Are You Exposed? Lock Data Down Now

If you work at a fair-sized company, chances are good that you use something called an enterprise content management system (ECM). This incredibly vital system stores data, lets people throughout the company collaborate on documents, and keeps track of what's happened to what data. In the language of IT, it creates metadata—data about data—such as audit trails, workflows, clinical trial dates, e-submission details, digital signatures, and other critical information.

Mike Fernandes

You can use this system for a lot of important functions: tracking every stage of the drug discovery and development process, preparing electronic submissions, tracking your business and making decisions based on real-time information, and ensuring that you're in compliance with FDA's 21 CFR Part 11 guidelines and the requirements of Sarbanes-Oxley, to name just a few.

If you lose what's in your ECM, you're in big trouble.

The great rule of computing, of course, is to back everything up, so that when a glitch takes place you can recover your information. There are a variety of systems for doing this sort of backup and recovery, so if you go to your IT team and ask whether your ECM system is fully protected, they'll probably tell you yes.

Unfortunately, it's often not quite that simple. In fact, many pharma companies today needlessly expose themselves to data losses that could result in clinical studies being placed on hold, withdrawn approval of initial new drug applications, and in extreme cases, criminal prosecution. The culprit: backup and recovery strategy.

Understanding Information Loss

It's important to understand that there are two different types of data loss, and two different kinds of information you can lose.

There are two types of data loss.

Full-system failures are the type most of us think of when we think about data loss. A catastrophe strikes, and everything on your system is destroyed. All organizations are aware of the potentially devastating impact of full-system failures and disasters, and most make a good faith effort to plan for recovery. Total system failure is, blessedly, relatively rare.

Be Prepared for the Worst

Partial information loss, the lesser known but more sinister form of loss, is quite another matter. A partial loss affects one, several, or thousands of pieces of information in an ECM system, rather than the entire system. It is caused by common, everyday incidents such as accidental user deletions, programmatic errors, malfeasance, corruptions, and viruses.

According to AIIM International and Strategic Research, events of this sort account for more than 80 percent of all ECM system information loss. And the results can be devastating, creating a crisis for both the IT and business sides of the company.

Take the case of a Fortune 200 pharmaceutical manufacturer's experience, when an IT administrator accidentally overwrote a folder that was linked to 15,000 other files containing more than 100,000 links to other content and metadata. Many of the files were documents necessary to support the company's current good manufacturing process guidelines, and their loss caused compliance risk with these and other FDA regulations. Manufacturing was shut down as the company scrambled to recover the documents, halting the production of drugs worth millions of dollars in revenue.

The company could have taken the ECM system offline and recovered the information using a traditional "cold" backup. But that wasn't a viable option; additions and changes made to the

system since backup was created almost a full day prior would have been permanently lost.

The company lost more than $100,000 in productivity, spending over 1,100 hours recovering content from backup tapes and re-creating employee work—but no original metadata could be recovered.

The loss of metadata is particularly damaging. If the relationships between content (documents) and their associated metadata are broken, the ability to quickly respond to business demands, audits, and e-discovery requirements (as well as complying with 21 CFR Part 11) becomes severely impaired or even impossible. Worse, on the drug development side, if you break the link between content and metadata, you've broken the chain of custody of your data. This can delay drug submissions or halt manufacturing, which can cost millions of dollars in lost revenue, brand exposure and market share.

In the case of this particular company, subsequent reviews of help desk incidents indicated that there had been 46 partial data loss incidents over a three-year period—more than one a month. All incidents had resulted in the permanent loss of metadata, and when recovery of content was attempted, the effort either took several days or was abandoned completely.

System Protection Now

To ensure that you are protected from all forms of information loss, it is essential that you have a strategy that

encompasses both recovery from full-system failures and disasters, and recovery from partial information loss incidents.

Enterprise backup and recovery solutions are an integral part of any recovery strategy. They're ideal for providing long term accessibility to data through archiving, and for recovering from those situations where the entire ECM system is down due to a full-

system failure or disaster. However, they're not ideal for restoring information lost or corrupted due to partial information loss incidents, because they require that the entire system be brought down and rolled back to the last known good backup. This takes employees offline, and causes the loss of any changes to the ECM system since the backup was created.

Continuous data processing, replication, and mirroring are solutions that provide various forms of "always-on" backup that capture changes as they are made, creating a snapshot at the point in time data is modified in a secondary location. Because these methods capture changes frequently or even as they happen, recovery typically doesn't require a rollback. In cases of partial data loss, however, these methods can compromise the integrity of documents and their related metadata because the loss is copied to the secondary location. If data is corrupted, damaged, or deleted in the ECM system, it will be in the same state in the secondary copy.

Granular, ECM-specific recovery solutions ensure that in the event of a partial information loss incident, only the lost or corrupted information can quickly be restored directly back into the ECM system in its original state, with no application downtime and minimal resources. In order to accomplish this, the solution must validate the integrity of the content and metadata when it's captured, and ensure that any corrupt information is rejected and flagged for IT, so the issues can be rectified before the information is needed to meet business demands, for an audit, or to comply with litigation processes.

In addition, granular recovery solutions can complement your existing traditional recovery solutions by significantly reducing the data loss window in the event of a full-system failure.

Micromanage Your Backup

Pharmaceutical companies expend significant time and resources implementing ECM systems in order to streamline the processes and workflows associated with the drug development and FDA e-submissions process.

Implementing a comprehensive backup and recovery strategy for your ECM system ensures that the complex information that is your company's stock in trade will never be compromised.

Mike Fernandes is director, product management and services, CYA Technologies. He can be reached at mfernandes@cya.com