Electronic Data Records Law | How to Win E-Discovery

Archive Storage Costs

Cloud? Content Addressable Storage?

Is it too expensive for an enterprise to store lots of email for many years? On this blog I have been arguing that the law motivates enterprises to keep email generously.  But generous retention (many records many years) raises questions about cost. 

Earlier on this blog, I have aired arguments about whether cloud computing or content addressable storage (creates an in-house cloud) are effective technologies for archiving email economically.

One commenter argued that the alleged allure of content addressable storage (CAS) is just vendor hype.  
To that, my friend Greg Smith at Messaging Architects responds:

“It is true that the cost of data centers, power, rack space and environmental conditions is a big cost to take into account when evaluating storage costs.   But too often the thought when organizations are looking at storing terra bytes of data is to look at traditional storage mechanisms such as a SAN.  Although a SAN is more economically feasible than local storage, it still requires the overhead of redundant drives for data protection . . . so that in effect some systems running a RAID 6 environment with an 8 drive array have a 25% operating overhead.  

“Contrast content addressable storage designed for secure, in-house, long term storage. Since it does not employ a traditional file system, it does not have the limitations or constraints of a traditional system. In fact CAS systems, which can provide storage for under $3 / GB, are designed to replicate and validate data to ensure that:

* corruption and disk failure do not destroy data, and

* in the event of hardware failure, replacement of hardware is not the lengthy rebuild process that would apply to a SAN.

“How does CAS compare with storage in an external cloud provider?  External cloud storage costs are low, and it is difficult for any form of in-house storage to compete with those low costs.  However, what remains to be seen is whether the accessibility and security of data in the external cloud will be satisfactory.  Let’s not forget that a cloud looks solid enough from the ground, but if you fly through it, it becomes translucent.

“Sometimes being able to physically touch something provides more assurance of its existence and accessibility.  Still, that perception may change over time.

“On the subject of external cloud storage, one other question is data ransom.  After you have stored many terra bytes of valuable data in a vendor’s cloud, the vendor will see that it can raise prices. When it does that, there is no choice but to pay these higher ongoing prices, as the cost to move that data in-house or to another provider may be prohibitive.”

–Benjamin Wright

How to store e-mail long term?

Is Archival Affordable?

As a mere lawyer, I struggle to understand from a technical perspective how to implement e-mail retention policy in a larger organization across many years.  Legally, the organization may – due to litigation hold or other factors – have incentive to store some (maybe a lot) of email a long time, more than five or seven years.

Legally speaking, the biggest mistake with e-mail records is to destroy them too early.  That could be spoliation.  So why not delay destruction a long time?

In theory long-term storage of much e-mail should not be expensive because the cost of raw storage is dropping.  However, IT folks in my SANS courses argue that the cost of storage media is not the key issue.  They argue the overhead associated with massive storage is high (they bring up things like expensive disk arrays).

City Government | Deleted Official E-mails

Expensive Forensic Recovery | Political Cover-up? | State Retention Schedule

Compliance with e-mail retention law is not easy.  Email in an enterprise like a municipal government is voluminous.  And at city hall failure to keep required electronic mail records can look like a scandal, where it appears officials are trying to cover-up the truth about something important.

Hence, the City of Boston has hired a forensic specialist to attempt to recover the destroyed e-mails of an aide to the mayor.  As part of the specialist's investigation, he or she might look on PC hard drives and in network backup tapes to locate erased records.

Forensic retrieval of email is costly.  The expense of forensics quickly exceeds whatever the city might have been saving by not implementing a comprehensive email archival system in the first place.  Had the city foreseen it would eventually be forced to restore deleted e-mails, it would have readily seen the need for good archival as a matter of good IT management.

The city did have an archival program, but it only pertained to emails that had resided in a user’s inbox for more than 90 days.  In other words, the program gave decision-making responsibility to users.  If users decided to delete records before 90 days, then the records would not be archived.  Among the problems with giving users such decision power is that users are rarely qualified (and rarely have the time) to exercise the required judgment.  Further, a user has a conflict of interest.  If a record reflects negatively on the user, she naturally prefers to destroy it.

Generally speaking, the Massachusetts state record retention schedules applicable to city government require that email correspondence of “no informational or evidential value” be retained a minimum of two years. 

But just instituting a policy that all e-mail be kept two years does not solve all of the city’s record retention problem.  Many e-mails must be retained for various longer periods, depending on content.  To examine email content and determine that this record needs X retention period and that record needs Y retention period is very, very difficult in practice.  In fact, I am skeptical that such one-by-one examination can be done effectively in government across many users over an extended period of time. (If you dear reader know of any case study showing it can be done, I cordially invite you to point me to that study!)

Under the auspices of Messaging Architects, I led a workshop on this topic with the stakeholders (IT, HR, legal department et al) in Jackson County, Oregon, February 2009. The conclusion was that the county government would archive all e-mail (spam excluded) more or less indefinitely.

–Benjamin Wright, Senior SANS Instructor for e-Discovery and e-Record Retention Law