Electronic Data Records Law | How to Win E-Discovery

Electronic Mail Storage In-House or In the Cloud

E-mail Archive Economics

How expensive is it to store many terabytes of enterprise email so they are available for survey, eDiscovery, investigations, internal control and purposes of institutional memory?  The raw cost of data storage is dropping, so is it not cheap to retain lots of email?

I explored this topic in an email exchange with Dennis Vlasich, Director of the IT Department at the municipality of Fontana, California.  Following is our conversation.  

I started by pointing Dennis to my earlier post suggesting "content addressable storage" for inexpensive retention of large volumes of e-mail.  

DENNIS replied: “It [the earlier post] is basically a standard storage vendor's sales pitch and we have a SAN that can accommodate terabytes of ‘cheap’ storage.  You have to factor in the rack space, power and HVAC requirements to keep the storage system running and I can assure you that because such technology [content addressable storage] is less expensive, its MTBF [mean time between failures] will be much shorter than other forms of storage, so what happens when it crashes?  The bottom line is cost versus benefit versus risk.”  [Update:  Greg Smith of Messaging Architects responds to Dennis' comment.]

To which BEN said: “As a lawyer, here's what I don't understand.  I use Yahoo as my business email provider.  I pay Yahoo a paltry $20 a year for this email service, and Yahoo provides me unlimited storage.  Unlimited.  And since I pay for my service, Yahoo does not attach ads to my email. I store on Yahoo all my business e-mail (including attachments) dating back to 2004.  (I also back it up in zip files I download onto a local hard drive.)  The economics suggest that storage costs -- including all the infrastructure and management -- are just not a big deal to Yahoo.  Why can't an in-house IT department at a mid-sized enterprise get anywhere close to the same economics that Yahoo (or AOL or Hotmail) gets?”

Then DENNIS said: “Yahoo probably restricts the size of your attachments to 1MB and your mail client (probably not Outlook) does not have the search and archiving capabilities of an Exchange/Outlook environment.  I don't know what your SLA [service level agreement] is with them, but I would guess they aren't guaranteeing that all your email will be available forever.

"The City of Los Angeles just switched from an in-house mail system (using Groupwise) to Google's gmail for 30,000 users.  Google charges $50 per year per user for up to 250 gigabyte of disk space, however they do not have any archiving capabilities yet.  

“Google, Yahoo, and other large providers are using data center technologies on a massive scale that brings the cost per MB down below what those of us in smaller shops can come even close to. The big problem they have is security.  Google had to guarantee LA to use dedicated servers and storage for all their email so it wouldn't co-mingle with their other customers, and for $7.5MM Google said OK.  Most of the rest of us cannot negotiate on that scale.

“The administration overhead for managed disk space is quite high because it requires routine backups with take time and resources which all cost money.”

BEN responded: “My attachments, both incoming and outgoing, are limited to 20MB, which is reasonably large.  The mail client I use is Yahoo's web mail page.  It provides quite a few search capabilities.  I have my 'archives' arranged as large chronological zip folders on my local hard drive. Still, Yahoo stores a full copy of all my emails back to 2004.”

DENNIS: “[Regarding the 1MB attachment limit] I guess I was thinking of the free version [of Yahoo mail].  It's been a while since I looked at those kinds of mail services. We're looking seriously at Google Apps with gmail (waiting to see how things work for LA).  The 250GB limit might be a problem for some of our users, but I figure by the time we're ready to move, there will be more space available or they will have an archiving feature.”

Update July 2010:  The General Services Administration has certified Google Apps as qualified for use by government agencies.  Typically, a US government implementation of Google Apps would keep data on servers in the continental US.

–-Benjamin Wright.    A senior instructor at the SANS Institute, Mr. Wright regularly serves as a public speaker at lunch and dinner meetings for local professional groups, such as IIA and ARMA.

FOIA Meets Electronic Mail

Populist Politics

Publish Internal Records on the Web?

Freedom of Information Act requests will eventually motivate governments will publish on the web large portions of their official records, including the emails and text messages of officials and administrators.  Federal, state and local governments are being swamped with FOIA requests for email records of officials like county commissioners.

As more and more of these records are released to the public, people publish them on the web.  AnnArbor.com, a news organization, has assembled and published more than 2700 pages of emails of the city council members for the municipality of Ann Arbor, Michigan.  The records cover messages exchanged among members during the council's public meetings.  The municipality had released the e-mail records, piecemeal, on paper in response to numerous FOIA demands over time.  The news outlet then gathered them from the various requesters, scanned them into digital form, organized them and published them on the web.  As a consequence government became more transparent in that city.

As technology advances, and as populist activists make more use of FOIA, it will become easier and more common to find and assemble records like these on the open Web.   Ever-greater transparency in public administration is an inexorable force in the 21st century.

Wise government leaders will embrace radical transparency.  Instead of dribbling records out in small units, they will jump straight to the endgame and publish an ever-growing swath of their internal records and communications directly on the web and make them easily searchable.  

Meanwhile many politicians and managers will live in denial and fail to understand how technology is changing society.  The Ann Arbor city council, for instance, explicitly decided in October not to publish on the web a database of emails similar to the one that annarbor.com compiled and published.

–Benjamin Wright

Mr. Wright teaches eDiscovery and eData Security Law at the SANS Institute, where he stresses the growing role of web publicity. 

City Government | Deleted Official E-mails

Expensive Forensic Recovery | Political Cover-up? | State Retention Schedule

Compliance with e-mail retention law is not easy.  Email in an enterprise like a municipal government is voluminous.  And at city hall failure to keep required electronic mail records can look like a scandal, where it appears officials are trying to cover-up the truth about something important.

Hence, the City of Boston has hired a forensic specialist to attempt to recover the destroyed e-mails of an aide to the mayor.  As part of the specialist's investigation, he or she might look on PC hard drives and in network backup tapes to locate erased records.

Forensic retrieval of email is costly.  The expense of forensics quickly exceeds whatever the city might have been saving by not implementing a comprehensive email archival system in the first place.  Had the city foreseen it would eventually be forced to restore deleted e-mails, it would have readily seen the need for good archival as a matter of good IT management.

The city did have an archival program, but it only pertained to emails that had resided in a user’s inbox for more than 90 days.  In other words, the program gave decision-making responsibility to users.  If users decided to delete records before 90 days, then the records would not be archived.  Among the problems with giving users such decision power is that users are rarely qualified (and rarely have the time) to exercise the required judgment.  Further, a user has a conflict of interest.  If a record reflects negatively on the user, she naturally prefers to destroy it.

Generally speaking, the Massachusetts state record retention schedules applicable to city government require that email correspondence of “no informational or evidential value” be retained a minimum of two years. 

But just instituting a policy that all e-mail be kept two years does not solve all of the city’s record retention problem.  Many e-mails must be retained for various longer periods, depending on content.  To examine email content and determine that this record needs X retention period and that record needs Y retention period is very, very difficult in practice.  In fact, I am skeptical that such one-by-one examination can be done effectively in government across many users over an extended period of time. (If you dear reader know of any case study showing it can be done, I cordially invite you to point me to that study!)

Under the auspices of Messaging Architects, I led a workshop on this topic with the stakeholders (IT, HR, legal department et al) in Jackson County, Oregon, February 2009. The conclusion was that the county government would archive all e-mail (spam excluded) more or less indefinitely.

–Benjamin Wright, Senior SANS Instructor for e-Discovery and e-Record Retention Law

Data Loss Prevention Strategy

Transparency as Compliance with Data Security Regulation

How are data holders to comply with the swelling riot of data security laws?  These laws include breach notification laws, which require that individuals and/or government be notified when the security of private data has been compromised.  Perfect compliance is impossible.

Almost all the states have adopted breach notice laws – though they are not uniform – and legislatures are expanding the scope of the laws.  

The original law from California (effective six years ago) focused on identity information – name plus social security number, driver’s license number or financial account number.  Then the California legislature expanded its law to also include breaches of medical data. That expansion became effective January 1, 2009.  Result?  In the first five months of 2009, California authorities were notified of a whopping 823 healthcare data breaches, mostly through self-reporting by healthcare entities. That’s just one industry, in one state, in five months.  And California authorities anticipate that the flow of notices will rise as people in the healthcare community become better aware of the new law. 

A data breach can occur in myriad ways:  a misdirected fax or e-mail, a hacking incident or snooping by an employee.

Meanwhile, we see floods of breach notices issued in other states and other industries – retail, nonprofit, colleges, financial, professional services, water districts, school districts, county government, municipal government, state government, federal government.  No organization is immune.  As these laws sit longer on the books, the flow of notices grows larger and larger.

Why are there so many notices?  The reason is that the laws assume that 100% data privacy can reasonably be achieved.  They further assume that any shortcoming (or suspected shortcoming) of data privacy should be an unusual event within well-managed enterprises.  The assumptions are wrong.  
The expectations of the public, and especially the expectations of policy makers, are out of touch with the reality of modern data management.  These outsized expectations contribute to a growing risk of monetary liability on the part of data holders.

So what are data holders to do?  Obviously they need to invest in data security, training and investigation – and they have been.  But banks, schools, utilities, clinics, hospitals, merchants and government agencies can throw massive investments at this problem and never meet present expectations.