Virtual Vote of Confidence for TAR

The Text Retrieval Conference (TREC) Legal Track, also known as the TREC Legal Track, is a government sponsored project designed to assess the ability of information retrieval techniques to meet the needs of the legal profession.  On July 13, 2012, TREC released its 2011 study results.  The results have been referred to as “a virtual vote of confidence for technology-assisted review.”

Generally, the TREC 2011 Legal Track was concerned with the identification of responsive documents as part of the e-discovery process.  The participants’ objective was to identify as nearly as practicable all documents from a collection that were responsive to a request for production in civil litigation.  Likewise, they were to minimize the number of unresponsive documents identified.

Participants ranked an entire data set which consisted of 685,592 documents.  They would have to estimate the probability of responsiveness to each of three topics, and also to provide a quantitative estimate of that probability.

“[T]he results show that the technology-assisted review efforts of several participants achieve recall scores that are about as high as might reasonably be measured using current evaluation methodologies. These efforts require human review of only a fraction of the entire collection, with the consequence that they are far more cost-effective than manual review,” the report states.

The term “technology-assisted review” refers to “any semi-automated process in which a human codes documents as relevant or not, and the system uses that information to code or prioritize further documents,” said TREC co-leader Gordon Cormack, of the University of Waterloo. Its meaning is far wider than just the software method known as predictive coding, he noted. The overview of the 2011 TREC Legal Track can be found here.

Total Recall and Precision

Recall is the proportion of responsive documents retrieved by information retrieval methods such as  keyword searches.  Precision is the proportion of the retrieved documents that are responsive.  The goal of  a document review is high precision.  That means a smaller number of non-relevant documents will have been gathered in the data set that require review.

Lawyers commonly use keyword searches to identify the universe of potentially relevant documents.  Once the potentially relevant data set is identified, it is de-duped and culled to further reduce the amount of data that will actually be reviewed and coded.

This tried and true method of data identification and reduction is no longer sufficient, because the overall volume of ESI that must be considered for relevance is continuously growing.  According to e-Discovery experts like Magistrate Judge Andrew Peck and Ralph Losey, keyword searching is an outdated methodology for identifying  potentially relevant documents.

Software companies offer technologies that go beyond keyword searching.  These new offerings include concept searching, email threading, review analytics, pattern recognition technology and predictive coding.  The new solutions are poised to replace  keyword searching as the primary method for identifying relevant data.

Unfortunately, at this time, there is no specific case law that upholds or rejects the new technology.  Nonetheless attorneys will “be called upon to demonstrate to opposing parties, courts and government agencies, that its chosen method and tool accurately captured a reasonably sufficient number of the relevant, nonprivileged ESI in existence, and that the remaining unreviewed and unproduced ESI is irrelevant.”  Since there is no gold standard for identifying relevant electronic information for review, lawyers will have to take measures into their own hands to ensure the defensibility and quality of their searches when using these new technologies to reduce the amount of data that must be reviewed .

Here are some suggestions:

1. Allow the needs of the legal matter to dictate what technology is optimal for recall and precision.
2. Document every step of the discovery process and protocol.
3. Combine search solutions by integrating keyword and conceptual searches with other methods like predictive coding.
4. Employ some statistical sampling methodology to establish that unreviewed and unproduced ESI is not relevant.
5. Conduct detailed discussions about the ESI search and sampling protocols at the meet-and-confer conferences.
6. If the parties cannot agree on protocols, parties should not hesitate to ask courts to enforce such an agreement in the spirit of FRCP 1,which is “to secure the just, speedy, and inexpensive determination of every action and proceeding.”
7. Engage project managers and review specialist that understand human review, the new technology and techniques.

Attorneys Ordered to Pay Vendor for Breach of a Document Review Contract

On January 5, 2012, Magistrate Judge Bernard Zimmerman signed an Amended Order ordering defendants to pay plaintiff $317,113.00, including costs and post judgment interest.  That Order followed a September 19, 2011 jury verdict against 15 defendants, who were a group of mass tort attorneys, that were sued by the technology vendor Cataphora for breach of a document review contract.

The underlying facts of this case center upon the problems with Chinese drywall that occurred around 2006. Homes throughout the Southeast were built with the Chinese drywall because there was a shortage of American drywall. The drywall was defective and caused several problems. For example, the drywall emitted fumes that smelled like sulfur or rotting eggs.  The fumes ruined metal appliances such as air-conditioning evaporator coils, pipes and copper wiring.

Multi-district litigation arose as a result of the defects.  A Plaintiff Steering Committee (“PSC”) consisting of the defendats was appointed by the court to handle the drywall cases.   In 2009, the PSC and Cataphora finalized negotiations for document review services.  Cataphora and Jerrold Seth Parker, the authorized representative for PSC, signed the contract. 

The contract involved a large upfront fixed fee in exchange for a lower rate throughout the rest of the 24-month work period.  Pursuant to that contract, Cataphora sent its first invoice for the upfront fixed fee of $366,000.  PSC refused to pay the initial sum.  Then, PSC canceled the contract with Cataphora.  After repeated attempts to collect the money due from PSC, Cataphora filed an action for breach of contract . 

In court, the defendants argued that they should not have to pay Cataphora because the contract was never valid. PSC argued that Cataphora included a success fee in the contract, which was illegal for non-attorneys to receive. They also claimed that the clause stating that the original fee was nonrefundable had been added at the last-minute without proper notification. Frank Pitre of Cotchett, Pitre & McCarthy, counsel for the defendants PSC, said Cataphora committed fraud by adding that particular clause so late in the game.

The four day trial for this matter began on September 12th, 2011.  Ultimately, the  jurors were not persuaded by PSC’s arguments.  They ruled in favor of the plaintiff’s Cataphora.  PSC filed several post trial motions.  After all the motions had been presented, the Magistrate Judge signed the amended order which echoed the jury’s verdict – the contract was indeed enforceable.

Court Will Not Compel Defendants to Share in the Costs of Obtaining Data

In Last Atlantis Capital LLC v. AGS Specialist Partners, Plaintiffs’ proposal that Defendants pay one-half of all costs incurred by Plaintiffs in pursuing third-party discovery was denied.  The Court had made the suggestion that “the Defendants aiding in half the costs of obtaining the audit trail data would be reasonable.” Relying on the Court’s suggestion, Plaintiffs submitted a Status Report.  The Report proposed that the Defendants should share in the costs of obtaining data from the Exchanges which Plaintiffs subpoenaed.

The Court allowed all Defendants to file a response to Plaintiffs’ Status Report and proposal.  Defendants rejected the proposal. They asserted that “there is neither reason nor precedent for requiring Defendants to share the costs incurred in connection with collecting data that Plaintiffs’ subpoenaed from third parties.”

Plaintiffs asserted that “Defendants’ need for the audit trail data is much greater than they claim.”  They also pointed to the Court’s prior suggestion, the Sedona Conference Commentary and several court opinions in order to prove their case against Defendants.

The Court found “this argument futile, for it is indeed the Plaintiffs’ burden to prove what they have alleged against the Defendants, and whatever costs are incurred in doing so, these costs must be borne by the party seeking the information-Plaintiffs.”  It opined that both the Sedona Commentary and case law relied upon by Plaintiff failed to support “the notion of a non-requesting, opposing party being required to aid a requesting party in funding its efforts to obtain third-party data.”  Moreover, the Court admitted that it had suggested that the ”proposal of cost sharing sounded ‘reasonable’. ” However,  the Court stated that this was “purely a suggestion for Defendants to take into consideration.”

The Court went further to acknowledge that ‘Defendants have substantially more resources than Plaintiffs’.  However, unfortunately for Plaintiffs, the Court did not consider this fact.  In the end, the Court determined that it would not “force the Defendants to pay for the evidence that Plaintiffs need in order to prove their case against Defendant.”

E-Discovery Lessons for 2011

Legal teams and judges faced a complex and ever-changing E-discovery landscape in 2011.  Here are some of the trends I noted throughout the year:

(1) In the midst of the trend of outsourcing e-discovery, a consistent and robust QC workflow is crucial. Performing the QC  ensures attorneys are accurately reviewing outsourced work and protects clients from future liability.   In J-M Manufacturing Company, Inc v. McDermott Will & Emery, J-M Manufacturing  sued its legal counsel, McDermott Will & Emery for e-discovery legal malpractice, on the grounds that they produced 3,900 privileged documents in a pending lawsuit.  Given the chance of future liability for producing privileged information to the other side, it makes sense to perform QC during all the phases of the electronic discovery process from initiation to completion.

(2) Lawyers should add claw back provisions to their standard lists of items that they consider in every case. Someday, when you inadvertently disclose a privileged document, you’ll be glad that you put Rule 502 claw back provisions on your list of items to consider in every case.  See Claw back – Trick or Treat, posted November 2, 2011.

(3) Counsel should not agree to search terms without knowing the sources of potentially relevant information.  For example, in I-Med Pharma, Inc. v. Biomatrix, Inc. No. 03-3677 (DRD), 2011 6140658 (D.N.J. Dec 9, 2011) the court affirmed the order of the Magistrate Judge excusing Plaintiff from the obligation of reviewing and producing millions of pages of documents recovered from unallocated space files.  The search terms agreed upon by the parties were not targeted to document custodians, relevant time periods or databases.  For these reasons, the Court found the burden of the review would outweigh any potential benefit and likelihood of finding relevant admissible evidence.

(4) Attorneys should be specific when making discovery requests.  In Dartnell Enterprises, Inc. v Hewlett-Packard Co., the Defendant Hewlett-Packard was ordered to produce responsive documents in their native electronic  format despite its production of hard copies of those documents.  The court made this determination because the disclosure requests by Plaintiff specifically requested that ESI be provided as part of the production.   The Defendant did not present any valid basis or reason for not complying with the disclosure requests in an electronic format.  Consequently, the Defendant was required to produce the requested documents in their native electronic format, including any metadata as well as a an index to the electronic files identifying the document produced in response to each demand.

(5) In 2011, the majority of the reported judicial opinions addressing electronic discovery focused upon sanctions.  In those cases, the sanctions were issued for preservation and spoliation issues.  The second most opined upon issue in e-discovery was cases addressing various procedural issues like searching protocol and cooperation.

I am looking forward to finding out the trends and issues in E-Discovery in 2012.  Keep reading!!!!