Virtual Vote of Confidence for TAR

The Text Retrieval Conference (TREC) Legal Track, also known as the TREC Legal Track, is a government sponsored project designed to assess the ability of information retrieval techniques to meet the needs of the legal profession.  On July 13, 2012, TREC released its 2011 study results.  The results have been referred to as “a virtual vote of confidence for technology-assisted review.”

Generally, the TREC 2011 Legal Track was concerned with the identification of responsive documents as part of the e-discovery process.  The participants’ objective was to identify as nearly as practicable all documents from a collection that were responsive to a request for production in civil litigation.  Likewise, they were to minimize the number of unresponsive documents identified.

Participants ranked an entire data set which consisted of 685,592 documents.  They would have to estimate the probability of responsiveness to each of three topics, and also to provide a quantitative estimate of that probability.

“[T]he results show that the technology-assisted review efforts of several participants achieve recall scores that are about as high as might reasonably be measured using current evaluation methodologies. These efforts require human review of only a fraction of the entire collection, with the consequence that they are far more cost-effective than manual review,” the report states.

The term “technology-assisted review” refers to “any semi-automated process in which a human codes documents as relevant or not, and the system uses that information to code or prioritize further documents,” said TREC co-leader Gordon Cormack, of the University of Waterloo. Its meaning is far wider than just the software method known as predictive coding, he noted. The overview of the 2011 TREC Legal Track can be found here.

Total Recall and Precision

Recall is the proportion of responsive documents retrieved by information retrieval methods such as  keyword searches.  Precision is the proportion of the retrieved documents that are responsive.  The goal of  a document review is high precision.  That means a smaller number of non-relevant documents will have been gathered in the data set that require review.

Lawyers commonly use keyword searches to identify the universe of potentially relevant documents.  Once the potentially relevant data set is identified, it is de-duped and culled to further reduce the amount of data that will actually be reviewed and coded.

This tried and true method of data identification and reduction is no longer sufficient, because the overall volume of ESI that must be considered for relevance is continuously growing.  According to e-Discovery experts like Magistrate Judge Andrew Peck and Ralph Losey, keyword searching is an outdated methodology for identifying  potentially relevant documents.

Software companies offer technologies that go beyond keyword searching.  These new offerings include concept searching, email threading, review analytics, pattern recognition technology and predictive coding.  The new solutions are poised to replace  keyword searching as the primary method for identifying relevant data.

Unfortunately, at this time, there is no specific case law that upholds or rejects the new technology.  Nonetheless attorneys will “be called upon to demonstrate to opposing parties, courts and government agencies, that its chosen method and tool accurately captured a reasonably sufficient number of the relevant, nonprivileged ESI in existence, and that the remaining unreviewed and unproduced ESI is irrelevant.”  Since there is no gold standard for identifying relevant electronic information for review, lawyers will have to take measures into their own hands to ensure the defensibility and quality of their searches when using these new technologies to reduce the amount of data that must be reviewed .

Here are some suggestions:

1. Allow the needs of the legal matter to dictate what technology is optimal for recall and precision.
2. Document every step of the discovery process and protocol.
3. Combine search solutions by integrating keyword and conceptual searches with other methods like predictive coding.
4. Employ some statistical sampling methodology to establish that unreviewed and unproduced ESI is not relevant.
5. Conduct detailed discussions about the ESI search and sampling protocols at the meet-and-confer conferences.
6. If the parties cannot agree on protocols, parties should not hesitate to ask courts to enforce such an agreement in the spirit of FRCP 1,which is “to secure the just, speedy, and inexpensive determination of every action and proceeding.”
7. Engage project managers and review specialist that understand human review, the new technology and techniques.

Quality Checking the Review

The Quality Check (QC) process should begin after a first pass review of the documents has been completed.  The QC team should consist of reviewers and project managers that have demonstrated a high level of understanding about the case and documents.  Although the QC process will be different for each case, there are some general steps that should be taken during every QC process.  Here is a quick QC workflow that can be used for any e-discovery matter:

1. Responsive and Not Privileged:  All documents tagged as “responsive” and “not privileged” should be checked to make sure that the set does not contain privileged documents.
2. Non-Responsive:  A small percentage, 10 to 15 percent, of the documents designated “non-responsive” should be reviewed to ensure that responsive documents are not being eliminated from the production.
3. Responsive and Redacted: These documents should be checked to confirm that the redaction is visible, proper and consistent on similar documents.
4. Emails and Attachments:  Check for inconsistencies and issues with the document coding of non-privileged attachments to privileged emails or non-responsive emails to responsive attachments.
5. Responsive and Privileged: These documents will receive a second look at the privilege log stage of the production.  Prior to that time, there is no need to review this set.

The percentage of the overall QC, whether 100 or 10 percent, depends upon the complexity of the data set.  So, the percentage will vary widely from one review to the next.  The most important point is to perform a QC and to do so early in the review process.

Clawback – Trick or Treat

Many litigators do not employ the clawback trick.  You just have to take a look at the circumstances surrounding J-M Manufacturing Company, Inc v. McDermott Will & Emery to bolster this point.  Now, this may be the case because the clawback provisions appear in the Federal Rules of Evidence, rather than the Federal Rules of Civil Procedure (FRCP).  Often, litigators do not focus on the FRCP because the Rules of Civil Procedure never prompt litigators to focus on them during the pretrial process.

Nonetheless, the Federal Rules of Evidence (FRE) 502 provides lawyers with a great deal of protection from inadvertent disclosures.  It states that the disclosure of attorney-client or work product information in a federal proceeding does not waive either privilege if:

(1) The disclosure was inadvertent;

(2) The holder of the privilege took reasonable steps to prevent the disclosure; and,

(3) The holder promptly took reasonable steps to rectify the error.

The FRE Advisory Committee Notes, explain that Rule 502 “contemplates enforcement of ‘claw-back’ and ‘quick peek’ arrangements as a way to avoid the excessive costs of pre-production review for privilege and work product.”  Plainly, under FRE 502, if a privileged document is inadvertently disclosed, the document MUST be returned.  Furthermore, if a federal court enters a clawback order, FRE 502(d) provides that the order can prevent the inadvertent disclosure from being a waiver not just between the parties to the agreement, but also “in any other Federal or State proceeding.”  So, parties should ask federal courts to enter clawback orders so that their clients have the broadest protection to the attorney-client and work product privileges.

Lawyers should stop missing this trick. They ought to add clawback provisions to their standard lists of items that they consider in every case. Someday, when you inadvertently disclose a privileged document, you’ll be glad that you put Rule 502 clawback provisions on your list of items to consider in every case.

Errors

A recent eDiscovery Errors Survey reveals that technology has not solved the challenges posed by electronically stored information (ESI).  In fact, technology is at the base of most of the mistakes that are made.   Lawyers and litigation support professionals were the subject of the survey to identify areas where their organizations experienced the most pain in terms of ESI.  The results were collected from across the USA, Europe and Australia.

Respondents were asked to classify the error as “frequently occurs”, “occasionally occurs”, “not very common” or “never occurs”.  Their responses revealed that the most common discovery error is related to the failure to effectively communicate across teams with key IT personnel and employees.  The survey exposed this as a frequent occurrence.  More specifically, 50% of the respondents indicated this as a major challenge.

Following closely, at 47% amongst the survey takers, was organizational failure to implement an adequate record retention plan. Failure to collect all pertinent data and badly thought out, or badly implemented, data retention policies were tied for discovery error creation.  Respondent’s stated that this occurred 40% of the time when discovery took place.  The other most common error during discovery was the failure to perform critical quality control like data sampling.  Missing data and spoliation accounted for the least amount of problems.  Interestingly, contract review teams were not listed as cause of common e-discovery errors.  Score one for the humans!