The Text Retrieval Conference (TREC) Legal Track, also known as the TREC Legal Track, is a government sponsored project designed to assess the ability of information retrieval techniques to meet the needs of the legal profession. On July 13, 2012, TREC released its 2011 study results. The results have been referred to as “a virtual vote of confidence for technology-assisted review.”
Generally, the TREC 2011 Legal Track was concerned with the identification of responsive documents as part of the e-discovery process. The participants’ objective was to identify as nearly as practicable all documents from a collection that were responsive to a request for production in civil litigation. Likewise, they were to minimize the number of unresponsive documents identified.
Participants ranked an entire data set which consisted of 685,592 documents. They would have to estimate the probability of responsiveness to each of three topics, and also to provide a quantitative estimate of that probability.
“[T]he results show that the technology-assisted review efforts of several participants achieve recall scores that are about as high as might reasonably be measured using current evaluation methodologies. These efforts require human review of only a fraction of the entire collection, with the consequence that they are far more cost-effective than manual review,” the report states.
The term “technology-assisted review” refers to “any semi-automated process in which a human codes documents as relevant or not, and the system uses that information to code or prioritize further documents,” said TREC co-leader Gordon Cormack, of the University of Waterloo. Its meaning is far wider than just the software method known as predictive coding, he noted. The overview of the 2011 TREC Legal Track can be found here.