Predictive Coding: The Perfect is the Enemy of the Good

My late friend, Browning Marean, loved this saying and applied it often to ESI disputes.  Actually, he preferred two other versions of the sentiment, this one being made popular by Voltaire in the late 1700’s.  Browning loved to quote Shakespeare, and so he often said the line from King Lear, where the Duke of Albany warns that in “striving to better, oft we mar what’s well.”  And, when at his desk overlooking San Diego Bay and the US Navy vessels displayed there, he would simply revert to “close enough for government work.”

Well the saying came to mind this week when another old friend, ESI stalwart Phil Favro, mentioned a new case on predictive coding.  He noted that the case stood for the proposition that the Federal Rules of Civil Procedure don’t require a perfect discovery response, just one that results from the responding party making a “reasonable inquiry.”

Phil’s comment is at  https://twitter.com/PhilipFavro, referring to another tweet by Jonathan Swerdloff.  Swerdloff’s tweet cites a July 20th National Law Review article on a production dispute and subsequent Order by Judge Buch of the US Tax Court in the case of Dynamo Holdings, Ltd. vs. Commissioner, 143 T.C. No. 9 (2014).

But I want to dive a little deeper than the focus on “perfection,” because I think more than that is in play here.  This dispute began when the parties agreed to, and followed, a framework for producing the electronically stored information using predictive coding on documents being restored from two separate backup tapes.

The parties agreed that the taxpayers would run a search for terms, determined by the IRS, on the potentially relevant documents.  Petitioners conducted a Boolean search on the tape data for 76 search terms provided by the IRS, which were then supposed to lead to seed sets composed of documents containing hits from the Boolean search.  However, the parties subsequently agreed that petitioners would randomly select two sets of 1,000 documents from the tape data.

The taxpayers then produced samples of randomly selected documents from this set of potentially relevant documents, from which the IRS used selections to create a predictive coding model. The IRS also selected a “recall rate” of 95 percent.  Recall rate is the percentage of all relevant documents in the search universe that are retrieved by the search method.

But when the taxpayers delivered the documents retrieved, using this predictive coding model (minus privileged documents), it appeared that there were LESS documents than were identified in the initial keyword search.  The IRS filed a Motion to Compel Production of the “missing” documents.

In his Order on the Motion, Judge Buch went into a discussion of the arguments about why there was a difference in document count.  I won’t go into that here; you can read the full Order at https://www.mwe.com/~/media/files/thought-leadership/blogs/Dynamo-v-Commissioner-Decision.pdf?la=en.

But what strikes me as most important, is his following discussion which begins at page 8 of the Order:

Likewise, “the Federal Rules of Civil Procedure do not require perfection.” Moore, 287 F.R.D. at 191.  Like the Tax Court Rules, the Federal Rule of Civil – 9 – Procedure 26(g) only requires a party to make a “reasonable inquiry” when making discovery responses.  The fact that a responding party uses predictive coding to respond to a request for production does not change the standard for measuring the completeness of the response.  Here, the words of Judge Peck, a leader in the area of e-discovery, are worth noting: “One point must be stressed – it is inappropriate to hold TAR [technology assisted review] to a higher standard than keywords or manual review. Doing so discourages parties from using TAR for fear of spending more in motion practice than the savings from using TAR for review.” Rio Tinto PLC v. Vale S.A., 306 F.R.D. 125, 129 (S.D.N.Y. 2015).

I’ve raised this point before at conferences and have been largely ignored.  However, doesn’t an ordinary reading of what Judge Peck says here lead tone to conclude that:

  1. We shouldn’t hold TAR to a higher standard than manual review
  2. Because TAR is faster than manual review, and
  3. Because the cost savings of what is faster is what is most important?

Well now I’m going to channel my inner Lee Corso, and because I hear him saying “Not so fast there, Terrible Tommy O’Connor.”  Isn’t the entire point of using TAR that it IS more accurate than manual review, and thus IS held to a higher standard?   In fact, right before his section in the Order on the myth of perfection, Judge Buch, at page 7, specifically states that holding up human review as the “gold standard” of review is a myth.

Are we to proceed then without discussing which method is more accurate, only because one is faster?  And thus, using the analysis of Judge Buch, speed of search is the determination of what is a “reasonable inquiry?”

I’m confused.  Even if TAR was NOT faster, if it has a higher standard of accuracy shouldn’t it be used?

I think what I’m reading here, is that Judge Buch is saying that the standard is “reasonable inquiry,” and Judge Peck already said that using TAR doesn’t introduce a new standard.  But doesn’t that open the door to say: “my expert will testify that my well-crafted keyword searches can get me 90% recall rate, and that is just as reasonable as TAR, so we should be able to do that rather than pay some expensive eDiscovery vendor to use their equally expensive software, just because its faster; that’s just not proportional, Your Honor”?

I hear the experts tuning up their dueling banjos and getting ready to go where angels dare not.

Leave a Reply