Month: March 2017


Advanced Discovery recently held a breakfast round-table with some law firms in San Diego. The idea was to meet with people and talk about what’s bothering them instead of me just being a talking head and going on and on about what I think is important.  Because really, nobody wants to come to a 7:00 AM meeting on a Friday to hear someone else talk: if they come to this meeting the have something on their mind that they want to discuss in a forum with their peers.

I started off the discussion by putting the topics I considered to be important on a flip board and then asking the folks in the room to tell me about their pain points.  The final result is captured below and its quite interesting.

My hot topics were:

  1. Collecting Social media
  2. Keyword searching: is it really dead?
  3. TAR/CAR/automated searching (or as I call it, SkyNet)
  4. Latest changes to the FRCP

The attendees were concerned with:

  1. Proportionality and costs
  2. Collecting off mobile devices
  3. Relevancy standards in the preservation process
  4. Review problems with social media collections: how to load and review
  5. Presentation/production of evidence

My immediate impression was that while I was talking in generalities that had some overlap with the attendees topics (the new rules, social media issues)  generally the law firms were much more focused on specific issues where I was talking about broad issues. That is, they were comfortable with the framework of the discussion but were technically savvy enough that they had very specific issues within that general framework that they wanted to talk through.

Cost issues were paramount within the general heading of proportionality.  The overall feeling was there was so much data being collected that narrowing it down to a manageable data set was problematic. The new rule changes were a welcome relief in this area by bringing proportionality front and center to help control not just costs but more importantly, the ability to focus on what was most important in their cases. Moving the matters forward in an efficient manner.

My focus on keyword searches was met with some feeling of skepticism but not because they felt TAR was not a useful technology. Rather they felt it still an up and coming technology whose value had not been proven.

Their objections were at several levels:

  1. They had not seen a clear definition of what TAR was and fell that the arguments about different tools, especially CAL, were too technical to convey to clients.
  2. They had not seen reliable case studies that showed them a repeatable work flow which implemented a verifiable process that they could use.
  3. They had not seen an extended case process which saved them money in a manner that their clients would accept. Although they understood that an early use of TAR could reduce costs downstream at the review stage, the immediate costs of TAR were so high as to exclude its use if keyword searches were still fairly reliable (per Judge Pecks discussion in the Hyles matter on the standard of reasonableness)

Likewise, their focus on social media was not so much on collection.  The feeling on that was “we hire forensics experts to collect things and we trust that they know what they’re doing.” Rather, they were concerned about two other aspects of the EDRM as they applied to social media: preservation and production.  These were specific “parts of the process” concerns: how do I properly preserve a dynamic ESI source and, once I do, how do I work with it, both in a review tool and in producing it.

And finally, a real focus on the rule changes was the issue of the standards of preservation. Once again, law firms were concerned with relevancy issue and very much liked the focus on the claims of the case.  Their emphasis was on practicality and the real nuts and bolts issues of using ESI in their cases.  Although they realized and in many cases were willing to discuss the larger philosophical issues, their real concern was the matter in front of them and how best to represent their clients in handing that case.

Next week I’ll take a look at the GC point of view with a report from the Houston version of the TGC Exchange which dealt with many of the same issues.

Predictive Coding: The Perfect is the Enemy of the Good

My late friend, Browning Marean, loved this saying and applied it often to ESI disputes.  Actually, he preferred two other versions of the sentiment, this one being made popular by Voltaire in the late 1700’s.  Browning loved to quote Shakespeare, and so he often said the line from King Lear, where the Duke of Albany warns that in “striving to better, oft we mar what’s well.”  And, when at his desk overlooking San Diego Bay and the US Navy vessels displayed there, he would simply revert to “close enough for government work.”

Well the saying came to mind this week when another old friend, ESI stalwart Phil Favro, mentioned a new case on predictive coding.  He noted that the case stood for the proposition that the Federal Rules of Civil Procedure don’t require a perfect discovery response, just one that results from the responding party making a “reasonable inquiry.”

Phil’s comment is at, referring to another tweet by Jonathan Swerdloff.  Swerdloff’s tweet cites a July 20th National Law Review article on a production dispute and subsequent Order by Judge Buch of the US Tax Court in the case of Dynamo Holdings, Ltd. vs. Commissioner, 143 T.C. No. 9 (2014).

But I want to dive a little deeper than the focus on “perfection,” because I think more than that is in play here.  This dispute began when the parties agreed to, and followed, a framework for producing the electronically stored information using predictive coding on documents being restored from two separate backup tapes.

The parties agreed that the taxpayers would run a search for terms, determined by the IRS, on the potentially relevant documents.  Petitioners conducted a Boolean search on the tape data for 76 search terms provided by the IRS, which were then supposed to lead to seed sets composed of documents containing hits from the Boolean search.  However, the parties subsequently agreed that petitioners would randomly select two sets of 1,000 documents from the tape data.

The taxpayers then produced samples of randomly selected documents from this set of potentially relevant documents, from which the IRS used selections to create a predictive coding model. The IRS also selected a “recall rate” of 95 percent.  Recall rate is the percentage of all relevant documents in the search universe that are retrieved by the search method.

But when the taxpayers delivered the documents retrieved, using this predictive coding model (minus privileged documents), it appeared that there were LESS documents than were identified in the initial keyword search.  The IRS filed a Motion to Compel Production of the “missing” documents.

In his Order on the Motion, Judge Buch went into a discussion of the arguments about why there was a difference in document count.  I won’t go into that here; you can read the full Order at

But what strikes me as most important, is his following discussion which begins at page 8 of the Order:

Likewise, “the Federal Rules of Civil Procedure do not require perfection.” Moore, 287 F.R.D. at 191.  Like the Tax Court Rules, the Federal Rule of Civil – 9 – Procedure 26(g) only requires a party to make a “reasonable inquiry” when making discovery responses.  The fact that a responding party uses predictive coding to respond to a request for production does not change the standard for measuring the completeness of the response.  Here, the words of Judge Peck, a leader in the area of e-discovery, are worth noting: “One point must be stressed – it is inappropriate to hold TAR [technology assisted review] to a higher standard than keywords or manual review. Doing so discourages parties from using TAR for fear of spending more in motion practice than the savings from using TAR for review.” Rio Tinto PLC v. Vale S.A., 306 F.R.D. 125, 129 (S.D.N.Y. 2015).

I’ve raised this point before at conferences and have been largely ignored.  However, doesn’t an ordinary reading of what Judge Peck says here lead tone to conclude that:

  1. We shouldn’t hold TAR to a higher standard than manual review
  2. Because TAR is faster than manual review, and
  3. Because the cost savings of what is faster is what is most important?

Well now I’m going to channel my inner Lee Corso, and because I hear him saying “Not so fast there, Terrible Tommy O’Connor.”  Isn’t the entire point of using TAR that it IS more accurate than manual review, and thus IS held to a higher standard?   In fact, right before his section in the Order on the myth of perfection, Judge Buch, at page 7, specifically states that holding up human review as the “gold standard” of review is a myth.

Are we to proceed then without discussing which method is more accurate, only because one is faster?  And thus, using the analysis of Judge Buch, speed of search is the determination of what is a “reasonable inquiry?”

I’m confused.  Even if TAR was NOT faster, if it has a higher standard of accuracy shouldn’t it be used?

I think what I’m reading here, is that Judge Buch is saying that the standard is “reasonable inquiry,” and Judge Peck already said that using TAR doesn’t introduce a new standard.  But doesn’t that open the door to say: “my expert will testify that my well-crafted keyword searches can get me 90% recall rate, and that is just as reasonable as TAR, so we should be able to do that rather than pay some expensive eDiscovery vendor to use their equally expensive software, just because its faster; that’s just not proportional, Your Honor”?

I hear the experts tuning up their dueling banjos and getting ready to go where angels dare not.