JW: In his article, Does WestlawNext Really Change Everything: The Implications of WestlawNext on Legal Research, Professor Ronald E. Wheeler, Jr. states that given the more focused results that are retrieved within WestlawNext, the product’s pricing scheme discourages researchers from “opening, skimming, and reading portions of numerous documents.” According to Prof. Wheeler, this fundamentally changes our researching behavior because Westlaw Classic actually lets you browse portions of documents without being charged. How do you respond to this criticism and the other points made in the article?
MD: I’ve spoken with Professor Wheeler at length and found him to be very smart and sincerely interested in advancing the state of scholarship in the area of legal research. We’re also very interested in that and fully support what he and other academic professionals are trying to accomplish with articles like these.
Regarding your specific question, I think that for the minority of our customers for whom this pricing model could come into play at all, it’s certainly possible that this pricing discourages some browsing. But the same exact arguments for this theory could be used to claim that the pricing model encourages browsing, since in Westlaw Classic, each additional search costs more, so we often see researchers limiting the number of content sets they search in, which typically means that cases and statutes will be searched and many secondary sources will be ignored. Since WestlawNext aggregates multiple content sets under a single low search price, researchers should be more likely to browse content beyond cases and statutes – which is exactly what we see in our usage logs. In WestlawNext, secondary source usage as a percentage of total usage is up 50% over the ratios we see in Westlaw Classic. We think that’s a very good thing for researchers.
As a librarian, I always encouraged both students and associates to leverage secondary sources in their research – the right secondary source can literally save hours of time, and that time savings is a cost savings to clients. When I first came to West, I tried to promote the use of secondary sources within Westlaw, and our first project to accomplish that in a big way was ResultsPlus. With it, we saw a significant boost in secondary source usage, but WestlawNext takes it to a whole new level, making sure that relevant secondary sources are not just suggested on the side, but made an integral part of the main search result.
In addition, if you want to browse content like ALR or jury verdicts in Westlaw Classic, and they’re not included in your subscription plan, you must pay an out-of-plan cost to search them before you can browse anything – and you’ll have to try this with database after database. In WestlawNext, ALL searching is included within subscription plans (even if the underlying content is not), and we show more of a document preview in WestlawNext than we do in Westlaw Classic, so customers can browse previews of content outside their subscription plan all day long without incurring extra charges. This not only encourages browsing, but it tends to reduce out-of-plan costs. I heard from a large law firm librarian the other day who said her out-of-plan costs were down 67% in WestlawNext compared with Westlaw Classic. This makes sense, since with WestlawNext you can search repeatedly and browse content previews all you want, and you only pay for the out-of-plan document you click on. This will almost always cost much less than what it would have cost to search and browse in Westlaw Classic. And in WestlawNext, any search that turns up nothing costs you nothing (in terms of incremental charges), but in Westlaw Classic, if your search results in no relevant documents, you still pay an out-of-contract cost for the search.
So, for the minority of WestlawNext customers for whom the pricing model would apply, does the new pricing encourage or discourage browsing? I think it’s fair to say that, like Westlaw Classic and similar systems, it encourages some browsing and discourages other browsing. We think it’s really good that secondary source usage is up so much. That’s the sort of usage we’d like to encourage, as most researchers will be much better off if they incorporate secondary sources in their research.
You also referred to Professor Wheeler’s concern about “more focused results” in WestlawNext. He notes the importance of being able to run a broad search and then narrow down to specific documents, and he implies that the inability to do that could cause researchers to miss important documents. He then goes on to say that:
Retrieving an initially focused WestlawNext result will require researchers to rethink how they approach research. WestlawNext does not afford researchers the luxury of sifting through or focusing on a large and wide-ranging set of search results. It has no mechanism for executing an initially broad search.
To test his theory, he ran a natural language search in WestlawNext and got 317 cases. He then ran the same natural language search in Westlaw Classic and reports that he got 804 cases, and he emphasizes that this is two and half times more than WestlawNext.
But this can’t be.
In Westlaw Classic, we limit all natural language search results to 100. So, with this example, comparing a natural language search in Westlaw Classic with a natural language search in WestlawNext results in three times as many cases in WestlawNext – the opposite of what he claims is an issue. In other words, with these results, the argument would be that Westlaw Classic results are too narrow, and researchers should use WestlawNext to execute a broader search to sift through a wide-ranging set of search results.
This is not to say, “A ha! See! WestlawNext is better!” We do think that WestlawNext is substantially better, but not for this reason. We don’t necessarily agree with the notion that more search results are better. We’ve heard from customers repeatedly that they don’t like large search results, and that with large search results, they either immediately execute another search or just browse through the top documents. What we’re trying to do with WestSearch is maximize the recall of relevant documents and rank them as near the top as we can, while, at the same time, removing irrelevant documents or pushing them down to the bottom of the list. Often, we’re trying to make result lists smaller, not larger.
In Professor Wheeler’s defense, he doesn’t actually claim that the narrower search results (even though they’re actually broader in most cases) bring about worse research results. He simply claims that it might, and that it should be studied – so the mix-up in his reporting of the natural language results in Westlaw Classic is a very minor issue. It doesn’t change the basic premise that this is worthy of study. I agree with him on that and think that WestlawNext will be superior.
Regarding the other points of the article, one theory Professor Wheeler has is that WestlawNext will make it harder to find esoteric content because we employ usage patterns as a feature of our algorithms. However, we think that WestlawNext makes it easier to find esoteric content because of the sophisticated language associations we make with our algorithms and because users don’t have to know where the answer is before running a search.
To illustrate his point, he runs a search for a Georgia issue in all jurisdictions in both WestlawNext and Westlaw Classic and notes with concern that the right Georgia statute is buried at rank #9 in the WestlawNext-generated statutes list. When I was an academic law librarian teaching legal research labs to students, I’d always encourage them to choose the right jurisdictional content for their search – go to the California cases database if you’re looking for a California case. This wasn’t just about cost, it was about getting the right information in the most efficient manner possible.
I’ve known many librarians through the years who teach the same thing, and a review of our Westlaw server logs shows that the vast majority of researchers choose databases specific to a jurisdiction in Westlaw Classic and run searches specific to a jurisdiction in WestlawNext.
For this example, Professor Wheeler states that there are four statutes needed to understand the issue. When we choose Georgia as the jurisdiction, as most users would, and then run the search, the four statutes specified as relevant by Professor Wheeler come up 1 – 4 in the WestlawNext statutes list. We consider that an excellent result.
Ideally, examples used to illustrate the theories would be common use cases as opposed to rare ones.
Whether a common or uncommon example were used, it’s important to remember that Professor Wheeler selected it to be illustrative and not to prove the theory. As he says, the only way to figure it out is with empirical research. A single example alone indicates very little. We look forward to new empirical studies, and we’re glad to help if we can do it in a way that does not detract from the research with a perception of bias. I understand Professor Wheeler is interested in such research, and it looks like Professor Peoples is as well.
Considering the rest of Professor Wheeler’s article, we found a lot with which we agree with him – like the importance of knowing the structure of U.S. law and the differences in type and quality of publications, the usefulness of secondary sources generally, and the need for a change in the way legal research is taught now that WestlawNext is widely available. We wish him the best in his continuing pursuits, and we’re open to collaborating if it can be done without the perception of bias.
JW: Lately we’ve seen a very lengthy criticism of WestlawNext by an anonymous author over at LLB. Any response to that?
MD: Like any business, we’re interested in growing and becoming more successful, and also in helping our customers do the same. We’ve found the best way to do that is to stay very close to our customers and to continue to invest in product and service innovation to maintain market leadership. We know that one of the best ways we can improve is to invite criticism and to be open to new ideas about how our products could be better, so we appreciate sincere studies of our products, like those conducted by Professor Wheeler, and we’re happy to respond in constructive ways and work together to find new solutions.
But the stuff over on LLB wasn’t like that at all. It was nonsensical and clearly biased.
I asked my team to go through the author’s claims, to run searches, test theories, etc. We found it to be more ridden with errors and gaps in logic than any other blog post or article we’ve seen. We’re tempted to post a complete dissection of it online, but our sense is that engaging the author would only escalate the nonsense exponentially.
For those who are tempted to wade through it, I’d ask them to consider the obvious bias. The two posts are anonymous, very lengthy and extremely negative. By contrast, nearly all other published reviews of WestawNext list both positive and negative aspects of the system, and without exception the author is willing to stand behind his or her words by name. For instance, Professor Wheeler, obviously unbiased, said in his article
My own research experiences with WestlawNext seem to support the company’s claims. I have performed numerous searches in WestlawNext, pursuing answers to what I thought were challenging research questions, and the results have been remarkable.
A separate reviewer noted:
[T]hat new search algorithm West has outfitted WLN with really does improve your results. My query was on a very fine point of insurance law – I was having some difficulty even understanding the question, let alone formulating a tight search for an answer (after this many years in the biz, it takes something else entirely to make me scratch my head). So I entered my mostly unformed inquiry into the search box and, to my surprise, the very first hit was directly on point. I can only imagine what WLN would do with one of my familiar searches.
I could go on for pages with similar quotes, especially if you consider all of the positive Twitter commentary. Add to this that WestlawNext was selected as Product of the Year by the American Association of Law Libraries and just recently came out as number one in the New York Law Journal reader survey. Westlaw / WestlawNext was also shown to be preferred among attorneys nearly two-to-one in the recent ABA Technology Survey.
Given all this, how likely is it that an anonymously posted, lengthy, all-negative critique is unbiased?
Beyond that, I’d encourage readers to check the author’s claims themselves. When we checked the author’s examples, we found one where, when we entered the author’s suggested query terms, we got the exact right materials at the top of the list (as defined by the author). In most other cases, obvious and simple modifications to the queries resulted in highly relevant results (similar to the GA example we discussed earlier).
Our customers are finding real value in WestlawNext. For an excellent example of that, see: http://www.westlawnext.com/westlawnext/reducing-complexity-report.pdf
JW: Any final comments?
MD: To those of your readers who would like to advance the state of legal research or the quality of discussion in this area by publishing an article or blog post about products in this industry – whether our products or those of our competitors – I humbly offer the following suggestions:
- Start by considering what the researcher is trying to accomplish, and then analyze or test the product’s ability to deliver what the researcher needs.When we test our product’s capabilities in the area of legal research, we typically focus on three things: speed, accuracy, and quality and depth of understanding.In terms of researcher goals and behaviors, analyze, test, and consider a broad spectrum of actual researchers, including novice, average, and advance, as opposed to the very rare “ideal” researcher.This is like the advice from behavioral economists to consider actual human behavior over the mythical perfectly rational human. We build commercial products to solve real-world problems for our customers. These customers often behave in a way that is different than what they were taught in school.
Critics often urge us to design solutions for the way customers should think and behave, rather than for the way they actually do think and behave. Purely academic notions are interesting to us, but they’re a secondary consideration.
- In terms of product capabilities, compare the product analyzed to reasonable alternatives. The standard should not be perfection but improvement.When comparing alternatives, remember that outcome-based comparisons need to account for a wide variety of skill-levels, practice area experience, issue complexity, and problem familiarity. The best way to deal with this is to increase sample size.
- As much as possible, strive for apples-to-apples comparisons.It always amazes us when comparisons are so wildly off the mark. For instance, with Westlaw Classic, you have a certain toolset for legal research. With WestlawNext, you have the same toolset (except for rarely used minor things like fax delivery) plus a new set of power tools. It kills us when we have critics say things like, “with Westlaw Classic, I pounded a nail in with a hammer, and it worked perfectly, but with WestlawNext, I tried pounding the same type of nail in with the side of your new fancy power saw, and it didn’t work very well.”We’re thinking, why didn’t you just use the hammer in WestlawNext?
- If you’re going to publish a lengthy article or blog post about one of our products, let us fact-check it.We’re happy to do it, and there is no downside to doing so. We won’t tell you what you can or can’t say, we’ll simply give you the facts that we know and other things you might want to consider. We may also give you some of our own opinions and conclusions, but those can’t possibly hurt your article – you can ignore whatever you like.Professional journalists who publish in traditional print publications nearly always do this with us. Bloggers who are just posting on their blogs rarely do. We’re happy to assist the bloggers as much as we do the traditional journalists.
Finally, I can’t let all of this discussion of legal research and WestlawNext reach an end without acknowledging my friend and colleague Peter Jackson. WestlawNext and the WestSearch algorithm carry the indelible mark of his unique expertise. Peter was one of the world’s brightest lights in the field of artificial intelligence and machine learning. Beyond that, he was a great partner and a cool guy, and we are honored to have worked with him on such a remarkable project as WestlawNext. He is missed by his many friends and colleagues at Thomson Reuters.
Jason, I really appreciate this opportunity to share our point of view with your readers. I’ve been a fan of your blog for a long time, and given your interest and experience in legal publishing, this is a great place to discuss the issues. Thank you very much.
[You can find Mike’s responses to the comments and blog posts about this interview in his follow-up piece here.]