Fuck reliability. Screw authentication.

by Jason Wilson on May 31, 2010

By Jason Wilson

I’ve written about this before, but Jacob Sayward’s recent article for AALL—iCite: Legal Research? There’s an app for that.—has me all kinds of pissed off. Not because he’s wrong or has misrepresented anything about the products he reviewed, but because his article is representative of reviewers of legal research products generally, and their unwillingness to question how accurate and reliable the digital content those publishers provide is, particularly statutory information. How hard is it to ask a publisher (if that is in fact what we consider apps developers to be) where they get statutes and the workflow processes they use to ensure the content’s reliability and authenticity. As lawyers and law librarians, did we all of a sudden not care about this? Did we wake up one day and say, “awwww shiiiiiit, I don’t care where it came from, as long as I can get it on my phone.” Are we so stupid that the only thing we care about is how wonderful and intuitive the GUI is? Because if that is true, we’re sunk for sure. And along with it, the profession.

[Image (cc) by Lomo-Cam]

______________________

Post Links:

Sayward, iCite: Legal Research? There’s an app for that.

Wilson, Screw authenticity. I just want it for free.

{ 5 comments }

Mikhail Koulikov June 1, 2010 at 7:08 pm

At this point, have there actually been any studies that have found outright errors in digital content provided by legal information publishers? I'm aware of at least a couple that take general scholarly databases to task for misrepresentation of content and incomplete content, but of course, these focus on coverage of journal articles, not statutes or laws.

(…plus, there's also that whole thing with collection development practices of evaluating databases along a number of different criteria. Reliability/authenticity is certainly one of them – but not the only one…)

jasnwilsn June 1, 2010 at 7:47 pm

To my knowledge, I have seen no studies concerning errors in "content," but am familiar with the types of studies you mention. And we need to be clear about content because to say "digital content" today is very redundant. Content is digital, output may or may not be. There is certainly a sufficient amount of anecdotal evidence suggesting that questions should be raised about standards for reliability and authentication, it's just not happening. If someone were to pick up one of my products and say "I wonder how they ensure the accuracy of the content included in this book," they can contact me. I'll tell them.

My concern is that many people are pumping out "platforms" for content taken from the web. I'm not suggesting they don't have processes in place for verifying the accuracy of that content, catching mistakes, pushing out updates, etc., it's just that no one is asking them about it.

@memathome June 3, 2010 at 2:54 pm

I recently wrote an article for LLAGNY Law Lines on I-Phone apps which will be published in July. I agree we as librarians need to start evaluating these products. I asked the developers the source of the content and most answered a government sponsored web site. I didn't publish that because the focus of my article was to let people know these apps exist. The reality is that there are mistakes in all platforms/media print, for-fee database, government websites, etc. When complaining about the quality of the NY State Criminal Law Handbook, a team of VPs from Lexis told us they make mistakes on purpose…………….

@memathome June 3, 2010 at 10:00 am

Both Lexis and Westlaw have warnings that they are not responsible for errors and omissions……………………..

jasnwilsn June 3, 2010 at 3:34 pm

Mistakes, while important, don't concern me as much as source and work-flow processes. If a vendor is spidering content from a government-sponsored site, I think reviewers should say that about the material. You can still rave about the functionality, but there should be some discussion (unless it is really obvious) about the authenticity, reliability, or correctness of the data.

Back in 2008, the Reporter of Decisions for California wrote a Viewpoint piece on how Lexis' editorial workflow (compared to Westlaw's) ensured that California lawyers were actually getting quicker access to the most "correct" version of California cases, by months. It's something of a revelation to read that because it can influence your purchasing decision.

This was a hot topic when Google Scholar with cases was announced. Where did they get their cases? How reliable is that data?

We can ask the same about all the apps that provide access, say, to the Federal Rules of Civil Procedure. How does the vendor know all the rules changes are included? Do they rely on the House version of the rules? Do apps include their own disclaimer that the information provided came from the House website? What about verification during the conversion process (final rules are in PDF)? How does (if at all) the vendor compare the versions to ensure fidelity?

We're so concerned with functionality that we miss what's under the hood. I can make a turd functional, but guess what? It's still smells.

{ 1 trackback }

Previous post:

Next post: