The R2K lawsuit: market corrections and scalp-taking

Markos Moulitsas of DailyKos announced Monday that he is suing Research 2000 for fraudulant activity, based on a statistical analysis conducted by Mark Grebner, Michael Weissman, and Jonathan Weissman.  I won’t comment on the details of their study here — Nate Silver has done a much better job of that already — but instead want to make a broader comment about the internet, markets, and “scalp-taking.”

I’ll note as a caveat that Research 2000 is launching a counter-suit.  The facts will be revealed in time, and the way things look today may not turn out to be the reality of the situation.  I don’t mean this blog entry to prejudge the results of this trial.

That said, it appears as though the progressive political blogosphere has just claimed a second scalp within the polling industry.  The first occurred back in the fall of 2009, when Nate Silver at 538 raised serious concerns about Strategic Vision.  Noticing serious anomalies in their data, as well as a lack of public information about the company itself, Silver asked some very public questions about whether they were fabricating their data.  The head of Strategic Vision cried foul and claimed he’d see Nate in court, but he then beat a hasty retreat and hasn’t been heard from since.

DailyKos has contracted with R2K since the 2008 election cycle, and has sent them a lot of business.  After Nate published his inaugural pollster rankings last month, Markos announced that he’d be rethinking the partnership with R2K (who fared poorly compared to other pollsters).  That apparently led to a few statisticians deciding to take a deeper look at R2K’s numbers, which revealed anomalies that would be consistent with mild cooking of the books and/or outright fraud.

Talking Points Memo took a deeper look at the head of R2K, Del Ali, and found that his background consists of 2 degrees in recreation.  That’s really pretty odd, to say the least.  You would expect the head of a major polling firm to have a background in, well, statistics.

And that leads us to the point I’d like to make: how is this possible?  Professional polling is a competitive and lucrative business, with longstanding industry leaders and standard-setting organizations.  Neither Strategic Vision nor R2K was a minor player — both were significant pollsters whose findings were reported by mainstream media sources.  Both (it appears) were somewhere between shady and fraudulent.  In a well-functioning market, incentives should exist for shaming and discrediting such actors.  The field of professional polling involves enough statistical wizardry and high enough stakes that, if such incentives operate anywhere, they should operate there.  And yet we now have seen two occasions in which, essentially because Nate Silver and company have made a hobby of advocating for responsible polling practices, major irregularities have been uncovered, with field-transforming impacts.

There’s a lesson here about just how robust the market mechanisms in various knowledge industries actually are.  Even in a field that has incentives for self-policing, even in a field tied to academic institutions like AAPOR that are full of people who have the means and motive to investigate such irregularities, there has been a distinct lack of accountability for years.  The lowered transaction costs of the internet has enabled skilled hobbyists to dramatically affect that market.  The internet itself doesn’t magically improve the polling industry (far from it), but it did create a new opportunity structure through which motivated volunteers could challenge and affect existing institutions.

Bravo to Nate Silver for his nearly one-man quest to improve the polling industry.  He didn’t have to take on this challenge, and I’m sure it’s made him plenty of enemies in the process.  Kudos to Markos Moulitsas as well for partnering with statistical researchers and readily admitting it once he learned there was a problem with his data.  The internet doesn’t make the industries perform more responsibly, it just creates new opportunities for motivated outsiders to mobilize knowledge/people/resources in new and interesting ways.  Between R2K and Strategic Vision, we have a good example of just how poorly the “statistical wizardry” industry was actually functioning, and also a case study in how networked volunteers can transform such industries.

As the old proverb goes, “may you live in interesting times…” Interesting times, indeed.

2 thoughts on “The R2K lawsuit: market corrections and scalp-taking

  1. Given these developments, wouldn’t it make some sense to form an independent body to continually run metanalyses of polling data? Community effort among non-pollster professional statisticians, stats and poli-sci grad students and post-docs as a professional responsibility kin to reviewing. It’d be both a good way to gain experience with the data and a way to assure continued confidence in the most public manifestation of the profession.

  2. Yeah, it would make a lot of sense. Honestly, one of the surprises in this sequence of events has been learning that such an independent body doesn’t *already* exist. Such analyses could easily become publication-worthy in their own right, I’d think, meaning there’s both a social value, a learning value, and a professional-advancement value.

    I’m not looped into the AAPOR circles, but it seems like this would be right in their wheelhouse…

Comments are closed.