The Analytics Floor

There was an interesting article in Politico yesterday, titled [gulp] “Honey, I shrunk the Obama data machine.”*  The article discusses next steps for the Democratic data machine in the leadup to the 2013 and 2014 elections.  The big question: can the Obama analytics tools translate to the state and congressional levels?

The answer (to paraphrase): “yes, but only some of them.”

When people talk about the #Demdata advantage in campaigns, they’re really talking about (at least) three distinct phenomena.  Two translate well to smaller campaigns, the third doesn’t.  The dividing line is something that I call the analytics floor.

(1) One of the biggest advantages Democrats hold over Republicans is the rich voter file that Democrats have developed.  Republicans are working to build their own national database, to sometimes-comedic ends.  That voter file can be exported to congressional campaigns, special elections, governor’s races, etc.  OFA alumni like Dan Wagner of Civis Analytics specialize in just this sort of data modeling.  Obama invested millions in developing the voter file and built a network of hundreds of experts in combining the voter file with polling data to produce much clearer maps of the electorate.  As those experts turn to consulting and expand their reach outward, the price of these services will become more affordable over time.

 

(2) A second advantage comes in the form of lessons learned through persuasion and turnout experiments.  The Analyst Institute was very busy during the 2012 election cycle, running tests to determine what sort of techniques and appeals can best sway undecided voters and motivate disinterested supporters.  These lessons in political behavior are transportable from one election to another — if they’ve determined that voter “report cards” drive people to the polls, that’s a lesson that can improve off-year elections as well.  Democrats have invested in cutting-edge social science, and are in no rush to share their findings with Republican competitors.  This advantage will echo into 2014 and beyond.

(3) The third facet of #DemData is what Daniel Kreiss calls “computational management.”  Computational management refers to the day-to-day role that analytics can play in campaign management, and the “culture of testing” it promotes.  The Obama campaign tested everything.  It tested e-mail subject lines.  It tested font sizes.  It tested niche television spots.  Data settled arguments and maximized investments.  Here’s where things get dicey.

Day-to-day inputs aren’t going to be available and/or useful to smaller campaigns the way they were to the Obama campaign.  If you’re running a mayoral race in Hartford, there will be one or two polls conducted *at most*, and they’ll probably come from a relatively unknown firm.  That’s exponentially less data than the Obama “cave” was working with.  If you’re running a Rockville City Council race, there may be no polling available.  And the number of people visiting your website/receiving your emails/reading your tweets is so small that you can’t run tests to find out which messages/frames/asks are most effective.  You need scale for computational management.  The analytics floor is the dividing line between large-scale and small-scale.

Computational management is a solution to large-scale problems, though.  Honestly, running for city council just isn’t that complicated.  Talk to your neighbors, earn the endorsements of community leaders, place a table at community events.  The districts are small enough that you will mostly be relying upon personalized political communication anyway.  The Hartford mayoral race is a bit more complicated, so data and modeling play a modest role.   Think of that as a rule: as we increase the size of the electorate, the power of the office, and the (resultant) money being spent on the election, the size and complexity of the campaign apparatus increases as well.

The Obama campaign’s biggest managerial innovation was using multiple forms of data to improve decision-making in this complex environment.  Analytics is a solution to the problems introduced at massive scale.  Below the analytics floor, the tools are less useful, but they’re also less necessary.

 

*C’mon Politico, a 24 year-old Rick Moranis reference?  You’re better than that.

Neglect and Uncle Sam, not the Internet, Killed the Middle Class

In an interview with Salon and his newest book, “digital visionary” (Salon’s words) Jaron Lanier claims that the internet has destroyed the middle class. Kodak employed 140,000 people, while at the point of its sale to Facebook, Instagram employed just 13, and (without much exaggeration) thus, the internet killed the middle class. QED.

What a crock.

Lanier is apparently incapable of stepping back from technological determinism and looking at the actual causes of our ballooning economic inequality — which, to cut to the chase, is primarily a result of our policy choices. Yet the role of government in determining the overall shape of the economy is too often understated or outright ignored by those who wring their hands about growing economic inequality.

With some noted exceptions, those who criticize Lanier still mostly point at the old standby twin bogeymen of automation and outsourcing. The HuffPost chat in which all of the guests are willing to challenge Lanier’s conclusions is typical on this count but hardly alone. To his credit, Buffalo State College economist Bruce Fisher starts heading in the right direction with his concerns about fostering and preserving the political and social engagement of those who are being left out, but he fails to take it the next step and discuss the major policy changes and political neglect that have brought us to this point.

The best explanation that I’ve seen of America’s growing wealth inequality is Winner-Take-All Politics, in which Jacob Hacker and Paul Pierson start with a simple look at other industrialized countries to show that inequality isn’t an inexorable outcome trade and automation. The Germans and Swedes certainly have similar chances to outsource their manufacturing and use technology to reduce labor forces.

Not only does the rest of the industrial world have the internet, too, better telecom policy means they generally have faster connections and cheaper prices. Yet as measured by the Gini Coefficient, a measure of economic inequality, their economies have far more equal distributions of income in take-home pay and wealth.

The wealth distribution in particular is just shocking — the US has a wealth Gini of .801 (where 1.000 is “one person owns everything”), the fifth highest among all included countries and almost exactly the same as the distribution of wealth across the entire planet (.803). Think about that for a second; we have the same radically unequal distribution of capital within the US as among the entire population of the world across all countries — from Hong Kong and Switzerland to Nigeria and Haiti.

With our paper-thin social safety net and highly unequal distribution of income and wealth, we’re left with an economy where tens of millions struggle to get by while wealthy Manhattanites are hiring handicapped “relatives” for $1,000 per day to be able to skip the lines at Disney World.

Across countless major policy areas —health care, education, financial regulation, taxation, support for the unemployed, and many more — the rest of the industrialized world generally does far more to make their societies fairer for all. Our shrinking protections for workers may be the greatest single cause of the shrinking middle class. Of course, this can be done badly — I would certainly not want to swing as far as Italy and Spain, where it’s nearly impossible to fire somebody once they’re a regular, fulltime employee. Yet we should not allow employers to fire union organizers with near impunity. We should not force organizers to wait for months between card check and votes to unionize so that employers can “educate” their captive audience workforce with the most pernicious disinformation and intimidation. We should not sit idly while nearly half of states fail to meet even “minimum workplace-safety inspection goals, due to state budget cuts and reduced staffing.”

It’s true that the middle class is being gutted in the US, but this is primarily due to how our political system turns the act of surviving and thriving into a high-wire act for an ever-larger slice of the population. Laid-off baby boomers, even those with desirable skills, are having a devil of a time finding work in a country where age discrimination is only nominally illegal. Meanwhile, our children attend public schools with an unconscionably unequal distribution of funding, so moving or being born into a more affordable neighborhood may cost kids their futures, too.

Teens and laid off workers alike are told that college is the route to a better future, but the cost of education is skyrocketing as states and the feds slash public investment in higher education. Many families — even many families with health insurance — are one major medical problem away from unemployment and bankruptcy. Since it’s totally legal to use credit reports and current employment status in making hiring decisions, being laid off or losing one’s job after a medical problem can quickly become a death spiral. None of this is due to outsourcing or automation, but is instead the result of a noxious combination of deliberate policy changes (the privileged seeking to strengthen their own hand) and policy drift (the rest of us sitting idly by or being ignored when we do speak up).

Frankly, I’m glad that Lanier has released this book, sloppy though it may be. (The people raving about this book as a carefully wrought masterpiece are deluding themselves — and not, as Lanier accuses others of doing, “diluting themselves”.) This is not primarily because he has some insights here and there, but because we need to talk about the gutting of the middle class as loudly and as frequently as possible. We must do so, however, in a way that examines how our collective decisions have gotten us to this point. That includes making international comparisons with other “laboratories of democracy” to see how we can do better.

After even a cursory glance abroad, we will see that we should stop returning to the too-easy explanations based on globalization and technology. These forces are at play across the world, and the other wealthy industrialized countries have generally not had the same dismal results. The more likely culprit is in the halls of government.

Dear Commissioner Copps: Thank You for Your Public Service

On Monday evening, the Hunter College Roosevelt House is hosting an event on media policy and reform, featuring former FCC Commissioner Michael Copps. Sadly, it’s in the middle of my Monday class, so I will be unable to attend — and it’s oversubscribed, so I can’t urge you to attend either.

Still, I’m really excited for my colleague Andrew Lund, who is leading the conversation with Mr. Copps, as well as the many Hunter students and faculty who will be able to attend. Thus, I wanted to share a bit about what I’d like them (and the world) to know about this great public servant.

To fully appreciate how exceptional Copps was as an FCC Commissioner, a role he fulfilled from 2001 to 2011, you need to know how thoroughly the Commission has traditionally been a “captured” agency — that is, generally doing the bidding of the industries that it was constructed, in principle, to regulate.

You should also know how the “revolving door” of government works: After working in government in a position of any real importance, many former public servants often take plum jobs in the private sector where they can leverage their regulatory knowledge and even their interpersonal connections to the advantage of their new employers.

Once he started his term at the FCC, Commissioner Copps knew that, after his time in government, he could easily walk into a plum job in the private sector. After all, this had been the route taken by many of his predecessors — as well as many of his colleagues who stepped down in the interim.

Unfortunately, when looking at the decisions that many of these FCC folks who turned that experience into very-well-paid private sector jobs, one could be forgiven for wondering whether many of them truly had the public interest at heart. Some of their decisions suggest that they were, at least in part, also thinking about their long-term earning potential. I won’t name names, but all of us who follow communication law reasonably closely know the most obvious examples.

When looking at Commissioner Copps’ decisions, however, nobody could possibly doubt that his true allegiance really was with the public for the full decade of his service. Media reform groups like Free Press and Public Knowledge finally had an unabashed, reliable ally with his hand on the levers of power, on issues from broadcasting to telecommunications to pluralism and diversity.

Want a sense of where Copps stands on the issues? Go listen to this interview with Democracy Now. Or this one. Read this collection of speeches or this collection of op-eds. Over and over again, you see him supporting the importance of using the power of the state to shape a more democratic, fair, and representative media system.

Copps is probably best known for his opposition to consolidation in ownership between media companies. He “was the one vote against approving Comcast’s takeover of AT&T’s cable systems in 2002” (p. 261), but this was just a warm-up.

The real sea change on ownership came in late 2002 and 2003, as then-Chair Michael Powell proposed a substantial roll-back in the rules against media consolidation. Copps and fellow Commissioner Jonathan Adelstein pushed to have substantial public discussion around the proposal, including multiple, well-publicized hearings. Powell said no — allowing just one hearing — so Copps and Adelstein went on tour, holding 13 unofficial hearings.

Through this and other efforts, working alongside public interest-minded NGOs, Copps helped bring major public attention to Powell’s proposal, ultimately bringing it to a halt. This slowed (though certainly did not stop) the process of media consolidation, through which ever fewer companies control ever more of our media landscape.

Copps has continued to be known for his opposition to media consolidation — though unfortunately, when Adelstein stepped down in 2009, Copps lost an important ally in the fight. Echoing the 2002 vote, Copps was the only Commissioner to vote against allowing Comcast to purchase NBC-Universal in 2011.

I would love to say a great deal more about Copps’ time at the FCC, but I’ll say just a few more words on one more issue: broadband regulation. He came in just in time to dissent from the FCC’s decisions to give away the keys to the kingdom on broadband interconnection, in the decision that led to the Brand X ruling by the Supreme Court.

The FCC ruled that broadband infrastructure companies — the folks who’ve used imminent domain and massive public subsidies as key tools as they’ve laid the cable, phone, or fiber lines over which broadband is transmitted — are not obligated to share their “last mile” systems with competitors. (This requirement for “interconnection” was already in place for landline local and long-distance telephone service, which led to an explosion of competition and plummeting prices.)

The Supremes held that the FCC was within their rights to make the decision, not that it had to come out that way; if Copps had won the day, we wouldn’t be dogging it in the horse latitudes of poor service, high prices, and slow broadband speeds as the world runs past us on all three counts. In the years after, Copps made the best of a bad regulatory position, serving as the most reliable vote for for mandatory network neutrality.

Again, though ownership and broadband policy are among his best-known issues, Copps was a tireless voice for the public interest on virtually every issue imaginable that came before the Commission. Even though he stepped down from the Commission over a year ago, he continues the work today.

Even as a former Commissioner who spent a decade being the thorniest thorn in the sides of those seeking to make a quick buck at the public’s expense, Mr. Copps could still quickly make a quick buck himself working for industry. There are a large number of companies, industry trade groups, and swanky D.C. law firms that would be quite happy to give him a huge salary, cushy office, and first class travel budget to speak on their behalf.

Instead, Copps has moved on to work for Common Cause, one of our nation’s strongest voices fighting for the best interests of ordinary people. This is just the latest in a long line of decisions in which he has chosen to fight for the public interest, even though it’s easier and more lucrative to fight for those who already have disproportionate money and influence.

For public interest advocates, Michael Copps was, at a minimum, the greatest FCC Commissioner since Nicholas Johnson retired nearly 40 years ago — and perhaps the greatest ever. His work at the Commission will be missed, but I look forward to seeing him continue to have a major role in pushing for a fairer, more just media system for many years to come.

One more point, for anybody who’s read this far: As of now, Copps’ Wikipedia page is a mere stump — the Wikipedia term for an article that is too short and needs to be expanded. In this case, a great deal more needs to be said in order to do its subject justice. I call on you to help me do this in the coming weeks. Mr. Copps was and remains a tireless and effective servant of the public, and this is but a small favor we can do in return.

Social Media and Internet-Mediated Organizations Can’t Replace Unions.

It’s been fascinating to watch the Wisconsin protests unfurl over the past 10 days.  Governor Scott Walker has chosen to stuff his budget repair proposal full of Trojan horse provisions, including “emergency” power to sell off state assets through no-bid contracts to his favorite corporate backers and an end to collective bargaining rights for all state employee unions who didn’t endorse him in the last election.  That’s not hyperbole on my part.  The governor is being exactly that crass.  Readers who are interested in more information on the topic should check out Stephanie Taylor’s excellent essay at Slate.com.

Social media has played an augmenting role in these protests.  There’s the pizza orders, which are pretty cool.  There’s the twitter- and blog-based information diffusion.  There are the solidarity events planned around the country, occurring throughout the past week and also this Saturday.  There’s the $300,000 raised by DailyKos, DFA and PCCC to support the “Wisonsin 14.” And of course there are the “mundane mobilization tools” used to coordinate events themselves.  But in general, whereas the focus on social media in the Arab protests has been so intense as to border on self-parody, no one has really spent much time talking about the internet’s role in this saga.  Nor should they.  The story here is pretty simple.  Walker is trying to destroy the central organizing structure for working class interests.  This isn’t about reduced benefits – the unions have already agreed to cuts – it’s about power, plain and simple.

The new generation of internet-mediated organizations can achieve a lot of things.  They’re optimized for the new media environment, both in terms of organizational overhead, staff structure, membership communications, and rapid tactical repertoire.  But they can’t organize workers in a specific industry or location to increase salary, working conditions, or benefits.  MoveOn isn’t going to sit across from management at the negotiating table.  Your local Meetup group sure doesn’t have that capacity either.  This is the most obvious space where “organizing without organizations” comes up short.  You need to build power if you’re going to confront power.

What’s more, MoveOn, DFA, PCCC, DailyKos, Living Liberally, New Organizing Institute and the rest of the netroots are fully aware of this.  Professional organizers, old-school and new-school, understand that the Wisconsin fight is about power.    Take away the unions, and the super-wealthy will be the only interests in America capable of aggregating massive resources to affect policy change.  It looks an awful lot like a coordinated, multi-year strategy to knock out every significant organization of the left (first ACORN, now Planned Parenthood and the Unions).

I feel the need to point this out because the short-version summary of my research is “the new media environment is transforming the interest group ecology of American politics.  We’re experiencing a ‘generation shift.’”  I want to be absolutely clear about this: what’s happening in Wisconsin isn’t about digital media or about generational displacement in the advocacy group system.  It is a coordinated Rightwing assault on the rights of citizens to organize in the workplace.  Internet-mediated organizations are doing all they can to support the unions in this fight, because they know full well that the unions fill a niche that internet-mediated issue generalists, online communities-of-interest, and neo-federated organizations cannot.  I cannot think of a single serious online organizer who believes otherwise.

Book Blogging: Moore’s Law and Politics

Note: I’ll be spending the next few months writing a book about the new generation of internet-mediated political groups.  This post will be my first “book blog,” in which I try out new ideas that I’m planning to include in the manuscript.  Book blog pieces will be less tied to the politics-of-the-day, and will be a bit lengthier.  They also give readers a window into the broader project as it develops.  As such, feedback is particularly appreciated.

I’ve written once before on this blog about Moore’s Law, the surprisingly accurate 1965 prediction that computing capacity would double every 18-to-24 months.  What I’ve noticed recently is that, while Moore’s Law is common knowledge within the tech community (you see it mentioned in almost every issue of Wired magazine). it’s much less well-understood in the political and social science communities.  Those crowds are aware, of course, that their computer from 4 years ago now seems ancient, slow, and lacking in storage space, but it appears to me that the deep political implications of Moore’s Law (which I’ll be calling “Moore’s Law Effects” in the book, unless someone wants to earn their way into the acknowledgments by suggesting a catchier name!) have largely gone overlooked.

I checked through the indexes of several major internet-and-politics books and, sure enough, there’s no mention of Moore’s Law.  Bruce Bimber’s Information and American Democracy, Matt Hindman’s Myth of Digital Democracy, Bimber and Davis’s Campaigning Online, Phil Howard’s New Media Campaigns and the Managed Citizen.  I’ll check a few others on Monday when I’m in the office, but I’m pretty sure there’s no mention of it in Kerbel’s Netroots, Davis’s Typing Politics, Chadwick’s Internet Politics or either of Cass Sunstein’s books either.  …These are good books I’m talking about here — award-winners that rightly deserve the praise they’ve received.  I’d be thrilled if my book ends up half as good as many of them.  Yet Moore’s Law doesn’t earn a single mention, nor does it show up in most of the influential articles in the field.  It just hasn’t entered the discourse.

The one exception I’ve found is a Berkeley Roundtable on the International Economy working paper by Zysman and Newman that eventually became the lead article of a co-edited volume, How Revolutionary is the Revolution.  It’s a political economy treatment of the digital era as a whole and seems pretty promising (amazon should have it to me by mid-week).  I really enjoyed the following quote in the working paper: “…Information technology represents not one, but a sequence of revolutions.  It is a continued and enduring unfolding of digital innovation, sustaining a long process of industrial adaptation and transition” (pg 8).  That “sequence of revolutions” line is what I think we’ve largely been missing when talking about digital politics.

Take Bimber and Davis’s Campaigning Online for instance.  They conducted first-rate research in the 2000 election cycle on citizen access to campaign websites.  The central finding was that, by and large, the only citizens who visit such sites are existing partisans.  The sites are useful for message reinforcement, rather than message persuasion.  As a result, Bimber and Davis conclude that the impact of the internet on political campaigns is pretty slight.  Web sites simply don’t reach undecided voters, so they aren’t of much use in determining election results.

Their book was released in September, 2003.  By that time, the Dean campaign had already attracted overwhelming media attention, leading observers everywhere to rethink the importance of mobilization.  It was an unlucky sequence of events, having a definitive work on the internet and American political campaigns come out just as the Dean campaign was overthrowing everything we thought we knew about the internet and American political campaigns.

Here’s the thing, though: Bimber and Davis weren’t wrong. The Internet of 2000 wasn’t particularly useful for mobilization.  John McCain raised a bit of online money around his primary, but online bill paying was still in its untrustworthy infancy, and the social web was still restricted to the lead adopter crowd who had heard of Pyra Labs.  The suite of technologies making up the Internet changed between 2000 and 2003.  It changed again between 2003/04 and 2006.  [Pop quiz: what was John Kerry's YouTube strategy in the '04 election?  (A: YouTube didn't exist until 2005.)]  And it continues to do so.  The internet of 2010 is actually a different medium than the internet of 2000.  The devices we use to access it have changed.  Cheap processing power and increasing bandwidth speeds let us access video and geolocational aspects that were prohibitively expensive and technically infeasible or impossible in 2000.  We’ve traveled through five iterations of Moore’s Law, and that means that the devices and architecture of the earlier internet have been overwritten (html to xml being just the tip of the iceberg).

The internet is a sequence of communications revolutions, and that is entirely because of Moore’s Law.  It makes the internet different than previous revolutions in information technology.  Consider: as the television or radio moved from 10% household penetration to 80% household penetration, how much did the technology itself change?  I’d argue it wasn’t much at all.  A television set from 1930 is fundamentally pretty similar to a television set from 1960.  The major changes of the 20th century can be counted on one hand – color television, remote control, vcr, maybe a couple others.  It is frequently noted that the internet’s penetration rate has been faster than these previous communications technologies.  But what rarely gets mentioned is that the internet itself has changed pretty dramatically in the process. (Need further convincing?  Watch the 1995 movie Hackers and listen for the reference to one character’s blazing-fast 28.8 kb modem.  LolCats and YouTube aren’t so fun at 28.8kbs speed.  Or read James Gleick’s 1995 New York Times Magazine essay “This is Sex?” in which he explains that the internet is a terrible place for pornography because search is so complicated and the pictures upload so slowly!)

Transitioning into the political sphere, it bears noting that every election since 1996 has been labeled “the internet election” or “the year of the internet” by a set of researchers and public intellectuals.  The paradox, of sorts, is that they have been right every time.  2012 will be different than 2010, 2008, 2006 2004, 2002, and 2000.  It will be a different medium, in which users engage in modified activities, and this will create new opportunities for campaigns and organizations to engage in acts of mobilization and persuasion.  The cutting-edge techniques of last year become mundane, encouraging organizations to maintain a culture of ostentatious innovation.

Now I’m not suggesting that the internet exists in some state of quantum uncertainty, where we can predict basically nothing in the future based on the past or present.  In fact, as Rasmus Kleis Nielsen points out, the tools that will have the biggest impact on campaign organizations will be the ones that have become mundane, reaching near-universal penetration rates and no longer subject to a steep learning curve.  (As we recently learned with Google Wave, e-mail is much a settled routine at this point.)  Indeed, one of the lessons here may be that we are on much safer grounds when studying individual internet-mediated tools that have reached near-universal adoption (within a given community).  The techno-centric studies of facebook, youtube, and twitter that are a recent fad of sorts are on much weaker ground, because those tools are themselves still pretty dramatically changing thanks to increasing adoption and the ongoing influence of Moore’s Law.

The other thing it tells us, however, is that we should focus attention on the new organizations and institutions being built out of the digital economy.  The continual waves of innovation made possible by Moore’s Law mean that existing industries do not solely need to adapt to a single change in communications media.  Rather, an existing market leader who hires the best consultants, purchases a fleet of state-of-the-art hardware and software, and spends two years developing their plan for the digital environment will suddenly find that the internet has changed in a few important ways, their hardware and software is outdated, and the plan those consultants developed has collected more dust than accolades.

Communications revolutions (or changes in “information regime,” if you prefer to avoid talk of revolution) create a classically disruptive moment for various sectors of the economy.  Rather than advantaging existing market leaders, whose R&D departments let them lead the way in sustaining innovations, disruptive moments tend to lead to the formation of new markets that undercut the old ones (this is classic Christensen).  Startups do better under those conditions, because they have low operating costs and no ingrained organizational routines.  And while individual areas of the internet eventually give way to  monopolies (particularly if we lose net neutrality and let major firms capture markets and tamp down on competition),  those monopolies aren’t as secure as they were in previous eras.  Just ask AOL, Compuserv, Microsoft or Yahoo. The wrong policy decisions can still basically kill the internet, but Moore’s Law creates a scenario in which ongoing disruptions continually advantage new entrants, experimenting with new things.

That, frankly, is why my focus has been on the rise of these internet-mediated advocacy groups.  It’s because they represent a disruption of the advocacy group system.  They embrace ostentatious innovation, keep their staffing and overhead small, and otherwise continue to act like a start-up (and are often founded by technologists with a background in startup culture).  They fiddle with membership and fundraising regimes, and develop new tactical repertoires unlike anything found among the older advocacy groups.  And Moore’s Law suggests that the internet is still in a state of becoming, that the emergence of these new institutions is much more substantial than the mass behavioral patterns found among citizens in the internet of 2010, which may very well be altered as Moore’s Law allows the internet to become something else in 2012.

Moore’s Law, disruption theory, and new developments at the organizational level.  That’s what I think has been missing from our understanding of the internet and American politics thus far.

What do you think?

The R2K lawsuit: market corrections and scalp-taking

Markos Moulitsas of DailyKos announced Monday that he is suing Research 2000 for fraudulant activity, based on a statistical analysis conducted by Mark Grebner, Michael Weissman, and Jonathan Weissman.  I won’t comment on the details of their study here — Nate Silver has done a much better job of that already — but instead want to make a broader comment about the internet, markets, and “scalp-taking.”

I’ll note as a caveat that Research 2000 is launching a counter-suit.  The facts will be revealed in time, and the way things look today may not turn out to be the reality of the situation.  I don’t mean this blog entry to prejudge the results of this trial.

That said, it appears as though the progressive political blogosphere has just claimed a second scalp within the polling industry.  The first occurred back in the fall of 2009, when Nate Silver at 538 raised serious concerns about Strategic Vision.  Noticing serious anomalies in their data, as well as a lack of public information about the company itself, Silver asked some very public questions about whether they were fabricating their data.  The head of Strategic Vision cried foul and claimed he’d see Nate in court, but he then beat a hasty retreat and hasn’t been heard from since.

DailyKos has contracted with R2K since the 2008 election cycle, and has sent them a lot of business.  After Nate published his inaugural pollster rankings last month, Markos announced that he’d be rethinking the partnership with R2K (who fared poorly compared to other pollsters).  That apparently led to a few statisticians deciding to take a deeper look at R2K’s numbers, which revealed anomalies that would be consistent with mild cooking of the books and/or outright fraud.

Talking Points Memo took a deeper look at the head of R2K, Del Ali, and found that his background consists of 2 degrees in recreation.  That’s really pretty odd, to say the least.  You would expect the head of a major polling firm to have a background in, well, statistics.

And that leads us to the point I’d like to make: how is this possible?  Professional polling is a competitive and lucrative business, with longstanding industry leaders and standard-setting organizations.  Neither Strategic Vision nor R2K was a minor player — both were significant pollsters whose findings were reported by mainstream media sources.  Both (it appears) were somewhere between shady and fraudulent.  In a well-functioning market, incentives should exist for shaming and discrediting such actors.  The field of professional polling involves enough statistical wizardry and high enough stakes that, if such incentives operate anywhere, they should operate there.  And yet we now have seen two occasions in which, essentially because Nate Silver and company have made a hobby of advocating for responsible polling practices, major irregularities have been uncovered, with field-transforming impacts.

There’s a lesson here about just how robust the market mechanisms in various knowledge industries actually are.  Even in a field that has incentives for self-policing, even in a field tied to academic institutions like AAPOR that are full of people who have the means and motive to investigate such irregularities, there has been a distinct lack of accountability for years.  The lowered transaction costs of the internet has enabled skilled hobbyists to dramatically affect that market.  The internet itself doesn’t magically improve the polling industry (far from it), but it did create a new opportunity structure through which motivated volunteers could challenge and affect existing institutions.

Bravo to Nate Silver for his nearly one-man quest to improve the polling industry.  He didn’t have to take on this challenge, and I’m sure it’s made him plenty of enemies in the process.  Kudos to Markos Moulitsas as well for partnering with statistical researchers and readily admitting it once he learned there was a problem with his data.  The internet doesn’t make the industries perform more responsibly, it just creates new opportunities for motivated outsiders to mobilize knowledge/people/resources in new and interesting ways.  Between R2K and Strategic Vision, we have a good example of just how poorly the “statistical wizardry” industry was actually functioning, and also a case study in how networked volunteers can transform such industries.

As the old proverb goes, “may you live in interesting times…” Interesting times, indeed.

Advocacy Groups and Haiti Disaster Relief

Scrolling through Twitter this morning, I noticed the following tweet from Tom Mattzie (@tommatzzie), formerly of MoveOn.org:

I hope all my progressive groups and friends remember #Haiti today. I’d be bummed if they didn’t.

Mattzie has backed up the talk himself, pledging to match up to $1,000 in disaster relief donations from his fellow twitter-donators.  As the day has progressed, I’ve already seen online appeals from Color of Change and MoveOn (both urging their lists to donate to groups such as Oxfam and Doctors without Borders).  Nothing so far from the single-issue political advocacy groups, though of course Red Cross and others have appropriately sprung into action.

I don’t mean this to be a critique of the single-issue groups, but it does bring one point to mind that bears examination.

In the presentation that I’ve been giving about my research, I use the phrase “Headline Chasing” to describe the distinctions between MoveOn-style targeted fundraising and the direct mail funding appeals that fueled advocacy groups for the past 40 years.  It’s an intentionally provocative term.  The new generation of advocacy groups organize around whatever issue is at the top of the public agenda, whereas the earlier generation of groups mobilize around specific issue topics, regardless to their immediate salience.  That proves very effective as a fundraising tactic, but it implies a sort of nimbleness and fluidity that may or may not be such a good thing.

I think today’s fundraising appeals are an important example of the unquestionably positive side of this “headline chasing.”  MoveOn isn’t making a buck off this tragedy.  They are mobilizing their large supporter list and asking them to help out through other organizations.  When tragedy strikes, tragedy rules the headlines.  And in that moment, unless the tragedy impacts an issue group’s central focus, the large majority of organizations remain silent, clearing out of the way while the red cross and others take center stage.  The new political economy of advocacy organizations allows the progressive netroots to get behind the red cross, doctors without borders, and other center-stage organizations and quietly help out.  Internet-mediated organizations are performing mitzvahs right now, because their structure allows them to.  Older organizations, progressive or not, remain sidelined because the logic of their structure demands it.

Let’s hope that organizations, governments, and individuals do all they can to come together in the wake of this tragedy.  A 7.0 earthquake is a reminder of just how fragile many social institutions actually can be.  Unpredictable tragedy like this can happen anywhere, and national boundaries should not stand in the way of efforts to aid our fellow human beings.

lessig on institutional corruption

Professor Lessig is presenting on Institutional Corruption today at the Kennedy School as his first public appearance at Harvard since his return a few months ago.

Professor Lessig likes to introduce three ideas to frame his talk today: 1) influence, 2) independence and 3) responsibility.

Relying on his framework of the four modalities of control that he used in Code, Professor Lessig explains how the law, markets, norms and architecture together exert influence, and that depending on your policy objectives, these four forces can be complementing or conflicting. He suggests that together they form an “economy of influence” that we need to understand if we want to make effective policy.

He continues to explain “independence”, in the sense that something is not dependent on something. Independence matters, because it means that you try to find the right answer for the right reason, as opposed to doing so for a wrong reason you might be dependent on.

Independence, however, does not mean dependence from everything. Lessig reframes independence as a “proper dependence”. In legal terms, it means that a judge depends on the law for her judgment. So independence is about defining proper dependence, and limiting improper dependence.

Responsibility is the third concept Lessig goes into. He tells us about a case he represented in 2006: Hardwicke vs ABS. It was a case that focused on a series of events concerning child abuse, all perpetrated by a single person. The question that was raised: Who is responsible? Lessig makes the argument that responsibility does not lie with the individual, that this individual has no power to reform, and that this is pathological. Instead, he makes the case that responsibility in this case is all the people who knew about the wrongdoings, but refused to pick up the phone. Nevertheless, the focus of the law was on the one pathological person. Lessig suggests it is more productive to focus responsibility on those who have the power to make changes, instead of those are pathological and are not in a position to reform. He notes it is ironic that the one person who is least likely to reform is held responsible, while the one entity who could do something about it, was immune.

He raises another example of “responsibility” gone awry. He cites Al Gore and his book “The Assault on Reason”, and lambasts its narrow perception of responsibility. It focuses on former president Bush, arguably the man least likely to reform, and instead forgets those who could have done something about it, suggesting that they also have been critically responsible.

His argument is one of “institutional corruption”. What it is not: what happened with Blagojevitch; it is not bribery, not “just politics”, not any violation of existing rules. Instead, institutional corruption is “a certain kind of influence situated within an economy of influence that has a certain effect, either it 1) weakens the effectiveness of the institution or 2) weakens public trust for the institution.

He explains the system of institutional corruption using the White House. Referring to Robert Kaiser’s book “So Damn Much Money”, he argues how the story of the government has dramatically changed in the past fifteen years and how the engine of this change has been the growth of the lobbying industry. He illustrates this with numbers: Lobbyists pay with cash which members use as support for their campaigns. The cost of campaigns have exploded over the years, and subsequently, members have become dependent on lobbyists for cash – he cites that lobbyists make up 30-70% of campaign budgets! This is not new, he carefully explains, but citing Kaiser again, what is new is the scale of this practice has gotten out of hand. Members /need/ and take /much more/, becoming /dependent/ on those who supply. This is only during the tenure, but institutional corruption also needs to be understood as something after tenure: 50% of senators translate their senate tenure into a career as lobbyist, while 42% of the house do the same. This suggests a business model, focused on life after government, that perpetuates itself, and influential people who end up becoming dependent on this system surviving, both during and after their time in Congress.

He goes on to give example after example of institutional corruption. He mentions the important work done by maplight.org that tracks money in politics, who have shown that members who voted to gut a bill had 3x times the contribution from lobbyists than those who voted against. Simply put, policies get bent to those who pay. He cites a study by Alexander, Scholz and Mazza measuring rates of return for lobbying expenditures, who conclude that ROI is a whopping 22,000%! He again cites Kaiser, who suggests that lobbying is a $9-12 billion industry.

Why does this matter? It matters if it
1) weakens effectiveness of institution or
2) weakens public trust of institution

In the first case, he argues how lobbying can shift policy. He cites a study by Hall and Deardorff “Lobbying as Legislative Subsidy” on how the work of congresspersons shift as a result of lobbying. Imagine you’re a congressperson and you see it as your goal to work on two issues: one is to stop piracy, the other is to help mums on welfare. The line of lobbyists that will happily help you with stopping piracy is long, whereas not so many will help you with the latter – so work of the congressperson shifts, and thus work of Congress shifts.

Lessig suggests it also bends policies. Does money really not change results? Citing the Sonny Bono case of October 27, 1998, he shows how in copyright lobbying power had a powerful influence in getting the copyright term extended for another twenty years. Does this advance the public good? A clear no. Lessig backs this up by telling how in the challenge at the Supreme Court, an impressive line-up of Nobel Prize winning economists, including Milton Friedman, supported this and that it would be a “no brainer” to sign the support that copyright extension did not advance the public good. But he concludes that there were “no brains” in the House. An easy case of institutional corruption. There are two explanations: Either they are idiots, or they are guided by something other than reason. He suggests of course it’s the latter. It is not misunderstanding that explains these cases.

Lessig continues to explain how corruption can be seen as weakening public trust. He tells us about how the head of the committee in charge of deciding the future of healthcare is getting $4 million from the healthcare industry. Or how a congressperson ended up opposing the public option even though the majority of his constituency supports it. The idea is not that there might be a direct link between the money and the vote, but that if you take money to do something that is against the public interest, people will automatically make that link, and this weakens public trust. If you don’t take money and you go against the popular vote, that won’t reek of corruption.

Lessig goes on to discuss different fields: medicine and the healthcare industry, citing research by Drummond Rennie from UCSF that shows how there is an overwhelming bias in favor of sponsor’s company drugs. How there are 2.5 doctors to 1 detailer (a detailer being someone who is like a lobbyist for the pharmaceuticals, promoting the drugs to doctors, often giving “gifts”). How the budget for detailing tripled in the past ten years.

Lessig asks us: how can we find out whether these claims are true? Do detailing practices either weaken the effectiveness of medicine, or weaken the public trust for it? What would it take to know?

There is also the issue of “the structure of fact finding” that Lessig suggests is corrupt. Again, he argues we need to understand whether this is a process by which results are affected or trust is weakened. He cites how sponsor funded research can cause delay, and mentions the case of “popcorn lung”.

Lessig makes a strong case that we need more than intuition. That we need a framework or metric to know for sure. Because we all have ideological commitments, that we need to escape this in order to have a proper understanding of corruption. This is, in short, the aim of his new project: The Lab. It should be a neutral ground with a framework that determines whether and when institutional corruption exists, to develop remedies for institutional corruption when it exists. He sees the initial work having three dimensions: 1) data – necessary to describe influence and track its change; 2) perception of institutional corruption and understand how it has changed;
and 3) causation – what can we say about what causes what in these contexts in alleged corruption. Having this information, we can then design remedies.

Moore’s Law and Sunk Costs: An E-government Dilemma

Moore’s Law,” first articulated in 1965, tells us that we will see a doubling in transistor capacity roughly every two years.  As predictions go, it has proven surprisingly durable, and is a handy conceptual framework for understanding why the internet is evolving so quickly.

File-Transistor_Count_and_Moore's_Law_-_2008

Consider: when I was in college a decade ago, I carried around a spare floppy disk for saving files and transfering them from one computer to another.  It could store about 100 megabites, which was a lot more storage than you’d find on campus webmail.  A few friends of mine had new computers with a whole GIGAbite of space.  It wasn’t entirely clear to me what one would do with all that space.

Today, my iPod has 120 Gigs.  I save my files to gmail and they reside in the clouds.  Wireless networked computing means I can access them anywhere, even on the road from an iPhone.  My computer, bought 4 years ago, is antiquated and slow.  It’s a mac that doesn’t even have embedded video.  And therein lies a disconnect for businesses and governments in the digital age.

Let’s say that the University I work at decides to invest in a complete upgrade of their computer systems.  This includes hardward and software, a new fleet of computers; custom-built e-learning tools.  (full disclosure: this post was inspired by the “MyCourses” e-learning training I attended this morning)  In so doing, they incur some substantial sunk costs.  Those computers and those software programs have got to be useable for several years.  Organizations and governments don’t move seamlessly up the line presented in the figure above.  They move in a stepwise fashion, investing in new tools every few years which they are in turn saddled with as the technology continues to evolve.

Why is this important?  Well for one thing, it helps to explain why large bureacracies will virtually always have outdated websites.  Purchasing a fleet of new computers or moving into a new area of web-related activity caries a cost, and no organization can afford to keep pace with the rapid expansion.  Sunk costs are a reality of egovernment and organizational adoption.

Particularly in terms of software, however, it seems to me to be a strong argument in favor of partnering with organizations like Google or, better yet, relying on open source software platforms.  Learning today about Brown University’s “My Courses” software platform (it is similar to the “blackboard” site at Penn… a password protected site for students to access the syllabus, readings, grades, etc), I couldn’t help but think that the more advanced functionalities could be accomplished for free with a Ning site.  I’m sure “My Courses” was cutting edge when Brown first made the investment, [well... I'm not sure, but I'm willing to believe it] but companies like Ning have been free to innovate — incentivized to do so, in fact — while Brown has incurred the sunk costs.

In a digital environment that evolves as quickly as the internet, we can draw a hardline distinction between the entities who have an incentive to continually create and innovate (tech giants, startups who hope to challenge tech giants, and open source communities) and the entities who proceed in stepwise fashion, incurring sunk costs, staying pat for several years, and then incurring another round of sunk costs.  It then follows that one of the benefits afforded to informal networks (policy networks, activist networks, political networks, communities-of-interest… Clay Shirky’s “organizing without organizations”) is that their lack of resources also leads to a lack of sunk costs, meaning that any online action they engage in is likely to be, and essentially remain at, the cutting edge.

Those are just a few early thoughts on the topic.  E-gov hasn’t been one of my major research areas thus far, but I’m warming up to it.  Thoughts/reactions/feedback would be appreciated (as always)

Andrew Lih on the Wikipedia Revolution

Do you want to know how Wikipedia was able to become such an incredible success? Who the people behind its success are? The best book to learn about the history and the culture of Wikipedia is Andrew Lih’s new book “The Wikipedia Revolution“, launched last week. He was at Harvard last night to give a talk and do an interview with Berkman Fellow and distinguished internet scholar David Weinberger.

Andrew shares with us his story of how he first came across Wikipedia – in many ways, it was a very different experience from most people. On February 9, 2003, Andrew was looking for his next research project – he has been studying online journalism and new media for a long time – and has been instrumental in creating the new media program at the Columbia J-school – he was told that he should take a look at this new site called Wikipedia – this amazing site that “anyone can edit”. Contrary to most people, he heard the principle first, before he saw the actual website. When he took the time to explore the site, he was immediately taken away with it, thinking “the crowd could not have written this” He looked at more pages, started using Wikipedia in class assignments, and became so fascinated with the project that he wanted to study it full-time.

“It works in practice, but not in theory” is often said of Wikipedia. And that’s definitely true if you consider its origin. Wikipedia started out of a project called Nupedia – in many ways this was projected to be a conventional encyclopedia. Started by Bomis, it envisioned a 7 step rigorous peer review – it would recruit volunteers to write its articles – and the hope was that most of these volunteers would have a PhD degree. That is, the original vision of an online encyclopedia was one with very high stringent requirements.

The big problem: after one year, Nupedia had the grand total of twelve (count ‘m) articles. Even worse, they were written by someone on the payroll. This was clearly not sustainable. Larry Sanger decided to intervene – realizing they needed something radical to at least get seed material. He turned to this thing he saw called wiki software – created by Ward Cunningham – wiki was a way for programmers to share best practices – it would be an online resource for programmers. The name came from the wiki wiki bus in Hawaii – meaning ‘quick’. The wiki software indeed produced quick results – as of recent, there are over 2.8 million entries in the English Wikipedia alone. So why does Wikipedia work? Andrew suggests five key factors: it was free – open – neutral – timely and social.

Andrew describes the piranha effect – the idea that one change in one corner can inspire other changes and create a torrent in the community. For example, in one particular week, 33,800 (count ‘m) articles were added in Wikipedia. This was largely from a huge body of census data from the US – a software robot was written to extract relevant information from this data and inject every possible town and city in Wikipedia. One such town was Apex, and it just happened that on one day SethIlys visited this page. It was a dry article – what he decided to do was – hey, why not put a map on there? A few keystrokes later, he had added his own handmade map – and in his own way, was able to contribute his knowledge to the world. Useless perhaps? Perhaps, but if he visited this page, why not someone else as well? This experience was really empowering to him. Once he started with one map, he figured, why not add others? And once he started, it did not make sense to stop – so like Forrest Gump – he kept on running. The strange thing was, others started running, too. Nearly all the US census location articles now have maps.

There’s a famous saying: “if you have a hammer, everything looks like a nail”. Andrew adds to that: “If there was ever a project that had lots and lots of unhammered nails, it was Wikipedia.” The dot map project was an inspiration – an exemplar – encouraging people to do things they never thought possible. And in many ways, Wikipedia itself is such a project as well – an exemplar.

David starts his interview with Andrew.

David: Let’s get this out of the way first, are you neutral about Wikipedia?

Andrew: No I’m not. I analyze neutrally. But I’m a big fan. I believe Wikipedia is one of the most fascinating creations man has ever made. But that doesn’t mean it doesn’t deserve scrutiny.

David: You think it was important enough to write a book about – an endorsement in itself. But let’s go to its origin myth – as all super heroes have one – the myth is often that idealists came together to do this democratic experiment and that the world’s greatest encyclopedia is the result. Is that right, where did it go wrong?

Andrew: Telling Nupedia’s story helps debunk a lot of this. It started as a failure. There was no way anyone knew how to do this. Even though the founders were very internet savvy and big fans of open source, it was not apparent that doing an encyclopedia in that style was the way to go. Only after a full year, did they decide to try it this way.

It’s also interesting that Wikipedia is always cited as an example of democracy, but the community itself never uses that word. It assumes good faith, it likes consensus, but it never ever uses the word democracy. As a matter of fact, a key thing in wiki is NOT to do voting. They discourage voting – they rather decide through discussion, not to rely on hard measures like voting.

David: What’s wrong with hard measures?

Andrew: The problem of gaming the vote without having meaningful discourse. One of the most contentious issue was the Danzig/Gdansk edit war. An edit war is what happens when you don’t converge on a neutral point of view – the result is that there is a constant flipping back and forth between different revisions of one article. This edit war was the catalyst of a lot of policy change – for example, the Three-Revert-Rule. But in this case, after a year of brutal edit war, voting was inevitable – it was a defining edit war in English Wikipedia history.

David: Can you talk about the flatness – that supposedly every voice is equal and there is no hierarchy – and its rules, the anti-rules and emergence of rules?

Andrew: The rule is that you shouldn’t have that many rules – having too many rules, you start to game the rules. There are rules nevertheless – neutral point of view, assume good faith, – the idea that your next contributor could be the most prolific one, so don’t bite the newbie. But these rules are soft ones and established during the early days – the community has changed quite a bit since 2001.Today it is no problem to get people to contribute. The problem is to get rid of bad stuff. The concern: is the community is still as vibrant as the early days?

David: There is an antipathy towards rules – the idea that rules tend to breed bad behavior – yet at the same time it is a warm-hearted community – assume good faith. To what extent is Wikipedia free of a certain political mindset in the structure of Wikipedia as an emergent community?

Andrew: The English Wikipedia, it’s a liberal progressive community, or libertarian. It is reflected in the early roots of Wiki – they met on Objectivist mailing lists. Jimbo (Jimmy Wales) is a straight forward libertarian – common in the geek community. The articles are generally of good quality nevertheless. But if you disagree, you can fork. One such response is Conservapedia.

David: Is it built along the same principles?

Andrew: No, but I wish it were. Articles are often written in direct opposition to Wikipedia articles.

David: Is it open to edit?

Andrew: Hmm, hard to say. More people are in control, they are not as inclusive.

David: What I like is the pragmatism of Wikipedia – a general dislike for rules, but if you need a rule to build an encyclopedia, then it’s fine.

Andrew: There are really five pillars, one of them is that Wikipedia is an encyclopedia. That might sound silly, but that wasn’t so in 2004. Wikipedia had grown as a community with lots of social aspects – there was a gaming lounge for example where people were playing virtual chess games. We had to shut that down – it was pretty cruel – but we are here to write encyclopedia articles and not to support MySpace activities.

David: It’s also a discussion about which articles are deleted – that Wikipedia is not an art project. It’s an encyclopedia, but sort of different – so the question becomes what an encyclopedia is in a digital age? It’s a sharp edged debate between the deletionists and inclusionists – what side do you fall on?

Andrew: The inclusionists’ argument is that wiki is not paper – why not have articles about anything under the sky? An article on an obscure issue does not take away from your general experience. The deletionists, also called exclusionists, argue that the value of an encyclopedia is that it is a set of articles. It’s no good to have an article where every single word is cross-linked, or that are not reliable – the key test here is – should we have an article on what we had for breakfast?

In the early days, I was considered an exclusionist. I argued that it does matter how selective you are – that articles need to be verifiable, high quality. Over the years, the community standards have shifted, to the point that I don’t think I have changed my stance that much, but where I am now being considered an inclusionist.

Now it is crucial to keep out the bad stuff – Wikipedia is now high profile – and recent policy changes are all about restrictions, restrictions, restrictions. It provides a much more different atmosphere than the early days – now much more stringent.

David: What gets people so passionate about this particular issue?

Andrew: It’s not just within one language – it’s across cultures as well – for example, the German Wikipedia has 900,000 articles – a long way to go before you hit the 2.8 million articles of the English Wikipedia. But the Germans are very happy with their 900,000 articles – they generally have a much more stringent standard. Wikipedia used to be known as the definitive guide to Pokemon – that would not fly in German Wikipedia. That’s their style. The German Wikipedia is more traditional – but also has a great reputation – the German government, libraries, and universities are all interested in working with Wikimedia Deutschland because their quality is so high.

That is to say, the inclusionist/exclusionist argument also varies widely depending on the cultural lens you use.

David: Is it a problem that neutrality happens only if there is enough homogeneity in the community? Or they will have to break off? Does Wikipedia reinforce a prevalent domain of discourse that everybody agrees on? And thus excluding other views?

Andrew: Certainly in some languages – the first twenty languages – the largest languages – are fairly well educated and multilingual – especially contributors for the English Wikipedia span the whole world – and there is diversity of view points. But after the twenty languages – the drop off is bigger – and people are more homogeneous.

David: Isn’t this the case in English Wikipedia as well? That is, neutrality hides a fork – people fork.

Andrew: Yes, but they create meaningless forks, that nobody links to, they fade away.

David: That is exactly the price that it exacts – marginalization of points of view out of mainstream – that they cannot get on the same page – lots of groups accuse Wikipedia of this.

Andrew: Jimbo said once that Neutral Point of View is a term of art – most things that work are not razor sharp. There is a lot of faith in the actual ground troops – that they stay within directive – and that hopefully the diverse community will take this in account and create reliable content.

David: Lets talk about the changing roles of authority. Being a big prof doesn’t matter – its bad form even if you say this.

Andrew: Editorial authority is even more interesting in Japanese Wikipedia – most are anonymous – this is because the dominant internet culture in Japan is based on anonymity. You could be discussing with anyone, a housewife or a prof, what matters is the quality of edits.

David: Let’s talk about Essjay.

Andrew: That was one of the bigger crisis. Essjay was a pseudonym – and on his user page it said that I can’t tell you who I am but I have a PhD in Theology and I work at an academic institution but would get into trouble if I tell you my real name. Was an incredible prolific contributor – over 10,000 edits and everybody generally accepts that they were good quality. He eventually got access to admin privileges – that is, he could check IP addresses of users behind the scenes, and only a dozen people can do that, had access to private data.

What happened was that the New Yorker was doing an article – by Stacy Schiff, a Pulitzer award winning reporter – she did an interview with Essjay – wrote a long piece. Then Essjay took a job with Wikia Wikimedia Foundation (EDIT: Andrew corrected me: he took a job with Wikia, the for-profit firm founded by Jimmy Wales and another Wikipedian Angela Beesley) – and to do so, he had to come clean – that he was a 20-year old with no PhD degree. This was a huge embarrassment to the New Yorker – it seemed that Stacy never even asked Essjay’s name just to fact check it.

Some people argue that Essjay lied to a reporter but had good contributions. Others pointed to the fact that he sometimes used his credentials to win arguments. It was a real soul searching for the community – a prized Wikipedian would lie to the outside world, to a Pulitzer award winning reporter, and raised issues with regard to having faith in each other in the community.

David: The increasing use of credentials – or the German system that now allows for the marking, a flagging of pages that are considered reliable – is this a trend that will continue?

Andrew: Germans lead on quality issues – they have a tighter community of admins, who almost act like a council – whereas the admins in the English Wikipedia function more like janitors. So why not have a flagged version – you could flag the last version of an article that is stable – and you show people the latest checked version. You get better quality but you lose that they are instantly updated. The Germans implemented this last year – quite a success – flagged 89% in first year. The English Wikipedia has interest to implement this but it is hard to get the community to reach consensus on anything at all. Right now it’s a total stalemate – it had a surge of initial support but now trickled down.

David: The common complain is that students go to Wikipedia and simply believe what is there. What is it that readers need to do not to be fooled by occasional vandalism? How scared should we be?

Andrew: Wikipedia should be the starting point, but not ending point. It should not be in citations, just like entries from the Britannica should not be cited.

David: How confident should we be when we use it to look things up>

Andrew: The critique that it is dangerous when 14 year olds take it as gospel is not fair. Most people are media savvy. And then there is a whole range of things the community implemented – for example, requiring sources – in 2003, 2004 you never had any article that was tagged ‘citation needed’, now you do everywhere – there is a team called the ‘citation needed patrol’. Standards have improved – but ultimately I think flagged versions should be put in some way – right now it looks like it will be used for entries of living persons – this is for libel reasons. We start there and see what happens.

Audience Questions

Question: Can you discuss failed Wiki projects?

Andrew: The battlefield of failed wikiprojects is vast. Wikitorial from the LA Times was a real disaster. There is an assumption that you put up a Wiki and the Wiki Magic will happen. The LA Times learned the hard way – if you have no robust community with admins that fight vandalism, it’s a recipe for disaster.

What you realize after all this failed projects – wiki is perfectly suited for encyclopedia. It’s like a bento box of writing.
Very structured writing and lends to crowdsourcing. Very modular. This is not true for a novel, for example. Penguin had a contest where they put up a Wiki and expected that the magic wiki crowd would write a novel – did not happen.
Those that do work: lots of sharing, step by step, modular structured style of writing. Certain type of content are like this, but lots don’t. A lot of other organizations learn the hard way.

Question: Why not make people use full names?

Andrew: There is always talk in community – now do we don’t need anonymous people anymore – they give us more problems than they are worth – lets start requiring higher standard. In the beginning – the original culture dominates – Wikipedia tends to be inclusive – anonymous users are the core value of “anyone can edit”.

David: What about pseudonyms?

Andrew: It makes you to be able to converse with this person, it allows interaction, although you don’t know the authenticity. You can still see all the edits. Interestingly, pseudonym users give less information than anonymous users – with anonymous users, an IP address is recorded, and that often provides geographic location, what organization you are part of, etc. The Wikiscanner used this to its advantage – found out that people in Congress, Ogilvy, all kinds of organizations were editing articles they probably should not be editing. It was a typical example of sunshine being the best disinfectant – it was a kind of watchdogging the crowd.

Question: If Wikipedia would have been run by company, would it have been different?

Andrew: If Wikipedia was a commercial company, no way it could have been successful – people contribute because it is a free license – same like with Linux – people know it wasn’t making a company rich. Example is the Spanish fork – in the early days, there were some rumours about the possibility of advertisements – the Spanish community went ballistic on the mention of ads – they literally took the ball and went home – started Encyclopedia Libre – convinced all contributors to leave. This incident set the tone for the community since then.

Question: Is the bulk of content made by a small number of people?

Andrew: The idea behind the 80/20 rule is that 80% is done by 20% of the people. But this is not necessarily true for Wikipedia – Aaron Swartz’s research shows that there is a wide swath of people that edit Wikipedia. While the distribution is still non-linear, it’s just not the case that there is an elite crowd who edits over hundred hours a week.

David: Aaron’s work shows that the creation of new articles, the bulk of it is done by a broad range of users – which makes intuitive sense.

Andrew: As far as where the community is now, we don’t have good numbers. Since October 2006, there is no authoritative dump of Wikipedia anymore – it takes more than a month to do a monthly dump. This leaves Wikipedia vulnerable – and you also can no longer do statistical analysis.

David: We should each download one page!

Question: Can you talk about Larry Sanger?

Andrew: Sanger has an odd role – he did set up most of the basic rules of Wikipedia – but over time also encouraged Wikipedia to be more elitist over time – and some started seeing him as a pariah, as the anti-Wikipedian. Citizendium is supposed to be Wikipedia done right – with a layer of expertise but still largely open. His main criterion seems to be maintainability. He thinks a lot of what is going on in Wikipedia is just bs – trying to turn vandals into productive members – he is saying, cut that out, work with experts who can cut through the junk. We’ll see what history will say about that.

(Question about the vote on license migration – got lost in the details)

David: Wikipedia experienced exponential growth – but what got us there may not be the right set of tools to move ahead.

Andrew: That’s why flagged is inevitable – not to grow further, but to maintain quality.

Question: How did the power structure evolve?

Andrew: The number of privileged positions have grown but tend to be technical rather than editorial oversight. As an admin – you can block users – but only in narrow situations. You can lock articles – but only temporary – for combating vandalism. Promotion is community decision, there are no hard metrics. Things considered include the number of edits, activities you engage in, social capital – these are all intentionally left vague – the decision is made on an interaction human human basis – it’s not like there is an eBay rating or Amazon ranking.

Question: Why are there different forks and how do they exist – is there a possibility to have one global Wikipedia instead of all these divides?

Andrew: You’re right that it is too easy to see the 2.8 million English entries as the super set from which other Wikipedia languages should be translated from. This set is missing lots of things on Chinese arts, history – things the Chinese Wikipedia has. But the problem is, you need bilingual folks, tools to discover which article is good in one language and has a bad counterpart in another ..

Question: Will the WikiMedia foundation do this?

Andrew: They are a great engine to raise funds.