I am concerned about researchers recruiting Wikipedia community volunteers as human subject participants in their research when they need people to take surveys or be interviewed. The major problems with research on the Wikipedia community is that time in research is time taken away from Wikipedia community activities and research done by outsiders to the community is unlikely to be relevant to the community and provide benefit justifying the time taken away. Other complications include the very large number of researchers wishing to talk with Wikipedians and the pressure of survey fatigue which limits the number of times the community members will be available for research.
Wikipedia is a well known project everywhere in the world where people use Internet, and I think that Wikipedia users know this. Many people also know that Wikipedia is community written, even if they do not know what that means or to what extent anyone can do things on Wikipedia. What is much less known is just how few people actually contribute content or edit Wikipedia. Wikipedia is one of the most popular websites because of readers. The most recent data I have is from 2010, saying that worldwide Wikipedia gets 365 million unique visitors a year. Right now, Alexa is reporting that 20% of this traffic is from the United States, 9% is from India, then other countries are less. A lot of people are reading.
Surveys and interviews take the time of Wikipedians, and frequently researchers seek the most talented and active Wikipedians. When those Wikipedians spend time doing the research, that time is probably cut from the time that they would otherwise be doing community activities. This is valuable when the research benefits the community, however, as time passes it is becoming more evident that many universities in this world have at least one researcher or student every year that wants to do some Wikipedia research. There will never be anyway way that all researchers will be able to contact and take time from only the community’s most active contributors, and even if they did, that would be a loss of valuable time for the community.
One of the problems with giving time is that too often, the researcher is unable to provide community benefit in return due to their research being so without understanding of the Wikimedia community that it fails to have community relevance and provide benefit. In many and perhaps most cases in which researchers plan to recruit human subjects, the research being conducted is not valid because the researchers have made no effort otherwise to understand any part of Wikipedia’s functionality or community norms. Instead, researchers use the research process to learn what they could come to understand themselves if they participated for a few hours in Wikimedia projects. For example, they might recruit human subjects to talk to them about how conversations happen on Wikipedia, when that information might be better gained by observation or participation rather than by recruiting volunteers to explain such things to them. In lots of other ways, researchers will often ask strange and meaningless questions if they have had no insight either from being in the Wikimedia community or getting comment from a community member before making the survey live. When human subjects are recruited for research, there ought to be some amount of understanding of the Wikimedia community and the value of their time.
To put this in context I am going to describe a community on English Wikipedia. WikiProject Medicine is the community of contributors who are most public and available about their support for developing Wikipedia’s articles on health topics. WikiProject Medicine is one of the most popular of the wikiprojects, which are communities of shared interests. The only more popular wikiprojects which are not administrative are military history, LGBT studies, video games, and mathematics, so WikiProject Medicine is among the largest specialized communities on Wikipedia. Wikipedia’s health articles are viewed by about 150,000,000 people per month. There are about 30,000 health articles on English Wikipedia. There is a fair amount of academic research on Wikipedia’s health articles. I am saying all of these things because I think all of this information would lead many people to believe that there must be many people developing this content and making all of Wikipedia’s health content work very well.
The truth is there are hardly any regular health editors at all on English Wikipedia, and almost all of the editing is done by people who pass through, do a bit, then leave. The stable and at hand editors are few. In 2013 28 editors had made at least 1000 edits to Wikipedia, 72 made 500, 129 made 250, and 375 made 100. A person who has made fewer than 100 edits to health articles on Wikipedia probably could not be called a regular health editor. WikiProject Medicine’s own forum gets, by my informal count, not more than 40 people there who post twice in a given month. Those people are probably constitute the community to whom anyone is likely to direct questions about health, as there is no other public forum making itself available to answer questions on developing health content. Looking at these figures, one might think that about 30-40 people are managing the world’s most popular source of health information. Thinking further, if health is one of the most popular communities on Wikipedia, then one might expect that many other communities are managed by even fewer people. “Managed” is not the right word here and perhaps there is no right word, because on Wikipedia, most work is effectively done by queues of volunteers who do not participate in organized community groups, and because the “managing” community groups actually have no hierarchical authority except that they choose to make themselves available for discussion.
In considering any other community group on Wikipedia which does anything, anticipating that not more than 30-40 people manage that task or function would not be an unreasonable assumption. To make the assumption more reasonable, anticipating that 30-40 people do 80% of the organized community tasks in any given work queue is probably the case almost everywhere on Wikipedia.
The problem with survey research and interviews on Wikipedia which are supposed to target a representative sample is that most researchers really expect to recruit about 50 people for their studies. Not only do they expect their study size to be larger than the pool of possible research participants, but also when they recruit a research participant that person’s time is so dear because if they were found in a public channel, then it is highly likely that they are one of the public community organizers. It is a major problem for the Wikipedia community if its most active participants are distracted in any way, so if such people participate in research, a benefit should be expected.
Research is further problematic because we are approaching a time when every university in the world every year is producing a student or researcher who wants to answer their personal question about Wikipedia through human subject research recruitment. I wish all researchers well, and research is a great benefit to the Wikipedia community. But when researchers come to Wikipedia, I wish for them to be mindful of what they intend to take from the community and be passionate and thoughtful to give back at least as much as they take.