My views on a typical exchange with conflict in editing health content on Wikipedia

At the 2015 WikiConference I met Diana Straussman, chair of the board of the Wiki Education Foundation. We had a conversation which I would describe as having two focus points – civility and non-medical information in Wikipedia health articles. As an example in our conversation, she told me that she had a student participating in her own instance of the Wikipedia Education Program who was seeking to edit the autism article. She asked for my thoughts. I looked over what was happening. I saw nothing unusual happening there, and that the events unfolding were typical. Because Diana was interested, and because I would like for her and anyone else interested in Wikipedia in the classroom to be informed about how medical content on Wikipedia is managed, I am going to give my views on what is happening here and why.

Here is a breakdown of events:

1. Diana as a professor creates an assignment for her students to contribute to Wikipedia in autumn 2015
2. One student makes a new Wikipedia account and plans for their first edits to be in the article on autism
3. Immediately before the student edits, the autism article is in a certain condition that Wikipedians call “featured“, which is the highest quality designation that the Wikipedia community assigns to anything. Since the article is featured, one can presume that some of Wikipedia’s most thoughtful and knowledgeable contributors will be at hand to respond to any comments on this article.
4. The student makes a proposal on the talk page to do a certain kind of editing, and immediately the student gets what seems to me to be status quo advice. All further conversation referenced below happens here. There is an exchange in which Diana joins in, and staff of the Wiki Education Foundation joins in, with Diana and the staff person proposing that something unorthodox happen. The conversation sort of halts with the student + two unorthodox positions on one side, and the one status quo Wikipedian on the other, with neither side really acknowledging the other.
5. The student makes these edits to the article contrary to the Wikipedian’s advice.
6. As is routine, within a day a Wikipedian (Doc James, the omnipresent health reviewer) reverts the content for being unorthodox.
7. This is a point where the Wikipedia volunteer wanted conversation to proceed to its end. There is a hope among regular Wikipedia reviewers that when new users have their content removed and get a note about why, those new users will pick up conversation and ask “What is the problem, and what is the quickest and easiest resolution?” In this case and in lots of others, for whatever reason, the outcome was that the new user whose content was removed simply quit communicating. This response is especially common from new editors like students, and is one of the reasons why the Wikipedia community hesitates to support a lot of student groups. Far too often, students feel comfortable requesting Wikipedia community volunteer time, but when they get it, they cease communication if it includes any criticism. This is true even when there is a sincere praise and only a little criticism – students in classes who feel watched by teachers are super sensitive to criticism. Not getting a response makes volunteer Wikipedians feel bad because they feel like the student is using them and their time without considering them as a human. In this case the conversation did go forward a bit but not in a way that satisfied anyone.
8. The student dropped out of the conversation on October 8. Because of the activity here and number of people looking, on October 21 a Wikipedian reviews what happened in a narrative (search for “for the record”, a comment by Flyer22). Other conversation had been happening but for various reasons the student did not appear again till October 30, which makes for a disjointed conversation.
9. I will stop here – at this point, student and professor are likely asking, “What happened – is this normal?” and also thinking, “Who are these reviewers, what do they know, and what gives them the right to remove my good information?”

Here are some questions that, if answered, could give insight to this exchange:

1. To what extent will any contribution to a Wikipedia article be precisely processed by Wikipedia’s crowdsourcing?
2. What would be the Wikipedian assessment of the content submitted?
3. Who reviews content submitted to Wikipedia?
4. What is “WikiProject Medicine” and to what extent does it matter?
5. How can anyone review content on Wikipedia?

I will take this points in turn.

To what extent will any contribution to a Wikipedia article be precisely processed by Wikipedia’s crowdsourcing?

An aspect of Wikipedia which I find attractive is its ability to institutionalize a large number of strangers in a decentralized way to deconstruct complicated situations to parts which can be managed objectively and give a consistent outcome. By this I mean that Wikipedia has a large number of rules, some more or less subjective, and it has no straightforward way of teaching these rules to its participants, but somehow all of its participants seem to converge on uniform interpretations of how Wikipedia should be. There is something gratifying and compelling about passing a complicated judgment on a complicated situation, then getting confirmation from a large number of strangers that they would have done the same thing. Likewise, it is gratifying to see a complicated problem and to imagine how one would respond to it, but then to see another person resolve the problem just as if they had shared the same thoughts and ideas.

One of the reasons why Wikipedia’s community of editors trusts it so much is that participants have a deep sense of community with each other, which builds over time as every day participants see acquaintances and strangers alike seemingly making all kinds of subjective decisions but which all lead to the same resolution. The trust that builds over time is that when there is a problem, someone in the crowd will manage it in a way which no one in the group could improve, given the context of the rules and near universal acceptance of those rules. So often, when something happens on Wikipedia, it looks like one person did it, but in fact, any number of people would have done the same action in the same way had they been the ones to encounter the situation first; and also, no one in the crowd would have handled the situation in other ways.

It is sometimes said that Wikipedians argue a lot with each other. It is definitely true that in Wikipedia’s massive scope and huge number of participants, there are lots of arguments to be found at any time. However, disagreement is not the norm, but rather it is just easy to spot. Wikipedians do not congregate to affirm each other’s decisions, which has its pluses and minuses. A plus to quiet confirmations is that without voiced agreement, multiple Wikipedians redundantly review the same agreed-upon actions and leave no mark on their review, which means that a lot of review goes into each action. Minuses to this system include that it grows doubt that review ever happened at all, and is a waste of labor after enough people have confirmed that an action is good, and also it does not permit research on efficiency in using volunteer labor.

What would be the Wikipedian assessment of the content submitted?

Taking the student’s contribution as an example, here is are my thoughts on this content. I feel that I am probably close to the Wikipedia hivemind on all of these points.

1. The student is failing to use citations when making assertions. This must mean that they came to Wikipedia with insufficient editing advice and were not prepared to accept feedback when it was offered. When mingling multiple sources in paragraphs and making bold claims Wikipedia reviewers expect a citation after every sentence. Without that, verifiability of information quality is impossible.
2. Failing verifiability is a deal breaker. It is one problem for the student to not perform the technical action of citing sources, and another for them to not engage in conversation that demonstrates some appreciation of the value of citations. There are two impasses here – one is improper editing, and one is lack of indication that the student knows why their edits cannot be accepted. The student really must come to understand this or further discussion will have no point.
3. It is not the student’s fault, but when they are making citations, it has to be in a way that Wikipedian can follow. Standard academic citations are fine. Wikipedia’s own automated tools will generate citations from a doi or PMID. If the student was at least attempting to use enough citations then the Wikipedia community will fix this.
4. The student was given advice to not do something but then they did it. That is fine. For whatever reason, a common response on Wikipedia when someone is directed to not do something, and they engage in conversation about doing it, and they are still told no, then they end the conversation and then do the thing anyway. This does not surprise me at all. It happens all the time. Their content gets removed then hopefully they will participate in further conversation. In my experience, people who engage in this behavior somehow have their minds go blank and forget having the conversation which is documented in print, and really expect the outcome they want. The unfortunate part about this is that when someone is told “do not do that”, then when they do it, they seem surprised that other people undo their action. There is no aggression here, because Wikipedians expect this illogical behavior even if it is inexplicable. However, the person who did the action often behaves awkwardly after this.
5. If printed on paper, the previous version of the Wikipedia article would be about 13 pages. With the student’s addition, it would be about 15 pages, meaning the student added at least 1.5 pages of text to this document. As I said, the article already had been reviewed as really high quality and well balanced. Wikipedia’s “neutral point of view” policy talks about due weight to subjects. In my opinion, the student’s contributions were biased and would break the neutrality of the article. Whereas the status quo version of the article gave a global presentation of the topic of autism, the students contributions discussed autism in the context of middle class America, detailing autism programs in the classroom, talking about US autism law, local certifications or programs for special needs students, and some context on US federal law with two mentions of how Mexico also follows some of this. When I see this, the thought that comes to my mind is “What does this have to do with India?” Since the information is not globally useful, it should not appear in this article, and the article definitely should not suddenly become weighted to serve the minority privileged class in the United States by taking about 10% of the article real estate.
6. There is plenty of room on Wikipedia, but it may not be in the place a person first posts. Good Wikipedia articles are about 8-15 pages when they are fully developed, with anything more than that requiring a the split or forking of the article into shorter concepts that can be considered independently. The student’s content, in my opinion, would be great for an article like “Management of autism in the United States”, because in an article with that title this kind of localized focus is more appropriate than an article for an international audience.
7. Another way to deconstruct the student’s contributions would be to look more deeply at the use of educational interventions as a way to treat, address, or manage autism. If the student had been writing about that, then that kind of content could have gone into the main autism article, but instead, the student was citing local case studies (primary sources). If the student used secondary sources or review articles to describe “effects of educational interventions on students with autism” then that would be appropriate to include. This did not happen.
8. Sometime else that the student could have done, even if it were United States focused, was started a length-proportionate section on policy and practices of autism in education, separating that from claims about its efficacy.
9. One of the most problematic aspects of the student’s contributions was that each claim tied to another in the tone of a persuasive essay. It seemed to me like saying, “This is the United States, and this health intervention is effective, and these policies bring these results.” It is a series of bold and persuasive claims to describe the state of US practice as a whole, and also to make health claims on what is “proven to be helpful” without good medical sourcing, and also to make claims about the outcomes of legislation. I understand why writers do this – people tend to want to see patterns and to derive narratives when they read multiple papers on the same topic. However, Wikipedia does not encourage these kinds of personal insights to be expressed in Wikipedia. Instead, Wikipedia presents the facts, and if it presents any interpretation, then it only does so by presenting and citing an interpretation that some expert already published. Student researchers get coaching to try to draw their own conclusions about the significance of research. In university culture, interpretations tend to bias toward believing that meaningful conclusions can be drawn from intuition about research data, and in Wikipedia culture, the community says “no essays” and is conservative about not ascribing meaning to primary data when the researchers who published it have not themselves explicitly drawn a conclusion from it.
10. Consider this source – it is a primary study by two researchers on about 30 students. The student said, “One study on the joint attention of autistic children in a classroom, showed that teachers “did not adjust their teaching to address” the issues of decreased play and joint attention behaviors.” and cited that paper as a source. From a Wikipedia reviewers perspective, this is not information worth sharing, because no conclusion can be drawn from primary research like this except to say “in this one case something happened”. There may well be a primary research paper which gives exactly the opposite result, so presenting information like this provides no reliable context for understanding anything except that the issue is being researched. Newspapers may tolerate writing which describes what happened to 30 people and leave the reader to wonder if those results might have broader implication, but Wikipedia in talking about health outcomes forbids original interpretations about primary studies or presenting results suggestively in a way that would lead people to believe there is a meaningful way to look at the data and draw a conclusion. Primary studies are great as a foundation for proposing more primary studies or making review articles, but especially in health they are not good as a basis for describing the state of health practice. This is not only a Wikipedia rule – this applies to all research. Wikipedians instinctively scrutinize health claims made based on primary sources. I would not want anyone making assumptions about the state of US health education based on what this one study reports.

Who reviews content submitted to Wikipedia?

The best answer to this is probably “Volunteers who are in sync with the Wikipedia hivemind”. There are broadly two kinds of Wikipedia contributors – content creators and functionaries. Most Wikipedia contributors are casual users who submit content, and may not engage with the project regularly. They do not routinely do things like copy editing, new user welcoming, mediating talk page discussion, or quick quality review in response to automated signals. In this case and in most cases, when someone is reviewing content for quality, they are probably an experienced Wikipedia and probably they have already gone through several thousand rounds of social vetting by not less than 100 other Wikipedia contributors. This level of review can be presumed for anyone who has made, say, 1000 edits to Wikipedia, which is the equivalent of about 3 months participation by someone who has given good consideration to about 3-5 articles. That level would still be called “new” but probably not called “newbie”.

What is “WikiProject Medicine” and to what extent does it matter?

WikiProject Medicine is a forum where people who edit Wikipedia’s medical content meet. This community also reviews edits to Wikipedia medical content. This community matters when anyone submits content to Wikipedia’s medical articles, because this community will review those edits and engage with the editor who posted them.

Among demographics who embrace Wikipedia community practices, like using citations, citing reliable sources, and keeping conversation brief, WikiProject Medicine is respected. The most common criticisms that I see directed at WikiProject Medicine is that it enforces the use of citations, is biased to evidence-based medicine, and that its members avoid conversations with people who want compromise on those points especially before trying the standard practices.

How can anyone review content on Wikipedia?

WikiProject Medicine members are mostly not experts. Internal surveys have shown that a qualified demographic does participate there, but that is not the point. The review process on Wikipedia is simplistic, and by following some simple rules, a person of almost any level of education can direct powerful, insightful, and compelling criticism at the work of people with the most technical knowledge. Here is the basic review process. For any given content –

1. Are citations presented for every statement made?
2. Are the sources presented reliable?
3. Does the content submitted mirror the presentation of that content in the cited source?
4. Is the content written in a way that a typical reader can understand?
5. If there is problem or doubt, will the editor discuss with the reviewer, keeping focus on the above questions?

Note that comprehension of the text is hardly a qualification for the person reviewing it. Asking these questions is a good life skill for assessing any sort of information, not just in Wikipedia.

An additional concern of reviewers is “Is this submission in the right place?” In the case of this student submission, it was eventually moved to its own page about education in the United States for people with a medical condition. The content was too specific to be of general interest. For example to the majority of people reading this are not in the United States, and have less need for information about US-specific resources. Also there is not supporting evidence that general readers want a major focus on education, instead of a summary as with other subtopics and links to more information.

Conclusion

After resolving the discussion and getting leave to post into an autism article focusing on “society” issues, it seems that the student left Wikipedia without posting their content. I do not think there was a good bye or explanation. They even said that they would do more, but did not go through. Maybe class ended, or maybe they lost interest. Maybe they did not like the Wikipedia review, or maybe they did not enjoy Wikipedia editing. Maybe they did not know what happened. Maybe they had second thoughts about the sources they were citing, because the discussion about their sources had hardly begun and there would have been problems with their statements not mirroring the conclusions of the primary sources they cited.

This did not come to a desirable outcome for the Wikipedia volunteers. It must not have been entirely positive for the student or teacher. It was a positive outcome for Wikipedia, because the quality control process worked as it should.

I regret to say that I have heard some people in Wikipedia and education say that medicine should be avoided, so that what happened here is not repeated. I think this is unfortunate, especially since what sometimes happens instead is that students are directed to try to edit unpopular, low-impact articles in Wikipedia on the premise that other Wikipedians will not review their work closely there during the class. To me, having peer review is a benefit of using Wikipedia, not a drawback, and posting content which gets read is the point of posting on Wikipedia instead of any private, unpopular media. There hardly has been any community or educator discussion or essays on what ought to be normal, and how the Wikipedia Education Program ought to be. I am not aware of anyone else trying to present a typical case of problems with student-Wikipedian interaction, as I am doing here.