Some online news sources regularly publish inaccurate, sensational articles that often receive disproportionate amounts of attention. When these articles are shared on social media platforms, they sometimes spread more quickly and widely than articles with more measured language and sourcing practices. Responses that work to verify claims or o ffer alternative views could reduce a perceived social consensus supporting the article [1] or even lead to changed beliefs [2]. Yet in online discussions, responding to unreliable articles could also have the eff ect of promoting those articles through algorithms that determine the relative prominence of information on a platform.

In this experiment we ask if using “sticky comments” encouraging norms of evidence-based discussion increases the chance that commenters will link to evidence. We also ask a second set of questions related to reddit’s ranking systems with two “arms,” two different kinds of sticky comments. In one arm of the experiment, we consider if encouraging skepticism causes reddit’s algorithm promoting posts from unreliable websites. We expect the other arm to reduce a post’s prominence in the site’s rankings.

The research site is r/worldnews, a 14.8 million subscriber subreddit community on the social news platform reddit, where links to news are often shared and discussed. Measured in November 2016, this English language community received 914 posts per day on average, 2.4% of which are from tabloid news sites that community members tend to report for being more sensational and less reliably sourced. Of all of these tabloid posts, 46% are permitted by moderators. Among all tabloid posts 78% have at least one comment and 28% have at least one comment with a link. Across all comments in discussions of tabloid posts, 5% of comments include at least one link.

Moderators also report that tabloid posts are often highly recommended in their subreddit by the reddit platform. Even small changes in the reception orprominence of these articles might have a substantial e ffect on the experience of the subreddit’s many readers. On the reddit platform, readers are able to influence the prominence of discussions. They can add “upvotes” or “downvotes” to the thread to influence its relative prominence in the platform’s ranking system. While reddit does not publicize the number of votes received by a discussion, software can regularly query the position of a post on the subreddit’s front page or the site-wide listing of top discussions. Furthermore, the platform does provide information on the “score” of a post, a number partially-based on upvotes and downvotes which is used to determine rankings. In an email on Jan 16, 2017, a reddit employee confirmed to me that: “between two posts of similar age, the one with a higher score will tend to be ranked more highly in the HOT algorithm on average,” especially for the default HOT listing on a subreddit.

What Outcomes Would We Expect?

We expect that posting a message encouraging people to link to evidence will cause people to be more likely link to evidence in their comments. In a previous experiment with r/science, we found that posting sticky comments with the subreddit rules caused newcomers to be more likely to follow subreddit rules. So we expect that our sticky comments will have some effect on commenting behavior.

In this experiment, we also wondered if encouraging fact-checking of submissions might actually cause those submissions to become more promoted by reddit algorithms than they would otherwise. These recommendation systems, which attempt to surface popular and interesting submissions, pay attention to many markers of user activity and use that information to promote some submissions over others. On good days, these algorithms draw user attention to meaningful content; on others, they can concentrate attention toward misinformation and harassment [5]. For this reason, moderators consequently pay close attention to the things that “trend” in their subreddits. In this experiment, we expected that:

How We Tested The Effect of Our Sticky Comments

To test the effect of our sticky comments, I used the /u/CivilServantBot, which continuously monitors all posts and comments in the subreddit, including the actions of moderators. The full experiment design is at osf.io/hmq5m/; here is a brief summary. During the experiment, this bot randomly assigned sticky comments to posts from domains that moderators identified, sites that met two criteria: (a) the subreddit receives large volumes of links from these sites, and (b) community members routinely complain about them for sensationalized headlines and unreliable information. The list does not include many US sites because r/worldnews disallows news from the US:

With this experiment, we tried two different kinds of sticky comments. The first encouraged people to link to further evidence about the topic:

A second sticky comment encouraged people to link to further evidence and also downvote genuinely-unreliable articles:

Within these domains, any post had an equal chance to receive (a) no sticky comment, (b) a the sticky comment encouraging links to evidence, or (c) the encouragement to downvote. Randomizations were conducted in balanced blocks of 12.

Outcome Variables

By comparing these three, we are able to make a causal inference about the effect of the sticky comment on the outcomes variables we care about:

  • the chance of a comment to include links to further evidence

    ** Excluding bots, comments removed by moderators, links to other reddit pages, and links to image sites like giphy and quickmeme (we kept imgur, since its sometimes used for publishing photos of breaking news)

  • We tried to monitor the highest rank achieved by a post in /r/worldnews/HOT, the default subreddit view

    ** Unfortunately, our code only collected the top 100 items, rather than the larger list we intended. So I use the score of the post after 24 hours. CivilServant sampled the post score every four minutes.

During the experiment, reddit made a change to the scores they report on the site. The site formerly held scores at a ceiling, but starting on December 6, the platform begain to report the ``true’’ score for a post. Since we were observing the scores over time, we were able to see the change in score caculations when they occurred. Here is a chart showing the change in score for two very popular posts.