Based on community feedback and to help us ensure we can best serve members, we've completed a thorough review of moderation in the
Politics, Religion, Social Issues (PRSI) forum. Additionally, and as a result of both this review and community feedback, we've made some changes to clarify and strengthen our rules around hate speech, discrimination and group slurs. If you're interested in just these changes but not in the full review, you can see them at the bottom of this post.
Background and scope
Over time, the proportion of moderator time spent in the PRSI forum has greatly increased. Since this forum is not the focus on the site, we discussed options to reduce this workload, including closing the forum completely. In April 2017, we decided to keep the forum open, but implemented a
new policy that tightens moderation in the PRSI forum. In particular, members making three violations of the
Rules for Appropriate Debate. within a six month period will generally permanently lose access to the PRSI forum, even though they may retain rights to the rest of the forums. The goal of this change was to focus moderator attention on the most important parts of the forums, without removing a forum that many members find valuable.
Reviewing this policy, and moderation in the PRSI forum as a whole, helps us to investigate whether the policy has been effective in its goals, as well as addressing potential concerns that members have raised over the last year.
The review sets out to answer the following questions:
- Is the policy meeting its goals of reducing moderator workload in the PRSI forum?
- Is there any political bias by the moderators, collectively or individually?
- Are processes being followed correctly by the moderation team, and is moderation fair and consistent?
- Are there changes to rules or policies that would make rules clearer to members or improve the way the forums are moderated?
In order to do the review, I analysed the following:
- Moderator documentation of the violations that lead to removal of PRSI access for all members since the new policy was enacted
- The underlying posts, reports and (where applicable) appeals for the above violations for the first 50 members who had their PRSI access removed
- Posts made in the PRSI forum by every member who had their PRSI access removed, in order to infer their political leaning to investigate possible moderator bias
- Post report statistics for the PRSI forum over the last several years
- PRSI forum activity in relation to the above
Is the policy meeting its goals of reducing moderator workload in the PRSI forum?
The primary goal of the new policy was to reduce moderator workload in the PRSI forum, with secondary goals of keeping the forum open to members who find it valuable and encouraging constructive discourse via the
Rules for Appropriate Debate.
Since the policy was enacted, 80 members have had their access to the PRSI forum removed, or an average of about three and a half per month. Over the first year of the policy, there was an increasing trend in the number of members with access removed per month, as would be expected as it transitioned in (for example, no one had their access removed in the first month due to the short timespan in which they could have violated the rules for appropriate debate). Since then, the trend has been less clear, but overall there appears to be a decreasing trend in the last year, suggesting some improvements in forum decorum. However, it's also possible that some of the recent improvement was at least in part due to it coinciding with a post-election period in the US where debates may have been less contentious.
View attachment 835316
For the next part of the analysis, I examined reports made in the PRSI forum. Perhaps of most relevance is the subset of these that are related to the Rules for Appropriate Debate (RfAD), which relate most commonly to rules about personal attacks and trolling - with only the very rare exception, these have been the reasons that members received "strikes" that were considered under the new policy. These reports are detected by keyword analysis in the reasons members provide when reporting posts, and are likely an underestimate of the true number, but provide useful data when examining trends.
The number of RfAD reports within the PRSI forum has been fairly stable since the new policy was enacted, but decreased slightly in the last six months:
View attachment 835317
For a more complete picture, another relevant variable is the PRSI forum activity in general. We don't have data on posts made in the forum over time, but have approximated data for the number of new threads created per month. Using this, we can calculate the estimated number of RfAD reports per PRSI thread to get an idea of the level of discourse:
View attachment 835318

There is an increasing trend in this rate for most of the period since the policy was enacted. This is due to the relatively more stable rate of reports while the forum activity in general has dropped significantly since the 2016 election (but is still significantly higher than 2015). There has been, however, a significant drop in this rate since late 2018, which coincided with a lower rate of reports rather than a drop in forum activity.
It is difficult to make any concrete conclusions from this data, as the period since the policy was enacted coincides with an unusually divisive period of US politics. This makes it hard to tease apart the effects of the policy and underlying changes in political discourse in society at large. For example, while the above graph shows a generally increasing trend in RfAD reports per thread from mid 2017 to late 2018, it's possible there could have been an even steeper trend had we not enacted the policy. Subjectively, staff have noted a lower workload from dealing with contacts regarding PRSI moderation.
Is there any political bias by the moderators, collectively or individually?
Our policy is that we allow all political opinions as long as they are expressed within the forum rules. Nevertheless, I reviewed the moderation records to examine whether there was any bias in practice.
Looking at PRSI moderation as a whole, I saw no evidence of bias. 53% of members with PRSI access removed had a conservative political leaning, compared to 47% with a liberal political leaning. The distribution of severity of violations was also similar on both sides. Although we don't have statistics on the political leanings of PRSI members as a whole, these numbers do not suggest any systemic bias.
Looking at particular moderators, I looked at the individual violations that lead to removal of PRSI access and noted the moderators who actioned each of these and the political leaning of the moderated members. I compared the statistics for each individual moderator to the overall moderation team statistics above, looking for statistically significant differences using a 95% confidence interval. I also examined many of the underlying violations, and found that the severity of the underlying violations handled by each moderator was similar regardless of political leaning of the moderated members. Overall, I found no evidence of political bias by any of the moderators.
We can't rule out bias in terms of what members report (e.g., members of a certain political persuasion being more likely to report posts they disagree with), nor do we have any evidence that this does happen, as we don't track data that would be needed to investigate this. However, even if it were the case, the findings above suggest that it does not appear to impact moderation in any noticeable way. Also of note is that reporting a post doesn't necessarily mean it will result in any moderation, since moderators review reported posts for compliance with the rules rather than blindly acting just because a report has been made. In 2017, roughly two thirds of reports in the PRSI forum resulted in some moderator action, and this dropped further to 56% in 2018 - considerably below the
2017 forum-wide rate of 82%.
Are processes being followed correctly by the moderation team, and is moderation fair and consistent?
Of the 80 members who have had their PRSI access removed under the policy, 30 contacted us either to request clarification on the moderation or to appeal it. Appeals frequently included attacks against us and accusations of bias, and we note that contacts taking this aggressive tone tend to reinforce the reasons that access was removed rather than help the member contacting us. However, in four cases, we reinstated PRSI access on appeal where we discovered that we had erred. Some members also requested access to be reinstated after their access having been revoked for over a year. Access was also reinstated in some, but not all, of these cases - we consider these requests on a case by case basic and use discretion based on a number of factors, such as the severity of the original violations, any evidence to suggest that the member has changed their behavior, and whether the member is active in other parts of the forums.
All 80 PRSI bans correctly met the criteria of the three violations occurring within a six month period (although we reserve the right to make exceptions to this time criteria to fit with the spirit of the rule). The average time period between the first and third violations was 85 days (just under three months).
There was some inconsistency around handling of strikes that had overlapping timeframes, e.g. where the third violation was posted before receiving a warning for the second violation. Four PRSI bans occurred under these circumstances, while another eventual ban included a case were a strike wasn't counted because of this. This doesn't include additional cases that have likely occurred where the member still has PRSI access, which I didn't review. There were also two members who received a warning minutes before they posted their next violation, so they might not have seen or read the warning yet. After discussion with staff, we have decided that a standard policy needn't be applied here, but rather discretion based on the severity of the violations and whether there would be a reasonable expectation for the member to be aware they were violations the rules for appropriate debate.
I examined approximately 150 individual violations to check whether they met the criteria as outlined in the policy. The vast majority met the criteria under the RfAD. In some of the early months of the policy, there were rare cases of moderation for other issues being considered as a strike, although there were no cases of this since the start of 2018. Overall, despite rare errors, moderation was consistent.
Is the policy fair and consistent?
About 600 members have had posts reported in the PRSI forum since the policy was enacted. Of the 20 most reported members, 65% have lost PRSI access. For the other 35%, I confirmed that none of them should have lost access under the policy. Examining some of the reports made against these members who did not lose access, reports appeared to have been handled correctly, and were most often not rule violations, or were for more minor rule violations that don't fall under the Rules for Appropriate Debate. Sometimes members would have a large number of (rejected) reports made against their posts by a small number of people of differing political views. These findings suggest that the policy is correctly targeting the most problematic members who create a disproportionate amount of moderator workload, while also confirming that members are not unfairly targeted just because those who disagree with them report their posts.
Although the policy was generally applied correctly, a secondary question is whether there are any modifications we can make to the policy to better achieve our goals. From a moderator perspective, we would like to spend fewer resources dealing with PRSI reports. From a community perspective, we would like to facilitate better conversations amongst members, as well as have the community as large feel confident in the moderation processes and staff.
Changes to rules and policies
As a result of the review and subsequent discussions with other staff, we are making the following changes and clarifications:
- We are creating a more explicit rule prohibiting hate speech, and group slurs/discrimination, under both the general rules and RfAD. Previously many of these violations were classified under different rules, such as trolling. Having a separate rule make our warnings about violations clearer to members, as well as making it clearer what we do and don't allow. The new rule also increases the scope of what we consider a violation, in line with changing community expectations. The new rule is as follows:
The new rule takes effect for any posts made after this announcement.
- Refusal to cite sources, which is part of the RfAD, will not generally be counted as a PRSI strike. However, it is still moderated under the forum rules as normal, and persistently violating this rule may be counted as a strike. This is consistent with how we have implemented the policy in practice, but has not previously been communicated to members.
- If we decide to reinstate a member's PRSI access after it was previously removed, there will be a reduced tolerance of rule violations; two violations of the RfAD within six months will result in a permanent loss of access from the PRSI forum, with no option of later reversal except in the case of moderator error.
- Although errors have been rare, we are making some minor internal process changes to reduce the risk of errors and improve oversight.
