LLMs vs. human moderators
Watch Your Language: Investigating Content Moderation with Large Language Models
Deepak Kumar, Yousef Anees AbuHashem, and Zakir Durumeric. ICWSM 2024.
Content moderation has always been a job for thousands of human reviewers. Right after ChatGPT came out, everyone started asking the obvious question: could a model just do it instead?
So we measured it. We took GPT-3, GPT-3.5, GPT-4, Gemini Pro, and LLAMA 2 and put them through two jobs a real moderator does: enforcing one community’s specific rules, and catching toxic content in general.
For the first, we spun up 95 subreddit-specific moderators by prompting GPT-3.5 with each community’s actual rules, then fed them real comments. For the second, we ran the models against commercial classifiers on a balanced set of 10,000 comments.
It can enforce rules, about two thirds of the time
- Median accuracy across 95 subreddits
- 64%
- Median precision
- 83%
- Best community (r/movies)
- 82% accuracy, 95% precision
That precision number is the encouraging one. When GPT-3.5 flagged a comment, it was usually right. The accuracy number is the sobering one: it missed roughly a third of violations.
Performance also swung wildly between communities. r/movies got close to human-moderator quality. Communities whose rules depend on conversational context or implicit meaning did far worse, because that is exactly where the model has to infer something nobody wrote down.
The most useful finding for anyone actually deploying this: the model did better with restrictive rules (don’t do X) than prescriptive ones (be welcoming, contribute constructively). If you are writing community policy with an LLM in the loop, phrasing matters more than you would expect.
On toxicity, it beats the incumbents
Commodity LLMs outperformed the commercial toxicity classifiers people are already paying for. GPT-3.5 was the most balanced, at 0.73 accuracy and 0.75 F1, against baselines that typically land around 0.60 to 0.65.
The surprise: scaling up barely helped. GPT-3, GPT-3.5, GPT-4, and Gemini Pro all landed in roughly the same place on toxicity detection, which looks a lot like a plateau. Bigger model, same answer.
What to do with this
Platform operators
Useful for communities with clear, restrictive rules, but keep a human in the loop. A 64% median accuracy means about one in three violations walks right past.
Policy designers
Write rules as restrictions rather than aspirations. Context-dependent rules will keep breaking automated systems no matter which model you point at them.
LLMs are a real step forward for moderation. They are not a replacement for judgment, and the gap shows up exactly where communities are most human.