Progression in a Language Annotation Game with a Purpose

Abstract

Within traditional games design, incorporating progressivedifficulty is considered of fundamental importance. But de-spite the widespread intuition that progression could haveclear benefits in Games-With-A-Purpose (GWAPs)–e.g., fortraining non-expert annotators to produce more complexjudgements– progression is not in fact a prominent featureof GWAPs; and there is even less evidence on its effects. Inthis work we present an approach to progression in GWAPsthat generalizes to different annotation tasks with minimal, ifany, dependency on gold annotated data. Using this methodwe observe a statistically significant increase in accuracy overrandomly showing items to annotators.