Keywords: Gamification, Text Classification, R Shiny, Interactive Machine Learning.
Webpages:
https://gmdn.shinyapps.io/Classification/ Supervised machine learning algorithms require a set of labelled examples to be trained; however, the labelling process is a costly, time consuming task. In the last years, mixed approaches that use crowd-sourcing and interactive machine learning (Amershi et al. 2014) have shown that it is possible to create annotated datasets at affordable costs (Morschheuser, Hamari, and Koivisto 2016). One major challenge in motivating people to participate in these labelling tasks is to design a system that promotes and enables the formation of positive motivations towards work as well as fits the type of the activity.
In this context, an approach named ‘gamification’ has become popular. Gamification is defined as ‘the use of game design elements in non-game contexts’ (Deterding et al. 2011), i.e. tipical game elements, like rankings, leaderboards, points, badges, etc, are used for purposes different from their normal expected employment. Nowadays, gamification spreads through a wide range of disciplines and its applications are implemented in different areas. For instance, an increasingly common feature of online communities and social media sites is a mechanism for rewarding user achievements based on a system of badges and points. They have been employed in many domains, including educational sites like
Khan Academy, and tourist review sites like
Tripadvisor. At the most basic level, these game elements serve as a summary of a user’s key accomplishments; however, experience with these sites also shows that users will put in non-trivial amounts of work to achieve particular badges, and as such, badges can act as powerful incentives (Anderson et al. 2013).
In this work, we present the recent studies of gamification in text classification and the development of a Web application written in Shiny (Di Nunzio, Maistro, and Zilio 2016). This application, initially designed to understand probabilistic models, has been redesigned as a game to gather labelled data from lay people, especially kids from primary and secondary schools, during the
European Researchers’ Night in September 2016 at the University of Padua. We have tested this application with a two-fold goal in mind: i) how the gamification of a classification problem can be used to understand what is the `price’ of labelling a small amount of objects for building a reasonably accurate classifier, ii) to analyze the classification performance given the presence of small sample sizes and little training. We will describe three different interfaces and the analysis of the results: a pilot experiment with PhD and post-doc students, a second experiment with primary and secondary school students, and a
third experiment with a computer instsalled in a Bank of the city center.
References Amershi, Saleema, Maya Cakmak, W. Bradley Knox, and Todd Kulesza. 2014. “Power to the People: The Role of Humans in Interactive Machine Learning.”
AI Magazine 35 (4): 105–20.
http://www.aaai.org/ojs/index.php/aimagazine/article/view/2513.
Anderson, Ashton, Daniel Huttenlocher, Jon Kleinberg, and Jure Leskovec. 2013. “Steering User Behavior with Badges.” In
Proceedings of the 22Nd International Conference on World Wide Web, 95–106. WWW ’13. New York, NY, USA: ACM. doi:
10.1145/2488388.2488398.
Deterding, Sebastian, Dan Dixon, Rilla Khaled, and Lennart Nacke. 2011. “From Game Design Elements to Gamefulness: Defining ‘Gamification’.” In
Proc. of the 15th International Academic Mindtrek Conference: Envisioning Future Media Environments, 9–15. MindTrek ’11. New York, NY, USA: ACM. doi:
10.1145/2181037.2181040.
Di Nunzio, Giorgio Maria, Maria Maistro, and Daniel Zilio. 2016. “Gamification for Machine Learning: The Classification Game.” In
Proceedings of the Third International Workshop on Gamification for Information Retrieval Co-Located with 39th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2016), Pisa, Italy, July 21, 2016., 45–52.
http://ceur-ws.org/Vol-1642/paper7.pdf.
Morschheuser, B., J. Hamari, and J. Koivisto. 2016. “Gamification in Crowdsourcing: A Review.” In
2016 49th Hawaii International Conference on System Sciences (Hicss), 4375–84. doi:
10.1109/HICSS.2016.543.
