Statistical Methods for Annotation Analysis

Silviu Paun got his Ph.D. from the University of Essex in 2017 with a thesis on topic models. Since then he has been at Queen Mary University of London. His research focuses on models of annotation, probabilistic and neural, for creating resources and to more efficiently train machine learning models. His models have been deployed to create the Phrase Detectives coreference corpus, one of the largest crowdsourced NLP corpora, created using the Phrase Detectives Game-With-A-Purpose.Ron Artstein received his Ph.D. in Linguistics from Rutgers University in 2002, held positions at the Technion–Israel Institute of Technology and the University of Essex, and is presently a research scientist at the Institute for Creative Technologies, University of Southern California. His current research focuses on the collection, annotation, and management of linguistic data for human–machine interaction, analysis of corpora, and the evaluation of implemented dialogue systems; he has published work on theoretical and computational linguistics, conversational dialogue systems, and human–agent and human–robot interaction.Massimo Poesio received his Ph.D. from the University of Rochester in 1994. He is a Professor in Computational Linguistics at Queen Mary University of London and a Turing Institute Fellow. His main interests are in anaphora resolution, disagreements in language interpretation, the use of games-with-a-purpose for creating NLP resources, and semantic interpretation in dialogue.

• Preface.- Acknowledgements.- Introduction.- Coefficients of Agreement.- Using Agreement Measures for CL Annotation Tasks.- Probabilistic Models of Agreement.- Probabilistic Models of Annotation.- Learning from Multi-Annotated Corpora.- Bibliography.- Authors' Biographies.