Simulating Information Retrieval Test Collections

David Hawking is an Honorary Professor in the College of Engineering and Computer Science at the Australian National University in Canberra. His research in Information Retrieval began in the early nineteen-nineties at the Australian National University where he worked on parallel and distributed IR, and proximity models of retrieval. With ANU colleagues, he coordinated the Very Large Collection and Web tracks at TREC and co-created a number of test collections, notably WT10g and VLC2. Between 1998 and 2008 he worked for the government research organisation CSIRO and, with colleagues, developed an internet and enterprise retrieval system which was spun off as Funnelback Pty Ltd. David worked as Chief Scientist for Funnelback until 2013 when he joined Microsoft and worked on the Bing search engine. He retired from paid work in 2018.Bodo Billerbeck is an applied data scientist at Microsoft Bing, working mostly on core search problems. His interests lie in finding often data-driven solutions to multiple problems in the search stack, including indexing, query reformulation, matching, answer selection and insertion, as well as evaluation. Since completing his Ph.D. at RMIT University in 2005 he briefly worked at Sensis.com.au, but soon moved on to Microsoft. After spending some time embedded at Microsoft Research in Cambridge, UK, he returned to Australia, and recently has come full circle and is enjoying an honorary fellow position at RMIT.Paul Thomas is an applied scientist at Microsoft. His research is in information retrieval: particularly in how people use web search systems and how we should evaluate these systems, as well as interfaces for search including search with different types of results, search on mobile devices, and search as conversation. He has previously worked at Australia’s CSIRO and the Australian National University.Nick Craswell is a research manager at Microsoft Bing. Since obtaining his Ph.D. from the Australian National University in 2000 on the topic of distributed information retrieval, Nick has worked on enterprise search, expert search, anchor text, click graphs, image search, offline and online evaluation of Web search, query-independent evidence in ranking, evaluation metrics and web ranking and neural ranking models. He has been a driving force in the TREC web tracks and enterprise tracks and was instrumental in creating various widely used test collections.

• Acknowledgments.- Symbols.- Introduction.- Evaluation Approaches.- Modeling Document Lengths.- Modeling Word Frequencies, Assuming Independence.- Modeling Term Dependence.- Modeling Word Strings.- Models of Corpus Growth.- Generation of Compatible Queries.- Proof of the Simulation Pudding.- Speed of Operation.- Leaking Confidential Information.- Discussion, Conclusions,\nobreakspace {.- Bibliography.- Authors' Biographies.