no code implementations • EMNLP (NLPOSS) 2020 • Nitin Madnani, Anastassia Loukina
For the last 5 years, we have developed and maintained RSMTool – an open-source tool for evaluating NLP systems that automatically score written and spoken responses.
no code implementations • 17 Aug 2020 • Anastassia Loukina, Keelan Evanini, Matthew Mulholland, Ian Blood, Klaus Zechner
However, these differences do not lead to differences in human or automated scores of English language proficiency.
no code implementations • WS 2020 • Anastassia Loukina, Nitin Madnani, Aoife Cahill, Lili Yao, Matthew S. Johnson, Brian Riordan, Daniel F. McCaffrey
The effect of noisy labels on the performance of NLP systems has been studied extensively for system training.
no code implementations • WS 2019 • Anastassia Loukina, Nitin Madnani, Klaus Zechner
We illustrate that total fairness may not be achievable and that different definitions of fairness may require different solutions.
no code implementations • ACL 2019 • Nitin Madnani, Beata Beigman Klebanov, Anastassia Loukina, Binod Gyawali, Patrick Lange, John Sabatini, Michael Flor
Literacy is crucial for functioning in modern society.
no code implementations • WS 2018 • Anastassia Loukina, Klaus Zechner, James Bruno, Beata Beigman Klebanov
In this paper we compare the performance of an automated speech scoring engine using two corpora: a corpus of almost 700, 000 randomly sampled spoken responses with scores assigned by one or two raters during operational scoring, and a corpus of 16, 500 exemplar responses with scores reviewed by multiple expert raters.
no code implementations • NAACL 2018 • Su-Youn Yoon, Aoife Cahill, Anastassia Loukina, Klaus Zechner, Brian Riordan, Nitin Madnani
In large-scale educational assessments, the use of automated scoring has recently become quite common.
no code implementations • NAACL 2018 • Anastassia Loukina, Van Rynald T. Liceralde, Beata Beigman Klebanov
Using a case study, we show that variation in oral reading rate across passages for professional narrators is consistent across readers and much of it can be explained using features of the texts being read.
no code implementations • WS 2017 • Anastassia Loukina, Nitin Madnani, Aoife Cahill
We consider the automatic scoring of a task for which both the content of the response as well its spoken fluency are important.
no code implementations • WS 2017 • Nitin Madnani, Anastassia Loukina, Aoife Cahill
We explore various supervised learning strategies for automated scoring of content knowledge for a large corpus of 130 different content-based questions spanning four subject areas (Science, Math, English Language Arts, and Social Studies) and containing over 230, 000 responses scored by human raters.
no code implementations • WS 2017 • Beata Beigman Klebanov, Anastassia Loukina, John Sabatini, Tenaha O{'}Reilly
This paper is a preliminary report on using text complexity measurement in the service of a new educational application.
no code implementations • WS 2017 • Nitin Madnani, Anastassia Loukina, Alina von Davier, Jill Burstein, Aoife Cahill
Automated scoring of written and spoken responses is an NLP application that can significantly impact lives especially when deployed as part of high-stakes tests such as the GRE® and the TOEFL®.
no code implementations • COLING 2016 • Anastassia Loukina, Su-Youn Yoon, Jennifer Sakano, Youhua Wei, Kathy Sheehan
In this paper we explore to what extent the difficulty of listening items in an English language proficiency test can be predicted by the textual properties of the prompt.