no code implementations • Findings (ACL) 2022 • Vatsal Raina, Mark Gales
In terms of an MRC system this means that the system is required to have an idea of the uncertainty in the predicted answer.
Ranked #8 on Reading Comprehension on ReClor
no code implementations • 16 Apr 2024 • Vatsal Raina, Mark Gales
Additionally, zero-shot comparative assessment is more effective at difficulty ranking than the absolute assessment and even the task transfer approaches at question difficulty ranking with a Spearman's correlation of 40. 4%.
1 code implementation • 1 Feb 2024 • Luran Wang, Mark Gales, Vatsal Raina
This work provides an information-theoretic framework to analyse the influence of inputs for text classification tasks.
1 code implementation • 15 Nov 2023 • Nataliia Molchanova, Vatsal Raina, Andrey Malinin, Francesco La Rosa, Adrien Depeursinge, Mark Gales, Cristina Granziera, Henning Muller, Mara Graziani, Meritxell Bach Cuadra
The results from a multi-centric MRI dataset of 334 patients demonstrate that our proposed measures more effectively capture model errors at the lesion and patient scales compared to measures that average voxel-scale uncertainty values.
no code implementations • 8 Nov 2023 • Vatsal Raina, Adian Liusie, Mark Gales
Specifically, we define quality in terms of the incorrectness, plausibility and diversity of the distractor options.
no code implementations • 22 Sep 2023 • Asma Farajidizaji, Vatsal Raina, Mark Gales
We also find greater drops in semantic and lexical similarity between the source and target texts with greater shifts in the readability.
no code implementations • 3 Jul 2023 • Vatsal Raina, Adian Liusie, Mark Gales
Multiple-choice reading and listening comprehension tests are an important part of language assessment.
no code implementations • 22 Jun 2023 • Adian Liusie, Vatsal Raina, Andrew Mullooly, Kate Knill, Mark J. F. Gales
Multiple choice exams are widely used to assess candidates across a diverse range of domains and tasks.
1 code implementation • 8 Jun 2023 • Potsawee Manakul, Yassir Fathullah, Adian Liusie, Vyas Raina, Vatsal Raina, Mark Gales
In this paper, we consider the challenge of summarizing patients' medical progress notes in a limited data setting.
1 code implementation • 10 Feb 2023 • Vatsal Raina, Nataliia Molchanova, Mara Graziani, Andrey Malinin, Henning Muller, Meritxell Bach Cuadra, Mark Gales
This work describes a detailed analysis of the recently proposed normalised Dice Similarity Coefficient (nDSC) for binary segmentation tasks as an adaptation of DSC which scales the precision at a fixed recall rate to tackle this bias.
1 code implementation • 13 Nov 2022 • Adian Liusie, Vatsal Raina, Mark Gales
Two metrics are described: the expected number of options, which measures whether a passage-free system can identify the answer a question using world knowledge; and the contextual mutual information, which measures the importance of context for a given question.
1 code implementation • 9 Nov 2022 • Nataliia Molchanova, Vatsal Raina, Andrey Malinin, Francesco La Rosa, Henning Muller, Mark Gales, Cristina Granziera, Mara Graziani, Meritxell Bach Cuadra
This paper focuses on the uncertainty estimation for white matter lesions (WML) segmentation in magnetic resonance imaging (MRI).
no code implementations • 23 Sep 2022 • Vatsal Raina, Mark Gales
Applying n-gram based approaches is challenging for this form of system as the reference set is unlikely to capture the full range of possible questions and answer options.
2 code implementations • 30 Jun 2022 • Andrey Malinin, Andreas Athanasopoulos, Muhamed Barakovic, Meritxell Bach Cuadra, Mark J. F. Gales, Cristina Granziera, Mara Graziani, Nikolay Kartashev, Konstantinos Kyriakopoulos, Po-Jui Lu, Nataliia Molchanova, Antonis Nikitakis, Vatsal Raina, Francesco La Rosa, Eli Sivena, Vasileios Tsarsitalidis, Efi Tsompopoulou, Elena Volf
This creates a need to be able to assess how robustly ML models generalize as well as the quality of their uncertainty estimates.
3 code implementations • 15 Jul 2021 • Andrey Malinin, Neil Band, Ganshin, Alexander, German Chesnokov, Yarin Gal, Mark J. F. Gales, Alexey Noskov, Andrey Ploskonosov, Liudmila Prokhorenkova, Ivan Provilkov, Vatsal Raina, Vyas Raina, Roginskiy, Denis, Mariya Shmatova, Panos Tigas, Boris Yangel
However, many tasks of practical interest have different modalities, such as tabular data, audio, text, or sensor data, which offer significant challenges involving regression and discrete or continuous structured prediction.
Ranked #2 on Weather Forecasting on Shifts
no code implementations • 9 Jul 2021 • Vatsal Raina, Mark J. F. Gales
The SQA task considered in this paper is to extract the answer from a candidate$\text{'}$s spoken response to a question in a prompt-response style language assessment test.
no code implementations • WS 2020 • Vatsal Raina, Mark Gales, Kate Knill
This paper examines one form of spoken language assessment; whether the response from the candidate is relevant to the prompt provided.