Search Results for author:
Found 47 papers, 6 papers with code
Language Technologies for Low Resource Languages: Sociolinguistic and Multilingual Insights
There is a growing interest in building language technologies (LTs) for low resource languages (LRLs).
A Case Study of Efficacy and Challenges in Practical Human-in-Loop Evaluation of NLP Systems Using Checklist
We conduct experiments for evaluating an offensive content detection system and use a data augmentation technique for improving the model using insights from Checklist.
METAL: Towards Multilingual Meta-Evaluation
This dataset is created specifically to evaluate LLM-based evaluators, which we refer to as meta-evaluation (METAL).
Private Benchmarking to Prevent Contamination and Improve Comparative Evaluation of LLMs
To solve this problem, we propose Private Benchmarking, a solution where test datasets are kept private and models are evaluated without revealing the test data to the model.
DOSA: A Dataset of Social Artifacts from Different Indian Geographical Subcultures
Generative models are increasingly being used in various applications, such as text generation, commonsense reasoning, and question-answering.
A Unified Framework and Dataset for Assessing Gender Bias in Vision-Language Models
Large vision-language models (VLMs) are widely getting adopted in industry and academia.
MAFIA: Multi-Adapter Fused Inclusive LanguAge Models
In this work, we aim to modularly debias a pretrained language model across multiple dimensions.
MAPLE: Multilingual Evaluation of Parameter Efficient Finetuning of Large Language Models
Prior work on multilingual evaluation has shown that there is a large gap between the performance of LLMs on English and other languages.
MEGAVERSE: Benchmarking Large Language Models Across Languages, Modalities, Models and Tasks
We also perform a study on data contamination and find that several models are likely to be contaminated with multilingual evaluation benchmarks, necessitating approaches to detect and handle contamination while assessing the multilingual performance of LLMs.
Representativeness as a Forgotten Lesson for Multilingual and Code-switched Data Collection and Preparation
Multilingualism is widespread around the world and code-switching (CSW) is a common practice among different language pairs/tuples across locations and regions.
Partial Rank Similarity Minimization Method for Quality MOS Prediction of Unseen Speech Synthesis Systems in Zero-Shot and Semi-supervised setting
That is the partial rank similarity is measured (PRS) rather than the individual MOS values as with the L1 loss.
Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation?
Large Language Models (LLMs) excel in various Natural Language Processing (NLP) tasks, yet their evaluation, particularly in languages beyond the top $20$, remains inadequate due to existing benchmarks and metrics limitations.
On Evaluating and Mitigating Gender Biases in Multilingual Settings
While understanding and removing gender biases in language models has been a long-standing problem in Natural Language Processing, prior research work has primarily been limited to English.
Breaking Language Barriers with a LEAP: Learning Strategies for Polyglot LLMs
Our results show substantial advancements in multilingual understanding and generation across a diverse range of languages.
MEGA: Multilingual Evaluation of Generative AI
Most studies on generative LLMs have been restricted to English and it is unclear how capable these models are at understanding and generating text in other languages.
DiTTO: A Feature Representation Imitation Approach for Improving Cross-Lingual Transfer
Zero-shot cross-lingual transfer is promising, however has been shown to be sub-optimal, with inferior transfer performance across low-resource languages.
Fairness in Language Models Beyond English: Gaps and Challenges
With language models becoming increasingly ubiquitous, it has become essential to address their inequitable treatment of diverse demographic groups and factors.
A Survey of Code-switching: Linguistic and Social Perspectives for Language Technologies
To fill this gap, we offer a survey of code-switching (C-S) covering the literature in linguistics with a reflection on the key issues in language technologies.
Benchmarking Evaluation Metrics for Code-Switching Automatic Speech Recognition
Code-switching poses a number of challenges and opportunities for multilingual automatic speech recognition.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
On the Calibration of Massively Multilingual Language Models
Massively Multilingual Language Models (MMLMs) have recently gained popularity due to their surprising effectiveness in cross-lingual transfer.
Beyond Static Models and Test Sets: Benchmarking the Potential of Pre-trained Models Across Tasks and Languages
Although recent Massively Multilingual Language Models (MMLMs) like mBERT and XLMR support around 100 languages, most existing multilingual NLP benchmarks provide evaluation data in only a handful of these languages with little linguistic diversity.
Multilingual CheckList: Generation and Evaluation
We compare the TEA CheckLists with CheckLists created with different levels of human intervention.
A Survey of Multilingual Models for Automatic Speech Recognition
Although Automatic Speech Recognition (ASR) systems have achieved human-like performance for a few languages, the majority of the world's languages do not have usable systems due to the lack of large speech datasets to train these models.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
Predicting the Performance of Multilingual NLP Models
Recent advancements in NLP have given us models like mBERT and XLMR that can serve over 100 languages.
On the Universality of Deep Contextual Language Models
Deep Contextual Language Models (LMs) like ELMO, BERT, and their successors dominate the landscape of Natural Language Processing due to their ability to scale across multiple tasks rapidly by pre-training a single model, followed by task-specific fine-tuning.
GCM: A Toolkit for Generating Synthetic Code-mixed Text
Code-mixing is common in multilingual communities around the world, and processing it is challenging due to the lack of labeled and unlabeled data.
Multilingual and code-switching ASR challenges for low resource Indian languages
For this purpose, we provide a total of ~600 hours of transcribed speech data, comprising train and test sets, in these languages including two code-switched language pairs, Hindi-English and Bengali-English.
mask-Net: Learning Context Aware Invariant Features using Adversarial Forgetting (Student Abstract)
Training a robust system, e. g., Speech to Text (STT), requires large datasets.
Cross-lingual and Multilingual Spoken Term Detection for Low-Resource Indian Languages
Since no publicly available dataset exists for Spoken Term Detection in these languages, we create a new dataset using a publicly available TTS dataset.
GLUECoS: An Evaluation Benchmark for Code-Switched NLP
We present results on all these tasks using cross-lingual word embedding models and multilingual models.
Learning not to Discriminate: Task Agnostic Learning for Improving Monolingual and Code-switched Speech Recognition
Recently, we showed that monolingual ASR systems fine-tuned on code-switched data deteriorate in performance on monolingual speech recognition, which is not desirable as ASR systems deployed in multilingual scenarios should recognize both monolingual and code-switched speech with high accuracy.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Learning to Recognize Code-switched Speech Without Forgetting Monolingual Speech Recognition
In this work, we show that fine-tuning ASR models on code-switched speech harms performance on monolingual speech.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Crowdsourcing Speech Data for Low-Resource Languages from Low-Income Workers
Unfortunately, collecting labelled speech data in any language is an expensive and resource-intensive task.
GLUECoS : An Evaluation Benchmark for Code-Switched NLP
We present results on all these tasks using cross-lingual word embedding models and multilingual models.
A New Dataset for Natural Language Inference from Code-mixed Conversations
Code-mixing is the use of more than one language in the same conversation or utterance, and is prevalent in multilingual communities all over the world.
Unsung Challenges of Building and Deploying Language Technologies for Low Resource Language Communities
In this paper, we examine and analyze the challenges associated with developing and introducing language technologies to low-resource language communities.
CoSSAT: Code-Switched Speech Annotation Tool
Code-switching refers to the alternation of two or more languages in a conversation or utterance and is common in multilingual communities across the world.
End-to-End ASR for Code-switched Hindi-English Speech
While the lack of data adversely affects the performance of end-to-end models, we see promising improvements with MTL and balancing the corpus.
A Survey of Code-switched Speech and Language Processing
Code-switching, the alternation of languages within a conversation or utterance, is a common communicative phenomenon that occurs in multilingual communities across the world.
Word Embeddings for Code-Mixed Language Processing
We compare three existing bilingual word embedding approaches, and a novel approach of training skip-grams on synthetic code-mixed text generated through linguistic models of code-mixing, on two tasks - sentiment analysis and POS tagging for code-mixed text.
Phone Merging For Code-Switched Speech Recognition
Though the best performance gain of 1. 2{\%} WER was observed with manually merged phones, we show experimentally that the manual phone merge is not optimal.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Language Modeling for Code-Mixing: The Role of Linguistic Theory based Synthetic Data
Training language models for Code-mixed (CM) language is known to be a difficult problem because of lack of data compounded by the increased confusability due to the presence of more than one language.
Automatic Speech Recognition (ASR)
Language Identification
+3
Automatic Detection of Code-switching Style from Acoustics
We hypothesize that it may be useful for an ASR system to be able to first detect the switching style of a particular utterance from acoustics, and then use specialized language models or other adaptation techniques for decoding the speech.
Automatic Speech Recognition (ASR)
Language Identification
+1
Polyglot Neural Language Models: A Case Study in Cross-Lingual Phonetic Representation Learning
We introduce polyglot language models, recurrent neural network models trained to predict symbol sequences in many different languages using shared representations of symbols and conditioning on typological information about the language to be predicted.
Speech Synthesis of Code-Mixed Text
Most Text to Speech (TTS) systems today assume that the input text is in a single language and is written in the same language that the text needs to be synthesized in.
