no code implementations • ACL 2022 • Yi Zhou, Masahiro Kaneko, Danushka Bollegala
Sense embedding learning methods learn different embeddings for the different senses of an ambiguous word.
1 code implementation • COLING 2022 • Koki Maeda, Masahiro Kaneko, Naoaki Okazaki
Correlations between IMPARA and human scores indicate that IMPARA is comparable or better than existing evaluation methods.
1 code implementation • AACL (WAT) 2020 • Zizheng Zhang, Tosho Hirasawa, Wei Houjing, Masahiro Kaneko, Mamoru Komachi
New things are being created and new words are constantly being added to languages worldwide.
no code implementations • AACL (WAT) 2020 • Hiroto Tamura, Tosho Hirasawa, Masahiro Kaneko, Mamoru Komachi
Subsequently, we pretrain a translation model on the augmented noisy data, and then fine-tune it on the clean data.
no code implementations • MTSummit 2021 • Raj Dabre, Aizhan Imankulova, Masahiro Kaneko
To this end and in this paper and we propose wait-k simultaneous document-level NMT where we keep the context encoder as it is and replace the source sentence encoder and target language decoder with their wait-k equivalents.
no code implementations • COLING 2022 • Masahiro Kaneko, Danushka Bollegala, Naoaki Okazaki
We study the relationship between task-agnostic intrinsic and task-specific extrinsic social bias evaluation measures for MLMs, and find that there exists only a weak correlation between these two types of evaluation measures.
no code implementations • LREC 2022 • Yujin Takahashi, Masahiro Kaneko, Masato Mita, Mamoru Komachi
This study investigates how supervised quality estimation (QE) models of grammatical error correction (GEC) are affected by the learners’ proficiency with the data.
1 code implementation • 17 Apr 2024 • Masahiro Kaneko, Youmi Ma, Yuki Wata, Naoaki Okazaki
In this study, we propose a Sampling-based Pseudo-Likelihood (\textbf{SPL}) method for MIA (\textbf{SaMIA}) that calculates SPL using only the text generated by an LLM to detect leaks.
no code implementations • 24 Mar 2024 • Masahiro Kaneko, Timothy Baldwin
In this paper, we conduct an experimental survey to elucidate the relationship between the leakage rate and both the output rate and detection rate for personal information, copyrighted texts, and benchmark data.
no code implementations • 25 Feb 2024 • Masanari Ohi, Masahiro Kaneko, Ryuto Koike, Mengsay Loem, Naoaki Okazaki
In this paper, we investigate the presence and impact of likelihood bias in LLM-based evaluators.
no code implementations • 22 Feb 2024 • Masahiro Kaneko, Danushka Bollegala, Timothy Baldwin
The existing evaluation metrics and methods to address these ethical challenges use datasets intentionally created by instructing humans to create instances including ethical problems.
no code implementations • 28 Jan 2024 • Masahiro Kaneko, Danushka Bollegala, Naoaki Okazaki, Timothy Baldwin
In this study, we examine the impact of LLMs' step-by-step predictions on gender bias in unscalable tasks.
no code implementations • 16 Jan 2024 • Masahiro Kaneko, Danushka Bollegala, Timothy Baldwin
Moreover, the performance degradation due to debiasing is also lower in the ICL case compared to that in the FT case.
no code implementations • 14 Nov 2023 • Mengsay Loem, Masahiro Kaneko, Naoaki Okazaki
Large Language Models (LLMs) can justify or critique their predictions through discussions with other models or humans, thereby enriching their intrinsic understanding of instances.
1 code implementation • 14 Nov 2023 • Ryuto Koike, Masahiro Kaneko, Naoaki Okazaki
Furthermore, our analysis indicates that the high instruction-following ability of LLMs fosters the large impact of such constraints on detection performance.
1 code implementation • 20 Sep 2023 • Masahiro Kaneko, Naoaki Okazaki
Generating explanations for GEC corrections involves aligning input and output tokens, identifying correction points, and presenting corresponding explanations consistently.
1 code implementation • 18 Sep 2023 • Panatchakorn Anantaprayoon, Masahiro Kaneko, Naoaki Okazaki
In Natural Language Inference (NLI), existing bias evaluation methods have focused on the prediction results of one specific label out of three labels, such as neutral.
no code implementations • 16 Sep 2023 • Masahiro Kaneko, Danushka Bollegala, Naoaki Okazaki
In this study, we compare the impact of debiasing on performance across multiple downstream tasks using a wide-range of benchmark datasets that containing female, male, and stereotypical words.
1 code implementation • 13 Sep 2023 • Daisuke Oba, Masahiro Kaneko, Danushka Bollegala
We show that, using CrowsPairs dataset, our textual preambles covering counterfactual statements can suppress gender biases in English LLMs such as LLaMA2.
1 code implementation • 21 Jul 2023 • Ryuto Koike, Masahiro Kaneko, Naoaki Okazaki
Experiments in the domain of student essays show that the proposed detector improves the detection performance on the attacker-generated texts by up to +41. 3 points F1-score.
no code implementations • 29 May 2023 • Mengsay Loem, Masahiro Kaneko, Sho Takase, Naoaki Okazaki
Large-scale pre-trained language models such as GPT-3 have shown remarkable performance across various natural language processing tasks.
no code implementations • 19 May 2023 • Masahiro Kaneko, Naoaki Okazaki
Experiments show that the proposed method achieves comparable performance to the baseline in four tasks, paraphrasing, formality style transfer, GEC, and text simplification, despite reducing the length of the target text by as small as 21%.
1 code implementation • 19 May 2023 • Masahiro Kaneko, Graham Neubig, Naoaki Okazaki
Humans work together to solve common problems by having discussions, explaining, and agreeing or disagreeing with each other.
no code implementations • 28 Jan 2023 • Masahiro Kaneko, Danushka Bollegala, Naoaki Okazaki
Prior works have relied on human annotated examples to compare existing intrinsic bias evaluation measures.
no code implementations • 6 Oct 2022 • Masahiro Kaneko, Danushka Bollegala, Naoaki Okazaki
We study the relationship between task-agnostic intrinsic and task-specific extrinsic social bias evaluation measures for Masked Language Models (MLMs), and find that there exists only a weak correlation between these two types of evaluation measures.
no code implementations • 27 Jul 2022 • Mengsay Loem, Sho Takase, Masahiro Kaneko, Naoaki Okazaki
Impressive performance of Transformer has been attributed to self-attention, where dependencies between entire input in a sequence are considered at every position.
no code implementations • 19 May 2022 • Masahiro Kaneko, Danushka Bollegala, Naoaki Okazaki
Different methods have been proposed to develop meta-embeddings from a given set of source embeddings.
1 code implementation • NAACL 2022 • Masahiro Kaneko, Aizhan Imankulova, Danushka Bollegala, Naoaki Okazaki
Unfortunately, it was reported that MLMs also learn discriminative biases regarding attributes such as gender and race.
1 code implementation • ACL 2022 • Masahiro Kaneko, Sho Takase, Ayana Niwa, Naoaki Okazaki
In this study, we introduce an Example-Based GEC (EB-GEC) that presents examples to language learners as a basis for a correction result.
1 code implementation • 14 Mar 2022 • Yi Zhou, Masahiro Kaneko, Danushka Bollegala
Sense embedding learning methods learn different embeddings for the different senses of an ambiguous word.
no code implementations • 17 Jan 2022 • Yujin Takahashi, Masahiro Kaneko, Masato Mita, Mamoru Komachi
This study investigates how supervised quality estimation (QE) models of grammatical error correction (GEC) are affected by the learners' proficiency with the data.
no code implementations • NAACL (ACL) 2022 • Mengsay Loem, Sho Takase, Masahiro Kaneko, Naoaki Okazaki
Through experiments, we show that ExtraPhrase improves the performance of abstractive summarization tasks by more than 0. 50 points in ROUGE scores compared to the setting without data augmentation.
no code implementations • NAACL 2021 • Seiichiro Kondo, Kengo Hotate, Masahiro Kaneko, Mamoru Komachi
It is assumed that this issue is caused by insufficient number of long sentences in the training data.
no code implementations • NAACL 2021 • Aomi Koyama, Kengo Hotate, Masahiro Kaneko, Mamoru Komachi
Therefore, GEC studies have developed various methods to generate pseudo data, which comprise pairs of grammatical and artificially produced ungrammatical sentences.
no code implementations • 15 Apr 2021 • Raj Dabre, Aizhan Imankulova, Masahiro Kaneko, Abhisek Chakrabarty
Parallel corpora are indispensable for training neural machine translation (NMT) models, and parallel corpora for most language pairs do not exist or are scarce.
1 code implementation • 15 Apr 2021 • Masahiro Kaneko, Danushka Bollegala
To overcome the above-mentioned disfluencies, we propose All Unmasked Likelihood (AUL), a bias evaluation measure that predicts all tokens in a test case given the MLM embedding of the unmasked input.
2 code implementations • EACL 2021 • Masahiro Kaneko, Danushka Bollegala
In comparison to the numerous debiasing methods proposed for the static non-contextualised word embeddings, the discriminative biases in contextualised embeddings have received relatively little attention.
1 code implementation • EACL 2021 • Masahiro Kaneko, Danushka Bollegala
Word embeddings trained on large corpora have shown to encode high levels of unfair discriminatory gender, racial, religious and ethnic biases.
no code implementations • COLING 2020 • Ikumi Yamashita, Satoru Katsumata, Masahiro Kaneko, Aizhan Imankulova, Mamoru Komachi
Cross-lingual transfer learning from high-resource languages (the source models) is effective for training models of low-resource languages (the target models) for various tasks.
no code implementations • COLING 2020 • Kengo Hotate, Masahiro Kaneko, Mamoru Komachi
In this study, we propose a beam search method to obtain diverse outputs in a local sequence transduction task where most of the tokens in the source and target sentences overlap, such as in grammatical error correction (GEC).
1 code implementation • COLING 2020 • Ryoma Yoshimura, Masahiro Kaneko, Tomoyuki Kajiwara, Mamoru Komachi
We propose a reference-less metric trained on manual evaluations of system outputs for grammatical error correction (GEC).
no code implementations • COLING 2020 • Masahiro Kaneko, Danushka Bollegala
Prior work investigating the geometry of pre-trained word embeddings have shown that word embeddings to be distributed in a narrow cone and by centering and projecting using principal component vectors one can increase the accuracy of a given set of pre-trained word embeddings.
no code implementations • Findings of the Association for Computational Linguistics 2020 • Masato Mita, Shun Kiyono, Masahiro Kaneko, Jun Suzuki, Kentaro Inui
Existing approaches for grammatical error correction (GEC) largely rely on supervised learning with manually created GEC datasets.
no code implementations • WS 2020 • Masahiro Kaneko, Aizhan Imankulova, Tosho Hirasawa, Mamoru Komachi
We introduce our TMU system that is submitted to The 4th Workshop on Neural Generation and Translation (WNGT2020) to English-to-Japanese (En→Ja) track on Simultaneous Translation And Paraphrase for Language Education (STAPLE) shared task.
1 code implementation • ACL 2020 • Masahiro Kaneko, Masato Mita, Shun Kiyono, Jun Suzuki, Kentaro Inui
The answer to this question is not as straightforward as one might expect because the previous common methods for incorporating a MLM into an EncDec model have potential drawbacks when applied to GEC.
Ranked #2 on Grammatical Error Correction on JFLEG
1 code implementation • WMT (EMNLP) 2020 • Aizhan Imankulova, Masahiro Kaneko, Tosho Hirasawa, Mamoru Komachi
Simultaneous translation involves translating a sentence before the speaker's utterance is completed in order to realize real-time understanding in multiple languages.
no code implementations • WS 2019 • Aizhan Imankulova, Masahiro Kaneko, Mamoru Komachi
We introduce our system that is submitted to the News Commentary task (Japanese{\textless}-{\textgreater}Russian) of the 6th Workshop on Asian Translation.
no code implementations • WS 2019 • Mio Arai, Masahiro Kaneko, Mamoru Komachi
Existing example retrieval systems do not include grammatically incorrect examples or present only a few examples, if any.
no code implementations • WS 2019 • Masahiro Kaneko, Kengo Hotate, Satoru Katsumata, Mamoru Komachi
Thus, it is not straightforward to utilize language representations trained from a large corpus, such as Bidirectional Encoder Representations from Transformers (BERT), in a form suitable for the learner{'}s grammatical errors.
no code implementations • ACL 2019 • Kengo Hotate, Masahiro Kaneko, Satoru Katsumata, Mamoru Komachi
In this paper, we propose a method for neural grammar error correction (GEC) that can control the degree of correction.
1 code implementation • ACL 2019 • Masahiro Kaneko, Danushka Bollegala
Word embeddings learnt from massive text collections have demonstrated significant levels of discriminative biases such as gender, racial or ethnic biases, which in turn bias the down-stream NLP applications that use those word embeddings.
no code implementations • 15 Apr 2019 • Masahiro Kaneko, Mamoru Komachi
In this work, we investigate the effect of utilizing information not only from the final layer but also from intermediate layers of a pre-trained language representation model to detect grammatical errors.
no code implementations • NAACL 2019 • Masato Mita, Tomoya Mizumoto, Masahiro Kaneko, Ryo Nagata, Kentaro Inui
This study explores the necessity of performing cross-corpora evaluation for grammatical error correction (GEC) models.
1 code implementation • WS 2018 • Masahiro Kaneko, Tomoyuki Kajiwara, Mamoru Komachi
We introduce the TMU systems for the second language acquisition modeling shared task 2018 (Settles et al., 2018).
1 code implementation • IJCNLP 2017 • Masahiro Kaneko, Yuya Sakaizawa, Mamoru Komachi
In this study, we improve grammatical error detection by learning word embeddings that consider grammaticality and error patterns.
Ranked #6 on Grammatical Error Detection on FCE