Browse SoTA > Natural Language Processing > Language Modelling

Language Modelling

656 papers with code · Natural Language Processing

Language modeling is the task of predicting the next word or character in a document.

* indicates models using dynamic evaluation; where, at test time, models may adapt to seen tokens in order to improve performance on following tokens. (Mikolov et al., (2010), Kraus et al., (2017))

( Image credit: Exploring the Limits of Language Modeling )

Benchmarks

Latest papers with code

DeLighT: Very Deep and Light-weight Transformer

3 Aug 2020sacmehta/delight

On the WMT'16 En-Ro low resource dataset, DeLighT delivers similar performance with 2. 8 times fewer parameters than baseline transformers.

LANGUAGE MODELLING MACHINE TRANSLATION

105
03 Aug 2020

Language Modelling for Source Code with Transformer-XL

31 Jul 2020Anon-111/Source-Code-Modelling

It has been found that software, like natural language texts, exhibits "naturalness", which can be captured by statistical language models.

LANGUAGE MODELLING

1
31 Jul 2020

NeuralQA: A Usable Library for Question Answering (Contextual Query Expansion + BERT) on Large Datasets

30 Jul 2020victordibia/neuralqa

Existing tools for Question Answering (QA) have challenges that limit their use in practice.

LANGUAGE MODELLING QUESTION ANSWERING

65
30 Jul 2020

What does BERT know about books, movies and music? Probing BERT for Conversational Recommendation

30 Jul 2020Guzpenha/ConvRecProbingBERT

Overall, our analyses and experiments show that: (i) BERT has knowledge stored in its parameters about the content of books, movies and music; (ii) it has more content-based knowledge than collaborative-based knowledge; and (iii) fails on conversational recommendation when faced with adversarial data.

LANGUAGE MODELLING RECOMMENDATION SYSTEMS

2
30 Jul 2020

Mirostat: A Perplexity-Controlled Neural Text Decoding Algorithm

29 Jul 2020basusourya/mirostat

Experiments show that for low values of k and p in top-k and top-p sampling, perplexity drops significantly with generated text length, which is also correlated with excessive repetitions in the text (the boredom trap).

LANGUAGE MODELLING

8
29 Jul 2020

FiSSA at SemEval-2020 Task 9: Fine-tuned For Feelings

24 Jul 2020barfsma/FiSSA

In this paper, we present our approach for sentiment classification on Spanish-English code-mixed social media data in the SemEval-2020 Task 9.

LANGUAGE MODELLING SENTIMENT ANALYSIS

1
24 Jul 2020

The Lottery Ticket Hypothesis for Pre-trained BERT Networks

23 Jul 2020TAMU-VITA/BERT-Tickets

For a range of downstream tasks, we indeed find matching subnetworks at 40% to 90% sparsity.

LANGUAGE MODELLING

14
23 Jul 2020

newsSweeper at SemEval-2020 Task 11: Context-Aware Rich Feature Representations For Propaganda Classification

21 Jul 2020paramansh/propaganda_detection

This paper describes our submissions to SemEval 2020 Task 11: Detection of Propaganda Techniques in News Articles for each of the two subtasks of Span Identification and Technique Classification.

LANGUAGE MODELLING NAMED ENTITY RECOGNITION

4
21 Jul 2020

InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training

15 Jul 2020CZWin32768/xnlg

In this work, we formulate cross-lingual language model pre-training as maximizing mutual information between multilingual-multi-granularity texts.

CONTRASTIVE LEARNING LANGUAGE MODELLING

74
15 Jul 2020

Pre-trained Word Embeddings for Goal-conditional Transfer Learning in Reinforcement Learning

10 Jul 2020maximecb/gym-miniworld

Reinforcement learning (RL) algorithms typically start tabula rasa, without any prior knowledge of the environment, and without any prior skills.

LANGUAGE MODELLING TRANSFER LEARNING WORD EMBEDDINGS

248
10 Jul 2020