Language Modelling
4526 papers with code • 51 benchmarks • 157 datasets
Language Modeling is the task of predicting the next word or character in a document. This technique can be used to train language models that can further be applied to a wide range of natural language tasks like text generation, text classification, and question answering.
Historically, language modelling was done with N-gram language models (which still have niche uses), but since the 2010s neural language models took over, and starting from the 2020s SOTA was achieved exclusively with large language models (LLMs).
A model's language modeling capability is measured using cross-entropy and perplexity. Some datasets to evaluate language modeling are WikiText-103, One Billion Word, Text8, C4, The Pile, among others.
Some notable state-of-the-art language models include:
Check below for all state-of-the-art models.
Here are some additional readings to go deeper on the task:
- Language Modeling - Lena Voita
( Image credit: Exploring the Limits of Language Modeling )
Libraries
Use these libraries to find Language Modelling models and implementationsDatasets
Subtasks
Latest papers
Evaluating Class Membership Relations in Knowledge Graphs using Large Language Models
A backbone of knowledge graphs are their class membership relations, which assign entities to a given class.
Nyonic Technical Report
This report details the development and key achievements of our latest language model designed for custom large language models.
Studying Large Language Model Behaviors Under Realistic Knowledge Conflicts
In cases where the models still fail to update their answers, we find a parametric bias: the incorrect parametric answer appearing in context makes the knowledge update likelier to fail.
Knowledge Graph Completion using Structural and Textual Embeddings
We demonstrate that our model achieves competitive results in the relation prediction task when evaluated on a widely used dataset.
Attacks on Third-Party APIs of Large Language Models
Large language model (LLM) services have recently begun offering a plugin ecosystem to interact with third-party API services.
CultureBank: An Online Community-Driven Knowledge Base Towards Culturally Aware Language Technologies
To enhance language models' cultural awareness, we design a generalizable pipeline to construct cultural knowledge bases from different online communities on a massive scale.
Setting up the Data Printer with Improved English to Ukrainian Machine Translation
To build large language models for Ukrainian we need to expand our corpora with large amounts of new algorithmic tasks expressed in natural language.
BattleAgent: Multi-modal Dynamic Emulation on Historical Battles to Complement Historical Analysis
This novel system aims to simulate complex dynamic interactions among multiple agents, as well as between agents and their environments, over a period of time.
Aligning LLM Agents by Learning Latent Preference from User Edits
In a typical setting such as writing assistants, the user interacts with a language agent to generate a response given a context, and may optionally edit the agent response to personalize it based on their latent preference, in addition to improving the correctness.
Detection of circular permutations by Protein Language Models
The protein language model can help us extract structural information from sequences.