Subword Segmentation

Byte Pair Encoding

Introduced by Sennrich et al. in Neural Machine Translation of Rare Words with Subword Units

Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word classes are translatable via smaller units than words, for instance names (via character copying or transliteration), compounds (via compositional translation), and cognates and loanwords (via phonological and morphological transformations).

Lei Mao has a detailed blog post that explains how this works.

Source: Neural Machine Translation of Rare Words with Subword Units

Papers


Paper Code Results Date Stars

Tasks


Task Papers Share
Language Modelling 58 7.63%
Retrieval 36 4.74%
Question Answering 32 4.21%
Large Language Model 30 3.95%
Semantic Segmentation 22 2.89%
In-Context Learning 15 1.97%
Sentence 13 1.71%
Prompt Engineering 13 1.71%
Object Detection 12 1.58%

Components


Component Type
🤖 No Components Found You can add them if they exist; e.g. Mask R-CNN uses RoIAlign

Categories