Search Results for author:
Found 6 papers, 0 papers with code
An Analysis of BPE Vocabulary Trimming in Neural Machine Translation
We explore threshold vocabulary trimming in Byte-Pair Encoding subword tokenization, a postprocessing step that replaces rare subwords with their component subwords.
Two Counterexamples to Tokenization and the Noiseless Channel
In Tokenization and the Noiseless Channel (Zouhar et al., 2023a), R\'enyi efficiency is suggested as an intrinsic mechanism for evaluating a tokenizer: for NLP tasks, the tokenizer which leads to the highest R\'enyi efficiency of the unigram distribution should be chosen.
SoftRegex: Generating Regex from Natural Language Descriptions using Softened Regex Equivalence
We continue the study of generating se-mantically correct regular expressions from natural language descriptions (NL).
On the Compression of Lexicon Transducers
Indeed, a delicate balance between comprehensiveness, speed, and memory must be struck to conform to device requirements while providing a good user experience. In this paper, we describe a compression scheme for lexicons when represented as finite-state transducers.
Online Infix Probability Computation for Probabilistic Finite Automata
Probabilistic finite automata (PFAs) are com- mon statistical language model in natural lan- guage and speech processing.
Incremental Computation of Infix Probabilities for Probabilistic Finite Automata
The problem of computing infix probabilities of strings when the pattern distribution is given by a probabilistic context-free grammar or by a probabilistic finite automaton is already solved, yet it was open to compute the infix probabilities in an incremental manner.
