Improving Large-scale Language Models and Resources for Filipino
In this paper, we improve on existing language resources for the low-resource Filipino language in two ways. First, we outline the construction of the TLUnified dataset, a large-scale pretraining corpus that serves as an improvement over smaller existing pretraining datasets for the language in terms of scale and topic variety. Second, we pretrain new Transformer language models following the RoBERTa pretraining technique to supplant existing models trained with small corpora. Our new RoBERTa models show significant improvements over existing Filipino models in three benchmark datasets with an average gain of 4.47% test accuracy across the three classification tasks of varying difficulty.
PDF Abstract LREC 2022 PDF LREC 2022 AbstractTasks
Datasets
CCAligned
Results from the Paper
Submit
results from this paper
to get state-of-the-art GitHub badges and help the
community compare results to other papers.
Methods
Absolute Position Encodings •
Adam •
Attention Dropout •
BERT •
BPE •
Dense Connections •
Dropout •
GELU •
Label Smoothing •
Layer Normalization •
Linear Layer •
Linear Warmup With Linear Decay •
Multi-Head Attention •
Position-Wise Feed-Forward Layer •
Residual Connection •
RoBERTa •
Scaled Dot-Product Attention •
Softmax •
Transformer •
Weight Decay •
WordPiece
