RobeCzech: Czech RoBERTa, a monolingual contextualized language representation model
We present RobeCzech, a monolingual RoBERTa language representation model trained on Czech data. RoBERTa is a robustly optimized Transformer-based pretraining approach. We show that RobeCzech considerably outperforms equally-sized multilingual and Czech-trained contextualized language representation models, surpasses current state of the art in all five evaluated NLP tasks and reaches state-of-the-art results in four of them. The RobeCzech model is released publicly at https://hdl.handle.net/11234/1-3691 and https://huggingface.co/ufal/robeczech-base.
PDF AbstractTasks
Datasets
Add Datasets
introduced or used in this paper
Methods
Adam •
Attention Dropout •
BERT •
Dense Connections •
Dropout •
GELU •
Layer Normalization •
Linear Layer •
Linear Warmup With Linear Decay •
Multi-Head Attention •
Residual Connection •
RoBERTa •
Scaled Dot-Product Attention •
Softmax •
Weight Decay •
WordPiece
