Search Results for author:
Found 5 papers, 4 papers with code
DiJiang: Efficient Large Language Models through Compact Kernelization
In an effort to reduce the computational load of Transformers, research on linear attention has gained significant momentum.
Rethinking Optimization and Architecture for Tiny Language Models
Several design formulas are empirically proved especially effective for tiny language models, including tokenizer compression, architecture tweaking, parameter inheritance and multiple-round training.
Towards Higher Ranks via Adversarial Weight Pruning
To this end, we propose a Rank-based PruninG (RPG) method to maintain the ranks of sparse weights in an adversarial manner.
Multiscale Positive-Unlabeled Detection of AI-Generated Texts
Recent releases of Large Language Models (LLMs), e. g. ChatGPT, are astonishing at generating human-like texts, but they may impact the authenticity of texts.
Source-Target Unified Knowledge Distillation for Memory-Efficient Federated Domain Adaptation on Edge Devices
Moreover, to prevent the compact model from forgetting the knowledge of the source data during knowledge distillation, a collaborative knowledge distillation (Co-KD) method is developed to unify the source data on the server and the target data on the edge device to train the compact model.
