Search Results for author:
Found 11 papers, 8 papers with code
DAM: Dynamic Adapter Merging for Continual Video QA Learning
Our DAM model outperforms prior state-of-the-art continual learning approaches by 9. 1% while exhibiting 1. 9% less forgetting on 6 VidQA datasets spanning various domains.
SELMA: Learning and Merging Skill-Specific Text-to-Image Experts with Auto-Generated Data
In this paper, we introduce SELMA: Skill-Specific Expert Learning and Merging with Auto-Generated Data, a novel paradigm to improve the faithfulness of T2I models by fine-tuning models on automatically generated, multi-skill image-text datasets, with skill-specific expert learning and merging.
ECoFLaP: Efficient Coarse-to-Fine Layer-Wise Pruning for Vision-Language Models
We first determine the sparsity ratios of different layers or blocks by leveraging the global importance score, which is efficiently computed based on the zeroth-order approximation of the global model gradients.
Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy
Sparsely activated Mixture-of-Experts (SMoE) has shown promise to scale up the learning capacity of neural networks, however, they have issues like (a) High Memory Usage, due to duplication of the network layers into multiple copies as experts; and (b) Redundancy in Experts, as common learning-based routing policies suffer from representational collapse.
Unified Coarse-to-Fine Alignment for Video-Text Retrieval
Specifically, our model captures the cross-modal similarity information at different granularity levels.
Ranked #11 on
Video Retrieval
on MSR-VTT
An Empirical Study of Multimodal Model Merging
In this paper, we expand on this concept to a multimodal setup by merging transformers trained on different modalities.
Vision Transformers are Parameter-Efficient Audio-Visual Learners
To do so, we propose a latent audio-visual hybrid (LAVISH) adapter that adapts pretrained ViTs to audio-visual tasks by injecting a small number of trainable parameters into every layer of a frozen ViT.
Ranked #4 on
Audio-visual Question Answering
on MUSIC-AVQA
LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning
LST saves 69% of the memory costs to fine-tune the whole network, while other methods only save 26% of that in similar parameter usages (hence, 2. 7x more memory savings).
VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks
Our results demonstrate that training the adapter with the weight-sharing technique (4. 18% of total parameters for image-text tasks and 3. 39% for video-text tasks) can match the performance of fine-tuning the entire model.
Training Neural Networks with Fixed Sparse Masks
In this paper, we show that it is possible to induce a fixed sparse mask on the model's parameters that selects a subset to update over many iterations.
Difference-Seeking Generative Adversarial Network
DSGAN considers the scenario that the training samples of target distribution, $p_{t}$, are difficult to collect.
