Search Results for author:
Found 347 papers, 250 papers with code
LEGO-ABSA: A Prompt-based Task Assemblable Unified Generative Framework for Multi-task Aspect-based Sentiment Analysis
This paper proposes a unified generative multi-task framework that can solve multiple ABSA tasks by controlling the type of task prompts consisting of multiple element prompts.
Ranked #5 on
Aspect-Based Sentiment Analysis (ABSA)
on TASD
(using extra training data)
Aspect-Based Sentiment Analysis
Aspect-Based Sentiment Analysis (ABSA)
Going “Deeper”: Structured Sememe Prediction via Transformer with Tree Attention
However, all existing sememe prediction studies ignore the hierarchical structures of sememes, which are important in the sememe-based semantic description system.
CodRED: A Cross-Document Relation Extraction Dataset for Acquiring Knowledge in the Wild
Existing relation extraction (RE) methods typically focus on extracting relational facts between entity pairs within single sentences or documents.
BMInf: An Efficient Toolkit for Big Model Inference and Tuning
In recent years, large-scale pre-trained language models (PLMs) containing billions of parameters have achieved promising results on various NLP tasks.
Do Pre-trained Models Benefit Knowledge Graph Completion? A Reliable Evaluation and a Reasonable Approach
In recent years, pre-trained language models (PLMs) have been shown to capture factual knowledge from massive texts, which encourages the proposal of PLM-based knowledge graph completion (KGC) models.
Pass off Fish Eyes for Pearls: Attacking Model Selection of Pre-trained Models
To validate our viewpoints, we design two methods to evaluate the robustness of FMS: (1) model disguise attack, which post-trains an inferior PTM with a contrastive objective, and (2) evaluation data selection, which selects a subset of the data points for FMS evaluation based on K-means clustering.
Cross-Lingual Contrastive Learning for Fine-Grained Entity Typing for Low-Resource Languages
In this paper, we propose a cross-lingual contrastive learning framework to learn FGET models for low-resource languages.
PreGSU-A Generalized Traffic Scene Understanding Model for Autonomous Driving based on Pre-trained Graph Attention Network
In this study, we propose PreGSU, a generalized pre-trained scene understanding model based on graph attention network to learn the universal interaction and reasoning of traffic scenes to support various downstream tasks.
UltraEval: A Lightweight Platform for Flexible and Comprehensive Evaluation for LLMs
The rapid development of LLMs calls for a lightweight and easy-to-use framework for swift evaluation deployment.
MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies
For data scaling, we introduce a Warmup-Stable-Decay (WSD) learning rate scheduler (LRS), conducive to continuous training and domain adaptation.
Advancing LLM Reasoning Generalists with Preference Trees
We introduce Eurus, a suite of large language models (LLMs) optimized for reasoning.
Robust and Scalable Model Editing for Large Language Models
Large language models (LLMs) can make predictions using parametric knowledge--knowledge encoded in the model weights--or contextual knowledge--knowledge presented in the context.
LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Images
To address the challenges, we present LLaVA-UHD, a large multimodal model that can efficiently perceive images in any aspect ratio and high resolution.
BurstAttention: An Efficient Distributed Attention Framework for Extremely Long Sequences
Effective attention modules have played a crucial role in the success of Transformer-based large language models (LLMs), but the quadratic time and memory complexities of these attention modules also pose a challenge when processing long sequences.
Mastering Text, Code and Math Simultaneously via Fusing Highly Specialized Language Models
Underlying data distributions of natural language, programming code, and mathematical symbols vary vastly, presenting a complex challenge for large language models (LLMs) that strive to achieve high performance across all three domains simultaneously.
StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Models
The virtual API server contains a caching system and API simulators which are complementary to alleviate the change in API status.
LLM-Oriented Retrieval Tuner
Dense Retrieval (DR) is now considered as a promising tool to enhance the memorization capacity of Large Language Models (LLM) such as GPT3 and GPT-4 by incorporating external memories.
Controllable Preference Optimization: Toward Controllable Multi-Objective Alignment
In practice, the multifaceted nature of human preferences inadvertently introduces what is known as the "alignment tax" -a compromise where enhancements in alignment within one objective (e. g., harmlessness) can diminish performance in others (e. g., helpfulness).
Beyond Natural Language: LLMs Leveraging Alternative Formats for Enhanced Reasoning and Communication
Natural language (NL) has long been the predominant format for human cognition and communication, and by extension, has been similarly pivotal in the development and application of Large Language Models (LLMs).
RepoAgent: An LLM-Powered Open-Source Framework for Repository-level Code Documentation Generation
Generative models have demonstrated considerable potential in software engineering, particularly in tasks such as code generation and debugging.
Unified View of Grokking, Double Descent and Emergent Abilities: A Perspective from Circuits Competition
Recent studies have uncovered intriguing phenomena in deep learning, such as grokking, double descent, and emergent abilities in large language models, which challenge human intuition and are crucial for a deeper understanding of neural models.
Cleaner Pretraining Corpus Curation with Neural Web Scraping
The web contains large-scale, diverse, and abundant information to satisfy the information-seeking needs of humans.
ProSparse: Introducing and Enhancing Intrinsic Activation Sparsity within Large Language Models
Some recent efforts have explored introducing ReLU or its variants as the substitutive activation function to help LLMs achieve activation sparsity and inference acceleration, but few can simultaneously obtain high sparsity and comparable model performance.
OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems
Notably, the best-performing model, GPT-4V, attains an average score of 17. 23% on OlympiadBench, with a mere 11. 28% in physics, highlighting the benchmark rigor and the intricacy of physical reasoning.
Ouroboros: Speculative Decoding with Large Model Enhanced Drafting
In this paper, we introduce Ouroboros, which constructs a phrase candidate pool from the verification process of LLMs to provide candidates for draft generation of the small model.
ActiveRAG: Revealing the Treasures of Knowledge via Active Learning
Retrieval Augmented Generation (RAG) has introduced a new paradigm for Large Language Models (LLMs), aiding in the resolution of knowledge-intensive tasks.
$\infty$Bench: Extending Long Context Evaluation Beyond 100K Tokens
Processing and reasoning over long contexts is crucial for many practical applications of Large Language Models (LLMs), such as document comprehension and agent construction.
Large Language Model-based Human-Agent Collaboration for Complex Task Solving
We construct a human-agent collaboration dataset to train this policy model in an offline reinforcement learning environment.
MatPlotAgent: Method and Evaluation for LLM-Based Agentic Scientific Data Visualization
Scientific data visualization plays a crucial role in research by enabling the direct display of complex information and assisting researchers in identifying implicit patterns.
LoRA-Flow: Dynamic LoRA Fusion for Large Language Models in Generative Tasks
Most prior works on LoRA combination primarily rely on task-level weights for each involved LoRA, making different examples and tokens share the same LoRA weights.
OneBit: Towards Extremely Low-bit Large Language Models
Model quantification uses low bit-width values to represent the weight matrices of models, which is a promising approach to reduce both storage and computational overheads of deploying highly anticipated LLMs.
Tell Me More! Towards Implicit User Intention Understanding of Language Model Driven Agents
Current language model-driven agents often lack mechanisms for effective user participation, which is crucial given the vagueness commonly found in user instructions.
InfLLM: Unveiling the Intrinsic Capacity of LLMs for Understanding Extremely Long Sequences with Training-Free Memory
To alleviate these issues, existing efforts employ sliding attention windows and discard distant tokens to achieve the processing of extremely long sequences.
UltraLink: An Open-Source Knowledge-Enhanced Multilingual Supervised Fine-tuning Dataset
Different from previous works that simply translate English instructions, we consider both the language-specific and language-agnostic abilities of LLMs.
MolTC: Towards Molecular Relational Modeling In Language Models
Molecular Relational Learning (MRL), aiming to understand interactions between molecular pairs, plays a pivotal role in advancing biochemical research.
ReLU$^2$ Wins: Discovering Efficient Activation Functions for Sparse LLMs
To find the most efficient activation function for sparse computation, we propose a systematic framework to examine the sparsity of LLMs from three aspects: the trade-off between sparsity and performance, the predictivity of sparsity, and the hardware affinity.
UniMem: Towards a Unified View of Long-Context Large Language Models
Although there exist various methods devoted to enhancing the long-context processing ability of large language models (LLMs), they are developed in an isolated manner and lack systematic analysis and integration of their strengths, hindering further developments.
Investigate-Consolidate-Exploit: A General Strategy for Inter-Task Agent Self-Evolution
This paper introduces Investigate-Consolidate-Exploit (ICE), a novel strategy for enhancing the adaptability and flexibility of AI agents through inter-task self-evolution.
Towards 3D Molecule-Text Interpretation in Language Models
Through 3D molecule-text alignment and 3D molecule-centric instruction tuning, 3D-MoLM establishes an integration of 3D molecular encoder and LM.
DebugBench: Evaluating Debugging Capability of Large Language Models
Previous evaluations of LLMs' debugging ability are significantly limited by the risk of data leakage, the scale of the dataset, and the variety of tested bugs.
Experiential Co-Learning of Software-Developing Agents
Recent advancements in large language models (LLMs) have brought significant changes to various domains, especially through LLM-driven autonomous agents.
GitAgent: Facilitating Autonomous Agent with GitHub by Tool Extension
As GitHub has hosted a multitude of repositories which can be seen as a good resource for tools, a promising solution is that LLM-based agents can autonomously integrate the repositories in GitHub according to the user queries to extend their tool set.
D-Bot: Database Diagnosis System using Large Language Models
Database administrators (DBAs) play an important role in managing, maintaining and optimizing database systems.
RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback
Multimodal Large Language Models (MLLMs) have recently demonstrated impressive capabilities in multimodal understanding, reasoning, and interaction.
Sparse Low-rank Adaptation of Pre-trained Language Models
Recognizing the need for more flexible adaptation, we extend the methodology of LoRA to an innovative approach we call sparse low-rank adaptation (SoRA) that enables dynamic adjustments to the intrinsic rank during the adaptation process.
INTERVENOR: Prompting the Coding Ability of Large Language Models with the Interactive Chain of Repair
INTERVENOR prompts Large Language Models (LLMs) to play distinct roles during the code repair process, functioning as both a Code Learner and a Code Teacher.
Ranked #15 on
Code Generation
on HumanEval
MAVEN-Arg: Completing the Puzzle of All-in-One Event Understanding Dataset with Event Argument Annotation
Understanding events in texts is a core objective of natural language understanding, which requires detecting event occurrences, extracting event arguments, and analyzing inter-event relationships.
NExT-Chat: An LMM for Chat, Detection and Segmentation
The development of large language models (LLMs) has greatly advanced the field of multimodal understanding, leading to the emergence of large multimodal models (LMMs).
ProAgent: From Robotic Process Automation to Agentic Process Automation
Empirical experiments are conducted to detail its construction and execution procedure of workflow, showcasing the feasibility of APA, unveiling the possibility of a new paradigm of automation driven by agents.
WebDRO: A Web-based Group-level Clustering and Reweighting Method for Unsupervised Dense Retrieval
In this paper, we introduce WebDRO, an efficient approach for clustering the web graph data and optimizing group weights to enhance the robustness of the pretraining process of dense retrieval models on web graphs.
MUSER: A Multi-View Similar Case Retrieval Dataset
Furthermore, the case similarities are typically measured solely by the textual semantics of the fact descriptions, which may fail to capture the full complexity of legal cases from the perspective of legal knowledge.
Variator: Accelerating Pre-trained Models with Plug-and-Play Compression Modules
Pre-trained language models (PLMs) have achieved remarkable results on NLP tasks but at the expense of huge parameter sizes and the consequent computational costs.
Rethinking Tokenizer and Decoder in Masked Graph Modeling for Molecules
Our results show that a subgraph-level tokenizer and a sufficiently expressive decoder with remask decoding have a large impact on the encoder's representation learning.
MARVEL: Unlocking the Multi-Modal Capability of Dense Retrieval via Visual Module Plugin
To facilitate the multi-modal retrieval tasks, we build the ClueWeb22-MM dataset based on the ClueWeb22 dataset, which regards anchor texts as queries, and exacts the related text and image documents from anchor-linked web pages.
Thoroughly Modeling Multi-domain Pre-trained Recommendation as Language
With the thriving of pre-trained language model (PLM) widely verified in various of NLP tasks, pioneer efforts attempt to explore the possible cooperation of the general textual information in PLM with the personalized behavioral information in user historical behavior sequences to enhance sequential recommendation (SR).
ReLM: Leveraging Language Models for Enhanced Chemical Reaction Prediction
Predicting chemical reactions, a fundamental challenge in chemistry, involves forecasting the resulting products from a given reaction process.
MolCA: Molecular Graph-Language Modeling with Cross-Modal Projector and Uni-Modal Adapter
MolCA enables an LM (e. g., Galactica) to understand both text- and graph-based molecular contents via the cross-modal projector.
Ranked #4 on
Molecule Captioning
on ChEBI-20
Boosting Inference Efficiency: Unleashing the Power of Parameter-Shared Pre-trained Language Models
Parameter-shared pre-trained language models (PLMs) have emerged as a successful approach in resource-constrained environments, enabling substantial reductions in model storage and memory costs without significant performance compromise.
Toolink: Linking Toolkit Creation and Using through Chain-of-Solving on Open-Source Model
We first validate the efficacy of Toolink in harnessing the model's creativity and CoS ability on ChatGPT.
Predicting Emergent Abilities with Infinite Resolution Evaluation
With PassUntil, we conduct a quantitative investigation into the scaling law of task performance.
UltraFeedback: Boosting Language Models with High-quality Feedback
However, the scarcity of diverse, naturalistic datasets of human preferences on LLM outputs at scale poses a great challenge to RLHF as well as feedback learning research within the open-source community.
Reformulating Vision-Language Foundation Models and Datasets Towards Universal Multimodal Assistants
The capabilities of MLLMs depend on two crucial factors: the model architecture to facilitate the feature alignment of visual modules and large language models; the multimodal instruction tuning datasets for human instruction following.
ConPET: Continual Parameter-Efficient Tuning for Large Language Models
First, Static ConPET can adapt former continual learning methods originally designed for relatively smaller models to LLMs through PET and a dynamic replay strategy, which largely reduces the tuning costs and alleviates the over-fitting and forgetting issue.
QASnowball: An Iterative Bootstrapping Framework for High-Quality Question-Answering Data Generation
Recent years have witnessed the success of question answering (QA), especially its potential to be a foundation paradigm for tackling diverse NLP tasks.
Instant Photorealistic Style Transfer: A Lightweight and Adaptive Approach
In this paper, we propose an Instant Photorealistic Style Transfer (IPST) approach, designed to achieve instant photorealistic style transfer on super-resolution inputs without the need for pre-training on pair-wise datasets or imposing extra constraints.
Text Matching Improves Sequential Recommendation by Reducing Popularity Biases
TASTE alleviates the cold start problem by representing long-tail items using full-text modeling and bringing the benefits of pretrained language models to recommendation systems.
Rational Decision-Making Agent with Internalized Utility Judgment
Central to the development of rationality is the construction of an internalized utility judgment, capable of assigning numerical utilities to each decision.
Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languages
Building a competitive counterpart in other languages is highly challenging due to the low-resource nature of non-English multimodal data (i. e., lack of large-scale, high-quality image-text data).
AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors
Autonomous agents empowered by Large Language Models (LLMs) have undergone significant improvements, enabling them to generalize across a broad spectrum of tasks.
ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate
Text evaluation has historically posed significant challenges, often demanding substantial labor and time cost.
LLM As DBA
Database administrators (DBAs) play a crucial role in managing, maintaining and optimizing a database system to ensure data availability, performance, and reliability.
ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs
Based on ToolBench, we fine-tune LLaMA to obtain an LLM ToolLLaMA, and equip it with a neural API retriever to recommend appropriate APIs for each instruction.
Ranked #3 on
Trajectory Planning
on ToolBench
Exploring Format Consistency for Instruction Tuning
Instruction tuning has emerged as a promising approach to enhancing large language models in following human instructions.
Communicative Agents for Software Development
At the core of this paradigm lies ChatDev, a virtual chat-powered software development company that mirrors the established waterfall model, meticulously dividing the development process into four distinct chronological stages: designing, coding, testing, and documenting.
CPET: Effective Parameter-Efficient Tuning for Compressed Large Language Models
Parameter-efficient tuning (PET) has been widely explored in recent years because it tunes much fewer parameters (PET modules) than full-parameter fine-tuning (FT) while still stimulating sufficient knowledge from large language models (LLMs) for downstream tasks.
OpenDelta: A Plug-and-play Library for Parameter-efficient Adaptation of Pre-trained Models
The scale of large pre-trained models (PTMs) poses significant challenges in adapting to downstream tasks due to the high optimization overhead and storage costs associated with full-parameter fine-tuning.
Won't Get Fooled Again: Answering Questions with False Premises
In this paper, we find that the PLMs already possess the knowledge required to rebut such questions, and the key is how to activate the knowledge.
Automatic Truss Design with Reinforcement Learning
Directly applying end-to-end reinforcement learning (RL) methods to truss layout design is infeasible either, since only a tiny portion of the entire layout space is valid under the physical constraints, leading to particularly sparse rewards for RL training.
Interactive Molecular Discovery with Natural Language
Natural language is expected to be a key medium for various human-machine interactions in the era of large language models.
KoLA: Carefully Benchmarking World Knowledge of Large Language Models
The unprecedented performance of large language models (LLMs) necessitates improvements in evaluations.
The Devil is in the Details: On the Pitfalls of Event Extraction Evaluation
In this paper, we check the reliability of EE evaluations and identify three major pitfalls: (1) The data preprocessing discrepancy makes the evaluation results on the same dataset not directly comparable, but the data preprocessing details are not widely noted and specified in papers.
Revisiting Out-of-distribution Robustness in NLP: Benchmark, Analysis, and LLMs Evaluations
Then we introduce BOSS, a Benchmark suite for Out-of-distribution robustneSS evaluation covering 5 tasks and 20 datasets.
Exploring the Impact of Model Scaling on Parameter-Efficient Tuning
Our investigations reveal that model scaling (1) mitigates the effects of the positions of tunable parameters on performance, and (2) enables tuning methods to achieve performance comparable to full-parameter fine-tuning by optimizing fewer tunable parameters.
Structure-Aware Language Model Pretraining Improves Dense Retrieval on Structured Data
SANTA proposes two pretraining methods to make language models structure-aware and learn effective representations for structured data: 1) Structured Data Alignment, which utilizes the natural alignment relations between structured data and unstructured data for structure-aware pretraining.
Exploring Lottery Prompts for Pre-trained Language Models
Consistently scaling pre-trained language models (PLMs) imposes substantial burdens on model adaptation, necessitating more efficient alternatives to conventional fine-tuning.
From Adversarial Arms Race to Model-centric Evaluation: Motivating a Unified Automatic Robustness Evaluation Framework
In our experiments, we conduct a robustness evaluation of RoBERTa models to demonstrate the effectiveness of our evaluation framework, and further show the rationality of each component in the framework.
Emergent Modularity in Pre-trained Transformers
In analogy to human brains, we consider two main characteristics of modularity: (1) functional specialization of neurons: we evaluate whether each neuron is mainly specialized in a certain function, and find that the answer is yes.
Plug-and-Play Document Modules for Pre-trained Models
By inserting document plugins into the backbone PTM for downstream tasks, we can encode a document one time to handle multiple tasks, which is more efficient than conventional encoding-task coupling methods that simultaneously encode documents and input queries using task-specific encoders.
Stochastic Bridges as Effective Regularizers for Parameter-Efficient Tuning
Since it is non-trivial to directly model the intermediate states and design a running cost function, we propose to use latent stochastic bridges to regularize the intermediate states and use the regularization as the running cost of PETs.
Plug-and-Play Knowledge Injection for Pre-trained Language Models
Experimental results on three knowledge-driven NLP tasks show that existing injection methods are not suitable for the new paradigm, while map-tuning effectively improves the performance of downstream models.
Augmentation-Adapted Retriever Improves Generalization of Language Models as Generic Plug-In
Retrieval augmentation can aid language models (LMs) in knowledge-intensive tasks by supplying them with external information.
Fusion-in-T5: Unifying Document Ranking Signals for Improved Information Retrieval
Common IR pipelines are typically cascade systems that may involve multiple rankers and/or fusion models to integrate different information step-by-step.
Enhancing Chat Language Models by Scaling High-quality Instructional Conversations
Fine-tuning on instruction data has been widely validated as an effective practice for implementing chat language models like ChatGPT.
Efficient Cross-Lingual Transfer for Chinese Stable Diffusion with Images as Pivots
IAP optimizes only a separate Chinese text encoder with all other parameters fixed to align Chinese semantics space to the English one in CLIP.
Recyclable Tuning for Continual Pre-training
In pilot studies, we find that after continual pre-training, the upgraded PLM remains compatible with the outdated adapted weights to some extent.
WebCPM: Interactive Web Search for Chinese Long-form Question Answering
We recruit annotators to search for relevant information using our interface and then answer questions.
Revisiting Graph Contrastive Learning for Anomaly Detection
In the paper, we delve into the misconception and propose Multi-GNN and Augmented Graph contrastive framework MAG, which unified the existing GCAD methods in the contrastive self-supervised perspective.
VPGTrans: Transfer Visual Prompt Generator across LLMs
While developing a new multimodal LLM (MLLM) by pre-training on tremendous image-text pairs from scratch can be exceedingly resource-consuming, connecting an existing LLM with a comparatively lightweight visual prompt generator (VPG) becomes a feasible paradigm.
Tool Learning with Foundation Models
Considering the lack of a systematic tool learning evaluation in prior works, we experiment with 18 representative tools and show the potential of current foundation models in skillfully utilizing tools.
Rethinking Dense Retrieval's Few-Shot Ability
Moreover, the dataset is disjointed into base and novel classes, allowing DR models to be continuously trained on ample data from base classes and a few samples in novel classes.
Decoder-Only or Encoder-Decoder? Interpreting Language Model as a Regularized Encoder-Decoder
Grounded on our analysis, we propose a novel partial attention language model to solve the attention degeneration problem.
Language-Specific Representation of Emotion-Concept Knowledge Causally Supports Emotion Inference
Humans no doubt use language to communicate about their emotional experiences, but does language in turn help humans understand emotions, or is language just a vehicle of communication?
READIN: A Chinese Multi-Task Benchmark with Realistic and Diverse Input Noises
In order to fill this important gap, we construct READIN: a Chinese multi-task benchmark with REalistic And Diverse Input Noises.
Semi-supervised Large-scale Fiber Detection in Material Images with Synthetic Data
Accurate detection of large-scale, elliptical-shape fibers, including their parameters of center, orientation and major/minor axes, on the 2D cross-sectioned image slices is very important for characterizing the underlying cylinder 3D structures in microscopic material images.
Decoder Tuning: Efficient Language Understanding as Decoding
With the evergrowing sizes of pre-trained models (PTMs), it has been an emerging practice to only provide the inference APIs for users, namely model-as-a-service (MaaS) setting.
Mul-GAD: a semi-supervised graph anomaly detection framework via aggregating multi-view information
For a more comprehensive conclusion, we further investigate the effect of the objective function and the number of fused views on detection performance.
Visually Grounded Commonsense Knowledge Acquisition
In this work, we present CLEVER, which formulates CKE as a distantly supervised multi-instance learning problem, where models learn to summarize commonsense relations from a bag of images about an entity pair without any human annotation on image instances.
MAVEN-ERE: A Unified Large-scale Dataset for Event Coreference, Temporal, Causal, and Subevent Relation Extraction
It contains 103, 193 event coreference chains, 1, 216, 217 temporal relations, 57, 992 causal relations, and 15, 841 subevent relations, which is larger than existing datasets of all the ERE tasks by at least an order of magnitude.
Finding Skill Neurons in Pre-trained Transformer-based Language Models
Furthermore, we demonstrate the skill neurons are most likely generated in pre-training rather than fine-tuning by showing that the skill neurons found with prompt tuning are also crucial for other fine-tuning methods freezing neuron weights, such as the adapter-based tuning and BitFit.
FPT: Improving Prompt Tuning Efficiency via Progressive Training
Inspired by these observations, we propose Fast Prompt Tuning (FPT), which starts by conducting PT using a small-scale partial PLM, and then progressively expands its depth and width until the full-model size.
Few-shot Classification with Hypersphere Modeling of Prototypes
Moreover, it is more convenient to perform metric-based classification with hypersphere prototypes than statistical modeling, as we only need to calculate the distance from a data point to the surface of the hypersphere.
COPEN: Probing Conceptual Knowledge in Pre-trained Language Models
We believe this is a critical bottleneck for realizing human-like cognition in PLMs.
Sparse Structure Search for Delta Tuning
Generally, DT methods exquisitely design delta modules (DT modules) which could be applied to arbitrary fine-grained positions inside PTMs.
Reduce Catastrophic Forgetting of Dense Retrieval Training with Teleportation Negatives
In this paper, we investigate the instability in the standard dense retrieval training, which iterates between model training and hard negative selection using the being-trained model.
A Close Look into the Calibration of Pre-trained Language Models
We observe a consistent change in calibration performance across six factors.
Exploring Mode Connectivity for Pre-trained Language Models
(3) How does the PLM's task knowledge change along the path connecting two minima?
Different Tunes Played with Equal Skill: Exploring a Unified Optimization Subspace for Delta Tuning
To fathom the mystery, we hypothesize that the adaptations of different DETs could all be reparameterized as low-dimensional optimizations in a unified optimization subspace, which could be found by jointly decomposing independent solutions of different DETs.
Why Should Adversarial Perturbations be Imperceptible? Rethink the Research Paradigm in Adversarial NLP
We discuss the deficiencies in previous work and propose our suggestions that the research on the Security-oriented adversarial NLP (SoadNLP) should: (1) evaluate their methods on security tasks to demonstrate the real-world concerns; (2) consider real-world attackers' goals, instead of developing impractical methods.
Automatic Label Sequence Generation for Prompting Sequence-to-sequence Models
Prompting, which casts downstream applications as language modeling tasks, has shown to be sample efficient compared to standard fine-tuning with pre-trained models.
Universal Vision-Language Dense Retrieval: Learning A Unified Representation Space for Multi-Modal Retrieval
To learn a unified embedding space for multi-modal retrieval, UniVL-DR proposes two techniques: 1) Universal embedding optimization strategy, which contrastively optimizes the embedding space using the modality-balanced hard negatives; 2) Image verbalization method, which bridges the modality gap between images and texts in the raw data space.
Improving Task Generalization via Unified Schema Prompt
Task generalization has been a long standing challenge in Natural Language Processing (NLP).
Effective Few-Shot Named Entity Linking by Meta-Learning
In this paper, we endeavor to solve the problem of few-shot entity linking, which only requires a minimal amount of in-domain labeled data and is more practical in real situations.
GACT: Activation Compressed Training for Generic Network Architectures
Training large neural network (NN) models requires extensive memory resources, and Activation Compressed Training (ACT) is a promising approach to reduce training memory footprint.
A Unified Understanding of Deep NLP Models for Text Classification
The rapid development of deep natural language processing (NLP) models for text classification has led to an urgent need for a unified understanding of these models proposed individually.
A Unified Evaluation of Textual Backdoor Learning: Frameworks and Benchmarks
However, we highlight two issues in previous backdoor learning evaluations: (1) The differences between real-world scenarios (e. g. releasing poisoned datasets or models) are neglected, and we argue that each scenario has its own constraints and concerns, thus requires specific evaluation protocols; (2) The evaluation metrics only consider whether the attacks could flip the models' predictions on poisoned samples and retain performances on benign samples, but ignore that poisoned samples should also be stealthy and semantic-preserving.
Sparse Structure Search for Parameter-Efficient Tuning
The searched structures preserve more than 99\% fine-tuning performance with 0. 01\% trainable parameters.
Prompt Tuning for Discriminative Pre-trained Language Models
Recent works have shown promising results of prompt tuning in stimulating pre-trained language models (PLMs) for natural language processing (NLP) tasks.
PEVL: Position-enhanced Pre-training and Prompt Tuning for Vision-language Models
We show that PEVL enables state-of-the-art performance of detector-free VLP models on position-sensitive tasks such as referring expression comprehension and phrase grounding, and also improves the performance on position-insensitive tasks with grounded inputs.
Ranked #1 on
Visual Commonsense Reasoning
on VCR (Q-AR) test
ProQA: Structural Prompt-based Pre-training for Unified Question Answering
Furthermore, ProQA exhibits strong ability in both continual learning and transfer learning by taking the advantages of the structural prompt.
Dimension Reduction for Efficient Dense Retrieval via Conditional Autoencoder
These embeddings need to be high-dimensional to fit training signals and guarantee the retrieval effectiveness of dense retrievers.
Ranked #1 on
Information Retrieval
on MS MARCO
P^3 Ranker: Mitigating the Gaps between Pre-training and Ranking Fine-tuning with Prompt-based Learning and Pre-finetuning
In this paper, we identify and study the two mismatches between pre-training and ranking fine-tuning: the training schema gap regarding the differences in training objectives and model architectures, and the task knowledge gap considering the discrepancy between the knowledge needed in ranking and that learned during pre-training.
Exploring the Universal Vulnerability of Prompt-based Learning Paradigm
Prompt-based learning paradigm bridges the gap between pre-training and fine-tuning, and works effectively under the few-shot setting.
A Roadmap for Big Model
With the rapid development of deep learning, training Big Models (BMs) for multiple downstream tasks becomes a popular paradigm.
Fine-Grained Scene Graph Generation with Data Transfer
Scene graph generation (SGG) is designed to extract (subject, predicate, object) triplets in images.
Ranked #1 on
Predicate Classification
on Visual Genome
Prototypical Verbalizer for Prompt-based Few-shot Tuning
However, manual verbalizers heavily depend on domain-specific prior knowledge and human efforts, while finding appropriate label words automatically still remains challenging. In this work, we propose the prototypical verbalizer (ProtoVerb) which is built directly from training data.
LEVEN: A Large-Scale Chinese Legal Event Detection Dataset
However, existing Legal Event Detection (LED) datasets only concern incomprehensive event types and have limited annotated data, which restricts the development of LED methods and their downstream applications.
Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Models
This necessitates a new branch of research focusing on the parameter-efficient adaptation of PLMs, dubbed as delta tuning in this paper.
Sememe Prediction for BabelNet Synsets using Multilingual and Multimodal Information
In this paper, we utilize the multilingual synonyms, multilingual glosses and images in BabelNet for SPBS.
ELLE: Efficient Lifelong Pre-training for Emerging Data
We experiment ELLE with streaming data from 5 domains on BERT and GPT.
A Simple but Effective Pluggable Entity Lookup Table for Pre-trained Language Models
Pre-trained language models (PLMs) cannot well recall rich factual knowledge of entities exhibited in large-scale corpora, especially those rare entities.
QuoteR: A Benchmark of Quote Recommendation for Writing
To facilitate the research on this task, we build a large and fully open quote recommendation dataset called QuoteR, which comprises three parts including English, standard Chinese and classical Chinese.
Training Free Graph Neural Networks for Graph Matching
We present a framework of Training Free Graph Matching (TFGM) to boost the performance of Graph Neural Networks (GNNs) based graph matching, providing a fast promising solution without training (training-free).
CUGE: A Chinese Language Understanding and Generation Evaluation Benchmark
We argue that for general-purpose language intelligence evaluation, the benchmark itself needs to be comprehensive and systematic.
Video as Conditional Graph Hierarchy for Multi-Granular Question Answering
To align with the multi-granular essence of linguistic concepts in language queries, we propose to model video as a conditional graph hierarchy which weaves together visual facts of different granularity in a level-wise manner, with the guidance of corresponding textual cues.
Ranked #23 on
Video Question Answering
on NExT-QA
Multi-modal application: Image Memes Generation
An Internet meme commonly takes the form of an image and is created by combining a meme template (image) and a caption (natural language sentence).
On Transferability of Prompt Tuning for Natural Language Processing
To explore whether we can improve PT via prompt transfer, we empirically investigate the transferability of soft prompts across different downstream tasks and PLMs in this work.
OpenPrompt: An Open-source Framework for Prompt-learning
Prompt-learning has become a new paradigm in modern natural language processing, which directly adapts pre-trained language models (PLMs) to $cloze$-style prediction, autoregressive modeling, or sequence to sequence generation, resulting in promising performances on various tasks.
Exploring Universal Intrinsic Task Subspace via Prompt Tuning
In the experiments, we study diverse few-shot NLP tasks and surprisingly find that in a 250-dimensional subspace found with 100 tasks, by only tuning 250 free parameters, we can recover 97% and 83% of the full prompt tuning performance for 100 seen tasks (using different training data) and 20 unseen tasks, respectively, showing great generalization ability of the found intrinsic task subspace.
Textual Backdoor Attacks Can Be More Harmful via Two Simple Tricks
In this paper, we find two simple tricks that can make existing textual backdoor attacks much more harmful.
Mind the Style of Text! Adversarial and Backdoor Attacks Based on Text Style Transfer
In this paper, we make the first attempt to conduct adversarial and backdoor attacks based on text style transfer, which is aimed at altering the style of a sentence while preserving its meaning.
bert2BERT: Towards Reusable Pretrained Language Models
However, large language model pre-training costs intensive computational resources and most of the models are trained from scratch without reusing the existing pre-trained models, which is wasteful.
Program Transfer for Answering Complex Questions over Knowledge Bases
In this paper, we propose the approach of program transfer, which aims to leverage the valuable program annotations on the rich-resourced KBs as external supervision signals to aid program induction for the low-resourced KBs that lack program annotations.
MoEfication: Transformer Feed-forward Layers are Mixtures of Experts
In this work, we study the computational patterns of FFNs and observe that most inputs only activate a tiny ratio of neurons of FFNs.
Few-shot Learning with Big Prototypes
A big prototype could be effectively modeled by two sets of learnable parameters, one is the center of the hypersphere, which is an embedding with the same dimension of training examples.
CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models
Pre-Trained Vision-Language Models (VL-PTMs) have shown promising capabilities in grounding natural language in image data, facilitating a broad variety of cross-modal tasks.
TIAGE: A Benchmark for Topic-Shift Aware Dialog Modeling
In research on dialog systems, the ability to actively and smoothly transition to new topics is often ignored.
PPT: Pre-trained Prompt Tuning for Few-shot Learning
To ensure the generalization of PPT, we formulate similar classification tasks into a unified task form and pre-train soft prompts for this unified task.
Non-Euclidean Analysis of Joint Variations in Multi-Object Shapes
This paper considers joint analysis of multiple functionally related structures in classification tasks.
Prompt-Learning for Fine-Grained Entity Typing
In this work, we investigate the application of prompt-learning on fine-grained entity typing in fully supervised, few-shot and zero-shot scenarios.
Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classification
Tuning pre-trained language models (PLMs) with task-specific prompts has been a promising approach for text classification.
More Robust Dense Retrieval with Contrastive Dual Learning
With contrastive learning, the dual training object of DANCE learns more tailored representations for queries and documents to keep the embedding space smooth and uniform, thriving on the ranking performance of DANCE on the MS MARCO document retrieval task.
CPM-2: Large-scale Cost-effective Pre-trained Language Models
We present a suite of cost-effective techniques for the use of PLMs to deal with the efficiency issues of pre-training, fine-tuning, and inference.
Evaluating Modules in Graph Contrastive Learning
The recent emergence of contrastive learning approaches facilitates the application on graph representation learning (GRL), introducing graph contrastive learning (GCL) into the literature.
Pre-Trained Models: Past, Present and Future
Large-scale pre-trained models (PTMs) such as BERT and GPT have recently achieved great success and become a milestone in the field of artificial intelligence (AI).
Turn the Combination Lock: Learnable Textual Backdoor Attacks via Word Substitution
Recent studies show that neural natural language processing (NLP) models are vulnerable to backdoor attacks.
Open Hierarchical Relation Extraction
To establish the bidirectional connections between OpenRE and relation hierarchy, we propose the task of open hierarchical relation extraction and present a novel OHRE framework for the task.
Sub-Character Tokenization for Chinese Pretrained Language Models
2) Pronunciation-based SubChar tokenizers can encode Chinese homophones into the same transliteration sequences and produce the same tokenization output, hence being robust to homophone typos.
Fully Hyperbolic Neural Networks
Hyperbolic neural networks have shown great potential for modeling complex data.
CLEVE: Contrastive Pre-training for Event Extraction
Event extraction (EE) has considerably benefited from pre-trained language models (PLMs) by fine-tuning.
Knowledge Inheritance for Pre-trained Language Models
Specifically, we introduce a pre-training framework named "knowledge inheritance" (KI) and explore how could knowledge distillation serve as auxiliary supervision during pre-training to efficiently learn larger PLMs.
Hidden Killer: Invisible Textual Backdoor Attacks with Syntactic Trigger
As far as we know, almost all existing textual backdoor attack methods insert additional contents into normal samples as triggers, which causes the trigger-embedded samples to be detected and the backdoor attacks to be blocked without much effort.
Automatic Construction of Sememe Knowledge Bases via Dictionaries
We use this method to build an English SKB and a French SKB, and conduct comprehensive evaluations from both intrinsic and extrinsic perspectives.
PTR: Prompt Tuning with Rules for Text Classification
This indicates that PTR is a promising approach to take advantage of both human prior knowledge and PLMs for those complicated classification tasks.
Manual Evaluation Matters: Reviewing Test Protocols of Distantly Supervised Relation Extraction
Distantly supervised (DS) relation extraction (RE) has attracted much attention in the past few years as it can utilize large-scale auto-labeled data.
Few-NERD: A Few-Shot Named Entity Recognition Dataset
In this paper, we present Few-NERD, a large-scale human-annotated few-shot NER dataset with a hierarchy of 8 coarse-grained and 66 fine-grained entity types.
Ranked #5 on
Named Entity Recognition (NER)
on Few-NERD (SUP)
Few-Shot Conversational Dense Retrieval
In this paper, we present a Conversational Dense Retrieval system, ConvDR, that learns contextualized embeddings for multi-turn conversational queries and retrieves documents solely using embedding dot products.
Lawformer: A Pre-trained Language Model for Chinese Legal Long Documents
Legal artificial intelligence (LegalAI) aims to benefit legal systems with the technology of artificial intelligence, especially natural language processing (NLP).
Is Multi-Hop Reasoning Really Explainable? Towards Benchmarking Reasoning Interpretability
However, we find in experiments that many paths given by these models are actually unreasonable, while little works have been done on interpretability evaluation for them.
Incentivizing Exploration in Linear Bandits under Information Gap
We study the problem of incentivizing exploration for myopic users in linear bandits, where the users tend to exploit arm with the highest predicted reward instead of exploring.
Visual Distant Supervision for Scene Graph Generation
In this work, we propose visual distant supervision, a novel paradigm of visual relation learning, which can train scene graph models without any human-labeled data.
Equality before the Law: Legal Judgment Consistency Analysis for Fairness
We propose to simulate judges from different groups with legal judgment prediction (LJP) models and measure the judicial inconsistency with the disagreement of the judgment results given by LJP models trained on different groups.
Infrastructure Assisted Constrained Connected Automated Vehicle Trajectory Optimization on Curved Roads: A Spatial Formulation on a Curvilinear Coordinate
This paper presents an infrastructure assisted constrained connected automated vehicles (CAVs) trajectory optimization method on curved roads.
UPRec: User-Aware Pre-training for Recommender Systems
Existing sequential recommendation methods rely on large amounts of training data and usually suffer from the data sparsity problem.
Representation Learning for Natural Language Processing
This book aims to review and present the recent advances of distributed representation learning for NLP, including why representation learning can improve NLP, how representation learning takes part in various important topics of NLP, and what challenges are still not well addressed by distributed representation.
CSS-LM: A Contrastive Framework for Semi-supervised Fine-tuning of Pre-trained Language Models
We then perform contrastive semi-supervised learning on both the retrieved unlabeled and original labeled instances to help PLMs capture crucial task-related semantic features.
OpenMatch: An Open Source Library for Neu-IR Research
OpenMatch is a Python-based library that serves for Neural Information Retrieval (Neu-IR) research.
Red Alarm for Pre-trained Models: Universal Vulnerability to Neuron-Level Backdoor Attacks
In this work, we demonstrate the universal vulnerability of PTMs, where fine-tuned PTMs can be easily controlled by backdoor attacks in arbitrary downstream tasks.
Better Robustness by More Coverage: Adversarial Training with Mixup Augmentation for Robust Fine-tuning
In this work, we propose a simple and effective method to cover a much larger proportion of the attack search space, called Adversarial and Mixup Data Augmentation (AMDA).
ERICA: Improving Entity and Relation Understanding for Pre-trained Language Models via Contrastive Learning
Pre-trained Language Models (PLMs) have shown superior performance on various downstream Natural Language Processing (NLP) tasks.
Few-Shot Text Ranking with Meta Adapted Synthetic Weak Supervision
To democratize the benefits of Neu-IR, this paper presents MetaAdaptRank, a domain adaptive learning method that generalizes Neu-IR models from label-rich source domains to few-shot target domains.
CPM: A Large-scale Generative Chinese Pre-trained Language Model
However, applying GPT-3 to address Chinese NLP tasks is still challenging, as the training corpus of GPT-3 is primarily English, and the parameters are not publicly available.
Try to Substitute: An Unsupervised Chinese Word Sense Disambiguation Method Based on HowNet
In this paper, we propose a new unsupervised method for HowNet-based Chinese WSD, which exploits the masked language model task of pre-trained language models.
Meta-Information Guided Meta-Learning for Few-Shot Relation Classification
Few-shot classification requires classifiers to adapt to new classes with only a few training instances.
Neural Gibbs Sampling for Joint Event Argument Extraction
Existing EAE methods either extract each event argument roles independently or sequentially, which cannot adequately model the joint probability distribution among event arguments and their roles.
ExpanRL: Hierarchical Reinforcement Learning for Course Concept Expansion in MOOCs
Within the prosperity of Massive Open Online Courses (MOOCs), the education applications that automatically provide extracurricular knowledge for MOOC users become rising research topics.
Hierarchical Reinforcement Learning
reinforcement-learning
+1
ONION: A Simple and Effective Defense Against Textual Backdoor Attacks
Nevertheless, there are few studies on defending against textual backdoor attacks.
TLab: Traffic Map Movie Forecasting Based on HR-NET
In our 2020 Competition solution, we further design multiple variants based on HR-NET and UNet.
