Search Results for author:
Found 76 papers, 35 papers with code
Multimodal Intent Discovery from Livestream Videos
We construct and present a new multimodal dataset consisting of software instructional livestreams and containing manual annotations for both detailed and abstract procedural intent that enable training and evaluation of joint video and text understanding models.
Keyphrase Prediction from Video Transcripts: New Dataset and Directions
To this end, this work studies new challenges of KP in transcripts of videos, an understudied domain for KP that involves informal texts and non-cohesive presentation styles.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
ViLBERTScore: Evaluating Image Caption Using Vision-and-Language BERT
In this paper, we propose an evaluation metric for image captioning systems using both image and text information.
Virtual Knowledge Graph Construction for Zero-Shot Domain-Specific Document Retrieval
We conduct three experiments 1) domain-specific document retrieval, 2) comparison of our virtual knowledge graph construction method with previous approaches, and 3) ablation study on each component of our virtual knowledge graph.
Offensive Content Detection via Synthetic Code-Switched Text
The prevalent use of offensive content in social media has become an important reason for concern for online platforms (customer service chat-boxes, social media platforms, etc).
UCSD-Adobe at MEDIQA 2021: Transfer Learning and Answer Sentence Selection for Medical Summarization
We show that both transfer learning methods combined achieve the highest ROUGE scores.
Scaling Up Video Summarization Pretraining with Large Language Models
Long-form video content constitutes a significant portion of internet traffic, making automated video summarization an essential research problem.
PEEB: Part-based Image Classifiers with an Explainable and Editable Language Bottleneck
CLIP-based classifiers rely on the prompt containing a {class name} that is known to the text encoder.
Fine-tuning CLIP Text Encoders with Two-step Paraphrasing
Contrastive language-image pre-training (CLIP) models have demonstrated considerable success across various vision-language tasks, such as text-to-image retrieval, where the model is required to effectively process natural language input to produce an accurate visual output.
Multi-Modal Video Topic Segmentation with Dual-Contrastive Domain Adaptation
Video topic segmentation unveils the coarse-grained semantic structure underlying videos and is essential for other video understanding tasks.
LRM: Large Reconstruction Model for Single Image to 3D
We propose the first Large Reconstruction Model (LRM) that predicts the 3D model of an object from a single input image within just 5 seconds.
Aspect-based Meeting Transcript Summarization: A Two-Stage Approach with Weak Supervision on Sentence Classification
Then we merge the sentences selected for a specific aspect as the input for the summarizer to produce the aspect-based summary.
Multilingual Sentence-Level Semantic Search using Meta-Distillation Learning
Our approach leverages meta-distillation learning based on MAML, an optimization-based Model-Agnostic Meta-Learner.
Boosting Punctuation Restoration with Data Generation and Reinforcement Learning
Punctuation restoration is an important task in automatic speech recognition (ASR) which aim to restore the syntactic structure of generated ASR texts to improve readability.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Learning Navigational Visual Representations with Semantic Map Supervision
Being able to perceive the semantics and the spatial structure of the environment is essential for visual navigation of a household robot.
ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning
The answer to this question requires a thorough evaluation of ChatGPT over multiple tasks with diverse languages and large datasets (i. e., beyond reported anecdotes), which is still missing or limited in current research.
Harnessing the Spatial-Temporal Attention of Diffusion Models for High-Fidelity Text-to-Image Synthesis
We then impose spatial attention control by combining the attention over the entire text description and that over the local description of the particular object in the corresponding pixel region of that object.
PR-MCS: Perturbation Robust Metric for MultiLingual Image Captioning
Vulnerability to lexical perturbation is a critical weakness of automatic evaluation metrics for image captioning.
Align and Attend: Multimodal Summarization with Dual Contrastive Losses
The goal of multimodal summarization is to extract the most important information from different modalities to form output summaries.
Ranked #3 on
Supervised Video Summarization
on SumMe
Extractive Text Summarization
Supervised Video Summarization
Moment Detection in Long Tutorial Videos
To study this problem, we propose the first dataset of untrimmed, long-form tutorial videos for the task of Moment Detection called the Behance Moment Detection (BMD) dataset.
Uncovering the Disentanglement Capability in Text-to-Image Diffusion Models
Based on this finding, we further propose a simple, light-weight image editing algorithm where the mixing weights of the two text embeddings are optimized for style matching and content preservation.
LiveSeg: Unsupervised Multimodal Temporal Segmentation of Long Livestream Videos
Livestream videos have become a significant part of online learning, where design, digital marketing, creative painting, and other skills are taught by experienced experts in the sessions, making them valuable materials.
Semantics-Consistent Cross-domain Summarization via Optimal Transport Alignment
Multimedia summarization with multimodal output (MSMO) is a recently explored application in language grounding.
Improving Local Features with Relevant Spatial Information by Vision Transformer for Crowd Counting
ViT is adapted on each patch to employ the attention mechanism across the 3 × 3 cells to count the number of people in the central cell.
Ranked #4 on
Crowd Counting
on ShanghaiTech A
Medical Question Understanding and Answering with Knowledge Grounding and Semantic Self-Supervision
The system first matches the summarized user question with an FAQ from a trusted medical knowledge base, and then retrieves a fixed number of relevant sentences from the corresponding answer document.
PiC: A Phrase-in-Context Dataset for Phrase Understanding and Semantic Search
While contextualized word embeddings have been a de-facto standard, learning contextualized phrase embeddings is less explored and being hindered by the lack of a human-annotated benchmark that tests machine understanding of phrase semantics given a context sentence or paragraph (instead of phrases alone).
Fine-grained Image Captioning with CLIP Reward
Toward more descriptive and distinctive caption generation, we propose using CLIP, a multimodal encoder trained on huge image-text pairs from web, to calculate multimodal similarity and use it as a reward function.
Ranked #26 on
Image Captioning
on COCO Captions
Factual Error Correction for Abstractive Summaries Using Entity Retrieval
In this paper, we propose an efficient factual error correction system RFEC based on entities retrieval post-editing process.
MHMS: Multimodal Hierarchical Multimedia Summarization
Multimedia summarization with multimodal output can play an essential role in real-world applications, i. e., automatically generating cover images and titles for news articles or providing introductions to online videos.
CAISE: Conversational Agent for Image Search and Editing
To our knowledge, this is the first dataset that provides conversational image search and editing annotations, where the agent holds a grounded conversation with users and helps them to search and edit images according to their requests.
Double Trouble: How to not explain a text classifier's decisions using counterfactuals synthesized by masked language models?
We find two reasons why IM is not better than LOO: (1) deleting a single word from the input only marginally reduces a classifier's accuracy; and (2) a highly predictable word is always given near-zero attribution, regardless of its true importance to the classifier.
Few-Shot Intent Detection via Contrastive Pre-Training and Fine-Tuning
In this work, we focus on a more challenging few-shot intent detection scenario where many intents are fine-grained and semantically similar.
StreamHover: Livestream Transcript Summarization and Annotation
With the explosive growth of livestream broadcasting, there is an urgent need for new summarization technology that enables us to create a preview of streamed content and tap into this wealth of knowledge.
A Gradually Soft Multi-Task and Data-Augmented Approach to Medical Question Understanding
Users of medical question answering systems often submit long and detailed questions, making it hard to achieve high recall in answer retrieval.
End-to-end Neural Coreference Resolution Revisited: A Simple yet Effective Baseline
Since the first end-to-end neural coreference resolution model was introduced, many extensions to the model have been proposed, ranging from using higher-order inference to directly optimizing evaluation metrics using reinforcement learning.
UMIC: An Unreferenced Metric for Image Captioning via Contrastive Learning
Also, we observe critical problems of the previous benchmark dataset (i. e., human annotations) on image captioning metric, and introduce a new collection of human annotations on the generated captions.
Learning by Planning: Language-Guided Global Image Editing
Recently, language-guided global image editing draws increasing attention with growing application potentials.
X-METRA-ADA: Cross-lingual Meta-Transfer Learning Adaptation to Natural Language Understanding and Question Answering
We extensively evaluate our framework on two challenging cross-lingual NLU tasks: multilingual task-oriented dialog and typologically diverse question answering.
A Context-Dependent Gated Module for Incorporating Symbolic Semantics into Event Coreference Resolution
Event coreference resolution is an important research problem with many applications.
Out of Order: How Important Is The Sequential Order of Words in a Sentence in Natural Language Understanding Tasks?
Encouraging classifiers to capture word order information improves the performance on most GLUE tasks, SQuAD 2. 0 and out-of-samples.
Natural Language Inference
Natural Language Understanding
+2
AutoNLU: An On-demand Cloud-based Natural Language Understanding System for Enterprises
With the renaissance of deep learning, neural networks have achieved promising results on many natural language understanding (NLU) tasks.
A Joint Learning Approach based on Self-Distillation for Keyphrase Extraction from Scientific Documents
Experimental results show that our approach consistently improves the performance of baseline models for keyphrase extraction.
Scene Graph Modification Based on Natural Language Commands
In this paper, we explore the novel problem of graph modification, where the systems need to learn how to update an existing scene graph given a new user's command.
A Benchmark and Baseline for Language-Driven Image Editing
To solve this new task, we first present a new language-driven image editing dataset that supports both local and global editing with editing operation and mask annotations.
PhraseCut: Language-based Image Segmentation in the Wild
We consider the problem of segmenting image regions given a natural language phrase, and study it on a novel dataset of 77, 262 images and 345, 486 phrase-region pairs.
Ranked #4 on
Referring Expression Segmentation
on PhraseCut
Bayesian Optimization for Selecting Efficient Machine Learning Models
The performance of many machine learning models depends on their hyper-parameter settings.
ISA: An Intelligent Shopping Assistant
Despite the growth of e-commerce, brick-and-mortar stores are still the preferred destinations for many people.
Open-Domain Question Answering with Pre-Constructed Question Spaces
Our reader-retriever first uses an offline reader to read the corpus and generate collections of all answerable questions associated with their answers, and then uses an online retriever to respond to user queries by searching the pre-constructed question spaces for answers that are most likely to be asked in the given way.
Efficient Deployment of Conversational Natural Language Interfaces over Databases
Many users communicate with chatbots and AI assistants in order to help them with various tasks.
History for Visual Dialog: Do we really need it?
Visual Dialog involves "understanding" the dialog history (what has been discussed previously) and the current question (what is asked), in addition to grounding information in the image, to generate the correct response.
KPQA: A Metric for Generative Question Answering Using Keyphrase Weights
To evaluate our metric, we create high-quality human judgments of correctness on two GenQA datasets.
DSTC8-AVSD: Multimodal Semantic Transformer Network with Retrieval Style Word Generator
Audio Visual Scene-aware Dialog (AVSD) is the task of generating a response for a question with a given scene, video, audio, and the history of previous turns in the dialog.
A Multimodal Dialogue System for Conversational Image Editing
In this paper, we present a multimodal dialogue system for Conversational Image Editing.
Adjusting Image Attributes of Localized Regions with Low-level Dialogue
Our system grounds language on the level of edit operations, and suggests options for a user to choose from.
Variational Hierarchical Dialog Autoencoder for Dialog State Tracking Data Augmentation
Recent works have shown that generative data augmentation, where synthetic samples generated from deep generative models complement the training dataset, benefit NLP tasks.
Rethinking Self-Attention: Towards Interpretability in Neural Parsing
Finally, we find that the Label Attention heads learn relations between syntactic categories and show pathways to analyze errors.
Ranked #1 on
Dependency Parsing
on Penn Treebank
A Simple but Effective BERT Model for Dialog State Tracking on Resource-Limited Systems
In a task-oriented dialog system, the goal of dialog state tracking (DST) is to monitor the state of the conversation from the dialog history.
Ranked #4 on
Dialogue State Tracking
on Wizard-of-Oz
A Gated Self-attention Memory Network for Answer Selection
Answer selection is an important research problem, with applications in many areas.
Propagate-Selector: Detecting Supporting Sentences for Question Answering via Graph Neural Networks
In this study, we propose a novel graph neural network called propagate-selector (PS), which propagates information over sentences to understand information that cannot be inferred when considering sentences in isolation.
Exploiting semi-supervised training through a dropout regularization in end-to-end speech recognition
In this paper, we explore various approaches for semi supervised learning in an end to end automatic speech recognition (ASR) framework.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Expressing Visual Relationships via Language
To push forward the research in this direction, we first introduce a new language-guided image editing dataset that contains a large number of real image pairs with corresponding editing instructions.
A Compare-Aggregate Model with Latent Clustering for Answer Selection
In this paper, we propose a novel method for a sentence-level answer-selection task that is a fundamental problem in natural language processing.
Ranked #8 on
Question Answering
on TrecQA
Dance Dance Generation: Motion Transfer for Internet Videos
This work presents computational methods for transferring body movements from one person to another with videos collected in the wild.
Supervised Transfer Learning for Product Information Question Answering
Popular e-commerce websites such as Amazon offer community question answering systems for users to pose product related questions and experienced customers may provide answers voluntarily.
A System for Automated Image Editing from Natural Language Commands
This work presents the task of modifying images in an image editing program using natural language written commands.
A Review on Deep Learning Techniques Applied to Answer Selection
Given a question and a set of candidate answers, answer selection is the task of identifying which of the candidates answers the question correctly.
A Simple End-to-End Question Answering Model for Product Information
When evaluating a potential product purchase, customers may have many questions in mind.
Conversational Image Editing: Incremental Intent Identification in a New Dialogue Task
We present {``}conversational image editing{''}, a novel real-world application domain combining dialogue, visual information, and the use of computer vision.
The Context-Dependent Additive Recurrent Neural Net
Contextual sequence mapping is one of the fundamental problems in Natural Language Processing (NLP).
A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents
Neural abstractive summarization models have led to promising results in summarizing relatively short documents.
Ranked #4 on
Unsupervised Extractive Summarization
on Pubmed
Abstractive Text Summarization
Unsupervised Extractive Summarization
Visual to Sound: Generating Natural Sound for Videos in the Wild
As two of the five traditional human senses (sight, hearing, taste, smell, and touch), vision and sound are basic sources through which humans understand the world.
AMC: Attention guided Multi-modal Correlation Learning for Image Search
According to the intent of query, attention mechanism can be introduced to adaptively balance the importance of different modalities.
Proposing Plausible Answers for Open-ended Visual Question Answering
One of the most interesting recent open-ended question answering challenges is Visual Question Answering (VQA) which attempts to evaluate a system's visual understanding through its answers to natural language questions about images.
