Search Results for author:
Found 16 papers, 11 papers with code
Exposing the Limits of Video-Text Models through Contrast Sets
We test the robustness of recent methods on the proposed automatic contrast sets, and compare them to additionally collected human-generated counterparts, to assess their effectiveness.
Agent AI: Surveying the Horizons of Multimodal Interaction
To accelerate research on agent-based multimodal intelligence, we define "Agent AI" as a class of interactive systems that can perceive visual stimuli, language inputs, and other environmentally-grounded data, and can produce meaningful embodied actions.
Localized Symbolic Knowledge Distillation for Visual Commonsense Models
Empirical results and human evaluations in a zero-shot setup demonstrate that our distillation method results in more precise VL models of reasoning compared to a baseline of passing a generated referring expression to an LLM.
ArK: Augmented Reality with Knowledge Interactive Emergent Ability
In this study, we develop an infinite agent that learns to transfer knowledge memory from general foundation models (e. g. GPT4, DALLE) to novel domains or scenarios for scene understanding and generation in the physical or virtual world.
Fusing Pre-Trained Language Models With Multimodal Prompts Through Reinforcement Learning
Language models are capable of commonsense reasoning: while domain-specific models can learn from explicit knowledge (e. g. commonsense graphs [6], ethical norms [25]), and larger models like GPT-3 manifest broad commonsense reasoning capacity.
The Abduction of Sherlock Holmes: A Dataset for Visual Abductive Reasoning
We present Sherlock, an annotated corpus of 103K images for testing machine capacity for abductive reasoning beyond literal image contents.
MERLOT: Multimodal Neural Script Knowledge Models
As humans, we understand events in the visual world contextually, performing multimodal reasoning across time to make inferences about the past, present, and future.
LLC: Accurate, Multi-purpose Learnt Low-dimensional Binary Codes
We further quantitatively measure the quality of our codes by applying it to the efficient image retrieval as well as out-of-distribution (OOD) detection problems.
Natural Language Rationales with Full-Stack Visual Reasoning: From Pixels to Semantic Frames to Commonsense Graphs
Natural language rationales could provide intuitive, higher-level explanations that are easily understandable by humans, complementing the more broadly studied lower-level explanations based on gradients or attention weights.
Identity-Aware Multi-Sentence Video Description
This auxiliary task allows us to propose a two-stage approach to Identity-Aware Video Description.
LSTM-based Anomaly Detection for Non-linear Dynamical System
Anomaly detection for non-linear dynamical system plays an important role in ensuring the system stability.
VisualCOMET: Reasoning about the Dynamic Context of a Still Image
In addition, we provide person-grounding (i. e., co-reference links) between people appearing in the image and people mentioned in the textual commonsense descriptions, allowing for tighter integration between images and text.
Joint High Dynamic Range Imaging and Super-Resolution from a Single Image
This paper presents a new framework for jointly enhancing the resolution and the dynamic range of an image, i. e., simultaneous super-resolution (SR) and high dynamic range imaging (HDRI), based on a convolutional neural network (CNN).
Adversarial Inference for Multi-Sentence Video Description
Among the main issues are the fluency and coherence of the generated descriptions, and their relevance to the video.
Generation of High Dynamic Range Illumination from a Single Image for the Enhancement of Undesirably Illuminated Images
This paper presents an algorithm that enhances undesirably illuminated images by generating and fusing multi-level illuminations from a single image. The input image is first decomposed into illumination and reflectance components by using an edge-preserving smoothing filter.
Efficient Generation of Motion Plans from Attribute-Based Natural Language Instructions Using Dynamic Constraint Mapping
We generate a factor graph from natural language instructions called the Dynamic Grounding Graph (DGG), which takes latent parameters into account.
Robotics
