Search Results for author:
Found 23 papers, 7 papers with code
Introducing v0.5 of the AI Safety Benchmark from MLCommons
We created a new taxonomy of 13 hazard categories, of which 7 have tests in the v0. 5 benchmark.
Elephants Never Forget: Memorization and Learning of Tabular Data in Large Language Models
We then compare the few-shot learning performance of LLMs on datasets that were seen during training to the performance on datasets released after training.
KITAB: Evaluating LLMs on Constraint Satisfaction for Information Retrieval
Motivated by rising concerns around factual incorrectness and hallucinations of LLMs, we present KITAB, a new dataset for measuring constraint satisfaction abilities of language models.
Diversity of Thought Improves Reasoning Abilities of LLMs
Large language models (LLMs) are documented to struggle in settings that require complex reasoning.
Attention Satisfies: A Constraint-Satisfaction Lens on Factual Errors of Language Models
We investigate the internal behavior of Transformer-based Large Language Models (LLMs) when they generate factually incorrect text.
Mitigating Spurious Correlations in Multi-modal Models during Fine-tuning
Spurious correlations that degrade model generalization or lead the model to be right for the wrong reasons are one of the main robustness concerns for real-world deployments.
Social Biases through the Text-to-Image Generation Lens
In this paper, we take a multi-dimensional approach to studying and quantifying common social biases as reflected in the generated images, by focusing on how occupations, personality traits, and everyday situations are depicted across representations of (perceived) gender, age, race, and geographical location.
Benchmarking Spatial Relationships in Text-to-Image Generation
We investigate the ability of T2I models to generate correct spatial relationships among objects and present VISOR, an evaluation metric that captures how accurately the spatial relationship described in text is generated in the image.
Advancing Human-AI Complementarity: The Impact of User Expertise and Algorithmic Tuning on Joint Decision Making
In addition, we found that users' perception of the AI's performance relative on their own also had a significant impact on whether their accuracy improved when given AI recommendations.
Who Goes First? Influences of Human-AI Workflow on Decision Making in Clinical Imaging
A critical aspect of interaction design for AI-assisted human decision making are policies about the display and sequencing of AI inferences within larger decision-making workflows.
Investigations of Performance and Bias in Human-AI Teamwork in Hiring
In AI-assisted decision-making, effective hybrid (human-AI) teamwork is not solely dependent on AI performance alone, but also on its impact on human decision-making.
Hierarchical Analysis of Visual COVID-19 Features from Chest Radiographs
Chest radiography has been a recommended procedure for patient triaging and resource management in intensive care units (ICUs) throughout the COVID-19 pandemic.
Understanding Failures of Deep Networks via Robust Feature Extraction
Traditional evaluation metrics for learned models that report aggregate scores over a test set are insufficient for surfacing important and informative patterns of failure over features and instances.
An Empirical Analysis of Backward Compatibility in Machine Learning Systems
In many applications of machine learning (ML), updates are performed with the goal of enhancing model performance.
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
However, prior studies observed improvements from explanations only when the AI, alone, outperformed both the human and the best team.
Is the Most Accurate AI the Best Teammate? Optimizing AI for Teamwork
To optimize the team performance for this setting we maximize the team's expected utility, expressed in terms of the quality of the final decision, cost of verifying, and individual accuracies of people and machines.
SQuINTing at VQA Models: Introspecting VQA Models with Sub-Questions
We quantify the extent to which this phenomenon occurs by creating a new Reasoning split of the VQA dataset and collecting VQA-introspect, a new dataset1 which consists of 238K new perception questions which serve as sub questions corresponding to the set of perceptual tasks needed to effectively answer the complex reasoning questions in the Reasoning split.
What You See Is What You Get? The Impact of Representation Criteria on Human Bias in Hiring
Although systematic biases in decision-making are widely documented, the ways in which they emerge from different sources is less understood.
A Case for Backward Compatibility for Human-AI Teams
We introduce the notion of the compatibility of an AI update with prior user experience and present methods for studying the role of compatibility in human-AI teams.
Metareasoning in Modular Software Systems: On-the-Fly Configuration using Reinforcement Learning with Rich Contextual Representations
We address the opportunity to maximize the utility of an overall computing system by employing reinforcement learning to guide the configuration of the set of interacting modules that comprise the system.
Towards Accountable AI: Hybrid Human-Machine Analyses for Characterizing System Failure
We present Pandora, a set of hybrid human-machine methods and tools for describing and explaining system failures.
On Human Intellect and Machine Failures: Troubleshooting Integrative Machine Learning Systems
We study the problem of troubleshooting machine learning systems that rely on analytical pipelines of distinct components.
Crowd Access Path Optimization: Diversity Matters
Based on this intuitive idea, we introduce the Access Path Model (APM), a novel crowd model that leverages the notion of access paths as an alternative way of retrieving information.
