Search Results for author:
Found 4 papers, 0 papers with code
Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs
For model evaluation, we establish a comprehensive benchmark encompassing all the aforementioned tasks.
ILuvUI: Instruction-tuned LangUage-Vision modeling of UIs from Machine Conversations
Multimodal Vision-Language Models (VLMs) enable powerful applications from their fused understanding of images and language, but many perform poorly on UI tasks due to the lack of UI training data.
AXNav: Replaying Accessibility Tests from Natural Language
Recently, Large Language Models (LLMs) have been used for a variety of tasks including automation of UIs, however to our knowledge no one has yet explored their use in controlling assistive technologies for the purposes of supporting accessibility testing.
Scout: Rapid Exploration of Interface Layout Alternatives through High-Level Design Constraints
Although exploring alternatives is fundamental to creating better interface designs, current processes for creating alternatives are generally manual, limiting the alternatives a designer can explore.
