Reference-Limited Compositional Learning: A Realistic Assessment for Human-level Compositional Generalization

To narrow the considerable gap between artificial and human intelligence, we propose a new task, namely reference-limited compositional learning (RLCL), which reproduces three core challenges to mimic human perception: compositional learning, few-shot, and few referential compositions. Building upon the setting, we propose two benchmarks that consist of multiple datasets with diverse compositional labels, providing a suitable and realistic platform for systematically assessing progress on the task. Moreover, we extend popular few-shot and compositional learning approaches to serve as baselines, and also introduce a simple method that achieves better performance in recognizing unseen compositions. Extensive experiments demonstrate that existing solutions struggle with the challenges imposed by the RLCL task, revealing substantial research space for pursuing human-level compositional generalization ability.