About

Benchmarks

No evaluation results yet. Help compare methods by submit evaluation metrics.

Datasets

Greatest papers with code

Learning to Execute Programs with Instruction Pointer Attention Graph Neural Networks

NeurIPS 2020 google-research/google-research

More practically, we evaluate these models on the task of learning to execute partial programs, as might arise if using the model as a heuristic function in program synthesis.

LEARNING TO EXECUTE PROGRAM REPAIR SYSTEMATIC GENERALIZATION

Multi-Object Representation Learning with Iterative Variational Inference

1 Mar 2019deepmind/deepmind-research

Human perception is structured around objects which form the basis for our higher-level cognition and impressive systematic generalization abilities.

REPRESENTATION LEARNING SYSTEMATIC GENERALIZATION VARIATIONAL INFERENCE

Prioritized Level Replay

8 Oct 2020maximecb/gym-minigrid

We introduce Prioritized Level Replay, a general framework for estimating the future learning potential of a level given the current state of the agent's policy.

SYSTEMATIC GENERALIZATION

The NetHack Learning Environment

NeurIPS 2020 facebookresearch/nle

Here, we present the NetHack Learning Environment (NLE), a scalable, procedurally generated, stochastic, rich, and challenging environment for RL research based on the popular single-player terminal-based roguelike game, NetHack.

NETHACK SCORE SYSTEMATIC GENERALIZATION

CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Text

IJCNLP 2019 facebookresearch/clutrr

The recent success of natural language understanding (NLU) systems has been troubled by results highlighting the failure of these models to generalize in a systematic and robust way.

INDUCTIVE LOGIC PROGRAMMING NATURAL LANGUAGE UNDERSTANDING RELATIONAL REASONING SYSTEMATIC GENERALIZATION

Systematic Generalization: What Is Required and Can It Be Learned?

ICLR 2019 rizar/systematic-generalization-sqoop

Numerous models for grounded language understanding have been recently proposed, including (i) generic models that can be easily adapted to any given task and (ii) intuitively appealing modular models that require background knowledge to be instantiated.

SYSTEMATIC GENERALIZATION VISUAL QUESTION ANSWERING

Compositional Networks Enable Systematic Generalization for Grounded Language Understanding

6 Aug 2020LauraRuis/groundedSCAN

Humans are remarkably flexible when understanding new sentences that include combinations of concepts they have never encountered before.

SYSTEMATIC GENERALIZATION

A Benchmark for Systematic Generalization in Grounded Language Understanding

NeurIPS 2020 LauraRuis/groundedSCAN

In this paper, we introduce a new benchmark, gSCAN, for evaluating compositional generalization in situated language understanding.

SYSTEMATIC GENERALIZATION

CLOSURE: Assessing Systematic Generalization of CLEVR Models

12 Dec 2019rizar/CLOSURE

In this work, we study how systematic the generalization of such models is, that is to which extent they are capable of handling novel combinations of known linguistic constructs.

FEW-SHOT LEARNING SYSTEMATIC GENERALIZATION TRANSFER LEARNING

Latent Compositional Representations Improve Systematic Generalization in Grounded Question Answering

1 Jul 2020benbogin/glt-grounded-latent-trees-qa

However, state-of-the-art models in grounded question answering often do not explicitly perform decomposition, leading to difficulties in generalization to out-of-distribution examples.

QUESTION ANSWERING SYSTEMATIC GENERALIZATION