OpenEQA

The OpenEQA dataset is a significant contribution in the field of Embodied Question Answering (EQA). Let me provide you with some details:

1. Definition:
2. Embodied Question Answering (EQA) involves understanding an environment well enough to answer questions about it in natural language.

3. EQA agents can achieve this understanding through either episodic memory (as seen in agents using smart glasses) or active exploration of the environment (as in the case of mobile robots).

4. OpenEQA Dataset:

5. OpenEQA is the first open-vocabulary benchmark dataset for EQA that supports both episodic memory and active exploration use cases.
6. It contains over 1600 high-quality human-generated questions drawn from more than 180 real-world environments.
7. The dataset consists of question-answer pairs ($Q, A^$) and corresponding episode histories* ($H$).
8. You can find the question-answer pairs in the file data/open-eqa-v0.json.

9. To access the episode histories, follow the instructions provided here.

10. Evaluation Protocol:

11. OpenEQA also provides an automatic evaluation protocol powered by language model-based evaluation (LLM).

12. This evaluation protocol correlates well with human judgment.

13. Foundation Models Evaluation:

14. Researchers evaluated several state-of-the-art foundation models, including GPT-4V, using the OpenEQA dataset.

15. The findings revealed that these models significantly lag behind human-level performance in EQA tasks.

16. Significance:

17. OpenEQA serves as a straightforward, measurable, and practically relevant benchmark for current-generation foundation models.
18. It poses a considerable challenge and inspires research at the intersection of Embodied AI, conversational agents, and world models.