Learning to execute or ask clarification questions

Collaborative tasks are ubiquitous activities where a form of communication is required in order to reach a joint goal. Collaborative building is one of such tasks. To this end, we wish to develop an intelligent builder agent in a simulated building environment (Minecraft) that can build whatever users wish to build by just talking to the agent. However, in order to achieve this goal, such agents need to be able to take the initiative by asking clarification questions when further information is needed. Existing work on Minecraft Corpus Dataset only learned to execute instructions neglecting the importance of asking for clarifications. In this paper, we extend the Minecraft Corpus Dataset by annotating all builder utterances into eight types, including clarification questions, and propose a new builder agent model capable of determining when to ask or execute instructions. Experimental results show that our model achieves state-of-the-art performance on the collaborative building task with a substantial improvement. We also provide baselines for the new tasks, learning to ask and the joint tasks, which consists in solving both collaborating building and learning to ask tasks jointly.