Aligning Text and Embodied Environments for Interactive Learning Mohit Shridhar, Xingdi (Eric) Yuan, Marc-Alexandre C?té, Yonatan Bisk, Adam Trischler, Matthew Hausknecht ICLR 2021

ALFWorld contains interactive TextWorld environments (C?té et. al) that parallel embodied worlds in the ALFRED dataset (Shridhar et. al). The aligned environments allow agents to reason and learn high-level policies in an abstract space before solving embodied tasks through low-level actuation.

For the latest updates, see: alfworld.github.io

Create a virtual environment (recommended)

Install with pip (python3.9+):

Note: Without the extra, it will only install the text version of ALFWorld. To enable visual modalities, use .

Download PDDL & Game files and pre-trained MaskRCNN detector (will be stored in ):

alfworld-download

Use to download pre-trained checkpoints and seq2seq data.

Play a Textworld game:

Play an Embodied-World (THOR) game:

Get started with a random agent:

import numpy as np
from alfworld.agents.environment import get_environment
import alfworld.agents.modules.generic as generic

# load config
config = generic.load_config()
env_type = config['env']['type'] # 'AlfredTWEnv' or 'AlfredThorEnv' or 'AlfredHybrid'

# setup environment
env = get_environment(env_type)(config, train_eval='train')
env = env.init_env(batch_size=1)

# interact
obs, info = env.reset()
while True:
    # get random actions from admissible 'valid' commands (not available for AlfredThorEnv)
    admissible_commands = list(info['admissible_commands']) # note: BUTLER generates commands word-by-word without using admissible_commands
    random_actions = [np.random.choice(admissible_commands[0])]

    # step
    obs, scores, dones, infos = env.step(random_actions)
    print("Action: {}, Obs: {}".format(random_actions[0], obs[0]))

Run

Installing from source is recommended for development.

Clone repo:

Install requirements:

# Note: Requires python 3.9 or higher
virtualenv -p $(which python3.9) --system-site-packages alfworld_env # or whichever package manager you prefer
source alfworld_env/bin/activate

pip install -e .[full]

Download PDDL & Game Files and pre-trained MaskRCNN detector:

export ALFWORLD_DATA=<storage_path>
python scripts/alfworld-download

Use to download pre-trained checkpoints and seq2seq data.

Train models:

Play around with TextWorld and THOR demos.

• Data: PDDL, Game Files, Pre-trained Agents. Generating PDDL states and detection training images.
• Agents: Training and evaluating TextDAgger, TextDQN, VisionDAgger agents.
• Explore: Play around with ALFWorld TextWorld and THOR environments.

• Python 3.9+

See requirements.txt for the prerequisites to run ALFWorld text-only version. See requirements-vis.txt for the prerequisites to run ALFWorld with both text and visual modalities. See requirements-full.txt for the full prerequisites to run experiments.

Tested on:

• GPU - GTX 1080 Ti (12GB)
• CPU - Intel Xeon (Quad Core)
• RAM - 16GB
• OS - Ubuntu 16.04

Pull vzhong's image: https://hub.docker.com/r/vzhong/alfworld

OR

Install Docker and NVIDIA Docker.

Modify docker_build.py and docker_run.py to your needs.

Build the image:

For local machines:

For headless VMs and Cloud-Instances:

python docker/docker_run.py --headless

# inside docker
tmux new -s startx  # start a new tmux session

# start nvidia-xconfig
sudo nvidia-xconfig -a --use-display-device=None --virtual=1280x1024

# start X server on DISPLAY 0
# single X server should be sufficient for multiple instances of THOR
sudo python ~/alfworld/docker/startx.py 0  # if this throws errors e.g "(EE) Server terminated with error (1)" or "(EE) already running ..." try a display > 0

# detach from tmux shell
# Ctrl+b then d

# source env
source ~/alfworld_env/bin/activate

# set DISPLAY variable to match X server
export DISPLAY=:0

# check THOR
python ~/alfworld/docker/check_thor.py

###############
## (300, 300, 3)
## Everything works!!!

You might have to modify in gen/constants.py depending on which display you use.

ALFWorld can be setup on headless machines like AWS or GoogleCloud instances. The main requirement is that you have access to a GPU machine that supports OpenGL rendering. Run startx.py in a tmux shell:

# start tmux session
tmux new -s startx

# start X server on DISPLAY 0
# single X server should be sufficient for multiple instances of THOR
sudo python ~/alfworld/scripts/startx.py 0  # if this throws errors e.g "(EE) Server terminated with error (1)" or "(EE) already running ..." try a display > 0

# detach from tmux shell
# Ctrl+b then d

# set DISPLAY variable to match X server
export DISPLAY=:0

# check THOR
python ~/alfworld/docker/check_thor.py

###############
## (300, 300, 3)
## Everything works!!!

You might have to modify in gen/constants.py depending on which display you use.

Also, checkout this guide: Setting up THOR on Google Cloud

ALFWorld

ALFRED

TextWorld

• ALFWorld - MIT License
• TextWorld - MIT License
• Fast Downward - GNU General Public License (GPL) v3.0

Questions or issues? File an issue or contact Mohit Shridhar