Neural Modules

Repository for training and using neural modules for hybrid procedural semantics projects. The project supports three datasets: CLEVR, CLEVR_Dialog, and MNIST_Dialog.

Table of Contents

In this document, you will find the following sections:

• Section 1. Setting up the Python environment for training the modules and running the IRL communication server.
• Section 2. Downloading the raw datasets
• Section 3. Downloading pretrained modules
• Section 4. Generating or downloading the data that is required for training the neural modules from scratch.
• Section 5. Using scripts for training the modules.
• Section 6. Using scripts for evaluating the trained modules.
• Section 7. Running the flask server for hybrid procedural semantics with IRL.
• Section 8. Running the modules locally [UNSUPPORTED]

1 Setting up the Python environment

Anaconda is one way to get up and running with Python environments. However, the full Anaconda toolkit installs 1500 Python packages and takes up 3GB of disk space. A good alternative, therefore, is Miniconda. Miniconda will only install the bare minimum to run Python. It is then up to you to manually install the packages that you really need.

Download Miniconda from this website and follow the installation instructions. If you are prompted to add Miniconda to your .bash_profile or .zshrc, accept. For the changes to take effect and use conda from the command line, you must restart the Terminal. At the time of writing, the Miniconda website suggests to install Miniconda with Python 3.9.

The environment.yml file in this repository makes it easy to recreate the neural-modules conda environment that was used to develop this project. This file lists all necessary packages and their version numbers. You can use the Makefile to setup this environment.

1. To recreate the neural-modules conda environment that was used to develop this project, run:
   • make create_environment
2. If the environment was successfully created, you can activate it by running
   • conda activate neural-modules
   • You should see the prompt of your Terminal change from (base) to (neural-modules).

Note: the Makefile will likely install CPU versions of the packages (e.g. pytorch). GPU installation will require to install the packages manually.

Optionally, the project also provides a requirements.dev file that specifies a set of development packages used to create this repository. We strongly recommend using these packages to adhere and keep the style of the project consistent.

2 Downloading the datasets

Datasets are placed in the data/ folder. This repository supports three datasets: CLEVR, CLEVR_Dialog, MNIST_Dialog. The CLEVR and CLEVR_Dialog datasets

2.1 CLEVR dataset

The CLEVR dataset can be downloaded through a bash script. From the root directory of the project, run the download script as follows:

bash scripts/download_data/clevr_raw_svn.sh

Otherwise, the CLEVR dataset can also be downloaded here from the original source.

The dataset should be stored in data/clevr. This folder has the following structure:

clevr/
| architectures/
| config.json/
| images/
| | train/
| | val/
| | test/
| questions/
| scenes/

2.2 Downloading the MNIST-Dialog dataset

The MNIST-Dialog dataset can be downloaded through a bash script. From the root directory of the project, run the download script as follows:

bash scripts/download_data/mnist_dialog_raw.sh

Otherwise, the MNIST-Dialog dataset can also be downloaded here from the original source.

The dataset should be stored in data/mnist_dialog. This folder has the following structure:

mnist_dialog/
| architectures/
| config.json/
| images/
| questions/
| scenes/

3 Downloading pretrained models

All pretrained models are downloaded from the Babel Corpora SVN. From the root directory of the project, run the download script as follows:

bash scripts/download_data/pretrained_models.sh -u <your-svn-username>

The pretrained models should be stored in models/. The CLEVR modules are stored in models/clevr. The MNIST_Dialog modules are stored in models/mnist_dialog.

4 Preprocessing the datasets

To train the modules we first require to preprocess the raw dataset scenes. This process differs depending on the chosen dataset.

4.1 Preprocessing CLEVR

The CLEVR (json) scene files are annotated with a mask for every object. The masks were obtained through a Mask R-CNN model pretrained on CLEVR-like images. The model and instructions on how to run it can be found here. Afterwards, we match the obtained masks with the JSON data. This process is automated and can be executed through the following command:

python scripts/preprocess_data/clevr/preprocess.py

The test set used in this project is different from the test set of the CLEVR dataset. This is due to the missing scene annotations from the CLEVR test set. These are however required to generate our training data. To compensate for this, we split the CLEVR training set in two parts of 60.000 scenes and 10.000 scenes respectively. The former serves as the training set and the latter as the validation set. The original validation set is then used as the test set.

The preprocess.py script works non-destructively. It will thus rename data/clevr/scenes to data/clevr/raw_scenes and hold all temporary files in this folder. Then, a new folder data/clevr/scenes is created to contain the three (train, val, test) annotated json scene files.

4.2 Preprocessing MNIST

The process for MNIST is similar to the previous section. Preprocessing is executed through the following two scripts:

python scripts/preprocess_data/mnist_dialog/split_data.py
python scripts/preprocess_data/mnist_dialog/generate_masks.py

4.3 Generating module specific datasets

The generate_module_data.py script can be used to generate training data for all modules at once. Run the following command from the root of the repository for each of the data files individually (train, val and test):

python scripts/preprocess_data/generate_module_data.py --experiment clevr
python scripts/preprocess_data/generate_module_data.py --experiment mnist_dialog

The generated module data is stored in data/<experiment>/module_data/.

Optionally, users can provide --max_samples argument to limit the amount of samples generated for each split. This can be useful to create small datasets during development. However, it is never recommended throwing away valid training samples.

5 Running the training script

Finally, each module needs to be trained separately. For this, run the scripts/train.py script. Below, we provide an example of how to train a query module on a GPU to query blue things:

python scripts/training/train_module.py \
	--experiment clevr \
	--use_cuda \
	--shuffle_data \
	--num_workers 8 \
	--num_epochs 500 \
	--max_samples_val 50000 \
	train \
	--batch_size 128 \
	--learning_rate 0.0001 \
	--module_config query_single.json \
	--module_name query_color_blue

The module_config argument specifies the amount of attention inputs required by the module. For example, a module trained to query if a single object is blue uses a query_single.json configuration. The configuration is internally used to configure the underlying deep learning architecture. Alternatively, some modules require as an input to reason about two objects. Therefore, these type of modules use the query_double.json configuration. For example, a query_relate_left module queries whether one object is left of another.

6 Evaluating a trained module

If a module is done training, one can evaluate its performance on the held-out test set using the scripts/evaluate/test_module.py script.

python scripts/evaluate/test_module.py \
        --experiment <experiment> \
        --checkpoint <path_to_checkpoint> \
        --batch_size 1

7. Running the Flask server

The Flask server makes use of a session when communicating with multiple instances, e.g. experiments using the neural modules that are running in parallel. Sessions are stored using Redis. To obtain Redis, run:

brew install redis

Run the following script and specify the experiment to start the Flask server.

python scripts/server/server.py --experiment <name>

8. Running the modules locally

The repository currently supports the evaluation of the modules locally. Currently, only the CLEVR VQA dataset is supported.

python scripts/evaluate/test_experiment.py --experiment clevr