A simple neural network module for relational reasoning

Title

A neural network module called Relation Networks (RNs) is introduced for relational reasoning.

Abstract

Relation Networks (RNs) are introduced as a plug-and-play module to enable neural networks to perform relational reasoning, achieving state-of-the-art performance on tasks like visual question answering (CLEVR), text-based question answering (bAbI), and dynamic physical systems.

1 Introduction

Relation Networks (RNs) are presented as a general solution for neural networks to tackle relational reasoning, outperforming standard architectures on tasks like visual question answering (CLEVR).

2 Relation Networks

Relation Networks (RNs) are a neural network module designed with a structure that explicitly computes relations between objects, offering strengths in learning relations, data efficiency, and order invariance.

3 Tasks

RN-augmented networks were applied to various tasks including visual question answering (CLEVR), text-based question answering (bAbI), and dynamic physical systems to demonstrate their versatility in relational reasoning.

3.2 Sort-of-CLEVR

Sort-of-CLEVR dataset was created to separate relational and non-relational questions, demonstrating that standard neural architectures struggle with relational questions while RNs can solve them.

3.3 bAbI

The bAbI text-based question answering dataset, comprising 20 tasks of various reasoning types, was used to evaluate RN-based architectures, which achieved success on 18 out of 20 tasks.

3.4 Dynamic physical systems

A dataset of simulated physical mass-spring systems was used to train RN models to infer connections and count systems based on observed ball movements, outperforming MLPs.

4 Models

RNs can integrate with conventional neural network modules like CNNs and LSTMs by treating their outputs as sets of objects, demonstrating flexibility in handling unstructured inputs for relational reasoning.

5 Results

RN-augmented models achieved state-of-the-art, super-human performance on the CLEVR visual question answering task, highlighting their effectiveness in relational reasoning.

5.2 CLEVR from state descriptions

The RN module demonstrated robustness and generality by achieving high accuracy on the CLEVR dataset when using state descriptions as input, showing applicability beyond visual problems.

5.3 Sort-of-CLEVR from pixels

RN-augmented CNNs significantly outperformed CNNs augmented with MLPs on relational questions in the Sort-of-CLEVR dataset, confirming the necessity of a dedicated relational reasoning component.

5.4 bAbI

The RN model achieved 18 out of 20 tasks passed on the bAbI dataset, notably succeeding on the basic induction task where other advanced models struggled.

5.5 Dynamic physical systems

RN models achieved high accuracy (93% for connection inference, 95% for counting) on dynamic physical systems tasks, demonstrating superior performance compared to MLPs and enabling transfer learning to unseen motion capture data.

6 Discussion and Conclusions

Relation Networks (RNs) are a versatile and powerful module that can be integrated into deep learning architectures to significantly improve performance on tasks requiring rich relational reasoning.

Acknowledgments

The authors express gratitude to individuals and teams for their feedback and discussions that contributed to the research.

Supplementary Material

This section provides supplementary details on related work, experimental setups for various tasks, and model architectures, highlighting the versatility and effectiveness of Relation Networks.

BCLEVR Training Details

The BCLEVR model was trained on 70000 CLEVR scenes and 699989 questions, achieving 96.8% accuracy with smaller models performing best.

Failure Cases and Limitations

The model's failure cases, often shared with humans, occur with heavy occlusion or when precise object positioning is required.

CCLEVR Training Details

The CCLEVR model, trained on state descriptions without a vision module, utilized a 256 unit LSTM and specific MLP configurations for question and relation processing.

Sort-of-CLEVR Dataset Details

The Sort-of-CLEVR dataset features 10000 images with relational and non-relational questions, where questions are encoded as binary strings.

Sort-of-CLEVR Model Architecture

The Sort-of-CLEVR model uses convolutional layers, question embeddings, and MLPs for relational reasoning, optimized with Adam and cross-entropy loss.

bAbI Model Details

For the bAbI task, a 32 unit LSTM processes support set sentences, feeding into an RN with MLPs for relation and answer processing, optimized via Adam and cross-entropy.

Dynamic Physical System Reasoning

The model reasons about dynamic physical systems for connection inference and counting tasks, using object properties across time-frames as input to the RN.