👋[12/27/22] Can We Teach Machines to Act Like Humans? (And Should We?)

NeurIPS 2022 Outstanding Paper Award, DeepMind

Motivation and Core Insight

Subdomain: Meta-learning — learning not just a task, but a distribution of tasks. Aims to understand underlying underlying abstractions common to tasks.

Problem: Machines learn too slowly, especially compared to humans. They might not be picking up underlying abstractions.

Insight:

• Humans have inherent inductive biases, which helps them learn faster.

• If we can train models with these biases, they can also learn faster.

• We can add these biases using auxiliary tasks — making it navigate multiple representations of a task to think more like a human.

Example and Underlying Design

We want an agent to sweep the grid as per “target board” below. Alongside this, asking it to generate natural language like top-right biases its movement to be explainable, and therefore more human-like.

Adding the idea of a “U-Shape” to the library of functions a program can do allows the model to use abstractions to reach the target board.

Scaled up, one can imagine this could make reinforcement learning more efficient, removing the need for massive amounts of data.

Auxiliary tasks included:

• Human-generated natural language descriptors

• Synthetic descriptors — pseudo-code

• Programs built using library learning (maintaining an ever-growing library of functions that can be composed when trying to solve tasks)

Evaluation

• The agent got better at tasks humans are great at, and got worse at the control group tasks that humans are bad at.

  • Grounding with human-generated descriptions leads to a human-like inductive bias

  • Grounding on synthetic descriptions does not lead to human-like bias – it improves performance uniformly rather than selectively

  • Grounding on library learning programs leads to a human-like inductive bias

• Level of abstraction of the task embeddings influences the extent to which the agent acquires a human inductive bias. The mode of information wasn’t the differentiating factor. An example from the paper:

  • 

  • The embeddings of these representations that were compressed through abstract concepts may therefore be distilling useful concepts into our meta-learning agent that enables it to acquire a human-like inductive bias.

Limitations and Future Work

• They expect that the effectiveness of our approach in artificial domain could translate to abstractions in real-world domains that are natural for humans— like mathematical reasoning, planning, or puzzle solving. Future work includes determining the space of tasks for which co-training with language and program abstractions improves an artificial neural agent’s performance.

• There is still work to do to determine how language and programs can influence acquisition of human inductive biases in more scaled up, naturalistic, and real-world settings.

• On the language side, the main bottleneck is that collecting human descriptions with sufficient coverage can become prohibitively expensive as the state space grows larger.

• On the program side, the need to define ad hoc base DSLs for each domain is an important limitation of any approach relying on program abstractions

The Discussion section is a great example of a paper admitting bottle-necks and limitations instead of sweeping them under the rug. Good science.

References

[1] Paper: https://arxiv.org/abs/2205.11558