# KnowPC: Knowledge-Driven Programmatic Reinforcement Learning for Zero-shot Coordination

**Authors:** Yin Gu, Qi Liu, Zhi Li, Kai Zhang  
**arXiv:** [2408.04336](https://arxiv.org/abs/2408.04336)  
**Date:** 2024-08-08  
**Summary page:** https://www.summarizepaper.com/en/paper/2408.04336v1/

## Key points

- The paper addresses the challenge of zero-shot coordination (ZSC) in cooperative AI.
- Deep reinforcement learning (DRL) combined with self-play or population-based methods is commonly used for ZSC, but relies on black-box neural networks as policy functions.
- The authors propose using interpretable programs instead of neural networks to represent the agent's policy for better interpretability and generalization ability.
- KnowPC is introduced as a Knowledge-driven Programmatic reinforcement learning approach for zero-shot coordination, utilizing a Domain-Specific Language (DSL) with program structures, conditional primitives, and action primitives.
- One challenge is the vast search space for high-performing programs, which is addressed by integrating an extractor and a reasoner to identify environmental transition knowledge and deduce preconditions of action primitives.
- Overall, KnowPC aims to enhance interpretability and generalization ability of cooperative AI agents by leveraging interpretable programs and environmental transition knowledge.

## Summary

The paper "KnowPC: Knowledge-Driven Programmatic Reinforcement Learning for Zero-shot Coordination" by Yin Gu, Qi Liu, Zhi Li, and Kai Zhang addresses the challenge of zero-shot coordination (ZSC) in the field of cooperative AI. ZSC involves training an agent to cooperate with an unseen partner in both familiar and novel environments. While deep reinforcement learning (DRL) combined with self-play or population-based methods has been a popular solution for ZSC, these approaches often rely on black-box neural networks as policy functions. However, neural networks lack interpretability and logic, making it challenging for partners such as humans to understand the learned policies and limiting their generalization ability. To overcome these limitations, the authors propose representing the agent's policy using interpretable programs instead of neural networks. Programs offer stable logic but are non-differentiable and difficult to optimize. To automatically learn such programs, they introduce KnowPC, a Knowledge-driven Programmatic reinforcement learning approach for zero-shot coordination. This method defines a Domain-Specific Language (DSL) that includes program structures, conditional primitives, and action primitives. One significant challenge in implementing KnowPC is the vast search space for high-performing programs, making it difficult to find efficient solutions. To address this issue, KnowPC integrates an extractor and a reasoner. The extractor identifies environmental transition knowledge from multi-agent interaction trajectories while the reasoner deduces the preconditions of each action primitive based on this transition knowledge. Overall,<Organization>, KnowPC aims to enhance the interpretability and generalization ability of cooperative AI agents by utilizing interpretable programs instead of black-box neural networks.By leveraging environmental transition knowledge and integrating extractor-reasoner components,<Organization> offers a promising approach to tackling the zero-shot coordination challenge in diverse cooperative scenarios.