DRG-MAPPO：面向协同空战的分层动态角色图多智能体强化学习框架

Junlin Liu

The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, CASIA

School of Artificial Intelligence, University of Chinese Academy of Sciences

E-mail

{liujunlin2025, yang.gao}@ia.ac.cn

Yang Gao

The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, CASIA

Chengwei Li

The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, CASIA

School of Artificial Intelligence, University of Chinese Academy of Sciences

E-mail

{liujunlin2025, yang.gao}@ia.ac.cn

Hui Chang

The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, CASIA

Xinchen Zhang

The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, CASIA

Zhijun Zhao

The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, CASIA

Hao Zhao

The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, CASIA

Abstract

Multi-Agent Reinforcement Learning (MARL) has emerged as a pivotal paradigm for complex decision-making in autonomous systems and air combat. While MARL has demonstrated significant potential in air combat, achieving sophisticated tactical coordination remains a non-trivial challenge. This difficulty is largely attributed to two primary limitations: (1) the absence of structured relational modeling hinders agents from capturing complex, time-varying interactions among battlefield entities; and (2) conventional flat architectures often lack the capability to explicitly model tactical roles, leading to ambiguous task allocation in highly dynamic environments. To address these challenges, we propose Hierarchical Dynamic Role-Graph Multi-Agent Proximal Policy Optimization (DRG-MAPPO), a novel MARL framework that integrates graph-based relational modeling with dynamic role assignment. Specifically, DRG-MAPPO constructs a graph-based representation of battlefield interactions and leverages graph attention mechanisms to extract critical relational features among allies, enemies, and threats. Subsequently, a high-level policy employs a dynamic role assignment mechanism to determine tactical responsibilities (e.g., “leader” and “supporter”). Conditioned on these roles and encoded graph-relational features, a low-level policy executes discrete maneuver actions, facilitating the joint optimization of tactical strategy and collaborative execution. Furthermore, a target-priority auxiliary task is designed to foster the emergence of behaviors such as focus-fire. Experimental results demonstrate that DRG-MAPPO achieves a state-of-the-art win rate of 87%, suggesting that our framework effectively balances relational modeling, interpretability, and optimization stability for cooperative air combat.

Keywords: 

Multi-Agent Reinforcement Learning Hierarchical Policy Graph Attention Network Role Assignment Air Combat.

1 Introduction

Modern air combat is undergoing a fundamental paradigm shift, evolving from isolated dogfights toward sophisticated multi-UAV (Unmanned Aerial Vehicle) swarm coordination [22, 1, 26, 27, 9]. In these highly contested environments, mission success relies heavily on complex tactical synergies such as flanking maneuvers, bait-and-switch, and cooperative suppression. Existing cooperative air-combat decision systems often rely on expert rules [4, 6], optimization approaches [2, 15], or hand-crafted tactical indicators [12, 18]. These methods can encode domain-specific knowledge and behave reliably in familiar situations, but they struggle to scale in highly dynamic engagements where missile, radar, and teammate interactions jointly dictate tactical outcomes.

Multi-Agent Reinforcement Learning (MARL) has emerged as a promising paradigm for autonomous decision-making due to its efficacy in handling high-dimensional continuous spaces [7, 19, 21, 10, 11, 23]. However, existing MARL frameworks still struggle to capture the complex, time-varying topological relations among battlefield entities due to two primary limitations. (1) First, the absence of structured relational modeling impedes deep situational awareness. Without it, agents struggle to precisely perceive rapid topological shifts, thereby missing optimal tactical windows (e.g., an enemy redirecting its threat). (2) Second, traditional flat architectures lack explicit tactical role modeling, leading to role confusion and preventing the emergence of asymmetric coordination (e.g., "Leader-Wingman Decoy" tactics).

To address these challenges, we propose Dynamic Role-Graph MAPPO (DRG-MAPPO), a hierarchical MARL framework that integrates graph-based relational modeling with dynamic role assignment. DRG-MAPPO decouples the decision-making process into a two-level hierarchy. Specifically, a graph attention mechanism first constructs structured battlefield representations to capture time-varying topological dependencies. Based on these representations, a high-level policy dynamically assigns explicit tactical responsibilities (e.g., “leader” and “supporter”) to resolve allocation ambiguity, while a low-level policy executes precise maneuvers conditioned on these roles. Furthermore, we design a target-priority auxiliary task to explicitly foster asymmetric cooperative behaviors, and introduce a temporal commitment mechanism that enforces behavioral consistency within fixed decision intervals to prevent destabilizing role oscillation during training. The main contributions are summarized as follows:

• We introduce DRG-MAPPO, a hierarchical MARL framework that decouples tactical role assignment from low-level maneuver control in multi-agent air combat.

• We design a graph-attention relational module and a target-priority auxiliary task, enabling agents to capture fluid battlefield topologies and seize optimal tactical windows.

• Empirical evaluations in a high-fidelity simulation environment demonstrate that DRG-MAPPO achieves an 87% win rate, significantly outperforming state-of-the-art MARL baselines while exhibiting robust and interpretable tactical coordination.

2 Related Work

2.1 Multi-Agent Reinforcement Learning in Air Combat.

MARL has increasingly replaced traditional rule-based and heuristic optimization methods in autonomous air combat, demonstrating significant potential in handling high-dimensional continuous state spaces [21]. Algorithms like MADDPG [13], MATD3 [25], and MAPPO [24] have established a strong foundation for centralized training with decentralized execution (CTDE) in multi-agent systems. Specifically, Wu et al. [22] proposed a context-aware feature fusion method to enhance cooperative perception among multiple UAVs, improving coordination under partial observability. Ding et al. [1] introduced a layer-delay dual-center MAPPO architecture to mitigate non-stationarity in multi-UAV decision-making. However, conventional MARL frameworks typically employ flat architectures that conflate strategic intent with low-level maneuver execution, lacking explicit mechanisms for modeling inter-agent topological relationships.

2.2 Hierarchical Reinforcement Learning (HRL).

To alleviate the decision-making bottleneck in long-horizon and complex tasks, HRL has been introduced into multi-agent air combat. By decoupling the decision process, conventional HRL frameworks typically utilize a high-level policy to generate abstract subtasks or spatial subgoals, while a low-level policy focuses on executing specific maneuver control commands to achieve these sub-objectives. Building on this paradigm, recent works [17, 8, 14] proposed hierarchical architectures to explicitly decouple high-level tactical decision-making from low-level maneuver control, further leveraging mechanisms like competitive self-play to optimize dual-aircraft formation engagements. Nevertheless, these existing hierarchical approaches often rely on heuristic state partitioning or static task assignments.

2.3 Graph-based Relational Modeling.

Effective coordination in multi-agent systems necessitates structured perception of inter-entity relationships. Graph Neural Networks (GNNs), particularly Graph Attention Networks (GAT), have been increasingly leveraged to model complex interactions and extract relational features in multi-agent decision-making. Jing et al. [5] employed graph convolutional networks within a MARL framework to capture the topological dependencies among machines and operations in flexible job shop scheduling. In the domain of UAV confrontation, Hu et al. [3] proposed a GNN-enhanced MARL approach that constructs interaction graphs among UAVs to facilitate coordinated engagement strategies. However, most existing graph-based multi-agent frameworks either limit their scope to communication topologies among friendly units or employ static graph structures, struggling to capture comprehensive, time-varying topological shifts in dense friend-or-foe engagement scenarios.

3 Preliminaries

3.1 Decentralized Partially Observable Markov Decision Process

We formulate the multi-UAV cooperative air combat problem as a Dec-POMDP, defined by the tuple $ℳ=⟨ℐ,𝒮,\left\{{\Omega }_i\right\},\left\{{𝒜}_i\right\},𝒯,𝒵,\left\{R_i\right\},\gamma ⟩$, where $ℐ=\left\{1,\dots ,N\right\}$ is the set of $N$ cooperative agents; $𝒮$ denotes the global state space; ${\Omega }_i$ and ${𝒜}_i$ represent the local observation space and action space of each agent, with $𝒜={\prod }_i{𝒜}_i$ being the joint action space. The transition function $𝒯:𝒮\times 𝒜\times 𝒮\to \left[0,1\right]$ governs the probability of reaching state $s_{t+1}$ from $s_t$ given joint action $a_t$. The observation function $𝒵:𝒮\times 𝒜\to \Omega {}_1\times \cdots \times \Omega {}_N$ maps the global state and joint action to individual observations $\{o_1,\dots ,o_N\}$ for each agent, where $o_i\in {\Omega }_i$. Each agent receives a scalar reward via reward function $R_i(s_t,a_t)$, and $\gamma \in [0,1)$ is the discount factor. The goal is to learn a joint policy $\pi$ that maximizes the expected discounted cumulative return for each agent $i$:

$\underset{\pi }{\max }{𝔼}_{\pi }\left[\sum _{t=0}^T{\gamma }^t{R}_i(s_t,a_t)\right],\forall i\in ℐ$

(1)

3.2 Multi-Agent Proximal Policy Optimization (MAPPO)

We build upon the MAPPO algorithm [24] with a parameter-sharing scheme to enhance sample efficiency. A shared actor network ${\pi }_{\theta }$ performs decentralized execution based on local observations, while a centralized critic $V_{\psi }$ evaluates global states during training. Dropping the agent index $i$ for brevity, the actor minimizes a clipped surrogate loss to bound policy updates:

${ℒ}_{actor}\left(\theta \right)=-{𝔼}_t\left[\min ({\rho }_t\left(\theta \right){\hat{A}}_t,\text{clip}({\rho }_t\left(\theta \right),1-ϵ,1+ϵ){\hat{A}}_t)\right]$

(2)

where ${\rho }_t\left(\theta \right)=\frac{{\pi }_{\theta }\left(a_t\right|o_t)}{{\pi }_{{\theta }_{\text{old}}}\left(a_t\right|o_t)}$ denotes the probability ratio between the current and the previous policy, and $ϵ$ restricts the magnitude of policy updates.

The objective function for the global critic aims to minimize the Mean Squared Error (MSE) between the state value estimation and the discounted empirical return ${\hat{G}}_t$:

${ℒ}_{critic}\left(\psi \right)={𝔼}_t\left[{\left(V_{\psi }\left(s_t\right)-{\hat{G}}_t\right)}^2\right]$

(3)

Furthermore, the advantage function ${\hat{A}}_t$ is calculated utilizing the Generalized Advantage Estimation (GAE) [16] to balance the variance and bias:

${\hat{A}}_t=\sum _{k=0}^{\infty }{\left(\gamma \lambda \right)}^k{\delta }_{t+k},{\delta }_t=r_t+\gamma V_{\psi }\left(s_{t+1}\right)-V_{\psi }\left(s_t\right)$

(4)

where ${\delta }_t$ represents the standard TD error, $\gamma$ is the discount factor, and $\lambda$ acts as the GAE smoothing parameter.

4 Simulation Environment

4.1 Observation Space

Unlike conventional monolithic state vectors, we model the battlefield at each step $t$ as a directed graph ${𝒢}_t=({𝒱}_t,{ℰ}_t)$ to explicitly capture entity interactions. As illustrated in Fig. 1, the node set is partitioned into four categories (${𝒱}_t={𝒱}_{\text{self}}\cup {𝒱}_{\text{ally}}\cup {𝒱}_{\text{enemy}}\cup {𝒱}_{\text{missile}}$), where the self-node encodes the agent’s intrinsic state and the remaining nodes encode relative features. The exact mathematical compositions and detailed definitions of all feature components are summarized in Table 1.

Self-node features. The self-state ${𝐟}_{\text{self}}\in {ℝ}^{10}$ captures the agent’s kinematic state and resource availability:

${𝐟}_{\text{self}}=\left[x,y,z,v,\theta ,\varphi ,\psi ,n_m,{\delta }_{\text{det}},{\sigma }_m\right]$

(5)

Ally-node features. For each ally $j\in {𝒱}_{\text{ally}}$, the relative feature vector ${𝐟}_{\text{ally}}^j\in {ℝ}^6$ is defined as:

${𝐟}_{\text{ally}}^j=\left[d_a^j,{\theta }_a^j,\Delta v_a^j,\Delta {\psi }_a^j,v_{c,a}^j,n_{m,a}^j\right]$

(6)

Enemy-node features. For each detected enemy $k\in {𝒱}_{\text{enemy}}$, the feature vector ${𝐟}_{\text{enemy}}^k\in {ℝ}^7$ encodes the tactical geometry:

${𝐟}_{\text{enemy}}^k=\left[d_e^k,{\theta }_e^k,\Delta v_e^k,v_{c,e}^k,{\varphi }_{\text{AA}}^k,\Delta h_e^k,\Delta {\psi }_e^k\right]$

(7)

Missile-node features. For each incoming missile $l\in {𝒱}_{\text{missile}}$, the threat feature vector ${𝐟}_{\text{missile}}^l\in {ℝ}^5$ is defined as:

${𝐟}_{\text{missile}}^l=\left[d_m^l,{\theta }_m^l,v_m^l,\Delta h_m^l,b_m^l\right]$

(8)

媒体内容 · 前往原文查看

Type	Symbol	Description	Type	Symbol	Description
Self	$x,y,z$	Position coordinates	Enemy	$d_e$	Relative distance
	$v$	Velocity magnitude		${\theta }_e$	Relative azimuth angle
	$\theta$	Pitch angle		$\Delta v_e$	Velocity difference
	$\varphi$	Roll angle		$v_{c,e}$	Closing speed
	$\psi$	Yaw angle		${\varphi }_{\text{AA}}$	Aspect angle
	$n_m$	Remaining missiles		$\Delta h_e$	Altitude difference
	${\delta }_{\text{det}}$	Detected by enemy (binary)		$\Delta {\psi }_e$	Heading deviation
	${\sigma }_m$	Last missile status	Ally	$d_a$	Slant distance
Missile	$d_m$	Relative distance		${\theta }_a$	Relative azimuth angle
	${\theta }_m$	Relative azimuth angle		$\Delta v_a$	Airspeed difference
	$v_m$	Approaching speed		$\Delta {\psi }_a$	Heading deviation
	$\Delta h_m$	Altitude difference		$v_{c,a}$	Closing speed
	$b_m$	Threat existence flag (binary)		$n_{m,a}$	Ally’s remaining missiles

4.2 Action Space

To reduce exploration costs, we eschew continuous or fine-grained control in favor of a tactical-level discrete action space. The low-level policy outputs a discrete command $a_i^t\in 𝒜$, which an underlying autopilot translates into flight trajectories, decoupling strategic decisions from low-level execution. The action set $𝒜={𝒜}_{\text{maneuver}}\cup {𝒜}_{\text{weapon}}$ comprises 12 commands covering tactical maneuvering (adjustment, pursuit, evasion) and missile engagement, as detailed in Table 2.

媒体内容 · 前往原文查看

Index	Category	Action Description	Index	Category	Action Description
0	Maneuver	Maintain current heading and altitude	6	Maneuver	Dive to decrease altitude
1		Turn left by 30∘ toward target	7		Accelerate to gain closing speed
2		Turn left by 60∘ toward target	8		Decelerate to reduce closing speed
3		Turn right by 30∘ toward target	9		Perform S-shaped lateral oscillation
4		Turn right by 60∘ toward target	10		Execute notch maneuver to evade missile
5		Climb to increase altitude	11	Weapon	Launch missile at the designated target

4.3 Reward Function

To address the sparse reward inherent in air combat, we design a composite reward combining a shared team signal with individual dense shaping terms:

$r_t^i=r_{\text{combat}}+\alpha r_{\text{adv},t}^i+\beta r_{\text{threat},t}^i+r_{\text{bound},t}^i$

(9)

The team-shared combat reward $r_{\text{combat}}$ provides $+R_{\text{win}}$ / $-R_{\text{lose}}$ at episode termination, and $+r_{\text{kill}}$ / $-r_{\text{death}}$ upon individual destruction events. The tactical advantage reward offers dense offensive guidance based on the Antenna Train Angle (ATA) and relative distance:

$r_{\text{adv},t}^i=\left(1-\frac{{\theta }_{\text{ATA}}^i}{\pi }\right)\cdot \exp \left(-\frac{d_e^i}{D_{\max }}\right)$

(10)

where ${\theta }_{\text{ATA}}^i\in \left[0,\pi \right]$ is the angle between agent $i$’s velocity vector and the line-of-sight to its nearest enemy, and $D_{\max }$ is the maximum engagement range. This term is maximized when the agent’s nose aligns with a nearby enemy.

The threat avoidance penalty compels defensive maneuvers when an incoming missile is detected:

$r_{\text{threat},t}^i=-1(b_m^i=1)\cdot \exp (-\frac{d_m^i}{D_{\text{safe}}})$

(11)

where $D_{\text{safe}}$ is the safety distance threshold.

A fixed boundary penalty is imposed when agents depart the predefined operational zone $ℬ$:

$r_{\text{bound},t}^i=-R_{\text{oob}}\cdot 1(p_t^i\notin ℬ)$

(12)

where $p_t^i$ is the position of agent $i$ and $R_{\text{oob}}$ is the out-of-boundary penalty constant. Notably, cooperative behaviors (e.g., focus-fire) are not explicitly rewarded but instead emerge from the synergy of the graph attention module, dynamic role assignment, and the target-priority auxiliary task, avoiding reward over-engineering.

5 Methods

5.1 DRG-MAPPO Framework

The DRG-MAPPO framework (Fig. 2) integrates three core components: (1) a graph-based relational encoder for capturing time-varying topological interactions among entities; (2) a hierarchical policy decoupling high-level tactical role assignment from low-level role-conditioned maneuvers; and (3) a target-priority auxiliary task injecting domain inductive biases to foster cooperation. The framework is optimized end-to-end via the CTDE paradigm, where the centralized critic leverages global states during training to stabilize optimization, while actors execute relying strictly on local observations.

5.2 Graph-Based Entity Relational Modeling

Entity graph construction. Rather than treating the raw observation as a monolithic feature vector, we decompose it into a local entity graph ${𝒢}_i=({𝒱}_i,{ℰ}_i)$ for each agent $i$. The node set ${𝒱}_i$ consists of four entity categories as defined in the observation space (Section 4.1): self, ally, enemy, and missile. Each node $k\in {𝒱}_i$ is characterized by its raw feature vector ${𝐱}_k$ extracted from the corresponding observation partition. Since entity types possess heterogeneous feature dimensions, all node features are zero-padded to a uniform dimension $d_{\text{raw}}$ and augmented with a learned type embedding ${𝐞}_{\text{type}\left(k\right)}\in {ℝ}^{d_t}$ that encodes entity category information. The combined features are projected into a shared hidden space via a two-layer MLP, $d_h$ is the hidden dimension and $[;]$ denotes concatenation:

$h_k=\text{MLP}\left(\left[\text{pad}\left(x_k\right);e_{\text{type}\left(k\right)}\right]\right),h_k\in {ℝ}^{d_h}$

(13)

Graph attention encoder. To capture the relational structure among entities, we implement a graph attention mechanism using a scaled dot-product formulation [20]. This approach operates over a fully-connected entity graph and was empirically found to provide more stable training dynamics, particularly for the small-scale graphs characteristic of our environment. Given the initial node representations $H^{\left(0\right)}={\left[h_1^{\left(0\right)},\dots ,h_{\left|{𝒱}_i\right|}^{\left(0\right)}\right]}^{\top }\in {ℝ}^{\left|{𝒱}_i\right|\times d_h}$, the attention-based relational update is computed as:

$Q=H^{\left(0\right)}{𝐖}_Q,K=H^{\left(0\right)}{𝐖}_K,V=H^{\left(0\right)}{𝐖}_V$

(14)

$A=\text{softmax}\left(\frac{Q{K}^{\top }}{\sqrt{d_h}}\right)$

(15)

$\tilde{H}=\text{LayerNorm}\left(H^{\left(0\right)}+AV\right),H^{\left(1\right)}=\text{LayerNorm}\left(\tilde{H}+\text{FFN}\left(\tilde{H}\right)\right)$

(16)

where ${𝐖}_Q,{𝐖}_K,{𝐖}_V\in {ℝ}^{d_h\times d_h}$ are learnable projection matrices, and $\text{FFN}(\cdot )$ denotes a two-layer feed-forward network with ReLU activation. The attention matrix $A\in {ℝ}^{\left|{𝒱}_i\right|\times \left|{𝒱}_i\right|}$ captures pairwise relational importance: entry $A_{jk}$ reflects how much entity $j$ attends to entity $k$. To facilitate multi-level relational reasoning, we derive three levels of representation from the output $H^{\left(L\right)}$ of the final graph attention layer:

$z_{\text{self}}^i=H^{\left(L\right)}\left[0\right],g_{\text{agent}}^i=\frac{1}{\left|{𝒱}_i\right|}\sum _k{H}^{\left(L\right)}\left[k\right],g_{\text{team}}=\frac{1}{N}\sum _{i=1}^N{g}_{\text{agent}}^i$

(17)

where $z_{\text{self}}^i\in {ℝ}^{d_h}$ is the graph-refined self-representation; $g_{\text{agent}}^i\in {ℝ}^{d_h}$ summarizes agent $i$’s local battlefield topology; and $g_{\text{team}}\in {ℝ}^{d_h}$ provides a team-level abstraction shared across agents.

5.3 Hierarchical Policy with Dynamic Role Assignment

Role policy (high-level). The high-level role policy assigns a discrete tactical role $r_i\in \left\{1,\dots ,K\right\}$ to each agent, enabling explicit division of labor. The role policy takes as input the multi-level relational representations and the global context:

$f_{\text{state}}={\text{MLP}}_{\text{state}}\left(s\right),c_{\text{global}}={\text{MLP}}_{\text{fuse}}\left(\left[f_{\text{state}};g_{\text{team}}\right]\right)$

(18)

${\pi }_{\text{role}}(r_i\mid o_i)=\text{softmax}\left({\text{MLP}}_{\text{role}}\left(\left[z_{\text{self}}^i;g_{\text{agent}}^i;c_{\text{global}}\right]\right)\right)$

(19)

The selected role index is mapped to a continuous representation through a learned role embedding table $E_{\text{role}}\in {ℝ}^{K\times d_r}$. Unlike one-hot encodings, the learned embedding allows the framework to capture latent similarities between roles and provides richer gradient signals to downstream policy layers.

$e_{\text{role}}^i=E_{\text{role}}\left[r_i\right],e_{\text{role}}^i\in {ℝ}^{d_r}$

(20)

Temporal commitment mechanism. In air combat, tactical roles correspond to sustained behavioral modes (e.g., maintaining offensive pursuit or providing suppressive cover) rather than frame-by-frame switching. To reflect this and prevent destabilizing role oscillation, we introduce a temporal commitment mechanism that re-samples roles only at fixed intervals:

$r_i^t=\{\begin{array}{cc}r{}_i{}^t\sim \pi {}_{\text{role}}(\cdot \mid o{}_i{}^t)& \text{if }tmod{T}_{\text{role}}=0\\ r_i^{t-1}& \text{otherwise}\end{array}$

(21)

where $T_{\text{role}}$ is the commitment horizon. This mechanism provides three benefits: (i) it enforces behavioral consistency within each commitment window; (ii) it enables teammates to anticipate each other’s sustained intentions, facilitating implicit coordination; and (iii) it reduces the effective decision frequency of the high-level policy, stabilizing training.

Role-conditioned action policy (low-level). Conditioned on the assigned role embedding, the low-level policy selects a discrete action from the tactical action space:

${\pi }_{\theta }(a_i\mid o_i,r_i)=\text{softmax}\left({\text{MLP}}_{\text{actor}}\left(\left[f_{\text{obs}}^i;z_{\text{self}}^i;g_{\text{agent}}^i;c_{\text{global}};e_{\text{role}}^i;{\text{id}}_i\right]\right)\right)$

(22)

where ${\text{id}}_i$ is a one-hot agent identity vector that enables parameter sharing while allowing agent-specific behaviors. The centralized critic estimates the state-value function using global information:

$V_{\psi }(s,r_i)={\text{MLP}}_{\text{critic}}\left(\left[f_{\text{state}};z_{\text{self}}^i;c_{\text{global}};e_{\text{role}}^i;{\text{id}}_i\right]\right)$

(23)

This design adheres to the CTDE principle: the actor (Eq. 22) uses only local features for decentralized execution, while the critic (Eq. 23) accesses the global state $s$ through $f_{\text{state}}$ during centralized training. Both are conditioned on the role embedding, ensuring that the value estimation accounts for the agent’s current tactical responsibility.

5.4 Target-Priority Auxiliary Task

To foster cooperative behaviors like focus-fire without complex reward shaping, we introduce an auxiliary target-prioritization task that provides dense supervisory signals. As shown in Eq. (24), the auxiliary head maps concatenated self-features $z_{\text{self}}^i$, graph-derived features $g_{\text{agent}}^i$, and role embeddings $e_{\text{role}}^i$ to a probability distribution $p_{\text{aux}}^i\in {ℝ}^{N_e}$ over $N_e$ enemy targets:

$p_{\text{aux}}^i=\text{softmax}\left({\text{MLP}}_{\text{aux}}\left(\left[z_{\text{self}}^i;g_{\text{agent}}^i;e_{\text{role}}^i\right]\right)\right)$

(24)

Ground-truth labels $y_{\text{target}}$ are derived from a heuristic priority scoring function:

$\text{score}(i,j)=d_{ij}-{\alpha }_1\cdot L_{\text{own}}^{ij}+{\alpha }_2\cdot L_{\text{enemy}}^{ij}-{\alpha }_3\cdot M_j$

(25)

where $d_{ij}$ is the distance to enemy $j$; $L_{\text{own}}^{ij},L_{\text{enemy}}^{ij}\in \{0,1\}$ denote radar lock statuses; and $M_j$ is the number of friendly missiles already tracking enemy $j$. The enemy with the lowest score is designated as the primary target. Crucially, the auxiliary head shares the graph attention encoder with the main policy and is optimized via cross-entropy (CE) loss: ${ℒ}_{\text{aux}}=\text{CE}(p_{\text{aux}},y_{\text{target}})$.

5.5 Joint Training Objective

The framework is trained end-to-end via the Adam optimizer. The joint objective combines PPO clipped surrogates with auxiliary and entropy terms:

$ℒ={ℒ}_{\text{actor}}+{\lambda }_r{ℒ}_{\text{role}}+{\lambda }_V{ℒ}_{\text{critic}}+\lambda {}_{\text{aux}}{}^{}ℒ_{\text{aux}}^{}-\sum {\lambda }_{H}H\left(\pi \right)$

(26)

where ${ℒ}_{\text{actor}}$ and ${ℒ}_{\text{role}}$ denote clipped losses for action and role policies; ${ℒ}_{\text{critic}}$ and ${ℒ}_{\text{aux}}$ represent the value MSE and auxiliary CE loss, respectively. $H\left(\pi \right)$ denotes entropy bonuses for both policy levels. Crucially, role gradients are computed only at re-sampling intervals ($tmod{T}_{\text{role}}=0$), using a distinct clipping coefficient ${ϵ}_r$ to accommodate the lower decision frequency of role assignments without undermining temporal commitment stability.

6 Experiments and Results

6.1 Simulation Setting

Our 2v2 Beyond-Visual-Range (BVR) engagement is modeled within a 200 km $\times$ 100 km operational theater. The scenario initializes with Blue and Red flights (two aircraft each) deployed at opposing southern and northern boundaries. Every combatant is equipped with four air-to-air missiles and maintains a standard combat profile: an altitude of 3 km and an initial cruise speed of 180 m/s. The simulation environment, developed in C++ and visualized via GTacview, enforces strict termination criteria: an episode ends upon total team elimination, boundary breach, or exceeding the maximum time limit. Victory is determined by numerical superiority at termination, while identical survival counts result in a draw. The maximum time limit is set to 800 seconds to ensure sufficient duration for tactical engagement and missile fly-out.

6.2 Main Results and Performance Analysis

媒体内容 · 前往原文查看

媒体内容 · 前往原文查看

To evaluate the effectiveness of DRG-MAPPO, we present the training curves of all methods over 2500 episodes in Fig. 3.

Win Rate. During the initial exploration phase (0–800 episodes), all methods exhibit a similar cold-start bottleneck, reflecting the inherent difficulty of learning complex 2v2 coordination from high-dimensional observations. After approximately 1250 episodes, DRG-MAPPO begins to significantly outperform the baselines, ultimately achieving a peak win rate of 87%. This performance gap underscores the synergy between relational modeling and hierarchical role assignment. While MAPPO+GAT and HAPPO yield moderate improvements by incorporating graph structures or hierarchical decomposition alone, they fall short of the integrated DRG-MAPPO architecture. In contrast, standard MAPPO lacks the structural inductive biases necessary for complex entity reasoning. Furthermore, the inferior performance of QMIX and IPPO highlights that value decomposition and independent learning are insufficient for the tight tactical coordination required in adversarial BVR engagements. This demonstrates that decoupling macro-level tactical role assignment from micro-level decision-making is vital for overcoming the exploration bottleneck and achieving robust multi-agent coordination in air combat.

Average Reward. The evolution of the average episodic reward, illustrated in Fig. 3(b), further corroborates the performance gains. During the initial exploration phase, all agents receive similar negative rewards (approximately $-450$), primarily due to frequent boundary breaches and missile attrition. As basic engagement behaviors emerge, the curves surpass the zero-threshold around episode 400. DRG-MAPPO ultimately converges to the highest asymptotic reward ($\phantom{}\sim 950$), outperforming the mid-tier methods, which plateau between 500 and 650. Consistent with findings in similar BVR benchmarks, standard MAPPO often struggles with exploration efficiency in discrete spaces, while independent learners like IPPO exhibit the lowest rewards due to severe non-stationarity.

6.3 Ablation Experiments

To quantify the individual contribution of each component, we evaluate several ablation variants of DRG-MAPPO. As illustrated in Fig. 4(a), the superior performance and tactical efficiency of DRG-MAPPO stem from the synergy of four core modules: (1) Graph-Based Modeling facilitates structured relational reasoning; (2) Dynamic Role Assignment fosters emergent division of labor; (3) Target-priority Auxiliary Task effectively shapes entity representations toward combat-critical features; and (4) Temporal Commitment prevents high-frequency role oscillations that would otherwise lead to tactical inconsistency and severe performance fluctuations.

媒体内容 · 前往原文查看

6.4 Confrontation Experiments

To further evaluate the generalization and robustness of the learned policies, we conduct a round-robin cross-play tournament among all models. Each pair of policies competes for 200 independent episodes; the resulting win-rate matrix is visualized in Fig. 4(b). As shown in the matrix, DRG-MAPPO achieves a superior average win rate of 69.9%, consistently outperforming all baselines in head-to-head matchups. This performance advantage demonstrates that our role-based coordination strategy generalizes effectively to diverse adversarial behaviors instead of overfitting to the specific patterns of a single training opponent. Furthermore, the minor deviations from 0.5 observed on the diagonal are attributed to the intrinsic stochasticity in the simulation environment.

6.5 Tactical Behavior Visualization

To qualitatively validate the interpretability and coordination of DRG-MAPPO, we visualize a 2v2 engagement episode. The trained DRG-MAPPO model controls the blue-side agents, while the red side employs a rule-based policy.

As shown in Fig. 5(a), asymmetric role assignment fosters an emergent bait-and-flank tactic. The Supporter deliberately maintains a forward trajectory to draw enemy fire, creating a tactical window for the Leader to execute a lateral flanking maneuver and establish a favorable off-axis attack geometry. Furthermore, Fig. 5(b) illustrates a coordinated focus-fire behavior. After evading incoming threats, both agents converge on a single adversary guided by the target-priority auxiliary task, launching missiles from complementary angles to maximize the kill probability. Notably, as the threat diminishes, the Supporter dynamically transitions from a defensive to an offensive posture and successfully strikes the target (shown in Fig. 5(c)), highlighting the temporal adaptability of our role commitment mechanism.

7 Conclusion

In this paper, we presented DRG-MAPPO, a hierarchical MARL framework designed to address coordination challenges in multi-UAV air combat. By integrating a graph-based relational encoder with a dynamic role assignment mechanism, our approach effectively decouples high-level tactical intent from low-level maneuver execution. Extensive simulations in a high-fidelity BVR environment demonstrate that DRG-MAPPO achieves a superior win rate of 87%. Furthermore, the emergence of advanced collaborative tactics, such as bait-and-flank and focus-fire, underscores the effectiveness of our framework in facilitating complex multi-agent decision-making. Future work will explore the scalability of this hierarchical paradigm in larger-scale swarm confrontations.

Acknowledgements

This research received no external funding. The authors express gratitude to the anonymous reviewers for their insightful comments.

Disclosure of Interests.

The authors have no competing interests to declare that are relevant to the content of this article.

References

• [1] Z. Ding, X. Wang, C. Cai, L. Jia, and Z. Xu (2025) Multi-uav intelligent decision-making method with layer delay dual-center mappo for air combat. Applied Intelligence 55 (11), pp. 811. Cited by: §1, §2.1.
• [2] H. Duan, P. Li, and Y. Yu (2015) A predator-prey particle swarm optimization approach to multiple ucav air combat modeled by dynamic game theory. IEEE/CAA Journal of Automatica Sinica 2 (1), pp. 11–18. Cited by: §1.
• [3] K. Hu, H. Pan, C. Han, J. Sun, D. An, and S. Li (2025) Graph neural network-enhanced multi-agent reinforcement learning for intelligent uav confrontation. Aerospace 12 (8), pp. 687. Cited by: §2.3.
• [4] S. Jiang, X. Yang, and L. Delin (2018) Cooperative combat decision-making research for multi uavs. Information and Control 47 (3), pp. 347–354. Cited by: §1.
• [5] X. Jing, X. Yao, M. Liu, and J. Zhou (2024) Multi-agent reinforcement learning based on graph convolutional network for flexible job shop scheduling. Journal of Intelligent Manufacturing 35 (1), pp. 75–93. Cited by: §2.3.
• [6] C. Kim, C. Ji, and B. S. Kim (2020) Development of a control law to improve the handling qualities for short-range air-to-air combat maneuvers. Advances in Mechanical Engineering 12 (7), pp. 1687814020936790. Cited by: §1.
• [7] W. Kong, D. Zhou, Y. Du, Y. Zhou, and Y. Zhao (2023) Hierarchical multi-agent reinforcement learning for multi-aircraft close-range air combat. IET Control Theory & Applications 17 (13), pp. 1840–1862. Cited by: §1.
• [8] W. Kong, D. Zhou, Y. Zhou, and Y. Zhao (2023) Hierarchical reinforcement learning from competitive self-play for dual-aircraft formation air combat. Journal of Computational Design and Engineering 10 (2), pp. 830–859. Cited by: §2.2.
• [9] C. Li, J. Liu, and Y. Gao (2026) Evolutionary enhanced multi-agent reinforcement learning for cooperative air combat. arXiv preprint arXiv:2605.25091. Cited by: §1.
• [10] S. Li, M. Chen, Y. Wang, and Q. Wu (2022) Air combat decision-making of multiple ucavs based on constraint strategy games. Defence Technology 18 (3), pp. 368–383. Cited by: §1.
• [11] J. Liu, J. Chen, Z. Song, S. Zhou, C. Lv, H. Wu, K. Jiang, J. Wu, B. Yu, and C. Zhou (2026) From proprietary to open-source: bridging the distribution gap via multi-agent protocol distillation in agentic search. arXiv preprint arXiv:2607.24280. Cited by: §1.
• [12] Y. Liu, D. Ding, M. Tan, Y. Luo, N. Li, and H. Zhou (2025) Tactical coordination-based decision making for unmanned combat aerial vehicles maneuvering in within-visual-range air combat. Aerospace 12 (3), pp. 193. Cited by: §1.
• [13] R. Lowe, Y. I. Wu, A. Tamar, J. Harb, O. Pieter Abbeel, and I. Mordatch (2017) Multi-agent actor-critic for mixed cooperative-competitive environments. Advances in neural information processing systems 30. Cited by: §2.1.
• [14] C. Lv, Y. Wei, J. Liu, Y. Gao, M. Liu, X. Wang, J. Wu, G. Wang, and C. Li (2026) PCSD: persistent consistency for self-distillation in agentic reinforcement learning. arXiv preprint arXiv:2608.01837. Cited by: §2.2.
• [15] W. Ruan, H. Duan, and Y. Deng (2022) Autonomous maneuver decisions via transfer learning pigeon-inspired optimization for ucavs in dogfight engagements. IEEE/CAA Journal of Automatica Sinica 9 (9), pp. 1639–1657. Cited by: §1.
• [16] J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel (2016) High-dimensional continuous control using generalized advantage estimation. In International Conference on Learning Representations (ICLR), Cited by: §3.2.
• [17] A. Selmonaj, O. Szehr, G. Del Rio, A. Antonucci, A. Schneider, and M. Rüegsegger (2023) Hierarchical multi-agent reinforcement learning for air combat maneuvering. In 2023 International Conference on Machine Learning and Applications (ICMLA), pp. 1031–1038. Cited by: §2.2.
• [18] H. Shin, J. Lee, H. Kim, and D. H. Shim (2018) An autonomous aerial combat framework for two-on-two engagements based on basic fighter maneuvers. Aerospace Science and Technology 72, pp. 305–315. Cited by: §1.
• [19] Z. Sun, H. Piao, Z. Yang, Y. Zhao, G. Zhan, D. Zhou, G. Meng, H. Chen, X. Chen, B. Qu, et al. (2021) Multi-agent hierarchical policy gradient for air combat tactics emergence via self-play. Engineering Applications of Artificial Intelligence 98, pp. 104112. Cited by: §1.
• [20] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin (2017) Attention is all you need. In Advances in Neural Information Processing Systems, Vol. 30, pp. 5998–6008. Cited by: §5.2.
• [21] X. Wang, Y. Wang, X. Su, L. Wang, C. Lu, H. Peng, and J. Liu (2024) Deep reinforcement learning-based air combat maneuver decision-making: literature review, implementation tutorial and future direction.. Artificial Intelligence Review 57 (1), pp. 1. Cited by: §1, §2.1.
• [22] J. Wu, N. Zhang, D. Li, J. Bi, and G. Han (2025) A context-aware feature fusion method for multi-uav cooperative air combat. IEEE Transactions on Intelligent Transportation Systems 26 (5), pp. 7197–7210. Cited by: §1, §2.1.
• [23] X. Wu, J. Liu, X. Liu, X. Zhu, J. Wang, L. Guo, X. Li, X. Cao, and X. Cai (2026) Contrastive reinforced policy optimization via privileged self-distillation. arXiv preprint arXiv:2607.28026. Cited by: §1.
• [24] C. Yu, A. Velu, E. Vinitsky, J. Gao, Y. Wang, A. Bayen, and Y. Wu (2022) The surprising effectiveness of ppo in cooperative multi-agent games. Advances in neural information processing systems 35, pp. 24611–24624. Cited by: §2.1, §3.2.
• [25] M. Zhan, J. Chen, C. Du, and Y. Duan (2021) Twin delayed multi-agent deep deterministic policy gradient. In 2021 IEEE International Conference on Progress in Informatics and Computing (PIC), pp. 48–52. Cited by: §2.1.
• [26] L. Zhu, J. Wang, Y. Wang, and Y. Ji (2022) Research on multi-aircraft cooperative combat based on deep reinforcement learning. In International Conference on Autonomous Unmanned Systems, pp. 1410–1420. Cited by: §1.
• [27] Y. Zihui, X. Liang, H. Yueqi, Y. Aiwu, J. Zhang, and W. Ning (2025) A sample selection mechanism for multi-ucav air combat policy training using multi-agent reinforcement learning. Chinese Journal of Aeronautics 38 (6), pp. 103391. Cited by: §1.