Ye Yuan

Rui Song

Weien Li

Zeyu Li

Haochen Liu

Xiangyu Kong

Changjiang Han

Yonghan Yang

Zichen Zhao

Zixuan Dong

Fuyuan Lyu

Bowei He

Haolun Wu

Jikun Kang

Xue Liu

McGill University,

Mila - Quebec AI Institute,

University of Cambridge,

MBZUAI - Mohamed bin Zayed University of Artificial Intelligence,

University of Toronto,

Salesforce

Corresponding to

ye.yuan3@mail.mcgill.ca

Abstract

Social deduction games have become a popular testbed for probing reasoning, deception, coordination, and belief modeling in Large Language Model (LLM) agents. However, most environments are scored only by game outcomes such as win rates and largely remain to text-only interaction, making it difficult to tell whether an agent’s language is actually grounded in what it perceived and did, or to identify the failure modes underlying its behavior. To address this gap, we introduce QUACK, an open-source environment and evaluation framework for auditing the grounding of agent language in multimodal social reasoning. QUACK evaluates agents at three levels: game outcomes, behavioral trajectories, and utterance-level consistency. Its core Statement Verification Pipeline reconstructs each agent’s ground-truth trajectory from engine logs and checks every discussion claim against it, automatically flagging spatial hallucination, unsupported accusation, deception collapse, and language-action inconsistency. Evaluating three frontier VLMs in both homogeneous and cross-model adversarial settings, we find that even the strongest agent hallucinates % of its verifiable spatial claims and makes over half of its accusations without grounded evidence. We release the full engine, evaluation framework, toolkit, and logs at https://github.com/AAAAA-Academia-Attractions/QUACK.

QUACK: Questioning, Understanding, and Auditing Communicated Knowledge in Multimodal Social Deduction Agents

Ye Yuan^{1, 2  ††}thanks: Corresponding to ye.yuan3@mail.mcgill.ca., Rui Song¹, Weien Li¹, Zeyu Li¹, Haochen Liu³, Xiangyu Kong^{1, 2}, Changjiang Han⁴, Yonghan Yang⁴, Zichen Zhao⁴, Zixuan Dong⁵, Fuyuan Lyu^{1, 2}, Bowei He⁴, Haolun Wu^{1, 2}, Jikun Kang⁶, Xue Liu^{4, 1, 2 1} McGill University, ² Mila - Quebec AI Institute, ³ University of Cambridge, ⁴ MBZUAI - Mohamed bin Zayed University of Artificial Intelligence, ⁵ University of Toronto, ⁶ Salesforce

1 Introduction

Large Language Models (LLMs) and Vision-Language Models (VLMs) are increasingly deployed as interactive agents that must perceive their environment, communicate with other agents, decide under uncertainty, and explain their behavior in natural language (Zhu et al., 2025; Yuan et al., 2026). In such settings, an agent’s language is only useful if it stays grounded: its statements about where it has been, who it has seen, and what it has done must remain faithful to its actual perception and actions (Koh et al., 2024). This shifts the central question beyond static question answering or single-turn instruction following toward whether an agent can maintain grounding over long horizons (Curvo, 2025; Barkur et al., 2025; Jones and Bergen, 2024; Banerjee et al., 2024). In a social deduction game, players hold hidden roles and must infer the hidden roles of others from their behavior and claims. It has therefore become a natural testbed for studying reasoning, deception, coordination, and belief modeling in multi-agent settings (Hu et al., 2025; Chi et al., 2024; Fu, 2025). Compared with traditional static benchmarks, social deduction environments combine hidden information, adversarial incentives, cooperation, strategic communication, and long-horizon interaction (Yu et al., 2025; Sarkar et al., 2025). Crucially, they also admit a recoverable ground truth against which an agent’s every utterance can, in principle, be checked.

Yet existing social deduction environments for LLM agents still face two limitations that make it hard to be directly evaluated. First, most prior work evaluates agents primarily through game outcomes such as win rates, survival rates, or voting accuracy (Light et al., 2023; Wang et al., 2023). These metrics reveal little about why an agent succeeded or failed: an agent may lose despite locally coherent reasoning, or win despite producing inconsistent or unsupported claims. Second, even works that move beyond outcome-level evaluation (Song et al., 2025) remain largely text-only (Shindo et al., 2026; Xu et al., 2024a; Song et al., 2025; O’Gara, 2023). Without grounded visual observations and reconstructable trajectories, it is difficult to determine whether an agent’s dialogue is consistent with what it actually perceived and did, and thus to distinguish correct reasoning from hallucinated evidence or merely plausible dialogue patterns. As a result, important reasoning failures remain hard to identify systematically.

To address this gap, we introduce QUACK, an open-source environment and evaluation framework for auditing grounded multimodal social reasoning in Vision-Language Model agents. QUACK is inspired by social deduction games such as Goose Goose Duck and recent works that leverage Among Us (Chi et al., 2024), but is purpose-built as a controlled research environment for grounded agent evaluation. Agents navigate configurable graph-based maps under partial observability, observe rendered global and local views, complete location-bound tasks, communicate through free-form discussion, and vote under hidden-role adversarial incentives. Critically, every episode is replayable through structured engine-level event logs, yielding a tick-by-tick ground-truth trajectory for each agent against which its statements can be verified.

Beyond the environment, the central contribution of QUACK is a Statement Verification Pipeline that turns this ground-truth trajectory into an automatic audit of agent language. It is embedded in a three-tier evaluation framework that measures game outcomes (Tier ), behavioral trajectories (Tier ), and utterance-level consistency (Tier ). While Tiers  and  provide standard outcome and behavioral context, the pipeline at Tier  reconstructs each agent’s trajectory from engine logs, extracts the structured claims embedded in its discussion utterances, and verifies each claim against the reconstructed world state. This operationalizes four grounding failures as concrete, automatically measurable quantities: spatial hallucination, unsupported accusation, deception collapse, and language-action inconsistency. Because the audit is fully automatic, we validate it against human annotation, confirming that the reported failure rates reflect agent behavior rather than verification noise.

Using QUACK, we evaluate frontier VLM-powered agents across games in both homogeneous and cross-model adversarial settings. Our experiments show that even strong VLM agents exhibit systematic and diagnosable failures when social reasoning must remain grounded in partially observed multimodal interaction: all three frontier models hallucinate a substantial fraction of their spatial claims and make the majority of their accusations without grounded evidence.

• We introduce QUACK, an open-source multimodal social deduction environment for auditing grounded reasoning in VLM agents, with partial observability and fully replayable logs.

• We propose a three-tier evaluation framework scoring game outcomes, behavioral trajectories, and utterance-level consistency, moving beyond win rates toward language grounding.

• We develop a Statement Verification Pipeline that checks each discussion utterance against the reconstructed ground-truth trajectory, operationalizing four grounding failures and validated against human annotation.

• Across three frontier VLMs, in homogeneous and cross-model adversarial play, we show these failures arise systematically.

2 Related Work

QUACK sits at the intersection of two lines of work. We discuss social deduction games as environments for studying multi-agent language behavior and the evaluation of social agents beyond game outcomes.

Social deduction games as environments.

A large body of work uses Werewolf/Mafia-style games to study deception, persuasion, and strategic communication in LLMs, ranging from empirical studies of prompting (Xu et al., 2024a) and reasoning enhancement (Wu et al., 2024b) to dedicated evaluation arenas (Bailis et al., 2024; Shibata et al., 2023) and text-based deception games (O’Gara, 2023). A parallel line targets hidden-role deduction in Avalon, emphasizing recursive reasoning and resistance to deception (Light et al., 2023; Wang et al., 2023), while the impostor-identification setting closest to ours is explored in text-based Among Us variants (Chi et al., 2024; Fu, 2025). Beyond prompting, some works move from playing to training, using reinforcement learning to acquire strategic play and communication (Xu et al., 2024b; Sarkar et al., 2025), and others embed deduction in broader trust-and-deception or social simulations (Curvo, 2025; Park et al., 2023). Almost all of these environments, however, are text-only: agents read and write natural language. The main multimodal resource, Werewolf Among Us (Lai et al., 2023), is an observational corpus of human gameplay for modeling persuasion, rather than an interactive environment in which a vision-language agent must perceive, act, and then justify its claims. QUACK fills this gap by coupling a playable, partially observed multimodal environment with reconstructable ground-truth trajectories.

Evaluating social agents.

Most social deduction benchmarks score agents by game outcomes such as win, survival, or voting accuracy (Light et al., 2023; Wang et al., 2023; Chi et al., 2024; Fu, 2025), which reveal little about why an agent succeeds or fails. Recent work pushes beyond outcomes toward strategy quality and human alignment (Song et al., 2025), explicit opponent and belief modeling (Yu et al., 2025; Premack and Woodruff, 1978), and collaboration-competition metrics in multi-agent settings (Zhu et al., 2025; Sarkar et al., 2025), while a related thread isolates deception itself, studying lie detection (Banerjee et al., 2024) and persuasion (Jones and Bergen, 2024). Multimodal evaluation, in contrast, is largely confined to static or single-agent tasks; visual question answering (Goyal et al., 2017), chart and document understanding (Masry et al., 2022), broad multimodal benchmarks (Liu et al., 2023; Yue et al., 2024), spatial reasoning (Chen et al., 2024), and navigation or web tasks (Anderson et al., 2018; Koh et al., 2024), where there is no adversarial multi-agent dialogue to keep grounded. Methodologically, our verification procedure connects to work on faithfulness and factual consistency in text generation (Ji et al., 2023), which decomposes an output into atomic claims and checks each against an external knowledge source (Thorne et al., 2018; Min et al., 2023), or retrieves evidence to attribute and revise unsupported content (Gao et al., 2023). Unlike these settings, QUACK verifies each claim against a recoverable, agent-specific ground-truth trajectory produced by an interactive, adversarial multi-agent environment. What none of these settings provide is an utterance-level check of whether an agent’s generated claims are faithful to its own perceived-and-acted trajectory. QUACK’s Statement Verification Pipeline supplies exactly this: it reconstructs each agent’s trajectory and verifies every discussion claim against it, turning grounding failures into directly measurable quantities rather than inferring them from final outcomes.

3 The QUACK Environment

We formalize QUACK as a partially observable Markov game (Littman, 1994) played by agents on a graph-structured map. This section defines the teams and roles (§3.1), the map and state space (§3.2), the multimodal observation space (§3.3), the agent (§3.4), the action space (§3.5), and the phase-structured transition dynamics and win conditions (§3.6). We discuss the formulation here and defer the exact agent prompts to Appendix A; full configuration values are released with the code.

3.1 Agents, Teams, and Roles

A game instance has agents partitioned into two hidden-role teams, the Geese (crew) and the Ducks (impostors). At game start, of the agents are sampled uniformly at random to be Ducks and the remaining are Geese. Each agent is privately told its own role, and Ducks are additionally told the identities of their fellow Ducks, whereas Geese know only the team sizes. Our experiments use the standard configuration , , but our environment inherently allows other configurations with different values of and .

Geese.

Each Goose is assigned a private set of location-bound tasks ( in our experiments), each anchored to a specific room. A Goose wins by either collectively completing all Goose tasks or by identifying and ejecting all Ducks through discussion and voting. Geese cannot kill.

Ducks.

Ducks win when the number of living Ducks is at least the number of living Geese (voting parity). A Duck may eliminate a co-located Goose (§3.5), subject to a cooldown, and is issued a set of fake tasks identical in form to a Goose’s so that its task-like behavior is indistinguishable from a Goose’s at the level of observable actions. Ducks must blend in during free roam and avoid suspicion during meetings. The environment advances in discrete time steps, which we call ticks. We set the cooldown as ticks by default.

3.2 Map and State Space

Map.

The environment is parameterized by a map , an undirected weighted graph whose nodes are rooms and whose edges are corridors. The weight is the number of ticks required to traverse the corridor between adjacent rooms and . Figure 1 left demonstrates an omniscient view of the game state. A subset of rooms carry tasks, and one designated room holds the emergency button, which can be used to call a meeting (elaborated later in §3.5). Our environment supports configurable maps, and the instance used in our experiments is a -room map with weighted corridors with travel times ticks.

State.

The global state at tick is

(1)

where is the current game phase (elaborated later in §3.6), is the set of bodies currently on the map (each a tuple of victim, room, and time of death), and collects per-tick communication and witnessed-movement buffers. Each agent’s individual state records its current room, whether it is in transit along a corridor, its task progress vector, its set of visited rooms, and, for Ducks, the remaining kill cooldown. The full state is serialized to a structured engine-level event log at every tick, enabling exact replay and trajectory reconstruction.

3.3 Observation Space

QUACK is partially observable: an agent never sees the global state. At each decision point agent receives a multimodal observation consisting of two rendered images and a structured textual summary.

Rendered views.

The global map image shows the full room layout for spatial orientation but reveals no other players, only the viewer’s own position and its own task markers. The local view image renders only what the agent can presently perceive: the players and bodies in its current room, together with movement events it witnesses this tick (players departing its room or arriving into it). Figure 1 top right illustrates the local view of each agent corresponding to the left omniscient view.

Structured summary.

The text symbolically encodes the agent’s perceptual state, including information the static images cannot convey: its transit status and destination, the movement events it witnesses this tick (which players departed its room or arrived into it, and in which direction), and the adjacent rooms together with their per-corridor travel costs . It also lists the agent’s own tasks and progress, any proximity chat spoken in the room this tick, and, for Ducks, the remaining kill cooldown. Figure 1 bottom right shows an example of the structured summary from Alice’s perspective. During meetings the observation is augmented with the meeting reason, the speaking order, the discussion transcript so far, and the list of known-dead players.

3.4 Agents

Each agent is an VLM-based policy that maps observations to actions and utterances. Because the game is long-horizon and partially observed, an agent cannot rely on a single observation: at every decision point it is conditioned not only on the current observation but also on a running memory of its own trajectory so far: the sequence of rooms it has occupied, the movements it has witnessed (which players it saw depart or arrive), the players it has encountered, and the transcripts and outcomes of previous meetings. During free roam the agent receives together with this memory and selects an action (and optional utterance); during meetings it additionally conditions on the running discussion transcript before producing its statement and vote. This design means an agent’s discussion claims are generated from its own accumulated, partial recollection of the game.

3.5 Action Space

The available actions depend on the phase, the agent’s role, and its local situation. The engine exposes the legal action set with each observation.

Free-roam actions.

During free roam an agent selects one action per tick from: ; to an adjacent room , which initiates a traversal lasting ticks; , which advances the task anchored to the current room by one tick (a task completes after a fixed number of consecutive ticks in its room); , available when a body is present in the agent’s room; and , available only in the emergency-button room while a shared meeting budget remains. A Duck whose cooldown has elapsed additionally has for each co-located Goose . Orthogonally to the chosen action, an agent may attach a free-form utterance , which is heard only by agents in the same room on that tick; this is the local, "proximity chat" channel.

Meeting actions.

When a meeting is convened, free roam halts and the action space switches to language. In the discussion phase each living agent speaks in turn over a fixed number of rounds, producing a free-form natural language utterance. In the subsequent voting phase each living agent casts a vote for a player to eject or abstains.

3.6 Transition Dynamics and Win Conditions

A game proceeds as an alternation between a free-roam phase and an event-triggered meeting phase, formalized as transitions over the phase variable .

Free roam.

On each free-roam tick the engine first advances all in-transit agents (decrementing remaining travel ticks and committing arrivals), decrements Duck cooldowns, and then queries living agents in a randomized order; each chosen action is applied immediately to the state, so an agent’s action can depend on movements already resolved this tick. Movement, kills, task progress, and proximity chat all mutate the state and emit corresponding events. The phase remains FreeRoam until a body is reported or an emergency meeting is called, or until a tick budget is exhausted.

Meeting.

A or action transitions the game to Discussion: all in-transit movement is cancelled, a speaking order is fixed (the caller first, the remaining living agents shuffled), and agents speak for a fixed number of rounds. The game then enters Voting; votes are tallied and the plurality target is ejected, with ties or a plurality-abstain resulting in no ejection (Ejection). If the game is not over, surviving agents are randomly redistributed across rooms and bodies are cleared, returning the game to FreeRoam. This respawn is logged explicitly so it can be reconstructed in replay.

Win conditions.

After every phase the engine checks termination. The Ducks win immediately if living Ducks reach parity with living Geese. The Geese win if all Ducks are ejected, if all Goose tasks are completed, or if the tick budget is reached with at least one Goose alive. On termination the phase becomes GameOver and the outcome and reason are recorded.

4 Automated Evaluation Framework

A central limitation of prior social-deduction benchmarks is that they score agents almost entirely by game outcomes, which reveal little about why an agent succeeded or failed as we discussed in §2. QUACK instead evaluates agents at three complementary levels, all computed automatically from the engine-level event log of each game: Tier  measures game outcomes, Tier  measures behavioral trajectories, and Tier  audits the groundedness of what agents say. Tiers  and  provide standard outcome and behavioral context; our core contribution is the Tier  Statement Verification Pipeline, which reconstructs each agent’s ground-truth trajectory and checks every claim it makes during discussion against that trajectory. We summarize the metrics at each tier in Appendix B.

4.1 Tier 1: Game Outcomes

Tier  records the standard outcome and summary statistics of a game directly from engine events: the winner and win condition, game length, task completion, kill and meeting counts, and survival. It also includes ejection accuracy, the fraction of ejections that removed an actual Duck, which serves as a coarse measure of collective deduction quality. These metrics situate a game but, by design, say nothing about the reasoning behind it.

4.2 Tier 2: Behavioral Trajectories

Tier  reconstructs each agent’s spatial trajectory from the event log and derives behavioral statistics that outcome metrics miss. For Geese, these include voting accuracy and skip rate, task efficiency (task progress relative to the movement undertaken), spatial coverage, and the latency between a body being killed and being reported. For Ducks, they include kill rate, cooldown utilization, the rate at which a Duck reports its own victim (self-report), and post-kill displacement (the distance a Duck travels away from its kill before the next meeting). Together, Tiers  and  characterize what agents did; they do not test whether what agents said is consistent with it.

4.3 Tier 3: Statement Verification

The core of our framework verifies, at the level of individual utterances, whether an agent’s discussion statements are grounded in what it actually perceived and did. The pipeline has two stages: claim extraction and claim verification against the reconstructed world state.

Claim extraction.

Each free-form discussion utterance is parsed by an LLM (GPT-5.5 in our experiment) into a set of structured, individually checkable claims (Pai et al., 2024; Wu et al., 2024a). We define five claim types: (1) location: the speaker asserts that a player was in a room, or, for an ordered multi-room path, a route, (2) sighting: the speaker saw another player in a room, (3) activity: a player was doing a task, traveling, or waiting in a room, (4) accusation: the speaker suspects another player of being a Duck, and (5) defense: the speaker vouches for a player. Each claim carries a subject, the relevant room(s) or target, and a temporal reference. Extraction is run with a fixed prompt (Appendix A), de-duplicated within each utterance, and cached so that re-evaluating a game reproduces the same set.

Claim verification.

Each extracted claim is checked against the agent’s reconstructed ground-truth trajectory for the relevant time window. We recover, tick by tick, every room each agent occupied, including rooms entered only briefly while passing through, and resolve each claim’s temporal reference to a window of ticks before verifying it. Every claim receives one of five verdicts: true, false, wrong_room (the right activity in the wrong place), near_miss (a duration claim, e.g. “I was there the whole time,” that is only briefly true), or unverifiable (no ground truth resolves the claim). Location and route claims are verified by presence/ordered occupancy in the window; Sighting claims by mutual visibility; Activity claims by the logged task and movement events in the claimed room and window; and Accusation claims along two orthogonal axes (detailed in the following). Each verdict is stored with its supporting evidence, so every judgment is auditable.

媒体内容 · 前往原文查看

Setting		Tier 1		Tier 2			Tier 3: Goose (crew)				Tier 3: Duck (impostor)
Goose	Duck	Goose win	Eject. acc.	Vote acc.	Cooldn. util.	Self- rep.	Goose truth.	Spat. hall.	Unsup. acc.	Lie det.	Duck truth.	Dec. rate	Dec. soph.
Homogeneous
Claude-Opus-4.7	Claude-Opus-4.7	90.0	75.0	76.3	66.7	8.3	72.3	10.2	57.8	46.7	38.8	11.2	0.0
Gemini-3.1-Pro	Gemini-3.1-Pro	66.7	61.7	67.8	69.8	1.7	81.6	15.5	47.9	83.3	61.8	27.3	3.1
GPT-5.5	GPT-5.5	76.7	66.7	69.7	45.0	2.8	74.9	12.4	52.6	73.3	62.4	20.5	1.7
Cross-model (adversarial)
Claude-Opus-4.7	Gemini-3.1-Pro	63.3	51.7	53.3	73.3	6.7	52.0	11.7	45.3	46.7	35.3	20.0	1.6
Claude-Opus-4.7	GPT-5.5	70.0	68.3	72.2	65.0	8.3	84.4	12.6	56.6	81.7	74.3	22.4	0.0
Gemini-3.1-Pro	Claude-Opus-4.7	76.7	68.3	69.8	62.2	5.6	78.9	20.8	51.8	80.0	74.8	24.6	1.6
Gemini-3.1-Pro	GPT-5.5	70.0	70.0	75.6	74.3	3.3	84.5	15.5	58.9	77.8	76.2	23.2	2.3
GPT-5.5	Claude-Opus-4.7	93.3	93.3	95.0	48.3	7.8	82.6	19.1	60.8	100.0	76.4	23.6	0.0
GPT-5.5	Gemini-3.1-Pro	73.3	66.7	74.1	69.2	10.0	80.0	17.6	50.1	87.2	63.5	26.1	1.3
All (270 games)		75.6	69.1	72.7	63.8	6.0	76.8	15.1	53.5	75.2	62.6	22.1	1.3

Validating the pipeline.

Because the audit is automatic, we assess its reliability along two axes. For precision, we draw random claims spanning all five types and have a human check, for each, both that the claim is faithfully extracted from the utterance and that its verdict is correct against the ground-truth trajectory; the pipeline is correct on of (). For recall, we draw random utterances and have a human list every claim each contains; the extractor recovers of (). Extraction is thus slightly conservative, occasionally dropping a claim, but the claims it does extract are both parsed and judged reliably, so the failure rates we report reflect agent behavior rather than verification noise.

Operationalizing grounding failures.

The verified claims let us turn four qualitative failure modes into directly measurable quantities: (1) Spatial hallucination: a Goose asserting a location or sighting that contradicts its own trajectory. (2) Unsupported accusation: accusing a player without grounded supporting evidence. We separate two axes that prior work conflates: an accusation’s outcome (did it target an actual Duck, giving accusation accuracy) and its groundedness (could the accuser actually have observed evidence against the target). (3) Deception collapse: a Duck producing easily falsifiable claims rather than subtle ones; we quantify this with the Duck deception rate and a deception sophistication score, the share of a Duck’s false claims that are near-misses rather than outright contradictions. (4) Language-action inconsistency: a stated activity or route that conflicts with the logged actions. Finally, by linking a Duck’s false claims in a meeting to the ejection that follows, we report a lie detection rate: among meetings in which a Duck told a verifiable lie, the fraction after which the Duck was ejected.

5 Experiments

We use QUACK to audit frontier VLM agents, focusing on the question our framework is built to answer: when social reasoning must stay grounded in partially observed multimodal interaction, where and how do VLM agents fail? After describing the setup (§5.1) and overall outcomes (§5.2), we organize our analysis around the four grounding failure modes operationalized in §4 (§5.3).

5.1 Experimental Setup

Models.

We evaluate three frontier vision-language models as agents: GPT-5.5, Gemini-3.1-Pro, and Claude-Opus-4.7. Each agent receives the multimodal observation of §3.3 and acts through the action interface of §3.5; the prompts are identical across models (Appendix A).

Settings.

We run two regimes on the -room map with agents and Duck. In the homogeneous regime all six agents are the same model ( settings). In the cross-model adversarial regime the Geese are one model and the Duck is another, over all ordered model pairs ( settings), isolating how a crew of one model fares against an impostor of another. We play games per setting, using the same set of random seeds for game initialization across settings, for games in total. Table 1 reports all settings; Table 2 aggregates the same set of metrics per model, pooling each model’s crew-side metrics over the settings in which it plays the Geese and its impostor-side metrics over the settings in which it plays the Duck ( games each). Unless noted, we report means over games, so that per-game extraction variance is averaged out at the reporting level.

5.2 Overall Outcomes

At the outcome level the games are well-balanced (Table 1). Complete results across three tiers are available in Appendix C. Across the settings, Geese win – of games and Ducks win –, so the social task is genuinely adversarial rather than trivially crew-favored. Task-related deduction is far from reliable, with ejection accuracy . As Ducks, the three models reach quite different win rates: Claude-Opus-4.7 succeeds as the impostor in only of games versus for Gemini-3.1-Pro (Table 2b).

Crucially, these outcome numbers say little on their own about the quality of an agent’s reasoning: two settings with comparable win rates can differ sharply in how grounded the underlying reasoning is. The clearest example is the strongest crew in our study, GPT-5.5, which wins of games as the Geese yet still hallucinates of its spatial claims and makes of its accusations without grounded evidence (Table 2a), failures that the win rate alone would never reveal. This is exactly the gap Tier  is designed to expose.

媒体内容 · 前往原文查看

(a) As Goose (crew). Pooled over all settings with this model as the crew ( each).
Model	Goose win	Eject. acc.	Vote acc.	Goose truth.	Spatial halluc.	Unsup. accus.	Lie detect.
Claude-Opus-4.7	74.4	65.0	67.3	69.6	11.5	53.2	58.4
Gemini-3.1-Pro	71.1	66.7	71.1	81.7	17.3	52.9	80.4
GPT-5.5	81.1	75.6	79.6	79.2	16.4	54.5	86.8
(b) As Duck (impostor). Pooled over all settings with this model as the Duck ( each).
Model	Duck win	Cooldn. util.	Self- report	Duck truth.	Decep. rate	Decep. soph.
Claude-Opus-4.7	13.3	59.1	7.2	63.3	19.8	0.5
Gemini-3.1-Pro	32.2	70.8	6.1	53.5	24.5	2.0
GPT-5.5	27.8	61.4	4.8	71.0	22.0	1.3

5.3 Grounding Failures (Tier 3)

Across all games, agents tell the verifiable truth most but far from all of the time: pooled Goose truthfulness is (Table 1). The interesting structure is in the failures, which fall cleanly into the four modes our pipeline operationalizes. A consistent theme is that the three frontier models share the same qualitative failure profile (Table 2), differing in degree rather than kind.

Spatial hallucination.

Even though agents are largely truthful, a substantial share of their spatial claims contradict their own trajectories: the pooled spatial hallucination rate is , i.e. roughly one in seven verifiable location/sighting claims is grounded-false. A representative example, a crew member reporting having seen a player who was already dead, is shown in Appendix D. This is the clearest evidence that the difficulty is genuinely long-horizon and partially observed: agents misremember or misreport where they were and whom they saw, the kind of error that outcome metrics cannot detect. The rate tracks model strength: as a crew, Claude-Opus-4.7 hallucinates least while GPT-5.5 and Gemini-3.1-Pro are markedly higher.

Unsupported accusation.

Accusations are both inaccurate and, more tellingly, mostly ungrounded. Appendix D gives an example where a crew member, by its own admission having seen no one, still names a suspect. In a six-player, one-Duck game, accusations land on an actual Duck less than half the time, but the sharper finding comes from separating groundedness from outcome: the pooled unsupported accusation rate is . More than half of all accusations are made without any evidence the accuser could actually have observed, regardless of whether they happen to be correct. Strikingly, this rate is remarkably stable across crews: manufacturing suspicion rather than reasoning from grounded observation is a consistent failure of all three frontier models.

Deception collapse.

On the Duck side, deception is frequent but crude. The pooled Duck deception rate is : roughly a fifth of a Duck’s verifiable claims are outright false. Appendix D shows a Duck fabricating a sighting of a player who was already dead. Critically, deception sophistication is near zero for every model in the Duck seat, meaning these lies are almost never subtle near-misses: they are flatly falsifiable against the ground truth. Ducks fabricate locations and tasks that the engine log directly contradicts, rather than constructing alibis that bend the truth. The most capable agents are thus no more sophisticated as liars. They merely lie at somewhat different rates. This "deception collapse" is precisely why a verification pipeline is informative: the lies exist and are mechanically detectable, even when the Geese fail to act on them.

Language-action inconsistency.

The same pattern appears in activity and route claims, where stated tasks and paths conflict with the logged actions. A recurring instance is a Duck claiming to have performed a task in a room where the log shows it performed none: a faked-task alibi that is internally fluent but inconsistent with what the agent actually did. Appendix D shows an example of this failure mode.

Are the lies caught?

Finally, we connect the surfaced lies back to outcomes. Among meetings in which a Duck told a verifiable lie, the Duck is subsequently ejected only (Table 1) of the time on average, and as low as (Table 2a) when Claude-Opus-4.7 is the crew. Even when a Duck’s statements are mechanically falsifiable against ground truth, Geese frequently fail to convert that into the correct ejection: a gap between the evidence available in principle and the deduction agents actually perform. Together, these results show that strong VLM agents exhibit systematic and diagnosable grounding failures that are invisible to win rates but surfaced by QUACK’s audit.

6 Conclusion and Discussion

We introduced QUACK, an open-source environment and evaluation framework for auditing whether the language of multimodal social-deduction agents stays grounded in what they actually perceived and did. Unlike prior social-deduction benchmarks, which score agents almost entirely by game outcomes, QUACK evaluates at three levels: game outcomes, behavioral trajectories, and utterance-level consistency. Its core Statement Verification Pipeline reconstructs each agent’s ground-truth trajectory from engine logs and checks every discussion claim against it. This turns four qualitative failure modes: spatial hallucination, unsupported accusation, deception collapse, and language-action inconsistency into directly measurable quantities. These failures are largely shared across the three models, differing in degree rather than kind, and several of them worsen under cross-model adversarial pressure. Crucially, none of them is visible from win rates alone: two agents with similar outcomes can differ sharply in how grounded their reasoning is, and only a statement-level audit surfaces the difference.

We see two broader takeaways. First, for social-deduction and multi-agent language settings more generally, groundedness is a distinct axis of capability that is not captured by task success and deserves to be measured directly. Second, social-deduction games are a uniquely convenient instrument for studying grounded generation: they pair strong incentives to make verifiable claims (and to lie) with a fully recoverable world state, a combination rarely available in open-ended language tasks. We hope QUACK serves both as a diagnostic for current agents and as a substrate for future work: for example, training agents whose discussion is explicitly optimized for groundedness, or extending the verification approach to richer environments.

Limitations

Our study has a few limitations that also point to future work. Claim extraction relies on an LLM and is slightly conservative. Although our human validation finds the pipeline both precise (extractions and verdicts correct on sampled claims) and high-recall ( claims; §4.3), the extractor occasionally drops a claim. Missed claims reduce coverage rather than corrupt the verdicts, so our reported rates may slightly undercount the total claims made, but the claims that are scored are both parsed and judged reliably. We do not isolate the contribution of the visual modality. Agents receive aligned image and text observations, and we do not run a text-only ablation; we therefore characterize the difficulty as long-horizon and partially observed rather than attributing it specifically to vision, and leave a controlled comparison to future work. Our experimental scope is bounded. We evaluate three models on a single -room map with a fixed configuration (, ). The one-Duck setting in particular yields fewer impostor-side claims per game, so Duck metrics rest on smaller samples than crew metrics. The environment supports larger maps, more agents, more impostors, and additional roles, and we expect the absolute numbers to shift with these factors even if the qualitative failure modes persist. Finally, verification is defined relative to the engine’s ground truth and our claim taxonomy. Claims that no logged event can resolve are marked unverifiable rather than scored, so the framework audits grounded, checkable statements and does not attempt to judge the full pragmatic content of free-form dialogue.