Jingxuan Wei

Xi Bai

Shan Liu

Caijun Jia

Zheng Sun

Xinglong Xu

Siyuan Li

Linzhuang Sun

Bihui Yu

Conghui He

Cheng Tan

tancheng@pjlab.org.cn

May 15, 2026

Abstract

Large vision-language models have significantly advanced GUI agents, enabling executable interaction across web, mobile, and desktop interfaces. Yet these gains largely rely on a forgiving region-tolerant paradigm, where many nearby pixels inside the same component remain valid. Precise geometric construction breaks this assumption: actions must land on points in continuous canvas space rather than tolerant regions. Because geometric primitives carry ontological dependencies, a local coordinate error can induce cascading topological failures that distort downstream objects and invalidate the final construction. We identify this regime as precision-sensitive GUI tasks, requiring point-level accuracy, geometry-aware verification, and robustness to dependency-driven error propagation. To benchmark it, we introduce PAGE Bench, with 4,906 problems and over 224K process-supervised, pixel-level GUI actions. We further propose PAGER, a topology-aware agent that decomposes construction into dependency-structured planning and pixel-level execution. Pixel-grounded supervised tuning establishes executable action grammar, while precision-aligned reinforcement learning mitigates rollout-induced exposure bias through state-conditioned geometric feedback. Experiments reveal a pronounced Semantic-Execution Gap: general multimodal models can exceed 88% action type accuracy yet remain below 6% task success. PAGER closes this gap, delivering higher task success than the strongest evaluated general baseline and raising step success rate from below 9% for GUI-specialized agents to over 62%, establishing a new state of the art for point-precise GUI control.

Cheng Tan,

Code   Website   Dataset

1 Introduction

Modern GUI agents increasingly turn software interfaces into action spaces for vision-language models. Recent systems operate across web, mobile, desktop, and broader computer-use environments by grounding multimodal instructions to interface elements and composing them into executable workflows [nguyen2025gui, chen2025guicourse, qin2025uitars, liu2026infiguiagent, zhao2025worldgui, yang2026probench, wang2026history]. This progress, however, is built mainly on a region-tolerant interaction paradigm: a button, link, input box, or menu item remains correct under many nearby click locations. The paradigm supports much of today’s GUI automation, but it leaves a basic capability boundary unresolved: can an agent still operate reliably when the target is a point in continuous visual space rather than a tolerant region?

We investigate this boundary through precise geometric construction. Given a geometry problem, the agent must construct points, segments, lines, circles, polygons, labels, and spatial relations on a GUI canvas. As illustrated in Figure 1, this setting is not merely a harder instance of GUI grounding; it changes the success geometry from region membership to point-level accuracy within a small pixel tolerance. More importantly, geometric operations are dependency-coupled: a misplaced point changes every line, circle, intersection, angle, or polygon that depends on it, so local coordinate errors propagate through the construction process like perturbations under a dependency Jacobian.

We call this regime precision-sensitive GUI tasks, where agents must move beyond region-level component selection toward point-precise manipulation. This regime sits at the intersection of GUI agents, geometric reasoning, and reinforcement learning, but none of these lines directly captures it. GUI agents mainly study semantic component grounding and workflow completion [lian2025uiagile, lee2025reguide, zhou2025guig1, liao2025beyondclicking]; geometric reasoning methods focus on diagram understanding, auxiliary construction, or formal validity in symbolic spaces [xu2025geosense, feng2025geobench, weng2025geosketch, wei2025geointr1]; and RL-based agents typically optimize discrete success, milestones, or target regions rather than continuous geometric precision [zhang2025r1vl, shi2025mobileguirl, xu2025mobilerl, lu2025uir1, xia2025guir1]. To make this missing capability measurable, we introduce PAGE Bench, a Precision-Aware GEometric GUI Benchmark for precise geometric construction. PAGE Bench contains 4,906 geometry problems, 53,277 high-level construction tasks, and 224,497 low-level GUI actions, with trajectories that preserve problem statements, ordered sub-tasks, canvas states, execution feedback, and pixel-level geometric annotations. Its evaluation therefore goes beyond final visual similarity, measuring process correctness, parameter precision, and final geometric validity.

We further propose PAGER, a Precision-Aware GEometric Reasoning framework for precision-sensitive GUI tasks. PAGER factorizes drawing into dependency-structured planning and pixel-level execution: the planner induces a construction graph and produces a topologically valid sub-task order, while the executor grounds each sub-task into concrete GUI actions conditioned on the current canvas state. Pixel-grounded supervised tuning first establishes executable action grammar and sequential drawing behavior. Since this imitation stage is teacher-forced, inference still suffers from exposure bias: small deviations move the rollout away from reference canvas states and can be amplified by downstream geometric dependencies. Precision-aligned reinforcement learning then optimizes action-type correctness, parameter accuracy, and rendered geometric validity, directly targeting the point-level bottleneck exposed by precision-sensitive drawing.

Experiments show that this task exposes a structural mismatch in existing agents. Strong general multimodal models often understand the intended operation, but fail to maintain the continuous parameters needed for a valid construction. Ablations further show that pixel-grounded SFT provides the execution prior, parameter-accuracy rewards drive continuous-space control, and combining action-type and parameter rewards yields the strongest task-level performance.

Our main contributions are as follows:

• We identify and formalize precision-sensitive GUI tasks, a class of GUI tasks that require point-level spatial accuracy, continuous-canvas manipulation, geometry-aware verification, and mitigation of cascading coordinate errors.

• We introduce PAGE Bench, to the best of our knowledge the first benchmark for evaluating GUI agents on precise geometric construction, with process-supervised trajectories, pixel-level annotations, and both process-level and final-result metrics.

• We propose PAGER, a dependency-structured planning and pixel-level execution framework trained with pixel-grounded supervised tuning and precision-aligned reinforcement learning. Experiments show that PAGER substantially improves precise geometric GUI execution over general VLMs and GUI-specialized agents.

2 Related Work

GUI Agents

GUI agent research maps multimodal instructions to executable actions across web, mobile, desktop, and broader computer-use environments. Early work builds perceptual-action abstractions: CogAgent [hong2024cogagent] improves high-resolution interface understanding, while CoCo-Agent [ma2024agent] structures mobile action prediction through environment perception and conditional decomposition. Recent systems such as UI-TARS [qin2025uitars] and GUI-Libra [yang2026guilibra] move toward native end-to-end execution with reasoning-aware action modeling. A related thread improves grounding accuracy and data efficiency through continuous-reward optimization, self-evolutionary reinforcement learning, spatial reasoning, test-time search, and difficulty-aware reward correction [lian2025uiagile, yuan2025segui, lee2025reguide, zhou2025guig1]; this trajectory also extends beyond clicking to text dragging [liao2025beyondclicking]. Benchmarking likewise shifts toward realistic and process-aware evaluation, including arbitrary-state desktop automation, broader computer-use, tool-use, and browsing settings [zhao2025worldgui, yang2026probench, mu2025gui360, fan2025mcptoolbenchpp, wei2025browsecomp]. Despite this progress, existing GUI agents mainly target semantic interface elements or tolerant regions, where success depends on component selection or workflow completion. Our work instead studies canvas-based precision-sensitive GUI tasks, where success requires point-level spatial accuracy, geometric validity, and mitigation of cascading error propagation induced by small coordinate deviations.

Geometric Reasoning

Geometric reasoning studies how models interpret diagrams, identify principles, and derive mathematically valid solutions from multimodal inputs. Diagnostic benchmarks analyze failures in principle identification, principle application, perception, planning, theorem use, and reflection [xu2025geosense, feng2025geobench]. Subsequent evaluations broaden the scope beyond plane geometry to 3D settings, larger diagram-based problem spaces, and visually aided mathematical reasoning [wang2025solidgeo, zhang2026geochallenge, ma2024visaidmath]. Another line pursues formalization and reliable data through verified data construction, formal proof systems, and formal-language-driven synthesis [fu2025trustgeogen, he2025matpbench, zhang2025geofm]. More recent methods make diagrams less static by incorporating auxiliary construction, geometric transformation, cross-modal rewards, dense sub-goal supervision, and staged reinforcement learning [weng2025geosketch, guo2025geovlmath, chen2026milestones, wei2025geointr1]. Despite these advances, existing work still mainly operates in symbolic space, where success is defined by recognition, proof, or formal construction. Our work bridges symbolic validity and physical execution by grounding geometric reasoning into pixel-space GUI actions that require logical correctness, point-level spatial accuracy, and mitigation of cascading error propagation.

3 Methodology

3.1 Preliminaries

We study precision-sensitive geometric GUI drawing, where an agent constructs a target figure on a continuous canvas. Given problem context with instruction and target image, the agent starts from canvas state and generates

(1)

where is the drawing environment, is the operation type, is the object type, and denotes typed parameters.

The task differs from region-tolerant GUI interaction in success geometry:

(2)

where is the executed pixel location, is a valid target region, and is a reference point. Geometric drawing follows the point-level criterion and exhibits dependency-coupled error propagation:

(3)

where captures construction dependencies and maps parameter errors to canvas perturbations. Thus, small coordinate deviations can affect downstream objects.

3.2 PAGER: Dependency-Structured Planning and Execution

As shown in Figure 2, PAGER factorizes drawing into planning and execution. The Planning Module induces a construction graph and a dependency-consistent sub-task list:

(4)

where contains primitives or relations, encodes dependencies, and is the transitive closure. The Task Execution Module grounds each sub-task into GUI actions:

(5)

where is the number of actions for , is action history, nested actions flatten to , and instantiates the Step Specification with pixel coordinates, geometric parameters, visual style, and label position.

3.3 Pixel-Grounded Supervised Tuning

Pixel-Precise Data Construction provides trajectories with sub-tasks, screenshots, histories, next actions, execution feedback, and spatial annotations. For visible window , geometric coordinates are projected to pixels by:

(6)

where and are canvas width and height. The same projection binds anchors of points, lines, circles, arcs, polygons, and labels to pixel targets. SFT optimizes:

(7)

where and is the number of reference actions for . SFT learns executable action grammar and state-conditioned action prediction, but teacher forcing uses reference screenshots while inference uses self-generated screenshots. Eq. 3 therefore motivates precision-aware rollout training.

3.4 Precision-Aligned Reinforcement Learning

RL Precision Optimization aligns the policy with action-type correctness, parameter accuracy, and rendered geometric validity. For each problem, the Planning Module produces , and policy-environment interaction induces rollout with length and rendered construction . Each sampled action is scored against:

(8)

where is the reference construction, and , , and are operation, object, and typed parameter spaces. The admissible set is built by a training-time geometric verifier and is not used during inference. The rollout reward is:

				(9)

For , operation-type matching grants and activates the parameter-accuracy term. The distance penalizes object mismatch and typed parameter error, including text consistency for type, region validity for click, and pixel deviation for paint; compares anchors, relations, and layout. The policy is optimized with the SFT policy as a KL anchor:

(10)

The KL term preserves executable behavior, while the reward targets the point-level criterion in Equation 2 and the cascading-error mechanism in Equation 3.

4 Dataset

4.1 Dataset Construction

As shown in Figure 3, PAGE Bench is constructed as a closed execution loop rather than a static collection. The full pipeline converts raw problems into executable construction trajectories in GeoGebra and retains only those instances that remain valid after execution and verification.

Problem collection and executable screening. A candidate pool is first assembled from public K–12 multimodal geometry resources [du2025mm]. Since many raw items support symbolic solving but not GUI-grounded construction, a model-assisted screening module selects problems whose solutions can be realized as ordered constructions in GeoGebra, and manual verification then removes under-specified statements, non-constructive formulations, and cases whose dependencies cannot be operationalized on the canvas. This stage yields construction-ready problems whose solution logic can be grounded in interface actions rather than free-form derivations.

Structured task generation and standardization. For each retained problem, a language-model-based authoring module produces a high-level task sequence represented as an ordered list of function+args operations. A subsequent standardization module parses the generated structures, rectifies malformed task strings, and normalizes the output into a canonical task list together with aligned metadata. The result is a structured intermediate representation that makes the intended geometric dependencies and execution order explicit.

Execution mapping and environment-grounded reconstruction. The standardized task list is next mapped to low-level GUI interactions in a live GeoGebra environment. Each abstract construction step is decomposed into a sequence of tool-category selection, tool selection, and parameterized canvas manipulation, yielding executable click, paint, and type actions. A unified interaction layer converts structured construction intent into browser-level operations, while coordinate normalization, geometry-to-pixel projection, and boundary-aware retry preserve executability under varying browser states and out-of-canvas conditions. In this way, symbolic construction plans are reconstructed as replayable interface trajectories.

Execution recording, post-execution filtering, and final packaging. During execution, the framework records, for each step, the screenshot, present task, previous actions, exe success, exe log, and next action, together with the executed action and its parameters. For click operations, the recorder additionally preserves the target bounding box, hit range, and normalized coordinates, providing the fine-grained spatial evidence required for later precision analysis. After execution, a final language-model-based filtering module compares recorded trajectories against rendered outcomes and removes inconsistent task sequences, failed executions, and geometrically invalid constructions. The retained benchmark therefore provides verified construction trajectories with fine-grained spatial provenance, making it possible to study point-level accuracy and cascading geometric errors in precision-sensitive GUI tasks.

4.2 Dataset Analysis

Figure 4 and Table 1 summarize the composition and process scale of PAGE Bench. PAGE Bench contains 4,906 problems with a 4,443/463 train-test split, including 2,049 multiple-choice and 2,857 open-ended instances. The 58.23% open-ended share emphasizes explicit construction rather than answer selection. Its ten-category multi-label taxonomy yields 25,301 annotations, or 5.16 tags per problem, indicating that most instances combine language-to-tool grounding, object construction, coordinate modeling, relation reasoning, multi-step planning, and auxiliary construction rather than isolated skills. Most problems come from Grades 8–10+, and intermediate or hard cases account for 94.11%, placing the benchmark in a construction-oriented and nontrivial reasoning regime.

媒体内容 · 前往原文查看

Corpus statistics			Process statistics
Statistic	Count		Statistic	Count
Total problems	4,906	100.00%	Total tasks	53,277	–
Train / Test	4,443 / 463	90.56 / 9.44	Avg. tasks / problem	10.86	–
Multiple-choice	2,049	41.77%	Total actions	224,497	–
Open-ended	2,857	58.23%	Avg. actions / problem	45.76	–
Skill categories	10	–	click actions	107,148	47.73%
Category tags	25,301	5.16 / prob.	paint actions	90,486	40.31%
Grade 8–10+	4,387	89.42%	type actions	26,863	11.97%
Intermediate / Hard	2,940 / 1,677	94.11%	click+paint	197,634	88.03%

On the process side, the corpus contains 53,277 high-level tasks and 224,497 GUI actions, averaging 10.86 tasks and 45.76 actions per problem. This trajectory length creates meaningful dependency chains, where early execution errors can influence later objects. Spatial operations dominate: click and paint contribute 88.03% of all actions, with paint directly requiring continuous-canvas control.

5 Experiment

5.1 Experimental Setup

We train PAGER from Qwen3-VL-8B [bai2025qwen3]. During supervised fine-tuning, we update the vision encoder, multimodal projector, and language backbone with a maximum input length of 8,192 tokens, per-device batch size 1, gradient accumulation 4, learning rate , 5% warmup, bfloat16 precision, and DeepSpeed ZeRO-2 over 8 GPUs for 1 epoch. The reinforcement-learning stage follows SFT and uses rejection sampling with 8 candidates per prompt; prompts with high outcome variance are retained to focus optimization on uncertain rollouts. All stages are implemented with torchrun on 8 NVIDIA A100 GPUs. Evaluation metrics are detailed in Appendix F.

We compare against open-source VLMs, including Qwen3-VL-8B [bai2025qwen3], DeepSeek-VL2 [wu2024deepseek], GLM-4.5V [hong2025glm], InternVL2.5-8B [zhu2025internvl3], KimiVL-A3B [team2025kimi], MiniCPM-V-2.6 [yao2024minicpm], and LLaVA-NeXT-8B [liu2024llavanext]; closed-source VLMs, including Claude-Sonnet-4.6 [Anthropic2026ClaudeSonnet4_6], GPT-5.4 [OpenAI2026GPT5_4], Qwen3.6-Plus [qwen3.6-35b-a3b], and Gemini-3.1-Pro [GoogleDeepMind2026Gemini3_1Pro]; and GUI-specialized agents, including UI-TARS [qin2025ui], OS-ATLAS [wu2024atlas], InfiGUI-R1-3B [liu2025infigui], GUI-Actor-7B [shakeel2026medspot], and OpenCUA-7B [wang2025opencua]. This benchmark set covers general multimodal reasoning, proprietary vision-language modeling, and interface-specialized action prediction.

5.2 Main Results

媒体内容 · 前往原文查看

Model	Action	Param	Step	Task	O-Comp.	S-Comp.	S-Vis.	S-Geo.	Middle	Final	Overall
Open-source VLMs
Qwen3-VL-8B [bai2025qwen3]	52.03	9.48	9.38	0.25	3.09	3.83	3.69	3.05	8.18	3.42	5.80
DeepSeek-VL2 [wu2024deepseek]	26.33	1.75	1.53	0.00	18.25	8.46	3.70	12.40	3.11	11.23	7.17
GLM-4.5V [hong2025glm]	78.91	12.17	11.76	0.00	5.44	7.93	2.89	13.41	11.46	7.27	9.37
InternVL2.5-8B [zhu2025internvl3]	40.34	1.48	1.33	0.00	9.23	7.19	1.72	10.85	4.45	7.44	5.94
KimiVL-A3B [team2025kimi]	46.60	0.59	0.57	0.00	1.70	7.12	1.24	14.05	4.83	5.70	5.27
MiniCPM-V-2.6-8B [yao2024minicpm]	45.87	1.06	0.71	0.00	16.01	4.81	1.17	8.18	4.84	8.12	6.48
LLaVA-NeXT-8B [liu2024llavanext]	45.77	1.69	1.59	0.00	8.04	5.13	1.74	8.75	5.06	6.05	5.56
Closed-source VLMs
Claude-Sonnet-4.6 [Anthropic2026ClaudeSonnet4_6]	\cellcolor[rgb].851,.953,.99295.85	36.44	36.38	1.11	7.04	9.46	3.11	15.39	21.17	8.65	14.91
GPT-5.4 [OpenAI2026GPT5_4]	88.04	31.71	31.41	0.56	10.99	9.59	3.53	15.39	18.59	9.96	14.28
Qwen3.6-Plus [qwen3.6-35b-a3b]	\cellcolor[rgb].851,.961,.83990.95	51.60	51.07	4.90	18.19	11.04	4.23	10.52	27.41	11.72	19.56
Gemini-3.1-Pro [GoogleDeepMind2026Gemini3_1Pro]	89.18	\cellcolor[rgb].851,.953,.99266.68	\cellcolor[rgb].851,.953,.99266.66	\cellcolor[rgb].851,.961,.8395.82	\cellcolor[rgb].851,.961,.83920.97	\cellcolor[rgb].851,.961,.83914.17	\cellcolor[rgb].851,.953,.9927.63	\cellcolor[rgb].851,.953,.99221.21	\cellcolor[rgb].851,.961,.83932.41	\cellcolor[rgb].851,.961,.83916.31	\cellcolor[rgb].851,.961,.83924.36
Specialized GUI Agents
UI-TARS [qin2025ui]	47.08	8.79	8.38	0.00	7.06	5.26	3.78	5.21	7.26	5.49	6.38
OS-ATLAS [wu2024atlas]	51.24	9.19	8.80	0.29	14.56	6.65	4.32	6.36	7.98	8.50	8.24
InfiGUI-R1-3B [liu2025infigui]	44.81	16.51	16.18	0.00	18.23	9.66	4.14	13.82	9.37	11.96	10.66
OpenCUA-7B [wang2025opencua]	55.86	10.57	9.85	0.12	15.39	8.92	3.11	10.77	8.69	10.07	9.38
GUI-Actor-7B [shakeel2026medspot]	47.26	5.31	5.02	0.00	18.66	6.04	1.42	7.58	6.26	9.21	7.74
PAGER	82.62	\cellcolor[rgb].851,.961,.83962.76	\cellcolor[rgb].851,.961,.83962.20	\cellcolor[rgb].851,.953,.99223.78	\cellcolor[rgb].851,.953,.99228.88	\cellcolor[rgb].851,.953,.99215.30	\cellcolor[rgb].851,.961,.8397.05	\cellcolor[rgb].851,.961,.83915.63	\cellcolor[rgb].851,.953,.99241.25	\cellcolor[rgb].851,.953,.99217.79	\cellcolor[rgb].851,.953,.99229.52

Table 2 shows that PAGER achieves the best Overall score, 29.52, improving over the strongest general baseline, Gemini-3.1-Pro, by 5.15 points, or 21.1%. It also obtains the highest Task, Middle, and Final scores, indicating stronger complete-rollout execution and better final geometric quality. Notably, these gains occur despite Gemini-3.1-Pro leading in Param and Step, suggesting that PAGER better converts local execution into task-level success. This indicates stronger trajectory-level stability rather than merely better single-step prediction.

The results expose a clear Semantic-Execution Gap. Closed-source VLMs often select the correct operation type: Claude-Sonnet-4.6 reaches 95.85 Action Accuracy, while GPT-5.4 and Gemini-3.1-Pro reach 88.04 and 89.18. However, their Task Success remains 1.11, 0.56, and 5.82, respectively. In contrast, PAGER reaches 23.78 Task Success, about Gemini-3.1-Pro. This shows that precise drawing is not bottlenecked by action semantics alone, but by state-conditioned parameter control and error accumulation across dependent construction steps.

Compared with GUI-specialized agents, PAGER further highlights the limitation of region-tolerant GUI. UI-TARS and OS-ATLAS remain below 9% Step Success, and the strongest GUI-agent baseline reaches only 16.18, whereas PAGER reaches 62.20. This indicates that component-level GUI grounding is too coarse for geometric construction, where exact points, rather than regions, determine validity. These results support the central motivation of this work: precision-sensitive geometric GUI control requires point-level execution, geometry-aware feedback, and robustness to cascading coordinate errors, rather than only component-level grounding.

5.3 Ablation Study

Table 3 analyzes the effect of the proposed components. PAGER-SFT already provides a strong execution prior, reaching 48.47 Parameter Accuracy, 47.91 Step Success, and 20.47 Overall, which confirms the value of pixel-grounded process supervision for learning executable drawing grammar. The reward ablations show that parameter alignment is the key bottleneck. Without the parameter-accuracy reward, the model gains little beyond SFT and even drops from 20.47 to 20.07 Overall, suggesting that action-level correctness alone cannot preserve geometric structure when coordinates, endpoints, radii, or labels drift. Without the action-type reward, the model still improves to 24.52 Overall and 15.90 Task Success, indicating that continuous-space precision is central to valid construction. The full PAGER combines both rewards and achieves the best performance across most metrics, improving Overall from 20.47 to 29.52 and Task Success from 4.48 to 23.78 over SFT. The two rewards are therefore complementary: action-type reward stabilizes semantic execution order, while parameter-accuracy reward improves point-level control. Their combination yields the most reliable alignment between action selection, parameter prediction, and final geometric validity.

媒体内容 · 前往原文查看

Model	Action	Param	Step	Task	O-Comp.	S-Comp.	S-Vis.	S-Geo.	Middle	Final	Overall
PAGER-SFT	75.14	48.47	47.91	4.48	27.70	11.24	5.69	\cellcolor[rgb].851,.953,.99217.48	24.63	16.32	20.47
PAGER o/w RL	74.72	52.88	52.40	5.62	20.11	10.99	4.35	16.64	26.61	13.53	20.07
PAGER o/w RL	78.77	59.14	58.70	15.90	22.67	11.54	4.33	14.22	35.07	13.97	24.52
PAGER	\cellcolor[rgb].851,.953,.99282.62	\cellcolor[rgb].851,.953,.99262.76	\cellcolor[rgb].851,.953,.99262.20	\cellcolor[rgb].851,.953,.99223.78	\cellcolor[rgb].851,.953,.99228.88	\cellcolor[rgb].851,.953,.99215.30	\cellcolor[rgb].851,.953,.9927.05	15.63	\cellcolor[rgb].851,.953,.992 \cellcolor[rgb].851,.953,.99241.25	\cellcolor[rgb].851,.953,.99217.79	\cellcolor[rgb].851,.953,.99229.52

5.4 Case Study and Error Analysis

Figure 5 presents a representative construction task: drawing rectangle with diagonals intersecting at , under and , and determining . The example requires accurate vertex placement, diagonal construction, and preservation of side, angle, and intersection constraints. PAGER constructs a geometrically consistent rectangle with valid side relations and central diagonal intersection, despite minor numerical deviations. GPT-5.4 captures the rough intent but distorts the quadrilateral, introduces redundant elements, and omits a key diagonal. Gemini-3.1-Pro exhibits stronger parameter drift: early vertex errors propagate into invalid long segments and break the rectangular structure. This case illustrates that failures in precise drawing often arise from unstable pixel-level parameters and weak constraint preservation, rather than from complete semantic misunderstanding. It therefore supports the quantitative finding that PAGER’s process supervision and precision-aligned optimization improve execution reliability under strict geometric constraints.

5.5 Consistency with Human Judgments

Figure 6 illustrates a clear spatial separation in model performance. Most existing MLLMs, including GPT-5.4 and Gemini-3.1-Pro, cluster in the lower-left region with both low automated scores and low human ratings. In contrast, PAGER occupies the top-right corner, achieving high automated success alongside superior human preference. The near-perfect correlation () demonstrates a strong alignment between our automated verification and expert judgment. Despite the precision-sensitive nature of the task, improvements in our metrics translate monotonically into human-perceived correctness, confirming that PAGE Bench captures genuine geometric validity rather than proxy signals.

媒体内容 · 前往原文查看

6 Conclusion and Limitations

We introduce precision-sensitive GUI tasks, where GUI success shifts from region-tolerant component selection to point-precise control in continuous visual space. Precise geometric construction exposes this regime because dependency-coupled primitives allow small coordinate errors to cascade into invalid downstream structures. PAGE Bench makes this failure mode measurable with process-supervised, pixel-level geometric interactions, and PAGER addresses it through dependency-structured planning and precision-aligned execution. Experiments reveal a clear Semantic-Execution Gap: existing agents often identify the correct operation but fail to realize it with the spatial precision required by geometry, while PAGER narrows this gap and advances GUI agents toward faithful execution of structured intent in pixel space. This work focuses on GeoGebra-style planar construction, so other precision-sensitive interfaces may require additional action grammars, environment adapters, and validity rules. This controlled setting isolates the core semantic-execution gap and provides verifier-backed supervision, while extending the same principle to broader domains such as CAD, diagram editing, and scientific visualization remains a natural direction.

References

Appendix A Performance Results with Radar Chart Analysis

To analyze model performance on precision-sensitive GUI tasks, we evaluate PAGER against fourteen baselines across three categories, as shown in Figure 7. The evaluation reveals a critical Semantic-Execution Gap in existing architectures. State-of-the-art closed models like Gemini-3.1-Pro and GPT-5.4 show strong high-level semantic planning, achieving Middle Process scores of 32.4 and 27.4 respectively. However, their Final Result scores drop drastically, with GPT-5.4 falling to 11.8. This steep degradation highlights that advanced general models lack the fine-grained continuous spatial reasoning needed to prevent dependency-driven error propagation. Leading open-weight architectures, including Qwen3-VL-8B and InternVL2.5-8B, cluster near the radar chart center, reflecting systemic struggles across all metrics. Similarly, GUI-specialized agents designed for interface automation, such as UI-TARS and OS-ATLAS, exhibit severely limited performance. Trained primarily on region-tolerant paradigms, these agents fail to adapt to tasks requiring strict pixel-level parameter accuracy and geometry-aware verification. Our proposed framework consistently outperforms all evaluated baselines. By factorizing construction into dependency-structured planning and pixel-level execution, PAGER achieves the highest Middle Process score of 41.3. Furthermore, it effectively closes the Semantic-Execution Gap, delivering a leading Final Result score of 17.8 and an Overall Score of 29.5. These results empirically validate that our precision-aligned reinforcement learning pipeline successfully maintains spatial accuracy and resists cascading topological failures.

Appendix B Fine-Grained Classification Results Analysis

To further investigate the specific bottlenecks of existing models on precision-sensitive GUI tasks, we decompose the evaluation into ten distinct geometric capabilities. Figure 8 illustrates the performance distribution across these dimensions, highlighting severe variances in how different architectures handle continuous spatial reasoning and ontological dependencies.

Capabilities including natural language to geometric tool mapping and basic geometric object construction represent the fundamental prerequisites for successful drawing. Our proposed PAGER achieves peak performance in these areas with scores of 61.12 and 59.52 respectively. Strong proprietary models like Gemini-3.1-Pro and Qwen3.6-Plus also show reasonable competence here. This indicates that grounding basic semantic instructions to specific drawing tools remains relatively manageable for advanced vision-language models, aligning with their established strengths in semantic parsing.

As evaluation shifts toward multi-step dependency and sequential planning ability alongside geometric relations and structural constraints, baseline performance drops precipitously. GPT-5.4 scores only 34.68 on sequential planning, while open-weight models like Qwen3-VL-8B drop below 23. This degradation empirically validates our hypothesis regarding dependency-coupled operations. Small coordinate deviations in early steps accumulate into cascading topological failures that standard region-tolerant agents cannot resolve. By explicitly factorizing construction into dependency-structured planning and pixel-level execution, PAGER successfully preserves these structural constraints and maintains a sequential planning score of 55.04.

The most pronounced capability gaps emerge in complex scenarios, specifically auxiliary elements and implicit object introduction, multi-object composite management, and real-world scenario geometric modeling. These categories require abstract spatial reasoning and strict geometry-aware verification. GUI-specialized agents like UI-TARS and OS-ATLAS experience near-complete failures in these domains, with scores consistently falling to near 10. Conversely, PAGER demonstrates robust adaptability in these challenging regimes. The precision-aligned reinforcement learning phase effectively mitigates rollout-induced exposure bias, enabling our agent to handle implicit auxiliary constructions and complex multi-object dependencies far more reliably than existing general multimodal paradigms.

Appendix C GeoGebra Screening Prompt

媒体内容 · 前往原文查看

Appendix D GeoGebra Automation Prompt

媒体内容 · 前往原文查看

Appendix E Dataset Quality Assurance Prompt

媒体内容 · 前往原文查看

Appendix F Metrics

To comprehensively evaluate a model’s geometric generative reasoning capability, we design a four-stage evaluation protocol covering the middle process of action execution, the final geometric result, a stratified analysis, and a unified overall score.

F.1 Middle Process Metrics (Rule-Based)

To measure the quality of the step-by-step action sequence produced by the agent, we define four fine-grained rule-based metrics computed directly from the predicted action log against the ground-truth annotation.

Let the predicted action sequence be and the ground-truth sequence be , where each step .

Action Type Accuracy (AA, Action) measures per-step correctness of the predicted action category:

(11)

Parameter Accuracy (PA, Param) evaluates whether each predicted action’s parameters are correct. For type actions, string equality (case-insensitive) is required. For click actions, the predicted coordinate must fall within the annotated bounding box:

(12)

For paint actions, correctness is determined by the Euclidean pixel distance on the screen being within a 5-pixel tolerance:

(13)

The per-step correctness indicators are aggregated as:

(14)

Step Success Rate (SSR, Step) requires both action type and parameter to be simultaneously correct at each step:

(15)

Task Success Rate (TSR, Task) is a binary indicator per task requiring all steps to be simultaneously successful, averaged across tasks:

(16)

where is the total number of tasks.

All four metrics are first computed per task, then averaged across tasks within each evaluation ID, and finally macro-averaged across all IDs to obtain the model-level score. The Middle Process Score (MPS, Middle) is then computed as a weighted combination:

(17)

The weights reflect the hierarchy of difficulty: task-level success is the strictest criterion and thus contributes most, while action type accuracy serves as a lower-level sanity check.

F.2 Final Result Metrics

We evaluate the quality of the model’s final geometric output from two complementary perspectives.

Objective Task Completion Score (OTC, O-Comp.).

Given the ground-truth GeoGebra construction and the model’s construction , where denotes the set of labeled points with coordinates and denotes the list of construction commands, we compute a geometric similarity score. For each model point , we find its closest ground-truth match and define a continuous point-match contribution via an exponential kernel:

(18)

where is the normalized coordinate tolerance. For the command sequence, we translate each model command’s inputs through the point correspondence mapping and check for a signature match against the ground-truth command set:

(19)

The task completion score combines both components, with geometric structure weighted more heavily than point placement:

(20)

VLM-Based Holistic Scores.

To capture aspects that resist purely symbolic evaluation—such as visual rendering fidelity, label legibility, and cross-modal consistency—we employ Gemini-2.5-Pro as an automated evaluator. Given the task description, ground-truth JSON construction data, model JSON data, ground-truth rendered image, and model rendered image, the VLM outputs scores in across two evaluation views.

Under the data logic view and the visual presentation view, the model independently scores three dimensions:

• Task Completion (TC, S-Comp.): whether all geometric elements defined in the task list are present and identifiable in the output.

• Visual Similarity (VS, S-Vis.): pixel-level rendering fidelity relative to the ground-truth image, including line thickness, color, label positioning, and coordinate scaling.

• Geometric Logic (GL, S-Geo.): mathematical correctness of geometric relationships such as perpendicularity, tangency, and midpoints, including detection of visual “hallucinations” where the image appears correct but the underlying coordinates are logically inconsistent.

An overall comprehensive view synthesizes the scores from both the data logic view and the visual presentation view into a final set of scores , where each dimension is the average of its counterparts across the two views.

The Final Result Score (FRS, Final) is a weighted combination of the rule-based and VLM-based scores:

(21)

F.3 Overall Score

To facilitate model comparison with a single scalar, we define the Overall Score (OS, Overall) as the equally-weighted average of MPS and FRS:

(22)

The overall score thus balances procedural correctness in the action sequence against the quality of the final geometric output, providing a unified ranking across all evaluated models.