Siting Li

Work done while interning at Amazon FAR.

Zhengyang Wang

Amazon FAR

Simon Shaolei Du

Xi Chen

Amazon FAR

Amazon FAR

Abstract

Image tokenizers define the “visual language” of unified multimodal models, yet are commonly studied through isolated metrics or generation-/understanding-only evaluations. These evaluations do not fully capture how visual tokens behave when modeled jointly with text. We build a controlled pure-autoregressive testbed and track task-specific validation losses during multimodal continual pretraining across text, image, text-to-image (T2I), and image-to-text (I2T) prediction. We examine how these losses scale and relate to downstream performance, then use them to study multimodal learnability—how well image and text tokens are jointly modeled—and tokenizer design. We find that (1) losses should be analyzed by task, since they exhibit distinct scaling behavior and rank tokenizers differently. (2) The loss–performance relationship depends on the predicted token space: for a fixed tokenizer, T2I and I2T losses correlate with generation quality, but across tokenizers, the T2I loss–performance relationship shifts with the image-token space, whereas I2T loss, computed over a shared text vocabulary, provides a more consistent signal. I2T loss also correlates with both generation and visual understanding performance after supervised finetuning. Using losses as a lens, we show that (3) better reconstruction does not necessarily yield lower task-specific losses or stronger downstream performance, and that (4) image tokenizer choice can affect text modeling under joint optimization. As case studies, we revisit three tokenizer design axes—the discriminator, semantic supervision, and vocabulary size—to examine their effects on joint modeling and downstream performance. Together, our testbed offers a complementary perspective on image tokenizers as visual languages, highlighting their interplay with text in joint multimodal training.

1 Introduction

Large-scale unified multimodal models have advanced rapidly in both visual generation and understanding by modeling images and text within a single framework. Among these, autoregressive (AR) modeling is a popular choice due to its compatibility with strong off-the-shelf language models [52, 33, 59, 9]. In models that autoregressively predict both text and image tokens, a key component is the discrete image tokenizer, which converts raw pixels into discrete symbols that can be modeled alongside text [61, 46, 35, 34]. In this sense, an image tokenizer is not merely a preprocessing module; it defines the “visual language” that the language model must learn, align with text, and use for both visual generation and understanding.

However, existing tokenizer studies primarily examine this component outside the unified modeling context, relying on isolated reconstruction-based metrics (e.g., rFID [18]) or ImageNet classification accuracy [70, 71], which do not necessarily translate into better downstream performance [64]. Others study tokenizers through generation-only or understanding-only downstream pipelines and report benchmark scores [55, 67]. These end-to-end evaluations are valuable, but they still leave open how tokenizers affect joint modeling behavior. In unified pure-AR multimodal models, image and text tokens are modeled jointly under a shared model and training objective. Joint training with text may affect image-token modeling, while the choice of image tokenizer may in turn affect cross-modal alignment and text modeling. Whether these effects can be inferred from downstream-agnostic metrics or single-axis pipelines alone remains unclear. Consequently, tokenizer design choices, such as compression ratio and auxiliary losses, may be optimized without fully accounting for their effects on joint image–text modeling.

Motivated by this gap, we study how the image tokenizer shapes multimodal modeling behavior in unified AR multimodal training. In Section 4, we develop a controlled, pure-AR continual pretraining recipe using publicly available images and Qwen3 as the language backbone [63]. Building upon this testbed, we use task-specific validation losses during continual pretraining to characterize tokenizer effects on text, image, text-to-image (T2I), and image-to-text (I2T) modeling. Applying this perspective requires addressing two questions. First, it is unclear whether task-specific losses exhibit similar scaling behavior during mixed image–text AR continual pretraining, or whether they must be interpreted separately. Second, it is unclear how these losses relate to downstream benchmark performance. Different image tokenizers define different prediction spaces, which may alter the loss–performance relationship. Moreover, visual benchmarks emphasize semantic faithfulness, perceptual quality, and cross-modal alignment, which are not directly measured by next-token prediction loss.

We therefore first examine how pretraining losses should be interpreted in unified multimodal training (Section 5). Losses on text, image, T2I, and I2T all decrease with increasing data and model size, but exhibit distinct scaling behavior and rank tokenizers differently across tasks, making an averaged loss too coarse for this analysis. We then relate losses to generation and understanding benchmarks. When the tokenizer is fixed, T2I loss correlates with generation quality. I2T loss also tracks generation quality, suggesting that it captures aspects of multimodal training progress beyond caption prediction. Across tokenizers, however, the T2I loss–performance relationship shifts with the visual token space; vocabulary normalization reduces part of this shift, and the remaining gap is related to reconstruction fidelity. In contrast, I2T loss is computed over shared text tokens and remains a more consistent cross-tokenizer signal. I2T loss measured before supervised finetuning also correlates with post-finetuning generation and general VQA performance across tokenizers, supporting its use as a diagnostic signal.

Using task-specific losses as a lens, we make several observations about how the image tokenizer affects multimodal modeling behavior (Section 6). First, reconstruction fidelity can diverge from multimodal learnability—how well image and text tokens are modeled under the shared AR objective—indicating that reconstruction metrics alone are insufficient for evaluating tokenizers in unified multimodal models. Second, tokenizer choice can affect text modeling through the image-token prediction objective under joint training, a cross-modal effect not captured by evaluating generation or understanding alone. We then revisit three tokenizer design axes through ablations and find that (1) a DINO-based discriminator improves reconstruction fidelity but does not consistently improve multimodal learnability or downstream performance, with VQAv2 performance declining; (2) semantic supervision encourages object-level image-token–word associations, improving multimodal learnability despite worse reconstruction fidelity; and (3) the relationship between vocabulary size and multimodal learnability is non-monotonic, though a larger vocabulary can still benefit downstream performance, potentially through higher reconstruction fidelity. Together, these results offer a complementary perspective on image tokenizers as visual languages, highlighting their interplay with text in joint multimodal training.

2 Related Work

Unified multimodal understanding and generation. There have been recent advancements in building unified multimodal models, especially models for both visual understanding (captioning and VQA) and generation (text-to-image), in the hope of their mutual benefit. Various architectures have been proposed for unified models to jointly model text and image, including pure-autoregressive (AR) models (e.g., Chameleon [52], Emu3.5 [9], Liquid [59], Janus-Pro [8], and LongCat-Next [53]), serial AR + diffusion models (e.g., BLIP3-o [7] and MetaQuery [40]), and hybrid AR + diffusion models (e.g., Transfusion [74] and BAGEL [10]). Among these models, some employ a unified architecture and tokenizer for understanding and generation (e.g., Chameleon and Liquid), while others use decoupled modules, Mixture-of-Experts (MoEs), or different inference modes for the two tasks, and some adopt separate image tokenizers, illustrating the difficulty in truly unifying the two tasks in both tokenization and downstream models [72]. We adopt a pure-AR setup to model both modalities through next-token prediction, providing a controlled setting for studying image tokenizer effects under a shared modeling objective.

Loss-based analysis of unified multimodal models. Loss-based metrics, such as Perplexity (PPL) or Bits-Per-Byte (BPB), are commonly tracked against compute to validate model efficiency: Kaplan et al. [23] established held-out cross-entropy as a predictable scaling signal, Magnusson et al. [36] adopt BPB because token-level perplexity is not comparable across tokenizers, and Gadre et al. [14] fit language-model loss directly to average downstream error. Aghajanyan et al. [1] pioneered this analysis for mixed-modal models by establishing foundational scaling laws for mixed-modal loss, demonstrating that joint cross-entropy follows predictable power laws. Chameleon uses compute-loss curves to prove the stability of its early-fusion architecture at scale [52]. More recently, Liquid [59] utilizes these loss-vs-compute trends to demonstrate that inter-modality interference within a unified token space diminishes as model capacity increases. BAGEL [10] and Shukor et al. [47] identify emerging reasoning properties and the advantage of early-fusion and MoEs when scaling up compute, respectively. Tong et al. [56] uses IsoFLOP analysis to uncover the scaling asymmetry between vision and language in unified models. The above works on multimodal training use loss to characterize scaling, optimization, or performance under a fixed token space; we instead ask how losses should be interpreted across image tokenizers and use losses to study image tokenizers’ downstream effects.

Studying image tokenizer properties. (1) Reconstruction. PSNR and SSIM [58] are pixel-wise metrics for image reconstruction quality, but are sensitive to noise and poorly align with human perception. Feature-based metrics, including rFID [18], IS [44], and LPIPS [68] on the ImageNet-1K validation set [11], consider semantic and distributional quality of reconstructed images. Recent tokenizer benchmarks such as VTBench [32] and TokBench [60] measure how tokenizers preserve text, identity, and details, which are important for downstream tasks like OCR-VQA [37]. However, better reconstruction does not necessarily translate into better downstream performance, and could conflict with better generation [64]. ETT [57] shares this motivation but tunes the tokenizer end-to-end with the downstream model, whereas we keep tokenizers frozen to compare fixed visual token spaces. (2) Generation. Tokenizers can be compared by training with the same generative model and measuring the generation quality by gFID on a fixed set [61, 67, 70, 31], but generation-only evaluation does not establish how tokenizer choice affects both generation and understanding under joint training. (3) Semantics for understanding. Zero-shot accuracy and linear probing accuracy on ImageNet-1K have been used for both continuous and discrete tokenizers [42, 39, 71, 70], but higher classification accuracy does not necessarily translate into better downstream visual understanding, as observed in UniTok [35]. GigaTok [62] suggests AR probing accuracy as a better proxy to predict downstream performance when training with a larger AR model. TA-Tok [17] directly measures downstream performance at varying data scales. Cambrian-1 [55] pioneered leveraging multimodal large language models as an interface for tokenizer evaluation, but is restricted to continuous tokenizers. Apart from the above, (4) codebook usage is often reported in earlier work, but recent tokenizers can achieve near 100% usage with techniques like entropy loss even when they have a large vocabulary [76].

3 Preliminaries

In this section, we review discrete image tokenizers and the key design axes along which they differ: model and codebook architecture, bitwise compression ratio, and training objectives. Table 1 summarizes the tokenizers used in our study.

The VQGAN-based image tokenizers studied here [12, 66] follow an encoder-quantizer-decoder architecture. The encoder maps an input RGB image $x\in {ℝ}^{H\times W\times 3}$ to a sequence of $K$ continuous vectors $f\in {ℝ}^{K\times D}$, where $K$ is the number of image tokens and $D$ is the feature dimension. The quantizer maintains a codebook $Z\in {ℝ}^{B\times D}$ of vocabulary size $B$ and replaces each encoder vector with its nearest codebook entry, yielding quantized vectors $z\in {ℝ}^{K\times D}$. The corresponding codebook indices form the discrete image-token sequence modeled by the autoregressive model. The decoder reconstructs an RGB image $\hat{x}\in {ℝ}^{H\times W\times 3}$ from $z$. A standard VQGAN is trained with a vector-quantization (VQ) loss and an autoencoding (AE) loss:

${ℒ}_{VQ}$

$\phantom{}={\Vert \text{sg}\left(f\right)-z\Vert }_2^2+\beta {\Vert f-\text{sg}\left(z\right)\Vert }_2^2,{ℒ}_{AE}={ℒ}_2(x,\hat{x})+{ℒ}_{𝒫}(x,\hat{x})+{ℒ}_{𝒢}(x,\hat{x}),$

(1)

where $\text{sg}(\cdot )$ denotes the stop-gradient operation, $\beta$ weights the commitment loss, and ${ℒ}_2,{ℒ}_{𝒫},{ℒ}_{𝒢}$ are the $L_2$ reconstruction loss, the LPIPS perceptual loss [68], and the adversarial (GAN) loss from the discriminator, respectively.

媒体内容 · 前往原文查看

Tokenizer	Size	$B$	${ℒ}_{sem}$	Disc.	Usage	rFID$\downarrow$
IBQ-1024	0.1B	1024		PatchGAN	99%	2.24
IBQ-8192	0.1B	8192		PatchGAN	98%	1.87
IBQ-16384	0.1B	16384		PatchGAN	96%	1.37
GigaTok	0.6B	16384	✓	PatchGAN	100%	0.81
GigaTok-DINO	0.6B	16384	✓	DINO	100%	0.51
UniTok†	0.8B	16384		DINOv2-S	100%	1.86
UniTok-sem†	0.8B	16384	✓	DINOv2-S	100%	2.23

Architecture. On the encoder/decoder choices, prior work adopts various architectures, including ConvNet [12, 51], ViT [66], and ViT-based models with learnable latent queries [67]. On the codebook design, some tokenizers use multiple sub-codebooks [41, 49, 35] to represent each spatial position with multiple indices. Modeling these indices introduces additional choices in sequence organization and prediction architecture, complicating controlled comparisons. We therefore focus on single-codebook tokenizers throughout this work.

Bitwise compression ratio. The bitwise compression ratio is determined by the input image resolution $H\times W$, the number of image tokens $K$, and the vocabulary size $B$. In this work, we fix the number of image tokens to $K=16\times 16$ and the input resolution to $H\times W=256\times 256$, and study the vocabulary size $B$ as the main compression-related axis using the IBQ tokenizer family [46] (Section 6.4). We defer to future work the study of varying $H\times W$ (e.g., any-resolution image tokenizers [34]) and varying $K$ (e.g., highly compressed 1D tokenizers [2, 24]), as changing image resolution alters the visual detail available to the model, while changing $K$ alters sequence length and the image-token budget during training, introducing additional factors into the comparison.

Training objectives. Beyond the standard losses in Eq. (1), we study two design choices in the training objective. On the supervision signal, recent tokenizers add semantic supervision, such as a CLIP contrastive loss [35], to encourage the tokens to capture high-level semantics; we study its effect on downstream joint modeling by training single-codebook UniTok variants without and with this loss, denoted UniTok and UniTok-sem, respectively [35], in Section 6.3. On the adversarial loss ${ℒ}_G$, some tokenizers replace the standard PatchGAN discriminator [22] with a DINO-based discriminator [4] to obtain lower rFID [54, 28]; we revisit whether this reconstruction gain carries over to downstream joint modeling using GigaTok with the two discriminators (denoted GigaTok and GigaTok-DINO) [62] in Section 6.1.

4 Framework Construction

In this section, we describe the framework for exploring the image tokenizer’s effect on downstream unified multimodal training. In Section 4.1, we introduce the training recipe, which extends a pretrained text language model into a unified autoregressive multimodal model through continual pretraining and supervised finetuning. In Section 4.2, we define the validation loss we use to measure modeling quality across tasks. Further training details and tokenizer information are provided in Appendix A.

4.1 Training Recipe

Overall setup. We extend pretrained Qwen3 language models [63] into unified autoregressive multimodal models by expanding their vocabularies to include discrete image tokens. Our main experiments use three dense model sizes: Qwen3-0.6B, 1.7B, and 4B. Given an image tokenizer with vocabulary size $B$, we add $B$ new learnable embeddings to the base model’s vocabulary, initialized from a multivariate normal distribution matching the original embeddings’ mean and covariance. We also extend the LM head to predict the newly added image tokens. We add special tokens ⟨boi⟩ and ⟨eoi⟩ to mark the start and end of image-token sequences. During training, we remove the text conditioning from $10\%$ of T2I samples and use ⟨unconditional⟩ to mark unconditional image generation, enabling classifier-free guidance (CFG) at evaluation [19]. The resulting model accepts and produces both text tokens and flattened image-token IDs. We continually pretrain it on mixed-modal data and then apply supervised finetuning (SFT) on multimodal instruction-following data, as detailed below.

Continual pretraining stage. In this stage, we train the unified model on large-scale image-text and pure-text data.

(1) Data mixture and preprocessing. Our largest data scale is 60M samples, comprising 6.6M pure-text samples from DataComp-LM [27] and 53.3M image-text samples from three sources: (a) LAION-Aesthetics [45], filtered by aesthetic score $\phantom{}\ge 5.5$ and recaptioned by InternVL3-1B [75]; (b) JourneyDB [50], recaptioned by GPT-3.5; and (c) BLIP3o-Pretrain-Short-Caption [7]. We resize all images to $256\times 256$ and pre-tokenize them before training.

(2) Loss and token sequence formatting. For pure-text samples, we adopt the standard cross-entropy loss for next-token prediction in training. For image-text samples, we initially assign $80\%$ to text-to-image (T2I) prediction and $20\%$ to image-to-text (I2T) prediction. As described above, $10\%$ of the T2I samples are converted to unconditional image generation. The conditional sequence formats are:

		$\text{T2I: {text}{prompt}⟨boi⟩{}\text{image tokens}\text{}⟨}\text{eoi}\text{⟩⟨}\text{eos}\text{⟩},$
		$\text{I2T: ⟨boi⟩{image tokens}⟨eoi⟩{prompt}{}\text{text}\text{}⟨}\text{eos}\text{⟩}.$

Following Liquid [59], we compute the cross-entropy loss only on the bold tokens (conditional next-token prediction), leaving the prompt and the conditioning modality unscored. The prompts are listed in Appendix A.2.

(3) Hyperparameters. We use a Warmup-Stable-Decay (WSD) schedule with a $0.03$ warmup ratio and linear decay over the last $20\%$ of steps. For the 0.6B model, we sweep the learning rate (lr) over $\{3\text{e-}5,1\text{e-}4\}$ and the batch size (bs) over $\{512,1024,2048\}$, forming a $2\times 3$ grid. Because different tasks favor different hyperparameter settings (Appendix C.2), we adopt a balanced setting of (lr $\phantom{}=3\text{e-}5$, bs $\phantom{}=512$) for the main experiments and reuse it for the 1.7B and 4B models. We also experimented with larger learning rates ($3\text{e-}4$ for 0.6B and $1\text{e-}4$ for 4B), but found that they caused unstable training with loss spikes.

Supervised finetuning (SFT) stage. We finetune the models for 2 epochs on 4.9M instruction-following samples. The mixture combines 1M LMSYS-Chat pure-text instructions [73], 2.9M multimodal instruction and captioning samples from Mini-Gemini [29], and 1M text-to-image samples (LAION-Aesthetics sampled from the continual-pretraining distribution, together with JourneyDB and BLIP3o-60K data [7]). We use a cosine schedule with a $0.03$ warmup ratio, a peak learning rate of $5\text{e-}5$, and batch size $1024$.

Further details about data of both stages, including the subset proportions, can be found in Appendix A.1. All images used in the training are publicly accessible.

媒体内容 · 前往原文查看

Model	Data	GenAI$\uparrow$	MJHQ-30K$\downarrow$	WISE$\uparrow$	VQA (Mean)$\uparrow$
Liquid-7B	90M	0.72	5.47	0.41	61.40
Ours (8B)	60M	0.73	10.55	0.38	60.83

Training recipe validation. We verify our training recipe by training a Qwen3-8B model with the Chameleon tokenizer (Table 2). Compared with Liquid-7B, our model reaches similar performance on GenAI-Bench, WISE, and VQA (averaged over VQAv2, GQA, TextVQA, and POPE), while trailing on MJHQ-30K. Full benchmark scores and image generation examples are provided in Appendix A.3. We view this experiment as a validation that the recipe serves as a reliable testbed for studying tokenizers, rather than as evidence of achieving superior performance.

4.2 Validation Loss

We use validation loss, defined as the mean negative log-likelihood over supervised tokens, to measure how well a model fits held-out data, following common practice in language-model scaling studies [23, 20, 1]. Let $N$ be the evaluation set. For a sequence $t\in N$, let $S\left(t\right)\subseteq \{1,\dots ,\left|t\right|\}$ denote its supervised positions: the tokens on which the loss is computed (the bold spans of the T2I/I2T formats in Section 4.1; all positions for pure-text samples). The model’s log-likelihood on the supervised tokens is

$l=\sum _{t\in N}\sum _{i\in S\left(t\right)}\ln p(t_i\mid t_{\phantom{}<i}),$

(2)

where conditioning tokens (e.g., the text prompt or the conditioning image) are never scored but still enter through the context $t_{\phantom{}<i}$. Using $T\left(N\right)={\sum }_{t\in N}\left|S\left(t\right)\right|$ for the number of supervised tokens, we report the mean per-token loss

$ℒ=-\frac{l}{T\left(N\right)}.$

(3)

Note on loss interpretation. Unlike the bits-per-byte loss common in language modeling, which upper-bounds a description length of the raw data [15, 36], our validation loss on image tokens is computed over the tokenizer’s discrete latent codes. Because image tokenization is lossy, the model’s likelihood on these codes cannot be converted into a description length for the original pixels. We therefore interpret $ℒ$ as a measure of latent-modeling quality for a fixed visual language. Comparisons across image tokenizers require accounting for differences in their token spaces, as examined in Section 5.2.

Evaluation protocol. We hold out $\mathrm{50,000}$ pure-text and $\mathrm{50,000}$ image-text samples, drawn from the training distribution, as the evaluation set. This yields validation losses for four tasks: text, unconditional image, text-to-image (T2I), and image-to-text (I2T). We apply the same loss-masking rules as in training: conditioning tokens provide context but do not contribute directly to the loss. Empirically, losses over different image sources behave similarly, so we report LAION-Aesthetics as our main proxy, denoted LAION-(Image, T2I, I2T); losses on the other sources are given in Appendix C.1. In loss-scaling plots, the $x$-axis FLOPs denote the compute used in continual mixed-modal pretraining.

5 Interpreting Loss in Unified Multimodal Training

Before using pretraining loss to study tokenizers, we first examine how it should be interpreted in unified multimodal training. We begin by asking whether a single unified loss metric can characterize tokenizer behavior in downstream modeling, or whether the task-specific losses must be analyzed separately (Section 5.1). We then examine whether these losses align with the performance on image generation benchmarks and how they can be calibrated across tokenizers (Section 5.2.1). Finally, we test how they relate to post-SFT performance (Section 5.2.2).

5.1 Task-Specific Loss Scaling

媒体内容 · 前往原文查看

媒体内容 · 前往原文查看

Our setting differs from standard language-only pretraining in three aspects: we continually train from a text-only pretrained model; the training is mixed-modal, spanning four distinct tasks; and the T2I and I2T losses are conditional. These differences make the loss harder to interpret: it is not obvious whether the four task-specific losses share a common scaling behavior or must be analyzed separately. We therefore first examine how each task’s loss scales with data and model size, and then ask whether any single signal derived from the loss yields a tokenizer ranking that is consistent across tasks.

All task-specific losses scale with data and model size. We plot validation loss against pretraining FLOPs for three tokenizers, varying data scale (Figure 2) and model size (Figure 3). Both figures show that the losses on all four tasks decrease smoothly and roughly follow a power law (a linear trend in the log-log plots) under both data and model scaling. For the data-scaling analysis, we use intermediate, pre-annealing checkpoints from a single training run for each tokenizer–model-size configuration, with lr $\phantom{}=3\text{e-}5$ and bs $\phantom{}=512$. Annealed checkpoints follow the same power-law trends, with annealing mainly reducing the text loss and leaving the image-side losses largely unchanged (Appendix C.3). Results across hyperparameters are deferred to Appendix C.2, and further continual-pretraining dynamics to Appendix B.

The scaling patterns differ across tasks. Although all four losses scale, they do so in qualitatively different ways, which makes an aggregated loss too coarse to serve as an analysis signal. (1) Text loss is largely governed by language-model initialization, while image-related losses are shaped more by multimodal continual pretraining. Under data scaling at a fixed model size (Figure 2), text loss changes only slightly, whereas image, T2I, and I2T losses decrease more substantially. This suggests that text-modeling quality is largely inherited from the pretrained language backbone, while image-related tasks benefit more from multimodal continual pretraining. Under model scaling (Figure 3), text loss also decreases, consistent with initializing larger models from stronger pretrained language backbones; this trend therefore cannot be attributed to multimodal continual pretraining alone [69]. (2) Image-token modeling drives T2I, while I2T reflects cross-modal alignment. The T2I loss closely tracks the unconditional image loss, indicating that image-token prediction accounts for much of the difficulty in T2I. The I2T loss, though also computed over text tokens, behaves differently from the text loss, indicating that conditioning on image tokens introduces a distinct cross-modal signal.

Tokenizer rankings differ across tasks under both data and model scaling. No single ordering of the three tokenizers holds across tasks. In Figure 2, UniTok has the highest text loss yet the lowest T2I loss, while GigaTok has the lowest text-related losses but the highest T2I loss. Consequently, no single task loss yields a tokenizer ranking that holds across tasks, and a simple average can obscure these task-dependent differences. These task-dependent rankings suggest trade-offs between text modeling and image-token prediction under the unified AR objective. Stronger semantic alignment, in turn, can help I2T even when a tokenizer is not optimal for pure-text modeling, as seen in the comparison between UniTok and IBQ-16384 in Figure 2.

5.2 Loss–Performance Relation

We next study how pretraining losses relate to benchmark performance. These validation losses measure next-token modeling quality, whereas downstream benchmarks emphasize semantic faithfulness, perceptual quality, and cross-modal alignment. We therefore ask the following questions:

• (Section 5.2.1) In pretraining, do the losses align with image-generation benchmark scores, and are these relationships consistent across tokenizers?

• (Section 5.2.2) Do the losses serve as signals of post-SFT benchmark performance?

Benchmarks. We evaluate the continually pretrained checkpoints under varying hyperparameters on GenAI-Bench [25] and MJHQ-30K [26], scored by VQAScore ($\uparrow$) and ${\log }_{10}$ gFID ($\downarrow$), respectively. We then apply our SFT recipe to the pretraining checkpoints after annealing and re-evaluate generation performance, together with general visual understanding on VQAv2 [16] and GQA [21]. Qualitative examples on GenAI-Bench and further benchmark details are in Appendix D.

5.2.1 Alignment between Losses and T2I Quality in Pretraining

媒体内容 · 前往原文查看

媒体内容 · 前往原文查看

媒体内容 · 前往原文查看

媒体内容 · 前往原文查看

We study the alignment between pretraining losses and T2I generation quality, which can be evaluated directly on the pretrained checkpoints without SFT.

Within a tokenizer, both T2I and I2T loss align with T2I quality. Since T2I generation predicts image tokens conditioned on text, one might expect the T2I validation loss to be the most relevant pretraining signal for generation quality. This holds when the tokenizer is fixed: as shown in Figure 4, T2I loss aligns strongly with GenAI-Bench VQAScore and MJHQ-30K gFID across data scales and hyperparameter settings. More surprisingly, I2T loss also correlates with generation quality, albeit more weakly, suggesting that it captures aspects of multimodal training progress beyond caption prediction.

Across tokenizers, the T2I loss–performance relation shifts with the image-token space. We next ask whether both alignments hold across tokenizers. I2T loss is computed over a shared text vocabulary, providing a common basis for comparison. Empirically, Figure 5 shows a more consistent relationship between I2T loss and generation quality across tokenizers. T2I loss, however, exhibits tokenizer-dependent shifts in its relationship with generation quality. Thus, T2I loss remains a useful modeling diagnostic, but the same loss need not correspond to the same image quality across tokenizers, since each defines a different image-token space.

Vocabulary normalization reduces the cross-tokenizer shift in the T2I loss–performance relation. We hypothesize that vocabulary size contributes to the numerical scale of T2I loss. A natural reference is ${\log }_{2}B$, the entropy of a uniform distribution over $B$ image tokens. We therefore define the vocabulary-normalized image loss as ${ℒ}^{\ast }=ℒ/{\log }_{2}B$, where $B$ denotes the image vocabulary size. Evaluated on IBQ tokenizers with $B\in \left\{1024,8192,16384\right\}$, ${ℒ}^{\ast }$ yields a more consistent T2I loss–performance relation across the IBQ variants (Figure 7). Further details are in Appendix C.5.

The residual cross-tokenizer gap is linked to reconstruction fidelity. Among tokenizers with the same vocabulary size ($16384$), Figure 7 shows that, at a fixed benchmark score, T2I loss follows the reverse order of the rFID ranking: a tokenizer with worse reconstruction fidelity reaches the same generation quality at a lower T2I loss. This association suggests that reconstruction fidelity may help explain the residual differences in the T2I loss–performance relationship at a fixed vocabulary size. These results motivate considering reconstruction fidelity alongside T2I loss when comparing generation performance across tokenizers.

5.2.2 Correlation between Losses and Post-SFT Performance

We ask whether pretraining losses remain a useful signal of benchmark performance after supervised finetuning, which introduces additional variation beyond pretraining.

媒体内容 · 前往原文查看

媒体内容 · 前往原文查看

Losses correlate with generation quality after SFT. The pretraining alignment of Section 5.2.1 carries over to post-SFT performance. When the tokenizer is fixed (GigaTok), both the T2I and I2T validation losses measured before SFT remain strongly correlated with post-SFT benchmark performance (Figure 8). Across tokenizers, the I2T loss again remains the more consistent signal, staying correlated with post-SFT generation performance (Figure 9).

I2T loss is informative about general visual understanding. Across tokenizers, hyperparameters, and training scales, the I2T loss shows a moderate correlation with post-SFT VQAv2 and GQA performance (Figure 9), indicating that it captures part of the capability that transfers to general VQA and can thus serve as a meaningful signal. The moderate correlation may partly reflect differences in both task and training stage: the loss measures caption prediction during pretraining, while the benchmarks measure question answering after SFT. Caption fit is therefore informative about VQA performance, but does not determine it. This imperfect correlation is unlikely to be explained solely by data-shuffling noise: in our reproducibility check of one GigaTok configuration, VQAv2 and GQA scores vary by less than $1\%$ relative across two seeds (Table 11). The relationship is less consistent for specialized benchmarks such as TextVQA, which additionally require capabilities such as OCR (Appendix C.4).

6 What Unified Training Reveals About Image Tokenizers

Building on our loss-based analysis of unified multimodal training (Section 5), we now use validation loss as a lens to reveal tokenizer properties that isolated metrics or single-axis evaluations miss: reconstruction fidelity can diverge from multimodal learnability, and the image token space can even affect text modeling (Section 6.2). We study these effects through three tokenizer design axes as case studies: the discriminator (Section 6.1), semantic supervision (Section 6.3), and vocabulary size (Section 6.4).

6.1 Does better reconstruction imply better unified multimodal learning?

The premise behind reconstruction-based tokenizer selection is that higher-fidelity reconstruction yields better downstream results. Recent analyses question this premise, showing that tokenizer design trades off reconstruction fidelity, compression, and latent-space learnability, though largely in the visual-generation-only setting [3, 64, 43, 62]. We revisit this fidelity–learnability–performance relationship in unified AR multimodal training, where we measure fidelity by rFID and take multimodal learnability to be how well the resulting image and text tokens are modeled under the shared AR objective, as reflected by the task-specific validation losses of Section 5.

媒体内容 · 前往原文查看

Tokenizer	Fidelity	Multimodal Learnability			Performance
	rFID$\downarrow$	Text$\downarrow$	I2T$\downarrow$	T2I$\downarrow$	GenAI$\uparrow$	VQAv2$\uparrow$
GigaTok-DINO	0.51	2.949	1.660	7.437	0.720	51.31
GigaTok	0.81	2.937	1.661	7.416	0.720	52.25
UniTok†	1.86	2.883	1.670	6.264	0.670	57.21
UniTok-sem†	2.23	2.873	1.655	5.848	0.690	61.28

Better reconstruction does not always lead to better unified multimodal learning. We compare several tokenizer pairs on reconstruction fidelity, multimodal learnability, and downstream performance in Table 3. Within each tokenizer family, better reconstruction does not consistently correspond to better downstream generation or understanding performance. Replacing the standard PatchGAN discriminator with a DINO-based discriminator improves GigaTok’s rFID from $0.81$ to $0.51$, yet this reconstruction gain does not translate into a better joint model: none of the validation losses improve significantly, GenAI is unchanged ($0.720$), and VQAv2 drops from $52.25$ to $51.31$. This complements the image-generation-only study of GigaTok [62], where the DINO-based discriminator is adopted to improve reconstruction. In contrast, semantic supervision in UniTok-sem worsens reconstruction fidelity yet improves all three reported validation losses and both reported downstream metrics (Section 6.3). Together, these results show that a reconstruction metric such as rFID alone is insufficient for tokenizer selection: we must also consider how the tokenizer affects multimodal learnability.

6.2 Does the image token space affect text modeling?

In unified AR training, text and image tokens are optimized under the same model and objective. This raises a question not addressed by evaluating generation or understanding alone: can the image tokenizer affect text modeling even when the text-only data, text tokenizer, and language backbone are all fixed?

媒体内容 · 前往原文查看

Image token space affects text modeling difficulty under joint training. We observe this effect in two tokenizer comparisons (lr=3e-5, batch size=512) in Figure 10 (Joint). First, UniTok and UniTok-sem share the same text tokenizer and training recipe, yet UniTok-sem achieves lower text loss under joint text-image training. Second, among IBQ variants, IBQ-1024 obtains lower text loss than IBQ-8192 despite using the same text token space. These differences suggest that changes in the image token space can affect the difficulty of text modeling. In these comparisons, adding semantic supervision to UniTok or reducing the IBQ vocabulary from 8192 to 1024 lowers text loss under joint training.

The text-side effect comes from image token prediction rather than captioning (I2T). We hypothesize that this text-side effect arises from interference between image token modeling and text token modeling. To test this, we remove the image token prediction objective by reformatting the T2I and image-generation samples into I2T order and train on only Text+I2T data with the same hyperparameter settings. Under this ablation, the text-loss gap largely disappears for both the UniTok pair and the IBQ pair (Figure 10, Text+I2T), indicating that the text-side difference is explained by the image token prediction objective in full unified training. Interestingly, the I2T gap in both pairs persists, suggesting that tokenizer choice affects how informative image tokens are for captioning through a pathway independent of image token prediction. In Appendix C.6, we further verify that the text-side effect is not caused by the I2T objective through a similar ablation on I2T data.

6.3 Semantic supervision: How does semantic alignment change multimodal learnability?

Semantic supervision is commonly added in image tokenizer training to encourage visual features to encode higher-level semantics. In Section 6.1, we observed that it improves the multimodal learnability of the resulting visual language, which in turn benefits downstream performance. We now ask where this improvement comes from: does semantic supervision make local image-token sequences easier to predict (“better local visual grammar”), or does it make individual image tokens more informative for captioning (“better visual words”)?

媒体内容 · 前往原文查看

媒体内容 · 前往原文查看

The I2T gain is concentrated on visually grounded object words. In the Text+I2T setting of Section 6.2, where image token prediction is ablated, we track each tokenizer’s I2T loss improvement over training and take the gap (${\Delta }_{\text{UniTok-sem}}-{\Delta }_{\text{UniTok}}$), for which larger values mean greater relative improvement of UniTok-sem. This gain concentrates on visually grounded object words (Figure 12, left): COCO object-category tokens improve far more than stopwords, general content tokens (all tokens apart from stopwords), or color tokens (CSS color names), indicating that semantic supervision makes image tokens more informative for predicting object-level words.

Semantic supervision strengthens object-level image-token–word association. As further evidence, the pointwise mutual information (PMI) between image tokens and words, averaged within three word groups on the validation set, is higher for UniTok-sem on COCO object words (Figure 12, right), suggesting that semantic supervision associates image tokens more tightly with object-level words.

Semantic supervision does not consistently reduce empirical image-token $n$-gram entropy. An alternative explanation is that semantic supervision makes image-token sequences easier to model. Following Chan et al. [5], we compute the empirical 1-, 2-, 3-, and 7-gram entropy of the image tokens produced by UniTok and UniTok-sem on the validation set (Figure 12). While UniTok-sem tokens have slightly lower entropy at 1- and 2-grams, their 3- and 7-gram entropy is higher than for UniTok. These measurements do not show a consistent reduction in local $n$-gram entropy, providing no clear evidence for a simpler local visual grammar under this diagnostic.

6.4 Vocabulary size: Is a larger vocabulary better for unified multimodal learning?

Vocabulary size is a key compression axis of discrete image tokenizers. Prior work shows that scaling up the vocabulary improves reconstruction fidelity by letting each image token carry more visual information [46], but it may also change the modeling difficulty of image tokens. Since reconstruction fidelity and multimodal learnability can diverge, we ask whether a larger vocabulary brings better unified multimodal learning. Within the IBQ family, we vary $B\in \left\{1024,8192,16384\right\}$ and compare image-side losses using the vocabulary-normalized image-token loss ${ℒ}^{\ast }$ (Section 5.2.1).

媒体内容 · 前往原文查看

Vocabulary size affects task-specific losses non-monotonically. As shown in Figure 13, the relationship between vocabulary size $B$ and loss is not simply larger-is-better. IBQ-16384 attains the best I2T loss, yet the intermediate size IBQ-8192 achieves the lowest normalized T2I and image losses among the three variants. Text loss, meanwhile, follows the reverse (also non-monotonic) trend relative to the image-side losses, again indicating that different tasks can prefer different visual token spaces.

媒体内容 · 前往原文查看

A larger vocabulary can benefit downstream performance, possibly through higher reconstruction fidelity. Comparing the IBQ variants on generation and understanding benchmarks after SFT (Figure 14), IBQ-16384 achieves the best performance among the three, despite IBQ-8192 having lower T2I and image losses. As with the rFID–performance relation in Section 5.2.1, this is plausibly due to IBQ-16384’s higher reconstruction fidelity. Beyond $B=16384$, however, it remains unclear whether reconstruction fidelity or multimodal learnability is the dominant factor, which we leave to future work.

7 Conclusion

We study image tokenizers in the context of unified AR multimodal models through a controlled, loss-based testbed. We first examine how validation loss relates to downstream performance and how the task-specific losses should be interpreted. Building on this, we show that reconstruction fidelity can diverge from multimodal learnability and that the image token space can affect text modeling under joint training. We also revisit discriminator choice, semantic supervision, and vocabulary size to examine their effects on unified multimodal learning. Our findings suggest that unified tokenizer design should account for both reconstruction fidelity and joint modeling difficulty across tasks, rather than optimizing any single proxy metric. These observations highlight the importance of studying image tokenizers as visual languages in interaction with text during joint multimodal training.

Acknowledgments and Disclosure of Funding

SSD acknowledges the support of NSF IIS 2143493, NSF IIS 2229881, Sloan Fellowship, and the AI2050 program at Schmidt Sciences. This material is based on the Chameleon tokenizer supported by the Chameleon Research License, Copyright (c) Meta Platforms, Inc. All Rights Reserved.

References

• [1] A. Aghajanyan, L. Yu, A. Conneau, W. Hsu, K. Hambardzumyan, S. Zhang, S. Roller, N. Goyal, O. Levy, and L. Zettlemoyer (2023) Scaling laws for generative mixed-modal language models. In International Conference on Machine Learning, pp. 265–279. Cited by: §2, §4.2.
• [2] R. Bachmann, J. Allardice, D. Mizrahi, E. Fini, O. F. Kar, E. Amirloo, A. El-Nouby, A. Zamir, and A. Dehghan (2025) FlexTok: resampling images into 1d token sequences of flexible length. In Forty-second International Conference on Machine Learning, Cited by: §3.
• [3] Black Forest Labs (2025) FLUX.2: analyzing and enhancing the latent space of FLUX – representation comparison. External Links: Link Cited by: §6.1.
• [4] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin (2021) Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9650–9660. Cited by: §3.
• [5] D. M. Chan, R. Corona, J. Park, C. J. Cho, Y. Bai, and T. Darrell (2024) Analyzing the language of visual tokens. arXiv preprint arXiv:2411.05001. Cited by: §6.3.
• [6] J. Chen, Q. Yu, X. Shen, A. Yuille, and L. Chen (2024) ViTamin: designing scalable vision models in the vision-language era. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12954–12966. Cited by: §A.4.
• [7] J. Chen, Z. Xu, X. Pan, Y. Hu, C. Qin, T. Goldstein, L. Huang, T. Zhou, S. Xie, S. Savarese, et al. (2025) BLIP3-o: a family of fully open unified multimodal models-architecture, training and dataset. arXiv preprint arXiv:2505.09568. Cited by: §2, §4.1, §4.1.
• [8] X. Chen, Z. Wu, X. Liu, Z. Pan, W. Liu, Z. Xie, X. Yu, and C. Ruan (2025) Janus-Pro: unified multimodal understanding and generation with data and model scaling. arXiv preprint arXiv:2501.17811. Cited by: §2.
• [9] Y. Cui, H. Chen, H. Deng, X. Huang, X. Li, J. Liu, Y. Liu, Z. Luo, J. Wang, W. Wang, et al. (2025) Emu3.5: native multimodal models are world learners. arXiv preprint arXiv:2510.26583. Cited by: §A.4, §1, §2.
• [10] C. Deng, D. Zhu, K. Li, C. Gou, F. Li, Z. Wang, S. Zhong, W. Yu, X. Nie, Z. Song, et al. (2025) Emerging properties in unified multimodal pretraining. arXiv preprint arXiv:2505.14683. Cited by: §2, §2.
• [11] J. Deng, W. Dong, R. Socher, L. Li, K. Li, and L. Fei-Fei (2009) ImageNet: a large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248–255. Cited by: §2.
• [12] P. Esser, R. Rombach, and B. Ommer (2021) Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12873–12883. Cited by: §3, §3.
• [13] S. Y. Gadre, G. Ilharco, A. Fang, J. Hayase, G. Smyrnis, T. Nguyen, R. Marten, M. Wortsman, D. Ghosh, J. Zhang, et al. (2023) DataComp: in search of the next generation of multimodal datasets. Advances in Neural Information Processing Systems 36, pp. 27092–27112. Cited by: §A.4.
• [14] S. Y. Gadre, G. Smyrnis, V. Shankar, S. Gururangan, M. Wortsman, R. Shao, J. Mercat, A. Fang, J. Li, S. Keh, et al. (2024) Language models scale reliably with over-training and on downstream tasks. arXiv preprint arXiv:2403.08540. Cited by: §2.
• [15] L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, et al. (2020) The pile: an 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027. Cited by: §4.2.
• [16] Y. Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh (2017) Making the V in VQA matter: elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6904–6913. Cited by: Appendix D, §5.2.
• [17] J. Han, H. Chen, Y. Zhao, H. Wang, Q. Zhao, Z. Yang, H. He, X. Yue, and L. Jiang (2025) Vision as a dialect: unifying visual understanding and generation via text-aligned representations. arXiv preprint arXiv:2506.18898. Cited by: §2.
• [18] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter (2017) GANs trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems 30. Cited by: §1, §2.
• [19] J. Ho and T. Salimans (2022) Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598. Cited by: §4.1.
• [20] J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark, et al. (2022) Training compute-optimal large language models. arXiv preprint arXiv:2203.15556. Cited by: §4.2.
• [21] D. A. Hudson and C. D. Manning (2019) GQA: a new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6700–6709. Cited by: Appendix D, §5.2.
• [22] P. Isola, J. Zhu, T. Zhou, and A. A. Efros (2017) Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1125–1134. Cited by: §3.
• [23] J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei (2020) Scaling laws for neural language models. arXiv preprint arXiv:2001.08361. Cited by: §2, §4.2.
• [24] D. Kim, J. He, Q. Yu, C. Yang, X. Shen, S. Kwak, and L. Chen (2025) Democratizing text-to-image masked generative models with compact text-aware one-dimensional tokens. arXiv preprint arXiv:2501.07730. Cited by: §3.
• [25] B. Li, Z. Lin, D. Pathak, J. Li, Y. Fei, K. Wu, T. Ling, X. Xia, P. Zhang, G. Neubig, et al. (2024) GenAI-Bench: evaluating and improving compositional text-to-visual generation. arXiv preprint arXiv:2406.13743. Cited by: Appendix D, Table 2, §5.2.
• [26] D. Li, A. Kamko, E. Akhgari, A. Sabet, L. Xu, and S. Doshi (2024) Playground v2.5: three insights towards enhancing aesthetic quality in text-to-image generation. External Links: 2402.17245 Cited by: Appendix D, Table 2, §5.2.
• [27] J. Li, A. Fang, G. Smyrnis, M. Ivgi, M. Jordan, S. Y. Gadre, H. Bansal, E. Guha, S. S. Keh, K. Arora, et al. (2024) DataComp-LM: in search of the next generation of training sets for language models. Advances in Neural Information Processing Systems 37, pp. 14200–14282. Cited by: §4.1.
• [28] X. Li, K. Qiu, H. Chen, J. Kuen, J. Gu, B. Raj, and Z. Lin (2024) Imagefolder: autoregressive image generation with folded tokens. arXiv preprint arXiv:2410.01756. Cited by: §3.
• [29] Y. Li, Y. Zhang, C. Wang, Z. Zhong, Y. Chen, R. Chu, S. Liu, and J. Jia (2025) Mini-Gemini: mining the potential of multi-modality vision language models. IEEE Transactions on Pattern Analysis and Machine Intelligence. Cited by: §4.1.
• [30] Y. Li, Y. Du, K. Zhou, J. Wang, W. X. Zhao, and J. Wen (2023) Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355. Cited by: Appendix D.
• [31] H. Lin, T. Wang, Y. Ge, Y. Ge, Z. Lu, Y. Wei, Q. Zhang, Z. Sun, and Y. Shan (2025) TokLIP: marry visual tokens to clip for multimodal comprehension and generation. arXiv preprint arXiv:2505.05422. Cited by: §2.
• [32] H. Lin, T. Geng, Z. Xu, and W. Zhao (2025) VTBench: evaluating visual tokenizers for autoregressive image generation. arXiv preprint arXiv:2505.13439. Cited by: §2.
• [33] H. Liu, W. Yan, M. Zaharia, and P. Abbeel (2024) World model on million-length video and language with blockwise ringattention. arXiv preprint arXiv:2402.08268. Cited by: §1.
• [34] J. Lu, L. Song, M. Xu, B. Ahn, Y. Wang, C. Chen, A. Dehghan, and Y. Yang (2025) AToken: a unified tokenizer for vision. arXiv preprint arXiv:2509.14476. Cited by: Appendix E, §1, §3.
• [35] C. Ma, Y. Jiang, J. Wu, J. Yang, X. Yu, Z. Yuan, B. Peng, and X. Qi (2025) UniTok: a unified tokenizer for visual generation and understanding. arXiv preprint arXiv:2502.20321. Cited by: §A.4, §1, §2, Table 1, §3, §3.
• [36] I. Magnusson, A. Bhagia, V. Hofmann, L. Soldaini, A. H. Jha, O. Tafjord, D. Schwenk, E. Walsh, Y. Elazar, K. Lo, et al. (2024) Paloma: a benchmark for evaluating language model fit. Advances in Neural Information Processing Systems 37, pp. 64338–64376. Cited by: §2, §4.2.
• [37] A. Mishra, S. Shekhar, A. K. Singh, and A. Chakraborty (2019) OCR-VQA: visual question answering by reading text in images. In 2019 international conference on document analysis and recognition (ICDAR), pp. 947–952. Cited by: §2.
• [38] Y. Niu, M. Ning, M. Zheng, W. Jin, B. Lin, P. Jin, J. Liao, C. Feng, K. Ning, B. Zhu, et al. (2025) WISE: a world knowledge-informed semantic evaluation for text-to-image generation. arXiv preprint arXiv:2503.07265. Cited by: Table 2.
• [39] M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, et al. (2023) DINOv2: learning robust visual features without supervision. arXiv preprint arXiv:2304.07193. Cited by: §2.
• [40] X. Pan, S. N. Shukla, A. Singh, Z. Zhao, S. K. Mishra, J. Wang, Z. Xu, J. Chen, K. Li, F. Juefei-Xu, et al. (2025) Transfer between modalities with metaqueries. arXiv preprint arXiv:2504.06256. Cited by: §2.
• [41] L. Qu, H. Zhang, Y. Liu, X. Wang, Y. Jiang, Y. Gao, H. Ye, D. K. Du, Z. Yuan, and X. Wu (2025) TokenFlow: unified image tokenizer for multimodal understanding and generation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 2545–2555. Cited by: §3.
• [42] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. (2021) Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748–8763. Cited by: §2.
• [43] V. Ramanujan, K. Tirumala, A. Aghajanyan, L. Zettlemoyer, and A. Farhadi (2024) When worse is better: navigating the compression-generation tradeoff in visual tokenization. arXiv preprint arXiv:2412.16326. Cited by: §6.1.
• [44] T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen (2016) Improved techniques for training gans. Advances in neural information processing systems 29. Cited by: §2.
• [45] C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, et al. (2022) LAION-5B: an open large-scale dataset for training next generation image-text models. Advances in neural information processing systems 35, pp. 25278–25294. Cited by: §4.1.
• [46] F. Shi, Z. Luo, Y. Ge, Y. Yang, Y. Shan, and L. Wang (2025) Scalable image tokenization with index backpropagation quantization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 16037–16046. Cited by: §A.4, §1, Table 1, §3, §6.4.
• [47] M. Shukor, E. Fini, V. G. T. da Costa, M. Cord, J. Susskind, and A. El-Nouby (2025) Scaling laws for native multimodal models. arXiv preprint arXiv:2504.07951. Cited by: §2.
• [48] A. Singh, V. Natarajan, M. Shah, Y. Jiang, X. Chen, D. Batra, D. Parikh, and M. Rohrbach (2019) Towards VQA models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8317–8326. Cited by: Appendix D.
• [49] W. Song, Y. Wang, Z. Song, Y. Li, H. Sun, W. Chen, Z. Zhou, J. Xu, J. Wang, and K. Yu (2025) DualToken: towards unifying visual understanding and generation with dual visual vocabularies. arXiv preprint arXiv:2503.14324. Cited by: §3.
• [50] K. Sun, J. Pan, Y. Ge, H. Li, H. Duan, X. Wu, R. Zhang, A. Zhou, Z. Qin, Y. Wang, et al. (2023) JourneyDB: a benchmark for generative image understanding. Advances in neural information processing systems 36, pp. 49659–49678. Cited by: §4.1.
• [51] P. Sun, Y. Jiang, S. Chen, S. Zhang, B. Peng, P. Luo, and Z. Yuan (2024) Autoregressive model beats diffusion: llama for scalable image generation. arXiv preprint arXiv:2406.06525. Cited by: §3.
• [52] C. Team (2024) Chameleon: mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818. Cited by: Appendix E, §1, §2, §2.
• [53] M. L. Team, B. Xiao, C. Wang, C. Li, C. Zhang, C. Peng, H. Yu, H. Yang, H. Yan, H. Sun, et al. (2026) Longcat-next: lexicalizing modalities as discrete tokens. arXiv preprint arXiv:2603.27538. Cited by: §2.
• [54] K. Tian, Y. Jiang, Z. Yuan, B. Peng, and L. Wang (2024) Visual autoregressive modeling: scalable image generation via next-scale prediction. Advances in neural information processing systems 37, pp. 84839–84865. Cited by: §3.
• [55] S. Tong, E. Brown, P. Wu, S. Woo, M. Middepogu, S. C. Akula, J. Yang, S. Yang, A. Iyer, X. Pan, A. Wang, R. Fergus, Y. LeCun, and S. Xie (2024) Cambrian-1: a fully open, vision-centric exploration of multimodal llms. External Links: 2406.16860 Cited by: §1, §2.
• [56] S. Tong, D. Fan, J. Nguyen, E. Brown, G. Zhou, S. Qian, B. Zheng, T. Vallaeys, J. Han, R. Fergus, et al. (2026) Beyond language modeling: an exploration of multimodal pretraining. arXiv preprint arXiv:2603.03276. Cited by: §2.
• [57] W. Wang, F. Zhang, Y. Cui, H. Diao, Z. Luo, H. Lu, J. Liu, and X. Wang (2025) End-to-end vision tokenizer tuning. arXiv preprint arXiv:2505.10562. Cited by: §2.
• [58] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli (2004) Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing 13 (4), pp. 600–612. Cited by: §2.
• [59] J. Wu, Y. Jiang, C. Ma, Y. Liu, H. Zhao, Z. Yuan, S. Bai, and X. Bai (2025) Liquid: language models are scalable and unified multi-modal generators. International Journal of Computer Vision. Cited by: §A.1, Table 6, §C.2, Appendix D, §1, §2, §2, §4.1, Table 2.
• [60] J. Wu, D. Luo, W. Zhao, Z. Xie, Y. Wang, J. Li, X. Xie, Y. Liu, and X. Bai (2025) TokBench: evaluating your visual tokenizer before visual generation. arXiv preprint arXiv:2505.18142. Cited by: §2.
• [61] Y. Wu, Z. Zhang, J. Chen, H. Tang, D. Li, Y. Fang, L. Zhu, E. Xie, H. Yin, L. Yi, et al. (2024) VILA-U: a unified foundation model integrating visual understanding and generation. arXiv preprint arXiv:2409.04429. Cited by: §1, §2.
• [62] T. Xiong, J. H. Liew, Z. Huang, J. Feng, and X. Liu (2025) GigaTok: scaling visual tokenizers to 3 billion parameters for autoregressive image generation. arXiv preprint arXiv:2504.08736. Cited by: §A.4, §2, Table 1, §3, §6.1, §6.1.
• [63] A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. (2025) Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: §1, §4.1.
• [64] J. Yao, B. Yang, and X. Wang (2025) Reconstruction vs. generation: taming optimization dilemma in latent diffusion models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 15703–15712. Cited by: §1, §2, §6.1.
• [65] S. Yin, C. Fu, S. Zhao, K. Li, X. Sun, T. Xu, and E. Chen (2024) A survey on multimodal large language models. National Science Review 11 (12), pp. nwae403. Cited by: Appendix D.
• [66] J. Yu, X. Li, J. Y. Koh, H. Zhang, R. Pang, J. Qin, A. Ku, Y. Xu, J. Baldridge, and Y. Wu (2021) Vector-quantized image modeling with improved VQGAN. arXiv preprint arXiv:2110.04627. Cited by: §3, §3.
• [67] Q. Yu, M. Weber, X. Deng, X. Shen, D. Cremers, and L. Chen (2024) An image is worth 32 tokens for reconstruction and generation. Advances in Neural Information Processing Systems 37, pp. 128940–128966. Cited by: Appendix E, §1, §2, §3.
• [68] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang (2018) The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586–595. Cited by: §2, §3.
• [69] Y. Zhang, B. McKinzie, Z. Gan, V. Shankar, and A. T. Toshev (2024) Pre-trained language models do not help auto-regressive text-to-image generation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 1281–1287. Cited by: §5.1.
• [70] Y. Zhao, F. Xue, S. Reed, L. Fan, Y. Zhu, J. Kautz, Z. Yu, P. Krähenbühl, and D. Huang (2025) QLIP: text-aligned visual tokenization unifies auto-regressive multimodal understanding and generation. arXiv preprint arXiv:2502.05178. Cited by: §1, §2.
• [71] A. Zheng, X. Wen, X. Zhang, C. Ma, T. Wang, G. Yu, X. Zhang, and X. Qi (2025) Vision foundation models as effective visual tokenizers for autoregressive image generation. arXiv preprint arXiv:2507.08441. Cited by: §1, §2.
• [72] D. Zheng, M. Zhang, H. Li, K. Zou, H. Liu, Z. Guo, K. Feng, Y. Liu, Y. Luo, Y. Feng, et al. (2025) Architecture decoupling is not all you need for unified multimodal model. arXiv preprint arXiv:2511.22663. Cited by: §2.
• [73] L. Zheng, W. Chiang, Y. Sheng, T. Li, S. Zhuang, Z. Wu, Y. Zhuang, Z. Li, Z. Lin, E. P. Xing, et al. (2023) LMSYS-Chat-1M: a large-scale real-world LLM conversation dataset. arXiv preprint arXiv:2309.11998. Cited by: §4.1.
• [74] C. Zhou, L. Yu, A. Babu, K. Tirumala, M. Yasunaga, L. Shamis, J. Kahn, X. Ma, L. Zettlemoyer, and O. Levy (2024) Transfusion: predict the next token and diffuse images with one multi-modal model. arXiv preprint arXiv:2408.11039. Cited by: §2.
• [75] J. Zhu, W. Wang, Z. Chen, Z. Liu, S. Ye, L. Gu, H. Tian, Y. Duan, W. Su, J. Shao, et al. (2025) InternVL3: exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479. Cited by: Table 4, §4.1.
• [76] L. Zhu, F. Wei, Y. Lu, and D. Chen (2024) Scaling the codebook size of vq-gan to 100,000 with a utilization rate of 99%. Advances in Neural Information Processing Systems 37, pp. 12612–12635. Cited by: §2.

Appendix A Training Details

A.1 Data Proportion

We detail the data mixture in Table 4 and 5. The SFT data is similar to Liquid [59], but we substitute the text-to-image part with public data from pretraining and use more DVQA data.

We adopt 1:8 as the text:image-text ratio in continual pretraining. This choice is influenced by 1:2 in Liquid, while they use the Chameleon tokenizer with a sequence length of 1024. We observe that it is important to keep the total number of image tokens seen during training at the same level to achieve comparable performance, so we use 1:8 as the final ratio.

媒体内容 · 前往原文查看

Data Source	# Data (M)	Filtering	Recaptioning
DataComp-LM	6.6	Random Sampling	–
LAION-Aesthetics	42	Aesthetic Score (5.5)	InternVL3-1B [75]
JourneyDB	6.5*	–	GPT3.5
BLIP3o-Pretrain-Short-Caption	4.8	–	–

媒体内容 · 前往原文查看

Data Source	# Data (M)
Mini-Gemini (VQA)	1.7*
Mini-Gemini (Caption)	1.2
LMSYS-Chat	1.0
LAION-Aesthetics	0.86
JourneyDB	0.08
BLIP3o-60K	0.06

A.2 Prompts

For visual generation, we uniformly sample the prompt from the same set of prompts as Liquid during pretraining: “Generate an image based on this description.”, “Create an image that captures the provided description.”, “Based on the previous text, produce a corresponding image.”, “Please illustrate the above text with a picture.”, “Translate the given description into a image.”, “Construct a visual representation of the above description.”, “Create a image that matches the text.”, “Formulate a visual expression that reflects the narrative just provided.”, “Give a visual depiction based on the above sentences.”, “Create an image using the information mentioned above as guidance.”. During loss calculation, we only use “Generate an image based on this description.” Note that for unconditional visual generation, we use the format ⟨unconditional⟩⟨boi⟩{image tokens}⟨eoi⟩⟨eos⟩ without any prompt.

For captioning, we use “The caption of this image is:” as the prompt.

A.3 Model Performance

We list the model performance on visual generation and understanding benchmarks in Table 6 and provide qualitative image generation results on GenAI-Bench prompts in Figure 15. Our 8B model trained on public images with less pretraining data achieves similar performance to Liquid-7B on most benchmarks, which verifies that the framework serves as a reusable and reliable testbed for studying tokenizers’ impact on downstream unified training.

媒体内容 · 前往原文查看

Model	Data	GenAI$\uparrow$	MJHQ-30K$\downarrow$	WISE$\uparrow$	VQAv2$\uparrow$	GQA$\uparrow$	TextVQA$\uparrow$	POPE$\uparrow$	MME$\uparrow$
Liquid-7B	90M	0.72	5.47	0.41	68	56.1	40.4	81.1	1107.2
Ours (8B)	60M	0.73	10.55	0.38	67.3	54.44	43.46	78.1	1038.12

✓ Success “A lone lighthouse standing guard on a rocky coastline.”	✓ Success “A red rose in full bloom next to a pink rosebud in a garden.”	✓ Success “Two apples on a kitchen counter.”
✓ Success “A painting where the mountain is depicted as taller than the trees in the foreground.”	$\times$ Failure “A knight with a feather plume helmet by a stone tower.”	$\times$ Failure “Three cookies on a plate.”

A.4 Tokenizer Information

IBQ (Index Backpropagation Quantization) [46] modifies standard vector quantization in VQGAN to enable scalable training of large codebooks. Instead of updating only the selected code entries, IBQ backpropagates gradients to all codebook embeddings jointly with the visual encoder, which helps maintain high codebook utilization and a more consistent latent space between encoded features and code vectors. A variant of IBQ is adopted in Emu3.5 [9] with semantic loss and 131072 as the vocabulary size. The authors release IBQ variants with vocabulary sizes of 1024, 8192, 16384, and 262144 without semantic loss. We adopt the first three since 262144 is much larger than the original vocabulary size in the base model, which might cause training difficulty.

GigaTok [62] is proposed for autoregressive image generation and studies the effect of tokenizer scale and architectural design on discrete visual representations. It is based on VQGAN with large encoder–decoder networks and incorporates DINO-based semantic loss. It examines design choices such as asymmetric encoder–decoder scaling, different discriminator architectures, and alternative tokenization structures, and reports their impact on reconstruction behavior and downstream image generation. We revisit their conclusion on discriminator architectures using GigaTok-B-L under our framework.

UniTok [35] is a unified discrete image tokenizer designed to support both image generation and visual understanding. It is also based on a VQGAN and introduces multi-codebook quantization, which partitions latent features into multiple subspaces and discretizes each with independent codebooks to increase representational capacity while maintaining stable training. The tokenizer is trained with both reconstruction and semantic loss (contrastive loss with CLIP text encoder) so that the resulting tokens capture fine-grained visual details as well as high-level semantic information. We train a single-codebook version of UniTok with vocabulary size 16384 and latent dimension 64, and ablate the usage of semantic loss, examining its effect on downstream scaling under our framework. Following the original paper, we initialize the model with ViTamin-L/16 [6]. Due to compute constraints, we train the model on DataComp-medium [13] for five epochs and use 4096 as the global batch size. We adopt a smaller learning rate (lr=2e-4 for tokenizer and 5e-6 for discriminator) with entropy loss weight 0.05 and semantic loss weight 0 or 0.05. The rFID, zero-shot accuracy, and linear probing accuracy on ImageNet-1K of the two UniToks are shown in Table 7: UniTok-sem is stronger than UniTok in image classification but is weaker in image reconstruction.

媒体内容 · 前往原文查看

Model	rFID$\downarrow$	ZS Accuracy$\uparrow$	LP Accuracy$\uparrow$
UniTok	1.86	5.34	15.36
UniTok-sem	2.23	46.79	59.42

Following the official code of UniTok, we adopt a zooming-then-cropping strategy for image preprocessing for UniTok and UniTok-sem, which is different from other tokenizers. This strategy is also applied in the pre-tokenization process in our training. Since we do not compare tokenizers from different families, this does not affect our conclusions.

A.5 Compute Resources

The experiments were conducted on an internal cluster using 1, 2, 4, or 8 NVIDIA H200 GPUs with 141GB memory. The estimated compute for continual pretraining is listed in the result plots. The full research project also involved preliminary experiments for adjusting the training recipe and hyperparameters.

Appendix B Supplementary Results on Training Dynamics

媒体内容 · 前往原文查看

Tokenizer	Text		T2I	Image
GigaTok	5.46e-04	1.96e-02	6.92e-03	5.15e-03
GigaTok-DINO	1.21e-04	-7.73e-03	3.66e-03	2.54e-03
IBQ-1024	-1.63e-03	1.11e-02	2.16e-02	1.57e-02
IBQ-8192	-2.24e-03	6.94e-03	1.21e-02	1.08e-02
IBQ-16384	-2.17e-03	8.35e-03	1.15e-02	9.01e-03
UniTok	-1.59e-03	1.01e-02	1.26e-02	1.15e-02
UniTok-sem	-2.01e-03	2.19e-02	1.13e-02	1.22e-02

媒体内容 · 前往原文查看

媒体内容 · 前往原文查看

We try quadratic fitting for the loss-data relationship in Figure 2 for all tokenizers and record the quadratic coefficients in Table 8: Most data scaling curves show convexity for LAION-I2T, LAION-T2I, and LAION-Image loss, but a slight concavity for text loss. Considering that the data scaling dots are checkpoints from the same runs, this implies different training dynamics across tasks. In Figure 16, we present the loss curves of a 4B model training including earlier checkpoints, and find the curve shapes aligned with the finding on quadratic coefficients. We hypothesize that the image-related modeling might dominate optimization in the early phase but slowly saturate later, giving rise to the text loss decaying rate.

We look into the gradient norm during a single run and check how gradient norm per modality changes over time in continual pretraining with GigaTok and IBQ-16384. In Figure 17, we find that the T2I-related gradient is larger than the text gradient at the beginning, but gradually decays throughout the training. The text gradient decays at a slower rate and reaches a similar level to the T2I gradient at the end. This supports the hypothesis about the convexity and concavity we observe in data scaling curves.

Appendix C Supplementary Loss Results

C.1 Loss over Different Image Sources

We observe very similar T2I loss and image loss trends across three data sources in Figure 18. The I2T loss trends are also similar between JourneyDB and BLIP3o-Short-Caption, but are slightly different from LAION-I2T, where (lr=3e-5, bs=512) surpasses other settings. We hypothesize that this is due to their different caption styles: recaptioned LAION-Aesthetics has longer and more detailed captions than the other two. Therefore, it would rely more on the models’ language ability.

媒体内容 · 前往原文查看

C.2 Loss Scaling Results for Different Hyperparameters

媒体内容 · 前往原文查看

To study the effect of hyperparameters on model fitting, we experiment with a 2×3 grid of (lr, bs) using GigaTok. In Figure 19, we observe that no hyperparameter setting dominates all tasks: Aggressive hyperparameter settings (high lr, small bs, represented by (1e-4,512)) are favored on LAION-I2T, LAION-T2I, and LAION-Image, but lags on text loss. Mild hyperparameters represented by (3e-5, 2048) maintain the property of the base model with better text fitting. Such conflict in hyperparameter preference echoes observations on the trade-off between maintaining the pure-text ability of the base model and accommodating it for image-related tasks [59].

C.3 Annealed Loss Results

媒体内容 · 前往原文查看

We plot the loss-FLOPs results after annealing along with the non-annealed results in Figure 20. On all tasks, the relative ranking among hyperparameters is barely changed after annealing compared with results before annealing in Figure 19. One notable exception is that lr=1e-4 has a greater loss decay during annealing: In LAION-I2T, (lr=3e-5, bs=512) reaches the lowest loss at the end of the constant learning rate stage, but is outperformed by (lr=1e-4, bs=512) after annealing. We observe that annealing mainly reduces the text loss but does not affect the image-related losses significantly. One hypothesis is that the loss on image tokens might lead to noisier or smaller gradients during the annealing stage than the loss on the text tokens.

C.4 Loss-VQA Performance on TextVQA

媒体内容 · 前往原文查看

We study the relationship between pretraining loss and a more specific VQA task: TextVQA. In Figure 21, I2T loss is negatively correlated for 0.6B models, but shows the opposite trend when we scale up the model size to 4B. As a side finding, we observe that optimal hyperparameters vary for different benchmarks: Aggressive hyperparameters win on VQAv2 and GQA, but mild hyperparameters lead to better TextVQA performance despite higher I2T loss. These results suggest that post-SFT understanding performance sometimes depends on additional capabilities beyond pretraining caption fit, such as OCR. On the other hand, treating the average score on several VQA benchmarks as the visual understanding performance might omit their distinct properties and could be highly sensitive to benchmark choices.

C.5 Details on Loss Normalization

For LAION-T2I-force and LAION-Image-force, we first set the output logits for non-image tokens to -inf between ⟨boi⟩ and ⟨eoi⟩. This ensures that the model can only choose from the image tokens. Then, we normalize the loss (log perplexity) by log vocabulary size. Empirically, we find that the effect of the first step (forcing image output) is negligible for the loss values, indicating that the model has learned to output image tokens between ⟨boi⟩ and ⟨eoi⟩ on the validation set even at the first checkpoint plotted in the results (trained with 12M data).

Note that we do not apply the same normalization to the text loss and I2T loss even though the total vocabulary sizes are not identical for the three IBQ variants. The reason is that we also find the first step (forcing text output) has a negligible effect on the loss values, which shows that the model only chooses from text tokens during text and I2T evaluation on the validation set. Hence, we do not normalize the losses by the total vocabulary size.

Comparison with empirical code entropy. Normalizing by ${\log }_{2}B$ assumes the maximal entropy of the image vocabulary, whereas real codebooks are used unevenly. To test whether ${\log }_{2}B$ is an adequate stand-in for the actual code distribution, we compare it with the empirical unigram entropy $H_1$ measured on the validation image tokens of all seven tokenizers (Table 9). $H_1$ lies within $1.3\%$ of ${\log }_{2}B$ for every tokenizer. Uneven code usage does occur: GigaTok’s rank-frequency distribution is visibly skewed, with head codes used roughly $40$ times as often as tail codes. This skew is nevertheless confined to a small fraction of code ranks and carries little probability mass, so GigaTok’s entropy stays close to ${\log }_{2}B$. The larger deficits of IBQ-16384 and UniTok-sem instead reflect lower utilization across a broader portion of the codebook.

媒体内容 · 前往原文查看

Tokenizer	${\log }_{2}B$	$H_1$	$H_1/{\log }_{2}B$
GigaTok	14.000	13.953	0.997
GigaTok-DINO	14.000	13.949	0.996
IBQ-1024	10.000	9.977	0.998
IBQ-8192	13.000	12.947	0.996
IBQ-16384	14.000	13.841	0.989
UniTok	14.000	13.859	0.990
UniTok-sem	14.000	13.821	0.987

Substituting $H_1$ for ${\log }_{2}B$. We then relate the normalized T2I loss to GenAI-all at the 0.6B scale for the three IBQ variants under both normalizations (Table 10). The two choices give nearly identical results: both recover a strong loss–performance relation, whereas the unnormalized loss explains essentially none of the benchmark variance and tokenizer identity accounts for most of it.

媒体内容 · 前往原文查看

Normalization	Pearson $r$	$R^2$ from loss	Additional $R^2$ from tokenizer identity
None	$+0.016$	0.000	0.972
Divide by ${\log }_{2}B$	$-0.957$	0.915	0.048
Divide by $H_1$	$-0.953$	0.909	0.055

Scope of the normalization. This normalization corrects the loss scale associated with vocabulary size; it does not calibrate losses across tokenizer architectures at a fixed $B$, which is consistent with the cross-tokenizer clustering in Figure 5. Our experiments also do not cover tokenizers with severe or complete codebook collapse; in such a regime the nominal vocabulary size would no longer represent the effective prediction space, and ${\log }_{2}B$ normalization may not be applicable.

C.6 Ablation on I2T Objective

媒体内容 · 前往原文查看

To verify the effect of the I2T objective in joint training, we ablate it by reformatting the I2T samples into T2I order (with drop rate $10\%$ which enables CFG) and train with only Text+T2I data with the same hyperparameter settings. This experiment setting resembles the ablation in Section 6.2 for studying the effect of the T2I objective.

In Figure 22, we observe that the gap between tokenizers on text and T2I is not reduced by ablating I2T data. Notably, the text loss increases for both UniTok and UniTok-sem when the I2T objective is ablated. This suggests that the I2T objective complements, rather than competes with, text modeling during joint training. As a comparison, the T2I loss slightly decreases when there is no I2T task.

Appendix D Supplementary Benchmark Information and Results

(1) Text-to-image generation. Following Liquid [59], we report (1) VQAScore on GenAI-Bench [25], calculated by prompting CLIP-FlanT5-XXL with whether the image aligns with the text and recording the probability of answering “Yes” (hence ranging from 0 to 1), and (2) gFID on MJHQ-30K [26].During inference, we use unfiltered softmax sampling and CFG scale=7.0. We adopt a unified template “{text} Generate an image based on this description.⟨boi⟩” for evaluation. Note that we force the model output to be an image token by setting the probability of text tokens to zero during sampling. Without this operation, we find that an under-optimized unified model could output mixed text tokens and image tokens, which even holds for larger public models like Liquid-7B. Some prompts and generated results are in Figure 23. The images are generated by the Qwen3-0.6B model trained with lr=1e-4, bs=512 on 60M data. The qualitative results align with the reported benchmark scores, as GigaTok and GigaTok-DINO produce images with the best quality and alignment, followed by UniTok and UniTok-sem with worse quality but good alignment.

(2) Visual understanding. We use VQAv2 [16] and GQA [21] as two general VQA benchmarks. TextVQA [48] is included in Appendix C.4 as a specific VQA benchmark. We also test the models on POPE [30] and MME [65] following Liquid, but find large variance among runs with different shuffling seeds (see Table 11 and analysis below). Therefore, we exclude them from our analysis.

Benchmark sensitivity to random seeds. All central experiments in this work use a single data-shuffling seed (42), which controls the ordering of training data while the language-model initialization is unchanged. To quantify how much of the benchmark spread this ordering accounts for, we additionally ran SFT on one GigaTok configuration (0.6B model, lr=1e-4, bs=1024) with data-shuffling seed 37 and compared the two runs (Table 11). VQAv2, GQA, GenAI-all, and MJHQ-30K agree to within $1.9\%$ relative difference, whereas POPE changes by $7.07\%$ and MME-P by $4.98\%$. This quantitatively motivates excluding POPE and MME-P from the loss-correlation analysis: their data-order sensitivity is substantially larger than that of the benchmarks we do analyze. We note that this is a two-seed comparison for a single GigaTok configuration rather than a multi-seed replication of every tokenizer comparison due to high computational cost.

媒体内容 · 前往原文查看

Benchmark	Seed 42	Seed 37	Abs. diff.	Rel. diff.
VQAv2$\uparrow$	51.43	51.47	0.04	0.08%
GQA$\uparrow$	43.11	42.71	0.40	0.93%
POPE$\uparrow$	61.40	65.90	4.50	7.07%
MME-P$\uparrow$	825.56	785.48	40.08	4.98%
GenAI-all$\uparrow$	0.710	0.700	0.010	1.42%
MJHQ-30K gFID$\downarrow$	9.2558	9.0812	0.1746	1.90%

Appendix E Limitations

First, our study is limited to seven discrete image tokenizers with fixed token length and single-codebook designs in consideration of a controlled experiment setting. For example, on the sequence length ($K$) axis, we observe that the mixed-modal training is sensitive to the number of image tokens seen. Since it is mathematically impossible to simultaneously hold both the number of image tokens seen and the number of images seen constant when $K$ varies, such a study would introduce confounding variables. We leave the study of tokenizers with sub-codebooks [34], larger input resolutions [52], and different sequence lengths [67] to future research.

Second, we use pretraining validation loss as a lens for studying joint modeling behavior, not as a replacement for downstream evaluation after SFT or for tokenizer selection. Although I2T loss provides a useful signal, it is less predictive for specialized tasks such as TextVQA.

Third, due to the high computational cost, we do not conduct a full scaling law study on unified multimodal AR training that starts from scratch with more model scales or derive compute-efficient frontiers. Future work could derive and compare image tokenizers’ effects on the compute-efficient frontier of multimodal AR models.

Fourth, analyses on the validation set, including PMI and $n$-gram entropy, are diagnostic rather than mechanistic, and high-order empirical entropy can suffer from finite-sample bias.

Fifth, our empirical findings are established for Qwen3-based pure-AR unified models. Holding the backbone family fixed is itself a control in our study, since it lets us vary the visual token space while keeping the downstream architecture and training recipe consistent. We therefore do not claim that every loss relationship or tokenizer ranking is unchanged under a different backbone family; validating them would require matched retraining and evaluation across tokenizers under that backbone. The analysis framework itself can be applied to another language-model family, and we view such cross-family validation as an important direction for future work.

Finally, our downstream evaluation covers a limited set of generation and VQA benchmarks; broader task, safety, and human-preference evaluations are needed to fully evaluate the models. In particular, our understanding evaluation covers general VQA (VQAv2 and GQA) and the OCR-oriented TextVQA, but includes no dedicated region-level grounding or fine-grained spatial-reasoning benchmark. We therefore do not claim that our findings extend to these specialized visual capabilities, and evaluating how the image token space affects them remains future work.

Appendix F Third-Party Attribution Notices

Portions of this work use third-party software released under the MIT License: we train UniTok and UniTok-sem variants using the official UniTok code (https://github.com/FoundationVision/UniTok), and we use the GigaTok and GigaTok-DINO tokenizers (https://github.com/SilentView/GigaTok). The copyright and permission notices below are reproduced as required by that license.

Copyright (c) 2024 FoundationVision

Copyright (c) 2025 GigaTok: Scaling Visual Tokenizers to 3 Billion Parameters for Autoregressive Image Generation Authors

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Appendix G Broader Impacts

This work studies image tokenizers in the context of unified pure-AR multimodal models. By showing how tokenizer design affects modeling, including cross-modal modeling and text-side behavior, our analysis may help researchers design and diagnose multimodal training beyond reconstruction or single-axis evaluation. Improved tokenizer analysis could contribute to stronger generation and understanding models, which may support creative, educational, and accessibility applications.

However, stronger multimodal models also raise risks such as biased outputs, misleading synthetic content, and misuse. Our experiments use publicly available images and do not introduce a deployed system, but future work using these insights should be paired with standard safety, bias, privacy, and responsible-release evaluations.