# Intel 论文提出 BITCOS 布局，三元大语言模型突破 1.58 比特存储障碍

- 来源：Hacker News 热门（buzzing.cc 中文翻译）
- 作者：matt_d
- 发布时间：2026-09-17 07:43
- AIHOT 分数：50
- AIHOT 链接：https://aihot.news/items/cmu4r4xwz04v6rokc5u15o7nr
- 原文链接：https://arxiv.org/abs/2609.16338

## AI 摘要

Intel 研究人员提出 BITCOS，一种由存在位图加压缩符号向量组成的三元权重布局，按零密度 z 计每权重仅需 2−z 比特。实测 29 个 SOTA 三元模型的零密度为 29.7%–51.5%，其中 26 个在 BITCOS 下比五 trit 打包存储更紧凑，最稀疏模型达 1.485 比特每权重。

## 正文

Breaking the 1.58-bit Barrier for Ternary LLMs

Evangelos Georganas, Alexander Heinecke, Pradeep Dubey

Intel Corporation

Abstract

Ternary Large Language Models (LLM) store every weight as one of three symbols {−1,0,+1}, so the cost of a ternary model is conventionally referenced to the information-theoretic log2⁡3≈1.585 bits per weight. The prevailing deployment format packs five ternary weights into one byte (five-trit packing), and due to the power-of-two group sizes used in practice this rounds up to 1.625 bits per weight. This effective storage bit-width treats the three symbols {−1,0,+1} as equiprobable. We measure the actual symbol distribution of 29 ternary LLM models and find that zeros account for up to 51.5% of all weights. Motivated by this finding, we introduce BITCOS, a simple distribution-adaptive layout comprised of a dense presence bitmap plus a compacted sign vector, and costs 2−z bits per weight element given a zero density z in the model’s weights. BITCOS stores weights more compactly than the five-trit packing in 26 of the 29 tested models, and reaches 1.485 bits per weight on the sparsest of them. BITCOS is amenable to efficient unpacking on modern processors and GPUs, and we present optimized unpacking sequences for AVX-512, AVX2 and Intel Xe2 GPUs. Measured against production state-of-the-art ternary matrix-vector multiplication kernels, at the zero densities real-world ternary models exhibit, the realized gain with our proposed layout is up to 1.28×. Finally, we illustrate end-to-end LLM inference results on 5 different platforms (client and server CPUs, integrated and discrete Xe2 GPUs) where decode throughput improves by up to 1.18× on CPUs and 1.27× on GPUs.

I Introduction

Ternary weight quantization restricts every weight to {−1,0,+1} scaled by a group-wise factor [1, 2]. Three equiprobable symbols carry log2⁡3≈1.585 bits of information, and this theoretical bound is the reference for ternary storage. Nevertheless, in practice the actual storage cost is determined by how those symbols are packed. Five trits (ternary digits) fit in a byte (35=243≤256), which approaches the bound at 8/5=1.6 bits per weight. However, deployed implementations quantize in blocks of power-of-two weights like 128, and 128 is not a multiple of 5: a block needs ⌈128/5⌉=26 payload bytes, so the rate stored in practice is 26×8/128=1.625 bits per weight. Still, this effective storage bit-width treats the three symbols {−1,0,+1} as equiprobable.

媒体内容 · 前往原文查看

Fig. 1: Effective bit-widths for various families of ternary LLM models, comparing the five-trit packing and the proposed BITCOS layout. The sloped line represents the BITCOS bit-width 2−z, while the plateau indicates the fixed five-trit bit-width of 1.625 bits per weight. Each marker corresponds to a model from Table I, placed at its measured zero density. Models with zero density z above 0.375 benefit from the BITCOS layout (green plot area), as such 26 out of the 29 models achieve lower effective bit-width with BITCOS.

媒体内容 · 前往原文查看

TABLE I: Effective bit-widths for SOTA ternary LLM models. “% 0” is the measured zero density z. “Symbols” column counts the ternary codes alone and “+scale” adds the measured 16-bit scale overhead. The symbols-only rate is 2−z for BITCOS and for every model 2.000 and 1.625 bits per weight for 2-bit and 5-trit packing respectively. “red.” is the size reduction of BITCOS over the corresponding format with scales included. A value >1 means BITCOS stores the model more compactly.

BITCOS 2-bit packing 5-trit per byte

Ternary model % 0 Symbols +scale +scale red. +scale red.

∙ BitNet b1.58 2B4T 42.19 1.578 — 2.000 1.27× 1.625 1.03×

■ Bonsai 1.7B 39.89 1.601 1.726 2.125 1.23× 1.750 1.01×

Bonsai 4B 37.71 1.623 1.748 2.125 1.22× 1.750 1.00×

Bonsai 8B 38.25 1.618 1.743 2.125 1.22× 1.750 1.00×

Bonsai 27B 29.66 1.703 1.828 2.125 1.16× 1.750 0.96×

▲ CAT-Q Qwen3-1.7B 51.48 1.485 1.610 2.125 1.32× 1.750 1.09×

CAT-Q Qwen3-8B 46.70 1.533 1.658 2.125 1.28× 1.750 1.06×

CAT-Q Qwen3-30B-A3B 32.88 1.671 1.796 2.125 1.18× 1.750 0.97×

CAT-Q Qwen3-32B 47.11 1.529 1.654 2.125 1.28× 1.750 1.06×

CAT-Q Qwen3-235B-A22B 34.07 1.659 1.784 2.125 1.19× 1.750 0.98×

⧫ ParetoQ 125M 41.07 1.589 1.613 2.023 1.25× 1.648 1.02×

ParetoQ 350M 41.58 1.584 1.598 2.014 1.26× 1.639 1.03×

ParetoQ 600M 43.27 1.567 1.579 2.012 1.27× 1.637 1.04×

ParetoQ 1B 44.18 1.558 1.569 2.010 1.28× 1.635 1.04×

ParetoQ 1.5B 47.10 1.529 1.537 2.008 1.31× 1.633 1.06×

▼ TriLM 99M 40.97 1.590 1.618 2.027 1.25× 1.652 1.02×

TriLM 190M 40.67 1.593 1.611 2.018 1.25× 1.643 1.02×

TriLM 390M 40.79 1.592 1.606 2.014 1.25× 1.639 1.02×

TriLM 560M 40.73 1.593 1.604 2.011 1.25× 1.636 1.02×

TriLM 830M 40.54 1.595 1.604 2.009 1.25× 1.634 1.02×

TriLM 1.1B 40.39 1.596 1.605 2.009 1.25× 1.634 1.02×

TriLM 1.5B 40.21 1.598 1.606 2.008 1.25× 1.633 1.02×

TriLM 2.4B 39.70 1.603 1.610 2.007 1.25× 1.632 1.01×

TriLM 3.9B 38.70 1.613 1.618 2.005 1.24× 1.630 1.01×

★ Maple 20B-A1B 40.67 1.593 1.609 2.016 1.25× 1.641 1.02×

◀ BitCPM-CANN 0.5B 37.67 1.623 1.636 2.012 1.23× 1.637 1.00×

BitCPM-CANN 1B 38.39 1.616 1.623 2.006 1.24× 1.631 1.01×

BitCPM-CANN 3B 38.06 1.619 1.624 2.005 1.23× 1.630 1.00×

BitCPM-CANN 8B 39.30 1.607 1.610 2.003 1.24× 1.628 1.01×

We measure the actual symbol distribution of 29 state-of-the-art (SOTA) ternary LLM models and find that zeros account for up to 51.5% of all weights; Table I lists the measured zero density of every model. More specifically we benchmarked seven ternary model families: i) BitNet [1, 2], the 2B-parameter model trained from scratch on 4T tokens that established the 1.58-bit reference point; ii) Bonsai [3], four dense checkpoints from 1.7B to 27B; iii) CAT-Q [4], five post-training quantizations of Qwen3 from 1.7B to 235B, including two mixture-of-experts models; iv) ParetoQ [5], five small checkpoints (125M–1.5B) from a study of low-bit quantization-aware training; v) TriLM [6], the nine-model Spectra suite (99M–3.9B) pretrained in ternary; vi) Maple [7], a 20B-A1B ternary mixture-of-experts reasoning model; and vii) BitCPM-CANN [8], four ternary checkpoints (0.5B–8B) trained from scratch. Motivated by the finding that the zero density of many ternary models is significantly higher than the density of either of the other two codes {−1,+1}, we introduce BITCOS (BITmap and COmpacted Signs): a simple distribution-adaptive layout for ternary tensors. Given a zero density z, BITCOS spends one presence bit on every weight entry and one sign bit only on the non-zero weights, thus the layout effectively achieves 2−z bits per weight. Such a layout improves the effective bit-width compared to the deployed five-trit packing once z>0.375 and is strictly more efficient than the widely-adopted 2-bit packing for all zero densities. Figure 1 places every model of Table I at its measured zero density, on whichever of the two rates it meets first: the sloped 2−z line when the BITCOS sparse layout is more efficient, or the 1.625 plateau when the five-trit fixed-rate packing has lower bit-width. Models with zero density z above 0.375 benefit from the BITCOS layout (green plot area), as such 26 out of the 29 models achieve lower effective bit-width with BITCOS compared to the five-trit packing layout.

However, the effective bit-width is only one aspect of the overall inference. The actual performance of ternary models during inference also depends on how efficiently the packed weights can be unpacked and utilized in matrix-vector multiplication kernels. The decode phase of LLM inference with a small batch-size is bandwidth-bound, so the time per token tracks the bit-width of the weight datatype [9], which is precisely the regime that on-device and agentic deployments exercise [10]. BITCOS not only reduces the storage cost but also is amenable to efficient unpacking and computation on modern CPUs and Intel GPUs. Measured end to end over 7 ternary LLM checkpoints and five platforms, decode throughput improves by up to 1.18× on a 64-core server CPU, 1.15× on a 24-core client CPU, 1.27× on a discrete Xe2 GPU and 1.22× on an integrated Xe2 GPU. The performance gain however is not universal: on an 8-core, bandwidth-rich client platform with enough bandwidth per core to leave the unpack sequence exposed, the smaller payload does not translate into performance benefits. Therefore, we develop a simple two-term roofline model to assess the limitations of the BITCOS-based kernels.

A novel ternary sparse layout (which we name BITCOS) comprising of a presence bitmap plus a compacted sign vector, that costs 2−z bits per weight for a zero density z and is amenable to efficient unpacking on modern CPUs and GPUs.

Optimized instruction sequences to unpack the proposed BITCOS layout and perform matrix multiplication operations efficiently on modern x86 CPUs (client CPUs with performance and efficiency cores, and server CPUs with high core counts) and Intel Xe2 GPUs (both integrated and discrete GPUs).

A roofline model to assess the efficacy and the limitations of our CPU microkernels.

Performance evaluation of the proposed layout with microbenchmarks and end-to-end LLM inference over seven ternary LLM checkpoints on modern server/client CPUs and integrated/discrete GPUs, illustrating decode throughput improvements by up to 1.18× on CPUs and 1.27× on GPUs.

II The BITCOS Layout: BITmap + COmpacted Signs

II-A Layout definition and storage cost

We propose the BITCOS layout that leverages the inherent unstructured sparsity of ternary weights and stores a ternary tensor as two pieces:

a presence bitmap of one bit per weight, set where the weight is non-zero; and

a sign vector of one bit per non-zero weight, in tensor order.

Figure 2 shows both pieces on a small 8×8 tensor. The bitmap has the full tensor length; the sign vector is compacted to the population count of the bitmap. Assuming a zero density of z, the cost per weight is therefore:

B⁡(z)= 1+(1−z)= 2−zbits (1)

媒体内容 · 前往原文查看

Fig. 2: The BITCOS layout on an 8×8 tensor. Each column of BITCOS is one mask whose bit r records whether row r is non-zero, so a decoder reads presence for a whole block in a single load. The compacted sign vector carries one bit per non-zero only, in the same (k,r) order, so the j-th sign bit belongs to the j-th set bit of the bitmap. Zeros consume a bitmap bit and nothing else, which is why the effective rate is 2−z. In this example, 30 out of the 64 weights are zero (z=0.469), so the tensor costs 64 presence plus 34 sign bits, or 1.531 bits per weight (which is below log2⁡3).

The cost per weight falls linearly in z, so the BITCOS format yields substantial benefits over 2-bit or five-trit packing at zero-heavy distributions. Table I compares the per-weight cost of different formats over the 29 measured checkpoints. Column “% 0” is the measured zero density z, column “Symbols” counts the ternary codes alone and the “+scale” columns add the measured 16-bit scale overhead of the corresponding model. The “red.” columns report the size reduction of BITCOS over the 2-bit and the five-trit per byte packing respectively; a value greater than 1 means BITCOS stores the model more compactly than that format. We conclude that BITCOS improves the effective bit-width compared to the deployed five-trit packing once z>0.375 (26 out of 29 ternary LLM models) and is strictly more efficient than the widely-adopted 2-bit packing for all zero densities.

II-B BITCOS unpack sequence with x86 AVX-512 instructions

The BITCOS layout reconstructs 16-bit weights in a short mask-driven sequence assuming the target ISA supports masks. In Figure 3 we illustrate such an exemplary sequence with AVX-512 instructions. Two vectors are prepared once per group of 128 reduction elements and then reused for every one of its iterations: zmm3 holding the 32 fp16 group scales, one per row, and zmm4 holding a copy with the sign bit set, obtained by a single vporq against a broadcast 0x8000. Because the sign of an IEEE half lives in the most significant bit and the group scales are non-negative by construction, that OR is an exact negation, so zmm3 and zmm4 hold +s and −s respectively for each of the 32 rows. The per-iteration work is then:

Load the presence mask. Read 32 bitmap bits into a general register and then into k1.

Place the signs. Take the next 32 bits of the sign stream and scatter them via pdep into the bit positions that k1 marks as present. The sign vector is stored compacted, so its j-th bit belongs to the j-th non-zero of the group’s bitmap. pdep is exactly the scatter that undoes the compaction (see Figure 4).

Select. A zero-masked move of zmm3 under k1 puts +s at every present lane and exact +0 elsewhere; a merge-masked move of zmm4 under the deposited mask overwrites the negative ones.

Figure 4 illustrates what that deposit does: pdep takes the low bits of its source in order and drops them at the positions the mask selects, leaving every unselected position zero. In the AVX-512 assembly, r13 walks the bitmap, r9 is the base of the sign stream, r8 the running bit position within it, r14 walks the activations, and zmm2 is one accumulator for the fused multiply-add (FMA) operation. In total we get 17 instructions, of which the weight unpacking portion corresponds to 3 instructions: one pdep and 2 masked moves. The remaining 14 are not unpacking: one loads the bitmap word, 5 compute the data-dependent address of the sign window, one shrx performs the unaligned 64-bit read and its alignment in a single operation, 2 are software prefetches, 2 advance the bit position by the population count, 2 move masks, and the last is the fused multiply-add (FMA), which absorbs the activation broadcast as an embedded operand.

媒体内容 · 前往原文查看

Fig. 3: AVX-512 instruction sequence to unpack the BITCOS layout for one 32-row block.

媒体内容 · 前往原文查看

Fig. 4: pdep parallel bit deposit. In this example, only the low four bits of a are consumed, one per set bit of the mask. The sign stream stores one bit per non-zero weight, so undoing that compaction means scattering those bits onto the set positions of the presence mask, which is the instruction’s definition.

II-C BITCOS unpack sequence with x86 AVX2 instructions

The sequence of Section II-B depends on two AVX-512 facilities: mask registers, which make a 32-bit presence pattern directly usable as a predicate, and FP16 compute instructions. The client CPUs we target support neither, thus we implemented a second kernel targeting the AVX2 ISA with AVX-VNNI-INT8 compute capabilities, which reconstructs the ternary weights {−1,0,+1} as int8 and contracts against int8 activations (see Figure 5).

For each of the 32 rows, let pi∈{0,1} indicate that the weight is present, and let ni∈{0,1} indicate that the present weight is negative. The bitmap supplies the pi bits. After pdep returns the compact signs to their row positions, its result supplies the ni bits. Since AVX2 has no mask registers, we materialize each presence and sign vector as byte masks Pi=−pi and Ni=−ni: a true lane is 0xFF (−1 as a signed byte), and a false lane is zero. The desired ternary byte is then

wi=pi−2​ni. (2)

In the example of Figure 2, the column k2 holds the ternary values (+1,+1,+1,+1,−1,0,−1,+1) for rows i=0​…​7, so the bitmap word for that column is 𝟶​𝚡​𝙳​𝙵 and the two predicates are

p=(1,1,1,1,1,0,1,1),n=(0,0,0,0,1,0,1,0). (3)

Equation (2) returns the unpacked column exactly: rows 0–3 and 7 give 1−0=+1, rows 4 and 6 give 1−2=−1, and the absent row 5 gives 0−0=0. Materialized as bytes these are P=(𝙵𝙵,𝙵𝙵,𝙵𝙵,𝙵𝙵,𝙵𝙵,𝟶𝟶,𝙵𝙵,𝙵𝙵) and, after the mask with 0xFE, T=(𝟶𝟶,𝟶𝟶,𝟶𝟶,𝟶𝟶,𝙵𝙴,𝟶𝟶,𝙵𝙴,𝟶𝟶), so vpsubb leaves (𝟶𝟷,𝟶𝟷,𝟶𝟷,𝟶𝟷,𝙵𝙵,𝟶𝟶,𝙵𝙵,𝟶𝟷), which is the original column as int8. Note that n is not what the format stores. Only the popcount⁡(p)=7 sign bits (0,0,0,0,1,1,0) are stored, so the sign of row 6 is the sixth stored bit rather than the seventh. Recovering the n of (3) from those seven bits is exactly the pdep of Figure 5.

媒体内容 · 前往原文查看

Fig. 5: AVX2 instruction sequence to unpack the BITCOS layout for one 32-row block.

媒体内容 · 前往原文查看

Fig. 6: Xe2 instruction sequence to unpack the BITCOS format. Register names are shortened and independent operations are grouped by function rather than scheduler order. Address setup, predicate formation, and register repacking are omitted. The 16 SIMD lanes correspond to different output columns, so one four-row group produces 4×16 weights. Section II-D walks through the sequence.

This alternative contraction algorithm has two implications. First, arithmetic moves from fp16 to integer: activations are quantized to int8 per group of 128, the dot product accumulates in int32, and the group scales are applied once per group instead of being blended into the weights. The weights are therefore reconstructed as the values ±1 rather than ±s, and the layout is stored in VNNI4 order so that each vpdpbssd (AVX-VNNI-INT8 compute) consumes four consecutive reduction elements for each of eight rows. Second, and more consequentially, the absence of mask registers means the presence pattern must be materialized as one byte per lane before it can select anything. That expansion consists of a broadcast, an in-lane shuffle and a compare against the bit-select constant, and it costs 5 instructions where AVX-512 spends merely 1 mask move instruction kmovd. Once those byte masks exist, the last two instructions implement Eq. (2): masking Ni with 0xFE forms Ti=−2​ni, and the vpsubb P,T,W computes Wi=Ti−Pi=pi−2​ni. Thus a positive present lane becomes +1, a negative present lane becomes −1, and an absent lane remains zero.

II-D BITCOS unpack sequence for Intel Xe2 GPUs

Both x86 sequences rely on pdep to scatter compact signs back to the rows marked present. Xe2 has no corresponding instruction, so the GPU kernel replaces that scatter with a small lookup table in shared local memory (SLM). We implement the kernel with the XeTLA templates [11], and use the conventional SYCL terminology of workgroups and subgroups throughout [12]. Figure 6 illustrates the assembly sequence for the Xe2 kernel.

In the assembly sequence of Figure 6, one load.ugm.d32x3 fetches the three consecutive sign words sufficient for two 32-row blocks. For each block, the contiguous bitmap load load_block2d.ugm bmp supplies one 32-bit presence word per output column. Two bfn instructions, the three-input bitwise Boolean operation of Xe2, select the adjacent low/high sign words, and an and, two shifts and an or align them at the current bit rank. The and reduces the rank to the in-word offset b, the shr aligns the low word by b and the shl brings in the b high bits that cross the word boundary. That last shift needs a count of 32−b, so the complementary count is taken modulo 32 and the or is predicated off in the one case the wrap gets wrong, b=0, where the high word contributes nothing. The block-wide cbit advances that rank by the number of nonzeros in all 32 rows. Each block is then decoded as eight four-row groups. A group extracts one presence nibble (4 bits), combines it with the next 4 bits of the aligned sign window, and uses the resulting byte offset for one SIMD16 load.slm.d32x2. The lookup returns four fp16 ternary codes per lane. The group-level cbit advances the sign window by only the bits actually consumed, an fp16 multiply applies the group scale, and one DPAS contraction is issued after four groups have supplied 16 reduction rows. The sequence touches two memories, load.ugm for the presence bitmap and sign vector in global memory and load.slm for the table in SLM.

The lookup table in SLM is indexed by an 8-bit key formed from two nibbles: the four presence bits m of the bitmap for rows 4​g​…​4​g+3, and the next four bits s of that column’s compact sign stream. Entry (m,s) holds four fp16 constants c0​…​c3, one per row, where

ci={0if ​mi=0,+1if ​mi=1​ and ​sri=0,−1if ​mi=1​ and ​sri=1,ri=∑j<imj, (4)

and ri is the rank of row i among the present rows of the nibble. The table is a precomputed, four-bit-wide pdep composed with the map from sign bit to ternary code; Figure 7 lists representative entries.

媒体内容 · 前往原文查看

Fig. 7: Representative entries of the 256-entry lookup table. The key is the presence nibble m together with a four-bit window s of the compact sign stream, and the entry holds four fp16 constants, shown with their bit patterns. The second and third rows are the first two groups worked through in Figure 9, with keys 𝟶​𝚡​𝙱​𝟼 and 𝟶​𝚡​𝟼𝟼.

媒体内容 · 前往原文查看

Fig. 8: Xe2 kernel mapping of a weight tile onto SIMD lanes. (a) Each grid column is one of the 16 output columns and is held by one SIMD lane, and each grid row is one register row. VNNI2 order puts a k-pair in every dword. The blue-shaded band is the output of a single load.slm.d32x2, which returns two dwords per lane and therefore fills 2 register rows, i.e. 4 reduction rows, in one lookup. (b) The presence bitmap and the compact sign vector in memory, for the 16 columns one subgroup owns: the bitmap words of adjacent columns are adjacent, whereas each column enters the sign vector at its own offset.

媒体内容 · 前往原文查看

Fig. 9: Three consecutive four-row lookups on one column. Each group forms an 8-bit key from its presence nibble and a four-bit window of the compact sign stream. The bracket under each window gives the resulting table index/key. The sign-stream consumption is disjoint, but because the cursor advances by popcount⁡(m), the read windows overlap. The figure follows a single column, that is one SIMD lane. The kernel runs 16 of these in parallel, each with its own presence word, its own entry point into the sign stream and its own cursor.

Because a column’s sign bits are compacted, reading them requires tracking a per-column read position, which we call that column’s cursor: the number of sign bits the groups above it have already consumed, equivalently the rank of its next non-zero row among the non-zeros seen so far. The cursor is an offset into a variable-rate stream, and it is the only piece of per-column state whose value the bitmap alone does not give away. The sign field of the key always takes four bits, because four is the most a four-row group can need, but only popcount⁡(m) of them are consumed; that population count is a single cbit in Figure 6. The cursor therefore advances by popcount⁡(m) and the next group re-reads whatever this one left behind, so successive read windows overlap even though the bits they consume are disjoint, as Figure 9 draws for three consecutive groups. This is why the key is formed by a shift and a mask of a running window rather than by indexing the stream. The kernel keeps the window in a register and shifts it right by popcount⁡(m) after each group (shr signs of Figure 6). The gather address is maintained more coarsely, where the kernel takes one cbit of the entire 32-bit presence word per block and adds that to the column rank. The two population counts cbit in Figure 6 therefore serve different purposes and are not redundant, i.e. the per-group one only drives the window shift, and the per-block one only drives the gather address. Keeping them apart is what keeps the loop-carried chain short, since the address depends on one count per thirty-two rows rather than on a chain of eight. The per-group count also depends only on the bitmap, so it can issue before the SLM lookup returns. Because m and s are four bits each, the table has 28=256 entries of four fp16 values, so its size is 2 KB in total. At kernel entry, the workgroup’s subgroups cooperatively initialize disjoint table entries in one SLM-resident LUT. The group size of four rows is not arbitrary. Four presence bits and four sign bits give a table small enough to sit in SLM, whereas an eight-row group would need 216 entries. Four is also compatible with the XMX/DPAS operand layout. The fp16 DPAS consumes the weight tensor in VNNI2 order, with reduction rows 2​m and 2​m+1 packed into one dword, so four consecutive rows are precisely two VNNI2 dwords. The entry is therefore stored in the order the tile needs, a single d32x2 lookup returns both dwords, the load.slm.d32x2 of Figure 6, and they are written into the unpacked tile with no shuffle or transpose, thus meeting the VNNI2 requirement without any extra instructions.

Figure 8(a) makes the mapping concrete. A subgroup builds a tile of sixteen weight columns by sg_k reduction rows, and because a dword holds two consecutive k of one column, one SIMD16 register is exactly one k-pair across all sixteen columns. One lookup therefore fills two adjacent register rows, that is four reduction rows of sixteen columns, or 64 weights per message. The sixteen lanes are sixteen output columns, so the sequence is vectorized along n while k is walked serially by the loop. A column’s cursor depends on the population counts of all its preceding groups, so neighboring lanes drift apart as they advance. Figure 8(b) shows the presence bitmap and the compact sign vector in memory. The presence bitmap is stored as ⌈K/32⌉×N words, each folding 32 reduction rows of one column, and the words of adjacent columns are themselves adjacent, so sixteen lanes read them with a single block load, the load_block2d.ugm of Figure 6. The sixteen sign cursors, by contrast, are unrelated addresses, so the sign words must be gathered per lane (the gather load.ugm.d32x3). One 32-row window needs two adjacent words, and the next block starts at most one word later, so the union of both windows is exactly three words and a single d32x3 serves two blocks rather than one. A fourth word is never consumed: the cursor can sit at most 31 bits into a word and two blocks consume at most 32 sign bits each, so the span reaches bit 31+64−1=94 at worst, still inside the third word. In group 0 of Figure 9 the bitmap nibble m=10112 marks rows 0, 1 and 3 as non-zero and row 2 as zero, and the next four compact sign bits are s=01102, so the key is table index 𝟶​𝚡​𝙱​𝟼. Row 0 takes sign s0=0 and row 1 takes s1=1, but row 2 is absent and consumes nothing, so row 3 takes s2=1 rather than s3. The entry is therefore (+1,−1,0,−1), and the cursor advances by popcount⁡(m)=3, and the following groups advance it by 2 and 3.

III Experimental results

III-A Experimental platforms

We use three x86 CPU platforms with different core counts, core types and memory bandwidth:

One socket of an Intel Xeon Platinum 8592+ CPU (referred to as EMR) with 64 cores. It has DDR5@4400 MT/s memory and a measured streaming read bandwidth of ∼245 GB/s. It supports Advanced Matrix Extensions (AMX) and AVX-512, including AVX-512-FP16.

An Intel Core Ultra 9 285K CPU (referred to as ARL) with 24 cores (8 performance cores and 16 efficiency cores). It has dual-channel DDR5 memory and a measured read bandwidth of ∼98 GB/s. It supports AVX-VNNI-INT8 but not AVX-512.

An Intel Core Ultra 7 258V CPU (referred to as LNL CPU) with 8 cores (4 performance and 4 efficiency cores). It has 32 GB of LPDDR5X memory and a measured read bandwidth of ∼108 GB/s. It supports AVX-VNNI-INT8 but not AVX-512.

The instruction sets are relevant to the results: EMR runs the AVX-512 kernel of Section II-B, and ARL and LNL run the AVX2 kernel of Section II-C. For the GPU evaluation we use two Xe2 GPU platforms:

The Intel Arc 140V, which is the integrated GPU of the LNL platform above. It has 8 Xe2 cores, and it uses the same LPDDR5X memory as the LNL CPU, and its measured read bandwidth is ∼108 GB/s.

An Intel Arc Pro B70 discrete GPU. It has 32 Xe2 cores and 32 GB of dedicated GDDR6 memory, with a measured read bandwidth of ∼500 GB/s.

III-B Results on the CPU platforms

We first introduce a roofline model to assess the efficacy and limitations of our CPU kernels, and then present the GEMV microbenchmarks and the end-to-end decode results.

III-B1 A roofline model for the BITCOS CPU kernels

媒体内容 · 前往原文查看

Fig. 10: BITCOS CPU roofline at zero density z=0.40. Each line is ebw=min⁡(β,B/γ) for one kernel and core type; the flat part is the instruction ceiling B/γ and the diagonal is the memory bandwidth limit. Markers place the five measured core groups. Emerald Rapids and the Arrow Lake performance cores sit on the diagonal (thus the kernels are bandwidth bound), the Arrow Lake efficiency cores sit at the knee, and Lunar Lake sits on the flat part (thus it is instruction bound).

媒体内容 · 前往原文查看

Fig. 11: Zero-density sweep for a 32​k×16​k matrix-vector multiplication on: (a) Emerald Rapids, (b) Arrow Lake and (c) Lunar Lake, against the flat LIBXSMM 2-bit reference. The shaded band marks z∈[0.297,0.515], the range spanned by the deployed checkpoints of Table I. Top: GEMV time. Bottom: effective bandwidth.

媒体内容 · 前往原文查看

TABLE II: BITCOS CPU roofline at zero density z=0.40. γ is the measured cost in cycles of one L1-resident microkernel iteration and β is the per-core share of the measured read bandwidth. The bounding/bottleneck term is set in bold.

# of

platform core type cores γ β B/γ bound by

EMR P 64 4.60 1.321 1.502 memory

ARL P 08 5.00 0.756 1.381 memory

ARL E 16 7.95 0.903 0.868 instructions (knee)

LNL P 04 5.00 3.224 1.381 instructions

LNL E 04 7.95 3.663 0.868 instructions

In this section we present a simple two-term bottleneck roofline model [13], similar to the one used for the fixed-width ternary kernels in prior work [9]. For both the AVX-512 and the AVX2 CPU microkernels (i.e. see Figures 3 and 5), one iteration of the innermost loop upconverts 32 ternary weight values. Let γ be the number of cycles that iteration costs when every operand is already in L1, and β the share of read bandwidth available to one core, in bytes per cycle. For these 32 weight entries, an iteration reads (in bytes):

B⁡(z)=4⏟bitmap+4​(1−z)⏟signs+0.5⏟fp16 scale=8.5−4​z (5)

so the time T per iteration and the bandwidth ebw a core can sustain are:

T=max⁡(B⁡(z)β,γ),ebw=min⁡(β,B⁡(z)γ). (6)

The BITCOS-based kernel is memory-bound while B⁡(z)/γ>β and instruction-bound otherwise. The fixed-width 2-bit kernels have a constant B while a BITCOS-based kernel does not, so its knee moves with the density of the model at hand since B⁡(z) depends on z. For the per core bandwidth β, we take the per-core share of the streaming read bandwidth of the corresponding platform. We measure γ empirically on each platform by running the microkernel loop over an L1-resident block. Table II illustrates the measured γ values for the various core types, and we also report the term B/γ for a zero density z=0.4 which implies B=6.9 bytes read per 32 weight entries. Arrow Lake and Lunar Lake have the same performance and efficiency cores and run the same AVX2 microkernel, so a single pair of measurements for γ on performance and efficiency cores serves both platforms. Table II also evaluates Equaå (6) for the three CPU platforms, and Figure 10 depicts the corresponding roofline. Emerald Rapids is memory-bound on all of its cores and Arrow Lake on its performance cores, so the BITCOS-based kernel converts its smaller payload into savings in execution time; the Arrow Lake efficiency cores sit at the knee, where the two terms are within 4% of each other. Lunar Lake is instruction-bound on both core types: with only eight cores sharing 108 GB/s, each core has 3.2–3.7 bytes per cycle available but can only consume 0.87–1.38 bytes per cycle, so roughly three quarters of the bandwidth the platform offers a core is not attainable for this kernel. A bandwidth-rich client platform with a limited number of cores is exactly the case where a cheaper decode (like the 2-bit kernels from prior work [9]) beats kernels with smaller payload and more expensive decode (like the BITCOS-based kernel of this work).

III-B2 GEMV microbenchmarks on Emerald Rapids

To test the efficacy of the BITCOS-based GEMV microkernel we experimented with a large 32768×16384 ternary weight matrix. Weights are replicated to a working set of at least 4 GB so that nothing is served from the last-level cache. We use group size 128, i.e. 128 entries along the inner-product dimension share one 16-bit scale, and we vary the zero density z. The 2-bit reference GEMV is the production LIBXSMM 2-bit microkernel for CPUs [9]. In Figure 11(a) top panel we illustrate the execution time of the GEMV on EMR, whereas on the bottom panel we show the corresponding effective bandwidth. We observe that the BITCOS format wins at every density, and converts most of its bit-width advantage into execution time savings (see Figure 11(a) bottom panel, where the effective bandwidth stays constant ∼225 GB/s for z up to 0.6). This behavior is also validated by our roofline model, where on EMR the BITCOS kernel operates in a bandwidth-bound regime for z<0.6. In these plots we highlight with a green area the zero densities of interest: the deployed ternary models/checkpoints we examined in Table I exhibit z∈[0.297,0.515]. For these zero densities, the observed speedup of the BITCOS-based GEMV over the 2-bit SOTA GEMV is in the range of 1.14–1.28×. For extreme zero densities (e.g. z=0.95) we observe that the per-iteration byte count drops, yielding B/γ=1.02, and the unpack instruction sequence starts being the bottleneck, thus restricting the effective bandwidth to 190.6 GB/s.

III-B3 GEMV microbenchmarks on Arrow Lake

We repeat the same GEMV benchmark on Arrow Lake (see Figure 11(b)) and the conclusions are the same as the ones on EMR: the BITCOS-based GEMV wins at every density, and converts most of its bit-width advantage into execution time savings, which is in alignment with our roofline analysis. Over the same band of deployed zero densities, z∈[0.297,0.515], the observed speedup of the BITCOS-based GEMV over the 2-bit SOTA GEMV is in the range of 1.13–1.27×, and the kernel holds 93.4–93.6 GB/s of effective bandwidth across that band.

媒体内容 · 前往原文查看

Fig. 12: Decode throughput in tokens per second on the three CPU platforms, batch one, over 7 ternary LLMs, with each model’s measured zero density z under its name. The orange and green bars correspond to the Prism ML fork of llama.cpp and appear only for the Bonsai family of models that the fork supports. The purple bar corresponds to the SOTA LIBXSMM 2-bit kernel [9] and the blue bar is BITCOS (this work), both inside the vLLM CPU backend. The green number above each cluster of bars is the speedup of BITCOS over the SOTA 2-bit kernel.

III-B4 GEMV microbenchmarks on Lunar Lake

Figure 11(c) illustrates the GEMV benchmark on Lunar Lake CPU, where our roofline model predicts that the BITCOS-based GEMV kernel is instruction-bound on both core types, and as such it is expected to be slower than the SOTA 2-bit GEMV. The measurements in Figure 11(c) bottom panel confirm that prediction: the BITCOS kernel sustains only 28.9 GB/s of effective bandwidth at z=0.40. The BITCOS kernel is slower than the 2-bit reference at every density, and its effective bandwidth never exceeds 33.3 GB/s against the 74.7 GB/s the two-bit kernel sustains. This result confirms our roofline analysis, and it is a cautionary tale for client platforms with a limited number of cores and high memory bandwidth per core: a 2-bit, cheaper decode GEMV kernel beats kernels with smaller payload and more expensive decode.

III-B5 End-to-end decode on the CPU platforms

媒体内容 · 前往原文查看

Fig. 13: Zero-density sweep for a 32​k×16​k matrix-vector multiplication on (a) the Arc 140V and (b) the Arc Pro B70, against the flat XeTLA int2 reference, with every point tuned independently. The shaded band marks z∈[0.297,0.515], the range spanned by the deployed checkpoints of Table I. Top: GEMV time. Bottom: effective bandwidth.

媒体内容 · 前往原文查看

Fig. 14: Decode throughput in tokens per second on the two Xe2 platforms, batch one, over 7 ternary LLMs, with each model’s measured zero density z under its name. The orange bar corresponds to the Prism ML fork of llama.cpp on its Vulkan backend and appears only for the Bonsai family of models that the fork supports. The purple bar corresponds to the SOTA XeTLA int2 kernel [9] and the blue bar is BITCOS (this work), both inside the vLLM XPU backend. The green number above each cluster of bars is the speedup of BITCOS over the SOTA int2 kernel.

We integrated both the 2-bit LIBXSMM kernel [9] and the BITCOS GEMV kernels into the vLLM CPU backend [14] and measured decode throughput on all three CPU platforms of Section III-A over 7 of the group-scaled LLM models of Table I. We also benchmarked the same 7 models with the Prism ML fork of llama.cpp11 1 https://github.com/PrismML-Eng/llama.cpp. Two formats in that build are relevant here: Q2_0, the fork’s 2-bit code with one fp16 scale per 128 weights, which is the same bit rate and group size as the SOTA 2-bit packing of prior work [9], and upstream’s TQ1_0, a five-trit per byte format.

Figure 12 shows the results of the end-to-end inference on the three CPU platforms. Each bar corresponds to the achieved decode throughput in tokens per second, for a single request of 256 output tokens, with each inference engine at its own best thread count. The green number above each cluster of bars is the speedup of BITCOS over the SOTA 2-bit kernel, and we conclude that the two memory-bound platforms (EMR and ARL) benefit from BITCOS on every model, while the instruction-bound LNL CPU platform does not see any benefit, which is consistent with the roofline analysis in Section III-B1. We also make the following observations regarding the llama.cpp bars. First, the five-trit per byte format TQ1_0 is faster than the 2-bit Q2_0 on the two client platforms, by 1.66–1.68× on Arrow Lake and 1.91–2.44× on Lunar Lake, whereas on the server socket the two converge within 4%. It is worth noting that the SOTA 2-bit kernels of prior work [9] move more data than the TQ1_0 kernels of llama.cpp, and yet they outperform them by up to 1.78×. On the other hand, our BITCOS-based kernel outperforms the SOTA 2-bit kernel on the two memory-bound platforms in the range of 1.10–1.18× on Emerald Rapids and 1.02–1.15× on Arrow Lake. Compared to the TQ1_0 five-trit per byte format (which in principle is memory efficient), BITCOS is 1.13–1.48× faster on Emerald Rapids and 1.46–1.74× faster on Arrow Lake. On the instruction-bound LNL CPU platform the SOTA 2-bit work delivers the best end-to-end results and BITCOS loses on every model as predicted by the roofline model and the microbenchmarks of the previous section.

III-C Results on the Xe2 GPU platforms

We first present the GEMV microbenchmarks on each GPU platform (integrated GPU Arc 140V and discrete Arc Pro B70) and then the end-to-end decode results.

III-C1 GEMV microbenchmarks on the Arc 140V

To test the efficacy of the BITCOS-based Xe2 GEMV microkernel of Section II-D we use the same large 32768×16384 ternary weight matrix as on the CPUs, group size 128, and a varying zero density z. Every BITCOS point is tuned independently over the candidate tiles and the int2 reference is tuned the same way. In Figure 13(a) top panel we illustrate the execution time of the GEMV on the Arc 140V, whereas on the bottom panel we show the corresponding effective bandwidth. We observe that the BITCOS format wins at every sampled density. In these plots we highlight with a green area the zero densities of interest, i.e. the z∈[0.297,0.515] that the deployed checkpoints of Table I exhibit. For these zero densities, the observed speedup of the BITCOS-based GEMV over the int2 state-of-the-art GEMV is in the range of 1.04–1.14×, and the kernel delivers 71.7–75.6 GB/s of effective bandwidth across that band. The effective bandwidth does not stay flat but falls steadily with z, from 87.2 to 65.5 GB/s across the full sweep, because the payload shrinks while the decode work per weight does not. This is why the realized speedup is smaller than the corresponding payload reduction.

III-C2 GEMV microbenchmarks on the Arc Pro B70

We repeat the same GEMV benchmark on the discrete Arc Pro B70 (see Figure 13(b)) and the conclusions are largely the same as the ones on the Arc 140V: the BITCOS-based GEMV is never slower than the int2 reference. Over the same band of deployed zero densities, z∈[0.297,0.515], the observed speedup is in the range of 1.01–1.12×, and the kernel holds 398.7–421.9 GB/s of effective bandwidth across that band. The effective bandwidth declines with z for the same reason as on the integrated GPU platform, from 476.6 to 342.6 GB/s. For example, at z=0.40 the measured speedup of 1.06× (BITCOS vs 2-bit kernel) falls short of the 1.23× byte ratio.

III-C3 End-to-end decode on the GPU platforms

We integrated both the int2 XeTLA kernel [9] and the BITCOS GEMV kernels into the same vLLM XPU backend [14] and measured decode throughput on both Xe2 platforms of Section III-A over the same 7 group-scaled LLM models of Table I. We also benchmarked the same 7 models with the Prism ML fork of llama.cpp, this time on its Vulkan backend, i.e. the vendor-neutral GPU path that is available in llama.cpp. Vulkan is the only backend of that fork which is available for the Xe2 GPUs, as the SYCL backend does not support the two relevant formats (Q2_0 and TQ1_0). Of those two formats only Q2_0, the 2-bit code with one fp16 scale per 128 weights, has a Vulkan kernel. The five-trit per byte TQ1_0 format does not have any supporting Xe2 GPU kernel.

Figure 14 shows the results of the end-to-end inference on the two Xe2 platforms. Each bar corresponds to the achieved decode throughput in tokens per second, for a single request of 256 output tokens. The green number above each cluster of bars is the speedup of BITCOS over the SOTA int2 kernel, and we conclude that both Xe2 platforms benefit from BITCOS on every model, by 1.09–1.22× on the integrated Arc 140V and 1.02–1.27× on the discrete Arc Pro B70. We also make the following observations regarding the llama.cpp bars. At identical 2-bit format, identical group size and identical checkpoint the two 2-bit packing methods, (Q2_0 and the XeTLA int2 packing [9]) converge on the integrated GPU part within 5% in the end-to-end inference results (Figure 14(a)). On the discrete GPU part (Figure 14(b)) the 2-bit XeTLA kernel delivers 1.46–2.12× speedup over the Q2_0-based inference. Our BITCOS-based inference outperforms the Q2_0 of llama.cpp by 1.11–1.18× on the Arc 140V and by 1.61–2.30× on the Arc Pro B70 and pushes the envelope of ternary LLM inference on Xe2 GPUs.

IV Related Work

IV-A Ternary Quantization of LLMs

There are two main approaches to producing ultra-low-bit LLMs: quantization-aware training (QAT) and post-training quantization (PTQ). QAT applies the ternary-weights constraint during pretraining or fine-tuning. Early work established that binary and ternary weights are viable for natural language generation [15], and the seminal BitNet work [1, 2] showed that ternary {−1,0,+1} weights keep the accuracy of full-precision weights at scale. A subsequent work, ParetoQ [5] showed that ternary and 2-bit formats live on the accuracy-size Pareto frontier, ahead of 1-bit and 4-bit formats, refining the earlier k-bit inference scaling laws that placed the optimum at 4 bits [16, 17]. The Spectra/TriLM suite [6] pretrained ternary models ranging from 99M to 3.9B parameters. A more recent QAT work, Tequila [18], removes the trapping behavior that makes ternary QAT unstable by re-activating deadzone-trapped weights in the training process. Finally, more recent ternary QAT work has produced SOTA accuracy ternary LLMs for various LLM architectures: Bonsai [3] models range from 1.7B to 27B parameters and offer multi-step reasoning, structured tool calls, vision tasks and agentic loops, while Maple [7] is a 20B SOTA Mixture-of-Experts (MoE) ternary LLM with 1B active parameters. Such compact, high-quality models are a key enabler of on-device agentic systems [10]. Post-training quantization (PTQ) does not involve training steps and weight gradient updates. Instead, PTQ quantizes a pre-trained model and calibrates on a few sequences. Compared to QAT, PTQ is generally faster and less data-intensive, but may result in lower accuracy. Early PTQ methods at 4 and 2 bits, such as AWQ [19] and QuIP [20], have not matched QAT at ultra-low bit-widths. Recent advancements in PTQ, CAT-Q [4] and TWLA [21], make ternary weights directly from a pre-trained model and substantially narrow the accuracy gap compared to the QAT methods. These QAT and PTQ methods are complementary to our work: QAT and PTQ methods provide the means to obtain high-quality ternary LLM models, while our work yields memory-efficient and performant kernels to serve such ternary LLM inference on CPU and GPU platforms.

IV-B Kernels and runtimes for ternary LLM inference

Bitnet.cpp [22] is the reference runtime for ternary LLMs and it is the companion runtime of the seminal ternary LLM work [1]. It is faster than stock llama.cpp [23], but recent work showed that it does not deliver performance close to roofline [9], thus in this work we compared against the SOTA runtime [9]. Earlier libraries targeted ternary and binary inference on edge devices by packing several low-bit weights per word and exploiting bit-serial or sign-based arithmetic, e.g. TernGEMM [24] and TABv2 [25]. One related approach for ternary LLM kernels replaces multiplication with table lookup [26, 27, 28]. T-MAC [29] pre-computes partial dot products, and uses the packed low-bit codes as indices in a table lookup. This approach is applicable to CPUs with low vector FMA throughput. On GPUs, prior work mixes 2-bit and 4-bit groups within a weight matrix and overlaps dequantization with the contraction to contain the accuracy loss [30]. The 2-bit reference in this work is the LIBXSMM [31]/XeTLA [11] kernels [9], and it is the strongest published baseline for CPU and Xe2 GPU platforms to date. Section III shows that the LIBXSMM/XeTLA kernels are equal to or faster than the best llama.cpp configurations on all platforms. Therefore, the reported gains of this work are measured against kernels that are roofline-optimal: the advantage of our work over these SOTA kernels stems from the fact that we exploit the inherent zero density of ternary LLM weights.

IV-C Use of sparsity in ternary LLM inference

Recent work [32] exploits sparsity in ternary LLMs by storing only the indices of the non-zero weights in a Ternary CSC format, since the sign alone describes a non-zero ternary weight, and replaces the multiplications with additions and subtractions of the activations, which in theory reduces arithmetic by 4× at 50% sparsity. This technique helps only in compute-bound cases: prefill is compute-bound and becomes 1.2–2.3× faster than libTorch on Spectra TriLM 1.1B, but decode at batch size of one yields bandwidth-bound GEMV operations and the performance is even worse than the one of libTorch. The ternary CSC kernels also need 75–88% sparsity to overtake dense cuBLAS at the layer level, well above the 29.7–51.5% that most ternary checkpoints exhibit (Table I). Finally, the proposed index format has a variable bit rate and reads the weights through irregular gathers, whereas our BITCOS work consists of a dense and positional bitmap, offering sequential accesses and the resulting format is amenable to vectorizable unpacking.

Recent work makes the ternary sparsity semi-structured, so that N:M kernels can exploit it. Unlike the one-shot unstructured pruning of dense LLMs [33], the zeros here are produced by the quantizer itself. Sparse-BitNet [34] observes that the 42% zeros of a pretrained BitNet model are unstructured, and trains ternary quantization jointly with a dynamic N:M mask, reporting up to 1.30× end to end speedup. Sherry [35] constrains every block of four weights to hold exactly one zero, which leaves 4×23=32 distinct blocks that a five-bit code stores exactly, i.e. 1.25 bits per weight. Both these recent lines of work replace the distribution that the quantizer produces and therefore require training: Sparse-BitNet must apply the mask over the full pretraining run, while Sherry fixes the zero density at 25% and enforces a 3:4 pattern. Enforcing semi-structured sparsity also costs accuracy: Sparse-BitNet reports that its 6:8 ternary models lose 0.17–0.32 perplexity and 0.8 to 3.8 points of downstream accuracy against their own dense ternary baselines. BITCOS instead uses the unstructured zeros that existing SOTA ternary LLM checkpoints already contain. It is a change of the storage layout only, it applies to an existing checkpoint, it needs neither retraining nor sparsity hardware, and it is bit-exact, because it decodes the same ternary values as the existing/original ternary model.

V Conclusion

We introduced BITCOS, a distribution-adaptive ternary layout of a presence bitmap plus a compacted sign vector that costs 2−z bits per weight at a zero density z. Across 29 SOTA ternary LLM checkpoints the zero density ranges from 29.7% to 51.5%, so BITCOS stores 26 of them more compactly than the five-trit packing and reaches 1.485 bits per weight on the sparsest, while against the 2-bit format in production it reduces weight traffic by 1.16–1.32× on all 29 models. Over the zero densities these checkpoints exhibit, the BITCOS GEMV is 1.14–1.28× faster than the state-of-the-art 2-bit kernel on a 64-core server Emerald Rapids CPU, 1.13–1.27× on a 24-core client Arrow Lake CPU, 1.04–1.14× on the integrated Intel Xe2 Arc 140V GPU (Lunar Lake GPU) and 1.01–1.12× on the Arc Pro B70 discrete Intel GPU. End to end in vLLM over 7 ternary LLMs, decode throughput improves by 1.10–1.18×, 1.02–1.15×, 1.09–1.22× and 1.02–1.27× on the same four platforms. As future work we plan to extend the BITCOS layout and its unpacking sequences to more CPU and GPU architectures.

References

[1] S. Ma, H. Wang, L. Ma, L. Wang, W. Wang, S. Huang, L. Dong, R. Wang, J. Xue, and F. Wei (2024) The era of 1-bit LLMs: all large language models are in 1.58 bits. Note: arXiv:2402.17764 External Links: Document Cited by: §I, §I, §IV-A, §IV-B.

[2] S. Ma, H. Wang, S. Huang, X. Zhang, Y. Hu, T. Song, Y. Xia, and F. Wei (2025) BitNet b1.58 2B4T technical report. Note: arXiv:2504.12285 External Links: Document Cited by: §I, §I, §IV-A.

[3] Prism ML (2026) Ternary Bonsai models. Note: https://huggingface.co/prism-ml Cited by: §I, §IV-A.

[4] S. Wang, C. Li, Y. Kang, J. Fan, and A. Yao (2026) CAT-Q: cost-efficient and accurate ternary quantization for LLMs. In Proc. Int. Conf. on Machine Learning (ICML), Note: arXiv:2606.26650 Cited by: §I, §IV-A.

[5] Z. Liu, C. Zhao, H. Huang, S. Chen, J. Zhang, J. Zhao, S. Roy, L. Jin, Y. Xiong, et al. (2025) ParetoQ: improving scaling laws in extremely low-bit LLM quantization. Note: arXiv:2502.02631Models available: https://huggingface.co/collections/facebook/mobilellm External Links: Document Cited by: §I, §IV-A.

[6] A. Kaushal, T. Vaidhya, A. K. Mondal, T. Pandey, A. Bhagat, and I. Rish (2024) Spectra: surprising effectiveness of pretraining ternary language models at scale. Note: arXiv:2407.12327Models available: https://huggingface.co/collections/SpectraSuite/trilms-unpacked External Links: Document Cited by: §I, §IV-A.

[7] DeepGrove (2026) Maple: a 20B-A1B ternary-weight reasoning model. Note: https://huggingface.co/deepgrove/maple-preview Cited by: §I, §IV-A.

[8] OpenBMB (2025) BitCPM-CANN: full-pipeline ternary quantized models trained on CANN. Note: https://huggingface.co/collections/openbmb/bitcpm-cann Cited by: §I.

[9] E. Georganas, D. Kalamkar, A. Heinecke, and P. Dubey (2026) Pushing the envelope of LLM inference with ultra-low-bit quantized models. Note: arXiv:2508.06753v3 External Links: Document Cited by: §I, Fig. 12, Fig. 14, §III-B1, §III-B1, §III-B2, §III-B5, §III-B5, §III-C3, §III-C3, §IV-B.

[10] P. Belcak, G. Heinrich, S. Diao, Y. Fu, X. Dong, S. Muralidharan, Y. C. Lin, and P. Molchanov (2025) Small language models are the future of agentic AI. Note: arXiv:2506.02153 External Links: Document Cited by: §I, §IV-A.

[11] Intel XeTLA: intel Xe templates for linear algebra. Note: https://github.com/intel/xetla Cited by: §II-D, §IV-B.

[12] R. Keryell, R. Reyes, and L. Howes (2015) Khronos SYCL for OpenCL: a tutorial. In Proc. 3rd Int. Workshop on OpenCL, pp. 1–1. External Links: Document Cited by: §II-D.

[13] S. Williams, A. Waterman, and D. Patterson (2009) Roofline: an insightful visual performance model for multicore architectures. Communications of the ACM 52 (4), pp. 65–76. External Links: Document Cited by: §III-B1.

[14] W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica (2023) Efficient memory management for large language model serving with PagedAttention. In Proc. 29th ACM Symp. on Operating Systems Principles (SOSP), pp. 611–626. External Links: Document Cited by: §III-B5, §III-C3.

[15] Z. Liu, B. Oguz, A. Pappu, Y. Shi, and R. Krishnamoorthi (2023) Binary and ternary natural language generation. In Proc. 61st Annual Meeting of the Association for Computational Linguistics (ACL), pp. 65–77. External Links: Document Cited by: §IV-A.

[16] T. Dettmers and L. Zettlemoyer (2023) The case for 4-bit precision: k-bit inference scaling laws. In International Conference on Machine Learning, pp. 7750–7774. Cited by: §IV-A.

[17] T. Kumar, Z. Ankner, B. F. Spector, B. Bordelon, N. Muennighoff, M. Paul, C. Pehlevan, C. Ré, and A. Raghunathan (2024) Scaling laws for precision. Note: arXiv:2411.04330 External Links: Document Cited by: §IV-A.

[18] H. Huang, D. Wu, R. Cen, G. Yu, Z. Li, K. Liu, J. Zhu, P. Chen, X. Liu, and D. Wu (2025) Tequila: trapping-free ternary quantization for large language models. Note: arXiv:2509.23809 External Links: Document Cited by: §IV-A.

[19] J. Lin, J. Tang, H. Tang, S. Yang, G. Xiao, and S. Han (2025) AWQ: activation-aware weight quantization for on-device LLM compression and acceleration. GetMobile: Mobile Computing and Communications 28 (4), pp. 12–17. External Links: Document Cited by: §IV-A.

[20] J. Chee, Y. Cai, V. Kuleshov, and C. D. Sa (2023) QuIP: 2-bit quantization of large language models with guarantees. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 36, pp. 4396–4429. Cited by: §IV-A.

[21] Z. Zhao, Z. Xu, Z. Chen, X. Hu, Z. Jiang, and D. Yang (2026) TWLA: achieving ternary weights and low-bit activations for LLMs via post-training quantization. In Proc. Int. Conf. on Machine Learning (ICML), Note: arXiv:2606.13054 Cited by: §IV-A.

[22] J. Wang, H. Zhou, T. Song, S. Cao, Y. Xia, T. Cao, J. Wei, S. Ma, H. Wang, and F. Wei (2025) bitnet.cpp: efficient edge inference for ternary LLMs. In Proc. 63rd Annual Meeting of the Association for Computational Linguistics (ACL), pp. 9305–9322. External Links: Document Cited by: §IV-B.

[23] llama.cpp: LLM inference in C/C++. Note: https://github.com/ggml-org/llama.cpp Cited by: §IV-B.

[24] S. Choi, K. Shim, J. Choi, W. Sung, and B. Shim (2021) TernGEMM: general matrix multiply library with ternary weights for fast DNN inference. In Proc. IEEE Workshop on Signal Processing Systems (SiPS), pp. 111–116. External Links: Document Cited by: §IV-B.

[25] G. Fu, O. Fischer, S. Zhu, and G. Alonso (2026) TABv2: a faster ternary and binary neural network inference library on the edge. IEEE Trans. Very Large Scale Integr. (VLSI) Syst., pp. 1–13. External Links: Document Cited by: §IV-B.

[26] D. Blalock and J. Guttag (2021) Multiplying matrices without multiplying. In Proc. Int. Conf. on Machine Learning (ICML), pp. 992–1004. Cited by: §IV-B.

[27] X. Tang, Y. Wang, T. Cao, L. L. Zhang, Q. Chen, D. Cai, Y. Liu, and M. Yang (2023) LUT-NN: empower efficient neural network inference with centroid learning and table lookup. In Proc. 29th Annual Int. Conf. on Mobile Computing and Networking (MobiCom), pp. 1–15. External Links: Document Cited by: §IV-B.

[28] D. C. Ganji, S. Ashfaq, E. Saboori, S. Sah, S. Mitra, M. AskariHemmat, A. Hoffman, A. Hassanien, and M. Léonardon (2023) DeepGEMM: accelerated ultra low-precision inference on CPU architectures using lookup tables. In Proc. IEEE/CVF Conf. on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 4656–4664. External Links: Document Cited by: §IV-B.

[29] J. Wei, S. Cao, T. Cao, L. Ma, L. Wang, Y. Zhang, and M. Yang (2025) T-MAC: CPU renaissance via table lookup for low-bit LLM deployment on edge. In Proc. 20th European Conf. on Computer Systems (EuroSys), pp. 278–292. External Links: Document Cited by: §IV-B.

[30] J. Li, J. Xu, S. Li, S. Huang, J. Liu, Y. Lian, and G. Dai (2024) Fast and efficient 2-bit LLM inference on GPU: 2/4/16-bit in a weight matrix with asynchronous dequantization. In Proc. 43rd IEEE/ACM Int. Conf. on Computer-Aided Design (ICCAD), pp. 1–9. External Links: Document Cited by: §IV-B.

[31] LIBXSMM: library for specialized dense and sparse matrix operations, and deep learning primitives. Note: https://github.com/libxsmm/libxsmm Cited by: §IV-B.

[32] S. Zhu, G. Fu, M. Kjoseva, and G. Alonso (2026) Efficient addition-based sparse GEMM for fast ternary large language model inference on edge devices. ACM Trans. Embedded Computing Systems 25 (4), pp. 60:1–60:29. External Links: Document Cited by: §IV-C.

[33] E. Frantar and D. Alistarh (2023) SparseGPT: massive language models can be accurately pruned in one-shot. In Proc. Int. Conf. on Machine Learning (ICML), pp. 10323–10337. Cited by: §IV-C.

[34] D. Zhang, X. Wu, S. Huang, Y. Wang, H. Shao, Y. Hao, Z. Chi, L. Dong, T. Song, Y. Xia, Z. Sui, and F. Wei (2026) Sparse-BitNet: 1.58-bit LLMs are naturally friendly to semi-structured sparsity. Note: arXiv:2603.05168 External Links: Document Cited by: §IV-C.

[35] H. Huang, D. Wu, Q. Hu, G. Yu, J. Yang, J. Zhu, X. Liu, and D. Wu (2026) Sherry: hardware-efficient 1.25-bit ternary quantization via fine-grained sparsification. Note: arXiv:2601.07892 External Links: Document Cited by: §IV-C.

Optimization Notice: Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. For more information go to http://www.intel.com/performance.

Intel, Xeon, and Intel Xeon Phi are trademarks of Intel Corporation in the U.S. and/or other countries.