Breaking the 1.58-bit Barrier for Ternary LLMs

Evangelos Georganas, Alexander Heinecke, Pradeep Dubey

Intel Corporation

Abstract

Ternary Large Language Models (LLM) store every weight as one of three symbols $\left\{-1,0,+1\right\}$, so the cost of a ternary model is conventionally referenced to the information-theoretic ${\log }_{2}3\approx 1.585$ bits per weight. The prevailing deployment format packs five ternary weights into one byte (five-trit packing), and due to the power-of-two group sizes used in practice this rounds up to $1.625$ bits per weight. This effective storage bit-width treats the three symbols $\left\{-1,0,+1\right\}$ as equiprobable. We measure the actual symbol distribution of 29 ternary LLM models and find that zeros account for up to $51.5\%$ of all weights. Motivated by this finding, we introduce BITCOS, a simple distribution-adaptive layout comprised of a dense presence bitmap plus a compacted sign vector, and costs $2-z$ bits per weight element given a zero density $z$ in the model’s weights. BITCOS stores weights more compactly than the five-trit packing in 26 of the 29 tested models, and reaches $1.485$ bits per weight on the sparsest of them. BITCOS is amenable to efficient unpacking on modern processors and GPUs, and we present optimized unpacking sequences for AVX-512, AVX2 and Intel Xe2 GPUs. Measured against production state-of-the-art ternary matrix-vector multiplication kernels, at the zero densities real-world ternary models exhibit, the realized gain with our proposed layout is up to $1.28\times$. Finally, we illustrate end-to-end LLM inference results on 5 different platforms (client and server CPUs, integrated and discrete Xe2 GPUs) where decode throughput improves by up to $1.18\times$ on CPUs and $1.27\times$ on GPUs.

I Introduction

Ternary weight quantization restricts every weight to $\left\{-1,0,+1\right\}$ scaled by a group-wise factor [1, 2]. Three equiprobable symbols carry ${\log }_{2}3\approx 1.585$ bits of information, and this theoretical bound is the reference for ternary storage. Nevertheless, in practice the actual storage cost is determined by how those symbols are packed. Five trits (ternary digits) fit in a byte ($3^5=243\le 256$), which approaches the bound at $8/5=1.6$ bits per weight. However, deployed implementations quantize in blocks of power-of-two weights like 128, and 128 is not a multiple of 5: a block needs $\lceil 128/5\rceil =26$ payload bytes, so the rate stored in practice is $26\times 8/128=1.625$ bits per weight. Still, this effective storage bit-width treats the three symbols $\left\{-1,0,+1\right\}$ as equiprobable.

媒体内容 · 前往原文查看

媒体内容 · 前往原文查看

			BITCOS		2-bit packing		5-trit per byte
	Ternary model	% 0	Symbols	$+$scale	$+$scale	red.	$+$scale	red.
$\bullet$	BitNet b1.58 2B4T	42.19	1.578	—	2.000	$1.27\times$	1.625	$1.03\times$
$■$	Bonsai 1.7B	39.89	1.601	1.726	2.125	$1.23\times$	1.750	$1.01\times$
	Bonsai 4B	37.71	1.623	1.748	2.125	$1.22\times$	1.750	$1.00\times$
	Bonsai 8B	38.25	1.618	1.743	2.125	$1.22\times$	1.750	$1.00\times$
	Bonsai 27B	29.66	1.703	1.828	2.125	$1.16\times$	1.750	$0.96\times$
$▲$	CAT-Q Qwen3-1.7B	51.48	1.485	1.610	2.125	$1.32\times$	1.750	$1.09\times$
	CAT-Q Qwen3-8B	46.70	1.533	1.658	2.125	$1.28\times$	1.750	$1.06\times$
	CAT-Q Qwen3-30B-A3B	32.88	1.671	1.796	2.125	$1.18\times$	1.750	$0.97\times$
	CAT-Q Qwen3-32B	47.11	1.529	1.654	2.125	$1.28\times$	1.750	$1.06\times$
	CAT-Q Qwen3-235B-A22B	34.07	1.659	1.784	2.125	$1.19\times$	1.750	$0.98\times$
$⧫$	ParetoQ 125M	41.07	1.589	1.613	2.023	$1.25\times$	1.648	$1.02\times$
	ParetoQ 350M	41.58	1.584	1.598	2.014	$1.26\times$	1.639	$1.03\times$
	ParetoQ 600M	43.27	1.567	1.579	2.012	$1.27\times$	1.637	$1.04\times$
	ParetoQ 1B	44.18	1.558	1.569	2.010	$1.28\times$	1.635	$1.04\times$
	ParetoQ 1.5B	47.10	1.529	1.537	2.008	$1.31\times$	1.633	$1.06\times$
$▼$	TriLM 99M	40.97	1.590	1.618	2.027	$1.25\times$	1.652	$1.02\times$
	TriLM 190M	40.67	1.593	1.611	2.018	$1.25\times$	1.643	$1.02\times$
	TriLM 390M	40.79	1.592	1.606	2.014	$1.25\times$	1.639	$1.02\times$
	TriLM 560M	40.73	1.593	1.604	2.011	$1.25\times$	1.636	$1.02\times$
	TriLM 830M	40.54	1.595	1.604	2.009	$1.25\times$	1.634	$1.02\times$
	TriLM 1.1B	40.39	1.596	1.605	2.009	$1.25\times$	1.634	$1.02\times$
	TriLM 1.5B	40.21	1.598	1.606	2.008	$1.25\times$	1.633	$1.02\times$
	TriLM 2.4B	39.70	1.603	1.610	2.007	$1.25\times$	1.632	$1.01\times$
	TriLM 3.9B	38.70	1.613	1.618	2.005	$1.24\times$	1.630	$1.01\times$
$★$	Maple 20B-A1B	40.67	1.593	1.609	2.016	$1.25\times$	1.641	$1.02\times$
$◀$	BitCPM-CANN 0.5B	37.67	1.623	1.636	2.012	$1.23\times$	1.637	$1.00\times$
	BitCPM-CANN 1B	38.39	1.616	1.623	2.006	$1.24\times$	1.631	$1.01\times$
	BitCPM-CANN 3B	38.06	1.619	1.624	2.005	$1.23\times$	1.630	$1.00\times$
	BitCPM-CANN 8B	39.30	1.607	1.610	2.003	$1.24\times$	1.628	$1.01\times$

We measure the actual symbol distribution of 29 state-of-the-art (SOTA) ternary LLM models and find that zeros account for up to $51.5\%$ of all weights; Table I lists the measured zero density of every model. More specifically we benchmarked seven ternary model families: i) BitNet [1, 2], the 2B-parameter model trained from scratch on 4T tokens that established the $1.58$-bit reference point; ii) Bonsai [3], four dense checkpoints from $1.7$B to $27$B; iii) CAT-Q [4], five post-training quantizations of Qwen3 from $1.7$B to $235$B, including two mixture-of-experts models; iv) ParetoQ [5], five small checkpoints ($125$M–$1.5$B) from a study of low-bit quantization-aware training; v) TriLM [6], the nine-model Spectra suite ($99$M–$3.9$B) pretrained in ternary; vi) Maple [7], a $20$B-A1B ternary mixture-of-experts reasoning model; and vii) BitCPM-CANN [8], four ternary checkpoints ($0.5$B–$8$B) trained from scratch. Motivated by the finding that the zero density of many ternary models is significantly higher than the density of either of the other two codes $\left\{-1,+1\right\}$, we introduce BITCOS (BITmap and COmpacted Signs): a simple distribution-adaptive layout for ternary tensors. Given a zero density $z$, BITCOS spends one presence bit on every weight entry and one sign bit only on the non-zero weights, thus the layout effectively achieves $2-z$ bits per weight. Such a layout improves the effective bit-width compared to the deployed five-trit packing once $z>0.375$ and is strictly more efficient than the widely-adopted 2-bit packing for all zero densities. Figure 1 places every model of Table I at its measured zero density, on whichever of the two rates it meets first: the sloped $2-z$ line when the BITCOS sparse layout is more efficient, or the $1.625$ plateau when the five-trit fixed-rate packing has lower bit-width. Models with zero density $z$ above $0.375$ benefit from the BITCOS layout (green plot area), as such 26 out of the 29 models achieve lower effective bit-width with BITCOS compared to the five-trit packing layout.

However, the effective bit-width is only one aspect of the overall inference. The actual performance of ternary models during inference also depends on how efficiently the packed weights can be unpacked and utilized in matrix-vector multiplication kernels. The decode phase of LLM inference with a small batch-size is bandwidth-bound, so the time per token tracks the bit-width of the weight datatype [9], which is precisely the regime that on-device and agentic deployments exercise [10]. BITCOS not only reduces the storage cost but also is amenable to efficient unpacking and computation on modern CPUs and Intel GPUs. Measured end to end over 7 ternary LLM checkpoints and five platforms, decode throughput improves by up to $1.18\times$ on a $64$-core server CPU, $1.15\times$ on a $24$-core client CPU, $1.27\times$ on a discrete Xe2 GPU and $1.22\times$ on an integrated Xe2 GPU. The performance gain however is not universal: on an 8-core, bandwidth-rich client platform with enough bandwidth per core to leave the unpack sequence exposed, the smaller payload does not translate into performance benefits. Therefore, we develop a simple two-term roofline model to assess the limitations of the BITCOS-based kernels.

1. A novel ternary sparse layout (which we name BITCOS) comprising of a presence bitmap plus a compacted sign vector, that costs $2-z$ bits per weight for a zero density $z$ and is amenable to efficient unpacking on modern CPUs and GPUs.

2. Optimized instruction sequences to unpack the proposed BITCOS layout and perform matrix multiplication operations efficiently on modern x86 CPUs (client CPUs with performance and efficiency cores, and server CPUs with high core counts) and Intel Xe2 GPUs (both integrated and discrete GPUs).

3. A roofline model to assess the efficacy and the limitations of our CPU microkernels.

4. Performance evaluation of the proposed layout with microbenchmarks and end-to-end LLM inference over seven ternary LLM checkpoints on modern server/client CPUs and integrated/discrete GPUs, illustrating decode throughput improvements by up to $1.18\times$ on CPUs and $1.27\times$ on GPUs.

II The BITCOS Layout: BITmap + COmpacted Signs

II-A Layout definition and storage cost

We propose the BITCOS layout that leverages the inherent unstructured sparsity of ternary weights and stores a ternary tensor as two pieces:

1. a presence bitmap of one bit per weight, set where the weight is non-zero; and

2. a sign vector of one bit per non-zero weight, in tensor order.

Figure 2 shows both pieces on a small 8$\times$8 tensor. The bitmap has the full tensor length; the sign vector is compacted to the population count of the bitmap. Assuming a zero density of $z$, the cost per weight is therefore:

$B\left(z\right)=\mathrm{\hspace{0.33em}1}+\left(1-z\right)=\mathrm{\hspace{0.33em}2}-z\text{bits}$

(1)

媒体内容 · 前往原文查看

The cost per weight falls linearly in $z$, so the BITCOS format yields substantial benefits over 2-bit or five-trit packing at zero-heavy distributions. Table I compares the per-weight cost of different formats over the 29 measured checkpoints. Column “% 0” is the measured zero density $z$, column “Symbols” counts the ternary codes alone and the “$+$scale” columns add the measured 16-bit scale overhead of the corresponding model. The “red.” columns report the size reduction of BITCOS over the 2-bit and the five-trit per byte packing respectively; a value greater than $1$ means BITCOS stores the model more compactly than that format. We conclude that BITCOS improves the effective bit-width compared to the deployed five-trit packing once $z>0.375$ (26 out of 29 ternary LLM models) and is strictly more efficient than the widely-adopted 2-bit packing for all zero densities.

II-B BITCOS unpack sequence with x86 AVX-512 instructions

The BITCOS layout reconstructs 16-bit weights in a short mask-driven sequence assuming the target ISA supports masks. In Figure 3 we illustrate such an exemplary sequence with AVX-512 instructions. Two vectors are prepared once per group of 128 reduction elements and then reused for every one of its iterations: zmm3 holding the 32 fp16 group scales, one per row, and zmm4 holding a copy with the sign bit set, obtained by a single vporq against a broadcast 0x8000. Because the sign of an IEEE half lives in the most significant bit and the group scales are non-negative by construction, that OR is an exact negation, so zmm3 and zmm4 hold $+s$ and $-s$ respectively for each of the 32 rows. The per-iteration work is then:

1. Load the presence mask. Read 32 bitmap bits into a general register and then into k1.

2. Place the signs. Take the next 32 bits of the sign stream and scatter them via pdep into the bit positions that k1 marks as present. The sign vector is stored compacted, so its $j$-th bit belongs to the $j$-th non-zero of the group’s bitmap. pdep is exactly the scatter that undoes the compaction (see Figure 4).

3. Select. A zero-masked move of zmm3 under k1 puts $+s$ at every present lane and exact $+0$ elsewhere; a merge-masked move of zmm4 under the deposited mask overwrites the negative ones.

Figure 4 illustrates what that deposit does: pdep takes the low bits of its source in order and drops them at the positions the mask selects, leaving every unselected position zero. In the AVX-512 assembly, r13 walks the bitmap, r9 is the base of the sign stream, r8 the running bit position within it, r14 walks the activations, and zmm2 is one accumulator for the fused multiply-add (FMA) operation. In total we get 17 instructions, of which the weight unpacking portion corresponds to 3 instructions: one pdep and 2 masked moves. The remaining 14 are not unpacking: one loads the bitmap word, 5 compute the data-dependent address of the sign window, one shrx performs the unaligned 64-bit read and its alignment in a single operation, 2 are software prefetches, 2 advance the bit position by the population count, 2 move masks, and the last is the fused multiply-add (FMA), which absorbs the activation broadcast as an embedded operand.

媒体内容 · 前往原文查看

媒体内容 · 前往原文查看

II-C BITCOS unpack sequence with x86 AVX2 instructions

The sequence of Section II-B depends on two AVX-512 facilities: mask registers, which make a 32-bit presence pattern directly usable as a predicate, and FP16 compute instructions. The client CPUs we target support neither, thus we implemented a second kernel targeting the AVX2 ISA with AVX-VNNI-INT8 compute capabilities, which reconstructs the ternary weights $\left\{-1,0,+1\right\}$ as int8 and contracts against int8 activations (see Figure 5).

For each of the 32 rows, let $p_i\in \left\{0,1\right\}$ indicate that the weight is present, and let $n_i\in \left\{0,1\right\}$ indicate that the present weight is negative. The bitmap supplies the $p_i$ bits. After pdep returns the compact signs to their row positions, its result supplies the $n_i$ bits. Since AVX2 has no mask registers, we materialize each presence and sign vector as byte masks $P_i=-p_i$ and $N_i=-n_i$: a true lane is 0xFF ($-1$ as a signed byte), and a false lane is zero. The desired ternary byte is then

$w_i=p_i-2n_i.$

(2)

In the example of Figure 2, the column $k_2$ holds the ternary values $(+1,+1,+1,+1,-1,0,-1,+1)$ for rows $i=0\dots 7$, so the bitmap word for that column is $\mathrm{𝟶}𝚡𝙳𝙵$ and the two predicates are

$p=(1,1,1,1,1,0,1,1),n=(0,0,0,0,1,0,1,0).$

(3)

Equation (2) returns the unpacked column exactly: rows $0$–$3$ and $7$ give $1-0=+1$, rows $4$ and $6$ give $1-2=-1$, and the absent row $5$ gives $0-0=0$. Materialized as bytes these are $P=(\mathrm{𝙵𝙵},\mathrm{𝙵𝙵},\mathrm{𝙵𝙵},\mathrm{𝙵𝙵},\mathrm{𝙵𝙵},\mathrm{𝟶𝟶},\mathrm{𝙵𝙵},\mathrm{𝙵𝙵})$ and, after the mask with 0xFE, $T=(\mathrm{𝟶𝟶},\mathrm{𝟶𝟶},\mathrm{𝟶𝟶},\mathrm{𝟶𝟶},\mathrm{𝙵𝙴},\mathrm{𝟶𝟶},\mathrm{𝙵𝙴},\mathrm{𝟶𝟶})$, so vpsubb leaves $(\mathrm{𝟶𝟷},\mathrm{𝟶𝟷},\mathrm{𝟶𝟷},\mathrm{𝟶𝟷},\mathrm{𝙵𝙵},\mathrm{𝟶𝟶},\mathrm{𝙵𝙵},\mathrm{𝟶𝟷})$, which is the original column as int8. Note that $n$ is not what the format stores. Only the $\mathrm{popcount}\left(p\right)=7$ sign bits $(0,0,0,0,1,1,0)$ are stored, so the sign of row $6$ is the sixth stored bit rather than the seventh. Recovering the $n$ of (3) from those seven bits is exactly the pdep of Figure 5.

媒体内容 · 前往原文查看

媒体内容 · 前往原文查看

This alternative contraction algorithm has two implications. First, arithmetic moves from fp16 to integer: activations are quantized to int8 per group of 128, the dot product accumulates in int32, and the group scales are applied once per group instead of being blended into the weights. The weights are therefore reconstructed as the values $\pm 1$ rather than $\pm s$, and the layout is stored in VNNI4 order so that each vpdpbssd (AVX-VNNI-INT8 compute) consumes four consecutive reduction elements for each of eight rows. Second, and more consequentially, the absence of mask registers means the presence pattern must be materialized as one byte per lane before it can select anything. That expansion consists of a broadcast, an in-lane shuffle and a compare against the bit-select constant, and it costs 5 instructions where AVX-512 spends merely 1 mask move instruction kmovd. Once those byte masks exist, the last two instructions implement Eq. (2): masking $N_i$ with 0xFE forms $T_i=-2n_i$, and the vpsubb P,T,W computes $W_i=T_i-P_i=p_i-2n_i$. Thus a positive present lane becomes $+1$, a negative present lane becomes $-1$, and an absent lane remains zero.

II-D BITCOS unpack sequence for Intel Xe2 GPUs

Both x86 sequences rely on pdep to scatter compact signs back to the rows marked present. Xe2 has no corresponding instruction, so the GPU kernel replaces that scatter with a small lookup table in shared local memory (SLM). We implement the kernel with the XeTLA templates [11], and use the conventional SYCL terminology of workgroups and subgroups throughout [12]. Figure 6 illustrates the assembly sequence for the Xe2 kernel.

In the assembly sequence of Figure 6, one load.ugm.d32x3 fetches the three consecutive sign words sufficient for two 32-row blocks. For each block, the contiguous bitmap load load_block2d.ugm bmp supplies one 32-bit presence word per output column. Two bfn instructions, the three-input bitwise Boolean operation of Xe2, select the adjacent low/high sign words, and an and, two shifts and an or align them at the current bit rank. The and reduces the rank to the in-word offset $b$, the shr aligns the low word by $b$ and the shl brings in the $b$ high bits that cross the word boundary. That last shift needs a count of $32-b$, so the complementary count is taken modulo 32 and the or is predicated off in the one case the wrap gets wrong, $b=0$, where the high word contributes nothing. The block-wide cbit advances that rank by the number of nonzeros in all 32 rows. Each block is then decoded as eight four-row groups. A group extracts one presence nibble (4 bits), combines it with the next 4 bits of the aligned sign window, and uses the resulting byte offset for one SIMD16 load.slm.d32x2. The lookup returns four fp16 ternary codes per lane. The group-level cbit advances the sign window by only the bits actually consumed, an fp16 multiply applies the group scale, and one DPAS contraction is issued after four groups have supplied 16 reduction rows. The sequence touches two memories, load.ugm for the presence bitmap and sign vector in global memory and load.slm for the table in SLM.

The lookup table in SLM is indexed by an 8-bit key formed from two nibbles: the four presence bits $m$ of the bitmap for rows $4g\dots 4g+3$, and the next four bits $s$ of that column’s compact sign stream. Entry $(m,s)$ holds four fp16 constants $c_0\dots c_3$, one per row, where

$c_i=\{\begin{array}{cc}0& \text{if }{m}_i=0,\\ +1& \text{if }{m}_i=1\text{ and }{s}_{r_i}=0,\\ -1& \text{if }{m}_i=1\text{ and }{s}_{r_i}=1,\end{array}{r}_i={\sum }_{j<i}{m}_j,$

(4)

and $r_i$ is the rank of row $i$ among the present rows of the nibble. The table is a precomputed, four-bit-wide pdep composed with the map from sign bit to ternary code; Figure 7 lists representative entries.

媒体内容 · 前往原文查看

媒体内容 · 前往原文查看

媒体内容 · 前往原文查看

Because a column’s sign bits are compacted, reading them requires tracking a per-column read position, which we call that column’s cursor: the number of sign bits the groups above it have already consumed, equivalently the rank of its next non-zero row among the non-zeros seen so far. The cursor is an offset into a variable-rate stream, and it is the only piece of per-column state whose value the bitmap alone does not give away. The sign field of the key always takes four bits, because four is the most a four-row group can need, but only $\mathrm{popcount}\left(m\right)$ of them are consumed; that population count is a single cbit in Figure 6. The cursor therefore advances by $\mathrm{popcount}\left(m\right)$ and the next group re-reads whatever this one left behind, so successive read windows overlap even though the bits they consume are disjoint, as Figure 9 draws for three consecutive groups. This is why the key is formed by a shift and a mask of a running window rather than by indexing the stream. The kernel keeps the window in a register and shifts it right by $\mathrm{popcount}\left(m\right)$ after each group (shr signs of Figure 6). The gather address is maintained more coarsely, where the kernel takes one cbit of the entire 32-bit presence word per block and adds that to the column rank. The two population counts cbit in Figure 6 therefore serve different purposes and are not redundant, i.e. the per-group one only drives the window shift, and the per-block one only drives the gather address. Keeping them apart is what keeps the loop-carried chain short, since the address depends on one count per thirty-two rows rather than on a chain of eight. The per-group count also depends only on the bitmap, so it can issue before the SLM lookup returns. Because $m$ and $s$ are four bits each, the table has $2^8=256$ entries of four fp16 values, so its size is $2$ KB in total. At kernel entry, the workgroup’s subgroups cooperatively initialize disjoint table entries in one SLM-resident LUT. The group size of four rows is not arbitrary. Four presence bits and four sign bits give a table small enough to sit in SLM, whereas an eight-row group would need $2^{16}$ entries. Four is also compatible with the XMX/DPAS operand layout. The fp16 DPAS consumes the weight tensor in VNNI2 order, with reduction rows $2m$ and $2m+1$ packed into one dword, so four consecutive rows are precisely two VNNI2 dwords. The entry is therefore stored in the order the tile needs, a single d32x2 lookup returns both dwords, the load.slm.d32x2 of Figure 6, and they are written into the unpacked tile with no shuffle or transpose, thus meeting the VNNI2 requirement without any extra instructions.

Figure 8(a) makes the mapping concrete. A subgroup builds a tile of sixteen weight columns by sg_k reduction rows, and because a dword holds two consecutive $k$ of one column, one SIMD16 register is exactly one $k$-pair across all sixteen columns. One lookup therefore fills two adjacent register rows, that is four reduction rows of sixteen columns, or $64$ weights per message. The sixteen lanes are sixteen output columns, so the sequence is vectorized along $n$ while $k$ is walked serially by the loop. A column’s cursor depends on the population counts of all its preceding groups, so neighboring lanes drift apart as they advance. Figure 8(b) shows the presence bitmap and the compact sign vector in memory. The presence bitmap is stored as $\lceil K/32\rceil \times N$ words, each folding $32$ reduction rows of one column, and the words of adjacent columns are themselves adjacent, so sixteen lanes read them with a single block load, the load_block2d.ugm of Figure 6. The sixteen sign cursors, by contrast, are unrelated addresses, so the sign words must be gathered per lane (the gather load.ugm.d32x3). One 32-row window needs two adjacent words, and the next block starts at most one word later, so the union of both windows is exactly three words and a single d32x3 serves two blocks rather than one. A fourth word is never consumed: the cursor can sit at most $31$ bits into a word and two blocks consume at most $32$ sign bits each, so the span reaches bit $31+64-1=94$ at worst, still inside the third word. In group $0$ of Figure 9 the bitmap nibble $m={1011}_2$ marks rows $0$, $1$ and $3$ as non-zero and row $2$ as zero, and the next four compact sign bits are $s={0110}_2$, so the key is table index $\mathrm{𝟶}𝚡𝙱\mathrm{𝟼}$. Row $0$ takes sign $s_0=0$ and row $1$ takes $s_1=1$, but row $2$ is absent and consumes nothing, so row $3$ takes $s_2=1$ rather than $s_3$. The entry is therefore $(+1,-1,0,-1)$, and the cursor advances by $\mathrm{popcount}\left(m\right)=3$, and the following groups advance it by $2$ and $3$.

III Experimental results

III-A Experimental platforms

We use three x86 CPU platforms with different core counts, core types and memory bandwidth:

• One socket of an Intel Xeon Platinum 8592+ CPU (referred to as EMR) with $64$ cores. It has DDR5@4400 MT/s memory and a measured streaming read bandwidth of $\phantom{}\sim 245$ GB/s. It supports Advanced Matrix Extensions (AMX) and AVX-512, including AVX-512-FP16.

• An Intel Core Ultra 9 285K CPU (referred to as ARL) with $24$ cores ($8$ performance cores and $16$ efficiency cores). It has dual-channel DDR5 memory and a measured read bandwidth of $\phantom{}\sim 98$ GB/s. It supports AVX-VNNI-INT8 but not AVX-512.

• An Intel Core Ultra 7 258V CPU (referred to as LNL CPU) with $8$ cores ($4$ performance and $4$ efficiency cores). It has $32$ GB of LPDDR5X memory and a measured read bandwidth of $\phantom{}\sim 108$ GB/s. It supports AVX-VNNI-INT8 but not AVX-512.

The instruction sets are relevant to the results: EMR runs the AVX-512 kernel of Section II-B, and ARL and LNL run the AVX2 kernel of Section II-C. For the GPU evaluation we use two Xe2 GPU platforms:

• The Intel Arc 140V, which is the integrated GPU of the LNL platform above. It has $8$ Xe2 cores, and it uses the same LPDDR5X memory as the LNL CPU, and its measured read bandwidth is $\phantom{}\sim 108$ GB/s.

• An Intel Arc Pro B70 discrete GPU. It has $32$ Xe2 cores and $32$ GB of dedicated GDDR6 memory, with a measured read bandwidth of $\phantom{}\sim 500$ GB/s.

III-B Results on the CPU platforms

We first introduce a roofline model to assess the efficacy and limitations of our CPU kernels, and then present the GEMV microbenchmarks and the end-to-end decode results.

III-B1 A roofline model for the BITCOS CPU kernels

媒体内容 · 前往原文查看

媒体内容 · 前往原文查看

媒体内容 · 前往原文查看

		# of
platform	core type	cores	$\gamma$	$\beta$	$B/\gamma$	bound by
EMR	P	64	4.60	1.321	1.502	memory
ARL	P	08	5.00	0.756	1.381	memory
ARL	E	16	7.95	0.903	0.868	instructions (knee)
LNL	P	04	5.00	3.224	1.381	instructions
LNL	E	04	7.95	3.663	0.868	instructions

In this section we present a simple two-term bottleneck roofline model [13], similar to the one used for the fixed-width ternary kernels in prior work [9]. For both the AVX-512 and the AVX2 CPU microkernels (i.e. see Figures 3 and 5), one iteration of the innermost loop upconverts $32$ ternary weight values. Let $\gamma$ be the number of cycles that iteration costs when every operand is already in L1, and $\beta$ the share of read bandwidth available to one core, in bytes per cycle. For these 32 weight entries, an iteration reads (in bytes):

$B\left(z\right)=\underset{\text{bitmap}}{\underbrace{4}}+\underset{\text{signs}}{\underbrace{4\left(1-z\right)}}+\underset{\text{fp16 scale}}{\underbrace{0.5}}=8.5-4z$

(5)

so the time $T$ per iteration and the bandwidth $e_{\mathrm{bw}}$ a core can sustain are:

$T=\max (\frac{B\left(z\right)}{\beta },\gamma ),e_{\mathrm{bw}}=\min (\beta ,\frac{B\left(z\right)}{\gamma }).$

(6)

The BITCOS-based kernel is memory-bound while $B\left(z\right)/\gamma >\beta$ and instruction-bound otherwise. The fixed-width 2-bit kernels have a constant $B$ while a BITCOS-based kernel does not, so its knee moves with the density of the model at hand since $B\left(z\right)$ depends on $z$. For the per core bandwidth $\beta$, we take the per-core share of the streaming read bandwidth of the corresponding platform. We measure $\gamma$ empirically on each platform by running the microkernel loop over an L1-resident block. Table II illustrates the measured $\gamma$ values for the various core types, and we also report the term $B/\gamma$ for a zero density $z=0.4$ which implies $B=6.9$ bytes read per 32 weight entries. Arrow Lake and Lunar Lake have the same performance and efficiency cores and run the same AVX2 microkernel, so a single pair of measurements for $\gamma$ on performance and efficiency cores serves both platforms. Table II also evaluates Equaå (6) for the three CPU platforms, and Figure 10 depicts the corresponding roofline. Emerald Rapids is memory-bound on all of its cores and Arrow Lake on its performance cores, so the BITCOS-based kernel converts its smaller payload into savings in execution time; the Arrow Lake efficiency cores sit at the knee, where the two terms are within $4\%$ of each other. Lunar Lake is instruction-bound on both core types: with only eight cores sharing $108$ GB/s, each core has $3.2$–$3.7$ bytes per cycle available but can only consume $0.87$–$1.38$ bytes per cycle, so roughly three quarters of the bandwidth the platform offers a core is not attainable for this kernel. A bandwidth-rich client platform with a limited number of cores is exactly the case where a cheaper decode (like the 2-bit kernels from prior work [9]) beats kernels with smaller payload and more expensive decode (like the BITCOS-based kernel of this work).

III-B2 GEMV microbenchmarks on Emerald Rapids

To test the efficacy of the BITCOS-based GEMV microkernel we experimented with a large 32768$\times$16384 ternary weight matrix. Weights are replicated to a working set of at least $4$ GB so that nothing is served from the last-level cache. We use group size 128, i.e. 128 entries along the inner-product dimension share one 16-bit scale, and we vary the zero density $z$. The 2-bit reference GEMV is the production LIBXSMM 2-bit microkernel for CPUs [9]. In Figure 11(a) top panel we illustrate the execution time of the GEMV on EMR, whereas on the bottom panel we show the corresponding effective bandwidth. We observe that the BITCOS format wins at every density, and converts most of its bit-width advantage into execution time savings (see Figure 11(a) bottom panel, where the effective bandwidth stays constant $\phantom{}\sim 225$ GB/s for $z$ up to $0.6$). This behavior is also validated by our roofline model, where on EMR the BITCOS kernel operates in a bandwidth-bound regime for $z<0.6$. In these plots we highlight with a green area the zero densities of interest: the deployed ternary models/checkpoints we examined in Table I exhibit $z\in \left[0.297,0.515\right]$. For these zero densities, the observed speedup of the BITCOS-based GEMV over the 2-bit SOTA GEMV is in the range of $1.14$–$1.28\times$. For extreme zero densities (e.g. $z=0.95$) we observe that the per-iteration byte count drops, yielding $B/\gamma =1.02$, and the unpack instruction sequence starts being the bottleneck, thus restricting the effective bandwidth to $190.6$ GB/s.

III-B3 GEMV microbenchmarks on Arrow Lake

We repeat the same GEMV benchmark on Arrow Lake (see Figure 11(b)) and the conclusions are the same as the ones on EMR: the BITCOS-based GEMV wins at every density, and converts most of its bit-width advantage into execution time savings, which is in alignment with our roofline analysis. Over the same band of deployed zero densities, $z\in \left[0.297,0.515\right]$, the observed speedup of the BITCOS-based GEMV over the 2-bit SOTA GEMV is in the range of $1.13$–$1.27\times$, and the kernel holds $93.4$–$93.6$ GB/s of effective bandwidth across that band.

媒体内容 · 前往原文查看

III-B4 GEMV microbenchmarks on Lunar Lake

Figure 11(c) illustrates the GEMV benchmark on Lunar Lake CPU, where our roofline model predicts that the BITCOS-based GEMV kernel is instruction-bound on both core types, and as such it is expected to be slower than the SOTA 2-bit GEMV. The measurements in Figure 11(c) bottom panel confirm that prediction: the BITCOS kernel sustains only $28.9$ GB/s of effective bandwidth at $z=0.40$. The BITCOS kernel is slower than the 2-bit reference at every density, and its effective bandwidth never exceeds $33.3$ GB/s against the $74.7$ GB/s the two-bit kernel sustains. This result confirms our roofline analysis, and it is a cautionary tale for client platforms with a limited number of cores and high memory bandwidth per core: a 2-bit, cheaper decode GEMV kernel beats kernels with smaller payload and more expensive decode.

III-B5 End-to-end decode on the CPU platforms

媒体内容 · 前往原文查看

媒体内容 · 前往原文查看

We integrated both the 2-bit LIBXSMM kernel [9] and the BITCOS GEMV kernels into the vLLM CPU backend [14] and measured decode throughput on all three CPU platforms of Section III-A over 7 of the group-scaled LLM models of Table I. We also benchmarked the same 7 models with the Prism ML fork of llama.cpp¹¹ 1 https://github.com/PrismML-Eng/llama.cpp. Two formats in that build are relevant here: Q2_0, the fork’s 2-bit code with one fp16 scale per $128$ weights, which is the same bit rate and group size as the SOTA 2-bit packing of prior work [9], and upstream’s TQ1_0, a five-trit per byte format.

Figure 12 shows the results of the end-to-end inference on the three CPU platforms. Each bar corresponds to the achieved decode throughput in tokens per second, for a single request of $256$ output tokens, with each inference engine at its own best thread count. The green number above each cluster of bars is the speedup of BITCOS over the SOTA 2-bit kernel, and we conclude that the two memory-bound platforms (EMR and ARL) benefit from BITCOS on every model, while the instruction-bound LNL CPU platform does not see any benefit, which is consistent with the roofline analysis in Section III-B1. We also make the following observations regarding the llama.cpp bars. First, the five-trit per byte format TQ1_0 is faster than the 2-bit Q2_0 on the two client platforms, by $1.66$–$1.68\times$ on Arrow Lake and $1.91$–$2.44\times$ on Lunar Lake, whereas on the server socket the two converge within $4\%$. It is worth noting that the SOTA 2-bit kernels of prior work [9] move more data than the TQ1_0 kernels of llama.cpp, and yet they outperform them by up to $1.78\times$. On the other hand, our BITCOS-based kernel outperforms the SOTA 2-bit kernel on the two memory-bound platforms in the range of $1.10$–$1.18\times$ on Emerald Rapids and $1.02$–$1.15\times$ on Arrow Lake. Compared to the TQ1_0 five-trit per byte format (which in principle is memory efficient), BITCOS is $1.13$–$1.48\times$ faster on Emerald Rapids and $1.46$–$1.74\times$ faster on Arrow Lake. On the instruction-bound LNL CPU platform the SOTA 2-bit work delivers the best end-to-end results and BITCOS loses on every model as predicted by the roofline model and the microbenchmarks of the previous section.

III-C Results on the Xe2 GPU platforms

We first present the GEMV microbenchmarks on each GPU platform (integrated GPU Arc 140V and discrete Arc Pro B70) and then the end-to-end decode results.

III-C1 GEMV microbenchmarks on the Arc 140V

To test the efficacy of the BITCOS-based Xe2 GEMV microkernel of Section II-D we use the same large 32768$\times$16384 ternary weight matrix as on the CPUs, group size 128, and a varying zero density $z$. Every BITCOS point is tuned independently over the candidate tiles and the int2 reference is tuned the same way. In Figure 13(a) top panel we illustrate the execution time of the GEMV on the Arc 140V, whereas on the bottom panel we show the corresponding effective bandwidth. We observe that the BITCOS format wins at every sampled density. In these plots we highlight with a green area the zero densities of interest, i.e. the $z\in \left[0.297,0.515\right]$ that the deployed checkpoints of Table I exhibit. For these zero densities, the observed speedup of the BITCOS-based GEMV over the int2 state-of-the-art GEMV is in the range of $1.04$–$1.14\times$, and the kernel delivers $71.7$–$75.6$ GB/s of effective bandwidth across that band. The effective bandwidth does not stay flat but falls steadily with $z$, from $87.2$ to $65.5$ GB/s across the full sweep, because the payload shrinks while the decode work per weight does not. This is why the realized speedup is smaller than the corresponding payload reduction.

III-C2 GEMV microbenchmarks on the Arc Pro B70

We repeat the same GEMV benchmark on the discrete Arc Pro B70 (see Figure 13(b)) and the conclusions are largely the same as the ones on the Arc 140V: the BITCOS-based GEMV is never slower than the int2 reference. Over the same band of deployed zero densities, $z\in \left[0.297,0.515\right]$, the observed speedup is in the range of $1.01$–$1.12\times$, and the kernel holds $398.7$–$421.9$ GB/s of effective bandwidth across that band. The effective bandwidth declines with $z$ for the same reason as on the integrated GPU platform, from $476.6$ to $342.6$ GB/s. For example, at $z=0.40$ the measured speedup of $1.06\times$ (BITCOS vs 2-bit kernel) falls short of the $1.23\times$ byte ratio.

III-C3 End-to-end decode on the GPU platforms

We integrated both the int2 XeTLA kernel [9] and the BITCOS GEMV kernels into the same vLLM XPU backend [14] and measured decode throughput on both Xe2 platforms of Section III-A over the same 7 group-scaled LLM models of Table I. We also benchmarked the same 7 models with the Prism ML fork of llama.cpp, this time on its Vulkan backend, i.e. the vendor-neutral GPU path that is available in llama.cpp. Vulkan is the only backend of that fork which is available for the Xe2 GPUs, as the SYCL backend does not support the two relevant formats (Q2_0 and TQ1_0). Of those two formats only Q2_0, the 2-bit code with one fp16 scale per $128$ weights, has a Vulkan kernel. The five-trit per byte TQ1_0 format does not have any supporting Xe2 GPU kernel.

Figure 14 shows the results of the end-to-end inference on the two Xe2 platforms. Each bar corresponds to the achieved decode throughput in tokens per second, for a single request of $256$ output tokens. The green number above each cluster of bars is the speedup of BITCOS over the SOTA int2 kernel, and we conclude that both Xe2 platforms benefit from BITCOS on every model, by $1.09$–$1.22\times$ on the integrated Arc 140V and $1.02$–$1.27\times$ on the discrete Arc Pro B70. We also make the following observations regarding the llama.cpp bars. At identical 2-bit format, identical group size and identical checkpoint the two 2-bit packing methods, (Q2_0 and the XeTLA int2 packing [9]) converge on the integrated GPU part within $5\%$ in the end-to-end inference results (Figure 14(a)). On the discrete GPU part (Figure 14(b)) the 2-bit XeTLA kernel delivers $1.46$–$2.12\times$ speedup over the Q2_0-based inference. Our BITCOS-based inference outperforms the Q2_0 of llama.cpp by $1.11$–$1.18\times$ on the Arc 140V and by $1.61$–$2.30\times$ on the Arc Pro B70 and pushes the envelope of ternary LLM inference on Xe2 GPUs.

IV Related Work

IV-A Ternary Quantization of LLMs

There are two main approaches to producing ultra-low-bit LLMs: quantization-aware training (QAT) and post-training quantization (PTQ). QAT applies the ternary-weights constraint during pretraining or fine-tuning. Early work established that binary and ternary weights are viable for natural language generation [15], and the seminal BitNet work [1, 2] showed that ternary $\left\{-1,0,+1\right\}$ weights keep the accuracy of full-precision weights at scale. A subsequent work, ParetoQ [5] showed that ternary and 2-bit formats live on the accuracy-size Pareto frontier, ahead of 1-bit and 4-bit formats, refining the earlier $k$-bit inference scaling laws that placed the optimum at 4 bits [16, 17]. The Spectra/TriLM suite [6] pretrained ternary models ranging from 99M to 3.9B parameters. A more recent QAT work, Tequila [18], removes the trapping behavior that makes ternary QAT unstable by re-activating deadzone-trapped weights in the training process. Finally, more recent ternary QAT work has produced SOTA accuracy ternary LLMs for various LLM architectures: Bonsai [3] models range from 1.7B to 27B parameters and offer multi-step reasoning, structured tool calls, vision tasks and agentic loops, while Maple [7] is a 20B SOTA Mixture-of-Experts (MoE) ternary LLM with 1B active parameters. Such compact, high-quality models are a key enabler of on-device agentic systems [10]. Post-training quantization (PTQ) does not involve training steps and weight gradient updates. Instead, PTQ quantizes a pre-trained model and calibrates on a few sequences. Compared to QAT, PTQ is generally faster and less data-intensive, but may result in lower accuracy. Early PTQ methods at 4 and 2 bits, such as AWQ [19] and QuIP [20], have not matched QAT at ultra-low bit-widths. Recent advancements in PTQ, CAT-Q [4] and TWLA [21], make ternary weights directly from a pre-trained model and substantially narrow the accuracy gap compared to the QAT methods. These QAT and PTQ methods are complementary to our work: QAT and PTQ methods provide the means to obtain high-quality ternary LLM models, while our work yields memory-efficient and performant kernels to serve such ternary LLM inference on CPU and GPU platforms.

IV-B Kernels and runtimes for ternary LLM inference

Bitnet.cpp [22] is the reference runtime for ternary LLMs and it is the companion runtime of the seminal ternary LLM work [1]. It is faster than stock llama.cpp [23], but recent work showed that it does not deliver performance close to roofline [9], thus in this work we compared against the SOTA runtime [9]. Earlier libraries targeted ternary and binary inference on edge devices by packing several low-bit weights per word and exploiting bit-serial or sign-based arithmetic, e.g. TernGEMM [24] and TABv2 [25]. One related approach for ternary LLM kernels replaces multiplication with table lookup [26, 27, 28]. T-MAC [29] pre-computes partial dot products, and uses the packed low-bit codes as indices in a table lookup. This approach is applicable to CPUs with low vector FMA throughput. On GPUs, prior work mixes 2-bit and 4-bit groups within a weight matrix and overlaps dequantization with the contraction to contain the accuracy loss [30]. The 2-bit reference in this work is the LIBXSMM [31]/XeTLA [11] kernels [9], and it is the strongest published baseline for CPU and Xe2 GPU platforms to date. Section III shows that the LIBXSMM/XeTLA kernels are equal to or faster than the best llama.cpp configurations on all platforms. Therefore, the reported gains of this work are measured against kernels that are roofline-optimal: the advantage of our work over these SOTA kernels stems from the fact that we exploit the inherent zero density of ternary LLM weights.

IV-C Use of sparsity in ternary LLM inference

Recent work [32] exploits sparsity in ternary LLMs by storing only the indices of the non-zero weights in a Ternary CSC format, since the sign alone describes a non-zero ternary weight, and replaces the multiplications with additions and subtractions of the activations, which in theory reduces arithmetic by $4\times$ at $50\%$ sparsity. This technique helps only in compute-bound cases: prefill is compute-bound and becomes $1.2$–$2.3\times$ faster than libTorch on Spectra TriLM 1.1B, but decode at batch size of one yields bandwidth-bound GEMV operations and the performance is even worse than the one of libTorch. The ternary CSC kernels also need $75$–$88\%$ sparsity to overtake dense cuBLAS at the layer level, well above the $29.7$–$51.5\%$ that most ternary checkpoints exhibit (Table I). Finally, the proposed index format has a variable bit rate and reads the weights through irregular gathers, whereas our BITCOS work consists of a dense and positional bitmap, offering sequential accesses and the resulting format is amenable to vectorizable unpacking.

Recent work makes the ternary sparsity semi-structured, so that N:M kernels can exploit it. Unlike the one-shot unstructured pruning of dense LLMs [33], the zeros here are produced by the quantizer itself. Sparse-BitNet [34] observes that the $42\%$ zeros of a pretrained BitNet model are unstructured, and trains ternary quantization jointly with a dynamic N:M mask, reporting up to $1.30\times$ end to end speedup. Sherry [35] constrains every block of four weights to hold exactly one zero, which leaves $4\times 2^3=32$ distinct blocks that a five-bit code stores exactly, i.e. $1.25$ bits per weight. Both these recent lines of work replace the distribution that the quantizer produces and therefore require training: Sparse-BitNet must apply the mask over the full pretraining run, while Sherry fixes the zero density at $25\%$ and enforces a 3:4 pattern. Enforcing semi-structured sparsity also costs accuracy: Sparse-BitNet reports that its 6:8 ternary models lose $0.17$–$0.32$ perplexity and $0.8$ to $3.8$ points of downstream accuracy against their own dense ternary baselines. BITCOS instead uses the unstructured zeros that existing SOTA ternary LLM checkpoints already contain. It is a change of the storage layout only, it applies to an existing checkpoint, it needs neither retraining nor sparsity hardware, and it is bit-exact, because it decodes the same ternary values as the existing/original ternary model.

V Conclusion

We introduced BITCOS, a distribution-adaptive ternary layout of a presence bitmap plus a compacted sign vector that costs $2-z$ bits per weight at a zero density $z$. Across 29 SOTA ternary LLM checkpoints the zero density ranges from $29.7\%$ to $51.5\%$, so BITCOS stores 26 of them more compactly than the five-trit packing and reaches $1.485$ bits per weight on the sparsest, while against the 2-bit format in production it reduces weight traffic by $1.16$–$1.32\times$ on all 29 models. Over the zero densities these checkpoints exhibit, the BITCOS GEMV is $1.14$–$1.28\times$ faster than the state-of-the-art 2-bit kernel on a 64-core server Emerald Rapids CPU, $1.13$–$1.27\times$ on a 24-core client Arrow Lake CPU, $1.04$–$1.14\times$ on the integrated Intel Xe2 Arc 140V GPU (Lunar Lake GPU) and $1.01$–$1.12\times$ on the Arc Pro B70 discrete Intel GPU. End to end in vLLM over 7 ternary LLMs, decode throughput improves by $1.10$–$1.18\times$, $1.02$–$1.15\times$, $1.09$–$1.22\times$ and $1.02$–$1.27\times$ on the same four platforms. As future work we plan to extend the BITCOS layout and its unpacking sequences to more CPU and GPU architectures.

References

• [1] S. Ma, H. Wang, L. Ma, L. Wang, W. Wang, S. Huang, L. Dong, R. Wang, J. Xue, and F. Wei (2024) The era of 1-bit LLMs: all large language models are in 1.58 bits. Note: arXiv:2402.17764 External Links: Document Cited by: §I, §I, §IV-A, §IV-B.
• [2] S. Ma, H. Wang, S. Huang, X. Zhang, Y. Hu, T. Song, Y. Xia, and F. Wei (2025) BitNet b1.58 2B4T technical report. Note: arXiv:2504.12285 External Links: Document Cited by: §I, §I, §IV-A.
• [3] Prism ML (2026) Ternary Bonsai models. Note: https://huggingface.co/prism-ml Cited by: §I, §IV-A.
• [4] S. Wang, C. Li, Y. Kang, J. Fan, and A. Yao (2026) CAT-Q: cost-efficient and accurate ternary quantization for LLMs. In Proc. Int. Conf. on Machine Learning (ICML), Note: arXiv:2606.26650 Cited by: §I, §IV-A.
• [5] Z. Liu, C. Zhao, H. Huang, S. Chen, J. Zhang, J. Zhao, S. Roy, L. Jin, Y. Xiong, et al. (2025) ParetoQ: improving scaling laws in extremely low-bit LLM quantization. Note: arXiv:2502.02631Models available: https://huggingface.co/collections/facebook/mobilellm External Links: Document Cited by: §I, §IV-A.
• [6] A. Kaushal, T. Vaidhya, A. K. Mondal, T. Pandey, A. Bhagat, and I. Rish (2024) Spectra: surprising effectiveness of pretraining ternary language models at scale. Note: arXiv:2407.12327Models available: https://huggingface.co/collections/SpectraSuite/trilms-unpacked External Links: Document Cited by: §I, §IV-A.
• [7] DeepGrove (2026) Maple: a 20B-A1B ternary-weight reasoning model. Note: https://huggingface.co/deepgrove/maple-preview Cited by: §I, §IV-A.
• [8] OpenBMB (2025) BitCPM-CANN: full-pipeline ternary quantized models trained on CANN. Note: https://huggingface.co/collections/openbmb/bitcpm-cann Cited by: §I.
• [9] E. Georganas, D. Kalamkar, A. Heinecke, and P. Dubey (2026) Pushing the envelope of LLM inference with ultra-low-bit quantized models. Note: arXiv:2508.06753v3 External Links: Document Cited by: §I, Fig. 12, Fig. 14, §III-B1, §III-B1, §III-B2, §III-B5, §III-B5, §III-C3, §III-C3, §IV-B.
• [10] P. Belcak, G. Heinrich, S. Diao, Y. Fu, X. Dong, S. Muralidharan, Y. C. Lin, and P. Molchanov (2025) Small language models are the future of agentic AI. Note: arXiv:2506.02153 External Links: Document Cited by: §I, §IV-A.
• [11] Intel XeTLA: intel Xe templates for linear algebra. Note: https://github.com/intel/xetla Cited by: §II-D, §IV-B.
• [12] R. Keryell, R. Reyes, and L. Howes (2015) Khronos SYCL for OpenCL: a tutorial. In Proc. 3rd Int. Workshop on OpenCL, pp. 1–1. External Links: Document Cited by: §II-D.
• [13] S. Williams, A. Waterman, and D. Patterson (2009) Roofline: an insightful visual performance model for multicore architectures. Communications of the ACM 52 (4), pp. 65–76. External Links: Document Cited by: §III-B1.
• [14] W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica (2023) Efficient memory management for large language model serving with PagedAttention. In Proc. 29th ACM Symp. on Operating Systems Principles (SOSP), pp. 611–626. External Links: Document Cited by: §III-B5, §III-C3.
• [15] Z. Liu, B. Oguz, A. Pappu, Y. Shi, and R. Krishnamoorthi (2023) Binary and ternary natural language generation. In Proc. 61st Annual Meeting of the Association for Computational Linguistics (ACL), pp. 65–77. External Links: Document Cited by: §IV-A.
• [16] T. Dettmers and L. Zettlemoyer (2023) The case for 4-bit precision: k-bit inference scaling laws. In International Conference on Machine Learning, pp. 7750–7774. Cited by: §IV-A.
• [17] T. Kumar, Z. Ankner, B. F. Spector, B. Bordelon, N. Muennighoff, M. Paul, C. Pehlevan, C. Ré, and A. Raghunathan (2024) Scaling laws for precision. Note: arXiv:2411.04330 External Links: Document Cited by: §IV-A.
• [18] H. Huang, D. Wu, R. Cen, G. Yu, Z. Li, K. Liu, J. Zhu, P. Chen, X. Liu, and D. Wu (2025) Tequila: trapping-free ternary quantization for large language models. Note: arXiv:2509.23809 External Links: Document Cited by: §IV-A.
• [19] J. Lin, J. Tang, H. Tang, S. Yang, G. Xiao, and S. Han (2025) AWQ: activation-aware weight quantization for on-device LLM compression and acceleration. GetMobile: Mobile Computing and Communications 28 (4), pp. 12–17. External Links: Document Cited by: §IV-A.
• [20] J. Chee, Y. Cai, V. Kuleshov, and C. D. Sa (2023) QuIP: 2-bit quantization of large language models with guarantees. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 36, pp. 4396–4429. Cited by: §IV-A.
• [21] Z. Zhao, Z. Xu, Z. Chen, X. Hu, Z. Jiang, and D. Yang (2026) TWLA: achieving ternary weights and low-bit activations for LLMs via post-training quantization. In Proc. Int. Conf. on Machine Learning (ICML), Note: arXiv:2606.13054 Cited by: §IV-A.
• [22] J. Wang, H. Zhou, T. Song, S. Cao, Y. Xia, T. Cao, J. Wei, S. Ma, H. Wang, and F. Wei (2025) bitnet.cpp: efficient edge inference for ternary LLMs. In Proc. 63rd Annual Meeting of the Association for Computational Linguistics (ACL), pp. 9305–9322. External Links: Document Cited by: §IV-B.
• [23] llama.cpp: LLM inference in C/C++. Note: https://github.com/ggml-org/llama.cpp Cited by: §IV-B.
• [24] S. Choi, K. Shim, J. Choi, W. Sung, and B. Shim (2021) TernGEMM: general matrix multiply library with ternary weights for fast DNN inference. In Proc. IEEE Workshop on Signal Processing Systems (SiPS), pp. 111–116. External Links: Document Cited by: §IV-B.
• [25] G. Fu, O. Fischer, S. Zhu, and G. Alonso (2026) TABv2: a faster ternary and binary neural network inference library on the edge. IEEE Trans. Very Large Scale Integr. (VLSI) Syst., pp. 1–13. External Links: Document Cited by: §IV-B.
• [26] D. Blalock and J. Guttag (2021) Multiplying matrices without multiplying. In Proc. Int. Conf. on Machine Learning (ICML), pp. 992–1004. Cited by: §IV-B.
• [27] X. Tang, Y. Wang, T. Cao, L. L. Zhang, Q. Chen, D. Cai, Y. Liu, and M. Yang (2023) LUT-NN: empower efficient neural network inference with centroid learning and table lookup. In Proc. 29th Annual Int. Conf. on Mobile Computing and Networking (MobiCom), pp. 1–15. External Links: Document Cited by: §IV-B.
• [28] D. C. Ganji, S. Ashfaq, E. Saboori, S. Sah, S. Mitra, M. AskariHemmat, A. Hoffman, A. Hassanien, and M. Léonardon (2023) DeepGEMM: accelerated ultra low-precision inference on CPU architectures using lookup tables. In Proc. IEEE/CVF Conf. on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 4656–4664. External Links: Document Cited by: §IV-B.
• [29] J. Wei, S. Cao, T. Cao, L. Ma, L. Wang, Y. Zhang, and M. Yang (2025) T-MAC: CPU renaissance via table lookup for low-bit LLM deployment on edge. In Proc. 20th European Conf. on Computer Systems (EuroSys), pp. 278–292. External Links: Document Cited by: §IV-B.
• [30] J. Li, J. Xu, S. Li, S. Huang, J. Liu, Y. Lian, and G. Dai (2024) Fast and efficient 2-bit LLM inference on GPU: 2/4/16-bit in a weight matrix with asynchronous dequantization. In Proc. 43rd IEEE/ACM Int. Conf. on Computer-Aided Design (ICCAD), pp. 1–9. External Links: Document Cited by: §IV-B.
• [31] LIBXSMM: library for specialized dense and sparse matrix operations, and deep learning primitives. Note: https://github.com/libxsmm/libxsmm Cited by: §IV-B.
• [32] S. Zhu, G. Fu, M. Kjoseva, and G. Alonso (2026) Efficient addition-based sparse GEMM for fast ternary large language model inference on edge devices. ACM Trans. Embedded Computing Systems 25 (4), pp. 60:1–60:29. External Links: Document Cited by: §IV-C.
• [33] E. Frantar and D. Alistarh (2023) SparseGPT: massive language models can be accurately pruned in one-shot. In Proc. Int. Conf. on Machine Learning (ICML), pp. 10323–10337. Cited by: §IV-C.
• [34] D. Zhang, X. Wu, S. Huang, Y. Wang, H. Shao, Y. Hao, Z. Chi, L. Dong, T. Song, Y. Xia, Z. Sui, and F. Wei (2026) Sparse-BitNet: 1.58-bit LLMs are naturally friendly to semi-structured sparsity. Note: arXiv:2603.05168 External Links: Document Cited by: §IV-C.
• [35] H. Huang, D. Wu, Q. Hu, G. Yu, J. Yang, J. Zhu, X. Liu, and D. Wu (2026) Sherry: hardware-efficient 1.25-bit ternary quantization via fine-grained sparsification. Note: arXiv:2601.07892 External Links: Document Cited by: §IV-C.

Optimization Notice: Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. For more information go to http://www.intel.com/performance.

Intel, Xeon, and Intel Xeon Phi are trademarks of Intel Corporation in the U.S. and/or other countries.