Abstract

Compressing large-scale neural networks is essential for deploying models on resource-constrained devices. Most existing methods adopt weight pruning or low-bit quantization individually, often resulting in suboptimal compression rates to preserve acceptable performance drops. We introduce a unified framework for simultaneous pruning and low-bit quantization via Bayesian variational learning (SQS), which achieves higher compression rates than prior baselines while maintaining comparable performance. The key idea is to employ a spike-and-slab prior to induce sparsity and model quantized weights using Gaussian Mixture Models (GMMs) to enable low-bit precision. Due to the intractability of the objective involving spike-and-slab priors with GMMs, we derive an efficient approximation that facilitates effective compression with minimal accuracy loss. In theory, we provide a consistent result for our proposed variational approach to a sparse and quantized deep neural network. Extensive experiments on compressing ResNet, BERT-base, Llama3.2, and Qwen2.5 models show that our method achieves higher compression rates than a line of existing methods with comparable performance drops.
Project page: https://comeusr.github.io/SQS_Webpage/.

1 Introduction

Deep Neural Networks (DNNs) have achieved state-of-the-art performance across a wide range of tasks but at the cost of significantly increased computational and memory requirements (Radford et al., 2018; Xu et al., 2020; Touvron et al., 2023; Kumar et al., 2025), making deployment on resource-constrained devices challenging. Model compression methods have therefore been proposed to reduce the size and computational complexity of DNNs while maintaining predictive accuracy, including pruning (LeCun et al., 1989; Han et al., 2016), weight quantization (Courbariaux et al., 2015; Rastegari et al., 2016; Frantar et al., 2023; Lin et al., 2024), knowledge distillation (Park et al., 2019; Gou et al., 2021), and neural architecture search (Liu et al., 2018; Wang et al., 2020b).

Among these, weight pruning and low-bit quantization are particularly effective and widely adopted for compressing DNNs (Buciluǎ et al., 2006; Choudhary et al., 2020; Liu et al., 2025a). Weight pruning eliminates redundant or unimportant weights by setting selected weights to zero, thereby reducing the number of active parameters without significantly altering the model architecture (You et al., 2019; Guo et al., 2016; Dong et al., 2017). On the other hand, quantization reduces the bit-width of numerical representations for inputs, outputs, and weights by converting high-precision formats (e.g., FP32) to lower-precision alternatives, such as FP8 or INT8. This quantization coarsens the model representation and yields significant reductions in memory footprint and computational overhead. It enhances efficiency in both training and inference across diverse architectures, including ResNet (Banner et al., 2018), Transformers (Sun et al., 2019), Large language models (Dettmers et al., 2023; Wang et al., 2025), and vision-language models (Wortsman et al., 2023).

However, quantization and pruning inevitably introduce distributional shifts from the original DNNs, often leading to performance degradation (Dong et al., 2022). To mitigate this, existing methods adopt conservative compression rates, limiting their applicability to resource-constrained environments (Wang et al., 2020c; Wang et al., 2020b; Bai et al., 2022; Frantar et al., 2022; Bai et al., 2023). Achieving high compression rates while maintaining acceptable performance remains an open question to explore.

To tackle the above problem, we introduce a unified framework: Sparse Quantized Sub-distribution compression (SQS), which unifies pruning and quantization within a single variational learning process. Instead of applying pruning and quantization separately, the key idea of SQS is joint pruning and quantization that learns a sparse, quantized sub-distribution over network weights through variational learning. To model the variational posterior, we adopt a spike-and-slab prior combined with a Gaussian Mixture Model (GMM): the spike component encourages sparsity for pruning, while the GMM component models a quantized weight distribution, effectively mitigating performance degradation. The training pipeline of SQS is in Figure 1. Theoretically, we show that under mild conditions, our SQS method finds a sparse and quantized neural network that converges to the true underlying target neural network with high probability.

In our experiments, we compare several recent state-of-the-art compression methods across a range of widely used neural networks, including ResNet, BERT-base, Llama3.2, and Qwen2.5. Our findings show that (1) under the same bit-width setting, our SQS achieves the highest compression rate, requiring fewer parameters than baselines. (2) At the same compression rate, our SQS achieves the smallest accuracy drop or F1 score drop among all approaches, with particularly strong performance at 2-bit and 4-bit precision.

Further ablation studies highlight the contributions of individual components: (1) The spike-and-slab distribution is more effective in promoting sparsity than Gaussian alternatives. (2) Bayesian averaging during inference outperforms greedy weight selection. (3) An outlier-aware window strategy better preserves informative weight outliers compared to uniform windowing, further improving performance.

• We propose SQS, a unified Bayesian framework for compressing full-precision DNNs into sparse, low-bit models. Unlike methods that perform pruning and quantization separately, SQS jointly learns which weights to remove and how to quantize the remaining weights using a spike-and-GMM variational distribution.

• We derive a tractable approximate objective for training SQS and provide theoretical guarantees showing that, under mild conditions, the learned sparse and quantized network converges to the target regression function.

• We conduct experiments on ResNet, BERT-base, Llama3.2, and Qwen2.5. Across these architectures, SQS achieves higher compression rates with comparable or smaller accuracy degradation than existing baselines. Ablation studies further validate the effectiveness of the spike-and-slab formulation, Bayesian averaging at inference time, and the outlier-aware windowing strategy.

2 Preliminaries

Low-bit Quantization uses discrete low-bit values to approximate full-precision floating-point values, primarily to reduce precision for more efficient storage and computation while preserving essential information (Gholami et al., 2022). Formally, it is defined as a mapping $Q:ℝ\to 𝒬=\{{\mu }_1,\dots ,{\mu }_K\}$, where the input is the full-precision weight and $𝒬$ denotes the set of low-bit discrete values. Representative quantization methods include deterministic quantization (Jacob et al., 2018), stochastic quantization (Courbariaux et al., 2015), and end-to-end learnable quantization (Dong et al., 2022).

Specifically, let $\theta =({\theta }_1,\dots ,{\theta }_T)\in {ℝ}^T$ represent the pre-trained full-precision weights of a deep neural network, with ${\theta }_i$ denoting the $i$-th weight. Given a quantization set $𝒬=\{{\mu }_1,\dots ,{\mu }_K\}$, a general stochastic quantization is a map $Q:ℝ\to {𝒫}_K\left(ℝ\right)$ from the real numbers to the space of probability distributions over $ℝ$ with finite support of cardinality $K$. For each weight ${\theta }_i$,

$Q\left({\theta }_i\right)={\mu }_k,$

$\text{with probability }{p}_{k_i},$

for $i=1,\dots ,T$. Here ${\mu }_k$ is the learnable parameter and $p_{k_i}$ is the corresponding probability that weight ${\theta }_i$ is quantized to weight ${\mu }_k$. A key challenge is the distribution divergence between the quantized weights and the original weights, leading to significant performance degradation (Dong et al., 2022). To mitigate this, Dong et al. (2022) propose to approximate the quantized weight distribution using a Gaussian Mixture Model (GMM):

$Q\left({\theta }_i\right)\approx \sum _{k=1}^K{\varphi }_k({\theta }_i,{\pi }_k)𝒩({\mu }_k,{\sigma }_k^2),$

(1)

where $𝒩({\mu }_k,{\sigma }_k^2)$ denotes a Gaussian distribution, and ${\varphi }_k({\theta }_i,{\pi }_k)$ is the weight for the $k$-th Gaussian component $𝒩({\mu }_k,{\sigma }_k^2)$. To control the sharpness of this mixture, a temperature-scaled softmax is applied to obtain ${\varphi }_k({\theta }_i,{\pi }_k)$, that is:

${\varphi }_k({\theta }_i,{\pi }_k)=\frac{\exp \left({\phi }_k({\theta }_i,{\pi }_k)/\tau \right)}{{\sum }_{j=1}^K\exp \left({\phi }_j({\theta }_i,{\pi }_j)/\tau \right)},$

(2)

where the temperature parameter $\tau >0$ controls the concentration of the distribution. As $\tau \to 0$, the GMM in Equation (1) approaches a single dominant Gaussian component. Given a prior distribution $({\pi }_1,\dots ,{\pi }_K)$ over the quantization set $𝒬$, the posterior component weight ${\phi }_k({\theta }_i,{\pi }_k)$ is:

${\phi }_k({\theta }_i,{\pi }_k)=\frac{\exp \left({\pi }_k𝒩({\theta }_i\mid {\mu }_k,{\sigma }_k^2)\right)}{{\sum }_{j=1}^K\exp \left({\pi }_j𝒩({\theta }_i\mid {\mu }_j,{\sigma }_j^2)\right)}.$

Additionally, with sufficiently small ${\sigma }_k^2$, the GMM approximates a multinomial distribution over $𝒬$, effectively bridging continuous and discrete quantization. For simplicity, we denote ${\varphi }_k\left({\theta }_i\right)$ as a shorthand for ${\varphi }_k({\theta }_i,{\pi }_k)$ throughout the remainder of this paper.

In our experiments, we find that the GMM-based compression method (Dong et al., 2022) still cannot achieve a high compression rate while maintaining a small performance drop, as it cannot efficiently encourage sparsity during training.

Variational learning. Given an observed dataset $D$, the goal of a Bayesian framework is to infer the true posterior distribution $\pi \left(\theta \right|D)\propto \pi \left(\theta \right)p\left(D\right|\theta )$, where $\pi \left(\theta \right)$ denotes the prior and $p\left(D\right|\theta )$ the likelihood. Since the posterior is generally intractable, variational learning (Jordan et al., 1999) is proposed to approximate it by selecting the closest distribution from a variational family $ℱ$ in terms of the Kullback–Leibler (KL) divergence (Csiszar, 1975):

$q^{\ast }\left(\theta \right)={\arg \min }_{q\left(\theta \right)\in ℱ}\mathrm{KL}(q\left(\theta \right)\parallel \pi (\theta \mid D)).$

(3)

Following (Blei et al., 2017), this optimization is equivalent to minimizing the negative Evidence Lower Bound (ELBO), defined as:

$\Omega \left(q\right):=-{𝔼}_{q\left(\theta \right)}\left[\log p\left(D\right|\theta )\right]+\mathrm{KL}(q\left(\theta \right)\parallel \pi \left(\theta \right)),$

(4)

where the first term measures how well the variational distribution $q\left(\theta \right)$ aligns with the log-likelihood of the observed data, and the second term regularizes $q\left(\theta \right)$ to stay close to the prior $\pi \left(\theta \right)$.

Our SQS method employs a variational family based on a spike-and-GMM distribution to approximate the sparse and quantized posterior. The first term in Equation (4) allows the spike-and-GMM to learn the posterior distribution given the data. For the second term, we adopt a spike-and-slab prior $\pi (\cdot )$ distribution to promote sparsity in the network weights.

3 Methodology

The objective is to approximate a full-precision neural network $f(\cdot ,\theta )$ with a Bayesian model $f(\cdot ,\tilde{\theta })$ that is both sparse and low-precision, while minimizing performance degradation. To achieve this, we employ a spike-and-slab distribution combined with a GMM to parameterize the variational posterior.

3.1 SQS: Variational learning for sparse and quantized sub-distribution

The spike-and-slab prior consists of a point mass at zero (spike) and a continuous distribution (slab) (Bai et al., 2020; Ishwaran and Rao, 2005). Formally, let $\gamma =({\gamma }_1,\dots ,{\gamma }_T)$ be a binary indicator vector, where each ${\gamma }_i$ determines whether the corresponding weight ${\theta }_i$ is preserved (${\gamma }_i=1$) or pruned (${\gamma }_i=0$). The prior for each weight ${\tilde{\theta }}_i$ is defined as:

	${\gamma }_i$	$\phantom{}\sim \mathrm{Bern}\left(\lambda \right),$
	${\tilde{\theta }}_i|{\gamma }_i$	$\phantom{}\sim {\gamma }_i𝒩(0,{\sigma }_0^2)+\left(1-{\gamma }_i\right){\delta }_0,$

where $\lambda$ is the prior probability of retaining a weight, and ${\sigma }_0^2$ is the prior variance of the Gaussian slab. Marginalizing out the binary variable ${\gamma }_i$, the prior distribution over ${\tilde{\theta }}_i$ becomes:

$\pi \left({\tilde{\theta }}_i\right)=\lambda 𝒩(0,{\sigma }_0^2)+\left(1-\lambda \right){\delta }_0,$

(5)

where $1-\lambda$ corresponds to the prior pruning probability. For example, in a DNN with a target sparsity of $90\%$, setting $\lambda =0.1$ implies that each weight has a $90\%$ prior probability of being pruned.

3.1.1 Training Procedure

To incorporate quantization into the variational family, we extend the spike-and-slab formulation by modeling the slab using a $K$-component GMM. Each variational distribution $q\left({\theta }_i\right)$ is then defined as:

	${\gamma }_i$	$\phantom{}\sim \mathrm{𝙱𝚎𝚛𝚗}\left({\tilde{\lambda }}_i\right)$
	${\tilde{\theta }}_i|{\gamma }_i$	$\phantom{}\sim {\gamma }_i\sum _{k=1}^K{\varphi }_k\left({\theta }_i\right)𝒩({\mu }_k,{\sigma }_k^2)+\left(1-{\gamma }_i\right){\delta }_0$

where ${\varphi }_k\left({\theta }_i\right)$ is the mixture weight for component $k$, and ${\tilde{\lambda }}_i$ is the variational probability of retaining weight ${\theta }_i$. The marginal variational distribution $q\left({\tilde{\theta }}_i\right)$ is:

$q\left({\tilde{\theta }}_i\right)={\tilde{\lambda }}_i\sum _{k=1}^K{\varphi }_k\left({\theta }_i\right)𝒩({\mu }_k,{\sigma }_k^2)+\left(1-{\tilde{\lambda }}_i\right){\delta }_0.$

(6)

Given this variational family, we define the learning objective based on the ELBO:

$\Omega \left(\tilde{\theta }\right)=-{𝔼}_{q\left(\tilde{\theta }\right)}\left[\log p\left(D\right|\tilde{\theta })\right]+\sum _{i=1}^T\mathrm{KL}(q\left({\tilde{\theta }}_i\right)\parallel \pi \left({\tilde{\theta }}_i\right)).$

(7)

Yet, computing Equation (7) is intractable, as no closed-form solution exists for the KL divergence between $q\left({\tilde{\theta }}_i\right)$ and the spike-and-slab prior $\pi \left({\tilde{\theta }}_i\right)$. To overcome this challenge, we propose the following approximation: For each coordinate $i$, the posterior mean is ${𝔼}_{q\left({\tilde{\theta }}_i\right)}\left[{\tilde{\theta }}_i\right]={\tilde{\lambda }}_i{\sum }_{k=1}^K{\mu }_k{\varphi }_k({\theta }_i,\pi ,\tau )$. Collecting these coordinate-wise means, define

${\theta }^{\mathrm{avg}}:={𝔼}_{q\left(\tilde{\theta }\right)}\left[\tilde{\theta }\right]={\left({\tilde{\lambda }}_i\sum _{k=1}^K{\mu }_k{\varphi }_k({\theta }_i,\pi ,\tau )\right)}_{i=1}^T\in {ℝ}^T.$

The approximate objective becomes

${\Omega }_{\mathrm{𝚊𝚙𝚡}}\left(\tilde{\theta }\right)=\phantom{}$

$-\log p(D\mid {\theta }^{\mathrm{avg}})+\sum _{i=1}^T\mathrm{KL}(\mathrm{𝙱𝚎𝚛𝚗}\left({\tilde{\lambda }}_i\right)\parallel \mathrm{𝙱𝚎𝚛𝚗}\left(\lambda \right))+\sum _{i=1}^T{\tilde{\lambda }}_i\mathrm{KL}(𝒩({\mu }_{k_i^{\ast }},{\sigma }_{k_i^{\ast }}^2)\parallel 𝒩(0,{\sigma }_0^2)),$

(8)

where $k_i^{\ast }={\arg \max }_{1\le k\le K}{\varphi }_k\left({\theta }_i\right)$. The first term uses the plug-in approximation $-{𝔼}_{q\left(\tilde{\theta }\right)}\left[\log p(D\mid \tilde{\theta })\right]\approx -\log p(D\mid {\theta }^{\mathrm{avg}}).$ Thus, the likelihood is evaluated using the complete parameter vector ${\theta }^{\mathrm{avg}}$. The second and the third terms provide an upper bound on the term ${\sum }_{i=1}^T\mathrm{KL}(q\left({\tilde{\theta }}_i\right)\parallel \pi \left({\tilde{\theta }}_i\right))$ by applying Lemma 3. Please refer to Appendix A for a detailed derivation of Equation (8).

3.1.2 Inference Procedure

In the inference stage, we first sample the sparse and quantized weights given the learned parameters and predict the output $\hat{y}$ for each testing input $x$. Let $\hat{q}(\cdot )$ denote the optimization solution of the above variational learning, associated with the optimal parameter estimations ${\{{\hat{\mu }}_i,{\hat{\sigma }}_i^2,{\hat{\lambda }}_i\}}_{i=1}^T$, and the corresponding ${\hat{\varphi }}_k$’s (for each ${\theta }_i$) are obtained from Equation (2). Then, the $i$-th quantized weight ${\tilde{\theta }}_i$ is sampled from the set of quantization levels $\{{\hat{\mu }}_1,\dots ,{\hat{\mu }}_K\}$ according to

$P({\tilde{\theta }}_i={\hat{\mu }}_k\mid {\theta }_i)={\hat{\varphi }}_k\left({\theta }_i\right),k=1,\dots ,K.$

(9)

Compared to sampling from $𝒩({\hat{\mu }}_k,{\hat{\sigma }}_k^2)$, posterior sampling in Equation (9) reduces memory consumption.

To enforce sparsity, we introduce a user-specified pruning parameter, the Non-zero rate. Each weight ${\theta }_i$ is associated with a score ${\hat{\lambda }}_i$, which reflects the likelihood of being retained. We deterministically prune by setting the $i$-th weight to zero if ${\hat{\lambda }}_i$ is smaller than the Non-zero-quantile of all ${\hat{\lambda }}_i$ values; otherwise, the weight is kept unchanged. Formally,

${\tilde{\theta }}_i=0,$

$\text{ if }{\hat{\lambda }}_i<\text{Non-zero}\text{ quantile of all }{\hat{\lambda }}_i,$

This deterministic rule provides exact control over the sparsity level, in contrast to stochastic pruning via posterior sampling (Bai et al., 2020; Sun et al., 2022), which does not guarantee a fixed sparsity rate and often requires an additional pruning step.

Bayesian averaging. Given a test input $x$, the predicted output $\hat{y}$ is computed using Bayesian averaging:

$\hat{y}=\frac{1}{M}\sum _{m=1}^{M}f(x,{\tilde{\theta }}^m)$

(10)

where ${\tilde{\theta }}^m$’s are $M$ many samples from the sparse quantized sub-distribution. In the following experiments, we set the $M$ default to 4. Our ablation study (in Figure 3) shows that Bayesian averaging consistently yields smaller accuracy degradation than the greedy alternative (detailed in Equation 11).

Greedy approach is to greedily select the most likely weight for making predictions on the test set. Specifically, for each quantized weight ${\theta }_i$, we choose the index $k_i^{\ast }$ corresponding to the highest posterior probability ${\hat{\varphi }}_{k_i^{\ast }}\left({\theta }_i\right)$. The quantized weight is then set to the mean ${\mu }_{k_i^{\ast }}$ of the selected component, and the predicted output $\hat{y}$ is computed using these selected means. Formally, this greedy inference strategy is given by:

$\hat{y}=f(x,{\hat{\mu }}_{k_i^{\ast }}),k_i^{\ast }=\arg \underset{k}{\max }{\hat{\varphi }}_k\left({\theta }_i\right),Q\left({\theta }_i\right)={\mu }_{k_i^{\ast }}.$

(11)

We empirically compare the greedy inference approach with Bayesian averaging in the ablation study shown in Figure 3.

Outlier-aware windowing. Recent studies show that the weight distribution of large language models (LLMs) often contains significant outliers (Wei et al., 2022). To address this, we use an outlier-aware windowing strategy to enhance the performance of SQS. Specifically, the full-precision weights are partitioned into four groups using window sizes determined by a modified interquartile range (IQR) rule (Dekking et al., 2006), which helps preserve large-magnitude weights during quantization. Each group is then quantized to $K$ representative values. As shown in the ablation study (Figure 2), this strategy outperforms the approach using equal-sized windows. Implementation details are provided in Appendix C, and the full procedure is summarized in Algorithm 1.

媒体内容 · 前往原文查看

1: // Variational Learning

2: Training data $D$; Full-precision weights $({\theta }_1,\dots ,{\theta }_T)$; #components in GMM $K$; initial temperature $\tau ,{\tau }^{\prime }$; prior variance ${\sigma }_0^2$.

3: Initialize trainable parameters ${\left\{({\hat{\mu }}_k,{\hat{\pi }}_k,{\hat{\sigma }}_k)\right\}}_{k=1}^K$;

4: while not converged do

5:   calculate the approximate objective ${\Omega }_{\mathrm{𝚊𝚙𝚡}}$; $⊳$ in Equation (8)

6:   update learnable parameters with stochastic gradient descent; return the sparse quantized weight sub-distribution $\hat{q}\left(\tilde{\theta }\right)$.

7: // Variational Inference

8: A sparse quantized weight distribution $\hat{q}\left(\tilde{\theta }\right)$; #Bayesian average $M$; percentage of non-zero weights Non-zero rate (%).

9: for $m\leftarrow 1$ to $M$ do

10:   sample quantized weight ${\tilde{\theta }}^m$ from the posterior $\hat{q}\left(\tilde{\theta }\right)$; $⊳$ in Equation (9)

11:   get pruned weight ${\tilde{\theta }}^m$ with Non-zero, according to ${\hat{\lambda }}_i$;

12: predict output $\hat{y}$ for each testing input $x$, using Bayesian average with $\{{\tilde{\theta }}^1,\dots ,{\tilde{\theta }}^M\}$; $⊳$ in Equation (10)

13: return The set of predicted outputs.

Remarks. DGMS (Dong et al., 2022) adopts Gaussian mixtures, but uses them primarily as a clustering mechanism. In contrast, our method leverages a principled Bayesian framework that supports posterior inference and enables Bayesian model averaging, enhancing robustness to quantization noise. Furthermore, by unifying pruning and quantization within a spike-and-GMM variational family, our approach creates a joint optimization space that encourages globally optimal solutions across both pruning and quantization.

3.1.3 Windowing strategy in quantization

We observe that weight distributions vary significantly across layers, including Gaussian and long-tailed forms. In particular, long-tailed distributions contain a small subset of weights with large magnitudes. Previous works (Nagel et al., 2020; Hubara et al., 2021; Frantar et al., 2022) have demonstrated that layer-wise compression methods lead to better performance.

To address performance degradation arising from such heterogeneous distributions, we extend our proposed method to support layer-wise quantization, where each group of weight parameters within a layer is assigned its own quantization set. This enables each layer to learn and utilize a distinct, trainable quantization set tailored to its distribution.

Equal-size windowing. For the equal window strategy, given a layer of weights $\theta$, we group the weights into 4 windows where each one has an equal window size $\frac{\max \left(\theta \right)-\min \left(\theta \right)}{4}$. Within each window, a $K$-component GMM is applied to approximate the weight distribution.

Outlier-aware windowing. For layers with long-tailed distributions, we further introduce an outlier-aware windowing strategy. Specifically, the weights $\theta$ in each layer are partitioned into four windows, with two dedicated to capturing the lower and upper tails of the distribution. To identify these tail regions, we apply a standard outlier detection rule based on the $5\times$ inter quartile range (IQR): let $q_1$ and $q_3$ denote the first and third quartiles of the weights, and define $\text{IQR}=q_3-q_1$. The outlier-aware windows are then defined as

$\left[\min \left(\theta \right),q_1-5\times \text{IQR}\right],\text{ and }\left[q_3+5\times \text{IQR},\max \left(\theta \right)\right]$

(12)

Within each of the four windows in every layer, we fit a $K$-component GMM to approximate the local weight distribution.

We adopt the layer-wise quantization scheme with outlier-aware windowing in all our experiments. This approach improves the preservation of extreme values during quantization and enhances robustness across layers. An ablation study evaluating the effectiveness of outlier-aware windowing is presented in Figure 2.

3.2 Theoretical Justification of SQS

For clarity, this section focuses on regression tasks with fully connected neural networks. We analyze the variational posterior of sparse and quantized neural networks, i.e., the optimization of Equation (7). We show that this variational posterior converges to a true regression function under some mild conditions.

Consider a regression problem with random covariates,

$Y_i=f_0\left(X_i\right)+{\epsilon }_i,\text{ for }i=1,\dots ,n,$

(13)

where $f_0:{\left[0,1\right]}^p\to ℝ$ is the underlying unknown true function, $X_i\sim 𝒰\left({\left[0,1\right]}^p\right)$ is sampled from a $p$-dimensional uniform distribution, ${\epsilon }_i\stackrel{iid}{\sim }𝒩(0,{\sigma }_{\epsilon }^2)$ is the noise term from a Gaussian distribution of zero mean and variance ${\sigma }_{\epsilon }^2$. Let $P_0$ denote the true underlying probability measure of the data, and $p_0$ denote the corresponding density function. An $L$-hidden-layer fully connected NN with constant layer width $N$ and parameters $W_{\ell }\in {ℝ}^{N\times N}$, $b_{\ell }\in {ℝ}^N$ and activation function $\sigma (\cdot )$ can be defined as:

$f_{\theta }\left(X\right)=W_{L+1}{\sigma }_{b_L}\left(W_L{\sigma }_{b_{L-1}}\dots {\sigma }_{b_1}\left(W_{1}X\right)\right)+b_{L+1}.$

(14)

For simplicity, ${\sigma }_{\epsilon }$ is assumed to be known. Let $s^{\ast }$ be the “oracle” sparsity level (see Equation 19 in Appendix B for formal definition) and $H(T,s,K)$ be the set of network weight parameters such that the network has a sparsity of $s$ and shares at most $K$ distinct values. Let $P_0$ and $P_{\theta }$ be the true data distribution and the distribution under parameter $\theta$, respectively.

Theorem 1 carries a proof sketch stating the two-step structure: Lemma 1 upper-bounds the variational objective with high probability; Lemma 2 converts that bound into convergence of the variational posterior in squared Hellinger distance.

Lemma 1.

Under Conditions 1-3, with high probability,

$\underset{q\left(\theta \right)\in ℱ}{inf}\{\mathrm{KL}(q\left(\theta \right)\parallel \pi \left(\theta \right|\lambda ))+{\int }_{{ℝ}^T}{l}_n(P_0,P_{\theta })q\left(\theta \right)d\theta \}\le Cn(r_n^{\ast }+{\xi }_n^{\ast })$

where $C$ is either some positive constant if $limn\left(r_n^{\ast }+{\xi }_n^{\ast }\right)=\infty$, or any diverging sequence if $limsupn\left(r_n^{\ast }+{\xi }_n^{\ast }\right)\ne \infty$. And $l_n(P_0,P_{\theta })$ is defined as:

$l_n(P_0,P_{\theta })=\frac{1}{2{\sigma }_{\epsilon }^2}\left({\Vert Y-f_{\theta }\left(X\right)\Vert }_2^2-{\Vert Y-f_0\left(X\right)\Vert }_2^2\right).$

For any $\delta >1$, let $s_n:=s^{\ast }{\log }^{2\delta -1}\left(n\right)$.

Lemma 2.

Under Conditions 1-5, if ${\sigma }_0^2$ is set to be constant and $\lambda \le T^{-1}\exp \{-Mn{r}_n^{\ast }/s_n\}$ for any positive diverging sequence $M\to \infty$, then with high probability, then we have

$\int d^2(P_{\theta },P_0)\hat{q}\left(\theta \right)d\theta \le C{\epsilon }_n^{\phantom{}\ast 2}+\frac{3}{n}\underset{q\left(\theta \right)\in ℱ}{inf}\{\mathrm{KL}(q\left(\theta \right)\parallel \pi \left(\theta \right|\lambda ))+\int l_n(P_0,P_{\theta })q\left(\theta \right)d\theta \},$

(15)

where $C$ is some constant, and

${\epsilon }_n^{\ast }:={\epsilon }_n(L,N,s^{\ast })=\sqrt{r_n(L,N,s^{\ast })}{\log }^{\delta }\left(n\right),\text{ for any }\delta >1.$

We refer to Appendix B.1 for the proof of Lemma 1 and Appendix B.2 for the proof of Lemma 2.

Theorem 1.

Let $r_n^{\ast }=\left(\left(L+1\right)s^{\ast }/n\right)\log N+\left(s^{\ast }/n\log \left(p\sqrt{n/s^{\ast }}\right)\right)$, ${\epsilon }_n^{\ast }=\sqrt{r_n^{\ast }}{\log }^{\delta }\left(n\right)$ for any $\delta >1$ from Lemma 2, and ${\xi }_n^{\ast }={inf}_{\theta \in H(T,s,K),{\Vert \theta \Vert }_{\infty }\le B}{\Vert f_{\theta }-f_0\Vert }_{\infty }^2$. Then, under mild conditions specified in the supplementary material, with high probability:

${\int }_{{ℝ}^T}{d}^2(P_{\theta },P_0)\hat{q}\left(\theta \right)𝑑\theta \le C{\epsilon }_n^{\phantom{}\ast 2}+C^{\prime }\left(r_n^{\ast }+{\xi }_n^{\ast }\right),$

(16)

where $d(\cdot ,\cdot )$ denotes the Hellinger distance, and $C$ and $C^{\prime }$ are some constants.

Sketch of Proof.

Based on prior work (Bai et al., 2020), the proof proceeds in two steps. Lemma 1 establishes a high-probability bound on the ELBO in Equation (7). Lemma 2 connects this bound to the convergence of the variational distribution toward the true full-precision posterior. Together, these results show that the variational posterior induced by our method converges to the true regression function with high probability. The full proof is in Appendix B. ∎

Remark. Similar to previous Bayesian sparse DNN results (Bai et al., 2020; Chérief-Abdellatif, 2020), the convergence rate of variational Bayes is determined by the deep neural network structure via 1) statistical estimation error ${\epsilon }_n^{\ast }$, 2) variational error $r_n^{\ast }$, and 3) approximation error ${\xi }_n^{\ast }$. The first two are positively related to the network capacity, while the third one is negatively related to the network capacity. The estimation error ${\epsilon }_n^{\ast }$ and variational error $r_n^{\ast }$ vanish as $n\to \infty$. Prior work (Beknazaryan, 2022) shows that under $B\ge 2$, $K\ge 6$, and $\beta$-Hölder smoothness of $f_0$, the approximation error ${\xi }_n^{\ast }$ also vanishes.

While the theoretical analysis mainly considers an $L$-hidden-layer fully connected NN with constant layer width $N$, our method SQS is empirically validated on a variety of models such as ResNets, BERT-based models, and LLMs (refer to Section 5).

4 Related Work

Weight pruning was initially introduced by LeCun et al. (1989), with further development by Hassibi et al. (1993) through a mathematical method known as the Optimal Brain Surgeon (OBS). This approach selects weights for removal from a trained neural network using second-order information. Subsequent improvements, as indicated by studies (Dong et al., 2017; Wang et al., 2019; Singh and Alistarh, 2020), have adapted OBS for large-scale DNNs by employing numerical techniques to estimate the second-order information required by OBS. Meanwhile, Louizos et al. (2018) introduced an $L_0$-regularized method to promote sparsity in DNNs. Frankle and Carbin (2019) established a critical insight that within a randomly initialized DNN, an optimal sub-network can be identified and extracted. Recently, Xia et al. (2024) showed that structured pruning combined with targeted retraining can significantly reduce computational costs while preserving robust performance for large language models. Concurrently, spike-and-slab distributions have been employed to promote sparsity in DNNs using Bayesian Neural Networks formulation (Deng et al., 2019; Blundell et al., 2015; Bai et al., 2020).

Low-bit quantization. Quantization improves DNN efficiency, particularly in resource-constrained environments (Sze et al., 2017; Frantar et al., 2023; Lin et al., 2024; Lin et al., 2025). Research in this field typically follows two paradigms: discontinuous-mapping and continuous-mapping quantization. Discontinuous-mapping methods project full-precision weights onto a low-bit grid using rounding operations (Gupta et al., 2015; Hubara et al., 2018; Wu et al., 2018; Louizos et al., 2019; Courbariaux et al., 2015; De Sa et al., 2018; Marchesi et al., 1993). The non-differentiability of these mappings necessitates the use of the straight-through estimator (STE) for gradient approximation (Courbariaux and Bengio, 2016; Rastegari et al., 2016). However, STE-based training may introduce pseudo-gradients, leading to training instability (Yin et al., 2019). Meanwhile, many researchers propose post-training quantization methods that have limited access to the training dataset (Wang et al., 2020a; Hubara et al., 2021; Li et al., 2021; Frantar et al., 2022; Frantar et al., 2023; Lin et al., 2024).

Continuous-mapping quantization offers an alternative that avoids pseudo-gradients, leading to more stable training (Yin et al., 2019; Nielsen et al., 2025). These methods often use variational learning (Ullrich et al., 2017; Louizos et al., 2017; Shayer et al., 2018) or Markov Chain Monte Carlo techniques (Roth and Pernkopf, 2018) to approximate discrete weight distributions. However, variational methods often require manual prior specification (Ullrich et al., 2017; Louizos et al., 2017; Shayer et al., 2018), while MCMC approaches can be memory-intensive (Roth and Pernkopf, 2018). DGMS (Dong et al., 2022) addresses these limitations through automated quantization using GMMs. Our work extends DGMS by integrating pruning and quantization into a unified framework, thereby achieving higher compression rates.

Joint pruning and quantization. A growing line of work optimizes sparsity and precision jointly: Bayesian formulations derive both from a single prior or posterior (Ullrich et al., 2017; Louizos et al., 2017; Achterhold et al., 2018), while non-Bayesian approaches rely on differentiable gates, second-order saliency, or joint policy search (Wang et al., 2020c; Frantar et al., 2022; Wang et al., 2020b; Bai et al., 2023). Bayesian Bits (Van Baalen et al., 2020) unifies pruning and quantization by gating a chain of residual terms that double the bit-width, with pruning as the $0$-bit case. Unlike its gates on a uniform grid, SQS learns the levels themselves as GMM means with a per-weight retention probability, giving a non-uniform codebook, exact sparsity control, and a posterior for Bayesian averaging.

Large language model compression. Recent work on compressing large models has pursued several complementary directions. SpinQuant (Liu et al., 2025b) applies learned rotations to weights and embeddings to reduce outliers, making models easier to quantize. LeanQuant (Zhang and Shrivastava, 2025) introduces a loss-aware post-training quantization method that learns adaptive affine transformations and non-uniform quantization grids, aiming to preserve outlier-sensitive weights. EfficientXpert (Zhao et al., 2025) uses LoRA-guided pruning to obtain domain-aware pruned models. Xu et al. (2025) targeted vision–language models and adaptively pruned attention heads using entropy-based effective rank and the Kolmogorov–Smirnov distance, reporting substantial FLOP reductions. In contrast, our SQS introduces a learnable codebook and a spike-and-slab posterior tailored to sparse, quantized weights in the low-precision regime.

5 Experiments

In this section, we show that our SQS achieves a much higher compression rate (see the second-to-last column in Tables 1-3) while incurring a comparable or smaller performance drop (see the last column in the same tables). Through ablation studies, we further validate that (1) under the same sparsity level, the spike-and-slab prior more effectively preserves model accuracy (see Table 4). (2) Under identical hyperparameter settings, we show that SQS with Bayesian averaging outperforms the greedy approach (see Figure 3).

5.1 Experiment settings

We evaluate all methods using two metrics: the compression rate and the performance drop (i.e., accuracy drop or F1 score drop). The compression rate is defined as the memory footprint of the compressed model over the original dense full-precision model:

$\text{Compression rate}=\frac{32\times \text{original weight counts}}{{\log }_{2}K\times \text{nonzero weight counts}+32\times K},$

(17)

where $K$ is the codebook size. Non-zero rate is the percentage of weights that are pruned to zero. In all our experiments, the Non-zero rate is configured as a hyperparameter to control the sparsity. Equation (17) accounts for the codebook and the quantized value indices assigned to the nonzero weights, but, following the convention adopted by the pruning baselines we compare against (e.g., L-OBS (Dong et al., 2017), PLATON (Zhang et al., 2022) and ExactOBS/OBC (Frantar et al., 2022)), it does not separately charge bits for the binary sparsity mask that records which weights are pruned. We adopt this convention so that the reported compression rates in Tables 1–3 remain directly comparable to the values reported for these baselines in their original papers, all of which similarly define compression as a function of remaining/quantized parameter count alone. We note that some deployment-oriented compression pipelines, such as (Han et al., 2016; Dettmers et al., 2024), instead charge explicit bits for the sparsity structure (e.g., via a bitmap or compressed-sparse-row index); under that stricter accounting, absolute compression rates for all pruning-based methods, including ours, would be lower, though the relative ranking among methods is unaffected. In practice, for the models we study, the product “$\log {}_{2}K\times$ nonzero weight counts” is usually much larger than $32\times K$, because the number of nonzero weights is very large and the codebook size is small. In this regime, the memory used to store the indices dominates, and the memory used by the codebook is very small.

We compare methods of different compression types (the “Compression type” column): “P+Q” denotes combined pruning and quantization, “P” denotes pruning only, and “Q” denotes quantization only.

In our experiments, we store the indices of the weights using INT4, and we perform computation in FP32. This setting is common in prior work, such as QLoRA (Dettmers et al., 2023) and learned codebook methods (van den Oord et al., 2017; Dong et al., 2022).

To ensure a fair comparison, each method is initialized with the same full-precision pre-trained model and is run with the same set of hyperparameters for compression. All methods are constrained to a maximum runtime of 24 hours. The resulting compressed models are then evaluated on the same test sets, and the key performance metrics are summarized in the corresponding tables. Appendix D provides detailed experimental configurations and baseline settings.

5.2 Experimental analysis

Compression on ResNet models. Table 1 summarizes the result of all methods for compressing ResNet-20, ResNet-32 and ResNet-56, evaluated on the CIFAR-10 dataset. On compressing the ResNet-20 model, our SQS attains a better compression rate than the baselines. On compressing ResNet-32 and ResNet-56 models, our SQS attains substantially higher compression rates while incurring smaller accuracy drops compared to the baselines. Optimizing pruning and quantization separately overlooks redundancies in each step; by merging them into a single optimization, we effectively eliminate these inefficiencies.

媒体内容 · 前往原文查看

ResNet-20	Methods	Compression	Bits	Non-zero rate	Compression	Top-1 accuracy
		type		(%)	rate	drop
	LQNets (Zhang et al., 2018)	Q	$2$	$100\%$	$16\times$	$1.20\%$
	DGMS (Dong et al., 2022)	P+Q	$2$	$56\%$	$29\times$	$0.87\%$
	SQS (Ours)	P+Q	$2$	$\mathrm{𝟓𝟎}\%$	$\mathrm{𝟑𝟐}\times$	$1.47\%$
(a) Compressing 32Bits ResNet-20 model on CIFAR-10 dataset with Top-1 accuracy $92.60\%$.
ResNet-32	Method	Compression	Bits	Non-zero rate	Compression	Top-1 accuracy
		type		(%)	rate	drop
	TTQ (Zhu et al., 2017)	Q	$2$	$100\%$	$16\times$	$1.90\%$
	DGMS (Dong et al., 2022)	P+Q	$2$	$59\%$	$27\times$	$1.30\%$
	SQS (Ours)	P+Q	$2$	$\mathrm{𝟓𝟎}\%$	32$\times$	$1.29\%$
(b) Compressing 32Bits ResNet-32 model on CIFAR-10 dataset with Top-1 accuracy $93.53\%$.
ResNet-56	Method	Compression	Bits	Non-zero rate	Compression	Top-1 accuracy
		type		(%)	rate	drop
	TTQ (Zhu et al., 2017)	Q	2	$100\%$	16$\times$	$1.06\%$
	L1 (Li et al., 2017)	P	32	$10\%$	10$\times$	$1.83\%$
	DGMS (Dong et al., 2022)	P+Q	$2$	$52\%$	$31\times$	$0.89\%$
	SQS (Ours)	P+Q	2	$\mathrm{𝟓𝟎}\%$	$\mathrm{𝟑𝟐}\times$	$0.84\%$
(c) Compressing 32Bits ResNet-56 model on CIFAR-10 dataset with Top-1 accuracy $94.37\%$.

媒体内容 · 前往原文查看

BERT-base	Methods	Compression	Bits	Non-zero rate	Compression	F1 score
		type		(%)	rate	drop
	GMP (Zhu and Gupta, 2017)	P	$32$	$50\%$	$2\times$	$22.89$
	L-OBS (Dong et al., 2017)	P	$32$	$50\%$	$2\times$	$10.86$
	ExactOBS (Frantar et al., 2022)	P	$32$	$25\%$	$4\times$	$6.43$
	PLATON (Zhang et al., 2022)	P	$32$	$20\%$	$5\times$	$2.20$
	OBQ (Frantar et al., 2022)	Q	$3$	$100\%$	$11\times$	$3.24$
	GPTQ (Frantar et al., 2023)	Q	$3$	$100\%$	$11\times$	$2.51$
	OBC (Frantar et al., 2022)	P+Q	$4$	$50\%$	$16\times$	$2.33$
	SQS (Ours)	P+Q	4	$\mathrm{𝟐𝟓}\%$	$\mathrm{𝟑𝟐}\times$	$1.66$

Compression on BERT-base model. We apply our compression method to the BERT-base model (Devlin et al., 2019) and evaluate its performance on the SQuAD v1.1 dataset (Rajpurkar et al., 2016). The evaluation metrics include the F1 score drop and the compression rate. As shown in Table 2, our method achieves the lowest F1 score drop and the highest compression rate, outperforming existing methods. This demonstrates the effectiveness of our SQS method in preserving accuracy under aggressive compression.

媒体内容 · 前往原文查看

Llama3.2	Method	Compression	Bits	Non-zero rate	Compression	Top-1 accuracy
		type		(%)	rate	drop
	AWQ (Lin et al., 2024)	Q	$4$	$100\%$	$8\times$	$0.46\%$
	DGMS (Dong et al., 2022)	P+Q	$6$	$82\%$	$7\times$	$46.67\%$
	SQS (Ours)	P+Q	$6$	$\mathrm{𝟐𝟓}\%$	$\mathrm{𝟐𝟏}\times$	$1.48\%$
(a) Compressing 32Bits Llama3.2-1B model on SST-2 dataset with Top-1 accuracy $94.72\%$.
Qwen2.5	Method	Compression	Bits	Non-zero rate	Compression	Top-1 accuracy
		type		(%)	rate	drop
	AWQ (Lin et al., 2024)	Q	4	100%	$8\times$	$1.54\%$
	DGMS (Dong et al., 2022)	P+Q	$6$	$\mathrm{𝟑𝟒}\%$	$\mathrm{𝟏𝟔}\times$	$50.80\%$
	SQS (Ours)	P+Q	$6$	$50\%$	$11\times$	$2.46\%$
(b) Compressing 32Bits Qwen2.5-0.5B model on SST-2 dataset with Top-1 accuracy $92.60\%$.

Compression on Llama and Qwen models. In Table 3, we compare our SQS with others on the SST-2 task in the GLUE benchmark using Llama3.2-1B and Qwen2.5-0.5B models, as our method could preserve the weight outliers, which are crucial in maintaining the performance (Lin et al., 2024). We further observe that DGMS (Dong et al., 2022) incurs a large performance drop for compressing Llama3.2-1B and Qwen2.5-0.5B, which occurs because the weight distribution of the self-attention layer is not Gaussian (see Figure 2(left)), and it fails to capture large magnitude weights. Furthermore, DGMS does not allow customization of the sparsity level; thus, it presents an unreasonable performance drop.

媒体内容 · 前往原文查看

ResNet-18	Bits	Non-zero rate (%)	Compression rate	Top-1 accuracy drop
				Gaussian prior	Spike-and-slab prior (Ours)
	$4$	$50\%$	$16\times$	$4.51\%$	$3.12\%$
	$4$	$40\%$	$20\times$	$5.60\%$	$3.21\%$
	$4$	$30\%$	$27\times$	$11.42\%$	$5.54\%$
	$4$	$20\%$	$40\times$	$44.04\%$	$5.59\%$

媒体内容 · 前往原文查看

5.3 Ablation studies

Prior selection: Gaussian vs. spike-and-slab. We evaluate how the choice of prior affects compression performance, comparing a Gaussian prior (Appendix D.2) to the spike-and-slab prior in Equation (5). Specifically, we compress ResNet-18 at varying sparsity levels by representing each layer’s weights with $K=16$ components, and evaluate accuracy on CIFAR-100.

As shown in Table 4, the spike-and-slab prior consistently outperforms the Gaussian prior across all sparsity levels. The gap becomes particularly pronounced at high sparsity, where the Gaussian prior suffers substantial degradation, suggesting that it may be less effective at inducing posterior sparsity in DNN weights. We leave a deeper investigation of this behavior to future work.

Windowing strategy: equal-size vs. outlier-aware window. We use the first-layer attention weights of Llama3.2-1B as a case study. The full-precision weights exhibit a pronounced long-tail distribution, where a small fraction of entries have large magnitudes. Additional statistics on layer-wise weight distributions are reported in Appendix D.3. Motivated by this observation, our SQS uses an outlier-aware window strategy to better fit the full-precision weight distribution in Equation (12). Figure 2 visualizes the resulting quantized weights under the equal-window and outlier-aware window strategies.

媒体内容 · 前往原文查看

Qwen2.5	Method	Windowing strategy	Bits	Non-zero rate (%)	Top-1 accuracy drop $\downarrow$
	SQS	Outlier-aware window	$6$	$50\%$	$2.46\%$
		Equal-size window	$6$	$50\%$	$5.40\%$

As shown in Table 5, outlier-aware windowing reduces the Top-1 accuracy drop on Qwen2.5-0.5B from 5.40 to 2.46 percentage points at the same bit width and nonzero rate. The equal-size strategy spreads its windows uniformly over the weight range, so the few large-magnitude entries in the long tail fall into wide windows and are coarsely quantized, which is precisely the tail region that Figure 2 shows to be distorted. Preserving these outlier weights is therefore a primary driver of the accuracy that SQS retains on heavy-tailed LLM weights.

As shown in Figure 2 (left, middle), the quantized weights produced by SQS under the outlier-aware window strategy more closely match the full-precision distribution than those obtained with an equal-window strategy. Figure 2 (right) further highlights that the outlier-aware window improves fidelity in the tails, capturing extreme-magnitude weights more accurately than equal windowing.

Inference strategies: Bayesian averaging vs. greedy approach. We evaluate the effectiveness of two inference strategies within our SQS framework: (1) Bayesian averaging as defined in Equation (10), and (2) greedy approach as defined in Equation (11). To ensure a fair comparison, we assess the performance of compressing ResNet-18 and ResNet-50 models while varying the number of Gaussian components. The sparsity level is fixed to zero (i.e., no pruning), so that all performance degradation arises purely from quantization.

As shown in Figure 3, using fewer components results in a larger accuracy drop. Under the same number of components, SQS with Bayesian averaging consistently achieves a smaller accuracy drop compared to the greedy approach.

媒体内容 · 前往原文查看

Effect of the number of Bayesian-averaging samples. As shown in Table 6, the number of averaged posterior samples $M$ directly affects the robustness of the compressed model: a single posterior sample ($M=1$) incurs a $3.76\%$ Top-1 accuracy drop, whereas averaging over $M=50$ samples reduces the drop to $2.63\%$. Most of this benefit is realized with only a few samples, as $M=5$ already attains a $2.86\%$ drop and further increasing $M$ yields only marginal improvement. This indicates that a small number of posterior samples suffices in practice, and that Bayesian averaging mainly improves robustness to quantization noise rather than providing a large accuracy gain.

媒体内容 · 前往原文查看

Inference strategy	$M=1$	$M=5$	$M=20$	$M=50$
Top-1 accuracy Drop (%) $\downarrow$	$3.76$	$2.86$	$2.85$	$2.63$

媒体内容 · 前往原文查看

Method	Compression	Weight bits	Non-zero	Effective	Compression	Top-1 accuracy
	type		rate (%)	bits/weight	rate	drop
Bayesian Bits	P+Q	$2/4/8$ (mixed)	$100\%$	$\phantom{}\approx 2.36$	$\approx 13.5\times$	$6.79\%$
SQS (Ours)	P+Q	$2$	$\mathrm{𝟓𝟎}\%$	$1.87$	$17.0\times$	$0.84\%$

Comparison with Bayesian Bits. In Table 7, effective bits per weight is the average storage cost per weight of the original dense model. We define $b_{\mathrm{eff}}=B_{\mathrm{comp}}/T$, where $T$ is the original dense model’s weight count and $B_{\mathrm{comp}}$ is the total bit cost counted for the compressed weight representation in this comparison. Relative to an FP32 baseline, the corresponding compression rate is $32T/B_{\mathrm{comp}}=32/b_{\mathrm{eff}}$.

We use the official Bayesian Bits implementation released by Qualcomm AI Research¹¹ 1 https://github.com/Qualcomm-AI-research/BayesianBits and follow the published CIFAR-10 recipe without modification. We use a batch size of $128$ and Adam to optimize the weights, quantization, and gate parameters. Bayesian Bits is trained from scratch for $70$ epochs for $2$ hours, whereas SQS is initialized from a pre-trained model and trained for $11$ epochs.

We compare Bayesian Bits with SQS on ResNet-56/CIFAR-10, for which the full-precision model achieves $94.37\%$ Top-1 accuracy. As shown in Table 7, SQS achieves higher Top-1 accuracy and greater compression with fewer effective bits per weight.

6 Conclusion

In this paper, we proposed a unified framework for compressing full-precision DNNs by combining pruning and quantization into one integrated optimization process through variational learning. Unlike conventional approaches that apply pruning and quantization sequentially—often resulting in suboptimal solutions—our method jointly explores a broader solution space, achieving significantly higher compression rates with comparable performance degradation. To address the intractability of the original objective, we introduce an efficient approximation that enables scalable optimization. We evaluate our method across a range of benchmarks, including ResNets, BERT-base, Llama3.2, and Qwen2.5. Experimental results demonstrate that our approach consistently outperforms existing baselines in compression rate while maintaining competitive accuracy, highlighting its potential for efficient deployment in resource-constrained environments.

Broader Impact

Model compression can reduce memory use and may lower the cost of deploying DNNs on resource-constrained hardware. These benefits should be weighed against training-to-deployment trade-offs. Our method requires additional optimization after pretraining, which can increase training time and energy use before deployment. Inference also involves a design choice: Bayesian averaging can improve robustness to quantization noise, but using multiple posterior samples may increase latency compared with a single compressed model. Practical deployments should therefore measure end-to-end latency, memory use, energy consumption, and accuracy under the target hardware and workload.

SQS does not change the task or data distribution of the original model, so its safety risks largely inherit those of the base model. Compression may make models easier to deploy more widely, including in settings where monitoring is limited. For applications involving sensitive, high-stakes, or user-facing decisions, compressed models should be evaluated for robustness, fairness, privacy leakage, and failure modes after compression, rather than assuming that performance on aggregate benchmarks is sufficient.

Limitations

SQS is a compression-aware training method that optimizes its objective using task-specific training data. In our LLM experiments, the base Llama3.2-1B and Qwen2.5-0.5B models are first fine-tuned on SST-2 before compression; omitting this task-adaptation step leads to substantial performance degradation. Consequently, the reported results characterize the compression of task-adapted models rather than the preservation of the models’ general-purpose capabilities. Evaluating whether SQS maintains performance on unrelated tasks or under distribution shifts is beyond the scope of this study. In addition, our theoretical analysis is currently restricted to regression problems with fully connected neural networks and does not directly cover transformer architectures, classification settings, or other model families.

AI Use Statement

The authors used AI to assist with proofreading, grammar and mathematical-notation consistency checks, and the identification of potential derivation, citation, and presentation issues. All suggestions were reviewed and verified by the authors, who take full responsibility for the accuracy, integrity, and content of the manuscript.

References

• Achterhold et al. (2018) J. Achterhold, J. M. Koehler, A. Schmeink, and T. Genewein Variational network quantization. In ICLR, Cited by: §4.
• Bai et al. (2020) J. Bai, Q. Song, and G. Cheng Efficient variational inference for sparse deep learning with theoretical guarantee. In NeurIPS, Vol. 33, pp. 466–476. Cited by: §B.2, §B.2, §B.2, Appendix B, Appendix B, §3.1.2, §3.1, §3.2, §4, Theorem 1.
• Bai et al. (2023) S. Bai, J. Chen, X. Shen, Y. Qian, and Y. Liu Unified data-free compression: pruning and quantization without fine-tuning. In ICCV, pp. 5876–5885. Cited by: §1, §4.
• Bai et al. (2022) Y. Bai, H. Wang, Z. Tao, K. Li, and Y. Fu Dual lottery ticket hypothesis. In ICLR, Cited by: §1.
• Banner et al. (2018) R. Banner, I. Hubara, E. Hoffer, and D. Soudry Scalable methods for 8-bit training of neural networks. In NeurIPS, Vol. 31, pp. 5151–5159. Cited by: §1.
• Beknazaryan (2022) A. Beknazaryan Function approximation by deep neural networks with parameters {0, $\pm$½, $\pm$1, 2}. Journal of Statistical Theory and Practice 16 (1), pp. 7. Cited by: §3.2.
• Blei et al. (2017) D. M. Blei, A. Kucukelbir, and J. D. McAuliffe Variational inference: a review for statisticians. Journal of the American statistical Association 112 (518), pp. 859–877. Cited by: §2.
• Blundell et al. (2015) C. Blundell, J. Cornebise, K. Kavukcuoglu, and D. Wierstra Weight uncertainty in neural network. In ICML, Vol. 37, pp. 1613–1622. Cited by: §4.
• Boucheron et al. (2013) S. Boucheron, G. Lugosi, and P. Massart Concentration inequalities: a nonasymptotic theory of independence. Oxford University press. Cited by: §B.2.
• Buciluǎ et al. (2006) C. Buciluǎ, R. Caruana, and A. Niculescu-Mizil Model compression. In KDD, pp. 535–541. Cited by: §1.
• Chérief-Abdellatif and Alquier (2018) B. Chérief-Abdellatif and P. Alquier Consistency of variational bayes inference for estimation and model selection in mixtures. Electronic Journal of Statistics 12 (2), pp. 2995 – 3035. Cited by: §A.1, Lemma 3.
• Chérief-Abdellatif (2020) B. Chérief-Abdellatif Convergence rates of variational inference in sparse deep learning. In ICML, Vol. 119, pp. 1831–1842. Cited by: §B.1, Appendix B, §3.2.
• Choudhary et al. (2020) T. Choudhary, V. Mishra, A. Goswami, and J. Sarangapani A comprehensive survey on model compression and acceleration. Artificial Intelligence Review 53, pp. 5113–5155. Cited by: §1.
• Courbariaux et al. (2015) M. Courbariaux, Y. Bengio, and J. David Binaryconnect: training deep neural networks with binary weights during propagations. In NeurIPS, Vol. 28, pp. 3123–3131. Cited by: §1, §2, §4.
• Courbariaux and Bengio (2016) M. Courbariaux and Y. Bengio BinaryNet: training deep neural networks with weights and activations constrained to +1 or -1. CoRR abs/1602.02830. Cited by: §4.
• Csiszar (1975) I. Csiszar $I$-Divergence Geometry of Probability Distributions and Minimization Problems. The Annals of Probability 3 (1), pp. 146 – 158. Cited by: §2.
• De Sa et al. (2018) C. De Sa, M. Leszczynski, J. Zhang, A. Marzoev, C. R. Aberger, K. Olukotun, and C. Ré High-accuracy low-precision training. arXiv preprint arXiv:1803.03383. Cited by: §4.
• Dekking et al. (2006) F.M. Dekking, C. Kraaikamp, H.P. Lopuhaä, and L.E. Meester A modern introduction to probability and statistics: understanding why and how. Springer Texts in Statistics, Springer London. External Links: LCCN 2004057700 Cited by: §3.1.2.
• Deng et al. (2019) W. Deng, X. Zhang, F. Liang, and G. Lin An adaptive empirical bayesian method for sparse deep learning. In NeurIPS, Vol. 32, pp. 5564–5574. Cited by: §4.
• Dettmers et al. (2023) T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer QLoRA: efficient finetuning of quantized llms. In NeurIPS, Vol. 36, pp. 10088–10115. Cited by: §1, §5.1.
• Dettmers et al. (2024) T. Dettmers, R. Svirschevski, V. Egiazarian, D. Kuznedelev, E. Frantar, S. Ashkboos, A. Borzunov, T. Hoefler, and D. Alistarh SpQR: A sparse-quantized representation for near-lossless LLM weight compression. In ICLR, Cited by: §5.1.
• Devlin et al. (2019) J. Devlin, M. Chang, K. Lee, and K. Toutanova BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL, pp. 4171–4186. Cited by: §D.1, §5.2.
• Dong et al. (2022) R. Dong, Z. Tan, M. Wu, L. Zhang, and K. Ma Finding the task-optimal low-bit sub-distribution in deep neural networks. In ICML, Vol. 162, pp. 5343–5359. Cited by: 2nd item, §1, §2, §2, §2, §3.1.2, §4, §5.1, §5.2, Table 1, Table 1, Table 1, Table 3, Table 3.
• Dong et al. (2017) X. Dong, S. Chen, and S. Pan Learning to prune deep neural networks via layer-wise optimal brain surgeon. In NeurIPS, Vol. 30, pp. 4857–4867. Cited by: §1, §4, §5.1, Table 2.
• Frankle and Carbin (2019) J. Frankle and M. Carbin The lottery ticket hypothesis: finding sparse, trainable neural networks. In ICLR, Cited by: §4.
• Frantar et al. (2023) E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh OPTQ: accurate quantization for generative pre-trained transformers. In ICLR, Cited by: 6th item, §1, §4, Table 2.
• Frantar et al. (2022) E. Frantar, S. P. Singh, and D. Alistarh Optimal brain compression: a framework for accurate post-training quantization and pruning. In NeurIPS, Vol. 35, pp. 4475–4488. Cited by: 5th item, §1, §3.1.3, §4, §4, §5.1, Table 2, Table 2, Table 2.
• Gholami et al. (2022) A. Gholami, S. Kim, Z. Dong, Z. Yao, M. W. Mahoney, and K. Keutzer A survey of quantization methods for efficient neural network inference. In Low-power computer vision, pp. 291–326. Cited by: §2.
• Gou et al. (2021) J. Gou, B. Yu, S. J. Maybank, and D. Tao Knowledge distillation: a survey. International Journal of Computer Vision 129 (6), pp. 1789–1819. Cited by: §1.
• Guo et al. (2016) Y. Guo, A. Yao, and Y. Chen Dynamic network surgery for efficient dnns. In NeurIPS, Vol. 29, pp. 1379–1387. Cited by: §1.
• Gupta et al. (2015) S. Gupta, A. Agrawal, K. Gopalakrishnan, and P. Narayanan Deep learning with limited numerical precision. In ICML, Vol. 37, pp. 1737–1746. Cited by: §4.
• Han et al. (2016) S. Han, H. Mao, and W. J. Dally Deep compression: compressing deep neural network with pruning, trained quantization and huffman coding. In ICLR, Cited by: §1, §5.1.
• Hassibi et al. (1993) B. Hassibi, D. G. Stork, and G. J. Wolff Optimal brain surgeon and general network pruning. In IEEE International Conference on Neural Networks, pp. 293–299. Cited by: 1st item, §4.
• Hershey and Olsen (2007) J. R. Hershey and P. A. Olsen Approximating the kullback leibler divergence between gaussian mixture models. In IEEE International Conference on Acoustics, Speech, and Signal Processing, pp. 317–320. Cited by: §A.1.
• Hubara et al. (2018) I. Hubara, M. Courbariaux, D. Soudry, R. El-Yaniv, and Y. Bengio Quantized neural networks: training neural networks with low precision weights and activations. Journal of Machine Learning Research 18 (187), pp. 1–30. Cited by: §4.
• Hubara et al. (2021) I. Hubara, Y. Nahshan, Y. Hanani, R. Banner, and D. Soudry Accurate post training quantization with small calibration sets. In ICML, Vol. 139, pp. 4466–4475. Cited by: 3rd item, §3.1.3, §4.
• Ishwaran and Rao (2005) H. Ishwaran and J. S. Rao Spike and slab variable selection: frequentist and bayesian strategies. The Annals of Statistics 33, pp. 730–773. Cited by: §3.1.
• Jacob et al. (2018) B. Jacob, S. Kligys, B. Chen, M. Zhu, M. Tang, A. Howard, H. Adam, and D. Kalenichenko Quantization and training of neural networks for efficient integer-arithmetic-only inference. In CVPR, pp. 2704–2713. Cited by: §2.
• Jordan et al. (1999) M. I. Jordan, Z. Ghahramani, T. S. Jaakkola, and L. K. Saul An introduction to variational methods for graphical models. Machine learning 37 (2), pp. 183–233. Cited by: §2.
• Kumar et al. (2025) T. Kumar, Z. Ankner, B. F. Spector, B. Bordelon, N. Muennighoff, M. Paul, C. Pehlevan, C. Re, and A. Raghunathan Scaling laws for precision. In ICLR, Cited by: §1.
• LeCun et al. (1989) Y. LeCun, J. Denker, and S. Solla Optimal brain damage. In NeurIPS, Vol. 2, pp. 598–605. Cited by: §1, §4.
• Li et al. (2017) H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf Pruning filters for efficient convnets. In ICLR, Cited by: Table 1.
• Li et al. (2021) Y. Li, R. Gong, X. Tan, Y. Yang, P. Hu, Q. Zhang, F. Yu, W. Wang, and S. Gu BRECQ: pushing the limit of post-training quantization by block reconstruction. In ICLR, Cited by: 4th item, §4.
• Lin et al. (2024) J. Lin, J. Tang, H. Tang, S. Yang, W. Chen, W. Wang, G. Xiao, X. Dang, C. Gan, and S. Han AWQ: activation-aware weight quantization for on-device LLM compression and acceleration. In Annual Conference on Machine Learning and Systems, Cited by: 1st item, §1, §4, §5.2, Table 3, Table 3.
• Lin et al. (2025) M. Lin, S. Guan, W. Jing, G. Botterweck, and A. Patane Stochastic weight sharing for bayesian neural networks. In AISTATS, Proceedings of Machine Learning Research, Vol. 258, pp. 4519–4527. Cited by: §4.
• Liu et al. (2018) C. Liu, B. Zoph, M. Neumann, J. Shlens, W. Hua, L. Li, L. Fei-Fei, A. Yuille, J. Huang, and K. Murphy Progressive neural architecture search. In ECCV, pp. 19–34. Cited by: §1.
• Liu et al. (2025a) K. Liu, Q. Zheng, K. Tao, Z. Li, H. Qin, W. Li, Y. Guo, X. Liu, L. Kong, G. Chen, Y. Zhang, and X. Yang Low-bit model quantization for deep neural networks: a survey. External Links: 2505.05530 Cited by: §1.
• Liu et al. (2025b) Z. Liu, C. Zhao, I. Fedorov, B. Soran, D. Choudhary, R. Krishnamoorthi, V. Chandra, Y. Tian, and T. Blankevoort SpinQuant: LLM quantization with learned rotations. In ICLR, Cited by: §4.
• Louizos et al. (2019) C. Louizos, M. Reisser, T. Blankevoort, E. Gavves, and M. Welling Relaxed quantization for discretized neural networks. In ICLR, Cited by: §4.
• Louizos et al. (2017) C. Louizos, K. Ullrich, and M. Welling Bayesian compression for deep learning. In NeurIPS, Vol. 30, pp. 3288–3298. Cited by: §4, §4.
• Louizos et al. (2018) C. Louizos, M. Welling, and D. P. Kingma Learning sparse neural networks through $L_0$ regularization. In ICLR, Cited by: §4.
• Marchesi et al. (1993) M. Marchesi, G. Orlandi, F. Piazza, and A. Uncini Fast neural networks without multipliers. IEEE transactions on Neural Networks 4 (1), pp. 53–62. Cited by: §4.
• Nagel et al. (2020) M. Nagel, R. A. Amjad, M. Van Baalen, C. Louizos, and T. Blankevoort Up or down? adaptive rounding for post-training quantization. In ICML, Vol. 119, pp. 7197–7206. Cited by: §3.1.3.
• Nielsen et al. (2025) J. Nielsen, P. Schneider-Kamp, and L. Galke Continual quantization-aware pre-training: when to transition from 16-bit to 1.58-bit pre-training for bitnet language models?. In ACL (Findings), pp. 13483–13493. Cited by: §4.
• Park et al. (2019) W. Park, D. Kim, Y. Lu, and M. Cho Relational knowledge distillation. In CVPR, pp. 3967–3976. Cited by: §1.
• Radford et al. (2018) A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever Improving language understanding by generative pre-training. Technical Report OpenAI. Cited by: §1.
• Rajpurkar et al. (2016) P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang SQuAD: 100,000+ questions for machine comprehension of text. In EMNLP, pp. 2383–2392. Cited by: §D.1, §5.2.
• Rastegari et al. (2016) M. Rastegari, V. Ordonez, J. Redmon, and A. Farhadi Xnor-net: imagenet classification using binary convolutional neural networks. In ECCV, pp. 525–542. Cited by: §1, §4.
• Roth and Pernkopf (2018) W. Roth and F. Pernkopf Bayesian neural networks with weight sharing using dirichlet processes. IEEE Transactions on Pattern Analysis and Machine Intelligence 42 (1), pp. 246–252. Cited by: §4.
• Shayer et al. (2018) O. Shayer, D. Levi, and E. Fetaya Learning discrete weights using the local reparameterization trick. In ICLR, Cited by: §4.
• Singh and Alistarh (2020) S. P. Singh and D. Alistarh Woodfisher: efficient second-order approximation for neural network compression. In NeurIPS, Vol. 33, pp. 18098–18109. Cited by: §4.
• Sun et al. (2019) X. Sun, J. Choi, C. Chen, N. Wang, S. Venkataramani, V. V. Srinivasan, X. Cui, W. Zhang, and K. Gopalakrishnan Hybrid 8-bit floating point (hfp8) training and inference for deep neural networks. In NeurIPS, Vol. 32, pp. 4901–4910. Cited by: §1.
• Sun et al. (2022) Y. Sun, Q. Song, and F. Liang Consistent sparse deep learning: theory and computation. Journal of the American Statistical Association 117 (540), pp. 1981–1995. Cited by: §3.1.2.
• Sze et al. (2017) V. Sze, Y. Chen, T. Yang, and J. S. Emer Efficient processing of deep neural networks: a tutorial and survey. Proceedings of the IEEE 105 (12), pp. 2295–2329. Cited by: §4.
• Touvron et al. (2023) H. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: open and efficient foundation language models. arXiv preprint arXiv:2302.13971. Cited by: §1.
• Ullrich et al. (2017) K. Ullrich, E. Meeds, and M. Welling Soft weight-sharing for neural network compression. In ICLR, Cited by: §4, §4.
• Van Baalen et al. (2020) M. Van Baalen, C. Louizos, M. Nagel, R. A. Amjad, Y. Wang, T. Blankevoort, and M. Welling Bayesian bits: unifying quantization and pruning. In NeurIPS, Vol. 33, pp. 5741–5752. Cited by: §4.
• van den Oord et al. (2017) A. van den Oord, O. Vinyals, and K. Kavukcuoglu Neural discrete representation learning. In NeurIPS, Vol. 30. Cited by: §5.1.
• Wang et al. (2019) C. Wang, R. Grosse, S. Fidler, and G. Zhang Eigendamage: structured pruning in the kronecker-factored eigenbasis. In ICML, Vol. 97, pp. 6566–6575. Cited by: §4.
• Wang et al. (2020a) P. Wang, Q. Chen, X. He, and J. Cheng Towards accurate post-training network quantization via bit-split and stitching. In ICML, Vol. 119, pp. 9847–9856. Cited by: 2nd item, §4.
• Wang et al. (2025) R. Wang, Y. Gong, X. Liu, G. Zhao, Z. Yang, B. Guo, Z. Zha, and P. Cheng Optimizing large language model training using FP4 quantization. In ICML, Proceedings of Machine Learning Research, Vol. 267, pp. 62937–62957. Cited by: §1.
• Wang et al. (2020b) T. Wang, K. Wang, H. Cai, J. Lin, Z. Liu, H. Wang, Y. Lin, and S. Han Apq: joint search for network architecture, pruning and quantization policy. In CVPR, pp. 2078–2087. Cited by: §1, §1, §4.
• Wang et al. (2020c) Y. Wang, Y. Lu, and T. Blankevoort Differentiable joint pruning and quantization for hardware efficiency. In ECCV, pp. 259–277. Cited by: §1, §4.
• Wei et al. (2022) X. Wei, Y. Zhang, X. Zhang, R. Gong, S. Zhang, Q. Zhang, F. Yu, and X. Liu Outlier suppression: pushing the limit of low-bit transformer language models. In NeurIPS, Vol. 35, pp. 17402–17414. Cited by: §3.1.2.
• Wortsman et al. (2023) M. Wortsman, T. Dettmers, L. Zettlemoyer, A. Morcos, A. Farhadi, and L. Schmidt Stable and low-precision training for large-scale vision-language models. In NeurIPS, Vol. 36, pp. 10271–10298. Cited by: §1.
• Wu et al. (2018) S. Wu, G. Li, F. Chen, and L. Shi Training and inference with integers in deep neural networks. In ICLR, Cited by: §4.
• Xia et al. (2024) M. Xia, T. Gao, Z. Zeng, and D. Chen Sheared LLaMA: accelerating language model pre-training via structured pruning. In ICLR, Cited by: §4.
• Xu et al. (2020) C. Xu, W. Zhou, T. Ge, F. Wei, and M. Zhou BERT-of-theseus: compressing BERT by progressive module replacing. In EMNLP, pp. 7859–7869. Cited by: §1.
• Xu et al. (2025) Z. Xu, Y. Zhang, J. Li, J. Guo, Q. Zhu, and H. Huang Towards efficient vlms: information-theoretic driven compression via adaptive structural pruning. arXiv preprint arXiv:2511.19518. Cited by: §4.
• Yin et al. (2019) P. Yin, J. Lyu, S. Zhang, S. J. Osher, Y. Qi, and J. Xin Understanding straight-through estimator in training activation quantized neural nets. In ICLR, Cited by: §4, §4.
• You et al. (2019) Z. You, K. Yan, J. Ye, M. Ma, and P. Wang Gate decorator: global filter pruning method for accelerating deep convolutional neural networks. In NeurIPS, Vol. 32, pp. 2130–2141. Cited by: §1.
• Zhang et al. (2018) D. Zhang, J. Yang, D. Ye, and G. Hua Lq-nets: learned quantization for highly accurate and compact deep neural networks. In ECCV, pp. 365–382. Cited by: Table 1.
• Zhang et al. (2022) Q. Zhang, S. Zuo, C. Liang, A. Bukharin, P. He, W. Chen, and T. Zhao Platon: pruning large transformer models with upper confidence bound of weight importance. In ICML, Vol. 162, pp. 26809–26823. Cited by: §5.1, Table 2.
• Zhang and Shrivastava (2025) T. Zhang and A. Shrivastava LeanQuant: accurate and scalable large language model quantization with loss-error-aware grid. In ICLR, Cited by: §4.
• Zhao et al. (2025) S. Zhao, M. Pitts, and Z. Qin EfficientXpert: efficient domain adaptation for large language models via propagation-aware pruning. arXiv preprint arXiv:2511.19935. Cited by: §4.
• Zhu et al. (2017) C. Zhu, S. Han, H. Mao, and W. J. Dally Trained ternary quantization. In ICLR, Cited by: Table 1, Table 1.
• Zhu and Gupta (2017) M. Zhu and S. Gupta To prune, or not to prune: exploring the efficacy of pruning for model compression. arXiv preprint arXiv:1710.01878. Cited by: Table 2.

Appendix A Derivation of Approximate Objective

媒体内容 · 前往原文查看

Symbol	Meaning
${\theta }_i$	Pre-trained full-precision weight that is fixed and provided as input to the compression procedure.
${\tilde{\theta }}_i$	Random sparse and quantized weight learned during compression.
$q\left({\tilde{\theta }}_i\right)$	Marginal variational distribution of ${\tilde{\theta }}_i$, defined by the spike-and-GMM model in Equation (6).
$\pi \left({\tilde{\theta }}_i\right)$	Spike-and-slab prior defined in Equation (5).
${\gamma }_i$; $\lambda$; ${\tilde{\lambda }}_i$	Keep/prune indicator, prior retention probability, and variational retention probability, respectively.
${\varphi }_k\left({\theta }_i\right)$	Responsibility of GMM component $k$ for weight $i$, evaluated as a function of the fixed weight ${\theta }_i$.
${\mu }_k,{\sigma }_k^2$	Learnable component mean and variance; the means ${\mu }_k$ define the quantization levels.
${\sigma }_0^2$	Variance of the Gaussian slab in the spike-and-slab prior.
$\widehat{(\cdot )}$	Quantity estimated after optimization, such as ${\hat{\mu }}_k$, ${\hat{\lambda }}_i$, or ${\hat{\varphi }}_k$.
$K$, $T$, $M$	Number of GMM components, total number of weights, and number of posterior samples used for Bayesian averaging, respectively.

A.1 An upper bound on the KL divergence between two mixtures

To simplify the ELBO and validate our approach, we reformulate a key lemma from previous work (Chérief-Abdellatif and Alquier, 2018, Lemma 6.1). This Lemma is a tool widely used in signal processing (Hershey and Olsen, 2007). We provide the proof for the sake of completeness.

Lemma 3(From Lemma 6.1 in (Chérief-Abdellatif and Alquier, 2018) ).

For any $K>0$, the KL divergence between any two mixture densities ${\sum }_{k=1}^K{w}_k{g}_k$ and ${\sum }_{k=1}^K{\tilde{w}}_k{\tilde{g}}_k$ is upper bounded by

$\mathrm{KL}(\sum _{k=1}^K{w}_k{g}_k\parallel \sum _{k=1}^K{\tilde{w}}_k{\tilde{g}}_k)\le \sum _{k=1}^K{w}_k\mathrm{KL}(g_k\parallel {\tilde{g}}_k)+\sum _{k=1}^K{w}_k\log \left(\frac{w_k}{{\tilde{w}}_k}\right)$

Proof.

We expand the KL divergence term by its definition and obtain:

	$\mathrm{KL}(\sum {}_{k=1}{}^{K}w{}_{k}g{}_k\parallel \sum {}_{k=1}{}^K\tilde{w}{}_k\tilde{g}{}_k)$	$\phantom{}=\int \left(\sum _{k=1}^K{w}_k{g}_k\right)\log \left(\frac{{\sum }_{k=1}^K{w}_k{g}_k}{{\sum }_{k=1}^K{\tilde{w}}_k{\tilde{g}}_k}\right)$
		$\phantom{}\le \int \sum _{k=1}^K{w}_k{g}_k\log \left(\frac{w_k{g}_k}{{\tilde{w}}_k{\tilde{g}}_k}\right)$
		$\phantom{}=\int \sum _{k=1}^K{w}_k{g}_k\log \left(\frac{g_k}{{\tilde{g}}_k}\right)+\int \sum _{k=1}^K{w}_k{g}_k\log \left(\frac{w_k}{{\tilde{w}}_k}\right)$
		$\phantom{}=\sum _{k=1}^K{w}_k\int g_k\log \left(\frac{g_k}{{\tilde{g}}_k}\right)+\sum _{k=1}^K{w}_k\log \left(\frac{w_k}{{\tilde{w}}_k}\right)\left(\int g_k\right)$
		$=\sum {}^K{}_{k=1}w{}_k\mathrm{KL}(g{}_k\parallel \tilde{g}{}_k)+\sum {}_{k=1}{}^{K}w{}_k\log \left(\frac{w_k}{{\tilde{w}}_k}\right),$

where the first inequality is due to Jensen’s inequality and the convexity of the function $x\log \left(x\right)$. This completes the proof. ∎

A.2 Derivation of Approximate Objective

We aim to approximate the ELBO objective:

$\Omega \left(\tilde{\theta }\right)=-\underset{\mathrm{𝙿𝚊𝚛𝚝}\mathrm{\hspace{0.33em}1}}{\underbrace{{𝔼}_{q\left(\tilde{\theta }\right)}\left[\log p\left(D\right|\tilde{\theta })\right]}}+\sum _{i=1}^T\mathrm{KL}(q\left({\tilde{\theta }}_i\right)\parallel \pi \left({\tilde{\theta }}_i\right)).$

(18)

where $\pi \left({\tilde{\theta }}_i\right)$ is defined in Equation (5) and $q\left({\tilde{\theta }}_i\right)$ is defined in Equation (6):

	$q\left({\tilde{\theta }}_i\right)$	$\phantom{}={\tilde{\lambda }}_i\sum _{k=1}^K{\varphi }_k\left({\theta }_i\right)𝒩({\mu }_k,{\sigma }_k^2)+\left(1-{\tilde{\lambda }}_i\right){\delta }_0,$
	$\pi \left({\tilde{\theta }}_i\right)$	$\phantom{}=\lambda 𝒩(0,{\sigma }_0^2)+\left(1-\lambda \right){\delta }_0.$

It is important to note that the KL divergence between the variational distribution and the spike-and-slab prior distribution does not have a closed-form solution.

Step 1: Approximate the expected log-likelihood. The first term ${𝔼}_{q\left(\tilde{\theta }\right)}\left[\log p(D\mid \tilde{\theta })\right]$ can be expensive to compute because it requires sampling from the spike-and-GMM distribution. We therefore use a plug-in approximation based on the posterior mean. For each coordinate $i=1,\dots ,T$,

${𝔼}_{q\left({\tilde{\theta }}_i\right)}\left[{\tilde{\theta }}_i\right]={\tilde{\lambda }}_i\sum _{k=1}^K{\mu }_k{\varphi }_k\left({\theta }_i\right).$

Collecting these coordinate-wise means gives the complete mean parameter vector

${\theta }^{\mathrm{avg}}:={𝔼}_{q\left(\tilde{\theta }\right)}\left[\tilde{\theta }\right]={\left({\tilde{\lambda }}_i\sum _{k=1}^K{\mu }_k{\varphi }_k({\theta }_i,\pi ,\tau )\right)}_{i=1}^T\in {ℝ}^T.$

The plug-in approximation is therefore

$\mathrm{𝙿𝚊𝚛𝚝}\mathrm{\hspace{0.33em}1}\approx \log p(D\mid {\theta }^{\mathrm{avg}}).$

Step 2: Upper Bound KL between spike-and-slab distributions. The KL divergence between the marginal variational posterior and the prior is intractable due to the presence of both the Dirac delta and the mixture components. To upper-bound the KL divergence between them, we apply Lemma 3 by matching component structure:

$\mathrm{KL}(q\parallel \pi )$

$\le \sum _{j=1}^2{w}_j\mathrm{KL}(g_j\parallel {\tilde{g}}_j)+\sum _{j=1}^2{w}_j\log \left(\frac{w_j}{{\tilde{w}}_j}\right)$

	with	$w_1={\tilde{\lambda }}_i,$		$g_1=\sum _{k=1}^K{\varphi }_k\left({\theta }_i\right)𝒩({\mu }_k,{\sigma }_k^2),$		${\tilde{w}}_1=\lambda ,$		${\tilde{g}}_1=𝒩(0,{\sigma }_0^2),$
		$w_2=1-{\tilde{\lambda }}_i,$		$g_2={\delta }_0,$		${\tilde{w}}_2=1-\lambda ,$		${\tilde{g}}_2={\delta }_0.$

Substituting into the bound, we obtain:

	$\mathrm{KL}(q\left({\tilde{\theta }}_i\right)\parallel \pi \left({\tilde{\theta }}_i\right))\le$	${\tilde{\lambda }}_i\mathrm{KL}(\sum _{k=1}^K{\varphi }_k\left({\theta }_i\right)𝒩({\mu }_k,{\sigma }_k^2)\parallel 𝒩(0,{\sigma }_0^2))+(1-{\tilde{\lambda }}_i)\underset{\phantom{}=0}{\underbrace{\mathrm{KL}(\delta {}_0\parallel \delta {}_0)}}$
		$+{\tilde{\lambda }}_i\log \frac{{\tilde{\lambda }}_i}{\lambda }+\left(1-{\tilde{\lambda }}_i\right)\log \frac{1-{\tilde{\lambda }}_i}{1-\lambda }.$

Combining the terms, we have:

$\mathrm{KL}(q\left({\tilde{\theta }}_i\right)\parallel \pi \left({\tilde{\theta }}_i\right))\le {\tilde{\lambda }}_i\mathrm{KL}(\sum _{k=1}^K{\varphi }_k\left({\theta }_i\right)𝒩({\mu }_k,{\sigma }_k^2)\parallel 𝒩(0,{\sigma }_0^2))+\mathrm{KL}(\mathrm{Bern}\left({\tilde{\lambda }}_i\right)\parallel \mathrm{Bern}\left(\lambda \right)),$

Note that the first term on the right-hand side, which is the KL divergence between the GMM and the Gaussian distribution, does not have a closed form. But it can be further upper-bounded as:

	$\mathrm{KL}$	$(\sum _{k=1}^K{\varphi }_k\left({\theta }_i\right)𝒩({\mu }_k,{\sigma }_k^2)\parallel 𝒩(0,{\sigma }_0^2))=\mathrm{KL}(\sum _{k=1}^K{\varphi }_k\left({\theta }_i\right)𝒩({\mu }_k,{\sigma }_k^2)\parallel \sum _{k=1}^K{\varphi }_k\left({\theta }_i\right)𝒩(0,{\sigma }_0^2))$
	$\le$	$\sum _{k=1}^K{\varphi }_k\left({\theta }_i\right)\mathrm{KL}(𝒩({\mu }_k,{\sigma }_k^2)\parallel 𝒩(0,{\sigma }_0^2))+\sum _{k=1}^K{\varphi }_k\left({\theta }_i\right)\underset{\phantom{}=0}{\underbrace{\log \left(\frac{{\varphi }_k\left({\theta }_i\right)}{{\varphi }_k\left({\theta }_i\right)}\right)}}$
	$=$	$\underset{\mathrm{𝙿𝚊𝚛𝚝}2}{\underbrace{\sum {}_{k=1}{}^K\varphi {}_k\left(\theta {}_i\right)\mathrm{KL}\left(𝒩\right(\mu {}_k,\sigma {}^2{}_k)\parallel 𝒩(0,\sigma {}^2{}_0\left)\right)}},$

where the inequality is obtained by Lemma 3. Empirically, we approximate the mixture KL by evaluating only the dominant component:

$\mathrm{𝙿𝚊𝚛𝚝}\mathrm{\hspace{0.33em}2}\approx \mathrm{KL}(𝒩({\mu }_{k_i^{\ast }},{\sigma }_{k_i^{\ast }}^2)\parallel 𝒩(0,{\sigma }_0^2)),\text{where }{k}_i^{\ast }={\arg \max }_{1\le k\le K}{\varphi }_k\left({\theta }_i\right)$

We approximate the inner sum over $k$ using the maximum-weight component, which is the $k^{\ast }$-th component.

A small temperature is needed to avoid a flat posterior distribution, which could introduce large differences between the training phase and inference phase.

Finally, putting all approximations together, we obtain:

${\Omega }_{\mathrm{𝚊𝚙𝚡}}\left(\tilde{\theta }\right)=-\log p(D\mid {\theta }^{\mathrm{avg}})+\sum _{i=1}^T\mathrm{KL}(\mathrm{Bern}\left({\tilde{\lambda }}_i\right)\parallel \mathrm{Bern}\left(\lambda \right))+\sum _{i=1}^T{\tilde{\lambda }}_i\mathrm{KL}(𝒩({\mu }_{k_i^{\ast }},{\sigma }_{k_i^{\ast }}^2)\parallel 𝒩(0,{\sigma }_0^2)),$

where ${\theta }^{\mathrm{avg}}$ is the mean parameter vector defined in Step 1, and $k_i^{\ast }={\arg \max }_{1\le k\le K}{\varphi }_k\left({\theta }_i\right)$. We thus obtain the result shown in Equation (8).

Appendix B Proof of Theorem 1

Consider an $L$- hidden-layer fully connected neural network with the ReLU activation function ${\sigma }_b:{ℝ}^d\to {ℝ}^d$ defined as ${\sigma }_b\left(X\right)=\max \left\{0,X-b\right\}$ on some dimension $d$ and parameter $b$. The number of neurons in each layer is defined as $p_i$ for $i=1,\dots ,L$. The weights and biases are denoted by $W_i\in {ℝ}^{N\times N}$ and $b_i\in {ℝ}^N$. Thus, given the parameters $𝐩=(p_1,\cdots ,p_L)$, let $\theta$ denote the vector obtained by stacking all entries of the weight matrices $W_i$ and bias vectors $b_i$, then the fully connected network can be presented as:

$f_{\theta }\left(X\right)=W_{L+1}{\sigma }_{b_L}\left(W_L{\sigma }_{b_{L-1}}\dots {\sigma }_{b_1}\left(W_{1}X\right)\right)+b_{L+1}.$

The DNN $f_{\theta }$ also introduces a probability measure of the data, which we denote as $P_{\theta }$, and $p_{\theta }$ is the corresponding density function; $p_{\theta }\left(D\right)$ would be the likelihood of the data $D$.

One can define the sparse parameter space with sparsity parameter $s$ as $\{\theta \in {ℝ}^T:{\Vert \theta \Vert }_0<s\}$, where $\theta$ has only $s$ many non-zero entries. Then we can further introduce the sparse and quantized weights space $H(T,s,K)$ as follows:

	$ℐ(T,s,K)$	$=\{I={[r_1,\dots ,r_T]}^{\top }|{\Vert I\Vert }_0\le s,r_i\in {\{0,1\}}^K,{\Vert r_i\Vert }_0\le 1\},$
	$H(T,s,K)$	$=\{\theta \in {ℝ}^T|\theta =I\cdot E,E\in {[-B,B]}^K,I\in ℐ(T,s,K)\},$

where $B$ is some constant that satisfies $B>2$ and $ℐ(T,s,K)$ is the indexing space; each element $I\in ℐ(T,s,K)$ consists of $s$ many $K$-dimensional one-hot rows, and the remaining $T-s$ rows are zero vectors indicating the corresponding weight is pruned. In such a way, any $\theta \in H(T,s,K)$ satisfies that ${\Vert \theta \Vert }_0\le s$ and $\theta$ only have $K$ many distinct entry values then the DNN $f_{\theta }(\cdot )=f(\cdot ,\theta )$ is sparse and quantized. The following conditions are assumed, similarly to (Bai et al., 2020):

Condition 1.

$p_i\equiv N\in {\mathbb{Z}}^{+}$ that can depend on n, and $limT=\infty$.

Condition 2.

$\sigma \left(x\right)$ is 1-Lipschitz continuous.

Condition 3.

The hyperparameter ${\sigma }_0^2$ is set to be some constant, and $\lambda$ satisfies

	$\log \left(\frac{1}{\lambda }\right)$	$\phantom{}=O\left(\left(L+1\right)\log N+\log \left(p\sqrt{n/s^{\ast }}\right)\right)$
	$\log \left(\frac{1}{1-\lambda }\right)$	$\phantom{}=O\left(\frac{s^{\ast }}{T}\left(\left(L+1\right)\log N+\log \left(p\sqrt{n/s^{\ast }}\right)\right)\right)$

Condition 4.

$\max \{s^{\ast }\log \left(p\sqrt{n/s^{\ast }}\right),\left(L+1\right)s^{\ast }\log N\}=o\left(n\right)$ and $r_n^{\ast }\asymp {\xi }_n^{\ast }$.

The “oracle” sparsity $s^{\ast }$ is defined in Equation (19).

Definition 1.

The true function $f_0$ defined in Equation (13) is $\beta$-Hölder smoothness if

$f_0\in {𝒞}_p^{\beta }\left(M\right)=\{$

$f:{[0,1]}^p\to ℝ:\sum _{\alpha :\left|\alpha \right|<\beta }\parallel {\partial }^{\alpha }f{\parallel }_{\infty }+\sum _{\alpha :\left|\alpha \right|=\lfloor \beta \rfloor }\underset{\stackrel{x,y\in {\left[0,1\right]}^p}{x\ne y}}{sup}\frac{\left|{\partial }^{\alpha }f\left(x\right)-{\partial }^{\alpha }f\left(y\right)\right|}{{\Vert x-y\Vert }_{\infty }^{\beta -\lfloor \beta \rfloor }}\le M\}.$

for some constant $M>0$.

Following previous paper (Bai et al., 2020), we define:

$s^{\ast }=\arg \underset{s}{\min }\left\{r_n(L,p,s)+{\xi }_n(L,p,s)\right\}$

(19)

where

	$r_n(L,p,s)$	$\phantom{}=\left(\left(L+1\right)s/n\right)\log N+\left(s/n\right)\log \left(p\sqrt{n/s}\right)$
	${\xi }_n(L,p,s)$	$\phantom{}=\underset{\theta \in H(T,s,K),{\Vert \theta \Vert }_{\infty }\le B}{inf}{\Vert f_{\theta }-f_0\Vert }_{\infty }^2.$

Correspondingly, we define $r_n^{\ast }=r_n(L,p,s^{\ast }),{\xi }_n^{\ast }={\xi }_n(L,p,s^{\ast })$.

In this section, we reformulate the variational distribution by introducing a latent index variable. For any $q\left(\tilde{\theta }\right)\in ℱ$, it has the following equivalent form:

	${\tilde{\theta }}_i|z_i,{\gamma }_i$	$\sim \gamma {}_i\sum {}_{k=1}{}^K\mathrm{𝟙}\{z{}_i=k\}𝒩(\mu {}_k,\sigma {}_k{}^2)+(1-\gamma {}_i)\delta {}_0,$		(20)
	$z_i$	$\phantom{}\sim \text{Categorical}({\varphi }_1\left({\theta }_i\right),{\varphi }_2\left({\theta }_i\right),\dots ,{\varphi }_K\left({\theta }_i\right)),$
	${\gamma }_i$	$\phantom{}\sim \text{Bernoulli}\left({\lambda }_i\right),$

In addition, for theoretical convenience, we further restrict the variational family to satisfy

Condition 5.

$\left|{\mu }_k\right|\le B$ and ${\sigma }_k^2\le \frac{1}{2\log \left(T/r_n^{\ast }{\log }^2\left(n\right)\right)}$.

Note that the requirement of $\left|{\mu }_k\right|\le B$ is fairly reasonable, as most of the existing approximation results (Chérief-Abdellatif, 2020) only need bounded DNN weights.

We restate a formal version of our Theorem 1 as follows:

Theorem 2.

Under Conditions 1-2 and 4-5, Let ${\sigma }_0^2$ be a constant and $-\log \lambda =\log \left(T\right)+\delta \left[\left(L+1\right)\log N+\log \sqrt{n}p\right]$ for any constant $\delta >0$, Then with high probability:

${\int }_{{ℝ}^T}{d}^2(P_{\theta },P_0)\hat{q}\left(\theta \right)𝑑\theta \le C{\epsilon }_n^{\phantom{}\ast 2}+C^{\prime }\left(r_n^{\ast }+{\xi }_n^{\ast }\right),$

(21)

where $d(\cdot ,\cdot )$ denotes the Hellinger distance, and $C$ and $C^{\prime }$ are some constants.

Proof.

The convergence in squared Hellinger distance follows directly from Lemmas 1 and Lemma 2, as the chosen value of $\lambda$ meets the necessary assumptions. ∎

Remark. Compared to prior results, Lemma 1 demonstrates that a spike-and-slab prior combined with a Gaussian mixture model (GMM) with finitely many components can effectively approximate the true underlying function. In contrast, Lemma 2 establishes that the statistical estimation error of the spike-and-GMM variational distribution vanishes as the sample size $n\to \infty$.

B.1 Proof of Lemma 1

Proof.

Let ${\theta }^{\ast }=\arg {\min }_{\theta \in H(L,N,s,K)}{\Vert f_{\theta }-f_0\Vert }_{\infty }^2$. By definition, they are only $K$ many unique non-zero number in ${\theta }^{\ast }$, denoted as ${\mu }_k^{\ast }\in ℝ$, for $k=1,\dots ,K$. In other words, for any ${\theta }_i^{\ast }\ne 0$, ${\theta }_i^{\ast }$ must choose from the quantization set $𝒬=\{{\mu }_1^{\ast },\dots ,{\mu }_K^{\ast }\}$, and we denote the choice index of ${\theta }_i^{\ast }$ from $𝒬$ as $\left(i\right)$, (i.e., ${\theta }_i^{\ast }={\mu }_{\left(i\right)}^{\ast }$). Now, given ${\theta }^{\ast }$, we construct $q^{\ast }\left(\tilde{\theta }\right)$ as follow:

	${\tilde{\theta }}_i|z_i^{\ast },{\gamma }_i^{\ast }$	$\sim {\gamma }_i^{\ast }\sum _{k=1}^K\mathrm{𝟙}\{z_i^{\ast }=k\}𝒩({\mu }_k^{\ast },{\sigma }_n^2)+(1-{\gamma }_i^{\ast }){\delta }_0,$
	$z_i^{\ast }$	$\phantom{}\sim \text{Categorical}({\varphi }_1\left({\theta }_i^{\ast }\right),{\varphi }_2\left({\theta }_i^{\ast }\right),\dots ,{\varphi }_K\left({\theta }_i^{\ast }\right)),$
	${\gamma }_i^{\ast }$	$\sim \text{Bernoulli}\left({\psi }_i^{\ast }\right),{\psi }_i^{\ast }=\mathrm{𝟙}\{{\theta }_i^{\ast }\ne 0\}$

where ${\sigma }_n^2=\frac{s^{\ast }}{32n}\log {\left(3N\right)}^{-1}{\left(2BN\right)}^{-2L}{\left\{\left(p+1+\frac{1}{BN-1}\right)+\frac{1}{{\left(2BN\right)}^2-1}+\frac{2}{{\left(2BN-1\right)}^2}\right\}}^{-1}$. ²² 2 Notice that ${\sigma }_n^2$ satisfies Condition 5 as with sufficiently large $n,N$ and $L$, $1/n\le 1/\log \left(n\right)$ and $1/N^L<1/\log \left(T\right)=1/\log \left(NL\right)$. Thus, we can have the following marginal distribution:

$q^{\ast }\left(\theta \right)=\sum \mathrm{𝟏}\{\gamma ={\gamma }^{\ast }\}\prod _{i=1}^T{\gamma }_i[{\varphi }_{\left(i\right)}\left({\theta }_i^{\ast }\right)𝒩({\mu }_{\left(i\right)}^{\ast },{\sigma }_n^2)+\sum _{k\ne \left(i\right)}{\varphi }_k\left({\theta }_i^{\ast }\right)𝒩({\mu }_k^{\ast },{\sigma }_n^2)]+(1-{\gamma }_i^{\ast }){\delta }_0$

Next we first need to bound $\int {\Vert f_{\theta }-f_{{\theta }^{\ast }}\Vert }_{\infty }^2{q}^{\ast }\left(\theta \right)$. We can also write ${\theta }^{\ast }=\left\{W_1^{\ast },b_1^{\ast },\dots ,W_{L+1}^{\ast },b_{L+1}^{\ast }\right\}$, then we define the following terms as:

	${\tilde{W}}_l$	$\phantom{}=\underset{i,j}{sup}\left|W_{l,i,j}-W_{l,i,j}^{\ast }\right|,$
	${\tilde{b}}_l$	$\phantom{}=\underset{i}{sup}\left|b_{l,i}-b_{l,i}^{\ast }\right|.$

Then, following the proof in (Chérief-Abdellatif, 2020, Proof of Theorem 7), we can have the following:

		$\int {\Vert f_{\theta }-f_{{\theta }^{\ast }}\Vert }_{\infty }^2{q}^{\ast }\left(𝑑\theta \right)$
		$\phantom{}\le 2N^{2L-2}{\left(d+1+\frac{1}{BN-1}\right)}^2(\sum _{l=1}^L{B}^{2l-2}\prod _{v=l+1}^L\int {\left(B+{\tilde{W}}_v\right)}^{2}q\left(𝑑\theta \right)\int {\tilde{W}}_l^2{q}_l\left(d{\theta }_l\right)CLOSE$
		$+2\sum _{l=1}^L\sum _{k=1}^{l-1}{B}^{l-1}{B}^{k-1}\prod _{v=l+1}^L\int {(B+{\tilde{W}}_v)}^{2}q\left(d\theta \right)\int {\tilde{W}}_l{q}_l\left(d{\theta }_l\right)\prod _{v=k+1}^l\int (B+{\tilde{W}}_v)q\left(d\theta \right)\int {\tilde{W}}_{k}q\left(d\theta \right))$
		$+2(\sum _{l=1}^L{D}^{2\left(L-l\right)}\prod _{v=l+1}^L\int {\left(B+{\tilde{W}}_v\right)}^{2}q\left(𝑑\theta \right)\int {\tilde{b}}_l^{2}q\left(𝑑\theta \right)CLOSE$
		$+2\sum _{l=1}^L\sum _{k=1}^{l-1}{D}^{L-l}{D}^{L-k}\prod _{v=l+1}^L\int {(B+{\tilde{W}}_v)}^{2}q\left(d\theta \right)\int {\tilde{b}}_{l}q\left(d\theta \right)\prod _{v=k+1}^l\int (B+{\tilde{W}}_v)q\left(d\theta \right)\int {\tilde{b}}_{k}q\left(d\theta \right)).$		(22)

Then next we need to upper bound the term:

$\int {\tilde{W}}_l{q}^{\ast }\left(𝑑\theta \right)=\int \underset{i,j}{sup}\left|W_{l,i,j}-W_{l,i,j}^{\ast }\right|q^{\ast }\left(𝑑\theta \right)$

We first bound the following, for some $t>0$,

	$\exp \left(𝔼\left[t\underset{i,j}{sup}\left|W_{l,i,j}-W_{l,i,j}^{\ast }\right|\right]\right)$	$\phantom{}\le 𝔼\underset{i,j}{sup}\exp \left(t\left|W_{l,i,j}-W_{l,i,j}^{\ast }\right|\right)$
		$\phantom{}\le N^2𝔼\left[\exp \left(t\left|W_{l,i,j}-W_{l,i,j}^{\ast }\right|\right)\right]$
		$\phantom{}=N^2𝔼\left[{𝔼}_{𝒩({\mu }_k^{\ast },{\sigma }_n^2)}\left[\exp \left(t\left|W_{l,i,j}-W_{l,i,j}^{\ast }\right|\right)|z_{l,i,j}=k\right]\right]$
		$\phantom{}=N^2\sum _{k=1}^K{\varphi }_k\left(W_{l,i,j}^{\ast }\right){𝔼}_{𝒩({\mu }_k^{\ast },{\sigma }_n^2)}\left[\exp \left(t\left|W_{l,i,j}-W_{l,i,j}^{\ast }\right|\right)\right]$
		$\phantom{}=N^2({\varphi }_{\left(i\right)}\left(W_{l,i,j}^{\ast }\right){𝔼}_{𝒩({\mu }_{\left(i\right)}^{\ast },{\sigma }_n^2)}\left[\exp \left(t\left|W_{l,i,j}-W_{l,i,j}^{\ast }\right|\right)\right]CLOSE$
		$+\sum _{k\ne \left(i\right)}^K{\varphi }_k\left(W_{l,i,j}^{\ast }\right)𝔼\left[\exp \left(t\right|W_{l,i,j}-W_{l,i,j}^{\ast }\left|\right)\right])$		(23)

Notice that by definition $W_{l,i,j}^{\ast }={\mu }_{\left(i\right)}^{\ast }$, thus we can bound the first term as:

	${𝔼}_{𝒩({\mu }_{\left(i\right)}^{\ast },{\sigma }_n^2)}\left[\exp \left(t\left|W_{l,i,j}-W_{l,i,j}^{\ast }\right|\right)\right]$	$\phantom{}={𝔼}_{𝒩({\mu }_{\left(i\right)}^{\ast },{\sigma }_n^2)}\left[\exp \left(t\left|W_{l,i,j}-{\mu }_{\left(i\right)}^{\ast }\right|\right)\right]$
		$\phantom{}={\int }_0^{\infty }P\left(\exp \left(t\left|W_{l,i,j}-{\mu }_{\left(i\right)}^{\ast }\right|>x\right)\right)𝑑x$
		$\phantom{}={\int }_0^{\infty }P\left(\left|W_{l,i,j}-{\mu }_i^{\ast }\right|>\frac{\log x}{t}\right)𝑑x$
		$\phantom{}={\int }_0^{\infty }2P\left(W_{l,i,j}-{\mu }_{\left(i\right)}^{\ast }>\frac{\log x}{t}\right)𝑑x$
		$\phantom{}={\int }_0^{\infty }2P\left(z>\frac{\log x}{t{\sigma }_n}\right)𝑑x$
		$\phantom{}=2\exp \left(\frac{t^2{\sigma }_n^2}{2}\right)$

Next, we bound the second term of the Equation (23):

	${𝔼}_{𝒩({\mu }_k^{\ast },{\sigma }_n^2)}\left[\exp \left(t\left|W_{l,i,j}-{\mu }_{\left(i\right)}^{\ast }\right|\right)\right]$	$\phantom{}={\int }_0^{\infty }P\left(\exp \left(t\left|W_{l,i,j}-{\mu }_{\left(i\right)}^{\ast }\right|>x\right)\right)𝑑x$
		$\phantom{}={\int }_0^{\infty }2P\left(W_{l,i,j}-{\mu }_{\left(i\right)}^{\ast }>\frac{\log x}{t}\right)𝑑x$
		$\phantom{}={\int }_0^{\infty }2P\left(W_{l,i,j}-{\mu }_k^{\ast }+{\mu }_k^{\ast }-{\mu }_{\left(i\right)}^{\ast }>\frac{\log x}{t}\right)𝑑x$
		$\phantom{}={\int }_0^{\infty }2P\left({\sigma }_{n}z+{\mu }_k^{\ast }-{\mu }_{\left(i\right)}^{\ast }>\frac{\log x}{t}\right)𝑑x$
		$\phantom{}=2\exp \left(t\left({\mu }_k^{\ast }-{\mu }_{\left(i\right)}^{\ast }\right)+\frac{{\sigma }_n^2{t}^2}{2}\right)$
		$\phantom{}\le 2\exp \left(4t+\frac{{\sigma }_n^2{t}^2}{2}\right)$

Notice that the last inequality is because of ${sup}_{i,j}\left({\mu }_i^{\ast }-{\mu }_j^{\ast }\right)\le 4$. And by choosing $t=\sqrt{2\log \left(3N^2\right)}/{\sigma }_n$, the Equation (23), can be bounded by:

	$\exp \left(𝔼\left[t\underset{i,j}{sup}\left|W_{l,i,j}-W_{l,i,j}^{\ast }\right|\right]\right)$	$\phantom{}\le 2N^2\left({\varphi }_{\left(i\right)}\left(W_{l,i,j}^{\ast }\right)\exp \left(t^2{\sigma }_n^2/2\right)+\sum _{k\ne \left(i\right)}{\varphi }_k\left(W_{l,i,j}^{\ast }\right)\exp \left(4t+t^2{\sigma }_n^2/2\right)\right)$
		$\phantom{}=2N^2\left(\exp \left(t^2{\sigma }_n^2/2\right)+\sum _{k\ne \left(i\right)}{\varphi }_k\left(W_{l,i,j}^{\ast }\right)\exp \left(4t+t^2{\sigma }_n^2/2\right)\right)$
		$\phantom{}\le 2N^2\left(\exp \left(t^2{\sigma }_n^2/2\right)+1/2\exp \left(t^2{\sigma }_n^2/2\right)\right)$
		$\phantom{}=3N^2\exp \left(t^2{\sigma }_n^2/2\right)$

The second inequality is obtained by setting $\tau$ such that ${\sum }_{k\ne \left(i\right)}{\varphi }_k\left(W_{l,i,j}^{\ast }\right)\le 1/2$. Note that given a fixed DNN structure, such a $\tau$ always exists as the ${\sum }_{k\ne \left(i\right)}{\varphi }_k\left(W_{l,i,j}^{\ast }\right)$ decrease monotonically to zero as $\tau$ decrease. Thus, we can have

$𝔼\left[\underset{i,j}{sup}\left|W_{l,i,j}-W_{l,i,j}^{\ast }\right|\right]$

$\phantom{}\le \frac{3N^2}{t}+\frac{t{\sigma }_n^2}{2}=\sqrt{2{\sigma }_n^2\log \left(3N^2\right)}\le \sqrt{8{\sigma }_n^2\log \left(3N\right)}$

Next we bound $\int {\tilde{W}}_l^2{q}^{\ast }\left(𝑑x\right)$, following similar procedure, for some $t>0$, we can have:

	$\exp \left(𝔼\left[t\underset{i,j}{sup}{\left(W_{l,i,j}-W_{l,i,j}^{\ast }\right)}^2\right]\right)$	$\phantom{}\le N^2𝔼\left[\exp \left(t{\left(W_{l,i,j}-W_{l,i,j}^{\ast }\right)}^2\right)\right]$
		$\phantom{}\le N^2𝔼\left[{𝔼}_{𝒩({\mu }_k,{\sigma }_n^2)}\left[t{\left(W_{l,i,j}-W_{l,i,j}^{\ast }\right)}^2|z_{l,i,j}=k\right]\right]$
		$\phantom{}=N^2({\varphi }_{\left(i\right)}\left(W_{l,i,j}^{\ast }\right){𝔼}_{𝒩({\mu }_{\left(i\right)}^{\ast },{\sigma }_n^2)}\left[\exp \left(t{\left(W_{l,i,j}-W_{l,i,j}^{\ast }\right)}^2\right)\right]CLOSE$
		$+\sum _{k\ne \left(i\right)}{𝔼}_{𝒩({\mu }_k,{\sigma }_n^2)}\left[\exp \left(t{(W_{l,i,j}-W_{l,i,j}^{\ast })}^2\right)\right])$		(24)

Note that with a slight abuse of notation, the $z_{l,i,j}$ in the above equation means the latent variable $z_i$ (introduced in (20)) corresponding to the weight $W_{l,i,j}$. The first term of Equation (24) can be bounded for $t<\frac{1}{2{\sigma }_n^2}$ as follow:

${𝔼}_{𝒩({\mu }_{\left(i\right)}^{\ast },{\sigma }_n^2)}\left[\exp \left(t{\left(W_{l,i,j}-W_{l,i,j}^{\ast }\right)}^2\right)\right]$

$\phantom{}={𝔼}_{𝒩({\mu }_{\left(i\right)}^{\ast },{\sigma }_n^2)}\left[\exp \left(t{\left(W_{l,i,j}-{\mu }_{\left(i\right)}^{\ast }\right)}^2\right)\right]=\frac{1}{\sqrt{1-2t{\sigma }_n^2}}.$

And the second term of Equation (24) can be bounded as:

	${𝔼}_{𝒩({\mu }_k,{\sigma }_n^2)}\left[\exp \left(t{\left(W_{l,i,j}-W_{l,i,j}^{\ast }\right)}^2\right)\right]$	$\phantom{}={𝔼}_{𝒩({\mu }_k,{\sigma }_n^2)}\left[\exp \left(t{\left(W_{l,i,j}-{\mu }_{\left(i\right)}^{\ast }\right)}^2\right)\right]$
		$\phantom{}={𝔼}_{𝒩({\mu }_k,{\sigma }_n^2)}\left[\exp \left(t\left(W_{l,i,j}-{\mu }_k^{\ast }+{\mu }_k^{\ast }-{\mu }_{\left(i\right)}^{\ast }\right)\right)\right]$
		$\phantom{}=𝔼\left[\exp \left(t{\left({\sigma }_{n}z+{\mu }_k^{\ast }-{\mu }_{\left(i\right)}^{\ast }\right)}^2\right)\right]$
		$\phantom{}=\frac{1}{\sqrt{1-2t{\sigma }_n^2}}\exp \left(\frac{{\left({\mu }_k^{\ast }-{\mu }_{\left(i\right)}^{\ast }\right)}^{2}t}{1-2t{\sigma }_n^2}\right)$
		$\phantom{}\le \frac{1}{\sqrt{1-2t{\sigma }_n^2}}\exp \left(\frac{16t}{1-2t{\sigma }_n^2}\right)$

The last inequality is again because of the property that ${sup}_{i,j}\left|{\mu }_i^{\ast }-{\mu }_j^{\ast }\right|\le 4$. Thus Equation (24) can be bounded by:

	$\exp \left(𝔼\left[t\underset{i,j}{sup}{\left(W_{l,i,j}-W_{l,i,j}^{\ast }\right)}^2\right]\right)$	$\phantom{}\le N^2\left(\frac{1}{\sqrt{1-2t{\sigma }_n^2}}+\sum _{k\ne \left(i\right)}{\varphi }_k\left(W_{l,i,j}^{\ast }\right)\frac{1}{\sqrt{1-2t{\sigma }_n^2}}\exp \left(\frac{16t}{1-2t{\sigma }_n^2}\right)\right)$
		$\phantom{}\le 2N^2\frac{1}{\sqrt{1-2t{\sigma }_n^2}},$

where the last inequality is because of choosing a small $\tau$ such ${\sum }_{k\ne \left(i\right)}{\varphi }_k\left(W_{l,i,j}^{\ast }\right)\le \exp \left(\frac{16t}{1-2t{\sigma }_n^2}\right)$, such $\tau$ always exist since ${\sum }_{k\ne \left(i\right)}{\varphi }_k\left(W_{l,i,j}^{\ast }\right)$ decrease monotonically to $0$ as $\tau \to 0$.

Thus we can bound $\int {\tilde{W}}_l^2{q}^{\ast }\left(𝑑x\right)$ by the following:

	$\int {\tilde{W}}_l^2{q}^{\ast }\left(𝑑x\right)=\int \underset{i,j}{sup}{\left(W_{l,i,j}-W_{l,i,j}^{\ast }\right)}^2𝑑x$	$\phantom{}\le \log \left(N^2\right)/t+\log \left(\frac{2}{\sqrt{1-2t{\sigma }_n^2}}\right)/t$
		$\phantom{}=4\log \left(N^2\right){\sigma }_n^2+4\log \left(\frac{2}{\sqrt{\frac{1}{2}}}\right){\sigma }_n^2$
		$\phantom{}\le 8{\sigma }_n^2\log \left(3N\right)$

By choosing $\sqrt{8{\sigma }_n^2\log \left(3N\right)}\le B$, we can have:

		$\int \left(B+{\tilde{W}}_l\right)q^{\ast }\left(𝑑\theta \right)\le 2B,$
		$\int {\left(B+{\tilde{W}}_l\right)}^2{q}^{\ast }\left(𝑑\theta \right)\le B^2+2B\sqrt{8{\sigma }_n^2\log \left(3N\right)}+8{\sigma }_n^2\log \left(3N\right)\le 4B^2.$

Similarly, we can have the following:

	$\int {\tilde{b}}_l{q}^{\ast }\left(𝑑\theta \right)$	$\phantom{}\le \sqrt{8{\sigma }_n^2\log \left(3N\right)},$
	$\int {\tilde{b}}_l^2{q}^{\ast }\left(𝑑\theta \right)$	$\phantom{}\le 8{\sigma }_n^2\log \left(3N\right).$

Combined with Equation (22), we can have:

	$\int {\Vert f_{\theta }-f_{{\theta }^{\ast }}\Vert }_{\infty }^{\ast }{q}^{\ast }\left(𝑑\theta \right)$	$\phantom{}\le 2N^{2L-2}{\left(p+1+\frac{1}{BN-1}\right)}^2(\sum _{\ell =1}^L{B}^{2\ell -2}{\left(4B^2\right)}^{L-\ell }8{\sigma }_n^2\log \left(3N\right)CLOSE$
		$+2\sum _{\ell =1}^L\sum _{k=1}^{\ell -1}{B}^{\ell -1}{B}^{k-1}{\left(4B^2\right)}^{L-\ell }\sqrt{8{\sigma }_n^2\log \left(3N\right)}{\left(2B\right)}^{\ell -k}\sqrt{8{\sigma }_n^2\log \left(3N\right)})$
		$+2(\sum _{\ell =1}^L{N}^{2\left(L-\ell \right)}{\left(4B^2\right)}^{L-\ell }8{\sigma }_n^2\log \left(3N\right)CLOSE$
		$+2\sum _{\ell =1}^L\sum _{k=1}^{\ell -1}{N}^{L-\ell }{N}^{L-k}{\left(4B^2\right)}^{L-\ell }\sqrt{8{\sigma }_n^2\log \left(3N\right)}{\left(2B\right)}^{\ell -k}\sqrt{8{\sigma }_n^2\log \left(3N\right)}).$

With some algebra, we can have:

	$\int$	${\Vert f_{\theta }-f_{{\theta }^{\ast }}\Vert }_2^2{q}_n^{\ast }\left(d\theta \right)$
		$\phantom{}\le 2N^{2L-2}{\left(d+1+\frac{1}{BN-1}\right)}^2\left(B^{2L-2}8{\sigma }_n^2\log \left(3N\right)\sum _{\ell =0}^{L-1}{4}^{\ell }+2B^{2L-2}8{\sigma }_n^2\log \left(3D\right)\sum _{\ell =1}^L\sum _{k=1}^{\ell -1}{2}^{L-\ell }{2}^{L-k}\right)$
		$+2\left(8{\sigma }_n^2\log \left(3N\right)\sum _{\ell =1}^L{\left(2BN\right)}^{2L-2\ell }+16{\sigma }_n^2\log \left(3N\right)\sum _{\ell =1}^L\sum _{k=1}^{\ell -1}{\left(2BN\right)}^{L-\ell }{\left(2BN\right)}^{L-k}\right)$
		$\phantom{}\le 2N^{2L-2}{\left(p+1+\frac{1}{BN-1}\right)}^2\left(B^{2L-2}8{\sigma }_n^2\log \left(3N\right)\frac{4^L-1}{4-1}+2B^{2L-2}8{\sigma }_n^2\log \left(3N\right)\sum _{\ell =1}^L{2}^{L-\ell }{2}^{L-\ell +1}\sum _{k=0}^{\ell -2}{2}^k\right)$
		$+2\left(8{\sigma }_n^2\log \left(3N\right)\sum _{\ell =0}^{L-1}{\left(2BN\right)}^{2\ell }+16{\sigma }_n^2\log \left(3N\right)\sum _{\ell =1}^L{\left(2BD\right)}^{L-\ell }{\left(2BN\right)}^{L-\ell +1}\sum _{k=0}^{\ell -2}{\left(2BN\right)}^k\right)$
		$\phantom{}\le 2N^{2L-2}{\left(d+1+\frac{1}{BN-1}\right)}^2\left(B^{2L-2}8{\sigma }_n^2\log \left(3N\right)\frac{4^L}{3}+2B^{2L-2}8{\sigma }_n^2\log \left(3N\right)\sum _{\ell =1}^L{2}^{L-\ell }{2}^{L-\ell +1}{2}^{\ell -1}\right)$
		$+2\left(8{\sigma }_n^2\log \left(3N\right)\frac{{\left(2BN\right)}^{2L}}{{\left(2BN\right)}^2-1}+16{\sigma }_n^2\log \left(3N\right)\sum _{\ell =1}^L{\left(2BN\right)}^{L-\ell }{\left(2BN\right)}^{L-\ell +1}\frac{{\left(2BN\right)}^{\ell -1}}{2BN-1}\right)$
		$\phantom{}\le 2N^{2L-2}{\left(d+1+\frac{1}{BN-1}\right)}^2\left(B^{2L-2}8{\sigma }_n^2\log \left(3N\right)\frac{4^L}{3}+2B^{2L-2}8{\sigma }_n^2\log \left(3N\right)2^L\sum _{\ell =0}^{L-1}{2}^{\ell }\right)$
		$+2\left(8{\sigma }_n^2\log \left(3N\right)\frac{{\left(2BN\right)}^{2L}}{{\left(2BN\right)}^2-1}+16{\sigma }_n^2\log \left(3N\right)\sum _{\ell =0}^{L-1}{\left(2BN\right)}^{\ell }\frac{{\left(2BN\right)}^L}{2BN-1}\right)$
		$\phantom{}\le 2N^{2L-2}{\left(p+1+\frac{1}{BN-1}\right)}^2\left(B^{2L-2}8{\sigma }_n^2\log \left(3N\right)\frac{4^L}{3}+2B^{2L-2}8{\sigma }_n^2\log \left(3N\right)2^{2L}\right)$
		$+2\left(8{\sigma }_n^2\log \left(3N\right)\frac{{\left(2BN\right)}^{2L}}{{\left(2BN\right)}^2-1}+16{\sigma }_n^2\log \left(3N\right)\frac{{\left(2BN\right)}^{2L}}{{\left(2BN-1\right)}^2}\right)$
		$\phantom{}=2N^{2L-2}{\left(p+1+\frac{1}{BN-1}\right)}^{2}8{\sigma }_n^2\log \left(3N\right)\left(B^{2L-2}\frac{4^L}{3}+2B^{2L-2}{2}^{2L}\right)$
		$+2\left(\frac{{\left(2BN\right)}^{2L}}{{\left(2BN\right)}^2-1}+2\frac{{\left(2BN\right)}^{2L}}{{\left(2BN-1\right)}^2}\right)8{\sigma }_n^2\log \left(3N\right),$

as $BN>2$,

	$\int$	${\Vert f_{\theta }-f_{{\theta }^{\ast }}\Vert }_{\infty }^2{q}^{\ast }\left(d\theta \right)$
		$\phantom{}\le 16{\sigma }_n^2\log \left(3N\right)\left\{N^{2L-2}{\left(p+1+\frac{1}{BN-1}\right)}^2\frac{7}{3}{B}^{2L-2}{2}^{2L}+{\left(2BN\right)}^{2L}\left(\frac{1}{{\left(2BN\right)}^2-1}+\frac{2}{{\left(2BN-1\right)}^2}\right)\right\}$
		$\phantom{}=16{\sigma }_n^2\log \left(3N\right)\left\{{\left(2BN\right)}^{2L}\frac{1}{{\left(BN\right)}^2}{\left(p+1+\frac{1}{BN-1}\right)}^2\frac{7}{3}+{\left(2BN\right)}^{2L}\left(\frac{1}{{\left(2BN\right)}^2-1}+\frac{2}{{\left(2BN-1\right)}^2}\right)\right\}$
		$\phantom{}\le 16{\sigma }_n^2\log \left(3N\right){\left(2BN\right)}^{2L}\left\{{\left(p+1+\frac{1}{BN-1}\right)}^2+\frac{1}{{\left(2BN\right)}^2-1}+\frac{2}{{\left(2BN-1\right)}^2}\right\}$
		$\phantom{}=\frac{s^{\ast }}{2n}\le r_n^{\ast }$

The last equality is due to the definition of ${\sigma }_n$, and the last inequality is due to the definition of $r_n^{\ast }$.

In the next step, we aim to bound the integral ${\int }_{{ℝ}^T}{l}_n(P_0,P_{\theta })𝑑\theta$. Note that by definition:

	$l_n(P_0,P_{\theta })$	$\phantom{}=\frac{1}{2{\sigma }_{\epsilon }^2}\left({\Vert Y-f_{\theta }\left(X\right)\Vert }_2^2-{\Vert Y-f_0\left(X\right)\Vert }_2^2\right)$
		$=\frac{1}{2{\sigma }_{\epsilon }^2}(\parallel Y-f_0\left(X\right)+f_0\left(X\right)-f_{\theta }\left(X\right)){\parallel }_2^2-\parallel Y-f_0\left(X\right){\parallel }_2^2)$
		$\phantom{}=\frac{1}{2{\sigma }_{\epsilon }^2}\left({\Vert f_{\theta }\left(X\right)-f_0\left(X\right)\Vert }_2^2+2⟨Y-f_0\left(X\right),f_0\left(X\right)-f_{\theta }\left(X\right)⟩\right),$

We can define the following:

	${ℛ}_1$	$\phantom{}={\int }_{{ℝ}^T}{\Vert f_{\theta }\left(X\right)-f_0\left(X\right)\Vert }_2^2{q}^{\ast }\left(\theta \right)\left(𝑑\theta \right),$
	${ℛ}_2$	$\phantom{}={\int }_{{ℝ}^T}⟨Y-f_0\left(X\right),f_0\left(X\right)-f_{\theta }\left(X\right)⟩q^{\ast }\left(\theta \right)\left(𝑑\theta \right).$

Since ${\Vert f_{\theta }\left(X\right)-f_0\left(X\right)\Vert }_2^2\le n{\Vert f_{\theta }-f_0\Vert }_{\infty }^2\le n\left(r_n^{\ast }+{\Vert f_{{\theta }^{\ast }}-f_0\Vert }_{\infty }^2\right)$, it follows that

${ℛ}_1\le n{r}_n^{\ast }+n{\Vert f_{{\theta }^{\ast }}-f_0\Vert }_{\infty }^2.$

Given $Y-f_0\left(X\right)=\epsilon \sim 𝒩(0,{\sigma }_{\epsilon }^{2}I)$, we have

${ℛ}_2={\epsilon }^T{\int }_{\Theta }\left(f_0\left(X\right)-f_{\theta }\left(X\right)\right)q^{\ast }\left(\theta \right)\left(𝑑\theta \right)\sim 𝒩(0,c_f{\sigma }_{\epsilon }^2),$

whereby the Cauchy-Schwarz inequality,

$c_f={\Vert {\int }_{\Theta }\left(f_0\left(X\right)-f_{\theta }\left(X\right)\right)q^{\ast }\left(\theta \right)\left(𝑑\theta \right)\Vert }_2^2\le {ℛ}_1$

Thus, ${ℛ}_2=O_p\left(\sqrt{{ℛ}_1}\right)$, and with high probability, ${ℛ}_2\le C_0^{\prime }{ℛ}_1$ for some positive constant $C_0^{\prime }$ if $limn\left(r_n^{\ast }+{\xi }_n^{\ast }\right)=\infty$, or for any diverging sequence $C_0^{\prime }$ if $limsupn\left(r_n^{\ast }+{\xi }_n^{\ast }\right)\ne \infty$. Therefore,

${\int }_{{ℝ}^T}{l}_n(P_0,P_{\theta })q^{\ast }\left(\theta \right)\left(𝑑\theta \right)\le C_1^{\prime }\left(n{r}_n^{\ast }+{\Vert f_{{\theta }^{\ast }}-f_0\Vert }_{\infty }^2\right)\text{w.h.p.}$

(25)

In the next step, we try to bound the $\mathrm{KL}$ divergence between $q^{\ast }$ and $\pi \left(\theta \right|\lambda )$,

		$\mathrm{KL}(q^{\ast }\left(\theta \right)\parallel \pi \left(\theta \right|\lambda ))$
	$\le$	$\log \left(\frac{1}{\pi \left({\gamma }^{\ast }\right)}\right)+\sum _{i=1}^T\mathrm{KL}[{\gamma }_i^{\ast }\left[\sum _k{\varphi }_k\left({\theta }_i^{\ast }\right)𝒩({\mu }_k^{\ast },{\sigma }_n^2)\right]+(1-{\gamma }_i^{\ast }){\delta }_0\parallel {\gamma }_i^{\ast }𝒩(0,{\sigma }_0^2)+(1-{\gamma }_i^{\ast }){\delta }_0]$		(26)
	$=$	$\log \frac{1}{{\lambda }^{s^{\ast }}{\left(1-\lambda \right)}^{T-s^{\ast }}}+\sum _{i=1}^T{\gamma }_i^{\ast }\mathrm{KL}[\sum _k{\varphi }_k\left({\theta }_i^{\ast }\right)𝒩({\mu }_k^{\ast },{\sigma }_n^2)\parallel 𝒩(0,{\sigma }_n^2)]$
	$\le$	$s^{\ast }\log \left(\frac{1}{\lambda }\right)+(T-s^{\ast })\log \left(\frac{1}{1-\lambda }\right)+\sum _{i=1}^T{\gamma }_i^{\ast }\sum _{k=1}^K{\varphi }_k\left({\theta }_i^{\ast }\right)\mathrm{KL}(𝒩({\mu }_k^{\ast },{\sigma }_n^2)\parallel 𝒩(0,{\sigma }_0^2))$		(27)
	$\le$	$s^{\ast }\log \left(\frac{1}{\lambda }\right)+\left(T-s^{\ast }\right)\log \left(\frac{1}{1-\lambda }\right)+\sum _{i=1}^T{\gamma }_i^{\ast }\sum _{k=1}^K{\varphi }_k\left({\theta }_i^{\ast }\right)\left[\frac{1}{2}\log \frac{{\sigma }_0^2}{{\sigma }_n^2}+\frac{{\sigma }_n^2+{\left({\mu }_k^{\ast }\right)}^2}{2{\sigma }_0^2}-\frac{1}{2}\right]$
	$\le$	$s^{\ast }\log \left(\frac{1}{\lambda }\right)+\left(T-s^{\ast }\right)\log \left(\frac{1}{1-\lambda }\right)+\sum _{i=1}^T{\gamma }_i^{\ast }\left[\frac{1}{2}\log \frac{{\sigma }_0^2}{{\sigma }_n^2}+\frac{{\sigma }_n^2+4}{2{\sigma }_0^2}-\frac{1}{2}\right]$
	$\le$	$C_{0}n{r}_n^{\ast }+\frac{s^{\ast }}{2}{\sigma }_n^2+\frac{s^{\ast }}{2{\sigma }_0^2}\left(B^2-1\right)+\frac{s^{\ast }}{2}\log \left(\frac{{\sigma }_0^2}{{\sigma }_n^2}\right)$		(28)
	$\le$	$\left(C_0+1\right)n{r}_n^{\ast }+\frac{s^{\ast }}{2{\sigma }_0^2}{B}^2+\frac{s^{\ast }}{2}\log \left(\frac{16n}{s^{\ast }}\log \left(3pN\right){\left(2BN\right)}^{2L+2}\left\{{\left(p+1+\frac{1}{BN-1}\right)}^2+\frac{1}{{\left(2BN\right)}^2-1}+\frac{2}{{\left(2BN-1\right)}^2}\right\}\right)$
	$\le$	$\left(C_0+2\right)n{r}_n^{\ast }+\frac{s^{\ast }}{2{\sigma }_0^2}{B}^2+\left(L+1\right)s^{\ast }\log \left(2BN\right)+\frac{s^{\ast }}{2}\log \log \left(3BN\right)+\frac{s^{\ast }}{2}\log \left(\frac{n}{s^{\ast }}{p}^2\right)$
	$\le$	$\left(C_0+3\right)n{r}_n^{\ast }+\left(L+1\right)s^{\ast }\log N+s^{\ast }\log \left(p\sqrt{\frac{n}{s^{\ast }}}\right)$
	$\le$	$C_{1}n{r}_n^{\ast },\text{ for sufficiently large }n.$

where the inequality (26) and (27) are follows from Lemma 3 and the inequality (28) is because of the fact that $B>2$ and ${\sum }_{i=1}^T{\gamma }_i^{\ast }=s^{\ast }$, thus combined with the result in equation (25), we finish the proof. ∎

B.2 Proof of Lemma 2

Proof.

Following previous work (Bai et al., 2020, proof of Lemma 4.2), we first define the space

	$H_n\left(\theta \right)$	$=\{\theta \in {ℝ}^T:{\Vert \theta \Vert }_0\le s_n,{\Vert \theta \Vert }_{\infty }\le B+1\},$
	$H_n^{\prime }\left(\theta \right)$	$=\{\theta \in {ℝ}^T:{\Vert \theta \Vert }_0>s_n,{\Vert \theta \Vert }_{\infty }\le B+1\},$
	$H_n^{\prime \prime }\left(\theta \right)$	$\phantom{}=\{\theta \in {ℝ}^T:{\Vert \theta \Vert }_{\infty }>B+1\}.$

By the above definitions, we now have:

${\int }_{{ℝ}^T}{d}^2(P_{\theta },P_0)\hat{q}\left(𝑑\theta \right)={\int }_{H_n\left(\theta \right)}{d}^2(P_{\theta },P_0)\hat{q}\left(𝑑\theta \right)+{\int }_{H_n^{\prime }\left(\theta \right)}{d}^2(P_{\theta },P_0)\hat{q}\left(𝑑\theta \right)+{\int }_{H_n^{\prime \prime }\left(\theta \right)}{d}^2(P_{\theta },P_0)\hat{q}\left(𝑑\theta \right).$

(29)

Lemma 4 presents a variational characterization of the $\mathrm{KL}$ divergence, originally due to Donsker and Varadhan. The proof is available in (Boucheron et al., 2013).

Lemma 4.

Let $\mu$ be any probability measure and $h$ a measurable function with $e^h\in L_1\left(\mu \right)$, then

$\log \int e^{h\left(\eta \right)}\mu \left(d\eta \right)=\underset{\rho }{sup}[\int h\left(\eta \right)\rho \left(d\eta \right)-\mathrm{KL}(\rho \parallel \mu )].$

We can define the truncation of distribution $\hat{q}(\cdot )$ on the set $H_n\left(\theta \right)$ denoted as $\check{q}(\cdot )$, (i.e. $\check{q}\left(\theta \right)=\hat{q}\left(\theta \right)\mathrm{𝟙}\{\theta \in H_n\left(\theta \right)\}/\hat{q}\left(H_n\left(\theta \right)\right)$ ), similarly we can also define the $\tilde{\pi }\left(\theta \right)=\pi \left(\theta \right)\mathrm{𝟙}\{\theta \in H_n\left(\theta \right)\}/\hat{q}\left(H_n\left(\theta \right)\right)$. By adopting the arguments from (Bai et al., 2020), and following steps analogous to those leading to Equation (17) therein, we obtain:

${\int }_{H_n\left(\theta \right)}\eta (P_{\theta },P_0)\tilde{\pi }\left(\theta \right)𝑑\theta \le \exp \left(C_{0}n{\epsilon }_n^{\phantom{}\ast 2}\right),\text{w.h.p.}$

(30)

for some constant $C_0>0$, where $\log \eta (P_{\theta },P_0)=l_n(P_{\theta },P_0)+\frac{n}{3}{d}^2(P_{\theta },P_0)$.

Then, given the Lemma 1 and equation (30), we can show that the first term can be bounded w.h.p. as:

$\begin{array}{cc}& \frac{n}{3\hat{q}\left(H_n\left(\theta \right)\right)}{\int }_{H_n\left(\theta \right)}{d}^2(P_{\theta },P_0)\hat{q}\left(\theta \right)𝑑\theta \\ & \le Cn\epsilon {}_n{}^{\phantom{}\ast 2}+\text{KL}\left(\check{q}\right(\theta )\parallel \tilde{\pi }(\theta \left)\right)-\int {}_{H_n\left(\theta \right)}l{}_n(P{}_{\theta },P{}_0)\check{q}\left(\theta \right)d\theta \\ & =Cn\epsilon {}_n{}^{\phantom{}\ast 2}+\frac{1}{\hat{q}\left(H_n\left(\theta \right)\right)}\left(\text{KL}\right(\hat{q}\left(\theta \right)\parallel \pi \left(\theta \right))-\int {}_{\Theta }l{}_n(P{}_{\theta },P{}_0\left)\hat{q}\right(\theta \left)d\theta \right)\\ & -\frac{1}{\hat{q}\left(H_n\left(\theta \right)\right)}\left({\int }_{H_n{\left(\theta \right)}^c}\log \frac{\hat{q}\left(\theta \right)}{\pi \left(\theta \right)}\hat{q}\left(\theta \right)d\theta -{\int }_{H_n{\left(\theta \right)}^c}{l}_n(P_{\theta },P_0)\hat{q}\left(\theta \right)d\theta \right)+\log \frac{\pi \left(H_n\left(\theta \right)\right)}{\hat{q}\left(H_n\left(\theta \right)\right)}.\end{array}$

(31)

Additionally, since $d^2(P_{\theta },P_0)\le 1$, the second and the third terms of Equation (29) are bounded by:

	${\int }_{\theta \in H_n^{\prime }\left(\theta \right)}{d}^2(P_{\theta },P_0)\hat{q}\left(\theta \right)𝑑\theta$	$\phantom{}\le {\int }_{\theta \in H_n^{\prime }\left(\theta \right)}\hat{q}\left(\theta \right)𝑑\theta =\hat{q}\left(H_n^{\prime }\left(\theta \right)\right),$
	${\int }_{\theta \in H_n^{\prime \prime }\left(\theta \right)}{d}^2(P_{\theta },P_0)\hat{q}\left(\theta \right)𝑑\theta$	$\phantom{}\le {\int }_{\theta \in H_n^{\prime \prime }\left(\theta \right)}\hat{q}\left(\theta \right)𝑑\theta =\hat{q}\left(H_n^{\prime \prime }\left(\theta \right)\right).$

Substituting the bound on the first term from Equation (31), together with the two bounds above, into Equation (29), we obtain with high probability:

$\begin{array}{cc}\int d^2(P_{\theta },P_0)\hat{q}\left(\theta \right)d\theta \le \phantom{}& \mathrm{\hspace{0.33em}3}\hat{q}\left(\Theta {}_n\right)C\epsilon {}^{\phantom{}\ast 2}{}_n+\frac{3}{n}\left(\mathrm{KL}\right(\hat{q}\left(\theta \right)\parallel \pi \left(\theta \right))-\int {}_{H_n\left(\theta \right)}l{}_n(P{}_{\theta },P{}_0\left)\hat{q}\right(\theta \left)d\theta \right)\\ & +\frac{3}{n}\int {}_{H_n^{\prime }{\left(\theta \right)}^c}l{}_n(P{}_{\theta },P{}_0)\hat{q}\left(\theta \right)d\theta +\frac{3}{n}\int {}_{H_n^{\prime }{\left(\theta \right)}^c}\log \frac{\pi \left(\theta \right)}{\hat{q}\left(\theta \right)}\hat{q}\left(\theta \right)d\theta \\ & +\frac{3\hat{q}\left(H_n\left(\theta \right)\right)}{n}\log \frac{\pi \left(H_n\left(\theta \right)\right)}{\hat{q}\left(H_n\left(\theta \right)\right)}+\hat{q}\left(H_n^{\prime }\left(\theta \right)\right)+\hat{q}\left(H_n^{\prime \prime }\left(\theta \right)\right).\end{array}$

(32)

Following the procedure in (Bai et al., 2020, Lemma 4.2, equation 20), we can show with high probability that:

	$\int d^2(P_{\theta },P_0)\hat{q}\left(\theta \right)𝑑\theta$	$\le 3\hat{q}\left(H_n\right)C{\epsilon }_n^{\phantom{}\ast 2}+\frac{3}{n}(\mathrm{KL}(\hat{q}\left(\theta \right)\parallel \pi \left(\theta \right))-\int l_n(P_{\theta },P_0)\hat{q}\left(\theta \right)d\theta )+\frac{3}{n}{\int }_{H_n^c}{l}_n(P_{\theta },P_0)\hat{q}\left(\theta \right)d\theta$
		$+\frac{3}{n}{\int }_{H_n^c}\log \frac{\pi \left(\theta \right)}{\hat{q}\left(\theta \right)}\hat{q}\left(\theta \right)d\theta +\frac{3\hat{q}\left(H_n\right)}{n}\log \frac{\pi \left(H_n\right)}{\hat{q}\left(H_n\right)}+\hat{q}\left(H_n^{\prime }\left(\theta \right)\right)+\hat{q}\left(H_n^{\prime \prime }\left(\theta \right)\right)$
		$=3\hat{q}\left(H_n\right)C{\epsilon }_n^{\phantom{}\ast 2}+\frac{3}{n}(\mathrm{KL}(\hat{q}\left(\theta \right)\parallel \pi \left(\theta \right))-{\int }_{{ℝ}^T}{l}_n(P_{\theta },P_0)\hat{q}\left(\theta \right)d\theta )+\frac{3}{n}{\int }_{H_n^c}{l}_n(P_{\theta },P_0)\hat{q}\left(\theta \right)d\theta$
		$+\frac{3}{n}{\int }_{H_n^c}\log \frac{\pi \left(\theta \right)}{\hat{q}\left(\theta \right)}\hat{q}\left(\theta \right)d\theta +\frac{3\hat{q}\left(H_n\right)}{n}\log \frac{\pi \left(H_n\right)}{\hat{q}\left(H_n\right)}+\hat{q}\left(H_n^{\prime }\right)+\hat{q}\left(H_n^{\prime \prime }\right)$
		$\le C{\epsilon }_n^{\phantom{}\ast 2}+\frac{3}{n}(\mathrm{KL}(\hat{q}\left(\theta \right)\parallel \pi \left(\theta \right))-{\int }_{{ℝ}^T}{l}_n(P_{\theta },P_0)\hat{q}\left(\theta \right)d\theta )+O(1/n)+\hat{q}\left(H_n^{\prime \prime }\right),$

where $C$ is some constant. Next, we show that $\hat{q}\left(H_n^{\prime \prime }\left(\theta \right)\right)=O\left({\epsilon }_n^{\phantom{}\ast 2}\right)$.

Lemma 5.

Given the Condition 5, $\hat{q}\left(H_n^{\prime \prime }\left(\theta \right)\right)=O\left({\epsilon }_n^{\phantom{}\ast 2}\right)$ holds.

Proof.

Let ${\hat{\sigma }}_m=\max \{{\hat{\sigma }}_1,\dots ,{\hat{\sigma }}_K\}$, then by definition of the variational distribution $q\left(\theta \right)$,we can know that:

$\hat{q}\left(H_n^{\prime \prime }\left(\theta \right)\right)\le \sum _{i=1}^T\hat{q}\left(\left|{\theta }_i\right|>\left(B+1\right)\right)=\sum _{i=1}^T\hat{q}\left({\theta }_i>\left(B+1\right)\right)+\hat{q}\left({\theta }_i<-\left(B+1\right)\right).$

By the definition of variational distribution $\hat{q}(\cdot )$, we know:

	$\hat{q}\left({\theta }_i>B+1\right)$	$\phantom{}\le {\int }_{B+1}^{\infty }\frac{1}{\sqrt{2\pi {\sigma }_m^2}}\exp \left(\frac{-{\left(t-B\right)}^2}{2{\sigma }_m^2}\right)𝑑t,$
	$\hat{q}\left({\theta }_i<-\left(B+1\right)\right)$	$\phantom{}\le {\int }_{-\infty }^{-\left(B+1\right)}\frac{1}{\sqrt{2\pi {\sigma }_m^2}}\exp \left(\frac{-{\left(t-B\right)}^2}{2{\sigma }_m^2}\right)𝑑t.$

And by Chernoff bound and the fact that ${\sigma }_m^2\le \frac{1}{2\log \left(T/{\epsilon }_n^{\phantom{}\ast 2}\right)}$, we can have:

${\int }_{B+1}^{\infty }\frac{1}{\sqrt{2\pi {\sigma }_m^2}}\exp \left(\frac{-{\left(t-B\right)}^2}{2{\sigma }_m^2}\right)𝑑t\le \frac{1}{2}\exp \left(-\frac{1}{2{\sigma }_m^2}\right)\le \frac{{\epsilon }_n^{\phantom{}\ast 2}}{2T}.$

Thus we show that $\hat{q}\left(H_n^{\prime \prime }\right)\le {\epsilon }_n^{\phantom{}\ast 2}$. ∎

Then by Lemma 5, we complete the proof. ∎

Appendix C Implementation of SQS

Pretrained model setting. Taking the compression of the Llama3.2-1B model as an example, we first download the pre-trained model from Hugging Face³³ 3 https://huggingface.co/meta-llama/Llama-3.2-1B using the Python package “transformers”. We then fine-tune the model on the considered SST-2 task before applying our SQS for compression. We find that omitting the fine-tuning step significantly degrades the performance of SQS.

For ResNet models, we use publicly available pre-trained models obtained by the Python package “timm’’ on the CIFAR-10 and CIFAR-100 datasets and directly apply our SQS for compression. For the BERT-base model, we download the pre-trained model from Hugging Face⁴⁴ 4 https://huggingface.co/huggingface-course/bert-finetuned-squad using the Python package “transformers”.

Initialization. To initialize the learnable parameters of our SQS method, denoted as ${\{{\mu }_k,{\sigma }_k,{\pi }_k\}}_{k=1}^K$, we employ the K-means algorithm. Specifically, the DNN weights of a given layer are first clustered into $K$ groups. For each group $G_k$, the mean ${\mu }_k$ and standard deviation ${\sigma }_k$ are computed as the empirical statistics of the weights in that group, while the mixture coefficient ${\pi }_k$ is set to the proportion of weights in group $k$ relative to the total number of weights in the layer.

We assume that K-means yields $K$ disjoint groups of weights, denoted as $\{G_1,\dots ,G_K\}$, such that ${\bigcup }_{k=1}^K{G}_k$ covers all weights in the selected layer. The initial parameters are then defined as:

${\mu }_k=\frac{1}{\left|G_k\right|}\sum _{{\theta }_i\in G_k}{\theta }_i,{\sigma }_k=\sqrt{\frac{1}{\left|G_k\right|-1}\sum _{{\theta }_i\in G_k}{\left({\theta }_i-{\mu }_k\right)}^2},{\pi }_k=\frac{\left|G_k\right|}{{\sum }_{j=1}^K\left|G_j\right|}.$

Implementing marginal $q\left({\theta }_i\right)$. To make the distribution differentiable, we reparameterize ${\tilde{\lambda }}_i$ (as defined in Equation 6) using the following equation:

${\tilde{\lambda }}_i=\frac{\exp \left({\tilde{s}}_i/{\tau }^{\prime }\right)}{1+\exp \left({\tilde{s}}_i/{\tau }^{\prime }\right)},$

(33)

where the temperature ${\tau }^{\prime }$ is set to a fixed constant to stabilize the training process. To better exploit the learned pruning parameters in later stages of training, we halve ${\tau }^{\prime }$ after completing half of the total training steps to stabilize the training and sharpen the retention probabilities around their learned optima.

Hyperparameter Configuration. The number of Gaussian components $K$ is not fixed across experiments; it is chosen per model and can be read off the Bits column of each experiment table, since $\text{Bits}={\log }_{2}K$. Specifically, we use $K=4$ ($2$ bits) for the ResNet models on CIFAR-10 in Table 1, $K=16$ ($4$ bits) for BERT-base on SQuAD v1.1 in Table 2, and $K=64$ ($6$ bits) for Llama3.2-1B and Qwen2.5-0.5B on SST-2 in Table 3. The ablation studies state their own $K$ in the corresponding table or figure caption. A larger $K$ reduces the performance drop at the cost of a lower compression rate; this trade-off is analyzed in the case study shown in Figure 3. For the experiments not reporting the Bayesian Averaging number $M$, the default is set to 4. All models are trained on an NVIDIA H100 GPU with 80 GB of memory.

During training and testing, for ResNet-18, ResNet-20, ResNet-32, ResNet-50, ResNet-56, BERT-base, Llama3.2-1B, and Qwen2.5-0.5B models, the settings are:

• Training Time: Approximately 30 minutes for ResNet models (ResNet-18 through ResNet-56); 4 hours for BERT-base; 24 hours for both Llama3.2-1B and Qwen2.5-0.5B.

• Optimizer: AdamW is used consistently across all models.

• Quantization Learning Rate: $5\times {10}^{-4}$ for ResNet-18; $5\times {10}^{-5}$ for all other models.

• Pruning Learning Rate: Fixed at $0.012$ for all models.

• Temperature Hyperparameters: $\tau$ in Equation (2) is set to $5\times {10}^{-4}$ and ${\tau }^{\prime }$ in Equation (33) is set to $0.0125$.

• Pruning Schedule: A polynomial schedule is used for all models.

Appendix D Experiment Settings

D.1 Experiment settings for benchmark with all baselines

Benchmark compression on ResNet models. We present experiments using ResNet architectures on the CIFAR-10 and CIFAR-100 datasets. When compressing ResNet models, our method requires fine-tuning over the training dataset, completing the compression process within $10$ epochs. To achieve high compression rates, we represent each layer’s weights with $4$ components (i.e., $K=4$ for each layer) and apply a sparsity level of $50\%$. As shown in Table 1, our method achieves 32× compression on all three ResNet models, with reported accuracy drops below 1.5 percentage points. For example, compressing ResNet-32 by a factor of $32\times$ yields a minimal accuracy reduction of $1.29\%$. Additionally, we compress ResNet-56 by a factor of $32\times$, observing an accuracy drop of only $0.84\%$. Compared to other methods, our approach achieves much higher compression rates with smaller decreases in accuracy.

Benchmark compression on BERT-base model. We further investigate our compression method on attention-based models. We apply our compression model on the BERT-base (Devlin et al., 2019) model and test it on the SQuAD V1.1 dataset (Rajpurkar et al., 2016). Similarly, we consider the F1 score drop and compression rate as the evaluation metrics. During the compression process, the BERT model is fine-tuned on the training dataset, with the entire procedure completed within $3$ epochs.

We compressed the BERT model using $K=16$ Gaussian components and pruned $75\%$ of its parameters, leading to a $32\times$ compression rate. We employed layer-wise quantization combined with unstructured pruning to attain these results.

Benchmark compression on Llama and Qwen models. Due to hardware limitations, we cannot run very large-scale LLMs, which are Llama3.1-8B and Qwen2.5-7B.

D.2 Experiment settings for ablation studies for SQS method

Impact of different priors. For comparison, we consider a zero-mean Gaussian distribution as the prior and replace the delta distribution with a Gaussian distribution in the variational family. That is, any $q^{\prime }\left(\theta \right)\in {ℱ}^{\prime }$ has the form:

${\tilde{\theta }}_i|{\gamma }_i\sim {\gamma }_i\sum _{k=1}^K{\varphi }_k\left({\theta }_i\right)𝒩({\mu }_k,{\sigma }_k^2)+\left(1-{\gamma }_i\right)𝒩(0,{\sigma }_0^2),$

${\gamma }_i\sim \text{Bern}\left({\tilde{\lambda }}_i\right).$

Based on this, we can get the modified marginal variational distribution $q^{\prime }\left({\tilde{\theta }}_i\right)$ as:

$q^{\prime }\left({\tilde{\theta }}_i\right)={\tilde{\lambda }}_i\sum _{k=1}^K{\varphi }_k\left({\theta }_i\right)𝒩({\mu }_k,{\sigma }_k^2)+\left(1-{\tilde{\lambda }}_i\right)𝒩(0,{\sigma }_0^2).$

(34)

Thus, following the same reasoning and derivation, we get the equation (8), The Gaussian prior can be obtained by:

	${\Omega }^{\prime }$	$=-{𝔼}_{q^{\prime }\left(\tilde{\theta }\right)}\left[\log p\left(D\right|\tilde{\theta })\right]+\sum _{i=1}^T\mathrm{KL}(q^{\prime }\left({\tilde{\theta }}_i\right)\parallel 𝒩(0,{\sigma }_0^2))$
		$=-{𝔼}_{q^{\prime }\left(\tilde{\theta }\right)}\left[\log p\left(D\right|\tilde{\theta })\right]+\sum _{i=1}^T\mathrm{KL}(q^{\prime }\left({\tilde{\theta }}_i\right)\parallel ({\tilde{\lambda }}_i+(1-{\tilde{\lambda }}_i))𝒩(0,{\sigma }_0^2))$
		$\le -{𝔼}_{q^{\prime }\left(\tilde{\theta }\right)}\left[\log p\left(D\right|\tilde{\theta })\right]+\sum _{i=1}^T{\tilde{\lambda }}_i\mathrm{KL}(\sum _{k=1}^K{\varphi }_k\left({\theta }_i\right)𝒩({\mu }_k,{\sigma }_k^2)\parallel 𝒩(0,{\sigma }_0^2)).$		(Gaussian prior)

We compare the impact of the above Gaussian prior with the proposed Spike-and-GMM priors and summarize the result in Table 4.

Description of Baselines. For the following lines of baselines, we use the reported results in their papers:

• Optimal Brain Surgeon (OBS) (Hassibi et al., 1993) selects weights for removal from a trained neural network using second-order information.

• BitSplit (Wang et al., 2020a) incrementally constructs quantized values using a squared error metric based on residual errors.

• AdaQuant (Hubara et al., 2021) utilizes STE for direct optimization.

• BRECQ (Li et al., 2021) integrates Fisher information into the optimization process and focuses on the joint optimization of layers within individual residual blocks.

• Exact Optimal Brain Quantization (OBQ) (Frantar et al., 2022) adapts second-order weight pruning methods to quantization tasks.

• GPTQ (Frantar et al., 2023) employs second-order information for error compensation on calibration sets to speed up generative models.

We adopt their implemented code and use the same setting for training and testing:

• AWQ (Lin et al., 2024) implements activation-aware quantization, selectively bypassing the quantization of key weights⁵⁵ 5 https://github.com/mit-han-lab/llm-awq. This method is training-free and does not need extra training on the selected dataset.

• DGMS (Dong et al., 2022) is an automated quantization method that utilizes Mixtures of Gaussians to avoid the aforementioned problem⁶⁶ 6 https://github.com/RunpeiDong/DGMS. We use their codebase and configure it with the same hyperparameters. Their algorithm is trained on the same dataset for fairness of comparison.

Definition of Evaluation Metrics. Let $K$ denote the number of shared weight vectors. Then, the metric Bits is defined as ${\log }_{2}K$.

D.3 Long-tailed Full-precision Weight Distribution of Llama3.2 and Qwen2.5 Models

We present the visualization of the long-tailed weight distributions for the Llama3.2 model in Figures 6 and 7, and for the Qwen2.5 model in Figures 4 and 5.

媒体内容 · 前往原文查看

媒体内容 · 前往原文查看

媒体内容 · 前往原文查看

媒体内容 · 前往原文查看