MoE Compression of LLM Weights

Problem

Compression results are usually reported as a size reduction. That number says nothing about whether the compressed model still produces equivalent outputs, which is the property that actually matters at inference time.

Approach

• 01Applied dynamic quantization across linear layers, reducing on-disk size from roughly 90MB to 52MB while preserving parameter count.
• 02Injected low-rank approximated linear layers using the top 32 singular values, storing compressed weights separately for inference.
• 03Built an evaluation suite running fixed-prompt inference across original and compressed models, logging latency, logit cosine similarity, parameter count, and disk size.
• 04Tracked semantic degradation through output cosine similarity while validating inference correctness.

Outcome

Roughly 42 percent on-disk reduction at a measured output cosine similarity of about 0.71 against the original model.