Vaania Kaul

Effect of Quantization and Pruning on ML Model Performance and Energy

Started Oct. 3, 2025

Abstract or project description

Dense retrieval models require significant compute and memory bandwidth, making them ideal candidates for quantization and pruning. This project evaluates how different optimization techniques such as binary vs scalar embedding quantization, light structured pruning, and compiled runtimes such as ONNX Runtime and TorchScript, affect the efficiency and performance of a standard dense retrieval architecture. Using benchmarks such as MS MARCO Passage Ranking, the study will measure retrieval accuracy, latency, throughput, and memory usage. Energy consumption will be measured on Apple Silicon using Zeus, enabling power comparisons across optimization settings. Results will then be extrapolated to predict how these techniques would behave on compute-in-memory (CIM) accelerators, which benefit more from low-precision operations.