Impact of Post-Workout Quantization on Local Inference of the Whisper Model on Consumer Hardware

Local execution of speech recognition models, such as Whisper, is limited by high computational cost. In this work, the feasibility of this execution via post-training quantization is investigated, with analysis of trade-offs between memory, latency and accuracy. Five precision schemas are evaluated on a CPU and a GPU that do not natively support low-precision operations. It is indicated by the results that the model is reduced by up to 73%, but without guaranteeing acceleration on non-specialized hardware. The degradation of the error rate is limited to 2.5 percentage points. In addition, it is evidenced that more accurate transcriptions are produced by CTranslate2 independently of compression. Finally, a local pipeline with transcription, voice detection, and diarization is proposed, which enables the processing of sensitive data without external services.