Edge CPU Voice Assistant

Outcomes

Context

Language coverage and privacy often matter more than peak tokens/sec for local assistants.

Challenge

Voice assistants usually assume GPUs or cloud STT/LLM/TTS. The requirement was a workable Romanian speech loop on CPU, privately, with a simple WebSocket API and test UI.

Approach

We packaged faster-whisper STT (including CTranslate2 conversion for regional Whisper weights), llama.cpp GGUF LLM inference, Piper Romanian TTS, and a FastAPI WebSocket voice endpoint. Docker Compose init jobs download/convert models into volumes on first boot.

Architecture

FastAPI WebSocket voice service, faster-whisper STT, llama.cpp LLM, Piper TTS, Docker Compose model volumes and init converters.

Scope

• —Speech recognition
• —Local LLM turns
• —Speech synthesis
• —WebSocket API
• —Model bootstrap

Constraints

• —CPU latency budgets
• —Model download size
• —Transformers→CTranslate2 conversion for Whisper variants

Solution highlights

• —Init containers for one-time model conversion
• —Small instruct GGUF for CPU practicality
• —Thin WebSocket protocol for clients

Technologies

Results

• —End-to-end STT→LLM→TTS loop on CPU
• —Romanian-first speech synthesis and recognition path
• —WebSocket API for product integration
• —Automated model fetch/convert on compose up

Lessons learned

• —Voice stacks are model-ops problems as much as ML problems.
• —Format conversion belongs in automation, not tribal README steps only.
• —CPU assistants need ruthless model size discipline.