STT→LLM→TTS on CPU is possible for many product loops — especially when language coverage and privacy beat peak latency.
The hidden work is model ops: Whisper format conversions, GGUF selection, TTS voice packs, and volume bootstrap in Compose.
Automate the ugly conversions
Init jobs that download and convert weights prevent README-only folklore.
Keep models out of image layers; use volumes.
Protocol over playground
A WebSocket voice API lets real clients integrate. A demo UI is not enough.
Takeaways
- Model ops is most of voice delivery.
- Pick small instruct models on purpose.
- Ship an integration protocol.