fusion-runtime runs speech recognition, a language model and speech synthesis in one process, so a reply starts playing while it is still being written. It is Apache-2.0, it wants Python 3.11 to 3.13, and the interesting part is not the architecture. It is the benchmark table.
Two numbers, because there are two honest answers
On one RTX 3090 with a quantised 7B model, Whisper small and Kokoro sharing the card: about 490 ms of processing from the end of a turn to audio coming back, 127 tokens per second, and interruptions honoured mid-sentence. Synthesis runs about eleven times faster than real time.
The author gives two totals and says which is which. A stopwatch started at your last syllable reads 991 ms, and about 500 ms of that is the runtime waiting through silence to decide you have finished, which elapses while you are still finishing the sentence. He points out that a voice stack advertising a number under 500 ms is almost always measuring from the moment it decided you had stopped, not from the moment you did. He also explains why the stages do not sum to the total, which is that transcribing what you have already said happens during that silence wait, and that the gap between the first token and the first audio is mostly the model writing a whole sentence before the speech can start.
Concurrency, honestly
One caller gets about 460 ms. Four callers get about 740 ms, which is the same range within run-to-run variance. Then it falls over: eight callers get 4600 ms and twelve get 7500 ms, with throughput plateauing around 0.7 turns per second. The bottleneck is one specific thing. A single in-process llama.cpp context decodes one reply at a time, so at twelve callers the model’s first token accounts for 4790 ms of a 5312 ms response while speech recognition stays at 76 ms. The fix is to move the language model out to vLLM or llama-server with slots and leave speech where it is, and whether that moves the knee is not yet measured.
Running it
frun up starts a server, and the browser client it serves is the same one you would embed in your own page. With no keys configured it binds to localhost only and refuses to listen anywhere else. A page gets a single-use session token, because a page cannot hold a secret. The agent file names the environment variable that holds an API key rather than the key itself, so it is safe to commit and frun doctor checks the machine before you blame the code.
My read
The latency section is the reason to read this. Projects rarely publish the number that makes them look worse, explain why the flattering version is dishonest, and show where the ceiling is. The concurrency table also tells you what one card actually buys, which is a phone line for a household rather than anything that answers for a business.
Source: fusion-runtime