PRODUCTION SCENARIO
A voice assistant repeats a large stable prompt prefix on every turn. It buffers the entire generated response before starting speech synthesis, so callers hear four to six seconds of silence before the first word.
Which change lets the caller hear speech before the full response is generated?
Answering here is anonymous. Nothing is saved unless you sign in.
Show answer and explanation
Answer: Stream the reply into speech synthesis sentence by sentence
Streaming is the one change that overlaps generation with playback: synthesis starts on the first finished sentence instead of the last token, which cuts the wait to about a second. Caching the prefix shortens prefill, and the caller still waits for the whole reply. A smaller model and a lower output cap shorten total generation and leave the buffer in place, so the silence gets shorter and never goes away. Measure first-audio latency separately from full-response latency.
Official Source: https://developers.openai.com/api/docs/guides/latency-optimization ↗