Identify which timeout actually fired
A connection timeout happens before a usable response is established. An idle-read timeout happens after the response starts but no bytes arrive for too long. An overall deadline stops a request even if chunks continue to arrive. Raising one value does not repair the other two.
- Log when the request started, when headers arrived, when the first event arrived, and when the last event arrived.
- Keep the endpoint, protocol, model, effort, status code, and attempt number with the timing record.
- Do not reduce every transport failure to “network error”; the recovery action depends on the stage.
A partial stream is not a completed answer
OpenAI Responses emits a response.completed event for a successful completed response. Anthropic Messages ends a normal SSE sequence with message_stop. Text deltas before those events are useful for display and recovery, but they do not prove the request completed.
If the socket closes early, preserve the partial text for diagnosis but mark the evaluation attempt as failed. Scoring a truncated answer as if it were complete silently rewards transport failures.
Streaming cannot override an absolute proxy limit
Regular events can prevent an idle connection from looking dead, but they cannot extend an absolute request cap enforced by a client, reverse proxy, edge platform, or provider. When a request repeatedly ends at nearly the same wall-clock time, inspect every hop rather than only the model timeout.
Retry transport failures without creating an endless run
A retry is a new model request. It may produce a different answer and may incur another charge even when the first response never reached your client. Keep retries bounded, use backoff for rate limits, and store each attempt separately.
For evaluation, select the answer according to a fixed retry policy declared before the run. Do not keep retrying only low-scoring answers, because that changes the comparison standard.