Skip to article
DikayaChat

LLM API timeouts: avoid paying for blind retries

Dikaya editorial team · Reviewed

An LLM API timeout tells you that your client stopped waiting. It does not prove that generation stopped, that nothing was saved, or that no cost was incurred. For developers integrating Dikaya, the useful default is one deliberate submission, an explicit “outcome unknown” state, and a manual decision after checking available evidence. This guide builds that workflow without pretending a local request ID makes an arbitrary API endpoint idempotent.

Key takeaways

  • A received HTTP error and a connection lost after submission are different kinds of evidence.
  • Disable automatic SDK retries while validating an integration; choose timeouts explicitly.
  • Keep application operation IDs for reconciliation, but do not assume public API replay protection.

First identify the boundary that failed

Suppose your application sends a writing request and waits 30 seconds. The socket closes before a complete answer arrives. The provider may have rejected the request, may still be generating, or may have completed an answer that never reached your application. One “failed” label collapses these possibilities and makes the next button dangerous: a user reasonably expects Retry to recover the same operation, while your code may start another paid generation.

Use three separate outcomes: a complete response, a known HTTP rejection, and uncertain delivery. MDN documents the same distinction for fetch: an HTTP 404 or 500 normally resolves to a Response, whereas a network failure rejects the promise. A status code alone still does not establish the financial outcome of every server error. Preserve the response and any accounting information rather than converting every non-200 into “not charged”.

Make SDK retries a visible choice

The OpenAI Python library documents two retries by default for several connection and HTTP failure classes. That behavior can be useful for some APIs, but an OpenAI-compatible base URL does not automatically inherit every upstream idempotency guarantee. During initial integration, set max_retries=0 so that one application call does not silently become multiple attempts. Pin the SDK version in your application and check its own release documentation when upgrading.

The example below makes one text Chat Completions call. Set DIKAYA_API_KEY in the process environment; never place a real key in browser JavaScript or a public repository. The 120-second client timeout is an example waiting policy, not a service latency guarantee. The explicit 512-token output cap is separate from the timeout and is not a cap on total input-plus-output cost.

import os
from openai import OpenAI, APIConnectionError, APIStatusError

client = OpenAI(
    base_url="https://dikaya.cc/v1",
    api_key=os.environ["DIKAYA_API_KEY"],
    max_retries=0,
    timeout=120.0,
)

try:
    result = client.chat.completions.create(
        model="seller-pro-2",
        messages=[{"role": "user", "content": "Give three checks for a CSV import."}],
        max_completion_tokens=512,
    )
except APIConnectionError:
    print("Outcome unknown. Reconcile before another submission.")
except APIStatusError as error:
    print(f"HTTP {error.status_code}; retain the response for review.")
else:
    print(result.choices[0].message.content or "")
    print(result.usage)

Keep a small operation journal

Before submitting, record a local operation ID, creation time, selected model, output cap and a fingerprint of the input. After receiving a response, add its status, usage when present, and whether the body was complete. Avoid copying sensitive prompts into routine error logs. A fingerprint helps identify accidental duplicate work without turning every log system into a second conversation archive.

That operation ID is your reconciliation handle, not a magic header. Do not invent an Idempotency-Key contract for /v1/chat/completions. The Dikaya browser chat has its own persisted run and request-ID machinery, but it is a separate internal workflow, not a promise that arbitrary public API calls are deduplicated. If your application needs durable jobs, implement its own submit-once state transition and make uncertain jobs stop for review.

  • Write “submitting” before the network call.
  • Record a received HTTP status without replacing it with a generic network error.
  • After a lost acknowledgement, mark the operation “unknown”, not “safe to retry”.
  • Require a deliberate decision to submit a new operation.

Treat an interrupted stream as a partial result

With streaming, visible text proves that some output reached you; it does not prove that the final usage or completion event arrived. Save the accumulated text as partial and keep the terminal state separate. A stop button or AbortController ends local waiting, but cannot by itself establish remote cancellation or a refund. Even a real server cancellation can leave already-consumed work to account for.

When resuming a task, show the partial result and ask whether to continue from it or start fresh. If you send “continue” with that text as context, this is a new request with its own token cost. Do not label it a free transport recovery. In browser chat, inspect the existing run state before creating another turn; changing the interface language must not replay the request.

Test uncertainty without spending on model calls

Use a local HTTP test server with four cases: immediate rejection, delayed success, connection close after accepting the body, and a stream that ends after its first chunk. Assert that only an explicit new action creates another submission. Count calls at the receiving server, not only in the client mock: a hidden SDK retry can otherwise escape your test.

A second test should close the page or restart the worker between “submitting” and the response. On restart, the application must preserve “unknown” rather than resetting the job to an empty draft. These tests exercise your state machine; they are not measurements of production model speed or service availability.

Reconcile before choosing the next action

Compare the operation time, model and available account usage/history. If the evidence is incomplete, contact account support with a concise operation description rather than forwarding the key. Do not infer a refund from an empty text response or a missing client log. Where the evidence cannot settle the outcome, keep that uncertainty visible.

The practical win is not “retry nothing forever”. It is knowing when your next click is a new piece of work. Start with one bounded request, verify the usage you actually receive, then add background execution and richer recovery only after the failure tests pass.

Questions and answers

Does a timeout mean my request was not charged?

No. A timeout describes the client’s waiting limit. The server may have accepted or completed the work. Check available run, usage and account evidence; if it is incomplete, retain an unknown outcome rather than promising a zero charge.

Can I reuse a local UUID to deduplicate public API calls?

A local UUID is useful for your application journal, but does not create server-side idempotency. Do not assume a header or request field is supported without a documented endpoint contract. Browser-chat run IDs belong to a separate internal workflow.

Is max_completion_tokens a complete dollar budget?

No. It constrains output subject to model and account limits. Input, cached input, reasoning accounting and any separate requests also matter. Calculate an estimate from the actual tariff and compare it with reported usage after a completed call.

Sources

AI-assisted editorial material. Examples are illustrative, not captured model responses or comparative tests. External sources do not replace the service terms.

Take this task to chat

Only the draft question is transferred. Nothing is sent until you choose to send it. New generation uses your account conditions.

Take this task to chat Documentation