LLM Provider Compatibility

​

Provider detection

1. Explicit provider (from DB ModelConfig.provider field) — highest priority. If the provider matches a known domain in the URL, no api_base is returned (LiteLLM routes natively). Otherwise, api_base is set to the relay URL.
2. Domain match against KNOWN_DOMAINS — official API endpoints are recognized by hostname.
3. URL path hint against PATH_PROVIDER_HINTS — common on relay platforms like UniAPI where /claude or /anthropic in the path indicates the upstream protocol.
4. Fallback — openai/ prefix (generic OpenAI-compatible).

​

tool_choice — the four modes

​

Where tool_choice is used

​

ReAct engine — tool_choice=“auto”

​

structured_llm_call — tool_choice=forced

​

Structured output: the four guarantee tiers

​

T1 and T2 are not interchangeable

​

What T2 costs

• A schema subset. OpenAI’s strict mode wants additionalProperties: false on every object and every property listed in required; optionality is expressed as a union with null rather than by omission from required. Root must be an object. There are caps on total properties and nesting depth. Most hand-written schemas need editing before they are eligible.
• Grammar compilation. The first request carrying a new schema pays a one-off compile latency on the provider side. Reusing a stable schema amortises it; generating a fresh schema per request does not.
• A new failure surface. A model that will not answer returns a refusal rather than a schema-shaped object, so the caller needs a branch for it.

​

What no tier guarantees

​

Where FIM One sits

1. The chain has no schema-gated rung to fall back to. When Level 1 fails, the next rung down is json_object, which guarantees only that the text parses. There is no intermediate step that still enforces fields.
2. Level 1’s guarantee is weaker than its name suggests. Without strict, native function calling is the pre-2024 non-strict variety: the model usually honours the schema and is not prevented from omitting a required field, inventing a key, or returning a string where a number was declared.
3. On anthropic/ routes, Level 2 is not really T3. The Anthropic Messages API has no response_format, so LiteLLM emulates JSON mode by injecting an assistant prefill. That is a prompt-layer device, which puts the effective guarantee between T4 and T3 rather than at T3. The Bedrock prefill trap documented below is this emulation failing loudly; the quieter cost is that the guarantee was never the one the parameter name implies.
4. Nothing validates the result against the schema afterwards. jsonschema is not a dependency. Whatever checking happens is per-call-site, inside the optional parse_fn, and its rigour varies by call site.
5. Failure is absorbed rather than surfaced. Nearly every caller passes default_value, so an exhausted chain returns a plausible-looking object instead of raising. StructuredCallResult.level_used records which rung produced the value, but no call site reads it, and on the default_value path it reports plain_text for a chain that in fact produced nothing.

structured_llm_call: fn=classify_mode level=native_fc attempt=first calls=1
structured_llm_call: fn=annotate_tables level=json_mode attempt=reformat_retry calls=3

​

Provider support for each tier

​

The Bedrock prefill trap

{"role": "assistant", "content": "{"}

ValidationException: This model does not support assistant message prefill.
The conversation must end with a user message.

1. The model is resolved with the anthropic/ prefix (via domain match or URL path hint).
2. response_format={"type":"json_object"} is passed (the json_mode code path in structured_llm_call).
3. The actual backend is AWS Bedrock (which rejects prefill).

​

The fix: json_mode_enabled

• DB-configured models: toggle in Admin → Models → Advanced settings. The flag is stored on ModelProviderModel.json_mode_enabled (default TRUE).
• ENV-configured models: set LLM_JSON_MODE_ENABLED=false in your environment.
• Effect: when disabled, abilities["json_mode"] returns False → response_format is never passed → no prefill → Bedrock works. The degradation chain becomes native_fc → plain_text, skipping the doomed json_mode call entirely.
• Cost of the skip: the model still returns valid JSON in practice, because the system prompt asks for it and extract_json() parses free-form content reliably on modern models. What is lost is the guarantee rather than the output: the chain now ends at T4, where nothing but the prompt constrains the result. On an anthropic/ route that loss is smaller than it looks, since the emulated JSON mode was never a decoder-level guarantee either.

​

Thinking models + forced tool_choice

tool_choice 'specified' is incompatible with thinking enabled

​

The fix: tool_choice_enabled

• DB-configured models: toggle in Admin → Models → Advanced → “Native Function Calling”. The flag is stored on ModelProviderModel.tool_choice_enabled (default TRUE).
• ENV-configured models: set LLM_TOOL_CHOICE_ENABLED=false in your environment.
• Effect: when disabled, abilities["tool_choice"] returns False → the degradation chain starts from Level 2 (json_mode) or Level 3 (plain_text), skipping native_fc entirely. This eliminates the ~10s penalty per structured call for incompatible models.
• ReAct agent unaffected: tool_choice_enabled only controls forced tool selection in structured_llm_call. The ReAct engine uses tool_choice="auto" (model freely decides), which works with all models regardless of this setting.

​

Provider Capability Matrix

​

Table A: Protocol routing

• store=false keeps the conversation stateless upstream, and include=["reasoning.encrypted_content"] asks for the encrypted payload to be returned. Without the include, the reasoning items arrive empty and replay silently becomes a no-op.
• A replayed reasoning item must have its server-side id stripped. With store=false nothing is persisted upstream, so echoing the id back earns Item with id 'rs_...' not found. Items are not persisted when 'store' is set to false. The encrypted_content blob carries the state on its own, so dropping the id costs nothing (sanitize_reasoning_item).

​

Table B: Conflicts and workarounds

​

Table C: Thinking protocol

​

Relay/proxy gotchas

​

Recommended per-model configuration

# Disable native_fc globally (for thinking-model-only deployments)
LLM_TOOL_CHOICE_ENABLED=false

# Disable json_mode globally (for Bedrock relay deployments)
LLM_JSON_MODE_ENABLED=false

​

Reasoning effort and thinking configuration

1. Temperature is handled for you. On an anthropic/ route with thinking active, _build_request_kwargs pins temperature to 1.0, which is what Bedrock demands. Models that reject sampling parameters outright (Opus 4.7 and 4.8, Fable 5, Mythos 5) have temperature removed from the request entirely, thinking or not. Do not set LLM_TEMPERATURE=1 by hand for this.
2. GPT-5.x keeps tools and reasoning together where it can. FIM One probes the Responses bridge first for GPT-5.x, because that is the only surface where the two combine. An endpoint with no usable /v1/responses route falls back to chat completions, the verdict is cached per endpoint, and on that path a request carrying tools sends an explicit reasoning_effort of none. Omitting the field is not equivalent, since the server default is not none.

​

Defensive parsing for structured output

1. Single object instead of array. Some models return {"steps": {"id": "1", "task": "..."}} (a single step object) instead of {"steps": [{"id": "1", "task": "..."}]} (an array). The parser detects this by checking for id or task keys and wraps the object in a list.
2. Double-encoded JSON string. When structured output falls through to json_mode (which lacks schema enforcement), some providers return the steps value as a JSON string rather than a native array — e.g., {"steps": "[{\"id\": \"1\", ...}]"}. This string may also contain literal newlines (from the model’s formatting) that break standard json.loads. The parser uses extract_json_value() (which includes _repair_json_strings) to handle:
   • Literal newlines inside JSON string values
   • Invalid escape sequences (common with LaTeX or code content)
   • Other serialization quirks from compatibility layers
3. Missing steps wrapper. The model may return a single step as the top-level object without the steps wrapper key. The parser detects id and task at the root level and wraps accordingly.

​

Prompt caching (cross-provider)

​

Architecture

• PromptSection — a named fragment with either a static content: str or a dynamic content: Callable
• PromptRegistry — a memoized store (static sections render once, dynamic sections re-render per call)
• DYNAMIC_BOUNDARY — a sentinel marker the registry inserts between the last static section and the first dynamic one, so callers can split the rendered prompt at the cache breakpoint

• Static prefix (~95% of the prompt) — identity, core guidelines, tool descriptions
• Dynamic suffix — current datetime, per-request language directive, handoff context

​

Capability detection

​

Cross-provider coverage

​

Observability

"cache": {
  "read_tokens": 1067,
  "creation_tokens": 0
}

​

Multi-turn cache ROI

​

Reasoning replay policy (modelless correctness)

​

Four policies

• anthropic_thinking — Claude family (including anthropic/, bedrock/anthropic, vertex_ai/claude). Thinking blocks MUST be replayed with signature attached; Anthropic rejects subsequent turns if the signature is missing or altered.
• informational_only — models that emit CoT but do NOT expect replay: DeepSeek reasoning mode (deepseek-reasoner on V3.2, and the older deepseek-r1 and R1-Distill ids the fragment table still matches), Qwen QwQ, Gemini flash-thinking, OpenAI o1 / o3 / o4. Their documentation explicitly says “do not send reasoning_content back in message history”. Sending it anyway:
  • Violates the provider contract (may start rejecting in future versions)
  • Silently invalidates their automatic prefix cache — message bytes mutate on every turn, breaking the hash
• openai_responses — GPT-5.x, matched on the gpt-5 fragment. Its reasoning state is not text but a sequence of opaque items carrying encrypted payloads, and only /v1/responses has a slot for them. On that protocol the items are replayed verbatim, which is what keeps the model’s chain of thought alive across tool rounds. The readable summary is still dropped from outgoing requests, so on the chat-completions fallback this behaves exactly like informational_only. Checked before the informational fragments, whose generic reasoning entry would otherwise swallow proxy-tagged GPT-5 ids.
• unsupported — the catch-all: models with no reasoning capability (GPT-4o, Gemini 1.5, Mistral, Llama), and reasoning models whose id matches no fragment (GLM, MiniMax, Kimi, Doubao). No field should be replayed either way, so this policy puts the same bytes on the wire as informational_only. It is also the safe default for unknown model ids.

​

Enforcement

​

For users

• If you switch agents between Claude and DeepSeek in the same conversation, history is stored with thinking blocks intact; on the next turn, the outgoing message shape adapts per the current model.
• If you use a proxy / custom BaseLLM subclass, make sure its model id is recognizable (contains one of the fragments) or the default unsupported policy will apply — which is safe but means Claude behind an unusual proxy might lose thinking replay. Add the model-id fragment to _CACHE_CAPABLE_MODEL_FRAGMENTS (in core/prompt/caching.py) and/or the reasoning policy lookup.

​

Troubleshooting