LLM Provider Compatibility

​

Provider detection

1. Explicit provider (from DB ModelConfig.provider field) — highest priority. If the provider matches a known domain in the URL, no api_base is returned (LiteLLM routes natively). Otherwise, api_base is set to the relay URL.
2. Domain match against KNOWN_DOMAINS — official API endpoints are recognized by hostname.
3. URL path hint against PATH_PROVIDER_HINTS — common on relay platforms like UniAPI where /claude or /anthropic in the path indicates the upstream protocol.
4. Fallback — openai/ prefix (generic OpenAI-compatible).

​

tool_choice — the four modes

​

Where tool_choice is used

​

ReAct engine — tool_choice=“auto”

​

structured_llm_call — tool_choice=forced

​

The Bedrock prefill trap

{"role": "assistant", "content": "{"}

ValidationException: This model does not support assistant message prefill.
The conversation must end with a user message.

1. The model is resolved with the anthropic/ prefix (via domain match or URL path hint).
2. response_format={"type":"json_object"} is passed (the json_mode code path in structured_llm_call).
3. The actual backend is AWS Bedrock (which rejects prefill).

​

The fix: json_mode_enabled

• DB-configured models: toggle in Admin → Models → Advanced settings. The flag is stored on ModelProviderModel.json_mode_enabled (default TRUE).
• ENV-configured models: set LLM_JSON_MODE_ENABLED=false in your environment.
• Effect: when disabled, abilities["json_mode"] returns False → response_format is never passed → no prefill → Bedrock works. The degradation chain becomes native_fc → plain_text, skipping the doomed json_mode call entirely.
• No quality loss: the model still returns valid JSON because the system prompt instructs it to. The plain_text level uses extract_json() to parse JSON from free-form content, which works reliably with modern models.

​

Thinking models + forced tool_choice

tool_choice 'specified' is incompatible with thinking enabled

​

The fix: tool_choice_enabled

• DB-configured models: toggle in Admin → Models → Advanced → “Native Function Calling”. The flag is stored on ModelProviderModel.tool_choice_enabled (default TRUE).
• ENV-configured models: set LLM_TOOL_CHOICE_ENABLED=false in your environment.
• Effect: when disabled, abilities["tool_choice"] returns False → the degradation chain starts from Level 2 (json_mode) or Level 3 (plain_text), skipping native_fc entirely. This eliminates the ~10s penalty per structured call for incompatible models.
• ReAct agent unaffected: tool_choice_enabled only controls forced tool selection in structured_llm_call. The ReAct engine uses tool_choice="auto" (model freely decides), which works with all models regardless of this setting.

​

Provider Capability Matrix

​

Table A: Protocol routing

• store=false keeps the conversation stateless upstream, and include=["reasoning.encrypted_content"] asks for the encrypted payload to be returned. Without the include, the reasoning items arrive empty and replay silently becomes a no-op.
• A replayed reasoning item must have its server-side id stripped. With store=false nothing is persisted upstream, so echoing the id back earns Item with id 'rs_...' not found. Items are not persisted when 'store' is set to false. The encrypted_content blob carries the state on its own, so dropping the id costs nothing (sanitize_reasoning_item).

​

Table B: Conflicts and workarounds

​

Table C: Thinking protocol

​

Relay/proxy gotchas

​

Recommended per-model configuration

# Disable native_fc globally (for thinking-model-only deployments)
LLM_TOOL_CHOICE_ENABLED=false

# Disable json_mode globally (for Bedrock relay deployments)
LLM_JSON_MODE_ENABLED=false

​

Reasoning effort and thinking configuration

1. Temperature is handled for you. On an anthropic/ route with thinking active, _build_request_kwargs pins temperature to 1.0, which is what Bedrock demands. Models that reject sampling parameters outright (Opus 4.7 and 4.8, Fable 5, Mythos 5) have temperature removed from the request entirely, thinking or not. Do not set LLM_TEMPERATURE=1 by hand for this.
2. GPT-5.x keeps tools and reasoning together where it can. FIM One probes the Responses bridge first for GPT-5.x, because that is the only surface where the two combine. An endpoint with no usable /v1/responses route falls back to chat completions, the verdict is cached per endpoint, and on that path a request carrying tools sends an explicit reasoning_effort of none. Omitting the field is not equivalent, since the server default is not none.

​

Defensive parsing for structured output

1. Single object instead of array. Some models return {"steps": {"id": "1", "task": "..."}} (a single step object) instead of {"steps": [{"id": "1", "task": "..."}]} (an array). The parser detects this by checking for id or task keys and wraps the object in a list.
2. Double-encoded JSON string. When structured output falls through to json_mode (which lacks schema enforcement), some providers return the steps value as a JSON string rather than a native array — e.g., {"steps": "[{\"id\": \"1\", ...}]"}. This string may also contain literal newlines (from the model’s formatting) that break standard json.loads. The parser uses extract_json_value() (which includes _repair_json_strings) to handle:
   • Literal newlines inside JSON string values
   • Invalid escape sequences (common with LaTeX or code content)
   • Other serialization quirks from compatibility layers
3. Missing steps wrapper. The model may return a single step as the top-level object without the steps wrapper key. The parser detects id and task at the root level and wraps accordingly.

​

Prompt caching (cross-provider)

​

Architecture

• PromptSection — a named fragment with either a static content: str or a dynamic content: Callable
• PromptRegistry — a memoized store (static sections render once, dynamic sections re-render per call)
• DYNAMIC_BOUNDARY — a sentinel marker the registry inserts between the last static section and the first dynamic one, so callers can split the rendered prompt at the cache breakpoint

• Static prefix (~95% of the prompt) — identity, core guidelines, tool descriptions
• Dynamic suffix — current datetime, per-request language directive, handoff context

​

Capability detection

​

Cross-provider coverage

​

Observability

"cache": {
  "read_tokens": 1067,
  "creation_tokens": 0
}

​

Multi-turn cache ROI

​

Reasoning replay policy (modelless correctness)

​

Four policies

• anthropic_thinking — Claude family (including anthropic/, bedrock/anthropic, vertex_ai/claude). Thinking blocks MUST be replayed with signature attached; Anthropic rejects subsequent turns if the signature is missing or altered.
• informational_only — models that emit CoT but do NOT expect replay: DeepSeek reasoning mode (deepseek-reasoner on V3.2, and the older deepseek-r1 and R1-Distill ids the fragment table still matches), Qwen QwQ, Gemini flash-thinking, OpenAI o1 / o3 / o4. Their documentation explicitly says “do not send reasoning_content back in message history”. Sending it anyway:
  • Violates the provider contract (may start rejecting in future versions)
  • Silently invalidates their automatic prefix cache — message bytes mutate on every turn, breaking the hash
• openai_responses — GPT-5.x, matched on the gpt-5 fragment. Its reasoning state is not text but a sequence of opaque items carrying encrypted payloads, and only /v1/responses has a slot for them. On that protocol the items are replayed verbatim, which is what keeps the model’s chain of thought alive across tool rounds. The readable summary is still dropped from outgoing requests, so on the chat-completions fallback this behaves exactly like informational_only. Checked before the informational fragments, whose generic reasoning entry would otherwise swallow proxy-tagged GPT-5 ids.
• unsupported — the catch-all: models with no reasoning capability (GPT-4o, Gemini 1.5, Mistral, Llama), and reasoning models whose id matches no fragment (GLM, MiniMax, Kimi, Doubao). No field should be replayed either way, so this policy puts the same bytes on the wire as informational_only. It is also the safe default for unknown model ids.

​

Enforcement

​

For users

• If you switch agents between Claude and DeepSeek in the same conversation, history is stored with thinking blocks intact; on the next turn, the outgoing message shape adapts per the current model.
• If you use a proxy / custom BaseLLM subclass, make sure its model id is recognizable (contains one of the fragments) or the default unsupported policy will apply — which is safe but means Claude behind an unusual proxy might lose thinking replay. Add the model-id fragment to _CACHE_CAPABLE_MODEL_FRAGMENTS (in core/prompt/caching.py) and/or the reasoning policy lookup.

​

Troubleshooting