Skip to content

LLM Providers Deep Dive ​

GhostAI uses three independent LLM layers. This page explains every provider option, how they connect, and how to configure them.

Quick Start (Minimum Viable Config) ​

Set your API keys and go. Routing is automatic -- every model reaches the right provider without any extra configuration.

bash
# Required: at least one provider key
OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...
GEMINI_API_KEY=...

# Recommended: OpenRouter as a universal fallback
OPENROUTER_API_KEY=sk-or-...

# Optional: additional direct providers
MISTRAL_API_KEY=...
XAI_API_KEY=...
DEEPSEEK_API_KEY=...
GROQ_API_KEY=...

# Routing is automatic (LLM_PROVIDER defaults to "auto")
# LLM_PROVIDER=auto

With this setup:

  • gpt-4o routes directly to OpenAI
  • claude-3.5-sonnet routes directly to Anthropic
  • gemini-2.5-flash routes directly to Gemini
  • Any model whose direct key is missing falls back to OpenRouter
  • No LLM_PROVIDER change needed when switching models

Architecture Overview ​

GhostAI uses a multi-source architecture: all configured providers are active simultaneously. Each provider with a configured API key contributes models to the unified model picker. Users select a model, and the system routes the request to the correct provider automatically.


Layer 1: Chat LLM (Multi-Source) ​

The main conversational AI that powers chat, tool calling, streaming responses, and model comparison. All providers with configured API keys are active at once; the model picker shows a unified list from every available source.

Routing Mode (LLM_PROVIDER) ​

LLM_PROVIDER controls how requests are routed to providers. The default is auto, which infers the correct provider from the model name and sends the request directly -- no manual routing needed.

bash
LLM_PROVIDER=auto        # (default) Direct-first: infer provider from model, fallback to OpenRouter
LLM_PROVIDER=openai      # Legacy: force all unrouted requests through OpenAI
LLM_PROVIDER=openrouter  # Legacy: force all unrouted requests through OpenRouter
LLM_PROVIDER=ollama      # Legacy: force all unrouted requests through Ollama

How auto routing works ​

When LLM_PROVIDER=auto (the default), the backend resolves each request through this fallback chain:

  1. Infer provider from model ID -- gpt-4o maps to OpenAI, claude-3.5-sonnet maps to Anthropic, gemini-2.5-flash maps to Gemini, etc. OpenRouter-prefixed IDs like anthropic/claude-3.5-sonnet are also recognized.
  2. Try the direct provider -- if the inferred provider's API key is configured, route directly to that provider's API.
  3. Fall back to OpenRouter -- if the direct key is missing but OPENROUTER_API_KEY is set, route through OpenRouter with the correct model prefix.
  4. Fall back to Ollama -- for unprefixed model names when OLLAMA_BASE_URL is configured.
  5. Ultimate fallback -- use the primary client (highest-priority configured provider).

This means you can set all your API keys once and every model automatically goes to the right place. No need to change LLM_PROVIDER when switching between models.

Auto-routing decision tree ​

Backward compatibility

Existing deployments with LLM_PROVIDER=openai continue to work exactly as before. The auto behavior only activates when explicitly set or when LLM_PROVIDER is omitted (new default).


Direct Provider Configuration ​

Each provider is enabled by setting its API key. Providers without a key are omitted from the model picker and cannot be used.

ProviderEnv VarBase URLNotes
OpenAIOPENAI_API_KEYapi.openai.comDefault primary provider
AnthropicANTHROPIC_API_KEYapi.anthropic.comClaude models
GeminiGEMINI_API_KEYgenerativelanguage.googleapis.comGoogle Gemini
MistralMISTRAL_API_KEYapi.mistral.aiMistral models
xAIXAI_API_KEYapi.x.aiGrok models
DeepSeekDEEPSEEK_API_KEYapi.deepseek.comDeepSeek models
GroqGROQ_API_KEYapi.groq.comFast inference
TogetherTOGETHER_API_KEYapi.together.xyzOpen models
PerplexityPERPLEXITY_API_KEYapi.perplexity.aiSearch-augmented
CohereCOHERE_API_KEYapi.cohere.comUses CohereAdapter (non-OpenAI API)
OpenRouterOPENROUTER_API_KEYopenrouter.aiAggregator: 200+ models
AWS Bedrock(BYOK only)bedrock-runtime.{region}.amazonaws.comPer-request key from Provider Keys (region + optional credentials). No env key.
Microsoft FoundryFOUNDRY_API_KEY or Entra ID{resource}.services.ai.azure.comAzure AI Foundry; also configurable as BYOK via Provider Keys. See Foundry guide.
Ollama(no key)OLLAMA_BASE_URLLocal, no API key
vLLM(no key)VLLM_BASE_URLLocal GPU server

Example: Enable Multiple Direct Providers ​

bash
# All of these providers will appear in the model picker
OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...
MISTRAL_API_KEY=...
DEEPSEEK_API_KEY=...
GROQ_API_KEY=...

Route Selection Configuration ​

When a user selects a model, GhostAI resolves the route (which provider to call). Models can be available through multiple routes (e.g., Claude via Anthropic direct or via OpenRouter).

How Routes Work ​

  1. Model picker: Queries each configured provider in parallel. Models are deduplicated by canonical ID (e.g., claude-3.5-sonnet).
  2. Route metadata: Each model carries a list of routes with source (e.g., anthropic-direct, openrouter) and model_id.
  3. Explicit route: If the user (or frontend) specifies a route, that route is used directly.
  4. Auto-inference (default): When no route is specified, the backend infers the correct provider from the model ID (e.g., claude-* -> Anthropic, gpt-* -> OpenAI, gemini-* -> Gemini). If that provider's key is configured, the request goes direct. Otherwise it falls back to OpenRouter, then Ollama, then the primary client.
  5. OpenAI-compatible and Anthropic-compatible API endpoints (/v1/chat/completions, /v1/messages) also use auto-inference, so third-party tools like Continue, OpenWebUI, and LangChain benefit from direct routing automatically.

Route Sources ​

SourceProviderRequires
openai-directOpenAIOPENAI_API_KEY
anthropic-directAnthropicANTHROPIC_API_KEY
gemini-directGeminiGEMINI_API_KEY
mistral-directMistralMISTRAL_API_KEY
xai-directxAIXAI_API_KEY
deepseek-directDeepSeekDEEPSEEK_API_KEY
groq-directGroqGROQ_API_KEY
together-directTogetherTOGETHER_API_KEY
perplexity-directPerplexityPERPLEXITY_API_KEY
cohere-directCohereCOHERE_API_KEY
bedrock-directAWS BedrockProvider Keys (BYOK): region + optional access key/secret
foundry-directMicrosoft FoundryFOUNDRY_API_KEY / Entra ID, or Provider Keys (BYOK)
openrouterOpenRouterOPENROUTER_API_KEY
ollamaOllamaOLLAMA_BASE_URL
vllmvLLMVLLM_BASE_URL

Provider Details ​

OpenAI ​

Standard OpenAI API. GPT-4o, GPT-4o-mini, o1, and other models.

bash
OPENAI_API_KEY=sk-...
DEFAULT_MODEL=gpt-4o-mini
SettingDefaultDescription
OPENAI_API_KEY(required)Your OpenAI API key
DEFAULT_MODELgpt-4o-miniDefault chat model (when OpenAI is primary)
AVAILABLE_MODELSgpt-4o,gpt-4o-mini,...Models shown when OpenAI is primary
LLM_TIMEOUT120Request timeout (seconds)
LLM_MAX_CONCURRENT20Max parallel API calls

Anthropic ​

Claude models via direct API.

bash
ANTHROPIC_API_KEY=sk-ant-...

Requires anthropic-version header; the client sets this automatically.

Gemini (Google) ​

Google Gemini models via Generative Language API (OpenAI-compatible endpoint).

bash
GEMINI_API_KEY=...

Base URL: https://generativelanguage.googleapis.com/v1beta/openai/

Mistral ​

Mistral AI models (Mistral Large, Mistral Small, etc.).

bash
MISTRAL_API_KEY=...

xAI ​

xAI Grok models.

bash
XAI_API_KEY=...

DeepSeek ​

DeepSeek models (DeepSeek V3, R1, etc.).

bash
DEEPSEEK_API_KEY=...

Groq ​

Groq's fast inference for Llama and other models.

bash
GROQ_API_KEY=...

Together ​

Together AI for open models (Llama, Qwen, etc.).

bash
TOGETHER_API_KEY=...

Perplexity ​

Perplexity's search-augmented models.

bash
PERPLEXITY_API_KEY=...

Cohere ​

Cohere models. Uses a custom CohereAdapter because Cohere's API differs from the OpenAI chat completions format. The adapter translates requests/responses so Cohere appears as a standard provider.

bash
COHERE_API_KEY=...

Microsoft Foundry (Azure AI) ​

Azure AI Foundry models via an OpenAI-compatible endpoint. Supports API key and Microsoft Entra ID authentication. Also available as a BYOK provider through Provider Keys.

bash
FOUNDRY_ENDPOINT=https://my-foundry.services.ai.azure.com
FOUNDRY_API_KEY=your-foundry-api-key
FOUNDRY_AUTH_MODE=api_key   # or: entra_id
SettingDefaultDescription
FOUNDRY_ENDPOINT(required)Base URL of your Azure AI Foundry resource
FOUNDRY_API_KEY(required for api_key mode)Azure AI Foundry API key
FOUNDRY_AUTH_MODEapi_keyAuthentication mode: api_key or entra_id
FOUNDRY_TENANT_ID(optional)Azure AD tenant ID (for Entra ID)

Models use the Azure deployment name directly (e.g., gpt-4.1, gpt-5). The backend dynamically discovers available deployments via GET {endpoint}/openai/v1/models and falls back to a static model list.

Foundry also supports PII referee, embeddings, image generation, TTS, STT, and video generation. See the Microsoft Foundry guide for full configuration.

Entra ID (Managed Identity)

Set FOUNDRY_AUTH_MODE=entra_id and leave FOUNDRY_API_KEY empty. The backend uses DefaultAzureCredential from the Azure Identity SDK, which automatically picks up Managed Identity on Azure, Azure CLI credentials locally, or environment-based service principal credentials.

OpenRouter (Aggregator) ​

Access 200+ models from every major provider through a single API key. Includes Claude, Gemini, Llama, DeepSeek, Mistral, and more. Useful when you want one key for many providers.

bash
OPENROUTER_API_KEY=sk-or-...
OPENROUTER_BASE_URL=https://openrouter.ai/api/v1
OPENROUTER_DEFAULT_MODEL=openai/gpt-4o-mini
SettingDefaultDescription
OPENROUTER_API_KEY(required)Your OpenRouter API key
OPENROUTER_BASE_URLhttps://openrouter.ai/api/v1API endpoint
OPENROUTER_DEFAULT_MODELopenai/gpt-4o-miniDefault model (vendor/model format)
OPENROUTER_AVAILABLE_MODELS(see config)Models shown when OpenRouter is primary

Ollama (Local) ​

Run open-source models locally. No API keys, no cloud calls, no cost. Uses Ollama's OpenAI-compatible API.

bash
OLLAMA_BASE_URL=http://localhost:11434
OLLAMA_DEFAULT_MODEL=llama3.2
SettingDefaultDescription
OLLAMA_BASE_URLhttp://localhost:11434Ollama daemon URL
OLLAMA_DEFAULT_MODELllama3.2Default model
OLLAMA_AVAILABLE_MODELSllama3.2,llama3.1,...Models shown in UI picker

Setup:

bash
# macOS
brew install ollama
ollama serve &        # Start daemon (runs on port 11434)
ollama pull llama3.2  # Download a model (~2GB)

# Linux
curl -fsSL https://ollama.com/install.sh | sh
ollama serve &
ollama pull llama3.2

Auto-Discovery

When Ollama is configured, the model picker automatically queries ollama list and shows all installed models -- no need to manually configure OLLAMA_AVAILABLE_MODELS.

Rich Metadata via /api/show ​

For each installed model, GhostAI calls Ollama's /api/show endpoint to fetch real metadata instead of relying on name-based heuristics. The enriched fields include:

FieldSourceExample
Context lengthmodel_info.*.context_length or parameters.num_ctx131072
Parameter countdetails.parameter_size7B, 70B
CapabilitiesArchitecture inspection (e.g., clip projector → vision)vision, tools
Quantizationdetails.quantization_levelQ4_K_M

Brand and Country Mapping ​

Ollama model descriptions display the correct vendor brand name and country flag based on the model family:

Family PrefixBrandFlag
llama, codellamaMetaUS
deepseekDeepSeekCN
mistral, mixtralMistral AIFR
gemmaGoogleUS
qwenAlibaba CloudCN
phiMicrosoftUS
command-rCohereCA

Model Management Endpoints ​

Platform admins can pull and delete Ollama models through the API without CLI access:

EndpointMethodDescription
/api/admin/ollama/modelsGETList models with enriched metadata
/api/admin/ollama/pullPOSTPull a new model by tag (e.g., llama3.2)
/api/admin/ollama/modelsDELETEDelete an installed model by name

These endpoints require platform admin role. The Model Browser UI exposes pull and delete actions for admin users.

vLLM (Local GPU Server) ​

Connect to a vLLM serving endpoint (OpenAI-compatible). For GPU-rich environments with multiple concurrent users.

bash
VLLM_BASE_URL=http://localhost:8000

The client appends /v1 for the OpenAI-compatible endpoint. No API key required.

llama-cpp (PII Only) ​

In-process GGUF inference. Used by the PII Referee layer only, not the chat LLM. See Layer 2 below.


Layer 2: PII Referee (PII_REFEREE_RUNTIME) ​

A separate local LLM that validates PII detections. When Presidio and the NER model disagree about whether text is sensitive, the referee casts the deciding vote. This runs independently of the chat LLM.

Platform Auto-Detection ​

When PII_REFEREE_MODEL_NAME=auto, GhostAI detects your hardware and picks the best model:

PlatformDetectionRecommended ModelRAM Needed
Apple Silicon (M1/M2/M3/M4)Darwin + arm64phi-3.5-mini (3.8B)2.3 GB
NVIDIA GPU (8GB+ VRAM)nvidia-smillama-3.1-8b (8B)4.5 GB
NVIDIA GPU (<8GB VRAM)nvidia-smiphi-3.5-mini (3.8B)2.3 GB
CPU with 4GB+ RAMFallbackphi-3.5-mini (3.8B)2.3 GB
CPU with <4GB RAMFallbackdistil-pii-1b (1B)0.7 GB

Model Registry ​

All models are US-sourced with verified GGUF downloads:

KeyModelVendorParamsVRAM/RAMBest For
llama-3.1-8bLlama 3.1 8B InstructMeta8B4.5 GBGPU systems (best accuracy)
phi-3.5-miniPhi-3.5 Mini InstructMicrosoft3.8B2.3 GBApple Silicon, general use
llama-3.2-3bLlama 3.2 3B InstructMeta3B2.0 GBBalanced CPU/GPU
distil-pii-1bDistil-PII Llama 1BMeta (base)1B0.7 GBLow-resource CPU
gemma-3-1bGemma 3 1B ITGoogle1B0.7 GBLightweight CPU

Runtime: llama-cpp (In-Process) ​

Loads a GGUF model directly into the Python process. No external daemon. Best for single-user deployments.

bash
PII_REFEREE_RUNTIME=llama-cpp
PII_REFEREE_MODEL_NAME=auto           # or: phi-3.5-mini, llama-3.1-8b, etc.
PII_REFEREE_MODELS_DIR=models         # GGUF download directory
PII_REFEREE_MODEL_CTX=4096            # Context window (tokens)
PII_REFEREE_MODEL_GPU_LAYERS=-1       # -1 = auto (Metal/CUDA), 0 = CPU only
PII_REFEREE_MODEL_THREADS=0           # 0 = auto-detect CPU cores
PII_REFEREE_WARMUP=true               # Pre-load model at startup

Pros: Zero external dependencies, single process, lowest latency. Cons: Blocks on inference (single-threaded), higher memory footprint in the app process.

Apple Silicon

On macOS with Apple Silicon, PII_REFEREE_MODEL_GPU_LAYERS=-1 automatically offloads all layers to Metal. This gives you GPU-speed inference with no configuration.

Runtime: Ollama (Daemon) ​

Calls a running Ollama daemon. Supports concurrency and is shared with the chat LLM if both use Ollama.

bash
PII_REFEREE_RUNTIME=ollama
PII_REFEREE_OLLAMA_URL=               # Defaults to OLLAMA_BASE_URL
PII_REFEREE_OLLAMA_MODEL=llama3.1:8b  # Ollama model tag
PII_REFEREE_MAX_CONCURRENT=4          # Parallel referee calls

Pros: Shared model cache with chat, concurrent inference, model hot-swap. Cons: Requires Ollama daemon running, extra process to manage.

Circuit Breaker

The Ollama backend includes a circuit breaker: after 5 consecutive failures, it stops calling Ollama for 30 seconds, then retries. This prevents cascade failures if Ollama crashes or becomes unresponsive.

Runtime: vLLM (GPU Server) ​

Connects to a vLLM serving endpoint (OpenAI-compatible). Best for GPU-rich environments with multiple concurrent users.

bash
PII_REFEREE_RUNTIME=vllm
PII_REFEREE_VLLM_URL=http://localhost:8000/v1
PII_REFEREE_VLLM_MODEL=meta-llama/Llama-3.1-8B-Instruct
PII_REFEREE_MAX_CONCURRENT=8

Pros: Highest throughput, continuous batching, tensor parallelism. Cons: Requires dedicated GPU server, more complex setup.

Runtime: Foundry (Azure Cloud) ​

Routes referee calls to a Foundry-hosted model. Useful when you want cloud-grade accuracy without running local models.

bash
PII_REFEREE_RUNTIME=foundry
PII_REFEREE_FOUNDRY_MODEL=gpt-4.1-mini

The referee reuses FOUNDRY_ENDPOINT and FOUNDRY_API_KEY / Entra ID credentials from the main Foundry configuration. Includes a circuit breaker (stops calling after 5 consecutive failures, retries after 30s). Per-tenant BYOK key resolution is supported (Cloud > Tenant > User).

Dedicated Prometheus metrics: ghostai_referee_foundry_tokens_total, ghostai_referee_foundry_cost_dollars.

Cloud Referee

Unlike local runtimes, Foundry referee calls send PII-candidate text to Azure. Audit logging for Foundry referee can be controlled with PII_REFEREE_AUDIT_FOUNDRY=true|false.


Layer 3: Embeddings ​

Embeddings for RAG (file search) and memory (conversation recall) default to local SentenceTransformers. Cloud backends (Bedrock, Foundry) are also available.

Local (default) ​

bash
MEMORY_EMBEDDING_BACKEND=local            # Default
MEMORY_EMBEDDING_MODEL=all-MiniLM-L6-v2   # 384-dim embeddings
MEMORY_EMBEDDING_DIM=384                   # Must match model output
RAG_EMBEDDING_DEVICE=auto                  # auto → MPS → CUDA → CPU

Foundry ​

bash
MEMORY_EMBEDDING_BACKEND=foundry
FOUNDRY_EMBEDDING_MODEL=text-embedding-3-large
FOUNDRY_EMBEDDING_ENDPOINT=               # Defaults to FOUNDRY_ENDPOINT

Uses the same FOUNDRY_ENDPOINT and authentication settings as chat. Embedding dimensions default to the model's native output (3072 for text-embedding-3-large). Also available for the content library via CONTENT_LIBRARY_EMBEDDING_BACKEND=foundry.


Common Deployment Combinations ​

Use auto-routing with multiple direct providers for chat; keep PII verification local.

bash
# Auto-routing sends each model to its native provider
# LLM_PROVIDER=auto  (default -- no need to set)
OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...
GEMINI_API_KEY=...
MISTRAL_API_KEY=...
DEEPSEEK_API_KEY=...
OPENROUTER_API_KEY=sk-or-...      # fallback for any model without a direct key
PII_AI_MODEL=local
PII_REFEREE_RUNTIME=llama-cpp
PII_REFEREE_MODEL_GPU_LAYERS=-1
PII_REFEREE_WARMUP=true

Data flow: gpt-4o goes directly to OpenAI, claude-3.5-sonnet goes directly to Anthropic, gemini-2.5-flash goes directly to Gemini, etc. Models from providers without a direct key fall back to OpenRouter. PII runs locally.

OpenRouter + Direct Anthropic + Ollama ​

OpenRouter for broad model access, direct Anthropic for best Claude latency, Ollama for local models.

bash
# LLM_PROVIDER=auto  (default)
OPENROUTER_API_KEY=sk-or-...
ANTHROPIC_API_KEY=sk-ant-...
OLLAMA_BASE_URL=http://localhost:11434
PII_AI_MODEL=local
PII_REFEREE_RUNTIME=ollama
PII_REFEREE_OLLAMA_MODEL=llama3.1:8b

Data flow: Claude models route directly to Anthropic; other cloud models fall back to OpenRouter; local models use Ollama. PII runs locally through Ollama.

Fully Cloud (Simplest) ​

bash
OPENAI_API_KEY=sk-...
PII_AI_MODEL=dedicated             # Use cloud LLM for PII too
PII_AI_DEDICATED_MODEL=gpt-4o-mini

Data flow: Chat + PII both go through OpenAI. Simplest setup, but PII text leaves your network.

Hybrid (Cloud Chat + Local PII) ​

bash
OPENAI_API_KEY=sk-...
PII_AI_MODEL=local
PII_REFEREE_RUNTIME=llama-cpp
PII_REFEREE_MODEL_GPU_LAYERS=-1
PII_REFEREE_WARMUP=true

Data flow: Chat goes through OpenAI, but PII verification runs locally. Sensitive text for PII detection never leaves the machine.

Fully Local (Air-Gapped) ​

bash
LLM_PROVIDER=ollama
OLLAMA_BASE_URL=http://localhost:11434
OLLAMA_DEFAULT_MODEL=llama3.2
PII_AI_MODEL=local
PII_REFEREE_RUNTIME=ollama
PII_REFEREE_OLLAMA_MODEL=llama3.1:8b

Data flow: Everything runs locally. Zero network calls. Works offline / air-gapped.

Multi-Provider via OpenRouter (Legacy) ​

bash
LLM_PROVIDER=openrouter             # explicit -- overrides auto-routing
OPENROUTER_API_KEY=sk-or-...
OPENROUTER_DEFAULT_MODEL=anthropic/claude-3.5-sonnet
PII_AI_MODEL=local
PII_REFEREE_RUNTIME=ollama

Data flow: All chat routes through OpenRouter. PII runs locally through Ollama. Consider switching to LLM_PROVIDER=auto and adding direct provider keys for better latency and lower per-token cost.


Threat Analysis LLM Requirements ​

Security threat analysis requires LLM access to classify detected findings (malicious, suspicious, or benign). The default model is gpt-4o-mini, configurable per organization in Admin Dashboard → Threat Analysis.

In air-gapped deployments where no LLM is available, threat analysis shows ai_status=unavailable and falls back to the original risk levels from the detectors. Encrypted archives and zip bombs remain high-risk regardless of AI availability.

SOC 2 Type II Certified | BAA Available | HIPAA-Ready