Appearance
LLM Providers Deep Dive
GhostAI uses three independent LLM layers. This page explains every provider option, how they connect, and how to configure them.
Quick Start (Minimum Viable Config)
Set your API keys and go. Routing is automatic -- every model reaches the right provider without any extra configuration.
bash
# Required: at least one provider key
OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...
GEMINI_API_KEY=...
# Recommended: OpenRouter as a universal fallback
OPENROUTER_API_KEY=sk-or-...
# Optional: additional direct providers
MISTRAL_API_KEY=...
XAI_API_KEY=...
DEEPSEEK_API_KEY=...
GROQ_API_KEY=...
# Routing is automatic (LLM_PROVIDER defaults to "auto")
# LLM_PROVIDER=autoWith this setup:
gpt-4oroutes directly to OpenAIclaude-3.5-sonnetroutes directly to Anthropicgemini-2.5-flashroutes directly to Gemini- Any model whose direct key is missing falls back to OpenRouter
- No
LLM_PROVIDERchange needed when switching models
Architecture Overview
GhostAI uses a multi-source architecture: all configured providers are active simultaneously. Each provider with a configured API key contributes models to the unified model picker. Users select a model, and the system routes the request to the correct provider automatically.
Layer 1: Chat LLM (Multi-Source)
The main conversational AI that powers chat, tool calling, streaming responses, and model comparison. All providers with configured API keys are active at once; the model picker shows a unified list from every available source.
Routing Mode (LLM_PROVIDER)
LLM_PROVIDER controls how requests are routed to providers. The default is auto, which infers the correct provider from the model name and sends the request directly -- no manual routing needed.
bash
LLM_PROVIDER=auto # (default) Direct-first: infer provider from model, fallback to OpenRouter
LLM_PROVIDER=openai # Legacy: force all unrouted requests through OpenAI
LLM_PROVIDER=openrouter # Legacy: force all unrouted requests through OpenRouter
LLM_PROVIDER=ollama # Legacy: force all unrouted requests through OllamaHow auto routing works
When LLM_PROVIDER=auto (the default), the backend resolves each request through this fallback chain:
- Infer provider from model ID --
gpt-4omaps to OpenAI,claude-3.5-sonnetmaps to Anthropic,gemini-2.5-flashmaps to Gemini, etc. OpenRouter-prefixed IDs likeanthropic/claude-3.5-sonnetare also recognized. - Try the direct provider -- if the inferred provider's API key is configured, route directly to that provider's API.
- Fall back to OpenRouter -- if the direct key is missing but
OPENROUTER_API_KEYis set, route through OpenRouter with the correct model prefix. - Fall back to Ollama -- for unprefixed model names when
OLLAMA_BASE_URLis configured. - Ultimate fallback -- use the primary client (highest-priority configured provider).
This means you can set all your API keys once and every model automatically goes to the right place. No need to change LLM_PROVIDER when switching between models.
Auto-routing decision tree
Backward compatibility
Existing deployments with LLM_PROVIDER=openai continue to work exactly as before. The auto behavior only activates when explicitly set or when LLM_PROVIDER is omitted (new default).
Direct Provider Configuration
Each provider is enabled by setting its API key. Providers without a key are omitted from the model picker and cannot be used.
| Provider | Env Var | Base URL | Notes |
|---|---|---|---|
| OpenAI | OPENAI_API_KEY | api.openai.com | Default primary provider |
| Anthropic | ANTHROPIC_API_KEY | api.anthropic.com | Claude models |
| Gemini | GEMINI_API_KEY | generativelanguage.googleapis.com | Google Gemini |
| Mistral | MISTRAL_API_KEY | api.mistral.ai | Mistral models |
| xAI | XAI_API_KEY | api.x.ai | Grok models |
| DeepSeek | DEEPSEEK_API_KEY | api.deepseek.com | DeepSeek models |
| Groq | GROQ_API_KEY | api.groq.com | Fast inference |
| Together | TOGETHER_API_KEY | api.together.xyz | Open models |
| Perplexity | PERPLEXITY_API_KEY | api.perplexity.ai | Search-augmented |
| Cohere | COHERE_API_KEY | api.cohere.com | Uses CohereAdapter (non-OpenAI API) |
| OpenRouter | OPENROUTER_API_KEY | openrouter.ai | Aggregator: 200+ models |
| AWS Bedrock | (BYOK only) | bedrock-runtime.{region}.amazonaws.com | Per-request key from Provider Keys (region + optional credentials). No env key. |
| Microsoft Foundry | FOUNDRY_API_KEY or Entra ID | {resource}.services.ai.azure.com | Azure AI Foundry; also configurable as BYOK via Provider Keys. See Foundry guide. |
| Ollama | (no key) | OLLAMA_BASE_URL | Local, no API key |
| vLLM | (no key) | VLLM_BASE_URL | Local GPU server |
Example: Enable Multiple Direct Providers
bash
# All of these providers will appear in the model picker
OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...
MISTRAL_API_KEY=...
DEEPSEEK_API_KEY=...
GROQ_API_KEY=...Route Selection Configuration
When a user selects a model, GhostAI resolves the route (which provider to call). Models can be available through multiple routes (e.g., Claude via Anthropic direct or via OpenRouter).
How Routes Work
- Model picker: Queries each configured provider in parallel. Models are deduplicated by canonical ID (e.g.,
claude-3.5-sonnet). - Route metadata: Each model carries a list of
routeswithsource(e.g.,anthropic-direct,openrouter) andmodel_id. - Explicit route: If the user (or frontend) specifies a route, that route is used directly.
- Auto-inference (default): When no route is specified, the backend infers the correct provider from the model ID (e.g.,
claude-*-> Anthropic,gpt-*-> OpenAI,gemini-*-> Gemini). If that provider's key is configured, the request goes direct. Otherwise it falls back to OpenRouter, then Ollama, then the primary client. - OpenAI-compatible and Anthropic-compatible API endpoints (
/v1/chat/completions,/v1/messages) also use auto-inference, so third-party tools like Continue, OpenWebUI, and LangChain benefit from direct routing automatically.
Route Sources
| Source | Provider | Requires |
|---|---|---|
openai-direct | OpenAI | OPENAI_API_KEY |
anthropic-direct | Anthropic | ANTHROPIC_API_KEY |
gemini-direct | Gemini | GEMINI_API_KEY |
mistral-direct | Mistral | MISTRAL_API_KEY |
xai-direct | xAI | XAI_API_KEY |
deepseek-direct | DeepSeek | DEEPSEEK_API_KEY |
groq-direct | Groq | GROQ_API_KEY |
together-direct | Together | TOGETHER_API_KEY |
perplexity-direct | Perplexity | PERPLEXITY_API_KEY |
cohere-direct | Cohere | COHERE_API_KEY |
bedrock-direct | AWS Bedrock | Provider Keys (BYOK): region + optional access key/secret |
foundry-direct | Microsoft Foundry | FOUNDRY_API_KEY / Entra ID, or Provider Keys (BYOK) |
openrouter | OpenRouter | OPENROUTER_API_KEY |
ollama | Ollama | OLLAMA_BASE_URL |
vllm | vLLM | VLLM_BASE_URL |
Provider Details
OpenAI
Standard OpenAI API. GPT-4o, GPT-4o-mini, o1, and other models.
bash
OPENAI_API_KEY=sk-...
DEFAULT_MODEL=gpt-4o-mini| Setting | Default | Description |
|---|---|---|
OPENAI_API_KEY | (required) | Your OpenAI API key |
DEFAULT_MODEL | gpt-4o-mini | Default chat model (when OpenAI is primary) |
AVAILABLE_MODELS | gpt-4o,gpt-4o-mini,... | Models shown when OpenAI is primary |
LLM_TIMEOUT | 120 | Request timeout (seconds) |
LLM_MAX_CONCURRENT | 20 | Max parallel API calls |
Anthropic
Claude models via direct API.
bash
ANTHROPIC_API_KEY=sk-ant-...Requires anthropic-version header; the client sets this automatically.
Gemini (Google)
Google Gemini models via Generative Language API (OpenAI-compatible endpoint).
bash
GEMINI_API_KEY=...Base URL: https://generativelanguage.googleapis.com/v1beta/openai/
Mistral
Mistral AI models (Mistral Large, Mistral Small, etc.).
bash
MISTRAL_API_KEY=...xAI
xAI Grok models.
bash
XAI_API_KEY=...DeepSeek
DeepSeek models (DeepSeek V3, R1, etc.).
bash
DEEPSEEK_API_KEY=...Groq
Groq's fast inference for Llama and other models.
bash
GROQ_API_KEY=...Together
Together AI for open models (Llama, Qwen, etc.).
bash
TOGETHER_API_KEY=...Perplexity
Perplexity's search-augmented models.
bash
PERPLEXITY_API_KEY=...Cohere
Cohere models. Uses a custom CohereAdapter because Cohere's API differs from the OpenAI chat completions format. The adapter translates requests/responses so Cohere appears as a standard provider.
bash
COHERE_API_KEY=...Microsoft Foundry (Azure AI)
Azure AI Foundry models via an OpenAI-compatible endpoint. Supports API key and Microsoft Entra ID authentication. Also available as a BYOK provider through Provider Keys.
bash
FOUNDRY_ENDPOINT=https://my-foundry.services.ai.azure.com
FOUNDRY_API_KEY=your-foundry-api-key
FOUNDRY_AUTH_MODE=api_key # or: entra_id| Setting | Default | Description |
|---|---|---|
FOUNDRY_ENDPOINT | (required) | Base URL of your Azure AI Foundry resource |
FOUNDRY_API_KEY | (required for api_key mode) | Azure AI Foundry API key |
FOUNDRY_AUTH_MODE | api_key | Authentication mode: api_key or entra_id |
FOUNDRY_TENANT_ID | (optional) | Azure AD tenant ID (for Entra ID) |
Models use the Azure deployment name directly (e.g., gpt-4.1, gpt-5). The backend dynamically discovers available deployments via GET {endpoint}/openai/v1/models and falls back to a static model list.
Foundry also supports PII referee, embeddings, image generation, TTS, STT, and video generation. See the Microsoft Foundry guide for full configuration.
Entra ID (Managed Identity)
Set FOUNDRY_AUTH_MODE=entra_id and leave FOUNDRY_API_KEY empty. The backend uses DefaultAzureCredential from the Azure Identity SDK, which automatically picks up Managed Identity on Azure, Azure CLI credentials locally, or environment-based service principal credentials.
OpenRouter (Aggregator)
Access 200+ models from every major provider through a single API key. Includes Claude, Gemini, Llama, DeepSeek, Mistral, and more. Useful when you want one key for many providers.
bash
OPENROUTER_API_KEY=sk-or-...
OPENROUTER_BASE_URL=https://openrouter.ai/api/v1
OPENROUTER_DEFAULT_MODEL=openai/gpt-4o-mini| Setting | Default | Description |
|---|---|---|
OPENROUTER_API_KEY | (required) | Your OpenRouter API key |
OPENROUTER_BASE_URL | https://openrouter.ai/api/v1 | API endpoint |
OPENROUTER_DEFAULT_MODEL | openai/gpt-4o-mini | Default model (vendor/model format) |
OPENROUTER_AVAILABLE_MODELS | (see config) | Models shown when OpenRouter is primary |
Ollama (Local)
Run open-source models locally. No API keys, no cloud calls, no cost. Uses Ollama's OpenAI-compatible API.
bash
OLLAMA_BASE_URL=http://localhost:11434
OLLAMA_DEFAULT_MODEL=llama3.2| Setting | Default | Description |
|---|---|---|
OLLAMA_BASE_URL | http://localhost:11434 | Ollama daemon URL |
OLLAMA_DEFAULT_MODEL | llama3.2 | Default model |
OLLAMA_AVAILABLE_MODELS | llama3.2,llama3.1,... | Models shown in UI picker |
Setup:
bash
# macOS
brew install ollama
ollama serve & # Start daemon (runs on port 11434)
ollama pull llama3.2 # Download a model (~2GB)
# Linux
curl -fsSL https://ollama.com/install.sh | sh
ollama serve &
ollama pull llama3.2Auto-Discovery
When Ollama is configured, the model picker automatically queries ollama list and shows all installed models -- no need to manually configure OLLAMA_AVAILABLE_MODELS.
Rich Metadata via /api/show
For each installed model, GhostAI calls Ollama's /api/show endpoint to fetch real metadata instead of relying on name-based heuristics. The enriched fields include:
| Field | Source | Example |
|---|---|---|
| Context length | model_info.*.context_length or parameters.num_ctx | 131072 |
| Parameter count | details.parameter_size | 7B, 70B |
| Capabilities | Architecture inspection (e.g., clip projector → vision) | vision, tools |
| Quantization | details.quantization_level | Q4_K_M |
Brand and Country Mapping
Ollama model descriptions display the correct vendor brand name and country flag based on the model family:
| Family Prefix | Brand | Flag |
|---|---|---|
llama, codellama | Meta | US |
deepseek | DeepSeek | CN |
mistral, mixtral | Mistral AI | FR |
gemma | US | |
qwen | Alibaba Cloud | CN |
phi | Microsoft | US |
command-r | Cohere | CA |
Model Management Endpoints
Platform admins can pull and delete Ollama models through the API without CLI access:
| Endpoint | Method | Description |
|---|---|---|
/api/admin/ollama/models | GET | List models with enriched metadata |
/api/admin/ollama/pull | POST | Pull a new model by tag (e.g., llama3.2) |
/api/admin/ollama/models | DELETE | Delete an installed model by name |
These endpoints require platform admin role. The Model Browser UI exposes pull and delete actions for admin users.
vLLM (Local GPU Server)
Connect to a vLLM serving endpoint (OpenAI-compatible). For GPU-rich environments with multiple concurrent users.
bash
VLLM_BASE_URL=http://localhost:8000The client appends /v1 for the OpenAI-compatible endpoint. No API key required.
llama-cpp (PII Only)
In-process GGUF inference. Used by the PII Referee layer only, not the chat LLM. See Layer 2 below.
Layer 2: PII Referee (PII_REFEREE_RUNTIME)
A separate local LLM that validates PII detections. When Presidio and the NER model disagree about whether text is sensitive, the referee casts the deciding vote. This runs independently of the chat LLM.
Platform Auto-Detection
When PII_REFEREE_MODEL_NAME=auto, GhostAI detects your hardware and picks the best model:
| Platform | Detection | Recommended Model | RAM Needed |
|---|---|---|---|
| Apple Silicon (M1/M2/M3/M4) | Darwin + arm64 | phi-3.5-mini (3.8B) | 2.3 GB |
| NVIDIA GPU (8GB+ VRAM) | nvidia-smi | llama-3.1-8b (8B) | 4.5 GB |
| NVIDIA GPU (<8GB VRAM) | nvidia-smi | phi-3.5-mini (3.8B) | 2.3 GB |
| CPU with 4GB+ RAM | Fallback | phi-3.5-mini (3.8B) | 2.3 GB |
| CPU with <4GB RAM | Fallback | distil-pii-1b (1B) | 0.7 GB |
Model Registry
All models are US-sourced with verified GGUF downloads:
| Key | Model | Vendor | Params | VRAM/RAM | Best For |
|---|---|---|---|---|---|
llama-3.1-8b | Llama 3.1 8B Instruct | Meta | 8B | 4.5 GB | GPU systems (best accuracy) |
phi-3.5-mini | Phi-3.5 Mini Instruct | Microsoft | 3.8B | 2.3 GB | Apple Silicon, general use |
llama-3.2-3b | Llama 3.2 3B Instruct | Meta | 3B | 2.0 GB | Balanced CPU/GPU |
distil-pii-1b | Distil-PII Llama 1B | Meta (base) | 1B | 0.7 GB | Low-resource CPU |
gemma-3-1b | Gemma 3 1B IT | 1B | 0.7 GB | Lightweight CPU |
Runtime: llama-cpp (In-Process)
Loads a GGUF model directly into the Python process. No external daemon. Best for single-user deployments.
bash
PII_REFEREE_RUNTIME=llama-cpp
PII_REFEREE_MODEL_NAME=auto # or: phi-3.5-mini, llama-3.1-8b, etc.
PII_REFEREE_MODELS_DIR=models # GGUF download directory
PII_REFEREE_MODEL_CTX=4096 # Context window (tokens)
PII_REFEREE_MODEL_GPU_LAYERS=-1 # -1 = auto (Metal/CUDA), 0 = CPU only
PII_REFEREE_MODEL_THREADS=0 # 0 = auto-detect CPU cores
PII_REFEREE_WARMUP=true # Pre-load model at startupPros: Zero external dependencies, single process, lowest latency. Cons: Blocks on inference (single-threaded), higher memory footprint in the app process.
Apple Silicon
On macOS with Apple Silicon, PII_REFEREE_MODEL_GPU_LAYERS=-1 automatically offloads all layers to Metal. This gives you GPU-speed inference with no configuration.
Runtime: Ollama (Daemon)
Calls a running Ollama daemon. Supports concurrency and is shared with the chat LLM if both use Ollama.
bash
PII_REFEREE_RUNTIME=ollama
PII_REFEREE_OLLAMA_URL= # Defaults to OLLAMA_BASE_URL
PII_REFEREE_OLLAMA_MODEL=llama3.1:8b # Ollama model tag
PII_REFEREE_MAX_CONCURRENT=4 # Parallel referee callsPros: Shared model cache with chat, concurrent inference, model hot-swap. Cons: Requires Ollama daemon running, extra process to manage.
Circuit Breaker
The Ollama backend includes a circuit breaker: after 5 consecutive failures, it stops calling Ollama for 30 seconds, then retries. This prevents cascade failures if Ollama crashes or becomes unresponsive.
Runtime: vLLM (GPU Server)
Connects to a vLLM serving endpoint (OpenAI-compatible). Best for GPU-rich environments with multiple concurrent users.
bash
PII_REFEREE_RUNTIME=vllm
PII_REFEREE_VLLM_URL=http://localhost:8000/v1
PII_REFEREE_VLLM_MODEL=meta-llama/Llama-3.1-8B-Instruct
PII_REFEREE_MAX_CONCURRENT=8Pros: Highest throughput, continuous batching, tensor parallelism. Cons: Requires dedicated GPU server, more complex setup.
Runtime: Foundry (Azure Cloud)
Routes referee calls to a Foundry-hosted model. Useful when you want cloud-grade accuracy without running local models.
bash
PII_REFEREE_RUNTIME=foundry
PII_REFEREE_FOUNDRY_MODEL=gpt-4.1-miniThe referee reuses FOUNDRY_ENDPOINT and FOUNDRY_API_KEY / Entra ID credentials from the main Foundry configuration. Includes a circuit breaker (stops calling after 5 consecutive failures, retries after 30s). Per-tenant BYOK key resolution is supported (Cloud > Tenant > User).
Dedicated Prometheus metrics: ghostai_referee_foundry_tokens_total, ghostai_referee_foundry_cost_dollars.
Cloud Referee
Unlike local runtimes, Foundry referee calls send PII-candidate text to Azure. Audit logging for Foundry referee can be controlled with PII_REFEREE_AUDIT_FOUNDRY=true|false.
Layer 3: Embeddings
Embeddings for RAG (file search) and memory (conversation recall) default to local SentenceTransformers. Cloud backends (Bedrock, Foundry) are also available.
Local (default)
bash
MEMORY_EMBEDDING_BACKEND=local # Default
MEMORY_EMBEDDING_MODEL=all-MiniLM-L6-v2 # 384-dim embeddings
MEMORY_EMBEDDING_DIM=384 # Must match model output
RAG_EMBEDDING_DEVICE=auto # auto → MPS → CUDA → CPUFoundry
bash
MEMORY_EMBEDDING_BACKEND=foundry
FOUNDRY_EMBEDDING_MODEL=text-embedding-3-large
FOUNDRY_EMBEDDING_ENDPOINT= # Defaults to FOUNDRY_ENDPOINTUses the same FOUNDRY_ENDPOINT and authentication settings as chat. Embedding dimensions default to the model's native output (3072 for text-embedding-3-large). Also available for the content library via CONTENT_LIBRARY_EMBEDDING_BACKEND=foundry.
Common Deployment Combinations
All Direct + Local PII (Recommended)
Use auto-routing with multiple direct providers for chat; keep PII verification local.
bash
# Auto-routing sends each model to its native provider
# LLM_PROVIDER=auto (default -- no need to set)
OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...
GEMINI_API_KEY=...
MISTRAL_API_KEY=...
DEEPSEEK_API_KEY=...
OPENROUTER_API_KEY=sk-or-... # fallback for any model without a direct key
PII_AI_MODEL=local
PII_REFEREE_RUNTIME=llama-cpp
PII_REFEREE_MODEL_GPU_LAYERS=-1
PII_REFEREE_WARMUP=trueData flow: gpt-4o goes directly to OpenAI, claude-3.5-sonnet goes directly to Anthropic, gemini-2.5-flash goes directly to Gemini, etc. Models from providers without a direct key fall back to OpenRouter. PII runs locally.
OpenRouter + Direct Anthropic + Ollama
OpenRouter for broad model access, direct Anthropic for best Claude latency, Ollama for local models.
bash
# LLM_PROVIDER=auto (default)
OPENROUTER_API_KEY=sk-or-...
ANTHROPIC_API_KEY=sk-ant-...
OLLAMA_BASE_URL=http://localhost:11434
PII_AI_MODEL=local
PII_REFEREE_RUNTIME=ollama
PII_REFEREE_OLLAMA_MODEL=llama3.1:8bData flow: Claude models route directly to Anthropic; other cloud models fall back to OpenRouter; local models use Ollama. PII runs locally through Ollama.
Fully Cloud (Simplest)
bash
OPENAI_API_KEY=sk-...
PII_AI_MODEL=dedicated # Use cloud LLM for PII too
PII_AI_DEDICATED_MODEL=gpt-4o-miniData flow: Chat + PII both go through OpenAI. Simplest setup, but PII text leaves your network.
Hybrid (Cloud Chat + Local PII)
bash
OPENAI_API_KEY=sk-...
PII_AI_MODEL=local
PII_REFEREE_RUNTIME=llama-cpp
PII_REFEREE_MODEL_GPU_LAYERS=-1
PII_REFEREE_WARMUP=trueData flow: Chat goes through OpenAI, but PII verification runs locally. Sensitive text for PII detection never leaves the machine.
Fully Local (Air-Gapped)
bash
LLM_PROVIDER=ollama
OLLAMA_BASE_URL=http://localhost:11434
OLLAMA_DEFAULT_MODEL=llama3.2
PII_AI_MODEL=local
PII_REFEREE_RUNTIME=ollama
PII_REFEREE_OLLAMA_MODEL=llama3.1:8bData flow: Everything runs locally. Zero network calls. Works offline / air-gapped.
Multi-Provider via OpenRouter (Legacy)
bash
LLM_PROVIDER=openrouter # explicit -- overrides auto-routing
OPENROUTER_API_KEY=sk-or-...
OPENROUTER_DEFAULT_MODEL=anthropic/claude-3.5-sonnet
PII_AI_MODEL=local
PII_REFEREE_RUNTIME=ollamaData flow: All chat routes through OpenRouter. PII runs locally through Ollama. Consider switching to LLM_PROVIDER=auto and adding direct provider keys for better latency and lower per-token cost.
Threat Analysis LLM Requirements
Security threat analysis requires LLM access to classify detected findings (malicious, suspicious, or benign). The default model is gpt-4o-mini, configurable per organization in Admin Dashboard → Threat Analysis.
In air-gapped deployments where no LLM is available, threat analysis shows ai_status=unavailable and falls back to the original risk levels from the detectors. Encrypted archives and zip bombs remain high-risk regardless of AI availability.