Appearance
Private Model Admin
The Private Model page in the Platform Admin Console gives platform admins full visibility and control over the local LLM used for PII verification and Private Chat. No data leaves the environment — all inference runs on-premise.
Navigation
- Open the Platform Admin Console (requires platform admin role)
- In the left sidebar, select Operations → Private Model
Sections
Status Card
The status card shows the current state of the local inference backend:
| Field | Description |
|---|---|
| Status indicator | Green dot = model loaded and ready; Red dot = not loaded |
| Deploy label | Human-readable label, e.g. "Local-CPU: phi-3.5-mini" or "Ollama: phi3.5" |
| Runtime | Active backend: llama-cpp, ollama, or vllm |
| Hardware backend | Apple Silicon (Metal), NVIDIA GPU (CUDA), or CPU |
| RAM | Total system RAM |
| GPU Memory | Available GPU memory (shown for Metal/CUDA only) |
| CPU Cores | Detected core count |
| Architecture | CPU architecture (e.g. arm64, x86_64) |
| Auto-Recommended Model | The model key that auto would select for this hardware |
Use Refresh to re-probe the backend status without reloading the page.
Model Registry
A table of all models available for private inference. All models are US-sourced.
| Column | Description |
|---|---|
| Model | Display name and registry key |
| Vendor | Model vendor and HQ location |
| Params | Parameter count (e.g. 3.8B) |
| RAM (Q4) | Minimum RAM required for Q4 quantized inference |
| Backends | Hardware backends the model is recommended for |
| Ollama Tag | Tag to use when pulling the model with Ollama |
| Status | Download status: Installed (green check) or Download button |
Badges:
- Active (green) — currently selected model
- Recommended (purple) — the model
autowould select for the detected hardware
Downloading Models
Models can be downloaded directly from the registry table:
- Find the model you want in the registry
- If the Status column shows Download, click the button
- A progress bar appears in the row, showing real-time bytes transferred and percentage
- When complete, the status changes to Installed (green check)
For llama-cpp runtime, downloads are fetched from HuggingFace Hub with optional SHA-256 checksum verification. For Ollama runtime, the standard Ollama pull is used.
If the effective model is not loaded, a Download Model button also appears in the Status Card header for quick access.
Troubleshooting downloads
- Progress stuck at 0% or "0 / ? MB": Ensure the backend has
huggingface-hubinstalled (it is inrequirements-heavy.txt). If you run without heavy dependencies, install withpip install huggingface-hub>=0.20.0in the backend environment. Check backend logs for errors (e.g. network, HuggingFace auth). - "Download error: …" in the UI: The message is the exception from the server. Check backend logs for the full traceback.
Test inference
The Test inference card lets you send a test prompt to the configured private model and see the response (or error) and latency. It uses the same inference path as PII verification and Private Chat, so a successful test confirms the backend can reach the model and run a completion.
- Prompt — Optional; default is "Reply with one word: OK".
- Max tokens — Optional; default 50 (1–500).
- Run test — Disabled only while a request is in progress. If the model is not loaded, a warning is shown but you can still run a test to see the backend error.
- Copy diagnostics — Copies current status, last test result, and timestamp to the clipboard for support tickets or Slack.
- Troubleshooting link — Opens the docs section for referee/private model issues.
If the test fails
- Check backend logs for a line starting with
Private model test:— it records success/failure, latency, model, and error. Grep CloudWatch or server logs for that string. - For referee/Ollama issues (404, model not loaded, connection refused), see the Referee / private model: 404 and "Private Model Not Available" section in the AWS DevOps Troubleshooting Guide (or your deployment’s equivalent).
Runtime Configuration
Configure how the private model runs. Changes are saved immediately but take effect on the next model reload — the running backend is not hot-restarted.
Runtime selector
| Option | Description |
|---|---|
llama-cpp | In-process GGUF inference. No separate daemon required. Best for Docker and edge deployments. |
ollama | Daemon-based inference via Ollama. Recommended for Private Chat. Easiest model management. |
vllm | Production GPU inference via a vLLM OpenAI-compatible endpoint. Best for high-throughput GPU servers. |
llama-cpp fields
| Field | Default | Description |
|---|---|---|
| Model | auto | Registry key to load, or auto to let the platform pick based on hardware |
| Model Path Override | (empty) | Absolute path to a GGUF file. Bypasses the registry and models_dir lookup. |
| Models Directory | models | Directory where downloaded GGUF files are stored |
| Context Length | 4096 | Maximum token context window |
| GPU Layers | -1 | Number of layers to offload to GPU. -1 = all layers. 0 = CPU only. |
| Threads | 0 | CPU threads for inference. 0 = auto-detect. |
| Max Concurrent Requests | 1 | Maximum simultaneous inference calls |
| Warmup on startup | off | Pre-load the model into memory when the backend starts |
ollama fields
| Field | Default | Description |
|---|---|---|
| Ollama Base URL | http://localhost:11434 | URL of the Ollama daemon |
| Ollama Model Tag | (empty) | Ollama model tag (e.g. phi3.5, llama3.2:3b, mistral) |
vllm fields
| Field | Default | Description |
|---|---|---|
| vLLM Base URL | http://localhost:8000/v1 | OpenAI-compatible vLLM endpoint |
| vLLM Model Name | (empty) | HuggingFace model name (e.g. meta-llama/Llama-3.2-3B-Instruct) |
API Reference
All endpoints require platform admin authentication.
| Endpoint | Method | Description |
|---|---|---|
/api/platform-admin/private-model/status | GET | Current runtime status, platform info, recommended model |
/api/platform-admin/private-model/registry | GET | Full model registry with download status |
/api/platform-admin/private-model/config | GET | Current pii_referee_* configuration values |
/api/platform-admin/private-model/config | PUT | Partially update runtime configuration |
/api/platform-admin/private-model/test | POST | Run a test completion. Body: {"prompt?: string", "max_tokens?: number"}. Response: { success, response?, error?, latency_ms, model_used? }. Always 200. |
/api/platform-admin/private-model/download | POST | Stream model download progress (SSE). Body: {"model_key": "phi-3.5-mini"} |
Example: GET /status
json
{
"runtime": "llama-cpp",
"model_name": "auto",
"is_loaded": true,
"deploy_label": "Local-Metal: phi-3.5-mini",
"platform": {
"backend": "metal",
"gpu_memory_gb": 36.0,
"cpu_cores": 12,
"ram_gb": 36.0,
"arch": "arm64"
},
"recommended_model": "phi-3.5-mini"
}Example: PUT /config
json
{
"runtime": "ollama",
"ollama_url": "http://localhost:11434",
"ollama_model": "phi3.5"
}Only fields included in the request body are updated. Omitted fields retain their current values.
Environment Variables
The configuration fields map directly to environment variables. Changes made via the UI override the values at runtime; persistent changes require updating the environment.
| UI Field | Environment Variable |
|---|---|
| Runtime | PII_REFEREE_RUNTIME |
| Model | PII_REFEREE_MODEL_NAME |
| Model Path Override | PII_REFEREE_MODEL_PATH |
| Models Directory | PII_REFEREE_MODELS_DIR |
| Context Length | PII_REFEREE_MODEL_CTX |
| GPU Layers | PII_REFEREE_MODEL_GPU_LAYERS |
| Threads | PII_REFEREE_MODEL_THREADS |
| Max Concurrent | PII_REFEREE_MAX_CONCURRENT |
| Warmup | PII_REFEREE_WARMUP |
| Ollama URL | PII_REFEREE_OLLAMA_URL |
| Ollama Model | PII_REFEREE_OLLAMA_MODEL |
| vLLM URL | PII_REFEREE_VLLM_URL |
| vLLM Model | PII_REFEREE_VLLM_MODEL |
Related
- Private Model Feature — End-user documentation
- Audit Logs — AI audit log viewer
- Deployment: LLM Providers — Full local LLM deployment guide