Skip to content

Private Model Admin ​

The Private Model page in the Platform Admin Console gives platform admins full visibility and control over the local LLM used for PII verification and Private Chat. No data leaves the environment — all inference runs on-premise.

  1. Open the Platform Admin Console (requires platform admin role)
  2. In the left sidebar, select Operations → Private Model

Sections ​

Status Card ​

The status card shows the current state of the local inference backend:

FieldDescription
Status indicatorGreen dot = model loaded and ready; Red dot = not loaded
Deploy labelHuman-readable label, e.g. "Local-CPU: phi-3.5-mini" or "Ollama: phi3.5"
RuntimeActive backend: llama-cpp, ollama, or vllm
Hardware backendApple Silicon (Metal), NVIDIA GPU (CUDA), or CPU
RAMTotal system RAM
GPU MemoryAvailable GPU memory (shown for Metal/CUDA only)
CPU CoresDetected core count
ArchitectureCPU architecture (e.g. arm64, x86_64)
Auto-Recommended ModelThe model key that auto would select for this hardware

Use Refresh to re-probe the backend status without reloading the page.

Model Registry ​

A table of all models available for private inference. All models are US-sourced.

ColumnDescription
ModelDisplay name and registry key
VendorModel vendor and HQ location
ParamsParameter count (e.g. 3.8B)
RAM (Q4)Minimum RAM required for Q4 quantized inference
BackendsHardware backends the model is recommended for
Ollama TagTag to use when pulling the model with Ollama
StatusDownload status: Installed (green check) or Download button

Badges:

  • Active (green) — currently selected model
  • Recommended (purple) — the model auto would select for the detected hardware

Downloading Models ​

Models can be downloaded directly from the registry table:

  1. Find the model you want in the registry
  2. If the Status column shows Download, click the button
  3. A progress bar appears in the row, showing real-time bytes transferred and percentage
  4. When complete, the status changes to Installed (green check)

For llama-cpp runtime, downloads are fetched from HuggingFace Hub with optional SHA-256 checksum verification. For Ollama runtime, the standard Ollama pull is used.

If the effective model is not loaded, a Download Model button also appears in the Status Card header for quick access.

Troubleshooting downloads

  • Progress stuck at 0% or "0 / ? MB": Ensure the backend has huggingface-hub installed (it is in requirements-heavy.txt). If you run without heavy dependencies, install with pip install huggingface-hub>=0.20.0 in the backend environment. Check backend logs for errors (e.g. network, HuggingFace auth).
  • "Download error: …" in the UI: The message is the exception from the server. Check backend logs for the full traceback.

Test inference ​

The Test inference card lets you send a test prompt to the configured private model and see the response (or error) and latency. It uses the same inference path as PII verification and Private Chat, so a successful test confirms the backend can reach the model and run a completion.

  • Prompt — Optional; default is "Reply with one word: OK".
  • Max tokens — Optional; default 50 (1–500).
  • Run test — Disabled only while a request is in progress. If the model is not loaded, a warning is shown but you can still run a test to see the backend error.
  • Copy diagnostics — Copies current status, last test result, and timestamp to the clipboard for support tickets or Slack.
  • Troubleshooting link — Opens the docs section for referee/private model issues.

If the test fails

  • Check backend logs for a line starting with Private model test: — it records success/failure, latency, model, and error. Grep CloudWatch or server logs for that string.
  • For referee/Ollama issues (404, model not loaded, connection refused), see the Referee / private model: 404 and "Private Model Not Available" section in the AWS DevOps Troubleshooting Guide (or your deployment’s equivalent).

Runtime Configuration ​

Configure how the private model runs. Changes are saved immediately but take effect on the next model reload — the running backend is not hot-restarted.

Runtime selector ​

OptionDescription
llama-cppIn-process GGUF inference. No separate daemon required. Best for Docker and edge deployments.
ollamaDaemon-based inference via Ollama. Recommended for Private Chat. Easiest model management.
vllmProduction GPU inference via a vLLM OpenAI-compatible endpoint. Best for high-throughput GPU servers.

llama-cpp fields ​

FieldDefaultDescription
ModelautoRegistry key to load, or auto to let the platform pick based on hardware
Model Path Override(empty)Absolute path to a GGUF file. Bypasses the registry and models_dir lookup.
Models DirectorymodelsDirectory where downloaded GGUF files are stored
Context Length4096Maximum token context window
GPU Layers-1Number of layers to offload to GPU. -1 = all layers. 0 = CPU only.
Threads0CPU threads for inference. 0 = auto-detect.
Max Concurrent Requests1Maximum simultaneous inference calls
Warmup on startupoffPre-load the model into memory when the backend starts

ollama fields ​

FieldDefaultDescription
Ollama Base URLhttp://localhost:11434URL of the Ollama daemon
Ollama Model Tag(empty)Ollama model tag (e.g. phi3.5, llama3.2:3b, mistral)

vllm fields ​

FieldDefaultDescription
vLLM Base URLhttp://localhost:8000/v1OpenAI-compatible vLLM endpoint
vLLM Model Name(empty)HuggingFace model name (e.g. meta-llama/Llama-3.2-3B-Instruct)

API Reference ​

All endpoints require platform admin authentication.

EndpointMethodDescription
/api/platform-admin/private-model/statusGETCurrent runtime status, platform info, recommended model
/api/platform-admin/private-model/registryGETFull model registry with download status
/api/platform-admin/private-model/configGETCurrent pii_referee_* configuration values
/api/platform-admin/private-model/configPUTPartially update runtime configuration
/api/platform-admin/private-model/testPOSTRun a test completion. Body: {"prompt?: string", "max_tokens?: number"}. Response: { success, response?, error?, latency_ms, model_used? }. Always 200.
/api/platform-admin/private-model/downloadPOSTStream model download progress (SSE). Body: {"model_key": "phi-3.5-mini"}

Example: GET /status ​

json
{
  "runtime": "llama-cpp",
  "model_name": "auto",
  "is_loaded": true,
  "deploy_label": "Local-Metal: phi-3.5-mini",
  "platform": {
    "backend": "metal",
    "gpu_memory_gb": 36.0,
    "cpu_cores": 12,
    "ram_gb": 36.0,
    "arch": "arm64"
  },
  "recommended_model": "phi-3.5-mini"
}

Example: PUT /config ​

json
{
  "runtime": "ollama",
  "ollama_url": "http://localhost:11434",
  "ollama_model": "phi3.5"
}

Only fields included in the request body are updated. Omitted fields retain their current values.

Environment Variables ​

The configuration fields map directly to environment variables. Changes made via the UI override the values at runtime; persistent changes require updating the environment.

UI FieldEnvironment Variable
RuntimePII_REFEREE_RUNTIME
ModelPII_REFEREE_MODEL_NAME
Model Path OverridePII_REFEREE_MODEL_PATH
Models DirectoryPII_REFEREE_MODELS_DIR
Context LengthPII_REFEREE_MODEL_CTX
GPU LayersPII_REFEREE_MODEL_GPU_LAYERS
ThreadsPII_REFEREE_MODEL_THREADS
Max ConcurrentPII_REFEREE_MAX_CONCURRENT
WarmupPII_REFEREE_WARMUP
Ollama URLPII_REFEREE_OLLAMA_URL
Ollama ModelPII_REFEREE_OLLAMA_MODEL
vLLM URLPII_REFEREE_VLLM_URL
vLLM ModelPII_REFEREE_VLLM_MODEL

SOC 2 Type II Certified | BAA Available | HIPAA-Ready