Appearance
Private Model
The Private Model is a privacy-preserving AI layer that runs entirely within your environment. No data is sent to external cloud services. It supports both PII verification (judging ambiguous entity spans) and Private Chat (full conversations with local models).
Overview
GhostAI's PII detection pipeline uses multiple layers — pattern matching, named entity recognition, and AI verification. By default, the AI verification step uses a cloud LLM. The Private Model replaces that cloud call with a local model that runs on your own hardware, ensuring sensitive text never leaves your network.
The Private Model also powers Private Chat: full conversations with local models for use cases where the entire exchange must stay on-premise.
Key Benefits
- Zero data leakage — All inference runs locally; nothing is sent to external providers
- Fast — Optimized to verify only ambiguous detections, not full documents
- Cross-platform — Works on Mac, Windows, and Linux
- Hardware-aware — Automatically selects the best model for your CPU or GPU
- US-sourced models — All models are from US-based vendors
How It Works
After pattern matching and NER detect potential PII, most entities are already high-confidence (clear matches like SSNs, credit cards, emails). The Private Model reviews only the ambiguous subset — entities where detection confidence falls in a gray zone. This keeps processing fast and focused.
The result is the same accuracy as cloud AI verification, with the privacy guarantee that your text never leaves your machine.
Default Behavior
Private Model is the default for both Detection and Verification on all accounts. No configuration is needed — your data stays private out of the box. Both new and existing users are automatically defaulted to Private Model.
Changing the Model
Via the UI
- Go to Settings → AI Detector tab
- Under Detection Model, select your preferred option (Private Model, Cloud API, or Custom Endpoint)
- Under Verification Model, select your preferred option
Via environment variable (server-wide default)
- Set
PII_AI_MODEL=localin your environment - Run:
python -m app.cli.pii_model download - The platform auto-detects your hardware and selects the best configuration
Per-Conversation Override
Use the AI model dropdown in the chat header bar to override the detection model for a single conversation. This follows the configuration hierarchy: Cloud → Tenant → User → Conversation → Subject.
Hardware Requirements
| Hardware | Support | Notes |
|---|---|---|
| Mac (Apple Silicon) | Metal acceleration | Recommended for local development |
| NVIDIA GPU | CUDA acceleration | Best performance for production |
| CPU only | Supported | Slower but fully functional |
The Private Model requires 2–5 GB of RAM depending on hardware. GPU-equipped machines automatically use a higher-quality model variant.
Configuration Hierarchy
The Private Model setting follows GhostAI's 5-tier configuration hierarchy:
- Cloud — Platform-wide default
- Tenant — Organization override
- User — Personal preference
- Conversation — Per-chat override via header dropdown
- Subject — Auto-triggered by detected topic
Organization admins can enforce the Private Model at the tenant level, ensuring all users in the organization use local-only verification.
Private Chat
The Private Model is not limited to PII verification. You can use it for full conversations when the entire exchange must stay on-premise. Private Chat routes all inference through local backends — no data is sent to external APIs.
Backend Options
| Backend | Best For | Notes |
|---|---|---|
| Ollama | Chat (recommended) | Easiest setup, daemon-based, supports streaming and model management |
| llama-cpp | Zero-dependency | In-process inference, no separate daemon; ideal for Docker and edge |
| vLLM | Production GPU | High throughput, continuous batching; for AWS g5/p4 or on-premise GPU servers |
Ollama is recommended for Private Chat because it handles model loading, streaming, and updates. Use llama-cpp when you want zero external dependencies. Use vLLM when you need production-scale GPU throughput.
Chat-Optimized Models
The registry includes chat-optimized models suitable for full conversations:
| Model | Size | Best For |
|---|---|---|
| Mistral 7B | 7B | Balanced quality and speed |
| Gemma 2 9B | 9B | Strong reasoning, Google-sourced |
| Qwen 2.5 7B | 7B | Multilingual, instruction-tuned |
These models are tuned for chat rather than PII verification and provide better conversational quality.
Model Browser Integration
Private models appear in the Model Browser with clear visual indicators:
- Green shield badge — Indicates the model is local and privacy-preserving
- "Data never leaves your machine" — Subtitle shown under the model name
Use the locality filter to show only local models, or browse all models and look for the green shield to identify Private options.
Ollama Rich Metadata
When Ollama is configured, GhostAI fetches detailed model metadata from Ollama's /api/show endpoint for each installed model. This replaces the previous approach of guessing capabilities from the model name. Each Ollama model now displays:
- Real context length — Actual
num_ctxfrom the model's parameters instead of a hardcoded default - Parameter count — True parameter size (e.g., 7B, 70B) from the model manifest
- Capabilities — Accurate vision and tool-calling support detected from the model architecture
- Brand attribution — Correct vendor names (DeepSeek, Meta, Mistral AI, Google, etc.) with country flag indicators
Ollama Model Management
Platform admins can manage Ollama models directly from the Model Browser UI without using the command line.
Pull New Models
- Open the Model Browser and navigate to the Ollama section
- Use the admin pull interface to specify an Ollama model tag (e.g.,
llama3.2,deepseek-r1:14b) - The model is downloaded in the background; progress is tracked via the UI
Delete Models
- In the Model Browser, locate the installed Ollama model
- Use the admin delete action to remove the model from the Ollama daemon
- The model is immediately removed from the catalog
Admin API Endpoints
| Endpoint | Method | Description |
|---|---|---|
/api/admin/ollama/models | GET | List all Ollama models with rich metadata |
/api/admin/ollama/pull | POST | Pull a new model by tag |
/api/admin/ollama/models | DELETE | Delete an installed model |
These endpoints require platform admin authentication. Non-admin users see Ollama models in the browser but cannot pull or delete them.
Platform Admin Console
Platform admins have a dedicated Private Model page in the Platform Admin Console for full visibility and control over the local inference backend.
Navigating to the Private Model Page
- Open the Platform Admin Console (platform admins only)
- In the left sidebar, click Operations → Private Model
What You Can Do
Status Card — real-time view of the local LLM:
- Green/red indicator showing whether the model is loaded and ready
- Active deploy label (e.g. "Local-CPU: phi-3.5-mini")
- Detected hardware: backend type (Apple Silicon / NVIDIA / CPU), RAM, GPU memory, CPU cores
- Auto-recommended model for the detected hardware
Model Registry — browse all available models:
- Full registry table with vendor, parameter count, RAM requirement, supported backends, and Ollama tag
- Active model highlighted with a green badge
- Hardware-recommended model highlighted with a purple badge
Runtime Configuration — change how the private model runs:
- Switch between
llama-cpp(in-process),ollama(daemon), andvllm(GPU server) - For llama-cpp: select model from registry (or
auto), set model path override, context length, GPU layers, threads, max concurrent requests, and warmup - For Ollama: set base URL and model tag
- For vLLM: set base URL and model name
- Save configuration — changes take effect on the next model reload
Private Model Admin API Endpoints
| Endpoint | Method | Description |
|---|---|---|
/api/platform-admin/private-model/status | GET | Current runtime status, platform info, recommended model |
/api/platform-admin/private-model/registry | GET | Full model registry |
/api/platform-admin/private-model/config | GET | Current pii_referee_* configuration values |
/api/platform-admin/private-model/config | PUT | Update runtime configuration |
All endpoints require platform admin authentication.
Related
- PII Detection — Full detection pipeline documentation
- Industry Verticals — Vertical-specific detection settings
- Private Model Admin — Platform admin reference for the Private Model page