Skip to content

Private Model ​

The Private Model is a privacy-preserving AI layer that runs entirely within your environment. No data is sent to external cloud services. It supports both PII verification (judging ambiguous entity spans) and Private Chat (full conversations with local models).

Overview ​

GhostAI's PII detection pipeline uses multiple layers — pattern matching, named entity recognition, and AI verification. By default, the AI verification step uses a cloud LLM. The Private Model replaces that cloud call with a local model that runs on your own hardware, ensuring sensitive text never leaves your network.

The Private Model also powers Private Chat: full conversations with local models for use cases where the entire exchange must stay on-premise.

Key Benefits ​

  • Zero data leakage — All inference runs locally; nothing is sent to external providers
  • Fast — Optimized to verify only ambiguous detections, not full documents
  • Cross-platform — Works on Mac, Windows, and Linux
  • Hardware-aware — Automatically selects the best model for your CPU or GPU
  • US-sourced models — All models are from US-based vendors

How It Works ​

After pattern matching and NER detect potential PII, most entities are already high-confidence (clear matches like SSNs, credit cards, emails). The Private Model reviews only the ambiguous subset — entities where detection confidence falls in a gray zone. This keeps processing fast and focused.

The result is the same accuracy as cloud AI verification, with the privacy guarantee that your text never leaves your machine.

Default Behavior ​

Private Model is the default for both Detection and Verification on all accounts. No configuration is needed — your data stays private out of the box. Both new and existing users are automatically defaulted to Private Model.

Changing the Model ​

Via the UI ​

  1. Go to Settings → AI Detector tab
  2. Under Detection Model, select your preferred option (Private Model, Cloud API, or Custom Endpoint)
  3. Under Verification Model, select your preferred option

Via environment variable (server-wide default) ​

  1. Set PII_AI_MODEL=local in your environment
  2. Run: python -m app.cli.pii_model download
  3. The platform auto-detects your hardware and selects the best configuration

Per-Conversation Override ​

Use the AI model dropdown in the chat header bar to override the detection model for a single conversation. This follows the configuration hierarchy: Cloud → Tenant → User → Conversation → Subject.

Hardware Requirements ​

HardwareSupportNotes
Mac (Apple Silicon)Metal accelerationRecommended for local development
NVIDIA GPUCUDA accelerationBest performance for production
CPU onlySupportedSlower but fully functional

The Private Model requires 2–5 GB of RAM depending on hardware. GPU-equipped machines automatically use a higher-quality model variant.

Configuration Hierarchy ​

The Private Model setting follows GhostAI's 5-tier configuration hierarchy:

  1. Cloud — Platform-wide default
  2. Tenant — Organization override
  3. User — Personal preference
  4. Conversation — Per-chat override via header dropdown
  5. Subject — Auto-triggered by detected topic

Organization admins can enforce the Private Model at the tenant level, ensuring all users in the organization use local-only verification.

Private Chat ​

The Private Model is not limited to PII verification. You can use it for full conversations when the entire exchange must stay on-premise. Private Chat routes all inference through local backends — no data is sent to external APIs.

Backend Options ​

BackendBest ForNotes
OllamaChat (recommended)Easiest setup, daemon-based, supports streaming and model management
llama-cppZero-dependencyIn-process inference, no separate daemon; ideal for Docker and edge
vLLMProduction GPUHigh throughput, continuous batching; for AWS g5/p4 or on-premise GPU servers

Ollama is recommended for Private Chat because it handles model loading, streaming, and updates. Use llama-cpp when you want zero external dependencies. Use vLLM when you need production-scale GPU throughput.

Chat-Optimized Models ​

The registry includes chat-optimized models suitable for full conversations:

ModelSizeBest For
Mistral 7B7BBalanced quality and speed
Gemma 2 9B9BStrong reasoning, Google-sourced
Qwen 2.5 7B7BMultilingual, instruction-tuned

These models are tuned for chat rather than PII verification and provide better conversational quality.

Model Browser Integration ​

Private models appear in the Model Browser with clear visual indicators:

  • Green shield badge — Indicates the model is local and privacy-preserving
  • "Data never leaves your machine" — Subtitle shown under the model name

Use the locality filter to show only local models, or browse all models and look for the green shield to identify Private options.

Ollama Rich Metadata ​

When Ollama is configured, GhostAI fetches detailed model metadata from Ollama's /api/show endpoint for each installed model. This replaces the previous approach of guessing capabilities from the model name. Each Ollama model now displays:

  • Real context length — Actual num_ctx from the model's parameters instead of a hardcoded default
  • Parameter count — True parameter size (e.g., 7B, 70B) from the model manifest
  • Capabilities — Accurate vision and tool-calling support detected from the model architecture
  • Brand attribution — Correct vendor names (DeepSeek, Meta, Mistral AI, Google, etc.) with country flag indicators

Ollama Model Management ​

Platform admins can manage Ollama models directly from the Model Browser UI without using the command line.

Pull New Models ​

  1. Open the Model Browser and navigate to the Ollama section
  2. Use the admin pull interface to specify an Ollama model tag (e.g., llama3.2, deepseek-r1:14b)
  3. The model is downloaded in the background; progress is tracked via the UI

Delete Models ​

  1. In the Model Browser, locate the installed Ollama model
  2. Use the admin delete action to remove the model from the Ollama daemon
  3. The model is immediately removed from the catalog

Admin API Endpoints ​

EndpointMethodDescription
/api/admin/ollama/modelsGETList all Ollama models with rich metadata
/api/admin/ollama/pullPOSTPull a new model by tag
/api/admin/ollama/modelsDELETEDelete an installed model

These endpoints require platform admin authentication. Non-admin users see Ollama models in the browser but cannot pull or delete them.

Platform Admin Console ​

Platform admins have a dedicated Private Model page in the Platform Admin Console for full visibility and control over the local inference backend.

  1. Open the Platform Admin Console (platform admins only)
  2. In the left sidebar, click Operations → Private Model

What You Can Do ​

Status Card — real-time view of the local LLM:

  • Green/red indicator showing whether the model is loaded and ready
  • Active deploy label (e.g. "Local-CPU: phi-3.5-mini")
  • Detected hardware: backend type (Apple Silicon / NVIDIA / CPU), RAM, GPU memory, CPU cores
  • Auto-recommended model for the detected hardware

Model Registry — browse all available models:

  • Full registry table with vendor, parameter count, RAM requirement, supported backends, and Ollama tag
  • Active model highlighted with a green badge
  • Hardware-recommended model highlighted with a purple badge

Runtime Configuration — change how the private model runs:

  • Switch between llama-cpp (in-process), ollama (daemon), and vllm (GPU server)
  • For llama-cpp: select model from registry (or auto), set model path override, context length, GPU layers, threads, max concurrent requests, and warmup
  • For Ollama: set base URL and model tag
  • For vLLM: set base URL and model name
  • Save configuration — changes take effect on the next model reload

Private Model Admin API Endpoints ​

EndpointMethodDescription
/api/platform-admin/private-model/statusGETCurrent runtime status, platform info, recommended model
/api/platform-admin/private-model/registryGETFull model registry
/api/platform-admin/private-model/configGETCurrent pii_referee_* configuration values
/api/platform-admin/private-model/configPUTUpdate runtime configuration

All endpoints require platform admin authentication.

SOC 2 Type II Certified | BAA Available | HIPAA-Ready