Appearance
PII Detection
GhostAI's multi-layer PII detection system identifies and protects over 64 types of sensitive data before it's sent to AI models.
How It Works
User Input → Presidio (Patterns) → NER Model (Context) → Code & Secrets → AI Enhancement → AI Verification / Local LLM Referee → TokenizationDetection Layers
| Layer | Method | Strength |
|---|---|---|
| Presidio | Pattern matching + rules | High precision for structured data (SSNs, emails, credit cards) |
| NER Model | Dymium NER (DeBERTa-v3-large PII model) | Context-aware detection of names, dates, locations, organizations |
| Code & Secrets | Language-specific pattern matching (34 languages) | Detects source code, API keys, database URLs, tokens |
| AI Enhancement | LLM-based analysis | Complex/ambiguous cases missed by pattern + NER |
| AI Verification | LLM-based entity validation | Filters false positives from other layers |
| Local LLM Referee | On-device LLM referee (Llama-3.1-8B / Phi-3.5 for CPU) | Judges only ambiguous spans — no data leaves the machine |
| Vertical NER | Clinical models | Industry-specific entity detection (PHI) |
| Anti-Exfiltration | Obfuscation detection | Catches base64, hex, ROT13, unicode tricks, and more |
File-Type-Specific Detection
GhostAI routes each file type through specialized extractors before PII detection:
| File Category | Extractor | Special Handling |
|---|---|---|
| PyPDF2 + pdfplumber + PyMuPDF | Form fields, annotations, OCR for scanned pages | |
| Word | python-docx + olefile | Comments, tracked changes, headers/footers, embedded OLE |
| Excel | openpyxl + olefile | All sheets (including hidden), cell comments |
| PowerPoint | python-pptx + olefile | Slides, speaker notes, embedded objects |
| Images | Tesseract OCR + CLIP | ID card/passport/credit card classification |
| Archives | Recursive (3 levels) | ZIP, TAR, 7z, RAR with bomb protection |
| stdlib email + olefile | Headers, multipart body, attachments | |
| Medical | pydicom + hl7 | DICOM patient tags, HL7 segments |
| Audio | Whisper (faster-whisper) | Speech-to-text transcription |
| Video | ffmpeg + Whisper + OCR | Audio track, frame sampling, subtitles |
| Database | sqlite3 + mdbtools | Table names, sample rows |
| Code | UTF-8 + secrets detector | 34 languages, API keys, credentials |
Post-Response PII Guard
After the LLM responds, GhostAI applies a post-response check before delivering the answer:
- Raw PII scan: Checks for SSNs, credit cards, emails, phone numbers that bypassed tokenization
- Encoded PII detection: Decodes base64 and hex blocks to look for hidden PII
- Token verification: Flags token-like strings not present in the session vault
- Auto-redaction: Detected PII in responses is replaced with
[REDACTED_*]markers
The guard runs in both streaming and non-streaming response paths. In the streaming path (generate_stream and generate_compare_stream), the guard inspects the accumulated raw response before detokenization. In the non-streaming path, it runs as a synchronous check before the final response is returned.
PII detection is also applied to data tool results in the zero-trust PDF pipeline: when data_query or bi_insight return results containing PII, those results are detected and tokenized before reaching the LLM. See PDF Generation for details.
HMAC Token Authentication
Privacy tokens include an HMAC-SHA256 signature derived from a server-side secret. During detokenization, each token's signature is verified before resolving it to real PII. This prevents an LLM from fabricating valid-looking tokens (e.g., [PERSON_abc123]) that would resolve to real sensitive data.
Tool Argument Validation
Before executing LLM-generated tool calls (execute_python, web_search, data_query, create_canvas, update_canvas), GhostAI validates the arguments for dangerous patterns:
- Python code: Blocks dangerous imports (
os,subprocess,shutil) and functions (exec,eval,compile) - SQL queries: Detects SQL injection patterns (
DROP,DELETE,UNION SELECT, stacked queries) - URLs: Rejects private IP ranges,
file://URIs, and known malicious domains - Canvas HTML: Blocks sandbox escapes (
document.cookie,window.opener,parent.postMessage,WebSocket,sendBeacon), warns on suspicious patterns (eval,fetch,localStorage), and enforces 150KB size limit
Detection Models
GhostAI uses multiple transformer-based models, each optimized for a different detection scenario. Models are pre-downloaded during Docker image build and cached for fast startup.
Dymium NER (Primary NER)
| Property | Value |
|---|---|
| Model | dymium/Dymium-NER-v1 |
| Architecture | DeBERTa-v3-large fine-tuned for PII |
| Purpose | Multi-type PII entity detection |
| Entities | ADDRESS, AUTH, DATE, EMAIL, ID_DOC, ID_FIN, ID_GOV, ID_REF, ORG, PERSON, PHONE, TIME, URL |
| Memory | ~800 MB |
Dymium NER is a DeBERTa-v3-large model fine-tuned specifically for PII detection across 13 entity types. It provides context-aware detection that goes beyond pattern matching — for example, understanding that "Dr. Jane Smith" is a person name even without surrounding PII indicators, detecting credentials and API keys as AUTH entities, and recognizing financial identifiers in unstructured text.
Clinical NER (Healthcare)
| Property | Value |
|---|---|
| Model | obi/deid_roberta_i2b2 |
| Architecture | RoBERTa fine-tuned on i2b2 de-identification corpus |
| Purpose | Protected Health Information (PHI) detection |
| Activation | Loaded when the Healthcare vertical is selected |
| Memory | ~500 MB |
The clinical NER model is trained on real de-identified medical records from the i2b2 shared task. It detects PHI categories required by HIPAA Safe Harbor including patient names, dates, provider names, facility names, medical record numbers, and more.
Private Model (Local Referee)
| Property | Value |
|---|---|
| Purpose | Local PII referee — verifies ambiguous entity spans without sending data externally |
| Runs on | Mac (Metal), NVIDIA GPU (CUDA), CPU |
| Memory | 2–5 GB depending on hardware |
The Private Model runs entirely within your environment and reviews only ambiguous detection results — entities where the pattern-matching and NER layers are not fully confident. It never sends data to external services. Private Model is the default for both detection and verification on all accounts.
The Private Model also supports Private Chat — full conversations with local models, not just PII verification. See the Private Model guide for chat capability, backend options (Ollama, llama-cpp, vLLM), and Model Browser integration.
GhostAI automatically selects the best model for your hardware when you enable the Private Model. GPU-equipped machines use a higher-quality model, while CPU and laptop deployments use a lighter variant optimized for speed.
Model Loading
Detection models are pre-packaged to avoid cold-start latency. At runtime, models are loaded lazily on first use and cached in memory. If a detector or vertical preset is not enabled, its corresponding model is never loaded.
Supported Entity Types
Identity Documents
| Entity Type | Examples | Confidence |
|---|---|---|
US_SSN | 123-45-6789 | 0.99 |
US_DRIVER_LICENSE | D1234567 (varies by state) | 0.85 |
US_PASSPORT | 123456789 | 0.90 |
US_ITIN | 9XX-XX-XXXX | 0.95 |
Financial Data
| Entity Type | Examples | Confidence |
|---|---|---|
CREDIT_CARD | 4111-1111-1111-1111 | 0.99 |
CREDIT_CARD_CVV | 123, 4567 | 0.80 |
US_BANK_NUMBER | Account numbers | 0.85 |
ROUTING_NUMBER | 021000021 (ABA 9-digit) | 0.90 |
SWIFT_CODE | BOFAUS3NXXX (8-11 chars) | 0.92 |
IBAN_CODE | DE89370400440532013000 | 0.95 |
CRYPTO | Bitcoin/Ethereum addresses | 0.90 |
PCI-DSS Compliance
Financial entities are automatically tagged with PCI-DSS compliance classification. Credit card numbers, CVVs, and bank account numbers trigger PCI-DSS alerts.
Contact Information
| Entity Type | Examples | Confidence |
|---|---|---|
EMAIL_ADDRESS | john.doe@company.com | 0.99 |
PHONE_NUMBER | (555) 123-4567 | 0.95 |
STREET_ADDRESS | 123 Main St, City, ST 12345 | 0.85 |
IP_ADDRESS | 192.168.1.1, IPv6 | 0.99 |
URL | https://example.com | 0.95 |
Personal Information
| Entity Type | Examples | Confidence |
|---|---|---|
PERSON | John Smith, Dr. Jane Doe | 0.90 |
DATE_TIME | 01/15/1980, January 15, 1980 | 0.85 |
LOCATION | New York, California | 0.80 |
ORGANIZATION | Acme Corp, FDA | 0.85 |
Healthcare (PHI)
| Entity Type | Examples | Compliance |
|---|---|---|
MEDICAL_LICENSE | NPI: 1234567890 | HIPAA |
MEDICAL_PHI | Patient IDs, MRNs | HIPAA |
MEDICAL_TERM | Diagnoses, procedures | HIPAA |
PROVIDER_NAME | Dr. Smith, MD; Jane Doe, NP | HIPAA |
FACILITY_NAME | Memorial Hospital, City Clinic | HIPAA |
HIPAA Safe Harbor
Healthcare entities are tagged with HIPAA 164.514 Safe Harbor de-identification markers. Title prefixes (Dr., MD, DO, NP, RN) are recognized to improve provider name detection.
Code & Secrets
| Entity Type | Examples | Confidence |
|---|---|---|
SOURCE_CODE | Code blocks (34 languages) | 0.95 |
SECRETS | API keys, passwords | 0.99 |
AWS_KEY | AKIAIOSFODNN7EXAMPLE | 0.99 |
DATABASE_URL | postgresql://user:pass@host/db | 0.95 |
DATABASE_URL | postgresql://user:pass@host/db | 0.95 |
Detection Settings
Global Threshold
Set the minimum confidence score for detection:
bash
PII_SCORE_THRESHOLD=0.5| Threshold | Behavior |
|---|---|
| 0.3 | Very sensitive - may have false positives |
| 0.5 | Balanced (default) |
| 0.7 | Conservative - fewer false positives |
| 0.9 | Very conservative - only high-confidence |
Per-Entity Thresholds
Fine-tune sensitivity for specific entity types via Settings → Thresholds:
json
{
"US_SSN": 0.3, // Very sensitive for SSNs
"PERSON": 0.7, // Less sensitive for names
"DATE_TIME": 0.5 // Balanced for dates
}Per-entity thresholds are persisted to your user profile and applied as a post-filter after Presidio detection. Presidio receives the global minimum threshold, and results are then filtered per-entity type. This is stored in the 5-tier configuration hierarchy (Cloud → Tenant → User) so organization admins can set baseline thresholds that users can further refine.
NER and Name Detection
The NER model uses an internal threshold that is lower than the global threshold, allowing it to catch names and other contextual entities that pattern matching alone would miss. After NER returns results, a global post-filter ensures all entities meet the configured threshold. Certain high-risk entity types (such as names in healthcare contexts) receive a confidence boost based on the active vertical, so they are reliably detected even at moderate confidence levels.
Subject/Topic Detection
GhostAI can auto-detect the subject of a conversation to trigger context-appropriate detection settings. When a message matches a subject preset (via keywords or patterns), the subject indicator appears above the chat input.
Built-in subject presets include domain-specific topics (medical, financial, legal, code) and the general personal_info preset, which triggers on common PII patterns:
- Name + contact information keywords ("colleague", "phone number", "email address")
- Email regex, phone number regex, SSN-like patterns
- Title + name patterns ("Dr. Smith", "Ms. Johnson")
Subject detection is invoked via POST /api/detect-subject and returns matched subjects with confidence scores and matched keywords. The active subject can boost detection sensitivity or switch execution modes.
Execution Modes
Choose how detectors run:
| Mode | Speed | Accuracy | Use Case |
|---|---|---|---|
| Parallel | Fastest | Good | High-volume, real-time |
| Series | Slowest | Best | Maximum accuracy |
| Grouped | Balanced | Very Good | Default recommendation |
Pipeline Configuration
Control detector ordering and grouping through Settings → Pipeline. Each detector has:
| Property | Description |
|---|---|
| Priority | Higher numbers run first. In series mode, determines execution order. |
| Group | In grouped mode, detectors with the same group number run in parallel. |
| Enabled | Toggle individual detectors on/off. |
Default pipeline:
| Detector | Default Priority | Default Group (grouped mode) |
|---|---|---|
| Code & Secrets | 100 | 0 |
| Presidio | 90 | 1 |
| Dymium NER | 80 | 1 |
| AI Detector | 70 | 2 |
Custom priority and group values override the defaults while respecting the selected execution mode pattern. These values are persisted per-user and resolved through the 5-tier configuration hierarchy.
Per-Vertical Entity Whitelisting
Each vertical has a curated set of enabled entity types. Users can toggle individual entity types on/off in Settings → PII Categories. Only enabled entity types are detected; disabled types produce no results. The list is configured per vertical in the backend; use GET /api/verticals to see the enabled entity list for each vertical.
Language Support
Configure detection languages through Settings → Languages. Supported languages include:
| Language | Presidio Support | NER Support |
|---|---|---|
English (en) | Full | Full (Dymium NER) |
Spanish (es) | Full | Limited (Presidio patterns only) |
French (fr) | Full | Limited |
German (de) | Full | Limited |
Italian (it) | Full | Limited |
Portuguese (pt) | Full | Limited |
Chinese (zh) | Partial | Not supported |
Japanese (ja) | Partial | Not supported |
Korean (ko) | Partial | Not supported |
Arabic (ar) | Partial | Not supported |
NER Language Limitation
The Dymium NER model is English-only. When non-English languages are selected, Presidio handles pattern-based detection for those languages, but NER context-based detection (names, organizations, locations) only operates on English text. Multi-language NER support is planned for a future release.
Custom Model Endpoint
For organizations with fine-tuned PII detection models, GhostAI supports routing AI detection calls to a custom OpenAI-compatible endpoint:
- Open Settings → AI Detector tab
- Select Custom Model Endpoint
- Enter your endpoint URL (e.g.,
https://your-model.example.com/v1)
The endpoint must implement the OpenAI Chat Completions API (/chat/completions). GhostAI sends the same detection prompts it would send to OpenAI or OpenRouter, so your model should be fine-tuned on PII detection prompts for best results.
Custom endpoints are gated by the custom_endpoint_allowed Cloud-level setting. Organization admins can disable this feature.
Redaction Style
Choose how detected PII is replaced before being sent to the LLM. Configure via Settings → Redaction tab:
| Style | Description | Example |
|---|---|---|
tokenize | Reversible token (default) | [SSN_abc123] |
mask | Replace with asterisks | ***-**-**** |
hash | Replace with hash value | [HASH:a1b2c3d4] |
encrypt | Replace with encrypted value | [ENC:...] |
remove | Remove entirely | (empty) |
partial | Show first/last characters | J*** D** |
The tokenize style is recommended for most use cases because it allows the AI's response to be detokenized back to the original value. Other styles provide stronger obfuscation but are not reversible.
AI Verification (False Positive Filtering)
AI Verification is an optional LLM-based pass that reviews entities detected by Presidio and NER before they are tokenized. Its purpose is to filter obvious false positives by asking the LLM whether each detected entity is truly PII given the surrounding context.
How It Works
Detected Entities (from Presidio + NER)
│
▼
┌─────────────────────┐
│ AI Verification │
│ LLM reviews each │
│ entity in context │
└─────────────────────┘
│
├── Confirmed → Kept (tokenized as usual)
└── Rejected → Removed (false positive filtered out)The LLM receives each candidate entity along with its surrounding text and decides whether it genuinely represents sensitive information. Rejected entities are logged and excluded from tokenization.
Verification Modes
| Mode | Behavior | Performance | Use Case |
|---|---|---|---|
off | No AI verification | No additional latency | Default when AI detector is off |
suspicious | Verify only low-confidence and FP-prone entities | ~50-200ms per message | Recommended balance of accuracy and speed |
all | Verify every detected entity through the LLM | ~100-500ms per message | Maximum false-positive reduction |
Suspicious mode targets entity types that are most prone to false positives:
DATE_TIME-- phrases like "20 year old" or "last 3 months" often get incorrectly flaggedSTREET_ADDRESS-- numbers like "529" in context ("529 plan") can trigger false matchesPERSON-- common words in certain contexts (e.g., "she have to do") may be misdetected- Low-confidence entities (score below 0.7) from any type
All mode sends every entity to the LLM for review, including high-confidence entities. This provides the strongest false-positive filtering but adds latency to every message.
Per-Conversation Control
You can set the verification mode per conversation by opening the View menu (sliders icon) in the chat header and adjusting AI Verify mode. This override applies only to the current conversation and does not change your default setting.
To change the default for all conversations, go to Settings → Detection Settings → AI Verify Mode.
Configuration
| Setting | Default | Description |
|---|---|---|
PII_AI_VERIFY_MODE | off | Default verification mode |
The effective verification mode follows the 5-tier configuration hierarchy:
Cloud → Tenant → User → Conversation → SubjectHigher tiers can set ceilings and floors to enforce organizational policy. For example, a tenant admin can require at least suspicious mode for all users.
Examples of Filtered False Positives
| Input Text | Without Verification | With Verification |
|---|---|---|
| "She is a 20 year old student" | DATE_TIME flagged on "20 year old" | Filtered out -- not a date |
| "She opened a 529 plan" | STREET_ADDRESS flagged on "529" | Filtered out -- not an address |
| "What does she have to do?" | PROVIDER_NAME flagged on "she have to do" | Filtered out -- not a name |
| "My SSN is 123-45-6789" | US_SSN flagged on "123-45-6789" | Confirmed -- real PII |
When to Enable Verification
If you see frequent false positives -- especially for DATE_TIME, STREET_ADDRESS, or PERSON entities -- enable suspicious mode first. Switch to all only if false positives persist on high-confidence entities.
Local Referee Alternative
For environments where data must never leave the machine, use the Private Model (local LLM referee) instead of external AI verification. Select "Private Model" in Settings → AI Detector tab, or set PII_AI_MODEL=local as a server-wide default. The local referee uses Llama-3.1-8B on GPU or Phi-3.5-mini on CPU. The Private Model also supports full Private Chat conversations. See the Private Model guide and the section below.
Local LLM Referee
The Local LLM Referee is a privacy-preserving alternative to external AI verification. Instead of sending text to an external LLM, it runs a small (1-8B parameter) model directly on the machine. The model only sees ambiguous entity spans with minimal context — it never receives the full document.
UI Configuration
Users can now select "Private Model" directly in the Settings → AI Detector tab for both the detection model and the verification model — no environment variable needed. The effective model follows the 5-tier configuration hierarchy (Cloud → Tenant → User → Conversation → Subject), so an admin can set the server-wide default while individual users override it from the UI. You can also set a per-conversation override using the AI model dropdown in the chat header bar.
Three-Pass Detection Pipeline
Pass 1: Presidio (regex/rules) → High-confidence entities
Pass 2: Dymium NER → Context-aware PII entities
Pass 3: Local LLM Referee → Judges only ambiguous spansHow It Works
Entities are split into three buckets based on their confidence score:
| Score Range | Action | LLM Used? |
|---|---|---|
| >= 0.85 | Accepted outright | No |
| < 0.35 | Rejected outright | No |
| 0.35 — 0.85 | Sent to LLM referee | Yes (minimal context) |
The agreement rule provides an additional optimization: if both Presidio and NER flag the same span with > 0.85 confidence, it is accepted without the LLM, even if individual scores fall in the gray band.
Dual-Lane Routing
- Lane A (Fast Path): For short text (< 1000 tokens). Runs Presidio + NER, then sends only ambiguous spans to the referee. Target: < 50ms for clear text, < 200ms with referee.
- Lane B (Document Path): For long documents (>= 1000 tokens). Chunks the text into 1000-token segments with 160-token overlap, processes each chunk through Presidio + NER, batches all ambiguous spans (20–60 per batch), and sends one batched referee call. Results are merged and deduplicated.
Model Registry (US-Sourced)
All models in the registry are from US-based vendors:
| Model | Vendor | Params | RAM (Q4) | Best For |
|---|---|---|---|---|
| llama-3.1-8b | Meta (Menlo Park CA) | 8B | 4.5 GB | CUDA 8GB+ (recommended for GPU) |
| phi-3.5-mini | Microsoft (Redmond WA) | 3.8B | 2.3 GB | Apple Silicon, CPU (recommended for CPU) |
| llama-3.2-3b | Meta (Menlo Park CA) | 3B | 2.0 GB | Apple Silicon, CPU |
| distil-pii-1b | Meta base (fine-tuned) | 1B | 0.7 GB | CPU (low RAM) |
| gemma-3-1b | Google (Mountain View CA) | 1B | 0.7 GB | CPU fallback |
Platform auto-detection selects the best model automatically. Override with PII_REFEREE_MODEL_NAME.
Runtime Options
| Runtime | How It Works | Best For |
|---|---|---|
| llama-cpp | In-process inference, zero network | Docker, edge, Mac dev |
| ollama | External daemon, OpenAI-compat API | Local dev with Ollama |
| vllm | GPU server, continuous batching | Production GPU (AWS g5/p4) |
CLI Commands
bash
# List available models and platform recommendation
python -m app.cli.pii_model list
# Download the auto-recommended model
python -m app.cli.pii_model download
# Download a specific model
python -m app.cli.pii_model download --model llama-3.1-8b
# Download via Ollama instead of HuggingFace
python -m app.cli.pii_model download --runtime ollama
# Check platform and model status
python -m app.cli.pii_model status
# Verify model integrity (SHA-256)
python -m app.cli.pii_model verifyConfiguration
| Setting | Default | Description |
|---|---|---|
PII_AI_MODEL | user | Set to local to enable the referee |
PII_REFEREE_RUNTIME | llama-cpp | Runtime backend |
PII_REFEREE_MODEL_NAME | auto | Model key or auto for platform detection |
PII_REFEREE_LOW_THRESHOLD | 0.35 | Below this → reject without LLM |
PII_REFEREE_HIGH_THRESHOLD | 0.85 | Above this → accept without LLM |
PII_REFEREE_ACCEPT_THRESHOLD | 0.90 | LLM must return ≥ this to keep |
PII_REFEREE_BATCH_SIZE | 40 | Spans per LLM batch call |
PII_REFEREE_AGREEMENT_SKIP | true | Skip LLM if Presidio + NER agree |
PII_LANE_THRESHOLD_TOKENS | 1000 | Lane A / Lane B boundary |
Setup Guide
Apple Silicon Mac (M2/M3/M4):
bash
pip install llama-cpp-python # auto-detects Metal
python -m app.cli.pii_model download
# Set PII_AI_MODEL=local, PII_REFEREE_RUNTIME=llama-cppNVIDIA GPU:
bash
CMAKE_ARGS="-DGGML_CUDA=on" pip install llama-cpp-python
python -m app.cli.pii_model download --model llama-3.1-8b
# Set PII_AI_MODEL=local, PII_REFEREE_RUNTIME=llama-cppCPU-only:
bash
pip install llama-cpp-python
python -m app.cli.pii_model download # auto-selects phi-3.5-mini or distil-pii-1b
# Set PII_AI_MODEL=local, PII_REFEREE_RUNTIME=llama-cppProduction GPU (vLLM):
bash
# In docker-compose.production.yml, enable the gpu profile:
docker compose --profile gpu up -d
# Set PII_REFEREE_RUNTIME=vllm, PII_REFEREE_VLLM_URL=http://vllm:8000/v1Expected Performance
On NVIDIA GPU with Llama-3.1-8B (or Apple Silicon/CPU with Phi-3.5-mini):
| Scenario | Latency | Notes |
|---|---|---|
| Short text, no ambiguous spans | < 50ms | No LLM call needed |
| Short text, with referee | ~100-300ms | LLM judges a few spans |
| 10-page document | ~1-3s | Chunked NER + batched referee |
| Model warm-up | ~2-3s | One-time at startup |
AI Audit Logging
Every LLM call made by the PII detection and verification pipeline can be recorded for compliance monitoring and troubleshooting. The audit system captures metadata about each AI call and, optionally, the raw prompts and responses.
Audit Levels
| Level | What's Recorded | Storage Impact |
|---|---|---|
none | No audit records | None |
summary | Model, text hash, entity count, latency (no raw text) | ~200 bytes/event |
full | Everything in summary + raw LLM prompt and response | 2-10 KB/event |
5-Tier Configuration with Ceiling and Floor
The effective audit level is resolved through the configuration hierarchy with ceiling and floor enforcement:
Cloud (ceiling/floor) → Tenant → User → Conversation → Subject| Setting | Default | Description |
|---|---|---|
PII_AI_AUDIT_LEVEL | summary | Base audit level |
PII_AI_AUDIT_FLOOR | none | Minimum level (cannot be reduced by lower tiers) |
PII_AI_AUDIT_CEILING | full | Maximum level (cannot be exceeded by lower tiers) |
Example: If the Cloud sets floor=summary and ceiling=summary, all tenants and users will record summary-level audits regardless of their individual settings. This is useful for compliance mandates (e.g., healthcare organizations that require audit trails).
Vendor Debug Access
Controls debug-level audit detail visibility.
| Setting | Default | Description |
|---|---|---|
PII_VENDOR_DEBUG_ACCESS | false | Controls debug-level audit detail visibility |
PII_VENDOR_DEBUG_ACCESS_ALLOWED | true | Cloud-level gate (must be true for any lower tier to enable) |
When disabled (default), detailed fields are stripped from API responses. Enable only when actively troubleshooting with GhostAI support.
Viewing Audit Logs
Navigate to Platform Admin Console > AI Audit to browse audit events. The viewer provides:
- Stats badges showing total events, detection/verification breakdown, and average latency
- Type filter to view detection events, verification events, or both
- Pagination with configurable page size (25, 50, or 100 per page)
- Expandable detail rows for full-level events (with vendor debug access enabled)
For the complete admin guide, see AI Audit Logs.
Real-Time Detection
Input Highlighting
As you type, detected PII is highlighted:
- Yellow background: PII detected
- Hover tooltip: Entity type and confidence
- Badge count: Total PII items in message
Detection Explanations
Click on any highlighted PII to see (available in both single chat and comparison mode):
- Entity type: What was detected
- Confidence score: How certain the detection is
- Detection method: Which layer detected it
- Pattern matched: The specific pattern (if applicable)
When PII is detected before you send (in the message preview), you can:
- Expand the list — Click the bar to expand and see each flagged item with the detected text (truncated) and its token. For attachments, each file lists its entities the same way.
- Locate in context — Hover or focus (keyboard/tab or tap) any item: the corresponding span is highlighted in the prompt (message PII) or the associated file is highlighted in the attachments list and scrolled into view if off-screen. The same preview behavior (expand, original text, hover/focus highlight) is available in Compare mode; per-message override (Don't redact / Add to safe terms) is not offered in Compare mode.
- Don't redact for this message — Exclude that span from redaction for this send only. The text is sent to the AI as-is for this message; high-risk types (SSN, credit card, etc.) cannot be excluded.
- Add to safe terms and send — Add the detected text to your Safe Terms so future messages won't flag it, and send this message with the term unredacted.
These options are available only when your organization allows per-message overrides (Admin Dashboard → Policies → Allow PII overrides per message). If the policy is disabled, use Settings → My Terms to add safe terms instead.
For a full technical and security overview, see the PII Per-Message Override whitepaper. For training a custom PII model on your data, see PII Fine-Tuning and the True ML Fine-Tuning whitepaper.
Compliance Tagging
Detected entities are automatically tagged with relevant compliance frameworks:
| Entity Type | Compliance Tags |
|---|---|
US_SSN | HIPAA, PCI-DSS, GDPR, CCPA |
CREDIT_CARD | PCI-DSS, GDPR |
MEDICAL_PHI | HIPAA, HITECH |
PERSON + DATE_TIME | HIPAA (PHI indicator) |
Feedback & Tuning
Mark False Positives
When GhostAI incorrectly flags something as PII:
- Click the highlighted item
- Select "Not PII"
- (Optional) Provide reason
Mark False Negatives
When GhostAI misses PII:
- Select the text that should be PII
- Click "Report Missed PII"
- Select the entity type
Impact on Quality Score
User feedback directly drives the PII Accuracy dimension (20% weight) in the Quality Dashboard:
| Feedback Volume | Score Calculation |
|---|---|
| 5+ feedback entries | Real accuracy: (1 - false_positive_rate) × 100 |
| 1-4 entries | Blended: weighted mix of real rate + 98% baseline |
| No feedback, PII active | Baseline: 98% (assumed good until feedback says otherwise) |
| No PII activity | Neutral: 100% (nothing to measure = nothing wrong) |
The more feedback users provide, the more accurate and data-driven the quality score becomes.
Automatic Recommendations
Based on feedback, GhostAI recommends threshold adjustments:
- View in Settings → Detection Accuracy
- One-click apply recommendations
- Export training data for custom models
Detection Cache
PII detection results are cached to avoid redundant Presidio processing under load (e.g., the same message retried or re-rendered).
Security Model
The cache is designed to never store plaintext PII:
| Aspect | Implementation |
|---|---|
| Cache key | SHA-256 hash of text + detection parameters. Original text is NOT stored. |
| Cached values | Entity positions (start/end), types, scores, and metadata only. Matched text fragments are NOT stored. |
| Cache hit | On hit, entity .text is re-sliced from the source using stored start:end offsets. |
| TTL | Entries expire after PII_CACHE_TTL seconds (default: 300s / 5 min). |
| Shutdown | Cache is explicitly cleared when the process shuts down. |
| Max size | PII_CACHE_SIZE entries (default: 2000). Oldest entries evicted when full. |
| Thread safety | Thread-safe via threading.Lock. |
Configuration
| Variable | Default | Description |
|---|---|---|
PII_CACHE_SIZE | 2000 | Max cached detection results |
PII_CACHE_TTL | 300 | Entry TTL in seconds |
PII_MAX_CONCURRENT | 8 | Max parallel detection threads |
Observability
Prometheus metrics exposed:
ghostai_pii_cache_hits_total— cache hit countghostai_pii_cache_misses_total— cache miss count
The /api/pii/cache endpoint returns current cache statistics (hit rate, size, TTL).
Chunk-Level PII Cache (RAG)
For RAG-indexed files, PII detection is cached per chunk hash. When a chunk is re-indexed (e.g., after file re-upload), the system checks the cache by SHA-256 hash of the chunk content. If the chunk has not changed, cached PII results are reused instead of re-running detection.
| Setting | Default | Description |
|---|---|---|
rag_pii_cache_enabled | true | Enable chunk-level PII caching for RAG |
Configure in Admin Dashboard → RAG / File Intelligence. Disable if you need to force re-detection on every index.
File-Level PII Scan Cache
When a file is scanned via POST /api/files/{file_id}/scan, the full set of PII detection results (entity types, positions, scores, original text, and tokens) is serialized and stored in the pii_scan_cache column of the attachments table.
During chat, the _prepare_attachments_for_llm function checks this cache before running PII detection on non-RAG attachments. If cached results exist, the expensive Presidio detection step is skipped entirely.
| Behavior | Detail |
|---|---|
| Cache storage | JSON blob on the attachments.pii_scan_cache column |
| Cache population | Automatic after POST /api/files/{file_id}/scan |
| Cache consumer | _prepare_attachments_for_llm in the chat pipeline |
| Scope | Non-RAG files only (RAG files use the chunk-level cache above) |
| Invalidation | Overwritten on re-upload/re-scan; not time-based |
This is especially impactful in comparison mode where the same file is sent to 2-4 models -- the PII scan runs once during upload and is reused for all model requests.
API Usage
Detect PII in Text
bash
curl -X POST http://localhost:8000/api/detect \
-H "Content-Type: application/json" \
-d '{
"text": "My SSN is 123-45-6789 and email is john@example.com",
"threshold": 0.5,
"ai_verify_mode": "suspicious"
}'Response:
json
{
"entities": [
{
"type": "US_SSN",
"text": "123-45-6789",
"start": 10,
"end": 21,
"confidence": 0.99,
"compliance": ["HIPAA", "PCI-DSS"]
},
{
"type": "EMAIL_ADDRESS",
"text": "john@example.com",
"start": 36,
"end": 52,
"confidence": 0.98,
"compliance": ["GDPR", "CCPA"]
}
]
}The ai_verify_mode parameter accepts off, suspicious, or all. When set, entities that fail verification are excluded from the response. See Detection API for all parameters including ai_audit_level and vendor_debug_access.
Update Thresholds
bash
curl -X PUT http://localhost:8000/api/settings/thresholds \
-H "Authorization: Bearer YOUR_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"US_SSN": 0.3,
"PERSON": 0.7
}'Public Figure Reconciliation
When public figure reconciliation is enabled, PERSON entities that match known public or historical figures are not tokenized. This prevents questions like "Who is Albert Einstein?" from being masked to "Who is [PERSON_1]?" — which would make the AI unable to answer.
The resolver uses a three-tier strategy:
- Built-in list — ~5,000 curated public and historical figures
- Wikidata API — real-time lookup against a structured knowledge graph
- LLM fallback — uses the configured model's training knowledge
Defense-in-Depth Safety Filters
Before any reconciliation is attempted, detected PERSON entities pass through multiple safety filters to prevent private individuals from being mistakenly identified as public figures:
| Filter | What It Does | Example |
|---|---|---|
| Co-occurrence Filter | Blocks reconciliation when correlated PII (email, phone, URL) appears near the name | "Sarah Johnson" + sarah.johnson@acme.com → stays masked |
| Multi-Detector Gate | When 2+ detectors agree a name is PII, only the curated builtin list can override | "Bob Thompson" flagged by both Presidio and NER → Wikidata/LLM cannot override |
| Confidence Threshold | Each tier has a minimum score to reconcile | LLM fallback must reach 0.7+ confidence |
These layered filters ensure that common names like "Sarah Johnson" or "James Brown" are not mistakenly passed through just because a Wikipedia entry exists — if correlated PII is nearby or multiple detectors agree, the system errs on the side of protecting privacy.
Reconciled names appear in the UI with a green "Public Figure" badge and are excluded from tokenization. Private person names continue to be detected and tokenized as usual.
Enable this feature in Settings → PII Detection → Public Figure Reconciliation.
Learn More
See the full Public Figure Reconciliation guide for configuration details, safety filter specifics, permission model, edge cases, and API usage.
Public Knowledge Reconciliation
Public Knowledge Reconciliation extends the same idea beyond people. When enabled, broadly-known LOCATION, ORGANIZATION, and EVENT entities (and contextual DATE_TIME) can bypass tokenization so the assistant can answer normal knowledge requests without over-redaction. ORGANIZATION reconciliation uses a curated list of well-known organizations plus optional Wikidata/LLM tiers — not a registry of public companies (e.g. SEC or stock exchange).
Examples:
- "Find restaurants in Utah" (
LOCATION) - "What does NASA do?" (
ORGANIZATION) - "Explain World War II" (
EVENT)
Security guardrails remain in place:
- Builtin-only tier is the default
- Co-occurrence and context filters block private-context reconciliation
- Multi-detector consensus gate prevents weak overrides
- External tiers use pre-filtering, hashed cache keys for unresolved values, and scrubbed logs
See Public Knowledge Reconciliation.
Public Context Passthrough
Public Context Passthrough is a post-reconciliation step that reduces over-redaction for messages containing only public-domain entities and no private data.
How it works:
- After all other reconciliation steps, the passthrough classifier checks if the message contains only public-domain entity types (
LOCATION,ORGANIZATION,EVENT,DATE_TIME) - If any private data (SSN, credit card, medical records, etc.) or
PERSONentities co-occur, full protection is maintained - If the message is purely public-domain, those entities pass through without tokenization
Examples:
| Message | Result |
|---|---|
| "Tell me about the 2024 Olympics in Paris" | Passthrough — only LOCATION and EVENT |
| "John Smith attended the Olympics in Paris" | Protected — PERSON detected |
| "What is the capital of France?" | Passthrough — only LOCATION |
| "My SSN is 123-45-6789, I live in Utah" | Protected — SSN co-occurs |
Each passthrough decision is logged with reconciled_as: public_context_passthrough for audit purposes.
Memory Integration
All memories pass through the PII detection pipeline before storage. This ensures that personal information in memories receives the same protection as chat messages. When a memory is extracted from a conversation, detected PII entities are tokenized using the same multi-layer pipeline (Presidio + NER + AI Verification), and compliance tags are preserved on the stored memory. Organization and group memories inherit the tenant's detection settings, ensuring consistent protection across shared knowledge bases.
Best Practices
- Start with defaults: The 0.5 threshold works well for most cases
- Enable AI verification: Use
suspiciousmode to catch common false positives with minimal latency - Use Private Model for sensitive environments: If PII must never leave the machine, select "Private Model" in Settings → AI Detector tab or set
PII_AI_MODEL=localto use the Private Model (local LLM referee) instead of external AI verification - Review false positives: Adjust per-entity thresholds based on your data, or use AI Verification to filter them automatically
- Enable compliance tagging: Helps with audit requirements
- Use feedback: The system improves with user feedback
- Consider your vertical: Healthcare may need lower thresholds than tech
- Monitor with audit logs: Enable at least
summaryaudit level for compliance visibility
Troubleshooting
For detailed troubleshooting guides, see:
- False Positives — Diagnosing and reducing false positive detections
- Performance — Optimizing detection speed and memory usage
- Common Issues — General troubleshooting
Quick Fixes
Too many false positives:
- Enable AI Verification in
suspiciousmode - Increase the threshold for the problematic entity type
- Check if you're in the right industry vertical
- Use the feedback system to report false positives
Missing PII detection:
- Lower the threshold for that entity type
- Verify the format matches expected patterns
- Check if the entity type is enabled
Slow detection:
- Switch to Parallel execution mode
- Reduce the number of enabled detectors
- Check system memory (models need ~4GB RAM)
Next Steps
- AI Audit Logs — Complete admin guide for audit log management
- Public Figure Reconciliation — Smart handling of public names
- Public Knowledge Reconciliation — Smart handling of public locations, organizations, and events
- Tokenization — How PII is protected
- Code Protection — Secrets and source code
- Image Intelligence — PII detection in images
- Document Preview — Visual redacted previews
- Anti-Exfiltration — Obfuscation detection