Appearance
AI Audit Logs
The AI Audit Log captures every LLM call made by the PII detection and verification pipeline. Use it to monitor AI usage, troubleshoot false positives, and satisfy compliance requirements.
Accessing the AI Audit Log
Navigate to Platform Admin Console > AI Audit tab. The tab is available to platform admins and cloud admins.
Event Types
| Type | Description | Key Fields |
|---|---|---|
| Detection | An LLM was used to detect PII entities in text | entity_count, model, latency_ms |
| Verification | An LLM was used to verify/filter entities found by other detectors | entities_in, entities_out, rejected_count |
| Local Referee | On-device LLM referee judged ambiguous spans | entities_in, entities_out, model (e.g. local:llama-3.1-8b), latency_ms |
Detection Events
Logged when the AI detector (ai_enabled=true) processes text through the LLM. The entity_count field shows how many PII entities the AI found.
Verification Events
Logged when AI verification (ai_verify_mode set to all or suspicious) reviews entities from Presidio or Dymium NER. Key metrics:
- entities_in: Number of entities sent to the LLM for review
- entities_out: Number of entities the LLM confirmed as genuine PII
- rejected_count: Number of false positives filtered out (entities_in - entities_out)
Local Referee Events
Logged when PII_AI_MODEL=local is configured and the on-device LLM referee processes ambiguous spans. The model field uses the format local:<model_key> (e.g. local:llama-3.1-8b).
Detection entries (local mode) contain a descriptive pipeline summary instead of an LLM prompt/response, since local-mode detection is handled by Presidio+NER (no LLM call):
- prompt:
[local-mode] Detection delegated to regex+NER pipeline. Active detectors: Presidio, NER. Threshold: 0.7. AI detector role: verification only. - response:
[no LLM call — local mode uses Presidio+NER for detection, local LLM for verification only]
Verification entries (local mode) contain the actual LLM referee prompt and response at full audit level:
- prompt: The full REFEREE_PROMPT sent to the local LLM, including span context and entity details
- response: The raw LLM output with per-span verdicts (KEEP/REJECT with confidence and reasoning)
- rejected_entities: A list of entities filtered out by the referee, each with
entity_type,text, andscore
Key metrics:
- entities_in: Total entities passed to the referee stage
- entities_out: Entities confirmed by the referee (accepted + high-confidence)
- rejected_count: Number of entities filtered out by the referee
- latency_ms: Total time for gray-band filtering, span batching, and LLM inference
Connector Events
Logged when SaaS connector operations occur (connect, sync, disconnect, webhook, action).
| Type | Description | Key Fields |
|---|---|---|
connector.connected | SaaS tool connected via OAuth | connector_id, provider_account_name, scopes |
connector.disconnected | Connection revoked, data cascade-deleted | connector_id, records_deleted, memories_deleted |
connector.sync_completed | Connector sync job finished | connector_id, records_synced, duration_ms, trigger |
connector.sync_failed | Connector sync job failed | connector_id, error, retry_count |
connector.webhook_received | Inbound webhook processed | connector_id, event_type, records_upserted |
connector.action_executed | AI invoked a connector write action | connector_id, action_name, success |
connector.visibility_changed | Connection access control scope updated | connector_id, old_scope, new_scope |
connector.health_degraded | Connection health status worsened | connector_id, old_status, new_status |
Threat Analysis Events
Security threat analysis operations emit audit events for AI-enriched findings:
| Type | Description | Key Fields |
|---|---|---|
| security_threat_ai_analyzed | AI analyzed a security finding | finding_type, original_risk_level, ai_verdict, ai_confidence, adjusted_risk_level, override_source, prompt_version |
| security_threat_downgraded | AI downgraded a finding's severity | Same as above, plus downgraded: true |
| threat_feedback_submitted | User submitted feedback on AI verdict | finding_type, feedback |
| threat_batch_reanalysis_triggered | Admin triggered batch re-analysis | scope |
Reading Audit Entries
Each event includes these fields:
| Field | Description |
|---|---|
timestamp | When the LLM call occurred |
call_type | detection or verification |
model | LLM model used (e.g., gpt-4o-mini) |
text_hash | SHA-256 hash of the input text (for correlation without exposing content) |
text_length | Character count of the input text |
latency_ms | Round-trip time for the LLM call in milliseconds |
audit_level | The resolved audit level when this event was recorded |
prompt | Raw LLM prompt (only at full audit level with vendor debug access) |
response | Raw LLM response (only at full audit level with vendor debug access) |
rejected_entities | Details of rejected entities (only at full level, verification events) |
Audit Levels
The audit level controls how much data is stored per event:
| Level | Stored Data | Use Case |
|---|---|---|
none | No events recorded | Minimize storage, no compliance need |
summary | Metadata only (model, hash, counts, latency) | Standard compliance monitoring |
full | Metadata + raw LLM prompts and responses | Deep debugging, regulatory audits |
5-Tier Configuration Hierarchy
The effective audit level is resolved through the 5-tier config hierarchy with ceiling and floor enforcement:
Cloud (ceiling/floor) > Tenant > User > Conversation > Subject- Floor (
PII_AI_AUDIT_FLOOR): Minimum audit level that cannot be reduced by lower tiers. Use for compliance mandates (e.g., healthcare requires at leastsummary). - Ceiling (
PII_AI_AUDIT_CEILING): Maximum audit level that lower tiers cannot exceed. Use to limit storage costs or prevent raw text capture.
Example: If the Cloud sets floor=summary and ceiling=summary, all tenants/users will record summary-level audits regardless of their individual settings.
Configure via environment variables or the Settings UI:
| Setting | Default | Description |
|---|---|---|
PII_AI_AUDIT_LEVEL | summary | Base audit level |
PII_AI_AUDIT_FLOOR | none | Minimum enforced level |
PII_AI_AUDIT_CEILING | full | Maximum allowed level |
Vendor Debug Access
Controls debug-level audit detail visibility.
| Setting | Default | Description |
|---|---|---|
PII_VENDOR_DEBUG_ACCESS | false | Controls debug-level audit detail visibility |
PII_VENDOR_DEBUG_ACCESS_ALLOWED | true | Cloud-level gate (must be true for any lower tier to enable) |
When vendor debug is disabled (default):
- Summary fields (model, hash, counts, latency) are always visible
- Detailed fields are stripped from API responses
- The "Show sensitive fields" checkbox in the UI is unchecked
When vendor debug is enabled:
- All detail fields are returned
- Enable only when actively troubleshooting with GhostAI support
- Audit the audit: enabling this itself is logged
Filtering and Pagination
The AI Audit tab supports:
- Type filter: View all events, detection only, or verification only
- Vendor debug toggle: Show/hide sensitive fields
- Pagination: Navigate through results with configurable page size (25, 50, or 100 per page)
- Refresh: Reload the latest events
Querying Audit Data via API
For programmatic access, use the REST API:
List Events
GET /api/audit/ai-events?limit=50&offset=0&call_type=verification&vendor_debug_access=trueParameters:
| Parameter | Type | Default | Description |
|---|---|---|---|
limit | integer | 50 | Results per page (1-500) |
offset | integer | 0 | Pagination offset |
call_type | string | — | Filter by detection or verification |
vendor_debug_access | boolean | false | Include raw text fields |
Get Statistics
GET /api/audit/ai-events/statsReturns aggregate counts and average latency:
json
{
"total": 1234,
"detection_count": 456,
"verification_count": 778,
"avg_latency_ms": 342.5
}Storage Considerations
none: No storage impactsummary: ~200 bytes per event (metadata only)full: 2-10 KB per event depending on text length and number of entities
For high-volume deployments, consider:
- Setting the audit ceiling to
summaryunlessfullis needed for compliance - Implementing a retention policy to purge old events (see Deployment Guide)
- Monitoring database size and scheduling periodic cleanup
RAG Retrieval Metrics
RAG file intelligence operations emit Prometheus metrics that complement the audit log:
| Metric | Description |
|---|---|
ghostai_rag_retrieval_total | Total retrieval operations (by file type and search mode) |
ghostai_rag_retrieval_duration_seconds | End-to-end retrieval latency |
ghostai_rag_chunks_retrieved | Chunks returned per query |
ghostai_rag_indexing_duration_seconds | File indexing duration |
ghostai_rag_pii_cache_hits_total | PII cache hits (indicates caching effectiveness) |
ghostai_rag_pii_cache_misses_total | PII cache misses |
These are available on the /metrics Prometheus endpoint and via OpenTelemetry spans (rag.index, rag.retrieve, rag.bm25, rag.hnsw). See Dashboards — RAG Observability for more.
SIEM Integration
All audit events can now be automatically streamed to external SIEM systems. GhostAI's SIEM integration pushes AI audit events (detection, verification, local referee, threat analysis) along with authentication, file processing, admin actions, and compliance events to your SIEM in real time. Configure integrations, event filters, and delivery settings via the SIEM Configuration guide.
Related Documentation
- PII Detection Features -- Detection pipeline overview
- File Intelligence -- RAG indexing and retrieval details
- Deployment Guide -- Environment variable configuration
- API Reference -- Full API documentation