Skip to content

AI Audit Logs ​

The AI Audit Log captures every LLM call made by the PII detection and verification pipeline. Use it to monitor AI usage, troubleshoot false positives, and satisfy compliance requirements.

Accessing the AI Audit Log ​

Navigate to Platform Admin Console > AI Audit tab. The tab is available to platform admins and cloud admins.

Event Types ​

TypeDescriptionKey Fields
DetectionAn LLM was used to detect PII entities in textentity_count, model, latency_ms
VerificationAn LLM was used to verify/filter entities found by other detectorsentities_in, entities_out, rejected_count
Local RefereeOn-device LLM referee judged ambiguous spansentities_in, entities_out, model (e.g. local:llama-3.1-8b), latency_ms

Detection Events ​

Logged when the AI detector (ai_enabled=true) processes text through the LLM. The entity_count field shows how many PII entities the AI found.

Verification Events ​

Logged when AI verification (ai_verify_mode set to all or suspicious) reviews entities from Presidio or Dymium NER. Key metrics:

  • entities_in: Number of entities sent to the LLM for review
  • entities_out: Number of entities the LLM confirmed as genuine PII
  • rejected_count: Number of false positives filtered out (entities_in - entities_out)

Local Referee Events ​

Logged when PII_AI_MODEL=local is configured and the on-device LLM referee processes ambiguous spans. The model field uses the format local:<model_key> (e.g. local:llama-3.1-8b).

Detection entries (local mode) contain a descriptive pipeline summary instead of an LLM prompt/response, since local-mode detection is handled by Presidio+NER (no LLM call):

  • prompt: [local-mode] Detection delegated to regex+NER pipeline. Active detectors: Presidio, NER. Threshold: 0.7. AI detector role: verification only.
  • response: [no LLM call — local mode uses Presidio+NER for detection, local LLM for verification only]

Verification entries (local mode) contain the actual LLM referee prompt and response at full audit level:

  • prompt: The full REFEREE_PROMPT sent to the local LLM, including span context and entity details
  • response: The raw LLM output with per-span verdicts (KEEP/REJECT with confidence and reasoning)
  • rejected_entities: A list of entities filtered out by the referee, each with entity_type, text, and score

Key metrics:

  • entities_in: Total entities passed to the referee stage
  • entities_out: Entities confirmed by the referee (accepted + high-confidence)
  • rejected_count: Number of entities filtered out by the referee
  • latency_ms: Total time for gray-band filtering, span batching, and LLM inference

Connector Events ​

Logged when SaaS connector operations occur (connect, sync, disconnect, webhook, action).

TypeDescriptionKey Fields
connector.connectedSaaS tool connected via OAuthconnector_id, provider_account_name, scopes
connector.disconnectedConnection revoked, data cascade-deletedconnector_id, records_deleted, memories_deleted
connector.sync_completedConnector sync job finishedconnector_id, records_synced, duration_ms, trigger
connector.sync_failedConnector sync job failedconnector_id, error, retry_count
connector.webhook_receivedInbound webhook processedconnector_id, event_type, records_upserted
connector.action_executedAI invoked a connector write actionconnector_id, action_name, success
connector.visibility_changedConnection access control scope updatedconnector_id, old_scope, new_scope
connector.health_degradedConnection health status worsenedconnector_id, old_status, new_status

Threat Analysis Events ​

Security threat analysis operations emit audit events for AI-enriched findings:

TypeDescriptionKey Fields
security_threat_ai_analyzedAI analyzed a security findingfinding_type, original_risk_level, ai_verdict, ai_confidence, adjusted_risk_level, override_source, prompt_version
security_threat_downgradedAI downgraded a finding's severitySame as above, plus downgraded: true
threat_feedback_submittedUser submitted feedback on AI verdictfinding_type, feedback
threat_batch_reanalysis_triggeredAdmin triggered batch re-analysisscope

Reading Audit Entries ​

Each event includes these fields:

FieldDescription
timestampWhen the LLM call occurred
call_typedetection or verification
modelLLM model used (e.g., gpt-4o-mini)
text_hashSHA-256 hash of the input text (for correlation without exposing content)
text_lengthCharacter count of the input text
latency_msRound-trip time for the LLM call in milliseconds
audit_levelThe resolved audit level when this event was recorded
promptRaw LLM prompt (only at full audit level with vendor debug access)
responseRaw LLM response (only at full audit level with vendor debug access)
rejected_entitiesDetails of rejected entities (only at full level, verification events)

Audit Levels ​

The audit level controls how much data is stored per event:

LevelStored DataUse Case
noneNo events recordedMinimize storage, no compliance need
summaryMetadata only (model, hash, counts, latency)Standard compliance monitoring
fullMetadata + raw LLM prompts and responsesDeep debugging, regulatory audits

5-Tier Configuration Hierarchy ​

The effective audit level is resolved through the 5-tier config hierarchy with ceiling and floor enforcement:

Cloud (ceiling/floor) > Tenant > User > Conversation > Subject
  • Floor (PII_AI_AUDIT_FLOOR): Minimum audit level that cannot be reduced by lower tiers. Use for compliance mandates (e.g., healthcare requires at least summary).
  • Ceiling (PII_AI_AUDIT_CEILING): Maximum audit level that lower tiers cannot exceed. Use to limit storage costs or prevent raw text capture.

Example: If the Cloud sets floor=summary and ceiling=summary, all tenants/users will record summary-level audits regardless of their individual settings.

Configure via environment variables or the Settings UI:

SettingDefaultDescription
PII_AI_AUDIT_LEVELsummaryBase audit level
PII_AI_AUDIT_FLOORnoneMinimum enforced level
PII_AI_AUDIT_CEILINGfullMaximum allowed level

Vendor Debug Access ​

Controls debug-level audit detail visibility.

SettingDefaultDescription
PII_VENDOR_DEBUG_ACCESSfalseControls debug-level audit detail visibility
PII_VENDOR_DEBUG_ACCESS_ALLOWEDtrueCloud-level gate (must be true for any lower tier to enable)

When vendor debug is disabled (default):

  • Summary fields (model, hash, counts, latency) are always visible
  • Detailed fields are stripped from API responses
  • The "Show sensitive fields" checkbox in the UI is unchecked

When vendor debug is enabled:

  • All detail fields are returned
  • Enable only when actively troubleshooting with GhostAI support
  • Audit the audit: enabling this itself is logged

Filtering and Pagination ​

The AI Audit tab supports:

  • Type filter: View all events, detection only, or verification only
  • Vendor debug toggle: Show/hide sensitive fields
  • Pagination: Navigate through results with configurable page size (25, 50, or 100 per page)
  • Refresh: Reload the latest events

Querying Audit Data via API ​

For programmatic access, use the REST API:

List Events ​

GET /api/audit/ai-events?limit=50&offset=0&call_type=verification&vendor_debug_access=true

Parameters:

ParameterTypeDefaultDescription
limitinteger50Results per page (1-500)
offsetinteger0Pagination offset
call_typestring—Filter by detection or verification
vendor_debug_accessbooleanfalseInclude raw text fields

Get Statistics ​

GET /api/audit/ai-events/stats

Returns aggregate counts and average latency:

json
{
  "total": 1234,
  "detection_count": 456,
  "verification_count": 778,
  "avg_latency_ms": 342.5
}

Storage Considerations ​

  • none: No storage impact
  • summary: ~200 bytes per event (metadata only)
  • full: 2-10 KB per event depending on text length and number of entities

For high-volume deployments, consider:

  1. Setting the audit ceiling to summary unless full is needed for compliance
  2. Implementing a retention policy to purge old events (see Deployment Guide)
  3. Monitoring database size and scheduling periodic cleanup

RAG Retrieval Metrics ​

RAG file intelligence operations emit Prometheus metrics that complement the audit log:

MetricDescription
ghostai_rag_retrieval_totalTotal retrieval operations (by file type and search mode)
ghostai_rag_retrieval_duration_secondsEnd-to-end retrieval latency
ghostai_rag_chunks_retrievedChunks returned per query
ghostai_rag_indexing_duration_secondsFile indexing duration
ghostai_rag_pii_cache_hits_totalPII cache hits (indicates caching effectiveness)
ghostai_rag_pii_cache_misses_totalPII cache misses

These are available on the /metrics Prometheus endpoint and via OpenTelemetry spans (rag.index, rag.retrieve, rag.bm25, rag.hnsw). See Dashboards — RAG Observability for more.

SIEM Integration ​

All audit events can now be automatically streamed to external SIEM systems. GhostAI's SIEM integration pushes AI audit events (detection, verification, local referee, threat analysis) along with authentication, file processing, admin actions, and compliance events to your SIEM in real time. Configure integrations, event filters, and delivery settings via the SIEM Configuration guide.

SOC 2 Type II Certified | BAA Available | HIPAA-Ready