Appearance
Reversible Tokenization
GhostAI uses session-scoped reversible tokenization to protect sensitive data while maintaining full AI functionality. Sensitive values are replaced with secure, opaque tokens before reaching any AI provider, and original values are restored in responses delivered back to the user.
How It Works
When a user sends a message containing sensitive data, GhostAI intercepts it and replaces each detected entity with a unique token. The AI model receives only tokens — never the original values. When the AI responds, GhostAI detects the tokens in the response and restores the original data before presenting it to the user.
This means the AI provider never sees real PII, financial data, or other protected information, yet the conversation remains coherent and useful.
Key Concepts
Session Scoping
Tokens are unique to each conversation session:
- The same sensitive value maps to the same token within a single session, ensuring consistent AI responses.
- Different sessions produce different tokens, preventing cross-session data leakage.
- Tokens expire automatically when the session ends.
Public Figure Exclusion
When public figure reconciliation is enabled, names that are resolved as public or historical figures (e.g., "Albert Einstein", "Marie Curie") are excluded from tokenization. These names pass through to the AI model untouched, ensuring accurate responses to questions about well-known people.
Only names identified as private individuals are tokenized.
Safety Filters
Not every public figure match results in exclusion from tokenization. GhostAI applies multiple safety filters before allowing a name to pass through. These filters guard against cases where a private individual shares a name with a public figure, especially when correlated personal data appears nearby.
TIP
Public figure exclusion does not affect other entity types. Financial data, contact information, and all non-person PII are always tokenized regardless of this setting.
Public Knowledge Exclusion
When public knowledge reconciliation is enabled, selected well-known entities such as locations, organizations, and events can also be excluded from tokenization. This keeps common public references readable while still protecting private data.
Examples of entities that may pass through untokenized:
- Well-known locations (e.g., "Utah")
- Recognized organizations (e.g., "NASA")
- Historical events (e.g., "World War II")
These exclusions are still subject to safety filters to prevent accidental exposure of private data.
Reversibility
Unlike anonymization or redaction, tokenization is fully reversible:
| Approach | Reversible | AI Usable | Compliant |
|---|---|---|---|
| Redaction | No | No | Yes |
| Anonymization | No | Limited | Yes |
| Tokenization | Yes | Yes | Yes |
Token Vault
Storage
Original values are stored in a secure, encrypted vault. GhostAI supports multiple vault backends to meet different deployment requirements:
- In-Memory — optimized for speed in single-instance deployments
- Database — persistent storage across application restarts
- Cloud Storage — distributed storage for multi-server environments
- Customer-Managed — bring-your-own-key (BYOK) storage for enterprise deployments
Retention
Administrators can configure token retention policies to balance compliance requirements with storage considerations. Options range from session-scoped (tokens are discarded when the session ends) to extended retention windows for audit and compliance purposes. Custom retention periods are available for enterprise deployments.
Encryption
Token-to-value mappings are encrypted at rest and in transit. Enterprise customers can use their own encryption keys (BYOK) for full control over key management.
Detokenization
When the AI responds with tokens, GhostAI automatically:
- Scans the response for tokens
- Looks up original values in the vault
- Replaces each token with its original value
- Validates that all tokens were successfully resolved
The user sees a natural response with their original data restored.
Consistency Features
Reference Tracking
GhostAI maintains entity consistency across different references within a conversation. If the AI refers to the same person by name, title, or pronoun, the system maps these references to the same underlying token so detokenization produces correct results.
Multi-Value Handling
When multiple entities of the same type appear in a conversation, each unique value receives its own distinct token. This ensures accurate resolution even in complex, multi-entity conversations.
API Endpoints
GhostAI exposes REST API endpoints for programmatic tokenization and detokenization:
POST /api/tokenize— tokenize text containing sensitive dataPOST /api/detokenize— restore original values from tokenized textGET /api/sessions/{session_id}/tokens— retrieve token metadata for a session
All API requests require authentication.
File Type Support
Tokenization works across multiple content types, including plain text, PDFs, and images. When processing documents such as PDFs, sensitive data is tokenized before reaching the AI model and restored in the final rendered output. This ensures consistent protection regardless of the input format.
Security
Cryptographic Token Integrity
Each token includes a cryptographic signature derived from a server-side secret. During detokenization, this signature is verified before any vault lookup occurs. Tokens that fail verification — including any fabricated by the AI model — are rejected and left unresolved, preventing unauthorized data access.
Token Leakage Prevention
- Tokens are opaque and meaningless without vault access
- Vault access requires authentication
- Tokens expire automatically based on retention policy
Audit Trail
Every tokenization and detokenization event is logged, including timestamp, session identifier, entity type, and action performed. This audit trail supports compliance reporting and incident investigation.
Compliance
Tokenization supports regulatory compliance across multiple frameworks:
- HIPAA — protected health information never leaves your environment in original form
- PCI-DSS — payment card data is tokenized before AI processing
- GDPR — personal data is protected throughout the AI interaction
- SOC 2 — full audit trail maintained for all tokenization events
Best Practices
- Use session-scoped tokens — avoid sharing tokens across sessions
- Set appropriate retention — balance compliance needs with storage considerations
- Enable audit logging — required for most compliance frameworks
- Encrypt the token vault — always enable vault encryption in production
- Monitor token activity — unusual token volumes may indicate misconfiguration
Troubleshooting
Tokens Not Resolving
- Verify the session ID is correct
- Confirm the token has not expired
- Ensure the vault backend is accessible
Performance
- Use in-memory vault for lowest latency
- Reduce retention periods to limit vault size
- Schedule periodic token cleanup for long-running deployments
Next Steps
- PII Detection — what gets tokenized
- Enterprise Vault — advanced storage options
- Compliance Reports — audit documentation