Skip to content

Reversible Tokenization ​

GhostAI uses session-scoped reversible tokenization to protect sensitive data while maintaining full AI functionality. Sensitive values are replaced with secure, opaque tokens before reaching any AI provider, and original values are restored in responses delivered back to the user.

How It Works ​

When a user sends a message containing sensitive data, GhostAI intercepts it and replaces each detected entity with a unique token. The AI model receives only tokens — never the original values. When the AI responds, GhostAI detects the tokens in the response and restores the original data before presenting it to the user.

This means the AI provider never sees real PII, financial data, or other protected information, yet the conversation remains coherent and useful.

Key Concepts ​

Session Scoping ​

Tokens are unique to each conversation session:

  • The same sensitive value maps to the same token within a single session, ensuring consistent AI responses.
  • Different sessions produce different tokens, preventing cross-session data leakage.
  • Tokens expire automatically when the session ends.

Public Figure Exclusion ​

When public figure reconciliation is enabled, names that are resolved as public or historical figures (e.g., "Albert Einstein", "Marie Curie") are excluded from tokenization. These names pass through to the AI model untouched, ensuring accurate responses to questions about well-known people.

Only names identified as private individuals are tokenized.

Safety Filters

Not every public figure match results in exclusion from tokenization. GhostAI applies multiple safety filters before allowing a name to pass through. These filters guard against cases where a private individual shares a name with a public figure, especially when correlated personal data appears nearby.

TIP

Public figure exclusion does not affect other entity types. Financial data, contact information, and all non-person PII are always tokenized regardless of this setting.

Public Knowledge Exclusion ​

When public knowledge reconciliation is enabled, selected well-known entities such as locations, organizations, and events can also be excluded from tokenization. This keeps common public references readable while still protecting private data.

Examples of entities that may pass through untokenized:

  • Well-known locations (e.g., "Utah")
  • Recognized organizations (e.g., "NASA")
  • Historical events (e.g., "World War II")

These exclusions are still subject to safety filters to prevent accidental exposure of private data.

Reversibility ​

Unlike anonymization or redaction, tokenization is fully reversible:

ApproachReversibleAI UsableCompliant
RedactionNoNoYes
AnonymizationNoLimitedYes
TokenizationYesYesYes

Token Vault ​

Storage ​

Original values are stored in a secure, encrypted vault. GhostAI supports multiple vault backends to meet different deployment requirements:

  • In-Memory — optimized for speed in single-instance deployments
  • Database — persistent storage across application restarts
  • Cloud Storage — distributed storage for multi-server environments
  • Customer-Managed — bring-your-own-key (BYOK) storage for enterprise deployments

Retention ​

Administrators can configure token retention policies to balance compliance requirements with storage considerations. Options range from session-scoped (tokens are discarded when the session ends) to extended retention windows for audit and compliance purposes. Custom retention periods are available for enterprise deployments.

Encryption ​

Token-to-value mappings are encrypted at rest and in transit. Enterprise customers can use their own encryption keys (BYOK) for full control over key management.

Detokenization ​

When the AI responds with tokens, GhostAI automatically:

  1. Scans the response for tokens
  2. Looks up original values in the vault
  3. Replaces each token with its original value
  4. Validates that all tokens were successfully resolved

The user sees a natural response with their original data restored.

Consistency Features ​

Reference Tracking ​

GhostAI maintains entity consistency across different references within a conversation. If the AI refers to the same person by name, title, or pronoun, the system maps these references to the same underlying token so detokenization produces correct results.

Multi-Value Handling ​

When multiple entities of the same type appear in a conversation, each unique value receives its own distinct token. This ensures accurate resolution even in complex, multi-entity conversations.

API Endpoints ​

GhostAI exposes REST API endpoints for programmatic tokenization and detokenization:

  • POST /api/tokenize — tokenize text containing sensitive data
  • POST /api/detokenize — restore original values from tokenized text
  • GET /api/sessions/{session_id}/tokens — retrieve token metadata for a session

All API requests require authentication.

File Type Support ​

Tokenization works across multiple content types, including plain text, PDFs, and images. When processing documents such as PDFs, sensitive data is tokenized before reaching the AI model and restored in the final rendered output. This ensures consistent protection regardless of the input format.

Security ​

Cryptographic Token Integrity ​

Each token includes a cryptographic signature derived from a server-side secret. During detokenization, this signature is verified before any vault lookup occurs. Tokens that fail verification — including any fabricated by the AI model — are rejected and left unresolved, preventing unauthorized data access.

Token Leakage Prevention ​

  • Tokens are opaque and meaningless without vault access
  • Vault access requires authentication
  • Tokens expire automatically based on retention policy

Audit Trail ​

Every tokenization and detokenization event is logged, including timestamp, session identifier, entity type, and action performed. This audit trail supports compliance reporting and incident investigation.

Compliance ​

Tokenization supports regulatory compliance across multiple frameworks:

  • HIPAA — protected health information never leaves your environment in original form
  • PCI-DSS — payment card data is tokenized before AI processing
  • GDPR — personal data is protected throughout the AI interaction
  • SOC 2 — full audit trail maintained for all tokenization events

Best Practices ​

  1. Use session-scoped tokens — avoid sharing tokens across sessions
  2. Set appropriate retention — balance compliance needs with storage considerations
  3. Enable audit logging — required for most compliance frameworks
  4. Encrypt the token vault — always enable vault encryption in production
  5. Monitor token activity — unusual token volumes may indicate misconfiguration

Troubleshooting ​

Tokens Not Resolving ​

  • Verify the session ID is correct
  • Confirm the token has not expired
  • Ensure the vault backend is accessible

Performance ​

  • Use in-memory vault for lowest latency
  • Reduce retention periods to limit vault size
  • Schedule periodic token cleanup for long-running deployments

Next Steps ​

SOC 2 Type II Certified | BAA Available | HIPAA-Ready