Skip to content

Chat Endpoints ​

Send messages to AI with automatic PII protection.

Send Message ​

Send a chat message with PII detection and protection.

POST /api/chat

Request ​

json
{
  "message": "My SSN is 123-45-6789. Can you help?",
  "session_id": "sess_abc123",
  "model": "gpt-4",
  "settings": {
    "threshold": 0.5,
    "tokenize": true,
    "stream": false
  }
}
FieldTypeRequiredDescription
messagestringYesUser message
session_idstringNoExisting session ID
modelstringNoLLM model to use
routestringNoPreferred route source (e.g. openai-direct, openrouter, ollama). Falls back to primary provider if route unavailable.
settings.thresholdnumberNoPII detection threshold
settings.tokenizebooleanNoEnable tokenization
settings.streambooleanNoEnable streaming
custom_system_promptstringNoPer-session custom system instructions (max 50,000 chars). PII-tokenized before injection.
assistant_idstringNoLoad instructions/config from a saved AI Assistant
max_tokensintegerNoOverride the model's max output tokens (1–65536)
exclude_entity_spansarrayNoPer-message override. Spans to exclude from PII redaction for this request only. Each item: { "start": number, "end": number, "entity_type": string, "source"?: string }. source is "message" for main text or "attachment:filename" for attachment content. High-risk types (SSN, credit card, etc.) are never excluded. Requires org policy pii_allow_per_message_override.
add_to_safe_terms_and_sendstring[]NoPer-message override. Terms to add to the user's Safe Terms and send unredacted this message. Deduplicated and normalized. Requires org policy pii_allow_per_message_override.

Response ​

json
{
  "response": "Based on your SSN 123-45-6789, here's my advice...",
  "session_id": "sess_abc123",
  "message_id": "msg_xyz789",
  "pii_detected": 1,
  "entities": [
    {
      "type": "US_SSN",
      "original": "123-45-6789",
      "token": "[US_SSN_a1b2c3]",
      "confidence": 0.99,
      "start": 10,
      "end": 21
    }
  ],
  "usage": {
    "prompt_tokens": 25,
    "completion_tokens": 150,
    "total_tokens": 175
  }
}

When the organization disables per-message overrides but the client sends exclude_entity_spans or add_to_safe_terms_and_send, the response includes pii_override_warning with a message directing users to add safe terms in Settings.

The actual route used for the request is logged in response metadata and diagnostic traces.

Chat UI rendering: Assistant responses are rendered as markdown in the chat UI. Fenced code blocks (e.g. ```python) are syntax-highlighted client-side (Prism, One Dark theme) with copy, download, and language label. Generated PDFs use server-side Pygments for code highlighting.

Example ​

bash
curl -X POST https://api.ghostai.com/v1/chat \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "message": "My name is John Smith and my email is john@example.com",
    "settings": {
      "threshold": 0.5
    }
  }'

Enhance Prompt ​

Enhance a rough prompt with detail, structure, and industry-specific context.

POST /api/chat/enhance

Request ​

json
{
  "message": "explain this lab result",
  "session_id": "sess_abc123"
}
FieldTypeRequiredDescription
messagestringYesRaw user prompt to enhance (1-10000 chars)
session_idstringNoCurrent session ID for conversation context

Response ​

json
{
  "enhanced": "As a clinical professional, I need you to interpret the following lab result. Explain each biomarker's significance, flag any values outside normal ranges, suggest potential differential diagnoses, and recommend follow-up tests. Present in a structured clinical note format.",
  "original": "explain this lab result",
  "model": "gpt-4o-mini"
}
FieldTypeDescription
enhancedstringThe context-aware enhanced prompt
originalstringThe original input (for undo)
modelstringModel used for enhancement

Context Used ​

The enhancement gathers context automatically from the authenticated user's session:

  • Industry vertical from organization configuration
  • Custom instructions (about_user, response_style)
  • Matter context (legal vertical: matter name, practice area)
  • Recent messages from the current conversation (last 6 messages)

Example ​

bash
curl -X POST https://api.ghostai.com/api/chat/enhance \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "message": "analyze these transactions",
    "session_id": "sess_abc123"
  }'

Streaming ​

Enable streaming for real-time responses.

POST /api/chat/stream

Request ​

json
{
  "message": "Tell me a story",
  "session_id": "sess_abc123",
  "stream": true,
  "route": "openai-direct"
}
FieldTypeRequiredDescription
messagestringYesUser message
session_idstringNoExisting session ID
streambooleanNoEnable streaming
routestringNoPreferred route source (e.g. openai-direct, openrouter, ollama). Falls back to primary provider if route unavailable.
exclude_entity_spansarrayNoPer-message override: spans to exclude from PII redaction for this request only. Same shape as non-streaming.
add_to_safe_terms_and_sendstring[]NoPer-message override: terms to add to Safe Terms and send unredacted. Same as non-streaming.

Response (Server-Sent Events) ​

data: {"type": "start", "session_id": "sess_abc123"}

data: {"type": "token", "content": "Once"}

data: {"type": "token", "content": " upon"}

data: {"type": "token", "content": " a"}

data: {"type": "token", "content": " time"}

data: {"type": "pii", "entity": {"type": "PERSON", "token": "[PERSON_x1]"}}

data: {"type": "tool_start", "tool_name": "web_search", "tool_id": "call_abc123"}

data: {"type": "tool_result", "tool_name": "web_search", "tool_id": "call_abc123", "result_preview": "Found 5 results..."}

data: {"type": "end", "usage": {"total_tokens": 50}, "time_to_first_token_ms": 250, "llm_total_time_ms": 3200, "performance": {"pii_detection_ms": 12.3, "tokenization_ms": 0.8, "prompt_build_ms": 1.2, "llm_ttft_ms": 250.0, "llm_streaming_ms": 2950.0, "tool_execution_ms": 0.0, "detokenization_ms": 0.5, "db_save_ms": 3.1, "total_ms": 3218.0}}

Public Figure Reconciliation in start Event ​

When public figure reconciliation is enabled, the start event includes an optional reconciled_public_figures array. Each entry describes a PERSON entity that was identified as a public figure and excluded from PII masking. Note: entities blocked by safety filters (co-occurrence filter, multi-detector consensus gate) will NOT appear here — they remain masked as PII.

data: {"type": "start", "session_id": "sess_abc123", "reconciled_public_figures": [{"entity_type": "PERSON", "text": "Albert Einstein", "start": 7, "end": 22, "reconciled_as": "Albert Einstein", "reconciled_source": "builtin", "reconciled_category": "scientist", "reconciled_confidence": 0.99}]}
FieldTypeDescription
entity_typestringAlways "PERSON"
textstringOriginal text span from the user message
startnumberStart character offset
endnumberEnd character offset
reconciled_asstringCanonical name of the matched public figure
reconciled_sourcestringResolution tier: "builtin", "wikidata", or "llm"
reconciled_categorystringCategory (e.g. "scientist", "politician")
reconciled_confidencenumberConfidence score (0-1)

Example: A message like "Who is Albert Einstein?" with public figure reconciliation enabled produces a start event containing:

json
{
  "type": "start",
  "session_id": "sess_abc123",
  "reconciled_public_figures": [
    {
      "entity_type": "PERSON",
      "text": "Albert Einstein",
      "start": 7,
      "end": 22,
      "reconciled_as": "Albert Einstein",
      "reconciled_source": "builtin",
      "reconciled_category": "scientist",
      "reconciled_confidence": 0.99
    }
  ]
}

The name "Albert Einstein" is not masked or tokenized — it passes through to the LLM as-is. See Detection Endpoints for full field descriptions.

PII Override Warning in start Event ​

When the organization disables per-message overrides (pii_allow_per_message_override = false) but the client sends exclude_entity_spans or add_to_safe_terms_and_send, the start event includes pii_override_warning (string). The client should display this message and not apply overrides for that request.

Public Knowledge Reconciliation in start Event ​

When public knowledge reconciliation is enabled, start can also include reconciled_public_knowledge entries for entities like LOCATION, ORGANIZATION, and EVENT.

json
{
  "type": "start",
  "session_id": "sess_abc123",
  "reconciled_public_knowledge": [
    {
      "entity_type": "LOCATION",
      "text": "Utah",
      "start": 25,
      "end": 30,
      "reconciled_as": "public_knowledge",
      "reconciled_source": "builtin",
      "reconciled_category": "us_state_or_territory",
      "reconciled_confidence": 1.0
    }
  ]
}

Performance Breakdown (Latency Waterfall) ​

The done event includes a performance object with granular timing for each phase of request processing. This powers the user-facing Performance Waterfall in the chat UI.

FieldDescriptionCategory
pii_detection_msTime to detect PII entities in the user messageGhostAI
tokenization_msTime to tokenize (replace) detected PIIGhostAI
prompt_build_msTime to build system prompt, custom instructions, tool awarenessGhostAI
llm_ttft_msTime from LLM request to first token receivedExternal LLM
llm_streaming_msTime spent streaming tokens (total LLM time minus TTFT)External LLM
tool_execution_msTotal time spent executing tools (web search, image gen)External
detokenization_msTime to restore PII tokens in the responseGhostAI
db_save_msTime to persist messages to the databaseGhostAI
total_msEnd-to-end request time from the server perspectiveTotal

Tool Execution Events ​

When the AI invokes tools (web search, image generation), additional SSE events are emitted:

Event TypeDescription
tool_startTool execution has begun, includes tool_name and tool_id
tool_resultTool finished, includes result_preview with a summary

Supported tools:

  • web_search — Real-time web search (Tavily, Brave, or Serper)
  • generate_image — Image generation via DALL-E 3
  • generate_pdf — PDF generation from markdown content
  • create_canvas — Generate interactive HTML canvas artifacts (CSP-sandboxed, PII-scanned)
  • update_canvas — Update existing canvas artifacts with version tracking

generate_pdf ​

Generate a professional PDF document from rich markdown content. Supports world-class rendering of mermaid diagrams, LaTeX math, styled tables, syntax-highlighted code, embedded images, blockquotes, and clickable links — all with graceful fallback if any element fails to render. Heavy assets (mermaid, math, images, code highlighting) are rendered in parallel for improved response time on asset-heavy documents.

Parameters:

ParameterTypeRequiredDescription
titlestringYesDocument title
contentstringYesRich markdown content. Supports: mermaid diagrams (```mermaid blocks rendered via mermaid.ink API), LaTeX math (block $$...$$ and inline $...$ rendered via matplotlib), pipe-delimited tables (with styled branded headers), syntax-highlighted code blocks (```lang rendered via Pygments), image embedding (URLs including GhostAI-generated images), blockquotes (with left border styling), and clickable links ([text](url)). All rich elements fall back gracefully if rendering fails.
templatestringNoTemplate: briefing, report, memo, summary (default: report)
page_sizestringNoPage size: letter, a4 (default: letter)
ocr_textbooleanNoAdd searchable text layer (default: false)

SSE Response:

json
{
  "type": "tool_result",
  "tool_name": "generate_pdf",
  "pdf_url": "/api/generated-pdfs/Report_a1b2c3d4.pdf",
  "filename": "Report_a1b2c3d4.pdf",
  "pages": 3,
  "contains_pii": false,
  "result_preview": "PDF generated: Report_a1b2c3d4.pdf"
}

When the PDF contains detokenized PII (zero-trust secure mode), contains_pii is true.

PDFs are served as static files at /api/generated-pdfs/.

Example (JavaScript) ​

javascript
const response = await fetch('https://api.ghostai.com/v1/chat/stream', {
  method: 'POST',
  headers: {
    'Authorization': 'Bearer YOUR_TOKEN',
    'Content-Type': 'application/json'
  },
  body: JSON.stringify({
    message: 'Tell me a story',
    stream: true
  })
});

const reader = response.body.getReader();
const decoder = new TextDecoder();

while (true) {
  const { done, value } = await reader.read();
  if (done) break;
  
  const chunk = decoder.decode(value);
  const lines = chunk.split('\n');
  
  for (const line of lines) {
    if (line.startsWith('data: ')) {
      const data = JSON.parse(line.slice(6));
      console.log(data);
    }
  }
}

Sessions ​

List Sessions ​

GET /api/sessions

Query parameters:

  • page - Page number (default: 1)
  • limit - Results per page (default: 20)

Response:

json
{
  "sessions": [
    {
      "id": "sess_abc123",
      "title": "Tax Questions",
      "created_at": "2024-01-15T10:30:00Z",
      "last_accessed": "2024-01-15T11:00:00Z",
      "message_count": 5,
      "pii_count": 3
    }
  ],
  "pagination": {
    "page": 1,
    "limit": 20,
    "total": 45
  }
}

Get Session ​

GET /api/sessions/{session_id}

Response:

json
{
  "id": "sess_abc123",
  "title": "Tax Questions",
  "created_at": "2024-01-15T10:30:00Z",
  "messages": [
    {
      "id": "msg_1",
      "role": "user",
      "content": "My SSN is [US_SSN_a1b2]...",
      "timestamp": "2024-01-15T10:30:00Z",
      "pii_entities": [...]
    },
    {
      "id": "msg_2",
      "role": "assistant",
      "content": "Based on your information...",
      "timestamp": "2024-01-15T10:30:05Z"
    }
  ],
  "token_count": 3
}

Delete Session ​

DELETE /api/sessions/{session_id}

Response:

json
{
  "deleted": true,
  "tokens_cleared": 5
}

Get Session Tokens ​

View token mappings for a session.

GET /api/sessions/{session_id}/tokens

Response:

json
{
  "session_id": "sess_abc123",
  "tokens": [
    {
      "token": "[US_SSN_a1b2]",
      "type": "US_SSN",
      "created_at": "2024-01-15T10:30:00Z"
    },
    {
      "token": "[PERSON_c3d4]",
      "type": "PERSON",
      "created_at": "2024-01-15T10:30:00Z"
    }
  ],
  "count": 2
}

WARNING

Original values are not returned for security. Use detokenize endpoint with session context.

Conversation Management ​

Rename Conversation ​

PUT /api/conversations/{session_id}

Request:

json
{
  "title": "My Custom Title"
}

Generate Smart Title ​

AI-generated title based on conversation content.

POST /api/conversations/{conversation_id}/generate-title

Request:

json
{
  "current_title": "My first message text..."
}

Response:

json
{
  "title": "Tax Planning Questions"
}

Custom Instructions ​

Get Custom Instructions ​

GET /api/settings/custom-instructions

Response:

json
{
  "about_user": "I'm a healthcare compliance officer...",
  "response_style": "Be concise and use bullet points",
  "enabled": true
}

Update Custom Instructions ​

PUT /api/settings/custom-instructions

Request:

json
{
  "about_user": "I'm a healthcare compliance officer at a 500-bed hospital",
  "response_style": "Be concise, use bullet points, cite regulations",
  "enabled": true
}

Custom instructions are injected into the system prompt for all subsequent conversations when enabled.

Chat Flow Debugger (Admin) ​

Diagnostic endpoints for platform admins to inspect the full lifecycle of chat requests. Requires is_platform_admin=true.

List Diagnostic Traces ​

GET /api/admin/chat-debug/traces

Query parameters:

ParameterTypeDefaultDescription
limitint50Max traces to return (1-200)
session_idstring-Filter by session ID
user_idstring-Filter by user ID
has_errorbool-Filter to traces with/without errors
has_tool_callsbool-Filter to traces with/without tool calls

Response:

json
{
  "traces": [
    {
      "trace_id": "abc-123",
      "session_id": "sess-456",
      "user_id": "user-789",
      "started_at": 1738900000.0,
      "model": "openai/gpt-4o-mini",
      "provider": "openrouter",
      "message_preview": "What is the latest AI news?",
      "duration_ms": 3200.5,
      "event_count": 10,
      "has_tool_calls": true,
      "has_fallback": false,
      "has_error": false,
      "status": "success",
      "tools_available": ["web_search", "generate_image", "create_canvas", "update_canvas"],
      "tools_enabled": ["web_search"],
      "finished": true
    }
  ],
  "count": 1
}

Get Trace Detail ​

GET /api/admin/chat-debug/traces/{trace_id}

Returns the full trace including all diagnostic events:

json
{
  "trace_id": "abc-123",
  "session_id": "sess-456",
  "events": [
    {
      "timestamp_ms": 0.0,
      "wall_time": 1738900000.0,
      "event_type": "pii_detect",
      "detail": { "entity_count": 3, "detection_time_ms": 12.5 }
    },
    {
      "timestamp_ms": 12.5,
      "wall_time": 1738900000.012,
      "event_type": "pii_tokenize",
      "detail": { "token_count": 3 }
    },
    {
      "timestamp_ms": 13.0,
      "wall_time": 1738900000.013,
      "event_type": "tools_resolved",
      "detail": { "available": ["web_search"], "final": ["web_search"], "tool_count": 1 }
    },
    {
      "timestamp_ms": 14.0,
      "wall_time": 1738900000.014,
      "event_type": "llm_request",
      "detail": { "model": "openai/gpt-4o-mini", "provider": "openrouter", "tool_count": 1 }
    },
    {
      "timestamp_ms": 850.0,
      "wall_time": 1738900000.85,
      "event_type": "llm_first_token",
      "detail": { "ttft_ms": 836.0, "tool_round": 0 }
    }
  ]
}

Event Types ​

EventDescription
pii_detectPII detection completed (entity count, timing)
pii_tokenizePII tokenization completed (token count)
tools_resolvedTool availability resolved after user filtering
llm_requestLLM API call initiated (model, provider, tools, message count)
llm_first_tokenFirst streaming token received (TTFT)
tool_callLLM requested a tool call (tool name, arguments)
tool_resultTool execution completed (result preview, timing)
fallback_no_toolsProvider rejected tools, retried without (error, stripped tools)
detokenizePII detokenization completed (response length)
llm_doneStream completed (total time, response length)
errorAn error occurred (message, type)

Clear All Traces ​

DELETE /api/admin/chat-debug/traces

Response:

json
{
  "cleared": 42,
  "message": "Cleared 42 traces"
}

File Pipeline Diagnostics ​

Diagnostic endpoint for monitoring worker health and file processing configuration. No authentication required (designed for health checks and monitoring).

Get Pipeline Status ​

GET /health/file-pipeline

Response:

json
{
  "worker_pid": 12345,
  "peak_rss_mb": 842.3,
  "pii_ready": true,
  "uvicorn_workers": "4",
  "pii_max_concurrent": 4,
  "rag_threshold_mb": 1.0,
  "rag_embedding_model": "all-MiniLM-L6-v2",
  "rag_embedding_device": "cpu",
  "rag_hybrid_search_enabled": true,
  "rag_chunk_size_chars": 2000,
  "rag_chunk_overlap_chars": 200,
  "rag_tabular_chunk_rows": 50,
  "rag_ann_threshold": 100000
}
FieldTypeDescription
worker_pidintOS process ID of the responding worker
peak_rss_mbfloatPeak resident set size (memory high-water mark) in MB
pii_readyboolWhether PII detection models are loaded and warm
uvicorn_workersstringNumber of configured Gunicorn/Uvicorn workers
pii_max_concurrentintMax concurrent PII detection operations
rag_threshold_mbfloatFile size threshold (MB) for RAG indexing
rag_embedding_modelstringSentence-transformer model used for embeddings
rag_embedding_devicestringDevice for embedding inference (cpu, cuda, mps)
rag_hybrid_search_enabledboolWhether BM25 + semantic fusion is active
rag_chunk_size_charsintCharacters per text chunk
rag_chunk_overlap_charsintOverlap between consecutive chunks
rag_tabular_chunk_rowsintRows per chunk for CSV/spreadsheet files
rag_ann_thresholdintSwitch to HNSW approximate search above this chunk count

Use this endpoint to:

  • Monitor worker memory pressure before and after file processing
  • Verify PII models are loaded after deployment
  • Confirm RAG configuration and embedding device match expectations
  • Check worker configuration matches expectations
bash
# Check worker memory
curl -s http://localhost:8000/health/file-pipeline | python3 -m json.tool

Ollama Model Management (Platform Admin) ​

Manage Ollama models from the API. All endpoints require is_platform_admin=true.

List Installed Ollama Models ​

GET /api/admin/ollama/models

Response:

json
{
  "models": [
    {
      "name": "llama3.1:8b",
      "size": 4661224676,
      "parameter_size": "8.0B",
      "quantization_level": "Q4_0",
      "modified_at": "2026-02-20T10:30:00Z",
      "digest": "a1b2c3d4..."
    }
  ]
}

Pull an Ollama Model ​

Pull a model from the Ollama library. Streams NDJSON progress events.

POST /api/admin/ollama/pull

Request:

json
{
  "model": "deepseek-r1:14b"
}
FieldTypeRequiredDescription
modelstringYesModel name to pull (e.g. llama3.1:8b, deepseek-r1:14b)

Response (NDJSON stream):

{"status":"pulling manifest"}
{"status":"downloading","digest":"sha256:abc...","total":4661224676,"completed":1024000}
{"status":"downloading","digest":"sha256:abc...","total":4661224676,"completed":2048000}
{"status":"verifying sha256 digest"}
{"status":"writing manifest"}
{"status":"success"}

Delete an Ollama Model ​

DELETE /api/admin/ollama/models

Request:

json
{
  "model": "llama3.1:8b"
}
FieldTypeRequiredDescription
modelstringYesName of the installed model to delete

Response:

json
{
  "status": "success",
  "deleted": "llama3.1:8b"
}

Returns 404 if the model is not installed. Returns 403 if the caller is not a platform admin.


Message Feedback ​

Submit and retrieve per-message feedback (thumbs up/down). Feedback is persisted in the message_feedback database table.

Submit Feedback ​

POST /api/chat/messages/{message_id}/feedback

Request:

json
{
  "vote": "up"
}
FieldTypeRequiredDescription
votestringYes"up" or "down"

Response:

json
{
  "message_id": "msg_xyz789",
  "vote": "up",
  "created_at": "2025-02-07T15:30:00Z"
}

Get Feedback ​

GET /api/chat/messages/{message_id}/feedback

Response:

json
{
  "message_id": "msg_xyz789",
  "vote": "up",
  "created_at": "2025-02-07T15:30:00Z"
}

Returns 404 if no feedback has been submitted for the message.

Example ​

bash
# Submit thumbs-up feedback
curl -X POST https://api.ghostai.com/api/chat/messages/msg_xyz789/feedback \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"vote": "up"}'

# Retrieve feedback
curl https://api.ghostai.com/api/chat/messages/msg_xyz789/feedback \
  -H "Authorization: Bearer YOUR_TOKEN"

Combined Answer (Multi-Model Deliberation) ​

Run a 3-stage deliberation pipeline that produces one clean answer from multiple models.

Combine Endpoint ​

POST /api/chat/combine

Triggers the full 3-stage pipeline (Initial → Deliberation → Blend) and streams the result via SSE.

Request ​

json
{
  "message": "Explain the pros and cons of microservices",
  "session_id": "sess_abc123",
  "models": ["openai/gpt-4o", "anthropic/claude-3.5-sonnet", "google/gemini-pro"],
  "blend_model": "openai/gpt-4o",
  "settings": {
    "threshold": 0.5,
    "tokenize": true
  }
}
FieldTypeRequiredDescription
messagestringYesUser prompt
session_idstringNoExisting session ID
modelsstring[]Yes2-4 model IDs to use for initial responses and deliberation
blend_modelstringNoModel to use for the final blend stage (defaults to first model)
settings.thresholdnumberNoPII detection threshold
settings.tokenizebooleanNoEnable tokenization

Response (Server-Sent Events) ​

data: {"type": "stage", "stage": "initial", "model": "openai/gpt-4o", "status": "streaming"}

data: {"type": "token", "stage": "initial", "model": "openai/gpt-4o", "content": "Micro"}

data: {"type": "stage", "stage": "initial", "model": "openai/gpt-4o", "status": "done"}

data: {"type": "stage", "stage": "deliberation", "model": "openai/gpt-4o", "status": "streaming"}

data: {"type": "token", "stage": "deliberation", "model": "openai/gpt-4o", "content": "After"}

data: {"type": "stage", "stage": "deliberation", "model": "openai/gpt-4o", "status": "done"}

data: {"type": "stage", "stage": "blend", "status": "streaming"}

data: {"type": "token", "stage": "blend", "content": "When"}

data: {"type": "done", "combined_text": "When considering microservices...", "usage": {"prompt_tokens": 850, "completion_tokens": 600, "total_tokens": 1450}, "stages_completed": ["initial", "deliberation", "blend"]}

SSE Event Types ​

Event TypeStageDescription
stageallStage lifecycle event — includes stage name, optional model, and status (streaming or done)
tokenallStreaming content token — includes stage, optional model, and content
piiallPII entity detected during any stage
done—Pipeline complete — includes combined_text, usage, and stages_completed
erroranyError during pipeline — includes message, stage, and optional model

Combined Mode on Streaming Endpoint ​

The existing streaming endpoint also supports combined mode via additional fields.

POST /api/chat/stream

Additional Request Fields ​

json
{
  "message": "Explain the pros and cons of microservices",
  "session_id": "sess_abc123",
  "stream": true,
  "combined_mode": true,
  "combined_models": ["openai/gpt-4o", "anthropic/claude-3.5-sonnet", "google/gemini-pro"]
}
FieldTypeRequiredDescription
combined_modebooleanNoEnable combined answer mode (3-stage pipeline). Default: false
combined_modelsstring[]No2-4 model IDs for the pipeline. Required when combined_mode is true

When combined_mode is true, the response SSE stream includes the same stage and token events as the /api/chat/combine endpoint. The final done event includes combined_text with the blended answer.

Example ​

bash
# Full combined answer pipeline
curl -N -X POST https://api.ghostai.com/api/chat/combine \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "message": "Explain the pros and cons of microservices",
    "models": ["openai/gpt-4o", "anthropic/claude-3.5-sonnet"],
    "blend_model": "openai/gpt-4o"
  }'
bash
# Combined mode via streaming endpoint
curl -N -X POST https://api.ghostai.com/api/chat/stream \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "message": "Explain the pros and cons of microservices",
    "stream": true,
    "combined_mode": true,
    "combined_models": ["openai/gpt-4o", "anthropic/claude-3.5-sonnet"]
  }'

Analytics (Admin) ​

Admin endpoints for viewing aggregated feedback and performance metrics. Requires admin privileges.

Vote Summary ​

Aggregated thumbs up/down counts across all messages.

GET /api/admin/analytics/votes

Query parameters:

ParameterTypeDefaultDescription
start_datestring-ISO 8601 start date filter
end_datestring-ISO 8601 end date filter

Response:

json
{
  "total_votes": 1250,
  "thumbs_up": 980,
  "thumbs_down": 270,
  "approval_rate": 0.784
}

Feedback Details ​

Detailed feedback records with filtering support.

GET /api/admin/analytics/feedback

Query parameters:

ParameterTypeDefaultDescription
start_datestring-ISO 8601 start date filter
end_datestring-ISO 8601 end date filter
votestring-Filter by vote type ("up" or "down")
user_idstring-Filter by user ID
limitint50Max records to return
offsetint0Pagination offset

Response:

json
{
  "feedback": [
    {
      "message_id": "msg_xyz789",
      "user_id": "user_123",
      "vote": "up",
      "model": "gpt-4o",
      "created_at": "2025-02-07T15:30:00Z"
    }
  ],
  "total": 1250,
  "limit": 50,
  "offset": 0
}

Latency Analytics ​

Historical performance metrics from the performance_metrics database table.

GET /api/admin/analytics/latency

Query parameters:

ParameterTypeDefaultDescription
start_datestring-ISO 8601 start date filter
end_datestring-ISO 8601 end date filter
modelstring-Filter by model name
granularitystringdayAggregation granularity (hour, day, week)

Response:

json
{
  "metrics": [
    {
      "period": "2025-02-07",
      "request_count": 450,
      "avg_total_ms": 3200.5,
      "p50_total_ms": 2800.0,
      "p95_total_ms": 5400.0,
      "avg_llm_ttft_ms": 850.0,
      "avg_pii_detection_ms": 12.3,
      "avg_ghostai_overhead_ms": 18.5
    }
  ],
  "summary": {
    "total_requests": 450,
    "avg_total_ms": 3200.5,
    "avg_ghostai_overhead_pct": 0.58
  }
}

File Attachments & RAG Retrieval ​

When a chat message includes file attachments (attachment_ids), the chat pipeline automatically integrates RAG retrieval for indexed files.

How RAG Works in Chat ​

  1. Single indexed file: The user's message is used as a semantic query. Relevant chunks are retrieved via hybrid BM25+semantic search (with RRF merge) and injected into the LLM context, replacing the full file text.

  2. Multiple indexed files: Cross-file retrieval activates automatically. Chunks from all attached files are scored together against the query, with a shared token budget. Results include source attribution per file.

  3. PII protection: Retrieved chunks pass through the full PII detection and tokenization pipeline before reaching the LLM. Chunk-level PII caching avoids redundant detection when the same chunks are retrieved across queries.

  4. Token-budget-aware: The number of chunks injected adapts to the model's context window — more for Gemini (1M tokens), fewer for GPT-4 (8K tokens). The system allocates 50% of the context window for file content.

Request Example ​

bash
curl -X POST https://api.ghostai.com/api/chat \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "message": "What are the top revenue customers?",
    "session_id": "sess_abc123",
    "attachment_ids": ["file_xyz789"],
    "model": "gpt-4o"
  }'

The response will be based on the most relevant chunks from the attached file, with all PII tokenized.


Compare Models ​

Compare responses from multiple LLMs side-by-side with the same PII-protected prompt.

POST /api/chat/compare

Request ​

FieldTypeRequiredDescription
messagestringYesUser message
modelsstring[]Yes1-4 model IDs to compare
session_idstringNoExisting session ID
attachmentsstring[]NoFile attachment IDs
enabled_toolsstring[]NoTools to enable (e.g. web_search, execute_python)
tool_modestringNoper_model (default) or shared
custom_system_promptstringNoPer-session system instructions (max 50K chars)
assistant_idstringNoLoad config from a saved AI Assistant
max_tokensintegerNoOverride max output tokens per model
max_tool_roundsintegerNoOverride max tool execution rounds (1-15)
memory_enabledbooleanNoEnable/disable memory recall

SSE Event Types ​

EventDescription
compare_processingConnection keepalive emitted immediately and every ~5s during PII detection. Fields: session_id, models, status (scanning, preparing_attachments). Prevents proxy timeouts on large documents.
compare_startSession initialized with PII detection results
compare_chunkStreaming content chunk from a model
compare_tool_startA model initiated a tool call
compare_tool_resultTool execution completed
compare_doneModel finished with performance metrics
compare_all_doneAll models complete

Large Document Handling ​

Comparison mode includes optimizations for large file attachments:

  • Lower RAG threshold: Files >= 10KB are automatically indexed for RAG retrieval in comparison mode (vs 1MB in regular chat), so only relevant chunks are sent to each model instead of the full document.
  • Async PII with heartbeats: PII detection runs as a background task with compare_processing heartbeat events emitted every 5 seconds, keeping the SSE connection alive through proxies.
  • PII scan caching: If a file was previously scanned via POST /api/files/{file_id}/scan, cached PII results are reused during chat, eliminating redundant detection.

Code Interpreter ​

The execute_python tool runs Python code in an isolated sandbox with no network access. Uploaded files are available at /workspace/. Output is PII-scanned before reaching the LLM.

The sandbox supports two pluggable backends:

  • Docker — For local dev and docker-compose deployments. Hardened with read-only filesystem, tmpfs mounts, pids_limit=100, no-new-privileges, restricted builtins, and import allowlist.
  • Lambda — For AWS ECS Fargate and Kubernetes deployments. Uses AWS Lambda with container images, S3 for file transfer, minimal IAM, and VPC isolation.

Security is enforced at multiple layers: tool_validator.py blocks eval/exec/compile/importlib at pre-execution validation; the sandbox entrypoint restricts builtins and only allows whitelisted imports. The feature is gated behind billing (Team plan+), has per-user rate limiting (default 20/min), and supports gradual rollout.

Sandbox Environment ​

  • Libraries: pandas, numpy, matplotlib, seaborn, scipy, nltk, textblob, scikit-learn, beautifulsoup4, lxml
  • Limits: 2 CPU cores, 4GB RAM, 120s timeout
  • Output: stdout/stderr captured, artifacts from /workspace/output/ collected

Configuration ​

VariableDefaultDescription
SANDBOX_ENABLEDtrueMaster toggle
SANDBOX_BACKENDdockerBackend: docker (local dev) or lambda (AWS/Kubernetes)
SANDBOX_IMAGEghostai-sandbox:latestDocker image (Docker backend)
SANDBOX_LAMBDA_FUNCTION—Lambda function name (Lambda backend)
SANDBOX_S3_BUCKET—S3 bucket for file transfer (Lambda backend)
SANDBOX_RATE_LIMIT20Per-user executions per minute
SANDBOX_TIMEOUT120Execution timeout (seconds)
SANDBOX_CPU_LIMIT2.0CPU cores per container (Docker backend)
SANDBOX_MEMORY_LIMIT4gRAM per container (Docker backend)
SANDBOX_MAX_CONCURRENT5Max parallel containers (Docker backend)

Prometheus metrics are emitted with a backend label; CloudWatch alarms and Grafana dashboards are available for Lambda deployments.


AI Assistants ​

Reusable AI assistant configurations with custom instructions, knowledge files, and default settings.

GET    /api/assistants          # List all (own + shared)
POST   /api/assistants          # Create new assistant
GET    /api/assistants/:id      # Get by ID
PUT    /api/assistants/:id      # Update
DELETE /api/assistants/:id      # Delete
POST   /api/assistants/:id/duplicate  # Clone

Assistant Schema ​

FieldTypeDescription
namestringDisplay name (required)
descriptionstringBrief description
instructionsstringSystem instructions (max 50K chars)
knowledge_file_idsstring[]Attached knowledge file IDs
starter_messagestringPre-filled message (max 10K chars)
max_tokensintegerOutput token limit override
max_tool_roundsintegerTool round limit override (1-15)
default_modelsstring[]Default model selection
is_sharedbooleanShare with organization
iconstringDisplay icon/emoji

SOC 2 Type II Certified | BAA Available | HIPAA-Ready