Skip to content

File Pipeline Troubleshooting ​

This guide covers diagnosing failures in the file upload, PII scan, RAG indexing, and chat stream pipeline. For general issues see Common Issues. For PII tuning see Performance.

Quick Reference: Symptoms and Root Causes ​

SymptomLikely CauseSection
"AI service temporarily unavailable"Worker OOM during chat stream (502)Worker OOM
"Retrying... attempt 2 of 2" then failureWorker crash mid-stream, frontend retries exhaustedWorker OOM
File shows "Redacted" but LLM gives generic responseRAG indexing failed or 0-chunk retrievalRAG Indexing Failures
"Scanning PII" stuck foreverScan timeout or backend unhealthyScan Timeout
Upload spinner never completesNetwork, auth, file size, or backend downUpload Failures
Chat works for small files but fails for large PDFsPyMuPDF memory spike exceeds worker budgetLarge File Memory
"Response was not JSON" (502)ALB/CloudFront timeout or worker crashGateway Timeouts

Using the File Pipeline Debugger ​

The File Pipeline Debugger in Platform Admin Console provides end-to-end diagnostics across 7 phases. Access it from Platform Admin Console > Upload Test.

Diagnostic Workflow ​

Run the debugger against the problematic file (or a similar file) and follow the phases:

Phase 0 -- Health Pre-flight

Calls /health, /health/ready, and /health/pii-model before starting. If any check fails, the test aborts with a clear message. Common failures:

  • /health/ready returns 503: PII models still loading. Wait for warmup to complete.
  • /health shows database unhealthy: Check database connectivity.
  • /health/pii-model shows loaded: false: NER model failed to download or ran out of memory.

Phase 1 -- Upload

Compresses (if applicable) and uploads the file. Tracks upload progress and duration. If this fails, the issue is network-level (auth, CORS, file size limits).

Phase 2 -- RAG Indexing Poll

Polls GET /api/files/{id}/metadata every 2 seconds (up to 30s) watching for is_indexed to transition from false to true. Displays the final chunk_count. Also shows RAG configuration (embedding model, chunk sizes, hybrid search flag) from the /health/file-pipeline pre-flight call.

  • If is_indexed never becomes true: RAG indexing crashed. Check backend logs for RAG indexing failed.
  • If chunk_count is 0: Text extraction produced no content (encrypted PDF, scanned images without OCR).
  • Check that rag_embedding_device is appropriate for your deployment (CPU vs GPU).

Phase 3 -- PII Scan + Text Preview

Runs the PII scan and fetches the file preview. Shows:

  • Extracted text length and first 500 characters (confirms extraction worked)
  • Sanitized text preview (confirms PII tokenization)
  • Security findings (macros, phishing URLs, JavaScript, steganography)
  • Entity count and types detected

If text length is 0 for a document file, text extraction failed.

Phase 4 -- Chat Stream Test

Sends a test message (default: "Summarize this file", configurable) with the attachment to POST /api/chat/stream. Captures every SSE event:

  • heartbeat -- confirms connection is alive during PII processing
  • token -- first token time (TTFT) and total token count
  • error -- full error payload from the backend
  • done -- total stream duration

Also displays RAG retrieval metrics from the backend logs: chunks retrieved, top similarity score, search mode (semantic/hybrid/hnsw), and whether the 0-chunk DB fallback was triggered.

If the stream returns a 502 or drops with no events, the worker crashed (OOM).

Phase 5 -- Chat Diagnostic Trace

Fetches the per-request trace from /api/admin/chat-debug/traces for the test session. Shows timing breakdown:

  • PII detection (ms)
  • Attachment preparation (ms) -- file count and filenames
  • RAG retrieval -- chunks retrieved, top score, search mode, budget utilization, fallback status
  • LLM call (ms)
  • Tool execution (ms)
  • Total duration (ms)

If the trace shows no events or is missing, the worker died before the trace could be recorded. See RAG Retrieval Details for field interpretation.

Phase 6 -- Memory Delta

Calls /health/file-pipeline before and after the test. Compares peak_rss_mb:

  • Delta < 50 MB: Normal
  • Delta 50-100 MB: Elevated, monitor for larger files
  • Delta 100-300 MB: High, OOM risk with concurrent requests
  • Delta > 300 MB: Critical, likely to cause SIGKILL under load

Error Classification ​

The debugger automatically classifies failures with color-coded badges:

BadgeHTTP StatusMeaningTypical Fix
OOM502Worker ran out of memory and was killedReduce file size, increase worker memory, or check for PyMuPDF memory spikes
Timeout504Request exceeded ALB/CloudFront timeoutIncrease origin_read_timeout, check for missing heartbeats
Unavailable503Backend not ready or overloadedWait for PII model warmup, check worker count
Rate Limit429LLM provider rate limitWait and retry, or switch to a different provider/model
Auth Error401/403Authentication or authorization failureCheck session cookie, API key, or admin permissions
Scan Timeout--PII scan exceeded 30sFile is very large; text cap and progressive scan should handle this
SSE Error--Backend sent an error event in the streamRead the error message for specifics

Common Failure Patterns ​

Worker OOM During Chat Stream ​

Symptoms: "AI service temporarily unavailable", 502 errors, "Retrying..." messages, backend logs show Worker (pid:XX) was sent SIGKILL! Perhaps out of memory?

Root Cause: The chat stream endpoint processes file attachments (text extraction, PII detection, redaction) within the request handler. For large image-heavy PDFs, PyMuPDF loads the entire document into memory (10-50x file size) causing the worker to exceed its memory limit.

Specific triggers:

  1. RAG 0-chunk fallback: When semantic retrieval returns 0 matching chunks (common with vague queries like "what is"), the code falls back to storage.get_text_content() which re-extracts the full PDF via PyMuPDF.
  2. Redundant PDF redaction: If create_redacted_pdf is called when a redacted version already exists, PyMuPDF loads the document again unnecessarily.
  3. Unbounded PII text input: Without a text cap, the full extracted text (potentially millions of characters) is fed to NER models.
  4. High concurrency: Multiple files processed in parallel (ThreadPoolExecutor with too many workers) multiplies peak memory.

Fixes:

  • The RAG fallback now reads pre-indexed chunks from the database instead of re-extracting
  • A 5MB file size guard prevents PyMuPDF from loading very large files during chat
  • Text input to PII models is capped at 200K characters
  • Redaction checks is_redacted() before re-processing
  • ThreadPoolExecutor reduced to 2 concurrent workers
  • Explicit gc.collect() calls after heavy operations

RAG Indexing Failures ​

Symptoms: File uploads successfully, shows "Redacted" and threat count, but the LLM gives a generic "Could you share more context?" response.

Diagnosis:

  1. Check is_indexed and chunk_count in the File Pipeline Debugger (Phase 2)
  2. If is_indexed is false: RAG indexing crashed or never ran. Check backend logs.
  3. If chunk_count is 0: Text extraction produced no usable content.
  4. If chunks exist but LLM ignores them: The chat stream may have failed before reaching the LLM.

Fixes:

  • Ensure the file exceeds the RAG threshold (default: 1 MB) to trigger indexing
  • For encrypted PDFs, provide the password or convert to an unencrypted format
  • For scanned documents, ensure OCR is enabled and Tesseract is installed

Scan Timeout ​

Symptoms: "Scanning PII" indicator stays visible indefinitely, or the File Pipeline Debugger shows scan phase as "Timed out" after 30 seconds.

Diagnosis:

  1. Check backend health (/health/ready) -- if PII models aren't loaded, scans will hang waiting
  2. Check file size -- very large files (>10MB text content) take longer to scan
  3. Check backend logs for Python exceptions during scan

Fixes:

  • Wait for PII model warmup to complete (check /health/ready)
  • The scan text cap (200K chars) limits processing time for very large files
  • Progressive scanning returns partial results while the full scan continues
  • The frontend now clears the "Scanning PII" indicator after 30s timeout

Upload Failures ​

Symptoms: Upload spinner never completes, "Upload failed" error.

Diagnosis:

  1. Check the Environment panel in the File Pipeline Debugger -- confirms API base, auth, file limits
  2. Check browser network tab for the actual HTTP error
  3. Check backend logs for file validation errors

Common causes:

  • File exceeds MAX_FILE_UPLOAD_SIZE (default: 50 MB)
  • File extension not in allowed list
  • No authentication cookie (session expired)
  • CORS misconfiguration
  • Backend not reachable (wrong API base URL)

Gateway Timeouts ​

Symptoms: "Response was not JSON" error, HTTP 502 or 504.

Causes:

  • CloudFront origin_read_timeout (default: 60s): If the backend doesn't send any data within 60 seconds, CloudFront drops the connection. The chat stream now sends heartbeat SSE events every 10 seconds to prevent this.
  • ALB idle timeout (default: 300s): Long-running streams can be cut off. Increase to match the longest expected request.
  • Worker crash: If the worker dies mid-request, the ALB returns 502.

Fixes:

  • Verify heartbeat events appear in the SSE event log (Phase 4 of the debugger)
  • Check CloudFront origin_read_timeout in Terraform config
  • Check ALB idle timeout in AWS console or Terraform
  • If 502 with no heartbeats, the worker crashed before sending any response -- see Worker OOM

Large File Memory Budget ​

Understanding memory usage helps predict OOM risk:

OperationMemory UsageDuration
PDF text extraction (PyMuPDF)10-50x file size (images loaded into RAM)1-10s
Office text extraction2-5x file size0.5-3s
Plain text loading1x file size<0.1s
PII NER model (Dymium NER)~800 MB resident (shared across requests)N/A
PII detection on text~2x text size (intermediate buffers)0.2-1s
RAG chunk DB readNegligible (SQL query)<0.1s
RAG semantic retrieval~50 MB (embedding model)0.1-0.5s
PDF redaction (PyMuPDF)10-50x file size (full document reload)2-15s

Example: 9.2 MB image-heavy PDF

  • PyMuPDF extraction: 200-500 MB peak
  • PII detection: ~5 MB (200K char cap)
  • RAG chunk read: <1 MB
  • Total with models: 700-1000 MB per worker

Worker budget: With 4 Gunicorn workers on a 16 GB ECS task, each worker gets roughly 3-4 GB after OS and model overhead. A single 500 MB spike is safe; two concurrent spikes can trigger SIGKILL.

Backend Diagnostic Endpoints ​

EndpointAuthPurpose
GET /healthNoneFull system health (DB, LLM, PII, provider keys)
GET /health/liveNoneLiveness probe (process running)
GET /health/readyNoneReadiness probe (PII models loaded)
GET /health/pii-modelNonePII model detail (name, runtime, loaded, platform)
GET /health/file-pipelineNoneWorker memory, PII config, RAG status
GET /api/admin/chat-debug/tracesPlatform AdminPer-request diagnostic traces
GET /api/admin/chat-debug/traces/{id}Platform AdminFull trace with all pipeline events

/health/file-pipeline Response ​

json
{
  "worker_pid": 12345,
  "peak_rss_mb": 842.3,
  "pii_ready": true,
  "uvicorn_workers": "4",
  "pii_max_concurrent": 4,
  "rag_threshold_mb": 1.0,
  "rag_embedding_model": "all-MiniLM-L6-v2",
  "rag_embedding_device": "cpu",
  "rag_hybrid_search_enabled": true,
  "rag_chunk_size_chars": 2000,
  "rag_chunk_overlap_chars": 200,
  "rag_tabular_chunk_rows": 50,
  "rag_ann_threshold": 100000
}

RAG Retrieval Details ​

RAG Configuration Reference ​

The /health/file-pipeline endpoint reports the RAG system configuration:

FieldDescriptionDefault
rag_threshold_mbFiles above this size are RAG-indexed1.0 MB
rag_embedding_modelSentence-transformer model for embeddingsall-MiniLM-L6-v2
rag_embedding_deviceDevice for embedding inference (cpu, cuda, mps)cpu
rag_hybrid_search_enabledWhether BM25 + semantic search fusion is activetrue
rag_chunk_size_charsCharacters per text chunk2000
rag_chunk_overlap_charsOverlap between consecutive chunks200
rag_tabular_chunk_rowsRows per chunk for CSV/spreadsheet files50
rag_ann_thresholdSwitch to HNSW approximate search above this chunk count100,000

Interpreting RAG Retrieval in Chat Traces ​

The chat diagnostic trace includes a rag_retrieval event emitted during _process_one_file. Check the trace detail in Phase 5 of the debugger (or via /api/admin/chat-debug/traces/{id}):

FieldMeaning
chunks_retrievedNumber of chunks selected for the LLM context
total_chunksTotal chunks indexed for the file
top_scoreHighest similarity/relevance score (0.0-1.0)
search_modeSearch strategy used: semantic, hybrid, hnsw, hnsw+hybrid
budget_utilization_pctFraction of the character budget consumed by selected chunks
duration_msTime spent on retrieval

Understanding 0-Chunk Fallback ​

When semantic retrieval returns 0 chunks (common with very short or vague queries like "what is"):

  1. Current behavior: The system reads all stored chunk texts directly from the database (no PyMuPDF re-extraction). This is memory-safe because it reads only pre-indexed text, never the raw PDF.
  2. Previous behavior (caused OOM): Fell back to storage.get_text_content() which re-extracted the full PDF via PyMuPDF, causing 200-500 MB memory spikes.
  3. If no chunks exist in DB: A 5 MB file-size guard prevents any get_text_content() call for large files, returning a user-friendly message instead.

Quick Diagnostic Commands ​

bash
# Check if backend is ready
curl -s http://localhost:8000/health/ready | python3 -m json.tool

# Check PII model status
curl -s http://localhost:8000/health/pii-model | python3 -m json.tool

# Check worker memory
curl -s http://localhost:8000/health/file-pipeline | python3 -m json.tool

# List recent chat traces with errors
curl -s "http://localhost:8000/api/admin/chat-debug/traces?has_error=true&limit=10"

# Check file metadata (is_indexed, chunk_count)
curl -s http://localhost:8000/api/files/{file_id}/metadata -b "session=..."

# Check full system health
curl -s http://localhost:8000/health | python3 -m json.tool

Escalation Checklist ​

If the File Pipeline Debugger cannot resolve the issue, collect the following before escalating:

  1. Export debugger results -- "Copy as JSON" from the File Pipeline Debugger
  2. File details -- name, size, content type, number of pages (for PDFs)
  3. Health endpoint output -- /health, /health/file-pipeline, /health/pii-model
  4. Backend logs -- last 100 lines around the failure timestamp
  5. Chat trace -- if available, the full trace JSON from /api/admin/chat-debug/traces/{id}
  6. Environment -- deployment type (Docker, ECS, Kubernetes), worker count, memory allocation

SOC 2 Type II Certified | BAA Available | HIPAA-Ready