Appearance
File Pipeline Troubleshooting
This guide covers diagnosing failures in the file upload, PII scan, RAG indexing, and chat stream pipeline. For general issues see Common Issues. For PII tuning see Performance.
Quick Reference: Symptoms and Root Causes
| Symptom | Likely Cause | Section |
|---|---|---|
| "AI service temporarily unavailable" | Worker OOM during chat stream (502) | Worker OOM |
| "Retrying... attempt 2 of 2" then failure | Worker crash mid-stream, frontend retries exhausted | Worker OOM |
| File shows "Redacted" but LLM gives generic response | RAG indexing failed or 0-chunk retrieval | RAG Indexing Failures |
| "Scanning PII" stuck forever | Scan timeout or backend unhealthy | Scan Timeout |
| Upload spinner never completes | Network, auth, file size, or backend down | Upload Failures |
| Chat works for small files but fails for large PDFs | PyMuPDF memory spike exceeds worker budget | Large File Memory |
| "Response was not JSON" (502) | ALB/CloudFront timeout or worker crash | Gateway Timeouts |
Using the File Pipeline Debugger
The File Pipeline Debugger in Platform Admin Console provides end-to-end diagnostics across 7 phases. Access it from Platform Admin Console > Upload Test.
Diagnostic Workflow
Run the debugger against the problematic file (or a similar file) and follow the phases:
Phase 0 -- Health Pre-flight
Calls /health, /health/ready, and /health/pii-model before starting. If any check fails, the test aborts with a clear message. Common failures:
/health/readyreturns 503: PII models still loading. Wait for warmup to complete./healthshows database unhealthy: Check database connectivity./health/pii-modelshowsloaded: false: NER model failed to download or ran out of memory.
Phase 1 -- Upload
Compresses (if applicable) and uploads the file. Tracks upload progress and duration. If this fails, the issue is network-level (auth, CORS, file size limits).
Phase 2 -- RAG Indexing Poll
Polls GET /api/files/{id}/metadata every 2 seconds (up to 30s) watching for is_indexed to transition from false to true. Displays the final chunk_count. Also shows RAG configuration (embedding model, chunk sizes, hybrid search flag) from the /health/file-pipeline pre-flight call.
- If
is_indexednever becomestrue: RAG indexing crashed. Check backend logs forRAG indexing failed. - If
chunk_countis 0: Text extraction produced no content (encrypted PDF, scanned images without OCR). - Check that
rag_embedding_deviceis appropriate for your deployment (CPU vs GPU).
Phase 3 -- PII Scan + Text Preview
Runs the PII scan and fetches the file preview. Shows:
- Extracted text length and first 500 characters (confirms extraction worked)
- Sanitized text preview (confirms PII tokenization)
- Security findings (macros, phishing URLs, JavaScript, steganography)
- Entity count and types detected
If text length is 0 for a document file, text extraction failed.
Phase 4 -- Chat Stream Test
Sends a test message (default: "Summarize this file", configurable) with the attachment to POST /api/chat/stream. Captures every SSE event:
heartbeat-- confirms connection is alive during PII processingtoken-- first token time (TTFT) and total token counterror-- full error payload from the backenddone-- total stream duration
Also displays RAG retrieval metrics from the backend logs: chunks retrieved, top similarity score, search mode (semantic/hybrid/hnsw), and whether the 0-chunk DB fallback was triggered.
If the stream returns a 502 or drops with no events, the worker crashed (OOM).
Phase 5 -- Chat Diagnostic Trace
Fetches the per-request trace from /api/admin/chat-debug/traces for the test session. Shows timing breakdown:
- PII detection (ms)
- Attachment preparation (ms) -- file count and filenames
- RAG retrieval -- chunks retrieved, top score, search mode, budget utilization, fallback status
- LLM call (ms)
- Tool execution (ms)
- Total duration (ms)
If the trace shows no events or is missing, the worker died before the trace could be recorded. See RAG Retrieval Details for field interpretation.
Phase 6 -- Memory Delta
Calls /health/file-pipeline before and after the test. Compares peak_rss_mb:
- Delta < 50 MB: Normal
- Delta 50-100 MB: Elevated, monitor for larger files
- Delta 100-300 MB: High, OOM risk with concurrent requests
- Delta > 300 MB: Critical, likely to cause SIGKILL under load
Error Classification
The debugger automatically classifies failures with color-coded badges:
| Badge | HTTP Status | Meaning | Typical Fix |
|---|---|---|---|
| OOM | 502 | Worker ran out of memory and was killed | Reduce file size, increase worker memory, or check for PyMuPDF memory spikes |
| Timeout | 504 | Request exceeded ALB/CloudFront timeout | Increase origin_read_timeout, check for missing heartbeats |
| Unavailable | 503 | Backend not ready or overloaded | Wait for PII model warmup, check worker count |
| Rate Limit | 429 | LLM provider rate limit | Wait and retry, or switch to a different provider/model |
| Auth Error | 401/403 | Authentication or authorization failure | Check session cookie, API key, or admin permissions |
| Scan Timeout | -- | PII scan exceeded 30s | File is very large; text cap and progressive scan should handle this |
| SSE Error | -- | Backend sent an error event in the stream | Read the error message for specifics |
Common Failure Patterns
Worker OOM During Chat Stream
Symptoms: "AI service temporarily unavailable", 502 errors, "Retrying..." messages, backend logs show Worker (pid:XX) was sent SIGKILL! Perhaps out of memory?
Root Cause: The chat stream endpoint processes file attachments (text extraction, PII detection, redaction) within the request handler. For large image-heavy PDFs, PyMuPDF loads the entire document into memory (10-50x file size) causing the worker to exceed its memory limit.
Specific triggers:
- RAG 0-chunk fallback: When semantic retrieval returns 0 matching chunks (common with vague queries like "what is"), the code falls back to
storage.get_text_content()which re-extracts the full PDF via PyMuPDF. - Redundant PDF redaction: If
create_redacted_pdfis called when a redacted version already exists, PyMuPDF loads the document again unnecessarily. - Unbounded PII text input: Without a text cap, the full extracted text (potentially millions of characters) is fed to NER models.
- High concurrency: Multiple files processed in parallel (
ThreadPoolExecutorwith too many workers) multiplies peak memory.
Fixes:
- The RAG fallback now reads pre-indexed chunks from the database instead of re-extracting
- A 5MB file size guard prevents PyMuPDF from loading very large files during chat
- Text input to PII models is capped at 200K characters
- Redaction checks
is_redacted()before re-processing - ThreadPoolExecutor reduced to 2 concurrent workers
- Explicit
gc.collect()calls after heavy operations
RAG Indexing Failures
Symptoms: File uploads successfully, shows "Redacted" and threat count, but the LLM gives a generic "Could you share more context?" response.
Diagnosis:
- Check
is_indexedandchunk_countin the File Pipeline Debugger (Phase 2) - If
is_indexedis false: RAG indexing crashed or never ran. Check backend logs. - If
chunk_countis 0: Text extraction produced no usable content. - If chunks exist but LLM ignores them: The chat stream may have failed before reaching the LLM.
Fixes:
- Ensure the file exceeds the RAG threshold (default: 1 MB) to trigger indexing
- For encrypted PDFs, provide the password or convert to an unencrypted format
- For scanned documents, ensure OCR is enabled and Tesseract is installed
Scan Timeout
Symptoms: "Scanning PII" indicator stays visible indefinitely, or the File Pipeline Debugger shows scan phase as "Timed out" after 30 seconds.
Diagnosis:
- Check backend health (
/health/ready) -- if PII models aren't loaded, scans will hang waiting - Check file size -- very large files (>10MB text content) take longer to scan
- Check backend logs for Python exceptions during scan
Fixes:
- Wait for PII model warmup to complete (check
/health/ready) - The scan text cap (200K chars) limits processing time for very large files
- Progressive scanning returns partial results while the full scan continues
- The frontend now clears the "Scanning PII" indicator after 30s timeout
Upload Failures
Symptoms: Upload spinner never completes, "Upload failed" error.
Diagnosis:
- Check the Environment panel in the File Pipeline Debugger -- confirms API base, auth, file limits
- Check browser network tab for the actual HTTP error
- Check backend logs for file validation errors
Common causes:
- File exceeds
MAX_FILE_UPLOAD_SIZE(default: 50 MB) - File extension not in allowed list
- No authentication cookie (session expired)
- CORS misconfiguration
- Backend not reachable (wrong API base URL)
Gateway Timeouts
Symptoms: "Response was not JSON" error, HTTP 502 or 504.
Causes:
- CloudFront
origin_read_timeout(default: 60s): If the backend doesn't send any data within 60 seconds, CloudFront drops the connection. The chat stream now sends heartbeat SSE events every 10 seconds to prevent this. - ALB idle timeout (default: 300s): Long-running streams can be cut off. Increase to match the longest expected request.
- Worker crash: If the worker dies mid-request, the ALB returns 502.
Fixes:
- Verify heartbeat events appear in the SSE event log (Phase 4 of the debugger)
- Check CloudFront
origin_read_timeoutin Terraform config - Check ALB idle timeout in AWS console or Terraform
- If 502 with no heartbeats, the worker crashed before sending any response -- see Worker OOM
Large File Memory Budget
Understanding memory usage helps predict OOM risk:
| Operation | Memory Usage | Duration |
|---|---|---|
| PDF text extraction (PyMuPDF) | 10-50x file size (images loaded into RAM) | 1-10s |
| Office text extraction | 2-5x file size | 0.5-3s |
| Plain text loading | 1x file size | <0.1s |
| PII NER model (Dymium NER) | ~800 MB resident (shared across requests) | N/A |
| PII detection on text | ~2x text size (intermediate buffers) | 0.2-1s |
| RAG chunk DB read | Negligible (SQL query) | <0.1s |
| RAG semantic retrieval | ~50 MB (embedding model) | 0.1-0.5s |
| PDF redaction (PyMuPDF) | 10-50x file size (full document reload) | 2-15s |
Example: 9.2 MB image-heavy PDF
- PyMuPDF extraction: 200-500 MB peak
- PII detection: ~5 MB (200K char cap)
- RAG chunk read: <1 MB
- Total with models: 700-1000 MB per worker
Worker budget: With 4 Gunicorn workers on a 16 GB ECS task, each worker gets roughly 3-4 GB after OS and model overhead. A single 500 MB spike is safe; two concurrent spikes can trigger SIGKILL.
Backend Diagnostic Endpoints
| Endpoint | Auth | Purpose |
|---|---|---|
GET /health | None | Full system health (DB, LLM, PII, provider keys) |
GET /health/live | None | Liveness probe (process running) |
GET /health/ready | None | Readiness probe (PII models loaded) |
GET /health/pii-model | None | PII model detail (name, runtime, loaded, platform) |
GET /health/file-pipeline | None | Worker memory, PII config, RAG status |
GET /api/admin/chat-debug/traces | Platform Admin | Per-request diagnostic traces |
GET /api/admin/chat-debug/traces/{id} | Platform Admin | Full trace with all pipeline events |
/health/file-pipeline Response
json
{
"worker_pid": 12345,
"peak_rss_mb": 842.3,
"pii_ready": true,
"uvicorn_workers": "4",
"pii_max_concurrent": 4,
"rag_threshold_mb": 1.0,
"rag_embedding_model": "all-MiniLM-L6-v2",
"rag_embedding_device": "cpu",
"rag_hybrid_search_enabled": true,
"rag_chunk_size_chars": 2000,
"rag_chunk_overlap_chars": 200,
"rag_tabular_chunk_rows": 50,
"rag_ann_threshold": 100000
}RAG Retrieval Details
RAG Configuration Reference
The /health/file-pipeline endpoint reports the RAG system configuration:
| Field | Description | Default |
|---|---|---|
rag_threshold_mb | Files above this size are RAG-indexed | 1.0 MB |
rag_embedding_model | Sentence-transformer model for embeddings | all-MiniLM-L6-v2 |
rag_embedding_device | Device for embedding inference (cpu, cuda, mps) | cpu |
rag_hybrid_search_enabled | Whether BM25 + semantic search fusion is active | true |
rag_chunk_size_chars | Characters per text chunk | 2000 |
rag_chunk_overlap_chars | Overlap between consecutive chunks | 200 |
rag_tabular_chunk_rows | Rows per chunk for CSV/spreadsheet files | 50 |
rag_ann_threshold | Switch to HNSW approximate search above this chunk count | 100,000 |
Interpreting RAG Retrieval in Chat Traces
The chat diagnostic trace includes a rag_retrieval event emitted during _process_one_file. Check the trace detail in Phase 5 of the debugger (or via /api/admin/chat-debug/traces/{id}):
| Field | Meaning |
|---|---|
chunks_retrieved | Number of chunks selected for the LLM context |
total_chunks | Total chunks indexed for the file |
top_score | Highest similarity/relevance score (0.0-1.0) |
search_mode | Search strategy used: semantic, hybrid, hnsw, hnsw+hybrid |
budget_utilization_pct | Fraction of the character budget consumed by selected chunks |
duration_ms | Time spent on retrieval |
Understanding 0-Chunk Fallback
When semantic retrieval returns 0 chunks (common with very short or vague queries like "what is"):
- Current behavior: The system reads all stored chunk texts directly from the database (no PyMuPDF re-extraction). This is memory-safe because it reads only pre-indexed text, never the raw PDF.
- Previous behavior (caused OOM): Fell back to
storage.get_text_content()which re-extracted the full PDF via PyMuPDF, causing 200-500 MB memory spikes. - If no chunks exist in DB: A 5 MB file-size guard prevents any
get_text_content()call for large files, returning a user-friendly message instead.
Quick Diagnostic Commands
bash
# Check if backend is ready
curl -s http://localhost:8000/health/ready | python3 -m json.tool
# Check PII model status
curl -s http://localhost:8000/health/pii-model | python3 -m json.tool
# Check worker memory
curl -s http://localhost:8000/health/file-pipeline | python3 -m json.tool
# List recent chat traces with errors
curl -s "http://localhost:8000/api/admin/chat-debug/traces?has_error=true&limit=10"
# Check file metadata (is_indexed, chunk_count)
curl -s http://localhost:8000/api/files/{file_id}/metadata -b "session=..."
# Check full system health
curl -s http://localhost:8000/health | python3 -m json.toolEscalation Checklist
If the File Pipeline Debugger cannot resolve the issue, collect the following before escalating:
- Export debugger results -- "Copy as JSON" from the File Pipeline Debugger
- File details -- name, size, content type, number of pages (for PDFs)
- Health endpoint output --
/health,/health/file-pipeline,/health/pii-model - Backend logs -- last 100 lines around the failure timestamp
- Chat trace -- if available, the full trace JSON from
/api/admin/chat-debug/traces/{id} - Environment -- deployment type (Docker, ECS, Kubernetes), worker count, memory allocation
Related Documentation
- Common Issues -- general troubleshooting
- Performance -- PII detection tuning
- File Processing -- supported formats and limits
- Dashboards -- Chat Flow Debugger and File Pipeline Debugger details