Appearance
File Processing
GhostAI scans and protects sensitive data in 90+ file formats before processing with AI models.
File Size Limits
| Limit | Size | Scope |
|---|---|---|
| Regular files | 50 MB | Per-file upload |
| Overall | 150 MB | Total upload size |
Files up to 150 MB are supported with optimizations for large-file handling (see Large File Optimization below).
Supported Formats
Documents
| Format | Extensions | Features |
|---|---|---|
| Text extraction, OCR for scanned pages | ||
| Word | .docx, .doc | Full text, headers, footers, comments |
| Excel | .xlsx, .xls | All sheets, formulas, comments |
| PowerPoint | .pptx, .ppt | Slides, notes, embedded content |
| OpenDocument | .odt, .ods, .odp | Full LibreOffice support |
| Rich Text | .rtf | Formatted text extraction |
| Plain Text | .txt, .md, .csv | Direct processing |
Images
| Format | Extensions | Features |
|---|---|---|
| JPEG | .jpg, .jpeg | OCR, EXIF stripping |
| PNG | .png | OCR, metadata removal |
| GIF | .gif | Frame extraction, OCR |
| WebP | .webp | OCR support |
| TIFF | .tif, .tiff | Multi-page support, OCR |
| BMP | .bmp | OCR support |
Archives
| Format | Extensions | Features |
|---|---|---|
| ZIP | .zip | Recursive extraction (3 levels) |
| 7-Zip | .7z | Password detection |
| TAR | .tar, .tar.gz, .tgz | Linux archives |
| RAR | .rar | Windows archives |
Email
| Format | Extensions | Features |
|---|---|---|
| Outlook | .pst, .ost | Full mailbox scanning |
| MBOX | .mbox | Unix mail archives |
| EML | .eml | Individual emails |
| MSG | .msg | Outlook messages |
Medical
| Format | Extensions | Features |
|---|---|---|
| DICOM | .dcm | Metadata extraction, PHI detection |
| HL7 | .hl7 | Message parsing, patient data |
Code
| Format | Extensions | Features |
|---|---|---|
| Source Code | .py, .js, .ts, etc. | 34 languages |
| Config | .json, .yaml, .env | Secrets detection |
| Database | .sql, .sqlite | Schema and data scanning |
Audio
| Format | Extensions | Features |
|---|---|---|
| MP3 | .mp3 | Whisper transcription, ID3 metadata |
| WAV | .wav | Whisper transcription |
| M4A | .m4a | Whisper transcription |
| OGG | .ogg | Whisper transcription |
| FLAC | .flac | Whisper transcription |
Video
| Format | Extensions | Features |
|---|---|---|
| MP4 | .mp4 | Audio transcription, frame OCR, subtitles |
| MOV | .mov | Audio transcription, frame OCR |
| WebM | .webm | Audio transcription, frame OCR |
| AVI | .avi | Audio transcription, frame OCR |
Data Formats
| Format | Extensions | Features |
|---|---|---|
| Parquet | .parquet | Columnar data extraction via pyarrow |
| TSV | .tsv | Tab-separated value parsing |
| HAR | .har | HTTP archive / API debug data |
How File Scanning Works
┌─────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐
│ Upload │───▶│ Extract │───▶│ Detect │───▶│ Tokenize │
│ File │ │ Text │ │ PII │ │ PII │
└─────────┘ └──────────┘ └──────────┘ └──────────┘
│ │
▼ ▼
┌──────────┐ ┌──────────┐
│ OCR │ │ Report │
│ (images) │ │ Results │
└──────────┘ └──────────┘Process Steps
- Upload: File received and validated
- Type Detection: Identify file format
- Text Extraction: Extract readable content
- OCR (if needed): Process images for text
- PII Detection: Scan for sensitive data
- Tokenization: Protect detected PII
- Cache PII Results: Store detection results in the
attachments.pii_scan_cachecolumn so they can be reused during chat without re-running Presidio - Preview: Show scan results to user
Image Intelligence
Content Detection
GhostAI detects sensitive content in images:
| Content Type | Detection | Confidence |
|---|---|---|
| ID Cards | Driver's licenses, IDs | 95% |
| Credit Cards | Card numbers visible | 92% |
| Passports | Passport pages | 90% |
| Medical Records | Charts, prescriptions | 88% |
| Faces | For de-identification | 95% |
OCR Capabilities
Optical Character Recognition for:
- Scanned documents
- Photographed text
- Screenshots
- Handwritten text (limited)
EXIF Metadata
Automatically detected and optionally stripped:
| Metadata | Risk | Action |
|---|---|---|
| GPS coordinates | Location tracking | Strip |
| Camera model | Device identification | Strip |
| Timestamp | Activity tracking | Strip |
| Creator name | Identity exposure | Strip |
Archive Scanning
Recursive Extraction
Archives are extracted up to 3 levels deep:
file.zip
├── documents/
│ ├── report.pdf (scanned)
│ └── data.xlsx (scanned)
├── images/
│ └── scan.jpg (OCR + scanned)
└── nested.zip
└── config.json (scanned)Security Features
- Zip bombs: Detected and blocked
- Password protection: Detected, user prompted
- Encrypted files: Flagged, not processed
- Size limits: 50 MB per file, 150 MB overall
Security Threat Inspection
GhostAI inspects uploaded files for security threats beyond PII:
Threat Detection Capabilities
| Threat | File Types | Detection Method | Status |
|---|---|---|---|
| VBA Macros | .doc, .xls, .ppt, .docm, .xlsm, .pptm | oletools pattern analysis | Active |
| Excel 4.0 (XLM) Macros | .xls, .xlsx, .xlsm | Macro sheet detection | Active |
| PDF JavaScript | PyMuPDF /JS and /JavaScript extraction | Active | |
| PDF Embedded Files | /EmbeddedFiles traversal | Active | |
| Steganography | .jpg, .png, .gif, .webp, .tif, .bmp | Chi-square + LSB analysis | Active |
| Phishing URLs | All text-bearing files | Typosquatting, IDN homograph, URL shorteners | Active |
| Encryption/Password | ZIP, 7z, RAR, PDF, Office, ODF, PST | 12 format-specific detectors | Active |
| Archive Bombs | .zip, .tar, .7z, .rar | Compression ratio, size, depth limits | Active |
| SVG Injection | .svg | Script, event handler, foreignObject removal | Active |
| CSV/XML Injection | .csv, .xml | Formula injection, XXE prevention | Active |
| Malware | All files | ClamAV integration (when available) | Active |
| Secrets | All text files | 20+ patterns (AWS, GitHub, Stripe, etc.) | Active |
Prompt Injection Defense
File content is scanned for prompt injection patterns before being sent to AI models:
- 11 injection pattern detectors: "ignore previous instructions", "system prompt:", role-play triggers, etc.
- RAG chunk sanitization: Each retrieved chunk is individually checked
- Instruction hierarchy delimiters: File content is wrapped in a single
[UNTRUSTED FILE CONTENT]header with compact[FILE: name]...[/FILE]wrappers per file. Multiple files share one header to save tokens. See Context Optimization for details. - Post-response PII guard: LLM output is scanned for leaked PII before delivery
Security Threat Analysis
Detected security findings are enriched by AI analysis that classifies them as malicious, suspicious, or benign. The AI evaluates context and intent to reduce false positives while preserving high-confidence threats.
Supported detector types for AI analysis include: VBA macros, Excel 4.0 (XLM) macros, PDF JavaScript, PDF embedded files, steganography, phishing URLs, CSV/XML injection, SVG injection, malware (ClamAV), and secrets. Encrypted archives and zip bombs are factual findings that skip AI analysis — they are always treated as high-risk based on structural detection.
Medical Format Support
DICOM Processing
Digital Imaging and Communications in Medicine:
- Patient tags: Name, ID, DOB extraction
- Study info: Dates, descriptions
- Institution: Facility names, addresses
- Burned-in text: OCR for image overlays
HL7 Parsing
Health Level 7 message processing:
- Patient segments: PID, NK1
- Visit info: PV1, PV2
- Observations: OBX segments
- Orders: ORC, OBR segments
Deep PII Scanning
GhostAI scans beyond surface-level text to find PII hidden in document metadata and structure:
| Document Type | Hidden Locations Scanned |
|---|---|
| Word (.docx) | Comments, tracked changes, footnotes, headers/footers, text boxes |
| Excel (.xlsx) | Cell comments, hidden sheets, named ranges, metadata |
| PowerPoint (.pptx) | Speaker notes, slide masters, embedded objects |
| Form fields, annotations, embedded files, metadata | |
| Images | EXIF metadata (GPS, device info), OCR text, embedded QR codes |
Deep Scanning
Deep scanning is automatic — no configuration needed. All hidden content locations are checked by default.
Large File Optimization
GhostAI optimizes processing for large files to deliver fast feedback while ensuring full coverage:
Client-Side Gzip Compression
Text files over 5 MB are automatically compressed with gzip before upload. Compression is transparent — no user action required. This reduces upload time and bandwidth for large documents.
Progressive PII Scanning
Large files receive a two-phase scan:
- Quick scan: First 1,000 rows are scanned immediately; results appear right away.
- Deep scan: Full file is scanned in the background; results update when complete.
Users see actionable PII findings quickly while the full scan completes.
RAG (Retrieval-Augmented Generation)
For files over 1 MB, GhostAI uses Retrieval-Augmented Generation at chat time:
- Content-type-aware chunking: Documents, spreadsheets, and code are chunked according to their structure.
- Semantic embedding: Chunks are embedded for similarity search.
- Cosine-similarity retrieval: Only the most relevant chunks are retrieved for each query.
Token-Budget-Aware Retrieval
Retrieval adapts to each model's context window. The system selects the optimal number of chunks to fit within the model's token budget, maximizing relevance without overflow.
No Content Truncation
The entire file is indexed and accessible. No content is dropped or truncated — every chunk is available for retrieval when relevant to the user's question.
RAG v2 Features
RAG v2 extends retrieval with production-grade optimizations:
| Feature | Description |
|---|---|
| Hybrid search | BM25 + semantic search with Reciprocal Rank Fusion for improved relevance |
| Cross-file retrieval | Retrieve across multiple attached files in a single query |
| Incremental re-indexing | SHA-256 hash-based diffing — only re-embed changed chunks |
| GPU-accelerated embedding | Auto-detect CUDA/MPS, batch encoding for faster indexing |
| Streaming ingestion | Stream CSV/TSV/TXT files >50 MB without loading into memory |
| Chunk-level PII cache | Cache PII detection per chunk hash to avoid redundant scans |
| HNSW approximate nearest neighbor | For files with >100K chunks — sublinear retrieval |
| Retrieval quality metrics | Prometheus counters/histograms and OpenTelemetry spans |
Configure RAG behavior in Admin Dashboard → RAG / File Intelligence: hybrid search, ANN, PII cache, streaming, and embedding device selector.
Visual Document Preview
GhostAI includes rich, in-browser document preview with automatic redaction. Click any uploaded file to open the visual preview modal:
- PDF: Native browser viewer with zoom, search, and page navigation
- Word/Excel/CSV: Rendered as formatted HTML tables and text
- PowerPoint: Converted to PDF via LibreOffice
- Images: Displayed with sensitive regions automatically blurred
- HTML/Markdown: Sandboxed, sanitized rendering
Toggle between redacted (default) and original views. Download redacted copies for safe sharing.
For full details, see the Document Preview guide.
File Preview Metadata
When a file is uploaded:
┌────────────────────────────────────────────────────────────────┐
│ 📄 patient_records.pdf │
├────────────────────────────────────────────────────────────────┤
│ Type: PDF Document │
│ Size: 2.4 MB │
│ Pages: 12 │
│ │
│ 🔍 Scan Results: │
│ ├── 3 Patient names (PERSON) │
│ ├── 3 SSNs (US_SSN) │
│ ├── 5 Dates of birth (DATE_TIME) │
│ ├── 2 Medical record numbers (MEDICAL_PHI) │
│ └── 1 Phone number (PHONE_NUMBER) │
│ │
│ Compliance: HIPAA (14 PHI items) │
│ │
├────────────────────────────────────────────────────────────────┤
│ [View Full Report] [Tokenize All] [Remove] │
└────────────────────────────────────────────────────────────────┘API Usage
Upload and Scan File
bash
curl -X POST http://localhost:8000/api/files/upload \
-H "Authorization: Bearer YOUR_TOKEN" \
-F "file=@document.pdf" \
-F "scan=true"Response:
json
{
"file_id": "file_abc123",
"filename": "document.pdf",
"size": 245678,
"mime_type": "application/pdf",
"scan_results": {
"pii_count": 14,
"entities": [
{"type": "PERSON", "count": 3},
{"type": "US_SSN", "count": 3},
{"type": "DATE_TIME", "count": 5}
],
"compliance_tags": ["HIPAA"],
"scan_time_ms": 1234
}
}Get Scan Results
bash
curl http://localhost:8000/api/files/file_abc123/scan \
-H "Authorization: Bearer YOUR_TOKEN"Download Tokenized File
bash
curl http://localhost:8000/api/files/file_abc123/tokenized \
-H "Authorization: Bearer YOUR_TOKEN" \
-o tokenized_document.pdfConfiguration
Environment Variables
bash
# Maximum file size (bytes) — per-file limit
MAX_FILE_SIZE=52428800 # 50MB
# Maximum total upload size (bytes) — overall limit
MAX_UPLOAD_SIZE=157286400 # 150MB
# OCR settings
OCR_ENABLED=true
OCR_LANGUAGES=eng,spa,fra
# Archive settings
ARCHIVE_MAX_DEPTH=3
ARCHIVE_MAX_FILES=100
# Medical format support
DICOM_ENABLED=true
HL7_ENABLED=trueSupported MIME Types
Configure which file types to accept:
json
{
"allowed_mime_types": [
"application/pdf",
"application/vnd.openxmlformats-officedocument.*",
"image/*",
"text/*"
],
"blocked_mime_types": [
"application/x-executable",
"application/x-msdownload"
]
}Security Considerations
File Validation
- MIME type verification
- Extension validation
- Magic byte checking
- Virus scan integration (optional)
Storage
- Encrypted at rest
- Temporary file cleanup
- Organization isolation
- Retention policies
Access Control
- User owns uploaded files
- Admin can view organization files
- API key scoping
Best Practices
- Enable OCR: Catch PII in scanned documents
- Set size limits: Prevent resource exhaustion
- Review archive settings: Balance depth vs. security
- Strip EXIF: Remove metadata by default
- Monitor scan times: Large files may need timeout adjustment
Troubleshooting
For comprehensive file pipeline diagnostics, use the File Pipeline Debugger in Platform Admin Console > Upload Test. It tests all 7 phases (health, upload, RAG indexing, scan, text preview, chat stream, memory) in a single run. See the File Pipeline Troubleshooting guide for detailed diagnosis workflows.
OCR Not Working
- Verify Tesseract is installed
- Check language packs are available
- Ensure image quality is sufficient
Large Files Timing Out
- Increase timeout settings
- Use async processing — progressive PII scanning handles large files in the background
- Client-side gzip compression reduces upload time for text files over 5 MB
- Check worker memory with
GET /health/file-pipeline— large image-heavy PDFs can spike memory 10-50x file size
Large Files Causing Worker Crashes (OOM)
Image-heavy PDFs can cause worker OOM during chat stream processing. See File Pipeline Troubleshooting for symptoms, root causes, and fixes.
Archive Extraction Fails
- Check for password protection
- Verify archive isn't corrupted
- Check nesting depth
Next Steps
- Document Preview — Rich visual previews with redaction
- Image Intelligence — Image content detection and redaction
- Content Fingerprinting — Duplicate detection and tracking
- PII Detection — Detection capabilities
- Anti-Exfiltration — Obfuscation in files