Skip to content

File Processing ​

GhostAI scans and protects sensitive data in 90+ file formats before processing with AI models.

File Size Limits ​

LimitSizeScope
Regular files50 MBPer-file upload
Overall150 MBTotal upload size

Files up to 150 MB are supported with optimizations for large-file handling (see Large File Optimization below).

Supported Formats ​

Documents ​

FormatExtensionsFeatures
PDF.pdfText extraction, OCR for scanned pages
Word.docx, .docFull text, headers, footers, comments
Excel.xlsx, .xlsAll sheets, formulas, comments
PowerPoint.pptx, .pptSlides, notes, embedded content
OpenDocument.odt, .ods, .odpFull LibreOffice support
Rich Text.rtfFormatted text extraction
Plain Text.txt, .md, .csvDirect processing

Images ​

FormatExtensionsFeatures
JPEG.jpg, .jpegOCR, EXIF stripping
PNG.pngOCR, metadata removal
GIF.gifFrame extraction, OCR
WebP.webpOCR support
TIFF.tif, .tiffMulti-page support, OCR
BMP.bmpOCR support

Archives ​

FormatExtensionsFeatures
ZIP.zipRecursive extraction (3 levels)
7-Zip.7zPassword detection
TAR.tar, .tar.gz, .tgzLinux archives
RAR.rarWindows archives

Email ​

FormatExtensionsFeatures
Outlook.pst, .ostFull mailbox scanning
MBOX.mboxUnix mail archives
EML.emlIndividual emails
MSG.msgOutlook messages

Medical ​

FormatExtensionsFeatures
DICOM.dcmMetadata extraction, PHI detection
HL7.hl7Message parsing, patient data

Code ​

FormatExtensionsFeatures
Source Code.py, .js, .ts, etc.34 languages
Config.json, .yaml, .envSecrets detection
Database.sql, .sqliteSchema and data scanning

Audio ​

FormatExtensionsFeatures
MP3.mp3Whisper transcription, ID3 metadata
WAV.wavWhisper transcription
M4A.m4aWhisper transcription
OGG.oggWhisper transcription
FLAC.flacWhisper transcription

Video ​

FormatExtensionsFeatures
MP4.mp4Audio transcription, frame OCR, subtitles
MOV.movAudio transcription, frame OCR
WebM.webmAudio transcription, frame OCR
AVI.aviAudio transcription, frame OCR

Data Formats ​

FormatExtensionsFeatures
Parquet.parquetColumnar data extraction via pyarrow
TSV.tsvTab-separated value parsing
HAR.harHTTP archive / API debug data

How File Scanning Works ​

┌─────────┐    ┌──────────┐    ┌──────────┐    ┌──────────┐
│ Upload  │───▶│ Extract  │───▶│ Detect   │───▶│ Tokenize │
│  File   │    │  Text    │    │   PII    │    │   PII    │
└─────────┘    └──────────┘    └──────────┘    └──────────┘
                    │                               │
                    ▼                               ▼
              ┌──────────┐                   ┌──────────┐
              │   OCR    │                   │  Report  │
              │ (images) │                   │ Results  │
              └──────────┘                   └──────────┘

Process Steps ​

  1. Upload: File received and validated
  2. Type Detection: Identify file format
  3. Text Extraction: Extract readable content
  4. OCR (if needed): Process images for text
  5. PII Detection: Scan for sensitive data
  6. Tokenization: Protect detected PII
  7. Cache PII Results: Store detection results in the attachments.pii_scan_cache column so they can be reused during chat without re-running Presidio
  8. Preview: Show scan results to user

Image Intelligence ​

Content Detection ​

GhostAI detects sensitive content in images:

Content TypeDetectionConfidence
ID CardsDriver's licenses, IDs95%
Credit CardsCard numbers visible92%
PassportsPassport pages90%
Medical RecordsCharts, prescriptions88%
FacesFor de-identification95%

OCR Capabilities ​

Optical Character Recognition for:

  • Scanned documents
  • Photographed text
  • Screenshots
  • Handwritten text (limited)

EXIF Metadata ​

Automatically detected and optionally stripped:

MetadataRiskAction
GPS coordinatesLocation trackingStrip
Camera modelDevice identificationStrip
TimestampActivity trackingStrip
Creator nameIdentity exposureStrip

Archive Scanning ​

Recursive Extraction ​

Archives are extracted up to 3 levels deep:

file.zip
├── documents/
│   ├── report.pdf (scanned)
│   └── data.xlsx (scanned)
├── images/
│   └── scan.jpg (OCR + scanned)
└── nested.zip
    └── config.json (scanned)

Security Features ​

  • Zip bombs: Detected and blocked
  • Password protection: Detected, user prompted
  • Encrypted files: Flagged, not processed
  • Size limits: 50 MB per file, 150 MB overall

Security Threat Inspection ​

GhostAI inspects uploaded files for security threats beyond PII:

Threat Detection Capabilities ​

ThreatFile TypesDetection MethodStatus
VBA Macros.doc, .xls, .ppt, .docm, .xlsm, .pptmoletools pattern analysisActive
Excel 4.0 (XLM) Macros.xls, .xlsx, .xlsmMacro sheet detectionActive
PDF JavaScript.pdfPyMuPDF /JS and /JavaScript extractionActive
PDF Embedded Files.pdf/EmbeddedFiles traversalActive
Steganography.jpg, .png, .gif, .webp, .tif, .bmpChi-square + LSB analysisActive
Phishing URLsAll text-bearing filesTyposquatting, IDN homograph, URL shortenersActive
Encryption/PasswordZIP, 7z, RAR, PDF, Office, ODF, PST12 format-specific detectorsActive
Archive Bombs.zip, .tar, .7z, .rarCompression ratio, size, depth limitsActive
SVG Injection.svgScript, event handler, foreignObject removalActive
CSV/XML Injection.csv, .xmlFormula injection, XXE preventionActive
MalwareAll filesClamAV integration (when available)Active
SecretsAll text files20+ patterns (AWS, GitHub, Stripe, etc.)Active

Prompt Injection Defense ​

File content is scanned for prompt injection patterns before being sent to AI models:

  • 11 injection pattern detectors: "ignore previous instructions", "system prompt:", role-play triggers, etc.
  • RAG chunk sanitization: Each retrieved chunk is individually checked
  • Instruction hierarchy delimiters: File content is wrapped in a single [UNTRUSTED FILE CONTENT] header with compact [FILE: name]...[/FILE] wrappers per file. Multiple files share one header to save tokens. See Context Optimization for details.
  • Post-response PII guard: LLM output is scanned for leaked PII before delivery

Security Threat Analysis ​

Detected security findings are enriched by AI analysis that classifies them as malicious, suspicious, or benign. The AI evaluates context and intent to reduce false positives while preserving high-confidence threats.

Supported detector types for AI analysis include: VBA macros, Excel 4.0 (XLM) macros, PDF JavaScript, PDF embedded files, steganography, phishing URLs, CSV/XML injection, SVG injection, malware (ClamAV), and secrets. Encrypted archives and zip bombs are factual findings that skip AI analysis — they are always treated as high-risk based on structural detection.

Medical Format Support ​

DICOM Processing ​

Digital Imaging and Communications in Medicine:

  • Patient tags: Name, ID, DOB extraction
  • Study info: Dates, descriptions
  • Institution: Facility names, addresses
  • Burned-in text: OCR for image overlays

HL7 Parsing ​

Health Level 7 message processing:

  • Patient segments: PID, NK1
  • Visit info: PV1, PV2
  • Observations: OBX segments
  • Orders: ORC, OBR segments

Deep PII Scanning ​

GhostAI scans beyond surface-level text to find PII hidden in document metadata and structure:

Document TypeHidden Locations Scanned
Word (.docx)Comments, tracked changes, footnotes, headers/footers, text boxes
Excel (.xlsx)Cell comments, hidden sheets, named ranges, metadata
PowerPoint (.pptx)Speaker notes, slide masters, embedded objects
PDFForm fields, annotations, embedded files, metadata
ImagesEXIF metadata (GPS, device info), OCR text, embedded QR codes

Deep Scanning

Deep scanning is automatic — no configuration needed. All hidden content locations are checked by default.

Large File Optimization ​

GhostAI optimizes processing for large files to deliver fast feedback while ensuring full coverage:

Client-Side Gzip Compression ​

Text files over 5 MB are automatically compressed with gzip before upload. Compression is transparent — no user action required. This reduces upload time and bandwidth for large documents.

Progressive PII Scanning ​

Large files receive a two-phase scan:

  1. Quick scan: First 1,000 rows are scanned immediately; results appear right away.
  2. Deep scan: Full file is scanned in the background; results update when complete.

Users see actionable PII findings quickly while the full scan completes.

RAG (Retrieval-Augmented Generation) ​

For files over 1 MB, GhostAI uses Retrieval-Augmented Generation at chat time:

  • Content-type-aware chunking: Documents, spreadsheets, and code are chunked according to their structure.
  • Semantic embedding: Chunks are embedded for similarity search.
  • Cosine-similarity retrieval: Only the most relevant chunks are retrieved for each query.

Token-Budget-Aware Retrieval ​

Retrieval adapts to each model's context window. The system selects the optimal number of chunks to fit within the model's token budget, maximizing relevance without overflow.

No Content Truncation ​

The entire file is indexed and accessible. No content is dropped or truncated — every chunk is available for retrieval when relevant to the user's question.

RAG v2 Features ​

RAG v2 extends retrieval with production-grade optimizations:

FeatureDescription
Hybrid searchBM25 + semantic search with Reciprocal Rank Fusion for improved relevance
Cross-file retrievalRetrieve across multiple attached files in a single query
Incremental re-indexingSHA-256 hash-based diffing — only re-embed changed chunks
GPU-accelerated embeddingAuto-detect CUDA/MPS, batch encoding for faster indexing
Streaming ingestionStream CSV/TSV/TXT files >50 MB without loading into memory
Chunk-level PII cacheCache PII detection per chunk hash to avoid redundant scans
HNSW approximate nearest neighborFor files with >100K chunks — sublinear retrieval
Retrieval quality metricsPrometheus counters/histograms and OpenTelemetry spans

Configure RAG behavior in Admin Dashboard → RAG / File Intelligence: hybrid search, ANN, PII cache, streaming, and embedding device selector.

Visual Document Preview ​

GhostAI includes rich, in-browser document preview with automatic redaction. Click any uploaded file to open the visual preview modal:

  • PDF: Native browser viewer with zoom, search, and page navigation
  • Word/Excel/CSV: Rendered as formatted HTML tables and text
  • PowerPoint: Converted to PDF via LibreOffice
  • Images: Displayed with sensitive regions automatically blurred
  • HTML/Markdown: Sandboxed, sanitized rendering

Toggle between redacted (default) and original views. Download redacted copies for safe sharing.

For full details, see the Document Preview guide.

File Preview Metadata ​

When a file is uploaded:

┌────────────────────────────────────────────────────────────────┐
│  📄 patient_records.pdf                                         │
├────────────────────────────────────────────────────────────────┤
│  Type: PDF Document                                             │
│  Size: 2.4 MB                                                   │
│  Pages: 12                                                      │
│                                                                 │
│  🔍 Scan Results:                                               │
│  ├── 3 Patient names (PERSON)                                   │
│  ├── 3 SSNs (US_SSN)                                           │
│  ├── 5 Dates of birth (DATE_TIME)                              │
│  ├── 2 Medical record numbers (MEDICAL_PHI)                    │
│  └── 1 Phone number (PHONE_NUMBER)                             │
│                                                                 │
│  Compliance: HIPAA (14 PHI items)                              │
│                                                                 │
├────────────────────────────────────────────────────────────────┤
│  [View Full Report]  [Tokenize All]  [Remove]                  │
└────────────────────────────────────────────────────────────────┘

API Usage ​

Upload and Scan File ​

bash
curl -X POST http://localhost:8000/api/files/upload \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -F "file=@document.pdf" \
  -F "scan=true"

Response:

json
{
  "file_id": "file_abc123",
  "filename": "document.pdf",
  "size": 245678,
  "mime_type": "application/pdf",
  "scan_results": {
    "pii_count": 14,
    "entities": [
      {"type": "PERSON", "count": 3},
      {"type": "US_SSN", "count": 3},
      {"type": "DATE_TIME", "count": 5}
    ],
    "compliance_tags": ["HIPAA"],
    "scan_time_ms": 1234
  }
}

Get Scan Results ​

bash
curl http://localhost:8000/api/files/file_abc123/scan \
  -H "Authorization: Bearer YOUR_TOKEN"

Download Tokenized File ​

bash
curl http://localhost:8000/api/files/file_abc123/tokenized \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -o tokenized_document.pdf

Configuration ​

Environment Variables ​

bash
# Maximum file size (bytes) — per-file limit
MAX_FILE_SIZE=52428800  # 50MB

# Maximum total upload size (bytes) — overall limit
MAX_UPLOAD_SIZE=157286400  # 150MB

# OCR settings
OCR_ENABLED=true
OCR_LANGUAGES=eng,spa,fra

# Archive settings
ARCHIVE_MAX_DEPTH=3
ARCHIVE_MAX_FILES=100

# Medical format support
DICOM_ENABLED=true
HL7_ENABLED=true

Supported MIME Types ​

Configure which file types to accept:

json
{
  "allowed_mime_types": [
    "application/pdf",
    "application/vnd.openxmlformats-officedocument.*",
    "image/*",
    "text/*"
  ],
  "blocked_mime_types": [
    "application/x-executable",
    "application/x-msdownload"
  ]
}

Security Considerations ​

File Validation ​

  • MIME type verification
  • Extension validation
  • Magic byte checking
  • Virus scan integration (optional)

Storage ​

  • Encrypted at rest
  • Temporary file cleanup
  • Organization isolation
  • Retention policies

Access Control ​

  • User owns uploaded files
  • Admin can view organization files
  • API key scoping

Best Practices ​

  1. Enable OCR: Catch PII in scanned documents
  2. Set size limits: Prevent resource exhaustion
  3. Review archive settings: Balance depth vs. security
  4. Strip EXIF: Remove metadata by default
  5. Monitor scan times: Large files may need timeout adjustment

Troubleshooting ​

For comprehensive file pipeline diagnostics, use the File Pipeline Debugger in Platform Admin Console > Upload Test. It tests all 7 phases (health, upload, RAG indexing, scan, text preview, chat stream, memory) in a single run. See the File Pipeline Troubleshooting guide for detailed diagnosis workflows.

OCR Not Working ​

  1. Verify Tesseract is installed
  2. Check language packs are available
  3. Ensure image quality is sufficient

Large Files Timing Out ​

  1. Increase timeout settings
  2. Use async processing — progressive PII scanning handles large files in the background
  3. Client-side gzip compression reduces upload time for text files over 5 MB
  4. Check worker memory with GET /health/file-pipeline — large image-heavy PDFs can spike memory 10-50x file size

Large Files Causing Worker Crashes (OOM) ​

Image-heavy PDFs can cause worker OOM during chat stream processing. See File Pipeline Troubleshooting for symptoms, root causes, and fixes.

Archive Extraction Fails ​

  1. Check for password protection
  2. Verify archive isn't corrupted
  3. Check nesting depth

Next Steps ​

SOC 2 Type II Certified | BAA Available | HIPAA-Ready