Appearance
Content Library
Multi-tenant document management with per-organization isolation, grant-based access control, hybrid search (BM25 + semantic), and compliance features.
Overview
Every organization gets its own content library where documents are indexed, chunked, embedded, and searchable. Content can be added via UI upload, REST API, or synced from external sources (filesystem, S3). The existing GhostAI internal library operates as a standard organization library within this system.
Architecture
Multi-Tenant Isolation
- Every
content_library_*table includes anorganization_idcolumn - All SQL queries include
WHERE organization_id = ?for application-level isolation - Postgres Row-Level Security (RLS) provides defense-in-depth at the database level
- RLS session variable
app.current_org_idis set per-connection - SQLite (dev/test) relies on application-level WHERE clauses only
Storage
- Files stored via
StorageBackendwith AES-256-GCM per-tenant encryption - Local filesystem and S3 backends supported
- Per-org storage quotas enforced on upload (configurable per tier)
- Default quotas: Team 1 GB, Business 10 GB, Enterprise 100 GB
Content Ingestion
Upload (UI + API)
Users with content_library:upload permission can upload files via:
- Drag-and-drop in the Content Library UI
- File picker button
- REST API:
POST /api/content-library/items/upload(multipart) - Batch upload:
POST /api/content-library/items/upload-batch
Upload pipeline:
- Extension validation (blocked:
.env,.pem,.key, etc.) - Size check against org storage quota
- Filename sanitization (strip paths, normalize unicode)
- Store via StorageBackend (encrypted at rest)
- Text extraction for binary formats (PDF, Office, email)
- Chunking and embedding for search indexing
Supported File Types
| Type | Extensions | Viewer |
|---|---|---|
| Markdown | .md | Rendered |
.pdf | Native viewer | |
| HTML | .html | Rendered |
| Code | .py | Syntax highlighted |
| Text | .txt, .log | Plain text |
| YAML | .yaml, .yml | Highlighted |
| Data | .csv, .tsv, .json | Formatted |
| Images | .png, .jpg, .svg, .webp, .gif | Image viewer |
| Documents | .docx, .doc, .odt, .rtf | Extracted text + download |
| Spreadsheets | .xlsx, .xls, .ods | Extracted text + download |
| Presentations | .pptx, .ppt, .odp | Extracted text + download |
.eml, .msg | Extracted text + download |
Sync Sources
Organizations can configure external content sources:
- Filesystem: Syncs from a local directory path (must be in
CONTENT_LIBRARY_ALLOWED_SYNC_ROOTS) - S3: Syncs from an S3 bucket/prefix (credentials encrypted with AES-256-GCM)
Source management endpoints:
GET /api/content-library/sources- List sources (credentials redacted)POST /api/content-library/sources- Create sourcePOST /api/content-library/sources/{id}/sync- Trigger syncDELETE /api/content-library/sources/{id}- Remove source
Version Management
PUT /api/content-library/items/{id}/upload- Replace with new version- Creates a version snapshot of the old content
- Re-extracts text, re-chunks, re-embeds
- Preserves bookmarks, grants, and user state
Search
Hybrid search combining three strategies:
- BM25 text search - Chunk-level keyword matching with term frequency scoring
- Title/metadata fallback - Title, description, tags, and path matching
- Semantic search - Embedding-based cosine similarity
- Postgres: Uses pgvector HNSW index for O(log n) approximate nearest neighbor search
- SQLite: Falls back to in-memory cosine similarity
- Results merged via Reciprocal Rank Fusion (RRF)
All search paths are org-scoped. Results are filtered by user access grants.
Access Control
RBAC Permissions
| Permission | Owner | Admin | Member | Viewer |
|---|---|---|---|---|
content_library:read | Yes | Yes | Yes | Yes |
content_library:upload | Yes | Yes | No | No |
content_library:manage | Yes | Yes | No | No |
content_library:audit | Yes | Yes | No | No |
Grant System
Fine-grained access within the org:
- Category grants: Access all items in a category
- Item grants: Access a specific item
- Implicit grants: Owner/admin have full access by role
- Expiring grants: Optional
expires_atfor time-limited access - Grant templates: Pre-defined category bundles for common roles
- Bulk grants: Apply templates to multiple users at once
Feature Flags
Content library gated by subscription tier via feature_flags.py:
- Team, Business, and Enterprise tiers have access
- Free tier does not have access
API Reference
Items
| Method | Endpoint | Permission | Description |
|---|---|---|---|
| GET | /api/content-library/categories | read | List categories with counts |
| GET | /api/content-library/items | read | List items (filterable) |
| GET | /api/content-library/items/{id} | read | Get item content |
| GET | /api/content-library/items/{id}/raw | read | Download raw file |
| POST | /api/content-library/items/upload | manage | Upload file |
| PUT | /api/content-library/items/{id}/upload | manage | Replace with new version |
| POST | /api/content-library/items/upload-batch | manage | Upload multiple files |
| DELETE | /api/content-library/items/{id} | manage | Delete item |
| GET | /api/content-library/search | read | Hybrid search |
Admin
| Method | Endpoint | Permission | Description |
|---|---|---|---|
| GET | /api/content-library/audit | audit | Audit log |
| GET | /api/content-library/analytics | audit | Usage analytics |
| GET | /api/content-library/grants | manage | List grants |
| POST | /api/content-library/grants | manage | Create grant |
| DELETE | /api/content-library/grants/{id} | manage | Revoke grant |
| GET | /api/content-library/quota | read | Storage usage |
Sources
| Method | Endpoint | Permission | Description |
|---|---|---|---|
| GET | /api/content-library/sources | manage | List sources |
| POST | /api/content-library/sources | manage | Create source |
| POST | /api/content-library/sources/{id}/sync | manage | Trigger sync |
| DELETE | /api/content-library/sources/{id} | manage | Delete source |
Compliance
GDPR
- Right to erasure:
purge_user_content_library_data()removes all user footprint (uploads, feedback, bookmarks, audit entries, collections, access requests) - Org deletion:
purge_org_content_library()cascades deletion of all content library data including StorageBackend files - FK CASCADE on
organization_idensures database-level cleanup
Data Retention
- Audit logs honor org-level
audit_log_retention_dayspolicy - Optional
expires_aton content items for automatic expiration - Expired items hidden from browse/search after grace period
Encryption
- Files encrypted at rest with AES-256-GCM (per-tenant key derivation)
- S3 sync source credentials encrypted with org-scoped AES-256-GCM keys
- Credentials never stored in plaintext, never returned in API responses
Configuration
| Setting | Default | Description |
|---|---|---|
content_library_home_org_id | (deprecated) | Legacy setting, replaced by per-org model |
content_library_allowed_sync_roots | null | JSON list of allowed filesystem paths for sync sources |
content_library_embedding_backend | local | Embedding backend: local or bedrock |
content_library_storage_limit_mb | 10240 | Per-org storage quota (org_policies table) |
Migration
On first startup after upgrade, migrate_to_multi_tenant() runs automatically:
- Backfills
organization_idon all existing content library rows - Creates a default filesystem source for the home org
- Safe to run multiple times (idempotent)