Skip to content

Content Library ​

Multi-tenant document management with per-organization isolation, grant-based access control, hybrid search (BM25 + semantic), and compliance features.

Overview ​

Every organization gets its own content library where documents are indexed, chunked, embedded, and searchable. Content can be added via UI upload, REST API, or synced from external sources (filesystem, S3). The existing GhostAI internal library operates as a standard organization library within this system.

Architecture ​

Multi-Tenant Isolation ​

  • Every content_library_* table includes an organization_id column
  • All SQL queries include WHERE organization_id = ? for application-level isolation
  • Postgres Row-Level Security (RLS) provides defense-in-depth at the database level
  • RLS session variable app.current_org_id is set per-connection
  • SQLite (dev/test) relies on application-level WHERE clauses only

Storage ​

  • Files stored via StorageBackend with AES-256-GCM per-tenant encryption
  • Local filesystem and S3 backends supported
  • Per-org storage quotas enforced on upload (configurable per tier)
  • Default quotas: Team 1 GB, Business 10 GB, Enterprise 100 GB

Content Ingestion ​

Upload (UI + API) ​

Users with content_library:upload permission can upload files via:

  • Drag-and-drop in the Content Library UI
  • File picker button
  • REST API: POST /api/content-library/items/upload (multipart)
  • Batch upload: POST /api/content-library/items/upload-batch

Upload pipeline:

  1. Extension validation (blocked: .env, .pem, .key, etc.)
  2. Size check against org storage quota
  3. Filename sanitization (strip paths, normalize unicode)
  4. Store via StorageBackend (encrypted at rest)
  5. Text extraction for binary formats (PDF, Office, email)
  6. Chunking and embedding for search indexing

Supported File Types ​

TypeExtensionsViewer
Markdown.mdRendered
PDF.pdfNative viewer
HTML.htmlRendered
Code.pySyntax highlighted
Text.txt, .logPlain text
YAML.yaml, .ymlHighlighted
Data.csv, .tsv, .jsonFormatted
Images.png, .jpg, .svg, .webp, .gifImage viewer
Documents.docx, .doc, .odt, .rtfExtracted text + download
Spreadsheets.xlsx, .xls, .odsExtracted text + download
Presentations.pptx, .ppt, .odpExtracted text + download
Email.eml, .msgExtracted text + download

Sync Sources ​

Organizations can configure external content sources:

  • Filesystem: Syncs from a local directory path (must be in CONTENT_LIBRARY_ALLOWED_SYNC_ROOTS)
  • S3: Syncs from an S3 bucket/prefix (credentials encrypted with AES-256-GCM)

Source management endpoints:

  • GET /api/content-library/sources - List sources (credentials redacted)
  • POST /api/content-library/sources - Create source
  • POST /api/content-library/sources/{id}/sync - Trigger sync
  • DELETE /api/content-library/sources/{id} - Remove source

Version Management ​

  • PUT /api/content-library/items/{id}/upload - Replace with new version
  • Creates a version snapshot of the old content
  • Re-extracts text, re-chunks, re-embeds
  • Preserves bookmarks, grants, and user state

Hybrid search combining three strategies:

  1. BM25 text search - Chunk-level keyword matching with term frequency scoring
  2. Title/metadata fallback - Title, description, tags, and path matching
  3. Semantic search - Embedding-based cosine similarity
    • Postgres: Uses pgvector HNSW index for O(log n) approximate nearest neighbor search
    • SQLite: Falls back to in-memory cosine similarity
  4. Results merged via Reciprocal Rank Fusion (RRF)

All search paths are org-scoped. Results are filtered by user access grants.

Access Control ​

RBAC Permissions ​

PermissionOwnerAdminMemberViewer
content_library:readYesYesYesYes
content_library:uploadYesYesNoNo
content_library:manageYesYesNoNo
content_library:auditYesYesNoNo

Grant System ​

Fine-grained access within the org:

  • Category grants: Access all items in a category
  • Item grants: Access a specific item
  • Implicit grants: Owner/admin have full access by role
  • Expiring grants: Optional expires_at for time-limited access
  • Grant templates: Pre-defined category bundles for common roles
  • Bulk grants: Apply templates to multiple users at once

Feature Flags ​

Content library gated by subscription tier via feature_flags.py:

  • Team, Business, and Enterprise tiers have access
  • Free tier does not have access

API Reference ​

Items ​

MethodEndpointPermissionDescription
GET/api/content-library/categoriesreadList categories with counts
GET/api/content-library/itemsreadList items (filterable)
GET/api/content-library/items/{id}readGet item content
GET/api/content-library/items/{id}/rawreadDownload raw file
POST/api/content-library/items/uploadmanageUpload file
PUT/api/content-library/items/{id}/uploadmanageReplace with new version
POST/api/content-library/items/upload-batchmanageUpload multiple files
DELETE/api/content-library/items/{id}manageDelete item
GET/api/content-library/searchreadHybrid search

Admin ​

MethodEndpointPermissionDescription
GET/api/content-library/auditauditAudit log
GET/api/content-library/analyticsauditUsage analytics
GET/api/content-library/grantsmanageList grants
POST/api/content-library/grantsmanageCreate grant
DELETE/api/content-library/grants/{id}manageRevoke grant
GET/api/content-library/quotareadStorage usage

Sources ​

MethodEndpointPermissionDescription
GET/api/content-library/sourcesmanageList sources
POST/api/content-library/sourcesmanageCreate source
POST/api/content-library/sources/{id}/syncmanageTrigger sync
DELETE/api/content-library/sources/{id}manageDelete source

Compliance ​

GDPR ​

  • Right to erasure: purge_user_content_library_data() removes all user footprint (uploads, feedback, bookmarks, audit entries, collections, access requests)
  • Org deletion: purge_org_content_library() cascades deletion of all content library data including StorageBackend files
  • FK CASCADE on organization_id ensures database-level cleanup

Data Retention ​

  • Audit logs honor org-level audit_log_retention_days policy
  • Optional expires_at on content items for automatic expiration
  • Expired items hidden from browse/search after grace period

Encryption ​

  • Files encrypted at rest with AES-256-GCM (per-tenant key derivation)
  • S3 sync source credentials encrypted with org-scoped AES-256-GCM keys
  • Credentials never stored in plaintext, never returned in API responses

Configuration ​

SettingDefaultDescription
content_library_home_org_id(deprecated)Legacy setting, replaced by per-org model
content_library_allowed_sync_rootsnullJSON list of allowed filesystem paths for sync sources
content_library_embedding_backendlocalEmbedding backend: local or bedrock
content_library_storage_limit_mb10240Per-org storage quota (org_policies table)

Migration ​

On first startup after upgrade, migrate_to_multi_tenant() runs automatically:

  1. Backfills organization_id on all existing content library rows
  2. Creates a default filesystem source for the home org
  3. Safe to run multiple times (idempotent)

SOC 2 Type II Certified | BAA Available | HIPAA-Ready