Skip to content

GhostAI Scaling Guide ​

This document describes how GhostAI scales on AWS (ECS Fargate + RDS PostgreSQL) and how to plan for higher load.

Current production profile ​

ComponentSettingNotes
Backend tasks3 min, 20 maxECS Fargate; auto-scale on CPU/target tracking
Backend size2048 CPU, 4096 MiBPer task
RDSdb.r6g.largePostgreSQL 15; max_connections = 200
ALBidle_timeout 300sPath-based routing to backend/frontend/docs

Database connections and the connection pool ​

The backend uses a PostgreSQL connection pool per process (psycopg2.pool.ThreadedConnectionPool):

  • Per task: minconn=2, maxconn=10 (fixed in code).
  • Total connections to RDS ≈ number of backend tasks × 10.

So:

  • 3 tasks → ~30 connections
  • 10 tasks → ~100 connections
  • 20 tasks → ~200 connections (at RDS limit)

Rule of thumb: Keep (backend task count × 10) below ~80% of RDS max_connections (200) to leave headroom for admin/monitoring. For 200 max_connections, that’s about 16 backend tasks at 10 connections each.

If you scale beyond that:

  1. Option A: Reduce pool size per task (e.g. change maxconn in app/services/database.py to 5–8), or
  2. Option B: Increase RDS max_connections in the Terraform RDS parameter group (and size the instance accordingly).

Without the pool, every DB call would open and close a new connection; under load that could spike toward 200 and cause 502/504. The pool makes connection usage predictable and bounded per task.


Horizontal scaling (more backend tasks) ​

  • ECS: Increase backend_desired_count, backend_max_count in Terraform (e.g. terraform.tfvars for prod).
  • Effect: More tasks → more concurrent requests; DB connections scale linearly (tasks × 10).
  • Watch: RDS connection count (CloudWatch DatabaseConnections); RDS alarm *-rds-connections-high fires at 160 (80% of 200).

Vertical scaling (bigger backend tasks) ​

  • ECS: Increase backend_cpu and backend_memory in Terraform.
  • Effect: Each task can handle more concurrent work (event loop + thread pool for PII/billing/model resolution). Does not change DB connection count per task (still 10).
  • Use when: You want higher throughput per task without adding more tasks.

Request timeouts and blocking work ​

To avoid 502/504 under load, slow or blocking work is isolated and time-bounded:

PathChangeTimeout
Billing check (chat stream)Run in thread5s; on timeout, allow request
Model resolution (chat stream)Run in thread10s; on timeout, use default model
Session + PII + code policy (chat stream)asyncio.gather wrapped in wait_for15s; on timeout, 503
PII detect (POST /api/detect)Session/tokenize in thread; whole handler15s; on timeout, 503

These prevent one slow DB or external call from blocking the event loop and causing ALB timeouts.


ALB and target health ​

  • ALB idle_timeout: 300s (connection to client).
  • Target group: Backend health check /health/live every 60s; unhealthy threshold 5.
  • Sticky sessions: 10 minutes for /api/* (OAuth callback must hit same backend).

If a task is blocked (e.g. before the pool was added, or on a rare blocking path), it may stop responding to health checks and get dropped; new requests then go to healthy tasks.


Scaling checklist ​

  • [ ] RDS: Monitor DatabaseConnections; if you add many backend tasks, ensure (tasks × 10) < ~160 or raise max_connections.
  • [ ] Backend tasks: Increase backend_desired_count / backend_max_count for more throughput; stay within RDS connection budget.
  • [ ] Backend size: Increase backend_cpu / backend_memory for more throughput per task without more connections.
  • [ ] Alarms: *-rds-connections-high (160), *-alb-backend-unhealthy, *-alb-5xx-errors; act before hitting limits.

See also ​

SOC 2 Type II Certified | BAA Available | HIPAA-Ready