Appearance
GhostAI Scaling Guide
This document describes how GhostAI scales on AWS (ECS Fargate + RDS PostgreSQL) and how to plan for higher load.
Current production profile
| Component | Setting | Notes |
|---|---|---|
| Backend tasks | 3 min, 20 max | ECS Fargate; auto-scale on CPU/target tracking |
| Backend size | 2048 CPU, 4096 MiB | Per task |
| RDS | db.r6g.large | PostgreSQL 15; max_connections = 200 |
| ALB | idle_timeout 300s | Path-based routing to backend/frontend/docs |
Database connections and the connection pool
The backend uses a PostgreSQL connection pool per process (psycopg2.pool.ThreadedConnectionPool):
- Per task:
minconn=2,maxconn=10(fixed in code). - Total connections to RDS ≈ number of backend tasks × 10.
So:
- 3 tasks → ~30 connections
- 10 tasks → ~100 connections
- 20 tasks → ~200 connections (at RDS limit)
Rule of thumb: Keep (backend task count × 10) below ~80% of RDS max_connections (200) to leave headroom for admin/monitoring. For 200 max_connections, that’s about 16 backend tasks at 10 connections each.
If you scale beyond that:
- Option A: Reduce pool size per task (e.g. change
maxconninapp/services/database.pyto 5–8), or - Option B: Increase RDS
max_connectionsin the Terraform RDS parameter group (and size the instance accordingly).
Without the pool, every DB call would open and close a new connection; under load that could spike toward 200 and cause 502/504. The pool makes connection usage predictable and bounded per task.
Horizontal scaling (more backend tasks)
- ECS: Increase
backend_desired_count,backend_max_countin Terraform (e.g.terraform.tfvarsfor prod). - Effect: More tasks → more concurrent requests; DB connections scale linearly (tasks × 10).
- Watch: RDS connection count (CloudWatch
DatabaseConnections); RDS alarm*-rds-connections-highfires at 160 (80% of 200).
Vertical scaling (bigger backend tasks)
- ECS: Increase
backend_cpuandbackend_memoryin Terraform. - Effect: Each task can handle more concurrent work (event loop + thread pool for PII/billing/model resolution). Does not change DB connection count per task (still 10).
- Use when: You want higher throughput per task without adding more tasks.
Request timeouts and blocking work
To avoid 502/504 under load, slow or blocking work is isolated and time-bounded:
| Path | Change | Timeout |
|---|---|---|
| Billing check (chat stream) | Run in thread | 5s; on timeout, allow request |
| Model resolution (chat stream) | Run in thread | 10s; on timeout, use default model |
| Session + PII + code policy (chat stream) | asyncio.gather wrapped in wait_for | 15s; on timeout, 503 |
PII detect (POST /api/detect) | Session/tokenize in thread; whole handler | 15s; on timeout, 503 |
These prevent one slow DB or external call from blocking the event loop and causing ALB timeouts.
ALB and target health
- ALB idle_timeout: 300s (connection to client).
- Target group: Backend health check
/health/liveevery 60s; unhealthy threshold 5. - Sticky sessions: 10 minutes for
/api/*(OAuth callback must hit same backend).
If a task is blocked (e.g. before the pool was added, or on a rare blocking path), it may stop responding to health checks and get dropped; new requests then go to healthy tasks.
Scaling checklist
- [ ] RDS: Monitor
DatabaseConnections; if you add many backend tasks, ensure (tasks × 10) < ~160 or raisemax_connections. - [ ] Backend tasks: Increase
backend_desired_count/backend_max_countfor more throughput; stay within RDS connection budget. - [ ] Backend size: Increase
backend_cpu/backend_memoryfor more throughput per task without more connections. - [ ] Alarms:
*-rds-connections-high(160),*-alb-backend-unhealthy,*-alb-5xx-errors; act before hitting limits.
See also
- AWS ECS deployment — architecture and setup
- Production deployment — production-specific notes
- Troubleshooting performance — latency and timeouts