Appearance
API Key Health Monitoring β
Overview β
GhostAI runs scheduled health checks against all 15 supported providers (12 LLM + 3 Search) to detect problems before they impact your workflows. An APScheduler background job validates each key tier on a fixed cadence β cloud keys every 15 minutes, organization keys every 30 minutes, and user keys once per hour. Each check makes a lightweight API call to the provider, tracks response latency and quota usage, and triggers alerts when issues arise.
Availability
Health monitoring is available on Team, Business, and Enterprise tiers. Starter and Personal users can trigger on-demand health checks from the provider key settings.
Health States β
Each provider key has one of four health states:
| State | Icon | Meaning | Common Causes |
|---|---|---|---|
| Healthy | π’ | Key is valid and performing normally | Successful validation with normal latency |
| Degraded | π‘ | Key works but has warnings | Low quota (<20% remaining) or high latency (>2s) |
| Unhealthy | π΄ | Key is invalid or failing | Revoked key, authentication error, billing issue |
| Unknown | βͺ | Key has not been checked yet | Newly added key, monitoring not yet run |
State Transitions β
ββββββββββββββββ
ββββββββΆβ Unknown ββββββββ
β ββββββββββββββββ β
β βΌ
βββββββββββ΄βββββ ββββββββββββββββ
β Unhealthy ββββββββββββββββββ Healthy β
ββββββββ¬ββββββββ ββββββββ¬ββββββββ
β β
β ββββββββββββββββ β
ββββββββΆβ Degraded ββββββββββββ
ββββββββββββββββKeys can transition between any states based on check results.
Monitoring Schedule by Tier β
| Key Tier | Check Frequency | Description |
|---|---|---|
| Cloud | Every 15 minutes | Platform-wide default keys β highest priority |
| Tenant (Organization) | Every 30 minutes | Organization shared keys |
| User (Personal) | Hourly (at :05 past) | Individual user keys |
On-Demand Checks
You can trigger an immediate health check at any time by clicking the Check button on a provider card. This bypasses the schedule and runs the check instantly.
What Gets Checked β
Each health check performs a lightweight API call to the provider:
LLM Providers β
| Provider | Check Method | What It Validates |
|---|---|---|
| OpenAI | GET /v1/models | Authentication, key status |
| Anthropic | GET /v1/models (Anthropic API) | Authentication, key status |
| Google Gemini | models.list() (GenAI SDK) | Authentication, key status |
| AWS Bedrock | Bedrock runtime / list-foundation-models | AWS credentials, region access |
| Mistral AI | GET /v1/models (OpenAI-compat) | Authentication, key status |
| Cohere | GET /v2/models | Authentication, key status |
| Perplexity | GET /v1/models (OpenAI-compat) | Authentication, key status |
| xAI (Grok) | GET /v1/models (OpenAI-compat) | Authentication, key status |
| DeepSeek | GET /v1/models (OpenAI-compat) | Authentication, key status |
| Groq | GET /v1/models (OpenAI-compat) | Authentication, key status |
| Together AI | GET /v1/models (OpenAI-compat) | Authentication, key status |
| OpenRouter | GET /models | Authentication, credit balance |
Search Providers β
| Provider | Check Method | What It Validates |
|---|---|---|
| Tavily | Minimal search query | Authentication, search quota |
| Brave Search | Minimal search query | Authentication, subscription |
| Serper | Minimal search query | Authentication, key status |
OpenAI-Compatible Validation
Many newer providers (Mistral, Perplexity, xAI, DeepSeek, Groq, Together) use OpenAI-compatible APIs. GhostAI validates these by calling their /v1/models endpoint with the provider's API key and base URL. All 15 providers listed above have dedicated validators in key_health_service.py β no provider is skipped during scheduled checks.
Alerts β
Health check failures trigger an escalating alert process:
| Consecutive Failures | Action |
|---|---|
| 1 | Key marked as unhealthy in the dashboard |
| 2 | In-app quality alert created; email sent to admins (cloud/tenant keys) |
| 3+ | Critical log entry for cloud-tier keys |
| Recovery | Alerts auto-resolved; failure counter reset |
Cloud Key Failures Are Critical
When a cloud-tier key fails, it affects all users on the platform. Cloud key failures are treated as critical severity and generate immediate admin notifications.
Email Notifications β
For cloud and tenant-tier keys with 2+ consecutive failures, GhostAI sends an email to the configured admin address containing:
- Provider name and tier
- Masked key display (e.g.,
sk-proj...7x4q) - Error message
- Number of consecutive failures
Quota Tracking β
For providers that report usage information, GhostAI tracks quota data:
| Field | Description |
|---|---|
| Quota Remaining | Credits or requests remaining in your plan |
| Quota Limit | Total credits or requests in your plan |
| Usage Bar | Visual indicator in the provider card |
Quota Thresholds β
| Usage Level | Bar Color | Status Impact |
|---|---|---|
| > 50% remaining | π’ Green | Healthy |
| 20-50% remaining | π‘ Amber | Healthy (watch zone) |
| < 20% remaining | π΄ Red | Degraded |
| 0% remaining | π΄ Red | Degraded (rate limited) |
Viewing Health Status β
Provider Keys Tab β
Each provider card in Settings > Provider Keys shows:
- Health dot β colored indicator (green/amber/red/gray)
- Health message β description of the current state
- Last check β relative timestamp (e.g., "5m ago")
- Consecutive failures β count of sequential failures (if > 0)
- Quota bar β visual remaining/total display
Health History β
Click the History button on a provider card to view recent health check results:
- Status (healthy/degraded/unhealthy)
- Timestamp of each check
- Response latency in milliseconds
- Error messages (if any)
- Quota at time of check
History Retention
Health check history is retained for 90 days and automatically purged nightly at 3:00 AM UTC to prevent unbounded database growth.
Key Expiry Monitoring β
In addition to health checks, GhostAI monitors key expiry dates:
| Condition | Display | Action |
|---|---|---|
| Key expires in > reminder period | No warning | Normal operation |
| Key expires within reminder period | Amber "Expiring" badge with days remaining | Plan key rotation |
| Key has expired | Red "Expired" badge | Rotate key immediately |
Expiry warnings appear on the provider card alongside health indicators. Set expiry dates and reminder periods when adding or updating a key. See Key Rotation & Expiry for details.
Health API β
Programmatic access to health data is available via the API:
bash
# Get health summary for all providers
GET /api/settings/provider-keys/health
# Get health detail for a specific provider
GET /api/settings/provider-keys/health/{provider}
# Get health check history
GET /api/settings/provider-keys/health/{provider}/history
# Trigger an immediate health check
POST /api/settings/provider-keys/{tier}/{provider}/checkSee the Provider Keys API Reference for full details.