Skip to content

API Key Health Monitoring ​

Overview ​

GhostAI runs scheduled health checks against all 15 supported providers (12 LLM + 3 Search) to detect problems before they impact your workflows. An APScheduler background job validates each key tier on a fixed cadence β€” cloud keys every 15 minutes, organization keys every 30 minutes, and user keys once per hour. Each check makes a lightweight API call to the provider, tracks response latency and quota usage, and triggers alerts when issues arise.

Availability

Health monitoring is available on Team, Business, and Enterprise tiers. Starter and Personal users can trigger on-demand health checks from the provider key settings.

Health States ​

Each provider key has one of four health states:

StateIconMeaningCommon Causes
Healthy🟒Key is valid and performing normallySuccessful validation with normal latency
Degraded🟑Key works but has warningsLow quota (<20% remaining) or high latency (>2s)
UnhealthyπŸ”΄Key is invalid or failingRevoked key, authentication error, billing issue
UnknownβšͺKey has not been checked yetNewly added key, monitoring not yet run

State Transitions ​

                    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
           β”Œβ”€β”€β”€β”€β”€β”€β–Άβ”‚   Unknown    │──────┐
           β”‚       β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜      β”‚
           β”‚                              β–Ό
 β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”                 β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
 β”‚  Unhealthy   │◀───────────────│   Healthy     β”‚
 β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜                 β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜
        β”‚                                 β”‚
        β”‚       β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”          β”‚
        └──────▢│   Degraded   β”‚β—€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Keys can transition between any states based on check results.

Monitoring Schedule by Tier ​

Key TierCheck FrequencyDescription
CloudEvery 15 minutesPlatform-wide default keys β€” highest priority
Tenant (Organization)Every 30 minutesOrganization shared keys
User (Personal)Hourly (at :05 past)Individual user keys

On-Demand Checks

You can trigger an immediate health check at any time by clicking the Check button on a provider card. This bypasses the schedule and runs the check instantly.

What Gets Checked ​

Each health check performs a lightweight API call to the provider:

LLM Providers ​

ProviderCheck MethodWhat It Validates
OpenAIGET /v1/modelsAuthentication, key status
AnthropicGET /v1/models (Anthropic API)Authentication, key status
Google Geminimodels.list() (GenAI SDK)Authentication, key status
AWS BedrockBedrock runtime / list-foundation-modelsAWS credentials, region access
Mistral AIGET /v1/models (OpenAI-compat)Authentication, key status
CohereGET /v2/modelsAuthentication, key status
PerplexityGET /v1/models (OpenAI-compat)Authentication, key status
xAI (Grok)GET /v1/models (OpenAI-compat)Authentication, key status
DeepSeekGET /v1/models (OpenAI-compat)Authentication, key status
GroqGET /v1/models (OpenAI-compat)Authentication, key status
Together AIGET /v1/models (OpenAI-compat)Authentication, key status
OpenRouterGET /modelsAuthentication, credit balance

Search Providers ​

ProviderCheck MethodWhat It Validates
TavilyMinimal search queryAuthentication, search quota
Brave SearchMinimal search queryAuthentication, subscription
SerperMinimal search queryAuthentication, key status

OpenAI-Compatible Validation

Many newer providers (Mistral, Perplexity, xAI, DeepSeek, Groq, Together) use OpenAI-compatible APIs. GhostAI validates these by calling their /v1/models endpoint with the provider's API key and base URL. All 15 providers listed above have dedicated validators in key_health_service.py β€” no provider is skipped during scheduled checks.

Alerts ​

Health check failures trigger an escalating alert process:

Consecutive FailuresAction
1Key marked as unhealthy in the dashboard
2In-app quality alert created; email sent to admins (cloud/tenant keys)
3+Critical log entry for cloud-tier keys
RecoveryAlerts auto-resolved; failure counter reset

Cloud Key Failures Are Critical

When a cloud-tier key fails, it affects all users on the platform. Cloud key failures are treated as critical severity and generate immediate admin notifications.

Email Notifications ​

For cloud and tenant-tier keys with 2+ consecutive failures, GhostAI sends an email to the configured admin address containing:

  • Provider name and tier
  • Masked key display (e.g., sk-proj...7x4q)
  • Error message
  • Number of consecutive failures

Quota Tracking ​

For providers that report usage information, GhostAI tracks quota data:

FieldDescription
Quota RemainingCredits or requests remaining in your plan
Quota LimitTotal credits or requests in your plan
Usage BarVisual indicator in the provider card

Quota Thresholds ​

Usage LevelBar ColorStatus Impact
> 50% remaining🟒 GreenHealthy
20-50% remaining🟑 AmberHealthy (watch zone)
< 20% remainingπŸ”΄ RedDegraded
0% remainingπŸ”΄ RedDegraded (rate limited)

Viewing Health Status ​

Provider Keys Tab ​

Each provider card in Settings > Provider Keys shows:

  • Health dot β€” colored indicator (green/amber/red/gray)
  • Health message β€” description of the current state
  • Last check β€” relative timestamp (e.g., "5m ago")
  • Consecutive failures β€” count of sequential failures (if > 0)
  • Quota bar β€” visual remaining/total display

Health History ​

Click the History button on a provider card to view recent health check results:

  • Status (healthy/degraded/unhealthy)
  • Timestamp of each check
  • Response latency in milliseconds
  • Error messages (if any)
  • Quota at time of check

History Retention

Health check history is retained for 90 days and automatically purged nightly at 3:00 AM UTC to prevent unbounded database growth.

Key Expiry Monitoring ​

In addition to health checks, GhostAI monitors key expiry dates:

ConditionDisplayAction
Key expires in > reminder periodNo warningNormal operation
Key expires within reminder periodAmber "Expiring" badge with days remainingPlan key rotation
Key has expiredRed "Expired" badgeRotate key immediately

Expiry warnings appear on the provider card alongside health indicators. Set expiry dates and reminder periods when adding or updating a key. See Key Rotation & Expiry for details.

Health API ​

Programmatic access to health data is available via the API:

bash
# Get health summary for all providers
GET /api/settings/provider-keys/health

# Get health detail for a specific provider
GET /api/settings/provider-keys/health/{provider}

# Get health check history
GET /api/settings/provider-keys/health/{provider}/history

# Trigger an immediate health check
POST /api/settings/provider-keys/{tier}/{provider}/check

See the Provider Keys API Reference for full details.

SOC 2 Type II Certified | BAA Available | HIPAA-Ready