Appearance
AI Protection Admin Full Lab
Eight steps. About 35 minutes. All calls use the org-admin AI Protection API under /api/admin/ai-protection as implemented in securechat/backend/app/routers/ai_protection_admin.py.
You will verify protection status, read and update organization policy (including audit logging), pull time-bounded metrics, discover Bedrock referee models, update referee configuration (triggering a hot-swap), inspect referee health, run an embedding rebuild job and poll it to completion, and confirm ai_protection_policy_updated in the org admin audit log.
Prerequisites
- [ ] GhostAI backend reachable (example:
http://localhost:8000). - [ ] API key for a user who is organization owner or admin (
require_admin). - [ ]
curlandpython3 -m json.tool(orjq) installed. - [ ] Optional: private inference (Ollama/vLLM) and Bedrock credentials on the server for richer Demo responses.
Export variables used in every step:
bash
export GHOSTAI_API_KEY="gst_your_org_admin_key"
export BASE_URL="http://localhost:8000"Sanity check:
bash
curl -s -H "Authorization: Bearer $GHOSTAI_API_KEY" \
"$BASE_URL/health" | python3 -m json.toolStep 1: Check protection status (~4 min)
Goal: Read private_model_available, private model health, inference target summary, and default cloud model for the current org.
bash
curl -s -H "Authorization: Bearer $GHOSTAI_API_KEY" \
"$BASE_URL/api/admin/ai-protection/status" | python3 -m json.toolInterpret the JSON:
| Field | Meaning |
|---|---|
private_model_available | Overall signal that private inference is usable (loaded or probe succeeded without target error). |
private_model_status | Object with model, runtime, deploy_label, loaded, or null if unavailable. |
inference_target | location, url, model, models_available, gpu, gpu_memory_gb, error, tls, tls_warning. |
default_cloud_model | Server default model id when using cloud (read-only). |
Success criteria: HTTP 200 and a coherent JSON document. Empty or null nested objects are valid when infrastructure is not configured.
Step 2: View and update org protection policy (~6 min)
Goal: GET current policy, PUT selective updates, confirm persistence with a second GET.
2.1 Read policy
bash
curl -s -H "Authorization: Bearer $GHOSTAI_API_KEY" \
"$BASE_URL/api/admin/ai-protection/policy" | python3 -m json.toolExpected keys: default_ai_model (cloud | private | custom), lock_ai_model, default_verify_mode (all | suspicious | off), lock_verify_mode.
2.2 Update policy
The server maps default_ai_model to internal storage: cloud → user, private → local, custom → dedicated.
bash
curl -s -X PUT -H "Authorization: Bearer $GHOSTAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"default_ai_model": "private",
"lock_ai_model": true,
"default_verify_mode": "all",
"lock_verify_mode": true
}' \
"$BASE_URL/api/admin/ai-protection/policy" | python3 -m json.tool2.3 Confirm
bash
curl -s -H "Authorization: Bearer $GHOSTAI_API_KEY" \
"$BASE_URL/api/admin/ai-protection/policy" | python3 -m json.toolAudit note: Any PUT that changes stored fields triggers db.log_admin_action with ai_protection_policy_updated. You will query the audit log in Step 9.
Step 3: View PII detection metrics (~4 min)
Goal: Pull org-scoped metrics for 7d, 30d, and 90d.
bash
for p in 7d 30d 90d; do
echo "=== period=$p ==="
curl -s -H "Authorization: Bearer $GHOSTAI_API_KEY" \
"$BASE_URL/api/admin/ai-protection/metrics?period=$p" | python3 -m json.tool
doneResponse shape:
| Field | Meaning |
|---|---|
period | Echo of 7d, 30d, or 90d. |
total_sessions_protected | Distinct sessions with tokenization activity in range. |
total_entities_detected | Count of tokenized audit rows in range (as defined by the implementation). |
entities_by_type | [{ "entity_type": "...", "count": N }, ...]. |
detection_model_usage | Friendly map: private, cloud, custom user counts. |
users_with_private_model, users_with_cloud, users_with_custom | Aggregates aligned with stored user AI model settings. |
user_overview | Per-user rows: user_id, email, name, ai_model, matches_org_default, entities_detected. |
Success criteria: HTTP 200 for each period; empty arrays and zeros are valid on a fresh org.
Step 4: Discover referee models (Bedrock) (~3 min)
Goal: List models the server can see for the configured Bedrock region (uses IAM keys from server settings).
bash
curl -s -H "Authorization: Bearer $GHOSTAI_API_KEY" \
"$BASE_URL/api/admin/ai-protection/referee/models" | python3 -m json.toolTypical shapes:
- When region and credentials work:
{ "models": [ ... ], "region": "us-east-1" }. - When region is unset:
{ "models": [], "message": "No Bedrock region configured" }. - On discovery errors:
{ "models": [], "error": "..." }.
Step 5: Configure referee (hot-swap) (~5 min)
Goal: Read current referee settings, apply an allowed update, observe hot-swap status in the response.
6.1 GET config
bash
curl -s -H "Authorization: Bearer $GHOSTAI_API_KEY" \
"$BASE_URL/api/admin/ai-protection/referee/config" | python3 -m json.toolFields include: runtime, bedrock_region, bedrock_model, bedrock_endpoint_url, fallback, audit_bedrock, max_concurrent.
6.2 PUT config
Only keys present in the body are applied. Allowed keys: runtime, bedrock_region, bedrock_model, bedrock_endpoint_url, fallback, audit_bedrock, max_concurrent.
Safe lab change (raises concurrency cap without switching cloud regions):
bash
curl -s -X PUT -H "Authorization: Bearer $GHOSTAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{"max_concurrent": 4}' \
"$BASE_URL/api/admin/ai-protection/referee/config" | python3 -m json.toolWhen at least one field changes, the server invokes rebuild_referee. The response includes an updated object; if the swap ran, updated may contain status (ready or failed: …).
Production warning: Referee configuration is process-wide. Coordinate changes on shared infrastructure.
Step 6: Check referee health (~3 min)
Goal: Inspect referee LLM load state, Bedrock subsection when applicable, and circuit metrics.
bash
curl -s -H "Authorization: Bearer $GHOSTAI_API_KEY" \
"$BASE_URL/api/admin/ai-protection/referee/health" | python3 -m json.toolTypical keys:
runtime,status,loaded,fallback- If
runtimeisbedrock:bedrock.region,bedrock.model,bedrock.audit_enabled metrics.circuit_statewhen the metrics module exposes the referee circuit gauge
Compare with Step 1 for GPU/URL detail — /status remains the home for inference_target and private stack probes.
Step 7: Trigger embedding rebuild and track the job (~7 min)
Goal: Start a re-embedding migration and poll until completed or a failure string.
8.1 Start job
target_backend: local or bedrock.scope: all, memory, or content-library.batch_size: positive integer (default in API is 100).
Local / all scopes (may be CPU-heavy on small dev machines):
bash
curl -s -X POST -H "Authorization: Bearer $GHOSTAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"target_backend": "local",
"scope": "memory",
"batch_size": 32
}' \
"$BASE_URL/api/admin/ai-protection/embeddings/rebuild" | python3 -m json.toolExample start response:
json
{
"job_id": "a1b2c3d4",
"status": "started",
"scope": "memory",
"target_backend": "local"
}Copy job_id into the next command.
8.2 Poll status
bash
JOB_ID="a1b2c3d4"
curl -s -H "Authorization: Bearer $GHOSTAI_API_KEY" \
"$BASE_URL/api/admin/ai-protection/embeddings/rebuild/$JOB_ID" | python3 -m json.toolWhile running: status is running; fields include total, completed, failed, target_backend, scope.
Done: status becomes completed or failed: <reason>.
Note: Job state is held in process memory (_reembed_jobs). Restarting the backend clears unknown job IDs; you would then receive {"error": "Job not found", "job_id": "..."}.
Step 8: Review the admin audit log (~5 min)
Goal: Confirm ai_protection_policy_updated after Step 2.
Org admin audit entries are served from the main admin router:
bash
curl -s -H "Authorization: Bearer $GHOSTAI_API_KEY" \
"$BASE_URL/api/admin/audit-log?limit=50" | python3 -m json.toolEach entry includes timestamp, action_type, admin_user_id, target_user_id, organization_id, details.
Filter with jq (if installed):
bash
curl -s -H "Authorization: Bearer $GHOSTAI_API_KEY" \
"$BASE_URL/api/admin/audit-log?limit=100" \
| jq '.[] | select(.action_type == "ai_protection_policy_updated")'Success criteria: At least one row whose action_type is ai_protection_policy_updated and whose details mentions the updated policy fields (for example default_ai_detection_model, lock_ai_detection_model, default_verify_mode, lock_verify_mode).
Appendix — Endpoint summary
| Step | Method | Path |
|---|---|---|
| 1 | GET | /api/admin/ai-protection/status |
| 2 | GET, PUT | /api/admin/ai-protection/policy |
| 3 | GET | /api/admin/ai-protection/metrics?period=30d (also 7d, 90d) |
| 4 | GET | /api/admin/ai-protection/referee/models |
| 5 | GET, PUT | /api/admin/ai-protection/referee/config |
| 6 | GET | /api/admin/ai-protection/referee/health |
| 7 | POST | /api/admin/ai-protection/embeddings/rebuild |
| 7 | GET | /api/admin/ai-protection/embeddings/rebuild/{job_id} |
| 8 | GET | /api/admin/audit-log |
Authentication: Authorization: Bearer $GHOSTAI_API_KEY on every request.
Troubleshooting
| Symptom | Likely cause |
|---|---|
| HTTP 403 on AI Protection routes | User is not org admin or owner, or lacks org membership. |
| Empty Bedrock model list | pii_referee_bedrock_region unset, invalid credentials, or network/AWS errors—see message / error on /referee/models. |
Job not found for embeddings | Backend restarted; start a new job. |