Skip to content

AI Protection Admin Full Lab ​

Eight steps. About 35 minutes. All calls use the org-admin AI Protection API under /api/admin/ai-protection as implemented in securechat/backend/app/routers/ai_protection_admin.py.

You will verify protection status, read and update organization policy (including audit logging), pull time-bounded metrics, discover Bedrock referee models, update referee configuration (triggering a hot-swap), inspect referee health, run an embedding rebuild job and poll it to completion, and confirm ai_protection_policy_updated in the org admin audit log.


Prerequisites ​

  • [ ] GhostAI backend reachable (example: http://localhost:8000).
  • [ ] API key for a user who is organization owner or admin (require_admin).
  • [ ] curl and python3 -m json.tool (or jq) installed.
  • [ ] Optional: private inference (Ollama/vLLM) and Bedrock credentials on the server for richer Demo responses.

Export variables used in every step:

bash
export GHOSTAI_API_KEY="gst_your_org_admin_key"
export BASE_URL="http://localhost:8000"

Sanity check:

bash
curl -s -H "Authorization: Bearer $GHOSTAI_API_KEY" \
  "$BASE_URL/health" | python3 -m json.tool

Step 1: Check protection status (~4 min) ​

Goal: Read private_model_available, private model health, inference target summary, and default cloud model for the current org.

bash
curl -s -H "Authorization: Bearer $GHOSTAI_API_KEY" \
  "$BASE_URL/api/admin/ai-protection/status" | python3 -m json.tool

Interpret the JSON:

FieldMeaning
private_model_availableOverall signal that private inference is usable (loaded or probe succeeded without target error).
private_model_statusObject with model, runtime, deploy_label, loaded, or null if unavailable.
inference_targetlocation, url, model, models_available, gpu, gpu_memory_gb, error, tls, tls_warning.
default_cloud_modelServer default model id when using cloud (read-only).

Success criteria: HTTP 200 and a coherent JSON document. Empty or null nested objects are valid when infrastructure is not configured.


Step 2: View and update org protection policy (~6 min) ​

Goal: GET current policy, PUT selective updates, confirm persistence with a second GET.

2.1 Read policy

bash
curl -s -H "Authorization: Bearer $GHOSTAI_API_KEY" \
  "$BASE_URL/api/admin/ai-protection/policy" | python3 -m json.tool

Expected keys: default_ai_model (cloud | private | custom), lock_ai_model, default_verify_mode (all | suspicious | off), lock_verify_mode.

2.2 Update policy

The server maps default_ai_model to internal storage: cloud → user, private → local, custom → dedicated.

bash
curl -s -X PUT -H "Authorization: Bearer $GHOSTAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "default_ai_model": "private",
    "lock_ai_model": true,
    "default_verify_mode": "all",
    "lock_verify_mode": true
  }' \
  "$BASE_URL/api/admin/ai-protection/policy" | python3 -m json.tool

2.3 Confirm

bash
curl -s -H "Authorization: Bearer $GHOSTAI_API_KEY" \
  "$BASE_URL/api/admin/ai-protection/policy" | python3 -m json.tool

Audit note: Any PUT that changes stored fields triggers db.log_admin_action with ai_protection_policy_updated. You will query the audit log in Step 9.


Step 3: View PII detection metrics (~4 min) ​

Goal: Pull org-scoped metrics for 7d, 30d, and 90d.

bash
for p in 7d 30d 90d; do
  echo "=== period=$p ==="
  curl -s -H "Authorization: Bearer $GHOSTAI_API_KEY" \
    "$BASE_URL/api/admin/ai-protection/metrics?period=$p" | python3 -m json.tool
done

Response shape:

FieldMeaning
periodEcho of 7d, 30d, or 90d.
total_sessions_protectedDistinct sessions with tokenization activity in range.
total_entities_detectedCount of tokenized audit rows in range (as defined by the implementation).
entities_by_type[{ "entity_type": "...", "count": N }, ...].
detection_model_usageFriendly map: private, cloud, custom user counts.
users_with_private_model, users_with_cloud, users_with_customAggregates aligned with stored user AI model settings.
user_overviewPer-user rows: user_id, email, name, ai_model, matches_org_default, entities_detected.

Success criteria: HTTP 200 for each period; empty arrays and zeros are valid on a fresh org.


Step 4: Discover referee models (Bedrock) (~3 min) ​

Goal: List models the server can see for the configured Bedrock region (uses IAM keys from server settings).

bash
curl -s -H "Authorization: Bearer $GHOSTAI_API_KEY" \
  "$BASE_URL/api/admin/ai-protection/referee/models" | python3 -m json.tool

Typical shapes:

  • When region and credentials work: { "models": [ ... ], "region": "us-east-1" }.
  • When region is unset: { "models": [], "message": "No Bedrock region configured" }.
  • On discovery errors: { "models": [], "error": "..." }.

Step 5: Configure referee (hot-swap) (~5 min) ​

Goal: Read current referee settings, apply an allowed update, observe hot-swap status in the response.

6.1 GET config

bash
curl -s -H "Authorization: Bearer $GHOSTAI_API_KEY" \
  "$BASE_URL/api/admin/ai-protection/referee/config" | python3 -m json.tool

Fields include: runtime, bedrock_region, bedrock_model, bedrock_endpoint_url, fallback, audit_bedrock, max_concurrent.

6.2 PUT config

Only keys present in the body are applied. Allowed keys: runtime, bedrock_region, bedrock_model, bedrock_endpoint_url, fallback, audit_bedrock, max_concurrent.

Safe lab change (raises concurrency cap without switching cloud regions):

bash
curl -s -X PUT -H "Authorization: Bearer $GHOSTAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"max_concurrent": 4}' \
  "$BASE_URL/api/admin/ai-protection/referee/config" | python3 -m json.tool

When at least one field changes, the server invokes rebuild_referee. The response includes an updated object; if the swap ran, updated may contain status (ready or failed: …).

Production warning: Referee configuration is process-wide. Coordinate changes on shared infrastructure.


Step 6: Check referee health (~3 min) ​

Goal: Inspect referee LLM load state, Bedrock subsection when applicable, and circuit metrics.

bash
curl -s -H "Authorization: Bearer $GHOSTAI_API_KEY" \
  "$BASE_URL/api/admin/ai-protection/referee/health" | python3 -m json.tool

Typical keys:

  • runtime, status, loaded, fallback
  • If runtime is bedrock: bedrock.region, bedrock.model, bedrock.audit_enabled
  • metrics.circuit_state when the metrics module exposes the referee circuit gauge

Compare with Step 1 for GPU/URL detail — /status remains the home for inference_target and private stack probes.


Step 7: Trigger embedding rebuild and track the job (~7 min) ​

Goal: Start a re-embedding migration and poll until completed or a failure string.

8.1 Start job

target_backend: local or bedrock.
scope: all, memory, or content-library.
batch_size: positive integer (default in API is 100).

Local / all scopes (may be CPU-heavy on small dev machines):

bash
curl -s -X POST -H "Authorization: Bearer $GHOSTAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "target_backend": "local",
    "scope": "memory",
    "batch_size": 32
  }' \
  "$BASE_URL/api/admin/ai-protection/embeddings/rebuild" | python3 -m json.tool

Example start response:

json
{
  "job_id": "a1b2c3d4",
  "status": "started",
  "scope": "memory",
  "target_backend": "local"
}

Copy job_id into the next command.

8.2 Poll status

bash
JOB_ID="a1b2c3d4"

curl -s -H "Authorization: Bearer $GHOSTAI_API_KEY" \
  "$BASE_URL/api/admin/ai-protection/embeddings/rebuild/$JOB_ID" | python3 -m json.tool

While running: status is running; fields include total, completed, failed, target_backend, scope.

Done: status becomes completed or failed: <reason>.

Note: Job state is held in process memory (_reembed_jobs). Restarting the backend clears unknown job IDs; you would then receive {"error": "Job not found", "job_id": "..."}.


Step 8: Review the admin audit log (~5 min) ​

Goal: Confirm ai_protection_policy_updated after Step 2.

Org admin audit entries are served from the main admin router:

bash
curl -s -H "Authorization: Bearer $GHOSTAI_API_KEY" \
  "$BASE_URL/api/admin/audit-log?limit=50" | python3 -m json.tool

Each entry includes timestamp, action_type, admin_user_id, target_user_id, organization_id, details.

Filter with jq (if installed):

bash
curl -s -H "Authorization: Bearer $GHOSTAI_API_KEY" \
  "$BASE_URL/api/admin/audit-log?limit=100" \
  | jq '.[] | select(.action_type == "ai_protection_policy_updated")'

Success criteria: At least one row whose action_type is ai_protection_policy_updated and whose details mentions the updated policy fields (for example default_ai_detection_model, lock_ai_detection_model, default_verify_mode, lock_verify_mode).


Appendix — Endpoint summary ​

StepMethodPath
1GET/api/admin/ai-protection/status
2GET, PUT/api/admin/ai-protection/policy
3GET/api/admin/ai-protection/metrics?period=30d (also 7d, 90d)
4GET/api/admin/ai-protection/referee/models
5GET, PUT/api/admin/ai-protection/referee/config
6GET/api/admin/ai-protection/referee/health
7POST/api/admin/ai-protection/embeddings/rebuild
7GET/api/admin/ai-protection/embeddings/rebuild/{job_id}
8GET/api/admin/audit-log

Authentication: Authorization: Bearer $GHOSTAI_API_KEY on every request.


Troubleshooting ​

SymptomLikely cause
HTTP 403 on AI Protection routesUser is not org admin or owner, or lacks org membership.
Empty Bedrock model listpii_referee_bedrock_region unset, invalid credentials, or network/AWS errors—see message / error on /referee/models.
Job not found for embeddingsBackend restarted; start a new job.

SOC 2 Type II Certified | BAA Available | HIPAA-Ready