Appearance
SIEM Integration Full Lab
Ten steps. Forty-five minutes. From empty org to governed SIEM telemetry.
This lab exercises the GhostAI SIEM integration entirely through the Admin API (/api/admin/siem). You will list vendor presets, create a webhook integration with X-GhostAI-Reauth-Token, browse the event catalog, send test and validation burst traffic, inspect health and delivery artifacts, enable pull ingestion with a Bearer pull token, preview formatted output, generate compliance reports, and run lifecycle operations (enable, disable, clone, rotate credentials, maintenance, migration).
Prerequisites
Before starting, ensure you have:
- [ ] Enterprise tier organization with the
siem_integrationfeature enabled - [ ] Admin API access — a Bearer token (
GHOSTAI_API_KEY) for a user in the target organization - [ ] Re-authentication token — value for
X-GHOSTAI-REAUTH-TOKEN(sensitive operations) - [ ] Target SIEM endpoint — HTTPS URL that accepts POST (e.g. Splunk HEC, mock server, or
https://httpbin.org/postfor format inspection only) - [ ]
curlandpython3 -m json.tool(orjq) installed - [ ] Optional: second terminal to watch your SIEM or mock receiver logs
Environment variables
bash
export GHOSTAI_BASE_URL="https://your-ghostai-host" # or http://localhost:8000
export GHOSTAI_API_KEY="gst_..." # org admin API key
export X_GHOSTAI_REAUTH_TOKEN="..." # re-auth token for create/deleteVerify prerequisites
bash
curl -s -H "Authorization: Bearer $GHOSTAI_API_KEY" \
"$GHOSTAI_BASE_URL/api/admin/siem/presets" | python3 -m json.toolExpected: JSON array of presets. If you receive 403 or a feature error, confirm Enterprise + siem_integration before continuing.
Step 1: Explore Presets
Duration: 4 minutes
Goal: Discover all SIEM presets, default formats, and deep metadata for Splunk.
1.1 List all presets
bash
curl -s -H "Authorization: Bearer $GHOSTAI_API_KEY" \
"$GHOSTAI_BASE_URL/api/admin/siem/presets" | python3 -m json.toolExpected: Entries including splunk, sentinel, datadog, elastic, qradar, chronicle, logscale, cortex_xsiam, sumologic, panther, exabeam, logrhythm, securonix, devo, aws_security_lake, and custom. Each item includes id, name, logo, and format.
1.2 Preset detail and setup fields
bash
curl -s -H "Authorization: Bearer $GHOSTAI_API_KEY" \
"$GHOSTAI_BASE_URL/api/admin/siem/presets/splunk" | python3 -m json.toolExpected: setup_steps, setup_fields, endpoint_hint, content_pack filename (if any), error_remediation map, and format defaulting to splunk_hec.
1.3 Optional: download a content pack
Only succeeds when the file exists on the server (SIEM_CONTENT_PACKS_PATH):
bash
curl -s -O -J -H "Authorization: Bearer $GHOSTAI_API_KEY" \
"$GHOSTAI_BASE_URL/api/admin/siem/content-packs/splunk"Step 2: Create a Webhook Integration
Duration: 6 minutes
Goal: Create a Splunk-oriented webhook integration; observe re-auth and encrypted credential storage (masked in responses).
Replace YOUR_HEC_URL and YOUR_HEC_TOKEN with real values (or use a mock URL for lab-only format testing).
bash
curl -s -X POST "$GHOSTAI_BASE_URL/api/admin/siem/integrations" \
-H "Authorization: Bearer $GHOSTAI_API_KEY" \
-H "X-GhostAI-Reauth-Token: $X_GHOSTAI_REAUTH_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"name": "Lab Splunk HEC",
"siem_preset": "splunk",
"delivery_type": "webhook",
"endpoint_url": "YOUR_HEC_URL",
"auth_type": "bearer",
"auth_value": "YOUR_HEC_TOKEN",
"format": "splunk_hec",
"batch_size": 100,
"flush_interval_seconds": 5,
"minimization_policy": "standard",
"event_filters": {}
}' | python3 -m json.toolExpected: HTTP 200 success body with an id field (integration UUID). auth_value appears masked (e.g. ••••abcd).
Save the integration ID:
bash
export SIEM_INTEGRATION_ID="<id-from-response>"Notes
minimization_policy:standard,minimal, orfullcontrols field richness in outbound events.event_filters: Arbitrary JSON object stored on the integration; extend with keys your organization standardizes (for example allowedcategoryorevent_typelists) when the dispatcher honors them.- Splunk preset sets HEC-friendly defaults; explicit
format": "splunk_hec"matches the demo script.
Step 3: Browse the Event Catalog
Duration: 5 minutes
Goal: Inventory streamable events, categories, detections, and one full event definition.
3.1 Categories with counts
bash
curl -s -H "Authorization: Bearer $GHOSTAI_API_KEY" \
"$GHOSTAI_BASE_URL/api/admin/siem/catalog/categories" | python3 -m json.tool3.2 All events (optional category filter)
bash
curl -s -H "Authorization: Bearer $GHOSTAI_API_KEY" \
"$GHOSTAI_BASE_URL/api/admin/siem/catalog" | python3 -m json.tool
curl -s -H "Authorization: Bearer $GHOSTAI_API_KEY" \
"$GHOSTAI_BASE_URL/api/admin/siem/catalog?category=authentication" | python3 -m json.tool3.3 Event detail (MITRE, compliance, sample)
bash
curl -s -H "Authorization: Bearer $GHOSTAI_API_KEY" \
"$GHOSTAI_BASE_URL/api/admin/siem/catalog/auth.login" | python3 -m json.toolExpected: mitre_attack, compliance array, sample_payload, fields, and description.
3.4 Detection rules bundle
bash
curl -s -H "Authorization: Bearer $GHOSTAI_API_KEY" \
"$GHOSTAI_BASE_URL/api/admin/siem/catalog/detections" | python3 -m json.toolStep 4: Test the Integration
Duration: 3 minutes
Goal: Send a single siem.test_sent event through the integration’s formatter to the remote endpoint.
bash
curl -s -X POST \
-H "Authorization: Bearer $GHOSTAI_API_KEY" \
"$GHOSTAI_BASE_URL/api/admin/siem/integrations/$SIEM_INTEGRATION_ID/test" | python3 -m json.toolExpected: http_status in the 2xx range for a healthy collector; response contains the first ~500 characters of the SIEM reply (e.g. Splunk {"text":"Success","code":0}).
Step 5: Run a Validation Burst
Duration: 4 minutes
Goal: POST a batch containing synthetic examples for every catalog event type using the integration’s active format.
bash
curl -s -X POST \
-H "Authorization: Bearer $GHOSTAI_API_KEY" \
"$GHOSTAI_BASE_URL/api/admin/siem/integrations/$SIEM_INTEGRATION_ID/validation-burst" | python3 -m json.toolExpected: http_status, response snippet, and events_sent equal to the number of entries in the platform catalog (large integer). If the receiver rejects oversized payloads, reduce batch-related settings on the integration or use a receiver with higher limits.
Step 6: Monitor Health
Duration: 6 minutes
Goal: Read aggregate health, score, delivery history, and actionable errors.
6.1 Health summary
bash
curl -s -H "Authorization: Bearer $GHOSTAI_API_KEY" \
"$GHOSTAI_BASE_URL/api/admin/siem/integrations/$SIEM_INTEGRATION_ID/health" | python3 -m json.toolExpected: events_sent, events_failed, events_expired, success_rate, latency fields, events_per_minute.
6.2 Health score
bash
curl -s -H "Authorization: Bearer $GHOSTAI_API_KEY" \
"$GHOSTAI_BASE_URL/api/admin/siem/integrations/$SIEM_INTEGRATION_ID/health/score" | python3 -m json.tool6.3 Recent deliveries
bash
curl -s -H "Authorization: Bearer $GHOSTAI_API_KEY" \
"$GHOSTAI_BASE_URL/api/admin/siem/integrations/$SIEM_INTEGRATION_ID/deliveries" | python3 -m json.tool6.4 Errors with remediation hints
bash
curl -s -H "Authorization: Bearer $GHOSTAI_API_KEY" \
"$GHOSTAI_BASE_URL/api/admin/siem/integrations/$SIEM_INTEGRATION_ID/errors" | python3 -m json.toolExpected: Each failed row may include remediation text from the preset’s playbook.
6.5 Webhook log (optional)
bash
curl -s -H "Authorization: Bearer $GHOSTAI_API_KEY" \
"$GHOSTAI_BASE_URL/api/admin/siem/integrations/$SIEM_INTEGRATION_ID/webhook-log?limit=20" | python3 -m json.tool6.6 Reconciliation and gaps
bash
curl -s -H "Authorization: Bearer $GHOSTAI_API_KEY" \
"$GHOSTAI_BASE_URL/api/admin/siem/integrations/$SIEM_INTEGRATION_ID/reconciliation?start_date=2026-03-01&end_date=2026-03-29" | python3 -m json.tool
curl -s -H "Authorization: Bearer $GHOSTAI_API_KEY" \
"$GHOSTAI_BASE_URL/api/admin/siem/integrations/$SIEM_INTEGRATION_ID/gaps" | python3 -m json.toolStep 7: Configure the Pull API
Duration: 5 minutes
Goal: Mint a pull token and page events with cursor / limit. Pull uses Bearer token auth and does not reuse the user API key.
7.1 Rotate (create) pull token
bash
curl -s -X POST \
-H "Authorization: Bearer $GHOSTAI_API_KEY" \
"$GHOSTAI_BASE_URL/api/admin/siem/integrations/$SIEM_INTEGRATION_ID/rotate-pull-token" | python3 -m json.toolExpected: pull_token string shown once. Store it securely.
bash
export SIEM_PULL_TOKEN="<pull_token-from-response>"7.2 Pull events
bash
curl -s -H "Authorization: Bearer $SIEM_PULL_TOKEN" \
"$GHOSTAI_BASE_URL/api/admin/siem/pull?cursor=0&limit=10" | python3 -m json.toolExpected: JSON with events, next_cursor, and has_more. Advance cursor to next_cursor until has_more is false.
Step 8: Event Filtering and Format Preview
Duration: 4 minutes
Goal: Inspect queued payloads reformatted as they would be sent on the wire.
The preview endpoint reads recent rows from the organization’s SIEM queue and applies the integration’s effective format (preset override included).
bash
curl -s -H "Authorization: Bearer $GHOSTAI_API_KEY" \
"$GHOSTAI_BASE_URL/api/admin/siem/integrations/$SIEM_INTEGRATION_ID/preview" | python3 -m json.toolExpected: format (e.g. splunk_hec) and events array of formatted strings or objects per FORMAT_REGISTRY.
Optional: compare envelope vs. wire format
bash
curl -s -H "Authorization: Bearer $GHOSTAI_API_KEY" \
"$GHOSTAI_BASE_URL/api/admin/siem/schema" | python3 -m json.tool
curl -s -H "Authorization: Bearer $GHOSTAI_API_KEY" \
"$GHOSTAI_BASE_URL/api/admin/siem/schema/splunk_hec" | python3 -m json.toolTo experiment with cef, leef, sentinel, or ocsf, create a second integration with "siem_preset": "custom" and set "format" accordingly, then re-run preview.
Step 9: Compliance Reporting
Duration: 4 minutes
Goal: Generate framework-scoped coverage maps for SOC 2, HIPAA, PCI-DSS, NIST 800-53, and GDPR.
9.1 SOC 2 report
bash
curl -s -H "Authorization: Bearer $GHOSTAI_API_KEY" \
"$GHOSTAI_BASE_URL/api/admin/siem/compliance/report?framework=soc2" | python3 -m json.tool9.2 Other frameworks
bash
for fw in hipaa pci_dss nist_800_53 gdpr; do
echo "=== $fw ==="
curl -s -H "Authorization: Bearer $GHOSTAI_API_KEY" \
"$GHOSTAI_BASE_URL/api/admin/siem/compliance/report?framework=$fw&start_date=2026-03-01&end_date=2026-03-29" | python3 -m json.tool
doneExpected: coverage_percentage, event_types keyed by event_type with controls and severity, and reconciliation data when the backend supplies rows for the org.
Step 10: Integration Lifecycle
Duration: 6 minutes
Goal: Practice operational controls—pause, resume, clone, rotate webhook secret, maintenance windows, and parallel migration.
10.1 Disable and enable (pause / resume delivery)
bash
curl -s -X POST -H "Authorization: Bearer $GHOSTAI_API_KEY" \
"$GHOSTAI_BASE_URL/api/admin/siem/integrations/$SIEM_INTEGRATION_ID/disable" | python3 -m json.tool
curl -s -X POST -H "Authorization: Bearer $GHOSTAI_API_KEY" \
"$GHOSTAI_BASE_URL/api/admin/siem/integrations/$SIEM_INTEGRATION_ID/enable" | python3 -m json.tool10.2 Clone
bash
curl -s -X POST \
-H "Authorization: Bearer $GHOSTAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{"name":"Lab Splunk HEC – clone"}' \
"$GHOSTAI_BASE_URL/api/admin/siem/integrations/$SIEM_INTEGRATION_ID/clone" | python3 -m json.toolSave the new id if you continue experimenting.
10.3 Rotate webhook credentials (validates against endpoint)
bash
curl -s -X POST \
-H "Authorization: Bearer $GHOSTAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{"auth_value":"YOUR_NEW_HEC_TOKEN"}' \
"$GHOSTAI_BASE_URL/api/admin/siem/integrations/$SIEM_INTEGRATION_ID/rotate-credentials" | python3 -m json.toolExpected: Success only if a test POST with the new secret returns HTTP 2xx.
10.4 Maintenance window
bash
curl -s -X POST \
-H "Authorization: Bearer $GHOSTAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{"start_time":"2026-03-30T02:00:00Z","end_time":"2026-03-30T04:00:00Z"}' \
"$GHOSTAI_BASE_URL/api/admin/siem/integrations/$SIEM_INTEGRATION_ID/maintenance" | python3 -m json.tool
curl -s -X POST -H "Authorization: Bearer $GHOSTAI_API_KEY" \
"$GHOSTAI_BASE_URL/api/admin/siem/integrations/$SIEM_INTEGRATION_ID/maintenance/cancel" | python3 -m json.tool10.5 Migration (parallel integration)
Creates a second integration for cutover testing:
bash
curl -s -X POST \
-H "Authorization: Bearer $GHOSTAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"target_preset": "elastic",
"target_endpoint_url": "https://elastic.example.com:9200/_bulk",
"target_auth_value": "YOUR_ELASTIC_API_KEY"
}' \
"$GHOSTAI_BASE_URL/api/admin/siem/integrations/$SIEM_INTEGRATION_ID/migrate" | python3 -m json.toolExpected: source and target integration objects (masked auth) with status: parallel_run.
To finalize cutover, call POST .../integrations/{target_id}/migrate/complete with JSON body true or false for decommission_source (the {target_id} is the new integration whose name starts with [Migration]).
10.6 Delete (requires re-auth)
bash
curl -s -X DELETE \
-H "Authorization: Bearer $GHOSTAI_API_KEY" \
-H "X-GhostAI-Reauth-Token: $X_GHOSTAI_REAUTH_TOKEN" \
"$GHOSTAI_BASE_URL/api/admin/siem/integrations/$SIEM_INTEGRATION_ID" | python3 -m json.toolSkip 10.6 if you want to keep the lab integration for further testing.
Troubleshooting
403 with reauth_required
Cause: Create, delete, or auth-value update without X-GhostAI-Reauth-Token.
Fix: Add the header with a valid re-auth token from your security policy.
401 on GET /api/admin/siem/pull
Cause: Wrong token type—user API keys are not pull tokens.
Fix: Call rotate-pull-token, then use Authorization: Bearer <pull_token> only on /pull.
Validation burst times out or returns 413 / 500
Cause: Receiver body size limits or slow SIEM.
Fix: Lower batch_size on the integration, use a staging collector, or filter events to reduce catalog breadth in testing environments.
Compliance report 400 Unknown framework
Cause: Invalid framework query value.
Fix: Use exactly hipaa, soc2, pci_dss, nist_800_53, or gdpr.
Preview returns empty events
Cause: No rows in siem_event_queue for the organization yet.
Fix: Generate traffic (chat, admin actions, or test/burst endpoints), then retry.
rotate-credentials fails with 400 and HTTP status in message
Cause: New secret failed live test POST against endpoint_url.
Fix: Verify token, URL, TLS, and network path before rotating.
Lab Summary
| Step | Topic | Duration |
|---|---|---|
| 1 | Explore presets (GET /presets, detail, content pack) | 4 min |
| 2 | Create webhook integration (POST /integrations + re-auth) | 6 min |
| 3 | Browse event catalog (/catalog, categories, detail, detections) | 5 min |
| 4 | Test integration (POST .../test) | 3 min |
| 5 | Validation burst (POST .../validation-burst) | 4 min |
| 6 | Monitor health (health, score, deliveries, errors, reconciliation, gaps) | 6 min |
| 7 | Configure pull API (rotate-pull-token, GET /pull) | 5 min |
| 8 | Filtering & format preview (GET .../preview, schema) | 4 min |
| 9 | Compliance reporting (GET /compliance/report) | 4 min |
| 10 | Lifecycle (disable/enable, clone, rotate-credentials, maintenance, migrate, delete) | 6 min |
| Total | ~47 min |
What's Next
- Customer-facing narrative: Practice the SIEM Integration Demo Script.
- Schema reference:
GET /api/admin/siem/schemaandGET /api/admin/siem/schema/{format}for envelope vs. wire formats. - MSP aggregation: If your org is an MSP parent, explore
GET /api/admin/siem/msp/eventswithchild_org_idafter backend enablement.
Event envelope fields: id, timestamp, organization_id, event_type, category, severity, outcome, actor, resource, details, compliance (plus optional sequence, prev_hash, source_table, signature). Supported wire formats: json, cef, leef, splunk_hec, sentinel, ocsf. Minimization: standard, minimal, full.