Skip to content

SIEM Integration Full Lab ​

Ten steps. Forty-five minutes. From empty org to governed SIEM telemetry.

This lab exercises the GhostAI SIEM integration entirely through the Admin API (/api/admin/siem). You will list vendor presets, create a webhook integration with X-GhostAI-Reauth-Token, browse the event catalog, send test and validation burst traffic, inspect health and delivery artifacts, enable pull ingestion with a Bearer pull token, preview formatted output, generate compliance reports, and run lifecycle operations (enable, disable, clone, rotate credentials, maintenance, migration).


Prerequisites ​

Before starting, ensure you have:

  • [ ] Enterprise tier organization with the siem_integration feature enabled
  • [ ] Admin API access — a Bearer token (GHOSTAI_API_KEY) for a user in the target organization
  • [ ] Re-authentication token — value for X-GHOSTAI-REAUTH-TOKEN (sensitive operations)
  • [ ] Target SIEM endpoint — HTTPS URL that accepts POST (e.g. Splunk HEC, mock server, or https://httpbin.org/post for format inspection only)
  • [ ] curl and python3 -m json.tool (or jq) installed
  • [ ] Optional: second terminal to watch your SIEM or mock receiver logs

Environment variables ​

bash
export GHOSTAI_BASE_URL="https://your-ghostai-host"   # or http://localhost:8000
export GHOSTAI_API_KEY="gst_..."                    # org admin API key
export X_GHOSTAI_REAUTH_TOKEN="..."                 # re-auth token for create/delete

Verify prerequisites ​

bash
curl -s -H "Authorization: Bearer $GHOSTAI_API_KEY" \
  "$GHOSTAI_BASE_URL/api/admin/siem/presets" | python3 -m json.tool

Expected: JSON array of presets. If you receive 403 or a feature error, confirm Enterprise + siem_integration before continuing.


Step 1: Explore Presets ​

Duration: 4 minutes
Goal: Discover all SIEM presets, default formats, and deep metadata for Splunk.

1.1 List all presets ​

bash
curl -s -H "Authorization: Bearer $GHOSTAI_API_KEY" \
  "$GHOSTAI_BASE_URL/api/admin/siem/presets" | python3 -m json.tool

Expected: Entries including splunk, sentinel, datadog, elastic, qradar, chronicle, logscale, cortex_xsiam, sumologic, panther, exabeam, logrhythm, securonix, devo, aws_security_lake, and custom. Each item includes id, name, logo, and format.

1.2 Preset detail and setup fields ​

bash
curl -s -H "Authorization: Bearer $GHOSTAI_API_KEY" \
  "$GHOSTAI_BASE_URL/api/admin/siem/presets/splunk" | python3 -m json.tool

Expected: setup_steps, setup_fields, endpoint_hint, content_pack filename (if any), error_remediation map, and format defaulting to splunk_hec.

1.3 Optional: download a content pack ​

Only succeeds when the file exists on the server (SIEM_CONTENT_PACKS_PATH):

bash
curl -s -O -J -H "Authorization: Bearer $GHOSTAI_API_KEY" \
  "$GHOSTAI_BASE_URL/api/admin/siem/content-packs/splunk"

Step 2: Create a Webhook Integration ​

Duration: 6 minutes
Goal: Create a Splunk-oriented webhook integration; observe re-auth and encrypted credential storage (masked in responses).

Replace YOUR_HEC_URL and YOUR_HEC_TOKEN with real values (or use a mock URL for lab-only format testing).

bash
curl -s -X POST "$GHOSTAI_BASE_URL/api/admin/siem/integrations" \
  -H "Authorization: Bearer $GHOSTAI_API_KEY" \
  -H "X-GhostAI-Reauth-Token: $X_GHOSTAI_REAUTH_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Lab Splunk HEC",
    "siem_preset": "splunk",
    "delivery_type": "webhook",
    "endpoint_url": "YOUR_HEC_URL",
    "auth_type": "bearer",
    "auth_value": "YOUR_HEC_TOKEN",
    "format": "splunk_hec",
    "batch_size": 100,
    "flush_interval_seconds": 5,
    "minimization_policy": "standard",
    "event_filters": {}
  }' | python3 -m json.tool

Expected: HTTP 200 success body with an id field (integration UUID). auth_value appears masked (e.g. ••••abcd).

Save the integration ID:

bash
export SIEM_INTEGRATION_ID="<id-from-response>"

Notes ​

  • minimization_policy: standard, minimal, or full controls field richness in outbound events.
  • event_filters: Arbitrary JSON object stored on the integration; extend with keys your organization standardizes (for example allowed category or event_type lists) when the dispatcher honors them.
  • Splunk preset sets HEC-friendly defaults; explicit format": "splunk_hec" matches the demo script.

Step 3: Browse the Event Catalog ​

Duration: 5 minutes
Goal: Inventory streamable events, categories, detections, and one full event definition.

3.1 Categories with counts ​

bash
curl -s -H "Authorization: Bearer $GHOSTAI_API_KEY" \
  "$GHOSTAI_BASE_URL/api/admin/siem/catalog/categories" | python3 -m json.tool

3.2 All events (optional category filter) ​

bash
curl -s -H "Authorization: Bearer $GHOSTAI_API_KEY" \
  "$GHOSTAI_BASE_URL/api/admin/siem/catalog" | python3 -m json.tool

curl -s -H "Authorization: Bearer $GHOSTAI_API_KEY" \
  "$GHOSTAI_BASE_URL/api/admin/siem/catalog?category=authentication" | python3 -m json.tool

3.3 Event detail (MITRE, compliance, sample) ​

bash
curl -s -H "Authorization: Bearer $GHOSTAI_API_KEY" \
  "$GHOSTAI_BASE_URL/api/admin/siem/catalog/auth.login" | python3 -m json.tool

Expected: mitre_attack, compliance array, sample_payload, fields, and description.

3.4 Detection rules bundle ​

bash
curl -s -H "Authorization: Bearer $GHOSTAI_API_KEY" \
  "$GHOSTAI_BASE_URL/api/admin/siem/catalog/detections" | python3 -m json.tool

Step 4: Test the Integration ​

Duration: 3 minutes
Goal: Send a single siem.test_sent event through the integration’s formatter to the remote endpoint.

bash
curl -s -X POST \
  -H "Authorization: Bearer $GHOSTAI_API_KEY" \
  "$GHOSTAI_BASE_URL/api/admin/siem/integrations/$SIEM_INTEGRATION_ID/test" | python3 -m json.tool

Expected: http_status in the 2xx range for a healthy collector; response contains the first ~500 characters of the SIEM reply (e.g. Splunk {"text":"Success","code":0}).


Step 5: Run a Validation Burst ​

Duration: 4 minutes
Goal: POST a batch containing synthetic examples for every catalog event type using the integration’s active format.

bash
curl -s -X POST \
  -H "Authorization: Bearer $GHOSTAI_API_KEY" \
  "$GHOSTAI_BASE_URL/api/admin/siem/integrations/$SIEM_INTEGRATION_ID/validation-burst" | python3 -m json.tool

Expected: http_status, response snippet, and events_sent equal to the number of entries in the platform catalog (large integer). If the receiver rejects oversized payloads, reduce batch-related settings on the integration or use a receiver with higher limits.


Step 6: Monitor Health ​

Duration: 6 minutes
Goal: Read aggregate health, score, delivery history, and actionable errors.

6.1 Health summary ​

bash
curl -s -H "Authorization: Bearer $GHOSTAI_API_KEY" \
  "$GHOSTAI_BASE_URL/api/admin/siem/integrations/$SIEM_INTEGRATION_ID/health" | python3 -m json.tool

Expected: events_sent, events_failed, events_expired, success_rate, latency fields, events_per_minute.

6.2 Health score ​

bash
curl -s -H "Authorization: Bearer $GHOSTAI_API_KEY" \
  "$GHOSTAI_BASE_URL/api/admin/siem/integrations/$SIEM_INTEGRATION_ID/health/score" | python3 -m json.tool

6.3 Recent deliveries ​

bash
curl -s -H "Authorization: Bearer $GHOSTAI_API_KEY" \
  "$GHOSTAI_BASE_URL/api/admin/siem/integrations/$SIEM_INTEGRATION_ID/deliveries" | python3 -m json.tool

6.4 Errors with remediation hints ​

bash
curl -s -H "Authorization: Bearer $GHOSTAI_API_KEY" \
  "$GHOSTAI_BASE_URL/api/admin/siem/integrations/$SIEM_INTEGRATION_ID/errors" | python3 -m json.tool

Expected: Each failed row may include remediation text from the preset’s playbook.

6.5 Webhook log (optional) ​

bash
curl -s -H "Authorization: Bearer $GHOSTAI_API_KEY" \
  "$GHOSTAI_BASE_URL/api/admin/siem/integrations/$SIEM_INTEGRATION_ID/webhook-log?limit=20" | python3 -m json.tool

6.6 Reconciliation and gaps ​

bash
curl -s -H "Authorization: Bearer $GHOSTAI_API_KEY" \
  "$GHOSTAI_BASE_URL/api/admin/siem/integrations/$SIEM_INTEGRATION_ID/reconciliation?start_date=2026-03-01&end_date=2026-03-29" | python3 -m json.tool

curl -s -H "Authorization: Bearer $GHOSTAI_API_KEY" \
  "$GHOSTAI_BASE_URL/api/admin/siem/integrations/$SIEM_INTEGRATION_ID/gaps" | python3 -m json.tool

Step 7: Configure the Pull API ​

Duration: 5 minutes
Goal: Mint a pull token and page events with cursor / limit. Pull uses Bearer token auth and does not reuse the user API key.

7.1 Rotate (create) pull token ​

bash
curl -s -X POST \
  -H "Authorization: Bearer $GHOSTAI_API_KEY" \
  "$GHOSTAI_BASE_URL/api/admin/siem/integrations/$SIEM_INTEGRATION_ID/rotate-pull-token" | python3 -m json.tool

Expected: pull_token string shown once. Store it securely.

bash
export SIEM_PULL_TOKEN="<pull_token-from-response>"

7.2 Pull events ​

bash
curl -s -H "Authorization: Bearer $SIEM_PULL_TOKEN" \
  "$GHOSTAI_BASE_URL/api/admin/siem/pull?cursor=0&limit=10" | python3 -m json.tool

Expected: JSON with events, next_cursor, and has_more. Advance cursor to next_cursor until has_more is false.


Step 8: Event Filtering and Format Preview ​

Duration: 4 minutes
Goal: Inspect queued payloads reformatted as they would be sent on the wire.

The preview endpoint reads recent rows from the organization’s SIEM queue and applies the integration’s effective format (preset override included).

bash
curl -s -H "Authorization: Bearer $GHOSTAI_API_KEY" \
  "$GHOSTAI_BASE_URL/api/admin/siem/integrations/$SIEM_INTEGRATION_ID/preview" | python3 -m json.tool

Expected: format (e.g. splunk_hec) and events array of formatted strings or objects per FORMAT_REGISTRY.

Optional: compare envelope vs. wire format ​

bash
curl -s -H "Authorization: Bearer $GHOSTAI_API_KEY" \
  "$GHOSTAI_BASE_URL/api/admin/siem/schema" | python3 -m json.tool

curl -s -H "Authorization: Bearer $GHOSTAI_API_KEY" \
  "$GHOSTAI_BASE_URL/api/admin/siem/schema/splunk_hec" | python3 -m json.tool

To experiment with cef, leef, sentinel, or ocsf, create a second integration with "siem_preset": "custom" and set "format" accordingly, then re-run preview.


Step 9: Compliance Reporting ​

Duration: 4 minutes
Goal: Generate framework-scoped coverage maps for SOC 2, HIPAA, PCI-DSS, NIST 800-53, and GDPR.

9.1 SOC 2 report ​

bash
curl -s -H "Authorization: Bearer $GHOSTAI_API_KEY" \
  "$GHOSTAI_BASE_URL/api/admin/siem/compliance/report?framework=soc2" | python3 -m json.tool

9.2 Other frameworks ​

bash
for fw in hipaa pci_dss nist_800_53 gdpr; do
  echo "=== $fw ==="
  curl -s -H "Authorization: Bearer $GHOSTAI_API_KEY" \
    "$GHOSTAI_BASE_URL/api/admin/siem/compliance/report?framework=$fw&start_date=2026-03-01&end_date=2026-03-29" | python3 -m json.tool
done

Expected: coverage_percentage, event_types keyed by event_type with controls and severity, and reconciliation data when the backend supplies rows for the org.


Step 10: Integration Lifecycle ​

Duration: 6 minutes
Goal: Practice operational controls—pause, resume, clone, rotate webhook secret, maintenance windows, and parallel migration.

10.1 Disable and enable (pause / resume delivery) ​

bash
curl -s -X POST -H "Authorization: Bearer $GHOSTAI_API_KEY" \
  "$GHOSTAI_BASE_URL/api/admin/siem/integrations/$SIEM_INTEGRATION_ID/disable" | python3 -m json.tool

curl -s -X POST -H "Authorization: Bearer $GHOSTAI_API_KEY" \
  "$GHOSTAI_BASE_URL/api/admin/siem/integrations/$SIEM_INTEGRATION_ID/enable" | python3 -m json.tool

10.2 Clone ​

bash
curl -s -X POST \
  -H "Authorization: Bearer $GHOSTAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"name":"Lab Splunk HEC – clone"}' \
  "$GHOSTAI_BASE_URL/api/admin/siem/integrations/$SIEM_INTEGRATION_ID/clone" | python3 -m json.tool

Save the new id if you continue experimenting.

10.3 Rotate webhook credentials (validates against endpoint) ​

bash
curl -s -X POST \
  -H "Authorization: Bearer $GHOSTAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"auth_value":"YOUR_NEW_HEC_TOKEN"}' \
  "$GHOSTAI_BASE_URL/api/admin/siem/integrations/$SIEM_INTEGRATION_ID/rotate-credentials" | python3 -m json.tool

Expected: Success only if a test POST with the new secret returns HTTP 2xx.

10.4 Maintenance window ​

bash
curl -s -X POST \
  -H "Authorization: Bearer $GHOSTAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"start_time":"2026-03-30T02:00:00Z","end_time":"2026-03-30T04:00:00Z"}' \
  "$GHOSTAI_BASE_URL/api/admin/siem/integrations/$SIEM_INTEGRATION_ID/maintenance" | python3 -m json.tool

curl -s -X POST -H "Authorization: Bearer $GHOSTAI_API_KEY" \
  "$GHOSTAI_BASE_URL/api/admin/siem/integrations/$SIEM_INTEGRATION_ID/maintenance/cancel" | python3 -m json.tool

10.5 Migration (parallel integration) ​

Creates a second integration for cutover testing:

bash
curl -s -X POST \
  -H "Authorization: Bearer $GHOSTAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "target_preset": "elastic",
    "target_endpoint_url": "https://elastic.example.com:9200/_bulk",
    "target_auth_value": "YOUR_ELASTIC_API_KEY"
  }' \
  "$GHOSTAI_BASE_URL/api/admin/siem/integrations/$SIEM_INTEGRATION_ID/migrate" | python3 -m json.tool

Expected: source and target integration objects (masked auth) with status: parallel_run.

To finalize cutover, call POST .../integrations/{target_id}/migrate/complete with JSON body true or false for decommission_source (the {target_id} is the new integration whose name starts with [Migration]).

10.6 Delete (requires re-auth) ​

bash
curl -s -X DELETE \
  -H "Authorization: Bearer $GHOSTAI_API_KEY" \
  -H "X-GhostAI-Reauth-Token: $X_GHOSTAI_REAUTH_TOKEN" \
  "$GHOSTAI_BASE_URL/api/admin/siem/integrations/$SIEM_INTEGRATION_ID" | python3 -m json.tool

Skip 10.6 if you want to keep the lab integration for further testing.


Troubleshooting ​

403 with reauth_required ​

Cause: Create, delete, or auth-value update without X-GhostAI-Reauth-Token.

Fix: Add the header with a valid re-auth token from your security policy.

401 on GET /api/admin/siem/pull ​

Cause: Wrong token type—user API keys are not pull tokens.

Fix: Call rotate-pull-token, then use Authorization: Bearer <pull_token> only on /pull.

Validation burst times out or returns 413 / 500 ​

Cause: Receiver body size limits or slow SIEM.

Fix: Lower batch_size on the integration, use a staging collector, or filter events to reduce catalog breadth in testing environments.

Compliance report 400 Unknown framework ​

Cause: Invalid framework query value.

Fix: Use exactly hipaa, soc2, pci_dss, nist_800_53, or gdpr.

Preview returns empty events ​

Cause: No rows in siem_event_queue for the organization yet.

Fix: Generate traffic (chat, admin actions, or test/burst endpoints), then retry.

rotate-credentials fails with 400 and HTTP status in message ​

Cause: New secret failed live test POST against endpoint_url.

Fix: Verify token, URL, TLS, and network path before rotating.


Lab Summary ​

StepTopicDuration
1Explore presets (GET /presets, detail, content pack)4 min
2Create webhook integration (POST /integrations + re-auth)6 min
3Browse event catalog (/catalog, categories, detail, detections)5 min
4Test integration (POST .../test)3 min
5Validation burst (POST .../validation-burst)4 min
6Monitor health (health, score, deliveries, errors, reconciliation, gaps)6 min
7Configure pull API (rotate-pull-token, GET /pull)5 min
8Filtering & format preview (GET .../preview, schema)4 min
9Compliance reporting (GET /compliance/report)4 min
10Lifecycle (disable/enable, clone, rotate-credentials, maintenance, migrate, delete)6 min
Total~47 min

What's Next ​

  • Customer-facing narrative: Practice the SIEM Integration Demo Script.
  • Schema reference: GET /api/admin/siem/schema and GET /api/admin/siem/schema/{format} for envelope vs. wire formats.
  • MSP aggregation: If your org is an MSP parent, explore GET /api/admin/siem/msp/events with child_org_id after backend enablement.

Event envelope fields: id, timestamp, organization_id, event_type, category, severity, outcome, actor, resource, details, compliance (plus optional sequence, prev_hash, source_table, signature). Supported wire formats: json, cef, leef, splunk_hec, sentinel, ocsf. Minimization: standard, minimal, full.

SOC 2 Type II Certified | BAA Available | HIPAA-Ready