Skip to content

Google Vertex AI ​

Route GhostAI chat, PII referee, embeddings, image generation, voice (TTS/STT), and video (Veo) through your Google Cloud project using Vertex AI.

Prerequisites ​

RequirementDetails
GCP projectActive project with billing enabled (Vertex AI is billed per use)
Vertex AI APIEnabled in Google Cloud Console → APIs & Services for the project
IAMService account or workload identity with Vertex AI User (roles/aiplatform.user) at minimum
NetworkBackend must reach https://{region}-aiplatform.googleapis.com
Python depsgoogle-auth (and related packages from requirements.txt) for ADC or service-account auth

Optional: deploy third-party models (Claude, Llama, Mistral, and others) from Model Garden in the Vertex AI section of the console; first-party Gemini models are available without a separate Model Garden deployment step in many regions.

Environment Variables ​

Add these to your .env (or configure equivalent fields in your secrets manager). Names follow the backend Settings model (uppercase env vars).

Core (chat + shared credentials) ​

bash
# Required for Vertex as a provider
VERTEX_PROJECT_ID=my-gcp-project
VERTEX_REGION=us-central1

# Auth: "adc" (Application Default Credentials) or "service_account"
VERTEX_AUTH_MODE=adc

# Required when VERTEX_AUTH_MODE=service_account — base64-encoded service account key JSON
# (same raw JSON as downloaded from GCP; encode as one line, no newlines in the value)
VERTEX_SERVICE_ACCOUNT_JSON=

When VERTEX_PROJECT_ID is set, GhostAI registers Vertex as an active provider and exposes compatible models in the unified model picker (subject to your plan and feature flags).

PII referee (Vertex) ​

bash
PII_REFEREE_RUNTIME=vertex
PII_REFEREE_VERTEX_MODEL=gemini-2.5-flash

# Optional overrides (otherwise inherit from core Vertex settings)
# PII_REFEREE_VERTEX_PROJECT_ID=
# PII_REFEREE_VERTEX_REGION=
# PII_REFEREE_VERTEX_SERVICE_ACCOUNT_JSON=

# Opt-in full prompt/response audit logging (default is metadata-only)
# PII_REFEREE_AUDIT_VERTEX=false

Feature flag

The Google Vertex AI PII Referee capability may be gated by subscription tier and the vertex_referee feature flag. Confirm with your admin if the referee backend does not activate after configuration.

Embeddings (memory, RAG, content library) ​

bash
MEMORY_EMBEDDING_BACKEND=vertex
MEMORY_EMBEDDING_DIM=768
MEMORY_EMBEDDING_MODEL=gemini-embedding-001

VERTEX_EMBEDDING_MODEL=gemini-embedding-001
VERTEX_EMBEDDING_DIM=768
# VERTEX_EMBEDDING_REGION=   # empty = reuse VERTEX_REGION

VERTEX_EMBEDDING_DIM must match MEMORY_EMBEDDING_DIM. Switching embedding backends typically requires re-embedding existing vectors (see admin embedding rebuild operations in your deployment).

For content library embeddings only:

bash
CONTENT_LIBRARY_EMBEDDING_BACKEND=vertex
CONTENT_LIBRARY_EMBEDDING_MODEL=gemini-embedding-001

Feature flag

Google Vertex AI Embeddings may require the vertex_embeddings flag on hosted tiers.

Image generation ​

bash
IMAGE_GENERATION_PROVIDER=vertex
VERTEX_IMAGE_MODEL=imagen-3.0-generate-002
# Or: VERTEX_IMAGE_MODEL=gemini-2.5-flash-image

Feature flag

Google Vertex AI Image Generation may require the vertex_image_gen flag on hosted tiers.

Voice (TTS / STT) ​

bash
TTS_PROVIDER=vertex
STT_PROVIDER=vertex

Vertex voice uses the same VERTEX_* project, region, and credentials as chat. Adjust fallbacks with TTS_FALLBACK_CHAIN / provider order if you use multiple engines.

Video (Veo) ​

Video generation uses Vertex-native APIs when your deployment is configured for Veo through the same GCP project and region. Retry behavior is controlled by:

bash
VIDEO_RETRY_ATTEMPTS=3
VIDEO_RETRY_BACKOFF=2.0

Authentication Modes ​

Application Default Credentials (ADC) ​

Set VERTEX_AUTH_MODE=adc and omit VERTEX_SERVICE_ACCOUNT_JSON (or leave it empty). The backend uses google.auth.default() with scope https://www.googleapis.com/auth/cloud-platform.

Typical environments:

  • GKE / Cloud Run — Workload Identity or attached service account
  • GCE — Instance service account
  • Local development — gcloud auth application-default login
bash
VERTEX_AUTH_MODE=adc
VERTEX_SERVICE_ACCOUNT_JSON=

Service account (JSON key) ​

Set VERTEX_AUTH_MODE=service_account and supply VERTEX_SERVICE_ACCOUNT_JSON as base64 of the full JSON key file from IAM → Service Accounts → Keys.

bash
VERTEX_AUTH_MODE=service_account
VERTEX_SERVICE_ACCOUNT_JSON=<base64 of key JSON>

This is the most portable option for VMs or Kubernetes clusters without workload identity, and matches the in-app Settings → API Keys → Google Vertex AI upload flow (stored encrypted at rest).

Model Naming Conventions ​

  • Gemini (first-party) — Use model IDs as shown in the GhostAI model picker (e.g. gemini-2.5-flash, gemini-2.5-pro), aligned with Vertex’s Generative AI model catalog for your region.
  • OpenAI-compatible chat endpoint — GhostAI targets Vertex’s OpenAI-compatible base URL:
    https://{region}-aiplatform.googleapis.com/v1/projects/{project}/locations/{region}/endpoints/openapi
    Model strings must be valid for that endpoint in your project/region.
  • Model Garden (third-party) — Deploy the model in your project; use the deployment / endpoint model name that Vertex returns for inference. If a name does not appear in the picker, confirm the deployment succeeded and that the region matches VERTEX_REGION.

Embeddings and image models use Vertex REST APIs; set VERTEX_EMBEDDING_MODEL and VERTEX_IMAGE_MODEL to the exact model IDs supported in your region (see Google’s Vertex AI documentation for Imagen and Gemini embedding versions).

PII Referee Configuration ​

  1. Configure core VERTEX_* credentials (project, region, auth).
  2. Set PII_REFEREE_RUNTIME=vertex and PII_REFEREE_VERTEX_MODEL to a fast, cost-appropriate Gemini model (default gemini-2.5-flash).
  3. Optionally set PII_REFEREE_VERTEX_PROJECT_ID / PII_REFEREE_VERTEX_REGION if the referee should use a different project or region than chat.
  4. Enable the vertex_referee feature for your org if your deployment enforces feature flags.

The referee shares the same authentication path as other Vertex clients (ADC or service account).

Embedding Backend Configuration ​

  1. Set MEMORY_EMBEDDING_BACKEND=vertex.
  2. Set MEMORY_EMBEDDING_DIM and VERTEX_EMBEDDING_DIM to the model’s output size (e.g. 768 for gemini-embedding-001 when configured for that dimensionality).
  3. Align MEMORY_EMBEDDING_MODEL / VERTEX_EMBEDDING_MODEL with the chosen Vertex embedding model.
  4. Plan a full re-embed of memory and RAG indexes after switching from local or another cloud backend.

For content library–only cloud embeddings, set CONTENT_LIBRARY_EMBEDDING_BACKEND=vertex and matching model fields.

Troubleshooting ​

403 Permission Denied / 401 Unauthorized ​

  • Confirm the Vertex AI API is enabled for VERTEX_PROJECT_ID.
  • Verify the identity has Vertex AI User (or a broader role that includes Vertex predict/generate) on the project.
  • For ADC, on GKE ensure Workload Identity is bound correctly; locally run gcloud auth application-default print-access-token to verify a token can be obtained.

Invalid / expired service account JSON ​

  • Re-encode the key file as base64 without extra whitespace; the decoded payload must be valid JSON with type, project_id, private_key, and client_email.
  • Rotated keys: update VERTEX_SERVICE_ACCOUNT_JSON (and any referee-specific override) and restart the backend.

Model not found or wrong region ​

  • Model availability is region-specific. Try a supported region (e.g. us-central1) or check the model’s regional availability in Google Cloud documentation.
  • For Model Garden endpoints, ensure the model is deployed in the same project and region as VERTEX_REGION.

Embeddings dimension mismatch ​

  • VERTEX_EMBEDDING_DIM, MEMORY_EMBEDDING_DIM, and existing DB / index schemas must agree. Mismatches cause indexing or query errors; fix dimensions and rebuild embeddings.

Referee not using Vertex ​

  • Check PII_REFEREE_RUNTIME=vertex (no typo).
  • Ensure vertex_project_id resolves: set PII_REFEREE_VERTEX_PROJECT_ID or VERTEX_PROJECT_ID.
  • Confirm the vertex_referee feature flag and subscription tier allow Vertex referee in your environment.

Timeouts or 429 / RESOURCE_EXHAUSTED ​

  • Increase LLM_TIMEOUT for long contexts.
  • For video, tune VIDEO_RETRY_ATTEMPTS and VIDEO_RETRY_BACKOFF.
  • Check GCP quotas and Vertex TPM/RPM limits in Cloud Console metrics.

SOC 2 Type II Certified | BAA Available | HIPAA-Ready