Appearance
Google Vertex AI
Route GhostAI chat, PII referee, embeddings, image generation, voice (TTS/STT), and video (Veo) through your Google Cloud project using Vertex AI.
Prerequisites
| Requirement | Details |
|---|---|
| GCP project | Active project with billing enabled (Vertex AI is billed per use) |
| Vertex AI API | Enabled in Google Cloud Console → APIs & Services for the project |
| IAM | Service account or workload identity with Vertex AI User (roles/aiplatform.user) at minimum |
| Network | Backend must reach https://{region}-aiplatform.googleapis.com |
| Python deps | google-auth (and related packages from requirements.txt) for ADC or service-account auth |
Optional: deploy third-party models (Claude, Llama, Mistral, and others) from Model Garden in the Vertex AI section of the console; first-party Gemini models are available without a separate Model Garden deployment step in many regions.
Environment Variables
Add these to your .env (or configure equivalent fields in your secrets manager). Names follow the backend Settings model (uppercase env vars).
Core (chat + shared credentials)
bash
# Required for Vertex as a provider
VERTEX_PROJECT_ID=my-gcp-project
VERTEX_REGION=us-central1
# Auth: "adc" (Application Default Credentials) or "service_account"
VERTEX_AUTH_MODE=adc
# Required when VERTEX_AUTH_MODE=service_account — base64-encoded service account key JSON
# (same raw JSON as downloaded from GCP; encode as one line, no newlines in the value)
VERTEX_SERVICE_ACCOUNT_JSON=When VERTEX_PROJECT_ID is set, GhostAI registers Vertex as an active provider and exposes compatible models in the unified model picker (subject to your plan and feature flags).
PII referee (Vertex)
bash
PII_REFEREE_RUNTIME=vertex
PII_REFEREE_VERTEX_MODEL=gemini-2.5-flash
# Optional overrides (otherwise inherit from core Vertex settings)
# PII_REFEREE_VERTEX_PROJECT_ID=
# PII_REFEREE_VERTEX_REGION=
# PII_REFEREE_VERTEX_SERVICE_ACCOUNT_JSON=
# Opt-in full prompt/response audit logging (default is metadata-only)
# PII_REFEREE_AUDIT_VERTEX=falseFeature flag
The Google Vertex AI PII Referee capability may be gated by subscription tier and the vertex_referee feature flag. Confirm with your admin if the referee backend does not activate after configuration.
Embeddings (memory, RAG, content library)
bash
MEMORY_EMBEDDING_BACKEND=vertex
MEMORY_EMBEDDING_DIM=768
MEMORY_EMBEDDING_MODEL=gemini-embedding-001
VERTEX_EMBEDDING_MODEL=gemini-embedding-001
VERTEX_EMBEDDING_DIM=768
# VERTEX_EMBEDDING_REGION= # empty = reuse VERTEX_REGIONVERTEX_EMBEDDING_DIM must match MEMORY_EMBEDDING_DIM. Switching embedding backends typically requires re-embedding existing vectors (see admin embedding rebuild operations in your deployment).
For content library embeddings only:
bash
CONTENT_LIBRARY_EMBEDDING_BACKEND=vertex
CONTENT_LIBRARY_EMBEDDING_MODEL=gemini-embedding-001Feature flag
Google Vertex AI Embeddings may require the vertex_embeddings flag on hosted tiers.
Image generation
bash
IMAGE_GENERATION_PROVIDER=vertex
VERTEX_IMAGE_MODEL=imagen-3.0-generate-002
# Or: VERTEX_IMAGE_MODEL=gemini-2.5-flash-imageFeature flag
Google Vertex AI Image Generation may require the vertex_image_gen flag on hosted tiers.
Voice (TTS / STT)
bash
TTS_PROVIDER=vertex
STT_PROVIDER=vertexVertex voice uses the same VERTEX_* project, region, and credentials as chat. Adjust fallbacks with TTS_FALLBACK_CHAIN / provider order if you use multiple engines.
Video (Veo)
Video generation uses Vertex-native APIs when your deployment is configured for Veo through the same GCP project and region. Retry behavior is controlled by:
bash
VIDEO_RETRY_ATTEMPTS=3
VIDEO_RETRY_BACKOFF=2.0Authentication Modes
Application Default Credentials (ADC)
Set VERTEX_AUTH_MODE=adc and omit VERTEX_SERVICE_ACCOUNT_JSON (or leave it empty). The backend uses google.auth.default() with scope https://www.googleapis.com/auth/cloud-platform.
Typical environments:
- GKE / Cloud Run — Workload Identity or attached service account
- GCE — Instance service account
- Local development —
gcloud auth application-default login
bash
VERTEX_AUTH_MODE=adc
VERTEX_SERVICE_ACCOUNT_JSON=Service account (JSON key)
Set VERTEX_AUTH_MODE=service_account and supply VERTEX_SERVICE_ACCOUNT_JSON as base64 of the full JSON key file from IAM → Service Accounts → Keys.
bash
VERTEX_AUTH_MODE=service_account
VERTEX_SERVICE_ACCOUNT_JSON=<base64 of key JSON>This is the most portable option for VMs or Kubernetes clusters without workload identity, and matches the in-app Settings → API Keys → Google Vertex AI upload flow (stored encrypted at rest).
Model Naming Conventions
- Gemini (first-party) — Use model IDs as shown in the GhostAI model picker (e.g.
gemini-2.5-flash,gemini-2.5-pro), aligned with Vertex’s Generative AI model catalog for your region. - OpenAI-compatible chat endpoint — GhostAI targets Vertex’s OpenAI-compatible base URL:
https://{region}-aiplatform.googleapis.com/v1/projects/{project}/locations/{region}/endpoints/openapi
Model strings must be valid for that endpoint in your project/region. - Model Garden (third-party) — Deploy the model in your project; use the deployment / endpoint model name that Vertex returns for inference. If a name does not appear in the picker, confirm the deployment succeeded and that the region matches
VERTEX_REGION.
Embeddings and image models use Vertex REST APIs; set VERTEX_EMBEDDING_MODEL and VERTEX_IMAGE_MODEL to the exact model IDs supported in your region (see Google’s Vertex AI documentation for Imagen and Gemini embedding versions).
PII Referee Configuration
- Configure core
VERTEX_*credentials (project, region, auth). - Set
PII_REFEREE_RUNTIME=vertexandPII_REFEREE_VERTEX_MODELto a fast, cost-appropriate Gemini model (defaultgemini-2.5-flash). - Optionally set
PII_REFEREE_VERTEX_PROJECT_ID/PII_REFEREE_VERTEX_REGIONif the referee should use a different project or region than chat. - Enable the vertex_referee feature for your org if your deployment enforces feature flags.
The referee shares the same authentication path as other Vertex clients (ADC or service account).
Embedding Backend Configuration
- Set
MEMORY_EMBEDDING_BACKEND=vertex. - Set
MEMORY_EMBEDDING_DIMandVERTEX_EMBEDDING_DIMto the model’s output size (e.g.768forgemini-embedding-001when configured for that dimensionality). - Align
MEMORY_EMBEDDING_MODEL/VERTEX_EMBEDDING_MODELwith the chosen Vertex embedding model. - Plan a full re-embed of memory and RAG indexes after switching from
localor another cloud backend.
For content library–only cloud embeddings, set CONTENT_LIBRARY_EMBEDDING_BACKEND=vertex and matching model fields.
Troubleshooting
403 Permission Denied / 401 Unauthorized
- Confirm the Vertex AI API is enabled for
VERTEX_PROJECT_ID. - Verify the identity has Vertex AI User (or a broader role that includes Vertex predict/generate) on the project.
- For ADC, on GKE ensure Workload Identity is bound correctly; locally run
gcloud auth application-default print-access-tokento verify a token can be obtained.
Invalid / expired service account JSON
- Re-encode the key file as base64 without extra whitespace; the decoded payload must be valid JSON with
type,project_id,private_key, andclient_email. - Rotated keys: update
VERTEX_SERVICE_ACCOUNT_JSON(and any referee-specific override) and restart the backend.
Model not found or wrong region
- Model availability is region-specific. Try a supported region (e.g.
us-central1) or check the model’s regional availability in Google Cloud documentation. - For Model Garden endpoints, ensure the model is deployed in the same project and region as
VERTEX_REGION.
Embeddings dimension mismatch
VERTEX_EMBEDDING_DIM,MEMORY_EMBEDDING_DIM, and existing DB / index schemas must agree. Mismatches cause indexing or query errors; fix dimensions and rebuild embeddings.
Referee not using Vertex
- Check
PII_REFEREE_RUNTIME=vertex(no typo). - Ensure
vertex_project_idresolves: setPII_REFEREE_VERTEX_PROJECT_IDorVERTEX_PROJECT_ID. - Confirm the
vertex_refereefeature flag and subscription tier allow Vertex referee in your environment.
Timeouts or 429 / RESOURCE_EXHAUSTED
- Increase
LLM_TIMEOUTfor long contexts. - For video, tune
VIDEO_RETRY_ATTEMPTSandVIDEO_RETRY_BACKOFF. - Check GCP quotas and Vertex TPM/RPM limits in Cloud Console metrics.
Related
- LLM providers deep dive — routing, BYOK, and multi-provider behavior
- Microsoft Foundry — similar pattern for Azure AI