Skip to content

Multi-Environment Support

streamt supports managing multiple environments (dev, staging, prod) with isolated configurations, protected environment safeguards, and per-environment secrets.

Overview

In multi-environment mode, each environment has its own:

  • Runtime configuration (Kafka clusters, Flink clusters, Schema Registry)
  • Safety settings (protected environments, destructive operation controls)
  • Deployment-state selection (inherited from the project or fully replaced)
  • Environment variables (via .env.{env} files)

Setup

Directory Structure

Create an environments/ directory in your project root:

my-project/
├── stream_project.yml      # Project definition (no runtime section)
├── environments/
│   ├── dev.yml             # Development environment
│   ├── staging.yml         # Staging environment
│   └── prod.yml            # Production environment
├── .env                    # Base environment variables
├── .env.dev                # Dev-specific variables
├── .env.staging            # Staging-specific variables
├── .env.prod               # Prod-specific variables
├── models/
└── sources/

Mode Detection

streamt automatically detects the mode:

  • Single-env mode: No environments/ directory → runtime: required in stream_project.yml
  • Multi-env mode: environments/ directory exists → runtime: comes from environment files

Environment File Format

Each environment file defines runtime configuration and safety settings:

environments/dev.yml
environment:
  name: dev
  description: Local development environment

runtime:
  kafka:
    bootstrap_servers: localhost:9092
  schema_registry:
    url: http://localhost:8081
  flink:
    default: local
    clusters:
      local:
        rest_url: http://localhost:8082
        sql_gateway_url: http://localhost:8084
environments/prod.yml
environment:
  name: prod
  description: Production environment
  protected: true  # Requires a reviewed plan and confirmation for apply

runtime:
  kafka:
    bootstrap_servers: ${PROD_KAFKA_SERVERS}
  schema_registry:
    url: ${PROD_SR_URL}
    username: ${PROD_SR_USER}
    password: ${PROD_SR_PASS}
  flink:
    default: prod-cluster
    clusters:
      prod-cluster:
        rest_url: ${PROD_FLINK_URL}

safety:
  confirm_apply: true         # Require --confirm flag in CI
  allow_destructive: false    # Block topic deletions, etc.
  require_reviewed_plan: true # Optional here: protected already implies it

Deployment-state precedence

deployment_state is a strict tagged provider block. Omitting it everywhere uses the local development backend. In multi-environment mode, a block in stream_project.yml is inherited by every environment that omits an override:

stream_project.yml
deployment_state:
  backend: postgres
  namespace: platform
  postgres:
    dsn_env: STREAMT_STATE_POSTGRES_DSN
    writer_role_env: STREAMT_STATE_POSTGRES_WRITER_ROLE
    schema: streamt

An environment may replace that complete block:

environments/dev.yml
deployment_state:
  backend: local

Replacement is whole-block, not a deep merge. A partial PostgreSQL override cannot borrow the root namespace or connection block and is rejected. This is intentionally different from root runtime, which is ignored whenever an environments/ directory exists.

The configuration contains only environment-variable names, not the DSN or role value. Online administrative commands resolve the values they need after .env, .env.<environment>, and the real process environment are applied; the real environment wins. Offline plan and validation read neither value. With the optional postgres package extra, state status can inspect an exact version-1 or version-2 store, confirmation-gated state init can create or register an empty address, and the separate confirmed state migrate-postgres-v2 command can atomically bind an external writer to an exact v1 store. Ordinary PostgreSQL plan/apply/adopt and recovery remain unavailable and fail safely without falling back to local state. See the PostgreSQL deployment-state migration guide before running the administrative migration.

Initialization confirmations bind to the effective environment after the whole-block precedence rules above have been applied. For a project named payments, the inherited platform namespace, and the prod environment:

streamt state init -p . -e prod \
  --confirm-project payments \
  --confirm-env prod \
  --confirm-address streamt-state://platform/payments/prod

All three confirmations must match exactly. Without a selected environment, the effective environment and required confirmation are default. Normal commands never initialize a store implicitly, and no CLI flag can replace the configured state backend.

CLI Usage

Targeting Environments

Use the --env flag to target a specific environment:

# Validate specific environment
streamt validate --env dev
streamt validate --env prod

# Plan deployment
streamt plan --env staging

# Apply changes
streamt apply --env dev

Environment Variable

Set STREAMT_ENV to avoid repeating --env:

export STREAMT_ENV=dev
streamt validate  # Uses dev environment
streamt plan      # Uses dev environment
streamt apply     # Uses dev environment

CLI Flag Priority

The --env flag always overrides STREAMT_ENV: 

export STREAMT_ENV=dev
streamt validate --env prod  # Uses prod, not dev

Validate All Environments

Validate all environments at once with --all-envs:

streamt validate --all-envs

This validates each environment sequentially and fails if any environment is invalid.

List Environments

View available environments:

streamt envs list

Output: 

dev          Local development environment
staging      Staging environment
prod         Production environment [protected]

Show Environment Config

View resolved configuration (secrets masked):

streamt envs show prod

Output: 

environment:
  name: prod
  description: Production environment
  protected: true
runtime:
  kafka:
    bootstrap_servers: prod-kafka.example.com:9092
  schema_registry:
    url: https://prod-sr.example.com
    username: admin
    password: '****'

Protected Environments

Mark critical environments as protected to require both a reviewed plan and explicit environment confirmation:

environments/prod.yml
environment:
  name: prod
  protected: true

Behavior

Direct apply is rejected before streamt constructs any deployer. Confirmation and force flags cannot bypass the review gate:

$ streamt apply --env prod
ERROR: Direct apply is disabled for environment 'prod'; a reviewed plan file is required.
$ streamt apply --env prod --confirm --force
ERROR: Direct apply is disabled for environment 'prod'; a reviewed plan file is required.

Create the online plan, review the saved JSON, then apply that exact file:

$ streamt plan --env prod --out prod.plan.json
$ streamt apply --env prod --plan prod.plan.json --confirm-env prod
WARNING: Deploying to protected environment 'prod'
Apply complete

The reviewed apply verifies the checksum, project and environment fingerprints, exact ownership-state backend/store/address/serial/checksum, current live actions, ownership requirements, and safety blockers before mutation. Protected-environment confirmation remains a separate check. In an interactive terminal, omit --confirm-env and type the environment name at the prompt.

Shared Environments

An environment does not need to be protected to require the review/apply protocol. Mark a shared staging or integration environment explicitly:

environments/staging.yml
environment:
  name: staging
  protected: false

safety:
  require_reviewed_plan: true

streamt does not infer policy from environment names. The explicit safety field is the supported way to gate an otherwise unprotected shared workflow.

Require remote ownership state

For environments that must never mutate while local ownership state is selected, enable the environment-only policy:

environments/staging.yml
safety:
  require_remote_state: true

The default is false, including for protected environments. When enabled, apply and adopt fail with E421_REMOTE_STATE_REQUIRED before confirmation, compilation, state access, or runtime deployer construction if the effective backend is local. --force cannot bypass the policy. Read-only online plan and state status remain available for diagnosis, and reviewed-plan/offline-plan validation errors retain their existing precedence.

Destructive Safety

Block destructive operations (topic deletions, connector removals) in critical environments:

environments/prod.yml
safety:
  allow_destructive: false

Behavior

# Blocked by default
$ streamt apply --env prod --plan prod.plan.json --confirm-env prod
ERROR: Destructive operations blocked for 'prod' environment. Use --force flag to override.

# Override with --force
$ streamt apply --env prod --plan prod.plan.json --confirm-env prod --force
WARNING: --force flag used, allowing destructive operations on 'prod'
Applying changes...

Environment Variables

.env File Precedence

Environment variables are loaded with this precedence (later wins):

  1. .env — Base variables (always loaded)
  2. .env.{environment} — Environment-specific overrides
  3. Actual environment variables — Highest priority

Example

.env
KAFKA_SERVERS=localhost:9092
LOG_LEVEL=INFO
.env.prod
KAFKA_SERVERS=prod-kafka.example.com:9092
LOG_LEVEL=WARN
environments/prod.yml
runtime:
  kafka:
    bootstrap_servers: ${KAFKA_SERVERS}  # Resolves to prod-kafka.example.com:9092

Override in CI/CD

# .env.prod has KAFKA_SERVERS=prod-kafka.example.com:9092
# But actual env var takes precedence
export KAFKA_SERVERS=custom-kafka.example.com:9092
streamt plan --env prod --out prod.plan.json
streamt apply --env prod --plan prod.plan.json --confirm-env prod

CI/CD Integration

GitHub Actions Example

.github/workflows/deploy.yml
# streamt:skip — GitHub Actions workflow, not streamt config
name: Deploy

on:
  push:
    branches: [main]

jobs:
  deploy-staging:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Deploy to Staging
        env:
          STAGING_KAFKA: ${{ secrets.STAGING_KAFKA }}
          STAGING_SR_URL: ${{ secrets.STAGING_SR_URL }}
        run: |
          streamt validate --env staging
          streamt plan --env staging --out staging.plan.json
          streamt apply --env staging --plan staging.plan.json

  deploy-prod:
    needs: deploy-staging
    runs-on: ubuntu-latest
    environment: production  # GitHub environment protection
    steps:
      - uses: actions/checkout@v4

      - name: Deploy to Production
        env:
          PROD_KAFKA_SERVERS: ${{ secrets.PROD_KAFKA }}
          PROD_SR_URL: ${{ secrets.PROD_SR_URL }}
        run: |
          streamt validate --env prod
          streamt plan --env prod --out prod.plan.json
          # --confirm-env verifies the environment name matches (safer for CI)
          streamt apply --env prod --plan prod.plan.json --confirm-env prod

Agent/LLM Automation

For programmatic use by LLM agents or automation tools, use --output json for structured output:

# Structured JSON output from any command
streamt -o json validate --env staging
streamt -o json plan --env prod --out prod.plan.json
streamt -o json list models
streamt -o json show model order_metrics

# Deploy the reviewed plan with name verification and JSON output
streamt -o json apply --env prod --plan prod.plan.json --confirm-env prod

Best Practices

1. Use Protected Environments for Production

environment:
  name: prod
  protected: true

safety:
  confirm_apply: true
  allow_destructive: false

2. Keep Secrets in .env Files

Never commit secrets to environment YAML files. Use variable references:

# streamt:skip
# Good: Reference variables
runtime:
  kafka:
    bootstrap_servers: ${KAFKA_SERVERS}
  schema_registry:
    password: ${SR_PASSWORD}

# .env.prod (gitignored)
KAFKA_SERVERS=prod-kafka:9092
SR_PASSWORD=secret123

3. Validate All Environments in CI

# In CI pipeline
streamt validate --all-envs

4. Use Descriptive Environment Names

environment:
  name: prod-us-east-1
  description: Production cluster in US East region

Migration from Single-Env Mode

To migrate an existing single-env project:

  1. Create environments/ directory
  2. Move runtime: from stream_project.yml to environments/dev.yml
  3. Create additional environment files as needed
  4. Update CI/CD to use --env flag

Before:

stream_project.yml
project:
  name: my-project
  version: "1.0.0"

runtime:
  kafka:
    bootstrap_servers: localhost:9092

sources:
  - name: events
    topic: events.raw.v1

After: 

# streamt:skip
project:
  name: my-project
  version: "1.0.0"

sources:
  - name: events
    topic: events.raw.v1

environments/dev.yml
environment:
  name: dev
  description: Development environment

runtime:
  kafka:
    bootstrap_servers: localhost:9092

Troubleshooting

"No environments configured"

ERROR: No environments configured. Remove --env flag or create environments/ directory.

You're using --env in single-env mode. Either: - Create an environments/ directory with environment files - Remove the --env flag

"Multiple environments found. Specify with --env"

ERROR: Multiple environments found. Specify with --env. Available: dev, staging, prod

In multi-env mode, you must specify which environment to use: 

streamt validate --env dev

Or set the environment variable: 

export STREAMT_ENV=dev
streamt validate

"Environment 'xyz' not found"

ERROR: Environment 'xyz' not found. Available: dev, staging, prod

Check that the environment file exists: environments/xyz.yml

"Environment name mismatch"

ERROR: Environment name mismatch: file is 'dev.yml' but environment.name is 'development'. They must match.

The name in the YAML must match the filename:

environments/dev.yml
environment:
  name: dev  # Must match filename 'dev.yml'