Deployment Guide

Overview

This guide covers deploying ML pipelines across all environments: sandbox (local), dev, staging, and production. The deployment architecture uses GitHub Actions for CI/CD with environment-specific service principals.

Deployment Architecture

4-Tier Environment Strategy

The platform uses a 4-tier deployment architecture (Sandbox → Dev → Staging → Prod) with progressive promotion gates. For the architectural rationale and detailed decision, see ADR-001: Four-Tier Deployment Architecture.

Deployment Flow: Sandbox (local) → Dev (auto on merge) → Staging (auto with approval) → Production (auto with approval)

Environment	Catalog	Deployed By	Service Principal	Purpose
Sandbox	{username}_sandbox	Developer (manual)	User credentials	Rapid local iteration
Dev	dev	GitHub Actions	ml-pipelines-dev	Integration testing
Staging	staging	GitHub Actions	ml-pipelines-staging	Pre-prod validation
Production	prod	GitHub Actions	ml-pipelines-prod	Live workloads

Prerequisites and Setup

1. Local Development Setup

Required Tools:

# Databricks CLI
databricks --version  # Should be >= 0.200.0

# UV package manager
uv --version

# AWS CLI (for S3 volume access)
aws --version

# Git
git --version

Configure Databricks Profiles:

File: ~/.databrickscfg

[ref-dev]
host = https://dbc-a72d6af9-df3d.cloud.databricks.com
auth_type = oauth

[ref-staging]
host = https://dbc-fab2a42a-8d11.cloud.databricks.com
auth_type = oauth

[ref-prod]
host = https://dbc-028d9e53-7ce6.cloud.databricks.com
auth_type = oauth

Authenticate:

databricks auth login --profile ref-dev
databricks auth login --profile ref-staging
databricks auth login --profile ref-prod

2. Service Principal Setup

See Service Principals Guide for detailed setup instructions.

Service Principals Required:

• ml-pipelines-dev-service-principal (UUID: 03ff99cd-a352-40bb-9d33-414c9ad9e7aa)

• ml-pipelines-staging-service-principal (UUID: 93bda7cf-b009-49d8-8e8d-046677c8597e)

• ml-pipelines-prod-service-principal (UUID: 2af4b95a-d80f-4da6-bfba-9ec5a8a8ec9f)

Verify Service Principals:

# Check service principal exists
databricks service-principals get 03ff99cd-a352-40bb-9d33-414c9ad9e7aa --profile ref-dev

# Check catalog permissions
databricks grants get catalog dev --profile ref-dev

3. GitHub Secrets

Required Secrets:

• GH_PAT - GitHub Personal Access Token (for checking out code with submodules)

GitHub Environments:

• development - For dev deployments

• staging - For staging deployments

• production - For production deployments

Sandbox Deployment (Developer Local)

Deploy to Your Sandbox

cd /Users/taylorlaing/Development/refresh-os/ml-pipelines

# Validate configuration
make validate

# Deploy to sandbox (creates {username}_sandbox catalog)
make deploy

What Happens:

1. Validates databricks.yml configuration

2. Determines your username (e.g., taylor)

3. Creates catalog taylor_sandbox if it doesn't exist

4. Creates S3 volume directories for your catalog

5. Builds Python wheel package with uv build --wheel

6. Deploys all pipelines and jobs to dev workspace

7. Resources named as [dev taylor] resource_name_sandbox

Verify Sandbox Deployment:

# List your catalog
databricks catalogs list --profile ref-dev | grep "$(whoami)_sandbox"

# List your pipelines
databricks pipelines list-pipelines --profile ref-dev | grep "$(whoami)"

# List your jobs
databricks jobs list --profile ref-dev | grep "$(whoami)"

Copy Sample Data (optional):

# Copy data from dev volumes to your sandbox for testing
make copy-data

Sandbox Cleanup

# Remove build artifacts
make clean

# Drop sandbox catalog (if needed)
databricks sql execute "DROP CATALOG $(whoami)_sandbox CASCADE" --profile ref-dev

Dev Deployment (Automatic on PR Merge)

Workflow Trigger

File: .github/workflows/ml_pipelines_dev_deploy.yml

Triggers:

• Push to main branch

• Manual workflow dispatch (optional)

Process:

1. PR opened → Bundle validation runs

2. PR merged to main → Dev deployment triggers automatically

3. Service principal ml-pipelines-dev deploys to dev workspace

Manual Dev Deployment

# From local machine (requires service principal auth)
make deploy-dev

Note: Local dev deployment requires service principal credentials, which are typically only available in CI/CD. For local testing, use sandbox instead.

Verify Dev Deployment

# Check dev pipelines
databricks pipelines list-pipelines --profile ref-dev | grep "dev_"

# Check dev catalog tables
databricks sql execute "SHOW TABLES IN dev.bronze" --profile ref-dev
databricks sql execute "SHOW TABLES IN dev.silver" --profile ref-dev

# Check pipeline status
databricks pipelines get <pipeline_id> --profile ref-dev

Troubleshooting Dev Deployment

Check GitHub Actions logs:

1. Go to repository → Actions tab

2. Click "Deploy to Development" workflow

3. Review step logs for errors

Common Issues:

• Bundle validation failed: Fix databricks.yml syntax errors

• Authentication failed: Verify service principal OIDC configuration

• Catalog not found: Check catalog permissions for service principal

• Wheel build failed: Check pyproject.toml dependencies

Staging Deployment (Automatic After Dev)

Workflow Trigger

File: .github/workflows/ml_pipelines_staging_deploy.yml

Triggers:

• Automatically after successful dev deployment

• Manual workflow dispatch (for hotfixes)

Condition: Only runs if dev deployment succeeded

Deployment Process

on:
  workflow_run:
    workflows: ['Deploy to Development']
    types: [completed]

jobs:
  deploy_staging:
    if: ${{ github.event.workflow_run.conclusion == 'success' }}

What Happens:

1. Dev deployment completes successfully

2. Staging workflow automatically triggered

3. Service principal ml-pipelines-staging deploys to staging workspace

4. All pipelines/jobs deployed to staging catalog

Verify Staging Deployment

# Check staging pipelines
databricks pipelines list-pipelines --profile ref-staging | grep "staging_"

# Check staging catalog
databricks sql execute "SHOW TABLES IN staging.silver" --profile ref-staging

# Verify data exists
databricks sql execute "SELECT COUNT(*) FROM staging.bronze.messages" --profile ref-staging

Staging-Specific Considerations

Paused Schedules: Staging jobs are paused by default to avoid unnecessary runs:

# In databricks.yml
targets:
  staging:
    presets:
      trigger_pause_status: PAUSED

Manual Trigger (if needed):

# Start a staging pipeline manually
databricks pipelines start-update <pipeline_id> --profile ref-staging

Production Deployment (Automatic After Staging)

Workflow Trigger

File: .github/workflows/ml_pipelines_prod_deploy.yml

Triggers:

• Automatically after successful staging deployment

• Manual workflow dispatch (for emergency deployments)

Condition: Only runs if staging deployment succeeded

Deployment Process

on:
  workflow_run:
    workflows: ['Deploy to Staging']
    types: [completed]

jobs:
  deploy_prod:
    environment: production  # Requires GitHub environment approval
    if: ${{ github.event.workflow_run.conclusion == 'success' }}

What Happens:

1. Staging deployment completes successfully

2. Production workflow automatically triggered

3. Service principal ml-pipelines-prod deploys to production workspace

4. All pipelines/jobs deployed to production catalog

5. Schedules are UNPAUSED for production workloads

Production Safety Checks

Unpaused Schedules: Only production runs on schedule automatically:

# In databricks.yml
targets:
  prod:
    presets:
      trigger_pause_status: UNPAUSED

Concurrency Limits:

targets:
  prod:
    presets:
      jobs_max_concurrent_runs: 5  # Limit concurrent runs

Verify Production Deployment

# Check production pipelines
databricks pipelines list-pipelines --profile ref-prod | grep "prod_"

# Check production catalog
databricks sql execute "SHOW TABLES IN prod.silver" --profile ref-prod

# Verify scheduled jobs
databricks jobs list --profile ref-prod | grep "prod_"

# Check job schedules
databricks jobs get <job_id> --profile ref-prod

Production Monitoring

Check Pipeline Health:

# Get recent pipeline updates
databricks pipelines list-updates <pipeline_id> --profile ref-prod

# Check for failures
databricks sql execute "
  SELECT
    pipeline_name,
    update_id,
    state,
    cause
  FROM system.compute.pipeline_events
  WHERE state = 'FAILED'
    AND timestamp > CURRENT_DATE() - INTERVAL 7 DAYS
" --profile ref-prod

Verification Steps for Each Environment

Post-Deployment Checklist

Sandbox Verification

# 1. Catalog exists
databricks catalogs get taylor_sandbox --profile ref-dev

# 2. Schemas created
databricks sql execute "SHOW SCHEMAS IN taylor_sandbox" --profile ref-dev

# 3. Pipelines deployed
databricks pipelines list-pipelines --profile ref-dev | grep taylor

# 4. Jobs deployed
databricks jobs list --profile ref-dev | grep taylor

Dev Verification

# 1. Pipelines running
databricks sql execute "
  SELECT
    pipeline_name,
    state,
    latest_update_time
  FROM system.compute.pipeline_events
  WHERE catalog = 'dev'
  ORDER BY latest_update_time DESC
" --profile ref-dev

# 2. Data freshness
databricks sql execute "
  SELECT
    MAX(created_date) as latest_data
  FROM dev.bronze.messages
" --profile ref-dev

# 3. Model endpoints active
databricks model-serving list --profile ref-dev

Staging Verification

# 1. Tables exist with data
databricks sql execute "
  SELECT
    table_name,
    COUNT(*) as row_count
  FROM staging.information_schema.tables t
  JOIN staging.silver.* s ON t.table_name = s.table_name
  GROUP BY table_name
" --profile ref-staging

# 2. Data quality checks pass
databricks sql execute "
  SELECT * FROM staging.silver.sentiment_features
  WHERE sentiment_score IS NULL
  LIMIT 10
" --profile ref-staging

Production Verification

# 1. All pipelines healthy
databricks pipelines list-pipelines --profile ref-prod

# 2. Scheduled jobs running
databricks jobs list-runs --profile ref-prod --limit 10

# 3. Data pipeline latency
databricks sql execute "
  SELECT
    CURRENT_TIMESTAMP() - MAX(timestamp) as data_lag
  FROM prod.bronze.messages
" --profile ref-prod

# 4. Model serving healthy
databricks model-serving get sentiment_analysis --profile ref-prod

Rollback Procedures

Rollback Strategy

1. Identify the problem (failed deployment, data corruption, model issues)

2. Assess impact (which environment is affected)

3. Choose rollback method (git revert, pipeline reset, data restore)

4. Execute rollback (steps below)

5. Verify (run verification checks)

Git-Based Rollback (Recommended)

For configuration or code issues:

# 1. Find the last working commit
git log --oneline

# 2. Revert to working state
git revert <bad_commit_sha>

# 3. Push revert commit
git push origin main

# 4. CI/CD automatically deploys reverted state
# Watch GitHub Actions for deployment status

Pipeline-Specific Rollback

Reset a pipeline to previous state:

# 1. Stop the pipeline
databricks pipelines stop <pipeline_id> --profile ref-prod

# 2. Reset to previous update
databricks pipelines reset <pipeline_id> --profile ref-prod

# 3. Trigger fresh update
databricks pipelines start-update <pipeline_id> --profile ref-prod

Data Rollback (Time Travel)

Restore data to previous version:

-- Check version history
DESCRIBE HISTORY prod.silver.sentiment_features;

-- Restore to previous version
RESTORE TABLE prod.silver.sentiment_features TO VERSION AS OF 42;

-- Or restore to timestamp
RESTORE TABLE prod.silver.sentiment_features TO TIMESTAMP AS OF '2025-01-15T10:00:00Z';

Model Rollback

Revert to previous model version:

from mlflow.tracking import MlflowClient

client = MlflowClient()

# 1. List model versions
versions = client.search_model_versions("name='prod.models.sentiment_analysis'")

# 2. Transition previous version to Production
client.transition_model_version_stage(
    name="prod.models.sentiment_analysis",
    version=3,  # Previous working version
    stage="Production",
    archive_existing_versions=True
)

Emergency Rollback (Full Environment)

For catastrophic failures:

# 1. Stop all pipelines
for pipeline_id in $(databricks pipelines list-pipelines --profile ref-prod --output json | jq -r '.[].pipeline_id'); do
  databricks pipelines stop $pipeline_id --profile ref-prod
done

# 2. Disable all job schedules
for job_id in $(databricks jobs list --profile ref-prod --output json | jq -r '.jobs[].job_id'); do
  databricks jobs update $job_id --pause true --profile ref-prod
done

# 3. Contact team and coordinate manual fix
# 4. Re-enable after verification

Common Deployment Issues

1. Bundle Validation Failures

Error: Error: databricks.yml validation failed

Causes:

• YAML syntax errors

• Invalid variable references

• Missing required fields

Solutions:

# Validate locally
make validate

# Check YAML syntax
yamllint databricks.yml

# Verify variable references
grep -r '\${var\.' databricks.yml resources/

2. Authentication Failures

Error: Error: OIDC authentication failed

Causes:

• Service principal not configured

• Incorrect Client ID

• OIDC trust not set up

Solutions:

# Verify service principal exists
databricks service-principals get <client_id> --profile ref-admin

# Check OIDC configuration in Databricks UI
# Account Console → Service Principals → [SP] → Authentication → GitHub OIDC

# Verify GitHub environment matches OIDC config

3. Permission Errors

Error: Error: Permission denied on catalog

Causes:

• Service principal lacks catalog permissions

• Catalog doesn't exist

• Wrong workspace

Solutions:

# Grant catalog permissions
databricks sql execute "GRANT ALL PRIVILEGES ON CATALOG dev TO '<service_principal_id>'" --profile ref-dev

# Verify permissions
databricks grants get catalog dev --profile ref-dev

4. Pipeline Deployment Failures

Error: Error: Pipeline deployment failed

Causes:

• Invalid SQL syntax

• Missing catalog references

• Schema conflicts

Solutions:

# Test SQL locally
databricks sql execute "$(cat resources/pipelines/silver/*/transformations/*.sql)" --profile ref-dev

# Use CREATE OR REPLACE to reset schema
# In SQL file: CREATE OR REPLACE STREAMING LIVE TABLE ...

# Check pipeline logs
databricks pipelines get <pipeline_id> --profile ref-dev

GitHub Actions Workflow Explanation

Workflow Structure

1. PR Validation (ml_pipelines_pr_validate.yml):

on:
  pull_request:
    branches: [main]
    paths:
      - 'databricks.yml'
      - 'resources/**'
      - 'src/**'

jobs:
  validate:
    steps:
      - Install Databricks CLI
      - Validate bundle (make validate-dev)
      - Comment on PR with results

2. Dev Deployment (ml_pipelines_dev_deploy.yml):

on:
  push:
    branches: [main]

jobs:
  deploy_dev:
    environment: development
    steps:
      - Checkout code
      - Install dependencies
      - Deploy bundle (make deploy-dev)

3. Staging Deployment (ml_pipelines_staging_deploy.yml):

on:
  workflow_run:
    workflows: ['Deploy to Development']
    types: [completed]

jobs:
  deploy_staging:
    environment: staging
    if: ${{ github.event.workflow_run.conclusion == 'success' }}
    steps:
      - Deploy bundle (make deploy-staging)

4. Production Deployment (ml_pipelines_prod_deploy.yml):

on:
  workflow_run:
    workflows: ['Deploy to Staging']
    types: [completed]

jobs:
  deploy_prod:
    environment: production
    if: ${{ github.event.workflow_run.conclusion == 'success' }}
    steps:
      - Deploy bundle (make deploy-prod)

Workflow Dependencies

PR Opened
    ↓
PR Validation (validate bundle)
    ↓
PR Merged to main
    ↓
Dev Deployment (automatic)
    ↓ (only if success)
Staging Deployment (automatic)
    ↓ (only if success)
Production Deployment (automatic)

Service Principal Authentication

For complete service principal setup and GitHub OIDC configuration, see:

• Service Principals Guide - Detailed setup instructions

• ADR-003: Service Principal Per Environment - Architecture decision

Key Benefits:

• Passwordless authentication (no secrets in GitHub)

• Environment-specific service principals for isolation

• Automatic token rotation and audit trail

• Scoped to specific repositories and environments

Best Practices

1. Always Test in Sandbox First

# Development flow
make deploy          # Test in sandbox
git add .
git commit -m "Add new feature"
git push origin feature-branch
# Create PR → Automatic validation
# Merge → Automatic dev deployment

2. Monitor Deployment Progress

# Watch GitHub Actions
# Repository → Actions → Click on workflow run

# Check Databricks pipeline status
databricks pipelines get <pipeline_id> --profile ref-dev

3. Use Feature Flags for Risky Changes

# In databricks.yml
configuration:
  "feature.new_model_enabled": "false"  # Toggle in production

4. Gradual Rollout

• Deploy to dev → Validate

• Deploy to staging → Validate with prod data

• Deploy to prod → Monitor closely

5. Keep Environments in Sync

• Same Databricks Runtime version

• Same library versions

• Same configuration (except environment-specific)

Related Documentation

This Repository

• Service Principals Setup - Detailed SP configuration

• Troubleshooting Guide - Common issues and solutions

• Local Development Guide - Sandbox workflow

• Configuration Reference - databricks.yml settings

Cross-Repository Documentation

• infra-core - Terraform infrastructure and service principal configuration

  • Manages: Databricks workspaces, Unity Catalog, VPC, cross-region replication

• api-core - API deployment and integration

  • Consumes: ML pipeline outputs via Databricks SQL endpoints

• app-web - Frontend deployment

  • Displays: Insights and analytics from ML pipelines

Emergency Contacts

For deployment issues:

• Taylor Laing ([email protected]) - Admin access

• Check #ml-pipelines Slack channel

• Review GitHub Actions logs

• Check Databricks workspace notifications