CLI Commands Reference

Overview

This document provides a complete reference for all command-line operations in the ML Pipelines repository, including Makefile targets and Databricks CLI commands.

Table of Contents

• Makefile Commands

• Databricks Bundle Commands

• Databricks Catalog Commands

• Databricks Pipeline Commands

• Databricks Job Commands

• Databricks Model Commands

• Databricks SQL Commands

• Databricks File System Commands

• AWS S3 Commands

• Common Workflows

Makefile Commands

Location: /Users/taylorlaing/Development/refresh-os/ml-pipelines/Makefile

help

Description: Show all available Makefile targets

make help

Output:

ML Pipelines - Databricks Asset Bundle Commands

Local Development:
  make validate         Validate bundle configuration for sandbox
  make deploy           Deploy to your personal sandbox catalog
  make copy-data        Copy sample data from dev to your sandbox volumes
  make test             Run Python tests

CI/CD Deployment (requires service principal auth):
  make deploy-dev       Deploy to dev environment
  make deploy-staging   Deploy to staging environment
  make deploy-prod      Deploy to production environment

Utility:
  make clean            Remove build artifacts

---

validate

Description: Validate bundle configuration for sandbox environment

make validate

What it does:

1. Runs databricks bundle validate -t sandbox

2. Checks YAML syntax

3. Verifies variable references

4. Validates resource configurations

When to use: Before deploying to catch configuration errors early

Example output:

Validating bundle configuration (sandbox target)...
Bundle validation passed

---

deploy

Description: Deploy to your personal sandbox catalog

make deploy

What it does:

1. Validates bundle configuration

2. Gets your username (e.g., taylor)

3. Creates catalog {username}_sandbox if needed

4. Creates S3 volume directories

5. Builds Python wheel with uv build --wheel

6. Deploys all resources to dev workspace

7. Resources named as [dev {username}] resource_name_sandbox

Variables:

• SHORT_NAME: Auto-detected from current user

• CATALOG_NAME: ${SHORT_NAME}_sandbox

• S3_BUCKET: ref-ml-core-dev-workspace-bucket

Example:

$ make deploy
Validating bundle configuration (sandbox target)...
Bundle validation passed
Deploying to your personal sandbox...
Checking if catalog taylor_sandbox exists...
Catalog already exists
Creating S3 volume directories for taylor_sandbox...
S3 directories created
Deploying bundle...
Sandbox deployment complete

---

copy-data

Description: Copy sample data from dev volumes to sandbox for testing

make copy-data

What it does:

1. Runs /Users/taylorlaing/Development/refresh-os/ml-pipelines/scripts/copy-dev-data-to-sandbox.sh

2. Copies data files from dev volumes to {username}_sandbox volumes

3. Useful for testing pipelines with real data

When to use: After initial sandbox deployment to populate test data

---

deploy-dev

Description: Deploy to dev environment (CI/CD only)

make deploy-dev

What it does:

1. Validates bundle for dev target

2. Deploys to dev workspace

3. Uses service principal authentication

Authentication: Requires DATABRICKS_AUTH_TYPE=github-oidc (set by GitHub Actions)

When to use:

• Automatically in GitHub Actions on PR merge

• Manual testing with service principal credentials (rare)

---

deploy-staging

Description: Deploy to staging environment (CI/CD only)

make deploy-staging

Similar to: deploy-dev but for staging environment

---

deploy-prod

Description: Deploy to production environment (CI/CD only)

make deploy-prod

Similar to: deploy-dev but for production environment

Warning: Use with caution - deploys to production

---

test

Description: Run Python tests using pytest

make test

What it does:

1. Runs uv run pytest

2. Executes all tests in tests/ directory

3. Shows test results and coverage

Example:

$ make test
Running Python tests...
============================= test session starts ==============================
collected 42 items

tests/test_schemas.py ..................                                 [ 42%]
tests/test_models.py ........................                            [100%]

============================== 42 passed in 2.34s ==============================
Tests passed

---

clean

Description: Remove build artifacts and cache

make clean

What it does:

1. Removes dist/ directory (wheel files)

2. Removes build/ directory

3. Removes *.egg-info directories

4. Removes __pycache__ directories

5. Deletes .pyc files

When to use:

• Before rebuilding from scratch

• When debugging build issues

• To reclaim disk space

Example:

$ make clean
Cleaning build artifacts...
Cleanup complete

---

validate-dev, validate-staging, validate-prod

Description: Validate bundle for specific environments

make validate-dev
make validate-staging
make validate-prod

What it does: Validates bundle configuration for the specified target

When to use:

• Testing environment-specific configurations

• Debugging CI/CD validation failures

Databricks Bundle Commands

bundle validate

Description: Validate bundle configuration

databricks bundle validate [OPTIONS]

Options:

• -t, --target <target>: Target environment (sandbox, dev, staging, prod)

• --var <key=value>: Override variable value

Examples:

# Validate sandbox (default)
databricks bundle validate

# Validate dev
databricks bundle validate -t dev

# Validate with variable override
databricks bundle validate -t dev --var="catalog_name=test_dev"

When to use: Before deployment to catch errors

---

bundle deploy

Description: Deploy bundle to Databricks workspace

databricks bundle deploy [OPTIONS]

Options:

• -t, --target <target>: Target environment

• --var <key=value>: Override variable value

• --force: Force deployment even if validation fails (use cautiously)

Examples:

# Deploy to sandbox (default)
databricks bundle deploy

# Deploy to dev
databricks bundle deploy -t dev

# Deploy with variable override
databricks bundle deploy -t sandbox --var="catalog_name=taylor_test"

What happens:

1. Validates configuration

2. Builds artifacts (Python wheels)

3. Uploads to workspace

4. Creates/updates resources (pipelines, jobs, volumes)

---

bundle destroy

Description: Delete all resources created by bundle

databricks bundle destroy [OPTIONS]

Options:

• -t, --target <target>: Target environment

• --auto-approve: Skip confirmation prompt

Examples:

# Destroy sandbox resources (with confirmation)
databricks bundle destroy -t sandbox

# Destroy without confirmation (dangerous!)
databricks bundle destroy -t sandbox --auto-approve

Warning: This deletes all pipelines, jobs, and volumes. Use with caution!

---

bundle run

Description: Run a job from the bundle

databricks bundle run <job_name> [OPTIONS]

Options:

• -t, --target <target>: Target environment

Examples:

# Run model registration job
databricks bundle run register_sentiment_analysis -t sandbox

# Run with specific target
databricks bundle run bronze_data_ingestion -t dev

Databricks Catalog Commands

catalogs list

Description: List all catalogs you have access to

databricks catalogs list [OPTIONS]

Options:

• --profile <profile>: Databricks profile to use

• --output <format>: Output format (json, table)

Examples:

# List catalogs in dev workspace
databricks catalogs list --profile ref-dev

# List as JSON
databricks catalogs list --profile ref-dev --output json

# Filter for sandbox
databricks catalogs list --profile ref-dev | grep sandbox

---

catalogs get

Description: Get details of a specific catalog

databricks catalogs get <catalog_name> [OPTIONS]

Examples:

# Get sandbox catalog details
databricks catalogs get taylor_sandbox --profile ref-dev

# Get as JSON
databricks catalogs get dev --profile ref-dev --output json

---

catalogs create

Description: Create a new catalog

databricks catalogs create <catalog_name> [OPTIONS]

Options:

• --storage-root <s3_path>: S3 location for catalog

• --comment <comment>: Catalog description

Examples:

# Create sandbox catalog
databricks catalogs create taylor_sandbox \
  --storage-root s3://ref-ml-core-dev-workspace-bucket/ \
  --profile ref-dev

# With comment
databricks catalogs create test_catalog \
  --storage-root s3://bucket/ \
  --comment "Test catalog for experimentation" \
  --profile ref-dev

---

catalogs delete

Description: Delete a catalog

databricks catalogs delete <catalog_name> [OPTIONS]

Options:

• --force: Delete even if not empty

Examples:

# Delete empty catalog
databricks catalogs delete test_catalog --profile ref-dev

# Force delete non-empty catalog
databricks catalogs delete taylor_sandbox --force --profile ref-dev

Warning: Force delete removes all schemas and tables!

Databricks Pipeline Commands

pipelines list-pipelines

Description: List all DLT pipelines

databricks pipelines list-pipelines [OPTIONS]

Options:

• --profile <profile>: Databricks profile

• --output <format>: Output format (json, table)

Examples:

# List all pipelines
databricks pipelines list-pipelines --profile ref-dev

# Filter by prefix
databricks pipelines list-pipelines --profile ref-dev | grep taylor

# Get as JSON and parse
databricks pipelines list-pipelines --profile ref-dev --output json | jq '.[] | {name, pipeline_id}'

---

pipelines get

Description: Get pipeline details

databricks pipelines get <pipeline_id> [OPTIONS]

Examples:

# Get pipeline details
databricks pipelines get abc123-pipeline-id --profile ref-dev

# Get as JSON
databricks pipelines get abc123-pipeline-id --profile ref-dev --output json

---

pipelines start-update

Description: Trigger a pipeline update (run)

databricks pipelines start-update <pipeline_id> [OPTIONS]

Options:

• --full-refresh: Full refresh of all tables

Examples:

# Start incremental update
databricks pipelines start-update abc123-pipeline-id --profile ref-dev

# Start full refresh
databricks pipelines start-update abc123-pipeline-id --full-refresh --profile ref-dev

---

pipelines stop

Description: Stop a running pipeline

databricks pipelines stop <pipeline_id> [OPTIONS]

Examples:

# Stop pipeline
databricks pipelines stop abc123-pipeline-id --profile ref-dev

---

pipelines reset

Description: Reset pipeline state (clear checkpoints)

databricks pipelines reset <pipeline_id> [OPTIONS]

Examples:

# Reset pipeline
databricks pipelines reset abc123-pipeline-id --profile ref-dev

Warning: This clears all streaming state. Next run will reprocess all data.

---

pipelines list-updates

Description: List recent pipeline updates

databricks pipelines list-updates <pipeline_id> [OPTIONS]

Options:

• --limit <n>: Number of updates to show

Examples:

# Show recent updates
databricks pipelines list-updates abc123-pipeline-id --profile ref-dev

# Show last 10 updates
databricks pipelines list-updates abc123-pipeline-id --limit 10 --profile ref-dev

---

pipelines get-update

Description: Get details of a specific pipeline update

databricks pipelines get-update <pipeline_id> <update_id> [OPTIONS]

Examples:

# Get update details
databricks pipelines get-update abc123-pipeline-id def456-update-id --profile ref-dev

Databricks Job Commands

jobs list

Description: List all jobs

databricks jobs list [OPTIONS]

Examples:

# List all jobs
databricks jobs list --profile ref-dev

# Filter by name
databricks jobs list --profile ref-dev | grep register

# Get as JSON
databricks jobs list --profile ref-dev --output json

---

jobs get

Description: Get job details

databricks jobs get <job_id> [OPTIONS]

Examples:

# Get job configuration
databricks jobs get 123456 --profile ref-dev

---

jobs run-now

Description: Trigger a job run

databricks jobs run-now <job_id> [OPTIONS]

Options:

• --notebook-params <json>: Parameters for notebook task

• --python-params <args>: Parameters for Python task

Examples:

# Run job immediately
databricks jobs run-now 123456 --profile ref-dev

# Run with parameters
databricks jobs run-now 123456 \
  --notebook-params '{"catalog":"dev","version":"v2"}' \
  --profile ref-dev

---

jobs list-runs

Description: List recent job runs

databricks jobs list-runs [OPTIONS]

Options:

• --job-id <job_id>: Filter by job

• --limit <n>: Number of runs to show

• --active-only: Show only running jobs

Examples:

# List recent runs for all jobs
databricks jobs list-runs --profile ref-dev --limit 20

# List runs for specific job
databricks jobs list-runs --job-id 123456 --profile ref-dev

# Show only active runs
databricks jobs list-runs --active-only --profile ref-dev

---

jobs get-run

Description: Get details of a specific job run

databricks jobs get-run <run_id> [OPTIONS]

Examples:

# Get run details
databricks jobs get-run 789012 --profile ref-dev

# Get as JSON and check status
databricks jobs get-run 789012 --profile ref-dev --output json | jq '.state.life_cycle_state'

---

jobs cancel-run

Description: Cancel a running job

databricks jobs cancel-run <run_id> [OPTIONS]

Examples:

# Cancel job run
databricks jobs cancel-run 789012 --profile ref-dev

Databricks Model Commands

model-serving list

Description: List model serving endpoints

databricks model-serving list [OPTIONS]

Examples:

# List all endpoints
databricks model-serving list --profile ref-dev

# Get as JSON
databricks model-serving list --profile ref-dev --output json

---

model-serving get

Description: Get model serving endpoint details

databricks model-serving get <endpoint_name> [OPTIONS]

Examples:

# Get endpoint details
databricks model-serving get sentiment_analysis --profile ref-dev

---

model-serving create

Description: Create a model serving endpoint

databricks model-serving create [OPTIONS]

Options:

• --name <name>: Endpoint name

• --config @<file>: Configuration file (JSON)

Examples:

# Create from config file
databricks model-serving create \
  --name sentiment_analysis \
  --config @model_config.json \
  --profile ref-dev

---

model-serving update

Description: Update model serving endpoint

databricks model-serving update <endpoint_name> [OPTIONS]

Examples:

# Update endpoint configuration
databricks model-serving update sentiment_analysis \
  --config @updated_config.json \
  --profile ref-dev

---

model-serving delete

Description: Delete model serving endpoint

databricks model-serving delete <endpoint_name> [OPTIONS]

Examples:

# Delete endpoint
databricks model-serving delete sentiment_analysis --profile ref-dev

Databricks SQL Commands

sql execute

Description: Execute SQL query

databricks sql execute <query> [OPTIONS]

Options:

• --profile <profile>: Databricks profile

• --warehouse-id <id>: SQL warehouse ID

Examples:

# Execute simple query
databricks sql execute "SELECT COUNT(*) FROM dev.bronze.messages" --profile ref-dev

# Multi-line query
databricks sql execute "
  SELECT
    DATE(timestamp) as date,
    COUNT(*) as count
  FROM dev.bronze.messages
  GROUP BY DATE(timestamp)
  ORDER BY date DESC
  LIMIT 10
" --profile ref-dev

# Create table
databricks sql execute "
  CREATE TABLE taylor_sandbox.gold.test_table AS
  SELECT * FROM dev.silver.sentiment_features LIMIT 100
" --profile ref-dev

---

sql queries list

Description: List saved SQL queries

databricks sql queries list [OPTIONS]

Examples:

# List all queries
databricks sql queries list --profile ref-dev

Databricks File System Commands

fs ls

Description: List files in workspace or DBFS

databricks fs ls <path> [OPTIONS]

Options:

• --absolute: Show absolute paths

• --long: Show detailed info (size, modified time)

Examples:

# List volumes
databricks fs ls /Volumes/ --profile ref-dev

# List specific volume
databricks fs ls /Volumes/taylor_sandbox/bronze/ --profile ref-dev

# List with details
databricks fs ls /Volumes/dev/bronze/raw_messages/ --long --profile ref-dev

# List bundle artifacts
databricks fs ls /Workspace/Users/[email protected]/.bundle/ml_pipelines/ --profile ref-dev

---

fs cp

Description: Copy files

databricks fs cp <source> <destination> [OPTIONS]

Options:

• --recursive: Copy directories

• --overwrite: Overwrite existing files

Examples:

# Copy file
databricks fs cp local_file.json /Volumes/dev/bronze/raw_messages/ --profile ref-dev

# Copy directory
databricks fs cp -r local_data/ /Volumes/dev/bronze/ --profile ref-dev

---

fs rm

Description: Remove files

databricks fs rm <path> [OPTIONS]

Options:

• --recursive: Remove directories

Examples:

# Remove file
databricks fs rm /Volumes/dev/bronze/test.json --profile ref-dev

# Remove directory
databricks fs rm -r /Volumes/taylor_sandbox/temp/ --profile ref-dev

AWS S3 Commands

s3 ls

Description: List S3 bucket contents

aws s3 ls <s3_uri> [OPTIONS]

Options:

• --profile <profile>: AWS profile to use

• --recursive: List recursively

Examples:

# List bucket root
aws s3 ls s3://ref-ml-core-dev-workspace-bucket/ --profile ref-ml-core

# List catalog volumes
aws s3 ls s3://ref-ml-core-dev-workspace-bucket/dev/volumes/ --profile ref-ml-core

# List recursively
aws s3 ls s3://ref-ml-core-dev-workspace-bucket/dev/volumes/bronze/ \
  --recursive --profile ref-ml-core

---

s3 cp

Description: Copy files to/from S3

aws s3 cp <source> <destination> [OPTIONS]

Options:

• --recursive: Copy directories

• --profile <profile>: AWS profile

Examples:

# Upload file
aws s3 cp local_data.json \
  s3://ref-ml-core-dev-workspace-bucket/dev/volumes/bronze/messages/ \
  --profile ref-ml-core

# Upload directory
aws s3 cp data/ \
  s3://ref-ml-core-dev-workspace-bucket/dev/volumes/bronze/messages/ \
  --recursive --profile ref-ml-core

# Download file
aws s3 cp \
  s3://ref-ml-core-dev-workspace-bucket/dev/volumes/bronze/messages/data.json \
  ./local_data.json \
  --profile ref-ml-core

---

s3api put-object

Description: Create S3 directory placeholder

aws s3api put-object --bucket <bucket> --key <key> [OPTIONS]

Options:

• --profile <profile>: AWS profile

• --no-cli-pager: Disable pager

Examples:

# Create volume directory
aws s3api put-object \
  --bucket ref-ml-core-dev-workspace-bucket \
  --key taylor_sandbox/volumes/bronze/messages/ \
  --profile ref-ml-core \
  --no-cli-pager

# Used by make deploy to create volume directories

---

s3api get-bucket-policy

Description: Get S3 bucket policy

aws s3api get-bucket-policy --bucket <bucket> [OPTIONS]

Examples:

# Get bucket policy
aws s3api get-bucket-policy \
  --bucket ref-ml-core-dev-workspace-bucket \
  --profile ref-ml-core

Common Workflows

Deploy Changes to Sandbox

# 1. Make code changes
code src/models/sentiment_analysis/model.py

# 2. Validate configuration
make validate

# 3. Deploy to sandbox
make deploy

# 4. Test pipeline
databricks pipelines start-update <pipeline_id> --profile ref-dev

# 5. Verify results
databricks sql execute "SELECT COUNT(*) FROM taylor_sandbox.silver.sentiment_features" --profile ref-dev

---

Create and Test New Pipeline

# 1. Create pipeline files
mkdir -p resources/pipelines/silver/new_pipeline/transformations
touch resources/pipelines/silver/new_pipeline/new_pipeline.pipeline.yml
touch resources/pipelines/silver/new_pipeline/transformations/01_transform.sql

# 2. Add pipeline configuration
code resources/pipelines/silver/new_pipeline/new_pipeline.pipeline.yml

# 3. Validate
make validate

# 4. Deploy to sandbox
make deploy

# 5. Find pipeline ID
databricks pipelines list-pipelines --profile ref-dev | grep new_pipeline

# 6. Run pipeline
databricks pipelines start-update <pipeline_id> --profile ref-dev

# 7. Monitor progress
databricks pipelines get <pipeline_id> --profile ref-dev

# 8. Check results
databricks sql execute "SELECT * FROM taylor_sandbox.silver.new_table LIMIT 10" --profile ref-dev

---

Register and Test Model

# 1. Create model registration job
code resources/jobs/model_registration/internal/my_model/register_my_model.job.yml

# 2. Deploy to sandbox
make deploy

# 3. Find job ID
databricks jobs list --profile ref-dev | grep register_my_model

# 4. Run registration job
databricks jobs run-now <job_id> --profile ref-dev

# 5. Monitor job
databricks jobs get-run <run_id> --profile ref-dev

# 6. Verify model registered
databricks sql execute "SHOW MODELS IN taylor_sandbox.models" --profile ref-dev

# 7. Test model
databricks sql execute "
  SELECT ai_query('taylor_sandbox_my_model', 'test input')
" --profile ref-dev

---

Debug Pipeline Failure

# 1. Get pipeline status
databricks pipelines get <pipeline_id> --profile ref-dev

# 2. List recent updates
databricks pipelines list-updates <pipeline_id> --limit 5 --profile ref-dev

# 3. Get failed update details
databricks pipelines get-update <pipeline_id> <update_id> --profile ref-dev

# 4. Check table exists
databricks sql execute "SHOW TABLES IN taylor_sandbox.silver" --profile ref-dev

# 5. Check data
databricks sql execute "
  SELECT * FROM taylor_sandbox.silver.problematic_table
  LIMIT 10
" --profile ref-dev

# 6. Fix issue (e.g., schema conflict)
# Use CREATE OR REPLACE in transformation SQL

# 7. Redeploy
make deploy

# 8. Retry pipeline
databricks pipelines start-update <pipeline_id> --profile ref-dev

---

Check Data Quality

# 1. Check row counts
databricks sql execute "
  SELECT
    'bronze.messages' as table_name,
    COUNT(*) as row_count
  FROM dev.bronze.messages
  UNION ALL
  SELECT
    'silver.sentiment_features' as table_name,
    COUNT(*) as row_count
  FROM dev.silver.sentiment_features
" --profile ref-dev

# 2. Check for nulls
databricks sql execute "
  SELECT
    COUNT(*) as total_rows,
    SUM(CASE WHEN sentiment_score IS NULL THEN 1 ELSE 0 END) as null_scores,
    SUM(CASE WHEN sentiment_score IS NULL THEN 1 ELSE 0 END) * 100.0 / COUNT(*) as null_pct
  FROM dev.silver.sentiment_features
" --profile ref-dev

# 3. Check data freshness
databricks sql execute "
  SELECT
    MAX(created_date) as latest_data,
    CURRENT_TIMESTAMP() as now,
    TIMESTAMPDIFF(HOUR, MAX(created_date), CURRENT_TIMESTAMP()) as hours_old
  FROM dev.bronze.messages
" --profile ref-dev

---

Cleanup Sandbox

# 1. List sandbox resources
databricks pipelines list-pipelines --profile ref-dev | grep taylor
databricks jobs list --profile ref-dev | grep taylor

# 2. Drop test tables
databricks sql execute "DROP TABLE IF EXISTS taylor_sandbox.gold.test_table" --profile ref-dev

# 3. Clean build artifacts
make clean

# 4. Optional: Drop entire sandbox catalog
databricks sql execute "DROP CATALOG taylor_sandbox CASCADE" --profile ref-dev

Tips and Tricks

Using JSON Output with jq

# Extract pipeline IDs
databricks pipelines list-pipelines --profile ref-dev --output json | jq -r '.[].pipeline_id'

# Get pipeline names and states
databricks pipelines list-pipelines --profile ref-dev --output json | \
  jq -r '.[] | "\(.name): \(.state)"'

# Filter running jobs
databricks jobs list-runs --profile ref-dev --output json | \
  jq '.runs[] | select(.state.life_cycle_state == "RUNNING") | {job_id, run_id}'

Using Shell Loops

# Stop all sandbox pipelines
for id in $(databricks pipelines list-pipelines --profile ref-dev --output json | \
            jq -r '.[] | select(.name | startswith("taylor_")) | .pipeline_id'); do
  databricks pipelines stop $id --profile ref-dev
done

# Trigger all pipelines
for id in $(databricks pipelines list-pipelines --profile ref-dev --output json | \
            jq -r '.[].pipeline_id'); do
  databricks pipelines start-update $id --profile ref-dev
done

Environment Variables

# Set default profile
export DATABRICKS_PROFILE=ref-dev
databricks current-user me  # Uses ref-dev automatically

# Set credentials for CI/CD
export DATABRICKS_AUTH_TYPE=github-oidc
export DATABRICKS_HOST=https://dbc-a72d6af9-df3d.cloud.databricks.com
export DATABRICKS_CLIENT_ID=03ff99cd-a352-40bb-9d33-414c9ad9e7aa

Related Documentation

• Configuration Reference - databricks.yml and resource YAMLs

• Local Development Guide - Sandbox workflow

• Deployment Guide - CI/CD deployment

• Troubleshooting Guide - Common issues

External Resources

• Databricks CLI Reference

• Databricks Bundle CLI

• AWS CLI Reference