Service Principal Setup Guide

ML Pipelines Environment-Specific Service Principals

Date: 2025-09-30 Purpose: Create isolated service principals for ML pipeline deployments across environments

---

Service Principals to Create

Create these three service principals in your Databricks account:

1. Development Service Principal

• Name: ml-pipelines-dev-service-principal

• Purpose: Deploy and run ML pipelines in dev workspace

• Workspace Access: Dev only

• Permissions: Full access to dev catalog

2. Staging Service Principal

• Name: ml-pipelines-staging-service-principal

• Purpose: Deploy and run ML pipelines in staging workspace

• Workspace Access: Staging only

• Permissions: Full access to staging catalog

3. Production Service Principal

• Name: ml-pipelines-prod-service-principal

• Purpose: Deploy and run ML pipelines in prod workspace

• Workspace Access: Prod only (most restrictive)

• Permissions: Full access to prod catalog

---

Creation Steps

Option A: Via Databricks UI (Quickest)

1. Navigate to Account Console

   • Go to https://accounts.cloud.databricks.com

   • Login as account admin

2. Create Service Principals

   • Click "User Management" → "Service Principals"

   • Click "Add Service Principal"

   • For each environment (dev, staging, prod):

     • Name: ml-pipelines-{env}-service-principal

     • Click "Add"

     • COPY THE APPLICATION ID (you'll need this)

3. Generate OAuth Secrets

   • Click on each service principal

   • Go to "OAuth Secrets" tab

   • Click "Generate Secret"

   • COPY SECRET (shown only once!)

   • Store in password manager

4. Assign to Workspaces

   • For ml-pipelines-dev-service-principal:

     • Go to "Workspaces" → Dev workspace → "Permissions"

     • Add service principal with "User" role

   • Repeat for staging and prod service principals

Option B: Via Databricks CLI (Recommended for IaC)

# Set up for account-level operations
export DATABRICKS_HOST="https://accounts.cloud.databricks.com"
export DATABRICKS_ACCOUNT_ID="90a79e25-1700-44e8-8e94-1aba556fb11b"

# Create dev service principal
databricks service-principals create \
  --display-name "ml-pipelines-dev-service-principal" \
  --profile ref-admin

# Create staging service principal
databricks service-principals create \
  --display-name "ml-pipelines-staging-service-principal" \
  --profile ref-admin

# Create prod service principal
databricks service-principals create \
  --display-name "ml-pipelines-prod-service-principal" \
  --profile ref-admin

# List all service principals to get IDs
databricks service-principals list --profile ref-admin

---

GitHub OIDC Configuration

For each service principal, you need to configure GitHub OIDC trust.

Steps for Each Service Principal:

1. Navigate to Service Principal Settings

   • Account Console → User Management → Service Principals

   • Click on the service principal (e.g., ml-pipelines-dev-service-principal)

2. Configure GitHub OIDC

   • Go to "Authentication" tab

   • Click "Configure GitHub OIDC"

   • GitHub Organization: refresh-os

   • GitHub Repository: ml-pipelines (or leave blank for org-wide)

   • GitHub Environment (optional but recommended):

     • For dev SP: development

     • For staging SP: staging

     • For prod SP: production

   • Click "Save"

3. Note the Client ID

   • The Application ID from step 1 is your OIDC Client ID

   • You'll use this in GitHub workflows

---

Information to Collect

After creating the service principals, fill out this table:

Service Principal	Application ID (UUID)	OIDC Client ID	Notes
ml-pipelines-dev-service-principal	____________________	Same as App ID	For GitHub dev workflow
ml-pipelines-staging-service-principal	____________________	Same as App ID	For GitHub staging workflow
ml-pipelines-prod-service-principal	____________________	Same as App ID	For GitHub prod workflow

---

Where to Use These IDs

1. databricks.yml Configuration

# /Users/taylorlaing/Development/refresh-os/ml-pipelines/databricks.yml

targets:
  dev:
    run_as:
      # Option A: Use service principal name (Databricks looks up by name)
      service_principal_name: "ml-pipelines-dev-service-principal"

      # Option B: Use application ID directly
      # application_id: "<DEV_SP_APPLICATION_ID>"

  staging:
    run_as:
      service_principal_name: "ml-pipelines-staging-service-principal"
      # application_id: "<STAGING_SP_APPLICATION_ID>"

  prod:
    run_as:
      service_principal_name: "ml-pipelines-prod-service-principal"
      # application_id: "<PROD_SP_APPLICATION_ID>"

Recommendation: Use service_principal_name (Option A) - cleaner and more readable

2. GitHub Workflow Files

File: .github/workflows/ml_pipelines_dev_deploy.yml

env:
  DATABRICKS_AUTH_TYPE: github-oidc
  DATABRICKS_HOST: https://dbc-a72d6af9-df3d.cloud.databricks.com
  DATABRICKS_CLIENT_ID: <DEV_SP_APPLICATION_ID>  # Replace with actual ID

File: .github/workflows/ml_pipelines_staging_deploy.yml

env:
  DATABRICKS_AUTH_TYPE: github-oidc
  DATABRICKS_HOST: https://dbc-fab2a42a-8d11.cloud.databricks.com
  DATABRICKS_CLIENT_ID: <STAGING_SP_APPLICATION_ID>  # Replace with actual ID

File: .github/workflows/ml_pipelines_prod_deploy.yml

env:
  DATABRICKS_AUTH_TYPE: github-oidc
  DATABRICKS_HOST: https://dbc-028d9e53-7ce6.cloud.databricks.com
  DATABRICKS_CLIENT_ID: <PROD_SP_APPLICATION_ID>  # Replace with actual ID

3. Terraform Configuration (Optional but Recommended)

File: /Users/taylorlaing/Development/refresh-os/infra-core/stacks/ml-databricks/main.tf

# Data sources to look up service principals by name
data "databricks_service_principal" "ml_pipelines_dev" {
  provider       = databricks.mws
  display_name   = "ml-pipelines-dev-service-principal"
}

data "databricks_service_principal" "ml_pipelines_staging" {
  provider       = databricks.mws
  display_name   = "ml-pipelines-staging-service-principal"
}

data "databricks_service_principal" "ml_pipelines_prod" {
  provider       = databricks.mws
  display_name   = "ml-pipelines-prod-service-principal"
}

# Grant service principals access to their respective catalogs
resource "databricks_grants" "dev_catalog_sp_grants" {
  provider = databricks.dev
  catalog  = "dev"

  grant {
    principal  = data.databricks_service_principal.ml_pipelines_dev.application_id
    privileges = ["ALL_PRIVILEGES"]
  }
}

resource "databricks_grants" "staging_catalog_sp_grants" {
  provider = databricks.staging
  catalog  = "staging"

  grant {
    principal  = data.databricks_service_principal.ml_pipelines_staging.application_id
    privileges = ["ALL_PRIVILEGES"]
  }
}

resource "databricks_grants" "prod_catalog_sp_grants" {
  provider = databricks.prod
  catalog  = "prod"

  grant {
    principal  = data.databricks_service_principal.ml_pipelines_prod.application_id
    privileges = ["ALL_PRIVILEGES"]
  }
}

---

Verification Steps

After creating service principals and updating configurations:

Test 1: CLI Authentication

# Test dev service principal
export DATABRICKS_HOST="https://dbc-a72d6af9-df3d.cloud.databricks.com"
export DATABRICKS_CLIENT_ID="<DEV_SP_APPLICATION_ID>"
export DATABRICKS_CLIENT_SECRET="<DEV_SP_SECRET>"

databricks auth login --client-id $DATABRICKS_CLIENT_ID --client-secret $DATABRICKS_CLIENT_SECRET
databricks catalogs list

# Should see 'dev' catalog in output

Test 2: GitHub OIDC

# Trigger a dev deployment via GitHub Actions
# Check workflow logs for successful OIDC authentication
# Look for: "Successfully authenticated with Databricks using GitHub OIDC"

Test 3: Catalog Permissions

# Verify service principal can write to its catalog
databricks sql execute "CREATE TABLE dev.bronze.test_table AS SELECT 1 as id" --profile ref-dev

# Verify it CANNOT write to other catalogs (should fail)
databricks sql execute "CREATE TABLE staging.bronze.test_table AS SELECT 1 as id" --profile ref-dev
# Expected: 403 Forbidden or Permission Denied

---

Security Best Practices

Principle of Least Privilege

• Each service principal only has access to its environment's workspace

• Each service principal only has permissions on its environment's catalog

• Service principals cannot access developer sandbox catalogs

• GitHub OIDC tied to specific GitHub environments

Secret Management

• NEVER commit service principal secrets to git

• Use GitHub OIDC (no secrets in GitHub at all)

• Store OAuth secrets in password manager (only needed for manual testing)

• Rotate secrets every 90 days (set calendar reminder)

Audit Trail

• All deployments run as service principals (not personal accounts)

• Service principal names clearly indicate purpose and environment

• Databricks audit logs track all service principal actions

• GitHub Actions logs provide deployment audit trail

---

Troubleshooting

Issue: "Service principal not found"

Cause: Service principal name typo or not created yet Fix: Verify service principal exists: databricks service-principals list --profile ref-admin

Issue: "OIDC authentication failed"

Cause: GitHub OIDC not configured or Client ID mismatch Fix:

1. Verify OIDC configuration in Databricks account console

2. Check Client ID matches Application ID in workflow file

3. Ensure GitHub environment name matches OIDC configuration

Issue: "Permission denied on catalog"

Cause: Service principal lacks catalog permissions Fix: Run Terraform grants or manually grant via SQL:

GRANT ALL PRIVILEGES ON CATALOG dev TO `<dev_service_principal_app_id>`;

Issue: "Cannot access workspace"

Cause: Service principal not assigned to workspace Fix: Account Console → Workspaces → [workspace] → Permissions → Add service principal

---

Next Steps

1. Create the three service principals (via UI or CLI)

2. Configure GitHub OIDC for each service principal

3. Fill in the Application IDs in the table above

4. Update GitHub workflow files with correct Client IDs

5. Update databricks.yml with service principal names

6. Add Terraform grants for catalog permissions

7. Test authentication via CLI

8. Test deployment via GitHub Actions

---

Document Version: 1.0 Last Updated: 2025-09-30 Author: Claude Code Status: Ready for Implementation