Secret Rotation Runbook

Purpose

This runbook provides step-by-step procedures for rotating service principal credentials, GitHub OIDC configurations, and other secrets used in the ML Pipelines infrastructure. Regular rotation maintains security and prevents unauthorized access.

When to Rotate

Scheduled Rotation (Recommended)

• Service Principal OIDC: Every 180 days (automatic via GitHub OIDC)

• GitHub Personal Access Tokens: Every 90 days

• Databricks Personal Access Tokens: Every 90 days (if used)

• AWS IAM Credentials: Every 90 days (if not using roles)

Immediate Rotation (Security Events)

Rotate immediately if:

• Credential exposure suspected or confirmed

• Employee offboarding (within 24 hours)

• Security audit findings

• Compromised GitHub repository

• Unauthorized access detected

• Compliance requirement

Pre-Rotation Checklist

Before rotating any credentials:

• Identify all systems using the credential

• Schedule maintenance window (if needed)

• Notify team of rotation schedule

• Backup current configuration (document current settings)

• Test procedure in dev environment first

• Prepare rollback plan (keep old credentials temporarily)

• Communication plan ready (Slack, email)

---

Procedure 1: Service Principal OIDC Configuration

Overview

GitHub OIDC uses federated authentication, eliminating the need for long-lived secrets. The OIDC token is automatically issued by GitHub Actions and validated by Databricks. This procedure updates the trust configuration.

When to Rotate

• Changing GitHub repository structure

• Updating environment names

• Modifying security policies

• Annually as best practice

Step-by-Step Procedure

Step 1: Document Current Configuration

1. Log into Databricks Account Console:

   https://accounts.cloud.databricks.com/

2. Navigate to: Service Principals → Select service principal

3. Click Authentication → GitHub OIDC

4. Document current settings:

   Issuer: https://token.actions.githubusercontent.com
   Audience: https://dbc-a72d6af9-df3d.cloud.databricks.com/oidc/v1/token
   Subject Pattern: repo:refresh-os/ml-pipelines:environment:development

5. Save configuration to secure location

Step 2: Verify GitHub Workflow Configuration

1. Check workflow files match OIDC config:

   grep -r "DATABRICKS_CLIENT_ID" .github/workflows/

2. Verify environment names:

   # In .github/workflows/ml_pipelines_dev_deploy.yml
   jobs:
     deploy_dev:
       environment: development  # Must match OIDC subject pattern

Step 3: Update OIDC Configuration (If Needed)

1. In Databricks Account Console → Service Principal → Authentication

2. Click Edit on GitHub OIDC configuration

3. Update fields:

   • Issuer: https://token.actions.githubusercontent.com (rarely changes)

   • Audience: Workspace-specific OIDC URL

   • Subject Pattern: repo:refresh-os/ml-pipelines:environment:{environment}

4. Click Save

Step 4: Test OIDC Authentication

1. Trigger test workflow in dev:

   # Push to test branch
   git checkout -b test-oidc-rotation
   git commit --allow-empty -m "Test OIDC authentication"
   git push origin test-oidc-rotation

2. Monitor GitHub Actions log for authentication:

   Requesting OIDC token from GitHub...
   Authenticating to Databricks...
   Authentication successful

3. If successful, merge test branch

4. If failed, check error message and rollback

Step 5: Verify All Environments

1. Test dev environment:

   # Trigger dev deployment
   git push origin main

2. Test staging environment (after dev succeeds)

3. Test production environment (after staging succeeds)

Step 6: Update Documentation

1. Update this runbook with new configuration (if changed)

2. Record rotation in security log:

   Date: 2025-10-03
   Action: OIDC configuration verified
   Environments: dev, staging, prod
   Result: Successful

Rollback Procedure

If OIDC authentication fails:

1. Revert to previous subject pattern in Databricks Account Console

2. Verify previous configuration works:

   # Re-run workflow
   gh workflow run ml_pipelines_dev_deploy.yml

3. If still failing, check:

   • Service principal still exists

   • Workspace URL correct

   • GitHub environment name matches

Verification

• Dev workflow completes successfully

• Staging workflow completes successfully

• Prod workflow completes successfully

• No authentication errors in logs

• All environments accessible

• Rotation documented

Timeline: 15-30 minutes (all environments)

---

Procedure 2: GitHub Personal Access Token (GH_PAT)

Overview

GitHub Personal Access Token is used in workflows for checking out code with submodules. This token should be rotated every 90 days.

When to Rotate

• Every 90 days (scheduled)

• Suspected exposure

• Employee offboarding

• Token expiration approaching

Step-by-Step Procedure

Step 1: Create New GitHub PAT

1. Go to GitHub → Settings → Developer settings → Personal access tokens → Tokens (classic)

2. Click Generate new token (classic)

3. Configure token:

   • Note: ML Pipelines CI/CD - 2025-Q4

   • Expiration: 90 days

   • Scopes:

     • repo (full control of private repositories)

     • workflow (update GitHub Action workflows)

4. Click Generate token

5. Copy token immediately (shown only once)

Step 2: Update GitHub Secret

1. Go to repository → Settings → Secrets and variables → Actions

2. Find GH_PAT secret

3. Click Update

4. Paste new token value

5. Click Update secret

Step 3: Test New Token

1. Trigger a workflow that uses GH_PAT:

   git commit --allow-empty -m "Test new GH_PAT"
   git push origin main

2. Monitor workflow for submodule checkout:

   - name: Checkout code
     uses: actions/checkout@v4
     with:
       submodules: true
       token: ${{ secrets.GH_PAT }}  # Should use new token

3. Verify no authentication errors

Step 4: Revoke Old Token

1. Wait 24 hours to ensure new token works

2. Go to GitHub → Settings → Developer settings → Personal access tokens

3. Find old token (by note with previous date)

4. Click Delete

5. Confirm deletion

Step 5: Update Rotation Schedule

1. Set reminder for 90 days:

   Next rotation: 2026-01-03
   Token note: ML Pipelines CI/CD - 2026-Q1

2. Document in security log

Rollback Procedure

If new token fails:

1. Re-update GH_PAT secret with old token (if not yet revoked)

2. Verify old token works:

   gh workflow run ml_pipelines_dev_deploy.yml

3. Investigate why new token failed:

   • Check token scopes

   • Verify token not expired

   • Ensure token associated with correct account

4. Create another new token with correct configuration

Verification

• Workflow runs successfully

• Submodules checked out properly

• No authentication errors

• Old token revoked (after 24hr)

• Next rotation scheduled

• Documentation updated

Timeline: 10-15 minutes

---

Procedure 3: Service Principal Permissions Audit

Overview

Periodically audit service principal permissions to ensure least privilege access and identify unused permissions.

When to Audit

• Quarterly (every 90 days)

• After major infrastructure changes

• Before compliance audits

• After adding new environments

Step-by-Step Procedure

Step 1: List All Service Principals

1. Get service principal IDs:

   ml-pipelines-dev: 03ff99cd-a352-40bb-9d33-414c9ad9e7aa
   ml-pipelines-staging: 93bda7cf-b009-49d8-8e8d-046677c8597e
   ml-pipelines-prod: 2af4b95a-d80f-4da6-bfba-9ec5a8a8ec9f

Step 2: Audit Catalog Permissions

For each environment:

# Dev catalog permissions
databricks grants get catalog dev --profile ref-dev

# Staging catalog permissions
databricks grants get catalog staging --profile ref-staging

# Production catalog permissions
databricks grants get catalog prod --profile ref-prod

Expected permissions:

Service Principal	Catalog	Permissions
ml-pipelines-dev	dev	ALL PRIVILEGES
ml-pipelines-staging	staging	ALL PRIVILEGES
ml-pipelines-staging	prod	USE CATALOG, SELECT (read-only)
ml-pipelines-prod	prod	ALL PRIVILEGES
ml-pipelines-prod	staging.models	USE CATALOG, SELECT (read-only)

Step 3: Audit Workspace Permissions

Check service principal can:

• Create and manage jobs

• Create and manage DLT pipelines

• Create and manage model serving endpoints

• Read/write to bundle directories

# List service principal workspace access
databricks service-principals get <sp_id> --profile ref-dev

Step 4: Audit S3 Permissions

Verify external location access:

# Dev external location
databricks external-locations get dev_volumes --profile ref-dev

# Check S3 bucket policy
aws s3api get-bucket-policy \
  --bucket ref-ml-core-dev-workspace-bucket \
  --profile ref-ml-core

Step 5: Document Findings

Create audit report:

# Service Principal Audit - 2025-10-03

## Summary
- 3 service principals reviewed
- All permissions align with least privilege
- No excess permissions found
- No security issues identified

## Findings by Environment

### Dev (ml-pipelines-dev)
- Catalog permissions: Correct
- Workspace permissions: Correct
- S3 permissions: Correct

### Staging (ml-pipelines-staging)
- Catalog permissions: Correct
- Workspace permissions: Correct
- S3 permissions: Correct
- Prod read access: Correct (read-only)

### Production (ml-pipelines-prod)
- Catalog permissions: Correct
- Workspace permissions: Correct
- S3 permissions: Correct
- Staging.models read access: Correct (read-only)

## Recommendations
- No changes needed
- Continue quarterly audits
- Next audit: 2026-01-03

Step 6: Remediate Issues (If Found)

If excess permissions found:

# Remove unnecessary permissions
databricks sql execute "
  REVOKE SELECT ON CATALOG dev FROM '<sp_id>'
" --profile ref-admin

If missing permissions:

# Grant required permissions
databricks sql execute "
  GRANT USE CATALOG ON prod TO '<staging_sp_id>'
" --profile ref-admin

Verification

• All service principals have correct catalog access

• No service principal has excessive permissions

• Cross-environment access limited (staging → prod read-only)

• S3 external locations properly configured

• Audit documented

• Issues remediated

Timeline: 30-45 minutes (all service principals)

---

Procedure 4: Emergency Credential Revocation

Overview

Use this procedure when credentials are compromised or suspected of compromise.

Immediate Actions (0-15 minutes)

Step 1: Revoke Compromised Credentials

For GitHub PAT:

1. Go to GitHub → Settings → Developer settings → Personal access tokens
2. Find compromised token
3. Click Delete
4. Confirm immediately

For Service Principal:

1. Databricks Account Console → Service Principals
2. Select compromised service principal
3. Click "Disable" (temporary)
4. Or "Delete" (permanent, requires recreation)

Step 2: Notify Team

Post in Slack immediately:

@channel SECURITY ALERT: Service principal credentials compromised

Affected: [dev/staging/prod] environment
Action taken: Credentials revoked
Impact: CI/CD deployments blocked temporarily
ETA for resolution: 30 minutes

Do not attempt deployments until all-clear issued.

Step 3: Assess Blast Radius

Check what systems are affected:

• CI/CD pipelines blocked?

• Manual deployments still work?

• Production systems impacted?

• Data access compromised?

Step 4: Review Access Logs

Check for unauthorized access:

# In Databricks Account Console
# Admin Console → Audit Logs
# Filter by service_principal_id and recent timeframe

Look for:

• Unexpected API calls

• Access from unknown IPs

• Data exports or downloads

• Permission changes

Recovery Actions (15-45 minutes)

Step 5: Create New Credentials

Follow Procedure 1 (OIDC) or Procedure 2 (PAT) above to create new credentials.

Step 6: Update All Systems

Update in priority order:

1. Production environment (highest priority)

2. Staging environment

3. Dev environment

4. Documentation

Step 7: Verify Security

• Old credentials confirmed revoked

• New credentials working

• No unauthorized access in logs

• All systems restored

• Team notified of resolution

Post-Incident Actions (1-24 hours)

Step 8: Root Cause Analysis

Document how compromise occurred:

• Was token committed to repository?

• Was token in logs?

• Was token shared in Slack?

• Was token phished?

Step 9: Implement Preventive Measures

Based on root cause:

• Add git-secrets pre-commit hook

• Update .gitignore

• Security training

• Process improvements

Step 10: Incident Report

Create incident report:

# Security Incident Report - Credential Compromise

## Incident Details
- Date: 2025-10-03 14:30 UTC
- Type: Service principal credential exposure
- Severity: High
- Duration: 45 minutes

## Timeline
- 14:30 - Credential exposure detected
- 14:32 - Credentials revoked
- 14:35 - Team notified
- 14:40 - New credentials created
- 15:00 - All systems restored
- 15:15 - Incident resolved

## Root Cause
[Detailed explanation of how credential was exposed]

## Impact
- CI/CD blocked for 30 minutes
- No unauthorized access detected
- No data exfiltration
- Production not affected

## Remediation
- Credentials rotated
- [Specific preventive measures]

## Lessons Learned
- [What went well]
- [What could be improved]
- [Action items]

Verification

• Incident timeline documented

• Root cause identified

• Preventive measures implemented

• Team trained on new procedures

• Monitoring enhanced

• Post-mortem completed

Timeline: 45-60 minutes (emergency rotation)

---

Post-Rotation Validation

After any credential rotation:

Validation Checklist

• CI/CD Pipelines: All workflows run successfully

• Manual Deployments: Local deployments work (if applicable)

• Service Principal Access: Can create/modify resources

• Data Access: Catalogs and tables accessible

• Model Endpoints: Can query model serving endpoints

• No Errors: No authentication errors in logs

• Documentation: Updated with new rotation date

• Team Notified: Rotation completion communicated

Test Commands

# Test dev deployment
make deploy-dev

# Test staging deployment
make deploy-staging

# Test production deployment
make deploy-prod

# Verify catalog access
databricks sql execute "SHOW CATALOGS" --profile ref-dev

# Test model endpoint
curl -X POST "https://dbc-028d9e53-7ce6.cloud.databricks.com/serving-endpoints/sentiment_analysis/invocations" \
  -H "Authorization: Bearer $DATABRICKS_TOKEN" \
  -d '{"inputs": ["test"]}'

---

Rotation Schedule Template

Use this template to track rotations:

# Credential Rotation Schedule

## GitHub PAT (GH_PAT)
- Current expiration: 2026-01-03
- Last rotation: 2025-10-03
- Next rotation: 2026-01-03
- Owner: Taylor Laing

## Service Principal OIDC
- Configuration verified: 2025-10-03
- Next audit: 2026-01-03
- Owner: Taylor Laing

## Databricks Personal Access Tokens
- Dev workspace: N/A (using OIDC)
- Staging workspace: N/A (using OIDC)
- Prod workspace: N/A (using OIDC)

## AWS IAM Roles
- Last reviewed: 2025-10-03
- Next review: 2026-01-03
- Owner: Infrastructure team

## Reminders Set
- [ ] Calendar reminder: 2025-12-15 (start planning)
- [ ] Email reminder: 2025-12-20
- [ ] Slack reminder: 2025-12-25

---

Related Documentation

• Service Principals Guide - Detailed setup and permissions

• Deployment Guide - CI/CD authentication

• Security Architecture - Security model overview

• Troubleshooting Guide - Authentication errors

• CI/CD Pipeline Architecture - GitHub OIDC details

---

Emergency Contacts

For credential rotation issues:

• Taylor Laing: [email protected] (Account Admin)

• #ml-pipelines: Slack channel for team support

• #security: Slack channel for security incidents

For security incidents:

• Immediately notify #security channel

• Follow incident response procedures

• Document all actions taken