ML Pipelines Overview

Welcome to the ML Pipelines documentation. This documentation is organized by audience to help you quickly find what you need.

Quick Start

New to the project? Start with the Project README for a quick overview, then proceed to the Developer Getting Started Guide.

Deploying to production? See the Operations Deployment Guide.

Understanding the architecture? Read the System Architecture document at the root level (ARCHITECTURE.md).

---

Documentation by Role

Developers

Start here: Getting Started Guide

Daily development workflows, coding standards, and troubleshooting:

• Getting Started - Onboarding checklist and first deployment

• Local Development - Sandbox workflow and iterative development

• Model Deployment - Model registration and ai_query integration

• DLT Pipelines - Delta Live Tables development guide

• Testing - Testing strategies and best practices

• Debugging - Common issues and solutions

• Code Standards - Coding conventions and PR process

Architects

Start here: System Architecture (root level)

System design, data flow, and architectural decisions:

• Data Flow - Bronze/Silver/Gold pipeline architecture

• Unity Catalog Design - Catalog structure and isolation strategy

• CI/CD Pipeline - Deployment automation and workflows

• Model Promotion - MLOps workflow from dev to production

• Security & Compliance - Security model and compliance framework

• Architecture Decision Records - ADRs documenting major decisions

Jobs and Orchestration

Start here: Jobs Overview

Orchestration, scheduling, and batch job documentation:

• Jobs Overview - Complete index of all Databricks Jobs

• Data Ingestion and Analysis Orchestration - Main orchestration workflow (7 tasks, 4 stages)

• Neon DB Replication - Reference data sync from PostgreSQL

DevOps/SRE

Start here: Deployment Guide

Deployment, infrastructure, and operational procedures:

• Deployment Guide - Step-by-step deployment to all environments

• Service Principals - Authentication configuration and management

• Infrastructure - Terraform and workspace management

• Monitoring - Observability and alerting setup

• Troubleshooting - Common issues and resolutions

• Runbooks - Operational procedures for incidents

• Disaster Recovery - Backup and recovery procedures

Executives

Start here: Platform Overview

High-level understanding of platform value and governance:

• Platform Overview - Business value and capabilities

• Compliance & Governance - Data governance and regulatory compliance

• Cost Optimization - Cost management and optimization

---

Reference Documentation

Quick reference for configurations, commands, and terminology:

• Configuration Reference - databricks.yml and pipeline YAML reference

• CLI Commands - Makefile and Databricks CLI usage

• Naming Conventions - Resource naming standards

• Glossary - Terminology and acronyms

---

Common Tasks

Local Development

1. Set up your sandbox environment

2. Deploy your first pipeline

3. Test changes locally

4. Create a pull request

Deploying to Production

1. Merge PR to deploy to dev

2. Validate in staging

3. Promote to production

4. Monitor production

Monitoring Orchestration

1. View orchestration job status

2. Check data freshness

3. Review task-level failures

4. Trigger manual run

Troubleshooting

1. Check common issues

2. Review runbooks

3. Check logs

4. Escalate if needed

---

Additional Resources

This Repository

• GitHub Workflows - CI/CD workflow documentation

• Contributing Guide - How to contribute to this project

• Changelog - Version history and changes

• Archived Documentation - Historical documents for context

Related Repositories

For complete system understanding, see documentation in related repositories:

• infra-core - Terraform infrastructure, VPC, networking, Databricks workspace setup

  • Path: /Users/taylorlaing/Development/refresh-os/infra-core/

  • Manages: VPC, subnets, security groups, service principals, Unity Catalog

• api-core - Backend REST API services

  • Path: /Users/taylorlaing/Development/refresh-os/api-core/

  • Consumes: ML pipeline outputs (sentiment scores, features, insights)

  • Provides: API endpoints for web application

• app-web - Frontend web application

  • Path: /Users/taylorlaing/Development/refresh-os/app-web/

  • Displays: Analytics dashboards, real-time insights from ML pipelines

---

Documentation Principles

This documentation follows these principles:

1. Audience-First: Organized by who needs the information

2. Task-Oriented: Focused on what you need to accomplish

3. Current: Updated to reflect the actual codebase

4. Cross-Referenced: Links between related documents

5. Searchable: Clear headings and consistent terminology

Need Help?

• Documentation Issues: Create an issue in the repository

• Technical Questions: Reach out to the development team

• Emergency: See Troubleshooting

---

Last updated: October 2025