Code Standards and PR Process

Overview

This guide defines coding conventions, standards, and the pull request process for the ML Pipelines project. Consistent code standards improve readability, maintainability, and collaboration.

Python Style Guide

PEP 8 Compliance

All Python code must follow PEP 8 style guidelines.

Key Principles:

• Indentation: 4 spaces (no tabs)

• Line length: 100 characters maximum (flexible to 120 for readability)

• Imports: Grouped and sorted (standard library, third-party, local)

• Naming: See naming conventions below

• Whitespace: Consistent spacing around operators and after commas

Enforcement:

# Format code with black
uv run black src/ tests/

# Check code style
uv run flake8 src/ tests/

# Sort imports
uv run isort src/ tests/

Naming Conventions

Variables and Functions

# Variables: snake_case
message_count = 10
user_email = "[email protected]"
sentiment_scores = [0.9, 0.8, 0.7]

# Functions: snake_case
def calculate_sentiment_score(text):
    """Calculate sentiment score from text."""
    pass

def preprocess_message(message_id, content):
    """Preprocess message content."""
    pass

Classes

# Classes: PascalCase
class SentimentAnalysisModel:
    """Sentiment analysis model implementation."""
    pass

class MessageProcessor:
    """Process and transform messages."""
    pass

Constants

# Constants: UPPER_SNAKE_CASE
MAX_MESSAGE_LENGTH = 10000
DEFAULT_BATCH_SIZE = 100
SUPPORTED_MODELS = ["sentiment", "emotion", "linguistic"]

Private Methods/Variables

class ModelTrainer:
    def __init__(self):
        self._internal_state = {}  # Private variable

    def _preprocess_data(self, data):
        """Private method (internal use only)."""
        pass

    def train(self, data):
        """Public method."""
        return self._preprocess_data(data)

Docstrings

Use Google-style docstrings for all public functions and classes.

Function Docstrings:

def calculate_sentiment_score(text: str, model_name: str = "default") -> float:
    """Calculate sentiment score using specified model.

    Args:
        text: Input text to analyze
        model_name: Name of sentiment model to use (default: "default")

    Returns:
        Sentiment score between 0.0 and 1.0

    Raises:
        ValueError: If text is empty or None
        ModelNotFoundError: If model_name doesn't exist

    Example:
        >>> calculate_sentiment_score("I love this product!", "roberta")
        0.95
    """
    if not text:
        raise ValueError("Text cannot be empty")
    # Implementation...

Class Docstrings:

class SentimentAnalysisModel:
    """Sentiment analysis model using RoBERTa and affect mapping.

    This model combines RoBERTa Go Emotions predictions with Plutchik's
    wheel of emotions to provide comprehensive sentiment analysis.

    Attributes:
        catalog_name: Unity Catalog name for model artifacts
        go_emotions_model: RoBERTa model for emotion classification
        affect_mapping: Configuration for emotion-to-affect mapping

    Example:
        >>> model = SentimentAnalysisModel("prod")
        >>> model.predict("I'm grateful for your help!")
        [{'sentiment': 'positive', 'score': 0.95, ...}]
    """

    def __init__(self, catalog_name: str):
        """Initialize sentiment analysis model.

        Args:
            catalog_name: Unity Catalog name (e.g., 'dev', 'prod')
        """
        self.catalog_name = catalog_name

Type Hints

Use type hints for function signatures and class attributes.

from typing import List, Dict, Optional, Union
from pyspark.sql import DataFrame

def process_messages(
    df: DataFrame,
    filter_by: Optional[str] = None,
    limit: int = 1000
) -> DataFrame:
    """Process messages DataFrame."""
    pass

def get_model_config(model_name: str) -> Dict[str, Union[str, int, float]]:
    """Get model configuration dictionary."""
    pass

class MessageTransformer:
    """Transform message data."""

    def __init__(self, columns: List[str]) -> None:
        self.columns = columns

    def transform(self, data: DataFrame) -> DataFrame:
        """Transform DataFrame."""
        pass

Import Organization

Import Order:

1. Standard library imports

2. Third-party imports

3. Local application imports

Example:

# Standard library
import os
import sys
from typing import List, Dict

# Third-party
import mlflow
import pandas as pd
from pyspark.sql import DataFrame
from pyspark.sql.functions import col, when

# Local
from models.internal.sentiment_analysis.model import SentimentAnalysisModel
from utils.data_validation import validate_schema
from config.table_schemas import MESSAGE_SCHEMA

Use isort to automatically organize imports:

uv run isort src/ tests/

File Organization

Directory Structure

ml-pipelines/
├── src/                           # Source code
│   ├── models/                    # Model implementations
│   │   ├── internal/             # Custom models
│   │   │   └── [model_name]/
│   │   │       ├── __init__.py
│   │   │       ├── model.py      # Model class
│   │   │       ├── training.py   # Training logic
│   │   │       └── registration.py # MLflow registration
│   │   └── external/             # External model wrappers
│   ├── pipelines/                # Pipeline logic (if used)
│   ├── table_schemas/            # Schema definitions
│   ├── utils/                    # Shared utilities
│   └── config/                   # Configuration files
├── resources/                     # Databricks resource definitions
│   ├── jobs/                     # Job YAML files
│   ├── pipelines/                # Pipeline YAML files
│   └── volumes/                  # Volume YAML files
├── tests/                        # Test code
│   ├── unit/                     # Unit tests
│   ├── integration/              # Integration tests
│   └── fixtures/                 # Test data and fixtures
└── docs/                         # Documentation

File Naming

• Python files: snake_case.py

  • sentiment_analysis.py

  • text_processing.py

  • model_registration.py

• YAML files: kebab-case.yml

  • register-sentiment-analysis.job.yml

  • bronze-ingestion.pipeline.yml

• Notebooks: snake_case.ipynb or snake_case.py (for Databricks)

  • data_exploration.ipynb

  • model_training.py

• Test files: test_[module_name].py

  • test_sentiment_model.py

  • test_data_validation.py

Code Documentation

Inline Comments

Use comments to explain why, not what.

Good Comments:

# Use two-stage pipeline to handle dynamic keys from model output
raw_result = ai_query('model', content)

# Retry on timeout because model endpoint may be scaling up
for attempt in range(max_retries):
    try:
        result = endpoint.predict(data)
        break
    except TimeoutError:
        time.sleep(2 ** attempt)

Bad Comments (obvious from code):

# Increment counter by 1
counter += 1

# Loop through messages
for message in messages:
    pass

TODOs and FIXMEs

Use standard markers for future work:

# TODO: Add support for multilingual text
# FIXME: Handle edge case when text contains only emojis
# NOTE: This is a temporary workaround until API v2 is released
# HACK: Override default timeout to prevent model serving issues

Search for TODOs:

grep -r "TODO" src/

PR Process

Creating a Pull Request

1. Create feature branch:

   git checkout -b ref-123-add-emotion-detection

2. Make changes and commit:

   git add src/models/internal/emotion_detection/
   git commit -m "Add emotion detection model
   
   - Implement EmotionDetectionModel class
   - Add model registration job
   - Update pipeline to use emotion detection
   - Add tests for emotion detection
   
   Resolves #123"

3. Push to GitHub:

   git push -u origin ref-123-add-emotion-detection

4. Open pull request:

   • Navigate to GitHub repository

   • Click "Compare & pull request"

   • Fill out PR template (see below)

PR Title Format

Always matches the ticket ID given by Linear, which prefixes tickets with ref-. When copying the branch name from the suggested named Linear gives, it will also include the title of the ticket converted into kebab-case.

• Example: A ticket is titled "Upgrade matplotlib package for security", the suggested branch name would look like ref-1234-upgrade-matplotlib-package-for-security

PR Description Template

## Summary
Brief description of what this PR does and why.

## Changes
- List of specific changes made
- Another change
- And another

## Testing
How was this tested?
- [ ] Unit tests added/updated
- [ ] Integration tests pass
- [ ] Tested in sandbox environment
- [ ] Tested with sample data

## Screenshots (if applicable)
Add screenshots for UI changes or pipeline results.

## Checklist
- [ ] Code follows style guidelines
- [ ] Documentation updated
- [ ] Tests added/updated
- [ ] All tests pass
- [ ] No new warnings or errors
- [ ] Bundle validation passes

## Related Issues
Closes #123
Related to #456

PR Review Process

Automatic Checks (CI/CD):

• Bundle validation (via GitHub Actions)

• Code style checks (if configured)

• Tests (if configured)

Manual Review:

• Code quality and readability

• Logic correctness

• Test coverage

• Documentation completeness

• Performance implications

• Security considerations

Review Checklist for Reviewers:

• Code is clear and well-documented

• Naming is consistent with conventions

• Changes are tested

• No hardcoded credentials or secrets

• Error handling is appropriate

• Performance implications considered

• Security implications considered

PR Approval Requirements

Dev Deployment:

• 1+ approval from team member

• All automated checks pass

• No unresolved comments

Staging/Prod Deployment:

• Lead developer approval required

• All tests pass

• Documentation updated

Git Workflow

Branch Strategy

main (default branch)
  ↓
  └─ ref-123-add-model-x     # New features

Commit Messages

Format:

Short summary (50 chars or less)

More detailed explanation if needed. Wrap at 72 characters.
Explain the problem this commit solves and how.

- Bullet points are okay
- Use present tense ("Add feature" not "Added feature")
- Reference issues: "Fixes #123" or "Related to #456"

Good Commit Messages:

Add two-stage pipeline pattern for sentiment analysis

The model returns dynamic keys that cause schema parse errors
in ai_query. Split into raw results stage and casting stage
to handle this properly.

Fixes #234

Bad Commit Messages:

fix bug
updated code
changes
wip

Keeping Branch Updated

# Update your branch with latest main
git checkout main
git pull origin main
git checkout ref-123-your-feature
git rebase main

# Or use merge if rebase is problematic
git merge main

Code Quality Tools

Recommended Tools

Linting and Formatting:

# Install dev dependencies
uv sync --dev

# Format code
uv run black src/ tests/

# Check code style
uv run flake8 src/ tests/

# Sort imports
uv run isort src/ tests/

# Type checking
uv run mypy src/

Pre-commit Hooks (recommended setup):

Create .pre-commit-config.yaml:

repos:
  - repo: https://github.com/psf/black
    rev: 24.1.1
    hooks:
      - id: black
        language_version: python3.13

  - repo: https://github.com/pycqa/flake8
    rev: 7.0.0
    hooks:
      - id: flake8
        args: ['--max-line-length=100', '--extend-ignore=E203,W503']

  - repo: https://github.com/pycqa/isort
    rev: 5.13.2
    hooks:
      - id: isort
        args: ['--profile', 'black']

Install pre-commit:

uv pip install pre-commit
pre-commit install

VS Code Settings

Recommended settings (.vscode/settings.json):

{
  "python.linting.enabled": true,
  "python.linting.flake8Enabled": true,
  "python.linting.flake8Args": [
    "--max-line-length=100",
    "--extend-ignore=E203,W503"
  ],
  "python.formatting.provider": "black",
  "python.formatting.blackArgs": [
    "--line-length=100"
  ],
  "editor.formatOnSave": true,
  "editor.codeActionsOnSave": {
    "source.organizeImports": true
  },
  "[python]": {
    "editor.defaultFormatter": "ms-python.black-formatter",
    "editor.formatOnSave": true,
    "editor.rulers": [100]
  }
}

Best Practices

Do's

Write self-documenting code: Use clear variable names and structure Add docstrings: Document all public functions and classes Use type hints: Make function signatures explicit Write tests: Test new features and bug fixes Keep functions small: Each function should do one thing well Handle errors gracefully: Use try/except with specific exceptions Log important events: Use logging instead of print statements Review your own code: Before requesting review, review your changes Update documentation: Keep docs in sync with code changes

Don'ts

Don't commit commented-out code: Delete it (git history preserves it) Don't use magic numbers: Define constants with meaningful names Don't ignore errors: Handle or propagate exceptions appropriately Don't hardcode values: Use configuration or environment variables Don't repeat code: Extract common logic into functions Don't commit secrets: Never commit API keys, passwords, tokens Don't skip tests: Untested code will break in production Don't merge without approval: Wait for code review Don't force push to main: Never use git push --force on main branch

Security Considerations

Secrets Management

Never commit:

• API keys

• Passwords

• Tokens

• Private keys

• Database credentials

Use instead:

• Databricks secrets

• Environment variables

• GitHub secrets (for CI/CD)

Check for secrets before commit:

# Search for potential secrets
grep -r "api_key" src/
grep -r "password" src/
grep -r "token" src/

Input Validation

Always validate user input:

def process_text(text: str) -> str:
    """Process text input with validation."""
    if not isinstance(text, str):
        raise TypeError("Input must be string")

    if not text.strip():
        raise ValueError("Input cannot be empty")

    if len(text) > MAX_TEXT_LENGTH:
        raise ValueError(f"Text exceeds max length {MAX_TEXT_LENGTH}")

    return text.strip()

Related Documentation

• Naming Conventions Reference - Complete naming standards

• Testing Guide - Testing requirements and strategies

• Contributing Guide - How to contribute to the project

• CI/CD Pipeline - Automated checks and deployment