- [[#Core Build Principles|Core Build Principles]]
- [[#Core Build Principles#Principle 1: Keep Build Fast|Principle 1: Keep Build Fast]]
- [[#Core Build Principles#Principle 2: Build Once, Deploy Anywhere|Principle 2: Build Once, Deploy Anywhere]]
- [[#Core Build Principles#Principle 3: Build as Code|Principle 3: Build as Code]]
- [[#Continuous Integration|Continuous Integration]]
- [[#Continuous Integration#CI Requirements|CI Requirements]]
- [[#Continuous Integration#MusicCorp CI Pipeline|MusicCorp CI Pipeline]]
- [[#Continuous Integration#CI Stages|CI Stages]]
- [[#Build Artifact Strategy|Build Artifact Strategy]]
- [[#Build Artifact Strategy#Primary Artifacts: Docker Images|Primary Artifacts: Docker Images]]
- [[#Build Artifact Strategy#Image Size Comparison|Image Size Comparison]]
- [[#Build Artifact Strategy#Artifact Storage|Artifact Storage]]
- [[#Source Code to Build to Service Mapping|Source Code to Build to Service Mapping]]
- [[#Source Code to Build to Service Mapping#Repository Structure|Repository Structure]]
- [[#Source Code to Build to Service Mapping#Build Dependency Graph|Build Dependency Graph]]
- [[#Source Code to Build to Service Mapping#Go Service (Independent)|Go Service (Independent)]]
- [[#Monorepo Strategy|Monorepo Strategy]]
- [[#Monorepo Strategy#MusicCorp Uses a Monorepo|MusicCorp Uses a Monorepo]]
- [[#Monorepo Strategy#Directory Conventions|Directory Conventions]]
- [[#Build Tooling|Build Tooling]]
- [[#Build Tooling#Python: uv|Python: uv]]
- [[#Build Tooling#Go: Standard Toolchain|Go: Standard Toolchain]]
- [[#Build Tooling#Make: Task Runner|Make: Task Runner]]
- [[#Pre-Commit Hooks|Pre-Commit Hooks]]
- [[#Pre-Commit Hooks#Configuration|Configuration]]
- [[#Pre-Commit Hooks#Setup|Setup]]
- [[#Pre-Commit Hooks#Hook Execution Flow|Hook Execution Flow]]
- [[#Docker Build Strategy|Docker Build Strategy]]
- [[#Docker Build Strategy#Multi-Stage Builds|Multi-Stage Builds]]
- [[#Docker Build Strategy#Go Service: Scratch Base|Go Service: Scratch Base]]
- [[#Docker Build Strategy#Layer Caching Strategy|Layer Caching Strategy]]
- [[#CI/CD Pipeline|CI/CD Pipeline]]
- [[#CI/CD Pipeline#GitHub Actions Workflow|GitHub Actions Workflow]]
- [[#CI/CD Pipeline#Pipeline Stages Explained|Pipeline Stages Explained]]
- [[#CI/CD Pipeline#Branch Strategy|Branch Strategy]]
- [[#Versioning Strategy|Versioning Strategy]]
- [[#Versioning Strategy#Current Approach: Static Version|Current Approach: Static Version]]
- [[#Versioning Strategy#Recommended: Git-Based Versioning|Recommended: Git-Based Versioning]]
- [[#Versioning Strategy#Semantic Versioning|Semantic Versioning]]
- [[#Versioning Strategy#Image Tagging Strategy|Image Tagging Strategy]]
- [[#Summary: Chapter 7 Principles in Practice|Summary: Chapter 7 Principles in Practice]]
- [[#Gaps and Recommendations|Gaps and Recommendations]]
- [[#Gaps and Recommendations#Currently Missing|Currently Missing]]
- [[#Gaps and Recommendations#Recommended Additions|Recommended Additions]]
- [[#Related Documentation|Related Documentation]]
## Core Build Principles
Newman emphasizes these key build principles:
### Principle 1: Keep Build Fast
Fast builds encourage frequent commits and quick feedback loops.
**In MusicCorp:**
- **uv** for Python package management (~10x faster than pip)
- **Multi-stage Docker builds** to cache dependencies
- **Parallel image loading** into Kind cluster
- **Pre-commit hooks** run only on changed files
```makefile
# Parallel image loading (from Makefile)
k8s-load:
kind load docker-image $(CATALOG_IMAGE) --name $(KIND_CLUSTER) & \
kind load docker-image $(INVENTORY_IMAGE) --name $(KIND_CLUSTER) & \
# ... all in parallel
wait;
```
### Principle 2: Build Once, Deploy Anywhere
Build artifacts should be immutable and promotable through environments.
**In MusicCorp:**
- Docker images are the deployment artifact
- Same image used in dev (Kind) and would be used in prod (ECS/EKS)
- Version tags: `localhost/musiccorp/order:0.0.1`
- No environment-specific build logic
### Principle 3: Build as Code
Build configuration should be version-controlled alongside application code.
**In MusicCorp:**
| Build Artifact | Configuration |
|----------------|---------------|
| Python deps | `pyproject.toml`, `uv.lock` |
| Go deps | `go.mod`, `go.sum` |
| Docker images | `zarf/docker/dockerfile.*` |
| K8s manifests | `zarf/k8s/base/`, `zarf/k8s/dev/` |
| Pre-commit | `.pre-commit-config.yaml` |
| CI pipeline | `.github/workflows/ci.yaml` |
---
## Continuous Integration
Newman defines CI as the practice of merging all developer working copies to a shared mainline frequently.
### CI Requirements
1. **Automated Build**: Triggered on every push
2. **Fast Feedback**: Know within minutes if build is broken
3. **Keep Build Green**: Broken builds are top priority to fix
4. **Trunk-Based Development**: Short-lived branches, frequent merges
### MusicCorp CI Pipeline
```yaml
# .github/workflows/ci.yaml
name: CI
on:
push:
branches: [main]
pull_request:
branches: [main]
jobs:
lint:
# Ruff linting + formatting
# Type checking with ty
test:
# Python unit tests
# Go unit tests
build:
# Docker image builds
# Verify images are buildable
```
### CI Stages
```
┌─────────────────────────────────────────────────────────────────────────┐
│ CI Pipeline │
├─────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ Lint │ │ Test │ │ Build │ │
│ │ │ │ │ │ │ │
│ │ - ruff │ │ - pytest │ │ - Docker │ │
│ │ - ruff-fmt │───▶│ - go test │───▶│ images │ │
│ │ - ty │ │ │ │ │ │
│ │ - yaml │ │ │ │ │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ [Fast: ~30s] [Medium: ~2m] [Slow: ~5m] │
│ │
└─────────────────────────────────────────────────────────────────────────┘
```
---
## Build Artifact Strategy
### Primary Artifacts: Docker Images
Each service produces a Docker image as its build artifact:
| Service | Image Name | Base |
|---------|------------|------|
| catalog | `musiccorp/catalog:x.x.x` | python:3.13-slim |
| inventory | `musiccorp/inventory:x.x.x` | python:3.13-slim |
| order | `musiccorp/order:x.x.x` | python:3.13-slim |
| order-go | `musiccorp/order-go:x.x.x` | scratch |
| payment | `musiccorp/payment:x.x.x` | python:3.13-slim |
| shipping | `musiccorp/shipping:x.x.x` | python:3.13-slim |
### Image Size Comparison
```
Python services (with dependencies): ~200MB
Go service (static binary): ~23MB
```
The Go service demonstrates the size advantage of compiled languages for microservices.
### Artifact Storage
| Environment | Registry |
|-------------|----------|
| Local (Kind) | `localhost/musiccorp/*` |
| Production | ECR: `<account>.dkr.ecr.<region>.amazonaws.com/musiccorp/*` |
---
## Source Code to Build to Service Mapping
Newman emphasizes clear mapping between code, builds, and running services.
### Repository Structure
```
building-microservices/ # Monorepo root
├── services/
│ ├── catalog/ # → musiccorp/catalog image
│ │ ├── app.py
│ │ ├── models.py
│ │ └── __init__.py
│ ├── inventory/ # → musiccorp/inventory image
│ ├── order/ # → musiccorp/order image
│ ├── order-go/ # → musiccorp/order-go image
│ │ ├── cmd/order/main.go
│ │ ├── internal/
│ │ ├── go.mod
│ │ └── Dockerfile
│ ├── payment/ # → musiccorp/payment image
│ ├── shipping/ # → musiccorp/shipping image
│ └── shared/ # Shared Python utilities (not an image)
├── shared/ # Cross-service shared code
├── zarf/
│ ├── docker/ # Dockerfiles for Python services
│ │ ├── dockerfile.catalog
│ │ ├── dockerfile.inventory
│ │ ├── dockerfile.order
│ │ ├── dockerfile.payment
│ │ └── dockerfile.shipping
│ └── k8s/ # Kubernetes manifests
└── pyproject.toml # Shared Python dependencies
```
### Build Dependency Graph
```
pyproject.toml
uv.lock
│
┌───────────────┼───────────────┐
▼ ▼ ▼
services/shared shared/events (common deps)
│ │
┌────┴────┬─────┬────┴────┬─────┐
▼ ▼ ▼ ▼ ▼
catalog inventory order payment shipping
│ │ │ │ │
▼ ▼ ▼ ▼ ▼
dockerfile dockerfile ... ... ...
│ │
▼ ▼
image image ...
```
### Go Service (Independent)
```
services/order-go/
├── go.mod # Own dependency management
├── go.sum
├── Dockerfile # Self-contained build
└── internal/ # No shared code dependencies
```
---
## Monorepo Strategy
Newman discusses the tradeoffs between monorepos and multi-repos.
### MusicCorp Uses a Monorepo
**Advantages realized:**
1. **Atomic Changes**: Update multiple services in one commit
2. **Shared Code**: `services/shared/` and `shared/` directories
3. **Consistent Tooling**: Single `pyproject.toml`, `.pre-commit-config.yaml`
4. **Simplified CI**: One pipeline for all services
5. **Easy Refactoring**: Move code between services easily
**Challenges addressed:**
| Challenge | Solution |
|-----------|----------|
| Build all on any change | Per-service Docker builds, selective CI |
| Coupling temptation | Clear service boundaries, database-per-service |
| Large repo size | Not an issue at current scale |
### Directory Conventions
```
services/ # Deployable services (each becomes a container)
├── catalog/ # Independent bounded context
├── order/ # Can import from shared/, services/shared/
└── shared/ # Service-level shared utilities (logging, db, metrics)
shared/ # True shared libraries
└── events.py # Kafka event utilities
zarf/ # "Deployment cruft" - infrastructure configs
├── docker/ # Dockerfiles
└── k8s/ # Kubernetes manifests
scripts/ # Developer utilities, not deployed
docs/ # Documentation
```
---
## Build Tooling
### Python: uv
**Why uv over pip/poetry:**
- 10-100x faster dependency resolution
- Lockfile for reproducible builds (`uv.lock`)
- Built-in Python version management
- Drop-in pip compatibility
**Usage in Docker:**
```dockerfile
# Copy uv binary from official image
COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv
# Install dependencies (frozen = use lockfile exactly)
RUN uv sync --frozen --no-dev
```
**Local development:**
```bash
uv sync # Install all deps including dev
uv run pytest # Run command in virtual env
uv add requests # Add dependency
uv lock # Update lockfile
```
### Go: Standard Toolchain
```bash
go mod download # Download dependencies
go build ./... # Build all packages
go test ./... # Run all tests
```
### Make: Task Runner
The `Makefile` provides a consistent interface for all build tasks:
```bash
make build # Build Docker images (compose)
make k8s-build # Build Docker images (k8s)
make test # Run tests
make k8s-run # Full cluster setup
```
---
## Pre-Commit Hooks
Pre-commit hooks catch issues before they enter version control.
### Configuration
```yaml
# .pre-commit-config.yaml
repos:
# Ruff - fast Python linter and formatter
- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.8.6
hooks:
- id: ruff
args: [--fix] # Auto-fix where possible
- id: ruff-format # Format code
# Ty - type checker
- repo: local
hooks:
- id: ty
name: ty
entry: uv run ty check
language: system
types: [python]
files: ^services/
pass_filenames: false
# General file quality
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v5.0.0
hooks:
- id: trailing-whitespace
- id: end-of-file-fixer
- id: check-yaml
- id: check-added-large-files
- id: check-merge-conflict
- id: debug-statements
```
### Setup
```bash
# Install pre-commit
uv sync # Installs pre-commit as dev dependency
# Install hooks
uv run pre-commit install
# Run manually on all files
uv run pre-commit run --all-files
```
### Hook Execution Flow
```
git commit
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ Pre-commit Hooks │
├─────────────────────────────────────────────────────────────────┤
│ │
│ 1. trailing-whitespace ──▶ Fix whitespace issues │
│ 2. end-of-file-fixer ──▶ Ensure files end with newline │
│ 3. check-yaml ──▶ Validate YAML syntax │
│ 4. check-merge-conflict ──▶ Catch unresolved conflicts │
│ 5. debug-statements ──▶ Catch print/debugger statements │
│ 6. ruff ──▶ Lint Python code │
│ 7. ruff-format ──▶ Format Python code │
│ 8. ty ──▶ Type check Python code │
│ │
│ Any failure ──▶ Commit blocked │
│ All pass ──▶ Commit proceeds │
│ │
└─────────────────────────────────────────────────────────────────┘
```
---
## Docker Build Strategy
### Multi-Stage Builds
All Python services use multi-stage Docker builds:
```dockerfile
# Stage 1: Build
FROM python:3.13-slim AS build
COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv
COPY pyproject.toml uv.lock ./
RUN uv sync --frozen --no-dev # Install deps (cached layer)
COPY services/order/ ./services/order/
# Stage 2: Production
FROM python:3.13-slim AS production
COPY --from=build /app/.venv /app/.venv
COPY --from=build /app/services /app/services
CMD ["python", "-m", "services.order.app"]
```
**Benefits:**
1. **Smaller images**: Only runtime artifacts in final image
2. **Better caching**: Dependencies layer cached separately
3. **Security**: Build tools not in production image
### Go Service: Scratch Base
```dockerfile
FROM golang:1.24-alpine AS builder
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 \
go build -ldflags="-w -s" -o /order ./cmd/order
FROM scratch
COPY --from=builder /order /order
ENTRYPOINT ["/order"]
```
**Result:** 23MB image containing only the static binary.
### Layer Caching Strategy
```dockerfile
# Layer 1: Base image (cached across builds)
FROM python:3.13-slim
# Layer 2: Dependencies (cached if pyproject.toml unchanged)
COPY pyproject.toml uv.lock ./
RUN uv sync --frozen --no-dev
# Layer 3: Application code (rebuilt on code changes)
COPY services/order/ ./services/order/
```
**Optimal change frequency:**
```
Base image ──▶ Rarely changes (months)
Dependencies ──▶ Sometimes (weekly)
Application ──▶ Frequently (every commit)
```
---
## CI/CD Pipeline
### GitHub Actions Workflow
```yaml
# .github/workflows/ci.yaml
name: CI
on:
push:
branches: [main]
pull_request:
branches: [main]
jobs:
lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Install uv
uses: astral-sh/setup-uv@v5
- name: Install dependencies
run: uv sync
- name: Run ruff linter
run: uv run ruff check .
- name: Run ruff formatter
run: uv run ruff format --check .
- name: Run type checker
run: uv run ty check services/
test-python:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Install uv
uses: astral-sh/setup-uv@v5
- name: Install dependencies
run: uv sync
- name: Run Python tests
run: uv run pytest -v
test-go:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Setup Go
uses: actions/setup-go@v5
with:
go-version: '1.24'
- name: Run Go tests
working-directory: services/order-go
run: go test -v ./...
build:
runs-on: ubuntu-latest
needs: [lint, test-python, test-go]
steps:
- uses: actions/checkout@v4
- name: Build catalog image
run: docker build -f zarf/docker/dockerfile.catalog -t catalog:ci .
- name: Build inventory image
run: docker build -f zarf/docker/dockerfile.inventory -t inventory:ci .
- name: Build order image
run: docker build -f zarf/docker/dockerfile.order -t order:ci .
- name: Build order-go image
run: docker build -f services/order-go/Dockerfile -t order-go:ci services/order-go
- name: Build payment image
run: docker build -f zarf/docker/dockerfile.payment -t payment:ci .
- name: Build shipping image
run: docker build -f zarf/docker/dockerfile.shipping -t shipping:ci .
```
### Pipeline Stages Explained
| Stage | Purpose | Duration | Blocking |
|-------|---------|----------|----------|
| **lint** | Code quality checks | ~30s | Yes |
| **test-python** | Python unit tests | ~1-2m | Yes |
| **test-go** | Go unit tests | ~30s | Yes |
| **build** | Docker image builds | ~5m | Yes |
### Branch Strategy
```
main (protected)
│
├── feature/add-auth ───▶ PR ───▶ CI ───▶ Merge
│
└── fix/order-bug ──────▶ PR ───▶ CI ───▶ Merge
```
- `main` is protected: requires passing CI
- Feature branches are short-lived
- PRs require CI green before merge
---
## Versioning Strategy
### Current Approach: Static Version
```makefile
VERSION := 0.0.1
CATALOG_IMAGE := $(BASE_IMAGE_NAME)/catalog:$(VERSION)
```
### Recommended: Git-Based Versioning
```makefile
VERSION := $(shell git describe --tags --always --dirty)
```
This produces versions like:
- `v1.0.0` - on a tag
- `v1.0.0-5-g1234abc` - 5 commits after tag
- `v1.0.0-5-g1234abc-dirty` - with uncommitted changes
### Semantic Versioning
For production, follow semver:
| Version | Meaning |
|---------|---------|
| `1.0.0` | Initial stable release |
| `1.0.1` | Patch: bug fixes |
| `1.1.0` | Minor: new features, backward compatible |
| `2.0.0` | Major: breaking changes |
### Image Tagging Strategy
```
# Development
localhost/musiccorp/order:0.0.1
localhost/musiccorp/order:latest
# CI builds
musiccorp/order:sha-a1b2c3d
# Production releases
musiccorp/order:v1.2.3
musiccorp/order:v1.2
musiccorp/order:v1
```
---
## Summary: Chapter 7 Principles in Practice
| Newman's Principle | MusicCorp Implementation |
|-------------------|--------------------------|
| **Keep build fast** | uv, multi-stage Docker, parallel loading |
| **Build once, deploy anywhere** | Docker images as artifacts |
| **Build as code** | Makefile, Dockerfiles, CI config in repo |
| **Continuous Integration** | GitHub Actions on every push/PR |
| **Trunk-based development** | Short-lived feature branches to main |
| **Immutable artifacts** | Versioned Docker images |
| **Monorepo** | Single repo with clear service boundaries |
| **Pre-commit validation** | Ruff, ty, YAML checks before commit |
---
## Gaps and Recommendations
### Currently Missing
1. **Python Unit Tests**: Only saga compensation test exists
2. **Integration Tests**: No automated E2E tests
3. **CD Pipeline**: No automated deployment to staging/prod
4. **Image Scanning**: No vulnerability scanning
5. **Artifact Registry**: No push to ECR/Docker Hub
### Recommended Additions
```yaml
# Future CI additions
- name: Trivy vulnerability scan
uses: aquasecurity/trivy-action@master
- name: Push to ECR
if: github.ref == 'refs/heads/main'
run: |
aws ecr get-login-password | docker login ...
docker push $ECR_REPO/order:$VERSION
```
---
## Related Documentation
- [08-deployment.md](08%20-%20k8s%20Deployment.md) - Deployment concepts (Chapter 8)
- [k8s-commands.md](k8s-commands.md) - Kubernetes command reference
- [event-schemas.md](event-schemas.md) - Kafka event documentation