- [[#Core Deployment Principles|Core Deployment Principles]]
- [[#Core Deployment Principles#Principle 1: Isolated Execution|Principle 1: Isolated Execution]]
- [[#Core Deployment Principles#Principle 2: Desired State Declaration|Principle 2: Desired State Declaration]]
- [[#Core Deployment Principles#Principle 3: Immutable Infrastructure|Principle 3: Immutable Infrastructure]]
- [[#Kubernetes Architecture Overview|Kubernetes Architecture Overview]]
- [[#Kubernetes Architecture Overview#Key Kubernetes Objects in MusicCorp|Key Kubernetes Objects in MusicCorp]]
- [[#Workload Types: Deployments vs StatefulSets|Workload Types: Deployments vs StatefulSets]]
- [[#Workload Types: Deployments vs StatefulSets#Deployments (Stateless Services)|Deployments (Stateless Services)]]
- [[#Workload Types: Deployments vs StatefulSets#StatefulSets (Stateful Infrastructure)|StatefulSets (Stateful Infrastructure)]]
- [[#Service Discovery and Networking|Service Discovery and Networking]]
- [[#Service Discovery and Networking#ClusterIP Services (Internal)|ClusterIP Services (Internal)]]
- [[#Service Discovery and Networking#Service-to-Service Communication|Service-to-Service Communication]]
- [[#Configuration Management|Configuration Management]]
- [[#Configuration Management#Environment Variables|Environment Variables]]
- [[#Configuration Management#Secrets|Secrets]]
- [[#Configuration Management#ConfigMaps|ConfigMaps]]
- [[#Health Checks and Self-Healing|Health Checks and Self-Healing]]
- [[#Health Checks and Self-Healing#Readiness Probes|Readiness Probes]]
- [[#Health Checks and Self-Healing#Liveness Probes|Liveness Probes]]
- [[#Health Checks and Self-Healing#PostgreSQL Health Checks|PostgreSQL Health Checks]]
- [[#Health Checks and Self-Healing#Graceful Shutdown|Graceful Shutdown]]
- [[#API Gateway Pattern|API Gateway Pattern]]
- [[#API Gateway Pattern#Nginx Ingress Controller|Nginx Ingress Controller]]
- [[#API Gateway Pattern#Ingress Resource Configuration|Ingress Resource Configuration]]
- [[#API Gateway Pattern#Path Rewriting|Path Rewriting]]
- [[#API Gateway Pattern#Benefits of API Gateway|Benefits of API Gateway]]
- [[#Database Per Service|Database Per Service]]
- [[#Database Per Service#Implementation in MusicCorp|Implementation in MusicCorp]]
- [[#Database Per Service#Database Initialization|Database Initialization]]
- [[#Database Per Service#Isolation Enforcement|Isolation Enforcement]]
- [[#Event-Driven Infrastructure|Event-Driven Infrastructure]]
- [[#Event-Driven Infrastructure#Kafka Configuration|Kafka Configuration]]
- [[#Event-Driven Infrastructure#Event Topics|Event Topics]]
- [[#Event-Driven Infrastructure#Event Flow|Event Flow]]
- [[#Observability Stack|Observability Stack]]
- [[#Observability Stack#Components|Components]]
- [[#Observability Stack#Metrics Collection|Metrics Collection]]
- [[#Observability Stack#Distributed Tracing|Distributed Tracing]]
- [[#Observability Stack#Correlation ID Propagation|Correlation ID Propagation]]
- [[#Local Development with Kind|Local Development with Kind]]
- [[#Local Development with Kind#Cluster Configuration|Cluster Configuration]]
- [[#Local Development with Kind#Port Mapping Strategy|Port Mapping Strategy]]
- [[#Manifest Organization with Kustomize|Manifest Organization with Kustomize]]
- [[#Manifest Organization with Kustomize#Directory Structure|Directory Structure]]
- [[#Manifest Organization with Kustomize#Base Kustomization|Base Kustomization]]
- [[#Manifest Organization with Kustomize#Dev Overlay|Dev Overlay]]
- [[#Manifest Organization with Kustomize#Benefits of Kustomize|Benefits of Kustomize]]
- [[#Manifest Organization with Kustomize#Common Operations|Common Operations]]
- [[#Summary: Chapter 8 Principles in Practice|Summary: Chapter 8 Principles in Practice]]
- [[#Related Documentation|Related Documentation]]
## Core Deployment Principles
Newman emphasizes several key deployment principles for microservices:
### Principle 1: Isolated Execution
Each microservice should run in isolation, with its own:
- Process space
- Network interface
- Dependencies
- Configuration
**In MusicCorp**: Each service runs as a separate Kubernetes Deployment with its own container, environment variables, and resource limits.
```yaml
# From zarf/k8s/base/order/base-order.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: order
spec:
template:
spec:
containers:
- name: order
image: order-image
env:
- name: SERVICE_NAME
value: "order"
- name: DATABASE_URL
valueFrom:
secretKeyRef:
name: postgres-secret
key: ORDER_DATABASE_URL
```
### Principle 2: Desired State Declaration
Kubernetes uses declarative configuration - you describe the desired state, and Kubernetes continuously works to maintain it.
**In MusicCorp**: All infrastructure is defined in YAML manifests under `zarf/k8s/`. The cluster continuously reconciles actual state with declared state.
### Principle 3: Immutable Infrastructure
Containers are immutable. To update, you replace the entire container rather than modifying it in place.
**In MusicCorp**: Updates are deployed by building new images and rolling out new pods:
```bash
make k8s-update # Builds new images, loads into Kind, restarts deployments
```
---
## Kubernetes Architecture Overview
```
┌─────────────────────────────────────────────────────────────┐
│ Kind Cluster │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ Control Plane Node │ │
│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────┐ │ │
│ │ │ API Server │ │ Scheduler │ │ Controller Manager │ │ │
│ │ └─────────────┘ └─────────────┘ └─────────────────────┘ │ │
│ │ ┌─────────────────────────────────────────────────────┐ │ │
│ │ │ etcd │ │ │
│ │ └─────────────────────────────────────────────────────┘ │ │
│ └─────────────────────────────────────────────────────────┘ │
│ │ │
│ ┌───────────────────────────┴───────────────────────────┐ │
│ │ Worker Processes │ │
│ │ │ │
│ │ ┌──────────────────────────────────────────────────┐ │ │
│ │ │ musiccorp-system namespace │ │ │
│ │ │ │ │ │
│ │ │ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ │ │
│ │ │ │ catalog │ │inventory│ │ order │ │ payment │ │ │ │
│ │ │ │ :5001 │ │ :5002 │ │ :5003 │ │ :5004 │ │ │ │
│ │ │ └─────────┘ └─────────┘ └─────────┘ └─────────┘ │ │ │
│ │ │ │ │ │
│ │ │ ┌─────────┐ ┌─────────────┐ ┌─────────────────┐ │ │ │
│ │ │ │shipping │ │ postgres │ │ kafka │ │ │ │
│ │ │ │ :5005 │ │ :5432 │ │ :9092 │ │ │ │
│ │ │ └─────────┘ └─────────────┘ └─────────────────┘ │ │ │
│ │ └──────────────────────────────────────────────────┘ │ │
│ │ │ │
│ │ ┌──────────────────────────────────────────────────┐ │ │
│ │ │ ingress-nginx namespace │ │ │
│ │ │ ┌────────────────────────────────────────────┐ │ │ │
│ │ │ │ nginx Ingress Controller │ │ │ │
│ │ │ │ :80, :443 │ │ │ │
│ │ │ └────────────────────────────────────────────┘ │ │ │
│ │ └──────────────────────────────────────────────────┘ │ │
│ └────────────────────────────────────────────────────────┘ │
└───────────────────────────────────────────────────────────────┘
```
### Key Kubernetes Objects in MusicCorp
| Object | Purpose | MusicCorp Usage |
|--------|---------|-----------------|
| **Namespace** | Logical isolation | `musiccorp-system` for all services |
| **Deployment** | Stateless workloads | Application services (catalog, order, etc.) |
| **StatefulSet** | Stateful workloads | PostgreSQL, Kafka |
| **Service** | Internal DNS/load balancing | One per microservice |
| **Ingress** | External HTTP routing | API Gateway pattern |
| **ConfigMap** | Non-sensitive config | Database init scripts, Prometheus config |
| **Secret** | Sensitive config | Database credentials |
| **PersistentVolumeClaim** | Persistent storage | Database and Kafka data |
---
## Workload Types: Deployments vs StatefulSets
### Deployments (Stateless Services)
Deployments are used for stateless application services. Pods are interchangeable and can be scaled horizontally.
**Characteristics:**
- Pods are identical and interchangeable
- Random pod naming (e.g., `catalog-7d8f9c6b4d-xj2kl`)
- Parallel scaling (all pods can start simultaneously)
- No stable network identity
- Data stored externally (databases, caches)
**MusicCorp Services using Deployments:**
| Service | Port | Role |
|---------|------|------|
| catalog | 5001 | Product information (read-heavy) |
| inventory | 5002 | Stock management |
| order | 5003 | Order orchestration (saga coordinator) |
| order-go | 5006 | Go implementation of order service |
| payment | 5004 | Payment processing |
| shipping | 5005 | Fulfillment coordination |
**Example: Order Service Deployment**
```yaml
# zarf/k8s/base/order/base-order.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: order
spec:
selector:
matchLabels:
app: order
template:
metadata:
labels:
app: order
annotations:
prometheus.io/scrape: "true" # Enable metrics scraping
prometheus.io/port: "5003"
prometheus.io/path: "/metrics"
spec:
terminationGracePeriodSeconds: 30 # Graceful shutdown window
containers:
- name: order
image: order-image
ports:
- name: http
containerPort: 5003
```
### StatefulSets (Stateful Infrastructure)
StatefulSets are used for stateful infrastructure that requires:
- Stable network identities
- Persistent storage
- Ordered deployment/scaling
- Ordered termination
**MusicCorp Services using StatefulSets:**
| Service | Port | Why StatefulSet? |
|---------|------|------------------|
| postgres | 5432 | Persistent data, stable identity for replication |
| kafka | 9092 | Ordered broker IDs, persistent message logs |
**Example: PostgreSQL StatefulSet**
```yaml
# zarf/k8s/base/postgres/base-postgres.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: postgres
spec:
serviceName: postgres # Creates DNS: postgres-0.postgres.musiccorp-system
replicas: 1
selector:
matchLabels:
app: postgres
template:
spec:
containers:
- name: postgres
image: postgres:16-alpine
volumeMounts:
- name: postgres-data
mountPath: /var/lib/postgresql/data
volumeClaimTemplates: # Automatic PVC per pod
- metadata:
name: postgres-data
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 1Gi
```
**Key StatefulSet Behaviors:**
1. **Stable Pod Names**: Pods get predictable names like `postgres-0`, `kafka-0`
2. **Stable DNS**: Each pod gets a DNS entry: `<pod>.<service>.<namespace>.svc.cluster.local`
3. **Ordered Operations**: Pods are created/deleted in order (0, 1, 2... or 2, 1, 0)
4. **Persistent Storage**: PVCs are not deleted when pods are deleted
---
## Service Discovery and Networking
Newman emphasizes that microservices need reliable service discovery. Kubernetes provides this through DNS-based service discovery.
### ClusterIP Services (Internal)
Every microservice exposes a ClusterIP Service, creating a stable internal DNS entry.
```yaml
# zarf/k8s/base/catalog/base-catalog.yaml
apiVersion: v1
kind: Service
metadata:
name: catalog
spec:
selector:
app: catalog
ports:
- name: http
port: 5001
targetPort: 5001
```
**DNS Resolution:**
```
catalog → 10.96.x.x (ClusterIP)
catalog.musiccorp-system → 10.96.x.x
catalog.musiccorp-system.svc.cluster.local → 10.96.x.x
```
### Service-to-Service Communication
Services reference each other using DNS names:
```yaml
# From order service deployment
env:
- name: CATALOG_URL
value: "http://catalog:5001"
- name: INVENTORY_URL
value: "http://inventory:5002"
- name: KAFKA_BOOTSTRAP_SERVERS
value: "kafka:9092"
```
**Communication Flow:**
```
┌───────────────────────────────────────────────────────────────────┐
│ Order Service Pod │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ HTTP Client: GET http://catalog:5001/albums/CD001 │ │
│ └───────────────────────────────┬─────────────────────────────┘ │
└──────────────────────────────────┼────────────────────────────────┘
│
▼
┌──────────────────────────┐
│ kube-dns / CoreDNS │
│ catalog → 10.96.152.185 │
└──────────────────────────┘
│
▼
┌──────────────────────────┐
│ catalog Service │
│ ClusterIP: 10.96.x.x │
│ Port: 5001 │
└────────────┬─────────────┘
│ (load balanced)
┌────────────┴─────────────┐
▼ ▼
┌─────────────┐ ┌─────────────┐
│ catalog-abc │ │ catalog-xyz │
│ :5001 │ │ :5001 │
└─────────────┘ └─────────────┘
```
---
## Configuration Management
### Environment Variables
Services receive configuration through environment variables, following the 12-factor app methodology.
**Types of Configuration:**
| Type | Source | Example |
|------|--------|---------|
| Static | Hardcoded in manifest | `SERVICE_NAME: "order"` |
| External service URLs | Hardcoded | `CATALOG_URL: "http://catalog:5001"` |
| Sensitive data | Kubernetes Secret | `DATABASE_URL` from `postgres-secret` |
| Runtime config | ConfigMap | Prometheus scrape config |
**Example: Order Service Configuration**
```yaml
env:
# Static configuration
- name: SERVICE_NAME
value: "order"
- name: PYTHONUNBUFFERED
value: "1"
# Service discovery
- name: CATALOG_URL
value: "http://catalog:5001"
- name: INVENTORY_URL
value: "http://inventory:5002"
- name: KAFKA_BOOTSTRAP_SERVERS
value: "kafka:9092"
# Sensitive - from Secret
- name: DATABASE_URL
valueFrom:
secretKeyRef:
name: postgres-secret
key: ORDER_DATABASE_URL
# Observability
- name: OTEL_EXPORTER_OTLP_ENDPOINT
value: "http://tempo:4317"
```
### Secrets
Database credentials are stored in a Kubernetes Secret:
```yaml
# zarf/k8s/base/postgres/base-postgres.yaml
apiVersion: v1
kind: Secret
metadata:
name: postgres-secret
type: Opaque
stringData:
POSTGRES_PASSWORD: "musiccorp_admin"
CATALOG_DATABASE_URL: "postgresql://catalog_user:catalog_pass@postgres:5432/musiccorp_catalog"
INVENTORY_DATABASE_URL: "postgresql://inventory_user:inventory_pass@postgres:5432/musiccorp_inventory"
ORDER_DATABASE_URL: "postgresql://order_user:order_pass@postgres:5432/musiccorp_order"
PAYMENT_DATABASE_URL: "postgresql://payment_user:payment_pass@postgres:5432/musiccorp_payment"
SHIPPING_DATABASE_URL: "postgresql://shipping_user:shipping_pass@postgres:5432/musiccorp_shipping"
```
**Security Notes:**
- In production, use external secret management (Vault, AWS Secrets Manager)
- Secrets are base64-encoded, not encrypted (encrypt etcd at rest)
- Use RBAC to restrict secret access
### ConfigMaps
Non-sensitive configuration like init scripts:
```yaml
# Generated by kustomization.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: postgres-init
data:
init-databases.sql: |
CREATE USER catalog_user WITH PASSWORD 'catalog_pass';
CREATE DATABASE musiccorp_catalog OWNER catalog_user;
-- ... more initialization
```
---
## Health Checks and Self-Healing
Newman emphasizes the importance of health checks for maintaining system reliability. Kubernetes implements this through probes.
### Readiness Probes
**Purpose:** Determine if a pod should receive traffic.
A pod is only added to Service endpoints when readiness probe succeeds.
```yaml
readinessProbe:
httpGet:
path: /health
port: 5003
initialDelaySeconds: 5 # Wait 5s before first check
periodSeconds: 10 # Check every 10s
timeoutSeconds: 5 # Fail if response takes >5s
successThreshold: 1 # 1 success = ready
failureThreshold: 2 # 2 failures = not ready
```
**Use Cases:**
- Waiting for database connection pool to initialize
- Warming caches
- Loading ML models
- Completing startup migrations
### Liveness Probes
**Purpose:** Determine if a pod should be restarted.
If liveness probe fails, Kubernetes kills and restarts the container.
```yaml
livenessProbe:
httpGet:
path: /health
port: 5003
initialDelaySeconds: 10 # Wait 10s before first check
periodSeconds: 10 # Check every 10s
timeoutSeconds: 5 # Fail if response takes >5s
successThreshold: 1 # 1 success = alive
failureThreshold: 3 # 3 failures = restart
```
**Use Cases:**
- Detecting deadlocks
- Detecting memory corruption
- Recovering from unrecoverable states
### PostgreSQL Health Checks
Databases use exec probes instead of HTTP:
```yaml
readinessProbe:
exec:
command: ["pg_isready", "-U", "postgres"]
initialDelaySeconds: 5
periodSeconds: 10
livenessProbe:
exec:
command: ["pg_isready", "-U", "postgres"]
initialDelaySeconds: 15
periodSeconds: 10
```
### Graceful Shutdown
The `terminationGracePeriodSeconds` setting allows pods to complete in-flight requests:
```yaml
spec:
terminationGracePeriodSeconds: 30
```
**Shutdown Sequence:**
1. Pod receives SIGTERM
2. Pod removed from Service endpoints (no new traffic)
3. Application handles SIGTERM (complete requests, close connections)
4. After 30s, SIGKILL if still running
---
## API Gateway Pattern
Newman describes the API Gateway as a single entry point that routes requests to appropriate services.
### Nginx Ingress Controller
MusicCorp uses nginx Ingress Controller to implement the API Gateway:
```
┌─────────────────────────┐
:80, :443 │ nginx Ingress │
────────────────▶│ Controller │
│ (ingress-nginx ns) │
└───────────┬────────────┘
│
┌────────────────────────┼─────────────────────────┐
│ │ │
▼ ▼ ▼
/api/catalog/* /api/orders/* /api/inventory/*
│ │ │
▼ ▼ ▼
┌───────────────┐ ┌───────────────┐ ┌───────────────┐
│ catalog:5001 │ │ order:5003 │ │ inventory:5002│
└───────────────┘ └───────────────┘ └───────────────┘
```
### Ingress Resource Configuration
```yaml
# zarf/k8s/base/ingress/base-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: musiccorp-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /$2
nginx.ingress.kubernetes.io/use-regex: "true"
spec:
ingressClassName: nginx
rules:
- http:
paths:
# /api/catalog/albums → catalog:5001/albums
- path: /api/catalog(/|$)(.*)
pathType: ImplementationSpecific
backend:
service:
name: catalog
port:
number: 5001
# /api/orders/orders → order:5003/orders
- path: /api/orders(/|$)(.*)
pathType: ImplementationSpecific
backend:
service:
name: order
port:
number: 5003
# ... more routes
```
### Path Rewriting
The annotation `nginx.ingress.kubernetes.io/rewrite-target: /$2` strips the `/api/<service>` prefix:
| External URL | Internal URL |
|--------------|--------------|
| `GET /api/catalog/albums` | `GET /albums` on catalog:5001 |
| `GET /api/orders/orders/123` | `GET /orders/123` on order:5003 |
| `POST /api/inventory/stock/CD001/reserve` | `POST /stock/CD001/reserve` on inventory:5002 |
### Benefits of API Gateway
1. **Single Entry Point**: Clients only need to know one URL
2. **Cross-Cutting Concerns**: SSL termination, rate limiting, auth (at edge)
3. **Path-Based Routing**: Clean URL structure
4. **Protocol Translation**: gRPC internally, REST externally (if needed)
---
## Database Per Service
Newman strongly advocates for database-per-service to ensure loose coupling.
### Implementation in MusicCorp
Single PostgreSQL instance with logical database isolation:
```
┌─────────────────────────────────────────────────────────────────┐
│ PostgreSQL Instance │
│ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │
│ │musiccorp_catalog│ │musiccorp_invent.│ │ musiccorp_order │ │
│ │ (catalog_user) │ │ (inventory_user)│ │ (order_user) │ │
│ │ │ │ │ │ │ │
│ │ ┌──────────┐ │ │ ┌──────────┐ │ │ ┌──────────┐ │ │
│ │ │ albums │ │ │ │ stock │ │ │ │ orders │ │ │
│ │ └──────────┘ │ │ └──────────┘ │ │ └──────────┘ │ │
│ └─────────────────┘ └─────────────────┘ └─────────────────┘ │
│ ┌─────────────────┐ ┌─────────────────┐ │
│ │musiccorp_payment│ │musiccorp_shippi.│ │
│ │ (payment_user) │ │ (shipping_user) │ │
│ │ │ │ │ │
│ │ ┌──────────┐ │ │ ┌──────────┐ │ │
│ │ │ payments │ │ │ │shipments │ │ │
│ │ └──────────┘ │ │ └──────────┘ │ │
│ └─────────────────┘ └─────────────────┘ │
└─────────────────────────────────────────────────────────────────┘
```
### Database Initialization
The init script creates isolated databases with separate credentials:
```sql
-- From postgres/init-databases.sql (via ConfigMap)
-- Catalog database
CREATE USER catalog_user WITH PASSWORD 'catalog_pass';
CREATE DATABASE musiccorp_catalog OWNER catalog_user;
GRANT ALL PRIVILEGES ON DATABASE musiccorp_catalog TO catalog_user;
-- Inventory database
CREATE USER inventory_user WITH PASSWORD 'inventory_pass';
CREATE DATABASE musiccorp_inventory OWNER inventory_user;
GRANT ALL PRIVILEGES ON DATABASE musiccorp_inventory TO inventory_user;
-- ... repeat for order, payment, shipping
```
### Isolation Enforcement
Each service only has credentials for its own database:
```yaml
# Order service only gets ORDER_DATABASE_URL
env:
- name: DATABASE_URL
valueFrom:
secretKeyRef:
name: postgres-secret
key: ORDER_DATABASE_URL # Only order database access
```
**Benefits:**
- Services cannot accidentally query each other's data
- Schema changes are isolated
- Independent scaling decisions
- Clear ownership boundaries
---
## Event-Driven Infrastructure
Newman emphasizes asynchronous communication for loose coupling between services.
### Kafka Configuration
MusicCorp uses Apache Kafka in KRaft mode (no Zookeeper):
```yaml
# zarf/k8s/base/kafka/base-kafka.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: kafka
spec:
template:
spec:
containers:
- name: kafka
image: apache/kafka:3.7.0
env:
# KRaft mode configuration
- name: KAFKA_NODE_ID
value: "1"
- name: KAFKA_PROCESS_ROLES
value: "broker,controller"
- name: KAFKA_CONTROLLER_QUORUM_VOTERS
value: "1@localhost:9093"
- name: KAFKA_LISTENERS
value: "PLAINTEXT://:9092,CONTROLLER://:9093"
- name: KAFKA_ADVERTISED_LISTENERS
value: "PLAINTEXT://kafka:9092"
- name: KAFKA_AUTO_CREATE_TOPICS_ENABLE
value: "true"
```
### Event Topics
| Topic | Publisher | Subscribers | Purpose |
|-------|-----------|-------------|---------|
| `order.placed` | Order | Payment | Triggers payment processing |
| `payment.received` | Payment | Order, Shipping | Payment confirmed |
| `payment.failed` | Payment | Order, Inventory | Triggers saga compensation |
| `shipment.dispatched` | Shipping | Order | Order shipped |
| `order.cancelled` | Order | - | Order cancelled |
| `stock.released` | Inventory | - | Stock released after failure |
### Event Flow
```
Order Placed → Payment Processing → Shipment → Completion
┌─────────┐ order.placed ┌─────────┐
│ Order │ ─────────────────────▶ │ Payment │
│ Service │ │ Service │
└────┬────┘ └────┬────┘
│ │
│ payment.received │
│ ◀────────────────────────────────┤
│ │
│ ┌────▼────┐
│ shipment.dispatched │Shipping │
│ ◀───────────────────────────│ Service │
│ └─────────┘
▼
Order COMPLETED
```
---
## Observability Stack
Newman emphasizes observability as critical for operating microservices. MusicCorp implements the LGTM stack (Loki, Grafana, Tempo, Mimir/Prometheus).
### Components
| Component | Purpose | Port |
|-----------|---------|------|
| **Prometheus** | Metrics collection | 9090 (host: 30909) |
| **Loki** | Log aggregation | 3100 (host: 30310) |
| **Tempo** | Distributed tracing | 3200 (host: 30320) |
| **Grafana** | Visualization | 3000 (host: 30300) |
| **Promtail** | Log shipping | DaemonSet |
### Metrics Collection
Services expose Prometheus metrics via annotations:
```yaml
# Pod annotations for Prometheus scraping
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "5003"
prometheus.io/path: "/metrics"
```
### Distributed Tracing
Services send traces to Tempo via OpenTelemetry:
```yaml
env:
- name: OTEL_EXPORTER_OTLP_ENDPOINT
value: "http://tempo:4317"
```
### Correlation ID Propagation
All services propagate the `X-Correlation-ID` header for request tracing:
```
Client → Ingress → Order → Catalog
└→ Inventory → Kafka
└→ Payment → Shipping
All requests share the same X-Correlation-ID
```
---
## Local Development with Kind
Kind (Kubernetes in Docker) provides a local Kubernetes cluster for development.
### Cluster Configuration
```yaml
# zarf/k8s/dev/kind-config.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
kubeadmConfigPatches:
- |
kind: InitConfiguration
nodeRegistration:
kubeletExtraArgs:
node-labels: "ingress-ready=true"
extraPortMappings:
# API Gateway
- containerPort: 80
hostPort: 80
- containerPort: 443
hostPort: 443
# Direct service access (development)
- containerPort: 5001
hostPort: 5001
- containerPort: 5002
hostPort: 5002
# ... more ports
# Observability
- containerPort: 30300
hostPort: 3000 # Grafana
- containerPort: 30909
hostPort: 9090 # Prometheus
```
### Port Mapping Strategy
| Type | Container Port | Host Port | Access |
|------|----------------|-----------|--------|
| Ingress | 80/443 | 80/443 | `http://localhost/api/...` |
| Services | 5001-5005 | 5001-5005 | `http://localhost:5001/...` |
| Grafana | 30300 | 3000 | `http://localhost:3000` |
| Prometheus | 30909 | 9090 | `http://localhost:9090` |
---
## Manifest Organization with Kustomize
Kustomize provides a structured way to manage Kubernetes manifests across environments.
### Directory Structure
```
zarf/k8s/
├── base/ # Environment-agnostic manifests
│ ├── namespace.yaml # musiccorp-system namespace
│ ├── catalog/
│ │ ├── base-catalog.yaml # Deployment + Service
│ │ └── kustomization.yaml
│ ├── inventory/
│ ├── order/
│ ├── order-go/
│ ├── payment/
│ ├── shipping/
│ ├── postgres/
│ │ ├── base-postgres.yaml
│ │ ├── init-databases.sql
│ │ └── kustomization.yaml
│ ├── kafka/
│ ├── ingress/
│ └── observability/
│ ├── prometheus/
│ ├── grafana/
│ ├── loki/
│ ├── tempo/
│ └── promtail/
│
└── dev/ # Development environment overlays
├── kind-config.yaml # Kind cluster configuration
├── catalog/
│ ├── dev-catalog-patch.yaml # Dev-specific patches
│ └── kustomization.yaml
├── inventory/
└── ...
```
### Base Kustomization
```yaml
# zarf/k8s/base/catalog/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- base-catalog.yaml
```
### Dev Overlay
```yaml
# zarf/k8s/dev/catalog/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base/catalog/
patches:
- path: ./dev-catalog-patch.yaml
images:
- name: catalog-image
newName: localhost/musiccorp/catalog
newTag: "0.0.1"
```
### Benefits of Kustomize
1. **No Templating**: Plain YAML, easier to read
2. **Overlay Model**: Base + environment-specific patches
3. **Image Transformations**: Override images without modifying base
4. **Built into kubectl**: `kubectl apply -k`
### Common Operations
```bash
# Preview rendered manifests
kustomize build zarf/k8s/dev/catalog
# Apply specific service
kubectl apply -k zarf/k8s/dev/catalog
# Apply all dev manifests
kubectl apply -k zarf/k8s/dev/
```
---
## Summary: Chapter 8 Principles in Practice
| Newman's Principle | MusicCorp Implementation |
|-------------------|--------------------------|
| **Isolated Execution** | Each service is a separate Deployment with own container |
| **Desired State** | Declarative YAML manifests, Kubernetes reconciliation |
| **Immutable Deployment** | Container images rebuilt, pods replaced on update |
| **Service Discovery** | Kubernetes DNS, ClusterIP Services |
| **Health Checks** | Readiness and liveness probes on all services |
| **API Gateway** | nginx Ingress Controller with path-based routing |
| **Database Per Service** | Separate PostgreSQL databases with isolated credentials |
| **Event-Driven** | Kafka for async communication between services |
| **Observability** | LGTM stack (Prometheus, Loki, Tempo, Grafana) |
| **Configuration** | Environment variables from ConfigMaps and Secrets |
---
## Related Documentation
- [07-build.md](07%20-%20Build%20Pipeline%20Guide.md) - Build pipeline concepts (Chapter 7)
- [k8s-commands.md](k8s-commands.md) - Quick reference for kubectl commands
- [event-schemas.md](event-schemas.md) - Kafka event format documentation
- [aws-deployment.md](aws-deployment.md) - Production AWS deployment guide