CI/CD and Automation

Loading…

CI/CD and Automation

Relevant source files

• .github/workflows/build-docs.yml
• .github/workflows/ci.yml

Purpose and Scope

This document describes the automated testing and documentation build systems that ensure code quality and maintain up-to-date documentation for the Docker MQTT Mosquitto Cloudflare Tunnel system.

The system implements two distinct automation pipelines:

1. Continuous Integration Pipeline - Validates that the Mosquitto service can successfully start on every commit
2. Documentation Build Pipeline - Generates and deploys wiki documentation to GitHub Pages

This page is organized into three main sections:

• Continuous Integration (Section 5.1) - Details the automated testing workflow
• Documentation Pipeline (Section 5.2) - Explains the documentation generation and deployment process
• Version Control Best Practices (Section 5.3) - Covers Git workflow and secret management in the context of CI/CD

For general system setup, see page 2 (Getting Started). For component details, see page 3 (Components).

Sources: .github/workflows/ci.yml, .github/workflows/build-docs.yml, .gitignore

Automation Pipeline Overview

The system uses GitHub Actions to automate quality assurance and documentation maintenance. Both pipelines execute in isolated GitHub-hosted runners and operate independently.

Pipeline Comparison

Sources: .github/workflows/ci.yml, .github/workflows/build-docs.yml

Automation Architecture

Diagram: GitHub Actions Pipeline Architecture

This diagram illustrates the two independent automation pipelines. The CI pipeline .github/workflows/ci.yml:1-43 executes on code changes, while the documentation pipeline .github/workflows/build-docs.yml:1-81 runs on a weekly schedule. Both use GitHub-hosted runners but have different trigger mechanisms and purposes.

Sources: .github/workflows/ci.yml, .github/workflows/build-docs.yml

5.1 Continuous Integration

The CI pipeline validates that the Mosquitto MQTT broker can successfully start in a Docker environment. This automated test runs on every push to main and on all pull requests targeting main.

Workflow Configuration

The workflow is defined in .github/workflows/ci.yml:1-43 with the following key elements:

Trigger Configuration .github/workflows/ci.yml:3-9:

Job Definition .github/workflows/ci.yml:11-13:

• Job Name: test-mosquitto
• Runner: ubuntu-latest
• Purpose: Validate Mosquitto container can start and reach running state

CI Execution Steps

Health Check Mechanism

Diagram: Health Check Loop Logic

The health check loop .github/workflows/ci.yml:28-39 implements a polling strategy with bounded retry:

This bash loop:

• Executes up to 10 iterations
• Uses docker inspect to query container state
• Checks if status contains “running”
• Waits 10 seconds between attempts
• Fails after 100 seconds total (10 × 10s)

Sources: .github/workflows/ci.yml:28-39

Why Only Mosquitto Is Tested

The CI pipeline starts only the mosquitto service, not the cloudflared service .github/workflows/ci.yml25:

Rationale:

• The cloudflared container requires CLOUDFLARE_TUNNEL_TOKEN environment variable
• This token is a secret that cannot be exposed in public CI logs
• The mosquitto service has no external dependencies and can run in isolation
• Starting Mosquitto validates the core MQTT broker functionality

The docker-compose.yml file docker-compose.yml:4-9 defines mosquitto as an independent service that doesn’t depend on cloudflared, enabling this isolated testing approach.

Sources: .github/workflows/ci.yml:25, docker-compose.yml:4-9

Failure Scenarios

When the CI workflow fails, it blocks pull request merges, preventing broken configurations from entering the main branch.

Sources: .github/workflows/ci.yml:28-42

5.2 Documentation Pipeline

The documentation pipeline automatically generates and deploys technical documentation to GitHub Pages. It uses the DeepWiki documentation generation system to create an mdBook-formatted static site from the repository’s codebase.

Workflow Configuration

The workflow is defined in .github/workflows/build-docs.yml:1-81 with distinct build and deploy jobs.

Trigger Configuration .github/workflows/build-docs.yml:3-7:

The pipeline executes in two scenarios:

1. Scheduled: Every Sunday at 00:00 UTC via cron schedule
2. Manual: Triggered through GitHub UI via workflow_dispatch

Permissions .github/workflows/build-docs.yml:9-12:

These permissions enable:

• Reading repository contents
• Writing to GitHub Pages deployment
• Generating OIDC tokens for secure deployment

Build Job

The build job .github/workflows/build-docs.yml:19-69 performs documentation generation:

Diagram: Documentation Build Job Flow

Step 1: Repository Resolution

.github/workflows/build-docs.yml:25-52 determines which repository to document:

This logic allows manual workflow dispatch to specify a different repository, defaulting to the current repository (github.repository context variable).

Step 2: Title Generation

.github/workflows/build-docs.yml:37-47 generates the documentation book title:

For this repository, the generated title would be: "docker-mqtt-mosquitto-cloudflare-tunnel Documentation"

Step 3: DeepWiki Generation

.github/workflows/build-docs.yml:59-64 invokes the DeepWiki documentation generator:

The jzombie/deepwiki-to-mdbook action:

• Analyzes the repository structure
• Generates markdown documentation pages
• Creates an mdBook configuration
• Outputs static HTML to ./output/book

Step 4: Artifact Upload

.github/workflows/build-docs.yml:66-69 uploads the generated documentation:

This creates a GitHub Actions artifact containing the compiled mdBook site, which is consumed by the deploy job.

Sources: .github/workflows/build-docs.yml:19-69

Deploy Job

The deploy job .github/workflows/build-docs.yml:71-81 publishes documentation to GitHub Pages:

Diagram: Documentation Deploy Job Flow

The deploy job configuration .github/workflows/build-docs.yml:71-80:

Key aspects:

• Dependency: needs: build ensures build completes first
• Environment: Publishes to github-pages environment
• Output: page_url provides the live documentation URL
• Idempotency: Overwrites existing GitHub Pages deployment

Sources: .github/workflows/build-docs.yml:71-81

Concurrency Control

.github/workflows/build-docs.yml:14-16 prevents concurrent documentation builds:

This configuration:

• Groups all Pages deployments under single concurrency group
• Queues new runs instead of canceling in-progress deployments
• Prevents race conditions in GitHub Pages publishing

Sources: .github/workflows/build-docs.yml:14-16

Documentation Update Frequency

The weekly schedule ensures documentation stays current without excessive CI usage:

Manual triggering is accessible via: Actions → Build and Deploy Documentation → Run workflow

Sources: .github/workflows/build-docs.yml:3-7

5.3 Version Control Best Practices

The repository implements strict separation between public configuration and sensitive secrets to prevent credential leakage in version control.

Gitignore Configuration

The .gitignore file .gitignore:1-2 enforces this separation:

.env
data/*

Rationale:

Secret Management Pattern

Diagram: Secret Management Flow in Version Control

This diagram shows how secrets flow from Cloudflare Dashboard to Docker containers without entering version control. The .gitignore file .gitignore1 acts as a security boundary.

Sources: .gitignore:1-2

Tracked vs Untracked Files

Git Workflow Integration

The CI pipeline .github/workflows/ci.yml:3-9 enforces version control best practices:

Protected Branch Pattern:

• All changes to main require pull request
• CI must pass before merge
• Prevents broken configuration from entering main branch

Pull Request Flow:

1. Developer creates feature branch
2. Pushes changes to remote
3. Opens pull request to main
4. CI workflow executes automatically .github/workflows/ci.yml:7-9
5. If CI passes, pull request can be merged
6. If CI fails, developer must fix issues

Environment Variable Security

The docker-compose.yml file docker-compose.yml:16-17 references the token without exposing it:

This pattern:

• Uses variable substitution instead of hardcoded values
• Reads from .env file at runtime (excluded by .gitignore)
• Never logs the token value in CI output
• Follows twelve-factor app methodology

Sources: docker-compose.yml:16-17, .gitignore:1

Data Directory Management

The data/* directory .gitignore2 is excluded for several reasons:

1. Size: Mosquitto may accumulate significant message data
2. Privacy: MQTT messages might contain sensitive IoT data
3. State: Runtime state should not be in version control
4. Portability: Each deployment should have independent state

The directory is automatically created by the Mosquitto container when it starts.

Sources: .gitignore:2

Commit Hygiene for CI/CD

DO:

• Commit workflow YAML files .github/workflows/
• Commit public configuration mosquitto.conf
• Update .env.sample when adding new required variables
• Include descriptive commit messages for CI changes

DON’T:

• Commit .env files with actual tokens
• Commit data/* runtime directories
• Hardcode secrets in workflow files
• Bypass CI by pushing directly to main

Security Checklist

Before committing changes related to CI/CD:

• Verify .gitignore excludes .env
• Confirm workflow files contain no hardcoded secrets
• Check that .env.sample is up-to-date
• Ensure CI workflow would catch your changes
• Review git status for untracked secret files
• Use git diff --cached to review staged changes

Sources: .gitignore:1-2, .github/workflows/ci.yml, .github/workflows/build-docs.yml

Testing Strategy

The automated testing focuses on validating successful service startup rather than functional MQTT testing:

Note that the cloudflared service is not tested in CI because it requires a valid CLOUDFLARE_TUNNEL_TOKEN, which is not available in the test environment. The CI validates only that the mosquitto service can successfully start in isolation.

Sources: .github/workflows/ci.yml

Development Commands

Developers use the following docker-compose commands during development:

These commands interact with the service definitions in docker-compose.yml:3-17

Sources: docker-compose.yml, .github/workflows/ci.yml

Configuration Management

Environment Variables

The system uses environment variable substitution in docker-compose.yml docker-compose.yml:14-17:

The CLOUDFLARE_TUNNEL_TOKEN variable must be defined in the .env file for the cloudflared service to authenticate with Cloudflare’s network.

Volume Mounts

The mosquitto configuration is provided via volume mount docker-compose.yml:7-8:

This allows developers to modify mosquitto.conf and restart the service without rebuilding any Docker images.

Sources: docker-compose.yml

Next Steps for Developers

After understanding this overview:

1. Setup Local Environment: Follow the detailed instructions in Local Development Setup to configure your workstation
2. Understand CI Pipeline: Review CI/CD Pipeline for details on how automated tests validate changes
3. Learn Version Control: Read Version Control Practices for guidelines on managing secrets and contributions

For advanced configuration topics such as ACLs and encryption, see Advanced Topics.

Sources: All files in this repository

Dismiss

Refresh this wiki

Enter email to refresh