Building a Complete DevOps Pipeline in My Homelab - with Terraform, Ansible

After setting up my initial Proxmox cluster (which I wrote about in my previous post), I wanted to take my homelab to the next level by implementing a full DevOps pipeline. This project has been incredible for learning industry-standard tools and practices in a controlled environment. I’ll walk you through how I set up a complete CI/CD pipeline with infrastructure as code, configuration management, containerization, and monitoring.

Architecture Overview

My DevOps homelab consists of a four-VM setup running on my Proxmox cluster. Here’s the architecture I implemented:

Infrastructure:

Proxmox Hypervisor running on my Dell OptiPlex and laptop nodes
- VM1: Jenkins Server (CI/CD) - 4GB RAM, 2 vCPUs, 50GB storage
- VM2: Application VM (Dev Environment) - 2GB RAM, 1 vCPU, 30GB storage
- VM3: Application VM (Production Environment) - 2GB RAM, 1 vCPU, 30GB storage
- VM4: Prometheus + Grafana Monitoring - 4GB RAM, 2 vCPUs, 40GB storage

Networking:

All VMs on a private subnet (192.168.100.0/24)
Jenkins server with SSH access to other VMs via private key authentication
Prometheus/Grafana VM with access to monitor metrics from all other VMs

0. Getting Started

Jenkins

In Proxmox, I deployed an Ubuntu server LXC with the resources mentioned above. I installed docker and followed the official Jenkins documentation. First downloaded and ran the Docker-in-Docker(dind) image and configured it to a docker network. I then created a custom Docker image which would install Jenkins and Docker CLI, as well as blueocean. After building and running both containers. I configured Jenkins to connect to my Github. I chose this set up as Jenkins will be able to deploy agents using dind which offers flexibility when executing jobs.

Here is how I set up my Jenkinsfile to execute terraform with Jenkins:

#Jenkinsfile
pipeline {
    agent {
        dockerfile {
            filename 'Dockerfile.agent'
            additionalBuildArgs '--no-cache'
            reuseNode true    
            args '-v /tmp:/tmp'
        }
    }
    
    parameters {
        string(name: 'PROXMOX_HOST', defaultValue: '192.168.1.10', description: 'Proxmox host IP address')
        string(name: 'VM_NAME', defaultValue: 'test-vm', description: 'Name for the new VM')
        string(name: 'VM_IP', defaultValue: '192.168.1.11', description: 'IP address for the new VM')
    }
    
    environment {
        PROXMOX_API_TOKEN_ID = credentials('proxmox-token-id')
        PROXMOX_API_TOKEN_SECRET = credentials('proxmox-token-secret')
        SSH_PUBLIC_KEY = credentials('homelab-ssh-public-key')
        TF_LOG = 'DEBUG'
        TF_LOG_PATH = 'terraform-debug.log'
    }
    
    stages {
        stage('Checkout') {
            steps {
                checkout scm
            }
        }
        
        stage('Test Connectivity') {
            steps {
                sh '''
                echo "Testing connectivity to Proxmox..."
                ping -c 3 ''' + params.PROXMOX_HOST + ''' || echo "Ping failed but continuing"
                
                echo "Testing API access..."
                curl -k -s -o /dev/null -w "Proxmox API HTTP Status: %{http_code}\\n" https://''' + params.PROXMOX_HOST + ''':8006/ || echo "API check failed but continuing"
                '''
            }
        }
        
        stage('Terraform Init') {
            steps {
                dir('terraform') {
                    sh 'terraform init -upgrade'
                }
            }
        }
        
        stage('Terraform Plan') {
            steps {
                dir('terraform') {
                    sh """
                    terraform plan \\
                      -var="proxmox_api_url=https://${params.PROXMOX_HOST}:8006/api2/json" \\
                      -var="proxmox_api_token_id=\${PROXMOX_API_TOKEN_ID}" \\
                      -var="proxmox_api_token_secret=\${PROXMOX_API_TOKEN_SECRET}" \\
                      -var="proxmox_node=pve-01" \\
                      -var="vm_name=${params.VM_NAME}" \\
                      -var="vm_ip=${params.VM_IP}" \\
                      -var="ssh_public_key=\${SSH_PUBLIC_KEY}" \\
                      -out=tfplan
                    """
                }
            }
        }
        
        stage('Terraform Apply') {
            steps {
                dir('terraform') {
                    sh 'terraform apply -auto-approve tfplan'
                }
            }
        }
        
        stage('Verify VM Creation') {
            steps {
                dir('terraform') {
                    sh 'terraform output'
                }
            }
        }
    }
    
    post { 
        always {
            echo "Cleaning up workspace..."
            
            // Clean up Terraform files inside the container
            sh '''
                # Print disk usage
                echo "Disk usage:"
                df -h /
                
                # Remove Terraform temporary files
                if [ -d "terraform" ]; then
                    cd terraform
                    rm -f terraform.tfstate.backup tfplan
                    cd ..
                fi
                
                # Clean workspace
                rm -rf .terraform
            '''
            
            cleanWs notFailBuild: true
            build job: 'docker-cleanup', wait: false, propagate: false
        }
        
        success { 
            echo "VM creation successful! VM '${params.VM_NAME}' has been created with IP ${params.VM_IP}" 
        } 
        failure { 
            echo "VM creation failed. Check the logs for details." 
        } 

    }    
}

1. Setting Up the Infrastructure with Terraform

First, I needed to provision the virtual machines. While Proxmox has a great UI, I wanted to practice Infrastructure as Code, so I used Terraform with the Proxmox provider.

Here’s how I created the Terraform configuration:

  
# main.tf
terraform {
  required_version = ">= 1.1.0"
  required_providers {
    proxmox = {
      source  = "telmate/proxmox"
      version = "3.0.1-rc3"
    }
  }
}

provider "proxmox" {
  pm_tls_insecure = true
  pm_api_url = var.proxmox_api_url
  pm_api_token_id = var.proxmox_api_token_id
  pm_api_token_secret = var.proxmox_api_token_secret
}

resource "proxmox_vm_qemu" "test_vm" {
  name        = var.vm_name
  desc        = "VM created via Terraform"
  target_node = var.proxmox_node
  
  # Clone from template
  clone       = var.template_name
  
  # VM settings
  agent                  = 1
  automatic_reboot       = true
  balloon                = 0
  bios                   = "seabios"
  boot                   = "order=scsi0;net0"
  cores                  = 2
  define_connection_info = true
  force_create           = false
  hotplug                = "network,disk,usb"
  memory                 = 2048
  numa                   = false
  onboot                 = false
  scsihw                 = "virtio-scsi-pci"
  sockets                = 1
  
  # Disk configuration - using disks block instead of disk
  disks {
    scsi {
      scsi0 {
        disk {
          backup   = true
          cache    = "none"
          discard  = true
          iothread = true
          size     = "20G"
          storage  = "local-lvm"
        }
      }
    }
  }
  
  # Network configuration
  network {
    bridge   = "vmbr0"
    model    = "virtio"
    firewall = false
  }
  
  # Cloud-init settings
  os_type    = "cloud-init"
  ipconfig0  = "ip=${var.vm_ip}/24,gw=${var.gateway}"
  sshkeys    = var.ssh_public_key
}

# Additional resources for other VMs...

I ran terraform apply and watched as my VMs were automatically provisioned on my Proxmox cluster. This approach allowed me to version control my infrastructure and make changes in a controlled manner.

2. Configuring Servers with Ansible

With the VMs provisioned, I needed to configure them. Ansible made this process straightforward and repeatable. I created an Ansible playbook structure with roles for each component:

ansible/
├── inventory.yml
├── site.yml
└── roles/
    ├── common/
    ├── jenkins/
    ├── app_server/
    └── monitoring/

My inventory file looked like this:

  
# inventory.yml
all:
  children:
    jenkins:
      hosts:
        jenkins-server:
          ansible_host: 192.168.100.10
    dev:
      hosts:
        dev-server:
          ansible_host: 192.168.100.20
    prod:
      hosts:
        prod-server:
          ansible_host: 192.168.100.30
    monitoring:
      hosts:
        monitoring-server:
          ansible_host: 192.168.100.40
  vars:
    ansible_user: ubuntu
    ansible_ssh_private_key_file: ~/.ssh/homelab_key

I then ran the playbook to configure all servers:

ansible-playbook -i inventory.yml site.yml

The Jenkins role installed and configured Jenkins with the necessary plugins, while the app server roles set up the environments for my application, including Node.js, Docker, and other dependencies.

3. Building a Simple Node.js Application

For this project, I developed a basic Node.js API that serves weather data. It’s simple but has enough complexity to demonstrate the CI/CD pipeline. The app uses Express.js and connects to a MongoDB database to store and retrieve weather records.

  
// app.js
const express = require("express");
const mongoose = require("mongoose");
const app = express();
const port = process.env.PORT || 3000;

// Connect to MongoDB
mongoose.connect(
  process.env.MONGODB_URI || "mongodb://localhost:27017/weatherdb"
);

// Define Weather schema
const weatherSchema = new mongoose.Schema({
  location: String,
  temperature: Number,
  conditions: String,
  timestamp: { type: Date, default: Date.now }
});

const Weather = mongoose.model("Weather", weatherSchema);

// Routes
app.use(express.json());

app.get("/", (req, res) => {
  res.json({ message: "Weather API is running" });
});

app.get("/weather/:location", async (req, res) => {
  try {
    const weather = await Weather.findOne({
      location: req.params.location
    }).sort({ timestamp: -1 });

    if (!weather) {
      return res.status(404).json({ error: "Weather data not found" });
    }

    res.json(weather);
  } catch (err) {
    res.status(500).json({ error: err.message });
  }
});

// Start server
app.listen(port, () => {
  console.log(`Server running on port ${port}`);
});

I included unit tests using Jest and integration tests to verify the API endpoints functioned correctly.

4. Dockerizing the Application

To ensure consistent deployments across environments, I containerized my application using Docker:

  
# Dockerfile
FROM node:18-alpine

WORKDIR /app

COPY package*.json ./
RUN npm ci --only=production

COPY . .

EXPOSE 3000

CMD ["node", "app.js"]

And created a docker-compose.yml file to define the application stack:

  
# docker-compose.yml
version: "3"
services:
  app:
    build: .
    ports:
      - "3000:3000"
    environment:
      - NODE_ENV=production
      - MONGODB_URI=mongodb://mongo:27017/weatherdb
    depends_on:
      - mongo

  mongo:
    image: mongo:6
    volumes:
      - mongo-data:/data/db
    ports:
      - "27017:27017"

volumes:
  mongo-data:

5. Setting Up CI/CD with Jenkins

This was the most exciting part - creating an automated pipeline to build, test, and deploy my application. I created a Jenkinsfile in my repository:

  
// Jenkinsfile
pipeline {
    agent any

    environment {
        DOCKER_HUB_CREDS = credentials('docker-hub-credentials')
        APP_IMAGE = 'joshdev/weather-api'
    }

    stages {
        stage('Checkout') {
            steps {
                checkout scm
            }
        }

        stage('Install Dependencies') {
            steps {
                sh 'npm ci'
            }
        }

        stage('Lint') {
            steps {
                sh 'npm run lint'
            }
        }

        stage('Test') {
            steps {
                sh 'npm test'
            }
        }

        stage('Build Docker Image') {
            steps {
                sh "docker build -t ${APP_IMAGE}:${BUILD_NUMBER} ."
                sh "docker tag ${APP_IMAGE}:${BUILD_NUMBER} ${APP_IMAGE}:latest"
            }
        }

        stage('Push Docker Image') {
            steps {
                sh "echo ${DOCKER_HUB_CREDS_PSW} | docker login -u ${DOCKER_HUB_CREDS_USR} --password-stdin"
                sh "docker push ${APP_IMAGE}:${BUILD_NUMBER}"
                sh "docker push ${APP_IMAGE}:latest"
            }
        }

        stage('Deploy to Dev') {
            steps {
                sh "ansible-playbook -i ansible/inventory.yml ansible/deploy.yml --limit dev -e 'app_version=${BUILD_NUMBER}'"
            }
        }

        stage('Integration Tests') {
            steps {
                sh 'npm run test:integration'
            }
        }

        stage('Deploy to Production') {
            when {
                branch 'main'
            }
            steps {
                sh "ansible-playbook -i ansible/inventory.yml ansible/deploy.yml --limit prod -e 'app_version=${BUILD_NUMBER}'"
            }
        }
    }

    post {
        always {
            sh 'docker logout'
            sh 'npm run clean'
        }
    }
}

For deployment, I created an Ansible playbook (deploy.yml) that pulls the Docker image and starts the containers:

  
# ansible/deploy.yml
---
- name: Deploy Weather API
  hosts: ""
  become: yes
  vars:
    app_version: latest

  tasks:
    - name: Create app directory
      file:
        path: /opt/weather-api
        state: directory
        mode: "0755"

    - name: Copy docker-compose file
      template:
        src: templates/docker-compose.j2
        dest: /opt/weather-api/docker-compose.yml

    - name: Pull latest images
      community.docker.docker_compose:
        project_src: /opt/weather-api
        pull: yes

    - name: Start containers
      community.docker.docker_compose:
        project_src: /opt/weather-api
        state: present

6. Setting Up Monitoring with Prometheus and Grafana

To monitor the health and performance of my application and infrastructure, I configured Prometheus to scrape metrics from all VMs and the Node.js application (using the prom-client library).

I created a Prometheus configuration file:

  
# prometheus.yml
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: "jenkins"
    static_configs:
      - targets: ["192.168.100.10:9100"] # Node exporter on Jenkins

  - job_name: "dev_server"
    static_configs:
      - targets: ["192.168.100.20:9100"] # Node exporter on Dev
      - targets: ["192.168.100.20:3000"] # Weather API metrics endpoint

  - job_name: "prod_server"
    static_configs:
      - targets: ["192.168.100.30:9100"] # Node exporter on Prod
      - targets: ["192.168.100.30:3000"] # Weather API metrics endpoint

In Grafana, I created dashboards to visualize:

System metrics (CPU, memory, disk usage)
Application metrics (request count, response times, error rates)
Jenkins build metrics (success/failure rates, build duration)

Challenges and Lessons Learned

This project wasn’t without its challenges. Here are some issues I encountered and how I resolved them:

Resource Constraints: My homelab has limited resources, so I had to optimize VM resource allocation to prevent overloading my Proxmox nodes. I ended up reducing the VM resources initially allocated and implementing CPU/memory limits.
Networking Issues: Initially, I struggled with VM connectivity in my private subnet. The solution was to correctly configure the bridges in Proxmox and ensure the network settings in Terraform matched my physical network.
Jenkins Pipeline Debugging: The Jenkins pipeline failed initially due to missing dependencies and permissions. I had to update the Ansible roles to ensure all required packages were installed and proper permissions were set.
Monitoring Setup: Getting Prometheus to properly scrape all endpoints took some trial and error, particularly in configuring the Node.js application to expose metrics in the right format.

Next Steps

This DevOps homelab has been an incredible learning experience, but I’m not stopping here. My next plans include:

Implementing GitOps with ArgoCD: Moving from Jenkins to a GitOps approach for even more declarative deployments.
Adding Kubernetes: Replacing individual VMs with a Kubernetes cluster for better scalability and resource utilization.
Enhancing Security: Implementing Vault for secrets management and enhancing overall security practices.
Expanding Monitoring: Adding ELK stack for centralized logging alongside Prometheus/Grafana for metrics.

Conclusion

Building this DevOps pipeline in my homelab has been invaluable for my professional growth. It’s one thing to read about DevOps practices, but actually implementing them in a working environment provides a much deeper understanding.

The most valuable lesson I’ve learned is how these tools work together to create a seamless pipeline from code to production. Each component has its specific role, but the real power comes from their integration into a cohesive workflow.

If you’re looking to enhance your DevOps skills, I highly recommend setting up a similar environment. Start small, iterate often, and don’t be afraid to break things—that’s how the best learning happens!

Happy automating! 🚀

Building a Complete DevOps Pipeline in My Homelab