Skip to content

Implement Jenkins HA

homepage-banner

In today’s fast-paced software development world, continuous integration and delivery (CI/CD) have become essential practices for delivering high-quality software at scale. Jenkins is a popular open-source automation server that enables developers to automate the CI/CD pipeline. However, as the number of builds and jobs increases, the risk of downtime and data loss also increases. That’s where Jenkins High Availability (HA) comes in. In this blog post, we’ll discuss the importance of Jenkins HA and a script to set it up.

Prerequisite

  • jq
  • etcd cluster
  • jenkins
  • rsync service
  • haproxy/elb

Code and Implementation

#!/usr/bin/env bash
ETCD_IP_LIST="etcd1.ip etcd2.ip etcd3.ip"
ETCD_PORT="2379"
EP=$(hostname)
KEY_NAME="j_master"
TTL=10
HBT=2
MASTER_IP=""
## startup
function rand_ip() {
  min=1
  max=$(echo $ETCD_IP_LIST | awk '{print NF}')
  num=$(($RANDOM + 1000000000))
  rand=$(($num % $max + $min))
  echo $ETCD_IP_LIST | cut -d" " -f$rand
}

function self_reg() {
  rc=$(curl -s $ETCD_IP:$ETCD_PORT/v2/keys/$KEY_NAME -XPUT -d value="$EP" -d prevExist="false" -d ttl=$TTL | jq '.["errorCode"]')
}

function get_master_ip() {
  MASTER_IP=$(curl -s $ETCD_IP:$ETCD_PORT/v2/keys/$KEY_NAME | jq '.["node"]["value"]')
}

function refresh_ttl() {
  x=$(curl -s $ETCD_IP:$ETCD_PORT/v2/keys/$KEY_NAME -XPUT -d ttl=$TTL -d refresh=true -d prevExist=true)
}

while :; do
  ETCD_IP=$(rand_ip)
  self_reg
  if [[ $rc -eq 105 ]]; then
    get_master_ip
    echo "[INFO] $(date '+%Y-%m-%d %H:%M:%S') MASTER_IP: $MASTER_IP"
    if [[ "$MASTER_IP" == "\"$EP\"" ]]; then
      echo "[INFO] $(date '+%Y-%m-%d %H:%M:%S') keep jenkins start"
      service jenkins start
      refresh_ttl
    else
      echo "[INFO] $(date '+%Y-%m-%d %H:%M:%S') service jenkins stop"
      service jenkins stop
      echo "[INFO] $(date '+%Y-%m-%d %H:%M:%S') sync data from $MASTER_IP"
      rsync -a ${MASTER_IP//\"/}::jenkins/var/lib/jenkins /var/lib/
      echo "[INFO] $(date '+%Y-%m-%d %H:%M:%S') data sync done"
    fi
  elif [[ "$rc" == "null" ]]; then
    echo "[INFO] $(date '+%Y-%m-%d %H:%M:%S') register success"
    continue
  fi
  sleep $HBT
done

Conclusion

Jenkins High Availability is essential for ensuring uninterrupted service and preventing data loss in large-scale CI/CD environments. Setting up Jenkins HA can be a complex process, but with the script we’ve discussed in this blog post, it’s much simpler. By using the script to set up a Jenkins cluster, you can ensure that your builds and jobs run smoothly without any disruption. So, if you’re running a large-scale CI/CD environment with Jenkins, consider implementing Jenkins High Availability to ensure continuous integration and delivery.

Leave a message