Implement Jenkins HA
In today’s fast-paced software development world, continuous integration and delivery (CI/CD) have become essential practices for delivering high-quality software at scale. Jenkins is a popular open-source automation server that enables developers to automate the CI/CD pipeline. However, as the number of builds and jobs increases, the risk of downtime and data loss also increases. That’s where Jenkins High Availability (HA) comes in. In this blog post, we’ll discuss the importance of Jenkins HA and a script to set it up.
Prerequisite
- jq
- etcd cluster
- jenkins
- rsync service
- haproxy/elb
Code and Implementation
#!/usr/bin/env bash
ETCD_IP_LIST="etcd1.ip etcd2.ip etcd3.ip"
ETCD_PORT="2379"
EP=$(hostname)
KEY_NAME="j_master"
TTL=10
HBT=2
MASTER_IP=""
## startup
function rand_ip() {
min=1
max=$(echo $ETCD_IP_LIST | awk '{print NF}')
num=$(($RANDOM + 1000000000))
rand=$(($num % $max + $min))
echo $ETCD_IP_LIST | cut -d" " -f$rand
}
function self_reg() {
rc=$(curl -s $ETCD_IP:$ETCD_PORT/v2/keys/$KEY_NAME -XPUT -d value="$EP" -d prevExist="false" -d ttl=$TTL | jq '.["errorCode"]')
}
function get_master_ip() {
MASTER_IP=$(curl -s $ETCD_IP:$ETCD_PORT/v2/keys/$KEY_NAME | jq '.["node"]["value"]')
}
function refresh_ttl() {
x=$(curl -s $ETCD_IP:$ETCD_PORT/v2/keys/$KEY_NAME -XPUT -d ttl=$TTL -d refresh=true -d prevExist=true)
}
while :; do
ETCD_IP=$(rand_ip)
self_reg
if [[ $rc -eq 105 ]]; then
get_master_ip
echo "[INFO] $(date '+%Y-%m-%d %H:%M:%S') MASTER_IP: $MASTER_IP"
if [[ "$MASTER_IP" == "\"$EP\"" ]]; then
echo "[INFO] $(date '+%Y-%m-%d %H:%M:%S') keep jenkins start"
service jenkins start
refresh_ttl
else
echo "[INFO] $(date '+%Y-%m-%d %H:%M:%S') service jenkins stop"
service jenkins stop
echo "[INFO] $(date '+%Y-%m-%d %H:%M:%S') sync data from $MASTER_IP"
rsync -a ${MASTER_IP//\"/}::jenkins/var/lib/jenkins /var/lib/
echo "[INFO] $(date '+%Y-%m-%d %H:%M:%S') data sync done"
fi
elif [[ "$rc" == "null" ]]; then
echo "[INFO] $(date '+%Y-%m-%d %H:%M:%S') register success"
continue
fi
sleep $HBT
done
Conclusion
Jenkins High Availability is essential for ensuring uninterrupted service and preventing data loss in large-scale CI/CD environments. Setting up Jenkins HA can be a complex process, but with the script we’ve discussed in this blog post, it’s much simpler. By using the script to set up a Jenkins cluster, you can ensure that your builds and jobs run smoothly without any disruption. So, if you’re running a large-scale CI/CD environment with Jenkins, consider implementing Jenkins High Availability to ensure continuous integration and delivery.