Jenkins Toolset Explained 2

One of the most useful feature of Jenkins Toolset is Compare. You can compare 2 jobs with your favorite diff tool. I use Beyond Compare for it.

You can see the differences in these 2 jobs in text. And you can sync all or parts of the job and push the change to the Jenkins master side right there on the tool.

This could be (or was) useful when I had to make similar change to an existing job from a new one.

The example is only showing the comparing 2 jobs on the same Jenkins master but it is possible to compare 2 different jobs on 2 different Jenkins master too.

To be continued…

Jenkins Toolset Explained 1

Like I wrote earlier, I created a WPF software to manage Jenkins jobs relatively easier if not a lot easier. In this article, I will explain the details of the features.

Architecture

Jenkins Toolset utilizes Jenkins’ REST APIs. The communication is through only HTTP or HTTPS. It supports both.

For the authentication, it uses token. We’ll go over how to do it below.

And you can make changes to the downloaded XML config files directly and the application keeps track on the changes.

Starting

When you start Jenkins Toolset, you have to enter the root URL for the target Jenkins instance.

When you hit enter there, you get a following error message at the bottom of the screen.

Jenkins Toolset doesn’t know who you are, so enter your username to the Jenkins instance.

Once you enter your username, you can click on Get Token button to get to the screen where you can generate your token to access the instance of the Jenkins via Jenkins Toolset.

Copy the token and paste it to the textbox on Jenkins Toolset. Once you have your username and token, you can now start to get the data from the Jenkins instance.

Edit Jobs

You can directly edit jobs in the XML form using Notepad++ or any text editor you would like. If you right click on a job or multiple jobs, you can open them in XML format.

Once, you make changes to the XML file, the changed jobs are marked in red meaning the change is still local and it’s ready to be uploaded.

You can right click the job in red and select Jenkins Server –> Push Job Changes to apply the change to the server side.

When you click Push Job Changes menu item, you get prompted if you are sure to push the change to the server side.

When the job push is successful, you will see a message and the red job turns back to the normal color.

And then, you can open the job with the browser and you can see the changes applied. This feature is very effective if you have hundreds of jobs and you want to mass update something and you wanna do it by doing find and replace operation. Notepad++ can do regex find and replace for each opened file.

To be continued…

Uploading Backup File to OCI’s Object Storage via Jenkins

I have had a need to upload a zip file for backup from a Windows agent to Oracle Cloud Infrastructure’s Object Storage. Here is what I did.

Installed OCI CLI for Windows. Please follow this link to install it on Windows. Then, Install Jenkins slave on the same machine. I have a step by step instruction on how to do it. Once you install it, make sure to change the account to run the slave as to the account you used to install OCI CLI. Otherwise, it won’t work.

On the Jenkins job, using Compress-Archive Cmdlet, you can zip up some directories into a zip file.

Compress-Archive -Path $zipPaths -DestinationPath $zipFile

Please note that Compress-Archive has a limitation of 2GB. I heard that it’s the limitation of the underlining API.

Now that you have the zip file, you can upload it to Object Storage like the following.

oci os object put -bn backup --file $zipFile -ns "yournamespace" `
	--parallel-upload-count 5 --part-size 20 --verify-checksum

I am recommending this method to a customer because Object Storage is a relatively cheap and secure storage on OCI. It also supports retention duration and also replication. Great feature for relatively reasonable service.

How to Make Your Mac a Jenkins Server and Agent

You may be working on a personal project. Maybe your company has a different CI/CD system from what you are used to or it’s hard to be onboarded with the CI/CD system. I’ve come up with a way to create your own personal Jenkins server and agent all in one macOS. The following diagram shows what I am envisioning.

This whole things can easily fit on a laptop or a desktop. Obviously the architecture could be applied to Windows and Linux as well. I have Mac as my main development machine, so I will use Mac as an example here.

Install Docker on Mac

Installing Docker on macOS is very easy.

  • First, download Docker for Mac from here.
  • (Double-)click the downloaded Docker.dmg file.
  • Drag and drop the Docker application file to /Applications directory.
  • Double-click Docker.app file in /Applications directory.
  • You may have to reboot your machine to get the daemon to work correctly.
  • Open Terminal.
  • Execute docker run -d -p 80:80 docker/getting-started
  • Once it’s successful, it means your Docker installation was successful.

One thing to note is that you don’t have to add your account to docker group so you don’t have to sudo for every docker command to execute and docker-compose is bundled with Docker.app on Mac, which is nicer than the installation process on Linux.

Jenkins Master

We will spin up the Jenkins master using docker-compose. Create a directory such as ~/jenkins.

Copy and paste the following contents in docker-compose.yaml file in ~/jenkins.

version: '3.8'
services:
    jenkins:
        image: jenkins/jenkins:2.263.1
        container_name: jenkins
        user: '1000'
        volumes:
            - ./jenkins_home:/var/jenkins_home
        ports:
            - '8080:8080'
            - '50000:50000'
        networks:
            - docker_network
        restart: unless-stopped
        environment:
            - JENKINS_JAVA_OPTIONS=-Djava.awt.headless=true
            - JENKINS_LOG=/var/jenkins_home/logs/jenkins.log
        #command: --enable-future-java
networks:
    docker_network:
        external:
            name: custom_network

Execute the following command to create a network for your Docker containers.

docker network create custom_network

After that, execute the following command from at ~/jenkins

docker-compose up

When you see the output like the following, you have Jenkins up and running

jenkins    | This may also be found at: /var/jenkins_home/secrets/initialAdminPassword
jenkins    |
jenkins    | *************************************************************
jenkins    | *************************************************************
jenkins    | *************************************************************
jenkins    |
jenkins    | 2021-01-08 07:21:22.224+0000 [id=44]	INFO	h.m.DownloadService$Downloadable#load: Obtained the updated data file for hudson.tasks.Maven.MavenInstaller
jenkins    | 2021-01-08 07:21:22.231+0000 [id=44]	INFO	hudson.util.Retrier#start: Performed the action check updates server successfully at the attempt #1
jenkins    | 2021-01-08 07:21:22.248+0000 [id=44]	INFO	hudson.model.AsyncPeriodicWork#lambda$doRun$0: Finished Download metadata. 9,056 ms
jenkins    | 2021-01-08 07:21:23.526+0000 [id=31]	INFO	jenkins.InitReactorRunner$1#onAttained: Completed initialization
jenkins    | 2021-01-08 07:21:23.554+0000 [id=20]	INFO	hudson.WebAppMain$3#run: Jenkins is fully up and running

At this point, you could hit the Jenkins master by navigating your browser to http://localhost:8080 but this is not what I want. I would like to have Dockerized NGINX in front of it and route the traffic to the Jenkins so that it is easy to implement SSL certs for it.

You can Ctrl+C to stop Jenkins and then…

docker-compose down

to completely remove the container.

NGINX as Reverse Proxy

As the diagram indicates, my plan is to have NGINX accept HTTP(S) traffic and route the traffic to the Jenkins instance via the Docker’s virtual network.

Please follow the steps below.

Create a directory like ~/nginx

mkdir ~/nginx

Copy and paste the following YAML to ~/nginx/docker-compose.yaml file and save the file.

version: '3.8'
services:
    reverse:
        image: nginx:latest
        volumes:
            - ./data/nginx.conf:/etc/nginx/nginx.conf
            - ./data/conf.d:/etc/nginx/conf.d
        ports:
            - "80:80"
            - "443:443"
        networks:
            proxynet:
        restart: always
networks:
    proxynet:
        name: custom_network

Next, create ~/nginx/data directory. While you are there, let’s create conf.d directory.

mkdir data
cd data
mkdir conf.d

Copy and paste the following contents to ~/nginx/data/nginx.conf file and save it.

user  nginx;
worker_processes  2;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;

events {
   worker_connections  1024;
   use epoll;
   accept_mutex off;
}

http {
   include       /etc/nginx/mime.types;
   proxy_set_header X-Real-IP $remote_addr;
   proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

   default_type  application/octet-stream;

   log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                     '$status $body_bytes_sent "$http_referer" '
                     '"$http_user_agent" "$http_x_forwarded_for"';

   access_log  /var/log/nginx/access.log  main;

   sendfile        on;
   #tcp_nopush     on;

   keepalive_timeout  65;

   client_max_body_size 300m;
   client_body_buffer_size 128k;

   gzip  on;
   gzip_http_version 1.0;
   gzip_comp_level 6;
   gzip_min_length 0;
   gzip_buffers 16 8k;
   gzip_proxied any;
   gzip_types text/plain text/css text/xml text/javascript application/xml application/xml+rss application/javascript application/json;
   gzip_disable "MSIE [1-6]\.";
   gzip_vary on;

   server {
       listen       80 default_server;
       listen       [::]:80 default_server;
       server_name  _;
       sendfile        on;
       #tcp_nopush     on;

       keepalive_timeout  65;

      client_max_body_size 300m;
      client_body_buffer_size 128k;

      gzip  on;
      gzip_http_version 1.0;
      gzip_comp_level 6;
      gzip_min_length 0;
      gzip_buffers 16 8k;
      gzip_proxied any;
      gzip_types text/plain text/css text/xml text/javascript application/xml application/xml+rss application/javascript application/json;
      gzip_disable "MSIE [1-6]\.";
      gzip_vary on;
   }
    include /etc/nginx/conf.d/*.conf;
}

Next, paste the following configuration to ~/nginx/conf.d/ssl.conf file.

upstream jenkins_upstream {
    server jenkins:8080;
}

server {
    server_name jenkins.local;
    listen 443 ssl;
    ssl_certificate /etc/nginx/conf.d/ssl/jenkins.local.crt;
    ssl_certificate_key /etc/nginx/conf.d/ssl/jenkins.local.key;

    location / {
        proxy_set_header        Host $host:$server_port;
        proxy_set_header        X-Real-IP $remote_addr;
        proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header        X-Forwarded-Proto $scheme;
        resolver 127.0.0.11;
        #proxy_redirect http:// https://;
        proxy_pass http://jenkins_upstream;
        # Required for new HTTP-based CLI
        proxy_http_version 1.1;
        proxy_request_buffering off;
        proxy_buffering off; # Required for HTTP-based CLI to work over SSL
        # workaround for https://issues.jenkins-ci.org/browse/JENKINS-45651
        add_header 'X-SSH-Endpoint' 'jenkins.local:50022' always;
    }
}

SSL Support

Since I’m just planning to run this locally, there isn’t a need for SSL but I would like to implement this for a good practice. And you never know, you may want to create a Jenkins instance in a real environment eventually.

I am planning to make the URL https://jenkins.local

Place the following files for SSL support. If you need to know how to create self-signed cert, please take a look my previous article.

To summarize, here are the files you have to create for NGINX to work in the nginx root directory.

├── data
│   ├── conf.d
│   │   ├── ssl
│   │   │   ├── jenkins.local.crt
│   │   │   └── jenkins.local.key
│   │   └── ssl.conf
│   └── nginx.conf
└── docker-compose.yml

Next, let’s edit /etc/hosts file so that https://jenkins.local will resolve to your local NGINX.

sudo vim /etc/hosts

Add the following line in the file.

127.0.0.1       jenkins.local

Now navigate to ~/jenkins and execute the following command.

docker-compose up -d

This starts the instance of Jenkins. Now navigate to ~/nginx and execute the following command.

docker-compose up -d

This starts the instance of NGINX as a reverse proxy that routes the traffic to the containerized Jenkins.

Access Jenkins UI and Initial Configuration

If you open your browser, and navigate it to https://jenkins.local/ you should see the following screen.

Beware that if you try to get the Administrator password from /var/jenkins_home/secrets/initialAdminPassword, your host side does not have the file at that location. Instead, you should cat the file at the following location.

~/jenkins/jenkins_home/secrets/initialAdminPassword

Once you get the admin password from the location, enter it to pass the initial screen to start the containerized Jenkins master.

Connect Jenkins Slave

If you take a look at the UI, you see that the Jenkins master has 2 executors by default. If you want your Mac to run builds, you can do so. Here are the steps.

Navigate to Manage Jenkins -> Manage Nodes and Clouds and then click New Node.

Enter mac for Node name and click Permanent Agent and then click OK.

And click Save on the next screen.

When you go back to Manage Nodes and Clouds, you will see the following screen.

Click mac (or whatever you created) and you will see the following screen.

You can definitely follow the instruction on the screen to connect your mac as a Jenkins slave. First download the agent.jar file from the same page and store it in ~/jenkins-slave directory.

Next execute the following command.

echo 15ef91ec4499d7be20f64fdca93995ab37cd23182982d8814ccb35b6df8254e7 > secret-file

This writes the secret to the secret-file in the directory. And then try to execute the following command to get your mac connected to Jenkins master as a slave.

java -jar agent.jar -jnlpUrl https://jenkins.local/computer/mac/slave-agent.jnlp -secret @secret-file -workDir "~/jenkins-slave"

You would expect that this should be the last step but you would get an error like the following.

ava -jar agent.jar -jnlpUrl https://jenkins.local/computer/mac/slave-agent.jnlp -secret @secret-file -workDir "~/dev/jenkins-slave"
Jan 13, 2021 10:31:43 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
INFO: Using ~/dev/jenkins-slave/remoting as a remoting work directory
Jan 13, 2021 10:31:43 PM org.jenkinsci.remoting.engine.WorkDirManager setupLogging
INFO: Both error and output logs will be printed to ~/dev/jenkins-slave/remoting
Exception in thread "main" java.io.IOException: Failed to validate a server certificate. If you are using a self-signed certificate, you can use the -noCertificateCheck option to bypass this check.
	at hudson.remoting.Launcher.parseJnlpArguments(Launcher.java:571)
	at hudson.remoting.Launcher.run(Launcher.java:347)
	at hudson.remoting.Launcher.main(Launcher.java:298)

This means that Java does not trust the self signed SSL certificate you created. I blogged about this in the past, so please refer to this blog article to resolve this issue.

Once the SSL cert issue is resolved, it connects to Jenkins master and it’s ready like the image below.

INFO: Agent discovery successful
  Agent address: jenkins.local
  Agent port:    50000
  Identity:      5c:25:1c:e8:1d:35:f1:e8:d1:34:b6:52:ac:e9:e5:ca
Jan 13, 2021 11:13:33 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Handshaking
Jan 13, 2021 11:13:33 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to jenkins.local:50000
Jan 13, 2021 11:13:33 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Trying protocol: JNLP4-connect
Jan 13, 2021 11:13:33 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Remote identity confirmed: 5c:25:1c:e8:1d:35:f1:e8:d1:34:b6:52:ac:e9:e5:ca
Jan 13, 2021 11:13:33 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connected

Recap

It took me a few nights to write up this article. To recap…

  • Dockerizing Jenkins master
  • Using NGINX as a reverse proxy
  • Implementing SSL cert
  • Jenkins slave

This give you an environment where you can manage your own Jenkins in Docker and do whatever you want!

Bulk Create Jenkins Nodes

If permanent slaves are required and hundreds of them, it’s very manual and tedious tasks. I have created a PowerShell script to create Jenkins nodes in bulk. CloudBees’ version of Jenkins seems to have nice API to create nodes but open source/free version doesn’t seem to have one though Jenkins CLI could be utilized to accomplish it. Please follow my previous blog article to make sure your Jenkins CLI works before moving on with this article. Please note that this script does not install Jenkins slaves on separate machines but I am planning to create a script to do that in the future.

Prerequisites

  • Windows (The script I am going to introduce utilizes PowerShell, so Windows is the safest choice of OS.)
  • Java
  • Donwloaded jenkins-cli.jar

The PowerShell Script Execution

The PowerShell script that I wrote can generate nodes in bulk. It can be found here in GitHub.

For example, execute the script with the parameters like below.

.\Jenkins_BulkCreateNodes.ps1 -JenkinsServerUrl https://jenkins.linux-mint.local/ -Username myadminuser -ApiToken 1197adsfasdfadsfasdfasdfasdfasdf3a -NodeCount 100 -NodeNamePattern "winnode-###" -OverwriteNodeIfExists $True

The command above creates nodes in Jenkins master from winnode-001 to winnode-100. The red cross icon you see below only indicates that they are offline and no actual slaves is communicating with them yet.

You can change the naming pattern specifying -NodeNamePattern.

By specifying -OverwriteNodeIfExists parameter to $True, it can delete the node and recreate. It can be dangerous, so its default value is $False and it’s an optional parameter.

The PowerShell Code

I am pasting all the code I have written for this operation. But if you just want the file, please go to my GitHub repo. All the details of each parameters are documented in the code.

<#
    .SYNOPSIS
       Generates Jenkins nodes.
    .DESCRIPTION
        This script bulk generates Jenkins nodes on the master so that nodes will be ready for Jenkins slaves to talk to them. 
        It utilizes Jenkins CLI, so Java (https://java.com/en/download/) and downloaded jenkins-cli.jar file from Jenkins master is required.
    .PARAMETER JenkinsServerUrl
        Required. Base URL of the Jenkins master. e.g. https://myjenkins.com/
    .PARAMETER Username
        The username to use for this process.
    .PARAMETER ApiToken
        Generated API Token of the username.
    .PARAMETER NodeCount
        Specifies how many nodes to created.
    .PARAMETER NodeNamePattern
        Specify the pattern of node name here. If 20 nodes to be created and "foobar-###" is passed, foobar-001 to foobar-020 would be generated.
    .PARAMETER Description
        Optional. Specify the description of each node. Use this field to specify what kind of fleet you are creating.
    .PARAMETER RemoteRootDir
        Optional. Defaults to "C:\slave". Specify any directory where you expect slave bits to reside on each node.
    .PARAMETER Mode
        Optional. Defaults to NORMAL.
    .PARAMETER NumExecutors
        Optional. Defaults to 1.
    .PARAMETER Labels
        Optional. e.g. "WINDOWS 2019". Separate labels by space.
    .PARAMETER JenkinsCliJarFilePath
        Optional. When you download jenkins-cli.jar, the file is usually at ~/Download/jenkins-cli.jar. Downaload the file from https://your-jenkins-server/jnlpJars/jenkins-cli.jar
        There is a download link on https://your-jenkins-server/cli
    .PARAMETER OverwriteNodeIfExists
        Optional. If the node to be created already exists on the Jenkins master, it deletes it and recreates it if $True is passed. If $False, it is skipped.
    .PARAMETER KeepNodeXmlFiles
        Optional. Defaults to $False. If $True is passed, the XML files created for the nodes will be kept in the same directory as this script. This option could be used to reuse the XML files in different environments.
    
    
    
#>
[CmdletBinding()]
param 
(
    [Parameter(Mandatory=$True)] 
    [string]$JenkinsServerUrl,
    [Parameter(Mandatory=$True)]
    [string]$Username,
    [Parameter(Mandatory=$True)]
    [string]$ApiToken,
    [Parameter(Mandatory=$False)]
    [int]$NodeCount = 5,
    [Parameter(Mandatory=$False)]
    [string]$NodeNamePattern = "winnode-###",
    [Parameter(Mandatory=$False)]
    [string]$Description = "Bulk gen'ed node",
    [Parameter(Mandatory=$False)]
    [string]$RemoteRootDir = "C:\slave",
    [Parameter(Mandatory=$False)]
    [string]$Mode = "NORMAL",
    [Parameter(Mandatory=$False)]
    [int]$NumExecutors = 1,
    [Parameter(Mandatory=$False)]
    [string]$Labels = "",
    [Parameter(Mandatory=$False)]
    [string]$JenkinsCliJarFilePath = "$env:HOMEPATH\Downloads\jenkins-cli.jar",
    [Parameter(Mandatory=$False)]
    [bool]$OverwriteNodeIfExists = $False,
    [Parameter(Mandatory=$False)]
    [bool]$KeepNodeXmlFiles = $False
)

$digitCount = ($NodeNamePattern.ToCharArray() | Where-Object {$_ -eq '#'} | Measure-Object).Count

ForEach ($index In 1..$NodeCount)
{
    $number = "{0:D$digitCount}" -f $index
    $nodeName = $NodeNamePattern -replace "#{$digitCount}", $number 
    $xmlSettings = New-Object System.Xml.XmlWriterSettings
    $xmlSettings.Indent = $true
    $xmlSettings.Encoding = [System.Text.Encoding]::UTF8
    $xmlSettings.OmitXmlDeclaration = $True
    $xmlSettings.ConformanceLevel = "Document"

    $sw = New-Object System.IO.StringWriter
    $xw = [System.Xml.XmlWriter]::Create($sw, $xmlSettings)
    
    try 
    {
        $xw.WriteStartElement("slave")

        $xw.WriteElementString("name", $nodeName)
        $xw.WriteElementString("description", $Description)
        $xw.WriteElementString("remoteFS", $RemoteRootDir)
        $xw.WriteElementString("numExecutors", $NumExecutors)
        $xw.WriteElementString("mode", $Mode)

        $xw.WriteStartElement("retentionStrategy")
        $xw.WriteAttributeString("class", 'hudson.slaves.RetentionStrategy$Always')
        $xw.WriteEndElement() # retentionStrategy

        $xw.WriteStartElement("launcher")
        $xw.WriteAttributeString("class", "hudson.slaves.JNLPLauncher")

        $xw.WriteStartElement("workDirSettings")
        $xw.WriteElementString("disabled", "false")
        $xw.WriteElementString("internalDir", "remoting")
        $xw.WriteElementString("failIfWorkDirIsMissing", "false")
        $xw.WriteEndElement() # workDirSettings
        $xw.WriteElementString("websocket", "false")
        $xw.WriteEndElement() # launcher

        $xw.WriteElementString("label", $Labels)
        $xw.WriteElementString("nodeProperties", "")
        $xw.WriteEndElement() #slave
        $xw.WriteEndDocument()
        $xw.Flush()
    }
    catch [System.Exception]
    {
        Write-Host $_
        exit 1
    }
    finally 
    {
        $xw.Close()
    }

    # Write out the node XML to file to be used for standard input below.
    Set-Content -Path "$nodeName.xml" -Value $sw.ToString() -Force

    # Check the existence of the node.

    $processInfo = New-Object System.Diagnostics.ProcessStartInfo
    $processInfo.FileName = "java"
    $processInfo.RedirectStandardError = $True
    $processInfo.RedirectStandardOutput = $True
    $processInfo.RedirectStandardInput = $True
    $processInfo.Arguments = "-jar $JenkinsCliJarFilePath -s $JenkinsServerUrl -auth $Username`:$ApiToken get-node $nodeName"
    $processInfo.UseShellExecute = $False

    $process = New-Object System.Diagnostics.Process
    $process.StartInfo = $processInfo
    $process.Start() | Out-Null
    $process.WaitForExit()
    $stdo = $process.StandardOutput

    If ($process.ExitCode -ne 0)
    {
        Start-Process -FilePath java -NoNewWindow -Wait -ArgumentList "-jar $JenkinsCliJarFilePath","-s $JenkinsServerUrl","-auth $Username`:$ApiToken","create-node" -RedirectStandardInput "$nodeName.xml"
        Write-Host "$nodeName created ($(Get-Date))"
    }
    else 
    {
        If ($OverwriteNodeIfExists)
        {
            Write-Host "Deleting $nodeName..."
            Start-Process -FilePath java -NoNewWindow -Wait -ArgumentList "-jar $JenkinsCliJarFilePath","-s $JenkinsServerUrl","-auth $Username`:$ApiToken","delete-node $nodeName"

            Write-Host "Recreating $nodeName..."
            Start-Process -FilePath java -NoNewWindow -Wait -ArgumentList "-jar $JenkinsCliJarFilePath","-s $JenkinsServerUrl","-auth $Username`:$ApiToken","create-node" -RedirectStandardInput "$nodeName.xml"
        }
        else 
        {
            Write-Host "Node $nodeName exists."   
        }

        If (!$KeepNodeXmlFiles)
        {
            Remove-Item -Path "$nodeName.xml" -Force
        }
    }
}

Next Step

The script I created can bulk create the “receivers” for the actual slaves to talk to the Jenkins master. I will write another PowerShell to script to automatically install and communicates with the Jenkins master.

Recap

Pull Requests, any suggestion or even forking is welcome. This should make Jenkins permanent node/slave administration a little easier.

Starting with Jenkins CLI

I previously wrote How to Install Jenkins Slave as Windows Service in this blog. It has been one of the most accessed articles on this site. Though the article worked for people who visited here, I thought of taking it to the next level. What if I come up with a way to easily install Jenkins Slave as Windows Service by running scripts? It would save so much time and effort without mistakes.

Before working on the whole script, I want to make sure Jenkins CLI works. Jenkins CLI is different from REST API of Jenkins and it needs some preliminary preparation.

Jenkins CLI

Jenkins CLI is available from Manage Jenkins -> Tools and Actions -> Jenkins CLI.

When I click Jenkins CLI, there is a list of commands available.

I’m going to try to see if help works for sanity check. Before running the command java -jar jenkins-cli.jar -s https://jenkins.linux-mint.local/ help , make sure to install Java and download jenkins-cli.jar from the Jenkins CLI page. When I ran it, I get the following error.

javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
<SNIP>
at hudson.cli.FullDuplexHttpStream.(FullDuplexHttpStream.java:73)
at hudson.cli.CLI.plainHttpConnection(CLI.java:361)
at hudson.cli.CLI._main(CLI.java:299)
at hudson.cli.CLI.main(CLI.java:96)
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439)
<SNIP>
… 20 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)
at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434)
… 25 more

This is because the SSL cert of the Jenkins master server is not trusted by Java. Let’s download the cert and get Java to trust it.

Download SSL Cert

Using openssl, we will download the SSL certificate as a file. Execute the following command.

openssl s_client -showcerts -connect jenkins.linux-mint.local:443 < /dev/null | openssl x509 -outform DER > jenkins.linux-mint.local.cer

Don’t mind some seemingly error message. Now you get a file jenkins.linux-mint.cer.

Trust the Cert

When you have Java on your system, you have a file called cacerts. Basically, you import the SSL cert you just downloaded into the cacerts file. Where is the file? Let’s find out. Execute the following command to locate cacerts.

sudo find /Library/Java -name cacerts

My system right now is a Mac and I happen to have the file at the following location.

/Library/Java/JavaVirtualMachines/jdk-14.0.2.jdk/Contents/Home/lib/security/cacerts

Execute the following command to import the SSL cert into cacerts. You wil be prompted if you really want to import it and type yes.

sudo keytool -import -v -trustcacerts -alias jenkins -file jenkins.linux-mint.local.cer -keystore /Library/Java/JavaVirtualMachines/jdk-14.0.2.jdk/Contents/Home/lib/security/cacerts -keypass changeit -storepass changeit

To check if it has been imported successfully, execute the following command. Enter the default password changeit if you haven’t changed.

keytool -list -keystore /Library/Java/JavaVirtualMachines/jdk-14.0.2.jdk/Contents/Home/lib/security/cacerts -alias jenkins

If you want to remove the certificate, you can execute the following command. (Do not execute it if you want to avoid the error I talked about earlier.)

keytool -delete -alias jenkins -keystore /Library/Java/JavaVirtualMachines/jdk-14.0.2.jdk/Contents/Home/lib/security/cacerts

Try Jenkins CLI

First, you need to generate API token for your user. Follow the steps below.

  1. Login to Jenkins master.
  2. Click on your username around the upper right corner.
  3. Click Configure.
  4. Click Add new Token button.
  5. Copy the generated token in clipboard.

Now try to execute the following command.

java -jar jenkins-cli.jar -s https://jenkins.linux-mint.local/ -auth [Your User]:[Your Token] help

Now you don’t get the error and you will see the list of available commands.

  add-job-to-view
    Adds jobs to view.
  build
    Builds a job, and optionally waits until its completion.
  cancel-quiet-down
    Cancel the effect of the "quiet-down" command.
   Resume using a node for performing builds, to cancel out the earlier "offline-node" command.
<SNIP>
  wait-node-online
    Wait for a node to become online.
  who-am-i
    Reports your credential and permissions.

Recap

I personally like using REST API of Jenkins better than Jenkins CLI but what I am planning to do may require Jenkins CLI. SSL protected Jenkins makes it harder to deal with it via its API but this makes it possible. Remember Java has its own keystore separate from the OS where it resides.

Revisiting Changing Jenkins Header Color

I previously wrote a blog article about changing Jenkins’ header color so that we can easily distinguish which instance of Jenkins you are dealing with. Some readers reported that the snippet to change the color was not working anymore after upgrading Jenkins. (Thank you for letting me know!)

Here are the complete steps again.

  1. Navigate to Manage Jenkins.
  2. Click Manage Plugins.
  3. Click Available tab.
  4. Install “Simple Theme” plugin by clicking Install without Restart button.
  5. Go to Manage Jenkins -> Configure System.
  6. Click Add under Theme section and select Extra CSS.
    jenkins simple theme
  7. Enter the following simple CSS to change the background color of the header. This is what has changed from the previous article.
    #header
    {
    background:red;
    }
  8. Save the change and when you refresh your browser, the header color has changed.

I don’t remember the exact HTML before but here is what the header looks like now and the CSS makes sense.

When I left my previous job, I didn’t have a Jenkins instance to test it out but now that I provisioned my own instance with Docker, I can test things with Jenkins again.

How to Make Jenkins Request User Data Available as an Environment Variable

Jenkins jobs serves many purposes. One of the important ones is to use them as self service type of jobs. Usually, you make them available to internal customers. When there are issues, the first thing you want to know may be who requested the job.

I know there is a plugin that makes BUILD_USER_ID available as a environment variable, but I will show you a little trick that doesn’t require another plugin.

First go to Configure section the job you are working on. Click Inject environment variables to the build process under Build Environment section.

Copy and paste the following code in Groovy Script section.

def map = ["REQUEST_USER": currentBuild.getCauses()[0]['userId']]
return map

When you run a build, the REQUEST_USER environment variable is available throughout your build. This is very convenient for logging purposes.  If you check the Environment Variables page for the build the key and the value are there.

Describing it in Job DSL, it looks like the following.

job('foo') {
    environmentVariables {
		loadFilesFromMaster(false)
		keepSystemVariables(true)
		keepBuildVariables(true)
		overrideBuildParameters(false)
        groovy('''def map = ["REQUEST_USER": currentBuild.getCauses()[0]['userId']]
return map''')
	}
}

 

How to Install Jenkins on AWS

I previously wrote an article on how to install Jenkins on Azure. I’ve also blogged about following related topics.

Now I’m going to try to write about Jenkins installation and configuration on AWS. I obviously don’t want to just reinvent the wheel or completely do things wrong, so I did my search and I found a documentation here. I’m going to refer to it and work on an instance actually on my personal AWS account.

Create EC2 (VM) Instance

Let’s create a VM instance to host the Jenkins master. I’m going to assume that you’ve already set up an AWS account.

With your browser, navigate to aws.com and click on EC2 under All services –> Compute.

Make sure to select Oregon as the region at the upper right area.

Click Launch Instance button.

I’m going to select “Amazon Linux 2 AMI (HVM), SSD Volume Type”.

In the next step, I’m going to select the VM template of t2 micro of vCPUs 1 and 1 GB memory. I think it’s enough to run an instance of Jenkins master but I would not select this configuration in actual enterprise environments for its sheer size to scale. Click Review and Launch button.

Click Launch button in the next step.

You are now prompted to select an existing key pair or create a new one. I am creating a new one for this one and naming it jenkins_master. Make sure to download it to your desktop for later use for your SSH session. Finally click Launch instances button.

SSH into the Created Instance

Let’s SSH into the created instance. First, navigate to EC2 -> Instances and select the instance you just created. And click Connect button.

By clicking Connect button, you will see the instruction on how to SSH into the instance like the following.

Open terminal and navigate to the directory where you store the .pem file and copy and paste the command in the Example: section.

ssh -i "jenkins_master.pem" ec2-user@yourhost.compute.amazonaws.com

Once the command is successful, you will see a screen like the following.

As you can see, Amazon Linux is an offspring of CentOS so run sudo yum -y update to update the packages.

Installing Jenkins Master

Now we can install Jenkins master. The latest version of Java supported by Jenkins is Java 11. I’m going to install OpenJDK 11 first.

$ sudo amazon-linux-extras install java-openjdk11

To check if Open JDK has been installed, execute the following command.

$ java --version

Now we are going to import Jenkins repo so that we can use yum to install Jenkins.

$ curl --silent --location http://pkg.jenkins-ci.org/redhat-stable/jenkins.repo | sudo tee /etc/yum.repos.d/jenkins.repo

$ sudo rpm --import https://jenkins-ci.org/redhat/jenkins-ci.org.key

Now execute the command to install Jenkins.

sudo yum install jenkins

Start Jenkins.

$ sudo systemctl start jenkins

Make sure that the Jenkins daemon starts by itself after reboot.

$ sudo systemctl enable jenkins

See if the Jenkins process is up and running by executing the following command.

You should see something like the following image.

$ ps aux | grep jenkins

–httpPort=8080 indicates that Jenkins master is listening to port 8080 with plain HTTP. I will review the configurations

Configuring Jenkins

We know that Jenkins exposes its UI on port 8080/tcp. I would like to use the default port of 80 for HTTP traffic for this instance of Jenkins. So I’m going to change the listening port by modifying the configuration file at /etc/sysconfig/jenkins

$ sudo vi /etc/sysconfig/jenkins

Now let’s make sure that JENKINS_PORT=”8080″ is present in the file. This means that you will be able to access the Jenkins UI via HTTP at the port 8080. Restart Jenkins master just in case.

$ sudo systemctl restart jenkins

Opening a Port

To access the running instance of Jenkins master on your browser, the port 80/tcp has to be open on AWS side. Here is how you can do it.

Navigate to EC2 dashboard and click Security Groups under NETWORK & SECURITY on the left menu.

Click Create Security Group button and you will see a popup like the image below. Click Add Rule button and select Custom TCP on Type column. Enter 8080 in the Port Range column. Click Create button.

Now we are going to assign this security group to the instance we created previously. Navigate to Instances and make sure that the target instance is checked. Click Actions button and select Networking –> Change Security Groups.

Select the security group that was just created and click Assign Security Groups button.

This opens the port 8080 to the target instance. Let’s try to access it!

There are two ways to access it. If you navigate to the EC2 Instances page and select the target instance, you will see the detailed information about the instance. You can see the public DNS and its IP address so enter an URL like below.

http://[your public DNS]:8080

When it’s successful, you should see a UI asking you to unlock Jenkins.

I have already documented the way to configure Jenkins here in my previous post, so please refer to it from here on.

Recap

We have learned how to spin up a VM and actually install Jenkins master on it on AWS. There are a lot more to get to the point where Jenkins master may be secured and usable on AWS. The principle is pretty much the same as the one on Azure, such as enabling SSL and getting slaves communicate with it via the secured communication. I will talk more about it later in my blog.

Getting Jenkins Jobs Recursively in Python via API

I’ve wanted to come up with a code that can get the whole list of Jenkins jobs recursively. I was able to write it but I’m not really liking it… View it in GitHub.

    def get_jobs(self, recursive=False):
        r = None
        if self.base_url.endswith("/"):
            api_url = self.base_url + "api/json"
        else:
            api_url = self.base_url + "/api/json"

        try:
            urllib3.disable_warnings()
            headers = {'Content-Type': 'application/json'}
            r = requests.get(url=api_url, headers=headers, auth=(self.username, self.api_token), verify=ssl.CERT_NONE)
            jenkins = json.loads(r.text)

            result = {}

            if recursive:
                folders = []
                while True:
                    self.process_result(jenkins['jobs'], folders, result)
                    if len(folders) == 0:
                        break

                    for index, folder in enumerate(folders):
                        r = requests.get(url=f"{folder}api/json", headers=headers,
                                         auth=(self.username, self.api_token), verify=ssl.CERT_NONE)
                        jenkins = json.loads(r.text)
                        self.process_result(jenkins['jobs'], folders, result)
                        folders.pop(index)
                    if len(folders) == 0:
                        break
            else:
                for job in jenkins['jobs']:
                    if job['_class'] != 'com.cloudbees.hudson.plugins.folder.Folder':
                        result[job['url']] = job['name']

            return result
        finally:
            if r is not None:
                r.close()

    def process_result(self, jobs, folders, result):
        for job in jobs:
            if job['_class'] == 'com.cloudbees.hudson.plugins.folder.Folder':
                folders.append(job['url'])
            else:
                result[job['url']] = job['name']

The reason I’m not really liking it because I’m repeating process_result twice in the code. At least I was successful in what I wanted to do and I didn’t use recursive function for recursive result but there has to be a better way to accomplish it. I’m going to dig more into this.

Edit: I realized that the reason why I have repeat the process_result function twice in the code is because Python doesn’t have support do…while. I could be wrong but I’m thinking that’s the core of the issue but if anyone out there can suggest a better way, please let me know. 🙂