How to Use awk

awk is one of the most used commands in bash environment. I can’t possibly cover everything about awk but I will write about an example that can be applied to many situations.

When I list files and directories in a directory, I execute…

ls -lah

That shows outputs the list like below.

total 296
drwxr-xr-x+ 50 hiriumi  staff   1.6K Aug 27 21:44 .
drwxr-xr-x   5 root     admin   160B Dec  5  2019 ..
-r--------   1 hiriumi  staff     7B Aug 18 22:06 .CFUserTextEncoding
-rw-r--r--@  1 hiriumi  staff    20K Aug 26 20:47 .DS_Store
drwx------  86 hiriumi  staff   2.7K Aug 26 16:57 .Trash
-rw-------   1 hiriumi  staff    36K Aug 29 17:44 .bash_history
-rw-r--r--   1 hiriumi  staff   788B Jul 19 17:31 .bash_profile
-rw-r--r--   1 hiriumi  staff   272B Mar 10 23:35 .bash_profile.backup
drwx------   3 hiriumi  staff    96B Apr 14 22:46 .cache
drwx------   5 hiriumi  staff   160B May 21 19:54 .config
drwx------   3 hiriumi  staff    96B Mar  7 13:40 .cups
drwxr-xr-x  14 hiriumi  staff   448B Mar  6 22:17 .dropbox
drwxr-xr-x  16 hiriumi  staff   512B Aug  1 14:25 .dvdcss
-rw-r--r--   1 hiriumi  staff    56B Mar  9 00:01 .gitconfig
drwxr-xr-x  15 hiriumi  staff   480B May 10 09:09 .iterm2
-rwxr-xr-x   1 hiriumi  staff    22K May 10 09:10 .iterm2_shell_integration.bash
drwxr-xr-x   6 hiriumi  staff   192B Mar  9 00:26 .kivy
<SNIP>

As you can see, if it’s a file, the line starts with “-” and if it’s a directory, it starts with “d”. Let’s just list the files using grep. I will show an example in awk after that.

ls -lah | grep '^-'

The following command outputs exactly the same list.

ls -lah | awk '/^-/ {print}'
-r--------   1 hiriumi  staff     7B Aug 18 22:06 .CFUserTextEncoding
-rw-r--r--@  1 hiriumi  staff    20K Aug 26 20:47 .DS_Store
-rw-------   1 hiriumi  staff    36K Aug 29 17:44 .bash_history
-rw-r--r--   1 hiriumi  staff   788B Jul 19 17:31 .bash_profile
-rw-r--r--   1 hiriumi  staff   272B Mar 10 23:35 .bash_profile.backup
-rw-r--r--   1 hiriumi  staff    56B Mar  9 00:01 .gitconfig
-rwxr-xr-x   1 hiriumi  staff    22K May 10 09:10 .iterm2_shell_integration.bash
-rw-------   1 hiriumi  staff    62B Jul 28 13:44 .lesshst
-rw-------   1 hiriumi  staff    29K Aug 27 21:44 .viminfo
-rw-r--r--   1 hiriumi  staff    10B Jun 29 21:40 .vimrc
-rw-r--r--   1 hiriumi  staff    32B May 22 20:29 .vuerc
-rw-------   1 hiriumi  staff    35B Mar  6 23:26 .zsh_history

What if you want to just output the file names? Here is what you can do. The command means print the 9th column separated by space.

ls -lah | awk '/^-/ {print $9}'
.CFUserTextEncoding
.DS_Store
.bash_history
.bash_profile
.bash_profile.backup
.gitconfig
.iterm2_shell_integration.bash
.lesshst
.viminfo
.vimrc
.vuerc
.zsh_history

The following command outputs the 1st and the 9th columns.

ls -lah | awk '/^-/ {print $1,$9}'
-r-------- .CFUserTextEncoding
-rw-r--r--@ .DS_Store
-rw------- .bash_history
-rw-r--r-- .bash_profile
-rw-r--r-- .bash_profile.backup
-rw-r--r-- .gitconfig
-rwxr-xr-x .iterm2_shell_integration.bash
-rw------- .lesshst
-rw------- .viminfo
-rw-r--r-- .vimrc
-rw-r--r-- .vuerc
-rw------- .zsh_history

What if you want a custom output for your documentation? I just want to list the file names with vertical bars. This can be used as a table in JIRA.

ls -lah | awk '/^-/ {print "|"$9"|"}'

Here is the sample output.

|.CFUserTextEncoding|
|.DS_Store|
|.bash_history|
|.bash_profile|
|.bash_profile.backup|
|.gitconfig|
|.iterm2_shell_integration.bash|
|.lesshst|
|.viminfo|
|.vimrc|
|.vuerc|
|.zsh_history|

Recap

awk is an old but very useful command on Linux/Mac. There is much more to this command so I will write in this blog as I come across more usages.

Bulk Create Jenkins Nodes

If permanent slaves are required and hundreds of them, it’s very manual and tedious tasks. I have created a PowerShell script to create Jenkins nodes in bulk. CloudBees’ version of Jenkins seems to have nice API to create nodes but open source/free version doesn’t seem to have one though Jenkins CLI could be utilized to accomplish it. Please follow my previous blog article to make sure your Jenkins CLI works before moving on with this article. Please note that this script does not install Jenkins slaves on separate machines but I am planning to create a script to do that in the future.

Prerequisites

  • Windows (The script I am going to introduce utilizes PowerShell, so Windows is the safest choice of OS.)
  • Java
  • Donwloaded jenkins-cli.jar

The PowerShell Script Execution

The PowerShell script that I wrote can generate nodes in bulk. It can be found here in GitHub.

For example, execute the script with the parameters like below.

.\Jenkins_BulkCreateNodes.ps1 -JenkinsServerUrl https://jenkins.linux-mint.local/ -Username myadminuser -ApiToken 1197adsfasdfadsfasdfasdfasdfasdf3a -NodeCount 100 -NodeNamePattern "winnode-###" -OverwriteNodeIfExists $True

The command above creates nodes in Jenkins master from winnode-001 to winnode-100. The red cross icon you see below only indicates that they are offline and no actual slaves is communicating with them yet.

You can change the naming pattern specifying -NodeNamePattern.

By specifying -OverwriteNodeIfExists parameter to $True, it can delete the node and recreate. It can be dangerous, so its default value is $False and it’s an optional parameter.

The PowerShell Code

I am pasting all the code I have written for this operation. But if you just want the file, please go to my GitHub repo. All the details of each parameters are documented in the code.

<#
    .SYNOPSIS
       Generates Jenkins nodes.
    .DESCRIPTION
        This script bulk generates Jenkins nodes on the master so that nodes will be ready for Jenkins slaves to talk to them. 
        It utilizes Jenkins CLI, so Java (https://java.com/en/download/) and downloaded jenkins-cli.jar file from Jenkins master is required.
    .PARAMETER JenkinsServerUrl
        Required. Base URL of the Jenkins master. e.g. https://myjenkins.com/
    .PARAMETER Username
        The username to use for this process.
    .PARAMETER ApiToken
        Generated API Token of the username.
    .PARAMETER NodeCount
        Specifies how many nodes to created.
    .PARAMETER NodeNamePattern
        Specify the pattern of node name here. If 20 nodes to be created and "foobar-###" is passed, foobar-001 to foobar-020 would be generated.
    .PARAMETER Description
        Optional. Specify the description of each node. Use this field to specify what kind of fleet you are creating.
    .PARAMETER RemoteRootDir
        Optional. Defaults to "C:\slave". Specify any directory where you expect slave bits to reside on each node.
    .PARAMETER Mode
        Optional. Defaults to NORMAL.
    .PARAMETER NumExecutors
        Optional. Defaults to 1.
    .PARAMETER Labels
        Optional. e.g. "WINDOWS 2019". Separate labels by space.
    .PARAMETER JenkinsCliJarFilePath
        Optional. When you download jenkins-cli.jar, the file is usually at ~/Download/jenkins-cli.jar. Downaload the file from https://your-jenkins-server/jnlpJars/jenkins-cli.jar
        There is a download link on https://your-jenkins-server/cli
    .PARAMETER OverwriteNodeIfExists
        Optional. If the node to be created already exists on the Jenkins master, it deletes it and recreates it if $True is passed. If $False, it is skipped.
    .PARAMETER KeepNodeXmlFiles
        Optional. Defaults to $False. If $True is passed, the XML files created for the nodes will be kept in the same directory as this script. This option could be used to reuse the XML files in different environments.
    
    
    
#>
[CmdletBinding()]
param 
(
    [Parameter(Mandatory=$True)] 
    [string]$JenkinsServerUrl,
    [Parameter(Mandatory=$True)]
    [string]$Username,
    [Parameter(Mandatory=$True)]
    [string]$ApiToken,
    [Parameter(Mandatory=$False)]
    [int]$NodeCount = 5,
    [Parameter(Mandatory=$False)]
    [string]$NodeNamePattern = "winnode-###",
    [Parameter(Mandatory=$False)]
    [string]$Description = "Bulk gen'ed node",
    [Parameter(Mandatory=$False)]
    [string]$RemoteRootDir = "C:\slave",
    [Parameter(Mandatory=$False)]
    [string]$Mode = "NORMAL",
    [Parameter(Mandatory=$False)]
    [int]$NumExecutors = 1,
    [Parameter(Mandatory=$False)]
    [string]$Labels = "",
    [Parameter(Mandatory=$False)]
    [string]$JenkinsCliJarFilePath = "$env:HOMEPATH\Downloads\jenkins-cli.jar",
    [Parameter(Mandatory=$False)]
    [bool]$OverwriteNodeIfExists = $False,
    [Parameter(Mandatory=$False)]
    [bool]$KeepNodeXmlFiles = $False
)

$digitCount = ($NodeNamePattern.ToCharArray() | Where-Object {$_ -eq '#'} | Measure-Object).Count

ForEach ($index In 1..$NodeCount)
{
    $number = "{0:D$digitCount}" -f $index
    $nodeName = $NodeNamePattern -replace "#{$digitCount}", $number 
    $xmlSettings = New-Object System.Xml.XmlWriterSettings
    $xmlSettings.Indent = $true
    $xmlSettings.Encoding = [System.Text.Encoding]::UTF8
    $xmlSettings.OmitXmlDeclaration = $True
    $xmlSettings.ConformanceLevel = "Document"

    $sw = New-Object System.IO.StringWriter
    $xw = [System.Xml.XmlWriter]::Create($sw, $xmlSettings)
    
    try 
    {
        $xw.WriteStartElement("slave")

        $xw.WriteElementString("name", $nodeName)
        $xw.WriteElementString("description", $Description)
        $xw.WriteElementString("remoteFS", $RemoteRootDir)
        $xw.WriteElementString("numExecutors", $NumExecutors)
        $xw.WriteElementString("mode", $Mode)

        $xw.WriteStartElement("retentionStrategy")
        $xw.WriteAttributeString("class", 'hudson.slaves.RetentionStrategy$Always')
        $xw.WriteEndElement() # retentionStrategy

        $xw.WriteStartElement("launcher")
        $xw.WriteAttributeString("class", "hudson.slaves.JNLPLauncher")

        $xw.WriteStartElement("workDirSettings")
        $xw.WriteElementString("disabled", "false")
        $xw.WriteElementString("internalDir", "remoting")
        $xw.WriteElementString("failIfWorkDirIsMissing", "false")
        $xw.WriteEndElement() # workDirSettings
        $xw.WriteElementString("websocket", "false")
        $xw.WriteEndElement() # launcher

        $xw.WriteElementString("label", $Labels)
        $xw.WriteElementString("nodeProperties", "")
        $xw.WriteEndElement() #slave
        $xw.WriteEndDocument()
        $xw.Flush()
    }
    catch [System.Exception]
    {
        Write-Host $_
        exit 1
    }
    finally 
    {
        $xw.Close()
    }

    # Write out the node XML to file to be used for standard input below.
    Set-Content -Path "$nodeName.xml" -Value $sw.ToString() -Force

    # Check the existence of the node.

    $processInfo = New-Object System.Diagnostics.ProcessStartInfo
    $processInfo.FileName = "java"
    $processInfo.RedirectStandardError = $True
    $processInfo.RedirectStandardOutput = $True
    $processInfo.RedirectStandardInput = $True
    $processInfo.Arguments = "-jar $JenkinsCliJarFilePath -s $JenkinsServerUrl -auth $Username`:$ApiToken get-node $nodeName"
    $processInfo.UseShellExecute = $False

    $process = New-Object System.Diagnostics.Process
    $process.StartInfo = $processInfo
    $process.Start() | Out-Null
    $process.WaitForExit()
    $stdo = $process.StandardOutput

    If ($process.ExitCode -ne 0)
    {
        Start-Process -FilePath java -NoNewWindow -Wait -ArgumentList "-jar $JenkinsCliJarFilePath","-s $JenkinsServerUrl","-auth $Username`:$ApiToken","create-node" -RedirectStandardInput "$nodeName.xml"
        Write-Host "$nodeName created ($(Get-Date))"
    }
    else 
    {
        If ($OverwriteNodeIfExists)
        {
            Write-Host "Deleting $nodeName..."
            Start-Process -FilePath java -NoNewWindow -Wait -ArgumentList "-jar $JenkinsCliJarFilePath","-s $JenkinsServerUrl","-auth $Username`:$ApiToken","delete-node $nodeName"

            Write-Host "Recreating $nodeName..."
            Start-Process -FilePath java -NoNewWindow -Wait -ArgumentList "-jar $JenkinsCliJarFilePath","-s $JenkinsServerUrl","-auth $Username`:$ApiToken","create-node" -RedirectStandardInput "$nodeName.xml"
        }
        else 
        {
            Write-Host "Node $nodeName exists."   
        }

        If (!$KeepNodeXmlFiles)
        {
            Remove-Item -Path "$nodeName.xml" -Force
        }
    }
}

Next Step

The script I created can bulk create the “receivers” for the actual slaves to talk to the Jenkins master. I will write another PowerShell to script to automatically install and communicates with the Jenkins master.

Recap

Pull Requests, any suggestion or even forking is welcome. This should make Jenkins permanent node/slave administration a little easier.

Starting with Jenkins CLI

I previously wrote How to Install Jenkins Slave as Windows Service in this blog. It has been one of the most accessed articles on this site. Though the article worked for people who visited here, I thought of taking it to the next level. What if I come up with a way to easily install Jenkins Slave as Windows Service by running scripts? It would save so much time and effort without mistakes.

Before working on the whole script, I want to make sure Jenkins CLI works. Jenkins CLI is different from REST API of Jenkins and it needs some preliminary preparation.

Jenkins CLI

Jenkins CLI is available from Manage Jenkins -> Tools and Actions -> Jenkins CLI.

When I click Jenkins CLI, there is a list of commands available.

I’m going to try to see if help works for sanity check. Before running the command java -jar jenkins-cli.jar -s https://jenkins.linux-mint.local/ help , make sure to install Java and download jenkins-cli.jar from the Jenkins CLI page. When I ran it, I get the following error.

javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
<SNIP>
at hudson.cli.FullDuplexHttpStream.(FullDuplexHttpStream.java:73)
at hudson.cli.CLI.plainHttpConnection(CLI.java:361)
at hudson.cli.CLI._main(CLI.java:299)
at hudson.cli.CLI.main(CLI.java:96)
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439)
<SNIP>
… 20 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)
at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434)
… 25 more

This is because the SSL cert of the Jenkins master server is not trusted by Java. Let’s download the cert and get Java to trust it.

Download SSL Cert

Using openssl, we will download the SSL certificate as a file. Execute the following command.

openssl s_client -showcerts -connect jenkins.linux-mint.local:443 < /dev/null | openssl x509 -outform DER > jenkins.linux-mint.local.cer

Don’t mind some seemingly error message. Now you get a file jenkins.linux-mint.cer.

Trust the Cert

When you have Java on your system, you have a file called cacerts. Basically, you import the SSL cert you just downloaded into the cacerts file. Where is the file? Let’s find out. Execute the following command to locate cacerts.

sudo find /Library/Java -name cacerts

My system right now is a Mac and I happen to have the file at the following location.

/Library/Java/JavaVirtualMachines/jdk-14.0.2.jdk/Contents/Home/lib/security/cacerts

Execute the following command to import the SSL cert into cacerts. You wil be prompted if you really want to import it and type yes.

sudo keytool -import -v -trustcacerts -alias jenkins -file jenkins.linux-mint.local.cer -keystore /Library/Java/JavaVirtualMachines/jdk-14.0.2.jdk/Contents/Home/lib/security/cacerts -keypass changeit -storepass changeit

To check if it has been imported successfully, execute the following command. Enter the default password changeit if you haven’t changed.

keytool -list -keystore /Library/Java/JavaVirtualMachines/jdk-14.0.2.jdk/Contents/Home/lib/security/cacerts -alias jenkins

If you want to remove the certificate, you can execute the following command. (Do not execute it if you want to avoid the error I talked about earlier.)

keytool -delete -alias jenkins -keystore /Library/Java/JavaVirtualMachines/jdk-14.0.2.jdk/Contents/Home/lib/security/cacerts

Try Jenkins CLI

First, you need to generate API token for your user. Follow the steps below.

  1. Login to Jenkins master.
  2. Click on your username around the upper right corner.
  3. Click Configure.
  4. Click Add new Token button.
  5. Copy the generated token in clipboard.

Now try to execute the following command.

java -jar jenkins-cli.jar -s https://jenkins.linux-mint.local/ -auth [Your User]:[Your Token] help

Now you don’t get the error and you will see the list of available commands.

  add-job-to-view
    Adds jobs to view.
  build
    Builds a job, and optionally waits until its completion.
  cancel-quiet-down
    Cancel the effect of the "quiet-down" command.
   Resume using a node for performing builds, to cancel out the earlier "offline-node" command.
<SNIP>
  wait-node-online
    Wait for a node to become online.
  who-am-i
    Reports your credential and permissions.

Recap

I personally like using REST API of Jenkins better than Jenkins CLI but what I am planning to do may require Jenkins CLI. SSL protected Jenkins makes it harder to deal with it via its API but this makes it possible. Remember Java has its own keystore separate from the OS where it resides.

How to Merge JSON Array

I struggled to find out a way to merge JSON arrays, so to save you time, I will show you an example here.

jq

I’m sure there are several ways to accomplish it with programming languages, I will use jq. Just run brew install jq in your terminal if you use Mac.

Sample Data

fruits1.json

{
    "data":[
      {"name":"banana"},
      {"name":"strawberry"},
      {"name":"orange"}
    ]
}

fruits2.json

{
    "data":[
      {"name":"cherry"},
      {"name":"blueberry"},
      {"name":"apple"}
    ]
}

Merging

Execute the following jq command to merge the 2 arrays in 2 different JSON data.

 jq -s '{ data: map(.data[]) }' fruits1.json fruits2.json

Here is the result.

{
  "data": [
    {
      "name": "banana"
    },
    {
      "name": "strawberry"
    },
    {
      "name": "orange"
    },
    {
      "name": "cherry"
    },
    {
      "name": "blueberry"
    },
    {
      "name": "apple"
    }
  ]
}

If you just want a compact version of the same JSON data.

jq -sc '{ data: map(.data[]) }' fruits1.json fruits2.json

Here is the result.

{"data":[{"name":"banana"},{"name":"strawberry"},{"name":"orange"},{"name":"cherry"},{"name":"blueberry"},{"name":"apple"}]}

Though I don’t like jq, I had to accomplish it in a bash script I was working on. I could not contain my joy when I got it. 🙂

Revisiting Changing Jenkins Header Color

I previously wrote a blog article about changing Jenkins’ header color so that we can easily distinguish which instance of Jenkins you are dealing with. Some readers reported that the snippet to change the color was not working anymore after upgrading Jenkins. (Thank you for letting me know!)

Here are the complete steps again.

  1. Navigate to Manage Jenkins.
  2. Click Manage Plugins.
  3. Click Available tab.
  4. Install “Simple Theme” plugin by clicking Install without Restart button.
  5. Go to Manage Jenkins -> Configure System.
  6. Click Add under Theme section and select Extra CSS.
    jenkins simple theme
  7. Enter the following simple CSS to change the background color of the header. This is what has changed from the previous article.
    #header
    {
    background:red;
    }
  8. Save the change and when you refresh your browser, the header color has changed.

I don’t remember the exact HTML before but here is what the header looks like now and the CSS makes sense.

When I left my previous job, I didn’t have a Jenkins instance to test it out but now that I provisioned my own instance with Docker, I can test things with Jenkins again.

Summary of sed

I sometimes visit the basic commands to refresh my memory. You work on so many different things in your real work, so it’s not possible to remember all the details of each command. I’ll go over sed in this blog article.

The description of sed in its man page says the following.

The sed utility reads the specified files, or the standard input if no files are specified, modifying the input as specified by a list of commands. The input is then written to the standard output.

The syntax looks like this.

sed [option] [script] [target file]

There are so many options and usages but I will just summarize the ones that are used often.

Sample Text File

Here is the content of sample.txt

Hello
Heeeeello
Helloooooo
foobar1
foobar2
foobar3

Print Certain Lines to Standard Output

sed -n 1,3p sample.txt 

The command above prints from the first line to the 3rd line in sample.txt file. The output looks like the following.

Hello
Heeeeello
Helloooooo

It is possible to list the lines that match regular expression.

sed -n /^He.*llo/p sample.txt

The result looks like the following.

Hello
Heeeeello
Helloooooo

Remove Lines

sed 1,3d sample.txt

The command above removes the lines from 1 to 3 and outputs the rest of the text to the standard output. Please note that the command will not remove those lines from the file itself but it just outputs the result in the standard output. The sample output shows like the following.

foobar1
foobar2
foobar3

It’s also possible to remove lines using regular expression. Please see the example below.

sed /^Hello/d sample.txt

The command above removes lines that start with “Hello”. The output will show like this.

Heeeeello
foobar1
foobar2
foobar3

Replace Texts

sed can replace certain text with something else based on regular expression like the following.

sed s/He.*llo/boo/g sample.txt

The result looks like the following.

boo
boo
booooooo
foobar1
foobar2
foobar3

How to Provision Jenkins on NGINX Reverse Proxy on Docker Container

I’ve wanted to provision Jenkins on NGINX reverse proxy for a long time. I would hit walls here and there, but I finally got things sorted out at least to the point where I can hit Jenkins master on NGINX reverse proxy on my browser both running on Docker container.

There is so much to cover, so I think this blog post is going to be quite long.

First of all, I need to make clear what I want to accomplish.

I want to have multiple instances of Jenkins hosted on a single Docker host. Depending on the host name the HTTP request came, NGINX reverse proxy routes the traffic to the appropriate Jenkins instance.

The diagram below represents the logical architecture.

Here is the list of things that will be covered in this blog entry.

  • Docker Network
  • Jenkins on Docker
  • NGINX on Docker as Reverse Proxy
  • Putting NGINX and Jenkins Together

Docker Network

As you can see, there are 3 containers in the diagram above. NGINX reverse proxy routes traffic to Jenkins instances. It means that NGINX needs to be able to communicate with Jenkins instance. The containers in the diagram exist within Docker network.

By default, Docker containers use bridge network meaning that Docker uses its own network though it is configurable. If you execute ip addr on your terminal, you’d see network interfaces like this.

4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:67:fe:f4:d5 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
103: br-1418967f1f10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 02:42:63:60:8f:d1 brd ff:ff:ff:ff:ff:ff
    inet 172.25.0.1/16 brd 172.25.255.255 scope global br-1418967f1f10
       valid_lft forever preferred_lft forever
    inet6 fe80::42:63ff:fe60:8fd1/64 scope link
       valid_lft forever preferred_lft forever

When you spin up a Docker container, each container has its own IP address within the Docker network. I provisioned a Jenkins container and I can check its IP address with the steps below.

  1. Check the container ID.
    docker ps -a
  2. Run the following command to get the IP address.
    docker inspect [container ID] | grep IPAddress
  3. I got 172.25.0.2 as the IP address of the container. This means that it’s within the range of the CIDR 172.25.0.1/16 listed in the br-1418967f1f10 bridge network above. Based on the CIDR, the network can host up to 65,536 containers which is way more than enough for normal operations.

Why is this important in this article. It’s because I am planning to provision NGINX as a Docker container and I want the NGINX container to communicate with Jenkins container within the Docker network without using the host network. It will increase the network security level to the next level. It is important to understand the network aspect of containerization.

Jenkins on Docker

First of all, I need to provision a Jenkins container. Here is the initial Docker Compose file (YAML) I created.

version: '3.8'
services:
    jenkins:
        image: jenkins/jenkins:lts
        container_name: jenkins
        user: '1000'
        volumes:
            - ./jenkins_home:/var/jenkins_home
        ports:
            - '8080:8080'
            - '2000:2000'
        restart: unless-stopped
        environment:
            - JENKINS_SLAVE_AGENT_PORT=2000
            - JENKINS_JAVA_OPTIONS=-Djava.awt.headless=true
            - JENKINS_LOG=/var/jenkins_home/logs/jenkins.log

Please note that you should run mkdir jenkins_home (create the directory) beforehand to make sure you won’t get an error because as you can see it in docker-compose.yaml, /var/jenkins_home in the container is mapped to the host’s ./jenkins_home directory where all the data is persisted.

If you save this file as docker-compose.yaml and run docker-compose up -d and then access it on your browser http://foobar:8080/, you should see the initial set up UI. If you want to stop the container, you can just run docker-compose down.

It would be operational enough to use the Jenkins container as the way it is. However, it’s limited in a sense that if you want to have another instance of Jenkins on the same machine, you would have to use another port like http://foobar:8081 and http://foobar:8082 if you need another instance. It would mean you would have to open more ports as you have more instances of Jenkins on the same Docker host. Also, it’s kind of ugly. By having NGINX reverse proxy in place in front of the Jenkins instances on the Docker host, it solves the problem.

Moreover, NGINX can be the front man to service SSL communication. It’s the best practice nowadays that even internal web applications are protected by encrypted communication by SSL. You may want to have URLs that looks like the following.

https://jenkins1.foobar and https://jenkins2.foobar

This way the URL is easier to remember and also protected by SSL communication.

NGINX on Docker as Reverse Proxy

Here is the NGINX’s docker-compose.yaml file that I’ve got.

version: '3.8'
services:
    reverse:
        image: nginx:latest
        volumes:
            - ./data/nginx.conf:/etc/nginx/nginx.conf
            - ./data/conf.d:/etc/nginx/conf.d
        ports:
            - "80:80"
            - "443:443"
        networks:
            proxynet:
        restart: always
networks:
    proxynet:
        name: custom_network

For the NGINX container, I am mapping the /etc/nginx/nginx.conf to the local ./data/nginx.conf file and /etc/nginx/conf.d directory to ./data/conf.d directory. Make sure that you have ./data and ./data/conf.d directories before you run docker-compose up -d.

I am also creating networks -> proxynet -> custom_network here in docker-compose.yaml because I want to have the Docker network where NGINX and Jenkins can communicate with each other.

Putting NGINX and Jenkins Containers Together

Here is the docker-compose.yaml file that can spin up NGINX and Jenkins within the same subnet within Docker. Notice that I commented out - 8080:8080 line. It means I stopped exposing the port 8080 of the Jenkins container to the host side because it is not necessary anymore. NGINX and Jenkins will communicate with each other within the Docker network subnet.

ersion: '3.8'
services:
    jenkins:
        image: jenkins/jenkins:lts
        container_name: jenkins
        user: '1000'
        volumes:
            - ./jenkins_home:/var/jenkins_home
        ports:
            #- '8080:8080'
            - '2000:2000'
        networks:
            proxynet:
        restart: unless-stopped
        environment:
            - JENKINS_SLAVE_AGENT_PORT=2000
            - JENKINS_JAVA_OPTIONS=-Djava.awt.headless=true
            - JENKINS_LOG=/var/jenkins_home/logs/jenkins.log
        #command: --enable-future-java
    reverse:
        image: nginx:latest
        volumes:
            - ./data/nginx.conf:/etc/nginx/nginx.conf
            - ./data/conf.d:/etc/nginx/conf.d
        ports:
            - "80:80"
            - "443:443"
        networks:
            proxynet:
        restart: always
networks:
    proxynet:
        name: custom_network

Also make sure you have the nginx.conf file under ./data directory.

user  nginx;
worker_processes  2;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;

events {
   worker_connections  1024;
   use epoll;
   accept_mutex off;
}

http {
   include       /etc/nginx/mime.types;
   proxy_set_header X-Real-IP $remote_addr;
   proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

   default_type  application/octet-stream;

   log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                     '$status $body_bytes_sent "$http_referer" '
                     '"$http_user_agent" "$http_x_forwarded_for"';

   access_log  /var/log/nginx/access.log  main;

   sendfile        on;
   #tcp_nopush     on;

   keepalive_timeout  65;

   client_max_body_size 300m;
   client_body_buffer_size 128k;

   gzip  on;
   gzip_http_version 1.0;
   gzip_comp_level 6;
   gzip_min_length 0;
   gzip_buffers 16 8k;
   gzip_proxied any;
   gzip_types text/plain text/css text/xml text/javascript application/xml application/xml+rss application/javascript application/json;
   gzip_disable "MSIE [1-6]\.";
   gzip_vary on;

   server {
       listen       80 default_server;
       listen       [::]:80 default_server;
       server_name  _;
       sendfile        on;
       #tcp_nopush     on;

       keepalive_timeout  65;

      client_max_body_size 300m;
      client_body_buffer_size 128k;

      gzip  on;
      gzip_http_version 1.0;
      gzip_comp_level 6;
      gzip_min_length 0;
      gzip_buffers 16 8k;
      gzip_proxied any;
      gzip_types text/plain text/css text/xml text/javascript application/xml application/xml+rss application/javascript application/json;
      gzip_disable "MSIE [1-6]\.";
      gzip_vary on;
   }
    include /etc/nginx/conf.d/*.conf;
}

Also you need to have the following ssl.conf file under ./data/conf.d directory file. Notice that this file does 2 important things. One is to enable SSL communication between NGINX and the client side. If you wonder how to create SSL certificate, I have written how to just do it in my previous blog post. The second thing is to route the HTTPS traffic to the server jenkins:8080. This communication with done within the Docker network subnet, so no traffic goes out to the host’s network. You can also see that the SSL certificate files are under conf.d/ssl directory.

upstream jenkins_upstream {
    server jenkins:8080;
}

server {
    server_name jenkins.linux-mint.local;
    listen 443 ssl;
    ssl_certificate /etc/nginx/conf.d/ssl/jenkins.linux-mint.local.crt;
    ssl_certificate_key /etc/nginx/conf.d/ssl/jenkins.linux-mint.local.key;

    location / {
        proxy_set_header        Host $host:$server_port;
        proxy_set_header        X-Real-IP $remote_addr;
        proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header        X-Forwarded-Proto $scheme;
        resolver 127.0.0.11;
        #proxy_redirect http:// https://;
        proxy_pass http://jenkins_upstream;
        # Required for new HTTP-based CLI
        proxy_http_version 1.1;
        proxy_request_buffering off;
        proxy_buffering off; # Required for HTTP-based CLI to work over SSL
        # workaround for https://issues.jenkins-ci.org/browse/JENKINS-45651
        add_header 'X-SSH-Endpoint' 'jenkins.linux-mint.local:50022' always;
    }
}

Once it’s all done, just execute the command docker-compose up -d in the directory where you have docker-compose.yaml. If it’s successful and you access the URL like https://jenkins.linux-mint.local, you should see the initial screen where you can start to manage the instance of Jenkins.

If you execute cat jenkins_home/secrets/initialAdminPassword, you should see the initial password to enter on the screen. And you just have to click Continue button, you are on to setting up Jenkins.

Recap

I have covered the steps to spin up Jenkins behind NGINX as a reverse proxy. Based on my experience, this is the most flexible and safest way to provision Jenkins. There are steps to provision agents (slaves) for it. I will cover the topic some other time.