Investing Time and Effort Upfront

One of the pitfalls that software engineers easily fall into is a lack of investment of time and effort upfront. We often make excuses like “We don’t have time.”

What do we need to think about before working on software? What does investing time and effort mean in software engineering? These just pop in my head. I’m sure there is a lot more…

  • Software Architecture
    • Maintenability
    • Expandability
    • Portability
  • Source Control Structure
    • Branching
    • Promotion Strategy
  • Continuous Integration
  • Continuous Delivery
  • Pipeline
  • Binary Storage (such as Artifactory)
  • Automated Testing (part of CI/CD)
  • Versioning
  • Test Environment
  • Debugging
  • Logging
  • Alerting System
  • Network Security

If you don’t plan and just go ahead and code, the wrong things remain there for years because “we just don’t have time.” No, we HAVE to make time to do things right and you (and I) are responsible for persuading the stakeholders that you are spending some extra time upfront. In the end we get to save so much time, energy and money.

It’s never perfect, but make sure to think about your software or even some scripts you want to write for automation purposes at many different levels upfront before you even start to write code and your software will be successful for a long time.

How to Install Jenkins Slave as Windows Service

If you are here through a search engine, I wrote an updated version of this article just recently.

If you are running Jenkins master on Azure, it automatically provisions Windows VM host but it’s not permanent. What if you want a permanent Windows slave? What if you want to configure a slave on an existing Windows machie? In this example, I am using my own Windows 10 desktop environment (not a good practice however) to make it a permanent Jenkins slave for this article.

I think the best way to run Jenkins slave on Windows is to install it as a Windows service. It can start automatically as Windows starts and we could programmatically check the status, start, stop and restart.

I found it quite hard to find step-by-step information on how to do it and this documentation is really old and it doesn’t work any more with much tighter security on Windows, so I’m going to summarize the right way here.

Creating a Node

  1. Logon to Jenkins with an administrative account.
  2. Go to Manage Jenkins.
  3. Click Manage Nodes.
    manage nodes
  4. Click New Node.
  5. Enter node name like “win-bld01” and click Permanent Agent and then click OK.
    enter node name
  6. Enter a path in Remote root directory of your choice and choose “Launch agent via Java Web Start” in Launch method.
    Launch agent via Java Web Start
  7. Click OK to save the setting.

Download and Configure Jenkins Slave Windows Service

  1. Download the latest Windows Service wrapper executable from http://repo.jenkins-ci.org/releases/com/sun/winsw/winsw/
    Download the latest Windows Service wrapper executable
    I’m downloading winsw-2.2.0-net4.exe in this example.
  2. Place it to the Custom WorkDir path and rename it to jenkins-slave.exe. I am using a directory C:\Users\[My User Account]\jenkins in this example.
  3. Create jenkins-slave.xml in the same directory with the following XML. Make sure to change the values in the XML according to your environment.
<service>
  <id>JenkinsSlave</id>
  <name>Jenkins agent</name>
  <description>This service runs an agent for Jenkins automation server.</description>
  <executable>C:\Users\amaterasu48\apps\java\bin\java.exe</executable>
  <arguments>-Xrs -jar &quot;%BASE%\slave.jar&quot; -jnlpUrl https://jks.westus.cloudapp.azure.com/computer/win-bld01/slave-agent.jnlp -secret e54b3a6b958f1e400e9809808sdfs098sdf0s9bcd4938d2fca496fbbea3e889 -workDir=C:\Users\amaterasu48\jenkins</arguments>
  <logmode>rotate</logmode>
  <onfailure action="restart">
    <download from="https://jks.westus.cloudapp.azure.com/jnlpJars/slave.jar" to="%BASE%\slave.jar">
      <extensions>
        <extension classname="winsw.Plugins.RunawayProcessKiller.RunawayProcessKillerExtension" enabled="true" id="killOnStartup">
          <pidfile>%BASE%\jenkins_agent.pid</pidfile>
          <stoptimeout>5000</stoptimeout>
          <stopparentfirst>false</stopparentfirst>
        </extension>
      </extensions>
    </download>
  </onfailure>
</service>

For more options, please refer to the official documentation.

Also make sure to create jenkins-slave.exe.config file with the following XML cotent to prevent the executable from running on the earlier version of the .NET Framework.

<configuration>
	<startup>
		<supportedruntime version="v4.0"/>
	</startup>
</configuration>

Install Jenkins Slave

  1. Open Windows command line as Administrator.
  2. Navigate to the directory where you have jenkins-slave.exe e.g. cd C:\Users\amaterasu48\jenkins
  3. Install the service by executing the following command.
    jenkins-slave.exe install
  4. Once it’s successful, you will see a message like this.
    2019-05-22 23:17:14,486 INFO - Installing the service with id 'JenkinsSlave'
  5. Right click Windows start button and select Run. Enter services.msc to open Windows Service management console. You will see the Jenkins agent Windows service we just installed.
    Windows Service management console
  6. Right click it and Start it.

Recap

It’s not a few simple steps but as long as this can be automated, it should not be so painful. Also, if your Jenkins slave has to talk to other resources within your network, it’s wiser to run the Windows service as a domain user that has permission to other servers.

Jenkins Slave Failed to Validate a Server Certificate

If you have enabled SSL on Jenkins, you probably get the following error when you start to have your slave start to communicate with the Jenkins master.

Exception in thread "main" java.io.IOException: Failed to validate a server certificate. If you are using a self-signed certificate, you can use the -noCertificateCheck option to bypass this check.

It means that the java process does not trust the SSL certificate and the application is not going to continue because some weird guy could be eavesdropping the communication between the slave and the Jenkins master. Let’s see how we can have the slave side trust the SSL certificate.

First of all, let’s download the publicly exposed SSL certificate using Chrome browser.

  1. Open Jenkins master using Chrome browser.
  2. Click the key icon and select Certificate (Valid).
  3. Save the .cer file in Details tab. I have saved the file to my desktop.
  4. Open your command line as an administrator on Windows.
  5. Execute the following command to save the SSL certificate to trust.
    openssl s_client -showcerts -connect jks.westus.cloudapp.azure.com:443 < /dev/null | openssl x509 -outform DER > azure-jenkins.com.cer
  6. Execute the command to trust the certificate.
    keytool -trustcacerts -keystore "%JAVA_HOME%\lib\security\cacerts" -storepass changeit -alias jks.westus.cloudapp.azure -import -file "C:\Users\amaterasu48\Desktop\azure-jenkins.cer"
  7. The path C:\Users\amaterasu48\Desktop\azure-jenkins.cer should be replaced with wherever you have the .cer file saved on your slave machine.
  8. Once the keytool command is successful, the Java process on the slave machine trusts the SSL cert on Jenkins master and it’s ready to communicate with the Jenkins master.

My batch file for the slave to communicate with the Jenkins master looks like this.

java -jar agent.jar -jnlpUrl https://jks.westus.cloudapp.azure.com/computer/win-bld01/slave-agent.jnlp -secret e54b3aasdfasdfasdfasdfsdfasdfasdfasdfsadfasdfbea3e889 -workDir "C:\Users\amaterasu48\workspaces"

You can get the command by adding a node on Jenkins master and saving it. I am using JNLP to communicate with the Jenkins master.

If you save the command and execute it, you will successfully start the communication between the slave machine and Jenkins master. Part of the log is in Japanese because I use a Windows machine in Japanese but you get the idea. You run the command to communicate with the Jenkins master successfully now.

  Agent address: jks.westus.cloudapp.azure.com
  Agent port:    5378
  Identity:      97:d8:c1:6f:31:e9:4e:07:58:bd:82:32:db:1b:7f:56
5月 20, 2019 11:50:44 午後 hudson.remoting.jnlp.Main$CuiListener status
情報: Handshaking
5月 20, 2019 11:50:44 午後 hudson.remoting.jnlp.Main$CuiListener status
情報: Connecting to jks.westus.cloudapp.azure.com:5378
5月 20, 2019 11:50:44 午後 hudson.remoting.jnlp.Main$CuiListener status
情報: Trying protocol: JNLP4-connect
5月 20, 2019 11:50:45 午後 hudson.remoting.jnlp.Main$CuiListener status
情報: Remote identity confirmed: 97:d8:c1:6f:31:e9:4e:07:58:bd:82:32:db:1b:7f:56
5月 20, 2019 11:50:45 午後 hudson.remoting.jnlp.Main$CuiListener status
情報: Connected
5月 20, 2019 11:50:48 午後 org.jenkinsci.remoting.util.AnonymousClassWarnings warn
警告: Attempt to (de-)serialize anonymous class org.jenkinsci.plugins.envinject.EnvInjectComputerListener$2; see: https://j
enkins.io/redirect/serialization-of-anonymous-classes/
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.jenkinsci.plugins.envinject.service.EnvInjectMasterEnvVarsSetter to field java.lang.reflect.Field.modifiers
WARNING: Please consider reporting this to the maintainers of org.jenkinsci.plugins.envinject.service.EnvInjectMasterEnvVarsSetter
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

To recap, the slave Java process must trust the Jenkins master’s SSL certificate to establish secure SSL connection.

Jenkins Pipeline Specifying Labels and Parallel Processing

In Jenkins Pipeline, it is possible to specify which slave to run each stage on. Before we go into the code, let’s see how we can set up a pipeline job.

  1. Create a pipeline job.
  2. Select Pipeline script from SCM.
  3. Set the Repository URL.
  4. Set Credentials to get the source code. In this example, I am using my credential for GitHub.

We will write all the pipeline code in Jenkinsfile file.

What I want to be able to do is to have stages and steps and I want to be able to specify which slaves to run them on. Here is how I can run Windows and Linux steps separately.

pipeline
{
    agent none
    stages
    {
        stage("win-step1")
        {
            agent
            {
                label "win"
            }
            steps
            {
                echo "Windows long running process..."
                sleep 10
            }
        }

        stage("linux-step1")
        {
            agent
            {
                label "linux"
            }
            steps
            {
                echo "Linux long running process..."
                sleep 10
            }
        }
    }
}

Right after pipeline declarative, agent is set to none. This is because we are specifying agent in each stage. Windows steps would run on slaves with the label “win” and Linux steps would run on slaves with the label “linux” obviously. Note that each step takes 10 seconds (simulating long running process that takes 10 seconds), so when I run the pipeline, it takes about 20 seconds. When you look at the result in Blue Ocean plugin, you can see the steps were executed sequentially.

If the second stage depends on the first one, this is totally fine, but what if stage 1 and stage 2 can be executed in parallel? What if each stage takes an hour? It would take 2 hours for entire pipeline to finish but if we can execute them in parallel, it would finish in just half the time. Let’s see how we can do it.

pipeline
{
    agent none
    stages
    {
        stage("Run builds on Windows and Linux")
        {
            parallel
            {
                stage("win-step1")
                {
                    agent
                    {
                        label "win"
                    }
                    steps
                    {
                        echo "Windows long running process..."
                        sleep 10
                    }
                }

                stage("linux-step1")
                {
                    agent
                    {
                        label "linux"
                    }
                    steps
                    {
                        echo "Linux long running process..."
                        sleep 10
                    }
                }
            }
        }
    }
}

Take a look at the result in Blue Ocean. You see the stages (steps) were executed in parallel and the execution time was reduced in half! This is beautiful!

If you surround the 2 stages with parallel and have another stage before parallel, Jenkins pipeline sends job requests to appropriate slaves and execute them in parallel. This technique will definitely save so much time. Pipeline should be designed with multiple slaves (agents) in mind for parallel processing.

Jenkins Pipeline

I have seen and used Jenkins Pipeline in the last few years. What I realized is that we should not overuse or not misuse Jenkins pipeline.

When you click on New Item in Jenkins, we see a bunch of job types. If you have more plugins, there could be a lot more. I personally like MultiJob plugin.

Although pipeline can do so many things and it’s tempting to just use pipeline for everything, but I think freestyle project should be used for most of the automation job you may have. Pipeline should be used to connect the dots as the name indicates what it is for.

I have seen pipeline kind of being misused in places and it made the job very complex. Pipeline is there to not only connect the dots of jobs, but to connect dev, QA and ops to carry the code all the way through the departments to production systems. Pipeline is not there just to run bunch of jobs sequentially and that’s what’s I’m seeing people misusing pipeline.

I believe it’s beneficial for any company to move code developed by devs as quickly as possible to production. So Jenkins jobs should be designed in the ways that can run in parallel on multiple slaves. Pipeline is there to support all that.

I could be wrong about what I’m thinking right now and I’m willing to accept it if I’m proven wrong but that’s what I believe right now.

Jenkins Slaves on Azure

The beauty of having Jenkins on Azure is that you really don’t have to manually configure Jenkins slaves unless you have complex requirements. By creating a job with a label “linux” or “win”, Azure automatically creates a slave for you and executes the job. Let’s see a very simple case.

  1. Click New Item.
  2. Create a Freestyle project with the name “linux-test”. Click OK button at the bottom of the screen to proceed.
  3. Click New Item.
  4. Create a Freestyle project with the name “linux-test”. Click OK button at the bottom of the screen to proceed.
  5. In the configuration screen, enter “linux” in Restrict where this project can be run.
  6. Select “Execute shell” task from Build section.
  7. Enter the following script in Command.
    echo "Hello from Azure Linux Jenkins slave!"
    cat /etc/os-release

  8. Click Save to commit the change.
  9. Click “Build Now” button .
  10. When you do, you will see a message in the Build History section saying that “(pending–‘Jenkins’ doesn’t have label ‘linux’)”. Don’t get deceived by the message. Actually, Jenkins and Azure are working very hard in the background to provision a Linux VM slave. You will start to see something like the following image after a little while.
  11. After provisioning is done, the job is executed successfully. If you take a look at the console log, you will see something like the following.
Started by user hogehogehoge
Building remotely on linux-agentfbcb70 (linux) in workspace /home/agentadmin/workspace/linux-test
[linux-test] $ /bin/sh -xe /tmp/jenkins1987715488800862551.sh
+ echo Hello from Azure Linux Jenkins slave!
Hello from Azure Linux Jenkins slave!
+ cat /etc/os-release
NAME="Ubuntu"
VERSION="16.04.6 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.6 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial
Finished: SUCCESS

The agents provisioned by the Jenkins job stay around for a while and accept requests as needed but when there is no job to do for a while, the system automatically decommissions the agents.

How is this possible? How does Jenkins know how to provision Linux and Windows agent? Let’s take a look at Manage Jenkins section.

However you want to provision the VM agents, it’s configurable on this screen.

By default, Jenkins on Azure is configured to provision Linux and Windows automatically. This means that the more agents it needs, the more agents it automatically adds in the pool and as requests die down, the system automatically decommission them. This is really cool in a sense that you are not really wasting resource on Azure when the VM agents are not used as much.

If you take a look at Cloud Statistics in Manage Jenkins section, you will see the history of the automatically provisioned VM agents.

Managing slaves can be troublesome based on the loads but Jenkins on Azure makes it very easy and scalable.

Changing Jenkins Header Color

Please visit the newer version of this article if you like. This article may not work for newer version of Jenkins.

When dealing with multiple instances of Jenkins, you sometimes forget which server you are working on. Imagine a situation where you thought you were testing a job on a pre prod Jenkins server but you executed the job against your production system and the system is completely down. It’s a scary thought.

By changing a little bit of the UI, you can easily tell which Jenkins server you are on. Here are the steps.

  1. Navigate to Manage Jenkins.
  2. Click Manage Plugins.
  3. Click Available tab.
  4. Install “Simple Theme” plugin by clicking Install without Restart button.
  5. Go to Manage Jenkins -> Configure System.
  6. Click Add under Theme section and select Extra CSS.
    jenkins simple theme
  7. Enter the following simple CSS to change the background color of the header.
    div#header
    {
    background:red;
    }
  8. Save the change and when you refresh your browser, the header color has changed.
  9. There are many named colors available that you can use for your header color. Take a look here.

Hope this reduces your (and my) mistakes.

How to Upgrade Jenkins on Azure

After you provision and configure your Jenkins instance on Azure, after a while you may get a notification in Manage Jenkins section like the following. I tested the steps on Azure but the same method can be applied to any Jenkins deployed on Ubuntu.

Here are the steps to upgrade the Jenkins master.

  1. Copy the link to the latest Jenkins war file.
  2. ssh into the VM on Azure where the Jenkins instance is hosted.
    $ ssh username@jenkinsvm.whicheverreagion.cloudapp.azure.com
  3. Run wget to download the file.
    $ wget http://updates.jenkins-ci.org/download/war/2.164.3/jenkins.war
  4. Stop jenkins daemon.
    $ sudo systemctl stop jenkins
  5. Copy the downloaded jenkins.war to /usr/share/jenkins
    $ sudo cp -f ~/jenkins.war /usr/share/jenkins
  6. Start jenkins daemon.
    $ sudo systemctl start jenkins
  7. Open the browser to the Jenkins (or refresh) to check the lower right corner for the version.

Test-NetConnection

One of the Cmdlet that I use often at work lately is Test-NetConnection. This one is good for your sanity check when you want your server to start to communicate with different endpoints. It supports ping test, TCP test, route tracing, and route selection diagnostics. 

$ Test-NetConnection -ComputerName google.com
ComputerName           : google.com
RemoteAddress          : 172.217.3.174
InterfaceAlias         : Ethernet
SourceAddress          : 192.168.1.23
PingSucceeded          : True
PingReplyDetails (RTT) : 14 ms

If you want to check if certain port is open, you can add -Port parameter to the command above and it checks the specified port.

$ Test-NetConnection -ComputerName google.com -Port 80
ComputerName     : google.com
RemoteAddress    : 172.217.3.174
RemotePort       : 80
InterfaceAlias   : Ethernet
SourceAddress    : 192.168.1.23
TcpTestSucceeded : True

Notice that TcpTestSucceeded data is True now. It means the current machine can reach the port 80 of the endpoint.