Jenkins Pipeline Specifying Labels and Parallel Processing

In Jenkins Pipeline, it is possible to specify which slave to run each stage on. Before we go into the code, let’s see how we can set up a pipeline job.

  1. Create a pipeline job.
  2. Select Pipeline script from SCM.
  3. Set the Repository URL.
  4. Set Credentials to get the source code. In this example, I am using my credential for GitHub.

We will write all the pipeline code in Jenkinsfile file.

What I want to be able to do is to have stages and steps and I want to be able to specify which slaves to run them on. Here is how I can run Windows and Linux steps separately.

pipeline
{
    agent none
    stages
    {
        stage("win-step1")
        {
            agent
            {
                label "win"
            }
            steps
            {
                echo "Windows long running process..."
                sleep 10
            }
        }

        stage("linux-step1")
        {
            agent
            {
                label "linux"
            }
            steps
            {
                echo "Linux long running process..."
                sleep 10
            }
        }
    }
}

Right after pipeline declarative, agent is set to none. This is because we are specifying agent in each stage. Windows steps would run on slaves with the label “win” and Linux steps would run on slaves with the label “linux” obviously. Note that each step takes 10 seconds (simulating long running process that takes 10 seconds), so when I run the pipeline, it takes about 20 seconds. When you look at the result in Blue Ocean plugin, you can see the steps were executed sequentially.

If the second stage depends on the first one, this is totally fine, but what if stage 1 and stage 2 can be executed in parallel? What if each stage takes an hour? It would take 2 hours for entire pipeline to finish but if we can execute them in parallel, it would finish in just half the time. Let’s see how we can do it.

pipeline
{
    agent none
    stages
    {
        stage("Run builds on Windows and Linux")
        {
            parallel
            {
                stage("win-step1")
                {
                    agent
                    {
                        label "win"
                    }
                    steps
                    {
                        echo "Windows long running process..."
                        sleep 10
                    }
                }

                stage("linux-step1")
                {
                    agent
                    {
                        label "linux"
                    }
                    steps
                    {
                        echo "Linux long running process..."
                        sleep 10
                    }
                }
            }
        }
    }
}

Take a look at the result in Blue Ocean. You see the stages (steps) were executed in parallel and the execution time was reduced in half! This is beautiful!

If you surround the 2 stages with parallel and have another stage before parallel, Jenkins pipeline sends job requests to appropriate slaves and execute them in parallel. This technique will definitely save so much time. Pipeline should be designed with multiple slaves (agents) in mind for parallel processing.

Author: admin

A software engineer in greater Seattle area

Leave a Reply

Your email address will not be published. Required fields are marked *