Devfest Lille 2017

Jenkins, BlueOcean and Declarative Pipelines

ToC

Motivations of this talk

Motivations

eng pig

Jenkins Project

Meet Jenkins

Jenkins is an open source automation server which enables developers around the world to reliably build, test, and deploy their software.

jenkins logo

What is Jenkins ?

samurai kk

  • #1 Continuous Integration and Delivery server

  • Created by Kohsuke Kawaguchi in 2006

    • Original project: "Hudson", renamed "Jenkins" in 2011

  • An independent and active community (jenkins.io)

    • 500+ releases to date

    • 150,000+ active installations

    • 300,000+ Jenkins servers

    • 1,200+ plugins

Jenkins Popularity: Through the Roof

ci usage rebel lab 2016

Source: RebelLabs Tools and Technologies Leaderboard 2016

Worldwide Adoption

jenkins worldwide adoption

Source: stats.jenkins.io

Jenkins in 2016

2016 was the year of Jenkins 2

Why Jenkins 2 ?

Jenkins 2 Goals

  • Target: CI → CD

  • No breaking changes from Jenkins 1

    • Smooth upgrade

    • Plugins compatibility

  • First time experience improvement

    • Brand new Wizard

  • Pipeline-as-Code:

    • Jenkinsfile stored in SCM

    • Groovy DSL: "Code your Pipeline"

Jenkins in 2017 ?

Jenkins in 2017

  • Declarative Pipeline

    • Still Jenkinsfile

    • Easier

    • Compatible with Scripted Pipeline

  • BlueOcean

    • Brand new GUI

    • Written in ReactJS

    • Opinionated

Hello

Whoami: Jean-Marc MEESSEN

Jmm

fishSticks

Whoami: Damien DUPORTAL

damien

CloudBees

<sales_pitch>

  • Software at the "Speed of Ideas", Hub of "Enterprise Jenkins and DevOps", providing:

    • Jenkins "Enterprise" Distribution

    • Services around Jenkins

  • Jenkins World 2017: THE Event for Everything Jenkins and DevOps

</sales_pitch>

Who are you ?

Prepare Lab Environment: Local VM Based

  • Requires VirtualBox >= 5.1.22

  • Requires Vagrant >= 1.9.4

  • From a Terminal, download the VM (1 time, ~1Gb):

    vagrant box add devfest-2017-jenkins \
  https://github.com/oufti-playground/lab-vm/releases/download/devfest-2017/jenkins-lab-demo.box

  • From the same Terminal, initialize the VM project:

    mkdir devfest-2017-jenkins
cd devfest-2017-jenkins
vagrant init -m -f devfest-2017-jenkins

Let’s Get Started: Local VM Based

  • Start the VM from the devfest-2017-jenkins folder:

    $ ls
  Vagrantfile
$ pwd
  .../devfest-2017-jenkins
$ vagrant up

  • Access your instance homepage:

    http://localhost:10000

That’s all for this chapter

Demo Application

Demo Application: Why ?

  • Goal: Illustrate a Software Supply Chain with a demo application

  • Challenge: So many languages/framework/toolchains

  • Solution:

    • Opinionated demo application (language, tooling, etc.)

    • Put everyone on same page with initial exercise

Demo Application: What ?

  • Web application

  • This is the Spring Boot Starter

  • Language: Java (OpenJDK 8)

  • Toolchain: Maven (Maven >= 3.3)

  • Source code stored inside a local Git repository

Demo Application: How ?

Demo Application: Access it

Demo Application: Check it

  • Maven configuration: pom.xml

  • Application Source code: src/main/java/

  • Application Templates/HTML: src/main/resources/

  • Application Test code: src/test/java

Demo Application: Get it

  • Open the DevBox, the Web based command line:

  • Copy the demoapp repository URL from GitServer

  • Run the following commands:

    # Get the git repository
git clone http://localhost:10000/gitserver/butler/demoapp.git
# Browse to the local repository
cd ./demoapp
# Check source code
ls -l
cat pom.xml

Demo Application: DevBox Tricks

  • Clean the window: clear

  • Show command history: history

  • CTRL + R: search the command history interactively

  • CTRL + C: cancel current command and clean line buffer

  • CTRL + A: jump to beginning of line

  • CTRL + E: jump to end of line

Demo Application: Maven

  • Maven TL;DR:

    • Provide a standardized workflow

    • pom.xml describe the application

  • Maven Command line : mvn, expects goals (workflow steps)

    mvn dependency:list

  • Can have flags (configuration on the fly)

    mvn dependency:list -fn

Demo Application: Compile it

  • Maven goal is compile

    • Resolve build dependencies

    • Process source code

    • Generate classes

  • Content put in the ./target folder:

    mvn compile
ls -l ./target

Demo Application: Unit-Test it

  • Maven goal is test

    • Execute compile goal

    • Compile Unit Test classes

    • Run Unit Test

  • Tests Reports put in the ./target/surefire-reports folder:

    mvn test
ls -l ./target/surefire-reports

Demo Application: Build it

  • Maven goal is package

    • Execute compile and test goals

    • Package the application as specified in pom.xml

  • The new artifact (generated packages) is stored in ./target

    mvn package
ls -ltrh ./target/

Demo Application: A note about Tests

test unit
test integration

Demo Application: Integration Testing

  • Maven goal is verify

    • Execute compile, test and package goals

    • Resolve integration test dependencies

    • Run Tests against the packaged application

  • Tests Reports stored in the ./target/failsafe-reports folder:

    mvn verify
ls -l ./target/failsafe-reports

That’s all for this chapter

Continuous Integration with Jenkins

aka "CI"

CI: Why ?

big bugs

Continuous Integration doesn’t get rid of bugs, but it does make them dramatically easier to find and remove.

— Martin Fowler

CI: What ?

Continuous Integration is a software development practice where members of a team integrate their work frequently, usually each person integrates at least daily, leading to multiple integrations per day.

— Martin Fowler - Continuous Integration

CI: How ?

  • Each integration is verified by an automated build (including test)

  • Integrate code often, at least daily, to make integration a non-event

  • Continuously build and integrate, with a feedback loop

ci howto

Continuous Integration with Jenkins

CI: Accessing Jenkins

CI: Jenkins BlueOcean

CI: Our First Pipeline Project

  • Create your 1st Pipeline:

    • Stored in Git

    • Fetch URL from the Gitserver

    • Add a User/password credential (butler / butler)

    • Pipeline is empty (for now): no Jenkinsfile

CI: Fast feedback with Webhooks

  • We want Fast feedback !

    • Pushed code to repository ? Tell Jenkins to build it now

  • Let’s use Webhook to the repository

    • HTTP request Gitserver → Jenkins

CI: Add a Gogs Webhooks

CI: Starting with Pipelines

CI: Declarative or Scripted Pipelines ?

  • Declarative

    • Easy syntax

    • Default syntax

    • Start with this one

  • Scripted

    • Original syntax (~3 years)

    • "Great Power == Great Responsibility"

    • Use it when Declarative starts to be weird

CI: BlueOcean Pipeline Editor

  • Provides the full round trip with SCM

  • No Pipeline ? Follow the wizard (not Gandalf, fool !)

  • Already have a Pipeline ? Edit, commit, run it

  • Needs a compliant SCM

    • Only Github with BO 1.0.1

    • Interested ? Open-Source: Contribute !

CI: Use the Pipeline Editor

CI: Exercise - Your First Pipeline

  • Use the BlueOcean Pipeline Editor and Gitserver

  • Create a Pipeline that have a single stage "Hello"

  • This stage have 1 step that prints the message "Hello World"

  • Copy/Paste this Pipeline in a new file Jenkinsfile on the repository root

  • A build will kick off immediately:

CI: Solution - Your first pipeline

pipeline {
  agent any
  stages {
    stage('Hello') {
      steps {
        echo 'Hello World !'
      }
    }
  }
}

CI: Exercise - Simple Build Pipeline

  • Exercise: Implement a simple build pipeline for demoapp

  • We want 4 stages, for the 4 Maven goals:

    • compile, test, package, verify

  • We need to build on the maven agent

CI: Solution - Simple Build Pipeline

pipeline {
  agent {
    node {
      label 'maven'
    }
  }
  stages {
    stage('Compile') {
      steps {
        sh 'mvn compile'
      }
    }
    stage('Unit Tests') {
      steps {
        sh 'mvn test'
      }
    }
    stage('Build') {
      steps {
        sh 'mvn package'
      }
    }
    stage('Integration Tests') {
      steps {
        sh 'mvn verify'
      }
    }
  }
}

CI: Exercise - Artifacts

  • We want to simplify to 2 stages, based on Unit Tests definition:

    • Build: compile, unit test and package the application

    • Verify: Run Integration Tests

  • We also want to archive the generated jar file

    • Only if the build is successful

  • Clues: Keywords post + success (not in Editor), and archiveArtifacts

CI: Solution - Artifacts

pipeline {
  agent {
    node {
      label 'maven'
    }
  }
  stages {
    stage('Build') {
      steps {
        sh 'mvn package'
      }
    }
    stage('Verify') {
      steps {
        sh 'mvn verify'
      }
    }
  }
  post {
    success {
      archiveArtifacts 'target/demoapp.jar'
    }
  }
}

CI: Exercise - Integration Tests Reports

  • We want the integration test reports to be published to Jenkins

    • Better feedback loop

  • If Integration Tests are failing, do NOT fail the build

    • Make it UNSTABLE instead

  • Clues:

    • Maven flag -fn ("Fails Never")

    • keyword junit (Pipeline keyword)

CI: Solution - Integration Tests Reports

pipeline {
  agent {
    node {
      label 'maven'
    }
  }
  stages {
    stage('Build') {
      steps {
        sh 'mvn clean compile test package'
      }
    }
    stage('Verify') {
      steps {
        sh 'mvn verify -fn'
        junit '**/target/failsafe-reports/*.xml'
      }
    }
  }
  post {
    success {
      archiveArtifacts 'target/demoapp.jar'
    }
  }
}

CI: Exercise - All Tests Reports

  • We now want all test reports published

    • Problem: how to handle Unit test failure ?

  • We also want to archive artifacts if build is unstable only due to the Verify stage

  • Clues: post can be used per stage

CI: Solution - All Tests Reports

pipeline {
  agent {
    node {
      label 'maven'
    }
  }
  stages {
    stage('Build') {
      steps {
        sh 'mvn clean compile test package'
      }
      post {
        always {
          junit '**/target/surefire-reports/*.xml'
        }
      }
    }
    stage('Verify') {
      steps {
        sh 'mvn verify -fn'
        junit '**/target/failsafe-reports/*.xml'
      }
      post {
        unstable {
          archiveArtifacts 'target/demoapp.jar'
        }
      }
    }
  }
  post {
    success {
      archiveArtifacts 'target/demoapp.jar'
    }
  }
}

CI: Failing Tests

  • Validate your changes by making your tests fails.

  • Edit each one and uncomment the failing block:

    • Integration: src/master/src/test/java/hello/ApplicationIT.java

    • Unit Tests: src/master/src/test/java/hello/ApplicationTest.java

  • Browse the top-level items "Changes", "Tests" and "Artifacts"

  • Do NOT forget to correct your tests at the end

That’s all for this chapter

Docker

to the Rescue

Docker: Why ?

the matrix from hell

Docker: What ?

container vs vm

Docker How ?

docker how

Docker: Demo Application’s Dockerfile

Docker: Building Demo Application

  • Using Devbox, from the demoapp work directory’s root

    • Checking images with docker images | grep registry_user

  • Build an image named registry_user/demoapp:latest from the repository root:

    docker build -t registry_user/demoapp:latest ./

  • Check again the images

Docker: Build and Smoke Test

/usr/local/bin/bats ./src/test/bats/docker.bats

That’s all for this chapter

Continuous Delivery with Jenkins

aka "CD"

CD: Why ?

How long would it take your organization to deploy a change that involves just one single line of code?

  • Reduce deployment risks

  • Allow more frequent user feedback

  • Make progress believable by everyone

CD: What ?

Continuous Delivery is the next step after Continuous Integration:

  • Every change to the system can be released for production

  • Delivery can be done at any time, on any environment

Your team prioritizes keeping the software deployable over working on new features

— Martin Fowler

CD is NOT Continuous Deployment

Both are always confused:

continuous depl vs delivery

CD: How ?

  • Having a collaborating working relationship with everyone involved

  • Using Deployment Pipelines, which are automated implementations of your application’s build lifecycle process

CD: Delivery Target

  • Production runs on Docker

  • Your Ops team use a Docker Registry

  • Expected Artifact:

    • Not a jar file

    • But a Docker image

CD: Exercise - Docker Test Suite

  • Goal: Run the Docker Test Suite

    • Using a single stage named "Docker", before Integration Tests

    • Using the agent labelled docker

    • Challenge: we need the jar file at "Docker time"

    • We do not need to archive artifact at the end, unless Integration Test is unstable

  • Clues: Keywords stash and unstash

CD: Solution - Docker Test Suite

pipeline {
  agent { node { label 'maven' }}
  stages {
    stage('Build') {
      steps {
        sh 'mvn package'
        stash(name: 'app', includes: 'target/demoapp.jar')
      }
      post { always { junit '**/target/surefire-reports/*.xml' }}
    }
    stage('Docker') {
      agent {
        label 'docker'
      }
      steps {
        unstash 'app'
        sh '/usr/local/bin/bats ./src/test/bats/docker.bats'
      }
    }
    stage('Verify') {
      steps {
        sh 'mvn verify -fn'
        junit '**/target/failsafe-reports/*.xml'
      }
      post { unstable { archiveArtifacts 'target/demoapp.jar' }}
    }
  }
}

CD: Exercise - Approval and Deploy

  • Goal: We want a Human Approval before Deploy

  • Add 2 stages named Approval and Deploy:

    • Approval will ask for a manual validation, after Integration Tests

    • Deploy will tag and push the Docker Image to the Docker registry at the URL localhost:5000:

  • Clues: Keyword input

  • Here is the Deploy shell code:

docker tag demoapp:$(git rev-parse --short HEAD) localhost:5000/registry_user/demoapp:latest
docker push localhost:5000/registry_user/demoapp:latest

CD: Solution - Approval and Deploy

pipeline {
  agent { node { label 'maven' }}
  stages {
    stage('Build') {
      steps { sh 'mvn package'
        stash(name: 'app', includes: 'target/demoapp.jar') }
      post { always { junit '**/target/surefire-reports/*.xml' }}
    }
    stage('Docker') {
      agent { label 'docker' }
      steps { unstash 'app'
        sh '/usr/local/bin/bats ./src/test/bats/docker.bats' }
    }
    stage('Verify') {
      steps { sh 'mvn verify -fn'
        junit '**/target/failsafe-reports/*.xml' }
      post { unstable { archiveArtifacts 'target/demoapp.jar' }}
    }
    stage('Approval') {
      agent none
      steps { input 'Is it OK to deploy demoapp ?' }
    }
    stage('Deploy') {
      agent { label 'docker' }
      steps {
        sh 'docker tag demoapp:$(git rev-parse --short HEAD) localhost:5000/registry_user/demoapp:latest'
        sh 'docker push localhost:5000/registry_user/demoapp:latest'
      }
    }
  }
}

CD: Exercise - Building with Docker

  • Goal: Use Docker to provide the build environment

    • Use the agent allocation to build and run builds within a Docker container

    • Use the Dockerfile.build from the repository

  • Clues: Keywords agent none, agent { dockerfile …​ label …​}

CD: Solution - Building with Docker

pipeline { agent none
  stages {
    stage('Build') {
      agent { dockerfile { filename 'Dockerfile.build'
        label 'docker'}}
      steps { sh 'mvn package'
        stash(name: 'app', includes: 'target/demoapp.jar') }
      post { always { junit '**/target/surefire-reports/*.xml' }}
    }
    stage('Docker') {
      agent { label 'docker' }
      steps { unstash 'app'
        sh '/usr/local/bin/bats ./src/test/bats/docker.bats' }
    }
    stage('Verify') {
      agent { dockerfile { filename 'Dockerfile.build'
        label 'docker'}}
      steps { sh 'mvn verify -fn'
        junit '**/target/failsafe-reports/*.xml' }
      post { unstable { archiveArtifacts 'target/demoapp.jar' }}
    }
    stage('Approval') {
      agent none
      steps { input 'Is it OK to deploy demoapp ?' }
    }
    stage('Deploy') {
      agent { label 'docker' }
      steps { sh 'docker tag demoapp:$(git rev-parse --short HEAD) localhost:5000/registry_user/demoapp:latest'
        sh 'docker push localhost:5000/registry_user/demoapp:latest' }
    }
  }
}

CD: Exercise - Scaling Pipeline

  • Goal: Share Pipeline across your teams

  • We want to use Shared Libraries

  • There is one autoconfigured named deploy

  • Use the annotation to load the Library, on master branch

  • Check the library here

  • Clues: Keywords @Library, script

CD: Solution - Scaling Pipeline

@Library('deploy@master') _
pipeline { agent none
  stages {
    stage('Build') {
      agent { dockerfile { filename 'Dockerfile.build'
        label 'docker'}}
      steps { sh 'mvn package'
        stash(name: 'app', includes: 'target/demoapp.jar') }
      post { always { junit '**/target/surefire-reports/*.xml' }}
    }
    stage('Docker') {
      agent { label 'docker' }
      steps { unstash 'app'
        sh '/usr/local/bin/bats ./src/test/bats/docker.bats' }
    }
    stage('Verify') {
      agent { dockerfile { filename 'Dockerfile.build'
        label 'docker'}}
      steps { sh 'mvn verify -fn'
        junit '**/target/failsafe-reports/*.xml' }
      post { unstable { archiveArtifacts 'target/demoapp.jar' }}
    }
    stage('Deploy') {
      agent none
      steps {
        script {
          deploy('demoapp','localhost:5000/registry_user')
        }
      }
    }
  }
}

CD: Exercise - Parallel Stages

  • Goal: Run Stages in parallels to gain time

    • We can safely run Docker Smoke and Integration Tests in parallel

    • To specify a specific agent, use Scripted Pipeline Block and the node allocation

  • Clues: Keywords parallel, script, node

  • WARNING: https://issues.jenkins-ci.org/browse/JENKINS-41334

CD: Solution - Parallel Stages

@Library('deploy@master') _
pipeline { agent none
  stages {
    stage('Build') {
      agent { dockerfile { filename 'Dockerfile.build'
          label 'docker' }}
      steps {
        sh 'mvn package'
        stash(name: 'app', includes: 'target/demoapp.jar')
      }
      post { always { junit '**/target/surefire-reports/*.xml' }}
    }
    stage('Tests') {
      steps {
        parallel ( "Integration Tests": { script {
          node('maven') { checkout scm
            sh 'mvn verify -fn'
            junit '**/target/failsafe-reports/*.xml'
        }}}, "Docker": { script {
          node('docker') {
            unstash 'app'
            withEnv(['DOCKER_HOST=tcp://docker-service:2375']) {
              sh '/usr/local/bin/bats ./src/test/bats/docker.bats'
        }}}})
      }
      post { unstable { archiveArtifacts 'target/demoapp.jar' }}
    }
    stage('Deploy') { agent none
      steps { script { deploy('demoapp','localhost:5000/registry_user') } }
    }
  }
}
thats all folks wallpaper
Devfest Lille 2017: Jenkins, BlueOcean and Declarative Pipelines © 2017 CloudBees, Inc. All Rights Reserved ToC Icon