VMware ESXi and vSphere Cluster Management

Jenkinsfile: Defining Jenkins Pipelines as Code

Learn how to create, structure, store, secure, test, and run Jenkinsfiles for reliable CI/CD pipelines using Declarative and Scripted Pipeline syntax.

A Jenkinsfile is a file in a source-code repository that defines a Jenkins Pipeline. It describes how Jenkins should check out code, build it, run tests, create artifacts, publish outputs, and deploy software.

Because the file is version-controlled, pipeline changes are reviewed, branched, tested, and traced like application changes. The conventional location is a file named Jenkinsfile at the repository root.

What a Jenkinsfile does

A Pipeline is an automated workflow made of ordered phases. A typical flow is:

1. Check out a commit from source control.
2. Install dependencies and select reproducible tools.
3. Compile or build the application.
4. Run unit and integration tests.
5. Publish test reports and retain useful artifacts.
6. Build and publish a container image or package.
7. Deploy to a non-production environment.
8. Validate the deployment, request approval when required, and deploy to production.

Jenkins can also hold pipeline logic in a job configuration edited through its interface. That approach is convenient for a quick experiment, but repository-managed logic provides code review, history, branch-specific behavior, and a recoverable source of truth.

Jenkins Pipeline fundamentals

A Pipeline job obtains a Jenkinsfile through its configured source-control definition. A single-branch Pipeline points to one repository and branch. A Multibranch Pipeline discovers branches and pull requests, then runs the Jenkinsfile found in each relevant branch or change request.

Builds can start from source-control polling, a webhook, a schedule, or an upstream job. Webhooks usually provide faster feedback than frequent polling. A Multibranch job normally needs repository credentials, branch and pull-request discovery rules, the expected Jenkinsfile path, and a source-control webhook.

Declarative and Scripted Pipeline syntax

Declarative Pipeline is usually the best starting point because its opinionated structure makes lifecycle behavior, visualization, validation, and team conventions easier to understand. Scripted Pipeline is appropriate when the workflow must dynamically create stages or branches, calculate execution paths, or use advanced Groovy logic. A Declarative script block can contain Scripted-style statements, but extensive use reduces Declarative’s clarity and validation benefits.

Declarative Pipeline structure

The outer pipeline block contains directives such as agent, stages, environment, options, and post.

pipeline {
  agent any
  stages {
    stage('Build') {
      steps {
        sh 'make build'
      }
    }
  }
}

Agents and workspaces

• agent any uses any available compatible agent.
• agent none allocates no global agent; each stage must select its own agent.
• agent { label 'linux-builder' } selects an agent with that label.
• agent { docker { image 'gradle:8.7-jdk17' } } runs work in a configured Docker environment.

Agents provide an operating system, installed tools, workspace capacity, and executors. The controller coordinates jobs and stores Jenkins state; agents perform build work. Do not assume a workspace is permanent: clean it, avoid sharing it between incompatible builds, and transfer required files when changing agents.

Stages, steps, and common directives

Environment, parameters, and options

environment supplies variables globally or to one stage. parameters defines user inputs such as a target environment or release version. Parameters should be few, documented, and validated; do not let an unchecked parameter select arbitrary commands or destinations.

Useful options include timeout, buildDiscarder for build retention, timestamps for log timing, disableConcurrentBuilds() to prevent overlapping deployments, and retry policies where appropriate.

Conditions, approvals, parallel work, and matrices

A when condition can inspect a branch, change request, tag, expression, or environment. post conditions include always, success, failure, unstable, aborted, and changed. Use always for cleanup and report collection that should happen regardless of the result.

stage('Deploy production') {
  when {
    branch 'main'
  }
  input {
    message 'Approve production deployment?'
  }
  steps {
    sh './deploy.sh production'
  }
}

parallel runs independent branches concurrently, such as unit and integration tests. A matrix runs combinations of declared axes, such as operating systems and runtime versions. Parallelism reduces elapsed time but consumes more executors and can overload shared test systems. Fail-fast behavior should be chosen deliberately.

Scripted Pipeline structure

node('linux') {
  try {
    stage('Checkout') {
      checkout scm
    }
    stage('Test') {
      sh './test.sh'
    }
  } catch (err) {
    currentBuild.result = 'FAILURE'
    throw err
  } finally {
    junit allowEmptyResults: true, testResults: 'reports/*.xml'
    cleanWs()
  }
}

A node block allocates an executor and workspace. stage labels work for visualization. Groovy provides variables, conditionals, loops, functions, and closures; scripted parallel execution can be expressed with a map of branch names to closures.

Use try, catch, and finally to handle errors and guarantee cleanup. Re-throw errors when the build must remain failed. Excessive Groovy can execute logic on the controller, consume controller resources, complicate restart behavior, and make security approval harder. Keep computation small and perform intensive work on agents.

A build, test, and artifact workflow

pipeline {
  agent any
  options {
    timeout(time: 30, unit: 'MINUTES')
    timestamps()
    buildDiscarder(logRotator(numToKeepStr: '20'))
    disableConcurrentBuilds()
  }
  stages {
    stage('Checkout') {
      steps {
        checkout scm
      }
    }
    stage('Build') {
      steps {
        sh './gradlew assemble'
      }
    }
    stage('Test') {
      steps {
        sh './gradlew test'
        junit 'build/test-results/test/*.xml'
      }
    }
  }
  post {
    always {
      archiveArtifacts artifacts: 'build/libs/*.jar', fingerprint: true
      cleanWs()
    }
    failure {
      echo 'Build or test failure requires investigation.'
    }
  }
}

Adapt commands and paths to the application. A complete delivery workflow may build a container image, scan it, publish it to a registry, publish packages, deploy to staging, run smoke tests, request production approval, and deploy using an idempotent operation. Add rollback or recovery procedures when a deployment can partially succeed. Notifications should identify the commit, failed stage, logs, and responsible team without exposing secrets.

Source-control integration

In a Multibranch Pipeline, checkout scm is a shortcut for the job’s configured source-control checkout. Branch builds normally use the Jenkinsfile from that branch. Pull-request behavior depends on the source-control plugin and discovery configuration, so verify whether the job builds the source branch, a merge result, or both.

Private repositories require a suitable read-only repository credential, such as an SSH key or access token. Pipeline changes should go through normal code review because a Jenkinsfile can change build commands, credentials usage, artifact publication, and deployment behavior.

Credentials and secret handling

Store passwords, tokens, private keys, certificates, and secret files in Jenkins Credentials. Reference them by a credentialsId; never place the secret itself in the Jenkinsfile. Scope credentials to the smallest suitable folder or job and separate development, staging, and production credentials.

withCredentials([string(credentialsId: 'registry-token', variable: 'REGISTRY_TOKEN')]) {
  sh '''
    set +x
    ./publish.sh "$REGISTRY_TOKEN"
  '''
}

Masking reduces accidental display but is not a guarantee: transformed values, command arguments, tool diagnostics, and poorly behaved programs may still expose secrets. Avoid Groovy interpolation of secret values, shell tracing, and echoing environment variables. If a secret appears in logs, revoke or rotate it immediately.

Agents, containers, and platform differences

Label-based selection ensures that a job runs where its required operating system and tools exist. Docker agents provide isolated, reproducible environments; pin image versions rather than relying on mutable latest tags. Confirm how the plugin mounts the workspace and whether nested container operations are supported.

Linux agents commonly use sh, Windows command agents use bat, and PowerShell agents use powershell or pwsh. Paths, quoting, exit codes, permissions, line endings, and available commands differ across platforms. A workspace may be reused after an interrupted build, so clean it or use a disposable environment when stale files could affect results.

Artifacts, reports, and outputs

archiveArtifacts retains files with a Jenkins build. Configure retention so storage does not grow without bound. junit publishes compatible XML test results, allowing Jenkins to show pass, failure, and trend information. Publish results even after test failures when files exist.

stash stores files temporarily for the current Pipeline run and unstash retrieves them on another stage or agent. Stashes are not a replacement for a durable artifact repository; publish release outputs to an artifact or container registry. Use fingerprint: true on archived files when tracing which builds used a particular output is important.

Reliability and failure handling

• Most shell steps fail the stage when their command returns a nonzero exit status. Check and handle expected nonzero results explicitly.
• Use pipeline- or stage-level timeouts to prevent hung builds.
• Retry only transient operations, such as a temporary network failure; do not blindly retry deterministic test or compile failures.
• Use Declarative post { always { ... } } or Scripted finally for cleanup and report publication.
• Use an unstable result for quality warnings when appropriate, but reserve failure for a delivery-blocking error.
• Handle manual aborts separately when notifications or cleanup need to distinguish them.
• Make deployment steps idempotent where possible so retrying converges on the desired state rather than duplicating side effects.

Pipeline reuse and maintainability

A Shared Library is versioned reusable Pipeline code. Conceptually, vars contains globally callable steps, src contains structured classes and supporting code, and resources contains non-code files. Pin a trusted library version where reproducibility matters.

@Library('delivery-library@v3.2.0') _

deliveryPipeline service: 'orders', deployable: true

Keep Jenkinsfiles focused on intent and move duplicated build, deployment, notification, and policy logic into reviewed library functions. Document required tools, credentials, parameters, agents, environment assumptions, and expected reports.

Security and governance

The Jenkins sandbox restricts operations available to untrusted Pipeline Groovy. A restricted operation may require script approval. Trusted libraries and approved code have greater authority, so they require stronger review and ownership.

Pull-request Jenkinsfiles are security-sensitive because contributors may alter commands executed by Jenkins. Do not expose production credentials or deployment permissions to untrusted change requests. Restrict who can approve deployments, separate permissions by environment, audit source-control changes and Jenkins build history, and avoid unsafe shell interpolation of branch names, parameters, or other untrusted input.

Validation, testing, and debugging

Use the Declarative Pipeline syntax validator for structural errors. Where available, add Jenkinsfile linting and Pipeline unit tests. Test small changes on a branch before changing production delivery paths. Build logs, stage visualization, console timestamps, and archived reports are the primary diagnostic evidence.

When debugging, identify whether the failure is caused by a plugin, agent, credential, checkout, command, workspace, or artifact path. Print safe tool versions and PATH values, but never print credentials. Check the earliest meaningful error rather than only the final cleanup failure.

Best-practice checklist

• Keep a reviewed Jenkinsfile at the repository root unless job configuration deliberately specifies another path.
• Prefer Declarative syntax and descriptive, focused stage names.
• Keep credentials outside source control and use least-privilege, environment-specific access.
• Use explicit timeouts, cleanup, report publication, and sensible build retention.
• Prefer pinned tool versions and isolated build environments.
• Use parameters sparingly and validate every externally supplied value.
• Archive useful outputs, but use durable artifact repositories for releases.
• Make deployments observable, repeatable, approval-aware, and recoverable.
• Review and test the Jenkinsfile as production code.

Related Jenkinsfile concepts

For broader implementation work, connect this lesson with Jenkinsfile pipeline design, Jenkins administration, Multibranch Pipeline configuration, Shared Libraries, credential management, containerized or Kubernetes agents, webhook configuration, artifact repositories, deployment strategies, infrastructure as code, and Pipeline testing.