Why does Jenkins think my build succeeded?

Jenkins is written in Java and because of that it is a bit challenging to run things. Jenkins takes a shortcut and doesn’t actually run the commands you type into the build step of a job. Instead, Jenkins writes the build step into a temporary file and calls shell to run it.

If shell exits with code zero, Jenkins thinks everything went fine and the build gets a blue ball. Any other exit code means the build failed and gets a red ball.

Let’s look at an example.

#!/bin/sh

make
echo “Build exit code was $?” 

What is the exit code of the script above?

It will be zero, always. Why? Because the exit code of a script is the exit code of the last command and the echo command will succeed with very high probability.

So you see, it is up to you to not hide the exit code of your build from Jenkins.

Fortunately there is an easy way to do it. The shell has a convenient option -e, which causes shell to stop running a script immediately when any command exits with non-zero exit code. This makes it easy to have Jenkins know when your script fails.

By the way, there’s also one more useful shell option you might want to know about, -x, which makes shell print every command it executes.

Actually, Jenkins will run your build script using /bin/sh -xe, if you only let it. If you start your build script with #!/bin/sh, Jenkins will respect your wish and execute the script with /bin/sh with no options. If you omit the hash-bang line, Jenkins will fall back to /bin/sh -xe, which, you will probably now agree, is a good choice.

The discussion above was unfortunately bit specific to Unix. Sorry.

Restore old job config

Today I discovered someone had overwritten job configuration changes I had carefully prepared. Apparently that someone had an old configuration open in his browser and decided to click on the Save button. I needed to revert back to the old configuration, but how to do it?

I had the Job Configuration History plugin installed, so it had preserved the old config.xml and it could show what has changed. Unfortunately it does not have a button to revert to an old configuration. 

But it does have a link for downloading the old config.xml file. I downloaded it.

Jenkins has a command-line interface, CLI. The same functionality can also be accessed over ssh. If you have turned on security, ssh wants to authenticate you and you must add your ssh public key to your Jenkins user information.

I find the ssh often easier to work with because the key-based authentication does not require me to type passwords.

Finally, to restore the old job configuration was a simple one-liner:

ssh -p NNNN jenkinshost update-job jobname < config.xml

The port where Jenkins listens for ssh protocol (NNNN) is random by default. You can either set it to a fixed port in the Jenkins configuration or you can look up the port number from Jenkins HTTP headers. Something like this will do it:

curl -sf -D/dev/stderr $JENKINS_URL 2>&1 >/dev/null | tr -d ‘\r’ | awk -F: ‘$1 == “X-SSH-Endpoint” {print $3}’

where $JENKINS_URL is the address of your Jenkins.

OK, maybe it is easier to configure Jenkins to use a fixed ssh port.

Hi, CloudBees

CloudBees newsletter features some Jenkins blogs, including Things you didn’t know about Jenkins. Thanks for spreading the word!

Built-in Groovy scripting

Jenkins has a built-in scripting language, Groovy. It can be of great help if you need to do something that is not provided by any setting or plugin and you do not feel you want to write a new plugin. 

The Groovy script has access to all the internals of Jenkins, so there is a lot you can do with it, both good and bad…

The internals of Jenkins are documented at http://javadoc.jenkins-ci.org/.

You can find a Groovy script console in your Jenkins by clicking on “Manage Jenkins” and “Script console”. You type in small Groovy scripts, hit cmd+Return and Jenkins will evaluate the script. (I’m sure there is a keyboard shortcut for PC users too.)

Jenkins also has a Groovy plugin which adds a build step which can run a Groovy script. And there is also a Groovy post-build plugin, which allows you run a Groovy script after the build is done.

Multiple Jenkins sharing the same directory

Sorry, it’s been a while since my last post. Let’s try to pick up the pace.

Some people like Jenkins so much they become dependent on it. Even to an extent where they cannot have Jenkins go down for 5 minutes. They would like to set up two instances of Jenkins running at the same time, but sharing the same JENKINS_HOME directory. This way one of them would be running even if the other fails.

I have bad news for you: Jenkins behaves really badly if you run multiple instances sharing the same JENKINS_HOME directory. Jenkins detects if you are doing this and complains loudly.

All is not lost for such people. Cloudbees, the vendor providing paid support for Jenkins, has a high availability plugin

Fingerprinting

Fingerprints are how Jenkins discovers dependencies among your builds. It will also allow you to show a build artifact like an executable to Jenkins at some later date and Jenkins will tell you where and how it was built.

A “fingerprint” is simply an MD5 checksum of a file. You should configure your Jenkins job to fingerprint all build artifacts:

  1. Open the job configuration page.
  2. Click on “Add post-build action”. Choose “Record fingerprints of files to track usage”.
  3. A new action will be added to the page. Click on the “Advanced” button of the new section. Check the box “Fingerprint all archived artifacts”. 

Jenkins will calculate MD5 checksums of all archived artifact files and remember them. 

If a build uses Copy Artifact step to grab files from another job, Jenkins will check the MD5 checksums and if it finds an earlier record of them, it knows where the files came from and it can show a dependency between the build that produced the file and the build which used the file.

This is very useful. Try it out!

Different ways to run Jenkins

Jenkins is easy to run and set up. In this post I will cover all the ways to run Jenkins.

You can download the jenkins.war file and then simply type into terminal window or command prompt:

java -jar jenkins.war

and now you have Jenkins running in port 8080. Stop it by pressing control + c.

On a Mac you can double-click the jenkins.war file and it will run. There is no user interface or any visible indication that Jenkins is running and listening in port 8080. To stop it, you need to kill the java process.

The jenkins.war file can, of course, be embedded into any servlet container, like any other web app distributed as .war file. You should follow the instructions of the servlet container on how to load and stop Jenkins. There are also some tips in the Jenkins wiki.

Jenkins has also native packages. These are installers in formats familiar to non-java developers. There is a Windows installer with a familiar installation wizard. There is a Mac installer, a package for Solaris, BSD and the various package managers for various Linux distributions.

The native packages typically set up Jenkins to run as a background daemon or system service and it is automatically started when the system boots up. Usually these packages also set up a new dedicated user account for running Jenkins.

And finally there is my own Jenkins package: Jenkins.app. This is a small Mac application that embeds Jenkins making it easy to install/uninstall, run and stop.

So there is something for everyone. And if you cannot decide which to try, just pick something that looks familiar. You can easily change the way you run Jenkins later if you need to.

Multi-configuration jobs

Multi-configuration (AKA matrix) jobs are the perfect tool when you need to run a job several times with a different parameter. Maybe you need to build a debug and release version of your code. Or maybe you need to build different versions for different customers. Or maybe you need to build your code on Windows and Mac.

Multi-configuration job is a build which spawns other builds, one for each configuration. The multi-configuration build does not end until all the configurations have been built.

In other words, if you set up a multi-configuration job that builds on Windows and Mac, you will see three builds being run: the “parent” build, then one build for Windows and one for Mac. The “parent” actually runs nothing. It just waits for the “child” builds to complete.

The “parent” build usually upsets people. People expect a build with two configurations to run twice, not three times. Then people get even more upset when they learn the parent build actually checks out the source code. It does not run any build steps, but it does check out the source if the job has been configured to check out something.

The reasons for this are known only to Jenkins developers, and probably they do not all know why multi-configuration jobs work like this.

Usually you do not have to care. If your codebase is small, the checkout costs nothing.

If your codebase is large, the checkout takes time and requires disk space. So get a big disk for your Jenkins and do not configure the job to delete the workspace before checkout. If the source from the previous build is left in the workspace, Jenkins can usually quickly update it with the latest commits without needing to do a full checkout.

To ensure you get clean builds, configure jobs to clean the workspace from all files that did not come from version control. Most Jenkins SCM plugins support it.

With these tips you should be able to live with multi-configuration jobs or at least tolerate them.

Git plugin, part 3

This is part 3 in a series of posts about the Jenkins Git plugin.

The Jenkins Git plugin is a bit different from the other Jenkins SCM plugins. Take a common use case: You want your build to update the build number into a file in the source tree and then commit the updated file back to version control.

(This is actually not a very good idea. If you want to track the build number, there are better ways to do it than committing it to the source tree.)

Usually you have something like this as part of your build script:

echo $BUILD_NUMBER > build_number.txt

git add build_number.txt

git commit -m ‘Updated build number’

git push

You will be surprised to see this automatic commit is missing. Did it fail? Where did it go?

The commit did not vanish, it’s just not where you expect it to be.

If you go to the Jenkins workspace directory and run git branch, you will be surprised:

> git branch

* (no branch)

  master

The star notes the current branch, which is apparently “no branch”. What is that? You were building from the master branch. You know you did. You wrote “origin/master” into the Jenkins job configuration.

The Git plugin manages the workspace in a peculiar way. It really built the commits from master, even if the workspace does not have the master branch checked out. I don’t fully understand the reason for this even after using Git plugin for four years. But this is the way it works.

If you want your build script to make commits, you have to first check out the branch where you want the commits to appear. You need to add git checkout master as the first command before starting to modify the files in the source tree.

And it really would be better if you didn’t have your build script commit back to version control. It is not needed. I also thought earlier it was. It wasn’t.

Git plugin, part 2

This is part 2 of a series of posts about the Jenkins Git plugin.

The Jenkins Git plugin is different also because it allows you to watch multiple repositories. All the other Jenkins SCM plugins work with only one source repository. The Git plugin allows you to configure multiple repositories in a job.

All the repositories must be clones of each other, so this does not allow you to build a project that needs code from multiple repositories. (For that there is Multiple SCMs plugin or Repo plugin or you could use git submodules.)

It is still a useful feature. You could use this e.g. to pull in commits from the workstations of your developers, merge them all and see if the result builds and passes tests.