Cleaning up Bamboo Agents

If you have a problem of bamboo agents running out of space and wondering why build expiry doesn’t help you – welcome to the club 🙂
I too have bumped into this problem several times and naively thought that this is configurable via Bamboo Admin by Build Expiration setting. Still, our jobs would fail with “fatal: write error: No space left on device” quite too soon. Agent’s home folder was full and running Build Expiration manually with “133 builds affected” result didn’t help a bit. Still full. Most, like in 99%, of space being taken by checked out source code.

I started looking for a solution more thoroughly and found that it happens for others too.

After some investigation I found that both results are stored on Bamboo server:

/var/www/bamboo/xml-data/builds/MYJOB1/download-data/build_logs/MYJOB1-260.log 
/var/www/bamboo/xml-data/builds/MYJOB1/results/buildresults260.xml

The only artifact we create is phpunit.xml (in junit format!) and this one is stored in the Bamboo agent’s build-dir:

/home/bamboo/agent-001/xml-data/build-dir/MYJOB1/test-reports/phpunit.xml

From the docs you can see the explanation. On server we have: xml-data/builds/JOB_KEY/results => contains the build results and it’s not allowed to be deleted. xml-data/builds/JOB_KEY/download-data => contains the logs. These are the thing Bamboo uses to show jobs’ info. As for agent we only have xml-data/build-dir/ => where the agent will check out the files and perform builds, looks not important.

You can find some docs on build expiry and deletion, but this will only affect build results on Bamboo server and artifacts on agent. Checked out source code is not an artifact and won’t be deleted.

There is a way to force clean build, but having 7 different jobs and 10 agents make it hard for same job to occur in same agent. Each job has it’s JOB_KEY and for folder to be cleaned same job needs to be executed on same agent. Eg once MYJOB1 executed on 1st agent, 500Mb of  code will stay on that agent until MYJOB1 is executed on it again. I had a thought to remove build-dir completely, but it triggers an artifact upload and ncover analysis error:

Unable to publish artifact [Test Results (JUnit)]: the source directory /home/bamboo/agent-001/xml-data/build-dir/MYJBOB1 does not exist.
Failed to execute plugin 'NCover Results Collector' with error: /home/bamboo/agent-001/xml-data/build-dir/MYJOB1 does not exist.

That means we need to remove only source code right after job’s completion. For that, check what is your “Checkout Directory” in “Source Code Checkout” task and add new “Script” type task to delete source code with following contents:

rm -rf ${bamboo.build.working.directory}/checkoutFolder
exit 0

Save it and move to “Final tasks” so it would be executed regardless if previous tasks failed. After checkout folder is removed, job folder’s size is just 500Kb. It should be tolerable for most CI systems 🙂

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s