Wayne Warren, Puppet Labs
Puppet Labs delivers on-premise software which consists of multiple interdependent component projects. Each individual project is subject to rigorous developer and QA validation involving unit tests, component & composite integration tests, and finally combined acceptance tests. Additionally, the tools we develop as part of our build and test automation suite must often be “smoke” tested against the products they are designed to test to ensure that deploying new versions of these tools doesn’t introduce instability into our overall testing and validation infrastructure —a single bug in just one of these tools has been known to impact over half of our development teams for days at a time while fixes are rolled out and build/test dependencies are updated in individual project repositories.
To automate this testing, we use Jenkins as our Continuous Integration system. However because of the large number of interdependent projects involved in building and testing our products, there is a heavy maintenance burden just to keep our jobs configured correctly. The Jenkins Web UI is tedious to approach even for a small number of jobs and the complexity increases as the number of projects, their versions, and people involved in managing them increases.
To begin addressing this maintenance burden, Puppet Labs has invested engineering time and effort into understanding, improving, and applying Jenkins Job Builder (JJB) to manage our job configurations. JJB is a python tool developed by the Openstack Infra developers that takes YAML as input, generates valid Jenkins Job configuration XML, then POSTs it to the specified Jenkins server to create or update Jenkins jobs. Its features include:
- modular support for Jenkins plugins
- support for parameterized job templates
- deletion as well as creation of managed jobs
Using Jenkins Job Builder has helped us:
- Mitigate configuration drift between jobs/pipelines
- Identify and decommission forgotten jobs
- Help engineers better understand and manage configurations
In this presentation you’ll learn how to use JJB to manage and version control your job configurations. You’ll walk away with an understanding of key concepts such as
- Generic job/pipeline types (unit, packaging, component integration, system integration)
- Language and framework specific implementations of generic job/pipeline types
- Job template name guidelines for optimal reuse
Additionally, I hope to cover some more specific implementation details of
- How we store JJB yaml in a git repo
- Configuration testing and deployment strategies
Target Audience: Intermediate
2015 Technical Paper, Wayne Warren, Paper, Slides, Notes, Video.
