DZone Interviews Jenkins Creator Kohsuke Kawaguchi
In conjunction with the release of our incredibly awesome Jenkins on PaaS Refcard, we decided to have another interview with a big hero in the Java community. As the creator of Hudson (now he develops the fork, Jenkins) during his days at Sun Microsystems, Kohsuke Kawaguchi quickly made a name for himself in the world of open source.
After leaving Sun in the wake of the Oracle acquisition, Kohsuke started his own Hudson-focused consulting startup, which he later decided to merge with a visionary new startup, CloudBees, which now provides a fully-loaded open source development stack for Java as a cloud service (and that stack includes an excellent Jenkins-as-a-service product).
So without further ado, here is what Kohsuke and I talked about:
DZone: What are some of the major changes that have come to Jenkins in the past year?
Kohsuke Kawaguchi: There are a number of things!
I'll start with plugins, which are what the users tend to notice more. In the past year, we added about 130 brand new plugins and countless updates to existing plugins.
There are so many that I can't track all of them, but for example, we added xcode plugin for better iOS builds, buildflow plugin was added that allows people to do choreography across a large number of jobs, build pipeline plugin gets a rebump in the UI. We added Subversion 1.7 support in the Subversion plugin, Active Directory plugin now really works flawlessly with forests and various different group types, and so on.
Then there's core, which serves as the foundation of everything else.
One area in the core that made a lot of progresses is the UI. A bunch of developers came to FOSDEM this Feburary this, and one afternoon we sat in a busy cafeteria and listed up a small number of important things that we thought would make a difference in the usability. Many of them have been subsequently implemented, as can be seen in this blog post, and those are now in the releases.
There are also numerous changes in the backend. For example, about a year ago we improved our plugin loading and wiring mechanism by introducing Guice, then we used this to allow people to install new plugins without restarting Jenkins. There are a whole bunch of new extension points, which in turn enable plugins to do more interesting things.
There are various performance improvements, too, such as classloading performance, memory footprint consumption, etc. One of them involves coming up with a clever data structure suitable for representing many similar strings efficiently. In Jenkins, this kind of patterns show up in a number of places, such as test reports.
DZone: What areas of development tooling has Jenkins expanded into other than being just a build tool and CI tool?
Kohsuke: I see a lot of people automating the entire process of the software build and delivery. Think of it as a blackbox where the source code changes come in from left and a working running application appear from right. This has been constantly made easier as more sophisticated plugins appear, such as integration with git branching and merging so that changes have to pass the test first before appearing in the main trunk, integration with IaaS providers to provision computers on-demand, and/or controlling a selenium grid from Jenkins.
There has been a number of presentations in Jenkins User Conference about how different people do thesse things. I think this is a very natural evolution of Jenkins as a general purpose automation platform. This enables smaller number of people to effectively harness the power of large number of computers.
I also see a lot of users using Jenkins to share batch tasks and other operational/maintenance tasks so that anyone in the group, not just the one who wrote it, can run it. Put another way, Jenkins as a glorified cron --- it's got a scheduler, logs, notifications, and multi-machine support, so it really is a far greater alternative to cron. Granted, this is not a sexy cutting-edge use case of Jenkins, but these are the kind of things people need every day, and it's a no-brainer!
DZone: You work for CloudBees - so what are some insights you can share about using Jenkins in the cloud?
Kohsuke: As I mentioned earlier, many of the emerging sophisticated practices rely on the abundance of the computing power (otherwise who can afford to run the full test cycle on every single inbound commit?) and the programmatic allocation and destruction of computers (AKA elasticity), which are two key properties of the "cloud" in my humble opinion. So I'm more convinced than ever that the combination of CI and cloud makes sense (and there are many others who seem to agree with me, given the rise of various hosted CI services.)
I think the ability for individual developers to take advantages of ever increasing computing power is one of the key challenges in our industry, and this is what Jenkins does well and can do even better.
Here at CloudBees, we've been busy making DEV@cloud better. For example, we've launched BuildHive, a low-end addition to DEV@cloud that allows you to start a CI service on your GitHub open-source projects with just a few clicks. It can automatically build pull requests, and it can act as a gatekeeper between a committer and a repository, allowing a committer to verify the changes with the tests first before the change lands on the repository.
On DEV@cloud, people can now bring their own slaves to Jenkins, doing builds that require special tools/libraries (such as iOS builds.) We've also improved the way our backend systems work, for example by using LXC to improve the time it takes to reuse slaves across tenants, and so on.
DZone: What are top 5 plugins that you wish users knew about?
Kohsuke: That would be...
and while this isn't a plugin, the matrix project functionality and the distributed build functionality in Jenkins.
These are all fairly generic features that have broad applicability regardless of the programming language or the apps that you are developing. I touch these topics in the training course that CloudBees provides, but too many people don't know about those.
But aside from those, there's this incredible diversity in what users do with Jenkins, that one can't really make a blanket statement that everyone needs plugin X or Y. I mean just in the software development proper, we got people doing embedded C++ projects to PHP websites, enterprise applications in Java to social game development in Ruby. Then there are people outside the software development, such as publishing, education, and so on, using Jenkins to develop books or training materials.
Fostering this diversity and helping people discover what others in a similar situation is doing is very important to me.
DZone: Are there any recent or upcoming plugins for Jenkins that have piqued your interest?
Kohsuke: One of the plugins that I recently did is the git-server plugin. While this server by itself doesn't provide the user visible features, this enables other plugins to turn Jenkins into a Git server. Nowadays I see many developer workflows built around git push/pull, and this plugin allows Jenkins to participate in this workflow in more interesting ways.
Speaking of workflow, creating a workflow on Jenkins by combining various jobs into doing something complicated is another emerging area that I've been very interested in. For example, there's BuildFlow plugin that allows you to use Groovy DSL to script together Jenkins jobs (often parameterized) into a workflow of a sort (say distribute/deploy apps and load test clients, run the load test, then tear down everything at the end.) There's also the Jenkow plugin that approaches a similar problem, but by integrating Jenkins into a full-blown BPMN workflow engine. So this can really model any development related workflow that not just involves Jenkins but all kinds of other things (say check-in approval process on a large software.)
Another area that's making a lot of progress lately is how to deal with a large number of jobs. For example, we did the template plugin in Jenkins Enterprise by CloudBees that help administrators manage a large number of similar jobs. In OSS, there's Job DSL plugin that allows people to use Groovy DSL to define lots of similar jobs without using GUI.
DZone: Have you seen the new JaCoCo continuous code coverage plugin? What's your initial review of it?
Kohsuke: I actually haven't had a chance to use it, but it's great that more and more plugins are written.
On a related node, I'd have liked to use this opportunity to decouple generic code coverage support (that involves visualization of data, source code overlay, etc.) from the parsing of a specific report format. In this way, supporting multiple code coverage tools become a whole lot easier. This is the pattern that worked well for static code analysis tools (FindBugs, CheckStyle, PMD and Android Lint plugins all share much of the code), and test report parsing tools.
Hopefully we can work with the developers of the Jenkins JaCoCo plugin to do this going down the road.
DZone: What are some upcoming changes to Jenkins?
Kohsuke: There are several things that I'd like to really work on. One is the start-up performance improvement of Jenkins by lazy-loading data. Put another way, fix up one of the implementation mistakes from the very early days. Ryan Campbell, my colleague, once told me that I have to do this for the sake of humanity! I keep distracted to this and that and never gotten around doing this yet, but I think all I need is like a few days disconnected from the e-mail/skype/IRC, but we all know those precious productivity time is hard to come by.
Another area that I really like to work on is to make it easier for people to find plugins they need/want and generally improve the update center / infrastructure to deal with ever increasing number of plugins. The original update center was developed when we had a few tens of plugins, and today we have more than 550 plugins. We have lots of ideas, like analyzing what plugins people are co-installing and recommend likely useful plugins, or have a review mechanism that allows people to guess the quality of the plugin, etc.
Then there's fixing bugs and adding little things here and there that people requested. If I look at the issue tracker today, there's more than 15000 tickets filed. While the pace of the development is accelerating, we have many tickets that are not getting attention that they deserve, and I'd like to do something about it.
The interesting thing about open-source project is that I can say what I think the community should be working on, but normally the community just does whatever it wants to do :-). So we'll see how much of these will come into reality next year!
DZone: Tell me about the Jenkins User Conference happening at the end of September.
Kohsuke: The real power of Jenkins comes from your learning how to combine plugins that you can use and making Jenkins work for you in your specific problem domain. Yet as I touched on earlier, there's this incredible diversity in what people do with Jenkins, that it's not like there's one solution that fits everyone.
So in the community, we are trying to encourage users to share what they are doing and how they put together pieces, in the hope that others find them inspirational and applicable to what they do. To me that's what Jenkins User Conference is about. Sometimes those talks come from developers of plugins, who are typically users themselves that decided to scratch their own itch.
We've been doing this all around the world in the last year, and at the end of this September, it is back to San Francisco, where it all started!
The event is situated back to back with JavaOne, to make it easier for people coming to JavaOne to attend. More information about the event can be found in http://www.cloudbees.com/juc2012.cb. I'm looking forward to seeing you there!