> Daily - or even more often. It's tempting to do continuous > builds, but you probably can't, because of source control issues > which I'll talk about in a minute.

Hmm.... I don't see anything about those issues later in the column. I'm curious, since a continuous build server is something on which I've done a little research, but I haven't got to the point of live implementation yet.

-- Anonymous, February 01, 2001

Answers

It basically comes down to the fact that programmers will sometimes do checkins that don't actually break the build, but which aren't of high enough quality to make even a daily build -- usually because they have several complicated sets of changes in several places which need to be checked in in stages.

Also, depending on how many bits you keep from each build, you will need an outrageous amount of storage if you make too many builds.

And finally, because life is just easier when there is a more finite number of recent builds that are being tested... e.g., everybody remembers that Tuesday's build was pretty good, and that QA is banging on Tuesday's build.

-- Anonymous, February 07, 2001

You might want to take a look at this article.

http://www.martinfowler.com/articles/continuousIntegration.html

-- Anonymous, February 02, 2001

Could we work in such a way that we never had to do multi-stage check- ins of complicated sets of changes? Could we get our build quality so consistent that nobody cared which particular Tuesday build QA got? (For that matter, could we get it so consistent that QA was happy to always be working with the latest build?) What benefits might we see if we could? Is it worth it?

I've got my own answers to these questions, but I'd like to know what you all think.

-- Anonymous, February 07, 2001

How do continuous builds avoid the obvious race condition where I check in a bunch of files, but the build machine is simultaneously syncing and misses a few of my check ins? At least with a daily build, everyone knows exactly when all their files need to be checked in by. Mozilla does continuous builds, so they must have solved this problem.

btw, I love daily builds. I would not want to work on a project that built less frequently. Plus, as mentioned above, daily builds provide QA a new well-known build to test each day. The QA process would get muddied if they get respones from developers such as, "why are you testing the 10:13am build? I fixed that bug in the 2:48pm build. Better restart your regression tests." ;) Having worked on both sides of the dev/QA fence, I can imagine this scene very easily.

-- Anonymous, February 09, 2001

On my old team, we avoided the race condition by using a little bell. You weren't allowed to check in unless you were holding the bell. When you wanted to check in, you grabbed the bell, checked in, and hit the Build button. (You also rang the bell when you were done, which got annoying pretty quickly. Next time I'm using a koosh ball. :)

I'm sure you can find a more, um, dignified solution, if you want to. The important thing was that the build was pretty quick, so it wasn't a problem for us to just check in one at a time.

I can certainly imagine the other scenario you describe, but in my head, it turns out differently. Developer says, "Yeah, I fixed that an hour ago. You'll see the fix when you run your tests tomorrow." Doesn't seem like such a big headache.

Even better (and here I'm being REALLY idealistic), can QA automate their tests? I'd like to see a QA cycle that takes MINUTES.

-- Anonymous, February 09, 2001

but a physical token (like a bell or koosh ball) does not prevent an automated build machine from syncing. For example, Mozilla's Tinderbox "is a flock of dedicated build machines that do nothing but continually check out the source tree and build it, over and over again." How do they prevent the build machines from syncing only some of a developer's updated files? Their Tinderbox home page does not seem to say..

Tinderbox home page

-- Anonymous, February 09, 2001

Does your project need that level of complexity? How many people have you got? How fast can you make your build process?

-- Anonymous, February 09, 2001

How do they prevent the build machines from syncing only some of a developer's updated files?

Aha! The correct answer is "they don't". That's the whole reason for the "status" part of Tinderbox. Tinderbox will show whether the build is currently "broken" or in a compilable state.

After doing some more work on this, I'm coming to the following strategy:

1. Build engines work opportunistically, either by notification of checkin if possible, or by periodic polling if not.

2. A build engine gives a window after first checkout. It waits until a configurable amount of time has happened since a checkin (say, on the order of fifteen minutes). At that point, it does a sync against the time of day when the last checkin was made. This prevents breakage due to people making new checkins while the sync is going on.

3. The build is made, packaged, and smoke tested. Errors in compilation, packaging, or smoke testing are reflected in status and possibly in emails--only to the developer who checked in the apparently-offending code. No need to bother everyone (or a beleaguered buildmaster or tech lead), and if the developer isn't the offender, she can determine what or who did it.

4. Periodic (daily or weekly, depending on the project) "blessed" builds are sacred. No code is allowed to be partially checked-in or modified during this time. This is the only time when "breaking the build" is a mortal sin.

5. QA typically uses the periodic "blessed" builds for testing. The interim builds are for development use or for QA use in special cases ("I've got a couple minutes; did Henry's checkin from this morning actually fix that problem?").

What do you think?

-- Anonymous, February 09, 2001

One of the reasons that the Mozilla Tinderbox is a continuous (or nearly sometimes so), process is that the build is geographically neutral. A lot of developers aren't even in the same timezone but three to the build machines, whether that's independants or NSCP employees.

However, I also run my own pull and build daily scheduled usually for the middle of the night PDT, though given some of the late night habits of some that breaks me as well so I end up aborting a build or running it later than usual.

Relying on Tinderbox when you have dependant builds of your own is dangerous and my fingertips are blistered from previous build problems. So the rule remains, and I'd carry this to any team, build your own crap before subjecting it to anyone else.

Sliver (Simon) That build is in addition to the build I run of my own developments on my own mangled trees.

-- Anonymous, March 08, 2001

Thanks, Simon! I was really hoping to hear from someone who'd worked with Tinderbox, since it seems to be the closest implementation I've ever seen to what I have in mind.

-- Anonymous, March 08, 2001

Tinderbox or something like it is absolutely necessary for an open, distributed project that runs on many different platforms such as Mozilla. If your own needs include multiple platforms and unattended builds then it may well suit you well.

The scripts for Tinderbox live in the tools tree in mozilla.

Simon

-- Anonymous, March 09, 2001

I just visited Martin Fowler's website (www.martinfowler.com) and saw that ThoughtWorks released a continuous build server at http://cruisecontrol.sourceforge.net/.

It's a java based solution with a servlet log and it uses customized ant tasks to check the source control system for updates. It says it's pretty easy to add new source control products (it already support cvs)

Chris

-- Anonymous, April 06, 2001

The "problem" of an automated build picking up code halfway through a checkin is not hard to solve... just have it wait for a "quiet period" of 30 seconds or whatever of no checked activity. I've used Cruise Control (which works well, but is missing a couple of thing), it works this way.

Getting confirmation from outside your own machine that whole system builds and passes unit tests is very nice... once you get used to it you will not want to give it up.

There is nothing to stop you from picking a build once a day or whatever to call your Daily Build to send over to some testers or whatever.

-- Anonymous, June 03, 2001