Tollef Fog Heen's blog

15:58 [Canonical] -!- Irssi: Disconnecting from server irc.canonical.com: [kthxbye!]
15:58 [Canonical] -!- Irssi: Connection lost to irc.canonical.com

From tomorrow on, I'm working full-time for Linpro, a Norwegian Linux consulting company.

Gunnar Wolf claims that Ubuntu ships a passwordless sudo by default. This would be an insane default configuration, so Ubuntu doesn't. What they do however is add the default user to the admin group which is allowed to use sudo.

Incidentially, Debian does the same thing (except it's just for the first user, not the admin group) if you don't set a root password.

One of the jobs of an archive administrator in Ubuntu processing sync requests. The job is fairly simple: read a sync request (in the form of a bug report), make sure it includes the relevant information and is either filed by or seconded by a person with the appropriate permissions. Then, it's downloading the source, injecting it into the correct queue and marking the bug as closed.

The by far most boring bit of this is actually closing the bugs: opening the bug report, clicking on the relevant task, marking as "Fix released", assigning to myself, pasting the update report from an editor and clicking "submit". Rinse and repeat, today for 73 bugs.

To help this, I started looking into writing a greasemonkey script. Just add a button besides the submit button which would then be labeled DTRT or something like it, but ran into some trouble which is really obvious: Javascript run in the content's security context can't access your clipboard. A small hack to greasemonkey.js fixed this and I now have a shiny GM_fromClipboard function. After playing around with this for a while, I thought it wouldn't help me at all since the javascript is called in the page content's security context, but any event listeners I add seemingly aren't. Nice. (This is of course due to the whole concept of closures and how Javascript works.)

Anyway, I ended up with a script that does the right thing. It needs a greasemonkey patch.

So, the Ubuntu release candidate was released today. As a release manager, it's a fascinating process. First the development where there is relativetly little central control: People work on their specs and my job as a relase manager is to roll new alpha/snapshot releases every couple of weeks. Those are lightly tested (does it boot and install on at least one machine?) and if a derivative or an architecture isn't ready, well, then it isn't ready.

Beta, the release candidate and the release are completely different beasts. We have test plans, people are assigned tests and so on. In addition, we have a freeze which in total lasts about a week for beta, two weeks for release. Every upload has to be hand-checked and approved. As the release grows nearer, the bugs have to be more severe in order for an upload to be approved and in the end it's more or less a full commitment "we have this, we have tested this thoroughly and there is no way we can do a full test and still release on schedule".

At some point, it gets scary. There is just one command left to run; sync-mirrors. No arguments, just the command. I pushed the button, and we are now live.

(and what we want to do to avoid similar problems in the future)

A few days ago, a broken xserver-xorg-core was uploaded to dapper-updates. This caused, unsurprisingly, a large amount of bug reports. A fixed package was uploaded about 12 hours after the first bug report came in. We also pointed all the $countrycode.archive.ubuntu.com (se.archive.ubuntu.com, au.archive.ubuntu.com, etc) to archive.ubuntu.com to speed up the propagation of the fix. Even so, it hit far, far too many users.

Incidentially, the distro team is currently in a sprint in Wiesbaden, Germany and we had a large discussion this morning about both how to handle such situations when they happen (and recover from them, mostly in the technical sense) and how to prevent them from happening in the first place. The latter is obviously more important as we won't have to recover if there is not a problem in the first place. Note that those ideas listed below were ideas and not finished procedures and we are going to write up a proper policy document.

Ideas for prevention ranged form using $distro-proposed and explicit call for testers of that to getting more code review for updates. The current review process for updates to $distro-updates is a review by the release manager who accepts it into $distro-updates. This update passed that review even though the update was faulty, so even if we had another reviewer, it might have passed that review too. Using $distro-proposed does not prevent the problem from affecting anybody, it just changes the group affected with the goal that the group choosing to enable $distro-proposed will hopefully be able to recover more easily.

Recovery ideas ranged from being able to put an update onto a user's machine whether he wanted it or not to snapshot/rollback support and having some kind of a "safe mode" where it would give you a kdrive-based VESA X server and tools to fix your system. I'm not sure what we would do if we managed to break one of those tools (or the "safe-mode" X server), though.

We all agreed that being open about the problem, the cause of the problem and how we are working to solve it is important. Downplaying the severity or making jokes or sarcastic comments about the fix ("Ooops, we did it again") is bad and something we shouldn't do. (And I don't think we did it this time either.) No response is equally bad and something we should work hard to avoid.

Hopefully, we will have a procedure in place in a little while which tells us how to handle such an emergency when it happens and we will be deploying safeguards to prevent it from happening again while not ending up paralysing us and making us unable to deploy updates to $distro-updates as this is something we need to be able to.

While this post is fairly critical of the current set of policies and procedures, I have tried hard to avoid pointing fingers at anybody in particular. I would much rather have us have a positive and constructive discussion about how to avoid similar problems in the future than a discussion on who is to blame. The points and views in this post is also those of my own and not any kind of official communication from Canonical or Ubuntu.

A bit later than the others, with a bit of manual hacking of debian-cd scripts as well as the publish-release script (so we didn't lose the other arches) and with a great effort from Fabio, Adam and Celso, Dapper for SPARC (with Niagara support) is now out. It got delayed due to the tg3 NIC on the T1000 and the kernel not playing to nicely together.

So, I did my first Ubuntu release today. Not a real release, but it felt real enough anyway. Flight 5, the fifth alpha release of Ubuntu 6.04 is out. Colin was nice (as usual) and helped me through the whole process and it was quite painless. Yay, fun. Even though it was work on a Saturday. Also a big thanks to Adam for tag-teaming with me and making sure that most of the issues were taken care of when I came in on Friday.

Also: PENGUINS. Even though "little penguin" is a silly name.

Everybody who has used an Ubuntu live cd over the last nine months or so has used casper. It started out as a special udeb, called by the debian-installer code to bootstrap a live environment. While d-i is fairly flexible, this was stretching the limits and not really a great solution. Amongst the problems were user interactivity halfway through the boot and a very slow boot.

In the middle of December, mdz asked me if I could take a look at implementing the SimplifiedLiveCD specification. As I had played a bit with casper already, I did. Casper is nothing like what it used to be, it now uses initramfs, so no user interactivity after the bootloader. It uses unionfs where available, which speeds it up a fair bit (compare to devmapper + cloop), and if the cd image has squashfs, it uses that too, which makes it even faster. Boot time improvements from around 368 to about 231 seconds is fairly good, but I hope to get it even lower.

What I really, really like about casper however is how hackable it is. I added cd integrity check in less than a day (modulo some bugs in usplash I had to fix). Today, I integrated it with the new usplash in initramfs, so we actually have progress in the initramfs as well. (Instead of "mounting root file system" taking about 40 seconds.)

Another neat feature is the persistence support. It will now look for filesystems with the label casper-cow (that will be changed to ubuntu-live-rw, I think) if persistent is seen on the kernel command line. This makes it easy to drag your setup around with just an USB key and any Ubuntu live cd.

Next out is getting keyboard selection better and more speedups.

Last night, I ended up hacking on the usplash and casper codebases until about 0500 (local time) in the morning (mostly due to the developer status meeting at 0200 UTC).

usplash had a fairly icky bug where it would choke and die if the fifo filled up and it got "incomplete" commands. No longer, it now uses a buffer which it fills up and then processes, handling partial commands and such correctly.

casper-md5check is a tool which does md5summing according to a list of files, similar to the regular md5sum program, but with one notable exception: it does progress information through usplash. So, since the live CD has a huge file which is the compressed live file system, just doing a per-file progress bar would be silly and inaccurate. It therefore does a size-based progress bar which looks quite neat.

If the debian-cd config has already been updated, just pulling down today's daily cd and choosing "integrity check" should show you the nice little hack.

Stephan Hermann complains about the missing hwinfo in Ubuntu. Well, it's not missing, it's just a few versions behind:

Package: hwinfo
[...]
Version: 8.38-3

Also, the -c parameter to wget is useful.