Tollef Fog Heen's blog

Before I start, I'll admit that I'm not a real RPM packager. Maype I'm approaching this from completely the wrong direction, what do I know?

I'm in the process of packaging Varnish 3.0.2 which includes mangling the spec file. The top of the spec file reads:

%define v_rc
%define vd_rc %{?v_rc:-%{?v_rc}}

Apparently, this is not legal, since we're trying to define v_rc as a macro with no body. It's however not possible to directly define it as an empty string which can later be tested on, you have to do something like:

%define v_rc %{nil}
%define vd_rc %{?v_rc:-%{?v_rc}}

Now, this doesn't work correctly either. %{?macro} tests if macro is defined, not whether it's an empty string so instead of two lines, we have to write:

%define v_rc %{nil}
%if 0%{?v_rc} != 0
%define vd_rc %{?v_rc:-%{?v_rc}}
%endif

The 0{?v_rc} != 0 workaround is there so that we don't accidentially end up with == 0 which would be a syntax error.

I think having four lines like that is pretty ugly, so I looked for a workaround and figured that, ok, I'll just rewrite every use of %{vd_rc} to %{?v_rc:-%{?v_rc}}. There are only a couple, so the damage is limited. Also, I'd then just comment out the v_rc definition, since that makes it clear what you should uncomment to have a release candidate version.

In my naivety, I tried:

# %define v_rc ""

# is used as a comment character in spec files, but apparently not for defines. The define was still processed and the build process stopped pretty quickly.

Luckily, doing # % define "" seems to work fine and is not processed. I have no idea how people put up with this or if I'm doing something very wrong. Feel free to point me at a better way of doing this, of course.

We use SugarCRM at work and I've complained about its not-very-RESTy REST interface. John Mertic a (the?) SugarCRM Community Manager asked me about what problems I'd had (apart from its lack of RESTfulness) and I said I'd write a blog post about it.

In our case, the REST interface is used to integrate Sugar and RT so we get a link in both interfaces to jump from opportunities to the corresponding RT ticket (and back again). This should be a fairly trivial exercise or so you would think.

The problems, as I see it are:

• Not REST-y.
• Exposes the database tables all the way through the REST interface
• Lack of useful documentation forcing the developer to cargo cult and guess
• Annoying data structures
• Forced pagination

My first gripe is the complete lack of REST in the URLs. Everything is just sent to https://sugar/service/v2/rest.php. Usually a POST, but sometimes a GET. It's not documented what to use where.

The POST parameters we send when logging in are:

method=>"login"
input_type=>"JSON"
response_type=>"JSON"
rest_data=>json($params)

$params is a hash as follows:

user_auth => {
            user_name => $USERNAME,
            password => $PW,
            version => "1.2",
},
application => "foo",

Nothing seems to actually care about the value of application, nor about the user_auth.version value. The password is the md5 of the actual password, hex encoded. I'm not sure why it is, as this adds absolutely no security, but it is. This is also not properly documented.

This gives us a JSON object back with a somewhat haphazard selection of attributes (reformatted here for readability):

{
     "id":"<hex session id>,
     "module_name":"Users",
     "name_value_list": {
             "user_id": {
                     "name":"user_id",
                     "value":"1"
             },
             "user_name": {
                     "name":"user_name",
                     "value":"<username>"
             },
             "user_language": {
                     "name":"user_language",
                     "value":"en_us"
             },
             "user_currency_id": {
                     "name":"user_currency_id",
                 "value":"-99"
             },
             "user_currency_name": {
                     "name":"user_currency_name",
                     "value":"Euro"
             }
     }
}

What is the module_name? No real idea. In general, when you get back an id and a module_name field, it tells you that the id exists is an object that exists in the context of the given module. Not here, since the session id is not a user.

The worst here is the name_value_list concept which is used all over the REST interface. First, it's not a list, it's a hash. Secondly, I have no idea what would be wrong by just using keys directly in the top level object, so the object would have looked somewhat like:

{
     "id":"<hex session id>,
     "user_id": 1,
     "user_name": "<username>,
     "user_language":"en_us",
     "user_currency_id": "-99",
     "user_currency_name": "Euro"
}

Some people might argue that since you can have custom field names this can cause clashes. Except, it can't, since they're all suffixed with _c.

So we're now logged in and can fetch all opportunities. This we do by posting:

method=>"get_entry_list",
input_type=>"JSON",
response_type=>"JSON",
rest_data=>to_json([
            $sid,
            $module,
            $where,
            "",
            $next,
            $fields,
            $links,
            1000
])

• $sid is our session id from the login
• $module is "Opportunities"
• $where is opportunities_cstm.rt_id_c IS NOT NULL. Yes, that's right. An SQL fragment right there and you have to know that you'll join the opportunities_cstm and opportunities tables because we are using a custom field. I find this completely crazy.
• $next starts out at 0 and we're limited to 1000 entries at a time. There is, apparently, no way to say "just give me all you have".
• $fields is an array, in our case consisting of id, name, description, rt_id_c and rt_status_c. To find out the field names, look at the database schema or poke around in the SugarCRM studio.
• $links is to link records together. I still haven't been able to make this work properly and just do multiple queries.
• 1000 is the maximum number of records. No, you can't say -1 and get everything.

Why this is a list rather than a hash? Again, I don't know. A hash would make more sense to me.

The resulting JSON looks like:

{
    "result_count" : 16,
    "relationship_list" : [],
    "entry_list" : [
       {
          "name_value_list" : {
             "rt_status_c" : {
                "value" : "resolved",
                "name" : "rt_status_c"
             },
             […]
          },
          "module_name" : "Opportunities",
          "id" : "<entry_uuid>"
       },
       […]
    ],
    "next_offset" : 16
}

Now, entry_list actually is a list here, which is good and all, but there's still the annoying name_value_list concept.

Last, we want to update the record in Sugar, to do this we do:

method=>"set_entry",
input_type=>"JSON",
response_type=>"JSON",
rest_data=>to_json([
    $sid,
    "Opportunities",
    $fields
])

$fields is not a name_value_list, but instead is:

{
    "rt_status_c" : "resolved",
    "id" : "<status text>"
}

Why this works and my attempts at using a proper name_value_list didn't work? I have no idea.

I think that pretty much sums it up. I'm sure there are other problems in there (such as the over 100 lines of support code for the about 20 lines of actual code that does useful work), though.

Just this morning, I was setting up TLS on a LDAP host, but slapd refused to start afterwards with a bizarre error message:

TLS init def ctx failed: -207

The key and certificate was freshly generated using openssl on my laptop (running wheezy, so OpenSSL 1.0.0d-3). After a bit of googling, I discovered that -207 is gnutls-esque for "Base64 error". Of course, the key looks just fine and decodes fine using base64, openssl base64 and even gnutls's own certtool.

Now, certtool also spits out what it considers the right base64 version of the key and I noticed it differed. Using the one certtool output seems to work, though, so if you ever run into this problem try running the key through certtool --infile foo.pem -k and use the base64 representation it outputs.

It's sometimes necessary to be able to access HTTP resources from inside VCL. Some use cases include authentication or authorization where a service validates a token and then tell Varnish whether to proceed or not.

To do this, we recently implemented libvmod_curl which is a set of cURL bindings for VCL so you can fetch remote resource easily. HTTP would be the usual method, but cURL also supports other protocols such as LDAP or POP3.

The API is very simple, to use it you would do something like:

require curl;

sub vcl_recv {
    curl.fetch("http://authserver/validate?key=" + regsub(req.url, ".*key=([a-z0-9]+), "\1"));
    if (curl.status() != 200) {
        error 403 "Go away";
    }
}

Other methods you can use are curl.header(headername) to get the contents of a given header and curl.body() to get the body of the response. See the README file in the source for more information.

A while ago, we got another machine for hosting Alioth and so we started thinking about how to use that machine. It's a used machine and not massively faster than the current hardware, so just moving everything over wouldn't actually get us that much of a performance upgrade.

However, Alioth is using FusionForge, which is supposed to be able to run on a cluster of machines. After all, this was originally built for SourceForge.net, which certainly does not run on a single host. So, a split of services is what we'll do.

This weekend, we're having a sprint in Collabora's office in Cambridge, actually implementing the split and doing a bit of general planning for the future.

Last afternoon (Friday), European time, we started the migration. The first step is to move all the data off the Xen guest on wagner, where Alioth is currently hosted. This finished a few minutes ago; it turns out syncing about 8.5 million files across almost 400G of data takes a little while.

The new host is called vasks and will host the database, run the main apache and be the canonical location for the various SCM repositories.

We are not decomissioning wagner, but it'll be reinstalled without Xen or other virtualisation which should help performance a bit. It'll host everything that has lower performance requirements such as cron jobs, mailing lists and so on.

I'll try to keep you all updated and feel free to drop by #alioth on irc.debian.org if you have any questions.

Every so often, we get bug reports about Varnish leaking memory. People have told Varnish to use 20 gigabytes for cache and they discover the process is eating 30 gigabytes of memory and they get confused about what's going on. So, let's take a look.

First, a little bit of history. Varnish 2.0 had a fixed per-object workspace which was used for both header manipulations in vcl_fetch as well as for storing the headers of the object when vcl_fetch was done. The default size of this workspace was 8k. If we assume an average object size of 20k, that is almost 1/3 of the store being overhead.

With 2.1, this changed. First, vcl_fetch doesn't have obj any longer, it only has beresp which is the backend response. At the end of vcl_fetch, the headers and other relevant bits of the backend response are copied into an object. This means we no longer have a fixed overhead, we use what we need. Of course, we're still subject to malloc's whims when it comes to page sizes and how it actually allocates memory.

Less overhead means more objects in the store. More objects in the store, means, everything else being equal, more overhead outside the store (for the hash buckets or critbit tree and other structs). This is where lots of people get confused, since what they see is just Varnish consuming more memory. When moving from 2.0 to 2.1, people should lower their cache size. How much depends on the amount of objects they have, but if they have many and small objects, a significant reduction might be needed. For a machine dedicated to Varnish, we usually recommend making the cache size be 70-75% of the memory of the machine.

A reasonable question to ask at this point is what all this overhead is being used for. Part of it is a per-thread overhead. Linux has a 10MB stack size by default, but luckily, most of it isn't allocated, so it only counts against virtual, not resident memory. In addition, we have a hash algorithm which has overhead and the headers from the objects are stored in the object itself and not in the stevedore (object store). Last, but by no means least, we usually see an overhead of around 1k per object, but I have seen up to somewhere above 2k. This doesn't sound like much, but when you're looking at servers with 10 million objects, 1k of overhead means 10 gigabytes of total overhead, leading to the confusion I talked about at the start.

Last night, I finally got my temperature sensors going, including a nice and shiny munin plugin giving me pretty graphs. So far, I only have a sensor in the loft, but I'll spend some days putting sensors in the rest of the house as well.

Robert McQueen asked me on twitter how this all was set up, so I figured I'd blog about it. The sensors I'm using are the DS18B20 ones from Dallas Semiconductor. You can probably buy them from your local electronics supplier, but mine charges around 75 NOK a piece, so I just bought some off Ebay. It takes a bit longer, but I paid about 1/10th the price.

For logging, I'm using my NAS, which is just a machine running Debian, an USB to serial adapter and an serial-to-1-wire adapter. Thanks a lot to Martin Bergek for the writeup and the ELFA part numbers for diodes.

Since I'm lazy, I ended up just writing a plugin for munin. It uses owfs, which I downloaded from mentors.debian.net. I also offered sponsorship for it, assuming a few small issues are cleaned up, so hopefully you can install using just Debian in the near future.

owfs is fairly easy to work with, and the plugin uses the aliased names if you provide aliases, so you can know what the temperature in a given location is, rather than having to remember 64 bit serial numbers.

GNOME's current sysadmin team is entirely volunteer-based, but as they are having problems finding enough (trusted) volunteers they are looking at hiring a part-time sysadmin. From looking at the GNOME wiki, it looks like they have had a meeting about the shortage of sysadmins. Citing from the minutes

The biggest problem that we've always had with the maintaining an active sysadmin team is the need for trust. If somebody shows up and wants to help out with a GNOME coding project, then it's easy to build up trust over time. Suggest a project, have the person send patches, review the patches, if the patches are good, eventually give them direct commit access. However, for sysadmin work, we get a lot of people who want to help out, but it's very hard for someone to contribute without being given a "dangerous" level of access to the GNOME systems.

Without having looked very hard, I would guess at the GNOME infrastructure being about as open as most proprietary software projects. There's no way for me, as a third party to take a look at their infrastructure, take a look at their ticket backlog and submit patches for problems. Similarly, their nagios setup is behind a password prompt, so there's no way for me to look at what services often have performance problems, suggest new monitors or point out any servers or services that are not monitored.

I'm not saying this to pick on GNOME, and as I'll touch on below, they do seem to mostly do the right thing, and as one of the Freedesktop.org sysadmins, I know we're not any better at least not yet.

One way to make it at least somewhat easier to contribute and get involved is to use a tool like Chef or Puppet and publishing the recipes. This won't magically make everything transparent, but it'll be a big step up. Ideally, the recipes should be complete enough that you can bootstrap a working system from them and so easier reproduce the infrastructure and any problems. It seems like GNOME is using puppet, but I couldn't find the recipes.

Moving a complete infrastructure from something managed by hand to something managed using automation tools is a fairly big and involved process. However, if you're serious about getting more people involved in your sysadmin team, I think it's one of the more reasonable ways to opening up. It also means that when one of your servers is stolen, catches fire or suffers other catastrophic failure you can rebuild the service much quicker.

My last point is to open up your ticket tracker. Most tickets aren't security sensitive, so provide a way for people to mark those tickets that are sensitive as such and make the rest public. The GNOME wiki makes this a bit confusing as it talks a bit about RT, but it seems like they actually use bugzilla for sysadmin tickets and just hide security-sensitive ones.

Russell Coker recently reviewed the Yubikey. The article mentions me, so I figured I'd correct a minor thing and respond to one of the comments.

First, the yubikey-server-c is my reimplementation of the Yubikey authentication protocol. Yubico provides two implementations, one in PHP and one in Java, neither which I'm particuarly interesting on building my system security on. Any bugs, misfeatures, etc in the C implementation are mine and mine alone.

Barak A. Pearlmutter, one of the commenters on Russell's blog writes:

i don’t understand. isn’t this thing vulnerable to eavesdropping and replaying? even if it has a counter which changes etc, the things it is talking to (web sites) can’t know that some generated string is being reused. and it doesn’t even have a clock, so these things can be old.

The way the Yubikey works is you have a central authentication server. This has a secret shared with the key. Setting this secret is the primary function of the personalisation tool. When you press the button, the key takes its internal state (various counters, uid field, etc) and encrypts this using AES-128. This is then sent to the application you are trying to access, be it Wordpress, SSH or something else. Said application then contacts the authentication server which decrypts the ticket, checks the values of the counters to make sure it's not a replay and responds with OK, bad ticket, replay and various other status codes. Based on this, the application grants or denies access.

There are really two places you could attack this: in the communication between the web browser and application or between application and authentication server. Both of those can be secured using SSL.

There is no way to use a single yubikey in multiple authentication realms without extra software. To do this, you would have a OpenID provider that uses the Yubikey for authentication, or you could have a Kerberos server with cross-realm trust.

As for the PAM modules and other tools so far not being packaged, yes, I know, I might fix it, but the current setup has the bits I use, as I use RADIUS authentication to get services to support both Yubikey and passwords.

I recently upgraded kemper.freedesktop.org to lenny. Collabora are nice enough to sponsor some of my sysadmin work for freedesktop and so making sure we are actually running a supported distribution was a good start. The actual dist-upgrade went fine, but when I rebooted with a 2.6.26 kernel, it just hung in the early boot phase. Luckily, a newer kernel worked fine. However, a newer kernel also breaks the NFS kernel server in Lenny. A short backport later, NFS was working fine, except annarchy (which NFS mounts from kemper) didn't have nfs-common installed at all, meaning it lacked mount.nfs. Ooops.

Now, bugs was broken. It used an SSH tunnel from annarchy to kemper, but the startup script was nowhere to be found. I replaced it with a trivial stunnel setup which has the added advantage of reconnecting if the tunnel goes down.

The ssh config had to be fixed slightly. We used to use an old and patched sshd that stored all the keys in a single file. I added a tiny script to split that again. We also had MkHomeDir in sshd's config, now replaced with pam_mkhomedir.

Another interesting thing I learnt is that the iLO ssh daemon chucks you out if you try to send enviromental options to it. Like, LANG which is sent by default. Slightly confusing, but easy enough to fix once I knew what the problem was.

In addition to kemper, I upgraded, but did not reboot fruit (the admin and LDAP host), due to not having the iLO password. I did not want to risk sitting there with a non-booting machine I could not fix. It's going to be rebooted at some later stage. I also did not have the iLO password for gabe, which runs mail and some other faff, so I'll have to schedule some more downtime in the near future.