System Administration

Debian Lenny: / on LVM2 on dm_crypt on raid

This is just Google-fodder:

If you use the lenny (beta2) installer to setup a system which uses / on LVM on dm_crypt on raid, then the first reboot fails with something like “cannot find LVM volumes”. After a few minutes you get dumped into a busybox shell.

Getting the system to boot

Run cryptsetup luksOpen /dev/md1 md1_crypt (or md0, depending on your setup). Exit busybox and the system should boot through. Then …

Fixing this for good

It seems to be that this Wiki page still applies. Thus you have to create a line in /etc/crypttab containing (in my case md1)

md1_crypt /dev/md1 none luks

and run update-initramfs -u to update the initrd.

That’s it.


A rant

I’m using the AvantGo client on my nokia phone to read newspapers on the subway from/to work (and while waiting for Clemens to go to sleep). Yesterday, it stopped working. Today I debugged it and discovered that the nameservers of A1 (Mobilkom Austria) simply cannot resolve The domain is fine, and IP connectivity towards them is also OK via the cell phone.

So I called up their hotline. So far, they can reproduce the problem. No solution, other than entering the IP address in the browser. If this isn’t just a bug, but a deliberate blocking, I’m going to raise hell. I’m in a bad mood.

[Update: they managed to fix this within a day. Good.]

And today’s Microsoft patchday seem to have broken switching from internal to external screen on my Thinkpad.


I’m downloading the latest Lenovo updates at speeds worthy of last century. Let’s see if that helps.

CERT System Administration

RT(IR) Command line ticket creation + Autoreply

This is one of the “simple” problems which always take longer to solve than expected.

The goal: Script RTIR to generate Investigations associated with an Incident.

In order to do that, I needed to solve the following who subproblems:

  1. Being able to use the “edit” feature in the command line “rt” client and capture the new ticket’s id
  2. Basically, I needed to capture the STDOUT of the “rt” command, and let also rt invoke vi. My solution:

    --- /usr/bin/rt 2007-10-31 11:52:20.000000000 +0100
    +++ /usr/local/bin/rt 2008-08-20 14:14:20.000000000 +0200
    @@ -1335,7 +1335,7 @@
    local $/ = undef;

    open(F, ">$file") || die "$file: $!\n"; print F $text; close(F);
    - system($editor, $file) && die "Couldn't run $editor.\n";
    + system("$editor $file < /dev/tty > /dev/tty") && die "Couldn't run $editor.\n";
    open(F, $file) || die "$file: $!\n"; $text = ; close(F);

  3. Add attachments during ticket creation
  4. The “Autoreply on Create” Scrip is supposed to mail out the newly created content to the Requestor, thus I needed to not only provide plaintext while the creation, but also all the attachments that need to be sent out to the Requestor.

    Adding text is simple, just use “Text: foo bar” to the Template. But attachments are simply not supported for “rt create” (and not that easy to add in the code). They are supported for “rt correspond”, so I can sidestep the “attachment on create” problem by simply splitting up the work into an empty “rt create” followed by a “rt correspond -a file”.

    There is just one gotcha: The Autoreply script will happily mail on ticket creation with an empty mail.

    Solution: Change the “Condition = On Create” for the Scrip to “Custom” with

    my $t = $self->TransactionObj;
    return 0 unless ($t->Type eq "Create");

    my $c = $t->ContentObj;
    return (defined($c)) ? 1 : 0;


DNS: The sky is falling

In April I speculated about the impending doom of the DNS.

Now we know what was in the works, and yes, it’s not a pretty picture.

My idea from april doesn’t work 1:1, as the attacker doesn’t attack a single target, sondern arbitrary other hostnames in the same domain.

Anyway, I spent the last days analyzing data from .at nameserver regarding patch discipline in Austria. You can read the depressing results here.

System Administration

New version of mod_epp released

I finally manged to crank out a new release of mod_epp. So, what’s new?

Last November Gavin Brown sent me a bug report regarding the handling of large answers. I provided a quick fix back then but did no official release.

While debugging that issue, I noticed that mod_epp didn’t generate the greeting on newer apaches. Tracing that, I found that the kernel didn’t return from accept on connection open, but on the first data reception. First I suspected the epoll interface, but some more intensive debugging found a more mundane reason: setsockopt is used to set:


Allows a listener to be awakened only when data arrives on the socket. Takes an integer value (seconds), this can bound the maximum number of attempts TCP will make to complete the connection.

Some more googling found the answer: you need to use the AcceptFilter directive to disable this optimization for the EPP socket.

This is not really documented, thus this bug report.

CERT System Administration

Watching logfiles with Cacti

Assume you have a number of patterns you might want to watch out for in various logfiles. These could be harmless, e.g. postfix/smtp\[[[:digit:]]+\]: .* status=sent which indicates that a mail was sent by postfix, or could be bad, e.g. indications of hardware troubles, firewall log entries or suspicious entries in proxy logs.

There are a number of tools which can help you do that, e.g. logwatch can produce nice reports for you.

In my case, I was less interested in individual log entries, but in a graphical representation of their frequency. For that, I need something similar to MRTG. Cacti is a decent graphical frontend to the core MRTG engine, rrdtool. It provides various data sources and graphing options all packed up into a neat web interface.

So, what needs to be done to get Cacti to graph match-rates of regexs in logfiles?


Please welcome: Elena

Just in case somebody was wondering why my blogging was non-existent recenty:


Born on May 22th, she certainly made my life even more interesting and, in close coordination with Clemens, the two children manage to consume my energy and spare time.

Up to date picture are at my gallery.


From the archives: Some thoughts on RUCUS

On Jan. 30th, 2008, I sent the following to the sipping list. It is a good summary of some of my thoughts on the RUCUS problem statement and on which basic premises need to be sorted out first.


New I-D on ENUM for loose-route SIP

I’ve submitted a new I-D defining an enumservices subtype for loose-route SIP according to J. Rosenberg’s UA loose route (which right now is one of multiple proposals to address one problem).

The basic idea is the following: SIP proxy should distinguish between “retargeting” and “routing”.

  • “retargeting” is done, whenever a proxy decides that this call should be directed towards a new destination. Consequently, the Request-URI changes.
  • “routing” happens, when a proxy forwards the SIP message to the next hop, but does not change the identification of the target.

In the context of ENUM, the destination of the call is identified by a E.164 phone number (whether that number is encoded in the user-part of a sip: URI or a in a tel: URI doesn’t really matter). That number is the key for the ENUM – lookup which returns (in most cases) a SIP-URI.

The current RFCs define this as a retargeting operation: the phone number is mapped to a SIP AoR, and from now on the call is towards that URI and the original phone number is no longer relevant.

If you look at what currently done in private/carrier-ENUM settings, then this is not how ENUM is used: In most cases there, ENUM returns the next-hop for this call towards the phone number. That next hop re-extracts the phone number from the Request-URI and applies his own number/prefix based routing to the call. In other words, this is “routing” operation.

My draft makes this explicit: if the service field in the NAPTR is “sip:lr”, then this records contains a next hop and does not rewrite the destination of the call. Here is the example from the I-D:

   To visualize the difference between how "sip" and "sip:lf" entries
   are interpreted, consider the following entries:

             @  IN NAPTR  ( 100 10 "u"
                            "!^.*$!!" .
             @  IN NAPTR  ( 100 10 "u"
                            "!^.*$!;lr!" .

   A SIP proxy dealing with a call to tel:+441154960496 can select
   either record.  The first leads to

           INVITE SIP/2.0

   being sent to the proxy responsible for  If the sip:lr
   record is used, then

           INVITE tel:+441154960496 SIP/2.0
           Route: <;lr>

   is sent to

May Weather

It’s time for the annual proof that god favors left over right, but this time it’s not that easy for him.

Some background: traditionally, the social democrats (which used to be called socialists) hold a rally on May 1st on the Vienna Ringstrasse. Over the last years, that has been more and more a relic of old times: various delegations of workers, unions, retirees clubs and whatnot else affiliated with the SPÖ marching and showing flags. Of course there are speeches, the (traditionally red) major and (if they are lucky) their own federal chancellor rally their people. After the official part, there is a big party in the Prater, with music for all tastes (incl. western&country (complete with line/square-dancing)). That’s usually very nice: the chestnut trees are in full bloom and …

… the sun shines. Always.

The major right-wing party also organizes annual festivities some other days in May: they hold in the the center of the city, complete with music and other attraction. But ..

… the weather sucks. Always.

So this year they are going for something new: They scheduled their Stadtfest for May 2nd – 4th. I see this as a challenge for Petrus: can he turn around the weather in just one night?

Oh, and there is another data-point regarding god’s political stance: The communist’s “Volksstimmenfest” also gets doused in rain every year.

[UPDATE, May 1st]

The ÖVP’s jinx worked, we didn’t have the usual bright day, there even were a few drops of rain. Let’s see what tomorrow will be like.

[UPDATE, May 3rd]

Well, well, well. It looks like the ÖVP got the better wetter this year. Will pigs fly next year?