Debian Lenny: / on LVM2 on dm_crypt on raid

This is just Google-fodder:

If you use the lenny (beta2) installer to setup a system which uses / on LVM on dm_crypt on raid, then the first reboot fails with something like “cannot find LVM volumes”. After a few minutes you get dumped into a busybox shell.

Getting the system to boot

Run cryptsetup luksOpen /dev/md1 md1_crypt (or md0, depending on your setup). Exit busybox and the system should boot through. Then …

Fixing this for good

It seems to be that this Wiki page still applies. Thus you have to create a line in /etc/crypttab containing (in my case md1)

md1_crypt /dev/md1 none luks

and run update-initramfs -u to update the initrd.

That’s it.

RT(IR) Command line ticket creation + Autoreply

This is one of the “simple” problems which always take longer to solve than expected.

The goal: Script RTIR to generate Investigations associated with an Incident.

In order to do that, I needed to solve the following who subproblems:

  1. Being able to use the “edit” feature in the command line “rt” client and capture the new ticket’s id
  2. Basically, I needed to capture the STDOUT of the “rt” command, and let also rt invoke vi. My solution:

    --- /usr/bin/rt 2007-10-31 11:52:20.000000000 +0100
    +++ /usr/local/bin/rt 2008-08-20 14:14:20.000000000 +0200
    @@ -1335,7 +1335,7 @@
    local $/ = undef;

    open(F, ">$file") || die "$file: $!\n"; print F $text; close(F);
    - system($editor, $file) && die "Couldn't run $editor.\n";
    + system("$editor $file < /dev/tty > /dev/tty") && die "Couldn't run $editor.\n";
    open(F, $file) || die "$file: $!\n"; $text = ; close(F);
    unlink($file);

  3. Add attachments during ticket creation
  4. The “Autoreply on Create” Scrip is supposed to mail out the newly created content to the Requestor, thus I needed to not only provide plaintext while the creation, but also all the attachments that need to be sent out to the Requestor.

    Adding text is simple, just use “Text: foo bar” to the Template. But attachments are simply not supported for “rt create” (and not that easy to add in the code). They are supported for “rt correspond”, so I can sidestep the “attachment on create” problem by simply splitting up the work into an empty “rt create” followed by a “rt correspond -a file”.

    There is just one gotcha: The Autoreply script will happily mail on ticket creation with an empty mail.

    Solution: Change the “Condition = On Create” for the Scrip to “Custom” with

    my $t = $self->TransactionObj;
    return 0 unless ($t->Type eq "Create");

    my $c = $t->ContentObj;
    return (defined($c)) ? 1 : 0;

New version of mod_epp released

I finally manged to crank out a new release of mod_epp. So, what’s new?

Last November Gavin Brown sent me a bug report regarding the handling of large answers. I provided a quick fix back then but did no official release.

While debugging that issue, I noticed that mod_epp didn’t generate the greeting on newer apaches. Tracing that, I found that the kernel didn’t return from accept on connection open, but on the first data reception. First I suspected the epoll interface, but some more intensive debugging found a more mundane reason: setsockopt is used to set:

TCP_DEFER_ACCEPT

Allows a listener to be awakened only when data arrives on the socket. Takes an integer value (seconds), this can bound the maximum number of attempts TCP will make to complete the connection.

Some more googling found the answer: you need to use the AcceptFilter directive to disable this optimization for the EPP socket.

This is not really documented, thus this bug report.

Watching logfiles with Cacti

Assume you have a number of patterns you might want to watch out for in various logfiles. These could be harmless, e.g. postfix/smtp\[[[:digit:]]+\]: .* status=sent which indicates that a mail was sent by postfix, or could be bad, e.g. indications of hardware troubles, firewall log entries or suspicious entries in proxy logs.

There are a number of tools which can help you do that, e.g. logwatch can produce nice reports for you.

In my case, I was less interested in individual log entries, but in a graphical representation of their frequency. For that, I need something similar to MRTG. Cacti is a decent graphical frontend to the core MRTG engine, rrdtool. It provides various data sources and graphing options all packed up into a neat web interface.

So, what needs to be done to get Cacti to graph match-rates of regexs in logfiles?

Continue reading Watching logfiles with Cacti

HP DLxxx, iLo2, XEN and remote consoles

As a general principle, servers with PC-style BIOS suck. If you ever worked with a real server with native support for a serial console, then anything else feels like a crude crutch.

HP servers with iLo2 are no exceptions.

At least unless you’re unwilling to shell out additional money for the feature packs.

You get a crude vt100 emulation during the boot sequence, but once the operating system starts, this falls back to a real serial device. The trick is thus to configure grub, the kernel, and the gettys in a way to get consistent console access all through the boot process.

Now add XEN and and root in an encrypted LVM volume to the mix and things start to become nasty. So here is my solution, for all those weary souls googling for a recipe for Debian (etch):


title Xen 3.0.3-1-i386-pae / Debian GNU/Linux, kernel 2.6.18-6-xen-686
root (hd0,0)
kernel /xen-3.0.3-1-i386-pae.gz dom0_mem=128M com1=115200,8n1,0x2f8,3
module /vmlinuz-2.6.18-6-xen-686 root=/dev/mapper/vg_main-lv_root ro console=ttyS0 xencons=ttyS0
module /initrd.img-2.6.18-6-xen-686

And don’t forget to note these parameters in the comments in /bot/grub/menu.lst unless you want to manually add them after each kernel upgrade. i.e.:


## Xen hypervisor options to use with the default Xen boot option
# xenhopt=dom0_mem=128M com1=115200,8n1,0x2f8,3

## Xen Linux kernel options to use with the default Xen boot option
# xenkopt=console=ttyS0 xencons=ttyS0

If you’ve forgotten that, you can save yourself the trip to the server room by manually editing the boot config using grub’s edit functionality.

Enabling login via ttyS0 is then just a matter of
T0:23:respawn:/sbin/getty -L ttyS0 115200 vt100
in /etc/inittab, and ttyS0 in /etc/securetty.

Once you enable ssh access in iLo2, you can reach the console of your server from any *ix shell via a simple ssh command.