Statistics and the Wall Street crash

I stumbled on an article about Nassim Taleb’s book “The Black Swan” recently. Quick summary: It’s very easy to make mistakes when applying the mathematical theory of statistics to the real world. Not everything in the world can be described as independent random variables with clearly defined properties.

So whenever you put real money behind your statistical reasoning, you better be really sure that your calculations are correct and that all the preconditions regarding applicability of the theory are met.

Furthermore, the rare event you are tempted to ignore (the “this won’t happen in my lifetime” syndrome) may have such bad results that you just shouldn’t discount it.

All this reminded me a bit of what I wrote 12 years ago in my master thesis on Pseudorandom Number Generation:

Classic statistical tests examine if the test statistic does not deviate from its expected value too much. If we are only interested in the expected outcome of a similar simulation problem, such one-level statistical tests are all we need in order to be confident about the accuracy of the simulation.

On the other hand we might be interested in the distribution of the simulation’s outcome. For this goal hitting the expected value is not enough, the variance of the result is now important, too. Thus we will demand the same behaviour from the test statistic, too.

Let us illustrate this principle with an example. We want to test the well known strategy of doubling the ante in a game of roulette. It is supposed to guarantee winning the initial ante and works like this: If we do not win in the first round (and therefore win twice the ante) the ante is doubled for the next round. If this round is won, we get back four times the initial ante while we invested three times the initial ante resulting in a net win of one ante. In case of bad luck we double the ante again hoping for eight times the ante for an investment of seven. As we hope that we will finally win before our capital is drained a net win seems to be certain.

In order to simulate this we need random numbers to determine whether we will win the current bet. The probabilities are 18/37 for winning and 19/37 for losing each round, respectively. It seems to be natural to use the lengths of runs as a test statistic to test our source of PRN for its fitness to simulate a real roulette table. The probability that the maximal run length in 500 tries is greater than 15, is smaller than all usual values for α, so according to the corresponding statistical test we should reject all sequences where such runs do occur.

When we now run the simulation with these prescreened sequences we will never ever experience a loss as long as we have enough money for 15 steps of doubling the ante. Thus we should conclude that the strategy works. As we know, this is not true. So what went wrong with our simulation ?

The statistical test considered it equally important whether the sequence in question was “well-behaved” or not, whereas the simulation assigned completely different weights to those cases. Thus the area that the test considered to be insignificant (smaller than α) played a major role in the simulation (more than 1/2).

There are some other cases of simulations where we are not so much interested in the average case, but in the extreme ones. Consider for example all those safety measures in power plants or other machinery where a rare sequence of occurrences might lead to catastrophic results. When simulating these security systems one must not a priori exclude unusual sequences.

Please note that the distinction between level-1 and level-2 tests (tests which test the distribution of the results of a level-1 test) is arbitrary. The test statistic of a level-2 test is just another function of the underlying set of PRN, too.

System Administration

Debian Lenny: / on LVM2 on dm_crypt on raid

This is just Google-fodder:

If you use the lenny (beta2) installer to setup a system which uses / on LVM on dm_crypt on raid, then the first reboot fails with something like “cannot find LVM volumes”. After a few minutes you get dumped into a busybox shell.

Getting the system to boot

Run cryptsetup luksOpen /dev/md1 md1_crypt (or md0, depending on your setup). Exit busybox and the system should boot through. Then …

Fixing this for good

It seems to be that this Wiki page still applies. Thus you have to create a line in /etc/crypttab containing (in my case md1)

md1_crypt /dev/md1 none luks

and run update-initramfs -u to update the initrd.

That’s it.


A rant

I’m using the AvantGo client on my nokia phone to read newspapers on the subway from/to work (and while waiting for Clemens to go to sleep). Yesterday, it stopped working. Today I debugged it and discovered that the nameservers of A1 (Mobilkom Austria) simply cannot resolve The domain is fine, and IP connectivity towards them is also OK via the cell phone.

So I called up their hotline. So far, they can reproduce the problem. No solution, other than entering the IP address in the browser. If this isn’t just a bug, but a deliberate blocking, I’m going to raise hell. I’m in a bad mood.

[Update: they managed to fix this within a day. Good.]

And today’s Microsoft patchday seem to have broken switching from internal to external screen on my Thinkpad.


I’m downloading the latest Lenovo updates at speeds worthy of last century. Let’s see if that helps.

CERT System Administration

RT(IR) Command line ticket creation + Autoreply

This is one of the “simple” problems which always take longer to solve than expected.

The goal: Script RTIR to generate Investigations associated with an Incident.

In order to do that, I needed to solve the following who subproblems:

  1. Being able to use the “edit” feature in the command line “rt” client and capture the new ticket’s id
  2. Basically, I needed to capture the STDOUT of the “rt” command, and let also rt invoke vi. My solution:

    --- /usr/bin/rt 2007-10-31 11:52:20.000000000 +0100
    +++ /usr/local/bin/rt 2008-08-20 14:14:20.000000000 +0200
    @@ -1335,7 +1335,7 @@
    local $/ = undef;

    open(F, ">$file") || die "$file: $!\n"; print F $text; close(F);
    - system($editor, $file) && die "Couldn't run $editor.\n";
    + system("$editor $file < /dev/tty > /dev/tty") && die "Couldn't run $editor.\n";
    open(F, $file) || die "$file: $!\n"; $text = ; close(F);

  3. Add attachments during ticket creation
  4. The “Autoreply on Create” Scrip is supposed to mail out the newly created content to the Requestor, thus I needed to not only provide plaintext while the creation, but also all the attachments that need to be sent out to the Requestor.

    Adding text is simple, just use “Text: foo bar” to the Template. But attachments are simply not supported for “rt create” (and not that easy to add in the code). They are supported for “rt correspond”, so I can sidestep the “attachment on create” problem by simply splitting up the work into an empty “rt create” followed by a “rt correspond -a file”.

    There is just one gotcha: The Autoreply script will happily mail on ticket creation with an empty mail.

    Solution: Change the “Condition = On Create” for the Scrip to “Custom” with

    my $t = $self->TransactionObj;
    return 0 unless ($t->Type eq "Create");

    my $c = $t->ContentObj;
    return (defined($c)) ? 1 : 0;


DNS: The sky is falling

In April I speculated about the impending doom of the DNS.

Now we know what was in the works, and yes, it’s not a pretty picture.

My idea from april doesn’t work 1:1, as the attacker doesn’t attack a single target, sondern arbitrary other hostnames in the same domain.

Anyway, I spent the last days analyzing data from .at nameserver regarding patch discipline in Austria. You can read the depressing results here.

System Administration

New version of mod_epp released

I finally manged to crank out a new release of mod_epp. So, what’s new?

Last November Gavin Brown sent me a bug report regarding the handling of large answers. I provided a quick fix back then but did no official release.

While debugging that issue, I noticed that mod_epp didn’t generate the greeting on newer apaches. Tracing that, I found that the kernel didn’t return from accept on connection open, but on the first data reception. First I suspected the epoll interface, but some more intensive debugging found a more mundane reason: setsockopt is used to set:


Allows a listener to be awakened only when data arrives on the socket. Takes an integer value (seconds), this can bound the maximum number of attempts TCP will make to complete the connection.

Some more googling found the answer: you need to use the AcceptFilter directive to disable this optimization for the EPP socket.

This is not really documented, thus this bug report.

CERT System Administration

Watching logfiles with Cacti

Assume you have a number of patterns you might want to watch out for in various logfiles. These could be harmless, e.g. postfix/smtp\[[[:digit:]]+\]: .* status=sent which indicates that a mail was sent by postfix, or could be bad, e.g. indications of hardware troubles, firewall log entries or suspicious entries in proxy logs.

There are a number of tools which can help you do that, e.g. logwatch can produce nice reports for you.

In my case, I was less interested in individual log entries, but in a graphical representation of their frequency. For that, I need something similar to MRTG. Cacti is a decent graphical frontend to the core MRTG engine, rrdtool. It provides various data sources and graphing options all packed up into a neat web interface.

So, what needs to be done to get Cacti to graph match-rates of regexs in logfiles?


Please welcome: Elena

Just in case somebody was wondering why my blogging was non-existent recenty:


Born on May 22th, she certainly made my life even more interesting and, in close coordination with Clemens, the two children manage to consume my energy and spare time.

Up to date picture are at my gallery.


From the archives: Some thoughts on RUCUS

On Jan. 30th, 2008, I sent the following to the sipping list. It is a good summary of some of my thoughts on the RUCUS problem statement and on which basic premises need to be sorted out first.


New I-D on ENUM for loose-route SIP

I’ve submitted a new I-D defining an enumservices subtype for loose-route SIP according to J. Rosenberg’s UA loose route (which right now is one of multiple proposals to address one problem).

The basic idea is the following: SIP proxy should distinguish between “retargeting” and “routing”.

  • “retargeting” is done, whenever a proxy decides that this call should be directed towards a new destination. Consequently, the Request-URI changes.
  • “routing” happens, when a proxy forwards the SIP message to the next hop, but does not change the identification of the target.

In the context of ENUM, the destination of the call is identified by a E.164 phone number (whether that number is encoded in the user-part of a sip: URI or a in a tel: URI doesn’t really matter). That number is the key for the ENUM – lookup which returns (in most cases) a SIP-URI.

The current RFCs define this as a retargeting operation: the phone number is mapped to a SIP AoR, and from now on the call is towards that URI and the original phone number is no longer relevant.

If you look at what currently done in private/carrier-ENUM settings, then this is not how ENUM is used: In most cases there, ENUM returns the next-hop for this call towards the phone number. That next hop re-extracts the phone number from the Request-URI and applies his own number/prefix based routing to the call. In other words, this is “routing” operation.

My draft makes this explicit: if the service field in the NAPTR is “sip:lr”, then this records contains a next hop and does not rewrite the destination of the call. Here is the example from the I-D:

   To visualize the difference between how "sip" and "sip:lf" entries
   are interpreted, consider the following entries:

             @  IN NAPTR  ( 100 10 "u"
                            "!^.*$!!" .
             @  IN NAPTR  ( 100 10 "u"
                            "!^.*$!;lr!" .

   A SIP proxy dealing with a call to tel:+441154960496 can select
   either record.  The first leads to

           INVITE SIP/2.0

   being sent to the proxy responsible for  If the sip:lr
   record is used, then

           INVITE tel:+441154960496 SIP/2.0
           Route: <;lr>

   is sent to