New version of mod_epp released

I finally manged to crank out a new release of mod_epp. So, what’s new?

Last November Gavin Brown sent me a bug report regarding the handling of large answers. I provided a quick fix back then but did no official release.

While debugging that issue, I noticed that mod_epp didn’t generate the greeting on newer apaches. Tracing that, I found that the kernel didn’t return from accept on connection open, but on the first data reception. First I suspected the epoll interface, but some more intensive debugging found a more mundane reason: setsockopt is used to set:


Allows a listener to be awakened only when data arrives on the socket. Takes an integer value (seconds), this can bound the maximum number of attempts TCP will make to complete the connection.

Some more googling found the answer: you need to use the AcceptFilter directive to disable this optimization for the EPP socket.

This is not really documented, thus this bug report.

Watching logfiles with Cacti

Assume you have a number of patterns you might want to watch out for in various logfiles. These could be harmless, e.g. postfix/smtp\[[[:digit:]]+\]: .* status=sent which indicates that a mail was sent by postfix, or could be bad, e.g. indications of hardware troubles, firewall log entries or suspicious entries in proxy logs.

There are a number of tools which can help you do that, e.g. logwatch can produce nice reports for you.

In my case, I was less interested in individual log entries, but in a graphical representation of their frequency. For that, I need something similar to MRTG. Cacti is a decent graphical frontend to the core MRTG engine, rrdtool. It provides various data sources and graphing options all packed up into a neat web interface.

So, what needs to be done to get Cacti to graph match-rates of regexs in logfiles?

Continue reading Watching logfiles with Cacti

Please welcome: Elena

Just in case somebody was wondering why my blogging was non-existent recenty:


Born on May 22th, she certainly made my life even more interesting and, in close coordination with Clemens, the two children manage to consume my energy and spare time.

Up to date picture are at my gallery.

New I-D on ENUM for loose-route SIP

I’ve submitted a new I-D defining an enumservices subtype for loose-route SIP according to J. Rosenberg’s UA loose route (which right now is one of multiple proposals to address one problem).

The basic idea is the following: SIP proxy should distinguish between “retargeting” and “routing”.

  • “retargeting” is done, whenever a proxy decides that this call should be directed towards a new destination. Consequently, the Request-URI changes.
  • “routing” happens, when a proxy forwards the SIP message to the next hop, but does not change the identification of the target.

In the context of ENUM, the destination of the call is identified by a E.164 phone number (whether that number is encoded in the user-part of a sip: URI or a in a tel: URI doesn’t really matter). That number is the key for the ENUM – lookup which returns (in most cases) a SIP-URI.

The current RFCs define this as a retargeting operation: the phone number is mapped to a SIP AoR, and from now on the call is towards that URI and the original phone number is no longer relevant.

If you look at what currently done in private/carrier-ENUM settings, then this is not how ENUM is used: In most cases there, ENUM returns the next-hop for this call towards the phone number. That next hop re-extracts the phone number from the Request-URI and applies his own number/prefix based routing to the call. In other words, this is “routing” operation.

My draft makes this explicit: if the service field in the NAPTR is “sip:lr”, then this records contains a next hop and does not rewrite the destination of the call. Here is the example from the I-D:

   To visualize the difference between how "sip" and "sip:lf" entries
   are interpreted, consider the following entries:

             @  IN NAPTR  ( 100 10 "u"
                            "!^.*$!!" .
             @  IN NAPTR  ( 100 10 "u"
                            "!^.*$!;lr!" .

   A SIP proxy dealing with a call to tel:+441154960496 can select
   either record.  The first leads to

           INVITE SIP/2.0

   being sent to the proxy responsible for  If the sip:lr
   record is used, then

           INVITE tel:+441154960496 SIP/2.0
           Route: <;lr>

   is sent to

May Weather

It’s time for the annual proof that god favors left over right, but this time it’s not that easy for him.

Some background: traditionally, the social democrats (which used to be called socialists) hold a rally on May 1st on the Vienna Ringstrasse. Over the last years, that has been more and more a relic of old times: various delegations of workers, unions, retirees clubs and whatnot else affiliated with the SPÖ marching and showing flags. Of course there are speeches, the (traditionally red) major and (if they are lucky) their own federal chancellor rally their people. After the official part, there is a big party in the Prater, with music for all tastes (incl. western&country (complete with line/square-dancing)). That’s usually very nice: the chestnut trees are in full bloom and …

… the sun shines. Always.

The major right-wing party also organizes annual festivities some other days in May: they hold in the the center of the city, complete with music and other attraction. But ..

… the weather sucks. Always.

So this year they are going for something new: They scheduled their Stadtfest for May 2nd – 4th. I see this as a challenge for Petrus: can he turn around the weather in just one night?

Oh, and there is another data-point regarding god’s political stance: The communist’s “Volksstimmenfest” also gets doused in rain every year.

[UPDATE, May 1st]

The ÖVP’s jinx worked, we didn’t have the usual bright day, there even were a few drops of rain. Let’s see what tomorrow will be like.

[UPDATE, May 3rd]

Well, well, well. It looks like the ÖVP got the better wetter this year. Will pigs fly next year?

Fun with conditional probabilities

Yes, mathematics is sometimes hard to understand. Probability theory is a prime candidate where “common sense” might mislead you. And once you venture into conditional probabilities (“given A is true, what’s the probability of B being true as well?”) things get tricky.

A good example is the following statement: “The orbit of our earth so incredibly fine tuned to the requirements of intelligent life, that this is a strong indication that a higher being was involved.”

This may sound reasonable at first glance, but it’s completely wrong. Let me rephrase the statement to make this clear:

“What is the probability that the environment here on earth is suitable for the development of life, given the fact that this question is asked by a person whose species evolved here on this planet?”

The answer is 1 (100% probable).

Or in other words, if the environment here on earth were hostile to us, we wouldn’t be here to ask this question. And indeed, there are billions of planets, where nobody is asking this question.

So no, these considerations do not provide an argument regarding the existence of god.

(Btw, the same reasoning applies to arguments concerning the fundamental constants of this universe. In that case, you just need to assume an infinite number of universes with varying parameters, as a number of physicists postulate.)

DNS: What to do if the sky is falling?

Having been treated to another iteration of “we need to deploy DNSSEC, otherwise DNS will fall apart due to rampant forgeries” talk recently, I started to think what other options resolvers have to protect themselves. See also bert’s draft.

First of all, is it possible to implement a successful forgery attack without the client noticing that anything is wrong? IMHO not. Most scenarios assume that the attacker sends a burst of forged replies to the recursor in the hope of hitting the right query-ID before the real answer arrives. So, even with a successful attack a lot of answers with mis-matched ID will arrive (and be rejected) before the forgery actually succeeds.

This spike in ID mismatches towards a query should tell the recursor that something fishy is going on. TODO: Instrument bind/powerdns/… to actually log such ID mismatches.

If this works, then the 99.9999% percent of normal DNS lookups which are not targetted by a forgery attack can continue without modification. The only question now is: what can a resolver do if he has a clear suspicion that he is the target of a DNS forgery attack?

There is a trivial solution: Redo the query a few times, preferably utilizing all authoritative servers.

Let’s do some simple math: assume that the forger has a 1% chance of a successful spoofing attack. What are his chances if the recursor only accepts the answer if he got the same data from n queries?

n = 2: 0.01 % chance
n = 3: 0.0001 % chance

That doesn’t sound so bad, does it? The downsides sound rather minimal: no impact on queries which are not target of an attack, a few more requests during an attack and a slight delay (sequentially querying! You can’t do that in parallel.) before answering back to the client.

A few optimizations come to mind: Bert’s draft mentions a few other options, e.g. source port (or even IP address) variability. These come with some added cost, so perhaps they could be utilized just for these cases.

What did I miss?

[Update: 2008/04/22]

After a chat with Klaus and Alex, two new thoughts on this:

  • There might be legitimate cases where multiple queries might give varying results. E.g. global load-balancing like Akamai or This might break the “ask multiple times and check for consistency” idea.
  • Another option is the following: if a forgery attack is detected, simply switch to TCP for this query / target nameserver.

SPIT: Where do we stand?

SPAM over Internet Telephony (SPIT) is a strange beast: Everybody knows it is a threat, but in real life SIP installations, it has hardly been observed. In other networks it is not uncommon, Skype needs to police its users to get a grip on abuse, and all Instant Messaging networks have to deal with IM spam, and once they offer voice capability, SPIT as well.

So, SPIT over SIP hasn’t been a problem so far. After all, there are millions of SIP devices out there replacing PSTN phone service. Why?

  • In many end-user installations, SIP is just a replacement for the old analog last-mile technology. There may be SIP accounts provisioned into these devices (ATAs, hardphones), but the user does not see them. Nor do they know their own SIP AoR. None of the people calling them knows their SIP AoR. It wouldn’t help them anyway, as these networks are usually not reachable via SIP from the Internet
  • The same is true for IP enabled PBXs: They talk SIP on the inside, and maybe to other
    PBXs in other offices of the same company (usually via some QoS-enabled VPN). In rare cases, people build SIP trunks to upstream carriers or even establish manual cross-links to the PBXs of other companies.
  • Fully-featured SIP installations which implement inter-domain SIP routing as standardized by the IETF in RFC 3263 are the exceptions. They are the enthusiast’s Asterisk installations, some enterprising VoIP operators (perhaps even using User-ENUM) and that’s it.

Summary: the vast majority of SIP devices out there is not reachable over the public Internet.

Corrolary: There is no SPIT problem, as the target audience is too small to make SPIT profitable.

This is some sort of vicious circle: Operators don’t want to open up their SIP ingress elements, because they fear SPIT. On the other hand, it is hard to build defenses against SPIT, whose characteristics and weak points you don’t really know. And it is completely impossible to prove that certain clever anti-SPIT measures work if you can’t test them in practice.

There has been no shortage on clever ideas concerning SPIT defenses. Academic papers galore, internet drafts, and even products claiming to solve the problem.

We are still in the same vicious circle.

Will RUCUS help?

HP DLxxx, iLo2, XEN and remote consoles

As a general principle, servers with PC-style BIOS suck. If you ever worked with a real server with native support for a serial console, then anything else feels like a crude crutch.

HP servers with iLo2 are no exceptions.

At least unless you’re unwilling to shell out additional money for the feature packs.

You get a crude vt100 emulation during the boot sequence, but once the operating system starts, this falls back to a real serial device. The trick is thus to configure grub, the kernel, and the gettys in a way to get consistent console access all through the boot process.

Now add XEN and and root in an encrypted LVM volume to the mix and things start to become nasty. So here is my solution, for all those weary souls googling for a recipe for Debian (etch):

title Xen 3.0.3-1-i386-pae / Debian GNU/Linux, kernel 2.6.18-6-xen-686
root (hd0,0)
kernel /xen-3.0.3-1-i386-pae.gz dom0_mem=128M com1=115200,8n1,0x2f8,3
module /vmlinuz-2.6.18-6-xen-686 root=/dev/mapper/vg_main-lv_root ro console=ttyS0 xencons=ttyS0
module /initrd.img-2.6.18-6-xen-686

And don’t forget to note these parameters in the comments in /bot/grub/menu.lst unless you want to manually add them after each kernel upgrade. i.e.:

## Xen hypervisor options to use with the default Xen boot option
# xenhopt=dom0_mem=128M com1=115200,8n1,0x2f8,3

## Xen Linux kernel options to use with the default Xen boot option
# xenkopt=console=ttyS0 xencons=ttyS0

If you’ve forgotten that, you can save yourself the trip to the server room by manually editing the boot config using grub’s edit functionality.

Enabling login via ttyS0 is then just a matter of
T0:23:respawn:/sbin/getty -L ttyS0 115200 vt100
in /etc/inittab, and ttyS0 in /etc/securetty.

Once you enable ssh access in iLo2, you can reach the console of your server from any *ix shell via a simple ssh command.