As part of our community network management service offering (a mouthful, I know - we manage a number of community WiFi networks), we sometimes have to employ measures to insure that some users don’t overrun the network with malware infections, peer to peer file sharing applications, etc. Instead of limiting what people are allowed to do on the network, like some ISPs have been getting heat for lately, I tend to prefer limiting how much they can do. I personally don’t care, and I don’t think that other ISPs should care, that a person is using Bittorrent, as such applications do have completely valid uses. I do care, however, if the use of that application, or an infection of some kind, is causing other users of that same network to suffer. So, this week I set out to employ the connlimit iptables match to help limit to some degree how many connections a runaway application can create through the firewall at one of our networks. Unfortunately, the process of getting to the point where I could include connlimit rules in my iptables configuration was not an easy one, so I thought I’d do my own writeup here to help those who may be looking for how to do something similar.

In most of my Linux systems, I use the Fedora distribution. Its free, easy to install, has much of what I need in the stock systems, and usually has a good combination of stability and the newest versions of packages. Unfortunately, as I discovered very quickly after my research in to how to best handle the connection limiting problem in iptables, the stock Fedora systems (Fedora Core 5 in this case, but its still true in Fedora 7 at least) do come with the iptables module for connlimit matching, but they do not come with the matching kernel module. So, I was going to have to roll my own kernel that included the connlimit module. Luckly, kernel building is not something I have to make a habit of since the kernels that come in all recent Linux distributions usually meet my needs. Sometimes, like in this instance, though, I do need to add modules that aren’t included in the stock kernels (and lately, those cases all seem to revolve around various network requirements). The issue here was that patching the kernel was not a simple matter.

The Netfilter folks have put together a system by which people can patch their Linux kernels with any of the netfilter modules they need. It’s called patch-o-matic. Unfortunately, much of the documentation on the Netfilter website relating to patch-o-matic is old and out of date. In fact, they’ve switched to the newer patch-o-matic-ng, but the link for the new patch-o-matic-ng system referenced the Netfilter Extensions HOWTO. This HOWTO, however, says nothing about the new patch-o-matic-ng, instead talking about the old system that no longer works or is even downloadable from what I can tell. It took quite a bit of digging through newsgroups, etc. to discover that the problem is that it appears that the Netfilter folks never started building and publishing new documentation from their source repository after switching to Subversion. The currently published documentation on the Netfilter website still references CVS revision numbers, but the current documentation is now kept in Subversions in it original SGML form and does not appear to be getting built into the HTML and other versions for public consumption. So, if you’re looking for the documentation for patch-o-matic-ng, you’ll have to peruse the original SGML formatted documentation, and the new HOWTO is available here.

So, finally having the correct documentation in hand, I proceeded. Given that the machine in question was still running Fedora Core 5 and was located on the west coast, making an update to a newer revision not completely feasible, it took some digging to find a mirror for the older distro’s source RPMs. Upon finding one, I grabbed the source RPMs for the latest kernel and iptables, as you’ll need both to work with patch-o-matic-ng. Here’s where the documentation’s usefulness comes to something of an end. If you, like me, are going to build a new RPM to apply to the system, you’ll have to apply the netfilter patches to a kernel, then use that to create a patch suitable for the RPM building process. To do this, you’ll need to run through several steps:

1. Install the kernel and iptables source RPMs, then run an rpmbuild -bp <specfile> on each of the kernel and iptables rpm spec files. This runs through the RPM build “prep” stage which unpacks the stock source and applies the patches that come with the source RPM. This is important so that you come away with clean patches for building your RPM. If you were to patch against a stock kernel, your patches may be difficult to apply when combined with the source from the source RPM that has already been somewhat patched.
2. Now that you have a fully prepped source tree, which will be located in /usr/src/redhat/BUILD/kernel-<version>/linux-<version>, make a backup copy in the /usr/src/redhat/BUILD/kernel-<version> directory, so that you have an original tree with which to make your own patches from.
3. Armed with the fully prepped source tree and a pristine backup of it, run through the patch-o-matic-ng instructions to apply the patches you need to the kernel tree. Be sure to feed it the paths to your prepped kernel and iptables sources in the BUILD directory. Also, if you’re looking for the connlimit patch, as I was, you’ll have to perform a ./runme –download followed by a ./runme external, a procedure which isn’t really documented in the HOWTO.
4. Now that you’ve patched your kernel tree with the Netfilter patches, create your patch files by running diffs between your backup tree and your newly patched tree. For me, this was a two step process, since the netfilter patching created some new files. Here’s what I ran (my backup/pristine source tree was in /usr/src/redhat/BUILD/kernel-2.6.20/linux-2.6.20.i386.old, and I had cd’d into the patched source directory of /usr/src/redhat/BUILD/kernel-2.6.20/linux-2.6.20.i386):
   1. for i in `find . -type f`; do diff -u ../linux-2.6.20.i386.old/$i $i; done > /tmp/connlimit.patch
   2. diff -u /dev/null net/ipv4/netfilter/ipt_connlimit.c >> /tmp/connlimit.patch
   3. diff -u /dev/null include/linux/netfilter_ipv4/ipt_connlimit.h >> /tmp/connlimit.patch
5. You now have a patchfile that can be used with your RPM building. You may have to doctor your patch just a bit to insure the directories referenced will work with your Patch statements you add to the spec file. Do any doctoring you need, then move the patch into the /usr/src/redhat/SOURCES directory.
6. Next, you’ll need to edit the .config files that are used to configure each kernel build (because there are a number of kernels that are built from that one source RPM). They’re located in the SOURCES directory, and each has a name similar to kernel-2.6.20-i586.config. In each one, add the line CONFIG_IP_NF_MATCH_CONNLIMIT=m. This will cause the new connlimit module to be compiled as a module.
7. Now you’re done. You can try to build your new kernel. You’ll need to specify your architecture on the rpmbuild line, so, in my case, I ran rpmbuild -bb –target i686 /usr/src/redhat/SPECS/kernel-2.6.spec. The i686 target for this kernel builds 19 separate RPMs, so the process takes quite a while (like 6 or so hours in my case). When its done, though, you should have the kernel you are looking for with your new iptables match modules in place.

The effort required was more than it should have been, in my opinion, but it was well worth it. I’m now able to limit the number of connections each user can create. The iptables man pages included some information to get you started, but there are some things that aren’t completely intuitive. First, each iptables rule does its own connection counting. You might figure this out if you look at other people’s rulesets, but no one seems to come out and say that. What this means is that you can set up rules for, say, different ports and the connlimit module only does matching in that case for that rule. The connection matching for that rule does not affect matching for any other connlimit rules. This was nice for me, because I was able to severely limit the number of outbound SMTP connections (to stop spambots from being able to create hundreds of simultaneous connections), but kind of lump everything else into a “you get so many connections before you can’t create anymore”. One other gotcha to keep in mind is that the connlimit kernel module keeps its own cache of connections (part of the reason you can apply different connection limits to each rule). As such, though, if a machine currently has a high number of connections prior to applying the connlimit rules, those pre-existing connections will not get counted against the limit for that client.

In the end, as I mentioned before, it took more effort to get this working than it should have, but it was well worth it, and tt has been satisfying to see the rules, especially the SMTP rules, have a positive affect on the availability of network resources - and hopefully cut down on the number of spam complaints we get.