To get fail2ban to work you first have to change the default date format in the ASSP log as it is not one of the formats supported by fail2ban. I changed LogDateFormat to ‘DD-MMM-YYYY hh:mm:ss’

then add the following to /etc/fail2ban/jail.conf

[assp-iptables]
enabled = true
filter  = assp
action  = iptables[name=ASSP, port=25, protocol=tcp]
#          sendmail-whois[name=ASSP, dest=email@domain.com]
logpath = /usr/share/assp/logs/maillog.txt
findtime = 40000
bantime = 21600
maxretry = 1

and create the following filter file at /etc/fail2ban/filter.d/assp.conf

# Fail2Ban configuration file for ASSP
#
# Author: Max Lyth
#

[Definition] 

# Option:  failregex
# Notes.:  regex to match the SMTP failure messages in the logfile. The
#          host must be matched by a group named "host". The tag "<HOST>" can
#          be used for standard IP/hostname matching and is only an alias for
#          (?:::f{4,6}:)?(?P<host>[\w\-.^_]+)
# Values:  TEXT
#
# Example: Jun-08-10 12:32:19 96739-11630 [Worker_2] [TLS-out] 81.192.188.147 <seoulodih98@iciberconveyor.com> [SMTP Error] 554 5.7.1 Extreme Bad IP Profile
failregex = \d{5}-\d{5} .*? <HOST> <.*?> .*?Extreme Bad IP Profile

# Option:  ignoreregex
# Notes.:  regex to ignore. If this regex matches, the line is ignored.
# Values:  TEXT
#
ignoreregex = 

This will ban senders on the firewall for 6 hours when they reach Extreme Bad IP. Because these bots are so dumb they will keep trying so I have another super filter that monitors the fail2ban log and bans offenders for 10 days once they have been banned 3 times in week. This superfilter also has the same effect for ssh attacks and other filters monitored by fail2ban.

Add the following to /etc/fail2ban/jail.conf

[fail2ban]
enabled	= true
filter	= fail2ban
action	= iptables-allports[name=fail2ban]
#	  sendmail-whois[name=fail2ban, dest=email@domain.com, sender=fail2ban@domain.com]
logpath = /var/log/fail2ban.log
# findtime: 1 week
findtime = 604800
# bantime: 10 days
bantime = 864000

Remove the comment on line 5 and set the email addresses if you want a whois mail notification of these persistent offenders.

then create the following filter file at /etc/fail2ban/filter.d/fail2ban.conf

# Fail2Ban configuration file
#
# Author: Tom Hendrikx
#
# $Revision$
#

[Definition]

# Option:  failregex
# Notes.:  regex to match the password failures messages in the logfile. The
#          host must be matched by a group named "host". The tag "<HOST>" can
#          be used for standard IP/hostname matching and is only an alias for
#          (?:::f{4,6}:)?(?P<host>\S+)
# Values:  TEXT
#

# Count all bans in the logfile
failregex = fail2ban.actions: WARNING \[(.*)\] Ban <HOST>

# Option:  ignoreregex
# Notes.:  regex to ignore. If this regex matches, the line is ignored.
# Values:  TEXT
#

# Ignore our own bans, to keep our counts exact.
# In your config, name your jail 'fail2ban', or change this line!
ignoreregex = fail2ban.actions: WARNING \[fail2ban\] Ban <HOST> 

http://www.yourbloghost.com/xmlrpc.php?rsd

then Firefox will probably flag an error. If you then view the page source then you probably have blank lines before the opening First thing to try is to deactivate all the plug-ins and revert back to standard theme and try again. If that fixes the XML error then re-enable the plugins one by one and then the theme and refresh the XML-RPC page between each to see where the problem goes away.
For me the theme and the plug-ins were fine so it must be something else, the mostly likely next cuplrit is custom php files. I did not have any of these but every blog has a wp-config.php and my problem lay here. I noticed two extra carriage returns after the closing ?>. After deleting this the XML RPC response parses correctly in Firefox and suddenly the iPhone App works again.

I used Carbon Copy Cloner to migrate the data to the external drive in two passes. The first pass was while the server was running and got 95% of the data to the external drive. When I was ready to transition I shut down all the services in Server Admin and then re-ran Carbon Copy Cloner in update mode.

Ten minutes later we were ready to go so I figured that it would be a simple task just to boot the MacMini from my new cloned external drive and we would be off. That mostly worked until I noticed a stream of errors in Console relating to OpenLDAP:

May 31 20:42:24 fs slapd[803]: @(#) $OpenLDAP: slapd 2.3.27 (Sep 29 2009 20:28:12) $
May 31 20:42:24 fs slapd[803]: overlay_config(): warning, overlay "dynid" already in list
May 31 20:42:24: --- last message repeated 4 times ---
May 31 20:42:24 fs slapd[803]: bdb_db_open: unclean shutdown detected; attempting recovery.
May 31 20:42:24 fs slapd[803]: bdb(dc= maxlyth,dc=com): Ignoring log file: /var/db/openldap/openldap-data/log.0000000015: magic number 88090400, not 40988
May 31 20:42:24 fs slapd[803]: bdb(dc=maxlyth,dc=com): Invalid log file: log.0000000015: Invalid argument
May 31 20:42:24 fs slapd[803]: bdb(dc= maxlyth,dc=com): PANIC: Invalid argument
May 31 20:42:24 fs slapd[803]: bdb(dc= maxlyth,dc=com): PANIC: DB_RUNRECOVERY: Fatal error, run database recovery
May 31 20:42:27 fs slapd[803]: bdb_db_open: Database cannot be recovered, err -30978. Restore from backup!
May 31 20:42:27 fs slapd[803]: bdb(dc= maxlyth,dc=com): DB_ENV->lock_id_free interface requires an environment configured for the locking subsystem
May 31 20:42:27 fs slapd[803]: bdb(dc= maxlyth,dc=com): txn_checkpoint interface requires an environment configured for the transaction subsystem
May 31 20:42:27 fs slapd[803]: bdb_db_close: txn_checkpoint failed: Invalid argument (22)
May 31 20:42:27 fs slapd[803]: backend_startup_one: bi_db_open failed! (-30978)
May 31 20:42:27 fs slapd[803]: bdb_db_close: alock_close failed
May 31 20:42:27 fs slapd[803]: slapd stopped.

This looked bad so I figured I’d roll-back and power back up the original server.

I did some research and it looked as though the ldap database was corrupted (or at least that what the messages were implying). I tried the usual ldap recovery tools after first backing up the ldap database:

sudo tar -zcf /var/db/openldap.tar /var/db/openldap
sudo db_recover -h /var/db/openldap/openldap-data/

Still the same errors so on the primary server I used Server Admin->Open Directory->Archive->Archive In: to create an archive of the working LDAP database which I transfered to the new server and then used the reciprocal Server Admin->Open Directory->Archive->Restore from:

It looked like the server was started but checking the logs and it was the same errors. So I tried Server Admin->Open Directory->Settings->General and hit the Change… button to switch the server mode to Standalone.

I then repeated switching it back to Open Directory Master and tried again Server Admin->Open Directory->Archive->Restore from: and this time it worked.

So I shut down the services agin on the live server, used Carbon Copy Cloner to sync the changes to an external drive, archived the LDAP database and then shutdown the original XServe. After booting the MacMini I demoted the OpenLDAP to standalone the promoted it back to OD Master. I then reimported the archived database, re-enabled all the services and I was done

The part number I was looking for was a Intel Dual Gigabit module for the server (Part # AXXGBIOMOD) and unlike all the rest of the components that made up the server this one was a real ordeal to track down.

Needless to say none of the major UK component vendors had this part in stock and it was going to need tobe special ordered with at least a 2 week lead time. Worse, none of the discount vendors carried it such as Dabs, Scan or Lambdatec so I ended paying top dollar. In the end I ordered it from Trust Systems and it cost me a princely £111.23.

Installation was simple but seeing as this was the first time the ESX server was going to go down for a while I took the opportunity to download the latest firmware from Intel which I copied to a USB flash. I shut down the server, opened the box and dropped in the daughter card; no screws required.

On starting up I following Intel’s firmware update instructions and this took about 20 minutes.

I also thought I might add another LUN to my Adaptec RAID and discovered that I had already reached the maximum. It seems that the Adaptec 5405 can only handle 4 LUNs, even with the updated firmware. I really wish they would document this somewhere so I could have planned for this.

No problem really as the extra disk is not pressing but it does mean that I will have to take the server down in the future to migrate all the VMs onto external storage so that I can re-partition.

On the bright side ESX4i had no problems finding and using the new Ethernet daughter card. Intel/VMware 1 – Adaptec nil (once again)

Adaptec’s web site lists the 5405 as being capable of multiple LUNs so my strategy was to initially deploy a small 8GB Raid5 LUN for ESX boot, a 96GB Raid0 stripe LUN as VM swap space and 250GB RAID5 for the initial VMs with the rest left available to use as a volume served by FreeNAS as a TimeMachine store for our office laptops.

On booting into the BIOS Adaptec console I started the management console and tried to create the first array. No problems there but when I tried to create the second array the console tells me there are no available to disks to create an array on. Hmm that’s not good because if a physical disk can only be part of one array then my partition plan goes out the window. I decide to format the whole array as a single 2.7TB Raid5 drive and install ESXi this seems to work fine until you realise there it shows only 0.7TB of available storage.

Loosing 2TB was not an option so I start trawling the web for solutions. I raise a ticket with Adaptec about lack of multiple LUNs but 4 days later there is no response. I search the knowledge-base, docs and forums but there is no mention of multiple LUN or multiple array support and the only place it comes up is marketing material as a feature of the card.

Finally I figure I should update the card firmware. There is no mention of multi LUN or array support in any of the revision notes but the strategy worked for the motherboard so I figured I’d give it a go and once a again the fix came from nowhere. Even better the new firmware now supports using an SSD as a hierarchical cache for the array and I have one spare SATA port on the card so this could well be a future upgrade option.

In the meantime I go back to my original disk allocation plan and install ESXi and it all seems to be working great. Happy days.

Putting the bits together was easy and we just needed to follow the instructions that came with SR1600. We were worried that we would be short of cables, thermal paste, riser cards, screws and accessories but the Intel chassis came with everything we needed. It may be ugly but it’s well thought out.

First thing to hit us when we switched it on was the noise. My co-workers looked aghast because it was intolerable, even through the wall of our computer room. I could not even hear myself think as I sought to figure out how to control it. I found updated firmware for the S5520UR motherboard on the Intel website and while it said nothing about the fan control in the read-me I figured it would do no harm.

We needed to install all 4 updates (even if you are not using netboot or the remote management) to get some relief and one of them (I think it was IPMI) took almost 2 hours. It was worth it because the server is now almost silent enough to directly share a workplace with.

Next up. Partitioning the disks.

This finally got to breaking point over the summer so we started planning an upgrade. The release of ESX4i as a free version was a deciding factor although we did evaluate both VirtualBox and XenServer. The biggest problem in leaving the Linux based VMWare Server to a bare metal hypervisor is that you don’t have the luxury of broad hardware support.

After trawling VMWare hardware compatibility list and having only a budget of £2000 we came up with the following:

1. Intel SR1600UR (comes with 2 socket with Tylersburg 5520 motherboard)
2. Intel E5520 Xeon Nehalem (no heatsink/thermal solution)
3. Adaptec SR5405 Unified SAS/SATA Raid controller + ABM-800 battery unit
4. 2 x Corsair xms3 6GB Ram kit (TR3X6G1333C7)
5. 3 x Seagate 1.5TB 7200.11 hard disk
6. Samsung slimline DVD drive

I could have bought an ESX pre-certified system from Dell for not much of a premium if I had the patience for a special offer to come around but I was not overly keen on buying their drives and RAM. What really made up my mind however was that the Intel box comes with zero CPUs but 2 heatsinks. Dell charges an outrageous premium for a CPU upgrade with heatsink whereas with the Intel I can buy an second Xeon from anyone at well under half the Dell price.

The final shopping list totalled around £1500 so was under budget and we will get a unit with 8 hyperthreaded cores (upgradable to 16), 12GB of RAM (upgradable to 24) and 2.7TB of storage. Hours of painstaking research said this would work so it’s ordered.

IPAcco is a free software package that helps a network admin to collect, visualize and analyze IP accounting data from the Cisco routers. Cisco routers themselves are capable of collecting IP accounting information.

Because IPAcco is based on TCL, MySQL, PHP and GD; this how-to covers configuration of all these elements and in particular, building and installing a GD capable version of PHP.

The inevitable death of GoLive after the Adobe-Macromedia meant that sooner or later the site was going to need reworking. However rather than pay Adobe for the GoLive-Dreamweaver crossgrade I figured I could get a much better site without shelling out any money if I switched to WordPress.

Problem is that I liked the layout of my old website and its variable width design and I could find no Themes I liked for WordPress although GlossyBlue from N.Design Studio came close. I decided to crack out my copy of the excellent CSSEdit and trusty php editor TextMate and set to work. I’m pretty pleased with the result (although I wasted hour trying to get it to work with IE and it’s still not that great) and even got the variable width to work as I wanted.

It’s going to take me a while to port my old HowTos to the new system so they are off-line for the time being (sorry) but let’s hope that this is the start of much more frequent updates.

I was also noticing my logs filling up with endless futile attempts from China to log in as root (the root account is disabled; duh!) to my unix box. While I was confident that my security wuld not be breached I wanted to be able to abruptly cut off these hackers so they would know they had been hacked in return.

I used the BSD built in host.deny feature as a simple way to cut-off hacking hosts and married it with a perl script from pettingers.org running as a daemon to manage and purge the hosts in the blacklist. I’ve written up instructions on how to implement the auto-blacklist in one of my HowTos.