thunkgeek.com

We have a bit of a tradition, here at Fibernet, that when the anniversary of your first day comes around you bring treats for the rest of the company. I decided, since this year makes a dozen for me, I’d up the ante a bit and start an Annual Potluck Belgian Waffle Festival. My mother-in-law bought me (at Erica’s suggestion) a pro-style belgian waffle iron a couple years ago for Christmas, so I brought that, and purchased a second one to help things move along. The rest of the company brought toppings to go on the waffles. I’m pretty sure everybody got full and had a good time:

My young co-worker just called my other young co-worker to his desk with the words “Come check out this screen saver!” When the second co-worker got there he said: “Awesome!! Flying toasters!”

Welcome to the 1980s, only this time it’s in OpenGL running on slackware. Irony.

At work having just finished helping three different clients on one night. Not too shabby.

Saw Ep III tonight. It was better than 1 and 2, but I still don’t think it had the magic. Maybe I am just too old now.

So I’m here working on spam filtering again. Thought I was done with this. Never assume, right? We decided a couple weeks ago to try to increase our filtering capacity by mirroring some of the block lists locally. Since then I’ve been fighting with the machines we setup to do this trying to find the right methodology and now the right hardware. I think maybe I finally have it all set just right. Ugh.

So we thought all was good in the land of spam and virus filtering. Then we signed up a customer who thought it would be a good deal to offload their mail to us because they were getting so much spam it was making their server crash. ‘Course they didn’t tell us that they were getting that much garbage. They probably had now actual idea just how much spam they were getting, actually. It took our load balanced system of five servers and basically thrashed it. Within a few hours load had more than tripled as far as the number of messages coming into the servers. Then they can’t handle the amount of mail coming in it ends up sticking in a queue directory till it can be sent through. The total queue size on the system shot up to about 300,000 messages within a day or so. Of course, once you get that many messages it becomes a burden on the server just to list the contents of the directory so it can work on sending them out. Add that to the already increased load of all the new incoming mail and you have five servers completely bogged down with trash. And, you know, that is the worst aspect of the whole problem… if this were legitimate mail coming in a an increased rate that would be ok. But to bog down everybody’s mail because some loser out there thinks the way to market his product that nobody wants is to send a million pieces of email that, again, nobody wants is just ludicrous. I hate to think how many hours of each day over the past month and a half or so I have spent tinkering with this. What a waste!

And just in case any spammers are reading this… get a life you morons.

I’m putting the finishing touches on our new traffic measurement system, including testing all the previous customers whose data and config I’ve had to convert over to it. It has been an exhausting process. Ugh!

You gotta love it when the problem you’ve been trying to solve is easily explained away by reading the documentation for a version of the software that was obsoleted approximately six years ago. Yeah. So, the mail filtering system seems to be working much better now that we fixed a little issue with our load balancer slowing down every single connection to a crawl. Arrrrghhh, I hate it when documentation is lacking.

For the pure sake of documentation, and since I haven’t posted in a while, I’ll just mention that I am still fighting the stinking spam filtering system. I can’t believe how much of my life is spent working on this thing… tweaking here, fixing there, rebooting as the case may call for it. I wouldn’t be surprised if the total percentage got close to 50% of every 24 hour day was spent working on or thinking about this thing. It’s getting nasty.

We added the third server this morning and things are looking good. It is not taking a lot of the traffic right now cause it is low on memory. As soon as we add some more I will load it up a bit more and see how it does. Response times to the system as a whole are already improved, going from around 20-30s for initial reponse to less than 10 in most cases, so we’re nearing the point where we could use this for outgoing mail as well without having people complain about their outgoing mail server not respond quick enough. I have to say though, it still blows me away that we’ve had to use so much processing power to handle all this. Live and learn right?

The servers are doing pretty well…. It can still take some time to connect to them and send an email, which is a problem because we are considering making this system handle outgoing mail as well. We’re in the process right now of building a third filtering machine to add to the mix. This will be the beefiest of the three in terms of processor and memory, so hopefully it will make a large enough difference that we can go ahead with the plan of filtering outgoing as well as incoming mail. I guess it all goes to show that no matter how much you test and plan and fret, you never know exactly what kind of results you are going to have in the real world. I really had no clue that this system would need to handle as many as 1000 concurrent connections. I should probably have tried a bit harder to come up with a good idea of how many connections the old server was handling, but I’m not sure how to really do that, especially over time. Maybe a netstat piped through a counting script every five minutes graphed over several days would have given me a decent idea. Anyway, that’s all in the past now and things are looking better and better. I just hope we don’t end up having to spend a lot of money to make this free service work!