bugs.gentoo.org status report, 2009/03/14 06h00 UTC
Yes guys, I know that Bugzilla is down.
Last night, while I was sleeping, we got a slew of IPs hitting the
dependency graph generation. This wouldn't have been a problem normally,
but they seemed to hit graphs that took an inordinate amount of memory
to generate with GraphViz (collectively 8GiB of RAM and 32GiB of swap).
This morning, I got into the box, saw some OOMs of the GraphViz
processes, killed off the remaining ones, banned the IPs, and then had
to rush off to a work meeting.
I came back this evening, to find the box not responding again, and my
last SSH shell was painfully slow then just hung - not died, but hung,
the TCP connection is still alive, but the shell isn't responding
(shortly after I had seen a loadavg exceeding 1k).
Whomever attacked it came back I think. And I can't get in to block them
right now. I've contacted the sponsor so that they can hard reboot the
box for me, but I don't expect any action from them for the next 5-6
hours at least.
Meantime, I'm enacting a plan B, to at least get us some slow Bugzilla
functionality, via the second bugzilla box that normally runs the
background computations (duplicates etc). I do however fully expect
whomever the attacker is to come right back at it - so I'm turning off
the dependency graphs.
Robin Hugh Johnson
Gentoo Linux Developer & Infra Guy
E-Mail : firstname.lastname@example.org
GnuPG FP : 11AC BA4F 4778 E3F6 E4ED F38E B27B 944E 3488 4E85