Re: [tsc-devel] Server outage

Sydney Dykstra | Fri, 19 Jun 2015 02:42:49 UTC

Ok, So here is how it has been going thus far.
     I ended up restoring the whole wiki from a backup we made, and it 
appears it has been spammed as far back as 80 days ago. But we had a 
increase in pages being made as of the last few days, which caused 
everything to bog out, and caused the server to crash.
     The issue ended up being a few of the color related TextChas, and 
also the metric one not being random enough. I removed them and added a 
bunch of random ones (In my opinion). If you get stuck with those 
textchas a search engine is your friend, or just find me on IRC. My nick 
at the moment is head8debian.
Currently the wiki as been up for 2 day with no successful break in 
attempts.
As Quintus noted, he had a drastic drop in resource usage when we shut 
the wiki down, so apparently it is a pretty big source of attention for 
spambots.

-Sydney


On 06/14/2015 05:09 AM, Quintus wrote:
> Hi everyone,
>
> today night between midnight and 08:00 UTC the alexandria server has
> been subject of both an SSH breakin attack and a massive spam attack on
> the wiki. It completely crashed shortly before 08:00 UTC. We solely use
> SSH publickey authentication, thus it was (near to) impossible to
> bruteforce the authentication mechanism. No unauthorised access thus
> happened.
>
> As of 09:00 UTC I got a chance to boot up the server again via the
> hoster’s control panel and reinstituted its services.
>
> For the wiki, I was unable to even reach the wiki page after I started
> it. It simply didn’t load anymore. Inspecting the directory, I see 6299
> pages listed for it, which I doubt are legitimate pages. I disabled the
> wiki for now. This is the peak of problems that arised from the moinmoin
> wiki software, which might be related to moinmoin itself or to the old
> version Debian oldstable ships; before ditching it entirely, I’d at
> least consider that possibility. Debian is known to not keep
> non-mainstream software up-to-date from a security point of view.
>
> The forum was entirely unaffected. It is up again, and it was only down
> due to the high load on the server (load average > 30). There’s not a single
> spam post on the forum as far as I can see.
>
> For those of you with SSH access, I have switched sshd to port 753 now,
> which should silence a number of the bruteforce login attempts.
>
> sydney, I’d appreciate it if you could PM me on IRC.
>
> Valete,
> Quintus
>