RHS Oct 27

by **Admin** Tue Oct 27, 2015 2:26 pm

Free forums taken down for an unscheduled maintenance; shouldn't be down too long. As for our past problems:

Hi everyone,

I am really sorry for the late reaction. I was at dinner and just signed on to find a good number of reports. Trust me if I tell you that I'm just as annoyed by the outages as all of you are.

Now, what's happening and why do we keep going down... you deserve an explanation.

As you know, we had a total outage of all our networks a couple weeks ago. This, in fact, caused a lot of damage. The outage was caused by a PDU overload during a routine maintenance. It was a PDU the board servers were connected to, so the outage caused data corruption. The following week was a bit of misery, because scripts were running on all servers, causing constant lags and white pages. These were in fact not caused by any issues, they were caused by the repair scripts running, which is taxing on the servers.

We had a second PDU overload earlier this week, this was a different one and thankfully, did not affect the actual board servers. In spite of an outage that lasted several hours, no repair scripts had to be run but the outage was frustrating to many, nonetheless.

On top of it all, we also experienced repeated issues with one of the cache servers, including the issue that caused the downtime this evening.

What to expect for the future:

Work is being done in the background to avoid situations such as this one. This work includes, but is not limited to, moving boards between servers to balance the loads. Last week, many boards were moved off C14. In the coming week, boards will be moved off C4. Once the servers are more balanced and the cache servers have had some maintenance, we will be in better shape.

I understand that there is worry, and I know that the boards mean a lot to you all. I also know that we've gone through quite a stretch of bad server performance here, and that some things still need to be done before we're completely out of the woods. Please rest assured that out first concern is to make the network stable again so that we can put these outages behind us.

I am truly sorry for these outages. Please hang in there.