Dara and I have been having to conduct critical unscheduled maintenance of our mail server today. This was precipitated by getting a voice mail from Comcast this morning asking us to call them, and noting a risk of possible cessation of our service. I went “WUT?” and called them, and was informed they’d noticed one of our machines carrying out DDoSes.
To which Dara and I both went “WUT?!” And after we thanked Comcast for the heads up, Dara moved forward with completely flattening the box so we could do a brand new clean install with the latest version of Debian Linux. (We were going to do some dist upgrades anyway over this forthcoming winter holiday, but this just escalated the priority.)
And we went down to Frys this afternoon to get a new hard drive and power supply too, since the box was down anyway. We’re both rather impressed that the hard drives previously in newmoon had dated back to 2004 (whoa) and 1998 (WHOA!). And we’ve brought home a new drive that’s acres more space than we had on newmoon before, not to mention much quieter. So the server room should have a significant reduction in noise now.
BUT. This also means we’ve had the server out all day, and it should be down probably for the rest of the evening. Hopefully it’ll be back up by tomorrow, but operating system upgrades can be dicey. So don’t be terribly surprised if we’re not up tomorrow morning.
ALSO: this impacts only our mail server. So the mailing lists we host (notably the LexFA list and the Filk list) are down, and users who have mail accounts on our system won’t be able to use those accounts till newmoon is back up. Web services are unaffected.
Please let Dara and me know if you have any questions, and watch her and my accounts for further updates!
Dara had to perform some maintenance on our web server last night, and it turned out to be an unexpectedly bumpy ride–thanks to unexpected changes in the process of updating to Apache 2.4. All this work kept her up until 4:30. AUGH.
Likewise, I have also updated all blogs we host to WordPress 4.2.1.
We THINK we’ve gotten everything working, but just in case, if anyone reading this has resources we host, doublecheck your stuff and let us know if anything looks screwy. And if anybody sees anything screwy with angelahighland.com in particular, let me know and I’ll check it out.
ATTENTION anybody who has a mail account on Murkworks.net:
Right now we’re having issues with any of our accounts sending mail out to Gmail addresses. Gmail is incorrectly dinging us as a spam sender, and either shunting mail from us into Spam folders or else outright bouncing it. Needless to say, this is massively annoying.
I’ve just been informed on Twitter that Google is already investigating similar reports, so this may or may not be clearing itself up soon. Dara’s been trying to investigate as well.
In the meantime, if you have alternate email addresses to use, you should do so. We apologize for the inconvenience. 🙁
We actually regained power overnight, but we didn’t get the servers back up and running until we got up this morning. However, we ARE now back up and running!
And Dara even took the opportunity to upgrade the network card in our webserver, so now hopefully our websites will be much more zippy for all you lovely visitors who come by to say hi. Let us know if you see any problems, won’t you?
Hope everybody made it through the windy action okay! And all hail the power company crews who’ve been working for the last several hours to get power back on for us all!
Heads up, you guys, we have a Wind Event coming in.
The PI is reporting about the incoming wind fun tomorrow night, so here’s the obligatory MURKWORKS.NET MAY GO OFFLINE TOMORROW NIGHT advisory. Dara and I will post if our servers have to go down.
ALSO: the PI’s saying one of our local meteorologists is advising people to go home early if possible. And the current High Wind Watch is talking about the winds hitting us possibly during tomorrow night’s commute. Batten down the hatches, this could get messy.
For those of you who didn’t see me post this to the social networks or to my LJ and Dreamwidth accounts, we had a windstorm last night and that took out power at my house for several hours.
Since my site is hosted on my home servers, that meant angelahighland.com was inaccessible for the duration. But now we’ve got power back and the servers came back up, so the site is live again.
Apologies to any visitors who tried to come by only to find the site down!
Dara needs to take down our web server this morning for security maintenance, so my website, hers, and all others we host will be temporarily inaccessible for a while this morning. Please stand by and we’ll post another update when web services are restored!
ETA: Web maintenance is now complete. Please let Dara or me know if any of our web resources look b0rked!
ETA #2: Second round of maintenance will be happening TONIGHT. Dara has to do our other server, door. So we’ll have another round of inaccessibility this evening as she works on that. Watch this space for further details.
ETA #3: Murkworks.net updates are confirmed finished for the evening, so far as we know. Dara finished before I got home, but we should be good. Please let Dara or me know if you see any weirdness visiting any of the sites we host.