Database Failure, 15 January 2016

I’m sorry.

TodaysMeet is an important part of many classrooms. For some students, it provides a critical way to participate and have a voice in their own education. I take that very seriously, and so it is my goal to always provide the best possible service. First and foremost, that means TodaysMeet should be available to use.

Today, I failed at that goal. TodaysMeet was unavailable for nearly six hours in the middle of the North American day (between 10:09am EST and 3:48pm EST). It’s the worst outage TodaysMeet has ever had, and it comes after a few weeks of (unrelated) issues.

Worse, a little over one minute’s worth of messages were lost. TodaysMeet rooms may be ephemeral, but that doesn’t mean it’s OK to treat your data with anything less than the utmost respect. Those messages are things students and teachers tried to say, and I lost them.

This was a rare and catastrophic failure, the sort that’s very difficult to prevent. But that doesn’t mean it’s impossible to plan for. This incident has demonstrated that I didn’t plan enough.

I am taking steps immediately to better plan for these sorts of problems, so when they do occur—which is, thankfully, not often—TodaysMeet will recover faster, and as few messages as possible are lost. There will probably be scheduled downtime in the near future to accomplish these changes. I will give as much notice as I can if that turns out to be necessary.

There will be a more technical write-up of the incident on the status page soon. In the meantime, I apologize.