Thursday July 09 2020 • posted by russ

Report on the July 2020 extended outage

The IRCCloud service experienced an extended period of downtime between around 22:40 on 07/07/2020 and 19:10 on 08/07/2020 (UTC) due to a fault with our internet service provider. This was the second time such an outage has occurred this year, and we are as frustrated as you are with this unacceptable situation.

This outage affected seven of the eight servers we use to handle our outgoing IRC connections. As we explained in our report on the previous outage, these outbound connection servers have a fairly unique networking configuration which means they are hosted by a specialist ISP, which we cannot quickly migrate away from.

We have built sufficient redundancy to ensure that we can survive the loss of one or two of these eight servers, but we don’t yet have the capability to survive the loss of all of them.

Although we haven’t yet received a full explanation from our ISP, it appears that this problem was caused by some kind of networking failure which required a technician to visit the datacenter to resolve. For reasons which are unclear, it took more than 12 hours for this to be arranged.

As we mentioned in our previous report, we have been working on making changes to our system so we can move to a new ISP. These changes are almost complete and we will now accelerate our migration away from this ISP, which we hope to complete within the next few months.

If you’re an IRCCloud subscriber, we’re happy to issue you a month’s refund in compensation for this downtime - drop us an email at team@irccloud.com with the email address associated with your account.