Last night we experienced approximately 12 hours of downtime between around 18:00 and 06:40 UTC, caused by a prolonged period of internet routing issues which our ISP has attributed to a failed line card in one of their routers. This was our longest period of downtime in many years and we’re very sorry for the disruption it caused.

Running a large service which interfaces with the venerable IRC protocol poses a different set of challenges to most modern web services: Firstly, we have to manage a large number of outbound IRC connections while ensuring as few disconnections as possible. Secondly, IRC networks expect our users to connect from a consistent set of IP addresses, and lastly, IRCCloud is subject to a high volume of distributed denial of service (DDoS) attacks.

These constraints mean that our outbound connection servers, which actually make your outbound IRC connections, have been hosted for years by a specialist DDoS-resistant hosting service provided by a major ISP. This is a costly part of our infrastructure, and it wouldn’t be economical for us to completely duplicate these servers elsewhere to mitigate against rare situations like the one last night. Switching to another ISP - even if we could find one to provide the required servers at short notice - would involve a long process of getting new IP addresses whitelisted by IRC networks.

Our current architecture also restricts us to running our outbound connection servers in relatively close proximity to the rest of our infrastructure (which is hosted on Amazon Web Services). Over the last few months we’ve been working on a significant update of our backend software to remove this restriction - in fact, we started rolling this update out yesterday.

These improvements will make it easier for us to investigate other approaches for our outbound connection servers in future, and we’ll certainly be discussing network redundancy with our ISP and future providers.

If you’re an IRCCloud subscriber, we’re happy to issue you a month’s refund in compensation for this downtime - drop us an email at team@irccloud.com.