Wednesday January 16 2013 • posted by RJ

Recent improvements

We’ve been busy improving IRCCloud - here are some recent changes we want to highlight.

As always, you can talk to us in #feedback if you have any questions or suggestions.

Alert sound

An alert sound now plays when you receive new highlights and PMs in the background. The sound effect is on by default, and you can disable it in settings.

Here’s the MP3 file for our sound.

Webkit desktop notifications

In Safari and Chrome, we’ve had desktop notifications for a while. We’ve switched to having them dismiss automatically after a few seconds. Clicking the popup will focus the message. There is a setting to disable autohide if you wish.

Desktop notification

User mode indicators

See inline if someone is an Op, has voice, etc. Defaults to coloured dots:

User mode dots

You can change it to traditional symbols (@, +, etc) in your settings:

User mode symbols

Commands to run on connect

Auto-send commands when you connect to the IRC server. For example, you can AUTH with the Q bot on QuakeNet this way:

Example of on-connect commands for AUTHing with Q

Improved nick tab-complete

Type the first few letters of a nickname and hit tab, to see our improved tab-complete menu.

Tab-complete menu

Press tab some more to cycle through the options.

Password security

We now store all user login passwords using bcrypt hashing. The previous salted MD5 hashes are removed and replaced with bcrypt the first time you login after your account was migrated.

HTTPS / SSL Security

IRCCloud continues to only be accessible via SSL (ie, https://). Also:

  • We have reviewed and updated our SSL cipher suite;
  • We are using HTTPS secure cookies;
  • We send strict transport security (HSTS) headers;
  • As of Chrome 22, the irccloud domain is on the force-https list.

Ignoring troublesome users

Often requested - now you can ignore people, and anything they say won’t display in the chat.

Basic usage:

  • /ignore bozo
  • /unignore bozo Advanced usage (masks):
  • /ignore <nick>!<user>@<host> (wildcard \* is supported)

There is also an ignore option in the nick context menu, which defaults to using \*!<user>@<host> for a more reliable ignore.

Check your current Ignore list from the Options menu. You can also use a blank /ignore command with no target to show the list of masks.

Channel management improvements

Conversations now automatically rename when the person you are talking to changes their nick.

Fixed a long-standing bug where channels failed to rejoin after a disconnect.

We also improved initial chanel joining to more reliably wait for confirmation of NickServ identification.

New shortcuts

  • /archive - leaves a channel if necessary, and archives it (or /close or /a or /wc “window close”)
  • /delete - leaves a channel if necessary, and deletes it (or /rm or /wd “window delete”)

Connectivity, IPv6, Hardware

We recently deployed 3 brand new connection pool machines, which we lovingly name after places in London that have outdoor pools. The new machines are called tooting, richmond and hillingdon.

Here is our current list of IP ranges, useful if you run an IRC network and want to whitelist IRCCloud.

We now favour IPv6 IRC connections if your network supports it. Lots of the major networks already do—Freenode for instance. This means we can assign each IRCCloud user a personal IP address and hostname on each of our connection pool machines. This makes it a lot easier to uniquely identify users on other networks.

General tweaks

  • Modal dialogs replaced with in-page overlays
  • Various bugs fixed when using irccloud with ZNC
  • Fixes for connecting to LDAP secured IRCDs that check usernames
  • Lots of little bug fixes

Thursday September 20 2012 • posted by James

Downtime post-mortem

Last week, on Thursday September 13, we attempted a planned upgrade of IRCCloud with an expected down time of up to 2 hours. It ended up taking all day, with partial-to-no availability for most users throughout.

Sorry for the poor service - we realise how much of a pain it is to be without your normal IRC client, and we’re busy working to make sure it doesn’t happen again.

Background

IRCCloud started out hosted on one server, where we changed settings and tuned the system as we went along. As part of growing the service, we recently invested time in moving all our configuration management to Chef, with automated deployments and upgrades provided by Jenkins (the subject of another post in future).

These changes allow us to easily deploy additional servers, safely upgrade the systems holding open your IRC connections, and run the site in a more robust and manageable way.

We tried to move the core service to our new infrastructure last week, but ultimately rolled back to the old system to properly address issues uncovered by the migration.

Alpha site

For the last 4 months we’ve been running an updated alpha version of the site with a small group of testers, in parallel with the beta.

The Alpha site lives on the new servers and infrastructure. Our plan was to migrate all users to the new site last week. This was going to involve some down time while we shut down user accounts and IRC connections, migrated them to a new database structure, and started them up again on the new system. Logs also needed migrating, but this was being done ahead of the downtime.

What happened

On Thursday morning we shut down the beta site, disconnecting everyone as planned, and began to migrate user accounts in the order they signed up. At the same time we redirected the irccloud.com domain to the new server, so people could immediately log in once migrated.

After resolving some initial problems with the upgrade procedure, we noticed the migration queue was being processed far slower than expected, owing to heavy load on the alpha server.

Throughout the day, we patched the system to address problems that were cropping up. With hindsight, we should definitely have aborted early on in the process and switched back to the old system.

We also had a couple of crashes caused by incorrectly set configuration parameters. Settings we had previously changed on the old system, but missed when building a new server. All our configuration is managed by Chef now, so mistakes like this will be much less likely when we deploy new servers in future.

At the end of the day, we had almost all users migrated to the new server. We left the remaining few migrations running over night. Things were working, but sluggish.

The following morning, rather than frantically patch the system in place, we decided to move everyone back to the old server. Although painful, this was better than inflicting slow and laggy performance on everyone for any longer. We should have done this much sooner.

Artificial load testing: too artificial

As part of testing performance on the new server, we generate artificial load with a test suite. It turns out our artificial tests were too artificial. The distribution of messages/joins/quits/parts/renames/etc we use for testing no longer resembles the traffic we see in the live environment.

We’ve fixed many of the issues that led to the server overloading and we’re improving our load-testing to better model the specifics of a full restart and ongoing operations. We’re also working to simulate other transient events better in our test environment (eg, netsplits, mass disconnect/reconnects, etc) before we migrate any more accounts to the new system.

Ongoing issues

We have a list of IRC networks where our session limit needs updating for the new servers. We’ve been contacting the relevant network admins this week to improve matters. Please email team@irccloud.com if you’re an IRC network admin and have any questions.

Some users are reporting a varying backlog gap in some channels between the end of August and near the time of the failed migration. This is a temporary display problem that should resolve itself as more activity in those channels pushes the gap out of cache. This might take a bit longer for quieter channels though.

Separately, we have a known issue on beta where backlog is inaccessible from before August 12th. This backlog is safe: it’s been moved to the new system, because we ran out of space on the current server. This is obviously an ugly solution and would have been fixed by the database migration. Since the rollback, this data is again unavailable. It will be accessible again once we migrate over to the new server.

If you’re having any other issues in the mean time, you can contact us via email, twitter, or in our #feedback channel on irc.irccloud.com.