Meet the new and improved itch app!

Hi Amos, you’re back on the blog! Last time you were here you talked about what was coming up for our desktop client, itch. What have you been working on since then?

Hey Spencer, I can’t believe it’s been over a year!

Looking back at that previous post, it seems what I’ve been up to is: more of the same, but better. butler has been doing great. We’ve recently reached 100’000 builds pushed with it. I remember when “32GiB” was a comically large upper bound we set for builds, now it’s not unusual for games to flirt with that limit.

In 2017 I brought up patch optimization, for even faster game updates – but it was still an experimental feature, hidden behind an option. It’s enabled by default in itch v25, and we’ve made it even faster since. The backend has seen a lot of changes as well: we’ve overhauled the entire itch.io API, I’ve rewritten build processing to be more parallel (thus faster!) and more reliable. For example, an 1.5GiB game takes about 40 seconds to process now.

The itch app and butler were originally custom-built for Finji, and remembering that always makes me smile. They’ve both come so far! I was checking the analytics around itch v25’s release, and the total amount of downloads for the app (including updates) was around 2.5 million! That’s crazy.

Last but not least, I’ve been enjoying working with Jesus more closely. He’s been incredibly helpful in getting important updates shipped (including itch v25). A breath of fresh air!

So what has changed in v25?

Well… how deep do you want me to go? I have a five-part postmortem in draft, and it barely covers the first six months of development!

It’d be quicker to list what hasn’t changed. It’s still cross-platform (available on Linux, macOS and Windows). It’s still open-source. It’s still translated in over 20 languages (although, many of those are partial translations). It can still be used offline. And it’s still definitely the best way to play itch.io games.

Apart from that, there’s not much left untouched. When I took over the desktop app project in late 2015, I had never done this before. I had worked on web stuff, and audio stuff, and compiler stuff, and game stuff – but not “desktop app that can install almost anything” stuff. I’ve tried to clean up things gradually – fixing bugs as they appear, trying to prepare for the next feature, cleaning up the shortcuts I took for a particular release, etc.

But it got complicated. It took longer and longer to get anything done within the codebase. Making new builds was slow, testing them was slow, regressions starting popping up over and over – I was not a happy camper. It got to a point where I would groan any time someone suggested a change to the app.

So when we reached a point where no one was complaining too loudly about lack of features, I started switching out essential parts of the app with other, better parts. I knew it was risky, and I knew I wouldn’t get it “right”, even after years of learning more about the problem. So I focused on just a few angles:

• It needs to be small
• It needs to be fast
• It needs to be reliable
• And most importantly: it needs to be easy to develop and easy to update

That all falls in “easier said than done” territory. That’s why I ended up replacing almost everything!

After 2 years of organic growth, the app’s dependencies (third-party components) were all over the place. I’ve always had a hard time deciding whether to use someone else’s work (and not reinvent the wheel), or to just roll our own. Starting that big overhaul meant re-evaluating all those decisions.

I went through all dependencies one by one and eliminated most of them. Then I took a closer look at packaged builds, identified unnecessary files, and wrote a cleanup tool to eliminate them.

I decided to adopt SQLite to store local data (all your games, collections, etc.), and then I rebuilt everything on top of it four times. This was a big win for startup time: v23 used to take over 10 seconds to start up for me because of the size of my library!

I evaluated multiple solutions for compiling and bundling TypeScript code – some of which came out during itch v25’s development – and found a compromise that’s both friendly in development and makes for small and fast production builds. For example, in itch v23, all the app’s code was loaded on start-up. Now it loads only what’s needed, when it’s needed. It makes a noticeable difference.

I took all the “business logic” (querying the itch.io API, installing, updating, launching, sandboxing, uninstalling) and moved it to butler. Instead of shoving everything in one project like I used to, I separated each concern in its own separate project – with separate tests and roadmaps.

I got rid of more stuff. We were using a third-party solution to distribute and update the app on Windows and macOS. But why not use our own solution? butler is solid now, right? So now, everything is distributed directly via itch.io. The app, its installer, butler, prerequisites like the DirectX runtime, the Linux sandboxing helper – everything. It’s dog food all the way down.

The app was almost ready. But it didn’t feel new – it just worked a lot better. So I started redoing parts of the user interface. In the process, I discovered and fixed more bugs, and added more features that were real easy to add now!

I’m omitting a lot here – it truly was a wild ride. And if you think I didn’t get technical enough, well, you’ll love the post-mortem 🙂

What do these changes mean for users?

First of all, please remove all traces of the previous version from your computer, grab v25 from https://itch.io/app and give it a shot. It should speak for itself.

But honestly? I hope nobody notices anything. The app should fade into the background, like a soundtrack. It’ll take up less space on your disk, less of your RAM, and less of your CPU. It’ll quietly do the work and let you experience all the wonderful stuff creators have been uploading to itch.io for the past five and a half years. If you encounter an issue, there’s a much better chance we’ll be able to track it down and fix it (did I mention v25 comes with a new feedback system?). Then again, there’s a good chance you won’t encounter it at all.

If you want to contribute to the app, it should be easier than ever to get started. If you just have a suggestion, it might just ship in a matter of hours, not weeks.

Do you have a favorite feature that’s been updated?

Yes! I get really excited about fundamental stuff though, so it’s going to be a niche answer.

Up until now, when installing a game, the app first downloaded an archive to disk, and then extracted it. It was fine for a while, but when you started installing games in the 10GiB club, it quickly became a problem. Worse, we kept the archive around, in case you wanted to do a clean re-install of the game.

But sometime in 2016, we figured out how to access remote files (that live on a server probably an ocean away from you) as if they were right here on your computer. That meant we could extract archives while downloading them.

But that wasn’t good enough. What if the download got interrupted somehow? What if you just wanted to pause it for a while? Would you have to start over again? That gets old quick, especially if your internet speeds are on the potato end of the spectrum. So we tried something crazy. What if we could save the state of the decompressor on your disk, so that we can resume any time, no matter what happens (downloads paused, app quit, computer reboot, catastrophic electrical failure).

And it worked. For .zip files at first, and then we figured it out for .tar.gz. And for .tar.bz2. And for brotli (the compression algorithm we use for butler patch file). And then we managed to convince 7-zip to do almost the same thing for all other archive formats we support. It took a lot of work, but everything is resumable now. When I’m down, I just hit that pause & resume button, and remind myself of all the pieces working together to make that happen seamlessly.

By the way, that’s how itch v25 determines how much disk you need to install a game! In that “Install” dialog, it opens the remote file, figures out what format it’s in, gathers information, and provides an (often pretty accurate) estimate. That’s why it takes a second or two – it’s doing a lot of work behind the scenes.

The work never ends, so I have to ask: What’s next for the itch desktop client?

Well, we’ve had a bunch of folks complain that the interface of the app is powered by Electron. So I’m thinking of getting rid of the interface altogether. No pictures for you. Just text. No, you know what? Not even text.

In all seriousness, I’m not sure yet. I’ve been working on this overhaul for so long that I need to take a breather and evaluate which of the many directions we should take now. I’d like to improve Linux compatibility. I’d like to resume work on capsule. I’d like to finally ship collection editing in-app. And I’d like to improve integration between the website and the app – they’ve been developed pretty independently of each other, and it shows.

There’s really no shortage of ideas. Time will tell!