Greetings Citizens, On Thursday, we opened Arena Commander’s multiplayer modes to a much larger segment of backers. Over the holiday weekend, our engineers kept careful track of server stats: number of successful connections, number of failed attempts, number of competed games, average latency and the like. We also spent time watching your Twitch feeds, which have proven to be an incredibly valuable tool for helping compare the real world experience to what we test in-house.
By releasing a new version of the game with many changes and also upping the number of simultaneous players, we exposed a variety of load cases and bugs we hadn’t seen before: ship movement and prediction issues, matchmaking bugs that include assigning excessive player counts to game servers and other tracking edge cases, and existing unused engine features causing problems.
The result is that with Patch 12.4, there is an unacceptable amount of lag and occurrence of rubber-banding (the effect when ships seem to ‘jump back’ in space.) While all of you know that Arena Commander is a pre-alpha module and that our aim is to determine and fix exactly these problems, they seem significant enough with this patch that we should take a moment to talk about how we’re aiming to repair them in the upcoming weeks.
I want to stress that this is part of our shared process, and it’s one of the things that makes Star Citizen’s development so unique. We hear about games that launch with dismal multiplayer issues every day. Unlike a traditionally published game, which launches fully formed into the wild, Arena Commander is letting us do real load balancing tests with the help of the community to flush out multiplayer and connectivity issues at this early stage. Getting this right now means having it right going forward; you’re helping ensure that future updates and modules improve on something that works instead of publish to a complete unknown.
The team was already preparing patch 12.5 for launch, which will add both fixes and a new surprise or two. With the rise of the new latency and rubber-banding issues, though, we knew we needed to up our focus. We had not seen these issues with the 60,000 players who had access to the previous patch, so our immediate thought was that one of our server-side changes may have caused them. On Monday morning, production sat down and developed a ‘to investigate’ checklist of possible causes of (and improvements for) for the issues we were seeing. Here’s a short sampling:
On Monday morning, the team went into action to determine the root cause for the issue. There were dozens of possibilities, all (or none) of which could be part of the issues we were observing. Bug reports submitted by dedicated backers were our first point for analysis. Like doctors trying to identify a mysterious illness, we looked for common environmental factors. Was there a significant geographic distance between players? Surprisingly, no: in many cases, players with almost no latency between them were still having issues.
The team moved on to examining our own code. Several possibilities presented themselves: leftover code was causing the game to attempt to connect to Crynetwork chat with each session, missing movement smoothing code that had been developed but not implemented in 12.4 and there were potential issues with the servers not correctly recognizing when they were full. Each of these possibilities became a JIRA task instructing Star Citizen’s engineers to investigate and repair a given item for the next patch.
We’ve already implemented a number of fixes which we believe will improve the multiplayer experience in 12.5, and we’re continuing investigating several other possibilities that we hope will improve the experience. Was one of these items the ‘magic bullet’ that fixes the multiplayer experience? Our tests are promising, but we won’t know for sure until we’ve patched Arena Commander and seen how it impacts players around the world. We know it can be frustrating not to see a ‘one button’ fix, but kind of extended process is necessary to preparing for the future launch of the Persistent Universe. Expect to see an improvement in 12.5, and greater improvements as we continue our work!
We’re learning how to make the investigative process better, too. On Tuesday, two of our programmers updated the server software to provide more detailed concurrency numbers. Knowing more specifically how many players are in the game in a given interval will be a valuable tool for investigating how the server behaves where changes are introduced in the future. Next, QA will use this tool to test 12.3 compared to 12.4 to try and discover where the performance changes were introduced.
In the meantime, I’d like to thank you all for your efforts in helping us discover these issues, both those waiting to play and those who have been dogfighting in less-than-ideal conditions. We’re working to make this the best possible experience, and that’s something we couldn’t do without the support and help of all of our backers. There will be bumps along the road as we improve the technology behind Star Citizen and the balance of the game itself… but we’re going to share the process by which we deal with all of these issues with you as it happens.
I’ve also had questions from several backers about international servers, expanding to places like Australia and Europe. Current indications is that this isn’t the cause of the problems we’re seeing with 12.4… but it is something on our roadmap, with plans to enable these servers by the time Arena Commander 1.0 launches!
— Chris Roberts