No Man’s Sky: A Performance Analysis

nms-google

Update: I’ve received some suggestions about using external v-sync controllers and allowing the game to rebuild shaders after updating drivers. I’ll update the post when I’ve had a chance to run some new benchmarks.

It’s hard to find any reviews and reports and No Man’s Sky without hearing about performance issues. Game stability aside, there are reports all over the place about FPS drops and stuttering while playing the game. A Google search for “No Man’s Sky PC performance issues” shows 1.24 million results, including an article from Polygon titled Don’t buy No Man’s Sky on PC yet.

What are these issues, exactly? Frame time stability seems to be a major one. At 60 frames per second, it takes 16.7 milliseconds to draw one frame. There’s always going to be some variance, so if one frame takes 15 ms and the next takes 18 you’re not going to notice much; everything will still be nice and smooth. However, if you’re averaging 16 ms per frame and suddenly get frame times bouncing between 16 ms and 50 ms (that is, sudden fluctuations between 62 FPS and 20 FPS) that sudden, drastic change becomes a very noticeable stutter.

There seem to be a few different causes of this frame time instability. One is that the game is constantly generating new terrain and lifeforms using algorithms using a process known as procedural generation. It’s important to point out that this does not mean it is randomly generated. Procedural generation uses a formula with some number of input variables to generate everything, so if the same variables are given (X, Y, and X coordinates in space, for example) the output will always be the same. This means that the game as a helluva lot of math to do all the time. Every time you go somewhere new, every time you land on a planet, and every time you warp to a new star system, the game needs to take all of the inputs, run it through the formula, and start generating every plant, animal, mountain, mineral, body of water, space pirate, landing pad… you can see where your computer’s processor might start to sweat a little. Traditional games have levels created by a designer, so everything is already determined and there’s very little math involved. Ammo crate there, blood demon there, and typically all loaded into system memory before the game even starts.

Another issue is that, with such a small studio creating the game, they simply could not optimize the game for such a wide variety of system configurations so some incompatibilities and strange default configuration options are going to make their way into the “final” product. Things like G-sync, a technology that requires a relatively modern video card and a new, expensive monitor, are enabled by default with no way to disable without digging through system configuration files. Or, the fact that the game is locked to 30 FPS by default on PC.

So how does the game actually perform? Here I’m testing the game under two scenarios: traveling from a space station to a planet’s surface and exploring the surface of the planet, harvesting resources. This should give us a pretty good understanding of how the game performs under the most demanding scenario as well as a more typical one. My testing system consists of an AMD FX-8350 overclocked to 4.4 GHz, an EVGA GTX 970 SC, and 16 GB of system memory. Running the game as it’s purest default settings at 1080p we get the following results:

Frame times in milliseconds with default settings @ 1080p.
Frame times in milliseconds with default settings @ 1080p.

So what we’re seeing here is how long it took to generate each video frame. By default the game is locked at 30 frames per second, so each frame should take about 33 milliseconds. Most of the time that’s true. Between both scenarios, the average frame time was 34.9 (28.6 frames per seconds). As a comparison let’s take a look at a modern juggernaut of real-time graphics: DOOM.

Frame times from No Man's Sky (default, 1080p) vs DOOM (ultra, 1080p).
Frame times from No Man’s Sky (default, 1080p) vs DOOM (ultra, 1080p). V-Sync was enabled on DOOM for a more accurate comparison.

Here I’ve taken the frame time data from earlier and overlayed that with the frame times of typical DOOM gameplay. I chose DOOM because it’s a showcase of what modern hardware and software are capable of. If you follow the red data you can see that there are very few frames that deviate from the pack. This uniformity gives a very smooth gameplay experience. Going back to the No Man’s Sky data you can see that, not only do they deviate more often, but much further as well.

If we uncork the performance by disabling v-sync and turning the max FPS off (I’m not entirely sure how these are different yet) we can see a huge jump in performance.

unlocked-framerate

Overall the frame times have improved, with the average jumping from 39.4 milliseconds (28.6 frames per second) to 20.7 ms (48.3 FPS). However, the distribution of stuttering, deviant frames is still the same.

Just to see what would happen with the maximum graphics settings used with a wider field of view (90 on foot, 100 in the ship) I got this:

Frame rates with all the graphics settings maxed out and wider field of view.

Frame rates with all the graphics settings maxed out and wider field of view.

While running around the planet and mining resources, the frame rate was a little less stable than with the default settings, which isn’t that surprising. There were a few more deviant frames causing stutters but not a significant amount. However, leaving and re-entering the atmosphere saw a tremendous surge in stuttering. Granted, most of your time isn’t spent traveling between planets and space, but there should be something we can do to make that transition smoother. Hello Games has an experimental patch available that is supposed to improve performance on some AMD CPUs and 8-core CPUs (of which mine is both). It also disables things that should have never been enabled in the first place, like G-sync, to address performance and compatibility issues. First, though, let’s update my video card drivers.

The nVidia GeForce Experience control panel is telling me my current driver is from a month ago, and that the newest driver “Provides the optimal experience for No Man’s Sky, Deus Ex: Mankind Divided, Obduction, F1 2016, and the Open Beta for Paragon”. So that’s a good start.

Frame times with maximum quality settings and the latest drivers from nVidia.
Frame times with maximum quality settings and the latest drivers from nVidia.

Before the driver update, the average frame time was 26.6 ms (37.6 FPS). Afterward, it jumped down to 17.7 ms (56.4 FPS), which is a staggering change. We can see that the deviant frames are reduced overall, though some frames took much, much longer than most. How much longer? All the charts so far have had a ceiling of 250 ms (4 FPS) for uniformity. If I remove that ceiling…

Frames that took forever to render.
Frames that took forever to render.

Woah, woah, woah. Those 5 frames that we couldn’t see before? In total, those 5 frames took 4.6 seconds total to render. During these tests I’ve been jumping back and forth to the same region of the same planet, so while I understand that the procedural generation does take an awful lot of resources, it’s also weird that it does not seem to be caching planet data anywhere. I don’t know how much drive space a cached planet or five might take but if it’s able to smooth the transition from space to surface, but it might be a good trade-off if we’re given the option in a later patch.

Speaking of patches, how does the experimental patch affect performance?

experimental-patch

Wow! As soon as I loaded No Man’s Sky after patching it I immediately felt a smoother frame rate but I wasn’t expecting this kind of result. The average frame time dropped a respectable 2.7 ms to 15.0 ms (66.6 FPS) from the pre-patch 17.7 ms (56.4 FPS). I should mention that after patching the game I did get a crash between the two benchmarks, which I haven’t had until now, so there might be increased frame time stability at the cost of game stability. That said, it was a graceful crash that didn’t take the rest of the system down with it, and I was able to jump back in with no problems.

As long as the crashes don’t come too often I think it’s pretty good trade for an experimental patch. The game has been out for a week and I’m anxious to see what fully supported patches bring to the game.

I think I’m Giving up on Pokemon GO (For Now)

In my last post I talked about some of the issues I, and the rest of the Pokemon GO community, are dealing with as Niantic continues to patch the game and roll out to the rest of the world. I’ve been keeping my spirits high and continuing to play, but on my walk to work today I realized something. There’s nothing to do anymore.

It sounds like I’m being a little harsh, and I might be. You can still do things, but I’m not compelled to. Going back to my last post, gym battles are literally pointless. I pass gyms held by other teams and have no motivation to take it over. Or, I’ll pass gyms held by my team and have no motivation to spar and increase the gym’s level. Without any meaningful tracking system, I can only know that somewhere within a several-hundred-meter bubble there is a pincer that will likely disappear before I ever find it. Over the past couple weeks, I’ve only opened the game on my walks to and from work, hatching eggs and catching Pokemon that happen to wonder in front of me. During these walks, I’ll have the phone at my side, the screen turning black to save power. When I feel the vibration and look at my phone, I’ll spin my map or try to tap on the Pokemon only to see that the app no longer responds to touch inputs. After restarting the application the Pokemon I wanted to catch are no longer there. In other instances, I’ll tap on the Pokemon that spawned next to a Poke Stop. When I leave the Poke Stop the Pokemon will have disappeared or seemingly transformed into something else (yesterday a Kingler and Clefable appeared to turn into a Nidoran and Weedle).

I was a Pokemon Go apologist, and I still might be, but at this point, the game is still an early beta with no enticing gameplay mechanics. The fact that the current build is version 0.31 more-or-less confirms that. Last I heard, Niantic has made over $130 million off of Pokemon go with an initial budget of about $20 million. This is only going to grow and grow and they roll out their game to more countries, so I don’t feel like taking a hiatus for a while is going to hurt the game at all. Hopefully, by the time I get to Korea in October the game will be more of a game and less of a meta walking simulator.

Pokemon GO: IVs and why Powering up is Broken

Update: The original post had a few inaccuracies and has been updated accordingly.

Pokemon GO can be broken down into three core game components:

  • Finding and catching Pokemon
  • Powering (leveling) up and evolving Pokemon
  • Battling at, and controlling, gyms

The “3 Foot” bug has been plaguing Pokemon GO for weeks, making it nearly impossible to track down individual Pokemon. If they’re on the radar at all you know they’re somewhere but the only way to track them down is by consistently monitoring where they fall within the list compared to the rest of the Pokemon (Pokemon at the front of the list should be closer to Pokemon at the end). As of version 0.31.0 the footprints have been removed altogether. Posting on the official Pokemon GO Facebook page, Niantic says “We have removed the ‘3-step’ display in order to improve upon the underlying design. The original feature, although enjoyed by many, was also confusing and did not meet our underlying product goals. We will keep you posted as we strive to improve this feature.” It makes sense to not display something that’s known to be broken, and it very well might be causing confusion for new players, but it also sounds like they might not reimplement the original functionality. Hopefully what this really means is they’ll come up with a better way of tracking Pokemon; something akin to the Poke Radar or Dowsing Rod items from the core series games.

Before I explain why I feel like powering up Pokemon is a wasted effort I should explain how powering up works, the risks involved, and some of the underlying game mechanics. Please note that when I say something like “we know” or “this is how it works”, what I really mean is “As a community, we currently believe, based on the data collected and analyzed by groups within the community…”.

When you catch or hatch a Pokemon it comes with a predetermined set of stats. For a Bulbasaur I just hatched those states are:

  • CP: 594
  • HP: 60
  • Type: Grass/Poison
  • Weight: 6.35 kg
  • Height: 0.65 m

We know that weight and height have no influence on any other stats, so we can ignore that. Also, the typing, “grass/poison”, is the same for all Bulbasaurs across the game. The only stats that matter, then, are CP and HP. 594 for a little baby Bulbasaur feels like a lot. To the player (me) it feels like he’s a monster capable of quite a lot. But, we have other information to consider. In order to level up this Bulbasaur, we can see that it will take 2,500 star dust and 2 candies, but how much star dust and candies do I need to spend in order to get him up to a CP 2000+ gym defender? Through whatever means the community has figured out how to calculate a Pokemon’s level and IVs. They have also determined what the maximum level and IVs are, so now we can know what the actual potential of a given Pokemon is. Using Poke Assistant’s IV Calculator tool I entered the required information and got this result:

iv calculator

There are multiple combinations that could result in my Bulbasaur’s stats, each listed below. Sometimes the level may change, too, creating even more possible combinations. The higher level a Pokemon is, typically, the fewer possible combinations there are. So we can see that, with all the possible combinations, my Bulbasaur is somewhere between 84.4% and 86.7% of it’s potential (100% being 15 for attack, defence, and stamina).

Hold up, I haven’t explained exactly what IVs are yet. IVs, or “individual values”, are used to determine how strong a Pokemon can be. In the core games they determine your Pokemon’s physical attack strength, special attack strength, physical defense, special defense, speed, and HP. Here it’s a little simplified, only determining attack, defense, and stamina (HP). Basically, if we have two Bulbasaurs that are both level 20, but one has higher IVs across attack, defense, and stamina, it will be stronger overall than one with lower IVs. This is important to know when selecting a Pokemon to evolve and power up.

Let’s assume you’ve been diligent about not spending your dust and candies on powering up your Pokemon. You’ve chosen your prized Pokemon that you want to pour all your hard work into. Maybe it’s a Golduck with a maximum potential of 91.1% (remember, there are likely multiple possible combinations of level and IV with no way of knowing which is correct without powering up). Currently, I have 98,835 star dust and 32 Psyduck candy. That should be plenty to power up this Golduck into an unstoppable powerhouse. Right?

To level this Golduck from 20 (though we’re hoping it’s currently 19) to its maximum level of 40.5 it would take 242,500 star dust and 41 candies. And that’s gambling it’s actually level 19 with good IVs, requiring even more. If it’s currently level 20 it’s IVs are, at best, average and not worth powering up since it would take all of our current resources, denying a Pokemon with much higher potential the chance to power up.

Note: Your Pokemon’s max level is proportionate to your trainer level. I wasn’t able to find any hard numbers online, so I calculated the potential IVs and levels for my Tauros and started powering it up. Cross-referencing the perfection percentages, before and after, I was able to figure out which one was correct and what the current level was. After powering up Tauros seven times his level increased from 20 to 23.5 (half a level per power-up), his CP rose 181 to 1211 and his HP went up by a staggering 8. My trainer level is 22, so it appears that the max Pokemon level is [trainer level] + 1.5. Seems weird that it isn’t the same as your trainer level, so it’s probably a bandaid for early level trainers. What all this means is that, while Pokemon have a level cap of 40.5, you would need to have a trainer level of 39 to reach that, which, for legitimate players, is going to be a long way off. More likely, your max Pokemon level is going to be around 25. For the Golduck in the previous example, powering up from level 20 to 25 would consume approximately 18,500 stardust.

With weeks of playing, I can invest reasonably in one Pokemon. It feels like so much work for very little payoff. It’s disheartening.

The third part of the game is gym battling. I’ve tried so many times to find a reason to do this but there honestly is none. I live in fairly large city with a heavy technical presence (Nintendo, Microsoft, The Pokemon Company International, ArenaNet, Valve, WarGaming, and more are all 15 minutes or less from my home). Where I live, gyms are impossible to hold for more than a couple hours at best. Many gyms change color multiple times per hour. With a 21 hour timer between collecting gym rewards, you’re lucky to get control of two gyms before collecting your reward for the day. The largest number of gyms I’ve been able to control before cashing in is three, and even then it was mere minutes before all three had fallen. So, what is your reward for holding gyms? 10 PokeCoins and 500 dust per gym. 500 dust?! You get 100 for each Pokemon you catch, so if you catch 20 Pidgies and Weedles in a short session that’s 2,000 dust. You can do that in an hour pretty easily, compared to 500 for a gym every 21 hours. It just doesn’t make any sense. While you do get XP for winning gym battles it doesn’t come close to what you can get for just catching Pokemon. So what’s the point? Bragging rights for your team?

We’ve identified two issues that could potentially be solved by a single change. In the core games, your Pokemon gain XP by battling. Get enough XP and they’ll level up. So why not apply this same mechanic to Pokemon GO’s gym battles? It would require making a few changes but let’s run through some options.

  1. Dust bonuses for winning gym battles
    Did you beat a Pokemon at a rival gym? Have some dust. Did you beat the gym leader? Get a dust bonus. Are you in control of the gym? Dust bonus! Of course, dust could be applied to other Pokemon, but this happens in the core games with Exp. Share, so it really wouldn’t be that different.
  2. Add an additional level-up mechanic
    Currently, the only way to power (level) up your Pokemon is by spending dust and candies, and it happens all at once. Add a counter to the Pokemon’s stats that tracks battles won. Each battle increments the counter by one, gym leaders by 2, 3, etc. depending on the level of the gym or CP of the defending Pokemon. Once that counter hits a certain number, determined by the current level, ding! Level up!
  3. Add an actual Pokemon XP mechanic
    This is probably the most work and least ideal from Niantic’s side. Having XP for both the trainer and Pokemon could be confusing, and would largely eliminate the need for dust, except as a Rare Candy to immediately gain levels.

This is a subtle change but could help trainers feel like their time and effort are going somewhere.