Dev Diary #397 - Shadows of the Shroud is Out, What's Next? (Performance)

by Eladrin

Hello everybody!

Shadows of the Shroud and the Stellaris 4.1 ‘Lyra’ update have been released, and today we’re going to go over some of our performance investigations and next steps moving forward.

Our 4.1.4 Open Beta found some critical issues, so will not be released this week. We hope to have a 4.1.5 Open Beta with some more fixes up before the weekend.

4.1.4 has a large number of fixes:

4.1.4 ‘Lyra’ Open Beta Notes

Improvements

• Removed district specializations from showing that you must not have the Wilderness origin.
• Added subtitles for secondary districts on Hive Worlds, Machine Worlds, Ring Worlds and Ecus.
• Added a Sol variant for Mindwardens.
• The Celestial Orrery system is now more likely to spawn.
• Modifiers from the Capital Productivity technologies should no longer be listed multiple times.
• Fledgling Dragons can now be hatched at a space fauna hatchery.
• Active accords for Endbringers empires now unlock Psionic ship components.
• Prevent AIs from taking the Mindwardens origin.
• Harmonized Mindwarden name lists

Balance

• Shroud Seals now give -50% fire rate to Psionic Weapons in their system.
• Buffed Tooth of the Eater with psionic components
• Create and Abolish special world designations (such as Resort or Thrall World) now take 180 days, giving you the opportunity to cancel the abolish decision before it happens if you selected it by accident.
• Death Priests now only gain +3 unity per 100 jobs if a sizeable percentage of your population is sacrificed.
• Resource Consolidation blockers are now more severe.
• Wilderness Empires will no longer be offered the Tomb World Adaptation since they can only end up on tomb worlds in very rare scenarios and it's mostly a dead tech.
• Civilian ships (such as science ships or construction ships) now automatically upgrade to psionic shields correctly.
• Gestalt empires can now research psionic cloaking devices if you have both Shadows of the Shroud and First Contact.
• The Aura of Whisperers now impacts cloaking strength.
• The secondary districts on the initial machine world for Resource Consolidation now have lower costs and upkeep before the world is finished.
• The Prototype Machinery blocker on the initial machine world for resource consolidation now costs 2500 alloys to remove.
• Shroud-Touched Regions now produce more Zro so you no longer start with a deficit.
• The Pierce the Shroud decision will no longer remove rare deposits.
• The Shroud-Forged origin now provides additional trait picks and points for the Shroud-Forged pops, allowing them to be robo-modded without removing any traits.
• Fully embracing the Shroud-Forged now sets them to be your primary species.
• Reduced the effectiveness of the Composer of Strands blockers and reworked their bonuses.
• The decision to add Shrouded Vegetation now takes a year to enact and can only be enacted a limited number of times per colony, scaling with planet size.
• Shroudshaper governors now provide telepath jobs scaling with their level, not the number of pops present.
• Death Cult and Superstitious Beliefs no longer block each other.
• Fortune Tellers now produce far less trade.
• The Realm of Madness event now gives insight on Zro Distillation tech.
• Teeth of the Eater covenant fleet now scales the number of ships on the time of the game.
• Shroud Observators and Extra-Dimensional Research Units from the Shroud-Forged origins are now job swaps for Biology Subroutine and Physics Subroutine jobs respectively. This means that they now fully benefit from modifiers affecting those jobs.
• Intensification: This Covenant Power for End of the Cycle is now on a 10 year cooldown instead of being on a one-time use.
• Tankbound civics now apply the following modifier: Upkeep from Jobs: -25%

Bugfix

• Improved repurposing of slaves (makes unemployed slaves show up properly and makes enslavement of specialists and rulers work better)
• Shroud seals are now dismantling properly even by other empires
• Fixed leader details not being present in events when they die.
• Fixed a number of weapon tags on space fauna mutations.
• Fix instances where The Instrument of Desire's aura effect would sometimes make planets' Planetary Ascension level go past 10.
• Composer of Strands' "Calling of Creation" now successfully completes after terraforming a world.
• The Baol, Gaia-Seeded and Under One Rule Terraform Insight effects now count towards the Composer of Strands' Calling to Terraform a planet.
• Instrument of Desire's Calling of Sculpture now properly completes when replacing a building.
• Multiple callings from the Cradle of Souls have been adjusted / fixed.
• Fixed assorted issues with the descriptions of federation perks for the Holy Covenant federation type.
• Production Overseers cannot be automated.
• Enforcers now correctly benefit from species traits that provided additional job efficiency to specialists that they were missing out on.
• Fix the generic Friendly Territory Tracking Mult modifier being ignored.
• Removed extra pre-ftl maintenance drone from game files.
• Fixed some pre-ftl hive mind jobs being unlocalised.
• Fixed Subterranean Urbanization and Coastal Hamlets not showing the correct icon.
• The Archives specialization on habitats now has the correct icon.
• Fixed being able to make Ecus into penal colonies, as you couldn't make penal colonies into Ecus.
• Typo fix in 'Lone Survivor' Event.
• Preventing Wilderness pops from getting Auto-Modding traits as Wilderness Empires no longer work jobs like regular empires.
• Add missing localizations for a few armies.
• Fixed job efficiency modifier for medical workers incorrectly stating it targeted farmers.
• Fixed the various strategic resource district specializations from showing up in technologies multiple times.
• Put a 1 year cooldown on the "Naval Activity in SYSTEMNAME" event so that people can't spam each other with it by starting and stopping the associated special project.
• Fix Armies getting stuck on colonies that get destroyed.
• Wilderness Empires now use exclusive defense armies, look at that unique icon!
• Natural Design empires will now be disgusted by the thought of ever using the weird vials inside of the Orbital Speed Demon.
• Mutation traditions can now add and remove the budding and crystallization traits.
• Corporate Death Cult and Chosen Executives are no longer mutually exclusive.
• Fixed issues with the calculations for sacrifice effects.
• You can now fire commanders who are governing planets.
• Viewing a pre-ftl planet no longer spams the error log.
• Pre-ftl brain drone jobs now correctly show icons in tooltips.
• Resource Consolidation blockers now take time to clear.
• Fixed Machine Intelligences being forced to have the Shroud-Forged origin if you didn't own The Machine Age.
• The Psionic Machine portrait will no longer have access to organic traits.
• The Mindwarden portrait now also has access to reptilian traits.
• You are once again only able to get one Warform. The Animator of Clay is not that generous.
• Fixed ascension perks being used for the category_last_picked_tradition trigger.
• Chosen and Shroudshaper leaders now benefit from Transcendent authorities.
• Fixed Transcendent Authorities not listing additional effects they grant to Psychic leader trait.
• Mind over Metal achievement now requires psionic cloaking devices if you have First Contact.
• Updated the draw weights for technologies that unlock monuments.
• AI empires with Bioships should now budget appropriately for armies.
• Assorted events that spawn armies and purge your population should now correctly set your population to be purged.
• Added a newline before the warning about jobs that cannot be favorited.
• Driven Assimilators with the Voidforged origin no longer start with far fewer generator districts than they should.
• Livestock can now work necrophyte jobs in hive mind empires.
• Livestock jobs now show the correct building icon.
• The Composer of Strands no longer adds Shroudstone to the Synaptic Lathe.
• The Chosen's homeworld is now correctly set up for the Life-Seeded origin.
• Shroud-Forged events will no longer attempt to convert a fractional amount of pops.
• Increase the assembly score for the Shroud-Forged pops.
• Machine and Robot modification technologies now provide the same number of trait points for both machine and mechanical species.
• Active recon fleet order now uses correct logic.
• The Strand Afflicted trait now grants Zro production to Biologist jobs.
• Corrected the notification for the Composer of Strand's aura surge to correctly refer to Shroudstone blockers (in English).
• Betharian Processing district specialization is no longer shown multiple times in the technology card.
• The caravaneers no longer provide misguided information about the Waste Reprocessing Center.
• Production Overseers now provide automated workforce efficiency.
• Death Cult sacrifices no longer auto-cancel.
• Mindwardens can no longer gain the Shroud Preacher trait.
• Fixed an exploit in Multiplayer where you could get certain event rewards as often as you could click the option before it went away.
• PSI Corps building can now be built on pleasure worlds, regardless of other present modifiers.
• Changed some shroud paragon ethics from fanatic to standard.
• Patron-Chosen Leaders now produce Influence.
• Endbringer related event no longer targets machine leaders when it shouldn't.
• Wilderness empires can no longer build Transit Hubs.
• Fixed Metallurgist and Artisan jobs from buildings on Penal Colonies. 
• Fixed Data-Driven Theorem Facility and Research Supply Depot getting instantly destroyed after being built.
• Sky/Empyrean Domes can now be built on Resort Worlds.
• Tankbound empires now have their rural districts automated on ringworlds and habitats.
• Ringworld districts providing worker jobs are now fully automated for Tankbound empires.
• Auras now properly affect Pirates, Space Fauna, Crisises and Fallen Empires.
• The Chamber of Silence now provides psionic job efficiency scaling with telepath jobs (as the Psi-Corps).
• Endbringers no longer have access to the Orbital Psi-Corps.
• The Orbital Psi-Corps now takes telepath drones into account.
• The Shrouded World in the Endbringers home system can no longer be a terraforming candidate.
• The Shrouded World in the Endbringers home system can no longer have a Zro deposit alongside a mineral or energy deposit (research deposits are fine).
• Fixed a bug when Repurposing slaves would not become Orderlies when changing their species' slavery type.
• Tankbound civic not removed systematically when losing an ideology war.
• Superstitious Beliefs cycles and effects removed when the civic is lost.
• Updated Concept Tooltips across a series of origins to be in line with 4.0 changes.
• The Unplugged traits now properly displays their descriptions.
• Updated species trait tooltips to be consistent with 4.0 changes.
• Removed the unintended energy upkeep on Habitat Orbitals.
• The Grand Battlement of Steel now properly provides soldier jobs.
• Updated Aristocratic Elite, Technocracy, and Exalted Priesthood tooltips to be in line with 4.0 changes.
• Added redundancy checks for the Contingency spawning in case one of the planned machine worlds is destroyed ahead of time.
• The Vivisection First Contact Events should now behave properly for Evolutionary Predators.
• Psionic Malfunction event can no longer kill Bio-Trophies for Rogue Servitors.
• Cradle of Souls aura surge now respects defense platform limits.
• Mindwarden Enclave's defense fleet doesn't spawn debris anymore.
• Sociocultural History should no longer appear for Wilderness empires.
• Fixed Evolutionary Predators stealing dna from Mindwarden species not rewarding reptilian dna.
• Experiment Engineer job uses the flat amount of crime considering the pops' happiness.
• Fixing outliner sometimes changing order of categories on show/hide/tab change.
• Fixed empires (most notably pre-FTL) being generated with the wrong habitability preference.
• Psionic species trait no longer show up in the empire creation menu if you're a machine species.
• Mindwarden empire will now be able to be force-spawned
• Advanced AI Empires galaxy parameter will now spawn the right amount of advanced empires
• Player origin randomization will now always allow non-advanced origins
• Allowed colonization of black needle planet when its associated Treasure Hunters empire is integrated or destroyed
• The Endbringers origin and Augmentation Bazaars civic are now incompatible, preventing a block on all ascension paths.
• Add branch-offices bonuses for almost all Shroud civics
• Shroud civics requirements improvements
• Cetana is now allowed even if a Shroud-Forged empire is in the galaxy as intended
• To prevent the Arc Furnace deposits from lingering upon its removal, the Arc Furnace now adds unique deposits to planets in the system.
• Prevent double base entry in some tooltips
• Add missing description for Assault Wardens army.
• Give Evangelising Zealots personality to the Exiled.
• Give Gestalt Chosen civics Growth node experience gain modifier.
• Fix Blind trait modifier not displayed in workforce tooltip.

UI

• Stopped showing unavailable ships in the naval capacity ship size limit tooltip.
• Resettlement window shows integer jobs amounts instead of floats.

Stability

• Prevented crash when no authority was available for a government during galaxy creation.
• Fixed issue with crashing when clicking on traits from the summary screen when editing a user empire.
• Fixed CTD in shroud view when resync has been done.
• Fixed CTD when disbanding armies.
• Fixed issues with planetview crashing when tooltiping certain elements and resync happens.
• Fixes OOS due to faulty reading growth data from savegame.

Post-release support will continue with more bug fixes and balance updates, with the next patch expected about a week from now.

Internal Performance Investigations

Performance remains the highest priority of the internal Stellaris team. A portion of the team will be fixing bugs, balance, and stability issues (especially multiplayer stability), but a dedicated subteam from the Custodians exists that is exclusively focusing on performance.

Today I’m going to go over some of our internal investigations and leads. It’s our goal to make steady progress on performance, but this is going to be a long haul and won’t be a “magic patch” that fixes everything. We have several angles to pursue, however. How modifiers are applied, fleet counts, UI impacts, and economic balance are four different major targets under investigation.

While investigating some 4.0 saves from the bug report forum, I noticed an unusually large number of ships in the late game saves (over 24000 ships!), and had the analytics team dig up some information. I posted this on our forums earlier this week, which compares the average number of ships in all “default empires” across a month each of 3.14 and 4.0 games:

The more powerful economies of 4.0 are causing us to reach previously end-game counts of ships a hundred years earlier. Some less rigorous checks of saves from our overnights and those provided in the bug-report forum also indicate that the 4.0 AI is much better at filling out their naval capacity and expanding it, but not nearly to the degree that players are. The AI is not as adept at stacking non-linear bonuses the way you guys are, so even if it’s several times as effective as it used to be, if you’re ten times as strong, it feels weaker overall.

In my next investigation, I took a Huge Grand Admiral game 200 years into the game and ran a series of tests, exploring how the number of ships in the galaxy affected the save. During these trials I also found that there was a substantial difference based on what the camera was looking at. Viewing Havenstar, the capital system of one of the largest empires, generally performed worse than viewing the entire galaxy map as an observer, and that itself was worse than staring at nothingness in an exploded azilash.

Huge Galaxy Grand Admiral stats in the 4.1.3 release build, 2400.01.01 - 2401.01.01.

Results from the above performance tests on 3.1.1. What you’re looking at in-game can affect one_year times.

We’re also seeing 4.1 performance being marginally better than 3.14 until a little bit after 2300, after which the two suddenly diverge. We’re currently in the process of generating more saves and analyzing the data to get a better understanding of exactly what’s happening here.

3.14 (blue) vs 4.1 (red) one_year times at various dates throughout the game  

These tests give us several directions to pursue - reducing the number of ships in the game through a combination of economic balancing and mechanical changes looks like it has some promise, but we also clearly have client-side improvements to investigate.

4.0.23 (blue) vs 4.1 (red) micro tick times at various dates throughout the game

4.1’s release day announcement was later than normal because we were concerned about whether 4.1 would harm performance. This was one of the performance graphs that contributed to the final approval for release.

I’ll now turn this over to Gosia to discuss the methodology and process.

How QA Does Performance Analysis

Hello everyone,

My name is Gosia, and I work as a QA Analyst on Stellaris. I’ve been with Paradox and Studio Green for some time, and I’m excited to introduce myself to you. o/

First and foremost, I would like to thank you all for your commitment, continuous support, and dedication to reporting all the issues Stellaris has. You have played a major role in fixing thousands of bugs throughout the years, and I can tell you, albeit you might not believe me, that we see you, we hear you, and we appreciate you. You’ve been an enormous help to the QA team - you have no idea! 

I was thinking about how to bite this topic in the dev diary and I think it would be best to start from scratch: do they even test it? Do they even care? What’s the process? 

The answer to all of those questions is: yes. Performance has had our attention even before 4.0 was released, and since then, we have been continuously improving our workflows and adapting the methodology to make every pop in the Galaxy happy.

The methodology and the process 

At Paradox we have machines ranging from low-spec to high-spec that we use to track performance 24/7. Both low and mid-spec machines have older generations of CPUs and GPUs, including specifics like a lower amount of RAM. This is to ensure we monitor how the game behaves on different desktops, investigate differences and issues in the performance, which lastly can result in fixes we all want.

Each week, our QA team runs performance sessions, rotating the responsibility among ourselves. 

Once the results are in, we review them with our game design and programming colleagues to ensure a well-rounded evaluation and the best outcome.

Let me dive into some details - this could be particularly interesting for those of you who are curious and like experimenting with performance testing yourselves. 

1. Preparing the performance run

At first, we need to make sure that this launching argument is set:

This keeps the random seed (all the random events in the Galaxy) the same on the save you made. 

Why is it so important?

Because our game is so complex, with so many different systems, different crises or situations might impact the performance and, as a natural consequence, invalidate the testing results. War in Heaven may not distress the game as heavily as Cetana, for example, and vice versa.

That is why having the same saves when executing the comparison between versions is crucial.

Of course, sometimes, due to the new content we release, it is not possible - only then we allow ourselves to run different saves BUT trying to make the settings and save as identical as possible. Keeping the same random seed will significantly reduce the divergence of the two galaxies.

From there, the only debug command that should ever be used is human_ai and nothing else. 

Any other command may falsify the results - i.e fast_forward might generate more issues that we are not aware of. The more natural, the better.

Once the performance machines are done, we gather the necessary files generated by the game and, using a special tool, generate graphs, as presented below.

For example, here, you can see that both versions, 4.0.17 and 4.0.22, run quite similarly, which means that we neither introduced new issues nor improved.

2. one_year, thirty_year commands

Once we have graphs, regardless of the results (improved or not), we load the save and do in-game checks.

One of them that is well known to you all is the one_year debug command that checks how fast the game can get through X years. We run thirty_years, either from the last save made or starting a completely new game and running in parallel on different PCs with different versions of the game. 

Think 2200 to 2230, compare, then again 2230 to 2260, compare until - if time allows - the end-game.

This type of test has exposed in a couple of runs that 4.1 generally is faster in the early game, but slows down heavily around mid-game.

It is essential to highlight here that it matters where you are looking - if it’s a system or galaxy view. If you run it only in the system view filled with cosmic storms or ships, the game will run slower. 

3. Script profiler

One of the most important checks, often exposing the biggest offenders of our performance.

Shows all the checks that we do in the script and code.

The values above show the total time, the number of hits (checks), and the average time to perform one check. 

As presented in the screenshot, the red square shows that the event.bio.755 has a very high total time value, with 1845 hits on a monthly tick. The average is also high if you look very closely.

One could suspect that BioGenesis is causing the distress and the aforementioned event, but if you expand bio.755, you will see that, in fact, the culprit is the UpdateModifiers (modifier threading) - and in many others, by the way.

This check is the most extensive, and sometimes, we sit there for hours, dwelling on the results and observing.

4. Overnight debug command

Overnight is not commonly used for performance tests, as it might easily generate wrong results, but what is good about overnight is its speed and the amount of saves it generates. Essentially, we can quickly get to 2500 and from there check saves made every 50 years to see what is happening in the Galaxy. It’s a good way of testing performance issues, if we notice that there is a breaking point in the graphs.

Currently known issues:

• UpdateModifiers check that happens every month heavily deteriorates performance and causes the game to stutter on the monthly tick
• The Pop Groups often do not merge when there are <100 pops
• The tick rate and performance drastically slows down during the mid-game which might be related to the above
• AI tends not to merge fleets 
  

Changes We’re Exploring

Back to Eladrin:

Modifiers:

As Gosia mentioned, the Custodians on the Performance Team are currently rewriting how Modifiers are calculated and improving their threading. We consider this a “potentially dangerous change” but it should have some positive (but hardware-dependent) impact, with greater impact on the late game.

Due to modifiers being a core system of the game and touching literally everything, we’re going to have to be very careful with deploying any changes here.

Ship Counts:

One of the common suggestions we see is reducing the number of ships and fleets while increasing their relative power. A mod was posted this week that happens to do precisely this - multiplying ship costs and power by ten. This is somewhat similar in nature to one of my internal design experiments, though my variant is based partially on how Hearts of Iron handles advanced vehicle design.

In Hearts of Iron, as you progress through the technology tree, you unlock progressively more advanced vehicles. In the case of planes, for instance, you’ll go from propeller-based Inter-war Airframes with pretty mediocre stats, up through Basic, Improved, Advanced, and Modern Airframes.

Our experiment with this includes advanced variants of the basic hull types which use more Naval Capacity than their base variants (but not Command Limit) and cost more to build and upkeep, but are significantly more powerful and have inherent shield and armor hardening. In our current experiment, the Mk.3 hulls were given an additional Auxiliary slot on the Stern ship section. 

This change reduced mid and end-game ship counts by approximately 25% with some of the benefits expected from that, but the inherent hardening and change in the relative value of ships also has an interesting effect on making bypass weapons less oppressive. The next iteration of this experiment will be more extreme and increase the difference between the tiers - I quite like the effect it has of making individual ships and fleets more important.

We’ve also investigated a system where enough of a specific type of ship would group together into a “squadron”, but our early experiments didn’t prove especially promising. This might get revived again later if ship counts remain high, or if there are other combat changes planned.

Economic and Modifier Balance:

The biggest design-side task is to analyze and curb some of the non-linear growth in the post-4.0 economy. While we want planet design and the decisions you make to remain rewarding, currently there are some elements that create excessive positive feedback loops or are otherwise above the desired curve. Like the Technology Open Beta, we need to bring some things back to a proper baseline.

Open Betas:

I expect that any of these will need to go through an Open Beta process with the intention of reducing risk, gathering feedback, and helping refine things.

Multiplayer Stability and Out of Syncs

We are aware that there are stability issues with multiplayer, and need your help to fix them. 

Reproducing desyncs in multiplayer is much harder than finding regular bugs and fixing them, as there are many different ways and causes the game can desync. Often one desync “hides” under another, so even if the patch notes say we’ve fixed one, it can often still remain an issue in a different form. Saves that reliably desync 100% of the time are pure gold from our perspective.

How does Stellaris multiplayer actually work?

Stellaris uses lock-step multiplayer. Essentially, a day is split up into 10 “turns”, and for each turn the client sends the instructions (commands) the player executed on their computer to the host, and a list of the relevant values for each command (such as country_resources, ship_count, etc.) to the host. The host then verifies the validity and executes that player’s instructions on the galaxy, and generates its own list of values and then compares the two lists. If everything goes as intended, and all the instructions are passed to the host, the values will be the same and the game will then continue to the next tick. If there is a mismatch between the two lists of values, the game declares a desync and generates an OOS report that is saved locally on the client and hosts computers. 

Why don’t you fix the desyncs?

We do actively fix desyncs with just about every version. Unfortunately, we also sometimes introduce new desyncs that look the same as the old desyncs, or uncover new desyncs that were “hiding” under the old desyncs. A desync is identified by a set of names (like RANDOM_COUNT, NUM_POP_GROUPS or ECONOMY_OVERLORD) that tell which part of the gamestate was misaligned: you can think about the desync name as the “symptoms”.

How do you fix desyncs?

When we get a bug report that has the two OOS reports (host and desynced client) our devs will compare the OOS reports and find the differences between them. They will then trace back through the code and find what didn’t synchronize between the host and client, sometimes this is a relatively easy fix, and other times just finding the culprit can take days.

I’ve experienced a desync, how do I report it?

Report your desync on the Bug Report Forums. In your bug report, please include an OOS report from the host and a client that desynced. If you have an autosave from the month before the desync, that will also help since sometimes these autosaves will desync every time – which helps us immensely when trying to reproduce desyncs on our end, as well as helps testing to be sure the desync is actually fixed.

Next Week

Next week we’ll likely be going over either the preliminary release notes or the actual release notes for 4.1.5, depending on how the timing and patching works out. 

If there are any major breakthroughs regarding anything in this dev diary we’ll also share more about them, but otherwise we’ll be going “headphones on” as we focus on these issues. I’ll likely elaborate more on some of the questions I expect this dev diary will create.

See you then, and thank you for playing Stellaris!