A Thousand Cuts Down to 5

How to Keep Unity’s Drawcalls and Other Bottlenecks in Check

A tale of lessons learned developing for a platform that literally caught fire

by

Cratesmith

(aka Kieran Lord)

Disclaimer

As usual I may swear and rant.

 

This was really hard to make into a cohesive talk. 

 

The name of the talk was going to have this really cool point in 5 steps...

 

However in re-writing this talk I found a better way to do it.

 

So lets fix up the title

Optimization Takes a whole team

How to Actually keep performance in check.

by

Cratesmith

(aka Kieran Lord who sucks at naming talks)

About me

I make games!

  • PS2/XB: Destroy All Humans 2
  • 360/PS3: The Dark Knight (Unreleased)
  • Wii: Next big thing (Unreleased)
  • Web: Alternator
  • PC/PS3: Vessel
  • iOS/Droid: Sim Cell
  • iOS/Droid: Codebreakers
  • PC/Mac: Kinect & VR support for Tail Drift
  • GearVR: Small  (unreleased)
  • Cardboard: RACQ Bike VR
  • GearVR: Zombie Nom Nom

 

Plus a bunch of middleware, consulting, professional education and R&D.

A deep dive into how to optimize for a particular kind of hardware or game.

 

Hard limits or rules on "don't use this feature" or "don't use more than X drawcalls"

What this talk isn't

A case study of a project that was a right nightmare to optimize all throughout development

 

The focus is going to be on looking at processes and guidelines that we used to finish it.

 

You're in a processes and workflow talk.

What this talk is

Overall philosophy

"Always shippable"

Everything put into a game should be in a state & quality where it could go into the final game.

 

Basic performance is a big part of that quality.

 

If performance is bad, work will need to be redone, crunched or cut.

 

(Obviously excluding prototyping & first passes)

Fig1. Solution not shippable

"Baseline optimization'

You need baseline performance in order to make the game.

 

That means the game must run at target frame rate during development.

 

"Early optimization is the root of  all evil"

  • I don't like this phrase, because people take it as gospel.
     
  • Deep optimization IS a rabbit hole that can stall development.
     
  • ...But if the game is running under target frame rate, it's a bug!
     
  • Worse, trying to regain acceptable performance late in development can be impossible!

Fig2. How poor performance affects designers & artists.

"Everyone is independent"

Performance is the entire team's responsibility.

 

Everyone in the team needs the tool & knowledge to manage performance for anything they're making independently of others. 

 

You need to make tools and teach other members of the team how to use them.

 

If performance problems, common errors and or busywork can be automated DO IT.

Fig3. A tool that saved hundreds of hours and took 5 minutes to make.

Case Study

GAme & PLATFORM

Programming guidelines

REndering Pipeline

Modelling/Texturing pipeline

Level design/Environment art pipeline

Build Pipeline

GAme: Zombie nom nom

First person walking game, always moving forward.

 

You're a zombie and you run into people to eat their brains.

 

The round ends if you go too long without eating any brains.

 

You look really dumb while playing it.

 

Small but visually complex environments (for GearVR)

Platform: The GEAR VR

Very rough S6 guidelines

  • 75k static polys
  • 8k dynamic polys
  • 25 draw calls per eye
  • Required 60fps
  • No Instancing
  • Not all support single pass stereo rendering
  • Note7 model can explode

 

Weak GPU but much more easily CPU bound (mostly rendering)

Case Study

GAme & PLATFORM

Programming guidelines

REndering Pipeline

Modelling/Texturing pipeline

Level design/Environment art pipeline

Build Pipeline

Goals

The goal was to write code that was quick to make/edit and had a good baseline performance. 

 

Always profile before/after each change.

 

Refactor or optimize areas that are running slow.

 

Focus on shortcuts that avoid major issues.

THe Profiler

Everyone on the project should be able to use the profiler.

 

But as a programmer, it should be used to test everything before it's put in the game.

 

It also shows when/where heap allocations are occuring in the callstack.

 

Never just rely on your gut feel that "it's faster". Always compare before/after in the profiler.

Tools and automation

Other roles need the tools to make content for the project as efficiently as possible.

 

If someone in a non-technical is spending time every iteration doing technical work... at least consider that for automation.

 

Don't just use the unity default features, turn it into an editor built around your team & game.

 

The most powerful feature unity has is how flexible it's editor scripting is.

Benchmarking & Limits

Everyone on the team needs to know at least rough limits of what they're working on (how many characters / sounds / fx / polys etc).

 

Finding those out/keeping them up to date is just as important as writing features.

 

Always ensure there's a small amount of breathing room in these numbers so that people can break them a little if needed.

Programming: problem areas

  • GC/Heap
  • Physics
  • Update()

GC/HEAP #1

Heap allocations can be VERY slow.

 

A programmer specific performance problem.

PCs aren't nearly as badly effected by this as as consoles & mobile.

 

If unchecked this can make porting a nightmare.

GC/HEAP #2

Garbage collection happens when you allocate something and the heap is full/fragemented, too much time has passed or if you call GC.Collect()

 

There's significant CPU spike as the garbage collector cleans up deleted objects on the heap.

 

You can manage the severity of the next GC spike by calling GC.Collect when CPU usage is low.


List<Objective> m_tempObjectives 
    = new List<Objective>();

public override void OnUpdate()
{
    base.OnUpdate();        
    m_tempObjectives.AddRange(m_objectives);
    foreach (var objective in m_tempObjectives)
    {
        if (objective != null)
        {
            objective.Update();                
        }
    }
    m_tempObjectives.Clear();
}

GC/HEAP #3

Go read the official Unity tutorial "Optimizing garbage collection in Unity Games" ... it's detailed! VERY detailed!

 

In Brief:

  • Avoid using functions that return new objects (such as arrays).
     
  • Understand the rules around boxing.
     
  • Avoid Linq (unless you're checking IL it's generating)
     
  • Pool & reuse allocations where possible.

Physics complexity

The cost of physics in CPU is heavily affected by on complexity and number of colliders near one another.

 

This can be cut down a lot.

 

Use layermasks extensively, especially for triggers & raycasts.

 

Also try to keep your physics environment "clean"

  • Use primitive colliders instead of meshes
  • Avoid mesh colliders for non-static objects
  • Use convex mesh colliders when possible

Update()

If you don't need to update something every frame... don't!

 

Break expensive logic out from required updates, in the update make it animate correctly and blend any changes.

 

Use coroutines, jobs, timers or managers to update that independently of the framerate/physics timestep.

Lessons Learned

Profiling everything before/after each change revealed that our "gut feel" couldn't be trusted at all. 

 

Getting everyone on board with GC alloc rules was a great time investment. 

 

Regularly updated benchmarks & tools made for other team saved both their time and programmer time (less support needed!)

Case Study

Programming guidelines

REndering Pipeline

Modelling/Texturing pipeline

Level design/Environment art pipeline

Build Pipeline

GAme & PLATFORM

This section

Before explaining the process we need to briefly discuss the rendering pipeline.

 

To just make the core gameplay we were pushing the GearVR's rendering limits.

 

The requirements of the render pipeline dictated the art style, asset pipelines and the team's workflows. 

Batching / Draw Calls

A drawcall is basically a SetPass call followed by one or more vertex buffers sent to the GPU.

 

SetPass is called between batches of objects to set textures & render parameters.

 

SetPass can be very slow on CPU

 

If the Render thread gets behind, the GPU can be left idle.

 

Batching is where unity tries to group objects together to minimize the number of SetPass calls.

 

Consoles, Standalone VR and mobile are much more affected by drawcalls than PC.

Batching vs Render API

Which render APIs your platform supports has a big impact on how expensive drawcalls are.

 

Unity is faster at submitting draw calls with the new APIs such as vulkan/metal/DX12 .

 

Additionally on some platforms they allow drawcall submission to be multithreaded heavily.

 

If you are only releasing for platforms that support this you'll generally be able to run many more drawcalls without becoming CPU bound.

 

Hardware support for these isn't universal however... and vulkan is presently unsupported on GearVR

Batching: STATIC

The first kind of batching unity supports is "Static Batching".

 

Works as "baked" batching for static geometry (though can be done at runtime).

 

Has significant memory overhead.

 

Reliably saves CPU time.

 

It performs significantly better with large "chunks" of geometry rather than hundreds of small objects.

Batching: Dynamic

Dynamic batching merges groups of meshes each frame CPU for rendering.

 

Limited to meshes of  ~300 verts or less, depending on your shader.

 

Only saves CPU on platforms where SetPass is very slow.

 

Test with/without dynamic batching.

 

Most games run faster with dynamic batching disabled!

Batching: Instancing

Draws a single mesh multiple times with different render parameters.

 

Very good performance compared to other batching methods.

 

This feature would have been perfect for Zombie...

 

...but it wasn't supported for GearVR at the time

Batching: For zombie

Zombie was unfortunately the worst case scenario.

  • Crowds of characters
  • Lots of dynamic objects
  • High drawcall CPU cost 
  • Instancing not supported

 

So everything had to be static / dynamic batched.

 

All assets needed to be authored, processed compiled to make maximum use of both batching types

 

We were constantly alternating between being CPU and GPU bound

Draw Calls: PROFILER

The profiler was the easiest way to track if we were CPU or GPU bound and what was the cause.

 

Gfx.WaitForCommands will indicate when the render thread is waiting for the GPU.

 

Long processing in Camera.Render usually suggested being bound by drawcalls.

 

Profiler does affect performance significantly on device. So we had a "framerate info" overlay built into dev builds. 

Frame Debugger

If we were CPU bound, we'd start by opening up the frame debugger to see what drawcalls could be cut or batched together.

 

It made tracking down when batching wasn't working a lot easier.

 

Additionally it helped us with ordering our rendering to minimize overdraw (the main way we were GPU bound)

 

Unfortunately the FD had issues with VR devices at the time, so we'd mostly use it in editor as a guide.

Lessons Learned

Static batching was always  a saving for CPU.

We had to prevent certain objects static batching to avoid overdraw.

 

We were in the unfortunate position where needed dynamic batching to make the game work.

 

A lot of the problems we faced on this project could have been avoided if the platform supported instancing or newer APIs 

Case Study

Programming guidelines

REndering Pipeline

Modelling/Texturing pipeline

Level design/Environment art pipeline

Build Pipeline

GAme & PLATFORM

Goals

The artists should be able to build models, put them into the game & test them.

 

Any technical busywork should be automated and have a fast iteration time.

 

The more of the time artists are focused on making content for the game (within constraints) the better the game gets.

Profiler again!

Teach artists how to use the profiler to check their work using the profiler!

 

Just the parts of the profiler relevant to their work.

 

How to check polygon budgets, drawcall restrictions etc.

Getting content in game

Everything had a specific, simple asset type.

 

Eg. All NPCs have a CharacterType asset as a central point for other assets that define how they look/sound/behave.

 

Testing characters, objects, effects etc in both empty scenes or levels in the game was supported and easy.

Atlasing

Drawcalls can only contain objects with the same material & params.

 

So if you need to reduce drawcalls, merging textures and materials is the first thing to do.

 

My preference is to generate altas textures, meshes & materials automatically in editor.

 

Unity does provide methods to combine textures for this, but getting a good automatic workflow takes a bit of editor scripting.

 

There's some tools on the Asset Store that do variations on this if you don't want to roll your own

Atlasing: Problems

The difficulty with atlases is keeping them up to date.

 

Editor scripts checked if any inputs to the atlas had changed. The atlas only auto-rebuilds if this has happened.

 

This kept iteration time down as reimporting atlases was quite slow.

Lessons learned

Auto-generating atlases was an easy win, and less work than expected.

 

Teaching artists to use the profiler was really valuable.

 

In future I'm going to make it so artists can export groups of models (eg, all characters) in a single fbx and import scripts break them up & fix transforms.

Case Study

Programming guidelines

REndering Pipeline

Modelling/Texturing pipeline

Level design/Environment art pipeline

Build Pipeline

GAme & PLATFORM

Goals

Everyone on the team had a hand in working on the levels. 

 

Despite the restrictions, the tools and limitations had to be fully understood by everyone.

 

We also wanted to use probuilder (an in-unity level/mesh editor) in the pipeline as the team was already familiar with it.

Environments: Building

Each level is built in grey box using probuilder.

 

Visual  meshes are arranged to cover the probuilder geometry.

 

Design focused geometry makes fast & clean collision and navigation meshes and is easier to tweak.

Environments: Saving

Editor scripts whenever the scene is saved

 

Navmeshes are built from probuilder geometry flagged as "colliders"

 

Occlusion is built for visual meshes (and probuilder objects flagged as "occluders")

 

Warnings are thrown for common mistakes.

Environments: Chunks

Static visual meshes are grouped into chunks by hand.

 

Chunks are parent objects that atlas & merge all visual meshes under them on play/build.

 

Selecting which meshes were in each chunk was a tradeoff between drawcalls (better static batching) and polycount

Environments: LIMITS

From benchmarking we knew rough "safe limits" for drawcalls and polycount. (75k static, 8k dynamic, 25 drawcalls/eye)

 

Everyone was shown how to use the profiler to track drawcalls & polycount.

 

If anyone could reorganize chunks/occlusion to fit more polys, they could add more detail.

 

Quality skyrocketed.

Environments: Markup

Everyone wanted to minimize polys to fit more stuff but we had "wasted polys" on objects that were never seen.

 

So we added "Backface Cull Volumes" to the scenes to strip polygons during the chunking process.

 

Any face inside the volume that matches the direction of the volume gets discarded.

 

We generally saved 20% of polys per level using these.

 

The trimmed polys still wasted space in the lightmap, a tradeoff for it being a simple workflow.

Environments: Culling

Chunks are parented to culling sectors. A parent with a bounding volume

 

A culling sector has a list of the other sectors that are visible when the camera is inside it.

 

Some sectors didn't have their own geometry, they were just "triggers" for visibility sets around doorways.

 

This is separate from unity's occlusion culling which is also used.

Lessons learned

We accidentally made a game out of fitting more detail into levels... and we got pretty good at it.

 

Automatic processes with simple markup volumes are really easy to explain and use.

 

Custom occlusion culling could have been easier to use but fit the needs of the project.

 

Having benchmarked limits that people can work against is essential (even if they're rough)

Case Study

Programming guidelines

REndering Pipeline

Modelling/Texturing pipeline

Level design/Environment art pipeline

Build Pipeline

GAme & PLATFORM

Goals

Wanted to make sure that everyone was testing on final hardware as much as possible

 

Or at the least checking everything in desktop VR.

 

People tend to avoid this as it's slower, espeically with GearVR build times.

 

Iteration time is key!

PRESS PLAY TO TEST

Every scene needs to work when you push play. No exceptions.

 

If any systems needed by the scene controller or any actor are auto-constructed.

BUILD & RUN ANY SCenE

Testing on device is slow enough as is. Being forced to make a full build is too much.

 

We usually just want to test one level, so we needed a way to one click build & run a single scene.

 

I'd maintained a fork of UBS (an opensource unity build manager) on previous projects which was expanded for this.

SKU types

We didn't have enough headsets go around. So we needed to support non-vr builds.

 

Expanded the build system to have a concept of plaforms (Skus)

 

Each SkuType was an editor script that configured the project & plugins to build or run in this mode.

 

Each SkuType also injects a prefab into each built scene based on the platform (eg, to handle different VR camera setups)

public class SkuTypeZombieGameNonVR 
: SkuTypeZombieGame
{
#if UNITY_EDITOR
    public override void EnablePlatform()
    {
        base.EnablePlatform();
        UnityEditor.PlayerSettings
            .virtualRealitySupported = false;        
    }
#endif

    public override bool vrSupported
    {
        get { return false; }
    }
}

Lessons learned

Unity generally has good iteration time during prototyping.

 

Keeping that iteration time up during development is up to you. 

 

In our case level specific builds/non-vr versions helped a lot but each project is different!

QUESTIONS?

Example: Pooled temporary Lists

foreach(var mr in gameObject.GetComponentsInChildren<MeshRenderer>())
{
    // do stuff!
}
using(var mrList = gameObject.GetComponentsInChildrenTempList<MeshRenderer>())
foreach(var mr in mrList)
{
    // do stuff!
}

Fig1. Creates a new array on the heap

Fig2. Example using a pooled temporary list instead

GC/HEAP: Advice

Example: Pooled temporary Lists

public class TempList<T> : IDisposable, IList<T>
{
    private static readonly Queue<TempList<T>> s_lists = new Queue<TempList<T>>();
    public readonly List<T> list = new List<T>();

    private TempList() {}

    // acquire a temporary list
    public static TempList<T> Get()
    {
        lock(s_lists) return s_lists.Count > 0 
                ? s_lists.Dequeue()
                : new TempList<T>();
    }

    // return a list back to the pool
    public void Dispose()
    {
	list.Clear();
        lock(s_lists) s_lists.Enqueue(this);
    }

    public static implicit operator List<T>(TempList<T> from)
    {
	return from != null ? from.list : null;
    }

    // ILIST IMPELMENTATION (call appropriate method on this.list)
}

Fig3. TempList implementation

GC/HEAP: Advice

Example: Pooled temporary Lists

public static TempList<T> GetComponentsInChildrenTempList<T>(this Component @this, 
    bool includeInactive=false)
{
	var tempList = TempList<T>.Get();
	if(@this!=null)
	{
		@this.GetComponentsInChildren<T>(includeInactive, tempList.list);
	}
	return tempList;
}



public static TempList<T> GetComponentsInChildrenTempList<T>(this GameObject @this, 
    bool includeInactive=false)
{
	var tempList = TempList<T>.Get();
	if(@this!=null)
	{
		@this.GetComponentsInChildren<T>(includeInactive, tempList.list);
	}
	return tempList;
}

Fig4. TempList extension methods

GC/HEAP: Advice

Atlasing: output Data

AtlasData stores info on how to map textures to

MeshGroups store all the meshes created for atlased models.

 

Prefabs are created for each of the atlased models.

 

AtlasMesh files created for each atlased model. Dummy assets that are lookup keys for the real models.

Atlasing: Versioning

In order to auto-rebuild atlases if they're needed we need to be able to track if contents have changed.

 

Building the atlas creates a hash asset which stores Unity's hash for each asset.

 

If the list of GUIDs/hashes for the AtlasGroup's inputs don't match the hash asset, the atlas is out of date

 

This worked really well! It's a reliable & general purpose solution for auto-generating assets.

Atlasing: AtlasGroup

Defines what's going to be built into an atlas and how.

 

Takes Model/Material list assets as inputs.

 

Used as a handle for the atlas by the rest of the game.

 

When built it creates a child folder for all the output assets.

Atlasing: Input LISTS

Recursive list assets for models & materials as inputs for atlases.

 

List of assets and a list of child lists.

 

Easy to maintain lists of objects/characters by hand, but tracking what levels use wasn't.

 

Saving a level creates/updates a material list for all static meshes in that level.

Batching LIGHT PROBES

Problem: We used lightprobes instead of dynamic lighting + dynamic batching.  

 

Light probes calculate SH values from the light probes for MeshRenderer's anchor transform.

 

Different SH values = different batches for dynamic batching.

Batching Light Probes

Characters drawcalls:

No anchor
1 character = up to 50 drawcalls

Waist bone as anchor

1 character = 1 drawcall.
(Same as using skinned meshes)

 

Multiple characters with shared anchor

up to 5 characters = 1 drawcall

Atlasing: Problems #2

We also had the problem that applying the atlas UVs to meshes at runtime was too slow on the GearVR.

 

So "pre-atlased" versions of models for characters & objects needed to be built with the atlas texture.

Each atlas needed to have a lookup table for a prefab to use instead of the model.

Light Probe Sectors #1

Environments marked up with light sectors, bounding volumes.

 

Objects/Characters look up which light sector they're in and use it's transform as their light probe anchor.

 

Light sectors can overlap! Objects use the sector with the highest priority.

 

Have to be tweaked by hand to make it look right.

Light Probe Sectors #2

Pros:

Solved our batching problems!

 

Also simplified adding light probes to levels. The sectors automatically created light probes for each bounding volume.

 

Cons:

Placement of light sectors to look good became a level design/art problem. More on that soon!

 

 

Still paying for transforming/culling/occlusion on each mesh seperately.

GC/HEAP Boxing

Boxing occurs when a value type (struct, int, bool etc) is passed into a function that expects an interface or object.

 

The value type creates a temporary System.Object that contains the value so that the function can treat it like an object.

 

This is very easy to do without realising it.

 

If a function that shouldn't be allocating is showing up on the profiler with GC Alloc this is the likely culprit.

void SurpriseAlloc()
{
    int cost = 5;
    string displayString 
        = String.Format("Price: {0} gold", cost);
}

public interface IInterface {}
public struct MyStruct : IInterface {}

void ThisIsBadToo()
{
    var val = new MyStruct();
    InterfaceBoxingExample(val) 
}

void InterfaceBoxingExample(IInterface val)
{
    Debug.Log("dang");
}
bool AnotherExample(object thing)
{
    int val = 5;
    return thing!=null && thing.Equals(val);
}

Raycasting: PRoBLEM

Raycasting is incredibly useful, however it's very expensive on CPU (and operates in the main thread!).

 

Caspule/Sweep casts are even worse!

 

It's very easy to cause spikes even on PC if too many raycasts occur in one frame.

A thousand cuts down to 5

By cratesmith

A thousand cuts down to 5

  • 1,381