Diagnostics

Such things are hard to debug.

Traditional methods such as diagnostic "print statements",
or even stepping through the execution in a debugger, are often a poor fit to this domain. If we attempt to apply the diagnostic tools of a 1970s-era batch-processing environment, then we are going to be caught in a painful mismatch!

For example, I often add a diagnostic object-selection system which allows me to "pick" game entities at runtime and have them dump information about themselves to the screen, perhaps in some kind of "speech bubble" window.

Likewise, we can add an interactive text console that allows query commands (etc.) to be sent into the game at runtime.

(it's not all pretty graphics)

For important "manager" objects I also implement dedicated
"debug pages" which allow them to dump relevant state info (e.g. a summary of all the active entity types from the entityManager) to an interactive onscreen overlay.

It then becomes possible to navigate all of the manager objects at run-time, to see what they are doing, and figure out what is actually happening inside their otherwise "invisible" internal data structures.

By making those visualisations interactive,
we can more easily "probe" a running system
to get a feel for its actual behaviour.

This lets us "drill in" to the core of an observed problem,
and helps us to localise the cause, which is pretty much
the first (and, often, the most significant)
step in any debugging exercise.

Wouldn't it be nice if we had those?

Well, we can! (sometimes).

Most of the games I've written implement such a system,
by simply recording all user-input to a little replay-buffer
(and, from there, to a file) and carefully designing the game code to be perfectly repeatable under equivalent inputs.

You've perhaps wondered why I always do
"gatherInputs" as a single, isolated step?

Answer : It lets me (almost trivially) do "replayInputs" too!

To make a robust replay system, you need to have a very
clear idea of what all of your inputs are, and when/where
they enter the system...

It's remarkably easy to forget about things like:
"time" (e.g. any calls to the live system clock or timers),
"randomness" (the "seeds" must be recorded), and
"async loading" (the arrival of an async load
is an "input", like any other).

Also, threads are a pain.

(Because of their unpredictable scheduling).

...if we can catch all of our inputs in some controlled fashion, and record them for future playback --- and if we build our code so that the evolution of its state is repeatably based only on those inputs (i.e. like a big "state machine") --- then we
can make a good diagnostic replay system.

By simply switching our input routines into a "playback" mode, they can be set to re-inject a stream of previously recorded inputs, and everything will proceed as before...

...except, this time, you can pause and single-step and investigate things with all the debug tools on hand! 

The key to making this work is to clearly separate the "logical" inputs of the replay stream from the live "diagnostic" inputs that you combine them with.

As long as the diagnostics are "passive", you can do whatever you like without damaging the integrity of the replay.

This includes stuff like single-stepping, changing the camera position, toggling wireframe on/off, and so on...

As long as the rendering methods don't perform any logic!

..it can actually be quite tricky to make this work,
but it's often worth it.

It requires discipline, certainly, but much of that discipline (e.g. separating the logic from the rendering, controlling "randomness", taming unpredictable threads)
is general good practice anyway.

Ah, if only...

Implementing rewind isn't really possible in the kind of replay system that I'm advocating here, because the update logic of a game is not directly "reversible" in general...

There is more than one way to arrive at a particular state in
the simulation, so there isn't any automatic way of working backwards to the previous ones without extra state-tracking.

However, explicit state-dumping does allow this, but it's more expensive to implement. A compromise is possible though.

If we dump out occasional state dumps (e.g. every 30 seconds or something), we can jump back to those, and then quickly play forward to the point that we actually care about.

This approach requires the state dumps to be "perfect" and "restorable" though, which means they have to be pretty big.

Other approaches rely on using approximate state dumps
(e.g. just positions), which then allows for easy forward-and-rewind (and, indeed, "random access") playback, but without enough detail to let you actually trace through the code.