GUNNER

A task runner optimized for performance and flexibility

(c) README

Facts

• Task runner for assets with caching system
• First commit by Hendrik Klindworth in 2014
• Written in JavaScript, runs on node.js
• Used by Forge and Elvenar

Concepts

• config.json - defines basic configuration
• gunnerfile.js - runnable script, defines tasks
• tasks - contain steps, can depend on other tasks
• steps - file processing job, depends on operation
• operation (AKA op) - piece of code to run a step
• cache - system for avoiding running steps
• cleanup - special mode to remove leftovers

Concepts: config.json

• 3 mandatory fields
  • srcDir - source
  • destDir - destination
  • cacheDir - cache data
• can contain any other fields
• accessible as gunner.config
• override with --setConfig
• srcDir will be the working directory

Concepts: gunnerfile.js

• normal javascript code
• executed internally by gunner 
• has prepared gunner environment
• defines tasks with gunner API

Concepts: tasks

• contain steps to process files
• defined with gunner.addTask
• executed sequentially
• order defined by addTask calls
• can depend on other tasks
  • dependencies specified in addTask
  • dependencies will be called first

Concepts: step

• actual file processing job
• defined as a part of task
• runs in parallel with other steps of a task
• object with 3 fields
  • src - glob string or function returning array of input file paths (relative to config.srcDir)
  • dest - destination path (relative to config.destDir)
  • op - operation to invoke on src files

Concepts: op(eration)

• named and versioned reusable piece of code
• invoked by gunner when processing steps
• takes source files and destination
• returns array of output files
• object with 3 fields
  • name - string with a simple op identifier
  • hash - hash string produced from version and given configuration, used for caching
  • exec - actual function with operation code, preferably as asynchronous as possible

Concepts: cache

• used to avoid processing unchanged assets
• one cache JSON file per task
• JSON object contains entries for steps
  • key is produced from op hash, input files and destination
  • value contains modification time and content hash for input and output files
    • modification time is a fast-path to avoid expensive content hash checks
• a step is re-run when any of input or output files are changed/added/missing
• cache file is updated after step processing

Concepts: cleanup

• invoke with the same configuration and --cleanup
• to be executed right after a normal run
• reads cache files for defined steps and gets "known" output files
• deletes any files in destDir that are not "known"
• also deletes cache files in cacheDir for unknown tasks

Best practices?

• one step per file for 1:1 processing
  • simple copy, image resize, etc.
• no writing same file in different steps!
  • messes up cache resulting in step re-runs and potential asset regeneration
• be mindful about op.hash
  • bump version when changing behaviour
  • normally hash step constructor arguments
    • avoid hashing callbacks! (e.g. gunner-flatten)
    • add constants by code to hash (e.g. magic strings/numbers)

How we use it

(in Forge Browser)

• two-stage processing with two gunner configurations:
• browser-asset​
  • collects assets from SVN checkout folders
  • places assets into folders as required by the client
  • simply copies ready-for-browser files, does additional processing for others
    • resize, convert and compress images
    • extract building animation "subframes"
    • convert sounds
• browser-texture
  • collects prepared assets from browser-asset​
  • combines assets into atlases (texture or sounds)
  • generates manifest files

How we use it

(in Forge Browser)

• after running browser-asset and browser-texture:
  • merge the output from both of them
  • run more manifest generation scripts
    • asset/reflib mapping
    • checksum symlinks
    • typed reflib code
  • commit and push the output to the game_foe_browser_resources repo
• probably the messiest part and a huge TODO:
  • could/should be done with gunner in a browser-texture run
  • no need for symlinks anymore
  • mappings are messy
  • lots (?) of assets unused by client
  • etc.

Questions?