Taking a page from the awesome Handmade Hero I decided to pre-process game assets in a format more suitable for game consumption. You can take a look at how Casey approaches it starting from Day 150 of his series.

Rival Fortress is a 3D game, so it uses a variety of assets type: textures, normal maps, fonts, meshes, shaders, etc., and each type benefits from a little preprocessing before being used by the actual game engine.

For this reason I wrote a simple command line tool that reads a JSON manifest that contains a list of the assets needed by the game, along with some metadata needed by the preprocessor, and spits out a PAK file that the game engine consumes.

The manifest looks like this:

[
  {
    "type": "vertexshader",
    "filename": "shaders/example.vert",
  },
  {
    "type": "fragmentshader",
    "filename": "shaders/example.frag"
  },
  {
    "type": "geometryshader",
    "filename": "shaders/example.geom"
  },
  {
    "type": "mesh",
    "filename": "meshes/cube.obj"
  },
  {
    "type": "mesh",
    "filename": "meshes/sphere.obj"
  },
  {
    "filename": "images/test.jpg",
    "type": "texture"
  },
  {
    "filename": "images/test2.tga",
    "type": "normalmap"
  },
]

The PAK file format

The file format I chose is similar to the old Quake PAK: a header, a directory, followed by the asset data.

The file begins with a header that is laid out like so:

  Header
  (4 bytes) signature = 'MPAK'
  (4 bytes, uint) version number
  (4 bytes, uint) asset count

After that is the directory of assets, that is an array of the following elements:

  Directory entry
  (4 bytes, uint) numerical ID
  (4 bytes, enum) asset type
  (8 bytes, uint64) size in bytes
  (8 bytes, uint64) byte offset within file

The numerical ID of the asset is a 32-bit FNV-1 hash of the filename of the asset as specified in the Manifest.json file. The preprocessor also spits out a .generated.h file, that is included in the build and contains an enum with all the assets and their ID. With this approach the string hash is computed offline and in no place in the actual game code are assets referred to by their filename, faster lookups by cutting out string comparisons.

Finally the actual asset data is dependent of the asset type, and contains any metadata related to the asset itself. For example, meshes include the vertex array data as well as uints for the number of verts, normals, texcoords and indices.

Preprocessing shaders

Unfortunately, unlike their DirectX counterparts, OpenGL shaders cannot be precompiled, so all we can do is some basic text manipulation and validation. To do so, the files are squished by removing excess whitespace and comments and compiled with OpenGL to check for syntax errors. The source code is then written into the PAK file.

Preprocessing images

Images are loaded into memory as a raw RGB or RGBA byte stream, flipped on the Y axis and written to the PAK file. In a future iteration of the asset preprocessor I’ll likely compress the images with DXTn as the PAK file is getting kinda bloated with uncompressed images.

Preprocessing meshes

I chose the Wavefront .obj file format for meshes because it is a very simple format. I wrote a simplified parser that can digest only triangulated meshes and has very simple material loading capabilities. I’ll revise it when I’ll get a firmer grip on the rendering requirements for Rival Fortress. Mesh vertex, normal and texture coordinate data is interleaved into a large float array for better cache locality. This will be used as a single Vertex Buffer Object (VBO) by OpenGL.

Preprocessing fonts

I still haven’t tackled font preprocessing, because I’m still unsure as to how I will handle text rendering in the game.