Progress isn’t made by early risers. It’s made by lazy men trying to find easier ways to do something.

The Resource Management System

Posted: September 22nd, 2013 | Author: | Filed under: Hermit, Tutorials | 1 Comment »

Typically, resource systems are created as an afterthought. When the need comes to load or save some piece of data, a bunch of file io code is written and left to languish until it gets so convoluted that some unfortunate soul is tasked to go clean it up. This is sad because as one of the main backbones of the engine, any design misdemeanours will have a tremendous ripple effect throughout the whole code base. After having put considerable thought into it, I present the design of the resource management system for the newly christened Hermit Engine. (Hermit, because it is designed by lone wolf coders for lone wolf coders ;))

Anatomy of a Resource System

Before we dive in, let’s take a look at what the resource system needs to achieve. Much more goes into it than simply loading and saving files. The resource system will be responsible for loading up all assets that are needed by other systems (shaders, textures, scripts, audio, geometry, etc.). This could come from a variety of sources, be it a simple folder structure, compressed archives, a database, a network stream or any other data source. It needs to ensure that all resources are loaded only once, and are reused when possible so as to keep the memory footprint as small as possible.  Ideally, it would be able to load resources at the beginning of levels so that no file io or allocations would happen during the main gameplay phase as these may cause stutters in frame rate. It needs to identify the different types of resources so that they can be funnelled off to the appropriate subsystems for usage. Also, when requested resources are not found, it should be able to substitute in default resources instead of simply crashing.

For Hermit Engine’s resource system, we will lean on 3 main pillars: The basic resource type, the resource pack, and the resource pack loader.

Resource Types

The Achilles heel for most resource systems are the humans that use them. Because of humans, we need human readable resource names. We need unwieldy folder structures so that the fleshy brained can find their stuff. As a result, we end up with elaborate string libraries, hash tables, directory traversal, and other icky constructs to deal with their archaic naming conventions. Remember the ripple effect we mentioned earlier? To all this, Hermit says, “Screw humans!”

Resource ID structure

The Hermit’s resource comprises of 3 pieces of data. The first is the resource identifier. Naturally, this won’t be a string. Instead, it will be a 32 or 64 bit integer (depending on how many resources you have). The highest few bits of will represent the resource type. The next few will indicate which resource pack it came from. This will eliminate the need to store some sort of global identifier counter. The few after that will be a subtype that each system can define. For example, the shader system might want to differentiate between vertex, pixel and geometry shaders. After that, you simply have a running number. If this number happens to be zero, that would be your default resource. Default resources will always be available even in the absence of data because they are procedurally generated (remember the whole point of this engine?).

This will let you easily enumerate all the resource types with a simple enum. Need default geometry? Just request for RESOURCETYPE_GEOMETRY. Need to know what resource type a given resource type is? Just & it with a high-bit mask. Want to sort resources by resource type for more efficient loading? Just use a simple int-compare on the resource id. Everything becomes simpler once the foundation is laid.

The second piece of data that the resource would need is the source information needed to generate the resource. For shaders, this would be the uncompiled script. For geometry, this would be spline and fractal fields. The last piece of the puzzle would naturally be the compiled data. A shader id in the case of shaders or a VBO id in the case of geometry. The source data is what gets saved to disk. During the game’s execution, the source data is read in when a resource is loaded, dished off to whatever system in order to get the compiled data, and then jettisoned since it is no longer useful. If it was still useful for any reason, it would be part of the compiled data. In the case of an editor (you were planning on having one, right?), the source information is kept in memory so that it may be displayed, edited and used to regenerate the compiled data.

“But I’m a human, how will I ever find my stuff amongst all this binary gibberish?!”

The engine won’t sully itself with this since coddling humans is not its problem. However, the editor will maintain a two-way human_name<–>resourceid map so that when information is presented to the human, they get to see their precious strings. After all, it’s the editor’s job is to act as an interface between human and machine.

Resource Packs

Resource packs are what they sound like. They are responsible for grouping related assets together and treating them as a batch as far as loading and unloading goes. They provide the structure which forces the human to think closely about what’s in memory at any one time. For a simple level-based game, you would typically have a pack for all the common elements, and a pack for each level. As you move from level to level, you unload old packs and load new ones. This guarantees that you won’t be doing any file io or memory allocations (with regard to resource handling anyway) that may cause stutters during the main gameplay. Furthermore, it accomplishes this without reference counting or any other clunky mechanism.

“But wait! I’m making a super huge MMO WoW-killer that needs to stream in assets on the fly. I can’t do things level-based!”

First off, if you are a lone-wolf programmer making a “super huge MMO”, it would be my duty to inform you that you are bat-shit crazy and should turn back while you still can. That said, if you still have your mind set on it, all you need are smaller packs. Your packs would contain all resources related to that particular entity: it’s geometry, script, audio, etc. This would handle the problem of making sure all assets are present before trying to display the object. Shared resources can be stored in zone-wide or global packs.

Regardless of how it’s done, the smart designer will keep most if not all dependencies within a pack so that they get loaded/unloaded together. Once everything is viewed modularly in terms of packs, inter-dependency management should become pretty easy. Furthermore, since we are lone-wolf programmers, we only need the modicum of support for stupid design—assert(IsDesignerBeingStupid()).

Pack Loaders

This is the part that does the file io. Its interface would only have three main methods: load a pack, unload a pack, and save a pack. You can have different loaders to cater to different needs. Here are a few examples:

A lot of people seem to like keeping all their assets in version control for some reason or other. You can have the loader write out to a directory structure which can then be checked in. If you really want to, you can also make use of the editor’s mapping to write them out into human filenames rather than just plain ids, complete with convoluted directory structure.

One of the better places to store data for editing would be in a database. This could be as simplistic SQLite or as elaborate as Oracle. This is handy for synchronization of assets between multiple machines, DLC management, statistics gathering, and those shady mass-edit operations that no self-respecting developer would ever admit to doing. The danger here is that a corrupt database could ruin your day.

For runtime, most resource packs would end up in compressed archives. The pack nature, makes this extremely easy. Instead of having to compress each resource individually, you simply compress the whole pack. The header would simply comprise of resourceid<–>offset_location pairs. When loading, you load the whole pack, decompress the whole thing, distribute offset pointers to each of the resource consumers to generate compiled data. After that, just free all the source data at once.

If you really want to, you could even have loader that reads off a network stream for dynamic runtime sharing. However, being the HERMIT engine, we network and share with nobody.

One Comment on “The Resource Management System”

  1. 1 Alex said at 2:15 pm on September 22nd, 2013:

    Hmm, what if the conventional rules of resources naming were made to prevent excessive pessimization? I mean, the integer thing works if you’re the only one who deals with the source code. But what if you license the engine to somebody else (along with the source code) and they would want to make some changes? Clearly, it’ll take them some time to figure out what is what.
    And one more thing: how do you set relations between loaded resources and objects that use them?

    An interesting idea overall 🙂 And I agree that resources management is often overlooked.

Leave a Reply