Don’t code today what you can’t debug tomorrow.

The Resource Management System

Posted: September 22nd, 2013 | Author: | Filed under: Hermit, Tutorials | 1 Comment »

Typically, resource systems are created as an afterthought. When the need comes to load or save some piece of data, a bunch of file io code is written and left to languish until it gets so convoluted that some unfortunate soul is tasked to go clean it up. This is sad because as one of the main backbones of the engine, any design misdemeanours will have a tremendous ripple effect throughout the whole code base. After having put considerable thought into it, I present the design of the resource management system for the newly christened Hermit Engine. (Hermit, because it is designed by lone wolf coders for lone wolf coders ;))

Anatomy of a Resource System

Before we dive in, let’s take a look at what the resource system needs to achieve. Much more goes into it than simply loading and saving files. The resource system will be responsible for loading up all assets that are needed by other systems (shaders, textures, scripts, audio, geometry, etc.). This could come from a variety of sources, be it a simple folder structure, compressed archives, a database, a network stream or any other data source. It needs to ensure that all resources are loaded only once, and are reused when possible so as to keep the memory footprint as small as possible.  Ideally, it would be able to load resources at the beginning of levels so that no file io or allocations would happen during the main gameplay phase as these may cause stutters in frame rate. It needs to identify the different types of resources so that they can be funnelled off to the appropriate subsystems for usage. Also, when requested resources are not found, it should be able to substitute in default resources instead of simply crashing.

For Hermit Engine’s resource system, we will lean on 3 main pillars: The basic resource type, the resource pack, and the resource pack loader.

Resource Types

The Achilles heel for most resource systems are the humans that use them. Because of humans, we need human readable resource names. We need unwieldy folder structures so that the fleshy brained can find their stuff. As a result, we end up with elaborate string libraries, hash tables, directory traversal, and other icky constructs to deal with their archaic naming conventions. Remember the ripple effect we mentioned earlier? To all this, Hermit says, “Screw humans!”

Resource ID structure

The Hermit’s resource comprises of 3 pieces of data. The first is the resource identifier. Naturally, this won’t be a string. Instead, it will be a 32 or 64 bit integer (depending on how many resources you have). The highest few bits of will represent the resource type. The next few will indicate which resource pack it came from. This will eliminate the need to store some sort of global identifier counter. The few after that will be a subtype that each system can define. For example, the shader system might want to differentiate between vertex, pixel and geometry shaders. After that, you simply have a running number. If this number happens to be zero, that would be your default resource. Default resources will always be available even in the absence of data because they are procedurally generated (remember the whole point of this engine?).

This will let you easily enumerate all the resource types with a simple enum. Need default geometry? Just request for RESOURCETYPE_GEOMETRY. Need to know what resource type a given resource type is? Just & it with a high-bit mask. Want to sort resources by resource type for more efficient loading? Just use a simple int-compare on the resource id. Everything becomes simpler once the foundation is laid.

The second piece of data that the resource would need is the source information needed to generate the resource. For shaders, this would be the uncompiled script. For geometry, this would be spline and fractal fields. The last piece of the puzzle would naturally be the compiled data. A shader id in the case of shaders or a VBO id in the case of geometry. The source data is what gets saved to disk. During the game’s execution, the source data is read in when a resource is loaded, dished off to whatever system in order to get the compiled data, and then jettisoned since it is no longer useful. If it was still useful for any reason, it would be part of the compiled data. In the case of an editor (you were planning on having one, right?), the source information is kept in memory so that it may be displayed, edited and used to regenerate the compiled data.

“But I’m a human, how will I ever find my stuff amongst all this binary gibberish?!”

The engine won’t sully itself with this since coddling humans is not its problem. However, the editor will maintain a two-way human_name<–>resourceid map so that when information is presented to the human, they get to see their precious strings. After all, it’s the editor’s job is to act as an interface between human and machine.

Resource Packs

Resource packs are what they sound like. They are responsible for grouping related assets together and treating them as a batch as far as loading and unloading goes. They provide the structure which forces the human to think closely about what’s in memory at any one time. For a simple level-based game, you would typically have a pack for all the common elements, and a pack for each level. As you move from level to level, you unload old packs and load new ones. This guarantees that you won’t be doing any file io or memory allocations (with regard to resource handling anyway) that may cause stutters during the main gameplay. Furthermore, it accomplishes this without reference counting or any other clunky mechanism.

“But wait! I’m making a super huge MMO WoW-killer that needs to stream in assets on the fly. I can’t do things level-based!”

First off, if you are a lone-wolf programmer making a “super huge MMO”, it would be my duty to inform you that you are bat-shit crazy and should turn back while you still can. That said, if you still have your mind set on it, all you need are smaller packs. Your packs would contain all resources related to that particular entity: it’s geometry, script, audio, etc. This would handle the problem of making sure all assets are present before trying to display the object. Shared resources can be stored in zone-wide or global packs.

Regardless of how it’s done, the smart designer will keep most if not all dependencies within a pack so that they get loaded/unloaded together. Once everything is viewed modularly in terms of packs, inter-dependency management should become pretty easy. Furthermore, since we are lone-wolf programmers, we only need the modicum of support for stupid design—assert(IsDesignerBeingStupid()).

Pack Loaders

This is the part that does the file io. Its interface would only have three main methods: load a pack, unload a pack, and save a pack. You can have different loaders to cater to different needs. Here are a few examples:

A lot of people seem to like keeping all their assets in version control for some reason or other. You can have the loader write out to a directory structure which can then be checked in. If you really want to, you can also make use of the editor’s mapping to write them out into human filenames rather than just plain ids, complete with convoluted directory structure.

One of the better places to store data for editing would be in a database. This could be as simplistic SQLite or as elaborate as Oracle. This is handy for synchronization of assets between multiple machines, DLC management, statistics gathering, and those shady mass-edit operations that no self-respecting developer would ever admit to doing. The danger here is that a corrupt database could ruin your day.

For runtime, most resource packs would end up in compressed archives. The pack nature, makes this extremely easy. Instead of having to compress each resource individually, you simply compress the whole pack. The header would simply comprise of resourceid<–>offset_location pairs. When loading, you load the whole pack, decompress the whole thing, distribute offset pointers to each of the resource consumers to generate compiled data. After that, just free all the source data at once.

If you really want to, you could even have loader that reads off a network stream for dynamic runtime sharing. However, being the HERMIT engine, we network and share with nobody.


Here we go again…

Posted: July 14th, 2013 | Author: | Filed under: Hermit, Projects, Tutorials | 1 Comment »

Once upon a time, there was a project called Darwena. This was to be a game engine built from the ground up with the aim of reducing our reliance on that other group of people that are kinda crucial to game development. (the artists!) That project started up fairly quickly, with cross-platform input and rendering implemented, even a little text. However, that project died just as quickly when I joined the ranks of Gameloft. Many factors contributed to that. There was the fear that as a generic employee in a generic evil empire, my work would simply get assimilated into the host. There was the lack of time and energy due to the crazy scheduling of a freshly started-up studio (albeit under the umbrella of said evil empire). Most of all, I start learning a whole lot and discovered (i.e. had my nose pushed into, gently) a lot of concepts that made my previous work look like the inane doodles of a three-year-old. (no offense to three-year-olds, you rock!)

Well, most of that has changed now. The wild schedule has been tamed. The powers that be have given their blessing to my pursuit of non-commercial project work. Most of all, I’ve learned enough that I just need to write something to congeal all those disparate concepts into a contiguous body of knowledge. Also, I now have a conducive work area in my apartment, all ready to get those creative juices flowing!

Design Goals

  • It will be cross-platform across Windows, OSX and Linux, though most work will be done in Linux. (because we can!)
  • It will be multi-threaded, and data-oriented, and component based. (I was told that component models tend to not work well. Time to test that!)
  • It will emphasize procedural generation. However, we’re not targeting the sort of procedural generation that we find in the demoscene which creates a lot of pretty stuff with very small code and assets. Instead, we will be generating assets non-realtime (unless things go exceedingly well), mutating and splicing them with genetic algorithms using human input as the fitness function, in order to produce not-so-abstract-looking visual and audio assets. This is not a new concept, but it sounds like fun and worth exploring.

Target Features

  • There will be an editor for creating content, possibly including game levels.
  • Text input and editing will be highly emphasized as the main means of control. The target users are programmers, so there’s no fancy drag-and-drop-connect-the-dots nonsense  going on. Command prompt ftw!
  • There will be a scripting language for fast prototyping, most likely AngelScript because the C++-like syntax might be easier to copy/paste into actual finalized code. Also, the exposed debugging interface would allow us to do breakpoints, watches, etc. within the scripts through our custom interface.
  • The first asset generation milestone would be creating X different assets with different mutations for the human to select and filter through.
  • The second asset generation milestone would allow splicing of two or more assets into one.
  • The third asset generation milestone would be finer-grained gene manipulation.

Basic Architecture

Despite all the fancy stuff, a game engine is still a game engine and will need the same basic things. Our design will emphasize mulththreading while trying to steer away from excessive mutex use or double-buffering of data. Instead, we apply the data-oriented paradigm and look to transform data from one form to another with each module. On a very high level, it may look something like this:

High Level Model

The Simulation module is basically a huge collection of cascaded step functions. This would can be thought of as a physics process. For example, you use acceleration to step velocity. You then use velocity to step position. However, this could apply to HSV color, audio frequency, emotional state, etc. This is very likely to eventually comprise of many submodules to handle different types of algorithmic relationships. I’ve still to sort this out in my head.

The Steering module takes the output from the simulation in order to generate control inputs for the next simulation step. It can also take additional input from its internal stored state or from external sources like human or network interfaces.

The Rendering module simply takes the output from the simulation and splats it on screen, speakers or whatever other output device so that we humans can perceive what’s going on in there. In a sense, this is just a slightly twisted MVC model.

The whole pipeline will run as if it were in a single thread but each of the modules that execute will be threaded. This will allow us to scale dynamically as more processors become available, and will also let us collapse back down into a single-threaded system for easy debugging. Each of the modules or submodules will load balance by simply dividing the number of tasks by the number of processors. We can do this as long as all the submodules are homogeneous in the sense that each task will take about the same amount of time to execute. The only exception to this is rendering which generally likes to run from the same thread, and is thus run in parallel with steering since they operate off the same immutable data. This may cause bubbles in the pipeline, but that’s another worry for another day.

Where are we now?

Currently, we have rendering up on all three platforms, as well as a modicum of input processing. Project/makefiles are generated through CMake. This includes the main engine library project, as well as a project for the editor that runs on the engine.

The next stop is the resource system. Stay tuned!


C vs C++ for Game Engine Code

Posted: May 12th, 2013 | Author: | Filed under: Soapbox, Tutorials | 2 Comments »

Until a year or so ago, I was a C++ purist. This was fair, because as a game programmer, generally you would want your code in C++ 99% of the time. The object-oriented style it promotes helps a lot in scalability and code management. Any super-time-critical code would mostly be handled by the engine anyway. Also, as a superset of C, there’s technically no reason why it can’t do everything that C can do. However, coming across game engine code written in C, most notably in HGE, Orx and RK, was an eye opener. There is a poetic simplicity and succinctness to the code. Just reading the code alone makes you instinctively feel that it should be faster, even if just by a smidgen. Much of this past year was spent ping-ponging between the two concepts, trying to figure out which one I like better.

Why C++ instead of C

Readbility

Namespaces are wonderful to have. Ultimately all you want to do is prevent naming collisions. C gets around this by generally having absurdly long function names like  BlahBlahEngine_GetIrritatingObject() which ultimately makes the code harder to read. In C++ GetIrritatingObject() is simple and to the point.

Operator overrides are also very nice when used sensibly. Case in point: which is easier to read?

//C Code
Vector a, b, c, d;
Vector_Set(a, 1, 2, 3);
Vector_Set(b, 3, 4, 5);
Vector_Set(c, Vector_Add(a, b));
Vector_Set(d, Vector_Mul(c, 3));

// C++ Code
Vector a(1, 2, 3);
Vector b(3, 4, 5);
Vector d = (a + b) * 3;

This readability can also be a detriment if badly used, because the programmer is not made aware of the cost of each operation. This, however, can be circumvented by simply having more alert programmers! 😀

RAII

Resource acquisition is initialization. This is one of the little tricks used by smart pointer classes to ensure that memory is cleaned up after them. In general, I don’t like smart pointers. Sure, they are a convenient way to manage memory, but abuse of them tends to lead to bad architecture (more on this later). However, I do like to write almost all my classes using the RAII paradigm. While it has the side-benefit of making your code exception-safe, it also encourages what I consider to be good architecture—in that everything is allocated at once (and therefore most likely in the same area of memory), and no strange allocation happens through different code paths.

C generally uses Init and Deinit functions. These are not exception-safe (i use the term liberally), so if your application gets shut down unexpectedly half-way, it will most likely leak memory. Some OS’s might sandbox you, but not all do.

Templates

Love or hate them, they are still useful for all sorts of things, most notably container classes. Their shortcomings are well documented—hard to debug, inconsistency between compiler implementations, etc. I equate them to C macros, but with more use cases and potential problems. Overall, they are a net gain, but I would still shy away from STL/Boost unless I know their exact implementation. I tend to use these more for my own custom classes, but the moment ugly template magic starts happening, I take pretty much any other alternative I can.

Why C instead of C++

Object vs Data Orientation

We all know object oriented code. After all, it is what was promoted when we were in school, even though at the time, we all secretly wanted to write everything in one big function and be done with it. The structure it enforces produces very scalable and flexible code. It also had the benefit of localizing all data for an object in the same spot of memory, which was and still generally is a good thing.

Data oriented code is the new (relatively speaking) paradigm of envisioning code modules as data manipulation. You take in an input and produce an output. By cleverly organizing your data in a somewhat unobject-like way, you can have the same operation acting on mass amounts of data that happen to be all located sequentially next to each other. As a wise man once summarized, you organize your data as a struct of arrays, rather than an array of structs. The benefit of this is that your data is more cache-optimized, and also lends itself more easily to parallel processing.

Now, it should be noted that both C and C++ are capable of implementing both paradigms, even at the same time (they are not mutually exclusive). However, by its nature, C++ nudges you to think in a more object-oriented way. C doesn’t nudge you at all, and has absolutely no qualms to letting you chop off your own foot if you so desire. It is, however, somewhat easier to head down the data-oriented route without the uneasy feeling that you are doing something naughty. I found the best way to not feel guilty is simply to stop feeling guilty.

The Singleton Problem

This is one of my biggest peeves with object-oriented languages. Let’s say you want to write an audio module, with functions like PlaySound(soundid) that everybody and their grandmother wants to call at some point. OO logic dictates that there should be some kind of SoundManager class, and since there should be only one of them, the Singleton pattern is followed. Thus every time you want to play a sound, you would do something like SoundManager::GetSingletonPtr()->PlaySound(soundid). Depending on your implementation, GetSingletonPtr could contain a branch to check if the singleton exists. Also, managing when your singleton object gets allocated, intiated, etc. is a pain. Of course, there are at least two dozen singleton variants that get around this in various ways. But that’s a lot of reading…

C, on the other hand, is made for modules. By embracing the “evilness” of a globally defined function, one can simply call SoundManager_PlaySound(soundid). Firstly, it’s as simple as just calling a function.  In fact, it is just calling a function! Secondly, your module would have already been inited and deinited at startup and shutdown giving you unequivocal control over the lifespan of the SoundManager.

I get around this by embracing the “evilness” of global objects. All these objects’ lifetimes are managed by a single RAII manager class, but pointers to them are available in the global scope. SOUNDMANAGER->PlaySound(soundid) is not bad, though still not as “clean” as the C way.

Convolution in the Name of Safety

C/C++ was generally regarded as one step up from assembly. When you want finer-grain control over what happens where, without explicitly dictating each opcode, they would be your languages of choice. Lately, C++ has been moving away from that shaky ground. The additional functionality like smart pointers, dynamic casts, templatized algorithms and libraries, are all there in essence to prevent you from screwing yourself over. The cost to that is that you are removed further from implementation details. When I malloc() something, I get a chunk of memory, When I new() something, I get an object, with it’s constructor called, and whatever happens in there. This is not a bad thing, and probably improves the quality of life for C++ programmers in general. The thing is, if you are going in this direction, why not use a language like Java or C# which was designed for this role in the first place?

Instead of writing code to prevent programmers from doing stupid things, why not have smart programmers that won’t do stupid things in the first place? Game programmers, especially engine programmers, are control freaks when it comes to what their code does. Sometimes, we do want to shoot our own foot off (because it’s amusing?), so let us and stop asking questions!

I get around this by simply not using the C++ features I don’t like. Yes, I use new and delete, since I know what they do (as long as they are my own classes). I use sprintf instead of stringstreams which are just clunky. I use arrays instead of vectors. I use the C string manipulation functions or simply roll my own instead of std::string. I use C casts instead because it is briefer, and architect so I won’t be in a position to cast the wrong pointer. That last one is particular poignant. With all the convenience tools taken out, you are forced to look at your architecture, design it sensibly, and generally be more aware of the different interactions. (The same can be said for avoiding VisualAssist, but that’s a rant for another day!)

 


Project Update!

Posted: February 6th, 2013 | Author: | Filed under: Projects | 3 Comments »

Hi everyone! It’s been a long time since I posted. In fact, it’s almost been ever since I joined Gameloft. The corporate hush-hushness has been keeping my lips sealed. However, now that several games have already hit the markets (for some time), now would be a good time to list them out and add them to my online portfolio. If you are looking for amazing revelations and insights, sorry. This is more of a “Look what I did!” post.

First, the ones that didn’t quite make it, and hence I can’t say much about. The first is  an the Android port for the very first game published by the Auckland studio. Unfortunately, the port was later abandoned for reasons we shall not say. I also worked on an experimental project which, for blogging purposes,  we shall call “Mystery Island”. Unfortunately,  Mystery Island did not make it past the prototype stage, but it was very pretty. (We were definitely carried by the amazing art team for that one!) Ah well, rough start!

Playful Minds and Men In Black 3

In the meantime, I did contribute to two games that did make it to the market. Playful Minds: Math, and Men In Black 3. While I wasn’t a main developer for either of these, I contributed via bug-fixing in the former and performance optimization in the latter. Apparently, that was good enough to get my name in the credits, so yay me! Playful Minds is an educational game that was inherited by our studio. A lot of sweat and tears went into making it as it was one of our first projects, and nobody was quite sure which way was up yet. MiB3, on the other hand, was developed in another studiol, and they were under tremendous pressure to release in time for the movie premier.

Wonder Zoo, My Little Pony and Littlest Pet Shop

Now for three games that I did play an active role in: Wonder Zoo, My Little Pony and Littlest Pet Shop. Note that they are all freemium games. Yes, the exact kind that you can get off the AppStore for free, but extract money from you later by exploiting your psychology. It’s a diabolical scheme, but a good coder puts himself beyond petty things like good and evil. It’s all just 1’s and 0’s in the end, and the weak-willed will always be susceptible to mind tricks anyway. They all run on both iOS and Android. We do not discriminate! (except against Blackberry, Palm and Windows… oops!)

All of these games were developed on a framework that I helped architect and wrap around Daniel Stephen’s RKEngine. (I can say that because it’s in the credits!) Note that when I say architect, what I really mean is that a few mates and I wrote some of the initial base classes and processes, but because our engine team was tiny in its infancy, everybody (and I mean EVERYBODY) chipped in bits and pieces. A true Frankenstein framework, but it LIVES!

Now would be a good time to give a shout-out to Remi Akilapa, who passed away unexpectedly during the very early formative years of this framework. He was a brilliant coder and engineer, who had a tremendous stabilizing influence before we found our footing. We kept each other sane through countless meetings with designers, artists and producers, sharing the mantle of technical leadership that neither of us really wanted to bear. Thankfully, I don’t  have as many meetings to attend now that the studio has had time to mature. I would much rather be writing code. RIP my friend!

MechWarrior: Tactical Command

Finally, a note on a game that was not made by Gameloft. MechWarrior: Tactical Command was released by Personae Studios in September 2012. I worked with them part-time for a year, departing roughly 2 years before the release date. By that time, we had  established their base engine tech, as well as a couple of (very) rough prototype levels that the team put together during one amazing week of glory. I don’t know how much of our work made it to publication, but I’m glad it did finally hit the store. The sad news is that it is iPad-only, so I won’t get to show it off.