Don’t code today what you can’t debug tomorrow.

Moving To The Country, Gonna Eat A Lot Of Lemons

Posted: April 17th, 2017 | Author: | Filed under: Hermit, Projects, Soapbox | 3 Comments »

What is the most resilient parasite? Bacteria? A virus? An intestinal worm? An idea. Resilient… highly contagious. Once an idea has taken hold of the brain it’s almost impossible to eradicate. An idea that is fully formed – fully understood – that sticks; right in there somewhere.

– Cobb (Inception, 2010)

That idea for me was to drop everything, move out into the middle of nowhere, and make games out of nothing. It’s not a new idea. It’s been stewing around in there for a long long time. When Gameloft Auckland closed its doors, that provided the impetus to move it along. I spent a good year tossing it over, discussing it with peers and (former) colleagues. The general sentiment was, “If I could, I would.” The catch was, of course, that we are never really truly prepared. Sometimes you have just got to take the plunge. So I jumped. What’s the worst that could happen?

A few were tempted to jump with me. There were quite a few late night discussions on where to go, what to do, how to set up a community of indies so that we can feed off each other’s creative juices. We even had a big road trip down to Taranaki just to scope out the place and see how comfortable we would be there. Reality has a way of crashing down on dreams, and that happened for many of my compatriots. They are still keen to come down in a couple of years when their earthly affairs are sorted, but for now, I am the pioneer.

So… the idea of moving out here is of course for the cheap rent, fresh air and uninterrupted coding time. I scoped out the cheapest places I could find, and spent a few days down in Hawera to look at places. Some were downright nasty, while the good ones were snatched from under my nose mere hours before I was due to inspect them. The very last place that I was scheduled to look at, however, wasn’t too shabby. It was a spacious three-room house with a reasonably large garden and yard. It was old, and needed some cleaning up, but certainly serviceable. The other catch was that it’s out in the farmlands 20 minutes drive from the city. It also came with a very bountiful lemon tree. That was the sign. Life literally gave me lemons. I took the deal—$150 a week… eat your heart out Auckland!

The move itself went fairly smoothly, but there was a lot to take care. It appears rural folk take great pride in their ruralness (rurality?). I had to sign up at the post office for Rural delivery. Internet is not through fibre or even copper lines, but via Rural Broadband (which is really just glorified 4G with a dedicated cell tower). If it tickled your fancy, you could even fill your car up with Rural Petrol. Then there was the war on the creepy crawlies. With my trusty spray can and dustbuster, I expanded my territory from room to room, eventually claiming the entire house. There are still guerilla insurgents every now and then, but nothing that can’t be shocked and awed. The garden is still wild territory though, but I have a good excuse…

In the two weeks that I’ve been here so far, not one, but two cyclones (or as they like to call it in the news, cyclone remnants) have rolled lengthwise through New Zealand, causing flooding and closing highways across the country. My choice in lodging was validated, given that the house was in no danger of flooding, and I didn’t need to get evacuated. How traumatic that would have been! So while it wasn’t the most auspicious of signs, there was plenty of silver fresnel.

One other thing that killed a lot of momentum in my personal efforts, was some contract work I’ve taken up. Before I left Auckland, I made arrangements with a few choice game companies to take up contract work from time to time, with the stipulation that the work either be lightweight (~2 days a week) or short (~2 weeks). As it turned out, one of them wanted me to start on something right away. So for the first two weeks I’ve been here, that has been the main focus. Looking on the brighter side of things, at least it gives me spending money to cover the cost of the move and settling in.

Nevertheless, having committed to my course, I’ve started a dev blog to track my progress which you can access by clicking here. Unlike this one, that will be focused on the project itself, with daily updates and weekly thoughts, which is why I have not been talking much about it here. Right now, it’s mostly full of excuses for my lack of progress, but hopefully things will kick into high gear once I’m settled in.

The Resource Management System

Posted: September 22nd, 2013 | Author: | Filed under: Hermit, Tutorials | 1 Comment »

Typically, resource systems are created as an afterthought. When the need comes to load or save some piece of data, a bunch of file io code is written and left to languish until it gets so convoluted that some unfortunate soul is tasked to go clean it up. This is sad because as one of the main backbones of the engine, any design misdemeanours will have a tremendous ripple effect throughout the whole code base. After having put considerable thought into it, I present the design of the resource management system for the newly christened Hermit Engine. (Hermit, because it is designed by lone wolf coders for lone wolf coders ;))

Anatomy of a Resource System

Before we dive in, let’s take a look at what the resource system needs to achieve. Much more goes into it than simply loading and saving files. The resource system will be responsible for loading up all assets that are needed by other systems (shaders, textures, scripts, audio, geometry, etc.). This could come from a variety of sources, be it a simple folder structure, compressed archives, a database, a network stream or any other data source. It needs to ensure that all resources are loaded only once, and are reused when possible so as to keep the memory footprint as small as possible.  Ideally, it would be able to load resources at the beginning of levels so that no file io or allocations would happen during the main gameplay phase as these may cause stutters in frame rate. It needs to identify the different types of resources so that they can be funnelled off to the appropriate subsystems for usage. Also, when requested resources are not found, it should be able to substitute in default resources instead of simply crashing.

For Hermit Engine’s resource system, we will lean on 3 main pillars: The basic resource type, the resource pack, and the resource pack loader.

Resource Types

The Achilles heel for most resource systems are the humans that use them. Because of humans, we need human readable resource names. We need unwieldy folder structures so that the fleshy brained can find their stuff. As a result, we end up with elaborate string libraries, hash tables, directory traversal, and other icky constructs to deal with their archaic naming conventions. Remember the ripple effect we mentioned earlier? To all this, Hermit says, “Screw humans!”

Resource ID structure

The Hermit’s resource comprises of 3 pieces of data. The first is the resource identifier. Naturally, this won’t be a string. Instead, it will be a 32 or 64 bit integer (depending on how many resources you have). The highest few bits of will represent the resource type. The next few will indicate which resource pack it came from. This will eliminate the need to store some sort of global identifier counter. The few after that will be a subtype that each system can define. For example, the shader system might want to differentiate between vertex, pixel and geometry shaders. After that, you simply have a running number. If this number happens to be zero, that would be your default resource. Default resources will always be available even in the absence of data because they are procedurally generated (remember the whole point of this engine?).

This will let you easily enumerate all the resource types with a simple enum. Need default geometry? Just request for RESOURCETYPE_GEOMETRY. Need to know what resource type a given resource type is? Just & it with a high-bit mask. Want to sort resources by resource type for more efficient loading? Just use a simple int-compare on the resource id. Everything becomes simpler once the foundation is laid.

The second piece of data that the resource would need is the source information needed to generate the resource. For shaders, this would be the uncompiled script. For geometry, this would be spline and fractal fields. The last piece of the puzzle would naturally be the compiled data. A shader id in the case of shaders or a VBO id in the case of geometry. The source data is what gets saved to disk. During the game’s execution, the source data is read in when a resource is loaded, dished off to whatever system in order to get the compiled data, and then jettisoned since it is no longer useful. If it was still useful for any reason, it would be part of the compiled data. In the case of an editor (you were planning on having one, right?), the source information is kept in memory so that it may be displayed, edited and used to regenerate the compiled data.

“But I’m a human, how will I ever find my stuff amongst all this binary gibberish?!”

The engine won’t sully itself with this since coddling humans is not its problem. However, the editor will maintain a two-way human_name<–>resourceid map so that when information is presented to the human, they get to see their precious strings. After all, it’s the editor’s job is to act as an interface between human and machine.

Resource Packs

Resource packs are what they sound like. They are responsible for grouping related assets together and treating them as a batch as far as loading and unloading goes. They provide the structure which forces the human to think closely about what’s in memory at any one time. For a simple level-based game, you would typically have a pack for all the common elements, and a pack for each level. As you move from level to level, you unload old packs and load new ones. This guarantees that you won’t be doing any file io or memory allocations (with regard to resource handling anyway) that may cause stutters during the main gameplay. Furthermore, it accomplishes this without reference counting or any other clunky mechanism.

“But wait! I’m making a super huge MMO WoW-killer that needs to stream in assets on the fly. I can’t do things level-based!”

First off, if you are a lone-wolf programmer making a “super huge MMO”, it would be my duty to inform you that you are bat-shit crazy and should turn back while you still can. That said, if you still have your mind set on it, all you need are smaller packs. Your packs would contain all resources related to that particular entity: it’s geometry, script, audio, etc. This would handle the problem of making sure all assets are present before trying to display the object. Shared resources can be stored in zone-wide or global packs.

Regardless of how it’s done, the smart designer will keep most if not all dependencies within a pack so that they get loaded/unloaded together. Once everything is viewed modularly in terms of packs, inter-dependency management should become pretty easy. Furthermore, since we are lone-wolf programmers, we only need the modicum of support for stupid design—assert(IsDesignerBeingStupid()).

Pack Loaders

This is the part that does the file io. Its interface would only have three main methods: load a pack, unload a pack, and save a pack. You can have different loaders to cater to different needs. Here are a few examples:

A lot of people seem to like keeping all their assets in version control for some reason or other. You can have the loader write out to a directory structure which can then be checked in. If you really want to, you can also make use of the editor’s mapping to write them out into human filenames rather than just plain ids, complete with convoluted directory structure.

One of the better places to store data for editing would be in a database. This could be as simplistic SQLite or as elaborate as Oracle. This is handy for synchronization of assets between multiple machines, DLC management, statistics gathering, and those shady mass-edit operations that no self-respecting developer would ever admit to doing. The danger here is that a corrupt database could ruin your day.

For runtime, most resource packs would end up in compressed archives. The pack nature, makes this extremely easy. Instead of having to compress each resource individually, you simply compress the whole pack. The header would simply comprise of resourceid<–>offset_location pairs. When loading, you load the whole pack, decompress the whole thing, distribute offset pointers to each of the resource consumers to generate compiled data. After that, just free all the source data at once.

If you really want to, you could even have loader that reads off a network stream for dynamic runtime sharing. However, being the HERMIT engine, we network and share with nobody.

Here we go again…

Posted: July 14th, 2013 | Author: | Filed under: Hermit, Projects, Tutorials | 1 Comment »

Once upon a time, there was a project called Darwena. This was to be a game engine built from the ground up with the aim of reducing our reliance on that other group of people that are kinda crucial to game development. (the artists!) That project started up fairly quickly, with cross-platform input and rendering implemented, even a little text. However, that project died just as quickly when I joined the ranks of Gameloft. Many factors contributed to that. There was the fear that as a generic employee in a generic evil empire, my work would simply get assimilated into the host. There was the lack of time and energy due to the crazy scheduling of a freshly started-up studio (albeit under the umbrella of said evil empire). Most of all, I start learning a whole lot and discovered (i.e. had my nose pushed into, gently) a lot of concepts that made my previous work look like the inane doodles of a three-year-old. (no offense to three-year-olds, you rock!)

Well, most of that has changed now. The wild schedule has been tamed. The powers that be have given their blessing to my pursuit of non-commercial project work. Most of all, I’ve learned enough that I just need to write something to congeal all those disparate concepts into a contiguous body of knowledge. Also, I now have a conducive work area in my apartment, all ready to get those creative juices flowing!

Design Goals

  • It will be cross-platform across Windows, OSX and Linux, though most work will be done in Linux. (because we can!)
  • It will be multi-threaded, and data-oriented, and component based. (I was told that component models tend to not work well. Time to test that!)
  • It will emphasize procedural generation. However, we’re not targeting the sort of procedural generation that we find in the demoscene which creates a lot of pretty stuff with very small code and assets. Instead, we will be generating assets non-realtime (unless things go exceedingly well), mutating and splicing them with genetic algorithms using human input as the fitness function, in order to produce not-so-abstract-looking visual and audio assets. This is not a new concept, but it sounds like fun and worth exploring.

Target Features

  • There will be an editor for creating content, possibly including game levels.
  • Text input and editing will be highly emphasized as the main means of control. The target users are programmers, so there’s no fancy drag-and-drop-connect-the-dots nonsense  going on. Command prompt ftw!
  • There will be a scripting language for fast prototyping, most likely AngelScript because the C++-like syntax might be easier to copy/paste into actual finalized code. Also, the exposed debugging interface would allow us to do breakpoints, watches, etc. within the scripts through our custom interface.
  • The first asset generation milestone would be creating X different assets with different mutations for the human to select and filter through.
  • The second asset generation milestone would allow splicing of two or more assets into one.
  • The third asset generation milestone would be finer-grained gene manipulation.

Basic Architecture

Despite all the fancy stuff, a game engine is still a game engine and will need the same basic things. Our design will emphasize mulththreading while trying to steer away from excessive mutex use or double-buffering of data. Instead, we apply the data-oriented paradigm and look to transform data from one form to another with each module. On a very high level, it may look something like this:

High Level Model

The Simulation module is basically a huge collection of cascaded step functions. This would can be thought of as a physics process. For example, you use acceleration to step velocity. You then use velocity to step position. However, this could apply to HSV color, audio frequency, emotional state, etc. This is very likely to eventually comprise of many submodules to handle different types of algorithmic relationships. I’ve still to sort this out in my head.

The Steering module takes the output from the simulation in order to generate control inputs for the next simulation step. It can also take additional input from its internal stored state or from external sources like human or network interfaces.

The Rendering module simply takes the output from the simulation and splats it on screen, speakers or whatever other output device so that we humans can perceive what’s going on in there. In a sense, this is just a slightly twisted MVC model.

The whole pipeline will run as if it were in a single thread but each of the modules that execute will be threaded. This will allow us to scale dynamically as more processors become available, and will also let us collapse back down into a single-threaded system for easy debugging. Each of the modules or submodules will load balance by simply dividing the number of tasks by the number of processors. We can do this as long as all the submodules are homogeneous in the sense that each task will take about the same amount of time to execute. The only exception to this is rendering which generally likes to run from the same thread, and is thus run in parallel with steering since they operate off the same immutable data. This may cause bubbles in the pipeline, but that’s another worry for another day.

Where are we now?

Currently, we have rendering up on all three platforms, as well as a modicum of input processing. Project/makefiles are generated through CMake. This includes the main engine library project, as well as a project for the editor that runs on the engine.

The next stop is the resource system. Stay tuned!