"More than just programming"

AntHive

04-05-2020

NQ Transmission: ANTHIVE

Greetings! NQIsaac this time. I'm going to cover a part of our development workflow that I'm rather proud of: the AntHive build system.

Why build a build system?

AntHive

Any fleshed out game will have a ton of components that go into its development:

These all need to be neatly packaged up for several reasons. First and foremost to distribute the game to other people they will need all the components to run it.

We work off of a development repository that contains all kinds of "extra" stuff -- unbuilt versions of this content that is needed for development, but needs additional processing to be part of the game we actually distribute through a service like Steam.

A second concern that often gets overlooked is that if you want the game to load quickly it needs to be set up efficiently. A well designed loading system can help the game start faster and either have no loading screens or very fast ones. Finally, and the reason I actually started on this project, when creating games you don't want to have to sit around waiting for ages for the game to build all the assets in order to test changes.

In reality the only actual requirement is the first one. The second improves the game for users and the third improves the day to day work for the developers. Nevertheless AntHive addresses all three.

Why create a new build system?

Ultimately the first and main concern stated above was already handled before AntHive but it had several issues that we wanted to address. The very first was that sprites were not well packed. 'Texture Atlas Packing' or just atlas packing is the process of combining all the individual sprites and frames of 2D animation into single large images. These atlases can be easily and efficiently loaded into the GPU for rendering. The original version didn't use any rectangle packing so there was a lot of wasted space meaning that we couldn't fit as many individual sprites per atlas. In regards to the second concern mentioned, the way it loaded things from the disk was inefficient as it involved many individual files that were iterated over to load up all the required resources. One of a few issues regarding the third concern was that to add a brand new texture, say a new enemy skin for example, it would require rebuilding the game, running it in a special mode, then rebuilding it again. It worked but it was quite a slow process. This made it frustrating to make large changes with any sort of speed.

The system was spread across c++, bash, batch, cmake, and python so it was very unwieldy to try and refactor. This means that there was a lot of room for improvement to make both our lives as developers as well as the experience for consumers better.

So Fast

How it works now

While I could discuss in depth the process of building it the interesting points are the features and benefits it adds. The first feature that AntHive added was the ability to actually pack textures so we can fit the most number of sprites per atlas. As the system is now done entirely in python this was thankfully trivial. In this first success it was made pretty clear that this system was a lot faster than the original way of building the game. This raised two thoughts: how do I make this as fast as possible and what else can I integrate.

In answer to the first question what it does it automatically iterates over the root directory down discovering 'hive' files that describe what kind of operation should be performed on each folder. Once it's figured out what it needs to do it then checks 'did any changes actually happen?'. This is because the original system would just rebuild everything regardless of whether or not it was actually changed. The new system very cleverly uses a combination of hashes and time stamps to make sure that nothing is done unless it has to. This means it takes approximately a quarter of a second to check everything instead of 5+ minutes.

The second answer starts with its three stages: pre, build, and post. A pre stage would be something like cleanup of files or ensuring that everything is in the right place. The build stage is the main group of tasks including the packing of meshes, sounds, atlases, shaders, and textures. The final stage actually runs the compiler generating out the executable. The stages are run quickly by parallelizing the tasks. At this point everything from the old system has been completely integrated and improved while integrating into the new system. This has substantially improved the speed that we can work on projects built with NQTech.

As a further improvement to the original system all the resources are now packed into a single resource file and compressed instead of spread over many individual files. This significantly improves the loading speed of the games as well as making the downloads smaller.

I hope this post has been able to shed some light on the largely unseen parts of game development that have surprisingly wide reaching effects. The ability to build custom solutions like this cleanly into our environment and engine is one of the things that NerveQuake prides itself on.


Posts

05-23-2020 -- VECTORLORD 1.1 Economy Update
05-01-2020 -- Why All Coordinate Systems Suck
04-22-2020 -- VECTORLORD OST
04-05-2020 -- AntHive
03-10-2020 -- VECTORLORD Visual Style and Supporting Photosensitive Players
03-08-2020 -- NerveQuake Introductions
03-06-2020 -- Transmission
NerveQuake Software Inc.