Me And Game Engines

I made so many game engines. Some were complex, others were simple, but all were horrible.

These engines always lacked a greater focus and a more scalable model. While I do not care much for abstractions or planning big from the beginning, I do have to say that the engines I created in the past always had a lifetime of one use and then were immediately discarded. The general design of these engines was more "old-school" than anything. By that I mean, the engine itself was embedded into the game. Many of the engine's systems were soldered onto the die (the game in this case). There is a great talk by Bill Clark that goes in-depth about the differences between an engine and a capital A engine. Essentially, from my point of view, an engine is just one that is built with the game itself. The engine is not a separate entity, but rather, it is the game. While, on the other hand, a capital A engine is your typical Unity, Unreal, and CryEngine (you thought I would say Godot, didn't you?). And, for the longest time, I built an engine within a game.

And why wouldn't I? I wanted to make a game, after all. And, perhaps, with the scale of the games I was making at the time, it worked. It did serve me well for a bit. However, as my ambitions grew to create games on a bigger scale, an engine was not sufficient anymore. I could not just load all of the resources that I would ever need in the game at initialization and keep them alive for the entire runtime of the game. My levels were bigger now. They had way more resources. They could not simply live in memory all at once without problems. On top of that, I needed a better and more intuitive way of creating and prototyping levels. I could not just place all of the objects into the world programmatically. I had to find a serialization system of some sort, and one that would handle hundreds, if not, thousands of entities. Suffice it to say, an engine had to leave and a new capital A engine had to take its place. One that would ease the creation process of my games while still being "out of my way" enough not to annoy me. I'll get into more about that later.

And, before truly starting to work on the engine, I had to decide my goals, ambitions, and, most importantly, my intentions with this engine. And, as you would come to know, my goals and ambitions with this engine are, quite frankly, fairly selfish. Well, let me back up a bit.

The Intentions

Previously, whenever I started a new game project, I would start completely from scratch. From zero. I would not carry any line of code from my previous projects. The reason is that I, simply, could not. The systems I "designed" at the time were very dependent on each other. Moreover, they were dependent on specific data structures only created in that game. While I could have perhaps changed some code around or written my systems to be game-agnostic from the beginning, I chose not to. You could call it inexperience since it was. I was constantly learning at the time. I would make a "renderer" only to find a better and more efficient way to make another renderer. I would make an entity system and then scrap it all because I wanted to experiment with another system. On top of that, I liked to switch genres all the time. Yet, if you go to my itch page (which I will not link out of shame), you can see that all my games are, ironically, one genre. Arcade-y games. It is not because I love arcade games so much. I would simply give up halfway through my original idea and decide to make an arcade-style game instead. The "engines" I made at the time were not thought through, as I have said before. They were lackluster at best. A mish-mash of different ideas and implementations that I had suddenly decided to add. But that was only a testament to my lack of experience and lack of patience.

While I could have used something like Godot or Unity, I did not really want to. I will not go into the reasons why I dislike game engines (because I already did) but just know that I dislike the GUI nature and the restricted workflow that all game engines seem to share. Understand, I am not trying to bash on game engines. I think they are fantastic tools for game developers. However, they are certainly not for me.

I believe that not all games are equal. Does every game have common components? Of course. That is quite obvious. However, not every game has the same game creation flow. The flow of prototyping and creating levels for a racing game is not the same as for a real-time strategy game. They share technical components, yes, but not everything is the same. I think that lumping all games in a general sense might hurt experimentation and might hinder the innovation of game design. Many game studios end up creating specialized tools for their games anyway even if they are using an already-existing engine like Unreal. And so, after somewhat of a long-winded rant, my intention with this project is not to create a general-purpose game engine. At the same time, as I said before, making an engine with each game might, in the long run, be inefficient as well. So, what then? Well, let's take a trip down memory lane, shall we? There's no special hat for that, don't worry. Just hop in.

You see, back in the day, developers used to also start from scratch. Unlike me, their reason was more valid. The hardware scene at the time was rapidly changing. CPUs were getting stronger and faster every clock cycle (or what seemed like it, at least). Every new game had to be rewritten since a new and better CPU entered the scene. Now, developers did not remove every piece of code they wrote and start literally from zero. Yet, most of the code was thrown out and replaced with code that adhered to the newer CPU's architecture. Code that could, in some way, bend the CPU to the developer's will by using some exploit or some special feature the CPU had. Not all the code was like that, of course.

But, as the growth and speed of CPUs started to plateau, developers started to keep their old tools and code to be reused. Every project now had a base to comfortably be built upon. Level editors, graphics tools and renderers, and even multiplayer code. Yet, these tools were never in one place. They were always separate programs that had some kind of output that would be fed into the game itself. The game was the main hub these tools would output to. Compare this to game engines of today, where most of the tools are built into the engine while the game is just a result of all of these tools working together. The engine is like the main hub for everything. The overworld, if you will. The engine will import the assets, compress the textures, edit the world, handle all of the gameplay scripting, and do other things like create animations or particle effects.

There is a reason for this--it is, simply, much easier for everyone involved. But, it does come with drawbacks. Mainly, as I said before, it locks you into an ecosystem that you are heavily relying upon. It somewhat limits you to what you can do. That is just not what I wanted to create. I wanted to create an engine to have all the basic needs and the setup to get a game up and running, but I did not want to limit any experimentation or any, perhaps, additional features or tweaks that can be made on the fly. I wanted the engine to be lightweight yet still have a robust set of features. I wanted the game to be the central hub of everything and not the engine. The engine is only there in the passenger seat, helping the game with directions rather than being the driver and ordering the game around.

Now, of course, the game still needed a set of tools to help speed up development. Level editors, for example, would have to be created. The meaning of a "level editor" differs from game to game. For an open-world RPG, that might mean a whole-world editor, while for a Mario-style game, that would mean a top-down 2D tile editor. Do understand that this is not the perfect way of making a game. But, for my use cases, it might as well be. It allows me to reuse tools and code while still giving me the freedom to experiment with different ideas and genres. It does not limit my flow in any way.

My intentions for this game engine are quite selfish now that I think about it. I do not care if other people use it or not. I do not care if it has the best and prettiest GUI editor. I do not want to use things like ECS (Entity Component System) for better entity management. I, frankly, only want to make this game engine for myself. I want to make games at the end of the day. But I want to make games the way I like to make them. More work, yes, but also way more fun.

The purposes for this engine's existence are a) Make it easier for my style of game development, b) Learn a great deal from the technical challenges, and c) A good show off for my portfolio. Of course. Why not?

In order to capture my vision for this engine, however, I have to use the right tools for the job. I did not want to choose a tool that would end up wasting my time and energy while not benefiting my intentions with this engine whatsoever. And so...

The Tools And Dependencies

You see, for the past several years I have used many programming languages and many more game frameworks and libraries. Programming languages like Java, C#, C++, and even, sadly, JavaScript (I know...). Game frameworks like LWJGL, SDL2, Raylib, MonoGame, SFML, and many more. Essentially, I have seen it all. Out of all of them, I think SDL2 was closer to what I was looking for, though, Raylib was the one I used the most at the beginning. And the reason I liked SDL more was because it was more"lower-level" than Raylib or SFML. Additionally, it had that C-style of programming that I have always been fond of. However, despite that, I decided to go against any of these libraries.

The one thing I knew I wanted to do in this engine was to reduce the amount of dependencies as much as possible. I would not dare to go full Handmade Hero-style, but I still wanted to use the minimum amount of required dependencies to get me started. I wanted the engine to be as lightweight as I could make it be. And while SDL does have a multitude of systems that would be beneficial for me, it still included a 2D renderer that I would for sure never use. I wanted to design and implement my own renderer, and having a "ghost" dependency in there just felt wrong to me. Besides that, SDL had its own way of handling textures, fonts, and audio files. While useful, it was very much unnecessary for me. I had in mind to create my own resource binary format which would, essentially, deprecate any need for .png files or the sort. Now, of course, I still need loaders to decode these image and audio files, but I already had a way better and smaller dependency for that in mind.

So what did I use, then? Well, I separated the engine dependencies into five categories. The categories are laid out as such:

1. *Operating System Dependencies*: This is for things like window creation, input handling, the file system, console logging, and any operating system-specific operation 

2. *Graphics*: This is, as the name implies, anything to do with rendering and graphics. So the graphics API (OpenGL in this case), any UI libraries, and so on. 

3. *Audio*: Obviously, it has anything to do with audio. Not _decoding_ or _encoding_, but _playing_ audio by giving the audio card samples and having an audio thread active in the background.

3. *Math*: This might be a stretch but, besides the obvious math library, I also added physics dependencies in this category. 

4. *Resource Loaders*: Image loaders, audio files decoders, 3D model format parsers, and so on.

Out of all of these, the first category--the operating system dependencies--is probably the one I thought about the most. Since SDL was out of the picture, I saw GLFW as a potential choice for handling window creation and input. An obvious choice, by many. And, seeing how I already had used it before, I thought it was obvious to me as well. Yet, there was a feeling I did not need it. After all, I decided from the start that I would only support Windows and Linux. Not for any particular reason other than I use Linux on a daily basis and Windows has the bigger market. I did not have a Mac machine lying around somewhere (poor. I'm poor, basically). And as for consoles, well, that was a long stretch. I did not see the possibility of me ever needing to port my games to consoles. At least not for the time being. And so, that means I only had to deal with the Win32, X11, and Wayland APIs. I say it as though it is an easy affair. It is not. Far from it. Especially if you had never dealt with these APIs before and had to start learning them, which was my case. So, instead, I picked a middle ground. I would use GLFW but I would design my API in such a way that would be easier to switch away from it in the future if needed. I'll write a more in-depth article about the window system and whatnot in the future.

As for the graphics, well, that was a point of contention, too. The only graphics API I truly know by heart is OpenGL. I had used it plenty of times before so it would not be difficult for me to integrate it into the engine. Yet, like the case with GLFW, I was torn. OpenGL is not, well, the most modern of graphics APIs out there at the moment. A better and more "modern" choice would, of course, be something like Vulkan. But Vulkan is a big beast. One that would take quite a long time to deal with. And, frankly, I had the itch to make a game for a long time. I did not want to be held by Vulkan any longer. So, once again, I decided to find a middle ground. I would create a more robust and "open-ended" graphics API that would wrap around OpenGL so that I can, in the future, substitute it if needed. That was the intention at least. I did, briefly, try to integrate DirectX11 at some point, but I ultimately failed. Miserably, I might add. You can still see some remnants of my attempts at implementing DirectX11 in the engine, in fact. But, again, I will go more in-depth about that in a future article.

Audio is another similar issue to graphics. There are plenty of audio APIs out there. Different APIs support different audio cards. I, however, had made up my mind on this one a long time ago. I decided to use MiniAudio, which is the audio library used by Raylib under the hood and one that I used plenty of times before. It is a single-header library that is super easy to integrate.

As for math, that was the easiest choice as of yet. No doubt, GLM is a "gold standard" at this point. For OpenGL it is, at least. But, like with a lot of the other APIs, I decided to build a wrapper around it rather than directly reference the library in the engine's code. And for physics, well, I had not come upon that answer just yet. I did try to make my own physics logic at some point. And while it was, surprisingly, successful, I wanted more than just a simple physics layer. I wanted something more complex and, more importantly, faster than my implementation. I have not decided upon a physics library yet. But I'll cross that bridge when I come to it.

And, finally, resource loaders. I have worked with many resource loaders in the past. That's to say that I know my way around them. For image loaders, I decided to go with the obvious stb_image. It is small, easy to use, and has only a single header. Meaning, like MiniAudio, I can easily integrate it into the engine. Besides that, it supports a wide array of image formats. Even HDR which was surprising. Audio file loaders are next. For this one, I decided to use a few small libraries. Specifically, the Mp3 and WAV loaders from dr_libs and the OGG loader from STB once again (Sean Barret to the rescue!) Each of these are a single-header library as well. Once again, very small and very easy to integrate. As for 3D models, I decided to use a huge dependency named ASSIMP. For me, that was a very hard sell. ASSIMP is huge but it supports plenty of 3D model formats. And besides that, none of the resource loaders are going to be present in the engine itself. But, rather, there is going to be a separate tool--one that I dubbed NBR (Nikola Binary Resource)--that would take only the minimum required data from these loaders and save it into a binary file (.nbr) which then would be read accordingly by the engine.

If you think that's weird, well...

The Design

In my dependencies talk earlier, I mentioned a lot of libraries that were "single-header". I'm sure you noticed I used a lot of them. But, what are they exactly?

In the most simple of terms, a "single-header" library is what it sounds like: There is only one header file (.h or .hpp) in the whole project. These libraries also have their implementation code (.c or .cpp) in the header file itself. Usually, the implementation code is hidden behind a #ifdef. So, in the case of stb_image, you would create a translation unit (.c or .cpp... you should know this by now) which only has the following code:

#define STB_IMAGE_IMPLEMENTATION 
#include <stb_image.h>

The implementation code would effectively be copied into the translation unit and then compiled normally. I love these kind of libraries. They are usually easy to use, very easy to integrate, and a lightweight dependency overall. I even created a library or two in the same vain just for fun. But why am I talking about single-header libraries now? Besides the fact that most of my dependencies are single-header, I wanted to somewhat imitate the spirit of single-header libraries while avoiding the need to jumble all my code into one header file. While it is convenient to have all the code in one place and, once again, it would be very easy to integrate by other folks, but, seeing how I am already an unorganized person, I will refrain from the complexity that comes with such a design. Instead, I wanted to have separate translation units for every module, but keep the idea of a single-header file for the definitions. Let me explain.

This engine really has three parts:

- *Core*: This is the _base_ of the whole engine. This is where the window, input, and event-handling systems live. The graphics API wrapper exists here as well. The logger, the asserts, many core typedefs, and so on. It is more of a "game engine-maker" if you will. 

- *Engine*: This is where mostly all of the code that will be used directly by the user (me) lives. The entities, the scenes, the resource manager, the application callbacks, the camera, the renderer, and so on. 

- *UI*: I have yet to make this part of the engine but it is essentially a wrapper around ImGui to handle the UI for any custom editors and such.

For each part of the engine, there exists a .hpp equivalent. There is a nikola_core.hpp, a nikola_engine.hpp, and a yet-to-be-created, nikola_ui.hpp. The nikola_engine.hpp depends on nikola_core.hpp, and nikola_ui.hpp depends on both.

Now, the great thing about this is that all the definitions of the project live in three places. I do not have to include twenty or so header files in each translation unit when I want to do such a trivial thing as rendering a cube mesh for example. However, there are, as I came to discover, two very important cons to this approach. The first is the obvious length (be mature, damn it) of these header files. For example, the nikola_core.hpp header file alone is 2000 lines of code. And there are still features to be added to it. Yet, with some thorough documentation and some separator comments (one of these bad boys, /// ----------------------), it could be managed pretty well. The main issue and the one that annoys me the most is that if there is any small change in any of the header files it means, potentially, a whole recompile of the engine. Now, the nikola_core.hpp does not have any heavy dependencies that would require a long time to recompile (in fact, this is the only section of the engine that compiles surprisingly fast). However, nikola_engine.hpp does have heavy dependencies. Specifically, since GLM is exclusively templated, the "engine" part of the project takes quite some time to compile. It isn't slow by any means. But it is frustrating to change something in the renderer definitions only to have the whole math section be recompiled. Perhaps I could have handled it in a better way, but, honestly, I do not mind it. The ease of use this approach gives me is well worth the recompilations. And, besides, the header files are not often changed anyway. So I do not have to recompile all the time.

The other important design decision I had to make was with resources. As I talked about before, I created a custom binary resource format specifically for the engine to use instead of the so-called "middle-man" formats. Or, the PNGs, JPEGs, OGGs, and so on. It is quite a common practice in the game engine space to handle resources that way. It keeps the engine separated from any need for these formats while giving the engineers room to approve the load times and compression of these resources. Quake used its proprietary format .mdl for model files. The Source Engine used the format .vtf for handling textures. And, of course, every modern engine packages its resources in some way or another so that the runtime (the exported game, essentially) can use it as efficiently as possible.

Now, I am not smart. Far from it. However, I decided to also partake in this "tradition" and make my own resource binary format that will, in the future, be subject to better optimizations for loading at runtime. Of course, as discussed before, there is no "runtime" concept in my engine. The runtime in these engines usually means the final exported game. Well, in my case, the engine is the game. So, effectively, I am always in runtime. So, I decided to make a small command line program to convert any given supported resources into the .nbr format. Keep in mind, the engine does not understand nor does it care about any other resource format. It can only parse and decode the .nbr resource format.

I will talk in-depth about the whole resource management system of my engine in a future article. But, for now, know that it is very primitive in its current state. I do not claim that .nbr compresses files to seemingly nothing. In fact, .nbr files do no compressing whatsoever. But, it is a start. And, surprisingly, I have noticed better startup times whenever I'm loading .nbr files as opposed to regular .obj files for example. But, I would have to test that theory later in the future.

Current Progress And The Future

So, where am I? And where am I going?

Currently, as I'm sure you can tell, the engine is still in its infant state. It can do a lot. Currently, it can open a window, accept input, render pixels, load models, and images, and render them even. But there is still a long way to go. For example, audio and fonts are still not fully implemented. While things like entities are not even a thing yet. However, if you are interested, I do have some interesting showcases on my website. You can also go to the engine's repo to check the code for yourself if you are interested.

It is hard to say what the future of this project is. I can say, though, that I am planning to stay with this engine for a while. Once again, my intentions are fairly selfish when it comes to this engine. And, by extension, the planned future for this engine is quite selfish as well. While you can use the engine for your own projects, I will not advertise it as such. In fact, I will not advertise it at all. Show some interesting demos and progress reports here and there, sure. But I will not go out of my way to market this engine as a product. At the end of the day, I'm making this engine solely for my own enjoyment and based entirely on my own philosophies. And, naturally, not everyone will share these sentiments.

Thanks for reading and have a good day/night