The Ghost of Refactors Past…

…has come to hunt me once again. Although this time it’s not because of mistakes that I did. Instead, the problem lies in something that I missed completely.

Can you spot the problem in the following code?

void SomeObject::save( Stream &s )
{
   std::size_t count = getElementCount();
   s.write( count );
}

Well, it turns out std::size_t is NOT PLATFORM INDEPENDENT. And here’s the twist: I knew that since, well, forever, but I never paid any attention to it. That is, until it became a problem. And the problem was EVERYWHERE in the code.

First thing first. The C++ standard has this to say about std::size_t:

typedef /*implementation-defined*/ size_t;

 

What’s that supposed to mean? Basically, std::size_t may have different precision depending on the platform. For example, in a 32-bit architecture, std::size_t may be represented as a 32-bit unsigned integer. Something similar happens in a 64-bit platform.

Wait.

std::size_t is supposed to help portability, right? That’s its whole purpose. And that’s true, of course. There’s nothing wrong with std::size_t itself.

Check the code above again. Go on, I’ll wait.

So, whenever we create a [binary] stream to serialize the scene, we can’t use std::size_t because it’s just plain wrong. It will be saved as a 32-bit integer in some platforms and 64-bit in others. Later, when the stream is loaded, the number of bytes read will depend on whatever the precision of the current platform is, regardless of what was used when saving. See the problem?

This means that we can’t share binary streams between different platforms because the data might be interpreted in different ways, leading to errors when generating scenes.

For the past few years, my main setup have been OS X and iOS, both 64-bit platforms. But one day I had to use streaming on 32-bit Android phones and, as you might have guessed by now, all hell break loose…

Entering crimild::types

I had to made a call here: either we can keep using std::size_t everywhere and handle the special case in the Stream class itself; or we can make use of fixed precision types for all values (specially integers) and therefore guaranteeing that the code will be platform independent.

I went for the second approach, which seems to me to be right choice. At the time of this writing, the new types are:

namespace crimild {

   using Int8 = int8_t;
   using Int16 = int16_t;
   using Int32 = int32_t;
   using Int64 = int64_t;

   using UInt8 = uint8_t;
   using UInt16 = uint16_t;
   using UInt32 = uint32_t;
   using UInt64 = uint64_t;

   using Real32 = float;
   using Real64 = double;

   using Bool = bool;
 
   using Size = UInt64;
}

As you can see, crimild::Size is always defined as a 64-bit unsigned integer regardless of the platform.

Yet that means I need to change every single type definition in the current code so it uses the new types. As you might have guessed, it’s a lot of work, so I’m going to do it on the fly. I think I already tackled the critical code (that is, streaming) by the time I’m writing this post, but there’s still much to be reviewed.

New code already makes use of the platform-independent types. For example, the new particle system is employing UInt32, Real64 and other new types and–

Oh, right, I haven’t talked about the new particle system yet… Well, spoiler alert: there’s a new particle system.

You want to know more? Come back next week, then 🙂

 

 

 

Progress Update – November 2016

This post summarizes all the things I’m currently working on, without any specific priority order (as usual).

Job System

Probably the biggest feature to be included in the next release (whenever that happens) is the new Job System.

Last year I implemented a Task Scheduler for asynchronous work and while it is (was?) useful, it does lack the mechanisms required for programs to properly work concurrently. In particular, there’s no way to wait for tasks to complete or group them together as one unit of work.

Enters the new Job System, in which we’re going to be able to schedule jobs (obviously), that may or may not be linked with other jobs (as in parent/child jobs) and wait for them to complete before continuing. A “work stealing” approach is being used internally and we can spawn as many worker threads as we need.

The new Job System will allow us to move to a more parallel architecture by implementing things like parallel visitors, multi-threaded render queues and more.

Surprisingly, the current state of this feature is quite advanced and looking good. I just pushed the main classes to the development branch and I expect to refactor the Simulation flow pretty soon.

Ray Tracer Improvements

I needed a way to test the new Job System before having to refactor the entire Simulation flow and the Ray Tracer seemed to be the perfect candidate.

Initial tests are promising, showing that the rendering time has been reduced by 60% (depending on the number of worker threads, of course). The image below took “only” 80 minutes to render in my Macbook using 8 worker threads.

output_800_600_100

I’ve included several fixes in the latest code and I’m going to move to phase two (actual geometry, lighting, textures, etc) probably before the end of the year.

Scripting

As it is right now, the scripting system is mostly used for scene building, specifying objects and components in script files using Lua. But I want more. I need more. I want to be able to create components from Lua files and interact between them and first class ones written in C++. I want to be able to create a whole new simulation or game without enforcing the developer to write and compile C++ code.

The improved scripting tools are still in a design phase and even when I would like to include them in the next release, I know they might not make it.

Improving Audio Support

This is something that I had in my list for a while and I’m giving it a try as a side project. At the moment, Crimild supports audio clips using OpenAL (including positional audio, of course) only in WAV format and there’s no way to play background music.

My initial goal is to work with OGG files. By the end of phase one I should be able not only to play music files but also to support some simple mixing mechanisms.

I don’t expect this feature to be included in the next release, but it should be done early next year because it’s something that I need for a different, super secret, project.

Scene Streaming

Yes, this is back. You know, that super useful thing let you save and load entire scenes to disk, using a binary file format (which was there five years ago but then disappeared one day without reason).

A working implementation for scene streaming was included in the last version (I’m not kidding, it’s there). I’m constantly revisiting it by adding support for more components and entities. The new binary file format has proven quite good, and loading times have been reduced in a great amount.

That’s it. Well, not really. There might be one or two more things in my TODO list right now, but the ones above are the most important ones. I’m planning on having a new release before the year’s done, including most of these features. So, stay tuned.

 

Introducing: Crimild’s Sandbox

I’ve been in autopilot mode during the last month or so and thus it was hard for me to keep the blog updated. A lot of exiting things have been added to Crimild lately (and to everyone’s surprise, just one or two little refactors) and I promise I will describe them during the following weeks. But for now let’s focus on the newest addition to the engine: the Sandbox

The Sandbox

The Sandbox

What is the Sandbox?

While we we’re discussing about our next project at Building Brothers, we realized that a tool was needed in order to speed up development process, not only on the technical aspects, but also on level design, assets and gameplay. Our Game Designer needs a tool to create a level and test any new feature, making changes in real-time if needed without having to wait to developer to implement them. Artists needed a way to update models and textures and verify that they look correctly without the need of a new build. And developers needed a tool to implement and test new features as fast as possible through the use of scripting and shader editors. And so the Sandbox was born.

Crimild’s Sandbox is a desktop application which goal is to help developers and designers alike by including a WYSIWYG scene editor, scripting and debugging tools and some other features that will be disclosed as they’re implemented. It currently runs only on Mac OS X, but it will be available on Windows and Linux as soon as I have the time to do so.

Loading and displaying a scene in the Sandbox

Loading and displaying scenes in the Sandbox

Is the Sandbox a Game Editor?

The short answer is “no” (although a more optimistic one would be “not yet”). As it is right now, the Sandbox is more of a testing tool than an editor. Of course, you can use it to build scenes and later import them into your projects. But the whole idea of the Sandbox is to be able to experiment with any new feature and then see how it behaves in real-time. Since the Sandbox uses the same rendering techniques as any other Crimild-based app, you can rest assure that if something works in the Sandbox, there is a pretty big chance that it will work on your program. Of course, platform-specific constrains may still be an issue but the Sandbox will help you isolate them.

What’s included in the box?

The Sandbox is still in a very early stage of development, with only a couple of features already working. Among them we have a Scene Hierarchy Viewer that let us understand how a scene is composed by displaying a tree of nodes. In addition, a Node Inspector displays information about a selected node in the scene, along with a list of components attached to it. If a ScriptComponent is found, a Script Editor is available for the user to edit the script in real-time without having to restart the application. A simple Pathfinding tool is also included, allowing level designers to place waypoints comforming a walkable path for characters.

Script Editor

Editing Lua scripts in real-time

The Sandbox has two modes: Editor Mode and Simulation Mode. The former one will let you create and change a scene using any of the available tools. You can add new nodes, components, cameras, scripts, etc. Once you have your scene ready, you enter Simulation Mode by clicking the Start Simulation button. In this mode, scripts and other behaviors are executed. It is possible to edit scripts while the simulation is running and see the changes take place in real-time.

Path edition tool

Path edition mode

New Crimild features

I’m going to dedicate full posts to each of the latest changes in Crimild, but here’s a brief introduction to the ones related to the Sandbox:

HTML interfaces with WebKit/Awesomium

This is perhaps the most interesting of the newest additions. I integrated the Awesomium framework in order to create HTML based user interfaces for the Sandbox. By using HTML, CSS and Javascript we can create new panels and editors in minutes instead of having to code an entire UI library for Crimild. Right now, all panels in the Sandbox are actually HTML Divs that can be dragged around the screen if needed. Awesomium is based on Chrome’s WebKit and renders all output into a memory buffer that can be later used as a texture. It’s very simple to use and the results are incredible (no, I’m not going to say “awesome”. Oh, I just did).

XML Scene Factory

The Scene Factory is a new addition to Crimild that let developers define new scenes using XML, declaring nodes and custom components, in addition to Lua scripts and OpenGL shaders. New scenes can be created and used in your programs without having to wait for a new build.

Scripting

I mentioned this several times in this post: Crimild has support for Lua scripts. Yeah!! A new ScriptComponent class has been added to the engine that handles script compilation and execution. Scripts can be created on the Sandbox and built on the fly without having to restart the application. Scripting support is a huge speed boost for the development process

Closing comments

Even in its current state, the Sandbox has proven to be a great tool for rapid prototyping of scenes. I successfully managed to position cameras and make them behave as I commanded by using Lua scripts. Also some simple pathfinding techniques have been tested on the Sandbox to be later used in our next projects.

As for the future, I’m planning on adding a lot of debugging tools and probably a shader program editor. Some sort of binary file format specification will be needed at some point in order to save our scene and import it in another project, but we can manage with XML so far. There are too many features that can be added in future iterations that is hard to tell how big this thing can become.