Delta Serialization

You change a default value in your code. Recompile. Run. Everything looks fine.

Except that every existing scene instance still has the old value. No warning. No error. Just broken data, silently waiting for you to notice.

During my 3rd year at Breda University of Applied Sciences, I built another game engine from scratch with a team. This exact problem kept biting us as our scenes grew larger and more complex. So I fixed it.

Problem:

User makes a new component:

struct MyComponent {
    int a = 42;
    float b = 3.14f;
    bool c = true;
    std::string d = "Hello";
};
JSON_REFLECT(MyComponent, a, b, c, d); /* relfection */

Adds the component to an entity and the scene now stores:

{
    "MyComponent": {
        "a": 42,
        "b": 3.14,
        "c": true,
        "d": "Hello"
    }
}

Now, the user decides that variable d is not supposed to be "Hello" but "Goodbye" by default, so they change the code to:

struct MyComponent {
    // ...
    std::string d = "Goodbye";
};

Except… all existing instances of the component still contain Hello instead of "Goodbye"…

The user now has to go back into the scene, and either re-add all instances, or manually modify the variable to "Goodbye". (tedious and error-prone)

Solution

Only serialize values that have been changed!

What if, instead of serializing everything every time, we check if a member value has changed? IF it has changed, we serialize it! For example:

MyComponent input{};
input.a = 100;
input.b = 2.71f;
// input.c and input.d remain unchanged
json j = JsonReflect::to_json(input);

Output:

{
    "a": 100,
    "b": 2.71
}

This allows the user to freely modify the default values, without having to manually “fix” the old data. Additionally, this can also reduce the amount of data that is being serialized, thus lowering scene/level size on disk!

Harder than it sounds

While the idea may sound simple, “just check if the value has changed”, it’s a bit more technical than that. I had 2 ideas in mind:

1. Dirty Flag

Whenever we change the value of a variable, we keep track of it by marking the value “dirty”. Then, when we want to serialize that value, we first check if the value is dirty. A quick prototype I made:

template<typename T, auto Default>
struct Tracked {
    T value = Default;
    bool dirty = false;

    Tracked& operator=(const T& v) {
        value = v;
        dirty = true;
        return *this;
    }
    operator const T& () const { return value; }
};

struct MyComponent {
    Tracked<int, 42> a;
    Tracked<float, 3.14f> b;
    Tracked<bool, true> c;
    /* std::string doesn't work */
};

If we now change the value of a member variable, the dirty flag will be set to true. E.g.:

MyComponent instance{};
// instance.a.dirty is false
instance.a = 100;
// instance.a.dirty is true

However, this approach has multiple issues:

1. Refactoring, requires a big refactor of the existing codebase.
2. Intrusive, it requires the user to own the objects; external objects won’t work.
3. Types not supported, std::string doesn’t compile and gives error C2762: 'Tracked': invalid expression as a template argument for 'Default
4. Memory alignment, it changes the size and how the object is stored in memory:

While this was an interesting idea, due to all the reasons stated above, I decided to go with the second approach, which is what I implemented in the end:

2. Default Construct Compare

Because I was already using visit_struct for reflection in my JsonReflect library, serialization looked like this for objects that had reflection:

template <typename T>
static json to_json(const T& value) {
    json j;
    /* Loop through each member variable */
    visit_struct::for_each(value, 
        [&](const char* name, const auto& member_value) {
            j[name] = to_json(member_value); /* serialize member */
        }
    );
    return j;
}

To read more about my JsonReflect library, check out my other blog post.

What I needed to achieve was a way to check if member_value was equal to its default value. Luckily, visit_struct offered a handy function called Binary Visitation where you can loop through 2 objects at the same time:

MyObject object_a;
MyObject object_b;
visit_struct::for_each(object_a, object_b, 
    [&](const char* name, const auto& member_a, const auto& member_b) {
        /* Access to both members at the same time */
    }
);

This allowed me to construct a temporary default value of type T, and use that to compare with:

template <typename T>
static json to_json(const T& value) {
    json j;

    /* Construct a default T */
    const T default_value{};

    /* Loop through each member variable */
    visit_struct::for_each(value, default_value,
        [&](const char* name, const auto& member_value, const auto& default_value) {
            /* Check if they are NOT the same */
            if(member_value != default_value) {
                j[name] = to_json(member_value); /* serialize */
            }
        }
    );
    return j;
}

This gave me my expected results:

MyComponent input{};
input.a = 100;
input.b = 2.71f;
// input.c and input.d remain unchanged
json j = JsonReflect::to_json(input);

Output:

{
    "a": 100,
    "b": 2.71
}

While looping through each member, it saw that both c and d were the same as their default values and decided to skip them in the serialization.

Results

Technical Details

If you’re still reading and have reached this point without getting scared away by all the templates, I assume you’re interested in the technical details of this approach. There were some things I had to consider when implementing this.

Opt-in type trait

Since I implemented this feature into my JsonReflect library, I didn’t want to break existing use cases or confuse users as to why sometimes their values are not serialized. Additionally, maybe users don’t want every type to have delta serialization. For this reason, I made the delta serialization opt-in only.

template <typename T>
static json to_json(const T& value) {
    json j;
    
    constexpr bool delta_serialize_v = delta_serialize<T>::value;
    if constexpr (delta_serialize_v) {
        // Compare serialization...
    } else{
        // Default serialization...
    }
    return j;
}

This means if users want their type to be delta serialized, they have to specialize this type trait:

template<>
struct JsonReflect::Detail::delta_serialize<MyComponent> : std::true_type {};

MyComponent will now be serialized with delta serialization enabled.

If you want to learn more about type traits, check out my other blog post where I go into more detail about what they are and how to implement them.

Default constructible

If you’re a C++ developer, you probably already know that some objects are not default constructible. This means that this line of code:

/* Construct a default T */
const T default_value{};

Doesn’t always work/compile. To account for this, I made another type trait called delta_default:

template<typename T>
struct delta_default {
    /* Default, assumes default-constructible */
    static T make() {
        static_assert(std::is_default_constructible_v<T>, "JsonReflect: T is not default-constructible. Implement/specialize delta_default<T> to provide a baseline instance.");
        return T{};
    }
};

By default, it checks if the type is default constructible using std::is_default_constructible_v.

If the type is default constructible, it just creates the object as normal and returns it.

If a type is not default constructible, it static asserts and lets the user know they need to implement a delta_default trait.

This means for a type that requires some constructor arguments, the user has to specialize a delta_default type trait:

struct NonDefaultConstructible {
	int a;
	float b;
	bool c;
    NonDefaultConstructible(int a, float b, bool c) : a(a), b(b), c(c) {}
};

template<>
struct JsonReflect::Detail::delta_default<NonDefaultConstructible> {
	static NonDefaultConstructible make() {
		return NonDefaultConstructible{ 0, 0.0f, true };
	}
};

The delta serialization now uses this make function to create the default value for comparison:

/* Construct a default T */
const T default_value = delta_default<T>::make();

Static vs re-allocate

In the to_json function, we construct an object to compare against. In the example, I do:

/* Construct a default T */
const T compare = delta_default<T>::make();

/* Loop through each member variable */
visit_struct::for_each(value, default_value, // ...

Depending on the object, allocating it every time we need to serialize can be quite expensive. For that reason, I leave it up to the user which one they want. They can define a macro called JSON_REFLECT_STATIC_FOR_DELTA which makes the compare object static, so it only gets allocated once and is reused for every serialization:

#if JSON_REFLECT_STATIC_FOR_DELTA
    static T compare = delta_default<T>::make();
#else
    const T compare = delta_default<T>::make();
#endif

Equal operator

To check if a member variable has changed, we need to compare it with the default value. For this, we need an equal operator (operator==) implemented for that type. If the type doesn’t have an equal operator, we can’t compare it and the code doesn’t compile.

To solve this, I first do some checks if a type is comparable. I check for both operator== and operator!=*.

*C++ 20 made it so if you have a operator==, you automatically get a operator!=. Since my library supports C++ 17 and up, I have to manually check for == AND !=.

/* == */
template<typename T, typename = void>
struct is_equality_comparable : std::false_type {};

template<typename T>
struct is_equality_comparable<T, std::void_t<
    decltype(std::declval<const T&>() == std::declval<const T&>())
    >> : std::true_type {};

/* != */
template<typename T, typename = void>
struct is_inequality_comparable : std::false_type {};

template<typename T>
struct is_inequality_comparable<T, std::void_t<
    decltype(std::declval<const T&>() != std::declval<const T&>())
    >> : std::true_type {};

template <typename T>
constexpr bool is_equality_comparable_v = is_equality_comparable<T>::value;

template <typename T>
constexpr bool is_inequality_comparable_v = is_inequality_comparable<T>::value;

Now, when looping through each member variable, I check if it has any of the operators:

/* Loop through each member variable */
visit_struct::for_each(value, default_value,
    [&](const char* name, const auto& member_value, const auto& default_value) {
        using Field_T = std::decay_t<decltype(member_value)>;

        if constexpr (is_equality_comparable_v<Field_T>) { /* check for == */
            if (member_value == default_value) return;
            j[name] = to_json(member_value);
        } else if constexpr (is_inequality_comparable_v<Field_T>) { /* check for != */
            if (member_value != default_value)
                j[name] = to_json(member_value);
        } else {
            static_assert(svh::always_false<Field_T>::value, "JsonSerializer Error: Type T is set to delta serialize but has no equality operator or inequality operator.");
        }
    }
);

This forces every member variable to have a == or != operator and static asserts if it doesn’t. Since this might not always be the case because users maybe don’t own the type. I made a fallback approach:

if constexpr (is_equality_comparable_v<Field_T>) { /* check for == */
    if (member_value == default_value) return;
    j[name] = to_json(member_value);
} else if constexpr (is_inequality_comparable_v<Field_T>) { /* check for != */
    if (member_value != default_value)
        j[name] = to_json(member_value);
} else {
#if JSON_REFLECT_ALLOW_JSON_COMPARE
    /* ! EXPENSIVE ! */
    /* No equality operator, serialize both and compare json */
    json field_json = to_json(member_value);
    json compare_json = to_json(default_value);
    if (field_json != compare_json) {
        j[name] = std::move(field_json);
        return;
    } else {
        return; /* No change, skip */
    }
#else
    static_assert(svh::always_false<Field_T>::value, "JsonSerializer Error: Type T is set to delta serialize but has no equality operator, inequality operator and macro JSON_REFLECT_ALLOW_JSON_COMPARE is disabled.");
#endif
}

This fallback simply serializes both values, and instead of comparing the values, it compares the JSON object instead. This way we can still compare values without having a == or != operator. However, this approach does have the downside that we need to serialize both values AND compare the JSON objects. This can take longer than a simple == or !=, but since it’s optional, it’s in my opinion worth having.

Conclusion

Delta serialization turned out to be one of those problems that looks trivial on the surface, “just check if the value changed”, but quietly hides a chain of edge cases the moment I tried to make it production-ready. Non-default-constructible types, missing equality operators, memory allocation costs, and backwards compatibility all needed an answer before it could actually ship.

It’s worth noting that none of this is specific to JSON. The same principle applies to binary formats, network packets, or save files. The 73% file size reduction is a JSON number, but the real win is format-agnostic.

My biggest takeaway is this: the moment you serialize everything unconditionally, you’re locking in your defaults forever, silently, and with no warning. Delta serialization breaks that lock, and that was worth a lot more than just 73% smaller files.

If you’re still reading and want to see the full implementation, you can check out my JsonReflect library or read my other blog post where I go into more detail about the library itself.