One Function to Serialize Them All

During block B of year 3, I have been working on a self-study project github.com/Sven-vh/JsonReflect, a reflection-based JSON serialization library for C++.

In this blog post, I will walk through my thought process of making my most important function of this library.

Problem

Just like my previous self-study ImReflect, this project heavily uses reflection and Tag Invoke to create a generic serialization and deserialization system.

For this project, I wanted to have the default C++ and STL types supported out of the box. This includes types like int, float, std::string, std::vector<T>, etc.

To do this. I need a serialize() function for every type I want to support by default.

Initial Implementation

I started with the primitives. Luckily github.com/nlohmann/json, JSON library I use, already supports these types by default. So my initial implementation looked like this:

using json = nlohmann::json;

json serialize(const int& value) {
    return json(value);
}

json serialize(const float& value) {
    return json(value);
}

json serialize(const bool& value) {
    return json(value);
}

Have 3 overloaded functions that all take different types.

Now whenever I call serialize() with an int, float, or bool, the correct function will be called and return a JSON object with the value inside.

While this works, and all 3 types correctly get serialized, it means that I would have to write an overload for every type. That may not sound like a lot, until you realize that there are quite a lot of default C++ types:

Integrals:

bool, char, signed char, unsigned char, wchar_t, char8_t, char16_t, char32_t, short, unsigned short, int, unsigned int, long, unsigned long, long long, unsigned long long

Floating points:

float, double, long double

Writing an implementation for each of those goes against my programmer instinct and breaks the DRY principle.

DRY - “Don’t Repeat Yourself”

I also noticed that the implementations of the functions are exactly the same. They all simply return a JSON object with their value inside.

Templates

With the template meta programming skills I learned during ImReflect, I knew I could make this more generic. Instead of writing an implementation for each type, I would take in a templated type!

template<typename T>
json serialize(const T& value) {
    return json(value);
}

Nice, All integrals and floating point numbers work with just a single function!

serialize(42);                      // works!
serialize(3.14f);                   // works!
serialize(true);                    // works!

Except…

This function takes in everything. So if we would give it, let’s say a vec3 or a custom struct that nlohmann doesn’t support/recognize, it would throw an error.

struct vec3 {
    float x, y, z;
};

serialize(vec3{1.0f, 2.0f, 3.0f}); // Compilation Error!

Even if we would make a specialization for vec3 later on, this function would still be considered during overload resolution and cause an ambiguity error:

template<typename T>
json serialize(const T& value) {
    return json(value);
}

template<>
json serialize(const vec3& value) {
    return json::array({value.x, value.y, value.z});
}

serialize(vec3{1.0f, 2.0f, 3.0f}); // Ambiguity Error!

To solve this, what if, instead of taking everything, we only accept integrals and floating points?

Type Traits & SFINAE

Here is where type traits and SFINAE come in. Instead of taking everything with template <typename T>, we can “filter” the types we want. For our example, this looks like:

template <typename T>
std::enable_if_t<std::is_arithmetic_v<T>, json> serialize(const T& value) {
    return json(value);
}

If you’ve never worked with templates, this can look quite complex. To simplify, we have 2 things: std::enable_if_t and std::is_arithmetic_v that make this work.

std::enable_if_t is basically an if check at compile time. It checks if the first template parameter is true, and if it is, it will return the second template parameter as the return type of the function. If the first parameter is false, this function will not be considered during overload resolution.

std::enable_if_t<CONDITION, RETURN TYPE>

We then use std::is_arithmetic_v to check if the template parameter is an arithmetic number. This type traits returns true if you give it any C++ integral or floating point type. See cppreference for more information.

Awesome! We now have a function that will only take in the numeric and does not crash when it encounters other types!

template <typename T>
std::enable_if_t<std::is_arithmetic_v<T>, json> serialize(const T& value) {
    return json(value);
}

template<>
json serialize(const vec3& value) {
    return json::array({value.x, value.y, value.z});
}

serialize(vec3{1.0f, 2.0f, 3.0f}); // works! calls vec3 specialization
serialize(42);                      // works! calls numeric version

Now this function is only possible because nlohmann supports these types by default and has a JSON constructor for them.

This got me thinking… What else does it support??

According to the documentation it supports by default:

That means, I can simply make some overloads/specializations for all of these types:

json serialize(const std::string& value) {
    return json(value);
}
template<typename T>
json serialize(const std::vector<T>& value) {
    return json(value);
}

template<typename T, std::size_t N>
json serialize(const std::array<T, N>& value) {
    return json(value);
}

Super simple!

serialize(std::string("Hello World"));          // works!
serialize(std::vector<int>{1,2,3,4,5});         // works!
serialize(std::array<float,3>{1.0f,2.0f,3.0f}); // works!

Except…

They all have the same implementations again! They simply return a JSON object that takes in the type value. My programmer instinct is again telling me that I am breaking the DRY principle here.

The Holy Grail: nlohmann

I started looking into the nlohmann JSON source code to see how they determine which type are and are not supported. After some digging, I found the nlohmann::detail::is_compatible_type type trait!

This type trait takes in a templated type and returns true if it’s supported by nlohmann. For example:

using namespace nlohmann::detail;

is_compatible_type<json, int>::value;                       // true
is_compatible_type<json, float>::value;                     // true
is_compatible_type<json, std::vector<int>>::value;          // true
is_compatible_type<json, std::map<std::string, int>>::value;// true

is_compatible_type<json, vec3>::value;                      // false
is_compatible_type<json, GameSettings>::value;              // false

This is perfect!

With this type trait, I can now super easily create a super generic function that takes in all supported types and simply return a JSON:

/* Check if type is json-compatible */
template<typename T>
inline constexpr bool is_json_compatible_v = nlohmann::detail::is_compatible_type<nlohmann::json, T>::value;

template<typename T>
std::enable_if_t<is_json_compatible_v<T>, json> serialize(const T& value) {
    return json(value);
}

This single function can serialize all supported types by nlohmann! No more code duplication. Additionally, I can still make specializations/overloads for custom types when needed.

using json = nlohmann::json;

template<typename T>
std::enable_if_t<is_json_compatible_v<T>, json> serialize(const T& value) {
    return json(value);
}

template<>
json serialize(const vec3& value) {
    return json::array({value.x, value.y, value.z});
}

serialize(42);                      // works! calls nlohmann version
serialize(std::string("Hello"));    // works! calls nlohmann version
serialize(vec3{1.0f, 2.0f, 3.0f});  // works! calls vec3 specialization

Conclusion

Using type traits and template meta programming, I was able to create a single generic function that can serialize all supported types by nlohmann/json. This drastically reduced code duplication and made the codebase much cleaner and easier to maintain.

See the full project on github.com/Sven-vh/JsonReflect