How to make Seraphis wallet related structs and classes serializable

[rbrunner7]

This text aims to be something like a manual how to make structs and classes related to the Seraphis wallet serializable, i.e. getting able to persist them to files. The approach described is not set in stone with all its details already, and as far as I remember we never had a workgroup meeting where we confirmed as a group "Yes, let's do it this way". Please do comment in this issue if you have questions or disagree with details or even the whole approach in general.

---

Here serialization means going from C++ objects in RAM as the wallet creates and processes them to data stored in files that you later can use to re-create these objects, in a process called deserialization.

For this job Monero uses a system inherited from the original CryptoNote code base in many places, and quite extensively in wallet2. It's based on a class called binary_archive. Its definition is here.

To make an object support serialization you add calls to macros like BEGIN_SERIALIZE_OBJECT and FIELD to the struct or class definition, like in this example from wallet2.h:

    struct pool_payment_details
    {
      payment_details m_pd;
      bool m_double_spend_seen;

      BEGIN_SERIALIZE_OBJECT()
        VERSION_FIELD(0)
        FIELD(m_pd)
        FIELD(m_double_spend_seen)
      END_SERIALIZE()
    };

Technically it's a very simple binary format that gets produced, basically only the "naked" data stored. That's why you need versioning to support several variants of structures with upward compatibility; e.g. there is no general way to query "Is field x present in the serialized data?".

As you can see e.g. here there is a large number of such systems in use, many of them certainly more modern and more versatile than this proprietary format that Monero uses, e.g. heavyweights like Protocol Buffers, originally from Google. But on the other hand binary_archive is mature, is simple, and kind of a "standard" as far as the Monero code base is concerned.

Now, with many versions that some classes went through in more than 10 years of Monero history, and with the complexity that some classes have, things can get pretty hairy. E.g. have a look at the serialization code for the transaction class here, or even worse for some RingCT related things here.

---

When the time came to add serialization capability to the structs and classes of the Seraphis library, in the light of this state of affairs in the existing code base, author @UkoeHB decided that adding serialization code directly to the objects is not a promising approach. Most importantly, as far as I understood, he wanted to keep any serialization related complexity that may develop over time out of the library proper.

The approach he chose is to work with dedicated serialization objects. In principle, every type in the Seraphis library that needs the capability to persist gets something like a partner type that gets serialized. Helper methods convert between the "real" types and the serialization types.

The result is 3 files serialization_demo_* in this folder of the library implementation.

An example: ser_SpEnoteV1 is the serializable partner of SpEnoteV1. The code to go from the latter to the former type is here.

You can also have a look at a recently merged PR to the library from @DangerousFreedom1984 here where he added support for JamtisDestinationV1. From the added code pieces you should be able to see quite clearly how it's done.