FR: Allow templates to access binary schema

[wibotwi]

If I have a demo.fbs with Demo table, then fb can generate embed schema as static method in struct DemoBinarySchema in file demo_bfbs_generated.h

My method is template<T> send(const T& data).
I need a way to access DemoBinarySchema from DemoT.

For example fbs already allows accessing table type from T type:

struct DemoT: ... {
    typedef Demo TableType;

This allows me to use T::TableType inside my send template method. I need similar approach for embed binary schema.

Thank you

[wibotwi]

TIL that schema is generated only for root_type. This is fine, the access should just be added for all types.
e.g. if demo.fbs has second table FooBar then it might have:

struct FooBarT: ... {
    typedef FooBar TableType;

    typedef DemoBinarySchema BinarySchema;  // points to Demo binary schema, because Demo is root type

or if it is a method:

struct FooBar ... {
    static std::string_view<uint8_t> binarySchema() { return DemoBinarySchema(); }   // FooBar::binarySchema() returns root schema
}

[dbaileychess]

First, what are you using the binary schema for that you need it for each individual table? That might help determine if you need this or not.

We only generate it for the root_type as defined here and we assume its at the start of the buffer. If there are multiple tables in your schema, you wouldn't be able to get a "clean" bfbs binary out of it. Because the reflection flatbuffer might share items between the two types. You would really need two fbs files with only a single root_type defined for each.

[wibotwi]

I don't need it to be generated for each table. The bfbs is not technically for a root type: it contains all definitions from fbs file. And it works perfectly, no need to change that.

What I need is a generic way to access it from the table class. All of tables should return pointer to the same bfbs.
I think best way is to do it with typedef:
every table should have "typedef BinarySchema = RootTypeBunarySchema".

Of course for it to work we need to generate include bfbs header. And this should be done only if embed parameter is provided.

[dbaileychess]

So you want the list of objects (i.e. tables) from the reflected schema?

I feel like you would want something like this:

struct FooBarT {
   typedef FooBar TableType;
 
   static reflection::Object* GetSchema();    
}

Where that would return you the reflection::Object * of table FooBar.

Is there a reason you would want the whole reflected schema over the particular object you are working with?

[wibotwi]

I have a function that I give to clients: "bool send(T& msg)"
This function does the following:

1. if this message type is first ever seen - send bfbs over the wire
2. serializes msg into buffer
3. sends the buffer over the wire

These messages are arrived to server side.
So server gets this:

1. First message from the wire (at start of day) - it gets bfbs and saves it.
2. For all next incoming messages server parses them using bfbs and reflection.

Server knows what information to search in messages by using attributes and other hints that it extracts from bfbs.
But server is not recompiled for each fbs. Server is 100% dynamic. We can consider server was started up 10 years ago and is still running.
But new clients with new fbs schemas pop up every day. And they stream live data to server. And server can process them all, because for every message it has a binary schema that it periodically receives from clients.

Implementation of send method is something like:

template<class T>
void send(const T& msg) {
    auto msgType = typeid(msg);
    if (_seenTypesMap.find(msgType) == map.end()) {
        // first time we see this msg type, so we need to send schema
        _bus.send(T::GetBinarySchema());    --- This is missing now.
    }

    auto* buffer = flatbuffers::serialize( msg );   // actual code is much bigger, but essentially it just does serizlization
    _bus.send(msgType, buffer);   // we send msgType with every message so that server knows how to parse it
}

[dbaileychess]

Gotcha, thanks for the background, your request makes more sense now.

So your server has no strongly-typed code related to your schema, it is 100% done be reflection? Almost sounds like you would like flexbuffers since it is schemaless.

Can you also provide the flags you are passing flatc and maybe a min example schema.

Of course for it to work we need to generate include bfbs header. And this should be done only if embed parameter is provided.

So we current put the binary schema in a <NAME>_bfbs<POSTFIX>.h file. You just want that to be auto included in the main generated file? The alternative is just to emit the binary schema in the same file. Generally for C++ we only make one file per flatc invocation.

[dbaileychess]

@wibotwi I put something up, let me know what you think.

[wibotwi]

wow, thanks!

But do we need both typedef and static BinarySchema& GetBinarySchema() ?
Both methods data() and size() are static.
If I have typedef I can access everything:

auto* binary_schema_data = MonsterT::BinarySchema::data();
auto binary_schema_size = MonsterT::BinarySchema::size();

// Verify the schema is OK. 
flatbuffers::Verifier verifierEmbeddedSchema(binary_schema_data,
                                               binary_schema_size);

[dbaileychess]

Updated the PR to reflect that change.

[wibotwi]

Looks perfect!
Thank you