Support newer Protobuf

[FlorianReimold]

Context

Currently, we have Protobuf 3.11 in the submodule, which is used for Windows. That version is quite old and newer versions would have better support for C#. Also, the old version of protobuf currently forces users to install old python protobuf wheels, as we ship the eCAL wheel with pre-generated pb2.py files.

Proposal

The goal here is to update the protobuf submodule to something more recent.

Tasks and updates

• Update Protobuf to 3.21 in the submodule https://github.com/eclipse-ecal/ecal/tree/feature/protobuf_3_21_12
• Fix Windows build
• Fix pre-generated pb2.py files of eCAL python samples
• Test python wheels

[FlorianReimold]

Looks like C# doesn't like the new protobuf. Maybe we just need to re-generate the files?

D:\a\ecal\_build\complete\lang\csharp\samples\pubsub\protobuf\protobuf\protobuf\animal.cs(41,13): error CS0234: The type or namespace name 'IBufferMessage' does not exist in the namespace 'Google.Protobuf' (are you missing an assembly reference?) [D:\a\ecal\_build\complete\lang\csharp\samples\pubsub\protobuf\protobuf\person_protobufs_cs.csproj]
D:\a\ecal\_build\complete\lang\csharp\samples\pubsub\protobuf\protobuf\protobuf\house.cs(41,13): error CS0234: The type or namespace name 'IBufferMessage' does not exist in the namespace 'Google.Protobuf' (are you missing an assembly reference?) [D:\a\ecal\_build\complete\lang\csharp\samples\pubsub\protobuf\protobuf\person_protobufs_cs.csproj]
D:\a\ecal\_build\complete\lang\csharp\samples\pubsub\protobuf\protobuf\protobuf\house.cs(150,14): error CS0234: The type or namespace name 'IBufferMessage' does not exist in the namespace 'Google.Protobuf' (are you missing an assembly reference?) [D:\a\ecal\_build\complete\lang\csharp\samples\pubsub\protobuf\protobuf\person_protobufs_cs.csproj]
D:\a\ecal\_build\complete\lang\csharp\samples\pubsub\protobuf\protobuf\protobuf\house.cs(210,14): error CS0234: The type or namespace name 'IBufferMessage' does not exist in the namespace 'Google.Protobuf' (are you missing an assembly reference?) [D:\a\ecal\_build\complete\lang\csharp\samples\pubsub\protobuf\protobuf\person_protobufs_cs.csproj]
D:\a\ecal\_build\complete\lang\csharp\samples\pubsub\protobuf\protobuf\protobuf\animal.cs(169,14): error CS0234: The type or namespace name 'IBufferMessage' does not exist in the namespace 'Google.Protobuf' (are you missing an assembly reference?) [D:\a\ecal\_build\complete\lang\csharp\samples\pubsub\protobuf\protobuf\person_protobufs_cs.csproj]
D:\a\ecal\_build\complete\lang\csharp\samples\pubsub\protobuf\protobuf\protobuf\animal.cs(243,14): error CS0234: The type or namespace name 'IBufferMessage' does not exist in the namespace 'Google.Protobuf' (are you missing an assembly reference?) [D:\a\ecal\_build\complete\lang\csharp\samples\pubsub\protobuf\protobuf\person_protobufs_cs.csproj]
D:\a\ecal\_build\complete\lang\csharp\samples\pubsub\protobuf\protobuf\protobuf\animal.cs(169,53): error CS0234: The type or namespace name 'WriteContext' does not exist in the namespace 'Google.Protobuf' (are you missing an assembly reference?) [D:\a\ecal\_build\complete\lang\csharp\samples\pubsub\protobuf\protobuf\person_protobufs_cs.csproj]
D:\a\ecal\_build\complete\lang\csharp\samples\pubsub\protobuf\protobuf\protobuf\animal.cs(169,10): error CS0538: 'IBufferMessage' in explicit interface declaration is not an interface [D:\a\ecal\_build\complete\lang\csharp\samples\pubsub\protobuf\protobuf\person_protobufs_cs.csproj]
D:\a\ecal\_build\complete\lang\csharp\samples\pubsub\protobuf\protobuf\protobuf\animal.cs(243,55): error CS0234: The type or namespace name 'ParseContext' does not exist in the namespace 'Google.Protobuf' (are you missing an assembly reference?) [D:\a\ecal\_build\complete\lang\csharp\samples\pubsub\protobuf\protobuf\person_protobufs_cs.csproj]
D:\a\ecal\_build\complete\lang\csharp\samples\pubsub\protobuf\protobuf\protobuf\animal.cs(243,10): error CS0538: 'IBufferMessage' in explicit interface declaration is not an interface [D:\a\ecal\_build\complete\lang\csharp\samples\pubsub\protobuf\protobuf\person_protobufs_cs.csproj]
D:\a\ecal\_build\complete\lang\csharp\samples\pubsub\protobuf\protobuf\protobuf\house.cs(150,53): error CS0234: The type or namespace name 'WriteContext' does not exist in the namespace 'Google.Protobuf' (are you missing an assembly reference?) [D:\a\ecal\_build\complete\lang\csharp\samples\pubsub\protobuf\protobuf\person_protobufs_cs.csproj]
D:\a\ecal\_build\complete\lang\csharp\samples\pubsub\protobuf\protobuf\protobuf\house.cs(150,10): error CS0538: 'IBufferMessage' in explicit interface declaration is not an interface [D:\a\ecal\_build\complete\lang\csharp\samples\pubsub\protobuf\protobuf\person_protobufs_cs.csproj]
D:\a\ecal\_build\complete\lang\csharp\samples\pubsub\protobuf\protobuf\protobuf\house.cs(210,55): error CS0234: The type or namespace name 'ParseContext' does not exist in the namespace 'Google.Protobuf' (are you missing an assembly reference?) [D:\a\ecal\_build\complete\lang\csharp\samples\pubsub\protobuf\protobuf\person_protobufs_cs.csproj]
D:\a\ecal\_build\complete\lang\csharp\samples\pubsub\protobuf\protobuf\protobuf\house.cs(210,10): error CS0538: 'IBufferMessage' in explicit interface declaration is not an interface [D:\a\ecal\_build\complete\lang\csharp\samples\pubsub\protobuf\protobuf\person_protobufs_cs.csproj]
D:\a\ecal\_build\complete\lang\csharp\samples\pubsub\protobuf\protobuf\protobuf\person.cs(45,13): error CS0234: The type or namespace name 'IBufferMessage' does not exist in the namespace 'Google.Protobuf' (are you missing an assembly reference?) [D:\a\ecal\_build\complete\lang\csharp\samples\pubsub\protobuf\protobuf\person_protobufs_cs.csproj]
D:\a\ecal\_build\complete\lang\csharp\samples\pubsub\protobuf\protobuf\protobuf\person.cs(249,14): error CS0234: The type or namespace name 'IBufferMessage' does not exist in the namespace 'Google.Protobuf' (are you missing an assembly reference?) [D:\a\ecal\_build\complete\lang\csharp\samples\pubsub\protobuf\protobuf\person_protobufs_cs.csproj]
D:\a\ecal\_build\complete\lang\csharp\samples\pubsub\protobuf\protobuf\protobuf\person.cs(391,14): error CS0234: The type or namespace name 'IBufferMessage' does not exist in the namespace 'Google.Protobuf' (are you missing an assembly reference?) [D:\a\ecal\_build\complete\lang\csharp\samples\pubsub\protobuf\protobuf\person_protobufs_cs.csproj]
D:\a\ecal\_build\complete\lang\csharp\samples\pubsub\protobuf\protobuf\protobuf\person.cs(249,53): error CS0234: The type or namespace name 'WriteContext' does not exist in the namespace 'Google.Protobuf' (are you missing an assembly reference?) [D:\a\ecal\_build\complete\lang\csharp\samples\pubsub\protobuf\protobuf\person_protobufs_cs.csproj]
D:\a\ecal\_build\complete\lang\csharp\samples\pubsub\protobuf\protobuf\protobuf\person.cs(249,10): error CS0538: 'IBufferMessage' in explicit interface declaration is not an interface [D:\a\ecal\_build\complete\lang\csharp\samples\pubsub\protobuf\protobuf\person_protobufs_cs.csproj]
D:\a\ecal\_build\complete\lang\csharp\samples\pubsub\protobuf\protobuf\protobuf\person.cs(391,55): error CS0234: The type or namespace name 'ParseContext' does not exist in the namespace 'Google.Protobuf' (are you missing an assembly reference?) [D:\a\ecal\_build\complete\lang\csharp\samples\pubsub\protobuf\protobuf\person_protobufs_cs.csproj]
D:\a\ecal\_build\complete\lang\csharp\samples\pubsub\protobuf\protobuf\protobuf\person.cs(391,10): error CS0538: 'IBufferMessage' in explicit interface declaration is not an interface [D:\a\ecal\_build\complete\lang\csharp\samples\pubsub\protobuf\protobuf\person_protobufs_cs.csproj]

[KerstinKeller]

The c# files are already generated as part of the build. Which proto version did you upgrade to?
I previously tested with 3.21.x (think x=12)..

[FlorianReimold]

Yep, that's the exact version that I used, too. git describe tells me it is v21.12

[KerstinKeller]

Now that I think about it, it is more likely that something with restoring the nuget protobuf references is going wrong now.
In the PR that introduced the protobuf related changes, also the method for restoring nuget references was changed. A more modern cmake approach was used.
However this has been in place for a while, so I wonder why it's not working in combination with the newer protobuf version.

[FlorianReimold]

OK, I managed to fix the build. It turns out that there are numerous hardcoded protobuf versions that one need to change manually and that need to match the version that is used in the build. I missed one of them.

[FlorianReimold]

So it compiles now, but at least executing the eCAL Samples fails with the following error mesage. The reason is that we have pre-generated pb2.py files checked in. The files are generated with protoc 3.0, if I recall correctly. I think the only proper way is to generate the pb2.py files at build time and not push them to the repo.

C:\eCAL\samples\src\python\pubsub\protobuf\person_snd>python person_snd.py
Traceback (most recent call last):
  File "C:\eCAL\samples\src\python\pubsub\protobuf\person_snd\person_snd.py", line 27, in <module>
    import person_pb2
  File "C:\eCAL\samples\src\python\pubsub\protobuf\person_snd\../_protobuf\person_pb2.py", line 16, in <module>
    import animal_pb2 as animal__pb2
  File "C:\eCAL\samples\src\python\pubsub\protobuf\person_snd\../_protobuf\animal_pb2.py", line 36, in <module>
    _descriptor.FieldDescriptor(
  File "C:\Users\Florian\AppData\Local\Programs\Python\Python39\lib\site-packages\google\protobuf\descriptor.py", line 561, in __new__
    _message.Message._CheckCalledFromGeneratedFile()
TypeError: Descriptors cannot not be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
 1. Downgrade the protobuf package to 3.20.x or lower.
 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates

---

Edit: after re-generating the pb2.py files with protoc.exe that came from the eCAL Installer that I tested, everything worked fine.

[FlorianReimold]

Possible solutions:

1. Remove generated files from repo and generate pb2.py files while building

• package with installer
• package with whl / python3-ecal5.deb
• Don't package at all

2. Don't care for now and let the user re-generate the files

---

We decided to leave the pb2.py files as they are at the moment. The user can re-generate them if required.
#1161 should find a better way to re-generate the files.

[DownerCase]

Working on some packaging work for eCAL in preparation of 6.0 and the C++ protobuf API has moved to v6.30 which drops C++14 support, and now requires C++17. According to the documentation it means there is only until the end of the year before the 5.x series is unsupported. https://protobuf.dev/support/version-support/#cpp

The main effect is that some instances of const std::string& are now std::string_view which breaks the protobuf helpers. Not a major problem but a good excuse to finally make use of C++17 😉