Handling of UTF8 command argument files regressed in v26.1 breaking Grpc.Tools

[tonydnewell]

What version of protobuf and what language are you using?
Version: v26.1
Language: C# (but any will see problem)

What operating system (Linux, Windows, ...) and version?
Windows - any version

What did you do?
This is causing the Grpc.Tools issue grpc/grpc#36518
Grpc.Tools is broken by the protocol buffers change.

Handling of non-ASCII characters in UTF-8 command line argument files is broken. It was working in v25.1 and is not working in V26.1. I suspect the cause of the issue to be the "fix" in #14197

Grpc.Tools V2.62.0 ships with protoc version 25.1
Grpc.Tools V2.63.0 ships with protoc version 26.1

Steps to reproduce:

• Create a directory on Windows containing a non-ASCII character in its name
• Use that directory in the a command line file (an @ file)

Example of @ file - test.rps

--plugin=protoc-gen-grpc=E:\work\grpc\protoctest\test-Dré\tools\grpc_csharp_plugin.exe
--grpc_out=E:\work\grpc\protoctest\test-Dré\grpc_out
--proto_path=E:\work\grpc\protoctest\test-Dré
hélloworld.proto

• example command line on Windows:

E:\work\grpc\protoctest\test-Dré>v1.63.0\protoc.exe @..\test-Dré\test.rsp

What did you expect to see
No errors

What did you see instead?

E:\work\grpc\protoctest\test-Dré>v1.63.0\protoc.exe @..\test-Dré\test.rsp
Failed to open argument file: ..\test-Dr�\test.rsp

** Results with older version (working):**

E:\work\grpc\protoctest\test-Dré>v1.62.0\protoc.exe @..\test-Dré\test.rsp

[acozzette]

Thank you for the bug report. Do you have any suggestions on how to fix it? It sounds like #14197 was supposed to fix this problem, so that is surprising if it actually is the root cause of the problem.

[slonopotamus]

It isn't clear what issue #14197 is fixing at all. Grpc.Tools V2.62.0 works perfectly for us when path contains non-ascii symbols, while V2.63.0 breaks. Maybe just git revert?

[tonydnewell]

Without trying to reproduce or examine the original issue that #14197 is trying to fix I'm not sure why this fix was needed.

The original code treated arguments on Windows (both command line and those read from a file) as just a string of UTF-8 bytes and thus no wchar needed for non-ascii characters. The original problem may just have been environmental - not using the correct Windows codepage when running the command (UTF-8 is codepage 65001).

My suggestion would be to revert the fix and then re-investigate the original issue.

[TopSwagCode]

I wish my parents naming me, could have been simple. But they ended up with "Joshua Jesper Krægpøth Ryder". This has fucked me several times. Just came back from vacation to find out my Dotnet work projects with GRPC / Proto files no longer build:

C:\Users\JoshuaJesperKrægpøth.nuget\packages\grpc.tools\2.64.0\tools\windows_x64\protoc.exe --csharp_out=obj\Debug\net8.0 --proto_path=C:\Users\JoshuaJesperKrægpøth.nuget\packages\grpc.tools\2.64.0\build\native\include --proto_path=Protos --dependency_out=obj\Debug\net8.0\12fbc7ffe642ca7e_common.protodep --error_format=msvs Protos/common.proto
0>Failed to open argument file: Error : C:\Users\JoshuaJesperKrµgp°th\AppData\Local\Temp\MSBuildTemp\tmpd22c16c07e134b838bcbee401f0f1fc2.rsp

Here's the kicker. I don't get to pick my windows name, since it's a work laptop and is predefined from sysadmins..... I am so happy they chose to pick Krægpøth instead of Ryder.....

Long rant aside, is there a estimate when this will be fixed ? :D

[acozzette]

This should hopefully be fixed now with #17581.