continue prost

[jokemanfire]

work continue with #173

Closes #258

[Tim-Zhang]

@jokemanfire I appreciate you taking on this work very much.
Why don't we directly reopen #173 and continue the work?
Just let me know if there are any difficulties.

Thanks agian.

[jokemanfire]

@jokemanfire I appreciate you taking on this work very much. Why don't we directly reopen #173 and continue the work? Just let me know if there are any difficulties.

Thanks agian.

There are some conflicts and API changes that need to be resolved and some code needs to be packaged to adapt to the current API. It's not easy to operate with the previous version of PR. I need to add some commit to complete this feature. I have communicated with the original author.

[jokemanfire]

@Tim-Zhang @justxuewei Local test all pass but i don't know how to install protoc in CI.

[Tim-Zhang]

@jokemanfire FYI
https://github.com/marketplace/actions/setup-protoc
https://www.andreamedda.com/posts/go-buf-github-actions/

[jokemanfire]

Basically enough @Tim-Zhang @justxuewei just take a look . :)

[justxuewei]

Nice work, thanks @jokemanfire! Could you cleanup your commits? It makes us easier to review and maintain.

1. Use git rebase instead of git merge.
2. Modify the commit title to {subsystem}: {brief title for your changes}, and give it a brief introduction in the commit message.
3. Split your commits, and don't mix bugfixes, refactoring and new features.
4. Please modify this PR title to something like "codegen: Introduce a new codegen using prost".

Thanks!

[jokemanfire]

Nice work, thanks @jokemanfire! Could you cleanup your commits? It makes us easier to review and maintain.好的工作，谢谢！你能清理一下你的提交吗？这让我们更容易审查和维护。

1. Use git rebase instead of git merge.使用 git rebase 代替 git merge 。
2. Modify the commit title to {subsystem}: {brief title for your changes}, and give it a brief introduction in the commit message.修改提交标题为 {subsystem}: {brief title for your changes} ，并在提交信息中简要介绍。
3. Split your commits, and don't mix bugfixes, refactoring and new features.拆分您的提交，不要混合修复错误、重构和新功能。
4. Please modify this PR title to something like "codegen: Introduce a new codegen using prost".请将此 PR 标题修改为类似于“codegen：引入使用 prost 的新代码生成器”的标题。

Thanks!  谢谢！

I have changed it.

[justxuewei]

Once this feature gets merged, I prefer to bump a major version to 2 to avoid some compatibility issues, as it nearly changes the entire codegen engine. WDYT? @jokemanfire @Tim-Zhang

[jokemanfire]

Once this feature gets merged, I prefer to bump a major version to 2 to avoid some compatibility issues, as it nearly changes the entire codegen engine. WDYT? 一旦这个功能合并，我更倾向于将主版本号提升到 2，以避免一些兼容性问题，因为它几乎改变了整个代码生成引擎。你怎么看？@jokemanfire @Tim-Zhang

This modification is indeed quite significant, and I agree. It depends on the maintainer's thoughts

[jokemanfire]

I tried to minimize conditional compilation as much as possible @Tim-Zhang @justxuewei

[jokemanfire]

@Tim-Zhang Can you take some time to advance this feature? In the previous puncture, it was found that the prost library uses less memory during operation than the Rust Protobuf library

[jokemanfire]

@jokemanfire I have left a few review comments, and btw please resolve the conflicts, thanks.

thanks , I will take some time to resolve it.

[Copilot]

Pull request overview

This pull request continues the migration path toward supporting tokio-rs/prost alongside the existing rust-protobuf backend, adding a new standalone prost-based codegen crate plus a new example2 tree to exercise the prost workflow, and updating CI/build plumbing to install protoc and test both backends.

Changes:

• Add prost + rustprotobuf mutually-exclusive feature support in the ttrpc crate, with conditional encode/decode paths across sync/async runtimes.
• Introduce a new standalone codegen/ crate (prost-build based) and a new example2/ project that uses it.
• Update CI and build scripts (including a new install_protoc.sh) to ensure protoc is available and both backends build/test appropriately.

Reviewed changes

Copilot reviewed 54 out of 55 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
ttrpc-codegen/Makefile	Add local check/test targets for the legacy codegen crate.
compiler/Makefile	Add local check/test targets for the compiler crate.
tests/run-examples.rs	Run examples conditionally for rustprotobuf vs prost and add per-example working directory support.
src/ttrpc.proto	Change proto package to ttrpc (aligns with prost codegen expectations).
src/sync/utils.rs	Add prost-specific request/response helpers and macro variants.
src/sync/server.rs	Add prost decode/encode handling in sync server request path.
src/sync/mod.rs	Export utils as public module (macros/helpers used externally).
src/sync/client.rs	Add prost-specific oversize/status handling in sync client.
src/proto.rs	Add Codec impl for prost messages and gate protobuf stream codecs behind feature flags.
src/lib.rs	Re-export sync helpers and enforce mutual exclusivity of prost/rustprotobuf.
src/error.rs	Make Error -> Response and get_status work for both backends.
src/context.rs	Construct metadata KeyValue values compatibly for both backends.
src/asynchronous/utils.rs	Add prost variants of async handler/client macros and gate existing ones.
src/asynchronous/server.rs	Adjust async server response/status paths for prost message layout/size checks.
src/asynchronous/client.rs	Make async client request conversion/status handling work for both backends.
README.md	Document how to use the prost backend and its differences.
Makefile	Split CI checks/tests to run rustprotobuf and prost configurations (and skip prost where needed on Windows).
install_protoc.sh	Add helper to install protoc across GitHub Actions runner OSes.
example2/utils.rs	Add shared utilities for the prost-based example project.
example2/server.rs	Add sync prost example server.
example2/protocols/sync/mod.rs	Include generated sync prost protocol modules.
example2/protocols/protos/streaming.proto	Add streaming proto schema for example2.
example2/protocols/protos/oci.proto	Add OCI proto schema for example2.
example2/protocols/protos/health.proto	Add health proto schema for example2.
example2/protocols/protos/google/protobuf/test_empty.proto	Add test Empty message for example2 protos.
example2/protocols/protos/google/protobuf/empty.proto	Vendor protobuf well-known Empty type for example2 compilation.
example2/protocols/protos/google/protobuf/descriptor.proto	Vendor protobuf descriptor type for example2 compilation.
example2/protocols/protos/github.com/kata-containers/agent/pkg/types/types.proto	Vendor kata agent types proto used by example2.
example2/protocols/protos/github.com/gogo/protobuf/gogoproto/gogo.proto	Vendor gogo proto options used by example2 schemas.
example2/protocols/protos/agent.proto	Vendor kata agent service proto used by example2.
example2/protocols/mod.rs	Define example2 protocol module layout.
example2/protocols/hack/update-generated-proto.sh	Add script for updating/regenerating vendored protos + rust output.
example2/protocols/asynchronous/mod.rs	Include generated async prost protocol modules.
example2/Makefile	Add build/build-examples/deps targets (including protoc install).
example2/client.rs	Add sync prost example client.
example2/Cargo.toml	Define example2 crate dependencies and examples (prost + ttrpc prost feature).
example2/build.rs	Generate sync/async prost services via new codegen crate.
example2/async_stream_server.rs	Add async streaming server example for prost.
example2/async_stream_client.rs	Add async streaming client example for prost.
example2/async_server.rs	Add async unary server example for prost.
example2/async_client.rs	Add async unary client example for prost.
example/protocols/sync/mod.rs	Adjust example protocol module exports for legacy layout.
example/protocols/asynchronous/mod.rs	Adjust example async protocol module exports for legacy layout.
example/Cargo.toml	Ensure example explicitly enables rustprotobuf backend feature.
codegen/src/util.rs	Add identifier casing/type-path helper utilities for prost codegen.
codegen/src/svcgen.rs	Implement prost-build ServiceGenerator for ttrpc services/clients (sync+async+streaming).
codegen/src/lib.rs	Export the codegen API surface.
codegen/src/codegen.rs	Implement codegen builder + header injection using prost-build and fd_set introspection.
codegen/README.md	Document how to use the new prost-based codegen crate.
codegen/Makefile	Add build/test/check/deps targets for codegen crate.
codegen/Cargo.toml	Define new standalone ttrpc-codegen (prost-based) crate.
Cargo.toml	Add optional prost deps, define prost vs rustprotobuf features, and exclude new standalone crates from workspace.
build.rs	Generate ttrpc protos via protobuf-codegen or prost-build depending on selected backend, with prost enum variant rewriting.
.gitignore	Ignore generated protocol Rust files while keeping mod.rs tracked for both example trees.
.github/workflows/bvt.yml	Install protoc in CI and build example2 as part of the build job.

Suppressed comments (2)

src/sync/utils.rs:182

• In the prost client macro, merge is called with an immutable slice. prost::Message::merge needs &mut input (e.g. &mut &[u8]), otherwise this won’t compile.
  src/asynchronous/utils.rs:317
• In the prost async client macro, merge is called with an immutable slice. prost::Message::merge requires &mut input (e.g. &mut &[u8]), otherwise it won’t compile.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 54 out of 55 changed files in this pull request and generated 2 comments.

Suppressed comments (10)

src/sync/utils.rs:113

• With the prost backend, prost::Message::merge expects a mutable Buf (e.g. &mut &[u8]). Passing an immutable &[u8] won’t type-check, so this macro expansion will fail to compile under --features prost.
  src/sync/utils.rs:182
• In the prost client_request! macro, prost::Message::merge requires a mutable Buf (e.g. &mut &[u8]). The current merge(&res.payload as &[u8]) will not compile with prost 0.13.
  src/asynchronous/utils.rs:317
• In the prost async_client_request! macro, prost::Message::merge requires a mutable Buf (e.g. &mut &[u8]). The current merge(&res.payload as &[u8]) won’t type-check with prost 0.13.
  tests/run-examples.rs:126
• In the prost test block, the sync server/client example is commented out, so CI no longer exercises the basic sync example under the prost backend even though example2 provides it.
  src/sync/client.rs:166
• For the prost backend, the oversize check currently only considers req.payload.len(), but the encoded request size also includes service, method, metadata, etc. This can bypass the max message size check with large metadata. Use the encoded size instead (via the existing Codec::size() implementation).
  src/sync/utils.rs:58
• The prost response_to_channel variant doesn’t enforce check_oversize like the rustprotobuf path does. That can send oversized responses and violate the library’s message size limits. Mirror the existing oversize handling before sending.

This issue also appears in the following locations of the same file:

• line 110
• line 179
  compiler/Makefile:8

• make -C compiler (as invoked by .github/workflows/bvt.yml) now defaults to running only the check target, which means the build job no longer actually builds/tests this crate by default. Add an explicit all default target (or restore the previous include-based targets) so CI and local make behave as expected.

.PHONY: check
check:
	cargo fmt --all -- --check
	cargo clippy --all-targets -- -D warnings

ttrpc-codegen/Makefile:8

• make -C ttrpc-codegen (as invoked by .github/workflows/bvt.yml) now defaults to running only the check target, so the build job no longer builds/tests this crate by default. Add an all default target so make continues to validate both linting and tests.
  Makefile:3
• PROTOC is introduced but not used anywhere in the Makefile (no $(PROTOC) references). This adds confusion for maintainers; either wire it into the relevant targets or remove it.

PROTOC ?= $(shell which protoc 2>/dev/null || echo $(HOME)/protoc/bin/protoc)

all: debug test

src/error.rs:17

• use crate::proto::{self, ...} imports proto::self but it’s never referenced, and the surrounding #[allow(unused_imports)] masks that. This can hide real unused-import issues; remove the unused import and the allow.

[Tim-Zhang]

Thanks for the continued updates. I hope we can stay in close communication and get this PR merged within the next two weeks.

The latest revision fixes several earlier issues, and the current checks are green. I also verified locally that the codegen tests and the main library tests for both protobuf backends pass.

The main remaining items are:

1. Preserve the existing ttrpc-codegen fluent API and add .prost() for backend selection instead of requiring users to migrate to CodegenBuilder.
2. Avoid unnecessary generated API differences, including Arc<Box<dyn Trait>>, backend-dependent macro signatures, and newly exposed internal helpers.
3. Keep the original protobuf package declarations. Rust module layout differences should be handled in the build or module glue rather than by changing protocol identities.
4. Keep the standalone Prost example, but rename example2 to example-prost and reuse the existing proto schemas. Removing the duplicated proto directory and update script would eliminate 2,252 of the 3,238 added example lines.
5. Make the example tests validate RPC results and execute the sync example in CI.
6. Make the codegen dependencies backend-specific, fix package-less service names, enforce size limits against complete encoded messages, and run the codegen crate tests explicitly in CI.

Acronym casing differences such as OnlineCPUMemRequest and OnlineCpuMemRequest are determined by the respective generators. I am not requesting automatic compatibility aliases for these types in this PR.

The PR also currently conflicts with the target branch and will need to be updated before merging.

[Tim-Zhang]

@jokemanfire Thanks for the updates. One last codegen issue:

[Tim-Zhang]

Multiple proto files can share the same package and therefore the same output file. Here, health.proto, agent.proto, and oci.proto all map to grpc.rs, so the generated file starts with the same header three times:

// This file is generated by ttrpc-codegen 1.0.0. Do not edit
// @generated

// This file is generated by ttrpc-codegen 1.0.0. Do not edit
// @generated

// This file is generated by ttrpc-codegen 1.0.0. Do not edit
// @generated

Please deduplicate rs_path before rewriting the file.