|
9 | 9 | [](https://github.com/bufbuild/homebrew-buf) |
10 | 10 | [][badges_slack] |
11 | 11 |
|
12 | | -<a name="features"></a> |
13 | | -The [`buf`][buf] CLI is the best tool for working with [Protocol Buffers][protobuf]. It provides: |
14 | | - |
15 | | -- A [linter][lint_usage] that enforces good API design choices and structure. |
16 | | -- A [breaking change detector][breaking_tutorial] that enforces compatibility at the source code or wire level. |
17 | | -- A [generator][generate_usage] that invokes your plugins based on [configuration files][templates]. |
18 | | -- A [formatter][format_usage] that formats your Protobuf files in accordance with industry standards. |
19 | | -- Integration with the [Buf Schema Registry][bsr], including full dependency management. |
| 12 | +Buf is the modern toolchain for [Protobuf][protobuf]. It replaces day-to-day `protoc` use with a fast compiler, module-aware workspaces, formatting, linting, breaking-change detection, code generation, dependency management, API calls, and a client for the [Buf Schema Registry][bsr]. |
20 | 13 |
|
21 | | -## Installation |
| 14 | +If you are still driving Protobuf with shell scripts around `protoc -I ...`, Buf is the upgrade you want: the same schema language, the same generated-code plugin model, fewer moving parts, and a direct path from local `.proto` files to governed, versioned APIs. |
22 | 15 |
|
23 | | -### Homebrew |
| 16 | +## Start |
24 | 17 |
|
25 | | -You can install `buf` using [Homebrew][brew] (macOS or Linux): |
| 18 | +Install `buf` with [Homebrew][brew]: |
26 | 19 |
|
27 | 20 | ```sh |
28 | 21 | brew install bufbuild/buf/buf |
29 | 22 | ``` |
30 | 23 |
|
31 | | -This installs: |
| 24 | +Initialize a workspace and run the checks you should expect every Protobuf repository to pass: |
| 25 | + |
| 26 | +```sh |
| 27 | +buf config init |
| 28 | +buf build |
| 29 | +buf format -w |
| 30 | +buf lint |
| 31 | +buf breaking --against '.git#branch=main' |
| 32 | +``` |
32 | 33 |
|
33 | | -- The `buf`, [`protoc-gen-buf-breaking`][breaking], and [`protoc-gen-buf-lint`][lint] binaries |
34 | | -- Shell completion scripts for [Bash], [Fish], [Powershell], and [zsh] |
| 34 | +Generate code from a checked-in `buf.gen.yaml` instead of a hand-maintained shell command: |
35 | 35 |
|
36 | | -### Other methods |
| 36 | +```sh |
| 37 | +buf generate |
| 38 | +``` |
37 | 39 |
|
38 | | -For other installation methods, see our [official documentation][install], which covers: |
| 40 | +For a guided walkthrough from an empty workspace to a working Connect service, run the [Buf CLI quickstart][cli_quickstart]. |
39 | 41 |
|
40 | | -- Installing `buf` via [npm] |
41 | | -- Installing `buf` on [Windows] |
42 | | -- Using `buf` as a [Docker image][docker] |
43 | | -- Installing as a [binary], from a [tarball], and from [source] through [GitHub Releases][releases] |
44 | | -- [Verifying] releases using a [minisign] public key |
| 42 | +## Why Buf wins |
45 | 43 |
|
| 44 | +<a name="features"></a> |
46 | 45 |
|
47 | | -## Usage |
| 46 | +| Protobuf work | With `protoc` and scripts | With Buf | |
| 47 | +| --- | --- | --- | |
| 48 | +| Finding files | Maintain `-I` paths and hope import order does not change behavior. | Declare modules once in `buf.yaml`; Buf discovers files and rejects ambiguous imports. | |
| 49 | +| Compiling | Manage a local `protoc` install and parse changing stderr output. | Use Buf's internal compiler, tested against `protoc` descriptor output and built for deterministic parallel compilation. | |
| 50 | +| Style | Rely on review comments or separate tooling. | Run `buf lint` locally, in editors, in CI, and on the BSR with 40+ built-in rules plus custom plugins. | |
| 51 | +| Compatibility | Find breakage after generated code fails, clients fail, or serialized data becomes unreadable. | Run `buf breaking` against Git, a BSR module, a tarball, a zip file, or a Buf image before merge. | |
| 52 | +| Code generation | Keep plugin binaries installed on every machine and encode behavior in long commands. | Put plugins, outputs, options, inputs, and managed-mode settings in `buf.gen.yaml`; use local or remote plugins. | |
| 53 | +| Dependencies | Copy `.proto` files between repositories or vendor them by hand. | Declare BSR module dependencies in `buf.yaml` and pin them in `buf.lock`. | |
| 54 | +| API consumers | Send people your schemas and generation instructions. | Publish to the BSR and let consumers install generated SDKs with `go get`, `npm install`, Maven, Gradle, `pip install`, NuGet, Cargo, SwiftPM, CMake, or an archive. | |
| 55 | +| Governance | Reimplement checks in every repository and hope every team keeps them enabled. | Enforce breaking-change, uniqueness, and custom policies at the BSR layer. | |
| 56 | + |
| 57 | +Core CLI features work without a BSR account. Signing in to the registry adds distribution, remote plugins, generated SDKs, hosted docs, dependency resolution for private modules, and server-side checks when you need them. |
| 58 | + |
| 59 | +## Core workflow |
| 60 | + |
| 61 | +Buf treats a directory tree of `.proto` files as a module, and a project as a workspace. A small `buf.yaml` is enough to make build, lint, breaking-change detection, generation, dependency resolution, and publishing agree on the same input. |
| 62 | + |
| 63 | +```yaml |
| 64 | +version: v2 |
| 65 | +modules: |
| 66 | + - path: proto |
| 67 | +lint: |
| 68 | + use: |
| 69 | + - STANDARD |
| 70 | +breaking: |
| 71 | + use: |
| 72 | + - FILE |
| 73 | +``` |
48 | 74 |
|
49 | | -Buf's help interface provides summaries for commands and flags: |
| 75 | +From there, the useful commands are deliberately boring: |
50 | 76 |
|
51 | 77 | ```sh |
52 | | -buf --help |
| 78 | +buf build |
| 79 | +buf format -w |
| 80 | +buf lint |
| 81 | +buf breaking --against '.git#branch=main' |
| 82 | +buf generate |
| 83 | +buf push |
53 | 84 | ``` |
54 | 85 |
|
55 | | -For more comprehensive usage information, consult Buf's [documentation][docs], especially these guides: |
| 86 | +`buf build` compiles the workspace. `buf lint` catches API-shape problems while the author is still editing. `buf breaking` compares the current schema against a previous version and flags source, JSON, or wire-format incompatibilities. `buf generate` runs `protoc` plugins from a checked-in template. `buf push` publishes named modules to the BSR. |
| 87 | + |
| 88 | +## Code generation |
| 89 | + |
| 90 | +`buf generate` is compatible with the normal `protoc` plugin model, but it moves generation into versioned configuration. This example generates Go Protobuf types and ConnectRPC handlers from `proto/` using remote plugins hosted on the BSR: |
| 91 | + |
| 92 | +```yaml |
| 93 | +version: v2 |
| 94 | +clean: true |
| 95 | +managed: |
| 96 | + enabled: true |
| 97 | + override: |
| 98 | + - file_option: go_package_prefix |
| 99 | + value: github.com/acme/weather/gen/go |
| 100 | +plugins: |
| 101 | + - remote: buf.build/protocolbuffers/go |
| 102 | + out: gen/go |
| 103 | + opt: paths=source_relative |
| 104 | + - remote: buf.build/connectrpc/gosimple |
| 105 | + out: gen/go |
| 106 | + opt: |
| 107 | + - paths=source_relative |
| 108 | + - simple |
| 109 | +inputs: |
| 110 | + - directory: proto |
| 111 | +``` |
56 | 112 |
|
57 | | -* [`buf breaking`][breaking_tutorial] |
58 | | -* [`buf build`][build_usage] |
59 | | -* [`buf generate`][generate_usage] |
60 | | -* [`buf lint`][lint_usage] |
61 | | -* [`buf format`][format_usage] |
62 | | -* [`buf registry`][bsr_quickstart] (for using the [BSR]) |
| 113 | +Remote plugins remove the need to install and maintain generator binaries on every developer machine or CI runner. Managed mode lets API producers keep language-specific file options out of `.proto` files while consumers still get correct generated package names for their target language. |
63 | 114 |
|
64 | | -## CLI breaking change policy |
| 115 | +Local plugins work too. If a plugin speaks the standard Protobuf plugin protocol, Buf can run it. |
65 | 116 |
|
66 | | -We will never make breaking changes within a given major version of the CLI. After `buf` reached v1.0, you can expect no breaking changes until v2.0. But as we have no plans to ever release a v2.0, we will likely never break the `buf` CLI. |
| 117 | +## Breaking changes |
67 | 118 |
|
68 | | -> This breaking change policy does _not_ apply to commands behind the `buf beta` gate, and you should expect breaking changes to commands like `buf beta registry`. The policy does go into effect, however, when those commands or flags are elevated out of beta. |
| 119 | +Protobuf compatibility is not one thing. Renaming a field can break generated source code while preserving the binary wire format; changing a field from `int32` to `string` breaks every existing serialized message. `buf breaking` makes that distinction explicit with rule categories for `FILE`, `PACKAGE`, `WIRE_JSON`, and `WIRE` compatibility. |
69 | 120 |
|
70 | | -## Our goals for Protobuf |
| 121 | +```sh |
| 122 | +buf breaking --against '.git#branch=main' |
| 123 | +``` |
71 | 124 |
|
72 | | -[Buf]'s goal is to replace the current paradigm of API development, centered around REST/JSON, with a **schema-driven** paradigm. Defining APIs using an [IDL] provides numerous benefits over REST/JSON, and [Protobuf] is by far the most stable and widely adopted IDL in the industry. We've chosen to build on this widely trusted foundation rather than creating a new IDL from scratch. |
| 125 | +`--against` accepts a Git branch, a BSR module, a tarball, a zip file, a local directory, or a prebuilt Buf image. That matters in real repositories: the same command works on a laptop, in CI, and in release automation. |
73 | 126 |
|
74 | | -But despite its technical merits, actually _using_ Protobuf has long been more challenging than it needs to be. The Buf CLI and the [BSR](#the-buf-schema-registry) are the cornerstones of our effort to change that for good and to make Protobuf reliable and easy to use for service owners and clients alike—in other words, to create a **modern Protobuf ecosystem**. |
| 127 | +## Buf Schema Registry |
75 | 128 |
|
76 | | -While we intend to incrementally improve on the `buf` CLI and the [BSR](#the-buf-schema-registry), we're confident that the basic groundwork for such an ecosystem is _already_ in place. |
| 129 | +<a name="the-buf-schema-registry"></a> |
77 | 130 |
|
78 | | -## The Buf Schema Registry |
| 131 | +[Buf Schema Registry][bsr] is a Protobuf-aware registry. It stores modules, verifies they compile, renders documentation, resolves dependencies, hosts remote plugins, produces generated SDKs, and can enforce schema checks before a breaking change reaches consumers. |
79 | 132 |
|
80 | | -The [Buf Schema Registry][bsr] (BSR) is a SaaS platform for managing your Protobuf APIs. It provides a centralized registry and a single source of truth for all of your Protobuf assets, including not just your `.proto` files but also [remote plugins][bsr_plugins]. Although the BSR provides an intuitive browser UI, `buf` enables you to perform most BSR-related tasks from the command line, such as [pushing] Protobuf sources to the registry and managing [users] and [repositories]. |
| 133 | +```sh |
| 134 | +buf push |
| 135 | +``` |
81 | 136 |
|
82 | | -> The BSR is not required to use `buf`. We've made the core [features] of the `buf` CLI available to _all_ Protobuf users. |
| 137 | +Pushing a module to the BSR gives your organization a source of truth for Protobuf APIs. Consumers can depend on the schema as a BSR module, install generated SDKs from their normal package manager, or use the BSR docs to inspect services, messages, fields, enums, references, and historical commits. |
83 | 138 |
|
84 | | -## More advanced CLI features |
| 139 | +## Related projects |
85 | 140 |
|
86 | | -While `buf`'s [core features][features] should cover most use cases, we've included some more advanced features to cover edge cases: |
| 141 | +Buf is most useful when schemas drive more than code generation. [ConnectRPC][connectrpc] uses Protobuf schemas to build simple HTTP APIs that support Connect, gRPC, and gRPC-Web without separate service definitions. [Protobuf-ES][protobuf_es] gives JavaScript and TypeScript users a modern Protobuf runtime and generator. [Protovalidate][protovalidate] puts validation rules in the schema and runs them consistently across languages. |
87 | 142 |
|
88 | | -* **Automatic file discovery**. Buf walks your file tree and builds your `.proto` files in accordance with your supplied [build configuration][build_config], which means that you no longer need to manually specify `--proto_paths`. You can still, however, specify `.proto` files manually through CLI flags in cases where file discovery needs to be disabled. |
89 | | -* **Fine-grained rule configuration** for [linting][lint_rules] and [breaking changes][breaking_rules]. While we do have recommended defaults, you can always select the exact set of rules that your use case requires, with [40 lint rules][lint_rules] and [53 breaking change rules][breaking_rules] available. |
90 | | -* **Configurable error formats** for CLI output. `buf` outputs information in `file:line:column:message` form by default for each lint error and breaking change it encounters, but you can also select JSON, MSVS, JUnit, and Github Actions output. |
91 | | -* **Editor integration** driven by `buf`'s granular error output. We currently provide linting integrations for both [Vim and Visual Studio Code][ide] and [JetBrains IDEs][jetbrains] like IntelliJ and GoLand, but we plan to support other editors such as Emacs in the future. |
92 | | -* **Universal Input targeting**. Buf enables you to perform actions like linting and breaking change detection not just against local `.proto` files but also against a broad range of other [Inputs], such as tarballs and ZIP files, remote Git repositories, and pre-built [image][images] files. |
93 | | -* **Speed**. Buf's internal Protobuf [compiler] compiles your Protobuf sources using all available cores without compromising deterministic output, which is considerably faster than `protoc`. This allows for near-instantaneous feedback, which is of special importance for features like [editor integration][ide]. |
| 143 | +One contract should drive the whole workflow: compile, lint, compatibility checks, generated clients and servers, validation, API calls, package publishing, and governed changes. |
94 | 144 |
|
95 | | -## Next steps |
| 145 | +## Installation |
96 | 146 |
|
97 | | -Once you've installed `buf`, we recommend completing the [CLI tutorial][cli-tutorial], which provides a broad but hands-on overview of the core functionality of the CLI. The tour takes about 10 minutes to complete. |
| 147 | +Homebrew installs the `buf`, [`protoc-gen-buf-breaking`][breaking_plugin], and [`protoc-gen-buf-lint`][lint_plugin] binaries, plus shell completion scripts for [Bash], [Fish], [PowerShell], and [zsh]. |
98 | 148 |
|
99 | | -After completing the tour, check out the remainder of the [docs] for your specific areas of interest. |
| 149 | +```sh |
| 150 | +brew install bufbuild/buf/buf |
| 151 | +``` |
100 | 152 |
|
101 | | -## Builds |
| 153 | +Other supported installation methods include [npm], [Windows], [Docker], [binary downloads], [tarballs], [source builds], and [minisign verification][verifying]. See the [installation docs][install] for the full list. |
102 | 154 |
|
103 | | -The following is a breakdown of the binaries by CPU architecture and operating system available through our [releases]: |
| 155 | +## CLI stability |
104 | 156 |
|
105 | | -| | Linux | MacOS | Windows | OpenBSD | FreeBSD | |
106 | | -| --- | --- | --- | --- | --- | --- | |
107 | | -| x86 (64-bit) | ✅ | ✅ | ✅ | ✅ | ✅ | |
108 | | -| ARM (64-bit) | ✅ | ✅ | ✅ | ✅ | ✅ | |
109 | | -| ARMv7 (32-bit) | ✅ | ❌ | ❌ | ❌ | ❌ | |
110 | | -| RISC-V (64-bit) | ✅ | ❌ | ❌ | ❌ | ❌ | |
111 | | -| ppc64le | ✅ | ❌ | ❌ | ❌ | ❌ | |
112 | | -| s390x | ✅ | ❌ | ❌ | ❌ | ❌ | |
| 157 | +Buf CLI releases do not make breaking changes within a major version. Since `buf` reached v1.0, you can expect no breaking changes until v2.0. We have no plans to release v2.0. |
113 | 158 |
|
114 | | -## Community |
| 159 | +This policy does not apply to commands behind the `buf beta` gate. Expect breaking changes for beta commands until they are promoted. |
| 160 | + |
| 161 | +## Documentation |
115 | 162 |
|
116 | | -For help and discussion around Protobuf, best practices, and more, join us on [Slack][badges_slack]. |
| 163 | +- [Buf CLI][cli] |
| 164 | +- [CLI quickstart][cli_quickstart] |
| 165 | +- [Modules and workspaces][modules_workspaces] |
| 166 | +- [Code generation][generate] |
| 167 | +- [Linting][lint] |
| 168 | +- [Breaking-change detection][breaking] |
| 169 | +- [Formatting][format] |
| 170 | +- [Calling APIs with `buf curl`][curl] |
| 171 | +- [Buf Schema Registry][bsr] |
| 172 | +- [Generated SDKs][generated_sdks] |
| 173 | +- [Remote plugins][remote_plugins] |
| 174 | +- [Schema checks][schema_checks] |
| 175 | +- [Migrating from `protoc`][migrate_from_protoc] |
| 176 | + |
| 177 | +## Community |
117 | 178 |
|
118 | | -For updates on the Buf CLI, [follow this repo on GitHub][repo]. |
| 179 | +For help and discussion around Protobuf, best practices, and Buf, join us on [Slack][badges_slack]. |
119 | 180 |
|
120 | | -For feature requests, bugs, or technical questions, email us at [dev@buf.build][email_dev]. For general inquiries or inclusion in our upcoming feature betas, email us at [info@buf.build][email_info]. |
| 181 | +For bugs, feature requests, and technical questions, open an issue in this repository or email [dev@buf.build][email_dev]. For general inquiries, email [info@buf.build][email_info]. |
121 | 182 |
|
122 | 183 | [badges_slack]: https://buf.build/links/slack |
123 | 184 | [bash]: https://www.gnu.org/software/bash |
124 | | -[binary]: https://buf.build/docs/cli/installation/#source |
125 | | -[breaking]: https://buf.build/docs/breaking/overview/ |
126 | | -[breaking_rules]: https://buf.build/docs/breaking/rules/ |
127 | | -[breaking_tutorial]: https://buf.build/docs/breaking/tutorial/ |
| 185 | +[binary downloads]: https://buf.build/docs/cli/installation/#github |
| 186 | +[breaking]: https://buf.build/docs/breaking/ |
| 187 | +[breaking_plugin]: https://buf.build/docs/breaking/ |
128 | 188 | [brew]: https://brew.sh |
129 | 189 | [bsr]: https://buf.build/docs/bsr/ |
130 | | -[bsr_plugins]: https://buf.build/plugins |
131 | | -[bsr_quickstart]: https://buf.build/docs/bsr/quickstart/ |
132 | | -[buf]: https://buf.build |
133 | | -[build_config]: https://buf.build/docs/build/usage/#key-concepts |
134 | | -[build_usage]: https://buf.build/docs/build/usage |
135 | | -[cli-tutorial]: https://buf.build/docs/cli/quickstart/ |
136 | | -[compiler]: https://buf.build/docs/reference/internal-compiler/ |
| 190 | +[cli]: https://buf.build/docs/cli/ |
| 191 | +[cli_quickstart]: https://buf.build/docs/cli/quickstart/ |
| 192 | +[connectrpc]: https://connectrpc.com |
| 193 | +[curl]: https://buf.build/docs/curl/ |
137 | 194 | [docker]: https://buf.build/docs/cli/installation/#docker |
138 | | -[docs]: https://buf.build/docs |
139 | 195 | [email_dev]: mailto:dev@buf.build |
140 | 196 | [email_info]: mailto:info@buf.build |
141 | | -[features]: #features |
142 | 197 | [fish]: https://fishshell.com |
143 | | -[format_usage]: https://buf.build/docs/format/style/ |
144 | | -[generate_usage]: https://buf.build/docs/generate/tutorial/ |
145 | | -[ide]: https://buf.build/docs/cli/editor-integration/ |
146 | | -[idl]: https://en.wikipedia.org/wiki/Interface_description_language |
147 | | -[images]: https://buf.build/docs/reference/images/ |
148 | | -[inputs]: https://buf.build/docs/reference/inputs/ |
| 198 | +[format]: https://buf.build/docs/format/ |
| 199 | +[generate]: https://buf.build/docs/generate/ |
| 200 | +[generated_sdks]: https://buf.build/docs/bsr/generated-sdks/ |
149 | 201 | [install]: https://buf.build/docs/cli/installation/ |
150 | | -[jetbrains]: https://buf.build/docs/cli/editor-integration/#jetbrains-ides |
151 | | -[lint]: https://buf.build/docs/lint/overview/ |
152 | | -[lint_rules]: https://buf.build/docs/lint/rules/ |
153 | | -[lint_usage]: https://buf.build/docs/lint/tutorial/ |
| 202 | +[lint]: https://buf.build/docs/lint/ |
| 203 | +[lint_plugin]: https://buf.build/docs/lint/ |
| 204 | +[migrate_from_protoc]: https://buf.build/docs/migration-guides/migrate-from-protoc/ |
| 205 | +[modules_workspaces]: https://buf.build/docs/cli/modules-workspaces/ |
154 | 206 | [npm]: https://buf.build/docs/cli/installation/#npm |
155 | | -[minisign]: https://github.com/jedisct1/minisign |
156 | | -[powershell]: https://docs.microsoft.com/en-us/powershell |
| 207 | +[powershell]: https://learn.microsoft.com/en-us/powershell/ |
157 | 208 | [protobuf]: https://protobuf.dev |
158 | | -[pushing]: https://buf.build/docs/bsr/module/publish/ |
159 | | -[releases]: https://buf.build/docs/cli/installation/#github |
160 | | -[repo]: https://github.com/bufbuild/buf/ |
161 | | -[repositories]: https://buf.build/docs/concepts/repositories/ |
162 | | -[source]: https://buf.build/docs/cli/installation/#source |
163 | | -[tarball]: https://buf.build/docs/cli/installation/#github |
164 | | -[templates]: https://buf.build/docs/configuration/v2/buf-gen-yaml/ |
165 | | -[users]: https://buf.build/docs/admin/manage-members/ |
| 209 | +[protobuf_es]: https://github.com/bufbuild/protobuf-es |
| 210 | +[protovalidate]: https://protovalidate.com |
| 211 | +[remote_plugins]: https://buf.build/docs/bsr/remote-plugins/ |
| 212 | +[schema_checks]: https://buf.build/docs/bsr/checks/ |
| 213 | +[source builds]: https://buf.build/docs/cli/installation/#source |
| 214 | +[tarballs]: https://buf.build/docs/cli/installation/#github |
166 | 215 | [verifying]: https://buf.build/docs/cli/installation/#github |
167 | 216 | [windows]: https://buf.build/docs/cli/installation/#windows |
168 | 217 | [zsh]: https://zsh.org |
0 commit comments