Use non-200 HTTP response status codes for API errors #7881
Description
When cargo and crates.io were first implemented, the HTTP response status codes were ignored. cargo assumed that crates.io would always respond with a "200 OK" and any error messages are detected from the response JSON body. When crates.io responded with a different status code the cargo was showing the raw JSON body instead of a regular error message.
This was changed in rust-lang/cargo#6771, which was released in cargo 1.34 (mid-2019). Since that release, cargo now supports 4xx and 5xx status codes too and extracts the error message from the JSON response, if available.
The crates.io team has decided to phase out these "200 OK" response status codes for error responses to make our API a little more intuitive. Since this can be seen as a breaking change to our API we will need to announce this change publicly, including a cut-off date. On this date we want to switch the API from the previous "200 OK" behavior to the new 4xx/5xx behavior.
Inside the crates.io source code we have been using a cargo_err() fn which returns a "200 OK" JSON response with a custom error message. These calls will have to be replaced with other errors.
To allow us to incrementally migrate our codebase while maintaining the previous behavior, a middleware was introduced in #7715, which ensures that all cargo-relevant endpoints always return "200 OK". #7776 then implemented a feature flag on top of the middleware, which will allow us to flip the switch on the cut-off date and test it out ahead of time on our staging environment.
The remaining work now is to replace all calls of cargo_err() with 4xx/5xx error responses and adjust the corresponding tests, or write new tests if none exist for these error scenarios yet.