proposal: net/url: preserve order of URL Query parameters instead of sorting them alphabetically

[tiger5226]

What version of Go are you using (go version)?

master

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

N/A

What did you do?

I was leveraging a go library for restful api communication, and ran into a frustrating issue where the url parameters were being sorted alphabetically. Normally, this does not matter. However, with the advent of HMAC Signing and many APIs using it now, this becomes a problem. The idea is that you sign the url which in a GET situation includes the url parameters and append a new hmac_sign parameter to the end. This works fine in many languages except Go.

I don't know for sure why the go team decided to do this. I suspect that the decision was made to use map[string]string to hold the url parameters. Since maps do not guarantee an order, a sort was done to make the return deterministic.

I would like to suggest this be backed by an array so it can be possible to preserve an order to support the above mentioned use case. This allows libraries to also support ordering while still depending on the core Go libraries.

What did you expect to see?

I expect that as I set parameters the order is maintained when I call the encode function.

What did you see instead?

Instead they are sorted alphabetically.

[odeke-em]

Thank you for filing this issue @tiger5226 and welcome to the Go project!

For posterity purposes when working on this, here is source code to demo @tiger5226's issue
https://play.golang.org/p/N4NOePil1iL or inlined below

package main

import (
	"fmt"
	"net/url"
)

func main() {
	s := "https://golang.org/net/http?scale=10&age=100&tok=aYxZPls28wz"
	u, _ := url.Parse(s)
	fmt.Printf("Encode:   %s\nRawQuery: %s\n", u.Query().Encode(), u.RawQuery)
}

which produces

Encode:   age=100&scale=10&tok=aYxZPls28wz
RawQuery: scale=10&age=100&tok=aYxZPls28wz

Luckily most of my services are in Go so the problem isn't out of my control.

Anyways to encounter the problem, @tiger5226 please use url.RawQuery which preserves the raw query like other languages do and when performing your MAC signing please use that and it should solve the issue, as we have this proposal up for examination.

[agnivade]

/cc @bradfitz

[rsc]

The choice of a map for storage basically makes this impossible. url.Values is a map. Values.Encode can either take the random order that map iteration returns (not ideal) or sort them (better).

The few uses that care about exact wire format concerns (including order) can use RawQuery directly. That's why it's there.

Sorry, but there's very little to do here.

[tiger5226]

@rsc I agree, the choice of map for storage makes this impossible. That is precisely what I am questioning. We don't need to use a map, or do we? Maybe I missed something that requires this. If so I get it then. Otherwise this was a choice someone made, probably before url signing was popular that they thought was irrelevant at the time. I am suggesting we can instead back it with a slice to support these use cases.

The business case for something like this is to allow others who create libraries to continue to use the default packages structs. I can definitely work around this and have. I am just offering up a better alternative to me avoiding the use of the url package's struct. It doesn't make sense for me to avoid that since both can be supported without a breaking change.

The few uses that care about exact wire format concerns (including order) can use RawQuery directly.

I respectfully disagree. This is pretty common. Please see online references to the practice.

https://www.limestonenetworks.com/support/knowledge-center/24/112/what_is_url_signing.html
https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-signed-urls.html
https://developers.google.com/maps/documentation/maps-static/get-api-key#sample-code-for-url-signing
https://oauth1.wp-api.org/docs/basics/Signing.html

It is becoming more and more popular. I respect your decision to close the issue. I guess, I expected a more robust consideration process, especially since I took the time to create the issue and I would offer to implement a solution to contribute.

[ianlancetaylor]

@tiger5226 I'm sorry you feel that your proposal was not sufficiently considered.

We can't change net/url.Values because that would break the Go 1 compatibility guarantee (https://golang.org/go1compat). Perhaps using a map was a mistake. I don't think it was, but it's not worth taking the time to discuss now because it can't be changed in any case.

More importantly, you haven't explained why you can't use RawQuery.

[tiger5226]

No, I certainly can. That is not a problem. I was just thinking of a better way. I agree we should NEVER break the Go 1 compatibility guarantee. I would not suggest changing that specific struct, just that we support both use cases. This can be separate from url.Values, like url.OrderedValues for example. I am proposing that we support ordered params without having to set a raw query.

I think maybe I am doing a poor job at explaining my suggestion. Let me collect some thoughts and put together a more comprehensive suggestion that better represents what I am saying.