cmd/compile: relax wasm/wasm32 function import signature type constraints

[johanbrandhorst]

Background

#59149 removed the package restrictions on the use of go:wasmimport, but established strict constraints on the types that can be used as input and result parameters. The motivation for this was that supporting rich types between the host and the client would require sophisticated and expensive runtime type conversions because of the mismatch between the 64 bit architecture of the client and the 32 bit architecture of the host.

With the upcoming 32 bit wasm port, this problem goes away, as both client and host will use 32 bit pointers. However, we can also support a limited set of types on 64 bit platforms, where no runtime conversion is necessary.

Proposal

Relax the constraints on types that can be used as input and result parameters with the go:wasmimport compiler directive. The exact allowed types would depend on whether wasm or wasm32 is used.

We define the "small integer types" group as the set of types described by [u]int[8|16]. The following types would be allowed as input parameters to any wasmimport/wasmexport function:

• bool
• int32, uint32, int64, uint64
• float32, float64
• string
• uintptr, unsafe.Pointer, *T where T is an allowed parameter type or one of the small integer types.
• *struct. All fields of the *struct must be allowed parameter types, struct, [...]T, or one of the small integer types.
  • If the struct has any fields, it must embed structs.HostLayout (see structs: add HostLayout "directive" type #66408).
  • Any struct fields must also embed structs.HostLayout (recursively).
  • *T, unsafe.Pointer, uintptr, and string types are only allowed in *struct on wasm32.
• *[...]T where T is an allowed type or one of the small integer types.

All input parameter types except string are also allowed as result parameter types.

The following types would remain disallowed as input and output parameter types:

• chan T
• complex64, complex128
• func
• interface
• map[T]U
• []T
• struct
• [...]T

The conventions established for use of pointers in CGO will be required when using pointers with wasmimport/wasmexport, e.g. the host can read Go memory, can write pointerless data (like the content of a byte buffer) but cannot write Go pointers to Go memory, and cannot hold on to Go pointers unless they are pinned.

Discussion

Compatibility guarantees

The Go spec does not specify the struct layout and leaves it up to implementations to decide. As such, we cannot provide a guaranteed ABI without having to change the spec or force future layout changes to provide runtime conversion of data. This proposal suggests making it clear to users through documentation that there are no guarantees of compatibility across versions of the Go compiler.

Type conversion rules

The following conversion rules would be automatically applied by the compiler for the respective parameter type:

Go Type	Parameter type (per Wasm spec)
bool	i32
int32, uint32 int64, uint64	i32, i32 i64, i64
float32, float64	f32, f64
string	Assigned to two call parameters as a (i32, i32) tuple of (pointer, len). Only allowed for input parameters.
uintptr, unsafe.Pointer, *T, *struct, *[...]T	i32, i32, i32, i32, i32

Strings

Strings are not allowed as result parameters as Wasm practically does not allow more than 1 result parameter.

Supporting GOARCH=wasm

The wasm architecture uses 64 bit pointers and integer sizes. As the host uses 32 bit pointers, this makes it impossible to allow certain types without costly runtime conversions, such as *struct types containing pointer fields. Since string types are also pointer types, *struct types containing string fields are also disallowed.

Supporting [u]int, [u]int8, [u]int16 as concrete parameters

The [u]int types are problematic as the size of them are not precisely defined, and may cause confusion when used with strictly 32 bit or 64 bit integers. The [u]int8 and [u]int16 types are problematic because we would be forced to automatically convert them to/from the i32 wasm representation, with potential loss of precision or overflow. They are still allowed as pointer type, array elements and struct fields.

Supporting slices, maps

Both slices and maps are disallowed because of the uncertainty around the memory underlying these types and interactions with struct and array rules. Users who wish to use slices can manually use (&slice, len(slice)) or unsafe.Pointer. There is no clear way to support passing or returning map data from the host other than by using unsafe.Pointer and making assumptions about the underlying data.

Related proposals

struct.Hostlayout

#66408 proposes a way for users to request that struct layout is host compatible. Our proposal depends on the definitions put forward in this proposal for struct parameters.

Future work

WASI Preview 2 (AKA WASI 0.2)

WASI Preview 2 defines its API in terms of the Component Model, with a rich type system and an IDL language, WIT. The Component Model also defines a Canonical ABI with a specification for lifting and lowering Component Model types into and out of linear memory. This proposal does not attempt to define the ABI for any hypothetical wasip2 target, and would leave such decisions for any future wasip2 proposal.

Supporting struct and [...]T by value

A previous version of this proposal included support for passing struct and [...]T types by value by expanding each field recursively into call parameters. This was removed in favor of a simpler initial implementation but could be re-added if users require it.

Contributors

@johanbrandhorst, @evanphx, @achille-roussel, @dgryski, @ydnar

CC @cherrymui @golang/wasm

@gabyhelp's overview of this issue: #66984 (comment)

[dr2chase]

"This follows the C struct value semantics" is just a hair vague; are 8-byte quantities (float64, int64, uint64) stored at a 4-byte or 8-byte alignment? It was my understanding (and the purpose of #66408) to specify a 4-byte alignment for fields of those types when they occur in structs passed to wasm32 (tagged structs.HostLayout).

(edited to note error, the host alignment for 8-byte integers and floats is 8 bytes).

[ydnar]

Ideally 8-byte values would always be 8-byte aligned in the wasm32 port.

[evanphx]

@dr2chase Looking at what clang does, it uses 8-byte alignment on 64bit quantities so we'd match that.

[dr2chase]

You are right, I got it backwards. But that is what you are expecting for anything that has pointers-to-it passed to the wasm host platform, yes?

[cherrymui]

Thanks for the proposal! A few questions:

• 8-byte alignment for 64-bit values, as mentioned above. cmd/compile: create GOARCH=wasm32 #63131 doesn't seem to have a definitive answer, and currently it seems the CL doesn't implement 64-bit alignment.
• If we don't always align 64-bit value to 8 bytes (which differs from current all 32-bit architectures we support and probably requires quite some work), we should align 64-bit value to 8 bytes when structs.HostLayout is specified. So structs: add HostLayout "directive" type #66408 is very related.
• structs and arrays. What is the ABI specification exactly? The C ABI on, say ELF AMD64, is pretty complex for passing structs and arrays. Small fields may be packed into one word. Large structs may be passed indirectly (stored on stack, passing a pointer to the callee). Do we have a specification for this?
• string. What does a string look like on Wasm/WASI side? I couldn't find its specification on WASI P1 doc. On Component Model doc https://github.com/WebAssembly/component-model/blob/main/design/mvp/CanonicalABI.md (which I guess is for WASI P2, not P1?), it specifies string is two i32, which is similar to Go's string, which is good. But also it allows three encodings, UTF-8, UTF-16, and "latin1+utf16" differentiated by a high bit. The second and third encoding are not compatible with Go strings. Do we require UTF-8 encoding? Or we don't allow passing Go strings directly?

Besides, for structs, arrays of structs, and pointer to structs, I would suggest we allow only structs with structs.HostLayout to be passed. The reason is that in the Go spec we don't require struct fields to be laid out in memory in source order, and it may well change in a future Go release. structs.HostLayout specifies a fixed layout. Structs without that marker can change. This gives a clear way to say which structs should have a fixed layout, which are okay to change.

Thanks.

[dr2chase]

Two other questions, first:

type w32thing struct {
    _ structs.HostLayout
    a uint8
    b uint16
}

Is this laid out a_bb or is it aaaabbbb? What sizes do I use for struct fields? I assume it is the smaller ones, but I wanted to verify this else it would be a problem.

Second, passing pointers to 8-byte primitive types to the host will be tricky unless those references come from fields in structures tagged with HostLayout -- otherwise, they may not be aligned. So

type wx struct {
   _ structs.HostLayout
  x int64
}
func f(x int64, w wx) {
  someWasmFunc(&x) // might not work, x might not be 8-byte aligned
  someWasmFunc(&w.x) // this will work because w is a wx and its x field is 8-byte aligned
  someOtherWasmFunc(&w) // if it used *wx for its parameter type instead of *int64
}

[johanbrandhorst]

Thanks for the quick feedback! I've tried to answer each question:

structs and arrays. What is the ABI specification exactly? The C ABI on, say ELF AMD64, is pretty complex for passing structs and arrays. Small fields may be packed into one word. Large structs may be passed indirectly (stored on stack, passing a pointer to the callee). Do we have a specification for this?

The specification falls out of the table of transformations (I think?). There current plan isn't to introduce any sort of magic around large structs or field packing. Structs fields are added as call parameters, from the first field to the last, according to the conversion rules for the type of the field. Examples:

type foo struct {
    a int
    b string
    c [2]float32
}

With a function signature of

//go:wasmimport some_module some_function
func wasmFunc(in foo) int

Would roughly translate to (in WAT format)

// $a is of type `i32` holding the value of `a`
// $b_addr is of type `i32` and is a pointer to the start of the bytes for the Go string `b`
// $b_len is of type `i32` and is the length in bytes to read from `$b_addr` to get the whole string
// $c_0 is of type `f32` and is the value of `c[0]`
// $c_1 is of type `f32` and is the value of `c[1]`
call $some_function (local.get $a) (local.get $b_addr) (local.get $b_len) (local.get $c_0) (local.get $c_1)

Struct fields would be expanded into call parameters before subsequent fields at the same level.

What does a string look like on Wasm/WASI side?

For wasip1, we will treat Go string parameters simply as a (*byte, int) tuple. There will be no encoding constraints, just as with regular Go strings. To the Wasm host, it will look identical to using struct { a *byte; b int } as a parameter. For wasip2, those constraints would have to be considered in a hypothetical future wasip2 proposal.

Making structs.HostLayout required for structs, arrays of structs and pointers to structs

This sounds like a great idea, and we should also extend it to pointers to 8 byte sized primitive types to guarantee alignment, as suggested by @dr2chase's last question. This would avoid any question around alignment issues for pointers. It hurts the ergonomics a little bit but that's a price worth paying, I think.

type w32thing struct {
_ structs.HostLayout
a uint8
b uint16
}

Is this laid out a_bb or is it aaaabbbb? What sizes do I use for struct fields? I assume it is the smaller ones, but I wanted to verify this else it would be a problem.

I'm a little confused by the question to be honest. If this type was used as an input to a Wasm call, it would look like this:

// $a is of type `i32`
// $b is of type `i32`
call $some_function (local.get $a) (local.get $b)

I suppose that might mean the memory looks like this: a___bb__? We're not passing a pointer to the struct or the fields, so we'd need to copy the values into locals, which will be of type i32 (I think)? Admittedly my grasp of this exact part of the code is a bit weak so I appreciate corrections.

[cherrymui]

Thanks for the response!

Structs fields are added as call parameters, from the first field to the last, according to the conversion rules for the type of the field.

This sounds like a reasonable choice. Is this ABI specified anywhere in Wasm/WASI docs? Or the Wasm side has to define the function taking parameters element-wise?

For wasip1, we will treat Go string parameters simply as a (*byte, int) tuple. There will be no encoding constraints, just as with regular Go strings. To the Wasm host, it will look identical to using struct { a *byte; b int } as a parameter.

This sounds reasonable as well. Is it specified anywhere in Wasm/WASI docs?

Thanks.

[johanbrandhorst]

This sounds like a reasonable choice. Is this ABI specified anywhere in Wasm/WASI docs? Or the Wasm side has to define the function taking parameters element-wise?

I don't know about this being an official ABI so much as just a consequence of the Wasm spec around function calls and how we can apply Go semantics to it. We're limited to the i32, i64, f32 and f64 value types, and the call instruction takes a function index and arguments from the stack. In order to simulate pass-by-value for structs, we have to flatten each field to one of the allowed value types.

This sounds reasonable as well. Is it specified anywhere in Wasm/WASI docs?

Not sure there's a doc anywhere, but practically, definitions like path_create_directory, which take a string parameter, use this pattern: https://cs.opensource.google/go/go/+/refs/tags/go1.22.2:src/syscall/fs_wasip1.go;l=230.

[dr2chase]

I guess my question is whether a pointer-to-struct is ever passed from Go to the WASM platform, and therefore, what expectations the WASM side has about the layout of the fields of that structure. structs.HostLayout is intended to obtain those expectations, but (1) do we even need to do this? We thought we did, and (2) we need to know what the expectations are. I think it was just that 64-bit floats and ints get 64-bit alignment.

I don't think this is for specifying the layout that gets passed to a WASM call if the struct is passed by value.