feat: added WASI runtime and js/go WASM packages

[niallnsec]

This PR builds on the initial browser runtime support that is already present in YARA-X from my earlier PR. The main purpose of this change is to tidy up the WASM runtime code now that there are multiple backends, and to add two new pieces on top of that existing support:

• a browser-facing JavaScript package, js-wasm
• a new set of Go bindings, go-wasm

Alongside that, I have added support for a wasm32-wasip1 build and expanded the test coverage for both the browser and Go/WASM paths.

Runtime Refactor
YARA-X now has three runtime backends for executing compiled rules:

• native wasmtime
• the browser runtime for wasm32-unknown-unknown
• a WASI-oriented runtime for wasm32-wasip1

As these pieces developed, the browser and WASI runtimes had started to duplicate too much of the same Wasmtime-shaped shim code. I have restructured that under lib/src/wasm/runtime so that:

• mod.rs is just the backend selection and re-export layer
• native.rs handles the native wasmtime path
• common.rs contains the shared compatibility layer used by the custom runtimes
• browser.rs contains browser-specific behaviour
• wasip1.rs contains WASI-specific behaviour

The aim was to separate shared machinery from backend-specific logic. That makes the layout easier to follow and should make future maintenance less awkward.

I also renamed the Cargo feature from wasip2-runtime to wasip1-runtime, which better reflects the target actually being built.

Browser JavaScript Package
One entirely new part of this PR is js-wasm.

Although YARA-X already had the underlying browser runtime support, it did not yet have a browser-facing JavaScript package that exposed that functionality in a usable form. This PR adds that package in-tree.

I also moved its standard packaging flow onto wasm-pack, with the default generated package in pkg/ and additional no-modules bundle output in dist/.

The JavaScript API is unchanged, but the build and test setup should now be much more familiar to anyone who has worked with Rust/WASM projects before.

Go/WASM Bindings
The other major addition in this PR is go-wasm.

I use YARA from Go heavily in my day-to-day work, pushing very large volumes of data through it every week. The existing CGO-based approach works, but CGO is awkward to maintain and deploy, and it complicates the build process far more than I would like. To borrow the usual phrase, "cgo is not Go". A pure Go build of YARA is extremely useful to me, and I think it is likely to be useful to others as well, because it makes the build system much simpler and allows it to run in many more places. (eg. github.com/Velocidex/yara-x-go)

That is the reason for the WASI guest build and the go-wasm bindings. The aim here was not just to prove that it can be done, but to produce something that is actually practical to use from normal Go code.

Why the experimental allocator exists
A large part of the go-wasm work was driven by scanning behaviour rather than just packaging.

The existing block-scanning approach in YARA is very limited, to the point that it is not really usable for a lot of the cases I care about. What I wanted was a way to keep using the normal scanning flow while still being able to supply data lazily, without having to load the whole input into memory up front.

That is what led to the experimental allocator and the userfaultfd work. It allows the standard scan calls to operate against guest memory that is populated on demand at runtime. In practice, that makes it possible to handle Go-style scanning workloads much more naturally, especially where the source data is large or where loading everything eagerly would be undesirable.

I think that is a much better fit for real Go code than relying on YARA's current block scanner.

wasm32 limits and wasm64
This line of work also led me to look at whether a wasm64 build would be practical.

Even with the userfaultfd approach, a wasm32 guest is still fundamentally limited by its address space. In practice that means the maximum size of a block-scannable file is still somewhat less than 4 GB. I investigated whether moving to wasm64 would let that limit be lifted, but at the moment that is not practical because of upstream dependency issues. So for now that ceiling remains in place.

I think it is worth being explicit about that limitation, even though I would have preferred to remove it.

Performance
It was also important to me that the Go bindings be as fast as possible, because my hope is to use them in production as a replacement for the C bindings rather than treating them as an experiment.

I spent a fair amount of time analysing and profiling the Go/WASM path to get it into a reasonable state. A number of the details in the runtime, particularly around synchronisation behaviour, came out of that work. The goal there was to reduce unnecessary copying and state movement between host and guest, and to make the normal scan path efficient enough to be taken seriously as a production option.

So while the pure-Go build and deployment system was one motivation, performance was equally important, if not even more so.

Testing
I also put a lot of effort into validating both the normal and experimental Go paths.

The Go module functionality tests have been factored into a shared reusable suite and are exercised through:

• the normal Scanner.Scan path
• the normal copied Scanner.ScanReaderAt path
• the experimental userfaultfd-backed ScanReaderAt path

That is especially useful because modules tend to access the scanned data in a much less predictable way than the main scanner. Reusing the existing module fixture corpus against these paths is a much better stress test of the lazy-paging behaviour than a simple sequential scan would be.

Note: I ran the Go/WASM tests in Docker in both the standard and privileged userfaultfd configurations.

Why I am proposing this
What I wanted to do here was not just add functionality, but leave the code in a shape that is easier to work with afterwards.

In practical terms, I think this improves things by:

• making the runtime layout clearer
• removing duplicated code between the custom runtimes
• adding a proper browser JS package on top of the existing browser runtime support
• adding a practical pure-Go binding layer
• improving regression coverage substantially
• giving high confidence in the Go/WASM code, especially around ScanReaderAt, performance, and the experimental allocator path

Maintenance
I appreciate that go-wasm is a fairly substantial addition, and I can understand that you may be concerned about the maintenance burden.

I am happy to make a clear commitment to maintaining the go-wasm code going forward. If there is concern about keeping that much code in-tree, I would also be willing to move it out of tree and maintain it separately instead.

My preference would be to keep it in-tree, because it benefits from staying close to YARA-X itself, sharing tests and versioning, and being validated alongside the rest of the project. But if you would rather reduce the amount of code carried in the main repository, I completely understand.

Finally, I am fairly new to Rust as most of my work is done in Go. I took some lessons from the modifications you made to my previous PR in the hopes that this one is a little better. Even so, I apologise if my relative inexperience with Rust has caused me to produce any Rust anti-patterns.

[plusvic]

This is too much code for a single PR, and it contains changes that are not necesarily related like the js-wasm crate and the go-wasm crate. I need to study the proposes changes and decide what to do, but I'll probably ask you to split this into more manageable chunks, each focusing on a more specific feature.

[niallnsec]

Totally understand. Its probably quite easy to split up into 3 pieces off the bat.

All of the changes under the lib directory are focused on the WASM runtime and don't really touch anything else. Its essentially a continuation of the previous change plus the extra support for building for a WASI target.

Then js-wasm and go-wasm are pretty self contained so I can take them out entirely and submit them individual or make them distinct packages outside of this repo if you'd prefer. I believe go-wasm contains the largest share of code in this PR which is one of the reasons I was already concerned about its size and maintenance burden.

[plusvic]

Totally understand. Its probably quite easy to split up into 3 pieces off the bat.

All of the changes under the lib directory are focused on the WASM runtime and don't really touch anything else. Its essentially a continuation of the previous change plus the extra support for building for a WASI target.

Then js-wasm and go-wasm are pretty self contained so I can take them out entirely and submit them individual or make them distinct packages outside of this repo if you'd prefer. I believe go-wasm contains the largest share of code in this PR which is one of the reasons I was already concerned about its size and maintenance burden.

For now I have this question: which advantages offers the wasip1 runtime over the browser one?.

After a brief investigation my conclusion is that wasip1 offers a less limited environment that supports, among other thing, access to the file system. Something that is not supported for WASM code compiled for the wasm32-unknown-unknown target. Which leads me to another question: What happens currently with the code that depends on std::fs?
For instance, the compiler sometimes needs to access the file system for reading included files, what happens to that logic when you compile the yara_x crate for the wasm32-unknown-unknown target?

I haven't spent enough time yet digging into the previous PR that added support for WASM, I would like to understand this better.

[niallnsec]

The main advantage of the wasip1 runtime over the browser one is mainly in the more capable host environment.

In both cases the generated rule WASM is executed without embedding Wasmtime inside yara-x. The difference is that:

• wasm32-unknown-unknown runs against the browser/JS WebAssembly environment, which is quite limited
• wasm32-wasip1 runs in a host environment that can provide capabilities such as filesystem access

Paths that use std::fs are not rewritten or stubbed out on wasm32-unknown-unknown at the moment. If they are called there they fail at runtime because there is no backing filesystem. To support them in the browsers WASM runtime we would have to create a virtual FS abstraction and shim it in via the runtime callback ABI.

Taking include handling as an example, the compiler still uses std::fs::read to load included files. On the browser target there is no virtual filesystem or include callback, so include statements fail with a compile error there. With the wasip1 runtime, those same code paths can work as long as the host provides filesystem access. In go-wasm I mount a filesystem into the guest, so using include directories there works as expected.

So generally speaking:

• browser runtime: good for executing rules in JS/browser environments that are highly restrictive
• wasip1 runtime: useful for embeddings like go-wasm, where the guest needs access to host capabilities and to behave more like a normal program

[niallnsec]

In order to try and make things a bit easier to review, I thought it might be helpful to explain in a bit more detail the design decisions that went into this PR and the previous one as well as the process I went through, as it was very much an iterative design.

The first objective I had was to make it possible to run yara-x in a browser, and WebAssembly is the natural way to do that. The first thing I tried was compiling the library for wasm32-unknown-unknown without making any structural changes. That failed because yara-x depends on wasmtime for executing generated rule Wasm, and Wasmtime is a host-side runtime rather than something that can sensibly be used when yara-x itself is being compiled as a Wasm guest.

Once I understood that properly, the obvious approach was to remove the dependency on Wasmtime for Wasm targets and let the surrounding host runtime execute the generated rule Wasm directly. In the browser case, that means using the host WebAssembly implementation exposed through JavaScript. js-sys and wasm-bindgen provide the necessary building blocks for that, so the browser work was mainly a matter of implementing a small shim that looks enough like the subset of the Wasmtime API that yara-x uses, while delegating the actual execution to the browser’s own WebAssembly runtime.

I did not try to extend the browser-side host integration beyond what is needed to get the core functionality of yara-x working. In particular, things like a virtual filesystem for loading included files, or support for scanning files by path in the browser, are out of scope for this work. The browser package is intended to compile rules from provided source strings and scan in-memory byte buffers. Filesystem-backed features are not currently part of that design.

Once I had the browser runtime working, it occurred to me that the same overall approach could be used to produce a set of Go bindings that do not require CGO.

One of the first things I ran into there was that wasm32-unknown-unknown is a very conservative target. It makes essentially no assumptions about the host environment. The standard library still exists for that target, but a number of OS-like APIs are either unavailable or simply fail at runtime because there is no defined host implementation behind them. That includes filesystem access, and it also includes things like SystemTime::now() on that target. In the browser build, where time support is still desirable, I handled that explicitly by using the JavaScript clock rather than relying on std::time.

Because wasm32-unknown-unknown makes so few assumptions about the host, any host-specific functionality has to be provided explicitly by the embedding environment. In the browser case that means interacting with JavaScript and the browser’s WebAssembly runtime through JS bindings and shared memory. That is why the browser path relies on js-sys and wasm-bindgen rather than on a standard system interface such as WASI.

When a full WASI environment is available on the host, wasm32-wasip1 is a much better fit for this kind of non-browser embedding. WASI Preview 1 gives the guest a standard syscall-like interface for things such as filesystem and stdio access, provided the host chooses to make those capabilities available. The important difference here is not that wasm32-wasip1 is inherently "faster", but that it is much more capable and much more natural for a library-style embedding that wants the guest to behave more like a normal program. The wazero runtime I used for the Go bindings has good support for WASI Preview 1, which meant a fair amount of that host integration was already available.

That said, the Go bindings still need their own host-guest ABI on top of WASI. This is an important distinction, because there are actually two different interfaces involved in this work.

The first is the internal runtime interface used by yara-x itself when it is compiled for wasm32-wasip1. That interface is defined in the WIT file and is the minimum host functionality required for the custom wasip1 rule runtime to operate. It covers things like validating modules, instantiating them, calling exports, and creating or synchronizing globals and memories. This is the interface the wasip1 runtime inside yara-x uses in place of Wasmtime.

The second is the public ABI used by go-wasm. That is separate. The Go bindings need a stable, Go-friendly interface for creating compilers and scanners, loading rules, setting globals, scanning buffers, and retrieving results. For that ABI I chose a simple handle-based transport with JSON used for higher-level structured data. The host side can pull byte buffers out of guest memory using integer handles, and structured results can be serialized in a way that is easy to consume from Go. That choice was mainly about portability, simplicity, and keeping the public boundary explicit.

A separate detail that came out of the performance work was synchronization of imported module state across the boundary between generated rule Wasm and the host runtime. Both the browser runtime and the wasip1 runtime maintain host-side views of globals and memories that need to stay coherent with what the generated rule Wasm expects to see. Unconditionally synchronizing all of that state around every callback would work, but it would be unnecessarily expensive. To avoid that, the functions exported to generated rule Wasm carry sync metadata which indicates whether state needs to be synchronized before the call, after the call, both, or neither. The custom runtimes then use that metadata when invoking callbacks so that synchronization only happens when it is actually needed. That was one of the main results of the profiling and analysis work I did while trying to make the Go bindings performant enough to be taken seriously as a production replacement for the C bindings.

That performance work was important to me from the start. I use YARA heavily from Go in day-to-day work, and while CGO works, it is awkward to maintain and awkward to deploy. A pure Go build is useful not just because it is cleaner in principle, but because it simplifies the build system substantially and makes it possible to use YARA in places where carrying a native C dependency is inconvenient or undesirable. My aim with go-wasm was therefore not just to prove that the idea works, but to get it to a point where it is practical enough to use in production.

A large part of that work led into the experimental allocator and the userfaultfd support. The motivation there was to get something closer to a usable block-scanning model for Go. The block scanner available natively in YARA is quite limited, to the point that it is not a realistic option for many of the workloads I care about. What I wanted instead was a way to keep using the standard scan calls while allowing data to be faulted into guest memory lazily rather than loading the entire input eagerly. userfaultfd provides a way to do that by backing guest memory with host-managed mappings and populating pages on demand. That is a much better fit for Go-style scanning code that often wants to abstract over streams, readers, and large addressable inputs.

That work is also what led me to look into the feasibility of a wasm64 build. Even with the userfaultfd approach, a wasm32 guest is still constrained by a 32-bit address space, which means the maximum practically scannable file is still somewhat less than 4 GB. I investigated whether moving the guest to wasm64 would remove that limit, but at the moment that is not practical because upstream Rust and WIT tooling is not yet in a shape that makes this path viable for this project. So for now that size limit remains in place.

Since both the browser runtime for wasm32-unknown-unknown and the custom runtime for wasm32-wasip1 needed to expose roughly the same external surface as the subset of Wasmtime used by yara-x, a fair amount of duplication had naturally built up between them. To address that, I moved the runtime code into its own module and split it into three layers: a thin selector module, a shared common layer for the Wasmtime shaped abstraction, and separate backend-specific implementations for the browser and wasip1 cases. The browser-specific code deals with JavaScript and the host WebAssembly API. The wasip1 specific code deals with the WIT-defined host bridge. For non-Wasm targets, the code continues to use Wasmtime directly.

I think this is a good structure because it leaves the vast majority of the library untouched and keeps the Wasm-specific support isolated to a clearly defined runtime abstraction layer.

There are still some important limitations, and they differ between the browser and wasip1 cases.

The browser target is the most constrained. There is no practical equivalent there to the native timeout and interruption mechanisms used in the non-Wasm build, so timeout configuration in the js-wasm package is currently treated as a no-op. More broadly, browser-side filesystem-dependent features are not implemented, so things such as loading included files from disk are not supported there.

The wasip1 case is better, but it is still not identical to native execution. In that path there is real timeout support, but it is host enforced and best effort rather than being equivalent to native preemptive interruption. In the Go embedding, deadlines are enforced through the host bridge and checked at the relevant interaction points. That is a meaningful improvement over the browser case, and it is good enough to be useful, but it is still not the same thing as having full native style interruption available everywhere.

Overall, the approach I ended up with was to keep the core of yara-x as intact as possible, replace the Wasmtime dependency only where that is necessary for Wasm targets, and isolate the target-specific behaviour in one runtime abstraction layer. That gave me a browser-capable build, a usable wasm32-wasip1 build for non-browser embeddings, and a path to pure-Go bindings without forcing invasive changes across the rest of the codebase.

[plusvic]

Thanks for your detailed explanation. I took some time to think carefully about this, and I came to these conclusions:

• I support the idea of having JS bindings for YARA-X, in the same spirit of the Golang and Python API. This is something that would be useful for anyone wanting to interface their JavaScript projects with YARA. For instance, the YARA playground discussed here could benefit from having such an API.

• I would remove the functions scanRules and validateRules from the API proposed in js-wasm. They deviate too much from the existing Rust, Go and Python APIs. My goal is that all the APIs share a common principle, which consists on a Compiler that produces Rules, and a Scanner that uses such Rules for scanning data. Those types are already present in js-wasm and they look fine.

• At least at this moment I prefer to leave the wasip1 runtime and go-wasm out. While I see the benefits of having Go bindings that don't involve the use of cgo, my concern is that it introduces a lot of complexity and an extra maintenance burden. I'm going to play with this a little more, I find your work in this area very interesting, but let's start with the contibutions that I think will have a broader impact.

So, if you want to work on a PR that includes only the JS bindings that would be really great.

[niallnsec]

Thanks for taking the time to review this. I have submitted a new PR with just the JS focused changes. If we can get that to a place where you are happy with it to be merged, then after I'll move just the wasip1 runtime build support into a followup PR without all the extensive go-wasm binding code if thats okay.