Revisit inference API to load and run a model

[jbingham]

Can we revisit the idea of an API to load and run a model?

I've written up a draft explainer for what a Web ML Inference API might look like, as well as a bunch of issues and questions around it.

The idea was discussed way back in issue 3, and probably even earlier by many people in the group. For various reasons, the group decided to pursue a graph API instead.

Why revisit now? The TensorFlow team raised some concerns that a graph API may not be the right level of abstraction, due to how fast ML is evolving, and their experience with the rapid growth in operations in TensorFlow models. After digging a bit to understand where this caution came from, I learned about the efforts around MLIR, and how the TensorFlow team sees that fitting into the picture. Also, I had a chance to talk with Greg and others at Microsoft about the original reasons not to go with an inference API, and it seems like things may have changed.

If this is an interesting enough topic to people, we could consider talking about it during the face-to-face in Berlin.

[huningxin]

Thanks @jbingham for reaching out and opening this discussion. I am interested in this topic. Especially, the explainer is extremely helpful to me, thanks for that.

The TensorFlow team raised some concerns that a graph API may not be the right level of abstraction

As my understanding, the concerns are mainly about the way that defines/constructs a graph/model by wiring operations, as EXAMPLE 2 shows in current spec.

If there is a consensus on model loading, a possible solution could be to allow creating a Model object by loading a URL:

partial interface NeuralNetworkContext {
  // Create a Model object by loading a file.
  Promise<Model> createModel(DOMString modelUrl);
};

With this, the EXAMPLE 2 would be changed to:

const modelUrl = 'url/to/ml/model';
const model = await nn.createModel(modelUrl);

Then the model object can be compiled and executed as same as EXAMPLE 3 and EXAMPLE 4 show.

What do you think?

[anssiko]

Thank you @jbingham for the proposal. Tagging @RafaelCintron and @gregwhitworth from Microsoft.

All - please review the proposal and provide your feedback in this issue.

I added this issue as a discussion topic for our next call that takes place tomorrow:
WebML CG Teleconference – 6 February 2020 - 15:00-16:00 UTC+0

The group has learned a lot since we last discussed this topic ~1 year ago in #3, so I feel now is a good time to revisit the issue. Given adequate support, I'll start the process to expand the Community Group's scope per the charter change process to explicitly bring this proposal in scope of this group.

We should definitely discuss this topic at our upcoming workshop in Berlin. It fits nicely into the workshop scope and touches many broad topics that have both near-term and long-term implications. It'd be great to have @jbingham give a lightning talk on the topic at the workshop. I encourage you to register and submit a position statement. Anyone else lurking on this repo should do the same :-)

[wchao1115]

Thanks @jbingham for putting together a very clearly articulated proposal.

I do agree that a loadmodel API is useful and I'm supportive of adding it. I also agree with @huningxin that it is complementary to the graph API and not a replacement of it, because from the design point of view, the graph API is just the next level of abstraction, and one that doesn't require a serialization format. Looking at it this way the model format is a convenience wrapper to help with packaging and distribution.

I think the more important discussion though is about whether the lowest-level primitive of this API should be at the operator level or something lower than the operator level, a question Jonathan rightfully raises in his proposal. This I believe is at the heart of this matter because this primitive is the contract that represents not only what the browser implementer must provide to the developers, but also what the underlying OS must provide to support the browsers. Also note that this issue applies even if we choose to have a single loadmodel API and without the graph API because the primitive would still live inside the file format that the loadmodel API would parse and subsequently execute.

I think although the operator-level primitives has its own share of evolution over the years, it does provide a reasonable contract between the provider and the consumer of ML. I'm one of the implementers of ML operators with DirectML so I understand this difficulties firsthand.

The operator-level primitives would allow the browsers the freedom to implement them as optimally as what the underlying OS can support while still producing consistent result across diverse OS and hardware platforms. Any ML expert will attest that nothing is more frustrating than having the same model behaving differently on different platform or hardware. In practice it'll be virtually impossible to test the model with every possible combination of OS and hardware, old and new.

The concern I have over a primitive, as a contract, at the level lower than the operator level, is that the lower you go, the harder the job of the browsers to provide reliable and consistent service will become. From the OS perspective, it is far more reliable to provide the browsers a contract that allows both the browsers and the OS to evolve in parallel over time; so as the new hardware or drivers show up, it automatically works with the existing browsers while the browsers continue to evolve and control how it wants to maintain or extend its commitment to the developers on its own timeline. This healthy ecosystem is hard to achieve if the browsers would insist on demanding a primitive at a level too low for the OS to do a good job at.

I think there are definitely considerations we can use in how we define the operators going forward so that it's more modular and easier to maintain over time. The current studies such as the conv2d compatibility exercise is helpful as a way to separate out core functionalities from the more auxiliary aspects of an operation already diversely defined by various API in the industry today. But to go lower to the level of a compiler's intermediate format maybe a bridge too far in my opinion.

[gramalingam]

I agree with @wchao1115 .

(a) Load-model vs the so-called "graph API" (I prefer to think of it as a model-builder API). A load-model API seems useful. But I agree that there is no conflict between the two, they can co-exist, if we think they are both useful. But if we decide to use just one of them, I am fine with that too.

(b) The "level" of the API: do we use a small set of primitive ops (what Jonathan calls instructions) or a large set of "big" ops? This definitely seems to be the harder, and more controversial, question.

With regards to the second question, let me mention one compromise that was adopted in ONNX (and I believe that the MLIR community is also thinking about something similar): ONNX has a notion of "functions": these are operations with a precisely defined semantics as well as a "lowering" into some expanded-code in terms of other operations. The idea is to enable an implementation that recognizes a function FooBar can use an optimized kernel/implementation for the op, while other implementations that do not recognize FooBar can simply replace FooBar by its expanded form. I think this is worth considering.

[jbingham]

Thanks, everyone for the comments.

@huningxin , yes, i definitely think we can reuse some of the API work you've done already -- and we should, wherever it makes sense. I'm not sure if the compilation step is necessary for a model-loading API, or if it could be a no-op. If it's needed, the way you've defined it looks pretty good to me.

Thanks @anssiko , I've sent a position statement for the Berlin meeting.

And @wchao1115, thanks for the thoughtful comments. I agree that these APIs are not mutually exclusive, and a key question is what level of abstraction the model format is at.

Yes, it would be possible to implement a model loader API on top of a graph API, at least if the model format is a graph of operations, like ONNX or TFlite. If the model format is lower level, implementing it on top of a graph API might require decompiling the model into a graph of operations, and then recompiling it before executing. I'm not sure if that would always be possible, and it could incur some performance hit. Not sure how big.

And thanks, @gramalingam , for introducing the concept of ONNX functions. I'd like to learn more about that. It sounds like this might address the concern about custom ops needing to be defined in javascript, by providing a way for them to be lower level than that.

There are definitely pros and cons to big ops and lower-level representations. Let's discuss further!

[huningxin]

yes, i definitely think we can reuse some of the API work you've done already -- and we should, wherever it makes sense.

Thanks @jbingham !

I'm not sure if the compilation step is necessary for a model-loading API, or if it could be a no-op. If it's needed, the way you've defined it looks pretty good to me.

IMO, in a model-loading API, the loaded model is supposed to be in a platform independent format, so developers would be able to inspect its representation. After compilation, the compiled model would be optimized by optimizer based on compilation preference. And its representation would be platform dependent and opaque to developers. So decoupling loaded model and compiled model seems to make sense to me.

[jbingham]

Yes, agree that model formats need to be platform-independent.

Does the browser need to do any compilation and expose a javascript API for compilation, or is it sufficient to validate the file during the load step, and then hand it off to the system ML compiler/runner, like WinML, TFlite, or CoreML?

Separate question:
If the system does not have an ML execution engine installed, what should the browser do?

Options:

1. Throw
2. Use a bundled fallback ML engine that's slower but can run the model
3. Provide a prompt to install a native execution engine

1, 3. There's precedent for the browser saying no for things that aren't available, like media formats that aren't installed. An alert could be provided with a link to a separate bundle that implements the runner.

2. For maximum portability, it would be nicest if there was a fallback.

But browser vendors are not ML experts. Realistically, they would have to bundle code written by other teams. Eg, the Chrome team would look to the TensorFlow team for help, the Edge team would ask the WinML team, Safari would ask CoreML, and Firefox would look to one of the ML frameworks.

Any bundled implementation could get out of sync with the system version. It would likely be big, increasing the browser download size.

If the default implementation isn't performant, it might not just be slower; it might be infeasible to run. A model that takes 10x longer to execute, or soaks up all of the memory, might be best not to run at all. A server-based fallback, or a lighter model, might be required in practice.

Anyway, just some things to think about!

[huningxin]

Thanks @jbingham , my comments,

Yes, agree that model formats need to be platform-independent.

Great. So do you think the Model interface can expose some platform-independent properties of a model, like name, format, version, inputs and outputs etc.,?

For model file validation and properties query, browser can also leverage OS ML API, e.g. LearningModel of WinML and FlatBufferModel of TF-Lite and MLModel of CoreML.

Does the browser need to do any compilation and expose a javascript API for compilation, or is it sufficient to validate the file during the load step, and then hand it off to the system ML compiler/runner, like WinML, TFlite, or CoreML?

I agree browser should leverage OS ML compiler/runner to compile the model. The Compilation interface allows configuring the options/preferences that help OS ML compiler decide the target device and optimization strategy, e.g. LearningModelSession of WinML allows to configure deviceToRunOn and learningModelSessionOptions, Interpreter of TF-Lite allows to configure options for GPU/NNAPI/DSP delegates, MLModel of CoreML allows to configure MLPredictionOptions

So IMO, this process would have two steps: 1) load model: transfer the model file from network, leverage OS ML API to verify model file, create in-memory model and populate its properties. 2) compile model: configure compilation options, run OS ML compiler with model and options, get the compiled model.

[wchao1115]

@jbingham I agree with @huningxin that a separate compilation step is necessary and that it should belong to the OS layer. In practice, unless in a naive implementation, the OS rarely consumes the model as-is, but rather transforms it into a form that's more suitable for the target hardware before executing it. There are also situations where the OS is unable to fulfill the request on a target hardware due to mismatches between what the model requires to run and what the target hardware can support. In this situation, the OS would fail the request, and the browser or the javascript code should provide a fallback option.

Related to fallback options, I believe the browser must provide a reasonable fallback on the CPU by default unless the default behavior is overridden by the developer. There are cases where the model must be executed at a certain performance level or the user experience may not make sense e.g. a word processor web app running an NLP spellchecking model on the client. In cases like this, falling back to a suboptimal performance on a weak CPU due to the lack of sufficient GPU support may directly degrade the end user's experience to the point that the feature must instead be disabled.

[jbingham]

@huningxin and @wchao1115 : Thanks for clarifying.

To recap: A compilation step will happen, implemented at the OS level. Model validation will also happen at the OS level.

I understand that the OS will use lots of ML model properties for validation and compilation. Now I'm trying to understand what is an implementation detail at the OS level, what the browser needs to know about, and what web developers need to know about.

Questions:

• What model properties does a web developer need to know about? These properties would need to be exposed in the javascript API surface, as properties of the ML model object.
• What model properties does the web browser need to know about? These properties would be part of the spec, but not necessarily exposed in javascript.

Maybe ideal would be if it's all at the OS level, and things are nice and simple for web developers and browser vendors. I expect that there will be some things that need to be exposed at those layers though.

Next topic:

My hunch is we'd want any CPU fallback implementation to be provided at the OS level as well. Reason: otherwise, if the browser has to ship with a full-functioned, compatible, performant CPU implementation, the requirement would be a big burden for browser vendors, because they would have to implement or bundle a very complex piece of code, which could be large on disk too.

Does that sound right?

[wchao1115]

@jbingham At a minimum, the web developer would need to understand how to use the model. This includes how to feed the model with the kind of data it is trained for, and how to interpret the model's output correctly.

The web browser, on the other hand, needs to be able to work with the underlying OS API to satisfy the model's requirements, which includes how to represent the data format used, how to carry out the operations expressed in the model graph, and how to respond to execution failures such as a missing functionality in the platform, to name a few. Again, this is orthogonal to the Javascript exposure i.e. the browser needs to be able to handle all of these aspects of model execution regardless of the exposure because all the OS API are OS-specific and the browser API is not.

I think there are good points to be made either for or against a CPU implementation in the OS. But I tend to agree that it may be more appropriate to leave it up to the OS how it should want to supply its CPU implementation to the browser e.g. ARM-based vs. Intel-based etc.

[jbingham]

@wchao1115 Sounds good. Yes, agree the browser needs to know enough to call the underlying OS APIs, with whatever information they require. I guess it's going to vary by OS, and the browser version running on each OS will need to deal with it.

There are a few work streams here:

• Defining the developer-facing API surface, which could eventually go into an official Proposal and a Web ML Working Group.
• Getting an end-to-end prototype working with one browser and OS. @huningxin is maybe done with this one :)
• Defining a model format. This group is looking at some of the issues to standardize op representations, and could potentially look at MLIR as it evolves. (It's still very early)
• Figuring out what's needed for the browser to talk to CoreML, WinML, TFlite, TensorFlow. Maybe the code for this translation layer can be shared by all the browsers, so that the ML API works on any browser running on any OS. And then if a new format comes along, all browsers can benefit.

[gramalingam]

There is already an API (along with ongoing discussions) about Compilation/Execution. I assume that "Loading a model" needs to just introduce an alternative way to create a Model (by directly loading, instead of constructing it via a builder API), and that it should be able to share the Compilation/Execution API?

[huningxin]

@jbingham , thanks for laying out the plan.

• Getting an end-to-end prototype working with one browser and OS. @huningxin is maybe done with this one :)

It makes sense to me.

The existing WebNN Proof-Of-Concept (POC) is based on Chromium and implements the model-builder API on top of Android NNAPI, Windows DirectML and macOS MPS/BNNS.

For the new model-loader API prototype, it would be based on WinML, TF-Lite and CoreML etc.,. So I'd like to propose to extend the existing Chromium-based POC and start with WinML.

What do you think?