High level vs low level

[gregwhitworth]

Hey everyone,

While reviewing this PR I had an issue but it's a horizontal issue that should be discussed outside of that PR, maybe a good agend item for our first telecon @anssiko. I have had very few folks desire a low level API for this based on the use cases I've been able to procure.

Additionally, I'm not sure that some of the items that are denoted as low-level in the linked commit I would define as low-level, rather that they are not using the pre-trained models exactly as is. In the discussions that I've had with many folks across Microsoft, the majority of client side use cases that people are looking at for production scenarios would not require a low level API. Effectively this comes back to the issue of who the customer is, a library author or web developer solving the problem. There are pros and cons to each approach we take and I don't know if we need to have a fundamentals line in the sand but I do think I'm seeing a larger desire for higher level APIs than a lower level one. Thoughts?

[anssiko]

@gregwhitworth, thanks for initiating this discussion! It would be very helpful if you could share some of the client side use cases you've procured.

I don't think we can yet confidently draw a line between low level and high level use cases for our purposes. We need more input. Our current charter explains the layering through the Extensible Web Manifesto principles. When the charter talks about low-level capabilities it attempts to refer to the EWM definition:

To enable libraries to do more, browser vendors should provide new low-level capabilities that expose the possibilities of the underlying platform as closely as possible.

As said, the use cases PR is an initial attempt and I hope the group will help refine it further. We probably learn more when we derive requirements out of the use cases too.

Maybe it should be noted, that for many high level use cases a library makes a lot of sense too. For example, face-api.js solves a high level use case, is implemented on top of the TensorFlow.js core API that itself could be a consumer of the Web Neural Network API. Out of scope for this group would be, however, to define a dedicated API for face detection since that's considered a high level use case (incubation is underway elsewhere as the Face Detection API). Borrowing the EWM terminology, the Face Detection API could be explained in terms of the low-level capabilities exposed by the Web Neural Network API.

This is exactly why I believe discussion around use cases (as opposed to jumping right to the "solution") will be time well spent as it forces us to put the needs of the consumer first, and identify who the (primary) consumer is.

You pointed out some low-level use cases are not using pre-trained models as is. That's good input and should be fed into the PR.

(Any clarifications or suggestions to the charter itself, please open an issue against the charter repo.)

[gregwhitworth]

We spoke about this on the telecon today, and I wanted to try and articulate my thoughts here a bit further. There are effectively three routes we can take with this API:

1. Operator
2. Load & Run Model
3. Solution Based

Non-normative definition of each:

Operator - Primary Customer [Framework Authors]

This would map more similiar to what is available in the Intel POC, and is somewhat similiar to what data scientists are used to utilzing in Tensorflow, PyTorch, WinML, etc. The current list of supported operators in the POC is currently here - but that does not mean that those are the only operators that need to be in the spec.

• Pros:
  • Scopes in the necessary operators that are required to support the usecases on the web.
  • Allows it to be more extensible and handle more complex capabilities
• Cons:
  • Has a pretty steep learning curve
  • For most web developers they'll have to include a library in order to utilize the technology

Load & Run Model - Primary Customer [Web Dev]

This would allow a web developer to load a model, and pass in inputs to be evaluated against the pre-trained model. A quick example of this would be something like this:

var evaluation = "some text"
var modelA = await loadModel(model.onnx);
var output = runModel(model.onnx, evaluation);
var modelB = await loadModel(modelB.onnx);
var answer = runModel(modelB, output);

• Pros:
  • Simple & intuitive
  • Will still require scoping of operators so implementors can plumb through the necessary operators to support various model formats for hardware accelleration
  • Will not require a JS library for model evaluation
• Cons:
  • Is not extensible without pre-training additional models as to affect the final output

Solution Based

This is taking the common use cases, as have been proposed in this PR and beginning to define JS APIs that solve those problems.

• Pros:
  • The most intuitive of the three
• Cons:
  • Is not extensible
  • Would require standardization on the return results from the models which will limit the capability of the models

Path forward

There is nothing stopping us from doing all three, but I personally prefer for us to explore specifying and implementing the operator and load model approaches. This aligns with the charter of being "low level" and will enable JS libraries to build out the high level solutions while allowing the platform to cater to use cases for building a framework or for evaluating an input against a pre-trained model.

I welcome any feedback you all may have.

[jonathanding]

@gregwhitworth Thanks a lot for raising this topic and personally I also think that we need a set of APIs (temporarily called it as high level APIs) to cover probably typical requirements of web app developers, i.e. load and run the pre-trained model.

The major benefits of High Level APIs are:

• Simple, intuitive to match the typical needs of web app developers (not framework developers)

• API is Stable and less dependant to domain specific world (machine learning in this case). It doesn't need to update the specification from time to time to reflect the latest achievements in machine learning world, such as new operators, etc.

• Faster to market. It has opportunity to land to browsers faster, by mapping the quite few apis to native stack. Even it isn't supported by browser yet, it is very easy to provide polyfills based on tensorflow.js and onnx.js etc.

High Level API for ML is about Model, similar as Video in HTML

A lot of high level usage (from web app developers) of models of machine learning, e.g. neural network models, are quite similar as media files such as video clips in Web i.e. <video>.

The high level APIs for ML would focus on Models, similar as API/Tag of processing videos. For ML model and a video clip in Web:

• They both could be considered as a kind of Resource

• Such Resource would have a set of high level JavaScript APIs, e.g.

  • For video it is HTMLMediaElement which consists of methods like play() and pause().

  • For ML model we could have something like keras.Model, with methods such as load(), compile(), predict(), fit() etc.

• Such Resource could have different format. The actual format is domain specific thus is defined by various standard bodies other than W3C.e.g.

  • For video it could be H.264 or AV1 etc.

  • For ML model it could be tensorflow, ONNX, caffe etc.

• JavaScript could be used to access the data of such Resource

  • For video there are canvas related API (at this stage) to get the frame data.

  • For models we even don't need standardized APIs. With the known schema of the models (e.g. Tensorflow / ONNX), we could have JavaScript libraries providing parsers, readers, writers to parse and even modify the model.

Necessity of High Level APIs for WebML

As you mentioned, typical machine learning usages in Web app need to be simple and intuitive

1. Typical Usages in Web app

   • pre-trained model -> Load and inference in web app

2. Less Typical in Web app

   • build, rather than load, the computation graph

   • significantly modify the topology of a loaded model

   • perform heavy training, rather than some lightweight fine tuning

A good design of High Level APIs could make 1st category (Typical Usages) much simpler and more elegant, while still support 2nd category (Less Typical).

Furthermore, high level apis could be easily hardware accelerated by almost directly mapping to underlying native software stack for machine learning, since they have model based APIs too.

Lastly, with high level APIs, we don't need to define many solution based APIs one by one (e.g. face detection, action recognition api, pose recognition api etc). They are simply models loaded and then inferenced.

Proposal

High level API could be defined by following wisdoms of <video> and keras.Model.

The model could be created from a URL in JavaScript, or defined by a HTML tag similar as <video>

<MLModel 
    id="ml-example" 
    source="a.com/a.model"
    preload="true" other-attrs="..." />

// Model is created in JavaScript from remote
let model_by_js = WebML.model.fromSource("a.com/a.model");

// Model is created in JavaScript from a memory buffer / array
let model_by_buffer = WebML.model.fromBuffer(...);

// Or defined in HTML 
let model_by_tag = document.getElementById("ml-example");

// preload already, so just use it
await model_by_tag.compile();
await model_by_tag.predict();
await model_by_tag.fit();

Other Considerations

Model Format - whether it need to be specified

This could be an open that whether the format should be specified. We have two options:

1. Specify a format for model in standard

   For example, agree on using ONNX. Web app developers could either convert their non-ONNX models to ONNX offline. Community could also build some converters in JavaScript as libraries, so that these models could be dynamically converted inside web app, which are more friendly to web app developers.

2. Similar as HTML5 video, no restrictions in spec, but allow multiple ML engines (similar as codecs) registered into browser.

   Developer may provide the same model but in different formats, browser use the first one that it supports.

   <MLModel id="ml-example" pre-load="true">
       <source="a.com/model.tf" type="tensorflow"/>
       <source="a.com/model.onnx" type="onnx"/>
       <source="a.com/model.coreml" type="coreml"/>
   </MLModel>

Query APIs

Similar as Media Query, there could be APIs to query ML capabilities of browser, e.g. what format (and version) it supports, how about the hardware acceleration etc.

Construct / Modify the Model with High Level APIs

As mentioned above, there are actually NO direct high level APIs to construct/modify a model. This is different vs. low level APIs which could directly compose the graph by adding operators etc.

However, with known schema (encoding format) of the model, one could still use pure JavaScript to directly construct/modify a model in raw format in memory buffer, and then load it from buffer to form a model for predicting / fitting. And it is expected that community would emerge JavaScript libraries to provide helpers (e.g. onnx writer etc.) to facilitate such internal manipulation on raw model.

[gregwhitworth]

@jonathanding thanks for response, I agree overall that we need a loadModel level and operator. I prefer to refrain from saying high and low level as I think both are providing web developers with what they want from the platform.

I do not think that we need to be focused on the formats that need to be supported at the present time, but the operators in which will be supported (we'll need to know this for both a loadModel and operator approach).

As we begin to identify formats that support these operators we can use them as examples and still leave this specification from defining which formats a UA needs to support as any format that supports the defined operators will be covered.

I am less inclined to support the notion of a HTML ML element, as unlike the media components we've been using for analogies the models do not inherently have a semantic meaning and will require JS to be of any use to the web developer and their users.

Thanks for the feedback, any other thoughts from others?

[Kangz]

In the spirit of the Extensible Web, I believe the API should be exclusively low-level and probably not even know about the concept of loading ML models.

There is no reason to expose a new HTML element because like @gregwhitworth pointed out, the model doesn't carry semantic value by itself and needs JS to be actually useful. Adding the concept of ML "codecs" to the Web platform won't make it easier for developers to do ML on the Web put will instead increase the complexity of their application because they will have to make choices. Video codecs are different because they represent actual hardware video decoding capabilities (as well as historical reasons).

The usability aspect of the loadModel approach is very compelling: in Web development today this type of functionality can be added via a single npm command. There are real drawbacks in that "operators" are needed for existing frameworks anyway and loadModel would add even more complexity to the Web platform on top of them. I don't know the market too well, but there doesn't seem to be a single agreed upon file format (coreml vs. TF vs. ONNX vs. ...) so choosing one of them will be a difficult political decision. There is also the issue of upgrading to newer version of the format and some versions only being supported on some browsers. Also debuggability.

[DanielMazurkiewicz]

Since on Thursday will be an online meeting want to point here another proposal for the "low level" API to discuss:
https://github.com/DanielMazurkiewicz/WebAI

[RafaelCintron]

I think we should concentrate on inferencing scenarios for v1 of the API. Training can come later.

I agree with @Kangz and @gregwhitworth that an HTML element is not necessary. The API can be a pure Javascript API like WebGL and WebAudio are today.

Both a loadModel-style (high level) API and operators-style (low level) API require "political decisions" on the part of the group. Even an operators-style API requires that we choose which operator definitions to align with. Different frameworks have different definitions.

As we evolve the API over time, there will need to be a way to query the API for the supported version. In case of a model-style API, this will be the model version. For an operator-style API, this will be and operator set version.

For both API level types, aligning to an existing organization such as ONNX is an attractive proposition to me. ONNX has been designed to be a common interchange format between existing frameworks. Multiple companies collaborate on formats and operators: Amazon, Facebook, Microsoft, etc. All of the major IHVs are a member. There already exist open source converters to and from ONNX and Tensorflow as well as CoreML formats/operators.

[tomoyukilabs]

For operators-style API, I agree with @RafaelCintron that alignment with a common interchange format like ONNX seems a good approach. It is more practical if developers can create custom operators that have not been supported by ONNX yet, for example.

Also, I agree that a way to confirm compatible version will be necessary for both model-style and operator-style APIs. I guess a method to check available formats will be needed as well if we want to allow a model-style API to support multiple formats.

[DanielMazurkiewicz]

Training can come later.

True, but backpropagation training algorithm is very well established algorithm and I don't see a single point why shouldn't it be implemented in first version of standard. Even if we will not make it to first version of standard I would still opt for making clear statement about how training api will look like - this will let us to avoid making shitty standards because of we didn't think of something and then have to adjust to what we just have.

For both API level types, aligning to an existing organization such as ONNX is an attractive proposition

Aldo there is plenty of more or less open standards, there is no clear winner and there is no clear leader (as it is usually with emerging technologies). So I'm totally against favoring any of existing standards in terms of using as web/javascript standard. We have a chance to make something good here, something that fits well javascript world. And I want to really opt to make it simple and designed for web.

Someone here metioned WebGL as example. WebGL is sort of good example of adopting existing non JS standard (openGL) that no one really uses directly :-) - tons of 3rd party libraries are needed in order to be able to work with it conveniently. It is nice to have super low level api access, but in terms of web would be much nicer to have something like Three.js API or a-frame built into a browsers and optimized and running at native speeds. Don't you agree?

It is also pity that no one before mentioned mature ML library made entirely in Javascript as something to start with for the web standard. :-) Have you heard of or seen brain.js library? This is really made by javascript guys for javascript world! :-) Maybe it isn't perfect, but as standardization group we don't have to copy 1 to 1 existing solutions, we can take best from them and compile into a standard. Look how simple and flexible is proposal API I made based on brain.js: API proposal

Also want to aware you that there is ongoing standardization project called webgpu with one of aims of providing access to gpu vulkan computations and spir-v, so keeping that in mind I wouldn't go too low with providing any sort of low level computation API in ML standard, it will be simply wasting and doubling work in our efforts, I would really focus on providing simple, nice, extensible ML API for well proven algorithms.

Cheers!

[RafaelCintron]

True, but backpropagation training algorithm is very well established algorithm and I don't see a single point why shouldn't it be implemented in first version of standard. Even if we will not make it to first version of standard I would still opt for making clear statement about how training api will look like.

The majority of the people we've spoken with about client side ML APIs are interested in inferencing. For this reason, I would prefer that we leave training out of the initial version. I agree anything we design for v1 should keep training in mind so we don't paint ourselves into a corner when it comes time to tackle it.

Aldo there is plenty of more or less open standards, there is no clear winner and there is no clear leader (as it is usually with emerging technologies). So I'm totally against favoring any of existing standards in terms of using as web/javascript standard. We have a chance to make something good here, something that fits well javascript world.

To be clear, I am proposing that we adopt ONNX for its model formats and operator definitions.

I agree the WebML API this group defines should be a first class JavaScript API.

Also want to aware you that there is ongoing standardization project called webgpu with one of aims of providing access to gpu vulkan computations and spir-v, so keeping that in mind I wouldn't go too low with providing any sort of low level computation API in ML standard, it will be simply wasting and doubling work in our efforts, I would really focus on providing simple, nice, extensible ML API for well proven algorithms.

WebGPU is slated to support compute shaders. However, compute shaders is one avenue to implement operators. There are additional platform APIs such as DirectML (on Windows) and others on Apple and Google platforms that will not be surfaced by WebGPU. The WebML API would make those additional capabilities available to web developers.

On devices where specialized hardware is not available, the user agent can run the web developer's graph by falling back to either compute shaders or a CPU backend as appropriate.

[DanielMazurkiewicz]

The majority of the people we've spoken with about client side ML APIs are interested in inferencing. For this reason, I would prefer that we leave training out of the initial version.

I guess we were talking just to different people, :-) because majority of people I've been talking to would be pleased with training capabilities that for example would let them adjust NNs "on the fly" (in gaming industry). And frankly saying - have you ever coded back-propagation training algorithm? This is really not a "rocket science", also plenty of open source implementations laying around.

To be clear, I am proposing that we adopt ONNX for its model formats and operator definitions.

To be clear - I'm saying that we don't need full set of operator definitions now, only basic, most common types of neurons/layers and nice, flexible, JSON based, not over complicated, model format that can be easily maintained by dev's and safely (no JS code inside models) exchanged between clients, servers, machines...

And to be clear again here - I'm not against of providing ultra low level api for ML, I think it could be added somewhere in future versions of standards and only after some experience with base version of API (it might simply end up that base ML api is just fine for 99.9% of use cases).

I think covering all these ONNX operators in web ML standard is pointless and unnecessary over complicating web ML at this moment.

On devices where specialized hardware is not available, the user agent can run the web developer's graph by falling back to either compute shaders or a CPU backend as appropriate.

I think every computation device should be for web ml first class device for running ML. The only sort of "falling back" could be only from and to sort of "simulation modes". Check "WebAI.getCapabilities" method from my API proposal where you can get all (or selected by required capabilities) computation devices capable of running NN and then decide on which of them to run NN.

If you take into consideration that modern computer systems, mobile processors, SoC's can contain multiple GPU, CPU, DSP (including all sort of "vector units" or "ML units") falling back just from GPU (or other "dedicated" device) to CPU makes no sense, when you can use them all simultaneously. (btw. there are even USB sticks offering computation power, so why not allowing to use them as well?)

Going further - if we will stick to simple JSON model format (and not mess with javascript inside models and not over complicate models) there is much, much higher chance that such a variety of computation hardware (and software computation libraries) will receive JSON model dedicated compilers/interpreters due to low implementation complexity. And that also means much higher chance for quick standard adoption.

Resuming: I really vote for sort of MVP standard that will do it's primary ML job

Cheers! :-)

PS. For the reference of what I mean by saying "simple JSON model format" - I made some examples in my API proposal.

[anssiko]

A word on scoping and coordination with my chair hat on:

• Scope of Work section of our charter defines the group's technical scope its participants have agreed to upon joining.

• Out of Scope section currently says: "Training capabilities are out of scope due to limited availability of respective platform APIs." Training API is a possible candidate for inclusion into a future "v2" API and we keep training in mind while designing the "v1" API focused on inferencing.

• Coordination section notes: "The WebML Community Group should coordinate with [WebGPU CG] to avoid overlap." We're lucky to have WebGPU participants in this group, including WebGPU CG co-chair @Kangz to help with that coordination.

• Amendments to this Charter section defines an agreed upon process for changing the group's scope.

Thank you all for great proposals and productive discussion to date.

[DanielMazurkiewicz]

@anssiko Thank you for reminding out of scopes!

What are procedures to change scope and out of scopes? Because as far as I know community the very first lib/shim to web ML will be adding training feature :-) I think it really makes no sense to leave it behind (unless ONNX will be adapted as standard - then it will be really hard to implement it)

EDIT: I'm thinking it over - it is totally WRONG idea of providing ML without training capabilities, because it will make standard totally dependent on some existing solutions. Whoever proposed it - it was EVIL proposition :-)