Add KHR_interactivity draft

[lexaknyazev]

Rendered version: https://github.com/KhronosGroup/glTF/blob/interactivity/extensions/2.0/Khronos/KHR_interactivity/Specification.adoc

[bhouston]

Great work! What sections are next? Also what could we release as a draft extension for SIGGRAPH 2023?

[aaronfranke]

If we have Pi, we should also have Tau. https://tauday.com/tau-manifesto The value Tau is the circle constant (equal to 2*Pi, or rather, Pi is half of Tau). Using Tau usually results in more readable code. Tau is supported in many programming languages such as C#, Java, Python, GDScript, Rust, Unreal Blueprints, and more, so it's useful for interoperability with other languages, especially Unreal Blueprints which is conceptually similar.

===== Tau

[cols="1h,1,2"]
|===
| Type | `math/tau` | The circle constant, the circumference of the unit circle in radians.
| Output value sockets | `float value` | 6.2831853071795862
|===

[aaronfranke]

In case more justification is needed, the KHR_interactivity document currently contains Degrees-To-Radians (math/rad) and Radians-To-Degrees (math/deg). These allow converting between degrees and radians. Tau can be used for a similarly important purpose, dividing or multiplying by this number allows converting between turns and radians, where 1 turn is τ radians. Tau can be used for more than just this, but the point is, if math/rad and math/deg are justified as a part of KHR_interactivity for converting angles, then math/tau is also justified for converting angles.

[Jmangles]

While writing unit tests I noticed that variable/interpolate has no caveat for types that have no "intermediary" state. For example it shows "T value" suggesting any type can be used but interpolating from false to true, integer 0 to 1, etc isn't a defined process.

I searched around the spec for "interpolate" to see if I could find a general mention of what to do in this case but wasn't able to find one. For now I'm assuming that interpolate can only be used on floatN and floatNxN types.

[lexaknyazev]

While writing unit tests I noticed that variable/interpolate has no caveat for types that have no "intermediary" state.

From the operation description:

If the referenced variable is integer or boolean, the node is invalid and the graph MUST be rejected.

[robertdorn83]

What's the status with the quaternion math op specs :)

[hybridherbst]

@lexaknyazev If you would be able to update this PR with the quaternion nodes that would be well appreciated! Thanks

[robertdorn83]

hey @lexaknyazev, is there a plan for adding something like quatLookRotation? or do i missing a workaround with the new quat nodes?

[jsorel]

Feedback from a GLTF user and, GLTF java binding developer:

While I can understand the choice of graph structures to define the operations, unfortunately I believe it is a short/mid term solution :

• because graphs gets unreadable when they get larger
• increasing boxes/operation types over the versions.
• bad integration in IDE

Has a more developer version been studied ? like LUA with a dedicated function set ?
lua has been in the game development industry for very long, and is also used in many tools and international standards because of it's easy learning curve and simple integration in existing tools.
Using the lua syntax would allow a direct interaction in IDE, many developers and game designer knowing it already.

I understand adding script support in the format is raising the bar a lot, but I believe it's still better then a cheap graph solution.

[robertdorn83]

@lexaknyazev , was there any discussion in the past about some color nodes, like create a color with a HSV node and vice versa. Can also be very practical for interactivity stuff :)

[taye]

Thank you to everyone working on this proposal. I'm building an AR effects library and need a system for programming interactivity. While the WASM proposal is compelling, for my own project I'm more interested in using a format that's easier to inspect without running a debugger and that's generally more intuitive to people who aren't experienced programmers. Visual scripting systems keep appearing in video game and AR editors, and I'd prefer to follow a standard than to make yet another system from scratch.

I've implemented KHR_interactivity for glTF-Transform and I've made a partial runtime implementation for Three.js using behave-graph. I've also made a custom extension which depends on of interactivity for face landmark effects. You can find the code here on Github. I'd like to develop my Three.js runtime further, but I'm not sure what are the current states of behave-graph and glTF-InteractivityGraph-AuthoringTool.

[taye]

@robertdorn83 There's a proposal for procedural textures, but it seems to be independent from interactivity graphs. It would be great to have only one graph-based scripting language spec. Perhaps it makes sense to split the scripting language specification from the platform interface specification.

[javagl]

There's a #2381, but it seems to be independent from interactivity graphs. It would be great to have only one graph-based scripting language spec.

I do/did not have the details of the procedural texture specs on the radar. But if does not use the same structures as this one, then this indeed raises a bunch of questions.

Perhaps it makes sense to split the scripting language specification from the platform interface specification.

Due to the lack of familiarity with the other spec, I cannot say where the right place might be for this "split". But given that most of the spec here is about the graph, its operations, and its execution (and only the last, tiny step is to ~"write some value into some property"), it sounds like there could be a huge overlap between this spec and the procedural materials one. Fragmentation on the level of the graph implementation could increase the implementation effort tremendously.

Note: I cannot say whether this is really technically feasible. But iff we could define a graph-based system that was powerful and versatile enough to describe interactivity and procedural materials, then it would be very likely also be powerful enough to handle any possible future extensions (like procedural geometry or whatnot...)

[lexaknyazev]

there could be a huge overlap between this spec and the procedural materials one

Procedural textures extension has different interoperability goals and constraints than this extension. Although it may sound convenient to unify them, there are many subtle operational discrepancies that could prevent that.

[emackey]

there could be a huge overlap between this spec and the procedural materials one

Procedural textures extension has different interoperability goals and constraints than this extension. Although it may sound convenient to unify them, there are many subtle operational discrepancies that could prevent that.

Agreed. Material graphs and behavioral graphs are different things. A material graph does not use control flow nodes or local variables, for example. A material graph gets inputs from the vertex attributes and textures of the actual mesh being rendered, and is evaluated at render time for that mesh, not during a physics/interactivity/heartbeat cycle between animation frames.

[Igor-dvr]

Is there interactivity Blender extension?
Any plans to implement the extension?

[robertdorn83]

@lexaknyazev I have some question about the animation/play node.
For a looping playback, "start" would be 0, and "end" would be positive infiniy. right?
But what if a animation should played reversed? Especially with looping.
Should it be "start" = animation.length and "end"= negative infinity?
But then, the spec says :
"If the startTime or endTime input values are NaN or if the startTime input value is infinite, activate the err output flow and skip the next steps."

[lexaknyazev]

But what if a animation should played reversed? Especially with looping.

To get infinite looping in reverse direction, set end to negative infinity and start to 0 (or any other finite value).

[robertdorn83]

@lexaknyazev

the term "index" for the input is not specific enough here. Which variable index? From the config array? Or the graph variable index?
Let's say whe have an config here with [3,6,9]. Would be the input names "3", "6", "9" OR "0", "1", "2" ?

[lexaknyazev]

@robertdorn83 The operation details located after the table make it quite clear that input value socket ids correspond to graph variable indices, that is 3, 6, and 9 in your example. We'll update the table for clarity.

[robertdorn83]

@lexaknyazev
Currently the readonly pointer for activeCamera contains only position and rotation. Would it make also sense to add matrix to it ? I know we can build a matrix out of position and rotation because its global space, but just for consistency with the node transforms.

Also, and we already talked about partly when i remember correctly, but it would be also more consistent to add the other gltf camera properties such as activeCamera.orthographic.xmag, activeCamera.orthographic.zfar, activeCamera.perspective.aspectRatio etc. These are all known standard gltf properties.
Also maybe adding activeCamera.isOrtho, activeCamera.viewMatrix, activeCamera.projectionMatrix.
My wishlist for christmas this year ;)

[Jmangles]

"goto to the step 11." Reads a bit odd. This step also instructs you to set isValid to false while Step 11 also instructs you to do the same

[lexaknyazev]

The step 11 checks if isValid is false; it doesn't set it.

[lexaknyazev]

The step 11 checks if isValid is false; it doesn't set it.

[lexaknyazev]

The step 11 checks if isValid is false; it doesn't set it.