QC Patches

[tobyspark]

Here is a survey of QC patches I made by trawling an old machine of mine, as a prompt for memory. A lot of the more interesting patches had a settings pane that did a lot, not shown here. Some of these were behind a flag (e.g. Log) and at least the Syphon ones are third party.

The actual list in my library is a lot longer, e.g. this misses all CoreImage, repository compositions, a bunch of spring animation and other Apple internal ones, etc. that are not ‘core’ by what I’m valuing here. And then there’s mine and other plugins, but those were more special sauce. Special mention to Kineme which had some could-be core stuff like command-line and data tools.

[vade]

The memories!!! This is great, thank you!!

[vade]

So sitting with this a bit some things stand out to me

1. As documented elsewhere, but worth surfacing as a discussion point, we dont have a Virtual type and type casting isnt really handled in Fabric

2. The Structure type plays two roles, as array and dictionary. We have a generic array, but no dictionary alternative. I think separating these is wise personally.

3. Settings, as you called out can do some work and change patch behavior in ways we cant right now. Need to ponder that

4. Mesh handling coupled with structure allows pulling keys / values out and doing work on more complicated data structures. This is tricky in Satin, as the main Mesh hosts submeshes (their own type) and would be akward to populate into Fabric? Im sure theres a way its just not obvious at this juncture.

5. Slew of patches I forgot about that are important (ive updated Node.md with some)

6. Macros - QC uses a ton of nested macros for various things, and they are re-entrant (nesting Replicate in Space for example). I like macros for iterator and repliate XYZ, but not for things like camera / fov / fog , etc? It makes it very hard to do certain kinds of maneuvers and im not connvinced its the most friendly. Im brought up of memories of havign to re nest and re architect patches due to macro nuances.

This has been super great to muse on! Thank you @tobyspark

[tobyspark]

Virtual – I'm not sure of the utility; I always set those nodes to a specific type. "Union type" ports seems a super good idea (as discussed elsewhere). And perhaps some notion of an opaque type might be useful, e.g. I might want to pass some custom data type around a family of custom nodes I've made myself.

Structure – Yep this seemed a little too cute to me at the time. Just have Array/List and Dictionary/Map (I guess we go with the Swift naming)

Settings – it's a small thing, but I loved that you got this bit of custom real-estate when you made a QCPlugin. Settings, yes, but also logo and whatnot.

Mesh – I defer to whatever makes sense for Metal / Satin. And being able to manipulate it all easily and performantly. 3D and advanced rendering was where QC was weak.

Macros - Probably. Fog just being something you drop into a scene makes sense, as what if you need more than one of these rendering environment features. But e.g. 3D transform conceptually makes a lot of sense as a macro, but that approach literally breaks up your view of the graph as you wrangle the sections around. I'm filing this under: I'll know it when I see it.

[vade]

So right now Fabric supports arbitrary types as node ports wrap 'generics' , you can pass literally whatever you want. Its more of a loose contract i have right now, so if we wanted audio, it would just be NodePort<MyCustomAudioSample>(name:"Input Sampes", type.:Inlet) or whatever - the underlying graph / node / node port infrastructure isnt tightly coupled to the data types we are making (ie its just convention) - just want to mention that.

One thing thats helped me work through this stuff : is there a thing you want to make / re-make to help focus work around concrete things? I prefer that over airy what if scenarios?

[tobyspark]

is there a thing you want to make / re-make to help focus work

Most immediately, the software the visual part of my Fuel a/v performance is made in is dead. I have the source to keep it alive for my purposes, but I’m not motivated to do any work there. I’d be much happier wrangling a modern code-base in Swift, and the effort to port both the show and what I need across might be worth it. (Or – a fool’s errand; time, resources, project traction etc.)

What I really need for that show is a something a bit like a cross between my old AppKit + QC app Screenrunner and Keynote. That might be best expressed as a *spark app with Fabric as QCRenderer, or perhaps some UI/document ideas integrated into Fabric itself. I’m not going to pitch that here, but I do think the idea has legs.

In terms of nodes and features, the needs for the above are relatively simple. A 3D world, with a lot of 2D → 3D expediency. Syphon in and out. I’d make a point-cloud node. And... rendering that looks good by default (e.g. anti-aliasing, translucency, soft shadows), ideally with a way to do post-processing volume fx.

What’s not simple, but really gets me going these days, is hybrid live camera stuff like this or this. Keijiro’s RCam (now v4) is both pioneer and reference here. That is beyond my graphical programming experience, though.

[vade]

Awesome. I love it.

So good news is loading up and rendering using Fabric / Satin is pretty easy in a Swift / App Kit for UI or Swift / SwiftUI for the UI -

I've tried to be reminiscent of the QCRenderer niceties there.

To make a SwiftUI view that renders a FabricDocument, you can do that like so:

App

@main
struct FabricApp: App {
    var body: some Scene {
        DocumentGroup(newDocument: FabricDocument()) { file in
            
            ContentView(document: file.$document)
        }
    }
}

ContentView:

struct ContentView: View {
    
    @Binding var document: FabricDocument
    
        var body: some View {
              SatinMetalView(renderer: document.graphRenderer)
        }

You can nab FabricDocument from the Fabric Editor code here, all it does is setup a default renderer and deserialize the graph format.

Thats the 'harness'

For the rendering, Fabric should be able to support Model loading for the car asset, shadows, a ground plane, and 'render to texture' via the deferred rendering node, and can now support depth of field and post processing.

I dont yet have Syphon but thats due to just not getting around to it.

I really do prefer working on a goal - if you send me assets (privately?) and give me a sense of how things were wired up we can baby step to it!

[vade]

As for some of the other stuff, ive done some work on realtime ML segmenting you can see here in v, which uses the same underlying node engine as Fabric

https://www.instagram.com/p/DMCT-KBsGry/?img_index=2

I plan on adding the some of that to Fabric :)

As for the other tooling like RCam etc we can get to it!

[vade]

messing with Sketchfab model (USDZ) and lighting / shadows with, depth of field :

[tobyspark]

...and all without breaking a sweat, I have no doubt. (Or rather, the fans spinning up)

I’m currently on a / blown-past-a writing deadline so I’m heads-down on that. I will download the repo next week. I have a plan (ish). I’ll be in touch.

[vade]

All good!

[vade]

@tobyspark - can you remind me how Signa and Pulsel worked and their input / settings options?

I cant find good resources online that talk about them, and i recall them being very powerful but for the life of me i cant find good docs.

[tobyspark]

[vade]

Amazing thank you, this is SUPER helpful

[tobyspark]

I think peak QC is probably this – right clicking to set a patch’s timebase to external, which creates a Patch time input port.

[vade]

Absolutely, this is also a thing i want to expose, we'll get there!

[aleith]

I never really understood why people used that to be honest!

One idea that was central to Vuo's initial spec design was that one issue with QC was that, even with Lazy Evaluation, it had to resolve the entire graph per frame, so if there was a particularly slow set of patches, they would drag the entire composition to a slow FPS rate even if the work being done in those patches didn't need to happen at the desired screen refresh rate. that led to Vuo having a push rather than a pull methodology, it meant variable execution times for nodes would allow asynchronous execution across the graph. they played around with this quite a bit before choosing to go with push not pull, with the caveat that a push base execution mode would allow for a variable-speed or async pull methodology to be built on top of it that wouldn't have the limitations QC had WRT slow patch execution dragging down the entire comp. Even custom non-blocking Kinmem patches to load images and things like that in didn't solve the QC speed issues.

Anyhow, long way of saying, if you could somehow not do what QC did in respect of total graph execution for each frame then that would be ideal.

[tobyspark]

I exposed the timebase of a node as an input a lot. And we’re both right, in the sense that QC was a tool different people could use for different things. For example, I had a lot of things which had “animating in”, “on”, “animating off” state. Driving a timeline node by a stopwatch hooked up to the state logic made most of that work.

Per the docs, push vs. pull is a locked-in decision at this point. FWIW, I’m with Anton on pull. On the particular point of execution being brought down by a “heavy” node, my take is let’s make sure there’s a sensible way of doing heavy work async, and keep it simple otherwise. Though really right now it’s a question of what Anton has time and inclination for =]

[vade]

So theres a few paths in theory with slow nodes

1. Async updates for a specific node - ie, func(x) takes a long time, we kick it to a background thread, single or double / triple buffer it if we can, and just return the latest whenever we get it. This assumes the function or use case has some affordance of input and output being decoupled in time and not synchronous.

2. Building some method to evaluate the graph in parallel chunks, with execution barriers as sync points where needed (ie a graph has multiple trees, and vertexes are sync points). This should be doable but requires a LOT of work, and still wont ameliorate a single node from slowing SOMETHING DOWN. I'd love to tackle this for a 2.0 or 3.0, but its not happening any time in the future, and requires preliminary work in Satin at a minimum to add threaded support to decouple rendering / update loops.

3. if you have a slow node, and you need it asap for the next step of evaluation, i mean, logically, you just have a non realtime process at that point. What is expected at that point other than a nobel prize at solving some hard problems™? Im half joking ofc, but in honesty that use case just simply means no system could accommodate it, node based or not. If your func(x) doesnt fit into your render budget, no amount of hemming and hawing is going to fix it.

Re Timebase and Execution Mode, ive added support for those in the latest PR, but they arent leveraged yet by the engine or UI

[vade]

hey @tobyspark

:)

We have a prototype for Settings now (node's vend a SwiftUI view which is placed in a containing view in a pop over). Nodes can control popover size, and dismissal happens only when you want (unlike normal pop overs)

We also have what appears to be dynamic port addition and subtraction working - so for example math parser seems to behave more or less how we want it to save for some minor redraw shit i need to deduce.

Getting close!

[tobyspark]

That is really great. Better than QC.

And you’re getting through the nodes I had in mind to make, which I can hardly feel churlish about. I... I’ll send you a screenshot that may make you laugh and somewhat explain me being quiet this past month or two.