Make `ScriptInstance.call` and company re-entrant

[Mercerenies]

Related to #501, though ScriptInstance is a bit of a special case.

Godot actually passes ScriptInstance around as void* for some reason, so on the Rust side we implement a trait called ScriptInstance whose methods take a &mut self. In particular, the method of most concern is ScriptInstance::call

fn call(
    &mut self,
    method: StringName,
    args: &[&Variant]
) -> Result<Variant, u32>

This method documentation makes the following note.

It’s important that the script does not cause a second call to this function while executing a method call. This would result in a panic.

Which makes any attempts to implement recursion in a Rust-side scripting language a non-starter. I think we can make this method re-entrant, and I'd like to start the discussion on how to do that.

Currently, we implement the GDExtension ScriptInterface type as ScriptInstanceData<T: ScriptInstance>, which contains a RefCell<T>. When we need to invoke a trait function like ScriptInstance::call, we take the ScriptInstanceData<T> and do

borrow_instance_mut(instance).call(method.clone(), args)

That is, we borrow the contents of the RefCell<T>, instance.inner, for the duration of the call. That makes it impossible to invoke call again on the same instance recursively from Godot until the original call exits.

Proposed Solution 1

My first thought is this. ScriptInstance is never used as a trait object, so it needn't be object-safe. So one solution is to change the signature of all of the ScriptInstance methods from

fn call(&mut self, ...) -> ...

to

fn call(instance: RefCell<Self>, ...) -> ...

Then the implementor of call can borrow mutably, decide what it needs to do, and then release the borrow if it's going to make a potentially-recursive call. This just puts the control (and responsibility) of borrowing correctly in the hands of the implementor. I'm not entirely sure how I feel about that, as instance: RefCell<Self> is a much more obtuse API than &mut self, but it definitely would get the job done.

Proposed Solution 2

We may be able to apply the GodotCell trick to ScriptInstance. I don't claim to fully understand the trickery exhibited by this new cell type, but it looks like it hinges on the fact that the self in these calls is actually owned by a Gd<Self>. In our case with ScriptInstance, the self we have is always owned by a ScriptInstanceData<Self>, so a similar trick may be viable.

[TitanNano]

I believe the script instance should either adopt GodotCell once it is ready, or an alternative I already proposed in the initial PR, is to make the entire script instance &self and delegate all mutation to the script instance implementation as interior mutability.

That said, script recursion should not require going back through the engine and should be contained in the script runtime.

EDIT: I missed that GodotCell has already been merged.

[Mercerenies]

That said, script recursion should not require going back through the engine and should be contained in the script runtime.

I'm not sure I agree with that. If I have an unknown object of type Variant, I'm going to call back into the Godot engine to invoke methods on it, not handle that myself. Sure, if it's explicit, direct recursion then I could do something clever and never leave Rust, but that doesn't handle mutual recursion, recursion that goes through GDScript, or calling something like Array.map or another higher-order Godot function with a function argument that calls back into self somehow. There are lots of legitimate reasons to recurse through Godot.

[TitanNano]

@Mercerenies All right, I was more thinking of direct recursion. But what you are describing is definitely an issue. Unfortunately, it is a general issue of gdext.

If I'm not mistaken, even with GodotCell call stack like
Class::method(&mut self) -> EngineClass::api_method() -> Class::method(&self)will cause a panic.

[Bromeon]

Unfortunately, it is a general issue of gdext.

If I'm not mistaken, even with GodotCell call stack like Class::method(&mut self) -> EngineClass::api_method() -> Class::method(&self)will cause a panic.

No, GodotCell should solve this 🙂

You can look at an integration test to get a feel. Here you have the chain first_calls -> call -> second.

gdext/itest/rust/src/object_tests/reentrant_test.rs

Lines 27 to 44 in e3644a0

	#[func]
	fn first_calls(&mut self) {
	self.first_called_pre = true;
	self.base_mut().call("second".into(), &[]);
	self.first_called_post = true;
	}
	#[func]
	fn first_signal(&mut self) {
	self.first_called_pre = true;
	self.base_mut().emit_signal("some_signal".into(), &[]);
	self.first_called_post = true;
	}
	#[func]
	fn second(&mut self) {
	self.second_called = true;
	}

[TitanNano]

@Bromeon sorry I didn't make my example super clear.

So yes, the new GodotCell implementation allows for re-entrance when making calls to the API if the base class. What I was trying to express though is any other scenario where a call to some engine API happens to call back into the same rust class again.

Something like this:

CustomScriptLanguage::method(&mut self) -> Engine::api_method() -> CustomScriptLanguage::get_type(&self)

Or a class calling a GDScript that calls back into the class due to some circular references:

impl CustomNode {
    pub fn method(&mut self) {
        self.ref_gdscript_child_node.call("some_method", &[]);
    }

    pub fn on_child_signal(&self) {...}
}

func some_method():
    self.child_signal.emit()

[TitanNano]

This discussion leads me to this idea of performing a "downgrade" of &mut self to &self before calling an engine API:

let base = self.base_mut();
let self_gd = base.to_godot().cast::<Self>();
let new_self = self_gd.bind();

[TitanNano]

@Mercerenies what are your thoughts on making the entire ScriptInstance take &self? I still believe it would give implementors the highest amount of control and flexibility.

[Mercerenies]

Can we do that? To be honest, I would've written that idea off as impossible, but I suppose if everything really is just clever interior mutability, then we could probably get away with it. If &selfing everything is remotely possible, then I'm on board with it. And (again, assuming there's no unexpected hiccups) it would certainly be the most straightforward thing to implement.

I also feel like we have too much direct access to ScriptInstance. Every other Godot type is a Gd<T>, so when we pass them around and use very non-Rust semantics to work with them, it's not surprising (since it's clearly a wrapper type). ScriptInstance is weird in that Godot gives us direct pointers to it, so I wonder if it should also be some kind of custom wrapper type (which we could then, of course, take by immutable reference).

[TitanNano]

Experimental branch where ScriptInstance is completely immutable: https://github.com/TitanNano/gdext/tree/jovan/script_instance_no_mut

I also feel like we have too much direct access to ScriptInstance.

I don't really follow. ScriptInstance is an abstraction for the GDExtension interface the engine provides / expects to expose a script instance to it. The engine is the only one making calls into an instance, there shouldn't be anyone else holding a reference to it. Where do you see it being passed around?

I would say ScriptInstance is not that much different from implementing something like IScriptExtension.

[Bromeon]

Using &self everywhere feels a bit like a cop-out: while it makes us no longer need to think about mutability, it pushes the responsibility onto the user, through interior mutability.

Furthermore, it doesn't follow our API design anywhere else -- Array, Dictionary etc. also allow &mut self access, even though they are internally reference-counted. It would also weaken type safety if ScriptInstance::set() would take &self: you can no longer pass around a &ScriptInstance to prevent accidental modification.

[Mercerenies]

I don't really follow. ScriptInstance is an abstraction for the GDExtension interface the engine provides / expects to expose a script instance to it. The engine is the only one making calls into an instance, there shouldn't be anyone else holding a reference to it. Where do you see it being passed around?

Let me put it another way. Our ScriptInstance trait isn't a Godot interface. It's something rust-gdextension made up. The Godot interface is actually GDExtensionScriptInstancePtr, which we effectively "subclass" on the Rust side as

struct ScriptInstanceData<T: ScriptInstance> {
    inner: RefCell<T>,
    gd_instance_ptr: *mut sys::GDExtensionScriptInstanceInfo,
}

And then whenever a method is called on that, we do inner.borrow().whatever() or inner.borrow_mut().whatever(). But ScriptInstance (the Rust-side trait that takes self) is our invention. So maybe those methods shouldn't be taking &self or &mut self at all and should instead be taking either ScriptInstanceData<Self> or something that resembles it. It still pushes the responsibility for interior mutability onto the user, but it makes it incredibly explicit that that's the case.

[TitanNano]

It would also weaken type safety if ScriptInstance::set() would take &self: you can no longer pass around a &ScriptInstance to prevent accidental modification.

@Bromeon I think this is a fair point. We would then still have to provide a way to make &mut ScriptInstance inaccessible (via GodotCell or otherwise) and integrators of scripting languages would have to work this into their implementation. Every call that is passed from a script to an engine API has to ensure that there is no mutable reference as there is always a change of reentrance and having the constraints of this library leaking into the script language itself is not desirable.

Do you have a suggestion how to approach this instead?

And then whenever a method is called on that, we do inner.borrow().whatever() or inner.borrow_mut().whatever(). But ScriptInstance (the Rust-side trait that takes self) is our invention. So maybe those methods shouldn't be taking &self or &mut self at all and should instead be taking either ScriptInstanceData<Self> or something that resembles it.

@Mercerenies I don't entirely agree with this. ScriptInstance is a safe version of the GDExtensionScriptInstance struct. ScriptInstanceData<Self> is a necessary wrapper to uphold all the safety requirements, and it should not be exposed to the public API in the same way none of the other GDExtension* types are directly exposed.

Exposing just a cell like &GodotCell<T: ScriptInstance> definitely would have its advantages, though.