N-API: Type safety for napi_get_value_external/napi_unwrap

[argv-minus-one]

Is your feature request related to a problem? Please describe.
napi_get_value_external is currently type-unsafe. It yields 32 or 64 arbitrary bits, with no guarantee as to whether they mean what the caller thinks they mean, nor which native module created the external. Even assuming that it's a pointer is unsafe and may result in segfault.

Describe the solution you'd like
Please add a way to attach type information to values created by napi_create_external, and a way to check that type information when calling napi_get_value_external. The “type information” should be some sort of unique identifier generated by Node.js with an opaque C type, so that no two native modules can ever accidentally use the same type identifier for different types of external values.

Suggested API:

// An opaque identifier.
typedef struct napi_external_type__* napi_external_type;

// Creates a new napi_external_type.
// Each call to this function yields a different napi_external_type.
napi_status napi_create_external_type(
    napi_env env,
    napi_external_type* type
);

// Like napi_create_external, but takes a napi_external_type parameter.
// Attaches it to the created napi_value.
napi_status napi_create_typed_external(
    napi_env env,
    void* data,
    napi_finalize finalize_cb,
    void* finalize_hint,
    napi_external_type type,
    napi_value* result
);

// Like napi_get_value_external, but takes a napi_external_type parameter.
// Throws TypeError if the given napi_value has a different (or no) napi_external_type.
napi_status napi_get_value_typed_external(
    napi_env env,
    napi_value value,
    napi_external_type type,
    void** result
);

Describe alternatives you've considered
Currently, I'm just assuming that napi_get_value_external gives me a pointer to something that's at least sizeof(void *) bytes long, and put a magic number at the beginning of the external data structure to identify its type. To make the magic number distinctive, it is a pointer to some data inside my module:

static const void *MAGIC = &MAGIC;

typedef struct {
    void *magic; // check if magic == MAGIC before use
    …
} my_data;

…

my_data *d;
napi_get_value_external(…, …, &d);
if (d->magic != MAGIC) {
    // bail
}

As I've said above, this will result in undefined behavior (probably segfault) if some other native module calls napi_create_external with a value that isn't a valid pointer, and that external value somehow gets fed to my native module. Nor is it actually guaranteed that my magic number won't happen to be at the beginning of some other module's unrelated data structure, though it is highly unlikely.

[devsnek]

is calling napi_get_value_external on something you don't own not a bug on its own? this looks like working as intended from my perspective. Additionally, having generic void* for random user data is a super super common pattern across the c and c++ ecosystems.

[argv-minus-one]

is calling napi_get_value_external on something you don't own not a bug on its own?

No, because once the external value passes through JavaScript land and comes back to my native module (which, as far as I can tell, is the whole point of externals), there is no guarantee of whether or not the external that gets passed back is one that my native module created. napi_unwrap has the same problem, by the way.

That's the whole point of adding type information to an external: to find out whether or not I own it, in a way that's guaranteed, not merely likely, to work.

Additionally, having generic void* for random user data is a super super common pattern across the c and c++ ecosystems.

Yes, but there's always some guarantee as to who owns which one.

For example, there is a guarantee that, when one of my externals is finalized, the void* passed to my finalizer function is very definitely the value of one of my own externals. No other native module can even find my finalizer (it's declared static, so it doesn't appear in my module's dylib/so/dll symbol table, and there are no pointers to it aside from those passed to N-API), let alone call it with some other value.

But there's no such guarantee for napi_get_value_external and napi_unwrap.

[bnoordhuis]

It's not that this feature couldn't be implemented but if you can't be sure the external that's passed to you is one you handed out, then you're arguably using it wrong.

An idiomatic way of solving that is by storing the external as a value in a WeakMap, where the key is the object you hand out to your downstream user:

// index.js
const addon = require('./build/Release/addon.node');
const externals = new WeakMap();
module.exports = MyClass;

class MyClass {
  constructor(...args) {
    externals.set(this, addon.newExternal(...args));
  }

  act() {
    const external = externals.get(this);
    return external && addon.act(external);
  }
}

[argv-minus-one]

@bnoordhuis, that is not a solution. Reasons:

1. Using WeakMap for this would be a waste of CPU time and memory.
2. Managing the WeakMap in a JS wrapper module is unsafe, because other JS modules can require a native add-on directly.
3. Even if I construct a WeakMap using N-API calls, there is no guarantee that global.WeakMap is the genuine built-in WeakMap constructor, and hasn't been replaced by some other JS code.
4. I can't use a native hash table like std::unordered_map, either, because N-API does not expose any (almost-)unique identifier for objects that's guaranteed not to change throughout an object's lifetime (along the lines of Java's System.identityHashCode()).

In short:

if you can't be sure the external that's passed to you is one you handed out, then you're arguably using it wrong.

It is currently impossible to be sure. That's the whole point of my proposal.

[devsnek]

Just for some more context, the napi_external api more-or-less mirrors v8's External api, which looks like this:

class V8_EXPORT External : public Value {
 public:
  static Local<External> New(Isolate* isolate, void* value);
  V8_INLINE static External* Cast(Value* obj);
  void* Value() const;
 private:
  static void CheckCast(v8::Value* obj);
};

This hasn't (as far as i am aware) ever been an issue for this api in v8, so i'm not yet convinced that this issue would suddenly appear for napi. We even use these v8 externals within node itself and they haven't been a source of unsafe behaviour.

[TimothyGu]

I think it’s good to first take a step back, and ask the question of what use cases there are for external that cannot be fulfilled by anything else, rather than jumping to the conclusion that N-API should provide such an interface.

How are you using externals, may I ask? Is the external object like the native counterpart for a JavaScript object?

In my experience, externals are a huge hammer that is best thought of as a tool of last resort. It is very unsafe, and also fairly memory-inefficient. (The underlying V8 External object is a full-on object, even though all it has is a pointer.) We had been able to get rid of most instances of externals in our code base in favor of something safer, like the equivalent of napi_wrap().

[argv-minus-one]

@devsnek: Well, using externals is safe if only one code base (the application embedding V8) is able to create externals. In that case, you own all externals. But that's not the case with Node, which allows multiple native modules to be loaded at the same time. I can ensure that all of my externals are pointers to a data structure with a type identifier at the start, but that's it—I can't control what sort of data other native modules put in their externals.

@TimothyGu: napi_wrap has the same problem. The documentation makes no guarantees, and napi_[un]wrap does not accept any sort of unique type identifier. napi_unwrap will happily unwrap any wrapper object, not just my wrapper objects.

I notice that the V8 implementation of N-API won't allow an instance method created with napi_define_class to be called on the wrong receiver object, so napi_unwrap of the thisArg seems safe, but that's it—there's no way to check whether a wrapper object passed as a parameter to a native function is of the correct type.

Perhaps it would be better to attach type information to napi_wrapped objects instead of externals? I'm fine with that too.

[TimothyGu]

Okay, I think adding some sort of support for type-checking for napi_wrapped objects is more workable. In particular, I think something like V8's FunctionTemplate::HasInstance() would be a worthy addition.

However, this is not without technically challenges. The way napi_define_class() and V8 APIs are defined, you don't really get access to the underlying FunctionTemplate from the defined class directly. How to essentially virtualize FunctionTemplate in an engine-independent way could be a challenge.

[argv-minus-one]

If that isn't feasible, a somewhat cumbersome but hopefully simple solution would be along the lines of my original proposal for typed externals: a function that generates opaque unique type identifiers, and a variant of napi_[un]wrap that accepts/checks those type identifiers. Since napi_wrap already somehow associates an arbitrary void * with an arbitrary JS object, I assume it can associate a numeric type identifier too? API suggestion:

// An opaque identifier.
typedef struct napi_wrapper_type__* napi_wrapper_type;

// Creates a new napi_wrapper_type.
// Each call to this function yields a different napi_wrapper_type.
napi_status napi_create_wrapper_type(
	napi_env env,
	napi_wrapper_type* type
);

// Like napi_wrap, but takes a napi_wrapper_type parameter.
// Attaches it to the created napi_value.
napi_status napi_wrap_typed(
	napi_env env,
	napi_value js_object,
	napi_wrapper_type type,
	void* native_object,
	napi_finalize finalize_cb,
	void* finalize_hint,
	napi_ref* result
);

// Like napi_unwrap, but takes a napi_wrapper_type parameter.
// Throws TypeError if the given object has a different (or no) napi_wrapper_type.
napi_status napi_unwrap_typed(napi_env env,
	napi_value js_object,
	napi_wrapper_type type,
	void** result
);

[bnoordhuis]

Managing the WeakMap in a JS wrapper module is unsafe, because other JS modules can require a native add-on directly.

To be clear, that's not a design consideration for Node.js. It's not a sandbox for executing untrusted code.

If the idea of a WeakMap or a JS shim is unpalatable to you (it's just idiomatic, it's not the only way to do things), then e.g.:

1. Store the pointers you wrap in an external in a std::set or whatever, and
2. Check the pointer is part of the set when you unwrap an external later on.

Perhaps it would be better to attach type information to napi_wrapped objects instead of externals?

That could work but it would have to be investigated how inheritance (class Explode extends YourClass {}) affects storage/retrieval of the private data. I know the answer for V8 but I'm unsure about e.g. Spidermonkey.

[gabrielschulhof]

Actually, napi_wrap() is entirely type-safe if used in conjunction with napi_define_class(), because V8 ensures that prototype functions cannot be detached from the prototype and called on objects other than instances of the class.

If you try to pass an object other than an instance of the class to one of its prototype methods, you get

TypeError: Illegal invocation
    at Object.<anonymous> (/home/nix/node/node-addon-examples/6_object_wrap/napi/addon.js:8:26)
    at Module._compile (internal/modules/cjs/loader.js:774:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:785:10)
    at Module.load (internal/modules/cjs/loader.js:641:32)
    at Function.Module._load (internal/modules/cjs/loader.js:556:12)
    at Function.Module.runMain (internal/modules/cjs/loader.js:837:10)
    at internal/main/run_main_module.js:17:11

To illustrate, take a look at https://github.com/nodejs/node-addon-examples/blob/master/6_object_wrap/napi/addon.js#L6 and insert this line:

obj.plusOne.apply({});

This will cause the above-mentioned exception.

The reason for this scrutiny is here:

node/src/js_native_api_v8.cc

Line 837 in 76bf7ee

v8::Signature::New(isolate, tpl));

That is, when we define a prototype method in N_API we add a v8::Signature() to its definition to ensure that the method can only be called when instances of the class or its subclasses are passed as the this parameter.

Thus, if you wrap instances of a class declared with napi_define_class() and receive the instances exclusively as this arguments to prototype methods, V8 will ensure that you never receive an instance containing the wrong kind of pointer.

Additionally you can verify using napi_instanceof() that any given JavaScript object is an instance of the constructor before you do a napi_unwrap() and cast the pointer.

So, basically, the type information you need is stored in V8 itself in the relationship between the class and its instances.

For externals, there is no such guarantee.