fs: support different path name encodings

[bnoordhuis]

Continuing from #3401, it's clear that the way node.js handles path name encodings is sub-optimal. What is not clear is how to fix it. This issue is for discussing possible solutions.

A quick recap of the current situation:

• node.js assumes UTF-8 in most - but not all - places.
• UTF-8 is fine on Windows. Libuv converts UTF-8 to and from UTF-16, which is what the kernel expects.
• UTF-8 is common but not universal on UNIX systems. Most file systems are character set agnostic, encodings are normally by convention. OS X's HFS+ is the most common exception to the rule.

Considerations:

• Conversions should be zero-byte safe because most C APIs operate on zero-terminated strings.
• JS strings are conceptually always UTF-16 but V8 accepts ISO-8859-1, UTF-8 and UTF-16 as input.
• Conversion (to JS string) from ISO-8859-1 is lossless but conversion from UTF-8 and UTF-16 is not: invalid byte sequences are replaced with U+FFFD.
• Inversely, conversion to UTF-8 and UTF-16 is lossless but conversion to ISO-8859-1 is not: out-of-range characters wrap around - which can be insecure, see the bullet point about C APIs.

[bnoordhuis]

Two possible solutions:

1. Assume UTF-8 unconditionally. May be reasonable but we probably still need special casing for HFS+. It normalizes file names to NFD. Without intervention the file name you read back may not be what you created.
2. Assume UTF-8 but let the user override the encoding. This is what we do elsewhere (crypto, http, streams, etc.) so there is precedent. Users can deal with more obscure encodings like CP-1252 by encoding to buffer and post-processing it with iconv or iconv-lite.

[Fishrock123]

@bnoordhuis what are some downsides of the proposed solutions?

[bnoordhuis]

1. Assume UTF-8 unconditionally. Does the wrong thing when the encoding isn't UTF-8.
2. Assume UTF-8 but let the user override the encoding. How would the user know what the correct encoding is? Not solvable in general but at least it's flexible.

With either solution we could add an environment variable or command line flag to override the default encoding.

[jorangreef]

Regarding the two options:

1. Assuming UTF-8 should not necessitate a special case for HFS+ after decoding to UTF-8. HFS+ NFD form should come through fine when decoding as UTF-8 and should always be passed on to the user as is, never normalized by Node, for the reasons given here: https://github.com/nodejs/new.nodejs.org/blob/master/locale/en/docs/guides/working-with-different-filesystems.md
2. This would be ideal.

I don't think there should be a global environment variable or command line flag to override the encoding (it's tempting), because encodings are all relative to the filesystem in play at the mounted subtree the user happens to be working with. For example, a user may use a single Node process to work with multiple different filesystems mounted in /Volumes all with different encodings for example. The encoding override should just be an optional functional argument (e.g. as for fs.writeFile or fs.readFile etc.).

[bnoordhuis]

NFD form should come through fine when decoding as UTF-8 and should always be passed on to the user as is

I'm ambivalent. The NFC/NFD dichotomy is confusing IMO in that the file you create with fs.open() can end up having a different name when you read it back with fs.readdir().

https://github.com/nodejs/new.nodejs.org/blob/master/locale/en/docs/guides/working-with-different-filesystems.md

Nice write-up!

I don't think there should be a global environment variable or command line flag to override the encoding (it's tempting), because encodings are all relative to the filesystem in play at the mounted subtree the user happens to be working with.

I don't exactly disagree but there is an (IMO reasonable) case to be made for common case convenience if there is going to be a default encoding anyway. It's something we can tackle later though.

[jorangreef]

Thanks Ben!

The NFC/NFD forms are surprising (hopefully the guide will help with that!) but I think it's not technically possible for Node to try and fix HFS+ now, and if Node tried it would be repeating the same mistake HFS+ made (implementing form-insensitivity by sacrificing form-preservation). It would also be equally confusing if Node normalized HFS+ NFD to NFC and users called ls via child_process.exec only to see different filenames. In any event, even if Node wanted to, the roundtrip from HFS+ NFD to NFC to NFD is not lossless because the standard has advanced (Node would have to use the same normalization table baked into HFS+). So I don't think we should bundle it here with encoding choice. Better for users to learn about different Unicode forms and how to compare them insensitively without sacrificing form preservation in the process.

[seishun]

node.js assumes UTF-8 in most - but not all - places.

This is somewhat misleading. UTF-8 is assumed everywhere in fs except one place, where Latin-1 is erroneously assumed.

Without intervention the file name you read back may not be what you created.

You might have already figured it out, but just in case anyone else is confused: it's not possible to make you sure read back the same filename you created. HFS+ normalizes file names when they are created, and it's a lossy conversion. For example, both "도시락" and "도시락" become "도시락", and one can't know afterwards which one was used originally. (The strings all look the same because normalization preserves the "look" of the string, duh. Try comparing their lengths in a JS console.)

Users can deal with more obscure encodings like CP-1252 by encoding to buffer and post-processing it with iconv or iconv-lite.

Has an actual user requested this, or is this pure theory?

[piscisaureus]

Note that both Windows (and Javascript too, for that matter) technically use UCS2 and not UTF16. That means that not all valid Windows filenames are expressible as UTF8.

If we are going to fix this "propertly" libuv should probably use WTF-8 on Windows.

[rvagg]

How does this sound for the "Known issues":

• Unicode characters in filesystem paths are not handled consistently across platforms or Node.js APIs. See #2088, #3401 and #3519.

@bnoordhuis?

[bnoordhuis]

@rvagg Looks good.