Files that have Unicode chars only after the 1024th byte get their Unicode mangled

[petertseng]

The situation is that I had a file whose first 1024 bytes didn't contain a Unicode character, but then a character after the 1024th byte did. An example of such a file is:

http://exercism.io/submissions/1e341848768141cf8eba94c6af6e55a7

Submitting this file mangles the Unicode character.

In contrast, this file has Unicode in the first 1024 bytes, so it is good (even the Unicode that appears after the first 1024 bytes is good)

http://exercism.io/submissions/dce4e3ddf0294034ad987ce7b86cdb38

(These are just example submissions in Hello World, but this affected my submission for a real exercise too, Counter in xgo)

I tracked this down to readFileAsUTF8String in api/iteration.go. This uses the https://godoc.org/golang.org/x/net/html/charset#DetermineEncoding function to determine the encoding, which reads the first 1024 bytes.

I'm not really sure what's the right solution here. I know that function was created for #182 to solve exercism/exercism#2303 so there obviously is a legitimate reason behind all this, I guess maybe now we just need to figure out how to deal with this case as well. I don't yet have a good solution, so I'll file this first and sleep on it for a bit.

[rmunn]

Another file whose Unicode was corrupted:

http://exercism.io/submissions/841caf9f0bcb4c95a5fc36e6e457efc4

It seems that the ♯, ♭ and ♮ characters in my code (on lines 47-49) were misinterpreted as CP-1252 instead of UTF-8, because Go's DetermineEncoding function saw nothing but ASCII in the first 1024 bytes of my code, and therefore defaulted to CP-1252, aka "Latin-1". Which, IMHO, is the wrong default. UTF-8 is the de facto standard of the Internet these days, because unlike Latin-1, it can represent all the characters of the Unicode standard. Therefore, if the first 1024 bytes don't yield enough information to know what the encoding is, UTF-8 should be the assumed default.

This would, of course, result in anyone submitting a Latin-1-encoded file whose first 1024 bytes are ASCII getting their file rejected for not being valid UTF-8, and/or getting mangled if they happen to have two or three special Latin-1 characters together that happen to form a valid UTF-8 sequence. But that's highly unlikely; more likely is that they'll get their code rejected for not being UTF-8. Which, IMHO, is precisely what need to happen -- because anyone who writes code in Latin-1 these days probably doesn't understand Unicode, and therefore desperately needs to get an error message telling them to read http://www.joelonsoftware.com/articles/Unicode.html.

So my recommendation is that the submission should be interpreted as UTF-8 first, by default. If (and only if) an error occurs, then use DetermineEncoding to guess the encoding and reinterpret the submission in the guessed encoding.

[kytrinyx]

So my recommendation is that the submission should be interpreted as UTF-8 first, by default. If (and only if) an error occurs, then use DetermineEncoding to guess the encoding and reinterpret the submission in the guessed encoding.

This sounds like a reasonable approach to me.

[petertseng]

I attempted to use the proposed suggestion:

So my recommendation is that the submission should be interpreted as UTF-8 first, by default. If (and only if) an error occurs, then use DetermineEncoding to guess the encoding

To test this out, I simply made my exercism binary always interpret as UTF-8, then submitted a UTF-16 file. Unfortunately, no error occurred, but the files were still mangled, which will complicate this suggestion.

You may see my submitted files at:

• http://exercism.io/submissions/b846f2733ee74552a4748d50d629e142
• http://exercism.io/submissions/3b05d215146b4e10a77d4dbfcbadc552

This may make life slightly more difficult. What if I made Suggestion B instead:

Always use DetermineEncoding, but if it reports that it was uncertain about its guess, then ignore that and proceed with UTF-8.

This is possible because DetermineEncoding returns a boolean to indicate whether it is certain of its guess. Note that in the case of UTF-16-LE and UTF-16-BE (based on BOM presence) it will report that it is certain.

There is also Suggestion C instead:

Use code similar to https://github.com/golang/net/blob/master/html/charset/charset.go#L57-L62 to detect whether the input file is either UTF-16-LE or UTF-16-BE. If it is, use them. If not, go with UTF-8.

Suggestion C was made before discovering the certainty boolean for Suggestion B.

Suggestion C differs from B in that we only pick out the parts from DetermineEncoding that we actually care about (we don't need to scan meta tags for Exercism submissions!).

I believe both B and C and narrow the acceptable encodings to Exercism. Theoretically we could have also accepted windows-1252 in the status quo (although it seems nobody here would complain about losing that!). We would also lose any other encoding that DetermineEncoding is liable to report, although note that DetermineEncoding was originally designed for HTML - it scans for a declared content-type (we always declare text/plain) or meta tags that indicate the charset, features which are unlikely to be present in an Exercism submission.

This is all very difficult for me because I don't understand the requirements. What encodings do we want to accept? Without an answer to this question, it's so hard for me to evaluate suggestions since I don't know if they drop something we want. However, my best guess is UTF-16-LE, UTF-16-BE, UTF-8.

---

I'm sorry to report that I may not get to any of this urgently since it's rare that I submit files with non-ASCII chars and I already modified my exercism binary locally to always UTF-8, since I know that's the right answer for me (I don't plan to create files in any other encoding). Unfortunately that approach isn't applicable to everyone.

[petertseng]

Amending Suggestion B since DetermineEncoding returns a boolean to indicate whether it is certain of the encoding it guesses.

Old suggestion:

Always use DetermineEncoding, but if it reports windows-1252, then ignore that and proceed with UTF-8.

New suggestion:

Always use DetermineEncoding, but if it reports that it was uncertain about its guess, then ignore that and proceed with UTF-8.

[kytrinyx]

This is all very difficult for me because I don't understand the requirements.

I don't really understand the requirements here either :(

[petertseng]

Then my line of reasoning will be the following:

• In the beginning (Before Deal with different file encodings when submitting solutions #182), it was implicitly the case that only UTF-8 was supported (and this has test coverage), since we called ioutil.ReadFile directly.
• Then after Deal with different file encodings when submitting solutions #182, we did a few things:
  • intentionally added support for UTF-16-LE and UTF-16-BE (and these two have test coverage)
  • unintentionally added support for windows-1252 (no test coverage)
  • broke support for UTF-8 files under the circumstances described here.
• Now, we fix UTF-8 files in all circumstances, and add test coverage for it. But we probably lose support for windows-1252.

So now the only risk if someone really really wanted to use windows-1252 (I have doubts, but I support I can't be sure). Compared to the beginning, we still gained UTF-16.