Nextercism & Encoding

[kytrinyx]

We've had issues with encoding in the past.

the character α is shown as α on the site, which is what α will look like when erroneously interpreted as latin-1. —exercism/exercism#2523

I had a file whose first 1024 bytes didn't contain a Unicode character, but then a character after the 1024th byte did. [...] Submitting this file mangles the Unicode character. —exercism/cli#309

@petertseng submitted a fix for these in exercism/cli#346.

The new CLI will not be dealing in encoding at all. It will submit the files as is, and then the backend will detect the encoding. So we might get it wrong again.

Right now we tested with the fixtures that are in the https://github.com/exercism/cli/tree/master/fixtures/iteration/python/leap directory, specifically

• https://github.com/exercism/cli/blob/master/fixtures/iteration/python/leap/long-utf8.py
• https://github.com/exercism/cli/blob/master/fixtures/iteration/python/leap/utf16be.py
• https://github.com/exercism/cli/blob/master/fixtures/iteration/python/leap/utf16le.py

These look like this on the new site, uploaded with the new client:

@petertseng I don't know how to view the UTF-16 files locally, to verify that they're rendering correctly on the site. Would you take a look at the screenshot for me?

So. If you've had files mangled in the past in fun and interesting ways, would you mind supplying an example file (and a screenshot of what it should end up looking like) so that we can test with it?

[iHiD]

Here's a screenshot with line numbers in case that's useful.

[NobbZ]

Does python actually peroperly deal with UTF-16 encoded files?

Meaning, can I do python utf16be.py and it will run without any syntax-errors? Not even Java was able to this when I tried it for the last time, and it encodes stuff UTF-16 internally back then…

I think best way is to keep them untouched and converted in the database. Both simple uninterpreted blobs. The latter is allowed to be NULL, also a third column which saves the currently choosen encoding.

Also we assume a default encoding of UTF-8 (which seems to be near to standard). We will deliver the original untouched if converted is NULL, if converted is not NULL we will deliver it.

If a user realizes that his submission is not properly displayed he can choose the encoding for the submission from the WEBGUI. In this case we convert the original submission from X to UTF-8.

This should be the most robust way do deal with the danger of randomly-encoded input.

Since we do not know what we get, we can't normalize before hand, because we do not know what we get, and we could end up with invalid and irrecoverable data.

[iHiD]

So, at the moment, we don't touch the files until we render them website at which point we temporarily convert them to utf8 for rendering. This is working well.

@kytrinyx - shall we smoke-test what happens when we send the files back to you via the API. If it works, then we don't need the extra column(s) @NobbZ suggests. If it doesn't work, then I suggest we take his approach.

[NobbZ]

The old way was to let the CLI guess the encoding and then normalize it before sending. Guessing the encoding is a bad idea, since different encoding could use the same sequence of bytes to actually represent different things.

By keeping the version of the user, we maintain the ability to reconstruct it in its original version, without the original user having to re-upload.

If problems occur, anyone (who has sufficient access rights) may choose to do an emergency conversion (simply remove all bytes not fitting ASCII)

[kytrinyx]

@NobbZ at the moment the files are stored untouched. We do the encoding thing on the fly when rendering.

If we run into any particularly troublesome encodings, we can consider adding a column to the database to let people specify what encoding it should be read as, and since we're not mutating the file that we receive, that should work for any existing files at that time.

@iHiD Yeah, let's do some smoke tests and make sure that these can take some round trips.

[NobbZ]

@kytrinyx But isn't the function api.readFileAsUTF8String() trying to actually guess the encoding and trying to normalize to UTF-8 while reading it in?

Why should we convert to UTF-8 while reading, when we wan't to send it untouched?

[kytrinyx]

@NobbZ That's in the old CLI. I've spent the past week rewriting it, and the new one uploads files directly instead of reading them in and sending them inside of JSON.

[NobbZ]

Yeah, the CLI that is currently available to the public… So I thought that was meant when @iHiD said "at the moment"…

Sorry for the confusion.

[kytrinyx]

@NobbZ we could have been more clear. We're super knee-deep in the nextercism prototype stuff, so everything "at the moment" for us is about that.

[petertseng]

I don't know how to view the UTF-16 files locally, to verify that they're

rendering correctly on the site. Would you take a look at the screenshot for me? Sorry, not able to view the UTF-16 files either for next 2 weeks. Limited.

[kytrinyx]

So far it seems like we've got this sorted. If there is a problem in the nextercism prototype, we'll open an issue when we discover it.