Fix UTF-8 Support

[sizzlemctwizzle]

@OpenUserJs/admin @OpenUserJs/backend @OpenUserJs/frontend
Related to #198 and a few other bugs, I just wanted to consolidate identifying and fixing a type of bug that might be common inside of OUJS. UTF-8 characters sometimes break things in production. It doesn't happen in development for some reason. It mostly has to do with encoding going wrong (or the wrong encoding function being used) during a redirect, but it might happen it other areas that have yet to be uncovered.

So far the errors I've been able to verify in production are:

• Import using GitHub (UTF-8 in GitHub path)
• Edit script description (UTF-8 in scriptname)

@Martii
Were there any others? Any other script manipulation I tried in production seems to work.

This is a blocking bug. I will not accept any unrelated PRs, except bug fixes, until this issue is resolved. Update: All issues that been discovered have been fixed, but there may be lingering undiscovered bugs so I'm leaving this issue open until we're confident this has been resolved.

Your mission (should you choose to accept it): Use UTF-8 characters in production in any way you can possibly imagine, and report back here with your results in the comments. This way we don't waste effort testing something twice and can identify the problem areas that need to be fixed.

[Martii]

Were there any others?

Not that I have seen yet. I could delete my affected scripts but not show the source or edit the source on production.

• Import using GitHub (UTF-8 in GitHub path)
• Edit script description (UTF-8 in scriptname)

Most notably on the POST routines... e.g. they do change the data store on production but come back with ISO 8859-1 string redirects on dev (which means they work as expected but maybe not correctly) and UTF-8 string redirects on production... which is very peculiar.

See also:

• Possibly similar event happening in Show script source messes up character encoding greasemonkey/greasemonkey#1940 with the browser... hopefully it's not a Fx issue... I will go try it out in gmport for SM in a bit and see if it is present... but I have tested importing in at least one other non-Moz browser so possibly not related but similar only.

[sizzlemctwizzle]

but not show the source or edit the source on production.

Could you give me a url? I haven't been able to reproduce.

Most notably on the POST routines...

Wait a minute... Since they're redirections and not views, we must not be setting the character set correctly.

[Martii]

I haven't been able to reproduce.

I've subsequently deleted my affected scripts... on resync the issue of being non-editable isn't present but the 400 is... I didn't want orphaned scripts on OUJS currently if a fix shows up... I can infer that we were/are storing in ISO 8859-1 instead of UTF-8 in the DB I think. If you pick my Hello World script (only one in UserScripts repo) it should do it as of testing earlier this morning. I haven't retried it with your recent commits and possible deployment yet.

EDIT: You might try in jesus2099's account with his scripts (specifically this one that had source before DB Migration) that have Unicode in them too. I can't view Source Code on a few of them which is how I detected which ones of mine to delete.

EDIT: CONFIRMED pro importing my Hello World script causes the 400 with new tab on current, as of just a moment ago, still.

[Martii]

btw production appears to be goofed up right now... with CSS at least. :\ This is challenging to find where things should be vs where they are. ;)

[Zren]

btw production appears to be goofed up right now... with CSS at least. :\

Getting

Resource interpreted as Stylesheet but transferred with MIME type text/plain: "https://openuserjs.org/css/common.css". openuserjs.org/:15
Resource interpreted as Stylesheet but transferred with MIME type text/plain: "https://openuserjs.org/css/bootstrap-custom.css". openuserjs.org/:14
Resource interpreted as Stylesheet but transferred with MIME type text/plain: "https://openuserjs.org/css/github.css".

I left out type="text/css" in the <link> tags, but it was working without them so I've got no idea why it just broke.

[sizzlemctwizzle]

so I've got no idea why it just broke.

I was trying something out in production. Reverted. Should be fixed now.

[Martii]

I left out type="text/css" in the <link> tags

Later those should probably go back in from where they left. MIME types are important to maintain with client side.

Reverted. Should be fixed now.

It is. :) Thanks.

[sizzlemctwizzle]

I've identified 7 scripts that are currently in production but can't be installed because of a fuckup in the AWS code I used to get rid of url namespaces.

• https://openuserjs.org/scripts/jesus2099/kasi._PLAIN_TEXT_LYRICS_歌詞コピー_純文本歌詞
• https://openuserjs.org/scripts/©TriMoon™/FB-App_Widescreen
• https://openuserjs.org/scripts/DitherSky/Skype-линкификатор
• https://openuserjs.org/scripts/AMZMA/Yet_Another_Download_Helper_for_Various_FileHosting,_Just_Helper_Dengan_Taburan_Kacang_Almond_Diselimuti_Coklat_Belgia_Yang_Langsung_Meleleh_Bila_Masuk_Di_Mulut._[Argggghhh}_(╯°Д°）╯︵_ɯuᴎƃɐɯ
• https://openuserjs.org/scripts/DutyCorp/Duty™_Engine
• https://openuserjs.org/scripts/jesus2099/JASRACへの直リンク
• https://openuserjs.org/scripts/Kelseydehaan/iBOOD_Autosubmit_to_sofortÜberweisung_(DE)(no_longer_PayPal)

This issue will not occur going forward. It is isolated to these scripts. I should be able to fix this soon.

[Martii]

There's also may be some on dev that I saw last night in my account too. Those are totally orphaned from what I can tell... e.g. can't delete them because no homepage access. I haven't check today with current HEAD yet.

• http://localhost:8080/scripts/marti/httplocalhost.localdomain/RFC_2606%C2%A73_-_license_and_licence_Unit_Test
• http://localhost:8080/scripts/marti/httpuserscripts.orgusers37004/uso_-_More_Pages2

from my dev script listings page. These were orphaned during some of the past commit fixes including dev DB migration from your discussion.