Wrong width for Hindi on macOS, but correct width on Linux

[tleonhardt]

I tried using wcwidth to calculate the length of the name for the city of Mumbai in Hindi (बॉम्बे हिंदी)

from wcwidth import wcswidth
wcswidth('बॉम्बे हिंदी')
9

On macOS 10.13.5 using Python 3.6.5, I see a visual width of 5 characters and a calculated width of 9 characters.

On Ubuntu 18.04 using Python 3.6.5, I see a visual width of 9 characters and a calculated width of 9 characters.

Thank you by the way for creating a very useful module!

[jquast]

Thank you very much for the specific example, definitely a bug to try to address !

[tleonhardt]

In case it is important for future troubleshooting, I was using iTerm 2 version 3.1.6 with Bash version 4.4.23

[shreevatsa]

Almost surely, the "correct" width on Linux is because of a broken terminal that does not display the characters correctly. On macOS, the terminal uses the system libraries for text rendering, and properly renders the combining characters. On Linux, the terminal does something horrible.

Here's a much simpler test case:

>>> from wcwidth import wcswidth
>>> wcswidth('का')
2

That's U+‎0915 DEVANAGARI LETTER KA followed by U+‎093E DEVANAGARI VOWEL SIGN AA

It is supposed to be displayed as a single grapheme (e.g. you should not be able to place the cursor between them to type, and definitely you should not see a dotted circle), but on Linux terminals I get very weird results, with क in one cell, and ा in another cell (with the dotted circle).

The wcswidth result of "2" is consistent with this weird result, but obviously incorrect.

I imagine it's similarly broken for all Indic scripts. In fact I cannot imagine how something like wcwidth, which only returns integers, is going to work for Indic, Arabic, etc scripts. Searching for [wcwidth indic] brings up some results like mintty/mintty#553 xtermjs/xterm.js#1468 xtermjs/xterm.js#72 and see the mentions of "Indic" at https://www.cl.cam.ac.uk/~mgk25/unicode.html -- looks like a lot of software is broken.

[jquast]

Hello new conversationalist :)

I know the details pretty well, but re-reading the last link for “Indic” somewhere in there makes pretty dire “no support”, anyway the terminal landscape is very rich today, but mostly libc/wcwidth(3) based with little font intelligence, but combining more than has been written here in past years, improving due to emoticon situation I suppose, folks want silly icons aligned.

I personally saw Hindi has the go-to Test characters, I have a combining browser in the repo, you’re welcome to use to modify and view “Indic” language effects,

looks like a lot of software is broken

Well, we can fix it! This library had some light use in terminal landscape for python languages and Python is very readable, folks may be copying us already and I hope they may, into the next future wcwidth(), so please join if you can help :)

I have a scheme for automatically detected Unicode support level by introspection using terminal report cursor position query..

Anyway if anyone is interested in taking on the bulk of the work I will transfer knowledge and always accept test-passing PR’s that make sense.

Otherwise I’m FOSS retired and unable to put in the hours, sorry and good luck

[jquast]

Wow, this really screws up in iTerm2, especially if navigating a cursor around the text, which is one of the better terminals for these things. In iTerm2, it is displayed in 1 cell, not 2, but wcwidth determines 2 for this example.

[Zarainia]

I am having similar problems with various WSL terminals. Gnome Terminal under an X-server or WSLg displays á (Latin letter a + combining accent) as width 1 (which is what wcwidth returns). Entering WSL from Windows, Windows Terminal and the default bash terminal display it with width 2, Hyper and Terminus displays it as width 1, but ucs-detect detects Unicode version as 13.0.0 for all of them, despite the results being different.

[dscrofts]

I too am having issues with this. Here's another specific example:

from wcwidth import wcswidth
wcswidth("चाइनीज")
6

When viewing this using the Kitty terminal, the text only occupies 4 columns, not 6. This seems to be a problem in general with most Hindi text that I've encountered.

Tested on macOS Big Sur 11.6.5 and Kitty 0.24.4

[jquast]

Zero-width characters used with the Hindi language have been resolved in today's release by #91.

I created a testing tool that verifies it, that at least in the case of "Universal Declaration of Human Rights" from https://unicode.org/udhr/ in Hindi, that wcwidth now agrees in measurement of all words with "kitty" and "mlterm" terminals.