Offline spell checker for VSCode

[bartosz-antosik]

Hello (first time contributing here)

There are few offline spell checkers among VSCode extensions, but they are based on seriously faulty JavaScript implementations of Hunspell spell checker.

Hunspell is nowadays probably the most widespread standard for spell check layer. It is used on MacOS, Linux and in some software (e.g. LibreOffice) on Windows. It is also used by both Atom and Sublime Text. There is an enormous collection of polished dictionaries for Hunspell.

There exists some JavaScript implementations that refer to Hunspell's name but in fact they do not implement critical functionality - lexical parser. I have verified these three:

hunspell-spellchecker
Typo.js
nspell

All three work more or less following a simple idea of loading the dictionary into memory (into a associative table, a.k.a. dictionary, object to be precise). They use the Hunspell's affixes (.aff file) to create ALL variants of the words found in the dictionary (.dic file) and then store them in the memory. When checking spelling dictionary is simply asked whether the word exist or not. Simple, but it has these implications:

1. Loading takes a lot of time;
2. It takes a lot of memory too;
3. Memory consumption causes them to crash under dictionaries with more expanded affix system (two out of three mentioned, third does not consume all of the affixes).

For example when running hunspell-spellchecker (there is a SpellChecker extension based on it) with English dictionary ("en_US", 62K+ words in dictionary) memory consumption is in peaks 500 MB and constantly above 250 MB. It crashes under Polish language dictionary ("pl_PL", 300K+ words in dictionary) after reaching about 1.5 GB memory consumed (there are reports about other dictionaries doing the same) with "JavaScript heap out of memory" message hidden well under the hood. Hunspell has a lexical parser which allows it to use these two sets (dictionary and affixes) "on the fly" without the need to merge them thus exploding memory consumption and load time.

There is a good spell checker component for node.js, which is actually a bindings for native spell checkers for MacOS (NSSpellChecker), Linux (Hunspell) and Windows (Spell Check API in windows 8+, Hunspell in earlier versions):

https://github.com/atom/node-spellchecker

It is alas a native module.

I have built a spell checker using this module. I will rather not publish it because it is quite pointless:

• The extension will (silently) stop working every time the electron or node get a version bump and I cannot guarantee I will always be around to rebuild binary dependencies quickly;
• Rebuilding binary dependencies is quite a hassle;
• I am unable to reasonably maintain binary dependencies for all three platforms (MacOS, Linux & Windows) - there already is an extension which uses this module, but it provides binary dependencies for MacOS only;
• Even If I would produce node-spellchecker module using node-pre-gyp with binaries for various platforms if I understand things correctly extension cannot (easily?) install dependent modules using npm (which could also imply having a proper C++ toolchain around in case node-pre-gyp packaged binaries are not sufficient). Binaries are packaged and simply get downloaded along with the extension.

So I would like you to consider doing something about it.

There are few paths I can imagine among them two are most obvious:

1. Build the node-spellchecker module along with the VSCode and make it available among "standard" modules that extension developers can count upon (this could result in more than one spell checker extension e.g. for spelling text or latex documents, comments in code etc.);
2. Provide a way to use native modules among extensions' dependencies.

I am most probably no one to discuss pros or cons of these alternatives, there are maybe other alternatives that I cannot see, but I think that with the evidence provided it is clear that unless something changes the answer to the question in the title is MOST PROBABLY NOT!

[Tyriar]

I'd personally like to see spell checking using native platform API bundled eventually.

[bartosz-antosik]

Do I get it correctly than by native you mean implemented in JavaScript/TypeScript?

I understand your point of view, I would probably prefer it too.

But please have a look on Hunspell GitHub page and consider how many years of active developement it took to get it where it is now. I doubt that someone could just "rewrite" it.

Anyway, for the time being existing spellcheckers are problematic and it may push away e.g. people who use VSCode to write technical docs, latex papers etc.

Having node-spellchecker accessible with compatible binaries, which can only realistically be achieved by bundling it with VSCode, would cure not even most but all of the above mentioned issues. Later when native API with comparable quality appears you can switch and it will not generate a lot of trouble for extensions developers because it is only a handful of calls.

[Tyriar]

I mean native as in platform; talking to macOS, Windows, Linux services if available, and falling back to another implementation if not.

[bartosz-antosik]

That's EXACTLY what node-spellchecker module does!

[bartosz-antosik]

Somehow I go impression that you tend to have everything in pure JavaScript and native modules support is not going to appear anytime soon. Sorry about misinterpretation.

[Tyriar]

Badly worded on my part 😄

[rebornix]

First of all, sincerely many thanks to @bartosz-antosik , your work is really thorough and an awesome guidance on spell checking. I spent some time investigating into this feature this iteration and here are my thoughts and todo items.

How we ship it

Firstly, the spell checking process should work in a separate process, without blocking the core or extension host pipeline.

Secondly, there are two ways to talk to native code: node native module or standalone script with interactive console. The former is easy to do as the only catch is you have to recompile every time if node/v8 version changes. To fix that we just need to put the extension into Code's folder, either in core or in our builtin extension folder. The benefit is obvious, we don't need to talk to C++ code painfully and stay inside NodeJS always but there are several issues that should be taken care of before we do that

1. Since we need to bundle Hunspell into Code, how does it affect our installer size/build time?
2. Right now node-spellchecker's api is not even async.
3. It should still be executed in a separate process. There are quite a few perf issues on Atom/node-spellchecker side.

The second way to solve this problem is running a standalone interactive script, which talks to system API, compiled in different architecture/platform. Then our NodeJS code, either Core or an extension can talk to it through standard IO or even better Socket. The script will be running in a new NodeJS process and we can easily make all the spell checking async.

I start with the second solution. Even though this problem is fixed perfectly, we still get quite a few issues around the experience and maturity of spell checking on different platforms, including but not limited to:

---

Spell Check API

On macOS and Windows (8 and above), the system provides builtin spell check support, their behaviors vary but they both support following common functionalities

• check whether a word is misspelled.
• get the ranges of mispelling in a text
• get the corrections for a mispelled word

In addition to above features, macOS, Hunspell and Windows disagree with each other on several APIs:

Ignore word

• Windows.
  • ✋ Right now node-spellchecker is using https://msdn.microsoft.com/en-us/library/windows/desktop/hh869774(v=vs.85).aspx . It will treats the provided word as though it were part of the original dictionary. The drawback of this API is once it adds the word to the dictionary, you can no longer delete it through API. A workaround is modifying file C:\Users---UserName---\AppData\Roaming\Microsoft\Spelling\en-US\default.dic and reload. I'll say let's use ignore
  • If we use https://msdn.microsoft.com/en-us/library/windows/desktop/hh869783(v=vs.85).aspx to ignore words, its lifetime is just the session.
  • ✋ node-spellchecker // NB: ISpellChecker has no way to remove words from the dictionary , it means node-spellchecker thinks right now you can't remove words from the dictionary, even though they are added by you :(
• Hunspell.
  • Life time: When using Hunspell, this will not modify the .dic file; custom words must be added each time the spellchecker is created. Use a custom dictionary file.
• macOS https://developer.apple.com/reference/appkit/nsspellchecker/1534837-learnword
  • Life time: until you ignore it.
• If we want to have the same experience in different platforms, we may want to use ignore on Windows and then users always have their ignore list on their preferred storage.

Conclusion: On Windows and Hunspell, ignore words temporarily and each time we initialize a spell check process, set the ignore list on the fly. As on macOS you can always remove words from the dictionary, let's trust it.

---

Builtin language support

• Builtin language on macOS and Windows
  • macOS: It depends on the language you install on your System. You can go to System Preference -> Keyboard->Text->Spelling and sort the dictionaries as macOS has automatic language detection, you may want to have a good sorting if you are depending on automatic language support.
  • Windows. Black box, what I learn from users feedback/bug report on Spell Check is it's part of language pack. We don't have a chance to touch anything, maybe that's why Atom uses Hunspell always on Windows.
• Builtin language dictionary we ship with Hunspell. Basically we can ship en_us and en_gb (less than 1Mb in total) and try to download dictionary when necessary.

What's the experience of setting up dictionaries for another language which has no builtin support?

• Hunspell, it's customizable as you can provide a path to your dictionary.
• macOS. Users can put dictionaries into ~/Library/Spell. We can't programmatically ask macOS Cocoa API to load dictionary from another place but we may want to touch that folder directly.
• Windows. Users have to install target language package.

How to spell check text which contains multiple languages, automatically?

System already has some native support, but they behave differently and the experience is not charming.

• macOS native . There is a configuration called automaticallyIdentifiesLanguages: Bool which controls whether the spell checker will atuomatically identify langauges or not. If you dont' specify a dictionary/locale explictly, this option is true by default.
• Windows 8 and above native. AFAIK, its builtin spell checker can have connection with keyboard/IME https://superuser.com/questions/480540/how-can-i-change-the-spell-check-and-auto-correction-language-of-ie10-windows8
• Linux/Hunspell
• how to specify what language to use in a specific scope of a specific type of file.

---

Dictionaries

• Hunspell Dictionaries we ship along with the product
• Dictionary compatibility: As you can't set another kind of dictionary to Windows, the only way to have a custom dictionary is using Hunspell.

Both Chrome and Firefox ship with en-US dictionary (for English users). Chrome will download any dictionary users require ( see https://cs.chromium.org/chromium/src/chrome/browser/spellchecker/spellcheck_hunspell_dictionary.cc?dr=C&q=chrome/dict&l=238 ), and Firefox fetches dictionaries from https://dxr.mozilla.org/mozilla-central/source/browser/app/profile/firefox.js#77.

Conclusion: Ship with en-US (because most of time you are coding in English) and maybe ship with one user preferred language (for example, maybe one day users can get a Chinese version of VS Code directly and it has Chinese dictionary builtin). For other requests, provide a stable/high available dictionary download service. Atom now downloads dictionaries from Google's service (which is used by Chrome), however that service is not available in some countries and regions.

Exception list/known Words

• How to ignore a word, where are they stored, life time? See above.
  They should be stored in a separate file and each time a spell checker instance is created, update the ignore list on the fly.

• How can users set a exception for known words, through context menu, command palette, or light bulb code action?

---

Spell Checker

• Can users choose their favorite Spell Checker, eg, use Hunspell even on Windows? It can be useful if users who work on the same product but on different platforms want to align with each other about spell checking.

We can ship Hunspell in all platforms and users can choose to use Hunspell or not.

---

Settings

Open questions about how we define the settings for spell checking.

• Setting for Dictionary, Path, etc.
• Can it be a setting file in workspace/user space?
• Can it be part of Workspace/User Setting?
• Scopes for spell checking
  • Based on programming lanuage id.
  • Based on TextMate Token Type(String, Comments, Other)
  • Based on TextMate Grammar (text.html.markdown, etc)
• Spell check actions
  • Light bulb and red squiggles
  • Context menu
  • Command: Fix mispelled words
  • Auto fix
  • Can spell checker be used for autocomplte/intellisense?

[bartosz-antosik]

Thanks @rebornix for kind words & analysis which I like a lot.

I would like to refer to few points of your analysis as it looks like maybe I do not understand one or more things.

Excuse me if I am very off at points but I know very little about node.js and the whole environment.

Synchronous/Asynchronous Interface

About this sync/async interface: are events (e.g. onDidOpenTextDocument, onDidChangeTextDocument, onDidChangeVisibleTextEditors) asynchronous or not?

If they are then then why bother if node-spellchecker's interface is or is not?

If they are not then not only spell checking engine should be asynchronous but all the extension code that reacts to events to parse text & select parts to spell that calls the engine should be too, should it not?

What takes time in spelling is parsing a document, possibly large, and eliminating parts that should not be spelled (suppose latex commands or parts of code that should be skipped to spell comments & strings etc.) And I recon it should be left up to the extension, not the speller, to decide on what to do with particular document type.

There is one more thing to consider here: Word lookup is quick. Suggestions are slow.

About spellcheckers that I used they are quick to look word up to test whether it is spelled correctly and slow (like over 10 times slower on average) to produce suggestions. Current approach e.g. in my spell checker extension is to spell & feed diagnostic collection with suggestions plus there is an option to just signal misspelled words and look up suggestions on provideCodeActions event.

So do I understand correctly that either all parts of the process should be async or it does not matter much whether spelling engine is?

Ignoring Words

About custom/known/ignored words: I would consider off loading this to the extension! Don't know about the rest of the world but I would love them to be manageable like rest of the VSCode's configuration. All three MS/iOS/hunspell place them no one knows where and it is additional pain to transfer them to another location or manage them in the context of the document type.

Language Scope in a Document

I like the idea of multiple languages inside one document a lot. It seemed to me crazy at first but the more I think about it it seems quite doable. The only way though I can think of is content/comment driven language switching. Again - the extension should decide about this, as this information can be, for instance, extracted from latex document quite other way than from other document type.

[rebornix]

@bartosz-antosik thanks for your reply. About async/sync problem, I'm referring to function calls to native code, they are no async right now. But it's not a problem as in nodejs, we can always use setTimeout or similar to mitigate it. Not a big deal.

Word Lookup/Suggestions

I like your idea of separating word look up and generate suggestions and thanks again for your perf testing. Postponing suggestion lookup to code action provider makes sure we only do minimal calculation. And you are right, this can be an option as the only catch of this feature is users can't have a general view of misspell suggestions in Problems View.

Another thing about perf is where to do the calculation, doing all the math in native code can be faster but sending a large portion of data to native code can cost time as well. We need good testing to find the balance.

Ignoring words

System Spell Checker stores the ignoring words on the fly and yes we'll hide them from users.

Multi language

macOS has its in-house language detect which works reasonable to me but Windows doesn't. Comments, strings and technical documents are the most possible cases that users may need multi-language support. We can either switch languages automatically, or maybe even spawn multiple spell check process for different languages.