Add Elixir language support #478

nietaki · 2017-03-02T11:12:08Z

Implements the changes for #319 .

This PR adds support for Elixir. As with some other languages, for simplicity and performance it doesn't support string interpolation highlighting, but should cover pretty much the rest of the language syntax.

For context: I have been working full time with Elixir for the last year and have tested the code with snippets from a handful of projects. I have submitted the highlighting for feedback on elixir forum and the Elixir slack channel and didn't get any issues raised (yet).

If you want to play around with the highlighting without checking out the code, go to this rawgit page - I wrote this to make implementing and testing this easier for myself.

googlebot · 2017-03-02T11:12:18Z

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed, please reply here (e.g. I signed it!) and we'll verify. Thanks.

If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
If you signed the CLA as a corporation, please let us know the company's name.

nietaki · 2017-03-02T11:14:15Z

I signed the CLA! :)

googlebot · 2017-03-02T11:14:17Z

CLAs look good, thanks!

sheharyarn · 2017-03-02T21:13:22Z

I'm so happy right now 😭 ❤️

nietaki · 2017-03-05T14:36:38Z

One bug and one improvement have been pointed out in the forums by @notriddle, please hold off on merging while I write the fixes.

nietaki · 2017-03-05T16:14:26Z

I don't think adding string interpolation support is really viable using the framework here, unless there's something about it that I'm missing. Take this (valid) example: "string #{ inspect(%{{"foo"} => "bar\""}) } interpolation" - there's not practical way of tracking where the string interpolation ends.

I think the state of the branch as it is right now might be as far as we get. Especially since using string literals inside a string interpolation expression is not a common use-case.

mikesamuel

Interrupted, but I thought I'd send out what I have.

mikesamuel · 2017-03-09T14:08:15Z

src/lang-ex.js

@@ -0,0 +1,78 @@
+/**
+ * @license
+ * Copyright (C) 2017 Google Inc.


Unless you're a Google employee, the copyright should be attributed to you and by putting that license on the file, you're saying you're ok with distributing your work under that license.

mikesamuel · 2017-03-09T14:16:42Z

src/lang-ex.js

+         // # comments
+         [PR['PR_COMMENT'], /^#.*/, null, '#'],
+         // a (possibly multiline) charlist
+         [PR['PR_LITERAL'], /^'(?:[^'\\]|\\.)*'?/, null, '\''],


\\. allows any escaped character except newlines. I mention this since I don't whether Elixir allows JS style line continuations

Thanks for pointing that out! The JS-style line continuations are supported and did break the highlighter, both in charlists and regular strings.

Now fixed and added to the tests.

mikesamuel · 2017-03-09T14:18:21Z

src/lang-ex.js

+          '!%&()*+,-;<=>?[\\]^{|}'],
+         // Borrowed from lang-erlang.js:
+         [PR['PR_LITERAL'],
+          /^(?:0o[0-7](?:[0-7]|_[0-7])*|0x[\da-f](?:[\da-f]|_[\da-f])*|\d(?:\d|_\d)*(?:\.\d(?:\d|_\d)*)?(?:e[+\-]?\d(?:\d|_\d)*)?)/i,


Ok, So 0o and 0O precede octal literals, and underscores can separate myriads?

You're right about the underscores. The 0x and 0o part shouldn't be case insensitive - I fixed that now.

mikesamuel · 2017-03-09T14:20:17Z

src/lang-ex.js

+         [PR['PR_ATTRIB_NAME'], /^(?:__(?:CALLER|ENV|MODULE|DIR)__)/],
+         // keywords
+         [PR['PR_KEYWORD'],
+          /^(?:alias|case|catch|def(?:delegate|exception|impl|macrop?|module|overridable|p?|protocol|struct)|do|else|end|fn|for|if|in|import|quote|raise|require|rescue|super|throw|try|unless|unquote(?:_splicing)?|use|when|with|yield)\b/],


Note to self, the |p? variant follows def so does not mean this token matches the empty string.

mikesamuel · 2017-03-09T14:28:05Z

src/lang-ex.js

+         // A double-quoted multi-line string
+         // or a triple double-quoted multi-line string.
+         [PR['PR_STRING'],
+          /^(?:"(?:(?:""(?:""?(?!")|[^\\"]|\\.)*"{0,3})|(?:[^"\\]|\\.)*"?))/],


The lookahead means that """""" (6 double quotes in a row) is not a double quoted string?

IIUC, this might match """x" but not the whole of """x"y"""?

Maybe just do two cases:

/^(?:"""(?:(?!""")[^\\]|\.)""")|""|"(?!")([^\\"]|\.)")/

GitHub mangled your suggestion a bit, I just reworked it on my own.

Since the string recognition isn't a shortcut pattern anymore, there isn't as much benefit in putting both """ """ and " " strings in one pattern, I separated them for simplicity.

Also: I looked into it more closely and a correct heredoc requires a newline after the opening """ - the compilation/interpreter fails otherwise. I Modified the code to take it into account.

mikesamuel · 2017-03-09T14:28:41Z

src/lang-ex.js

+         [PR['PR_PLAIN'], /^[$a-z]\w*[\!\?]?/],
+         // sigils with the same starting and ending character.
+         // Key part: X(?:[^X\r\n\\]|\\.)+X where X is the sigil character
+         [PR['PR_ATTRIB_VALUE'], /^~[A-Z](?:\/(?:[^\/\r\n\\]|\\.)+\/|\|(?:[^\|\r\n\\]|\\.)+\||"(?:[^"\r\n\\]|\\.)+"|'(?:[^'\r\n\\]|\\.)+')[A-Z]*/i],


Note to self, stopped here. Resume later.

mikesamuel

This looks broadly good. Thanks for doing this.

The only blocking issue is the copyright one, so feel free to treat all the other comments as FYI.

mikesamuel · 2017-03-09T17:32:46Z

tests/prettify_test_2.js

+'    `END`ATV~C&lt;hello&gt;`END`PLN\n' +
+'  `END`KWDend`END`PLN\n' +
+'\n' +
+'`END`KWDend`END')


Thanks for writing tests.

nietaki · 2017-03-09T18:17:44Z

Thanks for reviewing, I'll update the copyright and look into the issues you highlighted when I have a second, probably on the weekend.

nietaki · 2017-03-14T10:46:15Z

@mikesamuel How's this looking now?

mikesamuel · 2017-03-15T14:35:23Z

Merged

…

On Mar 14, 2017 6:46 AM, "Jacek Królikowski" ***@***.***> wrote: @mikesamuel <https://github.com/mikesamuel> How's this looking now? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#478 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAWg9vqA8Dvc3cOO7Iq93kHQZHMJvai_ks5rlm_5gaJpZM4MQ2nh> .

* First, crude version of the elixir language handler. * multiple elixir plugin improvements - attributes, atoms, strings, ... * elixir-lang - added support for number literals with underscores * elixir-lang: added support for atoms as keys in keyword lists * Elixir: added support for sigils, simplified the keywords regex * Elixir: added more keywords from Kernel.SpecialForms * Elixir: adding missing keywords and more permissive atom/variable names * Elixir: better support for binaries/bitstrings * Elixir: added support for the `iex` prompt for interactive examples * added a failing test for elixir * Elixir: fixed the failing test for elixir syntax highlighting * Elixir: Highlighting constructs like `%{"THIS": :foo}` as atoms * Elixir: fixed false-positive highlighting for 0XFF and 0O77 * Elixir: fixing the license text. * Elixir: fixing multiline charlists * Elixir: more robust multiline strings and charlists * Elixir: making string recognition simpler and more correct

nietaki added 11 commits February 20, 2017 23:50

First, crude version of the elixir language handler.

6953f23

multiple elixir plugin improvements - attributes, atoms, strings, ...

c933f87

elixir-lang - added support for number literals with underscores

937e87a

elixir-lang: added support for atoms as keys in keyword lists

ebbe645

Elixir: added support for sigils, simplified the keywords regex

df92df8

Elixir: added more keywords from Kernel.SpecialForms

5916b9b

Elixir: adding missing keywords and more permissive atom/variable names

f84ed3c

Elixir: better support for binaries/bitstrings

d478f11

Elixir: added support for the iex prompt for interactive examples

e78c848

added a failing test for elixir

1d51604

Elixir: fixed the failing test for elixir syntax highlighting

ede689e

nietaki mentioned this pull request Mar 2, 2017

Add support for Elixir #319

Closed

Elixir: Highlighting constructs like %{"THIS": :foo} as atoms

644e782

mikesamuel reviewed Mar 9, 2017

View reviewed changes

nietaki added 5 commits March 11, 2017 13:06

Elixir: fixed false-positive highlighting for 0XFF and 0O77

41c2cf2

Elixir: fixing the license text.

c078ac6

Elixir: fixing multiline charlists

b1b5f45

Elixir: more robust multiline strings and charlists

fcdb42c

Elixir: making string recognition simpler and more correct

0437005

amroamroamro added the enhancement label Mar 13, 2017

mikesamuel merged commit 760e6e7 into googlearchive:master Mar 15, 2017

sheharyarn mentioned this pull request Oct 7, 2018

Use correct language name for Elixir #562

Open

Add Elixir language support #478

Add Elixir language support #478

Uh oh!

Conversation

nietaki commented Mar 2, 2017

Uh oh!

googlebot commented Mar 2, 2017

Uh oh!

nietaki commented Mar 2, 2017

Uh oh!

googlebot commented Mar 2, 2017

Uh oh!

sheharyarn commented Mar 2, 2017

Uh oh!

nietaki commented Mar 5, 2017

Uh oh!

nietaki commented Mar 5, 2017

Uh oh!

mikesamuel left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mikesamuel left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nietaki commented Mar 9, 2017

Uh oh!

nietaki commented Mar 14, 2017

Uh oh!

mikesamuel commented Mar 15, 2017 via email

Uh oh!

Uh oh!