Correct identifier lexing

[matklad]

Rust uses _ and XID_Start, XID_Continue to define identifiers. We need to add lexer test for some edge cases with these classes.

[mystor]

Let's just depend on https://crates.io/crates/unicode-xid for now - that makes sure we don't get it wrong.

[matklad]

@mystor we already do that: https://github.com/matklad/libsyntax2/blob/master/Cargo.toml#L8, however, we don’t have thorough tests for this. While I am pretty sure that the crate is correct, I think it is still important to have independent tests for this :)

[theotherphil]

Is this still required now that this crate is using the rustc lexer?

(Sorry if all the 'is this done'/'this is done' comments on old issues are annoying - I'm looking through open issues to try and find something easy to work on.)

[lnicola]

@theotherphil maybe you want to try #1592?

[theotherphil]

Thanks for the suggestion - I’ve started work on another couple of issues, but I’ll have a look at that one if it’s still open by the time I’m done with these.

Edit: I couldn't work out how to get started on the other issue I tried, so I looked into your suggestion instead. I think I have a fix now in #1604

[matklad]

yeah, this can be just closed now!