Strange behavior with tokens of import nodes

[jrfaller]

Hi!

I tried the following code to parse import statements :

source = 'import asttokens as at'
atok = asttokens.ASTTokens(source, parse=True)

I get the following AST:

Module
  Import
    alias

Which is OK. My purpose is to gather the position of the identifiers asttokens and at in the code. Therefore I was trying to get the tokens composing the alias node, which sounds logic because asttokens as at is and alias definition. However when I ask for the tokens composing the alias node, using list(atok.get_tokens(atok.tree.body[0].names[0])) I get the following result :

[Token(type=1, string=u'import', start=(1, 0), end=(1, 6), line=u'import asttokens as at', index=0, startpos=0, endpos=6)]

Which is was not what I was expecting, I was expecting to have the ['asttokens', 'as', 'at'] tokens.

Is there something I am missing?

Best regards.

[dsagal]

You are right, the built-in ast module doesn't seem to annotate the "alias" node with any useful information, so it doesn't have any location info. One reasonable thing we could do is add a method to mark_tokens.py like:

def visit_alias(self, node, first_token, last_token):
   asname = self._code.find_token(first_token, token.NAME, node.asname)
   return (asname, asname)

This would map the "alias" node to the token with the alias name ("at" in your case). If you experiment with this and feel this is a useful change, a test and a pull request would be appreciated. Note also that there are somewhat harder situations, like from os import path as _path, wait as _wait.

[jrfaller]

Hi!

Thanks for the answer!

I will try to cope with this problem, if I manage to do something, I will send a PR!

Cheers.