Semantic Syntax is incorrectly highlighting arguments as commands

[Jaykul]

Any bare (unquoted) string argument that has a - in it is being highlighted as a command:

[justinytchen]

The problem is that there is no way to distinguish between a function name in a declaration and a parameter from just the token information alone. I think that we'll need to add other checks (i.e. position or looking into the AST)

[rjmholt]

I think the important part is that the Test token and the That-Thing token have different flags:

> $ast = [System.Management.Automation.Language.Parser]::ParseInput('get-banana this-thing', [ref]$tok
s, [ref]$errs)
C:\Users\Robert Holt\Documents\Dev\Microsoft\PowerShellEditorServices [async-ps-consumer +0 ~4 -0 !]
> $toks[0].TokenFlags
CommandName
C:\Users\Robert Holt\Documents\Dev\Microsoft\PowerShellEditorServices [async-ps-consumer +0 ~4 -0 !]
> $toks[1].TokenFlags
None

That's how PSReadLine is able to differentiate them I imagine

[SeeminglyScience]

@rjmholt I think what they're getting at is that the current regex based highlighter treats the name in a function definition like a command (which doesn't have the flag).

Personally I'd rather have the name in a function definition be parsed as a bare word than the current situation (that's how both the ISE and PSRL work currently). But a fix for both might be just keeping track of if the previous token kind was Function. Then also use TokenFlags.CommandName.

[justinytchen]

There's no way to differentiate between the Get-AA token (should be labelled as a command/function) and the That-Thing token (which should be labelled as something else) with only the Token info. So without using additional data, we either have the situation we have now, or Get-AA would not be highlighted

[SeeminglyScience]

@justinytchen You also could change this up a bit:

https://github.com/PowerShell/PowerShellEditorServices/blob/c40734354631043cf3c390d36c9b1ef6a80b3949/src/PowerShellEditorServices/Services/TextDocument/Handlers/PsesSemanticTokensHandler.cs#L51-L54

to be:

Token[] tokens = file.ScriptTokens;
for (int i = 0; i < tokens.Length; i++)
{
    Token token = tokens[i];
    PushToken(token, builder);
    if (token.Kind == TokenKind.Function)
    {
        // + bounds checks
        Token next = tokens[++i];
        PushTokenAsFunctionNameIfBareWord(next, builder);
    }
}

[rjmholt]

I think what they're getting at is that the current regex based highlighter treats the name in a function definition like a command (which doesn't have the flag).

Yes, I noticed that. But in @Jaykul's original issue description he seems to be referring to the CommandAst situation rather than the FunctionDefinitionAst situation.

The FunctionDefinitionAst scenario is easy to address: we look back a token and see if it's the function keyword. Given that the grammar defines the full nature of the context, rather than just the tokens, we can't do a simple token search for all tokens, but I believe we can in this case.

A more sophisticated approach is to track our position in both the AST and the token array simulataneously and for each token look into where we are in the AST to determine the token semantics.

I've written an example where we track through tokens as we move through an AST here.

Or the converse, where we find the AST element corresponding to our token:

    /// <summary>
    /// An AST visitor to find the smallest AST containing a given IScriptPosition.
    /// </summary>
    internal class FindAstFromPositionVisitor : AstVisitor2
    {
        private readonly IScriptPosition _position;


        private Ast _astAtPosition;


        /// <summary>
        /// Create a FindAstFromPositionVisitor around a given position,
        /// ready to find the smallest AST containing that position.
        /// </summary>
        /// <param name="position">The position to find the containing AST of.</param>
        public FindAstFromPositionVisitor(IScriptPosition position)
        {
            _position = position;
        }


        /// <summary>
        /// Retrieve the computed smallest containing AST after a visit has been performed.
        /// </summary>
        /// <returns>The smallest AST containing the originally given position in the visited AST.</returns>
        public Ast GetAst()
        {
            return _astAtPosition;
        }


        public override AstVisitAction VisitArrayExpression(ArrayExpressionAst arrayExpressionAst) => VisitAst(arrayExpressionAst);


        public override AstVisitAction VisitArrayLiteral(ArrayLiteralAst arrayLiteralAst) => VisitAst(arrayLiteralAst);


        public override AstVisitAction VisitAssignmentStatement(AssignmentStatementAst assignmentStatementAst) => VisitAst(assignmentStatementAst);


        public override AstVisitAction VisitAttribute(AttributeAst attributeAst) => VisitAst(attributeAst);


        public override AstVisitAction VisitAttributedExpression(AttributedExpressionAst attributedExpressionAst) => VisitAst(attributedExpressionAst);


        public override AstVisitAction VisitBaseCtorInvokeMemberExpression(BaseCtorInvokeMemberExpressionAst baseCtorInvokeMemberExpressionAst) => AstVisitAction.SkipChildren;


        public override AstVisitAction VisitBinaryExpression(BinaryExpressionAst binaryExpressionAst) => VisitAst(binaryExpressionAst);


        public override AstVisitAction VisitBlockStatement(BlockStatementAst blockStatementAst) => VisitAst(blockStatementAst);


        public override AstVisitAction VisitBreakStatement(BreakStatementAst breakStatementAst) => VisitAst(breakStatementAst);


        public override AstVisitAction VisitCatchClause(CatchClauseAst catchClauseAst) => VisitAst(catchClauseAst);


        public override AstVisitAction VisitCommand(CommandAst commandAst) => VisitAst(commandAst);


        public override AstVisitAction VisitCommandExpression(CommandExpressionAst commandExpressionAst) => VisitAst(commandExpressionAst);


        public override AstVisitAction VisitCommandParameter(CommandParameterAst commandParameterAst) => VisitAst(commandParameterAst);


        public override AstVisitAction VisitConfigurationDefinition(ConfigurationDefinitionAst configurationDefinitionAst) => VisitAst(configurationDefinitionAst);


        public override AstVisitAction VisitConstantExpression(ConstantExpressionAst constantExpressionAst) => VisitAst(constantExpressionAst);


        public override AstVisitAction VisitContinueStatement(ContinueStatementAst continueStatementAst) => VisitAst(continueStatementAst);


        public override AstVisitAction VisitConvertExpression(ConvertExpressionAst convertExpressionAst) => VisitAst(convertExpressionAst);


        public override AstVisitAction VisitDataStatement(DataStatementAst dataStatementAst) => VisitAst(dataStatementAst);


        public override AstVisitAction VisitDoUntilStatement(DoUntilStatementAst doUntilStatementAst) => VisitAst(doUntilStatementAst);


        public override AstVisitAction VisitDoWhileStatement(DoWhileStatementAst doWhileStatementAst) => VisitAst(doWhileStatementAst);


        public override AstVisitAction VisitDynamicKeywordStatement(DynamicKeywordStatementAst dynamicKeywordStatementAst) => VisitAst(dynamicKeywordStatementAst);


        public override AstVisitAction VisitErrorExpression(ErrorExpressionAst errorExpressionAst) => VisitAst(errorExpressionAst);


        public override AstVisitAction VisitErrorStatement(ErrorStatementAst errorStatementAst) => VisitAst(errorStatementAst);


        public override AstVisitAction VisitExitStatement(ExitStatementAst exitStatementAst) => VisitAst(exitStatementAst);


        public override AstVisitAction VisitExpandableStringExpression(ExpandableStringExpressionAst expandableStringExpressionAst) => VisitAst(expandableStringExpressionAst);


        public override AstVisitAction VisitFileRedirection(FileRedirectionAst redirectionAst) => VisitAst(redirectionAst);


        public override AstVisitAction VisitForEachStatement(ForEachStatementAst forEachStatementAst) => VisitAst(forEachStatementAst);


        public override AstVisitAction VisitForStatement(ForStatementAst forStatementAst) => VisitAst(forStatementAst);


        public override AstVisitAction VisitFunctionDefinition(FunctionDefinitionAst functionDefinitionAst) => VisitAst(functionDefinitionAst);


        public override AstVisitAction VisitFunctionMember(FunctionMemberAst functionMemberAst) => VisitAst(functionMemberAst);


        public override AstVisitAction VisitHashtable(HashtableAst hashtableAst) => VisitAst(hashtableAst);


        public override AstVisitAction VisitIfStatement(IfStatementAst ifStmtAst) => VisitAst(ifStmtAst);
    
        public override AstVisitAction VisitIndexExpression(IndexExpressionAst indexExpressionAst) => VisitAst(indexExpressionAst);


        public override AstVisitAction VisitInvokeMemberExpression(InvokeMemberExpressionAst methodCallAst) => VisitAst(methodCallAst);


        public override AstVisitAction VisitMemberExpression(MemberExpressionAst memberExpressionAst) => VisitAst(memberExpressionAst);


        public override AstVisitAction VisitMergingRedirection(MergingRedirectionAst redirectionAst) => VisitAst(redirectionAst);


        public override AstVisitAction VisitNamedAttributeArgument(NamedAttributeArgumentAst namedAttributeArgumentAst) => VisitAst(namedAttributeArgumentAst);


        public override AstVisitAction VisitNamedBlock(NamedBlockAst namedBlockAst) => VisitAst(namedBlockAst);


        public override AstVisitAction VisitParamBlock(ParamBlockAst paramBlockAst) => VisitAst(paramBlockAst);


        public override AstVisitAction VisitParameter(ParameterAst parameterAst) => VisitAst(parameterAst);


        public override AstVisitAction VisitParenExpression(ParenExpressionAst parenExpressionAst) => VisitAst(parenExpressionAst);


        public override AstVisitAction VisitPipeline(PipelineAst pipelineAst) => VisitAst(pipelineAst);


        public override AstVisitAction VisitPropertyMember(PropertyMemberAst propertyMemberAst) => VisitAst(propertyMemberAst);


        public override AstVisitAction VisitReturnStatement(ReturnStatementAst returnStatementAst) => VisitAst(returnStatementAst);


        public override AstVisitAction VisitScriptBlock(ScriptBlockAst scriptBlockAst) => VisitAst(scriptBlockAst);


        public override AstVisitAction VisitScriptBlockExpression(ScriptBlockExpressionAst scriptBlockExpressionAst) => VisitAst(scriptBlockExpressionAst);


        public override AstVisitAction VisitStatementBlock(StatementBlockAst statementBlockAst) => VisitAst(statementBlockAst);


        public override AstVisitAction VisitStringConstantExpression(StringConstantExpressionAst stringConstantExpressionAst) => VisitAst(stringConstantExpressionAst);


        public override AstVisitAction VisitSubExpression(SubExpressionAst subExpressionAst) => VisitAst(subExpressionAst);


        public override AstVisitAction VisitSwitchStatement(SwitchStatementAst switchStatementAst) => VisitAst(switchStatementAst);


        public override AstVisitAction VisitThrowStatement(ThrowStatementAst throwStatementAst) => VisitAst(throwStatementAst);


        public override AstVisitAction VisitTrap(TrapStatementAst trapStatementAst) => VisitAst(trapStatementAst);


        public override AstVisitAction VisitTryStatement(TryStatementAst tryStatementAst) => VisitAst(tryStatementAst);


        public override AstVisitAction VisitTypeConstraint(TypeConstraintAst typeConstraintAst) => VisitAst(typeConstraintAst);


        public override AstVisitAction VisitTypeDefinition(TypeDefinitionAst typeDefinitionAst) => VisitAst(typeDefinitionAst);


        public override AstVisitAction VisitTypeExpression(TypeExpressionAst typeExpressionAst) => VisitAst(typeExpressionAst);


        public override AstVisitAction VisitUnaryExpression(UnaryExpressionAst unaryExpressionAst) => VisitAst(unaryExpressionAst);


        public override AstVisitAction VisitUsingExpression(UsingExpressionAst usingExpressionAst) => VisitAst(usingExpressionAst);


        public override AstVisitAction VisitUsingStatement(UsingStatementAst usingStatementAst) => VisitAst(usingStatementAst);


        public override AstVisitAction VisitVariableExpression(VariableExpressionAst variableExpressionAst) => VisitAst(variableExpressionAst);


        public override AstVisitAction VisitWhileStatement(WhileStatementAst whileStatementAst) => VisitAst(whileStatementAst);


        private AstVisitAction VisitAst(Ast ast)
        {
            if (!AstContainsPosition(ast))
            {
                return AstVisitAction.SkipChildren;
            }


            _astAtPosition = ast;
            return AstVisitAction.Continue;
        }


        private bool AstContainsPosition(Ast ast)
        {
            return IsBefore(ast.Extent.StartScriptPosition, _position)
                && IsBefore(_position, ast.Extent.EndScriptPosition);
        }


        private static bool IsBefore(IScriptPosition left, IScriptPosition right)
        {
            return left.LineNumber < right.LineNumber
                || (left.LineNumber == right.LineNumber && left.ColumnNumber <= right.ColumnNumber);
        }
    }

[rjmholt]

Ok so the problem is actually that generic tokens with no flags are currently highlighted as functions.

Bareword arguments without -s are tokenised as identifiers rather than generic tokens, so they fall through to the grammar.

We can recognise a bareword identifier most correctly as an identifier or a generic token that is subordinate to a command token... Harder to do though.

We need to pick a new value based on this list. I would guess Member, Property or EnumMember would work.

[TylerLeonhardt]

Property is probably the least threatening because Member in the Monokai theme was lime green lol