fix: clean matched vars after chained and non-chained rule

[airween]

what

This PR changes the code behavior: now the engine cleans the MATCHED_VAR* variables after chained and non-chained rules too.

why

Until now if there was a single (non-chained) rule, and if any of the MATCHED_VAR* variable were filled, then the next rule which used them accessed the filled value, even the rule does not use chain action.

references

See issue #3382.

This PR fixes #3382.

other notes

please see commit 5572ac0; I added this change because the first test on Windows was failed. It seems like the argument processing order is non-deterministic, at least it's different on Windows (see the log: the first argument is the last from the QUERY_STRING, and the tests were success on all other platforms).

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[airween]

@mirkodziadzka-avi could you take a review on this?

[MirkoDziadzka]

Looks good to me, thanks.

Can we also update the documentation? I think this is at least as important than this change.

[airween]

Looks good to me, thanks.

Great, thanks!

Can we also update the documentation? I think this is at least as important than this change.

Indeed, after the merge I'm going to update it. Thank you!

[fzipi]

Thanks for adding so many tests.

[theseion]

Why this check? If I understand correctly, m_ruleId == 0 would be an invalid rule (caught exception). Would we even get here then? As far as I can tell, nothing bad would happen if m_ruleId == 0, since cleanMatchedVars() operates on the transaction only.

[airween]

If I understand correctly, m_ruleId == 0 would be an invalid rule

Yes, except if it's a chained rule. Generally, every chained rule (rules?) has (have?) only one unique id. In libmodsecurity3, despite you set up the id action at the first rule (in a chained rule), the last rule will own that id. Therefore this condition (m_ruleId > 0) tells us this is the end of a rule, no matter that's chained or not, we should clean the MATCHED_* variables.

[theseion]

I see. From the code (and the comment) it seems like the condition is redundant though, since end_exec: will always be evaluated for the last rule. So I still don't see why the extra condition is necessary.

[airween]

Because we need to clear MATCHED_* variables only if the rule is a standalone rule (not chained), or it's the last part of a chained rule.

This means if the rule is part if a chained rule, but not the last, then we don't clean these group of variables.

If you see the diff, we can say this solves the issue.

[theseion]

I understand, and that's exactly what I'm saying... this will only be true when all chained rules have been processed. This will only be true when the rule is not chained. In all cases,end_exec will only be evaluated when the last rule has been reached, whether it's the last in a chain or a single rule. Hence: end_exec is equivalent to m_ruleId > 0. Unless I still don't get it...

[airween]

Let me approach it from a different direction: if that condition were not there, the MATCHED_* variables would be deleted after each rule was evaulated. Not just after standalone rules, but in case of every part in chained rules. For eg. take a look at the initial rule in original issue:

SecRule ARGS pattern "chain,deny,id:28"
  SecRule MATCHED_VARS_NAMES "@eq ARGS:param"

and the request:

GET /?foo=1&bar=2&baz=pattern

If the condition were not there, then the first part of chained rule matches, MATCHED_* variables are set, but the engine clears them immediately after the end of processing, therefore the second part of the chained rule wont match, because MATCHED_VARS_NAMES is empty.

[MirkoDziadzka]

2 cents:

• I think when there are different opinions what the code is doing, having a test to confirm which interpretation is correct would be always nice ;-)
• How about clearing matched vars not at the end, but as the first action when starting to evaluate a rule (or chain of rules)? Should be semantically the same and we need only a single place to clean matched vars.

[airween]

When I worked on this fix I made several tests, this is why I could make this patch 😃.

Btw I try to figure out how can I solve the issue with your idea (clear vars before the evaluation). (As I remember I tried to go on this way, but I rejected for some reason... Let me check again.)

[airween]

Okay, I think I know why did I reject that solution.

Let's approach the issue from the transaction's direction. Transaction object calls the evaluate() function here, for eg. (there are several other call, in each phases).

The evaluate() is part of transaction's m_rules member, which is a RuleSet type, see here. m_rules->evaluate() has two arguments: phase and transaction.

It grabs the rules in phase that called, see here, and iterates through them. There is an auto cast here, but the rule which created here can be a RuleWithActions or RuleWithOperator.

But only the RuleWithOperator has cleanMatchedVars.

I didn't want to touch this, therefore I tried to find a solution at the end of the rule processing. But we can find a different solution, eg. we can move the cleanMatchedVars() to another class. Then I think we can make clean method easier.

[theseion]

Thanks for the explanation @airween. I still don't understand why the condition is necessary, but your tests prove that the code works, so that's good enough for me. I suggest you add a comment before the condition that explains what the condition is for. Then I'll be happy.