Skip to content

Refactor linear/quadratic expression compilers #3651

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 41 commits into from
Aug 4, 2025

Conversation

jsiirola
Copy link
Member

Fixes # .

Summary/Motivation:

The linear / quadratic / parameterized / templated expression walkers and compilers contain a large amount of repeated code. This PR refactors the walkers so that the bulk of repeated code can be deleted. The key changes are:

  • Move the "fixed" argument handlers from the parameterized walkers into the base (linear/quadratic) walkers
  • Don't assume that coefficients (that are floats in the base walkers and potentially expressions in other walkers) can be implicitly be case to bool or compared to 1. This change imparts a small performance hit (~1% for the LP writer; no noticeable effect on the other writers).

Other changes in this PR

  • Parameterized walkers are simple enough that they are now together in a single "parameterized.py" module
  • A slight change to how we process bounds in constraints and variables makes the code simpler and slightly more efficient (or at least no more inefficient)
  • The introduction of TemplateDataMixin classes for Constraint and Objectives allow us to templatize scalar components in addition to indexed components
  • Simplify the ScalarObjective class, and introduce an AbstractScalarObjective class (with the use of the @disable_methods decorator). This follows the pattern used elsewhere (e.g., in Constriant)
  • Fix a bug in how we pass InvalidNumbers through unary operators
  • Test the LinearStandardForm compiler on regular and templatized models
  • Fix a bug in the TemplateVarRecorder when the user provides a (partial) variable ordering.

While this PR will slow down the LP writer a little (1-2%), it resolves enough issues in the Parameterized and Templatized walkers that I think we should adopt it. There are a number of future developments in the works (that build on this PR) that should more than make up for this performance hit [moving the LP writer to leverage the standard form compiler to eliminate the need for sorting in the lp_writer and the use of dicts in the compiler for coefficient deduplication).

Changes proposed in this PR:

  • (see above)

Legal Acknowledgement

By contributing to this software project, I have read the contribution guide and agree to the following terms and conditions for my contribution:

  1. I agree my contributions are submitted under the BSD license.
  2. I represent I am authorized to make the contributions and grant the license. If my employer has rights to intellectual property that includes these contributions, I represent that I have received permission to make contributions and grant the required license on behalf of that employer.

jsiirola added 27 commits June 9, 2025 12:14
- fix BeforeChildDispatcher inheritance
- leverage constant_flag / multiplier_flag for checking coefficients
- changing argument ordering
- more consistent handling of 0*expressions and quadratic results
Copy link

codecov bot commented Jun 29, 2025

Codecov Report

❌ Patch coverage is 92.16966% with 48 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.18%. Comparing base (82906ae) to head (f8562a0).
⚠️ Report is 44 commits behind head on main.

Files with missing lines Patch % Lines
pyomo/repn/linear_template.py 76.36% 13 Missing ⚠️
pyomo/core/base/objective.py 77.14% 8 Missing ⚠️
pyomo/core/base/constraint.py 80.00% 7 Missing ⚠️
pyomo/repn/linear.py 95.13% 7 Missing ⚠️
pyomo/repn/util.py 80.00% 6 Missing ⚠️
pyomo/repn/quadratic.py 97.31% 5 Missing ⚠️
pyomo/core/base/var.py 95.83% 1 Missing ⚠️
pyomo/repn/parameterized.py 98.85% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3651      +/-   ##
==========================================
+ Coverage   89.08%   89.18%   +0.09%     
==========================================
  Files         891      890       -1     
  Lines      102899   102774     -125     
==========================================
- Hits        91671    91660      -11     
+ Misses      11228    11114     -114     
Flag Coverage Δ
builders 26.75% <25.44%> (+0.02%) ⬆️
default 85.77% <92.16%> (?)
expensive 34.11% <34.91%> (?)
linux 86.95% <92.16%> (-1.89%) ⬇️
linux_other 86.95% <92.16%> (+0.09%) ⬆️
osx 83.24% <92.16%> (+0.09%) ⬆️
win 85.14% <92.16%> (+0.08%) ⬆️
win_other 85.14% <92.16%> (+0.08%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Comment on lines +41 to +50
return val
return 2 # something not 0 or 1

@staticmethod
def multiplier_flag(val):
if val.__class__ in native_numeric_types:
if not val:
return 2
return val
return 2 # something not 0 or 1
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should these both return 2?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure. It really doesn't matter. The point (as indicated by the comment) is that it is just anything other 0 or 1. Because "2" is not unique (the val could actually be float(2)), there really isn't a driver to differentiate between 0, 2, and non-native numeric types.

Comment on lines -979 to +1019
self.assertIsNone(repn.quadratic)
self.assertEqual(repn.quadratic, None)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not just use assertIsNone?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was part of leftover debugging. The error you get when repn.quadratic is not None is remarkable unhelpful, whereas using assertEqual gives an error message that says what repn.quadratic actually is.

@emma58 emma58 self-requested a review July 8, 2025 12:36
@jsiirola jsiirola requested a review from mrmundt July 10, 2025 16:32
@github-project-automation github-project-automation bot moved this from Todo to Reviewer Approved in August 2025 Release Jul 16, 2025
@blnicho blnicho self-requested a review July 22, 2025 18:42
Copy link
Contributor

@emma58 emma58 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like this very much! A few questions in the comments, but I think this looks good (and I've actually been using it for awhile because I raided the cookie jar...)

@jsiirola jsiirola requested a review from emma58 August 4, 2025 17:53
@blnicho blnicho merged commit fa799a9 into Pyomo:main Aug 4, 2025
35 checks passed
@github-project-automation github-project-automation bot moved this from Reviewer Approved to Done in August 2025 Release Aug 4, 2025
@jsiirola jsiirola deleted the repn-refactor branch August 8, 2025 22:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
No open projects
Development

Successfully merging this pull request may close these issues.

5 participants