Skip to content

Conversation

sharkdp
Copy link
Contributor

@sharkdp sharkdp commented Apr 10, 2025

Summary

Track the reachability of nested scopes within their parent scopes. We use this as an additional requirement for emitting unresolved-reference diagnostics (and in the future, unresolved-attribute and unresolved-import). This means that we only emit unresolved-reference for a given use of a symbol if the use itself is reachable (within its own scope), and if the scope itself is reachable. For example, no diagnostic should be emitted for the use of x here:

if False:
    x = 1

    def f():
        print(x)  # this use of `x` is reachable inside the `f` scope,
                  # but the whole `f` scope is not reachable.

There are probably more fine-grained ways of solving this problem, but they require a more sophisticated understanding of nested scopes (see astral-sh/ty#210, in particular astral-sh/ty#210). But it doesn't seem completely unreasonable to silence this specific kind of error in unreachable scopes.

Test Plan

Observed changes in reachability tests and ecosystem.

@sharkdp sharkdp added the ty Multi-file analysis & type inference label Apr 10, 2025
Copy link
Contributor

mypy_primer results

Changes were detected when running on open source projects
mypy_primer (https://github.com/hauntsaninja/mypy_primer)
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/mypy_primer/mypy_primer/utils.py:33:24: Name `ILLEGAL_PATH_CHARS` used when not defined
- Found 709 diagnostics
+ Found 708 diagnostics

werkzeug (https://github.com/pallets/werkzeug)
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/werkzeug/src/werkzeug/middleware/shared_data.py:139:45: Name `fnmatch` used when not defined
- Found 770 diagnostics
+ Found 769 diagnostics

rich (https://github.com/Textualize/rich)
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:31:17: Name `WindowsCoordinates` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:31:47: Name `WindowsCoordinates` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:38:32: Name `_win32_console` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:44:34: Name `CURSOR_SIZE` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:48:13: Name `_win32_console` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:50:26: Name `StubScreenBufferInfo` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:52:13: Name `_win32_console` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:60:16: Name `LegacyWindowsTerm` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:61:40: Name `WindowsCoordinates` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:61:63: Name `CURSOR_Y` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:61:77: Name `CURSOR_X` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:64:16: Name `LegacyWindowsTerm` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:65:36: Name `WindowsCoordinates` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:66:17: Name `SCREEN_HEIGHT` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:66:36: Name `SCREEN_WIDTH` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:71:16: Name `LegacyWindowsTerm` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:87:16: Name `LegacyWindowsTerm` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:103:47: Name `DEFAULT_STYLE_ATTRIBUTE` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:111:16: Name `LegacyWindowsTerm` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:128:16: Name `LegacyWindowsTerm` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:145:16: Name `LegacyWindowsTerm` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:162:16: Name `LegacyWindowsTerm` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:179:16: Name `LegacyWindowsTerm` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:200:16: Name `LegacyWindowsTerm` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:202:17: Name `WindowsCoordinates` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:202:40: Name `CURSOR_Y` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:204:39: Name `SCREEN_WIDTH` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:207:27: Name `DEFAULT_STYLE_ATTRIBUTE` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:207:59: Name `SCREEN_WIDTH` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:218:16: Name `LegacyWindowsTerm` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:222:39: Name `SCREEN_WIDTH` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:222:54: Name `CURSOR_X` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:222:70: Name `CURSOR_POSITION` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:226:13: Name `DEFAULT_STYLE_ATTRIBUTE` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:227:20: Name `SCREEN_WIDTH` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:227:35: Name `CURSOR_X` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:228:19: Name `CURSOR_POSITION` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:239:16: Name `LegacyWindowsTerm` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:242:17: Name `WindowsCoordinates` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:242:36: Name `CURSOR_Y` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:245:39: Name `CURSOR_X` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:248:27: Name `DEFAULT_STYLE_ATTRIBUTE` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:248:59: Name `CURSOR_X` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:255:18: Name `WindowsCoordinates` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:256:16: Name `LegacyWindowsTerm` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:266:18: Name `WindowsCoordinates` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:267:16: Name `LegacyWindowsTerm` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:277:18: Name `WindowsCoordinates` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:278:16: Name `LegacyWindowsTerm` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:288:16: Name `LegacyWindowsTerm` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:293:34: Name `WindowsCoordinates` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:293:57: Name `CURSOR_Y` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:293:75: Name `CURSOR_X` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:300:16: Name `LegacyWindowsTerm` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:305:34: Name `WindowsCoordinates` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:305:57: Name `CURSOR_Y` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:305:75: Name `CURSOR_X` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:312:16: Name `LegacyWindowsTerm` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:317:34: Name `WindowsCoordinates` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:317:57: Name `CURSOR_Y` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:317:71: Name `CURSOR_X` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:324:33: Name `StubScreenBufferInfo` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:325:30: Name `COORD` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:325:36: Name `SCREEN_WIDTH` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:325:54: Name `CURSOR_Y` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:330:16: Name `LegacyWindowsTerm` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:334:34: Name `WindowsCoordinates` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:334:57: Name `CURSOR_Y` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:341:16: Name `LegacyWindowsTerm` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:344:34: Name `WindowsCoordinates` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:344:53: Name `CURSOR_Y` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:351:16: Name `LegacyWindowsTerm` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:354:34: Name `WindowsCoordinates` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:354:57: Name `CURSOR_Y` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:354:71: Name `CURSOR_X` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:361:35: Name `StubScreenBufferInfo` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:362:30: Name `COORD` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:362:39: Name `CURSOR_Y` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:367:16: Name `LegacyWindowsTerm` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:371:20: Name `WindowsCoordinates` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:371:43: Name `CURSOR_Y` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:371:61: Name `SCREEN_WIDTH` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:376:16: Name `LegacyWindowsTerm` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:385:48: Name `CURSOR_SIZE` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:389:16: Name `LegacyWindowsTerm` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:398:48: Name `CURSOR_SIZE` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:402:16: Name `LegacyWindowsTerm` used when not defined
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/rich/tests/test_win32_console.py:409:16: Name `LegacyWindowsTerm` used when not defined
- Found 886 diagnostics
+ Found 798 diagnostics

@sharkdp sharkdp marked this pull request as ready for review April 10, 2025 11:19
@AlexWaygood
Copy link
Member

werkzeug (https://github.com/pallets/werkzeug)
- warning[lint:unresolved-reference] /tmp/mypy_primer/projects/werkzeug/src/werkzeug/middleware/shared_data.py:139:45: Name `fnmatch` used when not defined
- Found 770 diagnostics
+ Found 769 diagnostics

This one is interesting -- looks like a buggy type annotation in werkzeug to me. Based on the type annotation of the disallow parameter, I think we're correct in thinking that the if disallow is not None branch is unreachable... but I can't imagine that you're really only allowed to pass None in to the disallow parameter (https://github.com/pallets/werkzeug/blob/7868bef5d978093a8baa0784464ebe5d775ae92a/src/werkzeug/middleware/shared_data.py#L103-L139)

@sharkdp
Copy link
Contributor Author

sharkdp commented Apr 10, 2025

This one is interesting -- looks like a buggy type annotation in werkzeug to me. Based on the type annotation of the disallow parameter, I think we're correct in thinking that the if disallow is not None branch is unreachable... but I can't imagine that you're really only allowed to pass None in to the disallow parameter (https://github.com/pallets/werkzeug/blob/7868bef5d978093a8baa0784464ebe5d775ae92a/src/werkzeug/middleware/shared_data.py#L103-L139)

Came to the exact same conclusion. I was also confused at first.

Edit: this might explain it. Maybe None was the only thing ever passed in at runtime. Let's just say it's Carl's fault.

image

Copy link
Member

@AlexWaygood AlexWaygood left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, this looks excellent

@sharkdp sharkdp enabled auto-merge (squash) April 10, 2025 11:54
@sharkdp sharkdp merged commit 4d50ee6 into main Apr 10, 2025
20 checks passed
@sharkdp sharkdp deleted the david/scope-reachability branch April 10, 2025 11:56
@carljm
Copy link
Contributor

carljm commented Apr 10, 2025

Love this!

dcreager added a commit that referenced this pull request Apr 10, 2025
* main: (30 commits)
  [red-knot] Silence `unresolved-import` in unreachable code (#17336)
  red_knot_python_semantic: move TODO comment
  red_knot_python_semantic: rename `lint()` and `report()`
  ruff_db: use `Annotation::get_message` in more places
  red_knot_python_semantic: tweak docs on building reporter builders
  red_knot_python_semantic: remove the "old" secondary message type
  red_knot_python_semantic: replace one use of "old" secondary diagnostic messages
  red_knot_python_semantic: update revealed type snapshots
  ruff_db: tweak how the revealed type diagnostic is rendered
  red_knot: add explicit test for concise `reveal_type` diagnostic
  red_knot_python_semantic: remove `InferContext::report_diagnostic`
  red_knot_python_semantic: add "reporter" API
  Bump 0.11.5 (#17337)
  [red-knot] Silence `unresolved-attribute` in unreachable code (#17305)
  Revert "[red-knot] Type narrowing for assertions (#17149)" (#17335)
  [red-knot] Type narrowing for assertions (#17149)
  [red-knot] avoid unnecessary evaluation of visibility constraint on definitely-unbound symbol (#17326)
  update cargo-dist (#17325)
  [red-knot] Fix double hovers/inlays in playground (#17334)
  [red-knot] Track reachability of scopes (#17332)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ty Multi-file analysis & type inference
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants