Sibling modules are unbound in qualified Menhir parsers

[brendanzab]

Expected Behavior

When using (include_subdirs qualified), a Menhir parser in a qualified subdirectory should be able to access its sibling modules. This is useful for if a programming language has a number of intermediate languages, each with its own parser and AST module, nested within their own subdirectories:

├── dune
├── dune-project
├── foo.ml
├── lang
│   ├── ast.ml
│   ├── dune
│   ├── lexer.ml
│   └── parser.mly
└── run.t

Actual Behavior

Sibling modules are unbound in qualified Menhir parsers:

File "lang/parser.mly", line 3, characters 8-16:
Error: Unbound module Ast
[1]

Reproduction

#11118

Specifications

• Version of dune (output of dune --version): 3.12.0-1285-gf7af2d0
• Version of ocaml (output of ocamlc --version): The OCaml toplevel, version 5.1.1
• Operating system (distribution and version): macOS 15.0.1

[brendanzab]

I tried to clarify the issue further in comment thread on the PR: #11118 (comment), but thought I’d try to spell it out more clearly here, in case it’s helpful to anyone in the future.

To summarise, given the following menhir file:

%token EOF

%start <Ast.expr> expr

%%

expr:
| EOF { Ast.Unit }

I’ve tried the following approaches to set up a nested menhir parser that can refer to sibling modules:

• Top-level dune file:

  ├── dune            (menhir stanza here)
  ├── dune-project
  ├── foo.ml
  └── lang
      ├── ast.ml
      ├── lexer.ml
      └── parser.mly
  

  • Using a path:

    (menhir
     (modules lang/parser))

    Fails with:

    Error: dune__exe__Lang/parser corresponds to an invalid module name
    -> required by _build/default/.foo.eobjs/dune__exe.ml-gen
    -> required by _build/default/.foo.eobjs/byte/dune__exe.cmi
    -> required by _build/default/lang/parser__mock.mli.inferred
    -> required by _build/default/lang/parser.ml
    -> required by alias lang/all
    -> required by alias default
    

  • Variations of (modules Lang.Parser), (modules Lang__Parser), (modules dune__exe__Lang__parser)

    I forget what the errors were here.

  • Using subdir:

    (subdir lang
     (menhir
      (modules parser)))

    Fails with:

    File "lang/parser.mly", line 3, characters 8-16:
    Error: Unbound module Ast
    [1]
    

• Nested dune file:

  ├── dune
  ├── dune-project
  ├── foo.ml
  └── lang
      ├── ast.ml
      ├── dune        (menhir stanza here)
      ├── lexer.ml
      └── parser.mly
  

  (menhir
   (modules parser)

  Fails with:

  File "lang/parser.mly", line 3, characters 8-16:
  Error: Unbound module Ast
  [1]
  

I’m not sure if any of those, or none of those are intended to work with (include_subdirs qualified).

[brendanzab]

This seems potentially related to #8989, though from what I understand, that issue is for the specific case of the .mly being the “group interface” of the module group, where as this issue is about .mly files being sibling modules within a module group. I’m not sure if solving one issue would fix the other, or vice-versa.

[brendanzab]

For a concrete use case, this project currently contains some modules that I’d like to convert to using (include_subdirs qualified): https://github.com/brendanzab/language-garden/tree/02ee6dd6276aacee30935d785f1ad440524a7180/compile-closure-conv/lib

[rmonat]

Bumping this. We are also encountering this issue in the Mopsa static analyzer, for our one our parsers (link to WIP MR)

[Alizter]

We have two choices on how to move forward with this:

1. We can detect (include_subdirs qualified) being used with the (menhir) and (ocamllex) stanzas and raise a user error indicating that this is not supported at the moment. This is strictly better than the current situation which is confusing.
2. We can support (include_subdirs qualified) and menhir working in tandem. I haven't looked at the details on what would need to change or even if it makes sense to do just yet.

To me, waiting around for 2 seems pointless unless somebody knows what needs to be done. I think 1 can easily be done and should provide a better user experience when encountered.