run=False in CLI prevents some config keys from being accepted

[cifkao]

🐛 Bug

Instantiating LightningCLI with run=False prevents some config keys (namely seed_everything and ckpt_path) from being accepted. This seems to be because these config keys are defined by a subcommand parser, and these are not initialized when run=False is passed.

This makes it impossible to load config files created by Lightning during training since they contain the seed_everything ckpt_path key.

To Reproduce

Boring model

error: 'Configuration check failed :: No action for destination key "ckpt_path" to check its value.'

Expected behavior

Should run without error.

Environment

• CUDA:
  • GPU:
    • Tesla K80
  • available: True
  • version: 11.1
• Packages:
  • numpy: 1.21.5
  • pyTorch_debug: False
  • pyTorch_version: 1.10.0+cu111
  • pytorch-lightning: 1.5.10
  • tqdm: 4.63.0
• System:
  • OS: Linux
  • architecture:
    • 64bit
  • processor: x86_64
  • python: 3.7.12
  • version: # 1 SMP Tue Dec 7 09:58:10 PST 2021

cc @carmocca @mauvilsa

[mauvilsa]

There is a typo in the reproduction script. It says seed_everyting and it should be seed_everything. The error shows --seed_everything which means that there is no problem in that argument. ckpt_path is an argument of the subcommand and it shouldn't be there, since the subcommands are disabled. In my view everything is working as expected.

It is not intended that you implement a CLI with subcommands, and then implement a second CLI without subcommands expecting that the saved config from the first CLI will work. The targeted reproducibility is only for the same CLI. And it should be this way because if you truly want to reproduce, the exact same version of the software should be used.

[cifkao]

Ah, my bad – fixed it.

Then the actual problem was with ckpt_path and my issue was this: I ran training from the command line (i.e. with run=True) and this created a config file with ckpt_path: null. I then tried to load the model in a notebook by instantiating the CLI with run=False, which fails due to this issue.

To me, this is not implementing a new CLI, it's using the same CLI in a different mode. With run=False, I would still expect the subcommands to be defined, just not executed. I don't know if this can be reasonably done with the way this is set up, and maybe I misunderstood how run=False was supposed to work, but I think the use case is quite reasonable.

[mauvilsa]

LightningCLI is a class intended to ease the implementation of command line tools. Any change in the python source code in general is a new CLI. For more context also read lightning_cli.html#notes-related-to-reproducibility.

With run=False only the classes are instantiated. It is up to the developer to implement which and how the Trainer methods are called. And to add to the argument parser any additional needed parameters.

[stale]

This issue has been automatically marked as stale because it hasn't had any recent activity. This issue will be closed in 7 days if no further activity occurs. Thank you for your contributions, Pytorch Lightning Team!

[JiangWenPL]

I would say loading a model from a configuration file in jupyter notebook is a quite common scenario for many research projects. For example, we may want to create colab demo for our research projects or run some interactive visualization or experiment on downstream tasks with our model.
A simple workaround for current version could be something like this:

class NBCLI(LightningCLI):
    def fit(self, *args, **kwargs):
        pass

You just need to instantiate this CLI instead of the CLI you are currently using. But still I think it would be good if those scenarios can be addressed by the LightningCLI or have some documentations or examples to make the LightningCLI much easier to use for researchers. I would like to help if you think instantiating the model with CLI follows the philosophy of this library.

[ribesstefano]

I'm still experiencing the same issue, i.e. code breaking when run=False. Even the example in the documentation breaks:

cli = LightningCLI(MyModel, run=False)  # True by default
# you'll have to call fit yourself:
cli.trainer.fit(cli.model)

[mauvilsa]

@Toekan responding to you here since the topic of #18885 is different and it fits better here.

Is there no easier way to load back in the whole state of the trainer or the model weights from the checkpoint file?

I do think that loading a model from a checkpoint should be easy to do and not depend on LightningCLI. Just because a model was trained with a CLI, there is no reason why loading the model should need the CLI. For this, some time ago I opened pull request #18105. You are free to look at it and comment. Not sure why it hasn't been reviewed in a long time, so you could ping people and express your interest in that feature.

After reading around and trying things out for hours, the only working way I could come up with was:

cli = LightningCLI(
      MyLitModelModule,
      MyLitDataModule,
      run=False,
  )

  model = cli.model.load_from_checkpoint(
      "lightning_logs/version_xx/checkpoints/my_checkpoint.ckpt"
      # Here I need to pass in every argument that expects an instantiated class by hand
      model=cli.model.model,
      loss_fn=cli.model.loss_fn,
      activation=cli.model.activation,
      train_metrics=cli.model.train_metrics,
      ...
  )

Is this the easiest way to achieve loading the model from a lightningCLI checkpoint? Having to pull every instantiated class from the instantiated cli, just to be able to do load_from_checkpoint is obviously a considerably worse experience than what run=True has to offer.

Extending a bit more my response in #12302 (comment). If you use run=False, then LightningCLI only instantiates classes. It is up to the developer to implement the rest. If you want to support loading from a checkpoint, then subclass LightningCLI and add a ckpt_path to the parser with type Optional[str]. And the loading of the checkpoint should be implemented, conditional on whether the ckpt_path is provided or not. The user experience of the CLI depends on what the developer decides to implement.

I understand the strict distinction you are trying to create between config files for configuration, a new CLI for changes in source code (very happy LightningCLI didn't go down the jinja route), but I find it hard to fully understand where checkpoints sit in this or why they have to be linked to trainer commands rather than to the trainer itself.

Hopefully my previous comments helped a bit.