Lightning init [app/pl-app] issues

[luca-medeiros]

🐛 Bug

Running lightning init app|pl-app raises internal issues related to installation.

To Reproduce

Reproduced on Colab:
https://colab.research.google.com/drive/1oY5hQNysCh4oLeFcxtU2frVZnkTU2U80?usp=sharing

Step by step:
pip install lightning

1. lightning init app test

Traceback (most recent call last):
  File "/usr/local/bin/lightning", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/lightning_app/cli/lightning_cli.py", line 218, in init_app
    cmd_init.app(name)
  File "/usr/local/lib/python3.7/dist-packages/lightning_app/cli/cmd_init.py", line 15, in app
    new_resource_name, name_for_files = _make_resource(resource_dir="app-template", resource_name=app_name)
  File "/usr/local/lib/python3.7/dist-packages/lightning_app/cli/cmd_init.py", line 42, in _make_resource
    os.rename(os.path.join(new_resource_name, "placeholdername"), os.path.join(new_resource_name, name_for_files))
FileNotFoundError: [Errno 2] No such file or directory: '/content/test/placeholdername' -> '/content/test/test'

placeholdername not inside of cli/app-template.

2. lightning init pl-app train_script.py (train_script.py contains the boring model)

⠋ Copying app files
Traceback (most recent call last):
  File "/usr/local/bin/lightning", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/lightning_app/cli/lightning_cli.py", line 254, in init_pl_app
    cmd_pl_init.pl_app(source_dir=source_dir, script_path=script_path, name=name, overwrite=overwrite)
  File "/usr/local/lib/python3.7/dist-packages/lightning_app/cli/cmd_pl_init.py", line 80, in pl_app
    shutil.copytree(template_dir, destination, ignore=shutil.ignore_patterns("node_modules", "build"))
  File "/usr/lib/python3.7/shutil.py", line 318, in copytree
    names = os.listdir(src)
FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/lib/python3.7/dist-packages/lightning_app/cli/pl-app-template'

Expected behavior

Creation of the sample apps from CLI.

Environment

• CUDA:
  - GPU:
  - GeForce RTX 2080 Ti
  - available: True
  - version: 10.2
• Packages:
  - numpy: 1.23.1
  - pyTorch_debug: False
  - pyTorch_version: 1.12.0+cu102
  - pytorch-lightning: 1.7.0dev
  - lightning: 2022.7.18
  - lightning-app: 0.5.2
  - tqdm: 4.64.0
• System:
  - OS: Linux
  - architecture:
  - 64bit
  - ELF
  - processor: x86_64
  - python: 3.8.13
  - version: 200-Ubuntu SMP Wed Jun 22 19:53:37 UTC 2022

Additional context

Analyzing further, I found out that the src/lightning_app/cli folder is 'incomplete' when pip installing.

While pl-app-template is completely missing, component-template, app-template, and react-ui-template contain only README files.

Discussed with the team in Slack and came to the conclusion that these directories are currently not added to the dist package:

• src/lightning_app/cli/app-template
• src/lightning_app/cli/component-template
• src/lightning_app/cli/react-ui-template
• src/lightning_app/cli/pl-app-template

Also, we could improve the lightning-app CLI CI tests too. But that might be more appropriate as a different issue.

[manskx]

A brief background, these directories contain templates for building lightning apps. When a user does lightning init <app_name> it copies these directories to the app destination and initializes the app (perhaps rename the app or install dependencies ....)

• src/lightning_app/cli/app-template
• src/lightning_app/cli/component-template
• src/lightning_app/cli/react-ui-template
• src/lightning_app/cli/pl-app-template

As @luca-medeiros mentioned, these directories are not added to the dist package, therefore the framework couldn't copy the template directories and initialize the app.

In this PR I added the templates directories to the package Manifest which is more like a temporary solution.

Soon, I think we would like to have some remote templates that are not part of the lightning package and downloaded on demand. @luca-medeiros will you be interested to help with remote templates?

[luca-medeiros]

@manskx
Appreciate the background explanation 👍. As the PR gets merged, will we see a new release soon?

I suppose #13757 deals with the remote templates you mentioned. It would be awesome to help out with that!

[stale]

This issue has been automatically marked as stale because it hasn't had any recent activity. This issue will be closed in 7 days if no further activity occurs. Thank you for your contributions - the Lightning Team!