Reckless: Features list for UI implementation

[ShahanaFarooqui]

The reckless plugin shows great promise as a plugin manager for UI-based workflows, but a few key features are needed to make it fully usable in this context. Below are the remaining requirements, listed in descending priority order:

Streaming Logs for install RPC

Real-Time Feedback: Enables progress tracking and immediate error display in UI by streaming log output during installation. Currently it provides logs only upon completion.

List already installed plugins on the node

UX Improvement: List installed plugins with enabled/disabled state and version numbers (if possible). It is helpful to improve plugin management like enable, disable, update, etc.

List Available Plugins from Added Sources

UX Improvement: Ability to fetch a list of available plugins from registered reckless sources to improve discoverability.

Clear Old Reckless Configurations

Safety Feature: A method to reset/clear old reckless configurations and .sources in case of issues.

List Reckless Options in lightning-cli listconfigs

Enhancement: It would be great (not required) if reckless options (like reckless-dir, reckless-lightning, reckless-conf, etc.) could be included in the lightning-cli listconfigs response for better visibility.

[ShahanaFarooqui]

While testing the newest reckless UV changes, I noticed a few things. Listing them below:

1:
lightning-cli reckless --version works fine but reckless --version throws error after the version number config file not found: <home>/.lightning/bitcoin/config. The version command should not require a config file.

2:
reckless -d <home>/l1 -l <home>/l1 --network=regtest -c <home>/l1/config --version shows messages that config file not found: <home>/l1/regtest-reckless.conf and config file not found: <home>/l1/.sources but doesn't inform the user once created.

3:
Default config file location for CLN is --lightning/config but reckless's default location is set to --lightning/--network/config. So reckless -l <home>/l1 --network=regtest --version creates the config file in <home>/l1/regtest/config even when CLN config already exists at <home>/l1/config.

4:
lightning-cli reckless source add throws error the reckless process has crashed rather than missing param error.

5:
lightning-cli reckless source add "abc" throwsWARNING: failed to add source abc. The error message can be clearer.

6:
reckless -l <home>/l1 --network=regtest source add "https://github.com/Ride-The-Lightning/c-lightning-REST.git" should either fail to add or remove the extension before adding it into the list. Consequent search for c-lightning-rest will fail because of .git extension.

7:
I am unable to install backup OR datastore plugins:

• lightning-cli reckless install backup and lightning-cli reckless -k "command"="install" "target/subcommand"='["backup"]' throw reckless returned invalid character in json output
• lightning-cli reckless install "datastore" throws:

"log": [
  "DEBUG: Searching for datastore",
  "DEBUG: fetching from gh API: https://api.github.com/repos/lightningd/plugins/contents/",
  "DEBUG: fetching from gh API: https://api.github.com/repos/lightningd/plugins/git/trees/14583768ff9631cca8d06d062c1fdf36ec5770f3",
  "DEBUG: InstInfo(datastore, https://github.com/lightningd/plugins, None, None, None, datastore), Source.GITHUB_REPO",
  "INFO: found datastore in source: https://github.com/lightningd/plugins",
  "DEBUG: entry: None",
  "DEBUG: sub-directory: datastore",
  "DEBUG: Retrieving datastore from https://github.com/lightningd/plugins",
  "DEBUG: Install requested from InstInfo(datastore, https://github.com/lightningd/plugins, None, None, None, datastore).",
  "INFO: cloning Source.GITHUB_REPO InstInfo(datastore, https://github.com/lightningd/plugins, None, None, None, datastore)",
  "DEBUG: missing dependency for InstInfo(datastore, /tmp/reckless-0243461488pi33wdk/clone, None, None, None, None)",
  "DEBUG: cloned_src: InstInfo(datastore, /tmp/reckless-0243461488pi33wdk/clone, None, datastore.py, requirements.txt, datastore/datastore)",
  "DEBUG: using latest commit of default branch",
  "DEBUG: checked out HEAD: aa77a053fb95eb1394157b8aeecd86f9b4e5e1d0",
  "DEBUG: using installer python3venv",
  "DEBUG: creating <home>/l1/reckless/datastore",
  "DEBUG: creating <home>/l1/reckless/datastore/source",
  "DEBUG: copying /tmp/reckless-0243461488pi33wdk/clone/datastore/datastore tree to <home>/l1/reckless/datastore/source/datastore",
  "DEBUG: linking source <home>/l1/reckless/datastore/source/datastore/datastore.py to <home>/l1/reckless/datastore/datastore.py",
  "DEBUG: InstInfo(datastore, <home>/l1/reckless/datastore, None, datastore.py, requirements.txt, source/datastore)",
  "DEBUG: configuring a python virtual environment (pip) in <home>/l1/reckless/datastore/.venv",
  "DEBUG: virtual environment created in <home>/l1/reckless/datastore/.venv.",
  "INFO: dependencies installed successfully",
  "DEBUG: virtual environment for cloned plugin: .venv",
  "INFO: plugin installed: <home>/l1/reckless/datastore",
  "DEBUG: activating datastore",
  "ERROR: reckless: datastore failed to start!",
  "ERROR: CLIError(-3 <home>/l1/reckless/datastore/datastore.py: disabled itself at init: there is a real datastore command)",
  "ERROR: dynamic activation failed: datastore not found in reckless directory"
]

8:
Failed installation still leaves the orphaned plugin folder in <home>/l1/reckless which:

• On install throws error ERROR: File exists: <home>/l1/reckless/<plugin>/source/<plugin>.
• On uninstall throws error ERROR: plugin entrypoint not found in <home>/l1/reckless/<plugin>.

[ShahanaFarooqui]

9:
lightning-cli reckless update throws error missing required parameter: target/subcommand rather than updating all plugins.

[endothermicdev]

Great feedback, thanks @ShahanaFarooqui! I think most of these are addressable; I'll see what I can do. Some of these raise UX issues that I haven't figured out how to deal with yet. I see how cleaning up the failed installation is better for the UI application. I haven't figured out a good way to install plugins like backup though - where they require configuration or external setup steps before you can test and activate them. I guess we could have another flag to skip testing and activation? It just seemed to me that could raise more issues - especially if using reckless via rpc could leave your node in a broken state. Would like to hear your thoughts.

[ShahanaFarooqui]

I haven't figured out a good way to install plugins like backup though - where they require configuration or external setup steps before you can test and activate them

IMHO we have two potential approaches for managing plugins that require configuration:
1: Extend the install RPC – Add a new parameter to pass configuration details during installation, enabling pre-activation setup.
2: Initial UI Filtering – If only a few plugins need configuration, we can exclude them in the UI for initial release, allowing more time to refine the implementation.