Unify usage of multiple callbacks

[Borda]

🚀 Feature

Simplified API, with callbacks... as e.g. Keras did, pass just list of callbacks to be executed and Trainer will call then when needed instead of having them specified
https://github.com/PyTorchLightning/pytorch-lightning/blob/b1040523b2180300574d961444b00abfa3c84195/pytorch_lightning/trainer/trainer.py#L65-L66

mentioned also in #825 (comment)

[Borda]

@ethanwharris lest split the discussion and talk about callbacks here...

[hadim]

I don't plan to incorporate checkpoint_callback and early_stop_callback callbacks in the new system to avoid making this #889 too big but that should be done in a future PR.

Same for the loggers logic, they should use the callback system IMO (all the custom callbacks I use are loggers...).

[Borda]

Same for the loggers logic, they should use the callback system IMO (all the custom callbacks I use are loggers...).

interesting idea, and true that logger is a special case of a callback, just not sure if I won't be too confusing for a user...

[hadim]

For the user, it's easy to make this change invisible. Just make the Logger class inherit from Callback and modify the appropriate loggers to use the new mechanism instead of the custom logger's functions. Everything should be backward compatible.

[Borda]

sure making the parents of loggers does not disturb any user, I was more-less thinking about passing callbacks and lobber in the same argument (list) instead of having an argument for callbacks and loggers separately

[hadim]

But loggers and callbacks are the same things so I don't see any issue and new users should no be disturbed IMO.

About existing users and backward compatibilities, if the documentation is clear about it, everything should be fine.

[ethanwharris]

I'm not convinced that there is an advantage to making Loggers into Callbacks. Currently there is a lot of inter-dependency between checkpointing and logging (the checkpointer directory depends on logger name / version for example), I can see that getting quite messy if everything is of the same class and invoked with the same command. There's also the problem of callback hell, where everything can become a callback if the API is flexible enough, which we should probably avoid. That said, I'm not against the idea if it doesn't disturb the user, just not sure what value it would bring in that case.

[Borda]

@PyTorchLightning/core-contributors what do you think? is it reasonable/beneficial? 🤖

[jeremyjordan]

my two cents:

sure making the parents of loggers does not disturb any user, I was more-less thinking about passing callbacks and lobber in the same argument (list) instead of having an argument for callbacks and loggers separately

I think it will be more user friendly to have the trainer keep these separate (eg. Trainer(..., logger=True, callbacks=[]))

There's also the problem of callback hell, where everything can become a callback if the API is flexible enough, which we should probably avoid.

+1

---

as I mentioned in #595, I think we should allow user defined callbacks passed in as a list and execute the callbacks in the order which they're provided in the list.

Trainer(..., callbacks=[custom_callback_1, custom_callback_2, model_checkpoint_callback])

one thing that we'll want to consider is what information we'll expose to the callbacks. do we want to expose all model attributes?

[williamFalcon]

@Borda i'm not sure i love this idea. this means you wouldn't get early stopping for free or checkpointing for free. most of the time users don't care about the details of those and a simple True is enough. This means now you'd have to specify all the details.

i think this clutters the API and makes it unnecessarily complex. We moved the callbacks to default arguments after a ton of feedback on people being annoyed they had to init them EVERY time.

and frankly, the keras callbacks are super unintuitive and confusing.

So, unless there's a clear API win, i'm going to push back on this.