[BUG] long_callback fails when bundling python code

[JonThom]

Describe your context
Please provide us your environment, so we can easily reproduce the issue.

• replace the result of pip list | grep dash below

dash                                2.0.0         A Python framework for building reactive web-apps. Developed b...
dash-bootstrap-components           1.0.2         Bootstrap themed components for use in Plotly Dash
dash-core-components                2.0.0         Core component suite for Dash
dash-table                          5.0.0         Dash table

Describe the bug

When using Pyinstaller to bundle a Dash app using long_callback as a single MacOS .app file, the app fails to register the long_callback.
The error occurs in

dash/dash/long_callback/managers/__init__.py

Line 38 in 78ca3ec

fn_source = inspect.getsource(fn)

with the call to inspect.getsource, which throws an OSError, stating that the source code cannot be retrieved.
In this case, the long_callback calls external libraries used to query a database.
The error only occurs when packaging with Pyinstaller.
Although the Pyinstaller packaging is probably a fairly rare use case, I was wondering why access to source is needed, and of course whether there might be an obvious workaround.

Expected behavior

Expect the long_callback to register correctly.

[Simon-12]

I run into the same problem under Windows 10.
Any updates on the issue here?

[JonThom]

@Simon-12 a workaround with pyinstaller is to write the long callbacks in a separate file and include it using 'datas'.
Downside is that this exposes the long callbacks source code to the user.
Let me know if you find any other way.

[Simon-12]

@JonThom thanks for the fast response.
I think i know what you mean. But, what you mean with in seperate file?
The whole callback function in a seperate .py file as a function?
What is with the path to the function?

I will test during the next week and will write a feedback here.

[JonThom]

@Simon-12

Here is a sketch of the suggested approach:
Here we create a filelong_callbacks.py in the root project folder (for simplicity; you probably want to put it somewhere else) with the following content:

def register_long_callbacks(app):
   @app.long_callback(
        Output("id1","prop1"),
        Input("id2","prop2"),
    )
    def lc1(input_1):
        ...
   

    @app.long_callback() ..

Then in the main app file, main_app_file.py, import that function and call it, passing the Dash app object

from .long_callbacks import register_long_callbacks

my_app = Dash()
register_long_callbacks(my_app)

...

Now, in the pyinstaller .spec file, include the long_callbacks.py in the datas argument to Analysis:

a = Analysis([
        'my_app/main_app_file.py'],
        pathex=[],
        binaries=[],
        datas=[
            ('path/to/long_callbacks.py', '.')
        ],

[alexcjohnson]

@JonThom I missed this issue when you first raised it, apologies and thanks for bringing this case to our attention. The problematic line you link to is in building a cache key for this function invocation, I believe the reason we're using the function source for that is so the cache will invalidate if you modify the callback code itself. Clearly not a concern if you've wrapped the app up and shipped it, it's really only for apps being actively developed.

@T4rk1n I bet we can fall back on callback_id when getsource fails - want to give that a shot? Actually... we might even want to always include callback_id in generating that cache key, and only add the source if it's available. It would be weird, but I think you could construct two functions with identical source and identical arguments that nevertheless behave differently - maybe because they're in different modules and call out to different inner functions, or maybe because there's a factory function creating these long callbacks and the meat of the operation is passed in to the outer scope.

[Simon-12]

@JonThom It was possible for me to implement the workaround and the error message disapered.
But when i start the long callback function nothing happens and the app magically restarts. I have no clue ...
Have anyone similar experiences?

[T4rk1n]

@JonThom @Simon-12

Downside is that this exposes the long callbacks source code to the user.

Just a warning, Pyinstaller doesn't obfuscate any part of the code, I can take the executable and get all the original source quite easily. The only way to keep the code secure is keeping it private and deploying the server.

[T4rk1n]

It would be weird, but I think you could construct two functions with identical source and identical arguments that nevertheless behave differently - maybe because they're in different modules and call out to different inner functions, or maybe because there's a factory function creating these long callbacks and the meat of the operation is passed in to the outer scope.

I agree the callback_id should be included in the cache key for callback factories that may share the same function but have different outputs/inputs.

[JonThom]

@JonThom @Simon-12

Downside is that this exposes the long callbacks source code to the user.

Just a warning, Pyinstaller doesn't obfuscate any part of the code, I can take the executable and get all the original source quite easily. The only way to keep the code secure is keeping it private and deploying the server.

Thanks @T4rk1n, I should have made clear that including the source as plain text just makes it even more trivial to access it.
I was wondering, have you tried different approaches to obfuscating python code? Could one hope for an approach where reversing it is at least a very laborious process? I do not have the knowledge to judge a priori whether this should be the case.
I have tried nuitka but overcoming the many errors arising with Dash and PySide required many workarounds and some issues I could not overcome.
I also considered pyarmor, but haven't tried it yet.

[T4rk1n]

I was wondering, have you tried different approaches to obfuscating python code?

No, I don't think obfuscation is a reliable way to protect code. If there is intellectual property that should be protected by licensing. Otherwise private code or secrets used to access databases belongs on servers.

[Simon-12]

I dont want to secure my code, i just want to distribute the app to my friends who are absolutely not familiar with programming.

I wrote a small example, just a simple dash app with a long callback: github.com/Simon-12/simple-dash
If some one want to test, just open a terminal and run inside the folder: pyinstaller specs.spec

When i start the long callback function nothing happens, the app magically restarts and opens a new tab.
If anyone has an idea, feel free to help.

Cheers

[DeKhaos]

I dont want to secure my code, i just want to distribute the app to my friends who are absolutely not familiar with programming.

I wrote a small example, just a simple dash app with a long callback: github.com/Simon-12/simple-dash If some one want to test, just open a terminal and run inside the folder: pyinstaller specs.spec

When i start the long callback function nothing happens, the app magically restarts and opens a new tab. If anyone has an idea, feel free to help.

Cheers

Hi @Simon-12 , have you solve this problem so far? Would be nice of you to share with me the workaround if you figured it out. I'm deploying a desktop dashboard using cx_freeze to convert python to .exe and I'm using lots of background callbacks and multiple pages. Every time I try to activate a background callback with a button, it doesn't do anything then restart the application as well as the console print.

[JonThom]

Hi @DeKhaos
Not sure for cx_freeze, but pyinstaller makes it possible to include 'data' files with the app, and as described above, I included the long callbacks in such a data file when using pyinstaller.

[DeKhaos]

@JonThom , I tried as you suggested, but I got the same result as @Simon-12, the error message doesn't show up but the background callback doesn't work either. Every time I triggered the Run button, the server restarts and nothing happens.

I used example 4 from Background callbacks and separate the callback to another file as you suggested and included to datas of main.spec.

main.py

import time
import os
from long_callbacks import register_long_callbacks
import dash
from dash import DiskcacheManager, Input, Output, html


# Diskcache for non-production apps when developing locally
import diskcache
cache = diskcache.Cache("./cache")
background_callback_manager = DiskcacheManager(cache)

app = dash.Dash(__name__, background_callback_manager=background_callback_manager)
register_long_callbacks(app)
app.layout = html.Div(
    [
        html.Div(
            [
                html.P(id="paragraph_id", children=["Button not clicked"]),
                html.Progress(id="progress_bar", value="0"),
            ]
        ),
        html.Button(id="button_id", children="Run Job!"),
        html.Button(id="cancel_button_id", children="Cancel Running Job!"),
    ]
)


if __name__ == "__main__":
    app.run_server(debug=False)

long_callbacks.py

import time
import os

import dash
from dash import DiskcacheManager, Input, Output, html

def register_long_callbacks(app):
    @app.callback(
        output=Output("paragraph_id", "children"),
        inputs=Input("button_id", "n_clicks"),
        background=True,
        running=[
            (Output("button_id", "disabled"), True, False),
            (Output("cancel_button_id", "disabled"), False, True),
            (
                Output("paragraph_id", "style"),
                {"visibility": "hidden"},
                {"visibility": "visible"},
            ),
            (
                Output("progress_bar", "style"),
                {"visibility": "visible"},
                {"visibility": "hidden"},
            ),
        ],
        cancel=Input("cancel_button_id", "n_clicks"),
        progress=[Output("progress_bar", "value"), Output("progress_bar", "max")],
        prevent_initial_call=True
    )
    def update_progress(set_progress, n_clicks):
        total = 5
        for i in range(total + 1):
            set_progress((str(i), str(total)))
            time.sleep(1)
    
        return f"Clicked {n_clicks} times"

[JonThom]

@DeKhaos

I have taken your code and made a working app using pyinstaller
Instructions in the README.
I used pyinstaller since I have no experience with cz_freeze and imagine the problem and solution is the same.
Hope it is of use.