Deployment on custom IP, problem with worker connection

[MrStrzelec]

Hi there!
First of all I'd like to say that I really appreciate your software.
But I'm having trouble with a worker.

Basics:
We don't use any kind of huggingface/docker, so it only works on localhost.
We've run everything as u said in the tutorial, but we're not able to run it correctly.

So we're running server via systemd - python manage.py runserver .
We'll also run worker as systemd - celery -A server worker --loglevel=info -P gevent --concurrency 1 -E
(to run the mercury server no matter is server is restarted)

And yet we're still waiting for Worker.

Do you have any idea what's going on?
Thank you very much!

[pplonski]

Hi @MrStrzelec,

Have you tried to start with mercury run? Additionally you can start server in verbose mode mercury run --verbose.

If you would like to run each part of architecture by your own I would recommend checking docker entrypoint script: https://github.com/mljar/mercury/blob/main/docker/mercury/entrypoint.sh

In your case, looks like you are missing: -Q celery,ws, please try to run celery -A server worker --loglevel=info -P gevent --concurrency 1 -E -Q celery,ws

Please let me know if it works for you.

[MrStrzelec]

Hey!

Server working.
I've managed also to change celery script and adjust to your advice but not working. I've got this error from celery (systemd).

Okt 30 09:22:45 Servername celery[1155]: return request("get", url, params=params, **kwargs)
Okt 30 09:22:45 Servername celery[1155]: File "/opt/jupyterhub/lib/python3.10/site-packages/requests/api.py", line 59, in request
Okt 30 09:22:45 Servername celery[1155]: return session.request(method=method, url=url, **kwargs)
Okt 30 09:22:45 Servername celery[1155]: File "/opt/jupyterhub/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
Okt 30 09:22:45 Servername celery[1155]: resp = self.send(prep, **send_kwargs)
Okt 30 09:22:45 Servername celery[1155]: File "/opt/jupyterhub/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
Okt 30 09:22:45 Servername celery[1155]: r = adapter.send(request, **kwargs)
Okt 30 09:22:45 Servername celery[1155]: File "/opt/jupyterhub/lib/python3.10/site-packages/requests/adapters.py", line 519, in send
Okt 30 09:22:45 Servername celery[1155]: raise ConnectionError(e, request=request)
Okt 30 09:22:45 Servername celery[1155]: requests.exceptions.ConnectionError: HTTPConnectionPool(host=Servername', port=5668): Max retries exceeded with url: /api/v1/worker/18db8815-183d-4f84-929b-40522256a3e2/387/12/nb (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb144005870>:Failed to establish a new connection: [Errno 111] Connection refused'))

[pplonski]

I have no idea right now what might be the issue. Is it running on 127.0.0.1?

[MrStrzelec]

It's running on a diffrent IP address. I thought i add this information.

python manage.py runserver IPADDRESS:PORT

[pplonski]

Could you please add --verbose argument to server start command? Let's check if worker is connecting to the correct IP address.

[MrStrzelec]

user@server:/opt/jupyterhub/lib/python3.10/site-packages/mercury$ python manage.py runsever --verbose IP:PORT
Traceback (most recent call last):

  File "/usr/lib/python3.10/logging/config.py", line 565, in configure
    handler = self.configure_handler(handlers[name])
  File "/usr/lib/python3.10/logging/config.py", line 746, in configure_handler
    result = factory(**kwargs)
  File "/usr/lib/python3.10/logging/__init__.py", line 1169, in __init__
    StreamHandler.__init__(self, self._open())
  File "/usr/lib/python3.10/logging/__init__.py", line 1201, in _open
    return open_func(self.baseFilename, self.mode,
PermissionError: [Errno 13] Permission denied: '/opt/jupyterhub/lib/python3.10/site-packages/mercury/django-errors.log'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):

 File "/opt/jupyterhub/lib/python3.10/site-packages/mercury/manage.py", line 22, in <module>
    main()
  File "/opt/jupyterhub/lib/python3.10/site-packages/mercury/manage.py", line 18, in main
    execute_from_command_line(sys.argv)
  File "/opt/jupyterhub/lib/python3.10/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/opt/jupyterhub/lib/python3.10/site-packages/django/core/management/__init__.py", line 416, in execute
    django.setup()
  File "/opt/jupyterhub/lib/python3.10/site-packages/django/__init__.py", line 19, in setup
    configure_logging(settings.LOGGING_CONFIG, settings.LOGGING)
  File "/opt/jupyterhub/lib/python3.10/site-packages/django/utils/log.py", line 76, in configure_logging
    logging_config_func(logging_settings)
  File "/usr/lib/python3.10/logging/config.py", line 811, in dictConfig
    dictConfigClass(config).configure()
  File "/usr/lib/python3.10/logging/config.py", line 572, in configure
    raise ValueError('Unable to configure handler '
ValueError: Unable to configure handler 'file'

It tried to run also mercury run --verbose IP:PORT and output is similar

[pplonski]

Looks like process that is running server doesnt have permission to write logs to the directory. This might be connected to #384 I will need some time to make it configurable.

Have you tried to deploy with docker-compose?

Have you already developed the notebook with mercury and deployment is the last step?

[MrStrzelec]

PermissionError: [Errno 13] Permission denied: '/opt/jupyterhub/lib/python3.10/site-packages/mercury/django-errors.log'

I've just changed permissions to this file for everyone and now it's works. Server running correctly and worker working correctly. W/O docker or huggingface. For some reasons Mercury showing that he cannot find this file, while JupyterNotebook working correctly. U know why?

[pplonski]

Great that it is working!

Do you have data file in the same directory as notebook? It should read file if it is in the same directory.

Alternative might be to provide full path to the file.

[MrStrzelec]

Hey!

Sorry i had a day off yesterday.

Yep, data is in the same directory as a notebook. I also tried with full path and also didn't worked.

Any other idea?
Thanks a Ton :)

[pplonski]

This is strange, could you provide full error message? Maybe try to print out the working directory before loading files, just to check the path. Try to add code:

import os
print(os.getcwd())

[MrStrzelec]

I think worker was not working at all, i've checked logs and found this:

[2023-11-02 12:33:20,930: INFO/MainProcess] Task apps.ws.tasks.task_start_websocket_worker[1cfbef57-5ef1-4ea6-b307-f1380f3a20aa] received
[2023-11-02 12:33:20,976: INFO/MainProcess] Task apps.ws.tasks.task_start_websocket_worker[1cfbef57-5ef1-4ea6-b307-f1380f3a20aa] succeeded in 0.04001394798979163s: None
NB 2023-11-02 12:33:22,661 Exception when notebook load, quit
Traceback (most recent call last):
  File "/opt/jupyterhub/lib/python3.10/site-packages/mercury/apps/../apps/nbworker/rest.py", line 36, in load_notebook
    raise Exception("Cant load notebook")
Exception: Cant load notebook

I think worker wouldn't find this file due to having problem with loading notebook.
I've check everything twice and didn't find anything.

[pplonski]

Do you have notebook file in the same directory in which you start worker?

[keyvan-najafy]

Hi
I am having the same problem

I have created a private fork without any changes and deployed for http via docker-compose on my local computer with default setting and changed nothing but I am getting similar errors to this #391 (comment)

I have used mercury-deploy-demo repo for notebooks and everything seems to be working. admin panel, welcome.md , and notebook icons in welcome page are getting displayed properly but when I click on any of notebooks I get following errors in terminal and I am redirected to notebooks page and notebook is visible in the background but page stays in loading state forever

what should I do ?

mercuryprivate-mercury-1  |   NB 2023-11-23 22:43:06,960 Exception when notebook load, quit
mercuryprivate-mercury-1  |   Traceback (most recent call last):
mercuryprivate-mercury-1  |   File "/opt/conda/lib/python3.10/site-packages/urllib3/connection.py", line 174, in _new_conn
mercuryprivate-mercury-1  |     conn = connection.create_connection(
mercuryprivate-mercury-1  |   File "/opt/conda/lib/python3.10/site-packages/urllib3/util/connection.py", line 95, in create_connection
mercuryprivate-mercury-1  |     raise err
mercuryprivate-mercury-1  |   File "/opt/conda/lib/python3.10/site-packages/urllib3/util/connection.py", line 85, in create_connection
mercuryprivate-mercury-1  |     sock.connect(sa)
mercuryprivate-mercury-1  | ConnectionRefusedError: [Errno 111] Connection refused
mercuryprivate-mercury-1  |
mercuryprivate-mercury-1  | During handling of the above exception, another exception occurred:
mercuryprivate-mercury-1  |
mercuryprivate-mercury-1  | Traceback (most recent call last):
mercuryprivate-mercury-1  |   File "/opt/conda/lib/python3.10/site-packages/urllib3/connectionpool.py", line 715, in urlopen
mercuryprivate-mercury-1  |     httplib_response = self._make_request(
mercuryprivate-mercury-1  |   File "/opt/conda/lib/python3.10/site-packages/urllib3/connectionpool.py", line 416, in _make_request
mercuryprivate-mercury-1  |     conn.request(method, url, **httplib_request_kw)
mercuryprivate-mercury-1  |   File "/opt/conda/lib/python3.10/site-packages/urllib3/connection.py", line 244, in request
mercuryprivate-mercury-1  |     super(HTTPConnection, self).request(method, url, body=body, headers=headers)
mercuryprivate-mercury-1  |   File "/opt/conda/lib/python3.10/http/client.py", line 1283, in request
mercuryprivate-mercury-1  |     self._send_request(method, url, body, headers, encode_chunked)
mercuryprivate-mercury-1  |   File "/opt/conda/lib/python3.10/http/client.py", line 1329, in _send_request
mercuryprivate-mercury-1  |     self.endheaders(body, encode_chunked=encode_chunked)
mercuryprivate-mercury-1  |   File "/opt/conda/lib/python3.10/http/client.py", line 1278, in endheaders
mercuryprivate-mercury-1  |     self._send_output(message_body, encode_chunked=encode_chunked)
mercuryprivate-mercury-1  |   File "/opt/conda/lib/python3.10/http/client.py", line 1038, in _send_output
mercuryprivate-mercury-1  |     self.send(msg)
mercuryprivate-mercury-1  |   File "/opt/conda/lib/python3.10/http/client.py", line 976, in send
mercuryprivate-mercury-1  |     self.connect()
mercuryprivate-mercury-1  |   File "/opt/conda/lib/python3.10/site-packages/urllib3/connection.py", line 205, in connect
mercuryprivate-mercury-1  |     conn = self._new_conn()
mercuryprivate-mercury-1  |   File "/opt/conda/lib/python3.10/site-packages/urllib3/connection.py", line 186, in _new_conn
mercuryprivate-mercury-1  |     raise NewConnectionError(
mercuryprivate-mercury-1  | urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f16bc30e3e0>: Failed to establish a new connection: [Errno 111] Connection refused
mercuryprivate-mercury-1  |
mercuryprivate-mercury-1  | During handling of the above exception, another exception occurred:
mercuryprivate-mercury-1  |
mercuryprivate-mercury-1  | Traceback (most recent call last):
mercuryprivate-mercury-1  |   File "/opt/conda/lib/python3.10/site-packages/requests/adapters.py", line 486, in send
mercuryprivate-mercury-1  |     resp = conn.urlopen(
mercuryprivate-mercury-1  |   File "/opt/conda/lib/python3.10/site-packages/urllib3/connectionpool.py", line 799, in urlopen
mercuryprivate-mercury-1  |     retries = retries.increment(
mercuryprivate-mercury-1  |   File "/opt/conda/lib/python3.10/site-packages/urllib3/util/retry.py", line 592, in increment
mercuryprivate-mercury-1  |     raise MaxRetryError(_pool, url, error or ResponseError(cause))
mercuryprivate-mercury-1  | urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=80): Max retries exceeded with url: /api/v1/worker/dd83117b-dcd9-4c54-8782-aa60c094f6ac/16/1/nb (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f16bc30e3e0>: Failed to establish a new connection: [Errno 111] Connection refused'))
mercuryprivate-mercury-1  |
mercuryprivate-mercury-1  | During handling of the above exception, another exception occurred:
mercuryprivate-mercury-1  |
mercuryprivate-mercury-1  | Traceback (most recent call last):
mercuryprivate-mercury-1  |   File "/app/mercury/apps/../apps/nbworker/rest.py", line 32, in load_notebook
mercuryprivate-mercury-1  |     response = requests.get(
mercuryprivate-mercury-1  |   File "/opt/conda/lib/python3.10/site-packages/requests/api.py", line 73, in get
mercuryprivate-mercury-1  |     return request("get", url, params=params, **kwargs)
mercuryprivate-mercury-1  |   File "/opt/conda/lib/python3.10/site-packages/requests/api.py", line 59, in request
mercuryprivate-mercury-1  |     return session.request(method=method, url=url, **kwargs)
mercuryprivate-mercury-1  |   File "/opt/conda/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
mercuryprivate-mercury-1  |     resp = self.send(prep, **send_kwargs)
mercuryprivate-mercury-1  |   File "/opt/conda/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
mercuryprivate-mercury-1  |     r = adapter.send(request, **kwargs)
mercuryprivate-mercury-1  |   File "/opt/conda/lib/python3.10/site-packages/requests/adapters.py", line 519, in send
mercuryprivate-mercury-1  |     raise ConnectionError(e, request=request)
mercuryprivate-mercury-1  | requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=80): Max retries exceeded with url: /api/v1/worker/dd83117b-dcd9-4c54-8782-aa60c094f6ac/16/1/nb (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f16bc30e3e0>: Failed to establish a new connection: [Errno 111] Connection refused'))