TextIO somehow causes a TypeError when using `multiprocessing` to spawn a Process.

[ceccopierangiolieugenio]

TextIO somehow causes a TypeError when using multiprocessing to spawn a Process.

TypeError: int() argument must be a string, a bytes-like object or a real number, not 'NoneType'

Just fyi.

Originally posted by @slook in #494 (comment)

[ceccopierangiolieugenio]

@slook
I cannot reproduce the error,
Do you have a simple example or the backtrace?
It could be that I am not reimplementing some required TextIO methods.

[slook]

ERROR:(MainThread) Error in sys.excepthook:
Traceback (most recent call last):
  File "/home/user/Git/slook/nicotine-plus/pynicotine/events.py", line 273, in emit
    raise error

  File "/home/user/Git/slook/nicotine-plus/pynicotine/events.py", line 265, in emit
    function(*args, **kwargs)

  File "/home/user/Git/slook/nicotine-plus/pynicotine/shares.py", line 764, in _quit
    self._scanner_process.terminate()

  File "/usr/lib/python3.11/multiprocessing/process.py", line 133, in terminate
    self._popen.terminate()
    ^^^^^^^^^^^^^^^^^^^^^

AttributeError: 'NoneType' object has no attribute 'terminate'

Original exception was:
Traceback (most recent call last):
  File "/home/user/Git/slook/nicotine-plus/./nicotine", line 24, in <module>
    sys.exit(load_module())
             ^^^^^^^^^^^^^

  File "/home/user/Git/slook/nicotine-plus/./nicotine", line 20, in load_module
    return pynicotine.run()
           ^^^^^^^^^^^^^^^^

  File "/home/user/Git/slook/nicotine-plus/pynicotine/__init__.py", line 233, in run
    exit_code = application.run(ci_mode)
                ^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/user/Git/slook/nicotine-plus/pynicotine/ttktui/__init__.py", line 43, in run
    return Application(ci_mode).run()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/user/Git/slook/nicotine-plus/pynicotine/ttktui/application.py", line 58, in run
    core.start()


  File "/home/user/Git/slook/nicotine-plus/pynicotine/core.py", line 207, in start
    events.emit("start")


  File "/home/user/Git/slook/nicotine-plus/pynicotine/events.py", line 273, in emit
    raise error


  File "/home/user/Git/slook/nicotine-plus/pynicotine/events.py", line 265, in emit
    function(*args, **kwargs)


  File "/home/user/Git/slook/nicotine-plus/pynicotine/shares.py", line 756, in _start
    self.rescan_shares(init=True, rescan=rescan_startup)


  File "/home/user/Git/slook/nicotine-plus/pynicotine/shares.py", line 1112, in rescan_shares
    self._scanner_process.start()


  File "/usr/lib/python3.11/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
                  ^^^^^^^^^^^^^^^^^

  File "/usr/lib/python3.11/multiprocessing/context.py", line 288, in _Popen
    return Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^

  File "/usr/lib/python3.11/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)


  File "/usr/lib/python3.11/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)


  File "/usr/lib/python3.11/multiprocessing/popen_spawn_posix.py", line 40, in _launch
    tracker_fd = resource_tracker.getfd()
                 ^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/lib/python3.11/multiprocessing/resource_tracker.py", line 75, in getfd
    self.ensure_running()


  File "/usr/lib/python3.11/multiprocessing/resource_tracker.py", line 129, in ensure_running
    pid = util.spawnv_passfds(exe, args, fds_to_pass)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/lib/python3.11/multiprocessing/util.py", line 450, in spawnv_passfds
    passfds = tuple(sorted(map(int, passfds)))
                    ^^^^^^^^^^^^^^^^^^^^^^^^^

TypeError: int() argument must be a string, a bytes-like object or a real number, not 'NoneType'

import multiprocessing
context = multiprocessing.get_context(method="spawn")
scanner_obj = Scanner()  # custom class as in a module of pynicotine core
self._scanner_process = context.Process(target=scanner_obj.run, daemon=True)
self._scanner_process.start()  # <-- Line 1112 of pynicotine/shares.py

Process().start()
https://github.com/slook/nicotine-plus/blob/ttk/pynicotine/shares.py#L1112

TTkStderrHandler(TextIO): # (TextIO): # doesn't work with multiprocessing
https://github.com/slook/nicotine-plus/blob/ttk/pynicotine/external/pyTermTk/TermTk/TTkCore/ttk.py#L52

Note: Application() is an instance of Thread() wherein TTk() is initialized, but the offending process isn't joined by the TTk app (the underlying pynicotine program core takes care of it).

In https://docs.python.org/3/library/multiprocessing.html#programming-guidelines it says...
"Beware of replacing sys.stdin with a “file like object”"
... so that probably has something to do with it.

[ceccopierangiolieugenio]

I still cannot reproduce the issue,
I tried to clone your repo but I don't know how to start it.
I'd like to investigate why this issue happen,
I used TextIO because the stderr is an IO type,
Surely there are some methods that need to be implemented to keep the compatibility.
I need to reproduce the issue in order to proceed with the investigation.
I think io.TextIOBase need to be used instead of TextIO in order to avoid this issue.

from:
lib/python3.13/multiprocessing/resource_tracker.py

            fds_to_pass = []
            try:
                fds_to_pass.append(sys.stderr.fileno())
            except Exception:
                pass

I have not fileno() implemented so None is returned,
I should raise an exception in order to not include stderr in the fds_to_pass
or use a proper TextIO subclass.

[slook]

I tried to clone your repo but I don't know how to start it.

./nicotine --tui

It can't really be started properly unless a network account and shared folder has already been setup (by using the default Gtk interface) because most of the needed dialogs don't exist yet. I don't think it is realistic for you to use this for testing purposes in its current state, as its quite a specialized app.

However it is possible to attempt a rescan (click on the "Rescan Shares" menu item) without having an account nor doing any prior configuration (then you should see "Rescan complete: 0 folders found" in the bottom status bar). You don't need to connect in order to test the functions which I have implemented (which isn't very much, as yet).

I have not fileno() implemented so None is returned,

https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods
"In particular, unnecessary file descriptors and handles from the parent process will not be inherited."

It seems like a rather unusual scenario, but there might be other use cases where spawning a process might be desired, hence why I only report to you for your benefit of knowledge. It works fine without TextIO, so I am happy to patch it out in my builds for now.

[ceccopierangiolieugenio]

yes, it looks that I need to extend the TextIOBase

class _TTkStderrHandler(io.TextIOBase)

Anyway, I am still trying to investigate it, K want to make a minimum code that would fail.
Any attempt at making a simple multiprocess for some reason doesn't fail.

[slook]

I just fixed my project so it now runs successfully even with an empty config (before it showed blank screen at startup because there was an input() prompt on stdout asking to choose a new username!) so now you can try testing if helpful for your investigations.

Its only dependency is Python >= 3.9 and the slightly modified TTk library is bundled inside the app (it won't run with main branch of TTk because of the needed threading patch). There is no need to build it or anything because it can be launched like a script.

I don't know exactly what's non-simple about the Scanner() process (I didn't write the program but I think it just a regular "spawn" type - that is the one and only module which uses multiprocessing) but who knows after all it is more than 20 years old and there are many patches in it! (you see it is apparently why the program was given that name like nicotine patches).

Start with ./nicotine --tui and then select the menu "Shares" -> "Rescan Shares" (it should complete to "0 folders" instantly).

Disable TextIO patch (where I had to comment out to make it work):
https://github.com/slook/nicotine-plus/blob/ttk/pynicotine/external/pyTermTk/TermTk/TTkCore/ttk.py#L52

My TTk Text User Interface components (including where the threading happens) are all inside this subfolder:
https://github.com/slook/nicotine-plus/tree/ttk/pynicotine/ttktui

[ceccopierangiolieugenio]

Yes, I managed to run it, and if I use TextIOBase, the crash doesn't happen
thanks