Detect POSIX-style paths on Windows

[cr1901]

I have a git binary installed on Windows (the MSYS2 variant, specifically) that does not understand Windows-style paths. When trying to install software using setuptools_scm, this leads to cryptic errors such as the following:

William@DESKTOP-H0PMN4M MINGW64 ~/Projects/FPGA/nmigen
$ python setup.py develop
Traceback (most recent call last):
  File "setup.py", line 10, in <module>
    setup(
  File "C:/msys64/mingw64/lib/python3.8/site-packages/setuptools/__init__.py", line 145, in setup
    return distutils.core.setup(**attrs)
  File "C:/msys64/mingw64/lib/python3.8/distutils/core.py", line 108, in setup
    _setup_distribution = dist = klass(attrs)
  File "C:/msys64/mingw64/lib/python3.8/site-packages/setuptools/dist.py", line 447, in __init__
    _Distribution.__init__(self, {
  File "C:/msys64/mingw64/lib/python3.8/distutils/dist.py", line 292, in __init__
    self.finalize_options()
  File "C:/msys64/mingw64/lib/python3.8/site-packages/setuptools/dist.py", line 740, in finalize_options
    ep.load()(self)
  File "C:/msys64/mingw64/lib/python3.8/site-packages/setuptools/dist.py", line 747, in _finalize_setup_keywords
    ep.load()(self, ep.name, value)
  File "c:/msys64/home/william/projects/fpga/nmigen/.eggs/setuptools_scm-3.5.0-py3.8.egg/setuptools_scm/integration.py", line 17, in version_keyword
    dist.metadata.version = _get_version(config)
  File "c:/msys64/home/william/projects/fpga/nmigen/.eggs/setuptools_scm-3.5.0-py3.8.egg/setuptools_scm/__init__.py", line 147, in _get_version
    parsed_version = _do_parse(config)
  File "c:/msys64/home/william/projects/fpga/nmigen/.eggs/setuptools_scm-3.5.0-py3.8.egg/setuptools_scm/__init__.py", line 103, in _do_parse
    version = _version_from_entrypoints(config) or _version_from_entrypoints(
  File "c:/msys64/home/william/projects/fpga/nmigen/.eggs/setuptools_scm-3.5.0-py3.8.egg/setuptools_scm/__init__.py", line 63, in _version_from_entrypoints
    version = _call_entrypoint_fn(root, config, ep.load())
  File "c:/msys64/home/william/projects/fpga/nmigen/.eggs/setuptools_scm-3.5.0-py3.8.egg/setuptools_scm/__init__.py", line 44, in _call_entrypoint_fn
    return fn(root, config=config)
  File "c:/msys64/home/william/projects/fpga/nmigen/.eggs/setuptools_scm-3.5.0-py3.8.egg/setuptools_scm/git.py", line 98, in parse
    wd = GitWorkdir.from_potential_worktree(config.absolute_root)
  File "c:/msys64/home/william/projects/fpga/nmigen/.eggs/setuptools_scm-3.5.0-py3.8.egg/setuptools_scm/git.py", line 33, in from_potential_worktree
    if not samefile(real_wd, wd):
  File "C:\msys64\mingw64/lib/python3.8/genericpath.py", line 100, in samefile
    s1 = os.stat(f1)
FileNotFoundError: [WinError 3] The system cannot find the path specified: '/home/William/Projects/FPGA/nmigen'

The error comes from the path git returns from the rev-parse command:

William@DESKTOP-H0PMN4M MINGW64 ~/Projects/FPGA/nmigen
$ git rev-parse --show-toplevel
/home/William/Projects/FPGA/nmigen

One potential solution is to pipe the output of git rev-parse through cygpath is we detect that the git being run is MSYS2 (or otherwise doesn't understand Windows-style paths), as unfortunately conversion between POSIX and Windows path is an involved problem:

William@DESKTOP-H0PMN4M MINGW64 ~/Projects/FPGA/nmigen
$ git rev-parse --show-toplevel | cygpath -w -f -
C:\msys64\home\William\Projects\FPGA\nmigen

The following workaround allows me to successfully run the egg-info, develop, and install successfully (with warning: no previously-included files found matching '.travis.yaml'). Obviously this patch isn't acceptable as-is, and this may not be the correct repo to bring this issue up, depending on the scope of supported git variants. However, I figure it's worth a shot.

William@DESKTOP-H0PMN4M MINGW64 ~/Projects/py-packages/setuptools_scm
$ git diff
diff --git a/src/setuptools_scm/file_finder_git.py b/src/setuptools_scm/file_finder_git.py
index 8b81d2c..f68c91a 100644
--- a/src/setuptools_scm/file_finder_git.py
+++ b/src/setuptools_scm/file_finder_git.py
@@ -17,6 +17,13 @@ def _git_toplevel(path):
                 universal_newlines=True,
                 stderr=devnull,
             )
+
+            out = subprocess.check_output(
+                ["cygpath", "-w", out],
+                cwd=(path or "."),
+                universal_newlines=True,
+                stderr=devnull,
+            )
         trace("find files toplevel", out)
         return os.path.normcase(os.path.realpath(out.strip()))
     except subprocess.CalledProcessError:
diff --git a/src/setuptools_scm/git.py b/src/setuptools_scm/git.py
index afefa34..43cc6c5 100644
--- a/src/setuptools_scm/git.py
+++ b/src/setuptools_scm/git.py
@@ -27,6 +27,9 @@ class GitWorkdir(object):
     @classmethod
     def from_potential_worktree(cls, wd):
         real_wd, _, ret = do_ex("git rev-parse --show-toplevel", wd)
+        if ret:
+            return
+        real_wd, _, ret = do_ex("cygpath -w {}".format(real_wd), wd)
         if ret:
             return
         trace("real root", real_wd)

William@DESKTOP-H0PMN4M MINGW64 ~/Projects/py-packages/setuptools_scm
$

[RonnyPfannschmidt]

at first glance this will require quite a number of back and forth path transformations,

its not something that i can hope to build and test (i have neither the env nor the msys expertise, plus i use linux)

you can use the SETUPTOOLS_SCM_PRETEND_VERSION env var to work around git not providing data for the windows env, but i would most strongly suggest to use a windows git on the windows side and a msys git on the msys side

[cr1901]

@RonnyPfannschmidt

i would most strongly suggest to use a windows git on the windows side and a msys git on the msys side

What about an msys git with an msys python (which also has the path mismatch problem)? Is that something you'd consider supporting? If you don't support this configuration, my workaround is probably better suited as a patch for MINGW-packages, and that you only explicitly support a windows git with a windows python.

[RonnyPfannschmidt]

im not sure whats the best approach there

if msys python uses windows paths instead of msys paths that sounds like a bug in msys python

the problem with supporting the configuration you mention is that i do not use windows or msys at all
and i have no idea how to set up a correct reliable CI for testing them either

so right now msys is not a system/setup i can support by myself, it needs help by someone using those systems and knowing them

[wjblanke]

@cr1901 we are trying to build wheels using msys2. We are seeing the exact same path issue you report. I've forked setuptools_scm and applied your patch. How did you replace setuptools_scm from msys2 with the forked version? Thanks!

[cr1901]

@wjblanke According to pip list:

setuptools-scm      0.1.dev814+g69eb0fa.d20200329 c:/msys64/home/william/projects/py-packages/setuptools_scm/src

Meaning I did the following from my setuptools_scm checkout:

William@DESKTOP-H0PMN4M MINGW64 ~/Projects/py-packages/setuptools_scm
$ python setup.py develop
running develop
running egg_info
writing src/setuptools_scm.egg-info/PKG-INFO
writing dependency_links to src/setuptools_scm.egg-info/dependency_links.txt
writing entry points to src/setuptools_scm.egg-info/entry_points.txt
writing requirements to src/setuptools_scm.egg-info/requires.txt
writing top-level names to src/setuptools_scm.egg-info/top_level.txt
reading manifest template 'MANIFEST.in'
warning: no previously-included files found matching '.travis.yaml'
writing manifest file 'src/setuptools_scm.egg-info/SOURCES.txt'
running build_ext
Creating c:/msys64/mingw64/lib/python3.8/site-packages/setuptools-scm.egg-link (link to src)
setuptools-scm 0.1.dev814+g69eb0fa.d20200401 is already the active version in easy-install.pth

Installed c:/msys64/home/william/projects/py-packages/setuptools_scm/src
Processing dependencies for setuptools-scm==0.1.dev814+g69eb0fa.d20200401
Finished processing dependencies for setuptools-scm==0.1.dev814+g69eb0fa.d20200401

William@DESKTOP-H0PMN4M MINGW64 ~/Projects/py-packages/setuptools_scm
$

[wjblanke]

Thanks @cr1901! We were able to get it to run in the toml with

requires = ["setuptools_scm@git+https://github.com/Chia-Network/setuptools_scm.git@master#egg=setuptools_scm", "setuptools>=42", "wheel", "setuptools_scm[toml]>=3.5.0"]

Unfortunately it doesn't look like pypi accepts mingw wheels.

HTTPError: 400 Client Error: Binary wheel 'chiapos-0.0.0-cp38-cp38-mingw.whl' has an unsupported platform tag 'mingw'. for url: https://test.pypi.org/legacy/

[cr1901]

@wjblanke I'm guessing "mingw" is explicitly for packages that are part of the MSYS2 Project, and you otherwise have to use any tag:

$ python -c "import wheel.pep425tags as w; print(w.get_supported())"
[('cp38', 'cp38', 'mingw'), ('cp38', 'none', 'mingw'), ('cp38', 'none', 'any'), ('cp3', 'none', 'any'), ('cp37', 'none', 'any'), ('cp36', 'none', 'any'), ('cp35', 'none', 'any'), ('cp34', 'none', 'any'), ('cp33', 'none', 'any'), ('cp32', 'none', 'any'), ('cp31', 'none', 'any'), ('cp30', 'none', 'any'), ('py3', 'none', 'mingw'), ('py38', 'none', 'any'), ('py3', 'none', 'any'), ('py37', 'none', 'any'), ('py36', 'none', 'any'), ('py35', 'none', 'any'), ('py34', 'none', 'any'), ('py33', 'none', 'any'), ('py32', 'none', 'any'), ('py31', 'none', 'any'), ('py30', 'none', 'any')]

[RonnyPfannschmidt]

personally i wont work on this as i dont have the system needed, im happy to review pr's with tests

i wonder if a gitlab ci build could be added with such a environment to ensure its not breaking

until someone comes up with a working co it cant be declared a supported feature

[cr1901]

@RonnyPfannschmidt Sorry, I wasn't expecting you to fix this, especially if you don't have the requisite system.

I've been trying to tie up a bunch of loose ends, and I haven't gotten around to adding a fix to this. Calling a binary to do parsing is a horrible approach. In the past I've used ctypes to load cygwin1.dll, but I also consider that unacceptable for a widely-used library.

im happy to review pr's with tests

I will take a look, though it may be a few days. You have a utils.py module- I'm sure I can stick something there.

working co

Working checkout?

[RonnyPfannschmidt]

Whops, I meant ci for continuous integration

[xiaozhongtian]

Same issue!

in MSYS2:

when i install dependencies by pip install -r requirement.txt. it works well.

pip install -e '.[dev]' by setup.py it crashed with error:

 s1 = os.stat(f1)
    FileNotFoundError: [WinError 3] Le chemin d▒acc▒s sp▒cifi▒ est introuvable: '/c/Users/XIAOZHO/digibiz/DE/digibiz_data_validation'