Skip to content

tox native codegen unused variables #1694

Open
@jfennick

Description

@jfennick

Expected Behavior

Using the instructions in the README, developers should be able to run tox locally without errors.

Actual Behavior

During the installation process, some autogenerated native code creates too many unused variables which crashes the entire script.

Workflow Code

conda create --name cwltool_test
conda activate cwltool_test
conda install pip
pip install .
cwltool --version
pip install -rtest-requirements.txt
pip install tox; tox -p

That last command generates the following traceback, and then hangs...

Full Traceback

⠼ [2] py310-unit | py310-mypyERROR: invocation failed (exit code 1), logfile: /Users/jakefennick/cwltool/.tox/py310-mypy/log/py310-mypy-12.log
========================================================================================================================= log start ==========================================================================================================================
py310-mypy recreate: /Users/jakefennick/cwltool/.tox/py310-mypy
py310-mypy installdeps: -rrequirements.txt, -rtest-requirements.txt, -rmypy-requirements.txt
py310-mypy installed: arcp==0.2.1,argcomplete==2.0.0,attrs==21.4.0,bagit==1.8.1,CacheControl==0.12.11,certifi==2022.6.15,charset-normalizer==2.1.0,coloredlogs==15.0.1,coverage==6.4.2,execnet==1.9.0,humanfriendly==10.0,idna==3.3,iniconfig==1.1.1,isodate==0.6.1,lockfile==0.12.2,lxml==4.9.1,mistune==0.8.4,mock==4.0.3,msgpack==1.0.4,mypy==0.961,mypy-extensions==0.4.3,networkx==2.8.4,packaging==21.3,pluggy==1.0.0,prov==1.5.1,psutil==5.9.1,py==1.11.0,pydot==1.4.2,pyparsing==3.0.9,pytest==7.1.2,pytest-cov==3.0.0,pytest-forked==1.4.0,pytest-mock==3.8.2,pytest-xdist==2.5.0,python-dateutil==2.8.2,rdflib==6.1.1,requests==2.28.1,ruamel.yaml==0.17.21,ruamel.yaml.clib==0.2.6,schema-salad==8.3.20220626185350,shellescape==3.8.1,six==1.16.0,tomli==2.0.1,types-mock==4.0.15,types-psutil==5.9.1,types-requests==2.28.1,types-setuptools==62.6.1,types-urllib3==1.26.16,typing_extensions==4.3.0,urllib3==1.26.10
py310-mypy run-test-pre: PYTHONHASHSEED='300611664'
py310-mypy run-test: commands[0] | make mypy mypyc PYTEST_EXTRA=
if ! test -f /Users/jakefennick/cwltool/.tox/py310-mypy/lib/python3.10/site-packages/ruamel/yaml/py.typed ; \
	then \
		rm -Rf typeshed/ruamel/yaml ; \
		ln -s /Users/jakefennick/cwltool/.tox/py310-mypy/lib/python3.10/site-packages/ruamel/yaml \
			typeshed/ruamel/ ; \
	fi  # if minimally required ruamel.yaml version is 0.15.99 or greater, than the above can be removed
MYPYPATH=$MYPYPATH:typeshed mypy cwltool/__init__.py cwltool/__main__.py cwltool/argparser.py cwltool/builder.py cwltool/checker.py cwltool/command_line_tool.py cwltool/context.py cwltool/cuda.py cwltool/cwlrdf.py cwltool/cwlviewer.py cwltool/docker.py cwltool/docker_id.py cwltool/env_to_stdout.py cwltool/errors.py cwltool/executors.py cwltool/expression.py cwltool/factory.py cwltool/flatten.py cwltool/job.py cwltool/load_tool.py cwltool/loghandler.py cwltool/main.py cwltool/mpi.py cwltool/mutation.py cwltool/pack.py cwltool/pathmapper.py cwltool/process.py cwltool/procgenerator.py cwltool/provenance.py cwltool/provenance_constants.py cwltool/provenance_profile.py cwltool/resolver.py cwltool/run_job.py cwltool/sandboxjs.py cwltool/secrets.py cwltool/singularity.py cwltool/singularity_utils.py cwltool/software_requirements.py cwltool/stdfsaccess.py cwltool/subgraph.py cwltool/task_queue.py cwltool/udocker.py cwltool/update.py cwltool/utils.py cwltool/validate_js.py cwltool/workflow.py cwltool/workflow_job.py tests/__init__.py tests/test_anon_types.py tests/test_check.py tests/test_conditionals.py tests/test_content_type.py tests/test_context.py tests/test_cuda.py tests/test_cwl_version.py tests/test_default_path.py tests/test_dependencies.py tests/test_docker.py tests/test_docker_info.py tests/test_docker_paths_with_colons.py tests/test_empty_input.py tests/test_environment.py tests/test_examples.py tests/test_ext.py tests/test_fetch.py tests/test_http_input.py tests/test_input_deps.py tests/test_iwdr.py tests/test_js_sandbox.py tests/test_load_tool.py tests/test_make_template.py tests/test_misc_cli.py tests/test_mpi.py tests/test_override.py tests/test_pack.py tests/test_parallel.py tests/test_path_checks.py tests/test_pathmapper.py tests/test_procgenerator.py tests/test_provenance.py tests/test_rdfprint.py tests/test_recursive_validation.py tests/test_relocate.py tests/test_schemadef.py tests/test_secrets.py tests/test_singularity.py tests/test_singularity_versions.py tests/test_stdout_stderr_log_dir.py tests/test_streaming.py tests/test_subgraph.py tests/test_target.py tests/test_tmpdir.py tests/test_toolargparse.py tests/test_trs.py tests/test_udocker.py tests/test_validate.py tests/test_validate_js.py tests/test_windows_warning.py tests/util.py
Success: no issues found in 99 source files
MYPYPATH=typeshed CWLTOOL_USE_MYPYC=1 pip install --verbose -e . \
		 && pytest -rs -vv 
Using pip 22.1.2 from /Users/jakefennick/cwltool/.tox/py310-mypy/lib/python3.10/site-packages/pip (python 3.10)
Obtaining file:///Users/jakefennick/cwltool
  Preparing metadata (setup.py): started
  Running command python setup.py egg_info
  running egg_info
  creating /private/var/folders/xg/1dp6d5j960v4s6vmv680jylr0000gp/T/pip-pip-egg-info-eysfuzd7/cwltool.egg-info
  writing /private/var/folders/xg/1dp6d5j960v4s6vmv680jylr0000gp/T/pip-pip-egg-info-eysfuzd7/cwltool.egg-info/PKG-INFO
  writing dependency_links to /private/var/folders/xg/1dp6d5j960v4s6vmv680jylr0000gp/T/pip-pip-egg-info-eysfuzd7/cwltool.egg-info/dependency_links.txt
  writing entry points to /private/var/folders/xg/1dp6d5j960v4s6vmv680jylr0000gp/T/pip-pip-egg-info-eysfuzd7/cwltool.egg-info/entry_points.txt
  writing requirements to /private/var/folders/xg/1dp6d5j960v4s6vmv680jylr0000gp/T/pip-pip-egg-info-eysfuzd7/cwltool.egg-info/requires.txt
  writing top-level names to /private/var/folders/xg/1dp6d5j960v4s6vmv680jylr0000gp/T/pip-pip-egg-info-eysfuzd7/cwltool.egg-info/top_level.txt
  writing manifest file '/private/var/folders/xg/1dp6d5j960v4s6vmv680jylr0000gp/T/pip-pip-egg-info-eysfuzd7/cwltool.egg-info/SOURCES.txt'
  reading manifest file '/private/var/folders/xg/1dp6d5j960v4s6vmv680jylr0000gp/T/pip-pip-egg-info-eysfuzd7/cwltool.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  no previously-included directories found matching 'cwltool/schemas/v1.0/salad/typeshed'
  no previously-included directories found matching 'cwltool/schemas/v1.0/salad/schema_salad/tests'
  no previously-included directories found matching 'cwltool/schemas/v1.1.0-dev1/salad/typeshed'
  no previously-included directories found matching 'cwltool/schemas/v1.1.0-dev1/salad/schema_salad/tests'
  no previously-included directories found matching 'cwltool/schemas/presentations'
  no previously-included directories found matching 'cwltool/schemas/site'
  no previously-included directories found matching 'cwltool/schemas/v1.0/examples'
  no previously-included directories found matching 'cwltool/schemas/v1.0/v1.0'
  no previously-included directories found matching 'cwltool/schemas/v1.1.0-dev1/examples'
  no previously-included directories found matching 'cwltool/schemas/v1.1.0-dev1/v1.1.0-dev1'
  warning: no previously-included files matching '*.py' found under directory 'cwltool/schemas'
  warning: no previously-included files found matching 'debian.img'
  warning: no previously-included files matching '*~' found anywhere in distribution
  warning: no previously-included files matching '*.pyc' found anywhere in distribution
  adding license file 'LICENSE.txt'
  writing manifest file '/private/var/folders/xg/1dp6d5j960v4s6vmv680jylr0000gp/T/pip-pip-egg-info-eysfuzd7/cwltool.egg-info/SOURCES.txt'
  Preparing metadata (setup.py): finished with status 'done'
Requirement already satisfied: setuptools in ./.tox/py310-mypy/lib/python3.10/site-packages (from cwltool==3.1.20220714234856) (62.6.0)
Requirement already satisfied: requests>=2.6.1 in ./.tox/py310-mypy/lib/python3.10/site-packages (from cwltool==3.1.20220714234856) (2.28.1)
Requirement already satisfied: ruamel.yaml<0.17.22,>=0.15 in ./.tox/py310-mypy/lib/python3.10/site-packages (from cwltool==3.1.20220714234856) (0.17.21)
Requirement already satisfied: rdflib<6.2.0,>=4.2.2 in ./.tox/py310-mypy/lib/python3.10/site-packages (from cwltool==3.1.20220714234856) (6.1.1)
Requirement already satisfied: shellescape<3.9,>=3.4.1 in ./.tox/py310-mypy/lib/python3.10/site-packages (from cwltool==3.1.20220714234856) (3.8.1)
Requirement already satisfied: schema-salad<9,>=8.2.20211104054942 in ./.tox/py310-mypy/lib/python3.10/site-packages (from cwltool==3.1.20220714234856) (8.3.20220626185350)
Requirement already satisfied: mypy-extensions in ./.tox/py310-mypy/lib/python3.10/site-packages (from cwltool==3.1.20220714234856) (0.4.3)
Requirement already satisfied: psutil>=5.6.6 in ./.tox/py310-mypy/lib/python3.10/site-packages (from cwltool==3.1.20220714234856) (5.9.1)
Requirement already satisfied: prov==1.5.1 in ./.tox/py310-mypy/lib/python3.10/site-packages (from cwltool==3.1.20220714234856) (1.5.1)
Requirement already satisfied: bagit>=1.6.4 in ./.tox/py310-mypy/lib/python3.10/site-packages (from cwltool==3.1.20220714234856) (1.8.1)
Requirement already satisfied: typing-extensions in ./.tox/py310-mypy/lib/python3.10/site-packages (from cwltool==3.1.20220714234856) (4.3.0)
Requirement already satisfied: coloredlogs in ./.tox/py310-mypy/lib/python3.10/site-packages (from cwltool==3.1.20220714234856) (15.0.1)
Requirement already satisfied: pydot>=1.4.1 in ./.tox/py310-mypy/lib/python3.10/site-packages (from cwltool==3.1.20220714234856) (1.4.2)
Requirement already satisfied: pyparsing!=3.0.2 in ./.tox/py310-mypy/lib/python3.10/site-packages (from cwltool==3.1.20220714234856) (3.0.9)
Requirement already satisfied: argcomplete in ./.tox/py310-mypy/lib/python3.10/site-packages (from cwltool==3.1.20220714234856) (2.0.0)
Requirement already satisfied: six>=1.9.0 in ./.tox/py310-mypy/lib/python3.10/site-packages (from prov==1.5.1->cwltool==3.1.20220714234856) (1.16.0)
Requirement already satisfied: networkx in ./.tox/py310-mypy/lib/python3.10/site-packages (from prov==1.5.1->cwltool==3.1.20220714234856) (2.8.4)
Requirement already satisfied: python-dateutil in ./.tox/py310-mypy/lib/python3.10/site-packages (from prov==1.5.1->cwltool==3.1.20220714234856) (2.8.2)
Requirement already satisfied: lxml in ./.tox/py310-mypy/lib/python3.10/site-packages (from prov==1.5.1->cwltool==3.1.20220714234856) (4.9.1)
Requirement already satisfied: isodate in ./.tox/py310-mypy/lib/python3.10/site-packages (from rdflib<6.2.0,>=4.2.2->cwltool==3.1.20220714234856) (0.6.1)
Requirement already satisfied: charset-normalizer<3,>=2 in ./.tox/py310-mypy/lib/python3.10/site-packages (from requests>=2.6.1->cwltool==3.1.20220714234856) (2.1.0)
Requirement already satisfied: idna<4,>=2.5 in ./.tox/py310-mypy/lib/python3.10/site-packages (from requests>=2.6.1->cwltool==3.1.20220714234856) (3.3)
Requirement already satisfied: certifi>=2017.4.17 in ./.tox/py310-mypy/lib/python3.10/site-packages (from requests>=2.6.1->cwltool==3.1.20220714234856) (2022.6.15)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in ./.tox/py310-mypy/lib/python3.10/site-packages (from requests>=2.6.1->cwltool==3.1.20220714234856) (1.26.10)
Requirement already satisfied: ruamel.yaml.clib>=0.2.6 in ./.tox/py310-mypy/lib/python3.10/site-packages (from ruamel.yaml<0.17.22,>=0.15->cwltool==3.1.20220714234856) (0.2.6)
Requirement already satisfied: mistune<0.9,>=0.8.1 in ./.tox/py310-mypy/lib/python3.10/site-packages (from schema-salad<9,>=8.2.20211104054942->cwltool==3.1.20220714234856) (0.8.4)
Requirement already satisfied: CacheControl<0.13,>=0.11.7 in ./.tox/py310-mypy/lib/python3.10/site-packages (from schema-salad<9,>=8.2.20211104054942->cwltool==3.1.20220714234856) (0.12.11)
Requirement already satisfied: lockfile>=0.9 in ./.tox/py310-mypy/lib/python3.10/site-packages (from schema-salad<9,>=8.2.20211104054942->cwltool==3.1.20220714234856) (0.12.2)
Requirement already satisfied: humanfriendly>=9.1 in ./.tox/py310-mypy/lib/python3.10/site-packages (from coloredlogs->cwltool==3.1.20220714234856) (10.0)
Requirement already satisfied: msgpack>=0.5.2 in ./.tox/py310-mypy/lib/python3.10/site-packages (from CacheControl<0.13,>=0.11.7->schema-salad<9,>=8.2.20211104054942->cwltool==3.1.20220714234856) (1.0.4)
Installing collected packages: cwltool
  Running setup.py develop for cwltool
    Running command python setup.py develop
    running develop
    /Users/jakefennick/cwltool/.tox/py310-mypy/lib/python3.10/site-packages/setuptools/command/easy_install.py:144: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
      warnings.warn(
    /Users/jakefennick/cwltool/.tox/py310-mypy/lib/python3.10/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
      warnings.warn(
    running egg_info
    writing cwltool.egg-info/PKG-INFO
    writing dependency_links to cwltool.egg-info/dependency_links.txt
    writing entry points to cwltool.egg-info/entry_points.txt
    writing requirements to cwltool.egg-info/requires.txt
    writing top-level names to cwltool.egg-info/top_level.txt
    reading manifest file 'cwltool.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    no previously-included directories found matching 'cwltool/schemas/v1.0/salad/typeshed'
    no previously-included directories found matching 'cwltool/schemas/v1.0/salad/schema_salad/tests'
    no previously-included directories found matching 'cwltool/schemas/v1.1.0-dev1/salad/typeshed'
    no previously-included directories found matching 'cwltool/schemas/v1.1.0-dev1/salad/schema_salad/tests'
    no previously-included directories found matching 'cwltool/schemas/presentations'
    no previously-included directories found matching 'cwltool/schemas/site'
    no previously-included directories found matching 'cwltool/schemas/v1.0/examples'
    no previously-included directories found matching 'cwltool/schemas/v1.0/v1.0'
    no previously-included directories found matching 'cwltool/schemas/v1.1.0-dev1/examples'
    no previously-included directories found matching 'cwltool/schemas/v1.1.0-dev1/v1.1.0-dev1'
    warning: no previously-included files matching '*.py' found under directory 'cwltool/schemas'
    warning: no previously-included files found matching 'debian.img'
    warning: no previously-included files matching '*~' found anywhere in distribution
    warning: no previously-included files matching '*.pyc' found anywhere in distribution
    adding license file 'LICENSE.txt'
    writing manifest file 'cwltool.egg-info/SOURCES.txt'
    running build_ext
    building '8f906987e2683b830d2d__mypyc' extension
    clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /Users/jakefennick/opt/anaconda3/envs/cwltool_test/include -fPIC -O2 -isystem /Users/jakefennick/opt/anaconda3/envs/cwltool_test/include -I/Users/jakefennick/cwltool/.tox/py310-mypy/lib/python3.10/site-packages/mypyc/lib-rt -Ibuild -I/Users/jakefennick/cwltool/.tox/py310-mypy/include -I/Users/jakefennick/opt/anaconda3/envs/cwltool_test/include/python3.10 -c build/__native_8f906987e2683b830d2d.c -o build/temp.macosx-10.9-x86_64-cpython-310/build/__native_8f906987e2683b830d2d.o -O3 -g1 -Werror -Wno-unused-function -Wno-unused-label -Wno-unreachable-code -Wno-unused-variable -Wno-unused-command-line-argument -Wno-unknown-warning-option
    build/__native_8f906987e2683b830d2d.c:56380:10: error: variable 'cpy_r_r654' set but not used [-Werror,-Wunused-but-set-variable]
        char cpy_r_r654;
             ^
    build/__native_8f906987e2683b830d2d.c:106245:10: error: variable 'cpy_r_r278' set but not used [-Werror,-Wunused-but-set-variable]
        char cpy_r_r278;
             ^
    build/__native_8f906987e2683b830d2d.c:108491:10: error: variable 'cpy_r_r205' set but not used [-Werror,-Wunused-but-set-variable]
        char cpy_r_r205;
             ^
    build/__native_8f906987e2683b830d2d.c:108471:10: error: variable 'cpy_r_r183' set but not used [-Werror,-Wunused-but-set-variable]
        char cpy_r_r183;
             ^
    build/__native_8f906987e2683b830d2d.c:110989:10: error: variable 'cpy_r_r18' set but not used [-Werror,-Wunused-but-set-variable]
        char cpy_r_r18;
             ^
    build/__native_8f906987e2683b830d2d.c:137993:10: error: variable 'cpy_r_r520' set but not used [-Werror,-Wunused-but-set-variable]
        char cpy_r_r520;
             ^
    build/__native_8f906987e2683b830d2d.c:153417:10: error: variable 'cpy_r_r174' set but not used [-Werror,-Wunused-but-set-variable]
        char cpy_r_r174;
             ^
    build/__native_8f906987e2683b830d2d.c:153430:10: error: variable 'cpy_r_r188' set but not used [-Werror,-Wunused-but-set-variable]
        char cpy_r_r188;
             ^
    build/__native_8f906987e2683b830d2d.c:153403:10: error: variable 'cpy_r_r159' set but not used [-Werror,-Wunused-but-set-variable]
        char cpy_r_r159;
             ^
    build/__native_8f906987e2683b830d2d.c:153379:10: error: variable 'cpy_r_r133' set but not used [-Werror,-Wunused-but-set-variable]
        char cpy_r_r133;
             ^
    build/__native_8f906987e2683b830d2d.c:153443:10: error: variable 'cpy_r_r202' set but not used [-Werror,-Wunused-but-set-variable]
        char cpy_r_r202;
             ^
    build/__native_8f906987e2683b830d2d.c:174703:10: error: variable 'cpy_r_r106' set but not used [-Werror,-Wunused-but-set-variable]
        char cpy_r_r106;
             ^
    build/__native_8f906987e2683b830d2d.c:174710:10: error: variable 'cpy_r_r114' set but not used [-Werror,-Wunused-but-set-variable]
        char cpy_r_r114;
             ^
    build/__native_8f906987e2683b830d2d.c:174670:10: error: variable 'cpy_r_r76' set but not used [-Werror,-Wunused-but-set-variable]
        char cpy_r_r76;
             ^
    build/__native_8f906987e2683b830d2d.c:179830:10: error: variable 'cpy_r_r191' set but not used [-Werror,-Wunused-but-set-variable]
        char cpy_r_r191;
             ^
    build/__native_8f906987e2683b830d2d.c:194430:10: error: variable 'cpy_r_r1546' set but not used [-Werror,-Wunused-but-set-variable]
        char cpy_r_r1546;
             ^
    build/__native_8f906987e2683b830d2d.c:194424:10: error: variable 'cpy_r_r1539' set but not used [-Werror,-Wunused-but-set-variable]
        char cpy_r_r1539;
             ^
    build/__native_8f906987e2683b830d2d.c:194413:10: error: variable 'cpy_r_r1527' set but not used [-Werror,-Wunused-but-set-variable]
        char cpy_r_r1527;
             ^
    build/__native_8f906987e2683b830d2d.c:213579:10: error: variable 'cpy_r_r46' set but not used [-Werror,-Wunused-but-set-variable]
        char cpy_r_r46;
             ^
    fatal error: too many errors emitted, stopping now [-ferror-limit=]
    20 errors generated.
    error: command '/usr/bin/clang' failed with exit code 1
    error: subprocess-exited-with-error
    
    × python setup.py develop did not run successfully.
    │ exit code: 1
    ╰─> See above for output.
    
    note: This error originates from a subprocess, and is likely not a problem with pip.
    full command: /Users/jakefennick/cwltool/.tox/py310-mypy/bin/python -c '
    exec(compile('"'"''"'"''"'"'
    # This is <pip-setuptools-caller> -- a caller that pip uses to run setup.py
    #
    # - It imports setuptools before invoking setup.py, to enable projects that directly
    #   import from `distutils.core` to work with newer packaging standards.
    # - It provides a clear error message when setuptools is not installed.
    # - It sets `sys.argv[0]` to the underlying `setup.py`, when invoking `setup.py` so
    #   setuptools doesn'"'"'t think the script is `-c`. This avoids the following warning:
    #     manifest_maker: standard file '"'"'-c'"'"' not found".
    # - It generates a shim setup.py, for handling setup.cfg-only projects.
    import os, sys, tokenize
    
    try:
        import setuptools
    except ImportError as error:
        print(
            "ERROR: Can not execute `setup.py` since setuptools is not available in "
            "the build environment.",
            file=sys.stderr,
        )
        sys.exit(1)
    
    __file__ = %r
    sys.argv[0] = __file__
    
    if os.path.exists(__file__):
        filename = __file__
        with tokenize.open(__file__) as f:
            setup_py_code = f.read()
    else:
        filename = "<auto-generated setuptools caller>"
        setup_py_code = "from setuptools import setup; setup()"
    
    exec(compile(setup_py_code, filename, "exec"))
    '"'"''"'"''"'"' % ('"'"'/Users/jakefennick/cwltool/setup.py'"'"',), "<pip-setuptools-caller>", "exec"))' develop --no-deps
    cwd: /Users/jakefennick/cwltool/
error: subprocess-exited-with-error

× python setup.py develop did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
make: *** [mypyc] Error 1
ERROR: InvocationError for command /usr/bin/make mypy mypyc PYTEST_EXTRA= (exited with code 2)

========================================================================================================================== log end ===========================================================================================================================

Your Environment

  • cwltool version: 3.1.20220714234856
    Check using cwltool --version

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions