fix system image library uninstallation on macos x86

[pmeier]

#7417 touched the general dev env creation for CI. Amongst other things, it now also install system libpng libraries on macOS x86. However, this leads to failures with our I/O tests: https://github.com/pytorch/vision/actions/runs/5057898641/jobs/9077254208. This uninstallation was part of the journey to get CMake running, but was no longer needed in the end.

Thanks to @NicolasHug observation, it turned out that the root cause was a broken uninstall logic and nothing to do with the PNG libs. This PR now fixes the error described in #7622 (comment).

cc @seemethere

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/vision/7622

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 32 New Failures

As of commit 01c1a95:

NEW FAILURES - The following jobs have failed:

• build / linux-job (gh)
• c-source / linux-job (gh)
• linux (linux.12xlarge, cpu) / linux-job (gh)
• linux (linux.g5.4xlarge.nvidia.gpu, cuda, 11.8) / linux-job (gh)
• macos (macos-12) / macos-job (gh)
• macos (macos-m1-12) / macos-job (gh)
• onnx / linux-job (gh)
• python-source-and-configs / linux-job (gh)
• python-types / linux-job (gh)
• unittests-extended / linux-job (gh)
• unittests-linux (3.10, linux.12xlarge, cpu) / linux-job (gh)
• unittests-linux (3.11, linux.12xlarge, cpu) / linux-job (gh)
• unittests-linux (3.8, linux.12xlarge, cpu) / linux-job (gh)
• unittests-linux (3.8, linux.g5.4xlarge.nvidia.gpu, cuda, 11.8) / linux-job (gh)
• unittests-linux (3.9, linux.12xlarge, cpu) / linux-job (gh)
• unittests-macos (3.10, macos-12) / macos-job (gh)
• unittests-macos (3.11, macos-12) / macos-job (gh)
• unittests-macos (3.8, macos-12) / macos-job (gh)
• unittests-macos (3.8, macos-m1-12) / macos-job (gh)
• unittests-macos (3.9, macos-12) / macos-job (gh)
• unittests-prototype (3.10, linux.12xlarge, cpu) / linux-job (gh)
• unittests-prototype (3.11, linux.12xlarge, cpu) / linux-job (gh)
• unittests-prototype (3.8, linux.12xlarge, cpu) / linux-job (gh)
• unittests-prototype (3.8, linux.g5.4xlarge.nvidia.gpu, cuda, 11.8) / linux-job (gh)
• unittests-prototype (3.9, linux.12xlarge, cpu) / linux-job (gh)
• unittests-windows (3.10, windows.4xlarge, cpu) / windows-job (gh)
• unittests-windows (3.11, windows.4xlarge, cpu) / windows-job (gh)
• unittests-windows (3.8, windows.4xlarge, cpu) / windows-job (gh)
• unittests-windows (3.8, windows.g5.4xlarge.nvidia.gpu, cuda, 11.8) / windows-job (gh)
• unittests-windows (3.9, windows.4xlarge, cpu) / windows-job (gh)
• windows (windows.4xlarge, cpu) / windows-job (gh)
• windows (windows.g5.4xlarge.nvidia.gpu, cuda, 11.8) / windows-job (gh)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[pmeier]

8c5b877 has a green CI for all CMake and unittest workflows.

[NicolasHug]

Stamping, but why is removing the uninstallation of png lib has an effect on the jpeg lib? IIUC the failures in https://pipelines.actions.githubusercontent.com/serviceHosts/e1cea170-7cad-4fd4-b94c-3e3f4452f685/_apis/pipelines/1/runs/60504/signedlogcontent/24?urlExpires=2023-05-24T10%3A29%3A04.2815900Z&urlSigningMethod=HMACV1&urlSignature=k7F0EHX1UgrEUKv1kqdLfWJGDUDX7mK%2Bx543dpy3418%3D relate to libjpeg, not libpng

[pmeier]

That is a really good question. I honestly didn't see that. My guess is that the new logic is somehow wrong and jpeg library is not uninstalled for some reason. Let me check.

[pmeier]

Log output

before:

2023-05-23T07:05:30.8266370Z ##[group]Uninstall system JPEG libraries on macOS
2023-05-23T07:05:43.3191720Z jpeg-turbo jpeg-xl openjpeg
2023-05-23T07:05:46.3387080Z Uninstalling jpeg-turbo... (44 files, 3.9MB)
2023-05-23T07:05:49.6651010Z Uninstalling jpeg-xl... (43 files, 19.4MB)
2023-05-23T07:05:52.1431560Z Uninstalling openjpeg... (536 files, 13.8MB)
2023-05-23T07:05:52.6152150Z ##[endgroup]

So I guess I'm right since the log after the patch never has the Uninstalling ... output coming from brew.

after:

2023-05-23T13:28:30.1339840Z + echo '::group::Uninstall system JPEG libraries on macOS'
2023-05-23T13:28:30.1340560Z ++ brew list
2023-05-23T13:28:30.1389040Z ++ grep -E 'jpeg|png'
2023-05-23T13:28:44.4600320Z + IMAGE_LIBS='jpeg-turbo
2023-05-23T13:28:44.4600770Z jpeg-turbo
2023-05-23T13:28:44.4697740Z jpeg-xl
2023-05-23T13:28:44.4698360Z jpeg-xl
2023-05-23T13:28:44.4732740Z libpng
2023-05-23T13:28:44.4733290Z libpng
2023-05-23T13:28:44.4790080Z openjpeg'
2023-05-23T13:28:44.4790790Z openjpeg
2023-05-23T13:28:44.4822750Z + echo 'jpeg-turbo
2023-05-23T13:28:44.4838960Z jpeg-xl
2023-05-23T13:28:44.4885750Z libpng
2023-05-23T13:28:44.4924170Z openjpeg'
2023-05-23T13:28:44.4970110Z + for lib in '"${IMAGE_LIBS}"'
2023-05-23T13:28:44.5014900Z + brew uninstall --ignore-dependencies --force 'jpeg-turbo
2023-05-23T13:28:44.5066140Z jpeg-xl
2023-05-23T13:28:44.5107200Z libpng
2023-05-23T13:28:44.5131170Z openjpeg'
2023-05-23T13:28:47.4642280Z + echo ::endgroup::

[pmeier]

Ok, there were two things coming together with

  for lib in "${IMAGE_LIBS}"; do
    brew uninstall --ignore-dependencies --force "${lib}" || true

1. I botched the loop by wrapping IMAGE_LIBS into a string. Thus, instead of iterating over the individual libs, we just had one string. For example:

   LIBS="foo bar baz"
   
   echo 1
   for lib in $LIBS; do
       echo "${lib}"
   done
   
   echo 2
   for lib in "${LIBS}"; do
       echo "${lib}"
   done

   Running this prints

   1
   foo
   bar
   baz
   2
   foo bar baz
   

2. While definitely a bug, this went undetected because we also ignored the error code brew uninstall ... || true.

I've fixed both things in 334b4e5

[pmeier]

Looking good now

  + echo '::group::Uninstall system JPEG libraries on macOS'
  ++ brew list
  ++ grep -E 'jpeg|png'
  + IMAGE_LIBS='jpeg-turbo
  jpeg-xl
  libpng
  openjpeg'
  + for lib in '$IMAGE_LIBS'
  + brew uninstall --ignore-dependencies --force jpeg-turbo
  Uninstalling jpeg-turbo... (44 files, 3.9MB)
  + for lib in '$IMAGE_LIBS'
  + brew uninstall --ignore-dependencies --force jpeg-xl
  Uninstalling jpeg-xl... (43 files, 19.4MB)
  + for lib in '$IMAGE_LIBS'
  + brew uninstall --ignore-dependencies --force libpng
  Uninstalling libpng... (27 files, 1.3MB)
  + for lib in '$IMAGE_LIBS'
  + brew uninstall --ignore-dependencies --force openjpeg
  Uninstalling openjpeg... (536 files, 13.8MB)
  + echo ::endgroup::

[NicolasHug]

Thanks!

[pmeier]

CMake and unittest jobs on macOS x86 are green!

[github-actions]

Hey @pmeier!

You merged this PR, but no labels were added. The list of valid labels is available at https://github.com/pytorch/vision/blob/main/.github/process_commit.py