fix(sca): avoid exception on malformed packages #14167
Conversation
florentinl
added
changelog/no-changelog
|
|
Bootstrap import analysisComparison of import times between this PR and base. SummaryThe average import time from this PR is: 286 ± 3 ms. The average import time from base is: 289 ± 3 ms. The import time difference between this PR and base is: -2.3 ± 0.1 ms. Import time breakdownThe following import paths have shrunk:
|
Performance SLOsCandidate: florentinl/APPSEC-58471/fix-malformed-metadata (044ca6a) 🔵 No Baseline Data (24 suites)🔵 coreapiscenario - 12/12 (2 unstable)🔵 No baseline data available for this suite
|
florentinl
removed
the
changelog/no-changelog
florentinl
force-pushed
the
florentinl/APPSEC-58471/fix-malformed-metadata
branch
8 times, most recently
from
7e1232d to
e6c5d9a
Compare
florentinl
force-pushed
the
florentinl/APPSEC-58471/fix-malformed-metadata
branch
from
e6c5d9a to
044ca6a
Compare
## Problem
Some dependencies may be malformed and may be missing a `METADATA` file
in its `*.dist-info` folder.
In this case, we still want to send the information about the packages
that are correctly formed.
Currently when a single package has no METADATA file in its dist-info
folder, the code raises an error and nothing is reported.
```python
for dist in importlib_metadata.distributions():
# PKG-INFO and/or METADATA files are parsed when dist.metadata is accessed
# Optimization: we should avoid accessing dist.metadata more than once
metadata = dist.metadata
name = metadata["name"].lower() # <- metadata["name"] is None and calling .lower() raises an error
version = metadata["version"]
if name and version:
pkgs[name] = version
```
## Context
I noticed this while investigating SCA in AWS Lambda, where the
`dist-info` files are removed to slim down the layer:
```Dockerfile
# remove *.dist-info directories except any entry_points.txt files
RUN find ./python/lib/$runtime/site-packages/*.dist-info -not -name "entry_points.txt" -type f -delete
```
https://github.com/DataDog/datadog-lambda-python/blob/f1a4cb4f52abd42f99a42d8764963fd0c0a9297d/Dockerfile#L28-L29
## Checklist
- [x] PR author has checked that all the criteria below are met
- The PR description includes an overview of the change
- The PR description articulates the motivation for the change
- The change includes tests OR the PR description describes a testing
strategy
- The PR description notes risks associated with the change, if any
- Newly-added code is easy to change
- The change follows the [library release note
guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html)
- The change includes or references documentation updates if necessary
- Backport labels are set (if
[applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting))
## Reviewer Checklist
- [x] Reviewer has checked that all the criteria below are met
- Title is accurate
- All changes are related to the pull request's stated goal
- Avoids breaking
[API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces)
changes
- Testing strategy adequately addresses listed risks
- Newly-added code is easy to change
- Release note makes sense to a user of the library
- If necessary, author has acknowledged and discussed the performance
implications of this PR as reported in the benchmarks PR comment
- Backport labels are set in a manner that is consistent with the
[release branch maintenance
policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)
Problem
Some dependencies may be malformed and may be missing a
METADATAfile in its*.dist-infofolder.In this case, we still want to send the information about the packages that are correctly formed.
Currently when a single package has no METADATA file in its dist-info folder, the code raises an error and nothing is reported.
Context
I noticed this while investigating SCA in AWS Lambda, where the
dist-infofiles are removed to slim down the layer:https://github.com/DataDog/datadog-lambda-python/blob/f1a4cb4f52abd42f99a42d8764963fd0c0a9297d/Dockerfile#L28-L29
Checklist
Reviewer Checklist