fix(pypi): rewrite RECORD file entries for extracted .data contents - #4025
Draft
rickeylev wants to merge 12 commits into
Draft
fix(pypi): rewrite RECORD file entries for extracted .data contents#4025rickeylev wants to merge 12 commits into
rickeylev wants to merge 12 commits into
Conversation
…and are readable Expand importlib_metadata_test to assert that all files returned by importlib.metadata.files() can be found on disk and read both as binary and UTF-8 text. Update test fixture RECORD files to list installed paths.
…und and read Assert that each file returned by importlib.metadata.files() exists and can be read, reproducing the issue with unrewritten RECORD files.
Verify that the list of files returned by importlib.metadata.files() matches the expected set of paths from the distribution's RECORD.
Sort expected_paths list alphabetically in source for improved readability.
When extracting wheels containing .data directories (purelib, platlib, scripts, headers, data), the files are moved into their respective target scheme directories and .data is deleted. Rewrite .dist-info/RECORD entries so that archive member paths prefixed with .data/ are updated to their installed paths relative to site-packages, allowing tools like importlib.metadata.files() to locate and read them. Also add Starlark unit tests for the RECORD rewriting logic.
rickeylev
changed the title
Unify extraction destinations and RECORD path rewrite prefixes into a single _DATA_CATEGORIES mapping in whl_extract.bzl. Also add a news entry fragment for the bug fix in bazel-contrib#4025.
…_test On Windows, virtual environments have a 2-level directory depth (Lib/site-packages) compared to POSIX 3-level depth (lib/pythonX.Y/site-packages), so relative paths in RECORD starting with .. cannot resolve via locate() on Windows. Skip locate() on Windows for files outside site-packages.
Virtual environment directory structure depth differs between Windows (`Lib/site-packages`) and POSIX (`lib/pythonX.Y/site-packages`). Because the host platform during repo extraction cannot determine the target platform, RECORD files must be generated per target platform. Move extracted RECORD files to `rewrite-record/` and rewrite relative paths for `.data` directories at build time via `gen_wheel_record`.
Update wheel_record_rewriter_test.sh to run .ps1 scripts via powershell.exe
when executing under Windows MSYS2 bash.
Update importlib_metadata_test.py to assert data_prefix ('../../') for
files originating in .data/data/bin/ and .data/data/include/ on Windows.
Convert input/output paths using cygpath -w when running powershell in wheel_record_rewriter_test.sh and ignore CRLF differences in diff. Handle .bat extension on Windows venv bin scripts when validating located files in importlib_metadata_test.py.
Collaborator
|
A few open questions:
|
Collaborator
Author
|
AFAIK, we're extracting the wheel the same as what other installers do, sans the later "install" part where it shows up into a binary's venv.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Extracting wheels with .data/ subdirectories moves files to their
target directories and deletes .data/, but left .dist-info/RECORD
referencing deleted .data/ paths. This causes tools like
importlib.metadata.files() to fail locating or reading files.
Per PEP 427 and PEP 376, rewrite .dist-info/RECORD entries during
wheel extraction to match installed locations relative to
site-packages.
importlib_metadata_test.
Work towards #3024