[tools] add utilities for deterministic port to kernels

[sayakpaul]

Related to #729.

I know we have been talking about having a new DSL. I didn't take that route in this PR. Instead, I decided to add atomic components that can be used in isolation as well as in the DSL. These operations include:

• Determining the latest release tag and then fetching it for a given kernel repository on GitHub.
• Relativizing imports as specified in https://huggingface.co/docs/kernels/main/en/kernel-requirements
• Fixing PyTorch op registration as specified in https://huggingface.co/docs/kernels/main/en/kernel-requirements through ASTs

Have added tests for each atomic operation, too. But didn't intentionally add them to the CI because we still haven't decided on the direction this whole utility would take.

Tip

I think for the initial review, it's more important to decide if we want to have this kind of atomic utilities and agree on their signatures rather than their actual implementation details. Best to maybe start with the README.md added in this PR to get a sense of how I have decided to structure it.

TODOs:

• Documentation. Planning on having either a short section or an entire page dedicated to "porting existing kernel projects to be kernel-builder compliant".
• Wire this up in AGENTS.md of kernels-community so that agents actually know which tools to use for deriving deterministic outputs rather than hits and misses.

[github-actions]

Coverage report — kernels/

Measured on: Python 3.10 / Torch 2.13.0.
Other CI configurations are not included in this number.
Hardware-gated code paths (ROCm/XPU/NPU/Darwin/Windows) are excluded or unreachable on the Linux+CUDA runner.

Total coverage: 84.6% — threshold: 80% — ✅

Per-file breakdown

Name	Stmts	Miss	Cover	Missing
src/kernels/__init__.py	10	0	100%
src/kernels/_system.py	6	1	83%	10
src/kernels/_versions.py	63	7	89%	46, 49, 52-53, 56-57, 100
src/kernels/backends.py	194	55	72%	40, 44, 48-51, 68, 90, 108, 117, 121, 125-127, 148, 170, 181, 188-191, 201, 205-225, 233, 256-276
src/kernels/compat.py	8	1	88%	5
src/kernels/deps.py	54	4	93%	58-59, 95, 98
src/kernels/layer/__init__.py	6	0	100%
src/kernels/layer/_interval_tree.py	103	4	96%	23, 52, 147, 150
src/kernels/layer/device.py	48	14	71%	42, 47-49, 91, 96-98, 101, 149, 152, 155-157
src/kernels/layer/func.py	81	7	91%	81, 111, 183, 301, 307, 320, 338
src/kernels/layer/globals.py	5	0	100%
src/kernels/layer/kernelize.py	74	8	89%	255, 281, 289-290, 296, 300, 316-318
src/kernels/layer/layer.py	210	16	92%	167, 210, 216, 229, 337, 417-418, 430, 439, 447, 458, 487, 491, 504, 557, 587
src/kernels/layer/mode.py	14	0	100%
src/kernels/layer/repos.py	130	34	74%	27, 33, 36-41, 61-62, 68, 71-74, 88, 92, 101-102, 108, 111-114, 121-122, 128, 131-134, 141-142, 148, 151-154, 235
src/kernels/lockfile.py	71	46	35%	37-104, 108-131
src/kernels/status.py	49	2	96%	23, 81
src/kernels/utils.py	313	55	82%	65, 77-81, 87-88, 246, 250, 253, 315, 323, 362-363, 401, 432, 437, 472, 701, 704, 706, 712, 725-726, 747-759, 763-770, 778, 782-792, 796-803, 841, 845, 864, 866
src/kernels/variants.py	270	22	92%	63, 94, 115, 145, 254-255, 297-300, 302, 386-393, 399-405, 436-442, 454-460, 559-561
src/kernels/verify.py	88	1	99%	32
TOTAL	1797	277	85%

Updated by the Test kernels workflow on commit 36ef3bd553c383c86a1e62e7039833a78ff2684c.

[drbh]

thanks for putting this together! I think the AST utilities here are useful, especially the import relativization and op registration work. however, after looking through this more, I don't think standalone tools driven by an agent are exactly the design we want for the larger porting workflow.

the long term goal is to have a maintainable definition of how an upstream kernel is converted into the kernel-builder format. ideally a port should look something like:

pinned upstream checkout + 
checked-in conversion steps + 
checked-in overlays = 
kernel-builder compatible tree

when upstream changes, we should be able to update the pin and rerun the same conversion from a fresh checkout. if upstream changed something the conversion depends on, it should fail at that specific step and require us to update the conversion. this gives us a reviewable history of how the port changes along with upstream.

with the current approach, each individual tool can be deterministic and safe to rerun, but the port as a whole is still stateful. the agent decides what to copy, move, delete, which tools to run, the order to run them in, and how to resolve anything the tools cannot handle. those decisions end up represented in the mutated source tree rather than in something we can replay later.

I think we should start with the higher level pipeline/DSL contract and make transformations like these operations within it. a lot of the AST work in this PR could still be reused there. the important difference is that the main interface should take pinned source inputs, run a checked-in conversion against a fresh tree, and produce the same output for the same inputs.

tldr; I think this PR contains useful primitives, but the thing we need to maintain is the conversion recipe, not the converted checkout. before settling on the standalone tool signatures, I think we should agree on that higher level execution model first.

[sayakpaul]

Closing for now.