Skip to content

gh-153946: Add API to defer automatic garbage collection#154188

Draft
pablogsal wants to merge 3 commits into
python:mainfrom
pablogsal:agent/defer-automatic-gc
Draft

gh-153946: Add API to defer automatic garbage collection#154188
pablogsal wants to merge 3 commits into
python:mainfrom
pablogsal:agent/defer-automatic-gc

Conversation

@pablogsal

@pablogsal pablogsal commented Jul 19, 2026

Copy link
Copy Markdown
Member

No description provided.

Copy link
Copy Markdown
Member Author

I ran this a few times on release builds pinned to one CPU, comparing aeedae8 with this branch. These timings are the median of the medians from three independent runs, with 15-20 samples and 10 iterations per sample.

Workload main This PR Change Automatic collections
Parse eight large stdlib files into AST objects 96.22 ms 78.20 ms 18.7% faster 8 per batch -> 0
marshal.loads(), 50,000 records 12.73 ms 10.49 ms 17.6% faster 1 per call -> 0

I also ran perf stat -r 5 on the same CPython workloads. The AST case went from 26.25B to 24.14B instructions, 8.1% fewer, and task clock dropped 18.5%. The marshal case went from 4.25B to 3.57B instructions, 15.9% fewer, and task clock dropped 17.8%.

I tried scheduling a collection directly from _PyGC_ResumeAutomaticCollection() as well. That made AST conversion about 3% slower than main, so this version keeps counting and waits for the next normal scheduling point. I think this is the right behavior here.

Comment on lines +2074 to +2080
// Close the race with a deferral that started before the world stopped.
if (state->reason == _Py_GC_REASON_HEAP &&
_Py_atomic_load_int(
&state->gcstate->automatic_collection_pause_count)) {
_PyEval_StartTheWorld(interp);
return false;
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need a similar check before this?

cpython/Python/gc.c

Lines 1486 to 1497 in fe0cc2b

/* update collection and allocation counters */
if (generation+1 < NUM_GENERATIONS) {
gcstate->generations[generation+1].count += 1;
}
for (i = 0; i <= generation; i++) {
gcstate->generations[i].count = 0;
}
/* merge younger generations with one we are currently collecting */
for (i = 0; i < generation; i++) {
gc_list_merge(GEN_HEAD(gcstate, i), GEN_HEAD(gcstate, generation));
}

This is a very shy question, so please treat it with a huge grain of salt. My hunch is that GC callbacks may still cause race, since they're just regular Python code and may release GIL:

cpython/Python/gc.c

Lines 1467 to 1472 in fe0cc2b

GC_STAT_ADD(generation, collections, 1);
struct gc_generation_stats stats = { 0 };
if (reason != _Py_GC_REASON_SHUTDOWN) {
invoke_gc_callback(tstate, "start", generation, &stats);
}

I interrogated Claude Opus to show me what is the actual place for invoking and returning from the callbacks, and that's how I arrived at 1486.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants