Skip to content

Give each mypy prek hook its own file list in CI - #70654

Merged
Lee-W merged 1 commit into
apache:mainfrom
FrankYang0529:airflow-fix-mypy-hook-concurrency
Aug 6, 2026
Merged

Give each mypy prek hook its own file list in CI#70654
Lee-W merged 1 commit into
apache:mainfrom
FrankYang0529:airflow-fix-mypy-hook-concurrency

Conversation

@FrankYang0529

@FrankYang0529 FrankYang0529 commented Jul 29, 2026

Copy link
Copy Markdown
Member

Why

  • prek runs hooks from different prek projects concurrently. Every one of them wrote its mypy file list to the same files/mypy_files.txt.
  • On v3-3-test run 30279407098, mypy-shared-configuration, mypy-shared-dagnode and mypy-shared-listeners all failed on non-existent path.

Their writing file time are close.

2026-07-27T17:28:58.9980347Z     Running mypy for folders: ['shared/configuration']
2026-07-27T17:28:58.9981475Z     You can check the list of files in: /home/runner/work/airflow/airflow/files/mypy_files.txt

2026-07-27T17:28:59.0832479Z     Running mypy for folders: ['shared/dagnode']
2026-07-27T17:28:59.0833606Z     You can check the list of files in: /home/runner/work/airflow/airflow/files/mypy_files.txt

2026-07-27T17:28:59.1323493Z     Running mypy for folders: ['shared/listeners']
2026-07-27T17:28:59.1324120Z     You can check the list of files in: /home/runner/work/airflow/airflow/files/mypy_files.txt
2026-07-27T17:28:59.0455195Z     mypy: Cannot read file 'shared/dagnode/tests/dagnode/__init__.pyteners/spec/taskinstance.py': No such file or directory

2026-07-27T17:28:59.1199107Z     mypy: Cannot read file 'shared/dagnode/tests/dagnode/__init__.pyteners/spec/taskinstance.py': No such file or directory

2026-07-27T17:28:59.1521972Z     mypy: Cannot read file 'shared/dagnode/tests/dagnode/__init__.pyteners/spec/taskinstance.py': No such file or directory

How

  • run_mypy_full_dist_local_venv_or_breeze_in_ci.py: Use suffix argument in write_file_list() to avoid overwriting same file.

Was generative AI tooling used to co-author this PR?
  • Yes - Claude Code

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

Signed-off-by: PoAn Yang <payang@apache.org>
@FrankYang0529
FrankYang0529 force-pushed the airflow-fix-mypy-hook-concurrency branch from 18c1b6b to 16e39f3 Compare July 29, 2026 08:31
@potiuk
potiuk marked this pull request as ready for review July 29, 2026 20:59
@potiuk

potiuk commented Jul 29, 2026

Copy link
Copy Markdown
Member

Good idea - I think prek only fairly recently started to run checks in parallel but that one seems like it wll solve that issue.

@Lee-W
Lee-W merged commit ddef168 into apache:main Aug 6, 2026
134 checks passed
@github-actions

github-actions Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Backport successfully created: v3-3-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status Branch Result
v3-3-test PR Link

github-actions Bot pushed a commit to aws-mwaa/upstream-to-airflow that referenced this pull request Aug 6, 2026
…70654)

(cherry picked from commit ddef168)

Co-authored-by: PoAn Yang <payang@apache.org>
Signed-off-by: PoAn Yang <payang@apache.org>
aws-airflow-bot pushed a commit to aws-mwaa/upstream-to-airflow that referenced this pull request Aug 6, 2026
…70654)

(cherry picked from commit ddef168)

Co-authored-by: PoAn Yang <payang@apache.org>
Signed-off-by: PoAn Yang <payang@apache.org>
@FrankYang0529
FrankYang0529 deleted the airflow-fix-mypy-hook-concurrency branch August 7, 2026 03:51
potiuk pushed a commit that referenced this pull request Aug 7, 2026
…#71236)

(cherry picked from commit ddef168)

Signed-off-by: PoAn Yang <payang@apache.org>
Co-authored-by: PoAn Yang <payang@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants