mirror of
https://github.com/kuhyx/testsAndMisc.git
synced 2026-11-21 04:02:17 +01:00
Cache pip and pre-commit envs in CI
Measured per-step timings, rather than guessing where the time went:
Pre-commit checks Install dependencies 63s (next slowest: 16s)
Python tests Install dependencies 49s (pytest itself: 35s)
Both workflows re-downloaded ~80 packages from PyPI on every run. The
Node workflows already cache npm; the Python ones were the outliers.
* cache: pip on actions/setup-python in both, keyed on
meta/requirements.txt -- the REAL file, not the root symlink, since
the key is hashed from the file at the given path and hashing a
symlink is not hashing its target. (That same symlink already cost
us a silently-skipped workflow earlier today.)
* ~/.cache/pre-commit via actions/cache@v4, keyed on
.pre-commit-config.yaml. pre-commit builds a virtualenv per hook repo
on first use, which costs more than the pip install itself, and it
only needs rebuilding when a hook's repo or rev changes.
The first run after this lands is a cache miss and will be no faster;
the saving starts from the second.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PJrqKERAo5QUeEHWS5Asgp
This commit is contained in:
parent
12058a4201
commit
1351a1460f
12
.github/workflows/pre-commit.yml
vendored
12
.github/workflows/pre-commit.yml
vendored
@ -15,6 +15,10 @@ jobs:
|
|||||||
- uses: actions/setup-python@v5
|
- uses: actions/setup-python@v5
|
||||||
with:
|
with:
|
||||||
python-version: "3.12"
|
python-version: "3.12"
|
||||||
|
# See the note in python-tests.yml: caches the pip download cache,
|
||||||
|
# keyed on the real meta/ path rather than the root symlink.
|
||||||
|
cache: pip
|
||||||
|
cache-dependency-path: meta/requirements.txt
|
||||||
|
|
||||||
# zsh backs the zsh-syntax hook. apt does not let a patch release be
|
# zsh backs the zsh-syntax hook. apt does not let a patch release be
|
||||||
# selected, so this is pinned only to what the runner image provides;
|
# selected, so this is pinned only to what the runner image provides;
|
||||||
@ -96,6 +100,14 @@ jobs:
|
|||||||
fi
|
fi
|
||||||
echo "prettier ${prettier_installed} and shfmt ${shfmt_installed} pinned and on PATH"
|
echo "prettier ${prettier_installed} and shfmt ${shfmt_installed} pinned and on PATH"
|
||||||
|
|
||||||
|
# pre-commit builds a virtualenv per hook repo on first use, which is
|
||||||
|
# slower than the pip install itself. Keyed on the hook definitions, so
|
||||||
|
# it invalidates exactly when a hook's repo or rev changes.
|
||||||
|
- uses: actions/cache@v4
|
||||||
|
with:
|
||||||
|
path: ~/.cache/pre-commit
|
||||||
|
key: pre-commit-${{ runner.os }}-py3.12-${{ hashFiles('.pre-commit-config.yaml') }}
|
||||||
|
|
||||||
- name: Install dependencies
|
- name: Install dependencies
|
||||||
run: |
|
run: |
|
||||||
python -m pip install --upgrade pip
|
python -m pip install --upgrade pip
|
||||||
|
|||||||
8
.github/workflows/python-tests.yml
vendored
8
.github/workflows/python-tests.yml
vendored
@ -43,6 +43,14 @@ jobs:
|
|||||||
- uses: actions/setup-python@v5
|
- uses: actions/setup-python@v5
|
||||||
with:
|
with:
|
||||||
python-version: "3.11"
|
python-version: "3.11"
|
||||||
|
# Cache the pip DOWNLOAD cache, keyed on the requirements file, so a
|
||||||
|
# run that changes nothing about dependencies reuses the wheels
|
||||||
|
# instead of re-fetching ~80 packages from PyPI. Points at
|
||||||
|
# meta/requirements.txt, not the root symlink: the cache key is
|
||||||
|
# hashed from the file at this path, and hashing a symlink is not
|
||||||
|
# the same as hashing its target.
|
||||||
|
cache: pip
|
||||||
|
cache-dependency-path: meta/requirements.txt
|
||||||
- name: Install system dependencies
|
- name: Install system dependencies
|
||||||
run: |
|
run: |
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|||||||
@ -0,0 +1,43 @@
|
|||||||
|
{
|
||||||
|
"intent": "Stop the two Python CI workflows re-downloading ~80 packages from PyPI on every run, which was the single slowest step in both.",
|
||||||
|
"scope": [
|
||||||
|
".github/workflows/python-tests.yml",
|
||||||
|
".github/workflows/pre-commit.yml",
|
||||||
|
"Non-goal: pinning dependencies or trimming the requirements list"
|
||||||
|
],
|
||||||
|
"changes": [
|
||||||
|
"Added cache: pip with cache-dependency-path: meta/requirements.txt to actions/setup-python in both workflows. The path is the real file, not the root requirements.txt symlink, because the cache key is hashed from the file at the given path.",
|
||||||
|
"Added an actions/cache@v4 step for ~/.cache/pre-commit in pre-commit.yml, keyed on .pre-commit-config.yaml, so hook virtualenvs are not rebuilt when no hook changed.",
|
||||||
|
"The Node workflows (poker-stakes, reverse-survivors) already cached npm; the Python ones were the outliers."
|
||||||
|
],
|
||||||
|
"verification": [
|
||||||
|
{
|
||||||
|
"command": "gh api .../actions/runs/32645134828/jobs -- per-step timings for Pre-commit checks",
|
||||||
|
"result": "pass",
|
||||||
|
"evidence": "Install dependencies = 63s, the dominant step; every other step 16s or less. This is the step the cache targets."
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"command": "gh api .../actions/runs/32645075080/jobs -- per-step timings for Python tests",
|
||||||
|
"result": "pass",
|
||||||
|
"evidence": "Install dependencies = 49s vs 35s actually running pytest; same dominant step"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"command": "python3 -c \"import yaml; yaml.safe_load(...)\" on both workflow files",
|
||||||
|
"result": "pass",
|
||||||
|
"evidence": "both parse as valid YAML"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"command": "pre-commit run --files .github/workflows/python-tests.yml .github/workflows/pre-commit.yml",
|
||||||
|
"result": "pass",
|
||||||
|
"evidence": "no hook reported a failure (prettier, check-yaml included)"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"risks": [
|
||||||
|
"A stale pip cache could in principle serve an old wheel; mitigated because the key is a hash of the requirements file, so any dependency change invalidates it.",
|
||||||
|
"The first run after this lands is a cache MISS and will be no faster; the saving appears from the second run onward."
|
||||||
|
],
|
||||||
|
"rollback": [
|
||||||
|
"Remove the cache lines from both setup-python blocks and delete the actions/cache step.",
|
||||||
|
"After rollback, confirm the Install dependencies step returns to ~50-60s."
|
||||||
|
]
|
||||||
|
}
|
||||||
Loading…
Reference in New Issue
Block a user