gh-122931 CI for Linux multiarch co-installability by stefanor · Pull Request #152831 · python/cpython

stefanor · 2026-07-01T22:23:31Z

As suggested in this thread, add some CI tooling to ensure that Python continues to be multi-arch co-installable.

Explicitly skip checking abi3.so and abi3t.so extensions, as they are currently not co-installable (#122931).

We extend reusable-ubuntu to perform an extra set of build+install+hash runs. This required some slightly different configure options to get more co-install-friendly builds. Those configure options broke some existing tests, but the fixes are trivial, and included.

From an earlier implementation that didn't use hash-manifests:

Example failure (without abi3 and abi3t ignored): https://github.com/stefanor/cpython/actions/runs/28550736484/job/84647793000
Example success (with them ignored): https://github.com/stefanor/cpython/actions/runs/28551134448/job/84648971317

Issue: Allow stable ABI extensions to include multiarch tuples in the filename #122931

…dule

stefanor · 2026-07-01T23:09:35Z

Note: This will require an extra ~165MiB * 5 test builds = ~820 MiB of artifact storage in GitHub CI, per test run. These are stored for 90 days by default. We could configure less...

Duh. Let me store hash manifests instead of storing the installs.

webknjaz · 2026-07-02T18:01:09Z

We do not do this in the main test suite execution, as achieving multi-arch co-installability requires some non-default configure parameters.

Do you mean the debug option?

But we could share builds, if desired for performance.

Seems like having another matrix will add 15–16 minutes of CI runtime.

We are trying to install into a prefix, any absolute path would not necessarily be writeable. e.g. if Python is configured with --libdir=/usr/lib/$(MULTIARCH)/ during a Debian build.

stefanor · 2026-07-03T21:10:05Z

We do not do this in the main test suite execution, as achieving multi-arch co-installability requires some non-default configure parameters.

Do you mean the debug option?

I meant a multiarch libdir and --with-build-details-suffix. And in fact I needed two extra commits to fix tests with these options:

d4c6100 that I probably should have included in build-details.json should be installed to different locations for nondebug/debug builds and have distinct base_interpreter values #131372
d390e7a which is already a PR I've already filed a while ago: Strip absolute --libdir paths from configure args in test_freeze #124916

stefanor · 2026-07-03T21:11:14Z

Seems like having another matrix will add 15–16 minutes of CI runtime.

It's part of the main test matrix, now.

rgommers

Thanks for working on this @stefanor! Overall this looks quite good to me - I only have a couple of small suggestions for robustness of the checks.

rgommers · 2026-07-08T13:50:20Z

    if internal:
        cflags.append('-DTEST_INTERNAL_C_API=1')

+    py_limited_api = limited or abi3t


This looks unrelated to the purpose of this PR, but is a correctness fix, so 👍🏼

It was necessary to tell setuptools to produce abi3/abi3t filenames.

webknjaz · 2026-07-08T14:14:28Z

+        description: OS to run the job
+        required: true
+        type: string


This is unrelated but is probably fine provided it's backported into the older branches.

Yeah, let's split this and the other unindent into another PR so we can more cleanly backport this, along with the other changes like:

- if: ${{ fromJSON(inputs.bolt-optimizations) }} + if: fromJSON(inputs.bolt-optimizations)

Which could maybe be just?

- if: ${{ fromJSON(inputs.bolt-optimizations) }} + if: inputs.bolt-optimizations

rgommers

All my comments were addressed and this LGTM now. Thanks @stefanor 🙏🏼

webknjaz · 2026-07-29T12:32:08Z

Looks like tiny timestamp differences are still happening here and there:

-   'date_time': (2026, 7, 29, 12, 1, 52),
?                                      ^

+   'date_time': (2026, 7, 29, 12, 1, 53),
?                                      ^

(https://github.com/python/cpython/actions/runs/30450012974/job/90569653484?pr=152831#step:20:9330)

pythonGH-122931 merged

webknjaz · 2026-07-29T13:42:34Z

        sudo apt-get install --no-install-recommends bolt-21
        echo PATH="$(llvm-config-21 --bindir):$PATH" >> $GITHUB_ENV
+    - name: Use the commit timestamp for timestamps the install
+      run: echo "SOURCE_DATE_EPOCH=$(git log -1 --format=%ct)" >> $GITHUB_ENV


Ideally, I'd put this as early as possible, right after the actions/checkout step since it's rather coupled.

Actually, that's causing the test failures that you mentioned earlier, so I'm going to move it later, after the tests.

hugovk · 2026-07-29T13:15:29Z

          "$BUILD_DIR/cross-python/bin/python3" -m test test_sysconfig test_site test_embed

+  linux-install-compare:
+    name: Ubuntu Co-install comparison


Suggested change

name: Ubuntu Co-install comparison

name: Ubuntu co-install comparison

hugovk · 2026-07-29T13:17:03Z

@@ -0,0 +1,3 @@
+CI Tests to ensure Debian `multi-arch co-installability
+<https://www.debian.org/doc/debian-policy/ch-controlfields.html#multi-arch>`_


Prefer anonymous Sphinx links, if we don't need to create a reference.

Suggested change

<https://www.debian.org/doc/debian-policy/ch-controlfields.html#multi-arch>`_

<https://www.debian.org/doc/debian-policy/ch-controlfields.html#multi-arch>`__

hugovk · 2026-07-30T10:50:02Z

+      debug:
+        description: Whether to create a Debug Build
+        required: false
+        type: boolean
+        default: true


We don't need a variable for this because we're not varying it:

Suggested change

debug:

description: Whether to create a Debug Build

required: false

type: boolean

default: true

hugovk · 2026-07-30T10:51:17Z

@@ -84,14 +102,15 @@ jobs:
        PROFILE_TASK='-m test --pgo --ignore test_unpickle_module_race'
        ../cpython-ro-srcdir/configure
        --config-cache


Re-instate:

Suggested change

--config-cache

--config-cache

--with-pydebug

hugovk · 2026-07-30T10:51:25Z

        --enable-slower-safety
        --enable-safety
        --with-openssl="$OPENSSL_DIR"
+        ${{ fromJSON(inputs.debug) && '--with-pydebug' || '' }}


Drop:

Suggested change

${{ fromJSON(inputs.debug) && '--with-pydebug' || '' }}

hugovk · 2026-07-30T11:35:20Z

+from typing import Any
+from pathlib import Path


Suggested change

from typing import Any

from pathlib import Path

from pathlib import Path

from typing import Any

hugovk · 2026-07-30T11:35:49Z

+from argparse import ArgumentParser
+from hashlib import file_digest
+from pathlib import Path
+from typing import Any, cast
+import gzip
+import json


Suggested change

from argparse import ArgumentParser

from hashlib import file_digest

from pathlib import Path

from typing import Any, cast

import gzip

import json

import gzip

import json

from argparse import ArgumentParser

from hashlib import file_digest

from pathlib import Path

from typing import Any, cast

hugovk · 2026-07-30T11:39:58Z

+
+
+def main() -> None:
+    p = ArgumentParser("Hash a python install for comparison later")


Suggested change

p = ArgumentParser("Hash a python install for comparison later")

p = ArgumentParser("Hash a Python install for comparison later")

hugovk · 2026-07-30T11:43:13Z

+    p.add_argument(
+        "-o",
+        "--output",
+        type=Path,
+        help="Output file (gzipped)",
+    )


This has no default: either make it required=True, make positional, or add a default. And let's list this after destdir.

hugovk · 2026-07-30T11:49:54Z

+    for platform, tagsets in tags_seen_by_platform.items():
+        if len(tagsets) >= 2:
+            break


platform never used:

Suggested change

for platform, tagsets in tags_seen_by_platform.items():

if len(tagsets) >= 2:

break

for tagsets in tags_seen_by_platform.values():

if len(tagsets) >= 2:

break

stefanor requested review from AA-Turner, JacobCoffee, ezio-melotti, hugovk, itamaro and webknjaz as code owners July 1, 2026 22:23

bedevere-app Bot mentioned this pull request Jul 1, 2026

Allow stable ABI extensions to include multiarch tuples in the filename #122931

Open

bedevere-app Bot added the awaiting review label Jul 1, 2026

stefanor force-pushed the ci-multiarch-co-install branch 2 times, most recently from a347830 to 3a52cb5 Compare July 1, 2026 22:38

stefanor added 9 commits July 1, 2026 18:39

In test_cext: Tell setuptools when we are generating a limited_api mo…

6a37083

…dule

Make debug builds configurable in reusable-ubuntu

15cd434

Tidy up YAML

db6e1a9

Make tests optional in reusable-ubuntu

216fae0

Allow reusable-ubuntu to install and upload results

f20a5b0

Matrix to run installs on Ubuntu

f059478

Compare multiple install results

2f5cd99

Skip comparing abi3 and abi3t for now

e6dd647

NEWS entry

b84e719

stefanor force-pushed the ci-multiarch-co-install branch from 3a52cb5 to b84e719 Compare July 1, 2026 22:39

Upload hashes rather than installs

3fce580