Enable latest versions of compilers on main

[neatudarius]

Enable latest versions of compilers on main

• Enable gcc-15
• Enable gcc-14
• Enable gcc-13
• Enable gcc-12
• Enable clang-20
• Enable clang-19
• Enable clang-18
• Enable clang-17
• Enable clang-16
• TBD msvc

[wusatosi]

Are you planning to shove all these into the CI?

We can create a nightly CI job for this, what will the build parameters be?
Note that building this with current version matrix (17,20,23,26) will result in a 40 job matrix.

[neatudarius]

Well, I think we need to discuss.

We have an action from our meetings (1-2 months ago) to define a matrix for environments.

It's absolutely cool that we have existing matrix, which I understand it just run "default " version of gcc/clang, but we also need to make sure we have support on a wide range of compiler version, in my opinion, but it depends on execution time. What's the total execution time right now for a single version of gcc+clang+msvc? What it will be if we add 3 extra versions for each compiler? (They should run in parallel, so maybe not a big overhead after spawn.)

Example latest 3-4 compiler version for gcc clang, msvc. CC: @steve-downey

To be honest, I don't actually care about the number of jobs, I do care about the total execution time on CI. Related to the nightly job, in my opinion we would want to run something more often.

@camio , do we pay a subscription for GitHub Actions? Do we pay extra if having multiple jobs?

[wusatosi]

You can actually see the time spent for actions in this tab: link.

Currently each job (one combination of the matrix, basically per compile command variant per C++ version per compiler) takes 29s to run, setup time is usually ~5s.
With 28 jobs, it takes GitHub Actions ~1 min - ~2 mins to run them, we are limited by queuing time (GitHub load) + parallelization.

I do want to point out that we do not have a lot to test in this test case, 29s could potentially be higher if the library consumer is developing complex library (longer compile time)/ library with timeout (longer testing time). Also we are going to test more platforms (macos, windows), this would dramatically increase the number of combinations.

I think we are on free tier of GitHub so our max parallelization is 20 jobs, see: here.
I believe we are not paying GitHub, open source project gets free tire.

My point is, I don't think it will be reasonable to trigger another matrix of (per compiler version per c++ version potentially per compile option variant per platform) on each commit, maybe only trigger (on main branch + in pr) / nightly.

Though realistically, it will probably take ~5-10 mins to run our current matrix + macos matrix + compiler matrix (assume one per version per compiler), this is definitely not horrible but not desirable as well.

[ClausKlein]

A real project takes about 10 min on CI, only 2 compiler, but with all sanitizer!
see https://github.com/beman-project/execution26/actions

[neatudarius]

A real project takes about 10 min on CI, only 2 compiler, but with all sanitizer! see https://github.com/beman-project/execution26/actions

Indeed, but let's keep a note this is a set of experimental libraries (which should become stable), which need to be tested on multiple platforms and compilers! It's not a production-like project, where you develop features for a product/platform etc and usually have a small set of platforms to support. This is actually about creating performant, portable, scalable C++ standard, which should be compliant at some point with an actual C++ Standard and should run with any environment. (TBD what that means - ofc, it's a finit set of environments, but a big/relevant one!)

TLDR: I think all ideas are good, but we need to recheck/redefine our goals (e.g., maybe we decide that we MUST or MUST NOT use multiple compilers when running tests on main), and then see what's an optimal implementation for these goals.

[camio]

@camio , do we pay a subscription for GitHub Actions? Do we pay extra if having multiple jobs?

We're not currently on a paid plan.

[steve-downey]

The CI shouldn't be a blocker for the normal dev cycle, so a few minutes doesn't seem unreasonable to run a full matrix periodically or on a PR.
Exemplar isn't a good project to use to decide on what the support matrix should be. It's far too simple. At a policy level minimum is probably more like trunk on one major compiler?
As a practical matter I think starting with current release compilers and C++23 and seeing how far back the library can be ported, then deciding what to do when it becomes difficult. Supporting standard library components is far more challenging with pre-concepts C++ is much more difficult and requires very different implementation techniques which may only approximate the specification. Deducing this adds an additional set of capabilities that are difficult to emulate.
LLVM makes tool chains available from their apt repository, and script to install the compilers from there. It looks like it's currently supporting back to 17, and through the unreleased 20.
There are PPAs for Ubuntu for GCC for newer than OS compilers. Be careful with them as they will destabilize a system, but are fine for isolated use in a container. We should be able to add support for GCC 15 that way, which is the current trunk version as of now.
It is definitely worth while running a range of language versions as there are occasional changes that cause incompatibilities. Finding those early and providing feedback to vendors and WG21 would be welcome.

[neatudarius]

I will try to consider all mentioned ideas and to come up with a solution for bemanproject/beman#34

[wusatosi]

Yeah looking back, I don't think big matrix is not that big of problem after all, I just don't want to let developer wait 10min/30min/1hr for a commit to clear CI : ( . Worked with a project with a 1hr30min pipeline and I got PTSD.

[steve-downey]

I have a project with a current pipeline of just under 3 days. I share your concern.

[neatudarius]

Yeah looking back, I don't think big matrix is not that big of problem after all, I just don't want to let developer wait 10min/30min/1hr for a commit to clear CI : ( . Worked with a project with a 1hr30min pipeline and I got PTSD.

I'm sure we can tweak the frequency. Agree, not want to take too many minutes.

[wusatosi]

I have a project with a current pipeline of just under 3 days. I share your concern.

omg I feel sorry for you.

[wusatosi]

Yeah looking back, I don't think big matrix is not that big of problem after all, I just don't want to let developer wait 10min/30min/1hr for a commit to clear CI : ( . Worked with a project with a 1hr30min pipeline and I got PTSD.

I'm sure we can tweak the frequency. Agree, not want to take too many minutes.

Yeah, let's keep the CI test associated per commit minimal small not take more than 10mins.

They say, while fully expecting the CI pipeline to blow up with macos and windows tests.

[wusatosi]

Hey @neatudarius since it looks like you've already added this feature in other repos, do you mind pushing a PR to exemplar first? I feel like that would help us evaluate the impact of this matrix a lot better.

[steve-downey]

I think we have enough evidence that if it's not tested it doesn't work? It looks to me that CI time is probably dominantly setting up the VM and installing compilers. Running the tests after successfully compiling is almost free. If the costs of CI get prohibitive, like if we get above 5 minutes for everything in parallel?, we should start considering trimming the matrix. But evidence first, then work on a fix. We probably don't need, for the packages we have, matrixing OS and compilers. Like GCC versions on Windows. Although we might consider that for something like networking, where the environment is probably significant.

…

On Tue, Nov 12, 2024, 06:58 Darius Neațu ***@***.***> wrote: A real project takes about 10 min on *CI*, only 2 compiler, but with all sanitizer! see https://github.com/beman-project/execution26/actions Indeed, but let's keep a note this is a set of experimental libraries (which should become stable), which need to be tested on multiple platforms and compilers! It's not a production-like project, where you develop features for a product/platform etc and usually have a small set of platforms to support. This is actually about creating performant, portable, scalable C++ standard, which should be compliant at some point with an actual C++ Standard and should run with any environment. (TBD what that means - ofc, it's a finit set of environments, but a big/relevant one!) TLDR: I think all ideas are good, but we need to recheck/redefine our goals (e.g., maybe we decide that we MUST or MUST NOT use multiple compilers when running tests on main), and then see what's an optimal implementation for these goals. — Reply to this email directly, view it on GitHub <#73 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAVNZ5TIE4A7QEC7DPNGVBT2AHUOXAVCNFSM6AAAAABRSKCXYGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINZQGM2DEOJXGE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[wusatosi]

This does sound like a valid use case for custom action image.
We can optimize out the installing compiler part by caching and using a custom docker image for actions.
Given we are getting more and more projects, this could be potentially beneficial.

I am assuming main unit test is an independent CI set from this.

Anyway we can try to set something up first and evaluate if we need to create docker image specifically for compiler check later.