Convergence test and validation experiment Buildkite pipeline #1223
Conversation
|
Quite concerning that some of the convergence tests do not pass on the GPU... This is probably due to #1170 since the simpler convergence tests that do not rely on a pressure solver seem to pass (also the fact that it passes forced flow free slip with doubly periodic (x, z) but not with a wall-bounded dimension (x, y)). Hmmm, but the Taylor-Green one is doubly periodic...
|
There was a problem hiding this comment.
This is great. Some minor comments:
-
test/test_convergence.jlis rather boiler-plate-y. Is there any way to use a list of convergence tests scripts? This will make it easier to add new tests in the future... -
Could make sense to raise an issue about converting convergence tests plots to
Plots.jlorCairoMakie.jlor whatever when this PR is merged, so we don't have to wrestlePyPlotin the test scripts.
Yeah it could be more automated. I'll give it a try once the tests are all passing.
I don't think it's worth spending much effort on switching plotting libraries (especially since the plots work now) but I'll use CairoMakie.jl for future plots. CairoMakie.jl doesn't support log axes yet (gotta hack it in for now) but hopefully it'll get added soon. |
|
Some tests fail.. |
|
All tests should pass (see https://buildkite.com/clima/oceananigans-validation-experiments/builds/96) but Tartarus went down so a lot of builds died leading to failing tests. |
| xlabel(L"N_x") | ||
| ylabel("\$L\$-norms of \$ | c_\\mathrm{sim} - c_\\mathrm{analytical} |\$") | ||
| removespines("top", "right") | ||
| legend = legend(loc="upper right", bbox_to_anchor=(1.4, 1.0), prop=Dict(:size=>6)) | ||
| lgd = legend(loc="upper right", bbox_to_anchor=(1.4, 1.0), prop=Dict(:size=>6)) |
|
44 successful checks 🚀 I think this PR is ready to be merged. All the existing convergence tests are now in CI and pass, and the plots are uploaded as Buildkite artifacts. This PR might conflict with @francispoulin's PR #1276 (I'm happy to fix conflicts no matter which PR is merged first). There are a few tests that take a long time. This can be shortened in a future PR as it might involve some trial and error and fiddling with rate of convergence tolerances. Other validation experiments should also be added. Right now I have set the validation pipeline to only run on master (and every night at 3am ET). |
Please go ahead and don't wait for me. I will work around whatever you have done here in my PR. |
This PR finally sets up a CI pipeline to run convergence tests and validation experiments for CPU and GPU on Buildkite.
Don't think this should run on every push like the main pipeline does and I couldn't figure out how to trigger it via a GitHub comment (see CliMA/slurm-buildkite#13).
We can trigger this pipeline manually from Buildkite and I've scheduled it to run every night at 3am EST (on the master branch).
Cool thing is that it uploads the convergence plots as artifacts so we can view them from Buildkite!
Resolves #1216