Usage with vitest browser breaks when using codspeed-runner

[JCQuintas]

Use Case

I know this is probably not a supported use-case

We want to benchmark the test in a real-browser environment to test closer to what the user will actually experience. Also, some of our planned tests would not work at all in a jsdom environment.

We looked at codspeed to help us detect performance changes between pull request, as to not slow down the MUI X Charts library unexpectedly.

Issue

Tests break when running them with codspeed-runner using vitest in browser mode with playwright

They break in the CodSpeedHQ/action, and when installing and running codspeed-runner manually. But they don't break if we run vitest bench

reproducible repo: https://github.com/JCQuintas/codspeed-mui-example
failed actions: https://github.com/JCQuintas/codspeed-mui-example/actions/runs/10074486926

[adriencaccia]

Hey @JCQuintas, thanks for the issue.

It seems that there will be a bit of work to support browser testing within CodSpeed, too bad that the error is not really helpful 😕.
I will need to investigate.

In the meantime, to still be able to leverage CodSpeed and detect performance changes, you can still set up CodSpeed using jsdom for tests where it is possible. I expect that performance changes will still be picked up in this simulated environment, even if the relative values will be different.