Skip to content

Conversation

@eregon
Copy link
Member

@eregon eregon commented Jan 15, 2026

  • Otherwise new failures do not get noticed by the PR breaking it.
  • Instead, just exclude the benchmarks known to fail.
  • Relates to 6d43a4c. There has been some failures which were due to the harness assuming Ractor is defined, this CI job helps catch such issues. There is also CRuby-specific code in run_benchmarks.rb, so this tests it's properly guarded.

I'd like to remove the continue-on-error for benchmarks on TruffleRuby in CI, otherwise any regression there will most likely be missed. A concrete example is #375 didn't notice it broke all benchmarks on TruffleRuby due to the continue-on-error. I fixed that #442.

If it fails again, and it's not an issue of the code in this repository then please @-mention me (@eregon) and:

  • If it's a single/few benchmarks then please add it to the --excludes list.
  • If it's many benchmarks or transient, please re-add continue-on-error: ${{ matrix.ruby == 'truffleruby' }} in the name: Test run_benchmarks.rb step.

Is that reasonable, WDYT?

These benchmarks run with ruby-version: truffleruby, so the latest TruffleRuby 33 release, so that's more stable than e.g. using truffleruby-head. That also contains an important fix for Bundler install, which prevented to make this PR before.

Independently of this I will run the same benchmarks in truffleruby/truffleruby's CI to make sure we don't regress there on master, and that way new TruffleRuby releases shouldn't break ruby-bench's CI.

* Otherwise new failures do not get noticed.
* Instead, just exclude the benchmarks known to fail.
* Relates to 6d43a4c.
  There has been a couple failures which were due to the harness
  assuming Ractor is defined, this CI job helps catch such issues.
  There is also CRuby-specific code in run_benchmarks.rb, so this
  tests it's properly guarded.
@eregon eregon requested a review from k0kubun January 15, 2026 11:31
Co-authored-by: Takashi Kokubun <takashikkbn@gmail.com>
@eregon eregon force-pushed the truffleruby-benchmark-ci branch from 82fff14 to 6016b63 Compare January 15, 2026 18:12
@eregon eregon merged commit ada5ca9 into ruby:main Jan 15, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants