feat: add t-test mode for statistical significance testing #133
Conversation
| text += styleText(["blue", "bold"], `${result.timeFormatted} total time`); | ||
| } | ||
|
|
||
| // TODO: produce confidence on stddev |
There was a problem hiding this comment.
Should this todo be cleared?
| @@ -0,0 +1,218 @@ | |||
| // Welch's t-test implementation for benchmark comparison | |||
There was a problem hiding this comment.
This might warrant a bibliography entry or three. What reference materials did you use?
There was a problem hiding this comment.
I thought about including some references, but that's actually well spread out there. I think it's just a matter of context.
There was a problem hiding this comment.
Well, my challenge to you is to keep in the back of your head if you ever see a particularly good article or video to come back here and add a link.
Or maybe just the Wikipedia link if nothing else fits the bill.
- Add ttest option to Suite that automatically sets repeatSuite=30 - Implement Welch's t-test for comparing benchmark results - Display significance stars (*, **, ***) based on p-values - Add T-Test Mode indicator in reporter output - Update TypeScript definitions with ttest and ReporterOptions - Add comprehensive tests for t-test utilities - Add statistical-significance example demonstrating the feature - Update documentation with usage and interpretation guide The t-test compares 30 independent runs of each benchmark to determine if performance differences are statistically significant, helping identify real improvements vs. random variance. Signed-off-by: RafaelGSS <rafael.nunu@hotmail.com>
RafaelGSS
force-pushed
the
add-ttest-feature
branch
from
f10f65a to
a8c60b5
Compare
|
PTAL @jdmarshall |
|
PTAL @H4ad @jdmarshall - Planning to land it today. |
H4ad
left a comment
H4ad
left a comment
There was a problem hiding this comment.
It's the same implementation of the one we have for node? If so, great to have it here.
Pretty much it. But instead of comparing binaries, we compare benchmark.fn. |
|
Oops. LGTM |
|
Holy cow is this a lot slower. Should I be adjusting the run count? |
The t-test compares 30 independent runs of each benchmark to determine if performance differences are statistically significant, helping identify real improvements vs. random variance.