pjin/hross/mt-verifiers-main-ray-dev-20251208 vs main #468
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Jiaqi Zeng <jiaqiz@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
- Add template_metadata field to MCQARunRequest for custom output regex - Implement _parse_answer_with_custom_regex function for flexible answer extraction - Update verify function to prioritize template_metadata.output_regex over grading_mode - Maintain full backward compatibility with existing grading modes - Support both letter matching and option text matching with custom regex patterns Signed-off-by: Pritam Gundecha <pgundecha@nvidia.com>
Signed-off-by: Pritam Gundecha <pgundecha@nvidia.com>
Signed-off-by: Pritam Gundecha <pgundecha@nvidia.com>
Add 6 test cases covering template_metadata feature including edge cases, priority handling, and fallback behavior. Update README with feature docs, examples, and fix incorrect input schema. Include example files with verified rollouts. Signed-off-by: Pritam Gundecha <pgundecha@nvidia.com>
Signed-off-by: Pritam Gundecha <pgundecha@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Jiaqi Zeng <jiaqiz@nvidia.com>
Signed-off-by: Jiaqi Zeng <jiaqiz@nvidia.com>
Signed-off-by: Hayley Ross <hross@nvidia.com>
Signed-off-by: Hayley Ross <hross@nvidia.com>
Signed-off-by: Hayley Ross <hross@nvidia.com>
Signed-off-by: Hayley Ross <hross@nvidia.com>
…atible with Python 3.12 Signed-off-by: Hayley Ross <hross@nvidia.com>
Signed-off-by: Hayley Ross <hross@nvidia.com>
Signed-off-by: Hayley Ross <hross@nvidia.com>
Signed-off-by: Hayley Ross <hross@nvidia.com>
Signed-off-by: Hayley Ross <hross@nvidia.com>
Signed-off-by: Hayley Ross <hross@nvidia.com>
…les and COMET cache path Signed-off-by: Hayley Ross <hross@nvidia.com>
Signed-off-by: Hayley Ross <hross@nvidia.com>
#193 --------- Signed-off-by: Sugam Devare <sdevare@nvidia.com>
Signed-off-by: Sugam Devare <sdevare@nvidia.com>
Signed-off-by: Jiaqi Zeng <jiaqiz@nvidia.com>
…ross/mt-verifiers
Signed-off-by: Hayley Ross <hross@nvidia.com>
…ics for BLEU and MetricX Signed-off-by: Hayley Ross <hross@nvidia.com>
Signed-off-by: Hayley Ross <hross@nvidia.com>
Signed-off-by: Hayley Ross <hross@nvidia.com>
Signed-off-by: Hayley Ross <hross@nvidia.com>
…oss/mt-verifiers/nemotron-ray-dev-20251208 Signed-off-by: Peter Jin <pjin@nvidia.com>
Signed-off-by: Peter Jin <pjin@nvidia.com>
Signed-off-by: Peter Jin <pjin@nvidia.com>
Signed-off-by: Peter Jin <pjin@nvidia.com>
Signed-off-by: Peter Jin <pjin@nvidia.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.