Skip to content

Conversation

@ahibrahimm
Copy link
Contributor

Description

Please add an informative description that covers that changes made by the pull request and link all relevant issues.

If an SDK is being regenerated based on a new API spec, a link to the pull request containing these API spec changes should be included above.

All SDK Contribution checklist:

  • The pull request does not introduce [breaking changes]
  • CHANGELOG is updated for new features, bug fixes or other significant changes.
  • I have read the contribution guidelines.

General Guidelines and Best Practices

  • Title of the pull request is clear and informative.
  • There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

  • Pull request includes test coverage for the included changes.

Copilot AI review requested due to automatic review settings December 8, 2025 16:47
@ahibrahimm ahibrahimm requested a review from a team as a code owner December 8, 2025 16:47
@github-actions github-actions bot added the Evaluation Issues related to the client library for Azure AI Evaluation label Dec 8, 2025
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes an issue where the success key was incorrectly nested inside the details object in the tool call success evaluator. The changes update both the prompt template to show the correct JSON structure and add fallback logic to handle cases where the LLM response still has success incorrectly placed in details.

Key Changes:

  • Corrected JSON formatting in the prompt template to move success key to the top level
  • Added fallback handling in the evaluator to extract success from details if needed
  • Improved JSON indentation consistency throughout the prompt examples

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
tool_call_success.prompty Fixed JSON structure in all examples to show success at the correct top level, removed trailing whitespace, and improved indentation consistency
_tool_call_success.py Added fallback logic to extract success key from details if it's incorrectly nested there

Comment on lines +165 to +168
details = llm_output.get('details', {})

if "success" not in llm_output and "success" in details:
success = details.get("success", False)
Copy link

Copilot AI Dec 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The variable details is extracted but not used elsewhere in the visible code. If details is not utilized beyond checking for the misplaced success key, consider removing the variable assignment and accessing llm_output.get('details', {}) directly in line 167 to avoid creating an unused variable.

Suggested change
details = llm_output.get('details', {})
if "success" not in llm_output and "success" in details:
success = details.get("success", False)
if "success" not in llm_output and "success" in llm_output.get('details', {}):
success = llm_output.get('details', {}).get("success", False)

Copilot uses AI. Check for mistakes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Evaluation Issues related to the client library for Azure AI Evaluation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants