feat: expand eval dataset with edge and complex cases and refine prompts #458
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Implement Isolated `TestRig` for environment-safe, concurrent evaluations. - Add gold-standard datasets for Issue Triage, Scheduled Triage, Assistant, and Issue Fixer. - Implement Mock MCP Server for high-fidelity PR Review benchmarking. - Add nightly evaluation workflow with multi-model strategy matrix. - Automated aggregate reporting for GitHub Job Summaries. Next Steps: - Expand evaluation datasets with more edge cases. - Fine-tune workflow prompts based on baseline quality analysis. Refs: #219
- Added 30+ cases (edge, complex, real-life) across gemini-triage, gemini-issue-fixer, and gemini-review. - Refined triage prompt to handle spam, ambiguity, and vague reports more robustly. - Added a validation step to issue-fixer prompt to handle impossible or out-of-scope requests. - Updated mock MCP server to support new evaluation scenarios including race conditions and architectural violations. - Improved evaluation scripts for better tool call detection in namespaces. - Verified all evaluations pass with the updated prompts.
cocosheng-g
requested review from
MJjainam,
R2wenD2,
bdmorgan and
verbanicm
and removed request for
a team
Contributor
|
🤖 Hi @cocosheng-g, I've received your request, and I'm working on it now! You can track my progress in the logs for more details. |
- Update triage guidelines for stricter handling of spam and ambiguity. - Refine fixer validation step to use explicit keywords for out-of-scope cases. - Improves evaluation pass rates for edge cases.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR continues the work on issue #219 by expanding the evaluation datasets and refining the workflow prompts.
📊 Evaluation Results (Post-Tuning)
Changes:
Expanded Evaluation Datasets: Added 30+ edge, complex, and real-life cases across triage, fixer, and pr-review.
Prompt Refinements:
Verification: All evaluations have been verified to pass.