fix: change time check

terryyz · terryyz · commit 1fc9b347c28c · 2024-06-18T20:34:13.000+08:00
diff --git a/bigcodebench/evaluate.py b/bigcodebench/evaluate.py
@@ -152,7 +152,7 @@ def evaluate(flags):
                     solution,
                     sample["_identifier"],
                     flags.min_time_limit,
-                    expected_time[task_id] if not flags.no_gt else 20
+                    expected_time[task_id] if not expected_time else 20
                 )
                 futures.append(executor.submit(check_correctness, *args))
                 completion_id[task_id] += 1

Original file line number	Diff line number	Diff line change
`@@ -152,7 +152,7 @@ def evaluate(flags):`
`152`	`152`	`solution,`
`153`	`153`	`sample["_identifier"],`
`154`	`154`	`flags.min_time_limit,`
`155`		`- expected_time[task_id] if not flags.no_gt else 20`
	`155`	`+ expected_time[task_id] if not expected_time else 20`
`156`	`156`	`)`
`157`	`157`	`futures.append(executor.submit(check_correctness, *args))`
`158`	`158`	`completion_id[task_id] += 1`