Skip to content

Commit 3d04478

Browse files
committed
doc: minor update
1 parent 098ee79 commit 3d04478

File tree

1 file changed

+4
-5
lines changed

1 file changed

+4
-5
lines changed

README.md

Lines changed: 4 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -91,8 +91,11 @@ pip install "git+https://github.com/bigcode-project/bigcodebench.git#egg=bigcode
9191

9292
We use the greedy decoding as an example to show how to evaluate the generated code samples via remote API.
9393

94+
> [!Note]
95+
>
96+
> Remotely executing on `BigCodeBench-Full` typically takes 6-7 minutes, and on `BigCodeBench-Hard` typically takes 4-5 minutes.
97+
9498
```bash
95-
# greedy decoding by default
9699
bigcodebench.evaluate \
97100
--model meta-llama/Meta-Llama-3.1-8B-Instruct \
98101
--split [complete|instruct] \
@@ -105,10 +108,6 @@ bigcodebench.evaluate \
105108
- The evaluation results will be stored in a file named `[model_name]--bigcodebench-[instruct|complete]--[backend]-[temp]-[n_samples]-sanitized_calibrated_eval_results.json`.
106109
- The pass@k results will be stored in a file named `[model_name]--bigcodebench-[instruct|complete]--[backend]-[temp]-[n_samples]-sanitized_calibrated_pass_at_k.json`.
107110

108-
> [!Note]
109-
>
110-
> Remotely executing on `BigCodeBench-Full` typically takes 5-7 minutes, and on `BigCodeBench-Hard` typically takes 3-5 minutes.
111-
112111
> [!Note]
113112
>
114113
> BigCodeBench uses different prompts for base and chat models.

0 commit comments

Comments
 (0)