Skip to content

Commit 1ed699b

Browse files
committed
Add comparison to lingua-1b repro
Signed-off-by: Peter St. John <[email protected]>
1 parent 5f9f912 commit 1ed699b

File tree

4 files changed

+11
-6
lines changed

4 files changed

+11
-6
lines changed

bionemo-recipes/recipes/llama3_native_te/README.md

Lines changed: 11 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -46,16 +46,21 @@ Alternatively, the dependencies can be installed manually in an environment with
4646

4747
### Performance Benchmarks
4848

49-
![Llama 3 Lingua 1B Validation](../../../docs/docs/assets/images/recipes/llama3_lingua_1b_validation.png)
50-
51-
We compared the performance and convergence of this Llama3 recipe (with FSDP2) against NeMo 2.0 (https://github.com/NVIDIA-NeMo/NeMo)
52-
on the Lingua-1B dataset. See [Training on Natural Language Data (Lingua Reproduction)](#lingua-reproduction) for more
53-
details. The figure above shows similar loss convergence and step time to the NeMo 2.0 training example, and the
54-
following table shows downstream performance on various tasks using the
49+
<p align="center">
50+
<img src="../../../docs/docs/assets/images/recipes/lingua-1b-loss-curve.png" alt="Llama 3 Lingua 1B Loss Curve" width="49%" />
51+
<img src="../../../docs/docs/assets/images/recipes/lingua-1b-step-time.png" alt="Llama 3 Lingua 1B Step Time" width="49%" />
52+
</p>
53+
54+
We compared the performance and convergence of this Llama3 recipe (with FSDP2) against NeMo 2.0
55+
(https://github.com/NVIDIA-NeMo/NeMo) and the [facebookresearch/lingua](https://github.com/facebookresearch/lingua)
56+
implementation on the DCLM Baseline 1.0 dataset. See [Training on Natural Language Data (Lingua
57+
Reproduction)](#lingua-reproduction) for more details. The figure above shows similar loss convergence and step time to
58+
the NeMo 2.0 training example, and the following table shows downstream performance on various tasks using the
5559
[lm-eval](github.com/eleutherai/lm-evaluation-harness) library.
5660

5761
| name | arc_challenge | arc_easy | boolq | copa | hella_swag | piqa | winogrande |
5862
| ------------------- | ------------- | -------- | ----- | ---- | ---------- | ----- | ---------- |
63+
| Llama3 1B (Lingua) | 32 | 69.32 | 62.94 | 77 | 47.16 | 73.34 | 63.06 |
5964
| Llama3 1B (NeMo2) | 32.34 | 67 | 59.79 | 78 | 59.09 | 74.43 | 59.83 |
6065
| Llama3 1B (Recipes) | 30.63 | 65.57 | 55.6 | 80 | 44.57 | 73.29 | 58.72 |
6166

194 KB
Loading
189 KB
Loading
-39.4 KB
Binary file not shown.

0 commit comments

Comments
 (0)