Skip to content

Commit 54ab3fc

Browse files
authored
docs: add info on running offline to FAQ (#555)
Added FAQ section on offline usage with guidance for configuring local caching and pre-populating data --------- Signed-off-by: Marta Stepniewska-Dziubinska <[email protected]>
1 parent fc304a0 commit 54ab3fc

File tree

2 files changed

+25
-0
lines changed

2 files changed

+25
-0
lines changed

docs/references/faq.md

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -318,3 +318,25 @@ nemo-evaluator-launcher run --config-name your_config --dry-run
318318
```
319319

320320
Reference: {ref}`configuration-overview`.
321+
322+
---
323+
324+
## **Can I use Evaluator without internet access?**
325+
326+
Yes. NeMo Evaluator uses datasets and model checkpoints from [Hugging Face Hub](https://huggingface.co/docs/hub/en/index). If a requested dataset or model is not available locally, it is downloaded from the Hub at runtime.
327+
328+
When working in an environment without internet access, configure a cache directory and pre-populate it with all required data before launching the evaluation.
329+
330+
See the [example configuration](https://github.com/NVIDIA-NeMo/Evaluator/blob/main/packages/nemo-evaluator-launcher/examples/slurm_vllm_advanced_hf_caching.yaml) with HF caching:
331+
332+
```{literalinclude} ../../packages/nemo-evaluator-launcher/examples/slurm_vllm_advanced_hf_caching.yaml
333+
:language: yaml
334+
:start-after: "# [docs-start-snippet]"
335+
:end-before: "# [docs-end-snippet]"
336+
```
337+
338+
Modify the example with actual paths for the mounts and run:
339+
340+
```bash
341+
nemo-evaluator-launcher run --config packages/nemo-evaluator-launcher/examples/slurm_vllm_advanced_hf_caching.yaml
342+
```

packages/nemo-evaluator-launcher/examples/slurm_vllm_advanced_hf_caching.yaml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,7 @@
3131
# Results from such test runs should NEVER be used to compare models or
3232
# report benchmark performance.
3333

34+
# [docs-start-snippet]
3435
defaults:
3536
- execution: slurm/default
3637
- deployment: vllm
@@ -81,3 +82,5 @@ evaluation:
8182
tasks:
8283
- name: ifeval # chat benchmark will automatically use v1/chat/completions endpoint
8384
- name: gsm8k # completions benchmark will automatically use v1/completions endpoint
85+
86+
# [docs-end-snippet]

0 commit comments

Comments
 (0)