Update AXLearn performance script #1593
Conversation
| f"\n=== Final metrics ===\n" | ||
| f"Tokens/s/device: {tokens_per_sec_gpu.mean()} +/- {tokens_per_sec_gpu.std()}\n" | ||
| f"Seqs/s/device: {seqs_per_sec_gpu.mean()} +/- {seqs_per_sec_gpu.std()}\n" | ||
| f"AvgTimestep: {times_arr.mean()} +/- {times_arr.std()}\n" |
There was a problem hiding this comment.
A bit confused, what's Timestep?
There was a problem hiding this comment.
Hey @gpupuck
Timestep the name is a bit misleading, it comes directly from AXLearn. It's how many seconds we take to perform a single training step.
BTW I was thinking of saving those metrics into a specific metric.log file, rather than writing them directly in the output log. WDYT?
There was a problem hiding this comment.
I think printing the metrics to stdout is good enough. It's easier to process the log if all the metrics are there.
There was a problem hiding this comment.
That's true
However, I am now running some tests on cloud and I need to save those metrics in a storage, to check them after the model has run
There was a problem hiding this comment.
Just do both? Print human-readable to stdout, dump machine-readable (json?) values to a separate file.
There was a problem hiding this comment.
You're right Olli
I ended up to that :) no need to change the script
Add the metrics to the output log file