More information

[fenollp]

Great initiative!

As I get it from your code these are all training times, right?
If that is the case could it be mentioned somewhere?
I think also adding percentages (fastest: 100%, then other lower %s)

Also, what about adding "application times"? ie: for each framework, benchmark the time it takes to apply the trained data on some input data (probably from the test set).

Also, time is nice but what about memory/IOps/CPU GPU%/...

[aizvorski]

@fenollp Thank you!

Yes these are all training times (forward + backward + weights update). Some other benchmarks only quote the forward+backward pass. I will change the description to make that clearer.

It would be nice to have "application times" but I wasn't focusing on that initially.

Memory, cpu, GPU% are all interesting. I have GPU memory % and GPU compute % results in the results/*dmon files, but I haven't made them into a table yet. All of these frameworks in practice seem to maximize use of the GPU compute, but they use very different amounts of memory.