zarr slower than npy, hdf5 etc?

[nschloe]

I got interested in the performance of zarr and did a comparison with npy, pickle, hdf5 etc. See https://stackoverflow.com/a/58942584/353337. To my surprise, I found zarr reads large arrays slower than npy. This is for random float data as well as more structured mesh data. I had expected zarr to take the cake using multiple cores. Perhaps this isn't a good test for zarr to show its strength either.

Code to reproduce the plot: https://gist.github.com/nschloe/3d3b1adb9ce9e2d68d1c2d1a23ffa06d

[alimanfoo]

Thanks for posting, interesting. Couple of factors may be at work here. In the read test you use zarr in a zip file. IIUC reading from zip adds some overhead because the CRCs are checked. I'd suggest comparing with zarr stored as files in a directory (as you do for the write test). Also when comparing zarr versus h5, if you wanted to be totally fair you'd need to use the same chunk sizes and the same compressor. In your benchmark you don't explicitly set the chunk size, so zarr and h5py will guess a chunk size from the data size, and both use the same algorithm for guessing chunk size but zarr is tuned to guess larger chunks. This may not make a difference in your benchmark but thought I'd mention. Also you don't explicitly set the compressor, and h5py and zarr have different defaults. Again may not make a difference here, but worth noting.

…

On Tue, 19 Nov 2019, 21:48 Nico Schlömer, ***@***.***> wrote: I got interested in the performance of zarr and did a comparison with npy, pickle, hdf5 etc. See https://stackoverflow.com/a/58942584/353337. To my surprise, I found zarr reads large arrays with random float data slower than npy. I had expected zarr to take the cake using multiple cores. Perhaps this isn't a good test for zarr to show its strength either. [image: hloNX] <https://user-images.githubusercontent.com/181628/69189352-7e0e1d00-0b1e-11ea-8387-8bfd37a59d77.png> — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#519?email_source=notifications&email_token=AAFLYQQL7N5CWMBCHYVFRF3QURNMLA5CNFSM4JPJ3YC2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4H2PIFJA>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFLYQQIYDYQH6Y6HZQMWD3QURNMLANCNFSM4JPJ3YCQ> .

[nschloe]

I've updated the original post to include zipped zarr vs directory zarr; same results.

I suppose it must be the default for the compressor then.

[vigji]

I am doing a similar testing and I was trying to understand whether in zarr it is possible to read partially the file to get only some slices of the array. If I compare performance on partial retrieval with hdf5 files, where I know this is possible, I get a much longer time to read small chunks of the data (smaller than the zarr chunk size), and this seem to come from the fact that zarr reads the entire chunk to return part of it. Is there any way to get a partial file reading in zarr?

[jakirkham]

@nschloe, are these using the same compressor and chunk size? It seems like they may not be, which could cause huge variability.

[nschloe]

@jakirkham I don't know; I just used the default values. See the code for how to reproduce the plot.

[constantinpape]

I don't know; I just used the default values.

Afaik, zarr uses blosc compression by default. h5py does not compress by default.
Also, h5py does not chunk the data if you don't specify chunks=True (or enable compression).
Numpy and pickle neither compress nor chunk, I don't know about pytables.
So the comparison is not very fair.

FWIW when I benchmarked z5, which implements the zarr spec bin C++, I found the performance
on par with hdf5 in single-threaded performance and better mult-threaded. Unfortunately I don't have the results right now, the code is here.

[nschloe]

Can I adjust chunking and compression in zarr.save? Can I enable multithreading in Python?

[constantinpape]

Can I adjust chunking and compression in zarr.save?

You will need to use zarr.save_array instead and call it like this:

zarr.save_array('out.zr', data, chunks=chunks, compressor=compressor)

Chunks needs to be a tuple with the chunk size you want. Compressor can either be None (= no compression) or one of the compressors from numcodecs.

Can I enable multithreading in Python?

As far as I am aware there is no convenience function to read/write multi-threaded in zarr-python and you will need to implement this yourself. (I might be wrong about this though, zarr-python has grown quite a bit since I looked at it in detail for the last time ....)

[rabernat]

To get multithreading in python, use zarr together with dask. That’s the recommended way to go.

…

Sent from my iPhone

On Nov 23, 2019, at 10:05 AM, Constantin Pape ***@***.***> wrote:  Can I adjust chunking and compression in zarr.save? You will need to use zarr.save_array instead and call it like this: zarr.save_array('out.zr', data, chunks=chunks, compressor=compressor) Chunks needs to be a tuple with the chunk size you want. Compressor can either be None (= no compression) or one of the compressors from numcodecs. Can I enable multithreading in Python? As far as I am aware there is no convenience function to read/write multi-threaded in zarr-python and you will need to implement this yourself. (I might be wrong about this though, zarr-python has grown quite a bit since I looked at it in detail for the last time ....) — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[nschloe]

use zarr together with dask

I see. I suppose you're using dask arrays then, which is perhaps easy to do since they support memory segmentation with their blocked arrays.

In my opinion, it's still worthwhile exploring the possibility of using multithreaded reads into numpy arrays. Those are the de-facto standard, every Python programmer knows them and lots of code is optimized for it. It should be possible, too: You know in advance how large the block in memory will be, and where each chunk will go.

[rabernat]

If you don’t mind the dask dependency, it would be simple to just use dask for the reading step. If you call dask.array.from_zarr and then coerce immediately to a numpy array, you will effectively get what you’re looking for.

…

Sent from my iPhone

On Nov 24, 2019, at 1:29 PM, Nico Schlömer ***@***.***> wrote:  use zarr together with dask I see. I suppose you're using dask arrays then, which is perhaps easy to do since they support memory segmentation with their blocked arrays. In my opinion, it's still worthwhile exploring the possibility of using multithreaded reads into numpy arrays. Those are the de-facto standard, every Python programmer knows them and lots of code is optimized for it. It should be possible, too: You know in advance how large the block in memory will be, and where each chunk will go. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

[jakirkham]

Would add I’m not aware of NumPy or pickle doing multithreaded reads or writes. Though am not sure if we are still discussing benchmarking.

[nschloe]

Would add I’m not aware of NumPy or pickle doing multithreaded reads or writes. Though am not sure if we are still discussing benchmarking.

They don't and yes. I want to see zarr benchmarks which show file sizes smaller than pickle/npy (they already are) and (almost) equally fast reads (not possible without multithreading it seems).

[jakirkham]

Well they are also writing a single file. I suppose one could have a Zarr file with a single chunk.

Anyways I think before we jump to conclusions we need a good benchmark. The first pass gave us a starting point albeit there are issues with it as pointed out. Can we do a second pass that integrates this feedback?

[nschloe]

Okay, well compression does make a difference after all, at least with file sizes. In particular with structured data, zarr is way ahead of the competition. The compression also explains the slower read times.

In particular with structured data, zarr is way ahead of the competition.

import os
import pickle
from pathlib import Path

import h5py
import matplotlib.pyplot as plt
import matplotx
import meshzoo
import numpy as np
import tables
import zarr


def save(data):
    np.save("out.npy", data)
    #
    f = h5py.File("out.h5", "w")
    f.create_dataset("data", data=data)
    f.close()
    #
    with open("out.pkl", "wb") as f:
        pickle.dump(data, f)
    #
    f = tables.open_file("pytables.h5", mode="w")
    gcolumns = f.create_group(f.root, "columns", "data")
    f.create_array(gcolumns, "data", data, "data")
    f.close()
    #
    zarr.save("out.zip", data)
    zarr.save("out.zarr", data)


def check_file_sizes(get_data):
    file_sizes = []
    mem_sizes = []
    for n in [2**k for k in range(24)]:
        print(n)
        data = get_data(n)
        mem_sizes.append(data.nbytes)
        save(data)
        file_sizes.append(
            [
                Path("out.npy").stat().st_size,
                Path("out.h5").stat().st_size,
                Path("out.pkl").stat().st_size,
                Path("pytables.h5").stat().st_size,
                Path("out.zip").stat().st_size,
                sum(
                    f.stat().st_size
                    for f in Path("out.zarr").glob("**/*")
                    if f.is_file()
                ),
            ]
        )

    labels = ["npy", "h5", "pickle", "pytables", "zarr zip", "zarr dir"]

    file_sizes = np.array(file_sizes).T

    # sort by last entry
    idx = np.argsort(file_sizes[:, -1])[::-1]
    file_sizes = [file_sizes[i] for i in idx]
    labels = [labels[i] for i in idx]

    with plt.style.context(matplotx.styles.dufte):
        for fs, label in zip(file_sizes, labels):
            plt.loglog(mem_sizes, fs, label=label)

        plt.xlabel("size in memory [B]")
        plt.title(f"file sizes ({get_data.name})")
        matplotx.ylabel_top("filesize [B]")
        matplotx.line_labels()

        r = "-".join(get_data.name.split(" "))
        plt.savefig(f"filesizes-{r}.png", transparent=True, bbox_inches="tight")
        # plt.show()
        plt.close()


def get_data_random_int(n):
    get_data_random_int.name = "random int"
    return np.random.randint(0, 1000, size=n)


def get_data_random_float(n):
    get_data_random_float.name = "random float"
    return np.random.rand(n)


def get_data_structured_int(n):
    get_data_structured_int.name = "structured int"
    points, cells = meshzoo.disk(6, int(np.sqrt(n)))
    return cells


def get_data_structured_float(n):
    get_data_structured_float.name = "structured float"
    points, cells = meshzoo.disk(6, int(np.sqrt(n)))
    return points


for f in [
    get_data_random_int,
    get_data_random_float,
    get_data_structured_int,
    get_data_structured_float,
]:
    check_file_sizes(f)