BUG: read_hdf is not read-threadsafe

[jreback]

xref: #2397
xref #14263 (example)

I think that we should protect the file open/closes with a lock to avoid this problem.

[toobaz]

I agree!: although my code assumes only write access, and so it would require some fixing and integration, do you think the approach makes sense?

[jreback]

@toobaz your soln 'works', but that is not a recommended way to do it at all. HDF5 by definition should only every have a single writer.

This helps with the SWMR case: https://www.hdfgroup.org/HDF5/docNewFeatures/NewFeaturesSwmrDocs.html

all that said read_hdf is not thread-safe for reading I think because the file handle needs to be opened in the master thread.

Here's an example using a multi-proc threadpool; needs testing with a regular threadpool as well; this currently segfaults

import numpy as np
import pandas as pd
from pandas.util import testing as tm
from multiprocessing.pool import ThreadPool

path = 'test.hdf'
num_rows = 100000
num_tasks = 4

def make_df(num_rows=10000):

    df = pd.DataFrame(np.random.rand(num_rows, 5), columns=list('abcde'))
    df['foo'] = 'foo'
    df['bar'] = 'bar'
    df['baz'] = 'baz'
    df['date'] = pd.date_range('20000101 09:00:00',
                               periods=num_rows,
                               freq='s')
    df['int'] = np.arange(num_rows, dtype='int64')
    return df

print("writing df")
df = make_df(num_rows=num_rows)
df.to_hdf(path, 'df', format='table')

# single threaded
print("reading df - single threaded")
result = pd.read_hdf(path, 'df')
tm.assert_frame_equal(result, df)

# multip
def reader(arg):
    start, nrows = arg

    return pd.read_hdf(path,
                       'df',
                       mode='r',
                       start=start,
                       stop=start+nrows)

tasks = [
    (num_rows * i / num_tasks,
     num_rows / num_tasks) for i in range(num_tasks)
    ]

pool = ThreadPool(processes=num_tasks)

print("reading df - multi threaded")
results = pool.map(reader, tasks)
result = pd.concat(results)

tm.assert_frame_equal(result, df)

[dragonator4]

Yes this is a duplicate of #14692. I have a small contribution to this discussion:

def reader(arg):
    start, nrows = arg
    with pd.HDFStore(path, 'r') as store:
        df = pd.read_hdf(store,
                         'df',
                         mode='r',
                         start=int(start),
                         stop=int(start+nrows))
    return df

Works without any problems in the code example above. It seems that as long as multiple connections are made to the same store, and only one query is placed through those connections, there is no error. Looking at the example I provided in #14692 again, and also the original version of reader given by @jreback above, multiple queries from the same connection cause all problems.

The fix may not involve placing locks on the store. Perhaps allowing df.read_hdf or other variants of accessing data from a store in read mode to make as many connection as required is the fix...

[jtf621]

I found another example of the issue. See #15274.

Note the segfault failure rate depends on the presence or absence of compression on the hdf file.

[rbiswas4]

I am running into this issue on both OSX and Linux (centos), trying to parallelize using joblib. even when I am not trying any write process part of SWMR. Is there a different way for a readonly solution? Is this a pandas issue or a h5.c issue? Thanks.

In my application I am trying to read different groups on different processors, and running into this error.
I am using

pandas                    0.19.1
pytables                  3.4.2
joblib                    0.11

The code which works when n_jobs=1 and fails when n_jobs=2 is 2 or more.

store = pd.HDFStore('ddf_lcs.hdf', mode='r')
params = pd.read_hdf('ddf_params.hdf')

tileIDs = params.tileID.dropna().unique().astype(np.int)

def runGroup(i, store=store, minion_params=params):
      print('group ', i)
      if i is np.nan:
          return
      low = i - 0.5
      high = i + 0.5
      print('i, low and high are ', i, low, high)
      key = str(int(i))
      lcdf = store.select(key)
      print('this group has {0} rows and {1} unique values of var'.format(len(lcdf), lcdf.snid.unique().size))
Parallel(n_jobs=1)(delayed(runGroup)(t) for t in tileIDs[:10])
store.close()

The error message I get is :

/usr/local/miniconda/lib/python2.7/site-packages/tables/group.py:1213: UserWarning: problems loading leaf ``/282748/table``::

  HDF5 error back trace

  File "H5Dio.c", line 173, in H5Dread
    can't read data
  File "H5Dio.c", line 554, in H5D__read
    can't read data
  File "H5Dchunk.c", line 1856, in H5D__chunk_read
    error looking up chunk address
  File "H5Dchunk.c", line 2441, in H5D__chunk_lookup
    can't query chunk address
  File "H5Dbtree.c", line 998, in H5D__btree_idx_get_addr
    can't get chunk info
  File "H5B.c", line 340, in H5B_find
    unable to load B-tree node
  File "H5AC.c", line 1262, in H5AC_protect
    H5C_protect() failed.
  File "H5C.c", line 3574, in H5C_protect
    can't load entry
  File "H5C.c", line 7954, in H5C_load_entry
    unable to load entry
 File "H5Bcache.c", line 143, in H5B__load
    wrong B-tree signature

End of HDF5 error back trace
...