Skip to content

chore: better error message when writing TBaskets that are too large #1415

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ariostas merged 5 commits into from
Apr 9, 2025
Merged

chore: better error message when writing TBaskets that are too large #1415

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
df14a52
Better error message for TBaskets that are too large
ariostas Apr 7, 2025
304b8f1
Mention max size in the message
ariostas Apr 8, 2025
e8ee610
Merge branch 'main' into ariostas/tbasket_error_message
ariostas Apr 8, 2025
20d8fb1
Define a constant for max TBasket size
ariostas Apr 9, 2025
54c3917
Merge branch 'main' into ariostas/tbasket_error_message
ariostas Apr 9, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions src/uproot/const.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@
# determines when a file is "big"
kStartBigFile = 2000000000

kMaxTBasketBytes = numpy.iinfo(numpy.int32).max

# used in unmarshaling
kByteCountMask = numpy.int64(0x40000000)
kByteCountVMask = numpy.int64(0x4000)
Expand Down
21 changes: 21 additions & 0 deletions src/uproot/writing/_cascadetree.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1395,6 +1395,13 @@ def write_np_basket(self, sink, branch_name, compression, array):
fObjlen = len(uncompressed_data)
fNbytes = fKeylen + len(compressed_data)

if max(fObjlen, fNbytes) > uproot.const.kMaxTBasketBytes:
raise ValueError(
f"Numpy array data of branch {branch_name} has an uncompressed size of {fObjlen} bytes "
f"and a compressed size of {fNbytes} bytes, which is too large to fit in a TBasket. "
"Neither of these sizes can exceed 2 GiB."
)

parent_location = self._directory.key.location # FIXME: is this correct?

location = self._freesegments.allocate(fNbytes, dry_run=False)
Expand Down Expand Up @@ -1470,6 +1477,13 @@ def write_jagged_basket(self, sink, branch_name, compression, array, offsets):
fObjlen = len(uncompressed_data)
fNbytes = fKeylen + len(compressed_data)

if max(fObjlen, fNbytes) > uproot.const.kMaxTBasketBytes:
raise ValueError(
f"Jagged array data of branch {branch_name} has an uncompressed size of {fObjlen} bytes "
f"and a compressed size of {fNbytes} bytes, which is too large to fit in a TBasket. "
"Neither of these sizes can exceed 2 GiB."
)

parent_location = self._directory.key.location # FIXME: is this correct?

location = self._freesegments.allocate(fNbytes, dry_run=False)
Expand Down Expand Up @@ -1559,6 +1573,13 @@ def write_string_basket(self, sink, branch_name, compression, array, offsets):
fObjlen = len(uncompressed_data)
fNbytes = fKeylen + len(compressed_data)

if max(fObjlen, fNbytes) > uproot.const.kMaxTBasketBytes:
raise ValueError(
f"String data of branch {branch_name} has an uncompressed size of {fObjlen} bytes "
f"and a compressed size of {fNbytes} bytes, which is too large to fit in a TBasket. "
"Neither of these sizes can exceed 2 GiB."
)

parent_location = self._directory.key.location # FIXME: is this correct?

location = self._freesegments.allocate(fNbytes, dry_run=False)
Expand Down