Merge pull request #1085 from horsto/s3_chunked

When retrieving data from S3, account for chunking

Author: dimitri-yatsenko

CHANGELOG.md

@@ -5,6 +5,7 @@
 - Fixed - `.ipynb` output in tutorials is not visible in dark mode ([#1078](https://github.com/datajoint/datajoint-python/issues/1078)) PR [#1080](https://github.com/datajoint/datajoint-python/pull/1080)
 - Changed - Readme to update links and include example pipeline image
 - Changed - Docs to add landing page and update navigation
+- Changed - `.data` method to `.stream` in the `get()` method for S3 (external) objects PR [#1085](https://github.com/datajoint/datajoint-python/pull/1085)
 - Fixed - Docs to rename `create_virtual_module` to `VirtualModule`
 
 ### 0.14.0 -- Feb 13, 2023

datajoint/s3.py

@@ -68,7 +68,9 @@ def fput(self, local_file, name, metadata=None):
     def get(self, name):
         logger.debug("get: {}:{}".format(self.bucket, name))
         try:
-            return self.client.get_object(self.bucket, str(name)).data
+            with self.client.get_object(self.bucket, str(name)) as result:
+                data = [d for d in result.stream()]
+            return b"".join(data)
         except minio.error.S3Error as e:
             if e.code == "NoSuchKey":
                 raise errors.MissingExternalFile("Missing s3 key %s" % name)