OSX native OSX-Unzipper won't work with s3 streams and large archives do not work at all #116
Description
I am using the latest release to create zip files containing s3 objects.
None of the Archives are working with the native unzipper:
Small Archives (<4GB) produce the error "Error 1 - Operation not permitted"
Large Archives (>4GB) produce the error "Error 2 - No such file or directory"
When using "The Unarchiver" the small archives are working - the large ones don't (error: The Archive is incomplete)
Both Archives are working fine with the terminal command "unzip".
I also tried a different approach using "McNetic/PHPZipStreamer" which is used in OwnCloud/NextCloud. And while also NOT working with the Native Unzipper - files produced with this library are working in "The Unarchiver" no matter what size. Their workaround is to use .tar files in Unix-Environments. But that's even more confusing for users and also is throwing a warning in chrome.
I would much rather like to use ZipStream and get to the bottom of this issue.
Let me know if you need s3-object-keys for testing.
This is the code i use:
<?php
use ZipStream\ZipStream;
use Symfony\Component\HttpFoundation\StreamedResponse;
use ZipStream\Option\Archive as ArchiveOptions;
use ZipStream\Option\File as FileOptions;
use ZipStream\Option\Method;
ini_set('max_execution_time', 0);
set_time_limit(0);
header('X-Accel-Buffering: no');
$downloadCollection = DownloadCollection::where('token', $token)->first();
if($downloadCollection) {
$region = config('s3.region') ?: '';
// Get Laravel Disk Instance
$disk = Storage::disk($region);
$adapter = $disk->getAdapter();
$client = $adapter->getClient();
$client->registerStreamWrapper();
$response = new StreamedResponse(function() use($downloadCollection) {
$options = new ArchiveOptions();
$options->setSendHttpHeaders(true);
$options->setContentDisposition('attachment');
$options->setContentType('application/x-zip');
$options->setEnableZip64(true);
$options->setZeroHeader(true); // Required. Otherwise Download happens before Dialog appears and resulting zip is very small and corrupt
$options->setHttpHeaderCallback('header');
# create a new zipstream object
$zip = new ZipStream('example.zip', $options);
foreach ($downloadCollection->collection as $download) {
$key = $download['key'];
$context = stream_context_create(array('s3' => array('seekable' => true)));
// Open a stream in read-only mode
if (!($stream = fopen("s3://bucket/{$key}", 'r', false, $context))) {
throw new \Exception('Could not open stream for reading file: ['.$key.']');
}
$fileOptions = new FileOptions();
$fileOptions->setMethod(Method::STORE());
$fileOptions->setSize($download['size']);
$zip->addFileFromStream($download['name'], $stream, $fileOptions);
}
$zip->finish();
});
return $response->send();
}This is the result from "unzip -Zv 'zipstream over 5gb.zip'"
Archive: zipstream over 5gb.zip
There is no zipfile comment.
End-of-central-directory record:
-------------------------------
Zip archive file size: 6059547926 (00000001692D5D16h)
Actual end-cent-dir record offset: 6059547828 (00000001692D5CB4h)
Expected end-cent-dir record offset: 6059547828 (00000001692D5CB4h)
(based on the length of the central directory and its expected offset)
This zipfile constitutes the sole disk of a single-part archive; its
central directory contains 37 entries.
The central directory is 3769 (0000000000000EB9h) bytes long,
and its (expected) offset in bytes from the beginning of the zipfile
is 6059544059 (00000001692D4DFBh).
Central directory entry #1:
---------------------------
Testfilename-redacted.abc
offset of local header from start of archive: 0
(0000000000000000h) bytes
file system or operating system of origin: OS/2 or NT HPFS
version of encoding software: 0.3
minimum file system compatibility required: MS-DOS, OS/2 or NT FAT
minimum software version required to extract: 4.5
compression method: none (stored)
file security status: not encrypted
extended local header: yes
file last modified on (DOS date/time): 2019 May 17 17:51:34
32-bit CRC value (hex): 18e94143
compressed size: 577444062 bytes
uncompressed size: 577444062 bytes
length of filename: 43 characters
length of extra field: 0 bytes
length of file comment: 0 characters
disk number on which file begins: disk 1
apparent file type: binary
non-MSDOS external file attributes: 000000 hex
MS-DOS file attributes (20 hex): arc
There is no file comment.
.
.
.
And this is the result from the small archive:
Archive: example.zip
There is no zipfile comment.
End-of-central-directory record:
-------------------------------
Zip archive file size: 91441 (0000000000016531h)
Actual end-cent-dir record offset: 91419 (000000000001651Bh)
Expected end-cent-dir record offset: 91419 (000000000001651Bh)
(based on the length of the central directory and its expected offset)
This zipfile constitutes the sole disk of a single-part archive; its
central directory contains 1 entry.
The central directory is 76 (000000000000004Ch) bytes long,
and its (expected) offset in bytes from the beginning of the zipfile
is 91343 (00000000000164CFh).
Central directory entry #1:
---------------------------
filenamredacted.abc
offset of local header from start of archive: 0
(0000000000000000h) bytes
file system or operating system of origin: OS/2 or NT HPFS
version of encoding software: 0.3
minimum file system compatibility required: MS-DOS, OS/2 or NT FAT
minimum software version required to extract: 4.5
compression method: none (stored)
file security status: not encrypted
extended local header: yes
file last modified on (DOS date/time): 2019 May 18 09:24:20
32-bit CRC value (hex): 39518077
compressed size: 91239 bytes
uncompressed size: 91239 bytes
length of filename: 30 characters
length of extra field: 0 bytes
length of file comment: 0 characters
disk number on which file begins: disk 1
apparent file type: binary
non-MSDOS external file attributes: 000000 hex
MS-DOS file attributes (20 hex): arc
There is no file comment.