Expose an API to get Resumable Upload Session URL

[tsawada]

I think it is a common practice to initiate resumable uploads on the server side, and hand the session URL to clients.
However, it seems that java-storage does not expose any methods to get the session URL for a resumable upload.

Describe alternatives you've considered
One can use Signed URL with single-request upload. However, for large files, resumable upload is preferable.

https://cloud.google.com/storage/docs/access-control/signed-urls#signing-resumable

[BenWhitehead]

Rather than directly exposing resumable upload URLs, we recommend using signed urls, and resumable upload signed URLs can be generated, and even used by this client via Storage#writer(URL).

We have an integration test that does just that:

java-storage/google-cloud-storage/src/test/java/com/google/cloud/storage/it/ITSignedUrlTest.java

Line 232 in 4f85af9

public void testUploadUsingSignedURL() throws Exception {

[tsawada]

@BenWhitehead Thank you for your reply.

Could you please share why you recommend using signed URLs? The documentation link I included in the description advises against using signed URLs.

[Using session URLs] avoids the complexity of having the server create a signed URL, as well as the complexity of having the client handle the signed URL and initiate the resumable upload.

In addition to the client-side complexity mentioned above, we have a few other reasons to prefer not using signed URLs:

• Signed URLs are not always available. Your credentials must either have a private key locally or be a ServiceAccountCredentials with iam.serviceAccounts.signBlob permission. Especially when developing locally using UserCredentials, this criterion is not met.
• Signed URLs can only be used with XML APIs, which require the bucket to have proper CORS settings when used from browsers. With the session URLs, we can use JSON APIs, which do not require CORS settings.
• Signed URLs expose the email address of the service account being used to sign the URLs, which includes project IDs as well as the account name. We prefer not to expose those information to the client-side for security reasons.

We use signed URLs for generating GET URLs, and they are indeed very useful. We would greatly appreciate it if the library could also expose session URLs, allowing us to utilize all the great features that the GCS API provides.

[BenWhitehead]

Our library does not present (and at this time does not plan on adding) an api that allows writing to a resumable upload that was started by another client or client instance. This is in no small part due to the complicated nature of the state machine required to safely and correctly upload to a resumable upload. As such, if an authorized storage instance is to provide a handle to perform a resumable upload to an unauthorized storage instance Storage#writer(URL) is the way to accomplish this.

If you question is more to the effect of "How can I start a resumable upload with my authorized client and hand the url off to an end user's web browser session to perform a single PUT?", the method this library uses to start resumable uploads is public, and could be invoked by your application code -- albeit missing transparent automatic retries as retries are managed at the com.google.cloud.storage.Storage layer, not the com.google.cloud.storage.spi.v1.StorageRpc layer.

import com.google.api.services.storage.model.StorageObject;
import com.google.cloud.storage.Storage;
import com.google.cloud.storage.StorageOptions;
import com.google.cloud.storage.spi.v1.StorageRpc;
import com.google.cloud.storage.spi.v1.StorageRpc.Option;
import com.google.common.collect.ImmutableMap;

public final class C {
  public static void main(String[] args) {
    try (Storage s = StorageOptions.http().build().getService()) {
      String location = ((StorageRpc) s.getOptions().getRpc())
        .open(
          StorageObject().setBucket("<bucket-name>").setName("<object-name>"), 
          ImmutableMap.of(Option.IF_GENERATION_MATCH, 0L) // prevent overwriting an existing object
        );
      System.out.println("location = " + location);
    }
  }
}

[tsawada]

@BenWhitehead Thank you so much! This is exactly the information we were looking for.

With your code we were able to initiate a resumable upload, but it seems the server has to set the Origin header to the open request, as the same value as the subsequent requests from the browsers.

https://cloud.google.com/storage/docs/performing-resumable-uploads

If you have enabled Cross-Origin Resource Sharing, you should also include an Origin header in both this and subsequent upload requests.

Looking at HttpStorageRpc.open I wasn't able to find a way to do so. If java-storage doesn't support this now, do you think adding ORIGIN to StorageRpc.Option enum is reasonable?

[BenWhitehead]

I'll need some more details about your bucket config in order to reproduce the error you're seeing and figure out a path forward. I'd prefer to avoid the need to add new StorageRpc.Option given the number of locations those options are referenced.

I've attempted to write an integration test that applies some CORS config to a bucket, but unfortunately I don't end up with a failed request to GCS.

The test:

java-storage/google-cloud-storage/src/test/java/com/google/cloud/storage/spi/v1/ITStartResumableUploadTest.java

Line 62 in 343ac60

public void cors() throws Exception {

The http logs

The POST to start the resumable upload

2025-02-07 16:59:07,829 CONFIG   com.google.api.client.http.HttpTransport           - -------------- REQUEST  --------------
POST https://storage.googleapis.com/upload/storage/v1/b/java-storage-grpc-rand-0b7fc87d-893b-4e4a-9535-3b1fbf6080eb/o?ifGenerationMatch=0&name=com.google.cloud.storage.spi.v1.ITStartResumableUploadTest.cors%5BHTTP%5D%5BPROD%5D-0001&uploadType=resumable
Accept-Encoding: gzip
Authorization: <Not Logged>
User-Agent: Google-HTTP-Java-Client/1.45.3 (gzip)
x-goog-api-client: gl-java/17.0.14__Debian gccl/--protobuf-3.25.5 gax/2.60.0 protobuf/3.25.5
x-upload-content-type: application/octet-stream
Content-Type: application/json; charset=UTF-8
Content-Length: 162
 
2025-02-07 16:59:07,832 CONFIG   com.google.api.client.http.HttpTransport           - Total: 162 bytes 
2025-02-07 16:59:07,832 CONFIG   com.google.api.client.http.HttpTransport           - {"bucket":"java-storage-grpc-rand-0b7fc87d-893b-4e4a-9535-3b1fbf6080eb","name":"com.google.cloud.storage.spi.v1.ITStartResumableUploadTest.cors[HTTP][PROD]-0001"} 
2025-02-07 16:59:07,908 CONFIG   com.google.api.client.http.HttpTransport           - -------------- RESPONSE --------------
HTTP/1.1 200 OK
Server: UploadServer
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
X-GUploader-UploadID: AHMx-iE8r1WSGNEdL-vdrmavRQ15rvXcHHGUI7_qBb8LmUm9-ORxUBNd9ROa29QvNMZXsgBLTZGypH0NQR2ohJ46aILbgNm-EvVPvh2Kp4TxcKE
Vary: X-Origin
Vary: Origin
Pragma: no-cache
Expires: Mon, 01 Jan 1990 00:00:00 GMT
Content-Length: 0
Date: Fri, 07 Feb 2025 16:59:07 GMT
Location: https://storage.googleapis.com/upload/storage/v1/b/java-storage-grpc-rand-0b7fc87d-893b-4e4a-9535-3b1fbf6080eb/o?ifGenerationMatch=0&name=com.google.cloud.storage.spi.v1.ITStartResumableUploadTest.cors%5BHTTP%5D%5BPROD%5D-0001&uploadType=resumable&upload_id=AHMx-iE8r1WSGNEdL-vdrmavRQ15rvXcHHGUI7_qBb8LmUm9-ORxUBNd9ROa29QvNMZXsgBLTZGypH0NQR2ohJ46aILbgNm-EvVPvh2Kp4TxcKE
Content-Type: text/plain; charset=utf-8

The PUT to finalize the upload

2025-02-07 16:59:07,913 CONFIG   com.google.api.client.http.HttpTransport           - -------------- REQUEST  --------------
PUT https://storage.googleapis.com/upload/storage/v1/b/java-storage-grpc-rand-0b7fc87d-893b-4e4a-9535-3b1fbf6080eb/o?ifGenerationMatch=0&name=com.google.cloud.storage.spi.v1.ITStartResumableUploadTest.cors%5BHTTP%5D%5BPROD%5D-0001&uploadType=resumable&upload_id=AHMx-iE8r1WSGNEdL-vdrmavRQ15rvXcHHGUI7_qBb8LmUm9-ORxUBNd9ROa29QvNMZXsgBLTZGypH0NQR2ohJ46aILbgNm-EvVPvh2Kp4TxcKE
Accept-Encoding: gzip
Content-Range: bytes */0
User-Agent: unauthed Google-HTTP-Java-Client/1.45.3 (gzip)
Content-Length: 0
 
2025-02-07 16:59:08,056 CONFIG   com.google.api.client.http.HttpTransport           - -------------- RESPONSE --------------
HTTP/1.1 200 OK
Server: UploadServer
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
ETag: CPywhbeEsosDEAE=
X-GUploader-UploadID: AHMx-iE8r1WSGNEdL-vdrmavRQ15rvXcHHGUI7_qBb8LmUm9-ORxUBNd9ROa29QvNMZXsgBLTZGypH0NQR2ohJ46aILbgNm-EvVPvh2Kp4TxcKE
Vary: X-Origin
Vary: Origin
Pragma: no-cache
Expires: Mon, 01 Jan 1990 00:00:00 GMT
Content-Length: 1273
Date: Fri, 07 Feb 2025 16:59:08 GMT
Content-Type: application/json; charset=UTF-8
 
2025-02-07 16:59:08,057 CONFIG   com.google.api.client.http.HttpTransport           - Total: 1,273 bytes 
2025-02-07 16:59:08,057 CONFIG   com.google.api.client.http.HttpTransport           - {
  "kind": "storage#object",
  "id": "java-storage-grpc-rand-0b7fc87d-893b-4e4a-9535-3b1fbf6080eb/com.google.cloud.storage.spi.v1.ITStartResumableUploadTest.cors[HTTP][PROD]-0001/1738947547977852",
  "selfLink": "https://www.googleapis.com/storage/v1/b/java-storage-grpc-rand-0b7fc87d-893b-4e4a-9535-3b1fbf6080eb/o/com.google.cloud.storage.spi.v1.ITStartResumableUploadTest.cors%5BHTTP%5D%5BPROD%5D-0001",
  "mediaLink": "https://storage.googleapis.com/download/storage/v1/b/java-storage-grpc-rand-0b7fc87d-893b-4e4a-9535-3b1fbf6080eb/o/com.google.cloud.storage.spi.v1.ITStartResumableUploadTest.cors%5BHTTP%5D%5BPROD%5D-0001?generation=1738947547977852&alt=media",
  "name": "com.google.cloud.storage.spi.v1.ITStartResumableUploadTest.cors[HTTP][PROD]-0001",
  "bucket": "java-storage-grpc-rand-0b7fc87d-893b-4e4a-9535-3b1fbf6080eb",
  "generation": "1738947547977852",
  "metageneration": "1",
  "contentType": "application/octet-stream",
  "storageClass": "STANDARD",
  "size": "0",
  "md5Hash": "1B2M2Y8AsgTpgAmY7PhCfg==",
  "crc32c": "AAAAAA==",
  "etag": "CPywhbeEsosDEAE=",
  "timeCreated": "2025-02-07T16:59:08.011Z",
  "updated": "2025-02-07T16:59:08.011Z",
  "timeStorageClassUpdated": "2025-02-07T16:59:08.011Z",
  "timeFinalized": "2025-02-07T16:59:08.011Z"
}

Would it be possible for you to share what headers are sent with the PUT requests (please strip any sensitive values/names before sharing) from your browser or other client?

[tsawada]

Thanks! I can get the details next week, but off the top of my head, I think what’s happening here is that:

• Browsers set Origin headers on PUT requests, and this cannot be modified or removed via the JavaScript API.
• GCS expects the Origin header value to be the same for both the POST and PUT requests.

I believe this applies regardless of whether the bucket has a custom CORS configuration or not.
I tried running your test with an Origin header on #2916 but of course a random outsider cannot trigger a CI...

[BenWhitehead]

I've run your changes in #2916 with the added origin header, but do not receive any permission error.

The request/response logging:

POST

POST https://storage.googleapis.com/upload/storage/v1/b/java-storage-e0bebc17-e37b-4e93-a08b-43e50333f0a9/o?ifGenerationMatch=0&name=hackery/63541c28-c8b2-4607-bc9b-3cc831c39089&uploadType=resumable
Accept-Encoding: gzip
Authorization: <Not Logged>
User-Agent: Google-HTTP-Java-Client/1.45.3 (gzip)
x-goog-api-client: gl-java/17.0.14__Debian gccl/--protobuf-3.25.5 gax/2.60.0 protobuf/3.25.5
x-upload-content-type: application/octet-stream
Content-Type: application/json; charset=UTF-8
Content-Length: 116
 
2025-02-10 22:05:55,028 CONFIG   com.google.api.client.http.HttpTransport           - Total: 116 bytes 
2025-02-10 22:05:55,029 CONFIG   com.google.api.client.http.HttpTransport           - {"bucket":"java-storage-e0bebc17-e37b-4e93-a08b-43e50333f0a9","name":"hackery/63541c28-c8b2-4607-bc9b-3cc831c39089"} 
2025-02-10 22:05:55,064 CONFIG   com.google.api.client.http.HttpTransport           - -------------- RESPONSE --------------
HTTP/1.1 200 OK
Server: UploadServer
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
X-GUploader-UploadID: AHMx-iGLXbpo8VUV8gIrnVcZxy_i6zfkynrlqFzqTB1fS0FVChjLndsq3kSRUMfRFOdY6SZFzArU2eWixuO4o49pZ8uIbDa46ZAHlFM_TaZELcE
Vary: X-Origin
Vary: Origin
Pragma: no-cache
Expires: Mon, 01 Jan 1990 00:00:00 GMT
Content-Length: 0
Date: Mon, 10 Feb 2025 22:05:55 GMT
Location: https://storage.googleapis.com/upload/storage/v1/b/java-storage-e0bebc17-e37b-4e93-a08b-43e50333f0a9/o?ifGenerationMatch=0&name=hackery/63541c28-c8b2-4607-bc9b-3cc831c39089&uploadType=resumable&upload_id=AHMx-iGLXbpo8VUV8gIrnVcZxy_i6zfkynrlqFzqTB1fS0FVChjLndsq3kSRUMfRFOdY6SZFzArU2eWixuO4o49pZ8uIbDa46ZAHlFM_TaZELcE
Content-Type: text/plain; charset=utf-8

PUT

2025-02-10 22:05:55,068 CONFIG   com.google.api.client.http.HttpTransport           - -------------- REQUEST  --------------
PUT https://storage.googleapis.com/upload/storage/v1/b/java-storage-e0bebc17-e37b-4e93-a08b-43e50333f0a9/o?ifGenerationMatch=0&name=hackery/63541c28-c8b2-4607-bc9b-3cc831c39089&uploadType=resumable&upload_id=AHMx-iGLXbpo8VUV8gIrnVcZxy_i6zfkynrlqFzqTB1fS0FVChjLndsq3kSRUMfRFOdY6SZFzArU2eWixuO4o49pZ8uIbDa46ZAHlFM_TaZELcE
Accept-Encoding: gzip
Content-Range: bytes */0
User-Agent: unauthed Google-HTTP-Java-Client/1.45.3 (gzip)
origin: example.com
Content-Length: 0
 
2025-02-10 22:05:55,194 CONFIG   com.google.api.client.http.HttpTransport           - -------------- RESPONSE --------------
HTTP/1.1 200 OK
Server: UploadServer
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
ETag: CKesucyOuosDEAE=
X-GUploader-UploadID: AHMx-iGLXbpo8VUV8gIrnVcZxy_i6zfkynrlqFzqTB1fS0FVChjLndsq3kSRUMfRFOdY6SZFzArU2eWixuO4o49pZ8uIbDa46ZAHlFM_TaZELcE
Vary: X-Origin
Vary: Origin
Pragma: no-cache
Expires: Mon, 01 Jan 1990 00:00:00 GMT
Content-Length: 1077
Date: Mon, 10 Feb 2025 22:05:55 GMT
Content-Type: application/json; charset=UTF-8
 
2025-02-10 22:05:55,195 CONFIG   com.google.api.client.http.HttpTransport           - Total: 1,077 bytes 
2025-02-10 22:05:55,195 CONFIG   com.google.api.client.http.HttpTransport           - {
  "kind": "storage#object",
  "id": "java-storage-e0bebc17-e37b-4e93-a08b-43e50333f0a9/hackery/63541c28-c8b2-4607-bc9b-3cc831c39089/1739225155130919",
  "selfLink": "https://www.googleapis.com/storage/v1/b/java-storage-e0bebc17-e37b-4e93-a08b-43e50333f0a9/o/hackery%2F63541c28-c8b2-4607-bc9b-3cc831c39089",
  "mediaLink": "https://storage.googleapis.com/download/storage/v1/b/java-storage-e0bebc17-e37b-4e93-a08b-43e50333f0a9/o/hackery%2F63541c28-c8b2-4607-bc9b-3cc831c39089?generation=1739225155130919&alt=media",
  "name": "hackery/63541c28-c8b2-4607-bc9b-3cc831c39089",
  "bucket": "java-storage-e0bebc17-e37b-4e93-a08b-43e50333f0a9",
  "generation": "1739225155130919",
  "metageneration": "1",
  "contentType": "application/octet-stream",
  "storageClass": "STANDARD",
  "size": "0",
  "md5Hash": "1B2M2Y8AsgTpgAmY7PhCfg==",
  "crc32c": "AAAAAA==",
  "etag": "CKesucyOuosDEAE=",
  "timeCreated": "2025-02-10T22:05:55.157Z",
  "updated": "2025-02-10T22:05:55.157Z",
  "timeStorageClassUpdated": "2025-02-10T22:05:55.157Z",
  "timeFinalized": "2025-02-10T22:05:55.157Z"
}

I think we'll need the bucket config in order to try and replicate things.

[BenWhitehead]

Once #2941 merges you will be able to set arbitrary headers for the open call, including Origin if necessary for the CORS issues you're running into.

[BenWhitehead]

version 2.49.0 has been cut, and should be available in maven central later today.