[Feature] Built-in S3 support

[HuakunShen]

Built-in S3 API is supported in bun 1.2
https://bun.sh/docs/api/s3

It would be nice if a similar API can be implemented in Deno.

[Hajime-san]

I think that is weird for a JavaScript runtime to maintain this code.

https://github.com/oven-sh/bun/blob/dd93f08215c6f39bfd7aa868fd9e304ca4451cad/src/s3/credentials.zig#L393-L394

[HuakunShen]

I think that is weird for a JavaScript runtime to maintain this code.

https://github.com/oven-sh/bun/blob/dd93f08215c6f39bfd7aa868fd9e304ca4451cad/src/s3/credentials.zig#L393-L394

Yeah, I also think it's weird for a js runtime, but both Deno and Bun are trying to become not only a JS runtime, but a all-in-one toolchain, right?
Deno has support for lint, fmt, test, kv, jsx.

Also, in another issue I mentioned that Deno is somehow super slow with @aws-sdk/client-s3 #27841
I don't know how I can use S3 with Deno now. (I didn't want to try those Deno S3 packages last updated ~4 years ago)

[hypnoagus]

I think that is weird for a JavaScript runtime to maintain this code.
https://github.com/oven-sh/bun/blob/dd93f08215c6f39bfd7aa868fd9e304ca4451cad/src/s3/credentials.zig#L393-L394

Yeah, I also think it's weird for a js runtime, but both Deno and Bun are trying to become not only a JS runtime, but a all-in-one toolchain, right? Deno has support for lint, fmt, test, kv, jsx.

Also, in another issue I mentioned that Deno is somehow super slow with @aws-sdk/client-s3 #27841 I don't know how I can use S3 with Deno now. (I didn't want to try those Deno S3 packages last updated ~4 years ago)

I agree the reason it feels right is what bun says in their docs "S3 is the de facto standard internet filesystem"

[dsherret]

I also think it's weird for a js runtime, but both Deno and Bun are trying to become not only a JS runtime, but a all-in-one toolchain, right?

An all-in-one toolchain is a little different than an all-in-one runtime. An issue with adding stuff like this into the runtime is it means all applications have a dependency on the code instead of just the ones that need it. It would cause the executable produced by deno compile (called denort) to always be X MB/KB bigger because it has S3 support built-in even though not everyone needs that. Over time that starts to add up (ex. built-in sqlite support is like 1.5MB, which is a bit of a shame that it's built-in with Deno.kv and now "node:sqlite". People argued for exposing this because we already used it behind the scenes, but I've refactored things in denort such that sqlite is only now used by the runtime for Deno.kv/"node:sqlite". That could have been a library only included when needed)

Another huge issue is the dependency can no longer be upgraded and managed independently of the runtime because it's built-in—in my opinion, it's a huge long term mistake to couple everything together... independent version management that libraries provide is a huge feature. Library authors being able to break APIs by increasing a major version number helps modernize and improve APIs over time and being able to stay on an old version of an API is extremely valuable. Imagine if Node had built-in all these APIs before template literals, async/await, and explicit resource managment were around? We'd be stuck with non-modern APIs built-in. Who knows what the future holds.

Also, in another issue I mentioned that Deno is somehow super slow with @aws-sdk/client-s3

I feel like it's better long term to invest time into identifying why certain libraries might be slow and innovating on low level stable primitives that enable library authors to achieve better performance. For example, in this case that will be possible to fix with this landing: hyperium/hyper#3818 -- should be in Deno by next week.

[hypnoagus]

I also think it's weird for a js runtime, but both Deno and Bun are trying to become not only a JS runtime, but a all-in-one toolchain, right?

An all-in-one toolchain is a little different than an all-in-one runtime. An issue with adding stuff like this into the runtime is it means all applications have a dependency on the code instead of just the ones that need it. It would cause the executable produced by deno compile (called denort) to always be X MB/KB bigger because it has S3 support built-in even though not everyone needs that. Over time that starts to add up (ex. built-in sqlite support is like 1.5MB, which is a bit of a shame that it's built-in with Deno.kv and now "node:sqlite". People argued for exposing this because we already used it behind the scenes, but I've refactored things in denort such that sqlite is only now used by the runtime for Deno.kv/"node:sqlite". That could have been a library only included when needed)

Another huge issue is the dependency can no longer be upgraded and managed independently of the runtime because it's built-in—in my opinion, it's a huge long term mistake to couple everything together... independent version management that libraries provide is a huge feature. Library authors being able to break APIs by increasing a major version number helps modernize and improve APIs over time and being able to stay on an old version of an API is extremely valuable. Imagine if Node had built-in all these APIs before template literals, async/await, and explicit resource managment were around? We'd be stuck with non-modern APIs built-in. Who knows what the future holds.

Also, in another issue I mentioned that Deno is somehow super slow with @aws-sdk/client-s3

I feel like it's better long term to invest time into identifying why certain libraries might be slow and innovating on low level stable primitives that enable library authors to achieve better performance. For example, in this case that will be possible to fix with this landing: hyperium/hyper#3818 -- should be in Deno by next week.

Okay but what about the argument of these two being de facto standards?

[HuakunShen]

I also think it's weird for a js runtime, but both Deno and Bun are trying to become not only a JS runtime, but a all-in-one toolchain, right?

An all-in-one toolchain is a little different than an all-in-one runtime. An issue with adding stuff like this into the runtime is it means all applications have a dependency on the code instead of just the ones that need it. It would cause the executable produced by deno compile (called denort) to always be X MB/KB bigger because it has S3 support built-in even though not everyone needs that. Over time that starts to add up (ex. built-in sqlite support is like 1.5MB, which is a bit of a shame that it's built-in with Deno.kv and now "node:sqlite". People argued for exposing this because we already used it behind the scenes, but I've refactored things in denort such that sqlite is only now used by the runtime for Deno.kv/"node:sqlite". That could have been a library only included when needed)

Another huge issue is the dependency can no longer be upgraded and managed independently of the runtime because it's built-in—in my opinion, it's a huge long term mistake to couple everything together... independent version management that libraries provide is a huge feature. Library authors being able to break APIs by increasing a major version number helps modernize and improve APIs over time and being able to stay on an old version of an API is extremely valuable. Imagine if Node had built-in all these APIs before template literals, async/await, and explicit resource managment were around? We'd be stuck with non-modern APIs built-in. Who knows what the future holds.

Also, in another issue I mentioned that Deno is somehow super slow with @aws-sdk/client-s3

I feel like it's better long term to invest time into identifying why certain libraries might be slow and innovating on low level stable primitives that enable library authors to achieve better performance. For example, in this case that will be possible to fix with this landing: hyperium/hyper#3818 -- should be in Deno by next week.

I wonder how Bun plans to deal with dependency and breaking API change problem. I believe S3 is stable enough though.

[petamoriken]

Since you say it is the de facto standard, why not encourage WinterTC to make it the official standard?

[hypnoagus]

Since you say it is the de facto standard, why not encourage WinterTC to make it the official standard?

I dont think that's needed, it's okay being the de facto standard for now

[Hajime-san]

After all, since the details of S3 are hidden by the cloud vendors, it makes the most sense to use the SDKs they provide directly to access its detailed functions.
For others, operationalizing tests would be very difficult.
Deno.kv is a product developed by Deno, so they can test it in production easily, but the same cannot be said for S3.

[HuakunShen]

https://jsr.io/@hk/s3
I built a library imitating Bun's API using @aws-sdk/client-s3.
Although it's impossible to reach the performance of bun's, at least the API's look simpler than the original package's commands.