feat(@jsii/spec): add a utility library to be consumed by jsii modules #3570

kaizencc · 2022-06-02T23:04:15Z

Adds a set of utility functions under @jsii/spec that can (and will) be leveraged by other jsii modules.

Currently has the following functions exposed:

getAssemblyFile()
writeAssembly()
loadAssemblyFromPath()
loadAssemblyFromFile()

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

kaizencc

@RomainMuller my game plan is to iterate on this PR to your fancy, and then modify the previous PR to use this library (and address the rest of your comments) when its merged.

packages/jsii/lib/assembler.ts

packages/@jsii/utils/lib/assembly.ts

kaizencc

If they both exist... there might be an issue -- in particular, this will ALWAYS prefer loading the uncompressed assembly, when it should likely prefer loading the compressed one if it exists...

A possible solution would be to ALWAYS load .jsii, but if there is a compressed assembly, that file basically contains something along the lines of:
{
  "schema": "jsii/file-redirect",
  "compression": "gzip",
  "filename": ".jsii.gz"
}
The loader would detect this... and proceed to load (and uncompress) .jsii.gz as indicated.

@RomainMuller if you want this, i am happy to do things this way. but as it stands, i simply changed to preferring the compressed file if it exists, and i don't really see what always writing the .jsii file will achieve, and in fact, i think it might be worse.

we also talked offline about hacking the .jsii file to throw a descriptive error when an old jsii version is being used. i couldn't really get this to work, because the only error i can throw is Unexpected token E in JSON at position 0, which gives us 1 character of description to work with 😂. if what we write in .jsii is actual JSON, then the loader will simply load the .jsii file normally. Since we don't really enforce validating the assembly, i think it's entirely possible that users get ugly non-descriptive errors when they try to access the nonexistent assembly['name'].

my conclusion is that the best we can do is throw the error that the .jsii file does not exist in these situations. At least that alerts the user that the issue is at the level of a missing .jsii file, and not that their .jsii file is malformed. let me know if that conclusion makes sense!

RomainMuller · 2022-06-03T08:35:05Z

i don't really see what always writing the .jsii file will achieve, and in fact, i think it might be worse.

Using that "redirect" placeholder gives us a new point where we can detect future "you need to upgrade your toolchain" situations... We can actually insert an unconditional validation of the contents of the schema property, and bail out with an error message that says you need to upgrade your toolchain or the assembly might have been corrupted/tampered with.

Some of the stuff we could end up doing in the future include (not that we are planning to do any of these at this time, but I could see those as possible future directions):

Other compression algorithms if they yield better compression ratios (ZStd, Brotli, Bzip, ...)
Breaking down an assembly into multiple documents (.jsii.tgz or .jsii.zip)
Per-language assemblies (.jsii-java, .jsii-python, ...)

Basically it would help crystallize the contract that you can ALWAYS start by looking at .jsii, and that may contain instructions on how to load other files.

We could also have a requires: { jsii: '>= X.Y.Z' } property in there, which we could check in the future (to throw "You are on jsii A.B.C, but this assembly requires jsii X.Y.Z. Please upgrade your toolchain"), and would hopefully serve as a self-diagnosis support tool for people using jsii releases earlier than the one which will introduce this feature.

packages/@jsii/utils/tsconfig.json

packages/@jsii/utils/.npmignore

…ath always loads SPEC_FILE_NAME first

packages/@jsii/utils/src/assembly.ts

kaizencc

As this nears the finish line... how do we feel about the name @jsii/utils? Currently utils is too broad of a term, I think. I plan to add functions for writing/loading .jsii.tabl.json files as well, maybe also writing/loading package.json files.

Since these functions seem to govern input/output, is there a better term that we can name this module? @jsii/io?

RomainMuller · 2022-06-15T07:33:14Z

packages/jsii/package.json

@@ -37,6 +37,7 @@
  "dependencies": {
    "@jsii/check-node": "0.0.0",
    "@jsii/spec": "^0.0.0",
+    "@jsii/utils": "^0.0.0",


There should be a related entry of references in tsconfig.json.

packages/@jsii/spec/package.json

mergify · 2022-06-28T08:21:07Z

Thank you for contributing! ❤️ I will now look into making sure the PR is up-to-date, then proceed to try and merge it!

mergify · 2022-06-28T08:21:31Z

Merging (with squash)...

This PR utilizes the functions added in #3570 everywhere in the jsii monorepo. The benefit of this is two-fold -- it refactors all places where we load/read assemblies and points them to the source of truth (introduced in #3570). This also means the logic for compressing assemblies will be available for all packages in the jsii monorepo if/when we flip that switch. --- By submitting this pull request, I confirm that my contribution is made under the terms of the [Apache 2.0 license]. [Apache 2.0 license]: https://www.apache.org/licenses/LICENSE-2.0

Updates `cdk-generate-synthetic-examples` to use the functions introduced in aws/jsii#3570 where appropriate.

This is part of the Reduce Package Size [project](https://github.com/aws/aws-cdk/projects/15). Assembly compression is supported in jsii v1.64.0 (see this jsii [PR](aws/jsii#3570) for details + discussion about how compressed assemblies are handled). This PR includes the following: - `--compress-assembly` flag turned on for `aws-cdk-lib` only - `run-rosetta.sh` depends on `cdk-generate-synthetic-examples@^0.1.14` which is the minimum version with compressed assembly [support](cdklabs/cdk-generate-synthetic-examples#16). ---- This `--compress-assembly` flag was tested in the following way: - this branch was pushed to `test-main-pipeline` and successfully built in the test pipeline. - the build artifact was downloaded from s3, unzipped, and verified to have the correct `.jsii` and `.jsii.gz` structure in both TypeScript and Python. - the build artifact was used to synth and deploy basic CDK apps in TypeScript and Python. - the build artifact was privately published to a codeartifact repository. - the codeartifact repository was used as a source to a local instance of construct hub. - verified that the published library was ingested correctly by the ingestion lambda and shows up on construct hub. - verified (via eye test, nothing too rigorous) that documentation was generated correctly on construct hub, including generated examples. ---- On code artifact, the package size shows up as 54 MB, compared to 194 MB, which is the size of the current package in npm. ---- ### All Submissions: * [ ] Have you followed the guidelines in our [Contributing guide?](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md) ### Adding new Unconventional Dependencies: * [ ] This PR adds new unconventional dependencies following the process described [here](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md/#adding-new-unconventional-dependencies) ### New Features * [ ] Have you added the new feature to an [integration test](https://github.com/aws/aws-cdk/blob/main/INTEGRATION_TESTS.md)? * [ ] Did you use `yarn integ` to deploy the infrastructure and generate the snapshot (i.e. `yarn integ` without `--dry-run`)? *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license* ---- ### All Submissions: * [ ] Have you followed the guidelines in our [Contributing guide?](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md) ### Adding new Unconventional Dependencies: * [ ] This PR adds new unconventional dependencies following the process described [here](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md/#adding-new-unconventional-dependencies) ### New Features * [ ] Have you added the new feature to an [integration test](https://github.com/aws/aws-cdk/blob/main/INTEGRATION_TESTS.md)? * [ ] Did you use `yarn integ` to deploy the infrastructure and generate the snapshot (i.e. `yarn integ` without `--dry-run`)? *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

kaizencc added 3 commits June 2, 2022 18:49

new module for jsii utility functions

0068b7f

update readme and package.json

74bd29c

prove that new module works

5bd3929

kaizencc requested a review from RomainMuller June 2, 2022 23:04

kaizencc commented Jun 2, 2022

View reviewed changes

packages/jsii/lib/assembler.ts Outdated Show resolved Hide resolved

packages/@jsii/utils/lib/assembly.ts Outdated Show resolved Hide resolved

kaizencc commented Jun 2, 2022

View reviewed changes

RomainMuller reviewed Jun 3, 2022

View reviewed changes

packages/@jsii/utils/tsconfig.json Outdated Show resolved Hide resolved

RomainMuller reviewed Jun 3, 2022

View reviewed changes

packages/@jsii/utils/.npmignore Outdated Show resolved Hide resolved

kaizencc added 4 commits June 3, 2022 11:48

move spec file names to @jsii/spec and re-export in @jsii/utils

759b7d7

update project structure for out-of-source building

ca54f78

getAssemblyFile returns path to SPEC_FILE_NAME, and loadAssemblyFromP…

c802eb2

…ath always loads SPEC_FILE_NAME first

add tests on error messages

7d75f79

kaizencc requested a review from RomainMuller June 3, 2022 19:09

kaizencc added 3 commits June 3, 2022 15:22

fix formatting for writing assembly

b6abc5d

add validation prop to load assembly

f60a57a

add validate to loadfrompath

7742e78

RomainMuller reviewed Jun 10, 2022

View reviewed changes

packages/@jsii/utils/src/assembly.ts Outdated Show resolved Hide resolved

packages/@jsii/utils/src/assembly.ts Outdated Show resolved Hide resolved

packages/@jsii/utils/src/assembly.ts Outdated Show resolved Hide resolved

kaizencc mentioned this pull request Jun 13, 2022

feat(jsii): compress .jsii file under feature flag #3560

Closed

kaizencc and others added 3 commits June 13, 2022 15:19

Merge branch 'main' into conroy/jsiiutils

3deb6f9

move compress into a property bag

cac5532

refactor schema validation error message

417df72

kaizencc requested a review from RomainMuller June 13, 2022 19:42

kaizencc commented Jun 13, 2022

View reviewed changes

kaizencc mentioned this pull request Jun 14, 2022

feat: use new read/load assembly functions everywhere #3600

Merged

fix test

f06ca4a

RomainMuller reviewed Jun 15, 2022

View reviewed changes

kaizencc and others added 3 commits June 27, 2022 20:32

Merge branch 'main' into conroy/jsiiutils

fc154f6

more @jsii/utils to @jsii/spec

550282a

update example usage

97e080f

kaizencc added 2 commits June 27, 2022 15:51

export functions from @jsii/spec

013d3d2

remove @jsii/utils

f7f60a4

kaizencc commented Jun 27, 2022

View reviewed changes

packages/@jsii/spec/package.json Show resolved Hide resolved

kaizencc requested a review from RomainMuller June 27, 2022 19:54

RomainMuller approved these changes Jun 28, 2022

View reviewed changes

mergify bot added the pr/ready-to-merge This PR is ready to be merged. label Jun 28, 2022

mergify bot merged commit 635404a into main Jun 28, 2022

mergify bot deleted the conroy/jsiiutils branch June 28, 2022 08:21

mergify bot removed the pr/ready-to-merge This PR is ready to be merged. label Jun 28, 2022

kaizencc changed the title ~~feat(@jsii/utils): add a utility library to be consumed by jsii modules~~ feat(@jsii/spec): add a utility library to be consumed by jsii modules Jun 28, 2022

kaizencc mentioned this pull request Jul 20, 2022

support creating and loading zipped assembly files #3561

Closed

2 tasks

kaizencc mentioned this pull request Aug 1, 2022

chore: update module with new jsii writeAssembly function cdklabs/cdk-generate-synthetic-examples#16

Merged

This was referenced Aug 5, 2022

chore: compress .jsii assembly files aws/aws-cdk#21480

Closed

feat(aws-cdk-lib): aws-cdk-lib assembly file is compressed aws/aws-cdk#21481

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(@jsii/spec): add a utility library to be consumed by jsii modules #3570

feat(@jsii/spec): add a utility library to be consumed by jsii modules #3570

Uh oh!

kaizencc commented Jun 2, 2022 •

edited

Loading

Uh oh!

kaizencc left a comment

Uh oh!

Uh oh!

Uh oh!

kaizencc left a comment

Uh oh!

RomainMuller commented Jun 3, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kaizencc left a comment

Uh oh!

RomainMuller Jun 15, 2022

Uh oh!

Uh oh!

mergify bot commented Jun 28, 2022

Uh oh!

mergify bot commented Jun 28, 2022

Uh oh!

Uh oh!

feat(@jsii/spec): add a utility library to be consumed by jsii modules #3570

feat(@jsii/spec): add a utility library to be consumed by jsii modules #3570

Uh oh!

Conversation

kaizencc commented Jun 2, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kaizencc left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

kaizencc left a comment

Choose a reason for hiding this comment

Uh oh!

RomainMuller commented Jun 3, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kaizencc left a comment

Choose a reason for hiding this comment

Uh oh!

RomainMuller Jun 15, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mergify bot commented Jun 28, 2022

Uh oh!

mergify bot commented Jun 28, 2022

Uh oh!

Uh oh!

kaizencc commented Jun 2, 2022 •

edited

Loading