feat: add support for .tf.json

[STollenaar]

Changes

Added support for the terraform json file format. Expanded the parseHCL function to determine if a given file is a .tf.json or regular .tf file and do a JSON parse into HCL before passing that output into the HCL parser, which format is further supported by the code base

Context

• Closes Support .tf.json Terraform JSON #11997

Documentation (please check one with an [x])

• I have updated the documentation, or
• No documentation update is required

How I've tested my work (please select one)

I have verified these changes via:

• Code inspection only, or
• Newly added/modified unit tests, or
• No unit tests but ran on a real repository, or
• Both unit tests + ran on a real repository

[viceice]

I don't think we need all those new fixtures. please reduce to minimum to satisfy coverage

[viceice]

I also think think you shouldn't convert to tf and parse again. you should directly transform to the parsed hcl structure.

isn't the hcl lib from terraform supporting those json files out of the box?

[viceice]

you can check if there's an additional library to @cdktf/hcl2json which support it

[STollenaar]

I don't think we need all those new fixtures. please reduce to minimum to satisfy coverage

Ok, I will remove a few fixtures. I tried to keep the same tests as we have for the regular .tf files to ensure complete coverage for both types

I also think think you shouldn't convert to tf and parse again. you should directly transform to the parsed hcl structure.

isn't the hcl lib from terraform supporting those json files out of the box?

so this is a bit tricky. When executing terraform directly it does support the json files. However, the syntax is a bit different. With the @cdktf/hcl2json when parsed there are some distinct differences on what the output will be. This is why a simple JSON.parse doesn't work as the rest of the renovate code would break.

Including a few screenshots on the subtle differences:

you can check if there's an additional library to @cdktf/hcl2json which support it

I checked around and I couldn't find a supported node library that would support the json to hcl parse. There are instances of GO packages, but that wouldn't work for this project.

[viceice]

ok, in this case you should directly convert the Json to the parsed hcl format and don't generate the hcl source code.

[STollenaar]

ok, in this case you should directly convert the Jain to the parsed hcl format and don't generate the hcl source code.

From the .tf.json spec you don't really know when something is an array vs an object. in the kubernetes resources there is the metadata block for example. This is seen as an array, but written as an object. There are many of these little nuances that make it harder to go directly from JSON to that parsed hcl format

[RahulGautamSingh]

Could you reduce the number of deps extracted in the tests? I can see many deps being extracted and most are similar. Best will be to only keeps deps which are unique somewhat.

If something similar can be done with the fixtures too, please consider doing it. We prefer adding fixtures only when its necessary.

Thanks!

[viceice]

reduce fixtures to minimum size

[STollenaar]

I initially resolved this by removing some coverage. But I do want to ask, why do we want to reduce the coverage which would make sure the behavior between the .tf and the .tf.json remains the same?

[RahulGautamSingh]

We prefer to inline small fixtures wherever possible as it's better for performance. But it should not come at cost of coverage

[viceice]

ok, in this case you should directly convert the Jain to the parsed hcl format and don't generate the hcl source code.

From the .tf.json spec you don't really know when something is an array vs an object. in the kubernetes resources there is the metadata block for example. This is seen as an array, but written as an object. There are many of these little nuances that make it harder to go directly from JSON to that parsed hcl format

I don't agree. we know we only support specific types with specific properties, so we can directly use zod schema to parse and transform the known supported objects and discard all other things.

[STollenaar]

ok, in this case you should directly convert the Jain to the parsed hcl format and don't generate the hcl source code.

From the .tf.json spec you don't really know when something is an array vs an object. in the kubernetes resources there is the metadata block for example. This is seen as an array, but written as an object. There are many of these little nuances that make it harder to go directly from JSON to that parsed hcl format

I don't agree. we know we only support specific types with specific properties, so we can directly use zod schema to parse and transform the known supported objects and discard all other things.

I changed it to use the zod schema. Tests pass so I think it does work correctly now. Might need to expand the test cases to cover all the supported terraform resources

[secustor]

@cdktf/hcl2json directly parses JSON files while converting.

From the .tf.json spec you don't really know when something is an array vs an object. in the kubernetes resources there is the metadata block for example. This is seen as an array, but written as an object. There are many of these little nuances that make it harder to go directly from JSON to that parsed hcl format

This is how the spec describes it in their syntax docs.

When a JSON object property is named after a nested block type, the value of this property represents one or more blocks of that type. The value of the property must be either a JSON object or a JSON array.

To implement this it should be also possible to do a simply normalization from objects to array starting from the second or third level of the JSON objects

[secustor]

Why is this any?

[STollenaar]

sorry, an oversight. Removed for a typed value

[secustor]

Suggested change

	function normalise(node: any): any {
	function normalise(node: unknown): unknown {

Do not use any.

If we are going this route we should do this for every block and not only the Kubernetes ones.

When a JSON object property is named after a nested block type, the value of this property represents one or more blocks of that type. The value of the property must be either a JSON object or a JSON array.

https://developer.hashicorp.com/terraform/language/syntax/json#nested-block-mapping

[STollenaar]

If we are going this route we should do this for every block and not only the Kubernetes ones.

Agreed, however renovate currently only supports the docker, helm and kubernetes resources for Terraform.

[secustor]

@cdktf/hcl2json directly parses JSON files while converting.

If you read that function, it actually only does a JSON.parse on it and then does a deep merge with the .tf files. If you were to do that with renovate you get errors since the extractor doesn't recognize that kind of syntax

I have read it and meant this in context of this comment and it is feasible to me to adapt the extractor logic to expect an array here, if it simplifies the code.

[RahulGautamSingh]

Confliced

[RahulGautamSingh]

Also, we should add .tf.json to the managerFilePatterns

[RahulGautamSingh]

We prefer to inline small fixtures wherever possible as it's better for performance. But it should not come at cost of coverage

[STollenaar]

Also, we should add .tf.json to the managerFilePatterns

added the .json items to this, also updated to include the .tofu files

[RahulGautamSingh]

LGTM

[viceice]

looks good but still too many fixtures which will slow down tests too much

[STollenaar]

looks good but still too many fixtures which will slow down tests too much

Is this latest change to inline fixtures better?