Bottlerocket EC2NodeClass: when only /dev/xvdb is set in blockDeviceMappings, /dev/xvda is created as 2Gi gp2 (expected defaults 4Gi gp3 per docs)

[strazhnyk]

Description

Description

Observed Behavior:
When using amiFamily: Bottlerocket and specifying spec.blockDeviceMappings with only the data volume (/dev/xvdb) to increase container storage, newly provisioned nodes have the root/control volume (/dev/xvda) created as 2GiB and gp2.
This happens even though Bottlerocket docs show the default root/control device as 4GiB (and we expect gp3 in our environment / according to our configuration expectations).

In other words: overriding only /dev/xvdb appears to result in an unexpected default for /dev/xvda (size/type differs from Bottlerocket defaults shown in docs).

Expected Behavior:
If blockDeviceMappings includes only /dev/xvdb, Karpenter should either:
1. Preserve Bottlerocket defaults for /dev/xvda (4GiB, and expected volume type), or
2. Clearly document that specifying any blockDeviceMappings overrides the entire default mapping and that /dev/xvda will be derived from the underlying AMI/template defaults (which in our case become 2Gi gp2).

Reproduction Steps (Please include YAML):
1.Apply an EC2NodeClass for Bottlerocket where we only define /dev/xvdb:

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: bottlerocket-arm64
spec:
  amiFamily: Bottlerocket
  role: <KarpenterNodeRole>
  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: <cluster-name>
  securityGroupSelectorTerms:
    - tags:
        karpenter.sh/discovery: <cluster-name>
  blockDeviceMappings:
    - deviceName: /dev/xvdb
      ebs:
        volumeSize: 100Gi
        volumeType: gp3
        encrypted: true
        deleteOnTermination: true

  tags:
    architecture: arm64

2. Create/ensure a NodePool that references this EC2NodeClass

---
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: apps-arm64-spot
spec:
  template:
    metadata:
      labels:
        "purpose": "apps-arm64-spot"
        "arm64": "true"
        "spot": "true"
    spec:
      requirements:
        - key: kubernetes.io/arch
          operator: In
          values: ["arm64"]
        - key: kubernetes.io/os
          operator: In
          values: ["linux"]
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot"]
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["t", "c", "m", "r"]
        - key: karpenter.k8s.aws/instance-generation
          operator: Gt
          values: ["4"]
      taints:
        - key: arm64_ready
          value: "true"
          effect: NoSchedule
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: bottlerocket-arm64
      expireAfter: "720h" # 30 * 24h = 720h
  limits:
    cpu: 1000
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 0s # set it to 0 to have the same behavior as in v1beta1.

3. Trigger provisioning (scale a deployment / create a pod that requires this NodePool).
4. Verify attached volumes on the created EC2 instance:

Result: /dev/xvda shows up as 2GiB gp2 (unexpected), while /dev/xvdb is correctly created as gp3 100GiB.

Versions:

• Karpenter Version: v1.2.0 (controller image tag)
• Chart Version: v1.2.0
• Kubernetes Version (kubectl version): v1.32 (kubectl= v1.31.1)

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[ryan-mist]

I was able to replicate 1 of the 2 issues presented. For me, the root volume (/dev/xvda) created as 2GiB rather than 4GiB, however, my root volume is gp3 type (not gp2). This may be because I was using the latest bottle rocket AMI. Can you also attach your amiSelectorTerms?

As for the root volume created with 2GiB rather than 4GiB, the root cause seems to be that we only use the default configuration for BlockDeviceMappings when no mapping is specified (code is here).

You can workaround this issue by adding a mapping for dev/xvda to your desired specifications.

[ryan-mist]

Bottlerocket docs show the default root/control device as 4GiB

By docs do you mean https://karpenter.sh/docs/concepts/nodeclasses/#bottlerocket-1?

I'm assuming that the reason the volume is created as 2GiB is that is what is configured in the AMI snapshot. See bottlerocket-os/bottlerocket#4376

[strazhnyk]

Bottlerocket docs show the default root/control device as 4GiB

By docs do you mean https://karpenter.sh/docs/concepts/nodeclasses/#bottlerocket-1?

I'm assuming that the reason the volume is created as 2GiB is that is what is configured in the AMI snapshot. See bottlerocket-os/bottlerocket#4376

Yes,by docs I mean https://karpenter.sh/docs/concepts/nodeclasses/#bottlerocket-1?

[strazhnyk]

I was able to replicate 1 of the 2 issues presented. For me, the root volume (/dev/xvda) created as 2GiB rather than 4GiB, however, my root volume is gp3 type (not gp2). This may be because I was using the latest bottle rocket AMI. Can you also attach your amiSelectorTerms?

As for the root volume created with 2GiB rather than 4GiB, the root cause seems to be that we only use the default configuration for BlockDeviceMappings when no mapping is specified (code is here).

You can workaround this issue by adding a mapping for dev/xvda to your desired specifications.

Thanks for looking into this!

What surprised me is the behavior difference between “no blockDeviceMappings specified” vs “partial mapping blockDeviceMappings”.

• If I leave blockDeviceMappings empty (just config without blockDeviceMappings) in the EC2NodeClass, the node comes up with the root volume settings that match the example/defaults shown in the documentation: https://karpenter.sh/docs/concepts/nodeclasses/#bottlerocket-1
• But when I add the mapping configuration only for, e.g. /dev/xvdb , the resulting /dev/xvda (root volume) shows up as 2GiB, while /dev/xvdb is created with configuration. That inconsistency is what made it look like a bug from my perspective, but if I mistake, let me know please :)

So from my side it felt like:

• “empty config” ⇒ applies doc-like defaults
• “partial config” ⇒ produces the described behavior

I used these AMIs:
ami-0e6e05cce6a7c3581
ami-0bd00f57d5606d9b4

[strazhnyk]

I was able to replicate 1 of the 2 issues presented. For me, the root volume (/dev/xvda) created as 2GiB rather than 4GiB, however, my root volume is gp3 type (not gp2). This may be because I was using the latest bottle rocket AMI. Can you also attach your amiSelectorTerms?

As for the root volume created with 2GiB rather than 4GiB, the root cause seems to be that we only use the default configuration for BlockDeviceMappings when no mapping is specified (code is here).

You can workaround this issue by adding a mapping for dev/xvda to your desired specifications.

Regarding the workaround, yes, thanks!
I use explicit /dev/xvda mapping alongside /dev/xvdb.
But the behavior, the configuration, in case of adding only one block, seemed strange to me.
However, maybe I'm wrong and judging by the code, it will work like this

[ryan-mist]

What surprised me is the behavior difference between “no blockDeviceMappings specified” vs “partial mapping blockDeviceMappings”.

I was a bit surprised as well, although, the docs do mention:

The following blockDeviceMapping defaults are used for each AMIFamily if no blockDeviceMapping overrides are specified in the EC2NodeClass

Which, since the default is used only when "no blockDeviceMapping overrides are specified", does indicate that this is expected behavior.

[strazhnyk]

What surprised me is the behavior difference between “no blockDeviceMappings specified” vs “partial mapping blockDeviceMappings”.

I was a bit surprised as well, although, the docs do mention:

The following blockDeviceMapping defaults are used for each AMIFamily if no blockDeviceMapping overrides are specified in the EC2NodeClass

Which, since the default is used only when "no blockDeviceMapping overrides are specified", does indicate that this is expected behavior.

Thanks a lot for the clarification! That makes sense now, and I understand why the behavior differs between “no overrides” vs “partial overrides.”
For now, I’m using a full blockDeviceMappings block to manage the node storage explicitly and avoid any unexpected defaults.