Skip to content

Weight mismatch error running brats_mri_axial_slices_generative_diffusion bundle on MONAI 1.5 #8544

@Neetong

Description

@Neetong

Describe the bug
When running the brats_mri_axial_slices_generative_diffusion bundle on MONAI 1.5, inference fails with a weight mismatch error. The autoencoder or diffusion model cannot load the pretrained weights (KeyError for certain encoder/decoder keys), e.g. 'encoder.blocks.10.attn.out_proj.weight'.

To Reproduce
Steps to reproduce the behavior:

  1. Install MONAI 1.5 and all required dependencies (PyTorch 2.6, NumPy 2.x).
  2. Download the brats_mri_axial_slices_generative_diffusion bundle from Hugging Face (version 1.1.4).
  3. Run inference using the provided inference.json file.

Expected behavior
The bundle should run successfully and generate synthetic 2D MRI images from Gaussian noise without any weight mismatch errors.

Screenshots
2025-08-20 18:18:30,604 - WARNING - Default logging file in configs/logging.conf does not exist, skipping logging.
2025-08-20 18:18:30,609 - INFO - --- input summary of monai.bundle.scripts.run ---
2025-08-20 18:18:30,609 - INFO - > workflow: 'ConfigWorkflow'
2025-08-20 18:18:30,610 - INFO - ---


KeyError Traceback (most recent call last)
/usr/local/lib/python3.12/dist-packages/monai/bundle/config_item.py in evaluate(self, globals, locals)
373 try:
--> 374 return eval(value[len(self.prefix) :], globals_, locals)
375 except Exception as e:

in

12 frames
KeyError: 'encoder.blocks.10.attn.out_proj.weight'

The above exception was the direct cause of the following exception:

RuntimeError Traceback (most recent call last)
/usr/local/lib/python3.12/dist-packages/monai/bundle/config_item.py in evaluate(self, globals, locals)
374 return eval(value[len(self.prefix) :], globals_, locals)
375 except Exception as e:
--> 376 raise RuntimeError(f"Failed to evaluate {self}") from e
377 warnings.warn(
378 f"\n\npdb: value={value}\n"

RuntimeError: Failed to evaluate ConfigExpression:
"$__local_refs['autoencoder_def'].load_old_state_dict(torch.load(__local_refs['load_autoencoder_path']))"

Environment

2025-08-20 18:34:37.879722: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:467] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1755714877.901508 23310 cuda_dnn.cc:8579] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1755714877.908249 23310 cuda_blas.cc:1407] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
W0000 00:00:1755714877.925178 23310 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1755714877.925212 23310 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1755714877.925215 23310 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1755714877.925217 23310 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.

Printing MONAI config...

MONAI version: 1.5.0
Numpy version: 2.3.2
Pytorch version: 2.6.0+cu124
MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False
MONAI rev id: d388d1c
MONAI file: /usr/local/lib/python3.12/dist-packages/monai/init.py

Optional dependencies:
Pytorch Ignite version: NOT INSTALLED or UNKNOWN VERSION.
ITK version: NOT INSTALLED or UNKNOWN VERSION.
Nibabel version: 5.3.2
scikit-image version: 0.25.2
scipy version: 1.16.1
Pillow version: 11.3.0
Tensorboard version: 2.19.0
gdown version: 5.2.0
TorchVision version: 0.21.0+cu124
tqdm version: 4.67.1
lmdb version: NOT INSTALLED or UNKNOWN VERSION.
psutil version: 5.9.5
pandas version: 2.2.2
einops version: 0.8.1
transformers version: 4.55.2
mlflow version: NOT INSTALLED or UNKNOWN VERSION.
pynrrd version: NOT INSTALLED or UNKNOWN VERSION.
clearml version: NOT INSTALLED or UNKNOWN VERSION.

For details about installing the optional dependencies, please visit:
https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies

================================
Printing system config...

System: Linux
Linux version: Ubuntu 22.04.4 LTS
Platform: Linux-6.1.123+-x86_64-with-glibc2.35
Processor: x86_64
Machine: x86_64
Python version: 3.12.11
Process name: python3
Command: ['python3', '-c', 'import monai; monai.config.print_debug_info()']
Open files: []
Num physical CPUs: 6
Num logical CPUs: 12
Num usable CPUs: 12
CPU usage (%): [4.8, 3.4, 2.8, 2.9, 3.7, 80.0, 2.7, 4.2, 3.0, 10.1, 2.3, 21.5]
CPU freq. (MHz): 2200
Load avg. in last 1, 5, 15 mins (%): [2.8, 1.8, 2.6]
Disk usage (%): 46.5
Avg. sensor temp. (Celsius): UNKNOWN for given OS
Total physical memory (GB): 53.0
Available memory (GB): 49.4
Used memory (GB): 2.9

================================
Printing GPU config...

Num GPUs: 1
Has CUDA: True
CUDA version: 12.4
cuDNN enabled: True
NVIDIA_TF32_OVERRIDE: None
TORCH_ALLOW_TF32_CUBLAS_OVERRIDE: None
cuDNN version: 90100
Current device: 0
Library compiled for CUDA architectures: ['sm_50', 'sm_60', 'sm_70', 'sm_75', 'sm_80', 'sm_86', 'sm_90']
GPU 0 Name: NVIDIA L4
GPU 0 Is integrated: False
GPU 0 Is multi GPU board: False
GPU 0 Multi processor count: 58
GPU 0 Total memory (GB): 22.2
GPU 0 CUDA capability (maj.min): 8.9

Additional context

  • Issue appears after updating to MONAI 1.5 and bundle version 1.1.4.
  • All other files (autoencoder, diffusion weights, configs) are unchanged from previous bundle versions.
  • This prevents the bundle from running properly on the latest MONAI version.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions