-
Notifications
You must be signed in to change notification settings - Fork 1.3k
Description
Describe the bug
When running the brats_mri_axial_slices_generative_diffusion bundle on MONAI 1.5, inference fails with a weight mismatch error. The autoencoder or diffusion model cannot load the pretrained weights (KeyError for certain encoder/decoder keys), e.g. 'encoder.blocks.10.attn.out_proj.weight'.
To Reproduce
Steps to reproduce the behavior:
- Install MONAI 1.5 and all required dependencies (PyTorch 2.6, NumPy 2.x).
- Download the brats_mri_axial_slices_generative_diffusion bundle from Hugging Face (version 1.1.4).
- Run inference using the provided inference.json file.
Expected behavior
The bundle should run successfully and generate synthetic 2D MRI images from Gaussian noise without any weight mismatch errors.
Screenshots
2025-08-20 18:18:30,604 - WARNING - Default logging file in configs/logging.conf does not exist, skipping logging.
2025-08-20 18:18:30,609 - INFO - --- input summary of monai.bundle.scripts.run ---
2025-08-20 18:18:30,609 - INFO - > workflow: 'ConfigWorkflow'
2025-08-20 18:18:30,610 - INFO - ---
KeyError Traceback (most recent call last)
/usr/local/lib/python3.12/dist-packages/monai/bundle/config_item.py in evaluate(self, globals, locals)
373 try:
--> 374 return eval(value[len(self.prefix) :], globals_, locals)
375 except Exception as e:
in
12 frames
KeyError: 'encoder.blocks.10.attn.out_proj.weight'
The above exception was the direct cause of the following exception:
RuntimeError Traceback (most recent call last)
/usr/local/lib/python3.12/dist-packages/monai/bundle/config_item.py in evaluate(self, globals, locals)
374 return eval(value[len(self.prefix) :], globals_, locals)
375 except Exception as e:
--> 376 raise RuntimeError(f"Failed to evaluate {self}") from e
377 warnings.warn(
378 f"\n\npdb: value={value}\n"
RuntimeError: Failed to evaluate ConfigExpression:
"$__local_refs['autoencoder_def'].load_old_state_dict(torch.load(__local_refs['load_autoencoder_path']))"
Environment
2025-08-20 18:34:37.879722: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:467] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1755714877.901508 23310 cuda_dnn.cc:8579] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1755714877.908249 23310 cuda_blas.cc:1407] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
W0000 00:00:1755714877.925178 23310 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1755714877.925212 23310 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1755714877.925215 23310 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1755714877.925217 23310 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
Printing MONAI config...
MONAI version: 1.5.0
Numpy version: 2.3.2
Pytorch version: 2.6.0+cu124
MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False
MONAI rev id: d388d1c
MONAI file: /usr/local/lib/python3.12/dist-packages/monai/init.py
Optional dependencies:
Pytorch Ignite version: NOT INSTALLED or UNKNOWN VERSION.
ITK version: NOT INSTALLED or UNKNOWN VERSION.
Nibabel version: 5.3.2
scikit-image version: 0.25.2
scipy version: 1.16.1
Pillow version: 11.3.0
Tensorboard version: 2.19.0
gdown version: 5.2.0
TorchVision version: 0.21.0+cu124
tqdm version: 4.67.1
lmdb version: NOT INSTALLED or UNKNOWN VERSION.
psutil version: 5.9.5
pandas version: 2.2.2
einops version: 0.8.1
transformers version: 4.55.2
mlflow version: NOT INSTALLED or UNKNOWN VERSION.
pynrrd version: NOT INSTALLED or UNKNOWN VERSION.
clearml version: NOT INSTALLED or UNKNOWN VERSION.
For details about installing the optional dependencies, please visit:
https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies
================================
Printing system config...
System: Linux
Linux version: Ubuntu 22.04.4 LTS
Platform: Linux-6.1.123+-x86_64-with-glibc2.35
Processor: x86_64
Machine: x86_64
Python version: 3.12.11
Process name: python3
Command: ['python3', '-c', 'import monai; monai.config.print_debug_info()']
Open files: []
Num physical CPUs: 6
Num logical CPUs: 12
Num usable CPUs: 12
CPU usage (%): [4.8, 3.4, 2.8, 2.9, 3.7, 80.0, 2.7, 4.2, 3.0, 10.1, 2.3, 21.5]
CPU freq. (MHz): 2200
Load avg. in last 1, 5, 15 mins (%): [2.8, 1.8, 2.6]
Disk usage (%): 46.5
Avg. sensor temp. (Celsius): UNKNOWN for given OS
Total physical memory (GB): 53.0
Available memory (GB): 49.4
Used memory (GB): 2.9
================================
Printing GPU config...
Num GPUs: 1
Has CUDA: True
CUDA version: 12.4
cuDNN enabled: True
NVIDIA_TF32_OVERRIDE: None
TORCH_ALLOW_TF32_CUBLAS_OVERRIDE: None
cuDNN version: 90100
Current device: 0
Library compiled for CUDA architectures: ['sm_50', 'sm_60', 'sm_70', 'sm_75', 'sm_80', 'sm_86', 'sm_90']
GPU 0 Name: NVIDIA L4
GPU 0 Is integrated: False
GPU 0 Is multi GPU board: False
GPU 0 Multi processor count: 58
GPU 0 Total memory (GB): 22.2
GPU 0 CUDA capability (maj.min): 8.9
Additional context
- Issue appears after updating to MONAI 1.5 and bundle version 1.1.4.
- All other files (autoencoder, diffusion weights, configs) are unchanged from previous bundle versions.
- This prevents the bundle from running properly on the latest MONAI version.