🐞 Device inconsistency when utilizing AUPRO metric

[samet-akcay]

Discussed in #2749

^{Originally posted by Narc17 June 6, 2025}

Describe the bug

Greetings!

I faced a tensor device inconsistent issue while applying AUPRO in Patchcore validation/testing. The model training was did utilize cuda and was succeeded. However, it raised error about two tensors on different devices in validation or testing. This issue only occurs on AUPRO metric. How can I slove this issue?

Thanks!

Dataset

MVTecAD

Model

PatchCore

Steps to reproduce the behavior

1. Create an Evaluator with AUPRO image and pixel level metrics.
   • Evaluator(test_metrics=[AUPRO(fields=["pred_score", "gt_label"], prefix="image_"), AUPRO(fields=["anomaly_map", "gt_mask"], prefix="pixel_", strict=False) ])
2. Initialize Patchcore with evaluator={evaluator created with step 1}
3. Initialize Engine and do Engine.fit() and Engine.test()
4. It will raise the error above during processing Engine.test()

OS information

OS information:

• OS: Ubuntu 24.04 (docker container)
• Python version: 3.11.12
• Anomalib version: 2.0.0
• PyTorch version: 2.1.2
• CUDA/cuDNN version: 12.6; driver version: 560.76
• GPU models and configuration: 1x GeForce RTX 3070 Ti Laptop
• Any other relevant information:
  • Using a sub-dataset from MVTec dataset, the hazelnut dataset.
  • Evaluator setup:
    • Evaluator(test_metrics=[AUPRO(fields=["pred_score", "gt_label"], prefix="image_"), AUPRO(fields=["anomaly_map", "gt_mask"], prefix="pixel_", strict=False) ])

Expected behavior

AUPRO metric could be handled GPU(cuda) computations.

Screenshots

No response

Pip/GitHub

pip

What version/branch did you use?

2.0.0

Configuration YAML

Did not utilize yaml configs.

Logs

Traceback (most recent call last):
  File "/src/anomalib_script.py", line 594, in <module>
    test_engine.test(datamodule=test_datamodule, model=test_lightning_model)
  File "/usr/local/lib/python3.11/dist-packages/anomalib/engine/engine.py", line 558, in test
    return self.trainer.test(model, dataloaders, ckpt_path, verbose, datamodule)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/lightning/pytorch/trainer/trainer.py", line 748, in test
    return call._call_and_handle_interrupt(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/lightning/pytorch/trainer/call.py", line 47, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/lightning/pytorch/trainer/trainer.py", line 788, in _test_impl
    results = self._run(model, ckpt_path=ckpt_path)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/lightning/pytorch/trainer/trainer.py", line 981, in _run
    results = self._run_stage()
              ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/lightning/pytorch/trainer/trainer.py", line 1018, in _run_stage
    return self._evaluation_loop.run()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/lightning/pytorch/loops/utilities.py", line 178, in _decorator
    return loop_run(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/lightning/pytorch/loops/evaluation_loop.py", line 142, in run
    return self.on_run_end()
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/lightning/pytorch/loops/evaluation_loop.py", line 254, in on_run_end
    self._on_evaluation_epoch_end()
  File "/usr/local/lib/python3.11/dist-packages/lightning/pytorch/loops/evaluation_loop.py", line 336, in _on_evaluation_epoch_end
    trainer._logger_connector.on_epoch_end()
  File "/usr/local/lib/python3.11/dist-packages/lightning/pytorch/trainer/connectors/logger_connector/logger_connector.py", line 195, in on_epoch_end
    metrics = self.metrics
              ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/lightning/pytorch/trainer/connectors/logger_connector/logger_connector.py", line 234, in metrics
    return self.trainer._results.metrics(on_step)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/lightning/pytorch/trainer/connectors/logger_connector/result.py", line 473, in metrics
    value = self._get_cache(result_metric, on_step)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/lightning/pytorch/trainer/connectors/logger_connector/result.py", line 437, in _get_cache
    result_metric.compute()
  File "/usr/local/lib/python3.11/dist-packages/lightning/pytorch/trainer/connectors/logger_connector/result.py", line 288, in wrapped_func
    self._computed = compute(*args, **kwargs)
                     ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/lightning/pytorch/trainer/connectors/logger_connector/result.py", line 253, in compute
    return self.value.compute()
           ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/torchmetrics/metric.py", line 699, in wrapped_func
    value = _squeeze_if_scalar(compute(*args, **kwargs))
                               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/anomalib/metrics/base.py", line 191, in compute
    return super().compute()  # type: ignore[misc]
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/anomalib/metrics/aupro.py", line 307, in compute
    fpr, tpr = self._compute()
               ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/anomalib/metrics/aupro.py", line 299, in _compute
    return self.compute_pro(cca=cca, target=target, preds=preds)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/anomalib/metrics/aupro.py", line 219, in compute_pro
    output_size = torch.where(fpr <= self.fpr_limit)[0].size(0)
                              ^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!

Code of Conduct

• I agree to follow this project's Code of Conduct

[github-actions]

Thanks for submitting this issue! It has been added to our triage queue. A maintainer will review it shortly.

[samet-akcay]

@sonhm3029 found the root cause for this here

#2749 (comment)

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[github-actions]

This issue was closed because it has been stalled for 14 days with no activity.