Does Pi0 load the wrong PaliGemma config in case of missing model_type?

[Qu3tzal]

Why is this line

lerobot/lerobot/common/policies/pi0/paligemma_with_expert.py

Line 116 in 2b71789

if "model_type" not in gemma_expert_config:

elif isinstance(self.paligemma_config, dict):
    # Override Pi0 default config for PaliGemma
    if "model_type" not in gemma_expert_config:
        paligemma_config["model_type"] = "paligemma"

    cfg_cls = CONFIG_MAPPING[paligemma_config["model_type"]]
    self.paligemma_config = cfg_cls(**paligemma_config)

and not

elif isinstance(self.paligemma_config, dict):
    # Override Pi0 default config for PaliGemma
    if "model_type" not in paligemma_config:
        paligemma_config["model_type"] = "paligemma"

    cfg_cls = CONFIG_MAPPING[paligemma_config["model_type"]]
    self.paligemma_config = cfg_cls(**paligemma_config)

?

So if we don't define the model type in the expert this triggers the reset of the PaliGemma model type?

[OpenJarvisAI]

self.pi0_paligemma.config.ignore_index

[Qu3tzal]

@OpenJarvisAI I don't think it's connected to the issue?

[OpenJarvisAI]

this repo has numerous bugs.