Skip to content

RecursionError occurs intermittently when triggering workflows using FarmVibes Python client. #226

@click2cloud-SanchitG

Description

@click2cloud-SanchitG

In which step did you encounter the bug?

Workflow execution

Are you using a local or a remote (AKS) FarmVibes.AI cluster?

Remote (AKS) cluster

Bug description

Description:

We are encountering an intermittent RecursionError: maximum recursion depth exceeded while using the FarmVibes Python client to submit workflows to a running FarmVibes cluster. The error appears after a few successful executions and seems to originate from deep recursive validation in pydantic when handling workflow configuration `data.```

Steps to reproduce the problem

How to Reproduce:

Set up a FarmVibes cluster with the Python client configured to connect.
Programmatically trigger multiple workflow runs in sequence using the client’s client.run() method (e.g., for different parameters).
After several successful executions, the error begins to surface. This suggests it builds up under certain runtime conditions or data patterns.

Observed Behavior:
The error occurs during internal processing of workflow submission, likely within create_new_run() on the server side, which the Python client calls under the hood.

The logs show a runaway recursive call in Pydantic dataclass validation, ultimately throwing:

RecursionError: maximum recursion depth exceeded while calling a Python object
RecursionError: maximum recursion depth exceeded while calling a Python object

Error (Truncated):


{"app_id":"terravibes-rest-api","instance":"terravibes-rest-api-677cb8097c-b5vvs","level":"ERROR","msg":"Failed to update workflow state\nTraceback (most recent call last):\n  File \"/opt/conda/lib/python3.11/site-packages/vibe_server/server.py\", line 378, in create_run\n    new_id, new_run = self.create_new_run(runConfig, run_ids)\n  File \"/opt/conda/lib/python3.11/site-packages/pydantic/dataclasses.py\", line 293, in pydantic.dataclasses._add_pydantic_validation_attributes.handle_extra_init\n  File \"/opt/conda/lib/python3.11/site-packages/pydantic/dataclasses.py\", line 293, in pydantic.dataclasses._add_pydantic_validation_attributes.handle_extra_init\n[Previous line repeated 934 more times]\n  File \"<string>\", line 12, in __init__\n  File \"/opt/conda/lib/python3.11/site-packages/pydantic/dataclasses.py\", line 312, in pydantic.dataclasses._add_pydantic_validation_attributes.new_post_init\n  File \"/opt/conda/lib/python3.11/site-packages/vibe_core/datamodel.py\", line 215, in __post_init__\n    super().__post_init__()\n  File \"/opt/conda/lib/python3.11/site-packages/pydantic/dataclasses.py\", line 312, in pydantic.dataclasses._add_pydantic_validation_attributes.new_post_init\n  File \"/opt/conda/lib/python3.11/site-packages/vibe_core/datamodel.py\", line 119, in __post_init__\n    add_pydantic_validation_attributes.new_post_init(self)\n  File \"/opt/conda/lib/python3.11/site-packages/pydantic/dataclasses.py\", line 445, in pydantic.dataclasses._add_pydantic_validation_attributes.new_post_init\n    self.__pydantic_model__.validate_model(self)\n  File \"/opt/conda/lib/python3.11/site-packages/pydantic/main.py\", line 1074, in pydantic.main.validate_model\n  File \"/opt/conda/lib/python3.11/site-packages/pydantic/fields.py\", line 881, in pydantic.fields.ModelField.validate\n  File \"/opt/conda/lib/python3.11/site-packages/pydantic/fields.py\", line 1091, in pydantic.fields.ModelField.validate_singleton\n  File \"/opt/conda/lib/python3.11/site-packages/pydantic/fields.py\", line 1012, in pydantic.fields.ModelField.validate_mapping_like\n  File \"/opt/conda/lib/python3.11/site-packages/pydantic/fields.py\", line 881, in pydantic.fields.ModelField.validate\n  File \"/opt/conda/lib/python3.11/site-packages/pydantic/fields.py\", line 1098, in pydantic.fields.ModelField.validate_singleton\n  File \"/opt/conda/lib/python3.11/site-packages/pydantic/class_validators.py\", line 337, in pydantic.class_validators._generic_validator_basic.Lambda13\n  File \"/opt/conda/lib/python3.11/site-packages/pydantic/validators.py\", line 62, in pydantic.validators.str_validator\nRecursionError: maximum recursion depth exceeded while calling a Python object","scope":"vibe_server.server.TerravibesProvider","time":"2025-06-09 08:50:26,246","type":"log","ver":"dev"}


Notes:

  • The issue is not immediate — it happens after a number of workflow runs (e.g., 5–10).
  • Restarting the FarmVibes server resets the problem temporarily.
  • From logs, it seems the issue lies in repeated application of validation logic in the workflow dataclasses (perhaps RunConfig or nested models).
  • We have workflows like soil salinity and sensor optimization configured.

Request:

  • Please review whether recursive dataclass construction or validation within the Vibe Server (via Pydantic) is causing unbounded depth.
  • Suggest a patch or workaround, such as limiting validator depth or modifying model construction to avoid circular references.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingremote clusterIssues encountered in remote (AKS) clustertriageIssues still not triaged by teamworkflowsIssues encountered when running workflows

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions