-
Notifications
You must be signed in to change notification settings - Fork 174
RecursionError occurs intermittently when triggering workflows using FarmVibes Python client. #226
Description
In which step did you encounter the bug?
Workflow execution
Are you using a local or a remote (AKS) FarmVibes.AI cluster?
Remote (AKS) cluster
Bug description
Description:
We are encountering an intermittent RecursionError: maximum recursion depth exceeded while using the FarmVibes Python client to submit workflows to a running FarmVibes cluster. The error appears after a few successful executions and seems to originate from deep recursive validation in pydantic when handling workflow configuration `data.```
Steps to reproduce the problem
How to Reproduce:
Set up a FarmVibes cluster with the Python client configured to connect.
Programmatically trigger multiple workflow runs in sequence using the client’s client.run() method (e.g., for different parameters).
After several successful executions, the error begins to surface. This suggests it builds up under certain runtime conditions or data patterns.
Observed Behavior:
The error occurs during internal processing of workflow submission, likely within create_new_run() on the server side, which the Python client calls under the hood.
The logs show a runaway recursive call in Pydantic dataclass validation, ultimately throwing:
RecursionError: maximum recursion depth exceeded while calling a Python object
RecursionError: maximum recursion depth exceeded while calling a Python object
Error (Truncated):
{"app_id":"terravibes-rest-api","instance":"terravibes-rest-api-677cb8097c-b5vvs","level":"ERROR","msg":"Failed to update workflow state\nTraceback (most recent call last):\n File \"/opt/conda/lib/python3.11/site-packages/vibe_server/server.py\", line 378, in create_run\n new_id, new_run = self.create_new_run(runConfig, run_ids)\n File \"/opt/conda/lib/python3.11/site-packages/pydantic/dataclasses.py\", line 293, in pydantic.dataclasses._add_pydantic_validation_attributes.handle_extra_init\n File \"/opt/conda/lib/python3.11/site-packages/pydantic/dataclasses.py\", line 293, in pydantic.dataclasses._add_pydantic_validation_attributes.handle_extra_init\n[Previous line repeated 934 more times]\n File \"<string>\", line 12, in __init__\n File \"/opt/conda/lib/python3.11/site-packages/pydantic/dataclasses.py\", line 312, in pydantic.dataclasses._add_pydantic_validation_attributes.new_post_init\n File \"/opt/conda/lib/python3.11/site-packages/vibe_core/datamodel.py\", line 215, in __post_init__\n super().__post_init__()\n File \"/opt/conda/lib/python3.11/site-packages/pydantic/dataclasses.py\", line 312, in pydantic.dataclasses._add_pydantic_validation_attributes.new_post_init\n File \"/opt/conda/lib/python3.11/site-packages/vibe_core/datamodel.py\", line 119, in __post_init__\n add_pydantic_validation_attributes.new_post_init(self)\n File \"/opt/conda/lib/python3.11/site-packages/pydantic/dataclasses.py\", line 445, in pydantic.dataclasses._add_pydantic_validation_attributes.new_post_init\n self.__pydantic_model__.validate_model(self)\n File \"/opt/conda/lib/python3.11/site-packages/pydantic/main.py\", line 1074, in pydantic.main.validate_model\n File \"/opt/conda/lib/python3.11/site-packages/pydantic/fields.py\", line 881, in pydantic.fields.ModelField.validate\n File \"/opt/conda/lib/python3.11/site-packages/pydantic/fields.py\", line 1091, in pydantic.fields.ModelField.validate_singleton\n File \"/opt/conda/lib/python3.11/site-packages/pydantic/fields.py\", line 1012, in pydantic.fields.ModelField.validate_mapping_like\n File \"/opt/conda/lib/python3.11/site-packages/pydantic/fields.py\", line 881, in pydantic.fields.ModelField.validate\n File \"/opt/conda/lib/python3.11/site-packages/pydantic/fields.py\", line 1098, in pydantic.fields.ModelField.validate_singleton\n File \"/opt/conda/lib/python3.11/site-packages/pydantic/class_validators.py\", line 337, in pydantic.class_validators._generic_validator_basic.Lambda13\n File \"/opt/conda/lib/python3.11/site-packages/pydantic/validators.py\", line 62, in pydantic.validators.str_validator\nRecursionError: maximum recursion depth exceeded while calling a Python object","scope":"vibe_server.server.TerravibesProvider","time":"2025-06-09 08:50:26,246","type":"log","ver":"dev"}
Notes:
- The issue is not immediate — it happens after a number of workflow runs (e.g., 5–10).
- Restarting the FarmVibes server resets the problem temporarily.
- From logs, it seems the issue lies in repeated application of validation logic in the workflow dataclasses (perhaps RunConfig or nested models).
- We have workflows like soil salinity and sensor optimization configured.
Request:
- Please review whether recursive dataclass construction or validation within the Vibe Server (via Pydantic) is causing unbounded depth.
- Suggest a patch or workaround, such as limiting validator depth or modifying model construction to avoid circular references.