Formalizing the PEcAn Event Data Specification

[dlebauer]

Description

PEcAn is adopting a 'standard' format for representing events including agronomic management and disturbance. This file type is intended to be analogous to the met data standard - an intermediate, model-independent representation that supports downstream translation into model-specific configurations.

Background

Though we have developed a format e.g.
fieldactivity_events.json
and JSON schema
ui_structure.json as part of the fieldactivity app app.

Details

Our use case is focused on agronomic events, but this could be extended to land use change, disturbance, etc.

Criteria:

• Links events to a specific location and date
• Builds on existing standards & works well with geospatial software
• Minimum information required: location, date, event type
• Allows additional attributes
• Text based (not required, but this is where we are going)

Request for feedback

The proposed specification is at
https://gist.github.com/dlebauer/5a2e505bca6afe66ca6aa44d9dd163f7

Please react to this thread w/ the appropriate emoji

• 👍 = Looks good to me!
• 👀 = I have questions (please comment)
• 💡 = I have ideas for improvement (please comment)

And comment as appropriate

We will discuss this at the next PEcAn biweekly, and I'd like to finalize this and begin implementing at that point.

cc @istfer @HenriKajasilta @mdietze

[mdietze]

A few comments:

1. I find the FeatureCollection part to be a bit confusing. For polygons, is the user expected to embed the entire geometry in the JSON? This seems excessive and duplicative with the polygon layer itself
2. For the management events themselves, I'd recommend adding the units to the variable names where appropriate to make the files more self documenting (e.g., fertilizer_total_amount could become fertilizer_total_amount_kg_m-2)
3. for the data that's stored in these files, how do we want to reconcile information that might hypothetically be provided by a farmer vs the variables that we aim to provide in the CARB project? There's a lot here that we'll never be able to provide and things that we will be able to provide that aren't here.
4. Similarly, there are things the model needs that aren't here, but it's not 100% clear to me where this file lives in the workflow between the RS and the model. Is the transition to model inputs supposed to be self contained or reliant or additional metadata
5. I'd recommend that areas be provided as fractional, not in absolute units
6. A lot of code lists are referenced by acronym but not linked / documented

[dlebauer]

1. FeatureCollection

I was attempting to follow geoJSON, but we could easily extract the properties to simplify it greatly

{
  "site": "9900000099",
  "events": [
    {
      "mgmt_operations_event": "tillage",
      "date": "2023-09-08",
      "tillage_practice": "tillage_practice_primary",

2. Units
   Good catch on omitting units. What is the primary advantage of adding them to the variable name vs storing them in a separate element?

3. Contents

how do we want to reconcile information that might hypothetically be provided by a farmer vs the variables that we aim to provide in the CARB project

Most of the spec, with all the details and options - is directly from the field activity schema. I wanted to start there and modify as necessary (as I did by putting it in a geojson).

We could add another (optional?) field that indicates the source of information and / or add this as metadata at the top if it all comes from a single source.

There's a lot here that we'll never be able to provide and things that we will be able to provide that aren't here. ... Similarly, there are things the model needs that aren't here ...

see below

4. Scope

it's not 100% clear to me where this file lives in the workflow between the RS and the model. Is the transition to model inputs supposed to be self contained or reliant or additional metadata.

I imagine that this would fit in the workflow thus

fieldactivity UI

fieldactivity ------------------------> events.json

manually curated, e.g. validation data

paper --------------------gapfiller---> events.json

remotely sensed

RS information -----------gapfiller---> events.json 

Where the gapfiller would take data from other sources, including extension guidance, usda stats, best practices etc.

Ideally the resulting events.json should be self contained - containing all information that a write.configs function will need to convert to model inputs. (Analogous to the met standard)

This brings up a larger question of 'what is the minimum information required' by this specification.

We can define what minimum required information is, e.g.

• all events require date and event type
• additional information
  • planting: crop type
  • harvest: crop type (+ fraction removed?)
  • fertilizer:

       (type + amount) 
       OR 
       ((mineral vs organic) AND 
               ((IF mineral amount of N) OR
                (IF organic, amount of N + C))
    

  • irrigation: amount

--> I've updated the 'minimum working example' accordingly

Minimum required for farmer-supplied vs remotely sensed and gap-filled will likely differ, e.g. a farmer can be expected to provide the type of fertilizer through a web interface, but a RS based pipeline may only have enough information to estimate of the amount of N, perhaps whether it is mineral or organic.

There are many challenges, e.g.

• models do have different needs; this isn't as consistent as e.g. met or soil
• gapfiller would have to be region specific; could get complicated.
• this can be a huge quagmire

5. Area units

I'd recommend that areas be provided as fractional, not in absolute units

That is a good point, especially given that models don't have a concept of area; I'll defer to the FMI team @HenriKajasilta and @istfer.

I don't think area needs to be required - we can assume 100% unless otherwise specified.

6. Code lists

A lot of code lists are referenced by acronym but not linked / documented

Those are from a single source, ICASA (vocabulary used by AGMIP) - listed under "Event specific properties" in the gist. There is a more user-friendly lookup table in the fieldactivity source code: https://github.com/PecanProject/fieldactivity/blob/main/inst/extdata/display_names.csv

I think using the 'display name' from that file could be an option - this would help make it more clear what happened.

[mdietze]

1. units: they could be entered in a different field, but they should be in the file not in separate metadata. One advantage of including them in the name is that it makes it clear what the standard is, while adding a units variable implies that any units are acceptable and then we need to add unit conversions to the workflow
2. contents: this wasn't a question about provenance tracking (did the data come from a farmer or R.S.) but a much more practical concern that the numbers I plan to provide aren't all documented here and the scope of this standard implies that one could be building down-stream workflows that depend on variables that will never be provided. I know we don't want the file format to be SIPNET specific, but why don't we start with the variables we're actually generating and then expand as we add more variables?
3. scope/workflow: If the goal is to ensure that this file is not model specific, I think this will require the injection of additional information after the generation of the file (e.g. translating from vegetation class to harvest fractions). This might be fine, but there's a non-zero chance that either information will be lost, or workflows will become unnecessarily complicated, or we might end up generating model-specific files that exist in parallel to the standard. This might be a purely hypothetical concern -- I haven't thought through the gory details of where this standard fits into the proposed CARB/MAGIC monitoring workflows.
4. Codes: should we establish a dependency on a "standard" that's based on a complicated excel spread sheet that doesn't look to be particularly machine readable (e.g. use of blank rows to indicate change in variable type; different info in different tables such that one has to manually search across tabs to know which table contains a particular set of codes.)

[dlebauer]

Notes from PEcAn meeting June 30 2025

• Storage options
  • Each file could correspond to a single site and ensemble realization. This would make geoJSON unnecessary
  • but ... to save space, GeoJSON allows explicitly assigning the same event time series to multiple sites by using a MultiPoint geometry type.
  • Could store (and read from) parquet JSON
• Required fields: event_type, date, location
• Optional fields vary by event_type, common examples include
  • tillage: optional intensity (continuous or categorical) + method
  • fertilizer: type, amount N, mineral vs organic
  • harvest: crop, fraction removed
• Separately consider (and store):
  • Probabilistic / observational data such as 'tillage probability'
  • Model-ready realizations --> deterministic event.json files with one ensemble member per run
• Upstream observational data (e.g. RS-based change in tillage index NDTI) may be retained alongside derived quantities.
• GeoJSON format

We may need:

• A field to track intensity_method and or source for values like tillage intensity
• Ability to store arbitrary information from upstream that may be useful (including measurement method, data source, detection probabilities)
• Naming convention, e.g.(ENS--.json?

I have updated the spec¹ to include

• ensemble number with each event time series
• guiding principles for development
• table of required and (a subset of) optional fields

Next steps:

• start prototyping mock event types Create mock monitoring inputs for testing ccmmf/organization#144

Footnotes

1. https://gist.github.com/dlebauer/5a2e505bca6afe66ca6aa44d9dd163f7 ↩

[HenriKajasilta]

Below are some brief comments about the event format and the discussions that have already taken place.

• Units: In the Fieldactivity UI, the units are displayed in the form headers for the users. However, as you discussed, it would make sense to display them in the properties.
• Events: Currently, this is not the case in Fieldactivity either, but having each planting/harvesting event displayed separately, rather than in a list form (check the extended example for a planting event), would make them simpler to handle. This is planned to change in Fieldactivity too.
• Area: I am not sure why we asked for the area. It was at least easy for farmers to fill in, and it could give us an indication of geometric changes, if there had been variation between the years. As discussed, it is not currently an efficient way to check which share of the field has been harvested, and the amounts are anyway displayed per area (mostly per hectare).

[istfer]

Hi David,

Thank you for the spec, the general idea behind it and the proposed implementation look great to me.

A few comments:

• Just a note that field activity app is historically built to replace some excel sheets farmers were filling in Finland, so some elements such as area and observations are relics from those excels (not that we used them in simulation workflows)
• I also liked the idea to store arbitrary information from upstream, especially data source 
• I agree with Mike's point about ICASA dictionary not being machine readable, but I still support sticking to it since it is the main standard used by many crop modelers (I think ICOS will also start using this vocabulary for crop sites)
• One thing that came up in our discussions is that some parcels might be changing from one year to another. In that case for example site id, lat/lon might be the same but the polygon of the field (to derive data from RS for example) might be different. I don't know if this is an issue you encountered or if it would cause complications with this spec, but mentioning it here to think out loud
• One other thing that has been on my mind is also to have some sort of summary/timeline section about what is in the field for the duration of the simulation / events file. What I mean is, when reading these files, it can be difficult to work out what is actually on the field at all times, need to find sowing & harvest events, parse *_crop elements, reconstruct the timeline etc. Would some section like this be useful (it certainly is for STICS but I don't know about it being a part of the standard)?

    "rotation": [
      {
        "planted_crop1": "-99.0", # bare ground
        "planted_crop2": "-99.0",
        "rotation_begin": "2022-01-01",
        "rotation_end": "2022-03-31"
      },
      {
        "planted_crop1": "RYE",
        "planted_crop2": "-99.0",
        "rotation_begin": "2022-04-01",
        "rotation_end": "2022-09-21"
      },
      {
        "planted_crop1": "OAT", # oat with cover crop
        "planted_crop2": "WCL",
        "rotation_begin": "2022-09-01",
        "rotation_end": "2023-09-27"
      },
      {
        "planted_crop1": "WCL", #oat removed, cover crop remains
        "planted_crop2": "-99.0",
        "rotation_begin": "2023-09-27",
        "rotation_end": "2024-04-01"
      },
   ],

[dlebauer]

@istfer and @mdietze - ICASA has just migrated from XLSX to CSV, does this allay concerns of machine-readability?

[mdietze]

Conversion from XLSX to CSV does not by itself make the information machine-readable. Additional key criteria would be whether they’ve made the data “tidy” or not, and whether things are joined into one large table or still exist and many smaller tables. The latter is technically “machine readable” but may still be a major pain in the butt to work with (i.e. it doesn’t alleviate the extent to which expert knowledge is needed to be able to understand what info is in what tables and how to join them). This information doesn’t appear to be of sufficient size and complexity to really justify having to do complex database joins and lots of complex conditional logic in the coding (i.e. it strikes me as something that really could/should exist as a single tidy table) On Jul 16, 2025, at 7:44 PM, David LeBauer ***@***.***> wrote: @istfer<https://github.com/istfer> and @mdietze<https://github.com/mdietze> - ICASA has just migrated from XLSX to CSV<DSSAT/ICASA-Dictionary#5>, does this allay concerns of machine-readability? — Reply to this email directly, view it on GitHub<#3551 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAXQJ6TFFTLFFSWGCSEWOYL3IZ6NPAVCNFSM6AAAAACABIC5OGVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTGNZXHE4DOOI>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[dlebauer]

yes, they have made the data tidy / normalized: https://github.com/DSSAT/ICASA-Dictionary/pull/5/files; https://github.com/DSSAT/ICASA-Dictionary/tree/main/CSV

[dlebauer]

I've opened a PR #3621 with an MVP events.json schema and a script that generates examples; and some examples.

What it includes from this discussion:

• site_id - no GeoJSON
• variable names that include units (amount_mm, leaf_c_kg_m2)
• units follow standard_vars (kg/m2; mm for irrigation)
• every event has a event type + date
• planting and harvest have crop identifier

This is what is still TODO (post MVP)

• add optional fraction_area_0to1
• make crop required

These are nice to have

• rotation object that summarizes timeline and makes it clear what is in the field on a given date (as suggested by Istem)
• (optional) mapping to ICASA crops

Not included (per discussion)

• geojson geometry (can be looked up from site_id)
• dependency on ICASA

[dlebauer]

#3621 implemented a new events schema specification has been implemented in modules/data.land/inst/events_schema_v0.1.0.json

Rotations can be handled by planting + harvest events followed by planting; it is possible to add an optional rotation object - and / or a function that can extract this information from an events.json file.

[ashiklom]

For tracking purposes, flagging that additional schema changes are being proposed in #3836.

[dlebauer]

I'll go ahead and close this since we have moved from the design/discussion phase and v0.1 is implemented.