CGI data products update (everything)

[agreenbaum]

Hi,

Please find metadata definitions for CGI products. New files for L3, L4, and calibration products. I tried to reduce redundancy of telemetry tied to earlier product when there is a 1-to-1 mapping (e.g., one L2a per L1 file), based on discussion with Scott.

• Level 1: Updated mas/yr unit for RApm, DECpm, added ISACQ keyword, added additional description to FCMLOOP, FSMLOS, FSMPRFL

cgi_l1_metadata_defintions.csv

• Level 2a: Removed all L1-inherited telemetry since there will be one L2a per L1

cgi_l2a_metadata_definitions.csv

• Level 2b: L2bs are either in analog mode where one L2b file is made for every L2a, or photon counting mode where multiple L2as are combined into a single L2b. This update removes much of the L1 metadata but kept the most important: VISITID, VISTYPE, ISHOWFSC, ISPC, ISACQ, SATSPOTS, and all the PAM telemetry. Added MJDSTART/MJDEND though it sounds like this could be computed from the preceding data products, more discussion may be needed. A few additional keywords are added related to the "DRP" (data reduction pipeline) processing.

cgi_l2b_metadata_defintions.csv

• Level 3: One L3 is created per L2b. At this stage, I've identified the most important telemetry that needs to be tied to the L3 products from earlier levels. I've included the WCS keywords that are computed at this step, because I assume those are needed.

cgi_l3_metadata_defintions.csv

• Level 4: At level 4 frames are combined, We'll need to preserve the PAM names, target (TARGET) and WFS information (HOWFSLNK). Some additional DRP keywords at this level are included.

cgi_l4_metadata_defintions.csv

• Ancillary: No changes

cgi_anc_metadata_definitions.csv

• Calibration: A note about the calibration keyword "CALTYPE" I've listed it as an ENUM, but more calibration types than this list are planned (e.g. for spectroscopy, polarimetry, etc.) so if it's easier to leave this as an open string I'm happy to do that.

cgi_cal_metadata_defintions.csv

[agreenbaum]

@scfleming and others met with us yesterday, and it sounds like removing all of the "propagated" metadata was maybe the wrong direction. I need a little guidance on both how to specify this in these .csv definitions for your development, and what that means for the actual json message metadata that we send.
Some of these fields get tricky when frames are combined which can happen for level 2b (photon-counting), level 3, and obviously level 4. I think I'm still not quite getting how the filesets work, maybe that is the key.

[agreenbaum]

Note: noticed a typo yesterday FMLOOP is supposed to be FSMLOOP.

[scfleming]

@agreenbaum if you wanted to go over filesets again, I can meet with you do demo how it works for Roman WFI, and I think you could then apply that to CGI as they are very similar. Happy to schedule a brief session at a time that works for you, just let me know a couple options.