`wfi_image.meta`'s `required` does not quite follow conventions

[WilliamJamieson]

This section of wfi_image.meta:

rad/latest/wfi_image.yaml

Lines 16 to 38 in b9c2b75

	- type: object
	properties:
	background:
	tag: asdf://stsci.edu/datamodels/roman/tags/sky_background-1.0.0
	cal_logs:
	tag: asdf://stsci.edu/datamodels/roman/tags/cal_logs-1.0.0
	cal_step:
	tag: asdf://stsci.edu/datamodels/roman/tags/l2_cal_step-1.2.0
	outlier_detection:
	tag: asdf://stsci.edu/datamodels/roman/tags/outlier_detection-1.0.0
	photometry:
	tag: asdf://stsci.edu/datamodels/roman/tags/photometry-1.0.0
	source_catalog:
	tag: asdf://stsci.edu/datamodels/roman/tags/source_catalog-1.0.0
	statistics:
	tag: asdf://stsci.edu/datamodels/roman/tags/statistics-1.0.0
	wcs:
	title: WCS object
	anyOf:
	- tag: tag:stsci.edu:gwcs/wcs-*
	- type: "null"
	required: [photometry, wcs]

Seems to run counter to our normal development patterns for example the statistics schema includes archive information but it is not required. Similarly, we expect from

rad/latest/l3_common.yaml

Lines 144 to 280 in b9c2b75

	background:
	title: Table of Background Level 2 (L2) Metadata
	description: >
	Table of background metadata from input calibrated rate
	image files. Background metadata contains measured
	sky background levels, the measurement method and
	if the file was background corrected.
	tag: tag:astropy.org:astropy/table/table-1.*
	basic:
	title: Table of Basic Level 2 (L2) Metadata
	description: >
	Table of basic metadata from input calibrated rate
	image files. Basic metadata encompass a wide variety of
	information such as pipeline version numbers and origin of the
	file.
	tag: tag:astropy.org:astropy/table/table-1.*
	cal_step:
	title: Table of Calibration Step Level 2 (L2) Metadata
	description: >
	Table of calibration step metadata from input
	calibrated rate image files. Calibration step metadata indicate
	which calibration steps were applied to the data during
	detector-level processing.
	tag: tag:astropy.org:astropy/table/table-1.*
	coordinates:
	title: Table of Coordinates Level 2 (L2) Metadata
	description: >
	Table of coordinates metadata from input calibrated
	rate image files. Coordinates metadata indicate the type of
	coordinate system used (e.g., the International Celestial
	Reference System or ICRS).
	tag: tag:astropy.org:astropy/table/table-1.*
	ephemeris:
	title: Table of Ephemeris Level 2 (L2) Metadata
	description: >
	Table of ephemeris metadata from input calibrated rate
	image files. Ephemeris metadata describe the location and
	movement of the spacecraft during the observations.
	tag: tag:astropy.org:astropy/table/table-1.*
	exposure:
	title: Table of Exposure Level 2 (L2) Metadata
	description: >
	Table of exposure metadata from input calibrated rate
	image files. Exposure metadata contain information about the
	exposure such as start time and readout pattern.
	tag: tag:astropy.org:astropy/table/table-1.*
	guide_star:
	title: Table of Guide Star Level 2 (L2) Metadata
	description: >
	Table of guide star metadata from input calibrated
	rate image files. Guide star metadata describe the star and
	window used for guiding on that particular Wide Field Instrument
	(WFI) detector.
	tag: tag:astropy.org:astropy/table/table-1.*
	instrument:
	title: Table of Instrument Level 2 (L2) Metadata
	description: >
	Table of instrument metadata from input calibrated
	rate image files. Instrument metadata contain information about
	the Wide Field Instrument (WFI) detector that the data
	correspond to as well as the optical element used during the
	observation.
	tag: tag:astropy.org:astropy/table/table-1.*
	observation:
	title: Table of Observation Level 2 (L2) Metadata
	description: >
	Table of observation metadata from input calibrated
	rate image files. Observation metadata contain programmatic
	identifier information such as the pass number, execution plan
	number, and visit number.
	tag: tag:astropy.org:astropy/table/table-1.*
	photometry:
	title: Table of Photometry Level 2 (L2) Metadata
	description: >
	Table of photometry metadata from input calibrated rate
	image files. Photometry metadata contains pixel
	area information, flux density conversion factors and
	uncertainty.
	tag: tag:astropy.org:astropy/table/table-1.*
	pointing:
	title: Table of Pointing Level 2 (L2) Metadata
	description: >
	Table of pointing metadata from input calibrated rate
	image files. Pointing metadata contains information
	about the telescope pointing.
	tag: tag:astropy.org:astropy/table/table-1.*
	program:
	title: Table of Program Level 2 (L2) Metadata
	description: >
	Table of program metadata from input calibrated rate
	image files. Program metadata contains information
	about the observing program.
	tag: tag:astropy.org:astropy/table/table-1.*
	rcs:
	title: Table of RCS Level 2 (L2) Metadata
	description: >
	Table of relative calibration system metadata from
	input calibrated rate image files. RCS metadata contains
	information about state of the relative calibration system.
	tag: tag:astropy.org:astropy/table/table-1.*
	ref_file:
	title: Table of Reference File Level 2 (L2) Metadata
	description: >
	Table of reference file metadata from input calibrated rate
	image files. Reference file metadata contains information
	about reference files used during calibration of the
	L2 files.
	tag: tag:astropy.org:astropy/table/table-1.*
	source_catalog:
	title: Table of Source Catalog Level 2 (L2) Metadata
	description: >
	Table of source catalog metadata from input calibrated rate
	image files. Source catalog metadata contains information
	about the measured source catalogs for the L2 files.
	tag: tag:astropy.org:astropy/table/table-1.*
	velocity_aberration:
	title: Table of Velocity Aberration Level 2 (L2) Metadata
	description: >
	Table of velocity aberration metadata from input calibrated rate
	image files. Velocity aberration metadata contains information
	about the calculated velocity aberration.
	tag: tag:astropy.org:astropy/table/table-1.*
	visit:
	title: Table of Visit Level 2 (L2) Metadata
	description: >
	Table of visit metadata from input calibrated rate
	image files. Visit metadata contains information
	about the dither, visit type, and planned exposures.
	tag: tag:astropy.org:astropy/table/table-1.*
	wcsinfo:
	title: Table of WCS Info Level 2 (L2) Metadata
	description: >
	Table of World Coordinate System (WCS) info metadata from
	input calibrated rate image files. WCS info metadata contains
	summary information about the sky footprint and relationship
	with the detectors.
	tag: tag:astropy.org:astropy/table/table-1.*

that background, cal_step and source_catalog should be populated in the image files. Indeed we see these being force populated when generating a fake image model to test the L3 pipeline code in RCAL: https://github.com/spacetelescope/romancal/blob/d97b36297c715c1cfc0c6c2a0f7d9a50ec03c4d1/romancal/conftest.py#L239-L241. This is because create_fake_data only populates fields listed in required, so even though the L3 code references these they aren't populated because they aren't marked as required.

I understand that his is because several of these metadata fields are not populated at the end of several of the L2 pipeline steps once they start passing an ImageModel around as those steps are often calculating and filling this information in. Since each step needs to be able to write a valid datamodel out incase people are running segments of the pipeline, those are omitted to enable that process.

The simplest approach would be to have the current wfi_image and then a second schema, say wfi_image_product(?), which is what the final step of the L2 pipeline produces. That schema can simply use an allOf to that contains a required block which finalizes all the required outputs of the L2 pipeline.

I haven't looked closely at the L3 pipeline so a similar issue might need to be addressed there too.

[schlafly]

I think we should probably require statistics and initially populate it with NaNs / invalid data.

For the background, this really shouldn't be required; we don't want it in our L2 products and it only comes in during the L3 pipeline.
For cal_logs, the pipeline assumes it exists so it can append logs. I don't think we want to require the logs and we don't archive them, so that would say we continue to special case this.
For cal_step, we probably should require this. Let me confer with Tyler, though.

That would only fix one of your three cases, though. Note that the linked lines are about background, cal_step, and cal_logs, not source_catalog, though I agree that source_catalog can end up in a similar place. That's not one that we would want to require; tweakreg uses it but nothing else does, and we don't require it at present.

If we decided that we did want to archive these, I guess I would make them required and populate them with NaNs or empty strings initially.

[WilliamJamieson]

I think we should probably require statistics and initially populate it with NaNs / invalid data.

For the background, this really shouldn't be required; we don't want it in our L2 products and it only comes in during the L3 pipeline. For cal_logs, the pipeline assumes it exists so it can append logs. I don't think we want to require the logs and we don't archive them, so that would say we continue to special case this. For cal_step, we probably should require this. Let me confer with Tyler, though.

That would only fix one of your three cases, though. Note that the linked lines are about background, cal_step, and cal_logs, not source_catalog, though I agree that source_catalog can end up in a similar place. That's not one that we would want to require; tweakreg uses it but nothing else does, and we don't require it at present.

If we decided that we did want to archive these, I guess I would make them required and populate them with NaNs or empty strings initially.

This issue has nothing to do with cal_logs so it is outside the scope of discussion, I linked to the entire sections of the relevant schemas. The issue is about statistics, background, cal_step, and source_catalog. If I understand your comment correctly, it boils down to:

• make statistics required and let it be filled with invalid data.
• make cal_step required and let it be constructed as we would expect.
• If background shouldn't be in L2 products then we should probably remove it from the individual_image_meta tables.
• If source_catalog shouldn't be in L2 products then we should probably remove it from the individual_image_meta tables.

If we do want the background and source_catalog in the tables then we should have a new datamodel that represents the image models post L2 pipeline with L3 modifications. wfi_l3_image or something like that.

[schlafly]

background needs to stay in the individual_image_meta. It's not in the L2 products since it's computed by the L3 pipeline and added to the individual exposures at that time, at which point it goes into the L3 individual_image_meta.

source_catalog will be in almost all L2 products we make. I'm okay with setting it to have some sentinel value / junk value but it's just not obvious to me what having the field with junk vs. not having the field at all buys.

Agreed about statistics and cal_step.

I brought up cal_logs since it was one of the extra fields you linked here
https://github.com/spacetelescope/romancal/blob/d97b36297c715c1cfc0c6c2a0f7d9a50ec03c4d1/romancal/conftest.py#L239-L241
while source_catalog was not.