Challenge 10 - A Web Application for Interactive Visualisation of Verification Results

[RubenRT7]

Challenge 10 - A Web Application for Interactive Visualisation of Verification Results

Stream 1 - Data Visualization and visual narratives for Earth Sciences applications

Goal

Develop a single-page web application and associated backend to allow the flexible exploration and plotting of scientific data. The primary use case will CAMS forecast verification data, i.e. making plots that compare forecasts with observations, but ideally the system will be general enough to have wide applicability.

Mentors and skills

• Mentors: Luke Jones, Cihan Sahin, Dimitar Tasev, James Varndell (all ECMWF)

• Skills required:

  • Web development in Javascript (preferably React, Material UI, Plotly)
  • Python

  ---

Challenge description

The Copernicus Atmospheric Monitoring Service (CAMS) makes daily forecasts and historic reanalyses of the planet's atmospheric composition, e.g. pollution, dust, greenhouse gas concentrations, etc. Verification is the process of comparing computer model forecasts with real-world observations so we can see how well the model is performing.

Our verification software currently spends a long time generating a huge number of plots, most of which don't get looked at. A solution to this problem would be a web application which would display a series of menus allowing a user to select precisely what they are interested in, and then allows them plot the associated data on demand.

This would necessitate a single-page frontend application with a Python-based backend. In order to be consistent with our other developments we envisage the frontend written in Javascript, preferably using Material UI on top of React. The backend could use Django, Flask, or anything similar. The job of producing the actual verification data would be ours - your backend would just have to call our Python to obtain the data, before preparing it for transfer to the frontend and rendering it as a plot.

Ideally, the code will be as general as possible. That is to say, the names and contents of the navigation menus will not be hard-coded anywhere in your app but will be provided by our backend code. If done this way this app would be useful beyond the immediate use case and could find widespread use across ECMWF.

We anticipate you would probably use Plotly to make the plots. The types we need include:

• Line plots of verification statistics (root mean square error, bias, etc.) as a function of time
• Line plots of verification statistics as a function of forecast lead time
• Map plots of verification statistics as a function of observation location ( 🌎 )
• Line plots of raw forecast and observation values at individual observation locations ( 📍 )
• 2D "heatmaps" of forecast vs observation values

( 🌎 ) ( 📍 )

These plots should be as interactive as possible, allowing a user to change scale, zoom in, etc. An important feature will be that a click on a point in a map plot ( 🌎 ) would generate a plot showing the time-series at that site ( 📍 ).

The number of features we'd like to have is extensive and accomplishing them all would probably be unfeasible in the time available, so we'll work with you to define a "minimum viable product" (MVP) and a list of "nice-to-haves" that could be completed if time allows. It may be important to consider the nice-to-haves even if they can't be achieved as they may influence the design of the MVP. For example, we would very much like the option to make a plot non-interactively (i.e. with a batch script that saves the plot to file without ever using a browser) which may have important implications for the system design.

We envisage that the project would start with some initial intensive discussions to communicate the vision and answer questions, with bi-weekly meetings thereafter.

This project, if successful, would be invaluable to CAMS scientists and possibly to many others in ECMWF. We really hope you can help us with our goals and that you get some rewarding experience in return!

[hadekale]

Hi! My team and I are very interested in making a proposal for challenge 10.

We are a team consisting of high school students from Turkey currently in the process of graduating, however we have plenty of free time thanks to the way our school term is structured. What we don't have is plenty of experience, though we aim to make up for it with our dedication and passion. During our discussion on how to make a good proposal, a few points of uncertainty arose.

Firstly, we are aware of the need to come up with a timeline and plan in general. Yet it is not clear how to make a plan without having a clear vision, which will apparently be discussed after the application as "(you will) work with (us) to define a "minimum viable product" (MVP) and a list of "nice-to-haves" that could be completed if time allows." and as "the project would start with some initial intensive discussions to communicate the vision and answer questions.". From this, we understand that making a proposal for the structure of such a web page is practically pointless, as it will be discussed later with the mentors.

The lack of information on proposal details (as well as the desire to prove our abilities) pushed us in the direction of making a demo. The idea is to make a basic demo with exemplary data showing the plot types given in the github page of the challenge. Would this indeed be helpful or would you suggest us to rather focus on another aspect? Should we still come up with some suggestions for the web page structure or what would you recommend us to mainly include in our proposal? We would appreciate some guidance on this issue.

Furthermore, you have stated that the verification software generates unnecessary plots and you want to rather supply the user with the plot on demand. If the plots generated by your verification software are publicly accessible we would like to have a look at them.

As an explanation, we searched the data page of the CAMS website however couldn't find any pre-generated verification plots. The plots in here, which are indeed verification plots, seem to be generated with server-side data and client-side visualisation via highcharts. If you mean there is pre generated data for cities, this normally shouldn't be a problem as the data must be a few megabytes per date (3 [models] * 4 [pollutants] * ~50 [cities] * 151002 [data points per city per 4 day period] * 4 [bytes per float] = 7.200.000 bytes ~7 megabytes and if done dynamically, 1/6 as much).

Of course, the application could be made more versatile, nonetheless, we fail to see why it would take your software a long time to generate the data for the plots on this website. As such we ask for a clarification of your plot generation problem, if this is indeed important.

Thanks for your answer in advance,
Hakan Deniz Kale, on behalf of my team

[LukeJones123]

Hi Hakan,

Thanks for your interest. I'm very happy to answer your questions. You're right that a very detailed plan is not practical until we've discussed how to proceed in detail, so just a rough outline is fine. Perhaps it could just be the order in which you expect to work on things with an estimate of when you would expect to have some kind of bare minimum functionality, when a prototype MVP, when a more polished version of the MVP, etc. Since you have multiple people in your team you could outline who is going to work on what. We'd just like to see that you've structured your thoughts a bit. It's not too important - plans often have to change anyway. More important are your technical skills.

You're right that there's not a lot of detail in the proposal - a word limit was imposed to stop us writing vast amounts of text. I will now demonstrate how much I can write when there is no word limit :)

Unfortunately we can't share the current web page we currently use to look at the plots because it's external, but here's a screenshot...

You can see all the menus on the right. For us, an "experiment" is a model configuration - a specific version of the code and settings used to make a forecast. We have the current operational configuration (making today's official forecast) and we are also constantly making new experimental configurations to test new ideas. Each of these experiments is assigned an ID ("oper_ebas" is the one shown here) and that's what can be selected in the Experiment menu. The Field-group is the parameter (or list of parameters if we're superimposing more than one) to show. Sometimes an additional wavelength menu is shown if the parameters are dependent on that (e.g. aerosol optical depth). You can see check-boxes for the various statistics we plot. The "As a function of" radio buttons allow us to switch between statistics as a function of site (maps), time and forecast lead time. Time-series plots for individual sites can be shown by clicking on a site in a map plot or selecting the site in the Site menu on the bottom right. Note that the "Show", "As a function of" and "Area" widgets aren't relevant when viewing a time-series for just a single site, so clicking on those takes you back to an area/time mean statistic plot.

It might be worth explaining a little bit more about the experiment menu actually. It won't just show single experiment IDs because when we compare experiments we have to calculate all the statistics together at the same time to ensure the results for each experiment are exactly comparable, so the string entries in the Experiment menu may be single IDs or groups of them. We also may want to verify the same group of experiments more than once with different input observations or settings, so a qualifier can be added to group of experiment IDs to distinguish them in the menu (and file system). I call this whole thing - a list of N experiment IDs plus an optional qualifier - the "experiment set". Before the user arrives at this page they are shown a landing page listing all the available experiment sets and they click on the one they're interested in. That takes them to this page, where the Experiment menu shows the subset of experiment sets which have a plot for the current selection (date period, field group, etc). So in the screenshot example, the Experiment menu won't show any experiment sets which don't have plots for December 2021.

We don't necessarily need an exact replica of this webpage but this gives you an idea of what kind of functionality we're looking for. You're correct that the link you quoted to policy.atmosphere.copernicus is also an example of this type of work. Yes, those plots do appear to be generated on demand but the interface is a bit limited. Other examples of this type of thing are the CAMS Global Validation Server: https://global-evaluation.atmosphere.copernicus.eu and AeroVal: https://cams2-82.aeroval.met.no. Both of these make plots on demand. I like AeroVal but it uses proprietary software (Highcharts). Our solution must only consist of free software.

If you want to make a demo then that could be helpful but please bear in mind that we can't say who will be accepted for the challenge at this stage so there may be no financial reward for that. You're welcome to borrow our existing web page structure, that of any of the other examples, or come up with your own if you think you can make improvements. One nice-to-have but potentially difficult feature might be the ability to split the page so that, for any single widget, you can show two values simultaneously. For example, if I want to compare SO2 and O3 side-by-side I could split the Field-group widget into two. Then I select SO2 in one and O3 in the other and I see plots for both, perhaps even superimposed if I want. And if I change the date period, it changes for both. I don't mention this because it's something we urgently need but to illustrate that there's room for new ideas and imagination in the interface. Any new ideas that help us investigate, compare and analyse more quickly and easily are welcome, but not essential.

On the issue of the number of plots and the time taken to generate them, the example you found shows many fewer plots than we need. The total number of observational sites that we use is probably of the order of 10,000. If you make a timeseries plot for every site for every month of a 20 year reanalysis (240 months) for, say, 8 different forecast parameters you can see that you're already have many millions of plots, and that's just for the single-site timeseries plots of one experiment. When you consider the fraction that will actually be viewed will be probably less than 1% you see that on-demand plotting makes more sense.

Plus, pre-generated plots are static - you can't zoom, change the colours, scales, etc. Much better to just generate the underlying data, make the plots on demand and get exactly what you want.

Well that's a long answer but it was a long series of questions :) I hope I've answered them all and please don't hesitate to ask any more or for further clarification.

Thanks,

Luke.

[hadekale]

Thanks again for the answer @LukeJones123 ! It was quite comprehensive and we think we have a quite good picture of the project in mind.

One part that we are curious about though is the data itself and the format it will be provided in. In the demo, we want to use the data from the Atmosphere Data Store to generate the displayed plots. The data here is provided in the GRIB or NetCDF formats.

We wanted to ask if the principal data provided for the application will be in the same formats.

With curiosity,
Hakan Deniz Kale

[LukeJones123]

Hi Hakan,

At the moment the data output by our verification system is in ASCII text format but it could easily be converted to netCDF. GRIB wouldn't be appropriate.

The system currently outputs a statistics data file for each plot it intends to make, as well as files containing the "raw" forecast and observation (fc-ob) value pairs that were used to create those statistics. In order for the new system to be flexible we wouldn't want to output pre-calculated statistics, because that immediately limits the plots you can make. Instead we would want the system to use the raw fc-ob pairs to calculate the required statistics "on the fly", so we anticipate working from just the fc-ob pairs files. Here's an example of one...

pairs_site_oper_global_20231201-20240229_500_inst.txt

It contains the fc-ob values for a single parameter for a single experiment verified for a 3-month period. If more parameters are verified or there's more than one experiment in the "experiment set" then the data for these will be in separate files. You can see the format is a lot of header lines of the form "#name=value" containing metadata, the last of which is the column titles, followed by a table of column-separated data giving validity (wallclock UTC) time, forecast range (hours since start of forecast), site ID, site latitude, site longitude, forecast value and observation value.

To make a plot of, for example, bias as a function of location, you would have to calculate the average of fc minus ob for each site for a restricted list of forecast ranges. To make a plot of bias as a function of forecast range, you would have to calculate the average fc minus ob for each forecast range (for a restricted list of sites if limiting to a geographical area).

I hope that's clear.

Luke.

[LukeJones123]

Actually, you might find it useful to see a few of these files and it would also save you having to make your own test dataset, so here you go...

example_data.tar.gz

I hope you can unpack a tar.gz file. If not, let me know your preferred format. You'll find in there data for two months (date period is given as yyyymmdd-yyyymmdd) for an experiment set using two experiments ("oper" and "mc0074"). The verified parameters are aerosol optical depth ("aod"), carbon monoxide (CO), nitrogen dioxide (NO2) and sulphur dioxide (SO2). AOD is a function of wavelength so those files have an extra field in the name giving the wavelength in nanometers. The file naming convention isn't important - it could be changed. What is important is that the file naming convention and the software are flexible with regards the "dimensions" of the data. Here the dimensions are experiment and parameter for some data files, but experiment, parameter and wavelength for others. In future we need the freedom to be able to add other dimensions if we need, so this can't be fixed.

Any questions, let me know.

Luke.