NSRDB API returning bad data for tmy-2017

[kandersolar]

Was a change made to the tmy-2017 NSRDB dataset? We noticed part of our test suite failing (pvlib/pvlib-python#886 (comment)) because the TMY API is returning somewhat different data than it used to. It could be that other datasets are affected as well; our tests just use tmy-2017 as an example, so that's the only one I've looked at.

Here's a script to reproduce what I'm seeing. It shows the difference from the old version of what the API returned (stored as a CSV in our test suite). Seems like some, but not all, of the data columns have changed. It also appears that the date labels are shifted by one day because two days are marked Feb 28.

Click to expand

import requests
import pandas as pd
import io

# read older dataset from before the change
url_old_data = 'https://raw.githubusercontent.com/pvlib/pvlib-python/master/pvlib/data/test_psm3_tmy-2017.csv'
df_old = pd.read_csv(url_old_data, index_col=0)

# retrieve current dataset from API
params = {'api_key': API_KEY,
          'full_name': NAME,
          'email': EMAIL,
          'affiliation': AFFILIATION,
          'reason': REASON,
          'mailing_list': 'false',
          'wkt': 'POINT(-108.5449 40.5137)',
          'names': 'tmy-2017',
          'attributes': 'air_temperature,dew_point,dhi,dni,ghi,surface_albedo,surface_pressure,wind_direction,wind_speed',
          'leap_day': 'false',
          'utc': 'false',
          'interval': 60}
url = 'https://developer.nrel.gov/api/nsrdb_api/solar/nsrdb_psm3_tmy_download.csv'

response = requests.get(url, params, timeout=30)
fbuf = io.StringIO(response.content.decode('utf-8'))

# pop off the header lines
fbuf.readline()
fbuf.readline()
df_new = pd.read_csv(fbuf)

print((df_old == df_new).all())
print(len(df_old.loc[(df_old.Month==2) & (df_old.Day==28)]))
print(len(df_new.loc[(df_new.Month==2) & (df_new.Day==28)]))

[PjEdwards]

Hi Kevin, yes we have been rolling out an updated version of these APIs, however we did not anticipate a change in output data. Let me contact the team and see if there are any known differences between the files in HSDS and the files sitting on discs at NREL.

I've tried a few other points and I'm not able to replicate the Feb 28th issue. Still debugging that one. I'll be in touch!

[PjEdwards]

OK, date formatting side effect when the TMY year for February is a leap year but for March is not. I'll fix that now. Still working on the data variations.

[kandersolar]

Hi @PjEdwards just curious, any news on the data variations? Thanks!

[PjEdwards]

@kanderso-nrel I am unable to track down the details, but there was a disconnect between the most recent version of the TMY data and the version being served through these APIs. The models are being refined periodically and at some point about a year and a half ago we fell out of sync. Going forward we are now serving data for both HSDS and the APIs from the same raw data files in S3 so this won't be an issue in the future. The current data is the most accurate and correct that we have to offer. The version previously served was also valid, but was an iteration or two out of date.

[kandersolar]

Thanks for the info! It makes sense that the current TMY could get out of sync, but I had the understanding that the year-marked TMY data (e.g. tmy-2017) were "set in stone" without any concept of updating over time, so I wouldn't have thought that there was anything to lose sync with in that case. Is that an inaccurate or incomplete understanding?

And a side note -- is there a preference between using the APIs or HSDS for public access?

Thanks again.

[PjEdwards]

You're right, 'set in stone' is what we intended. I don't work on the modeling side so the details are unclear, but essentially the 2017 TMY was run a second time shortly after the first run to create the definitive version, and I was unlucky enough to pick up and copy over the first run in the brief window before the second was available. While we (obviously) weren't aware of this specific situation at the time, moving all of our services to operate on a single set of raw data files was one of our motivations for the updates we recently released.

There is no preference on our side. The HSDS access provides users with a much more flexible way of slicing out data spatially and temporally, and gives access to much larger portions of the data per request. On the other hand, the API is easier to use and delivers the data pre-digested in the CSV files, but the API is rate limited based on hardware limitations* and can only deliver a full year of data per request. Purely personal choice based on use case. I typically recommend folks use HSDS when they want really big chunks of data, i.e. all of New York for all years at max temporal resolution... several TB worth. This would take several days or more of timed requests to fetch via the API within the rate limits, but would be fairly easy to snag going direct to HSDS.

• Incidentally I recently doubled the rate limits for the API requests, and in the near future will be doubling it again when the rest of our new servers come online.

[kandersolar]

Got it. I think from my perspective this is resolved -- feel free to close the issue if you agree.