Fix GNSS data retrieval and downloading functions#794
Conversation
Updated the get_stats_by_llh function to prioritize IGS20 coordinates over legacy data. Enhanced the download_UNR function to include a fallback to the legacy archive when downloading tropospheric delays. Context: There appears to be ongoing migration of data in the UNR archive such that data from much of the eastern hemisphere is disrupted after 2023. This PR is intended to circumvent this.
Added entry for PR #794 to address data migration issues in the UNR archive affecting the eastern hemisphere.
| url_igs20 = f'{baseURL}gps_timeseries/IGS20/llh/llh.out' | ||
| col_names = ['ID', 'Lat', 'Lon', 'Hgt_m'] | ||
|
|
||
| # Read legacy list |
There was a problem hiding this comment.
@sssangha would it make more sense to wrap all of the legacy calls within a single try/except that checks for stations for which IGS20 was successful for all stations and then pulls legacy data only for the subset of stations that are not available? That way the IGS20 is still the default and all the work needed if legacy stations are required is skipped unless they are actually needed.
There was a problem hiding this comment.
Hello @jlmaurer I cleaned up the logic a bit here to streamline it and lead with igs20, however we don't know what is missing from igs20 from the subset alone since all references to missing stations are gone. So I need to also pull in the legacy file to be sure nothing slipped through the cracks
| # Try IGS20 first | ||
| filepath = download_url(url_igs20, save_loc) | ||
|
|
||
| # If IGS20 is missing, try legacy |
There was a problem hiding this comment.
Does this work if a subset of stations is available but others are not?
There was a problem hiding this comment.
Good catch! Before if a user requests a list of 10 stations and the 3rd one is missing from both archives, the script will crash immediately. It will fail to process the remaining 7 stations, even if they are perfectly valid and exist in the archives. This was the existing behavior.
Since we're iterating over stations/years, we should not raise an exception that halts the entire script. Instead, I tweaked it now to log a warning that the specific station/year failed, and use continue to move on to the next iteration.
Added logging to test cases for download_UNR function and updated assertions.
|
@jlmaurer following up on this. Do my adjustments/comments address your concerns? Thanks |
2 participants
Updated the get_stats_by_llh function to prioritize IGS20 coordinates over legacy data.
Enhanced the download_UNR function to include a fallback to the legacy archive when downloading tropospheric delays.
Context: There appears to be ongoing migration of data in the UNR archive such that data from much of the eastern hemisphere is disrupted after 2023. This PR is intended to circumvent this.
Screenshots (if appropriate):
Type of change
Checklist: