JOSS paper preparation - #1249
Conversation
|
I will automatically update this comment whenever this PR is modified
|
jules32
left a comment
jules32
left a comment
There was a problem hiding this comment.
Hi! Great work on this Danny! A few commits and some suggestions to consider.
Co-authored-by: Julia Stewart Lowndes <julia@openscapes.org>
There was a problem hiding this comment.
We could symlink this in to our docs!
There was a problem hiding this comment.
@mfisher87, want to create an issue for it?
There was a problem hiding this comment.
@mfisher87 – pinging you again on this :)
I would say let's not wait. We've demonstrated impact and I think that matters more. Alternatively, let's just go 1.0.0 in the short term and be OK with quickly moving to a 2.0.0 release with breaking changes. I think both are fine, but the latter sets more a precedent of maintainers taking the user impact of breaking changes too lightly. |
Co-authored-by: Matt Fisher <3608264+mfisher87@users.noreply.github.com>
I'm fine with either too. I also think the decision could be on hold until one of the two things – (i) co-author reviews/revisions, (ii) development for v1.0.0 – is completely ready-to-go. |
danielfromearth
assigned
betolink,
itcarroll,
chuckwondo,
jhkennedy,
jrbourbeau,
JessicaS11,
battistowx,
asteiker and
Sherwin-14 and unassigned
betolink,
itcarroll and
chuckwondo
|
Hey all, it's been a couple weeks since activity here, so pinging to keep this moving. Would be great to have a complete draft ready to submit before Northern Hemisphere summer! If there's not a specific note next to your username, a general read-through and comments are welcome: @andypbarrett |
jules32
left a comment
jules32
left a comment
There was a problem hiding this comment.
Thank you @danielfromearth ! I've added the award number. Thanks for leading this!
Co-authored-by: Daniel Kaufman <114174502+danielfromearth@users.noreply.github.com> Co-authored-by: Julia Stewart Lowndes <julia@openscapes.org> Co-authored-by: Amy Steiker <47193922+asteiker@users.noreply.github.com>
|
Friendly ping for co-authors who haven't yet had a chance to review (or at least, approve): @andypbarrett @jhkennedy @jrbourbeau @battistowx @Sherwin-14 @betolink @chuckwondo Things have been coming together and I think we are getting close to a complete draft that's ready. Would be great to have everyone's eyes on it, even briefly, before we finalize. Could you each take a look in the next week or two? In particular, please confirm your name, affiliation, and ORCID are correct in the author list. And of course, all other comments welcome. If timing doesn't work, just comment as such so we know where things stand. Thanks! |
Co-authored-by: Jessica Scheick <JessicaS11@users.noreply.github.com>
|
Related: https://earthaccess.zulipchat.com/#narrow/channel/480557-general/topic/JOSS.3F/with/590557057 We're considering / planning going for pyopensci review first, which will give us a stronger review and expedite the JOSS acceptance process if accepted to pyopensci. Thanks @sampottinger for sharing this with me :) |
betolink
left a comment
betolink
left a comment
There was a problem hiding this comment.
I left some comments and suggestions but nothing major, I think this is a good draft so I'm approving as is. Thanks for leading the effort @danielfromearth
|
|
||
| 3. **Access**: Attempts to detect at runtime whether the process is running within AWS `us-west-2` | ||
| and automatically selects the optimal access path -- direct S3 reads for in-region | ||
| access or HTTPS downloads otherwise. Users can manually specify an access path if needed. Files can be opened as `fsspec`-compatible |
There was a problem hiding this comment.
I like the concise way of presenting this, maybe we can add that being format-agnostic and python file-like object compatible makes the library interoperable with the rest of the scientific python ecosystem (aka Pydata/Pangeo)
| open-source tools -- `python-cmr` for search, `fsspec` and `s3fs` for file I/O, | ||
| VirtualiZarr and kerchunk for virtual datasets -- rather than reimplementing their | ||
| functionality. The library's unique contribution is the NASA-specific integration | ||
| layer that binds these tools together. |
There was a problem hiding this comment.
This is the awesomeness, integrating and simplifying the steps a scientist usually do when working with NASA data. Maybe adding an example of time to science reduction both in lines of code and speed through performance optimizations via fsspec and virtualizarr. Tempo or ICESat-2 can be used for this, before N minutes, now N seconds. Before 10 lines of code, now 1.
There was a problem hiding this comment.
Hmm, @betolink, I'm not really sure what would be appropriate for pre-earthaccess code if we want that compared with earthaccess-enabled code... would it be something like this TEMPO S3 Access notebook, which uses boto3, requests, s3fs, etc?
There was a problem hiding this comment.
Sorry for the late reply, yes that would be a good comparison, the boto3, requests, CMR response parsing, s3s credentials etc. @danielfromearth
Thanks for the ping @danielfromearth! I've been traveling the last few weeks and will finally make it in back to the office on Monday. I'll have a look ASAP, but I suspect it's already in good shape judging from my quick glance here. |
Co-authored-by: Jessica Scheick <JessicaS11@users.noreply.github.com>
jhkennedy
left a comment
jhkennedy
left a comment
There was a problem hiding this comment.
Well, it turns out I do have a bit of feedback 😊 . I think it's in a very good place, and really would be fine to submit with or without my feedback.
Other than the specific things discussed below, I have a pretty big concern around publishing this discussing the automatic cloud-detection logic. That's something we know is technically infeasible to do reliably and we've decided to rip out:
https://github.com/earthaccess-dev/earthaccess/blob/main/docs/governance/decisions/231-aws-us-west-2-checking-method.md
So I'd like to either not mention it or abstract that away in the manuscript language.
Since I have a lot of feedback, I could open a PR into this PR with how I'd resolve my comments, if that's easier. Just let me know.
| error, and DAAC-specific configurations further compound the challenge. | ||
|
|
||
| NASA's ongoing migration to the Earthdata Cloud adds further complexity, as researchers | ||
| must now contend with two possible access paradigms, traditional HTTPS downloads and S3-based |
There was a problem hiding this comment.
I think this sentence should be moved up into the previous paragraph before (5) and (6), or part of a stand alone paragraph with (5) and (6).
There was a problem hiding this comment.
would be resolved by https://github.com/earthaccess-dev/earthaccess/pull/1249/changes#r3538571932
| - **earthdatalogin** [@earthdatalogin_r] provides similar authentication and access | ||
| functionality for the R programming ecosystem. The two projects share a common motivation and | ||
| serve as complementary tools for their respective language communities. |
There was a problem hiding this comment.
🤔 are there other R/Julia things we should call out?
There was a problem hiding this comment.
Happy to add others - anyone know?
| NASA's Earth science data archive is one of the largest and most diverse collections of | ||
| Earth observation data in the world, used by over ten million researchers, educators, | ||
| and decision-makers globally [@nasa_esds_data_metrics]. However, the complexity of the underlying data infrastructure | ||
| presents a significant barrier to scientific productivity. A typical data access workflow |
There was a problem hiding this comment.
I'm not sure I like how we've ordered the "data access workflow". Right now we have:
- auth
- "discover"
- parse metadata
- sessions + redirects
- is cloud?
- S3 credentials
I think (1) and (4) should be combined and indeed that's how we discuss it on L196
https://github.com/earthaccess-dev/earthaccess/pull/1249/changes#diff-e504eb580b095a7e65428b098183a581e475f0fb316db95287eacd7d4f344424R196
Similarly, (5) and (6) are also optional and only for in-place cloud access with performance constraints or if you want to use S3 aware tools, and really, fit into (1) and (4) as well, which is also discussed this way on L196.
I also think (2) is better described as "search" and (2) + (3) is what I would call discovery. At least for me, I am always parsing metadata as part of what I'd call discovery -- typically searching broadly and then refining with sensor/bands/variable/etc, so that I end up with the actual set of granules I want to use in my workflow. I don't really see why getting the access URLs are special compared to getting any of the other metadata along the way.
We don't talk about data preparation at all, except as features of Harmony and Icepyx, which seems like a missed opportunity.
I think I'd restructure this like:
- Discovery
- Auth (EDL, S3, Sessions + redirects)
- Access
- Data prep (includes virtual datasets and transformations)
which is similar to the Software Design section. Note, I've put auth after discovery since you generally only need it to access data, unless you're trying to discover restricted datasets... so It could go before or after discovery, I think it just flows a little better narrative-ly after, but 🤷 .
There was a problem hiding this comment.
I think this is a great revision you're suggesting, @jhkennedy. My immediate gut thought about ordering was that Auth goes before Discovery, but that's mostly because in code I usually run earthaccess.login() before anything else. Your point about it really only being needed to access data (restricted datasets aside) is enough for me to feel comfortable putting it where you suggest. And you're right that as written the list doesn't totally align with how we describe things later at line 196 – auth and the session/redirect handling are really one concern, and the cloud-detection and S3-credential steps are really a single conditional concern.
The one spot where I hesitate is restructuring this Statement-of-Need section to exactly match the four-layer Software Design section (Discovery / Auth / Access / Data prep). I believe the two sections serve different purposes: Statement-of-Need showing how much a researcher has to deal with today, where part of the point is the number of separate steps involved, while Software-Design shows the cleaner abstraction earthaccess provides on top of that. If both sections use the same four-stage structure, it seems to me that the problem statement gets slightly weaker as the two sections overlap more.
Revised suggestion
So, I suppose I'd lean toward
- keeping it as a numbered list of steps (still one fewer than before [5 instead of 6]: Search / Metadata / Auth / S3 setup / Access),
- putting discovery first (before auth) as you suggested, and
- fixing the other clarification points you raised (auth + session handling as one thing; cloud-detection and S3-credential steps merged into a single conditional step).
Even though I agree "search" and "parse metadata" are together discovery, I think part of the challenge in other workflows is that they are handled separately – and so that can be part of the statement of need. So, how about something like this?
A typical data access workflow requires a researcher to: (1) search the CMR API to discover relevant datasets and granules; (2) parse the returned metadata to extract the appropriate access URLs; (3) authenticate with NASA Earthdata Login and maintain authenticated sessions that handle tokens and cross-domain redirects; (4) for cloud-hosted data they wish to access directly while working in-region, determine that the data are cloud-hosted and obtain and periodically renew temporary AWS S3 credentials; and (5) retrieve the data, either by downloading it over HTTPS or streaming it directly from S3.
On data prep
On data preparation, I'm a bit hesitant to add it as a stage here, since transformation/subsetting isn't really something earthaccess itself does (as opposed to Harmony/icepyx). Listing it as a step might even read as if we're highlighting a gap in what earthaccess covers. I think we'd be better off referring to it in passing in the State-of-the-field section, which connects to your other comments there about tying tools to workflow steps. What do you think? Do you still want a data-prep step in this list, or should we keep it light?
Other Thoughts? Happy to move towards the four-stage structure you suggested earlier, if you think that's clearer than this revision.
There was a problem hiding this comment.
Great discussion! I really like where this is going. I'm going to build on @danielfromearth's suggested rewrite:
A typical data access workflow requires a researcher to: (1) search the CMR API to discover relevant datasets and products; (2) parse the returned metadata to identify specific granules and extract the appropriate access URLs; (3) authenticate with NASA Earthdata Login and maintain authenticated sessions that handle tokens and cross-domain redirects; (4) for in-region, Earthdata cloud-hosted data, confirm that the data are cloud-hosted and obtain and periodically renew temporary AWS S3 credentials; and (5) retrieve the data by downloading it over HTTPS, streaming it directly from S3, or passing the information into a third-party package for customization services. Researchers must know enough about the technical implementation of multiple APIs to perform these steps accurately, each of which introduces opportunities for error. DAAC-specific configurations and NASA's ongoing migration to the Earthdata Cloud compound the challenge, as researchers contend with multiple access paradigms, often within a single analysis workflow.
Then we start with the "During workshops organized..." as a new paragraph.
There was a problem hiding this comment.
That sounds good to me!
There was a problem hiding this comment.
That sounds good to me as well. The only thing I'd change is (4):
(4) for Earthdata cloud-hosted data, obtain and periodically renew temporary AWS S3 credentials
or to keep the in-region caveat, something like:
(4) for Earthdata hosted in the same AWS region you're working in, obtain and periodically renew temporary AWS S3 credentials
because of the reasons I mentioned here:
#1249 (review)
I have a pretty big concern around publishing this discussing the automatic cloud-detection logic. That's something we know is technically infeasible to do reliably and we've decided to rip out:
https://github.com/earthaccess-dev/earthaccess/blob/main/docs/governance/decisions/231-aws-us-west-2-checking-method.mdSo I'd like to either not mention it or abstract that away in the manuscript language.
Also, within the narrative, deciding which URLs to work with happens in (2) so by (4) you've already made this decision anyways.
There was a problem hiding this comment.
How about a slight change of wording to keep the data outside of Earthdata (as the NASA program/project), and to keep the researcher/3rd-person point of view:
"for Earthdata Cloud data hosted in the same AWS region as the researcher's compute environment, obtain and periodically renew temporary AWS S3 credentials
@jhkennedy , how does that sound to you?
| presents a significant barrier to scientific productivity. A typical data access workflow | |
| presents a significant barrier to scientific productivity. A typical data access workflow requires a researcher to: (1) search the CMR API to discover relevant datasets and products; (2) parse the returned metadata to identify specific granules and extract the appropriate access URLs; (3) authenticate with NASA Earthdata Login and maintain authenticated sessions that handle tokens and cross-domain redirects; (4) for Earthdata Cloud data hosted in the same AWS region as the researcher's compute environment, obtain and periodically renew temporary AWS S3 credentials; and (5) retrieve the data by downloading it over HTTPS, streaming it directly from S3, or passing the information into a third-party package for customization services. Researchers must know enough about the technical implementation of multiple APIs to perform these steps accurately, each of which introduces opportunities for error. DAAC-specific configurations and NASA's ongoing migration to the Earthdata Cloud compound the challenge, as researchers contend with multiple access paradigms, often within a single analysis workflow. |
There was a problem hiding this comment.
That works for me!
nitpicky wise, I don't think users should really care or know about the difference between:
- NASA Earthdata (the project)
- NASA Earthdata Cloud (where we store the data)
- NASA Earthdata Cloud data (the actual data)
All that is just "NASA Earthdata" to me and NASA Earthdata Cloud data is a mouthful. I guess if you want to not use Earthdata there the alternative I would use is "Earth science data" which is what's done on Earthdata Search's landing page:
https://search.earthdata.nasa.gov/
EARTHDATA SERACH
Search NASA's 2.2 billion+ Earth observations
Use keywords and filter by time and spatial area to search NASA's Earth science data
And is how it's termed on the main Earthdata page:
https://www.earthdata.nasa.gov/
EARTHDATA
Your Gateway to NASA Earth Observation Data
The Earth Science Data Systems (ESDS) Program provides open access to NASA’s archive of Earth science data, empowering researchers and decision makers to better understand and protect our home planet.
(I like Earth Science data over Earth observation data because CMR has more than observational data in it)
|
|
||
| # AI usage disclosure | ||
|
|
||
| No generative AI tools were used in the development of the `earthaccess` software; all architectural and design decisions were made exclusively by the authors and contributors. |
There was a problem hiding this comment.
Hmm, is this true anymore? @betolink have you been using Claude for the virtulizarr work?
I wonder if we need to adopt an AI policy and say something like "...developers may use AI tools but are responsible for their contributions...".
There was a problem hiding this comment.
I don't see any problem with modifying this to include development work that leveraged AI. Regarding an AI policy: this question might give another nudge to the AI policy discussion of #1294.
Co-authored-by: Joseph H Kennedy <me@jhkennedy.org>
| year = {2026}, | ||
| publisher = {American Geophysical Union (AGU)}, | ||
| doi = {10.1029/2025JC023511} | ||
| } |
There was a problem hiding this comment.
| } | |
| } | |
| @article{icesat2, | |
| title = {The {Ice, Cloud, and land Elevation Satellite-2 (ICESat-2)}: Science requirements, concept, and implementation}, | |
| journal = {Remote Sensing of Environment}, | |
| volume = {190}, | |
| pages = {260-273}, | |
| year = {2017}, | |
| issn = {0034-4257}, | |
| doi = {10.1016/j.rse.2016.12.029}, | |
| url = {https://www.sciencedirect.com/science/article/pii/S0034425716305089}, | |
| author = {Thorsten Markus and Tom Neumann and Anthony Martino and Waleed Abdalati and Kelly Brunt and Beata Csatho and Sinead Farrell and Helen Fricker and Alex Gardner and David Harding and Michael Jasinski and Ron Kwok and Lori Magruder and Dan Lubin and Scott Luthcke and James Morison and Ross Nelson and Amy Neuenschwander and Stephen Palm and Sorin Popescu and CK Shum and Bob E. Schutz and Benjamin Smith and Yuekui Yang and Jay Zwally}, | |
| keywords = {ICESat-2, Land ice, Sea ice, Vegetation, Climate change, Satellite mission}, | |
| } |
12 participants
Manuscript draft
This PR is intended for revisions and improvements to the manuscript draft being prepared for submission to the Journal of Open Source Software (JOSS).
Paper format: The manuscript is prepared as a Markdown (
paper.md) file with references in apaper.bibfile, following the JOSS formatting guidelines.For a PDF preview: With docker installed locally, a PDF preview of the draft manuscript can be generated, by running the following from the earthaccess root directory (as described in the JOSS guidelines's docker section):
📚 Documentation preview 📚: https://earthaccess--1249.org.readthedocs.build/en/1249/