Command-line invocation as part of a pipeline

[davidbenncsiro]

I'm working toward the use of HyperCP as part of a RAMSES TriOS pipeline.

I've been locally modifying run_Sample_Data.py for this purpose, which seems to work fine, at least single-threaded with multi-level (L0 to L2).

My natural inclination would be to make a variant of this with command-line arguments.

I initially started to use Main.py's command-line arguments, but notice that there seems to be a mismatch between the actual parameter values passed to the Command object constructor and the formal parameters of Command.__init__ in Main.py.

So, my question is: what approach would you recommend for a command-line invocation of HyperCP? Some options are:

1. Start with run_Sample_Data.py, generalise and add command-line arguments.
2. Start with the Multi-Level processing option code in the GUI and convert to be command-line.
3. Fix any errors in the command-line handling of Main.py. This may be closely associated with 2.

Option 3 would probably be simplest.

I note also the comment under __main__ in Main.py:

######## This section is not up to date. Scripted calls to Command are preferred. ##########

[davidbenncsiro]

I've made local changes to Main.py (Command class mostly) that works fine, at least for TriOS.

I would be happy to create a branch and put the code through a PR if you are willing.

Patch attached.

hypercp-command-line-patch.txt

[oceancolorcoder]

Ah. You have waded into a long-neglected backwater here. For background, I built HyperInSPACE for use with Satlantic/SeaBird and still primarily use it to process HyperSAS rather than TriOS triplets. So, once I got the multithread, scripted calls to Command working for my data, I allowed the incomplete TriOS multithreading script -- and the even older command-line approach (first introduced by ACRI-ST a few years ago) -- to become "demand driven." And here we are!

Thank you for your efforts to bring the command-line handling of Main.py up to date. It would be great to have this working again, though to use it as a multithread batch, you would obviously need another scripted call of some kind, which is why I've stuck with versions of the run_Sample_Data.py for SeaBird, personally, though I know others prefer the command-line approach.

I would very much welcome your input as a PR. However, I would request that you first test a few things.

Please keep in mind that raw (.mlb) TriOS data comes in file triplets per collection -- one file for each instrument -- while pySAS, SolarTracker, SoRad, and DALEC have single files per collection. This leads to some minor challenges in building and processing python lists of raw directory listings that can be stepped through in increments of 1. (This obviously ceases to be a problem when batching from L1A upward.) I would therefore request that you confirm that your edited version can still accomplish the following:

1. run_Sample_Data.py multi-level scripting of a single-file RAW directory (such as Data/Sample_Data/pySAS/RAW) and a multi-file RAW directory of HyperSAS-like data, such as pySAS. I can provide you with additional raw files for you to place in Data/Sample_Data/pySAS/RAW for this test.
2. run_Sample_Data.py single-level, multithread scripting of L1BQC to L2 for pySAS.
3. Command-line batch of the pySAS sample data with single-level and multi-level processing.
4. 1-3 for the Sample_Data/Manual_TriOS dataset.

Are you okay with testing these prior to submitting a PR? I really appreciate your interest and effort on this!

[davidbenncsiro]

Thanks @oceancolorcoder. Yes, I did wonder whether what I had changed would generalise beyond TriOS.

Happy to help. I just need to be a bit careful about balancing time spent on this vs the project-specific bugs and issues I need to work on. I'm sure you know the problem. :)

Yes, please provide pySAS raw files for this test, perhaps as an attached zip here, or in the repo?

When the time comes for the PR, is your group's practice to work on a branch from dev or fork? Let me know what your OSS collaboration approach is.

[oceancolorcoder]

@davidbenncsiro, I understand the struggle to maintain a productive balance very well, though I wouldn't say I've always achieved it.

There are currently three types of raw sample data files, together with calibrations and configurations, provided in the repository: Manual_TriOS, pySAS, and SolarTracker. The latter two are automated SeaBird systems. Data files (including ancillary) are provided in HyperCP/Data/Sample_Data and configurations and calibrations in HyperCP/Config. These sample files are what we typically do most of our testing and training on. The SolarTracker is somewhat redundant to the pySAS, so testing on pySAS and TriOS should suffice.

If you think you would want to make significant contributions to the project in the future, and would like to formalize the collaboration for any reason, we encourage you to review the HyperCP Collaboration Guidelines provided with the repository. This is not a process we typically pursue for an individual, minor change, but we would consider it if it helped on your end. Thanks again.

[davidbenncsiro]

@oceancolorcoder, thanks. I have read over the Collaboration Guidelines.

I am acting on behalf of an organisation and would have to pursue any formalisation in that context. If the fork+PR approach outlined there is acceptable for small fixes or improvements, I'm happy to take that approach.

In the worst case, I can do the testing and provide you with the final patch. Whatever works best.

I'm certainly open to formalising the collaboration if my organisational collaborators endorse that.

[oceancolorcoder]

@davidbenncsiro a contribution of this scope would not normally necessitate a formal abstract submission, but I wanted to inform you of that option if you chose, particularly if you had longer term plans to participate in development. I am happy to handle this update as a simple fork-PR, and would gladly welcome your contribution. I'm happy to iterate with you on the testing and debugging.

[davidbenncsiro]

Thanks @oceancolorcoder

[oceancolorcoder]

@davidbenncsiro did you have any updates on the command line patch that I could relay at this week's meeting tomorrow? Cheers,
Dirk

[davidbenncsiro]

@oceancolorcoder First, apologies for the silence for 3 weeks. This is still very much on my list of things to do. Other tasks have taken priority is all (a very broad statement I know).

I do have a patch and would be happy to fork/branch with it to get the ball rolling, if you are happy (which I think you are). What I have not yet done is to test for anything other than TriOS, which I was going to do before proceeding.

Happy to do the fork/branch right away and go from there if you like, then proceed with more testing. In the meantime, I have attached the patch I've been working with, so you have something (hopefully!) by the time of your meeting.

hypercp-issue-312.txt

[oceancolorcoder]

@davidbenncsiro I understand completely. That all sounds great. Just submit a PR to nasa:dev from your fork once you've completed testing. Thanks!

[oceancolorcoder]

Hi @davidbenncsiro, just doing some housekeeping here and noticed this old issue. Any updates you'd like to submit as a PR to nasa:dev? Cheers!

[davidbenncsiro]

Hi @oceancolorcoder, interestingly, I've started looking at this again in the last week. I am likely to have either something to contribute or questions or both, soon, i.e. within the next few weeks.