Use runtime to select the hardware variant

[sonnyp]

As part of #182, this is a WIP/attempt to stop building multiple images and use runtime config to select adafruithat or planktoscopehat

On the base image:

• PlanktoScope/hardware.json is not present anymore by default
• PlanktoScope/config.json is not present anymore by default
• The hardware controller is not enabled by default

After setting the hardware version in the dashboard

• PlanktoScope/hardware.json is created (that was already the case)
• PlanktoScope/config.json is created (already the case)
• The hardware controller is enabled
• The hardware controller is restarted (that was already the case)

Flow

What's new:

• "Choose default software config"
• "Enable hardware controller"
• the catch and set msg.payload (to JSON null) are new

Todo

1. solve hardware.json
2. solve config.json
3. Why is home not showing hardware type select?
4. replicate on adafruithat flow
5. solve default image name "xxx+.zip"
6. test adafruithat
7. test planktoscopehat
8. figure out https://github.com/PlanktoScope/install.planktoscope.community

Test

Use the image of this PR or

cd ~/PlanktoScope
sudo systemctl disable --now planktoscope-org.device-backend.controller-planktoscopehat.service 
sudo systemctl disable --now planktoscope-org.device-backend.controller-adafruithat.service 
rm hardware.json config.json 
git checkout build-single-image # this pr
git pull
sudo systemctl stop nodered

[sonnyp]

This will be sudo systemctl enable planktoscope-org.device-backend.controller-adafruithat.service on the adafruithat flow

But otherwise it should be the same thing

[sonnyp]

unrelated but I forgot to remove it in #577

it can also be removed in the adafruithat - it's the label for the "stepper type" hardware setting

[sonnyp]

The last blocker is the selection of the node-red flow which happens in https://github.com/PlanktoScope/PlanktoScope/blob/43289d154eada28b74364a0c65c646f123ca8443/software/node-red-dashboard/settings.js and reads hardware from /usr/share/planktoscope/installer-config.yml.

I see 2 options

A. Merge the adafruithat and planktoscopehat flows

Risky, possibly requires merging adafruithat and planktoscopehat backend
Flows are already quite complex.

B. Move hardware selection to device-portal

This is actually the landing page of the PlanktoScope so I think it's a natural move. There is also a proposal for first boot wizard PlanktoScope/device-portal#54

device-backend and node-red would remain disabled/stopped until hardware.json and config.json are created by device-portal

flows would become significantly simpler

That's my favorite but I'm a bit uncomfortable with device-portal being written in Go as only @ethanjli seems to be familiar with it and may have much less time for PlanktoScope soon.

device-portal seems fairly simple, perhaps a quick rewrite before adding more to it would make sense? Happy to volunteer. We can consider merging that with backend in the future.

@ethanjli WDYT?

[ethanjli]

B. Move hardware selection to device-portal

I'd be happy to assist with any rewrite you want to do of device-portal, because I'd like to eventually transition out of maintaining the PlanktoScope OS's landing page. Assuming we don't add a first-boot wizard (or other interactive functionality) to device-portal, the migration will be fairly simple: right now we're just getting a bit of information from various sources (e.g. HTTP request headers, so we can detect the hostname being used to access the landing page) and then passing that to a single web page template. So if you want to do a rewrite of device-portal in the future, I'd prefer to find some lower-effort way (outside of device-portal) to (at least temporarily) provide the minimal subset of "first-boot wizard" functionality which we'd need for this current PR.

@ethanjli [...] may have much less time for PlanktoScope soon.

To put into writing something I had previously only mentioned verbally (and in much less detail), I'd like to focus on lower-level/reusable OS subsystems, such as maintenance of the CI workflows for OS builds, working on the software update system, improving networking-related behaviors, etc. Much of that work is already being directly reused in a downstream project (e.g. by importing packages from pallet-standard, and by maintaining a slightly-diverged copy of CI workflow definitions and OS setup scripts I had written for the PlanktoScope repo) which I will be taking active (employed) responsibility for as part of the openUC2 project. Wherever there continues to be sufficient alignment between the RPi-based OSes for openUC2 and PlanktoScope, I can work by upstream-first contributions (e.g. fixing problems in PlanktoScope OS and then propagating them to openUC2's OS by bumping package imports wherever possible, or by manual edits of copied files otherwise) and/or by eventual upstreaming of improvements from downstream. So I think there may be certain subsystems of PlanktoScope OS that I will be able to sustain (employed) time to help maintain. Of course, plans may continue to evolve over the next several months, so nothing is guaranteed on my end for any planning on your end - but I can keep you updated as any changes become clear.

A. Merge the adafruithat and planktoscopehat flows

Risky, possibly requires merging adafruithat and planktoscopehat backend

My long-term hope would be to merge the adafruithat and planktoscopehat flows, but I agree with you that it's too big/risky of a change to do it all in this PR. I think we'd first want to integrate a separate PR which unifies the entrypoint to the Python backend. Now that I think about it, perhaps the backend merger could be done incrementally by the following steps:

1. Change the entrypoint for the backend to a script (perhaps at {repo root}/device-backend/control/main.py ?) which loads hardware.json and then selects {repo root}/device-backend/control/adafruithat/main.py or {repo root}/device-backend/control/planktoscopehat/main.py depending on the contents of hardware.json. Also unify the two systemd services into a single systemd service, and update the relevant parts of the Node-RED dashboards. Would this have any interactions (positive or negative) with PlanktoScope v3.0 hardware support?
2. Incrementally refactor device-backend (e.g. starting top-down by unifying .../adafruithat/main.py and .../planktoscopehat/main.py, and/or starting bottom-up by unifying particular hardware drivers which are literally copy-pasted between the variants) until they're a single codebase. This unification can happen on a separate timeline independent of any changes in Node-RED.

I see 2 options

I just thought of a third option, inspired by your work in #573, and I wonder if it might be reasonable as a temporary solution for a more incremental process of change (or maybe even a longer-term solution): make the base OS image default to a separate Node-RED flow file which is just a first-boot wizard (e.g. in {repo root}/software/node-red-dashboard/setup-wizard). The wizard would populate config.json/hardware.json based on user input, do anything else to set up the system for the adafruithat/planktoscopehat dashboard, and trigger a restart of Node-RED. If/when we make config schema changes in the future (e.g. as part of #290), maybe it could even be used to help migrate older config.json/etc. files to newer schemas (but maybe that's not worth the trouble).

Possible consequences/impacts:

• We won't have to merge the adafruithat and planktoscopehat Node-RED flows in this PR. We can do that later.
• device-portal won't need to change or expand its scope beyond just being a landing page. This helps keep a future rewrite simple, without keeping the current PR blocked.
• We would have a third (or is it fourth now?) Node-RED codebase. Even so, it could be simple: for now it'd just be a subset of what is currently in the home page of the planktoscopehat Node-RED dashboard.
• Our existing Node-RED codebases won't have to contain functionality related to version-switching, except perhaps to switch back to the setup wizard (e.g. by changing hardware version-related fields in config.json/hardware.json back to null and then triggering a restart of Node-RED, perhaps from a red button in the "Administration" page or the "Hardware Settings" page).
• The UX will be basically the same as what we currently have on the planktoscopehat image: go to the landing page, click on the link for the Node-RED dashboard, and go through the first-boot setup for the dashboard.
• Since Thibaut is taking responsibility for the GUI and since he prefers working in Node-RED, this would allow him to be involved in prototyping/implementing new UX things for the first-boot setup process in the future.

If we want to be ambitious with the UX, maybe it's even feasible to modify the Node-RED settings.js file to automatically select the appropriate flow file (setup wizard vs. adafruithat vs. planktoscopehat) based on 1) whether config.json/hardware.json exist, and 2) what information is specified in config.json/hardware.json. Then the first-boot wizard only needs to copy the appropriate json files and make Node-RED restart. Then FairScope can just flash the main lite image and copy the appropriate config.json/hardware.json files onto the SD card in the factory, and then we can stop building fairscope-latest images while still providing a simpler first-boot UX for FairScope customers while still resolving the concerns discussed in #543 (comment).

[sonnyp]

One more option for replacing device-portal; use the new dashboard https://github.com/PlanktoScope/dashboard

node-RED dashboard 2 uses Vue templates so it might be a good candidate - I'll prototype

[sonnyp]

#663