ssh: add an ssh-dev feature and keep host keys on /var for scarthgap by jetm · Pull Request #276 · avocado-linux/meta-avocado

jetm · 2026-08-13T00:38:15Z

A freshly flashed board has no way in. /etc/shadow ships root:*:, and even
once openssh is installed the image cannot serve SSH at all, so getting a shell
has meant a bespoke sequence per board.

Two commits, two independent gaps.

Host keys were unreachable. sshd resolves HostKey to ${sysconfdir}/ssh,
which on this image is the read-only rootfs. sshd_check_keys cannot create a
key there, so the unit fails, sshd starts keyless, and every client is dropped
at key exchange with Connection reset by peer - a symptom that reads like a
network fault rather than a missing file. On top of that, oe-core's
read_only_rootfs_hook sees no pre-generated key in /etc/ssh and points sshd
at sshd_config_readonly with keys on tmpfs, so the device presents a new host
key on every boot. Both halves are needed: clearing SSHD_OPTS alone still
writes keys to the read-only path, and repointing HostKey alone is ignored
while sshd is running the readonly config.

No build-time switch. ssh-dev is a kas overlay, so any board can ask:

bakar build meta-avocado/kas/machine/<board>.yml:meta-avocado/kas/feature/ssh-dev.yml

Deliberately not an AVOCADO_FEATURE_GROUPS entry. A feature group populates
the feed - networking builds the openssh RPMs and stops there, because the
base image is minimal by design and gains capability through extensions. That
is right for a product and wrong for a bench board, where the point is to have
a shell before any extension has been installed. ROOTFS_IMAGE_EXTRA_INSTALL
is the seam the image recipe already leaves for that.

Dev only - it accepts root with an empty password. The feature file says so at
the top.

Same files as the wrynose version

Byte-identical to #275. The bbappend branches on whether the recipe wrote the
HostKey lines at all, so one file is correct on both openssh versions in the
layer: 10.3p1 writes them from OPENSSH_HOST_KEY_DIR one per key type enabled
in PACKAGECONFIG, so rewriting in place preserves those gates; 9.6p1, which
scarthgap carries, leaves them commented and rewrites only
sshd_config_readonly, so sshd runs on its compiled-in defaults and a sed
matches nothing. Either branch alone fails silently on the other version - the
sed is a no-op on 9.6p1, and an unconditional explicit list would resurrect a
key type PACKAGECONFIG had turned off on 10.3p1.

Verification

Verified on an FRDM-IMX93 against the wrynose BSP: sshd reachable over the
network, and the ed25519 host key fingerprint unchanged across a reboot that
changed boot_id, so the key is reused rather than regenerated.

Not built or booted on scarthgap. The 9.6p1 branch above was derived by reading
that recipe's do_install, not by running it, so the parse and the produced
sshd_config are unconfirmed on this branch.

Added 2026-08-15: a console-only sibling, and a dedupe

Two extra commits, folded in here rather than opened as their own PR since they
touch the same file and the same base.

kas/feature/dev-root-login.yml is the console-only counterpart of this
fragment - the toggle and nothing else. It had been carried as an untracked file
on the bench through the whole Jetson device-tree-overlay bring-up, because a
stock image sets root's password field to * and /proc/device-tree, the only
thing that proves an overlay reached the running kernel, is readable only from a
shell on the board.

It is a separate fragment rather than a flag on this one because ssh-dev appends
openssh-sshd, openssh-keygen and openssh-sftp-server to the base rootfs so
oe-core's image features have a config to edit. On a board reached over a serial
cable that is a larger image and a network service for no gain. Reach for
ssh-dev when you need a shell over the network; reach for dev-root-login when
you need a login prompt and nothing else.

ssh-dev.yml now includes it rather than setting AVOCADO_DEV_ROOT_LOGIN
again, the way containers.yml pulls its layer through virtualization.yml.

That dedupe closes a gap this PR had on its own, which is the part worth a
reviewer's attention. Flipping the toggle re-signs avocado-users' do_install,
so the PR service can hand back a revision below one already published;
packagedata then fails version went backwards, and if the feed holds the
older package at a higher revision, dnf resolves to it and the image keeps
root:*. The build stays green and the board cannot be logged into - over SSH
or on the console - and the symptom is indistinguishable from the feature not
working. This PR flipped the same toggle and carried none of that handling.

The QA relaxation now inherited from the new fragment covers it. The PR bump
that is sometimes additionally needed ships commented out, with a
devtool-debt: marker: the right value depends on the reader's feed rather than
on the file, so a fixed one would be correct once and wrong after.

Not resolved through a real kas dump - the include is YAML-valid and matches
the containers.yml precedent exactly, but it has not been run through kas on
this branch. Worth a reviewer's eye given every ssh-dev build now depends on it
resolving.

The equivalent change is not yet on the wrynose side (#275); it should follow
once the shape is agreed here.

The device presents a new SSH host key on every boot. Every client's known_hosts breaks on every reboot, and `avocado container dev up` bootstraps the device over SSH, so a session cannot reconnect to a device that has rebooted - which is the normal case for a fleet. oe-core's read_only_rootfs_hook finds no pre-generated key in /etc/ssh and concludes the device cannot keep one, so it points sshd at sshd_config_readonly with keys under /var/run/ssh, on tmpfs. That inference is right for a stateless image and wrong for this one: the rootfs is read-only but /var is writable and persistent, which is a third case oe-core does not model. Its choice is between shipping a key in the image, which we will not do, and having no key survive at all. The shipped sshd_config already points HostKey at /var/lib/ssh, so only the override needs undoing. sshd_check_keys sources /etc/default/ssh as shell and its generate_key does mkdir -p on the key directory, so appending the persistent values is enough - keys are generated once on first boot and reused after. This runs from IMAGE_PREPROCESS_COMMAND rather than ROOTFS_POSTPROCESS_COMMAND, and that is the whole difficulty. A recipe-level append to the latter does not order after the entry rootfs-postcommands.bbclass contributes: measured here, read_only_rootfs_hook runs last of the three. Placed there this function ran before oe-core had created the file, its guard skipped, and the volatile values were written afterwards - a silent no-op that inspected as a working patch. Verified by reading the produced rootfs and sourcing the result rather than by reading the recipe. Confirmed on an FRDM-IMX93 running the wrynose BSP: the ed25519 host key fingerprint is unchanged across a reboot that changed /proc/sys/kernel/ random/boot_id, so the key is being reused rather than regenerated. Signed-off-by: Javier Tia <javier@peridio.com>

Getting a shell on a freshly flashed board took a bespoke sequence per board: there was no way to ask for SSH access at build time, and the image as built could not serve SSH at all even once openssh was present. Two independent gaps, hence two pieces. The host keys were unreachable. sshd resolves HostKey to ${sysconfdir}/ssh, which on this image is the read-only rootfs, so sshd_check_keys cannot create one: the unit fails, sshd starts keyless, and every client is dropped at key exchange with "Connection reset by peer" - a symptom that reads like a network fault rather than a missing file. Point HostKey at /var/lib/ssh, which is writable and persistent, so the key survives a reboot. A drop-in cannot do this: HostKey accumulates rather than replaces, so the unreachable paths would remain in the list. The bbappend branches on whether the recipe wrote those lines at all, and that is what makes the same file correct on both openssh versions in the layer. 10.3p1 writes them from OPENSSH_HOST_KEY_DIR, one per key type enabled in PACKAGECONFIG, so rewriting in place preserves those gates. 9.6p1, which is what scarthgap carries, leaves them commented and rewrites only sshd_config_readonly, so sshd runs on its compiled-in defaults and a sed matches nothing - there the keys have to be named. Either branch alone fails silently on the other version. This is complementary to the SYSCONFDIR=/var/lib/ssh line the preceding commit appends to /etc/default/ssh, not a duplicate of it. SYSCONFDIR only decides where sshd_check_keys does its mkdir -p, while the key paths come from `sshd -G` reading HostKey - which is why that append alone still wrote keys to the read-only path. The gap has been invisible because the sshd-dev extension ships an entire sshd_config of its own with the right paths, so only an image carrying base-image openssh hits it. The feature itself is a kas overlay so any board can ask for it: bakar build .../machine/<board>.yml:.../feature/ssh-dev.yml Deliberately not an AVOCADO_FEATURE_GROUPS entry. A feature group populates the feed - `networking` builds the openssh RPMs and stops there, because the base image is minimal by design and gains capability through extensions. That is correct for a product and useless for a bench board, where the point is to have a shell before any extension has been installed. ROOTFS_IMAGE_EXTRA_INSTALL is the seam the image recipe already leaves for putting a package in the base rootfs. Console and SSH need separate switches: AVOCADO_DEV_ROOT_LOGIN rewrites `root:*:` to `root::` at package build, and oe-core's allow-empty-password rewrites nullok_secure to nullok across pam.d at rootfs assembly. Without the second, an empty password authenticates on the console and is refused over SSH, because nullok_secure admits an empty password only from a tty listed in /etc/securetty. Verified on an FRDM-IMX93 against the wrynose BSP, where this same file produced a reachable sshd and a host key that survived a reboot. Not built or booted on scarthgap - the 9.6p1 branch above was derived by reading that recipe's do_install, not by running it. Signed-off-by: Javier Tia <javier@peridio.com>

This has been carried as an untracked file on the bench for the whole Jetson device-tree-overlay bring-up, because a stock image sets root's password field to `*` and /proc/device-tree - the only thing that proves an overlay reached the running kernel - is readable only from a shell on the board. An unshared file means the next person rediscovers both the toggle and the feed interaction below. ssh-dev.yml already sets the same toggle, so this is not the only way to get a login. It is the right one for a UART-attached board: ssh-dev appends sshd, keygen and sftp-server to the base rootfs so oe-core's image features have a config to edit, which on a board reached over a serial cable buys a larger image and a network service and nothing else. The PR bump ships commented out rather than active. Demoting the version-going-backwards check is not sufficient when the feed already holds a higher-PR avocado-users: dnf resolves to the older package, the image keeps `root:*`, and the build stays green - a failure indistinguishable from the feature not working. The value that fixes that depends on the reader's feed rather than on this file, so a fixed one would be correct once and wrong after. The debt marker names the condition that should replace it with a derived PR. Signed-off-by: Javier Tia <javier@peridio.com>

Both fragments need AVOCADO_DEV_ROOT_LOGIN, and two overlays setting the same variable independently drift - the second one to change is the one nobody notices. Include the fragment instead, the way containers.yml pulls its layer through virtualization.yml rather than pinning it twice. The include also closes a gap ssh-dev had on its own. Flipping the toggle re-signs avocado-users' do_install, so the PR service can hand back a revision below one already in the feed; packagedata then fails "version went backwards", and if the feed holds the older package at a higher revision dnf resolves to it and the image keeps root:*. ssh-dev flips the same toggle and carried none of that handling, so it was one stale feed away from a green build that cannot be logged into - over SSH or on the console. Including a fragment a config already pulls in directly is harmless: kas keys local_conf_header by section name, so the section resolves once rather than being emitted twice. Signed-off-by: Javier Tia <javier@peridio.com>

Any image built with the networking feature installs openssh-sshd, and that sshd cannot start. It listens on 22, then every connection dies at kex_exchange_identification, and journalctl -u sshd@ reports no entries at all. There is no sshd account. oe-core's openssh creates one through the useradd class, declaring USERADD_PARAM:${PN}-sshd = "--system --no-create-home --home-dir /var/run/sshd --shell /bin/false --user-group sshd". This recipe then installs a fixed /etc/passwd and /etc/group over the top, which discards it along with any other useradd-created system user. OpenSSH refuses to run without its privilege-separation account and exits before writing a banner or a log line, so the unit looks healthy and the journal has nothing - which is what makes this expensive to diagnose rather than merely broken. Add the account, mirroring that USERADD_PARAM's home directory and shell so the two do not drift. uid and gid 992 are free here and stay in the system range. Scoped to the account alone. #276 already carries the host-key work for this branch - installing openssh-sshd and moving the keys onto /var - and this is the piece it does not have. The two are independent: a host key in a writable directory still gets no sshd without the account, and the account alone still leaves the key on a read-only path, so neither supersedes the other and the order they merge in does not matter. The clobber itself is left alone and marked as debt. The real fix is to stop shipping a whole passwd and pin only the uid/gid this distro cares about, letting useradd own the rest - a change to how every Avocado image gets its users, which wants its own verification rather than riding in here. Found on the wrynose line while diagnosing an sshd that reported "Finished OpenSSH Key Generation" and still dropped every connection. Adding the account there produced sshd:x:992:992 in the built rootfs and a running server; scarthgap reaches the same failure by the same route, since it ships the same avocado-users and installs openssh through packagegroup-avocado-feature-networking.

jetm · 2026-08-24T15:09:13Z

Heads-up from taking this to hardware: the host-key work here is necessary but not sufficient on its own — sshd still will not start, because there is no sshd privilege-separation account.

avocado-users installs a fixed /etc/passwd and /etc/group over whatever the useradd class produces, so openssh's USERADD_PARAM:${PN}-sshd = "... --user-group sshd" is discarded. OpenSSH refuses to run without that account and exits before writing a banner or a log line, so the symptom is indistinguishable from the missing-key one: connections dropped at kex_exchange_identification, and journalctl -u sshd@ completely empty.

#304 adds the account against this branch. The two changes are independent and neither supersedes the other — a key in a writable directory still gets no sshd without the account, and the account alone still leaves the key on a read-only path — so merge order does not matter, but sshd will not come up until both have landed.

Flagging it mainly so this PR does not read as not-working when validated on its own.

Any image built with the networking feature installs openssh-sshd, and that sshd cannot start. It listens on 22, then every connection dies at kex_exchange_identification, and journalctl -u sshd@ reports no entries at all. There is no sshd account. oe-core's openssh creates one through the useradd class, declaring USERADD_PARAM:${PN}-sshd = "--system --no-create-home --home-dir /var/run/sshd --shell /bin/false --user-group sshd". This recipe then installs a fixed /etc/passwd and /etc/group over the top, which discards it along with any other useradd-created system user. OpenSSH refuses to run without its privilege-separation account and exits before writing a banner or a log line, so the unit looks healthy and the journal has nothing - which is what makes this expensive to diagnose rather than merely broken. Add the account, mirroring that USERADD_PARAM's home directory and shell so the two do not drift. uid and gid 992 are free here and stay in the system range. Scoped to the account alone. #276 already carries the host-key work for this branch - installing openssh-sshd and moving the keys onto /var - and this is the piece it does not have. The two are independent: a host key in a writable directory still gets no sshd without the account, and the account alone still leaves the key on a read-only path, so neither supersedes the other and the order they merge in does not matter. The clobber itself is left alone and marked as debt. The real fix is to stop shipping a whole passwd and pin only the uid/gid this distro cares about, letting useradd own the rest - a change to how every Avocado image gets its users, which wants its own verification rather than riding in here. Found on the wrynose line while diagnosing an sshd that reported "Finished OpenSSH Key Generation" and still dropped every connection. Adding the account there produced sshd:x:992:992 in the built rootfs and a running server; scarthgap reaches the same failure by the same route, since it ships the same avocado-users and installs openssh through packagegroup-avocado-feature-networking.

mobileoverlord

Approving. Verified beyond CI, which only parses the container-sdk config and never loads kas/feature/*:

Resolved machine/imx93-frdm.yml:feature/ssh-dev.yml with kas dump on a merge of this branch onto current scarthgap. Both feature/dev-root-login and feature/ssh-dev land in local_conf_header and meta-avocado-distro is in the layer set, so the include works. On the branch as-is it cannot resolve because its base still pins the orphaned oe-core 06b1c4f; that goes away on merge.
Traced the 9.6p1 path: the shipped sshd_config keeps #HostKey commented, so the bbappend takes the printf branch. read_only_rootfs_hook runs in do_rootfs and the IMAGE_PREPROCESS append runs in do_image afterwards, so SSHD_OPTS= is the last assignment for both sshd_check_keys (sourced) and sshd@.service (EnvironmentFile). /var is its own partition in fstab.

Non-blocking:

This diverges from wrynose, which points sshd_config_readonly at /var/lib/ssh and leaves SSHD_OPTS alone. 9.6p1 has no OPENSSH_HOST_KEY_DIR_READONLY_CONFIG, but the same shape is one sed on sshd_config_readonly in the bbappend, with no image hook and no ordering argument. Worth doing if this file is touched again.
return 0 inside a do_install:append body returns from the whole do_install, so a later append from another layer would be skipped in the no-sshd_config case. An if/else around the two branches avoids that.
The include precedent on scarthgap is kas/base.yml -> kas/vendor/oe.yml; containers.yml here does not include virtualization.yml.
The feed's openssh-sshd RPM now ships HostKey lines pointing at /var/lib/ssh, so extensions using the packaged sshd_config on the read-only rootfs get working keygen too.

jetm added 2 commits August 12, 2026 18:35

jetm force-pushed the ssh-dev-scarthgap branch from 0cfa25e to 5635ac1 Compare August 13, 2026 00:42

jetm changed the title ~~ssh: add an ssh-dev feature and keep host keys on /var~~ ssh: add an ssh-dev feature and keep host keys on /var for scarthgap Aug 13, 2026

jetm requested a review from mobileoverlord August 13, 2026 19:31

jetm added 2 commits August 15, 2026 14:03

jetm mentioned this pull request Aug 24, 2026

avocado-users: add the sshd privilege-separation account for scarthgap #304

Merged

jetm mentioned this pull request Aug 24, 2026

ssh: add an ssh-dev feature and keep host keys on /var for wrynose #275

Merged

jetm mentioned this pull request Aug 24, 2026

Bring up the OP-TEE fTPM on imx93-frdm #297

Closed

jetm requested a review from nicksinas September 4, 2026 17:18

mobileoverlord approved these changes Sep 15, 2026

View reviewed changes

mobileoverlord merged commit 6f19439 into scarthgap Sep 15, 2026
6 checks passed

mobileoverlord deleted the ssh-dev-scarthgap branch September 15, 2026 15:26

Navigation Menu

ssh: add an ssh-dev feature and keep host keys on /var for scarthgap - #276

ssh: add an ssh-dev feature and keep host keys on /var for scarthgap#276
mobileoverlord merged 4 commits into
scarthgapfrom
ssh-dev-scarthgap

jetm commented Aug 13, 2026 •

edited

Loading

Uh oh!

jetm commented Aug 24, 2026

Uh oh!

mobileoverlord left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

jetm commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Same files as the wrynose version

Verification

Added 2026-08-15: a console-only sibling, and a dedupe

Uh oh!

jetm commented Aug 24, 2026

Uh oh!

mobileoverlord left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jetm commented Aug 13, 2026 •

edited

Loading