diff --git a/CHANGELOG.md b/CHANGELOG.md index cbd1c61..008f847 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,6 @@ # Changelog -All notable changes to avocado-ext-template are documented in this file. +All notable changes to avocado-ext-logging are documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..d645695 --- /dev/null +++ b/LICENSE @@ -0,0 +1,202 @@ + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/README.md b/README.md index 83f39af..0a1d1a5 100644 --- a/README.md +++ b/README.md @@ -1,48 +1,21 @@ -# ext-template +# ext-logging -GitHub template repo for new Avocado extensions. **Use this template** → then work -through the checklist below. +Journald log streaming over MQTT. -## New-extension checklist - -1. Rename the repo `ext-` (or `bsp-`). -2. `avocado.yaml` — rename the extension key to `avocado-ext-`, fill in - `summary`/`description`, set `supported_targets` (and `default_target` for a BSP), - list your `packages`. Leave the `sdk.image` line alone. -3. `.github/workflows/test.yml` — set the matrix `target` to something the extension - actually supports (`qemux86-64` is fine for target-agnostic extensions). -4. `.github/workflows/release.yml` — same, plus add matrix rows for every feed you - publish into. -5. `CHANGELOG.md` — replace the placeholder `0.1.0` entry. -6. Delete this section from the README and describe the extension instead (see - "Using this extension" below — keep that part). -7. Repo secrets `AVOCADO_CONNECT_TOKEN` and `AVOCADO_CONNECT_ORG` must be set (org-level - secrets cover this if the repo is in `avocado-linux`). - -Release: tag the commit with the exact `avocado.yaml` version, e.g. `git tag 0.1.0 && git push --tags`. - -## Conventions - -- **SDK image is release-scoped, not channel-scoped.** - `docker.io/avocadolinux/sdk:{{ env.AVOCADO_DISTRO_RELEASE }}` → `avocadolinux/sdk:2026` - on a 2026 CI leg, `avocadolinux/sdk:2024` on a 2024 one. Do not append - `-{{ env.AVOCADO_DISTRO_CHANNEL }}` — there is no such tag on any release. The channel - selects the *package feed*; the image tag is per-release only. -- **CI comes from `avocado-linux/actions@v1`**, pinned. The matrix lives in the caller so - each repo owns its own target/feed combinations. -- **Feed today is `2024`/`edge-next`.** `avocado-linux/actions@v1` *defaults* to - `2026`/`edge`, but the 2026 feed currently publishes only `jetson-agx-thor`, so the - workflows here pass 2024 explicitly. Move a target to 2026 as it lands there. +systemd-journald is the collector (already running on Avocado OS). This extension +(1) persists the journal to `/var` with a rolling window and (2) streams it to an +MQTT broker by piping `journalctl -o json` to `mosquitto_pub` — one journal entry +per message. No custom agent code. ## Using this extension -`ext-template` is an [Avocado](https://avocadolinux.org) extension — a reusable fragment of -build- and runtime-configuration that you compose into your own Avocado project. To use it, -declare it as a package-sourced extension in your `avocado.yaml` and add it to a runtime: +`ext-logging` is an [Avocado](https://avocadolinux.org) extension — a reusable fragment +of build- and runtime-configuration that you compose into your own Avocado project. +Declare it as a package-sourced extension and add it to a runtime: ```yaml extensions: - avocado-ext-template: + avocado-ext-logging: source: type: package version: "*" # or pin an exact version @@ -50,15 +23,36 @@ extensions: runtimes: my-runtime: extensions: - - avocado-ext-template + - avocado-ext-logging ``` -Then install and build: +Then: ```sh -avocado install # fetches + installs the SDK, extensions and runtime deps from your config -avocado build # builds the SDK compile steps, extensions and runtime images +avocado install +avocado build ``` -`avocado install` pulls the extension from your target's package feed and merges its -config into your project; `avocado build` then produces the runtime. +## Configuration + +`/etc/avocado-log/avocado-log.env`: + +- `LOG_BROKER` / `LOG_PORT` — MQTT broker (default `127.0.0.1:1883`). +- `LOG_USER` / `LOG_PASSWORD` — optional plaintext auth; set both to enable. No + TLS in M1, so these (and the journal) cross the wire in cleartext. +- `LOG_TOPIC` — default `avocado/logs/{host}` (`{host}` → hostname). +- `JOURNAL_ARGS` — journalctl filter flags. Default `-b -n all -f -o json` (whole + current boot, then follow). Narrow with `-p err`, `-u foo.service`, `--grep RE`. + +The journald rolling window (`Storage=persistent`, size/retention) lives in +`/etc/systemd/journald.conf.d/10-avocado-log.conf`. + +Restart after editing: `systemctl restart avocado-log`. +Watch on the broker: `mosquitto_sub -t 'avocado/logs/#' -v`. + +## Milestone 1 scope + +Direct-broker path. **No TLS, no redaction, no cursor** (a broker drop replays the +boot on reconnect), **no backend gating**. Point it only at a broker you trust on a +trusted network. Filtering/redaction/rate-limiting + a single authenticated uplink +are the job of a future `avocado-logd`; see the logging RFC in `avocado-cli`. diff --git a/avocado.yaml b/avocado.yaml index dc5de79..a411323 100644 --- a/avocado.yaml +++ b/avocado.yaml @@ -1,35 +1,40 @@ -# Which targets this extension supports. '*' = any target. -# For a BSP, replace with the single board target and set default_target. supported_targets: '*' extensions: - # Rename to avocado-ext- (or avocado-bsp- for a BSP). - avocado-ext-template: + avocado-ext-logging: version: 0.1.0 release: r0 - summary: One-line summary - description: One-line description + summary: Journald log streaming over MQTT + description: >- + Ships device logs to an MQTT broker. systemd-journald is the collector + (already running on Avocado OS); this extension persists the journal to + /var with a rolling window and streams it out by piping `journalctl -o + json` to mosquitto_pub. What is shipped (unit, level, grep, boot window) + and where it is sent are set in /etc/avocado-log/avocado-log.env. license: Apache-2.0 - url: https://github.com/avocadolinux/avocado-os + url: https://github.com/avocado-linux/ext-logging vendor: Avocado Linux + package_files: + - overlay + - avocado.yaml + + overlay: overlay + # Runtime packages pulled from the target feed. - packages: {} - - # Optional, delete what you don't use: - # - # enable_services: - # - foo.service - # - # on_merge: - # - systemctl start --no-block foo.service - # - # sdk: - # packages: - # nativesdk-foo: '*' - -# Release-scoped, not channel-scoped: this resolves to avocadolinux/sdk:2026. -# The SDK image is per-release only — appending the channel points at a tag -# that does not exist. + packages: + mosquitto-clients: '*' + + enable_services: + - avocado-log.service + + on_merge: + - systemctl restart --no-block systemd-journald.service + - systemctl start --no-block avocado-log.service + + on_unmerge: + - systemctl stop avocado-log.service + +# Release-scoped, not channel-scoped: resolves to avocadolinux/sdk:. sdk: image: docker.io/avocadolinux/sdk:{{ env.AVOCADO_DISTRO_RELEASE }} diff --git a/opt/_avocado/qemux86-64/sdk/x86_64/var/cache/expired_repos.json b/opt/_avocado/qemux86-64/sdk/x86_64/var/cache/expired_repos.json new file mode 100644 index 0000000..0637a08 --- /dev/null +++ b/opt/_avocado/qemux86-64/sdk/x86_64/var/cache/expired_repos.json @@ -0,0 +1 @@ +[] \ No newline at end of file diff --git a/opt/_avocado/qemux86-64/sdk/x86_64/var/lib/rpm/rpmdb.sqlite b/opt/_avocado/qemux86-64/sdk/x86_64/var/lib/rpm/rpmdb.sqlite new file mode 100644 index 0000000..9c945a2 Binary files /dev/null and b/opt/_avocado/qemux86-64/sdk/x86_64/var/lib/rpm/rpmdb.sqlite differ diff --git a/opt/_avocado/qemux86-64/sdk/x86_64/var/lib/rpm/rpmdb.sqlite-shm b/opt/_avocado/qemux86-64/sdk/x86_64/var/lib/rpm/rpmdb.sqlite-shm new file mode 100644 index 0000000..fe9ac28 Binary files /dev/null and b/opt/_avocado/qemux86-64/sdk/x86_64/var/lib/rpm/rpmdb.sqlite-shm differ diff --git a/opt/_avocado/qemux86-64/sdk/x86_64/var/lib/rpm/rpmdb.sqlite-wal b/opt/_avocado/qemux86-64/sdk/x86_64/var/lib/rpm/rpmdb.sqlite-wal new file mode 100644 index 0000000..e69de29 diff --git a/overlay/etc/avocado-log/avocado-log.env b/overlay/etc/avocado-log/avocado-log.env new file mode 100644 index 0000000..2e4673b --- /dev/null +++ b/overlay/etc/avocado-log/avocado-log.env @@ -0,0 +1,23 @@ +# Avocado Log — MQTT log streaming config. +# Restart after editing: systemctl restart avocado-log + +# Broker. Set LOG_BROKER to your MQTT host. No TLS in Milestone 1 — the journal +# (and any LOG_USER/LOG_PASSWORD below) crosses the wire in cleartext, so point +# only at a broker you trust on a trusted network. +LOG_BROKER=10.0.2.2 +LOG_PORT=1883 +# Optional plaintext username/password auth. Set both to enable; leave unset for +# an anonymous broker. Passed to mosquitto_pub -u/-P (visible in `ps` on-device). +# LOG_USER= +# LOG_PASSWORD= + +# Topic. {host} is replaced with the device hostname by the unit. +LOG_TOPIC=avocado/logs/{host} + +# WHAT to ship — journalctl filter flags. Defaults: whole current boot (-b) then +# follow live (-f), all backlog (-n all), one JSON object per line. +# -p err minimum priority (emerg..debug) +# -u foo.service only this unit (repeatable) +# --grep RE message regex +# The set of logs is controlled here, not in code. +JOURNAL_ARGS=-b -n all -f -o json diff --git a/overlay/etc/systemd/journald.conf.d/10-avocado-log.conf b/overlay/etc/systemd/journald.conf.d/10-avocado-log.conf new file mode 100644 index 0000000..e8443d9 --- /dev/null +++ b/overlay/etc/systemd/journald.conf.d/10-avocado-log.conf @@ -0,0 +1,9 @@ +# Avocado Log — persist the journal to disk with a rolling window. +# Without this, Avocado's journald is volatile (RAM only) and boot logs vanish. +# Persisted under /var (the writable store on Avocado's read-only rootfs). +# The window size/duration are the user-facing knobs. +[Journal] +Storage=persistent +SystemMaxUse=256M +MaxRetentionSec=7d +ForwardToSyslog=no diff --git a/overlay/usr/lib/systemd/system/avocado-log.service b/overlay/usr/lib/systemd/system/avocado-log.service new file mode 100644 index 0000000..8cc4693 --- /dev/null +++ b/overlay/usr/lib/systemd/system/avocado-log.service @@ -0,0 +1,23 @@ +[Unit] +Description=Avocado Log — stream journald to MQTT +After=network-online.target systemd-journald.service +Wants=network-online.target + +[Service] +EnvironmentFile=/etc/avocado-log/avocado-log.env +# journald (collector) piped straight to mosquitto_pub. One JSON line = one message. +# ponytail: on broker drop the whole pipe restarts and -b -n all replays the boot +# (duplicates). Fine for M1; avocado-logd tracks a journal cursor to resume clean. +ExecStart=/bin/sh -c 'H=$(hostname); T=$(printf %%s "$LOG_TOPIC" | sed "s/{host}/$H/"); \ + if [ -n "$LOG_USER" ]; then \ + journalctl $JOURNAL_ARGS | mosquitto_pub -h "$LOG_BROKER" -p "$LOG_PORT" -t "$T" -u "$LOG_USER" -P "$LOG_PASSWORD" -l; \ + else \ + journalctl $JOURNAL_ARGS | mosquitto_pub -h "$LOG_BROKER" -p "$LOG_PORT" -t "$T" -l; \ + fi' +Restart=always +RestartSec=5 +User=root +Group=systemd-journal + +[Install] +WantedBy=multi-user.target