RFC: Replace partitions.conf with a structured YAML source format

[igoropaniuk]

Overview

qcom-ptool currently consumes a custom line-format text file (partitions.conf) as the source of truth for partition layouts, with each line being either a --disk directive or a --partition directive expressed as getopt-style option strings. The format predates the project's growth from one SoC to twenty boards across multiple storage classes (UFS, eMMC, NVMe, SPI-NOR), and a number of structural problems have surfaced as a result.

This RFC proposes replacing partitions.conf with a structured YAML source format and lays out the consequences.

1. Problems with the current format

1.1 The format itself

A representative slice of platforms/qrb5165-rb5/ufs/partitions.conf:

--disk --type=ufs --size=137438953472 --write-protect-boundary=0 \
       --sector-size-in-bytes=4096 --grow-last-partition

--partition --lun=0 --name=rootfs --size=79691776KB \
            --type-guid=1B81E7E6-F50D-419B-A739-2AEEF8DA3335 \
            --filename=rootfs.img
--partition --lun=1 --name=xbl_a --size=3584KB \
            --type-guid=DEA0BA2C-CBDD-4805-B4F9-F428251C3E98 \
            --filename=xbl.elf

Concrete issues that surface in day-to-day work:

• Two layers of bespoke parsing. Every line is first split on whitespace, then re-tokenised through getopt.gnu_getopt with a hand-maintained list of long options. Adding a new option means editing both layers - and the tokeniser silently drops unknown options because getopt rejects them with an exception that the caller catches and discards.
• No types, no schema. Sizes are accepted in five different forms ("1024", "1KB", "1Kb", "1k", "1MB", "1GB") and parsed by a regex that returns the wrong answer for surprising inputs ("foo123KB" returns 123). GUIDs are bare strings with no validation. Booleans are encoded as the strings "true" / "false".
• Heavy repetition that the format cannot factor out. The SPI-NOR partition list for glymur-crd declares the same eight-partition "primary + HMAC + backup + backup HMAC" pattern for nine logical items (QWESLICSTORE, DPP, SSD, SMBIOS, ddr, limits, SYSFW_VERSION, SOCCP_DATA, ...), giving 36 near-identical lines. Mistakes in any one of them cannot be caught by the format and only surface when the device fails to boot.
• Multiple partition.conf for multi-storage devices (like glymur-crd)

1.2 The directory layout

The repository convention is one directory per (board, storage) combination, each with its own partitions.conf:

platforms/
|-- qrb5165-rb5/
|   \-- ufs/
|       \-- partitions.conf   # one storage, multiple LUNs
|-- glymur-crd/
|   |-- nvme/
|   |   \-- partitions.conf   # HLOS storage
|   \-- spinor/
|       \-- partitions.conf   # boot / secure-boot storage
|-- ...

This works for the build (Makefile iterates platforms/*/*/partitions.conf) but pushes board-level decisions out into duplicated text. There is no file that says "this is glymur-crd; here are its two storages" - the board is a directory listing, not a declarative artifact. As a result:

• Common partition groups (boot-firmware pair, secure-boot HMAC quadruplets) are copied verbatim across boards. glymur-crd, sm8750-mtp and kaanapali-mtp all carry near-identical XBL_SC / BOOT_FW1 / BOOT_FW2 + _BACKUP blocks.
• Variant axes (Debian vs QLI rootfs sizes; XBL v2 vs v3 firmware layouts) cannot be expressed at all. A board can only describe one combination of (HLOS, boot-firmware) per directory.
• A duplicate-and-edit migration is the only way to add a new variant. We saw this when qcs615-ride and qcm6490-idp got emmc siblings to their ufs directories - a full copy of the partition list with ad-hoc edits.

1.3 What this blocks

Goal	Current cost	Why the format gets in the way
Share common partitions across boards	Copy-paste across N files	No include / extends mechanism
Build the same board for Debian and QLI	Duplicate the directory	No way to overlay variant-specific changes
Support multi-storage boards declaratively	Two sibling dirs, two builds	Format has no notion of "a board has multiple storages"
Generate UFS provisioning XML alongside rawprogram*.xml / patch*.xml / wipe XMLs	Hand-maintained, drifts from partitions.conf	The source has no first-class concept of UFS LUN geometry beyond the inline --lun= argument; cross-LUN provisioning attributes (bLUEnable, bLUWriteProtect, bMemoryType, ...) have nowhere to live
Validate input before emitting XML	Best-effort regex	No schema
Use board data to generate contents.xml	Parts of contents.xml are hardcoded; only one class of partitions can be generated either for main storage or for UFS, but the other class (SAIL, SPI-NOR) is not generated	With the right format we could generate contents.xml with multiple storage (eMMC/UFS/NVMe + optional SAIL + optional SPI-NOR)
Explicit transitions across boot firmware versions	Only one version of the boot firmware can be supported, no transition/upgrade mechanism	We could have one set of partitions per boot firmware branch/version, leveraged through includes, allowing tracing required changes across boot firmware upgrades or branch changes
Differences in HLOS features	Only one set of HLOS partitions; we create partitions that are not supported (e.g. persist on Debian) and have no upgrade mechanism	We could have one set of partitions per HLOS and HLOS version, leveraged through includes
Hardware vs software	No way of distinguishing software requirements vs hardware constraints	We could express hardware max sizes on one side and software requirements (e.g. min partition size) on the other
Generate FvUpdate.xml for UEFI FMP capsule updates	Hand-maintained per-board in meta-qcom bbappends; partition data is duplicated between qcom-ptool and the capsule definition	With the right format, capsule entries are derived from the partition spec directly (one GUID, one name, one filename) - no drift between rawprogram.xml and FvUpdate.xml

2. Proposal

Replace partitions.conf with a structured YAML source file.

3. Why YAML

Strengths:

• Nesting is structural - partitions is a real child of its storage entry. No order-sensitivity problems.
• Anchors (&base) and aliases (*base) handle intra-file repetition cleanly: the xbl_a / xbl_b pair, or the four _BACKUP / _HMAC partitions of a logical item, can share a base mapping.
• Flow style keeps long partition lists compact (one partition per line for the ~66-entry glymur-crd SPI-NOR side).
• Tooling is ubiquitous. Every contributor's editor already knows YAML; CI lint/schema tooling (yamllint, JSON Schema via jsonschema) is mature.

Known footguns we need to mitigate up front:

• Implicit typing in YAML 1.1 can misparse partition data:

  flag: no              # parsed as: false
  enabled: off          # parsed as: false
  guid: 0123456789ABCD  # may be parsed as integer in some libs
  partition_count: 010  # YAML 1.1: octal -> 8

• Anchors are document-scoped only.

4. File structure

Proposed end state:

platforms/
|-- _common/
|   |-- secure-boot-hmac-pairs.yaml   # QWESLICSTORE/DPP/SSD/SMBIOS/ddr/limits
|   |-- boot-firmware-pair.yaml       # XBL_SC, BOOT_FW1, BOOT_FW2 + _BACKUP
|   |-- recovery.yaml                 # RecoveryGPT, recoveryinfo, ...
|   \-- ramdump-pair.yaml             # XBL_RAMDUMP, TZAPPS, MULTIIMGQTI
|-- variants/
|   |-- hlos/
|   |   |-- qcom-deb-images.yaml      # rootfs: 32 GiB, rootfs.img
|   |   \-- meta-qcom.yaml            # rootfs: 60 GiB, qli-rootfs.img, persist
|   \-- boot-fw/
|       |-- xbl-v2.yaml
|       \-- xbl-v3.yaml
|-- boards/
|   |-- glymur-crd.yaml
|   |-- qrb5165-rb5.yaml
|   |-- qcs615-ride.yaml
|   |-- qcs615-ride-rev3.yaml         # derived board: extends qcs615-ride
|   \-- ...
\-- legacy/
    \-- apq8016-sbc/                  # un-migrated boards keep .conf here
        \-- emmc/
            \-- partitions.conf

Build invocation:

qcom-ptool gen_partition \
    --board    platforms/boards/glymur-crd.yaml \
    --hlos     debian \
    --boot-fw  xbl-v3 \
    -o platforms/glymur-crd/nvme/partitions.xml \
    -o platforms/glymur-crd/spinor/partitions.xml

The dispatcher resolves all extends: chains (board-level and storage-level), applies variant overlays last, and emits one partitions.xml per declared storage.

4.1 File contents examples

boards/glymur-crd.yaml:

platform:
  name: glymur-crd

storage:
  # -----------------------------------------------------------------
  # NVMe -- HLOS storage (rootfs size/filename supplied by hlos variant)
  # -----------------------------------------------------------------
  - id: nvme0
    type: nvme
    size: 68719476736            # 64 GiB
    sector-size: 512
    write-protect-boundary: 65536
    grow-last-partition: true
    partitions:
      - name: efi
        size: "524288KB"
        type-guid: "C12A7328-F81F-11D2-BA4B-00A0C93EC93B"
        filename: efi.bin
      - name: rootfs
        type-guid: "B921B045-1DF0-41C3-AF44-4C6F280D3FAE"
        # size + filename come from variants/hlos/{debian,qli}.yaml

  # -----------------------------------------------------------------
  # SPI-NOR -- boot / platform-config
  # Inherits four shared partition blocks via `extends`.
  # -----------------------------------------------------------------
  - id: spinor0
    type: spinor
    size: 67108864               # 64 MiB
    sector-size: 4096
    write-protect-boundary: 0
    extends:
      - _common/secure-boot-hmac-pairs.yaml
      - _common/boot-firmware-pair.yaml
      - _common/recovery.yaml
      - _common/ramdump-pair.yaml
    partitions:
      # Board-specific partitions on top of the common sets.
      - { name: cdt, size: "4KB",   type-guid: "A19F205F-CCD8-4B6D-8F1E-2D9BC24CFFB1", filename: cdt.bin }
      - { name: SD_MGR, size: "528KB", type-guid: "5E463172-D0AC-4DD4-91B8-CD3EE1281579" }
      # ...

_common/boot-firmware-pair.yaml:

partitions:
  - name: XBL_SC
    size: "2520KB"
    type-guid: "DEA0BA2C-CBDD-4805-B4F9-F428251C3E98"
    filename: xbl_s.melf
  - name: XBL_SC_BACKUP
    size: "2520KB"
    type-guid: "7A3DF1A3-A31A-454D-BD78-DF259ED486BE"
    filename: xbl_s.melf
  - name: BOOT_FW1
    size: "12288KB"
    type-guid: "CD6CDFAB-B3F7-46C6-BFFE-1A1D2B8B7BA0"
    filename: bootfw1.bin
  # ...

variants/hlos/debian.yaml:

# Debian-flavoured HLOS: 32 GiB rootfs, stock filename.
storage:
  - id: nvme0
    partitions:
      - name: rootfs
        size: "33554432KB"
        filename: rootfs.img

variants/boot-fw/xbl-v3.yaml:

# Boot-firmware deltas applied when --boot-fw xbl-v3 is selected.
# The base board file declares the v2 layout as the common ground;
# this overlay bumps sizes and appends partitions that only exist
# in v3. Keyed by stable identifier so the merge knows what to
# replace vs. what to append.
storage:
  - id: spinor0
    partitions:
      # BOOT_FW1 grew from 12 MiB to 16 MiB in v3.
      - name: BOOT_FW1
        size: "16384KB"
        filename: bootfw1-v3.bin

      # v3-only partitions; appended to whatever the base board declared.
      - name: vm-data
        size: "8192KB"
        type-guid: "1A2B3C4D-5E6F-7081-9203-A4B5C6D7E8F9"
        filename: vm-data.bin
      - name: SOCCP_DATA
        size: "256KB"
        type-guid: "F1E2D3C4-B5A6-9788-7665-543322110010"
        filename: soccp_data.bin

4.2 Composition mechanisms

Two composition mechanisms are proposed, operating at different levels.

4.2.1 Storage-level extends: - shared partition groups

Used inside a storage entry to include shared partition lists from _common/. The loader reads each referenced file, concatenates its partitions: list into the parent storage's partitions: list, then layers inline partitions from the board file on top.

Resolution semantics:

• Order: included files are processed in extends: list order; inline partitions: come last.
• Same-named overrides: if the board file declares a partition with the same name: as one in an included file, the board file's entry wins (deep-merged field-by-field, not whole-replaced - so a board can override just size: and inherit the rest).
• Transitivity: _common/*.yaml files may themselves use extends: to compose smaller pieces.
• Scope: applies inside storage entries (and inside variant overlays for the same reason).

This handles the "boot-firmware pair + HMAC quadruplet" repetition seen across glymur-crd, sm8750-mtp and kaanapali-mtp.

4.2.2 Board-level extends: - derived boards

Used at the top of a board file to inherit an entire base board and override only what differs. The loader resolves this before any storage-level processing.

# boards/qcs615-ride-rev3.yaml
extends: boards/qcs615-ride.yaml

# Everything from qcs615-ride is inherited verbatim.
# Override only the rootfs size -- silicon respin doubled the eMMC.
storage:
  - id: ufs0
    partitions:
      - name: rootfs
        size: "65536MB"

Resolution semantics:

• Deep-merge by stable identifier: storage entries matched by id, partition entries matched by name. Anything unmatched in the derived file is appended.
• Single base only: a board file may extend at most one other board file. No multiple inheritance, no diamond resolution. Keeps semantics predictable and contributors don't need to learn MRO rules.
• Resolution order overall: base board -> derived board -> storage-level extends: -> variant overlays (--hlos, --boot-fw).
• No deletion in v1: if a derived board needs to remove a partition from its base, the base should be refactored to not include it (or the partition moved to a _common/ block that the derived board doesn't include). A delete: true flag is deliberately omitted to avoid a slippery slope toward a full templating language.

This handles the long tail of board-family / revision variation that doesn't fit a clean variants/ axis:

• rb1-core-kit vs rb1-vision-kit -- same SoC, one extra mezzanine
• qcs615-ride vs qcs615-ride-rev3 -- same board, silicon respin with different DDR
• qcm6490-idp-32g vs qcm6490-idp-64g -- same IDP, different eMMC SKU

4.2.3 How the two mechanisms compose

They layer cleanly because they live at different levels:

# boards/qcs615-ride-rev3.yaml
extends: boards/qcs615-ride.yaml      # board-level: inherit everything

storage:
  - id: ufs0
    partitions:                       # partition-level: override one entry
      - name: rootfs
        size: "65536MB"

  - id: spinor0
    extends:                          # storage-level: still works
      - _common/boot-firmware-pair-v3.yaml   # different firmware in rev3
    partitions:
      - { name: cdt, ... }

4.2.4 Discoverability

Derived boards are harder to read than self-contained ones because the full partition list isn't visible in one file. A qcom-ptool show --board <name> subcommand will print the fully-resolved spec (after all extends: and variants are applied) so reviewers and integrators can see what's actually emitted without manually chasing the chain.

5. Additional emitters

A motivating use case for moving to a structured format: today both the UFS provisioning XML and FvUpdate.xml (for UEFI FMP capsule updates) are hand-maintained outside qcom-ptool, with partition data duplicated and drifting.

With a structured source, per-LUN attributes (bLUEnable, bMemoryType, bLUWriteProtect, bDataReliability, bProvisioningType, wContextCapabilities, ...) and per-partition capsule flags become first-class fields, and a single gen_partition invocation emits everything from the same source. For example:

storage:
  - id: ufs0
    type: ufs
    size: 137438953472
    sector-size: 4096
    luns:
      - id: 0
        enable: true
        memory-type: normal
        write-protect: none
        data-reliability: false
        provisioning-type: thin
        partitions:
          - name: rootfs
            size: "79691776KB"
            type-guid: "1B81E7E6-F50D-419B-A739-2AEEF8DA3335"
            filename: rootfs.img
      - id: 1
        enable: true
        memory-type: enhanced-1
        write-protect: power-on
        partitions:
          - name: xbl_a
            size: "3584KB"
            type-guid: "DEA0BA2C-CBDD-4805-B4F9-F428251C3E98"
            filename: xbl.elf
            capsule:
              update: true
              backup: xbl_b

The emitter consumes the same internal LoadedSpec and produces:

• partitions.xml (existing - GPT layout)
• rawprogram*.xml (existing - flash instructions)
• patch*.xml (existing)
• provisioning.xml (new - UFS LUN configuration)
• FvUpdate.xml (new - UEFI capsule layout, driven by per-partition capsule: flags)

Adding capsule support means each capsule-updatable partition is declared once in the partition spec; the FvUpdate emitter walks the spec and resolves the backup: reference by looking up the named partition. This eliminates the current pattern in meta-qcom where partition name + GUID + disk type have to be repeated in a recipe varflag separately from the qcom-ptool source.

---

Feedback welcome on

1. YAML format itself
2. The composition / extends semantics
3. The board-level capsule metadata (FwVersion, LSV, ESRT GUID, FlashType) - top-level capsule: section in the board file, or board-level overlay under _common/capsule/?
4. Migration ordering - I was planning to start with some multi-storage boards, for example with Glymur-CRD as a PoC
5. Splitting board definitions out of qcom-ptool into a separate repo. Today qcom-ptool conflates two concerns: the tool (parser, composition layer, XML emitters) and the data (per-board YAML/partitions.conf files under platforms/). These have very different release cadences and contributor profiles - tool changes are infrequent and reviewed by a small set of maintainers, while board definitions land continuously with every new SoC, SKU, or storage variant and pull in BSP/platform owners as reviewers. Co-locating them means board PRs churn the tool's repo history, board ownership is hard to encode in CODEOWNERS, and downstream consumers of the tool (other BSPs, internal builds, distro forks) have to pull in a growing set of upstream board definitions they don't care about. A split - e.g. qcom-ptool (the tool, versioned and released as a Python package) + qcom-platforms (the YAML board catalog, with its own ownership and review rules) - would let each side evolve independently and would also make the modularity story cleaner: _common/ and variants/ are clearly platform data, not tool source. The YAML migration is a natural time to do this split, since the directory layout is changing anyway.

[igoropaniuk]

Loïc Minier (@lool) Dmitry Baryshkov (@lumag) Viswanath Kraleti (@vkraleti) Sumit Garg (@b49020) Ricardo Salveti (@ricardosalveti) happy to hear your feedback on this

[ndechesne]

this looks really nice. some comments

1. the ordering of partitions given the amount of flexibility there is with the extends mechanism. worst case we will leverage less extends..
2. using extends inside a storage and a disk can be misleading/confusing, especially since they don't really do the same things. perhaps extends and includes?