feat: add segmentation spec by plopezlpz · Pull Request #91 · logos-messaging/specs

plopezlpz · 2025-10-19T12:45:57Z

Added the specs for Application level segmentation of messages to be able to send messages larger than 150kb using Waku by partitioning them at source and reconstructing them on the recipient

Already implemented in https://github.com/waku-org/nim-chat-sdk/blob/main/chat_sdk/segmentation.nim

jm-clius

I have a few comments below, none of them major. Thanks!

jazzz

Overall, I think the idea has good bones.

message_hash, segment_index, segment_count provide base functionality for segmentation within an encrypted context.

Relaxing some of the implementation constraints would make it easier for other engineers to incorporate this work into their projects.

jazzz · 2025-10-21T22:13:06Z

+### Reed–Solomon
+
+Implementations that apply parity **SHALL** use fixed-size shards of length `segmentSize`.  
+The last data chunk **MUST** be padded to `segmentSize` for encoding.  


What is the motivation for padding to segment size? I can see this being beneficial in some cases, but curious what makes it required?

Padding is a requirement of Reed-Solomon, all segments must be of the same size

fryorcraken

I think we can and should make the spec mostly Waku agnostic. Waku can still be mentioned in the motivation.

igor-sirotin · 2026-02-03T23:56:20Z

@plopezlpz please address the comments and let's get this merged 🚢
This holds the closing of Deliverable

jm-clius

Good job! I think this can be merged once the few outstanding comments are addressed.
In my mind they are:

make the spec Waku-agnostic (can be mentioned in Motivation section)
remove "sender public key" and just mention more generically that index can take place by sender as well as entire message hash.

jm-clius

I think this is good to merge now! Thanks. I'd say future updates can be done in follow-up PRs. Minor comments below.

jm-clius · 2026-04-29T12:39:26Z

+
+- `entire_message_hash.length == 32`
+- **Data segments:**
+  `segments_count >= 1` **AND** `index < segments_count`


Perhaps wise to also define a maximum segments_count (e.g. 256) here, especially because receivers will allocate an in-memory cache based on expected number of segments.

@jm-clius it's defined here:

specs/standards/application/segmentation.md

Lines 165 to 166 in 9bcf528

- `maxTotalSegments` — maximum number of total shards (data + parity) per message.

Implementation-specific parameter, fixed. The reference implementation uses **256**.

Or do you think we should introduce hard limits for implementations on spec level 🤔

Ivansete-status

Some nits so far

jazzz

Thank you for getting this across the line.

The changes look good, and I think the core here is solid.

My only hang up is that the "Backwards compatibility" section currently has more questions than answers. I would highly recommend removing it for now, and then address it in a follow up.

However I won't block this PR.

igor-sirotin · 2026-04-30T13:15:29Z

Here's what I'll change:

Forget about Status implementation. We're writing a new thing for ourselves.
Require wrapping single-segment messages (i.e. segments_count >= 1)
Remove Backwards Compatibility section
Add is_parity proto field instead of segment_count=0 as implicit sign of parity shards. Adding 2 bytes is not harmful, but brings clarity.

Co-authored-by: Hanno Cornelius <68783915+jm-clius@users.noreply.github.com>

igor-sirotin · 2026-05-05T22:10:49Z

I'm gonna merge now.

My initial goal was to address the comments in this PR, as it was ready and mostly approved, so I didn't bother much. I agree there're more problems to the current spec. I will address them in separate PRs, after moving it to https://github.com/logos-co/logos-lips.

Moved SEGMENTATION spec from https://github.com/logos-messaging/specs. Original PR: logos-messaging/specs#91 Changes included: - frontmatter format - convert links - fix RFC2119 terminology styling (`MUST` instead of `**MUST**`) - typos fixed Co-authored-by: Copilot <copilot@github.com>

plopezlpz requested review from jm-clius and kaichaosun October 19, 2025 12:45

kaichaosun reviewed Oct 21, 2025

View reviewed changes

Comment thread standards/application/segmentation.md Outdated

Comment thread standards/application/segmentation.md Outdated

Comment thread standards/application/segmentation.md Outdated

plopezlpz requested a review from kaichaosun October 21, 2025 14:40

plopezlpz mentioned this pull request Oct 21, 2025

Create Segmentation Library logos-messaging/pm#318

Open

4 tasks

jazzz reviewed Oct 21, 2025

View reviewed changes

Comment thread standards/application/segmentation.md Outdated

jm-clius requested changes Oct 21, 2025

View reviewed changes

jazzz reviewed Oct 21, 2025

View reviewed changes

jazzz reviewed Oct 22, 2025

View reviewed changes

Comment thread standards/application/segmentation.md Outdated

fryorcraken reviewed Oct 22, 2025

View reviewed changes

Comment thread standards/application/segmentation.md Outdated

Comment thread standards/application/segmentation.md Outdated

Comment thread standards/application/segmentation.md Outdated

Comment thread standards/application/segmentation.md Outdated

plopezlpz requested a review from jm-clius October 27, 2025 08:11

fryorcraken approved these changes Oct 28, 2025

View reviewed changes

jm-clius approved these changes Feb 4, 2026

View reviewed changes

Comment thread standards/application/segmentation.md Outdated

jazzz mentioned this pull request Feb 9, 2026

Chat Private #93

Merged

igor-sirotin self-assigned this Apr 23, 2026

igor-sirotin mentioned this pull request Apr 23, 2026

docs: Reliable Channel API spec #104

Merged

igor-sirotin force-pushed the feat/segmentation-spec branch from 0ff0c97 to bef0d20 Compare April 28, 2026 20:56

igor-sirotin requested review from jazzz and osmaczko April 28, 2026 20:59

igor-sirotin added this to Logos Delivery Apr 28, 2026

igor-sirotin moved this to Code Review / QA in Logos Delivery Apr 28, 2026

igor-sirotin requested a review from jm-clius April 28, 2026 21:23

osmaczko reviewed Apr 29, 2026

View reviewed changes

Comment thread standards/application/segmentation.md Outdated

jm-clius approved these changes Apr 29, 2026

View reviewed changes

Ivansete-status reviewed Apr 29, 2026

View reviewed changes

Comment thread standards/application/segmentation.md

Comment thread standards/application/segmentation.md

Comment thread standards/application/segmentation.md Outdated

Comment thread standards/application/segmentation.md Outdated

Comment thread standards/application/segmentation.md Outdated

jazzz reviewed Apr 29, 2026

View reviewed changes

Comment thread standards/application/segmentation.md Outdated

jazzz approved these changes Apr 30, 2026

View reviewed changes

Comment thread standards/application/segmentation.md Outdated

Comment thread standards/application/segmentation.md Outdated

igor-sirotin requested a review from osmaczko April 30, 2026 14:24

plopezlpz and others added 22 commits May 5, 2026 23:06

fix: spell

11a3e79

fix: pr feedback

d906362

fix: spelling

7346d36

fix: pr feedback

e706dcb

fix: pr feedback

d142aa0

fix: remove limit from spec, leave in the implementation

ab04427

fix: add wire format before descriptions

79e1d01

fix: rename

8a35369

fix: sort wordslist

4f3c1d9

address comments, improvements

4af5b4e

feat: backwards compatibility mode

b3e52e7

fix: transport-agnostic

09133e0

suggest encryption

91ac51d

fix: spell checks

f18b36f

fix: optional reed-solomon

eafe7c6

Co-authored-by: Hanno Cornelius <68783915+jm-clius@users.noreply.github.com>

fix: address comments

e9a257f

fix: remove backwards compatibility section

bf20d64

fix: is_parity flag, strict encoding all segments

06a9921

address comments, iteration 1

f7970ee

address comments part 2

4109430

address comments, part 3

9526ab2

sort wordslist

786f111

igor-sirotin force-pushed the feat/segmentation-spec branch from 34085c0 to 786f111 Compare May 5, 2026 22:07

igor-sirotin merged commit 63aa0d3 into master May 5, 2026
1 check passed

igor-sirotin deleted the feat/segmentation-spec branch May 5, 2026 22:11

github-project-automation Bot moved this from Code Review / QA to Done in Logos Delivery May 5, 2026

This was referenced May 5, 2026

docs(messaging): moved segmentation spec logos-co/logos-lips#331

Merged

Simplify segmentation spec logos-co/logos-lips#332

Open

	- `maxTotalSegments` — maximum number of total shards (data + parity) per message.
	Implementation-specific parameter, fixed. The reference implementation uses 256.

Conversation

plopezlpz commented Oct 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jm-clius left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jazzz left a comment

Choose a reason for hiding this comment

Uh oh!

jazzz Oct 21, 2025

Choose a reason for hiding this comment

Uh oh!

igor-sirotin May 5, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fryorcraken left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

igor-sirotin commented Feb 3, 2026

Uh oh!

jm-clius left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jm-clius left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jm-clius Apr 29, 2026

Choose a reason for hiding this comment

Uh oh!

igor-sirotin Apr 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Ivansete-status left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jazzz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

plopezlpz commented Oct 19, 2025 •

edited

Loading

igor-sirotin Apr 29, 2026 •

edited

Loading