bump max companion contacts to 700 for select boards

[recrof]

No description provided.

[liamcottle]

App won't be able to show max contact counts higher than ~500, due to this line:

MeshCore/examples/companion_radio/MyMesh.cpp

Line 835 in 77257a3

out_frame[i++] = MAX_CONTACTS / 2; // v3+

Firmware tells clients what the max contacts count is via a uint8_t where the actual value is halved, and the client doubles it to know the real value.

However with values greater than 255*2, this would overflow and the app would report an incorrect amount of max contacts.

We should probably append a new uint16_t (max 65535 contacts) or uint32_t (max 4,294,967,295 contacts) to the existing RESP_CODE_DEVICE_INFO and clients can use the new value if available, and fallback to the old one if not...

Also, has this increased contacts change been tested on any devices? Wondering if there will be any file system issues due to the increased FS write sizes.

[oltaco]

TLDR: On NRF52, our 100kB ExtraFS partition effectively caps contacts at ~450 due to LittleFS headroom, and devices with external flash are ultimately capped at ~640 contacts by RAM. These numbers assume we take it to the absolute limit with zero breathing room left which is probably a bad idea :)

One option could be to migrate to a larger ExtraFS partition at the cost of less room for the firmware itself to grow in size, but it probably won't appease most contact hoarders anyway so I'm not really sure if there's much to be gained. (There is one potential benefit, with a larger ExtraFS we could move to chunked contacts without having to reduce max contacts.)

I'll go over the nitty gritty details below so that people can understand better what we are dealing with.

1. Storage space

Most NRF52 devices are using a 100 kB ExtraFS partition for storing adv_blobs, contacts, and channels. There's some filesystem overhead to account for as well; the superblock, directories, and files all require an additional 2 blocks for metadata and possibly more for large files? (The file metadata requirements are what has kept me from pursuing a chunked contacts system, because we would lose storage space to metadata blocks and the maximum contacts would need to be reduced).

file / purpose	Record size (bytes)	Max records	Bytes required	Blocks required
superblk	—	—	—	2
rootdir	—	—	—	2
channels2	68	40	2,720	8
adv_blobs	180	100	18,000	38
contacts3	152	350	53,200	106
Min blocks required	—	—	—	156

Blocks in use are ~156, leaving 44 blocks remaining. This does not take into account any slack/reserve space that LittleFS may need to operate correctly (metadata, wear levelling?).

There is not much concrete data on how much slack is required, but I think I've seen it suggested that it needs at least ~7–10 free blocks to operate reliably otherwise you risk corruption. It might be possible to operate with as little as 2 free blocks but I think eventually it will fall apart.

At 451 contacts, we would be left with 14 free blocks, which should (?) be adequate but we would need to test for reliability before making that call. The hard limit is probably around there or maybe slightly more.

The final storage consideration is whether we want to leave ourselves room for new files. If we consume all remaining space now, we effectively paint ourselves into a corner with respect to future features that may require additional files.

Some NRF52 devices do have an external 2 MB flash chip that we already use, but even on those devices the practical contact limit is still constrained by RAM usage, which is covered below.

2. RAM

NRF52 devices only have 256kB of RAM, and all contacts are loaded into RAM at runtime.

The ContactInfo struct is currently 184 bytes (last time I checked). At 350 contacts, this consumes ~62 kB of RAM, and bumping it up to 451 would use ~81kB. This usage is not reflected in the RAM calculations produced by platformio.ini at compile time, since the compiler does not know how many contacts will be loaded by the device at runtime.

Total RAM usage varies by device and enabled features. As an example, the current WioL1 dev firmware appears to use ~140 kB of RAM. With 350 contacts loaded, this increases total usage to ~203 kB, leaving roughly 50 kB of headroom or if we bumped up to 451 contacts we'd be left with 36kB of headroom but I am probably missing some other RAM usage not accounted for so the the numbers might be less.

If my calculations are correct then the theoretical maximum contacts supported by WioL1 (which has an external 2MB flash for filesystem storage), would be ~644 contacts with no RAM left.

[liamcottle]

Allowing for contacts stored at the application layer instead of firmware would allow for unlimited contacts, at the expense of not being able to receive messages while offline unless the encrypted packet is stored in memory and synced later. Currently we store the decrypted message in memory and apps can fetch them when they connect.

[oltaco]

That's the the only realistic approach for storing lots of contacts that I can see. Storing offline messages in an encrypted state would mean you couldn't display them on the node UI but you wouldn't need to store all contacts on the phone either, you could store contacts on the node as usual and just use the phone as overflow storage, or only store favourites on the device... or any variation really :)

[recrof]

closing this for now, as we have other means how to deal with full contact list in the works(#1379). If @liamcottle's app can get support of resolving repeater names/paths from discover, the issue with full contact list won't be an issue.