feat(develop-docs): TelemetryBuffer Process Terminations

develop-docs/sdk/telemetry/telemetry-buffer/index.mdx

@@ -57,7 +57,7 @@ The BatchProcessor MUST forward all spans and logs in memory to the transport to
 2. When the user calls  `SentrySDK.close()`, the BatchProcessor MUST forward all data in memory to the transport. SDKs SHOULD keep their existing closing behavior.
 3. When the application shuts down gracefully, the BatchProcessor SHOULD forward all data in memory to the transport. The transport SHOULD keep its existing behavior, which usually stores the data to disk as an envelope. It is not required to call transport `flush`. This is mostly relevant for mobile SDKs already subscribed to these hooks, such as [applicationWillTerminate](https://developer.apple.com/documentation/uikit/uiapplicationdelegate/applicationwillterminate(_:)) on iOS.
 4. When the application moves to the background, the BatchProcessor SHOULD forward all data in memory to the transport and stop the timer. The transport SHOULD keep its existing behavior, which usually stores the data to disk as an envelope. It is not required to call transport `flush`. This is mostly relevant for mobile SDKs.
-5. We're working on concept for crashes, and will update the specification when we have more details.
+5. Mobile SDKs MUST minimize data loss when sudden process terminations occur. Refer to the [Mobile Telemetry Buffer](/sdk/telemetry/telemetry-buffer/mobile-telemetry-buffer) section for more details.
 
 The detailed specification is written in the [Gherkin syntax](https://cucumber.io/docs/gherkin/reference/). The specification uses spans as an example, but the same applies to logs or any other future telemetry data.
 

develop-docs/sdk/telemetry/telemetry-buffer/mobile-telemetry-buffer.mdx

@@ -4,4 +4,109 @@ description: Detailed mobile telemetry buffer design.
 sidebar_order: 4
 ---
 
-To be defined — full spec lives here.
+<Alert level="warning">
+  🚧 This concept is approved but not yet implemented in any SDKs, and it’s still being validated. If something feels unclear, too complex, or doesn’t work as expected, please open an issue or PR and tag @philipphofmann for review. Feedback and improvements are welcome while we confirm the approach makes sense. 🚧
+</Alert>
+
+For the common specification for the telemetry buffer, refer to the [Telemetry Buffer](/sdk/telemetry/telemetry-buffer/) page. This page describes the mobile-specific implementation of the telemetry buffer. The most important difference is that the mobile telemetry buffer is designed to minimize data loss when sudden process terminations occur, such as crashes or watchdog terminations.
+
+Each SDK environment is unique. Therefore, SDKs have three options to choose from to minimize data loss. As their number increases, the options get more complex. The first option is the simplest, and the last option is the most complicated. SDKs SHOULD implement the least complex option that is suitable for their environment.
+
+## 1. Flush All Data
+
+When the SDK detects a sudden process termination, it MUST put all remaining items in the telemetry buffer into one envelope and flush it. If your SDK has an offline cache, it MAY flush the envelope to disk and skip sending it to Sentry, if it ensures to send the envelope the next time the SDK starts. The telemetry buffer MUST keep its existing logic described in the [Telemetry Buffer Specification](/sdk/telemetry/telemetry-buffer/#specification) page.
+
+Suppose your SDK can't reliably detect sudden process terminations, or it can't reliably flush envelopes to Sentry or disk when a sudden process termination happens. In that case, it SHOULD implement the [FileStream Cache](#2-filestream-cache) or the [DoubleRotatingBuffer](#3-doublerotatingbuffer). It's acceptable to start with this option as a best effort interim solution before adding one of the more complex options.
+
+## 2. FileStream Cache
+
+SDKs for which blocking the main thread is a nogo, such as Android and Apple, SDKs MUST NOT implement this option. They SHOULD implement the [DoubleRotatingBuffer](#3-doublerotatingbuffer).
+
+With this option, the telemetry buffer stores the data on the calling thread directly to disk. The SDK SHOULD store the telemetry buffer files in a folder that is a sibling of the `envelopes` or `replay` folder, named `telemetry-buffer`. This folder is scoped per DSN, so SDKs ensure not mixing up data for different DSNs. In the `telemetry-buffer` folder, the SDK MUST store two types of cache files:
+
+- **`cache`** - The file the processor is actively writing to
+- **`flushing`** - The file being converted to an envelope and sent to Sentry
+
+When the timeout expires or the cache file hits the size limit, the telemetry buffer renames the `cache` file to `flushing`, creates a new `cache` file for incoming data, converts the data in the `flushing` file to an envelope, sends it to Sentry, and then deletes the `flushing` file. When the SDK starts again, it MUST check if there are any cache files in the cache directory (both `cache` and `flushing`) and if so, it MUST load the data from the files and send it to Sentry.
+
+
+## 3. DoubleRotatingBuffer
+
+SDKs should only consider implementing this option when options [1](#1-flush-all-data) or [2](#2-filestream-cache) are insufficient to prevent data loss within their ecosystem. We recommend this option only if SDKs are unable to reliably detect sudden process terminations or consistently store envelopes to disk during such terminations.
+
+The telemetry buffer uses two buffers to minimize data loss in the event of an abnormal process termination:
+* **Crash-Safe List**: A list stored in a crash-safe space to prevent data loss during detectable abnormal process terminations.
+* **Async IO Cache**: When a process terminates without the SDK being able to detect it, the crash-safe list loses all its elements. Therefore, the telemetry buffer uses a second buffer, the async IO cache, that stores elements to disk on a background thread to avoid blocking the calling thread, which ensures minimal data loss when such terminations occur.
+
+As the telemetry buffer MUST prevent data loss during flushing, it uses a double-buffering solution. The crash-safe list has two lists `crash-safe-list-1` and `crash-safe-list-2`, and the async IO cache has two files `async-io-cache-1` and `async-io-cache-2`. When `crash-safe-list-1` is full, the telemetry buffer stores any new incoming items in `crash-safe-list-2` until it successfully stores items from `crash-safe-list-1` to disk as an envelope. Then it can delete items in `crash-safe-list-1`. The same applies to the async IO cache.
+
+### Telemetry Buffer Files
+
+The SDK SHOULD store the telemetry buffer files in a folder that is a sibling of the `envelopes` or `replay` folder, named `telemetry-buffer`. This folder is scoped per DSN, so SDKs ensure not mixing up data for different DSNs. The `telemetry-buffer` folder MAY contain the following files:
+
+- `async-io-cache-1` and `async-io-cache-2` - The async IO cache files.
+- `detected-termination-x` - The file containing items from the crash-safe list from a previous detected abnormal termination.
+- `envelope-x` - The envelope that the telemetry buffer is about to move to the envelopes cache folder, so the SDK can send it to Sentry, where `x` is the an increasing index of the file starting from 0.
+
+
+### Receiving Items
+
+The telemetry buffer has two lists `crash-safe-list-1` and `crash-safe-list-2` and two files `async-io-cache-1` and `async-io-cache-2`. When it receives items, it performs the following steps:
+
+1. Put the item into the crash-safe `crash-safe-list-1` on the calling thread.
+2. On a background thread, store the item in the `async-io-cache-1`.
+
+### Flushing
+
+When the `crash-safe-list1` exceeds the [above described](#specification) 1MiB in size or the timeout exceeds, the telemetry buffer performs the following flushing steps:
+
+1. Store new incoming items to the `crash-safe-list-2` and `async-io-cache-2`.
+2. Put the items of `crash-safe-list-1` into an envelope named `envelope-x`.
+3. Delete the items in `crash-safe-list-1` and `async-io-cache-1`.
+4. Move the `envelope-x` to the envelopes cache folder, in which all the other envelopes are stored, so the SDK can send it to Sentry.
+
+The telemetry buffer stores the `envelope-x` not directly in the envelope cache folder because, if an abnormal process termination occurs before deleting the items `crash-safe-list-1` and `async-io-cache-1`, the SDKs might send duplicate items.
+
+
+### Abnormal Process Termination
+
+When SDKs detect an abnormal process termination, they MUST write the items in both `crash-safe-list-1` and `crash-safe-list-2` to the `detected-termination-x` file where `x` is the an increasing index of the file starting from 0.
+
+When the process terminates abnormally and the SDKs can't detect it, the SDKs lose items from the crash-safe lists, which we consider preferable to blocking the calling thread, which could be the main thread. However, the SDKs don't lose items from the async IO cache.
+
+So the SDK MAY lose items due to undetectable abnormal process terminations that occur immediately after receiving an item, but it won't lose items due to detectable abnormal process terminations.
+
+```
+// SDKs keep this log
+Sentry.logger.info("We might crash now.");
+
+crash();
+```
+
+```
+// SDKs might lose this log
+Sentry.logger.info("We are going to allocate loads of memory now.");
+
+// Something allocating a lot of memory leading to a watchdog termination due to out of memory.
+allocateAllMemory();
+```
+
+### SDK Initialization
+
+Whenever the SDKs initialize, they MUST check if there is any data in the telemetry buffer folder that needs to be recovered. They MUST perform the following steps when initializing:
+
+1. Load all items from `async-io-cache-1`, `async-io-cache-2` and `detected-termination-x` into memory if they exist. When the application terminates normally, these files don't exist.
+2. Deduplicate the items based on the IDs of the items and store the deduplicated items in the `envelope-x` file.
+3. Create new `async-io-cache-1` and `async-io-cache-2` files, and delete the `detected-termination-x` file.
+4. Now the telemetry buffer can start receiving new items.
+5. Move the `envelope-x` to the envelopes cache folder.
+
+### SDK Closes
+
+Whenever the users closes the SDK or the application terminates normally, the telemetry buffer MUST perform the steps described in the [Flushing](#flushing) section and the SDK MUST delete all items in the `async-io-cache-1` and `async-io-cache-2` files.
+
+## Miscellaneous
+
+The telemetry buffer maintains its logic of batching multiple logs and spans together into a single envelope to avoid multiple HTTP requests.
+
+Hybrid SDKs pass every log and span down to the native SDKs, which will put every log and span in their telemetry buffer and its cache when logs and spans are ready for sending, meaning after they go through beforeLog, integrations, processors, etc.