Add E-Document Core developer documentation

src/Apps/W1/EDocument/App/CLAUDE.md

@@ -0,0 +1,64 @@
+# E-Document Core
+
+E-Document Core is Business Central's foundation framework for electronic document exchange. It sits between BC's posting engine and external e-invoicing networks (PEPPOL, country clearance systems, etc.), providing the plumbing so that connector and format apps don't have to reinvent lifecycle tracking, error handling, or status management. Think of it as the "middleware" that turns a posted sales invoice into a tracked, auditable electronic document that flows through configurable services.
+
+## Quick reference
+
+- **ID range**: 6100-6199, 6208-6209, 6231-6232, 6234
+- **Dependencies**: None -- this is the base layer. Everything else depends on it.
+- **Namespace**: `Microsoft.eServices.EDocument`
+
+## How it works
+
+The app is organized in three conceptual layers. The **Core** layer owns the E-Document record, its lifecycle states, logging, and workflow orchestration. The **Document Format** layer (plugged in via the `E-Document` interface and the `E-Document Format` enum) handles serialization -- turning a BC document into XML/JSON and vice versa. The **Service Integration** layer (plugged in via `IDocumentSender`, `IDocumentReceiver`, and friends on the `Service Integration` enum) handles the actual HTTP communication with external networks.
+
+Documents flow bidirectionally. **Outbound**: when a sales invoice is posted, `EDocumentSubscribers` fires, creating an E-Document record, exporting it through the configured format, then routing it through a BC Workflow (`EDOC` category) that determines which service(s) send it. Sending can be synchronous or asynchronous -- if the connector implements `IDocumentResponseHandler`, the framework polls for responses via `GetResponse`. **Inbound**: a service's `IDocumentReceiver` fetches documents from an API, then each document flows through a multi-stage pipeline: Structure (convert PDF to XML via ADI or similar) -> Read into Draft (parse into `E-Document Purchase Header/Line`) -> Prepare Draft (resolve vendor, items, GL accounts) -> Finish Draft (create the actual BC purchase invoice).
+
+There are two import processing versions. **V1** (legacy, behind `#if not CLEAN27` guards) is a monolithic path where `GetCompleteInfoFromReceivedDocument` on the format interface does everything at once -- parse, resolve, and create the BC document in a single call. **V2** breaks this into four discrete steps with undo capability, each driven by a separate interface (`IStructureReceivedEDocument`, `IStructuredFormatReader`, `IProcessStructuredData`, `IEDocumentFinishDraft`). V2 is the active development path; V1 exists only for backward compatibility.
+
+Batch processing supports two modes: **Threshold** (accumulate N documents, then send as one blob) and **Recurrent** (job queue fires on a schedule, sends whatever is pending). Each mode creates a single export blob from multiple E-Documents via `CreateBatch` on the format interface.
+
+## Structure
+
+- `src/Document/` -- The E-Document table itself, status enums, direction/type enums, and the inbound/outbound list pages
+- `src/Service/` -- E-Document Service configuration, supported document types, and service participants
+- `src/Integration/` -- The V2 integration interfaces (`IDocumentSender`, `IDocumentReceiver`, etc.), context codeunits (`SendContext`, `ReceiveContext`, `ActionContext`), and the send/receive runners
+- `src/Processing/` -- The heavy lifting: export (`EDocExport`), import pipeline (`ImportEDocumentProcess`), AI-powered matching (Copilot tools), order matching, and all the V2 import interfaces
+- `src/Processing/Import/Purchase/` -- Draft purchase document tables and history tables used by V2 import
+- `src/Logging/` -- E-Document Log, Integration Log (HTTP request/response), and Data Storage (blob table)
+- `src/Mapping/` -- Field-level value mapping (find/replace on RecordRef fields during export/import)
+- `src/Workflow/` -- BC Workflow integration: EDOC category setup, flow processing, step arguments
+- `src/Helpers/` -- Error helper, JSON helper, import helper, log helper
+- `src/Format/` -- PEPPOL BIS 3.0 import handler
+- `src/DataExchange/` -- Data Exchange Definition integration for PEPPOL
+- `src/ClearanceModel/` -- QR code handling for clearance-model countries (posted invoice extensions with QR viewer)
+- `src/Extensions/` -- Page/table extensions that embed E-Document functionality into standard BC pages
+
+## Documentation
+
+- [docs/data-model.md](docs/data-model.md) -- How the data fits together
+- [docs/business-logic.md](docs/business-logic.md) -- Processing flows and gotchas
+- [docs/extensibility.md](docs/extensibility.md) -- Extension points and how to customize
+- [docs/patterns.md](docs/patterns.md) -- Recurring code patterns (and legacy ones to avoid)
+
+## Things to know
+
+- The E-Document table's `Status` field (3 values: In Progress / Processed / Error) is an **aggregate** derived from the per-service `E-Document Service Status` (24+ granular values). Each service status value implements `IEDocumentStatus` to declare which aggregate it maps to. Don't set E-Document.Status directly -- modify the service status and call `ModifyEDocumentStatus`.
+
+- The `E-Document Service Status` enum is `Extensible = true` with `DefaultImplementation = IEDocumentStatus = "E-Doc In Progress Status"`. This means any new enum value added by an extension defaults to "In Progress" unless it explicitly declares a different implementation.
+
+- E-Documents link to their source BC document via `Document Record ID` (a RecordId field). This is fragile across company renames and data migrations. The `Table ID` field exists separately for FlowField lookups.
+
+- The V2 import pipeline stores state across steps using enum fields on the E-Document record itself (`Structure Data Impl.`, `Read into Draft Impl.`, `Process Draft Impl.`). Each step's output determines which implementation the next step uses. This is a form of runtime dispatch chain.
+
+- `Commit()` is called deliberately before `Codeunit.Run()` in export and send paths. This is the error isolation pattern -- if the format/connector implementation throws a runtime error, the E-Document record survives and the error is logged rather than rolling back everything.
+
+- Duplicate detection uses the combination of `Incoming E-Document No.` + `Bill-to/Pay-to No.` + `Document Date`. The `IsDuplicate` method checks this and logs telemetry. Deleting non-duplicate E-Documents requires user confirmation.
+
+- The `E-Doc. Data Storage` table is a blob store. E-Documents reference it twice: `Unstructured Data Entry No.` (the original received file, e.g. PDF) and `Structured Data Entry No.` (the parsed structured version, e.g. XML). The structuring step may produce a new Data Storage entry, or reuse the original if the document was already structured.
+
+- Mapping (`E-Doc. Mapping` table) works at the RecordRef/FieldRef level. It applies find/replace transformations on field values during export or import, per service. The `For Import` flag distinguishes export mappings from import mappings.
+
+- The `Service Participant` table maps Customer/Vendor codes to external identifiers (PEPPOL participant IDs, etc.) per service. This is how the framework resolves "who does this document go to on the external network."
+
+- V1 integration code is wrapped in `#if not CLEAN26` / `#if not CLEAN27` guards and is being actively removed. The `Service Integration` field (old enum) is obsolete; use `Service Integration V2` instead.

src/Apps/W1/EDocument/App/docs/business-logic.md

@@ -0,0 +1,165 @@
+# Business logic
+
+This document covers how documents flow through the E-Document Core framework -- the actual processing sequences, decision points, and error handling. Read [data-model.md](data-model.md) first for table relationships.
+
+## Outbound flow
+
+When a sales invoice (or credit memo, service invoice, reminder, etc.) is posted, the BC posting engine fires events that `EDocumentSubscribers` handles. The subscriber reads the Document Sending Profile for the customer, and if it's configured for "Extended E-Document Service Flow," the export pipeline kicks in.
+
+```mermaid
+flowchart TD
+    A[Document posted] --> B[Get Document Sending Profile]
+    B --> C{Electronic Document = Extended Flow?}
+    C -- No --> Z[Exit -- not an e-document]
+    C -- Yes --> D[Get Workflow + Services from flow]
+    D --> E[Create E-Document record]
+    E --> F{For each service}
+    F --> G{Document type supported?}
+    G -- No --> F
+    G -- Yes --> H{Batch processing enabled?}
+    H -- Yes --> I[Set status: Pending Batch]
+    H -- No --> J[Export via format interface]
+    J --> K{Export succeeded?}
+    K -- Yes --> L[Status: Exported]
+    K -- No --> M[Status: Export Error]
+    L --> N[Start E-Document Created Flow]
+    N --> O[Workflow: Send step]
+    O --> P[Commit before send]
+    P --> Q[Call IDocumentSender.Send]
+    Q --> R{Connector implements IDocumentResponseHandler?}
+    R -- No --> S[Status: Sent]
+    R -- Yes --> T[Status: Pending Response]
+    T --> U[Job Queue polls GetResponse]
+    U --> V{Response received?}
+    V -- No --> U
+    V -- Yes --> S
+```
+
+The key codeunits involved:
+
+- **`EDocExport.CreateEDocument`** (in `EDocExport.Codeunit.al`) -- Creates the E-Document record, populates it from the source document header via RecordRef field reads, inserts service status records for each supporting service, then exports non-batch documents immediately.
+- **`EDocExport.ExportEDocument`** -- Applies field mappings, then calls `EDocumentCreate.Run()` (which invokes the format interface's `Create` method inside a `Codeunit.Run` error boundary). The exported blob is logged.
+- **`EDocIntegrationManagement.Send`** -- Retrieves the exported blob from the log, sets up `SendContext` with a default status of "Sent," then runs `SendRunner` inside a `Codeunit.Run` error boundary. After send, it logs the HTTP request/response and updates the service status.
+- **`EDocumentGetResponse`** -- A job queue handler that polls `IDocumentResponseHandler.GetResponse`. Returns `true` when the service confirms receipt; `false` keeps polling.
+
+The workflow engine (`EDocumentWorkFlowProcessing`) routes the E-Document through configured workflow steps. Each step can target a different service (stored in `Workflow Step Argument."E-Document Service"`). The framework supports email sending steps too, where the exported document is attached to an email.
+
+### Batch sending
+
+When `Use Batch Processing` is enabled on a service, individual E-Documents are exported but not immediately sent. Instead:
+
+1. **Threshold mode**: When the count of pending-batch documents reaches `Batch Threshold`, `EDocRecurrentBatchSend` fires and calls `ExportEDocumentBatch` to create a single combined blob, then sends it.
+2. **Recurrent mode**: A job queue entry runs on a schedule (`Batch Start Time` + `Batch Minutes between runs`), sending whatever is pending.
+
+The format interface's `CreateBatch` method receives all E-Documents and their mapped source records at once, producing a single `TempBlob`.
+
+## Inbound V2 flow
+
+V2 import processing is a four-step pipeline where each step transitions the E-Document to a new `Import Processing Status`. Every step is individually undoable, allowing users to fix data and re-run.
+
+```mermaid
+flowchart TD
+    A[Service: ReceiveDocuments] --> B[For each document in batch]
+    B --> C[Create E-Document record]
+    C --> D[Service: DownloadDocument]
+    D --> E[Store blob in Data Storage]
+    E --> F["Status: Unprocessed"]
+
+    F --> G["Step 1: Structure received data"]
+    G --> H{Already structured?}
+    H -- Yes --> I["Reuse unstructured as structured"]
+    H -- No --> J["Call IStructureReceivedEDocument"]
+    J --> K["Store structured blob"]
+    K --> L["Determine Read into Draft impl"]
+    I --> M["Status: Readable"]
+    L --> M
+
+    M --> N["Step 2: Read into Draft"]
+    N --> O["Call IStructuredFormatReader.ReadIntoDraft"]
+    O --> P["Populate E-Doc Purchase Header/Lines"]
+    P --> Q["Status: Ready for draft"]
+
+    Q --> R["Step 3: Prepare draft"]
+    R --> S["Call IProcessStructuredData.PrepareDraft"]
+    S --> T["Resolve vendor, items, accounts"]
+    T --> U["Status: Draft ready"]
+
+    U --> V["Step 4: Finish draft"]
+    V --> W["Call IEDocumentFinishDraft.ApplyDraftToBC"]
+    W --> X["Create BC Purchase Invoice"]
+    X --> Y["Link E-Document to Purchase Invoice"]
+    Y --> ZZ["Status: Processed"]
+```
+
+The import processing statuses form an ordered sequence:
+
+| Status | After step | Meaning |
+|--------|-----------|---------|
+| Unprocessed | (initial) | Document received, blob stored |
+| Readable | Structure received data | Structured content available |
+| Ready for draft | Read into Draft | Draft purchase header/lines populated |
+| Draft ready | Prepare draft | Vendor, items, accounts resolved |
+| Processed | Finish draft | BC document created and linked |
+
+**`ImportEDocumentProcess`** (codeunit 6104) is the orchestrator. It's a `Codeunit.Run` target configured with a step to execute and an optional undo flag. The `ConfigureImportRun` method sets the step, and then `OnRun` dispatches to the appropriate local procedure.
+
+### Receiving
+
+The receive path starts in `EDocIntegrationManagement`. For V2:
+
+1. `IDocumentReceiver.ReceiveDocuments` is called, populating a `Temp Blob List` with metadata for each available document.
+2. For each metadata blob, a new E-Document record is created (direction = Incoming, status = Created).
+3. `IDocumentReceiver.DownloadDocument` fetches the actual content, storing it in `ReceiveContext.GetTempBlob()`.
+4. The blob is saved to `E-Doc. Data Storage` and linked as the E-Document's `Unstructured Data Entry No.`.
+5. If the service has `Automatic Import Processing` enabled, the pipeline starts immediately.
+
+### Undo behavior
+
+Each step can be undone, which reverts the E-Document to the previous status:
+
+- **Undo Finish Draft**: calls `IEDocumentFinishDraft.RevertDraftActions` (deletes the created BC document), clears `Document Record ID`.
+- **Undo Prepare Draft**: deletes header mappings, clears vendor assignment, resets `Document Type` to None.
+- **Undo Structure**: clears `Structured Data Entry No.`.
+- **Undo Read into Draft**: handled by re-running -- the draft tables are overwritten.
+
+The `E-Document Log` entry for an undone step gets its `Step Undone` flag set to `true`.
+
+### V1 legacy path
+
+V1 documents (where `Import Process = "Version 1.0"`) bypass the four-step pipeline entirely. When `ImportEDocumentProcess` detects V1, it only responds to the "Finish Draft" step, delegating to `EDocImport.V1_ProcessEDocument`. This calls the old format interface methods (`GetBasicInfoFromReceivedDocument` and `GetCompleteInfoFromReceivedDocument`) in sequence. There is no undo capability in V1.
+
+## Error recovery
+
+Errors are handled at two levels:
+
+1. **Runtime errors** in format/connector code are caught by the `Codeunit.Run` pattern. The framework calls `Commit()` before running the interface code, so if it throws, the E-Document and its state survive. The error text is captured via `GetLastErrorText()` and logged through `EDocumentErrorHelper.LogSimpleErrorMessage`.
+
+2. **Business errors** are logged by connectors calling `EDocumentErrorHelper` directly. These don't throw runtime errors but still transition the service status to an error state.
+
+Error statuses include: `Export Error`, `Sending Error`, `Cancel Error`, `Imported Document Processing Error`, `Approval Error`. Each maps to the aggregate `E-Document Status::Error` via the `E-Doc Error Status` implementation of `IEDocumentStatus`.
+
+Recovery paths:
+
+- **Re-export**: Call `EDocExport.Recreate` to re-run the export step.
+- **Re-send**: The workflow can be restarted, or the send step can be retried.
+- **Re-import**: Use the undo mechanism to revert to an earlier step, fix data, and re-run forward.
+
+## Order matching
+
+For incoming invoices that reference a purchase order, the PO matching feature resolves draft purchase lines against existing PO lines or receipt lines.
+
+The matching can be manual (user selects PO lines on `EDocSelectPOLines` page) or AI-assisted (Copilot). The Copilot path (`EDocPOCopilotMatching`) uses Azure OpenAI function calling to propose matches based on descriptions, quantities, and amounts.
+
+Configuration lives in `E-Doc. PO Matching Setup` (table in `PurchaseOrderMatching/`). It defines whether to match against PO lines, receipt lines, or both, and the tolerance levels for amount/quantity differences.
+
+Matches are stored in `E-Doc. Purchase Line PO Match` as a junction between draft lines, PO lines, and receipt lines (all via SystemId). When `Finish Draft` runs, these matches determine whether to update an existing PO or create a new purchase invoice.
+
+## Batch import processing
+
+Auto-import is configured per service via `Auto Import`, `Import Start Time`, and `Import Minutes between runs`. The `EDocumentBackgroundJobs` codeunit manages the job queue entries:
+
+- `HandleRecurrentImportJob` creates or updates a recurring job queue entry.
+- `HandleRecurrentBatchJob` does the same for batch sending.
+- Job queue entry IDs are stored on the service record and cleaned up on service deletion.
+
+When auto-import fires, it calls the receive flow, then optionally processes documents through the pipeline based on `Automatic Import Processing` and the service's `GetDefaultImportParameters` configuration.