Improve Apple Pencil drawing quality in Brush Tool

[christianms-itx]

The WeaveBrushToolAction produces low-quality strokes when using Apple Pencil on iPad. Strokes are thin, jagged at high speeds, and exhibit pressure jitter. The root cause is that the brush tool only captures one point per animation frame via standard pointermove, ignoring the high-frequency input data Apple Pencil provides (up to 240Hz).

This issue proposes 7 targeted changes to significantly improve stylus input quality.

---

Problem Analysis

Issue	Root Cause	Impact
Jagged strokes at speed	Only 1 point captured per frame (~60Hz) while Pencil reports at ~240Hz	Angular appearance, missing curve detail
Thin/invisible strokes	No minimum width floor; low pressure → subpixel width	Strokes disappear at light touch
Pressure jitter	Raw pressure used directly, no smoothing	Inconsistent stroke width, wobbly edges
Perceived latency	No predictive rendering	Stroke tip lags ~20-30ms behind pencil
Accidental palm marks	No palm rejection while pen is active	Unwanted dots/strokes from palm contact
Browser gesture interference	No touch-action: none on canvas	Accidental scroll/zoom during drawing

---

Proposed Changes

Change 1: Capture coalesced pointer events

File: packages/sdk/src/actions/brush-tool/brush-tool.ts
Method: setupEvents() → handlePointerMove handler (line ~144)

What: Use PointerEvent.getCoalescedEvents() to capture all intermediate points the browser batched between frames. Apple Pencil on iPad delivers ~4 points per frame at 60fps.

Current code:

const handlePointerMove = (e: Konva.KonvaEventObject<PointerEvent>) => {
  // ...guards...
  const pointPressure = this.getEventPressure(e);
  this.handleMovement(pointPressure);
  e.evt.stopPropagation();
};

Proposed code:

const handlePointerMove = (e: Konva.KonvaEventObject<PointerEvent>) => {
  // ...guards...
  const coalescedEvents = e.evt.getCoalescedEvents ? e.evt.getCoalescedEvents() : [];
  if (coalescedEvents.length > 1) {
    for (const ce of coalescedEvents) {
      const pressure = (ce.pointerType === 'pen') ? (ce.pressure || 0.5) : 0.5;
      this.handleMovementFromEvent(pressure, ce);
    }
  } else {
    const pointPressure = this.getEventPressure(e);
    this.handleMovement(pointPressure);
  }
  e.evt.stopPropagation();
};

Why: Without coalesced events, points between frames are lost forever. This is the single most impactful change — it captures ~4x more data points from the Pencil, making strokes dramatically smoother at high drawing speeds.

Note: handleMovementFromEvent is a new overload of handleMovement that accepts a raw PointerEvent and computes the canvas-local position from clientX/clientY using the stage's bounding rect and the container's inverse absolute transform, instead of relying on Konva's getPointerPosition() (which only reflects the latest event, not coalesced sub-events).

---

Change 2: Predictive rendering for reduced latency

File: packages/sdk/src/actions/brush-tool/brush-tool.ts
Method: setupEvents() → handlePointerMove handler (line ~144)

What: After processing real points, use PointerEvent.getPredictedEvents() to render a provisional "tip" point that reduces perceived latency. On the next real frame, predicted points are discarded and replaced by actual data.

Proposed addition (after coalesced events processing):

const predictedEvents = e.evt.getPredictedEvents ? e.evt.getPredictedEvents() : [];
if (predictedEvents.length > 0) {
  const last = predictedEvents[predictedEvents.length - 1];
  const predPressure = (last.pointerType === 'pen') ? (last.pressure || 0.5) : 0.5;
  this.handleMovementFromEvent(predPressure, last, /* isPredicted */ true);
}

Implementation detail: Track a _predictedCount counter. When isPredicted=true, increment it. Before adding any real (non-predicted) point, slice off the last _predictedCount elements from strokeElements and reset the counter. This ensures predicted points are always provisional and never persist in the final stroke data.

Why: The browser can predict where the stylus is going based on velocity/acceleration. This eliminates ~20-30ms of visual latency with no accuracy cost (predictions are always replaced by real data).

---

Change 3: Pressure smoothing with velocity-adaptive EMA

File: packages/sdk/src/actions/brush-tool/brush-tool.ts
Method: getEventPressure() (line ~72)

What: Replace raw pressure pass-through with an Exponential Moving Average (EMA) whose smoothing factor adapts to drawing speed.

Current code:

private getEventPressure(e: Konva.KonvaEventObject<PointerEvent>) {
  if (e.evt.pointerType && e.evt.pointerType === 'pen') {
    return e.evt.pressure || 0.5;
  }
  return 0.5;
}

Proposed code:

private _lastSmoothedPressure = 0.5;
private _lastPointerPos: { x: number; y: number } | null = null;
private _lastPointerTime = 0;

private getEventPressure(e: Konva.KonvaEventObject<PointerEvent>): number {
  const now = performance.now();
  let velocity = 0;

  if (this._lastPointerPos && now - this._lastPointerTime > 0) {
    const dx = e.evt.clientX - this._lastPointerPos.x;
    const dy = e.evt.clientY - this._lastPointerPos.y;
    velocity = Math.hypot(dx, dy) / (now - this._lastPointerTime) * 1000; // px/s
  }
  this._lastPointerPos = { x: e.evt.clientX, y: e.evt.clientY };
  this._lastPointerTime = now;

  // Fast movement → higher alpha (less smoothing, more responsive)
  // Slow movement → lower alpha (more smoothing, eliminates jitter)
  const alpha = Math.min(Math.max(velocity / 1500, 0.15), 0.6);

  let raw: number;
  if (e.evt.pointerType === 'pen') {
    raw = e.evt.pressure || 0.5;
  } else {
    raw = 0.5;
  }

  this._lastSmoothedPressure = alpha * raw + (1 - alpha) * this._lastSmoothedPressure;
  return Math.max(this._lastSmoothedPressure, 0.15); // floor prevents invisible strokes
}

Why: Raw pressure from Apple Pencil fluctuates frame-to-frame causing visible width oscillation. A fixed EMA over-smooths fast strokes (losing expressiveness) or under-smooths slow ones (keeping jitter). Velocity-adaptive alpha solves both cases.

---

Change 4: Minimum stroke width floor in rendering

File: packages/sdk/src/nodes/stroke/stroke.ts
Method: drawRibbonWithDash() (line ~119)

What: Ensure the computed half-width never falls below 0.5px.

Current code:

const w0 = (baseW * p0.pressure) / 2;
const w1 = (baseW * p1.pressure) / 2;

Proposed code:

const w0 = Math.max((baseW * p0.pressure) / 2, 0.5);
const w1 = Math.max((baseW * p1.pressure) / 2, 0.5);

Why: When strokeWidth=1 and pressure=0.1, the rendered width is 0.05px — effectively invisible. A floor of 0.5px ensures strokes are always visible regardless of pressure, while still allowing pressure variation above the threshold.

---

Change 5: Palm rejection during pen input

File: packages/sdk/src/actions/brush-tool/brush-tool.ts
Method: setupEvents() → handlePointerDown and handlePointerUp

What: Add a _penActive flag that blocks touch-originated pointerdown events while a pen stroke is in progress.

Proposed additions:

In class properties:

private _penActive = false;

In handlePointerDown:

// After existing guards, before processing:
if (e.evt.pointerType === 'touch' && this._penActive) return;
if (e.evt.pointerType === 'pen') this._penActive = true;

In handlePointerUp:

this._penActive = false;

Why: When drawing with Apple Pencil, the palm resting on the screen can generate pointerdown events with pointerType: 'touch'. These create unwanted dots or strokes. While iPadOS has built-in palm rejection, it's not 100% reliable in web views — this provides a defensive layer.

---

Change 6: Set touch-action: none on canvas container

File: packages/sdk/src/actions/brush-tool/brush-tool.ts
Method: setupEvents() (line ~79)

What: Set CSS touch-action: none on the stage container when the brush tool initializes.

Proposed addition (at the beginning of setupEvents):

stage.container().style.touchAction = 'none';

Why: Without this, the browser may interpret pen/touch input as scroll, zoom, or navigation gestures before the JS handler fires. This adds latency and can cause the stroke to be interrupted. touch-action: none tells the browser that the application handles all pointer input directly.

---

Change 7: Reset smoothing state on stroke start

File: packages/sdk/src/actions/brush-tool/brush-tool.ts
Method: handleStartStroke() (line ~218)

What: Reset the EMA pressure state and predicted point counter when a new stroke begins.

Proposed addition (at the beginning of handleStartStroke):

this._lastSmoothedPressure = 0.5;
this._lastPointerPos = null;
this._lastPointerTime = 0;
this._predictedCount = 0;

Why: Without resetting, the first few points of a new stroke inherit the smoothed pressure from the end of the previous stroke, causing an incorrect initial width.

[markm39]

In your proposed code, getEventPressure() currently uses e.evt.pressure || 0.5. If a pen event reports pressure === 0, that gets treated as 0.5. I would think you should use nullish/type check instead:

const pressure =
  e.evt.pointerType === 'pen' && typeof e.evt.pressure === 'number'
    ? e.evt.pressure
    : 0.5;