fix(typing): route Safari Google Docs dictation through the clipboard #960
+174
−29
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -123,7 +123,13 @@ final class TypingService { | |
| private static let pasteboardSessionSemaphore = DispatchSemaphore(value: 1) | ||
| private static let pasteboardRestoreQueue = DispatchQueue(label: "TypingService.PasteboardRestore", qos: .utility) | ||
| private static var focusSnapshot: FocusSnapshot? | ||
|
|
||
| private static let pasteOnlyBundleIdentifiers: Set<String> = ["com.mitchellh.ghostty"] | ||
| private static let webKitBrowserBundleIdentifiers: Set<String> = [ | ||
| "com.apple.Safari", "com.apple.SafariTechnologyPreview", | ||
| ] | ||
| private static let keypressDrivenEditorTitles = ["Google Docs", "Google Slides"] | ||
| private static let axMessagingTimeoutSeconds: Float = 0.25 | ||
|
|
||
| private var textInsertionMode: SettingsStore.TextInsertionMode { | ||
| SettingsStore.shared.textInsertionMode | ||
|
|
@@ -303,36 +309,75 @@ final class TypingService { | |
| return Self.isCurrentlyFocusedElement(element, expectedPID: pid) | ||
| } | ||
|
|
||
| private func pasteOnlyTarget(preferredTargetPID: pid_t?) -> (pid: pid_t, reason: String)? { | ||
| if let preferredTargetPID, preferredTargetPID > 0 { | ||
| guard let reason = self.pasteOnlyReason(forPID: preferredTargetPID) else { return nil } | ||
| return (preferredTargetPID, reason) | ||
| } | ||
|
|
||
| if let focusedPID = self.getSystemFocusedElementAndPID()?.pid, | ||
| let reason = self.pasteOnlyReason(forPID: focusedPID) | ||
| { | ||
| return (focusedPID, reason) | ||
| } | ||
|
|
||
| if let frontmostPID = NSWorkspace.shared.frontmostApplication?.processIdentifier, | ||
| let reason = self.pasteOnlyReason(forPID: frontmostPID) | ||
| { | ||
| return (frontmostPID, reason) | ||
| } | ||
|
|
||
| return nil | ||
| } | ||
|
|
||
| private func pasteOnlyReason(forPID pid: pid_t) -> String? { | ||
| guard pid > 0, let app = NSRunningApplication(processIdentifier: pid) else { return nil } | ||
| return Self.pasteOnlyReason( | ||
| bundleIdentifier: app.bundleIdentifier, | ||
| focusedWindowTitle: Self.focusedWindowTitle(forPID: pid) | ||
| ) | ||
| } | ||
|
|
||
| /// Safari turns one synthesized unicode key event into a single `keypress` carrying only the | ||
| /// first character. Editors that build their text from `keypress`, which is what Google Docs and | ||
| /// Slides appear to do, therefore drop everything after it and need the clipboard path. The | ||
| /// title is looked up lazily so targets that match on bundle ID alone, and the far more common | ||
| /// targets that match nothing, never pay for an Accessibility round trip. | ||
| static func pasteOnlyReason( | ||
| bundleIdentifier: String?, | ||
| focusedWindowTitle: @autoclosure () -> String? | ||
| ) -> String? { | ||
| guard let bundleIdentifier else { return nil } | ||
|
|
||
| if Self.pasteOnlyBundleIdentifiers.contains(bundleIdentifier) { | ||
| return "bundleID=\(bundleIdentifier)" | ||
| } | ||
|
|
||
| guard Self.webKitBrowserBundleIdentifiers.contains(bundleIdentifier), | ||
| let title = focusedWindowTitle(), | ||
| let editor = Self.keypressDrivenEditorTitles.first(where: { title.hasSuffix($0) }) | ||
|
Comment on lines
+356
to
+358
There was a problem hiding this comment.
The routing predicate treats Safari’s page-controlled window title as proof that the destination is Google Docs or Slides. Any ordinary page whose title ends with one of those strings can therefore move otherwise directly typed dictation through How this was verified: Safari’s page-derived accessibility title is matched by suffix alone, after which the dictated string is written to the general pasteboard with only advisory markers. Knowledge Base Used: Dictation processing and typing Prompt To Fix With AIThis is a comment left during a code review.
Path: Sources/Fluid/Services/TypingService.swift
Line: 356-358
Comment:
**Page Title Controls Clipboard**
The routing predicate treats Safari’s page-controlled window title as proof that the destination is Google Docs or Slides. Any ordinary page whose title ends with one of those strings can therefore move otherwise directly typed dictation through `NSPasteboard.general`, where observers that do not honor the advisory transient markers can read or retain it. Verify the Google origin or another non-spoofable document identity before selecting the clipboard path.
**How this was verified:** Safari’s page-derived accessibility title is matched by suffix alone, after which the dictated string is written to the general pasteboard with only advisory markers.
**Knowledge Base Used:** [Dictation processing and typing](https://app.greptile.com/altic/-/custom-context/knowledge-base/altic-dev/fluidvoice/-/docs/dictation-processing-and-typing.md)
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly. |
||
| else { | ||
| return nil | ||
| } | ||
| return "document=\(editor)" | ||
| } | ||
|
|
||
| private static func focusedWindowTitle(forPID pid: pid_t) -> String? { | ||
| guard AXIsProcessTrusted(), pid > 0 else { return nil } | ||
|
|
||
| let appElement = AXUIElementCreateApplication(pid) | ||
| AXUIElementSetMessagingTimeout(appElement, Self.axMessagingTimeoutSeconds) | ||
|
|
||
| guard let window = Self.copyAXElementAttribute(from: appElement, attribute: kAXFocusedWindowAttribute as CFString) | ||
| ?? Self.copyAXElementAttribute(from: appElement, attribute: kAXMainWindowAttribute as CFString) | ||
| else { | ||
| return nil | ||
| } | ||
|
|
||
| AXUIElementSetMessagingTimeout(window, Self.axMessagingTimeoutSeconds) | ||
| return Self.stringAXAttribute(from: window, attribute: kAXTitleAttribute as CFString) | ||
| } | ||
|
|
||
| /// Activation options used to restore focus to the external target app after dictation. | ||
| /// `.activateAllWindows` is intentionally omitted: raising every window of a multi-window | ||
| /// app (e.g. WebStorm) destroys the user's window layout on each dictation (issue #748). | ||
|
|
@@ -553,14 +598,14 @@ final class TypingService { | |
| self.log("[TypingService] Attempting to type text: \"\(text.prefix(50))\(text.count > 50 ? "..." : "")\"") | ||
|
|
||
| if self.textInsertionMode == .standard, | ||
| let pasteOnlyTarget = self.pasteOnlyTarget(preferredTargetPID: preferredTargetPID) | ||
| { | ||
| self.log("[TypingService] Paste-only target detected in standard mode (PID \(pasteOnlyTarget.pid), \(pasteOnlyTarget.reason)); forcing Reliable Paste path") | ||
| if self.tryReliablePasteInsertion(text, preferredTargetPID: pasteOnlyTarget.pid) { | ||
| self.log("[TypingService] SUCCESS: Paste-only Reliable Paste path completed") | ||
| return true | ||
| } | ||
| self.log("[TypingService] Paste-only Reliable Paste path fell through to direct-typing fallbacks") | ||
| } | ||
|
|
||
| if self.textInsertionMode == .reliablePaste { | ||
|
|
@@ -874,11 +919,15 @@ final class TypingService { | |
| releasesPasteboardSessionOnReturn = false | ||
| Self.pasteboardRestoreQueue.async { | ||
| defer { Self.pasteboardSessionSemaphore.signal() } | ||
| let verificationStartedAt = ProcessInfo.processInfo.systemUptime | ||
| let verification = self.waitForFocusedTextVerification( | ||
| from: focusedTextSnapshot, | ||
| expectedText: text, | ||
| timeoutMicros: restoreDelayMicros | ||
| ) | ||
| self.bench( | ||
| "paste_verification result=\(verification.rawValue) elapsedMs=\(Self.elapsedMs(since: verificationStartedAt)) chars=\(text.count)" | ||
| ) | ||
| let pasteboard = NSPasteboard.general | ||
|
|
||
| // Avoid clobbering user clipboard changes that happened after our insertion. | ||
|
|
@@ -910,7 +959,7 @@ final class TypingService { | |
| } | ||
| self.bench("paste_target_prepared elapsedMs=\(Self.elapsedMs(since: targetStartedAt))") | ||
|
|
||
| return self.withTemporaryPasteboardString(text, restoreDelayMicros: 1_500_000) { | ||
|
There was a problem hiding this comment.
When the target app's event loop takes more than 1.5 seconds to handle the posted Cmd+V and the focused-text snapshot cannot verify completion—as is common for web editors— Useful? React with 👍 / 👎. There was a problem hiding this comment.
This Safari-specific fix also reduces the clipboard restoration window from five seconds to 1.5 seconds for every reliable, fallback, and AppleScript paste target, including the same changes at lines 1087 and 1127. If a slow destination has not consumed the transient paste when verification times out, the previous clipboard contents are restored and no dictated text may be inserted even though dispatch was reported as successful. This violates the repository requirement that Swift changes stay within the stated scope and avoid risking existing features, so the requirement must be satisfied before merging. Retain the established timeout or validate and test the shorter window independently. Rule Used: What: Ensure macOS Swift PR changes match the stated scope, don’t introduce unrelated UI/UX/theming work, and don’t risk breaking existing features. Why: Keeps reviews focused, prevents scope creep (especially UI/UX), and avoids regressions or inc... (source) Knowledge Base Used: Dictation processing and typing Prompt To Fix With AIThis is a comment left during a code review.
Path: Sources/Fluid/Services/TypingService.swift
Line: 962
Comment:
**Clipboard Window Shortened Globally**
This Safari-specific fix also reduces the clipboard restoration window from five seconds to 1.5 seconds for every reliable, fallback, and AppleScript paste target, including the same changes at lines 1087 and 1127. If a slow destination has not consumed the transient paste when verification times out, the previous clipboard contents are restored and no dictated text may be inserted even though dispatch was reported as successful. This violates the repository requirement that Swift changes stay within the stated scope and avoid risking existing features, so the requirement must be satisfied before merging. Retain the established timeout or validate and test the shorter window independently.
**Rule Used:** What: Ensure macOS Swift PR changes match the stated scope, don’t introduce unrelated UI/UX/theming work, and don’t risk breaking existing features. Why: Keeps reviews focused, prevents scope creep (especially UI/UX), and avoids regressions or inc... ([source](https://app.greptile.com/altic/-/custom-context?memory=c54a31bd-761f-45ed-8fcb-a3cb1158d02e))
**Knowledge Base Used:** [Dictation processing and typing](https://app.greptile.com/altic/-/custom-context/knowledge-base/altic-dev/fluidvoice/-/docs/dictation-processing-and-typing.md)
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly. |
||
| let dispatchStartedAt = ProcessInfo.processInfo.systemUptime | ||
| let vKey = Self.pasteVirtualKeyCode | ||
| let keyResolvedAt = ProcessInfo.processInfo.systemUptime | ||
|
|
@@ -1035,7 +1084,7 @@ final class TypingService { | |
| /// More reliable but slightly slower - copies text to clipboard then pastes | ||
| private func insertTextViaClipboard(_ text: String) -> Bool { | ||
| self.log("[TypingService] Starting clipboard-based insertion") | ||
| return self.withTemporaryPasteboardString(text, restoreDelayMicros: 1_500_000) { | ||
| let dispatchStartedAt = ProcessInfo.processInfo.systemUptime | ||
| let vKey = Self.pasteVirtualKeyCode | ||
| let keyResolvedAt = ProcessInfo.processInfo.systemUptime | ||
|
|
@@ -1075,7 +1124,7 @@ final class TypingService { | |
| return false | ||
| } | ||
|
|
||
| return self.withTemporaryPasteboardString(text, restoreDelayMicros: 1_500_000) { | ||
| let escapedAppName = appName.replacingOccurrences(of: "\"", with: "\\\"") | ||
| let script = """ | ||
| tell application "System Events" | ||
|
|
@@ -1348,11 +1397,10 @@ final class TypingService { | |
| let pollMicros: useconds_t = 50_000 | ||
| let expectedLength = max(1, (expectedText as NSString).length) | ||
| let tolerance = max(2, expectedLength / 5) | ||
| let deadline = ProcessInfo.processInfo.systemUptime + Double(timeoutMicros) / 1_000_000 | ||
|
|
||
| while ProcessInfo.processInfo.systemUptime < deadline { | ||
| usleep(pollMicros) | ||
|
|
||
| guard let current = self.captureFocusedTextSnapshot(), | ||
| current.pid == snapshot.pid | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,93 @@ | ||
| import AppKit | ||
| @testable import FluidVoice_Debug | ||
| import XCTest | ||
|
|
||
| // Safari turns one synthesized unicode key event into a single `keypress` carrying only the first | ||
| // character, so editors that build their text from `keypress` insert "H" for "Hello world". Only | ||
| // Google Docs and Slides in a WebKit browser lose text this way, and only those targets may be | ||
| // pushed off the direct-typing path onto the clipboard. | ||
|
|
||
| final class TypingServicePasteOnlyRoutingTests: XCTestCase { | ||
| private func reason(_ bundleIdentifier: String?, _ title: String?) -> String? { | ||
| TypingService.pasteOnlyReason(bundleIdentifier: bundleIdentifier, focusedWindowTitle: title) | ||
| } | ||
|
|
||
| func testKnownBundleIdentifierMatchesWithoutAWindowTitle() { | ||
| XCTAssertEqual( | ||
| self.reason("com.mitchellh.ghostty", nil), | ||
| "bundleID=com.mitchellh.ghostty", | ||
| "apps that never accept synthesized typing must match on bundle ID alone, before any AX lookup" | ||
| ) | ||
| } | ||
|
|
||
| func testTheWindowTitleIsOnlyReadWhenTheBundleIdentifierCouldMatch() { | ||
| var lookups = 0 | ||
| func reasonCountingLookups(_ bundleIdentifier: String?) -> String? { | ||
| TypingService.pasteOnlyReason( | ||
| bundleIdentifier: bundleIdentifier, | ||
| focusedWindowTitle: { | ||
| lookups += 1 | ||
| return "Quarterly notes - Google Docs" | ||
| }() | ||
| ) | ||
| } | ||
|
|
||
| _ = reasonCountingLookups("com.apple.Notes") | ||
| XCTAssertEqual(lookups, 0, "a native app must not trigger an Accessibility round trip") | ||
|
|
||
| _ = reasonCountingLookups("com.mitchellh.ghostty") | ||
| XCTAssertEqual(lookups, 0, "a bundle ID match resolves before the title is ever needed") | ||
|
|
||
| _ = reasonCountingLookups("com.apple.Safari") | ||
| XCTAssertEqual(lookups, 1, "only a WebKit browser reads the focused window title") | ||
| } | ||
|
|
||
| func testSafariMatchesGoogleDocsAndSlides() { | ||
| XCTAssertEqual( | ||
| self.reason("com.apple.Safari", "Quarterly notes - Google Docs"), | ||
| "document=Google Docs", | ||
| "Docs in Safari drops everything after the first character and needs the clipboard path" | ||
| ) | ||
| XCTAssertEqual( | ||
| self.reason("com.apple.Safari", "Launch deck - Google Slides"), | ||
| "document=Google Slides", | ||
| "Slides truncates the same way Docs does" | ||
| ) | ||
| } | ||
|
|
||
| func testGoogleSheetsKeepsTheDirectTypingPath() { | ||
| XCTAssertNil( | ||
| self.reason("com.apple.Safari", "Budget - Google Sheets"), | ||
| "Sheets inserts the full string in Safari, so forcing it onto the clipboard would be a regression" | ||
| ) | ||
| } | ||
|
|
||
| func testChromiumAndGeckoKeepTheDirectTypingPath() { | ||
| XCTAssertNil( | ||
| self.reason("com.google.Chrome", "Quarterly notes - Google Docs"), | ||
| "Chromium fires no keypress and inserts the full string" | ||
| ) | ||
| XCTAssertNil( | ||
| self.reason("org.mozilla.firefox", "Quarterly notes - Google Docs"), | ||
| "Gecko fires one keypress per character and inserts the full string" | ||
| ) | ||
| } | ||
|
|
||
| func testOrdinaryBrowsingDoesNotMatch() { | ||
| XCTAssertNil( | ||
| self.reason("com.apple.Safari", "Apple"), | ||
| "a WebKit browser alone is not enough; only the affected documents may be rerouted" | ||
| ) | ||
| } | ||
|
|
||
| func testNativeAppsAreNeverForcedOntoTheClipboard() { | ||
| XCTAssertNil( | ||
| self.reason("com.apple.Notes", "Quarterly notes - Google Docs"), | ||
| "a matching window title in a native app must not trigger the browser rule" | ||
| ) | ||
| XCTAssertNil( | ||
| self.reason(nil, nil), | ||
| "an unidentifiable target keeps the default path" | ||
| ) | ||
| } | ||
| } |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When the Google account/UI language localizes the product name (for example, Safari exposes a title ending in the localized equivalent of Google Docs), this exact English suffix check returns
nil, so the affected editor stays on the direct-typing path and continues dropping all but the first character. The routing signal needs to be independent of the localized window title, or explicitly support localized variants.Useful? React with 👍 / 👎.