Skip to content

Fix client dispatcher deadlock on SendRequest during stop#6

Open
andig wants to merge 1 commit into
masterfrom
fix/client-dispatcher-deadlock
Open

Fix client dispatcher deadlock on SendRequest during stop#6
andig wants to merge 1 commit into
masterfrom
fix/client-dispatcher-deadlock

Conversation

@andig

@andig andig commented Jun 8, 2026

Copy link
Copy Markdown
Member

DefaultClientDispatcher.SendRequest does a blocking send on the requestChannel notification channel:

d.mutex.RLock()
d.requestChannel <- true
d.mutex.RUnlock()

The caller chain holds the callback-queue mutex across this send (chargePoint.SendRequestAsynccallbackqueue.TryQueue holds callbacksMutex.Lock() while running try()SendRequest). Once the message pump has stopped draining requestChannel (during/after teardown), this send blocks forever, so the callbacks mutex is never released and the concurrent clearCallbacks on stop (asyncCallbackHandler<-stopC) deadlocks on the same mutex. The send can also panic on a closed channel after Stop.

This surfaced as a 10-minute hang in a downstream test suite (charge point connect/send/stop cycling), with one goroutine blocked at dispatcher.go on requestChannel <- true while holding the callbacks mutex, and another blocked acquiring that mutex in clearCallbacks.

requestChannel is only a wake-up doorbell — the request enqueued in requestQueue is the source of truth, and the pump re-checks the queue on every wake (paced by readyForDispatch/responses, not one dispatch per doorbell). So a dropped doorbell is harmless. This change:

  • makes the doorbell send non-blocking (select { case d.requestChannel <- true: default: }), so SendRequest never blocks;
  • adds a stopped signal channel (mirroring the server dispatcher's stoppedC), and bails out of SendRequest when stopping to avoid sending on the closed channel;
  • holds the read lock for the whole call (mutually exclusive with Stop's write lock) so requestChannel cannot be closed mid-send.

Adds a regression test (TestClientSendRequestAfterStop) that blocks/fails without the fix and passes with it. Existing ./ocppj/... suite stays green.

DefaultClientDispatcher.SendRequest did a blocking send on the requestChannel
notification channel while the caller (chargePoint.SendRequestAsync via the
callback queue) holds the callbacks mutex. Once the message pump has stopped
draining the channel, that send blocks forever, so the mutex is never released
and the concurrent clearCallbacks on teardown deadlocks. The send could also
panic on a closed channel after Stop.

The notification channel is only a wake-up doorbell; the request enqueued in
requestQueue is the source of truth and the pump re-checks the queue on every
wake. Make the doorbell send non-blocking and add a stopped signal so
SendRequest bails out once the dispatcher is stopping instead of blocking or
sending on a closed channel.

Adds a regression test that hangs without the fix.
@coveralls

Copy link
Copy Markdown

Coverage Report for CI Build 27122869506

Coverage increased (+0.01%) to 82.919%

Details

  • Coverage increased (+0.01%) from the base build.
  • Patch coverage: 2 uncovered changes across 1 file (16 of 18 lines covered, 88.89%).
  • No coverage regressions found.

Uncovered Changes

File Changed Covered %
ocppj/dispatcher.go 18 16 88.89%

Coverage Regressions

No coverage regressions found.


Coverage Stats

Coverage Status
Relevant Lines: 4508
Covered Lines: 3738
Line Coverage: 82.92%
Coverage Strength: 18.03 hits per line

💛 - Coveralls

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants