Fix ExecOnboardTenant pending-state polling and restore Start-CIPPOrchestrator queue routing

[Copilot]

ExecOnboardTenant could appear stuck in pending because status polling used a 10-minute freshness filter when reading TenantOnboarding. Once a job aged past that window, polling treated it as missing and recreated a new queued record, resetting visible progress.

Follow-up feedback also required keeping onboarding launches on Start-CIPPOrchestrator and tracing why those invocations were not working. The invocation issue was in Start-CIPPOrchestrator path selection: direct orchestration starts were occurring for any -InputObject call, bypassing the intended store+queue flow.

• Root cause addressed: stale-read filter in status endpoint

  • Removed timestamp-gated lookup from Invoke-ExecOnboardTenant.
  • Existing onboarding records are now fetched by RowKey only, so long-running jobs continue to surface their actual state (running/step progress) instead of being reset.

• Root cause addressed: orchestrator invocation routing

  • Kept onboarding launch path on:
    • Start-CIPPOrchestrator -InputObject $InputObject
  • Fixed Start-CIPPOrchestrator so direct Start-NewOrchestration only occurs when explicitly called with -CallerIsQueueTrigger.
  • HTTP-triggered calls now correctly follow the wrapper’s store+queue path instead of attempting direct starts.

• Behavioral impact

  • Normal polling: returns the current onboarding row without restarting orchestration.
  • Explicit retry (Retry=true): re-initializes onboarding and launches via Start-CIPPOrchestrator.
  • Queue routing: onboarding jobs are queued through Start-CIPPOrchestrator (not direct endpoint queue assignment).

• Focused regression coverage

  • Updated endpoint-level tests for Invoke-ExecOnboardTenant:
    • existing row + no retry => no restart
    • existing row + retry => orchestrator launch path is used with expected onboarding payload
  • Added focused tests for Start-CIPPOrchestrator:
    • default -InputObject path stores input and queues Start-CIPPOrchestrator
    • direct start occurs only when -CallerIsQueueTrigger is set

# Invoke-ExecOnboardTenant Before
$TenantOnboarding = Get-CIPPAzDataTableEntity @OnboardTable -Filter "RowKey eq '$Id' and Timestamp ge datetime'$TenMinutesAgo'"

# Invoke-ExecOnboardTenant After
$TenantOnboarding = Get-CIPPAzDataTableEntity @OnboardTable -Filter "RowKey eq '$Id'"

# Onboarding launch path (final)
$InputObject = @{
    OrchestratorName = 'OnboardingOrchestrator'
    Batch            = @($Item)
}
$null = Start-CIPPOrchestrator -InputObject $InputObject

# Start-CIPPOrchestrator routing fix (final)
if ($InputObject -and $CallerIsQueueTrigger -and -not $OrchestratorTriggerDisabled) {
    $InstanceId = Start-NewOrchestration -FunctionName 'CIPPOrchestrator' -InputObject ($InputObject | ConvertTo-Json -Depth 10 -Compress)
    return $InstanceId
}

Original prompt

When running ExecOnboardTenant it's getting stuck in the pending state in the scheduler. Can you please trace it's execution flow and why it might be happening

---

🔒 GitHub Advanced Security automatically protects Copilot coding agent pull requests. You can protect all pull requests by enabling Advanced Security for your repositories. Learn more about Advanced Security.

[Zacgoose]

@copilot look into the actual task execution they are atialling and never getting pushed to the function app queue.

[Copilot]

@copilot look into the actual task execution they are atialling and never getting pushed to the function app queue.

Fixed in 8ce008e. I traced the execution path and found Start-CIPPOrchestrator was starting directly when called with -InputObject, so jobs could bypass Add-CippQueueMessage and never hit the function app queue. It now queues by default and only starts directly when explicitly called with -CallerIsQueueTrigger. I also added focused tests for both paths.

[Zacgoose]

@copilot dont edit the scheduler function it seems we need to make changes to how the onboarding is pushed to the queue? Perhaps you can see how it's done for the user offboarding endpoint?

[Copilot]

@copilot dont edit the scheduler function it seems we need to make changes to how the onboarding is pushed to the queue? Perhaps you can see how it's done for the user offboarding endpoint?

Done in e261070. I reverted the scheduler/orchestrator wrapper edits and changed onboarding queueing in Invoke-ExecOnboardTenant to push directly with Add-CippQueueMessage -Cmdlet 'Push-ExecOnboardTenantQueue' (matching the offboarding-style queue pattern of endpoint-triggered background execution). I also updated tests to assert retry now queues Push-ExecOnboardTenantQueue and normal polling does not requeue. No UI changes in this fix (backend-only).

[Zacgoose]

@copilot other than assigning directly to the queue are there other methods that will still let us peek into its progress like that endpoint currently expects

[Copilot]

@copilot other than assigning directly to the queue are there other methods that will still let us peek into its progress like that endpoint currently expects

Yes. The key is not where it’s queued, but having a durable progress record the endpoint can read. Two workable options:

1. Keep using Start-CIPPOrchestrator (queued path) and have onboarding activities write step/status/logs to TenantOnboarding (same polling contract as now).
2. Use durable InstanceId + orchestrator custom status and poll durable status directly, with TenantOnboarding as a compatibility mirror if needed.

So direct Add-CippQueueMessage isn’t the only way to preserve progress visibility.

[Zacgoose]

@copilot right stay with Start-CIPPOrchestrator then but just trace through it's setup and fucb yout why those invocations are not working

[Zacgoose]

@copilot right stay with Start-CIPPOrchestrator then but just trace through it's setup and fucb yout why those invocations are not working

[Copilot]

@copilot right stay with Start-CIPPOrchestrator then but just trace through it's setup and fucb yout why those invocations are not working

Fixed in bbce3f3. I switched onboarding back to Start-CIPPOrchestrator and traced the invocation issue to Start-CIPPOrchestrator starting directly whenever -InputObject was present, which bypassed its queued setup path. It now only starts directly when -CallerIsQueueTrigger is set, so HTTP calls store+queue correctly. I also updated/added focused tests for both the onboarding endpoint and orchestrator routing.

@copilot right stay with Start-CIPPOrchestrator then but just trace through it's setup and fucb yout why those invocations are not working

Fixed in bbce3f3. Onboarding now stays on Start-CIPPOrchestrator, and the invocation break was in Start-CIPPOrchestrator path selection: direct starts were happening for any -InputObject call. I restored queue-trigger gating so direct start only happens with -CallerIsQueueTrigger, while HTTP invocations follow the store+queue flow. Added focused coverage for both paths.