All routes intermittently fail (running pode in pode.web, unclear pode.web is related)

[thoffmannaspenware]

Describe the Bug

Pode routes intermittently fail. We are on 0.8.3 of Pode.Web, and 2.12.1 of pode. We have seen this 5-6x over a month. Unfortunately, it's so intermittent we don't really know the cause. But I do have a stack trace. We are running in multi-threaded mode, about 10 threads.

Steps To Reproduce

Unknown unfortunately. It does seem to span all threads, or at least many threads.

Expected Behavior

Pode continues to serve routes

Platform

• OS: Linux
• Browser: Edge, but all really
• Versions:
  • Pode.Web: 0.8.3
  • Pode: 2.12.1

Additional Context

========== BEGIN PODE LOG ==========
ThreadId: 3
Server: 1e6e6f272947
Category: OperationStopped: (:) [Get-PodeRoute], NullReferenceException
Message: Object reference not set to an instance of an object.
StackTrace: at Get-PodeRoute, /home/site/wwwroot/ps_modules/Pode/2.12.1/Public/Routes.ps1: line 2408
at Test-PodeWebRoute, /home/site/wwwroot/ps_modules/Pode.Web/0.8.3/Private/Helpers.ps1: line 443
at Register-PodeWebComponentEventInternal, /home/site/wwwroot/ps_modules/Pode.Web/0.8.3/Private/Events.ps1: line 48
at Register-PodeWebEvent, /home/site/wwwroot/ps_modules/Pode.Web/0.8.3/Public/Events.ps1: line 30
at <ScriptBlock>, <No file>: line 10
at Invoke-PodeScriptBlock, /home/site/wwwroot/ps_modules/Pode/2.12.1/Public/Utilities.ps1: line 623
at <ScriptBlock>, /home/site/wwwroot/ps_modules/Pode.Web/0.8.3/Public/Pages.ps1: line 514
at Invoke-PodeScriptBlock, /home/site/wwwroot/ps_modules/Pode/2.12.1/Public/Utilities.ps1: line 615
at <ScriptBlock>, <No file>: line 125

Date: 2026-04-06 22:12:57
Level: Error
ThreadId: 3
Server: 1e6e6f272947
Category: System.Collections.Specialized
Message: Object reference not set to an instance of an object.
StackTrace:    at System.Collections.Specialized.OrderedDictionary.OrderedDictionaryEnumerator.get_Current()
   at System.Management.Automation.EnumerableOps.PropertyGetter(PSGetMemberBinder binder, IEnumerator enumerator)
   at Get-PodeRoute(Closure, FunctionContext)

Date: 2026-04-06 22:12:58
Level: Error
ThreadId: 7
Server: 1e6e6f272947
Category: InvalidOperation: (System.Collections.…ictionaryEnumerator:OrderedDictionaryEnumerator) [Find-PodeRoute], RuntimeException
Message: An error occurred while enumerating through a collection: Object reference not set to an instance of an object..
StackTrace: at Find-PodeRoute, /home/site/wwwroot/ps_modules/Pode/2.12.1/Private/Routes.ps1: line 71
at <ScriptBlock>, /home/site/wwwroot/ps_modules/Pode/2.12.1/Private/Middleware.ps1: line 257
at Invoke-PodeScriptBlock, /home/site/wwwroot/ps_modules/Pode/2.12.1/Public/Utilities.ps1: line 615
at Invoke-PodeMiddleware, /home/site/wwwroot/ps_modules/Pode/2.12.1/Private/Middleware.ps1: line 41
at <ScriptBlock>, <No file>: line 97

Date: 2026-04-06 22:50:21
Level: Error
ThreadId: 3
Server: 1e6e6f272947
Category: OperationStopped: (:) [Get-PodeRoute], NullReferenceException
Message: Object reference not set to an instance of an object.
StackTrace: at Get-PodeRoute, /home/site/wwwroot/ps_modules/Pode/2.12.1/Public/Routes.ps1: line 2408
at Test-PodeWebRoute, /home/site/wwwroot/ps_modules/Pode.Web/0.8.3/Private/Helpers.ps1: line 443
at Register-PodeWebComponentEventInternal, /home/site/wwwroot/ps_modules/Pode.Web/0.8.3/Private/Events.ps1: line 48
at Register-PodeWebEvent, /home/site/wwwroot/ps_modules/Pode.Web/0.8.3/Public/Events.ps1: line 30
at <ScriptBlock>, <No file>: line 10
at Invoke-PodeScriptBlock, /home/site/wwwroot/ps_modules/Pode/2.12.1/Public/Utilities.ps1: line 623
at <ScriptBlock>, /home/site/wwwroot/ps_modules/Pode.Web/0.8.3/Public/Pages.ps1: line 514
at Invoke-PodeScriptBlock, /home/site/wwwroot/ps_modules/Pode/2.12.1/Public/Utilities.ps1: line 615
at <ScriptBlock>, <No file>: line 125

Date: 2026-04-06 22:50:21
Level: Error
ThreadId: 3
Server: 1e6e6f272947
Category: System.Collections.Specialized
Message: Object reference not set to an instance of an object.
StackTrace:    at System.Collections.Specialized.OrderedDictionary.OrderedDictionaryEnumerator.get_Current()
   at System.Management.Automation.EnumerableOps.PropertyGetter(PSGetMemberBinder binder, IEnumerator enumerator)
   at Get-PodeRoute(Closure, FunctionContext)

Date: 2026-04-06 22:50:21
Level: Error
ThreadId: 7
Server: 1e6e6f272947
Category: OperationStopped: (:) [Get-PodeRoute], NullReferenceException
Message: Object reference not set to an instance of an object.
StackTrace: at Get-PodeRoute, /home/site/wwwroot/ps_modules/Pode/2.12.1/Public/Routes.ps1: line 2408
at Test-PodeWebRoute, /home/site/wwwroot/ps_modules/Pode.Web/0.8.3/Private/Helpers.ps1: line 443
at Register-PodeWebComponentEventInternal, /home/site/wwwroot/ps_modules/Pode.Web/0.8.3/Private/Events.ps1: line 48
at Register-PodeWebEvent, /home/site/wwwroot/ps_modules/Pode.Web/0.8.3/Public/Events.ps1: line 30
at <ScriptBlock>, <No file>: line 10
at Invoke-PodeScriptBlock, /home/site/wwwroot/ps_modules/Pode/2.12.1/Public/Utilities.ps1: line 623
at <ScriptBlock>, /home/site/wwwroot/ps_modules/Pode.Web/0.8.3/Public/Pages.ps1: line 514
at Invoke-PodeScriptBlock, /home/site/wwwroot/ps_modules/Pode/2.12.1/Public/Utilities.ps1: line 615
at <ScriptBlock>, <No file>: line 125

Date: 2026-04-06 22:50:21
Level: Error
ThreadId: 7
Server: 1e6e6f272947
Category: System.Collections.Specialized
Message: Object reference not set to an instance of an object.
StackTrace:    at System.Collections.Specialized.OrderedDictionary.OrderedDictionaryEnumerator.get_Current()
   at System.Management.Automation.EnumerableOps.PropertyGetter(PSGetMemberBinder binder, IEnumerator enumerator)
   at Get-PodeRoute(Closure, FunctionContext)

Date: 2026-04-06 22:50:21
Level: Error
ThreadId: 4
Server: 1e6e6f272947
Category: InvalidOperation: (System.Collections.…ictionaryEnumerator:OrderedDictionaryEnumerator) [Find-PodeRoute], RuntimeException
Message: An error occurred while enumerating through a collection: Object reference not set to an instance of an object..
StackTrace: at Find-PodeRoute, /home/site/wwwroot/ps_modules/Pode/2.12.1/Private/Routes.ps1: line 71
at <ScriptBlock>, /home/site/wwwroot/ps_modules/Pode/2.12.1/Private/Middleware.ps1: line 257
at Invoke-PodeScriptBlock, /home/site/wwwroot/ps_modules/Pode/2.12.1/Public/Utilities.ps1: line 615
at Invoke-PodeMiddleware, /home/site/wwwroot/ps_modules/Pode/2.12.1/Private/Middleware.ps1: line 41
at <ScriptBlock>, <No file>: line 97

========== END PODE LOG ==========

[thoffmannaspenware]

I discovered a bit more about this issue, although not a lot. This seems to happen on pages and not routes. We have a ctx page that looks at azure context, and a healthz/ready that just says 'ok'. Both of those worked, but pages failed.

ctx is authenticated, healthz/ready is not, so that isn't likely the cause. I think the only relevant bit is being a page vs a route.

We plan to add a health check page that is basically empty so we can confirm this and also add autoheal behavior off of this health page.

It's not at all clear to me if this is a defect in podes engine, something we are doing with execution or context or code on the various pages, or both. I just don't have enough information unfortunately.

[thoffmannaspenware]

We looked a bit further at this, and discovered a few interesting things:
a minimal page that returns 'OK' called 'healthpage' works
strangely, /doesnotexist yields the 500, when it would typically yield a 404

The state seems to be perpetual. Once it is 'in this state' it is busted until a restart

We have a page that is mostly wrapped in try catch. It loads for the most part, but no contents, just a blank page

Here is that code:

Add-PodeWebPage -Name '99-GetConsumedOriginGroups' -DisplayName 'Consumed Origin Groups' -Icon 'counter' -Group "Frontdoor" -ScriptBlock {
    Write-Verbose 'Consumed FrontDoor Origin Groups Entered' -Verbose

    Set-AwpAccountContext

    try {
        Write-Verbose 'Consumed FrontDoor Origin Groups Entered' -Verbose

        New-PodeWebCard -Name "Consumed FrontDoor Origin Groups" -Content @(
            New-PodeWebTable -Name "Counts" -ScriptBlock {
                # Run both origin queries in parallel
                [string]$scriptblock = Get-AwParallelScriptBlock;

                $results = @('UAT', 'Prod') | ForEach-Object -Parallel {
                    $using:scriptblock | Invoke-Expression
                    @{
                        "Consumed $_ Origin Groups" = "$((Get-AwFrontDoorOriginGroup -CommonEnvironment $_).count) / 600"
                    }
                }

                $results | Format-AwFlatDictionary
            }
        )
    }
    catch {
        Show-PodeWebToast -Icon 'alert-circle' -Title "Error" -Duration 3600000 -Message $_
    }
}

So that page loads, but the toast does not present the error (or present at all for that matter), and the error in the pode log is the same one I have in this ticket already.

What is so odd to me is that a 404 becomes a 500, most pages throw a 500 (both single and group pages), but some pages do load either partially or completely.

We are running in multithreaded, thread collisions or lack of safety is entirely possible.

[thoffmannaspenware]

We've gotten much further, and have learned some sort of nullable object is entering the $PodeContext.Server.Routes.Values.Values, specifically in the 'post' property of $PodeContext.Server.Routes.Values. .Values calls fail, due to some nullable or bad primitive that doesn't belong.

Unclear if our code added that off of some bad write-host or write-output or something, or if this is an issue in pode itself or what

[thoffmannaspenware]

Fun summary of our efforts to interrogate the object in a live debug of the server

[thoffmannaspenware]

Our current theory is this is probably thread safety related. We are running 20 threads. Are you using concurrent dictionaries for the $PodeContext.Server.Routes and affiliated friends? It seems like that is read from quite frequently, and possibly written to as well across threads. Speculative on my part, but maybe helpful.

This very contrived code will reproduce similar behavior, where we see low level failures in powershell with object references not found.

$shared = [System.Collections.Specialized.OrderedDictionary]::new()

$jobs = 1..10 | ForEach-Object {
    Start-ThreadJob -ArgumentList $shared, $_ -ScriptBlock {
        param($dict, $jobId)

        $errors = [System.Collections.Generic.List[object]]::new()

        1..5000 | ForEach-Object {
            $i = $_
            $key = "job${jobId}_$($i % 200)"

            try {
                if (($i % 3) -eq 0) {
                    if ($dict.Contains($key)) {
                        $dict[$key] = [ordered]@{
                            Job  = $jobId
                            Iter = $i
                            When = [datetime]::UtcNow
                        }
                    }
                    else {
                        $dict.Add($key, [ordered]@{
                            Job  = $jobId
                            Iter = $i
                            When = [datetime]::UtcNow
                        })
                    }
                }
                elseif (($i % 3) -eq 1) {
                    if ($dict.Contains($key)) {
                        $dict.Remove($key)
                    }
                }
                else {
                    $null = $dict.Count
                    $null = $dict.Keys
                    $null = $dict.Values
                    $null = $dict.GetEnumerator()
                }
            }
            catch {
                $errors.Add([pscustomobject]@{
                    Job   = $jobId
                    Iter  = $i
                    Type  = $_.Exception.GetType().FullName
                    Error = $_.Exception.Message
                })
            }
        }

        $errors
    }
}

$results = $jobs | Receive-Job -Wait -AutoRemoveJob

$results | Format-Table -AutoSize

"`nFinal type: $($shared.GetType().FullName)"
try { "Final count: $($shared.Count)" } catch { "Count failed: $($_.Exception.Message)" }
try { "Keys count: $($shared.Keys.Count)" } catch { "Keys failed: $($_.Exception.Message)" }
try { "Values count: $($shared.Values.Count)" } catch { "Values failed: $($_.Exception.Message)" }
try { $null = $shared.GetEnumerator(); "GetEnumerator OK" } catch { "GetEnumerator failed: $($_.Exception.Message)" }

Then call shared in a terminal. You see failures all the way down to the powershell level:

PS C:\git> $shared
format-default: Object reference not set to an instance of an object.

PS C:\git> $shared.Values.Values
InvalidOperation: An error occurred while enumerating through a collection: Object reference not set to an instance of an object..

[Badgerati]

Hi @thoffmannaspenware,

Thanks for looking into this further :)

I was also looking into this earlier myself, and I believe you're correct: it's a treading issue.

The null reference error on .Values and .Keys on Routes is the hint, and Pode.Web does create Routes during the running of the Server - so a lot of threads would exasperate the issue.

The Routes are all currently stored in an OrderedDictionary - notably not thread-safe. I'm looking into changing this to be tread-safe, but sadly can't simply make it a ConcurrentDictionary as the ordering is required.

There's a couple of options I need to test, but I'm currently leaning into making a ConcurrentOrderedDictionary - so that data structure itself is thread-safe, and I'm not spamming the powershell code with Lock-PodeObject everywhere 😅

[thoffmannaspenware]

Thanks so much for the reply! Nice to hear from the legend himself :). That makes sense. Bummer about the ordering being important, that definitely makes things harder.

Is there a call we can trigger to rebuild the routes when we get into this state (apart from a system restart)?

It seems like this hack below may work around the problem by deliberately avoiding low level function PowerShell calls. But I don't know what will happen to the poor route that was in mid write, or whatever happened that mangled item index 62 in the example I screenshotted. We just dropped this into our test environment with some overrides. It's unclear yet if it is stable. We shall see. Currently we only encounter this once every few days. Enough of a problem that we dug in more. Maybe most routes will work now, but some will fail. Perhaps the dictionary will slowly degrade. I'm not sure.

Also, do you do professional support / contracting / consulting for Pode (bug hunt fixes, code reviews, etc)? I don't have approval currently, but it's worth having that answered. It's not totally outside the realm of possibility if you are willing and support becomes more critical.

function Get-PodeRoute {
    [CmdletBinding()]
    [OutputType([System.Object[]])]
    param(
        [Parameter()]
        [ValidateSet('', 'Connect', 'Delete', 'Get', 'Head', 'Merge', 'Options', 'Patch', 'Post', 'Put', 'Trace', '*')]
        [string]
        $Method,

        [Parameter()]
        [string]
        $Path,

        [Parameter()]
        [string[]]
        $EndpointName
    )

    # start off with every route

    try {
        Write-Verbose "route hack!!" -Verbose
        $methodRoutes = @(
            $PodeContext.Server.Routes.get,
            $PodeContext.Server.Routes.post,
            $PodeContext.Server.Routes.put,
            $PodeContext.Server.Routes.patch,
            $PodeContext.Server.Routes.delete,
            $PodeContext.Server.Routes.connect,
            $PodeContext.Server.Routes.head,
            $PodeContext.Server.Routes.merge,
            $PodeContext.Server.Routes.options,
            $PodeContext.Server.Routes.trace,
            $PodeContext.Server.Routes.static,
            $PodeContext.Server.Routes.signal,
            $PodeContext.Server.Routes.'*'
        ) 
       # the index avoids the access problems with .Keys and .Values.  This may be further burying the problem.
        $routes = @(foreach ($mr in $methodRoutes) {
                for ($i = 0; $i -lt $mr.Count; $i++) {
                    if ($mr[$i]) {
                        $mr[$i]
                    }
                }
            })
    }
    catch {
        throw "failed to get routes: $_"
    }
    
    # $routes = @()
    # foreach ($route in $PodeContext.Server.Routes.Values.Values) {
    #     $routes += $route
    # }

    # if we have a method, filter
    if (![string]::IsNullOrWhiteSpace($Method)) {
        $routes = @(foreach ($route in $routes) {
                if ($route.Method -ine $Method) {
                    continue
                }

                $route
            })
    }

    # if we have a path, filter
    if (![string]::IsNullOrWhiteSpace($Path)) {
        $Path = Split-PodeRouteQuery -Path $Path
        $Path = Update-PodeRouteSlash -Path $Path
        $Path = Resolve-PodePlaceholder -Path $Path

        $routes = @(foreach ($route in $routes) {
                if ($route.Path -ine $Path) {
                    continue
                }

                $route
            })
    }

    # further filter by endpoint names
    if (($null -ne $EndpointName) -and ($EndpointName.Length -gt 0)) {
        $routes = @(foreach ($name in $EndpointName) {
                foreach ($route in $routes) {
                    if ($route.Endpoint.Name -ine $name) {
                        continue
                    }

                    $route
                }
            })
    }

    # return
    return $routes
}

Also you said route order was important. This works on my local, but I have done VERY little testing. Enough to see that it starts and some buttons work. This is not ordered as I'm sure you know.
C:\git\awPwshDashboard\src\ps_modules\Pode\2.12.1\Private\Context.ps1

    # routes for pages and api
    $ctx.Server.Routes = [hashtable]::Synchronized(@{
            # common methods
            get     = [hashtable]::Synchronized(@{})
            post    = [hashtable]::Synchronized(@{})
            put     = [hashtable]::Synchronized(@{})
            patch   = [hashtable]::Synchronized(@{})
            delete  = [hashtable]::Synchronized(@{})
            # other methods
            connect = [hashtable]::Synchronized(@{})
            head    = [hashtable]::Synchronized(@{})
            merge   = [hashtable]::Synchronized(@{})
            options = [hashtable]::Synchronized(@{})
            trace   = [hashtable]::Synchronized(@{})
            static  = [hashtable]::Synchronized(@{})
            signal  = [hashtable]::Synchronized(@{})
            '*'     = [hashtable]::Synchronized(@{})
        })

Thanks again for taking a look!

[Badgerati]

Is there a call we can trigger to rebuild the routes when we get into this state (apart from a system restart)?

There isn't, you could use `Restart-PodeServer so it restarts the server for you automatically - but that's obviously not what you're after :)

Also you said route order was important

Yep, originally Pode did use plain hashtables for storing Routes as in most circumstances it is "ordered" - at least seemingly. It depends on what mood the hash code generator is in, and the order the keys happen to go into the hashtable. Some people had issues where routes weren't in the order Add-PodeRoute was being called, so it was changed to [ordered]@{}

Depending on your scenario, chances are using [hashtable]::Syncronized(@{}) will be enough for you.

Also, do you do professional support / contracting / consulting for Pode

I'm afraid not; between a full time job and family I just don't have the time, sorry! Closest you'll get it Pode's Discord server, there's myself and other people on there who will be happy to answer questions 😄