[BUG]: Download package - Is timing out on different tasks

[teddy515]

We are running Self Hosted Agents on a windows server.
It worked fine last week, and now one "Download Package" or more is always timing out during our run.
https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/reference/download-package-v1?view=azure-pipelines

I have tried to update our build agents, and the problem still persists.
Agent v.3.244.1
Agent v.3.254.0

I have tried fetching the artifact from the build server, and it works everytime, and as the picture shows, there are some artifacts that can be downloaded, and some that cannot.
Next run, there could be others that works/not working.

Has anyone else experienced this problem?

Versions

Agent.Version: 4.254.0
Agent.OSVersion: 10.0.20348
Windows Server 2022 Standard

Agent.Version: 3.244.1
Agent.OSVersion: 10.0.20348
Windows Server 2022 Standard

Environment type (Please select at least one enviroment where you face this issue)

• Self-Hosted
• Microsoft Hosted
• VMSS Pool
• Container

Azure DevOps Server type

dev.azure.com (formerly visualstudio.com)

Azure DevOps Server Version (if applicable)

No response

Operation system

No response

Version controll system

No response

Relevant log output

2025-04-23T14:36:34.4326833Z ##[section]Starting: Download Package - Document Output OnPrem
2025-04-23T14:36:34.4334722Z ==============================================================================
2025-04-23T14:36:34.4334874Z Task         : Download package
2025-04-23T14:36:34.4335001Z Description  : Download a package from a package management feed in Azure Artifacts
2025-04-23T14:36:34.4335125Z Version      : 1.247.1
2025-04-23T14:36:34.4335246Z Author       : ms-vscs-rm
2025-04-23T14:36:34.4335318Z Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/download-package
2025-04-23T14:36:34.4335468Z ==============================================================================
2025-04-23T14:36:36.6507320Z Found tool in cache: artifacttool 0.2.387 x64
2025-04-23T14:36:41.7138152Z ##[error]Request timeout: /XXX/_apis/Packaging
2025-04-23T14:36:41.7907031Z ##[error]Unhandled: Request timeout: /XXX/_apis/Packaging
2025-04-23T14:36:41.7913021Z ##[section]Finishing: Download Package - Document Output OnPrem

[catadumitru]

I think it's a general service degradation issue for Azure DevOps services, we're getting these:

• Failed to connect to dev.azure.com port 443 after 21058 ms: Couldn't connect to server
• ##[error]Script failed with error: Error: connect ETIMEDOUT 13.107.42.20:443
  at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1611:16) {
  errno: -110,
  code: 'ETIMEDOUT',
  syscall: 'connect',
  address: '13.107.42.20',
  port: 443
  }

[ch0wm3in]

We are experiencing something similar with very slow selfhosted pipeline reporting back to dev.azure.com (upwards of 8 minutes to just report a github checkout completed across all selfhosted agents, the actual job completes fairly quickly when initiated but in steps in can take a looong time)

When looking at the job page it has all kinds of different warnings(these are everywhere on multiple agents..)

[ch0wm3in]

I have cross posted this to VS community for AZDO Post

[jwikman]

We also get different timeouts on our self-hosted agents when calling DevOps APIs or git endpoints.

fatal: unable to access 'https://dev.azure.com/[org]/[project]/_git/[repo]/': Failed to connect to dev.azure.com port 443 after 21059 ms: Could not connect to server
##[warning]Git fetch failed with exit code 128, back off 9.291 seconds before retry.

##[debug]SearchUrl: https://almsearch.dev.azure.com/[org]/_apis/search/workitemsearchresults?api-version=5.1-preview.1
Invoke-RestMethod : A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.

The "Create Work Item" task sometimes fails with this:

2025-04-23T09:41:28.4134811Z ##[section]Starting: Create User Story WI for delivery
2025-04-23T09:41:28.4144374Z ==============================================================================
2025-04-23T09:41:28.4144638Z Task         : Create Work Item
2025-04-23T09:41:28.4144771Z Description  : Creates a new work item from a build or release.
2025-04-23T09:41:28.4144973Z Version      : 2.3.3
2025-04-23T09:41:28.4145088Z Author       : Microsoft
2025-04-23T09:41:28.4145252Z Help         : [[Docs]](https://github.com/MicrosoftPremier/VstsExtensions/blob/master/CreateWorkItem/en-US/overview.md)
2025-04-23T09:41:28.4145541Z ==============================================================================
2025-04-23T09:41:50.2857934Z connect ETIMEDOUT 13.107.42.20:443
2025-04-23T09:41:50.2861280Z ##[error]connect ETIMEDOUT 13.107.42.20:443
2025-04-23T09:41:50.2961526Z ##[section]Finishing: Create User Story WI for delivery

The VSCode Marketplace API fails with timeouts now and then: https://marketplace.visualstudio.com/_apis/public/gallery/extensionquery?api-version=3.0-preview.1

It is really intermittent, and some pipelines succeeds, but some fails.
It started yesterday morning.

Agents running in Sweden, if that matters...

[ch0wm3in]

Hmm we are running agents from the Nordics as well primarily Denmark if that matters.

[teddy515]

Our agents are also running from servers hosted in Denmark.

[DunkRing]

Our agents is also impacted, based in Denmark.

Failed to publish telemetry data. Error The HTTP request timed out after 00:01:40.
UsePythonVersion

Failed to publish telemetry data. Error The HTTP request timed out after 00:01:40.
Login to Azure and Set Environment Variables

[mthoger]

Our agents is also impacted, based in Denmark.
Timeouts all over the place.

maybe its time to update the devops status website, with the actual service health status.
this have been near useless since yesterday.

[jwikman]

Please, everyone having this issue, go to the issue registered on the Developer Community site and vote to get higher visibility.

[kfuglsang]

Same issue here. Also on self-hosted agents in Denmark.

[jeppetn]

same here experiencing several timeout issues, also denmark

[mmrazik]

We are experiencing something similar with very slow selfhosted pipeline reporting back to dev.azure.com (upwards of 8 minutes to just report a github checkout completed across all selfhosted agents, the actual job completes fairly quickly when initiated but in steps in can take a looong time)

When looking at the job page it has all kinds of different warnings(these are everywhere on multiple agents..)

@ch0wm3in are you saying that it is both git checkout from github as well as the ADO endpoints (for pushing the telemetry) that are timing out? Or is the "git fetch failed" in your screenshot from fetching from ADO?