[BUG]: Update fails with "Checksum validation failed"

[cremor]

What happened?

I'm trying to update my self-hosted agent. I'm using the "Update all agents" button in the agent pool web UI on Azure DevOps for this. The process hangs at "Downloading 4.258.1 agent".

The agent logs show a checksum problem.

Versions

Agent version 4.255.0
Windows Server 2019 1809

Environment type (Please select at least one enviroment where you face this issue)

• Self-Hosted
• Microsoft Hosted
• VMSS Pool
• Container

Azure DevOps Server type

dev.azure.com (formerly visualstudio.com)

Azure DevOps Server Version (if applicable)

No response

Operation system

No response

Version controll system

No response

Relevant log output

[2025-07-07 06:06:30Z INFO Agent] Refresh message received, kick-off selfupdate background process.
[2025-07-07 06:06:30Z INFO SelfUpdater] Version '4.258.1' of 'agent' package available in server.
[2025-07-07 06:06:30Z INFO SelfUpdater] Current running agent version is 4.255.0
[2025-07-07 06:06:30Z INFO SelfUpdater] An update is available.
[2025-07-07 06:06:30Z INFO Terminal] WRITE LINE: Agent update in progress, do not shutdown agent.
[2025-07-07 06:06:30Z INFO MessageListener] Sent GetAgentMessage to keep alive agent 23, session '09c0547e-028c-4131-99ba-eed9aabeb223'.
[2025-07-07 06:06:30Z INFO Terminal] WRITE LINE: Downloading 4.258.1 agent
[2025-07-07 06:06:31Z INFO HostContext] Well known directory 'Bin': 'C:\agent\bin'
[2025-07-07 06:06:31Z INFO HostContext] Well known directory 'Root': 'C:\agent'
[2025-07-07 06:06:31Z INFO HostContext] Well known directory 'Work': 'D:\agent\_work'
[2025-07-07 06:06:31Z INFO SelfUpdater] Attempt 1: save latest agent into D:\agent\_work\_update\agent1.zip.
[2025-07-07 06:06:31Z INFO SelfUpdater] Download agent: begin download
[2025-07-07 06:06:32Z INFO SelfUpdater] Download agent: finished download
[2025-07-07 06:06:34Z WARN SelfUpdater] Checksum validation failed
  Expected hash: '4f854d13f57af8a7ddb2b034945fc84751fc77281ab18beae12b40e8f76974e9'
  Actual hash: 'CF-9C-FC-3D-C7-A1-08-6B-72-0D-CF-37-0B-1C-B0-87-6C-BB-2B-A6-2B-75-ED-EB-DF-6A-17-94-57-B8-4E-EA'
[2025-07-07 06:06:34Z INFO SelfUpdater] Attempt 2: save latest agent into D:\agent\_work\_update\agent1.zip.
[2025-07-07 06:06:34Z INFO SelfUpdater] Download agent: begin download
[2025-07-07 06:06:35Z INFO SelfUpdater] Download agent: finished download
[2025-07-07 06:06:36Z WARN SelfUpdater] Checksum validation failed
  Expected hash: '4f854d13f57af8a7ddb2b034945fc84751fc77281ab18beae12b40e8f76974e9'
  Actual hash: 'CF-9C-FC-3D-C7-A1-08-6B-72-0D-CF-37-0B-1C-B0-87-6C-BB-2B-A6-2B-75-ED-EB-DF-6A-17-94-57-B8-4E-EA'
[2025-07-07 06:06:36Z INFO SelfUpdater] Attempt 3: save latest agent into D:\agent\_work\_update\agent1.zip.
[2025-07-07 06:06:36Z INFO SelfUpdater] Download agent: begin download
[2025-07-07 06:06:37Z INFO SelfUpdater] Download agent: finished download
[2025-07-07 06:06:39Z WARN SelfUpdater] Checksum validation failed
  Expected hash: '4f854d13f57af8a7ddb2b034945fc84751fc77281ab18beae12b40e8f76974e9'
  Actual hash: 'CF-9C-FC-3D-C7-A1-08-6B-72-0D-CF-37-0B-1C-B0-87-6C-BB-2B-A6-2B-75-ED-EB-DF-6A-17-94-57-B8-4E-EA'
[2025-07-07 06:06:39Z INFO Agent] Ignore agent update exception. System.Threading.Tasks.TaskCanceledException: Agent package checksum validation failed.
There are possible reasons why this happened:
  1) The agent package was compromised.
  2) The agent package was not fully downloaded or was corrupted during the download process.
You can skip checksum validation for the agent package by setting the environment variable DISABLE_HASH_VALIDATION=true
   at Microsoft.VisualStudio.Services.Agent.Listener.SelfUpdater.DownloadLatestAgent(CancellationToken token) in D:\a\_work\1\s\src\Agent.Listener\SelfUpdater.cs:line 340
   at Microsoft.VisualStudio.Services.Agent.Listener.SelfUpdater.SelfUpdate(AgentRefreshMessage updateMessage, IJobDispatcher jobDispatcher, Boolean restartInteractiveAgent, CancellationToken token) in D:\a\_work\1\s\src\Agent.Listener\SelfUpdater.cs:line 82
   at Microsoft.VisualStudio.Services.Agent.Listener.Agent.RunAsync(AgentSettings settings, Boolean runOnce) in D:\a\_work\1\s\src\Agent.Listener\Agent.cs:line 424
[2025-07-07 06:06:39Z INFO Agent] Auto update task finished at backend, there is no available agent update needs to apply, continue message queue looping.

[sanjuyadav24]

Hi, thank you for bringing this to our attention. Could you please share the size of the downloaded file?

[cremor]

It seems like the download is automatically deleted when the hash mismatch is detected. The folder D:\agent\_work\_update\ was empty at the time when I copied the log.

[HeikoSCE]

We have the same behaviour with
agent 4.252
on Windows Server 2022
on 6 Azure hosted VMs
each with a self hosted agent:

[2025-07-15 15:46:49Z INFO Agent] Refresh message received, kick-off selfupdate background process.
[2025-07-15 15:46:49Z INFO SelfUpdater] Version '4.258.1' of 'agent' package available in server.
[2025-07-15 15:46:49Z INFO SelfUpdater] Current running agent version is 4.252.0
[2025-07-15 15:46:49Z INFO SelfUpdater] An update is available.
[2025-07-15 15:46:49Z INFO Terminal] WRITE LINE: Agent update in progress, do not shutdown agent.
[2025-07-15 15:46:49Z INFO Terminal] WRITE LINE: Downloading 4.258.1 agent
[2025-07-15 15:46:49Z INFO HostContext] Well known directory 'Bin': 'C:\agent\bin'
[2025-07-15 15:46:49Z INFO HostContext] Well known directory 'Root': 'C:\agent'
[2025-07-15 15:46:49Z INFO HostContext] Well known directory 'Work': 'C:\agent\_work'
[2025-07-15 15:46:49Z INFO SelfUpdater] Attempt 1: save latest agent into C:\agent\_work\_update\agent1.zip.
[2025-07-15 15:46:49Z INFO SelfUpdater] Download agent: begin download
[2025-07-15 15:46:50Z INFO MessageListener] Sent GetAgentMessage to keep alive agent 69, session 'fa180df6-df4e-4b9c-9324-bab2f6854ff6'.
[2025-07-15 15:46:50Z INFO SelfUpdater] Download agent: finished download
[2025-07-15 15:46:50Z WARN SelfUpdater] Checksum validation failed
  Expected hash: '4f854d13f57af8a7ddb2b034945fc84751fc77281ab18beae12b40e8f76974e9'
  Actual hash: 'CF-9C-FC-3D-C7-A1-08-6B-72-0D-CF-37-0B-1C-B0-87-6C-BB-2B-A6-2B-75-ED-EB-DF-6A-17-94-57-B8-4E-EA'
[2025-07-15 15:46:50Z INFO SelfUpdater] Attempt 2: save latest agent into C:\agent\_work\_update\agent1.zip.
[2025-07-15 15:46:50Z INFO SelfUpdater] Download agent: begin download
[2025-07-15 15:46:51Z INFO SelfUpdater] Download agent: finished download
[2025-07-15 15:46:51Z WARN SelfUpdater] Checksum validation failed
  Expected hash: '4f854d13f57af8a7ddb2b034945fc84751fc77281ab18beae12b40e8f76974e9'
  Actual hash: 'CF-9C-FC-3D-C7-A1-08-6B-72-0D-CF-37-0B-1C-B0-87-6C-BB-2B-A6-2B-75-ED-EB-DF-6A-17-94-57-B8-4E-EA'
[2025-07-15 15:46:51Z INFO SelfUpdater] Attempt 3: save latest agent into C:\agent\_work\_update\agent1.zip.
[2025-07-15 15:46:51Z INFO SelfUpdater] Download agent: begin download
[2025-07-15 15:46:52Z INFO SelfUpdater] Download agent: finished download
[2025-07-15 15:46:52Z WARN SelfUpdater] Checksum validation failed
  Expected hash: '4f854d13f57af8a7ddb2b034945fc84751fc77281ab18beae12b40e8f76974e9'
  Actual hash: 'CF-9C-FC-3D-C7-A1-08-6B-72-0D-CF-37-0B-1C-B0-87-6C-BB-2B-A6-2B-75-ED-EB-DF-6A-17-94-57-B8-4E-EA'
[2025-07-15 15:46:52Z INFO Agent] Ignore agent update exception. System.Threading.Tasks.TaskCanceledException: Agent package checksum validation failed.
There are possible reasons why this happened:
  1) The agent package was compromised.
  2) The agent package was not fully downloaded or was corrupted during the download process.
You can skip checksum validation for the agent package by setting the environment variable DISABLE_HASH_VALIDATION=true
   at Microsoft.VisualStudio.Services.Agent.Listener.SelfUpdater.DownloadLatestAgent(CancellationToken token) in D:\a\_work\1\s\src\Agent.Listener\SelfUpdater.cs:line 340
   at Microsoft.VisualStudio.Services.Agent.Listener.SelfUpdater.SelfUpdate(AgentRefreshMessage updateMessage, IJobDispatcher jobDispatcher, Boolean restartInteractiveAgent, CancellationToken token) in D:\a\_work\1\s\src\Agent.Listener\SelfUpdater.cs:line 82
   at Microsoft.VisualStudio.Services.Agent.Listener.Agent.RunAsync(AgentSettings settings, Boolean runOnce) in D:\a\_work\1\s\src\Agent.Listener\Agent.cs:line 424
[2025-07-15 15:46:52Z INFO Agent] Auto update task finished at backend, there is no available agent update needs to apply, continue message queue looping.

I find remarkable that the actual hash is always (all 6 machines + the one of the original post, all trials) the same. So it seems not to be an incomplete download.

And the second point: A repetition of the update after a period of 30 Minutes to 24h worked fine. So it seems to be a timing related issue. Maybe Microsofts Defender is blocking access to the file?

[njkoopman]

I've got the same issue:

[2025-07-24 06:14:34Z WARN SelfUpdater] Checksum validation failed
Expected hash: '4f854d13f57af8a7ddb2b034945fc84751fc77281ab18beae12b40e8f76974e9'
Actual hash: 'CF-9C-FC-3D-C7-A1-08-6B-72-0D-CF-37-0B-1C-B0-87-6C-BB-2B-A6-2B-75-ED-EB-DF-6A-17-94-57-B8-4E-EA'

[sanjuyadav24]

Hi
Thanks for the reporting!

The Azure Pipelines Agent checksum validation failure (Issue #5262) affecting updates to version 4.258.1 has been resolved.