Skip to content

fix(Foundation): MD5Engine and MD4Engine hang on inputs over 4 GB - #5601

Open
r3wretrhy wants to merge 1 commit into
pocoproject:mainfrom
r3wretrhy:5597-md5-md4-large-input
Open

r3wretrhy wants to merge 1 commit into
pocoproject:mainfrom
r3wretrhy:5597-md5-md4-large-input

Conversation

@r3wretrhy

Copy link
Copy Markdown

Closes #5597

MD5Engine::updateImpl() and MD4Engine::updateImpl() take a std::size_t length but used unsigned int loop counters. With more than 4 GB in a single update() call, i + 63 < inputLen never becomes false and the call does not return. A second problem sits in the bit count: (UInt32) inputLen >> 29 casts before shifting, so the high word of the length lost everything above bit 31 of inputLen, and the digest was wrong even with wider counters.

  • Foundation/MD5Engine, MD4Engine: i, index and partLen are std::size_t; the high word is (UInt32) (inputLen >> 29).
  • Tests: testLargeInput in MD5EngineTest and MD4EngineTest (long tests, -long) hash 4 GB + 100 zero bytes from a read-only anonymous mmap, so no physical memory is used. Unix 64-bit only. Expected digests come from md5sum and openssl dgst -md4.

Tests: without the fix both new tests hang; with it they pass. Foundation-testrunner -all passes except DirectoryWatcherTest (inotify is not available in my container; unrelated). ASan+UBSan clean on the MD tests; clang-tidy and cppcheck show no new findings.

Follow-up: SHA1Engine::updateImpl() has the same (UInt32 ) count >> 29 pattern; I left it out to keep this change small.

AI disclosure: this change was prepared with an AI coding agent following AGENTS.md, and reviewed by me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

MD5Engine::updateImpl function hangs when input buffer size > 4GB

1 participant