Don't silently corrupt large integers when exporting to XLS/XLSX - #662
Don't silently corrupt large integers when exporting to XLS/XLSX#662agu2347 wants to merge 1 commit into
Conversation
Both the legacy XLS format (via xlwt) and XLSX (via openpyxl) store numeric cell values internally as IEEE-754 doubles, which can only represent integers exactly up to 2**53 in magnitude. Writing a larger integer as a numeric cell silently rounds it to the nearest representable double, e.g. 123456789012345677 becomes 123456789012345680 on read-back -- a wrong value with no error or warning, as reported in the issue (which showed the same class of corruption with a different example integer). This is a fundamental limitation of both spreadsheet formats' numeric storage, not something fixable while keeping the cell as a spreadsheet 'number' -- it affects any writer for these formats. The reasonable, uncontroversial fix is to detect when an integer's magnitude exceeds what a double can represent exactly and, only in that case, write it as its exact decimal string instead of silently writing a wrong number. Integers within the safe range are completely unaffected and remain numeric cells, exactly as before. Added a shared spreadsheet_safe_value() helper in tablib.utils (bool is explicitly excluded, since it's an int subclass but should keep its normal boolean handling), and applied it at the one cell-writing call site in each of _xls.py and _xlsx.py. Added a regression test for each format that reproduces the exact corruption with a large integer (a value chosen to *not* happen to be exactly representable as a double, unlike the original issue's example which turned out to have enough trailing zero bits to round-trip correctly on its own -- I verified this by computing its required mantissa bits before choosing a genuinely-unsafe test value), and confirms normal small integers are untouched. Confirmed both new tests fail with the original code, reproducing the exact reported-style corruption, and pass with the fix. Ran the full existing test suite: 182 passed (180 baseline + 2 new), no regressions. Fixes jazzband#197
MohammedAlkindi
left a comment
There was a problem hiding this comment.
This works. Flagging first that #650 fixes the same issue the same way and has been open since 2026-07-04, so whoever reviews this should probably decide between them rather than read them separately. Both say "Fixes #197" and both implement @claudep's 2019 suggestion on that issue, "detect such values and force them to str".
I ran both against master on Windows 11, Python 3.13.13, openpyxl 3.1.5, reading every value back out of the exported file.
master #650 #662
2**53 exactly 9007199254740992 ok ok ok
2**53 + 1 9007199254740992 CORRUPT ok ok
568883383628111872 5.688833836281119e+17 CORRUPT ok ok
negative of that 5.688833836281119e+17 CORRUPT ok ok
small int 12345 ok ok ok
xls body (xlwt path) 5.688833836281119e+17 CORRUPT ok ok
bool stays bool True ok ok ok
I could not construct a case where the two branches differ. Same boundary (exclusive above 2**53, so 2**53 itself stays numeric), same bool exclusion, same behaviour on the XLS path.
Test counts, each against its own merge base, because both branches are behind master and the raw numbers are misleading otherwise:
#650 base 173 passed -> branch 175 passed (+2, 12 commits behind master)
#662 base 179 passed -> branch 181 passed (+2, 5 commits behind master)
Neither regresses anything. The same single pre-existing failure, UtilsGetDateTestCase.test_getDate_datetime_timestamp, is present on master and on both branches.
So the choice is style, not correctness:
- #650 is 20 lines and inlines the guard in
_xls.pyand_xlsx.py. Smallest possible change, duplicated in two places. - #662 is 74 lines and extracts
spreadsheet_safe_value()intotablib/utils.pywith a namedMAX_EXACT_INT_IN_DOUBLEand a docstring saying whyboolis excluded. No duplication, and a natural home if another format ever needs the same guard.
One small placement difference, which did not change any result I could measure but is worth a glance. In _xls.py, #662 calls the helper inside the final else: branch, after the date and time styling branches, while #650 converts at the top of the cell loop before any branch. Integers do not reach the date branches, so they agree today; #650's placement is the one that stays correct if a future branch is added above.
Not a maintainer, just reporting what ran here.
Fixes #197.
Both the legacy XLS format (via
xlwt) and XLSX (viaopenpyxl) store numeric cell values internally as IEEE-754 doubles, which can only represent integers exactly up to2**53in magnitude. Writing a larger integer as a numeric cell silently rounds it to the nearest representable double -- e.g.123456789012345677becomes123456789012345680on read-back -- a wrong value with no error or warning, as reported in the issue (which showed the same class of corruption with a different example integer).This is a fundamental limitation of both spreadsheet formats' numeric storage, not something fixable while keeping the cell as a spreadsheet "number" -- it affects any writer for these formats (Excel itself included). The reasonable, uncontroversial fix is to detect when an integer's magnitude exceeds what a double can represent exactly and, only in that case, write it as its exact decimal string instead of silently writing a wrong number. Integers within the safe range are completely unaffected and remain numeric cells, exactly as before.
Implementation: added a shared
spreadsheet_safe_value()helper intablib.utils(boolis explicitly excluded, since it's anintsubclass but should keep its normal boolean handling), and applied it at the one cell-writing call site in each of_xls.pyand_xlsx.py.Testing: added a regression test for each format that reproduces the exact corruption with a large integer. I picked a value that's genuinely not exactly representable as a double -- the original issue's example number turned out to have enough trailing zero bits in binary to round-trip correctly on its own, which I discovered by computing its required mantissa bits, so I used a different value that reliably demonstrates the general bug class. Both new tests fail with the original code (reproducing the exact reported-style corruption) and pass with the fix. Also verified small, normal-range integers are completely untouched (still numeric cells, not converted to strings).
Ran the full existing test suite: 182 passed (180 baseline + 2 new), no regressions.