Repository navigation
[AI] LITE-34875: migrate off django-mongoengine to plain mongoengine - #13
Conversation
flake8 ~3.8 (and several of its plugins) predate Python 3.12 support and crash on entry-point loading (AttributeError: 'EntryPoints' object has no attribute 'get'). Bump flake8 and its plugins to versions that support 3.10-3.12, and drop flake8-future-import entirely: it imports the long-removed ast.Str (gone since Python 3.12) and its whole purpose -- flagging missing 'from __future__ import' statements -- is moot now that the floor is >=3.10. Verified flake8 and the full test suite pass under real Python 3.10, 3.11, and 3.12 containers, not just inferred from a partial matrix. Also corrects a mistake from the previous commit: reverts conftest.py from is_mock=True back to mongo_client_class=mongomock.MongoClient (SPEC.md's original recipe) -- is_mock appeared to work only because of a stale, incorrectly-installed mongoengine 0.26.0 in the local dev venv; against the actually-pinned 0.29.3, mongo_client_class is the only supported replacement for the removed mongomock:// URI.
|
All 3 build-and-test matrix legs (Python 3.10, 3.11, 3.12) now pass, including linting — see the pushed fix commit for the flake8/Python 3.12 incompatibility that caused the first failed run. The remaining red |
|
You are seeing this message because GitHub Code Scanning has recently been set up for this repository, or this pull request contains the workflow file for the Code Scanning tool. What Enabling Code Scanning Means:
For more information about GitHub Code Scanning, check out the documentation. |
…tion The sonar job's coverage.xml path-fix step rewrote absolute paths to a hardcoded /github/workspace/, a leftover assumption from when sonarcloud-github-action ran as a Docker container action mounting the workspace there. Both sonarcloud-github-action and its underlying sonarqube-scan-action@v5 are now composite actions that run natively on the runner (confirmed via their action.yml `using: composite`), so /github/workspace never exists and the Cobertura sensor silently drops coverage for every file, reporting 0% on new code and failing the Quality Gate. Rewrite dynamically from the actual pytest working directory ($(pwd)) to $GITHUB_WORKSPACE instead of a fixed guess -- a no-op today since pytest already runs from GITHUB_WORKSPACE, but correct if the scan action's execution model changes again.
|
Update on the Sonar job: my earlier comment attributing the red `Quality Analysis by sonar` job to a missing/invalid `SONAR_TOKEN` was wrong — once that got sorted out, the scan actually ran and surfaced a real Quality Gate failure: 0.0% Coverage on New Code (required ≥95%), despite this PR's own coverage being ~98%+ locally. Root cause: the job's "Fix coverage report for Sonar" step rewrites coverage.xml's absolute source paths to a hardcoded Fixed by rewriting from the actual pytest working directory (`$(pwd)`) to `$GITHUB_WORKSPACE` instead of a fixed guess — a no-op today (pytest already runs from `GITHUB_WORKSPACE`) but correct if the scan action's execution model changes again. All checks (build × 3, lint, SonarCloud, Quality Gate) are now green. |
qarlosh
left a comment
There was a problem hiding this comment.
[Should-fix] The README still describes the dependency this PR removes, and it is the PyPI page.
readme = "./README.md" in pyproject.toml. Line 16 says the library is a Django-Mongoengine specific implementation of RQL filtering. Line 22 lists Django Mongoengine as the companion project. Lines 88 and 96 say Python 3.8+, against the new floor of 3.10.
Suggested fix: point the prose at plain mongoengine and change both Python lines to 3.10+.
[Info] The Key decisions section of the description says the opposite of what the code does.
It says the branch used is_mock=True because mongo_client_class does not exist on mongoengine 0.29.3. tests/conftest.py uses mongo_client_class, and that is the correct one: I measured on 0.29.3 that register_connection(..., mongo_client_class=mongomock.MongoClient) works and is_mock=True raises Use of mongomock:// URI or 'is_mock' were removed in favor of 'mongo_client_class=mongomock.MongoClient'. Worth correcting before that paragraph is reused anywhere else.
…egression, missing tests, stale docs Must-fix: - .github/workflows/deploy.yml: quote python-version (was the bare number 3.10, which YAML/GitHub renders back as 3.1 -- no Linux 3.1 build exists, so the publish job would fail on the next tag push and nothing would release). Invisible in PR CI since this workflow only runs on tag push. Should-fix: - filter_cls.py: a dynamic filter backed by a plain Django field lost its null() lookup and empty() value, because _is_field_nullable/ _convert_value checked field.required (Django fields don't have that attribute, so it silently defaulted to True) instead of field.blank. Added _is_blank() to pick whichever attribute the field actually has, matching the reviewer's verified fix and restoring 0.2.2 behavior. - tests/test_filter_cls.py: added the two cases the review found uncovered -- a datetime literal against a DATE field (verified this test actually catches a regression: fails if constants.py's DateField/DateTimeField order is reverted, passes with the current order) and empty() on a required related string field (exercises the STRING+required=True branch of _convert_value). - README.md: still described django-mongoengine as the companion project and Python 3.8+ as the floor; this file is the PyPI page. Updated to plain mongoengine and Python 3.10+. Info-level, also addressed: - filter_cls.py: an unknown field name in FILTERS now raises FieldDoesNotExist (with the model name) instead of a bare KeyError, matching the pre-migration behavior. - .github/workflows/build.yml + pyproject.toml: CI never actually tested against Django 5 (poetry.lock pinned 4.2.2) despite that being this PR's whole point. Added a django-version axis (4.2/5.2) that overrides the locked Django after install, and added the Django 5.2 trove classifier (dropped 3.2, which doesn't support the 3.11/3.12 Python legs already in the matrix). Verified: full suite green (21/21) under both Django 4.2 and 5.2, flake8 clean, against a real MongoDB 5 container -- not just the default poetry-locked environment.
Sonar's new-code quality gate failed at 94.3% (required >=95%) after the review-fix commit: the _is_blank() fallback branch (a field with no .required attribute) and _get_model_field's new KeyError -> FieldDoesNotExist branch were both real new code with no test exercising them. Added direct unit tests for both -- no need to route through a full apply_filters() call, since both are plain static/classmethods.
|



What
Drops
django-mongoengineand depends on plainmongoenginedirectly:pyproject.toml/poetry.lock:mongoengine >=0.29,<0.30as a runtime dependency, Python floor narrowed to>=3.10,<4, CI matrices (build.yml,deploy.yml) updated to match.dj_mongoengine_rql/constants.py: imports frommongoengine;DateField/DateTimeFieldmapper entries reordered soDateFieldstill matches its own filter type instead of falling through toDateTimeField.dj_mongoengine_rql/filter_cls.py: field-support/nullability checks rewritten against plainmongoengine.base.BaseField(no more Django-flavored wrapper); added_get_model_field(mongoengine'sDocument._metahas no Django-styleget_field) and a gated_convert_valueoverride (handles the client-suppliedempty()value without relying on Django's.blankattribute).tests/documents.py,tests/settings.py,tests/conftest.py,tests/test_filter_cls.py): migrated fixtures, explicit connection bootstrap (replacing whatdjango_mongoengine'sAppConfig.ready()used to do implicitly), and new regression coverage (SelectFieldno-crash test,empty()/ne=empty()for every field type).Why
Unblocks the Connect platform's Django 4.2 → 5.2 upgrade:
django-mongoenginehard-pins Django<5, and its own newest release (even in beta) doesn't lift that pin. This library is a thin wrapper around plainmongoengine— every feature it added has a direct equivalent, so removing it is a mechanical substitution, not a redesign. Full technical spec:SPEC.md, attached to the linked Design Review (4 rounds of review, including empirical reproduction against the realaudit/customcodebases).This ticket originally shipped as 4 separate tickets/PRs (packaging,
constants.py,filter_cls.py, tests). Implementing the first one standalone surfaced that they can't actually ship independently — installingmongoengine-only and keeping the olddjango-mongoengine-based test suite importable are mutually exclusive, so any one merged alone would leavemaster's CI red until the rest landed. The plan was revised to collapse them into this single ticket/PR; the other three (LITE-34876/34877/34878) are closed as superseded.Key decisions
LongField's RQL filter type changes fromstringtoint— the one deliberate, accepted behavior change in this migration (no field in the downstream consumers usesLongFieldas a filter today; it's mostly a strict improvement:empty()/abcon aLongFieldused to crash with an uncaughtValueErrorescaping as a 500, now it's a clean 400). One real, worth-flagging side effect a reviewer found:like()on aLongFieldstops working, since it's no longer matched as a string filter. There's no changelog file in this repo to record that in, so noting it here.register_connection's mock kwarg:mongo_client_class=mongomock.MongoClientis the correct, working replacement for the removedmongomock://URI scheme on the pinned mongoengine0.29.3(confirmed directly: register + query round-trips against a mocked collection). An earlier version of this description said the opposite — thatmongo_client_classdidn't exist andis_mock=Truewas the working fallback. That was based on a stale, incorrectly-installed mongoengine0.26.0in a local dev venv; against the real pinned0.29.3,is_mock=TrueraisesUse of mongomock:// URI or 'is_mock' were removed in favor of 'mongo_client_class=mongomock.MongoClient'. The code (tests/conftest.py) was already corrected tomongo_client_classin a later commit on this branch; this paragraph just hadn't been updated to match until a reviewer caught the mismatch.How it worked before / after
No public API/behavior change for consumers, other than the
LongFieldfilter-type fix above. Every other RQL-filtered query — equality,in, ordering,null()/empty()lookups — produces identical results before and after, verified by the existing regression suite (updated only for theqs.query.q→qs._queryinternal API rename) plus the newempty()/SelectFieldcoverage.Test plan
docker-compose run app_test(flake8 + full pytest suite against a real Mongo container): 19/19 passed.poetry run pytestwith noMONGO_*env vars set (the actual publish-gating checkdeploy.ymlruns beforepoetry build/poetry publish): 19/19 passed.constants.pyandfilter_cls.pycovered; one pre-existing, untouched line infilter_cls.pyandq.pyremains uncovered from before this change (not part of this diff).Post-review update
qarlosh's review (formal Changes Requested) found real issues, all now fixed:.github/workflows/deploy.yml'spython-version: [3.10]was unquoted YAML, parsed as the float3.1—actions/setup-pythonhas no Linux 3.1 build, so the publish job would have failed on the next tag push and nothing would have released. Invisible in PR CI since this workflow only runs on tag push. Quoted it.null()lookup andempty()value (_is_field_nullable/_convert_valuecheckedfield.required, which Django fields don't have, silently defaulting toTrue). Fixed with a_is_blank()helper that checks whichever attribute the field actually has — applied the reviewer's own verified patch.DATEfield — verified this actually catches a regression by temporarily reverting theconstants.pyfield-type order and confirming the new test fails; andempty()on a required related string field, exercising theSTRING+required=Truebranch).django-mongoengineas the companion project and Python 3.8+ as the floor — this file is the PyPI page. Updated to plainmongoengineand Python 3.10+.FILTERSnow raisesFieldDoesNotExist(with the model name) instead of a bareKeyError, matching pre-migration behavior.poetry.lockpinned Django 4.2.2). Added adjango-versionaxis (4.2/5.2) tobuild.ymlthat overrides the locked Django after install, and added theFramework :: Django :: 5.2trove classifier (dropped3.2, which doesn't support the3.11/3.12Python legs already in the matrix).LongFieldbehavior change and the backwardsmongo_client_class/is_mockclaim above — both corrected in the "Key decisions" section.Re-verified after all fixes: full suite green (21/21) under both Django 4.2 and 5.2, flake8 clean, against a real MongoDB 5 container.