Skip to content

[AI] LITE-34875: migrate off django-mongoengine to plain mongoengine - #13

Merged
jonatrios merged 5 commits into
masterfrom
cr/LITE-34875
Sep 17, 2026
Merged

jonatrios merged 5 commits into
masterfrom
cr/LITE-34875

Conversation

@jonatrios

@jonatrios jonatrios commented Aug 18, 2026 •

Copy link
Copy Markdown
Contributor

This PR was generated by Claude via the design-review-implementer workflow, implementing one task from the Design Review plan.

What

Drops django-mongoengine and depends on plain mongoengine directly:

  • pyproject.toml/poetry.lock: mongoengine >=0.29,<0.30 as a runtime dependency, Python floor narrowed to >=3.10,<4, CI matrices (build.yml, deploy.yml) updated to match.
  • dj_mongoengine_rql/constants.py: imports from mongoengine; DateField/DateTimeField mapper entries reordered so DateField still matches its own filter type instead of falling through to DateTimeField.
  • dj_mongoengine_rql/filter_cls.py: field-support/nullability checks rewritten against plain mongoengine.base.BaseField (no more Django-flavored wrapper); added _get_model_field (mongoengine's Document._meta has no Django-style get_field) and a gated _convert_value override (handles the client-supplied empty() value without relying on Django's .blank attribute).
  • Test suite (tests/documents.py, tests/settings.py, tests/conftest.py, tests/test_filter_cls.py): migrated fixtures, explicit connection bootstrap (replacing what django_mongoengine's AppConfig.ready() used to do implicitly), and new regression coverage (SelectField no-crash test, empty()/ne=empty() for every field type).

Why

Unblocks the Connect platform's Django 4.2 → 5.2 upgrade: django-mongoengine hard-pins Django <5, and its own newest release (even in beta) doesn't lift that pin. This library is a thin wrapper around plain mongoengine — every feature it added has a direct equivalent, so removing it is a mechanical substitution, not a redesign. Full technical spec: SPEC.md, attached to the linked Design Review (4 rounds of review, including empirical reproduction against the real audit/custom codebases).

This ticket originally shipped as 4 separate tickets/PRs (packaging, constants.py, filter_cls.py, tests). Implementing the first one standalone surfaced that they can't actually ship independently — installing mongoengine-only and keeping the old django-mongoengine-based test suite importable are mutually exclusive, so any one merged alone would leave master's CI red until the rest landed. The plan was revised to collapse them into this single ticket/PR; the other three (LITE-34876/34877/34878) are closed as superseded.

Key decisions

  • LongField's RQL filter type changes from string to int — the one deliberate, accepted behavior change in this migration (no field in the downstream consumers uses LongField as a filter today; it's mostly a strict improvement: empty()/abc on a LongField used to crash with an uncaught ValueError escaping as a 500, now it's a clean 400). One real, worth-flagging side effect a reviewer found: like() on a LongField stops working, since it's no longer matched as a string filter. There's no changelog file in this repo to record that in, so noting it here.
  • register_connection's mock kwarg: mongo_client_class=mongomock.MongoClient is the correct, working replacement for the removed mongomock:// URI scheme on the pinned mongoengine 0.29.3 (confirmed directly: register + query round-trips against a mocked collection). An earlier version of this description said the opposite — that mongo_client_class didn't exist and is_mock=True was the working fallback. That was based on a stale, incorrectly-installed mongoengine 0.26.0 in a local dev venv; against the real pinned 0.29.3, is_mock=True raises Use of mongomock:// URI or 'is_mock' were removed in favor of 'mongo_client_class=mongomock.MongoClient'. The code (tests/conftest.py) was already corrected to mongo_client_class in a later commit on this branch; this paragraph just hadn't been updated to match until a reviewer caught the mismatch.

How it worked before / after

No public API/behavior change for consumers, other than the LongField filter-type fix above. Every other RQL-filtered query — equality, in, ordering, null()/empty() lookups — produces identical results before and after, verified by the existing regression suite (updated only for the qs.query.q → qs._query internal API rename) plus the new empty()/SelectField coverage.

Test plan

  • docker-compose run app_test (flake8 + full pytest suite against a real Mongo container): 19/19 passed.
  • poetry run pytest with no MONGO_* env vars set (the actual publish-gating check deploy.yml runs before poetry build/poetry publish): 19/19 passed.
  • Coverage: all new/changed lines in constants.py and filter_cls.py covered; one pre-existing, untouched line in filter_cls.py and q.py remains uncovered from before this change (not part of this diff).

Post-review update

qarlosh's review (formal Changes Requested) found real issues, all now fixed:

  • Must-fix: .github/workflows/deploy.yml's python-version: [3.10] was unquoted YAML, parsed as the float 3.1 — actions/setup-python has no Linux 3.1 build, so the publish job would have failed on the next tag push and nothing would have released. Invisible in PR CI since this workflow only runs on tag push. Quoted it.
  • Should-fix: a dynamic filter backed by a plain Django field lost its null() lookup and empty() value (_is_field_nullable/_convert_value checked field.required, which Django fields don't have, silently defaulting to True). Fixed with a _is_blank() helper that checks whichever attribute the field actually has — applied the reviewer's own verified patch.
  • Should-fix: added the two test cases the review found uncovered (a datetime literal against a DATE field — verified this actually catches a regression by temporarily reverting the constants.py field-type order and confirming the new test fails; and empty() on a required related string field, exercising the STRING+required=True branch).
  • Should-fix: README described django-mongoengine as the companion project and Python 3.8+ as the floor — this file is the PyPI page. Updated to plain mongoengine and Python 3.10+.
  • Info: an unknown field name in FILTERS now raises FieldDoesNotExist (with the model name) instead of a bare KeyError, matching pre-migration behavior.
  • Info: CI never actually tested against Django 5 despite that being this PR's whole point (poetry.lock pinned Django 4.2.2). Added a django-version axis (4.2/5.2) to build.yml that overrides the locked Django after install, and added the Framework :: Django :: 5.2 trove classifier (dropped 3.2, which doesn't support the 3.11/3.12 Python legs already in the matrix).
  • Info: the LongField behavior change and the backwards mongo_client_class/is_mock claim above — both corrected in the "Key decisions" section.

Re-verified after all fixes: full suite green (21/21) under both Django 4.2 and 5.2, flake8 clean, against a real MongoDB 5 container.

flake8 ~3.8 (and several of its plugins) predate Python 3.12 support
and crash on entry-point loading (AttributeError: 'EntryPoints' object
has no attribute 'get'). Bump flake8 and its plugins to versions that
support 3.10-3.12, and drop flake8-future-import entirely: it imports
the long-removed ast.Str (gone since Python 3.12) and its whole
purpose -- flagging missing 'from __future__ import' statements -- is
moot now that the floor is >=3.10. Verified flake8 and the full test
suite pass under real Python 3.10, 3.11, and 3.12 containers, not just
inferred from a partial matrix.

Also corrects a mistake from the previous commit: reverts conftest.py
from is_mock=True back to mongo_client_class=mongomock.MongoClient
(SPEC.md's original recipe) -- is_mock appeared to work only because
of a stale, incorrectly-installed mongoengine 0.26.0 in the local dev
venv; against the actually-pinned 0.29.3, mongo_client_class is the
only supported replacement for the removed mongomock:// URI.
@jonatrios

Copy link
Copy Markdown
Contributor Author

All 3 build-and-test matrix legs (Python 3.10, 3.11, 3.12) now pass, including linting — see the pushed fix commit for the flake8/Python 3.12 incompatibility that caused the first failed run.

The remaining red Quality Analysis by sonar job is unrelated to this PR: it fails at the SonarScanner step with Failed to query JRE metadata... check sonar.token or SONAR_TOKEN, which is a missing/invalid SONAR_TOKEN repo secret. Confirmed pre-existing — the same job fails identically on this repo's last PR from October 2025, unrelated to any code change. Needs a repo admin to check/rotate the SONAR_TOKEN secret; not something fixable from this PR's diff.

@github-advanced-security

Copy link
Copy Markdown

You are seeing this message because GitHub Code Scanning has recently been set up for this repository, or this pull request contains the workflow file for the Code Scanning tool.

What Enabling Code Scanning Means:

  • The 'Security' tab will display more code scanning analysis results (e.g., for the default branch).
  • Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results.
  • You will be able to see the analysis results for the pull request's branch on this overview once the scans have completed and the checks have passed.

For more information about GitHub Code Scanning, check out the documentation.

…tion

The sonar job's coverage.xml path-fix step rewrote absolute paths to a
hardcoded /github/workspace/, a leftover assumption from when
sonarcloud-github-action ran as a Docker container action mounting the
workspace there. Both sonarcloud-github-action and its underlying
sonarqube-scan-action@v5 are now composite actions that run natively on
the runner (confirmed via their action.yml `using: composite`), so
/github/workspace never exists and the Cobertura sensor silently drops
coverage for every file, reporting 0% on new code and failing the
Quality Gate. Rewrite dynamically from the actual pytest working
directory ($(pwd)) to $GITHUB_WORKSPACE instead of a fixed guess -- a
no-op today since pytest already runs from GITHUB_WORKSPACE, but
correct if the scan action's execution model changes again.
@jonatrios

Copy link
Copy Markdown
Contributor Author

Update on the Sonar job: my earlier comment attributing the red `Quality Analysis by sonar` job to a missing/invalid `SONAR_TOKEN` was wrong — once that got sorted out, the scan actually ran and surfaced a real Quality Gate failure: 0.0% Coverage on New Code (required ≥95%), despite this PR's own coverage being ~98%+ locally.

Root cause: the job's "Fix coverage report for Sonar" step rewrites coverage.xml's absolute source paths to a hardcoded /github/workspace/..., which was the correct target back when sonarcloud-github-action ran as a Docker container action mounting the workspace there. Both sonarcloud-github-action and the sonarqube-scan-action@v5 it now delegates to are composite actions (confirmed via their action.yml) — they run natively on the runner, so /github/workspace never exists. The Cobertura sensor couldn't resolve any of the 4 source files against that path and silently dropped all coverage measures, reporting 0%.

Fixed by rewriting from the actual pytest working directory (`$(pwd)`) to `$GITHUB_WORKSPACE` instead of a fixed guess — a no-op today (pytest already runs from `GITHUB_WORKSPACE`) but correct if the scan action's execution model changes again. All checks (build × 3, lint, SonarCloud, Quality Gate) are now green.

@qarlosh qarlosh left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Should-fix] The README still describes the dependency this PR removes, and it is the PyPI page.

readme = "./README.md" in pyproject.toml. Line 16 says the library is a Django-Mongoengine specific implementation of RQL filtering. Line 22 lists Django Mongoengine as the companion project. Lines 88 and 96 say Python 3.8+, against the new floor of 3.10.

Suggested fix: point the prose at plain mongoengine and change both Python lines to 3.10+.


[Info] The Key decisions section of the description says the opposite of what the code does.

It says the branch used is_mock=True because mongo_client_class does not exist on mongoengine 0.29.3. tests/conftest.py uses mongo_client_class, and that is the correct one: I measured on 0.29.3 that register_connection(..., mongo_client_class=mongomock.MongoClient) works and is_mock=True raises Use of mongomock:// URI or 'is_mock' were removed in favor of 'mongo_client_class=mongomock.MongoClient'. Worth correcting before that paragraph is reused anywhere else.

Comment thread .github/workflows/deploy.yml Outdated
Comment thread dj_mongoengine_rql/filter_cls.py Outdated
Comment thread tests/test_filter_cls.py
Comment thread dj_mongoengine_rql/constants.py
Comment thread .github/workflows/build.yml
Comment thread dj_mongoengine_rql/filter_cls.py Outdated
Comment thread pyproject.toml
…egression, missing tests, stale docs

Must-fix:
- .github/workflows/deploy.yml: quote python-version (was the bare
  number 3.10, which YAML/GitHub renders back as 3.1 -- no Linux
  3.1 build exists, so the publish job would fail on the next tag
  push and nothing would release). Invisible in PR CI since this
  workflow only runs on tag push.

Should-fix:
- filter_cls.py: a dynamic filter backed by a plain Django field lost
  its null() lookup and empty() value, because _is_field_nullable/
  _convert_value checked field.required (Django fields don't have
  that attribute, so it silently defaulted to True) instead of
  field.blank. Added _is_blank() to pick whichever attribute the
  field actually has, matching the reviewer's verified fix and
  restoring 0.2.2 behavior.
- tests/test_filter_cls.py: added the two cases the review found
  uncovered -- a datetime literal against a DATE field (verified this
  test actually catches a regression: fails if constants.py's
  DateField/DateTimeField order is reverted, passes with the current
  order) and empty() on a required related string field (exercises
  the STRING+required=True branch of _convert_value).
- README.md: still described django-mongoengine as the companion
  project and Python 3.8+ as the floor; this file is the PyPI page.
  Updated to plain mongoengine and Python 3.10+.

Info-level, also addressed:
- filter_cls.py: an unknown field name in FILTERS now raises
  FieldDoesNotExist (with the model name) instead of a bare KeyError,
  matching the pre-migration behavior.
- .github/workflows/build.yml + pyproject.toml: CI never actually
  tested against Django 5 (poetry.lock pinned 4.2.2) despite that
  being this PR's whole point. Added a django-version axis
  (4.2/5.2) that overrides the locked Django after install, and
  added the Django 5.2 trove classifier (dropped 3.2, which doesn't
  support the 3.11/3.12 Python legs already in the matrix).

Verified: full suite green (21/21) under both Django 4.2 and 5.2,
flake8 clean, against a real MongoDB 5 container -- not just the
default poetry-locked environment.
Sonar's new-code quality gate failed at 94.3% (required >=95%) after
the review-fix commit: the _is_blank() fallback branch (a field with
no .required attribute) and _get_model_field's new KeyError ->
FieldDoesNotExist branch were both real new code with no test
exercising them.

Added direct unit tests for both -- no need to route through a full
apply_filters() call, since both are plain static/classmethods.
@sonarqubecloud

Copy link
Copy Markdown

@jonatrios
jonatrios requested a review from qarlosh September 17, 2026 08:33
@jonatrios
jonatrios merged commit b09e7d7 into master Sep 17, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants