Hydration error with oneToMany relation indexed by id

[danfraticiu]

Bug Report

Q	A
Version	3.52
Previous Version if the bug is a regression	-

Summary

I have this recurring intermittent hydration issue since I've upgraded doctrine/orm from 2 to 3 (starting with 3.3.3, currently on 3.5.2). ObjectHydrator.php:502 throws ErrorException: Warning: Undefined array key "id_11"

How to reproduce

Unable to reproduce, error only happens occasionally, inside the messenger handlers on the async queue (Doctrine transport if that matters), the error does not trigger if the worker restarts and retries the message. Also I was not able to reproduce it locally.

Stack trace:

ErrorException: Warning: Undefined array key "id_10"
#0 /vendor/doctrine/orm/src/Internal/Hydration/ObjectHydrator.php(502): Doctrine\ORM\Internal\Hydration\ObjectHydrator::hydrateRowData
#1 /vendor/doctrine/orm/src/Internal/Hydration/ObjectHydrator.php(147): Doctrine\ORM\Internal\Hydration\ObjectHydrator::hydrateAllData
#2 /vendor/doctrine/orm/src/Internal/Hydration/AbstractHydrator.php(171): Doctrine\ORM\Internal\Hydration\AbstractHydrator::hydrateAll
#3 /vendor/doctrine/orm/src/Persisters/Entity/BasicEntityPersister.php(1005): Doctrine\ORM\Persisters\Entity\BasicEntityPersister::loadCollectionFromStatement
#4 /vendor/doctrine/orm/src/Persisters/Entity/BasicEntityPersister.php(1792): Doctrine\ORM\Persisters\Entity\BasicEntityPersister::loadOneToManyCollection
#5 /vendor/doctrine/orm/src/UnitOfWork.php(2722): Doctrine\ORM\UnitOfWork::loadCollection
#6 /vendor/doctrine/orm/src/PersistentCollection.php(636): Doctrine\ORM\PersistentCollection::doInitialize
#7 /vendor/doctrine/orm/src/PersistentCollection.php(186): Doctrine\ORM\PersistentCollection::initialize
#8 /vendor/doctrine/collections/src/AbstractLazyCollection.php(157): Doctrine\Common\Collections\AbstractLazyCollection::toArray
#9 /src/Domain/Clients/Client.php(282): Shift\Domain\Clients\Client::getServiceAgreements
#10 /src/ClientBundle/Message/SendServiceAgreementEnhancementApprovalRemindersHandler.php(37): Shift\ClientBundle\Message\SendServiceAgreementEnhancementApprovalRemindersHandler::__invoke
#11 /vendor/symfony/messenger/Middleware/HandleMessageMiddleware.php(152): Symfony\Component\Messenger\Middleware\HandleMessageMiddleware::callHandler
#12 /vendor/symfony/messenger/Middleware/HandleMessageMiddleware.php(91): Symfony\Component\Messenger\Middleware\HandleMessageMiddleware::handle
#13 /vendor/symfony/messenger/Middleware/SendMessageMiddleware.php(71): Symfony\Component\Messenger\Middleware\SendMessageMiddleware::handle
#14 /src/Messenger/ThrottleMessengerMiddleware.php(49): Shift\Messenger\ThrottleMessengerMiddleware::handle
#15 /vendor/symfony/doctrine-bridge/Messenger/DoctrinePingConnectionMiddleware.php(34): Symfony\Bridge\Doctrine\Messenger\DoctrinePingConnectionMiddleware::handleForManager
#16 /vendor/symfony/doctrine-bridge/Messenger/AbstractDoctrineMiddleware.php(42): Symfony\Bridge\Doctrine\Messenger\AbstractDoctrineMiddleware::handle
#17 /vendor/symfony/messenger/Middleware/DeduplicateMiddleware.php(28): Symfony\Component\Messenger\Middleware\DeduplicateMiddleware::handle
#18 /vendor/symfony/messenger/Middleware/FailedMessageProcessingMiddleware.php(34): Symfony\Component\Messenger\Middleware\FailedMessageProcessingMiddleware::handle
#19 /vendor/symfony/messenger/Middleware/DispatchAfterCurrentBusMiddleware.php(68): Symfony\Component\Messenger\Middleware\DispatchAfterCurrentBusMiddleware::handle
#20 /vendor/symfony/messenger/Middleware/RejectRedeliveredMessageMiddleware.php(41): Symfony\Component\Messenger\Middleware\RejectRedeliveredMessageMiddleware::handle
#21 /vendor/symfony/messenger/Middleware/AddBusNameStampMiddleware.php(35): Symfony\Component\Messenger\Middleware\AddBusNameStampMiddleware::handle
#22 /vendor/symfony/messenger/MessageBus.php(69): Symfony\Component\Messenger\MessageBus::dispatch
#23 /vendor/symfony/messenger/RoutableMessageBus.php(51): Symfony\Component\Messenger\RoutableMessageBus::dispatch
#24 /vendor/symfony/messenger/Worker.php(170): Symfony\Component\Messenger\Worker::handleMessage
#25 /vendor/symfony/messenger/Worker.php(119): Symfony\Component\Messenger\Worker::run
#26 /vendor/symfony/messenger/Command/ConsumeMessagesCommand.php(260): Symfony\Component\Messenger\Command\ConsumeMessagesCommand::execute
#27 /vendor/symfony/console/Command/Command.php(318): Symfony\Component\Console\Command\Command::run
#28 /vendor/symfony/console/Application.php(1110): Symfony\Component\Console\Application::doRunCommand
#29 /vendor/symfony/framework-bundle/Console/Application.php(123): Symfony\Bundle\FrameworkBundle\Console\Application::doRunCommand
#30 /vendor/symfony/console/Application.php(359): Symfony\Component\Console\Application::doRun
#31 /vendor/symfony/framework-bundle/Console/Application.php(77): Symfony\Bundle\FrameworkBundle\Console\Application::doRun
#32 /vendor/symfony/console/Application.php(194): Symfony\Component\Console\Application::run
#33 /bin/console(34)

My application code that triggers the error is this:

return array_values($this->serviceAgreements->toArray());

There serviceAgreements is an index by id oneToMany relation (<one-to-many field="serviceAgreements" target-entity="Shift\Domain\ServiceAgreements\ServiceAgreement" mapped-by="client" index-by="id" orphan-removal="true" fetch="EXTRA_LAZY">). The same error happens with other oneToMany relations.

ObjectHydrator::hydrateRowData receives a row like this:

{
    bambora_pre_authorization_transaction_id_25: [Filtered],
    campaigns_limit_36: 1,
    charge_limit_34: 2900.0,
    client_id_30: [Filtered],
    created_at_28: "2023-12-19 18:41:58",
    created_by_id_31: 053b6764-63e0-11ea-b868-0242ac170003,
    declined_at_26: null,
    declined_by_id_33: null,
    declined_reason_27: null,
    deleted_at_21: null,
    file_url_20: [Filtered],
    id_29: 0199f6e1-b704-72ef-9ffb-8702e1d63efe,
    payment_information_required_24: 1,
    period_end_38: 2024-02-20 06:59:59,
    period_start_37: 2023-12-19 20:40:56,
    sales_tax_rate_35: 0.05,
    signed_at_23: 2023-12-20 20:06:20,
    signed_by_id_32: 4064556a-71dd-11eb-bb43-0242ac140002,
    signing_due_date_22: 2023-12-19 18:41:58,
    type: one-time
}

notice the row has key id_29 but ORM expects id_10.

[danfraticiu]

After some deep(er) diving, I managed to reproduce the issue locally: is trigger by modifying doctrine filters (confirmed with Gedmo\SoftDeleteable\Filter\SoftDeleteableFilter and a blank filter created just to debugging).

1. fetch the oneToMany relation
2. disable (or otherwise modify) an EntityManager filter
3. call EntityManager::clear()
4. fetch again fails with Undefined array key

[danfraticiu]

I managed to isolate the issue and reproduce it, it happens when:

1. you have an indexed OneToMany relation and the child is a single table inheritance entity
2. the collection is loaded
3. modify the entityManager::filters
4. cause the collection to be fetched again (ex: clear the entityManager)
   • this will cause the SingleTablePersister to generate new columns aliases (doctrine/orm/src/Persisters/Entity/SingleTablePersister.php:42)
   • and then \Doctrine\ORM\Query\ResultSetMapping::addIndexBy selects the first alias it finds (src/Query/ResultSetMapping.php:263) , which is no longer the alias in the SQL query.

---

The solution:

in \Doctrine\ORM\Query\ResultSetMapping::addIndexBy select the last alias found for the index field. But I'm not sure if this will cause issues in other situations as I'm not very familiar with the internals of the ORM. What I can say for sure is that it doesn't brake any of the tests. 🤷

I will create a PR soon.

[perk11]

Also experiencing this issue. In my case it's caused by disabling and enabling the filter in the loop

[greg0ire]

What I can say for sure is that it doesn't brake any of the tests

I just authorized the CI to run the code you submitted in #12228, and it actually breaks tests.

Also experiencing this issue. In my case it's caused by disabling and enabling the filter in the loop

@perk11 it would be great if you could contributed a failing test based on that scenario or on the scenario described by @danfraticiu , to go with #12228

[igrizzli]

I have researched this issue:

In https://github.com/doctrine/orm/blob/3.5.x/src/Persisters/Entity/BasicEntityPersister.php#L1259 we are adding entity result multiple times if we are changing filter state in runtime.

May be it have sense to clear RSM before populating it second time?

[perk11]

I work with @igrizzli. Can someone please review if the fix suggested by him makes sense? We can provide a PR with the fix and the test then.