Object reference table key collision: an object is packed as a back-reference to a different object #191

Summary

msgpack_pack() sometimes writes an object as a back-reference to a different, earlier object. The output has no warning or error. When you unpack it, that array slot contains the wrong object. In our production cache this replaced one entity in a list of a few thousand with an enum case.

Cause

msgpack_var_add() in msgpack_pack.c builds its identity key for an object like this:

(((size_t)Z_OBJCE_P(var_noref) << 5)
 | ((size_t)Z_OBJCE_P(var_noref) >> (sizeof(long) * 8 - 5))) + (long)Z_OBJ_HANDLE_P(var_noref)

This key is not unique. Two live objects A and B of different classes get the same key when:

handle(B) - handle(A) == 32 * (ce(A) - ce(B))

Class entries of userland classes are usually only a few hundred bytes apart. So with a few tens of thousands of objects in one payload, a collision is likely. The colliding object is then emitted as MSGPACK_SERIALIZE_TYPE_OBJECT_REFERENCE ({nil: 4, 0: <idx>}) pointing at the other object.

Reproduction

<?php
class Item {
    public Status $status = Status::Active;
}

enum Status: string {
    case Active = 'active';
}

$items = [];
for ($i = 0; $i < 200000; $i++) {
    $items[] = new Item();
}

$unpacked = msgpack_unpack(msgpack_pack($items));
foreach ($unpacked as $idx => $value) {
    if (!$value instanceof Item) {
        echo "index $idx: expected Item, got " . get_debug_type($value) . "\n";
    }
}
echo "done\n";

$ php -d memory_limit=2G repro.php
index 19199: expected Item, got Status
done

Expected: only done. unserialize(serialize($items)) round-trips correctly.

Notes:

• The exact index depends on how far apart the two class entries are in memory. Here ce(Status) - ce(Item) = 600 bytes, so the colliding handle is 1 + 600 * 32 = 19201, which is index 19199. Two things make it reproduce reliably:
  • Item is declared before Status, so its class entry has the lower address.
  • Handles are consecutive.
• The bug is not specific to enums: any two objects of different classes can collide. Enums just make it easy to have one object referenced from every element.
• In real applications handles are reused, so it can hit lists with only a few thousand objects. We saw it at index 5039 of 5123.

Environment:

• msgpack 3.0.1, and current master (4a68908)
• PHP 8.3.33 and 8.4.26 (NTS, Linux x86_64), with and without opcache
• Default INI settings (msgpack.php_only=1)

Suggested fix

Use a collision-free identity, as ext/standard/var.c does: key objects by Z_OBJ_HANDLE or by the zend_object * address alone, in a key space that cannot overlap with the array keys. Any object created during packing, such as the result of __serialize()/__sleep(), must stay alive until packing ends so its handle or address cannot be reused.