Skip to content

rmpv: decoding the reserved byte 0xc1 returns Value::Nil instead of an error #392

Open

Description

@hidingmyname

Location (3Hren/msgpack-rust @ default branch cf88001):

        Marker::Reserved => Value::Nil,
        Marker::Reserved => ValueRef::Nil,

Problem
rmpv::decode::read_value decodes the byte 0xc1, which the MessagePack spec marks as "never used" (spec.md L105), as Value::Nil (and read_value_ref as ValueRef::Nil). The result is indistinguishable from a real nil (0xc0), also when the byte is nested inside a container.

Expected: Err(_), since 0xc1 is not a valid MessagePack format byte. rmp-serde in this repository already does this: its deserializer returns Error::TypeMismatch(Marker::Reserved) (rmp-serde/src/decode.rs:618). The same bytes are therefore rejected by rmp_serde but accepted as nil by rmpv.

Reproduction

use rmpv::decode;

let r = decode::read_value(&mut &[0xc1u8][..]);
// Ok(Nil)

let r = decode::read_value(&mut &[0x91u8, 0xc1][..]); // fixarray(1) [0xc1]
// Ok(Array([Nil]))

Test output:

thread 'reserved_byte_should_error' panicked at rmpv-tests/tests/test_reserved_marker.rs:10:5:
Expected an error when decoding reserved byte 0xc1, but got: Nil
thread 'reserved_byte_inside_array_should_error' panicked at rmpv-tests/tests/test_reserved_marker.rs:24:5:
Expected an error when decoding reserved byte 0xc1 inside array, but got: Array([Nil])

Environment: master cf88001 (rmpv 1.3.1), rustc 1.90.0.

Already implemented in msgpack/msgpack-python
msgpack-python is another implementation of the same MessagePack spec; the spec's "never used" byte is the primary basis, and msgpack-python is precedent. PR: msgpack/msgpack-python#331 — "Add StackError and FormatError" (merged 2018-11-20T04:12:49Z, merge commit 44254dd35e8). msgpack-python already rejected unknown header bytes; this PR gave them a dedicated FormatError ("Unknown header: 0xc1") in both the pure-Python and C decoders and added tests for exactly the two inputs above (\xc1 and \x91\xc1). The StackError part of the PR is unrelated.

How msgpack-python implemented it:

--- a/msgpack/fallback.py
+++ b/msgpack/fallback.py
@@ -561,7 +589,7 @@ def _read_header(self, execute=EX_CONSTRUCT):
                 raise ValueError("%s exceeds max_map_len(%s)", n, self._max_map_len)
             typ = TYPE_MAP
         else:
-            raise ValueError("Unknown header: 0x%x" % b)
+            raise FormatError("Unknown header: 0x%x" % b)
         return typ, n, obj
 
     def _unpack(self, execute=EX_CONSTRUCT):
--- a/msgpack/unpack_template.h
+++ b/msgpack/unpack_template.h
@@ -229,7 +208,8 @@ static inline int unpack_execute(unpack_context* ctx, const char* data, Py_ssize
                 case 0xdf:  // map 32
                     again_fixed_trail(NEXT_CS(p), 2 << (((unsigned int)*p) & 0x01));
                 default:
-                    goto _failed;
+                    ret = -2;
+                    goto _end;
                 }
             SWITCH_RANGE(0xa0, 0xbf)  // FixRaw
                 again_fixed_trail_if_zero(ACS_RAW_VALUE, ((unsigned int)*p & 0x1f), _raw_zero);
--- a/test/test_except.py
+++ b/test/test_except.py
@@ -19,13 +19,34 @@ def test_raise_on_find_unsupported_value():
...
+    with raises(FormatError):
+        unpackb(b"\xc1")  # (undefined tag)
+
+    with raises(FormatError):
+        unpackb(b"\x91\xc1")  # fixarray(len=1) [ (undefined tag) ]
...

Suggested fix

Return an error from both Marker::Reserved arms instead of a Nil value. To avoid a breaking change to the exhaustive rmpv::decode::Error enum, the error is reported as InvalidMarkerRead with an io::ErrorKind::InvalidData error:

diff --git a/rmpv/src/decode/value.rs b/rmpv/src/decode/value.rs
--- a/rmpv/src/decode/value.rs
+++ b/rmpv/src/decode/value.rs
@@ -200,7 +200,12 @@ fn read_value_inner<R>(rd: &mut R, depth: u16) -> Result<Value, Error> where R:
             let (ty, vec) = read_ext_body(rd, len, depth)?;
             Value::Ext(ty, vec)
         },
-        Marker::Reserved => Value::Nil,
+        Marker::Reserved => {
+            return Err(Error::InvalidMarkerRead(io::Error::new(
+                io::ErrorKind::InvalidData,
+                "reserved marker 0xc1",
+            )))
+        },
     };
 
     Ok(val)
diff --git a/rmpv/src/decode/value_ref.rs b/rmpv/src/decode/value_ref.rs
--- a/rmpv/src/decode/value_ref.rs
+++ b/rmpv/src/decode/value_ref.rs
@@ -255,7 +255,12 @@ fn read_value_ref_inner<'a, R>(rd: &mut R, depth: u16) -> Result<ValueRef<'a>, E
             let (ty, vec) = read_ext_body(rd, len as usize, depth)?;
             ValueRef::Ext(ty, vec)
         },
-        Marker::Reserved => ValueRef::Nil,
+        Marker::Reserved => {
+            return Err(Error::InvalidMarkerRead(io::Error::new(
+                ErrorKind::InvalidData,
+                "reserved marker 0xc1",
+            )))
+        },
     };
 
     Ok(val)

The existing from_reserved test in rmpv/tests/decode_ref.rs asserted the old Nil result, so it is changed to expect InvalidMarkerRead. Verified on cf88001 (rustc 1.90.0): the test above fails without this patch and passes with it; all rmpv and rmpv-tests tests pass with it (253), and rmp-serde is unaffected.

I can submit a PR for this.

Related: searched for 0xc1, reserved marker, Reserved/Nil, invalid marker; no existing report. #136 (2017 test-coverage PR) notes that Marker::Reserved was left untested in value.rs; #388 / #367 are unrelated.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions