Repository navigation
Description
Location (3Hren/msgpack-rust @ default branch cf88001):
Marker::Reserved => Value::Nil, Marker::Reserved => ValueRef::Nil,Problem
rmpv::decode::read_value decodes the byte 0xc1, which the MessagePack spec marks as "never used" (spec.md L105), as Value::Nil (and read_value_ref as ValueRef::Nil). The result is indistinguishable from a real nil (0xc0), also when the byte is nested inside a container.
Expected: Err(_), since 0xc1 is not a valid MessagePack format byte. rmp-serde in this repository already does this: its deserializer returns Error::TypeMismatch(Marker::Reserved) (rmp-serde/src/decode.rs:618). The same bytes are therefore rejected by rmp_serde but accepted as nil by rmpv.
Reproduction
use rmpv::decode;
let r = decode::read_value(&mut &[0xc1u8][..]);
// Ok(Nil)
let r = decode::read_value(&mut &[0x91u8, 0xc1][..]); // fixarray(1) [0xc1]
// Ok(Array([Nil]))Test output:
thread 'reserved_byte_should_error' panicked at rmpv-tests/tests/test_reserved_marker.rs:10:5:
Expected an error when decoding reserved byte 0xc1, but got: Nil
thread 'reserved_byte_inside_array_should_error' panicked at rmpv-tests/tests/test_reserved_marker.rs:24:5:
Expected an error when decoding reserved byte 0xc1 inside array, but got: Array([Nil])
Environment: master cf88001 (rmpv 1.3.1), rustc 1.90.0.
Already implemented in msgpack/msgpack-python
msgpack-python is another implementation of the same MessagePack spec; the spec's "never used" byte is the primary basis, and msgpack-python is precedent. PR: msgpack/msgpack-python#331 — "Add StackError and FormatError" (merged 2018-11-20T04:12:49Z, merge commit 44254dd35e8). msgpack-python already rejected unknown header bytes; this PR gave them a dedicated FormatError ("Unknown header: 0xc1") in both the pure-Python and C decoders and added tests for exactly the two inputs above (\xc1 and \x91\xc1). The StackError part of the PR is unrelated.
How msgpack-python implemented it:
--- a/msgpack/fallback.py
+++ b/msgpack/fallback.py
@@ -561,7 +589,7 @@ def _read_header(self, execute=EX_CONSTRUCT):
raise ValueError("%s exceeds max_map_len(%s)", n, self._max_map_len)
typ = TYPE_MAP
else:
- raise ValueError("Unknown header: 0x%x" % b)
+ raise FormatError("Unknown header: 0x%x" % b)
return typ, n, obj
def _unpack(self, execute=EX_CONSTRUCT):
--- a/msgpack/unpack_template.h
+++ b/msgpack/unpack_template.h
@@ -229,7 +208,8 @@ static inline int unpack_execute(unpack_context* ctx, const char* data, Py_ssize
case 0xdf: // map 32
again_fixed_trail(NEXT_CS(p), 2 << (((unsigned int)*p) & 0x01));
default:
- goto _failed;
+ ret = -2;
+ goto _end;
}
SWITCH_RANGE(0xa0, 0xbf) // FixRaw
again_fixed_trail_if_zero(ACS_RAW_VALUE, ((unsigned int)*p & 0x1f), _raw_zero);
--- a/test/test_except.py
+++ b/test/test_except.py
@@ -19,13 +19,34 @@ def test_raise_on_find_unsupported_value():
...
+ with raises(FormatError):
+ unpackb(b"\xc1") # (undefined tag)
+
+ with raises(FormatError):
+ unpackb(b"\x91\xc1") # fixarray(len=1) [ (undefined tag) ]
...Suggested fix
Return an error from both Marker::Reserved arms instead of a Nil value. To avoid a breaking change to the exhaustive rmpv::decode::Error enum, the error is reported as InvalidMarkerRead with an io::ErrorKind::InvalidData error:
diff --git a/rmpv/src/decode/value.rs b/rmpv/src/decode/value.rs
--- a/rmpv/src/decode/value.rs
+++ b/rmpv/src/decode/value.rs
@@ -200,7 +200,12 @@ fn read_value_inner<R>(rd: &mut R, depth: u16) -> Result<Value, Error> where R:
let (ty, vec) = read_ext_body(rd, len, depth)?;
Value::Ext(ty, vec)
},
- Marker::Reserved => Value::Nil,
+ Marker::Reserved => {
+ return Err(Error::InvalidMarkerRead(io::Error::new(
+ io::ErrorKind::InvalidData,
+ "reserved marker 0xc1",
+ )))
+ },
};
Ok(val)
diff --git a/rmpv/src/decode/value_ref.rs b/rmpv/src/decode/value_ref.rs
--- a/rmpv/src/decode/value_ref.rs
+++ b/rmpv/src/decode/value_ref.rs
@@ -255,7 +255,12 @@ fn read_value_ref_inner<'a, R>(rd: &mut R, depth: u16) -> Result<ValueRef<'a>, E
let (ty, vec) = read_ext_body(rd, len as usize, depth)?;
ValueRef::Ext(ty, vec)
},
- Marker::Reserved => ValueRef::Nil,
+ Marker::Reserved => {
+ return Err(Error::InvalidMarkerRead(io::Error::new(
+ ErrorKind::InvalidData,
+ "reserved marker 0xc1",
+ )))
+ },
};
Ok(val)The existing from_reserved test in rmpv/tests/decode_ref.rs asserted the old Nil result, so it is changed to expect InvalidMarkerRead. Verified on cf88001 (rustc 1.90.0): the test above fails without this patch and passes with it; all rmpv and rmpv-tests tests pass with it (253), and rmp-serde is unaffected.
I can submit a PR for this.
Related: searched for 0xc1, reserved marker, Reserved/Nil, invalid marker; no existing report. #136 (2017 test-coverage PR) notes that Marker::Reserved was left untested in value.rs; #388 / #367 are unrelated.
Activity