Skip to content

Commit 5eac204

Browse files
committed
Fix decoding large json arrays
Without the fix, the included test fails with ``` AssertionError: Result length not requested length: Expected=2. Actual=0. Position: 500079. Data Length: 500079 ``` But I suspect (and have seen) also other errors, depending on specific contents of the byte stream, e.g. ("Json type $random is not handled"). Int32/UInt32 values in json arrays are encoded inline in the "large" encoding mode, but they are stored using 4 bytes, not 8. So the `read_uint64()` calls are wrong here, `read_uint32()` works. Also the check for `large` in that case is redundant, as `large` is always True for inlined int32/uint32 values. Relevant parts in mysql server code are functions `serialize_json_array`, `should_inline_value` and `insert_offset_or_size` in `sql/json_binary.cc` [1]. I believe this fixes issue #311. [1] https://github.com/mysql/mysql-server/blob/5.7/sql/json_binary.cc
1 parent 1376c6b commit 5eac204

File tree

2 files changed

+12
-2
lines changed

2 files changed

+12
-2
lines changed

pymysqlreplication/packet.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -402,9 +402,9 @@ def read_binary_json_type_inlined(self, t, large):
402402
elif t == JSONB_TYPE_UINT16:
403403
return self.read_uint32() if large else self.read_uint16()
404404
elif t == JSONB_TYPE_INT32:
405-
return self.read_int64() if large else self.read_int32()
405+
return self.read_int32()
406406
elif t == JSONB_TYPE_UINT32:
407-
return self.read_uint64() if large else self.read_uint32()
407+
return self.read_uint32()
408408

409409
raise ValueError('Json type %d is not handled' % t)
410410

pymysqlreplication/tests/test_data_type.py

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -513,6 +513,16 @@ def test_json_large(self):
513513

514514
self.assertEqual(event.rows[0]["values"]["value"], to_binary_dict(data))
515515

516+
def test_json_large_array(self):
517+
"Test json array larger than 64k bytes"
518+
if not self.isMySQL57():
519+
self.skipTest("Json is only supported in mysql 5.7")
520+
create_query = "CREATE TABLE test (id int, value json);"
521+
large_array = dict(my_key=[i for i in range(100000)])
522+
insert_query = "INSERT INTO test (id, value) VALUES (1, '%s');" % (json.dumps(large_array),)
523+
event = self.create_and_insert_value(create_query, insert_query)
524+
self.assertEqual(event.rows[0]["values"]["value"], to_binary_dict(large_array))
525+
516526
def test_json_large_with_literal(self):
517527
if not self.isMySQL57():
518528
self.skipTest("Json is only supported in mysql 5.7")

0 commit comments

Comments
 (0)