Support for variable-length, non-greedy Payload() #47
Description
tl;dr: It would be nice if Payload() consumed data until the first None/null-character if no length is specified, rather than the last instance of it as it appears to currently be doing.
I'm running into a problem with a Structure containing two null-delimited strings. The protocol doesn't specify information about the length of these strings. I wrote my own protocol parser before discovery this library, so I already have the protocol broken down into logical segments/Structures, but I was hoping to get rid of my hack job in favour of using Suitcase to parse the individual fields.
Reading the docs, I see this is probably an explanation for my situation:
Parameters: length_provider – The LengthField with which this variable length payload is associated. If not included, it is assumed that the length_provider should consume the remainder of the bytes available in the string. This is only valid in cases where the developer knows that they will be dealing with a fixed sequence of bytes (already boxed).
What would be the best way to work within the framework with such a constraint? Is there any way to make Payload() be lazy rather than greedy?
success.py:
from suitcase.structure import Structure
from suitcase.fields import (
SBInt8,
SBInt16,
UBInt32,
SBInt64,
Payload,
Magic
)
class Header(Structure):
a = UBInt32()
b = SBInt8()
c = Payload()
e = SBInt64()
f = SBInt16()
g = SBInt16()
h = SBInt16()
i = SBInt16()Success output:
In [1]: import success
In [2]: s = success.Header()
In [3]: s.unpack(b'\x00\x00\x007\n123.45.67.89-8888\x00Some-String-2\x00\x00\x00\x01Z\xa5\xfb\xc8\xab\x00\x0e\x00\x15\x00\x01\x00\x00')
In [4]: s
Out[4]:
Header (
a=55,
b=10,
c=b'123.45.67.89-8888\x00Some-String-2\x00',
e=1488843425963,
f=14,
g=21,
h=1,
i=0,
)
failure.py:
from suitcase.structure import Structure
from suitcase.fields import (
SBInt8,
SBInt16,
UBInt32,
SBInt64,
Payload,
Magic
)
class Header(Structure):
header_size = UBInt32()
version = SBInt8()
nis_id = Payload()
msg_id = Payload()
timestamp = SBInt64()
event_size = SBInt16()
job_discard_size = SBInt16()
num_jobs = SBInt16()
num_discards = SBInt16()Failure output:
In [1]: import failure
In [2]: f = failure.Header()
In [3]: f.unpack(b'\x00\x00\x007\n123.45.67.89-8888\x00Some-String-2\x00\x00\x00\x01Z\xa5\xfb\xc8\xab\x00\x0e\x00\x15\x00\x01\x00\x00')
---------------------------------------------------------------------------
SuitcaseParseError Traceback (most recent call last)
<ipython-input-3-91ea443f72a0> in <module>()
----> 1 f.unpack(b'\x00\x00\x007\n123.45.67.89-8888\x00Some-String-2\x00\x00\x00\x01Z\xa5\xfb\xc8\xab\x00\x0e\x00\x15\x00\x01\x00\x00')
/home/mpelikan/.local/lib/python3.6/site-packages/suitcase/structure.py in unpack(self, data, trailing)
339
340 def unpack(self, data, trailing=False):
--> 341 return self._packer.unpack(data, trailing)
342
343 def pack(self):
/home/mpelikan/.local/lib/python3.6/site-packages/suitcase/structure.py in unpack(self, data, trailing)
62 def unpack(self, data, trailing=False):
63 stream = BytesIO(data)
---> 64 self.unpack_stream(stream)
65 stream.tell()
66 if trailing:
/home/mpelikan/.local/lib/python3.6/site-packages/suitcase/structure.py in unpack_stream(self, stream)
150 "%r we tried to read %s bytes but "
151 "we were only able to read %s." %
--> 152 (_name, length, len(data)))
153 try:
154 field.unpack(data)
SuitcaseParseError: While attempting to parse field 'd' we tried to read None bytes but we were only able to read 32.