create PBF to GeoJSON translator lamda

[kpwebb]

@migurski Just wanted to flag that the global z12 PBF tile set is up on CloudFront:

http://tiles.sharedstreets.io/12-{x}-{y}.{reference|intersection|geometry|metadata}.pbf

I'd love to work with you all to put a PBF -> GeoJSON lambda in front of these tiles using your GeoJSON spec. I'm curious if that's something you all would be willing to take on writing and we could host as part of the core SharedStreets API?

[migurski]

This is great, Kevin! We’re currently running our experimental live service using Overpass API + OSMnx. It works well but relies on the kindness of a very-dynamic backing service.

[migurski]

I’m new to PBF, so I have some beginner questions about reading data based on a few minutes of tinkering.

I’ve found this .proto definition for SharedStreets, and compiled it to a Python file using protoc. This is the resulting Python file.

Here’s how I’m attempting to read PBF data, and the error I see:

>>> import sharedstreets_pb2 as ss
>>> data = open('12-656-1582.reference.pbf', 'rb').read()
>>> ss.SharedStreetsReference().ParseFromString(data)

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/migurski/Documents/SharedStreets/.venv-mvt/lib/python3.6/site-packages/google/protobuf/message.py", line 185, in ParseFromString
    self.MergeFromString(serialized)
  File "/Users/migurski/Documents/SharedStreets/.venv-mvt/lib/python3.6/site-packages/google/protobuf/internal/python_message.py", line 1083, in MergeFromString
    if self._InternalParse(serialized, 0, length) != length:
  File "/Users/migurski/Documents/SharedStreets/.venv-mvt/lib/python3.6/site-packages/google/protobuf/internal/python_message.py", line 1109, in InternalParse
    new_pos = local_SkipField(buffer, new_pos, end, tag_bytes)
  File "/Users/migurski/Documents/SharedStreets/.venv-mvt/lib/python3.6/site-packages/google/protobuf/internal/decoder.py", line 850, in SkipField
    return WIRETYPE_TO_SKIPPER[wire_type](buffer, pos, end)
  File "/Users/migurski/Documents/SharedStreets/.venv-mvt/lib/python3.6/site-packages/google/protobuf/internal/decoder.py", line 799, in _SkipGroup
    new_pos = SkipField(buffer, pos, end, tag_bytes)
  File "/Users/migurski/Documents/SharedStreets/.venv-mvt/lib/python3.6/site-packages/google/protobuf/internal/decoder.py", line 850, in SkipField
    return WIRETYPE_TO_SKIPPER[wire_type](buffer, pos, end)
  File "/Users/migurski/Documents/SharedStreets/.venv-mvt/lib/python3.6/site-packages/google/protobuf/internal/decoder.py", line 820, in _RaiseInvalidWireType
    raise _DecodeError('Tag had invalid wire type.')
google.protobuf.message.DecodeError: Tag had invalid wire type.

Should this work?

[kpwebb]

Ah! So the SharedStreets tiles use length delimited encoding (the technique $GOOG recommends for files over ~1mb to allow stream processing of data). Turns out the Python Protobuf library doesn't have helper functions to parse these messages, but I found this example on how do to it:

https://www.datadoghq.com/blog/engineering/protobuf-parsing-in-python/

Just posted a quick example here using your compiled version of the .proto file:

https://github.com/sharedstreets/sharedstreets-python/blob/master/test.py

[kpwebb]

Oh, also, just made https://github.com/sharedstreets/sharedstreets-python/ public. @migurski I'll add you to the repo now and let's move the Python specific stuff over there.

[migurski]

Thank you, Kevin—pip install sharedstreets is a much better experience!

[kpwebb]

Still a WIP, but check out the test.py file in the repo as an example on how to open the PBF files.

[kpwebb]

But lots of interest in Python interfaces so let's build this out to support common needs with tile data!

[migurski]

I have had a chance to get into this, see sharedstreets/sharedstreets-python#3.

For our use, OSM street names are helpful. Would it be possible to get those into the OSM metadata? It’s not otherwise super useful since we're never tying things back to OSM by node or way ID, but it is very helpful to be able to show meaningful street and intersection names.

I’ve also noticed that the sample data is very detailed, with lots of service alleys and driveways included. We’ve been targeting slightly larger streets for our use, and I wonder if it would make sense to provide several different cuts of the data with varying road class cutoffs? This might also be helpful for future applications like hierarchical tile-based routing where you might want big, coarse freeway networks at lower zoom levels.