UnicodeDecodeError for LotV beta replay

[StoicLoofah]

From this replay of Puck in a PvZ for the new LotV beta

http://lotv.spawningtool.com/4/download/

Python 3.4.0 (default, Apr 11 2014, 13:05:18) 
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sc2reader
>>> replay = sc2reader.load_replay('replays/lotv.SC2Replay')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/kevin/sc2reader/sc2reader/factories/sc2factory.py", line 85, in load_replay
    return self.load(Replay, source, options, **new_options)
  File "/home/kevin/sc2reader/sc2reader/factories/sc2factory.py", line 137, in load
    return self._load(cls, resource, filename=filename, options=options)
  File "/home/kevin/sc2reader/sc2reader/factories/sc2factory.py", line 146, in _load
    obj = cls(resource, filename=filename, factory=self, **options)
  File "/home/kevin/sc2reader/sc2reader/resources.py", line 271, in __init__
    self._read_data(data_file, self._get_reader(data_file))
  File "/home/kevin/sc2reader/sc2reader/resources.py", line 601, in _read_data
    self.raw_data[data_file] = reader(data, self)
  File "/home/kevin/sc2reader/sc2reader/readers.py", line 33, in __call__
    ) for i in range(data.read_bits(5))],
  File "/home/kevin/sc2reader/sc2reader/readers.py", line 33, in <listcomp>
    ) for i in range(data.read_bits(5))],
  File "/home/kevin/sc2reader/sc2reader/decoders.py", line 252, in read_aligned_string
    return self._buffer.read_string(count, encoding)
  File "/home/kevin/sc2reader/sc2reader/decoders.py", line 108, in read_string
    return self.read_bytes(count).decode(encoding)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfc in position 33: invalid start byte

Currently working on fixing it, and I'll try to post a fix if I can figure things out (probably not). Just thought I would share so we could work on these things together

[StoicLoofah]

Alright, I am of the impression that we just need blizzard to tell us what has changed in the format to be able to fix this. Or maybe we just need someone smarter than me looking at this.

[GraylinKim]

I mean, it would certainly be helpful if they just told us what changed. They have no record of providing information on non-official builds though so I wouldn't hold your breath. I'll try to take a look at it later this week. If things haven't changed too dramatically I can probably spot the difference.

[dsjoerg]

@koalaling just posted a new s2protocol for LotV: Blizzard/s2protocol@2c1d7ff

[StoicLoofah]

What a champ! I'll take a stab at it, though I'm not actually good at this stuff, so if you don't hear more from me, it is because I failed

[GraylinKim]

I've started doing work for this, you can follow my progress in the LotV branch. It isn't quite able to parse the replays yet and its not clear how many non-parsing related changes will be required to make for a good read.

You should checkout the changes in the readers file. Pretty interesting I'd say. Maybe this will also parse Heroes of the Storm replays?

[EHadoux]

I guess hero_talent_tree_selection_panel_toggled_event is pretty clear to me!

[StoicLoofah]

Wow, quick work! I'm playing around with it a bit to see what i can find.

One more thing I'll note for your TODO list is to set the expansion

sc2reader/sc2reader/resources.py

Line 344 in 722ae4f

else:

Thanks for your continued effort to maintain this library!

[StoicLoofah]

So this probably won't make you feel really comfortable, but I'm using this branch in production right now since I want to run more replays through it. So far, so good! I'll report any issues I come up against as they pop up.

For others who are building on top of sc2reader, you can check out my branch of spawningtool where I'm going through the types that blizzard is using

https://github.com/StoicLoofah/spawningtool/tree/32-lotv

[StoicLoofah]

I haven't dug much into it, but I got an archon mode replay that isn't quite working correctly

http://lotv.spawningtool.com/81/download/

As a sample of what this looks like:

>>> replay.players
[Player 1 - Amshel (Zerg), Player 2 - DrennoC (Zerg), Player 3 - Filleriste (Terran), Player 4 - Sudhish (Terran)]
>>> replay.players[0].units[-10:]
[Zergling [A240005], Zergling [9540004], Zergling [C840004], Zergling [1BC0007], Zergling [A440004], Zergling [C380004], Zergling [C100007], Zergling [C040002], Zergling [BA00005], Larva [4840002]]
>>> replay.players[1].units[-10:]
[Marine [3980003], Marine [6040003], Marauder [A4C0004], Marine [D00003], Marine [AE00004], Marine [A240004], Marauder [A540009], Marine [AA40003], Marine [AAC0002], Marine [AE80006]]

So the 2 zerg players are on the first team, but the units are associated with those 2 players rather than with the players on the 2 teams respectively.

This possibly should be addressed separately, and I haven't looked deeply into why this is happening, though my gut says that blizzard got this working through some creative indexing within the lobby that is throwing us off. I am as useless as ever in actually understanding how to fix these things but am happy to dig more or provide better reporting as desired

[EHadoux]

It may seems obvious but what is the size of the players list? If it is 4 it's awkward but if it's 2, the two players of each team may have been merged. In the former case, is players[0].units equals to players[2].units? I cannot test it myself ATM

[StoicLoofah]

Another fun fact that isn't too systemic in sc2reader but is probably a big assumption for others is that there are 16 frames per second. One change was that Blizzard slowed down the game clock so that it matches real-time. I verified the differences by comparing unit build times between

https://docs.google.com/spreadsheets/d/1JtL6Wd9q5Qxm3KEewnOXSz1KxXp6d6OF6KmrgO183Lw/edit#gid=410665689
https://github.com/StoicLoofah/spawningtool/blob/43c2ca0574c9dac3ce365853419a634962afa2ff/spawningtool/constants.py

Doing the math, it seems like they sped it up by roughly 1.38, which naively is around 22 frames per second. Of course, I have no idea whether they actually implemented it that way since I haven't easily been able to find a replay and matching VOD (and don't have beta access to test it), but that may be something we have to integrate into our thinking as well.

[StoicLoofah]

@EHadoux currently the size of players is 4

[EHadoux]

I'm pretty sure your maths are right as 1.38 was the speed factor between real time and fastest speed (is that the right name? I have the game in French) AFAIK. I watched some streams and it seems like the clock went slower.

[GraylinKim]

Okay @StoicLoofah I pushed a couple small changes. A fixed an attribute mapping that was causing parse failure and fixed the replay.expansion attribute. Please forward all errors that pop into your logs to my email so I can diagnose.

If you guys can fill-in the correct values for this table we can fix the game timings for LotV as well:

GAME_SPEED_FACTOR = {
    'Slower':   0.6,
    'Slow':     0.8,
    'Normal':   1.0,
    'Fast':     1.2,
    'Faster':   1.4
}

It seems that I have a choice between looking into archon mode, adding data for the new units/abilities, and wrapping data for the new events. Anyone have a preference?