Improve README

README.md

@@ -8,6 +8,13 @@
 
 This is the source code for the
 [Forged Alliance Forever](https://www.faforever.com/) lobby server.
+Click here to go to the
+[Server Python API Documentation](https://faforever.github.io/server/).
+
+-   For the Lua game mod see
+[faforever/fa](https://github.com/FAForever/fa).
+-   For the official FAF client see
+[faforever/downlords-faf-client](https://github.com/FAForever/downlords-faf-client)
 
 ## Overview
 The lobby server is the piece of software sitting at the very core of FAF,
@@ -27,6 +34,9 @@ The main responsibilities of the lobby server are:
 
 -   To perform rating calculations and updates
 
+In production, the lobby server is deployed behind a websocket bridge
+[faforever/ws_bridge_rs](https://github.com/FAForever/ws_bridge_rs).
+
 ## Support development
 
 Post a bounty on Issue Hunt. You can reward and financially help developers who
@@ -73,61 +83,80 @@ $ pipenv run tests
 $ pipenv run devserver
 ```
 
-# Network Protocol
-**NOTE: This section of the README is outdated. The QString based message
-format has been deprectaed and will be replaced by a UTF-8 json + newline
-format in version 2. Many commands are missing from here. For a more complete
-list see [https://faforever.github.io/server/](https://faforever.github.io/server/).**
-
-The protocol is mainly JSON-encoded maps, containing at minimum a `command` key,
-representing the command to dispatch.
-
-The wire format uses [QDataStream](http://doc.qt.io/qt-5/qdatastream.html) (UTF-16, BigEndian).
-
-For the lobbyconnection, each message is of the form:
-```
-ACTION: QString
-```
-With most carrying a footer containing:
-```
-LOGIN: QString
-SESSION: QString
-```
-
-## Incoming Packages
-
-##### Mod Vault
-
--   (deprecated) `{command: modvault, type: start}`: show the last 100 mods
--   (deprecated) `{command: modvault, type: like, uid: <uid>}`: check if user liked the mod, otherwise increase the like counter
--   (deprecated) `{command: modvault, type: download, uid: <uid>}`: notify server about a download (for download counter), does not start the download
-
-##### Social
--   `{command: social_add, friend|foe: <player_id>}`: Add a friend or foe
--   `{command: social_remove, friend|foe: <player_id>}`: Remove a friend or foe
-
-##### Avatar
--   `{command: avatar, action: list_avatar}`: Send a list of available avatars
--   `{command: avatar, action: select, avatar: <avatar_url>}`: Select a valid avatar for the player
-
-##### ICE Servers
-
--   (deprecated) `{command: ice_servers}`: Send ICE TURN/STUN servers - Returns: `{command: ice_servers, : <ice servers>, date_created: <date token was created in ISO 8601 format>, ttl: <ttl in seconds>}`
-
-#### Parties
--   `{command: invite_to_party, recipient_id: <...>}`: Invite this player to a party
--   `{command: accept_party_invite, sender_id: <...>}`: Accept the party invite from the given player
--   `{command: kick_player_from_party, kicked_player_id: <...>}`: Kick a player from a party you own
--   `{command: leave_party}`: Leave the party you are currently in
-
-##### Misc
-
--   (deprecated) `{command: ask_session}`: response with a welcome command and a valid session (can be delayed)
--   `{command: hello, version: <...>, login: <...>, password: <...>, unique_id: <...>, (session: <...>)}`: Log in to the server
-
-##  Stream (Deprecated)
-
-The stream API is deprecated, but currently the following message types are supported:
-
--   `PING`: response with a `PONG`
--   `PONG`: internal state changed to ponged
+# For Client Developers
+The official FAF client code is available at
+[faforever/downlords-faf-client](https://github.com/FAForever/downlords-faf-client).
+This can be used as a reference when implementing your own custom client.
+
+## Important Notes
+In order to avoid having your client break unexpectedly with a new server
+release, your server <-> client communication code must adhere to the following
+rules:
+-   Unrecognized server messages are ignored.
+-   Unrecognized fields in messages are ignored.
+
+This ensures that your client continues to function when new features are
+implemented on the server side. A new feature might mean that the server will
+include a new field in an existing message, or start sending an entirely new
+message all together. Such changes are considered to be backwards compatible
+additions to the server protocol.
+
+You can read more about the protocol API versioning
+[here](CONTRIBUTING.md#version-numbers).
+
+## Server Protocol
+There are two layers to the server protocol:
+1. **The wire format.**
+This is how messages are serialized to bytes and sent over the network stream.
+
+  *NOTE: in production, the client connects to the server via the websocket
+  bridge [faforever/ws_bridge_rs](https://github.com/FAForever/ws_bridge_rs)
+  rather than connecting directly to the TCP port.*
+
+2. **Application level messages.**
+Also sometimes called 'commands', messages are used to exchange state between
+the client and server. The client will need to implement appropriate logic for
+interpreting each message and reacting to it by sometimes updating UI elements,
+internal state, or launching or terminating external processes.
+
+### Wire Format
+Each message is serialized to a [JSON](https://www.json.org/) object followed
+by an ASCII newline byte (`b"\n"` or `b"\x0a"`). For additional information see
+the server API documentation:
+[SimpleJsonProtocol](https://faforever.github.io/server/protocol/simple_json.html)
+
+### Application messages
+1. **Request / response type commands.**
+  Most messages that are sent from the client to the server will be acknowledged
+  with a response message. The naming for these messages varies by command. For
+  instance the `ask_session` command will respond with a `session` command
+  containing the session id.
+2. **Asynchronous commands.**
+  Many messages are generated by activity of other users, or asynchronous
+  processes running on the server. These can generate 'broadcast' messages sent
+  out to many connected clients, or direct messages to a specific client without
+  being triggered by a request. Generally 'broadcast' messages are name `*_info`
+  and are used to synchronize the internal states of the server and client, and
+  often signal the need to update some UI elements. For instance the `game_info`
+  message sends updated information about a game that can either be in the lobby
+  state, be actively playing, or have ended.
+
+
+Work is ongoing to document these messages in a comprehensive way. For now, all
+commands that can be sent from the client -> server can be found via the server
+API documentation:
+[LobbyConnection](https://faforever.github.io/server/lobbyconnection.html)
+under the `command_*` methods. Check the source code for what fields the message
+is expected to have and any possible responses.
+
+It may also be useful to look at the definitions in the
+[faf-java-commons](https://github.com/FAForever/faf-java-commons/tree/develop/lobby/src/main/kotlin/com/faforever/commons/lobby)
+to see how the official client is deserializing messages from the server.
+
+### Deprecations
+Some fields or entire message classes may become deprecated and marked for
+removal over time. Actual removal is very rare as it can cause potential
+breakages of outdated clients. Currently, deprecated fields and messages will
+be marked with a `# DEPRECATED` comment in the server code and an explanation of
+how to migrate to the new functionality that is replacing the deprecated field
+or message.

server/__init__.py

@@ -75,7 +75,31 @@
 that go into more detail.
 
 ## Protocol
-TODO
+There are two layers to the server protocol:
+
+1. **The wire format.**
+This is how messages are serialized to bytes and sent over the network stream.
+2. **Application level messages.**
+Also sometimes called 'commands', messages are used to exchange state between
+the server and each client. Every command is part of a logical 'command flow'
+which is the expected sequence of messages exchanged between the server and
+client.
+
+### Wire Format
+See `server.protocol`.
+
+### Application messages
+See `server.lobbyconnection`.
+
+Terms:
+
+- **Command**: A message sent between the client and server. Every message is
+expected to deserialize to a python dict with at least one key `"command"`
+indicating the type of message. For example: `{"command": "ping"}`.
+- **Command Flow**: An expected logical sequence of commands. This is encoded in
+the application logic of each command handler.
+
+TODO: expand
 
 # Legal
 - Copyright © 2012-2014 Gael Honorez

server/protocol/__init__.py

@@ -1,5 +1,12 @@
 """
 Protocol format definitions
+
+Every message sent between the lobby server and a client is expected to
+deserialize to a python dictionary. The structure of that dictionary depends on
+the application logic (see `server.lobbyconnection`).
+
+This module defines the classes that handle the wire format, i.e. how messages
+are serialized to bytes and sent across the network.
 """
 
 from .gpgnet import GpgNetClientProtocol, GpgNetServerProtocol

server/protocol/protocol.py

@@ -1,3 +1,5 @@
+"""Protocol base class"""
+
 import contextlib
 import json
 from abc import ABCMeta, abstractmethod

server/protocol/qdatastream.py

@@ -1,3 +1,24 @@
+r"""DEPRECATED: Legacy [QDataStream](http://doc.qt.io/qt-5/qdatastream.html)
+(UTF-16, BigEndian) encoded data format.
+
+For the lobbyconnection, each message is of the form:
+```
+ACTION: QString
+```
+With most carrying a footer containing:
+```
+LOGIN: QString
+SESSION: QString
+```
+
+# Example:
+```python
+>>> QDataStreamProtocol.encode_message({"command": "ping"})
+b'\x00\x00\x00\x0c\x00\x00\x00\x08\x00P\x00I\x00N\x00G'
+
+```
+"""
+
 import base64
 import json
 import struct

server/protocol/simple_json.py

@@ -1,3 +1,13 @@
+r"""A simple newline terminated JSON data format.
+
+# Example:
+```python
+>>> SimpleJsonProtocol.encode_message({"command": "ping"})
+b'{"command":"ping"}\n'
+
+```
+"""
+
 import json
 
 from .protocol import DisconnectedError, Protocol, json_encoder