API

The node-strtok API is quite small: it has only one method.

strtok.parse(<stream>, <callback>)

• The <stream> argument should be an instance of net.Stream (or any EventEmitter that pumps out data events with a Buffer payload).
• The <callback> should have the signature function (<value>, <typeCb>). The <value> argument is the value of the token just read from the stream, or undefined before anything has been parsed from the stream. This callback should return the type of token to be read next from the stream (e.g. UINT32_BE, an instance of BufferType, etc). The <typeCb> argument is explained in the context of the specal DEFER token type.

The remainder of this documentation gives further details on implementing this second <callback> argument to parse(). It is this callback that ultimately implements the parsing logic for the protocol by consuming values provided to it and deciding what type of value needs to be parsed next.

A simple example

Let's look at an example of a callback that processes a simple protocol for transmitting an array of unsigned 32-bit integers encoded in big-endian format. The wire format is an initial unsigned octet indicating the number of 32-bit values, followed by each value.

var vals = []
var nvals = -1;

function(value) {
    if (value === undefined) {
        return strtok.UINT8;
    }

    if (nvals === -1) {
        nvals = value;
    } else {
        nvals.push(value);
    }

    return (nvals > vals.length) ?
        strtok.UINT32_BE :
        strtok.DONE;
};

Here our callback first checks the value itself, where an undefined value indicates that we haven't parsed anything yet. In that case, we return strtok.UINT8, indicating to strtok.parse() that we wish to read an unsigned octet from the wire and get called back once this is complete.

If our value is not undefined, we have something that was read off of the wire. Our nvals state allows us to interpret what to do with this value: either we haven't yet read the number of elements in our array, in which case that's what this value represents, or we have and this is an array element.

Once we've processed the value, we need to make a decision about what to do next with the stream: do we continue pulling values off of it? or are we done. In the former case, we respond with strtok.UINT32_BE, as this is the type of our payload elements. If we are done, we respond with the special token type strtok.DONE.

Note that we've left out the second argument to our callback, as we don't end up using it.

Token Types

A wide variety of token types are supported out of the box by node-strtok. Consult the token types reference for more information on these types. In particular, note that these types can be used directly (i.e. in third-party code unrelated to strtok.parse()) to parse and generate token values. This is done using the get() and put() methods, respectively. See Direct token usage for more information on direct usage.

Significantly, there are some special tokens change control flow in strtok.parse(), so-called "special tokens".

Special Token Types

There are a handful of "special" token types which have special meaning when returned from the protocol callback: DONE and DEFER.

The DONE token type indicates to strtok.parse() that the protocol parsing loop has come to an end, and that no more data need be read off of the stream. This causes strtok.parse() to stop listening for data events on its stream. The callback will not be invoked again. In addition, upon receiving DONE, strtok.parse() may have excess data buffers which it has pulled off of the stream, but which it did not consume while being directed by the protocol callback. Rather than dropping this data on the floor, strtok.parse() will synthesize and emit data events on the stream which it was operating on. The idea is to facilitate protocol stacks which want to use node-strtok to parse some portion of the protocol and then hand off processing to some other codepath(s). In this case, it is expected that the protocol callback will add data event listeners to the stream immediately before returning DONE.

The DEFER token indicates that the protocol doesn't know what type of token to read from the stream next. Perhaps the protocol needs to consult some out-of-process datastructure, or wait for some other event to occur. To support this case, the protocol callback is invoked with a <typeCb> callback as its second argument. This <typeCb> must be invoked with the desired token type once the protocol layer has figured this out. Note that between the time DEFER is returned and the callback is invoked, strtok.parse() is buffering all data received from the stream.