Tokens

Token type objects are one of the key concepts in node-strtok.

Note that each of these built-in token types supports a put() method for direct usage.

Numerical tokens

node-strtok supports a wide variety of numerical tokens out of the box:

• UINT8
• UINT16_BE, UINT16_LE
• UINT32_BE, UINT32_LE
• INT8
• INT16_BE
• INT32_BE

One might notice that there is no support for 64-bit tokens, since JavaScript seems to limit value size to less than 2^64. Rather than wrapping up an additional math library to handle this, I wanted to stick with JavaScript primitives. Maybe this will change later if this becomes important.

Complex tokens

The token types returned from the protocol callback are simply objects with

• a get() method that takes a Buffer and an offset and returns the token value
• a len field that indicates the number of bytes to consume

Any JavaScript object that meets this criteria can be considered a type and returned from the strtok.parse() callback or provided to the type callback when invoked after returning strtok.DEFER.

node-strtok ships with two built-in types for supporting common behavior that take advantage of this

• BufferType -- consume a fixed number of bytes from the stream and return a Buffer instance containing these bytes.
• StringType -- consume a fixed number of bytes from the stream and return a string with a specified encoding.

Implementing such types is extremely simple. The BufferType implementation is given below:

var BufferType = function(l) {
    var self = this;

    self.len = l;

    self.get = function(buf, off) {
        return buf.slice(off, off + this.len);
    };
};
exports.BufferType = BufferType;