nmsg documentation

Introduction

The NMSG format is an efficient encoding of typed, structured data into payloads which are packed into containers which can be transmitted over the network or stored to disk. libnmsg is the reference implementation of this format and provides an extensible interface for creating and parsing messages in NMSG format. The NMSG format relies on Google Protocol Buffers to encode the payload header. Individual NMSG payloads are distinguished by assigned vendor ID and message type values and libnmsg provides a modular interface for registering handlers for specific message types. libnmsg makes it easy to build new message types using a Protocol Buffers compiler.

See also

http://code.google.com/p/protobuf/

http://code.google.com/p/protobuf-c/

---

libnmsg

libnmsg provides a reference C implementation of an NMSG parser and generator. It contains core functions for reading and writing NMSG units and can be extended at runtime by plugin modules implementing new message types.

Core I/O functions

input.h and output.h provide the single-threaded input and output interfaces. io.h provides a multi-threaded interface for multiplexing data between inputs and outputs. message.h provides an interface for creating and inspecting message payloads.

nmsg_msg message module interface

The nmsg_msg message module interface is implemented by msgmod.h. Plugins in external shared objects provide an nmsg_msgmod_t structure in order to implement new message types.

Auxiliary functions

• alias.h
• asprintf.h
• chalias.h
• ipdg.h
• pcap_input.h
• rate.h
• strbuf.h
• timespec.h
• zbuf.h

---

Wire format

An NMSG unit consists of a small fixed-length header which precedes a variable length part that may contain one or more payloads or, if a single payload is too large, the variable length part may contain a single message fragment.

NMSG is designed for transport over UDP sockets or storage in on-disk files. Individual UDP datagrams may transport only a single NMSG unit, while the file format is simply a series of NMSG units concatenated together. The variable length part of an NMSG unit transported over UDP is usually much smaller than those stored on disk (NMSG_WBUFSZ_MAX versus NMSG_WBUFSZ_JUMBO or NMSG_WBUFSZ_ETHER). NMSG units stored on disk also do not contain message fragments.

The fixed-length NMSG header is ten octets long and consists of a magic value, a bit field of flags, a protocol version number, and the length of the variable length data part.

The variable length data part is interpreted as a Protocol Buffer message.

Octet 0-3	Octet 4	Octet 5	Octet 6-9	Remainder
Magic	Flags	Protocol Version	Length	Data

Magic value

The magic value (NMSG_MAGIC) is always the four octet sequence 'N', 'M', 'S', 'G'.

Flags

This is a bit field of flags. Currently two values are defined. NMSG_FLAG_ZLIB indicates that the data content has been compressed. NMSG_FLAG_FRAGMENT indicates that the data content starts a special fragmentation header.

NMSG_FLAG_ZLIB

This flag indicates that zlib compression has been applied to the variable length part. If the NMSG_FLAG_FRAGMENT flag is not also set, then the entire variable length part should be deflated with zlib and interpreted as an NmsgPayload.

NMSG_FLAG_FRAGMENT

This flag indicates that the variable length part should be interpreted as an NmsgFragment. After reassembly, the data should be interpreted as an NmsgPayload. If the NMSG_FLAG_ZLIB flag is also set, then the reassembled data should be deflated and then interpreted as an NmsgPayload.

Note that when creating a compressed, fragmented NMSG unit, compression should be applied before fragmentation.

Protocol Version

This value (NMSG_PROTOCOL_VERSION) is currently 2.

Length

This value is an unsigned 32 bit integer in network byte order indicating the length in octets of the variable length data part.

Data

The variable length data part is encoded using Google Protocol Buffers. The file nmsg/nmsg.proto in the source distribution describes the two message types Nmsg and NmsgFragment that can appear in NMSG units:

syntax = "proto2";
package nmsg;
 
message Nmsg {
    repeated NmsgPayload    payloads = 1;
    repeated uint32         payload_crcs = 2;
    optional uint32         sequence = 3;
    optional uint64         sequence_id = 4;
}
 
message NmsgFragment {
    required uint32         id = 1;
    required uint32         current = 2;
    required uint32         last = 3;
    required bytes          fragment = 4;
    optional uint32         crc = 5;
}
 
message NmsgPayload {
    required uint32         vid = 1;
    required uint32         msgtype = 2;
    required int64          time_sec = 3;
    required fixed32        time_nsec = 4;
    optional bytes          payload = 5;
    optional uint32         source = 7;
    optional uint32         operator = 8;
    optional uint32         group = 9;
}

If no flags are set, then the data part is an Nmsg protobuf message. If only the NMSG_FLAG_ZLIB flag is set, then the data part is a zlib compressed Nmsg protobuf message. The Nmsg protobuf message is a container message for one or more NmsgPayload messages. If only the NMSG_FLAG_FRAGMENT flag is set, then the data part is an NmsgFragment protobuf message.

Nmsg and NmsgPayload protobuf messages

Nmsg messages contain one or more NmsgPayload messages.

The vid field of NmsgPayload messages is the vendor ID. The currently defined vendor IDs are listed in vendors.h and assigned by Farsight. The msgtype field is a vendor-specific value and together the (vid, msgtype) tuple defines the type of the data contained in the payload field.

The time that the data encapsulated in the payload field was generated is stored in the time_sec and time_nsec fields. The number of nanoseconds since the Unix epoch is split across the two fields, with the integer number of seconds stored in the time_sec field and the nanoseconds part stored in the time_nsec field.

The source, operator, and group fields are optional fields that can be used by cooperating senders and receivers to classify the payload.

NmsgFragment protobuf messages

NmsgFragment messages are used to encapsulate fragments of an Nmsg message if the serialized message is too large for the underlying transport. This enables NMSG payloads to avoid the size restrictions of UDP/IP transport.

If a serialized Nmsg message is too large for the underlying transport, it is split into fragments which are carried in the fragment field of a sequence of NmsgFragment messages. A random value is selected for the id field of these fragments. The 0-indexed current field indicates the ordering of the fragments, and the last field indicates the total number of fragments. Once a receiver has received all the fragments for a given fragment id, the fragmented payload should be extracted from the payload field of each fragment and concatenated into a buffer.

If the sender performed compression of the Nmsg message before fragmentation, then all fragments should have the NMSG_FLAG_ZLIB field set and the receiver must perform decompression of the reassembled buffer. The result of decompression should be interpreted as an Nmsg message.

If the sender did not perform compression before fragmentation, then the buffer should be directly interpreted as an Nmsg message.