Scalable XMPP bots with erlang and exmpp, part I

Introducing EXMPP

In this series of articles we will introduce exmpp, the long awaited high-performance XMPP library for erlang, released by ProcessOne weeks ago. While doing so, we will learn how to use the library to build a highly scalable XMPP bot. As a library, exmpp is oriented towards high performance and low memory usage, while being general enough to be useful in different scenarios (clients, servers, components).

In this first installment we will learn the basis of exmpp, and start with our bot implementation.

Data structures

When working with exmpp, there are two sets of data structures (exmpp uses erlang records for its data structures) that you will be working with most of the time: the ones that represents XML fragments, and the one that represent JIDs.

#xmlel{}

This record represents an element node in a XML tree. It contains information on the element’s default namespace, tag name, attributes and child nodes. As the time of this writing, the definition of xmlel{} is:

-record(xmlel, {
ns = undefined :: xmlname() | undefined,
declared_ns = [] ,
name :: xmlname(),
attrs = [] :: [xmlattr()],
children = [] :: [#xmlel{} | xmlcdata()] | undefined
}).

declared_ns is used for namespace declarations, useful when you need to know if a specific namespace is defined specifically at a given element, even if that namespace is not used anywhere. Yes, there are such cases . Most of the time, you won’t care about that field. The rest of the fields are self documenting.

#xmlattr{}

As you guessed, this is exmpp’s representation of an XML attribute.

-record(xmlattr, {
ns = undefined :: xmlname() | undefined,
name :: xmlname(),
value :: binary()
}).

#xmlcdata{}

A text node.

-record(xmlcdata, {
cdata = <<>> :: binary()
}).

E. g. this stanza:

<message to='test@conference.example.org' type='groupchat'>
<body>test</body>
</message>

is represented by exmpp as:

[{xmlel,undefined,[],message,
[{xmlattr,undefined,to,<<"test@conference.example.org">>},
{xmlattr,undefined,type,<<"groupchat">>}],
[{xmlel,undefined,[],body,[],[{xmlcdata,<<"test">>}]}]}]

#jid{}

Other heavily used data type is exmpp’s JID representation.

-record(jid, {
orig_jid :: binary() | undefined,  
prep_node :: binary() | undefined,  
prep_domain :: binary() | undefined, 
prep_resource :: binary() | undefined
}).

prep_node, prep_domain and prep_resource fields correspond to each portion of the JID, after being passed through the corresponding stringprep profile.

E.g.

JID EXMPP representation
node@domain/resource
{jid,<<"node@domain/resource">>,<<"node">>,<<"domain">>,<<"resource">>}
Node@DOmain/reSOURCE
{jid,<<"Node@DOmain/reSOURCE">>,<<"node">>,<<"domain">>,<<"reSOURCE">>}
conference.domain
(no node or resource)
{jid,<<"conference.domain">>,undefined,<<"conference.domain">>,undefined}

Basic API

Most of the time, you don’t need to know the exact data representation, as exmpp provides a rich set of APIs to work with, that will make your life much easier. Not only you don’t need to work directly at the data representation level, you are encouraged to not doing so. By using the API you get all the benefit of data encapsulation, and allow your code to easily adapt in the case of a future change in the physical representation.

This is specially true for JIDs (even the definition of the #jid records is in an internal header file that your code isn’t supposed to import), as their use is so pervasive in XMPP entities, than a small improvement on its physical representation can lead to big gains for the entire application. The current representation is a compromise between memory and CPU usage, that was found to work well in our tests, but it isn’t written on stone.

If you browse the exmpp source code, you will note that it is splitted in a set of directories, according to its intended usage target:

  • src/core Contains the core functionality on exmpp. Here you will find functions for XML parsing and serialization, for working with the XML tree, with JIDs, etc.
  • src/client Functions in this module are targeted to build and interpret stanzas generated at the client side of the XMPP protocol.
  • src/server Functions in this module are targeted to build and interpret stanzas generated at the server side of the XMPP protocol.
  • src/network Contains the required functionality to establish network connections between XMPP clients and servers.

Some important modules are:

File Description
core/exmpp.erl Library initialization
core/exmpp_xml.erl XML parsing and manipulation
core/exmpp_jid.erl JID parsing and manipulation
core/exmpp_stanza.erl High level functions applicable to any stanza type
core/exmpp_message.erl High level functions for manipulating <message/> stanzas
core/exmpp_iq.erl High level functions for manipulating <iq/> stanzas
core/exmpp_presence.erl High level functions for manipulating <presence/> stanzas

Look at the exmpp sources for the complete list.

What do you say? Too much reading without opening an erlang console? You are right, let’s start!

1> exmpp:start().
ok
2> M = exmpp_message:normal(<<"Hello world!">>).
{xmlel,'jabber:client',[],message,
[{xmlattr,undefined,type,<<"normal">>}],
[{xmlel,'jabber:client',[],body,[],
[{xmlcdata,<<"Hello world!">>}]}]}
3> T = exmpp_xml:document_to_iolist(M).
[[60,"message",
[[32,"xmlns",61,34,"jabber:client",34],
[32,"type",61,34,"normal",34]],
62,
[[60,"body",[],62,[<<"Hello world!">>],60,47,"body",62]],
60,47,"message",62]]
4> io:format("~s\n", [T]).
<message xmlns="jabber:client" type="normal"><body>Hello world!</body></message>
ok

Then parse it again:

5> [M2] = exmpp_xml:parse_document(iolist_to_binary(T)).
[{xmlel,'jabber:client',
[{'jabber:client',none}],
message,
[{xmlattr,undefined,type,<<"normal">>}],
[{xmlel,'jabber:client',[],body,[],
[{xmlcdata,<<"Hello world!">>}]}]}]
6> io:format("~s\n", [exmpp_xml:document_to_iolist(exmpp_stanza:set_recipient(M2, <<"john@testserver">>))]).
<message xmlns="jabber:client" type="normal" to="john@testserver"><body>Hello world!</body></message>
ok

set_recipient/2 can also work with JID structures:

8> io:format("~s\n", [exmpp_xml:document_to_iolist(exmpp_stanza:set_recipient(M2, exmpp_jid:make(<<"john">>, <<"testserver">>, <<"testresource">>)))]).
<message xmlns="jabber:client" type="normal" to="john@testserver/testresource"><body>Hello world!</body></message>
ok
io:format("~s\n", [exmpp_xml:document_to_iolist(exmpp_stanza:set_recipient(M2, exmpp_jid:parse(<<"john@testserver/testresource">>)))]).
<message xmlns="jabber:client" type="normal" to="john@testserver/testresource"><body>Hello world!</body></message>
ok

At this point you might be wondering why some values are represented as atoms in the xml structure (for example the ‘jabber:client’ namespace, the ‘type’ attribute or the ‘body’ tag). This is the consequence of an important design decision within exmpp. The fact is that exmpp knows more about XMPP than a general purpose XML parser. It knows the ‘jabber:client’ namespace, and it knows that it will be used in all c2s stanzas. So why waste memory repeating it again and again each time?

Exmpp uses erlang atoms to represent those well-known values, thus saving memory and processing time. The set of known tags, namespaces and attributes is taken from the XMPP specification and the published XEPs.

Non-blocking parser

In our examples so far, we were parsing an entire document at one single step. This isn’t of great help for a streaming protocol like XMPP. But that was only a special case, exmpp is built around stream parsers, and prepared to receive its input chunk by chunk:

1> exmpp:start().
ok
2> P = exmpp_xml:start_parser().
{xml_parser,[{max_size,infinity},
{root_depth,0},
{names_as_atom,true},
{emit_endtag,false}],
#Port<0.712>}
3> exmpp_xml:parse(P, <<"<message to='someu">>).
continue
4> exmpp_xml:parse(P, <<"ser@somedomain' type='chat'><body>Hello!</body>">>).
continue
5> exmpp_xml:parse(P, <<"</message>">>).
[{xmlel,undefined,[],message,
[{xmlattr,undefined,to,<<"someuser@somedomain">>},
{xmlattr,undefined,type,<<"chat">>}],
[{xmlel,undefined,[],body,[],[{xmlcdata,<<"Hello!">>}]}]}]

So far, so good. exmpp_xml:start_parser/0 creates a parser using the default options. One of such options is {root_depth, 0}, that instructs exmpp to return the parsed data only after the entire document has been parsed (it returns the element at depth 0, the root element).

We are closer, but still do not have what we need. In XMPP, all stanzas are children of the opening <stream> element, exmpp can’t wait for the ending </stream>. What we really want is to receive entire stanzas, that are at depth 1:

1> P = exmpp_xml:start_parser([{root_depth,1}]).
{xml_parser,[{max_size,infinity},
{root_depth,1},
{names_as_atom,true},
{emit_endtag,false}],
#Port<0.724>}
2> exmpp_xml:parse(P, <<"<stream><message to=">>).
[{xmlel,undefined,[],stream,[],undefined}]
3> exmpp_xml:parse(P, <<"'someuser@somedomain'><body>">>).
continue
4> exmpp_xml:parse(P, <<"Hello!</body></message>">>).
[{xmlel,undefined,[],message,
[{xmlattr,undefined,to,<<"someuser@somedomain">>}],
[{xmlel,undefined,[],body,[],[{xmlcdata,<<"Hello!">>}]}]}]

In the above example, we got the opening stanza as soon as exmpp founds it. Note that the children field is set to ‘undefined’, to inform us that at this point we haven’t parsed the children nodes yet.

In contrast, the <message/> stanza is delivered by exmpp only when the entire stanza has been parsed, as it is at the desired root_depth. Most of the cases this is the desired behavior (besides 0, you can change the root_depth option to any other positive integer, not only 1).

Streams

We have seen how to manipulate and parse stanzas. Exmpp also provides helper modules to establish and interact with XMPP streams, in different scenarios. These modules are in charge of establishing the corresponding network connection, handling authentication, initializing and feeding the parser, and all the basic things that are required by any entity communicating with the XMPP network.

Module Description
network/exmpp_session.erl Client TCP/TLS session
network/exmpp_bosh.erl Client bosh session
network/exmpp_component.erl External component session

In the examples directory of exmpp’s source distribution you will find a module named echo_client.erl. It is a simple bot that connects to the server as a normal user, and echoes back any message you send to him. It is small enough to be included here:

start() ->
spawn(?MODULE, init, []).

stop(EchoClientPid) ->
EchoClientPid ! stop.


init() ->
application:start(exmpp),
%% Start XMPP session: Needed to start service (Like
%% exmpp_stringprep):
MySession = exmpp_session:start(),
%% Create XMPP ID (Session Key):
MyJID = exmpp_jid:make("echo", "localhost", random),
%% Create a new session with basic (digest) authentication:
exmpp_session:auth_basic_digest(MySession, MyJID, "password"),
%% Connect in standard TCP:
_StreamId = exmpp_session:connect_TCP(MySession, "localhost", 5222),
session(MySession, MyJID).

%% We are connected. We now log in (and try registering if authentication fails)
session(MySession, _MyJID) ->
%% Login with defined JID / Authentication:
try exmpp_session:login(MySession)
catch
throw:{auth_error, 'not-authorized'} ->
%% Try creating a new user:
io:format("Register~n",[]),
%% In a real life client, we should trap error case here
%% and print the correct message.
exmpp_session:register_account(MySession, "password"),
%% After registration, retry to login:
exmpp_session:login(MySession)
end,
%% We explicitly send presence:
exmpp_session:send_packet(MySession,
exmpp_presence:set_status(
exmpp_presence:available(), "Echo Ready")),
loop(MySession).

%% Process exmpp packet:
loop(MySession) ->
receive
stop ->
exmpp_session:stop(MySession);
%% If we receive a message, we reply with the same message
Record = #received_packet{packet_type=message, raw_packet=Packet} ->
io:format("~p~n", [Record]),
echo_packet(MySession, Packet),
loop(MySession);
Record ->
io:format("~p~n", [Record]),
loop(MySession)
end.

%% Send the same packet back for each message received
echo_packet(MySession, Packet) ->
From = exmpp_xml:get_attribute(Packet, from, <<"unknown">>),
To = exmpp_xml:get_attribute(Packet, to, <<"unknown">>),
TmpPacket = exmpp_xml:set_attribute(Packet, from, To),
TmpPacket2 = exmpp_xml:set_attribute(TmpPacket, to, From),
NewPacket = exmpp_xml:remove_attribute(TmpPacket2, id),
exmpp_session:send_packet(MySession, NewPacket).

Next Steps

In a future  article we will learn about XMPP bots and external components, and will learn how to build a highly scalable bot powered by erlang and exmpp.