Introducing the XMPP application server: The Twitter example

ejabberd is probably the first XMPP application server ever released. This article introduces this new concept and explains how to build a distributed event-based infrastructure for social networking. The example application that illustrate this article is a distributed Twitter-like microblogging platform.

Introduction

Since I designed the first version of the pluggable pubsub module for ejabberd in early 2007, I had in mind to turn it into a powerful application server. I have already blogged about the power of the new API of ejabberd 2.0 pubsub engine. However, this single article does not do any justice to this idea and how it can change the face of the web.

Customizable services based on pubsub is XMPP (eXtensible Messaging and Presence Protocol) at its full speed. The XMPP protocol has been designed since the beginning to be a near real time routing / distribution engine. A piece in the puzzle was missing however. This is where I think our new plugin based pubsub API fills the gap and turn ejabberd into the first XMPP application server.

As a proof of concept and as a way to robustify our API, we have written the Personal Eventing via Pubsub (PEP) as a plugin of our pubsub engine. It is a big specification and should clearly show the potential of the framework.

The XMPP application server: The Twitter case

The PEP implementation was still not enough to demonstrate that ejabberd is a fault-tolerant, highly scalable application development platform for the Web 2.0 era. I decided to start a serie of articles demonstrating how to build a distributed Twitter-like microblogging platform based on ejabberd 2.0.

Step 1

The goals of the first development step are the following:

  • Write a new ejabberd service as a pubsub extension: this is a XMPP Twitter-like application code itself.
  • Show that you can integrate with other pieces of the XMPP framework and not only the event distribution: The code uses the service browser as a user interface, building a fully dynamic hierarchy.
  • Write the near realtime publishing / distribution code. To show how powerfull it is, I wrote the code in a way that node creation and subscriptions are not necessary. You only have to publish content and it is distributed to anyone that can see your presence. The end result, like PEP, is closely integrated to the user roster.

The architecture is distributed across as many user base (domain) as you want: A user can get events from users having their account in a large number of different servers.

Writing a new plugin to define custom node hierarchy

I wanted to build a special service hierarchy so that the application integrates nicely in XMPP service discovery. I wanted a dynamic discovery tree whose content was depending on who was doing the query. This would make the application more user friendly and the example much more impressive.

To write a custom node hierarchy with our API, you simply have to write a nodetree plugin. I have written the nodetree_twitter module to handle our special hierachy looking like the following:

  • Root node: “twitter”
  • Special node: MyStream. This is a collection displaying your latest published posts.
  • Special node: FriendsStream. It contains the latest posts of my friends.
  • Special node: MyFriends. It contains the list of your friends and allow you to see the latest posts of a given contact.

Friends are taken from the instant messaging contacts list: I defined this friends list as “the people that already share their presence with me”. This way users do not have to manage the cumbersome subscription process specially for this application: Their friends from their contact list are cleverly reused.

Note that the node hierarchy is not defined in the Pubsub specification (XEP-0060), but is totally needed to make the application flexible. This pubsub node hierarchy does not violate the Pubsub specification, so we are fine with it.

This is the result as shown by Psi’s service discovery::

Psi: Service Discovery

The contact Blaine (thank you for your comments, by the way) is coming directly from my roster.

Posting entries

Posting an event is simply sending a small piece of XML code in ejabberd. As Psi does not support publishing events to our special virtual nodes (yet ;)), we will send it from Psi XML console.

For example, I can send the following piece of code in the XML console. I publish the post on my special pubsub node MyStream:

<iq type='set'
to='pubsub.localhost'
id='publish1'>
<pubsub xmlns='http://jabber.org/protocol/pubsub'>
<publish node='/twitter/MyStream'>
<item>Demoing my Twitter-like code</item>
</publish>
</pubsub>
</iq>

The message is received instantly by all the contacts that have subscribed to my presence. No polling mechanism is needed. This is the beauty of using ejabberd XMPP App Server.

The received message is the following:

<message from="pubsub.localhost" to="test1@localhost" >
<event xmlns="http://jabber.org/protocol/pubsub#event">
<items node="/twitter/FriendsStream" >
<item id="4B2C8277256F" >Demoing my Twitter-like code</item>
</items>
</event>
</message>

Note that the dynamic hierarchy is very powerful: We publish on our special node. The message seems to come from another special node on the other end, for the sake of consistency.

Persistence

I did nothing special for persistence of the published items. It is handled as a default by the ejabberd pubsub framework.

We can browse our new service to check that the blog post has been actually published:

Psi: Service discovery

I did not yet write the part of the code to browse friends posts. This will be part of a second article.

Conclusions

The end result is the basis of a Twitter-like application. It is fully distributed at the web scale. You can run this application on different servers and get events from people with account on different servers. It is compliant with ejabberd clustering and high scalability features. Most of all, thanks to our Pubsub API, I wrote this first code in 3 hours. It tooks something like 200 to 300 lines of code to write this extremely powerful infrastructure. Yes, that’s amazing, I agree, as I was surprised myself by how easy and fast it was to write !

After a bit of clean-up, the code will be added to ProcessOne contributions Subversion repository. If you are impatient to play with the code, please drop me a mail :)

In a next article, I will present the second stage of this application. I will add better and complete browsing of the items and publish directly for the service browser thank to adhoc commands supported in Psi.

In the meantime, I hope to get this experiment running with client developers: It would be very nice see the messages received properly displayed in OneTeam and in Psi. Stay tuned !

digg_url = ‘http://www.process-one.net/en/blogs/article/introducing_the_xmpp_application_server/';