ProcessOne Blog Articles about realtime systems: XMPP MQTT SIP Swift Go Kotlin and more! 2020-02-19T12:49:12Z https://blog.process-one.net/feed/ WordPress Mickaël Rémond <![CDATA[ejabberd on macOS with Homebrew package manager]]> https://blog.process-one.net/?p=6474 2020-02-19T12:49:12Z 2020-02-19T12:49:12Z Homebrew is now the de facto package manager for open source software on macOS. Thus, we have discontinued our macOS binary installer to focus on improving further ejabberd experience on macOS by polishing Homebrew ejabberd package.

We recently updated ejabberd on Homebrew and we will keep it in sync with ejabberd official releases. As of today, if you install ejabberd with Homebrew, you will always get the latest release.

If you have any suggestion on how to make ejabberd with Homebrew experience better, please open a Github issues, with the tag Packaging:Homebrew.

We have not yet tested it with Homebrew on Linux or on WSL, but still, pleae submit issues if you find some.

Here is the cheat sheet regarding ejabberd management with Homebrew.

Installing ejabberd

The following command will install ejabberd and its dependencies:

$ brew install ejabberd

Starting ejabberd and launching it on macOS start

brew services start ejabberd

Starting ejabberd

Alternatively, the following command start ejabberd in background, with trying to restart it automatically after reboot:

$ /usr/local/sbin/ejabberdctl start

You can check the status with the command:

$ /usr/local/sbin/ejabberdctl status
The node ejabberd@localhost is started with status: started
ejabberd 20.01 is running in that node

If ejabberd is not running the status command will display a nodedown error:

$ /usr/local/sbin/ejabberdctl status
Failed RPC connection to the node ejabberd@localhost: nodedown

Commands to start an ejabberd node:
  start      Start an ejabberd node in server mode
  debug      Attach an interactive Erlang shell to a running ejabberd node
  iexdebug   Attach an interactive Elixir shell to a running ejabberd node
  live       Start an ejabberd node in live (interactive) mode
  iexlive    Start an ejabberd node in live (interactive) mode, within an Elixir shell
  foreground Start an ejabberd node in server mode (attached)

Optional parameters when starting an ejabberd node:
  --config-dir dir   Config ejabberd:    /usr/local/etc/ejabberd
  --config file      Config ejabberd:    /usr/local/etc/ejabberd/ejabberd.yml
  --ctl-config file  Config ejabberdctl: /usr/local/etc/ejabberd/ejabberdctl.cfg
  --logs dir         Directory for logs: /usr/local/var/log/ejabberd
  --spool dir        Database spool dir: /usr/local/var/lib/ejabberd
  --node nodename    ejabberd node name: ejabberd@localhost

Stopping ejabberd

/usr/local/sbin/ejabberdctl stop

Configuration file

To change the configuration file, you can edit /usr/local/etc/ejabberd/ejabberd.yml

If you have upgraded ejabberd and want to apply possible change to config file, you can check the new ejabberd.yml default file ejabberd.yml-new in the same directory.

Checking the logs

To troubleshoot ejabberd, you can check the logs in the directory /usr/local/var/log/ejabberd/

]]>
0
Mickaël Rémond <![CDATA[Your Development Infrastructure Needs Green & Lean Computing]]> https://blog.process-one.net/?p=6467 2020-02-06T14:02:06Z 2020-02-06T14:02:06Z Software energy consumption is on the rise. The Internet and its online services now account for more than 7% of the planet’s electric consumption. Usage continues to grow extremely rapidly while energy consumption increases almost linearly with usage.

As developers, it’s not possible to ignore the problem of sustainability. As a company providing online applications and services for users, we must also make a commitment to reverse the trend.

However, CIOs are currently focusing on the use of data centres supplied with more renewable energy and limiting the usage by users. But in order to have a real impact, the scope of action must be extended:

  • to the design of applications.
  • to the choice of energy-efficient applications.
  • to development approaches that integrate energy issues from the design stage, with partners who are aware of this process.

How can you reduce the energy footprint of your application?

ProcessOne introduces a new offering aimed at helping you to act on several dimensions on the path towards the real Green IT. This new service consists of three elements:

  1. Benchmark and Diagnostic. In order to define a remediation strategy, it’s critical to measure the energy efficiency of an application and to map the most energy consuming areas.

  2. Optimization. This is an iterative process, carried out with the client’s development team.

  3. Measurement and Iteration. The benchmark must be repeated after optimization to measure the optical impact in terms of gross performance, energy balance and operating cost reduction.

Where to start?

First, learn more about our Green IT initiative and offering. Then, feel free to contact us and chat how we can make your IT systems more environmentally sustainable!

]]>
0
Jérôme Sautret <![CDATA[ejabberd 20.01]]> https://blog.process-one.net/?p=6451 2020-02-10T23:36:57Z 2020-01-29T14:30:40Z We are pleased to announce ejabberd version 20.01. This release focuses on OAUTH support, logging performances improvement and configuration helper, as well as the usual various bug fixes.

New Features and improvements

OAUTH

We are now supporting OAUTH to authenticate with the embedded MQTT server.

This will only work for MQTT 5.0 connections. A client MUST set “Authentication Method” property of CONNECT packet to “X-OAUTH2” and MUST set the token in “Authentication Data” property of the same CONNECT packet. The server responses as usual with CONNACK.

On the server, you just have to enable OAUTH authentication like for XMPP, as described in the post about understanding ejabberd OAuth support.

Logger

We are now using the new logger introduced in OTP22, instead of lagger. This should greatly improve performance of the logging infrastructure.

Configuration

A new configuration parser/validator was added, to check ejabberd.yml syntax and find configuration issues.

SCRAM support

The routine to convert password to SCRAM format now works for all backends, not only SQL.

CockroachDB

We’ve started to add to support to use CockroachDB as database backend.

Technical changes

Database schema changes

To migrate from 19.09 to 20.01, you have to use the following commands on your existing database, after you’ve made a backup of it:

MySQL

If you are using the legacy mysql.sql schema:

ALTER TABLE  oauth_client CHANGE `client` `client_id` text PRIMARY KEY;
ALTER TABLE  oauth_client CHANGE `secret` `client_name` text NOT NULL;

If you are using the newer mysql.new.sql schema:

CREATE TABLE oauth_client (
   client_id varchar(191) NOT NULL PRIMARY KEY,
   client_name text NOT NULL,
   grant_type text NOT NULL,
   options text NOT NULL

PostgreSQL

CREATE TABLE oauth_client (
    client_id text PRIMARY KEY,
    client_name text NOT NULL,
    grant_type text NOT NULL,
    options text NOT NULL
);
ALTER TABLE oauth_client RENAME COLUMN client TO client_id;
ALTER TABLE oauth_client RENAME COLUMN secret TO client_name;

Download and install ejabberd 20.01

The source package and binary installers are available at ProcessOne. If you installed a previous version, there are no additional upgrade steps, but as a good practice, plase backup your data.

As usual, the release is tagged in the Git source code repository on Github. If you suspect that you’ve found a bug, please search or fill a bug report in Issues.


Full changelog
===========

* New features
– Implement OAUTH authentication in MQTT
– Make logging infrastructure use new logger introduced
in Erlang (requires OTP22)
– New configuration parser/validator
– Initial work on being able to use CockroachDB as database backend
– Add gc command
– Add option to disable using prepared statements on Postgresql
– Implement routine for converting password to SCRAM format for all backends not only SQL
– Add infrastructure for having module documentation directly in individual module source code
– Generate man page automatically
– Implement copy feature in mod_carboncopy

* Fixes
– Make webadmin work with configurable paths
– Fix handling of result in xmlrpc module
– Make webadmin work even when accessed through not declared domain
– Better error reporting in xmlrpc
– Limit ammount of results returned by disco queries to pubsub nodes
– Improve validation of configured JWT keys
– Fix race condition in Redis/SQL startup
– Fix loading order of third party modules
– Fix reloading of ACL rules
– Make account removal requests properly route response
– Improve handling of malformed inputs in send_message command
– Omit push notification if storing message in offline storage failed
– Fix crash in stream management when timeout was not set

]]>
0
Remi Corniere <![CDATA[go-xmpp v0.4.0]]> https://blog.process-one.net/?p=6440 2020-01-30T09:58:13Z 2020-01-14T22:20:18Z A new version of the go-xmpp library, which can be used to write XMPP clients or components in Go, has been released. It’s available on GitHub.

You can find the full changelog here : CHANGELOG
Some noteworthy features are the support of three new extensions :
– XEP-0060: Publish-Subscribe
– XEP-0050 : Ad-Hoc Commands
– XEP-0004: Data Forms
for both component and client.
Callbacks for error managment were also added to the client and component.

New Extensions

PubSub

Let’s create a very simple client that is both owner of a node, and subscribed to it.
This example assumes that you have a locally running jabber server listening on port 5222, like ejabberd.
First, let’s create the client. We need to provide a user JID, our identity, then the address of the server we wish to connect to.
Let’s also get a little ahead and put the node name and the service name in our constants :

const (
  domain = "mycomponent.localhost"
  address = "localhost:8888"
  nodeName = "example_node"
  serviceName = "pubsub.localhost"
)

Now, we need to fill a Config struct, that will be passed to the NewClient method :
“`go
config := xmpp.Config{
TransportConfiguration: xmpp.TransportConfiguration{
Address: serverAddress,
},
Jid: userJID,
Credential: xmpp.Password(“pass123”), // For the sake brievety
Insecure: true,
}

]]>
0
Mickaël Rémond <![CDATA[Towards Lean Computing: Integrating Energy Consumption in Application Design]]> https://blog.process-one.net/?p=6431 2020-01-10T14:27:19Z 2020-01-10T14:13:47Z In the quest for better energy efficiency, the applications themselves are often overlooked. Our industry likes to pretend that all technologies are equivalent and has forgotten that the choice of programming languages, frameworks and architecture have a major impact on energy consumption.

The observation

Today we are all aware of the environmental impact of the objects, tools, devices we use. We all know that when we use our car, this usage will have a “carbon” impact. A whole industry is working to reduce the energy impact of the appliances we use. Our environmental awareness is leading us to moderate our use, by using appliances that consume less energy, or depend on a mix with more renewable energies. Finally, we even consider alternatives, such as replacing the car with a public transit, or cycling.

Taking into account the carbon footprint of our actions is more evident in the transport field. In the IT sector, we are only at the beginning, but our massive use of computing power and data turned the Internet into a major energy drain.

In 2017, Greenpeace considered Internet electric consumption to represent 7% of the world’s electricity consumption. The report is based on 2016 data. It is very likely that we are well beyond that point today. The growth in data traffic exceeds the advances in energy efficiency of network equipment. The “Low Tech Magazine” explains the problem very well: About Low Tech Magazine.

In other words, it means that all the world’s IT departments, phones, data centers, etc. emit more carbon than air transport, for example.

Keeping Internet infrastructure and usage under control

The awareness is growing, but the solutions being explored focus on only two lines of action, forgetting a third but fundamental one.

The first line of action on the energy consumption of the Internet is geared at infrastructure. The energy consumption of the infrastructure is dealt with at two levels:

  1. Reducing energy consumption in data centers. One example is OVH’s work on hydraulic cooling systems, replacing both the fans in the servers and the air conditioning in the data centres.
  2. The source of electric supply. An effort is being made by large groups such as Facebook, or Google to buy renewable electricity, i.e. electricity produced from renewable energy.

The second line of action to reduce the energy impact is usage limitation. From a user’s point of view, we have all seen awareness-raising campaigns recommending that people decrease their use of e-mail or even reduce the resolution of videos consumed to save bandwidth and electricity. This is an important step forward, but we must now go beyond it, to reduce the energy impact of everyday use.

Designing lightweight and efficient applications

Cloud computing has changed the way applications are designed. It’s an industry, with most of the major players now being business infrastructure providers. Cloud players offer increasingly sophisticated services and have simplified the use of online capabilities.

So much so that it is common today to design architecture that can scale-up automatically. It is even commonly accepted that human time is the most expensive and valuable resource. Consequently, it is better to put more and bigger servers to run one’s service than to spend time optimizing it. The environmental cost of time-to-market pressure is real. Cloud computing has made it too easy to leverage ever-increasing resources.

In this race to consume more and more, our industry has forgotten that this attitude is a major contributor to greenhouse gas emissions. This attitude is the IT equivalent of intensive agriculture. We need a new approach.

A third line of action: Measure and optimize

Another line of action to reduce the impact of the Internet on the planet is to rediscover optimization know-how, to be well aware of the impact of architecture decisions on our applications.

Many parameters have to be taken into account and can have a real impact on the environment:

  1. The type of architecture. Some architectures are more efficient than others. Intensive use of middleware and microservices can have a negative impact on the energy balance of a service.
  2. The panel of technologies. We have to be realistic, some technologies are convenient for development, but very energy consuming. For example, at ProcessOne, we have taken over the management of a messaging service on our infrastructure. The application was previously running on 30 servers. We now operate it on 4 servers of the same size. The choice of implementation technology makes a huge difference (i.e. Erlang, Elixir, Go).
  3. The set of features. Big data, the massive mining of data, is a great promise of wealth creation. However, the business case does not not always exist and storing and processing huge volumes of data just in case you may need it doesn’t seem rational. You have to think ahead about what you want to analyze and why. This approach is not really compatible with the General Data Protection Regulations (GDPR).

Conclusion

At the end of the day, companies have an important responsibility in the energy consumption of the Internet. They cannot be done with it by just shifting the burden to infrastructure providers and users. They have to take their share of the responsibility.

If you want to know how to address the problem, ProcessOne can certainly help you, by working on the whole process, and by helping you benchmark your application to monitor its performance.

So, are you ready to tackle the challenge with me and reducing the energy footprint of your applications?

]]>
0
Mickaël Rémond <![CDATA[ProcessOne: What to expect in 2020?]]> https://blog.process-one.net/?p=6420 2019-12-23T08:51:05Z 2019-12-23T08:51:05Z As this is the end of the year, I’d like to take the opportunity to share with you how ProcessOne has changed over time and how it is approaching the next decade.

ProcessOne was created to simplify the implementation of large-scale, real-time messaging services. In 2005, the most advanced tool was clearly Erlang. Since then, several technological revolutions have taken place, including the rise of the cloud and mobile.

We have been involved in these revolutions in our own way and built innovative projects for almost two decades, with some major milestones, including ejabberd development and some major customer projects.

ProcessOne Major Milestones

Gradually, we invested more extensively in the real-time domain and extended our technical scope (Erlang, then Elixir, Go and Swift).

But, in the end, it is our capacity for innovation as a whole that brings most value for our customers. This is how we have been led to develop a process and new expertise: Consulting, Data Science, and Mobile Development. We are now able to iterate on an objective and help our customers to imagine their next innovation, develop it and take it over on their own after the production launch.

ProcessOne Innovation offering

In the meantime, the time to move into the next decade is approaching smoothly.
I wish you a wonderful end of the year.

— Mickaël Rémond, ProcessOne Founder

]]>
0
Mickaël Rémond <![CDATA[SwiftNIO Workshop in Paris at dotSwift Conference]]> https://blog.process-one.net/?p=6413 2019-12-12T14:31:47Z 2019-12-12T14:31:47Z

I have partnered with the good folks at dotConferences to put together some contents for a SwiftNIO workshop. The workshop will take place in Paris on February the 4th, the day after the dotSwift conference.

If you are coming to Paris for the dotSwift conference, you can make the best of your trip by also attending my SwiftNIO workshop. If you are interested by Swift Server-side or Swift cross platform development, this a good opportunity to get you started.

What you’ll learn

SwiftNIO involves many advanced concepts, that may not always be obvious for Swift developer. The concept have been refined during the 10+ years of Netty development, so despite SwiftNIO being still quite young, it comes with a steep learning curve.

The workshop will guide you through the concepts behind SwiftNIO and the architecture of the project. From there it will show how to build you client library framework as a Swift Package and how to test it on Linux. Finally, it will show how to write a mini server-side framework. The example will be illustrated with a simple but real protocol for Messaging: STOMP (Simple Text Oriented Messaging Protocol).

The workshop will covers the following topics:

  • Concepts & principles of Swift NIO
  • Designing networking application: The role of threading & async / await
  • Understanding Futures & Promises
  • Going deeper into async code: Mastering the functional operators
  • Leveraging SwiftNIO provided components for HTTP
  • Building a simple client library (illustrated by STOMP protocol)
  • Building a simple server-side framework (illustrated by STOMP protocol)

You can learn more and book your workshop here: https://www.dotswift.io/training-swiftnio

]]>
0
Jérôme Sautret <![CDATA[go-xmpp 0.3.0]]> https://blog.process-one.net/?p=6401 2020-01-09T12:45:09Z 2019-11-29T16:08:20Z A new version of the go-xmpp library, which can be used to write XMPP clients or components in Go, as been released. It’s available on GitHub.

Upon new features, it adds a websocket transport. For this reason, the minimum go version to use it is now 1.13. It also adds a SendIQ method, to send iq stanza and receive the response asynchronously on a channel.
On the component side, it fixes a SIGSEGV in xmpp_component (#126) and adds more tests for the Component code.

A small example

Writing an XMPP component

Speaking of components, here is a simple example on how to create a simple one. As a reminder, components are external services that can communicate with an XMPP service using the Jabber Component Protocol as described in XEP-0114.

A component has its own XMPP domain and must know the server address and service port:

     const (
        domain  = "mycomponent.localhost"
        address = "localhost:8888"
    )

The options needed when creating a new component are defined as follow (secret must match the one define in the server config):

    opts := xmpp.ComponentOptions{
        TransportConfiguration: xmpp.TransportConfiguration{
            Address: address,
            Domain:  domain,
        },
        Domain:   domain,
        Secret:   "secret",
    }

To actually create the simplest component, just create a default route, and pass it to NewComponent, as well as the above options:

    router := xmpp.NewRouter()
    c, err := xmpp.NewComponent(opts, router)

Connect establishes the XMPP connection to the server, and authenticates with it:

      err := c.Connect()

Now we can try to send a disco iq to the server:

       iqReq := stanza.NewIQ(stanza.Attrs{Type: stanza.IQTypeGet,
        From: domain,
        To:   "localhost",
        Id:   "my-iq1"})
    disco := iqReq.DiscoInfo()
    iqReq.Payload = disco

In order to get the response asynchronously, the sendIq will return a channel where we expect to receive the result iq. We also need to pass it a context to set a timeout:

    ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
    res, _ := c.SendIQ(ctx, iqReq)

Now we just have to wait for our response:

    select {
    case iqResponse := <-res:
        // Got response from server
        fmt.Print(iqResponse.Payload)
    case <-time.After(100 * time.Millisecond):
        cancel()
        panic("No iq response was received in time")
    }

Full example

The full program that runs a component, connects it to a XMMP server and perform a disco on it to display the result is thus as is:

package main

import (
    "context"
    "fmt"
    "time"

    xmpp "github.com/FluuxIO/go-xmpp"
    "gosrc.io/xmpp/stanza"
)

const (
    domain  = "mycomponent.localhost"
    address = "build.vpn.p1:8888"
)

// Init and return a component
func makeComponent() *xmpp.Component {
    const (
        domain  = "mycomponent.localhost"
        address = "build.vpn.p1:8888"
    )
    opts := xmpp.ComponentOptions{
        TransportConfiguration: xmpp.TransportConfiguration{
            Address: address,
            Domain:  domain,
        },
        Domain: domain,
        Secret: "secret",
    }
    router := xmpp.NewRouter()
    c, err := xmpp.NewComponent(opts, router)
    if err != nil {
        panic(err)
    }
    return c
}

func main() {
    c := makeComponent()

    // Connect Component to the server
    fmt.Printf("Connecting to %v\n", address)
    err := c.Connect()
    if err != nil {
        panic(err)
    }

    // make a disco iq
    iqReq := stanza.NewIQ(stanza.Attrs{Type: stanza.IQTypeGet,
        From: domain,
        To:   "localhost",
        Id:   "my-iq1"})
    disco := iqReq.DiscoInfo()
    iqReq.Payload = disco

    // res is the channel used to receive the result iq
    ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
    res, _ := c.SendIQ(ctx, iqReq)

    select {
    case iqResponse := <-res:
        // Got response from server
        fmt.Print(iqResponse.Payload)
    case <-time.After(100 * time.Millisecond):
        cancel()
        panic("No iq response was received in time")
    }
}

]]>
0
Mickaël Rémond <![CDATA[Building Realtime Streaming Architectures]]> https://blog.process-one.net/?p=6389 2019-11-25T17:28:59Z 2019-11-26T09:28:41Z Realtime is not only about client interactions. We have been using XMPP & MQTT since a long time to connect people and things together.

However, there is another use case for realtime that is a little less known: realtime streaming architectures. This is a design pattern that you can use to make the core of your applications or your information system realtime.

The principle is simple: you decouple components into isolated services that talk to each other through a data bus, very often Kafka, using the publish & subscribe pattern.

This approach is very powerful, as you can tolerate a component being down for a short maintenance, with the confidence that the system will catch up. The pattern is also useful to tune the computing capacity at each stage of the processing pipeline. And finally, the pattern is also popular to decouple services and make them able to evolve independently.

This is just the technical benefits, but from a high level business perspective, it simply means that you get a more resilient system, that is able to process events in realtime, with a system serving as a basis of new innovative offerings for customers. This leads to a significant business edge.

I had already presented back in 2017 at DotGo Conference what we we had been doing, at a technical level, to build realtime streaming architectures:

Today, we are able to share more by publishing a case study, showing the type of architecture we have been building for Colissimo, a leading postal service in France. Go ahead and read Colissimo case study.

Do not hesitate to contact us, if you need guidance on how to build, improve, troubleshoot or rework such an architecture.

]]>
0
Marek Foss <![CDATA[Real-time Radar #27]]> https://blog.process-one.net/?p=6005 2019-11-21T10:33:37Z 2019-11-21T10:33:37Z ProcessOne curates the Real-time Radar – a newsletter focusing on articles about technology and business aspects of real-time solutions. Here are the articles we found interesting in Issue #27. To receive this newsletter straight to your inbox on the day it is published, subscribe here.

GopherCon 2019 Highlights

ProcessOne CEO had the chance to attend once more the big main annual Go conference: Gophercon 2019. For the first time this year, location changed from Denver to San Diego. Ocean, mild climate and extraordinary venue for the party (on the USS Midway) contributed to a relax and friendly atmosphere.

The State of the XMPP Community

This keynote gives an overview of what various XMPP projects have been doing (implementation wise) over the last ~12 month and will try to explain why some features have been adopted more quickly than others.

Cisco Scores Big with a New IETF-Approved Internet Standard

This June, Cisco achieved a milestone when the Internet Engineering Task Force (IETF) declared their XMPP-Grid architecture an official Internet standard?for security information exchange.

The Death of Jabber

Looking at the two previous links, it seems open XMPP and Cisco XMPP are going forward quite well – albeit in different directions. However, here’s an opposite point of view.

Deep Dive Into MQTT

Message Queuing Telemetry Transport (MQTT) has been relevant for years. It’s enjoying even wider attention now thanks to the explosive growth of IoT, with both consumer and industrial economies deploying distributed networks, edge computing and data-emitting devices as part of everyday operations.

Machine Vision Meets MQTT Messaging

This author covered recognizing objects with the JeVois sensor and sending the data to text-to-speech Processing scripts. Next, they describe passing the recognized object name and location data to other machines and systems, using MQTT.

]]>
0