Hello everyone. In this post I will try to provide you with a quick example on how to implement push notifications for your Android app using MQTT protocol. I will NOT discuss here why an application might need push notifications or the advantages of Push over Pull. I assume that you know exactly what I mean by push notifications are and why you might need them. However, before jumping in straight to the good stuff, let’s go over how it all started.
Introduction
It’s been around 4 months now since I’ve started developing apps on the Android platform. It began with me scoring a free Nexus One phone at one of the Android Developer Labs. Obviously, I couldn’t resist trying to hack around with some code, so I downloaded the SDK and dove in. I guess in some sense, that’s exactly what Google was hoping for when they starting giving out free phones. While it might sound like I got lucky, in the end Google is the one who won.
Anyway, developing for the Android platform turned out to the a pleasure. The SDK was easy to setup, easy to use and and easy to understand. Putting together your first app was a breeze. I was very impressed.
Unfortunately, I soon realized that Android is not perfect. One of the things that really disappointed me was the lack of a native method for performing push notifications. Over the past year push notifications became almost a standard in the mobile space thanks to Apple. Even though BlackBerry utlilized Push since god knows when, it was Apple that really brought Push mainstream. Obviously, lack of native Push on Android seems like a huge drawback. Naturally, I started looking around for a solution. After Googling through dozens and dozens of blogs and message boards, I’ve realized that there are 3 generally accepted ways to implement push notifications for your Android app. All of which are non-trivial, hacky and have their own disadvantages. Let’s go over the list:
- Poll? The name obviously tells you that it’s really not even push. The idea here is to periodically poll the server for new messages from a background local or remote service. The more often you poll the closer you get to the real-time push.
- SMS Android allows you to intercept SMS messages. Your server sends a specially encoded SMS to your phone, whenever there is something new. Your app intercepts all messages, looks for the ones from the server, then pops up a notification.
- Persistent TCP/IP The phone initiates a long-lived mostly idle TCP/IP connection with the server and maintains it by occasionally sending keepalive messages. Whenever there is something new on the server, it sends a messages to the phone over the TCP connection.
The first two methods have significant disadvantages that we cannot do anything about. However, the third method’s drawbacks are not as severe. It seems like with enough work and a good design, the persistent TCP/IP method can work. After all, that’s how GMail, GTalk and Google Voice implement their real-time updates. In fact, many developers out there agree that it is probably the best way to go until Google actually takes the matter in their own hands.
Persistent TCP/IP
After more Googling around I was able to come across three reasonable efforts to implement push notifications using a persistent TCP/IP connection:
- Josh Guilfoyle talks about how to create a most-idle TCP/IP connection with a long keep-alive timer based on the AlarmManager. He provides some really cool sample code with a service that runs in the background and makes connections. http://devtcg.blogspot.com/2009/01/push-services-implementing-persistent.html
- Dave Rea recently started the Deacon project, which is aimed at developing a 3rd party library for Android push notifications using comet technology based on the Meteor server. The project is still in a very early stage (at the time of this post), but looks quite promising. http://deacon.daverea.com/
- Dale Lane had done a number of presentations, where he talked about using the IBM developed MQTT protocol for implementing Android push notifications. He also provides some really useful sample Android code. http://dalelane.co.uk/blog/?p=938
While all of the work done by these guys is incredible, none of their results are quite ready for drop-in use by other developers. In my effort to implement push notifications, I decided to put the pieces of the puzzle together and combine their results to produce a relatively stable way of implementing push. The example that I provide you with further, is a combination of Josh Guilfoyle’s TestKeepAlive project and Dale Lane’s MQTT work. I borrow quite a bit of code from those guys, so they should get most of the credit. Anyways, enough for the introduction, let’s get to the good stuff.
My Idea
The problem with the TestKeepAlive project is that it creates a raw TCP connection, which means that you need write your own server to take care of push on the other side. While it’s, without a question, doable, it is exactly why TestKeepAlive is far from a working solution. On the other hand, the MQTT example shown by Dale Lane uses the IBM’s MQTT broker to handle the server work. To backup a little, MQTT stands for MQ Telemetry Transport, which is a protocol developed by IBM. Let’s take a quick look at the man page:
mqtt is a publish/subscribe messaging protocol intended that is designed to be lightweight. It is useful for use with low power sensors, but is applicable to many scenarios.
Did you see the part about ‘low power’? So did I. Basically, the reason why one might consider using MQTT is that it was designed to be very lightweight, so that it doesn’t consume much power. This is ideal for a mobile push solution as it addresses many battery life related concerns about persistent TCP/IP connections. Obviously, MQTT also has some disadvantages such as privacy, but we can talk about that later.
So, my idea consists of taking a KeepAliveService and replacing the raw TCP/IP connection with an MQTT connection. In this case, each device can simply subscribe to a unique topic which is based on its device ID. Now, assuming that your server knows the device ID, it can push data to the device over MQTT by publishing to that unique topic.
Architecture
In my example, I utilize a PHP script as a server. This uses the Simple Asynchronous Messaging library (see project SAM http://project-sam.awardspace.com/) to publish MQTT messages to the broker on which I host on my server. Let’s have a look at the overall system diagram:
wmqtt.jar is a simple drop-in implementation of MQTT protocol provided by IBM. It can be downloaded from http://www-01.ibm.com/support/docview.wss?rs=171&uid=swg24006006. The file that you download has a bunch of different stuff. Just look for the right jar file. You can include this jar as a part of your Android app.
Really Small Message Broker (RSMB) is a simple MQTT broker also provided by IBM http://www.alphaworks.ibm.com/tech/rsmb. It runs on port 1883 by default. In our architecture it accepts messages from the server and passes them on to the right devices. RSMB can also be replaced by the Mosquitto server http://mosquitto.atchoo.org/.
SAM is a drop-in PHP library for MQTT and other stuff. You can either get it as PECL extension or download the source online http://pecl.php.net/package/sam/download/0.2.0.
send_mqtt.php is a simple PHP script that accepts messages over POST and uses SAM to pass-on messages to the broker.
Sample Code and Demo
The goal of my work on push notifications was to develop a working demo, which is what all other examples out there lack. I’m happy to say that I accomplished my objective. You can download the sample android app on GitHub.
This app (shown on the left) has a TextView and two buttons. The TextView contains your device ID and the buttons are used to start and stop the push notifications service. Once you have the app on your phone, start the service. Then go to http://tokudu.com/demo/android-push/ and enter the device ID in the first text box and enter the message text in the textarea below. Press “Send Push Message” and you should get a notification on your phone. It’s as easy as that.
You can see the source code for andoid-push in this GitHub project. It contains the aforementioned send_mqtt.php script.
If you didn’t get a notification, make sure you have network connectivity. It can also be that the broker is down on my server (see server status on the page). If that’s the case, please post a comment and I will look into it bringing the broker back up.
Final Thoughts and Comments
MQTT is definitely not the best way to implement push for Android, but it does work. One of the main drawbacks of MQTT is that anyone who knows the IP and the PORT at which the broker is running can connect and intercept your Push messages. So it’s probably a good idea to encrypt them. Alternatively, you could write your own broker and introduce some sort of authentication to MQTT.
The code I provide here for the push service still needs more testing. Reliability is definitely the main question. I think the code can definitely be improved to better handle connectivity loss and other erroneous situations. You are welcome to post your comments here regarding how it can be improved.
Also let me know if you find any bad bugs. Good luck testing!
Anton Lopyrev
Follow me on twitter @tokudu
May 16th, 2010 at 2:43 pm
[...] MQTT for push in Android apps, you’ll probably want to head over to Anton L’s blog post How to Implement Push Notifications for Android. He has a sample Android app that uses the IBM Java library to implement push notifications using [...]
May 18th, 2010 at 7:10 am
Anton -
Great post. Push Notifications will be a growing part of the application eco-system. Particulalry on Android where messaging can be incredibly rich (including intents).
At Xtify we have seen real adoption of our SDK by developers eager to take advantage of the Android OS.
Our service provides for FREE push notificationa to Android devices. Any developer can download our SDK, compile into their application and immediately have a full push platform available.
Our service is free because we expect that over time the analytical and geo tools we are building will be more valuable.
You can have a go at it here: developer.xtify.com
May 19th, 2010 at 12:52 pm
Thanks Anton! Your post took me a big step forward. I used mosquitto as message broker. Got my proof of concept working without problems based on your article.
Regards
Klaus
P.S.: I found the smiley at the bottom of this page.
May 20th, 2010 at 7:36 am
Good job. Many thanks.
P.S. server is down.
May 20th, 2010 at 8:51 am
Thank you, it should be back up now
May 21st, 2010 at 11:56 am
Nice work, however you were right Google did take matters into their own hands and the Android Cloud to Device Messaging Framework is poised to be the defacto standard for push messages.
http://code.google.com/android/c2dm/
May 21st, 2010 at 12:26 pm
I’m very excited of try it out. However, if you wish your app to support devices 1.5 + you might still consider using mqtt.
May 23rd, 2010 at 4:27 am
very interesting post, i’m actually looking at those different solutions. yours seems very good, but did you try xmmp solution with the asmack library?
Android Cloud to Device Messaging Framework works from android 2.2 and i am targeting sdk from 1.5.
Furthermore, is mqtt implementation a good solution if i want differents devices to communicate to share data or multimedia sessions, i think that xmmp with jingle could maybe do the trick. I’m still investigating so do you have any idea of a simple way to do this?
May 23rd, 2010 at 10:43 am
XMPP could be a viable solution, after all, that’s what Google Talk is based on. I think it uses libjingle. I haven’t played around with it myself, but I have seen it on blogs online. Take a look at this for instance:
http://credentiality2.blogspot.com/2010/03/xmpp-asmack-android-google-talk.html.
Let me know if you get it to work nicely.
May 24th, 2010 at 10:00 am
actually, it works pretty well. i do use google app engine for the native xmmp client service allowing 1 milion plus notifications. xmpp smack client on the devices and an xmpp serveur. only thing is i got some annoying timely disconnections but i’m working on it actually.
I’m about to try mqtt to see which one is easier to deploy and maintain. I’ll let you know.
Other thing interesting with xmpp is the implementation of multimedia session between users, but not yet on the mobile devices library i guess.
May 25th, 2010 at 4:38 am
Thanks Anton.
You got me working and gave a direction…..
May 29th, 2010 at 11:58 am
Thanks! Tokudu!!! I got many helps.!!
but, I have a problem.
This demo app was running very well one week ago, but now is not running.
In other words, When I enter site ” http://tokudu.com/demo/android-push/ ”
and click “send push message”, my android phone doesn’t react.
Have to I re-install demo app??? or server port number change???
I don’t know what problem is. I need your help~
May 29th, 2010 at 12:57 pm
Hi there,
Try re-downloading the source code from GitHub. I changed somethings, so that’s why it’s probably not working. I just tested it myself, it’s working fine for me.
Good luck.
June 3rd, 2010 at 12:53 pm
Hey,
I was just wondering if mqtt (mosquitto for instance) allows multiple subscriptions from the same IP, but to different topics? I seem to get disconnects when I try to connect though to a different topic.
June 5th, 2010 at 5:04 pm
Yes it does. However, you need to make sure you have a different client name.
June 7th, 2010 at 9:53 pm
Fantastic post, thanks thk! Just wanted to mention that for some reason on my Nexus One, the “connectivity changed” event fires as soon as the service starts and makes the client think that the network connection has been lost. I had to comment out the “registerReceiver” and “unregisterReceiver” lines to get it to work. It works great now though (until I lose network connectivity, I suppose).
June 9th, 2010 at 11:56 am
Hi,
I succesfully made an apk of the sourcecode and i can start the application on my HTC desire. However when i fill in the device id and message on your website, the notification will not be displayed.
Do you know why this happens?
June 9th, 2010 at 12:45 pm
I have copied your code over to a scripting service that is “hosting” my page. I changed only the IP address to be the one where my page is hosted (I think). The problem is when I try to use it, the page comes up but it says that it is offline. I turned debug on and it says: Server status: –>SAMConnection_MQTT()
<–SAMConnection_MQTT(). What should I look at or what could be my problem?
Thanks for the wonderful tutorial by the way.
June 9th, 2010 at 1:57 pm
[22:42:20] Opened log.
[22:42:20] Service started with intent=Intent { act=tokudu.START cmp=com.tokudu.demo/.PushService }
[22:42:20] Starting service…
[22:42:20] Connecting…
[22:42:21] Connection established to 209.124.50.185 on topic tokudu/200145745b9784f9
[22:42:21] Connectivity changed: connected=false
June 9th, 2010 at 2:21 pm
@ alan: what version of PHP are you running? Try to turn on your warnings and see what’s being output.
@Joene from the log file it looks like it connects successfully, but then the network connectivity is lost. Have a look at Geoff’s comment above, he might have had a similar problem. Seems like the connection monitor is experiencing some issues. You can try to rework some code near “BroadcastReceiver mConnectivityChanged” and see when it starts working for you.
June 10th, 2010 at 11:10 am
PHP API 20041225
PHP Extension 20060613
If this isn’t what you want go to the following webpage where there is a lot of information:
http://alanma.scripts.mit.edu/QotD/
also: what address should I use for
fsockopen(ip_address, 1883);
I think my problem is occuring because that returns false
June 10th, 2010 at 11:58 am
found this also: PHP Version 5.2.13
June 10th, 2010 at 12:15 pm
Notice: (alanma) Use of undefined constant SAM_HOST - assumed ‘SAM_HOST’ in index.php
Notice: (alanma) Use of undefined constant SAM_PORT - assumed ‘SAM_PORT’ in index.php
Warning: (alanma) fsockopen() [function.fsockopen]: unable to connect to 18.98.4.32:1883 (Connection timed out)
I’m guessing I have the IP wrong… How can I find out what address to put there?
June 11th, 2010 at 12:13 am
Hi Anton,
I am having following problem when i deployed the php script..
Server status:
Warning: fsockopen() [function.fsockopen]: unable to connect to 127.0.0.1:1883 (A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. ) in C:\wamp\www\tokudu\SAM\MQTT\sam_mqtt.php on line 640
Fatal error: Maximum execution time of 60 seconds exceeded in C:\wamp\www\tokudu\SAM\MQTT\sam_mqtt.php on line 640
Am i missing something here…
I am using wamp server which has php 5.3.0
June 11th, 2010 at 12:27 pm
There’s a bug in the SAM MQTT code - they assume that the network data reads are always of the correct length. This can sometimes cause problems with mosquitto for some reason, which might be the reason for the disconnections. This is discussed briefly (including a fix) at http://chemicaloliver.net/programming/mqtt-mosquitto-and-php/
June 17th, 2010 at 3:19 am
Sir,
The blog entry was extremely helpful to get started with Push notifications.
I spent some time looking into the basic function calls to push messages via the IBM really small broker.
I have a query.How did you make the ‘push messaging’ localized to a specific device id?..The publish function for the client has only four parameters namely
Topic,payload,qos and the retained flag i.e.:
“publish(java.lang.String thisTopic, byte[] thisMessage, int thisQoS, boolean retained)”
Since the source of the IBM broker is not available , how is the default broadcasting of the broker stopped and ability to push messages to a particular device with a specific device id can be implemented?
June 18th, 2010 at 11:13 am
@veejayc You need to change the IP address to the IP address of the server, where you have the MQTT broker running.
@Roger Thanks for that link. I was wondering why the mosquitto server was not working for me.
@Varun, I embed the device ID as part of the topic. Have a look at the source code
June 23rd, 2010 at 5:21 am
Thanks for your helps!
and I have a question.
Instead of using PHP and PHP library, is it possible to implement server layer using java and java library or api?
if possible, do you know solution?
thanks.
June 23rd, 2010 at 6:07 pm
Yeah for sure! You can simply use the same wmqtt.jar file that you include as a part of your android app and use the same function calls.
June 24th, 2010 at 8:09 am
Just wonder do you need to have the app running to receive the push?
Thanks
June 30th, 2010 at 11:53 pm
thk thanks for your extreamly helpful blog!
I’ve testing the push notification on my HTC Desire and it worked just fine two month ago but now I don’t receive the notification. Is there any problem on the server? or is it just me?
July 1st, 2010 at 1:36 pm
Thank-you for the extremely informative post Anton. At the end of your article you say that MQTT is not the best way to implement push for Android, is there a better solution?
Paul
July 15th, 2010 at 8:49 am
Thanks a lot for helping the community !!
So i started to make some work based on yours, to be able to make broadcast and terminal to terminal communication, but i’m worried about something.
Is there any limitation (other than with my own server) in the “(SAM+MQTT) solution” on the number of clients that can be registered at the same time ?
PS And about other limitations, if it can help some people, i saw the MQTT client ID cannot be more that 6 characters (the target part before the slash).
Thanks, and congrats again for the work !
July 21st, 2010 at 9:01 pm
MQTT does have a limitation on the number of clients, but it’s due to the number of sockets that the Linux kernel supports. By default it’s around 1032, but it can be changed in the kernel to a really large number when needed.
The MQTT client ID limit is I believe 8 characters, or smth like that.
July 27th, 2010 at 12:36 pm
For those of you that prefer a simple implementation and/or are not interested in maintaining your own servers, give Xtify a try.
Xtify is an out-of-the-box push notification service - download and include the Xtify SDK with your application. Then you can use our web console to configure message campaigns or call our webservice for system / device generated messages.
You can even create geo-triggered push notification campaigns by adding some locations and rules.
Xtify is also compatible with iPhone and Blackberry so you have one place to push across your entire user base.
August 2nd, 2010 at 11:08 pm
[...] How to Implement Push Notifications for Android (tags: mobile android rtw) [...]
August 6th, 2010 at 4:13 pm
A clarification: according to the MQTT spec, the length of the client id should be between 1 and 23 characters. Mosquitto doesn’t enforce the upper limit though.
I can also comment on the limitations on the number of clients that can be connected to a broker. I believe (although I may be wrong) that RSMB is limited to approximately 1024 clients connected at once. Mosquitto on the other hand is capable of having greater than 1024 clients connected, although this needs a small amount of configuring on the server to increase the limits imposed by the OS.
August 10th, 2010 at 3:04 am
How to Implement Push Notifications for Android
August 11th, 2010 at 7:51 pm
Tokudu Android Push Demo:
PHP Version 5.2.10
Apache 2.0 Handler
Server status:
Warning: fsockopen() [function.fsockopen]: unable to connect to 127.0.0.1:1883 (ѕопытка установить соединение была безуспешной, т.к. от другого компьютера за требуемое врем€ не получен нужный отклик, или было разорвано уже установленное соединение из-за неверного отклика уже подключенного компьютера. ) in C:\SERVER\www\lesson\android\PhpMQTTClient\SAM\MQTT\sam_mqtt.php on line 640
August 24th, 2010 at 1:53 am
Handling Notifications…
What support do platforms provide?……