D'oh! pabluk's blog

Building a serverless Telegram bot

Mon, 03 Sep 2018 18:48:00

With the arrival of FaaS and all the serverless infrastructure providers (AWS, Google Cloud, Azure, etc) now is easy to deploy our own Telegram bot.

A Webhook-based bot it's the perfect use case for serverless computing, using for example Google Cloud Functions we'll have all these advantages:

Our function will be associated to a public endpoint (IPv4 and IPv6) over HTTPS (using a valid certificate provided by Google)
Pay only for function invocations and its compute resources consumption, so it could be cheaper than paying for a server powered on all the time
No server management
Scales automatically

Requirements

Before start building your bot you'll need to:

Register a new bot following these instructions core.telegram.org/bots in order to get an API token
Using a Google Cloud account, register a new project, enable the Cloud Functions API and install the Cloud SDK following the official quickstart

Code

In a few lines of Python you can code a simple echo bot that will reply with the same message that it receives:

# main.py
import os
import telegram

def webhook(request):
    bot = telegram.Bot(token=os.environ["TELEGRAM_TOKEN"])
    if request.method == "POST":
        update = telegram.Update.de_json(request.get_json(force=True), bot)
        chat_id = update.message.chat.id
        # Reply with the same message
        bot.sendMessage(chat_id=chat_id, text=update.message.text)
    return "ok"

The only dependency needed is the amazing python-telegram-bot library:

# requirements.txt 
python-telegram-bot==11.1.0

Notes about Android and IPsec on Linux

Sat, 23 Apr 2016 21:35:00

Here's some personal notes about the excellent Jeff Sharkey's article Deploying a pure-IPsec PKI VPN server for Android devices

My setup is a little bit different to that of Jeff, I have a 1Gbps fiber plan, a Linux server natted behind my ISP's router and my IPsec VPN server is running on a Linux container (LXC), so this container doesn't have a public IP address. The OS used in all the servers is Ubuntu Trusty 14.04. On the VPN client side I'm using Android 6.0 Mashmallow.

Server configuration

In addition to installing the packages

$ sudo apt-get install ipsec-tools racoon

I also installed the following packages

$ sudo apt-get install iptables openssl tcpdump

When installing the Debian racoon package, you have to chose the type of configuration, in my case I chose the direct mode.

I had to add 2 new port forwarding rules on my ISP's router to sent the traffic UDP on the ports 500 and 4500 to my Linux server, and on it I had to the same thing using iptables

$ sudo iptables -t nat -I PREROUTING -p udp -m udp --dport 4500 -j DNAT --to-destination 10.0.3.91:4500
$ sudo iptables -t nat -I PREROUTING -p udp -m udp --dport  500 -j DNAT --to-destination 10.0.3.91:500

Here the IP address 10.0.3.91 is assigned to the container running the IPsec VPN server.

In the /etc/racoon/racoon.conf file I had to change the public IP address used by Jeff by the container's private IP address:

listen {
    isakmp 10.0.3.91[500];
    isakmp_natt 10.0.3.91[4500];
}

` For more details about the configuration options you can see the racoon.conf's man page.

About the RSA keys, don't forget to convert your server's private key into RSA format using

$ openssl rsa -in myserver.key -out myserver-rsa.key

As the documentation says the output filename should not be the same as the input filename, so you also need to change it in your /etc/racoon/racoon.conf.

Android configuration

Here some details when configuring your VPN profile on your phone:

You don't need to install the adb tool to push your .p12 certificate file, you can download it from an URL, Android will detect the format and it will install it using the Android's certificate installer.
If you want to use the Always-on VPN option, you need to specify a server's public IP address (instead of its fqdn) and you also must specify an IP address for the DNS servers.

Troubleshooting

If you find some problems, you can always take a look at the /var/log/syslog file and use tcpdump to inspect your network traffic.

It's also highly recommended to read the Section 2 of the IPsec HOWTO to understand the theory behind IPsec and its Linux Kernel implementation.

You Build It, You Run It

Sun, 05 Apr 2015 20:38:00

You Build It, You Run It

Giving developers operational responsibilities has greatly enhanced the quality of the services, both from a customer and a technology point of view. The traditional model is that you take your software to the wall that separates development and operations, and throw it over and then forget about it. Not at Amazon. You build it, you run it. This brings developers into contact with the day-to-day operation of their software. It also brings them into day-to-day contact with the customer. This customer feedback loop is essential for improving the quality of the service.

source: Werner Vogel, CTO at Amazon.com from Programming Amazon EC2

8-bit art in Paris

Sun, 29 Mar 2015 20:33:00

The 8-bit version of Mona Lisa at 20 Rue du Louvre, Paris.

DNS resolution for LXC in Ubuntu 14.04

Sun, 27 Jul 2014 15:28:00

Working with LXC (Linux containers) in Ubuntu is very easy but by default you need to know the IP address of new containers to connect to services (ssh, database, webserver, etc.) running on them.

With some minor configuration you can connect to your containers using a domain name like that

$ sudo lxc-create --name container1
$ sudo lxc-start --name container1 --daemon
$ ssh ubuntu@container1.lxc

In Ubuntu 14.04 Trusty Tahr, lxc-create use by default the Ubuntu template and it will create a user called ubuntu, to change it take a look at the template options with lxc-create -t ubuntu -h.

To set up the internal DNS resolution on your machine, you must edit /etc/default/lxc-net and uncomment the line

LXC_DOMAIN="lxc"

Also you need to create the file /etc/NetworkManager/dnsmasq.d/lxc.conf with the following content

server=/lxc/10.0.3.1

This will redirect DNS queries for *.lxc hosts to the dnsmasq instance running on 10.0.3.1 that manage DHCP and DNS for containers.

After that restart networking related services

$ sudo service lxc-net stop
$ sudo service lxc-net start
$ sudo service network-manager restart

For the lxc-net service you can't use the restart command, you must use the stop/start commands to reload the configuration.

If you had some containers running, do not forget to restart them

$ sudo lxc-stop --name container1
$ sudo lxc-start --name container1 --daemon

Finally to check that everything works you can use, for example, the ping command and you must see something like this

$ ping -c 3 container1.lxc
PING container1.lxc (10.0.3.8) 56(84) bytes of data.
64 bytes from 10.0.3.8: icmp_seq=1 ttl=64 time=0.072 ms
64 bytes from 10.0.3.8: icmp_seq=2 ttl=64 time=0.125 ms
64 bytes from 10.0.3.8: icmp_seq=3 ttl=64 time=0.113 ms

--- psql.lxc ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 0.072/0.103/0.125/0.024 ms

Comment utiliser les photographies aériennes de Nantes avec Leaflet

Sat, 04 Jan 2014 01:15:00

Ce mini post explique comment on peut utiliser les Photographies aériennes de la Loire-Atlantique pour créer des cartes interactives avec Leafletjs.

Les Photographies aériennes de la Loire-Atlantique font partie des jeux de données Open Data de la région des Pays de la Loire et pour utiliser et accéder aux données il faut acepter leur licence.

Pour accéder aux images le site vuduciel.loire-atlantique.fr délivre un flux WMS (Web Map Service) qui peut être intégré facilement avec Leaflet en utilisant une TileLayer.WMS. Par exemple avec quelques lignes de HTML on peut générer une carte et ajouter un marqueur :

<html>
<head>
    <link rel="stylesheet" href="http://cdn.leafletjs.com/leaflet-0.7.1/leaflet.css" />
    <style>
        #map { height: 480px; }
    </style>
</head>
<body>
    <div id="map"></div>
    <script src="http://cdn.leafletjs.com/leaflet-0.7.1/leaflet.js"></script>
    <script>
        var map = L.map('map').setView([47.2162, -1.5492], 14);
        var laURL = "http://services.vuduciel.loire-atlantique.fr/geoserver/ows/"
        var loireAtlantique = L.tileLayer.wms(laURL, {
            layers: 'ORTHO44:jp2',
            format: 'image/jpeg',
            attribution: "Open Data D\u00e9partement de Loire-Atlantique"
        });
        var marker = L.marker([47.2162, -1.5492]);
        marker.bindPopup("Ch&acirc;teau des ducs de Bretagne").openPopup();

        loireAtlantique.addTo(map);
        marker.addTo(map);
    </script>
</body>
</html>

voici le résultat avec le code de l'exemple.

Si on a besoin de modifier les images livrées par le WMS ou de servir les images depuis notre propre serveur web de façon statique on peut utiliser par exemple landez qui permet de télécharger les images (tiles) d'une zone particulière de la carte, les modifier et les héberger dans un répertoire de notre serveur web.

Disclaimer : ma langue maternelle est l'espagnol, donc n'hésitez pas à me corriger s'il y a des fautes d'orthographe :)

Do not program defensively

Thu, 19 Dec 2013 08:49:00

Do not program defensively

Most of the code in the system should be written with the assumption that the input data to the function in question is correct. Only a small part of the code should actually perform any checking of the data. This is usually done when data enters the system for the first time, once data has been checked as it enters the system it should thereafter be assumed correct.

source: Program Development Using Erlang

Testing your REST client in Python

Fri, 27 Sep 2013 12:38:00

When you start to write a client for a REST API in Python at beginning it's easy to test it using a Python interactive session, but at some point you'll have to write tests, at that moment you'll see that it's not easy to test your code against live data from the RESTful web API. You may encounter various problems, for example, you can have network problems when tests run, the web server may be temporarily down or tests become slow due to network latency.

A solution to this problem is to use Mock objects, they simulate the behavior of your real objects in a controlled way, so in this case a mock object may simulate the behavior of the urlopen function (from the urllib2 module) and return something like an HTTP response (a file-like object) without hit the real REST API. The file-like object returned may map a RESTful resource to a file which contains a pre-saved response from the real web server.

To show the idea I wrote a simple REST client for the Github API. Here's what the directory structure looks like

$ tree project
project
├── client.py
└── tests/
    ├── test_client.py
    └── resources/
        └── users/
            └── test_user

3 directories, 3 files

I use nose as test runner and Foord's Mock library to create mock objects. You can install them into a virtualenv by typing

$ pip install nose mock

Here's the content of the client.py file

import json
from urllib2 import urlopen


class ClientAPI(object):
    def request(self, user):
        url = "https://api.github.com/users/%s" % user
        response = urlopen(url)

        raw_data = response.read().decode('utf-8')
        return json.loads(raw_data)

As you can see, it calls urlopen, parse the JSON data from the HTTP response and return a Python dictionary.

All links are created equal

Tue, 10 Sep 2013 13:11:00

“On an open Internet, where all links are created equal, good ideas win. Anyone, anywhere can share an idea that can be seen by millions.”
—Alexis Ohanian, Reddit co-founder

8-bit art in Nantes II

Sun, 18 Aug 2013 20:07:00

Mario Bros in the streets of Nantes. Can you guess where I found it?

How to use the Percolate API with Python

Fri, 19 Jul 2013 08:21:00

Recently, I've been working on migrate a Django project from Solr to Elasticsearch, both of them are great search servers based on Apache Lucene, but Elasticsearch has an interesting feature called Percolate, that's missing on Solr.

Percolate is the reverse operation of indexing and then searching. Instead of sending docs, indexing them, and then running queries. One sends queries, registers them, and then sends docs and finds out which queries match that doc.

So, here an example of percolation from the Percolate API documentation with Python. See the setup section to known how to setup Elasticsearch and get it running.

First, you'll want to install pyelasticsearch

$ pip install pyelasticsearch

Open a terminal and type python to use the Python interactive console

>>> from pyelasticsearch import ElasticSearch
>>> es = ElasticSearch('http://localhost:9200')

You need create a new index, named test

>>> es.create_index('test')
{u'acknowledged': True, u'ok': True}

Now, you must specify a query to index. As the _percolator index is also an index, we use the index method to index it, doc_type is the name of the index created before (test) and kuku is our query id

>>> query = {'query': {'term': {'field1': 'value1'}}}
>>> es.index(index='_percolator', doc_type='test', doc=query, id='kuku')
{u'_type': u'test', u'_id': u'kuku', u'ok': True, u'_version': 1, u'_index': u'_percolator'}

Finally to test a percolate request we need call the percolate method with a document

>>> doc = {'doc': {'field1': 'value1'}}
>>> es.percolate(index='test', doc_type='type1', doc=doc)
{u'matches': [u'kuku'], u'ok': True}

We can see that kuku matches with our document.

For more details see the Elasticsearch's reference API and the documentation of pyelasticsearch.

The value of breaking the law

Wed, 17 Jul 2013 08:20:00

The value of breaking the law

Interesting post and essay about the value and the ability to break the law.

source: Bruce Schneier's blog

8-bit art in Nantes

Thu, 11 Jul 2013 12:12:00

One of the ghosts from the Pac-Man video game that I discover in the streets of Nantes.

Running Scrapy on Amazon EC2

Tue, 26 Mar 2013 16:21:00

Sometimes can be useful to crawl sites with Scrapy using temporary resources on the cloud, and Amazon EC2 is perfect for this task. You can launch an Ubuntu OS instance and schedule your spiders using the Scrapyd API. With boto, a python interface to Amazon Web Services, you can launch instances and install the scrapy daemon using the user data feature to run a script on boot.

First, you need an AWS account with your access keys, a EC2 security group accepting TCP connections on port 6800 and a key pair for the selected region. After that you must choose an Ubuntu EC2 image, here you can find a list of Ubuntu AMIs.

Ubatar and the Ubuntu App Showdown

Sun, 08 Jul 2012 20:11:00

The last 3 weeks I was working on developing an application to participate in a contest called Ubuntu App Showdown.

My application is called Ubatar and its main objective is to provide a solution to this idea. Here you can see some videos of the latest version of Ubatar fulfilling its purpose.

During these 3 weeks I learned many things, reading a lot of source code, looking for examples and discussing with other developers. I can only say that I really enjoyed and was an incredible experience. Thanks to all the Ubuntu development team and the community at large.

And this is not the end of the project, there are still many things to improve, if you want to help please contact me via this form. Also any questions or suggestions can be sent to Questions for Ubatar in Launchpad.

World IPv6 day

Wed, 06 Jun 2012 21:53:00

To celebrate the world IPv6 day I decided to activate IPv6 on this blog. Now you can visit my site with native IPv6 connectivity.

If you want to know more about IPv6 you can start reading this overview.

Installing Solr on Debian 6.0

Tue, 22 Feb 2011 19:29:00

Update: this is an old blog post, Solr is now included in Debian since 7.0, it can be installed through apt-get by calling apt-get install solr-tomcat.

Recently I've been working on a Django project that uses the Haystack search API and the Solr search engine to perform full text searching on indexed data.

This project was initially deployed on a Debian 5.0 (Lenny) server with some issues with Tomcat, but using the latest stable release of Debian (code named Squeeze) install Tomcat 6 and Solr is very easy.

Just run the following

apt-get install tomcat6 tomcat6-admin

download and install the latest version of Solr

wget http://apache.cict.fr//lucene/solr/1.4.1/apache-solr-1.4.1.tgz
tar xvfz apache-solr-1.4.1.tgz
cd apache-solr-1.4.1
cp dist/apache-solr-1.4.1.war /var/lib/tomcat6/webapps/solr.war
cp -fr example/solr /var/lib/tomcat6/
chown -R tomcat6:tomcat6 /var/lib/tomcat6/solr

after install Solr you need to restart the tomcat6 service

/etc/init.d/tomcat6 restart

And voilà, Solr is running. Point your browser to http://localhost:8080/solr/admin this should show the Solr Admin interface.

Screenkey 0.2 released!

Wed, 16 Jun 2010 11:02:00

A few days ago I released the first version of Screenkey which was reviewed positively by OMG! Ubuntu!, Korben and Linux und Ich.

Now, I'm happy to announce the release of Screenkey 0.2. This release includes

A preferences dialog to adjust size and position
Two keyboard modes (normal and raw)
Command line options (see --help)
Some bug fixes

The next release will include support for Launchpad translations, save session to file and more keyboard modes (specific vim and emacs key bindings).

Thanks to Ivan Makfinsky for Fedora packaging and thanks to everyone who reported problems and made suggestions for improvement!

Le greffon OpenSubtitles de Totem

Tue, 25 May 2010 22:18:00

Si vous n'avez pas encore mis à jour votre système Ubuntu vers la dernière version 10.04 Lucid Lynx et si vous utilisez le greffon de téléchargement de sous-titres du lecteur vidéo Totem, vous remarquerez que le greffon ne marche plus. Cela est dû à un changement de l'API XML-RPC du site opensubtitles.org.

Mais bon pour le corriger, sans mettre à jour votre système Ubuntu 9.10, vous pouvez ouvrir un terminal et lancez la commande suivante

cd /usr/lib/totem/plugins/opensubtitles/
sudo sed -i 's/www/api/' opensubtitles.py

Ainsi, vous pouvez continuer à télécharger vos sous-titres sans problèmes.

Another Counter... and J2ME

Thu, 08 Apr 2010 23:28:00

Just to try Java Wireless Toolkit, also know as J2ME, on my Samsung C3050 mobile phone I wrote a silly and simple application called Another Counter, wich as its name says is used to count things, like a tally counter, then you can count people, events, measure traffic, etc.

A version of the toolkit for Linux can be downloaded from here, it includes build tools, utilities, examples, and a device emulator. With some knowledge of Java programming and reading the API is easy to write small applications like this.