Catégorie: Informatique

proxyta.net

Informatique,
Tags: web

NB: All of this is already done and available on proxyta.net. You can just clone it and use it. This post is only the step by step explanation.

Who needs a reverse proxy, anyway ?

If you have ever managed a server with more than one web service on it, you are already familiar with some reverse proxy, as the good old apache, the efficient nginx, and/or the brand new traefik.

My point is that if you want to test stuffs as if they were in production (and I hope you do), while not really testing them in your actual production environment (and I hope you don't), you would benefit from using a reverse proxy on your dev setup too.

Obviously, you will not add any complexity to your dev setup, unless there is something really simple, really easy, and really solid. Now guess what… Yup, that's the topic of this post :)

Bonus: if you do this, you can seemlessly switch a setup from test to prod and vice versa, without even letting your services know it !

Bootstrap your environment

To implement this is really easy, thanks to docker & traefik. All you need is to declare a network on which you will put your reverse proxy container and all the services that you want to expose. Let's call it "proxytanet":

docker network create proxytanet

Then, use a simple, ready-to-go, traefik.toml configuration file:

defaultEntryPoints = ["http"]

[entryPoints]
  [entryPoints.http]
    address = ":80"

[docker]
  endpoint = "unix:///var/run/docker.sock"
  domain = "docker.localhost"
  watch = true
  exposedbydefault = false

[api]

And, to finish, a docker-compose.yml for it:

version: '3'

services:
  traefik:
    image: traefik:alpine
    restart: unless-stopped
    ports:
      - 80:80
      - 8080:8080
    networks:
      - proxytanet
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./traefik.toml:/etc/traefik/traefik.toml

networks:
  proxytanet:
    external: true

If you don't already have something binded on the ports 80 & 8080 of your host, a docker-compose up -d will finish this setup. You can now access to traefik's dashboard, on http://localhost:8080.

NB: This setup is available in proxyta.net/dev

Setup your services

As traefik is listening to docker.sock, you can plug any service only by declaring a few labels and putting the container on your proxytanet network. Long story short, in the docker-compose.yml of your app:

version: '3'

services:
  your_app:
    []
    labels:
      traefik.enable: "true"
      traefik.frontend.rule: "Host: your_app.${DOMAIN_NAME:-local}, www.your_app.${DOMAIN_NAME:-local}"
    networks:
      - proxytanet

networks:
  proxytanet:
    external: true

In most cases, your service also need to access to other private containers, like a database or a redis, or a memcached. Then, you also have to put it in the (usually implicit) default network of its docker-compose.yml file:

version: '3'

services:
  db:
    []
  your_app:
    []
    labels:
      traefik.enable: "true"
      traefik.frontend.rule: "Host: your_app.${DOMAIN_NAME:-local}, www.your_app.${DOMAIN_NAME:-local}"
      traefik.docker.network: "proxytanet"
    networks:
      - proxytanet
      - default

networks:
  proxytanet:
    external: true

Just don't forget to tell traefik on which of the networks it has to listen, with the third label in this example.

One more thing

You're almost done: the last thing you need is to configure a DNS that will redirect your_app.local to your ip. In a dev / test environment, it's just a matter of one line in /etc/hosts

127.0.0.1 your_app.local www.your_app.local

As you may have noticed, the configuration is your_app.${DOMAIN_NAME:-local}. In other words, if you have a DOMAIN_NAME variable (defined in your environment or in a .env file) set to exemple.org, traefik will make your service available on your_app.example.org.

Production setup

In production, you can keep the architecture and the proxytanet network, all you need to change is the traefik.toml configuration to generate SSL certificates with Let's Encrypt:

defaultEntryPoints = ["http", "https"]

[entryPoints]
  [entryPoints.http]
    address = ":80"
    [entryPoints.http.redirect]
      entryPoint = "https"
  [entryPoints.https]
    address = ":443"
    [entryPoints.https.tls]
    [entryPoints.https.redirect]
      regex = "^https://www.(.*)"
      replacement = "https://$1"

[docker]
  endpoint = "unix:///var/run/docker.sock"
  domain = "docker.localhost"
  watch = true
  exposedbydefault = false

[acme]
  storage = "acme.json"
  entryPoint = "https"
  OnHostRule = true
  [acme.httpChallenge]
    entryPoint = "http"

And the associated docker-compose.yml:

version: '3'

services:
  traefik:
    image: traefik:alpine
    restart: unless-stopped
    ports:
      - 80:80
      - 443:443
    networks:
      - proxytanet
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./traefik.toml:/etc/traefik/traefik.toml
      - ./acme.json:/acme.json
    command: --acme.email=${ACME_EMAIL}

networks:
  proxytanet:
    external: true

(don't forget to setup the ACME_EMAIL variable in your environment or a .env file, te receive informations from Let's Encrypt.)

NB: This setup is available in proxyta.net/prod-le ; and if you prefer to use your own certificates rather than Let's Encrypt's, you can check proxyta.net/prod-ssl

Examples

This method has been developped in the context of oxyta.net, a CHATONS (for Collectif des Hébergeurs Alternatifs, Transparents, Ouverts, Neutres et Solidaires, which can be nicely translated into KITTENS, for Keen Internet Talented Teams Engaged in Network Service) from our friends of framasoft.

Hence the name.

In this CHATONS, we want to host several services for several organizations (french "associations loi 1901"), so we provide some exemples for different well known services:

And I can also provide exemples for a django project (incoming…)

That's all folks

Thanks for reading, and now, if you have any remarks, suggestions, or questions, feel free to participate in proxyta.net !



Django: Choices & Enum

Informatique,
Tags: django, python, web

Choices can be efficiently represented in Django: you can use a small CharField of keys stored in the database, associated with verboses descriptions, which can even be translated.

Here is the example from the official doc:

from django.db import models

class Student(models.Model):
    FRESHMAN = 'FR'
    SOPHOMORE = 'SO'
    JUNIOR = 'JR'
    SENIOR = 'SR'
    YEAR_IN_SCHOOL_CHOICES = (
        (FRESHMAN, 'Freshman'),
        (SOPHOMORE, 'Sophomore'),
        (JUNIOR, 'Junior'),
        (SENIOR, 'Senior'),
    )
    year_in_school = models.CharField(max_length=2,
                                      choices=YEAR_IN_SCHOOL_CHOICES,
                                      default=FRESHMAN)

    def is_upperclass(self):
        return self.year_in_school in (self.JUNIOR, self.SENIOR)

year_in_school is then really convinient to use, because you get the HTML's <select> that exactly fits your needs, and Django also generates for you a get_year_in_school_display() method that outputs the verbose version you are looking for.

But, hey, who wants to write all of those constants ?
Here is my version for the same functionnalities, you'll quickly understand my point:

from django.db import models
from enum import IntEnum

def enum_to_choices(enum):
    return ((item.value, item.name) for item in list(enum))

class Student(models.Model):
    YEARS_IN_SCHOOL = IntEnum('years_in_school', 'freshman sophomore junior senior')
    year_in_school = models.IntegerField(choices=enum_to_choices(YEARS_IN_SCHOOL),
                                         default=1)

    def is_upperclass(self):
        return self.year_in_chool >= YEARS_IN_SCHOOL.junior

I think that years in school may benefits of beeing represented with numbers, but in other use cases, you can also use models.CharField and Enum


Let us play with D-Bus \o/ (quick example with python and spotify)

Informatique,
Tags: python, scripts

D-Bus is a free and open-source inter-process communication (IPC) system, allowing multiple, concurrently-running computer programs (processes) to communicate with one another.

But if you are here, you probably already know what it is, and you may want to know how to quickly use it to talk with your applications.

You have probably already heard about tools like dbus-send or qdbus, but you first need to know which messages you can send.

The solution's name is D-Feet. It's a graphical tool that allows you to seamlessly and easily use D-Bus introspection functions to quickly find which methods and properties are available for your applications.
If you need a console tool, you can also try mdbus2.

Now, one quick example: let's say that you would like to control your spotify client.

  1. Launch Spotify
  2. Type "spotify" in D-Feet's search field. If nothing appears, check that you are looking in the "Session Bus", and not the "System Bus"
  3. You should easily find methods such as "Play", "Pause", "PlayPause", "Next", "Previous", "Stop" and a few other ones that are pretty self-explaining. Here, you can double clic on methods to call them, or on readable properties to read them.
  4. Now launch a python, with dbus-python installed (dbus-python >= 1.0 works with py2 & py3) and imported: >>> import dbus
  5. Get an interface object
bus = dbus.SessionBus()
proxy = bus.get_object('org.mpris.MediaPlayer2.spotify', '/org/mpris/MediaPlayer2')
interface = dbus.Interface(proxy, dbus_interface='org.mpris.MediaPlayer2.Player')

Where org.mpris.MediaPlayer2.spotify is the "bus name" you can see on the left column of D-Feet, /org/mpris/MediaPlayer2 is the "object path" in the first level of the tree in the right column, and org.mpris.MediaPlayer2.Player is the "dbus interface" which will allow you to directly call methods.

  1. What now ? Nothing, you're done ! Wanna switch play/pause ? interface.PlayPause() ! Wanna stop ? interface.Stop()
  2. Ok, ok, if you need to get or set properties, you have to use the getter and setter in the interface org.freedesktop.DBus.Properties:
props_iface = Interface(proxy, dbus_interface='org.freedesktop.DBus.Properties')
props_iface.GetAll("org.mpris.MediaPlayer2.Player")

But it seems to be a really poor example because the spotify client just does not implement all the standard functionnalities of org.mpris.MediaPlayer2:( (tried interface.Play() ? Sorry :P)

Oh, and three more things:

  • You can find my script in py3 on github
  • if you really want bash:
dbus-send --print-reply --dest=org.mpris.MediaPlayer2.spotify \
    /org/mpris/MediaPlayer2 org.mpris.MediaPlayer2.Player.PlayPause
  • Another example to connect your bluetooth device: github

i3 free workspace

Informatique,
Tags: python, scripts, window-manager

A few years ago, I decided to give a try to some Tiling Window Manager. I began with Awesome, which was… Awesome. But sometimes, you want something new, and I tried XMonad. Great, but kind of hard to configure if you are not eating Haskell for your breakfast.

And a few days ago, I followed some friends and set up i3.

It is almost perfect for me, but it missed one important feature : beeing able to switch to (or move a container to) the next (or previous) free workspace. I think it is really cool to go to the next used workspace with mod4 + , but I needed my mod4 +ctrl + .

So, here is the quick-and-dirty script that let you do that https://github.com/nim65s/scripts/blob/master/i3-free-workspace.py

Then, in your .i3/config:

bindsym $mod+Control+Left exec i3-free-workspace.py switch prev
bindsym $mod+Control+Right exec i3-free-workspace.py switch next
bindsym $mod+Control+Shift+Left exec i3-free-workspace.py move prev
bindsym $mod+Control+Shift+Right exec i3-free-workspace.py move next

Well, with this quality of code, I don't think that it could help anybody, but why not ?


Git pre-commit hook & git commit --all

Informatique,
Tags: git, scripts

It's generally good when a task you have to do many times a day is really fast. For example, when I want to make a commit, I just type gc <message>, because I made gc an alias for git commit -am. And when you have to commit, it's hard to go faster.

But sometimes, fast is not enough.

Today, I just tried to add a pre-commit hook that checks pep8 for the python files I’m about to commit. Easy.

The man page says that pre-commit is bypassed when we use the option --no-verify, so everything should be OK.

BUT sadly git commit -a also bypasses the pre-commit hook. And I don't wanna loose my time adding manually all the modified files in the repo. git add * is definitively not a solution.

However, the solution is quite easy: let the pre-commit script do the right git add stuff !

By "the right stuff", I mean git update-index --add, because it seems that on some systems with some versions of git, git add in a pre-commit adds files but for the next commit… 

Here is my implementation of this solution: https://github.com/nim65s/scripts/blob/master/pre-commit

Now, you may want to add this pre-commit to all your git repositories in every computer you use… Well, just look at this one https://github.com/nim65s/scripts/blob/master/pre-commit-deploy.sh (it just takes a ~/.gitrepos which contains all the repositories you use on the current computer… ie the config file for this script https://github.com/nim65s/scripts/blob/master/gitup.sh)