Software Engineer

Author: Matthias Schoettle (Page 1 of 8)

Django, django_tables2 and Bootstrap Table

I was always intrigued by Django. After all, it’s slogan is “The web framework for perfectionists with deadlines”. Last year I started a project for a client who needed a web app to manage a digital printing workflow. I evaluated Django and did their tutorial (which is really well made by the way). Since the project also required lots of data processing of different data sources (CSV, XML, etc.) Python made a lot of sense. So in the end the choice was to use Django.

I needed to create several tabIes showing data from the Django models. In this post I explain how I combined django_tables2 (for the table definitions) and Bootstrap Table (for visualizing the tables and client-side table features).

Continue reading

COVID-19 Montréal Dashboard

Thanks in particular to the great German podcast Das Coronavirus-Update from NDR with two fantastic virologists (Christian Drosten and Sandra Ciesek) I gained a better understanding of the pandemic and this coronavirus (SARS-CoV-2) and learned quite a lot new stuff. Maybe because of this, I felt for a while that there was a lack here (in Québec/Montréal) about how the data is presented/visualized. But I had this realization “late” (Fall 2020) and didn’t have past data so I didn’t want to start something from scratch.

One day I randomly came across a repository on GitHub of a dashboard for Montreal. Unfortunately, the website hadn’t been updated in a while so I reached out to Jeremy Moreau who created it. We had a chat and I started helping him adding the data, adjusting to new data formats and automating the data retrieval and processing. Adding the missing data from the past was possible thanks to the Wayback Machine and (for the most part) to the hard work by Jean-Paul Soucy archiving all Canada-wide data for the COVID-19 Canada Open Data Working Group.

Continue reading

SEPAQ Availability Scraper

Recently, we were trying to find an available camp site on SEPAQ during the summer. We were late to the party, though, and most (interesting) sites were already booked or had single days left here and there.

Finding the remaining sites at flexible dates is actually quite cumbersome since you need to go to each camp site (and click through a calendar week by week there) or go to a single spot of a camp site to see its availability calendar. This is especially cumbersome if you are flexible in terms of the dates and the park.

Long story short, I looked at how to get the availability of the camp sites and hacked together a scraper. It recursively finds all camping spots and downloads the availability for each of them. Once they are downloaded it can parse them and filter for available spots (with minimum days and a desired date range).

You can find the code here: https://github.com/mschoettle/sepaq-availability-scraper

There’s definitely some things that could be improved but it got the job done.

Setting up msmtp

For a long time I struggled with setting up cronjobs properly. It always took a lot of trial and error. Most of the times this was due to an environment problem (or a typo). You could test your cronjob but it would also be nice to be notified whenever something goes wrong (such as your backup script failing).

I had actually seen this output in the syslog before but never really cared much about it:

(CRON) info (No MTA installed, discarding output)

cron sends an email whenever a cronjob has an output, unless of course it cannot send the email.

So I recently set up msmtp on my server(s). I started with these instructions but in the end this is how it is working for me:

  1. Install msmtp: apt install msmtp
  2. Create the config file msmtprc in /etc (you might also need one for each user who wants to send email in ~/.msmtprc)
    • I tried to use gpg for password management to avoid storing the password as plain text. However, gpg encrypted passwords can not be decrypted with cron/sendmail. If you can, use a dedicated email account for this purpose.
    • Set chmod for those files to 0600, otherwise msmtp will complain (“sendmail: /home/<user>/.msmtprc: contains secrets and therefore must have no more than user read/write permissions”)
  3. For cron to know where to send email to you need to do one of the following:
    • specify default email in /etc/aliases
    • specify MAILTO=recipient@domain.tld in the crontab
  4. Finally, cron uses sendmail to send out emails.

    For msmtp v1.8.8+ install msmtp-mta and set the set_from_header configuration setting to on. The from address can be set to “something <server@domain.tld>” to customize the name display.

    For older versions, you could just install the msmtp-mta package. However, you then get a From header in the emails as “root (Cron Daemon) <>”. If you want to customize this, create a sendmail alias in /usr/sbin. This is especially handy if you set this up for several servers and want to see which server an email is coming from. Follow these instructions to properly set it up.
msmtp configuration
# find out more about the configuration here: https://marlam.de/msmtp/msmtprc.txt
# Set default values for all following accounts.
defaults
auth           on
tls            on
tls_starttls   on
tls_trust_file /etc/ssl/certs/ca-certificates.crt
logfile        /var/log/msmtp
aliases /etc/aliases
set_from_header on

# server
account        server
host           mail.domain.tld
port           587
from           "cron@srv <server@domain.tld>"
user           server@domain.tld
password       

# Set a default account
account default : server

Set Up Debian

Here are the steps I use to set up and configure a fresh install of Debian on a server.

  1. Log in as root: ssh root@<ip or domain.tld>
  2. Change the root password: passwd
  3. Update the system: apt update && apt upgrade
  4. Configure timezone: dpkg-reconfigure tzdata
  5. Configure locales: dpkg-reconfigure locales
  6. Install your favourite text editor (here nano) and make it the default:
    1. apt install nano
    2. update-alternatives --config editor

Now, create a user for yourself that you will be using and give this user rights to run commands that require root privileges:

  1. Create new user: adduser <username>
  2. If sudo is not installed: apt install sudo
  3. Add your user to the sudo group: usermod -aG sudo <username>
  4. Now, try to log in from a second terminal using that user
  5. Optional (but strongly recommended): Add your public key to log in without a password: ssh-copy-id <username>@<server IP/domain>

Now that you have your own user, let’s harden the SSH daemon by changing the port and restricting root access from the outside.

  1. Edit the SSHD config: nano /etc/ssh/sshd_config
  2. Change Port to something other than the default 22
  3. Change PermitRootLogin to no
  4. If you want to disable logins by password and only allow key-based authentication, change PasswordAuthentication to no
  5. Restart SSHD: sudo systemctl restart ssh
  6. Important: Try to log in from another terminal first to ensure it is working as intended (use ssh -p <newPort> if you changed the port)

Now, install a firewall (here ufw) to only open the ports that you really need:

  1. Install ufw: apt install ufw
  2. Create rule to allow SSH port: ufw allow <sshPort>/tcp (if you use the default port you can also use ufw allow OpenSSH)
  3. You can also rate limit a port (6 or more connections within 30 seconds): ufw limit <port>/tcp
  4. Ensure that your rule is correct, otherwise you will lock yourself out in the next step
  5. Enable ufw: ufw enable
  6. Try to log in from another terminal to verify it is still working.

That’s pretty much it. You might also want to set up msmtp so that you receive email from your system, cron etc. There are also the following packages I find useful and install:

  • htop: Allows to interactively monitor the system resources and processes.
  • icdiff: A nice tool providing side-by-side comparison with color highlighting.
  • dnsutils: Essential for diagnosing/testing network stuff. For example, it provides dig.
  • ntp: Time synchronization.
  • curl
  • ncdu: Nice tool to find big files.
  • tree: A nice tool to show directories in a tree-like format.
« Older posts

© 2024 Matthias Schoettle

Theme by Anders NorenUp ↑