I was always intrigued by Django. After all, it’s slogan is “The web framework for perfectionists with deadlines”. Last year I started a project for a client who needed a web app to manage a digital printing workflow. I evaluated Django and did their tutorial (which is really well made by the way). Since the project also required lots of data processing of different data sources (CSV, XML, etc.) Python made a lot of sense. So in the end the choice was to use Django.
I needed to create several tabIes showing data from the Django models. In this post I explain how I combined
django_tables2 (for the table definitions) and Bootstrap Table (for visualizing the tables and client-side table features).
Thanks in particular to the great German podcast Das Coronavirus-Update from NDR with two fantastic virologists (Christian Drosten and Sandra Ciesek) I gained a better understanding of the pandemic and this coronavirus (SARS-CoV-2) and learned quite a lot new stuff. Maybe because of this, I felt for a while that there was a lack here (in Québec/Montréal) about how the data is presented/visualized. But I had this realization “late” (Fall 2020) and didn’t have past data so I didn’t want to start something from scratch.
One day I randomly came across a repository on GitHub of a dashboard for Montreal. Unfortunately, the website hadn’t been updated in a while so I reached out to Jeremy Moreau who created it. We had a chat and I started helping him adding the data, adjusting to new data formats and automating the data retrieval and processing. Adding the missing data from the past was possible thanks to the Wayback Machine and (for the most part) to the hard work by Jean-Paul Soucy archiving all Canada-wide data for the COVID-19 Canada Open Data Working Group.
Recently, we were trying to find an available camp site on SEPAQ during the summer. We were late to the party, though, and most (interesting) sites were already booked or had single days left here and there.
Finding the remaining sites at flexible dates is actually quite cumbersome since you need to go to each camp site (and click through a calendar week by week there) or go to a single spot of a camp site to see its availability calendar. This is especially cumbersome if you are flexible in terms of the dates and the park.
Long story short, I looked at how to get the availability of the camp sites and hacked together a scraper. It recursively finds all camping spots and downloads the availability for each of them. Once they are downloaded it can parse them and filter for available spots (with minimum days and a desired date range).
You can find the code here: https://github.com/mschoettle/sepaq-availability-scraper
There’s definitely some things that could be improved but it got the job done.
For a long time I struggled with setting up cronjobs properly. It always took a lot of trial and error. Most of the times this was due to an environment problem (or a typo). You could test your cronjob but it would also be nice to be notified whenever something goes wrong (such as your backup script failing).
I had actually seen this output in the syslog before but never really cared much about it:
(CRON) info (No MTA installed, discarding output)
cron sends an email whenever a cronjob has an output, unless of course it cannot send the email.
So I recently set up msmtp on my server(s). I started with these instructions but in the end this is how it is working for me:
- Install msmtp:
apt install msmtp
- Create the config file
/etc (you might also need one for each user who wants to send email in
- I tried to use gpg for password management to avoid storing the password as plain text. However, gpg encrypted passwords can not be decrypted with cron/sendmail. If you can, use a dedicated email account for this purpose.
- For cron to know where to send email to you need to do one of the following:
- specify default email in
MAILTOemail@example.com in the crontab
- Finally, cron uses
sendmail to send out emails. You could just install the msmtp-mta package. However, you then get a From header in the emails as “root (Cron Daemon) <>”. If you want to customize this, create a
sendmail alias in
/usr/sbin. This is especially handy if you set this up for several servers and want to see which server an email is coming from. Follow these instructions to properly set it up. Note that starting with msmtp v1.8.8 there is a configuration setting (
set_from_header) for this.
# find out more about the configuration here: https://marlam.de/msmtp/msmtprc.txt
# Set default values for all following accounts.
# Set a default account
account default : server
Here are the steps I use to set up and configure a fresh install of Debian on a server.
- Log in as root:
ssh root@<ip or domain.tld>
- Change the root password:
- Update the system:
apt update && apt upgrade
- Configure timezone:
- Configure locales:
- Install your favourite text editor (here nano) and make it the default:
apt install nano
update-alternatives --config editor
Now, create a user for yourself that you will be using and give this user rights to run commands that require root privileges:
- Create new user:
- If sudo is not installed:
apt install sudo
- Add your user to the sudo group:
usermod -aG sudo <username>
- Now, try to log in from a second terminal using that user
- Optional (but strongly recommended): Add your public key to log in without a password:
ssh-copy-id <username>@<server IP/domain>
Now that you have your own user, let’s harden the SSH daemon by changing the port and restricting root access from the outside.
- Edit the SSHD config:
Port to something other than the default
- If you want to disable logins by password and only allow key-based authentication, change
- Restart SSHD:
sudo systemctl restart ssh
- Important: Try to log in from another terminal first to ensure it is working as intended (use
ssh -p <newPort> if you changed the port)
Now, install a firewall (here ufw) to only open the ports that you really need:
- Install ufw:
apt install ufw
- Create rule to allow SSH port:
ufw allow <sshPort>/tcp (if you use the default port you can also use
ufw allow OpenSSH)
- You can also rate limit a port (6 or more connections within 30 seconds):
ufw limit <port>/tcp
- Ensure that your rule is correct, otherwise you will lock yourself out in the next step
- Enable ufw:
- Try to log in from another terminal to verify it is still working.
That’s pretty much it. You might also want to set up msmtp so that you receive email from your system, cron etc. There are also the following packages I find useful and install:
- htop: Allows to interactively monitor the system resources and processes.
- icdiff: A nice tool providing side-by-side comparison with color highlighting.
- dnsutils: Essential for diagnosing/testing network stuff. For example, it provides dig.
- ntp: Time synchronization.
- ncdu: Nice tool to find big files.