Category: Uncategorized

Thanks in particular to the great German podcast Das Coronavirus-Update from NDR with two fantastic virologists (Christian Drosten and Sandra Ciesek) I gained a better understanding of the pandemic and this coronavirus (SARS-CoV-2) and learned quite a lot new stuff. Maybe because of this, I felt for a while that there was a lack here (in Québec/Montréal) about how the data is presented/visualized. But I had this realization “late” (Fall 2020) and didn’t have past data so I didn’t want to start something from scratch.

One day I randomly came across a repository on GitHub of a dashboard for Montreal. Unfortunately, the website hadn’t been updated in a while so I reached out to Jeremy Moreau who created it. We had a chat and I started helping him adding the data, adjusting to new data formats and automating the data retrieval and processing. Adding the missing data from the past was possible thanks to the Wayback Machine and (for the most part) to the hard work by Jean-Paul Soucy archiving all Canada-wide data for the COVID-19 Canada Open Data Working Group.

Recently, we were trying to find an available camp site on SEPAQ during the summer. We were late to the party, though, and most (interesting) sites were already booked or had single days left here and there.

Finding the remaining sites at flexible dates is actually quite cumbersome since you need to go to each camp site (and click through a calendar week by week there) or go to a single spot of a camp site to see its availability calendar. This is especially cumbersome if you are flexible in terms of the dates and the park.

Long story short, I looked at how to get the availability of the camp sites and hacked together a scraper. It recursively finds all camping spots and downloads the availability for each of them. Once they are downloaded it can parse them and filter for available spots (with minimum days and a desired date range).

You can find the code here: https://github.com/mschoettle/sepaq-availability-scraper

There’s definitely some things that could be improved but it got the job done.

For a long time I struggled with setting up cronjobs properly. It always took a lot of trial and error. Most of the times this was due to an environment problem (or a typo). You could test your cronjob but it would also be nice to be notified whenever something goes wrong (such as your backup script failing).

I had actually seen this output in the syslog before but never really cared much about it:

(CRON) info (No MTA installed, discarding output)

cron sends an email whenever a cronjob has an output, unless of course it cannot send the email.

So I recently set up msmtp on my server(s). I started with these instructions but in the end this is how it is working for me:

1. Install msmtp: apt install msmtp
2. Create the config file msmtprc in /etc (you might also need one for each user who wants to send email in ~/.msmtprc)
   • I tried to use gpg for password management to avoid storing the password as plain text. However, gpg encrypted passwords can not be decrypted with cron/sendmail. If you can, use a dedicated email account for this purpose.
   • Set chmod for those files to 0600, otherwise msmtp will complain (“sendmail: /home/<user>/.msmtprc: contains secrets and therefore must have no more than user read/write permissions”)
3. For cron to know where to send email to you need to do one of the following:
   • specify default email in /etc/aliases
   • specify MAILTO=recipient@domain.tld in the crontab
4. Finally, cron uses sendmail to send out emails.
   
   For msmtp v1.8.8+ install msmtp-mta and set the set_from_header configuration setting to on. The from address can be set to “something <server@domain.tld>” to customize the name display.
   
   For older versions, you could just install the msmtp-mta package. However, you then get a From header in the emails as “root (Cron Daemon) <>”. If you want to customize this, create a sendmail alias in /usr/sbin. This is especially handy if you set this up for several servers and want to see which server an email is coming from. Follow these instructions to properly set it up.

msmtp configuration

# find out more about the configuration here: https://marlam.de/msmtp/msmtprc.txt
# Set default values for all following accounts.
defaults
auth           on
tls            on
tls_starttls   on
tls_trust_file /etc/ssl/certs/ca-certificates.crt
logfile        /var/log/msmtp
aliases /etc/aliases
set_from_header on

# server
account        server
host           mail.domain.tld
port           587
from           "cron@srv <server@domain.tld>"
user           server@domain.tld
password       

# Set a default account
account default : server

When connecting to the database with the postgres user I realized it accepts any password or no password even though the user has a password set. I don’t know if this happens also when using the installer to install PostgreSQL, in my case I used initdb to set it up. I remember it mentioned something regarding “trust” after setting it up but didn’t take much notice until I realized it accepts any password.

In pg_hba.conf it adds all local connections to be trusted which means connecting from the same host doesn’t need to authenticate.

host    all             all             127.0.0.1/32            trust

If you don’t like that just change it to another method, for example md5.