Matthias Schoettle

Software Developer

Notes on traefik v2, Nextcloud, etc.

Now that the Raspberry Pi is set up and Docker ready to be used, Gitea is running nicely. However, without TLS and just accessible by the IP address and port. So before setting up Nextcloud, I wanted to get a reverse proxy ready that also takes care of TLS termination. I use traefik (v2.1) which supports/integrates with Docker. Here I document how I configured it to put all (this includes Pi-Hole and my router’s web interface) my services behind the reverse proxy with TLS.

At the end, I’ll briefly note how Nextcloud is set up.

Continue reading

Migrating Data to Nextcloud

If you need to migrate your data to Nextcloud you probably don’t want to upload all your files through the web interface. I suggest to first try it on with a small amount of data (for example, one folder) to verify that it works. Since Nextcloud runs with Docker in my case some of the commands are specific to that, but if you don’t you can just use the main command that is executed.

As previously noted, my data was “in the cloud” so not already on an external drive (besides my backup of course). In general, although it depends on the size of your data, I therefore suggest to copy the data on to an external drive instead of copying it over the network from your machine to your server. Unless the server is not physically accessible of course (such as a home server). Even then, you can use the same procedure I used.

For the external drive, I formatted it as ExFat so that my Mac can write to it. On the Raspberry Pi, I had to install exfat-fuse. I tried FAT32 as it is supported out-of-the-box by both but it has a limiation on maximum file size. I considered using ext4 but write support on the Mac supposedly is not stable.

If you are copying it directly to the Nextcloud destination, skip this step: First, copy the data to the external drive: rsync -rvia --exclude ".DS_Store" <src> <dest>. We are using rsync to synchronize the files. The important argument is a which preserves attributes, such as when the files were last modified. Once it is done, plug it in to your server and mount it.

Now copy it to the final destination. If you are using a Docker bind mount, you can directly copy them there. If you are using a named volume, I believe you need to go through a temporary container to be able to copy files there. When Nextcloud adds files and directories it sets certain permissions (0755 for directories, 0644 for files). Compared to the rsync command above we can fortunately add another argument that allows setting these right away while copying the files.

rsync -rvia --chmod=D755,F644 --exclude ".DS_Store" <src> <dest>

I left the --exclude here in case you skipped my first step. Also note that there is a difference between specifying a trailing slash on the source directory or not.

Now, hand ownership of the data to www-data (or whichever user your webserver is set to): docker exec chown -R www-data:www-data /var/www/html/data/<user>/files

Nextcloud however does not really “know” about these files yet. Specifically, there is nothing yet in the database regarding these files. To change that, make Nextcloud scan for files and add them: docker exec -u www-data nextcloud-app php /var/www/html/console.php files:scan --all

Depending on the size of your data this can run a few minutes (in my case ~3:30 minutes for almost 20000 files). At the end you should see something like the following output and the files should be shown to you by Nextcloud.

Starting scan for user 1 out of 2 (user1)
Starting scan for user 2 out of 2 (user2)
+---------+-------+--------------+
| Folders | Files | Elapsed time |
+---------+-------+--------------+
| 460     | 16589 | 00:03:27     |
+---------+-------+--------------+

Original Source: Tutorial: How to migrate mass data to a new NextCloud server

Notes on Docker

I’ve never really followed the hype around Docker but to be honest also never really taken the time to look into it more. That’s until my friend Harald told me that he is using it on his Raspberry Pi to run some services. What sounded appealing is that you can reproduce builds, you are not “polluting” the host system, you can keep all configs etc. in one place, and move your services somewhere else quickly. The latter is especially interesting when you want to reinstall the host system. Furthermore, you can put the build as well as configuration in version control. Of course, you are adding another layer of complexity in the mix. I thought I’d give it a try. Here are some notes pertinent to the setup with my Raspberry Pi.

Continue reading

Notes on Setting Up My Raspberry Pi

I recently purchased a Raspberry Pi 4 Model B for some projects at home. Here are some notes on how I set it up/what I did. They are mostly for my future self but might be helpful to others that try to do something similar.

Basically, the main goal was to use it for Pi-Hole (network-wide ad blocking on the DNS level), Gitea (self-hosted git service), and to set up a private cloud for all our data (using Nextcloud) instead of having them with one of the cloud providers. So far we used Boxcryptor to encrypt the data and store it in the cloud but it makes it a bit inconvenient for pictures since you can’t see a preview (thumbnail). Since there is potentially a lot of data I intended to store the data on an external drive. The Pi4 has USB3 so I mounted a solid state drive (SSD) using a SATA-to-USB3 enclosure/adapter.

Continue reading

Making EMF models serialized in XMI available in JSON with emfjson

This summer we had two interns in our team who started creating a web application for our TouchCORE tool. I have wanted this for a long time. Not only does it allow you to model in your browser, you can also do this collaboratively! For now class diagrams are supported but more supported will be added in the future (for example for sequence diagrams and feature models).

Re-creating the complete application right away was not feasible for this project. Also, we are all about reuse and I had envisioned in the architecture to reuse the backend parts when we build another user interface. Therefore, the goal was to keep our current backend based on the Eclipse Modeling Framework (EMF) and making it available through an API to the web application. Fortunately, there is already support for EMF and JSON by emfjson. The main use case however is to replace the XML/XMI serialization/deserialization with JSON.

There is a way to keep XMI on the backend and making it available through JSON. This allows to keep the XMI serialization which is still being used by the existing application. It requires to use unique IDs when serializing, otherwise you need to manage your own ID mapping (see below).

The emfjson-jackson setup is as follows:

ObjectMapper mapper = new ObjectMapper();
// Optional
mapper.configure(SerializationFeature.INDENT_OUTPUT, true);
        
module = new EMFModule();
module.configure(EMFModule.Feature.OPTION_USE_ID, true);
// Optional
module.configure(EMFModule.Feature.OPTION_SERIALIZE_TYPE, true);

module.setIdentityInfo(new EcoreIdentityInfo("_id"));
mapper.registerModule(module);

The use of IDs along with the EcoreIdentityInfo ensures that the IDs used by EMF during serialization are used. Now, assuming you loaded an existing model from a resource for any EObject of that model you can get the JSON by calling:

mapper.writeValueAsString(eObject)

To do the reverse, you can identify an object by its ID. To retrieve the EObject for a given ID EMF of course has a handy utility method. For this, you need the Resource of your model.

resource.getEObject(id)

To get the ID of an object you can use resource.getURIFragment(eObject).

If you have cross-references in your models to other models, the URI you get is of the form “anotherModel.ext#id”. You can use the ResourceSet to get the resource of the other model or the cross-referenced object using the resource set’s getResource(URI, boolean) and getEObject(URI, boolean) methods.

You can also maintain your own ID mapping by passing a custom ValueWriter as a second parameter to EcoreIdentityInfo, for example:

module.setIdentityInfo(new EcoreIdentityInfo("_id",
    new ValueWriter<EObject, Object>() {
        @Override
        public Object writeValue(EObject eObj, SerializerProvider context) {
            // TODO: return ID of object
        }
    }
));

Furthermore, you need to handle references between objects specifically by customizing the reference serializer:

module.setReferenceSerializer(new JsonSerializer<EObject>() {
    @Override
    public void serialize(EObject eObject, JsonGenerator generator, SerializerProvider serializer) throws IOException {
        generator.writeString(/* TODO: Get the ID of eObject */);
    }
});

There is one caveat though. That is when using EMF notifications on the backend to notify the frontend that something changed. If an object is deleted, EMF does not give you an ID for this deleted object anymore. That’s why we in the end used the custom ID solution.

As I researched some links for this post I noticed that there is now a whole Eclipse project to make EMF available in the cloud (EMF.cloud). It would be interesting to evaluate whether these technologies (Sprotty as the web-diagramming framework and GLSP as the graphical language server protocol) are more suitable instead of a custom-baked solution. Currently we use Spring Boot on the backend with web sockets and Angular with mxgraph on the frontend.

« Older posts

© 2020 Matthias Schoettle

Theme by Anders NorenUp ↑