Blog

Fix Docker no space left on device

9 september 2024 20:04

When doing a lot with Docker locally every developer find themselves with the no space left on device error message. Even though the error doesn't occur it's good to often clean up your workspace which might save you quite some Gigabytes on your hard drive.Below you'll find an example how to clean up your Docker disc space (were images takes significant the most space).

Squash Django migrations

9 september 2024 20:03

Maintaining a Django (and Wagtail) platform over multiple years comes with some challenges. Although Django is build to scale there are some things to consider while keeping your codebase clean. One of them is the amount of migration files checked into Git. Especially Wagtail migrations are growing hard when building a lot of page types and content blocks (due the fact StreamFields internally supports migrations as well).Luckely Django has support to squash migrations. This makes squashing a lot easier (remember connecting to remote databases and manualy apply migrations with Django South) and fully compatible with your common used Continuous Deployment architectures.

Setup a SSH tunnel

9 september 2024 19:54

Often you need to migrate production data back to your development environment.In our case we use virtual private networks with strict private- and public subnets in order to secure our infrastructure, e.g. access to the database or hotfixing something via a remote shell (keep 12-factor's admin processes in mind).However, that makes accessing production data quick and easily (so no developer unfriendly RDS snapshot S3 transfers) hard as we can't access a resource in a private subnet from the outside.Gladly we have something called bastion hosts with gives us a helping hand.In the example below we have a postgres database running on a RDS instance in a private subnet. On the other hand we have a tiny EC2 instance in a public subnet which accepts incoming traffic over SSH (port 22) retricted to our IP address. The same instance (the bastion host) is allowed to access the database in the private subnet.

Migrating Wagtail ModelAdmin to the new snippet admin views

6 september 2024 12:58

With the new snippet admin views the usage of modeladmin becomes outdated. The snippets now leverage the preview & history abilities from Wagtail, similar to the workflow of pages:PreviewDraft & publish workflowSearchFiltersHowever we find that it's most important to lift upon the core features of the Wagtail CMS so we can translate these snippets as well in the future.Let's see how we can migrate from our existing modeladmin setup and how it looks.First of all we have the current modeladmin configuration for a common used Person model:

Add Plausible to your Terraform CloudFront distribution

6 september 2024 12:54

At Vicktor we take data very serious, that's why we have fully adopted Plausible as a simple and privacy-friendly Google Analytics alternative."Plausible is lightweight and open source web analytics. No cookies and fully compliant with GDPR, CCPA and PECR. Made and hosted in the EU, powered by European-owned cloud infrastructure 🇪🇺."For our infrastructure-as-code we use Terraform in order to manage a lot of Amazon Web Services resources. CloudFront is the CDN we use which sits in front of all our websites.With the instructions below you should be able to setup Plausible into your infrastructure in a best practice way and make use of your own domain name to prevent adblockers block plausible events.The first step is to add plausible.io as an origin into the aws_cloudfront_distribution resource:

Beautiful asserts with your Django Test Client

6 september 2024 12:49

Structural pattern matching with Python 3.10 (AKA "Switch/Case")

6 september 2024 12:49

A Private PyPI server with AWS CodeArtifact

6 september 2024 12:48

IntroductionWhen you develop a variety of reusable Python packages or apps you soon will be facing the limits of git dependencies. And you don't want to release company / project specific packages to the public PyPI. Setting up a private PyPI server can be very time consuming and maintenance / backups are required. Can we use AWS CodeaArtifact to create our own package index?What is AWS CodeArtifactAWS CodeArtifact is a hosted service from AWS which can hold several package formats, like npm, PyPI, Maven, NuGet and generic package formats. The PyPI feature is interesting as it allows us to package and distribute reusable Python applications properly. These packages (wheels) are installable with [pip](https://pip.pypa.io/) or [poetry](https://python-poetry.org/) (or any other python package manager which supports wheels and/or a PyPI server).When to use a private PyPI serverThere are several reasons why packages with a (private) PyPI server could be beneficial:- You can "compile" front-end artifacts (images, javascript, css etc) into (binary and/or compressed) assets and have them properly distributed inside a [wheel](https://peps.python.org/pep-0427/) (which is a Python Package). Otherwise you'll need to rebuild your assets when you install dependencies from source distributions (via git dependency links).- The same goes for other artifacts, like translation files (no more binary `.mo` files in your repository!)- You wrote Python extensions in another language like C(++) / Rust or any other language and you want to compile these only when you update the extensions.- It speeds up your CI / CD pipeline: Reusable packages need to be build and released only once, instead of each time you need to build your project.- It forces you to separate concerns: Reusable apps can also be tested in isolated sandboxes, making the code less dependent on your project requirements. A "package oriented mindset" is a sustainable way of managing software in the long term.And there are many more advantages, like forcing you to implement (proper) versioning, implement (auto-)update strategies (dependabot) or even decide to open source one of your private packages.Create a new repositoryCreating a new AWS Codeartifact repository is fairly simple from the AWS console: `CodeArtifact -> Create Repository`:Or if you are using [Terraform](https://www.terraform.io/), the most basic configuration would be:```bashresource "aws_codeartifact_domain" "mydomain" {domain = "mydomain"}resource "aws_codeartifact_repository" "myrepo" {repository = "myrepo"domain = aws_codeartifact_domain.mydomain.domain}data "aws_codeartifact_repository_endpoint" "pypi_endpoint" {domain = aws_codeartifact_domain.mydomain.domainrepository = aws_codeartifact_repository.myrepo.repositoryformat = "pypi"}```AccessWhen an AWS CodeArtifact repository is created, the URL for accessing the (pypi) repository will be in the following format:```shellhttps://<domain>-<account>.d.codeartifact.<aws-region>.amazonaws.com/pypi/<repository>```An example using the terraform configuration above in the `eu-west-1` region:```shellhttps://mydomain-111122223333.d.codeartifact.eu-west-1.amazonaws.com/pypi/myrepo```For using the index server to query / download packages, you'll need to add a suffix `/simple` to this URL:```https://mydomain...amazonaws.com/pypi/myrepo/ # <-- for publishinghttps://mydomain...amazonaws.com/pypi/myrepo/simple # <-- for querying / downloading```AuthenticationAWS CodeArtifact uses JWT tokens for authentication. These tokens are valid for a maximum of 12 hours, but expiration times can be shorter too (suitable for CI environments). You need to setup the following permissions for the AWS user / IAM role to be able to query the endpoint and download the packages:```- codeartifact:GetAuthorizationToken- codeartifact:ReadFromRepository- sts:GetServiceBearerToken````When an AWS IAM user / role has these permissions, you can query for a token with the [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html). Let's export the whole command as an environment variable. You could add this to your `.bashrc` or `.zshrc`:```shell$ export AWS_CODEARTIFACT_TOKEN_COMMAND=`aws codeartifact get-authorization-token --domain mydomain --domain-owner 111122223333 --query authorizationToken --output text````Now you can use this token as a password with `aws` as username.With poetry```shell$ poetry source add --priority=supplemental aws-codeartifact-myrepo https://mydomain-111122223333.d.codeartifact.eu-west-1.amazonaws.com/pypi/myrepo/simple$ poetry config http-basic.aws-codeartifact-myrepo aws $(eval $AWS_CODEARTIFACT_TOKEN_COMMAND)```With pip```shell$ pip install -i https://aws:$(eval $AWS_CODEARTIFACT_TOKEN_COMMAND)@mydomain-111122223333.d.codeartifact.eu-west-1.amazonaws.com/pypi/myrepo/simple <my-private-package>````Or, you could set the credentials for a specific site like so:```shellpip config set site.index-url https://aws:$(eval $AWS_CODEARTIFACT_TOKEN_COMMAND)@mydomain-606718280940.d.codeartifact.eu-west-1.amazonaws.com/pypi/myrepo/simple/```NetRCPip and poetry also work with [netrc](https://pip.pypa.io/en/stable/topics/authentication/#netrc-support). I wrote a simple [update-netrc CLI](https://github.com/maerteijn/update-netrc) to set the credentials for a specific host:```shell$ update-netrc update http://mydomain-111122223333.d.codeartifact.eu-west-1.amazonaws.com/pypi/myrepo/simple --login aws --password $(eval $AWS_CODEARTIFACT_TOKEN_COMMAND)```This command can be easily integrated in CI systems like Github actions or Gitlab CI with a token which is valid for a limited time.PublishingPublishing packages is fairly easy as CodeArtifact is 100% compatible with the PyPI API. Your AWS account / IAM role needs the following permissions to allow uploading packages:```bash- codeartifact:GetAuthorizationToken- codeartifact:GetRepositoryEndpoint- codeartifact:PublishPackageVersion- codeartifact:PutPackageMetadata- sts:GetServiceBearerToken```First we export the "publishable" repository as an environment variable:```shell$ export AWS_CODEARTIFACT_PYPI_REPOSITORY_URL=https://mydomain-111122223333.d.codeartifact.eu-west-1.amazonaws.com/pypi/myrepo```With poetryWith poetry you should configure a repository and then you can use the poetry CLI to publish the package:```shell# Add the repository and configure the token$ poetry source add --priority=supplemental aws-codeartifact-myrepo-publish $AWS_CODEARTIFACT_PYPI_REPOSITORY_URL$ poetry config http-basic.aws-codeartifact-myrepo-publish aws $(eval $AWS_CODEARTIFACT_TOKEN_COMMAND)```Now we can publish the package after building it:```shell$ poetry build$ poetry publish --repository aws-codeartifact-myrepo-publish```With TwineWith [Twine](https://github.com/pypa/twine) you can upload your package with the CLI in a single line:```shell$ twine upload --repository-url $AWS_CODEARTIFACT_PYPI_REPOSITORY_URL --username aws --password $(eval $AWS_CODEARTIFACT_TOKEN_COMMAND) mypackage.whl```Useful links- [https://jasonstitt.com/private-packages-codeartifact-poetry-workflow](https://jasonstitt.com/private-packages-codeartifact-poetry-workflow), an excellent tutorial with [poetry](https://python-poetry.org/)- [https://docs.aws.amazon.com/codeartifact/latest/ug/python-configure-pip.html](https://docs.aws.amazon.com/codeartifact/latest/ug/python-configure-pip.html), configure pip with AWS CodeArtifact