On May 10, 2024, the venerable British science fiction institution Doctor Who will inaugurate a new era. Again.¹ While waiting for the new series to begin, I thought it would be fun to recount my introduction to Doctor Who:

Late one Saturday night, sometime in the early-to-mid 1990s, I happened on a program on my local PBS station that I thought was utterly bizarre. It was some kind of Sci-Fi, but clearly (a) British, (b) made in the past, and (c) made on a tight budget. The aliens were just actors painted head-to-toe in green makeup with what looked like torn-up knit sweaters for hair. The show featured a giant squid-like monster that was badly composited in long shots while closeups involved only an occasional giant tentacle obviously made of foam rubber. (It must have smelled terrible when it got wet.) I watched a few minutes then changed the channel not having any idea what I had just seen.²

Sometime later, again late one Saturday night in the early-to-mid 1990s, I happened on a program on my local PBS station. It was very clearly (a) British, (b) made in the past, and (c) made on a tight budget but this time the main plot captured my imagination. There was a group of mechanical-looking robots and another group of human-looking robots that had been locked in a centuries-long conflict. The conflict had long ago ground to a stalemate as each side’s battle computer was totally logical and could therefore predict and counter the strategies of the other side’s battle computer. In spite of the dodgy special effects and even dodgier costumes for the human-like robots (once again using knitwear for hair), the premise hooked me.

I didn’t know it at the time, but those two shows I happened on were part of the same long-running British Sci-Fi show, Doctor Who. The two episodes³ I had seen were The Power of Kroll and Destiny of the Daleks, respectively. After that second encounter, I made it a point to watch my local PBS station on Saturday nights.⁴ (It wasn’t until years later that I saw The Power of Kroll again and remembered scenes from that weird show I had once seen.)

I’ve written a lot recently about Swift, but Python remains the programming language that I use the most. And given that Visual Studio Code’s Dev Containers feature has become my favorite way to manage my development environments, it occurred to me that I should detail my Dev Container setup for Python. As a reminder, Dev Containers allow you to use one or more Docker containers as a development environment, enabling you to have a fully isolated, similar-to-production, Linux-based environment in which to code. And all of the configuration files can be committed to your repository, allowing all team members to consistently use the same environment.

First, The Inevitable Aside on Python Packaging

One aspect of Python that many developers find extremely compelling is the vibrant and active community of third-party, open source packages. If you ever find yourself running into a problem and thinking, “Surely someone out there has figured this out before,” there’s a very good chance that an installable package is available that solves it for you. Indeed, at the time this post was written, there are 415,359 packages available in Python’s canonical package repository, the Python Package Index (PyPI).¹

Unforunately, while Python has long had this cornucopia of installable packages, the details of installing those packages has been... not great. By default, packages are installed in a directory that is global to Python; any Python project you use or work on accesses the same global set of packages. That’s fine until you happen to work on multiple projects with conflicting dependencies. So then came virtual environments which allowed you to have separate pockets of Python on your system, each with their own set of packages. But virtual enviroments didn’t help you keep track of the packages you needed to install, so along came pip-tools, which would write out your dependencies to a text file. But pip-tools didn’t help you manage your virtual environments or help you manage conflicts among your dependencies’ dependencies so along came pipenv and poetry to help you with all of that.

You might be forgiven for thinking that none of this packaging messiness applies to Dev Containers, since they give each project a fully isolated virtual machine. And indeed, it is possible to simply use the global Python environment inside each project’s Dev Container. I have found this solution to be lacking, however, for the following reasons:

1. I invaribly run into permissions problems trying to install in the Dev Container’s global Python environment as it is owned by root while development usually happens under the vscode user. This can be worked around, but it tends to leave things in a quirky state.

2. It is still good practice to track your project’s dependencies and to pin known compatible verions of them in a way that lends itself to reliable replication. And it is still good practice to update your dependencies and your dependencies’ dependencies on a regular basis. The pipenv and poetry tools were designed for just this sort of thing and they excel at it.

3. Other developers on your team may not be using Dev Containers, and your production environment certainly won’t, so you still need good dependency hygiene even outside of a Dev Container context.

All of this is to say that I still recommend using virtual environments and dependency managers inside of Dev Containers. More specifically, I strongly recommend using either pipenv or poetry to manage your dependency tree. They are both excellent, so it is difficult to universally recommend one over the other. That said, poetry has adopted the pyproject.toml file as the way to record top-level dependencies. pyproject.toml has become the recommended way to record a project’s metadata, so these days I tend to use poetry.

Now, the Actual Dev Container Setup

First, the docker-compose.yml file creates our overall container environment:

.devcontainer/docker-compose.yml

version: '3.8'
services:
  app:
    build:
      context: ..
      dockerfile: .devcontainer/Dockerfile
      args:
        VARIANT: "3.10"
        NODE_VERSION: "none"
    volumes:
      - ..:/workspace:cached
    command: sleep infinity
    network_mode: service:db
    user: vscode
    env_file: .env
  db:
    image: postgres:14.5
    restart: unless-stopped
    env_file: postgres.env
volumes:
  postgres-data:

Based mostly on Microsoft’s default, this compose file specifies 3.10 as the target version of Python² and avoids installing Node. Some key differences from the default version of the file:

• I specify an environment file from which environment variables will be set when the container builds. This file — .devcontainer/.env — must be present in the .devcontainer folder and valid or else the container will fail to build.
• I prefer to pin the version of PostgreSQL I use rather than just latest, as I want to use the same version that is in my production environment.
• I also use an environment file — .devcontainer/postgres.env — to specify the database username and password. Be sure to add *.env to your .gitignore file so you don’t accidentally include the environment files in your repository.

The contents of your .env file will depend on the kind of project you are working on; you might not even need it at all. The postgres.env is rather simple:

.devcontainer/postgres.env

POSTGRES_DB={{ preferred DB name }}
POSTGRES_USER={{ preferred DB username }}
POSTGRES_PASSWORD={{ preferred DB password }}

Next, the Dockerfile for the main development container:

.devcontainer/Dockerfile

ARG VARIANT=3
FROM mcr.microsoft.com/vscode/devcontainers/python:0-${VARIANT}
ENV PYTHONUNBUFFERED 1
ARG NODE_VERSION="none"
RUN if [ "${NODE_VERSION}" != "none" ]; then su vscode -c "umask 0002 && . /usr/local/share/nvm/nvm.sh && nvm install ${NODE_VERSION} 2>&1"; fi
RUN apt-get update && export DEBIAN_FRONTEND=noninteractive \
    && apt-get -y install --no-install-recommends \
    postgresql-client
RUN /usr/local/py-utils/bin/pipx install --system-site-packages --pip-args '--no-cache-dir --force-reinstall' isort && \
    /usr/local/py-utils/bin/pipx install --system-site-packages --pip-args '--no-cache-dir --force-reinstall' poetry && \
    pip install --upgrade pip
COPY .devcontainer/config/pypoetry_config.toml /home/vscode/.config/pypoetry/config.toml
RUN chown -R vscode:vscode /home/vscode/.config && \
    /usr/local/py-utils/bin/poetry completions bash > /etc/bash_completion.d/poetry.bash-completion && \
    python -m venv /workspace/.venv --prompt feria && \
    chown -R vscode:vscode /workspace/.venv

Again, this is based mostly on Microsoft’s default but with a number of noteworthy changes:

• The postgresql-client OS package is installed in order to make the psql command available in the development environment.
• pipx, which comes pre-installed in the Python image, is used to globally install isort and poetry.
• A configuration file for poetry is copied into the container (more on that in a bit).
• Shell completion hints for poetry in the bash shell are installed.
• The development virtual environment is created. As indicated above, this can be skipped, but I find it easier to work with a virtual environment rather than the root-owned global Python environment in the container. By placing the virtual environment inside the /workspace folder, it will survive rebuilds of the container and some utilities in VS Code will detect it automatically.

The configuration file for poetry simply tells poetry not to create new virtual environments. This is not absolutely necessary. If you prefer, you can omit both this file and the virtual environment creation step in the Dockerfile; poetry will then create a virtual environment for you when invoked. I just prefer to have control over where the virtual environment is created for the reasons mentioned above.

.devcontainer/config/pypoetry_config.toml

[virtualenvs]
create = false

Finally, the devcontainer.json file, based on Microsoft’s default:

.devcontainer/devcontainer.json

{
    "name": "{{ project name }}",
    "dockerComposeFile": "docker-compose.yml",
    "service": "app",
    "workspaceFolder": "/workspace",
    "settings": {
        "editor.formatOnSave": true,
        "python.analysis.extraPaths": [
            "/workspace/source"
        ],
        "python.defaultInterpreterPath": "/workspace/.venv/bin/python",
        "python.formatting.blackPath": "/usr/local/py-utils/bin/black",
        "python.formatting.provider": "black",
        "python.languageServer": "Pylance",
        "python.linting.enabled": true,
        "python.linting.mypyPath": "/usr/local/py-utils/bin/mypy",
        "python.linting.pylintEnabled": true,
        "python.linting.pylintPath": "/workspace/.venv/bin/pylint",
        "python.testing.pytestPath": "/usr/local/py-utils/bin/pytest",
        "isort.path": [
            "/usr/local/py-utils/bin/isort"
        ],
        "rewrap.wrappingColumn": 88,
        "[python]": {
            "editor.codeActionsOnSave": {
                "source.organizeImports": true
            }
        }
    },
    "extensions": [
        "ms-python.python",
        "ms-python.vscode-pylance",
        "bungcip.better-toml"
    ],
    "forwardPorts": [
        {{ your project’s preferred port, default 8000 for Django }}
    ],
    "postCreateCommand": "VIRTUAL_ENV=\"/workspace/.venv\" PATH=\"$VIRTUAL_ENV/bin:$PATH\" poetry install --no-interaction --no-ansi --with dev",
    "remoteUser": "vscode"
}

Including some VS Code settings here can help your team establish consistent practices around code formatting and linting. Just as with the customizations above, it is not necessary but I find it to be good practice.

• A number of widely-used Python development utilities, like Black and pytest come pre-installed in the Python dev container image; their paths are referenced in the devcontainer.json file. I have activated the ones I prefer to use.
• Although Pylint also comes pre-installed, I find it works better when installed inside the virtual environment; that is the version that is referenced. We installed isort earlier, so that is also referenced.
• I install a TOML extension for highlighting the pyproject.toml file; be sure to include any other extensions you like to use.
• Finally, the postCreateCommand automatically uses poetry to install any dependencies, including dev dependencies.

As mentioned previously, Swift 5.7 (currently in beta) looks to be another significant release. I’ve written previously about using the Dev Containers feature of Visual Studio Code to do server-side Swift development. The setup then was rather complicated as there were no pre-built Docker containers for the then-current beta version of Swift, 5.5. More recently, I’ve been curious to give Swift 5.7 a try, so I wanted to update the Dev Container configuration I was using previously. The good news now is that pre-built containers of the Swift 5.7 beta are available, so the setup is significantly simpler.

As before, we’ll start with the dev container configuration file itself:

.devcontainer/devcontainer.json

{
    "name": "{{ project name }}",
    "dockerComposeFile": "docker-compose.yml",
    "service": "app",
    "workspaceFolder": "/workspace",
    "settings": {
        "lldb.library": "/usr/lib/liblldb.so"
    },
    "extensions": [
        "sswg.swift-lang"
    ],
    "forwardPorts": [
        {{ your project’s preferred port, default 8080 for Vapor }},
        {{ your database’s port, default 5432 for PostgreSQL }}
    ],
    "remoteUser": "vscode"
}

This differs from the dev container template for Swift because we are going to use a docker-compose.yml file. Using docker-compose will allow us to bring up a database container at the same time as the rest of our project. Note also that we are using a different Swift extension than last time. This is the new extension provided by the Swift Server Work Group that I mentioned in my last post.

Here, then, is the corresponding docker-compose file:

.devcontainer/docker-compose.yml

version: '3.8'
services:
  app:
    build:
      context: ..
      dockerfile: .devcontainer/Dockerfile
      args:
        NODE_VERSION: "none"
    volumes:
      - ..:/workspace:cached
    command: sleep infinity
    network_mode: service:db
    env_file: .env
    privileged: true
  db:
    image: postgres:14.4
    restart: unless-stopped
    volumes:
      - postgres-data:/var/lib/postgresql/data
      - ./db-scripts:/docker-entrypoint-initdb.d
    environment:
      POSTGRES_USER: {{ your preferred DB user }}
      POSTGRES_DB: {{ your preferred DB name }}
      POSTGRES_PASSWORD: {{ your DB user’s password }}
volumes:
  postgres-data: null

There are a few things to note here. First, if you would like to have a version of Node available (presumably for front-end development), replace the “none” in NODE_VERSION: "none" with your preferred version. Next, note the inclusion of an environment variable definition file (.env); this isn’t necessary but can be handy. Next, note the privileged: true line; this is necessary for some of LLDB’s functionality (including the Swift REPL) and should never be left in a production configuration. Finally, note the ./db-scripts:/docker-entrypoint-initdb.d line; this allows us to insert startup scripts into the PostgreSQL container. This is specific to PostgreSQL’s Docker container configuration so if you are using a different database it won’t work. I’ve added it in order to automatically create a test database for use in running the automated test suite. The directory referenced, db-scripts, contains a single file:

.devcontainer/db-scripts/create_test_db.sh

#!/bin/bash
set -e
set -u
echo "  Creating test database."
psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" <<-EOSQL
    CREATE DATABASE {{ your preferred DB name }}_test;
    GRANT ALL PRIVILEGES ON DATABASE {{ your preferred DB name }}_test TO {{ your preferred DB user }};
EOSQL

Now on to the Dockerfile:

.devcontainer/Dockerfile

FROM swiftlang/swift:nightly-5.7-focal
ARG INSTALL_ZSH="false"
ARG UPGRADE_PACKAGES="true"
ARG USERNAME=vscode
ARG USER_UID=1000
ARG USER_GID=$USER_UID
COPY .devcontainer/library-scripts/common-debian.sh /tmp/library-scripts/
RUN apt-get update && export DEBIAN_FRONTEND=noninteractive \
    && /bin/bash /tmp/library-scripts/common-debian.sh "${INSTALL_ZSH}" "${USERNAME}" "${USER_UID}" "${USER_GID}" "${UPGRADE_PACKAGES}" "true" "true" \
    && apt-get autoremove -y && apt-get clean -y && rm -rf /var/lib/apt/lists/* && rm -rf /tmp/library-scripts
ARG NODE_VERSION="none"
ENV NVM_DIR=/usr/local/share/nvm
ENV NVM_SYMLINK_CURRENT=true \
    PATH=${NVM_DIR}/current/bin:${PATH}
COPY .devcontainer/library-scripts/node-debian.sh /tmp/library-scripts/
RUN bash /tmp/library-scripts/node-debian.sh "${NVM_DIR}" "${NODE_VERSION}" "${USERNAME}" \
    && rm -rf /var/lib/apt/lists/* /tmp/library-scripts
COPY .devcontainer/library-scripts/install-vapor.sh /tmp/library-scripts/
RUN bash /tmp/library-scripts/install-vapor.sh \
    && rm -rf /tmp/library-scripts

This Dockerfile references three scripts in a library-scripts directory. Two of these come directly from the standard dev container template for Swift, common-debian.sh and node-debian.sh. The third is a simple script to install the Vapor toolbox; it is completely optional and you can skip it if you like. Here is that script:

.devcontainer/library-scripts/install-vapor.sh

#!/bin/bash
cd /opt
git clone https://github.com/vapor/toolbox.git
cd toolbox
git checkout "{{ current toolbox version }}"
make install

When last I wrote about the Swift programming language and the Vapor web framework, I focused on how the server-side development experience was on the cusp of becoming much, much better by the introduction of the async and await keywords. It’s been a minute, several new versions of Swift and Vapor have been released, so I think it’s time to write an update.

Swift 5.5 was officially released on September 20, 2021 and did indeed include the initial implementation of concurrency as a language-level feature, including the async and await keywords. This was followed up by the release of Swift 5.6 on March 14, 2022. Swift 5.6 was something of a smaller release, seemingly focused more on improvements to the type system. Swift 5.7 is now in the beta phase of development and builds on the concurrency and type system features of the last two versions along with a number of other features like a new regular expression engine, syntax improvements for generic typing, and more. Swift 5.7 feels like another big release and includes a number of changes that will improve the ergonomics of using Swift. Based on the pattern of recent Swift releases, version 5.7 seems destined for release sometime in September.

Vapor, too, has been updated since that last post. The async/await branch was merged into the mainline Vapor release with version 4.50.0 on October 26, 2021¹. I had previously feared that Vapor would be unable to adopt the async/await pattern until one of its key dependencies — SwiftNIO — did. Happily, that proved not to be the case as Vapor was able to wrap the concurrency features of the SwiftNIO library, making the Vapor API async/await compatible.

Finally, there has been a significant improvement when it comes to the code editing experience outside of Xcode. I previously reported disappointment with Swift’s canonical Language Server Protocol (LSP) implementation, SourceKit-LSP — going so far as to call it “nearly useless”. I’m thrilled to be able to say now that is absolutely no longer the case. Combined with the Swift Server Work Group’s new plugin for Visual Studio Code, working with Swift in VS Code is now much more enjoyable.² I still wouldn’t consider SourceKit-LSP to be speedy, exactly, but it is now responsive enough to be generally very useful. I have also found it to be much more reliable, able to work without crashing and able to register locally-defined types much more readily.

As the owner of its platforms, Apple can through shear force of will ensure that Swift is a viable choice of language for development on Apple platforms.³ That isn’t true, however, of other contexts. I’ve long felt that Swift had great potential as a server-side language on non-Apple platforms but that potential wasn’t being realized. The adoption of async/await feels like a significant step forward in that regard. I haven’t written about it much in this post but Swift 5.7 also feels like a significant step forward with first-class regular expression support, simplified syntax for generic typing, updates to the concurrency features, and more. Only time will tell if Swift will see broader adoption as a server-side language but I’m finally starting to feel that there’s a good foundation now for that to happen.

I have glimpsed the future of async and await in Swift and it is gloriously mundane, just as it should be.

Eager as I am to see language-level support for asynchronous programming land in Swift, I was very excited to see the Vapor project launch branches of key components. Naturally, I wanted to try them out.

Although some asynchronous features landed in Swift 5.4, it seems like Swift 5.5, currently in an alpha stage, is likely to be a watershed release when it comes to built-in support for the co-routine style of concurrency enabled by the async and await keywords. Naturally, then, I wanted to test things out in Swift 5.5. In order to do so both in a Linux environment (I’m mostly interested in Swift for server-side programming) and without affecting my day-to-day environment, I decided to take advantage of Microsoft Visual Studio Code’s Dev Containers feature.

Swift 5.5 in Visual Studio Code’s Dev Containers

VS Code’s Dev Containers are a great way to engage in server-side (i.e. Linux-based) development from your normal desktop environment — be that Mac or Windows. But there is some setup involved, especially if you are going to use an alpha-stage version of Swift for which pre-built container images are not yet available. There’s a lot to say about Dev Containers and how they work but I’ll leave most of that to Microsoft’s resources. The TL;DR version is this: if you add a .devcontainer folder to your project with a few key configuration files, you can have VS Code automatically build a complete Linux-based, Docker-based development environment. If you are doing server-side development and need access to databases or other similar resources, you can use Docker Compose to include those dependencies. And if you commit that folder to your source repository, anyone on your team can easily and automatically get the same environment.

Here, then, is my server-side Swift Dev Container configuration:

.devcontainer/devcontainer.json

{
    "name": "Swift 5.5 Playground",
    "dockerComposeFile": "docker-compose.yml",
    "service": "app",
    "workspaceFolder": "/workspace",
    "settings": {
        "terminal.integrated.defaultProfile.linux": "/bin/bash",
        "lldb.adapterType": "bundled",
        "lldb.executable": "/usr/bin/lldb",
        "sde.languageServerMode": "sourcekit-lsp",
        "sourcekit-lsp.serverPath": "/usr/bin/sourcekit-lsp"
    },
    "extensions": [
        "vknabel.vscode-swift-development-environment",
        "vadimcn.vscode-lldb",
        "ms-azuretools.vscode-docker",
        "mtxr.sqltools",
        "mtxr.sqltools-driver-pg"
    ],
    "forwardPorts": [5432, 8080],
    "remoteUser": "vscode"
}

This is, more or less, a standard VS Code configuration file with a few additional keys specific to Dev Containers. You may notice the inclusion of Swift’s language server, SourceKit-LSP; we’ll talk more about that later. Before that, we should take a look at that docker-compose.yml file:

.devcontainer/docker-compose.yml

version: '3'
services:
  app:
    build:
      context: ..
      dockerfile: .devcontainer/Dockerfile
    volumes:
      - ..:/workspace:cached
    # Overrides default command so things don't shut down after the process ends.
    command: sleep infinity
    # Runs app on the same network as the database container, allows "forwardPorts" in devcontainer.json function.
    network_mode: service:db
    # Uncomment the next line to use a non-root user for all processes.
    user: vscode
    env_file: .env
  db:
    image: postgres:13.2
    restart: unless-stopped
    volumes:
      - postgres-data:/var/lib/postgresql/data
    environment:
      POSTGRES_USER: playground
      POSTGRES_DB: playground
      POSTGRES_PASSWORD: {make up a password}
volumes:
  postgres-data:

The db section ensures that we have a working instance of the estimable open source SQL database, PostgreSQL. (Note also in the devcontainer.json file that Postgres’ port — 5432 — is forwarded so we can access the database outside of the container and even outside of VS Code.) The main action, though, is in the app section; this incorporates the container in which we will do our development. A .env will load any environment variables we need (handy for database connection information), a workspace volume is defined so that our code will live both inside and outside the container, and the Dockerfile that defines the container is called out. Let’s take a look at that next:

.devcontainer/Dockerfile

FROM ubuntu:20.04

RUN export DEBIAN_FRONTEND=noninteractive DEBCONF_NONINTERACTIVE_SEEN=true && apt-get -q update && \
    apt-get -q install -y \
    binutils \
    build-essential \
    git \
    gnupg2 \
    libc6-dev \
    libcurl4 \
    libedit2 \
    libgcc-9-dev \
    libpython3.8 \
    libsqlite3-0 \
    libsqlite3-dev \
    libstdc++-9-dev \
    libxml2 \
    libz3-dev \
    pkg-config \
    tzdata \
    zlib1g-dev \
    && rm -r /var/lib/apt/lists/*

# Everything up to here should cache nicely between Swift versions, assuming dev dependencies change little

# gpg --keyid-format LONG -k FAF6989E1BC16FEA
# pub   rsa4096/FAF6989E1BC16FEA 2019-11-07 [SC] [expires: 2021-11-06]
#       8A7495662C3CD4AE18D95637FAF6989E1BC16FEA
# uid                 [ unknown] Swift Automatic Signing Key #3 <swift-infrastructure@swift.org>
ARG SWIFT_SIGNING_KEY=8A7495662C3CD4AE18D95637FAF6989E1BC16FEA
ARG SWIFT_PLATFORM=ubuntu
ARG OS_MAJOR_VER=20
ARG OS_MIN_VER=04
ARG SWIFT_WEBROOT=https://swift.org/builds/swift-5.5-branch

ENV SWIFT_SIGNING_KEY=$SWIFT_SIGNING_KEY \
    SWIFT_PLATFORM=$SWIFT_PLATFORM \
    OS_MAJOR_VER=$OS_MAJOR_VER \
    OS_MIN_VER=$OS_MIN_VER \
    OS_VER=$SWIFT_PLATFORM$OS_MAJOR_VER.$OS_MIN_VER \
    SWIFT_WEBROOT="$SWIFT_WEBROOT/$SWIFT_PLATFORM$OS_MAJOR_VER$OS_MIN_VER"

RUN echo "${SWIFT_WEBROOT}/latest-build.yml"

RUN set -e; \
    # - Grab curl here so we cache better up above
    export DEBIAN_FRONTEND=noninteractive \
    && apt-get -q update && apt-get -q install -y curl && rm -rf /var/lib/apt/lists/* \
    # - Latest Toolchain info
    && export $(curl -s ${SWIFT_WEBROOT}/latest-build.yml | grep 'download:' | sed 's/:[^:\/\/]/=/g')  \
    && export $(curl -s ${SWIFT_WEBROOT}/latest-build.yml | grep 'download_signature:' | sed 's/:[^:\/\/]/=/g')  \
    && export DOWNLOAD_DIR=$(echo $download | sed "s/-${OS_VER}.tar.gz//g") \
    && echo $DOWNLOAD_DIR > .swift_tag \
    # - Download the GPG keys, Swift toolchain, and toolchain signature, and verify.
    && export GNUPGHOME="$(mktemp -d)" \
    && curl -fsSL ${SWIFT_WEBROOT}/${DOWNLOAD_DIR}/${download} -o latest_toolchain.tar.gz \
    ${SWIFT_WEBROOT}/${DOWNLOAD_DIR}/${download_signature} -o latest_toolchain.tar.gz.sig \
    && curl -fSsL https://swift.org/keys/all-keys.asc | gpg --import -  \
    && gpg --batch --verify latest_toolchain.tar.gz.sig latest_toolchain.tar.gz \
    # - Unpack the toolchain, set libs permissions, and clean up.
    && tar -xzf latest_toolchain.tar.gz --directory / --strip-components=1 \
    && chmod -R o+r /usr/lib/swift \
    && rm -rf "$GNUPGHOME" latest_toolchain.tar.gz.sig latest_toolchain.tar.gz \
    && apt-get purge --auto-remove -y curl

# Print Installed Swift Version
RUN swift --version

RUN echo '[ ! -z "$TERM" -a -r /etc/motd ] && cat /etc/motd' \
    >> /etc/bash.bashrc; \
    echo " ################################################################\n" \
    "#\t\t\t\t\t\t\t\t#\n" \
    "# Swift Nightly Docker Image\t\t\t\t\t#\n" \
    "# Tag: $(cat .swift_tag)\t\t#\n" \
    "#\t\t\t\t\t\t\t\t#\n"  \
    "################################################################\n" > /etc/motd

# [Option] Install zsh
ARG INSTALL_ZSH="false"
# [Option] Upgrade OS packages to their latest versions
ARG UPGRADE_PACKAGES="true"

# Install needed packages and setup non-root user. Use a separate RUN statement to add your own dependencies.
ARG USERNAME=vscode
ARG USER_UID=1000
ARG USER_GID=$USER_UID
COPY .devcontainer/library-scripts/common-debian.sh /tmp/library-scripts/
RUN apt-get update && export DEBIAN_FRONTEND=noninteractive \
    && /bin/bash /tmp/library-scripts/common-debian.sh "${INSTALL_ZSH}" "${USERNAME}" "${USER_UID}" "${USER_GID}" "${UPGRADE_PACKAGES}" "true" "true" \
    && apt-get -y install --no-install-recommends lldb python3-minimal libpython3.7 \
    && apt-get autoremove -y && apt-get clean -y && rm -rf /var/lib/apt/lists/* && rm -rf /tmp/library-scripts

# Install SourceKit-LSP
RUN git clone -b "release/5.5" https://github.com/apple/sourcekit-lsp.git \
    && cd sourcekit-lsp \
    && swift build -Xcxx -I/usr/lib/swift -Xcxx -I/usr/lib/swift/Block \
    && cp .build/debug/sourcekit-lsp /usr/bin/

Because the Swift project isn’t yet publishing pre-built Docker containers for Swift 5.5, this Dockerfile is necessarily quite involved. Most of it is taken from Swift’s own Dockerfile for 5.5 with another section taken from Microsoft’s Dev Container Dockerfile for Swift and a final section to compile SouceKit-LSP. Note also the inclusion of library-scripts/common-debian.sh which you can find in Microsoft’s Dev Containers repository and must be included in your .devcontainer folder.

With the dev container definition complete, you can tell VS Code to build and enter your development environment by selecting the command “Remote-Containers: Open Folder in Container”. The first time you run this, it will involve downloading base containers and building your container on top, so it will take a while.

Finally, the Code

With the development environment finally ready, we can turn to actual code. My main goal was to take the regular Vapor template project and tweak it just enough to try out the async and await keywords. I ended up recreating the template manually, but there’s no reason you can’t start with the Vapor toolbox instead. With the toolbox installed, you run the command vapor new {project name}; be sure to answer yes to the question about using Fluent and to select PostgreSQL as the database engine.

To begin, we need to customize the Package.swift file to pull from the relevant async-await branches of the Vapor sources:

Package.swift

// swift-tools-version:5.5
import PackageDescription

let package = Package(
    name: "VaporAsyncTry",
    dependencies: [
        .package(url: "https://github.com/vapor/vapor.git", .branch("async-await")),
        .package(url: "https://github.com/vapor/fluent-kit.git", .branch("async-await")),
        .package(url: "https://github.com/vapor/fluent.git", .branch("main")),
        .package(url: "https://github.com/vapor/fluent-postgres-driver.git", .branch("main"))
    ],
    targets: [
        .target(
            name: "App",
            dependencies: [
                .product(name: "FluentKit", package: "fluent-kit"),
                .product(name: "Fluent", package: "fluent"),
                .product(name: "FluentPostgresDriver", package: "fluent-postgres-driver"),
                .product(name: "Vapor", package: "vapor"),
            ],
            swiftSettings: [
                .unsafeFlags([
                    "-Xfrontend", "-enable-experimental-concurrency",
                    "-Xfrontend", "-disable-availability-checking",
                ])
            ]
        ),
        .executableTarget(
            name: "Run",
            dependencies: [.target(name: "App")]
        ),
        .testTarget(name: "AppTests", dependencies: [
            .target(name: "App"),
            .product(name: "XCTVapor", package: "vapor"),
        ])
    ]
)

Notice also the swiftSettings section with the "-enable-experimental-concurrency" clause. This setting was introduced in Swift 5.4 in order to enable the earliest versions of async await support. I’m hopeful that this setting will not be necessary in the final release of Swift 5.5, but for now it does seem to be needed.

Now, at long last, we can update the example TodoController from Vapor’s default template to take advantage of async and await:

Sources/App/Controllers/TodoController.swift

import Fluent
import Vapor

struct TodoController: RouteCollection {
    func boot(routes: RoutesBuilder) throws {
        let todos = routes.grouped("todos")
        todos.get(use: index)
        todos.post(use: create)
        todos.group(":todoID") { todo in
            todo.put(use: update)
            todo.delete(use: delete)
        }
    }

    func index(request: Request) async throws -> [Todo] {
        return try await Todo.query(on: request.db).all()
    }

    func create(request: Request) async throws -> Todo {
        let todo = try request.content.decode(Todo.self)
        try await todo.save(on: request.db)
        return todo
    }

    func update(request: Request) async throws -> Todo {
        guard let todo = try await Todo.find(request.parameters.get("todoID"), on: request.db) else {
            throw Abort(.notFound)
        }
        let updatedTodo = try request.content.decode(Todo.self)
        todo.title = updatedTodo.title
        try await todo.save(on: request.db)
        return todo
    }

    func delete(request: Request) async throws -> HTTPStatus {
        guard let todo = try await Todo.find(request.parameters.get("todoID"), on: request.db) else {
            throw Abort(.notFound)
        }
        try await todo.delete(on: request.db)
        return HTTPStatus.ok
    }
}

The first thing to notice is that none of the return values are wrapped in EventLoopFuture. Next, observe that there are no nested callbacks, no need for map or flatMap. Function declarations include the async keyword and calls to the database are preceded by await but otherwise this looks just like synchronous code.

One thing I couldn’t get working was Vapor’s unwrap(or: Abort(.notFound)) shortcut for handling database queries that may or may not return a value. Instead, I had to use the somewhat more verbose guard let ... else { throw Abort(.notFound) } construction. Nevertheless, I consider this to be a huge improvement over the nested callbacks previously required.

Regarding sourcekit-lsp

Many — but by no means all — programmers like to have a little boost while programming in the form of intelligent auto-complete, pop-up documentation, etc. Many code editors like VS Code include robust support for these features for many programming languages. Often in the past, each editor had a different API for enabling these features, meaning that a plugin written for TextMate for example wouldn’t work with SublimeText. Microsoft, of all companies, is trying to fix that with their Language Server Protocol, or LSP, specification. LSP allows programming languages to offer universal plugins that work with any text editor supporting LSP.

The Swift programming language has embraced LSP in the form of SourceKit-LSP. In theory, SourceKit-LSP allows editors like VS Code to have all of the same auto-complete features for Swift that Apple’s own Xcode has. In practice, however, SourceKit-LSP has in the past proven to be so slow, buggy, and incomplete as to be nearly useless.

As you can see above, as part of this experiment, I wanted to try out SourceKit-LSP again to see if any progress has been made. I, for one, would very much like to have these features for Swift programming outside of Xcode (Xcode continues to use a different, proprietary engine to drive these features). Alas, I am sorry to say that little progress has been made. SourceKit-LSP remains so slow, buggy, and incomplete as to be nearly useless.

Although my day job is (usually) programming in Python, I have been experimenting with Swift for a few years now. I am neither a Mac nor an iOS developer; these days I’m almost exclusively a backend developer. It should come as no surprise then that I’m most interested in Swift as a potential backend, server-side language. As such, I think I have a somewhat different perspective from many other developers using Swift. Even compared to others interested in server-side Swift, I think I’m outside the norm, as most members of that community that I’ve encountered have been iOS developers who would also like to program the backend for their apps in the same programming language as they use in the apps themselves.

My dalliances with Swift have always come in fits and starts. For me, the language has long been alluring but elusive: I really admire the concepts of the language and its potential but trying to use it outside of the Apple ecosystem has frequently proven very frustrating. That was especially true in the early days, because of the shortcuts that Apple had taken to get Swift out the door.

A Brief Digression on Apple’s Shortcuts

Swift was first released in 2014, as a proprietary programming language available only for Apple’s platforms. Later in 2015, Apple released it as open source, and released a Linux port at the same time. The Linux port, in particular, made it clear that Apple had taken some shortcuts with the initial implementation and release of Swift. Specifically, Swift’s standard library — the built-in functionality that comes with the language — was astonishingly small. It basically consists of just the built-in data types like Int, Double, Array, and Dictionary. Most modern, high-level languages have standard libraries that are far larger. Python’s standard library, for example, includes modules for dates, data parsing, and networking from low-level socket handling to higher-level HTTP serving. But if Swift’s standard library didn’t include any of those things, how were application developers for Apple’s platforms already able to write apps in Swift?

In turned out that on Apple’s platforms, Swift is intimately connected to the runtime environment of Apple’s previously preferred language, Objective-C. Apple’s messaging around this indicated this was to enable seamless interoperability between Swift and Objective-C code but it also meant that Swift code had ready access to Apple’s existing rich set of libraries written in Objective-C. Notably among these libraries is the Foundation framework — a fairly extensive collection of frequently-needed functionality that most of us developers take for granted.

On Linux, however, Swift is entirely separate from the Objective-C runtime and does not use Apple’s system libraries. To Apple’s credit, Swift on Linux was not left entirely in the lurch; there is a Swift-native version of Foundation. The oldest commit¹ dates to 2012, before Swift had even been publicly announced. This version of Foundation, however, “is a substantial reimplementation of the same API” as the old Objective-C version of Foundation and as of this writing, it remains incomplete.² Earlier versions of Swift’s Foundation implementation were pretty rough with missing functionality and even some incompatibilities with the Objective-C version. Today, however, I tend to think that it is, generally speaking, sufficiently complete and compatible as to be generally useable.

Foundation represents what most developers would consider essential functionality for a programming language. It, together with libdispatch and XCTest, constitute Swift’s Core Libraries; libraries considered so essential that they are distributed with Swift itself on non-Apple platforms. That Foundation, in particular, is now useable on non-Apple platforms means that Swift has passed what was previously a significant hurdle to its adoption on Linux.

Swift Today

The Swift project has ambitions well beyond Apple’s platforms. As stated on the Swift homepage, the “goal of the Swift project is to create the best available language for uses ranging from systems programming, to mobile and desktop apps, scaling up to cloud services.” Swift’s utility for systems, mobile, and desktop programming is evident from it’s use by Apple for Apple platforms. So where does Swift stand now for server-side use? Apple, after all, doesn’t really have a server platform³.

It took some time after the open source release of Swift, but in late 2016 the Swift project announced the formation of the Swift Server Work Group, sometimes abbreviated as SSWG. The SSWG’s remit is to create “APIs [that] provide low level ‘server’ functions as the basic building blocks for developing server-side capabilities”. The first fruit borne of the SSWG’s efforts was SwiftNIO, a non-blocking network I/O library. Since then, the group has sponsored a number of useful libraries.

It’s honestly a little hard for me to believe, but before SwiftNIO’s initial release in 2018, each of the early web frameworks in Swift — specifically, Kitura, Perfect, and Vapor — implemented their own low-level networking interfaces. No wonder, then, that SwiftNIO has been enthusiastically adopted, even by those frameworks that previously did that work themselves.

There’s just one problem. SwiftNIO, as denoted by the “N” in its name, is non-blocking. Swift, however, doesn’t have built-in support non-blocking (often referred to as asynchronous) operations, so this achieved by means of a coroutine engine embedded in SwiftNIO itself. Without syntax for asynchronous code in the language itself, however, using SwiftNIO becomes a tedious exercise in callback hell (a.k.a. the pyramid of doom).

Unfortunately, this isn’t just a problem for direct uses of SwiftNIO; anything built on top of SwiftNIO will necessarily inherit the need to incorporate the callback style of coding. The Swift-based web framework Vapor is a prime example of the contagious nature of SwiftNIO’s callback hell. Vapor 4 is built on top of SwiftNIO, including the database layers. As a result, any database operations in Vapor require the use of callbacks. Take, then, this example of a Vapor 4 web request handler to update a row in the Acronym table:

func updateHandler(_ req: Request) throws -> EventLoopFuture<Acronym> {
    let updatedAcronym = try req.content.decode(Acronym.self)
    return Acronym.find(req.parameters.get("acronymID"), on: req.db)
    .unwrap(or: Abort(.notFound)).flatMap { acronym in
        acronym.short = updatedAcronym.short
        acronym.long = updatedAcronym.long
        return acronym.save(on: req.db).map {
            acronym
        }
    }
}

Notice the return type is wrapped in EventLoopFuture. Notice that there are multiple return statements, starting on only the second line of code. Notice the use of flatMap and map to set the callback function (and notice that you must figure out which is appropriate in certain situations as they are subtly different). Notice the deep nesting. None of this code interacts with SwiftNIO directly but with other libraries that do.

The good news is that a solution is in the works. Other languages have adopted the keywords async and await to resolve the syntactic oddities of asynchronous code and the Swift community has resolved to follow suit⁴. Swift Evolution Proposal SE-0296, which includes these keywords, has been accepted and implemented. It remains unclear when this feature will be released properly; it is included in Swift version 5.4 currently in beta but is not enabled by default⁵. Even after Swift itself is updated, SwiftNIO will need to be updated to adopt it, followed in turn by the web frameworks like Vapor. Work on SwiftNIO’s update has started but of course it cannot be released until Swift itself is ready. When the updates to Swift, SwiftNIO, and Vapor are all released, the example above could look something like this:

async func updateHandler(_ req: Request) throws -> Acronym {
    let updatedAcronym = try req.content.decode(Acronym.self)
    let acronym = await Acronym.find(req.parameters.get("acronymID"), on: req.db)
    .unwrap(or: Abort(.notFound))
    acronym.short = updatedAcronym.short
    acronym.long = updatedAcronym.long
    await acronym.save(on: req.db)
    return acronym
}

No nested callbacks. Only one return statement at the end of the function. The return type for the function doesn’t have to be wrapped in EventLoopFuture. No trying to sort out the difference between flatMap and map. But for the use of the async and await keywords, it looks just like synchronous code.

I understand why SwiftNIO was written as an asynchronous library; that is where the entire industry is going. But shoehorning an asynchronous library into a synchronous programming language is awkward, to say the least. In my own humble opinion, there’s just not much point to trying to use SwiftNIO and the frameworks built on top of it right now. Relying on nested callbacks is error-prone, difficult to read, and difficult to debug. More to the point, asynchronous code that uses the async and await keywords is so much better that all asynchronous code in Swift will inevitably have to be rewritten as soon as they are available. I’d really, really like to start using Swift for server-side code right now but I just don’t feel like I can until async and await are available.

Researchers from the V-Dem Institute at the University of Gothenburg in Sweden have published a new study which seeks to quantify various aspects of political parties throughout the world. From the paper itself:

V-Party’s Illiberalism Index shows that the Republican party in the US has retreated from upholding democratic norms in recent years. Its rhetoric is closer to authoritarian parties, such as AKP in Turkey and Fidesz in Hungary. Conversely, the Democratic party has retained a commitment to longstanding democratic standards.

Julian Borger reports on the paper for The Guardian:

The study, published on Monday, shows the party has followed a similar trajectory to [Hungarian political party] Fidesz, which under Viktor Orbán has evolved from a liberal youth movement into an authoritarian party that has made Hungary the first non-democracy in the European Union.

India’s Bharatiya Janata Party (BJP) has been transformed in similar ways under Narendra Modi, as has the Justice and Development party (AKP) in Turkey under Recep Tayyip Erdoğan and the Law and Justice party in Poland. Trump and his administration have sought to cultivate close ties to the leadership of those countries.

[...]

“The data shows that the Republican party in 2018 was far more illiberal than almost all other governing parties in democracies,” the V-Dem study found. “Only very few governing parties in democracies in this millennium (15%) were considered more illiberal than the Republican party in the US.”

You may have seen claims that democracy itself is on the ballot this election. You may have thought that such claims were overblown. I submit to you that they are very, very true.

This Vapor tip is similar to the previous one, though a bit simpler. Of course, one of the most common data types to store in a database backend is text of some kind or other. The Vapor 4 documentation on models encourages you to use Swift’s String type in the model definition:

Sources/App/Models/Galaxy.swift

final class Galaxy: Model {
    // Name of the table or collection.
    static let schema = "galaxies"

    // Unique identifier for this Galaxy.
    @ID(key: .id)
    var id: UUID?

    // The Galaxy's name.
    @Field(key: "name")
    var name: String

    // Creates a new, empty Galaxy.
    init() { }

    // Creates a new Galaxy with all properties set.
    init(id: UUID? = nil, name: String) {
        self.id = id
        self.name = name
    }
}

And the migration documentation encourages you to use Vapor’s .string type in the database column definition:

Sources/App/Migrations/CreateGalaxy.swift

struct CreateGalaxy: Migration {
    // Prepares the database for storing Galaxy models.
    func prepare(on database: Database) -> EventLoopFuture<Void> {
        database.schema("galaxies")
            .id()
            .field("name", .string)
            .create()
    }

    // Optionally reverts the changes made in the prepare method.
    func revert(on database: Database) -> EventLoopFuture<Void> {
        database.schema("galaxies").delete()
    }
}

The .string type here, though, gets translated to an SQL column type of TEXT, which allows strings of unlimited length. I found that odd as many other ORMs, and indeed previous versions of Vapor, default to using a VARCHAR column type. I asked in the Vapor Discord about that a while ago and I was told that the change was made in order to maximize compatibility across database implementations; SQLite, in particular, doesn’t have a concept of a VARCHAR type¹ (although it does alias that name to TEXT so you won’t get an error if you try to define a VARCHAR column). It was also pointed out that in many modern SQL database implementations, TEXT and VARCHAR columns are stored in the same way so there isn’t much of a performance penalty for using a TEXT column. (PostgreSQL’s documentation on text types calls this out specifically.)

All of this is indeed true, but there are nevertheless perfectly valid reasons to prefer VARCHAR columns over TEXT. Often, limiting the length of a string is highly desirable, especially in the context of a system that accepts user input. Knowing that certain strings will never be longer than n characters can be very helpful in laying out the UI and also helpfully limits the scope of edge case testing. It’s also relevant to indexing decisions as many index types are limited to the number of bytes that can be indexed per row.²

With all that background out of the way, how then do we define a VARCHAR column in Vapor 4? Vapor 4 doesn’t include a .varchar type in the same way that it has the .string type, so we’ll need to work around that. The model definition doesn’t need to change; Swift doesn’t particularly care if the string length is limited on the backend. The change, then, needs to be made in the migration, and it really is quite a simple change:

Sources/App/Migrations/CreateGalaxy.swift

struct CreateGalaxy: Migration {
    // Prepares the database for storing Galaxy models.
    func prepare(on database: Database) -> EventLoopFuture<Void> {
        database.schema("galaxies")
            .id()
            .field("name", .custom("VARCHAR(100)"))
            .create()
    }

    // Optionally reverts the changes made in the prepare method.
    func revert(on database: Database) -> EventLoopFuture<Void> {
        database.schema("galaxies").delete()
    }
}

As you can see, all we did was change .string to .custom("VARCHAR(100)"). The .custom type here allows us to pass in arbitrary SQL, so you will need to make certain that (1) the syntax is correct for your chosen database engine (you are using the same database engine in development as in production, right?) and (2) that you aren’t including user-defined input (that would be an exceptionally strange thing to do in a migration anyway).

All code tested with Swift 5.2, Vapor 4.5, Fluent 4.0.0-rc2.2, and PostgreSQL 12 on both macOS 10.15 and Ubuntu Linux 18.04.

Vapor has essentially become the web framework for the Swift programming language so I’ve been doing quite a bit of experimenting with it. Unfortunately, the documentation for the latest version is still rather incomplete as of this writing. As such, I thought I would add to the general body of knowledge about Vapor available on the web.

For a project I’ve started working on, I wanted to add an enumerated type (more commonly known simply as an ENUM) to my PostgreSQL database, use that ENUM as the type on a column on one of my tables, and have a default value. The first thing, then, was to define the ENUM itself in Swift:

Sources/App/Models/Account.swift

enum BillingStatus: String, Codable {
    case current
    case pastDue
}

Then I could define a model that uses the ENUM:

Sources/App/Models/Account.swift

final class Account: Model {
    static let schema = "accounts"

    @ID(key: .id)
    var id: UUID?

    @Enum(key: "billing_status")
    var billingStatus: BillingStatus

    init() {}

    init(
        id: IDValue? = nil,
        billingStatus: BillingStatus = .current
    ) {
        self.id = id
        self.billingStatus = status
    }
}

Lastly, I needed to define the migration code. This is a little more tricky that you might at first suppose for two reasons: (1) the ENUM type must be created before it can be used as a column type on the table, and (2) Vapor 4’s migrations are asynchronous, so extra steps are needed to ensure a consistent order of operations. To handle this situation, I used flatMap to provide a callback block that will execute as soon as the ENUM definition block has finished. The inverse must be also done in the revert method.

Sources/App/Migrations/CreateAccount.swift

import Fluent

struct CreateAccount: Migration {
    func prepare(on database: Database) -> EventLoopFuture<Void> {
        return database.enum("billing_status")
            .case("current")
            .case("past_due")
            .create()
            .flatMap { billing_status in
                return database.schema("accounts")
                    .id()
                    .field(
                        "billing_status",
                        billing_status,
                        .required,
                        .custom("DEFAULT 'current'")
                    )
                    .create()
            }
    }

    func revert(on database: Database) -> EventLoopFuture<Void> {
        return database.schema("accounts").delete().flatMap {
            return database.enum("billing_status").delete()
        }
    }
}

I used Vapor’s .custom method to specify that current is the default, mirroring the default used in the model code’s init method. You may well feel that specifying a default in both the database and the application layers is redundant but I prefer to keep the two in sync where feasible.

All code tested with Swift 5.2, Vapor 4.5, and Fluent 4.0.0-rc2.2 on both macOS 10.15 and Ubuntu Linux 18.04.

Tom MacWright writing on his blog:

The emerging norm for web development is to build a React single-page application, with server rendering. The two key elements of this architecture are something like:

1. The main UI is built & updated in JavaScript using React or something similar.
2. The backend is an API that that application makes requests against.

This idea has really swept the internet. It started with a few major popular websites and has crept into corners like marketing sites and blogs.

I’m increasingly skeptical of it.

I’ve always been skeptical of it but it is very interesting to read this from the perspective of someone who had embraced React more fully than I had. Mr. MacWright has worked on the front-end for both Mapbox Studio and Observable, both of which are largely implemented in React.

Mr. MacWright identifies four areas where single-page applications (SPAs) have settled on some “messy optimizations” for frequently-encountered problems with SPAs:

1. Bundle splitting
2. Server-side rendering
3. APIs
4. Data fetching

But I’m at the point where I look at where the field is and what the alternative patterns are — taking a second look at unloved, unpopular, uncool things like Django, Rails, Laravel — and think what the heck is happening. We’re layering optimizations upon optimizations in order to get the SPA-like pattern to fit every use case, and I’m not sure that it is, well, worth it.

[...]

But the cultural tides are strong. Building a company on Django in 2020 seems like the equivalent of driving a PT Cruiser and blasting Faith Hill’s “Breathe” on a CD while your friends are listening to The Weeknd in their Teslas. Swimming against this current isn’t easy, and not in a trendy contrarian way.

I’m old enough to remember when frameworks like Django and Ruby on Rails felt cutting-edge but they have both now become boring, stable projects. Is it such a bad thing, though, to build on such a tried-and-true foundation?