Friday, May 21, 2021

Async in Swift

I have glimpsed the future of async and await in Swift and it is gloriously mundane, just as it should be.

Eager as I am to see language-level support for asynchronous programming land in Swift, I was very excited to see the Vapor project launch branches of key components. Naturally, I wanted to try them out.

Although some asynchronous features landed in Swift 5.4, it seems like Swift 5.5, currently in an alpha stage, is likely to be a watershed release when it comes to built-in support for the co-routine style of concurrency enabled by the async and await keywords. Naturally, then, I wanted to test things out in Swift 5.5. In order to do so both in a Linux environment (I’m mostly interested in Swift for server-side programming) and without affecting my day-to-day environment, I decided to take advantage of Microsoft Visual Studio Code’s Dev Containers feature.

Swift 5.5 in Visual Studio Code’s Dev Containers

VS Code’s Dev Containers are a great way to engage in server-side (i.e. Linux-based) development from your normal desktop environment — be that Mac or Windows. But there is some setup involved, especially if you are going to use an alpha-stage version of Swift for which pre-built container images are not yet available. There’s a lot to say about Dev Containers and how they work but I’ll leave most of that to Microsoft’s resources. The TL;DR version is this: if you add a .devcontainer folder to your project with a few key configuration files, you can have VS Code automatically build a complete Linux-based, Docker-based development environment. If you are doing server-side development and need access to databases or other similar resources, you can use Docker Compose to include those dependencies. And if you commit that folder to your source repository, anyone on your team can easily and automatically get the same environment.

Here, then, is my server-side Swift Dev Container configuration:

.devcontainer/devcontainer.json

{
    "name": "Swift 5.5 Playground",
    "dockerComposeFile": "docker-compose.yml",
    "service": "app",
    "workspaceFolder": "/workspace",
    "settings": {
        "terminal.integrated.defaultProfile.linux": "/bin/bash",
        "lldb.adapterType": "bundled",
        "lldb.executable": "/usr/bin/lldb",
        "sde.languageServerMode": "sourcekit-lsp",
        "sourcekit-lsp.serverPath": "/usr/bin/sourcekit-lsp"
    },
    "extensions": [
        "vknabel.vscode-swift-development-environment",
        "vadimcn.vscode-lldb",
        "ms-azuretools.vscode-docker",
        "mtxr.sqltools",
        "mtxr.sqltools-driver-pg"
    ],
    "forwardPorts": [5432, 8080],
    "remoteUser": "vscode"
}

This is, more or less, a standard VS Code configuration file with a few additional keys specific to Dev Containers. You may notice the inclusion of Swift’s language server, SourceKit-LSP; we’ll talk more about that later. Before that, we should take a look at that docker-compose.yml file:

.devcontainer/docker-compose.yml

version: '3'
services:
  app:
    build:
      context: ..
      dockerfile: .devcontainer/Dockerfile
    volumes:
      - ..:/workspace:cached
    # Overrides default command so things don't shut down after the process ends.
    command: sleep infinity
    # Runs app on the same network as the database container, allows "forwardPorts" in devcontainer.json function.
    network_mode: service:db
    # Uncomment the next line to use a non-root user for all processes.
    user: vscode
    env_file: .env
  db:
    image: postgres:13.2
    restart: unless-stopped
    volumes:
      - postgres-data:/var/lib/postgresql/data
    environment:
      POSTGRES_USER: playground
      POSTGRES_DB: playground
      POSTGRES_PASSWORD: {make up a password}
volumes:
  postgres-data:

The db section ensures that we have a working instance of the estimable open source SQL database, PostgreSQL. (Note also in the devcontainer.json file that Postgres’ port — 5432 — is forwarded so we can access the database outside of the container and even outside of VS Code.) The main action, though, is in the app section; this incorporates the container in which we will do our development. A .env will load any environment variables we need (handy for database connection information), a workspace volume is defined so that our code will live both inside and outside the container, and the Dockerfile that defines the container is called out. Let’s take a look at that next:

.devcontainer/Dockerfile

FROM ubuntu:20.04

RUN export DEBIAN_FRONTEND=noninteractive DEBCONF_NONINTERACTIVE_SEEN=true && apt-get -q update && \
    apt-get -q install -y \
    binutils \
    build-essential \
    git \
    gnupg2 \
    libc6-dev \
    libcurl4 \
    libedit2 \
    libgcc-9-dev \
    libpython3.8 \
    libsqlite3-0 \
    libsqlite3-dev \
    libstdc++-9-dev \
    libxml2 \
    libz3-dev \
    pkg-config \
    tzdata \
    zlib1g-dev \
    && rm -r /var/lib/apt/lists/*

# Everything up to here should cache nicely between Swift versions, assuming dev dependencies change little

# gpg --keyid-format LONG -k FAF6989E1BC16FEA
# pub   rsa4096/FAF6989E1BC16FEA 2019-11-07 [SC] [expires: 2021-11-06]
#       8A7495662C3CD4AE18D95637FAF6989E1BC16FEA
# uid                 [ unknown] Swift Automatic Signing Key #3 <swift-infrastructure@swift.org>
ARG SWIFT_SIGNING_KEY=8A7495662C3CD4AE18D95637FAF6989E1BC16FEA
ARG SWIFT_PLATFORM=ubuntu
ARG OS_MAJOR_VER=20
ARG OS_MIN_VER=04
ARG SWIFT_WEBROOT=https://swift.org/builds/swift-5.5-branch

ENV SWIFT_SIGNING_KEY=$SWIFT_SIGNING_KEY \
    SWIFT_PLATFORM=$SWIFT_PLATFORM \
    OS_MAJOR_VER=$OS_MAJOR_VER \
    OS_MIN_VER=$OS_MIN_VER \
    OS_VER=$SWIFT_PLATFORM$OS_MAJOR_VER.$OS_MIN_VER \
    SWIFT_WEBROOT="$SWIFT_WEBROOT/$SWIFT_PLATFORM$OS_MAJOR_VER$OS_MIN_VER"

RUN echo "${SWIFT_WEBROOT}/latest-build.yml"

RUN set -e; \
    # - Grab curl here so we cache better up above
    export DEBIAN_FRONTEND=noninteractive \
    && apt-get -q update && apt-get -q install -y curl && rm -rf /var/lib/apt/lists/* \
    # - Latest Toolchain info
    && export $(curl -s ${SWIFT_WEBROOT}/latest-build.yml | grep 'download:' | sed 's/:[^:\/\/]/=/g')  \
    && export $(curl -s ${SWIFT_WEBROOT}/latest-build.yml | grep 'download_signature:' | sed 's/:[^:\/\/]/=/g')  \
    && export DOWNLOAD_DIR=$(echo $download | sed "s/-${OS_VER}.tar.gz//g") \
    && echo $DOWNLOAD_DIR > .swift_tag \
    # - Download the GPG keys, Swift toolchain, and toolchain signature, and verify.
    && export GNUPGHOME="$(mktemp -d)" \
    && curl -fsSL ${SWIFT_WEBROOT}/${DOWNLOAD_DIR}/${download} -o latest_toolchain.tar.gz \
    ${SWIFT_WEBROOT}/${DOWNLOAD_DIR}/${download_signature} -o latest_toolchain.tar.gz.sig \
    && curl -fSsL https://swift.org/keys/all-keys.asc | gpg --import -  \
    && gpg --batch --verify latest_toolchain.tar.gz.sig latest_toolchain.tar.gz \
    # - Unpack the toolchain, set libs permissions, and clean up.
    && tar -xzf latest_toolchain.tar.gz --directory / --strip-components=1 \
    && chmod -R o+r /usr/lib/swift \
    && rm -rf "$GNUPGHOME" latest_toolchain.tar.gz.sig latest_toolchain.tar.gz \
    && apt-get purge --auto-remove -y curl

# Print Installed Swift Version
RUN swift --version

RUN echo '[ ! -z "$TERM" -a -r /etc/motd ] && cat /etc/motd' \
    >> /etc/bash.bashrc; \
    echo " ################################################################\n" \
    "#\t\t\t\t\t\t\t\t#\n" \
    "# Swift Nightly Docker Image\t\t\t\t\t#\n" \
    "# Tag: $(cat .swift_tag)\t\t#\n" \
    "#\t\t\t\t\t\t\t\t#\n"  \
    "################################################################\n" > /etc/motd

# [Option] Install zsh
ARG INSTALL_ZSH="false"
# [Option] Upgrade OS packages to their latest versions
ARG UPGRADE_PACKAGES="true"

# Install needed packages and setup non-root user. Use a separate RUN statement to add your own dependencies.
ARG USERNAME=vscode
ARG USER_UID=1000
ARG USER_GID=$USER_UID
COPY .devcontainer/library-scripts/common-debian.sh /tmp/library-scripts/
RUN apt-get update && export DEBIAN_FRONTEND=noninteractive \
    && /bin/bash /tmp/library-scripts/common-debian.sh "${INSTALL_ZSH}" "${USERNAME}" "${USER_UID}" "${USER_GID}" "${UPGRADE_PACKAGES}" "true" "true" \
    && apt-get -y install --no-install-recommends lldb python3-minimal libpython3.7 \
    && apt-get autoremove -y && apt-get clean -y && rm -rf /var/lib/apt/lists/* && rm -rf /tmp/library-scripts

# Install SourceKit-LSP
RUN git clone -b "release/5.5" https://github.com/apple/sourcekit-lsp.git \
    && cd sourcekit-lsp \
    && swift build -Xcxx -I/usr/lib/swift -Xcxx -I/usr/lib/swift/Block \
    && cp .build/debug/sourcekit-lsp /usr/bin/

Because the Swift project isn’t yet publishing pre-built Docker containers for Swift 5.5, this Dockerfile is necessarily quite involved. Most of it is taken from Swift’s own Dockerfile for 5.5 with another section taken from Microsoft’s Dev Container Dockerfile for Swift and a final section to compile SouceKit-LSP. Note also the inclusion of library-scripts/common-debian.sh which you can find in Microsoft’s Dev Containers repository and must be included in your .devcontainer folder.

With the dev container definition complete, you can tell VS Code to build and enter your development environment by selecting the command “Remote-Containers: Open Folder in Container”. The first time you run this, it will involve downloading base containers and building your container on top, so it will take a while.

Finally, the Code

With the development environment finally ready, we can turn to actual code. My main goal was to take the regular Vapor template project and tweak it just enough to try out the async and await keywords. I ended up recreating the template manually, but there’s no reason you can’t start with the Vapor toolbox instead. With the toolbox installed, you run the command vapor new {project name}; be sure to answer yes to the question about using Fluent and to select PostgreSQL as the database engine.

To begin, we need to customize the Package.swift file to pull from the relevant async-await branches of the Vapor sources:

Package.swift

// swift-tools-version:5.5
import PackageDescription

let package = Package(
    name: "VaporAsyncTry",
    dependencies: [
        .package(url: "https://github.com/vapor/vapor.git", .branch("async-await")),
        .package(url: "https://github.com/vapor/fluent-kit.git", .branch("async-await")),
        .package(url: "https://github.com/vapor/fluent.git", .branch("main")),
        .package(url: "https://github.com/vapor/fluent-postgres-driver.git", .branch("main"))
    ],
    targets: [
        .target(
            name: "App",
            dependencies: [
                .product(name: "FluentKit", package: "fluent-kit"),
                .product(name: "Fluent", package: "fluent"),
                .product(name: "FluentPostgresDriver", package: "fluent-postgres-driver"),
                .product(name: "Vapor", package: "vapor"),
            ],
            swiftSettings: [
                .unsafeFlags([
                    "-Xfrontend", "-enable-experimental-concurrency",
                    "-Xfrontend", "-disable-availability-checking",
                ])
            ]
        ),
        .executableTarget(
            name: "Run",
            dependencies: [.target(name: "App")]
        ),
        .testTarget(name: "AppTests", dependencies: [
            .target(name: "App"),
            .product(name: "XCTVapor", package: "vapor"),
        ])
    ]
)

Notice also the swiftSettings section with the "-enable-experimental-concurrency" clause. This setting was introduced in Swift 5.4 in order to enable the earliest versions of async await support. I’m hopeful that this setting will not be necessary in the final release of Swift 5.5, but for now it does seem to be needed.

Now, at long last, we can update the example TodoController from Vapor’s default template to take advantage of async and await:

Sources/App/Controllers/TodoController.swift

import Fluent
import Vapor

struct TodoController: RouteCollection {
    func boot(routes: RoutesBuilder) throws {
        let todos = routes.grouped("todos")
        todos.get(use: index)
        todos.post(use: create)
        todos.group(":todoID") { todo in
            todo.put(use: update)
            todo.delete(use: delete)
        }
    }

    func index(request: Request) async throws -> [Todo] {
        return try await Todo.query(on: request.db).all()
    }

    func create(request: Request) async throws -> Todo {
        let todo = try request.content.decode(Todo.self)
        try await todo.save(on: request.db)
        return todo
    }

    func update(request: Request) async throws -> Todo {
        guard let todo = try await Todo.find(request.parameters.get("todoID"), on: request.db) else {
            throw Abort(.notFound)
        }
        let updatedTodo = try request.content.decode(Todo.self)
        todo.title = updatedTodo.title
        try await todo.save(on: request.db)
        return todo
    }

    func delete(request: Request) async throws -> HTTPStatus {
        guard let todo = try await Todo.find(request.parameters.get("todoID"), on: request.db) else {
            throw Abort(.notFound)
        }
        try await todo.delete(on: request.db)
        return HTTPStatus.ok
    }
}

The first thing to notice is that none of the return values are wrapped in EventLoopFuture. Next, observe that there are no nested callbacks, no need for map or flatMap. Function declarations include the async keyword and calls to the database are preceded by await but otherwise this looks just like synchronous code.

One thing I couldn’t get working was Vapor’s unwrap(or: Abort(.notFound)) shortcut for handling database queries that may or may not return a value. Instead, I had to use the somewhat more verbose guard let ... else { throw Abort(.notFound) } construction. Nevertheless, I consider this to be a huge improvement over the nested callbacks previously required.

Regarding sourcekit-lsp

Many — but by no means all — programmers like to have a little boost while programming in the form of intelligent auto-complete, pop-up documentation, etc. Many code editors like VS Code include robust support for these features for many programming languages. Often in the past, each editor had a different API for enabling these features, meaning that a plugin written for TextMate for example wouldn’t work with SublimeText. Microsoft, of all companies, is trying to fix that with their Language Server Protocol, or LSP, specification. LSP allows programming languages to offer universal plugins that work with any text editor supporting LSP.

The Swift programming language has embraced LSP in the form of SourceKit-LSP. In theory, SourceKit-LSP allows editors like VS Code to have all of the same auto-complete features for Swift that Apple’s own Xcode has. In practice, however, SourceKit-LSP has in the past proven to be so slow, buggy, and incomplete as to be nearly useless.

As you can see above, as part of this experiment, I wanted to try out SourceKit-LSP again to see if any progress has been made. I, for one, would very much like to have these features for Swift programming outside of Xcode (Xcode continues to use a different, proprietary engine to drive these features). Alas, I am sorry to say that little progress has been made. SourceKit-LSP remains so slow, buggy, and incomplete as to be nearly useless.


Sunday, February 21, 2021

On the State of Swift (Early 2021) or: Awaiting Async

Although my day job is (usually) programming in Python, I have been experimenting with Swift for a few years now. I am neither a Mac nor an iOS developer; these days I’m almost exclusively a backend developer. It should come as no surprise then that I’m most interested in Swift as a potential backend, server-side language. As such, I think I have a somewhat different perspective from many other developers using Swift. Even compared to others interested in server-side Swift, I think I’m outside the norm, as most members of that community that I’ve encountered have been iOS developers who would also like to program the backend for their apps in the same programming language as they use in the apps themselves.

My dalliances with Swift have always come in fits and starts. For me, the language has long been alluring but elusive: I really admire the concepts of the language and its potential but trying to use it outside of the Apple ecosystem has frequently proven very frustrating. That was especially true in the early days, because of the shortcuts that Apple had taken to get Swift out the door.

A Brief Digression on Apple’s Shortcuts

Swift was first released in 2014, as a proprietary programming language available only for Apple’s platforms. Later in 2015, Apple released it as open source, and released a Linux port at the same time. The Linux port, in particular, made it clear that Apple had taken some shortcuts with the initial implementation and release of Swift. Specifically, Swift’s standard library — the built-in functionality that comes with the language — was astonishingly small. It basically consists of just the built-in data types like Int, Double, Array, and Dictionary. Most modern, high-level languages have standard libraries that are far larger. Python’s standard library, for example, includes modules for dates, data parsing, and networking from low-level socket handling to higher-level HTTP serving. But if Swift’s standard library didn’t include any of those things, how were application developers for Apple’s platforms already able to write apps in Swift?

In turned out that on Apple’s platforms, Swift is intimately connected to the runtime environment of Apple’s previously preferred language, Objective-C. Apple’s messaging around this indicated this was to enable seamless interoperability between Swift and Objective-C code but it also meant that Swift code had ready access to Apple’s existing rich set of libraries written in Objective-C. Notably among these libraries is the Foundation framework — a fairly extensive collection of frequently-needed functionality that most of us developers take for granted.

On Linux, however, Swift is entirely separate from the Objective-C runtime and does not use Apple’s system libraries. To Apple’s credit, Swift on Linux was not left entirely in the lurch; there is a Swift-native version of Foundation. The oldest commit1 dates to 2012, before Swift had even been publicly announced. This version of Foundation, however, “is a substantial reimplementation of the same API” as the old Objective-C version of Foundation and as of this writing, it remains incomplete.2 Earlier versions of Swift’s Foundation implementation were pretty rough with missing functionality and even some incompatibilities with the Objective-C version. Today, however, I tend to think that it is, generally speaking, sufficiently complete and compatible as to be generally useable.

Foundation represents what most developers would consider essential functionality for a programming language. It, together with libdispatch and XCTest, constitute Swift’s Core Libraries; libraries considered so essential that they are distributed with Swift itself on non-Apple platforms. That Foundation, in particular, is now useable on non-Apple platforms means that Swift has passed what was previously a significant hurdle to its adoption on Linux.

Swift Today

The Swift project has ambitions well beyond Apple’s platforms. As stated on the Swift homepage, the “goal of the Swift project is to create the best available language for uses ranging from systems programming, to mobile and desktop apps, scaling up to cloud services.” Swift’s utility for systems, mobile, and desktop programming is evident from it’s use by Apple for Apple platforms. So where does Swift stand now for server-side use? Apple, after all, doesn’t really have a server platform3.

It took some time after the open source release of Swift, but in late 2016 the Swift project announced the formation of the Swift Server Work Group, sometimes abbreviated as SSWG. The SSWG’s remit is to create “APIs [that] provide low level ‘server’ functions as the basic building blocks for developing server-side capabilities”. The first fruit borne of the SSWG’s efforts was SwiftNIO, a non-blocking network I/O library. Since then, the group has sponsored a number of useful libraries.

It’s honestly a little hard for me to believe, but before SwiftNIO’s initial release in 2018, each of the early web frameworks in Swift — specifically, Kitura, Perfect, and Vapor — implemented their own low-level networking interfaces. No wonder, then, that SwiftNIO has been enthusiastically adopted, even by those frameworks that previously did that work themselves.

There’s just one problem. SwiftNIO, as denoted by the “N” in its name, is non-blocking. Swift, however, doesn’t have built-in support non-blocking (often referred to as asynchronous) operations, so this achieved by means of a coroutine engine embedded in SwiftNIO itself. Without syntax for asynchronous code in the language itself, however, using SwiftNIO becomes a tedious exercise in callback hell (a.k.a. the pyramid of doom).

Unfortunately, this isn’t just a problem for direct uses of SwiftNIO; anything built on top of SwiftNIO will necessarily inherit the need to incorporate the callback style of coding. The Swift-based web framework Vapor is a prime example of the contagious nature of SwiftNIO’s callback hell. Vapor 4 is built on top of SwiftNIO, including the database layers. As a result, any database operations in Vapor require the use of callbacks. Take, then, this example of a Vapor 4 web request handler to update a row in the Acronym table:

func updateHandler(_ req: Request) throws -> EventLoopFuture<Acronym> {
    let updatedAcronym = try req.content.decode(Acronym.self)
    return Acronym.find(req.parameters.get("acronymID"), on: req.db)
    .unwrap(or: Abort(.notFound)).flatMap { acronym in
        acronym.short = updatedAcronym.short
        acronym.long = updatedAcronym.long
        return acronym.save(on: req.db).map {
            acronym
        }
    }
}

Notice the return type is wrapped in EventLoopFuture. Notice that there are multiple return statements, starting on only the second line of code. Notice the use of flatMap and map to set the callback function (and notice that you must figure out which is appropriate in certain situations as they are subtly different). Notice the deep nesting. None of this code interacts with SwiftNIO directly but with other libraries that do.

The good news is that a solution is in the works. Other languages have adopted the keywords async and await to resolve the syntactic oddities of asynchronous code and the Swift community has resolved to follow suit4. Swift Evolution Proposal SE-0296, which includes these keywords, has been accepted and implemented. It remains unclear when this feature will be released properly; it is included in Swift version 5.4 currently in beta but is not enabled by default5. Even after Swift itself is updated, SwiftNIO will need to be updated to adopt it, followed in turn by the web frameworks like Vapor. Work on SwiftNIO’s update has started but of course it cannot be released until Swift itself is ready. When the updates to Swift, SwiftNIO, and Vapor are all released, the example above could look something like this:

async func updateHandler(_ req: Request) throws -> Acronym {
    let updatedAcronym = try req.content.decode(Acronym.self)
    let acronym = await Acronym.find(req.parameters.get("acronymID"), on: req.db)
    .unwrap(or: Abort(.notFound))
    acronym.short = updatedAcronym.short
    acronym.long = updatedAcronym.long
    await acronym.save(on: req.db)
    return acronym
}

No nested callbacks. Only one return statement at the end of the function. The return type for the function doesn’t have to be wrapped in EventLoopFuture. No trying to sort out the difference between flatMap and map. But for the use of the async and await keywords, it looks just like synchronous code.

I understand why SwiftNIO was written as an asynchronous library; that is where the entire industry is going. But shoehorning an asynchronous library into a synchronous programming language is awkward, to say the least. In my own humble opinion, there’s just not much point to trying to use SwiftNIO and the frameworks built on top of it right now. Relying on nested callbacks is error-prone, difficult to read, and difficult to debug. More to the point, asynchronous code that uses the async and await keywords is so much better that all asynchronous code in Swift will inevitably have to be rewritten as soon as they are available. I’d really, really like to start using Swift for server-side code right now but I just don’t feel like I can until async and await are available.


  1. The commit message on that commit ends with the line, Swift SVN r3405 which would seem to indicate that perhaps the GitHub repository was created by importing a previous Subversion repository. It is possible that older commits may have been lost or truncated in the process. 

  2. It is unclear at this point how much of the remaining unimplemented API will ever be completed. Foundation dates back to the OpenStep partnership between NeXT and Sun Microsystems of the early ’90s but was, in turn, built from earlier antecedents dating to the earliest days of the NeXTSTEP operation system in the late ’80s. There is undoubtedly cruft that isn’t really still needed in a modern context. Many components of Foundation are also superseded either by the Swift standard library or by other libraries for Swift like SwiftNIO

  3. Yes, I know. It doesn’t count. 

  4. The async and await keywords are, in fact, just one component of a larger, comprehensive suite of concurrency-related proposals for Swift. See also SE-0297: Concurrency Interoperability with Objective-C, SE-0298: Async/Await: Sequences, SE-0300: Continuations for interfacing async tasks with synchronous code, SE-0302: ConcurrentValue and @concurrent closures, as well as the forum post outlining the overall Swift Concurrency Roadmap

  5. Support for the async and await keywords can be activated with the command line flags -Xfrontend -enable-experimental-concurrency. I haven’t tested to see if it actually works. 


Thursday, October 29, 2020

Republicans Start to Resemble Autocratic Parties ↦

Researchers from the V-Dem Institute at the University of Gothenburg in Sweden have published a new study which seeks to quantify various aspects of political parties throughout the world. From the paper itself:

V-Party’s Illiberalism Index shows that the Republican party in the US has retreated from upholding democratic norms in recent years. Its rhetoric is closer to authoritarian parties, such as AKP in Turkey and Fidesz in Hungary. Conversely, the Democratic party has retained a commitment to longstanding democratic standards.

Julian Borger reports on the paper for The Guardian:

The study, published on Monday, shows the party has followed a similar trajectory to [Hungarian political party] Fidesz, which under Viktor Orbán has evolved from a liberal youth movement into an authoritarian party that has made Hungary the first non-democracy in the European Union.

India’s Bharatiya Janata Party (BJP) has been transformed in similar ways under Narendra Modi, as has the Justice and Development party (AKP) in Turkey under Recep Tayyip Erdoğan and the Law and Justice party in Poland. Trump and his administration have sought to cultivate close ties to the leadership of those countries.

[...]

“The data shows that the Republican party in 2018 was far more illiberal than almost all other governing parties in democracies,” the V-Dem study found. “Only very few governing parties in democracies in this millennium (15%) were considered more illiberal than the Republican party in the US.”

You may have seen claims that democracy itself is on the ballot this election. You may have thought that such claims were overblown. I submit to you that they are very, very true.


Wednesday, May 27, 2020

Defining a VARCHAR Column in Vapor 4

This Vapor tip is similar to the previous one, though a bit simpler. Of course, one of the most common data types to store in a database backend is text of some kind or other. The Vapor 4 documentation on models encourages you to use Swift’s String type in the model definition:

Sources/App/Models/Galaxy.swift

final class Galaxy: Model {
    // Name of the table or collection.
    static let schema = "galaxies"

    // Unique identifier for this Galaxy.
    @ID(key: .id)
    var id: UUID?

    // The Galaxy's name.
    @Field(key: "name")
    var name: String

    // Creates a new, empty Galaxy.
    init() { }

    // Creates a new Galaxy with all properties set.
    init(id: UUID? = nil, name: String) {
        self.id = id
        self.name = name
    }
}

And the migration documentation encourages you to use Vapor’s .string type in the database column definition:

Sources/App/Migrations/CreateGalaxy.swift

struct CreateGalaxy: Migration {
    // Prepares the database for storing Galaxy models.
    func prepare(on database: Database) -> EventLoopFuture<Void> {
        database.schema("galaxies")
            .id()
            .field("name", .string)
            .create()
    }

    // Optionally reverts the changes made in the prepare method.
    func revert(on database: Database) -> EventLoopFuture<Void> {
        database.schema("galaxies").delete()
    }
}

The .string type here, though, gets translated to an SQL column type of TEXT, which allows strings of unlimited length. I found that odd as many other ORMs, and indeed previous versions of Vapor, default to using a VARCHAR column type. I asked in the Vapor Discord about that a while ago and I was told that the change was made in order to maximize compatibility across database implementations; SQLite, in particular, doesn’t have a concept of a VARCHAR type1 (although it does alias that name to TEXT so you won’t get an error if you try to define a VARCHAR column). It was also pointed out that in many modern SQL database implementations, TEXT and VARCHAR columns are stored in the same way so there isn’t much of a performance penalty for using a TEXT column. (PostgreSQL’s documentation on text types calls this out specifically.)

All of this is indeed true, but there are nevertheless perfectly valid reasons to prefer VARCHAR columns over TEXT. Often, limiting the length of a string is highly desirable, especially in the context of a system that accepts user input. Knowing that certain strings will never be longer than n characters can be very helpful in laying out the UI and also helpfully limits the scope of edge case testing. It’s also relevant to indexing decisions as many index types are limited to the number of bytes that can be indexed per row.2

With all that background out of the way, how then do we define a VARCHAR column in Vapor 4? Vapor 4 doesn’t include a .varchar type in the same way that it has the .string type, so we’ll need to work around that. The model definition doesn’t need to change; Swift doesn’t particularly care if the string length is limited on the backend. The change, then, needs to be made in the migration, and it really is quite a simple change:

Sources/App/Migrations/CreateGalaxy.swift

struct CreateGalaxy: Migration {
    // Prepares the database for storing Galaxy models.
    func prepare(on database: Database) -> EventLoopFuture<Void> {
        database.schema("galaxies")
            .id()
            .field("name", .custom("VARCHAR(100)"))
            .create()
    }

    // Optionally reverts the changes made in the prepare method.
    func revert(on database: Database) -> EventLoopFuture<Void> {
        database.schema("galaxies").delete()
    }
}

As you can see, all we did was change .string to .custom("VARCHAR(100)"). The .custom type here allows us to pass in arbitrary SQL, so you will need to make certain that (1) the syntax is correct for your chosen database engine (you are using the same database engine in development as in production, right?) and (2) that you aren’t including user-defined input (that would be an exceptionally strange thing to do in a migration anyway).

All code tested with Swift 5.2, Vapor 4.5, Fluent 4.0.0-rc2.2, and PostgreSQL 12 on both macOS 10.15 and Ubuntu Linux 18.04.


  1. SQLite handles data types altogether differently than other database engines. In fact, SQLite doesn’t enforce column types at all and will happily store text in a column marked as INT. Instead, column types are used for certain kinds of implicit type conversions in a system that SQLite calls “type affinity”. These deviations from SQL norms are why you shouldn’t use SQLite in your development environment; you really should use the same database engine in development as you do in production. 

  2. It is possible to limit the length of a TEXT column using a CHECK constraint but this appears to have generally worse performance than using a VARCHAR column. Historically, increasing the character limit on a VARCHAR column would require a whole table rewrite, so this method was also a means to avoid that. For PostgreSQL specifically, however, such changes to VARCHAR columns haven’t required table rewrites since version 9.2


Tuesday, May 26, 2020

Creating a Database Enum in Vapor 4

Vapor has essentially become the web framework for the Swift programming language so I’ve been doing quite a bit of experimenting with it. Unfortunately, the documentation for the latest version is still rather incomplete as of this writing. As such, I thought I would add to the general body of knowledge about Vapor available on the web.

For a project I’ve started working on, I wanted to add an enumerated type (more commonly known simply as an ENUM) to my PostgreSQL database, use that ENUM as the type on a column on one of my tables, and have a default value. The first thing, then, was to define the ENUM itself in Swift:

Sources/App/Models/Account.swift

enum BillingStatus: String, Codable {
    case current
    case pastDue
}

Then I could define a model that uses the ENUM:

Sources/App/Models/Account.swift

final class Account: Model {
    static let schema = "accounts"

    @ID(key: .id)
    var id: UUID?

    @Enum(key: "billing_status")
    var billingStatus: BillingStatus

    init() {}

    init(
        id: IDValue? = nil,
        billingStatus: BillingStatus = .current
    ) {
        self.id = id
        self.billingStatus = status
    }
}

Lastly, I needed to define the migration code. This is a little more tricky that you might at first suppose for two reasons: (1) the ENUM type must be created before it can be used as a column type on the table, and (2) Vapor 4’s migrations are asynchronous, so extra steps are needed to ensure a consistent order of operations. To handle this situation, I used flatMap to provide a callback block that will execute as soon as the ENUM definition block has finished. The inverse must be also done in the revert method.

Sources/App/Migrations/CreateAccount.swift

import Fluent

struct CreateAccount: Migration {
    func prepare(on database: Database) -> EventLoopFuture<Void> {
        return database.enum("billing_status")
            .case("current")
            .case("past_due")
            .create()
            .flatMap { billing_status in
                return database.schema("accounts")
                    .id()
                    .field(
                        "billing_status",
                        billing_status,
                        .required,
                        .custom("DEFAULT 'current'")
                    )
                    .create()
            }
    }

    func revert(on database: Database) -> EventLoopFuture<Void> {
        return database.schema("accounts").delete().flatMap {
            return database.enum("billing_status").delete()
        }
    }
}

I used Vapor’s .custom method to specify that current is the default, mirroring the default used in the model code’s init method. You may well feel that specifying a default in both the database and the application layers is redundant but I prefer to keep the two in sync where feasible.

All code tested with Swift 5.2, Vapor 4.5, and Fluent 4.0.0-rc2.2 on both macOS 10.15 and Ubuntu Linux 18.04.


Sunday, May 17, 2020

“Second-Guessing the Modern Web” ↦

Tom MacWright writing on his blog:

The emerging norm for web development is to build a React single-page application, with server rendering. The two key elements of this architecture are something like:

  1. The main UI is built & updated in JavaScript using React or something similar.
  2. The backend is an API that that application makes requests against.

This idea has really swept the internet. It started with a few major popular websites and has crept into corners like marketing sites and blogs.

I’m increasingly skeptical of it.

I’ve always been skeptical of it but it is very interesting to read this from the perspective of someone who had embraced React more fully than I had. Mr. MacWright has worked on the front-end for both Mapbox Studio and Observable, both of which are largely implemented in React.

Mr. MacWright identifies four areas where single-page applications (SPAs) have settled on some “messy optimizations” for frequently-encountered problems with SPAs:

  1. Bundle splitting
  2. Server-side rendering
  3. APIs
  4. Data fetching

But I’m at the point where I look at where the field is and what the alternative patterns are — taking a second look at unloved, unpopular, uncool things like Django, Rails, Laravel — and think what the heck is happening. We’re layering optimizations upon optimizations in order to get the SPA-like pattern to fit every use case, and I’m not sure that it is, well, worth it.

[...]

But the cultural tides are strong. Building a company on Django in 2020 seems like the equivalent of driving a PT Cruiser and blasting Faith Hill’s “Breathe” on a CD while your friends are listening to The Weeknd in their Teslas. Swimming against this current isn’t easy, and not in a trendy contrarian way.

I’m old enough to remember when frameworks like Django and Ruby on Rails felt cutting-edge but they have both now become boring, stable projects. Is it such a bad thing, though, to build on such a tried-and-true foundation?


Friday, May 8, 2020

DOJ Moves to Drop Charges Against Michael Flynn ↦

Spencer S. Hsu, Devlin Barrett, and Matt Zapotosky reporting for The Washington Post:

The Justice Department moved Thursday to drop charges against President Trump’s former national security adviser Michael Flynn, a stunning reversal that prompted fresh accusations from law enforcement officials and Democrats that the criminal justice system was caving to political pressure from the administration.

[...]

Shortly before the Justice Department abandoned Flynn’s prosecution, the line prosecutor on the case, Brandon Van Grack, formally withdrew — just as the Stone prosecutors had.

In the new filing, Timothy Shea, the U.S. attorney for the District of Columbia, wrote that “continued prosecution of this case would not serve the interests of justice,” but current and former law enforcement officials said the decision was a betrayal of long-standing Justice Department principles. Shea, who was tapped by Barr to lead the U.S. attorney’s office, was the only lawyer to sign the filing; no career attorneys affixed their names to it.

It’s quite telling that the motion to dismiss was signed by a political appointee and not by any of the career attorneys who had been working the case previously. This is obviously a political decision to appease President Trump and the lies about a “lack of a legitimate investigative basis for the interview of Mr. Flynn” are a particularly flimsy excuse.

Under Attorney General William Barr, the Department of Justice has abandoned the pursuit of impartial justice and has instead devolved into an extension of Trump’s misguided will.


Thursday, May 7, 2020

Trump Doubles Down on Ending Obamacare ↦

Devlin Barrett reporting for The Washington Post:

“We want to terminate health care under Obamacare,” Trump told reporters Wednesday, the last day for his administration to change its position in a Supreme Court case challenging the law. “Obamacare, we run it really well. . . . But running it great, it’s still lousy health care.”

While the president has said he will preserve some of the Affordable Care Act’s most popular provisions, including guaranteed coverage for preexisting medical conditions, he has not offered a plan to do so, and his administration’s legal position seeks to end all parts of the law, including those provisions.

The comments are related to a case now scheduled to be argued in front of the Supreme Court which seeks to invalidate the entirety of the Affordable Care Act, commonly referred to as Obamacare. The Court is unlikely to issue a ruling on the case before the November election.

The latest ACA suit was organized by Republican attorneys general in Texas and other states. When the Trump administration declined to defend the law, a coalition of Democratic-led states entered.

The case began after the Republican-led Congress in 2017, unable to secure the votes to abolish the law, reduced to zero the penalty for a person not buying health insurance. Lawyers for the state of Texas argued that in doing so, Congress had removed the essential tax element that the Supreme Court had previously ruled made the program constitutional.

A district judge in Texas agreed and said the entire law must fall. Eventually the Trump administration agreed with that assessment.

In the middle of a pandemic, Trump wants to kill health care coverage for millions of Americans. The previous efforts to repeal Obamacare were so toxic, the Republicans in Congress couldn’t pass a bill to do it in spite of being the majority in both houses during Trump’s first two years as President. Now they are asking the courts to do what they couldn’t. If it wasn’t already accompanied by a deluge of inhumanity from this Administration, it would be breathtakingly cruel.


Thursday, May 7, 2020

Trump Demotes Government Scientist for Raising Alarm ↦

Ricardo Alonso-Zaldivar, Michael Balsamo, and Colleen Long reporting for the Associated Press:

Dr. Rick Bright, former director of the Biomedical Advanced Research and Development Authority, alleges he was reassigned to a lesser role because he resisted political pressure to allow widespread use of hydroxychloroquine, a malaria drug pushed by President Donald Trump. [...]

“I witnessed government leadership rushing blindly into a potentially dangerous situation by bringing in a non-FDA approved chloroquine from Pakistan and India, from facilities that had never been approved by the FDA,” Bright said Tuesday on a call with reporters. “Their eagerness to push blindly forward without sufficient data to put this drug into the hands of Americans was alarming to me and my fellow scientists.”

Dr. Bright further alleges that his attempts to spur action early on met with resistance from Trump Administration officials, including Health and Human Services Secretary Alex Azar and Assistant Secretary for Preparedness and Response Robert Kadlec. Specific efforts to acquire N95 respirator masks as early as January and February were ignored. He also alleges that there pressure to award contracts to the well-connected:

“Time after time I was pressured to ignore or dismiss expert scientific recommendations and instead to award lucrative contracts based on political connections,” Bright said in the call with reporters. “In other words, I was pressured to let politics and cronyism drive decisions over the opinions of the best scientists we have in government.”

So often, underneath the lies and incompetence constantly coming from the Trump Administration, we also find grift. He is without a doubt the most corrupt President of the modern era and very likely ever.


Thursday, May 7, 2020

Amazon VP Resigns over Whistleblower Firings ↦

Kate Cox reporting for Ars Technica:

Amazon VP Tim Bray, who had been with the company for more than five years, has resigned in protest of Amazon’s treatment of warehouse workers and the firing of other employees who spoke out.

The company fired multiple warehouse and office workers in recent weeks amid organizing efforts to improve conditions in the company’s distribution centers, where individuals have contracted COVID-19. Firing the whistleblowers is “evidence of a vein of toxicity running through the company culture,” Bray said in a blog post explaining his departure. “I choose neither to serve nor drink that poison.”

It’s quite rare for an executive to resign over worker treatment and even more rare to do so publicly. I feel that Tim Bray made the right move here.