Prompt the canonical site URL during bootstrap script to rewrite website crawlers.

This commit is contained in:
2026-08-02 02:57:43 +02:00
parent a5ee02bb0b
commit 422560ec10
5 changed files with 112 additions and 18 deletions
+7 -1
View File
@@ -46,6 +46,11 @@ You'll be asked for:
| --- | --- | --- |
| **Site display name** (PascalCase) | `Loud` | Server name (`LoudWebsite`), `Loud.xcodeproj`, README title. |
| **Project slug** (lowercase) | `loud-ams` | Container owner, database name/user, Compose project name. |
| **Canonical site URL** (scheme + host) | `https://loud.amsterdam` | The `Sitemap:` reference in `robots.txt` and the `<loc>` entry in `sitemap.xml`. |
The canonical URL defaults to the reserved placeholder `https://site.example.com`;
leaving it there is allowed (the script warns), so you can bootstrap before the
domain is decided and set it later in the two crawler files.
The script rewrites the placeholder occurrences in place, renames
`Site.xcodeproj`, optionally starts a fresh git history, and then removes
@@ -57,7 +62,8 @@ Bootstrap leaves reminders, but in short:
1. **Copy & localization**`Services/Website/Sources/Library/Catalogs/Localizable.xcstrings`
and the landing / 404 pages under `Services/Website/Sources/Library`.
2. **Static assets**`Services/Website/Resources/Static` (CSS, JS, favicon, icons, manifest).
2. **Static assets**`Services/Website/Resources/Static` (CSS, JS, favicon, icons).
The manifest's `name` / `short_name` are empty and are not rewritten by bootstrap.
3. **Secrets** — set a real `DATABASE_PASSWORD` in a git-ignored
`Services/Website/.env` (the committed `.env.local` is an example only).
4. **Domain models** — the `Persistence` package ships a sample
+45
View File
@@ -24,6 +24,11 @@ fi
XCODEPROJ="./Site.xcodeproj"
# The placeholder origin the crawler files ship with. It is an RFC 2606 reserved
# domain, so an un-bootstrapped copy can never point a crawler at a real site.
PLACEHOLDER_URL="https://site.example.com"
PLACEHOLDER_URL_PATTERN='https://site\.example\.com'
# --- Prompts ------------------------------------------------------------------
printf 'Site display name (PascalCase, e.g. Berlin) [Site]: '
@@ -35,9 +40,31 @@ printf 'Project slug (lowercase, used for owner/db/compose, e.g. loud-berlin) [%
read -r SLUG
SLUG="${SLUG:-$DEFAULT_SLUG}"
printf 'Canonical site URL (scheme + host, e.g. https://berlin.example.com) [%s]: ' "$PLACEHOLDER_URL"
read -r URL
URL="${URL:-$PLACEHOLDER_URL}"
URL="${URL%/}" # the rewrites append their own path, so drop a trailing slash
# The URL is interpolated into a sed replacement (with `|` as the delimiter) and
# into the crawler files, so reject anything that is not a plain scheme + host.
case "$URL" in
http://* | https://*) ;;
*)
echo "error: the canonical site URL must start with http:// or https:// (got \"$URL\")." >&2
exit 1
;;
esac
case "$URL" in
*[!-A-Za-z0-9:/._~]*)
echo "error: the canonical site URL contains unexpected characters (got \"$URL\")." >&2
exit 1
;;
esac
echo
echo " display name : $NAME (server name \"${NAME}Website\", ${NAME}.xcodeproj)"
echo " slug : $SLUG (container owner, database name/user, compose project)"
echo " canonical URL: $URL (robots.txt sitemap reference, sitemap.xml entry)"
echo
printf 'Apply these values? [y/N]: '
read -r CONFIRM
@@ -95,6 +122,14 @@ rewrite "$W/docker-compose.override.yml" \
rewrite "$W/Makefile" \
-e "s/),site)/),${SLUG})/g"
# Crawler files: the absolute origin they must carry. `|` is the sed delimiter,
# since both the placeholder and the replacement contain slashes.
rewrite "$W/Resources/Static/robots.txt" \
-e "s|${PLACEHOLDER_URL_PATTERN}|${URL}|g"
rewrite "$W/Resources/Static/sitemap.xml" \
-e "s|${PLACEHOLDER_URL_PATTERN}|${URL}|g"
# Service README.
rewrite "$W/README.md" \
-e "s/^# Site Website\$/# ${NAME} Website/" \
@@ -147,6 +182,7 @@ Next steps:
- $W/Sources/Library/Catalogs/Localizable.xcstrings (copy)
- $W/Sources/Library (landing / 404 pages)
- $W/Resources/Static (css, js, favicon, icons)
- $W/Resources/Static/site.webmanifest (name / short_name)
2. Set a real database password in a git-ignored $W/.env
(the committed .env.local defaults the password to the slug — do NOT ship that).
3. Point the git remote at your new repository:
@@ -157,3 +193,12 @@ Next steps:
The Persistence package still ships an ExampleRecord / ExampleRepository sample
model — replace it with your real domain models when you add persistence.
EOF
if [ "$URL" = "$PLACEHOLDER_URL" ]; then
cat <<EOF
WARNING: the canonical site URL was left as the placeholder "${PLACEHOLDER_URL}".
Set the real origin in $W/Resources/Static/robots.txt and sitemap.xml before
going live, or crawlers will be pointed at a reserved example domain.
EOF
fi
+58 -15
View File
@@ -5,8 +5,10 @@ The **Site** public website service — a [Hummingbird](https://github.com/hummi
The service:
- Serves the landing page at `GET /` (rendered once per supported language with [Elementary](https://github.com/elementary-swift/elementary) and cached).
- Negotiates each request's language from its `Accept-Language` header against the languages in the `WebsiteLibrary` String Catalog, falling back to the default (`en`); pages are served from the per-language cache with `Content-Language` and `Vary: Accept-Language` headers.
- Builds every page on the shared `Page` scaffolding from `Infrastructure`, which assembles the document head around the page's own markup: the viewport declaration, the optional `description` summary and `rel="canonical"` link, the Open Graph / Twitter link-preview tags, and the JSON-LD structured-data script (see [Page metadata](#page-metadata)).
- Answers a liveness check at `GET /health` with a static JSON payload, and a readiness check at `GET /health/ready` that reports whether the database is reachable (`200` ready / `503` unavailable).
- Serves static files (CSS, JS, icons, manifest, `robots.txt`) from `Resources/Static` via Hummingbird's `FileMiddleware`, tagged with media-type-specific `Cache-Control`; the production image ships minified copies (see [Static assets](#static-assets)).
- Answers `HEAD` on every `GET` route: the router is built with `.autoGenerateHeadEndpoints`, so uptime monitors and crawlers probing with `HEAD` get the route's status and headers instead of a `404`.
- Serves static files (CSS, JS, icons, manifest, `robots.txt`, `sitemap.xml`) from `Resources/Static` via Hummingbird's `FileMiddleware`, tagged with media-type-specific `Cache-Control`; the production image ships minified copies (see [Static assets](#static-assets)).
- Returns a custom not-found (404) HTML page, localized like the landing page, for any request that matches neither a route nor a static file.
- Compresses responses (gzip/deflate) above a configurable size when the client advertises support.
- Stamps a hardened set of security headers on every response.
@@ -22,11 +24,11 @@ Two SwiftPM targets:
| Target | Kind | Path | Role |
| --- | --- | --- | --- |
| `Website` | executable | `Sources/App` | Entry point: reads configuration, builds the persistence service, and either serves the website or runs the migrate-and-exit mode. |
| `WebsiteLibrary` | library | `Sources/Library` | Controllers, the pages, the `StaticFile` asset catalog, the request context, and the `*+Defaults` extensions and configuration-key constants that supply the site's specifics to `Infrastructure`. |
| `WebsiteLibrary` | library | `Sources/Library` | Controllers, the pages (`IndexPage`, `NotFoundPage`) and their shared `Page` defaults, the `StaticFile` asset catalog, the request context, the String Catalog, and the `*+Defaults` extensions and configuration-key constants that supply the site's specifics to `Infrastructure`. |
The `Website` executable depends on four local packages:
- `Localization` (`Packages/Localization`) — the `Localize` and `Negotiate` helpers and the `LanguageList` of catalog languages (used by `WebsiteLibrary`).
- `Infrastructure` (`Packages/Infrastructure`) — the shared Hummingbird toolkit: the `RouterController` protocol and `addController` result-builder extension for declarative routing, the security/vary/rate-limit/localization/not-found middlewares, the `Page` and `Asset` scaffolding, the pre-rendered localized HTML responses, and the `FingerprintAssets` version-token derivation. The service supplies its specifics (String Catalog bundle, pages, icon metadata) through the `*+Defaults` extensions in `WebsiteLibrary`.
- `Infrastructure` (`Packages/Infrastructure`) — the shared Hummingbird toolkit: the `RouterController` protocol and `addController` result-builder extension for declarative routing, the security/vary/rate-limit/localization/not-found middlewares, the `Page` and `Asset` scaffolding, the `SocialCard` and `StructuredData` head-metadata types, the pre-rendered localized HTML responses, and the `FingerprintAssets` version-token derivation. The service supplies its specifics (String Catalog bundle, pages, icon metadata) through the `*+Defaults` extensions in `WebsiteLibrary`.
- `Persistence` (`Packages/Persistence`) — the Fluent-based data layer: the `Driver` selector, the `Service` factory that builds the `Fluent` service, the `PrepareDB` registrar that declares the migrations, and the `Probe` consulted by the readiness check; the models, migrations, and repositories stay internal to the package. It has no dependency on `swift-configuration`; the executable maps the `database.*` keys onto the driver.
- `Utility` (`Packages/Utility`) — small shared helpers with no server dependencies, currently the `NormalizeEmail` method.
@@ -45,6 +47,21 @@ RootController (GET / → landing page)
HealthController (GET /health → liveness, GET /health/ready → readiness)
```
The router is created with `.autoGenerateHeadEndpoints`, so each of those `GET` routes gets a `HEAD` sibling for free.
### Page metadata
Each page conforms to `Infrastructure`'s `Page` protocol and supplies only its `title`, `content`, `stylesheets`, and `scripts`; the protocol assembles the document around them and renders the head in a fixed order: the viewport declaration, the `summary`, the `canonicalURL` link, the `socialCard` tags, the `structuredData` script, then the page `metadata` and the stylesheet links. The body is the content followed by the script tags.
Four of those are optional and **omitted by default** — the reference site leaves them unset, so a generated site fills in what it needs by overriding them on `IndexPage` (or on the shared `Page+Defaults` extension, for site-wide values):
| Property | Renders as | Notes |
| --- | --- | --- |
| `summary` | `<meta name="description">` | The page's one-line description. |
| `canonicalURL` | `<link rel="canonical">` | Absolute URL. |
| `socialCard` | Open Graph + Twitter `<meta>` tags | A `SocialCard` — title, summary, URL, site name, locale, share image. Scrapers require absolute URLs, so the page composes them from its own origin. |
| `structuredData` | `<script type="application/ld+json">` | A `StructuredData` graph of schema.org nodes; `StructuredData(name:url:logo:profiles:)` builds the site-wide `Organization` + `WebSite` pair. The payload is an inert data block, so the `Content-Security-Policy` does not apply to it. |
What the pages *do* ship is in `Page+Defaults` (`Sources/Library/Internal/Extensions`), shared by every page: the document `lang`, the favicon / SVG icon / apple-touch-icon / web-manifest links, and the two `theme-color` metas.
## Configuration
Configuration is read through [swift-configuration](https://github.com/apple/swift-configuration) from
the following sources, **highest precedence first**:
@@ -68,9 +85,11 @@ A dotted config key maps to an environment variable by upper-casing, splitting c
| Config key | Environment variable | Default | Description |
| --- | --- | --- | --- |
| `cache.maxAge.asset` | `CACHE_MAX_AGE_ASSET` | `31536000` (1 year) | `max-age` for fingerprinted assets (CSS, JS) and fonts; also marked `immutable`. The pages reference CSS/JS through content-versioned URLs (`?v=<token>`), so a deploy busts them by changing the URL. |
| `cache.maxAge.text` | `CACHE_MAX_AGE_TEXT` | `3600` (1 hour) | `max-age` for unversioned text assets (e.g. `robots.txt`); also marked `must-revalidate`. |
| `cache.maxAge.text` | `CACHE_MAX_AGE_TEXT` | `3600` (1 hour) | `max-age` for the remaining `text/*` assets (e.g. `robots.txt`), which keep unversioned URLs; also marked `must-revalidate`. |
| `cache.maxAge.image` | `CACHE_MAX_AGE_IMAGE` | `604800` (1 week) | `max-age` for images (ICO, PNG, SVG). |
| `cache.maxAge.default` | `CACHE_MAX_AGE_DEFAULT` | `86400` (1 day) | `max-age` for everything else (e.g. the web manifest). |
| `cache.maxAge.default` | `CACHE_MAX_AGE_DEFAULT` | `86400` (1 day) | `max-age` for everything else — including `site.webmanifest` (`application/manifest+json`) and `sitemap.xml` (`application/xml`), neither of which is `text/*`. |
The groups are matched in order, so the specific media types (`text/css`, `text/javascript`, `font/*`) are resolved before the general `text/*` category.
### Response compression
| Config key | Environment variable | Default | Description |
@@ -141,6 +160,8 @@ A direct run picks up the local development overrides from `.env.local` (in-memo
Or via the Makefile / Docker (uses `docker-compose.override.yml`, which builds from source and sets `LOG_LEVEL=debug`):
```sh
make pkg-build # swift build
make pkg-outdated # list the SPM dependencies that can be updated
make pkg-update # update the SPM dependencies
make site-run # run locally with hot reload (hb watch)
make site-mount # docker compose up --build --detach
make site-unmount # docker compose down + remove the local image
@@ -168,8 +189,8 @@ A local MariaDB for development lives behind the `database` Compose profile (so
make db-mount # start MariaDB (docker compose --profile database up --wait mariadb)
make db-migrate # run migrations against it
make db-shell # open a SQL shell on it
make db-unmount # stop and remove the container (data kept in Tests/DB)
make db-reset # stop and remove the container
make db-unmount # stop and remove the container (keeps the data volume)
make db-reset # stop and remove the container, and delete its data volume
DATABASE_DRIVER=mysql make site-mount # run the site against MariaDB
```
> **Note:** because the data lives in the bind-mounted `Tests/DB` folder rather than a named volume, `db-reset`'s `--volumes` flag does **not** clear it. To start from an empty database, delete `Tests/DB` by hand.
@@ -185,7 +206,9 @@ make pkg-test
# = swift test --disable-xctest --enable-code-coverage --enable-swift-testing --parallel
```
Tests use the [Swift Testing](https://developer.apple.com/documentation/testing/) framework. The `Tests/Website.xctestplan` covers the service's two targets `WebsiteTests` (the executable/integration tests) and `WebsiteLibraryTests` (the library unit tests) — plus the local packages' suites: `InfrastructureTests`, `PersistenceTests`, `LocalizationTests`, and `UtilityTests`.
Tests use the [Swift Testing](https://developer.apple.com/documentation/testing/) framework. `make pkg-test` runs the service package's own two targets: `WebsiteTests` (the executable/integration tests) and `WebsiteLibraryTests` (the library unit tests).
`Tests/Website.xctestplan` — the plan the `Site.xcodeproj` scheme runs — adds the vendored packages' suites on top of those two: `InfrastructureTests`, `PersistenceTests`, `LocalizationTests`, and `UtilityTests`. From the command line, each of those is run from its own package directory (`swift test` in `Packages/<Name>`).
The `Persistence` package has its own suite (run it from `Packages/Persistence`). Its tests run against the in-memory backend by default; the MySQL integration test is skipped unless a database is pointed at via `MYSQL_TEST_HOST` (with optional `MYSQL_TEST_PORT`/`NAME`/`USERNAME`/`PASSWORD`), so `swift test` stays runnable with no database:
```sh
@@ -195,7 +218,14 @@ cd ../../Packages/Persistence && MYSQL_TEST_HOST=127.0.0.1 swift test
```
## Deployment
The production image is built for `linux/amd64` in release mode with a statically linked Swift runtime and jemalloc, runs as a non-root `hummingbird` user, and exposes port `8080` (`ENTRYPOINT ./Website --http-host 0.0.0.0 --http-port 8080`).
The production image is built in release mode with a statically linked Swift runtime and jemalloc, runs as a non-root `hummingbird` user, and exposes port `8080`. `make img-check` pins the build to `linux/amd64`; `make img-release` builds for `IMAGE_PLATFORM`.
The executable is the `ENTRYPOINT` and its serving flags are the `CMD`:
```dockerfile
ENTRYPOINT ["./Website"]
CMD ["--http-host", "0.0.0.0", "--http-port", "8080"]
```
The split is what makes the migrate-and-exit invocation below work: `docker compose run --rm website --database-migrate` replaces the `CMD` flags without having to override the entrypoint.
Verify the image builds for its `linux/amd64` target without tagging or publishing:
```sh
@@ -214,10 +244,14 @@ docker compose -f docker-compose.yml up -d
```
### Static assets
`Resources/Static` holds the site's stylesheets and scripts under `css/` and `js/`, paired by name: `shared.*` is loaded by every page, alongside a per-page `index.*` and `not-found.*`. The remaining files — the icons, `site.webmanifest`, `robots.txt`, and `sitemap.xml` — sit at the root. Every one of them is a case of the `StaticFile` enumeration, which is what the pages derive their URLs from.
The image build optimizes the files under `Resources/Static` in its `assets` stage, in place, with pinned optimizer versions so asset output is reproducible for a given Dockerfile commit:
- CSS and JS are minified with [esbuild](https://esbuild.github.io).
- PNG images are losslessly recompressed with [oxipng](https://github.com/oxipng/oxipng) — the output is pixel-identical, only encoded smaller.
- The SVG icon is minified with [svgo](https://github.com/svg/svgo).
- CSS and JS are minified with [esbuild](https://esbuild.github.io) (every file in `css/` and `js/`).
- PNG images are losslessly recompressed with [oxipng](https://github.com/oxipng/oxipng), recursively — the output is pixel-identical, only encoded smaller.
- SVGs are minified with [svgo](https://github.com/svg/svgo), recursively.
The PNG and SVG passes walk the tree (`--recursive`), so images added in a subdirectory are optimized without touching the Dockerfile.
Files keep their names and paths, so the URLs derived from the `StaticFile` enumeration are unaffected. The sources in the repository stay readable and unminified: a direct `swift run` serves them as-is, while any image build — including the local `make site-mount` one, which builds the same Dockerfile — serves the optimized copies.
@@ -228,6 +262,13 @@ Preview the optimized output locally — requires only Docker and writes to the
make ast-minify
```
#### Crawler files
`robots.txt` and `sitemap.xml` need an absolute origin, which the template ships as the placeholder `https://site.example.com` — an [RFC 2606](https://www.rfc-editor.org/rfc/rfc2606) reserved domain, so an un-bootstrapped copy can never point a crawler at a real site. The bootstrap script prompts for the canonical site URL and rewrites both files with it; it warns if the placeholder is left in place.
To change the origin later, edit the `Sitemap:` line in `robots.txt` and the `<loc>` entries in `sitemap.xml`. Add a `<loc>` per public page as the site grows — nothing generates the sitemap at runtime.
> **Still manual:** `site.webmanifest` ships empty `name` / `short_name` fields; bootstrap does not fill them in.
### Icons
All icons are renditions of the star mark in `icon.svg`, which is the canonical source — there is no external design file to regenerate from.
@@ -238,7 +279,9 @@ All icons are renditions of the star mark in `icon.svg`, which is the canonical
| `icon-192.png`, `icon-512.png` | 192/512 | `site.webmanifest` install icons and splash screens | None (no platform mechanism); full artwork, light rendering, on a transparent background. |
| `apple-touch-icon.png` | 180×180 | iOS home-screen bookmarks | None (fetched once, outside any page context); full artwork, deliberately opaque — iOS fills transparent regions with black. |
The landing page pairs the icons with two `theme-color` metas: `#0c0710` (media-qualified for dark, listed first — the first matching entry wins) and `#fafafa` as the fallback.
The icon and manifest links come from the shared `Page+Defaults` extension, so every page carries them, alongside two `theme-color` metas: `#fafafa` unqualified, then `#0c0710` qualified with `(prefers-color-scheme: dark)`.
> **Note:** a browser applies the *first* `theme-color` whose media query matches, so the unqualified light value currently wins on both themes. Put the dark, media-qualified meta first in `Page+Defaults` if the dark value should take effect.
The raster icons are committed binaries, regenerated from `icon.svg` on demand via a throwaway container (no local toolchain needed) — e.g. for the 512px rendition:
```sh
@@ -256,8 +299,8 @@ The Makefile and Compose files read these from a `.env` file (or the environment
| `HOST_CONTAINER` | Container registry host (e.g. `registry.example.com`). |
| `HOST_OWNER` | Registry namespace / owner. |
| `HOST_USER`, `HOST_PASSWORD` | Registry credentials for `make img-release`. |
| `IMAGE_NAME`, `IMAGE_TAG` | Image name and tag. |
| `IMAGE_PLATFORM` | Build platform (e.g. `linux/amd64`). |
| `IMAGE_NAME`, `IMAGE_TAG` | Image name and tag. `IMAGE_TAG` is the fallback for `make img-release`'s `version=` argument. |
| `IMAGE_PLATFORM` | Build platform for `make img-release` (e.g. `linux/amd64`). `make img-check` pins `linux/amd64` regardless. |
| `HOST_PORT` | Host port mapped to the container's `8080` (default `8080`). |
| `LOG_LEVEL` | Runtime log level (default `info`). |
| `HTTP_SERVER_NAME` | Runtime server name (default `SiteWebsite`). |
+1 -1
View File
@@ -4,4 +4,4 @@
User-agent: *
Disallow:
Sitemap: https://loud.amsterdam/sitemap.xml
Sitemap: https://site.example.com/sitemap.xml
@@ -1,6 +1,6 @@
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://loud.amsterdam/</loc>
<loc>https://site.example.com/</loc>
</url>
</urlset>