rock-n-code/ccn
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Activity

Prompt the canonical site URL during bootstrap script to rewrite website crawlers.

Browse Source
This commit is contained in:
javier
2026-08-02 02:57:43 +02:00
parent a5ee02bb0b
commit 422560ec10
5 changed files with 112 additions and 18 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Services/Website/README.md
+58 -15
View File
@@ -5,8 +5,10 @@ The **Site** public website service — a [Hummingbird](https://github.com/hummi
The service:
- Serves the landing page at `GET /` (rendered once per supported language with [Elementary](https://github.com/elementary-swift/elementary) and cached).
- Negotiates each request's language from its `Accept-Language` header against the languages in the `WebsiteLibrary` String Catalog, falling back to the default (`en`); pages are served from the per-language cache with `Content-Language` and `Vary: Accept-Language` headers.
- Builds every page on the shared `Page` scaffolding from `Infrastructure`, which assembles the document head around the page's own markup: the viewport declaration, the optional `description` summary and `rel="canonical"` link, the Open Graph / Twitter link-preview tags, and the JSON-LD structured-data script (see [Page metadata](#page-metadata)).
- Answers a liveness check at `GET /health` with a static JSON payload, and a readiness check at `GET /health/ready` that reports whether the database is reachable (`200` ready / `503` unavailable).
- Serves static files (CSS, JS, icons, manifest, `robots.txt`) from `Resources/Static` via Hummingbird's `FileMiddleware`, tagged with media-type-specific `Cache-Control`; the production image ships minified copies (see [Static assets](#static-assets)).
- Answers `HEAD` on every `GET` route: the router is built with `.autoGenerateHeadEndpoints`, so uptime monitors and crawlers probing with `HEAD` get the route's status and headers instead of a `404`.
- Serves static files (CSS, JS, icons, manifest, `robots.txt`, `sitemap.xml`) from `Resources/Static` via Hummingbird's `FileMiddleware`, tagged with media-type-specific `Cache-Control`; the production image ships minified copies (see [Static assets](#static-assets)).
- Returns a custom not-found (404) HTML page, localized like the landing page, for any request that matches neither a route nor a static file.
- Compresses responses (gzip/deflate) above a configurable size when the client advertises support.
- Stamps a hardened set of security headers on every response.
@@ -22,11 +24,11 @@ Two SwiftPM targets:
| Target | Kind | Path | Role |
| --- | --- | --- | --- |
| `Website` | executable | `Sources/App` | Entry point: reads configuration, builds the persistence service, and either serves the website or runs the migrate-and-exit mode. |
| `WebsiteLibrary` | library | `Sources/Library` | Controllers, the pages, the `StaticFile` asset catalog, the request context, and the `*+Defaults` extensions and configuration-key constants that supply the site's specifics to `Infrastructure`. |
| `WebsiteLibrary` | library | `Sources/Library` | Controllers, the pages (`IndexPage`, `NotFoundPage`) and their shared `Page` defaults, the `StaticFile` asset catalog, the request context, the String Catalog, and the `*+Defaults` extensions and configuration-key constants that supply the site's specifics to `Infrastructure`. |
The `Website` executable depends on four local packages:
- `Localization` (`Packages/Localization`) — the `Localize` and `Negotiate` helpers and the `LanguageList` of catalog languages (used by `WebsiteLibrary`).
- `Infrastructure` (`Packages/Infrastructure`) — the shared Hummingbird toolkit: the `RouterController` protocol and `addController` result-builder extension for declarative routing, the security/vary/rate-limit/localization/not-found middlewares, the `Page` and `Asset` scaffolding, the pre-rendered localized HTML responses, and the `FingerprintAssets` version-token derivation. The service supplies its specifics (String Catalog bundle, pages, icon metadata) through the `*+Defaults` extensions in `WebsiteLibrary`.
- `Infrastructure` (`Packages/Infrastructure`) — the shared Hummingbird toolkit: the `RouterController` protocol and `addController` result-builder extension for declarative routing, the security/vary/rate-limit/localization/not-found middlewares, the `Page` and `Asset` scaffolding, the `SocialCard` and `StructuredData` head-metadata types, the pre-rendered localized HTML responses, and the `FingerprintAssets` version-token derivation. The service supplies its specifics (String Catalog bundle, pages, icon metadata) through the `*+Defaults` extensions in `WebsiteLibrary`.
- `Persistence` (`Packages/Persistence`) — the Fluent-based data layer: the `Driver` selector, the `Service` factory that builds the `Fluent` service, the `PrepareDB` registrar that declares the migrations, and the `Probe` consulted by the readiness check; the models, migrations, and repositories stay internal to the package. It has no dependency on `swift-configuration`; the executable maps the `database.*` keys onto the driver.
- `Utility` (`Packages/Utility`) — small shared helpers with no server dependencies, currently the `NormalizeEmail` method.
@@ -45,6 +47,21 @@ RootController (GET / → landing page)
HealthController (GET /health → liveness, GET /health/ready → readiness)
```
The router is created with `.autoGenerateHeadEndpoints`, so each of those `GET` routes gets a `HEAD` sibling for free.
### Page metadata
Each page conforms to `Infrastructure`'s `Page` protocol and supplies only its `title`, `content`, `stylesheets`, and `scripts`; the protocol assembles the document around them and renders the head in a fixed order: the viewport declaration, the `summary`, the `canonicalURL` link, the `socialCard` tags, the `structuredData` script, then the page `metadata` and the stylesheet links. The body is the content followed by the script tags.
Four of those are optional and **omitted by default** — the reference site leaves them unset, so a generated site fills in what it needs by overriding them on `IndexPage` (or on the shared `Page+Defaults` extension, for site-wide values):
| Property | Renders as | Notes |
| --- | --- | --- |
| `summary` | `<meta name="description">` | The page's one-line description. |
| `canonicalURL` | `<link rel="canonical">` | Absolute URL. |
| `socialCard` | Open Graph + Twitter `<meta>` tags | A `SocialCard` — title, summary, URL, site name, locale, share image. Scrapers require absolute URLs, so the page composes them from its own origin. |
| `structuredData` | `<script type="application/ld+json">` | A `StructuredData` graph of schema.org nodes; `StructuredData(name:url:logo:profiles:)` builds the site-wide `Organization` + `WebSite` pair. The payload is an inert data block, so the `Content-Security-Policy` does not apply to it. |
What the pages *do* ship is in `Page+Defaults` (`Sources/Library/Internal/Extensions`), shared by every page: the document `lang`, the favicon / SVG icon / apple-touch-icon / web-manifest links, and the two `theme-color` metas.
## Configuration
Configuration is read through [swift-configuration](https://github.com/apple/swift-configuration) from
the following sources, **highest precedence first**:
@@ -68,9 +85,11 @@ A dotted config key maps to an environment variable by upper-casing, splitting c
| Config key | Environment variable | Default | Description |
| --- | --- | --- | --- |
| `cache.maxAge.asset` | `CACHE_MAX_AGE_ASSET` | `31536000` (1 year) | `max-age` for fingerprinted assets (CSS, JS) and fonts; also marked `immutable`. The pages reference CSS/JS through content-versioned URLs (`?v=<token>`), so a deploy busts them by changing the URL. |
| `cache.maxAge.text` | `CACHE_MAX_AGE_TEXT` | `3600` (1 hour) | `max-age` for unversioned text assets (e.g. `robots.txt`); also marked `must-revalidate`. |
| `cache.maxAge.text` | `CACHE_MAX_AGE_TEXT` | `3600` (1 hour) | `max-age` for the remaining `text/*` assets (e.g. `robots.txt`), which keep unversioned URLs; also marked `must-revalidate`. |
| `cache.maxAge.image` | `CACHE_MAX_AGE_IMAGE` | `604800` (1 week) | `max-age` for images (ICO, PNG, SVG). |
| `cache.maxAge.default` | `CACHE_MAX_AGE_DEFAULT` | `86400` (1 day) | `max-age` for everything else (e.g. the web manifest). |
| `cache.maxAge.default` | `CACHE_MAX_AGE_DEFAULT` | `86400` (1 day) | `max-age` for everything else — including `site.webmanifest` (`application/manifest+json`) and `sitemap.xml` (`application/xml`), neither of which is `text/*`. |
The groups are matched in order, so the specific media types (`text/css`, `text/javascript`, `font/*`) are resolved before the general `text/*` category.
### Response compression
| Config key | Environment variable | Default | Description |
@@ -141,6 +160,8 @@ A direct run picks up the local development overrides from `.env.local` (in-memo
Or via the Makefile / Docker (uses `docker-compose.override.yml`, which builds from source and sets `LOG_LEVEL=debug`):
```sh
make pkg-build # swift build
make pkg-outdated # list the SPM dependencies that can be updated
make pkg-update # update the SPM dependencies
make site-run # run locally with hot reload (hb watch)
make site-mount # docker compose up --build --detach
make site-unmount # docker compose down + remove the local image
@@ -168,8 +189,8 @@ A local MariaDB for development lives behind the `database` Compose profile (so
make db-mount # start MariaDB (docker compose --profile database up --wait mariadb)
make db-migrate # run migrations against it
make db-shell # open a SQL shell on it
make db-unmount # stop and remove the container (data kept in Tests/DB)
make db-reset # stop and remove the container
make db-unmount # stop and remove the container (keeps the data volume)
make db-reset # stop and remove the container, and delete its data volume
DATABASE_DRIVER=mysql make site-mount # run the site against MariaDB
```
> **Note:** because the data lives in the bind-mounted `Tests/DB` folder rather than a named volume, `db-reset`'s `--volumes` flag does **not** clear it. To start from an empty database, delete `Tests/DB` by hand.
@@ -185,7 +206,9 @@ make pkg-test
# = swift test --disable-xctest --enable-code-coverage --enable-swift-testing --parallel
```
Tests use the [Swift Testing](https://developer.apple.com/documentation/testing/) framework. The `Tests/Website.xctestplan` covers the service's two targets — `WebsiteTests` (the executable/integration tests) and `WebsiteLibraryTests` (the library unit tests) — plus the local packages' suites: `InfrastructureTests`, `PersistenceTests`, `LocalizationTests`, and `UtilityTests`.
Tests use the [Swift Testing](https://developer.apple.com/documentation/testing/) framework. `make pkg-test` runs the service package's own two targets: `WebsiteTests` (the executable/integration tests) and `WebsiteLibraryTests` (the library unit tests).
`Tests/Website.xctestplan` — the plan the `Site.xcodeproj` scheme runs — adds the vendored packages' suites on top of those two: `InfrastructureTests`, `PersistenceTests`, `LocalizationTests`, and `UtilityTests`. From the command line, each of those is run from its own package directory (`swift test` in `Packages/<Name>`).
The `Persistence` package has its own suite (run it from `Packages/Persistence`). Its tests run against the in-memory backend by default; the MySQL integration test is skipped unless a database is pointed at via `MYSQL_TEST_HOST` (with optional `MYSQL_TEST_PORT`/`NAME`/`USERNAME`/`PASSWORD`), so `swift test` stays runnable with no database:
```sh
@@ -195,7 +218,14 @@ cd ../../Packages/Persistence && MYSQL_TEST_HOST=127.0.0.1 swift test
```
## Deployment
The production image is built for `linux/amd64` in release mode with a statically linked Swift runtime and jemalloc, runs as a non-root `hummingbird` user, and exposes port `8080` (`ENTRYPOINT ./Website --http-host 0.0.0.0 --http-port 8080`).
The production image is built in release mode with a statically linked Swift runtime and jemalloc, runs as a non-root `hummingbird` user, and exposes port `8080`. `make img-check` pins the build to `linux/amd64`; `make img-release` builds for `IMAGE_PLATFORM`.
The executable is the `ENTRYPOINT` and its serving flags are the `CMD`:
```dockerfile
ENTRYPOINT ["./Website"]
CMD ["--http-host", "0.0.0.0", "--http-port", "8080"]
```
The split is what makes the migrate-and-exit invocation below work: `docker compose run --rm website --database-migrate` replaces the `CMD` flags without having to override the entrypoint.
Verify the image builds for its `linux/amd64` target without tagging or publishing:
```sh
@@ -214,10 +244,14 @@ docker compose -f docker-compose.yml up -d
```
### Static assets
`Resources/Static` holds the site's stylesheets and scripts under `css/` and `js/`, paired by name: `shared.*` is loaded by every page, alongside a per-page `index.*` and `not-found.*`. The remaining files — the icons, `site.webmanifest`, `robots.txt`, and `sitemap.xml` — sit at the root. Every one of them is a case of the `StaticFile` enumeration, which is what the pages derive their URLs from.
The image build optimizes the files under `Resources/Static` in its `assets` stage, in place, with pinned optimizer versions so asset output is reproducible for a given Dockerfile commit:
- CSS and JS are minified with [esbuild](https://esbuild.github.io).
- PNG images are losslessly recompressed with [oxipng](https://github.com/oxipng/oxipng) — the output is pixel-identical, only encoded smaller.
- The SVG icon is minified with [svgo](https://github.com/svg/svgo).
- CSS and JS are minified with [esbuild](https://esbuild.github.io) (every file in `css/` and `js/`).
- PNG images are losslessly recompressed with [oxipng](https://github.com/oxipng/oxipng), recursively — the output is pixel-identical, only encoded smaller.
- SVGs are minified with [svgo](https://github.com/svg/svgo), recursively.
The PNG and SVG passes walk the tree (`--recursive`), so images added in a subdirectory are optimized without touching the Dockerfile.
Files keep their names and paths, so the URLs derived from the `StaticFile` enumeration are unaffected. The sources in the repository stay readable and unminified: a direct `swift run` serves them as-is, while any image build — including the local `make site-mount` one, which builds the same Dockerfile — serves the optimized copies.
@@ -228,6 +262,13 @@ Preview the optimized output locally — requires only Docker and writes to the
make ast-minify
```
#### Crawler files
`robots.txt` and `sitemap.xml` need an absolute origin, which the template ships as the placeholder `https://site.example.com` — an [RFC 2606](https://www.rfc-editor.org/rfc/rfc2606) reserved domain, so an un-bootstrapped copy can never point a crawler at a real site. The bootstrap script prompts for the canonical site URL and rewrites both files with it; it warns if the placeholder is left in place.
To change the origin later, edit the `Sitemap:` line in `robots.txt` and the `<loc>` entries in `sitemap.xml`. Add a `<loc>` per public page as the site grows — nothing generates the sitemap at runtime.
> **Still manual:** `site.webmanifest` ships empty `name` / `short_name` fields; bootstrap does not fill them in.
### Icons
All icons are renditions of the star mark in `icon.svg`, which is the canonical source — there is no external design file to regenerate from.
@@ -238,7 +279,9 @@ All icons are renditions of the star mark in `icon.svg`, which is the canonical
| `icon-192.png`, `icon-512.png` | 192/512 | `site.webmanifest` install icons and splash screens | None (no platform mechanism); full artwork, light rendering, on a transparent background. |
| `apple-touch-icon.png` | 180×180 | iOS home-screen bookmarks | None (fetched once, outside any page context); full artwork, deliberately opaque — iOS fills transparent regions with black. |
The landing page pairs the icons with two `theme-color` metas: `#0c0710` (media-qualified for dark, listed first — the first matching entry wins) and `#fafafa` as the fallback.
The icon and manifest links come from the shared `Page+Defaults` extension, so every page carries them, alongside two `theme-color` metas: `#fafafa` unqualified, then `#0c0710` qualified with `(prefers-color-scheme: dark)`.
> **Note:** a browser applies the *first* `theme-color` whose media query matches, so the unqualified light value currently wins on both themes. Put the dark, media-qualified meta first in `Page+Defaults` if the dark value should take effect.
The raster icons are committed binaries, regenerated from `icon.svg` on demand via a throwaway container (no local toolchain needed) — e.g. for the 512px rendition:
```sh
@@ -256,8 +299,8 @@ The Makefile and Compose files read these from a `.env` file (or the environment
| `HOST_CONTAINER` | Container registry host (e.g. `registry.example.com`). |
| `HOST_OWNER` | Registry namespace / owner. |
| `HOST_USER`, `HOST_PASSWORD` | Registry credentials for `make img-release`. |
| `IMAGE_NAME`, `IMAGE_TAG` | Image name and tag. |
| `IMAGE_PLATFORM` | Build platform (e.g. `linux/amd64`). |
| `IMAGE_NAME`, `IMAGE_TAG` | Image name and tag. `IMAGE_TAG` is the fallback for `make img-release`'s `version=` argument. |
| `IMAGE_PLATFORM` | Build platform for `make img-release` (e.g. `linux/amd64`). `make img-check` pins `linux/amd64` regardless. |
| `HOST_PORT` | Host port mapped to the container's `8080` (default `8080`). |
| `LOG_LEVEL` | Runtime log level (default `info`). |
| `HTTP_SERVER_NAME` | Runtime server name (default `SiteWebsite`). |