Operations

Migrations

The schema is managed with EF Core migrations, applied explicitly:

python scripts/migrations.py

The script asks which environment file to load (for the connection string: the owner’s HEIMDALL_MIGRATIONS_CONNECTIONSTRING when the file sets it, HEIMDALL_DATA_CONNECTIONSTRING otherwise — see Database roles ), then offers list, create (generate), and apply. It needs dotnet tool restore to have been run once, for the pinned EF Core CLI tool. Authoring a migration is covered in Contributing .

The API process never migrates, and refuses to start when migrations are pending. The seeder checks GetPendingMigrationsAsync first and throws with the missing migration names:

The database is missing 2 migration(s): 20260811_AddAuditLog, 20260812_AddTwoFactorEmailCode.
Apply them with scripts/migrations.py before starting the API.

Automatic migration on startup is convenient exactly until two instances start at once, or until a deployment silently applies a migration nobody reviewed. An explicit step makes schema change a decision rather than a side effect.

The shipped container image does migrate, unless told not to. docker/entrypoint.sh runs the migration bundle built into the image before it starts the API, so a container pointed at an empty database comes up working. That is the default: HEIMDALL_RUN_MIGRATIONS is true when unset, and true in all four docker/*.env.example templates; the entrypoint migrates only when it is unset, empty or exactly true, and any other value (false, False, 0, no, TRUE) skips the migration, after which the API refuses to start while migrations are pending. The bundle connects with HEIMDALL_MIGRATIONS_CONNECTIONSTRING (the schema owner) when that is set and with HEIMDALL_DATA_CONNECTIONSTRING otherwise, and the entrypoint removes the owner’s string before the API starts (see Database roles ). Set HEIMDALL_RUN_MIGRATIONS=false and run python scripts/migrations.py as its own deploy step when a deployment runs more than one replica (two containers applying migrations at start-up race each other), when a migration needs review or timing first (see Migrating big tables ), or when the owner’s password should not be on the container at all. The argument above is against the API process migrating itself; the image’s default is the same choice made at deployment, opt-out, and a deployment that wants the explicit step takes it with this one variable.

Migrating big tables

Most tables are small. audit_log is not: nearly every command inserts a row and nothing deletes one, so it only grows, and person is read by every login. A migration that locks either one stalls the API for as long as it runs. When a migration touches a table that may hold millions of rows:

• Build indexes with CREATE INDEX CONCURRENTLY. A plain CREATE INDEX takes a SHARE lock and blocks every insert and update for the whole build. CONCURRENTLY cannot run inside a transaction, so issue it as migrationBuilder.Sql(..., suppressTransaction: true), with IF NOT EXISTS so an interrupted run can be repeated, and DROP INDEX CONCURRENTLY IF EXISTS in Down. A concurrent build that fails leaves an index marked INVALID (\d heimdall.audit_log); drop it before applying again. 20261021183451_AddPersonLowerEmailIndex is the example to copy (it also drops the superseded index concurrently); 20261021161812_AddAuditRetentionIndexes builds two partial indexes on audit_log the same way. 20261014045512_AddAuditLogTargetIndex predates the rule and builds its index with a plain CREATE INDEX: apply nothing like it to a large trail.
• Add a column as nullable, or with a default. ADD COLUMN ... NOT NULL DEFAULT <constant> is a catalog change on PostgreSQL 11 and later, with no table rewrite. A NOT NULL column with no default, or a default that is volatile, forces a rewrite under an ACCESS EXCLUSIVE lock. To tighten an existing column, add it nullable, backfill in batches, then set NOT NULL.
• Backfill in batches, never in one UPDATE. A single UPDATE person SET ... over the whole table (as 20260909163322_AnonymiseDeletedIdentities does, once, on a small table) holds its row locks until it commits, writes every row’s new version at once and bloats the table. Update a bounded slice at a time (WHERE id IN (SELECT id ... LIMIT 1000), or by id range), committing between slices, and suppressTransaction: true is what allows that inside a migration.
• Mind the per-row triggers. person, google_user, scope and the other tables named in Shared public ids carry two row-level triggers each from 20261015035042_ProtectIdentityPublicIds and 20261015041235_ProtectEntityPublicIds: *_public_id_is_immutable (BEFORE UPDATE ... FOR EACH ROW) fires on every updated row, whichever column the update sets, and *_public_id_is_unused (BEFORE INSERT) looks the id up in the other tables. A bulk backfill therefore pays a function call per row, plus those lookups on insert, on top of the write itself; budget for it, keep the slices small, and expect a backfill to run several times slower than the same update on a table without them.
• Take the lock timeout you can afford. Statements that need a brief lock (ALTER TABLE, a trigger) queue behind long transactions and block everything queued after them; SET lock_timeout = '5s' makes them fail fast and retry rather than stall the API.
• Apply it as its own step. Run it with HEIMDALL_RUN_MIGRATIONS=false and scripts/migrations.py, off peak, rather than letting a container’s start-up carry a long build. Authoring rules are in Contributing .

Shared public ids — before ProtectIdentityPublicIds (#199)

ProtectIdentityPublicIds and ProtectEntityPublicIds make public_id immutable on person, google_user, scope, application, application_credential, scope_permission, role and pending_registration, refuse a new row whose public_id another of those tables holds or a hard delete tombstoned, and refuse a new person or Google User whose id an audit entry names. They guard future writes only and do not look at what is already stored. The API only ever issues fresh random ids, so a duplicate can exist only if somebody wrote one by hand in SQL. To check for a hand-made duplicate, run both queries before migrating; each should return no rows.

-- 1. Any public_id that rows of two or more guarded tables hold.
WITH ids (tbl, public_id) AS (
    SELECT 'person', public_id FROM heimdall.person
    UNION ALL SELECT 'google_user', public_id FROM heimdall.google_user
    UNION ALL SELECT 'scope', public_id FROM heimdall.scope
    UNION ALL SELECT 'application', public_id FROM heimdall.application
    UNION ALL SELECT 'application_credential', public_id FROM heimdall.application_credential
    UNION ALL SELECT 'scope_permission', public_id FROM heimdall.scope_permission
    UNION ALL SELECT 'role', public_id FROM heimdall.role
    UNION ALL SELECT 'pending_registration', public_id FROM heimdall.pending_registration
)
SELECT public_id, string_agg(tbl, ', ' ORDER BY tbl) AS held_by
FROM ids
GROUP BY public_id
HAVING count(*) > 1;

-- 2. Any person or Google User whose id audit entries named before the identity existed, so those
--    entries were about something else. The minute absorbs clock skew between the API and the database.
SELECT 'person' AS tbl, p.public_id, count(*) AS earlier_entries
FROM heimdall.person p
JOIN heimdall.audit_log a ON a.target_id = p.public_id
WHERE a.created_at < p.created_at - interval '1 minute'
GROUP BY p.public_id
UNION ALL
SELECT 'google_user', g.public_id, count(*)
FROM heimdall.google_user g
JOIN heimdall.audit_log a ON a.target_id = g.public_id
WHERE a.created_at < g.created_at - interval '1 minute'
GROUP BY g.public_id;

A row from either query is worth investigating before anything else: it is either a hand-made fix-up or somebody preparing to make audit entries clearable. No rows proves less than it seems. Query 1 sees only rows that are there now. Query 2 trusts created_at, which an insert can set, so a backdated plant passes it. Neither sees a plant that has already run its course — target cleared, identity tombstoned and deleted — or a scope or other entity sitting on an id that erased_subject records. Once migrated, hand-written SQL that rewrites a public_id — a support fix-up, or a restore that remaps ids — fails with restrict_violation.

Database roles — TH-18

The supported setup runs the migrations as an owner role and the API as a login that owns nothing (NFR-29). The owner holds the schema and everything in it, including the audit triggers and their functions; the API’s login gets the row privileges it needs and nothing more. Two roles per environment, which the operator creates with the scripts in scripts/sql:

Why it matters. The triggers that keep audit_log append-only bind every role but the table’s owner and a superuser: an owner may ALTER TABLE … DISABLE TRIGGER USER, drop a trigger or replace its function without being a superuser, and then rewrite or delete the trail. A deployment whose API login owns the tables — the single login the deployment guide used to create, and the setup an environment keeps until it is separated as below — therefore gets no protection from the triggers against the API’s own credentials. In the two-role setup the API’s login is refused all of that, and the property the triggers exist for, that an action can never be denied, holds against it too.

Nothing changes until an operator opts in. Without HEIMDALL_MIGRATIONS_CONNECTIONSTRING the entrypoint migrates with the API’s own connection string, as it always has.

Use one pair per environment, as the deployment guide does for logins: heimdall_owner and heimdall_svc for production, heimdall_development_owner and heimdall_development_svc, and so on. The commands below are for production; substitute the names for another environment.

What the runtime login may do

Connect, look objects up, and read and write rows: SELECT, INSERT, UPDATE and DELETE on every heimdall table except audit_log, which gets no DELETE; USAGE and SELECT on their sequences; SELECT on the migrations history in public, which the start-up seeder reads to refuse a schema that is behind. Nothing else: no ownership, no TRUNCATE, REFERENCES, TRIGGER or MAINTAIN anywhere, no CREATE on the database or a schema, no membership of the owner. No EXECUTE is granted: the API calls no function of its own, and the trigger functions run when their triggers fire. That covers everything the API does at runtime, attempt_notice_budget, google_id_token_use and the budget columns on person included — the functional suite runs as such a login.

audit_log keeps UPDATE because the retention pass writes the rows it clears whole; the triggers accept an update only when it clears the attribution or a person target once due (NFR-21), and the login cannot change the triggers.

Tables later migrations add get the same grants with no grant in the migration. The scripts set default privileges for the owner (ALTER DEFAULT PRIVILEGES FOR ROLE heimdall_owner IN SCHEMA heimdall GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES …, and the sequences and public’s equivalents), so every table and sequence the owner creates is granted as it is created — a migration cannot forget, because it does nothing. Default privileges are recorded against the role that creates the object, so they apply to the owner’s objects alone: never run a migration as a superuser or as the runtime login, or its tables are not granted, and the API’s first query of them fails with permission denied. When a migration adds a table the API must not write freely — another append-only table, say — its revoke is added to scripts/sql/runtime-grants.sql, beside audit_log’s, and the operator runs new-deployment.sql again after that migration.

The scripts

The names are psql variables, owner, runtime and database; the scripts hold no password, and the operator sets each one with \password afterwards. Both setup scripts can be run again: what exists is left alone and the grants are reset. Copy the whole scripts/sql directory to the host and run the scripts with -f from there, readable by postgres: the two setup scripts include runtime-grants.sql by a path relative to themselves.

Checking the setup

As the runtime login, from the host (the loopback rule in pg_hba.conf covers it):

psql "host=127.0.0.1 dbname=heimdall user=heimdall_svc" -X -f scripts/sql/check-runtime-login.sql

It ends with NOTICE: heimdall_svc was refused all … statements. A list instead names each statement the login was not refused, and the script exits with an error. Run as the owner or a superuser, it lists them all: that is the wrong session, not a broken setup, and nothing was changed.

And as a superuser, connected to the environment’s database: each query must return no rows, and every column of the last one must be false.

-- 1. Nothing in this database, and no database, still belongs to the runtime login.
SELECT classid::regclass AS catalog, objid FROM pg_shdepend
WHERE deptype = 'o' AND refobjid = 'heimdall_svc'::regrole
  AND dbid IN (0, (SELECT oid FROM pg_database WHERE datname = 'heimdall'));

-- 2. The database, the schema, every table, sequence and index in heimdall and public, and every
--    function in heimdall belong to the owner.
SELECT 'database' AS kind, datname AS name, pg_get_userbyid(datdba) AS owner
FROM pg_database WHERE datname = 'heimdall' AND datdba <> 'heimdall_owner'::regrole
UNION ALL
SELECT 'schema', nspname, pg_get_userbyid(nspowner)
FROM pg_namespace WHERE nspname = 'heimdall' AND nspowner <> 'heimdall_owner'::regrole
UNION ALL
SELECT 'relation', n.nspname || '.' || c.relname, pg_get_userbyid(c.relowner)
FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace
WHERE n.nspname IN ('heimdall', 'public') AND c.relowner <> 'heimdall_owner'::regrole
UNION ALL
SELECT 'function', n.nspname || '.' || p.proname, pg_get_userbyid(p.proowner)
FROM pg_proc p JOIN pg_namespace n ON n.oid = p.pronamespace
WHERE n.nspname = 'heimdall' AND p.proowner <> 'heimdall_owner'::regrole;

-- 3. The runtime login's privileges and flags.
SELECT has_table_privilege(r.oid, 'heimdall.audit_log', 'DELETE') AS audit_delete,
       has_table_privilege(r.oid, 'heimdall.person', 'TRUNCATE') AS truncate,
       has_schema_privilege(r.oid, 'heimdall', 'CREATE') AS create_in_heimdall,
       has_schema_privilege(r.oid, 'public', 'CREATE') AS create_in_public,
       has_database_privilege(r.oid, 'heimdall', 'CREATE') AS create_schema,
       pg_has_role(r.oid, 'heimdall_owner', 'MEMBER') AS member_of_owner,
       r.rolsuper, r.rolcreaterole, r.rolcreatedb, r.rolbypassrls
FROM pg_roles r WHERE r.rolname = 'heimdall_svc';

A new deployment

1. As a superuser, from the directory scripts/sql was copied to:

   sudo -u postgres psql -X -v owner=heimdall_owner -v runtime=heimdall_svc -v database=heimdall \
       -f new-deployment.sql
   

2. Set both passwords, which the env file will hold: sudo -u postgres psql -c '\password heimdall_owner', and the same for heimdall_svc. Give both a pg_hba.conf rule like the one in the deployment guide (heimdall_owner only needs it while the container migrates; a deployment that migrates out of band gives it a local rule instead).

3. In the env file, set DB_USER/DB_PASSWORD to the runtime login and HEIMDALL_MIGRATIONS_CONNECTIONSTRING to the owner’s whole connection string, with the same Search Path and DB_CONNECTION_EXTRA as the API’s: Host=host.docker.internal;Port=5432;Database=heimdall;Username=heimdall_owner;Password=…;Search Path=public. Deploy. The entrypoint migrates as the owner, and the API starts as the runtime login, which the default privileges have already granted the new tables.

4. Run step 1 again. The first migration has created audit_log by now, and the second run takes away the DELETE on it that the default privileges gave the login. Until then its trigger refuses every DELETE anyway.

5. Run the checks above.

Separating the migration owner from the runtime login

A deployment created from the earlier guide has one login, heimdall_svc, owning the database and everything in it, and the audit triggers do not hold against it. separate-owner.sql moves the ownership to a new owner and grants the login back what the API needs, in one transaction. The API keeps running throughout: it sees the old ownership or the new one with its grants, never a mix.

Before you start. The environment must run a release whose entrypoint reads HEIMDALL_MIGRATIONS_CONNECTIONSTRING (1.6.0 or later); on an older image the variable is ignored. Take scripts/sql from this release or later. Once step 2 commits, a container start that migrates as heimdall_svc fails, because the login no longer owns what the migrations change: a deploy, or Docker restarting a crashed container. The running API is unaffected. So do steps 2 to 5 in one sitting and hold the environment’s deploys meanwhile.

1. Back up, as a superuser. The custom format keeps ownership and grants, so a restore puts back exactly what was there:

   sudo -u postgres pg_dump --format=custom --file=/var/backups/heimdall-$(date -u +%Y%m%dT%H%M%SZ).dump heimdall
   sudo -u postgres pg_restore --list /var/backups/heimdall-<timestamp>.dump | head
   

2. Run the script as the database admin, from the directory scripts/sql was copied to:

   sudo -u postgres psql -X -v owner=heimdall_owner -v runtime=heimdall_svc -v database=heimdall \
       -f separate-owner.sql
   

   It takes a lock on every table it changes and holds them until COMMIT, so requests wait for that moment. If a long transaction holds a table for more than five seconds it gives up and rolls back whole, saying canceling statement due to lock timeout; run it again at a quieter moment. It refuses, changing nothing, when the login also owns another database or a tablespace, which REASSIGN OWNED would move as well: change their owner first. Running it again after it succeeded is harmless.

3. Set the owner’s password: sudo -u postgres psql -c '\password heimdall_owner'. Add its pg_hba.conf rule (host heimdall heimdall_owner 172.16.0.0/12 scram-sha-256 on the VPS) and sudo systemctl reload postgresql.

4. Store the owner’s connection string as HEIMDALL_MIGRATIONS_CONNECTIONSTRING, with the same Search Path and DB_CONNECTION_EXTRA as the API’s: Host=host.docker.internal;Port=5432;Database=heimdall;Username=heimdall_owner;Password=…;Search Path=public. Under yggdrasil, type it at the hidden prompt, or pipe it in, never on the command line:

   ygg vars set heimdall-api@production HEIMDALL_MIGRATIONS_CONNECTIONSTRING=-
   

   The name marks it a secret. A value cannot contain a single quote. Elsewhere, put it in the environment’s env file. Leave DB_USER and DB_PASSWORD as they are.

5. Redeploy the environment as usual. The start-up log shows entrypoint: applying EF Core migrations with HEIMDALL_MIGRATIONS_CONNECTIONSTRING....

6. Verify: the container reports healthy (/healthcheck); a sign-in works, and so does an administrative change, which writes an audit entry; check-runtime-login.sql passes as heimdall_svc, and the superuser queries above return what they should.

Rolling back. If the deploy in step 5 fails because the owner cannot connect — a wrong password, or no pg_hba.conf rule — fix the connection string or the rule and redeploy; the database is fine as it is. To return to the single login, reverse the ownership first, as a superuser connected to the environment’s database, with the same retry as step 2:

BEGIN;
SET LOCAL lock_timeout = '5s';
ALTER DATABASE heimdall OWNER TO heimdall_svc;
REASSIGN OWNED BY heimdall_owner TO heimdall_svc;
COMMIT;

then remove the variable (ygg vars unset heimdall-api@production HEIMDALL_MIGRATIONS_CONNECTIONSTRING) and redeploy. Removing the variable alone does not roll back: with the ownership left with the owner the API runs, but the entrypoint migrates as heimdall_svc and fails, so that combination needs HEIMDALL_RUN_MIGRATIONS=false and migrations applied out of band as the owner. The grants left behind by the reversal name the login on objects it owns again, which changes nothing. The backup from step 1 is the last resort: pg_restore --clean --if-exists --dbname=heimdall from it, as a superuser, with the API stopped.

Where the owner’s password lives. With HEIMDALL_MIGRATIONS_CONNECTIONSTRING set, the owner’s connection string is in the env file and the container’s configuration, beside the other secrets there. The entrypoint hands it to the migration bundle through the bundle’s own environment, never its command line, and removes it before the API starts, so the API process never holds it. A deployment that wants it off the container altogether sets HEIMDALL_RUN_MIGRATIONS=false and migrates out of band with scripts/migrations.py, from an environment file that sets HEIMDALL_MIGRATIONS_CONNECTIONSTRING — the script, like the entrypoint, migrates with it when it is set.

Start-up sequence

flowchart TD
    A[Configure Serilog] --> B[Load environment configuration]
    B --> C[Configure Web API + custom model-state response]
    C --> D0{Diagnostics on and<br/>a public host?}
    D0 -->|"Yes, and HEIMDALL_ALLOW_<br/>SENSITIVE_DIAGNOSTICS ≠ true"| D0X["Turn them off —<br/>one warning"]
    D0 -->|No, or allowed| D[Register dependencies]
    D0X --> D
    D --> D1{Production and<br/>Mailgun unconfigured?}
    D1 -->|Yes| DX["Refuse to start —<br/>tokens would be logged in plaintext"]
    D1 -->|No| E[Configure security + JWT + rate limiting]
    E --> E1{Token secret set?}
    E1 -->|No| EX["Refuse to start —<br/>every request would fail with IDX10703"]
    E1 -->|Yes| F[Build app + middlewares + Swagger]
    F --> G[Start services → DatabaseSeeder]
    G --> G1{Migrations pending?}
    G1 -->|Yes| GX[Refuse to start — names them]
    G1 -->|No| H[Ensure the three Role rows]
    H --> I{Any system admin?}
    I -->|No| J["Create one from<br/>HEIMDALL_MASTER_USER_*<br/>— refuses to start if unset"]
    I -->|Yes| K[Ready]
    J --> K

The seeder is idempotent, so it runs on every start-up: it ensures every Roles member exists as a row and that at least one system administrator exists to sign in as. It never applies migrations.

Each of those refusals replaces a failure that would otherwise appear far from its cause — an opaque IDX10703 on every authenticated request, a plaintext token in a production log, or a runtime error against a column that does not exist yet. Each is an unhandled exception at start-up: its message is the last thing the container logs, and the process exits non-zero, so the container restarts and the health check never turns green.

Diagnostics on a public host — NFR-28

Every environment but Production asks for EF Core’s sensitive data logging and detailed errors, which put query parameter values in the log, and Development and Local also ask for the developer exception page, which answers a failed request with its details. That suits a developer’s machine.

On an instance that serves a public host they are turned off, all three together, and the instance starts as usual with one warning:

This instance serves a public host (PUBLIC_HOST=heimdall-api-dev.example.com, UI_HOST=heimdall-dev.example.com),
so EF Core sensitive data logging, detailed errors and the developer exception page are off in the
"Development" environment. Set HEIMDALL_ALLOW_SENSITIVE_DIAGNOSTICS=true to keep them on where the
instance holds made-up people only

One decision, taken once at start-up, sets both EF Core’s options and the exception page, so neither can stay on while the other is off. It covers the exception page the host adds by itself in Development too: where the decision turns the page off, a failed request gets the API’s usual 500 body, “Internal server error, please try again later”.

A host is public when any of these names anything but localhost, a name under .localhost, or a loopback address: PUBLIC_HOST or UI_HOST (the names Traefik routes to the instance, which docker-compose.yml passes into the container), the host of HEIMDALL_EMAIL_VERIFICATION_URL or HEIMDALL_PASSWORD_RESET_URL, or any origin in HEIMDALL_CORS_ALLOWED_ORIGINS. A URL or an origin counts by its host, and one written without a scheme is read as a host name; a template placeholder ([[...]]) does not count. With none of them public, nothing changes: a developer machine and the functional suite keep their diagnostics.

HEIMDALL_ALLOW_SENSITIVE_DIAGNOSTICS=true keeps them on with a public host, and logs a warning at every start. Set it only where the environment holds no real person’s data. The templates leave it empty.

Health checks — UC-30

All three also answer at the root, /healthcheck, /healthcheck/ready and /healthcheck/detailed; the container health check calls /healthcheck and yggdrasil’s status page calls the others. Use the /api/ addresses through Traefik: on the web UI’s host, a path outside /api/ is served by heimdall-ui, not the API.

{
  "status": "Healthy",
  "services": [
    { "name": "Database", "status": "Healthy" }
  ]
}

The aggregate status is Healthy only when every entry is healthy (FR-HC-05).

Liveness and readiness answer different questions. Liveness says the process is running; it reads no database, so an instance that has lost its database still passes it. Use it where failing means a restart. Readiness says the instance can serve requests: it asks DatabaseHealthCheck alone and answers 200, or 503 when the database does not respond. Use it where failing means stop routing to the instance, which is what an orchestrator or load balancer in front of more than one instance wants. Its body is the same envelope as liveness’s with only the bare status as data, "Healthy" or "Unhealthy", never a service name or a cause (the cause is in the log, at Warning, from DatabaseHealthCheck). Like liveness it is anonymous and carries no rate limit, but it does not read the database per call: the answer, healthy or not, is held for 5 seconds and shared by every caller in that window, so a flood of probes costs the database one small read per 5 seconds and, while it is down, writes one Warning per window rather than one per request. That is shorter than a typical 15-second probe interval, so the probe still sees a change at its next call.

Which probe Compose uses. The container health check in docker-compose.yml calls liveness (/healthcheck), not readiness. Each environment runs one API instance, and Traefik routes only to healthy containers, so a readiness probe would take the only instance out of routing during a database outage: the web UI’s same-origin /api/* calls could then fall through to heimdall-ui’s catch-all (200 with HTML), and the public host would answer Traefik 404s instead of the API’s JSON 503. Switch the Compose probe to /healthcheck/ready only when more than one instance runs behind Traefik, so a healthy one can take the traffic.

The liveness and readiness endpoints are anonymous because a load balancer cannot authenticate; the detailed one is System-Admin-only because “which of my dependencies is down” is operational intelligence, which a bare status is not.

Each dependency is one IServiceHealthCheck registration. DatabaseHealthCheck issues a trivial read through the repository abstraction and catches everything — an unreachable database throws on execution, and the check reports unhealthy rather than propagating, so the aggregate can still be reported (AF-30c). Adding a verification is one more registration; the detailed handler resolves them all as IEnumerable<IServiceHealthCheck>.

Metrics — Prometheus

The API publishes its metrics in Prometheus text format at GET /metrics on port 9464 — and on that port only. They are collected with OpenTelemetry: request rate, duration and status per route (http.server.*), Kestrel connections, outbound HttpClient calls (Mailgun), the .NET runtime (GC, heap, thread pool, exceptions), EF Core, and the Npgsql connection pool. Every series carries service_name="heimdall-api".

The endpoint is recognised by the port the connection arrived on, never by the Host header. On the VPS (development, homologation, production) the API runs behind Traefik, which forwards the client’s own Host header — a check on it would be a check on a value any caller chooses. The local port of the socket is the one thing a caller cannot pick: Traefik reaches the container on 8080, Prometheus reaches it directly on 9464 over a private Docker network, and a request for /metrics on 8080 falls through to the API, which has no such route (401 without a token, 404 with one).

That guarantee holds only while 9464 stays private, so:

• Never publish 9464. docker-compose.yml maps only the API’s port; the image listens on both (ASPNETCORE_HTTP_PORTS=8080;9464). Put Prometheus on a network it shares with the api service and scrape it there — under yggdrasil, heimdall-api.<environment>:9464 on the telemetry network.
• Never route Traefik to 9464. With two ports exposed, give Traefik the service port explicitly (traefik.http.services.<name>.loadbalancer.server.port=8080) rather than letting it pick.
• Moving the port means moving both. A HEIMDALL_METRICS_PORT other than 9464 must also be added to ASPNETCORE_HTTP_PORTS, or Kestrel never listens where the endpoint waits.

The endpoint is mounted first in the pipeline, as a terminal branch: a scrape never meets CORS, rate limiting, authentication or authorization, and — not being a controller action — it is not in the OpenAPI document. The rest of the API is still served on 9464 too; only /metrics is special there, which is harmless for a port nothing outside the private network can reach.

A minimal scrape job:

scrape_configs:
  - job_name: heimdall-api
    static_configs:
      - targets: ["api:9464"]

Running from source (dotnet run), the API listens on 5177 only, so /metrics is not served; add the port to the URLs it listens on (--urls "http://localhost:5177;http://localhost:9464") to try it.

Logging

Serilog, configured before anything else so that even configuration failures are recorded:

• Console — JSON formatted.
• Rolling files — <HEIMDALL_LOG_DIRECTORY>/log-<yyyyMMdd>.json, one file per day, flat in the directory. HEIMDALL_LOG_DIRECTORY defaults to logs.

EF Core diagnostics are enabled outside Production only. SensitiveDataLogging and DetailedErrors print parameter and column values — password hashes, salts, email addresses — so they stay off where those logs are retained.

Server errors

Every 5xx the API answers leaves one line, so the trace id a caller holds leads to a cause. An unhandled exception is logged at Error with its stack trace by ExceptionMiddleware. A 5xx a handler returns as a result, such as a persistence error mapped to 500, is logged at Error by ServerErrorLoggingFilter as Request to {Route} answered {StatusCode} (trace {TraceId}): the route template (api/person/{publicId}, never the path with its ids), the status and the trace id, and nothing of the query string, body, headers or caller. A 503 is the exception, logged at Warning: here it is always a deliberate answer (a saturated password-hash gate), not a fault. A readiness probe’s 503 gets no such line at all: the database check’s own The database health check failed Warning, one per 5-second window, is the only record.

Retention and redaction — NFR-22

Log files are written flat under the log directory, one per day, and removed once older than HEIMDALL_RETENTION_LOG_DAYS (365 by default). They used to be nested in year/month directories via Serilog’s Map sink, which created a new sink per month — a retention limit bounds files within a sink, so it bounded each month’s directory and never removed a month. The flat sink is the shape where the limit applies.

No log statement writes an email address. Where a line needs to refer to one so an operator can correlate, it writes a stable non-reversible reference ({EmailRef}) instead. LogRedactionTests scans the source and fails if an {Email} placeholder reappears anywhere.

The reference is pseudonymisation, not anonymisation: somebody who already suspects an address can hash it and look for it. What it defeats is reading addresses out of the logs in bulk.

Rate limiting and lockout

Brute force is bounded in two independent places, because each covers what the other misses.

Per caller — a fixed-window limiter of 10 requests per minute, partitioned by client IP (an IPv6 caller by its /56, since a subscriber is routinely handed a whole /56 and counting per /64 would give them 256 partitions), applied to AuthController’s anonymous, credential-checking endpoints: login, password recovery, password reset, email verification, Google sign-in, and second-factor verification and resend. Rejections answer 429 with a Retry-After header (whole seconds) and the usual error envelope. None of these requires a bearer token, so nothing else bounds how fast one caller can hit them, and each login attempt costs a full Argon2id verification (600 MB / 16 threads by the hashing library’s defaults) — so memory exhaustion is realistic without it. The authenticated endpoints that check a password or a second factor — 2FA enable, confirm, disable and recovery-code regeneration, the erasure request, and the person update (PUT /api/persons/{id}, which checks the password when a person changes their own email) — share the same limit: a bearer token is not the password, and without it whoever held one could test passwords or six-digit codes against them unthrottled. The person update is limited whether or not it checks a password, administrators' updates included. The verification-link resend (POST /api/auth/resend-verification) shares it too, because every call mails the account’s address.

Per administrator — account creation. The four endpoints that create a person with a password or invite an existing administrator — POST /api/persons, POST /api/scopes/{id}/persons, POST /api/scopes/{id}/owners and POST /api/scopes/{id}/owners/{personId} — the first three derive an Argon2id hash, and all need nothing but an administrator’s token. They are limited to 30 requests a minute per acting person and 30 a minute per address, both at once; a rejection is the same 429 with Retry-After. Only administrators’ access tokens count against the address, so other requests — without a token, or with a User’s, a two-factor challenge or a subject-rights token — cannot use up an address administrators share. Thirty a minute is more than anyone creates by hand, and a bulk import should pace itself to it. For traffic from many addresses at once, see Bounding a distributed flood at the edge .

Per account — a failure budget, which is what an attacker spread across many source addresses defeats the limiter with:

Every budget, and every single-use item — recovery codes, reset and verification tokens, the challenge, the TOTP step — is charged or spent by one conditional UPDATE whose WHERE clause is the rule itself (“unused”, “under the cap”, “still outstanding”), and before the comparison it guards rather than after it. A burst of parallel requests therefore cannot outrun a budget or spend one code twice: the database lets exactly as many through as the rule allows. A login attempt is counted in failed_login_attempts while it is in flight and cleared if the password turns out to be right.

A lockout is a window rather than a latch an administrator clears: reaching the threshold needs nothing but wrong guesses, so a permanent lock would hand any anonymous caller a denial of service against any address they know. It answers with UC-11’s ordinary InvalidCredentials, and spends the same Argon2id work a real check would, so it is not observable — by message or by timing — to a caller who does not already know the password.

Releasing a second-factor lock or a re-authentication pause

Both run out on their own: a second-factor lock after at most 24 hours, a re-authentication pause after 15 minutes. Heimdall has no endpoint to release either early, on purpose. For a support case that cannot wait, an operator can clear them in the database.

Replace the id with the person’s public_id (the id the API returns for them):

-- Second-factor lock at sign-in (FR-AU-12): the count, the lock, and the repeat count that sets
-- how long the next lock lasts.
UPDATE heimdall.person
SET failed_second_factor_attempts = 0,
    second_factor_locked_out_until = NULL,
    second_factor_lockouts = 0
WHERE public_id = '00000000-0000-0000-0000-000000000000';

-- Re-authentication pause (FR-AU-13): the count, the window and the pause.
UPDATE heimdall.person
SET failed_reauthentication_attempts = 0,
    reauthentication_window_started_at = NULL,
    reauthentication_paused_until = NULL
WHERE public_id = '00000000-0000-0000-0000-000000000000';

Each statement should report UPDATE 1. Run only the one you need. The person table’s triggers guard public_id alone, so these columns are writable by the API’s own role.

Bounding a distributed flood at the edge

Every password check passes through PasswordHashGate: four Argon2id derivations at a time per instance, about seven hundred a minute, the rest refused with 503 after a ten-second wait. The per-caller limit admits ten a minute, so about seventy unrelated networks sending login attempts with made-up addresses keep the gate full — and while they do, real sign-ins wait and fail. The API cannot stop this itself (Threat Model Document, TH-03): those callers look like real ones, and a global budget in the API would only let the attacker spend it and deny everybody at a lower cost. The bound belongs at the edge, where the request costs nothing to refuse.

On the VPS that edge is Traefik (v3.6), and yggdrasil 0.7.4 and later ship this limit: the labels below are in its stacks/heimdall-api.proxy.yml, beside the heimdall-api-${YGG_ENVIRONMENT} and heimdall-api-${YGG_ENVIRONMENT}-same-origin routers. On an older yggdrasil, or any other overlay that declares Heimdall’s routers, add them there. Every name carries ${YGG_ENVIRONMENT}, because the environments share one Traefik.

labels:
  # The credential routes, on both hosts: the API's own and the web UI's same-origin /api/.
  # (?i): Traefik matches paths case-sensitively and ASP.NET does not, so /api/AUTH/login must not
  # slip past on the main router. Every method but GET, HEAD and OPTIONS, so the profile and admin
  # pages' GETs under /api/auth/ stay on the other routers. The safe methods are excluded rather than
  # POST named: Traefik matches a method case-sensitively too, and ASP.NET routes `post` to [HttpPost].
  - traefik.http.routers.heimdall-api-${YGG_ENVIRONMENT}-auth.rule=(Host(`${PUBLIC_HOST}`) || Host(`${UI_HOST}`)) && PathRegexp(`(?i)^/api/auth/`) && !Method(`GET`) && !Method(`HEAD`) && !Method(`OPTIONS`)
  - traefik.http.routers.heimdall-api-${YGG_ENVIRONMENT}-auth.entrypoints=websecure
  # The chain is replaced, not extended, so keep the headers every other router has.
  - traefik.http.routers.heimdall-api-${YGG_ENVIRONMENT}-auth.middlewares=secure-headers@file,heimdall-api-${YGG_ENVIRONMENT}-auth-limit
  - traefik.http.routers.heimdall-api-${YGG_ENVIRONMENT}-auth.service=heimdall-api-${YGG_ENVIRONMENT}
  # Per source: 10 a minute with a burst of 10, an IPv6 caller counted by its /56, as the API does.
  - traefik.http.middlewares.heimdall-api-${YGG_ENVIRONMENT}-auth-limit.ratelimit.average=10
  - traefik.http.middlewares.heimdall-api-${YGG_ENVIRONMENT}-auth-limit.ratelimit.period=1m
  - traefik.http.middlewares.heimdall-api-${YGG_ENVIRONMENT}-auth-limit.ratelimit.burst=10
  - traefik.http.middlewares.heimdall-api-${YGG_ENVIRONMENT}-auth-limit.ratelimit.sourcecriterion.ipstrategy.ipv6subnet=56

• TLS is set on the websecure entrypoint, so the router needs nothing more. No priority is needed: Traefik ranks the longer rule first, and it is longer than the main router’s.
• A client system on the same host stays off the edge. A client system that signs its users in through Heimdall reaches it from one source, so through the edge all of its users would share one bucket of 10 a minute. Inside yggdrasil it calls Heimdall at http://heimdall-api.<environment>:8080/, the API’s alias on the edge network, and never through Traefik. fortuna-api 3.1.0 and later do so. Fortuna’s caller key is trusted, so Heimdall counts each of its users’ sign-ins under that user’s own address (A client system calling for its users ).
• A relay that has to cross the edge needs a source of its own. A client system on another host that relays its users’ sign-ins needs a router and a bucket of its own, matched on a source only it has and sized for its users’ sign-ins in a minute at their busiest; never one bucket shared by every consumer, and never the public bucket. Do not match it with ClientIP when its ClientHost in Traefik’s access log is a Docker bridge gateway or a NAT router’s address — which it can be when it reaches Traefik through the public host, and which other traffic then shows too (hairpinned clients, or IPv6 clients that Docker’s port proxy forwards): everything behind that address would land in the relay’s large bucket. Route it by a private network to Traefik or a host name of its own instead.
• Check the addresses first. Before relying on any of this, confirm in Traefik’s access log that ClientHost shows real client addresses, IPv6 ones included, rather than a proxy’s or Docker’s. A limiter keyed on the wrong address puts everybody in one bucket.
• This is the same per-source limit the API applies, moved in front of it: a refused request then never reaches an instance, and one Traefik counts across every replica behind it, where each instance’s own limiter counts alone.
• It does not, by itself, stop a flood from enough unrelated networks: that is the definition of one. What does is visibility and a decision: the hash gate’s 503s and the CREDENTIAL_VERIFICATION_SHEDDING security signal show it starting; the source networks are in Traefik’s access log; and a flood that persists is for the host firewall or an upstream filtering service, which can drop whole networks before they reach Traefik. Raising the replica count raises the gate’s total in proportion while it lasts.
• Do not add a global limit (one with sourcecriterion.requesthost=true) to the credential routes. It turns the flood into a denial of every caller at a lower cost, which is exactly what the API declines to do.

Edge 429 or the API’s. Either limit can refuse a credential request with a 429, and the two answers differ. The edge counts with a token bucket, ten at once and then one more every six seconds; the API with a fixed window of ten a minute. A burst therefore meets the edge’s 429 first, and a client that retries after a few seconds can get one request past the edge only to meet the API’s 429, with a longer Retry-After.

A client must not expect a JSON body on a 429, and should wait out Retry-After whichever limit answered. heimdall-ui shows its rate-limited message for both, with no server message for the edge’s. Fortuna’s users never meet the edge’s 429, because Fortuna does not cross the edge: it relays the API’s own.

Bounds on the mail one address receives

Heimdall sends mail from the operator’s Mailgun domain, so mail an attacker can trigger costs the operator quota and sender reputation as well as annoying its recipient. A per-caller limit does not bound what one recipient receives from many callers. The sends a caller can repeat are bounded as follows; mail an administrator causes — the links of UC-51 and UC-52, a new account’s verification link, the notice when they change someone’s address — is trusted to administrators, as their other powers over the accounts they manage are:

The self-registration and recovery bounds are silent by design: the caller gets the same answer whatever happened, so the bound cannot be used to learn whether an address is registered.

The per-scope cap on live pending registrations is there because the per-IP rate limit does not bound a caller with many source addresses, and the purge removes a registration only once it expires: without it the table, and the confirmation mail, grow with whatever arrives. A registration turned away by the cap is logged at Warning with the marker REGISTRATION_CAP_REACHED and the scope’s public id — never the address. A scope that reaches it is either being flooded or has outgrown the cap: look at its pending registrations (UC-49) before raising HEIMDALL_REGISTRATION_MAX_PENDING_PER_SCOPE. Any positive integer is accepted; an unusable value falls back to 1000 with a start-up warning. The count is read and then written, so instances processing registrations for one scope at the same moment can each add one past the cap — a small overshoot, at most one per instance.

docker logs heimdall-api 2>&1 | grep REGISTRATION_CAP_REACHED

A client system calling for its users — HEIMDALL_TRUSTED_CALLER_KEYS

A client system such as Fortuna’s API signs its users in on their behalf, so Heimdall sees one address for all of them — and the 10-a-minute limit on the anonymous endpoints is then shared by every user it serves. It can instead tell Heimdall whose request this is. Heimdall counts the request under that user’s address when it carries a key from HEIMDALL_TRUSTED_CALLER_KEYS:

• HEIMDALL_TRUSTED_CALLER_KEYS is a comma-separated list. Entries are trimmed and empty ones ignored. Each key must be at least 32 characters — they are secrets, so a shorter one stops start-up with an error naming its position, not its value. Generate them randomly, for example openssl rand -base64 48. Unset, no caller is trusted and the headers are ignored.
• The key is compared in constant time against every configured key. A wrong or missing key, or an address that does not parse, is not an error: the request is counted under the connection address as before. The headers can only choose the bucket; they authenticate nobody and open no endpoint.
• Rotating a key needs no downtime: list the new key beside the old (HEIMDALL_TRUSTED_CALLER_KEYS=<old>,<new>), restart Heimdall, move the client system to the new key, then remove the old one and restart again.
• Send the headers only from the client system’s server, over TLS, or over a private network nothing public routes to, never from a browser or a mobile app, and keep the key in that system’s secret store.
• A named address is counted in its own caller: namespace, apart from direct connections, so a key holder cannot spend the budget of a real caller by naming their address.
• A leaked key lets its holder pick their own bucket, which only spreads credential guesses over invented addresses; the per-account budgets still count them. That includes the six authenticated endpoints the same limiter guards — 2fa/enable, 2fa/confirm, 2fa/disable, 2fa/recovery-codes/regenerate, erasure-request and PUT /api/persons/{id} — which share the re-authentication budget above, whatever addresses the guesses come from (Threat Model Document, TH-32 and TH-36). The seventh, resend-verification, is bounded per person by the account-email budget below, whatever bucket it is counted in.
• Start-up logs how many keys are configured, never the keys, and neither header’s value is logged.

Database connection encryption — NFR-25

The API warns at start-up when HEIMDALL_DATA_CONNECTIONSTRING does not require TLS. Npgsql’s default SSL Mode is Prefer, which silently falls back to an unencrypted connection when the server offers none — so a deployment can be sending every credential and address in clear text while looking correctly configured. Set SSL Mode=Require, VerifyCA or VerifyFull.

Turn it on once TLS is confirmed. It is opt-in rather than the default so that enabling it is a decision somebody makes, not one that happens to them during an upgrade — but a deployment that has verified TLS and left this off is one warning away from not noticing when the setting is lost.

Data retention — NFR-19

Personal data is kept only as long as its purpose requires (GDPR Art. 5(1)(e), LGPD Art. 15). The schedule — every table holding personal data, its period, and the reason for it — is the Data Retention Schedule Document , and it is the source of truth these settings are configured against.

The schedule marks which periods are enforced and which are not yet, so it never describes a guarantee the code does not make. Read it there rather than inferring enforcement from the presence of a setting here.

Single-use tokens are purged on a schedule. A hosted service dispatches a purge hourly by default, removing password reset tokens, email verification tokens, and two-factor email codes that are past their expiry by the grace period — and, with them, unconfirmed pending registrations, spent application credentials, self-registration attempt-notice budgets whose day has ended, and the records of used Google ID tokens past their expiry by the grace period. It also removes scope owner invitations 90 days after they end, or 90 days after they expire if never answered. Each table is taken a bounded batch at a time, oldest first, and each batch is its own short delete — a purge is delete pressure on the same tables the login and recovery paths write to, and the write path degrades with table size (SRD §6.3.2).

Each pass drains its backlog, up to a cap per run. The token purge, the identity anonymisation and the audit pseudonymisation each repeat their batch while it comes back full, up to HEIMDALL_RETENTION_MAX_BATCHES_PER_TICK batches per table in one run. At the defaults (500 rows, 20 batches, hourly) that is a ceiling of 10,000 rows per table an hour, 240,000 a day — where one batch per run capped a table at 12,000 a day, which a busy audit trail outgrows for good. A backlog larger than the cap is left for the next run, and the run’s log line ends with how many rows are still due after this run: a number that does not fall from run to run is a backlog that is not draining, and the remedy is a larger batch or more batches per run. The anonymisation’s retry of blocked erasure requests (below) takes its batches the same way, each starting after the last request of the one before, so requests that stay blocked cannot keep a newer one from being retried.

The grace period is not zero, deliberately. UC-13 answers a presented token with TokenInvalid, TokenExpired or TokenAlreadyUsed, and purging the moment a token expires collapses the last two into the first — telling someone following a stale link that their token never existed. A live token is never at risk: the cutoff is strictly in the past.

Every instance runs its own purge and none coordinates with the others (NFR-06). The delete re-reads its batch, so an instance whose rows another already removed simply deletes fewer than it selected. Each run writes one audit entry (NFR-09) as an anonymous write — the evidence that the schedule was enforced, and when. A run that removes nothing is a success; most runs find nothing to do.

Logically deleted identities are anonymised on a schedule. A soft delete is a restriction of processing, not an erasure — the name, the address and the credential material all stay in the row. A second pass gives it a terminal state: once the retention window has elapsed, the record is anonymised in place, and the person’s single-use tokens and two-factor configuration go with it.

The window depends on why the record was deleted. Where the data subject asked (UC-42), it is 30 days — GDPR Art. 12(3)’s outer limit rather than a chosen period, and refused if configured higher. Where an administrator deleted it, it is 90 days, a reversal window. Records deleted before this existed carry no reason and take the longer one.

A subject’s request stores its own deadline when it is made, and that stored value wins over the configured window: a later change to the setting must not move an obligation already owed. Where NFR-12 blocked a request — the subject was a scope’s last active owner — the pass retries it once ownership has been transferred, and where NFR-32 blocked one — the subject was the last active System Admin — once another System Admin exists. It dates the suspension from the request rather than from the moment the block cleared, so a long-blocked request is due immediately rather than handed a fresh deadline. GET /api/auth/erasure-requests is the queue that surfaces those before they are late.

Anonymisation overwrites rather than removes, because NFR-07 requires every foreign key to keep resolving and the audit trail, the scope join rows and an owner’s applications all point at the record. Hard deletion (UC-10) remains available for callers that want the row gone.

The audit trail stops naming people on its own schedule. A third pass clears the actor attribution from entries past their attribution period, and immediately from entries naming an identity that has been anonymised. On the same two grounds it clears an entry’s target when that target is a person or Google User — a self-service command’s own account, or the person an administrator acted on. Targets naming anything else are kept. The entry itself is never removed and what it says happened is never altered — what happened, when and how often all survive.

The rule is enforced by a database trigger rather than by the pass, because this is the one table whose immutability is a security control. It permits the attribution, and a target naming a person, to be set to null and nothing else: no deletion, no truncation, no change to any other column, and no reassignment to a different value. A thirty-day floor stops somebody clearing their own attribution, or the person they acted on, for something they did this week; it is an anti-tamper backstop, not the retention period, which is eighteen months. A deployment upgrading to AllowClearingAuditTarget has its older person targets cleared as a backlog: a batch at a time, up to HEIMDALL_RETENTION_MAX_BATCHES_PER_TICK batches a run, over as many runs as that takes.

The pass finds what is due without reading the whole trail. Entries past the period are read through two partial indexes on created_at — over the entries that still have an actor, and over those that still have a target — and entries naming an anonymised or hard-deleted identity are found from those identities, so a run with nothing due costs a few index probes rather than a scan of eighteen months of entries. The one part that still grows with the trail is the check of old targets: an entry past the period whose target is a scope, an application or anything else that is not a person keeps its target for good, and each run looks at those again. The AddAuditRetentionIndexes migration builds the two indexes with CREATE INDEX CONCURRENTLY, so audit writes carry on while it runs; on a large trail it takes a while. If it is interrupted, drop the index it leaves marked invalid (\d heimdall.audit_log shows INVALID) before applying it again.

A deployment that sets none of these still gets the published periods — an unset variable must not mean “keep forever”. A malformed or out-of-range value falls back to the default and is named in a start-up warning rather than failing start-up: a typo in a retention setting should not take down the identity provider every client system authenticates against. Falling back silently would be worse than either, since an operator would believe a setting was applied.

Each bound is enforced for a reason. A grace period beyond ten years is a stray unit rather than a policy. An interval below a minute turns a purge into continuous delete pressure on the tables the login path writes to; one above seven days is past what PeriodicTimer accepts, and the exception it would throw inside a hosted service stops the host by default.

How much a pass can clear, and how to see a backlog

Each pass clears at most HEIMDALL_RETENTION_PURGE_BATCH_SIZE rows per batch and at most HEIMDALL_RETENTION_MAX_BATCHES_PER_TICK batches per table in one run, and runs every HEIMDALL_RETENTION_PURGE_INTERVAL_MINUTES. The ceiling for one table is therefore:

rows per day = batch size × max batches per tick × (1440 ÷ interval in minutes)

At the defaults (500 × 20 × 24) that is 10,000 rows per table per hour and 240,000 per day. The cap applies to each table or kind of row on its own, not to the pass: the token purge allows it for each of its eight tables (password reset tokens, email verification tokens, two-factor email codes, pending registrations, application credentials, attempt-notice budgets, used Google ID tokens, scope owner invitations); identity anonymisation for persons and for Google Users separately; audit pseudonymisation for the entries and, separately, the tombstones. Recompute it with your settings: a 250-row batch, 10 batches and a 30-minute interval give 250 × 10 × 48 = 120,000 per day. A table that is fed faster than its ceiling (a busy audit_log is the likely one) never drains, and the rows past the retention period pile up, which is a retention obligation silently lapsing; raise the batch size or the batches per tick before the interval, since a longer batch is cheaper than a more frequent run.

Seeing it. Each run that removed anything, or left anything due, logs one line at Information with the rows still due after it:

A run with nothing removed and nothing due logs nothing. A “still due” number that does not fall from run to run is a backlog that is not draining. Identity anonymisation prints an unknown number where counting the backlog itself failed (logged at Warning as Counting the identities still due failed). There is no backlog metric: heimdall_scheduled_pass_rows_processed_total per pass (see below) divided over time shows a pass working at its ceiling, which is rate(...) close to the figure above, and is the cue to read the log line.

When a scheduled pass fails

Every scheduled pass (token retention, audit pseudonymisation, identity anonymisation and security detection) logs a failed run at Error level with the marker SCHEDULED_PASS_FAILED and the pass name. A run fails when its command returns errors, as when the database refuses a write, and when it throws. The loop carries on and retries on the next tick. Match on the marker, not on the prose.

docker logs heimdall-api 2>&1 | grep SCHEDULED_PASS_FAILED

The same passes publish two metrics on the Prometheus endpoint , labelled by pass:

A pass that has not succeeded since the API started has no sample for the gauge. To alert on a retention obligation silently lapsing, alert on the age of the last success, for example time() - heimdall_scheduled_pass_last_success_timestamp > 6 * 3600 for a pass that runs hourly, or on absent(heimdall_scheduled_pass_last_success_timestamp{pass="identity-anonymisation"}). Give an absent() rule a for: of at least one pass interval plus a margin (for example for: 75m on an hourly pass), and expect it to fire for a pass that is disabled, because a disabled pass never sets the gauge.

An identity anonymisation run that retries a blocked erasure request and fails is reported the same way, but it does not stop the run anonymising the identities that are due: the failure is logged with the request’s person and the run’s errors, and the identities anonymised in that run are still named in the erasure ledger line.

Security signal detection — NFR-26

The API records strong signals and, until now, nothing read them. SecurityMonitoringService reads them on a schedule and writes what exceeds its thresholds to the log at warning level.

Every line carries the marker SECURITY_SIGNAL and a stable Kind. Match on those, not on the prose — the wording is written for whoever reads the line afterwards and will change.

Every threshold is a rate, not a total: twenty refusals over a year is somebody who forgets their password, twenty in a quarter of an hour is somebody trying things.

Signals repeat on every run while the condition persists. That is deliberate — an attack that continues is more interesting on its tenth tick than its first, and a detector that goes quiet while the thing it detects is still happening is worse than none. Deduplicate in the alerting rule if you want quiet, not here.

A signal is not a breach. It is a reason to look. What follows one is the Incident Response Document , which carries the notification deadlines and the register.

What a signal actually looks like

A warning-level log line, written to stdout and to the files in the log directory:

SECURITY_SIGNAL REPEATED_REFUSALS: One identity had 23 writes refused in the last
00:15:00. Sustained refusals are what somebody probing for what they may do looks
like. (count 23, actor 3f2a91c4-7d55-4e1a-9b2f-08c7d1e6a934)

To see what has been raised so far:

docker logs heimdall-api 2>&1 | grep SECURITY_SIGNAL

grep -h SECURITY_SIGNAL /path/to/logs/log-*.json

Collecting them — scripts/security_signals.py

Detection that tells nobody is not detection: both breach notification clocks run from awareness, so a signal sitting in a log file has not been detected. This job is what closes that. It reads the log files, finds the signals it has not reported before, and mails them to one address through the Mailgun account the API already uses.

python3 scripts/security_signals.py --stdout      # see what it would send, change nothing
python3 scripts/security_signals.py               # report new signals by e-mail
python3 scripts/security_signals.py --stdout --all  # ignore the watermark, print everything

Scheduling it. Every fifteen minutes matches the monitoring window’s default, so nothing waits longer to be reported than it waited to be detected:

[email protected]
*/15 * * * * cd /srv/heimdall && /usr/bin/python3 scripts/security_signals.py --quiet

The MAILTO is not decoration. The job exits non-zero when it cannot deliver, and prints the signals it was holding to stderr — so a broken collector announces itself instead of looking like a quiet week. Without a MAILTO, the failure of the thing that catches failures is itself uncaught. --quiet keeps the ordinary no-signal run silent, so the only mail cron sends is a real problem.

It needs to read the log files. docker-compose.yml keeps them in a named volume (logs:/app/logs), which the host cannot read directly — so run the job against that volume rather than against a host path:

docker run --rm \
  --volumes-from heimdall-api \
  -e HEIMDALL_LOG_DIRECTORY=/app/logs \
  -e HEIMDALL_SIGNAL_ALERT_TO -e MAILGUN_API_KEY -e MAILGUN_DOMAIN \
  -v "$PWD/scripts:/scripts:ro" python:3-slim \
  python3 /scripts/security_signals.py --quiet

The alternative is to change the volume to a bind mount and run the job on the host, which is simpler to schedule and puts the logs somewhere a backup can reach them. Either works; what does not work is scheduling it on the host and pointing it at a path the container owns.

The state file is written next to the logs by default, so it survives in the volume. Point HEIMDALL_SIGNAL_STATE_FILE somewhere writable if the log directory is read-only — and somewhere persistent, because losing it means the next run re-reports the history.

Exit codes: 0 ran and delivered whatever it found · 1 signals were found and could not be delivered · 2 the collection itself could not run.

Why it keeps a watermark

Signals repeat on every monitoring tick while a condition persists — that is deliberate in the service, and it would be intolerable in an inbox. The job records how far it has read, so each signal is reported once. An alert that resends its entire history is one that gets filtered into a folder within a week, and a filtered alert is the same non-detection the job exists to close.

Two details follow from that, and both are the conservative direction:

• The watermark advances only after delivery succeeds. A Mailgun outage delays signals rather than consuming them.
• An unreadable state file is treated as a first run, so the history is re-reported. Noisy, and the alternative — a truncated file silently meaning “everything is already handled” — is detection switching itself off without saying so.

Signals raised in the same monitoring tick share a timestamp exactly, so the watermark carries the digests of everything at its own instant. Without that, one of a pair raised together would be lost.

What is in the alert

The marker, the kind, the count, the timestamp, and the actor’s PublicId. That is a pseudonym — it identifies an account to somebody holding the database and to nobody else — so the alert can be read on a phone without carrying anybody’s e-mail address into a mailbox. NFR-22 keeps addresses out of the logs to begin with; the alert inherits that rather than re-deciding it. Do not extend the job to include addresses.

The alert also says, in its own body, that the moment you read it is when awareness happened — the timestamp that goes in the Art. 33(5) register, not the time the signal was raised.

What to alert on

Everything below is a log line or a metric the API already writes; nothing needs new instrumentation. Match on the marker or the quoted text, not on the surrounding prose, and on the level shown. Logs are JSON on the console and in the log directory; see Logging .

An alert on Warning or above catches all but the last. The prose around a marker will change; the markers and the pass names are stable.

Query costs to watch

Three costs depend on how much data a deployment holds. None needs a change today; each says what to look at and when to act, so a slow screen is not a surprise.

• Name and email filters scan. The list endpoints filter with x.Name.ToLower().Contains(name) (and the same for email) in the scopes, persons, owners, admins, Google Users, System Admins, applications and permissions listings. A match with a leading wildcard cannot use a b-tree index. For a scope’s own listings that is fine: the query is limited to one scope’s rows first. The one place it can matter is a System Admin listing across every scope. Act only if those listings are slow with real data: add a pg_trgm GIN index on the lower-cased column, built CONCURRENTLY as in Migrating big tables .
• Every page of the audit listing counts the table. ListAuditLogQueryHandler pages with PaginateAsync, which issues a COUNT(*) with the same filters on each request (library behaviour). Filtered by actor, target, action or date range that is cheap; on an unfiltered request it counts all of audit_log, a table that only grows. Watch the listing’s latency as the trail grows; if the unfiltered count becomes slow, move the listing to keyset pagination on (created_at, id) or return an approximate total, and tell the UI, which shows a total.
• Every authenticated request reads the actor. ActorLivenessFilter makes one or two indexed queries (by public_id) per authenticated request so that a deleted or restricted person’s token stops working at once. That is deliberate (see Scaling ); it is the price of immediate revocation. Do not cache it to save the reads: a cache is exactly the window in which a deleted person’s token still works.

Cross-origin requests

HEIMDALL_CORS_ALLOWED_ORIGINS lists the browser front ends allowed to call the API, comma separated, as scheme and host (https://app.example.com). With the variable unset, no cross-origin request is allowed.

Refusing by default is deliberate. The same-origin policy is what stops a page on an unrelated origin from reading an authenticated response, and an identity API is the last place to switch it off: any site the caller visited could otherwise read /api/persons/{id} with a token it scraped, and drive the anonymous endpoints from every visitor’s browser at once. A missing entry costs a front end its access until an operator adds one — visible, and quickly fixed. Defaulting to “any origin” would instead leave a deployment open with nothing to indicate it.

Server-to-server callers are unaffected: CORS is a browser rule, and non-browser clients send no Origin header.

Client addresses behind a proxy

On the VPS every connection reaches the API from Traefik, so the connection’s address is Traefik’s. The caller’s own address travels in X-Forwarded-For, and the scheme they used in X-Forwarded-Proto. HEIMDALL_TRUSTED_PROXIES lists the proxies those headers are believed from — addresses and CIDR networks, comma separated:

Two things read the result: the rate limiter, which partitions on it, and the request log, which records it (Started request with TraceId … from <address> on every request). Unset, nothing is trusted and both see Traefik — one rate-limit bucket for every caller, and a log that names the proxy. Start-up warns when that is the case.

Only the headers’ last hop is read: Traefik appends the address it saw to whatever the caller sent, so the rightmost entry is the only one it vouches for, and an earlier entry a caller forged is ignored. Never list a range a caller can originate from — anything trusted here can claim any address it likes. X-Forwarded-Host is not read; nothing in the API decides on the host name.

Integrations

Email delivery (Mailgun)

Region: United States. This is a deliberate choice, not an inherited default — Mailgun also offers an EU region. Data is hosted in Brazil, so every send is an international transfer under LGPD Art. 33 whichever region is picked; the choice is about which second jurisdiction is involved. The mechanism relied on is recorded in §7 of the Data Protection Document . Changing the region means updating that section and the sub-processor list in the Data Processing Agreement .

Verification (UC-06) and password reset (UC-12) emails, and 2FA email codes, go out through Mailgun via ArturRios.Messaging . Delivery is enabled only when both credentials are present:

Each page finishes the job by posting its token back — the verification page to POST /api/auth/verify-email, the reset page to POST /api/auth/password-reset with the new password. If no link is configured the email carries the bare token instead, which still works.

Pointing a scope at a client application’s screens

The two variables above are one link for the whole deployment. A client system such as Fortuna has its own /verify-email and /password-reset screens, and its users’ emails must lead there. A scope can carry its own link bases, which replace the global ones for that scope’s Users only:

PUT /api/scopes/{id}/email-links
Authorization: Bearer <System Admin or owning Scope Admin token>

{
  "emailVerificationUrl": "https://app.example.com/verify-email",
  "passwordResetUrl": "https://app.example.com/password-reset"
}

• Each must be an absolute http or https URL of at most 2048 characters; anything else is a 400. The token is appended as a token query parameter, as for the global links.
• The body is the whole state of the two links: one left out, null or empty is cleared, and the global link applies to that email again. They have their own endpoint, not fields of PUT /api/scopes/{id}, because that update replaces every field and would erase them whenever a client that does not know them sent it.
• The verification link is used for the sign-up confirmation (UC-47) and for address verification (UC-06, UC-15); the password-reset link for password recovery (UC-12). A Google User receives neither.
• Administrators have no scope, so their emails always use the global links.
• The client’s screens finish the job as the global ones do: the verification screen posts the token to POST /api/auth/verify-email (with the chosen password for a sign-up confirmation), the reset screen to POST /api/auth/password-reset.
• Both are returned on every scope response (GET /api/scopes, GET /api/scopes/{id} and the write endpoints).

Password recovery is answered before it is worked. POST /api/auth/password-recovery only validates the request and puts it on an in-process queue; a background worker then looks the address up, issues the token and sends the email. So the response takes the same time whether or not the address is registered — otherwise the Mailgun round trip a registered address waited for would give away what the uniform answer hides (AF-12a). Two consequences for operators:

• A restart drops whatever is still queued — at most a few seconds’ worth. Nobody is told; the person asks again, as they would for an email that never arrived.
• The queue holds 1,024 requests. Beyond that a request is dropped and the log says The password recovery queue is full — a sign that Mailgun is down or very slow, not that the endpoint is under attack (the rate limiter caps that first).

Google Sign-In

Unset, the API registers a verifier that refuses every token with a 401 and warns once at start-up. Unlike the email fallback this is not a convenience mode: verification needs an audience to check against (NFR-13), so a verifier with no configured client could only reject everything or trust everything. Every other endpoint is unaffected, as is any scope that never enabled the feature.

Google sign-in also requires the scope itself to have it switched on, through PUT /api/scopes/{id}/google-signin — it is off by default. See Google Sign-In .

Signing key

Heimdall signs every token it issues (person, subject-rights-only, two-factor challenge and application) with ES256, using the private key in HEIMDALL_AUTH_SIGNING_KEY. The key is one value per environment: the one-line base64 of a P-256 PKCS#8 PEM. Only Heimdall holds it; everything that verifies a token holds public keys only (Threat Model TH-43).

1. Generate the key: openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256 -out signing.pem
2. Set the variable to base64 -w0 signing.pem, keep the PEM only as long as you need it, and delete the file. The key is a secret and goes in the environment’s variable store, never in a template or a log.
3. Publish. Nothing to do: the public half is served at GET /.well-known/jwks.json as soon as the API runs. Each token’s header carries a kid, the RFC 7638 thumbprint of its key, derived from the key and never configured.

The API refuses to start without a P-256 private key in the variable, with a message that names the variable and never the value (NFR-31). At start-up it logs Tokens: signed ES256 with key {KeyId}; published keys {KeyIds}; HS256 tokens {LegacyState}, which is the line to check after a deploy: the key id it names is the one a consumer will see in the key set. LegacyState is always refused: no HS256 token is accepted.

Rotating the key

A rotation replaces the key without signing anyone out. The old key stays published and its tokens stay valid; it never signs again.

1. Set HEIMDALL_AUTH_SIGNING_KEY_PREVIOUS to the value currently in HEIMDALL_AUTH_SIGNING_KEY, set HEIMDALL_AUTH_SIGNING_KEY to a newly generated key, and redeploy. The key set now lists both, the new one first. New tokens carry the new kid; tokens issued before the redeploy keep working.
2. Wait the longest token lifetime: the longest of HEIMDALL_AUTH_TOKEN_EXPIRATION_IN_SECONDS (an hour by default), HEIMDALL_APPLICATION_TOKEN_EXPIRATION_IN_SECONDS (ten minutes by default) and the two-factor challenge token’s fixed ten minutes, so that no token signed with the old key can still be alive.
3. Clear HEIMDALL_AUTH_SIGNING_KEY_PREVIOUS and redeploy again. The old key leaves the key set.

Every instance must carry the same pair, so roll the change out to all of them before step 3: an instance that has already dropped the old key refuses tokens its neighbours still accept. Both variables are held to the same format, and an empty previous variable (including one holding just "") means no previous key.

Emergency rotation

When the old key leaked, do not keep it: set HEIMDALL_AUTH_SIGNING_KEY to a new key, leave HEIMDALL_AUTH_SIGNING_KEY_PREVIOUS empty, and redeploy. Withdrawing a key invalidates its tokens at once, which signs everybody out. That is the correct response to a compromised key and the wrong one to a routine replacement. Then follow the Incident Response Document .

Consumers learn of the new key without being told, but not at once. Fortuna keeps trusting the leaked key until a token with the new kid reaches it: that first token replaces Fortuna’s cached key set, throttled to at most one fetch a minute, and Fortuna ignores Cache-Control, so the max-age=300 on the key set does not matter. Any fresh sign-in issues a new-kid token, so sign in once through Fortuna right after the redeploy to make that happen promptly. Until then tokens with the new kid are refused; once it lands, tokens signed with the leaked key (unknown kid) are refused.

Ending the overlap

This is history: the overlap ended with heimdall-api 4.0.0.

3.x signed every token ES256 but, while HEIMDALL_AUTH_TOKEN_SECRET was set, still accepted the HS256 tokens Heimdall had issued before the upgrade, signed with that secret or _PREVIOUS, and logged each one as Verified a legacy HS256 {Reader} token. The secrets were to be removed no earlier than 24 hours after the 3.0.0 production deploy, and only once no such line had appeared for the longest token lifetime, with every consumer verifying through the key set.

4.0.0 no longer reads either variable: every reader refuses an HS256 token, whatever it is signed with, and the compose file no longer passes them. While either is still set in the API’s own environment, start-up logs {Variable} is set but no longer read; remove it at Warning, naming the variable and never its value. A container started from the 4.0 compose file never sees them, so it never logs that warning, and its silence says nothing about the store: under yggdrasil, check the store with ygg vars list <scope> --keys, which prints names only. Remove both from the environment. In yggdrasil, Fortuna’s FORTUNA_AUTH_TOKEN_SECRET is a ${ref:} to Heimdall’s value, so Fortuna’s references are removed first (fortuna-api 4.0.0 no longer reads it), then Heimdall’s value is unset.

Rolling back

Rolling 4.0.0 back to 3.x needs nothing: 3.x does not require the secret, and without it 3.x refuses HS256 tokens too. A rollback to 2.x is no longer possible once the secret has been removed: 2.x signs and verifies HS256 with it alone, its compose file requires it, and consumers at 4.0.0 or later no longer verify HS256.

Consumers

• Where to fetch the keys. Inside the platform, use the internal edge URL http://heimdall-api.<environment>:8080/.well-known/jwks.json. Anyone else uses https://<PUBLIC_HOST>/.well-known/jwks.json; the UI host does not serve it.
• Pin ES256. Accept the ES256 algorithm only, never the one a token’s alg header names, and resolve the key by kid.
• Refetch on an unknown kid, at most once a minute, and replace the cached set with the one fetched. The response says Cache-Control: public, max-age=300 and application/json; the body is a raw RFC 7517 key set, not the usual response envelope.
• Mind the budget. The key set shares the AuthAnonymous rate limit with the other anonymous credential endpoints: 10 requests a minute per address. A consumer that fetches only on an unknown kid stays far below it.
• Keep checking the issuer and the audience; the signature alone does not say a token was issued for you.

Environment variables at a glance

A present but unusable value of HEIMDALL_AUTH_TOKEN_EXPIRATION_IN_SECONDS, HEIMDALL_APPLICATION_TOKEN_EXPIRATION_IN_SECONDS or HEIMDALL_ADMIN_EMAIL_COOLDOWN_IN_SECONDS is logged as a warning at start-up, naming the variable, the rejected value and the default applied; an absent one is just the default and logs nothing.

Application tokens (HEIMDALL_APPLICATION_TOKEN_AUDIENCE, UC-56) are validated with zero clock skew, so instances whose clocks differ can refuse a freshly issued token for the length of the drift: keep instance clocks synchronised (NTP).

The account-email cooldown (HEIMDALL_ADMIN_EMAIL_COOLDOWN_IN_SECONDS) began as the administrators' (UC-51/UC-52) and keeps that name, but it now also sets the cooldown of a person’s own account-email budget (FR-PE-16) — resend, email setup code, email-change notice — and of anonymous password recovery (UC-12). A shorter value lets people retry sooner. 0 removes the cooldown: a person’s own sends are then bounded only by the budget’s five per window and the per-IP limit, and recovery only by the per-IP limit, so keep it on wherever the API faces the internet. For recovery the cooldown is held to the reset link’s lifetime (HEIMDALL_PASSWORD_RESET_TOKEN_EXPIRATION_IN_SECONDS), so a long cooldown can never leave the owner without a working link.

For the administrators’ sends and for recovery, the cooldown is derived from the issued token rows (ExpiresAt minus the token lifetime), not from a separate record. Two consequences: if HEIMDALL_RETENTION_TOKEN_GRACE_DAYS is 0 and a token lifetime is shorter than the cooldown, an expired token from inside the window may already have been purged, and the cooldown can undercount. And it is best-effort: simultaneous administrator requests for the same person can both pass it. Recovery requests are worked one at a time per instance, so only a second instance could overlap one. A person’s own budget is a conditional write on their row and has neither weakness.

Environments/.env in the Web API project is a tracked template listing every variable; real values live in per-environment files that are gitignored.

Scaling

The API is designed for horizontal scaling (NFR-06), and the authentication design is what makes that cheap: tokens are stateless and validated from their claims, so no session store or sticky routing is needed. The rate limiter’s window is per instance as a result, which is why the per-account failure budgets above live in the database instead.

A token still carries no revocation list, but it no longer outlives the identity it names: ActorLivenessFilter resolves the caller on every authenticated request and refuses a token whose person or Google User has been deleted (FR-AU-05, FR-GO-12) or is under a restriction of processing (NFR-24). That costs one indexed read per request — the price of making logical deletion and restriction take effect immediately rather than whenever the token happens to expire. The four endpoints through which a subject exercises their own rights — data export, erasure request, restriction and lifting their own restriction — still admit a restricted or suspended (not yet anonymised) subject, and leave the answer to the use case.

The full operational specification is the Operations & Infrastructure Document .