Why I Run Postgres Migrations on Container Startup, Not From CI

Open any 2026 guide on Kubernetes database migrations and you'll find the same advice in the same order: 'do not run migrations from your application startup.' The rationale is correct on the surface (race conditions, advisory-lock deadlocks, container failures cascading into unavailability) and wrong underneath, because the rationale assumes the only alternatives are an unguarded db.migrate() in your entrypoint script versus a fully decoupled CI/CD migration job. There is a third option, and on the agent backends I've shipped it has been the simplest deploy story by a wide margin. This is what container-startup migrations look like when you actually engineer them.

What people are actually trying to avoid

The 'never run migrations on startup' line collapses four distinct problems into one piece of advice. Each one is real. Each one has a fix. The fix is small and well-known to anyone who has read the Postgres advisory-locks docs. Here are the four:

• ▸Race conditions with N replicas. If five pods boot simultaneously and each runs alembic upgrade head, you get five concurrent migrations against the same schema. Postgres's locking will queue them, but the failure modes (deadlocks, partial migrations) are real.
• ▸Advisory-lock timeouts on failure. When a migration fails mid-flight, the connection holding the advisory lock can stay open for 10-15 minutes before the database server reaps it. During that window, every subsequent boot attempt sees a 'pending' lock and stalls.
• ▸PgBouncer transaction-pooling incompatibility. Advisory locks are session-scoped. PgBouncer in transaction pooling mode releases the underlying connection between transactions, which means the session-scoped lock disappears between calls. Migrations either fail to acquire or fail to release.
• ▸Hard failure surface. A migration that fails on container startup means the container fails to start. With a rolling deploy, you've potentially lost serving capacity. The deploy can deadlock if the new pods can't come up and the orchestrator won't kill old ones.

All four have direct, well-documented fixes. The reason most teams don't apply them is that adopting an init-container pattern or a CI migration job costs less reading than adopting the fixes does. That's a fair tradeoff for a team with a release engineer. For a small team where the deploy story has to fit in a founding engineer's head while they're also shipping the agent loop, the math is different.

The 30-line pattern that resolves all four

The pattern is a leader-elected advisory lock acquired against the application's main connection (not through PgBouncer's transaction pooler) at startup, with bounded timeout, explicit release on failure, and a separate connection string for migration use only. It looks like this:

┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐
│  pod 1   │  │  pod 2   │  │  pod 3   │  │  pod 4   │  │  pod 5   │
│  (boot)  │  │  (boot)  │  │  (boot)  │  │  (boot)  │  │  (boot)  │
└────┬─────┘  └────┬─────┘  └────┬─────┘  └────┬─────┘  └────┬─────┘
     │             │             │             │             │
     │   pg_try_advisory_lock(<MIGRATION_LOCK_KEY>)           │
     ▼             ▼             ▼             ▼             ▼
   wins         waits         waits         waits         waits
     │
     ▼
   alembic upgrade head  ◄──── direct connection, NOT PgBouncer
     │
     ▼
   pg_advisory_unlock(<MIGRATION_LOCK_KEY>)
     │
     ▼
   ──────────►  all pods proceed; new schema is the only schema
                they ever see

Five details make this work in production:

• ▸Use pg_try_advisory_lock, not pg_advisory_lock. The non-blocking variant returns a boolean. The four pods that don't win the lock should NOT block on the lock acquisition: they should poll the schema version in a separate cheap query (SELECT version_num FROM alembic_version) and proceed once it matches the version embedded in the deployed image. Blocking on the lock is what causes the cascade outage; polling the schema version doesn't.
• ▸Open a direct connection for the migration, bypassing PgBouncer. Add a second DATABASE_URL_MIGRATION env var that points straight at the Postgres host (or at PgBouncer in session pooling mode). The application's normal connection pool can stay on the transaction pooler. Migrations get their own session-aware path.
• ▸Wrap the lock acquisition + migration + release in a single try / finally with a hard timeout (I use 5 minutes). If the migration hangs, the connection is force-closed, the lock is reaped by Postgres on connection death, the next pod retries.
• ▸Make migrations idempotent. Every Alembic revision should be safe to re-run on a partially-applied state. This is more discipline than tooling: you write migrations as 'add column IF NOT EXISTS', 'create index IF NOT EXISTS', 'backfill rows that haven't been backfilled yet'. The result is that a half-completed migration can be re-applied cleanly.
• ▸Record both the schema version AND the deploy version in a small table the app reads on boot. The app refuses to start if the schema version is older than the version it expects. This is the safety belt for the case where a developer rolls back the app deploy without rolling back the schema (the schema only ever moves forward in this design).

I shipped 17 migrations on this pattern over a 14-week stretch with zero unplanned downtime. The total amount of bespoke code is about 80 lines of Python in a single migrate.py module that the FastAPI startup hook calls. The total amount of bespoke ops infrastructure is zero (no extra Kubernetes Job, no extra CI job, no init container).

When the CI/init-container pattern is actually right

I want to be specific about when not to do this, because the official advice exists for real reasons:

• ▸When you have a release engineer or a platform team. The init-container or CI-job pattern is a higher-quality default once someone owns the deploy story. The container-startup pattern depends on every developer remembering the lock semantics. That doesn't scale past one or two services.
• ▸When your migrations include long-running operations (CREATE INDEX CONCURRENTLY on a 100M-row table, large backfills) that cannot finish inside a 5-minute startup window. Long migrations belong in their own pipeline. Pretending they're short by stuffing them into a startup hook is how you discover deploy timeouts at 3 AM.
• ▸When you run multiple services that share a database. The advisory lock has to be coordinated across all of them. At that point you've reinvented a release engineer.
• ▸When your environment can't honor the deploy ordering. The container-startup pattern requires that the new schema is fully compatible with both the old and new app code (the standard zero-downtime expand/contract pattern). If your release tooling can't enforce that, do migrations separately.

What I'd change if I rebuilt it

• ▸Add a structured 'migration mode' env var. Today the same Python module decides whether to run migrations or skip based on a heuristic. An explicit MIGRATE_ON_BOOT=true|false|dry-run would make local-dev faster (skip the lock dance) and CI testing cleaner (run migrations against an empty test DB without contention).
• ▸Emit migration timing as a structured log line that gets ingested into the deploy dashboard. The hardest debugging case I hit was a slow migration making rolling deploys take longer than the orchestrator's healthcheck patience; visibility into per-migration runtime would have caught that earlier.
• ▸Treat a failed migration as a deploy-blocking event with explicit rollback semantics. Today, the 'rollback' path is `re-deploy the previous image, accept the schema is ahead'. That works because of the always-forwards rule, but it's worth writing down as part of the runbook so a future on-call doesn't reinvent the protocol under stress.

The bigger lesson

Operational simplicity is its own form of correctness for small teams. The 'right' answer to a coordination problem is often the answer that fits in one engineer's head, not the answer the platform team would build at scale. The container-startup migration pattern is operationally simpler than init containers + CI jobs + advisory-lock-aware tooling, but only because we did the engineering on the four coordination problems instead of routing around them. The simplicity is earned, not free.

If a hiring manager asks me how I think about deploy infra for a small team, this is the answer. Not because the pattern is exotic, but because the willingness to engineer past the default best-practice advice (when you can articulate why) is what 'founding engineer' means in a deploy context.

References

• ▸Postgres docs: pg_try_advisory_lock and session-vs-transaction-scoped advisory locks
• ▸PostgresAI: "Zero-downtime Postgres schema migrations need this: lock_timeout and retries"
• ▸IBM mcp-context-forge issue #4051: Alembic advisory lock hangs through PgBouncer transaction pooling
• ▸DEV: "37 Alembic Migrations, Zero Downtime: How We Moved a Live SaaS From Single-Tenant to Multi-Tenant"
• ▸Heroku Release Phase: official docs (devcenter.heroku.com)
• ▸Codefresh / Bytebase / Atlas: the prevailing CI-pipeline-first pattern (referenced as the contrast)