REF · RANDE-COREACCESS · UNLOGGED
← write-ups

r3plic4.esc

A managed-database admin account is a cage with a friendly name. This is the story of the one guard between that cage and the machine — and the single session flag that makes it look the other way.

impact

A deliberately de-privileged managed-database admin escalates to full PostgreSQL superuser and reads and writes the host filesystem (operating-system command execution reachable) — a complete escape from the managed sandbox, using only features the platform ships as sanctioned functionality.

Every managed database sells the same quiet promise, and nobody ever prints it on the invoice: real power on top, and a floor you can never reach through. You get an account that calls itself admin. You create schemas, load extensions, run whatever queries you like. And somewhere beneath all of it runs a line — the one where your database ends and their host begins — that you are simply never meant to cross.

That line is a trust boundary, and on a mature platform it is enforced in depth: in a dozen independent code paths, by the engine itself, each one a separate lock on the same door.

In one place, it is enforced by a single trigger. And the platform hands every tenant the switch that turns triggers off.

This is the anatomy of how a sealed box stops being sealed — assembled entirely from features the product advertises as conveniences, with no memory corruption, no parser trick, and no primitive more exotic than writing one row.

I didn’t come for the database. I came for the contradiction.

The findings worth having are rarely where the scanner points. They live in the gap between what a system claims about itself and what it quietly is. So on a new managed-database target I test nothing at first. I sit with the account I’ve been handed, adopt the platform’s own threat model — the tenant is a hostile principal; the host and the control plane are the assets — and ask the least interesting question I can think of, which usually turns out to be the most important one:

Why is this admin not a superuser?

Because it isn’t, and the removal is deliberate and thorough. The platform provisions the tenant as a role created without superuser, while the real superuser and the provider’s internal roles stay reserved and unreachable. That single decision is then defended across the whole surface. The stored password verifiers are legible only to a superuser. The server-side file-read routines sit behind privileges the tenant was never granted. The classic copy-to-a-program execution path is closed. Reach for anything that touches the host and the engine refuses, cleanly, in every path you try.

[[diagram:pg-sandbox-wall]]

That wall is not incidental. Someone took superuser away from this role on purpose and then spent real engineering making the removal hold across dozens of unrelated code paths. It is the most valuable sentence you can find on a target, because it inverts the entire exercise. The question stops being what can the admin do — the vendor answered that, deliberately and in detail — and becomes a question about composition:

They stripped this role of superuser and defended the decision everywhere it’s enforced directly. So which superuser powers did they have to hand back — indirectly, through some feature — to make the product worth paying for, and does any single feature re-open a path the direct controls all close?

A sealed box that does nothing is trivial to build and impossible to sell. A sealed box that lets you install a time-series engine, replicate data, schedule maintenance, and tune the server — that is the product. Each of those conveniences is a hole cut through the floor and covered with a grate, on the standing promise that the grate holds. The work is finding a grate that doesn’t quite fit its hole.

Two conveniences, and the seam between them

The extension surface drew my attention first. A role that may not hold superuser but may load native-code modules into the backend is already a contradiction — and extensions are precisely where superuser capability gets re-packaged into “safe,” scoped functions. If power was handed back anywhere, it was handed back here.

Two extensions mattered.

The first was a time-series engine — timescaledb — because it schedules and runs background jobs, and anything that runs code later, on your behalf has to resolve a security-critical question: under whose authority does that deferred code execute? The engine’s answer is honest and, evaluated on its own, reasonable. Every job is a row in a catalog table, bgw_job, and one column of that row names the owner role. When a job fires, the scheduler launches a background worker that adopts the identity named in that column — it becomes that role — and only then invokes the job’s check function and its procedure. Your retention policy runs as whoever configured it; this is the mechanism that makes that true.

Now read it as an authorization statement, because that is what it is: the identity a job’s code runs as is a value in a table row. Deferred execution had turned an identity decision into ordinary data.

The second extension was the platform’s own management helper — call it mgmt_extras — a collection of convenience wrappers whose entire reason to exist is to let a non-superuser perform actions a non-superuser normally cannot. Among them, one wrapper that changes session_replication_role. In stock Postgres that setting is superuser-only, and the restriction is not cosmetic. The replica value is the one a logical-replication apply worker runs under, and it exists to suspend the integrity machinery — triggers, and the constraint checks implemented as triggers — on the theory that the primary already enforced every invariant once, so the replica must not enforce them a second time. It is, in plain terms, the switch that disarms the database’s own safety interlocks. Postgres reserves it for superusers for exactly that reason.

Set those two facts side by side. One says a job runs under whatever role a row names. The other says I can disarm the machinery the database uses to enforce its own rules. Reviewed in isolation, each is documented, sanctioned, and dull — and that is the trap. Every feature review answers is this feature safe? under a single-feature threat model. The vulnerability doesn’t live inside either feature. It lives in the seam between them, in the composed model that no single review owns.

I didn’t have a bug yet. I had a shape. The shape of one.

The one guard

The engine’s authors were not careless about this. You cannot simply write a row into the job catalog naming the superuser as its owner. A trigger — validate_job_role — sits on the catalog and checks, on every write, that the owner you are declaring is a role you are actually a member of. It exists for one purpose: to hold the invariant a principal may only schedule code to run as a role it is entitled to assume.

So I tried the honest path first — the refusal that tells you the boundary is real and you’ve found it — and named the superuser as owner directly. The trigger did exactly its job: owner is not a member of the current role. Denied.

Good. The invariant was intact, and now I had its precise location. And its location was the whole finding. That same invariant — you may only act as a role you belong to — is one the engine already enforces natively at the authorization layer, the instant a session tries to assume another role. TimescaleDB could not lean on that native check here, because the job’s identity is adopted later, by a background worker, long after the row is written. So it re-implemented the invariant where it could: as a trigger on a table. A control-plane authorization rule, relocated into the data plane.

That relocation is the vulnerability, stated structurally. Because a data-plane trigger is not a law of the engine — it is a row that fires under the executor’s rules, and one of those rules is a switch the tenant was independently handed.

[[diagram:replica-blind]]

Here is that rule, exactly. Every trigger carries a firing disposition. The default disposition fires in origin sessions — normal operation — and is skipped when the session flips to replica; only a trigger explicitly armed to always fire survives the flip. This is the executor faithfully implementing replication semantics: during apply, default-fire triggers must not re-run. The owner check carries the default disposition. It was written as a business rule, not a security control, and it has no idea it was ever load-bearing.

Everything reduced to one property of the engine, true by design and lethal in composition: a trigger only guards you while it is firing.

Asking the guard to look the other way

Before the working idea, I spent the obvious ones — the part write-ups usually omit. A direct call to the file-read routines: denied on privilege. A shell through the copy-to-a-program path: closed. Every front door was locked exactly as designed, and that was the point. The locks were never the problem. I didn’t need a door; I needed the one guard watching them to look away for the length of a single write.

So I moved session_replication_role to replica through the wrapper the platform provided. That wrapper runs with its definer’s rights — a superuser’s — which is the entire reason a non-superuser can throw the switch at all. The interlock key was left hanging on the door, inside a convenience extension. With replica mode engaged, the owner check fell silent, its disposition telling the executor not to fire. I wrote the exact row it had refused a moment earlier, this time owning it as the superuser, and the write completed unchallenged. Then I returned the session to origin, so every other trigger in the database resumed firing and nothing downstream would ever register that a control had blinked.

A planted row is inert until something reads it. The scheduler only rebuilds its model of the catalog on a restore cycle — so I handed it one. The pair of routines intended for wrapping a dump-and-restore — the pre-restore and post-restore entry points — force that reload: the scheduler re-reads the job catalog, finds my job, and — precisely as designed — runs its check function under the identity in the owner column. That column named the superuser. My stored procedure ran with full superuser authority.

[[diagram:owner-spoof-chain]]

Note what class of bug this is, because it decides how seriously to take it. There is no corrupted pointer, no race that has to be won on the first try, no environment-specific offset. It is a deterministic authorization-composition failure: a logic bug that reproduces on demand, portably, from a standing start of valid credentials for my own service plus two extensions the platform lists as features. No second vulnerability. No user interaction. Those are the bugs that survive patches to their neighbors.

The proof

[[diagram:replica-poc-terminal]]

root:x:0:0. A role that could not read a single byte off that filesystem two minutes earlier returned the host’s password file, out of a table my payload created and granted back to me. The boundary was gone.

What that superuser actually holds

Reading one file is the demonstration, not the impact. The context I had landed in was the full database superuser, and from there the sandbox does not weaken — it stops existing. Each rung below crosses a boundary the direct controls were built to hold:

  • Every credential on the cluster. The role catalog now yields the stored password verifier for every role, the platform’s internal roles included — material to crack or replay offline, with no rate limit and no clock.
  • The host filesystem, for reading. The server-side file-read routines now answer for arbitrary paths: service configuration, and the TLS private key the service presents to its own clients — key material whose disclosure outlives any single session.
  • The host filesystem, for writing. Large-object export writes arbitrary bytes to disk as the database’s operating-system user, including a shared object placed where the backend can load it.
  • Native code execution. A superuser can register a C-language function backed by a shared object on disk. The copy-to-a-program door was bolted; this one stands open. Combine the write primitive above with a bound C function and you are running native code inside the backend process, on the host, wholly outside anything the managed product was designed to permit.

I stopped at the file read, and the scoping is deliberate. Superuser, host-file read and write, and the native-code path are each confirmed present; standing up a live shell from them is engineering, not research, and it trades nothing for real blast-radius risk against production. I did not probe cross-tenant reach or attempt to escape the container to the underlying node — those are separate boundaries and separate claims, and I make only the one the evidence supports: a non-superuser tenant becomes the database superuser and commands the host filesystem, with native code execution reachable from there.

Why it worked

Remove the extension names and the finding compresses to a single sentence: a privilege boundary was enforced by an ordinary trigger, and the same low-trust principal was independently granted the ability to disable ordinary triggers.

Neither grant is wrong in isolation. Running a job as its owner is reasonable. Exposing replication behavior to an operator has legitimate uses. The failure is architectural, and it has a precise name. A security-critical invariant — a principal may only cause code to run as a role it belongs to — was enforced in the data plane, by a trigger, rather than in the control plane, by the engine’s own authorization check. Data-plane enforcement inherits the data plane’s escape hatches, and this one had a spectacular hatch: a tenant-reachable switch that suspends trigger firing wholesale. Enforcement and the control that governs enforcement were reachable by the same adversary. That is a confused deputy, wearing the schema instead of a URL: the tenant can’t spoof an owner, but it can convince the engine to stop checking.

The remediation is layered, and each layer independently closes the hole — which is exactly what defense in depth is supposed to look like. Enforce owner-membership where the scheduler actually adopts the identity, in the engine’s job-launch path, where tenant SQL cannot reach the lever. Failing that, arm the owner-check trigger to fire even under replica mode, so replication semantics cannot silence a security control. And most fundamentally, a non-superuser role should never have been able to move session_replication_role; that wrapper is the interlock key, and it belongs behind the same wall as everything it can undo.

This is the recurring anatomy of a managed-service escape, and it is a design lesson before it is a bug. The provider builds a careful wall, then hands the tenant a tool whose documented purpose is harmless but whose mechanism quietly includes “…and it also weakens the wall.” Composition is where isolated correctness goes to die. The service betrays itself with its own conveniences.

What I carried out with me

The lesson worth keeping isn’t “time-series jobs” or “replica mode.” It’s the question I walked in with, sharpened into an auditor’s heuristic:

For every rule that keeps me in my box — what enforces it, in which plane, and can I, from inside the box, reach the mechanism that does the enforcing?

A guard is only a guard while it’s watching. Find the switch that makes it blink, and the wall was never a wall — only a habit the engine could be talked out of.

references