Skip to content

Recover on Lambda

Sign in with the recovery password on a Lambda installation when no directory can vouch for anybody, then turn it off.

Before you start

  • You need ssm:GetParameter on /sluis/<instance>/internal/oidc/recovery-password, and kms:Decrypt on ParameterKeyArn when set.

  • Lambda has no API server, so recovery is a password. The browser, cookie and 429 refusals of lost operator access apply here.

  • sluispulumi.NewLambda generates a 40-character password that an apply never rotates. It stores it as the SSM SecureString /sluis/<instance>/internal/oidc/recovery-password, and the service reads it at cold start and every five minutes. The RecoveryPasswordParameter output holds the name, never the value.

  • Without recovery.passwordSecret (secrets), recovery on Lambda logs an ERROR and builds nothing.

  • Recovery is on by default, and the setup checklist shows Turn off the recovery password while it is.

  • Every attempt is recorded as roster.recovery.signed_in (audit actions). With the audit trail unwritable, no session is issued.

  • API.KeepDefaultEndpoint: true skips the custom domain's mutual TLS. Use it for a cutover acceptance run only. Ten refused attempts make one instance answer 429 for a minute, so consider capping reserved concurrency.

Steps

  1. Read the password. It is 40 characters.
# Refuses a terminal unless --show; --profile names an AWS profile.
sluisctl recovery password --instance <instance> --region <region> | pbcopy

# Without sluisctl: layout v5 stores {"value":"<base64>"}; layout v4 keeps the
# text at /sluis/<instance>/internal/config/recovery/password, readable as is.
aws ssm get-parameter --with-decryption \
  --name /sluis/<instance>/internal/oidc/recovery-password \
  --query Parameter.Value --output text | jq -r .value | base64 --decode

  1. Open https://<issuer host>/console/login, expand Recovery sign-in and paste the password. You sign in as recovery, an operator by construction. Use this page even when console.client sends everybody else to the issuer's page.

  2. Fix the directory group, then turn recovery off. Sign in through your identity provider and press Disable recovery in Settings (turn off recovery), or change the configuration. Read the Pulumi preview before the apply.

off := false
sluispulumi.NewLambda(ctx, "acme", &sluispulumi.LambdaArgs{
 // ...
 Recovery: &sluispulumi.RecoveryArgs{Enabled: &off},
})

Or write recovery: {enabled: false} in the configuration; the two must agree. The parameter stays.

  1. Optionally rotate the password: run pulumi up --replace on the RandomPassword (<name>-recovery-password) after reading the preview.

Verify

The trail has roster.recovery.signed_in with outcome success (read the audit trail). After step 3 the password is refused and recorded as denied.

Roll back

Sign out. To turn recovery back on, set Enabled true: the password is unchanged. After a turn-off from the console, also set a new recovery.generation in Config. Rotate the password if anyone pasted it elsewhere.

Decided in

Recovery.