Failed update to gitlab 17.3

$ cat /srv/gitlab/logs/reconfigure/1723739323.log
# Logfile created on 2024-08-15 16:28:43 +0000 by logger.rb/v1.5.3
[2024-08-15T16:28:43+00:00] INFO: Started Cinc Zero at chefzero://localhost:1 with repository at /opt/gitlab/embedded (One version per cookbook)
[2024-08-15T16:28:43+00:00] INFO: *** Cinc Client 18.3.0 ***
[2024-08-15T16:28:43+00:00] INFO: Platform: x86_64-linux
[2024-08-15T16:28:43+00:00] INFO: Cinc-client pid: 33
[2024-08-15T16:28:43+00:00] INFO: Setting the run_list to ["recipe[gitlab-ee]"] from CLI options
[2024-08-15T16:28:43+00:00] INFO: Run List is [recipe[gitlab-ee]]
[2024-08-15T16:28:43+00:00] INFO: Run List expands to [gitlab-ee]
[2024-08-15T16:28:43+00:00] INFO: Starting Cinc Client Run for gitlab.archlinux.org
[2024-08-15T16:28:43+00:00] INFO: Running start handlers
[2024-08-15T16:28:43+00:00] INFO: Start handlers complete.
[2024-08-15T16:28:44+00:00] INFO: Loading cookbooks [gitlab-ee@0.0.1, package@0.1.0, gitlab@0.0.1, consul@0.1.0, patroni@0.1.0, pgbouncer@0.1.0, spamcheck@0.1.0, runit@5.1.7, logrotate@0.1.0, postgresql@0.1.0, redis@0.1.0, monitoring@0.1.0, registry@0.1.0, mattermost@0.1.0, gitaly@0.1.0, praefect@0.1.0, gitlab-kas@0.1.0, gitlab-pages@0.1.0, letsencrypt@0.1.0, nginx@0.1.0, acme@4.1.6, crond@0.1.0]
[2024-08-15T16:28:44+00:00] INFO: Generating default secrets
[2024-08-15T16:28:45+00:00] INFO: Generating /etc/gitlab/gitlab-secrets.json file
[2024-08-15T16:28:45+00:00] WARN: gitlab-rails does not have a log_group or default logdir mode defined. Setting to 0700.
[2024-08-15T16:28:45+00:00] INFO: Skipped selecting an init system because it was explicitly disabled
[2024-08-15T16:28:45+00:00] WARN: gitlab-shell does not have a log_group or default logdir mode defined. Setting to 0700.
[2024-08-15T16:28:45+00:00] WARN: gitlab-sshd does not have a log_group or default logdir mode defined. Setting to 0700.
[2024-08-15T16:28:45+00:00] WARN: logrotate does not have a log_group or default logdir mode defined. Setting to 0700.
[2024-08-15T16:28:45+00:00] WARN: logrotate does not have a log_group or default logdir mode defined. Setting to 0700.
[2024-08-15T16:28:45+00:00] WARN: puma does not have a log_group or default logdir mode defined. Setting to 0700.
[2024-08-15T16:28:45+00:00] WARN: gitlab-rails does not have a log_group or default logdir mode defined. Setting to 0700.
[2024-08-15T16:28:45+00:00] WARN: gitlab-shell does not have a log_group or default logdir mode defined. Setting to 0700.
[2024-08-15T16:28:45+00:00] WARN: gitlab-workhorse does not have a log_group or default logdir mode defined. Setting to 0700.
[2024-08-15T16:28:45+00:00] WARN: gitlab-pages does not have a log_group or default logdir mode defined. Setting to 0700.
[2024-08-15T16:28:45+00:00] WARN: gitlab-kas does not have a log_group or default logdir mode defined. Setting to 0700.
[2024-08-15T16:28:45+00:00] WARN: gitaly does not have a log_group or default logdir mode defined. Setting to 0700.
[2024-08-15T16:28:45+00:00] WARN: mailroom does not have a log_group or default logdir mode defined. Setting to 0700.
[2024-08-15T16:28:45+00:00] WARN: gitaly does not have a log_group or default logdir mode defined. Setting to 0700.
[2024-08-15T16:28:45+00:00] WARN: postgresql does not have a log_group or default logdir mode defined. Setting to 0700.
[2024-08-15T16:28:45+00:00] WARN: postgresql does not have a log_group or default logdir mode defined. Setting to 0700.
[2024-08-15T16:28:45+00:00] WARN: gitlab-kas does not have a log_group or default logdir mode defined. Setting to 0700.
[2024-08-15T16:28:45+00:00] WARN: crond does not have a log_group or default logdir mode defined. Setting to 0700.
[2024-08-15T16:28:45+00:00] WARN: puma does not have a log_group or default logdir mode defined. Setting to 0700.
[2024-08-15T16:28:45+00:00] WARN: sidekiq does not have a log_group or default logdir mode defined. Setting to 0700.
[2024-08-15T16:28:45+00:00] WARN: gitlab-workhorse does not have a log_group or default logdir mode defined. Setting to 0700.
[2024-08-15T16:28:45+00:00] WARN: mailroom does not have a log_group or default logdir mode defined. Setting to 0700.
[2024-08-15T16:28:45+00:00] WARN: gitlab-pages does not have a log_group or default logdir mode defined. Setting to 0700.
[2024-08-15T16:28:45+00:00] WARN: registry does not have a log_group or default logdir mode defined. Setting to 0700.
[2024-08-15T16:28:46+00:00] WARN: only_if block for file[/var/opt/gitlab/gitlab-rails/etc/gitlab-registry.key] returned a string, did you mean to run a command?
[2024-08-15T16:28:46+00:00] WARN: only_if block for templatesymlink[Create a gitlab_incoming_email_secret and create a symlink to Rails root] returned a string, did you mean to run a command?
[2024-08-15T16:28:46+00:00] WARN: only_if block for templatesymlink[Create a gitlab_pages_secret and create a symlink to Rails root] returned a string, did you mean to run a command?
[2024-08-15T16:28:46+00:00] WARN: only_if block for templatesymlink[Create a gitlab_kas_secret and create a symlink to Rails root] returned a string, did you mean to run a command?
[2024-08-15T16:28:46+00:00] INFO: link[/opt/gitlab/service/logrotate] created
[2024-08-15T16:28:46+00:00] WARN: redis does not have a log_group or default logdir mode defined. Setting to 0700.
[2024-08-15T16:28:46+00:00] INFO: link[/opt/gitlab/service/redis] created
[2024-08-15T16:28:46+00:00] INFO: link[/opt/gitlab/service/gitaly] created
[2024-08-15T16:28:47+00:00] INFO: link[/opt/gitlab/service/postgresql] created
[2024-08-15T16:28:47+00:00] INFO: file[/opt/gitlab/service/postgresql/supervise/status] owner changed to 996
[2024-08-15T16:28:47+00:00] INFO: file[/opt/gitlab/service/postgresql/supervise/status] group changed to 996
[2024-08-15T16:28:47+00:00] INFO: file[/opt/gitlab/service/postgresql/supervise/status] updated atime and mtime to 2024-08-15 16:28:47 +0000
[2024-08-15T16:28:47+00:00] INFO: file[/opt/gitlab/service/postgresql/log/supervise/status] owner changed to 996
[2024-08-15T16:28:47+00:00] INFO: file[/opt/gitlab/service/postgresql/log/supervise/status] group changed to 996
[2024-08-15T16:28:47+00:00] INFO: file[/opt/gitlab/service/postgresql/log/supervise/status] updated atime and mtime to 2024-08-15 16:28:47 +0000
[2024-08-15T16:28:47+00:00] INFO: link[/opt/gitlab/service/gitlab-kas] created
[2024-08-15T16:28:47+00:00] WARN: gitlab-rails does not have a log_group or default logdir mode defined. Setting to 0700.
[2024-08-15T16:29:03+00:00] INFO: Running queued delayed notifications before re-raising exception
[2024-08-15T16:29:03+00:00] ERROR: Running exception handlers
[2024-08-15T16:29:03+00:00] ERROR: Exception handlers complete
[2024-08-15T16:29:03+00:00] FATAL: Stacktrace dumped to /opt/gitlab/embedded/cookbooks/cache/cinc-stacktrace.out
[2024-08-15T16:29:03+00:00] FATAL: /srv/gitlab/logs/reconfigure/1723739323.log---------------------------------------------------------------------------------------
[2024-08-15T16:29:03+00:00] FATAL: PLEASE PROVIDE THE CONTENTS OF THE stacktrace.out FILE (above) IF YOU FILE A BUG REPORT
[2024-08-15T16:29:03+00:00] FATAL: ---------------------------------------------------------------------------------------
[2024-08-15T16:29:03+00:00] FATAL: Mixlib::ShellOut::ShellCommandFailed: rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash_hide_env[migrate gitlab-rails database] (gitlab::database_migrations line 20) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash"  ----
STDOUT: rake aborted!
StandardError: An error has occurred, this and all later migrations canceled:

PG::InternalError: ERROR:  no unpinned buffers available
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/batched_background_migration_helpers.rb:92:in `queue_batched_background_migration'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20240704030747_queue_backfill_new_audit_event_tables.rb:13:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:40:in `within'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:21:in `exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:123:in `run_block'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:134:in `block in run_block_with_lock_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:141:in `block in read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:228:in `retry_with_backoff'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:130:in `read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:129:in `run_block_with_lock_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:97:in `run'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:52:in `ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/runner_backoff/active_record_mixin.rb:21:in `execute_migration_in_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:28:in `block in with_advisory_lock_connection'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:25:in `with_advisory_lock_connection'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:138:in `configure_database'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:107:in `configure_pg_databases'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:94:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'

Caused by:
ActiveRecord::StatementInvalid: PG::InternalError: ERROR:  no unpinned buffers available
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/batched_background_migration_helpers.rb:92:in `queue_batched_background_migration'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20240704030747_queue_backfill_new_audit_event_tables.rb:13:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:40:in `within'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:21:in `exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:123:in `run_block'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:134:in `block in run_block_with_lock_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:141:in `block in read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:228:in `retry_with_backoff'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:130:in `read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:129:in `run_block_with_lock_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:97:in `run'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:52:in `ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/runner_backoff/active_record_mixin.rb:21:in `execute_migration_in_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:28:in `block in with_advisory_lock_connection'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:25:in `with_advisory_lock_connection'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:138:in `configure_database'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:107:in `configure_pg_databases'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:94:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'

Caused by:
PG::InternalError: ERROR:  no unpinned buffers available
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/batched_background_migration_helpers.rb:92:in `queue_batched_background_migration'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20240704030747_queue_backfill_new_audit_event_tables.rb:13:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:40:in `within'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:21:in `exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:123:in `run_block'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:134:in `block in run_block_with_lock_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:141:in `block in read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:228:in `retry_with_backoff'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:130:in `read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:129:in `run_block_with_lock_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:97:in `run'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:52:in `ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/runner_backoff/active_record_mixin.rb:21:in `execute_migration_in_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:28:in `block in with_advisory_lock_connection'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:25:in `with_advisory_lock_connection'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:138:in `configure_database'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:107:in `configure_pg_databases'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:94:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
Running db:migrate rake task
main: == [advisory_lock_connection] object_id: 53640, pg_backend_pid: 455
main: == 20240704030747 QueueBackfillNewAuditEventTables: migrating =================
main: == [advisory_lock_connection] object_id: 53640, pg_backend_pid: 455
STDERR: 
---- End output of "bash"  ----
Ran "bash"  returned 1

added scopebug label

assigned to @gromit

changed the description

changed the incident status to Resolved

Posted an upstream issue, let's see how this goes: https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/8682

Just to note it down somewhere, this is how one can access the gitlab db here:

$ docker exec -it -u gitlab-psql gitlab /opt/gitlab/embedded/bin/psql -h "/var/opt/gitlab/postgresql" -d gitlabhq_production

closed with commit 27f6c835

changed the incident status to Resolved by closing the incident

mentioned in commit 27f6c835