Migrating Gitlab Omnibus instance to new server
Since almost the beginning, we are using Gitlab as our main git repository server. After running on source for several years, we've migrated to the Omnibus installation. Since then, we've experienced almost no issues when upgrading between instances and we are using more and more of the provided services: Mattermost, CI, Registry.
Recently, we've finished a migration to a new server (simple Hetzner Cloud VPN, in case you are interested). For that case, I've needed to migrate all data, including the Mattermost Chat history to the new server efficiently.
1. Setup and config adjustments
- Install Gitlab omnibus on the new server, e.g.
apt install gitlab-ce
- Create a ssh-key identity on the new server
ssh-keygenand append the
authorized_keysof the old server's privileged user.
- Upgrade Gitlab to the latest version on your old instance. Migration is only possible between same version
- Copy the contents of
/etc/gitlabto the new server, including the
gitlab.rb. Now would be a good time for changing defaults inside.
- Use the built-in command
gitlab-ctl diff-configto see, what settings changed between versions. I would recommend to copy over all default values and comments, to future diffs are much more clean.
- E.g. we switched back from S3 container-registry to local registry to same a good amount of money1
- We've also moved from a proxied nginx with external Letsencrypt to Gitlab provisioned Letsenrypt.
- Use the built-in command
# the old host, must have direct ssh access # generate ssh-keygen on new host and copy over into authorized_keys firstname.lastname@example.org' echo "$gitlab - syncing content of /etc/gitlab" mkdir -p /etc/gitlab/ rsync -a $gitlab:/etc/gitlab/. /etc/gitlab/. gitlab-ctl reconfigure
2. Doing the migration
Best thing is, prepare everything as a script which you just run or copy & paste the individual commands. Here is the script I've followed on the migration. We take the thing appart and I link the full script as a Gist.
1. Seeting the login of the old server for convenience into the variable
#!/bin/bash # ATTENTION: COPY&PASTE RUN EVERY LINE BY HAND AND VERIFY THE OUTPUT # the old host, must have direct ssh access # generate ssh-keygen on new host and copy over into authorized_keys email@example.com'
Preferred: Copy over ssh server identity, otherwise you will receive tons of warnings when using the ssh interface of your future Gitlab server.
# copy ssh hostkey, so that the ssh connection from clients still work cd /etc/ssh scp $gitlab:/etc/ssh/ssh_host_* . service ssh restart
2. Stopping all processes that access the database on the target host
gitlab-ctl stop without arguments. We need the postgresql daemon.
# Stopping everything on remote system echo "$gitlab - Gitlab Stop" ssh $gitlab "gitlab-ctl stop nginx" ssh $gitlab "gitlab-ctl stop sidekiq" ssh $gitlab "gitlab-ctl stop mattermost"
3. Create a on demand backup
echo "$gitlab - Make Backup" ssh $gitlab "gitlab-backup create SKIP=lfs"
4. Copy over backup
# copy over last backup backup_file=`ssh $gitlab "cd /var/opt/gitlab/backups && ls *.tar | tail -n 1"` echo "Sync - /var/opt/gitlab/backups/" mkdir -p /var/opt/gitlab/backups/ scp -p $gitlab:/var/opt/gitlab/backups/$backup_file /var/opt/gitlab/backups/$backup_file chown git:git /var/opt/gitlab/backups/*.tar
5. Copy over registry
The registry is not included in the Gitlab backup, copy it over, if you are using local hosted (skip this step if you are using S3 for container images):
mkdir -p /var/opt/gitlab/registry/docker rsync $ssh:/var/opt/gitlab/registry/docker/. /var/opt/gitlab/registry/docker/. chown registry -R /var/opt/gitlab/registry/docker
5. Importing backup
echo "local - Stopping gitlab" gitlab-ctl stop unicorn gitlab-ctl stop sidekiq gitlab-backup restore
Syncing the Mattermost data dir (including all attachments and Mattermost config and plugins).
# Mattermost data dir echo "Sync - /var/opt/gitlab/mattermost" mkdir -p /var/opt/gitlab/mattermost/ rsync -a $gitlab:/var/opt/gitlab/mattermost/. /var/opt/gitlab/mattermost/. sudo chown -R mattermost /var/opt/gitlab/mattermost/
Creating a live Mattermost backup on the remove server.
# Mattermost sql echo "$gitlab - Creating mattermost backup" ssh $gitlab 'sudo -u gitlab-psql -- /opt/gitlab/embedded/bin/pg_dump -h /var/opt/gitlab/postgresql mattermost_production | gzip ' > /root/mattermost-db.sql.gz
Importing the db:
echo "local - reimport mattermost db" sudo -u mattermost /opt/gitlab/embedded/bin/dropdb -U gitlab_mattermost -h /var/opt/gitlab/postgresql -p 5432 mattermost_production sudo -u gitlab-psql /opt/gitlab/embedded/bin/createdb -U gitlab-psql -h /var/opt/gitlab/postgresql -p 5432 mattermost_production gunzip < /root/mattermost-db.sql.gz | sudo -u mattermost /opt/gitlab/embedded/bin/psql -U gitlab_mattermost -h /var/opt/gitlab/postgresql -p 5432 mattermost_production
7. Finish Backup and restart
gitlab-ctl reconfigure gitlab-ctl restart gitlab-rake gitlab:check SANITIZE=true
Don't forget the Backup cronjob if you've used any, keep in mind that the default backup does not include Mattermost history nor the container registry.
0 2 * * 1-6 /opt/gitlab/bin/gitlab-rake gitlab:backup:create CRON=1 SKIP=lfs
Bonus: Problem when upgrading the latest Gitlab on old server broke because of ulimits rule
Before migrating, we've upgraded the Gitlab to the newest version. That broke the Gitlay daemon, which did not start again.
$ gitlab-ctl tail gitaly ./run: 11: ulimit: error setting limit (Operation not permitted)
Problem was, that Gitaly start tries to change ulimit rules and fails if that is not possible, which was, in our case, because we used a Proxmox/LXC container. See this Gitlab issue.
gitaly['open_files_ulimit'] = false in
/etc/gitlab/gitlab.rb and reconfigure.
our CI infrastructure is on demand and pulled the images almost every time. I've tried adjusting the S3 cache, but that only worked for the docker base images, like ruby, Postgresql, but not our custom ones. Adding this saves us about 50 EUR per months. With the awesome rclone command, I've downloaded the images back from s3 into the local
rclone sync s3-registry:pludoni-gitlab-docker/docker/ /var/opt/gitlab/registry/docker/.) ↩