基于K8S的GitLab自动定时备份

发表于 | 更新于: | 分类于 ， | 阅读次数:
字数统计: 734

说明

  • 参考官方文档 Back up and restore GitLab
  • 使用Gitlab-CE镜像部署到K8S，Gitlab版本12.1.3-ce.0
  • 传统部署是使用Omnibus方式安装Gitlab，一般通过crontab+shell脚本的方式定期备份Gitlab
  • 在K8S环境下，没法这么做，于是想到用CronJob调用kubectl执行容器内部命令的方式备份Gitlab

准备RBAC

创建Role

1
2
3
4
5
6
7
8
9
10
11
12
kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  namespace: gitlab
  name: gitlab-backup
rules:
- apiGroups: [""]
  resources: ["pods", "pods/log"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["pods/exec"]
  verbs: ["create"]

创建ServiceAccount

1
2
3
4
5
6
7
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    app: gitlab-backup
  name: gitlab-backup
  namespace: gitlab

创建RoleBinding

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  labels:
    app: gitlab-backup
  name: gitlab-backup
  namespace: gitlab
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: gitlab-backup
subjects:
- kind: ServiceAccount
  name: gitlab-backup
  namespace: gitlab

创建ConfigMap

这里用来设置时区Asia/Shanghai

1
2
3
4
5
6
7
apiVersion: v1
kind: ConfigMap
metadata:
  name: timezone
  namespace: gitlab
binaryData:
  Shanghai: VFppZjIAAAAAAAAAAAAAAAAAAAAAAAACAAAAAgAAAAAAAAAcAAAAAgAAAAigl6KAoXkE8MhZXoDJCflwydO9AMsFivDLfEAA0js+8NOLe4DUQq3w1UUiANZMv/DXPL8A2AZmcNkd8oDZQXzwHrpSIB9pm5AgfoSgIUl9kCJnoSAjKV+QJEeDICUSfBAmJ2UgJvJeECgHRyAo0kAQAAEAAQABAAEAAQABAAEAAQABAAEAAQABAAEAAQAAfpABAAAAcIAABENEVABDU1QAAAAAAFRaaWYyAAAAAAAAAAAAAAAAAAAAAAAAAwAAAAMAAAAAAAAAHQAAAAMAAAAM/////342Qyn/////oJeigP////+heQTw/////8hZXoD/////yQn5cP/////J070A/////8sFivD/////y3xAAP/////SOz7w/////9OLe4D/////1EKt8P/////VRSIA/////9ZMv/D/////1zy/AP/////YBmZw/////9kd8oD/////2UF88AAAAAAeulIgAAAAAB9pm5AAAAAAIH6EoAAAAAAhSX2QAAAAACJnoSAAAAAAIylfkAAAAAAkR4MgAAAAACUSfBAAAAAAJidlIAAAAAAm8l4QAAAAACgHRyAAAAAAKNJAEAIBAgECAQIBAgECAQIBAgECAQIBAgECAQIBAgECAABx1wAAAAB+kAEEAABwgAAITE1UAENEVABDU1QAAAAAAAAACkNTVC04Cg==

创建CronJob

  • kubectl没有官方镜像，需要自己构建
  • 每天1点0分启动任务
  • 根据标签app=gitlab过滤出需要的Pod名字
  • 调用kubectl执行命令
  • 备份默认存放在/var/opt/gitlab/backups，因此最好将此目录挂载到外部存储，避免备份丢失！
  • Gitlab备份命令不包含/etc/gitlab/目录，因此需要手工备份，这里就不展示了
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: gitlab-backup-schedule
  namespace: gitlab
spec:
  schedule: "0 1 * * *"
  failedJobsHistoryLimit: 1
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 3
  suspend: false
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccount: gitlab-backup
          serviceAccountName: gitlab-backup
          dnsConfig:
            options:
            - name: single-request-reopen
          containers:
          - name: gitlab-backup
            image: kubectl:v1.14.8
            command:
            - /bin/sh
            - -c
            - |-
              #!/bin/bash
              # For More Information, https://docs.gitlab.com/ce/raketasks/backup_restore.html#back-up-gitlab
              # For GitLab 12.1 and earlier, use `gitlab-rake gitlab:backup:create`
              # For later, use `gitlab-backup create`
              echo "======Start Backup GitLab======"
              POD_NAME=$(kubectl -n gitlab get pods -l app=gitlab -o jsonpath='{.items[*].metadata.name}')
              kubectl -n gitlab exec ${POD_NAME} -- gitlab-rake gitlab:backup:create
              echo "======End Backup GitLab======"
              echo "======Clean Expired Backup======"
              # find files which modified before 15 days ago and delete its
              kubectl -n gitlab exec ${POD_NAME} -- find /var/opt/gitlab/backups -name *tar -mtime +15 -exec ls {} \;
              kubectl -n gitlab exec ${POD_NAME} -- find /var/opt/gitlab/backups -name *tar -mtime +15 -exec rm -f {} \;
            volumeMounts:
            - name: timezone
              mountPath: /etc/localtime
              subPath: Shanghai
          volumes:
          - name: timezone
            configMap:
              name: timezone
          restartPolicy: OnFailure
          dnsPolicy: ClusterFirst
          securityContext: {}
          terminationGracePeriodSeconds: 30

验证备份

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
======Start Backup GitLab======
2020-08-11 13:24:30 +0000 -- Dumping database ...
Dumping PostgreSQL database gitlabhq_production ... [DONE]
2020-08-11 13:24:40 +0000 -- done
2020-08-11 13:24:40 +0000 -- Dumping repositories ...
* xxxxxx/yyyyyy (@hashed/45/23/4523540f1504cd17100c4835e85b7eefd49911580f8efff0599a8f283be6b9e3) ... [DONE]
[SKIPPED] Wiki
* xxxxxx/yyyyyy (@hashed/45/23/4523540f1504cd17100c4835e85b7eefd49911580f8efff0599a8f283be6b9e3) ... [DONE]
[SKIPPED] Wiki
 * xxxxxx/yyyyyy (@hashed/45/23/4523540f1504cd17100c4835e85b7eefd49911580f8efff0599a8f283be6b9e3) ... [DONE]
2020-08-11 13:30:05 +0000 -- done
2020-08-11 13:30:05 +0000 -- Dumping uploads ...
2020-08-11 13:30:06 +0000 -- done
2020-08-11 13:30:06 +0000 -- Dumping builds ...
2020-08-11 13:30:06 +0000 -- done
2020-08-11 13:30:06 +0000 -- Dumping artifacts ...
2020-08-11 13:30:06 +0000 -- done
2020-08-11 13:30:06 +0000 -- Dumping pages ...
2020-08-11 13:30:06 +0000 -- done
2020-08-11 13:30:06 +0000 -- Dumping lfs objects ...
2020-08-11 13:30:06 +0000 -- done
2020-08-11 13:30:06 +0000 -- Dumping container registry images ...
2020-08-11 13:30:06 +0000 -- [DISABLED]
Creating backup archive: 1597152606_2020_08_11_12.1.3_gitlab_backup.tar ... done
Uploading backup archive to remote storage  ... skipped
Deleting tmp directories ... done
done
done
done
done
done
done
done
Deleting old backups ... skipping
Warning: Your gitlab.rb and gitlab-secrets.json files contain sensitive data
and are not included in this backup. You will need these files to restore a backup.
Please back them up manually.
Backup task is done.
======End Backup GitLab======
======Clean Expired Backup======