搭建简单的gitlab服务,在k8s环境

待填坑: 搭建runner的方案

背景

• 需要维护一个私有的git仓库, gitlab比较常用而且功能强大
• k8s环境比较容易维护

方案

• k8s环境使用已有的环境,相关搭建不在本文范围
• crowd使用已有的服务, 作为auth的服务, 这里可以按需配置
• gitlab使用单个docker启动(比较简单,目前可以满足需求)
  • GitLab Docker images
• gitlab的auth使用atlassian的crowd
• gitlab添加runner(由于k8s类型的runner还没调研,此处使用的是裸机的dind docker来做runner)
  • Run GitLab Runner in a container
  • 启用test coverage visualization
    • 
  • 启用test report
    • 
• gitlab启用pages(由于新版的gitlab支持pages的accesss-control,特别适合用来做内部代码文档管理)
  • GitLab Pages administration

搭建

{{}} 包含的是jinja的模板,用来传入参数,相关内容与本文无关` ### 预定义值 - 域名使用的是 `{{domain}}, 实际代表比如xxx.com

• gitlab的网页使用的是 git.xxx.com 或者写作 git.{{domain}}
• gitlab-pages使用的基础域名是 pages.xxx.com 或者写作 pages.{{domain}}
• {{ service_name }} 可以使用 gitlab

配置gitlab

ConfigMap

kind: ConfigMap
metadata:
  name: "{{service_name}}-{{k8s_deployment_stage}}-config"
  namespace: {{ envs.namespace }}
apiVersion: v1
data:
  # 使用国内的gem镜像
  gemrc: |
    ---
    :backtrace: false
    :bulk_threshold: 1000
    :sources:
    - https://gems.ruby-china.com/
    :update_sources: true
    :verbose: true
  
  # Question 1: 后续会挂载到gitlab的certs
  {{domain}}.pem: |
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----

• Question 1: 由于gitlab-pages开启access-control之后, 会向gitlab的发起请求,此时会请求git.{{domain}}, 如果没有加上相关的证书, 会出现异常 Post /oauth/token: x509: certificate signed by unknown authority
  • using-a-custom-certificate-authority-ca
    • install-custom-public-certificates
    • 官方是表示挂载到/etc/gitlab/trusted-certs, 不过在docker的gitlab中启动时, 会移动证书到/opt/gitlab/embedded/ssl/certs/, 所以我们直接挂载到/opt/gitlab/embedded/ssl/certs/{{domain}}.pem 就行了
  • 此处的证书可以使用{{domain}}的通配证书,也可以使用git.{{domain}}的单域名证书,可以根据自己的需要添加

Deployment

kind: Deployment
apiVersion: apps/v1
metadata:
  name: "{{ service_name }}-{{ k8s_deployment_stage }}"
spec:
  replicas: {{ replicas }}
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: "{{service_name}}"
      version: "{{ k8s_deployment_stage }}"
  template:
    metadata:
      labels:
        app: "{{service_name}}"
        version: "{{k8s_deployment_stage}}"
    spec:
      volumes:
      - name: config
        configMap:
          name: "{{ service_name }}-{{k8s_deployment_stage}}-config"
          items:
          - key: gemrc
            path: gemrc
          - key: {{domain}}.pem
            path: {{domain}}.pem
      - name: logs
        emptyDir: {}
      # Question 2: 需要额外挂载临时目录
      - emptyDir:
          medium: Memory
        name: tmp
      - name: pv-data
        persistentVolumeClaim: 
          claimName: "{{ service_name }}-pv-data"
      - name: pv-config
        persistentVolumeClaim: 
          claimName: "{{ service_name }}-pv-config"
      containers:
      - image: "{{ instance_image }}"
        imagePullPolicy: IfNotPresent
        name: main
        resources:
          requests:
            memory: "50Mi"
            cpu: "10m"
          limits:
            memory: "8Gi"
            cpu: "8"
        ports:
        - containerPort: 80
        - containerPort: 22
        # 由于使用外部的ssl, gitlab的容器不涉及ssl证书
        # - containerPort: 443
        volumeMounts:
        - name: config
          mountPath: /root/.gemrc
          subPath: gemrc
        - name: config
          mountPath: /opt/gitlab/embedded/ssl/certs/{{domain}}.pem
          subPath: {{domain}}.pem
        - name: pv-config
          mountPath: /etc/gitlab
        - name: pv-data
          mountPath: /var/opt/gitlab
        - name: logs
          mountPath: /var/log/gitlab
        - name: tmp
          mountPath: /tmp
        env:
        - name: GITLAB_OMNIBUS_CONFIG
          value: |
            # 只开启http, ssl使用k8s提供的ingress
            external_url 'https://git.{{domain}}'
            letsencrypt['enable'] = false
            nginx['redirect_http_to_https'] = false
            nginx['listen_port'] = 80
            nginx['listen_https'] = false

            # Question 4
            gitlab_rails['omniauth_enabled'] = true
            gitlab_rails['omniauth_enabled'] = true
            gitlab_rails['omniauth_allow_single_sign_on'] = true
            gitlab_rails['omniauth_block_auto_created_users'] = false

            gitlab_rails['omniauth_providers'] = [
                {
                  "name" => "crowd",
                  "args" => {
                    "crowd_server_url" => "https://crowd.{{domain}}/crowd",
                    "application_name" => "{{crowd_app_name}}",
                    "application_password" => "{{crowd_app_password}}"
                  }
                }
            ]

            pages_external_url 'https://pages.{{domain}}'
            gitlab_pages['enable'] = true
            gitlab_pages['inplace_chroot'] = true
            gitlab_pages['access_control'] = true
            # Question 3
            gitlab_pages['external_http'] = ['0.0.0.0:8010']
            pages_nginx['redirect_http_to_https'] = false
            pages_nginx['listen_https'] = false
            pages_nginx['enable'] = false

• Question 2:
  • 使用 hostPath 挂载的tmp出现权限问题
  • 使用 emptryDir 需要设置成 Memory 才会是tmpfs模式, 否则无法被gitlab使用
• Question 3:
  • 这里添加8010作为外部访问pages的端口, 可以按需使用端口, 由于时间关系没有尝试其他方案

PVC

---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: "{{ service_name }}-pv-data"
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: "{{ storage_class }}"
  resources:
    requests:
      storage: 200Gi

---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: "{{ service_name }}-pv-config"
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: "{{ storage_class }}"
  resources:
    requests:
      storage: 20Gi

Service

apiVersion: v1
kind: Service
metadata:
  name: "{{service_name}}"
  labels:
    name: "{{service_name}}"
spec:
  ports:
  - port: 80
    targetPort: 80
    name: http
  - port: 8010
    targetPort: 8010
    name: http-page
  - port: 22
    targetPort: 22
    name: ssh
    # Question 4
    nodePort: 32222
  type: NodePort
  selector:
    app: "{{service_name}}"

• Question 4:
  • 这里开放了ssh的端口, 需要在裸机上将22端口映射到对应的nodePort, 以供外部访问, 具体的转发方案可以使用iptables或者socat等一众工具

Ingress

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: "{{service_name}}"
  namespace: {{ envs.namespace }}
  annotations:
    kubernetes.io/ingress.class: traefik
spec:
  rules:
  - host: "git.{{domain}}"
    http:
      paths:
      - path: /
        backend:
          serviceName: "{{service_name}}"
          servicePort: "http"
  # Question 5
  - host: "*.pages.{{domain}}"
    http:
      paths:
      - path: /
        backend:
          serviceName: "{{service_name}}"
          servicePort: "http-page"

• Question 5:
  • pages在外部访问时, 是根据group来访问相应的二级域名, 比如group名字为 dev, 项目为 backend-api , 那么pages启用之后访问的地址是 https://dev.pages.{{domain}}/backend-api
  • 另外在访问pages并登陆的过程中, 也需要访问projects.pages.{{doamin}}, 因此需要准备相应的ssl证书

其他

• 过程中也遇到以下问题
  • dial tcp: lookup gitlab.example.com
  • https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4243
  • 解决方式:
    • https://docs.gitlab.com/ee/administration/pages/#dial-tcp-lookup-gitlabexamplecom-and-x509-certificate-signed-by-unknown-authority

搭建完成

• pages, test 等都需要gitlab-ci的支持, 这部分就不在本文中记录了, 编写相关的gitlab-ci之后, 就可以实现一个项目CI时的测试覆盖, 以及项目文档维护的需求