Kubernetes용 GitLab 에이전트를 설치하고 설정한 후 에이전트를 사용하여 클러스터에 GitLab Runner를 설치할 수 있습니다.

이 GitOps 워크플로우를 사용하면 리포지터리에 GitLab Runner 설정 파일이 포함되어 있으며 클러스터가 자동으로 업데이트됩니다.

1. GitLab Runner에 대한 Helm 차트 값을 검토하세요.

2. runner-chart-values.yaml 파일을 생성하세요. 예:

   # 러너를 등록할 GitLab 서버 URL(프로토콜 포함)
   # ref: https://docs.gitlab.com/runner/commands/#gitlab-runner-register
   #
   gitlabUrl: https://gitlab.my.domain.example.com/
   
   # GitLab 서버에 새 러너를 추가하기 위한 등록 토큰
   # GitLab 인스턴스에서 이 값을 가져오세요
   # 자세한 내용: https://docs.gitlab.com/ci/runners/
   #
   runnerRegistrationToken: "yrnZW46BrtBFqM7xDzE7dddd"
   
   # RBAC 지원을 위한 설정:
   rbac:
       create: true
   
   # 모든 컨테이너에 privileged 플래그를 활성화하여 실행합니다
   # Docker 명령을 실행해야 할 경우 docker:dind 이미지를 실행할 수 있는 플래그입니다
   # 활성화하기 전에 문서를 읽으세요:
   # https://docs.gitlab.com/runner/executors/kubernetes/#using-dockerdind
   runners:
       privileged: true
   

3. 클러스터 에이전트로 GitLab Runner 차트를 설치하기 위한 단일 매니페스트 파일을 생성하세요:

   helm template --namespace GITLAB-NAMESPACE gitlab-runner -f runner-chart-values.yaml gitlab/gitlab-runner > runner-manifest.yaml
   

   GITLAB-NAMESPACE를 네임스페이스로 교체하세요. 예시를 확인하세요.

4. runner-manifest.yaml 파일을 편집하여 ServiceAccount의 namespace를 포함하세요. helm template의 출력에는 생성된 리소스에 ServiceAccount 네임스페이스가 포함되지 않습니다.

   ---
   # Source: gitlab-runner/templates/service-account.yaml
   apiVersion: v1
   kind: ServiceAccount
   metadata:
     annotations:
     name: gitlab-runner-gitlab-runner
     namespace: gitlab
     labels:
   ...
   

5. Kubernetes 매니페스트를 보관하는 리포지터리에 runner-manifest.yaml을 푸시하세요.

6. GitOps를 사용하여 에이전트가 러너 매니페스트를 동기화하도록 설정하세요. 예:

   gitops:
     manifest_projects:
     - id: path/to/manifest/project
       paths:
       - glob: 'path/to/runner-manifest.yaml'
   

이제 에이전트가 매니페스트 업데이트를 위해 리포지터리를 확인할 때마다 클러스터가 GitLab Runner를 포함하도록 업데이트됩니다.

러너 매니페스트 예시#

이 예시는 샘플 러너 매니페스트 파일을 보여줍니다. 프로젝트의 요구 사항에 맞게 직접 manifest.yaml 파일을 생성하세요.

---
# Source: gitlab-runner/templates/service-account.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
  name: gitlab-runner-gitlab-runner
  labels:
    app: gitlab-runner-gitlab-runner
    chart: gitlab-runner-0.58.2
    release: "gitlab-runner"
    heritage: "Helm"
---
# Source: gitlab-runner/templates/secrets.yaml
apiVersion: v1
kind: Secret
metadata:
  name: "gitlab-runner-gitlab-runner"
  labels:
    app: gitlab-runner-gitlab-runner
    chart: gitlab-runner-0.58.2
    release: "gitlab-runner"
    heritage: "Helm"
type: Opaque
data:
  runner-registration-token: "FAKE-TOKEN"
  runner-token: ""
---
# Source: gitlab-runner/templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: gitlab-runner-gitlab-runner
  labels:
    app: gitlab-runner-gitlab-runner
    chart: gitlab-runner-0.58.2
    release: "gitlab-runner"
    heritage: "Helm"
data:
  entrypoint: |
    #!/bin/bash
    set -e
    mkdir -p /home/gitlab-runner/.gitlab-runner/
    cp /scripts/config.toml /home/gitlab-runner/.gitlab-runner/

    # Register the runner
    if [[ -f /secrets/accesskey && -f /secrets/secretkey ]]; then
      export CACHE_S3_ACCESS_KEY=$(cat /secrets/accesskey)
      export CACHE_S3_SECRET_KEY=$(cat /secrets/secretkey)
    fi

    if [[ -f /secrets/gcs-application-credentials-file ]]; then
      export GOOGLE_APPLICATION_CREDENTIALS="/secrets/gcs-application-credentials-file"
    elif [[ -f /secrets/gcs-application-credentials-file ]]; then
      export GOOGLE_APPLICATION_CREDENTIALS="/secrets/gcs-application-credentials-file"
    else
      if [[ -f /secrets/gcs-access-id && -f /secrets/gcs-private-key ]]; then
        export CACHE_GCS_ACCESS_ID=$(cat /secrets/gcs-access-id)
        # echo -e used to make private key multiline (in google json auth key private key is one line with \n)
        export CACHE_GCS_PRIVATE_KEY=$(echo -e $(cat /secrets/gcs-private-key))
      fi
    fi

    if [[ -f /secrets/runner-registration-token ]]; then
      export REGISTRATION_TOKEN=$(cat /secrets/runner-registration-token)
    fi

    if [[ -f /secrets/runner-token ]]; then
      export CI_SERVER_TOKEN=$(cat /secrets/runner-token)
    fi

    if ! sh /scripts/register-the-runner; then
      exit 1
    fi

    # Run pre-entrypoint-script
    if ! bash /scripts/pre-entrypoint-script; then
      exit 1
    fi

    # Start the runner
    exec /entrypoint run --user=gitlab-runner \
      --working-directory=/home/gitlab-runner

  config.toml: |
    concurrent = 10
    check_interval = 30
    log_level = "info"
    listen_address = ':9252'
  configure: |
    set -e
    cp /init-secrets/* /secrets
  register-the-runner: |
    #!/bin/bash
    MAX_REGISTER_ATTEMPTS=30

    for i in $(seq 1 "${MAX_REGISTER_ATTEMPTS}"); do
      echo "Registration attempt ${i} of ${MAX_REGISTER_ATTEMPTS}"
      /entrypoint register \
        --non-interactive

      retval=$?

      if [ ${retval} = 0 ]; then
        break
      elif [ ${i} = ${MAX_REGISTER_ATTEMPTS} ]; then
        exit 1
      fi

      sleep 5
    done

    exit 0

  check-live: |
    #!/bin/bash
    if /usr/bin/pgrep -f .*register-the-runner; then
      exit 0
    elif /usr/bin/pgrep gitlab.*runner; then
      exit 0
    else
      exit 1
    fi

  pre-entrypoint-script: |
---
# Source: gitlab-runner/templates/role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: "Role"
metadata:
  name: gitlab-runner-gitlab-runner
  labels:
    app: gitlab-runner-gitlab-runner
    chart: gitlab-runner-0.58.2
    release: "gitlab-runner"
    heritage: "Helm"
rules:
- apiGroups: [""]
  resources: ["*"]
  verbs: ["*"]
---
# Source: gitlab-runner/templates/role-binding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: "RoleBinding"
metadata:
  name: gitlab-runner-gitlab-runner
  labels:
    app: gitlab-runner-gitlab-runner
    chart: gitlab-runner-0.58.2
    release: "gitlab-runner"
    heritage: "Helm"
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: "Role"
  name: gitlab-runner-gitlab-runner
subjects:
- kind: ServiceAccount
  name: gitlab-runner-gitlab-runner
  namespace: "gitlab"
---
# Source: gitlab-runner/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: gitlab-runner-gitlab-runner
  labels:
    app: gitlab-runner-gitlab-runner
    chart: gitlab-runner-0.58.2
    release: "gitlab-runner"
    heritage: "Helm"
spec:
  replicas: 1
  selector:
    matchLabels:
      app: gitlab-runner-gitlab-runner
  template:
    metadata:
      labels:
        app: gitlab-runner-gitlab-runner
        chart: gitlab-runner-0.58.2
        release: "gitlab-runner"
        heritage: "Helm"
      annotations:
        checksum/configmap: a6623303f6fcc3a043e87ea937bb8399d2d0068a901aa9c3419ed5c7a5afa9db
        checksum/secrets: 32c7d2c16918961b7b84a005680f748e774f61c6f4e4da30650d400d781bbb30
        prometheus.io/scrape: 'true'
        prometheus.io/port: '9252'
    spec:
      securityContext:
        runAsUser: 100
        fsGroup: 65533
      terminationGracePeriodSeconds: 3600
      initContainers:
      - name: configure
        command: ['sh', '/config/configure']
        image: gitlab/gitlab-runner:alpine-v13.4.1
        imagePullPolicy: "IfNotPresent"
        env:

        - name: CI_SERVER_URL
          value: "https://gitlab.qa.joaocunha.eu/"
        - name: CLONE_URL
          value: ""
        - name: RUNNER_REQUEST_CONCURRENCY
          value: "1"
        - name: RUNNER_EXECUTOR
          value: "kubernetes"
        - name: REGISTER_LOCKED
          value: "true"
        - name: RUNNER_TAG_LIST
          value: ""
        - name: RUNNER_OUTPUT_LIMIT
          value: "4096"
        - name: KUBERNETES_IMAGE
          value: "ubuntu:16.04"

        - name: KUBERNETES_PRIVILEGED
          value: "true"

        - name: KUBERNETES_NAMESPACE
          value: "gitlab"
        - name: KUBERNETES_POLL_TIMEOUT
          value: "180"
        - name: KUBERNETES_CPU_LIMIT
          value: ""
        - name: KUBERNETES_CPU_LIMIT_OVERWRITE_MAX_ALLOWED
          value: ""
        - name: KUBERNETES_MEMORY_LIMIT
          value: ""
        - name: KUBERNETES_MEMORY_LIMIT_OVERWRITE_MAX_ALLOWED
          value: ""
        - name: KUBERNETES_CPU_REQUEST
          value: ""
        - name: KUBERNETES_CPU_REQUEST_OVERWRITE_MAX_ALLOWED
          value: ""
        - name: KUBERNETES_MEMORY_REQUEST
          value: ""
        - name: KUBERNETES_MEMORY_REQUEST_OVERWRITE_MAX_ALLOWED
          value: ""
        - name: KUBERNETES_SERVICE_ACCOUNT
          value: ""
        - name: KUBERNETES_SERVICE_CPU_LIMIT
          value: ""
        - name: KUBERNETES_SERVICE_MEMORY_LIMIT
          value: ""
        - name: KUBERNETES_SERVICE_CPU_REQUEST
          value: ""
        - name: KUBERNETES_SERVICE_MEMORY_REQUEST
          value: ""
        - name: KUBERNETES_HELPER_CPU_LIMIT
          value: ""
        - name: KUBERNETES_HELPER_MEMORY_LIMIT
          value: ""
        - name: KUBERNETES_HELPER_CPU_REQUEST
          value: ""
        - name: KUBERNETES_HELPER_MEMORY_REQUEST
          value: ""
        - name: KUBERNETES_HELPER_IMAGE
          value: ""
        - name: KUBERNETES_PULL_POLICY
          value: ""
        volumeMounts:
        - name: runner-secrets
          mountPath: /secrets
          readOnly: false
        - name: scripts
          mountPath: /config
          readOnly: true
        - name: init-runner-secrets
          mountPath: /init-secrets
          readOnly: true
        resources:
          {}
      serviceAccountName: gitlab-runner-gitlab-runner
      containers:
      - name: gitlab-runner-gitlab-runner
        image: gitlab/gitlab-runner:alpine-v13.4.1
        imagePullPolicy: "IfNotPresent"
        lifecycle:
          preStop:
            exec:
              command: ["/entrypoint", "unregister", "--all-runners"]
        command: ["/bin/bash", "/scripts/entrypoint"]
        env:

        - name: CI_SERVER_URL
          value: "https://gitlab.qa.joaocunha.eu/"
        - name: CLONE_URL
          value: ""
        - name: RUNNER_REQUEST_CONCURRENCY
          value: "1"
        - name: RUNNER_EXECUTOR
          value: "kubernetes"
        - name: REGISTER_LOCKED
          value: "true"
        - name: RUNNER_TAG_LIST
          value: ""
        - name: RUNNER_OUTPUT_LIMIT
          value: "4096"
        - name: KUBERNETES_IMAGE
          value: "ubuntu:16.04"

        - name: KUBERNETES_PRIVILEGED
          value: "true"

        - name: KUBERNETES_NAMESPACE
          value: "gitlab"
        - name: KUBERNETES_POLL_TIMEOUT
          value: "180"
        - name: KUBERNETES_CPU_LIMIT
          value: ""
        - name: KUBERNETES_CPU_LIMIT_OVERWRITE_MAX_ALLOWED
          value: ""
        - name: KUBERNETES_MEMORY_LIMIT
          value: ""
        - name: KUBERNETES_MEMORY_LIMIT_OVERWRITE_MAX_ALLOWED
          value: ""
        - name: KUBERNETES_CPU_REQUEST
          value: ""
        - name: KUBERNETES_CPU_REQUEST_OVERWRITE_MAX_ALLOWED
          value: ""
        - name: KUBERNETES_MEMORY_REQUEST
          value: ""
        - name: KUBERNETES_MEMORY_REQUEST_OVERWRITE_MAX_ALLOWED
          value: ""
        - name: KUBERNETES_SERVICE_ACCOUNT
          value: ""
        - name: KUBERNETES_SERVICE_CPU_LIMIT
          value: ""
        - name: KUBERNETES_SERVICE_MEMORY_LIMIT
          value: ""
        - name: KUBERNETES_SERVICE_CPU_REQUEST
          value: ""
        - name: KUBERNETES_SERVICE_MEMORY_REQUEST
          value: ""
        - name: KUBERNETES_HELPER_CPU_LIMIT
          value: ""
        - name: KUBERNETES_HELPER_MEMORY_LIMIT
          value: ""
        - name: KUBERNETES_HELPER_CPU_REQUEST
          value: ""
        - name: KUBERNETES_HELPER_MEMORY_REQUEST
          value: ""
        - name: KUBERNETES_HELPER_IMAGE
          value: ""
        - name: KUBERNETES_PULL_POLICY
          value: ""
        livenessProbe:
          exec:
            command: ["/bin/bash", "/scripts/check-live"]
          initialDelaySeconds: 60
          timeoutSeconds: 1
          periodSeconds: 10
          successThreshold: 1
          failureThreshold: 3
        readinessProbe:
          exec:
            command: ["/usr/bin/pgrep","gitlab.*runner"]
          initialDelaySeconds: 10
          timeoutSeconds: 1
          periodSeconds: 10
          successThreshold: 1
          failureThreshold: 3
        ports:
        - name: metrics
          containerPort: 9252
        volumeMounts:
        - name: runner-secrets
          mountPath: /secrets
        - name: etc-gitlab-runner
          mountPath: /home/gitlab-runner/.gitlab-runner
        - name: scripts
          mountPath: /scripts
        resources:
          {}
      volumes:
      - name: runner-secrets
        emptyDir:
          medium: "Memory"
      - name: etc-gitlab-runner
        emptyDir:
          medium: "Memory"
      - name: init-runner-secrets
        projected:
          sources:
            - secret:
                name: "gitlab-runner-gitlab-runner"
                items:
                  - key: runner-registration-token
                    path: runner-registration-token
                  - key: runner-token
                    path: runner-token
      - name: scripts
        configMap:
          name: gitlab-runner-gitlab-runner

문제 해결#

오류: associative list with keys has an element that omits key field "protocol"#

Kubernetes v1.19의 버그로 인해, GitLab Runner 또는 기타 애플리케이션을 Kubernetes용 GitLab 에이전트로 설치할 때 이 오류가 발생할 수 있습니다. 해결 방법:

• Kubernetes 클러스터를 v1.20 이상으로 업그레이드하세요.

• containers.ports 하위 섹션에 protocol: TCP를 추가하세요:

  ...
  ports:
    - name: metrics
      containerPort: 9252
      protocol: TCP
  ...