Docker is great. GitLab is also great. Running GitLab in Docker is awesome. Taking advantage of CI/CD and a registry is the amazing.

There are, however, some momentswhen things just don't seem to go right. These moments are generally just after upgrading or migrating. So this post is mostly a note-to-self of what can go astray.

This is my current docker-compose.yml file:

version: "2.4"

services:
  gitlab:
    image: gitlab/gitlab-ce
    container_name: gitlab
    restart: unless-stopped
    hostname: ${HOST_URL}
    environment:
      GITLAB_OMNIBUS_CONFIG: |
        external_url 'https://${HOST_URL}'
        letsencrypt['enabled'] = false

        gitlab_rails['time_zone'] = 'Australia/West'
        gitlab_rails['gitlab_email_enabled'] = true
        gitlab_rails['gitlab_email_from'] = 'gitlab@mydomain.com'
        gitlab_rails['gitlab_email_display_name'] = 'GitLab'

        nginx['listen_port'] = 80
        nginx['listen_https'] = false

        # email server config
        gitlab_rails['smtp_enable'] = true
        gitlab_rails['smtp_address'] = "smtp.sendgrid.net"
        gitlab_rails['smtp_port'] = 587
        gitlab_rails['smtp_user_name'] = "apikey"
        gitlab_rails['smtp_password'] = '${SENDGRID_PASSWORD}'
        gitlab_rails['smtp_domain'] = "smtp.sendgrid.net"
        gitlab_rails['smtp_authentication'] = "login"
        gitlab_rails['smtp_enable_starttls_auto'] = true
        gitlab_rails['smtp_tls'] = false

        # analytics
        gitlab_rails['extra_piwik_url'] = '${STATS_URL}'
        gitlab_rails['extra_piwik_site_id'] = '3'

        # registry_external_url 'https://${REGISTRY_URL}'
        gitlab_rails['registry_enabled'] = true
        gitlab_rails['registry_host'] = '${REGISTRY_URL}'
        gitlab_rails['registry_api_url'] = 'http://registry:5000'
        gitlab_rails['registry_path'] = '/var/opt/gitlab/registry'
        gitlab_rails['registry_issuer'] = 'omnibus-gitlab-issuer'
        registry['internal_key'] = "${REGISTRY_KEY}"

    ports:
      - "22:22"
      - 80
    volumes:
      - /srv/gitlab/config:/etc/gitlab
      - /srv/gitlab/logs:/var/log/gitlab
      - /srv/gitlab/data:/var/opt/gitlab
    - /srv/registry:/var/opt/gitlab/registry
    networks:
      - internal
      # - mail
      - web
    labels:
      traefik.backend.healthcheck.path: "/-/liveness?token=${token}"
      traefik.docker.network: "web"
      traefik.enable: "true"
      traefik.frontend.entryPoints: "http,https"
      traefik.frontend.headers.SSLRedirect: "true"
      traefik.frontend.redirect.entryPoint: "https"
      traefik.frontend.rule: "Host:${HOST_URL}"
      traefik.port: 80
    cpus: 3

  runner:
    image: gitlab/gitlab-runner:alpine
    container_name: gitlab-runner
    restart: unless-stopped
    networks:
      - internal
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /srv/gitlab-runner/config:/etc/gitlab-runner
    cpus: 2

  registry:
    image: registry:2
    container_name: registry
    restart: unless-stopped
    ports:
      - 5000
    networks:
      - internal
      - web
    depends_on:
      - gitlab
    volumes:
      - /srv/registry:/var/lib/registry
    - ./registry-config.yml:/etc/docker/registry/config.yml
      - ./certs:/certs
    labels:
      traefik.docker.network: "web"
      traefik.enable: "true"
      traefik.frontend.entryPoints: "http,https"
      traefik.frontend.headers.SSLRedirect: "true"
      traefik.frontend.redirect.entryPoint: "https"
      traefik.frontend.rule: "Host:${REGISTRY_URL}"
      traefik.port: 5000

networks:
  web:
    external: true
  internal:
    external: false

with registry-config.yml:

version: 0.1
log:
  level: info
  formatter: text
auth:
  token:
    realm: https://git.mydomain.com/jwt/auth
    service: container_registry
    issuer: omnibus-gitlab-issuer
    rootcertbundle: /certs/auth.crt
http:
  addr: :5000
  secret: secret
  headers:
    X-Content-Type-Options: [nosniff]
health:
  storagedriver:
enabled: true
    interval: 60s
    threshold: 3
storage:
  filesystem:
    rootdirectory: /var/lib/registry
  delete:
    enabled: true

Permissions

Permissions are generally the cause of most pitfalls.

ssh

If you can't push or pull a repo, then it's possible that the ssh keys have incorrect permissions. Running docker-compose exec gitlab update-permissions is worth a shot, but doesn't always get it.

If the problem persists, scour the logs for a warning message about the permissions on the ssh keys. Then docker-compose exec gitlab bash into the container, find the corresponding directory and update the permissions. 600 is generally the mode that is required for ssh keys.

registry

When I was unable to docker login to my registry from the commandline, this site had the solution for me.

Check the registry logs for a message like this:

level=warning msg="error authorizing context: authorization token required"

Check the certificate permissions. The following permissions work:

michael@server:~/gitlab/certs$ sudo chmod 0640 auth.key
michael@server:~/gitlab/certs$ sudo chmod 0750 auth.crt

michael@server:~/gitlab/certs$ ls -al
total 16
drwxrwxr-x 2 michael michael 4096 Aug 24 06:33 .
drwxrwxr-x 3 michael michael 4096 Aug 24 09:23 ..
-rwxr-x--- 1 root    root    1818 Aug 24 06:35 auth.crt
-rw-r----- 1 root    root    3268 Aug 24 06:35 auth.key

Conclusion

Make sure the permissions are right.

Hopefullythis helps you, future me.
Regards,
Present me.