Today, I finally solved a problem which cost me a few hours of research and I'd like to share my findings so that you don't have to waste the same amount of time. The setup: A server with the GitLab omnibus installed, as well as a working domain. Instead of using the omnibus' built-in nginx web server, I used a custom nginx as I also serve other subdomains on the same server. As a basis I used the following nginx configuration file found here:
## GitLab
##
## Modified from nginx http version
## Modified from http://blog.phusion.nl/2012/04/21/tutorial-setting-up-gitlab-on-debian-6/
## Modified from https://raymii.org/s/tutorials/Strong_SSL_Security_On_nginx.html
##
## Lines starting with two hashes (##) are comments with information.
## Lines starting with one hash (#) are configuration parameters that can be uncommented.
##
##################################
## CONTRIBUTING ##
##################################
##
## If you change this file in a Merge Request, please also create
## a Merge Request on https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests
##
###################################
## configuration ##
###################################
##
## See installation.md#using-https for additional HTTPS configuration details.
upstream gitlab-workhorse {
server unix:/var/opt/gitlab/gitlab-workhorse/socket fail_timeout=0;
}
## Redirects all HTTP traffic to the HTTPS host
server {
## Either remove "default_server" from the listen line below,
## or delete the /etc/nginx/sites-enabled/default file. This will cause gitlab
## to be served if you visit any address that your server responds to, eg.
## the ip address of the server (http://x.x.x.x/)
listen 0.0.0.0:80;
listen [::]:80 ipv6only=on default_server;
server_name YOUR_SERVER_FQDN; ## Replace this with something like gitlab.example.com
server_tokens off; ## Don't show the nginx version number, a security best practice
return 301 https://$http_host$request_uri;
access_log /var/log/nginx/gitlab_access.log;
error_log /var/log/nginx/gitlab_error.log;
}
## HTTPS host
server {
listen 0.0.0.0:443 ssl;
listen [::]:443 ipv6only=on ssl default_server;
server_name YOUR_SERVER_FQDN; ## Replace this with something like gitlab.example.com
server_tokens off; ## Don't show the nginx version number, a security best practice
root /opt/gitlab/embedded/service/gitlab-rails/public;
## Strong SSL Security
## https://raymii.org/s/tutorials/Strong_SSL_Security_On_nginx.html & https://cipherli.st/
ssl on;
ssl_certificate /etc/nginx/ssl/gitlab.crt;
ssl_certificate_key /etc/nginx/ssl/gitlab.key;
# GitLab needs backwards compatible ciphers to retain compatibility with Java IDEs
ssl_ciphers "ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA:ECDHE-RSA-DES-CBC3-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4";
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 5m;
## See app/controllers/application_controller.rb for headers set
## [Optional] Enable HTTP Strict Transport Security
## HSTS is a feature improving protection against MITM attacks
## For more information see: https://www.nginx.com/blog/http-strict-transport-security-hsts-and-nginx/
# add_header Strict-Transport-Security "max-age=31536000; includeSubDomains";
## [Optional] If your certficate has OCSP, enable OCSP stapling to reduce the overhead and latency of running SSL.
## Replace with your ssl_trusted_certificate. For more info see:
## - https://medium.com/devops-programming/4445f4862461
## - https://www.ruby-forum.com/topic/4419319
## - https://www.digitalocean.com/community/tutorials/how-to-configure-ocsp-stapling-on-apache-and-nginx
# ssl_stapling on;
# ssl_stapling_verify on;
# ssl_trusted_certificate /etc/nginx/ssl/stapling.trusted.crt;
# resolver 208.67.222.222 208.67.222.220 valid=300s; # Can change to your DNS resolver if desired
# resolver_timeout 5s;
## [Optional] Generate a stronger DHE parameter:
## sudo openssl dhparam -out /etc/ssl/certs/dhparam.pem 4096
##
# ssl_dhparam /etc/ssl/certs/dhparam.pem;
## Individual nginx logs for this GitLab vhost
access_log /var/log/nginx/gitlab_access.log;
error_log /var/log/nginx/gitlab_error.log;
location / {
client_max_body_size 0;
gzip off;
## https://github.com/gitlabhq/gitlabhq/issues/694
## Some requests take more than 30 seconds.
proxy_read_timeout 300;
proxy_connect_timeout 300;
proxy_redirect off;
proxy_http_version 1.1;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-Ssl on;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_pass http://gitlab-workhorse;
}
}
The idea was to register a GitLab Runner, but every time I tried so, I got the error message x509: certificate signed by unknown authority. While you find some answers on Google, they did not seem to help me much. When I switched back to GitLab internal nginx, everything was working flawlessly, so it had to be an issue with my nginx configuration.
As I had access to another gitlab instance with almost the same configuration, I first inspected the certificate
chain within the browser. Both instances seemed completely fine with no differences, except that one could register
runners and the other one could not. Next I used openssl s_client -connect sub.domain.com:443 to verify
the ssl/tls settings. The problematic GitLab instance returned
code: 21 (unable to verify the first certificate) while the other one simply said that everything was
okay. But this indicator made me wonder, why on earth it is not possible to verify the first certificate.
Long story short: It turned out that while adjusting the above nginx configuration, I only referenced the
cert.pem, but not the fullchain.pem, which caused the issue. Simple but magnificent.
Restarting nginx and the error was gone. Takeaway: Always stay alert when simply adjusting a template and think
about what you're doing 😉.