Change GitLab Port

nano /etc/gitlab/gitlab.rb

Change external_url from yourdomain.com to yourdomain.com:9999

gitlab-ctl reconfigure

References
http://serverfault.com/questions/585528/set-gitlab-external-web-port-number