Table of Contents
Intro
This post walks through using GitLab CI’s Kubernetes Cluster feature to deploy built container images to Kubernetes.
This is an update to my old guide which uses the in GitLab 10.3
deprecated Kubernetes integration feature, see: GitLab + Kubernetes: Perfect Match for Continuous Delivery with Container.
NOTE Please check the requirements before beginning.
Requirements
- Kubernetes Cluster
- GitLab instance
- GitLab Container Registry enabled.
- GitLab CI runner configured and enabled.
- The CI runners must be able to access the Kubernetes apiserver.
kubectl
configured with Kubernetes cluster access.- Kubernetes
ServiceAccount
- That has specific permissions, for more information see Step 2 - Get ServiceAccount Token from Kubernetes.
NOTE In this post the Kubernetes namespace
presentation-gitlab-k8s
will be used for “everything”.
GitLab CI Kubernetes Cluster Feature
The GitLab CI Kubernetes Cluster feature is the successor of the deprecated and beginning with 10.3d
disabled Kubernetes project integration.
Thankfully it is “100%” backwards compatible.
Though I have to note that I find it a bit “mehh” that you can only create/add one Kubernetes cluster in the GitLab community edition (CE).
Step 1 - Download and “import” example Repository
The repository with the files used in this blog post are available on GitHub: galexrt/presentation-gitlab-k8s.
You can use the GitLab repository import functionality to import the repository. If you imported the repository into your GitLab, you should already see GitLab CI begin to do it’s work, but fail on the release_upload
and at latest on the deploy_dev
task, as you shouldn’t have the Kubernetes integration configured and activated before you even read the post yet ;)
NOTE If you have now/already imported the repository, jump to Step 2 - Get ServiceAccount Token from Kubernetes-
When creating the repository, keep it empty! Don’t add a README
or anything at all to it.
Go ahead and clone my repository with the files locally. To import the repository the remote needs to be changed. For this we run the following commands:
|
|
Now we can begin with the GitLab Kubernetes integration.
Step 2 - Get ServiceAccount Token from Kubernetes
NOTE This step is definetely a bit different for newer clusters as you need to get a
ServiceAccount
token from an account with enough permissions to create, modify and delete the following objects in Kubernetes: Create, modify and deleteDeployment
,Service
,Ingress
.For more information and a prescription, talk to your cluster administrator about a
ServiceAccount
that matches these requirements.
For Kubernetes 1.6
and higher with role-based access control (RBAC) enabled you need to have a ServiceAccount
with the correct permissions, to deploy in the namespace of your choice.
For Kubernetes 1.5
and below, you just need to a) create a ServiceAccount
(see note below) or b) use the default existing one in the namespace of your choice.
NOTE It is recommended to create a new
ServiceAccount
for each application!For information on how create a
ServiceAccount
, please refer to the Kubernetes documentation here:
- Kubernetes
1.5
and below: https://kubernetes.io/docs/admin/service-accounts-admin/- Kubernetes
1.6
and higher (with RBAC enabled): https://kubernetes.io/docs/admin/authorization/rbac/
In my case even if it not the best way, I’ll go with the default ServiceAccount
created in the namespace where I will run the application.
For that I check what secrets exist, then get the secret and base64 decode it.
|
|
YOUR_DECODED_TOKEN
somewhere safe.
Step 3 - Get the Kubernetes CA Certificate
At least for my cluster that I setup with the kubernetes/contrib
Ansible deployment the Kubernetes CA certificate is located in /etc/kubernetes/certs/ca.crt
.
So a simple cat
does the thing ;)
|
|
kubectl
config from your provider or administrator, you can find the CA certificate location in there at the certificate-authority
key.
In most cases the so called kubeconfig
will be located at ~/.kube/config
.
|
|
If your kubeconfig
does not contain a certificate-authority:
, but a certificate-authority-data:
take the contents of it and run base64 -d
on it. That will base64 decode the CA certificate for you.
For more information on kubeconfig
, see the Kubernetes documentation for “access” to Kubernetes cluster here: https://kubernetes.io/docs/tasks/access-application-cluster/authenticate-across-clusters-kubeconfig/.
For other cluster “types”/deployments, please refer to your cluster administrator or guide.
Save the CA certificate somewhere safe with the token from Step 2 - Get ServiceAccount Token from Kubernetes.
Step 4 - Create a Kubernetes cluster in GitLab CI
You will now need the ServiceAccount
token, the CA certificate, Kubernetes API server address and the namespace you want to run the application in.
In your GitLab’s sidebar, go to CI / CD
-> Kubernetes
and you should get to this page:
GitLab CI Kubernetes cluster - Cluster list page
Now click the Add Kubernetes cluster
button and you get this page:

GitLab CI Kubernetes cluster - Create GKE cluster or add existing cluster page
On this page you can decide, if you want to create a new Google GKE cluster or add an existing cluster.
Click Add an existing Kubernetes cluster
button, so we can the Kubernetes cluster we just gathered the information for.

GitLab CI Kubernetes cluster - Add existing cluster form
The Kubernetes cluster name
can be “anything”, pick something you are able to identify the cluster later on again in case of issues. The other fields should be filled with the information we gathered it in the previous steps.
The Project namespace
is optional though the .gitlab-ci.yml
shown above/used here “must” have a Kubernetes namespace provided, so set it to presentation-gitlab-k8s
(or your own value, but you need to change all manifests in the repository to match this one then).
Click Add Kubernetes cluster
to add the cluster to GitLab and you now have the Kubernetes integration activated and ready.
Step 5 - Add a .gitlab-ci.yml
to your project
NOTE Replace
registry.example.com
is the address to your GitLab container registry.s3.example.com
is just a minio instance where I upload the artifact to an “external” destination for demonstration. To remove this step just delete therelease_upload
structure. Replace{gitlab,s3,registry}.example.com
with your corresponding domain name!WARNING You should first commit when you are done with adding the manifests from the sixth step too!
The .gitlab-ci.yml
is based on the official GitLab CI Go template.
A job in the .gitlab-ci.yml
file looks like this:
|
|
test
stage.
To specify the stages to be run, you put a simple list of the names anywhere in the file:
|
|
You can specify the image to be used to run the commands on a global level or on a per job basis. To extend the given job example, see below how you can specify the image:
|
|
NOTE For other parts of the
.gitlab-ci.yml
, please check the comments in the file below or just checkout the GitLab CI documentation for all possible settings/parameters here: https://docs.gitlab.com/ce/ci/yaml/README.html.
In the .gitlab-ci.yml
4 stages are defined test
, build
, release
and deploy
:
test
stage simply runs go test to test the example Golang application in this case.build
stage compiles the application and tells GitLab CI that the end binaryapp
is an artifact to be preserved in GitLab and the build containers.release
stage in which theimage_build
job, builds the Docker image and pushes it into the GitLab Container Registry. In therelease
stage, I also upload the artifactapp
into a S3.deploy
stage for branches always deploys to thedev
environment, for tags it will be deployed todev
and the manually triggered intolive
environment.
The whole .gitlab-ci.yml
file looks like this:
|
|
There are special control keys like when
and only
that allow for limiting the runs of the CI, to for example with only: ["tags"]
to run for created tags only and so on.
More on this topic can be found at the GitLab CI YAML file documentation here: https://docs.gitlab.com/ce/ci/yaml/README.html
I hope you can what it does by looking at the script
parts of the jobs and the stages that will be run.
Step 6 - Add Docker login information to Kubernetes
To be able to deploy the built image from the GitLab registry later on, you need to add the Docker login information for the GitLab Registry as a Secret
to Kubernetes. You need to have kubectl
downloaded and usable on your system for that.
The command for creating the Docker login secret is:
|
|
YOUR_PULLSECRET_NAME_HERE
). You will need to put it into the Deployment
manifest that is coming up next.
Step 7 - Create Kubernetes manifests
Now you are creating the Kubernetes manifests for your application and add them to your repository.
Create the Deployment
manifest (deployment.yaml
):
|
|
NOTE Don’t forget to replace
YOUR_SECRET_NAME_HERE
with the actual name of your Docker login secret created in the previous step.
This is a basic Kubernetes Deployment
manifest. For more information on Deployment
manifests please check the Kubernetes Docs page here: https://kubernetes.io/docs/concepts/workloads/controllers/deployment/
Placeholders like __CI_ENVIRONMENT_SLUG__
and __VERSION__
are used for templating this single manifest for the multiple environments we want to achieve.
For example later the __CI_ENVIRONMENT_SLUG__
get’s replaced by dev
or live
(environment name) and __VERSION__
with the built image tag.
To be able to connect to the generated Pod
s of the Deployment
, a Service
is also required.
A Service
manifest looks like this, includes the placeholders already (service.yaml
):
|
|
8000
. The port is named http-metrics
as in my case of Kubernetes cluster I use the prometheus-operator which creates the “auto-discovery” config for Prometheus for example to monitor all Service
s with a port named http-metrics
.
The Kubernetes Service
documentation can be found here: https://kubernetes.io/docs/concepts/services-networking/service/
But now we would be only able to connect to the cluster from the inside and not the outside. That’s what Ingress
es are for. As the name implies they provide a way of allowing traffic to kind of flow into the cluster to a certain Service
.
The following manifest contains so called “annotations” that would automatically get a Let’sencrypt certificate for it and deploy it into the “loadbalancer”. The file is named (ingress.yaml
).
|
|
Ingress
documentation can be found here: https://kubernetes.io/docs/concepts/services-networking/ingress/.
To be able to reach the domain names, you need to already have the DNS names created. With the current manifest you would need to create __CI_ENVIRONMENT_SLUG__-presentation-gitlab-k8s.example.com
, where __CI_ENVIRONMENT_SLUG__
live
and dev
. Resulting in dev-presentation-gitlab-k8s.example.com
and live-presentation-gitlab-k8s.example.com
to be created by yourself.
NOTE The deployment stage could be expanded to use the DNS providers API to create the domain name for you or the external-dns operator from the Kubernetes incubator project could be used.
Now that we have gone through all the manifests in the repository, we can move on to the next step.
Step 8 - Make a change, push and watch the magic happen!
Now that you have the manifests and the .gitlab-ci.yml
file in the repository or from the imported one, you can make a change to the code or just create a file by running the following commands:
|
|
Now you should see GitLab creating a new pipeline for your change and start running through the stages, which you specified in the .gitlab-ci.yml
, with their jobs.
GitLab CI - Pipelines list
GitLab CI - Commit Pipeline list view
When you now go to the pipeline, you should see a view like this:
GitLab CI - Running Pipeline Overview
The last stage shows if you did everything correct. If it passes you now have successfully deployed your application to your Kubernetes cluster.
A successful stage review
deploy looks like this:
GitLab CI - Pipeline deploy_review job successful
If any of the build/steps fail for you, you may have misconfigured your .gitlab-ci.yml
or the GitLab CI Kubernetes integration can’t reach the configured Kubernetes cluster. Make sure connectivity from the GitLab CI Runners to the Kubernetes cluster is given!
For Troubleshooting see the below section for more details on some possible issues.
Troubleshooting
Pipeline stuck on pending
If the build pipeline is stuck in pending, it could be that your GitLab CI runner aren’t properly configured with your GitLab CI instance.
Build failure
- If you made any changes to the
.gitlab-ci.yml
, use the “CI Lint” functionality available on the GitLab Repo pipeline page in the top right corner to check for any syntax issues. - Did you replace all domain names
{gitlab,s3,registry}.zerbytes.net
with your own domains?
Unable to reach the app review URL/deployed project
- Did you replace all domain names
{gitlab,s3,registry}.zerbytes.net
with your own domains? - Is your Ingress (class) correctly setup in the
Ingress
object/Kubernetes cluster?
Summary
I hope this helps you, using the GitLab CI Kubernetes cluster feature for your Continuous Delivery of your application(s) to Kubernetes. For questions about the post, please leave a comment below, thanks!
Have Fun!