Table of Contents
This a cross post of a post I wrote for the ownCloud Blog, the original post can be found here: Running ownCloud in Kubernetes With Rook Ceph Storage – Step by Step.
Thanks to them for allowing me to write and publish the post on their blog!
Let's prepare for the Kubernetes madness!
Kubernetes Cluster Access
As written in the first part, it is expected that you have (admin) access to a Kubernetes cluster already.
If you don't have a Kubernetes cluster, you can try using the following projects xetys/hetzner-kube on GitHub, Kubespray and others (Kubernetes documentation).
minikube is not enough when started with the default resources, be sure to give minikube extra resources otherwise you will run into problems! Be sure to add the following flags to the
minikube start command:
--memory=4096 --cpus=3 --disk-size=40g.
You should have
cluster-admin access to the Kubernetes cluster! Other access can also work, but due to the nature of objects that are created along the way it is easier to have the
WARNING Only follow this section, if your Kubernetes cluster does not have an Ingress controller yet.
We are going to install the Kubernetes NGINX Ingress Controller.
The instructions shown here are for an environment without
LoadBalancer Service type support (e.g., bare metal, “normal” VM provider, not cloud), for installation instructions for other environments checkout Installation Guide - NGINX Ingress Controller.
As these are bare metal installation instructions, the NGINX Ingress controller will be available through a Service of type
NodePort. This Service type exposes one or more ports on all Nodes in the Kubernetes cluster.
To get that port run:
In that output you can see the NodePorts for HTTP and HTTPS on which you can connect to the NGINX Ingress controller and ownCloud later.
Though as written you probably want to look into a more “solid” way to expose the NGINX Ingress controller(s), for bare metal where there is no Kubernetes LoadBalancer integration one can consider using
hostNetwork option for that: Bare-metal considerations - NGINX Ingress Controller.
Through the whole installation we will create 4 Namespaces:
rook-ceph- For the Rook run Ceph cluster + the Rook Ceph operator (will be created in Rook Ceph storage).
owncloud- For ownCloud and the other operators, such as Zalando's Postgres Operator and KubeDB for Redis.
ingress-nginx- If you don't have an Ingress controller running yet, the namespace is used for the Ingress NGINX controller (it is already created in the Ingress Controller steps).
Rook Ceph Storage
Now on to running Ceph in Kubernetes, using the Rook.io project.
In the following sections make sure to use the available
-test suffixed files if you have less than 3 Nodes which are available to any application / Pod (e.g., depending on your cluster the masters are not available for Pods).
(You can change that, for that be sure to dig into the
spec.placement.tolerations and the Operator environment variables for the discover and agent daemons. Running application Pods on the masters is not recommended though)
The operator will take care of starting up the Ceph components one by one and also preparing of disks and health checking.
You can check on the Pods to see how it looks:
rook-discover-* Pods are each one on each Node of your Kubernetes cluster, as they are discovering the disks of the Nodes so the operator can plan the actions for a given
CephCluster object which comes up next.
This is the definition of Ceph cluster that will be created in Kubernetes. It contains the lists and options on which disks to use and on which Nodes.
If you wanna see some example CephCluster objects to see what is possible, be sure to checkout Rook v1.0 Documentation - CephCluster CRD.
INFO Use the
cluster-test.yaml when your Kubernetes cluster has less than 3 schedulable Nodes (e.g., minikube)!
When using the
cluster-test.yaml only one
mon is started. If that mon is down for whatever reason, the Ceph Cluster will come to a halt to prevent any data “corruption”.
This will now cause the operator to start the Ceph cluster after the specifications in the CephCluster object.
To see which Pods have already been created by the operator, you can run (output example from a three node cluster):
Block storage (RBD)
Before creating the CephFS filesystem, let's create a block storage pool with a StorageClass. The StorageClass is for the PostgreSQL and if you want even the Redis cluster.
INFO Use the
storageclass-test.yaml when your Kubernetes cluster has less than 3 schedulable Nodes!
In case of a block storage Pool there are no additional Pods that will be started, we'll verify that the block storage Pool has been created in the Toolbox section.
One more thing to do is, to set the created StorageClass as default in the Kubernetes cluster by running the following command:
Now you are ready to move onto the storage for the actual data to be stored in ownCloud!
CephFS is the filesystem that Ceph offers, with its POSIX compliance it is a perfect fit to be used with ownCloud.
INFO Use the
filesystem-test.yaml when your Kubernetes cluster has less than 3 schedulable Nodes!
Creation of the CephFS will cause, so called MDS daemons, MDS Pods to be started.
This will create a Pod which will allow us to run Ceph commands. It will be use to quickly check the Ceph clusters status.
kubectl exec to enter the Rook Ceph Toolbox Pod:
In the Rook Ceph Toolbox Pod, run the following command to get the Ceph cluster health status (example output from a 7 Node Kubernetes Rook Ceph cluster):
You can also get it by using
That even shows you some additional information directly through
kubectl instead of having to read the
ceph -s output.
This is how it should look Pod wise now in your
rook-ceph Namespace (example output from a 3 Node Kubernetes cluster):
The important thing is that the
ceph -s output or the
kubectl get cephcluster output shows that the
HEALTH_OK and that you have OSD Pods running (
ceph -s output line:
osd: 3 osds: 3 up, 3 in (where 3 is basically the amount of OSD Pods).
Should you not have any OSD Pod, make sure all your Nodes are
Ready and schedulable (e.g., no taints preventing “normal” Pods to run) and make sure to checkout the logs of the
rook-ceph-osd-prepare-* and if existing
rook-ceph-osd-[0-9]* Pods. If you don't have any Pods related to
rook-ceph-osd-* look into the
rook-ceph-operator-* logs for error messages, be sure to go over each line to make sure you don't miss an error message.
Moving on to the PostgreSQL for ownCloud.
Zalando's PostgreSQL operator does a great job for running PostgreSQL in Kubernetes.
First thing to create is the PostgreSQL Operator which brings the CustomResourceDefinitions, remember the custom Kubernetes objects, with itself. Using the Ceph block storage (RBD) we are going to create a redundant PostgreSQL instance for ownCloud to use.
That is the operator created, moving on to the PostgreSQL custom resource object that will cause the operator to create a PostgreSQL instance for use in Kubernetes:
The CustomResourceDefinition exists? Perfect, continue with the creation:
It will take a bit for the two PostgreSQL Pods to appear, but in the end you should have two
Running status? That looks good.
Now that the database is running, let's continue to the Redis.
To run a Redis cluster we need the KubeDB Operator, installing it can done using a bash script or Helm.
To keep it quick'n'easy we'll use their bash script for that:
(You can remove the script afterwards:
For more information on the bash script and / or the Helm installation, checkout KubeDB.
Moving on to creating the Redis cluster, run:
It will take a few seconds for the first Redis Pod(s) to be started, to check that it worked look for Pods with
redis-owncloud- in their name:
That is how it should like now.
Now the final “piece”, ownCloud.
owncloud/ contains all the manifests we need.
- ConfigMap and Secret for basic configuration of the ownCloud.
- Deployment to get ownCloud Pods running in Kubernetes.
- Service and Ingress to expose ownCloud to the internet.
- CronJob to run the ownCloud cron task execution (e.g., cleanup and others), instead of having the cron run per instance.
The ownCloud Deployment currently uses a custom built image (
galexrt/owncloud-server:latest) which has a fix for a clustered Redis configuration issue (pull request has been opened https://github.com/owncloud-docker/base/pull/95).
The admin username is
myowncloudadmin and can be changed in the
owncloud/owncloud-configmap.yaml file. Be sure to restart both ownCloud Pods after changing values in the ConfigMaps and Secrets.
If you want to change the admin password, edit the
owncloud/owncloud-secret.yaml file line
OWNCLOUD_ADMIN_PASSWORD. The values in a Kubernetes Secret object are base64 encoded (e.g.,
echo -n YOUR_PASSWORD | base64 -w0)!
To know when your ownCloud is up'n'running check the logs, e.g.:
Installing server database... will take some time depending on your network, storage and other factors.
[Sun Aug 04 13:26:18.986558 2019] [core:notice] [pid 190] AH00094: Command line: '/usr/sbin/apache2 -f /etc/apache2/apache2.conf -D FOREGROUND' you should be able to reach your ownCloud instance through the NodePort Service Port (on HTTP) or through the Ingress (default address
owncloud.example.com). If you are using the Ingress from the example files, be sure to edit it to use a (sub-) domain pointing to the Ingress controllers in your Kubernetes cluster.
You now have a ownCloud instance running.
To further improve the experience of running ownCloud in Kubernetes, you will probably want to checkout Jetstack's cert-manager project on GitHub to get yourself Letsencrypt certificates for your Ingress controller.
cert-manager allows you to request Let's Encrypt certificates easily through Kubernetes custom objects and keep them uptodate.
Meaning the ownCloud will then be reachable via HTTPS which combined with the ownCloud encryption makes it pretty secure.
For more information on using TLS with Kubernetes Ingress, checkout Ingress - Kubernetes.
Pod Health Checks
owncloud/owncloud-deployment.yaml there is a
livenessProbe in the Deployment sepc but commented out.
After the ownCloud has been installed and you have verified it is running, you can go ahead and uncomment those lines and use
kubectl apply /
kubectl replace (don't forget to specify the Namespace
When changing the upload filesize on the ownCloud instance itself through the environment variables, be sure to also update the Ingress controller with the “max upload file size”.
Other configuration options
When wanting to change config options, you need to provide them through environment variables. The environment variables are given to the ownCloud Deployment in the
A list of all available environment variables can be found here:
Updating ownCloud in Kubernetes
It is the same procedure as with running ownCloud with, e.g.,
To update ownCloud you need to scale down the Deployment to
replicas), then update the image, wait for the one single Pod come up again and then scale up the ownCloud Deployment again to, e.g.,
2 or more.
This is the end of the two part series on running ownCloud in Kubernetes.