A Kubernetes Operator is a software component that extends the Kubernetes platform to manage specific types of applications and their underlying infrastructure. Operators provide a high-level and abstract way to manage the entire application stack, including deployment, scaling, upgrades, and maintenance of the application components. By encapsulating the operational knowledge of a specific application into a reusable package, operators simplify the deployment and management of complex applications within a Kubernetes environment.
Operators are custom controllers that watch the changes in the state of the application and take necessary actions to keep the application running as expected, providing a declarative interface for managing the application and ensuring a desired state is achieved. Kubernetes natively doesn’t have the knowledge required to automate the process of deploying every single stateful application, Operator custom CRD’s will extend Kubernetes API to overcome this issue.
The Prometheus Operator is a software component that extends the Kubernetes platform to manage Prometheus, an open-source monitoring and alerting system. The Prometheus Operator provides a simple and easy way to deploy, configure, and manage Prometheus instances within a Kubernetes cluster.
With the use of Prometheus Operator, users can automate the deployment and management of Prometheus instances, reducing the time and effort required to configure and maintain monitoring for their applications. The Prometheus Operator also provides built-in support for features such as rolling updates, scaling, and backup and restore, making it easier for users to ensure that their monitoring systems are reliable and scalable.
Monitoring focuses on predefined metrics to identify when a specific platform or application is performing within expectations. The metrics tracked generally focus on things like resource utilisations.
Observability combines metrics, logs and traces a specialised form of logging to instrument applications in a way that makes it simpler to troubleshoot issues without having to rely solely on a limited set of metrics that have been pre-defined to monitor a specific process or function. Observability provides proactive prevention with the help of collected metrics.
Deploying Prometheus Operator and kube-prometheus in an Azure Kubernetes Service (AKS) cluster using Prometheus operator Helm.
$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts 
$ helm install prometheus prometheus-community/kube-prometheus-stack --namespace monitoring --create-namespace
$ helm list --all-namespaces
Let’s examine all the Kubernetes components that are deployed with the Prometheus Operator Helm chart.
$ kubectl get pods --namespace monitoring 
Note: The componentes that have “-operator-” in their name are managed by the operator itself.
$ kubectl get service --namespace monitoring
$ kubectl get daemonset --namespace monitoring 
Node Exporter: Node Exporter is a Prometheus exporter that collects hardware and operating system metrics from a target node in a Kubernetes cluster. It exposes metrics related to CPU usage, memory, disk usage, network activity, and other system statistics. Node Exporter is typically deployed as a DaemonSet in Kubernetes, which ensures that it runs on every node in the cluster. These metrics can then be scraped by Prometheus to monitor the performance and health of the nodes in the cluster.
$ kubectl get deployments --namespace monitoring
In the Kubernetes deployments, you can find three main components: grafana, kube-prometheus-operator and kube-state-metrics.
$ kubectl get replicaset --namespace monitoring
$ kubectl get statefulset --namespace monitoring
Two statefulsets are deployed: one for the Alert Manager and the other for the Prometheus server.
Alertmanager is a component in the Prometheus ecosystem that is responsible for handling alerts sent by client applications such as Prometheus server. It manages those alerts, groups them, and routes them to different receivers (such as email, Slack, PagerDuty, etc.) based on their labels and filtering rules.
The Prometheus server is a core component of the toolkit that scrapes and stores time series data from monitored endpoints, and provides a query language and web UI for data analysis and visualisation. The server also supports alerting based on defined rules and threshold conditions.
$ kubectl port-forward deployment/prometheus-grafana 3000 -n monitoring
To access the Grafana dashboard, open http://localhost:3000 in your browser and use the default Grafana credentials, which can be obtained by following these steps.
$ kubectl logs prometheus-grafana-55f68d49c5-cstfz -c grafana -n monitoring | grep -e 'Listen' -e 'default admin'
$ kubectl get secret prometheus-grafana -n monitoring -oyaml
$ echo "cHJvbS1vcGVyYXRvcg==" | base64 --decode
To access the Prometheus UI, open http://localhost:9090 in your browser:
$ kubectl describe statefulset prometheus-prometheus-kube-prometheus-prometheus -n monitoring | grep -A 5 'Containers'
$ kubectl port-forward prometheus-prometheus-kube-prometheus-prometheus-0 9090 -n monitoring
In a nutshell, deploying the Prometheus Operator to monitor Kubernetes clusters using Helm is a straightforward process. Helm allows us to easily install, upgrade, and manage the Prometheus Operator and its components, including Alertmanager and Grafana, with just a few simple commands. By following the steps outlined in this blog, you can quickly deploy and configure the Prometheus Operator, and start collecting and visualising metrics for your Kubernetes clusters. With the Prometheus Operator, you can gain valuable insights into your clusters’ health and performance, and make informed decisions to optimise their performance and reliability.
Looking for help with your Infrastructure or want help with your DevOps implementation strategy? Reach out to us and see how we can help.
Experienced DevOps Engineer with a demonstrated history of working in the computer software industry. Skilled in AWS, Terraform, Jenkins, Prometheus and Grafana, New Relic, Github, SQL, Linux, and K8s. Strong information technology professional with a B-Tech focused in Information Technology from Dr N.G.P. Institute of Technology.