Cloud Experts Documentation

Federating System and User metrics to Azure Blob storage in Azure Red Hat OpenShift

This content is authored by Red Hat experts, but has not yet been tested on every supported configuration.

By default Azure Red Hat OpenShift (ARO) stores metrics in Ephemeral volumes, and its advised that users do not change this setting. However its not unreasonable to expect that metrics should be persisted for a set amount of time.

This guide shows how to set up Thanos to federate both System and User Workload Metrics to a Thanos gateway that stores the metrics in Azure Blob Container and makes them available via a Grafana instance (managed by the Grafana Operator).

ToDo - Add Authorization in front of Thanos APIs

Pre-Prequsites

  1. An ARO cluster

  2. Set some environment variables to use throughout to suit your environment

    Note: AZR_STORAGE_ACCOUNT_NAME must be unique

    export AZR_RESOURCE_LOCATION="eastus"
    export AZR_RESOURCE_GROUP="openshift"
    export CLUSTER_NAME="openshift"
    export UNIQUE="$(cat /dev/urandom | LC_ALL=C tr -dc 'a-z0-9' | fold -w 5 | head -n 1)"
    export AZR_STORAGE_ACCOUNT_NAME="arometrics${UNIQUE}"
    export NAMESPACE="aro-thanos-af"
    

Azure Preperation

  1. Create an Azure storage account

    modify the arguments to suit your environment

    az storage account create \
      --name $AZR_STORAGE_ACCOUNT_NAME \
      --resource-group $AZR_RESOURCE_GROUP \
      --location $AZR_RESOURCE_LOCATION \
      --sku Standard_RAGRS \
      --kind StorageV2
    
  2. Get the account key and update the secret in thanos-store-credentials.yaml

    AZR_STORAGE_KEY=$(az storage account keys list -g $AZR_RESOURCE_GROUP \
      -n $AZR_STORAGE_ACCOUNT_NAME --query "[0].value" -o tsv)
    
  3. Create a namespace to use

    oc new-project $NAMESPACE
    
  4. Add the MOBB chart repository to your Helm

    helm repo add mobb https://rh-mobb.github.io/helm-charts/
    
  5. Update your repositories

    helm repo update
    
  6. Use the mobb/operatorhub chart to deploy the grafana operator

    helm upgrade -n $NAMESPACE grafana-operator \
      mobb/operatorhub --install \
      --values https://raw.githubusercontent.com/rh-mobb/helm-charts/main/charts/aro-thanos-af/files/grafana-operator.yaml
    
  7. Wait for the Operator to be ready

    oc rollout status -n $NAMESPACE \
      deployment/grafana-operator-controller-manager
    
  8. Use Helm deploy the OpenShift Patch Operator

    helm upgrade -n patch-operator patch-operator --create-namespace \
      mobb/operatorhub --install \
      --values https://raw.githubusercontent.com/rh-mobb/helm-charts/main/charts/aro-thanos-af/files/patch-operator.yaml
    
  9. Wait for the Operator to be ready

    oc rollout status -n patch-operator \
      deployment/patch-operator-controller-manager
    
  10. Deploy ARO Thanos Azure Blob container Helm Chart (mobb/aro-thanos-af)

    **> Note: enableUserWorkloadMetrics=true will overwrite configs for cluster and userworkload metrics. If you have customized them already, you may need to modify patch-monitoring-configs.yaml in the Helm chart to include your changes.

    helm upgrade -n $NAMESPACE aro-thanos-af \
      --install mobb/aro-thanos-af \
      --set "aro.storageAccount=$AZR_STORAGE_ACCOUNT_NAME" \
      --set "aro.storageAccountKey=$AZR_STORAGE_KEY" \
      --set "aro.storageContainer=$CLUSTER_NAME" \
      --set "aro.clusterName=$CLUSTER_NAME" \
      --set "enableUserWorkloadMetrics=true"
    

Validate Grafana is installed and seeing metrics from Azure Blob storage

  1. get the Route URL for Grafana (remember its https) and login using username admin and the password password.

    oc -n $NAMESPACE get route grafana-route
    
  2. Once logged in go to Dashboards->Manage and expand the aro-thanos-af group and you should see the cluster metrics dashboards. Click on the Use Method / Cluster Dashboard and you should see metrics. \o/.

    Note: If it complains about a missing datasource run the following: oc annotate -n $NAMESPACE grafanadatasource aro-thanos-af-prometheus "retry=1"

    screenshot of grafana with federated cluster metrics

Cleanup

  1. Uninstall the aro-thanos-af chart

    helm delete -n $NAMESPACE aro-thanos-af
    
  2. Uninstall the federated-metrics-operators chart

    helm delete -n $NAMESPACE federated-metrics-operators
    
  3. Delete the aro-thanos-af namespace

    oc delete namespace $NAMESPACE
    
  4. Delete the storage account

    az storage account delete \
      --name $AZR_STORAGE_ACCOUNT_NAME \
      --resource-group $AZR_RESOURCE_GROUP
    

Interested in contributing to these docs?

Collaboration drives progress. Help improve our documentation The Red Hat Way.

Red Hat logo LinkedIn YouTube Facebook Twitter

Products

Tools

Try, buy & sell

Communicate

About Red Hat

We’re the world’s leading provider of enterprise open source solutions—including Linux, cloud, container, and Kubernetes. We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Subscribe to our newsletter, Red Hat Shares

Sign up now
© 2023 Red Hat, Inc.