Grafana

The Materialize Terraform modules can deploy a monitoring stack alongside your Materialize deployment. When you set enable_observability = true, the modules install:

The stack comes from the materialize-monitoring ⧉ charts. The Terraform modules also create the object storage and the cloud identities the stack needs, so you do not have to configure scrape targets, data sources, or dashboards yourself.

This stack was introduced in TF v10.0.0, replacing an earlier Prometheus-and-Grafana pair that collected metrics only. TF v10.1.0 then added durable state for Grafana and a load balancer to reach it on. If you are upgrading from before v10.0.0, read Upgrading from the previous stack first.

Before you begin

Ensure you have:

• A Materialize deployment created with the Materialize Terraform modules.

• Terraform ⧉ installed.

• kubectl ⧉ installed and configured to connect to your cluster.

Upgrading from the previous stack

Before TF v10.0.0, enable_observability = true installed a single Prometheus and a Grafana from kubernetes/modules/prometheus and kubernetes/modules/grafana. Those two modules were removed in v10.0.0 — not deprecated in place — and replaced by a monitoring module per cloud.

Other things that change on that upgrade:

• If you referenced kubernetes/modules/prometheus or kubernetes/modules/grafana directly rather than through an example, that reference breaks. Pin the previous major until you have migrated to the monitoring module for your cloud.

• The prometheus_url output is gone, replaced by metrics_url (Thanos Query) and logs_url (Loki). Thanos Query is Prometheus-API-compatible, so consumers of the old URL work against the new one — only the host and port change.

• grafana_url and grafana_admin_password keep their names and meaning.

• New cloud resources are created: object storage for each backend (logs and metrics), plus a per-backend cloud identity bound to the in-cluster ServiceAccount.

• If you set install_metrics_server = false on the operator module, set install_metrics_server = true on the monitoring module in the same change. The Materialize Console depends on the metrics API for cluster metrics.

For the per-cloud module blocks and the full upgrade procedure, see the upgrade guide for your cloud: AWS, Azure, or GCP.

Step 1. Enable observability

The simple example for each cloud takes an enable_observability variable, which defaults to false.

1. In your terraform.tfvars, set:

   enable_observability = true
   

2. Apply the configuration:

   terraform apply
   

   The apply creates the object storage and cloud identities for metrics and logs, and installs the stack into the monitoring namespace.

Starting in v10.1.0, the examples also create two resources for Grafana itself whenever enable_observability is on:

Both are billable, and both are sized as small as the cloud offers (db.t4g.micro on AWS, db-f1-micro on GCP, B_Standard_B1ms on Azure). See Step 2 for the load balancer and Step 3 for the database.

If you instantiate the modules in your own Terraform rather than using an example, add the monitoring module for your cloud (see the Terraform installation guide ⧉), and turn on the operator’s scrape annotations so its pods are collected:

module "operator" {
  # ...
  helm_values = {
    observability = {
      enabled = true
      prometheus = {
        scrapeAnnotations = {
          enabled = true
        }
      }
    }
  }
}

Step 2. Access Grafana

Retrieve the admin password from the Terraform output. You need it for either access method below:

terraform output -raw grafana_admin_password

Through the load balancer

Starting in v10.1.0, the examples put Grafana behind an L4 load balancer. It follows the same internal_load_balancer and ingress_cidr_blocks variables as the Materialize load balancer, so by default it is internal and allowlisted to the same ranges.

1. Read the address:

   terraform output -raw grafana_url
   

   grafana_url is the hostname you supplied, else the load balancer’s own address, else the in-cluster Service. grafana_load_balancer_address gives you just the load balancer.

   NOTE: On GCP and Azure the cloud assigns the address asynchronously, so a fresh apply can still report the in-cluster name. The next plan picks it up. Set ip on grafana_load_balancer to pre-allocate the address and have it known at plan time.

2. Open the address in a browser and log in as admin.

To make Grafana’s own share links, alert notification links, and OAuth redirect URIs correct, set grafana_host to a hostname you control. Nothing in the modules publishes DNS for that name — that record is yours to create.

To skip the load balancer entirely and keep Grafana on a ClusterIP Service, set grafana_load_balancer = null on the monitoring module block.

Through port forwarding

Port forwarding stays the private path, and is the only option when the load balancer is internal and you are outside the network.

1. Forward a local port to the Grafana service:

   kubectl -n monitoring port-forward svc/grafana 3000:80
   

2. Open http://localhost:3000 in a browser and log in as admin with the password from above.

Step 3. Persist Grafana’s own state

Grafana keeps users, service accounts and API tokens, annotations, dashboard versions, preferences, and alert-rule state in its own database — separate from the metrics in Thanos and the logs in Loki.

The chart default is SQLite on an emptyDir, so all of it is lost on every restart, upgrade, and reschedule. Starting in v10.1.0 the examples provision a dedicated PostgreSQL instance for it instead, whenever enable_observability is on. Confirm it:

terraform output -raw grafana_database_endpoint

To keep the previous SQLite behaviour, set grafana_database = null on the monitoring module block. To point at a database you already run, leave grafana_database = null and set the grafana_database_host, grafana_database_port, grafana_database_name, grafana_database_user, grafana_database_password, and grafana_database_ssl_mode variables instead.

Step 4. Open the Materialize dashboards

The dashboards and their data sources are installed by Grafana Operator from the released chart, so they track the chart version rather than a copy you maintain.

To confirm that they were installed:

kubectl -n monitoring get grafanamanifest,grafanadatasource

For the list of dashboards and what each one covers, see Grafana dashboards ⧉.

Connect existing tooling

If you already run Grafana, or want to point other tools at the collected data, the examples output the metric and logging query endpoints:

terraform output -raw metrics_url
terraform output -raw logs_url

Advanced configuration

The monitoring modules expose additional options, including sizing profiles, retention, node placement, and raw Helm value overrides. For these, and for installing the stack without the Materialize Terraform modules, see:

• Terraform installation guide ⧉, for the full set of module variables.

• Helm installation guide ⧉, for installing the stack with Helm rather than Terraform.

• Production best practices ⧉, for the throughput envelope each sizing profile assumes and what to scale when metric/logging queries feel slow.

Alerting

The stack includes Alertmanager for recording and routing alerts. For guidance on the initial set of metrics and suggested thresholds, see Alerting.