Observability tools

View as Markdown

The monitoring stack that the Materialize Terraform modules install keeps metrics in Thanos and logs in Loki, and serves both through Grafana. You can read that data in a browser, through an AI agent, or from the command line. This page compares those tools and shows how to set each one up.

This page assumes the monitoring stack is already running and that you can reach Grafana. To enable the stack and sign in to Grafana, see Grafana.

Compare the tools

Every tool reads the same metrics, logs, and alerts. The tools differ in who they suit, what access they need, and how much they ask of you:

Grafana Grafana MCP server gcx
Interface Browser AI agent, over MCP Command line
Audience Everyone who reads dashboards Anyone who works with an AI agent Scripts, and agents that run shell commands
Grafana access Browser login Service account token Service account token
Ease of use Highest: nothing to install High after setup, but check its answers Moderate: PromQL and LogQL

All three reach the data through Grafana. None of them needs Kubernetes access unless you can’t reach Grafana’s address directly, for example because its load balancer is internal and you are outside the network. In that case, port-forward Grafana as Access Grafana describes.

Pick a tool by what you are doing:

  • Reading a dashboard, or sharing a link: use Grafana in a browser.
  • Asking questions in plain language, or having an AI agent investigate: use the Grafana MCP server.
  • Scripting queries, or working from a terminal: use gcx.

To query Thanos, Loki, or Alertmanager directly, for example while Grafana is down, see Connect existing tooling for the stores’ addresses, and Backend APIs ⧉ in the monitoring stack’s documentation for example queries.

WARNING! Every tool on this page reads all of the telemetry the stack holds. A Grafana role limits what you can change, not what you can read, so any Grafana login or service account token reads every metric and every log.

Names the tools share

Scripts and AI agents reach the data through Grafana’s data sources, by UID:

Data source Type UID Holds
Thanos prometheus mzmon-thanos Metrics
Loki loki mzmon-loki Logs and Kubernetes events
Alertmanager alertmanager mzmon-alertmanager Firing alerts and silences

The following names also come up often:

  • Dashboard UIDs start with mz-mon-. For example, mz-mon-env-top is the Materialize Environment Overview dashboard.

  • Folder UIDs are mzmon-materialize for the Materialize dashboards and mzmon-infra for the cluster dashboards.

  • The environment label, materialize_cloud_organization_name, scopes Materialize metrics to one environment. Self-Managed deployments carry it too, despite its name.

Use Grafana in a browser

Grafana needs no setup beyond signing in. For its address and the admin password, see Access Grafana.

Start from these places:

  • Dashboards: the Materialize folder holds the per-environment dashboards. Start with Materialize Environment Overview.

  • Explore: query the Thanos data source in PromQL, or the Loki data source in LogQL.

  • Alerting: Alerting > Silences shows and manages silences in the bundled Alertmanager. See Silence alerts.

A dashboard’s URL carries its time range and variables, so you can share what you see with anyone who can sign in.

Create a service account token

The Grafana MCP server and gcx sign in to Grafana with a service account token rather than a password. To create one:

  1. Sign in to Grafana as admin.

  2. Go to Administration > Users and access > Service accounts, and click Add service account.

  3. Enter a display name, such as mcp-yourname, and click Create.

  4. Check that the service account’s role is Viewer, and change it with the role picker if it is not.

  5. Click Add service account token, then Generate token. Copy the token, and store it somewhere safe.

Create one service account for each person or agent, so you can revoke one without the rest. The Viewer role is enough to read dashboards and run queries.

NOTE: Grafana keeps service accounts and tokens in its own database. If your deployment still keeps that database on the chart’s default SQLite, every token disappears when Grafana restarts. See Persist Grafana’s own state.

Connect an AI agent

The Grafana MCP server ⧉, mcp-grafana, connects an AI agent such as Claude Code, Claude Desktop, or Cursor to Grafana over the Model Context Protocol (MCP). The agent can then search and read dashboards, run PromQL and LogQL through Grafana’s data sources, and read alert rules.

The Grafana MCP server reads the monitoring stack. To let an agent read Materialize’s own system catalog instead, see MCP Server for developer. An agent can use both.

Set up the server

  1. Create a service account token.

  2. Install uv ⧉, which downloads and runs mcp-grafana on demand. For the container image, release binaries, and Helm chart, see Set up the Grafana MCP server ⧉.

  3. Add the server to your agent. For Claude Code, run:

    claude mcp add-json grafana '{
      "command": "uvx",
      "args": ["mcp-grafana", "--disable-write"],
      "env": {
        "GRAFANA_URL": "<grafana-url>",
        "GRAFANA_SERVICE_ACCOUNT_TOKEN": "<service-account-token>"
      }
    }'
    

    For Claude Desktop, Cursor, and other agents, put the same command, args, and env in the agent’s MCP configuration.

Replace the following:

  • <grafana-url>: Grafana’s address, from terraform output -raw grafana_url. If you port-forward Grafana instead, use http://localhost:3000, and keep the port-forward running while the agent works.

  • <service-account-token>: the token you created. To keep it out of the agent’s configuration, set GRAFANA_SERVICE_ACCOUNT_TOKEN_FILE to the path of a file that holds it instead.

NOTE: mcp-grafana reports anonymous usage statistics to Grafana Labs by default: which tools ran and how the server is configured. The reports never include queries, log lines, credentials, or your Grafana’s address. To turn them off, add --usage-stats=disabled to args.
NOTE: Be sure to use a latest release of mcp-grafana. MCP servers continuously receive updates and improvements, so using the latest release ensures you have the newest features and bug fixes.

Keep the agent read-only

--disable-write removes the tools that create, update, or delete anything, such as dashboards and annotations, and limits the generic grafana_api_request tool to reads. Keep it on for investigation.

Log lines reach the agent as tool output, and a log line can carry text that reads like an instruction. With --disable-write, an agent that follows one cannot change anything through the server.

Grafana Operator re-pushes the Materialize dashboards from the released chart, so an agent’s edit to one does not last. To customize a dashboard, copy it to a new UID first.

What the agent can do

The following parts of mcp-grafana work against the bundled Grafana:

  • Dashboards: find, summarize, and read dashboards, including the queries behind each panel.
  • Prometheus: run PromQL against the Thanos data source, and list metrics and labels.
  • Loki: run LogQL against the Loki data source, list labels, and read stream statistics.
  • Alerting: list alert rules and their state.
  • Navigation: build a Grafana link to a dashboard, a panel, or an Explore query, so you can check the agent’s answer in a browser.

The tools that render a panel as an image need the Grafana Image Renderer, which the stack does not run, so they fail.

Give the agent context

The agent knows Grafana, not Materialize. Point it at the names the tools share and at the documentation it can read:

The following prompts are places to start:

  • “Open the mz-mon-env-top dashboard in Grafana and summarize the health of environment <environment>.”
  • “Which Materialize clusters have used more than 80% of their memory limit in the last 6 hours? Show me the PromQL you ran.”
  • “Find error lines from environmentd in the last hour, group them by message, and give me an Explore link for each group.”

Replace <environment> with the environment’s materialize_cloud_organization_name value.

Check the agent’s answers

An agent reads the same data you do, and can misread it in the same ways:

  • Ask for the query. A PromQL or LogQL expression, with its time range, is something you can run again. A summary is not.
  • Ask for a link, so you can see the panel the agent read.
  • Confirm a label before building on it. A query that filters on a label the series does not carry returns nothing, which looks healthy.
  • Don’t read quiet as healthy. The stack installs only some of the bundled alert rules by default, so the absence of firing alerts doesn’t mean everything is healthy.

Use gcx from the command line

gcx ⧉ is Grafana Labs’ command-line tool. It reaches dashboards, data sources, and alerting through Grafana’s API, so it needs a service account token and no Kubernetes access. An AI agent that can run shell commands can use it too.

  1. Install gcx, following Install gcx ⧉.

  2. Create a service account token.

  3. Log in, and name the connection mzmon:

    gcx login mzmon --server <grafana-url> --token <service-account-token>
    
  4. Set the stack’s data sources as the defaults, so queries don’t need a -d flag:

    gcx config set contexts.mzmon.datasources.prometheus mzmon-thanos
    gcx config set contexts.mzmon.datasources.loki mzmon-loki
    

Replace <grafana-url> and <service-account-token> as in Set up the server.

Then query metrics and logs, and read dashboards:

gcx --context mzmon metrics query 'up{namespace="materialize-environment"}' --since 1h
gcx --context mzmon logs query '{namespace="materialize-environment", app="environmentd"} |= "error"' --since 1h
gcx --context mzmon dashboards get mz-mon-env-top -o yaml

Replace materialize-environment with the namespace your Materialize environment runs in.

💡 Tip: Add --share-link to a query to print it as a Grafana Explore URL that you can send to someone working in a browser.
NOTE:

gcx switches to JSON output when an AI agent such as Claude Code or Cursor runs it. It also bundles agent skills that teach an agent its commands:

gcx agent skills install setup-gcx debug-with-grafana manage-dashboards

See also

Back to top ↑