Observability tools
View as MarkdownThe monitoring stack that the Materialize Terraform modules install keeps metrics in Thanos and logs in Loki, and serves both through Grafana. You can read that data in a browser, through an AI agent, or from the command line. This page compares those tools and shows how to set each one up.
This page assumes the monitoring stack is already running and that you can reach Grafana. To enable the stack and sign in to Grafana, see Grafana.
Compare the tools
Every tool reads the same metrics, logs, and alerts. The tools differ in who they suit, what access they need, and how much they ask of you:
| Grafana | Grafana MCP server | gcx |
|
|---|---|---|---|
| Interface | Browser | AI agent, over MCP | Command line |
| Audience | Everyone who reads dashboards | Anyone who works with an AI agent | Scripts, and agents that run shell commands |
| Grafana access | Browser login | Service account token | Service account token |
| Ease of use | Highest: nothing to install | High after setup, but check its answers | Moderate: PromQL and LogQL |
All three reach the data through Grafana. None of them needs Kubernetes access unless you can’t reach Grafana’s address directly, for example because its load balancer is internal and you are outside the network. In that case, port-forward Grafana as Access Grafana describes.
Pick a tool by what you are doing:
- Reading a dashboard, or sharing a link: use Grafana in a browser.
- Asking questions in plain language, or having an AI agent investigate: use the Grafana MCP server.
- Scripting queries, or working from a terminal: use
gcx.
To query Thanos, Loki, or Alertmanager directly, for example while Grafana is down, see Connect existing tooling for the stores’ addresses, and Backend APIs ⧉ in the monitoring stack’s documentation for example queries.
Names the tools share
Scripts and AI agents reach the data through Grafana’s data sources, by UID:
| Data source | Type | UID | Holds |
|---|---|---|---|
| Thanos | prometheus |
mzmon-thanos |
Metrics |
| Loki | loki |
mzmon-loki |
Logs and Kubernetes events |
| Alertmanager | alertmanager |
mzmon-alertmanager |
Firing alerts and silences |
The following names also come up often:
-
Dashboard UIDs start with
mz-mon-. For example,mz-mon-env-topis the Materialize Environment Overview dashboard. -
Folder UIDs are
mzmon-materializefor the Materialize dashboards andmzmon-infrafor the cluster dashboards. -
The environment label,
materialize_cloud_organization_name, scopes Materialize metrics to one environment. Self-Managed deployments carry it too, despite its name.
Use Grafana in a browser
Grafana needs no setup beyond signing in. For its address and the admin
password, see Access Grafana.
Start from these places:
-
Dashboards: the Materialize folder holds the per-environment dashboards. Start with Materialize Environment Overview.
-
Explore: query the Thanos data source in PromQL, or the Loki data source in LogQL.
-
Alerting: Alerting > Silences shows and manages silences in the bundled Alertmanager. See Silence alerts.
A dashboard’s URL carries its time range and variables, so you can share what you see with anyone who can sign in.
Create a service account token
The Grafana MCP server and gcx sign in to Grafana with a service account
token rather than a password. To create one:
-
Sign in to Grafana as
admin. -
Go to Administration > Users and access > Service accounts, and click Add service account.
-
Enter a display name, such as
mcp-yourname, and click Create. -
Check that the service account’s role is Viewer, and change it with the role picker if it is not.
-
Click Add service account token, then Generate token. Copy the token, and store it somewhere safe.
Create one service account for each person or agent, so you can revoke one without the rest. The Viewer role is enough to read dashboards and run queries.
Connect an AI agent
The Grafana MCP server
⧉,
mcp-grafana, connects an AI agent such as Claude Code, Claude Desktop, or
Cursor to Grafana over the Model Context Protocol (MCP). The agent can then
search and read dashboards, run PromQL and LogQL through Grafana’s data sources,
and read alert rules.
The Grafana MCP server reads the monitoring stack. To let an agent read Materialize’s own system catalog instead, see MCP Server for developer. An agent can use both.
Set up the server
-
Install uv ⧉, which downloads and runs
mcp-grafanaon demand. For the container image, release binaries, and Helm chart, see Set up the Grafana MCP server ⧉. -
Add the server to your agent. For Claude Code, run:
claude mcp add-json grafana '{ "command": "uvx", "args": ["mcp-grafana", "--disable-write"], "env": { "GRAFANA_URL": "<grafana-url>", "GRAFANA_SERVICE_ACCOUNT_TOKEN": "<service-account-token>" } }'For Claude Desktop, Cursor, and other agents, put the same
command,args, andenvin the agent’s MCP configuration.
Replace the following:
-
<grafana-url>: Grafana’s address, fromterraform output -raw grafana_url. If you port-forward Grafana instead, usehttp://localhost:3000, and keep the port-forward running while the agent works. -
<service-account-token>: the token you created. To keep it out of the agent’s configuration, setGRAFANA_SERVICE_ACCOUNT_TOKEN_FILEto the path of a file that holds it instead.
mcp-grafana reports anonymous usage statistics to Grafana Labs by default:
which tools ran and how the server is configured. The reports never include
queries, log lines, credentials, or your Grafana’s address. To turn them off,
add --usage-stats=disabled to args.
mcp-grafana.
MCP servers continuously receive updates and improvements, so using the latest release ensures you have the newest features and bug fixes.
Keep the agent read-only
--disable-write removes the tools that create, update, or delete anything,
such as dashboards and annotations, and limits the generic grafana_api_request
tool to reads. Keep it on for investigation.
Log lines reach the agent as tool output, and a log line can carry text that
reads like an instruction. With --disable-write, an agent that follows one
cannot change anything through the server.
Grafana Operator re-pushes the Materialize dashboards from the released chart, so an agent’s edit to one does not last. To customize a dashboard, copy it to a new UID first.
What the agent can do
The following parts of mcp-grafana work against the bundled Grafana:
- Dashboards: find, summarize, and read dashboards, including the queries behind each panel.
- Prometheus: run PromQL against the Thanos data source, and list metrics and labels.
- Loki: run LogQL against the Loki data source, list labels, and read stream statistics.
- Alerting: list alert rules and their state.
- Navigation: build a Grafana link to a dashboard, a panel, or an Explore query, so you can check the agent’s answer in a browser.
The tools that render a panel as an image need the Grafana Image Renderer, which the stack does not run, so they fail.
Give the agent context
The agent knows Grafana, not Materialize. Point it at the names the tools share and at the documentation it can read:
llms.txt⧉ indexes the Materialize documentation, including this page.llms.txtfor the monitoring stack ⧉ indexes the monitoring stack’s own documentation: its dashboards, queries, and troubleshooting guides.
The following prompts are places to start:
- “Open the
mz-mon-env-topdashboard in Grafana and summarize the health of environment<environment>.” - “Which Materialize clusters have used more than 80% of their memory limit in the last 6 hours? Show me the PromQL you ran.”
- “Find error lines from
environmentdin the last hour, group them by message, and give me an Explore link for each group.”
Replace <environment> with the environment’s
materialize_cloud_organization_name value.
Check the agent’s answers
An agent reads the same data you do, and can misread it in the same ways:
- Ask for the query. A PromQL or LogQL expression, with its time range, is something you can run again. A summary is not.
- Ask for a link, so you can see the panel the agent read.
- Confirm a label before building on it. A query that filters on a label the series does not carry returns nothing, which looks healthy.
- Don’t read quiet as healthy. The stack installs only some of the bundled alert rules by default, so the absence of firing alerts doesn’t mean everything is healthy.
Use gcx from the command line
gcx ⧉ is
Grafana Labs’ command-line tool. It reaches dashboards, data sources, and
alerting through Grafana’s API, so it needs a service account token and no
Kubernetes access. An AI agent that can run shell commands can use it too.
-
Install
gcx, following Install gcx ⧉. -
Log in, and name the connection
mzmon:gcx login mzmon --server <grafana-url> --token <service-account-token> -
Set the stack’s data sources as the defaults, so queries don’t need a
-dflag:gcx config set contexts.mzmon.datasources.prometheus mzmon-thanos gcx config set contexts.mzmon.datasources.loki mzmon-loki
Replace <grafana-url> and <service-account-token> as in Set up the
server.
Then query metrics and logs, and read dashboards:
gcx --context mzmon metrics query 'up{namespace="materialize-environment"}' --since 1h
gcx --context mzmon logs query '{namespace="materialize-environment", app="environmentd"} |= "error"' --since 1h
gcx --context mzmon dashboards get mz-mon-env-top -o yaml
Replace materialize-environment with the namespace your Materialize
environment runs in.
--share-link to a query to print it as a Grafana Explore URL that you can
send to someone working in a browser.
gcx switches to JSON output when an AI agent such as Claude Code or Cursor
runs it. It also bundles agent skills that teach an agent its commands:
gcx agent skills install setup-gcx debug-with-grafana manage-dashboards
See also
- Grafana, for enabling and reaching the bundled Grafana.
- How logs and metrics are stored, for where the data lives and how long it is kept.
- Observability tools ⧉ in the monitoring stack’s documentation.