mz-debug self-managed

mz-debug self-managed debugs Kubernetes-based Materialize deployments. It collects:

Logs and resource information from pods, daemonsets, and other Kubernetes resources.
Snapshots of system catalog tables from your Materialize instance.

By default, the tool will automatically port-forward to collect system catalog information. You can disable this by specifying your own connection URL via --mz-connection-url.

Requirements

mz-debug requires kubectl v1.32.3+. Install kubectl if you do not have it installed.

Syntax

mz-debug self-managed [OPTIONS]

Options

`mz-debug self-managed` options

Option	Description
`--k8s-namespace <NAMESPACE>`	Required. The Kubernetes namespace of the Materialize instance.
`--mz-instance-name <MZ_INSTANCE_NAME>`	Required. The Materialize instance to target.
`--dump-k8s <boolean>`	If `true`, dump debug information from the Kubernetes cluster. Defaults to `true`.
`--additional-k8s-namespace <NAMESPACE>`	Additional k8s namespaces to dump.
`--k8s-context <CONTEXT>`	The Kubernetes context to use. Defaults to the `KUBERNETES_CONTEXT` environment variable.
`--k8s-dump-secret-values <boolean>`	If `true`, include unredacted secrets in the dump. Use with caution. Defaults to `false`.
`-h`, `--help`	Print help information.

`mz-debug` global options

Option	Description
`--dump-heap-profiles <boolean>`	If `true`, dump heap profiles (.pprof.gz) from your Materialize instance. Defaults to `true`.
`--dump-prometheus-metrics <boolean>`	If `true`, dump prometheus metrics from your Materialize instance. Defaults to `true`.
`--dump-system-catalog <boolean>`	If `true`, dump the system catalog from your Materialize instance. Defaults to `true`.
`--mz-username <USERNAME>`	The username to use to connect to Materialize. Can also be set via the `MZ_USERNAME` environment variable.
`--mz-password <PASSWORD>`	The password to use to connect to Materialize if password authentication is enabled. Can also be set via the `MZ_PASSWORD` environment variable.
`--mz-connection-url <URL>`	The Materialize instance’s PostgreSQL connection URL. Defaults to `postgres://127.0.0.1:6875/materialize?sslmode=prefer`.

Output

The mz-debug outputs its log file (tracing.log) and the generated debug files into a directory named mz_debug_YYYY-MM-DD-HH-TMM-SSZ/ as well as zips the directory and its contents mz_debug_YYYY-MM-DD-HH-TMM-SSZ.zip.

The generated debug files are in two main categories: Kubernetes resource files and system catalog files.

Kubernetes resource files

Under mz_debug_YYYY-MM-DD-HH-TMM-SSZ/, the following Kubernetes resource debug files are generated:

Resource Type	Files
Workloads	`pods/{namespace}/.yaml` `logs/{namespace}/{pod}.current.log` `logs/{namespace}/{pod}.previous.log` `deployments/{namespace}/.yaml` `statefulsets/{namespace}/.yaml` `replicasets/{namespace}/.yaml` `events/{namespace}/.yaml` `materializes/{namespace}/.yaml`
Networking	`services/{namespace}/.yaml` `networkpolicies/{namespace}/.yaml` `certificates/{namespace}/*.yaml`
Storage	`persistentvolumes/.yaml` `persistentvolumeclaims/{namespace}/.yaml` `storageclasses/*.yaml`
Configuration	`roles/{namespace}/.yaml` `rolebinding/{namespace}/.yaml` `configmaps/{namespace}/.yaml` `secrets/{namespace}/.yaml` `serviceaccounts/{namespace}/*.yaml`
Cluster-level	`nodes/.yaml` `daemonsets/.yaml` `mutatingwebhookconfigurations/{namespace}/.yaml` `validatingwebhookconfigurations/{namespace}/.yaml` `customresourcedefinitions/*.yaml`

Each resource type directory also contains a describe.txt file with the output of kubectl describe for that resource type.

System catalog files

mz-debug outputs system catalog files if --dump-system-catalog is true (the default).

The generated files are in system-catalog sub-directory as *.csv files and contains:

Core catalog object definitions
Cluster and compute-related information
Data freshness and frontier metric
Source and sink metrics
Per-replica introspection metrics (under {cluster_name}/{replica_name}/*.csv)

For more information about each relation, view the system catalog.

Prometheus metrics

mz-debug outputs snapshots of prometheus metrics per service (i.e. environmentd) if --dump-prometheus-metrics is true (the default). Each file is stored under prom_metrics/{service}.txt.

Memory profiles

By default, mz-debug outputs heap profiles for each service in profiles/{service}.memprof.pprof.gz. To turn off this behavior, you can set --dump-heap-profiles to false.

Prerequisite: Get the Materialize instance name

To use mz-debug, you need to specify the Kubernetes namespace (--k8s-namespace) and the Materialize instance name (--mz-instance-name). To retrieve the Materialize instance name, you can use kubectl. For example, the following retrieves the name of the Materialize instance(s) running in the Kubernetes namespace materialize-environment:

kubectl --namespace materialize-environment get materializes.materialize.cloud

The command should return the NAME of the Materialize instance(s) in the namespace:

NAME
12345678-1234-1234-1234-123456789012

Examples

Debug a Materialize instance running in a namespace

The following example uses mz-debug to collect debug information for the Materialize instance (12345678-1234-1234-1234-123456789012 obtained in the Prerequisite) running in the Kubernetes namespace materialize-environment:

mz-debug self-managed --k8s-namespace materialize-environment \
--mz-instance-name 12345678-1234-1234-1234-123456789012

Include information from additional kubernetes namespaces

When debugging a Materialize instance, you can also include information from other namespaces via --additional-k8s-namespace. The following example collects debug information for the Materialize instance running in the Kubernetes namespace materialize-environment as well as debug information for the namespace materialize:

mz-debug self-managed --k8s-namespace materialize-environment \
--mz-instance-name 12345678-1234-1234-1234-123456789012 \
--additional-k8s-namespace materialize