mz-debug self-managed

mz-debug self-managed debugs Kubernetes-based Materialize deployments. It collects:

Logs and resource information from pods, daemonsets, and other Kubernetes resources.
Snapshots of system catalog tables from your Materialize instance.

By default, the tool will automatically port-forward to collect system catalog information. You can disable this by setting --auto-port-forward false and specifying your own connection URL via --mz-connection-url.

Requirements

mz-debug requires kubectl v1.32.3+. Install kubectl if you do not have it installed.

Syntax

mz-debug self-managed [OPTIONS]

Options

`mz-debug self-managed` options

Option	Description
`--dump-k8s`	If `true`, dump debug information from the Kubernetes cluster. Defaults to `true`.
`--k8s-namespace <NAMESPACE>`	The namespaces to dump. Specify multiple times to dump multiple namespaces. Required if `--dump-k8s` is true.
`--k8s-context <CONTEXT>`	The Kubernetes context to use. Defaults to the `KUBERNETES_CONTEXT` environment variable.
`--k8s-dump-secret-values`	If `true`, include unredacted secrets in the dump. Use with caution. Defaults to `false`.
`--auto-port-forward`	If `true`, automatically port-forward the external SQL port. Defaults to `true`.
`--port-forward-local-address <IP address>`	The address to listen on for port-forwarding. Defaults to `127.0.0.1`.
`--port-forward-local-port <PORT>`	The port to listen on for port-forwarding. Defaults to `6875`.
`-h`, `--help`	Print help information.

`mz-debug` global options

Option	Description
`--dump-heap-profiles <boolean>`	If `true`, dump heap profiles (.pprof.gz) from your Materialize instance. Defaults to `true`.
`--dump-prometheus-metrics <boolean>`	If `true`, dump prometheus metrics from your Materialize instance. Defaults to `true`.
`--dump-system-catalog <boolean>`	If `true`, dump the system catalog from your Materialize instance. Defaults to `true`.
`--mz-connection-url <URL>`	The Materialize instance’s PostgreSQL connection URL. Defaults to `postgres://127.0.0.1:6875/materialize?sslmode=prefer`.

Output

The mz-debug outputs its log file (tracing.log) and the generated debug files into a directory named mz_debug_YYYY-MM-DD-HH-TMM-SSZ/ as well as zips the directory and its contents mz_debug_YYYY-MM-DD-HH-TMM-SSZ.zip.

The generated debug files are in two main categories: Kubernetes resource files and system catalog files.

Kubernetes resource files

Under mz_debug_YYYY-MM-DD-HH-TMM-SSZ/, the following Kubernetes resource debug files are generated:

Resource Type	Files
Workloads	`pods/{namespace}/.yaml` `logs/{namespace}/{pod}.current.log` `logs/{namespace}/{pod}.previous.log` `deployments/{namespace}/.yaml` `statefulsets/{namespace}/.yaml` `replicasets/{namespace}/.yaml` `events/{namespace}/.yaml` `materializes/{namespace}/.yaml`
Networking	`services/{namespace}/.yaml` `networkpolicies/{namespace}/.yaml` `certificates/{namespace}/*.yaml`
Storage	`persistentvolumes/.yaml` `persistentvolumeclaims/{namespace}/.yaml` `storageclasses/*.yaml`
Configuration	`roles/{namespace}/.yaml` `rolebinding/{namespace}/.yaml` `configmaps/{namespace}/.yaml` `secrets/{namespace}/.yaml` `serviceaccounts/{namespace}/*.yaml`
Cluster-level	`nodes/.yaml` `daemonsets/.yaml` `mutatingwebhookconfigurations/{namespace}/.yaml` `validatingwebhookconfigurations/{namespace}/.yaml` `customresourcedefinitions/*.yaml`

Each resource type directory also contains a describe.txt file with the output of kubectl describe for that resource type.

System catalog files

mz-debug outputs system catalog files if --dump-system-catalog is true (the default).

The generated files are in system-catalog sub-directory as *.csv files and contains:

Core catalog object definitions
Cluster and compute-related information
Data freshness and frontier metric
Source and sink metrics
Per-replica introspection metrics (under {cluster_name}/{replica_name}/*.csv)

For more information about each relation, view the system catalog.

Prometheus metrics

mz-debug outputs snapshots of prometheus metrics per service (i.e. environmentd) if --dump-prometheus-metrics is true (the default). Each file is stored under prom_metrics/{service}.txt.

Memory profiles

By default, mz-debug outputs heap profiles for each service in profiles/{service}.memprof.pprof.gz. To turn off this behavior, you can set --dump-heap-profiles to false.

Examples

Debug the `materialize-environment` namespace

mz-debug self-managed --k8s-namespace materialize-environment

Debug the `materialize` namespace

mz-debug --dump-system-catalog=false self-managed --k8s-namespace materialize --auto-port-forward false

Debug namespaces without automatic port-forwarding

mz-debug self-managed \
    --k8s-namespace materialize \
    --k8s-namespace materialize-environment \
    --auto-port-forward false \
    --mz-connection-url 'postgres://root@127.0.0.1:6875/materialize?sslmode=disable'

Because the materialize namespace does not contain Materialize instances, the debug tool will be unable to dump the system catalog.