Install on Azure
Self-managed Materialize requires: a Kubernetes (v1.29+) cluster; PostgreSQL as a metadata database; and blob storage.
The tutorial deploys Materialize to Azure Kubernetes Service (AKS) with a PostgreSQL database as the metadata database and Azure Blob Storage for blob storage. The tutorial uses Materialize on Azure Terraform modules to:
- Set up the Azure Kubernetes environment
- Call terraform-helm-materialize module to deploy Materialize Operator and Materialize instances to that AKS cluster
The Terraform modules used in this tutorial are intended for evaluation/demonstration purposes and for serving as a template when building your own production deployment. The modules should not be directly relied upon for production deployments: future releases of the modules will contain breaking changes. Instead, to use as a starting point for your own production deployment, either:
-
Fork the repo and pin to a specific version; or
-
Use the code as a reference when developing your own deployment.
For simplicity, this tutorial stores various secrets in a file as well as prints them to the terminal. In practice, refer to your organization’s official security and Terraform/infrastructure practices.
Prerequisites
Azure subscription
If you do not have an Azure subscription to use for this tutorial, create one.
Azure CLI
If you don’t have Azure CLI installed, install Azure CLI.
Terraform
If you don’t have Terraform installed, install Terraform.
kubectl
If you do not have kubectl
, install kubectl
.
Python (v3.12+) and pip
If you don’t have Python (v3.12 or greater) installed, install it. See
Python.org. If pip
is not included with
your version of Python, install it.
Helm 3.2.0+
If you don’t have Helm version 3.2.0+ installed, install. For details, see to the Helm documentation.
jq (Optional)
Optional. jq
is used to parse the AKS cluster name and region from the
Terraform outputs. Alternatively, you can manually specify the name and region.
If you want to use jq
and do not have jq
installed, install.
A. Authenticate with Azure
-
Open a Terminal window.
-
Authenticate with Azure.
az login
The command opens a browser window to sign in to Azure. Sign in.
-
Select the subscription and tenant to use. After you have signed in, back in the terminal, your tenant and subscription information is displayed.
Retrieving tenants and subscriptions for the selection... [Tenant and subscription selection] No Subscription name Subscription ID Tenant ----- ------------------- ------------------------------------ ---------------- [1]* ... ... ... The default is marked with an *; the default tenant is '<Tenant>' and subscription is '<Subscription Name>' (<Subscription ID>).
Select the subscription and tenant.
-
Set
ARM_SUBSCRIPTION_ID
to the subscription ID.export ARM_SUBSCRIPTION_ID=<subscription-id>
B. Set up Azure Kubernetes environment and install Materialize
The Terraform modules used in this tutorial are intended for evaluation/demonstration purposes and for serving as a template when building your own production deployment. The modules should not be directly relied upon for production deployments: future releases of the modules will contain breaking changes. Instead, to use as a starting point for your own production deployment, either:
-
Fork the repo and pin to a specific version; or
-
Use the code as a reference when developing your own deployment.
Materialize on Azure Terraform module for deploys a sample infrastructure on Azure with the following components:
Component | Version |
---|---|
AKS cluster | All |
Azure Blob Storage for persistence | All |
Azure Database for PostgreSQL Flexible Server for metadata storage | All |
Required networking and security configurations | All |
Managed identities with proper RBAC permissions | All |
Materialize Operator | All |
Materialize instances (Deployed during subsequent runs after the Operator is running) | All |
cert-manager and a self-signed ClusterIssuer .
ClusterIssuer is
deployed on subsequent runs after the cert-manager is running.
|
v0.3.0+ |
Load balancers for each Materialize instance | v0.3.1+ |
The tutorial uses the main.tf
found in the examples/simple/
directory,
which requires minimal user input. For details on the examples/simple/
infrastructure configuration (such as the node instance type, etc.), see the
examples/simple/main.tf.
For more configuration options, you can run the main.tf
file at the root of
the
repository
instead. When running with the root main.tf
, see Azure required
configuration.
Terraform version | Notable changes |
---|---|
v0.3.4 |
|
v0.3.1 |
|
v0.3.0 |
|
v0.2.0 |
|
v0.1.4 |
|
-
Open a Terminal window.
-
Fork the Materialize’s sample Terraform repo.
-
Set
MY_ORGANIZATION
to your github organization name, substituting your organization’s name for<enter-your-organization>
:MY_ORGANIZATION=<enter-your-organization>
-
Clone your forked repo and checkout the
v0.3.4
tag. For example,-
If cloning via SSH (substitute
YOUR_ORGANIZATION
with your organization’s name):git clone --depth 1 -b v0.3.4 git@github.com:${MY_ORGANIZATION}/terraform-azurerm-materialize.git
-
If cloning via HTTPS (substitute
YOUR_ORGANIZATION
with your organization’s name):git clone --depth 1 -b v0.3.4 https://github.com/${MY_ORGANIZATION}/terraform-azurerm-materialize.git
-
-
Go to the
examples/simple
folder in the Materialize Terraform repo directory.cd terraform-azurerm-materialize/examples/simple
💡 Tip:The tutorial uses the
main.tf
found in theexamples/simple/
directory, which requires minimal user input. For details on theexamples/simple/
infrastructure configuration (such as the node instance type, etc.), see the examples/simple/main.tf.For more configuration options, you can run the
main.tf
file at the root of the repository instead. When running with the rootmain.tf
, see Azure required configuration. -
Optional. Create a virtual environment, specifying a path for the new virtual environment:
python3 -m venv <path to the new virtual environment>
Activate the virtual environment:
source <path to the new virtual environment>/bin/activate
-
Install the required packages.
pip install -r requirements.txt
-
Create a
terraform.tfvars
file (you can copy from theterraform.tfvars.example
file) and specify:-
The prefix for the resources. Prefix has a maximum of 12 characters and contains only alphanumeric characters and hyphens; e.g.,
mydemo
. -
The location for the AKS cluster.
prefix="enter-prefix" // maximum 12 characters, containing only alphanumeric characters and hyphens; e.g. mydemo location="eastus2"
💡 Tip:The tutorial uses the
main.tf
found in theexamples/simple/
directory, which requires minimal user input. For details on theexamples/simple/
infrastructure configuration (such as the node instance type, etc.), see the examples/simple/main.tf.For more configuration options, you can run the
main.tf
file at the root of the repository instead. When running with the rootmain.tf
, see Azure required configuration. -
-
Initialize the terraform directory.
terraform init
-
Use terraform plan to review the changes to be made.
terraform plan
-
If you are satisfied with the changes, apply.
terraform apply
To approve the changes and apply, enter
yes
.Upon successful completion, various fields and their values are output:
Apply complete! Resources: 25 added, 0 changed, 0 destroyed. Outputs: aks_cluster = <sensitive> connection_strings = <sensitive> kube_config = <sensitive> load_balancer_details = {} resource_group_name = "mydemo-rg"
-
Configure
kubectl
to connect to your cluster:-
<cluster_name>
. Your cluster name has the form<your prefix>-aks
; e.g.,mz-simple-aks
. -
<resource_group_name>
, as specified in the output.
az aks get-credentials --resource-group <resource_group_name> --name <cluster_name>
Alternatively, you can use the following command to get the cluster name and resource group name from the Terraform output:
az aks get-credentials --resource-group $(terraform output -raw resource_group_name) --name $(terraform output -json aks_cluster | jq -r '.name')
To verify that you have configured correctly, run the following command:
kubectl cluster-info
For help with
kubectl
commands, see kubectl Quick reference. -
-
By default, the example Terraform installs the Materialize Operator and, starting in v0.3.0, a
cert-manager
. Verify the installation and check the status:Verify the installation and check the status:
kubectl get all -n materialize
Wait for the components to be in the
Running
state:NAME READY STATUS RESTARTS AGE pod/materialize-mydemo-materialize-operator-74d8f549d6-lkjjf 1/1 Running 0 36m
NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/materialize-mydemo-materialize-operator 1/1 1 1 36m
NAME DESIRED CURRENT READY AGE replicaset.apps/materialize-mydemo-materialize-operator-74d8f549d6 1 1 1 36m
Verify the installation and check the status:
kubectl get all -n cert-manager
Wait for the components to be in the
Running
state:NAME READY STATUS RESTARTS AGE pod/cert-manager-8576d99cc8-xqxbc 1/1 Running 0 4m22s pod/cert-manager-cainjector-664b5878d6-wc4tz 1/1 Running 0 4m22s pod/cert-manager-webhook-6ddb7bd6c5-vrm2p 1/1 Running 0 4m22s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/cert-manager ClusterIP 10.1.227.230 <none> 9402/TCP 4m22s service/cert-manager-cainjector ClusterIP 10.1.222.156 <none> 9402/TCP 4m22s service/cert-manager-webhook ClusterIP 10.1.84.207 <none> 443/TCP,9402/TCP 4m22s
NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/cert-manager 1/1 1 1 4m23s deployment.apps/cert-manager-cainjector 1/1 1 1 4m23s deployment.apps/cert-manager-webhook 1/1 1 1 4m23s
NAME DESIRED CURRENT READY AGE replicaset.apps/cert-manager-8576d99cc8 1 1 1 4m23s replicaset.apps/cert-manager-cainjector-664b5878d6 1 1 1 4m23s replicaset.apps/cert-manager-webhook-6ddb7bd6c5 1 1 1 4m23s
If you run into an error during deployment, refer to the Troubleshooting.
-
Once the Materialize operator is deployed and running, you can deploy the Materialize instances. To deploy Materialize instances, create a
mz_instances.tfvars
file with the Materialize instance configuration.For example, the following specifies the configuration for a
demo
instance.cat <<EOF > mz_instances.tfvars materialize_instances = [ { name = "demo" namespace = "materialize-environment" database_name = "demo_db" cpu_request = "1" memory_request = "2Gi" memory_limit = "2Gi" } ] EOF
Starting in v0.3.0, the Materialize on Azure Terraform module also deploys, by default, a self-signed
ClusterIssuer
. TheClusterIssuer
is deployed after thecert-manager
is deployed and running.Starting in v0.3.1, the Materialize on Azure Terraform module also deploys, by default, Load balancers for Materialize instances (i.e., the
create_load_balancer
flag defaults totrue
). The load balancers, by default, are configured to be internal (i.e., theinternal_load_balancer
flag defaults totrue
).💡 Tip:If upgrading from a deployment that was set up using an earlier version of the Terraform modules, additional considerations may apply when using an updated Terraform modules to your existing deployments.
See Materialize on Azure releases for notable changes.
-
Run
terraform plan
with both.tfvars
files and review the changes to be made.terraform plan -var-file=terraform.tfvars -var-file=mz_instances.tfvars
The plan should show the changes to be made, with a summary similar to the following:
Plan: 9 to add, 1 to change, 0 to destroy.
-
If you are satisfied with the changes, apply.
terraform apply -var-file=terraform.tfvars -var-file=mz_instances.tfvars
To approve the changes and apply, enter
yes
.Upon successful completion, you should see output with a summary similar to the following:
Apply complete! Resources: 9 added, 1 changed, 0 destroyed. Outputs: aks_cluster = <sensitive> connection_strings = <sensitive> kube_config = <sensitive> load_balancer_details = { "demo" = { "balancerd_load_balancer_ip" = "192.0.2.10" "console_load_balancer_ip" = "192.0.2.254" } } resource_group_name = "mydemo-rg"
-
Verify the installation and check the status:
kubectl get all -n materialize-environment
Wait for the components to be ready and in the
Running
state.NAME READY STATUS RESTARTS AGE pod/db-demo-db-l6ss8 0/1 Completed 0 2m21s pod/mz62lr3yltj8-balancerd-6d5dd6d4cf-r9nf4 1/1 Running 0 111s pod/mz62lr3yltj8-cluster-s2-replica-s1-gen-1-0 1/1 Running 0 114s pod/mz62lr3yltj8-cluster-u1-replica-u1-gen-1-0 1/1 Running 0 114s pod/mz62lr3yltj8-console-bfc797745-6nlwv 1/1 Running 0 96s pod/mz62lr3yltj8-console-bfc797745-tk9vm 1/1 Running 0 96s pod/mz62lr3yltj8-environmentd-1-0 1/1 Running 0 2m4s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/mz62lr3yltj8-balancerd ClusterIP None <none> 6876/TCP,6875/TCP 111s service/mz62lr3yltj8-balancerd-lb LoadBalancer 10.1.201.77 192.0.2.10 6875:30890/TCP,6876:31750/TCP 2m4s service/mz62lr3yltj8-cluster-s2-replica-s1-gen-1 ClusterIP None <none> 2100/TCP,2103/TCP,2101/TCP,2102/TCP,6878/TCP 114s service/mz62lr3yltj8-cluster-u1-replica-u1-gen-1 ClusterIP None <none> 2100/TCP,2103/TCP,2101/TCP,2102/TCP,6878/TCP 114s service/mz62lr3yltj8-console ClusterIP None <none> 8080/TCP 96s service/mz62lr3yltj8-console-lb LoadBalancer 10.1.130.212 192.0.2.254 8080:30379/TCP 2m4s service/mz62lr3yltj8-environmentd ClusterIP None <none> 6875/TCP,6876/TCP,6877/TCP,6878/TCP 111s service/mz62lr3yltj8-environmentd-1 ClusterIP None <none> 6875/TCP,6876/TCP,6877/TCP,6878/TCP 2m5s service/mz62lr3yltj8-persist-pubsub-1 ClusterIP None <none> 6879/TCP 2m4s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/mz62lr3yltj8-balancerd 1/1 1 1 111s deployment.apps/mz62lr3yltj8-console 2/2 2 2 96s NAME DESIRED CURRENT READY AGE replicaset.apps/mz62lr3yltj8-balancerd-6d5dd6d4cf 1 1 1 111s replicaset.apps/mz62lr3yltj8-console-bfc797745 2 2 2 96s NAME READY AGE statefulset.apps/mz62lr3yltj8-cluster-s2-replica-s1-gen-1 1/1 114s statefulset.apps/mz62lr3yltj8-cluster-u1-replica-u1-gen-1 1/1 114s statefulset.apps/mz62lr3yltj8-environmentd-1 1/1 2m4s NAME STATUS COMPLETIONS DURATION AGE job.batch/db-demo-db Complete 1/1 10s 2m21s
If you run into an error during deployment, refer to the Troubleshooting.
-
Open the Materialize Console in your browser:
Starting in v0.3.1, for each Materialize instance, Materialize on Azure Terraform module also deploys load balancers (by default, internal) with the following listeners, including a listener on port 8080 for the Materialize Console:
Port Description 6875 For SQL connections to the database 6876 For HTTP(S) connections to the database 8080 For HTTP(S) connections to Materialize Console The load balancer details are found in the
load_balancer_details
in the Terraform output.The example uses a self-signed ClusterIssuer. As such, you may encounter a warning with regards to the certificate. In production, run with certificates from an official Certificate Authority (CA) rather than self-signed certificates.
-
Find your console service name.
MZ_SVC_CONSOLE=$(kubectl -n materialize-environment get svc \ -o custom-columns="NAME:.metadata.name" --no-headers | grep console) echo $MZ_SVC_CONSOLE
-
Port forward the Materialize Console service to your local machine:1
( while true; do kubectl port-forward svc/$MZ_SVC_CONSOLE 8080:8080 -n materialize-environment 2>&1 | tee /dev/stderr | grep -q "portforward.go" && echo "Restarting port forwarding due to an error." || break; done; ) &
The command is run in background.
- To list the background jobs, usejobs
.
- To bring back to foreground, usefg %<job-number>
.
- To kill the background job, usekill %<job-number>
. -
Open a browser and navigate to https://localhost:8080 (or, if you have not enabled TLS, http://localhost:8080).
The example uses a self-signed ClusterIssuer. As such, you may encounter a warning with regards to the certificate. In production, run with certificates from an official Certificate Authority (CA) rather than self-signed certificates.
-
The port forwarding command uses a while loop to handle a known Kubernetes issue 78446, where interrupted long-running requests through a standard port-forward cause the port forward to hang. The command automatically restarts the port forwarding if an error occurs, ensuring a more stable connection. It detects failures by monitoring for “portforward.go” error messages. ↩︎
💡 Tip: If you experience long loading screens or unresponsiveness in the Materialize Console, we recommend increasing the size of themz_catalog_server
cluster. Refer to the Troubleshooting Console Unresponsiveness guide. -
Next steps
-
From the Console, you can get started with the Quickstart.
-
To start ingesting your own data from an external system like Kafka, MySQL or PostgreSQL, check the documentation for sources.
Cleanup
To delete the whole sample infrastructure and deployment (including the Materialize operator and Materialize instances and data), run from the Terraform directory:
terraform destroy
When prompted to proceed, type yes
to confirm the deletion.
terraform destroy
command is unable to delete the subnet because it
is in use, you can rerun the terraform destroy
command.
See also
- Materialize Operator Configuration
- Troubleshooting
- [Appendix: Azure deployment guidelines](/installation/install-on-azure/ appendix-deployment-guidelines)
- Installation