Manage cloud resources

The Terraform modules below provide the cloud infrastructure foundation Materialize needs to communicate with components outside of Materialize itself. The Materialize provider allows users to manage Materialize resources in the same programmatic way.

You can use the modules to establish the underlying cloud resources and then use the Materialize provider to build Materialize-specific objects. A few use cases are captured in the sections below.

EC2 SSH bastion host

Another method for source connection is to use a bastion host to allow SSH communication to and from Materialize.

The EC2 SSH bastion module allows you to configure an EC2 instance with security groups and an SSH keypair. These components form the foundation you need to have a secure, centralized access point between your data source and Materialize.

After using the module, you can configure the materialize_connection_ssh_tunnel resource with the module output, allowing Materialize an end-to-end connection to your source. The provider will configure the same Materialize objects as the CREATE CONNECTION statement.

Amazon RDS for PostgreSQL

You can also create an RDS instance from which you can track and propagate changes. The AWS RDS Postgres module creates a VPC, security groups, and an RDS instance in AWS. You can use these AWS components to create a database with data you want to process in Materialize.

After you run the module, you can create a secret, connection, and source with the Materialize provider for an end-to-end connection to this instance as a new source. The Materialize provider will create these objects just like the CREATE SECRET, CREATE CONNECTION, and CREATE SOURCE statements in SQL. The secret, connection, and source resources would be similar to the example Terraform configuration below with output from the module:

resource "materialize_secret" "example_secret" {
  name  = "secret"
  value = <RDSpassword>
}

resource "materialize_connection_postgres" "example_postgres_connection" {
  name = "example_postgres_connection"
  host = <RDShostname>
  port = 5432
  user {
    secret {
      name          = "example"
      database_name = "database"
      schema_name   = "schema"
    }
  }
  password {
    name          = "example"
    database_name = "database"
    schema_name   = "schema"
  }
  database = "example"
}

resource "materialize_source_postgres" "example_source_postgres" {
  name        = "source_postgres"
  schema_name = "schema"
  cluster_name = "quickstart"
  postgres_connection {
    name = "pg_connection"
    # Optional parameters
    # database_name = "postgres"
    # schema_name = "public"
  }
  publication = "mz_source"
  table = {
    "schema1.table_1" = "s1_table_1"
    "schema2_table_1" = "s2_table_1"
  }
}