This tutorial shows how to write messages from your
Managed Service for Apache Kafka cluster to Cloud Storage using Kafka
Connect.

Kafka Connect manages data movement between your Kafka cluster and other
systems. In this tutorial, you create a
[Connect cluster](https://docs.cloud.google.com/managed-service-for-apache-kafka/docs/connect-cluster/create-connect-cluster.md.txt)
and a [Cloud Storage Sink connector](https://docs.cloud.google.com/managed-service-for-apache-kafka/docs/connect-cluster/create-cloud-storage-connector.md.txt).
The Cloud Storage Sink connector reads messages from Kafka and writes
them to a Cloud Storage bucket.

## Before you begin

### Console


1.

   Make sure that you have the following role or roles on the project:


   **Managed Kafka Cluster Editor** ,
   **Managed Kafka Connect Cluster Editor** ,
   **Managed Kafka Connector Editor** ,
   **Managed Kafka Topic Editor** ,
   **Storage Admin**

   #### Check for the roles

   1.
      In the Google Cloud console, go to the **IAM** page.

      [Go to IAM](https://console.cloud.google.com/projectselector/iam-admin/iam?supportedpurview=project)
   2. Select the project.
   3.
      In the **Principal** column, find all rows that identify you or a group that
      you're included in. To learn which groups you're included in, contact your
      administrator.

   4. For all rows that specify or include you, check the **Role** column to see whether the list of roles includes the required roles.

   #### Grant the roles

   1.
      In the Google Cloud console, go to the **IAM** page.

      [Go to IAM](https://console.cloud.google.com/projectselector/iam-admin/iam?supportedpurview=project)
   2. Select the project.
   3. Click **Grant access**.
   4.
      In the **New principals** field, enter your user identifier.

      This is typically the email address for a Google Account.

   5. Click **Select a role**, then search for the role.
   6. To grant additional roles, click **Add
      another role** and add each additional role.
   7. Click **Save**.

<br />

### gcloud


1.
   Grant roles to your user account. Run the following command once for each of the following
   IAM roles:
   `
   roles/managedkafka.clusterEditor,
   roles/managedkafka.connectClusterEditor,
   roles/managedkafka.connectorEditor,
   roles/managedkafka.topicEditor,
   roles/storage.admin
   `

   ```bash
   gcloud projects add-iam-policy-binding PROJECT_ID --member="user:USER_IDENTIFIER" --role=ROLE
   ```

   Replace the following:
   - `PROJECT_ID`: Your project ID.
   - `USER_IDENTIFIER`: The identifier for your user account. For example, `myemail@example.com`.
   - `ROLE`: The IAM role that you grant to your user account.

<br />

## Create a Cloud Storage bucket

In this step, you create a Cloud Storage bucket. This bucket stores the data
that the Cloud Storage Sink connector reads from Kafka.

### Console

1. Go to the **Cloud Storage** \> **Buckets** page.


   [Go to Buckets](https://console.cloud.google.com/storage/browser)

   <br />

2. Click **Create**.

3. On the **Create a bucket** page, enter a name that meets the
   [bucket naming requirements](https://docs.cloud.google.com/storage/docs/buckets.md.txt#naming).
   Cloud Storage bucket names must be globally unique.

4. Click **Create**.

### gcloud

To create a Cloud Storage bucket, run the
[`gcloud storage buckets create`](https://docs.cloud.google.com/sdk/gcloud/reference/storage/buckets/create.md.txt)
command.

```bash
gcloud storage buckets create gs://BUCKET_NAME --location=BUCKET_LOCATION
```

Replace the following:

- <var translate="no">BUCKET_NAME</var>: a name for your Cloud Storage
  bucket that meets the
  [bucket naming requirements](https://docs.cloud.google.com/storage/docs/buckets.md.txt#naming).
  Cloud Storage bucket names must be globally unique.

- <var translate="no">BUCKET_LOCATION</var>: the
  [location](https://docs.cloud.google.com/storage/docs/locations.md.txt#available-locations) for the bucket. For
  example, `US`.

## Create Managed Service for Apache Kafka resources

In this section, you create the following Managed Service for Apache Kafka
resources:

- A Kafka cluster with a topic.
- A Connect cluster with connectors a Cloud Storage Sink connector.

### Create a Kafka cluster

In this step, you create a
[Managed Service for Apache Kafka cluster](https://docs.cloud.google.com/managed-service-for-apache-kafka/docs/create-cluster.md.txt).
Creating a cluster can take up to 30 minutes.

### Console

1. Go to the **Managed Service for Apache Kafka** \> **Clusters** page.

   [Go to Clusters](https://console.cloud.google.com/managedkafka/clusters)
2. Click **Create**.
3. In the **Cluster name** box, enter a name for the cluster.
4. In the **Region** list, select a location for the cluster.
5. For **Network configuration** , configure the subnet where the cluster is accessible:
   1. For **Project**, select your project.
   2. For **Network**, select the VPC network.
   3. For **Subnet**, select the subnet.
   4. Click **Done**.
6. Click **Create**.

After you click **Create** , the cluster state is `Creating`. When the cluster
is ready, the state is `Active`.

### gcloud


To create a Kafka cluster, run the
[`managed-kafka clusters
create`](https://docs.cloud.google.com/sdk/gcloud/reference/managed-kafka/clusters/create) command.

```bash
gcloud managed-kafka clusters create KAFKA_CLUSTER \
--location=REGION \
--cpu=3 \
--memory=3GiB \
--subnets=projects/PROJECT_ID/regions/REGION/subnetworks/SUBNET_NAME \
--async
```

Replace the following:

- `KAFKA_CLUSTER`: a name for the Kafka cluster
- `REGION`: the location of the cluster
- `PROJECT_ID`: your project ID
- `SUBNET_NAME`: the subnet where you want to create the cluster, for example `default`


For information about supported locations, see
[Managed Service for Apache Kafka locations](https://docs.cloud.google.com/managed-service-for-apache-kafka/docs/locations.md.txt).

The command runs asynchronously and returns an operation ID:

```bash
Check operation [projects/PROJECT_ID/locations/REGION/operations/OPERATION_ID] for status.
```


To track the progress of the create operation, use the
[`gcloud managed-kafka
operations describe`](https://docs.cloud.google.com/sdk/gcloud/reference/managed-kafka/operations/describe) command:

```bash
gcloud managed-kafka operations describe OPERATION_ID \
  --location=REGION
```


When the cluster is ready, the output from this command includes the entry `state:
ACTIVE`. For more information, see
[Monitor the
cluster creation operation](https://docs.cloud.google.com/managed-service-for-apache-kafka/docs/create-cluster#monitor_cluster).

### Create a Kafka topic

After the Managed Service for Apache Kafka cluster is created, create a
Kafka topic.

### Console

1. Go to the **Managed Service for Apache Kafka** \>
   **Clusters** page.


   [Go to Clusters](https://console.cloud.google.com/managedkafka/clusters)

   <br />

2. Click the name of the cluster.

3. In the cluster details page, click
   **Create Topic**.

4. In the **Topic name** box, enter a name for the topic.

5. Click **Create**.

### gcloud

To create a Kafka topic, run the
[`managed-kafka topics create`](https://docs.cloud.google.com/sdk/gcloud/reference/managed-kafka/topics/create)
command.

```bash
gcloud managed-kafka topics create KAFKA_TOPIC_NAME \
--cluster=KAFKA_CLUSTER \
--location=REGION \
--partitions=10 \
--replication-factor=3
```

Replace the following:

- `KAFKA_TOPIC_NAME`: the name of the Kafka topic to create
- `KAFKA_CLUSTER`: the name of the Kafka cluster
- `REGION`: the region where you created the Kafka cluster

### Create a Connect cluster

In this step, you create a Connect cluster. Creating a Connect cluster can take
up to 30 minutes.

Before you start this step, make sure the Managed Service for Apache Kafka
cluster is fully created.

### Console

1. Go to the **Managed Service for Apache Kafka** \>
   **Connect Clusters** page.


   [Go to Connect Clusters](https://console.cloud.google.com/managedkafka/connectClusters)

   <br />

2. Click **Create**.

3. For the **Connect cluster name** , enter a string. Example:
   `my-connect-cluster`.

4. For **Primary Kafka cluster**, select the Kafka that you created earlier.

5. Click **Create**.

While the cluster is being created, the cluster state is `Creating`. When the
cluster has finished being created, the state is `Active`.

### gcloud

To create a Connect cluster, run the
[`gcloud managed-kafka connect-clusters create`](https://docs.cloud.google.com/sdk/gcloud/reference/alpha/managed-kafka/connect-clusters/create.md.txt)
command.

```bash
gcloud managed-kafka connect-clusters create CONNECT_CLUSTER \
  --location=REGION \
  --cpu=12 \
  --memory=12GiB \
  --primary-subnet=projects/PROJECT_ID/regions/REGION/subnetworks/SUBNET_NAME \
  --kafka-cluster=KAFKA_CLUSTER \
  --async
```

Replace the following:

- `CONNECT_CLUSTER`: a name for the Connect cluster
- `REGION`: the region where you created the Kafka cluster
- `PROJECT_ID`: your project ID
- `SUBNET_NAME`: the subnet where you created the Kafka cluster
- `KAFKA_CLUSTER`: the name of your Kafka cluster

The command runs asynchronously and returns an operation ID:

```bash
Check operation [projects/PROJECT_ID/locations/REGION/operations/OPERATION_ID] for status.
```

To track the progress of the create operation, use the
[`gcloud managed-kafka operations describe`](https://docs.cloud.google.com/sdk/gcloud/reference/managed-kafka/operations/describe)
command:

```bash
gcloud managed-kafka operations describe OPERATION_ID \
  --location=REGION
```

For more information, see
[Monitor the cluster creation operation](https://docs.cloud.google.com/managed-service-for-apache-kafka/docs/connect-cluster/create-connect-cluster.md.txt#monitor_cluster).

### Grant IAM roles

Grant the **Storage Object Creator** Identity and Access Management (IAM) role to the
Managed Kafka service account. This role allows the connector to write files to
Cloud Storage.

### Console

1. In the Google Cloud console, go to the **IAM** page.


   [Go to IAM](https://console.cloud.google.com/iam-admin/iam)

   <br />

2. Select **Include Google-provided role grants**.

3. Find the row for **Managed Kafka Service Account** and click
   **Edit principal**.

4. Click **Add another role** and select the **Storage Object Creator** role.

5. Click **Save**.

For more information about granting roles, see
[Grant an IAM role by using the console](https://docs.cloud.google.com/iam/docs/grant-role-console.md.txt).

### gcloud

To grant an IAM role to the service account, run the
[`gcloud projects add-iam-policy-binding`](https://docs.cloud.google.com/sdk/gcloud/reference/projects/add-iam-policy-binding)
command.

```bash
gcloud projects add-iam-policy-binding PROJECT_ID \
    --member=serviceAccount:service-PROJECT_NUMBER@gcp-sa-managedkafka.iam.gserviceaccount.com \
    --role=roles/storage.objectCreator
```

Replace the following:

- `PROJECT_ID`: your project ID
- `PROJECT_NUMBER`: your project number

To find your project number, use the
[`gcloud projects describe`](https://docs.cloud.google.com/sdk/gcloud/reference/projects/describe)
command.

### Create a Cloud Storage Sink connector

In this step, you create a
[Cloud Storage Sink connector](https://docs.cloud.google.com/managed-service-for-apache-kafka/docs/connect-cluster/create-pubsub-source-connector.md.txt).
This connector reads messages from one or more Kafka topics and writes them to
Cloud Storage.

### Console

1. Go to the **Managed Service for Apache Kafka** \>
   **Connect Clusters** page.


   [Go to Connect Clusters](https://console.cloud.google.com/managedkafka/connectClusters)

   <br />

2. Click the name of the Connect cluster.

3. Click **Create connector**.

4. For the **Connector name** , enter a string. Example: `pubsub-source`.

5. In the **Connector plugin** list, select `Cloud Storage Source`.

6. For **Topics** , select Kafka topic that you created previously and click
   **OK**.

7. For **GCS Bucket** , click **Browse**.

8. Select the Cloud Storage bucket that you created previously and
   click **Select**.

9. Click **Create**.

### gcloud

To create a Cloud Storage Sink connector, run the
[`gcloud managed-kafka connectors create`](https://docs.cloud.google.com/sdk/gcloud/reference/managed-kafka/connectors/create)
command.

```bash
gcloud managed-kafka connectors create STORAGE_CONNECTOR_NAME \
  --location=REGION \
  --connect-cluster=CONNECT_CLUSTER \
  --configs=connector.class=io.aiven.kafka.connect.gcs.GcsSinkConnector,\
file.name.prefix=,\
format.output.type=json,\
gcs.bucket.name=BUCKET_NAME,\
gcs.credentials.default=true,\
key.converter=org.apache.kafka.connect.storage.StringConverter,\
tasks.max=3,\
topics=KAFKA_TOPIC_NAME,\
value.converter=org.apache.kafka.connect.json.JsonConverter,\
value.converter.schemas.enable=false
```

Replace the following:

- `STORAGE_CONNECTOR_NAME`: a name for the connector, such as `storage-sink-connector`
- `CONNECT_CLUSTER`: the name of your Connect cluster
- `REGION`: the region where you created the Connect cluster
- `PROJECT_ID`: your project ID
- `BUCKET_NAME`: the name of your Cloud Storage bucket
- `KAFKA_TOPIC_NAME`: the name of your Kafka topic

## View results

To view the results, send messages to the Kafka topic. There are various ways to
send messages to Managed Service for Apache Kafka, including:

- [Use the Kafka command-line tools](https://docs.cloud.google.com/managed-service-for-apache-kafka/docs/quickstart.md.txt).
- [Use a Java producer](https://docs.cloud.google.com/managed-service-for-apache-kafka/docs/quickstart-java.md.txt).
- [Use a Python producer](https://docs.cloud.google.com/managed-service-for-apache-kafka/docs/quickstart-python.md.txt)
- [Stream messages from Pub/Sub by using Kafka Connect](https://docs.cloud.google.com/managed-service-for-apache-kafka/docs/connect-cluster/kafka-connect-pubsub-to-gcs)

The Cloud Storage Sink connector reads the Kafka messages and writes
them to files in Cloud Storage. It might take a few minutes for the
first file to be written.

To view the output files, perform the following steps.

### Console

1. Go to the **Cloud Storage** \> **Buckets** page.


[Go to Buckets](https://console.cloud.google.com/storage/browser)

<br />

1. In the list of buckets, click the name of the bucket that you created
   earlier.

2. If the Cloud Storage Sink connector successfully wrote files, they
   are listed in the **Bucket details** page. Select one of more files.

3. Click **Download** to download the selected files.

### gcloud

To list the output files, run the
[`gcloud storage ls`](https://docs.cloud.google.com/sdk/gcloud/reference/storage/ls) command:

```bash
gcloud storage ls gs://BUCKET_NAME
```

To download the files, run the
[`gcloud storage cp`](https://docs.cloud.google.com/sdk/gcloud/reference/storage/cp) command.

```bash
gcloud storage cp --recursive gs://BUCKET_NAME .
```

Replace <var translate="no">BUCKET_NAME</var> with the name of your
Cloud Storage bucket.

## Clean up


To avoid incurring charges to your Google Cloud account for the resources used in this
tutorial, either delete the project that contains the resources, or keep the project and
delete the individual resources.

### Console

1. Delete the Cloud Storage bucket.

   1. Go to the **Cloud Storage** \> **Buckets** page.


      [Go to Buckets](https://console.cloud.google.com/storage)

      <br />

   2. Select the bucket and click **Delete**.

2. Delete the Connect cluster.

   1. Go to the **Managed Service for Apache Kafka** \>
      **Connect Clusters** page.


      [Go to Connect Clusters](https://console.cloud.google.com/managedkafka/connectClusters)

      <br />

   2. Select the Connect cluster and click **Delete**.

3. Delete the Kafka cluster.

   1. Go to the **Managed Service for Apache Kafka** \>
      **Clusters** page.


      [Go to Clusters](https://console.cloud.google.com/managedkafka/clusters)

      <br />

   2. Select the Kafka cluster and click **Delete**.

### gcloud

1. To delete the Cloud Storage bucket and its objects, use the
   [`gcloud storage rm`](https://docs.cloud.google.com/sdk/gcloud/reference/storage/rm) command.

   ```bash
   gcloud storage rm gs://BUCKET_NAME --recursive
   ```
2. To delete the Connect cluster, use the
   [`gcloud managed-kafka connect-clusters delete`](https://docs.cloud.google.com/sdk/gcloud/reference/managed-kafka/connect-clusters/delete)
   command.

   ```bash
   gcloud managed-kafka connect-clusters delete CONNECT_CLUSTER \
     --location=REGION --async
   ```
3. To delete the Kafka cluster, use the
   [`gcloud managed-kafka clusters delete`](https://docs.cloud.google.com/sdk/gcloud/reference/managed-kafka/clusters/delete)
   command.

   ```bash
   gcloud managed-kafka clusters delete KAFKA_CLUSTER \
     --location=REGION --async
   ```

## What's next

- [Troubleshoot a Cloud Storage Sink connector](https://docs.cloud.google.com/managed-service-for-apache-kafka/docs/connect-cluster/troubleshoot-cloud-storage-connector.md.txt).
- Learn more about the [Cloud Storage Sink connector](https://docs.cloud.google.com/managed-service-for-apache-kafka/docs/connect-cluster/create-cloud-storage-connector.md.txt).
- Learn more about [Kafka Connect](https://docs.cloud.google.com/managed-service-for-apache-kafka/docs/kafka-connect-overview.md.txt).