# Manage Data Access audit log costs

> [!IMPORTANT]
> **Key Point:** To prevent unexpected high costs, restrict service account logging.

Bigtable is typically used for large, high-volume workloads. As a
result, if you don't manage the log volume, Bigtable can generate
an **extremely high number** of DATA_READ and DATA_WRITE logs, leading to
unexpectedly high [log storage costs](https://docs.cloud.google.com/stackdriver/pricing#logging-costs). If you use Data Access
audit logging, you should take steps to manage the log volume.

When you follow the best practices for [Bigtable authentication](https://docs.cloud.google.com/bigtable/docs/authentication), most Data Access audit log activity is generated by *service
accounts* . A [service account](https://docs.cloud.google.com/iam/docs/service-accounts) is an account that an
application uses to authenticate and make API calls to Google Cloud
services such as Bigtable. Managing service account logs is the
most important step to reduce log volume. You might want to also limit logs
using other criteria.

You can enable Data Access audit logging for Bigtable in the
following ways:

- Using the Google Cloud console
  - [Individual services](https://docs.cloud.google.com/logging/docs/audit/configure-data-access#config-console) (for example, only Bigtable)
  - [Default config](https://docs.cloud.google.com/logging/docs/audit/configure-data-access#config-console-default) all services in a Google Cloud project (not just Bigtable)
- [Using the Cloud Logging API](https://docs.cloud.google.com/logging/docs/audit/configure-data-access#config-api)

After you enable audit logging, take the following steps to restrict the volume
of logs.

## Identify service accounts

First, identify the service accounts that you don't need logs for. The list of
service accounts that are not useful and should not be logged depends on your
application and business needs.To get a list of service accounts that have
Cloud Bigtable API (Data API) permissions, you can [search IAM
policies](https://docs.cloud.google.com/asset-inventory/docs/searching-iam-policies) for your organization. You can also view them on
the [IAM Permissions](https://console.cloud.google.com/iam-admin/iam) Google Cloud console
page on the **Principals** tab.

## Set up log restrictions

Next, set up your log restrictions. There are two ways to manage your
Bigtable log volume by limiting service account logs. You can
either *exempt* service accounts using audit configuration, or you can *exclude*
service account logs using logs exclusion filters. For each method, you can
either use the [Cloud Logging API](https://docs.cloud.google.com/logging/docs/reference/api-overview)
or the Google Cloud console.

### Exempt service accounts using audit configuration

Exempting service accounts using [audit configuration](https://docs.cloud.google.com/logging/docs/audit/configure-data-access) is the
**recommended approach** because it lets you prevent certain logs from being
generated in the first place. For detailed instructions, see the following:

- [Configuring Data Access audit logs with the API](https://docs.cloud.google.com/logging/docs/audit/configure-data-access#config-api)
- [Configuring Data Access audit logs with the Google Cloud console](https://docs.cloud.google.com/logging/docs/audit/configure-data-access#config-console-exempt)

### Exclude service accounts using exclusion filters

[Exclusion filters](https://docs.cloud.google.com/logging/docs/routing/overview#exclusions) let you specify
logs to be excluded from ingestion into your logs buckets. In this approach,
logs are discarded *after they have been created* , so they still impose a
processing load on the Bigtable service components that serve
your data. Because of this load, we recommend that you use audit configuration
instead. For more information on setting up filters using the
Google Cloud console and the API, see [Create a sink](https://docs.cloud.google.com/logging/docs/export/configure_export_v2#creating_sink).

## Estimate Data Access audit log costs

Because Bigtable is typically used for large, high-volume workloads,
it has the potential to generate an extremely high number of logs. Before you
enable Data Access audit logging for Bigtable, you should estimate and
understand the [Cloud Audit Logs](https://docs.cloud.google.com/logging/docs/audit) ingestion and storage costs
that audit logging can incur each month.

Your Data Access audit logging costs are directly related to the number of
Bigtable requests that you choose to log each month. The
following table shows rough estimates of the [Cloud Audit Logs costs](https://cloud.google.com/products/observability/pricing) that you can expect based on
your average requests per second and the length of time that you store your
logs, assuming that you log all data requests. See [Calculating your costs](https://docs.cloud.google.com/bigtable/docs/audit-log-estimate-costs#calculating-costs) for a detailed explanation of how these estimates are
calculated.

| **Average requests per second** | **Log retention time** | **Approximate monthly cost** |
|---|---|---|
| 1,000 | 30 | $1,197 |
| 1,000 | 90 | $1,246 |
| 10,000 | 30 | $12,195 |
| 10,000 | 90 | $12,684 |
| 100,000 | 30 | $122,177 |
| 100,000 | 90 | $124,621 |

## Calculate your costs

Start with the following assumptions:

- The number of seconds in an average month is about 2,628,000.
- The average audit size is around 1 kb.
- You are not charged for the first 50 GiB of audit logs that are ingested per month, and after you pass that amount, you're charged $0.50/GiB.
- Storage is free for 30 days; after that you're charged $0.01/GiB for storage.

The method described on this page provides a gross estimate based on all
traffic. In production, you are encouraged to
[restrict service account logging](https://docs.cloud.google.com/bigtable/docs/audit-log-estimate-costs#identify-accounts).

> [!NOTE]
> **Note:** To keep calculations simple, this page estimates log volume in GB, then converts to GiB. Cloud Logging is billed per GiB.

### Calculate your monthly log volume

First, estimate the average amount of logs your traffic will generate in an
average month.

1. Gather the *average number of requests per second* that your application sends to Bigtable over the course of a month.
   - If you use client-side metrics, you can use them to determine your average queries per second (QPS) for the last month.
   - If you prefer to use your instance's system insights page in the Google Cloud console, use it to determine the average values for read requests and write requests per second over the last month, then add those two values together.
2. Multiply the requests per second by 2,628,000 to get the *average
   requests per month*.
3. Divide that number by 10e6, or 1,000,000. The result is the *estimated
   monthly log volume in GB* that you might generate each month.
4. Multiply the monthly log volume in GB by .93 to get the approximate *monthly log volume in GiB*.

### Calculate your ingestion costs

1. Subtract 50 GiB from the monthly log volume in GiB that you calculated. There is no charge for the first 50 GiB.
2. Multiply the remainder by $0.50 to arrive at your estimated monthly ingestion costs.

### Calculate your storage costs

1. If you plan to let your logs expire after 30 days, your cost for storage is $0.00.
2. If you store your logs for longer than 30 days, your storage costs can be estimated by multiplying the monthly log volume by $0.01. These costs start to incur after the first month.

## Detailed example

### 5,000 request per second, logs retained for 90 days

In this example, suppose that your average number of requests per second is
5,000 and you plan to keep your logs for 90 days. Using the steps on this page,
you calculate the following estimates:

- Multiply 5,000 by 2,628,000 to arrive at 13,140,000,000 requests per month.
- Divide 13,140,000,000 by 10e6 to arrive at roughly 13,140 GB of monthly log volume.
- Convert that number to GiB by multiplying it by .93 to arrive at 12,220.
- Subtract 50 GiB from your monthly log volume to get 12,170 GiB.
- Multiply by $0.50 to get $6,085 in ingestion costs.
- For the first month that your logs exist, the storage cost is $0.
- The second month, the log storage cost is 12,170 multiplied by $0.01, or about $122.
- Every month after the second month, the monthly storage cost is double that, or $244.
- After the second month, your estimated Data Access audit logging costs would be around $6,329 per month.

Presented in equation form, this looks like (((((5,000 rps \* 2,628,000
sec)/1,000,000) \* .93) - 50 GiB) \* $0.50) + $122 = $6,207.

In this example, your monthly Data Access logging costs are around **$6,329 per
month**.

## What's next

- [See what a Data Access audit log entry looks like.](https://docs.cloud.google.com/bigtable/docs/audit-log-example)