You can use the best practices listed here as a quick reference of what to keep
in mind when building an application that uses Datastore. If you are
just starting out with Datastore, this page might not be the best
place to start, because it does not teach you the basics of how to use
Datastore. If you are a new user, we suggest that you start with
[Getting Started with Datastore](https://docs.cloud.google.com/datastore/docs/datastore-api-tutorial).

> [!NOTE]
> **Note:** This page describes system behavior for Datastore databases that have not yet upgraded to Firestore in Datastore mode.
>
>
> [Firestore](https://docs.cloud.google.com/firestore) is the new version of
> Datastore and
> [removes
> several Datastore limitations](https://docs.cloud.google.com/datastore/docs/firestore-or-datastore#in_datastore_mode).

## General

- Always use UTF-8 characters for namespace names, kind names, property names, and custom key names. Non-UTF-8 characters used in these names can interfere with Datastore functionality. For example, a non-UTF-8 character in a property name can prevent creation of an index that uses the property.
- Do not use a forward slash (`/`) in kind names or custom key names. Forward slashes in these names could interfere with future functionality.
- Avoid storing sensitive information in a Cloud Project ID. A Cloud Project ID might be retained beyond the life of your project.
- As a data compliance best practice, we recommend not storing sensitive information in Datastore entity names or entity property names.

## API calls

- Use [batch operations](https://docs.cloud.google.com/datastore/docs/concepts/entities#batch_operations) for your reads, writes, and deletes instead of single operations. Batch operations are more efficient because they perform multiple operations with the same overhead as a single operation.
- If a [transaction](https://docs.cloud.google.com/datastore/docs/concepts/transactions) fails, ensure you try to rollback the transaction. The rollback minimizes retry latency for a different request contending for the same resource(s) in a transaction. Note that a rollback itself might fail, so the rollback should be a best-effort attempt only.
- Use asynchronous calls where available instead of synchronous calls. Asynchronous calls minimize latency impact. For example, consider an application that needs the result of a synchronous `lookup()` and the results of a query before it can render a response. If the `lookup()` and the query do not have a data dependency, there is no need to synchronously wait until the `lookup()` completes before initiating the query.

## Entities

- Group highly related data in [entity groups](https://docs.cloud.google.com/datastore/docs/concepts/entities#entity_groups). Entity groups enable [ancestor queries](https://docs.cloud.google.com/datastore/docs/concepts/queries#ancestor_queries), which return strongly consistent results. Ancestor queries also rapidly scan an entity group with minimal I/O because the entities in an entity group are stored at physically close places on Datastore servers.
- Avoid writing to an entity group more than once per second. Writing at a sustained rate above that limit makes eventually consistent reads more eventual, leads to time outs for strongly consistent reads, and results in slower overall performance of your application. A batch or transactional write to an entity group counts as only a single write against this limit.
- Do not include the same entity (by key) multiple times in the same commit. Including the same entity multiple times in the same commit could impact Datastore latency.

## Keys

- Key names are autogenerated if not provided at entity creation. They are allocated so as to be evenly distributed in the keyspace.
- For a key that uses a custom name, always use UTF-8 characters except a forward slash (`/`). Non-UTF-8 characters interfere with various processes such as importing a Datastore backup into [Google BigQuery](https://docs.cloud.google.com/bigquery). A forward slash could interfere with future functionality.
- For a key that uses a numeric ID:
  - Do not use a negative number for the ID. A negative ID could interfere with sorting.
  - Do not use the value `0`(zero) for the ID. If you do, you will get an automatically allocated ID.
  - If you wish to manually assign your own numeric IDs to the entities you create, have your application obtain a block of IDs with the `allocateIds()` method. This will prevent Datastore from assigning one of your manual numeric IDs to another entity.
- If you assign your own manual numeric ID or custom name to the entities
  you create, do not use monotonically increasing values such as:

      1, 2, 3, ...,
      "Customer1", "Customer2", "Customer3", ....
      "Product 1", "Product 2", "Product 3", ....

  If an application generates large traffic, such sequential numbering could
  lead to hotspots that impact Datastore latency. To avoid the
  issue of sequential numeric IDs, obtain numeric IDs from the `allocateIds()`
  method. The `allocateIds()` method generates well-distributed sequences of
  numeric IDs.
- By specifying a key or storing the generated name, you can later perform a
  consistent `lookup()` on that entity without needing issue a query to find the entity.

## Indexes

- If a property will never be needed for a query, [exclude the property from indexes](https://docs.cloud.google.com/datastore/docs/concepts/indexes#unindexed_properties). Unnecessarily indexing a property could result in increased latency to achieve consistency, and increased [storage costs of index entries](https://docs.cloud.google.com/datastore/docs/concepts/storage-size#index_entry_size).
- Avoid having too many [composite indexes](https://docs.cloud.google.com/datastore/docs/concepts/indexes#composite_indexes). Excessive use of composite indexes could result in increased latency to achieve consistency, and increased [storage costs of index entries](https://docs.cloud.google.com/datastore/docs/concepts/storage-size#index_entry_size). If you need to execute ad hoc queries on large datasets without previously defined indexes, use [Google BigQuery](https://docs.cloud.google.com/bigquery).
- Do not index properties with monotonically increasing values (such as a `NOW()` timestamp). Maintaining such an index could lead to hotspots that impact Datastore latency for applications with high read and write rates. For further guidance on dealing with monotonic properties, see [High read/write rates for a narrow key range](https://docs.cloud.google.com/datastore/docs/best-practices#high_readwrite_rates_to_a_narrow_key_range) below.

## Properties

- Always use UTF-8 characters for properties of type [string](https://docs.cloud.google.com/datastore/docs/concepts/entities#text_string). A non-UTF-8 character in a property of type string could interfere with queries. If you need to save data with non-UTF-8 characters, use a [byte string](https://docs.cloud.google.com/datastore/docs/concepts/entities#byte_string).
- Do not use dots in property names. Dots in property names interfere with indexing of [embedded entity properties](https://docs.cloud.google.com/datastore/docs/concepts/entities#embedded_entity).

## Queries

- If you need to access only the key from query results, use a [keys-only query](https://docs.cloud.google.com/datastore/docs/concepts/queries#keys-only_queries). A keys-only query returns results at lower latency and cost than retrieving entire entities.
- If you need to access only specific properties from an entity, use a [projection query](https://docs.cloud.google.com/datastore/docs/concepts/queries#projection_queries). A projection query returns results at lower latency and cost than retrieving entire entities.
- Likewise, if you need to access only the properties that are included in the query filter (for example, those listed in an `order by` clause), use a [projection query](https://docs.cloud.google.com/datastore/docs/concepts/queries#projection_queries).
- Do not use offsets. Instead use [cursors](https://docs.cloud.google.com/datastore/docs/concepts/queries#cursors_limits_and_offsets). Using an offset only avoids returning the skipped entities to your application, but these entities are still retrieved internally. The skipped entities affect the latency of the query, and your application is billed for the read operations required to retrieve them.
- If you need strong consistency for your queries, use an [ancestor query](https://docs.cloud.google.com/datastore/docs/concepts/queries#ancestor_queries). (To use ancestor queries, you first need to [structure your data for strong consistency](https://docs.cloud.google.com/datastore/docs/concepts/structuring_for_strong_consistency#structuring_your_data_for_consistency).) An ancestor query returns strongly consistent results. Note that a non-ancestor [keys-only](https://docs.cloud.google.com/datastore/docs/concepts/queries#keys-only_queries) query followed by a `lookup()` does not return strong results, because the non-ancestor keys-only query could get results from an index that is not consistent at the time of the query.

## Designing for scale

### Updates to a single entity group

A single entity group in Datastore should not be updated too rapidly.

If you are using Datastore, Google recommends that you design your
application so that it will not need to update an entity group more than once per second.
Remember that an entity with no parent and no children is its own entity
group. If you update an entity group too rapidly then your Datastore writes will
have higher latency, timeouts, and other types of error. This is known as
contention.

Datastore write rates to a single entity group can sometimes exceed the one per second
limit so load tests might not show this problem.

### High read/write rates to a narrow key range

Avoid high read or write rates to Datastore keys that are lexicographically
close.

Datastore is built on top of Google's NoSQL database,
[Bigtable](https://docs.cloud.google.com/bigtable), and is subject to Bigtable's performance
characteristics. Bigtable scales by sharding rows onto separate tablets, and
these rows are lexicographically ordered by key.

If you are using Datastore, you can get slow writes due to a
hot tablet if you have a sudden increase in the write rate to a small
range of keys that exceeds the capacity of a single tablet
server. Bigtable will eventually split the key space to support high
load.

The limit for reads is typically much higher than for writes,
unless you are reading from a single key at a high rate. Bigtable
cannot split a single key onto more than one tablet.

Hot tablets can apply to key ranges used by both entity keys and
indexes.

In some cases, a Datastore hotspot can have wider impact to an
application than preventing reads or writes to a small range of
keys. For example, the hot keys might be read or written during instance
startup, causing loading requests to fail.

By default, Datastore allocates keys using a scattered
algorithm. Thus you will not normally encounter hotspotting on
Datastore writes if you create new entities at a high write rate using
the default ID allocation policy. There are some corner cases where
you can hit this problem:

- If you create new entities at a very high rate using the legacy sequential ID
  allocation policy.

- If you create new entities at a very high rate and you are allocating your own
  IDs which are monotonically increasing.

- If you create new entities at a very high rate for a kind which previously had
  very few existing entities. Bigtable will start off with all entities on the
  same tablet server and will take some time to split the range of keys onto
  separate tablet servers.

- You will also see this problem if you create new entities at a high rate with
  a monotonically increasing indexed property like a timestamp, because these
  properties are the keys for rows in the index tables in Bigtable.

- Datastore prepends the namespace and the kind of the root entity group to the
  Bigtable row key. You can hit a hotspot if you start to write to a new
  namespace or kind without gradually ramping up traffic.

If you do have a key or indexed property that will be monotonically
increasing then you can prepend a random hash to ensure that the keys
are sharded onto multiple tablets.

Likewise, if you need to query on a monotonically increasing (or decreasing)
property using a sort or filter, you could instead index on a new property, for which you
prefix the monotonic value with a value that has high cardinality across the dataset, but is
common to all the entities in the scope of the query you want to perform.
For instance, if you want to query for entries by timestamp but only need to
return results for a single user at a time, you could prefix the timestamp with the
user id and index that new property instead. This would still permit queries and
ordered results for that user, but the presence of the user id would ensure the
index itself is well sharded.

For a more detailed explanation of this issue,
see
[Ikai Lan's blog posting on saving monotonically increasing values in Datastore](http://ikaisays.com/2011/01/25/app-engine-datastore-tip-monotonically-increasing-values-are-bad/).

### Ramping up traffic

Gradually ramp up traffic to new Datastore kinds or portions of the keyspace.

You should ramp up traffic to new Datastore kinds gradually in
order to give Bigtable sufficient time to split tablets as the traffic
grows. We recommend a maximum of 500 operations per second to a new
Datastore kind, then increasing traffic by 50% every 5 minutes. In
theory, you can grow to 740K operations per second after 90 minutes
using this ramp up schedule. Be sure that writes are distributed
relatively evenly throughout the key range. Our SREs call this the
"500/50/5" rule.

This pattern of gradual ramp up is particularly important if you
change your code to stop using kind A and instead use kind B. A naive
way to handle this migration is to change your code to read kind B,
and if it does not exist then read kind A. However, this could cause a
sudden increase in traffic to a new kind with a very small portion of
the keyspace. Bigtable might be unable to efficiently split tablets
if the keyspace is sparse.

The same problem can also occur if you migrate your entities to use
a different range of keys within the same kind.

The strategy that you use to migrate entities to a new kind or key
will depend on your data model. Below is an example strategy, known as
"Parallel Reads". You will need to determine whether or not this
strategy is effective for your data. An important consideration will
be the cost impact of parallel operations during the migration.

Read from the old entity or key first. If it is missing, then you
could read from the new entity or key. A high rate of reads of
non-existent entities can lead to hotspotting, so you need to be sure
to gradually increase load. A better strategy is to copy the old
entity to the new then delete the old. Ramp up parallel reads
gradually to ensure that the new key space is well split.

A possible strategy for gradually ramping up reads or writes to a
new kind is use a deterministic hash of the user ID to get a random
percentage of users that write new entities. Be sure that the result
of the user ID hash is not skewed either by your random function or by
your user behavior.

Meanwhile, run a Dataflow job to copy all your data from the old entities or
keys to the new ones. Your batch job should avoid writes to sequential keys in
order to prevent Bigtable hotspots. When the batch job is complete, you can read
only from the new location.

A refinement of this strategy is to migrate small batches of users
at one time. Add a field to the user entity which tracks the migration
status of that user. Select a batch of users to migrate based on a
hash of the user ID. A Mapreduce or Dataflow job will migrate the keys
for that batch of users. The users that have an in-progress migration
will use parallel reads.

Note that you cannot easily roll back unless you do dual writes of
both the old and new entities during the migration phase. This would
increase Datastore costs incurred.

### Deletions

Avoid deleting large numbers of Datastore entities across a small range of keys.

Bigtable periodically rewrites its tables to remove deleted
entries, and to reorganize your data so that reads and writes are more
efficient. This process is known as a compaction.

If you delete a large number of Datastore entities across a small
range of keys then queries across this part of the index will be
slower until compaction has completed. In extreme cases, your queries
might time out before returning results.

It is an anti-pattern to use a timestamp value for an indexed
field to represent an entity's expiration time. In order to retrieve
expired entities, you would need to query against this indexed field,
which likely lies in an overlapping part of the keyspace with index
entries for the most recently deleted entities.

You can improve performance with "sharded queries", that prepend a
fixed length string to the expiration timestamp. The index is sorted
on the full string, so that entities at the same timestamp will be
located throughout the key range of the index. You run multiple
queries in parallel to fetch results from each shard.

A fuller solution for the expiration timestamp issue is to use a
"generation number" which is a global counter that is periodically
updated. The generation number is prepended to the expiration
timestamp so that queries are sorted by generation number, then shard,
then timestamp. Deletion of old entities occurs at a prior
generation. Any entity not deleted should have its generation number
incremented. Once the deletion is complete, you move forward to the
next generation. Queries against an older generation will perform
poorly until compaction is complete. You might need to wait for several
generations to complete before querying the index to get the list of
entities to delete, in order to reduce the risk of missing results due
to eventual consistency.

### Sharding and replication

Use sharding or replication for hot Datastore keys.

You can use replication if you need to read a portion of the key
range at a higher rate than Bigtable permits. Using this strategy, you
would store N copies of the same entity allowing N times higher rate
of reads than supported by a single entity.

You can use sharding if you need to write to a portion of the key
range at a higher rate than Bigtable permits. Sharding breaks up an
entity into smaller pieces.

Some common mistakes when sharding include:

- Sharding using a time prefix. When the time rolls over to the next prefix then
  the new unsplit portion becomes a hotspot. Instead, you should gradually roll
  over a portion of your writes to the new prefix.

- Sharding just the hottest entities. If you shard a small proportion of the
  total number of entities then there might not be sufficient rows between the
  hot entities to ensure that they stay on different splits.

## What's next

- Learn about [Datastore Limits](https://docs.cloud.google.com/datastore/docs/concepts/limits).
- Learn about Google Cloud Platform [Best Practices for Enterprise Organizations](https://docs.cloud.google.com/architecture/framework).