This principle in the operational excellence pillar of the
[Google Cloud Well-Architected Framework](https://docs.cloud.google.com/architecture/framework)
helps you to ensure operational readiness and performance of your cloud
workloads. It emphasizes establishing clear expectations and commitments for
service performance, implementing robust monitoring and alerting, conducting
performance testing, and proactively planning for capacity needs.

## Principle overview

Different organizations might interpret operational readiness differently.
Operational readiness is how *your* organization prepares to successfully
operate workloads on Google Cloud. Preparing to operate a complex,
multilayered cloud workload requires careful planning for both go-live and
[day-2](https://www.googlecloudcommunity.com/gc/Community-Blogs/Simplify-Day-2-Cloud-Operations-with-Google-Cloud-Solutions-and/ba-p/495539)
operations. These operations are often called *CloudOps*.

## Focus areas of operational readiness

Operational readiness consists of four focus areas. Each focus area consists of
a set of activities and components that are necessary to prepare to operate a
complex application or environment in Google Cloud. The following table
lists the components and activities of each focus area:

> [!NOTE]
> **Note:** The recommendations in the operational excellence pillar of the Well-Architected Framework are relevant to one or more of these operational-readiness focus areas.

| Focus area of operational readiness | Activities and components |
|---|---|
| Workforce | - Defining clear roles and responsibilities for the teams that manage and operate the cloud resources. - Ensuring that team members have appropriate skills. - Developing a learning program. - Establishing a clear team structure. - Hiring the required talent. |
| Processes | - Observability. - Managing service disruptions. - Cloud delivery. - Core cloud operations. |
| Tooling | Tools that are required to support CloudOps processes. |
| Governance | - Service levels and reporting. - Cloud financials. - Cloud operating model. - Architectural review and governance boards. - Cloud architecture and compliance. |

## Recommendations

To ensure operational readiness and performance by using CloudOps, consider the
recommendations in the following sections. Each recommendation in this document
is relevant to one or more of the
[focus areas of operational readiness](https://docs.cloud.google.com/architecture/framework/operational-excellence/operational-readiness-and-performance-using-cloudops#focus-areas-of-operational-readiness).

### Define SLOs and SLAs

A core responsibility of the cloud operations team is to define service level
objectives (SLOs) and service level agreements (SLAs) for all of the critical
workloads. This recommendation is relevant to the governance
[focus area of operational readiness](https://docs.cloud.google.com/architecture/framework/operational-excellence/operational-readiness-and-performance-using-cloudops#focus-areas-of-operational-readiness).

SLOs must be specific, measurable, achievable, relevant, and time-bound (SMART),
and they must reflect the level of service and performance that you want.

- **Specific**: Clearly articulates the required level of service and performance.
- **Measurable**: Quantifiable and trackable.
- **Achievable**: Attainable within the limits of your organization's capabilities and resources.
- **Relevant**: Aligned with business goals and priorities.
- **Time-bound**: Has a defined timeframe for measurement and evaluation.

For example, an SLO for a web application might be "99.9% availability" or
"average response time less than 200 ms." Such SLOs clearly define the required
level of service and performance for the web application, and the SLOs can be
measured and tracked over time.

SLAs outline the commitments to customers regarding service availability,
performance, and support, including any penalties or remedies for noncompliance.
SLAs must include specific details about the services that are provided, the
level of service that can be expected, the responsibilities of both the service
provider and the customer, and any penalties or remedies for noncompliance. SLAs
serve as a contractual agreement between the two parties, ensuring that both
have a clear understanding of the expectations and obligations that are
associated with the cloud service.

Google Cloud provides tools like
[Cloud Monitoring](https://docs.cloud.google.com/monitoring)
and service level indicators (SLIs) to help you define and track SLOs.
Cloud Monitoring provides comprehensive monitoring and observability
capabilities that enable your organization to collect and analyze metrics that
are related to the availability, performance, and latency of cloud-based
applications and services. SLIs are specific metrics that you can use to measure
and track SLOs over time. By utilizing these tools, you can effectively monitor
and manage cloud services, and ensure that they meet the SLOs and SLAs.

Clearly defining and communicating SLOs and SLAs for all of your critical cloud
services helps to ensure reliability and performance of your deployed
applications and services.

### Implement comprehensive observability

To get real-time visibility into the health and performance of your cloud
environment, we recommend that you use a combination of
[Google Cloud Observability tools](https://cloud.google.com/products/operations)and
third-party solutions. This recommendation is relevant to these
[focus areas of operational readiness](https://docs.cloud.google.com/architecture/framework/operational-excellence/operational-readiness-and-performance-using-cloudops#focus-areas-of-operational-readiness):
processes and tooling.

Implementing a combination of observability solutions provides you with a
comprehensive observability strategy that covers various aspects of your cloud
infrastructure and applications. Google Cloud Observability is a unified platform for
collecting, analyzing, and visualizing metrics, logs, and traces from various
Google Cloud services, applications, and external sources. By using
Cloud Monitoring, you can gain insights into resource utilization,
performance characteristics, and overall health of your resources.

To ensure comprehensive monitoring, monitor important metrics that align with
system health indicators such as CPU utilization, memory usage, network traffic,
disk I/O, and application response times. You must also consider
business-specific metrics. By tracking these metrics, you can identify potential
bottlenecks, performance issues, and resource constraints. Additionally, you can
set up alerts to notify relevant teams proactively about potential issues or
anomalies.

To enhance your monitoring capabilities further, you can integrate third-party
solutions with Google Cloud Observability. These solutions can provide additional
functionality, such as advanced analytics, machine learning-powered anomaly
detection, and incident management capabilities. This combination of
Google Cloud Observability tools and third-party solutions lets you create a robust and
customizable monitoring ecosystem that's tailored to your specific needs. By
using this combination approach, you can proactively identify and address
issues, optimize resource utilization, and ensure the overall reliability and
availability of your cloud applications and services.

### Implement performance and load testing

Performing regular performance testing helps you to ensure that your
cloud-based applications and infrastructure can handle peak loads and maintain
optimal performance. Load testing simulates realistic traffic patterns. Stress
testing pushes the system to its limits to identify potential bottlenecks and
performance limitations. This recommendation is relevant to these
[focus areas of operational readiness](https://docs.cloud.google.com/architecture/framework/operational-excellence/operational-readiness-and-performance-using-cloudops#focus-areas-of-operational-readiness):
processes and tooling.

Tools like
[Cloud Load Balancing](https://docs.cloud.google.com/load-balancing)
and
[load testing services](https://docs.cloud.google.com/load-balancing/docs/backend-service-load-testing)
can help you to simulate real-world traffic patterns and stress-test your
applications. These tools provide valuable insights into how your system behaves
under various load conditions, and can help you to identify areas that require
optimization.

Based on the results of performance testing, you can make decisions to optimize
your cloud infrastructure and applications for optimal performance and
scalability. This optimization might involve adjusting resource allocation,
tuning configurations, or implementing caching mechanisms.

For example, if you find that your application is experiencing slowdowns during
periods of high traffic, you might need to increase the number of virtual
machines or containers that are allocated to the application. Alternatively, you
might need to adjust the configuration of your web server or database to improve
performance.

By regularly conducting performance testing and implementing the necessary
optimizations, you can ensure that your cloud-based applications and
infrastructure always run at peak performance, and deliver a seamless and
responsive experience for your users. Doing so can help you to maintain a
competitive advantage and build trust with your customers.

### Plan and manage capacity

Proactively planning for future capacity needs---both organic or inorganic---helps
you to ensure the smooth operation and scalability of your cloud-based systems.
This recommendation is relevant to the processes
[focus area of operational readiness](https://docs.cloud.google.com/architecture/framework/operational-excellence/operational-readiness-and-performance-using-cloudops#focus-areas-of-operational-readiness).

Planning for future capacity includes understanding and managing
[quotas](https://docs.cloud.google.com/docs/quotas/view-manage)
for various resources like compute instances, storage, and API requests. By
analyzing historical usage patterns, growth projections, and business
requirements, you can accurately anticipate future capacity requirements. You
can use tools like
[Cloud Monitoring](https://docs.cloud.google.com/monitoring)
and
[BigQuery](https://docs.cloud.google.com/bigquery)
to collect and analyze usage data, identify trends, and forecast future demand.

Historical usage patterns provide valuable insights into resource utilization
over time. By examining metrics like CPU utilization, memory usage, and network
traffic, you can identify periods of high demand and potential bottlenecks.
Additionally, you can help to estimate future capacity needs by making growth
projections based on factors like growth in the user base, new products and
features, and marketing campaigns. When you assess capacity needs, you should
also consider business requirements like SLAs and performance targets.

When you determine the resource sizing for a workload, consider factors that can
affect utilization of resources. Seasonal variations like holiday shopping
periods or end-of-quarter sales can lead to temporary spikes in demand. Planned
events like product launches or marketing campaigns can also significantly
increase traffic. To make sure that your primary and disaster recovery (DR)
system can handle unexpected surges in demand, plan for capacity that can
support graceful failover during disruptions like natural disasters and
cyberattacks.

Autoscaling is an important strategy for dynamically adjusting your cloud
resources based on workload fluctuations. By using autoscaling policies, you can
automatically scale compute instances, storage, and other resources in response
to changing demand. This ensures optimal performance during peak periods while
minimizing costs when resource utilization is low. Autoscaling algorithms use
metrics like CPU utilization, memory usage, and queue depth to determine when to
scale resources.

### Continuously monitor and optimize

To manage and optimize cloud workloads, you must establish a process for
continuously monitoring and analyzing performance metrics. This recommendation
is relevant to these
[focus areas of operational readiness](https://docs.cloud.google.com/architecture/framework/operational-excellence/operational-readiness-and-performance-using-cloudops#focus-areas-of-operational-readiness):
processes and tooling.

To establish a process for continuous monitoring and analysis, you track,
collect, and evaluate data that's related to various aspects of your cloud
environment. By using this data, you can proactively identify areas for
improvement, optimize resource utilization, and ensure that your cloud
infrastructure consistently meets or exceeds your performance expectations.

An important aspect of performance monitoring is regularly reviewing logs and
traces. Logs provide valuable insights into system events, errors, and warnings.
Traces provide detailed information about the flow of requests through your
application. By analyzing logs and traces, you can identify potential issues,
identify the root causes of problems, and get a better understanding of how your
applications behave under different conditions. Metrics like the round-trip time
between services can help you to identify and understand bottlenecks that are in
your workloads.

Further, you can use performance-tuning techniques to significantly enhance
application response times and overall efficiency. The following are examples of
techniques that you can use:

- **Caching**: Store frequently accessed data in memory to reduce the need for repeated database queries or API calls.
- **Database optimization**: Use techniques like indexing and query optimization to improve the performance of database operations.
- **Code profiling**: Identify areas of your code that consume excessive resources or cause performance issues.

By applying these techniques, you can optimize your applications and ensure
that they run efficiently in the cloud.