本文說明如何為事件導向 Cloud Run functions 設定重試功能。
如「重試事件」一文所述,如果 Pub/Sub 訊息目的地無法確認訊息,Pub/Sub 預設會以指數輪詢延遲時間再次傳送訊息。指數輪詢可讓您在每次重試之間,逐步增加延遲時間。但這可能不是您在特定實作中想要的行為。
重試次數屬性並非在函式本身實作,而是在叫用函式的 Eventarc 觸發程序實作,因此更具彈性。也就是說,對於 Cloud Run 目的地 (包括使用 Cloud Run Admin API 或 Cloud Functions v2 API 建立的 Cloud Run 函式),您可以設定單次傳送嘗試,不進行重試。在 Cloud Run 頁面中,透過 Google Cloud 控制台建立 Eventarc 觸發條件時,系統會預設採用這項設定。詳情請參閱「使用 Eventarc 建立觸發條件」。
事件導向函式無法完成的原因
事件導向函式可能會因函式程式碼本身擲回的錯誤,而無法順利完成。可能原因包括:
- 函式包含錯誤且執行階段擲回例外狀況。
- 函式無法連線至服務端點,或在嘗試連線時逾時。
- 函式刻意擲回例外狀況 (例如,當參數驗證失敗時)。
- Node.js 函式傳回遭拒絕的 promise,或將非
null值傳遞至回呼。 - 在極少數情況下,函式可能會因內部錯誤而提早結束,根據預設,函式可能會也可能不會自動重試。
在上述任一情況下,函式都會停止執行並傳回錯誤。對於事件觸發條件所產生的訊息,系統設有重試政策 (Retry policies),您可根據函式需求進行自訂。
設定重試政策
視 Cloud Run 函式需求而定,您可考慮透過與 Eventarc 觸發條件相關聯的 Pub/Sub 訂閱重試政策,設定重試政策。如此就能設定下列任意組合:
- 將重試期限從 7 天縮短至 10 分鐘。
- 變更指數輪詢重試策略的最短和最長輪詢時間。
- 將重試策略變更為立即重試。
- 設定 dead-letter 主題。
- 設定傳送嘗試次數上限和下限。
如要設定重試政策,請按照下列步驟操作:
- 編寫 HTTP 函式。
- 使用 Pub/Sub API 建立 Pub/Sub 訂閱項目,將函式的網址指定為目標。
如需其他最佳做法,例如將可重試的事件導向函式設為冪等,請參閱 Eventarc 說明文件中的重試事件。
想進一步瞭解如何直接設定 Pub/Sub,請參閱 Pub/Sub 說明文件中的失敗處理方式。
最佳做法
本節將說明使用重試的最佳做法。
使用重試來處理暫時性錯誤
由於您的函式會不斷重試,直到執行作業成功為止,因此應該先透過測試消除像是程式設計錯誤這一類的永久性錯誤,然後再啟用重試。重試最適合用來處理經過重試之後極有可能解決的間歇性或暫時性失敗,例如不穩定的服務端點或逾時。
設定結束條件以避免無限的重試循環
使用重試時,最佳做法是避免函式不斷循環。您可以在函式開始處理「之前」,加入妥善定義的結束條件來達成此目標。請注意,只有在函式順利啟動並能評估結束條件時,這項技術才能發揮作用。
有效的方法是捨棄時間戳記早於某特定時間的事件。針對永久性或失敗時間比預期時間長的情況,這樣做有助於避免過度執行。
例如,下方程式碼片段會捨棄 10 秒之前的所有事件:
Node.js
const functions = require('@google-cloud/functions-framework');
/**
* Cloud Event Function that only executes within
* a certain time period after the triggering event
*
* @param {object} event The Cloud Functions event.
* @param {function} callback The callback function.
*/
functions.cloudEvent('avoidInfiniteRetries', (event, callback) => {
const eventAge = Date.now() - Date.parse(event.time);
const eventMaxAge = 10000;
// Ignore events that are too old
if (eventAge > eventMaxAge) {
console.log(`Dropping event ${event} with age ${eventAge} ms.`);
callback();
return;
}
// Do what the function is supposed to do
console.log(`Processing event ${event} with age ${eventAge} ms.`);
// Retry failed function executions
const failed = false;
if (failed) {
callback('some error');
} else {
callback();
}
});Python
from datetime import datetime, timezone
# The 'python-dateutil' package must be included in requirements.txt.
from dateutil import parser
import functions_framework
@functions_framework.cloud_event
def avoid_infinite_retries(cloud_event):
"""Cloud Event Function that only executes within a certain
time period after the triggering event.
Args:
cloud_event: The cloud event associated with the current trigger
Returns:
None; output is written to Cloud Logging
"""
timestamp = cloud_event["time"]
event_time = parser.parse(timestamp)
event_age = (datetime.now(timezone.utc) - event_time).total_seconds()
event_age_ms = event_age * 1000
# Ignore events that are too old
max_age_ms = 10000
if event_age_ms > max_age_ms:
print("Dropped {} (age {}ms)".format(cloud_event["id"], event_age_ms))
return "Timeout"
# Do what the function is supposed to do
print("Processed {} (age {}ms)".format(cloud_event["id"], event_age_ms))
return # To retry the execution, raise an exception here
Go
// Package tips contains tips for writing Cloud Functions in Go.
package tips
import (
"context"
"fmt"
"log"
"time"
"github.com/GoogleCloudPlatform/functions-framework-go/functions"
"github.com/cloudevents/sdk-go/v2/event"
)
func init() {
functions.CloudEvent("FiniteRetryPubSub", FiniteRetryPubSub)
}
// MessagePublishedData contains the full Pub/Sub message
// See the documentation for more details:
// https://cloud.google.com/eventarc/docs/cloudevents#pubsub
type MessagePublishedData struct {
Message PubSubMessage
}
// PubSubMessage is the payload of a Pub/Sub event.
// See the documentation for more details:
// https://cloud.google.com/pubsub/docs/reference/rest/v1/PubsubMessage
type PubSubMessage struct {
Data []byte `json:"data"`
}
// FiniteRetryPubSub demonstrates how to avoid inifinite retries.
func FiniteRetryPubSub(ctx context.Context, e event.Event) error {
var msg MessagePublishedData
if err := e.DataAs(&msg); err != nil {
return fmt.Errorf("event.DataAs: %w", err)
}
// Ignore events that are too old.
expiration := e.Time().Add(10 * time.Second)
if time.Now().After(expiration) {
log.Printf("event timeout: halting retries for expired event '%q'", e.ID())
return nil
}
// Add your message processing logic.
return processTheMessage(msg)
}
Java
import com.google.cloud.functions.CloudEventsFunction;
import io.cloudevents.CloudEvent;
import java.time.Duration;
import java.time.ZoneOffset;
import java.time.ZonedDateTime;
import java.util.logging.Logger;
public class RetryTimeout implements CloudEventsFunction {
private static final Logger logger = Logger.getLogger(RetryTimeout.class.getName());
private static final long MAX_EVENT_AGE = 10_000;
/**
* Cloud Event Function that only executes within
* a certain time period after the triggering event
*/
@Override
public void accept(CloudEvent event) throws Exception {
ZonedDateTime utcNow = ZonedDateTime.now(ZoneOffset.UTC);
ZonedDateTime timestamp = event.getTime().atZoneSameInstant(ZoneOffset.UTC);
long eventAge = Duration.between(timestamp, utcNow).toMillis();
// Ignore events that are too old
if (eventAge > MAX_EVENT_AGE) {
logger.info(String.format("Dropping event with timestamp %s.", timestamp));
return;
}
// Process events that are recent enough
// To retry this invocation, throw an exception here
logger.info(String.format("Processing event with timestamp %s.", timestamp));
}
}C#
using CloudNative.CloudEvents;
using Google.Cloud.Functions.Framework;
using Google.Events.Protobuf.Cloud.PubSub.V1;
using Microsoft.Extensions.Logging;
using System;
using System.Threading;
using System.Threading.Tasks;
namespace TimeBoundedRetries;
public class Function : ICloudEventFunction<MessagePublishedData>
{
private static readonly TimeSpan MaxEventAge = TimeSpan.FromSeconds(10);
private readonly ILogger _logger;
// Note: for additional testability, use an injectable clock abstraction.
public Function(ILogger<Function> logger) =>
_logger = logger;
public Task HandleAsync(CloudEvent cloudEvent, MessagePublishedData data, CancellationToken cancellationToken)
{
string textData = data.Message.TextData;
DateTimeOffset utcNow = DateTimeOffset.UtcNow;
// Every PubSub CloudEvent will contain a timestamp.
DateTimeOffset timestamp = cloudEvent.Time.Value;
DateTimeOffset expiry = timestamp + MaxEventAge;
// Ignore events that are too old.
if (utcNow > expiry)
{
_logger.LogInformation("Dropping PubSub message '{text}'", textData);
return Task.CompletedTask;
}
// Process events that are recent enough.
// If this processing throws an exception, the message will be retried until either
// processing succeeds or the event becomes too old and is dropped by the code above.
_logger.LogInformation("Processing PubSub message '{text}'", textData);
return Task.CompletedTask;
}
}Ruby
require "functions_framework"
FunctionsFramework.cloud_event "avoid_infinite_retries" do |event|
# Use the event timestamp to determine the event age.
event_age_secs = Time.now - event.time.to_time
event_age_ms = (event_age_secs * 1000).to_i
max_age_ms = 10_000
if event_age_ms > max_age_ms
# Ignore events that are too old.
logger.info "Dropped #{event.id} (age #{event_age_ms}ms)"
else
# Do what the function is supposed to do.
logger.info "Handling #{event.id} (age #{event_age_ms}ms)..."
failed = true
# Raise an exception to signal failure and trigger a retry.
raise "I failed!" if failed
end
endPHP
/**
* This function shows an example method for avoiding infinite retries in
* Google Cloud Functions. By default, functions configured to automatically
* retry execution on failure will be retried indefinitely - causing an
* infinite loop. To avoid this, we stop retrying executions (by not throwing
* exceptions) for any events that are older than a predefined threshold.
*/
use Google\CloudFunctions\CloudEvent;
function avoidInfiniteRetries(CloudEvent $event): void
{
$log = fopen(getenv('LOGGER_OUTPUT') ?: 'php://stderr', 'wb');
$eventId = $event->getId();
// The maximum age of events to process.
$maxAge = 10; // 10 seconds
// The age of the event being processed.
$eventAge = time() - strtotime($event->getTime());
// Ignore events that are too old
if ($eventAge > $maxAge) {
fwrite($log, 'Dropping event ' . $eventId . ' with age ' . $eventAge . ' seconds' . PHP_EOL);
return;
}
// Do what the function is supposed to do
fwrite($log, 'Processing event: ' . $eventId . ' with age ' . $eventAge . ' seconds' . PHP_EOL);
// infinite_retries failed function executions
$failed = true;
if ($failed) {
throw new Exception('Event ' . $eventId . ' failed; retrying...');
}
}區分可重試的函式和嚴重錯誤
如果您的函式已啟用重試,那麼所有未處理的錯誤都會觸發重試。請確保程式碼會擷取不應導致重試的任何錯誤。
Node.js
const functions = require('@google-cloud/functions-framework');
/**
* Register a Cloud Event Function that demonstrates
* how to toggle retries using a promise
*
* @param {object} event The Cloud Event for the function trigger.
*/
functions.cloudEvent('retryPromise', cloudEvent => {
// The Pub/Sub event payload is passed as the CloudEvent's data payload.
// See the documentation for more details:
// https://cloud.google.com/eventarc/docs/cloudevents#pubsub
const base64PubsubMessage = cloudEvent.data.message.data;
const jsonString = Buffer.from(base64PubsubMessage, 'base64').toString();
const tryAgain = JSON.parse(jsonString).retry;
if (tryAgain) {
throw new Error('Retrying...');
} else {
console.error('Not retrying...');
return Promise.resolve();
}
});
/**
* Cloud Event Function that demonstrates
* how to toggle retries using a callback
*
* @param {object} event The Cloud Event for the function trigger.
* @param {function} callback The callback function.
*/
functions.cloudEvent('retryCallback', (cloudEvent, callback) => {
// The Pub/Sub event payload is passed as the CloudEvent's data payload.
// See the documentation for more details:
// https://cloud.google.com/eventarc/docs/cloudevents#pubsub
const base64PubsubMessage = cloudEvent.data.message.data;
const jsonString = Buffer.from(base64PubsubMessage, 'base64').toString();
const tryAgain = JSON.parse(jsonString).retry;
const err = new Error('Error!');
if (tryAgain) {
console.error('Retrying:', err);
callback(err);
} else {
console.error('Not retrying:', err);
callback();
}
});Python
import base64
import json
import functions_framework
from google.cloud import error_reporting
error_client = error_reporting.Client()
@functions_framework.cloud_event
def retry_or_not(cloud_event):
"""Cloud Event Function that demonstrates how to toggle retries.
Args:
cloud_event: The cloud event with a Pub/Sub data payload
Returns:
None; output is written to Cloud Logging
"""
# The Pub/Sub event payload is passed as the CloudEvent's data payload.
# See the documentation for more details:
# https://cloud.google.com/eventarc/docs/cloudevents#pubsub
encoded_pubsub_message = cloud_event.data["message"]["data"]
# Retry based on a user-defined parameter
try_again = json.loads(base64.b64decode(encoded_pubsub_message).decode())["retry"]
try:
raise RuntimeError("I failed you")
except RuntimeError:
error_client.report_exception()
if try_again:
raise # Raise the exception and try again
else:
pass # Swallow the exception and don't retry
Go
package tips
import (
"context"
"errors"
"fmt"
"log"
"github.com/GoogleCloudPlatform/functions-framework-go/functions"
"github.com/cloudevents/sdk-go/v2/event"
)
func init() {
functions.CloudEvent("RetryPubSub", RetryPubSub)
}
// MessagePublishedData contains the full Pub/Sub message
// See the documentation for more details:
// https://cloud.google.com/eventarc/docs/cloudevents#pubsub
type MessagePublishedData struct {
Message PubSubMessage
}
// PubSubMessage is the payload of a Pub/Sub event.
// See the documentation for more details:
// https://cloud.google.com/pubsub/docs/reference/rest/v1/PubsubMessage
type PubSubMessage struct {
Data []byte `json:"data"`
}
// RetryPubSub demonstrates how to toggle using retries.
func RetryPubSub(ctx context.Context, e event.Event) error {
var msg MessagePublishedData
if err := e.DataAs(&msg); err != nil {
return fmt.Errorf("event.DataAs: %w", err)
}
name := string(msg.Message.Data)
if name == "" {
name = "World"
}
// A misconfigured client will stay broken until the function is redeployed.
client, err := MisconfiguredDataClient()
if err != nil {
log.Printf("MisconfiguredDataClient (retry denied): %v", err)
// A nil return indicates that the function does not need a retry.
return nil
}
// Runtime error might be resolved with a new attempt.
if err = FailedWriteOperation(client, name); err != nil {
log.Printf("FailedWriteOperation (retry expected): %v", err)
// A non-nil return indicates that a retry is needed.
return err
}
return nil
}
Java
import com.google.cloud.functions.CloudEventsFunction;
import com.google.gson.Gson;
import com.google.gson.JsonElement;
import com.google.gson.JsonObject;
import functions.eventpojos.PubSubBody;
import io.cloudevents.CloudEvent;
import java.nio.charset.StandardCharsets;
import java.util.Base64;
import java.util.logging.Logger;
public class RetryPubSub implements CloudEventsFunction {
private static final Logger logger = Logger.getLogger(RetryPubSub.class.getName());
// Use Gson (https://github.com/google/gson) to parse JSON content.
private static final Gson gson = new Gson();
@Override
public void accept(CloudEvent event) throws Exception {
if (event.getData() == null) {
logger.warning("No data found in event!");
return;
}
// Extract Cloud Event data and convert to PubSubBody
String cloudEventData = new String(event.getData().toBytes(), StandardCharsets.UTF_8);
PubSubBody body = gson.fromJson(cloudEventData, PubSubBody.class);
String encodedData = body.getMessage().getData();
String decodedData =
new String(Base64.getDecoder().decode(encodedData), StandardCharsets.UTF_8);
// Retrieve and decode PubSubMessage data into a JsonElement.
// Function is expecting a user-supplied JSON message which determines whether
// to retry or not.
JsonElement jsonPubSubMessageElement = gson.fromJson(decodedData, JsonElement.class);
boolean retry = false;
// Get the value of the "retry" JSON parameter, if one exists
if (jsonPubSubMessageElement != null && jsonPubSubMessageElement.isJsonObject()) {
JsonObject jsonPubSubMessageObject = jsonPubSubMessageElement.getAsJsonObject();
if (jsonPubSubMessageObject.has("retry")
&& jsonPubSubMessageObject.get("retry").getAsBoolean()) {
retry = true;
}
}
// Retry if appropriate
if (retry) {
// Throwing an exception causes the execution to be retried
throw new RuntimeException("Retrying...");
} else {
logger.info("Not retrying...");
}
}
}C#
using CloudNative.CloudEvents;
using Google.Cloud.Functions.Framework;
using Google.Events.Protobuf.Cloud.PubSub.V1;
using Microsoft.Extensions.Logging;
using System;
using System.Text.Json;
using System.Threading;
using System.Threading.Tasks;
namespace Retry;
public class Function : ICloudEventFunction<MessagePublishedData>
{
private readonly ILogger _logger;
public Function(ILogger<Function> logger) =>
_logger = logger;
public Task HandleAsync(CloudEvent cloudEvent, MessagePublishedData data, CancellationToken cancellationToken)
{
bool retry = false;
string text = data.Message?.TextData;
// Get the value of the "retry" JSON parameter, if one exists.
if (!string.IsNullOrEmpty(text))
{
JsonElement element = JsonSerializer.Deserialize<JsonElement>(data.Message.TextData);
retry = element.TryGetProperty("retry", out var property) &&
property.ValueKind == JsonValueKind.True;
}
// Throwing an exception causes the execution to be retried.
if (retry)
{
throw new InvalidOperationException("Retrying...");
}
else
{
_logger.LogInformation("Not retrying...");
}
return Task.CompletedTask;
}
}Ruby
require "functions_framework"
FunctionsFramework.cloud_event "retry_or_not" do |event|
try_again = event.data["retry"]
begin
# Simulate a failure
raise "I failed!"
rescue RuntimeError => e
logger.warn "Caught an error: #{e}"
if try_again
# Raise an exception to return a 500 and trigger a retry.
logger.info "Trying again..."
raise ex
else
# Return normally to end processing of this event.
logger.info "Giving up."
end
end
endPHP
use Google\CloudFunctions\CloudEvent;
function tipsRetry(CloudEvent $event): void
{
$cloudEventData = $event->getData();
$pubSubData = $cloudEventData['message']['data'];
$json = json_decode(base64_decode($pubSubData), true);
// Determine whether to retry the invocation based on a parameter
$tryAgain = $json['some_parameter'];
if ($tryAgain) {
/**
* Functions with automatic retries enabled should throw exceptions to
* indicate intermittent failures that a retry might fix. In this
* case, a thrown exception will cause the original function
* invocation to be re-sent.
*/
throw new Exception('Intermittent failure occurred; retrying...');
}
/**
* If a function with retries enabled encounters a non-retriable
* failure, it should return *without* throwing an exception.
*/
$log = fopen(getenv('LOGGER_OUTPUT') ?: 'php://stderr', 'wb');
fwrite($log, 'Not retrying' . PHP_EOL);
}後續步驟
- 部署 Cloud Run 函式
- 從 Pub/Sub 事件建立觸發條件
- 從 Cloud Storage 事件建立觸發條件
- 使用 Eventarc 從 Pub/Sub 觸發函式
- 使用 Eventarc,透過 Cloud Storage 觸發函式