Spanner 佇列會至少傳送一次訊息,並專為多個接收端之間的平行處理作業而設計。當系統將同一個實體 (例如客戶帳戶、訂單或裝置) 的多則訊息傳送至佇列時,不同的 worker 可以同時提取及處理這些訊息。即使先傳送一則訊息,網路延遲或重試也可能導致第一個 worker 速度變慢,而第二個 worker 則完成後續訊息。
如果整個佇列中的每則訊息都嚴格按照全域順序處理,只要有一個 worker 速度緩慢,就可能導致整個佇列停滯 (這種現象稱為「隊頭阻塞」)。不過,大多數應用程式都不需要讓整個資料庫中的每則訊息都在單一全域佇列中等待。相反地,您只需要依序執行「相同」實體的工作,而不同實體的工作則會平行執行。您可以根據工作負載,使用下列模式依實體處理訊息。
鏈結訊息,建立循序工作流程
如果實體的背景工作包含依序執行的步驟 (例如 step-1,接著是 step-2 和 step-3),且外部要求不會同時將競爭工作新增至該工作流程,則您會希望每個步驟都在上一個步驟完成後才開始。
如要強制執行逐步排序,請一開始只將第一個步驟排入佇列。worker 完成目前步驟的處理作業後,會確認目前的訊息 (DELETE ...ASSERT_ROWS_MODIFIED1),並在同一個讀寫交易中插入下一個步驟的訊息。
由於系統在目前步驟完成並在同一筆交易中確認後,才會建立下一個步驟的訊息,因此同一工作流程的兩個步驟絕不會同時存在於佇列中,也不會由並行 worker 依序處理。
舉例來說,當員工完成新進人員訓練工作流程的 step-1 並排定 step-2 時:
GoogleSQL
-- Inside the worker transaction after finishing step-1:
-- 1. Acknowledge the current step
DELETE FROM OnboardingQueue
WHERE CustomerId = 456 AND StepName = 'step-1'
ASSERT_ROWS_MODIFIED 1;
-- 2. Enqueue the next step in the sequence
INSERT INTO OnboardingQueue (CustomerId, StepName, DeliverTime, Payload)
VALUES (456, 'step-2', CURRENT_TIMESTAMP(), b'{"stage": "provision_resources"}');
PostgreSQL
-- Inside the worker transaction after finishing step-1:
-- 1. Acknowledge the current step
DELETE FROM onboardingqueue
WHERE customerid = 456 AND stepname = 'step-1'
ASSERT_ROWS_MODIFIED 1;
-- 2. Enqueue the next step in the sequence
INSERT INTO onboardingqueue (customerid, stepname, deliver_time, payload)
VALUES (456, 'step-2', CURRENT_TIMESTAMP, CAST('{"stage": "provision_resources"}' AS bytea));
使用排序的工作資料表和喚醒佇列
不過,如果您需要特定實體的所有工作按照提交順序執行,即使外部要求、Webhook 或使用者動作在先前的工作仍在處理時新增工作,這個模式仍支援符合單一分割寫入容量的工作負載。
如要依實體順序處理並行工作,請將已排序的工作儲存在交錯資料表中,並僅將佇列用於輕量喚醒通知,藉此擴充髒標記模式。
首先,在追蹤有效理賠的上層實體資料表中,交錯排列已排序的工作資料表 (使用修訂版本時間戳記) 和喚醒佇列。為避免各分割區出現熱點,請選擇可平均分配實體的主鍵 (例如 UUID 或位元反轉序列):
GoogleSQL
CREATE SEQUENCE CustomerIdSeq OPTIONS (sequence_kind = 'bit_reversed_positive');
CREATE TABLE Customers (
CustomerId INT64 NOT NULL DEFAULT (GET_NEXT_SEQUENCE_VALUE(SEQUENCE CustomerIdSeq)),
CustomerName STRING(1024),
IsProcessing BOOL NOT NULL DEFAULT (FALSE),
ActiveClaimId STRING(36),
ProcessingExpiresAt TIMESTAMP,
) PRIMARY KEY (CustomerId);
CREATE TABLE CustomerWorkItems (
CustomerId INT64 NOT NULL,
CreatedAt TIMESTAMP NOT NULL DEFAULT (PENDING_COMMIT_TIMESTAMP())
OPTIONS (allow_commit_timestamp = true),
BatchIndex INT64 NOT NULL DEFAULT (0),
WorkType STRING(128) NOT NULL,
WorkPayload JSON NOT NULL,
) PRIMARY KEY (CustomerId, CreatedAt, BatchIndex),
INTERLEAVE IN PARENT Customers ON DELETE CASCADE;
CREATE QUEUE CustomerDirtyQueue (
CustomerId INT64 NOT NULL,
MessageId STRING(36) NOT NULL,
Payload BYTES(MAX) NOT NULL,
) PRIMARY KEY (CustomerId, MessageId),
INTERLEAVE IN PARENT Customers ON DELETE CASCADE;
PostgreSQL
CREATE SEQUENCE customeridseq BIT_REVERSED_POSITIVE;
CREATE TABLE customers (
customerid bigint NOT NULL DEFAULT nextval('customeridseq'),
customername varchar(1024),
isprocessing boolean NOT NULL DEFAULT false,
activeclaimid varchar(36),
processingexpiresat timestamptz,
PRIMARY KEY (customerid)
);
CREATE TABLE customerworkitems (
customerid bigint NOT NULL,
createdat spanner.commit_timestamp NOT NULL DEFAULT spanner.pending_commit_timestamp(),
batchindex bigint NOT NULL DEFAULT 0,
worktype varchar(128) NOT NULL,
workpayload jsonb NOT NULL,
PRIMARY KEY (customerid, createdat, batchindex)
) INTERLEAVE IN PARENT customers ON DELETE CASCADE;
CREATE QUEUE customerdirtyqueue (
customerid bigint NOT NULL,
messageid varchar(36) NOT NULL,
payload bytea NOT NULL,
PRIMARY KEY (customerid, messageid)
) INTERLEAVE IN PARENT customers ON DELETE CASCADE;
有了這個結構定義,傳送者和工作站就能使用下列工作流程進行協調:
- 插入排序任務和喚醒訊息:在傳送端交易中,將任務插入
CustomerWorkItems(使用PENDING_COMMIT_TIMESTAMP()和以零為基準的BatchIndex,在同一交易中排序多個任務),並將含有空白酬載 (b'') 的單一喚醒訊息插入CustomerDirtyQueue。 - 只要三個步驟,就能聲明、處理及釋出實體:
- 步驟 1 (在交易 1 中宣告實體):當 worker 收到來自
RECEIVE_CustomerDirtyQueue的喚醒訊息時,會執行簡短的讀寫交易:- 檢查
Customers並確認傳入的喚醒訊息 (DELETE ... ASSERT_ROWS_MODIFIED 1)。如果另一個 worker 已持有有效宣告 (IsProcessing = TRUE和ProcessingExpiresAt > CURRENT_TIMESTAMP()),請提交交易 1 並退出。作用中的 worker 會先檢查剩餘工作,再釋出宣告。 - 否則,請從
CustomerWorkItems讀取最舊的一批工作 (例如LIMIT 25),並依CreatedAt ASC, BatchIndex ASC排序,然後刪除任何剩餘的過期備用訊息 (其MessageId為先前的ActiveClaimId)。 - 如果傳回工作,請在
Customers上設定IsProcessing = TRUE、ActiveClaimId和ProcessingExpiresAt(例如 10 秒後),並將備用訊息 (MessageId = ActiveClaimId) 插入排定在ProcessingExpiresAt的CustomerDirtyQueue。如果沒有剩餘工作,請重設IsProcessing = FALSE、ActiveClaimId = NULL和ProcessingExpiresAt = NULL。
- 檢查
- 步驟 2 (在交易外處理工作):交易 1 提交後,worker 會在應用程式程式碼中依序執行工作。如果工作失敗,worker 會提早停止批次,並仍執行步驟 3,以便在重試失敗的工作前刪除已完成的工作。
- 步驟 3 (刪除已完成的工作,並釋出交易 2 中的實體):worker 開啟第二個短暫的讀寫交易,使用
ASSERT_ROWS_MODIFIED 1從CustomerDirtyQueue刪除備用訊息 (ActiveClaimId),從CustomerWorkItems刪除已完成的工作,如果CustomerWorkItems中還有其他工作,則插入新的喚醒訊息,並重設Customers上的IsProcessing = FALSE、ActiveClaimId = NULL和ProcessingExpiresAt = NULL。
- 步驟 1 (在交易 1 中宣告實體):當 worker 收到來自
這個模式會嚴格保留每個實體的順序和高處理量,原因如下:
- 提交時間戳記排序:將工作儲存在
CustomerWorkItems中,並以(CustomerId, CreatedAt, BatchIndex)做為索引鍵,即可依交易提交時間戳記排序每項工作,即使多個傳送者同時寫入也沒問題。 - 單一分割區位置:
交錯
Customers中的CustomerWorkItems和CustomerDirtyQueue會將實體的宣告狀態、工作和喚醒訊息保留在同一個分割區中。將CustomerDirtyQueue.Payload留空 (b'') 也能盡量減少佇列儲存空間和記憶體內傳送負擔。 - 短暫的資料庫鎖定:在步驟 2 期間保持讀寫交易開啟,會鎖定
Customers資料列,並封鎖並行傳送者。將宣告 (交易 1) 和完成 (交易 2) 分成兩筆短交易,可確保一次只有一個 worker 處理實體,且不會在應用程式程式碼執行時保留資料庫鎖定。 - 當機復原和隔離:立即確認交易 1 中的初始喚醒訊息,並在
ProcessingExpiresAt排定備用訊息 (ActiveClaimId),將固定的 10 秒RECEIVE訊息租約替換為可設定的實體宣告。如果 worker 在步驟 2 期間當機,備用訊息就會到期並喚醒另一個 worker。如果 slow worker 在宣告到期後完成,且另一個 worker 已接管,則在交易 2 中使用ASSERT_ROWS_MODIFIED 1刪除ActiveClaimId會失敗,並顯示OUT_OF_RANGE。讓該錯誤傳播 (或明確復原) 會在寫入提交前,終止交易 2。
下列陳述式會在單一傳送者交易中插入兩項排序工作和一則喚醒訊息:
GoogleSQL
-- 1. Insert ordered tasks for the entity using a 0-based BatchIndex
INSERT INTO CustomerWorkItems (CustomerId, CreatedAt, BatchIndex, WorkType, WorkPayload)
VALUES
(456, PENDING_COMMIT_TIMESTAMP(), 0, 'ORDER_CREATED', JSON '{"order_id": 1001}'),
(456, PENDING_COMMIT_TIMESTAMP(), 1, 'PAYMENT_CAPTURED', JSON '{"order_id": 1001}');
-- 2. Send a single lightweight wake-up message to the interleaved queue
INSERT INTO CustomerDirtyQueue (CustomerId, MessageId, DeliverTime, Payload)
VALUES (456, 'msg-uuid-1', CURRENT_TIMESTAMP(), b'');
PostgreSQL
-- 1. Insert ordered tasks for the entity using a 0-based batchindex
INSERT INTO customerworkitems (customerid, createdat, batchindex, worktype, workpayload)
VALUES
(456, spanner.pending_commit_timestamp(), 0, 'ORDER_CREATED', '{"order_id": 1001}'::jsonb),
(456, spanner.pending_commit_timestamp(), 1, 'PAYMENT_CAPTURED', '{"order_id": 1001}'::jsonb);
-- 2. Send a single lightweight wake-up message to the interleaved queue
INSERT INTO customerdirtyqueue (customerid, messageid, deliver_time, payload)
VALUES (456, 'msg-uuid-1', CURRENT_TIMESTAMP, CAST('' AS bytea));
下列 SQL 陳述式說明 worker 如何接收喚醒訊息、在交易 1 中宣告實體,以及在交易 2 中釋出實體:
GoogleSQL
-- Stream wake-up messages from the queue:
SELECT CustomerId, MessageId, Payload, DeliverTime, SpannerLeaseToken
FROM RECEIVE_CustomerDirtyQueue(max_duration=>'20m', max_batch_size=>10);
-- Transaction 1 (claim the entity):
-- 1. Check whether another worker holds an active claim on the entity
SELECT
IsProcessing,
ActiveClaimId,
IFNULL(ProcessingExpiresAt > CURRENT_TIMESTAMP(), FALSE) AS HasActiveClaim
FROM Customers
WHERE CustomerId = 456;
-- Always acknowledge the incoming wake-up message in Transaction 1.
-- Let OUT_OF_RANGE propagate to roll back the transaction so duplicate workers exit.
-- If IsProcessing = TRUE and HasActiveClaim = TRUE, commit after this DELETE and exit.
DELETE FROM CustomerDirtyQueue
WHERE CustomerId = 456 AND MessageId = 'msg-uuid-1'
ASSERT_ROWS_MODIFIED 1;
-- 2. Only if no active claim exists, read the next ordered batch of work items
SELECT CreatedAt, BatchIndex, WorkType, WorkPayload
FROM CustomerWorkItems
WHERE CustomerId = 456
ORDER BY CreatedAt ASC, BatchIndex ASC
LIMIT 25;
-- 3. Always clean up any remaining expired fallback message
-- (when ActiveClaimId IS NOT NULL AND ActiveClaimId != 'msg-uuid-1'):
DELETE FROM CustomerDirtyQueue
WHERE CustomerId = 456 AND MessageId = 'expired-claim-uuid';
-- 4. If step 2 returns work items, schedule a 10-second fallback message
-- and mark the entity as busy. (If step 2 returns no rows, clear any
-- expired claim fields on Customers instead.)
INSERT INTO CustomerDirtyQueue (CustomerId, MessageId, DeliverTime, Payload)
VALUES (456, 'claim-uuid-2', TIMESTAMP_ADD(CURRENT_TIMESTAMP(), INTERVAL 10 SECOND), b'');
UPDATE Customers
SET IsProcessing = TRUE,
ActiveClaimId = 'claim-uuid-2',
ProcessingExpiresAt = TIMESTAMP_ADD(CURRENT_TIMESTAMP(), INTERVAL 10 SECOND)
WHERE CustomerId = 456;
-- Transaction 2 (after processing the batch in order outside a transaction):
-- 1. Verify the claim is still active by deleting the fallback message.
-- Let OUT_OF_RANGE propagate to roll back the transaction if another worker took over.
DELETE FROM CustomerDirtyQueue
WHERE CustomerId = 456 AND MessageId = 'claim-uuid-2'
ASSERT_ROWS_MODIFIED 1;
-- 2. Delete the completed work items
DELETE FROM CustomerWorkItems
WHERE CustomerId = 456
AND CreatedAt = TIMESTAMP '2026-05-01T12:00:00.123456Z'
AND BatchIndex IN (0, 1);
-- 3. Check whether any unprocessed work items remain:
SELECT 1 FROM CustomerWorkItems
WHERE CustomerId = 456
LIMIT 1;
-- Run this INSERT only if the preceding SELECT returned a row:
INSERT INTO CustomerDirtyQueue (CustomerId, MessageId, DeliverTime, Payload)
VALUES (456, 'continuation-uuid-3', CURRENT_TIMESTAMP(), b'');
-- 4. Release the entity claim
UPDATE Customers
SET IsProcessing = FALSE, ActiveClaimId = NULL, ProcessingExpiresAt = NULL
WHERE CustomerId = 456 AND ActiveClaimId = 'claim-uuid-2';
PostgreSQL
-- Stream wake-up messages from the queue:
SELECT customerid, messageid, payload, deliver_time, spanner_lease_token
FROM spanner.receive_customerdirtyqueue(
max_batch_size=>10, priority=>NULL, max_duration=>'20m');
-- Transaction 1 (claim the entity):
-- 1. Check whether another worker holds an active claim on the entity
SELECT
isprocessing,
activeclaimid,
COALESCE(processingexpiresat > CURRENT_TIMESTAMP, false) AS hasactiveclaim
FROM customers
WHERE customerid = 456;
-- Always acknowledge the incoming wake-up message in Transaction 1.
-- Let OUT_OF_RANGE propagate to roll back the transaction so duplicate workers exit.
-- If isprocessing = true and hasactiveclaim = true, commit after this DELETE and exit.
DELETE FROM customerdirtyqueue
WHERE customerid = 456 AND messageid = 'msg-uuid-1'
ASSERT_ROWS_MODIFIED 1;
-- 2. Only if no active claim exists, read the next ordered batch of work items
SELECT createdat, batchindex, worktype, workpayload
FROM customerworkitems
WHERE customerid = 456
ORDER BY createdat ASC, batchindex ASC
LIMIT 25;
-- 3. Always clean up any remaining expired fallback message
-- (when activeclaimid IS NOT NULL AND activeclaimid != 'msg-uuid-1'):
DELETE FROM customerdirtyqueue
WHERE customerid = 456 AND messageid = 'expired-claim-uuid';
-- 4. If step 2 returns work items, schedule a 10-second fallback message
-- and mark the entity as busy. (If step 2 returns no rows, clear any
-- expired claim fields on customers instead.)
INSERT INTO customerdirtyqueue (customerid, messageid, deliver_time, payload)
VALUES (456, 'claim-uuid-2', CURRENT_TIMESTAMP + INTERVAL '10 SECOND', CAST('' AS bytea));
UPDATE customers
SET isprocessing = true,
activeclaimid = 'claim-uuid-2',
processingexpiresat = CURRENT_TIMESTAMP + INTERVAL '10 SECOND'
WHERE customerid = 456;
-- Transaction 2 (after processing the batch in order outside a transaction):
-- 1. Verify the claim is still active by deleting the fallback message.
-- Let OUT_OF_RANGE propagate to roll back the transaction if another worker took over.
DELETE FROM customerdirtyqueue
WHERE customerid = 456 AND messageid = 'claim-uuid-2'
ASSERT_ROWS_MODIFIED 1;
-- 2. Delete the completed work items
DELETE FROM customerworkitems
WHERE customerid = 456
AND createdat = '2026-05-01T12:00:00.123456Z'::timestamptz
AND batchindex IN (0, 1);
-- 3. Check whether any unprocessed work items remain:
SELECT 1 FROM customerworkitems
WHERE customerid = 456
LIMIT 1;
-- Run this INSERT only if the preceding SELECT returned a row:
INSERT INTO customerdirtyqueue (customerid, messageid, deliver_time, payload)
VALUES (456, 'continuation-uuid-3', CURRENT_TIMESTAMP, CAST('' AS bytea));
-- 4. Release the entity claim
UPDATE customers
SET isprocessing = false, activeclaimid = NULL, processingexpiresat = NULL
WHERE customerid = 456 AND activeclaimid = 'claim-uuid-2';
使用時間批次處理功能,簡化突發性喚醒訊息
當實體在幾秒內頻繁收到大量更新時,如果為每筆交易傳送個別的喚醒訊息,會造成不必要的佇列寫入和多餘的喚醒傳送。
如要在寫入時合併喚醒訊息,請將時間批次處理與確定性 MessageId 結合:
- 將
MessageId設為時間範圍的 bucket ID (例如 5 秒 bucket)。 - 將
DeliverTime四捨五入至下一個時間範圍,再加上使用FARM_FINGERPRINT(或spanner.farm_fingerprint) 計算的每個實體偏移量。 - 在 GoogleSQL 中使用
INSERT OR IGNORE,或在 PostgreSQL 中使用INSERT ... SELECT ... FROM ... WHERE NOT EXISTS,插入喚醒訊息。
在 5 秒的時間範圍內,同一實體的每筆交易都會計算相同的 MessageId 和 DeliverTime。所有交易都會將工作插入 CustomerWorkItems,但只有視窗中的第一筆交易會將喚醒訊息寫入 CustomerDirtyQueue。新增每個實體的偏移量,可將放送時間分散在 5 秒的間隔內,這樣不同的實體就不會在同一秒觸發。
舉例來說,下列傳送者交易會將喚醒訊息合併為 5 秒的值區:
GoogleSQL
-- 1. Insert the ordered work item
INSERT INTO CustomerWorkItems (CustomerId, CreatedAt, BatchIndex, WorkType, WorkPayload)
VALUES (456, PENDING_COMMIT_TIMESTAMP(), 0, 'METRIC_RECORDED', JSON '{"v": 42}');
-- 2. Coalesce wake-up messages into a 5-second window with a per-customer offset
INSERT OR IGNORE INTO CustomerDirtyQueue (CustomerId, MessageId, DeliverTime, Payload)
VALUES (
456,
CONCAT('bucket-', CAST(DIV(UNIX_SECONDS(CURRENT_TIMESTAMP()), 5) * 5 AS STRING)),
TIMESTAMP_SECONDS(
DIV(UNIX_SECONDS(CURRENT_TIMESTAMP()), 5) * 5 + 5
+ MOD(ABS(FARM_FINGERPRINT(CAST(456 AS STRING))), 5)
),
b''
);
PostgreSQL
-- 1. Insert the ordered work item
INSERT INTO customerworkitems (customerid, createdat, batchindex, worktype, workpayload)
VALUES (456, spanner.pending_commit_timestamp(), 0, 'METRIC_RECORDED', '{"v": 42}'::jsonb);
-- 2. Coalesce wake-up messages into a 5-second window with a per-customer offset
INSERT INTO customerdirtyqueue (customerid, messageid, deliver_time, payload)
SELECT
456,
'bucket-' || bucket.start_sec::text,
to_timestamp(
bucket.start_sec + 5
+ (abs(spanner.farm_fingerprint(456::text)) % 5)
),
CAST('' AS bytea)
FROM (
SELECT (floor(EXTRACT(EPOCH FROM CURRENT_TIMESTAMP))::bigint / 5) * 5 AS start_sec
) AS bucket
WHERE NOT EXISTS (
SELECT 1 FROM customerdirtyqueue
WHERE customerid = 456
AND messageid = 'bucket-' || bucket.start_sec::text
);
減少並行喚醒訊息造成的鎖定爭用
當同一個忙碌實體的多個喚醒訊息同時到期時,多個 worker 可以同時啟動交易 1,並爭奪同一個 Customers 資料列的讀寫鎖定。
如要減少熱門實體的鎖定爭用情形,請在啟動交易 1 之前執行無鎖定的單一讀取 (client.Single()),檢查 IsProcessing = TRUE 和 ProcessingExpiresAt > CURRENT_TIMESTAMP()。如果實體忙碌中,請先隨機短暫延遲 (例如 10 到 50 毫秒),再執行交易 1 重新檢查 Customers,並確認重複的喚醒訊息。
單一讀取作業會檢查實體的聲明狀態,而不會取得讀寫鎖定,隨機延遲則會錯開後續的讀寫交易,因此 worker 不會在同一毫秒內爭用 Customers 資料列。刪除喚醒訊息前,請務必重新檢查交易 1 內的 IsProcessing,這樣一來,如果活躍工作站剛好在單次讀取後完成,喚醒訊息就不會在仍有未處理的新工作時遭到刪除。
舉例來說,下列 Go 函式會在實體已忙碌時套用隨機延遲:
func maybeJitterBeforeClaim(ctx context.Context, client *spanner.Client, customerID int64) error {
iter := client.Single().Query(ctx, spanner.Statement{
SQL: `SELECT IsProcessing AND IFNULL(ProcessingExpiresAt > CURRENT_TIMESTAMP(), FALSE) AS IsBusy
FROM Customers WHERE CustomerId = @cid`,
Params: map[string]interface{}{"cid": customerID},
})
defer iter.Stop()
row, err := iter.Next()
if errors.Is(err, iterator.Done) {
return nil
}
if err != nil {
return err
}
var isBusy bool
if err := row.Columns(&isBusy); err != nil {
return err
}
if isBusy {
time.Sleep(time.Duration(10+rand.Intn(40)) * time.Millisecond)
}
return nil
}
處理長時間執行的批次和外部呼叫
如果一批工作的處理時間超過初始 10 秒 ProcessingExpiresAt 視窗,或呼叫外部 API (例如 Payment Gateway 或 Webhook),且在資料庫交易取消時未復原,則必須防止重複的外部動作和過早宣告到期。
如要保護外部呼叫,並延長慢速批次的實體宣告:
- 將確定性冪等金鑰傳遞至外部 API:合併
CustomerId、CreatedAt和BatchIndex(例如456:2026-05-01T12:00:00.123456Z:0),並在步驟 2 將該金鑰傳遞至下游服務。 - 使用定期心跳訊號延長實體宣告:在交易 1 中設定較長的初始
ProcessingExpiresAt和備用DeliverTime(例如 30 或 60 秒),或在宣告到期前執行背景心跳訊號交易 (例如在 10 秒的宣告視窗中每 3 到 4 秒執行一次)。每筆心跳交易都會刪除目前的備用訊息 (ActiveClaimId) 和ASSERT_ROWS_MODIFIED 1,插入新的備用訊息和更新後的未來DeliverTime,並更新Customers上的ActiveClaimId和ProcessingExpiresAt。在啟動交易 2 前,請先停止活動訊號。
由於步驟 2 是在資料庫交易外執行,且佇列傳遞至少會執行一次,因此在外部 API 呼叫後,worker 停止運作會導致下一個 worker 重試工作。傳遞確定性冪等性金鑰,可讓外部服務忽略重複的要求。詳情請參閱「交易式寄件匣」。定期更換備用訊息並延長 ProcessingExpiresAt,可在 worker 正常運作時保持實體宣告有效,同時在 worker 異常終止時允許其他 worker 接管。
舉例來說,下列心跳訊號交易會將有效實體宣告延長 10 秒:
GoogleSQL
-- Run every 3 to 4 seconds on a 10-second claim window before starting Transaction 2:
DELETE FROM CustomerDirtyQueue
WHERE CustomerId = 456 AND MessageId = 'claim-uuid-2'
ASSERT_ROWS_MODIFIED 1;
INSERT INTO CustomerDirtyQueue (CustomerId, MessageId, DeliverTime, Payload)
VALUES (456, 'claim-uuid-3', TIMESTAMP_ADD(CURRENT_TIMESTAMP(), INTERVAL 10 SECOND), b'');
UPDATE Customers
SET ActiveClaimId = 'claim-uuid-3',
ProcessingExpiresAt = TIMESTAMP_ADD(CURRENT_TIMESTAMP(), INTERVAL 10 SECOND)
WHERE CustomerId = 456 AND ActiveClaimId = 'claim-uuid-2';
PostgreSQL
-- Run every 3 to 4 seconds on a 10-second claim window before starting Transaction 2:
DELETE FROM customerdirtyqueue
WHERE customerid = 456 AND messageid = 'claim-uuid-2'
ASSERT_ROWS_MODIFIED 1;
INSERT INTO customerdirtyqueue (customerid, messageid, deliver_time, payload)
VALUES (456, 'claim-uuid-3', CURRENT_TIMESTAMP + INTERVAL '10 SECOND', CAST('' AS bytea));
UPDATE customers
SET activeclaimid = 'claim-uuid-3',
processingexpiresat = CURRENT_TIMESTAMP + INTERVAL '10 SECOND'
WHERE customerid = 456 AND activeclaimid = 'claim-uuid-2';
透過 shard 和過時讀取功能,擴充大量實體
如果單一實體收到的寫入作業超過一個 Spanner 分割區可處理的數量,即使您仍須按照修訂順序處理該實體的工作,將每項工作寫入單一 Customers 分割區也會建立寫入熱點。
如要擴充高量實體的寫入處理量,同時保留全域排序,請將寫入作業分散到 N 個 shard,並使用過時讀取,透過定期清除 worker 非同步合併這些 shard:
- 將工作和簡化資料表進行 shard:建立含有
PRIMARY KEY (ShardId, CustomerId)的CustomerShards父項資料表 (ShardId從0到N - 1),並交錯嵌入以(ShardId, CustomerId, CreatedAt, BatchIndex)為索引鍵的 shard 工作資料表。將ShardId設為主要鍵資料欄,可將寫入作業分散到N個分割區。為避免重試交易收到新的PENDING_COMMIT_TIMESTAMP()時產生重複的工作列,請一併交錯使用以(ShardId, CustomerId, BatchId)為索引鍵的批次簡化資料表,並套用TTL政策。 - 將每個產生器批次轉送至一個 shard,不寫入佇列:在每個產生器交易中,從不重複
BatchId(例如FARM_FINGERPRINT) 的雜湊計算ShardId,並插入簡化資料表和 shard 工作資料表,不插入佇列。對BatchId進行雜湊處理,可將每個產生器交易保留在單一分割區中,並消除每次寫入佇列的負擔。 - 使用過時讀取和進度游標,依序掃描 shard:在獨立的
CustomerSweeperState資料表中,維護有效的宣告和(LastSweptAt, LastBatchIndex, LastShardId)游標,並使用交錯的喚醒佇列,其中包含每個實體的一個週期性掃描器訊息。觸發時,worker 會宣告實體 (使用與使用排序工作表和喚醒佇列相同的宣告和備用訊息模式),使用 10 秒精確過時讀取 (依CreatedAt ASC, BatchIndex ASC, ShardId ASC排序),查詢游標後的所有N個 shard,依序處理批次,並提交完成交易,刪除已處理的資料列、推進游標,並排定下一次掃描。使用過時讀取可避免在各分區中進行中的生產者寫入作業遭到封鎖,而游標會略過在過時時間範圍內處理及刪除的資料列。
後續步驟
- 瞭解如何使用 Spanner 佇列,包括最佳做法和監控。
- 探索更多 Spanner 佇列情境和範例。
- 瞭解僅處理一次和最多確認一次。
- 使用精細的佇列存取控管機制設定存取控管。