Package @google-cloud/bigquery-storage (6.3.0)

Classes

managedwriter.JSONWriter

A StreamWriter that can write JSON data to BigQuery tables. The JSONWriter is built on top of a Writer, and it simply converts all JSON data to protobuf messages then calls Writer's appendRows() method to write to BigQuery tables. It maintains all Writer functions, but also provides an additional feature: schema update support, where if the BigQuery table schema is updated, users will be able to ingest data on the new schema after some time (in order of minutes).

managedwriter.Writer managedwriter

managedwriter.Writer

A BigQuery Storage API Writer that can be used to write data into BigQuery Table using the Storage API.

managedwriter

managedwriter.WriterClient

BigQuery Write API Client. The Write API can be used to write data to BigQuery.

This class provides the ability to make remote calls to the backing service through method calls that map to API methods.

For supplementary information about the Write API, see: https://cloud.google.com/bigquery/docs/write-api

managedwriter

reader.ArrowTableReader

A BigQuery Storage API Reader that can be used to read data from BigQuery Tables using the Storage API in Arrow format.

reader

reader.ReadClient

BigQuery Read API Client. The Read API can be used to read data to BigQuery.

This class provides the ability to make remote calls to the backing service through method calls that map to API methods.

For supplementary information about the Read API, see: https://cloud.google.com/bigquery/docs/read-api

reader

reader.ReadStream

ReadStream is responsible for reading data from a GRPC read stream connection against the Storage Read API readRows method.

EventEmitter reader

reader.TableReader

A BigQuery Storage API Reader that can be used to reader data into BigQuery Table using the Storage API.

reader

Variables

_default

_default: {
    v1: typeof v1;
    BigQueryReadClient: typeof v1.BigQueryReadClient;
    BigQueryWriteClient: typeof v1.BigQueryWriteClient;
    managedwriter: typeof managedwriter;
    WriterClient: typeof managedwriter.WriterClient;
    reader: typeof reader;
    ReadClient: typeof reader.ReadClient;
}

managedwriter.BufferedStream

BufferedStream = "BUFFERED"

BufferedStream is a form of checkpointed stream, that allows you to advance the offset of visible rows via Flush operations.

managedwriter

managedwriter.CommittedStream

CommittedStream = "COMMITTED"

CommittedStream appends data immediately, but creates a discrete stream for the work so that offset tracking can be used to track writes.

managedwriter

managedwriter.DefaultStream

DefaultStream = "DEFAULT"

DefaultStream most closely mimics the legacy bigquery tabledata.insertAll semantics. Successful inserts are committed immediately, and there's no tracking offsets as all writes go into a "default" stream that always exists for a table.

managedwriter

managedwriter.PendingStream

PendingStream = "PENDING"

PendingStream is a stream in which no data is made visible to readers until the stream is finalized and committed explicitly.

managedwriter

ReadClient

ReadClient: typeof reader.ReadClient

reader.ArrowFormat

ArrowFormat: DataFormat

Return data in Apache Arrow format.

reader

reader.AvroFormat

AvroFormat: DataFormat

Return data in Apache Avro format.

reader

WriterClient

WriterClient: typeof managedwriter.WriterClient

Functions

adapt.convertBigQuerySchemaToStorageTableSchema(schema)

export declare function convertBigQuerySchemaToStorageTableSchema(schema: ITableSchema): StorageTableSchema;

Converts a bigquery Schema into the protobuf-based TableSchema used by the BigQuery Storage WriteClient.

Parameter
Name Description
schema ITableSchema

a BigQuery TableSchema StorageTableSchema
Returns
Type Description
StorageTableSchema

adapt.convertStorageSchemaToProto2Descriptor(schema, scope, opts)

export declare function convertStorageSchemaToProto2Descriptor(schema: TableSchema, scope: string, ...opts: AdaptOption[]): DescriptorProto;

Builds a DescriptorProto for a given table schema using proto2 syntax.

If a column name is not compatible as a protobuf field name, it will be replaced with a placeholder name. The placeholder name is generated by removing all invalid characters from the column name and replacing all dashes with underscores. If the column name doesn't have any valid characters, we generate a placeholder name using the field number field{fieldNumber}.

If a column is required, but has a defaultValueExpression set, the resulting protobuf field will be optional, so the backend service can fill data with the given expression when no value is set.

Parameters
Name Description
schema TableSchema

a BigQuery Storage TableSchema.
scope string

scope to namespace protobuf structs.
opts AdaptOption[]
Returns
Type Description
DescriptorProto

DescriptorProto

adapt.normalizeDescriptor(dp)

export declare function normalizeDescriptor(dp: DescriptorProto): DescriptorProto;

Builds a self-contained DescriptorProto suitable for communicating schema information with the BigQuery Storage write API. It's primarily used for cases where users are interested in sending data using a predefined protocol buffer message.

Parameter
Name Description
dp DescriptorProto

DescriptorProto to be bundled. DescriptorProto
Returns
Type Description
DescriptorProto

adapt.withChangeSequenceNumber()

export declare function withChangeSequenceNumber(): AdaptOption;

Add pseudocolumn _CHANGE_SEQUENCE_NUMBER for BigQuery Change Data Capture. Used to change behavior of ordering records with same primary key. See more: https://cloud.google.com/bigquery/docs/change-data-capture#manage_custom_ordering

Returns
Type Description
AdaptOption

adapt.withChangeType()

export declare function withChangeType(): AdaptOption;

Add pseudocolumn _CHANGE_TYPE for BigQuery Change Data Capture. Used to define the type of change to be professed for each row. The pseudocolumn _CHANGE_TYPE only accepts the values UPSERT and DELETE. See more: https://cloud.google.com/bigquery/docs/change-data-capture#specify_changes_to_existing_records

Returns
Type Description
AdaptOption

managedwriter.parseStorageErrors(err)

export declare function parseStorageErrors(err: gax.GoogleError): StorageError[];

The BigQuery Storage API service augments applicable errors with service-specific details in the form of a StorageError message.

Parameter
Name Description
err gax.GoogleError
Returns
Type Description
StorageError[]

{google.cloud.bigquery.storage.v1.StorageError}

managedwriter.setLogFunction(logger)

export declare function setLogFunction(logger: ((msg: string) => void) | null): void;

Sets or disables the log function for all active Firestore instances.

Parameter
Name Description
logger ((msg: string) => void) | null

A log function that takes a message (such as console.log) or null to turn off logging.
Returns
Type Description
void

reader.setLogFunction(logger)

export declare function setLogFunction(logger: ((msg: string) => void) | null): void;

Sets or disables the log function for all active Firestore instances.

Parameter
Name Description
logger ((msg: string) => void) | null

A log function that takes a message (such as console.log) or null to turn off logging.
Returns
Type Description
void

Type Aliases

ReadClient

type ReadClient = reader.ReadClient;

reader.DataFormat

export type DataFormat = protos.google.cloud.bigquery.storage.v1.IReadSession['dataFormat'];

WriterClient

type WriterClient = managedwriter.WriterClient;