BigQuery API Connector Overview

The Workflows connector defines the built-in functions that can be used to access other Google Cloud products within a workflow.

This page provides an overview of the individual connector. There is no need to import or load connector libraries in a workflow—connectors work out of the box when used in a call step.

BigQuery API

A data platform for customers to create, manage, share and query data. To learn more, see the BigQuery API documentation.

You can use a page token in a workflow to paginate through the results of a query to a BigQuery dataset.

BigQuery connector sample

YAML

# This workflow demonstrates how to use the BigQuery connector:
# Create a dataset and then insert a table with data from a public dataset
# Delete both the table and dataset
# Expected output: "SUCCESS"
- init:
    assign:
      - project_id: ${sys.get_env("GOOGLE_CLOUD_PROJECT_ID")}
      - dataset_id: "example_dataset"
      - table_id: "example_table"
      - query: "SELECT * FROM `bigquery-public-data.usa_names.usa_1910_2013` LIMIT 5000;"
      - create_disposition: "CREATE_IF_NEEDED"  # creates table if it doesn't exist
      - write_disposition: "WRITE_TRUNCATE"  # truncates table if it already exists
- create_dataset:
    call: googleapis.bigquery.v2.datasets.insert
    args:
      projectId: ${project_id}
      body:
        datasetReference:
          datasetId: ${dataset_id}
          projectId: ${project_id}
        access[].role: "roles/bigquery.dataViewer"
        access[].specialGroup: "projectReaders"
- insert_table_into_dataset:
    call: googleapis.bigquery.v2.jobs.insert
    args:
      projectId: ${project_id}
      body:
        configuration:
          query:
            query: ${query}
            destinationTable:
              projectId: ${project_id}
              datasetId: ${dataset_id}
              tableId: ${table_id}
            create_disposition: ${create_disposition}
            write_disposition: ${write_disposition}
            allowLargeResults: true
            useLegacySql: false
- delete_table_from_dataset:
    call: googleapis.bigquery.v2.tables.delete
    args:
      projectId: ${project_id}
      datasetId: ${dataset_id}
      tableId: ${table_id}
- delete_dataset:
    call: googleapis.bigquery.v2.datasets.delete
    args:
      projectId: ${project_id}
      datasetId: ${dataset_id}
- the_end:
    return: "SUCCESS"

JSON

[
  {
    "init": {
      "assign": [
        {
          "project_id": "${sys.get_env(\"GOOGLE_CLOUD_PROJECT_ID\")}"
        },
        {
          "dataset_id": "example_dataset"
        },
        {
          "table_id": "example_table"
        },
        {
          "query": "SELECT * FROM `bigquery-public-data.usa_names.usa_1910_2013` LIMIT 5000;"
        },
        {
          "create_disposition": "CREATE_IF_NEEDED"
        },
        {
          "write_disposition": "WRITE_TRUNCATE"
        }
      ]
    }
  },
  {
    "create_dataset": {
      "call": "googleapis.bigquery.v2.datasets.insert",
      "args": {
        "projectId": "${project_id}",
        "body": {
          "datasetReference": {
            "datasetId": "${dataset_id}",
            "projectId": "${project_id}"
          },
          "access[].role": "roles/bigquery.dataViewer",
          "access[].specialGroup": "projectReaders"
        }
      }
    }
  },
  {
    "insert_table_into_dataset": {
      "call": "googleapis.bigquery.v2.jobs.insert",
      "args": {
        "projectId": "${project_id}",
        "body": {
          "configuration": {
            "query": {
              "query": "${query}",
              "destinationTable": {
                "projectId": "${project_id}",
                "datasetId": "${dataset_id}",
                "tableId": "${table_id}"
              },
              "create_disposition": "${create_disposition}",
              "write_disposition": "${write_disposition}",
              "allowLargeResults": true,
              "useLegacySql": false
            }
          }
        }
      }
    }
  },
  {
    "delete_table_from_dataset": {
      "call": "googleapis.bigquery.v2.tables.delete",
      "args": {
        "projectId": "${project_id}",
        "datasetId": "${dataset_id}",
        "tableId": "${table_id}"
      }
    }
  },
  {
    "delete_dataset": {
      "call": "googleapis.bigquery.v2.datasets.delete",
      "args": {
        "projectId": "${project_id}",
        "datasetId": "${dataset_id}"
      }
    }
  },
  {
    "the_end": {
      "return": "SUCCESS"
    }
  }
]

Module: googleapis.bigquery.v2.datasets

Functions
delete Deletes the dataset specified by the datasetId value. Before you can delete a dataset, you must delete all its tables, either manually or by specifying deleteContents. Immediately after deletion, you can create another dataset with the same name. # IAM Permissions Requires the bigquery.datasets.delete permission on the dataset.
get Returns the dataset specified by datasetID. # IAM Permissions Requires the bigquery.datasets.get permission on the dataset.
insert Creates a new empty dataset. # IAM Permissions Requires the bigquery.datasets.create permission on the project.
list Lists all datasets in the specified project to which the user has been granted the READER dataset role. # IAM Permissions Requires no specific IAM permission(s) to use this method. Results are filtered to only include datasets on which the caller has the bigquery.datasets.get permission.
patch Updates information in an existing dataset. The update method replaces the entire dataset resource, whereas the patch method only replaces fields that are provided in the submitted dataset resource. This method supports RFC5789 patch semantics. # IAM Permissions Requires the following IAM permission(s) to use this method: - bigquery.datasets.update on the dataset. - bigquery.datasets.get on the dataset.
undelete Undeletes a dataset which is within time travel window based on datasetId. If a time is specified, the dataset version deleted at that time is undeleted, else the last live version is undeleted. # IAM Permissions Requires the following IAM permission(s) to use this method: - bigquery.datasets.create on the project. - bigquery.datasets.get on the dataset.
update Updates information in an existing dataset. The update method replaces the entire dataset resource, whereas the patch method only replaces fields that are provided in the submitted dataset resource. # IAM Permissions Requires the bigquery.datasets.update permission on the dataset.

Module: googleapis.bigquery.v2.jobs

Functions
cancel Requests that a job be cancelled. This call will return immediately, and the client will need to poll for the job status to see if the cancel completed successfully. Cancelled jobs may still incur costs. # IAM Permissions Requires the bigquery.jobs.update permission on the job resource. If the user matches the creator of the job, the bigquery.jobs.create permission on the project is required instead.
delete Requests the deletion of the metadata of a job. This call returns when the job's metadata is deleted. # IAM Permissions Requires the bigquery.jobs.delete permission on the job resource.
get Returns information about a specific job. Job information is available for a six month period after creation. Requires that you're the person who ran the job, or have the Is Owner project role. # IAM Permissions Requires the bigquery.jobs.get permission on the job resource. If the user matches the creator of the job, the bigquery.jobs.create permission on the project is required instead.
getQueryResults RPC to get the results of a query job. # IAM Permissions Requires the following IAM permission(s) to use this method: - bigquery.jobs.get on the job. - bigquery.tables.getData on the destination table. If the user matches the creator of the job, the following IAM permission(s) are required instead: - bigquery.jobs.create on the project. - bigquery.tables.getData on the destination table.
insert Starts a new asynchronous job. This API has two different kinds of endpoint URIs, as this method supports a variety of use cases. * The Metadata URI is used for most interactions, as it accepts the job configuration directly. * The Upload URI is ONLY for the case when you're sending both a load job configuration and a data stream together. In this case, the Upload URI accepts the job configuration and the data as two distinct multipart MIME parts. # IAM Permissions Requires the bigquery.jobs.create permission on the project resource. Additional permissions are required depending on the job type: - Load, Export, and Copy jobs: Generally require data-level permissions such as bigquery.tables.export or access to external storage. - Query jobs: Permissions are dependent on the SQL statement. Complex queries (DDL, DCL) may require additional permissions to create reservations, modify IAM policies, or update project settings.
list Lists all jobs that you started in the specified project. Job information is available for a six month period after creation. The job list is sorted in reverse chronological order, by job creation time. Requires the Can View project role, or the Is Owner project role if you set the allUsers property. # IAM Permissions Requires no specific IAM permission(s) to use this method. Users are able to list the jobs they created. Additional access is granted based on the following permissions: - Users with the bigquery.jobs.listAll permission can list all jobs with all metadata. - Users with the bigquery.jobs.list permission can list all jobs, but with redacted information for jobs they did not create.
query Runs a BigQuery SQL query synchronously and returns query results if the query completes within a specified timeout. # IAM Permissions Requires the bigquery.jobs.create permission on the project resource. Data-level permissions are highly dependent on the SQL statement being executed. While standard queries require data access (such as bigquery.tables.getData), complex operations like DDL or DCL may require permissions to manage reservations, IAM policies, or project settings.

Module: googleapis.bigquery.v2.models

Functions
delete Deletes the model specified by modelId from the dataset. # IAM Permissions Requires the bigquery.models.delete permission on the model.
get Gets the specified model resource by model ID. # IAM Permissions Requires the bigquery.models.getMetadata permission on the model.
list Lists all models in the specified dataset. Requires the READER dataset role. After retrieving the list of models, you can get information about a particular model by calling the models.get method. # IAM Permissions Requires the bigquery.models.list permission on the dataset.
patch Patch specific fields in the specified model. # IAM Permissions Requires the bigquery.models.updateMetadata permission on the model.

Module: googleapis.bigquery.v2.projects

Functions
getServiceAccount RPC to get the service account for a project used for interactions with Google Cloud KMS. Requires the bigquery.jobs.create permission on the project resource. This permission is required to authorize the retrieval of the project's service identity for technical management tasks like encryption configuration.
list RPC to list projects to which the user has been granted any project role. Users of this method are encouraged to consider the Resource Manager API, which provides the underlying data for this method and has more capabilities.

IAM Permissions Requires no specific IAM permission(s) to use this

method. The results are filtered to only include projects on which the caller has been granted a project-level role such as a BigQuery predefined IAM role or a basic role such as Viewer or Owner.

Module: googleapis.bigquery.v2.routines

Functions
delete Deletes the routine specified by routineId from the dataset. # IAM Permissions Requires the bigquery.routines.delete permission on the routine.
get Gets the specified routine resource by routine ID. # IAM Permissions Requires the bigquery.routines.get permission on the routine.
getIamPolicy Gets the access control policy for a resource. Returns an empty policy if the resource exists and does not have a policy set.
insert Creates a new routine in the dataset. # IAM Permissions Requires the bigquery.routines.create permission on the dataset.
list Lists all routines in the specified dataset. Requires the READER dataset role. # IAM Permissions Requires the bigquery.routines.list permission on the dataset.
setIamPolicy Sets the access control policy on the specified resource. Replaces any existing policy. Can return NOT_FOUND, INVALID_ARGUMENT, and PERMISSION_DENIED errors.
testIamPermissions Returns permissions that a caller has on the specified resource. If the resource does not exist, this will return an empty set of permissions, not a NOT_FOUND error. Note: This operation is designed to be used for building permission-aware UIs and command-line tools, not for authorization checking. This operation may "fail open" without warning.
update Updates information in an existing routine. The update method replaces the entire Routine resource. # IAM Permissions Requires the bigquery.routines.update permission on the routine.

Module: googleapis.bigquery.v2.rowAccessPolicies

Functions
batchDelete Deletes provided row access policies. # IAM Permissions Requires the following IAM permission(s) on the table: - bigquery.rowAccessPolicies.delete - bigquery.rowAccessPolicies.setIamPolicy
delete Deletes a row access policy. # IAM Permissions Requires the following IAM permission(s) on the table: - bigquery.rowAccessPolicies.delete - bigquery.rowAccessPolicies.setIamPolicy
get Gets the specified row access policy by policy ID. # IAM Permissions Requires the bigquery.rowAccessPolicies.get permission on the table.
getIamPolicy Gets the access control policy for a resource. Returns an empty policy if the resource exists and does not have a policy set.
insert Creates a row access policy. # IAM Permissions Requires the following IAM permission(s) on the table: - bigquery.rowAccessPolicies.create - bigquery.rowAccessPolicies.setIamPolicy - bigquery.tables.getData
list Lists all row access policies on the specified table. # IAM Permissions Requires the bigquery.rowAccessPolicies.list permission on the table.
testIamPermissions Returns permissions that a caller has on the specified resource. If the resource does not exist, this will return an empty set of permissions, not a NOT_FOUND error. Note: This operation is designed to be used for building permission-aware UIs and command-line tools, not for authorization checking. This operation may "fail open" without warning.
update Updates a row access policy. # IAM Permissions Requires the following IAM permission(s) on the table: - bigquery.rowAccessPolicies.update - bigquery.rowAccessPolicies.setIamPolicy - bigquery.tables.getData

Module: googleapis.bigquery.v2.tabledata

Functions
insertAll Streams data into BigQuery one record at a time without needing to run a load job. # IAM Permissions Requires the following IAM permission(s) to use this method: - bigquery.tables.updateData on the table. - bigquery.tables.get on the table. - bigquery.datasets.get on the dataset.
list List the content of a table in rows. # IAM Permissions Requires the bigquery.tables.getData permission on the table.

Module: googleapis.bigquery.v2.tables

Functions
delete Deletes the table specified by tableId from the dataset. If the table contains data, all the data will be deleted. # IAM Permissions Requires the bigquery.tables.delete permission on the table.
get Gets the specified table resource by table ID. This method does not return the data in the table, it only returns the table resource, which describes the structure of this table. # IAM Permissions Requires the bigquery.tables.get permission on the table.
getIamPolicy Gets the access control policy for a resource. Returns an empty policy if the resource exists and does not have a policy set.
insert Creates a new, empty table in the dataset. # IAM Permissions Requires the bigquery.tables.create permission on the dataset.
list Lists all tables in the specified dataset. Requires the READER dataset role. # IAM Permissions Requires the bigquery.tables.list permission on the dataset.
patch Updates information in an existing table. The update method replaces the entire table resource, whereas the patch method only replaces fields that are provided in the submitted table resource. This method supports RFC5789 patch semantics. # IAM Permissions Requires the following IAM permission(s) on the table: - bigquery.tables.update - bigquery.tables.get
setIamPolicy Sets the access control policy on the specified resource. Replaces any existing policy. Can return NOT_FOUND, INVALID_ARGUMENT, and PERMISSION_DENIED errors.
testIamPermissions Returns permissions that a caller has on the specified resource. If the resource does not exist, this will return an empty set of permissions, not a NOT_FOUND error. Note: This operation is designed to be used for building permission-aware UIs and command-line tools, not for authorization checking. This operation may "fail open" without warning.
update Updates information in an existing table. The update method replaces the entire Table resource, whereas the patch method only replaces fields that are provided in the submitted Table resource. # IAM Permissions Requires the bigquery.tables.update permission on the table.