This document includes the best practices and guidelines for BigQuery when running generative AI workloads on Google Cloud. Use BigQuery with Vertex AI to store data. Using BigQuery with Vertex AI can significantly enhance your ML workflow because you can simplify data access, enable scalable analysis, and use its ML capabilities.
Consider the following use cases for BigQuery with Vertex AI:
- Seamless integration: BigQuery and Vertex AI are tightly integrated, letting you access and analyze your data directly within the Vertex AI platform. This integration eliminates the need for data movement, streamlines your ML workflow, and reduces friction.
- Scalable data analysis: BigQuery offers a petabyte-scale data warehouse, letting you analyze massive datasets without worrying about infrastructure limitations. This scalability is critical for training and deploying ML models that require vast amounts of data.
- SQL-based ML: BigQuery ML lets you use familiar SQL commands to train and deploy models directly within BigQuery. This feature lets data analysts and SQL practitioners use ML capabilities without requiring advanced coding skills.
- Online and batch predictions: BigQuery ML supports online and batch predictions. You can run real-time predictions on individual rows or generate predictions for large datasets in batch mode. This flexibility permits diverse use cases with varying latency requirements.
- Reduced data movement: With BigQuery ML, you don't need to move your data to separate storage or compute resources for model training and deployment. This reduced movement simplifies your workflow, reduces latency, and minimizes cost associated with data transfer.
- Model monitoring: Vertex AI provides comprehensive model monitoring capabilities, letting you track the performance, fairness, and explainability of your BigQuery ML models. Model monitoring helps you ensure that your models are performing as expected and address potential issues.
- Pretrained models: Vertex AI offers access to pretrained models, including those for natural language processing and computer vision. You can use these models within BigQuery to enhance your analysis and extract deeper insights from your data.
- Cost-effective solution: BigQuery ML offers a cost-effective, flexible way to train and deploy ML models. You only pay for the resources you use, making it an affordable option for organizations of all sizes.
- Advanced analytics capabilities: BigQuery provides tools for advanced analytics, including geospatial analysis and forecasting. These tools let you combine ML with other analytical techniques for deeper data exploration and richer insights.
- Enhanced collaboration: By using BigQuery with Vertex AI, data scientists, ML engineers, and analysts can collaborate seamlessly on ML projects. This collaboration helps create a more integrated and efficient approach to tackling complex data problems.
Required BigQuery controls
The following controls are strongly recommended when using BigQuery.
Ensure BigQuery datasets aren't publicly readable or set to allAuthenticatedUsers
| Google control ID | BQ-CO-6.1 |
|---|---|
| Category | Required |
| Description | Restrict access to the information in a BigQuery dataset to specific users only. To configure this protection, you must set up detailed roles. |
| Applicable products |
|
| Path | cloudasset.assets/assetType |
| Operator | == |
| Value |
|
| Type | String |
| Related NIST-800-53 controls |
|
| Related CRI profile controls |
|
| Related information |
Ensure BigQuery tables aren't publicly readable or set to allAuthenticatedUsers
| Google control ID | BQ-CO-6.2 |
|---|---|
| Category | Required |
| Description | Restrict access to the information in a BigQuery table to specific users only. To configure this protection, you must set up detailed roles. |
| Applicable products |
|
| Path | cloudasset.assets/iamPolicy.bindings.members |
| Operator | anyof |
| Value |
|
| Type | String |
| Related NIST-800-53 controls |
|
| Related CRI profile controls |
|
| Related information |
Optional BigQuery controls
These controls are optional. Consider enforcing them when they apply to your specific use cases.
Encrypt individual values in a BigQuery table
| Google control ID | BQ-CO-6.3 |
|---|---|
| Category | Optional |
| Description | If your organization requires that you encrypt individual values within a BigQuery table, use the Authenticated Encryption with Associated Data (AEAD) encryption functions. |
| Applicable products |
|
| Related NIST-800-53 controls |
|
| Related CRI profile controls |
|
| Related information |
Use authorized views for BigQuery datasets
| Google control ID | BQ-CO-6.4 |
|---|---|
| Category | Optional |
| Description | Authorized views let you share a subset of data in a dataset to specific users. For example, an authorized view lets you share query results with particular users and groups without giving them access to the underlying source data. |
| Applicable products |
|
| Related NIST-800-53 controls |
|
| Related CRI profile controls |
|
| Related information |
Use BigQuery column-level security
| Google control ID | BQ-CO-6.5 |
|---|---|
| Category | Optional |
| Description | Use BigQuery column-level security to create policies that check at query time whether a user has proper access. BigQuery provides fine-grained access to sensitive columns using policy tags or type-based classification of data. |
| Applicable products |
|
| Related NIST-800-53 controls |
|
| Related CRI profile controls |
|
| Related information |
Use BigQuery row-level security
| Google control ID | BQ-CO-6.6 |
|---|---|
| Category | Optional |
| Description | Use row-level security and access policies to enable fine-grained access control to a subset of data in a BigQuery table. |
| Applicable products |
|
| Related NIST-800-53 controls |
|
| Related CRI profile controls |
|
| Related information |
Use BigQuery resource charts
| Google control ID | BQ-CO-7.1 |
|---|---|
| Category | Optional |
| Description | BigQuery resource charts let BigQuery administrators observe how their organization, folder, or reservation uses BigQuery slots and how their queries perform. |
| Applicable products |
|
| Related NIST-800-53 controls |
|
| Related CRI profile controls |
|
| Related information |
What's next
Review Cloud Billing controls.
See more Google Cloud security best practices and guidelines for generative AI workloads.