Notebooks and programmatic analysis tools

While SQL is a powerful query language, programming languages like Python, Java, or R provide syntaxes and statistical functions that can be more expressive for data analysis tasks. Notebook environments offer a highly flexible alternative to spreadsheets for complex data exploration and machine learning (ML) workflows.

BigQuery provides built-in notebook integration as well as support for several other programmatic analysis solutions.

Colab Enterprise notebooks

You can use Colab Enterprise notebooks in BigQuery to perform end-to-end data science and ML workflows in a single, integrated interface. Unlike standard SQL editors, notebooks let you combine SQL queries with Python code, rich text, and visualizations to tell a comprehensive story with your data. Colab Enterprise notebooks are ideal for the following use cases:

End-to-end ML workflows: build, evaluate, and deploy BigQuery ML models within a single interface.
Data exploration: clean and analyze large datasets using BigQuery DataFrames.
Collaborative research: track version history and share notebooks with colleagues using Identity and Access Management (IAM).

Colab Enterprise notebooks offer the following benefits:

AI-powered development: leverage Gemini Enterprise Agent Platform for assistive code development.
Seamless Python integration: use the BigQuery DataFrames API without any additional setup.
Familiar editor features: benefit from SQL auto-completion similar to the BigQuery SQL editor.
Integrated visualizations: use interactive DataFrame visualizations or libraries like matplotlib and seaborn directly in your workflow.
SQL-Python interoperability: execute SQL in cells that reference Python variables.

Understand cells

A notebook is made up of cells that you can edit. The following types of cells are supported in Colab Enterprise notebooks:

Text cell: use a text cell to add explanations and images to your notebook in Markdown.
Code cell: use a code cell to add Python to your notebook. You can run each code cell individually. A code cell can reference any variables created in another cell that you've already run.
SQL cell: use a SQL cell to run GoogleSQL queries. The output of the query is automatically saved as a DataFrame with the same name as the title of the cell. You can run multiple SQL statements in a single SQL cell, but only the results of the last statement are saved to a DataFrame.

You can refer to Python variables in expressions or use BigQuery DataFrames as tables in your query by enclosing the variable name in braces ({ }):
```
# Refer to the Python variable my_threshold in a SQL expression.
SELECT * FROM my_dataset.my_table WHERE x > {my_threshold};

# Reference previous query results to iterate on your queries.
SELECT * FROM {df};
```
Visualization cell: use a visualization cell to automatically generate a visualization of any DataFrame in your notebook. You can modify which columns are displayed and select from various chart types and aggregations. You can also choose custom colors, data labels, and titles.

Notebook gallery

The notebook gallery is a central hub for discovering and using prebuilt notebook templates. It includes fundamental templates for SQL, Python, Apache Spark, and DataFrames to help you perform common tasks like data preparation, analysis, and visualization.

Runtime management and security

Notebooks use Colab Enterprise runtimes, which are Compute Engine virtual machines allocated to a specific user to enable code execution. Access to notebooks is controlled using IAM roles. To detect vulnerabilities in the Python packages used in your notebooks, you can install and use the Notebook Security Scanner.

Monitoring and regions

All code assets in BigQuery Studio are stored in a default region, and updating this setting changes the region for any newly created code assets. Colab Enterprise notebooks are available in all regions where BigQuery Studio is available.

To monitor notebook slot usage, you can view your Cloud Billing report and apply a filter for the goog-bq-feature-type label with the value BQ_STUDIO_NOTEBOOK.

Notebook capabilities are available only in the Google Cloud console.

Pricing

For pricing information about Colab Enterprise notebooks, see Notebook runtime pricing.

BigQuery DataFrames

BigQuery DataFrames is a set of open source Python libraries that let you take advantage of BigQuery data processing by using familiar Python APIs. BigQuery DataFrames implements the pandas and scikit-learn APIs by pushing the processing down to BigQuery through SQL conversion. This design lets you use BigQuery to explore and process terabytes of data and train ML models, all with Python APIs.

BigQuery DataFrames offers the following benefits:

More than 750 pandas and scikit-learn APIs implemented through transparent SQL conversion to BigQuery and BigQuery ML APIs.
Deferred execution of queries for enhanced performance.
Extending data transformations with user-defined Python functions to let you process data in the cloud. These functions are automatically deployed as BigQuery remote functions.
Integration with Gemini Enterprise Agent Platform to let you use Gemini models for text generation.