Course
digicode: OBQDF
Orchestrate BigQuery Workloads with Dataform
Dataform is a service for data analysts to develop, test, version control, and schedule complex SQL workflows for data transformation in BigQuery.
Duration
1 day
Price
850.–
exkl. 8.1% MWST
Course facts
- Understanding the components of Dataform core
- Creating tables and views in BigQuery using Dataform
- Documenting BigQuery tables and views
- Understanding BigQuery security settings using Dataform
- Using assertions to validate data in Dataform workflows
- Executing Dataform SQL workflows in an automated fashion
In this course you will explore the components of Dataform core, learn how to define tables and dependencies in SQLX, document BigQuery tables and views, understand BigQuery security settings and how to manage these with Dataform, write assertions, execute SQL workflows, and explore additional advanced use cases.
1 Dataform Core Components
- SQL workflow
- Repositories and workspaces
- Default files and folders
- Compiled graphs
- Understand the components of Dataflow core.
2 Table Definitions and Dependencies
- Declare a data source
- Create a table
- Create an incremental table
- Set partitioning and clustering options
- Create an empty table
- Create an external BigLake table
- Create views and materialized views
- Define dependencies
- Create tables and views in BigQuery using Dataform
3 Document BigQuery Tables and Views
- Use column descriptions
- Use globally defined JavaScript constants
- Add labels
- Document BigQuery tables and views
- Lab: Build SQL Workflows with Dependencies in Dataform
4 BigQuery Security Settings
- IAM dataset and table/view access
- Column-level security
- Row-level security
- Understand BigQuery security settings using Dataform
5 Assertions
- Use built-in assertions
- Create manual assertions
- Use assertions to validate data in Dataform workflows
- Lab: Work with Assertions and BigQuery Security Settings in Dataform
6 SQL Workflow Executions
- Dataform code lifecycle
- What happens during compilation
- Customize and schedule compilation results
- Execute workflows (UI, Cloud Scheduler, Cloud Composer)
- Logging and monitoring
- Execute Dataform SQL workflows in an automated fashion
- Lab: Automate and Monitor SQL Workflow Executions in Dataform
7 Advanced Use Cases
- Create a BigLake table after file upload using Cloud Run functions
- Build a Machine Learning pipeline with BigQuery ML
- Work with Slowly Changing Dimensions Type 2
- Explore additional use cases for Dataform
- Lab: Create a BigLake Table with Dataform Using Cloud Run Functions
Knowledge of SQL data analysis and BigQuery as discussed in BigQuery for Data Analysts