Data ingestion in BigQuery

Introduction to BigQuery

Matt Forrest

Field CTO

Methods of loading data

  1. Loading data in the BigQuery Studio
  2. Using the bq command line tools
  3. Using the LOAD DATA command in SQL
Introduction to BigQuery

Using the BigQuery Studio

Data loading interface in BigQuery

Introduction to BigQuery

Using the BigQuery Studio

Data loading interface in BigQuery

Introduction to BigQuery

Using the BigQuery Studio

Data loading interface in BigQuery

Introduction to BigQuery

Using the bq command line tools

Example using the bq command line

bq load \
    dataset.table \

    gs://mybucket/mydata.csv \

    --source_format=CSV \ 
    --autodetect
1 https://cloud.google.com/bigquery/docs/bq-command-line-tool#loading_data
Introduction to BigQuery

Using LOAD DATA in SQL

An example LOAD DATA statement:

LOAD DATA INTO dataset.table

  FROM FILES(
    uris = ['gs://mybucket/mydata.csv']

    format='CSV',
    skip_leading_rows=1
  )
Introduction to BigQuery

Data ingestion considerations

Using LOAD DATA

  1. Cannot use local data.
  2. Subject to load data limits.

Using bq and BigQuery Studio

  1. Local data can be used, but files must be under 100 MB.
  2. Subject to load data limits.
1 https://cloud.google.com/bigquery/quotas#load_jobs
Introduction to BigQuery

Let's practice!

Introduction to BigQuery

Preparing Video For Download...