Data ingestion in BigQuery

Introduzione a BigQuery

Matt Forrest

Field CTO

Methods of loading data

  1. Loading data in the BigQuery Studio
  2. Using the bq command line tools
  3. Using the LOAD DATA command in SQL
Introduzione a BigQuery

Using the BigQuery Studio

Data loading interface in BigQuery

Introduzione a BigQuery

Using the BigQuery Studio

Data loading interface in BigQuery

Introduzione a BigQuery

Using the BigQuery Studio

Data loading interface in BigQuery

Introduzione a BigQuery

Using the bq command line tools

Example using the bq command line

bq load \
    dataset.table \

    gs://mybucket/mydata.csv \

    --source_format=CSV \ 
    --autodetect
1 https://cloud.google.com/bigquery/docs/bq-command-line-tool#loading_data
Introduzione a BigQuery

Using LOAD DATA in SQL

An example LOAD DATA statement:

LOAD DATA INTO dataset.table

  FROM FILES(
    uris = ['gs://mybucket/mydata.csv']

    format='CSV',
    skip_leading_rows=1
  )
Introduzione a BigQuery

Data ingestion considerations

Using LOAD DATA

  1. Cannot use local data.
  2. Subject to load data limits.

Using bq and BigQuery Studio

  1. Local data can be used, but files must be under 100 MB.
  2. Subject to load data limits.
1 https://cloud.google.com/bigquery/quotas#load_jobs
Introduzione a BigQuery

Let's practice!

Introduzione a BigQuery

Preparing Video For Download...