Ekstraksi

Pengantar Data Engineering

Vincent Vankrunkelsven

Data Engineer, DataCamp

Mengekstrak data: apa artinya?

Diagram yang merepresentasikan fase ekstraksi sederhana

Ekstrak dari berkas teks

Tidak terstruktur

Teks biasa
Mis. bab dari buku

Call me Ishmael. Some years ago, never 
mind how long precisely, having little or
no money in my purse, and nothing particular
to interest me on shore, I thought ....

Berkas flat

Baris = rekam
Kolom = atribut
Mis. .tsv atau .csv

Year,Make,Model,Price
1997,Ford,E350,3000.00
1999,Chevy,"Venture Extended Edition",4900.00
1999,Chevy,"Venture Extended Edition",5000.00
1996,Jeep,Grand Cherokee,4799.00

JSON

JavaScript Object Notation
Semi-terstruktur
Atomik
- number
- string
- boolean
- null
Komposit
- array
- object

{
  "an_object": {
    "nested": [
      "one",
      "two",
      "three",
      {
        "key": "four"
      }
    ]
  }
}

import json

result = json.loads('{"key_1": "value_1", 
                      "key_2":"value_2"}')

print(result["key_1"])

value_1

Data di Web

Permintaan (request)

Diagram yang merepresentasikan model request–response

Contoh

Buka Google
Kirim permintaan ke Server Google
Google merespons dengan laman web

Data di Web lewat API

Kirim data dalam format JSON
API: application programming interface
Contoh
- GitHub API

{ "login": "datacamp", "type": "Organization", "created_at": "2013-05-14T13:01:20Z" }

Hackernews API

import requests


response = requests.get("https://hacker-news.firebaseio.com/v0/item/16222426.json")
print(response.json())

{'by': 'neis', 'descendants': 0, 'id': 16222426, 'score': 17, 'time': 1516800333, 'title': .... }

Data dalam database

Database aplikasi

Transaksi
Penambahan atau perubahan
OLTP
Berorientasi baris

Database analitis

OLAP
Berorientasi kolom

Ekstraksi dari database

String/URI koneksi

postgresql://[user[:password]@][host][:port]

Gunakan di Python

import sqlalchemy
connection_uri = "postgresql://repl:password@localhost:5432/pagila" 
db_engine = sqlalchemy.create_engine(connection_uri)


import pandas as pd
pd.read_sql("SELECT * FROM customer", db_engine)

Ayo berlatih!

Pengantar Data Engineering