Studi Kasus: Membuat Repository Laporan

Pengantar AWS Boto di Python

Maksim Pecherskiy

Data Engineer

Produk akhir

Langkah-langkah

Siapkan data

Unduh file bulanan dari bucket data mentah
Satukan menjadi satu CSV
Buat DataFrame teragregasi

Langkah-langkah

Buat laporan

Tulis DataFrame ke CSV dan HTML
Hasilkan plot Bokeh, simpan sebagai HTML

Langkah-langkah

Unggah laporan ke situs yang dapat dibagikan

Buat bucket gid-reports
Unggah ketiga file bulanan ke S3
Buat file index.html yang mencantumkan semua file
Dapatkan URL situs!

Bucket data mentah

File privat
CSV harian dari permintaan App
Data mentah

Bucket permintaan Gid

Baca file data mentah

# Create list to hold our DataFrames
df_list = []


# Request the list of csv's from S3 with prefix; Get contents
response = s3.list_objects(
  Bucket='gid-requests', 
  Prefix='2019_jan')


# Get response contents
request_files = response['Contents']

Baca file data mentah

# Iterate over each object
for file in request_files:
    obj = s3.get_object(Bucket='gid-requests', Key=file['Key'])


    # Read it as DataFrame
    obj_df = pd.read_csv(obj['Body'])


    # Append DataFrame to list
    df_list.append(obj_df)

Baca file data mentah

# Concatenate all the DataFrames in the list
df = pd.concat(df_list)


# Preview the DataFrame
df.head()

Pratinjau DataFrame

Buat laporan agregat

Lakukan agregasi
df.to_csv('jan_final_report.csv')
df.to_html('jan_final_report.html')
jan_final_chart.html

Laporan teragregasi

Bucket laporan

Situs bucket
Dapat diakses publik
Data teragregasi dan laporan HTML

Bucket Laporan

Unggah CSV agregat

# Upload Aggregated CSV to S3
s3.upload_file(Filename='./jan_final_report.csv', 
                      Key='2019/jan/final_report.csv', 
                      Bucket='gid-reports',
                      ExtraArgs = {'ACL': 'public-read'})

Unggah tabel HTML

# Upload HTML table to S3
s3.upload_file(Filename='./jan_final_report.html', 
                      Key='2019/jan/final_report.html', 
                      Bucket='gid-reports',
                      ExtraArgs = {
                        'ContentType': 'text/html',
                        'ACL': 'public-read'})

Unggah grafik HTML

# Upload Aggregated Chart to S3
s3.upload_file(Filename='./jan_final_chart.html', 
                      Key='2019/jan/final_chart.html', 
                      Bucket='gid-reports',
                      ExtraArgs = {
                        'ContentType': 'text/html',
                        'ACL': 'public-read'})

Laporan terunggah

Buat index.html

# List the gid-reports bucket objects starting with 2019/
r = s3.list_objects(Bucket='gid-reports', Prefix='2019/')


# Convert the response contents to DataFrame
objects_df = pd.DataFrame(r['Contents'])


# Create a column "Link" that contains website url + key
base_url = "https://gid-reports."
objects_df['Link'] = base_url + objects_df['Key']

Buat index.html

# Write DataFrame to html
objects_df.to_html('report_listing.html', 
                   columns=['Link', 'LastModified', 'Size'],
                   render_links=True)

Tulis DF ke html

Unggah index.html

# Upload the file to gid-reports bucket root.
s3.upload_file(
  Filename='./report_listing.html', 
  Key='index.html', 
  Bucket='gid-reports',
  ExtraArgs = {
    'ContentType': 'text/html', 
    'ACL': 'public-read'
  })

Dapatkan URL index!

URL situs bucket *


"http://gid-reports.index.html"

Mari kita ubah sedikit!

Pengantar AWS Boto di Python