Keeping objects secure

Introduction to AWS Boto in Python

Maksim Pecherskiy

Data engineer

Why care about permissions?

df = pd.read_csv('https://gid-staging.potholes.csv')

Permissions overview

Introduction to AWS Boto in Python

Why care about permissions?

Permission Allowed!

# Generate the boto3 client for interacting with S3
s3 = boto3.client('s3', region_name='us-east-1', 
                         aws_access_key_id=AWS_KEY_ID, 
                         aws_secret_access_key=AWS_SECRET)

# Use client to download a file
s3.download_file(
  Filename='potholes.csv',
  Bucket='gid-requests', 
  Key='potholes.csv')
Introduction to AWS Boto in Python

AWS Permissions Systems

IAM

Bucket Policy

ACL

Presigned URL

Introduction to AWS Boto in Python

AWS Permissions Systems

IAM

Bucket Policy

ACL

Presigned URL

Introduction to AWS Boto in Python

ACLs

ACL

Introduction to AWS Boto in Python

ACLs

Upload File

s3.upload_file(
  Filename='potholes.csv', Bucket='gid-requests', Key='potholes.csv')

Set ACL to 'public-read'

s3.put_object_acl(
  Bucket='gid-requests', Key='potholes.csv', ACL='public-read')
Introduction to AWS Boto in Python

Setting ACLs on upload

Upload file with 'public-read' ACL

s3.upload_file(
  Bucket='gid-requests', 
  Filename='potholes.csv', 
  Key='potholes.csv', 
  ExtraArgs={'ACL':'public-read'})
Introduction to AWS Boto in Python

Accessing public objects

S3 Object URL Template

https://{bucket}.{key}

URL for Key='2019/potholes.csv'

https://gid-requests.2019/potholes.csv
Introduction to AWS Boto in Python

Generating public object URL

Generate Object URL String

url = "https://{}.{}".format(
  "gid-requests", 
  "2019/potholes.csv")

'https://gid-requests.2019/potholes.csv'

# Read the URL into Pandas
df = pd.read_csv(url)
Introduction to AWS Boto in Python

How access is decided

Access hierarchy

Introduction to AWS Boto in Python

How access is decided

Access Hierarchy

Introduction to AWS Boto in Python

Review

IAM

Bucket Policy

ACL

Presigned URL

Introduction to AWS Boto in Python

Review

Set ACL to 'public-read'

s3.put_object_acl(
  Bucket='gid-requests', Key='potholes.csv', ACL='public-read')

Set ACL to 'private'

s3.put_object_acl(
  Bucket='gid-requests', Key='potholes.csv', ACL='private')
Introduction to AWS Boto in Python

Review

Upload file with 'public-read' ACL

s3.upload_file(
  Bucket='gid-requests', 
  Filename='potholes.csv', 
  Key='potholes2.csv', 
  ExtraArgs={'ACL':'public-read'})
Introduction to AWS Boto in Python

Review

Generate Object URL String

url = "https://{}.{}".format(
  "gid-requests", 
  "2019/potholes.csv")

'https://gid-requests.2019/potholes.csv'

# Read the URL into Pandas
df = pd.read_csv(url)
Introduction to AWS Boto in Python

Let's practice!

Introduction to AWS Boto in Python

Preparing Video For Download...