Keeping objects secure

Einführung in AWS Boto mit Python

Maksim Pecherskiy

Data engineer

Why care about permissions?

df = pd.read_csv('https://gid-staging.potholes.csv')

Permissions overview

Einführung in AWS Boto mit Python

Why care about permissions?

Permission Allowed!

# Generate the boto3 client for interacting with S3
s3 = boto3.client('s3', region_name='us-east-1', 
                         aws_access_key_id=AWS_KEY_ID, 
                         aws_secret_access_key=AWS_SECRET)

# Use client to download a file
s3.download_file(
  Filename='potholes.csv',
  Bucket='gid-requests', 
  Key='potholes.csv')
Einführung in AWS Boto mit Python

AWS Permissions Systems

IAM

Bucket Policy

ACL

Presigned URL

Einführung in AWS Boto mit Python

AWS Permissions Systems

IAM

Bucket Policy

ACL

Presigned URL

Einführung in AWS Boto mit Python

ACLs

ACL

Einführung in AWS Boto mit Python

ACLs

Upload File

s3.upload_file(
  Filename='potholes.csv', Bucket='gid-requests', Key='potholes.csv')

Set ACL to 'public-read'

s3.put_object_acl(
  Bucket='gid-requests', Key='potholes.csv', ACL='public-read')
Einführung in AWS Boto mit Python

Setting ACLs on upload

Upload file with 'public-read' ACL

s3.upload_file(
  Bucket='gid-requests', 
  Filename='potholes.csv', 
  Key='potholes.csv', 
  ExtraArgs={'ACL':'public-read'})
Einführung in AWS Boto mit Python

Accessing public objects

S3 Object URL Template

https://{bucket}.{key}

URL for Key='2019/potholes.csv'

https://gid-requests.2019/potholes.csv
Einführung in AWS Boto mit Python

Generating public object URL

Generate Object URL String

url = "https://{}.{}".format(
  "gid-requests", 
  "2019/potholes.csv")

'https://gid-requests.2019/potholes.csv'

# Read the URL into Pandas
df = pd.read_csv(url)
Einführung in AWS Boto mit Python

How access is decided

Access hierarchy

Einführung in AWS Boto mit Python

How access is decided

Access Hierarchy

Einführung in AWS Boto mit Python

Review

IAM

Bucket Policy

ACL

Presigned URL

Einführung in AWS Boto mit Python

Review

Set ACL to 'public-read'

s3.put_object_acl(
  Bucket='gid-requests', Key='potholes.csv', ACL='public-read')

Set ACL to 'private'

s3.put_object_acl(
  Bucket='gid-requests', Key='potholes.csv', ACL='private')
Einführung in AWS Boto mit Python

Review

Upload file with 'public-read' ACL

s3.upload_file(
  Bucket='gid-requests', 
  Filename='potholes.csv', 
  Key='potholes2.csv', 
  ExtraArgs={'ACL':'public-read'})
Einführung in AWS Boto mit Python

Review

Generate Object URL String

url = "https://{}.{}".format(
  "gid-requests", 
  "2019/potholes.csv")

'https://gid-requests.2019/potholes.csv'

# Read the URL into Pandas
df = pd.read_csv(url)
Einführung in AWS Boto mit Python

Let's practice!

Einführung in AWS Boto mit Python

Preparing Video For Download...