Scheduling your scripts with Cron

Introduction to Bash Scripting

Alex Scriven

Data Scientist

Why schedule scripts?

 

There are many situations where scheduling scripts can be useful:

  1. Regular tasks that need to be done. Perhaps daily, weekly, multiple times per day.
    • You could set yourself a calendar-reminder, but what if you forget!?
  2. Optimal use of resources (running scripts in early hours of morning)

Scheduling scripts with cron is essential to a working knowledge of modern data infrastructures.

Introduction to Bash Scripting

What is cron?

 

Cron has been part of unix-like systems since the 70's. Humans have been lazy for that long!

The name comes from the Greek word for time, chronos.

It is driven by something called a crontab, which is a file that contains cronjobs, which each tell crontab what code to run and when.

Introduction to Bash Scripting

Crontab - the driver of cronjobs

 

You can see what schedules (cronjobs) are currently programmed using the following command:

crontab -l

crontab: no crontab for user

Seems we need to make a schedule (cronjob) then!

Introduction to Bash Scripting

Crontab and cronjob structure

This great image from Wikipedia demonstrates how you construct a cronjob inside the crontab file. You can have many cronjobs, one per line.

Structure of a crontab

  • There are 5 stars to set, one for each time unit
  • The default, * means 'every'
Introduction to Bash Scripting

Cronjob example

Let's walk through some cronjob examples:

5 1 * * * bash myscript.sh

  • Minutes star is 5 (5 minutes past the hour). Hours star is 1 (after 1am). The last three are *, so every day and month
    • Overall: run every day at 1:05am.

 

15 14 * * 7 bash myscript.sh

  • Minutes star is 15 (15 minutes past the hour). Hours star is 14 (after 2pm). Next two are * (Every day of month, every month of year). Last star is day 7 (on Sundays).
    • Overall: run at 2:15pm every Sunday.
Introduction to Bash Scripting

Advanced cronjob structure

If you wanted to run something multiple times per day or every 'X' time increments, this is also possible:

  • Use a comma for specific intervals. For example:
    • 15,30,45 * * * * will run at the 15,30 and 45 minutes mark for whatever hours are specified by the second star. Here it is every hour, every day etc.
  • Use a slash for 'every X increment'. For example:
    • */15 * * * * runs every 15 minutes. Also for every hour, day etc.
Introduction to Bash Scripting

Your first cronjob

Let's schedule a script called extract_data.sh to run every morning at 1.30am. Your steps are as follows:

  1. In terminal type crontab -e to edit your list of cronjobs.
    • It may ask what editor you want to use. nano is an easy option and a less-steep learning curve than vi (vim).
  2. Create the cronjob:
    • 30 1 * * * extract_data.sh
Introduction to Bash Scripting

Your first cron job

3. Exit the editor to save it

If this was using nano (on Mac) you would use ctrl + o then enter then ctrl + x to exit.

You will see a message crontab: installing new crontab

4. Check it is there by running crontab -l.

30 1 * * * extract_data.sh

Nice work!

Introduction to Bash Scripting

Let's practice!

Introduction to Bash Scripting

Preparing Video For Download...