Sign in

Follow along with the jupyter notebook here

I recently documented how I created a simple data engineering ETL project using twitter, python and AWS. Briefly, the project extracts data from yahoo finance and twitter, cleans it, and dumps it into an Amazon RDS instance running postgres. The script is deployed onto and EC2 instance, and runs every 30 minutes using cron.

Click here to read more about the simple beginners project (repo included).

After about 2 weeks of running, the project was successful and I managed to build up a nice amount of data. …

datadummy

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store