Project Overview

  • In this project I was involved in setting up the cloud architecture from scratch on AWS, to support data and ML projects
  • This involved setting up ETL jobs involving S3, Glue, Athena and Airflow. And analytics involving Quicksight, jupyter notebooks etc.
  • The architecture was designed keeping in mind scale, speed, security, ease and varieties of use of the data
    • Scale: Parallelization and scaling abilities enabled through both the selection of services and columnar data formats
    • Speed: Data modeling to improve speed of querying and DAG task design to enable speed and modularity of execution on Airflow
    • Security: IAM, VPN, Bucket Policies and User Group controls setup to ensure granular access to data backend
    • Ease and Varieties of Use: Enabling quick querying through Athena, dashboarding through Quicksight and further ML modeling support through the versatility of stored data in terms of access methods, ability for data transformation and support for advanced analytics
  • With the solid data platform foundations, the project smoothly proceeded on to advanced analytics and NLP use cases
  • I had the responsibilty or research, modeling and API development for these analytics and machine learning projects
  • In addition to technical functionalities, this work also enabled market readiness to work on more sophisticated products

Skills

Solution Architect

Though at this point, I had only a Machine Learning Specialty certifications on AWS, GCP and Azure, I picked up Solution Architecting responsibilities and skills working on this project, which involved security and budget aspects in addition to the technical abilities of the architecture

Hands-on Technical Leadership

In addition to researching and deciding on the tech stack and architectural plans for improved Data and ML literacy, I was also the sole owner and implementer of these pipelines end-to-end, helping me validate and be confident about the direction I had set for us

Strategic ML

There was no doubt that our organization had to be data literate and get into advanced analytics to make our mark in the market - which is why I was brought in. In addition to that, as a lean team with several client commitments, it was important to identify, plan and undertake strategic analytics projects