Weekly Video Updates

Mentor:

Shawn Deggans

https://www.linkedin.com/in/shawn-deggans/

Objective:

To develop a flexible, scalable, and cloud-agnostic data platform leveraging Databricks and Delta Lake, providing a foundation for real-world data processing, analytics, and machine learning tasks.

Purpose:

• Strategic Enhancement: Demonstrate how the platform serves as a strategic asset for Applied Curiosity, enhancing our data solution offerings across various industries.
• Operational Efficiency: Highlight the commitment to leveraging advanced, cloud- agnostic solutions that provide flexibility, scalability, and operational efficiency Innovation and
• Value Delivery: Underline the platform's role in fostering innovation, accelerating the data-to-insight cycle, and delivering actionable insights to clients. Approach
• Cloud Agnosticism: Design the platform to be deployable across AWS, Azure, and GCP, ensuring operational flexibility and cost-effectiveness.
• Integration of Leading Technologies: Utilize Databricks for analytics and machine learning, and Delta Lake for reliable data storage. Infrastructure as Code (IaC): Adopt Pulumi for scripting infrastructure setups, enabling repeatable and consistent deployments using Python as the primary programming language.

Tools and Services

1. Databricks: For data processing, analytics, and collaborative data science workspaces.
2. Delta Lake: To bring reliability to data lakes, managed within cloud providers' storage services.
3. Pulumi: As the IaC tool to manage cloud resources, ensuring quick deployment and infrastructure management.
4. Additional Tools: Integration with Git repositories for version control and CI/CD processes.

Requirements