Data Intensive Applications TextBook - bepec.in

Course Content

Free Textbook Download

0/2

Data Intensive Applications TextBook

The Data Warehouse Toolkit Textbook

End-to-End Real-Time Projects & Internship Projects

0/10

Data Engineering Project Pipeline using AWS, S3, PySpark & SnowFlake

AirFlow, AWS S3, Glue, PySpark & Amazon RedShift Data Engineering Project Pipeline

AWS S3, Glue, PySpark, Athena & Quick Sight Project Pipeline

Kafka, EMR, PySpark & Amazon RedShift End-to-End Project

HDFS Data Lake, PySpark & MongoDB Project Pipeline

Star Schema & SnowFlake Schema Design using SnowFlake Data Warehouse Pipeline

Kafka, Azure Data Factory, Azure Databricks & Synapse Analytics Project Pipeline

Automatic Ingestion using SnowPipe, S3 & SnowFlake

SnowPark, SnowFlake, Stored Procedures Data Science Project Pipeline Design

SnowFlake Cron Jobs, Aggregations using SQL & Reporting using Power BI Pipeline

Introduction to Data Warehouse

0/6

What Is Data Warehouse?

Important Data Warehouse Concepts

Difference between OLAP vs OLTP

Data Ecosystem Mindmap

Overview of Data Warehouse

Data Warehouse Architecture

Mastering Python

0/17

Introduction to Python

Structure of Any Programming Language like Value, Variables, Functions & Libraries

Different In-Built Data Structures like List, Tuple, Set & Dict

Mastering Conditional Flow

Loops like While & For

List Comprehension

Mastering User-Defined Functions

Regular Expressions and Lambda

Strings and Arrays

Procedural Programming vs OOPS

Different Libraries like Pandas & Numpy

Instance Variable, Class Variable, Class Method

Association vs Composition & Aggregation

Encapsulation, Inheritance

Polymorphism, Method OverLoading, Method Overriding

Introduction to SnowFlake

0/4

What is Snowflake?

SnowFlake Architecture

Snowflake Virtual Warehouse

SnowFlake Object Hierarchy

Mastering MySQL

0/27

Introduction to MySQL

Creating Database in MySql

Star Schema and SnowFlake Schema Design

Forward and Reverse Engineering

Creating ER diagrams

Deep Dive into MySQL

SELECT, UPDATE, DELETE Operations

Clauses in MySql Part-1

Clauses in MySql Part-2

Clauses in MySQL Part-3

MySQL Data Handling Part-1

MySQL Data Handling Part-2

Data Integrity & Referential Integrity

Data Normalisation

First & Second Normal Form

Functional Dependency, Transitive Dependency & 3rd Normal Form

Boyce-Codd Normal Form

Denormalization

Temporary Table, Common Table Expression, Recursive CTE

When to use Temporary Table, CTE & Recursive CTE

Subquery in MySQL

Stored Functions

Stored Procedures

Triggers in MySQL

Create Events in MySQL

Different Functions in MySQL

Big Data & Streaming Tools

0/10

Introduction to Big Data

Distributed Computing & Hadoop Architecture

Storing Data on Hadoop -HDFS

Processing – MapReduce

Working on YARN

Working on Sqoop

Working on Impala

Working with Flume

Working with Zookeeper

Big Data Scheduling

0/2

Big Data Scheduling with Airflow

Mastering HIVE

0/5

Introduction to HIVE

HIVE Architecture

Data Storage in HIVE

Loading Data into Partitions

Big Data on Cloud

0/5

Introduction to Cloud – Models, Service Categories, AWS security, IAM

Databases on AWS

AWS Data Collection & DataLake

0/10

Data Warehousing on AWS

DynamoDB, Lambda

AWS Data pipelines and data nodes

Activity, Pre-condition and Schedule

Introduction to DataLake

Learn about Data Lake vs. Data Warehouse

Key components of a Data Lake Architecture

Query files directly using SQL

Amazon Kinesis

0/5

Amazon Kinesis and Kinesis Stream

Kinesis data stream architecture and components

Data producer and Consumer

Kinesis stream emitting data to AWS services and Kinesis connector library

Kinesis firehose

Data Loading/Ingestion and Extraction

0/9

What is Data Ingestion?

Data Ingestion in Real-World

Create an Integration Object to Connect Snowflake with AWS S3

Ingesting CSV from S3 to Snowflake

Ingest JSON from S3 to Snowflake

How to do Continuous Data Ingestion in Snowflake

SnowPipe: Implementation of Snow Pipe

SnowPipe: Billing Estimation and Key Considerations for Data Ingestion

Unloading of Data from Snowflake to S3

Big Data on Azure

0/7

Introduction to Cloud – Models, Service Categories

Overview of Azure Synapse and Architecture

Azure SQL, Creating SQL Pool

Data Warehousing on Azure

Azure SQL Data Warehouse, HDInsight

Cosmos, Azure Functions

Azure VM, Blob Storage

DataBricks and Data Factory

0/13

Azure Stream Analytics

Learn how to input data into Azure ASA

Learn how to create queries and data analysis pipelines with Azure ASA

Learn how to output the results of the data analysis pipeline to various output formats

Learn how to work with Azure ASA and Microsoft PowerBI

Azure Data Factory

Microsoft Azure Data Lake

Different tools and scenarios to ingest data in to Data Lake

Data Lifecycle and Architecture around Data Lake

Use Azure Databricks and HDInsight to process data in ADLS

Monitor the performance of your Data lake

Azure Data Factory

Azure Functions

PySpark Mastering

0/8

Core PySpark architecture

Deployment modes

RDD and RDD Operations, DAG

Dataframes, Hive Integration, data manipulation with PySpark SQL

PySpark Streaming

Structured Streaming

Amazon EMR and Running all Hadoop jobs on AWS EMR

Kafka

0/14

Kafka Architecture

Kafka Components

Where is Kafka Used?

Kafka Installation

Types of Kafka Clusters

Configuring Single Node Single Broker Cluster

Kafka Internals & Load Balancing

Mastering MongoDB

0/10

Introduction to Mongo DB

Installation of MongoDB

Basics Operations of MongoDB

Create Operations in MongoDB

Update Operations in MongoDB

Read Operations in MongoDB

Delete Operations in MongoDB

Query & Projection Operations

Python with MongoDB

Spark with MongoDB

Tables in Snowflake

0/6

Different Types of Tables in Snowflake

Creating Tables in Snowflake

Materialised Views, Views & Secure Views in Snowflake

Creating Views in Snowflake

Creating Secure Views in Snowflake

In-depth about Views in Snowflake

Tasks and Query Scheduling

0/3

Introduction to Tasks

Creating Standalone and Dependent Tree of Tasks

Billing and Query History for Tasks

Streams and Change Data Capture

0/8

Introduction to Streams

Implementation of Standard Streams

Implementation of Append-Only Streams

Streams in a Transaction

Data Retention and Staleness

Change Tracking Using ” Changes “

End-to-End Stream Creation

End-to-End Pipeline Demonstration

Snowflake – External Functions

0/8

Introduction to External Functions

What is AWS Lambda Functions?

Write & Deploy AWS Lambda Function

What is IAM Role in AWS?

Creating IAM Role in AWS

Create API Gateway

Secure and Deploy API Gateway

Create External Function in Snowflake

Snowflake with Python, Spark, and Airflow on AWS

0/13

Connect Python with Snowflake in Your Local Machine

Introduction to AWS Glue

Deploy and Execute Python Script to AWS Glue

Parameterize Your Python Script on AWS Glue

Python Pandas with Snowflake on AWS Glue

What Is Pushdown in Spark 3.1?

What is Spark? Introduction to PySpark

Deploy a PySpark Script Using AWS Glue

What is Airflow?

Set Up Managed Airflow Cluster on AWS

Configure Snowflake Connectivity in Airflow

Deploy a PySpark Transformation job in AWS Glue

Set Up Airflow DAG

Real-Time Streaming with Kafka and Snowflake

0/8

Why to go with Real-Time Streaming

Introduction to Kafka

Download the Necessary JAR Files

Set Up Kafka in your local system

Set Up Kafka Snowflake Connector

Set Up Encryption Keys for Kafka-Snowflake Connectivity

Streaming Data in Action

Snowflake – Data Protection and Governance

0/5

What Is Time Travel and Failsafe in Snowflake?

Time Travel and Data Recovery

Column Level Dynamic Data Masking

What Is Row Level Security?

Create and Implement Row Level Access Policy

Data Engineer Course with Career Transition

Exercise Files

Book-2Designing-data-intensive-applications.pdf

Size: 23.82 MB

0% Complete