Apache Spark & Scala

You will learn about the various infrastructure layouts and understand both Development and Operations (DevOps):

• Apache Spark & Scala Download[PDF] 0/17

  • Lecture

    Module 1:Scala for Apache Spark

    • this module we will learn about fundamentals of programming language
    • Topics:
    • 1.Brief about programming languages
    • 2.Brief comparison of Scala and Java
    • 3.Brief overview of the Scala language
    • 4.How to compile a Scala program
    • 5.The Scala shell (interpreter)
    • 6.Brief overview of tooling
    • 7.developing Scala in an IDE SBT, the Scala Build Tool
    • 8.Basic Scala syntax
    • 9.Scala variables, including mutable vs. immutable values
    • 10.Basic Scala types (primitives, tuples)
    • 11.Control flow (loops, conditionals)
    • 12.Functions, and lambdas
    • 13.Scoping
    • 14.Object-oriented programming in Scala classes, traits and inheritance methods
    • 15.Scala collections and the common operations on them (the basis of the Spark RDD API)
    • 16.Type inference
    • 17.Imports
    • 18.Overview of functional vs. imperative
    • 19.programming Case classes
    • 20.Pattern matching
    • 21.For-comprehensions

    25 hour(s)
  • Lecture

    Module 2: Real time Environment

    • In this module we learn about real time production environment called Amazon web Services
    • Topics:
    • 1.Brief about aws
    • 2.Account creation
    • 3.virtual server creation(ec2)
    • 4.Installation of apache spark on ec2
    • 5.Brief understanding of aws services which are mostly used by Data Engineer/ Data Scientist.

    25 hour(s)
  • Lecture

    Module 3:Apache Spark Core

    • In this module will understand why Apache Spark ? and the difference between spark and mapreduce and the fundamentals of apache spark with real time examples.
    • Topics:
    • 1.Spark Capabilities and Ecosystem
    • 2.Basic Spark Components
    • 3.Resilient Distributed Datasets (RDD) Fundamentals.
    • 4.Purpose and Structure of RDDs
    • 5.Operation Resilient Distributed Datasets (RDD).
    • 6.Transformations and actions (map, flatMap, filter, reduce, reduceByKey ,etc). In dept of RDD programming API
    • 
      
    • Solving real time case studies using above learning.
    • a)Analysis of crime dataset using above learning
    • b)Analysis of adult dataset.
    • c)Getting insight financial data.
    • There will be 4 case study assignments with proper problem statement and dataset.

    25 hour(s)
  • Lecture

    Module 4 : SparkSQL and DataFrames

    • In this module will learn in detail of apache spark with sql.
    • Topics:
    • 1.Spark SQL and DataFrame Uses
    • 2.Creating DataFrames
    • 3.Reading csv,text,parquet and json dataset.
    • 4.Query with DataFrame API and SQL
    • 5.Joining data frames.
    • 6.Caching and Re-using DataFrames
    • 7.Caching and Re-using DataFrames
    • 8.Solving Case study using above learning.
    • There will be lots of case studies based on student interested domains(bank,insurance etc) on open source dataset available

    25 hour(s)
  • Lecture

    Module 5 : Apache Spark Internals

    • This module is advanced part of apache spark.
    • Topics:
    • 1.In-depth discussion of spark architecture.
    • 2.Jobs, Stages, and Tasks
    • 3.Partitions and Shuffles
    • 4.Data Locality
    • 5.Job Performance
    • 6.Visualizing DAG Execution
    • 7.Observing Task Scheduling
    • 8.Analyze Spark jobs using the administration UIs

    25 hour(s)
  • Lecture

    Module 6:Apache spark Streaming

    • This module is for handling streaming data using apache spark
    • Topics:
    • 1.Creating DStreams from Sources
    • 2.Operating on DStream Data
    • 3.Reading from TCP
    • 4.Reading from Kafka

    25 hour(s)
  • Lecture

    Module 7:

    • Best practice and optimisation based on Students solutions of assignments given by student.

    25 hour(s)
  • Lecture

    Module 8:Machine Learning

    • In this module will learn how to leverage apache spark in machine learning
    • Topics:
    • 1.Basic Principles of Machine Learning
    • 2.Spark ML API Patters
    • 3.Built-in Featuring and Algorithm APIs
    • 4.There will be 10 case real time machine learning studies with real time dataset.

    25 hour(s)
  • Lecture

    Module 9 : Integration with other tools

    • In this module will learn how to leverage apache spark in machine learning
    • Topics:
    • 1.Integration with Kafka
    • 2.Integration with Cassandra
    • 3.Integration with Hive

    25 hour(s)
  • Lecture

    Module 10 :

    • Real time amazon EMR cluster creation.

    25 hour(s)
  • Lecture

    Module 11 :

    • There will 2 Projects based student interested domains.

    25 hour(s)
  • Lecture

    Module 12 :

    • Resume and interview preparation.

    25 hour(s)