Apache Spark and Scala Training and Certification

• Weekdays – 5 to 6 weeks
• Weekend – 7 to 8 weeks
• FastTrack – 3 to 4 weeks

• Weekdays – 5 to 6 weeks
• Weekend – 7 to 8 weeks
• FastTrack – 3 to 4 weeks

• Introducing Scala and deployment of Scala for Big Data applications and Apache Spark analytics
• Scala REPL and Lazy Values
• Control Structures in Scala
• Directed Acyclic Graph (DAG)
• First Spark Application Using SBT/Eclipse
• Spark Web UI and Spark in Hadoop Ecosystem.

• The importance of Scala
• The concept of REPL (Read Evaluate Print Loop)
• Deep dive into Scala pattern matching
• Type interface, higher-order function, currying, traits
• Application space and Scala for data analysis

• Learning about the Scala Interpreter
• Static object timer in Scala and testing string equality in Scala
• Implicit classes in Scala
• The concept of currying in Scala and various classes in Scala

• Functional Programming
• Higher Order Functions
• Anonymous Functions
• Learning about the Classes concept
• Getters and Setters
• Custom Getters and Setters
• Properties with only Getters
• Understanding the constructor overloading
• Various abstract classes and the hierarchy types in Scala
• The concept of object equality and the val and var methods in Scala

• Understanding sealed traits, wild, constructor, tuple, variable pattern and constant pattern

• Understanding traits in Scala
• The advantages of traits
• Linearization of traits, the Java equivalent, and avoiding of boilerplate code

• Implementation of traits in Scala and Java and handling of multiple traits extending

• Introduction to Scala collections
• Classification of collections
• The difference between Iterator and Iterable in Scala and example of list sequence in Scala

• The two types of collections in Scala
• Mutable and Immutable collections
• Understanding lists and arrays in Scala
• The list buffer and array buffer
• Queue in Scala and double-ended queue Deque, Stacks
• Sets, Maps and Tuples in Scala

• Introduction to Scala packages and imports
• The selective imports
• The Scala test classes
• Introduction to JUnit test class
• JUnit interface via JUnit 3 suite for Scala test
• Packaging of Scala applications in Directory Structure and examples of Spark Split and Spark Scala

• What is Big Data?
• Big Data Customer Scenarios
• Limitations and Solutions of Existing Data Analytics
• Architecture with Uber Use Case
• How Hadoop Solves the Big Data Problem?
• What is Hadoop?
• Hadoop’s Key Characteristics
• Hadoop Ecosystem and HDFS
• Hadoop Core Components
• Rack Awareness and Block Replication
• YARN and its Advantage
• Hadoop Cluster and its Architecture
• Hadoop: Different Cluster Modes
• Big Data Analytics with Batch & Real-time Processing
• Why Spark is needed?
• What is Spark?
• How Spark differs from other frameworks?

• Introduction to Spark and how Spark overcomes the drawbacks of working MapReduce
• Understanding in-memory MapReduce, interactive operations on MapReduce
• Spark stack, fine vs. coarse-grained update
• Spark stack and Spark Hadoop YARN
• HDFS Revision
• YARN Revision
• The overview of Spark and how it is better Hadoop
• Deploying Spark without Hadoop
• Spark history server and Cloudera distribution

• Spark installation guide and Spark configuration
• Memory management
• Executor memory vs. driver memory and working with Spark Shell
• The concept of resilient distributed datasets (RDD)
• Learning to do functional programming in Spark and the architecture of Spark

• Spark RDD and creating RDDs
• RDD partitioning, operations, and transformation in RDD
• Deep dive into Spark RDDs
• The RDD general operations, a read-only partitioned collection of records, using the concept of RDD for faster and efficient data processing
• RDD action for collect, count, collects map, save-as-text-files and pair RDD functions Aggregating Data with Pair RDDs
• Understanding the concept of Key-Value pair in RDDs
• Learning how Spark makes MapReduce operations faster
• Various operations of RDD
• MapReduce interactive operations
• Fine and coarse-grained update and Spark stack

• Comparing the Spark applications with Spark Shell
• Creating a Spark application using Scala or Java
• Deploying a Spark application
• Scala built application
• Creation of mutable list
• Set and set operations, list, tuple, concatenating list
• Creating application using SBT
• Deploying application using Maven
• The web user interface of Spark application, a real-world example of Spark and configuring of Spark

• Learning about Spark parallel processing
• Deploying on a cluster
• Introduction to Spark partitions
• File-based partitioning of RDDs
• Understanding of HDFS and data locality
• Mastering the technique of parallel operations
• Comparing repartition and coalesce and RDD actions

• The execution flow in Spark, understanding the RDD persistence overview
• Spark execution flow, and Spark terminology
• Distribution shared memory vs. RDD and RDD limitations
• Spark shell arguments and Distributed persistence
• RDD lineage
• Key-Value pair for sorting implicit conversions like CountByKey, ReduceByKey, SortByKey and AggregateByKey

• Introduction to Machine Learning
• Types of Machine Learning, introduction to MLlib
• Various ML algorithms supported by MLlib
• Linear Regression and Logistic Regression
• Decision Tree and Random Forest
• K-means clustering techniques, building a Recommendation Engine

• Why Kafka and What is Kafka
• Kafka architecture
• Kafka workflow
• Configuring Kafka cluster
• Basic operations
• Kafka monitoring tools
• Integrating Apache Flume and Apache Kafka

• Introduction to Spark Streaming
• Features of Spark Streaming
• Spark Streaming workflow
• Initializing StreamingContext
• Discretized Streams (DStreams)
• Input DStreams and Receivers, transformations on Dstreams
• Output Operations on Dstreams
• Windowed Operators and why it is useful
• Important Windowed Operators and Stateful Operators.

• Introduction to various variables in Spark like shared variables and broadcast variables
• Learning about accumulators
• The common performance issues and troubleshooting the performance problems

• Learning about Spark SQL
• The context of SQL in Spark for providing structured data processing
• JSON support in Spark SQL
• Working with XML data
• Parquet files
• Creating Hive context and writing Data Frame to Hive
• Reading JDBC files and understanding the Data Frames in Spark
• Creating Data Frames
• Manual inferring of schema
• Working with CSV files and reading JDBC tables, Data Frame to JDBC
• User-defined functions in Spark SQL
• Shared variables and accumulators
• Learning to query and transform data in Data Frames
• How Data Frame provides the benefit of both Spark RDD and Spark SQL and deploying
Hive on Spark as the execution engine

• Learning about the scheduling and partitioning in Spark
• Hash partition and range partition
• Scheduling within and around applications
• Static partitioning and dynamic sharing
• Fair scheduling and Map partition with index, the Zip, GroupByKey
• Spark master high availability
• Standby masters with ZooKeeper
• Single-node Recovery with Local File System and High Order Functions

Industry: - Digital news portal

Problem Statement: -Provide personalized news pages to web visitors over bing or yahoo

Platform like yahoo or bing usually work to provide very personalized experience to the user based on their likes or dislikes. For example yahoo uses Machine Learning based algorithm running on Spark to figure out what individual users are intersted in reading, along with categorizing news articles which helps in figuring out about what sort of users could be interested in reading. To do same 120 Lines of Spark ML algorithm required written in scala. We will let our learner to develop such algos to make them business ready.

Industry: - Entertainment

Problem Statement: - Develop a movie recommendation system for user

Here we will deploy Apache spark for recommending movie to user.This project will let you gain more hands on experince over Machine Learning Library, MLlib by deploying collaborative filtering, clustering, regression, and dimensionality reduction. At the end you will have experience with streaming data, sampling, testing and statistics.

Industry: - Data analysis

Problem Statement: - Develop an interactive real time analytics tool for user

Apache spark also supports interactive analysis in addition to bundle of features. It process exploratory queries without sampling which results into faster processing. You can read APi very easily with interactive data analysis.It is available in Scala. MapReduce is made to handle batch processing and SQl on Hadoop engines which are usually slow.As a result if you are having live data for identification queries it can perform very fast. With structured streaming web analytics can be performed by allowing client to run user friendly query with visitors.

Industry: - Miscllaneous

Problem Statement: -Use wikipedia data set for data exploratory

This project will give hands on experince on Spark Sql.It will implementind and practised by combining it with ETL applications, real-time analysis of data, performing batch analysis, deploying machine learning, creating visualizations and processing of graphs.You can use twitter or Wikipedia data set.