课程描述：

4天的课程包涵了解Apache Spark的基础知识及其与Hadoop整体生态系统的集成方式。
本课程将重温HDFS的基础内容，学习如何使用Sqoop/Flume摄取数据，利用Spark处理分布式数据，学习在Impala和Hive上数据建模，以及在数据存储方面的最佳实践。 

培训对象：

企业管理者、CIO、CTO、政府信息部门官员、项目（开发）经理、咨询顾问；IT经理，IT咨询顾问，IT支持专家；系统工程师、数据中心管理员、云计算管理员及想加入云计算队伍的您

学员基础：

具备基本Linux系统管理经验；不需要事先掌握Hadoop相关知识
 

认证证书：

通过考试可获得Cloudera Certified Administrator for Apache Hadoop (CCAH) 

课程目标： 
 

How data is distributed, stored, and processed in a Hadoop cluster  
How to use Sqoop and Flume to ingest data  
How to process distributed data with Apache Spark  
How to model structured data as tables in Impala and Hive  
How to choose the best data storage format for different data usage patterns  
Best practices for data storage

课程内容： 

Introduction to Hadoop and the Hadoop Ecosystem  
Problems with Traditional Large-scale Systems Hadoop!  
The Hadoop EcoSystem  
 

Hadoop Architecture and HDFS  
Distributed Processing on a Cluster  
Storage: HDFS Architecture  
Storage: Using HDFS  
Resource Management: YARN Architecture  
Resource Management: Working with YARN  
 

Importing Relational Data with Apache Sqoop  
Sqoop Overview  
Basic Imports and Exports  
Limiting Results  
Improving Sqoop’s Performance  
Sqoop 2  
 

Introduction to Impala and Hive  
Introduction to Impala and Hive  
Why Use Impala and Hive?  
Comparing Hive to Traditional Databases  
Hive Use Cases  
 

Modeling and Managing Data with Impala and Hive  
Data Storage Overview  
Creating Databases and Tables  
Loading Data into Tables  
HCatalog  
Impala Metadata Caching  
 

Data Formats  
Selecting a File Format
Hadoop Tool Support for File Formats  
Avro Schemas  
Using Avro with Hive and Sqoop  
Avro Schema Evolution  
Compression  
 

Data Partitioning  
Partitioning Overview  
Partitioning in Impala and Hive  
 

Capturing Data with Apache Flume  
What is Apache Flume?  
Basic Flume Architecture  
Flume Sources  
Flume Sinks  
Flume Channels  
Flume Configuration  
 

Spark Basics  
What is Apache Spark?  
Using the Spark Shell  
RDDs (Resilient Distributed Datasets)  
Functional Programming in Spark  
 

Working with RDDs in Spark  
A Closer Look at RDDs  
Key-Value Pair RDDs  
MapReduce  
Other Pair RDD Operations  
 

Writing and Deploying Spark Applications  
Spark Applications vs. Spark Shell  
Creating the SparkContext  
Building a Spark Application (Scala and Java)  
Running a Spark Application  
The Spark Application Web UI  
Configuring Spark Properties  
Logging  
 

Parallel Programming with Spark  
Review: Spark on a Cluster  
RDD Partitions  
Partitioning of File-based RDDs  
HDFS and Data Locality  
Executing Parallel Operations  
Stages and Tasks  
 

Spark Caching and Persistence  
RDD Lineage  
Caching Overview  
Distributed Persistence  
 

Common Patterns in Spark Data Processing  
Common Spark Use Cases  
Iterative Algorithms in Spark  
Graph Processing and Analysis  
Machine Learning  
Example: k-means  
 

Preview: Spark SQL  
Spark SQL and the SQL Context  
Creating DataFrames  
Transforming and Querying DataFrames  
Saving DataFrames  
Comparing Spark SQL with Impala