架設Spark
Hadoop Server 分工內容
hadoop1 Master
hadoop3 Worker
hadoop4 Worker
配置Hadoop1、3、4
配置Scala
下載Scala到hadoop User家目錄
$ wget https://downloads.lightbend.com/scala/2.12.6/scala-2.12.6.tgz
$ tar -zxvf scala-2.12.6.tgz
$ mv scala-2.12.6 scala設定必要環境變數
請在.bashrc新增以下環境變數內容
vi ~/.bashrc 變數內容
export SCALA_HOME=/home/hadoop/scala
export PATH=$SCALA_HOME/bin:$PATH
export SPARK_HOME=/home/hadoop/spark
export PATH=$SPARK_HOME/bin:$PATH使環境變數生效
$ source ~/.bashrc檢查Scala版本號
$ scala -version
Scala code runner version 2.12.6 -- Copyright 2002-2018, LAMP/EPFL and Lightbend, Inc.安裝Spark
下載Spark 2.3.1到Hadoop1 Server的hadoop User家目錄
要下載Source Code版本，原因是要打包沒有Hive版本的Spark
$ cd ~
$ wget http://apache.stu.edu.tw/spark/spark-2.3.1/spark-2.3.1-bin-hadoop2.7.tgz
$ tar -zxvf spark-2.3.1.tgz
$ ~/spark/dev/make-distribution.sh --name "hadoop2-without-hive" --tgz "-Pyarn,hadoop-provided,hadoop-2.7,parquet-provided,orc-provided"
$ cp ~/spark/spark-2.3.1-bin-hadoop2-without-hive.tgz ~/
$ tar -zxvf spark-2.3.1-bin-hadoop2-without-hive.tgz
$ mv spark-2.3.1-bin-hadoop2-without-hive spark配置spark-env.sh
$ cp ~/spark/conf/spark-env.sh.template ~/spark/conf/spark-env.sh
$ vi ~/spark/conf/spark-env.sh配置內容
export SCALA_HOME=/home/hadoop/scala
export JAVA_HOME=/usr
export HADOOP_HOME=/home/hadoop/hadoop
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export SPARK_HOME=/home/hadoop/spark
export SPARK_MASTER_IP=hadoop1
export SPARK_EXECUTOR_MEMORY=1G
export SPARK_DIST_CLASSPATH=$(hadoop classpath)配置slaves
$ vi ~/spark/conf/slaves配置內容
hadoop3
hadoop4將Spark從hadoop1 Server複製到其他Server的hadoop User家目錄
$ scp -r ~/spark hadoop3:/home/hadoop/
$ scp -r ~/spark hadoop4:/home/hadoop/啟動Spark
在hadoop1 Server執行指令
$ ~/spark/sbin/start-all.sh 測試
瀏覽hadoop1 Spark網站
看網站是否有順利開啟，以及是否有2個worker
http://hadoop1:8080/ 在hadoop1 Server測試單機執行圓周率任務
$ ~/spark/bin/spark-submit --class org.apache.spark.examples.SparkPi --master local ~/spark/examples/jars/spark-examples_2.11-2.3.1.jar 在hadoop1 Server測試獨立Spark集群模式執行圓周率任務
$ ~/spark/bin/spark-submit --class org.apache.spark.examples.SparkPi --master spark://hadoop1:7077 ~/spark/examples/jars/spark-examples_2.11-2.3.1.jar 在hadoop1 Server測試yarn-cluster集群模式執行圓周率任務
$ ~/spark/bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-client ~/spark/examples/jars/spark-examples_2.11-2.3.1.jarLast updated