Let us see different modes in which we can launch spark shell
Execution Modes
- Local
- Stand Alone
- Mesos
- YARN
- In production typically we use YARN or Mesos
- Mesos is used in exclusive Spark clusters
- YARN is used in distributions such as Cloudera, Hortonworks, MapR etc
On your PC
- You need to have spark set up – click here (if you need to set up environment)
- sbin contains scripts to start and stop services
- bin contains commands such as spark-shell
- We can launch spark in scala context using spark-shell command
- By default it will be launched in local mode
- spark-shell can also be launched in
- Stand-alone mode
- $SPARK_HOME/sbin/start-master.sh
- $SPARK_HOME/sbin/start-slave.sh
- Make sure $SPARK_HOME/sbin is appended to PATH
- Start master
start-master.sh
- Start slave
start-slave.sh spark://Apples-MacBook-Pro.local:7077
- Go to localhost:8080
- Get the URL for spark master similar to this
- Launch spark-shell, like this
spark-shell --master spark://Apples-MacBook-Pro.local:7077
- Mesos mode – out of scope
- YARN mode – we will see that on labs.itversity.com
- Stand-alone mode
On labs.itversity.com
- Enroll for the lab
- Login to gw01.itversity.com
- We can launch in local mode (default) or yarn mode
- spark-shell will launch it in local mode
- For YARN mode, you can check instructions over here
- For certifications such as HDPCD Spark and CCA 175, it is better to use YARN mode
Start the discussion at discuss.itversity.com