Launching Spark Shell

Let us see different modes in which we can launch spark shell

Execution Modes

• Local
• Stand Alone
• Mesos
• YARN
• In production typically we use YARN or Mesos
• Mesos is used in exclusive Spark clusters
• YARN is used in distributions such as Cloudera, Hortonworks, MapR etc

On your PC

• You need to have spark set up – click here (if you need to set up environment)
• sbin contains scripts to start and stop services
• bin contains commands such as spark-shell
• We can launch spark in scala context using spark-shell command
• By default it will be launched in local mode
• spark-shell can also be launched in
  • Stand-alone mode
    • $SPARK_HOME/sbin/start-master.sh
    • $SPARK_HOME/sbin/start-slave.sh
    • Make sure $SPARK_HOME/sbin is appended to PATH
    • Start master start-master.sh
    • Start slave start-slave.sh spark://Apples-MacBook-Pro.local:7077
    • Go to localhost:8080
    • Get the URL for spark master similar to this 
    • Launch spark-shell, like this spark-shell --master spark://Apples-MacBook-Pro.local:7077
  • Mesos mode – out of scope
  • YARN mode – we will see that on labs.itversity.com

On labs.itversity.com

• Enroll for the lab
• Login to gw01.itversity.com
• We can launch in local mode (default) or yarn mode
• spark-shell will launch it in local mode
• For YARN mode, you can check instructions over here

Using labs.itversity.com

• For certifications such as HDPCD Spark and CCA 175, it is better to use YARN mode