Skip to content

Upgrading from Java7 to Java8 on the Cloudera Quickstart VM

October 31, 2016

Purpose

The purpose of this post is to describe how to set Java8 as the version of Java to use in the Cloudera Quickstart VM and as the version of java to use in Hadoop. The reason you might want to do this is so that you can run Spark jobs using the Java8 libraries and features (like lambda operations, etc).

For more information you can follow the Guide to the Cloudera Quickstart VM post:

https://softwaresanders.wordpress.com/2016/10/24/getting-started-with-the-cloudera-quickstart-vm/

High Level Steps

  1. Shutdown services
  2. Upgrade to Java8
  3. Update configurations to use Java8
  4. Restart services

In depth Steps

  1. SSH to the Cloudera Quickstart VM
  2. Stop Hadoop Services
    • If you’re using Cloudera Manager:
      1. Login to the Cloudera Manager
      2. Stop to Cloudera Management Services
      3. Stop the Cluster
      4. SSH into the machine
      5. Login as root
        sudo su
        
      6. Stop the Cloudera SCM Services from the command line
        service cloudera-scm-agent stop
        service cloudera-scm-server stop
        
    • If you’re not using Cloudera Manager:
      1. SSH into the machine
      2. Login as root
        sudo su
        
      3. Execute the stop service commands
        service hadoop-hdfs-datanode stop
        service hadoop-hdfs-journalnode stop
        service hadoop-hdfs-namenode stop
        service hadoop-hdfs-secondarynamenode stop
        service hadoop-httpfs stop
        service hadoop-mapreduce-historyserver stop
        service hadoop-yarn-nodemanager stop
        service hadoop-yarn-proxyserver stop
        service hadoop-yarn-resourcemanager stop
        service hbase-master stop
        service hbase-regionserver stop
        service hbase-rest stop
        service hbase-solr-indexer stop
        service hbase-thrift stop
        service hive-metastore stop
        service hive-server2 stop
        service impala-catalog stop
        service impala-server stop
        service impala-state-store stop
        service oozie stop
        service solr-server stop
        service spark-history-server stop
        service sqoop2-server stop
        service sqoop-metastore stop
        service zookeeper-server stop
        
  3. Install and Configure JDK 1.8
    1. SSH into the machine
    2. Login as root
    3. Change directory to where you want to place the JDK resources
      1. We’ll assume its under /usr/java/
        cd /usr/java/
        
    4. Download the JDK
      wget --no-check-certificate --no-cookies --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u60-b27/jdk-8u60-linux-x64.tar.gz
      
    5. Unzip the JDK
      gunzip jdk-8u60-linux-x64.tar.gz
      tar -xvf jdk-8u60-linux-x64.tar jdk1.8.0_60/
      
    6. Edit the /etc/bashrc file to set java8 as the primary java
      1. Set the bottom of the file to resemble the following:
        export JAVA_HOME=/usr/java/jdk1.8.0_60
        export JRE_HOME=${JAVA_HOME}
        export JDK_HOME=${JAVA_HOME}
        export ANT_HOME=/usr/local/apache-ant/apache-ant-1.9.2
        export M2_HOME=/usr/local/apache-maven/apache-maven-3.0.4
        export PATH=/usr/local/firefox:/sbin:$JAVA_HOME/bin:$ANT_HOME/bin:$M2_HOME/bin:$PATH
        
    7. Apply the changes to the /etc/bashrc
      source ~/.bashrc
      
    8. Validate the correct version of
      1. Check Java Version
        java -version
        
      2. Expected Output:
        java version "1.8.0_60"
        
  4. Set Configs
    1. Edit the following files
      nano /etc/default/cloudera-scm-server
      nano /etc/default/hadoop
      nano /etc/default/hadoop-0.20-mapreduce
      nano /etc/default/hadoop-hdfs-datanode
      nano /etc/default/hadoop-hdfs-journalnode
      nano /etc/default/hadoop-hdfs-namenode
      nano /etc/default/hadoop-hdfs-secondarynamenode
      nano /etc/default/hadoop-yarn-nodemanager
      nano /etc/default/spark
      nano /etc/default/impala
      nano /etc/default/zookeeper
      nano /etc/default/solr
      
    2. Add the following line to the bottom of each
      export JAVA_HOME=/usr/java/jdk1.8.0_60
      
  5. Restart the Hadoop Services
    • If you’re using Cloudera Manager:
      1. SSH into the machine
      2. Login as root
      3. Start the Cloudera SCM Services from the command line
        service cloudera-scm-agent start
        service cloudera-scm-server start
        
      4. Login to Cloudera Manager
      5. Start the Cluster
      6. Start the Cloudera Management Services
    • If you’re not using Cloudera Manager:
      • A fast way to do this is to just restart the Virtual Machine. The services are setup to startup when the Virtual Machine starts.
  6. Validate that all the services are running with Java8
    1. Print the Hadoop processes
      ps -eaf | grep hadoop
      
    2. You should see all the lines resemble the following:
      • hdfs      5288     1 27 21:53 ?        00:00:04 /usr/java/jdk1.8.0_60/bin/…
Advertisements
One Comment
  1. Dan permalink

    Thank you for the wonderful post.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: