Sqoop Installation

[vc_row][vc_column][vc_text_separator title=”Chapter 3: Sqoop Installation” color=”green” style=”double” border_width=”2″ add_icon=”true”][/vc_column][/vc_row][vc_row][vc_column][vc_column_text]

Depending on the framework you are using you will have to configure Sqoop accordingly. Let me take you through all separately.

[/vc_column_text][/vc_column][/vc_row][vc_row][vc_column][vc_cta h2=”” h4=”Here I am considering three ways those people normally use-” style=”3d” el_width=”xl”]
  • Own configured Hadoop – Custom configuration
  • Cloudera CDH
  • HortonWorks
[/vc_cta][/vc_column][/vc_row][vc_row][vc_column][vc_column_text]

Now I will explain all in detail here-

[/vc_column_text][/vc_column][/vc_row][vc_row][vc_column][vc_custom_heading text=”1. Custom Sqoop Installation” font_container=”tag:h2|text_align:left|color:%2325bc2d” use_theme_fonts=”yes”][/vc_column][/vc_row][vc_row][vc_column][vc_column_text]

Here I am considering that you have already installed Hadoop and Java on your system and now you just want to install Sqoop to proceed further.

Follow the below steps for Sqoop installation when you have configured your own clusters-

[/vc_column_text][/vc_column][/vc_row][vc_row][vc_column][vc_custom_heading text=”Step 1: Download Sqoop” font_container=”tag:h3|text_align:left|color:%231e73be” use_theme_fonts=”yes”][/vc_column][/vc_row][vc_row][vc_column][vc_column_text]In this tutorial, we are using the latest version of fully functional Sqoop 1.4.6. You can download this Sqoop version from here.

You will find multiple files here and you will have to download the sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz file.[/vc_column_text][/vc_column][/vc_row][vc_row][vc_column][vc_custom_heading text=”Step 2: Start with the Sqoop installation” font_container=”tag:h3|text_align:left|color:%231e73be” use_theme_fonts=”yes”][/vc_column][/vc_row][vc_row][vc_column][vc_column_text]Now you will have to first untar the above file. Use the below command on your command line interface (eg. Putty) to untar the file.[/vc_column_text][/vc_column][/vc_row][vc_row][vc_column][vc_message message_box_color=”turquoise” icon_fontawesome=”fa fa-code”]$tar -xvf sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz

$ su

password:[/vc_message][/vc_column][/vc_row][vc_row][vc_column][vc_column_text]Now you can move this file to the standard folder “/usr/lib/sqoop” directory using the below command-[/vc_column_text][/vc_column][/vc_row][vc_row][vc_column][vc_message message_box_color=”turquoise” icon_fontawesome=”fa fa-code”]# mv sqoop-1.4.6.bin__hadoop-2.0.4-alpha /usr/lib/sqoop[/vc_message][/vc_column][/vc_row][vc_row][vc_column][vc_custom_heading text=”Step 3: Configure bashrc file” font_container=”tag:h3|text_align:left|color:%231e73be” use_theme_fonts=”yes”][/vc_column][/vc_row][vc_row][vc_column][vc_column_text]Now you will have to set the SQOOP_HOME path to the path where you put the Sqoop tar file. In our previous step, we have put it to “/usr/lib/sqoop” and so set the Sqoop_Home location to this location.

Use the below command to make this happen-[/vc_column_text][/vc_column][/vc_row][vc_row][vc_column][vc_message message_box_color=”turquoise” icon_fontawesome=”fa fa-code”]#Sqoop export SQOOP_HOME=/usr/lib/sqoop export PATH=$PATH:$SQOOP_HOME/bin[/vc_message][/vc_column][/vc_row][vc_row][vc_column][vc_column_text]As we know whenever you will make changes to the bashrc file, you will have to execute it to reflect the changes. Use the below command to do that-[/vc_column_text][/vc_column][/vc_row][vc_row][vc_column][vc_message message_box_color=”turquoise” icon_fontawesome=”fa fa-code”]$ source ~/.bashrc[/vc_message][/vc_column][/vc_row][vc_row][vc_column][vc_custom_heading text=”Step 4: Configure Sqoop now” font_container=”tag:h3|text_align:left|color:%231e73be” use_theme_fonts=”yes”][/vc_column][/vc_row][vc_row][vc_column][vc_column_text]So far we have untar the tar file and have also set the Sqoop_Home location. Now let’s perform the Sqoop basic file configuration.

To do this, we will have to make changes to sqoop-env.sh which you will find under Sqoop_Home/conf folder.

First of all, redirect to conf folder and copy the template file using the below two commands-[/vc_column_text][/vc_column][/vc_row][vc_row][vc_column][vc_message message_box_color=”turquoise” icon_fontawesome=”fa fa-code”]$ cd $SQOOP_HOME/conf

$ mv sqoop-env-template.sh sqoop-env.sh[/vc_message][/vc_column][/vc_row][vc_row][vc_column][vc_column_text]Now open sqoop-env.sh file and make the changes as per the below 2 lines-[/vc_column_text][/vc_column][/vc_row][vc_row][vc_column][vc_message message_box_color=”turquoise” icon_fontawesome=”fa fa-code”]export HADOOP_COMMON_HOME=/usr/local/hadoop

export HADOOP_MAPRED_HOME=/usr/local/hadoop[/vc_message][/vc_column][/vc_row][vc_row][vc_column][vc_custom_heading text=”Step5: Configure MySQL” font_container=”tag:h3|text_align:left|color:%231e73be” use_theme_fonts=”yes”][/vc_column][/vc_row][vc_row][vc_column][vc_column_text]As we need an RDBMS like MySQL or Oracle SQL for the data transfer between RDBMS and Hadoop and so let’s install MySQL as well.

First of all download mysql-connector-java-5.1.30.tar.gz file from this link.

Now untar the file and move to the /usr/lib/sqoop/lib directory.[/vc_column_text][/vc_column][/vc_row][vc_row][vc_column][vc_message message_box_color=”turquoise” icon_fontawesome=”fa fa-code”]$ tar -zxf mysql-connector-java-5.1.30.tar.gz

$ su

password:

# cd mysql-connector-java-5.1.30

# mv mysql-connector-java-5.1.30-bin.jar /usr/lib/sqoop/lib[/vc_message][/vc_column][/vc_row][vc_row][vc_column][vc_column_text]You are all done now, just verify Sqoop and you are good to go.[/vc_column_text][/vc_column][/vc_row][vc_row][vc_column][vc_custom_heading text=”Step 6: Verify Sqoop” font_container=”tag:h3|text_align:left|color:%231e73be” use_theme_fonts=”yes”][/vc_column][/vc_row][vc_row][vc_column][vc_message message_box_color=”turquoise” icon_fontawesome=”fa fa-code”]$ cd $SQOOP_HOME/bin

$ sqoop-version[/vc_message][/vc_column][/vc_row][vc_row][vc_column][vc_message color=”success” message_box_color=”success” icon_fontawesome=”fa fa-check”]That’s all!!

Now let me tell you about the Sqoop installation in Cloudera and HortonWorks.[/vc_message][/vc_column][/vc_row][vc_row][vc_column][vc_custom_heading text=”2. Install Sqoop in HortonWorks” font_container=”tag:h2|text_align:left|color:%23189b15″ use_theme_fonts=”yes”][/vc_column][/vc_row][vc_row][vc_column][vc_column_text]Usually, if you are using HortonWorks image then you will find Sqoop and MySQL already installed and directly start using it from the command line.

You should check this link for the details on Install Sqoop in HortonWorks.[/vc_column_text][/vc_column][/vc_row][vc_row][vc_column][vc_custom_heading text=”3. Sqoop installation in Cloudera” font_container=”tag:h2|text_align:left|color:%23189b15″ use_theme_fonts=”yes”][/vc_column][/vc_row][vc_row][vc_column][vc_column_text]As you have already used Cloudera CDH and so you need to again install Sqoop. You can check the details using the below commands on command line-[/vc_column_text][/vc_column][/vc_row][vc_row][vc_column][vc_message message_box_color=”turquoise” icon_fontawesome=”fa fa-code”]$ sqoop help$ sqoop version

$ sqoop import[/vc_message][/vc_column][/vc_row][vc_row][vc_column][vc_column_text]You can check the further details on Cloudera official site using this link.

[/vc_column_text][/vc_column][/vc_row] Previous: Sqoop ArchitectureChapter 4: Sqoop Import