SQOOP ( For Structured data)

(What is Hadoop and its ecosytem tools - this is the place for all your answers - Know about Hadoop - A Big Data Handling Framework )

Sqoop : 

Sqoop is a command line interface application for transferring data between relational Database and Hadoop

where Hadoop refers hdfs (hadoop distributed file system) , Hive(Datawarehouse) , Hbase(NoSqlDB) etc

Suppose your Manager came and said hey man onshore team need some tables on hdfs (i.e fetch records from MYSQL table Customers and store into hdfs)  and this need to deliver today itself .

what are the things you need to ask ?

1. Credentials required .
2. Table belongs to which database .

we will add questions as we progress ...

Replied:
Onshore coordinator has shared below image with credentials in separate mail from source system where you can find table belongs to which database i.e InsuranceEuropeDB.

Now lets start our coding ..

New thing SQOOP to explore

Start with : Sqoop help

It will provide all the commands supported by current Sqoop version .

Lets make a connection with mysql and check whether able to connect with requested database .

we can use above two commands i.e . 1) list-database 2) list-tables

sometimes we forgot what are the other things we need to pass with the command . Dont worry about that ask help from SQOOP .

                                           sqoop help <command>

Sample :

Syntax : sqoop list-databases --connect jdbc:mysql://<ip address> --username <username> --password <password>
Command:
sqoop list-databases --connect jdbc:mysql://127.0.0.1 --username root --password password

As we can see that internally its calling SHOW DATABASE only . Waoo I can see database "InsuranceEuropeDB" this is awesome ;)

Also I am highlighted the Consider using -P instead  ( in place of --password <password>) , don't worry guys ,lets try it!!

Command:
sqoop list-databases --connect jdbc:mysql://127.0.0.1 --username root -P   

Press enter then it will ask password to type , it will not be visible . (Me : Great Sqoop you are taking care of this ... Friend who know Sqoop : hmm so use -P only ( observe that P is capital with single hyphen))

This is great I cannot see password .

Let proceed for next command to list tables .

Me : Opps nothing came why ? I use list-tables ,connect ,username , password all still !!

Friend who know Sqoop : Hey Man , mention database also so that it will fetch tables name from that database .

Me : Yes I got it .. Thanks

Me : Sqoop is awesome . After lunch I will do import , till evening I have to provide them hdfs path (having Customers table data )(Next blog ) .


Hope you like my blogs . Please comment and like . Ask doubts if you have related to SQOOP .

Happy Coding
Amit Shyamal Dass