BigData

Run Hadoop In Galles

Courtesy from Daniel Waybright

(1) Copy up text files to your directory with scp from local machine. Looks like:

 $ scp <localDir> user@galles.alliance.unm.edu:<remoteDir>

(1.5) SSH into Galles

 $ ssh you@galles.alliance.unm.edu

(2) SSH into 163:

 $ ssh galles163

(3) Attempt to load files into your HDFS directory. Should be $$/users/yourUserName$$

  • Run this to see the list of filesystem commands available
 $ /usr/local/hadoop/bin/hadoop dfs
  • View what is in the HDFS filesystem using $$-ls$$
 $ /usr/local/hadoop/bin/hadoop dfs -ls /
  • Copy a file you made into your HDFS folder ($$-put$$ should work just as well)
 /usr/local/hadoop/bin/hadoop dfs -copyFromLocal <local_src> <hdfs_dst>
  • If it works, great! If not, and see something like this:
 copyFromLocal: org.apache.hadoop.security.AccessControlException: 
 Permission denied: user=userName, access=WRITE,   
 inode="userName":root:supergroup:rwxr-xr-x 

Send to CARC help desk immediately! You will be stuck here.

(4) Now that we have a text file in our HDFS folder, let's try a test run with the built-in example of wordcount:

 $ /usr/local/hadoop/bin/hadoop jar /usr/local/hadoop/hadoop-examples-1.2.1.jar wordcount <hdfs_text_path> <give_hdfs_output_path>

Notes: make sure the output path DOES NOT EXIST before executing.

If it works, great! If not, and see something like this:

 Exception in thread "main" java.io.IOException: Permission denied
    at java.io.UnixFileSystem.createFileExclusively(Native Method)
    at java.io.File.checkAndCreate(File.java:1704)
    at java.io.File.createTempFile(File.java:1792)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:119)

Send to CARC help desk immediately! You will be stuck here.

(5) You now have successfully ran a hadoop program!