Hadoop :: File not found exception when running using jar
I am facing a very strange issue here while running Map-Reduce job on a
2-node cluster. I am able to run basic map-reduce example directly on HDFS
and HBase, however, when I try to run the below driver code...I get very
weird exceptions like this:
13/09/09 23:30:31 INFO mapred.JobClient: map 0% reduce 0%
13/09/09 23:30:39 INFO mapred.JobClient: Task Id :
attempt_201309092214_0012_m_000002_0, Status : FAILED
Error initializing attempt_201309092214_0012_m_000002_0:
java.io.FileNotFoundException: File
/home/xx.xx/xx/Hadoop/hadoop-1.2.1/bin/partitions_a9be1170-cab1-4f5a-b5a3-a15c11a9b173
does not exist.
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:402)
at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:255)
at
org.apache.hadoop.filecache.TaskDistributedCacheManager.setupCache(TaskDistributedCacheManager.java:179)
at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1320)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1311)
at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1226)
at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2603)
at java.lang.Thread.run(Thread.java:662)
13/09/09 23:30:39 WARN mapred.JobClient: Error reading task
outputhttp://slaveNode:50060/tasklog?plaintext=true&attemptid=attempt_201309092214_0012_m_000002_0&filter=stdout
13/09/09 23:30:39 WARN mapred.JobClient: Error reading task
outputhttp://slaveNode:50060/tasklog?plaintext=true&attemptid=attempt_201309092214_0012_m_000002_0&filter=stderr
13/09/09 23:30:39 INFO mapred.JobClient: Task Id :
attempt_201309092214_0012_m_000003_0, Status : FAILED
Error initializing attempt_201309092214_0012_m_000003_0:
java.io.FileNotFoundException: File
/home/xx.xx/xxx/Hadoop/hadoop-1.2.1/bin/partitions_a9be1170-cab1-4f5a-b5a3-a15c11a9b173
does not exist.
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:402)
at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:255)
Below is my driver code:
public static void main(String[] args){
String inputFile = args[0];
String outFile = args[1];
String tableName = args[2];
Path inputPath = new Path(inputFile);
Path outPath = new Path(outFile);
BufferedWriter br = null;
FileWriter wr = null;
try {
wr = new
FileWriter("/home/xx.xx/Documents/Map-Reduce/mapred_hdfs_distributed.log",
true);
br = new BufferedWriter(wr);
} catch (IOException e) {
System.out.println("Error writing to file :: " + e.getMessage());
}
System.out.println("[TS - START] :: " + new Date());
br.write("[TS - START] :: " + new Date() + "\n");
String createTable = "create table " + tableName + " (row_key bigint not
null, m.row_value varchar(50) CONSTRAINT pk PRIMARY KEY (row_key))";
String upsertTable = "upsert into " + tableName + " (row_key,
m.row_value) values(?,?)";
Create a Phoenix table in HBase
createPTable(createTable);
System.out.println("[TS - Table created] :: " + new Date());
br.write("[TS - Table created] :: " + new Date() + "\n");
Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(conf);
//Set the configurations
conf.set("CREATE-TABLE-ddl", createTable);
conf.set("UPSERT-TABLE-ddl", upsertTable);
conf.set("mapred.job.tracker", "10.0.45.184:54311");
Job job = new Job(conf, "MapReduce - Phoenix bulk insert");
job.setJarByClass(MapReduce.class);
job.setInputFormatClass(TextInputFormat.class);
FileInputFormat.addInputPath(job, inputPath);
fs.delete(outPath);
FileOutputFormat.setOutputPath(job, outPath);
job.setMapperClass(MapReduce.MyMap.class);
job.setMapOutputKeyClass(ImmutableBytesWritable.class);
job.setMapOutputValueClass(KeyValue.class);
SchemaMetrics.configureGlobally(conf);
HTable hTable = new HTable(conf, tableName.toUpperCase());
// Auto configure partitioner and reducer
HFileOutputFormat.configureIncrementalLoad(job, hTable);
job.waitForCompletion(true);
System.out.println("[TS - M-R complete] :: " + new Date());
br.write("[TS - M-R complete] :: " + new Date() + "\n");
// Load generated HFiles into table
LoadIncrementalHFiles loader = new LoadIncrementalHFiles(conf);
loader.doBulkLoad(new Path(outFile), hTable);
System.out.println("[TS - END] :: " + new Date());
br.write("[TS - END] :: " + new Date() + "\n");
br.close();
}
The same code works perfectly fine on a single node pseudo distributed
cluster.
Thanks in advance !
No comments:
Post a Comment