Styjun

So I noticed an odd behavior of my map-reduce code today. Spent 3 hours trying to figure it out, still got nothing.

I am trying to find the answers for 3 simple questions based on this dataset:

The Academy Awards, 1927-2015

• How many awards were given in a particular year?


• Which actor/actress
  has received the most awards overall?


• Which film has received the
  most awards in a ceremony?

I wrote my map reduce code and noticed that the mapper is running for different number of lines for each code.

For Q1 - 3251 lines, Q2 - 3251 lines, and Q3 - 33 lines!

I don't understand why this is happening.

Driver

public static void main(String[] args) throws Exception 

 String inputFilePath = "./database.csv";
 String outputFilePath = "./<BASED_ON_QUESTION>";

 Configuration conf = new Configuration();
 Job job = Job.getInstance(conf, "<BASED_ON_QUESTION>");
 job.setJarByClass(YearlyAwards.class);
 job.setMapperClass(TokenizerMapper.class);
 job.setCombinerClass(IntSumReducer.class);
 job.setInputFormatClass(TextInputFormat.class);
 job.setOutputFormatClass(TextOutputFormat.class);
 job.setOutputKeyClass(Text.class);
 job.setOutputValueClass(IntWritable.class);

 FileInputFormat.addInputPath(job, new Path(inputFilePath));

 try 
 File f = new File(outputFilePath);
 FileUtils.forceDelete(f);
 catch (Exception e) 

 

 FileOutputFormat.setOutputPath(job, new Path(outputFilePath));
 System.exit(job.waitForCompletion(true) ? 0 : 1);

Mapper

Q1

public static class TokenizerMapper extends Mapper<Object, Text, Text, IntWritable> 
 Integer count = 0;

 public void map(Object key, Text value, Context context) throws IOException, InterruptedException 

 String[] quoteLessVal = value.toString().split(""");
 value = new Text(String.join("", quoteLessVal));
 String[] values = value.toString().split(",");
 String year = values[0];
 String win = values[3];
 count += 1;

 if (!win.equals("")) 
 context.write(new Text(year), new IntWritable(new Integer(win)));
 
 

 @Override
 protected void cleanup(Mapper<Object, Text, Text, IntWritable>.Context context)
 throws IOException, InterruptedException 
 super.cleanup(context);
 System.out.println(count);
 

Q2

public static class TokenizerMapper extends Mapper<Object, Text, Text, IntWritable> 
 Integer count = 0;

 public void map(Object key, Text value, Context context) throws IOException, InterruptedException 

 count += 1;

 String[] quoteLessVal = value.toString().split(""");
 value = new Text(String.join("", quoteLessVal));
 String[] values = value.toString().split(",");
 String name = values[4];
 String win = values[3];

 if (!win.equals("")) 
 context.write(new Text(name), new IntWritable(new Integer(win)));
 
 

 @Override
 protected void cleanup(Mapper<Object, Text, Text, IntWritable>.Context context)
 throws IOException, InterruptedException 
 super.cleanup(context);
 System.out.println(count);
 

Q3

public static class TokenizerMapper extends Mapper<Object, Text, Text, IntWritable> 
 Integer count = 0;

 public void map(Object key, Text value, Context context) throws IOException, InterruptedException 

 count += 1;

 String[] quoteLessVal = value.toString().split(""");
 value = new Text(String.join("", quoteLessVal));
 String[] values = value.toString().split(",");
 String name = values[5];
 String win = values[3];
 count += 1;

 if (!win.equals("")) 
 context.write(new Text(name), new IntWritable(new Integer(win)));
 
 

 @Override
 protected void cleanup(Mapper<Object, Text, Text, IntWritable>.Context context)
 throws IOException, InterruptedException 
 super.cleanup(context);
 System.out.println(count);
 

Reducer

(Fairly standard for all Qs)

public static class IntSumReducer extends Reducer<Text, IntWritable, Text, IntWritable> 

 public void reduce(Text key, Iterable<IntWritable> values, Context context)
 throws IOException, InterruptedException 
 Integer count = 0;

 for (IntWritable val : values) 
 count += 1;
 
 System.out.println(key + " > " + count);

 context.write(key, new IntWritable(count));
 

I think the code's execution is halting for some reason, as I am not getting any output file (part-r-00000) in the output folder!