Geek asks and answers

Hadoop cannot install gears> 1

I use Hadoop for a university assignment, and I have code that works, but I run into a little problem.

I am trying to set the number of gears to 19 (which is 0.95 *, as the docs suggest). However, when I look at my work in the task tracker, it says only 1 gear.

System.err.println("here");
job.setNumReduceTasks(19);
System.err.println(job.getNumReduceTasks());

Valid Values:

here
19

But in the final release, I get:

12/05/16 11:10:54 INFO mapred.JobClient:     Data-local map tasks=111
12/05/16 11:10:54 INFO mapred.JobClient:     Rack-local map tasks=58
12/05/16 11:10:54 INFO mapred.JobClient:     Launched map tasks=169
12/05/16 11:10:54 INFO mapred.JobClient:     Launched reduce tasks=1

Parts of the reprint that I rewrote:

  • Mapper
  • Gearbox
  • Markup
  • Grouping comparator.

My first thought was that the separator returned the same value for each key. I check it, and it wasn’t.

I also verified that the grouper is working correctly.

I'm not sure what else could be causing this. If anyone can help, it will be very appreciated.

Java, , , , .

PS: , , - . PS: , , , - , .

: TejasP :

Hadoop ? ( ).

, , jobtracker. 1 . , . 1 . Xml

HADOOP ?

, env, , .

env | grep HADOOP
HADOOP_HOME=/mnt/biginsights/opt/ibm/biginsights/IHC
HADOOP_CONF_DIR=/mnt/biginsights/opt/ibm/biginsights/hadoop-conf

node node? , ? ?

(10) :

Nodes: 10
Map Task Capacity: 20
Reduce Task Capacity: 20
Blacklisted Nodes: 0

setNumReduceTasks? , set, , (19), 1.

, ( ? ). . , . , .

+3
2

, :

  • Hadoop ? (., )
  • HADOOP ?
  • node node?
  • , ? ?
  • setNumReduceTasks? , ( ). . , . , .
+1

, LocalJobRunner ( , ). , > 1, 1. LocalJobRunner.java

int numReduceTasks = job.getNumReduceTasks();
if (numReduceTasks > 1 || numReduceTasks < 0) {
      // we only allow 0 or 1 reducer in local mode
      numReduceTasks = 1;
      job.setNumReduceTasks(1);
}
+2

More articles:


All Articles