Saturday 3 January 2015

How to pass small number of configuration parameters to a mapper and reducer ?

Hadoop is having several configurable properties that will be present in several xml and properties files. The main configuration files in hadoop are core-site.xml, mapred-site.xml, hdfs-site.xml, yarn-site.xml. The parameters in these configuration files are set while installing the cluster. This will be done by the administrator.

If a developer while developing the mapreduce programs want to modify some of the configuration parameter, he can do it from the program itself. The way to modify these values from the program is by instantiating the configuration class and setting the configuration values by passing the parameter and value as key-value pairs to the program.

The syntax is as shown below
Configuration conf = new Configuration();
conf.set("key1","value1");
conf.set("key2","value2");

No comments:

Post a Comment

How to check the memory utilization of cluster nodes in a Kubernetes Cluster ?

 The memory and CPU utilization of a Kubernetes cluster can be checked by using the following command. kubectl top nodes The above command...