Profiling a MapReduce program using HProf

You can profile a MapReduce program using Hprof. Some settings are required for this in your driver program as shown below

1 2 3 4

conf.setBoolean("mapred.task.profile", true); conf.set("mapred.task.profile.params", "-agentlib:hprof=cpu=samples," + "heap=sites,depth=6,force=n,thread=y,verbose=n,file=%s"); conf.set("mapred.task.profile.maps", "0-2");

Mount HDFS using NFS on a Windows Client

Configure settings for HDFS NFS gateway:

NFS gateway uses the same configurations as used by the NameNode and DataNode. Configure the following properties based on your application's requirements:

1. Edit the hdfs-default.xml file on your NFS gateway machine and modify the following property:

   <property>
     <name>dfs.namenode.accesstime.precision</name>
     <value>3600000</value>
     <description>The access time for HDFS file is precise up to this value. 
                  The default value is 1 hour. Setting a value of 0 disables
                  access times for HDFS.
     </description>
   </property>

2. 
   Add the following property to hdfs-site.xml:
   <property>    
       <name>dfs.nfs3.dump.dir</name>    
       <value>/tmp/.hdfs-nfs</value> 
   </property>
   

3. 
   Start the NFS gateway service.
   
   Three daemons are required to provide NFS service: rpcbind (or portmap), mountd and nfsd. 
   The NFS gateway process has both nfsd and mountd. It shares the HDFS root "/" as the only export. We recommend using the portmap included in NFS gateway package as shown below:
   
   
   
   Stop nfs/rpcbind/portmap services provided by the platform:
   service nfs stop
   service rpcbind stop
   
   
   
   
   Start the included portmap package (needs root privileges):
   hadoop portmap
   
   OR
   hadoop-daemon.sh start portmap
   
   
   
   
   Start mountd and nfsd.
   
   No root privileges are required for this command. However, verify that the user starting the Hadoop cluster and the user starting the NFS gateway are same.
   hadoop nfs3
   
   
   OR
   hadoop-daemon.sh start nfs3
   
   
   
   

4. 
   
   Verify that the HDFS namespace is exported and can be mounted.
   showmount -e $nfs_server_ip                         
   
   You should see output similar to the following:
   Exports list on $nfs_server_ip :
   / *
   
   
   
   
   

5. Mount the export “/”  on a windows client.

   mount -o nolock $your_ip:/!  W: