Compile Hadoop from Source

Compile Hadoop 2.x help you fix error Unable to load native-hadoop library

  1. Install Requirements

    • Java Oracle: otherwise, while maven build you may be occur exception below if not using java oracle
      Failed to execute goal on project hadoop-annotations:
      Could not resolve dependencies for project org.apache.hadoop:hadoop-annotations:jar:2.7.1:
      Could not find artifact jdk.tools:jdk.tools:jar:1.7 at specified path
      /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.85.x86_64/jre/../lib/tools.jar
      
    • Maven, libssl-dev, zlib-development, cmake
    • Protocol Buffers: Version must be v2.5.0, otherwise while maven build you can occur exception below
      Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:2.7.1:protoc (compile-protoc)
      on project hadoop-common: org.apache.maven.plugin.MojoExecutionException:
      protoc version is 'libprotoc 2.6.1', expected version is '2.5.0'
      
      wget https://github.com/google/protobuf/releases/download/v2.5.0/protobuf-2.5.0.tar.gz
      tar -xzf protobuf-2.5.0.tar.gz
      cd protobuf-2.5.0
      ./configure
      make
      make check
      make install
      protoc --version
      

      While install Protocol Buffers, if you occur exception below, this mean you using old gcc version

      error: uninitialized member 'google::protobuf::util::converter::DataPiece::
      

      Upgrade gcc

      cd /etc/yum.repos.d
      wget http://people.centos.org/tru/devtools-2/devtools-2.repo
      yum --enablerepo=testing-devtools-2-centos-6 install devtoolset-2-gcc devtoolset-2-gcc-c++
      # export CC=/opt/rh/devtoolset-2/root/usr/bin/gcc
      # export CPP=/opt/rh/devtoolset-2/root/usr/bin/cpp
      # export CXX=/opt/rh/devtoolset-2/root/usr/bin/c++
      # gcc --version # still old version, but CC, CPP, CXX used in protobuf build
      ln -s /opt/rh/devtoolset-2/root/usr/bin/* /usr/local/bin/
      hash -r
      gcc --version
      

      While install Protocol Buffers, if you occur exception below, this mean you using old binutils version

      error: uninitialized member 'google::protobuf::util::converter::DataPiece::
      

      Upgrade binutils

      yum --enablerepo=testing-devtools-2-centos-6 install devtoolset-2-binutils
      ln -s /opt/rh/devtoolset-2/root/usr/bin/as /usr/local/bin/as
      hash -r
      as --version
      
  2. Download Hadoop Source

    wget http://mirrors.maychuviet.vn/apache/hadoop/common/hadoop-2.7.1/hadoop-2.7.1-src.tar.gz
    tar -xzf hadoop-2.7.1-src.tar.gz
    
  3. Compile Hadoop from Source

    cd hadoop-2.7.1-src
    mvn package -Pdist,native -DskipTests -Dtar
    

Comments

Popular posts from this blog

Reduce TIME_WAIT Socket Connections