Hadoop自動編譯Java MapReduce源碼並運行輸出output

2024.03.14 网络 0 评

上課學Big Data需要撰寫MapReduce Java程序，寫了一個小腳本輔助運行。

#!/bin/bash

javaclassname=$1
localinput=$2
inputpath=$3
outputpath=$4

# Function to delete .class files
function delete_class_files {
    if [[ -e ./${javaclassname}*.class ]]; then
        rm ./${javaclassname}*.class
    fi
}

# Call the function to delete .class files
delete_class_files

if hdfs dfs -test -e ${outputpath}
then
    hdfs dfs -rm -r -f ${outputpath}
fi

hdfs dfs -mkdir -p ${inputpath}
hdfs dfs -copyFromLocal ${localinput}/* ${inputpath}
hadoop com.sun.tools.javac.Main ${javaclassname}.java
jar cf ${javaclassname}.jar ${javaclassname}*.class
yarn jar ${javaclassname}.jar ${javaclassname} ${inputpath} ${outputpath}
hdfs dfs -cat ${outputpath}/*

這個腳本可以自動檢測當前目錄是否有多餘的class文件，然後檢查hdfs裡面有沒有已經存在的output文件，然後自動編譯打包並運行jar，然後打印output的內容。

如果提示找不到類的話，應該是沒添加環境路徑，運行以下Command

sed -i '/export JAVA_HOME=\/usr\/lib\/jvm\/java-8-openjdk-amd64/a export PATH=$PATH:$JAVA_HOME/bin\nexport HADOOP_CLASSPATH=${JAVA_HOME}/lib/tools.jar' /usr/local/hadoop/etc/hadoop/hadoop-env.sh

順便做了兩個通過start-all.sh和stop-all.sh自動輸入密碼的小腳本

#!/bin/bash 
password=$1

expect << EOF  
    set timeout 30
    spawn start-all.sh
    expect { 
        "yes/no" { send "yes\n";exp_continue } 
        "password" { send "$password\n" }
    } 
    expect { 
        "yes/no" { send "yes\n";exp_continue } 
        "password" { send "$password\n" }
    }
    expect { 
        "yes/no" { send "yes\n";exp_continue } 
        "password" { send "$password\n" }
    }
    expect { 
        "yes/no" { send "yes\n";exp_continue } 
        "password" { send "$password\n" }
    }
EOF

杯子茶室

关注有趣的事物

Hadoop自動編譯Java MapReduce源碼並運行輸出output