上課學Big Data需要撰寫MapReduce Java程序,寫了一個小腳本輔助運行。
#!/bin/bash
javaclassname=$1
localinput=$2
inputpath=$3
outputpath=$4
# Function to delete .class files
function delete_class_files {
if [[ -e ./${javaclassname}*.class ]]; then
rm ./${javaclassname}*.class
fi
}
# Call the function to delete .class files
delete_class_files
if hdfs dfs -test -e ${outputpath}
then
hdfs dfs -rm -r -f ${outputpath}
fi
hdfs dfs -mkdir -p ${inputpath}
hdfs dfs -copyFromLocal ${localinput}/* ${inputpath}
hadoop com.sun.tools.javac.Main ${javaclassname}.java
jar cf ${javaclassname}.jar ${javaclassname}*.class
yarn jar ${javaclassname}.jar ${javaclassname} ${inputpath} ${outputpath}
hdfs dfs -cat ${outputpath}/*
這個腳本可以自動檢測當前目錄是否有多餘的class文件,然後檢查hdfs裡面有沒有已經存在的output文件,然後自動編譯打包並運行jar,然後打印output的內容。
如果提示找不到類的話,應該是沒添加環境路徑,運行以下Command
sed -i '/export JAVA_HOME=\/usr\/lib\/jvm\/java-8-openjdk-amd64/a export PATH=$PATH:$JAVA_HOME/bin\nexport HADOOP_CLASSPATH=${JAVA_HOME}/lib/tools.jar' /usr/local/hadoop/etc/hadoop/hadoop-env.sh
順便做了兩個通過start-all.sh和stop-all.sh自動輸入密碼的小腳本
#!/bin/bash
password=$1
expect << EOF
set timeout 30
spawn start-all.sh
expect {
"yes/no" { send "yes\n";exp_continue }
"password" { send "$password\n" }
}
expect {
"yes/no" { send "yes\n";exp_continue }
"password" { send "$password\n" }
}
expect {
"yes/no" { send "yes\n";exp_continue }
"password" { send "$password\n" }
}
expect {
"yes/no" { send "yes\n";exp_continue }
"password" { send "$password\n" }
}
EOF
Hello,
PRIVATE MP3 FTP SERVER 0-DAY Access https://0daymusic.org
SOME DETAILS ON PREMIUM ACCOUNT:
* Reseller payment method:AltCoins, Webmoney, Perfect Money.
* Choose payment method: BitCoin, Bank wire, Western Union.
* Server's capacity: 347 TB MP3, FLAC, LIVESETS, Music Videos.
* Support: FTP, FTPS (File Transfer Protocol Secure), SFTP and HTTP, HTTPS.
* Account delivery time: 1 to 48 hours.
* More 15 years Of Archives.
* Overal server's speed: 1 Gb/s.
* Easy to use: Most of genres are sorted by days.