Thursday, July 30, 2020

Hadoop Environment Setting


# Java Environment Variables
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export PATH=$PATH:$JAVA_HOME/bin

#Cassandra Environment Variables
export CASSANDRA_HOME=/usr/local/softwares/cassandra-3.11.7
export PATH=$PATH:$CASSANDRA_HOME/bin

#Spark Environment Variables
export SPARK_HOME=/usr/local/softwares/spark-2.4.6-hadoop2.7
export PATH=$PATH:$SPARK_HOME/bin

#Kafka Environment Variables
export KAFKA_HOME=/usr/local/softwares/kafka_2.12-2.4.1
export PATH=$PATH:$KAFKA_HOME/bin

#Hadop Environment Variables

export HADOOP_HOME=/usr/local/softwares/hadoop-2.8.5
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export HADOOP_HDFS_HOME=$HADOOP_HOME
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native

export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native"

# Hive Environment Variables
export HIVE_HOME=/usr/local/softwares/apache-hive-2.3.5
export PATH=$PATH:$HIVE_HOME/bin

No comments:

Post a Comment

Recent Post

Databricks Delta table merge Example

here's some sample code that demonstrates a merge operation on a Delta table using PySpark:   from pyspark.sql import SparkSession # cre...