从零开始搭建个人大数据集群——环境准备篇
从零开始搭建个人大数据集群(1)——zookeeper
从零开始搭建个人大数据集群(2)——HDFS
从零开始搭建个人大数据集群(3)——YARN
从零开始搭建个人大数据集群(4)——HIVE
从零开始搭建个人大数据集群(5)——HBASE
从零开始搭建个人大数据集群(6)——SPARK
从零开始搭建个人大数据集群(7)——KAFKA
安装前的准备
1.安装好并启动kafka集群
2.下载好flink-1.13.0-bin-scala_2.12.tgz
解压
cd /opt/packages
tar -zxf flink-1.13.0-bin-scala_2.12.tgz ./apps/
cd ../apps
ln -s flink-1.13.0 flink
高可用配置
flink-conf.yaml
#==============================================================================
# Common
#==============================================================================
jobmanager.rpc.address: hd1
jobmanager.rpc.port: 6123
jobmanager.memory.process.size: 1600m
taskmanager.memory.process.size: 1728m
taskmanager.numberOfTaskSlots: 1
parallelism.default: 1
#==============================================================================
# High Availability
#==============================================================================
# The high-availability mode. Possible options are 'NONE' or 'zookeeper'.
#
high-availability: zookeeper
high-availability.storageDir: hdfs://ns1/flink/ha/
high-availability.zookeeper.quorum: hd1:2181,hd2:2181,hd3:2181
high-availability.zookeeper.path.root: /flink
high-availability.cluster-id: /FlinkCluster
state.backend: filesystem
state.checkpoints.dir: hdfs://ns1/flink/checkpoints
jobmanager.execution.failover-strategy: region
masters
hd1:8081
hd2:8081
workers
hd3
hd4
hd5
环境变量 /etc/profile
注意这里需要export HADOOP_CLASSPATH
,否则flink无法兼容hadoop
JAVA_HOME=/usr/local/jdk
ZOOKEEPER_HOME=/opt/apps/zookeeper
HADOOP_HOME=/opt/apps/hadoop
HADOOP_COMMON_HOME=${HADOOP_HOME}
HADOOP_HDFS_HOME=${HADOOP_HOME}
HADOOP_MAPRED_HOME=${HADOOP_HOME}
HADOOP_YARN_HOME=${HADOOP_HOME}
HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop
export HADOOP_CLASSPATH=`${HADOOP_HOME}/bin/hadoop classpath`
KAFKA_HOME=/opt/apps/kafka
export LD_LIBRARY_PATH=$HADOOP_HOME/lib/native:/usr/lib64
HBASE_HOME=/opt/apps/hbase
HBASE_LIBRARY_PATH=${HBASE_HOME}/lib/native/Linux-amd64-64
HIVE_HOME=/opt/apps/hive
HIVE_CONF_DIR=${HIVE_HOME}/conf
SPARK_HOME=/opt/apps/spark
FLINK_HOME=/opt/apps/flink
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HBASE_HOME/bin:$HIVE_HOME/bin:$ZOOKEEPER_HOME/bin:$SPARK_HOME/bin:$MAVEN_HOME/bin:$KAFKA_HOME/bin:$FLINK_HOME/bin
启动
# 启动zookeeper
zkman start
# 启动hdfs
start-all.sh
# 启动flink
start-cluster.sh
启动成功后访问http://hd1:8081
Q.E.D.