从零开始搭建个人大数据集群——环境准备篇
从零开始搭建个人大数据集群（1）——zookeeper
从零开始搭建个人大数据集群（2）——HDFS
从零开始搭建个人大数据集群（3）——YARN
从零开始搭建个人大数据集群（4）——HIVE
从零开始搭建个人大数据集群（5）——HBASE
从零开始搭建个人大数据集群（6）——SPARK
从零开始搭建个人大数据集群（7）——KAFKA

安装前的准备

1.安装好并启动kafka集群
2.下载好flink-1.13.0-bin-scala_2.12.tgz

解压

cd /opt/packages
tar -zxf flink-1.13.0-bin-scala_2.12.tgz ./apps/
cd ../apps
ln -s flink-1.13.0 flink

高可用配置

flink-conf.yaml

#==============================================================================
# Common
#==============================================================================
jobmanager.rpc.address: hd1
jobmanager.rpc.port: 6123

jobmanager.memory.process.size: 1600m
taskmanager.memory.process.size: 1728m
taskmanager.numberOfTaskSlots: 1

parallelism.default: 1

#==============================================================================
# High Availability
#==============================================================================

# The high-availability mode. Possible options are 'NONE' or 'zookeeper'.
#
high-availability: zookeeper
high-availability.storageDir: hdfs://ns1/flink/ha/
high-availability.zookeeper.quorum: hd1:2181,hd2:2181,hd3:2181
high-availability.zookeeper.path.root: /flink
high-availability.cluster-id: /FlinkCluster

state.backend: filesystem
state.checkpoints.dir: hdfs://ns1/flink/checkpoints
jobmanager.execution.failover-strategy: region

masters

hd1:8081
hd2:8081

workers

hd3
hd4
hd5

环境变量 /etc/profile

注意这里需要export HADOOP_CLASSPATH，否则flink无法兼容hadoop

JAVA_HOME=/usr/local/jdk
ZOOKEEPER_HOME=/opt/apps/zookeeper
HADOOP_HOME=/opt/apps/hadoop
HADOOP_COMMON_HOME=${HADOOP_HOME}
HADOOP_HDFS_HOME=${HADOOP_HOME}
HADOOP_MAPRED_HOME=${HADOOP_HOME}
HADOOP_YARN_HOME=${HADOOP_HOME}
HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop
export HADOOP_CLASSPATH=`${HADOOP_HOME}/bin/hadoop classpath`
KAFKA_HOME=/opt/apps/kafka
export LD_LIBRARY_PATH=$HADOOP_HOME/lib/native:/usr/lib64

HBASE_HOME=/opt/apps/hbase
HBASE_LIBRARY_PATH=${HBASE_HOME}/lib/native/Linux-amd64-64
HIVE_HOME=/opt/apps/hive
HIVE_CONF_DIR=${HIVE_HOME}/conf
SPARK_HOME=/opt/apps/spark
FLINK_HOME=/opt/apps/flink
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HBASE_HOME/bin:$HIVE_HOME/bin:$ZOOKEEPER_HOME/bin:$SPARK_HOME/bin:$MAVEN_HOME/bin:$KAFKA_HOME/bin:$FLINK_HOME/bin

启动

# 启动zookeeper
zkman start
# 启动hdfs
start-all.sh
# 启动flink
start-cluster.sh

启动成功后访问http://hd1:8081

Q.E.D.