Day01-ElasticSearch的单点部署，集群部署，多实例部署，es-head和postman环境搭建

0、elasticsearch的简单介绍

elasticstack：

• elastic$earch
  • 日志存储和检索
• filebeat：
  • 用于日志收集
• logstash：
  • 日志转换，也可以用于日志采集
• kibana：
  • 用于日志展示
• tb(pb)级别日志处理：
  • kafka
  • zookeeper

elasticsearch:

• 单点部署

• 集群部署二

• es集群的常见术语

  • 分片
  • 副本
  • 索引
  • 文档
  • …

• 集群的api基础操作

  • 索引管理
  • 文档管理

• 集群环境准备：

ip地址	配置	主机名
10.0.0.101	cpu:2c memory:4g disk:50g+	elk101.oldboyedu.com
10.0.0.102	cpu:2c memory:4g disk:50g+	elk102.oldboyedu.com
10.0.0.103	cpu:2c memory:4g disk:50g+	elk103.oldboyedu.com

配置好后拍快照

1、elasticsearch的单点部署

(1)下载es软件包
https://www.elastic.co/cn/downloads

(2)老男孩线下同学下载

curl -o elasticsearch-7.17.5-x86_64.rpm  http://192.168.15.253/elasticstack/softwares/rpm/elasticsearch-7.17.5-x86_64.rpm

(3)安装es

rpm -ivh elasticsearch-7.17.5-x86_64.rpm 

(4)修改es的配置文件

vim /etc/elasticsearch/elasticsearch.yml
...
# es服务监听对外暴露服务的地址
network.host: 0.0.0.0

# 指定es集群的节点ip
discovery.seed_hosts: ["10.0.0.101"]

# 指定参与master选举的节点
cluster.initial_master_nodes: ["10.0.0.101"]

(5)启动es服务

systemctl enable --now elasticsearch

(6)验证节点是否正常工作

[root@elk101 ~]# ss -ntl
state       recv-q send-q         local address:port                        peer address:port     
...
listen      0      16384                   [::]:9200                                [::]:*        
listen      0      16384                   [::]:9300                                [::]:*        

• 面试题1:
  • 9200端口作用:
    • 对es集群外部提供http/https服务。可以理解为对客户端提供服务。
  • 9300端口作用:
    • 对es集群内部进行数据通信传输端口。走的时候tcp协议。

(7)客户端验证

curl 10.0.0.101:9200

如果遇到集群的uuid为"_na_"情况时，可以执行如下操作:

# 两块网卡时，打开指定参与master选举的节点
vim /etc/elasticsearch/elasticsearch.yml
cluster.initial_master_nodes: ["10.0.0.101"]
# 然后执行
systemctl stop elasticsearch.service
rm -rf /var/lib/elasticsearch/* /var/log/elasticsearch/* /tmp/*
systemctl start elasticsearch.service
curl 10.0.0.101:9200

2、elasticsearch的集群部署

(1)下载es软件包
https://www.elastic.co/cn/downloads

(2)老男孩线下同学下载

curl -o elasticsearch-7.17.5-x86_64.rpm http://192.168.15.253/elasticstack/softwares/rpm/elasticsearch-7.17.5-x86_64.rpm

(3)所有节点安装es

rpm -ivh elasticsearch-7.17.5-x86_64.rpm 

(4)elk101.oldboyedu.com做数据清空

systemctl stop elasticsearch.service
rm -rf /var/lib/elasticsearch/* /var/log/elasticsearch/* /tmp/*	

(5)修改es的配置文件

vim /etc/elasticsearch/elasticsearch.yml 
...

# 指定es集群的名称
cluster.name: oldboyedu-linux85

# es服务监听对外暴露服务的地址
network.host: 0.0.0.0

# 指定es集群的节点ip
discovery.seed_hosts: ["10.0.0.101","10.0.0.102","10.0.0.103"]

# 指定参与master选举的节点
cluster.initial_master_nodes: ["10.0.0.101","10.0.0.102","10.0.0.103"]

(6)将配置文件分发到其他两个节点

scp /etc/elasticsearch/elasticsearch.yml  10.0.0.102:/etc/elasticsearch/elasticsearch.yml
scp /etc/elasticsearch/elasticsearch.yml  10.0.0.103:/etc/elasticsearch/elasticsearch.yml

(6)所有节点启动es服务

systemctl enable --now elasticsearch

(7)验证es集群节点是否正常工作

[root@elk101.oldboyedu.com ~]# curl 10.0.0.101:9200/_cat/nodes   # *在谁那谁就是老大
10.0.0.103 11 71 51 1.28 0.38 0.13 cdfhilmrstw - elk103.oldboyedu.com
10.0.0.102 15 73 46 1.10 0.34 0.12 cdfhilmrstw - elk102.oldboyedu.com
10.0.0.101  6 92  1 0.07 0.11 0.12 cdfhilmrstw * elk101.oldboyedu.com

# 半数以上机器存活则集群正常，为了防止脑裂，半数以上机器不存活的话，集群不提供对外服务

[root@elk101.oldboyedu.com ~]# 
[root@elk101.oldboyedu.com ~]# curl 10.0.0.101:9200/_cat/nodes?v
ip         heap.percent ram.percent cpu load_1m load_5m load_15 m node.role   master name
10.0.0.103           12          71   0    0.72    0.34     0.13 cdfhilmrstw -      elk103.oldboyedu.com
10.0.0.102           16          73   0    0.61    0.31     0.12 cdfhilmrstw -      elk102.oldboyedu.com
10.0.0.101            7          92   0    0.04    0.10     0.11 cdfhilmrstw *      elk101.oldboyedu.com

heap.percent   # 堆内存使用百分比
ram.percent    # 内存百分比

3、基于二进制部署elasticsearch

3.1 准备阶段

(1)elk101配置集群免密登录及同步脚本
1.1 修改主机列表

cat >> /etc/hosts <<'eof'
10.0.0.101 elk101.oldboyedu.com
10.0.0.102 elk102.oldboyedu.com
10.0.0.103 elk103.oldboyedu.com
eof

1.2 elk101节点上生成密钥对

ssh-keygen -t rsa -p '' -f ~/.ssh/id_rsa -q

1.3 elk101配置所有集群节点的免密登录

for ((host_id=101;host_id<=103;host_id++));do ssh-copy-id elk${host_id}.oldboyedu.com ;done

1.4 链接测试

ssh 'elk101.oldboyedu.com'
ssh 'elk102.oldboyedu.com'
ssh 'elk103.oldboyedu.com'

1.5 所有节点安装rsync数据同步工具

yum -y install rsync

1.6 编写同步脚本

cat > /usr/local/sbin/data_rsync.sh <<'eof'
#!/bin/bash

# auther: jason yin

if  [ $# -ne 1 ];then
   echo "usage: $0 /path/to/file(绝对路径)"
   exit
fi 

# 判断文件是否存在

if [ ! -e $1 ];then
    echo "[ $1 ] dir or file not find!"
    exit
fi

# 获取父路径

fullpath=`dirname $1`

# 获取子路径

basename=`basename $1`

# 进入到父路径

cd $fullpath

for ((host_id=102;host_id<=103;host_id++))
  do
    # 使得终端输出变为绿色
    tput setaf 2
    echo ===== rsyncing elk${host_id}.oldboyedu.com: $basename =====
    # 使得终端恢复原来的颜色
    tput setaf 7
    # 将数据同步到其他两个节点
    rsync -apz $basename  `whoami`@elk${host_id}.oldboyedu.com:$fullpath
    if [ $? -eq 0 ];then
      echo "命令执行成功!"
    fi
done
eof

1.7 给脚本授权

chmod +x /usr/local/sbin/data_rsync.sh

3.2 部署阶段

(2)elk101下载es软件包

curl -o elasticsearch-7.17.5-linux-x86_64.tar.gz http://192.168.15.253/elasticstack/softwares/binary/elasticsearch-7.17.5-linux-x86_64.tar.gz

(3)所有节点创建运行es服务的用户

useradd -u 2023 oldboyedu

(4)elk101创建elasticsearch的工作目录
4.1 可以如下操作

mkdir -pv /oldboyedu/{data,logs,softwares}/es7
chown oldboyedu:oldboyedu -r /oldboyedu/{softwares,data,logs}/es7/
ll /oldboyedu/{data,logs,softwares}/es7 -d

4.2 也可以如下操作（推荐）

install -d /oldboyedu/{data,logs,softwares}/es7 -o oldboyedu -g oldboyedu

(5)elk101解压软件包

tar xf elasticsearch-7.17.5-linux-x86_64.tar.gz -c /oldboyedu/softwares/es7/

(6)elk101修改配置文件

[root@elk101.oldboyedu.com ~]# egrep -v "^#|^$" /oldboyedu/softwares/es7/elasticsearch-7.17.5/config/elasticsearch.yml 
cluster.name: oldboyedu-linux85-binary
path.data: /oldboyedu/data/es7
path.logs: /oldboyedu/logs/es7
network.host: 0.0.0.0
discovery.seed_hosts: ["elk101.oldboyedu.com","elk102.oldboyedu.com","elk103.oldboyedu.com"]
cluster.initial_master_nodes: ["elk101.oldboyedu.com","elk102.oldboyedu.com","elk103.oldboyedu.com"]

(7)elk101同步程序目录

[root@elk101.oldboyedu.com ~]# data_rsync.sh  /oldboyedu

(8)elk101节点针对es基础调优
8.1 修改文件打开数量上线，修改后需要断开会话

[root@elk101.oldboyedu.com ~]# cat /etc/security/limits.d/es7.conf

*	soft	nofile	65535
*	hard	nofile	131070
*	hard	nproc	8192

[root@elk101.oldboyedu.com ~]# ulimit -sn
65535

[root@elk101.oldboyedu.com ~]# ulimit -hn
131070

8.2 调大内核虚拟内存映射值

[root@elk101.oldboyedu.com ~]# cat  /etc/sysctl.d/es.conf
vm.max_map_count=524288

[root@elk101.oldboyedu.com ~]# sysctl -q vm.max_map_count
vm.max_map_count = 65530

[root@elk101.oldboyedu.com ~]# sysctl -f  /etc/sysctl.d/es.conf    # 生效
vm.max_map_count = 524288

[root@elk101.oldboyedu.com ~]# sysctl -q vm.max_map_count
vm.max_map_count = 524288

8.3 同步调优文件

[root@elk101.oldboyedu.com ~]# data_rsync.sh  /etc/sysctl.d/es.conf
[root@elk101.oldboyedu.com ~]# data_rsync.sh  /etc/security/limits.d/es7.conf 
[root@elk101.oldboyedu.com ~]# data_rsync.sh /etc/hosts

(9)所有节点启动服务

9.1 先断开连接

让文件打开数量上限生效。

9.2 使得内核参数生效

sysctl -f  /etc/sysctl.d/es.conf	

9.3 后台启动es服务

chown oldboyedu:oldboyedu -r /oldboyedu/{softwares,data,logs}/es7/
su oldboyedu -c '/oldboyedu/softwares/es7/elasticsearch-7.17.5/bin/elasticsearch -d'	

(10)验证服务是否正常

[root@elk101.oldboyedu.com ~]# curl 10.0.0.101:9200
{
  "name" : "elk101.oldboyedu.com",
  "cluster_name" : "oldboyedu-linux85-binary",
  "cluster_uuid" : "l0vl0fymsmsun_mnv3dh7g",
  "version" : {
    "number" : "7.17.5",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "8d61b4f7ddf931f219e3745f295ed2bbc50c8e84",
    "build_date" : "2022-06-23t21:57:28.736740635z",
    "build_snapshot" : false,
    "lucene_version" : "8.11.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "you know, for search"
}

[root@elk101.oldboyedu.com ~]# curl 10.0.0.101:9200/_cat/nodes
10.0.0.101 27 96 38 0.68 0.26 0.14 cdfhilmrstw * elk101.oldboyedu.com
10.0.0.102 23 88 38 0.49 0.21 0.14 cdfhilmrstw - elk102.oldboyedu.com
10.0.0.103 21 88 42 1.25 0.42 0.22 cdfhilmrstw - elk103.oldboyedu.com

补充：若集群uuid出现"_na_"状态时，做如下动作，所有节点操作

pkill java
rm -rf  /oldboyedu/data/es7/* /oldboyedu/logs/es7/* /tmp/*

3.3 使用systemctl管理es服务

1.elk101停止es服务

pkill java

2.elk101编写es启动脚本

cat > /usr/lib/systemd/system/es7.service <<eof
[unit]
description=oldboyedu linux85 es7
after=network.target

[service]
type=simple
execstart=/oldboyedu/softwares/es7/elasticsearch-7.17.5/bin/elasticsearch
user=oldboyedu
limitnofile=131070

[install]
wantedby=multi-user.target
eof

3.elk101加载脚本

systemctl daemon-reload

4.elk101设置开机自启动

systemctl enable --now es7

5.elk101查看服务是否正常运行

systemctl status es7
ss -ntl
curl 10.0.0.101:9200/_cat/nodes

6.其他节点使用脚本

data_rsync.sh /usr/lib/systemd/system/es7.service

7.其他节点使用脚本管理

pkill java
systemctl enable --now es7
systemctl status es7

3.4 使用oracle jdk管理es服务

(1)elk101下载jdk环境

curl -o jdk-8u291-linux-x64.tar.gz http://192.168.15.253/elasticstack/softwares/binary/jdk-8u291-linux-x64.tar.gz

(2)elk101解压jdk软件包

tar xf jdk-8u291-linux-x64.tar.gz -c /oldboyedu/softwares/

(3)elk101配置系统环境变量

[root@elk101.oldboyedu.com ~]# cat /etc/profile.d/jdk.sh 
#!/bin/bash

export java_home=/oldboyedu/softwares/jdk1.8.0_291
export path=$path:$java_home/bin

[root@elk101 ~]# source /etc/profile.d/jdk.sh
[root@elk101 ~]# which java
/oldboyedu/softwares/jdk1.8.0_291/bin/java
[root@elk101.oldboyedu.com ~]# java -version
java version "1.8.0_291"
java(tm) se runtime environment (build 1.8.0_291-b10)
java hotspot(tm) 64-bit server vm (build 25.291-b10, mixed mode)

(4)elk101修改启动脚本

cat > /usr/lib/systemd/system/es7.service <<eof
[unit]
description=oldboyedu linux85 es7
after=network.target

[service]
type=simple
environment=java_home=/oldboyedu/softwares/jdk1.8.0_291
execstart=/oldboyedu/softwares/es7/elasticsearch-7.17.5/bin/elasticsearch
user=oldboyedu
limitnofile=131070
limitnproc=8192

[install]
wantedby=multi-user.target
eof

systemctl daemon-reload
systemctl start es7

(5)elk101重启服务

systemctl restart es7

(6)elk101查看服务是否正常

[root@elk101.oldboyedu.com ~]# curl 10.0.0.101:9200/_cat/nodes
10.0.0.101 17 74 6 0.17 0.28 0.19 cdfhilmrstw - elk101.oldboyedu.com
10.0.0.102 11 90 0 0.03 0.03 0.05 cdfhilmrstw * elk102.oldboyedu.com
10.0.0.103 21 91 0 0.00 0.01 0.05 cdfhilmrstw - elk103.oldboyedu.com
root@elk101 ~]# jps
3572 jps
3371 elasticsearch
[root@elk101 ~]# ps -ef|grep java
oldboye+   3371      1 99 14:02 ?        00:00:35 /oldboyedu/softwares/jdk1.8.0_291/bin/java -xshare:auto -des.networkaddress.cache.ttl=60 -des.networkaddress.cache.negative.ttl=10 -xx:+alwayspretouch -xss1m -djava.awt.headless=true -dfile.encoding=utf-8 -djna.nosys=true -xx:-omitstacktraceinfastthrow -dio.netty.nounsafe=true -dio.netty.nokeysetoptimization=true -dio.netty.recycler.maxcapacityperthread=0 -dio.netty.allocator.numdirectarenas=0 -dlog4j.shutdownhookenabled=false -dlog4j2.disable.jmx=true -dlog4j2.formatmsgnolookups=true -djava.locale.providers=spi,jre -xx:+useconcmarksweepgc -xx:cmsinitiatingoccupancyfraction=75 -xx:+usecmsinitiatingoccupancyonly -djava.io.tmpdir=/tmp/elasticsearch-1143701341521397671 -xx:+heapdumponoutofmemoryerror -xx:heapdumppath=data -xx:errorfile=logs/hs_err_pid%p.log -xx:+printgcdetails -xx:+printgcdatestamps -xx:+printtenuringdistribution -xx:+printgcapplicationstoppedtime -xloggc:logs/gc.log -xx:+usegclogfilerotation -xx:numberofgclogfiles=32 -xx:gclogfilesize=64m -xms1024m -xmx1024m -xx:maxdirectmemorysize=536870912 -des.path.home=/oldboyedu/softwares/es7/elasticsearch-7.17.5 -des.path.conf=/oldboyedu/softwares/es7/elasticsearch-7.17.5/config -des.distribution.flavor=default -des.distribution.type=tar -des.bundled_jdk=true -cp /oldboyedu/softwares/es7/elasticsearch-7.17.5/lib/* org.elasticsearch.bootstrap.elasticsearch
root       3590   2034  0 14:03 pts/0    00:00:00 grep --color=auto java

(7)elk101同步配置到其他节点

data_rsync.sh /etc/profile.d/jdk.sh
data_rsync.sh /usr/lib/systemd/system/es7.service
data_rsync.sh /oldboyedu/softwares/jdk1.8.0_291

(8)其他节点重启服务

systemctl daemon-reload  && systemctl restart es7

3.5 修改es环境的堆(heap)内存大小

(1)elk101修改配置文件

[root@elk101.oldboyedu.com ~]# vim /oldboyedu/softwares/es7/elasticsearch-7.17.5/config/jvm.options
...
-xms256m
-xmx256m

(2)elk101重启服务并验证堆内存大小，若不配置，默认1gb

[root@elk101 ~]# systemctl restart es7
[root@elk101 ~]# jps
3987 jps
3753 elasticsearch
[root@elk101 ~]# jmap -heap 3753|grep maxheapsize
   maxheapsize              = 268435456 (256.0mb)

(3)elk101同步环境到其他节点

data_rsync.sh /oldboyedu/softwares/es7/elasticsearch-7.17.5/config/jvm.options

(4)其他节点重启服务并验证堆内存大小

[root@elk102.oldboyedu.com ~]# systemctl restart es7

[root@elk102.oldboyedu.com ~]# jmap -heap `jps | awk '/elasticsearch/{print $1}'` | grep maxheapsize
   maxheapsize              = 268435456 (256.0mb)

3.6 es集群的多实例部署

• 需要解决的问题
  • 程序安装目录
  • 数据目录
  • 日志目录
  • 端口冲突：9200|9300

(1)所有节点准备es6的工作目录

install -d /oldboyedu/{data,logs,softwares}/es6 -o oldboyedu -g oldboyedu

(2)elk101下载es6软件包

curl -o elasticsearch-6.8.23.tar.gz http://192.168.15.253/elasticstack/softwares/binary/elasticsearch-6.8.23.tar.gz

(3)elk101解压软件包
tar xf elasticsearch-6.8.23.tar.gz -c /oldboyedu/softwares/es6/
chown oldboyedu:oldboyedu -r /oldboyedu/softwares/es6/elasticsearch-6.8.23/
(4)elk101修改配置文件

[root@elk101.oldboyedu.com ~]# yy /oldboyedu/softwares/es6/elasticsearch-6.8.23/config/elasticsearch.yml 
cluster.name: oldboyedu-linux85-es6
node.name: elk101.oldboyedu.com
path.data: /oldboyedu/data/es6
path.logs: /oldboyedu/logs/es6
network.host: 0.0.0.0
http.port: 19200
transport.tcp.port: 19300
discovery.zen.ping.unicast.hosts: ["elk101.oldboyedu.com","elk102.oldboyedu.com","elk103.oldboyedu.com"]
discovery.zen.minimum_master_nodes: 2
[root@elk101.oldboyedu.com ~]# 

(5)elk101修改堆内存大小

[root@elk101.oldboyedu.com ~]# vim /oldboyedu/softwares/es6/elasticsearch-6.8.23/config/jvm.options 
...
-xms256m
-xmx256m

(6)elk101编写启动脚本

cat > /usr/lib/systemd/system/es6.service <<eof
[unit]
description=oldboyedu linux85 es6
after=network.target

[service]
type=simple
environment=java_home=/oldboyedu/softwares/jdk1.8.0_291
execstart=/oldboyedu/softwares/es6/elasticsearch-6.8.23/bin/elasticsearch
user=oldboyedu
limitnofile=131070

[install]
wantedby=multi-user.target
eof

(7)elk101同步配置文件

data_rsync.sh /oldboyedu/softwares/es6/
data_rsync.sh /usr/lib/systemd/system/es6.service

(8)修改各节点的配置文件

[root@elk102.oldboyedu.com ~]# vim  /oldboyedu/softwares/es6/elasticsearch-6.8.23/config/elasticsearch.yml 
...
node.name: elk102.oldboyedu.com


[root@elk103.oldboyedu.com ~]# vim  /oldboyedu/softwares/es6/elasticsearch-6.8.23/config/elasticsearch.yml 
...
node.name: elk103.oldboyedu.com

(9)启动服务

systemctl daemon-reload && systemctl enable --now es6

(10)查看服务状态

[root@elk101.oldboyedu.com ~]# curl 10.0.0.101:19200/_cat/nodes
10.0.0.102 50 77  1 0.58 0.25 0.14 mdi - elk102.oldboyedu.com
10.0.0.101 59 82  1 0.37 0.25 0.13 mdi * elk101.oldboyedu.com
10.0.0.103 44 76 42 0.46 0.22 0.12 mdi - elk103.oldboyedu.com

查看集群节点的api：

curl 10.0.0.101:9200/_cat/nodes
[root@elk101 ~]# curl 10.0.0.101:9200/_cat/nodes
10.0.0.103 43 75 0 0.00 0.01 0.05 cdfhilmrstw - elk103.oldboyedu.com
10.0.0.101 35 80 0 0.02 0.03 0.05 cdfhilmrstw - elk101.oldboyedu.com
10.0.0.102 57 84 0 0.00 0.01 0.05 cdfhilmrstw * elk102.oldboyedu.com

查看集群状态:

curl 10.0.0.101:9200/_cluster/health 2>/dev/null | jq
[root@elk101 ~]# curl 10.0.0.101:9200/_cluster/health 2>/dev/null | jq
{
  "cluster_name": "oldboyedu-linux85-binary",
  "status": "green",
  "timed_out": false,
  "number_of_nodes": 3,
  "number_of_data_nodes": 3,
  "active_primary_shards": 2,
  "active_shards": 4,
  "relocating_shards": 0,
  "initializing_shards": 0,
  "unassigned_shards": 0,
  "delayed_unassigned_shards": 0,
  "number_of_pending_tasks": 0,
  "number_of_in_flight_fetch": 0,
  "task_max_waiting_in_queue_millis": 0,
  "active_shards_percent_as_number": 100
}

3.7 常见报错

1.java.lang.runtimeexception: can not run elasticsearch as root	
报错原因:
	不能以root用户启动es服务。
解决方案:
	使用普通用户启动服务即可。

2.bootstrap check failure [1] of [3]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65535]
报错原因:
	程序默认的的文件打开数量上线过低。
解决方案:
	调大文件打开数量上限即可。

3.bootstrap check failure [2] of [3]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
报错原因：
	内核参数虚拟内存映射过低。
解决方案：
	调大内核虚拟内存映射值即可。

4.bootstrap check failure [3] of [3]: max number of threads [3795] for user [oldboyedu] is too low, increase to at least [4096]
报错原因：
	程序打开的线程数量设置过低。
解决方案:
	调大程序打开的线程数量即可。

5.initial heap size [268435456] not equal to maximum heap size [1031798784]; this can cause resize pauses and prevents mlockall from locking the entire heap
报错原因：
	初始化堆内存和最大堆内存大小不一致。
解决方案:
	观察配置是否生效，建议将"-xms"和"-xmx"值配置一致。

今日作业:

• 完成课堂的所有练习并整理思维导图;
• 使用zabbix监控es服务

扩展作业:
- 使用ansible的playbook一键搭建es6和es7多实例服务