learn - 项目架构演变-02（日志采集篇）
日志收集也是重要的一部分，虽然不紧急，但要有。
learn - 项目架构演变-02（日志采集篇）
创作背景
我负责的项目中，有很多是从0到1，也有很多是大型项目。都会有对日志的集中采集。
为什么要集中日志？1.不需要每台机器去观察；2.将来服务横向扩容依然能定位线上情况；3也为了从物理机到云端部署过度；4若是有宕机未恢复情况，仍然会有99%的日志供参考。
另外一点，本篇主要是以0到1项目，在小型公司里面节省成本情况下来过渡使用的。
容器搭建步骤E+K
elasticsearch (verion=8.7.0)
1
2
3
4
5
6
7
8
9
10
11
12
docker run -d \
--name ek-elasticsearch \
-p 9200:9200 \
-p 9300:9300 \
-e "discovery.type=single-node" \
-e "ES_JAVA_OPTS=-Xms84m -Xmx256m" \
-e "ELASTIC_PASSWORD=abcdefg" \
-e "bootstrap.password=abcdefg" \
-v /opt/elk-data/es/data:/usr/share/elasticsearch/data \
-v /opt/elk-data/es/plugins:/usr/share/elasticsearch/plugins \
--memory 2g \
docker.elastic.co/elasticsearch/elasticsearch:8.7.0
创建一个非elastic账户：kibana_admin 密码：abcdefg。
1
2
3
4
5
#添加个角色可以管理nginx日志
curl -X PUT -u "elastic:wfy2025@XY" "http://localhost:9200/_security/role/kibana_nginx" -k -H "Content-Type: application/json" -d'{"cluster":["all"],"indices":[{"names":["nginx*"],"privileges":["all"]}]}'
# 添加个账号或者修改
curl -X POST -u "elastic:wfy2025@XY" "http://localhost:9200/_security/user/kibana_admin" -k -H "Content-Type: application/json" -d'{"password":"abcdefg","roles":["kibana_admin","kibana_system","kibana_nginx"]}'
kibana
/opt/elk-data/kibana/config/kibana.yml配置账户和密码
1
2
3
4
5
6
7
8
9
10
11
#
# ** THIS IS AN AUTO-GENERATED FILE **
#
# Default Kibana configuration for docker target
server.host: "0.0.0.0"
server.shutdownTimeout: "5s"
elasticsearch.hosts: [ "http://wfy.360club.net:9200" ]
monitoring.ui.container.elasticsearch.enabled: true
elasticsearch.username: "kibana_admin"
elasticsearch.password: "abcdfg"
进行映射本地文件容器
1
2
3
4
5
6
docker run -d \
--name ek-kibana \
-p 5601:5601 \
-v /opt/elk-data/kibana/config/kibana.yml:/usr/share/kibana/config/kibana.yml \
-e "ELASTICSEARCH_HOSTS=http://192.168.99.215:9200" \
docker.elastic.co/kibana/kibana:8.7.0
配置被采集的机器
nginx
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
http {
include mime.types;
default_type application/octet-stream;
map $http_userid $userid_value {
default "0";
'~^(.*)$' $1;
}
map $upstream_status $up_status_value {
default "0";
'~^(.*)$' $1;
}
map $upstream_response_time $up_response_time_value {
default "0";
'~^(.*)$' $1;
}
map $upstream_header_time $up_header_time_value {
default "0";
'~^(.*)$' $1;
}
map $upstream_connect_time $up_connect_time_value {
default "0";
'~^(.*)$' $1;
}
log_format log_json escape=json
'{'
'"@timestamp":"$time_iso8601",'
'"host":"$host",'
'"http_host":"$http_host",'
'"server_name":"$server_name",'
'"server_ip":"$server_addr",'
## '"client_ip":"$remote_addr",'
'"client_ip":"$proxy_add_x_forwarded_for",'
'"x_forwarded_for":"$http_x_forwarded_for",'
'"user_id":"$http_userid",'
'"request_method":"$request_method",'
'"request_uri":"$request_uri",'
'"request_length":"$request_length",'
'"request_time":$request_time,'
'"status":$status,'
'"referrer":"$http_referer",'
'"user_agent":"$http_user_agent",'
'"upstream_addr":"$upstream_addr",'
'"upstream_status":$up_status_value,'
'"upstream_response_time":$up_response_time_value,'
'"upstream_connect_time":"$up_connect_time_value",'
'"upstream_header_time":"$up_header_time_value"'
'}';
##.....
server {
access_log /usr/local/openresty/nginx/logs/blog.360club.net.access.json.log log_json;
.....
filebeat
1
2
3
4
wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.7.0-linux-x86_64.tar.gz
tar -xvf filebeat-8.7.0-linux-x86_64.tar.gz
mv filebeat-8.7.0-linux-x86_64 /usr/local/filebeat-8.7.0
编写一个/usr/local/filebeat-8.7.0/nginx.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
filebeat.inputs:
- type: filestream
id: my-nginx-id
enabled: true
paths:
- /usr/local/openresty/nginx/logs/*.json.log
parsers:
- ndjson:
overwrite_keys : true
tags:
- "nginx"
start_position: beginning
close_inactive : 24h
ignore_older : 25h
close_timeout : 25h
clean_inactive : 72h
output.elasticsearch:
hosts: ["wfy.360club.net:9200"]
username: "elastic"
password: "abcdefg"
indices:
- index: "nginx-prod-%{+YYYY.MM.dd}"
when.contains:
tags: "nginx"
setup.template.enabled: false
setup.template.name: "nginx-prod"
setup.template.pattern: "nginx-prod-*"
setup.ilm.enable: false
启动filebeat脚本
1
2
3
4
5
#!/bin/sh
#/usr/local/filebeat-8.7.0/filebeat -e -c /usr/local/filebeat-8.7.0/nginx.yml
cd /opt/elk-data/
nohup /usr/local/filebeat-8.7.0/filebeat -c /usr/local/filebeat-8.7.0/nginx.yml 2>&1 & disown
使用
1.记得要对索引定期删除数据。
logrotate
logrotate是一个用于管理日志文件的工具，主要功能包括日志轮转、压缩、删除和发送等，旨在节省磁盘空间并方便日志管理。
每天对nginx的json日志文件进行分割。
more /etc/logrotate.d/nginx
1
2
3
4
5
6
7
8
9
10
11
12
13
14
/usr/local/openresty/nginx/logs/*.log {
daily
missingok
rotate 2
notifempty
dateext
dateformat -%Y%m%d
sharedscripts
postrotate
if [ -f /var/run/nginx.pid ]; then
kill -USR1 `cat /var/run/nginx.pid`
fi
endscript
}
每天会执行，并且按照要求去执行script
1
2
3
~]# ll /usr/local/openresty/nginx/logs/*.log*
-rw-r--r-- 1 nobody root 79279 May 7 02:20 wangfuyu.access.log
-rw-r--r-- 1 nobody root 79279 May 6 02:20 wangfuyu.access.log-20250506
定期工具
若让filebeat自动创建es索引map，将会首先判断首次的数据格式，会把request_time 设置成string 或者long，然而我们想要的是float,所以我们要定期创建新索引，并且定期删除旧索引。
Map 字段
cat /opt/elk-data/nginx-index.map.json
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
{
"settings":{
"index":{
"number_of_shards":1,
"number_of_replicas":0
}
},
"mappings": {
"properties": {
"@timestamp": {
"type": "date"
},
"agent": {
"properties": {
"ephemeral_id": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"id": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"type": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"version": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},
"client_ip": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"ecs": {
"properties": {
"version": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},
"host": {
"properties": {
"name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},
"http_host": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"input": {
"properties": {
"type": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},
"log": {
"properties": {
"file": {
"properties": {
"path": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},
"offset": {
"type": "long"
}
}
},
"referrer": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"request_length": {
"type": "long"
},
"request_method": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"request_time": {
"type": "float"
},
"request_uri": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"server_ip": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"server_name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"status": {
"type": "long"
},
"tags": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"upstream_addr": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"upstream_connect_time": {
"type": "float"
},
"upstream_header_time": {
"type": "float"
},
"upstream_response_time": {
"type": "float"
},
"upstream_status": {
"type": "long"
},
"user_agent": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"user_id": {
"type": "long",
"fields": {
"keyword": {
"type": "keyword"
}
}
},
"x_forwarded_for": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}
shell
日志文件切换索引，下次再单独整理一个es专栏。
1
2
3
4
5
6
7
8
9
10
11
12
#把数据切到另一个索引上
curl -u "elastic:wfy2025@XY" -H "Content-Type: application/json" -XPOST "http://localhost:9200/_reindex" -d'{"source":{"index":"nginx-test-2025.05.12"},"dest":{"index":"nginx-test-2025.05.12.2"}}'
#删除索引
curl -u "elastic:wfy2025@XY" -H "Content-Type: application/json" -XDELETE "http://localhost:9200/nginx-test-2025.05.12"
#添加别名
curl -u "elastic:wfy2025@XY" -H "Content-Type: application/json" -XPOST "http://localhost:9200/_aliases" -d'{"actions":[{"add":{"index":"nginx-test-2025.05.12.2","alias":"nginx-test-2025.05.12"}}]}'
本文由作者按照 CC BY 4.0 进行授权