本系列为作者学习ES官网文档时做的笔记
Elasticsearch 是 Elastic Stack 核心的分布式搜索和分析引擎
向 Elasticsearch 发送请求
REST API :
- 各种语言客户端或者curl等HTTP工具
- Kibana控制台：菜单=》Dev Tools =》Console
添加数据
从数据生成称为索引文档的 JSON 对象添加到 Elasticsearch
添加单个文档
POST _doc：将单个日志条目生成索引文档并添加到 logs-my_app-default数据流，数据流不存在会自动创建
1
2
3
4
5
6
7
|
POST logs-my_app-default/_doc
{
"@timestamp": "2099-05-06T16:21:15.000Z",
"event": {
"original": "192.0.2.42 - - [06/May/2099:16:21:15 +0000] \"GET /images/bg.jpg HTTP/1.0\" 200 24736"
}
}
|
响应包括 Elasticsearch 为文档生成的元数据：
- _index对应文档索引名称
- _id为该索引文档的唯一标志
1
2
3
4
5
6
7
8
9
10
11
12
13
14
|
{
"_index" : ".ds-logs-my_app-default-2022.12.17-000001",
"_type" : "_doc",
"_id" : "gBSIHoUBm1dGIH3uyvpV",
"_version" : 1,
"result" : "created",
"_shards" : {
"total" : 2,
"successful" : 1,
"failed" : 0
},
"_seq_no" : 0,
"_primary_term" : 1
}
|
添加多个文档
POST _bulk：在一个请求中添加多个文档。批量数据必须是换行符分隔的 JSON (NDJSON)。每行必须以换行符 ( \n) 结尾，包括最后一行。
1
2
3
4
5
|
PUT logs-my_app-default/_bulk
{ "create": { } }
{ "@timestamp": "2099-05-07T16:24:32.000Z", "event": { "original": "192.0.2.242 - - [07/May/2020:16:24:32 -0500] \"GET /images/hm_nbg.jpg HTTP/1.0\" 304 0" } }
{ "create": { } }
{ "@timestamp": "2099-05-08T16:25:42.000Z", "event": { "original": "192.0.2.255 - - [08/May/2099:16:25:42 +0000] \"GET /favicon.ico HTTP/1.0\" 200 3638" } }
|
搜索数据
GET _search：进行搜索，body配置搜索选项
query:搜索过滤
1
2
3
4
5
6
7
|
GET logs-my_app-default/_search
{
"query": {
"match_all": { }
},
...
}
|
- query._source:是否包含原始json
- query.fields:选择字段，响应结果中hits.hits会包含一个field
请求
1
2
3
4
5
6
7
8
9
|
GET logs-my_app-default/_search
{
...
"fields": [
"@timestamp"
],
"_source": false,
...
}
|
响应
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
|
{
...
"hits" : {
"total" : {
"value" : 3,
"relation" : "eq"
},
"max_score" : null,
"hits" : [
{
...
"fields" : {
"@timestamp" : [
"2099-05-08T16:25:42.000Z"
]
},
...
},
]
}
}
|
- query.range:特定时间或 IP 范围使用range查询，可以使用日期和数字来定义相对时间范围
1
2
3
4
5
6
7
8
9
10
11
|
GET logs-my_app-default/_search
{
"query": {
"range": {
"@timestamp": {
"gte": "2099-05-05",
"lt": "2099-05-08"
}
}
},
...
|
1
2
3
4
5
6
7
8
9
10
11
12
|
GET logs-my_app-default/_search
{
"query": {
"range": {
"@timestamp": {
"gte": "now-1d/d",
"lt": "now/d"
}
}
},
...
}
|
- query.bool:组合多个查询，通常使用filter
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
|
GET logs-my_app-default/_search
{
...
"query": {
"bool": {
"filter": [
{
"range": {
"@timestamp": {
"gte": "2099-05-05",
"lt": "2099-05-08"
}
}
},
{
"range": {
"source.ip": {
"gte": "192.0.2.0",
"lte": "192.0.2.240"
}
}
}
]
}
},
...
|
sort:排序
1
2
3
4
5
6
7
8
9
|
GET logs-my_app-default/_search
{
...
"sort": [
{
"@timestamp": "desc"
}
]
}
|
响应
- hits.hits：匹配的文档，_source为原始json对象，默认最多为10个文档
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
|
{
...
"hits" : {
"total" : {
"value" : 3,
"relation" : "eq"
},
"max_score" : null,
"hits" : [
{
"_index" : ".ds-logs-my_app-default-2022.12.17-000001",
"_type" : "_doc",
"_id" : "hRSRHoUBm1dGIH3uQPoI",
"_score" : null,
"_source" : {
"@timestamp" : "2099-05-08T16:25:42.000Z",
"event" : {
"original" : """192.0.2.255 - - [08/May/2099:16:25:42 +0000] "GET /favicon.ico HTTP/1.0" 200 3638"""
}
},
"sort" : [
4081940742000
]
},
...
]
}
}
|
runtime_mappings:从非结构化内容中提取字段
见官网
- runtime_mappings.xxx.script:提取脚本，支持groovy，js，python等
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
|
GET logs-my_app-default/_search
{
"runtime_mappings": {
"source.ip": {
"type": "ip",
"script": """
String sourceip=grok('%{IPORHOST:sourceip} .*').extract(doc[ "event.original" ].value)?.sourceip;
if (sourceip != null) emit(sourceip);
"""
}
},
...
"fields": [
"@timestamp",
"source.ip"
],
...
}
|
aggs:聚合数据
- aggs.xxx:将数据汇总为指标、统计数据或其他分析，比如对匹配结果的某个字段求平均值avg等
1
2
3
4
5
6
7
8
9
10
11
12
|
GET logs-my_app-default/_search
{
...
"aggs": {
"average_response_size":{
"avg": {
"field": "http.response.body.bytes"
}
}
},
...
}
|
响应中的aggregations记录结果
1
2
3
4
5
6
7
8
|
{
...
"aggregations" : {
"average_response_size" : {
"value" : 12368.0
}
}
}
|
探索更多搜索选项
常用搜索选项
清理
删除测试数据流及其支持索引
1
|
DELETE _data_stream/logs-my_app-default
|
kibana设置中文
在kibana配置文件etc/kibana.yaml中加一个