ServiceInput示例插件
简介
service_http_server
input
插件可以接收来自unix socket、http/https、tcp的请求,并支持sls协议、otlp等多种协议。
版本
配置参数
参数 | 类型 | 是否必选 | 说明 |
---|---|---|---|
Type | String | 是 | 插件类型,固定为service_http_server |
Format | String | 否 | 数据格式。 支持格式:`sls`、`prometheus`、`influxdb`、`otlp_logv1`、 `otlp_metricv1`、`pyroscope`、`statsd` v2版本支持格式:`raw`、`prometheus`、`otlp_logv1`、`otlp_metricv1`、`otlp_tracev1` 说明:`raw`格式以原始请求字节流传输数据 |
Address | String | 否 | 监听地址。 |
Path | String | 否 | 接收端点, 如Format 为 `otlp_logv1` 时, 默认端点为`/v1/logs` 。 |
ReadTimeoutSec | String | 否 | 读取超时时间。 默认取值为:`10s`。 |
ShutdownTimeoutSec | String | 否 | 关闭超时时间。 默认取值为:`5s`。 |
MaxBodySize | String | 否 | 最大传输 body 大小。 默认取值为:`64k`。 |
UnlinkUnixSock | String | 否 | 启动前如果监听地址为unix socket,是否进行强制释放。 默认取值为:`true`。 |
FieldsExtend | Boolean | 否 | 是否支持非integer以外的数据类型(如String) 目前仅针对有 String、Bool 等额外类型的 influxdb Format 有效 |
QueryParams | []String | 否 | 需要解析到Group.Metadata中的请求参数。 解析结果会以KeyValue放入Metadata。默认取值为`[]`,即不解析。 仅v2版本有效 |
QueryParamPrefix | String | 否 | 解析请求参数时需要添加的key前缀,如_query_param_ 。前缀会直接拼接在每个QueryParam前,无额外连接符,默认取值为空,即不增加前缀。 仅v2版本有效 |
HeaderParams | []String | 否 | 需要解析到Group.Metadata中的header参数。 解析结果会以KeyValue放入Metadata。默认取值为`[]`,即不解析。 仅v2版本有效 |
HeaderParamPrefix | String | 否 | 解析Header参数时需要添加的key前缀,如_header_param_ 。前缀会直接拼接在每个HeaderParam前,无额外连接符,默认取值为空,即不增加前缀。 仅v2版本有效 |
DisableUncompress | Boolean | 否 | 禁用对于请求数据的解压缩, 默认取值为:false 目前仅针对Raw Format有效 仅v2版本有效 |
Tags | map[String]String | 否 | 输出数据默认携带标签 仅v1版本有效 |
DumpData | Boolean | 否 | [开发使用] 将接收的请求存储于本地文件, 默认取值为:false |
DumpDataKeepFiles | Int | 否 | [开发使用] Dump文件保留文件数目, 文件按小时滚动, 此参数默认值为5, 表示保留5小时Dump 参数 |
AllowUnsafeMode | Boolean | 否 | 是否允许unsafe模式的Decode,启用该模式,Decoder将可能利用go unsafe技术来加速解码,目前仅当Format=prometheus时有效(注:暂不支持Exemplar、Histogram) |
样例
接收 OTLP 日志
- 采集配置
enable: true
inputs:
- Type: service_http_server
Format: "otlp_logv1"
Address: "http://127.0.0.1:12345"
flushers:
- Type: flusher_stdout
OnlyStdout: true
- 输入
使用 opentelemetry-java-sdk构造数据,基于ExampleConfiguration.java、OtlpExporterExample.java 进行如下代码改造。
ExampleConfiguration
进行 OTLP SDK
初始化。
static OpenTelemetrySdk initHTTPOpenTelemetry() {
// Include required service.name resource attribute on all spans and metrics
Resource resource =
Resource.getDefault()
.merge(Resource.builder().put(SERVICE_NAME, "OtlpExporterExample").build());
OpenTelemetrySdk openTelemetrySdk =
OpenTelemetrySdk.builder()
.setLogEmitterProvider(
SdkLogEmitterProvider.builder()
.setResource(resource)
.addLogProcessor(SimpleLogProcessor
.create(OtlpHttpLogExporter
.builder()
.setEndpoint("http://127.0.0.1:12345/v1/logs")
.build()))
.build())
.buildAndRegisterGlobal();
Runtime.getRuntime()
.addShutdownHook(new Thread(openTelemetrySdk.getSdkLogEmitterProvider()::shutdown));
return openTelemetrySdk;
}
OtlpExporterExample
构造数据。
OpenTelemetrySdk openTelemetry = ExampleConfiguration.initHTTPOpenTelemetry();
LogEmitter logger = openTelemetry.getSdkLogEmitterProvider().get("io.opentelemetry.example");
logger
.logRecordBuilder()
.setBody("log body1")
.setAllAttributes(
Attributes.builder()
.put("k1", "v1")
.put("k2", "v2").build())
.setSeverity(Severity.INFO)
.setSeverityText("INFO")
.setEpoch(Instant.now())
.setContext(Context.current())
.emit();
- 输出
{
"time_unix_nano": "1663913736115000000",
"severity_number": "9",
"severity_text": "INFO",
"content": "log body1",
"attributes": "{\"k1\":\"v1\",\"k2\":\"v2\"}",
"resources": "{\"service.name\":\"OtlpExporterExample\",\"telemetry.sdk.language\":\"java\",\"telemetry.sdk.name\":\"opentelemetry\",\"telemetry.sdk.version\":\"1.18.0\"}",
"__time__": "1663913736"
}
接收 OTLP Logs/Metrics/Traces (v2)
注意:目前v2 pipeline尚不支持otlp logs请求,会丢弃数据。
- 采集配置
enable: true
version: v2
inputs:
- Type: service_http_server
Format: "otlp_logv1"
Address: "http://127.0.0.1:12344"
- Type: service_http_server
Format: "otlp_metricv1"
Address: "http://127.0.0.1:12345"
- Type: service_http_server
Format: "otlp_tracev1"
Address: "http://127.0.0.1:12346"
flushers:
- Type: flusher_stdout
OnlyStdout: true
接收字节流数据
- 采集配置
enable: true
version: v2
inputs:
- Type: service_http_server
Format: "raw"
Address: "http://127.0.0.1:12345"
QueryParams:
- "QueryKey"
QueryParamPrefix: "_param_prefix_"
HeaderParams:
- "HeaderKey"
HeaderParamPrefix: "_header_prefix_"
flushers:
- Type: flusher_stdout
OnlyStdout: true
Tags: true
- 输入
手动构造请求
curl --location --request POST 'http://127.0.0.1:12345?QueryKey=queryValue' --header 'HeaderKey: headerValue' --header 'Content-Type: text/plain' --data 'test_measurement,host=server01,region=cn value=0.5'
- 输出
[Event] event 1, metadata map[_header_prefix_HeaderKey:headerValue _param_prefix_QueryKey:queryValue], tags map[__hostname__:579ce1e01dea]
{
"eventType":"byteArray",
"name":"",
"timestamp":0,
"observedTimestamp":0,
"tags":{
},
"byteArray":"test_measurement,host=server01,region=cn value=0.5"
}
接收Pyroscope Agent 数据
- Agent 兼容性说明
Agent | 协议 | 是否兼容 |
---|---|---|
pyroscopde/nodjs | pprof | 是 |
pyroscopde/.net | pprof | 是 |
pyroscopde/.net-new | pprof | 是 |
pyroscopde/java | JFR | 是 |
pyroscopde/go | pprof | 是 |
pyroscopde/php | raw tire | 是 |
pyroscopde/ebpf | raw tire | 是 |
pyroscopde/rust | raw groups | 是 |
pyroscopde/ruby | raw groups | 是 |
pyroscopde/python | raw groups | 是 |
- 采集配置 使用v1 版本表述使用protocol.Log 传递数据
enable: true
version: v1
inputs:
- Type: service_http_server
Format: "pyroscope"
Address: "http://:4040"
Endpoint: "/ingest"
Cluster: "sls-mall"
TagsInGroup: false
flushers:
- Type: flusher_stdout
OnlyStdout: true
Tags: true
输入
编译 Pyroscope Golang Example
go build -o main .
启动Pyroscope Golang Example
./main
输出
2023-02-21 14:15:20 {"name":"runtime.malg /Users/evan/sdk/go1.19.4/src/runtime/proc.go","stack":"runtime.newproc1 /Users/evan/sdk/go1.19.4/src/runtime/proc.go\nruntime.newproc.func1 /Users/evan/sdk/go1.19.4/src/runtime/proc.go\nruntime.systemstack /Users/evan/sdk/go1.19.4/src/runtime/asm_amd64.s","stackID":"afb4871c4cb30267","language":"go","type":"profile_mem","dataType":"CallStack","durationNs":"0","profileID":"e3620581-930d-4c51-8e52-cb752e75f1be","labels":"{\"__name__\":\"simple.golang.app\",\"_sample_rate_\":\"100\",\"cluster\":\"s=s-mall\"}","units":"count","valueTypes":"alloc_objects","aggTypes":"sum","val":"1260.00","__time__":"1676960120"}
2023-02-21 14:15:20 {"name":"runtime.malg /Users/evan/sdk/go1.19.4/src/runtime/proc.go","stack":"runtime.newproc1 /Users/evan/sdk/go1.19.4/src/runtime/proc.go\nruntime.newproc.func1 /Users/evan/sdk/go1.19.4/src/runtime/proc.go\nruntime.systemstack /Users/evan/sdk/go1.19.4/src/runtime/asm_amd64.s","stackID":"afb4871c4cb30267","language":"go","type":"profile_mem","dataType":"CallStack","durationNs":"0","profileID":"e3620581-930d-4c51-8e52-cb752e75f1be","labels":"{\"__name__\":\"simple.golang.app\",\"_sample_rate_\":\"100\",\"cluster\":\"s=s-mall\"}","units":"bytes","valueTypes":"alloc_space","aggTypes":"sum","val":"524496.00","__time__":"1676960120"}
2023-02-21 14:15:20 {"name":"runtime.malg /Users/evan/sdk/go1.19.4/src/runtime/proc.go","stack":"runtime.newproc1 /Users/evan/sdk/go1.19.4/src/runtime/proc.go\nruntime.newproc.func1 /Users/evan/sdk/go1.19.4/src/runtime/proc.go\nruntime.systemstack /Users/evan/sdk/go1.19.4/src/runtime/asm_amd64.s","stackID":"afb4871c4cb30267","language":"go","type":"profile_mem","dataType":"CallStack","durationNs":"0","profileID":"e3620581-930d-4c51-8e52-cb752e75f1be","labels":"{\"__name__\":\"simple.golang.app\",\"_sample_rate_\":\"100\",\"cluster\":\"s=s-mall\"}","units":"count","valueTypes":"inuse_objects","aggTypes":"sum","val":"1260.00","__time__":"1676960120"}
2023-02-21 14:15:20 {"name":"compress/flate.(*compressor).init /Users/evan/sdk/go1.19.4/src/compress/flate/deflate.go","stack":"compress/flate.NewWriter /Users/evan/sdk/go1.19.4/src/compress/flate/deflate.go\ncompress/gzip.(*Writer).Write /Users/evan/sdk/go1.19.4/src/compress/gzip/gzip.go\nruntime/pprof.(*profileBuilder).flush /Users/evan/sdk/go1.19.4/src/runtime/pprof/proto.go\nruntime/pprof.(*profileBuilder).pbSample /Users/evan/sdk/go1.19.4/src/runtime/pprof/proto.go\nruntime/pprof.(*profileBuilder).build /Users/evan/sdk/go1.19.4/src/runtime/pprof/proto.go\nruntime/pprof.profileWriter /Users/evan/sdk/go1.19.4/src/runtime/pprof/pprof.go","stackID":"d4e9448662480cdb","language":"go","type":"profile_mem","dataType":"CallStack","durationNs":"0","profileID":"e3620581-930d-4c51-8e52-cb752e75f1be","labels":"{\"__name__\":\"simple.golang.app\",\"_sample_rate_\":\"100\",\"cluster\":\"s=s-mall\"}","units":"count","valueTypes":"alloc_objects","aggTypes":"sum","val":"177.00","__time__":"1676960120"}