Releases: housepower/clickhouse_sinker
Releases · housepower/clickhouse_sinker
v3.2.0
发布说明
新功能
- 新增IPV4和IPV6类型的支持
- 移除多任务根据lag做assgin的复杂逻辑，需要显式打开rebalanceByLags 开关才能使用该功能
- 支持异步插入
- 支持ClickHouse集群发现功能，当ClickHouse集群发生了扩容或者缩容，sinker能动态感知到，并自动调整节点配置，无需人工修改配置重启任务；该功能为实验性质，需要通过 discovery.enabled: true开启
{ "discovery":{ "enabled": true, "checkInterval": 60 } }
优化项
- 增加kafka 高级配置，避免频繁rebalance导致不工作
{ "kafka":{ "properties":{ "heartbeat.interval.ms": 3000, "session.timeout.ms": 120000, "rebalance.timeout.ms": 120000, "request.timeout.ms": 60000 } } } - clickhouse client 默认开启lz4压缩
- 增加 recordPoolSize 用来优化内存占用
- 动态列加列时使用 aleter_sync = 0, 转为异步执行
- 如果 http-port没有指定，禁用http功能，避免安全漏洞
- 第一次运行时，随机选择副本，而不是每次都选择第一个
- 当每次发现新列时，不重启consumer，减少rebalance的次数
- 增加 ClickHouse客户端参数readTimeout配置，防止大sql运行超时
- kafka增加
max.poll.interval.ms参数，当长时间没有消费消费者组时，主动脱离该消费者组 - 所有错误码均会切换节点重试，防止报错任务卡住
缺陷修复
- 当指定http-port时，同一台机器运行多个示例会报错的问题
- 修改动态列会多次重复运行相同的sql的问题
- sinker配置多个任务，当其中一个任务初始化失败，后续任务都不能运行的问题
- 新加标签时，如果series为空，该标签不能添加到__labels__字段中的问题
Cannot convert NULL value to non-Nullable type导致任务卡住无法继续的问题
v3.1.8
What's Changed
- Bump golang.org/x/net from 0.22.0 to 0.23.0 by @dependabot in #194
Full Changelog: v3.1.6...v3.1.8
Contributors
dependabot
Assets 4
v3.1.6
What's Changed
- Allow to specify http host with config by @sashayakovtseva in #179
- Read credentials from envs or command line by @sashayakovtseva in #180
- v3.1.0 release by @YenchangChan in #184
- Bump golang.org/x/net from 0.9.0 to 0.17.0 by @dependabot in #181
- fix: typo by @testwill in #183
- feat: DynamicSchema add opt NotNullable by @exfly in #182
- ci/cd failed issue by @YenchangChan in #187
- Bump golang.org/x/crypto from 0.14.0 to 0.17.0 by @dependabot in #185
- Updated documentation and packages by @orginux in #186
- fix: systest issue by @YenchangChan in #188
- master by @YenchangChan in #190
- bump clickhouse-go to 2.21.0 by @YenchangChan in #191
- Bump google.golang.org/protobuf from 1.32.0 to 1.33.0 by @dependabot in #192
New Contributors
- @sashayakovtseva made their first contribution in #179
- @dependabot made their first contribution in #181
- @testwill made their first contribution in #183
- @exfly made their first contribution in #182
- @orginux made their first contribution in #186
Full Changelog: v3.0.6...v3.1.6
Contributors
testwill, sashayakovtseva, and 4 other contributors
Assets 4
v3.0.6
v3.0.5
Changelog
- b786986 Add a new http endpoint /debug/vars for viewing contents of seriesMap
- 05af994 Automatically end sinker if it remains inactive in last 10 mins
- 605533f chore - add test file for TestJksToPem
- 7b0e833 chore - no more input when context canceled
- 8125e26 chore - update the makefile to calculate case coverage
- 10dad19 feat - two new properties to manage reloading of bmseriesMap
v3.0.3
v3.0.2
Changelog
- 55c6db0 Combine nacos log into sinker log
- 180029f calculate topic lags every 10 seconds
- 685f94c fix clickhouse acquire conn timeout issue
- 45afaf3 make lint happy
- 57d39c4 reload bmseries every day to offload out-of-date record
- f4eb186 update dmseries quota map when applying new config, retry query when facing ErrAcquireConnTimeout error
- 595df6d update the systest script to cover the blacklist&whitelist
v3.0.1
Changelog
- 187fea3 Create containers with initial size
- 4751006 Exit immediately when receiving one more termination signal
- f15f2cf Fix changing TimeUnit config property does not trigger config reload
- 6dc62be Fix sinker crash because of illegal TimeZone issue & Decimal parsing error
- 2b7874d Ignore the SIGHUP signal
- 2af3853 Refine the error msg
- d41eef1 feat - Limit the fetch size and poll size based on the BufferSize property
- 07ec921 fix - Metrics from GoCollector and ProcessCollector are now being pushed to metric-push-gateway-addrs
- 5837a98 remove the rowpool & rowspool, reduce the GLBWritingPool size to lower memory pressure
v3.0.0
Changelog
- e34be13 Add support of ingesting multi-value metrics
- 74821f1 Allow self-defined series table name
- dfc271d Build up the Consumers map when normalizing the Config
- faad076 Delete Quota when stopping corresponding tasks
- 6a52833 Fix - invalid autocommit options specified when a group was not specified
- 0a4add9 Group the tasks by consumerGroup property to reduce number of kafka client
- a8befbc Offset shift should only be calculated once per task
- ac3ce76 Update the doc and allow customization of DatabaseName in task level
- 379f657 calculate the shard number based on message offset when shardingKey was not specified
- b452c6d format hjson files
- d15caf7 restart the consumergroup when facing offset commit error