分类目录归档:statsd/graphite

graphite_relay sharding

当statsd发送超量的metrics到graphite中,graphite单节点无法负载的情况,可以使用consistent-hashing的模式来将数据分片到backend中。同样的consistent-hashing模式下可以自动剔除/加入节点。

官方文档:

carbon-relay.py serves two distinct purposes: replication and sharding.

When running with RELAY_METHOD = rules, a carbon-relay.py instance can run in place of a carbon-cache.py server and relay all incoming metrics to multiple backend carbon-cache.py‘s running on different ports or hosts.

In RELAY_METHOD = consistent-hashing mode, a DESTINATIONS setting defines a sharding strategy across multiple carbon-cache.py backends. The same consistent hashing list can be provided to the graphite webapp via CARBONLINK_HOSTS to spread reads across the multiple backends.

本例模拟一个双后端的carbon-cache instance.单机器运行,使用carbon-cache.py 的instance功能

config文件:
#####carbon.conf
[cache:a]
LINE_RECEIVER_PORT = 2203
PICKLE_RECEIVER_PORT = 2204
CACHE_QUERY_PORT = 7102
LOCAL_DATA_DIR = /opt/graphite/storage/whisper_a/
[cache:b]
LINE_RECEIVER_PORT = 2205
PICKLE_RECEIVER_PORT = 2206
CACHE_QUERY_PORT = 7202
LOCAL_DATA_DIR = /opt/graphite/storage/whisper_b/
[relay]
LINE_RECEIVER_INTERFACE = 0.0.0.0
LINE_RECEIVER_PORT = 2003
PICKLE_RECEIVER_INTERFACE = 0.0.0.0
PICKLE_RECEIVER_PORT = 2004
DESTINATIONS = 127.0.0.1:2204:a, 127.0.0.1:2206:b 
启动服务:

启动instance:a
xmx@xiemx-test:/opt/graphisudo ./carbon-cache.py --instance=a --config=/opt/graphite/conf/carbon.conf start
Starting carbon-cache (instance a)

启动instance:b
xmx@xiemx-test:/opt/graphite/bin$ sudo ./carbon-cache.py --instance=b --config=/opt/graphite/conf/carbon.conf start
Starting carbon-cache (instance b)

启动relay:
xmx@xiemx-test:/opt/graphite/conf$ sudo /opt/graphite/bin/carbon-relay.py --debug --config=/opt/graphite/conf/carbon.conf start
Starting carbon-relay (instance a)
08/09/2017 16:02:16 :: [console] Using sorted write strategy for cache
08/09/2017 16:02:16 :: [clients] connecting to carbon daemon at 127.0.0.1:2204:a
08/09/2017 16:02:16 :: [clients] connecting to carbon daemon at 127.0.0.1:2206:b
08/09/2017 16:02:16 :: [console] twistd 16.4.1 (/usr/bin/python 2.7.6) starting up.
08/09/2017 16:02:16 :: [console] reactor class: twisted.internet.epollreactor.EPollReactor.
08/09/2017 16:02:16 :: [console] Starting factory CarbonClientFactory(127.0.0.1:2206:b)
08/09/2017 16:02:16 :: [clients] CarbonClientFactory(127.0.0.1:2206:b)::startedConnecting (127.0.0.1:2206)
08/09/2017 16:02:16 :: [console] Starting factory CarbonClientFactory(127.0.0.1:2204:a)
08/09/2017 16:02:16 :: [clients] CarbonClientFactory(127.0.0.1:2204:a)::startedConnecting (127.0.0.1:2204)
08/09/2017 16:02:16 :: [console] CarbonReceiverFactory starting on 2003
08/09/2017 16:02:16 :: [console] Starting factory 
08/09/2017 16:02:16 :: [console] CarbonReceiverFactory starting on 2004
08/09/2017 16:02:16 :: [console] Starting factory 
08/09/2017 16:02:16 :: [clients] CarbonClientProtocol(127.0.0.1:2206:b)::connectionMade
08/09/2017 16:02:16 :: [clients] CarbonClientFactory(127.0.0.1:2206:b)::connectionMade (CarbonClientProtocol(127.0.0.1:2206:b))
08/09/2017 16:02:16 :: [clients] Destination is up: 127.0.0.1:2206:b
08/09/2017 16:02:16 :: [clients] CarbonClientProtocol(127.0.0.1:2204:a)::connectionMade
08/09/2017 16:02:16 :: [clients] CarbonClientFactory(127.0.0.1:2204:a)::connectionMade (CarbonClientProtocol(127.0.0.1:2204:a))
08/09/2017 16:02:16 :: [clients] Destination is up: 127.0.0.1:2204:a

测试:

模拟5个客户端同时发送100个key

模拟node掉线重连

graphite-web数据聚合展示

修改local_settings.py
CARBONLINK_HOSTS = ["127.0.0.1:7102:a", "127.0.0.1:7202:b"]

启动django
sudo PYTHONPATH=/opt/graphite/webapp django-admin.py runserver 0.0.0.0:5000 --settings=graphite.settings

statsd cluster proxy

通过UDP proxy程序将前端的数据通过一定的hash算法将相同的metric发送的固定的后aggregation数据。proxy代理支持健康检测自动剔除/加入后端statsd。

1.配置
本例展示一个3节点的statsd后端且将聚合数据发送到standout方便观测。

node1
######config文件:
{
 port: 8127
, mgmt_port: 8227
, backends: [ "./backends/console" ]
}

启动:
xiemx➜  statsd : master ✘ :✹✭ ᐅ  node stats.js config8127.js
8 Sep 11:06:36 - [81604] reading config file: config8127.js
8 Sep 11:06:36 - server is up INFO
 
node2
######config文件:
{
 port: 8128
, mgmt_port: 8228
, backends: [ "./backends/console" ]
}

启动:
xiemx➜  statsd : master ✘ :✹✭ ᐅ  node stats.js config8128.js
8 Sep 11:07:09 - [81665] reading config file: config8128.js
8 Sep 11:07:09 - server is up INFO
 
node3
######config文件:
{
 port: 8129
, mgmt_port: 8229
, backends: [ "./backends/console" ]
}

启动:
xiemx➜  statsd : master ✘ :✹✭ ᐅ  node stats.js config8129.js
8 Sep 11:07:45 - [81723] reading config file: config8129.js
8 Sep 11:07:45 - server is up INFO
proxy
######config文件:
{
nodes: [
{host: '127.0.0.1', port: 8127, adminport: 8227},
{host: '127.0.0.1', port: 8128, adminport: 8228},
{host: '127.0.0.1', port: 8129, adminport: 8229}
],
server: './servers/udp',
host:  '0.0.0.0',
port: 8125,
mgmt_port: 8126,
forkCount: 0,
checkInterval: 1000,
cacheSize: 10000,
deleteIdleStats: true
}

启动:
xiemx➜  statsd : master ✘ :✹✭ ᐅ  node proxy.js proxyconfig.js
8 Sep 11:09:02 - [81938] reading config file: proxyconfig.js
8 Sep 11:09:02 - INFO: [81938] server is up

测试:
5个线程同时推送500个metric到代理查看分片情况

测试命令:
for i in $(seq 1 500);do echo "Ezbuy-$i:1|c" | nc -u -w0 127.0.0.1 8125;done

节点自动剔除和加入: