Netdata/MongoDB/API
From charlesreid1
The Netdata url schema exposes all metrics being measured by Netdata as a JSON-exportable REST url.
Notes
Querying Netdata API with Python
To obtain the data that Netdata is reading, then, is a simple matter of making a URL request and translating the result into JSON. This is a breeze with the Python 3 requests library:
import requests, json
my_url = 'http://10.6.0.1:19999/api/v1/allmetrics?format=json&help=yes'
r = requests.get(url=my_url)
# dump resulting json
with open('output.json','w') as f:
json.dump( r.json(), f, indent=4 )
# print resulting json
print(r.json())
This displays a huge dictionary full of key-value pairs - all the quantities netdata is monitoring.
At this point, the data can be inserted into the database, or it can be parsed to extract particular quantities of interest. Each key has a timestamp associated with it, in Unix epoch format (e.g., 1518321718).
$ head -n30 output.json
{
"ipv4.tcpofo": {
"name": "ipv4.tcpofo",
"context": "ipv4.tcpofo",
"units": "packets/s",
"last_updated": 1518321718,
"dimensions": {
"TCPOFOQueue": {
"name": "inqueue",
"value": 0.0
},
"TCPOFODrop": {
"name": "dropped",
"value": 0.0
},
"TCPOFOMerge": {
"name": "merged",
"value": 0.0
},
"OfoPruned": {
"name": "pruned",
"value": 0.0
}
}
},
"cgroup_happy_mongo.merged_ops": {
"name": "cgroup_happy_mongo.merged_ops",
"context": "cgroup.merged_ops",
"units": "operations/s",
"last_updated": 1518321718,
Parsing Netdata Output
Example Netdata Output
The result of a single API call to Netdata is an extremely large JSON full of measurements. There are 248 keys, each containing dictionaries of their own. Here are the key names:
In [1]: import requests, json
In [2]: my_url = 'http://10.6.0.1:19999/api/v1/allmetrics?format=json&help=yes'
...:
In [3]: r = requests.get(url=my_url)
In [4]: with open('output.json','w') as f:
...: json.dump( r.json(), f, indent=4 )
...:
In [5]: d = r.json()
In [6]: print(len(d.keys()))
248
In [7]: print(d.keys())
dict_keys(['ipv4.tcpofo', 'cgroup_happy_mongo.merged_ops', 'cgroup_mex.merged_ops', 'cgroup_mex.throttle_serviced_ops', 'cgroup_mex.throttle_io', 'cgroup_mex.net_packets_eth0', 'cgroup_mex.serviced_ops', 'cgroup_mex.net_eth0', 'cgroup_mex.io', 'cgroup_mex.mem_usage', 'cgroup_mex.pgfaults', 'cgroup_mex.mem_activity', 'cgroup_mex.writeback', 'cgroup_mex.mem', 'cgroup_mex.cpu_per_core', 'cgroup_mex.cpu', 'cgroup_happy_mongo.queued_ops', 'cgroup_happy_mongo.throttle_serviced_ops', 'cgroup_happy_mongo.throttle_io', 'cgroup_happy_mongo.serviced_ops', 'cgroup_happy_mongo.io', 'cgroup_happy_mongo.mem_usage', 'cgroup_happy_mongo.pgfaults', 'cgroup_happy_mongo.mem_activity', 'cgroup_happy_mongo.net_packets_eth0', 'cgroup_happy_mongo.writeback', 'cgroup_happy_mongo.net_eth0', 'cgroup_happy_mongo.mem', 'cgroup_happy_mongo.cpu_per_core', 'cgroup_happy_mongo.cpu', 'ipv4.sockstat_tcp_mem', 'net_packets.docker0', 'net.docker0', 'sensors.coretemp-isa-0000_temperature', 'cpu.cpu1_cpuidle', 'cpu.cpu0_cpuidle', 'cpu.cpufreq', 'netdata.runtime_sensors', 'netdata.runtime_cpuidle', 'netdata.runtime_cpufreq', 'disk_svctm.dm-1', 'disk_avgsz.dm-1', 'disk_await.dm-1', 'disk_svctm.dm-0', 'disk_avgsz.dm-0', 'disk_await.dm-0', 'disk_svctm.sda', 'disk_avgsz.sda', 'disk_await.sda', 'groups.pipes', 'groups.sockets', 'groups.files', 'netdata.compression_ratio', 'netdata.response_time', 'groups.lwrites', 'groups.lreads', 'netdata.net', 'groups.pwrites', 'netdata.requests', 'netdata.clients', 'netdata.server_cpu', 'netdata.plugin_proc_cpu', 'groups.preads', 'groups.minor_faults', 'netdata.plugin_proc_modules', 'system.ipc_semaphore_arrays', 'system.ipc_semaphores', 'groups.major_faults', 'system.io', 'disk_iotime.dm-1', 'groups.cpu_system', 'disk_util.dm-1', 'groups.cpu_user', 'disk_backlog.dm-1', 'disk_ops.dm-1', 'disk.dm-1', 'disk_iotime.dm-0', 'groups.processes', 'disk_util.dm-0', 'disk_backlog.dm-0', 'groups.threads', 'disk_qops.dm-0', 'disk_ops.dm-0', 'disk.dm-0', 'disk_iotime.sda', 'groups.vmem', 'disk_mops.sda', 'disk_util.sda', 'groups.mem', 'disk_backlog.sda', 'groups.cpu', 'disk_qops.sda', 'disk_ops.sda', 'disk.sda', 'netfilter.conntrack_sockets', 'cpu.cpu1_softnet_stat', 'users.pipes', 'cpu.cpu0_softnet_stat', 'system.softnet_stat', 'users.sockets', 'ipv6.ect', 'users.files', 'ipv6.icmptypes', 'ipv6.icmpmldv2', 'ipv6.icmpneighbor', 'ipv6.icmprouter', 'users.lwrites', 'users.lreads', 'ipv6.icmperrors', 'ipv6.icmp', 'users.pwrites', 'ipv6.mcastpkts', 'ipv6.mcast', 'users.preads', 'users.minor_faults', 'users.major_faults', 'ipv6.udperrors', 'ipv6.udppackets', 'users.cpu_system', 'ipv6.packets', 'system.ipv6', 'users.cpu_user', 'ipv4.udplite_errors', 'ipv4.udplite', 'users.processes', 'users.threads', 'ipv4.udperrors', 'users.vmem', 'ipv4.udppackets', 'ipv4.tcphandshake', 'users.mem', 'ipv4.tcpopens', 'users.cpu', 'ipv4.tcperrors', 'ipv4.tcppackets', 'ipv4.tcpsock', 'apps.pipes', 'apps.sockets', 'ipv4.icmpmsg', 'ipv4.icmp_errors', 'ipv4.icmp', 'ipv4.errors', 'ipv4.fragsin', 'ipv4.fragsout', 'apps.files', 'ipv4.packets', 'ipv4.ecnpkts', 'ipv4.bcastpkts', 'ipv4.mcastpkts', 'apps.lwrites', 'ipv4.bcast', 'ipv4.mcast', 'system.ipv4', 'ipv6.sockstat6_raw_sockets', 'ipv6.sockstat6_udp_sockets', 'ipv6.sockstat6_tcp_sockets', 'ipv4.sockstat_udp_mem', 'ipv4.sockstat_udp_sockets', 'ipv4.sockstat_tcp_sockets', 'apps.lreads', 'ipv4.sockstat_sockets', 'system.net', 'net_packets.wlx7cdd906c3ef0', 'net.wlx7cdd906c3ef0', 'net_packets.master', 'net.master', 'mem.slab', 'mem.kernel', 'apps.pwrites', 'mem.writeback', 'mem.committed', 'system.swap', 'apps.preads', 'mem.available', 'system.ram', 'mem.pgfaults', 'system.pgpgio', 'apps.minor_faults', 'cpu.cpu1_softirqs', 'cpu.cpu0_softirqs', 'apps.major_faults', 'system.softirqs', 'apps.cpu_system', 'cpu.cpu1_interrupts', 'apps.cpu_user', 'apps.processes', 'cpu.cpu0_interrupts', 'apps.threads', 'services.merged_io_ops_write', 'services.merged_io_ops_read', 'services.queued_io_ops_write', 'services.queued_io_ops_read', 'services.throttle_io_ops_write', 'services.throttle_io_ops_read', 'services.throttle_io_write', 'services.throttle_io_read', 'services.io_ops_write', 'services.io_ops_read', 'services.io_write', 'services.io_read', 'services.mem_usage', 'services.cpu', 'netdata.plugin_diskspace_dt', 'netdata.plugin_diskspace', 'system.interrupts', 'apps.vmem', 'system.entropy', 'disk_inodes._boot', 'system.active_processes', 'disk_space._boot', 'system.load', 'disk_inodes._run_lock', 'system.uptime', 'disk_space._run_lock', 'apps.mem', 'cpu.core_throttling', 'disk_inodes._dev_shm', 'system.processes', 'system.forks', 'system.ctxt', 'disk_space._dev_shm', 'system.intr', 'netdata.private_charts', 'disk_inodes._', 'apps.cpu', 'netdata.tcp_connected', 'disk_space._', 'netdata.apps_children_fix', 'cpu.cpu1', 'netdata.tcp_connects', 'disk_inodes._run', 'netdata.plugin_tc_time', 'netdata.apps_fix', 'netdata.statsd_packets', 'disk_space._run', 'cpu.cpu0', 'netdata.plugin_tc_cpu', 'netdata.statsd_bytes', 'netdata.apps_sizes', 'disk_inodes._dev', 'netdata.statsd_reads', 'netdata.apps_cpu', 'disk_space._dev', 'system.cpu', 'netdata.plugin_cgroups_cpu', 'netdata.statsd_events', 'netdata.statsd_metrics', 'system.idlejitter'])
Netdata Storage Schema
One option to store all of this data is to create a single table to hold all of the data obtained from Netdata. This would look like a table with 248 columns, with each row of the table corresponding to a single request that went to Netdata for which data was returned, and each cell of the table holds a large dictionary.
An alternative approach is to use the keys to create tables. Each key creates a new table, with the nested dictionary creating a table.
In [8]: d['cpu.cpu1']
Out[8]:
{'context': 'cpu.cpu',
'dimensions': {'guest': {'name': 'guest', 'value': 0.0},
'guest_nice': {'name': 'guest_nice', 'value': 0.0},
'idle': {'name': 'idle', 'value': 98.989899},
'iowait': {'name': 'iowait', 'value': 0.0},
'irq': {'name': 'irq', 'value': 0.0},
'nice': {'name': 'nice', 'value': 0.0},
'softirq': {'name': 'softirq', 'value': 0.0},
'steal': {'name': 'steal', 'value': 0.0},
'system': {'name': 'system', 'value': 1.010101},
'user': {'name': 'user', 'value': 0.0}},
'last_updated': 1518323931,
'name': 'cpu.cpu1',
'units': 'percentage'}
In [9]: d['ipv4.sockstat_tcp_sockets']
Out[9]:
{'context': 'ipv4.sockstat_tcp_sockets',
'dimensions': {'alloc': {'name': 'alloc', 'value': 26.0},
'inuse': {'name': 'inuse', 'value': 11.0},
'orphan': {'name': 'orphan', 'value': 0.0},
'timewait': {'name': 'timewait', 'value': 0.0}},
'last_updated': 1518323931,
'name': 'ipv4.sockstat_tcp_sockets',
'units': 'sockets'}
Recipe
The recipe to implement a table for each key is as follows:
create mongodb collection named netstat
for key in netstat_data_dict.keys:
d = netstat_data_dict[key]
create collection table with name key
insert d into table