Monitoreando remotos

Solución versión 1.0

Que tenemos ? 

Nagios: un sistema de monitoreo

• hosts
• contacts
• alerts
• checks/plugins

Que nos da ?

monitoreo de hosts: status up/down

Qué no nos da?

• visualización de los datos
  • no tiene soporte nativo para gráficos
  • es un sistema de monitoreo, no de reporting

Arquitectura

filosofia unix

¿ Qué tenemos ?

• hosts
• datos (cambian en el tiempo)

¿ Qué queremos ?

• monitorear los hosts
• visualizar los datos 

Datos

Almacenamiento de datos que cambian en el tiempo:

Time Series Database

opciones:

• rrd
• InfluxDB
• OpenTSDB
• MySQL
• ....

Visualización

¿ Time Series Database ?

Grafana 2.0

buena integración con InfluxDB

Obtener Datos

consultas snmp

quien monitorea? nagios

solución: nagios check/plugin

que es un nagios plugin? un script

en que lenguaje se hace el script?

Nagios plugin

¿bash o python?

python tiene librerias para conectar a influxdb y usar snmp.

bash llama a comandos externos en ambos casos

python sería más "performante"

Python - uptime

[osvaldo@na3lmon1 ~]$ snmpget -v 2c -c ${COMMUNITY} -Oa  11.192.100.155 \
    $oid_nombre $oid_NetworkName $oid_RSSI $oid_SNR $oid_uptime
SNMPv2-SMI::enterprises.4130.5.2.9.0 = STRING: "CL-1630 (CM2)."
SNMPv2-SMI::enterprises.4130.2.1.1.2.1.2.0 = STRING: "***********"
SNMPv2-SMI::enterprises.4130.2.1.1.3.2.4.0 = INTEGER: -63
SNMPv2-SMI::enterprises.4130.2.1.1.3.2.5.0 = INTEGER: 24
SNMPv2-SMI::enterprises.4130.5.2.5.0 = STRING: "40 days, 03 hours."
[osvaldo@na3lmon1 ~]$

>>> import dateparser
>>> dateparser.parse('1 hour ago')
datetime.datetime(2015, 10, 15, 12, 39, 22, 618141)
>>> dateparser.parse('40 days, 03 hours.')
datetime.datetime(2015, 9, 5, 10, 39, 22, 618141)
>>>
>>> dateparser.parse('06 hrs, 38 min')
datetime.datetime(2015, 10, 15, 7, 1, 22, 618141)
>>> dateparser.parse('32 days, 06 hours')
datetime.datetime(2015, 9, 13, 7, 39, 22, 618141)
>>>

https://pypi.python.org/pypi/dateparser

Python - uptime

(master) osvaldo@mbp:remotos-sur $ pip install dateparser
Collecting dateparser
  Downloading dateparser-0.3.0-py2.py3-none-any.whl
Requirement already satisfied (use --upgrade to upgrade): PyYAML in 
        /usr/local/lib/python2.7/site-packages (from dateparser)
Collecting python-dateutil>=2.3 (from dateparser)
  Downloading python_dateutil-2.4.2-py2.py3-none-any.whl (188kB)
    100% | | 192kB 546kB/s
Collecting six>=1.5 (from python-dateutil>=2.3->dateparser)
  Downloading six-1.10.0-py2.py3-none-any.whl
Installing collected packages: six, python-dateutil, dateparser
Successfully installed dateparser-0.3.0 python-dateutil-2.4.2 six-1.10.0

instalamos dateparser

Python - uptime

>>> hh = dateparser.parse('06 hrs, 38 min')
>>> time.mktime(hh.timetuple())
1444903282.0

>>> int(hh.strftime('%s'))
1444903282
>>>

>>> str(datetime.datetime.fromtimestamp(1444903282))
'2015-10-15 07:01:22'
>>>

usando dateparser

Python - uptime

instalando dateparser en el server remoto (sin conexion a internet)

1. descargamos los paquetes localmente
2. copiamos los paquetes al server remoto
3. instalamos los paquetes en el server remoto

Python - uptime

# online host
PIP_CACHE=/var/tmp/pip
mkdir ${PIP_CACHE}
pip install --download=${PIP_CACHE} dateparser

instalando dateparser en el server remoto (sin conexion a internet)

Python - uptime

pero si lo haces en una mac, hay paquetes (PyYML?!) que no te van a funcionar en un linux

$ ls /var/tmp/pip/
PyYAML-3.11-cp27-none-macosx_10_10_x86_64.whl  
six-1.10.0-py2.py3-none-any.whl 
dateparser-0.3.0-py2.py3-none-any.whl          
python_dateutil-2.4.2-py2.py3-none-any.whl

1. descargamos los paquetes

2. copiarlos para el server remoto

3. instalar 

# online host

# primero instalar pip

yum install pip

# pero el host offline no le funciona yum

# https://dl.fedoraproject.org/pub/epel/6/x86_64/repoview/python-pip.html
wget https://dl.fedoraproject.org/pub/epel/6/x86_64/python-pip-7.1.0-1.el6.noarch.rpm -P /var/tmp/pip/
rsync -av -e ssh /var/tmp/pip nagios:/var/tmp/



# offline host

yum --nogpgcheck localinstall /var/tmp/pip/python-pip-7.1.0-1.el6.noarch.rpm 

# instalemos el paquete dateparser
PIP_CACHE=/var/tmp/pip
sudo pip install --no-index --find-links=file://$PIP_CACHE  dateparser

hagámoslo desde en un server CentOS

Python - uptime

# virtualenvwrapper tiraba error

TypeError: __init__() got an unexpected keyword argument 'stream'
virtualenvwrapper.sh: There was a problem running the initialization hooks.

# solucion: instalar la version anterior (descargar, copiar, instalar)

pip install virtualenvwrapper==4.7.0

# dataparser da error en CentOS 6
# solucion: probar version anterior

pip install --download=${PIP_CACHE} dateparser==0.2.1

# sigue dado error

>>> import dateparser
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/osvaldo/src/python/virtualenvs/nagioscs-pip
    /lib/python2.6/site-packages/dateparser/__init__.py", line 4, in <module>
    from date import DateDataParser

hagámoslo desde en un server CentOS 6

Python - uptime

decido guardar el uptime como string y continuo con la parte de snmp

Python - uptime

sudo yum install net-snmp-python

# online host
sudo yum install --downloadonly --downloaddir=/tmp net-snmp-python

# me faltaba el plugin de yum, lo instalo
sudo yum install yum-plugin-downloadonly

# offline host
yum --nogpgcheck localinstall /var/tmp/pip/python-pip-7.1.0-1.el6.noarch.rpm 
# instalo el paquete pero encuentro dependencias, las descargo, copio, instalo ...
yum --nogpgcheck localinstall /var/tmp/pip/lm_sensors-libs-3.1.1-17.el6.x86_64.rpm
yum --nogpgcheck localinstall /var/tmp/pip/net-snmp-libs-5.5-54.el6_7.1.x86_64.rpm
yum --nogpgcheck localinstall /var/tmp/pip/net-snmp-python-5.5-54.el6_7.1.x86_64.rpm

# imposible, tenia monton de dependencias. mas sencillo usar proxy

# instalo y configuro un proxy en un server con conexion a internet: squid

#/etc/yum.conf
proxy=http://proxy.acme.com:3128

yum -y install pysnmp

instalando paquetes

Python - snmp

# http://pysnmp.sourceforge.net/quick-start.html#fetch-snmp-variable

[root@nagios ~]# python
Python 2.6.6 (r266:84292, Jan 22 2014, 09:42:36)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from pysnmp.hlapi import *
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named hlapi

# claro, error ...
# en la página aclaran:
# " (you’d need at least version 4.3.0 to run code from this page)."
# la versión que se instala en CentOS es anterior

probando código

Python - snmp

[root@nagios ~]# python
Python 2.6.6 (r266:84292, Jan 22 2014, 09:42:36)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from pysnmp.entity.rfc3413.oneliner import cmdgen
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.6/site-packages/pysnmp/entity/rfc3413/oneliner/cmdgen.py", line 1, in <module>
    from pysnmp.entity import engine, config
  File "/usr/lib/python2.6/site-packages/pysnmp/entity/engine.py", line 2, in <module>
    from pysnmp.proto.rfc3412 import MsgAndPduDispatcher
  File "/usr/lib/python2.6/site-packages/pysnmp/proto/rfc3412.py", line 3, in <module>
    from pyasn1.compat.octets import null
ImportError: No module named compat.octets


# pareciera problema de versiones ... 
# http://stackoverflow.com/questions/24356398/no-module-named-compat-octets

probando código (otro ejemplo)

Python - snmp

Installation

~~~
go get github.com/soniah/gosnmp
~~~

http_proxy=http://proxy.acme.com:3128 go get github.com/soniah/gosnmp


alias go='http_proxy=127.0.0.1:8080 go'

no hace nada ....

probando con Go (la forma desesperada)

Python - Go

# http://www.emilsit.net/blog/archives/how-to-use-the-git-protocol-through-a-http-connect-proxy/

#@server
sudo yum -y install socat
sudo vi /usr/local/bin/gitproxy-socat
sudo chmod 755 /usr/local/bin/gitproxy-socat
git config --global core.gitproxy gitproxy

#@squid server

#add to /etc/squid3/squid.conf
acl SSL_ports port 9418         # git
acl Safe_ports port 9418        # git

# nada ...
http_proxy=http://proxy.acme.com:3128 
export http_proxy
git config --global http.proxy $http_proxy

error again

Initialized empty Git repository in /home/osvaldo/src/go/src/github.com/soniah/gosnmp/.git/
error: Received HTTP code 302 from proxy after CONNECT while accessing https://github.com/soniah/gosnmp/info/refs

probando con Go (la forma desesperada)

Python - Go

http_proxy=http://proxy.acme.com:3128 
export http_proxy
https_proxy=http://proxy.acme.com:3128 
export https_proxy
git config --global http.proxy $http_proxy
git config --global https.proxy $https_proxy

export GIT_TRACE_PACKET=1
export GIT_TRACE=1
export GIT_CURL_VERBOSE=1


git config --global http.sslverify false


import "github.com/soniah/gosnmp"

# nada ...

probando con Go (la forma desesperada)

Python - Go

OUT=$(mktemp /var/tmp/nagiostmpdata.XXXXXXXXXX) || { echo "Failed to create temp file"; exit 1; }

cat <<_EOF_>$OUT
networkname,host=${HOST} value="${NetworkName}"
rssi,host=${HOST} value=${RSSI}
snr,host=${HOST} value=${SNR}
_EOF_

#UPTIME,host=${HOST},name={NAME} value=${UPTIME}
curl -i -XPOST 'http://localhost:8086/write?db=mydb' --data-binary @${OUT}

it just works

Python - Bash

# yeap:
curl -i -XPOST 'http://localhost:8086/write?db=mydb' 
  --data-binary 'networkname,host=11.192.100.155,region=cs value="^pxC{bqYMraP3}&."'

# nop:
curl -i -XPOST 'http://localhost:8086/write?db=mydb'  
  -d '{"points" : [{ "measurement" : "rssi", 
  "fields" : { "value" : -65 }, "tags" : { "region" : "cs", "ip" :  "11.192.100.155" } } ]}'

# yeap:
curl -G 'http://localhost:8086/query?pretty=true' 
  --data-urlencode "db=mydb" --data-urlencode "q=SELECT value FROM rssi"

# nop:
curl -X POST 'http://localhost:8086/db/mydb/series' 
  -d '{"name": "rssi", "columns": ["value"], "points": [[-66]]}'

probando como cargar los datos

Python - Bash

algunas formas funcionaban, otras no. al final me quede con la generación de un archivo y cargarlo desde ahi (diapositiva anterior)

se complicaba cargar el networkname en InfluxDB

InfluxDB - Datos

Solución

cargar datos númericos en InfluxDB y los de texto en MySQL

# -----------------
# System Name: PB-2015 (PB-500)
# -----------------
define host{
       use                     generic-host-remote  ; Inherit default values from a template
       #host_name               PB-2015  ; The name we're giving to this server
       #host_name               PB-2015 ; The name we're giving to this server
       host_name               PB-2015 ; The name we're giving to this server
       #host_name               PB-2015
       alias                   PB-2015 (PB-500) ; A longer name for the server
       address                 10.11.12.13   ; IP address of the server
       hostgroups              remotes_inet900;
       statusmap_image         switch.gd2
       icon_image              switch.gif
       contact_groups          admins
}

hosts definition = objects.cfg

Nagios - Datos

10.11.12.13,online,PB-2015 (PB-500),PBO, -65, 21,43 days, 02 hours

Data file (csv): objects.csv

Nagios - Datos

{% for host in item %}
# -----------------
# System Name: {{ host.name[5:] }}
# -----------------
define host{
       use                     moxa-switch  ; Inherit default values from a template
       host_name               {{ host.name }} ; The name we're giving to this server
       alias                   {{ host.name }} ; A longer name for the server
       address                 {{ host.ip }}   ; IP address of the server
       hostgroups              AP
       statusmap_image         switch.gd2
       icon_image              switch.gif
       contact_groups          admins
}

{% endfor %}

 Template file: objects.cfg.j2

Error: The name of host 'EH-4161 (D)' contains one or more illegal characters.

       host_name               {{ host.name | regex_replace('^(.*)\(.*$', '\\1') | regex_replace('^(.*)\(.*$', '\\1') }} 


Error: The name of host 'MEa-4012 EC)' contains one or more illegal characters.

hay un hostname que tiene un ")" pero no tiene "(" por lo que la regex fallaba
tomo lo que esta hasta el primer espacio en blanco y descarto el resto
pero no ...
Warning: Duplicate definition found for host 'MEA' 

fgrep 'MEA' data/na3lmon1/csv/remotes.csv
11.192.112.75,online,MEA 4013 (ME1),ME1, -75, 22,46 days, 01 hours
11.192.112.157,online,MEA 3069(ME),ME, -75, 21,46 days, 01 hours
11.192.117.225,online,MEA-3184 (ME),ME, -77, 24,1 days, 04 hours
11.192.112.187,online,MEA-3186 (ME),ME, -80, 22,20 days, 06 hours

algunos tienen guion, otros no ....

host_name               {{ host.name | regex_replace('^([a-zA-Z0-9\-\s]+)(\(.*$)', '\\1') | regex_replace('^(.*)\(.*$', '\\1') }} ;
ese no sirve ...
Error: The name of host 'MEa-4012 EC)' contains one or more illegal characters.

falta de estandarización en los host name

Nagios - Datos

# pruebo con un if/then/else

{% if '-' in host.name  %}
       host_name               {{ host.name | regex_replace('^([a-zA-Z0-9\-]+)\s.*$', '\\1') | regex_replace('^(.*)\(.*$', '\\1') }} ; The name we're giving to this server
{% else %}
       host_name               {{ host.name | regex_replace('^([a-zA-Z0-9\-\s]+)(\(.*$)', '\\1') | regex_replace('^(.*)\(.*$', '\\1') }} ; The name we're giving to this server
{% endif  %}

---

no sirve
Error: The name of host 'LH-3006(d)' contains one or more illegal characters.
Error: The name of host 'ME-4031(D)' contains one or more illegal characters.

falta de estandarización en los host name

Nagios - Datos

{% for host in item %}
{% if 'name' in host  %}
# -----------------
# System Name: {{ host.name }}
# -----------------
define host{
       use                     generic-host-remote  ; Inherit default values from a template
       #host_name               {{ host.name | regex_replace('^(.*)\(.*$', '\\1') | regex_replace('^(.*)\(.*$', '\\1') }} 
       #host_name               {{ host.name | regex_replace('^([a-zA-Z0-9\-]+)\s.*$', '\\1') | regex_replace('^(.*)\(.*$', '\\1') }} 
{% if '-' in host.name  %}
       host_name               {{ host.name | regex_replace('^([a-zA-Z0-9\-]+).*$', '\\1') | regex_replace('^(.*)\(.*$', '\\1') }} 
{% else %}
       host_name               {{ host.name | regex_replace('^([a-zA-Z0-9\-\s]+)(\(.*$)', '\\1') | regex_replace('^(.*)\(.*$', '\\1') }} 
{% endif  %}
       #host_name               {{ host.name | regex_replace('^(.*)(\(.*$)', '\\1') | regex_replace('^(.*)\(.*$', '\\1') }}
       alias                   {{ host.name }} ; A longer name for the server
       address                 {{ host.ip }}   ; IP address of the server
       hostgroups              remotes_inet900;
       statusmap_image         switch.gd2
       icon_image              switch.gif
       contact_groups          admins
}

{% endif  %}
{% endfor %}

Template file

Nagios - Datos

http://gitlab.sinopecarg.com.ar/infra/tempura/blob/master/data/na3lmon1/templates/remotes.cfg.j2

bonus point: se desarrollo un plugin de ansible (python) 

Retos

• impacto en el ancho de banda al tener >1k hosts realizando chequeos en simultáneo.

• distribuir los chequeos en el tiempo para minizar dicho impacto.

• impacto de la consulta snmp en el host

Nagios - Obtener datos

Demo

http://tcs.sinopecarg.com.ar/

Gracias