Wednesday, September 7, 2016

Use Slack bot to monitor the server


I used to install Datadog or other SaaS to monitor my Linux boxes on the cloud. Most times they are just overkill for my tiny servers with only 1GB or 2GB memory. Actually what I am most interested is the up-and-running processes, or/and the exact memory usage. And I need a mobile solution to monitor on-the-go.
Now with the coming of Slack bot, and its real time Python client, I can just use a simple Python script to realize the purposes.
from slackclient import SlackClient
from subprocess import  getoutput
import logging
import time

message_channel = '#my-server-001'
api_key = 'xoxb-slack-token'
client = SlackClient(api_key)

if client.rtm_connect():
    while True:
        last_read = client.rtm_read()
        if last_read:
            try:
                parsed = last_read[0]['text']
                if parsed and 'status' in parsed:
                    result = getoutput('pstree')
                    result += '\n\n' + getoutput('free -h')
                    client.rtm_send_message(message_channel, str(result))
            except Exception as e:
                logging.error(e)
        time.sleep(1)
Then I use systemd or other tools to daemonize it. No matter where and when I am, I enter status at the #my-server-001 channel on my phone, I will instantly get the result like -
systemd-+-accounts-daemon-+-{gdbus}
       |                 `-{gmain}
       |-agetty
       |-cron
       |-dbus-daemon
       |-fail2ban-server---2*[{fail2ban-server}]
       |-login---bash
       |-nginx---nginx
       |-postgres---5*[postgres]
       |-python---sh---pstree
       |-redis-server---2*[{redis-server}]
       |-rsyslogd-+-{in:imklog}
       |          |-{in:imuxsock}
       |          `-{rs:main Q:Reg}
       |-sshd-+-3*[sshd---sshd---bash]
       |      `-sshd---sshd
       |-2*[systemd---(sd-pam)]
       |-systemd-journal
       |-systemd-logind
       |-systemd-timesyn---{sd-resolve}
       |-systemd-udevd
       `-uwsgi---uwsgi---5*[uwsgi]

             total        used        free      shared  buff/cache   available
Mem:           2.0G        207M        527M         26M        1.2G        1.7G
Swap:          255M          0B        255M

Good math, bad engineering

As a formal statistician and a current engineer, I feel that a successful engineering project may require both the mathematician’s abilit...