Elasticsearch Notes

The ELK stack consists of, roughly, three parts. The consumer, logstash; the Storage, elasticsearch; and the viewer, kibana.

Elasticsearch Commands
Temperature monitor (with or without nagios)
Rolling Upgrade
Nmap Logging

Elasticsearch Commands

Display cluster details

curl -XGET 'localhost:9200/_cluster/health/?pretty'

Create template to limit shards and replcias

curl -XPUT 'localhost:9200/_template/template_forall?pretty' -H 'Content-Type: application/json' -d'
{
    "template" : "*",
    "order" : 0,
    "settings" : {
        "number_of_shards" : 1,
        "number_of_replicas" : 1
    }
}
'

Display shard details

curl -XGET 'localhost:9200/_cat/shards/logstash-2017.06.06/?human&pretty'

logstash-2017.06.06 0 p STARTED 148712 186.6mb 10.1.10.149 elk1
logstash-2017.06.06 0 r STARTED 148714 186.7mb 10.1.10.144 elk0

Finding Problem Shards ~~~ curl -XGET localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason| grep UNASSIGNED ~~~

Delete an index

curl -XDELETE 'localhost:9200/twitter?pretty'

Troubleshooting Shard allocaiton issues

We’ve ran into some issues with shard allocation once in a while. Here are some good references to help diagnose:

https://www.datadoghq.com/blog/elasticsearch-unassigned-shards/

curl -XGET 'localhost:9200/_cat/shards/?human&pretty' | grep UNASSIGNED

curl -XPUT 'localhost:9200/indexname-2018.06.15/_settings?pretty' -H 'Content-Type: application/json' -d'{"index":{"number_of_replicas" : 0}}'

Temperature monitor (with or without nagios)

I wrote up a little python script that reads the gpio ports on the raspberry pi for a DS18B20 temperature sensor. It’s probably not perfect, but it seems to work.

 python gpio_check.py --syslogserver 192.168.1.144 --syslogport 1514 --site SlowDownHome --sensor FloorTemp --sensorsystem 28-000006152042 --sensortype temp --measure f -c 65 -w 60

output looks something like

<SlowDownHome, FloorTemp> Warning: 64.2866 is greater than 60 for SlowDownHome - FloorTemp

OK, the logstash bits now…

input {
  tcp{
    port => 1514
    tags => ["temp"]
  }
  udp{
    port => 1514
    tags => ["temp"]
  }
}
filter {
  if "temp" in [tags] {
    grok {
      match => { "message" => "<%{WORD:site}, %{WORD:sensor}> %{WORD:status}: %{BASE16FLOAT:ftemp:float}"  }
    }
  }
}

Another optional config (if you have multiple types of messages coming in with a differnt format)

input {
  tcp{
    port => 1514
    tags => ["syslog"]
  }
  udp{
    port => 1514
    tags => ["syslog"]
  }
}
filter {
  if "syslog" in [tags] {
    grok {
      match => { "message" => "<%{WORD:site}, %{WORD:sensor}> %{WORD:status}: %{BASE16FLOAT:ftemp:float}"  }
      add_tag => ["temp"]
    }
  }
}

Rolling upgrade (v5.x)

curl -XPUT 'localhost:9200/_cluster/settings?pretty' -H 'Content-Type: application/json' -d'
{
    "transient": {
          "cluster.routing.allocation.enable": "none"
            }
}
'

Sync Flush

curl -XPOST 'localhost:9200/_flush/synced?pretty'

Do upgrade bits

Reenable cluster routing

curl -XPUT 'localhost:9200/_cluster/settings?pretty' -H 'Content-Type: application/json' -d'
{
    "transient": {
          "cluster.routing.allocation.enable": "all"
            }
}
'

Check health of the cluster

curl -XGET 'localhost:9200/_cat/health?pretty'

Elasticsearch Queries

{"query": {"bool":{"must": [{"match":{"field_name": "fieldvalue"}},{"match": {"host":"hostname"}}],"filter":{"range":{ "@timestamp": { "gte": "now-120m", "lte": "now" } }}}},"size":1,"sort":[{"@timestamp":{"order":"desc"}}]}

Nmap logging

Not sure this works any longer… There wasn’t a codec available when last I checked… but that was before now; maybe there is one now?.

install logstash codec

/usr/share/logstash/bin/logstash-plugin install logstash-codec-nmap

config

input {
  http {
    port => 8080
    codec => "nmap"
    tags => ["nmap"]
  }
}
output {
    stdout { codec => rubydebug }  
}

nmap command

nmap 192.168.1.1-254 -oX - | curl -H "x-nmap-target: Test" http://192.168.1.144:8080 --data-binary @-

output looks something like this

{
  "_index": "nmap-2017.05.20",
  "_type": "nmap_host",
  "_id": "AVwmqrxjnyQGj4R3m1i5",
  "_version": 1,
  "_score": null,
  "_source": {
    "addresses": [
      {
        "type": "ipv4",
        "addr": "192.168.1.126"
      },
      {
        "type": "mac",
        "addr": "55:55:55:55:55:55"
      }
    ],
    "traceroute": null,
    "type": "nmap_host",
    "ports": [
      {
        "number": 8008,
        "reason": "syn-ack",
        "protocol": "tcp",
        "service": {
          "protocol": null,
          "product": null,
          "hostname": null,
          "fingerprint_method": "table",
          "confidence": 3,
          "name": "http",
          "fingerprint": null,
          "device_type": null,
          "ssl": false
        },
        "state": "open"
      },
      {
        "number": 8009,
        "reason": "syn-ack",
        "protocol": "tcp",
        "service": {
          "protocol": null,
          "product": null,
          "hostname": null,
          "fingerprint_method": "table",
          "confidence": 3,
          "name": "ajp13",
          "fingerprint": null,
          "device_type": null,
          "ssl": false
        },
        "state": "open"
      },
      {
        "number": 9000,
        "reason": "syn-ack",
        "protocol": "tcp",
        "service": {
          "protocol": null,
          "product": null,
          "hostname": null,
          "fingerprint_method": "table",
          "confidence": 3,
          "name": "cslistener",
          "fingerprint": null,
          "device_type": null,
          "ssl": false
        },
        "state": "open"
      }
    ],
    "mac": "55:55:55:55:55:55",
    "times": {
      "3781": 0,
      "rttvar": 0
    },
    "hostname": null,
    "ipv4": "192.168.1.126",
    "ipv6": null,
    "@version": "1",
    "host": "192.168.1.113",
    "id": "560d18f4-9127-4f71-a049-15158410f07f-h6",
    "headers": {
      "http_accept": "*/*",
      "content_type": "application/x-www-form-urlencoded",
      "http_x_nmap_target": "Test",
      "request_path": "/",
      "http_version": "HTTP/1.1",
      "request_method": "POST",
      "http_host": "192.168.1.144:8080",
      "request_uri": "/",
      "content_length": "12828",
      "http_expect": "100-continue",
      "http_user_agent": "curl/7.54.0"
    },
    "address": "192.168.1.126",
    "os": null,
    "ip": "192.168.1.126",
    "end_time": "2017-05-20T16:22:13.000Z",
    "version": "7.40",
    "uptime": null,
    "tags": [
      "nmap"
    ],
    "start_time": "2017-05-20T16:22:10.000Z",
    "@timestamp": "2017-05-20T16:22:22.440Z",
    "arguments": "nmap -oX - 192.168.1.1-254",
    "scan_id": "560d18f4-9127-4f71-a049-15158410f07f",
    "status": {
      "reason": "arp-response",
      "state": "up"
    }
  },
  "fields": {
    "end_time": [
      1495297333000
    ],
    "start_time": [
      1495297330000
    ],
    "@timestamp": [
      1495297342440
    ]
  },
  "sort": [
    1495297342440
  ]
}