Discovered logstash last year, finally I have played around logstash and elasticsearch during weekend, pretty easy to walk through Getting Started. However when I try to customize pattern for work-related log, its documentation does not come with good example and its API changed between 1.2 and 1.3… some search result from google using deprecated version. Anyway here is only document what I played so far.
Preparation:
Download a “flat” jar (logstash integrated with elasticsearch now): http://download.elasticsearch.org/logstash/logstash/logstash-1.3.3-flatjar.jar
required curl, if it does not installed yet: $ sudo apt-get install curl
Tasting:
– Walk through “Getting started with logstash (standalone server example)”
– For production, input would better be log4j rather than file, here just a test.
config file:
Since the log4j timestamp does not use ISO format, need to define it. see http://logstash.net/docs/1.3.3/filters/grok#patterns_dir
create a folder named “pattern”, create a file inside and save with follow content.
|
1 |
LOGDATETIME %{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND} |
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 |
input { stdin { type => "stdin-type" } file { type => "xmllog" # path must be absolute path => "/workspaces/logstatsh/log4j-xml.log" } } filter { grok { patterns_dir => "./patterns" # sample log: 2014-03-08 11:50:02.001 INFO [loggingHandler] (logger.java:123) - <?xml .....> match => [ "message", "%{LOGDATETIME:logDateTime}%{SPACE}%{LOGLEVEL:level}%{SPACE}\[%{DATA:thread}\]%{SPACE}\(%{JAVACLASS:class}:%{NUMBER:line}\)%{SPACE}-%{SPACE}(?<xmlData>.*$)" ] # remove fields don't care, e.g. line number remove_field => [ "line" ] } date { # matched pattern will replace "@timestamp" field to logDateTime (parsed above) match => ["logDateTime", "YYYY-MM-dd HH:mm:ss.SSS"] } xml { source => "xmlData" target => "parsedXml" } } output { stdout { debug => true } elasticsearch { embedded => true } } |
N.B. in xml filter, remember to define target => “xxxx”! Otherwise console debug logged xmlparsefailure and NoMethodException. With it, all first level children xml tags will be indexed and fields will be created automatically, no need to setup XPath.
okay, now run $ java -jar logstash-1.3.3-flatjar.jar agent -f test.conf -- web
By paste following line to ‘standard in’ or listened log file: (xml from w3school example)
|
1 |
2014-03-09 19:01:03.005 INFO [loggingHandler] (logger.java:123) - <note><to>Tove</to><from>Jani</from><heading>Reminder</heading><body>Don't forget me this weekend!</body></note> |
then, by debug=true, logstash shell console display information. And it will send to elasticsearch with following result.
Now go to http://localhost:9292/index.html#/dashboard/file/logstash.json to see the result!
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 |
{ "_index": "logstash-2014.03.09", "_type": "stdin-type", "_id": "94ycZEdfQDCF_c5PxP-zlg", "_score": null, "_source": { "message": "2014-03-09 19:01:03.005 INFO [loggingHandler] (logger.java:123) - <note><to>Tove</to><from>Jani</from><heading>Reminder</heading><body>Don't forget me this weekend!</body></note>", "@version": "1", "@timestamp": "2014-03-09T19:01:03.005+10:00", "type": "stdin-type", "host": "hang321-mintvm", "logDateTime": "2014-03-09 19:01:03.005", "level": "INFO", "thread": "loggingHandler", "class": "logger.java", "xmlData": "<note><to>Tove</to><from>Jani</from><heading>Reminder</heading><body>Don't forget me this weekend!</body></note>", "parsedXml": { "to": [ "Tove" ], "from": [ "Jani" ], "heading": [ "Reminder" ], "body": [ "Don't forget me this weekend!" ] } }, "sort": [ 1394355663005, 1394355663005 ] } |