elasticsearch - Parsing XML files to LogStash -
i have following logstash conf file:
input { file { path => "c:\dashboard\elmah\*.xml" start_position => "beginning" type => "error" codec => multiline { pattern => "^<\?error .*\>" negate => true => "previous" } sincedb_path => "c:\dashboard\elmah" } } filter { xml { source => "error" xpath => [ "/error/@errorid", "errorid", "/error/@type", "type", "/error/@message", "message", "/error/@time", "time", "/error/@user", "user" ] store_xml => true } } output { elasticsearch { action => "index" host => "localhost" index => "stock" workers => 1 } stdout { codec => rubydebug } }
when run bin/logstash -f agent.conf not error no data gets inserted elasticsearch. example of file is: https://www.dropbox.com/s/6oni2zhorsdtz6p/error-2015-06-26203423z-3026bd43-07d6-44d6-a6cf-6d27b28a607e.xml?dl=0
how logstash read in collection of external xml files?
logstash debug output:
please see here: https://www.dropbox.com/s/g7g1154uvf9fr1f/outputlog2.txt?dl=0
i'm not sure can use file input here -- i've seen used monitor files changes, not monitor new files. unless xml files updated, don't think anything. remember logstash watching new log lines typically.
most people write tools following process whole files in batch:
https://github.com/elastic/elasticsearch-river-wikipedia
https://github.com/andrewvc/wikiparse
https://github.com/elastic/stream2es
those tools, last one, seem closer use case.
Comments
Post a Comment