elasticsearch - Parsing XML files to LogStash -


i have following logstash conf file:

input {   file  {     path => "c:\dashboard\elmah\*.xml"     start_position => "beginning"     type => "error"     codec => multiline      {         pattern => "^<\?error .*\>"         negate => true         => "previous"     }     sincedb_path => "c:\dashboard\elmah"   } }  filter  {     xml      {         source => "error"         xpath =>          [             "/error/@errorid", "errorid",             "/error/@type", "type",             "/error/@message", "message",             "/error/@time", "time",             "/error/@user", "user"         ]         store_xml => true     } }  output  {     elasticsearch      {          action => "index"         host => "localhost"         index => "stock"         workers => 1     }     stdout      {         codec => rubydebug     } } 

when run bin/logstash -f agent.conf not error no data gets inserted elasticsearch. example of file is: https://www.dropbox.com/s/6oni2zhorsdtz6p/error-2015-06-26203423z-3026bd43-07d6-44d6-a6cf-6d27b28a607e.xml?dl=0

how logstash read in collection of external xml files?

logstash debug output:

please see here: https://www.dropbox.com/s/g7g1154uvf9fr1f/outputlog2.txt?dl=0 

i'm not sure can use file input here -- i've seen used monitor files changes, not monitor new files. unless xml files updated, don't think anything. remember logstash watching new log lines typically.

most people write tools following process whole files in batch:

https://github.com/elastic/elasticsearch-river-wikipedia

https://github.com/andrewvc/wikiparse

https://github.com/elastic/stream2es

those tools, last one, seem closer use case.


Comments

Popular posts from this blog

android - Gradle sync Error:Configuration with name 'default' not found -

java - Andrioid studio start fail: Fatal error initializing 'null' -

html - jQuery UI Sortable - Remove placeholder after item is dropped -