Logstash: how to get logs in a readable format (part 3 of 3: Troubleshooting)


troubleshootingIn this third and final Logstash post on how to get logs in a readable format, I will talk about troubleshooting:

Troubleshooting

Verifying a config file

Note that you can verify a conf file by being in the logstash bin directory and running

logstash agent -f c:\ProgramData\logstash\<specific conf file> --configtest

When I first ran this I left off the specific conf file and so it validated every conf file I had.

It is worth realising that logstash first concatenates every logstash conf file relevant for that server (in its config directory e.g. C:\ProgramData\Logstash\ ) and then runs Logstash.  Thus if you don’t specify a file the line number of any error will probably be greater than the number of lines in any file.

Equally confusing is that the Logstash service will stop if it finds an error in any of the concatenated conf files.  Thus if you arent getting logs it could be that logstash has stopped and this could have nothing to do with your conf file.

Reasons why logs may not be appearing in Kibana

As stated above, Logstash is very fragile (to errors in its conf file) so its always worth checking if logstash is actually running – restart it. If on hitting restart, the command changes to start then its not healthy.  Conftest your conf files.  NB I had expected in these circumstances to see logstash as a stopped service, but that didnt seem to be the case.

Since I was developing my conf file I took to commenting out bits of it.  This wasnt always a good idea as comments are a bit fragile too. eg

mutate {
add_field => {
"instanceid" => "ie-7942b5e"
"environment" => "c03"
#   "role" => "eCommerce"
"owningcluster" => "CNR"

}
}

doesnt work and will stop logstash.  It seems that a comment in the middle of setting values in an array is bad – putting it before any fields are assigned is OK

Logs appearing twice

At some point the same log entry appeared twice in Kibana.  This was caused by a zombie java process still running logstash.  Killing it solved the problem.  Of course it was much easier to spot a rogue java process on a windows machine (stop logstash and it was the only java process still running)…or you could right click on the Name column in Task Manager and enable display of the command line.

Re-running test files

It is useful when testing to be able to work with a test file with a small number of log entries and be able to replay them time and again.

The internet says that the way to do this is to set the sincedb_path to “/dev/null”

This, alas, is a Unix null device and doesn’t work in Windows.  I couldn’t get a a windows equivalent to work.

The best I managed was to stop logstash and then cut out all rows in the file I was using.  Then, when logstash restarts it recognises that its read-up-to marker it greater than the lines in the file and resets it accordingly.

Then you can simply paste back the rows and save and they will be replayed.

It’s as well to subtly change an element of the data between each run so you can be sure that any data you are looking at has been generated by this run.

Click here to read part 1 on “Parsing”

Click here to read part 2 on “Basic Principles”

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s