Kentaro Kuribayashi's blog

Software Engineering, Management, Books, and Daily Journal.

More appropriate LTSV usage and fluent-plugin-extract_query_params

In id:stanaka's blog post on LTSV, he wrote they use %r for req key to record request methods, request uri, and protocol. I think the value %r provides, for example "GET /foo/bar?hoge=fuga HTTP1.1", is not easy to treat. I'd rather propose to use the format below:

For Apache:

method:%m\tpath:%U%q\tprotocol:%H

For nginx:

method:$request_method\tpath:$request_uri\tprotocol:$server_protocol

Using this format, the log will be emitted as below:

method:GET path:/foo/bar?hoge=fuga protocol:HTTP1.1

It's more convenient to do with by some unix toolkits or fluentd, isn't it? I strongly recommend you choose it.

Besides, you can use my fluent plugin named fluent-plugin-extract_query_params if you adopt the format described above.

<match access_log>
  type extract_query_params

  key                   path
  add_tag_prefix extracted.
  only                  hoge
</match>

With this configuration, the log will be emitted as below:

extracted.access_log => {
  "method"  : "GET",
  "path"       : "/foo/bar?hoge=fuga",
  "protocol" : "HTTP1.1",
  "hoge"      : "fuga"
}

I think this is really innovative for fluent world.