Installing Graylog2 logging server with Ansible and configure Rails properly
Out Of Date Warning
This article was published on 28/02/2014, this means the content may be out of date or no longer relevant.
You should verify that the technical information in this article is still up to date before relying upon it for your own purposes.
Intrigued by the screencast video about the Graylog2 daemon , I decided to try Graylog2 as a central logging server for our application log files. In this post, I want to show, how I used Ansible to easily configure a server with all the necessary daemons. In the second part, I want to give some information on how to get the most of it using Rails, by including information like name of the current logged in user.
Installation with Ansible
Luckily, Ansible-Galaxy provides some roles, we can just include into our Ansible playbook. For this playbook, I assume, you want to run the logging server on a single host. Each of the four parts, ElasticSearch, MongoDB, Graylog-Server, and Graylog-Web, could live under a different host. However, for our use case, having everything on the same host should be fine for a long time.
1. playbook.yml and inventory
---
- hosts: logging
# playbook.yml
sudo: true
vars:
allow_ips: # All app-server ips, that should be allowed to connect to the daemon
- 1.2.3.4
- 2.3.4.5
roles:
- role: gpstathis.ansible-elasticsearch
elasticsearch_version: 0.90.10
elasticsearch_heap_size: 1g
elasticsearch_max_open_files: 65535
elasticsearch_timezone: "Europe/Berlin"
elasticsearch_node_max_local_storage_nodes: 1
elasticsearch_index_mapper_dynamic: "true"
elasticsearch_cluster_name: graylog4
elasticsearch_node_name: elasticsearch-ansible-node
elasticsearch_memory_bootstrap_mlockall: "true"
elasticsearch_plugins: []
- role: bennojoy.mongo_mongod
mongod_replication: false
mongod_port: 2701
mongod_datadir_prefix: "/data/"
- role: ansible-graylog2
graylog_version: 0.20.1
graylog2_secret: "Put some very long random string here"
# Generate a sha256 hashed password:echo -n mypassword | shasum -a 256
graylog2_admin_password: "TODO: put a sha256 password here"
tasks:
- shell: 'iptables -L | grep 12201'
ignore_errors: true
register: iptables_has_port
- shell: 'iptables -A INPUT -p tcp --dport 12201 -s {{item}} -j ACCEPT && iptables -A INPUT -p udp --dport 12201 -s {{item}} -j ACCEPT'
with_items: allow_ips
when: iptables_has_port | failed
- shell: 'iptables -A INPUT -p tcp --dport 12201 -j DROP && iptables -A INPUT -p udp --dport 12201 -j DROP'
when: iptables_has_port | failed
Add the server to your inventory file (e.g. production):
# production
[logging]
logging.myserver.com ansible_ssh_user=root #... more options like ansible_ssh_port etc
2. Install missing dependencies
Make sure, your Ansible version is not too old (at least 1.4). As I am writing this, 1.5 should be released soon:
$ ansible-playbook --version
ansible-playbook 1.4.5
Install the missing Ansible Galaxy roles:
sudo ansible-galaxy install -i gpstathis.ansible-elasticsearch bennojoy.mongo_mongod
Install a third role, Graylog2, which is not in Ansible Galaxy right now:
mkdir -p roles
git clone https://github.com/nicolai86/ansible-graylog2.git roles/ansible-graylog2
3. Run the playbook + configure input
ansible-playbook -i production playbook.yml
Afterwards, navigate to http://logging.myserver.com:9000 and log in with admin and your password from above. There, under System -> Inputs, define a GELF UDP Input on the default port 12201. This port is protected by our iptables-Rules
If anything went wrong, have a look into the graylog2 log files on the server (/var/log/graylog2-server/console.log and /var/log/graylog2-web/console.log)
Getting logging input from Rails to Graylog2
Based on this Knowledge Base entry, we use gelf and lograge for the task:
# Gemfile
gem "gelf"
gem "lograge"
# bundle, of course
At the end of the config/environments/production.rb paste the logging configuration (I tried different kinds of initializer files and locations, but found it must be at the end of the environment-config. Otherwise the logger would still use stdout in development). Here, we depart from the knowledge base configuration, which is a bit thin and outdated.
# config/environments/production.rb
MyApp::Application.configure do
# ...
app_name = config.action_mailer.default_url_options.try(:fetch, :host) || Rails.application.class.name.underscore.split('/').tap{|i|i.pop}.join('/')
# or set to whatever you want
config.lograge.enabled = true
config.lograge.formatter = Lograge::Formatters::Graylog2.new
# config.lograge.log_format = :graylog2 # DEPRECATED
config.lograge.custom_options = ->(event) {
opts = event.payload.dup.delete_if{|k,v|v.nil?}.symbolize_keys
params = event.payload[:params]
if (params.present?) and (p = params.except("controller", "action",'utf8')) and (p.present?)
opts[:params] = p
else
opts.delete :params
end
opts[:duration] = event.duration
if event.payload[:exception]
quoted_stacktrace = %Q('#{Array(event.payload[:stacktrace]).to_json}')
opts[:stacktrace] = quoted_stacktrace
end
opts
}
config.lograge.ignore_custom = ->(event) {
event.payload[:bot]
}
config.logger = GELF::Logger.new("logging.yourserver.com", 12201, "WAN", { :host => app_name, :environment => Rails.env })
if Rails.env.production?
ActiveRecord::Base.logger = Logger.new('/dev/null')
end
end
As you can see, we:
- Define an
app_name(automatically by host-name or Rails application name) - Copy the event payload from the ActiveSupport notification into our log and do some parameter filtering. We don't need the supply the
controllerandactionparameters, because we get those as top level payload information anyway. - Define a custom ignore function, e.g. ignore all requests matched as bot
This was only the first part. We need an ApplicationController method for providing all those parameters we care about, including IP, HTTP-refer(r)er, User-Agent, logged in user. Here a starting example:
class ApplicationController < ActionController::Base
# ...
def append_info_to_payload(payload)
super
payload[:ip] = request.remote_ip
payload[:user_name] = current_user.try(:name) if defined?(current_user)
payload[:user_agent] = request.user_agent
payload[:referer] = request.env['HTTP_REFERER']
payload[:session] = session.to_hash.except("_csrf_token", "session_id")
payload[:bot] = (request.user_agent.blank? ||
request.user_agent[/bot\W|index|spider|crawl|wget|slurp|Mediapartners-Google/i])
end
end
Add whatever parameters you want and they will be delivered 1:1 to the logging interface. The bot matching is quite naive, and could be made better, but it gives you a good idea.
Commit, deploy and wait, and logs should fill your Graylog2!
Done! Happy logging.
