Graylog is an open-source log management tool which helps you to collect, index and analyze any machine logs centrally. This guide will help you to install Graylog on CentOS 7 / RHEL 7.
Components
- MongoDB – Being a database to store the configurations and meta information.
- Elasticsearch – It stores the log messages received from the Graylog server and provides a facility to search them whenever required. Elasticsearch is a resource monger as it does indexing of data, so allocate more memory and use SAS or SAN disks.
- Graylog server – This does the parsing of logs that are coming from various inputs and provides built-in Web Interface to handle those logs.
Prerequisites
Make sure your system is configured with EPEL repository.
Install the required packages for Graylog installation.
yum -y install wget pwgen
Elasticsearch requires Java to be installed on the machine. So, install either OpenJDK or Oracle JDK.
yum install -y java-1.8.0-openjdk-headless
Verify the Java version.
java -version
Output:
openjdk version "1.8.0_191" OpenJDK Runtime Environment (build 1.8.0_191-b12) OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode)
Install Elasticsearch
Elasticsearch is one of the important components in the Graylog setup. It stores the data coming from Graylog input and displays the messages whenever a user request over the Graylog built-in web interface.
Elasticseach is mainly used here to index the data/logs and provide the searching functionality when the Graylog web interface request for any information.
This guide covers the basic configuration required for Graylog.
Import the GPG signing key before the installation.
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
Create a repository file to get the latest version of Elasticsearch package from the official repository.
vi /etc/yum.repos.d/elasticsearch.repo
Add the below content to above file.
[elasticsearch-6.x] name=Elasticsearch repository for 6.x packages baseurl=https://artifacts.elastic.co/packages/6.x/yum gpgcheck=1 gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch enabled=1 autorefresh=1 type=rpm-md
Now, install the Elasticsearch package using yum command.
yum install -y elasticsearch
Reload the systemctl daemon and enable Elasticsearch to start automatically on the system startup.
systemctl daemon-reload systemctl enable elasticsearch
To make Elasticsearch work with Graylog setup, we need to set the cluster name to graylog. Edit the elasticsearch.yml file.
vi /etc/elasticsearch/elasticsearch.yml
Update it like shown below.
cluster.name: graylog
Restart the Elasticsearch.
systemctl restart elasticsearch
Give a minute to let the Elasticsearch get fully restarted. Elastisearch should be now listening to 9200 for processing HTTP requests. Use the CURL command to check the response.
curl -X GET http://localhost:9200
Cluster name should be graylog.
{
"name" : "DF8QK3-",
"cluster_name" : "graylog",
"cluster_uuid" : "_wAgUfN9RJeQ0npCKBswVA",
"version" : {
"number" : "6.6.0",
"build_flavor" : "default",
"build_type" : "rpm",
"build_hash" : "a9861f4",
"build_date" : "2019-01-24T11:27:09.439740Z",
"build_snapshot" : false,
"lucene_version" : "7.6.0",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}
Perform a health check of Elasticsearch cluster.
curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'
Make sure the cluster status comes as green.
{
"cluster_name" : "graylog",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}
Install MongoDB
MongoDB acts as a database to store the configurations and meta information. We will use MongoDB’s official repository to get the latest version of MongoDB.
vi /etc/yum.repos.d/mongodb-org-4.0.repo
Add repository information to the above file.
[mongodb-org-4.0] name=MongoDB Repository baseurl=https://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/4.0/x86_64/ gpgcheck=1 enabled=1 gpgkey=https://www.mongodb.org/static/pgp/server-4.0.asc
Install the community edition of MongoDB using the following command.
yum install -y mongodb-org
Start the MongoDB service and enable it to the system start-up.
systemctl start mongod systemctl enable mongod
Install Graylog
Graylog server accepts and processes the log messages coming from the various inputs, displays data to requests that come from the graylog web interface.
Download and Install graylog 3.x repository.
rpm -Uvh https://packages.graylog2.org/repo/packages/graylog-3.0-repository_latest.rpm
Install the Graylog server using the following command.
yum -y install graylog-server
Edit the server.conf file to begin the graylog configuration.
vi /etc/graylog/server/server.conf
Use the following command to create a secret
pwgen -N 1 -s 96
Output:
OH9wXpsNZVBA8R5vJQSnkhTB1qDOjCxAh3aE3LvXddtfDlZlKYEyGS24BJAiIxI0sbSTSPovTTnhLkkrUvhSSxodTlzDi5gP
Place the secret like below in server.conf
password_secret = OH9wXpsNZVBA8R5vJQSnkhTB1qDOjCxAh3aE3LvXddtfDlZlKYEyGS24BJAiIxI0sbSTSPovTTnhLkkrUvhSSxodTlzDi5gP
Set the hash password for the root user, i.e., admin of graylog. You would need this password to login into the graylog web.
If you ever want to change/reset the forgotten password of admin, you can edit/update the server.conf with a hashed password.
Generate a hashed password using the below command. Replace yourpassword with the choice of yours.
echo -n yourpassword | sha256sum
Output:
e3c652f0ba0b4801205814f8b6bc49672c4c74e25b497770bb89b22cdeb4e951
Place the hash password.
root_password_sha2 = e3c652f0ba0b4801205814f8b6bc49672c4c74e25b497770bb89b22cdeb4e951
You can set up an email address for the admin user.
root_email = "holhol24.web@gmail.com"
Set timezone of root (admin) user.
root_timezone = UTC
Set only one master node by defining the below variable, the default setting is true.
If this server is your second Graylog node in your environment, set it to false to make this node as a slave. Master node does some periodic tasks that slave nodes won’t perform.
is_master = true
Set the number of log messages to keep per index. It is recommended to have several smaller indices instead of larger ones.
elasticsearch_max_docs_per_index = 20000000
The following parameter defines to have a total number of indices. If this number is reached, the old index will be deleted.
elasticsearch_max_number_of_indices = 20
Shards setting rely on the number of nodes in the particular Elasticsearch cluster. If you have only one node, set it as 1.
elasticsearch_shards = 1
This defines the number of replicas for your indices. If you have only one node in the Elasticsearch cluster, set it as 0.
elasticsearch_replicas = 0
Install Graylog web interface
From the Graylog version 2.x, the web interface is served directly by the Graylog server. Enable the Graylog web interface by editing the server.conf file.
vi /etc/graylog/server/server.conf
Modify the entries to let Graylog Web Interface to connect to the Graylog server. Replace 192.168.1.10 with your system IP address.
http_bind_address = 192.168.1.10:9000
Restart Graylog service.
systemctl daemon-reload systemctl restart graylog-server
Make Graylog server to start automatically on system startup.
systemctl enable graylog-server
You can check out the server startup logs; it will be useful for you to troubleshoot Graylog for any issues.
tailf /var/log/graylog-server/server.log
If everything goes well, you should see below message in the server.log file.
2019-02-25T21:00:23.157+05:30 INFO [ServerBootstrap] Graylog server up and running.
Firewall
In CentOS 7 / RHEL 7, firewall rules are set to block most of the traffic coming from the external machines by default. You would need to add allow rules for Graylog to work correctly.
firewall-cmd --permanent --add-port=9000/tcp
Reload firewalld to take an effect of the new rules.
firewall-cmd --reload
Access Graylog web interface
The web interface will now be listening on port 9000. Point your browser to.
http://ip.add.re.ss:9000
Login with username admin and the password you configured at root_password_sha2 on server.conf.
Once you logged in, you would see the getting started page.
Click on System >> Overview to know the status of Graylog server.
Create Graylog Inputs
Graylog input needs to be configured to receive the logs from external sources, i.e., syslog or any logging system. For this example, we will set up an input for receiving logs from syslog on port number UDP 514.
But, there is a problem. If you start to try an input on one of the privileged ports, any TCP/UDP port number below 1024, you would see permission denied error in Graylog logs.
To resolve the issue, we will create an input to start on port 1514, and we will redirect the traffic comes to 514 to 1514 with firewall rules.
Click System >> Inputs >> select Syslog UDP and then click Launch new input.
Fill the values shown below.
Node: Select your Graylog Node
Title: Name your input
Bind address: 0.0.0.0 (Leave the default one)
Port: 1514
Ensure the created Syslog input is in running state.
Configure Rsyslog
Once you have created the inputs, configure Rsyslog or forward any system logs to your Graylog server.
Edit the Rsyslog configuration file.
vi /etc/rsyslog.conf
Typical Rsyslog configuration will look like below to send logs on UDP 514.
If you do not wish to use UDP 514 and want to send logs directly to UDP 1514 of Graylog, then you can change the port number in the Rsyslog configuration file
*.info;mail.none;authpriv.none;cron.none @192.168.1.10:514
Restart the Rsyslog service to send logs.
systemctl restart rsyslog
Firewall
Configure the firewall based on Rsyslog configuration you did on the previous step.
Port 514
Execute the below commands on the Graylog server to redirect the traffic that comes on port UDP 514 to UDP 1514 of Graylog input.
firewall-cmd --add-masquerade --permanent firewall-cmd --add-forward-port=port=514:proto=udp:toport=1514 --permanent firewall-cmd --reload
Port 1514
Execute the below commands on the Graylog server to allow port number UDP 1514 on the firewall to receive logs from Rsyslog.
firewall-cmd --permanent --add-port=1514/udp firewall-cmd --reload
View Logs using Graylog
Wait for a few minutes. You should start receiving log messages from the client machine when the event is generated.
Following screenshot shows the logs received by Graylog (Graylog console >> Search).
Conclusion
You have successfully installed Graylog 3.0 on CentOS 7 / RHEL 7 and configured Graylog to receive Rsyslog logs from external sources. As a further read, you can try configuring Nginx or Apache as a reverse proxy and set up HTTPS for the Graylog web interface.
