As you know, providing all-in-one server security, BitNinja protects 3000+ Linux web servers worldwide, capturing 100 million incidents a month and keeping 1.7 million suspicious IP addresses in its blocklists to protect you and your customers. Mixing that up with custom infrastructures, configurations, and software in each sometimes leads to a high load on the server.

Despite the best intentions - we always endeavor for the best solutions, and we test our novelties constantly before and after each and every release -, these things happen. Most of them are temporary, and everything falls back to normal very soon, but there are a few cases when it’s not.

We’re not letting you down this time, either. So we’d like to give you a short guide about handling and preventing these – because in some cases you can eliminate this side-effect!

Of course, the very first step should always be to ensure the root of the problem. After excluding every other possible reason, you can use this guide to handle any BitNinja-related issues.

And, of course, we’re always open to answering your questions.

Symptoms

We can distinguish the reason for the problem based on the symptoms and our logging system. When we receive a complaint, first of all, we have to know what kind of high load can we’re talking about.

There are huge differences between high CPU, memory, disk, and network usage.

Main reasons

We can often group these cases by the ‘guilty module.’

IPset

In the past, this was the most common cause of high load. While a spike in the system load is normal right after setup – it generally drops back to normal – within a few minutes. The higher load is because BitNinja stores the Greylist – and other IP lists in kernel space – and the server has to rearrange its memory for storing the IP sets. Usually, this is a relatively quick procedure.

Using BitNinja simultaneously with other programs that aggressively modify the IPset, is known to cause issues. For example, we know that Dome9 drops every iptables rule – other than its own – and as a result, Dome9 is incompatible with BitNinja, which may cause a high load in some cases.

Log Analysis

This module tends to be a bit CPU-hungry –but don’t be afraid.

Improper log rotation may cause similar symptoms. Usually, you won’t notice it at all – however, when your server generates hundreds of log entries in one second, the SenseLog module (Log Analysis) has to - in turn - work harder, which can raise the CPU usage by a few percentages. In addition, the size of the log file also factors in.

This is because the module currently utilizes regular expression matches on the analyzed log lines to protect the server against WordPress, Joomla, and cPanel brute-force attacks, Shellshock code injection, and directory traversal attacks, as well as many other potential vulnerabilities.

Suppose you notice this issue and you’re not hosting any Joomla or WordPress websites on your server. In that case, you try disabling some of the supervisors responsible for preventing attacks against the services mentioned above. You can configure the module by modifying the config.ini file in the /etc/bitninja/SenseLog folder in the Enable/Disable Supervisors section. If you would like to disable the Joomla-login filter, you should add the following line:

disabled[] = ‘ApacheJoomlaLogin’

Note 1: You can find the description of every supervisor on our documentation site.

After you have finished editing the configuration file, you should simply reload the SenseLog module using BitNinja CLI:

bitninjacli --module=SenseLog –-reload

Note 2: We are planning to make this workflow more user-friendly soon.

Malware Detection

When you first start the Malware Detection module, the system has to reinitialize inotify processes. This is a CPU and disc-intensive task and can take hours to complete. We suggest manual scanning right after you turn on this module.

Malware Detection should traverse all directories and set a flag to monitor them. We advise enabling this module during an idle period when a slightly increased load or I/O performance drop is not causing any problems. You can avoid potential future inconveniences by checking smaller directories instead of the whole /home folder.

Are you hosting many dynamic and changing files on your server and noticing the high CPU consumption? 

It’s because of the Malware detection too. The inotify process has to communicate with the kernel super intensively while the scanning runs—using only one CPU thread.

Note 3: We’re currently working on implementing multi-threading of the processes, which will accelerate the entire process.

Other cases – High network traffic

A drop in the network throughout may occur during an incoming attack - due to the nature of BitNinja protection.

What should you do?

We are always happy to help you and will be able to accelerate the process of our investigation into your concern if you make sure to send us all of the following details regarding your particular case.

• In the difficult period, all the files from the server’s /var/log/bitninja folder contain useful information for our experts.
• By sending the result of the command below executed during a high-load period, we can see the incoming connections, which will help us find out more about the nature of the problem:netstat -n --inet --inet6 
• With the output of the top command, we can see how many BitNinja-related processes are running – and how many resources they are using.

Upcoming solutions: Maldet and LogAnalysis refactoring

Our development team is working on both Malware Detection and SenseLog modules to help reduce resource consumption.

We began with custom memory allocation of Malware Detection, but unfortunately, it was not enough to obtain a definitive result so far. But… we won’t give up!

Both of the modules mentioned above will be refactored and accelerated by the end of this year with multi-thread running.