As you know, providing all-in-one server security, BitNinja protects 3000+ Linux web servers worldwide, capturing 100 million incidents a month and keeping 1.7 million suspicious IP addresses in its blocklists to protect you and your customers. Mixing that up with custom infrastructures, configurations, and software in each leads to high load problems sometimes. Despite the best intentions – we always endeavor for the best solutions and we test our novelties constantly before and after each and every release -, these things happen. Most of them are temporary and everything falls back to normal very soon, but there are a few cases when it’s not. We’re not letting you down this time either. So we’d like to give you a short guide about how to handle and prevent these – because in some cases you can really eliminate this side-effect! Of course, the very first step always should be to make sure, what the root of the problem is. After excluding every other possible reason, you can use this guide to handle any BitNinja related issues. And of course, we’re always open to answer your questions.
We can distinguish the reason of the problem based on the symptoms and our logging system. When we receive a complaint, first of all, we have to know what kind of high load can we’re talking about. There are huge differences between high CPU, memory, disk or network usage.
In lots of cases, we can group these cases by the ‘guilty module’.
In the past, this was the most common cause of high load. While a spike in the system load is normal right after setup – it generally drops back to normal – within a few minutes. The reason for the higher load is because BitNinja stores the Greylist – and other IP lists in kernel space – and the server has to rearrange its memory for storing the IP sets. Normally, this is a relatively quick procedure. Using BitNinja simultaneously with other programs that aggressively modify the IPset, is known to cause issues as well. For example, we know that Dome9 drops every iptables rules – other than its own – and as a result, Dome9 is incompatible with BitNinja, and this may cause high load in some cases.
This module tends to be a bit CPU-hungry –but don’t be afraid. Usually you won’t notice it at all – however, when your server generates hundreds of log entries in one second, the SenseLog module (Log Analysis) has to – in turn – work harder which can raise the CPU usage by a few percentages. Improper log rotation may cause similar symptoms. In addition, the size of the log file also factors in. The reason for this is because the module that is currently utilizing regular expression, matches on the analyzed log lines to protect the server against WordPress, Joomla and cPanel brute-force attacks, Shellshock code injection and directory traversal attacks, as well as many other potential vulnerabilities. If you notice you are having this issue and you’re not hosting any Joomla or WordPress websites on your server – you try disabling some of the supervisors that are responsible for preventing attacks against the above-mentioned services. You can configure the module by modifying the config.ini file located in the /etc/bitninja/SenseLog folder, in the section called Enable/Disable Supervisors. If you would like to disable the Joomla-login filter you should add the following line:
disabled[] = ‘ApacheJoomlaLogin’
Note 1: You can find the description of every supervisor on our documentation site. After you have finished editing the configuration file, you should simply reload the SenseLog module using BitNinja CLI:
bitninjacli --module=SenseLog –-reload
Note 2: We are planning to make this workflow more user-friendly soon.
We suggest manual scanning right after you turn on this module. When you start the Malware Detection module the first time, the system has to reinitialize inotify processes. This is a CPU and disc intensive task and can take hours to complete. Malware Detection should traverse all directories and set a flag on it to be monitored. We advise enabling this module during an idle period when a slightly increased load or I/O performance drop is not causing any problems. You can avoid some potential, future inconveniences, by checking smaller directories instead of the whole /home folder. Are you hosting a lot of dynamic and changing files on your server, and notice the CPU consumption is high? It’s because of the Malware detection too. The inotify process has to communicate with the kernel super intensively, while the scanning runs—using only one thread of the CPU. Note 3: We’re currently working on the implementation of multi-threading of the processes, which will accelerate the entire process.
During an incoming attack – due to the nature of BitNinja protection a drop in the network throughout may occur.
We are always happy to help you and will be able to accelerate the process of our investigation into your concern if you make sure to send us all of the following details regarding your particular case.
Our development team is working on both of Malware Detection and SenseLog modules to help reduce resource consumptions. We began with custom memory allocation of Malware Detection, but unfortunately, it was not enough to obtain a definitive result so far. But… we won’t give up! Both of the above-mentioned modules will be refactored and accelerated by the end of this year with multi-thread running.
Start the 7-day free trial with full functionality without spending a cent.
After the “Hello, Peppa!” zero-day botnet, our Attack Vector Miner detected another zero-day...
At the end of the last year, we made...
