About a month ago I was dealing with a customer's complain about policy installation taking way too much time. It is a valid concern for many organizations, apparently. If you think your policy is taking too long to install, there are several things to consider.
How it is done
Policy installation is a bit more complex that some people think. Let's see what's happening when you press an Install button in the GUI:
- Policy package is saved
- If you have Management High Availability in place with automatic sync, MGMT HA synchronization is invoked. Your whole MGMT database is being collected, zipped and transferred to the standby Management Server and then unpacked. Only when it is done (or failed, depending on the conditions), you go to the next step.
- Policy Verification. Your rulebase is being checked for logic errors and shadowed rules.
- Policy compilation. Management server calls a compilation process, that collects the rulebase info, objects descriptions, security settings for IPS and Threat Prevention, Application Control, etc and prepares all necessary files to for transfer to a Security Gateway. Files are zipped for transfer.
- Now the policy files are sent to the gateway through SIC encrypted channel.
- Gateway receives the files and unzip them. Once done, actual policy installation is about to begin.
- Finally, GW is replacing existing security rules with the new one from the received package. Depending on the settings, this process may include all open connections to be re-matched with the new policy. Once done, GW reports to MGMT the success or failure
In summary, steps 1 to 5 are happening on the Management Server. Only the last three steps are happening on the Gateway side.
Why can it be slow?
There are several bottlenecks at the Management Server side: HDD access speed, RAM and CPU. All of them may slow down multiple operations with the files and compilation of the policy package.
For example, if MGMT server is also doing logging, with high amount of logs disk access will be queued, and both verification and compilation may take significantly longer time than without logging. If your management CPUs are already running high (with logging, SmartLog indexing, any other CPU intensive operations) that would slow down policy installation.
If by any chance you are running Windows based Management Server, a compulsory antivirus software will slow down policy installation enormously, scanning and rescanning again all MGMT database files that are being opened, created and changed on the way.
Management HA sync adds five to ten minutes to the process. If this is your case, you may want to change MGMT HA settings to manual sync only. My customer mentioned at the beginning of this article was using fake management server objects for logging external FWs. Every time they were installing policies, MGMT HA process tried to talk to non-existing objects, finally failing and giving up. For that customer, the main pitfall was management high availability itself.
Bigger policy takes longer time to compile. More checks, more objects to touch, bigger files to create.
File transfer from MGMT to GW may take some time if the connectivity between elements of your security system is slow.
Once the files are on the gateway, it usually takes less than a minute to install policy. Again, bigger the policy, slower the installation time. If your policy has more than 3000 rules, you may expect installation would take time.
Will it be faster with R80?
Yes and no.
For all R77.x and below it will be exactly the same process as before.
With some later R80.X version (but not R80.10, apparently), we expect that gateway will be able to receive only the delta changes of the policy package. In this case every consecutive policy push after the first one should take less time.
Want to know more?
All above is thoroughly discussed in my Troubleshooting course. If you are interested in it, do not hesitate to let me know. My email is varera (at) gmail . Looking forward to hear from you.
One problem we often see at our customer's installations is lack of write performance on the security management server (SMS). If the SMS has only one physical HDD (Smart-1 205 and 210) or poor disk i/o in virtualization, the logging volume is high and SmartLog and SmartEvent is active, you may experience nearly a complete overload of the SMS machine during policy compilation and very high load values. Sometimes it can be helpful to reduce log volume but this maybe will produce a poorer visibility of the customer's actual security situation.
ReplyDeleteBtw.: The policy installation flow process is explained in sk101226: https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk101226
Yes, there is an SK, but there are some dependencies missing there, such as MGMT HA; for example. It also requires elaboration of each step. Unfortunately, blog format is not good either.
DeleteHi, chance upon this post. Wondering if there’s a way that checkpoint can skip verification if it detects that verification was done manually prior to installation and no publish changes was made after that ‘final verification’? Also, you mentioned that R80.10 will not be receiving that delta change push, wondering what version will since it is currently at R80.10 still 9 months after this post...
ReplyDeleteno, you cannot bypass verification. also, you do not want to.
DeleteЗдравствуйте, спасибо за крутую статью, интересует один момент (может глупый конечно), но как протестировать качество коннекта между GW и SMS? (это упомянуто вами здесь: File transfer from MGMT to GW may take some time if the connectivity between elements of your security system is slow)
ReplyDeleteNetwork bandwidth between MGMT and GW is rarely an issue. In fact, I have not see such a case even once in my 20+ years with Check Point.
DeleteHowever, you can check it easily. Send a file from MGMT to GW via scp/sftp and see how fast it goes.