Wednesday, January 26, 2011

Merging two SmartCenter servers, quick HOWTO

Sometimes customers want to merge two or more SmartCenter servers into a single management server. The reasons for that are usually operational. I can't say I am a fan of this idea in general, but let's face it, from time to time we have to do something we do not like.

So, here is the way to do it right.

You will need:
  • Two SmartCenter Servers - provided by customer
  • VMware workstation or ESX server - find it yourself
  • Check Point installation media or ISO files and HFA files - download them in advance from Check Point site
  • Standard evaluation licenses (optional) - ask you partner
You will NOT need:

  • tools like ofiller or Confwiz. Wee will only use one classic Check Point DB utility

Things to take into consideration before you proceed:

  • SMC server versions
If servers are on different versions, you need some additional steps on the way. The best is to have them on the same version and HFA level, if possible.
  • Local users and user groups. 
They cannot be merged. Ones on the target machine will remain. The source machine's users and groups will be lost. Choose your source and target wisely, because you will have to recreate these settings manually.

Users and usergroups are merged by using fwm export and fwm import commands on the source an target SMC servers respectively. Thanks to Tore Solberg for pointing this out on LinkedIn.

  • VPNs

You will have to tune both Site to site and Remote Access VPN definitions during the migration. basically, keeping VPNs intact in many cases even more difficult then taking care of the users and groups.

Migration steps:

  • Backup everything. Then backup again.
  • Prepare DB export files on both SMC servers with upgrade_export
  • Install two new SmartCenter servers on your VMware. Choose any IP address you want. Use exact versions of your actual machines, so you could import DB later without any issue.
    License them with evaluation licenses, if you cannot use actual production IP addresses.
  • Import two databases on both VMware machines using upgrade_import. Check they are running, and you can connect to them with SmartDashboard without any problem.
    It is important you have both servers functional before you start messing around.
  • Now it is a good time to decide who will be the target machine, and who is to be source.
    It is all about complexity. Less FWs managed, less users defined, less policy packages configured. In some cases customer says you in advance, which SMC will be decommissioned. Choosing the right approach is important because you will have to redefine manually all local users and user groups from source SMC.
  • On target SMC prepare the files. You will need to copy Objects_5_0.C file to a certain folder. Then use cm_merge utility to export all policy packages you need. Usually there are more of them, so consult with your customer about things he wants to keep.
  • Copy these files (DB file and policies) to the target SMC. Use the same cp_merge to merge the objects. Then use it again to import all policy packages. Easy, right? By the end of this step you should have in on your VMware one operational SMC with merged objects databases and all necessary policy packages. Reminder: users and usergroups should be created before this step.
  • Export DB from this machine with upgrade_export utility. On this point you are done with labs and simulation, it is time to change your production systems.
  • Import DB prepared in the previous step to your target production server. Run regression tests. Its own FWs should continue sending logs. SIC should work, you must be able to push policy on those FWs.
  • Now it is time for more interesting task. You already have FW objects from your source SMC, but they are not responding. Reset and re-establish SIC with them. Voila!
  • Take care about VPNs. Tune communities, change all needed parameters. It might not be as easy as it sounds, but it is not different from building a new VPN system, so you will manage.
  • Once you have all GWs operational, all VPNs up and all logs coming, it is time to clean your database. Remove old source SMC object from it and, if necessary, double objects.
Please let me know if you have any further questions.

Stay tuned.


  1. very nice howto.
    my 5 cents:
    If you define the target SC like a secondary SC on the source machine and push policy you can do the work without SIC reset.

  2. sorry, but no. the fingerprint of the new SC is completely different. do not forget, we are talking about merging two different SMC servers, not just migrating on a new IP address

  3. hmm... Ok, I am taking i back you right.

  4. Hi I am peforming exactly the same procedure as you mention above, I can merge the objects file without a problem on the target SC, however my policy import fails with a 'segmentation fault'. Have you come across this before?

  5. Hi Paul, I had the exact same problem. Turns out I found that the user I was using was the wrong user to access the database. Try changing the user, use the GUI user to do the merge.

  6. Me too, I have the exact same segmentation fault issue, but in P1 environment only, it is all working fine with standard Smartcenters...

  7. Fred, this procedure is not supported fo P1.

  8. Hi Valeri,

    Thanks a lot for this article.

    I've just tried the procedure and it works great... but I do have a problem.

    The management that I'm trying to merge into the other one has VSX objects in it, and when they get merged into the "master" using cp_merge the objects are all wrong. If I try to edit them I get:

    "Internal error: Cannot get object XXXX from table vs_slot_objects"

    Any ideas on how to succesfully migrate this?


    1. There is one limitation to that. You cannot merge databases, if they have VSX defined. Remove VSX, merge, recreate.

  9. Thanks for the info Valeri!

    Do you know if there are also limitations with ClusterXL? I also get errors with that.

    cetcfw001 : Validation error in field \'Number of cluster members\' of element #1 at object \'cetcfw001\' @ \'Network Objects\' --> The referenced object \'cetcfw001a\' from table \'network_objects\' does not exist in the database

    cetcfw001b : Validation error in field \'Cluster Object\' at object \'cetcfw001b\' @ \'Network Objects\' --> The referenced object \'cetcfw001\' from table \'network_objects\' does not exist in the database

    cetcfw001a : Validation error in field \'Cluster Object\' at object \'cetcfw001a\' @ \'Network Objects\' --> The referenced object \'cetcfw001\' from table \'network_objects\' does not exist in the database

    Where cetcfw001 is the Cluster object and cetcfw001a/cetcfw001b are the actual gateway nodes.

    1. Hi Pere!

      No, I am not aware of such limitation. It looks like some of the objects were not merged. Is it a VSX cluster or just a physical one?

    2. Hello Valeri,

      I have the same problem when merging databases with cluster objects with cp_merge. Is there a chance to merge the databases with cluster objects?

      Yevgeniy, Frankfurt

  10. Hi Valeri!

    It's a standard Cluster, not VSX.

    However I am merging mgmt1 (which has standard Cluster) into mgmt2 (which has VSX cluster).

    Maybe cp_merge doesn't like VSX in any of the mgmt stations.

    I'll try a few more things to see if I can get it to work.

    Another question if you don't mind: what's the easy way of backing up VSX configs/objects and then restoring it all without having to do it all manually? Do you have any document reference on how to do that? vsx_util seems only useful for exporting things, not importing!

    Thanks a lot Valeri for your help!

    1. Hi again!

      If there is one, I do not know it. I have heard, CP is currently testing a tool that would allow you to create multiple VS objects through CLI. Theoretically, that would allow you to dump info into a text file and then use it to re-build. I have not seen the tool, and it is not yet officially released.

  11. Thanks Valeri!

    And I guess no tool for being able to add/remove static routes to VSX via CLI either, right?

    I find VSX (which is new to me) quite a manual approach to things!