Hardware diagnostics ordeal with 21400 appliances

Through a long and painful support process with one of my customers we have had to run hardware diagnostic tool on 21400 appliances with optical 10GB NICs.

It is supposed to be an easy tool to use. There is SK97251 describing the usage, and even the Administration guide for the tool.

But guess what?

First, the built-in R77.10 diagnostics did not work. When loading, it was crashing with kernel panic message. So we have resorted to the USB-based tool, as described in the SK article above.

The main point of the exercise was to check if any of the optical interfaces has a problem. We have spent 2 hours trying to make those tests.

The main problem is that the tool does not seem to know anything about these 10GB ports, and not so much about regular Ethernet ones either.

One is supposed to plug a loopback adapter to the tested port. As the tool is using non-standard interface names, it is blinking the port to test before proceeding. It seems the tool is calling interfaces almost randomly, not in accord with standard Gaia interface names.

Interface eth1-01 became eth0, eth1-02 - eth2. Port eth1 was not ever blinking, same for eth4 and many other interfaces on the way. Finally, eth1-08 became eth20. The funniest part is, if you are unable to identify an interface and were not fast enough to skip its verification, the whole test fails.

Obviously, no optical ports were ever blinking during the procedure.

I suspect our problems are actually described in the quoted article as

• On 21000 Appliances with SAM card, testing of the SAM card is not supported.

It is not really clear to me if these 10GB cards are in fact SAM or not. In the catalog they are marked as "acceleration ready", whatever it means.

I am pretty sure that Check Point support engineers were never running HW diag tool on 21400 with optical cards. I suspect QA never did either. I really wish they did. That would save me yet another Saturday spent in a datacenter. 

10GB bonded interfaces with jumbo frames, some notes.

Today, with new Check Point appliances, one can design a decent DC firewall. With multiple 10GB interfaces and bonds, one can fit Nexus network core throughput easily.

There is one caveat though. There are multiple known issues with 10GB interfaces on Check Point appliances, especially concerning bonding.

Firstly, there is an issue with LACP if jumbo frames are in use, described in sk86980. I have to say the solution is incomplete. For example, i mentions only three particular software versions, and mine was not listed there. 

Secondly, there is a stability issue with jumbo frames, mentioned in sk99113. 

Luckily, driver update from the second SK fixes LACP with jumbo frames too.

Amount of different versions that need patching (from R75.40 till R77.10, including quite recent R75.47) for the matter clearly shows that Check Point developers were not too concerned about jumbo frames for a long time. It looks like high end DC testing scenarios were not part of the regular QA test cycles. 

I can only hope this will change in the future.

Extending logging partition on your management station

One of the major challenges when managing Check Point security systems is about log storage. If you are not careful and/or proactive, you can run out space eventually.

It does not matter, how much free space you plan for your logging partition, one day it may not be enough. If this day comes, what are you going to do?

Of course, you can always delete some old logs or just get a bigger box. If latter, then you have to run through a long migration procedure.

What about adding a new storage? There is a way to change Gaia partitioning with LVM management tool, as described in SK95566.

There is even an article about extending logging partition to a new disk drive. Take a look on SK94671. It describes VMware case, but can be in general applicable to any additional HD.

CPUG 2015 - let's make it happen

Do you miss annual CPUG conference? I do. Today, with the new owner of CPUG.org, it is time to plan it again.

We as CPUG community shell make it happen. Here are some questions for you, please take a minute to respond:

1. Did you participate in CPUGcon in the past? Are you going to participate in  CPUGcon 2015? If yes, will you invite and encourage to participate you colleagues, friends, clients and partners?
2. What would be ideal location for you? Mind we are talking about European event.
3. Could you help CPUG to find the best suitable location? We would like to accommodate around 80-100 people. We might need one or two conference rooms, with some reasonable priced hotels around and not too far from an airport. Any thoughts?

All, please feel free to share your opinions and suggestions. 

Static routes issue with R75.47 on 13500 appliances

I am in the middle of preparing FW migration for one of my favourite customers, who wants to replace open servers with 13500 to benefit multi 10GB connectivity.

One unexpected thing on the way was configuring new static routes on these appliances after clean installation of R75.47. I have tried both CLISH and WebUI, but routes were not in effect. I could see routes in the WebUI but not on clish with "show routes" and not on expert with "netstat -rn". It was like routes were never making it to IP stack. I have seen that on early Gaia versions, but on R75.47 it seems impossible, right?

It took some time to triple-check all the settings. It was all properly configured, except for a default GW that was never  defined. I have punched it in, still unable to see any static routes defined on OS level.

Only after installing a driver fix from sk99113 and consecutive reboot all static routes start appearing and working.

I have checked this situation on the second appliance. Strangely, static routes were not appearing even after drivers installed, if default GW was not defined.

So to configure static routes properly on 13500 running R75.47, one has to have two things done:

1. Drivers from sk99113

2. Default route in place

The only explanation I have for the matter is that 13500 with R75.47 was never a part of QA cycle. 

R77.20 gateway forward compatibility, SIC!

Something revolutionary is happening with Check Point releases lately. For example, it was never supported to have a Firewall software version higher that Management server's one.

Now it is no longer an issue. Quoting from R77.20 release notes:

CPUG is back to life

Great news for the community: CPUG is back to life. Read by the link if you are interested to know the details.

Thanks to Netanium and Eric Anderson for their efforts.

CPUG will be back, as we all hope

Hello, world.

I do not want to be too optimistic, but according to some sources, CPUG will be back very soon. No more details are available for now.

Setting proxy ARP for bonded interface

One of the interesting challenges is about setting proxy ARP on a FW  bond interface to facilitate manual NAT rules.

There is a very good SK article about proxy ARP configuration that covers both physical firewalls and VSX. There is only one problem with that: it is not applicable to bond interfaces. HA or LS bond type, does not matter. The main issue is that one cannot use either MAC address of the NICs in bond, as frames may sometimes go through another physical link.

So is it absolutely impossible to use a combination of proxy ARP, manual NAT rules and bond interfaces?

Not exactly. Here are several steps that you need to do.

1. Set up VSX and not a physical FW. Even if you do not have VSX licenses, physical FW license will allow you to run a single Virtual System. That is all you need.
2. Instead of connecting your bond interface to the Virtual System (VS), define a Virtual Switch (license will also allow that) and connect it to the bond.
3. Create a virtual link (warp) between VS and the Virtual Switch.
4. Go to CLI, check MAC address of the warp link and use it in local.arp file. Do not forget, you have to set unique ARP entries for each cluster member.

Problem solved.

CCSM Q&A

Uri Lewitus has commented on my previous post about CCMA/CCSM certification, and his comment is valuable enough to be quoted as a separate post. This is what he says about CCMA and CCSM:

Q: What happened to CCMA?
A: The CCMA lab is no longer available. It is being replaced by the online, multiple-choice written exam for the CCSM. The new exam, when available will be the 156-120.77. The new CCSM should be available Late September, Early October.

Q: Which is the prerequisite for CCSM?
A: CCSM will have a CCSE pre-requisite.

Q: What is the cost of the CCSM exam and where do I take it?
A: CCSM exam cost has not yet been set.

Q: How do I pay for the CCSM lab?
A: Payment will be managed through Pearson VUE voucher system.

Q: Does the CCMA and/or CCSM certification supersede the CCSE certification?
A: Yes.

Q: What is require to recertify/renew CCSM?
A: CCSM renewal consists of maintaining current CCSA and CCSE certifications. Certification expiration of two years or more automatically results in expiration of the CCMA older than four years.

CCMA is no more or Should I rename this blog?

Have you seen the certification page on CP site lately? If you have not, go take a look.

The funniest part is CCMA is no longer on the list. Although CCMA page still exists on the site, there is no link to it from the training portal anymore. Apparently Master Architect certification is to retire. New CCSM (Security Master?) will come instead. There is not much information about this new certification, but according to Don Paterson's post in LinkedIn, this new exam will not have the lab part.

Now a have a bunch of questions:

1. What happens to CCMA that is still valid for some years? Will it be converted to CCMS eventually?
2. When CCSM will be out?
3. Why Check Point keeps revolting certification process every 5 years? Are they even interested in having something as serious and respected on the field as Cisco certification route?
4. Should I rename my blog any time soon?  

Gaia: how to cron binaries depending on shared libraries

I have recently came across an issue that scheduling custom scripts with cron may not work properly on Gaia for binaries that require shared libraries to run.

For example, sendmail uses libProdUtils.so and requires access to this lib file when scripted. Path is defined for your bash shell, but not for cron. So if you make a simple script like this one:

#!/bin/bash 
/opt/CPsuite-R77/fw1/bin/sendmail -t 1.1.1.1 -m /var/tmp/testmail.txt

to send an email on a particular even, it works perfectly from bash CLI but fails when running through cron job.

To fix it, you need the script to call shell parameters explicitly. To do it, add source /etc/bashrc before executing any other command:

#!/bin/bash 

source /etc/bashrc
/opt/CPsuite-R77/fw1/bin/sendmail -t 1.1.1.1 -m /var/tmp/testmail.txt

By the way, Check Point own sendmail sucks big time and can only run when using message file and not as just a CLI command, even with 100% correct syntax. But this is something for another post.

CPUG - message from Barry

Barry J. Stiefel has reach out to me the last night. He has asked me to share with you all his side of the story. Here is what he has to say to us:

"I am unable to re-open the discussion board while I am still the owner. Negotiations over a possible sale continue... it is a forced decision.

Some background: I was recruited into a full-time career position with a major Check Point competitor and they are requiring that I divest myself from CPUG. I have already started the job and they want it all done now. That’s what’s forcing all this."

I have nothing to add.

Reminder: sign petition to save CPUG here, please

Petition to save CPUG

Hello all.

If you care about CPUG future and existence, please sign this petition:

Thank you.

---------
Update. If you cannot open the document for editing, just leave a comment here below, I will add your details to the petition as soon as I can. Sorry for this inconvenience. 

Can we save CPUG?

Wow, I never through I would write something like that. But it is happening, now.

As you probably know already, CPUG is down. Barry Stiefel shut down the site the last week. I do not want to discuss why he did that, it's pointless. It is done.

Forum that was arguably the best independent technical board around Check Point is off. Resource with (quoting Barry's sales pitch message) "80,526 immediately useful technical posts.., 18,664 unique discussion threads, 26,208 currently active members..." has vanished out of existence.

We all know Barry for years, and I believe most of us appreciate his efforts for running that board and promoting it. We are also grateful to Dameon Welch-Abernathy, who's phoneboy knowledge base was the foundation for CPUG.

Barry has a right to get paid for his effort. But what about these 26208 active members? What about people contributing to this site? What about Dameon? He never asked for a dime from Barry when handing over the data.

There is twist here. Someone owning a public free resource wants to shut it down. He does it without warning, without any attempt to reach out to the community of his users and discuss his case. He just shuts it down, willing to collect offers.

Offers for what? Public knowledge base he "owns"? Engine? Trademarks?

After years of using, nurturing and developing this site, what can we do as a community? Wait till a new sugar daddy comes around and buys it off Barry's?

Is all this even real? I try to shake a feeling this is just a practical joke of some kind.

Can we do something? Can we start collecting donations to buy off the site or at least its content? Or should we look for a sponsor? Seriously, what can we do?

UPDATE. If you care about CPUG future and existence, please sing this petition.

R77.10 MDM clean installation fails in the lab

From time to time I am hearing complains about failing Multi Domain Managemet lab installations with R77.10.

Basically, after seemingly correct installation of MDM machine on VMware one cannot connect to MDM with Smart Domain Management tool. mdsstat command output is completely empty. mdsconfig command shows part of initial configuration wizard and then goes down with a segmentation fault. Management machine remains not configured, no matter what you do.

The reason for this is that it is not enough space in the root partition for MDS to be configured correctly. You have to configure at least 20 GB for your system partition to succeed with the installation.

Strangely enough, this is not mentioned in the Release Notes document. RN states 3.5 GB to be the minimal requirement for /root. That is true for any other installation but Multi Domain Management.

It is also not clear to me why this issue occurs in the first place, as MDM does not really use that much space anyway.

I hope Check Point fixes this soon, at least by amending the Release Notes. Before that, make sure your MDM virtual machine has enough space configured for the system partition. 

FW module is lost after reboot, analysis

Some of my colleague have experiences a strange failure on Gaia-based Check Point appliances lately.

On certain point, after reboot, FW module is not accessible and can only controlled via physical console connection. It comes up with some weird initial policy that does not allow HTTPS, SSL and/or SIC anymore. You can unload it with "fw unloadlocal" from console only. If one runs "fw stat" command, the message reports failure to connect to FW.

I have analysed the issue and found that it is related to the fact /etc/hosts file is missing the host entry for the FW.

The scenario is now clear for me. This only happens when you remove or disable an interface that was used to define MGMT IP address during the first time configuration wizard. Gaia is generating /etc/hosts automatically, and if management interface is removed or changed, hosts entry associated with the first NIC is also removed. After reboot OS cannot communicate with FW anymore, and the module connectivity shuts down completely.

To fix this, after re-defining management interface go to hosts configuration in WebUI and make sure the new management IP address is properly defined there with the module hostname. Same can be done from CLISH. Do not try to edit the file form bash with VI, this will not work.

I did not manage to find any SecureKnowledge entry for this scenario.

Notes about sync redundancy

During the last Advanced Check Point Troubleshooting course I have been asked about best practices to build sync redundancy with Gaia.

The question is not a simple as it sounds. The classic textbooks for ClusterXl recommend using two or more independent synchronisation interfaces marked as First and Second Sync. Although it was true for older versions, R7x changed the play.

Sk92804 "Sync Redundancy in ClusterXL" clearly states using multiple sync interfaces is obsolete. The new best practice is to build a bond interface defined as sync.

Now simple, you say? Not really. Using bond interfaces with Check Point is tricky. There are at least three SecureKnowledge articles that you should keep in mind, mostly for CP appliances:

• State of Sync interface configured on Bond interface is 'DOWN' for each Virtual System
  Solution ID: sk100450
• SecurePlatform / Gaia OS crashes on 12000 / 21000 appliance during configuration of Bond interface
  Solution ID: sk69442
• Incorrect count of Bond slaves in use after physical link down
  Solution ID: sk98160

Each one of them requires a fix. Only after three support fixes your sync should be fine.

Kernel debug flags revealed

As I have been advised yesterday night, Check Point has published another extremely interesting document: Kernel Debug Flags. It is a comprehensive list of all kernel modules (chains) of R77 and debug flags for them.

Anyone dealing with kernel troubleshooting may want to download this at once.

Sergei Shir, Check Point TAC engineer and SecureKnowledge content developer is responsible for this brilliant material. Thanks, Sergei, for writing this document and sharing it with the community.

Distribution note: Although the document is classified as "Restricted", it is available for anyone with a valid User Center account. Sergei has personally asked me to share this document with the community.

Forwarding Management logs from CMAs to CLMs

If you only log your GWs to CLMs and not to CMAs, it is not exactly convenient having Management audit logs still residing on CMAs.

sk27042 is addressing this matter, but it is grossly outdated. Here is a procedure to forward audit logs to CMAs that works for versions R75.40 and up:

• Make sure that the CMA is not specified as a Log Server for any Security gateway. If it is, these Security gateways should be reconfigured to redirect their logs to somewhere else (for instance to the CLM). 
• Use GUIDbEdit, connect to CMA in question, under "network objects" find . In the object properties, find log_server parameter and set the value to false. Then find use_loggers_and_masters parameter and change its property to true. Save DB and exit GUIDBEdit.
• Log in to CMA with smartDashboard and open CMA object, then go to Logs tab. 
• If the settings there are greyed out, change settings to control the log settings using SmartDashboard (press “here” link in the tab). Set up primary and secondary log location as required. 
• Install database on all MGMT objects.

Log into CLM with SmartTracker and check you now have Management logs coming in.

iPad Document security - follow up

Some people from Check Point are still sending me private messages concerning my post about document security iPad app. Some even try to make it personal. Some others are just interested in getting to the bottom of it.

To both groups: I did not make it up, and I have honestly described the issue. Some of you even have the detailed step by step explanation of my experience.

I am sorry if you are upset by that.

I could give a hand to second group to reproduce the issue. I am still not ready to do in on my personal iPad, because I am using it for business purposes, and I have to have it operational. Any other option will be carefully considered.

My door remains open for you.

Transparent Kerberos SSO with multiple GWs

Check Point Identity Awareness is a neat feature, especially with browser based transparent SSO authentication. It is also a challenging one. There is a lot of configurations to do on AD side, and that is not a strong domain for some of FW administrators. For example, I specifically started working with Check Point to be as far away as possible from a turmoil of Windows administration.

Jokes aside, there is something that Check Point documentation is not covering clear enough.

With Kerberos, one has to configure Kerberos Principal Name with a use account. Identity Awareness admin Guide is covering this point fairly well, on pages 58-59 (R77 version of the document). There is a caveat though. The document is written under assumption there is only a single FW or clsuter enforcing user identity with Kerberos. Ktpass command should how to map Kerberos parameters to the user account in the document are only working for a single portal URL.

What if one has more than one GW? Ktpass is no use here. Instead, administrators have to edit servicePrincipalName with Multi-valued String Editor to add multiple URLs there to enable IA working for the same user through multiple Identity Awareness enabled gateways. To simplify the config, just refer to this screenshot bellow.

Great collections of Check Point "How to" links

If you have not seen it yet, there is a great collection of "how to" solutions and guides in SK65385. It has been last updated 14th of March 2014, and I can guess, Check Point is adding new entries there from time to time.

If you do not know, how to start with some complex issue, that would be a good place to start. As complexity of SecureKnowledge grows fast, we need such high level references.

Great job, SecureKnowledge teem, great job indeed!

Stay away from Document security app on iPad

I have had to install Check Point Document Security application on my iPad the other night. I did not want to, but Check Point guys have sent me some secured doc. So I had to. Big, big mistake.

This morning my iPad was fully charged, this afternoon the battery was completely drained. Considering it was just lying on the table half a day, that was odd. Obviously some app was stuck using CPU. So I have managed to charge it again to around 15% and started to kill applications one by one.

When I was trying ti kill Document Security, iPad crashed. I have charged it again, same drill. I could not close this bloody app! More, I could not charge it to a decent percentage anymore. Once charged to 10%, iPad was booting, then some app was eating all the power again.

It took me some tricks to finally kill it. Guess what? Once I have done that, the battery percentage jumped from 9% to 23%.

I guess there is just a few people writing Apple apps in Check Point. I would sincerely ask them to do some QA. SecureKnowledge app is great, but it is crashing with iOS 7. But at least you have other means to get your support cases, one can access them from Safari. With document security one has to use an app, there is no other way around.

So guys, please stop releasing half backed good, that gives a wrong impression. Pretty please...


------------- UPDATE--------------

I have been contacted by Check Point concerning this issues. It seems they could not reproduce it, even after my deliberate explanation of my case. And guess what, my SecureKnowledge app is not crashing anymore. That's a miracle.

3D Security report tool is now fully integrated into R77.10

If you have been following my posts about 3D Security Analysis Report tool, there are good news. Check Point has made an effort to integrate the tool completely with R77.10 release.

No more packages to install, just fire up the version, open Event Analysis GUI and check 3D Security Analysis Report under View options.

You will still need MS Office installed on the machine generating reports.

VSX R6x to R7x migration tips

As you all know, VSX R65 is out of support, and VSX R67 is about to be. Check Point is recommending customers to move to R7x versions as quick as possible. Here are some tips about the migration procedure:

1. Remember - there is no upgrade in place. To "upgrade" VSX one has to re-install with the clean target version and to run vsx_util upgrade/reconfigure on MGMT part.
2. Run lab trials of vsx_util part of operation. In every second case of my VSX upgrades MGMT part of migration fails or hangs. If this happens to you in production, you are doomed. Replicate your SMS or MDM in the lab and run vsx_util upgrade. Correct all the error on the way, if necessary.
3. Use multi-step upgrade on MGMT side, if you skip a version. In my practice, running vsx_util upgrade R67 -> R75.40VS and then again R75.40VS -> R77 is much safer than just going all the way up. In most of the cases vsx_util just hangs for half an our and then fails, if version is skipped.
4. Take extreme caution if you are going to replace your old cluster with new HW. In case of replacing open servers with Check Point appliances and vice versa one has to run vsx_util change_interfaces to rename all VSX interfaces before pushing configuration with vsx_util reconfigure to the new machine. In many cases you can successfully rename interfaces with the script, but old interfaces are not removed. If this happens, open VSX cluster object with SmartDashboard and remove them manually before running vsx_util reconfigure.

And the last tip. If you are reading this, and it does not make any sense, read through VSX administration manual and upgrade guides.

MDM and VSX are my favourite products with Check Point, but they are also the most delicate and fragile ones. Any mistake in the process can cost you dearly.

If you are interested, I am inviting all readers to my classroom to MDM and VSX training. We discuss the architecture of both products, best practices with them, maintenance and upgrade procedures. To get more information, send me a personal message here or just use our Training Center enquiry form

Check Point Quick Tips blog

There is an interesting blog about Check Point tricks: "Check Point Quick Tips by Kishin Fatnani", check it out.

Kishin is a great Check Point trainer and engineer. His training and consulting centre is based in Mumbai, India.

Palo Alto acquires an Advanced Threat specialised startup

It has been announced that that Palo Alto has acquired Morta Security, a company specialised in APT. It is the second most interesting acquisition announced this year in the field of APT, after FireEye has acquired Mandiant for over $1 billion, reportedly.