Monday, November 6, 2017

Kernel debug Best Practices or "Why "fw ctl zdebug..." should not be used"

Over last several days I have seen rapidly growing amount of posts at CPUG and CP Community where "fw ctl zdebug..." command was mentioned, used and advised.

Although some of you already know my position for the matter, I have decided to write a post about the growing custom to use zdebug instead of employing full fw ctl debug mechanism.

Kernel debug in general

Check Point FW is essentially a Linux-based system with a kernel module inserted between drivers and OS IP stack. If you do not know what I am talking about, you may want to look into this post with an explanatory video for the matter.

Extracting information about kernel based security decisions is rather tricky, so Check Point developed an elaborate tool to read some info about various FW kernel modules actions.

In a nutshell, each kernel module has multiple debug flags that force code to start printing out some information. I have numerous posts in this blog explaining different flags, tips and tricks with kernel debug and also providing links to CP kernel debug documents.

Debug buffer

It is important to understand FW kernel is always printing out some debug messages. For most of the kernel modules, error and warning flags are active, and the output goes to /var/log/messages by default. This is not practical for debug, so before starting kernel debug, an engineer needs to set a buffer which would receive debug output instead of /var/log/messages file.

To do so, the following command is used: fw ctl debug -buf XXXXX, where XXXXX is the buffer size in KB. The maximum possible buffer today is 32 MB, but I advise my students to use 99999 to make sure they get maximum buffer possible anyway.

Kernel can be very chatty, so having a bigger buffer would ensure less kernel messages being lost.

Debug modules and flags

FW kernel is a complex structure. It is built with multiple modules. Each of the modules has its own flags. One can run a single debug session with multiple flags raised for several modules. To raise debug flags, one use one or several commands of this type:

fw ctl debug -m (module name) (+|-) (list of flags)

It is essential that + and - options allow you to raise and remove flags on the fly, even during an already running debug session. List of modules and flags can be found by the first link in this post.

Printing info out of buffer

Raising flags is not enough, as to get information, you need to start reading buffer out with this command:

fw ctl kdebug -f (with some options)

There will be A LOT of information, so never do this on the console. Use SSH session or redirect to a file.

Stopping debug

Once you collected the relevant info, you need to reset kernel debug to the default settings, otherwise you FW will continue printing out tons of unnecessary info. To do so, run

fw ctl debug 0

What is fw ctl zdebug then?

fw ctl zdebug is an internal R&D macros to cut corners when developing and testing new features in the sterile environment. It is equivalent to the following sequence of commands:

fw ctl debug -buf 1024
fw ctl debug (your options)
fw ctl kdebug -f
-------(waiting for Ctrl-C)
fw ctl debug 0

Why is this a problem?

If you are still reading this post and get to this line, you probably think zdebug is a god sent miracle. It simplifies so many things, it is the only way to run debug in production environment! Right? 

Wrong. To make it plain, here is the list of problematic point with this way of doing things:

1. The buffer is way too small. Lots and lots of messages might be just lost because buffers does not have enough room to hold them before read.
2. It is not flexible enough. Running debug in production requires lots of consideration and certain amount of caution. After all, you are asking FW kernel to do extra things, lots of them. The best practice is to start with a single flag or two and expand area of research in the fly trying to catch an issue. This is impossible to do with fw ctl zdebug macros.
3. It is too simple to use. You could say, what a funny argument. Yet, let's think about it. To master kernel debug as described above, one has to understand kernel structure, dependencies, flags and modules. You don't have to do any of that to run fw ctl zdebug drop, and many people do jsut that. 

And guess what, this is also the simplest way to bring your busy production FW cluster down. So no, do not try this at home or at your place of work, if job security is important for you. 

Support CPET project and this blog with your donations to 

Monday, October 30, 2017

Check Point researches dissect IOTroops Botnet

Check Point security research team has recently posted an elaborate and impressive report about IOTroops botnet.

The details and depth are fascinating. Highly recommended to read.

Support CPET project and this blog with your donations to 

Monday, October 16, 2017

Check Point is finally fixing issue with CCSM continuity, somewhat

At the beginning of the year I was posting about CCSM continuity hiccup (1, 2, 3).

In a nutshell, there were two main issues:

1. CP did not managed to let people with expiring CCSM re-certificate in time and did not provide any graceful extension.
2. Due to a clerical error, some people were getting 4 years of CCSM certification while others were having only 2.

I have taken liberty to contact Check Point Education Services managers and to discuss the issue. There was a quite long threat with dozens of emails back and forth, and finally in February Check Point Certification manager has publicly acknowledged the mentioned issue.

However, Check Point did not provide any solution for the matter at that time. Moreover, they have rejected my private proposal to make a one time correction of CCSM validity for all certified specialists from 2 to 4 years, that would resolve both continuity and consistency problems in one shot.

This situation has undermined public trust and appeal of having the highest certification level with Check Point. Also, new partnership program does not require having any CCSMs, even for support partners. It seemed to me that the company did not have any solid strategy to develop an advanced certification at this point.

In my humble opinion, CCSM is nothing but a stripped down version of older CCSE Plus certification, and cannot be even compared with flawed but very challenging CCMA exams that Check Point eventually failed as well.

That said, there are signs the company is finally coming to its senses and trying to reverse the situation.

At the end of September all CCSMs have received an email from Check Point Certification manager Jason Tugwell granting an extension of CCSM status for all people having their certificates expired between the beginning of 2017 and up to end of March 2018.

Everyone in this group, including those whose certification has lapsed already, are granted extension of CCSM status till end of June 2018.

Although this is not making right 2 years of certification versus 4 years for some, but it is still covering the continuity lapse, under condition Check Point Education Services will be able to develop and release new CCSM course and exam till the second quarter of 2018.

Just to make it clear, the extension notice should be received by all CCSM professionals whose certification expires between January 2017 and March 2018. If you are one of them but did not receive such notice, please talk to account managment at Check Point to fix it.

Support CPET project and this blog with your donations to 

Wednesday, September 6, 2017

Your ultimate landing page for Advanced Tech Reference Guides

Check Point SecureKnowledge database is vast. It has hundreds of thousands of articles and documents. Sometimes, it takes a bit of an effort to find there what you are looking for.

Yet, it sometimes yields fantastic results. Here is something you may want to add to your bookmarks: a landing page for accessing ATRGs - Advanced Technical Reference Guides.

So far, it has links to 36 ATRGs. Whenever you want to learn a feature in depth, this is something you want to visit.

Also, it now has three new documents:

Many thanks to Sergei Shir for sharing this information.

Support CPET project and this blog with your donations to 

Monday, August 7, 2017

Capsule Docs on Mac? Forget about it...

Last year I was writing about my rather unpleasant experience around Capsule Docs on Mac. It is time to add another chapter to that story.  

I have made yet another attempt to use the tool on Mac. With my 10.12.6 Sierra machine it fails even more miserable than before. With the latest client (still Alpha, mind you!), I cannot even open a document.

Although I am logged in and even can open the same document on Windows with the same credentials, I am getting "Insufficient permissions"...

How hard can it be, really? What should happen for Check Point to start getting Mac user seriously?

In case you ask, the only reason for me to even touch Capsule Doc Viewer is that Check Point Education Services discontinued paper courseware, forcing both students and instructors to use e-kits with Capsule Docs protection. I will address this subject later on.

Support CPET project and this blog with your donations to 

Sunday, July 30, 2017

CPET session 3 - video is published

Thanks all who could join.

The session subject is Kernel Debug, best practices

CPET project relies on your support. 
Participate in the talks and help us with your donations to 
Follow us on Facebook and Twitter.