Monday, February 20, 2012

VSX - no logs from some of Virtual Systems

Last weekend I was assisting my customer in migrating VSX cluster to a new HW. The migration went smooth, but there was a weird problems - some Virtual Systems on the new cluster were not sending logs to their log servers.

I am facing this problem more than once, so I guess someone else can have this too. Here are some tips, what to do.

The issue itself might be related to VS creation process, as mentioned in sk43973 and sk61545 (both solutions are about the same problem). Although the case claims the issue is fixed in VSX R67 GA, I have experienced in on R67.10

First, check you do not have any issue with cpld process on bpth sides: VSX and CMA. Check the processes are up.

Then see if VS is actually talking to its CMA. To do that, run netstat -na. If logs are being sent, you will see an established connection between CMA and VS on port 257. Detailed explanation about how to do it can be found in sk38848.

Now, this is the key point. If you have the established connections, logs are coming. Install DB on your log server and re-open SmartView Tracker client, you should now see them.

If connection is not established, something is quite wrong with the logging process. In my last case, certain Virtual System was not sending logs, if active on the first cluster member, but was sending them after a failover. CPSTOP/CPSTART did not fix the issue, but the reboot of the cluster did.

If you cannot afford rebooting the cluster, you can try killing cplogd - special daemon responsible for logging in VSX environment. To do that use "kill -9 " command from the expert shell.

So, the recommendations are:

1. Check connectivity and communication on port 257.
2. If there is traffic from VS, install DB on the log server.
3. If there is no communication between VSX and CMA, reboot the cluster members one by one.
4. If neither one of the steps helped, open a support case and describe all you did thoroughly.

Good luck

4 comments:

  1. I've seen restarting logging services resolve similar issues (though not normally with new installs). On VSX it's handled by cplogd instead of fwd.

    ReplyDelete
  2. True, this is also an option.

    I will update the post.

    ReplyDelete
  3. I have seen this a couple times recently as well. I was never able to put my finger on what exactly fixed it. But I had done most everything that was discussed here and it just seemed to magically start working. Another painful thing I ran across is what is described in sk65615 - "R75.20 SmartView Monitor incorrectly shows the status of Virtual System / Virtual Router in VSX cluster". Have you seen this issue?

    ReplyDelete
  4. Sometimes cplogd cannot fulfill a handshake with CMA's fwd. I have not seen the mentioned SVM issue just yet.

    ReplyDelete