Greenfield Technologies

Your Progress OpenEdge Database Expert

Traps and Kills

From "A DBA's Guide to Databases Under Linux" ©2000 Syngress Media
ISBN 1-928994-04-0

"Hung" Users, "Runaways" And Other Psychos

The simplest (and safest) way to handle _progres processes (self service user sessions) needing some sort of external attention is to do nothing.

If that isn't an option then in all cases your next step should be to have the user slowly back away from the keyboard keeping their hands in sight at all times. This (hopefully) prevents any problems arising from simultaneous conflicting attempts to remedy the situation. Then your best bet is kill -2 (SIGINT). Kill -2 simply raises the STOP condition in the target 4gl procedure and causes the -p startup procedure to be rerun. The process won't die but it won't be doing anything bad either. And if the problem is something along the lines of "I'm hung" the user will probably get their session back without having to go through a UNIX login again (if it's a network or hung PC problem you're out of luck).

If kill -2 isn't what you need - perhaps because you limit users to one login each and there is a network problem so the idle session is blocking access - then you should still use kill -2 to get the process to a stable state. Once it is stabilized (the .lg file says that any transaction backout is complete and it is no longer accumulating CPU time and performing IO operations) then kill -1 (SIGHUP) should very safely terminate it just as if the user had shut off their terminal. Kill -15 (SIGTERM) has much the same effect.

Using a series of "kill" commands isn't a good idea unless you put plenty of time between them and monitor both the .lg file and the process status. If you interrupt Progress' cleanup routines and the interruption to an interrupt isn't handled correctly you could easily crash the db.

Trapping signals with the shell "trap" command is a bad idea. It will defeat the proper operation of these signals and lead to a very unstable system. If you shell out of a _progres session, type "trap" and get any output back then your system is at risk. The most common problem in this scenario is "runaway" processes -- processes that attempt to consume all available CPU. If you have traps get rid of them.

Kill -8 (SIGFPE) varies in behavior depending on the release of Progress. In some releases it just acts like an unhandled signal - that is it dumps core and the process dies without any serious attempt to cleanup which is essentially the same as kill -9. This is bad because you could be holding a latch and thus bring down the db. In more recent releases (including the Linux 8.3b release) it cleans up as if SIGHUP were sent but also creates a procore file.

Kill -16 (SIGUSR1) in recent releases causes the process to create a procore file and a protrace file without otherwise impacting the process (the process keeps on running). This is a very neat feature although it's not well supported (it isn't documented anywhere and TS is often ignorant of its existence.) The protrace file is very handy -- it contains a 4gl stack trace including line numbers showing where the process was at the time that it was signaled. This is very handy for tracking down what a user was (really) doing when they became "hung".

Kill -9 (SIGKILL) is never a good idea. It also isn't ever necessary. And it doesn't "always work" either - there is a myth that kill -9 is "good" because it "always works". This isn't true, a process blocked in certain states will not respond to anything - including kill -9.

When you use kill -9 the killed process cannot clean up (this is a UNIX feature - kill -9 is defined as being uncatchable). Watchdog notices that the client is gone and then attempts to shut down the session and cleanup on behalf of the client. Watchdog succeeds at cleaning up quite a lot of the time. Sometimes a critical resource is being held (protected by a latch) and there is no way to clean up because the necessary data was in the vanished memory space of the client. In that event watchdog brings down the db (if you aren't running a watchdog process the broker will perform this function). That isn't as big an issue as it sounds like since if a latch was being held (and they always are when watchdog crashes the db) then nobody was doing anything except spinning on it anyway so your users are effectively frozen - they just haven't noticed yet.

The corruption that many people think that they see after using kill -9 comes from some of the other excitement that often happens along with it. In no event does "bad data" get written to disk either in the .bi file or the .db file as a direct result of any sort of a kill command. Stuff like deleting live .lk files, hosing the location of the .bi file, unwisely using dbrpr or using the -F option can lead to data corruption (you have to ignore a lot of warning messages to succeed). Environments where kill -9 is used are usually very uncontrolled and chaotic. Lots of stuff happens that isn't reported or well thought out and it's difficult to get to the bottom of things. So kill -9 gets blamed.

So in summary - kill -9 will certainly crash your database sooner or later, it's like playing Russian Roulette. It will not, by itself, corrupt your database. It does, however, lay the groundwork for other activities to be executed which may damage your data. If you feel that you must kill a process first use kill -2, then, if necessary, kill -1. If you need more than that then something is very seriously wrong and you should get help.