TROUBLE SHOOTING IN WAS

        TROUBLE SHOOTING IN WAS



-----> When we got any issue in WAS we have to trouble shoot that issue 

-----> Suppose Page Cannot be Displayed  HTTP 404 we have to solve this issue.

-----> First we have to check the browser of our HTTP URL whether it is giving correct URL or not from the address bar.



-----> If it is correct URL then we have to check the LOGS



1) JVM LOGS , (System.Out System.err) 

2) PROCESS LOGS (NATIVE LOGS) (stdout, stderr) (native_stdout , native_stderr)

3) IBM service log (activitylog)

4) Http Server plugin (http_plugin.log)



----->  First try to hit the web server if that is good then hit the application server directly if both of them work then there is an issue with plugin.sometimes the web server will be in hung state kill the web server processes which can be found 

-----> After all checking this things we have to logout from our application and we have to stop the server and then starts the server.

Troubleshooting Out of Memory errors in WebSphere :

Effects of running Out  of Memory

   
   ----> The garbage collection (GC) process struggles to free memory. GC is running all the time

     ----->  The long and continuous GC cycles cause high CPU in the application server

    

     ----->  The Application server is not able to process request as fast as they come in. This creates queuing in the Web and Application Servers.

  

     ---->  The JVM eventually stops responding and crashes. Requests are failed over to the next server.

When an Out of Memory occurs, three primary pieces of evidence are left at the scene.

----> Verbose garbage collection log (How it happened)

   ----->  Heapdump (What was in memory when it happened)

   ---->  Javacore (What was running when it  happened).

Tools use to analyse the three evidence

     
    ---->  Garbage collection log - The IBM Support Assistance (ISA) provides the Garbage Collection and Memory Visualizer Tool to open the verboseGC log file

     ----> Java heapdump  (heapdump.phd)  - Use Memory Analyzer Tool (MAT) in IBM Support  Assistant (ISA)

    -----> Javacore - IBM Thread and Monitor Dump Analyzer for Java

Categorization Out of Memory problems

    
 ----> Java heap exhaustion - The JVM cannot allocate an object because it is out of memory and no more memory could be made available by the garbage collector.

    -----> Large object allocation - The application requesting a very large object  which Java cannot accommodate in the heap.

   -----> Native memory allocation failure  - The memory space for the operating system process that correspond to Java has two main areas

        
   -----> The Java Heap which contains the instances of Java objects and is maintained by Garbage Collection

        
  ----->  The Native Heap which contains - Compiled JIT code, Malloc allocation by application JNI code, Threads to map to Java threads and    Native DB2 or MQ libraries

        
   -----> These two areas compete for the same process space - Making the Java heap too large could leave too little native memory. If you see  malloc errors in the native_stderr.log you are running into a native OOM problem.

Common scenarios that could  lead to OOM

1. Typically, heap exhaustion is caused by
               - Large categories and lack of pagination or filtering (“show all”)
               - Improperly sized cache (in-memory cache is too large)
               - Unbounded search
               - Scheduler processing a large job
               - Processing large backend messages
               - Improperly sized Java heap (too small)
2. Typically, OOM due to a large object allocation is caused by
               - A 3rd-party catalog integration returning all products at once
               - Inbound web service receiving large messages
3. Typically, a native memory error is caused by
              - Improperly sized Java heap ( too big )

HEAP DUMP AND THREAD DUMP ISSUES: 

----->  When we trigger a thread dump on an application server, all Active thread information will be dump into file which is normally known as Thread Dump.

-----> In this Dump file , we can get information about thread like Active thread,hung ,
          thread, Dead lock, runnable and inactive thread info.



COMMAND TO GENERATE THREAD DUMP:

KILL -3 PID_OF_SERVER.

----> When our application server reports the thread Hung Conditions in SystemOut.log file then we generate thread dump to analyze the reason behind thread Hungs.

-----> We are generally having thread Hung situation when any threads coming to system being responded.

-----> When Web Sphere Application Server detects that a thread has been active longer than the time define by the thread monitors and write the error code below in SystemOut.log file.


Heap Dump  :

----->  Heap Dump is a text file which keeps records of all  objects in the Java Heap.

-----> It contains the information like size and address of all objects , as well as addresses of all the objects references.

-----> Before generating the Heap Dump we need to enable the Garbage Collection to get only live objects information in Dump.

-----> By default when ever we get OutOfMemoryException occurs Web Sphere Application Server create Heap Dump in profiles home directory.

-----> Heap Dump is nothing but snap shot of our JVM'S memory.

-----> Here we can have all active Java objects and their activities which object doing what and how much memory it is occupying. 


GENERATING HEAP DUMPS MANUALLY  :  

--->   When generating multiple heap dumps manually for memory leak analysis, make sure that significant objects are leaked in between the two heap dumps. This approach enables problem determination tools to identify the source of the memory leak.

-----> On a Java virtual machines (JVM) in WebSphere Application Server, you cannot enable automated heap dump generation. You might want to designate certain times to take heap dumps because of the overhead involved. On JVM in WebSphere Application Server, you can manually produce heap dumps by using the generateHeapDump operation on WebSphere Application Server managed beans (MBeans) that are special Java beans.


Procedure  :


STEP  1  :  Invoke the generateHeapDump operation on a JVM MBean, for example,


STEP 2  :  <wsadmin> set objectName [$AdminControl queryNames
WebSphere:type=JVM,process=<serverName>,node=<nodeName>, *]

STEP 3 :<wsadmin> $ AdminControl invoke $objectName generateHeapDump

How to look for CPU heap issues for WAS ?

A)  ----> (start the wsadmin console... you'll be prompted for login/pwd credentials...)

cd \ProgramFiles\IBM\WebSphere\AppServer\bin
wsadmin

(setup for the DUMPS... specifying application server to dump...)
wsadmin> set jvm [$AdminControl completeObjectName type=JVM,process=server1,*]

(when ready to dump... execute the following to get a javacore file...)
wsadmin> $AdminControl invoke $jvm dumpThreads

                  OR

(when ready to dump... execute the following to get a heapdump file...)
wsadmin> $AdminControl invoke $jvm generateHeapDump

javacore/heapdump file will be in the following directory:
\ProgramFiles\IBM\WebSphere\AppServer\profiles\ST_Advanced_Profile

----> Where to look for errors : (SystemOut, SystemErr, ffdc)

----> Logging/tracing usually found in /WebSphere/AppServer/profiles/profile_name/logs/

---->  Server logs: 

      1)  Look in system error logs, found in /WebSphere/AppServer/profiles/profile_name/logs/server1/SystemErr.log

2) Look in system out logs, found in /WebSphere/AppServer/profiles/profile_name/logs/server1/SystemOut.log

----->  ffdc logs: 

    Look in ffdc logs, found in /WebSphere/AppServer/profiles/profile_name/logs/ffdc