EPM Standalone Log Analysis Utility

In 11.1.2.3 a new command line utility was released that analyses ODL compliant logs to help troubleshoot problems and find route causes.

If you are not aware of the utility here is the Oracle documented information on it:

The Log Analysis Utility is a command line utility that helps you quickly identify the cause of issues reported by EPM System components by analyzing the applicable log files. Because this utility automates log file analysis, you do not need to manually locate and scan through EPM System log files to identify issues. Information required to troubleshoot the issue or to escalate it to Oracle Support is quickly available by running this utility. Generally, run on the server where Oracle Hyperion Foundation Services is installed, this utility accesses and analyzes log files on all the servers identified in the Oracle Hyperion Shared Services Registry of an EPM System instance.

Using the Log Analysis Utility you can:

• List EPM System errors that occurred within a time period. System issues are related to services, intercomponent communication errors, and user directory communication errors.
  
  
• List functional issues that occurred within a time period. Functional issues are related to EPM System component functionalities; for example, failure during an Oracle Essbase calculation run or the forms load process in Oracle Hyperion Planning or Oracle Hyperion Financial Management.
  
  
• Trace an Execution Context ID (ECID) through log files to trace user sessions across EPM System components. ECID is a unique identifier that is used to correlate events that are part of the same request execution flow. ECID is an Oracle standard unique ID.

The good news is this utility has been now been released as a standalone version that can be used to analyse logs from any 11.1.1.x to 11.1.2.x deployments and does not have to be run on EPM servers as it just requires access to the logs.

Though be aware that it is only ODL compliant logs it will run against so the older the EPM version the less of these type of logs will exist

The standalone utility can be downloaded as a patch (17425397) from Oracle support:

Once downloaded it is simple to get up and running with it.

Extract the utility and then edit loganalysis.bat/sh to set to a valid JAVA_HOME, it is worth noting that the utility is currently compiled with JDK 1.6.0_35 so I am not sure how well it will run against older versions of Java such as the default ones used in 11.1.1.x

Now I am not going to go through all the various parameters that can be used with the utility as the readme and documentation provide sufficient information to get a good understanding on what functionality is available.

I tested the utility against an 11.1.2.2 environment and at first I thought it would run just by setting the EPM_ORACLE_HOME variable but the utility errors out.

 

As this is a standalone utility it requires the use of –d parameter which defines the directory path to start searching from, the utility will search all the directories below the base directory.

As an example I stopped the database for all the EPM components and then started up Essbase which should generate errors because the communication to the Shared Services registry will not be available.

The command I used was:

loganalysis.bat -system -tday 1 -d E:\Oracle\Middleware\user_projects\epmsystem1\diagnostics\logs\essbase

This basically means the utility will search the logs starting in E:\Oracle\Middleware\user_projects\epmsystem1\diagnostics\logs\essbase and run a report for any ERROR and INCIDENT_ERROR types generated within the last day.

Once the utility has been run it will generate a html report in the same directory as the utility and then open it.

The report has picked up 12 errors and by looking at the messages straight away it points to an issue connecting to the Shared Services registry.

Running a similar report against the planning logs highlights issues with the database connection.

loganalysis.bat -system -tday 1 -d E:\Oracle\Middleware\user_projects\domains\EPMSystem\servers\Planning0\logs

The report also includes all the logs files which have been processed.

The utility has functionality to allow searching for a string using the –s parameter which I gave a quick test to search for the following “Out of memory” error:

[2013-09-20T19:39:20.127+01:00] [Profitability0] [ERROR] [EPMPCM-03028] [oracle.EPMPCM.web] [tid: 16] [userId: ] [ecid: 0000K1FlAWh7a6i5p7K6yY1HyxpJ0004_4,0] [SRC_CLASS: com.hyperion.profitability.web.filter.PersistenceFilter] [APP: PROFITABILITY#11.1.2.0] [SRC_METHOD: doFilter] Rolling back current transaction because of an exception[[javax.servlet.ServletException: java.lang.OutOfMemoryError: allocLargeObjectOrArray: [C, size 1179664

I ran the following:

loganalysis.bat -system -d E:\Oracle\Middleware\user_projects\domains\EPMSystem\servers\ -s "java.lang.OutOfMemoryError"

Unfortunately the utility did not pick up the error as it does not seem to search the supplemental detail in the log entry and the search needs to be focused on the initial error message for example:

loganalysis.bat -system -d E:\Oracle\Middleware\user_projects\domains\EPMSystem\servers -s "Rolling back current"

I am hoping the search functionality will be improved to allow a full search of the log entries which I understand will impact the running time but it would be nice to have the option, also allow multiple search strings and wildcards (I did not see anything mentioned about these in the documentation)

The utility does have some nice options like be able to search by different error types so it can be used to pick up functional related areas so let’s see when the last time the PLANS Essbase application was last started and its process ID

loganalysis.bat -functional -d E:\Oracle\Middleware\user_projects\epmsystem1\diagnostics\logs\essbase -s "Application [PLANS] started"

The utility can be set to run against multiple directory paths and can also analyse by ECID so in theory once you have the ECID you could trace a user’s session across the system.

So it is definitely worth having a look at the utility as it is easy to set up, it does not have to be run on an EPM server, it can be used across multiple EPM versions and certainly can save time trawling through the mass of logs.

I would be interested to know your views on the utility and how you feel it can be improved because I know the developers are looking for feedback which I will pass it on so maybe the enhancement requests will be included in future releases.