EPM Standalone Log Analysis Utility

In 11.1.2.3 a new command line utility was released that analyses ODL compliant logs to help troubleshoot problems and find route causes.

If you are not aware of the utility here is the Oracle documented information on it:

The Log Analysis Utility is a command line utility that helps you quickly identify the cause of issues reported by EPM System components by analyzing the applicable log files. Because this utility automates log file analysis, you do not need to manually locate and scan through EPM System log files to identify issues. Information required to troubleshoot the issue or to escalate it to Oracle Support is quickly available by running this utility. Generally, run on the server where Oracle Hyperion Foundation Services is installed, this utility accesses and analyzes log files on all the servers identified in the Oracle Hyperion Shared Services Registry of an EPM System instance.

Using the Log Analysis Utility you can:

• List EPM System errors that occurred within a time period. System issues are related to services, intercomponent communication errors, and user directory communication errors.
  
  
• List functional issues that occurred within a time period. Functional issues are related to EPM System component functionalities; for example, failure during an Oracle Essbase calculation run or the forms load process in Oracle Hyperion Planning or Oracle Hyperion Financial Management.
  
  
• Trace an Execution Context ID (ECID) through log files to trace user sessions across EPM System components. ECID is a unique identifier that is used to correlate events that are part of the same request execution flow. ECID is an Oracle standard unique ID.

The good news is this utility has been now been released as a standalone version that can be used to analyse logs from any 11.1.1.x to 11.1.2.x deployments and does not have to be run on EPM servers as it just requires access to the logs.

Though be aware that it is only ODL compliant logs it will run against so the older the EPM version the less of these type of logs will exist

The standalone utility can be downloaded as a patch (17425397) from Oracle support:

Once downloaded it is simple to get up and running with it.

Extract the utility and then edit loganalysis.bat/sh to set to a valid JAVA_HOME, it is worth noting that the utility is currently compiled with JDK 1.6.0_35 so I am not sure how well it will run against older versions of Java such as the default ones used in 11.1.1.x

Now I am not going to go through all the various parameters that can be used with the utility as the readme and documentation provide sufficient information to get a good understanding on what functionality is available.

I tested the utility against an 11.1.2.2 environment and at first I thought it would run just by setting the EPM_ORACLE_HOME variable but the utility errors out.

 

As this is a standalone utility it requires the use of –d parameter which defines the directory path to start searching from, the utility will search all the directories below the base directory.

As an example I stopped the database for all the EPM components and then started up Essbase which should generate errors because the communication to the Shared Services registry will not be available.

The command I used was:

loganalysis.bat -system -tday 1 -d E:\Oracle\Middleware\user_projects\epmsystem1\diagnostics\logs\essbase

This basically means the utility will search the logs starting in E:\Oracle\Middleware\user_projects\epmsystem1\diagnostics\logs\essbase and run a report for any ERROR and INCIDENT_ERROR types generated within the last day.

Once the utility has been run it will generate a html report in the same directory as the utility and then open it.

The report has picked up 12 errors and by looking at the messages straight away it points to an issue connecting to the Shared Services registry.

Running a similar report against the planning logs highlights issues with the database connection.

loganalysis.bat -system -tday 1 -d E:\Oracle\Middleware\user_projects\domains\EPMSystem\servers\Planning0\logs

The report also includes all the logs files which have been processed.

The utility has functionality to allow searching for a string using the –s parameter which I gave a quick test to search for the following “Out of memory” error:

[2013-09-20T19:39:20.127+01:00] [Profitability0] [ERROR] [EPMPCM-03028] [oracle.EPMPCM.web] [tid: 16] [userId: ] [ecid: 0000K1FlAWh7a6i5p7K6yY1HyxpJ0004_4,0] [SRC_CLASS: com.hyperion.profitability.web.filter.PersistenceFilter] [APP: PROFITABILITY#11.1.2.0] [SRC_METHOD: doFilter] Rolling back current transaction because of an exception[[javax.servlet.ServletException: java.lang.OutOfMemoryError: allocLargeObjectOrArray: [C, size 1179664

I ran the following:

loganalysis.bat -system -d E:\Oracle\Middleware\user_projects\domains\EPMSystem\servers\ -s "java.lang.OutOfMemoryError"

Unfortunately the utility did not pick up the error as it does not seem to search the supplemental detail in the log entry and the search needs to be focused on the initial error message for example:

loganalysis.bat -system -d E:\Oracle\Middleware\user_projects\domains\EPMSystem\servers -s "Rolling back current"

I am hoping the search functionality will be improved to allow a full search of the log entries which I understand will impact the running time but it would be nice to have the option, also allow multiple search strings and wildcards (I did not see anything mentioned about these in the documentation)

The utility does have some nice options like be able to search by different error types so it can be used to pick up functional related areas so let’s see when the last time the PLANS Essbase application was last started and its process ID

loganalysis.bat -functional -d E:\Oracle\Middleware\user_projects\epmsystem1\diagnostics\logs\essbase -s "Application [PLANS] started"

The utility can be set to run against multiple directory paths and can also analyse by ECID so in theory once you have the ECID you could trace a user’s session across the system.

So it is definitely worth having a look at the utility as it is easy to set up, it does not have to be run on an EPM server, it can be used across multiple EPM versions and certainly can save time trawling through the mass of logs.

I would be interested to know your views on the utility and how you feel it can be improved because I know the developers are looking for feedback which I will pass it on so maybe the enhancement requests will be included in future releases.

EAS - single sign-on to Essbase using separate Shared Services instances

I am back with a blog that was inspired from a question posted on the OTN forums, the poster asked if it was possible to use single sign in EAS to connect to two Essbase servers which are managed by separate Shared Services instances.

The initial answer is no because each Shared Services instance will use a unique CSS token encryption key but the answer changes to yes it is possible with the help of a utility.

Before I go through the process on how to achieve the single sign-on between environments I want to stress that this is just an example which may not be supported and it is not something I endorse so think about the consequences and security aspects before attempting.

I will be using 11.1.2.3 in the example but it should also be valid on 11.1.2.2

Both environments have been configured to use the same external directory in Shared Services as I don’t believe it will work with native directory accounts.

 An AD user called essbaseadmin has been provisioned with the Essbase administrator role on both environments.

I have already added the first Essbase server using SSO in EAS and as this it is using the same Shared Services instance there are no problems connecting to it.

Now let’s add an additional Essbase server which is managed by a separate Shared Services instance.

The “Use Single Sign On” option was selected.

Connecting to the Essbase server fails with the standard login fails due to invalid login credentials error.

Usually you would just add the server without using single sign-on and enter the username and password to connect with, I don’t see the problem with this and it should be the way to connect but that is just my opinion.

Anyway back to the original question and how to achieve the single sign-on, if you take a look on a foundation server in the directory:
<MIDDLEWARE_HOME>\EPMSystem11R11\common\CSS\11.1.2.0
you will see a zipped up file called regSyncUtil_EPM-to-EPM.zip

The archive file contains a pretty unknown utility which allows the migration of the CSS token encryption key between two Shared Services registries so a CSS token generated on the source environment should validate on the destination environment.

If you have ever gone through the process of single sign-on integration between Workspace and OBIEE then you will have used the same utility to migrate the encryption key.

Before running the utility there are a couple of steps that need to be completed.

• Copy reg.properties from the source environment to the src folder.
  
  
• Edit runRegSyncUtil.bat and update the ORACLE_HOME,ORACLE_HOME,JAVA_HOME variables to contain the correct full paths

If running the utility is successful the key should be read from the source registry and imported into the destination registry.

If you compare a registry report before and after running the utility you will see the CSSHandlerKey2 property value is updated.

Now that is done the Essbase server can be added again in EAS.

Once again the “Use Single Sign On” option is selected.

And there we go the user is able to log straight into the Essbase server using SSO.

This concept can no doubt be used for other products that generate and pass a SSO token.