Monday, 9 March 2015

EPM 11.1.2.4 Maintenance Release

I have finally got round to putting a post together around the 11.1.2.4 maintenance release, I would first like to stress this is not going to be a step by step process guide and to be honest the maintenance release is no different than applying in previous releases which I have already covered here and here.

There is also substantial documentation available here which should cover the full process.

Before diving into applying a maintenance release there are a number of important factors to consider like the supported paths to 11.1.2.4


Personally I am not a big fan of maintenance releases for reasons which I will try and cover and in many cases the end goal of running 11.1.2.4 with upgraded applications from a previous release can be achieved with a clean install.

Similar questions to the following examples should be asked first to ascertain whether a maintenance release versus a clean install is the best option:
  • Is the hardware that EPM is running on going to be fit for x number of years.
  • Is it time to rethink how your current release of EPM is distributed and whether it is the right fit for the next release.
  • Do you want to change operating system, for example windows 2012 server is now supported.
  • Is it time for a server rebuild required after accumulating years of patches and fixes at OS and Software level.
  • Do any additional products require bringing into the environment
  • Is there a requirement move to a different database type e.g. SQL Server to Oracle.
  • Is it a good time to perform a review of applications, reports, security before moving to the next release.
  • Is a clean install not a better option as a maintenance release pollutes rather than cleans.
  • Can you bring down your live system while a maintenance release is being applied.
  • Do you have time to fully test the system before bringing it back online.
  • What happens is something goes wrong in the maintenance release process.
There are many more of these types of questions and if whoever you have chosen to perform the upgrade is not asking them then you should really reconsider changing them before ending up in a bad place.

Maintenance releases do have there place but you have to be sure it is right for your deployment, I mean you may have recently had a clean install on new hardware of 11.1.2.3 and are an essbase/planning customer, as there have not been many infra changes in this area it could be a good fit for this in-place upgrade.

My main problem with maintenance releases is they are basically a bolt on to a previous version and many files that are no longer required are not removed, the Shared Services registry can be a mess and the registry cleaner does not really fix this, the system can be slower than a clean install and if it goes wrong it is not only a pain to fix but you are not sure if it is down to the maintenance release or a bug in the new release.

I prefer the way other Oracle products work where you can install a new release to a new Middleware home and then just run an upgrade assistant, once confirmed working the old Middleware home can be deleted.

The reason I am going to be trying out the maintenance release today from 11.1.2.3 is I want to see if I hit any issues, I am interested to know what happens with FDM as it is no longer an option in 11.1.2.4 with FDMEE as the replacement, also due to all the architectural changes with HFM I am keen to understand whether the maintenance release cleans up any of the existing components that are no longer required, I am pretty sure I know the answer but you never know I might be shocked.

It is very important to have a full rollback procedure in case the upgrade process goes wrong, I have seen many posts in the past where the person applying the release has totally overlooked this and ended up a complete mess.

I will also point out that housekeeping is definitely a good option before hitting the installer.

The maintenance release does not remove any client installations so these should really be removed before starting.

There is a bit of confusing statement in the documentation which has been there for the last few releases:
  • Apply any required PSEs before proceeding with maintenance the installation.
I have never been sure to what that is referring to as a maintenance release should drop down all the required files and files from a previous release should not be that important

It is a good time to run the registry cleaner utility before applying the maintenance to confirm whether the Shared Services registry is in good shape to proceed, it doesn’t always help much but it does no harm to run it.


All good, on to the install though as I mentioned this is not a step by step document and I would prefer to dig into what is happening when applying which may get a little technical.


So the installer has been started with the same user as the original installer and all the prerequisite checks have been met, at this point it is no different than a clean install but what always interests me is how the installer knows a version of EPM has been installed.


The middleware home setting is not updateable indicating a previous release exists.

The way the installer determines a previous version is there is by first checking two system variables.


If the EPM_ORACLE_HOME and HYPERION_HOME variables do not exist the Oracle inventory is then searched for EpmSystem home.


If none of the variables are found then it is deemed a clean install and the Middleware Home can be entered.


If any of the three variables are found then the assumption is a previous version of EPM has been installed.

If a valid version exists the only option will be to apply the maintenance release but how does the installer work out whether a valid version is installed, well this is where the file .oracle.products comes into play.


The file is basically in XML format and contains information about what products components have been installed and the version.


If any products are found that are in the supported upgrade path the apply maintenance reason will be the only available option as the maintenance has to be applied before any new installs.

The file also determines what list of products are displayed to be upgraded in the following screen:


There is an important note at this point

“You must apply the maintenance release to all EPM System products in the deployment. You cannot apply the maintenance release to only some products.”

To show how the file works I will go through a few changes.


In the above example I have set the version to 11.1.2.4 for Essbase Studio Server Samples.


The installer now believes the samples are now at 11.1.2.4 so need to apply the maintenance release.

In the next example I have removed all the children from the essbase node.


Run the installer again and it now thinks no Essbase related products have been installed.


If I remove .oracle.products then installer will be conned into thinking nothing has been installed and hence no option to apply the maintenance release.


Once a maintenance release has been installed this is what you are looking for before proceeding.


Just before completion you will notice the following in the status bar:


At this point not only is .oracle.products being updated but also the files under EPM_ORACLE_HOME\inventory are being updated to 11.1.2.4

An example of the components inventory after the maintenance install.


And if you take another look at .oracle.products the version been updated and the install type has been changed from NEW to MAINTENANCE.


It is interesting at this point to see if anything has happened with FDM in the file.


As expected there have been no changes to the FDM product components.

The maintenance install does not interact in any way with the Shared Services Registry or make any changes under EPM_ORACLE_INSTANCE.

The Shared Services registry comes into play the first time the configuration utility is started and in the background property values are being changed for all the installed components.

Here are some examples of the changes that happen the first time the utility is started but no configuration has taken place.


The utility has gone through the registry and set property values for each product to “PendingMaint

These changes now determine what components are automatically selected and the status in the next configuration screen.


The “PendingMaint” value is very important in a maintenance release as when a product is configured the utility is updating the database structure to bring it in line with PS4 release.

Once a configuration is complete you can see the updates that have been made to the registry.


The registry has been updated to change the status to "Configured" and version updated to "11.1.2.4.0"

There are also important updates being made on the file system under the EPM_ORACLE_INSTANCE directory during the configuration, I am not going to go into them but need to be taken into account in a rollback situation.

As I eluded to earlier a maintenance release does not clean up the previous release and an example is the OCM configuration which no longer exists in 11.1.2.4, after applying the maintenance you are still left with the OCM files, windows service and entries in the registry.



So if you add the products into the mix then you can see it is getting messy compared to a clean install.

If you take HFM as another example you will see registry entries that are no longer required if you compare a clean install to a maintenance.



As for FDM all Shared Services registry entries remain untouched in a 11.1.2.4 maintenance.

Taking a look at the file structure for HFM you will see how polluted a maintenance release gets.


Above is only an example of the exe files in the FM server directory after a maintenance compared to a clean install below.


In IIS you will see that nothing has been removed for HFM even though it is not required in 11.1.2.4


It is the same story with DCOM objects with the maintenance version with all the objects left.


In a clean install of 11.1.2.4 it is a different picture.


The windows services for HFM are cleaned up though with only two remaining.


Going back to the configuration process I know I have mentioned a rollback procedure a few times and I have good reason to as I got hit by an issue when configuring the foundation database.


Inspecting the logs highlighted the issue but not the reason why.

Error: {0}[[
com.hyperion.hit.registry.exceptions.RegistryException: Unable to create registry.
    at com.hyperion.hit.registry.RegistryConnection.createRegistry(RegistryConnection.java:358)
    at com.hyperion.hit.registry.RegistryUtils.initUpgradeRegistry(RegistryUtils.java:339)
    at com.hyperion.hit.registry.RegistryUtils.upgradeRegistry(RegistryUtils.java:141)
    at com.hyperion.foundation.config.FoundationDbConfigurator.configure(FoundationDbConfigurator.java:87)
    at com.hyperion.config.wizard.impl.RunAllTasks.executeDbConfigTask(RunAllTasks.java:685)
    at com.hyperion.config.wizard.impl.RunAllTasks.execute(RunAllTasks.java:306)
    at com.hyperion.config.wizard.impl.RunnAllTasksState.run(RunnAllTasksState.java:92)
    at java.lang.Thread.run(Thread.java:662)
Caused by: java.sql.BatchUpdateException: error occurred during batching: ORA-00001: unique constraint (FOUNDATION.PK_ROLES) violated


After a while of researching I did finally manage to find the cause and it was because a planning patch had been applied in 11.1.2.3 that made changes to Shared Services roles, the roles have not made it into 11.1.2.4 so caused a serious failure.

Luckily I could rollback remove the changes to the Shared Service database and then configure successfully.

The configuration utility does not automatically select to upgrade HFM application from an earlier release and has to be done manually.


Once again it is interesting to understand what is happening behind the scenes, I won’t go into full detail but basically a batch file is called which initiates a java class.

The Java builds up a command line to call the HFM application upgrade tool.


In 11.1.2.4 there is no GUI available for the tool and it has to be run silently using command line just like what the Java program generates from reading the Shared Services registry.

Here is an example of the command line that is generated:

E:\Oracle\Middleware\EPMSystem11R1\products\FinancialManagement\Server\HFM Application Upgrade_x64.exe
 -silent -dbname=FUSION -dbport=1521 -dbuser=HFM –dbpassword=HFMx3ds3s1 -dbtype=Oracle -dbserver=HFMTNS -eoi=E:\Oracle\Middleware\user_projects\epmsystem1


The HFM database is also updated


Logs are generated are in EPM_ORACLE_INSTANCE\diagnostics\logs\upgrades\hfm


Yes the password to the database is written to the log :)

Once the upgrade has been completed the Java program will register the application in Shared Services if it is not already registered.

The utility can be run multiple times from command line or the configuration utility by manipulating the database table.

On the FDM side of things it is still possible to log into the applications after the maintenance release and I even managed to load data from FDM to 11.1.2.4 essbase though I am not saying that it is supported and how long it will work for, it won’t obviously work with the HFM adaptors due to 11.1.2.4 HFM moving into the Java world.

I think I am going to wrap up the 11.1.2.4 maintenance post now as it has taken a fair bit of effort to put together even though I have removed a lot of content which was not only to stop boring the hell out of you but I also don’t want to give away too many secrets :)

Thursday, 5 February 2015

EPM 11.1.2.4 - Java versions and why Windows Server 2012 is not correctly recognised

Today’s post looks at an issue with 11.1.2.4 not correctly picking up Windows 2012 and explores the reasoning behind it.

If installing EPM 11.1.2.4 on Windows 2012 Server the following OS information is shown in the installer prerequisite checks:


So the installer picks up the operating system as “Windows NT (unknown)” which I did mention in my post about the 11.1.2.4 installation and configuration.

This has a knock on effect as when you configure the EPM Registry the same values are written to the registry.


Running the deployment report outputs the same information as it is read from the EPM registry.


So why is this happening? Well we will need to look closer at the versions of Java being deployed with 11.1.2.4, Java being an integral part of EPM.

I think it’s worth highlighting the versions of Java automatically that are deployed with each release of 11.1.2



Each release up to 11.1.2.4 has been 12 months apart and in every case the versions of Java have been updated.

So how about 11.1.2.4 which was released nearly two years after 11.1.2.3, you certainly would expect later releases of Java to be deployed.


Worryingly the versions are exactly the same as the previous release.

Even more concerning is the warning message you receive when trying to download a version of Java 6.

“WARNING: These older versions of the JRE and JDK are provided to help developers debug issues in older systems. They are not updated with the latest security patches and are not recommended for use in production.”

It stands out even further when you look at the Java release update table to see how far behind 11.1.2.4 is.


The last public update for Java 6 was update 45 was way back at the beginning of 2013 and update 35 which 11.1.2.4 is using was released in August 2012.

Just this month the pre-release of Java 9 was made available.

There have been a huge amount of security fixes since the versions of Java in 11.1.2.4 and on this world we live in, security should be taken extremely seriously.

In fact it’s not just the versions of Java that have remained static as it is the same story across the middleware components.

I understand that 11.1.1.7 middleware components which are installed with 11.1.2.4 do support Java 7 so it seems strange why this version was not introduced

Anyway let’s get back to the reason why the OS is being picked up incorrectly.

The Java code behind EPM uses the System class and Properties object to generate OS information, if you want more detailed information then have a look here.

I wrote a simple Java program using the os.name and os.version keys to output OS information like generated in EPM.

First using the EPM Java JDK version.


As you see it is generating incorrect information, now using a later JDK 7 release.


So this highlights it is the down to the version of Java which is causing the issue.

I looked into this further and noticed that this issue was resolved in Java 6 Update 38.


Update 38 was released in December 2012.

As a test I thought I would start up the EPM installer using a later version of Java but first checked out the version being used.


The version of Java that the installer uses is 6 update 29 which is even older than the version that is deployed.



Using a later version of Java the operating system information is now generated correctly, please not this was just a test and I don’t suggest trying to install or configure with different versions.

If you are interested in updating the registry so the OS information is correct then you can simply use the EPM system registry tool.



The above is based on a single host but can easily updated against multiple hosts.

Once the registry is updated the registry and deployment report will display the correct information.



From an infrastructure perspective if you take HFM out of the picture,  is 11.1.2.4 really 11.1.2.3 in disguise :)