Monday, 9 March 2015

EPM Maintenance Release

I have finally got round to putting a post together around the maintenance release, I would first like to stress this is not going to be a step by step process guide and to be honest the maintenance release is no different than applying in previous releases which I have already covered here and here.

There is also substantial documentation available here which should cover the full process.

Before diving into applying a maintenance release there are a number of important factors to consider like the supported paths to

Personally I am not a big fan of maintenance releases for reasons which I will try and cover and in many cases the end goal of running with upgraded applications from a previous release can be achieved with a clean install.

Similar questions to the following examples should be asked first to ascertain whether a maintenance release versus a clean install is the best option:
  • Is the hardware that EPM is running on going to be fit for x number of years.
  • Is it time to rethink how your current release of EPM is distributed and whether it is the right fit for the next release.
  • Do you want to change operating system, for example windows 2012 server is now supported.
  • Is it time for a server rebuild required after accumulating years of patches and fixes at OS and Software level.
  • Do any additional products require bringing into the environment
  • Is there a requirement move to a different database type e.g. SQL Server to Oracle.
  • Is it a good time to perform a review of applications, reports, security before moving to the next release.
  • Is a clean install not a better option as a maintenance release pollutes rather than cleans.
  • Can you bring down your live system while a maintenance release is being applied.
  • Do you have time to fully test the system before bringing it back online.
  • What happens is something goes wrong in the maintenance release process.
There are many more of these types of questions and if whoever you have chosen to perform the upgrade is not asking them then you should really reconsider changing them before ending up in a bad place.

Maintenance releases do have there place but you have to be sure it is right for your deployment, I mean you may have recently had a clean install on new hardware of and are an essbase/planning customer, as there have not been many infra changes in this area it could be a good fit for this in-place upgrade.

My main problem with maintenance releases is they are basically a bolt on to a previous version and many files that are no longer required are not removed, the Shared Services registry can be a mess and the registry cleaner does not really fix this, the system can be slower than a clean install and if it goes wrong it is not only a pain to fix but you are not sure if it is down to the maintenance release or a bug in the new release.

I prefer the way other Oracle products work where you can install a new release to a new Middleware home and then just run an upgrade assistant, once confirmed working the old Middleware home can be deleted.

The reason I am going to be trying out the maintenance release today from is I want to see if I hit any issues, I am interested to know what happens with FDM as it is no longer an option in with FDMEE as the replacement, also due to all the architectural changes with HFM I am keen to understand whether the maintenance release cleans up any of the existing components that are no longer required, I am pretty sure I know the answer but you never know I might be shocked.

It is very important to have a full rollback procedure in case the upgrade process goes wrong, I have seen many posts in the past where the person applying the release has totally overlooked this and ended up a complete mess.

I will also point out that housekeeping is definitely a good option before hitting the installer.

The maintenance release does not remove any client installations so these should really be removed before starting.

There is a bit of confusing statement in the documentation which has been there for the last few releases:
  • Apply any required PSEs before proceeding with maintenance the installation.
I have never been sure to what that is referring to as a maintenance release should drop down all the required files and files from a previous release should not be that important

It is a good time to run the registry cleaner utility before applying the maintenance to confirm whether the Shared Services registry is in good shape to proceed, it doesn’t always help much but it does no harm to run it.

All good, on to the install though as I mentioned this is not a step by step document and I would prefer to dig into what is happening when applying which may get a little technical.

So the installer has been started with the same user as the original installer and all the prerequisite checks have been met, at this point it is no different than a clean install but what always interests me is how the installer knows a version of EPM has been installed.

The middleware home setting is not updateable indicating a previous release exists.

The way the installer determines a previous version is there is by first checking two system variables.

If the EPM_ORACLE_HOME and HYPERION_HOME variables do not exist the Oracle inventory is then searched for EpmSystem home.

If none of the variables are found then it is deemed a clean install and the Middleware Home can be entered.

If any of the three variables are found then the assumption is a previous version of EPM has been installed.

If a valid version exists the only option will be to apply the maintenance release but how does the installer work out whether a valid version is installed, well this is where the file .oracle.products comes into play.

The file is basically in XML format and contains information about what products components have been installed and the version.

If any products are found that are in the supported upgrade path the apply maintenance reason will be the only available option as the maintenance has to be applied before any new installs.

The file also determines what list of products are displayed to be upgraded in the following screen:

There is an important note at this point

“You must apply the maintenance release to all EPM System products in the deployment. You cannot apply the maintenance release to only some products.”

To show how the file works I will go through a few changes.

In the above example I have set the version to for Essbase Studio Server Samples.

The installer now believes the samples are now at so need to apply the maintenance release.

In the next example I have removed all the children from the essbase node.

Run the installer again and it now thinks no Essbase related products have been installed.

If I remove .oracle.products then installer will be conned into thinking nothing has been installed and hence no option to apply the maintenance release.

Once a maintenance release has been installed this is what you are looking for before proceeding.

Just before completion you will notice the following in the status bar:

At this point not only is .oracle.products being updated but also the files under EPM_ORACLE_HOME\inventory are being updated to

An example of the components inventory after the maintenance install.

And if you take another look at .oracle.products the version been updated and the install type has been changed from NEW to MAINTENANCE.

It is interesting at this point to see if anything has happened with FDM in the file.

As expected there have been no changes to the FDM product components.

The maintenance install does not interact in any way with the Shared Services Registry or make any changes under EPM_ORACLE_INSTANCE.

The Shared Services registry comes into play the first time the configuration utility is started and in the background property values are being changed for all the installed components.

Here are some examples of the changes that happen the first time the utility is started but no configuration has taken place.

The utility has gone through the registry and set property values for each product to “PendingMaint

These changes now determine what components are automatically selected and the status in the next configuration screen.

The “PendingMaint” value is very important in a maintenance release as when a product is configured the utility is updating the database structure to bring it in line with PS4 release.

Once a configuration is complete you can see the updates that have been made to the registry.

The registry has been updated to change the status to "Configured" and version updated to ""

There are also important updates being made on the file system under the EPM_ORACLE_INSTANCE directory during the configuration, I am not going to go into them but need to be taken into account in a rollback situation.

As I eluded to earlier a maintenance release does not clean up the previous release and an example is the OCM configuration which no longer exists in, after applying the maintenance you are still left with the OCM files, windows service and entries in the registry.

So if you add the products into the mix then you can see it is getting messy compared to a clean install.

If you take HFM as another example you will see registry entries that are no longer required if you compare a clean install to a maintenance.

As for FDM all Shared Services registry entries remain untouched in a maintenance.

Taking a look at the file structure for HFM you will see how polluted a maintenance release gets.

Above is only an example of the exe files in the FM server directory after a maintenance compared to a clean install below.

In IIS you will see that nothing has been removed for HFM even though it is not required in

It is the same story with DCOM objects with the maintenance version with all the objects left.

In a clean install of it is a different picture.

The windows services for HFM are cleaned up though with only two remaining.

Going back to the configuration process I know I have mentioned a rollback procedure a few times and I have good reason to as I got hit by an issue when configuring the foundation database.

Inspecting the logs highlighted the issue but not the reason why.

Error: {0}[[
com.hyperion.hit.registry.exceptions.RegistryException: Unable to create registry.
    at com.hyperion.hit.registry.RegistryConnection.createRegistry(
    at com.hyperion.hit.registry.RegistryUtils.initUpgradeRegistry(
    at com.hyperion.hit.registry.RegistryUtils.upgradeRegistry(
    at com.hyperion.config.wizard.impl.RunAllTasks.executeDbConfigTask(
    at com.hyperion.config.wizard.impl.RunAllTasks.execute(
Caused by: java.sql.BatchUpdateException: error occurred during batching: ORA-00001: unique constraint (FOUNDATION.PK_ROLES) violated

After a while of researching I did finally manage to find the cause and it was because a planning patch had been applied in that made changes to Shared Services roles, the roles have not made it into so caused a serious failure.

Luckily I could rollback remove the changes to the Shared Service database and then configure successfully.

The configuration utility does not automatically select to upgrade HFM application from an earlier release and has to be done manually.

Once again it is interesting to understand what is happening behind the scenes, I won’t go into full detail but basically a batch file is called which initiates a java class.

The Java builds up a command line to call the HFM application upgrade tool.

In there is no GUI available for the tool and it has to be run silently using command line just like what the Java program generates from reading the Shared Services registry.

Here is an example of the command line that is generated:

E:\Oracle\Middleware\EPMSystem11R1\products\FinancialManagement\Server\HFM Application Upgrade_x64.exe
 -silent -dbname=FUSION -dbport=1521 -dbuser=HFM –dbpassword=HFMx3ds3s1 -dbtype=Oracle -dbserver=HFMTNS -eoi=E:\Oracle\Middleware\user_projects\epmsystem1

The HFM database is also updated

Logs are generated are in EPM_ORACLE_INSTANCE\diagnostics\logs\upgrades\hfm

Yes the password to the database is written to the log :)

Once the upgrade has been completed the Java program will register the application in Shared Services if it is not already registered.

The utility can be run multiple times from command line or the configuration utility by manipulating the database table.

On the FDM side of things it is still possible to log into the applications after the maintenance release and I even managed to load data from FDM to essbase though I am not saying that it is supported and how long it will work for, it won’t obviously work with the HFM adaptors due to HFM moving into the Java world.

I think I am going to wrap up the maintenance post now as it has taken a fair bit of effort to put together even though I have removed a lot of content which was not only to stop boring the hell out of you but I also don’t want to give away too many secrets :)