I have finally got round to putting a post together around the 11.1.2.4 maintenance release, I would first like to stress this is not going to be a step by step process guide and to be honest the maintenance release is no different than applying in previous releases which I have already covered here and here.
There is also substantial documentation available here which should cover the full process.
Before diving into applying a maintenance release there are a number of important factors to consider like the supported paths to 11.1.2.4
Personally I am not a big fan of maintenance releases for reasons which I will try and cover and in many cases the end goal of running 11.1.2.4 with upgraded applications from a previous release can be achieved with a clean install.
Similar questions to the following examples should be asked first to ascertain whether a maintenance release versus a clean install is the best option:
Maintenance releases do have there place but you have to be sure it is right for your deployment, I mean you may have recently had a clean install on new hardware of 11.1.2.3 and are an essbase/planning customer, as there have not been many infra changes in this area it could be a good fit for this in-place upgrade.
My main problem with maintenance releases is they are basically a bolt on to a previous version and many files that are no longer required are not removed, the Shared Services registry can be a mess and the registry cleaner does not really fix this, the system can be slower than a clean install and if it goes wrong it is not only a pain to fix but you are not sure if it is down to the maintenance release or a bug in the new release.
I prefer the way other Oracle products work where you can install a new release to a new Middleware home and then just run an upgrade assistant, once confirmed working the old Middleware home can be deleted.
The reason I am going to be trying out the maintenance release today from 11.1.2.3 is I want to see if I hit any issues, I am interested to know what happens with FDM as it is no longer an option in 11.1.2.4 with FDMEE as the replacement, also due to all the architectural changes with HFM I am keen to understand whether the maintenance release cleans up any of the existing components that are no longer required, I am pretty sure I know the answer but you never know I might be shocked.
It is very important to have a full rollback procedure in case the upgrade process goes wrong, I have seen many posts in the past where the person applying the release has totally overlooked this and ended up a complete mess.
I will also point out that housekeeping is definitely a good option before hitting the installer.
The maintenance release does not remove any client installations so these should really be removed before starting.
There is a bit of confusing statement in the documentation which has been there for the last few releases:
All good, on to the install though as I mentioned this is not a step by step document and I would prefer to dig into what is happening when applying which may get a little technical.
So the installer has been started with the same user as the original installer and all the prerequisite checks have been met, at this point it is no different than a clean install but what always interests me is how the installer knows a version of EPM has been installed.
The middleware home setting is not updateable indicating a previous release exists.
The way the installer determines a previous version is there is by first checking two system variables.
If the EPM_ORACLE_HOME and HYPERION_HOME variables do not exist the Oracle inventory is then searched for EpmSystem home.
If none of the variables are found then it is deemed a clean install and the Middleware Home can be entered.
If any of the three variables are found then the assumption is a previous version of EPM has been installed.
If a valid version exists the only option will be to apply the maintenance release but how does the installer work out whether a valid version is installed, well this is where the file .oracle.products comes into play.
The file is basically in XML format and contains information about what products components have been installed and the version.
If any products are found that are in the supported upgrade path the apply maintenance reason will be the only available option as the maintenance has to be applied before any new installs.
The file also determines what list of products are displayed to be upgraded in the following screen:
There is an important note at this point
“You must apply the maintenance release to all EPM System products in the deployment. You cannot apply the maintenance release to only some products.”
To show how the file works I will go through a few changes.
In the above example I have set the version to 11.1.2.4 for Essbase Studio Server Samples.
The installer now believes the samples are now at 11.1.2.4 so need to apply the maintenance release.
In the next example I have removed all the children from the essbase node.
Run the installer again and it now thinks no Essbase related products have been installed.
If I remove .oracle.products then installer will be conned into thinking nothing has been installed and hence no option to apply the maintenance release.
Once a maintenance release has been installed this is what you are looking for before proceeding.
Just before completion you will notice the following in the status bar:
At this point not only is .oracle.products being updated but also the files under EPM_ORACLE_HOME\inventory are being updated to 11.1.2.4
An example of the components inventory after the maintenance install.
And if you take another look at .oracle.products the version been updated and the install type has been changed from NEW to MAINTENANCE.
It is interesting at this point to see if anything has happened with FDM in the file.
As expected there have been no changes to the FDM product components.
The maintenance install does not interact in any way with the Shared Services Registry or make any changes under EPM_ORACLE_INSTANCE.
The Shared Services registry comes into play the first time the configuration utility is started and in the background property values are being changed for all the installed components.
Here are some examples of the changes that happen the first time the utility is started but no configuration has taken place.
The utility has gone through the registry and set property values for each product to “PendingMaint”
These changes now determine what components are automatically selected and the status in the next configuration screen.
The “PendingMaint” value is very important in a maintenance release as when a product is configured the utility is updating the database structure to bring it in line with PS4 release.
Once a configuration is complete you can see the updates that have been made to the registry.
The registry has been updated to change the status to "Configured" and version updated to "11.1.2.4.0"
There are also important updates being made on the file system under the EPM_ORACLE_INSTANCE directory during the configuration, I am not going to go into them but need to be taken into account in a rollback situation.
As I eluded to earlier a maintenance release does not clean up the previous release and an example is the OCM configuration which no longer exists in 11.1.2.4, after applying the maintenance you are still left with the OCM files, windows service and entries in the registry.
So if you add the products into the mix then you can see it is getting messy compared to a clean install.
If you take HFM as another example you will see registry entries that are no longer required if you compare a clean install to a maintenance.
Taking a look at the file structure for HFM you will see how polluted a maintenance release gets.
There is also substantial documentation available here which should cover the full process.
Before diving into applying a maintenance release there are a number of important factors to consider like the supported paths to 11.1.2.4
Personally I am not a big fan of maintenance releases for reasons which I will try and cover and in many cases the end goal of running 11.1.2.4 with upgraded applications from a previous release can be achieved with a clean install.
Similar questions to the following examples should be asked first to ascertain whether a maintenance release versus a clean install is the best option:
- Is the hardware that EPM is running on going to be fit for x number of years.
- Is it time to rethink how your current release of EPM is distributed and whether it is the right fit for the next release.
- Do you want to change operating system, for example windows 2012 server is now supported.
- Is it time for a server rebuild required after accumulating years of patches and fixes at OS and Software level.
- Do any additional products require bringing into the environment
- Is there a requirement move to a different database type e.g. SQL Server to Oracle.
- Is it a good time to perform a review of applications, reports, security before moving to the next release.
- Is a clean install not a better option as a maintenance release pollutes rather than cleans.
- Can you bring down your live system while a maintenance release is being applied.
- Do you have time to fully test the system before bringing it back online.
- What happens is something goes wrong in the maintenance release process.
There are many more of these types of questions and if whoever you have chosen to perform the upgrade is not asking them then you should really reconsider changing them before ending up in a bad place.
Maintenance releases do have there place but you have to be sure it is right for your deployment, I mean you may have recently had a clean install on new hardware of 11.1.2.3 and are an essbase/planning customer, as there have not been many infra changes in this area it could be a good fit for this in-place upgrade.
My main problem with maintenance releases is they are basically a bolt on to a previous version and many files that are no longer required are not removed, the Shared Services registry can be a mess and the registry cleaner does not really fix this, the system can be slower than a clean install and if it goes wrong it is not only a pain to fix but you are not sure if it is down to the maintenance release or a bug in the new release.
I prefer the way other Oracle products work where you can install a new release to a new Middleware home and then just run an upgrade assistant, once confirmed working the old Middleware home can be deleted.
The reason I am going to be trying out the maintenance release today from 11.1.2.3 is I want to see if I hit any issues, I am interested to know what happens with FDM as it is no longer an option in 11.1.2.4 with FDMEE as the replacement, also due to all the architectural changes with HFM I am keen to understand whether the maintenance release cleans up any of the existing components that are no longer required, I am pretty sure I know the answer but you never know I might be shocked.
It is very important to have a full rollback procedure in case the upgrade process goes wrong, I have seen many posts in the past where the person applying the release has totally overlooked this and ended up a complete mess.
I will also point out that housekeeping is definitely a good option before hitting the installer.
The maintenance release does not remove any client installations so these should really be removed before starting.
There is a bit of confusing statement in the documentation which has been there for the last few releases:
- Apply any required PSEs before proceeding with maintenance the installation.
I have never been sure to what that is referring to as a maintenance release should drop down all the required files and files from a previous release should not be that important
It is a good time to run the registry cleaner utility before applying the maintenance to confirm whether the Shared Services registry is in good shape to proceed, it doesn’t always help much but it does no harm to run it.
It is a good time to run the registry cleaner utility before applying the maintenance to confirm whether the Shared Services registry is in good shape to proceed, it doesn’t always help much but it does no harm to run it.
All good, on to the install though as I mentioned this is not a step by step document and I would prefer to dig into what is happening when applying which may get a little technical.
So the installer has been started with the same user as the original installer and all the prerequisite checks have been met, at this point it is no different than a clean install but what always interests me is how the installer knows a version of EPM has been installed.
The middleware home setting is not updateable indicating a previous release exists.
The way the installer determines a previous version is there is by first checking two system variables.
If the EPM_ORACLE_HOME and HYPERION_HOME variables do not exist the Oracle inventory is then searched for EpmSystem home.
If none of the variables are found then it is deemed a clean install and the Middleware Home can be entered.
If any of the three variables are found then the assumption is a previous version of EPM has been installed.
If a valid version exists the only option will be to apply the maintenance release but how does the installer work out whether a valid version is installed, well this is where the file .oracle.products comes into play.
The file is basically in XML format and contains information about what products components have been installed and the version.
If any products are found that are in the supported upgrade path the apply maintenance reason will be the only available option as the maintenance has to be applied before any new installs.
The file also determines what list of products are displayed to be upgraded in the following screen:
There is an important note at this point
“You must apply the maintenance release to all EPM System products in the deployment. You cannot apply the maintenance release to only some products.”
To show how the file works I will go through a few changes.
In the above example I have set the version to 11.1.2.4 for Essbase Studio Server Samples.
The installer now believes the samples are now at 11.1.2.4 so need to apply the maintenance release.
In the next example I have removed all the children from the essbase node.
Run the installer again and it now thinks no Essbase related products have been installed.
If I remove .oracle.products then installer will be conned into thinking nothing has been installed and hence no option to apply the maintenance release.
Once a maintenance release has been installed this is what you are looking for before proceeding.
Just before completion you will notice the following in the status bar:
At this point not only is .oracle.products being updated but also the files under EPM_ORACLE_HOME\inventory are being updated to 11.1.2.4
An example of the components inventory after the maintenance install.
And if you take another look at .oracle.products the version been updated and the install type has been changed from NEW to MAINTENANCE.
It is interesting at this point to see if anything has happened with FDM in the file.
As expected there have been no changes to the FDM product components.
The maintenance install does not interact in any way with the Shared Services Registry or make any changes under EPM_ORACLE_INSTANCE.
The Shared Services registry comes into play the first time the configuration utility is started and in the background property values are being changed for all the installed components.
Here are some examples of the changes that happen the first time the utility is started but no configuration has taken place.
The utility has gone through the registry and set property values for each product to “PendingMaint”
These changes now determine what components are automatically selected and the status in the next configuration screen.
The “PendingMaint” value is very important in a maintenance release as when a product is configured the utility is updating the database structure to bring it in line with PS4 release.
Once a configuration is complete you can see the updates that have been made to the registry.
The registry has been updated to change the status to "Configured" and version updated to "11.1.2.4.0"
There are also important updates being made on the file system under the EPM_ORACLE_INSTANCE directory during the configuration, I am not going to go into them but need to be taken into account in a rollback situation.
As I eluded to earlier a maintenance release does not clean up the previous release and an example is the OCM configuration which no longer exists in 11.1.2.4, after applying the maintenance you are still left with the OCM files, windows service and entries in the registry.
So if you add the products into the mix then you can see it is getting messy compared to a clean install.
If you take HFM as another example you will see registry entries that are no longer required if you compare a clean install to a maintenance.
As for FDM
all Shared Services registry entries remain untouched in a 11.1.2.4
maintenance.
Taking a look at the file structure for HFM you will see how polluted a maintenance release gets.
Above is only an example of the exe files in the FM server directory after a maintenance compared to a clean install below.
In IIS you will see that nothing has been removed for HFM even though it is not required in 11.1.2.4
It is the same story with DCOM objects with the maintenance version with all the objects left.
In a clean install of 11.1.2.4 it is a different picture.
The windows services for HFM are cleaned up though with only two remaining.
Going back to the configuration process I know I have mentioned a rollback procedure a few times and I have good reason to as I got hit by an issue when configuring the foundation database.
Inspecting the logs highlighted the issue but not the reason why.
Error: {0}[[
com.hyperion.hit.registry.exceptions.RegistryException: Unable to create registry.
at com.hyperion.hit.registry.RegistryConnection.createRegistry(RegistryConnection.java:358)
at com.hyperion.hit.registry.RegistryUtils.initUpgradeRegistry(RegistryUtils.java:339)
at com.hyperion.hit.registry.RegistryUtils.upgradeRegistry(RegistryUtils.java:141)
at com.hyperion.foundation.config.FoundationDbConfigurator.configure(FoundationDbConfigurator.java:87)
at com.hyperion.config.wizard.impl.RunAllTasks.executeDbConfigTask(RunAllTasks.java:685)
at com.hyperion.config.wizard.impl.RunAllTasks.execute(RunAllTasks.java:306)
at com.hyperion.config.wizard.impl.RunnAllTasksState.run(RunnAllTasksState.java:92)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.sql.BatchUpdateException: error occurred during batching: ORA-00001: unique constraint (FOUNDATION.PK_ROLES) violated
After a while of researching I did finally manage to find the cause and it was because a planning patch had been applied in 11.1.2.3 that made changes to Shared Services roles, the roles have not made it into 11.1.2.4 so caused a serious failure.
Luckily I could rollback remove the changes to the Shared Service database and then configure successfully.
The configuration utility does not automatically select to upgrade HFM application from an earlier release and has to be done manually.
Once again it is interesting to understand what is happening behind the scenes, I won’t go into full detail but basically a batch file is called which initiates a java class.
The Java builds up a command line to call the HFM application upgrade tool.
In 11.1.2.4 there is no GUI available for the tool and it has to be run silently using command line just like what the Java program generates from reading the Shared Services registry.
Here is an example of the command line that is generated:
E:\Oracle\Middleware\EPMSystem11R1\products\FinancialManagement\Server\HFM Application Upgrade_x64.exe
-silent -dbname=FUSION -dbport=1521 -dbuser=HFM –dbpassword=HFMx3ds3s1 -dbtype=Oracle -dbserver=HFMTNS -eoi=E:\Oracle\Middleware\user_projects\epmsystem1
The HFM database is also updated
Logs are generated are in EPM_ORACLE_INSTANCE\diagnostics\logs\upgrades\hfm
Yes the password to the database is written to the log :)
Once the upgrade has been completed the Java program will register the application in Shared Services if it is not already registered.
The utility can be run multiple times from command line or the configuration utility by manipulating the database table.
On the FDM side of things it is still possible to log into the applications after the maintenance release and I even managed to load data from FDM to 11.1.2.4 essbase though I am not saying that it is supported and how long it will work for, it won’t obviously work with the HFM adaptors due to 11.1.2.4 HFM moving into the Java world.
I think I am going to wrap up the 11.1.2.4 maintenance post now as it has taken a fair bit of effort to put together even though I have removed a lot of content which was not only to stop boring the hell out of you but I also don’t want to give away too many secrets :)
Thanks for the post John! Very useful research for future installations.
ReplyDeleteGreat job John, some really useful info in there.
ReplyDeleteThanks,
Andy T
Very usefull info and the questions you put in are simply superb. I installed cleanly for hfm consolidation timings, which are simply great but I saw smartview pulls are much much slower even after tuning. any idea?
ReplyDeletePrasad
Thanks John. Appreciate the time it takes to put this all together.
ReplyDeleteJohn- I am applying the maintenance release on 11.1.2.3.500. Do you remember name of role you took our from shared services registry ? Could you please provide me the name of role or the planning patch no.
ReplyDeleteThank you
Sanja
Sanja,
ReplyDeleteIt is planning 11.1.2.3.502 - patch 20358563
Hi John, thanks for the fine post. I've tried to use HFMCopyApp to copy a HFM application from 11.1.2.1 to 11.1.2.4 and then run the application update utility as you showed. It runs thru without any errors but the application doesn't show in either Shared Service nor in the Consolidation Administraton Applications Register task. Is this path to migrate an application a dead-end? I'd like to keep the application complete with PM, locked periods and audit logs.
ReplyDeleteNice one. thanks John.
ReplyDeleteThis post was a lifesaver!
ReplyDeleteWhen I got the PK_ROLES error, I had already installed the maintenance release and was not in a place to be able to roll back and keep my schedule.
I downloaded the "bad" patch (20358563), unzipped it and searched it for the .sql. Opened it and used the statements there to remove the entries from Shared Services schema. At that point, I was able to configure successfully and all was good!