Sunday, 24 March 2019

Hybrid FDMEE and upcoming Oracle Cloud changes

If you are not already aware, there are security protocol changes being made in the 19.05 Cloud release that will affect hybrid FDMEE implementations.

An excerpt from the Cloud Readiness document:

“Starting with the May 2019 update (Release 19.05) to EPM Services, Oracle will support only Transport Layer Security protocol version 1.2 (TLS 1.2) to communicate with EPM Cloud. To ensure the highest level of authentication and data encryption security, TLS 1.0 and TLS 1.1 will no longer be supported after your environments are updated on May 3, 2019.”

Now I have already read an article on this topic and Oracle support documents which outline how to be compliant with the TLS changes, but in my opinion the guidance has been made overcomplicated and confusing.

In this post I am going to provide my take on what is required to be ready for when the security changes are implemented. The objective is not to overly complicate matters, nor to advise to apply every critical patch update across your whole EPM estate. The idea is that you already have a functioning hybrid FDMEE implementation and you need to know what requires updating which will have minimal impact to your environment.

Please note this is just my opinion, so feel free to disagree and challenge my view.

Basically, FDMEE integrations with EPM Cloud are achieved using the REST API, communication between FDMEE and the Cloud is only from the FDMEE Java web application. This gives a clue to where to focus first and that is with Java. If there have been no updates to the version of Java deployed with FDMEE then it will be running Java 6 update 35. This version of Java is so old it does not support TLS 1.2 so when the upcoming changes are made, FDMEE integrations with Oracle Cloud will fail.

The Oracle Java roadmap highlights just how far behind is. If you put it into perspective Java 12 has recently been released and is running on Java 6 which extended support ended in December 2018 so there will no further patch updates.

If you want to read about this in more detail, then take a look at the Oracle support document – “Oracle Java SE 6 and JRockit End of Support (Doc ID 2244851.1)

So, what are the options when it comes to updating the version of Java to support TLS 1.2? Well, only supports Java 6 and 7 so you can forget about anything higher.

If you have already updated you EPM environment to Java 6 update 121+ (don’t hold me on that version though) or to Java 7 then you should be ready for TLS 1.2, though to be sure you should really test to confirm.

Before making any changes, I double checked the supported protocols against the EPM Cloud instance with the help of an SSL labs test.

Only TLS 1.2 is enabled which is what I wanted to see. If you have time and would like to test out on an Oracle Cloud instance, then raise an SR and they will provide temporary access.

In FDMEE I added a new target Cloud application.

Instead of the application selection window being displayed, an SSL error was generated.

With TLS debugging enabled, the logs confirmed that the issue was related to communication being initiated with the TLS 1.0 protocol.

[ACTIVE] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)', READ: TLSv1 Alert, length = 2
[ACTIVE] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)', RECV TLSv1 ALERT:  fatal, handshake_failure

I will go through what I believe is a simple method for updating Java to a version that should support TLS 1.2, it is not the one which Oracle sometimes advises and involves finding and replacing Java versions in files.

I am going to start out with Java 7 as that makes more sense to upgrade to. My example will be on Windows, but the concept should be similar for other operating systems.

If you look in the middleware home directory, you will likely see the following Java versions.

It is only the JDK which will require updating to Java 7, this is because JRockit does not exist after 6 so we will configure to only use the JDK.

If you are looking to download Java versions from Oracle support, then I recommend the document – “All Java SE Downloads on MOS (Doc ID 1439822.1)”.

I am going to be updating Java to the following version.

The patch is simple to install as it is just an executable.

I set the install path to be the middleware home directory, so it is in the same location as the existing Java versions.

There are now three Java versions.

The Java 6 directories were renamed.

Now the next step is to use “mklink” to create a symbolic link to the new Java location. The old Java 6 directories will be set as the link name. This means anything referring to jrockit_160_37 and jdk160_35 will be directed to the new Java installation, so no need to mess about trying to find and replace Java versions in files.  Also, if you update the version of Java in the future, it is as simple as recreating the symbolic links.

The syntax for “mklink” is:

mklink /d <link> <target>

/d - Creates a directory symbolic link.
<Link> -Specifies the name of the symbolic link that is being created.
<Target> - Specifies the path that the new symbolic link refers to.

It is as simple as running the following for the JDK.

mklink /d E:\Oracle\Middleware\jdk160_35 E:\Oracle\Middleware\jdk170_211

Then for JRockit:

mklink /d E:\Oracle\Middleware\jrockit_160_37 E:\Oracle\Middleware\jdk170_211

In the middleware home directory there are now two symbolic links created.

The properties show that the target is the Java 7 directory.

For example, if I go to the Java 6 JDK directory it might look like I am in that directory, but it is pointing to the Java 7 directory.

This makes life much easier as there is no need to go searching through files to repoint to the new Java installation.

Please note if your environment is across multiple servers you will need to update Java in each of them.

There is one more change that needs to be made before I can test. Each of the Java web applications will be accessing a Java file in a JRockit directory which will need pointing to the equivalent JDK directory.

To make this change, the Windows registry editor is required as well as the key for each of the web applications, for example:

There will be a “JavaDll” string pointing to a JRockit directory.

This can be updated to the equivalent JDK directory, notice that this is kept as the symbolic link location so no modifications are required if Java is upgraded again.

This change will need to be made in the registry for each of the Java web applications.

All done and ready to test once the services have started.

In FDMEE a new Cloud application is added.

This time the result is much better, and no error is displayed.

With TLS debugging enabled, the log this time confirms the communication is using the TLS 1.2 protocol.

[ACTIVE] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)', READ: TLSv1.2 Handshake, length = 87

To be sure everything is working as expected I tested out refreshing metadata.

This was successful.

Next, refreshing members.

No problems with this.

Now for a data load from on-premise FDMEE to EPM Cloud.

I am testing out with a numeric load first which will create a file that is in the format for a direct load to Essbase.

The file will be uploaded using the REST API and then again with the REST API a job will be run to load the data file.

I tested out the same data load but this time using the all data types method. This will create a file in the Outline Load Utility (OLU) format and the utility will be called to load the file through the planning layer.

Once again, no problems.

Next, on to extracting data from EPM Cloud and loading to on-premise FDMEE.

All good. I did also test a custom Jython script which makes REST API calls to EPM Cloud and there were no issues.

So all I have done is update the version of Java and everything seems to be working well.

If you are adamant in staying on Java 6 then it is a similar process to update. The only difference is that JRockit can be also be updated.

The final Java 6 patch versions are:

Patch 28414647: Oracle JDK 6 Update 211
Patch 28414796: JDK160_211 Oracle JRockit R28.3.20

If you follow the same process I have previously been through, then you should end up with something like:

Create the symbolic links using “mklink”

You should end up with the following:

There is no need to update the Windows registry as JRockit is still being used.

I carried out the same tests in FDMEE and did not come across any issues.

Oracle do suggest patching WebLogic but from my testing it didn’t look like it was required for TLS 1.2. I can only assume it is for enhanced security reasons.

If you look at the web application logs, you will notice how old the version of WebLogic is that is deployed by default with

<Version: WebLogic Server  Tue Nov 15 08:52:36 PST 2011 1441050 >

If you want to patch WebLogic, the process is pretty simple and there are lots of examples on the internet. Patches are applied using the BEA Smart Update (bsu) command line utility.

The patch I am going to apply is:

Patch 28710912: SU Patch 7HKN: WLS PATCH SET UPDATE

This was downloaded and extracted.

Then the patch can be applied with the BSU utility through command line. I will leave it to you to find out what each of the parameters are :)

If you have Java web applications deployed across multiple servers then the patch should be applied to each of these.

Once you start up a web application the log should contain the patch information.

<Version: WebLogic Server PSU Patch for BUG28710912 Tue Nov 13 06:00:38 PST 2018
WebLogic Server  Tue Nov 15 08:52:36 PST 2011 1441050 >

So there you go, updating Java might be enough to get you ready for the changes. Let me know if you see different results.