EPM Cloud - Integration Agent Part 3 - Extracting data

Back again to continue with the series looking at the EPM Cloud Integration Agent. In the last part I went through getting up and running with the agent in asynchronous and synchronous modes.

Now that the agent is running, it is time to look at our first integration. In this post I will go through extracting data from a remote relational database and loading to EPM Cloud.

The first example will be against an Oracle database, but I will also cover SQL Server in the next post. The good news is you are not restricted to these two database types, you have the power to connect to any source.

Before starting out, you need to be sure that you can connect to the source database from the machine running the agent. I simply confirmed this by telnetting to the Oracle database server and port.

For simplicity, I have created a database view which queries the tables and data I am interested in and produces the desired output. It doesn’t have to be a view, data can be extracted using a SQL query against the source tables.

The next stage is to create a SQL query in Data Integration that will be executed by the agent to extract the data.

Go to Data Integration > Actions > Query

Add a new query.

Enter a new name for the query and the SQL statement.

You will notice that I have ~SCENARIO~ in the SQL statement. This is a bind variable and must be enclosed with a tilde, this allows you to filter the data extract. It is possible to set the value either at data load rule or application level. These can also be updated using custom event scripting and I will provide an example in the next part.

The most important part about the SQL statement is the column names. They will have to match with what is created in the data source which I will get on to shortly.

It is possible to use alias names for the column in the SQL query.

Once the query has been created it is possible to edit, delete or create additional queries.

The next step is to create a comma separated file with one line containing all the possible columns that can be returned from various SQL queries. The file I have created has more columns than the SQL query above, this is because I intend to have additional queries that return different data sets.

As I just mentioned it is important that the names match to those that will be returned from the SQL queries.

The name of the file will become the name of the target application in Data Management. I know it is confusing calling it a target application when it is a source.

The mindset that you need to be in is to treat the integration as a file-based integration even though you are connecting to a relational source. The agent will execute the SQL query against the source database, extract the data and create a file, this file will contain column headers that match the columns in the SQL query. The file will be automatically uploaded to the cloud and imported where the column headers in the file will be matched to what has been defined in the cloud.

It should become clearer as I go through the full process.

To create the Data Source, go to Data Management as there is not yet full parity in Data Integration.

Go to Target Application > Add > Data Source

From the dropdown select “On Premise Database”, this doesn’t mean the database must be on-premise, it can cloud or on-premise if the agent has good connectivity to it.

This will open the file selection window. Upload the file with the column names and select it.

The file name now has been converted into the application name. A prefix can be added if there is already an application with the same name.

An application will be created with dimensions based on the header columns in the selected file.

I know it is a little confusing because they are really source columns and not dimensions.

The target dimension class should be left as “Generic” and no “Data Table Column Name” assignments are required. If your SQL extracts are going to include additional columns, then you can manually add them at any point to the application.

Next on to the Application Filters tab which provides options to configure the connection to the source database.

Data Extract Query – This is the name of the SQL query to execute. It is possible to define this at application or rule level.

Delimiter – This will be the delimiter that is used when the agent creates the data extract file.

Credential Store – This provides the option to either store database connection information in the cloud or in a file. If “Cloud” is selected the connection information is entered in the application filters. If File is chosen, a file will need to be created in the agent’s application data config directory in the same location as the agent initialisation file. The database user’s password in the file will require to be encrypted using the “encryptpassword” script.

For this example, I am going to go for the “Cloud” option and will cover “File” in the next example.

JDBC Driver – This is the driver for connecting to the source database. Currently there is only “Oracle” and “Microsoft SQL Server”, but I am sure this will increase in future releases. This doesn’t mean you can connect to only those database types though, with the use of event scripting you can connect to pretty much any source.

JDBC URL – This is the database connection string. The Oracle syntax I will be using is

jdbc:oracle:thin:@dbhost:port/service

Username/Password are for the user you want to connect to the database with.

Fetch Size – This is the number of records that the JDBC driver fetches with each call to the database when executing the SQL query.

This is my completed set of property values.

In my SQL query I was filtering on scenario and included a bind variable. To create a bind variable, select “Edit”.

This will then display all the application filters.

A new filter is added for the bind variable. The name must match to that in the SQL query. The prompt will be displayed at the display level, which can be application/rule.

The display order should be a value which is not already being used. It will be the order they are displayed in the UI. The validation type should be set to “None”.

I also updated the display level for the data exact query to be “Application and Rule”. This will mean you can override the query defined at the application in the data load rule.

The integration can now be built up as if building a standard file-based one.

The delimiter should match what has been set in the application filters.

To keep things simple, I have used expressions to map the source values directly to the target so no need for member mappings.

The data load rule includes the bind variable and data extract query as they were set at rule level.

I set scenario to “Actual” which will mean when the SQL query is executed this value will be used as the bind variable.

The equivalent in Data Integration is in Options and part of the Filters tab.

Now we are ready to run the data load rule. The agent is currently running in asynchronous mode and polling the cloud every 30 seconds.

The import stage for the integration is run.

In process details you will see the process is running.

In the process log you can view the database extract process.

10:00:56,078 INFO  [AIF]: ************************************************************************************
10:00:56,078 INFO  [AIF]: *On Prem DB Extract: Started                                               *
10:00:56,078 INFO  [AIF]: ************************************************************************************
10:00:56,203 DEBUG [AIF]: Adding DB credentials
10:00:56,204 DEBUG [AIF]: Adding DB bind variables
10:00:56,205 DEBUG [AIF]: Adding DB query
10:00:56,207 INFO  [AIF]: Original Extract Query:SELECT ENTITY,PRODUCT,ACCOUNT,AMOUNT 
FROM V_ACTDATA 
WHERE SCENARIO=~SCENARIO~
10:00:56,207 DEBUG [AIF]: Adding delimiter
10:00:56,208 DEBUG [AIF]: Adding fetchSize
10:00:56,212 INFO  [AIF]: Retrieved EPM Cluster name:EPMAGENT_ASYNC
10:00:56,213 INFO  [AIF]: Retrieved Cluster Mode:ASYNC
10:00:56,216 INFO  [AIF]: Calling agent extract ASYNC mode: BEGIN
10:00:56,216 INFO  [AIF]: Request Payload Length:469
10:00:56,231 DEBUG [AIF]: Updating process steps: QUEUED
10:00:56,234 DEBUG [AIF]: Agent Status: QUEUED

The database credentials, SQL query and bind variable, delimiter and fetch size are collated. This information will be passed to the agent once it requests the information.

The cluster name and mode are retrieved, the process is queued until the agent makes the next poll to the Cloud.

The agent makes a poll to the Cloud and as a process has been initiated, the response from the Cloud will return the Process (Job) ID.

As a Process ID has been returned the agent now makes a request to the Cloud for the details of the job. The information collated earlier is returned to the agent (all encrypted).

The agent will connect to the source database, execute the SQL query with the bind variable value, create an extract data file, upload the file to the Cloud, and upload the process log.

The details of this process are contained in the agent process log. The log includes the SQL query and the bind variable value.

10:01:06:086 :: *************** Starting execution for sessionId:945 at: Oct-06 10:01:06:086***************
10:01:06:086 :: Process Extract Request with data:--------------Executor Request Data--------------
 jobType:EXTRACT
 sessionID:945
 Data Source Name:EPMAGENT_ORA_DEMO
 credStore:CLOUD
 query:SELECT ENTITY,PRODUCT,ACCOUNT,AMOUNT 
FROM V_ACTDATA 
WHERE SCENARIO=~SCENARIO~
 outputType:FILE
 delimiter:,
--------------Bind Variables--------------
Name: SCENARIO, Value:Actual, Type:STRING
--------------------------------------------
10:01:06:086 :: --------------- Executing Custom Event :BefExtract ---------------
10:01:09:539 :: Script: C:\EPMAgent\appdata\scripts\BefExtract.py Not found.
10:01:10:148 :: Adding String Bind Variable: SCENARIO with value: Actual
10:01:10:227 :: Writing extract output to file: C:\EPMAgent\appdata\data\945.dat
10:01:10:258 :: --------------- Completed execution for sessionId:945 ---------------
10:01:10:258 :: --------------- Executing Custom Event :AftExtract ---------------
10:01:10:258 :: Script: C:\EPMAgent\appdata\scripts\AftExtract.py Not found.
10:01:10:258 :: --------------- Executing Custom Event :BefUpload ---------------
10:01:10:258 :: Script: C:\EPMAgent\appdata\scripts\BefUpload.py Not found.
10:01:10:258 :: Commencing file upload to cloud for file:C:\EPMAgent\appdata\data\945.dat
10:01:11:602 :: File Upload successfully complete
10:01:11:602 :: --------------- Executing Custom Event :AftUpload ---------------
10:01:11:602 :: Script: C:\EPMAgent\appdata\scripts\AftUpload.py Not found.
10:01:12:148 :: Commencing file upload of log file:C:\EPMAgent\appdata\logs\945.log
10:01:12:242 :: *************** End execution for sessionId:945 ***************

The agent has been started with the Jython home variable populated.

As this has been populated the agent will attempt to run the four event scripts. I have not created any event scripts yet so no events will be executed. I will be starting to introduce event scripts from the next post.

If the Jython home property was left blank the agent would not attempt to run the scripts.

If the Groovy home property was defined, the agent would attempt run four Groovy event scripts.

The process log is generated in the application data logs folder along with the main agent log. The format for log is .log

The agent process log is uploaded to the Cloud and appended to the main process log.

INFO  [AIF]: Appending EPM agent log to process: BEGIN
INFO  [AIF]: Appending EPM agent log to process:: END

The data extract file is generated in the application data folder. The format for the data file is .dat

The extract file will include column headers.

The column headers are matched in the Cloud. This is why it is important that the columns returned by the SQL query are the same as the ones defined in the application.

DEBUG [AIF]: EPMFDM-140274:Message - Datasource Column Position in header: AMOUNT -> 4

The remaining steps of the import and validate process will be same as file-based integration.

For each cluster there will be additional queue and process information available in Data Integration.

If the agent is running in synchronous mode, the only difference would be the Cloud makes a direct call to the agent through the Web URL to execute the job. There, process ID is sent directly to the agent so there is no waiting for a poll.

So now we can extract data from a remote Oracle database and load this to EPM Cloud. This is all done using default functionality and no custom scripting.

In the next part I will extract data from a SQL Server database and start to look at event scripts.

EPM Cloud - Integration Agent Part 2 - Get up and running

Moving swiftly on to the next part in this series on the EPM Cloud Integration Agent. In the last part I went through an introduction to the agent and in this post, I am going to dive straight in and go through the process of getting the agent up and running.

I will be going through an example on a Windows machine, but the concept will be similar on Linux/Unix type operating systems.

A summary of the tasks is:

• Install JRE 8+
• Install Groovy or Jython (optional)
• Download Agent
• Extract Agent
• Create application folders
• Generate encrypted password for EPM Cloud
• Populate agent initialisation file
• Copy EPM Cloud SSL certificate to agent folder
• Start agent

The agent requires a Java Runtime Environment (JRE) 8+ to run. JRE 1.8 update 221 was downloaded and installed.

If you want to use custom scripting and override default functionality in the form of event scripts, then either Groovy or Jython can be installed. If you want to stick with the default functionality and no scripting, then there is no need to install. It is also possible to implement your own Java classes without the need for Groovy/Jython.

I am not going to say which to install as it is personal preference or maybe company policy that defines this. If you work with on-premise FDMEE then you are probably going to be more used to Jython, for this reason I am going to install Jython but will no doubt cover both in future blog posts.

Currently the latest version of Jython is 2.7.1 and can be downloaded here.

To install Jython, the jar file can simply be executed, and the installation steps followed.

Next, the agent needs to be downloaded from Data Integration.

Go to Data Exchange.

Select Actions and there will be an option to download the agent.

Once downloaded, the zip file simply needs to be extracted to a suitable location.

The bin directory is where you need to start to configure and run the agent. There are scripts available for both Windows or *nix type platforms.

“createAppFolder” – will create the application folder structure, it can be generated within the same or separate location to the agent. It is possible to create multiple application folders if you want to run more than one agent and keep them segregated.

For simplicity, I am going to create the application folders in the same location as the agent in a directory named “appdata”.

Running the script will create the following structure.

The config directory contains an initialisation file for the start parameters of the agent, this will need populating before starting the agent, I will get on to that shortly.

The data directory will hold the data files that are created after extracting the source data. The files will be uploaded to EPM Cloud as part of the integration process. You will need to consider housekeeping to maintain this directory over time.

The logs directory will contain a log for the agent and each process, the process log also gets uploaded to EPM Cloud and appended to the main process log.

I will be covering the data and logs in the next part when I look at extracting data.

The scripts directory can contain any of the four available event scripts. As I mentioned earlier, event scripts are optional and once again I will be covering them in a later post.

Back to the agent initialisation file, which from now on I will refer to as ini file.

The file will be automatically populated with the EPM_AGENT_HOME and EPM_APP_DATA_HOME property values.

The EPM_APP_DATA_HOME directory is the one created earlier with the “createAppFolder” script.

AGENT_NAME is a unique name for the agent and can only contain alphanumeric characters. It is optional to define it in the file as it can set at runtime.

CLUSTER defines the name that the agent belongs to. Multiple agents can belong to a cluster and there can be one or many clusters. Like the agent name it can only contain alphanumeric characters.

Clusters provide a way of organising integrations. It may be that you want to use a cluster for only certain integration types, for example one cluster for extracting data from an Oracle database and another for connecting to other cloud providers. You can assign rules, locations or target applications to clusters. I plan on covering this in more detail in a future post.

Clusters are created in Data Management and can either be defined to operate in synchronous or asynchronous modes. I gave an overview of these modes in the previous post.

A cluster must exist before starting the agent. By default, the ini file will be populated with a cluster named “EPMCLUSTER”, this cluster also pre-exists in Data Integration.

Just for demo purposes I am going to have two clusters, one for each type of mode.

To create, view, edit and delete clusters, first go to Data Exchange.

Then select Agent.

There will already be “EPMCLUSTER” created, which has been assigned to asynchronous mode. I deleted this cluster, but it is your choice if you want to keep and use it.

Now a cluster can be added.

Two clusters were created, one for each mode.

I am going to keep it simple and configure the agent in asynchronous mode first, then update the ini file and restart the agent in synchronous mode.

I could have easily had two ini files, one for the agent in asynchronous and the other for synchronous.

Moving on to the next property in the ini file.

PORT is the port the agent will run on and the default is 9090.

CLOUD URL, DOMAIN and credentials should be easy to understand.

You will, however, need to encrypt the users’ password with “encryptpassword” script in the agent bin directory.

The encrypted password must be manually copied to the ini file. It is worth noting that every time the cloud password expires or is changed, the password will need to be encrypted again and the ini file updated.

It is a shame that the encryption method is not the same as EPM Automate. This means if you are running EPM Automate on the same machine as the agent, you will need to manage multiple encrypted passwords. In a perfect world, the encrypted password could be held in one place and be shared between the agent and EPM Automate.

On to the remaining properties.

LOG_LEVEL can be either ALL, INFO or ERROR.

POLL_INTERVAL is for asynchronous mode only, it is the time interval in seconds that the agent checks whether a process has been initiated in the cloud.

REGISTER_WITH_IP defines whether to register the agent against the host name or IP address.

EXECUTION_POOL_SIZE is the number of executions that the agent can run in parallel.

JYTHON_HOME and GROOVY_HOME is optional, they specify the installation location. Earlier I installed Jython to C:\jython2.7.1\ so this is used for the JYTHON_HOME parameter value.

CUSTOM_MEM_ARGS sets the minimum and maximum JVM memory sizes. The default values are just a guide and can be tuned after monitoring memory usage from running integrations.

CUSTOM_JAVA_OPTIONS allows you to add additional Java parameters, for example if the agent needs to go through a proxy to reach EPM Cloud.

CUSTOM_CLASS_PATH provides the option to add additional jar files to Java classpath, for example additional JDBC drivers.

CUSTOM_INTERFACE_CLASS_NAME is only required if custom Java classes are implemented to override default functionality through events.

Here is an example of my completed ini file.

There is one remaining task required before being able to start the agent. The EPM Cloud SSL certificate needs to be exported and copied to the agent cert folder.

I am going to use an example using Chrome, but it is a similar process for other browsers.

Log into one of your EPM Cloud instances and click the padlock icon.

Select Site settings > Certificate

This will open the certificate

Click the Details tab and select ‘Copy to File’.

Save the file to the agent cert folder.

Please be aware that the SSL certificates will expire so make a note of the valid date, you will need to replace it close to the expiration date or the agent will stop working.

Now we are ready to run the agent in asynchronous mode.

Update 19.12 release - The agent can now be run as a windows service. I have put together a post about this and you can read all about it here.

If you want to start the agent from command line, then you can use the “epmagent” script in the agent bin directory.

The syntax is:

epmagent <path to ini>

if the agent name is not defined in the ini file or you want to override it, you can set the agent name at runtime with the following syntax:

epmagent <agent name> <path to ini>

When the agent starts up, the initialisation file and SSL certificate will be read. The agent will be registered in the cloud against the cluster.

If you go back into Data Integration and select the cluster you will see the agent has been registered.

The Physical URL will be the machine and port the agent is running on.

In asynchronous mode the Web URL is not important, this only comes into play with synchronous mode, which I will get on to shortly.

As the agent is running in asynchronous mode, it will poll the cloud to check whether any processes have been initiated. I had set the interval poll to be 30 seconds.

Each time a poll is made the “Last Ping” will be updated in the Cloud. At least this gives some indication that agent status is healthy.

The agent name can be defined at start-up which will override the ini file.

There will now be two agents registered against the cluster even though only one is running.

Agents will stay registered against a cluster, if they are no longer required, they can be simply deleted from the cluster in Data Integration.

You might be wondering whether there is an option available to run the agent as a Windows service. I have raised this and apparently it is being looked into, so it might appear in a future release.

Let us move on to synchronous mode. I am just going to update the ini file to change the cluster name. I could have run both the agents at the same time, but I will be covering multiple agents, clusters and assignments in more detail in a future post.

I restarted the agent so it will pick up the new cluster name.

You can see that the agent mode is now synchronous.

In Data Integration the agent will be registered.

In the last post I provided an overview of the two modes so please refer to that for more details, but as the agent is running in synchronous mode the cloud requires an inbound connection over HTTPS to the agent. This would usually be achieved by running a load balancer or https with an internet facing URL which will proxy to the agent and port.

In my testing, I have a domain name routing to an AWS application load balancer. The Load Balancer has an SSL certificate applied which is registered against the domain name. The load balancer has been configured to only accept HTTPS traffic from Oracle Cloud, this is then proxied to the machine running the agent. The machine running the agent only accepts traffic on the agent port from the Load Balancer.

As you can see, when the agent is first registered with the cloud the Web URL is the same as the Physical URL. The Web URL can be updated by typing the internet facing URL for the agent.

It would be good if there was an option to also set the Web URL from the agent side when it starts up. It is possible to update using REST which will be covered in a separate blog post.

At this point there is no option to check whether EPM Cloud can connect to the agent. It is not until the point when you run an integration that you will find out, though to be honest depending on the problem there is not a great amount of detail in terms of logging. Personally, I think there should be an option to test the agent and check the connectivity. Maybe one for a future release?

It is possible to put the internet facing URL for the agent into a browser and check whether you get a valid response, this doesn’t guarantee that it will work from the cloud though.

You really need to decide whether you need synchronous over asynchronous. Are you happy to wait for polling the interval which can be set or do you have to run processes instantly? Synchronous mode will certainly require working closely with a network team.

Anyway, now that we have the agent running it is time to move on to extracting data. Watch out for the next part arriving very soon.

EPM Cloud - Integration Agent Part 1 - Introduction

One of the problems with data integration in EPM Cloud is that the list of data sources has been quite limited. Unless you want to integrate with an Oracle source like Financials, HCM or NetSuite then you are restricted to generating a file in the correct format and uploading. If there is the option of on-premise FDMEE then the hybrid functionality is a workaround for these limitations. Unfortunately, this is not viable for the majority of EPM Cloud customers.

These limiting factors have almost all been eliminated with the release of the Integration Agent which has arrived in EPM Cloud 19.10.

Here is an excerpt from the 'What’s New' documentation providing an insight to the details of the agent.

“The EPM Integration Agent, which provides connectivity between the EPM Cloud and on-premises sources including SQL data sources like Oracle and Microsoft SQL Server, E-Business Suite GL, and PeopleSoft GL, is now available:

In addition to easy access and setup for these sources, the agent may also be extended using scripting to connect to virtually any data source including 3rd party REST API based systems, or systems that can be accessed using a Jython or Groovy script. The agent, implemented as a data source for Data Management and Data Integration, responds and provides the specified information when a user executes data load rules.  Agent based integrations may be scheduled like any other data rule which provides users the flexibility to define integrations to any source in an automated fashion.”

One thing I disagree with in the first paragraph of the above statement is the reference to on-premise. The SQL data source can be in the cloud or on-premise, it doesn’t matter as long as the agent has good connectivity to it. The same goes for the agent, it can be run in the cloud or on-premise with the basic requirement that it has HTTPS connectivity to EPM Cloud.

The beauty of the agent is that with either Jython or Groovy custom scripting it goes beyond the default functionality of connecting and extracting data from SQL sources. It has opened up a whole new area of possibilities and I know there is going to be a lot of interest in it.

I personally think the release of the agent is one of the biggest enhancements in a long time for EPM Cloud integration.

In this first part I am going to give a quick overview of the agent before moving on to getting up and running with it. There is no point in overloading with too much information at this stage as it will all start to make more sense from the next post. There is a lot to cover so I am unsure how many parts there will be.

The agent is a lightweight Java application which is similar in concept to EPM Automate. As it is Java-based, it can run on most operating systems. Currently it stands at just over 6MB in size. The agent acts as a HTTP server and servlet engine with the use of Jetty, no surprises that the bulk of the functionality is built around a REST API.

In my testing I have been running the agent on a low tier AWS machine which has network connectivity to EPM Cloud and various data sources.

The agent requires a JRE to run which should be version 8+ and is not included in the download. This means either a supported JRE will need to be installed before running the agent, or if you are concerned about Java licensing restrictions and EPM Automate has been installed on the same machine, it is possible to point to the version of Java that is deployed with EPM Automate. I can imagine in a lot of cases the agent will be run on the same machine as EPM Automate.

There are configuration parameters for the agent that are defined in a file, I will be covering this in the next post.

The agent can currently be downloaded from Data Integration but I believe the longer-term plan is to move it into the downloads section but we will see.

By default, the agent will connect to relational data sources using JDBC and execute SQL queries to extract the required data set. A data extract file will be generated which is uploaded to EPM Cloud. The SQL query is defined in Data Integration. The database connection can either be set up in Data Management or in a file stored in the agent config directory.

There is also the option to extract data from EBS and Peoplesoft which are basically pre-seeded SQL queries. Though to be honest it might be better to fine tune the SQL to fit your requirements and use that instead.

There is no requirement to use the custom scripting option if you want to stick with default functionality to extract data. To be able to use custom scripting then either Jython or Groovy will need to be installed on the machine hosting the agent.

If you have used custom scripting in on-premise FDMEE then it is very similar to event scripts. The scripts allow you to override the default functionality and connect to any data source. I will cover this in a future post.

There are four events where custom scripting can be applied:

• Before extract – Before the agent connects to the data source and executes the SQL query
• After extract- After the agent has connected to the data source and produced an extract data file
• Before upload – Before the data file is uploaded to EPM Cloud
• After upload – After the data file has been uploaded to EPM Cloud.

Processes are executed in the same way from either Data Management, Data Integration, EPM Automate or REST.

It is possible to run multiple agents and cluster agents, once again I will cover this in a later post.

The agent can run in two different modes, the first being:

Asynchronous mode 

This is probably going to be the most common type of deployment as the agent only requires outbound connectivity over HTTPS to EPM Cloud, similar to EPM Automate requirements.

In this mode the agent will poll EPM Cloud using REST to check whether a load rule has been executed. The poll interval is defined in the agent configuration file.

If no rule has been executed the REST response will not return a Job ID, nothing will then happen until the next time the agent polls the cloud.

If a rule has been executed a payload is returned containing the Job ID. The agent makes another REST call with the job ID and the response contains the connectivity information, the SQL data source and the SQL query to execute.

The agent will then connect to the data source, execute the query and generate a data file. This file is uploaded to EPM Cloud using REST.

A REST call is made to inform that the process was successful.

The agent process log file is also uploaded to the cloud and appended to the main process log.

Standard functionality then takes over and the data file will be imported and validated.

Synchronous mode 

This requires EPM Cloud to connect over http(s) to the agent. This would usually be fronted with an internet facing load balancer or HTTP server which directs traffic to the EPM Agent on the designated port. The machine running the EPM Agent would not need to be internet facing and would be configured to only accept traffic from the load balancer/HTTP server.

An example being:

It does not have to follow the above design, there could be just a load balancer or a HTTP server or both. It is even possible to connect directly to the agent but that is not something I would promote for security reasons.

In my testing I ran an AWS application load balancer over HTTPS, this proxied requests to the EPM Agent machine and port. The machine hosting the agent was configured to only accept traffic from the load balancer.

What I will say is currently there is not much in the way of functionality or logging when testing the agent in synchronous mode. If you are considering this option, then you will need to work closely with a network team to configure, and put monitoring in place in case of configuration issues.

In Synchronous mode when a data load rule is executed, a REST request is made from EPM Cloud to the internet facing URL for the agent. The payload of the REST call contains the job ID. The agent then operates in the same way as Asynchronous mode. A request will be made based on the Job ID for the SQL connectivity and query information. The query is executed, file produced, uploaded to the cloud. The process status is updated, and process log file uploaded.

I think the decision of which mode to use really depends on whether you want to go down the route of configuring an internet facing solution and being able to run a process instantly or can wait for the poll interval. Remember the poll interval can be defined at agent level.

I will go into much more detail of how the asynchronous and synchronous modes work in a future post.

Communication between the agent and EPM Cloud is not only secured using HTTPS and TLS but also the REST payloads are encrypted using a public/private key management system.

Each time the agent is started it will generate a private and public key pair.

The private key is stored in the agent’s memory. The public key is uploaded to EPM Cloud and stored.

The REST payloads from EPM Cloud are encrypted using the public key. This can then be decrypted with the private key of the agent.

This means the agent can only be paired with one EPM Cloud instance and any attempt to communicate with the agent without the key and encryption will not work.

With that brief intro covered it is time to move on to the good stuff and get the agent up and running. Part two is available here.