Time for the next instalment in the series looking at the EPM Cloud Integration Agent. In this post I will look at running multiple agents and then move on to cluster assignments. I am going to assume that you have been following my previous posts on the agent and you understand the concept of agents and clusters. If not, I recommend reading through the first two parts.
Up to now, my examples have been using a single agent which has been assigned to either an asynchronous or synchronous cluster. To provide high availability and load balancing options you may wish to run multiple agents which can be assigned to one or more clusters. The agents could coexist on a single server or be spread across numerous servers. Depending on where the source data resides, the agents and clusters could be deployed across different geographical locations.
In this first example I am going to run two agents on one server and another two on a server in a different location. They will all be assigned to a single asynchronous cluster.
For simplicity, I am going to name the agents from AGENT1 through to AGENT4.
An asynchronous cluster has been created in Data Integration which the agents will be registered to.
The first agent is configured using an initialisation file and assigned to the above. Please refer to part two of this series for more details on all the parameters in the ini file.
The agent is started, registers to EPM Cloud and starts polling the job queue. Notice the agent is running on machine 1.
For the second agent, the ini file is copied to a new name. The agent name is set as AGENT2 and because the first agent will be occupying port 9090, the port for second agent is updated to 9095.
The agent is started and registers successfully, then starts polling.
Within Data Integration you can see that both the agents have been registered.
There is a slight difference that the agent uses a 24 hour clock format while Data Integration displays a 12 hour format.
On to the second server. The third agent is set up a similar way to the first agent. The only difference is the agent name.
The fourth agent is set up, once again the port has been updated to avoid a clash.
In Data Integration, the four agents are now registered across two servers.
When using asynchronous mode, the agent to first to make a poll to EPM Cloud after an integration has been initiated will be the agent that runs the process.
To demonstrate this, the first agent was started with a 60 second polling interval.
The second agent was started around 30 seconds later, again with a 60 second polling interval.
An integration was started.
AGENT1 makes the first poll after the integration has been executed so this agent runs the process.
AGENT2 makes a poll next and, as there have been no additional integrations started, it does nothing until the next poll interval.
If an integration had been started, then the next agent to poll would start the integration process.
In Data Integration, the queue shows that the first agent processed the integration.
If frequent integrations are being executed, then running multiple agents helps spread out the processing load between the agents. Also, if any of the agents are down or need restarting or if one of the servers is out of action, then it does not impact the integrations being run from the Cloud.
Moving on to running multiple agents in synchronous mode.
In the next example I am going to run three agents on the same server, but these could as easily be spread across multiple servers and locations.
A cluster has been created in Data Integration which operates in synchronous mode.
The first agent is configured using the same initialisation file as the previous example, but set to register with the synchronous cluster.
The second is set to register in sync mode and run on a different port.
The third agent, still running on the same machine but on a free port, registers in sync mode.
The registered agents can be viewed in Data Integration against the synchronous cluster.
As this is synchronous mode, the agents must be accessible through an internet facing URL. Just like in part two of this series, I configured a load balancer which was available over HTTPS through an internet facing URL. The load balancer was configured to only allow HTTPS traffic from Oracle Cloud. Depending on the agent URL, the load balancer would proxy through to the correct agent. The machine running the agents only accepts traffic from the load balancer on the designated ports.
In synchronous mode, agents are assigned jobs in a round robin technique using the order they are registered.
To demonstrate this, I registered AGENT3 first, then AGENT2 and then finally AGENT1.
An integration was then run. In the process log you can see the cluster and mode are retrieved, the next agent in the queue is assigned.
INFO [AIF]: Retrieved EPM Cluster name:EPMAGENT_SYNC
INFO [AIF]: Retrieved Cluster Mode:SYNC
INFO [AIF]: Calling agent extract SYNC mode: BEGIN
INFO [AIF]: Getting agent name from queue...
INFO [AIF]: Cluster Name:EPMAGENT_SYNC
INFO [AIF]: Retrieved EPM Agent name:AGENT3
As AGENT3 was the first agent to be registered, the integration is run against it.
The integration is run again, the process log again shows which agent is going to be assigned to process the integration.
INFO [AIF]: Getting agent name from queue...
INFO [AIF]: Cluster Name:EPMAGENT_SYNC
INFO [AIF]: Retrieved EPM Agent name:AGENT2
As expected, the next in line is AGENT2.
Three agents have been registered so let us run the integration again and check the process log.
INFO [AIF]: Getting agent name from queue...
INFO [AIF]: Cluster Name:EPMAGENT_SYNC
INFO [AIF]: Retrieved EPM Agent name:AGENT1
No shock that AGENT1 is assigned the integration.
In Data Integration you can see the queue order for the processes that have just been run.
To be complete, the integration is run one final time and it is back to AGENT3 processing the integration.
That covers multiple agents running in both asynchronous and synchronous modes.
Time to move on to cluster assignments, which provide the ability to assign integrations to different clusters. They can be assigned at either Integration (data rule), Location or Target application level.
As an example, I have created two clusters. One cluster will be designated to run integrations where the source is relational databases, the other will be for cloud data sources.
It doesn’t matter if the clusters are synchronous or asynchronous. The clusters can be registered with one or more agents and will operate in the same way as my previous examples.
I am going to keep it as simple as possible and there will be one agent registered to each cluster. The agents will coexist on a single machine. The clusters have been defined in asynchronous mode.
The cloud agent is configured with the initialisation file to point to the cloud cluster.
The agent is started and registers successfully.
The SQL agent is configured to the SQL cluster.
The agent is started and registers successfully.
In Data Integration, the cloud agent is shown to be registered to the cloud cluster.
The SQL agent is shown to be registered to the SQL cluster.
Within the agent cluster page there is an assignments tab.
The options available are to assign a type and value. Assignments can also be added or deleted.
The type defined the level of integration to assign to the cluster. These can be set at either Application, Integration (data rule) or Location level.
For this first assignment I am going to set it at Location level. Once this has been selected, the available locations are can be selected from the entity dropdown.
I selected “EPM_AGENT_ORA” which is an integration with a source Oracle database. The integration was covered in part three of the series.
I added another assignment at integration (data rule level) which extracts workforce data from a source SQL server database. This integration was covered in part four of the series.
This means if any integrations are run at the defined location or integration level, they can be processed by the agents that have been registered to the SQL cluster.
For the cloud cluster I added an assignment at location level.
This location includes an integration which extracts exchange rates using an API and was covered in part five.
Now the assignments have been applied, I ran the integration to extract exchange rates.
In the process log you can see the cluster assignments are retrieved and integration is assigned to the cloud cluster.
22:12:26,732 INFO [AIF]: Getting agent name from assignments...
22:12:26,732 INFO [AIF]: Getting agent from cluster assignments using Rule Id:77 Location Id:77 Source AppId:68
22:12:26,733 INFO [AIF]: Retrieved EPM Cluster name:CLOUD_CLUSTER
22:12:26,734 INFO [AIF]: Retrieved Cluster Mode:ASYNC
22:12:26,734 INFO [AIF]: Calling agent extract ASYNC mode: BEGIN
The queue also shows that the integration has been assigned to the cloud cluster. As the cluster is set up in asynchronous mode, there is no agent shown yet until an agent from the cluster makes a poll to EPM Cloud.
If the cluster was defined in synchronous mode, an agent would be assigned and the integration would be run directly against that agent.
The SQL agent makes a poll to EPM Cloud first. As this agent is not registered to the cloud cluster, no job details are returned from the Cloud and no action will be taken.
As the cloud agent is registered to the cloud cluster, the next time it makes a poll to the queue, job details are returned, and the agent executes the integration process.
Next to run an integration which extracts data from a source Oracle database. The location the integration is part of has been assigned to the SQL cluster.
The queue shows the integration has been assigned to the SQL cluster.
The SQL agent is registered to the SQL cluster, next time the agent makes a poll to the queue, job details are returned, and the agent processes the integration.
Now on to running the integration which extracts workforce database from a source SQL server database.
This integration has been assigned to the SQL cluster so appears in the queue.
The next time the SQL agent makes a poll, job details will be returned, and the agent will process the integration.
Assignment types have an order of precedence when integrations are initiated. Integration takes precedence over location, location takes precedence over an application.
To demonstrate this, I added the integration that extracts data from a source Oracle database to the cloud cluster.
This means the integration is assigned to the cloud cluster and the location that the integration is part of is assigned to the SQL cluster.
The integration was run again.
This time the integration is added to the cloud cluster queue. This is because integration takes precedence over location.
The cloud agent then runs the integration process.
So what happens if an integration is run that does not fall under any of the assignments?
I removed the workforce extract integration from the SQL cluster.
This means the integration is not covered by any of the assignments.
When the integration is run it instantly fails.
Checking the process log highlights that no cluster assignments were found so the process fails.
INFO [AIF]: Getting agent name from assignments...
INFO [AIF]: Getting agent from cluster assignments using Rule Id:75 Location Id:76 Source AppId:62
INFO [AIF]: Retrieved EPM Cluster name:null
ERROR [AIF]: Cluster not found for process id:1176
ERROR [AIF]: Unexpected error in importData:Cluster not found for process id: 1176
I think that should cover enough detail about running multiple agents and cluster assignments.
Coming up next, drill down to source data.
No comments:
Post a Comment
Note: only a member of this blog may post a comment.