GoldenGate 12.3: Microservice Architecture (MA)

The Microservices Architecture (MA) for Oracle GoldenGate is a new REST API Microservices-based architecture that allows you to install, configure, monitor, and manage Oracle GoldenGate services using a web-based UI.

Really there are two versions of GoldenGate now: classic and microservice. Classic architecture has standard extract, replicat, pump and receiver. It is managed by classic ggsci. Microservice Architecture (MA) has different types of processes and managed using Admin Client or using web UI. See architecture of GoldenGate MA below

imageOracle GoldenGate MA is designed with the industry-standard HTTP communication protocol and JSON data interchange format.

Classic architecture was managed using ggsci console and had weak authentication and authorization tools. Oracle GoldenGate MA has ability to verify identity using basic authentication and using SSL client certificates.

GoldenGate MA processes

Oracle GoldenGate MA  uses different types of processes to perform same tasks as GoldenGate Classic. Let’s talk a little bit about new processes:

Service Manager. Something like (and replacement of) Manager process.  This is watchdog for other processes.

Administration Server. Something like ggsci console. Operates as central control entity. You use it to create and manage other processes. The key feature of Administration Server is REST API which can be accesses from any HTTP or HTTPS client.

You can add, delete or alter GoldenGate processes, edit configuration files, add users and assign roles using Administration Server.

Receiver Server. Something like collector. It can receive trail files from remote server. However it replaces multiple collectors because it is multithreaded. Receiver was designed to be protocol agnostic – so it supports HTTPS, HTTP, UDT (reliable UDP) and classic GoldenGate TCP transports. By default it uses HTTPS protocol.

Distribution Server. Something like pump.  But again this multithreaded process which can handle multiple trail at the same time. So it will replace multiple pumps. And again it supports multiple protocols: WebSockets for HTTPS-based streaming, which relies on SSL security, UDT, SOCKS5, HTTP. It also support Passive mode to initiate connection from remote side.

Performance Metrics Server. This is process which collects and saves information from other processes (extracts, replicats, etc). All GoldenGate processes push information to Performance Metrics Server. Now this is the only processes which writes data to GoldenGate datastore (Berkley DB). You can use Performance Metrics Server to query various metrics, view logs, process statuses, monitor system utilization, etc.

Admin Client. It is a command line utility (similar to ggsci) used to create, configure and manage processes. Admin Client uses REST API to accomplish its tasks.


Admin Client has much more function and very usable in distributed configuration than ggsci. See table below:

GGSCI Admin Client
Can manage local deployments only all MA deployment
Type of access to server ssh or local console http or https
Connection to DBMS Required NOT required
Security Operation System security MA security
Connected by default requires separate CONNECT command
Login to database using USERID+PASSWORD, USERIDALIAS USERIDALIAS
Sequence of extract commands REGISTER EXTRACT before ADD EXTRACT REGISTER EXTRACT after ADD EXTRACT
Security of communications unsecure secure (uses SSL)
Transfer data using Pump Distribution Server

GoldenGate MA key directories and variables

GoldenGate MA directory structure was revised according to Linux Foundation Filesystem Hierarchy Standard. GoldenGate installation consist of read-only and read-write directories.

imageSee description of each directory used by GoldenGate below:

Oracle Database Home (ORACLE_HOME). Points to Oracle Database home which is extracted or replicated by GG (/database_install_location)

Oracle GoldenGate Home (OGG_HOME). Read-Only directory contains GoldenGate binaries, libraries, etc (/ogg_install_location)

Deployment etc home (OGG_ETC_HOME). Directory containing configuration artifacts (/ogg_deployment_location/etc)

Deployment configuration home (OGG_CONF_HOME). Directory containing configuration artifacts. (OGG_ETC_HOME/conf)

Deployment security home (OGG_SSL_HOME). Directory containing security artifacts. (OGG_ETC_HOME/ssl)

Deployment variable home (OGG_VAR_HOME). Directory containing logs, reports, temporary files (?) (/ogg_deployment_location/var).

Deployment data home (OGG_DATA_HOME). Directory containing data (trail files) (OGG_VAR_HOME/lib/data).

You can change default location of stored files. Directory structure was designed to make upgrade easier.

Steps to implement Microservices Architecture

There are steps you need to finish before using GoldenGate MA:

Install MA

Start Service Manager

Start Services

Start and Use Admin Client

Let’s talk about each step.

Install MA

There is separate document describing Installation of GG MA (https://docs.oracle.com/goldengate/c1230/gg-winux/OGGIN/toc.htm)

1. First of all download installation media for GoldenGate MA from edelivery.oracle.com or otn.oracle.com. GoldenGate MA has separate distribution.

2. Unzip and start installation using runInstaller

3. Choose database version

4. Choose directory for installation.

Installation is very easy.

Configure Service Manager

Service Manager is starting point for activities related to GoldenGate configuration. For example you can use it to create or edit deployments. One Service Manager will manage multiple deployments of GoldenGate.

1. First of all lets set OGG_HOME variable. It will be used later

export OGG_HOME=/u01/app/oracle/product/gg/12.3_ma/db122

2. Change directory and run oggca.sh to configure GoldenGate Service Manager (it can be secure or unsecure, for example)

[oracle@scgg1gg123 bin]$ cd $OGG_HOME/bin
[oracle@scgg1gg123 bin]$ ./oggca.sh

3. First screen will ask us about general Service Manager parameters. I used separate directory for deployments  because it makes easier to upgrade GoldenGate. Also I chose to use port 10000 and to register Service Manager as daemon (good news!).  On the next screen choose to create new deployment. Then choose name for deployment and directory where deployment will be created.

imageimageimageimage

4. Enter Oracle Database specific variables. Then Enter information to create admin account

imageimage

5. For simplicity we will choose to not use SSL/TLS security.

image

6. Enter ports which will be used by different processes. It is easier to enter port for Administration Server and other ports will be enter by incrementing previous port by 1

image

7. Specify default schema which will be used by GoldenGate (for example, ggadmin as we use usually). Then check other parameter and finish configuration

imageimageimage

8. Run create script as root. Wait until it finished and press OK.

image

9. Now you can login to Service Manager and see running services.

imageimage

10. Service Manager restarts automatically If you reboot server. This is because configurator created daemon OracleGoldenGate.

image

11. Repeat everything in this section for the second database (on another or the same server)

Configure Credentials and checkpoint tables

GoldenGate uses database connections during its work. Also we need connection to database to create some types of processes, add trandata (supplemental logging) and create checkpoint tables. Let’s prepare credentials.

1. Login to GoldenGate Service Manager on the source server using URL  http://scgg1:10000/. Then click on Administration Server.

image

2. Enter menu by click it in the left upper corner and choose configuration

imageimage

3. Click “+” to create new credentials. Then enter required information.

Credential Domain (choose any name for domain): mydomain
Credential Alias (choose name for credentials new credentials, i.e. username_dbname): ggadmin_cdb1
User ID (username and tns string for database, i.e. username@hostname:port/service_name): c##ggadmin@localhost:1521/cdb1
Password and Verify Password: passwords

image

4. Test you credentials by clicking “Login database” icon:
image

5. If login was successful you will 3 new tables: Checkpoint, Trandata and Heartbeat. Let’s use them.

6. Create checkpoint table
image

7. Add trandata for you schema

image

8. Add heartbeat tables if you want (it is not required for GoldenGate but very useful)

image

9. Finally you will have the following picture.

image

10. Configure Global checkpoint table by clicking tab “Parameter file”. Choose GLOBALS and add line “checkpointtable ggadmin.checkpoint”

image

11. Click Overview to return to Administration Server configuration page.

12. Repeat previous steps on both source and destination databases.

Configure extract process

We will configure standard one-way GoldenGate replication. Everything will be done through web console. 

1. If you go to Administration Server we can see no extracts were created. Click on “+” button to create new extract. Then choose Integrated Extract. Click Next. Enter required information

imageimage

2. On the “Parameter file” page enter other parameter which required to run extract. For example, you can enter table specification. Then click “Create and Run button”

image

6. You can get error. In my case I entered incorrect username. Also I’ve set subdirectory for trail files but didn’t created it. So press OK, correct error and retry to start process.

imageimage

7. Then run some inserts and look into detatail by pressing Action->Details. See multiple tabs which contain different information for extract.

imageimageimage

Configure distribution path

Now we need to create process to distribute trail data to different server. This work is done by Distribution Service. So return to GoldenGate Service Manager (http://scgg1:10000/) and click Distribution Server. You will get to Distribution Server page

image

1. Let’s create new distribution path by clicking “+” button. Enter required information and click “Create and Run”. I chose ogg protocol to send data because it require no additional configuration. You can loging to destination host to know Receiver Server port. My port was 10003.

imageimage

2. We can look into statistics (Actions->Detail->Statistics) to see that all data generated early was sent.

image

3. Now let’s go to destination and create replicat to apply data. My destination GoldenGate instance is available at http://scgg2:10000/. We can check that data were received from source by clicking “Receiver Server”

imageimageimage

Configure Replicat

1. Return to GoldenGate Service Manager and click Administration Server. Then click “+” to create Replicat

image

2. Enter required information.

imageimage

3. Edit parameter file. For example, I’ve added MAP and INSERTALLRECORDS directive

replicat ra
useridalias ggadmin_orcl2 domain mydomain

insertallrecords
MAP orcl1.ggtest.*, TARGET ggtest1.*;

image

4.Click  button “Create and Run” and wait for staring process

5. Run some workload and check statistics of Replicat. It should work

image

Conclusion

We just finished configuration of GoldenGate in Microservice Architecture mode. It was done using web interface. This interface is great for one time task and demonstrating GoldenGate. However you should use AdminClient utility for production cases – it is more useful for production cases.

Добавить комментарий

Ваш e-mail не будет опубликован. Обязательные поля помечены *