GoldenGate 12.3.0.1.2 was released

GoldenGate 12.3.0.1.2 (or December release) was uploaded to edelivery.oracle.com. If you use GoldenGate Microservice Architecture then it is essential to download new version because it has some new features and bugs fixed.

I couldn’t find instructions how to upgrade GG 12.3 Initial Release to GG 12.3.0.1.2 December Release. So I chose the following method:

1) Install GG 12.3.0.1.2 to new directory (for example to directory /u01/app/oracle/product/gg/12.3_ma/db122_december  — this is GG for DB 12c)

2) Move existing directory (I moved /u01/app/oracle/product/gg/12.3_ma/db122 —> /u01/app/oracle/product/gg/12.3_ma/db122_initial)

3) Create link  for old directory which will point to last installed version (link /u01/app/oracle/product/gg/12.3_ma/db122 ——> /u01/app/oracle/product/gg/12.3_ma/db122_december)

Restart everything. You should see new version in Service Manager

image

Introduction and configuring Oracle Data Integrator for Big Data (Cloudera Hadoop)

imageApache Hadoop is designed to handle and process data that is typically from data sources that are non-relational and data volumes that are beyond what is handled by relational databases.

Oracle Data Integrator is a transparent and heterogeneous Big Data Integration technology based on an open and lightweight ELT architecture. It runs a diverse set of workloads, including Spark, Spark Streaming and Pig transformations, to enable customers solve their most complex and time sensitive data transformation and data movement challenges. It is a core component of Oracle Data Integration solutions, integrating seamlessly with the rest of Oracle’s Data Integration and Business Application solutions

Oracle Data Integrator for Big Data provides the following benefits to customers:

  • It brings expanded connectivity to various Big Data source such as Apache Kafka or Cassandra
  • It decreases time to value for Big Data projects
  • It provides a future proof Big Data Integration technology investment
  • It streamlines and shortens the Big Data development and implementation process

Currently ODI supports

  • Generation of Pig Latin transformations: users can choose Pig Latin as their transformation language and execution engine for ODI mappings. Apache Pig is a platform for analyzing large data sets in Hadoop and uses the high-level language Pig Latin for expressing data analysis programs.
  • Generation of Spark and Spark Streaming transformations: ODI mappings can also generate PySpark. Apache Spark is a transformation engine for large-scale data processing. It provides fast in-memory processing of large data sets. Custom PySpark code can be added through user-defined functions or the table function component.
  • Orchestration of ODI Jobs using Oozie: users have a choice between using the traditional ODI Agent or Apache Oozie as orchestration engines for jobs such as mappings, packages, scenarios, or procedures. Apache Oozie allows fully native execution on Hadoop infrastructures without installing an ODI agent for orchestration. Users can utilize Oozie tooling to schedule, manage, and monitor ODI jobs. ODI uses Oozie’s native actions to execute Hadoop processes and conditional branching logic

You can use Oracle Data Integrator to design the ‘what’ of an integration flow and assign knowledge modules to define the ‘how’ of the flow in an extensible range of mechanisms. The ‘how’ is whether it is Oracle, Teradata, Hive, Spark, Pig, etc.

Let’s configure Oracle Data Integrator for Cloudera Hadoop. You don’t need to install any components on your Hadoop Cluster. It is enough to have remote connection to manage all jobs on Hadoop.

Continue reading ‘Introduction and configuring Oracle Data Integrator for Big Data (Cloudera Hadoop)’ »

GoldenGate 12.3: Microservice Architecture (MA)

The Microservices Architecture (MA) for Oracle GoldenGate is a new REST API Microservices-based architecture that allows you to install, configure, monitor, and manage Oracle GoldenGate services using a web-based UI.

Really there are two versions of GoldenGate now: classic and microservice. Classic architecture has standard extract, replicat, pump and receiver. It is managed by classic ggsci. Microservice Architecture (MA) has different types of processes and managed using Admin Client or using web UI. See architecture of GoldenGate MA below

imageOracle GoldenGate MA is designed with the industry-standard HTTP communication protocol and JSON data interchange format.

Classic architecture was managed using ggsci console and had weak authentication and authorization tools. Oracle GoldenGate MA has ability to verify identity using basic authentication and using SSL client certificates.

GoldenGate MA processes

Oracle GoldenGate MA  uses different types of processes to perform same tasks as GoldenGate Classic. Let’s talk a little bit about new processes:

Service Manager. Something like (and replacement of) Manager process.  This is watchdog for other processes.

Administration Server. Something like ggsci console. Operates as central control entity. You use it to create and manage other processes. The key feature of Administration Server is REST API which can be accesses from any HTTP or HTTPS client.

You can add, delete or alter GoldenGate processes, edit configuration files, add users and assign roles using Administration Server.

Receiver Server. Something like collector. It can receive trail files from remote server. However it replaces multiple collectors because it is multithreaded. Receiver was designed to be protocol agnostic – so it supports HTTPS, HTTP, UDT (reliable UDP) and classic GoldenGate TCP transports. By default it uses HTTPS protocol.

Distribution Server. Something like pump.  But again this multithreaded process which can handle multiple trail at the same time. So it will replace multiple pumps. And again it supports multiple protocols: WebSockets for HTTPS-based streaming, which relies on SSL security, UDT, SOCKS5, HTTP. It also support Passive mode to initiate connection from remote side.

Performance Metrics Server. This is process which collects and saves information from other processes (extracts, replicats, etc). All GoldenGate processes push information to Performance Metrics Server. Now this is the only processes which writes data to GoldenGate datastore (Berkley DB). You can use Performance Metrics Server to query various metrics, view logs, process statuses, monitor system utilization, etc.

Admin Client. It is a command line utility (similar to ggsci) used to create, configure and manage processes. Admin Client uses REST API to accomplish its tasks.


Continue reading ‘GoldenGate 12.3: Microservice Architecture (MA)’ »

GoldenGate 12.3: announcement, new features and installation

Oracle has release new version of GoldenGate 12.3 in 18 August. This is very long awaited version – it postponed 2 or 3 times because of some very important new features. See some useful links for GoldenGate 12.3:

Continue reading ‘GoldenGate 12.3: announcement, new features and installation’ »

GoldenGate Studio: Quick Start

imageIntroduction

You work with GoldenGate then you know that there was tool GoldenGate Director for configuration, deployment, monitoring and management. Really these responsibilities are distributed among development and support. So there should be two tools: for development and for support. Oracle is following this ideology and has created two tools:

 

  • GoldenGate Monitor (or GG plugin for OEM if you are OEM fan) — for management and monitoring
  • GoldenGate Studio — for design and deployment

What do we need from development tools? I have the following list in my head:

  • WYSIWYG/drag’n’drop interface
  • easy moving configuration through the path Dev/QA/Prod
  • versioning
  • collective working

GoldenGate has these features. It was created to be great scalable development tool.

Continue reading ‘GoldenGate Studio: Quick Start’ »

Oracle BigData Lite VM introduction

imageVirtual Machine Oracle BigData Lite is vm created by Oracle for development and demo purposes of BigData (Cloudera) and Oracle technologies working together.

You can download BigData Lite VM from Oracle site. This vm includes following software (for vm version 4.4):

Continue reading ‘Oracle BigData Lite VM introduction’ »

Configuring GoldenGate to replicate data from MSSQL Standard Edition (CDC, Change Data Capture)

Introduction

imageOne of GoldenGate 12.2 new features is replication support for MSSQL Standard Edition. In the previous GoldenGate 12.1 release, GoldenGate supported capture of DML only from SQL Server Enterprise Edition, due to the method of enabling Supplemental Logging, which is done via a feature only available to Enterprise Editions of SQL Server. Fortunately, beginning with Oracle GoldenGate 12.2, GoldenGate is able to enable Supplemental Logging for Standard Edition instances via a different mechanism. That mechanism to enable Supplemental Logging is done via certain SQL Server Replication components. These SQL Server Replication components are required to be installed and configured in order to enable Supplemental Logging,

Also it is very interesting that MSSQL doesn’t offer Change Data Capture for MSSQL Standard Edition. So if we have 100 offices with MSSQL and we need to collect only changed data to Data Warehouse then it is required to install MSSQL Enterprise Edition in every office because CDC is only available in MSSQL EE.

Continue reading ‘Configuring GoldenGate to replicate data from MSSQL Standard Edition (CDC, Change Data Capture)’ »

GoldenGate Monitor 12.2: how to change password for internal communications

When install GoldenGate Monitor we create special credentials used by GoldenGate Agent and GoldenGate Monitor to secure communications.

image

Also we set this credentials while configuring GoldenGate Agent using pw_agent_util.sh.

But is not well documented how these credentials are used and how change them. So let fill these gap.

Continue reading ‘GoldenGate Monitor 12.2: how to change password for internal communications’ »

Configuring GoldenGate Monitor 12.2

I already wrote about configuring GoldenGate agent for ODI and Enterprise Manager. Now I would like to talk about configuring it for GoldenGate Monitor. GoldenGate is interesting tool for GoldenGate monitoring. It is more powerful and flexible than GoldenGate plugin for Enterprise Manager but it is still standalone tool not integrated with OEM. So it has its own advantages and disadvantages.

From the technical standpoint configuring GoldenGate Agent for Monitor is a little bit different than configuring GoldenGate Agent for OEM and ODI. Also I see some inaccuracies in documentation and step-by-step configuration is not crystal clear. That’s why I writing this blog post — I would like to have short step-by-step instruction how to configure GoldenGate Agent for Monitor.

So let’s look at GoldenGate monitoring framework architecture

image

We can see

  1. GoldenGate Monitor has its own Server with Repository (which is Oracle Database)
  2. Each GoldenGate instance has Agent (Monitor Agent)
  3. Monitor Server and Monitor agents communicates through JMX (which can also be encrypted by SSL)
  4. Agent communicates with Manager to get information about GoldenGate state

Continue reading ‘Configuring GoldenGate Monitor 12.2’ »