I’ve done quite some real time logical replication projects in the past, either using Oracle Golden Gate or EDB replication server. Build in logical replication in PostgreSQL (which is available since PostgreSQL 10) can be used as well when both, the source and the target are PostgreSQL instances. While being at the DOAG conference and exhibition 2019 I got in contact with people from DBPLUS and they provide a product which is called “Data Replicator”. The interesting use case for me is the real time replication from Oracle to PostgreSQL as the next project for such a setup is already in the pipe so I thought I’ll give it try.
The “Data Replicator” software needs to be installed on a Windows machine and all traffic will go through that machine. The following picture is stolen from the official “Data Replicator” documentation and it pretty well describes the architecture when the source system is Oracle:
As “Data Replicator” will use Oracle LogMiner, no triggers need to be installed on the source system. Installing something on a validated system might become tricky so this already is a huge benefit compared to some other solutions, e.g. SymmetricDS. When you know GoldenGate the overall architecture is not so much different: What GoldeGate calls the extract is the “Reader” in Data Replicator and the replicat becomes the “Applier”.
The installation on the Windows machine is so simple, that I’ll just be providing the screenshots without any further comments:
In the background three new services have been created and started by the installation program:
There is the replication manager which is responsible for creating replication processes. And then there are two more services for reading from source and writing data to the target. In addition the graphical user interface was installed (which could also be running on another windows machine) which looks like this once you start it up:
Before connecting with the GUI you should do the basic configuration by using the “DBPLUS Replication Manager Configuration” utility:
Once that is done you can go back to the client and connect:
The initial screen has not much content, except for the possibility to create a new replication and I really like that: No overloaded, very hard to initially understand interface but easy and tidy. With only one choice it is easy to go forward so lets create a new replication:
Some concept here: Very clean interface, only 5 steps to follow. My source system is Oracle 19.3 EE and all I have to do is to provide the connection parameters, admin user and a new user/password combination I want to us for the logical replication:
Asking “Data Replicator” to create the replication user, and all is fine:
SQL> r 1* select username,profile from dba_users where username = 'REPLUSR' USERNAME PROFILE ------------------------------ ------------------------------ REPLUSR DEFAULT
Of course some system privileges have been granted to the user that got created:
SQL> select privilege from dba_sys_privs where grantee = 'REPLUSR'; PRIVILEGE ---------------------------------------- SELECT ANY TRANSACTION LOGMINING SELECT ANY DICTIONARY SELECT ANY TABLE
Proceeding with the target database, which is PostgreSQL 12.1 in my case:
As you can see there is no option to create a user on the target. What I did is this:
postgres=# create user replusr with login password 'xxxxxxx'; CREATE ROLE postgres=# create database offloadoracle with owner = 'replusr'; CREATE DATABASE postgres=#
Once done, the connection succeeds and can be saved:
That’s all for the first step and we can proceed to step two:
I have installed the Oracle sample schemas for this little demo and as I only want to replicate these I’ve changed the selection to “REPLICATE ONLY SELECTED SCHEMAS AND TABLES”.
Once more this is all that needs to be done and the next step would be to generate the report for getting an idea of possible issues:
The reported issues totally make sense and you even get the commands to fix it, except for the complaints about the unique keys, of course (If you go for logical replication you should anyway make sure that each table either contains a primary key or at last a unique key). Once the Oracle database is in archive mode and supplemental log data was added the screen will look fine (I will ignore the two warnings as they are not important for this demo):
The next step is to define the “Start Options” and when you select “automatic” you’ll have to specify the options for the transfer server:
There is a small configuration utility for that as well:
When you are happy with it, provide the details in the previous screen and complete the replication setup by providing a name in the last step:
That’s all you need to do and the replication is ready to be started:
… and then it immediately fails because we do not have a valid license. For getting a trial license you need to provide the computer ID which can be found in the information section:
Provide that to DBPLUS and request a trial license. Usually they are responding very fast:
Starting the replication once more:
You’ll see new processes on the PostgreSQL side:
postgres@centos8pg:/home/postgres/ [121] ps -ef | grep postgres root 1248 769 0 12:58 ? 00:00:00 sshd: postgres [priv] postgres 1252 1 0 12:58 ? 00:00:00 /usr/lib/systemd/systemd --user postgres 1256 1252 0 12:58 ? 00:00:00 (sd-pam) postgres 1262 1248 0 12:58 ? 00:00:00 sshd: postgres@pts/0 postgres 1263 1262 0 12:58 pts/0 00:00:00 -bash postgres 1667 1 0 12:58 ? 00:00:00 /u01/app/postgres/product/12/db_0/bin/postgres -D /u02/pgdata/12 postgres 1669 1667 0 12:58 ? 00:00:00 postgres: checkpointer postgres 1670 1667 0 12:58 ? 00:00:00 postgres: background writer postgres 1671 1667 0 12:58 ? 00:00:00 postgres: walwriter postgres 1672 1667 0 12:58 ? 00:00:00 postgres: autovacuum launcher postgres 1673 1667 0 12:58 ? 00:00:00 postgres: stats collector postgres 1674 1667 0 12:58 ? 00:00:00 postgres: logical replication launcher postgres 2560 1667 0 14:40 ? 00:00:00 postgres: replusr offloadoracle 192.168.22.1(40790) idle postgres 2562 1667 0 14:40 ? 00:00:00 postgres: replusr offloadoracle 192.168.22.1(40800) idle postgres 2588 1263 0 14:40 pts/0 00:00:00 ps -ef postgres 2589 1263 0 14:40 pts/0 00:00:00 grep --color=auto postgres
… and you’ll see LogMiner proceses on the Oracle side:
LOGMINER: summary for session# = 2147710977 LOGMINER: StartScn: 2261972 (0x00000000002283d4) LOGMINER: EndScn: 18446744073709551615 (0xffffffffffffffff) LOGMINER: HighConsumedScn: 0 LOGMINER: PSR flags: 0x0 LOGMINER: Session Flags: 0x4000441 LOGMINER: Session Flags2: 0x0 LOGMINER: Read buffers: 4 LOGMINER: Region Queue size: 256 LOGMINER: Redo Queue size: 4096 LOGMINER: Memory LWM: limit 10M, LWM 12M, 80% LOGMINER: Memory Release Limit: 0M LOGMINER: Max Decomp Region Memory: 1M LOGMINER: Transaction Queue Size: 1024 2019-11-22T14:05:54.735533+01:00 LOGMINER: Begin mining logfile for session -2147256319 thread 1 sequence 8, /u01/app/oracle/oradata/DB1/onlinelog/o1_mf_2_gxh8fbhr_.log 2019-11-22T14:05:54.759197+01:00 LOGMINER: End mining logfile for session -2147256319 thread 1 sequence 8, /u01/app/oracle/oradata/DB1/onlinelog/o1_mf_2_gxh8fbhr_.log
In the details tab there is more information about what is currently going on:
Although it looked quite good at the beginning there is the first issue:
Oracle data type is unknown: OE.CUST_ADDRESS_TYP Stack trace: System.ArgumentException: Oracle data type is unknown: OE.CUST_ADDRESS_TYP at DbPlus.DataTypes.Oracle.OracleDataTypes.Get(String name) at DbPlus.Replicator.Tracking.TableCopyParameterCreator.c__DisplayClass22_0.g__MapSourceColumnType|1(TableColumn sourceColumn, String targetColumnName) at DbPlus.Replicator.Tracking.TableCopyParameterCreator.c__DisplayClass25_0.g__GetColumnMapping|4(TableColumn sourceColumn) at System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext() at System.Linq.Enumerable.WhereEnumerableIterator`1.MoveNext() at System.Linq.Buffer`1..ctor(IEnumerable`1 source) at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source) at DbPlus.Replicator.Tracking.TableCopyParameterCreator.c__DisplayClass25_0.b__5() at DbPlus.Replicator.Alerts.AsyncTransientErrorHandler.Execute[T](Func`1 operation) at DbPlus.Replicator.Tracking.TableCopyParameterCreator.GetTableCopyParameters(ReplicatedTable sourceTable) at DbPlus.Replicator.Tracking.ReplicatedTablesTracker.b__61_2(ReplicatedTable table) at DbPlus.Replicator.Tracking.ReplicatedTablesTracker.ExecuteOneWithTableLock(Func`1 source, Action`1 operation, Nullable`1 timeout) at DbPlus.Replicator.Tracking.ReplicatedTablesTracker.ExecuteAllWithTableLock(Func`1 source, Action`1 operation, Nullable`1 timeout) at DbPlus.Replicator.Tracking.ReplicatedTablesTracker.b__61_0() at DbPlus.Replicator.Alerts.AsyncTransientErrorHandler.Block(Action action) at DbPlus.Replicator.Tracking.ReplicatedTablesTracker.StartDataTransfer() at DbPlus.Replicator.Tracking.ReplicatedTablesTracker.ProcessRemoteOperations() at DbPlus.Tasks.Patterns.TaskTemplates.c__DisplayClass0_0.<g__Run|0>d.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at DbPlus.Tasks.Patterns.TaskGroup.Run(CancellationToken cancellationToken) at DbPlus.Replicator.Tracking.ReplicatedTablesTracker.Run() at DbPlus.Replicator.ComponentModel.Component.RunInternal()
As with all logical replication solutions custom types are tricky and usually not supported. What I will be doing now is to replicate the “HR” and “SH” schemas only, which do not contain any custom type:
Once again, starting the replication, next issue:
Oracle data type is unknown: ROWID Stack trace: System.ArgumentException: Oracle data type is unknown: ROWID at DbPlus.DataTypes.Oracle.OracleDataTypes.Get(String name) at DbPlus.Replicator.Tracking.TableCopyParameterCreator.c__DisplayClass22_0.g__MapSourceColumnType|1(TableColumn sourceColumn, String targetColumnName) at DbPlus.Replicator.Tracking.TableCopyParameterCreator.c__DisplayClass25_0.g__GetColumnMapping|4(TableColumn sourceColumn) at System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext() at System.Linq.Enumerable.WhereEnumerableIterator`1.MoveNext() at System.Linq.Buffer`1..ctor(IEnumerable`1 source) at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source) at DbPlus.Replicator.Tracking.TableCopyParameterCreator.c__DisplayClass25_0.b__5() at DbPlus.Replicator.Alerts.AsyncTransientErrorHandler.Execute[T](Func`1 operation) at DbPlus.Replicator.Tracking.TableCopyParameterCreator.GetTableCopyParameters(ReplicatedTable sourceTable) at DbPlus.Replicator.Tracking.ReplicatedTablesTracker.b__61_2(ReplicatedTable table) at DbPlus.Replicator.Tracking.ReplicatedTablesTracker.ExecuteOneWithTableLock(Func`1 source, Action`1 operation, Nullable`1 timeout) at DbPlus.Replicator.Tracking.ReplicatedTablesTracker.ExecuteAllWithTableLock(Func`1 source, Action`1 operation, Nullable`1 timeout) at DbPlus.Replicator.Tracking.ReplicatedTablesTracker.b__61_0() at DbPlus.Replicator.Alerts.AsyncTransientErrorHandler.Block(Action action) at DbPlus.Replicator.Tracking.ReplicatedTablesTracker.StartDataTransfer() at DbPlus.Replicator.Tracking.ReplicatedTablesTracker.ProcessRemoteOperations() at DbPlus.Tasks.Patterns.TaskTemplates.c__DisplayClass0_0.<g__Run|0>d.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at DbPlus.Tasks.Patterns.TaskGroup.Run(CancellationToken cancellationToken) at DbPlus.Replicator.Tracking.ReplicatedTablesTracker.Run() at DbPlus.Replicator.ComponentModel.Component.RunInternal()
Lets check which column(s) and table(s) that is/are:
SQL> SELECT owner, table_name, column_name from dba_tab_columns where data_type = 'ROWID' and owner in ('HR','SH');
OWNER TABLE_NAME COLUMN_NAME
-------------------- ------------------------------ ------------------------------
SH DR$SUP_TEXT_IDX$U RID
SH DR$SUP_TEXT_IDX$K TEXTKEY
Such columns can be easily excluded:
Starting over again, next issue:
At least the schemas need to exist on the target, so:
postgres=# \c offloadoracle postgres You are now connected to database "offloadoracle" as user "postgres". offloadoracle=# create schema sh; CREATE SCHEMA offloadoracle=# create schema hr; CREATE SCHEMA offloadoracle=#
Next try:
On the source side:
SQL> grant flashback any table to REPLUSR; Grant succeeded. SQL>
On the target side:
offloadoracle=# grant all on schema hr to replusr; GRANT offloadoracle=# grant all on schema sh to replusr; GRANT
Finally most of the tables are replicating fine now:
There are a few warnings about missing unique keys and some tables can not be replicated at all:
For now I am just going to exclude the failed tables as this is fine for the scope of this post:
… an my replication is fine. A quick check on the target:
offloadoracle=# select * from sh.products limit 3;
prod_id | prod_name | prod_desc | prod_subcategory | prod_subcategory_id | prod_subcategory_desc | prod_category | prod_category_id | prod_category_desc | prod_weight_class | prod_unit_of_measure | prod_pac>
---------+-------------------------------+-------------------------------+------------------+---------------------+-----------------------+-----------------------------+------------------+-----------------------------+-------------------+----------------------+--------->
13 | 5MP Telephoto Digital Camera | 5MP Telephoto Digital Camera | Cameras | 2044.0000000000 | Cameras | Photo | 204.0000000000 | Photo | 1 | U | P >
14 | 17" LCD w/built-in HDTV Tuner | 17" LCD w/built-in HDTV Tuner | Monitors | 2035.0000000000 | Monitors | Peripherals and Accessories | 203.0000000000 | Peripherals and Accessories | 1 | U | P >
15 | Envoy 256MB - 40GB | Envoy 256MB - 40Gb | Desktop PCs | 2021.0000000000 | Desktop PCs | Hardware | 202.0000000000 | Hardware | 1 | U | P >
(3 rows)
lines 1-7/7 (END)
… confirms the data is there. As this post is already long enough here some final thoughts: The installation of “Data Replicator” is a no-brainer. I really like the simple interface and setting up a replication between Oracle and PostgreSQL is quite easy. Of course you need to know the issues you can run into with logical replication (missing unique or primary keys, not supported data types, …) but this is the same topic for all solutions. What I can say for sure is, that I never was as fast for setting up a demo replication as with “Data Replicator”. More testing to come …
Cet article Real time replication from Oracle to PostgreSQL using Data Replicator from DBPLUS est apparu en premier sur Blog dbi services.