io.debezium.connector.postgresql

Class RecordsStreamProducer

java.lang.Object

io.debezium.connector.postgresql.RecordsProducer

io.debezium.connector.postgresql.RecordsStreamProducer

@ThreadSafe
public class RecordsStreamProducer
extends RecordsProducer

A RecordsProducer which creates records from a Postgres streaming replication connection and messages.

Author:

Horia Chiorean (hchiorea@redhat.com)

Field Summary

Fields

Modifier and Type	Field and Description
private static String	CONTEXT_NAME
private ExecutorService	executorService
private ReplicationConnection	replicationConnection
private AtomicReference<ReplicationStream>	replicationStream

Fields inherited from class io.debezium.connector.postgresql.RecordsProducer

logger, sourceInfo, taskContext

Constructor Summary

Constructors

Constructor and Description
RecordsStreamProducer(PostgresTaskContext taskContext, SourceInfo sourceInfo) Creates new producer instance for the given task context

Method Summary

Modifier and Type	Method and Description
private Object[]	columnValues(List<PgProto.DatumMessage> messageList, TableId tableId, boolean refreshSchemaIfChanged)
protected void	commit() Notification that offsets have been committed to Kafka.
protected Object	extractValueFromMessage(PgProto.DatumMessage datumMessage) Converts the Protobuf value for a plugin message to a Java value based on the type of the column from the message.
protected void	generateCreateRecord(TableId tableId, Object[] rowData, Consumer<org.apache.kafka.connect.source.SourceRecord> recordConsumer)
protected void	generateDeleteRecord(TableId tableId, Object[] oldRowData, Consumer<org.apache.kafka.connect.source.SourceRecord> recordConsumer)
protected void	generateUpdateRecord(TableId tableId, Object[] oldRowData, Object[] newRowData, Consumer<org.apache.kafka.connect.source.SourceRecord> recordConsumer)
private void	process(PgProto.RowMessage message, Long lsn, Consumer<org.apache.kafka.connect.source.SourceRecord> consumer)
private boolean	schemaChanged(List<PgProto.DatumMessage> messageList, Table table)
protected void	start(Consumer<org.apache.kafka.connect.source.SourceRecord> recordConsumer) Starts up this producer.
protected void	stop() Requests that this producer be stopped.
private void	streamChanges(Consumer<org.apache.kafka.connect.source.SourceRecord> consumer)
private TableSchema	tableSchemaFor(TableId tableId)

Methods inherited from class io.debezium.connector.postgresql.RecordsProducer

clock, createEnvelope, schema, topicSelector

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

CONTEXT_NAME

private static final String CONTEXT_NAME

See Also:

Constant Field Values

executorService

private final ExecutorService executorService

replicationConnection

private final ReplicationConnection replicationConnection

replicationStream

private final AtomicReference<ReplicationStream> replicationStream

Constructor Detail

RecordsStreamProducer

public RecordsStreamProducer(PostgresTaskContext taskContext,
                             SourceInfo sourceInfo)

Creates new producer instance for the given task context

Parameters:

taskContext - a PostgresTaskContext, never null

sourceInfo - a SourceInfo instance to track stored offsets

Method Detail

start

protected void start(Consumer<org.apache.kafka.connect.source.SourceRecord> recordConsumer)

Description copied from class: RecordsProducer

Starts up this producer. This is normally done by a PostgresConnectorTask instance. Subclasses should start enqueuing records via a separate thread at the end of this method.

Specified by:

start in class RecordsProducer

Parameters:

recordConsumer - a consumer of SourceRecord instances, may not be null

streamChanges

private void streamChanges(Consumer<org.apache.kafka.connect.source.SourceRecord> consumer)

commit

protected void commit()

Description copied from class: RecordsProducer

Notification that offsets have been committed to Kafka.

Specified by:

commit in class RecordsProducer

stop

protected void stop()

Description copied from class: RecordsProducer

Requests that this producer be stopped. This is normally a request coming from a PostgresConnectorTask instance

Specified by:

stop in class RecordsProducer

process

private void process(PgProto.RowMessage message,
                     Long lsn,
                     Consumer<org.apache.kafka.connect.source.SourceRecord> consumer)
              throws SQLException

Throws:

SQLException

generateCreateRecord

protected void generateCreateRecord(TableId tableId,
                                    Object[] rowData,
                                    Consumer<org.apache.kafka.connect.source.SourceRecord> recordConsumer)

generateUpdateRecord

protected void generateUpdateRecord(TableId tableId,
                                    Object[] oldRowData,
                                    Object[] newRowData,
                                    Consumer<org.apache.kafka.connect.source.SourceRecord> recordConsumer)

generateDeleteRecord

protected void generateDeleteRecord(TableId tableId,
                                    Object[] oldRowData,
                                    Consumer<org.apache.kafka.connect.source.SourceRecord> recordConsumer)

columnValues

private Object[] columnValues(List<PgProto.DatumMessage> messageList,
                              TableId tableId,
                              boolean refreshSchemaIfChanged)
                       throws SQLException

Throws:

SQLException

schemaChanged

private boolean schemaChanged(List<PgProto.DatumMessage> messageList,
                              Table table)

tableSchemaFor

private TableSchema tableSchemaFor(TableId tableId)
                            throws SQLException

Throws:

SQLException

extractValueFromMessage

protected Object extractValueFromMessage(PgProto.DatumMessage datumMessage)

Converts the Protobuf value for a plugin message to a Java value based on the type of the column from the message. This value will be converted later on if necessary by the PostgresValueConverter.converter(Column, Field) instance to match whatever the Connect schema type expects. Note that the logic here is tightly coupled (i.e. dependent) on the Postgres plugin logic which writes the actual Protobuf messages.

Parameters:

datumMessage - a PgProto.DatumMessage instance; never null

Returns:

the value; may be null