Stream data from ServiceNow instances

Datastream supports replicating change events from a ServiceNow instance. ServiceNow is a cloud-based platform for enterprise operations, and you can replicate data from it just as you would from other sources that Datastream supports.

This page contains information about:

  • The key terms that you need to understand when replicating from ServiceNow.
  • The behavior of how Datastream handles data that it pulls from a ServiceNow instance.
  • The versions of ServiceNow that Datastream supports.
  • Known limitations for using ServiceNow as a source.

Key terms

ServiceNow operates using tables, records, and fields:

  • A ServiceNow instance is a dedicated, isolated ServiceNow cloud environment, for example, company.service-now.com.
  • A table is a primary data structure in ServiceNow, equivalent to a database table.
  • A record is an individual entry in a table, equivalent to a row in a database.
  • A field is an individual data element within a record, equivalent to a database column.
  • The connector uses the Table API as the primary RESTful interface to access data directly from ServiceNow tables.
  • Offset-based pagination is a method of navigating through records in a ServiceNow table by passing an offset number to the Table API. This method is sensitive to record deletions during pagination, which can cause the connector to skip data.
  • Keyset-based pagination is a method where the connector manages pagination by tracking unique record values, such as identifiers or timestamps, and their order. This method is effective even if there are record deletions during pagination.

Behavior

The ServiceNow source support in Datastream relies on the ServiceNow REST Table API. Datastream supports two primary replication methods to ensure data integrity and performance.

Historical backfill

  • When replicating change data from a ServiceNow instance, Datastream performs a full historical synchronization of all existing records in a table.
  • To ensure data consistency and prevent record loss during large extractions, the connector uses keyset-based pagination. The connector manages this process by tracking unique record values and their order, which makes the extraction robust to record deletions and prevents the connector from skipping or duplicating records.
  • Datastream sorts data by sys_updated_on and sys_id record properties to provide a stable order for the initial load. For the sys_journal_field table, which contains immutable records and lacks a sys_updated_on field, the connector sorts by sys_created_on and sys_id properties.

Incremental synchronization

  • When replicating change data from a ServiceNow instance, Datastream captures changes, such as inserts and updates, that occur after the initial backfill.
  • Server-side incremental synchronization: the connector queries the ServiceNow API for records modified since the last successful synchronization using a filter on the sys_updated_on and sys_id properties (or sys_created_on and sys_id for the sys_journal_field table).
  • The synchronization process uses a compound cursor consisting of sys_updated_on (or sys_created_on for the sys_journal_field table) and sys_id markers. The sys_updated_on and sys_created_on markers ensure that only records with a timestamp equal to or later than a specified time are synchronized in the next cycle, while the sys_id marker ensures consistency for records with equal timestamps.

Versions

The ServiceNow connector works with the ServiceNow REST Table API (V2). The functionality that Datastream supports is validated against the Zurich release.

Objects replicated

The ServiceNow connector uses a dynamic discovery mechanism to identify the entire set of tables (objects) within an instance. To ensure that you replicate only high-value business data, the connector applies a multi-layered filtering process to exclude internal architecture, technical logs, and system metadata tables.

Filtering logic

  • Web service access: you can only replicate tables that you explicitly configure with Web Services Access enabled (ws_access=true). If this configuration is missing, the connector can't retrieve record data from the table.
  • Excluded superclasses: the connector excludes tables that inherit from the sys_metadata system configuration table or are part of a rotated table hierarchy to prevent replicating technical metadata or inconsistent snapshots.
  • Internal system prefix: the connector filters out tables starting with specific technical prefixes representing platform plumbing or legacy features.
  • The Golden List (system exceptions): while the connector excludes most sys_ tables, it explicitly allowlists critical business entities, such as sys_user, sys_choice, and sys_journal_field.

System prefixes filtered by default

Category Prefixes Description
Core internals v_, ts_, imp_, ecc_, wf_, sh$, bin_, var_, gs_ Virtual tables, search indexes, import staging, ECC queues, and workflow contexts.
Analytics & AI pa_, ua_, ml_, ais_, promin_, di_ Performance Analytics scores, vendor telemetry, ML vectors, and AI search profiles.
Technical modules sa_, sla_, atf_, cxs_, automation_, ids_ Service Mapping patterns, SLA engine logs, and security audit trails.
Legacy & admin pf_, clone_, upgrade_, rollback_, expert_, dscy_ Instance patching history, upgrade logs, and obsolete legacy modules.

Allowed system tables

  • Users & security: sys_user, sys_user_group, sys_user_grmember, sys_user_role, sys_user_has_role.
  • Core reference: sys_choice (drop-down options), sys_company, sys_domain, sys_location.
  • Audit trails: sys_journal_field (work notes and comments history).

Known limitations

Known limitations for using ServiceNow as a source include:

  • Datastream doesn't support capturing deleted records.
  • Datastream excludes many internal platform tables by default, for example, tables starting with ts_, v_, or imp_.
  • You can only replicate tables that you explicitly configure with Web Services Access enabled (ws_access=true). This is the default for most tables.
  • The connector replicates field values in the way that they appear in the underlying ServiceNow database instance. These values are referred to as actual values. This means that the actual values that Datastream replicates to the destination might differ from the values displayed in the ServiceNow user interface (referred to as display values). For more information, see the ServiceNow knowledge base.

What's next