Data is imported from SITS:Vision to Data Engine to provide a single source of information for reporting and analytics. Data Engine imports data from the SITS:Vision database via the SITS agent.
The SITS agent is a .NET application that runs on a server within the same network as the SITS:Vision database for both on-premises and cloud-hosted environments. The SITS agent supports multiple instances on the same server. Therefore, a SITS agent can connect to different SITS:Vision databases, such as the SITS:Vision databases for live and test environments. The SITS agent is compatible with all supported versions of SITS:Vision. The SITS agent automatically detects the version of SITS:Vision and the associated Tribal data store scripts to ensure correct operation with Data Engine.
The SITS agent creates a secure connection to the Data Engine using a client secret that is generated by Tribal. The client secret is entered manually when setting up the SITS agent for the first time. Once the connection to Data Engine is established, the client secret key is rotated automatically. For security reasons, the client secret is then rotated automatically every seven days. The SITS agent creates a secure connection between the SITS:Vision database and the Data Engine database using an encrypted connection string. For more information about client secrets and connection strings, go to Azure Key Vault secrets 
.
Then, the SITS agent imports data from the SITS:VIsion database. The first import of data from SITS:Vision contains all data from the defined SITS:Vision database entities.
After the first import of data, the SITS agent monitors the SITS:Vision database entities for changes and then creates batches of data to load as follows:
- Change tracking
-
The SITS agent tracks changes in the SITS:Vision data based on an offset from the current time. This ensures that the SITS agent does not track changes for any transactions that are in progress. For example, if the change tracking interval is 10 seconds, the SITS agent does not track changes in the SITS data that are within 10 seconds of the current time. The default change tracking interval is 10 seconds.
Note that the change tracking interval should not be changed without consultation with professional services, as transactions taking longer than 10 seconds to complete may indicate issues with the SITS:Vision database.
-
Polling
-
The SITS agent creates batches of changed data at a frequency defined by the polling interval. The default polling interval is 15 seconds.
Data from the SITS:Vision database is transformed to JSON objects by the SITS agent. The JSON objects hold the batches of data that are defined by logical groupings of data, message types, such as Applicant or Course. The JSON objects are then sent, by the SITS agent, to Azure Event Hub via Advanced Message Queueing Protocol (AMQP). For more information on the Advanced Message Queueing Protocol, go to Azure AMQP .
The Azure Event Hub is a data streaming platform and event ingestion service. For more information on Azure Event Hub, go to Azure Event Hub . After the JSON objects are loaded by the Azure Event Hub, an Azure Function, the mediator, transforms and loads the JSON objects into the Data Engine operational data store. For more information on Azure Functions, go to Azure Functions.
Errors with data are logged on messages as follows:
- SITS agent
-
The SITS agent checks and logs errors for issues such as duplicate records, child records without parent records.
-
Mediator
-
The mediator checks and logs errors with the data of records such as invalid dates, missing required data, and so on.
The diagram Importing data from SITS:Vision shows how data is extracted from the SITS:Vision database by the SITS agent. The extracted data is sent to the Azure Event Hub by the SITS agent. Then loaded into the Data Engine operational data store by an Azure Function.
