10 April 2024

How data transparency works

Data analysts connect with Submissions through a dedicated integration endpoint for data transparency. The data transparency endpoint enables data analysts to self-manage their permissions and access Submissions data without having an impact on the performance of the Submissions user interface. The data transparency endpoint returns data from the following:

  • Submissions SMS table

  • Submissions Staging table plus overrides

  • Submissions Government table

  • TCSI

The first GET call, called an unkeyed GET, returns a list of all the smsRef that have changed since a set date and time passed as a parameter to the GET. If the date and time parameter is not set, the parameter defaults to retrieve changes in the last 24 hours. Then, a GET for each returned smsRef, called a GET by smsRef, retrieves details from Submissions and TCSI. All data is returned in JSON format.

A change includes any DELETE of a data group record, including child packets.

Unkeyed GET

The unkeyed GET parameter fromDateTime should be set so that only records that have been created, updated, sent to TCSI, purged, or deleted are returned in the period required. If the parameter is not sent with the request, the endpoint returns records that were created, updated, sent to TCSI, purged or deleted in the last 24 hours. Using the fromDateTime value in the unkeyed GET reduces the chance of duplicated records or records missing a smsRef being returned in the unkeyed GET.

  • This endpoint returns arrays of 1,000 smsRef by default, and the endpoint supports skip and take.

  • No sorting can be applied to this GET call, and the records are sorted by student smsRef.

The unkeyed GET takes the fromDateTime value and compares it with each of the following values for every student parent and child packet in Submissions as detailed in the table Unkeyed GET.

Unkeyed GET

Parameter

Table

Parent/child

CreatedOn

SMS

Staging

Government

Parent, child

Parent, child

Parent, child

UpdatedOn

SMS

Staging

Government

Parent, child

Parent, child

Parent, child

ReceivedFromSMSDate

SMS

Staging

Government

Parent, child

Parent, child

Parent, child

sentToGovDate

SMS

Staging

Government

Parent, child

Parent, child

Parent, child

PurgeRequestedOn

Purge

Parent only

Where the parent or child packet has been created, updated, sent to TCSI, or purged since the fromDateTime value, or where the parent has been deleted since the fromDateTime value, the smsRef of the parent is returned in the results of the GET call.

Error handling

If a call to the data transparency endpoint fails, one of the following errors is returned:

  • If the API name is incorrect, the GET call returns error 400.

  • If the fromDateTime value is invalid, the GET call returns error 400.

  • If the endpoint is unavailable, the GET call returns error 500.

  • If the you don't have an Intelligent license level, the GET call returns error 400.

GET by smsRef

Once the unkeyed GET has returned a list of created, updated, sent to TCSI, purged, or deleted student smsRef, calling the same endpoint with the smsRef specified returns data from Submissions and TCSI related to that student.

The keyed GET is accessed via /api/aucompliance/datatransparency/<data group>/:smsRef and supports the following data groups:

  • Student

  • Unit enrolment

  • Course admission

  • OS-HELP loan

  • SA-HELP loan

Note that the parameter includeTcsiData is true by default. However, if set to false, no TCSI data is returned for each element. If the student hasn't been sent to TCSI or has been deleted from TCSI, all TCSI objects are null.

The data returned includes the following:

Common information

Common information to the data group, such as the smsRef, whether the data matches across all sources, the UID from TCSI (where it exists), any Government errors against the record, whether the record has been deleted from Submissions, the provider type, and metadata such as:

  • When the record was received from the SMS.

  • Who created the record.

  • When the record was created in each of the Submissions tables.

  • Who updated the record.

  • When the record was updated in each of the Submissions tables.

  • When the record was last submitted to TCSI.

  • Details of any purges that have been actioned against the record.

Specific information

Specific information for each element within the data group, such as:

  • Values stored against the element in the Submissions tables for SMS, staging and Government, the value stored in TCSI (where the record exists in TCSI).

  • Whether this element is included in the matching checks. Note that this has been set for each element by Tribal and cannot be changed.

  • Whether the data matches across staging, plus overrides, and TCSI.

  • Whether there are any overrides present.

  • The user or system process that triggered a change, such as audit details.

The endpoint conducts data matching with the elements providing insight into whether the data matches between Submissions and TCSI. At the common level, the checks are as follows:

  • allMatched indicates whether the matchedElements value for each of the data groups, parent or child, is TRUE. This only appears at the parent level.

  • matchedElements at the data group level is an overall indicator for whether within that data group, each element's isMatched value is TRUE. If all isMatched values are TRUE then matchedElements is TRUE. If one or more isMatched values are FALSE, then matchedElements is FALSE.

  • matchedElementCount gives a count of how many elements have isMatched=TRUE.

  • unmatchedElementCount gives a count of how many elements have isMatched=FALSE.

For each element, the matching logic is as follows:

  • isMatched at the data group level is a check to see if the SMS, Staging (including overrides), and Government values match the element value in TCSI. This is to help with the data verification and troubleshooting.

  • If the data has not been sent to TCSI, the isMatched value will be FALSE as the null in TCSI does not match the value in Submissions.

  • Override matching logic looks at Staging, Government, and TCSI data only and ignores the SMS data. If there is an override that matches across Staging, Government, and TCSI, the isMatched indicator for that element is TRUE.

  • CHESSN values do not influence the matchedElements at the common level. This is because the government is phasing out CHESSN, and the matching across systems will be less predictable than other identifiers such as the Student Identification Code or USI.

  • The value ignoreInMatching is always TRUE for E488 CHESSN and always FALSE for all other elements.

Error handling

If a call to the data transparency endpoint fails, one of the following errors is returned:

  • If the API name is incorrect, the GET call returns error 400.

  • If the endpoint is unavailable, the GET call returns error 500.

  • If the customer doesn't have an Intelligent license level, the GET call returns error 400.

  • If TCSI is returning data too slowly (5 seconds or longer), the GET call returns error 500.

  • If the record is not found in Submissions, the GET call returns error 404.