Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add override for RAW table date column in delta loads #2052

Merged
merged 9 commits into from
Nov 12, 2024
Merged

Add override for RAW table date column in delta loads #2052

merged 9 commits into from
Nov 12, 2024

Conversation

JFriel
Copy link
Collaborator

@JFriel JFriel commented Oct 31, 2024

Proposed Change

DLS ran into issue where Remote Table attacher using a delta load with custom SQL would have adifferent date column in the select vs what was in the RAW due to joins i.e. d.date -> date

Added an optional override to specify the column name in raw alongside the existing column name in the select from remote

Type of change

What types of changes does your code introduce? Tick all that apply.

  • Bugfix (non-breaking change which fixes an issue)
  • New Feature (non-breaking change which adds functionality)
  • Breaking Change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation-Only Update
  • Other (if none of the other choices apply)

Checklist

By opening this PR, I confirm that I have:

  • Ensured that the PR branch is in sync with the target branch (i.e. it is automatically merge-able)
  • Created or updated any tests if relevant
  • Have validated this change against the Test Plan
  • Requested a review by one of the repository maintainers
  • Have written new documentation or updated existing documentation to detail any new or updated functionality and how to use it
  • Have added an entry into the changelog

@JFriel JFriel requested a review from a team October 31, 2024 09:41
@JFriel JFriel merged commit 0779ce3 into release/8.4.0 Nov 12, 2024
5 checks passed
@JFriel JFriel deleted the task/load_in_questionable_tables branch November 12, 2024 15:43
JFriel added a commit that referenced this pull request Nov 29, 2024
@JFriel @rdteviotdale
@jas88 @bpeacock001 @JBaird00183 @dependabot
Release: 8.4.0 (#2006)
b1457d0 
* add changelog

* Bugfix/rdmp 253 filter ordering (#2007)

* add start

* partial

* tidy up code

* filter updates

* update db migrations

* update correct db

* update changelog

* add base creation sql

* fix create

* tidy up

* update insert

* tidy up

* interim

* style updates

* add some info

* start of where clause

* improved

* improved dates

* improve bookends

* add changelog

* update changelog

* improve ui with frequency

* tidy up for tests

* add summary

* rename chart

* interim people

* add people label

* better where

* move items around

* fix up tests

* tidy up code

* tidy up

* made db independant

* add changelog

* add missing file

* Bugfix/rdmp 259 delta load off by one issue (#2024)

* fix off by one issue

* update tests

* update changelog

* check for multi-server query

* update changelog

* update changelog

* Add override for RAW table date column in delta loads (#2052)

* fix table attacher special column issue

* add override

* add extra override

* fix tests

* tidy up

* add docs

* update deps

* add safe ref

* interim

* working check

* tidy up

* add changelog

* fix typos

* update typo

* fix typos

* fix typos

* Changes made to popup buttons (#2076)

* Changes made to popup buttons

* Bump the aws-sdk group with 4 updates (#2074)

Bumps the aws-sdk group with 4 updates: [AWSSDK.S3](https://github.com/aws/aws-sdk-net), [AWSSDK.SecurityToken](https://github.com/aws/aws-sdk-net), [AWSSDK.SSO](https://github.com/aws/aws-sdk-net) and [AWSSDK.SSOOIDC](https://github.com/aws/aws-sdk-net).


Updates `AWSSDK.S3` from 3.7.407 to 3.7.407.1
- [Release notes](https://github.com/aws/aws-sdk-net/releases)
- [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD)
- [Commits](https://github.com/aws/aws-sdk-net/commits)

Updates `AWSSDK.SecurityToken` from 3.7.401.5 to 3.7.401.6
- [Release notes](https://github.com/aws/aws-sdk-net/releases)
- [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD)
- [Commits](https://github.com/aws/aws-sdk-net/commits)

Updates `AWSSDK.SSO` from 3.7.400.56 to 3.7.400.57
- [Release notes](https://github.com/aws/aws-sdk-net/releases)
- [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD)
- [Commits](https://github.com/aws/aws-sdk-net/commits)

Updates `AWSSDK.SSOOIDC` from 3.7.400.56 to 3.7.400.57
- [Release notes](https://github.com/aws/aws-sdk-net/releases)
- [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD)
- [Commits](https://github.com/aws/aws-sdk-net/commits)

---
updated-dependencies:
- dependency-name: AWSSDK.S3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: aws-sdk
- dependency-name: AWSSDK.SecurityToken
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: aws-sdk
- dependency-name: AWSSDK.SSO
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: aws-sdk
- dependency-name: AWSSDK.SSOOIDC
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: aws-sdk
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Update CHANGELOG.md

* Tidied up code

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Task/rdmp 32 regex redaction (#2009)

* interim

* interim

* redact and add some tests

* add missing interface

* add regex redaction keys

* add helper

* add concept

* update interfaces

* add icons and fix restore

* add missing files

* add docs

* add regions

* add ignore

* add speed test

* improve speed

* improve speed

* update padding

* stress test

* add test case

* add fk names

* faster data load

* faster update

* more timers

* super fast redaction

* tidy up code

* woking keys

* improved

* add limiting

* shared helper

* shared

* interim

* working

* add summary

* restore tests working

* add tests

* start og ui update

* start of ui

* basic configuraion

* working add

* user interface

* update for tests

* update test

* tests

* tidy up

* updates

* fix merge

* ad dicons

* add missing icons

* tidyup

* fix tests

* add missing file

* update migration number

* add changelog

* add timeout

* fix build

* interim

* tidy up

* tidy up code

* add publish

* add todo

* add redact all

* add column

* update table

* add docs

* update markdown

* fix typo

* fix typos

* tidy up tests

* tidy up tests

* Include regex redaction doc in VS Solution file

* Update ExecuteCommandPerformRegexRedactionOnCatalogueTests.cs

Syntax cleanup

* Update ThrowImmediatelyActivator.cs

Remove obsolete comment

* Update ExecuteCommandPerformRegexRedactionOnCatalogueTests.cs

Start being green and recycling

* Update ExecuteCommandPerformRegexRedactionOnCatalogueTests.cs

Finish recycling test objects

---------

Co-authored-by: James A Sutherland <[email protected]>

* codeql updates

* update client xml

* code ql updates

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: rdteviotdale <[email protected]>
Co-authored-by: James A Sutherland <[email protected]>
Co-authored-by: Brian <[email protected]>
Co-authored-by: JBaird00183 <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@rdteviotdale rdteviotdale rdteviotdale approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@JFriel @rdteviotdale