Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Runtime Variables YYYY, MM, and DD value are prefixed with underscore #345

Closed
karhendana opened this issue Jul 27, 2024 · 1 comment
Closed

Comments

@karhendana
Copy link

Issue Description

  • Description of the issue:
    When using runtime variables YYYY, MM, and DD in object configuration, resulting values are prefixed with underscore (_).
    e.g. {YYYY}{MM}{DD} becomes _2024_07_27, instead of 20240727.

  • Sling version (sling --version): 1.2.14

  • Operating System (linux, mac, windows): linux

  • Replication Configuration:

source: MYDB
target: 'local://'

defaults:
  mode: full-refresh
  object: 'file:///tmp/{YYYY}{MM}{DD}/{stream_schema}_{stream_table}/*.parquet'
  target_options:
    format: parquet

streams:
  myschema.mytable:
  • Log Output (please run command with -d):
2024-07-27 14:05:33 INF Sling Replication [1 streams] | MYDB -> local://

2024-07-27 14:05:33 INF [1 / 1] running stream myschema.mytable
2024-07-27 14:05:33 DBG Sling version: 1.2.14 (linux amd64)
2024-07-27 14:05:33 DBG type is db-file
2024-07-27 14:05:33 DBG using source options: {"empty_as_null":false,"null_if":"NULL","datetime_format":"AUTO","max_decimals":-1}
2024-07-27 14:05:33 DBG using target options: {"header":true,"compression":"auto","concurrency":7,"datetime_format":"auto","delimiter":",","file_max_rows":0,"file_max_bytes":0,"format":"parquet","max_decimals":-1,"use_bulk":true,"add_new_columns":true,"column_casing":"source"}
2024-07-27 14:05:33 DBG opened "oracle" connection (conn-oracle-kSu)
2024-07-27 14:05:34 INF connecting to source database (oracle)
2024-07-27 14:05:34 INF reading from source database
2024-07-27 14:05:35 DBG select * from "myschema"."mytable"
2024-07-27 14:05:35 INF writing to target file system (file)
2024-07-27 14:05:35 DBG opened "file" connection (conn-file-FEE)
2024-07-27 14:05:35 DBG writing to file:///tmp/_2024_07_27/myschema_mytable/part.01 [fileRowLimit=100000 fileBytesLimit=50000000 compression=auto concurrency=4 useBufferedStream=false fileFormat=parquet]
9s 75,392 6746 r/s 4.3 MB | 37% MEM | 23% CPU 2024-07-27 14:05:46 DBG wrote 1.5 MB: 92153 rows [7,179 r/s]
10s 84,865 9605 r/s 5.1 MB | 37% MEM | 14% CPU 
2024-07-27 14:05:46 INF wrote 92153 rows [7,178 r/s] to file:///tmp/_2024_07_27/myschema_mytable/*.parquet
2024-07-27 14:05:46 INF execution succeeded

2024-07-27 14:05:47 INF Sling Replication Completed in 13s | MYDB -> local:// | 1 Successes | 0 Failures
@flarco
Copy link
Collaborator

flarco commented Jul 27, 2024

Thanks for reporting. Fixed: cdb67de

@flarco flarco closed this as completed Jul 27, 2024
@flarco flarco mentioned this issue Aug 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants