MongoDB: Various fixes and improvements #216
Conversation
amotl
force-pushed
the
mongodb-better-1
branch
3 times, most recently
from
fcb2ab2
to
d224db7
Compare
| def date_converter(value): | ||
| if isinstance(value, int): | ||
| return value | ||
| dt = datetime.strptime(value[:-5], "%Y-%m-%dT%H:%M:%S.%f") | ||
| iso_match = _TZINFO_RE.match(value[-5:]) | ||
| if iso_match: | ||
| sign, hours, minutes = iso_match.groups() | ||
| tzoffset = int(hours) * 3600 + int(minutes) * 60 | ||
| if sign == "-": | ||
| dt = dt + timedelta(seconds=tzoffset) | ||
| else: | ||
| dt = dt - timedelta(seconds=tzoffset) | ||
| else: | ||
| raise Exception("Can't parse datetime string {0}".format(value)) | ||
| dt = dateparser.parse(value) | ||
| return calendar.timegm(dt.utctimetuple()) * 1000 |
There was a problem hiding this comment.
This has been trimmed to just rely on dateutil.parser completely. Do you think it will be okay, because that code here just tried to achieve similar things manually before?
There was a problem hiding this comment.
Just a small thing that it may be worth mentioning in the docs, the schema translation part takes the name of the MongoDB collection for the CREATE TABLE statement but the data load part takes the table name from the CRATEDB_SQLALCHEMY_URL so these 2 have to match for ctk load table to work.
amotl
force-pushed
the
mongodb-better-1
branch
2 times, most recently
from
4814bf1
to
d055214
Compare
|
Hi Hernan. Thanks for your suggestion. I was sure there are anomalies, but I can't spot them, probably because of operational blindness. Can I humbly ask you to submit a corresponding suggestion how and where to improve the documentation, either by commenting on this patch, or by submitting a separate one? Thanks a stack! |
amotl
force-pushed
the
mongodb-better-1
branch
from
d055214
to
85afd95
Compare
|
The MongoDB Table Loader will now also accept Zyp transformations for re-shaping data. |
amotl
force-pushed
the
mongodb-better-1
branch
from
85afd95
to
4f4a5e4
Compare
About
The venerable migr8 program did not receive much love to get the details right. This patch intends to catch up on this, in order to make
ctk load table mongodb://...hold up to its promise of better DWIM.Preview
pip install --upgrade 'cratedb-toolkit[mongodb] @ git+https://github.com/crate/cratedb-toolkit.git@mongodb-better-1'What's Inside
Bugfixes
migr8 export.Exception: Can't parse datetime string 2022-06-11T04:37:53.200Z_idinput field toidcolumn instead of dropping it.[MongoDB I/O] Propagation of
_idfield is lost #215Details
migr8 exportpython-dateutil_idinput field toidcolumn instead of dropping itmigr8 extractwrite to stdout by defaultmigr8 translateread from stdin by defaultmigr8 extractandmigr8 exportaccept the--limitoptionmigr8 translate--transformation, see Zyp Transformations.