Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: fixed serialization of DataFrame with empty (NaN) values, fixed escaping whitespaces, fixed order of tags #123

Merged
merged 2 commits into from
Jul 13, 2020

Conversation

rolincova
Copy link
Contributor

@rolincova rolincova commented Jul 3, 2020

Closes #118

  • Fixed serialization of DataFrame with empty (NaN) values
  • Fixed serialization of \n, \r and \t
  • Fixed order of tags
  • CHANGELOG.md updated
  • Rebased/mergeable
  • Tests pass
  • Sign CLA (if not already signed)

@rolincova rolincova requested a review from bednar July 3, 2020 06:12
@codecov-commenter
Copy link

codecov-commenter commented Jul 3, 2020

Codecov Report

Merging #123 into master will increase coverage by 0.21%.
The diff coverage is 96.87%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #123      +/-   ##
==========================================
+ Coverage   88.71%   88.93%   +0.21%     
==========================================
  Files          22       23       +1     
  Lines        1826     1852      +26     
==========================================
+ Hits         1620     1647      +27     
+ Misses        206      205       -1     
Impacted Files Coverage Δ
...fluxdb_client/client/write/dataframe_serializer.py 96.66% <96.66%> (ø)
influxdb_client/client/write_api.py 97.53% <100.00%> (+0.91%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f74d183...cd01183. Read the comment docs.

Copy link
Contributor

@bednar bednar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please also fix escaping of white spaces:

    def test_tag_escaping_key_and_value(self):
        from influxdb_client.extras import pd, np

        now = pd.Timestamp('2020-04-05 00:00+00:00')

        data_frame = pd.DataFrame(data=[["new\nline", "carriage\nreturn", "t\tab", np.int64(100.5)], ],
                                  index=[now + timedelta(hours=1), ],
                                  columns=["new\nline", "carriage\rreturn", "t\tab", "l\ne\rv\tel"])

        points = data_frame_to_list_of_points(data_frame=data_frame,
                                              point_settings=PointSettings(),
                                              data_frame_measurement_name='h\n2\ro\t_data',
                                              data_frame_tag_columns={"new\nline", "carriage\rreturn", "t\tab"})

        self.assertEqual(1, len(points))
        self.assertEqual("h\\n2\\ro\\t_data,carriage\\rreturn=carriage\\nreturn,new\\nline=new\\nline,t\\tab=t\\tab l\\ne\\rv\\tel=2i",
                         points[0])

influxdb_client/client/write/dataframe_serializer.py Outdated Show resolved Hide resolved
@7yl4r
Copy link

7yl4r commented Jul 7, 2020

Looks like the 3.7 check might just need to be rerun; it's a 500 error from the influx server.

@bednar bednar changed the title fix: Fixed serialization of DataFrame with empty (NaN) values #118 fix: fixed serialization of DataFrame with empty (NaN) values, fixed escaping whitespaces, fixed order of tags Jul 8, 2020
@bednar
Copy link
Contributor

bednar commented Jul 8, 2020

Hi @7yl4r,

we want to also fix a problem with serialization of whitespaces and order of tags (https://v2.docs.influxdata.com/v2.0/write-data/best-practices/optimize-writes/#sort-tags-by-key)

I hope that we are be able to merge it soon.

Regards

Copy link
Contributor

@bednar bednar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please also fix order of tags:

def test_tags_order(self):
    from influxdb_client.extras import pd, np

    now = pd.Timestamp('2020-04-05 00:00+00:00')

    data_frame = pd.DataFrame(data=[["c", "a", "b", np.int64(2)], ],
                              index=[now + timedelta(hours=1), ],
                              columns=["c", "a", "b", "level"])

    points = data_frame_to_list_of_points(data_frame=data_frame,
                                          point_settings=PointSettings(),
                                          data_frame_measurement_name='h2o',
                                          data_frame_tag_columns={"c", "a", "b"})

    self.assertEqual(1, len(points))
    self.assertEqual("h2o,a=a,b=b,c=c level=2i 1586048400000000000", points[0])

https://v2.docs.influxdata.com/v2.0/write-data/best-practices/optimize-writes/#sort-tags-by-key

@bednar bednar added the bug Something isn't working label Jul 10, 2020
@rolincova rolincova force-pushed the fix/dataframe-ignore-empty-values branch from 2ebf5a3 to a014150 Compare July 13, 2020 08:47
Copy link
Contributor

@bednar bednar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@bednar bednar merged commit d2472f4 into master Jul 13, 2020
@bednar bednar deleted the fix/dataframe-ignore-empty-values branch July 13, 2020 09:06
@bednar bednar added this to the 1.9.0 milestone Jul 13, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

errors saving dataframes since 1.8.0
4 participants