Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add CLDR JSON data to testdata and remove from provider_cldr #513

Merged
merged 6 commits into from
Mar 2, 2021

Conversation

sffc
Copy link
Member

@sffc sffc commented Feb 27, 2021

I made this change because:

  1. I need to open multiple CLDR JSON files for the numbers data provider, and it was cumbersome to set that up manually
  2. Reading from the CLDR JSON directory structure increases code coverage and eliminates the redundant FromStr code path on the CLDR data providers
  3. Keeps the CLDR JSON test data up to date

I copied the data for all of our test locales. I could have limited it to only the locales we were using in provider_cldr tests, but I did all the locales because it allows the ICU4X JSON test data to be generated without hitting any remote server, which could allow us to add additional end-to-end tests on icu4x-cldr-export.

@sffc sffc requested a review from zbraniecki February 27, 2021 02:45
@sffc sffc requested a review from a team as a code owner February 27, 2021 02:45
@codecov-io
Copy link

codecov-io commented Feb 27, 2021

Codecov Report

Merging #513 (49f24ab) into master (3be597c) will increase coverage by 0.78%.
The diff coverage is 60.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #513      +/-   ##
==========================================
+ Coverage   73.35%   74.13%   +0.78%     
==========================================
  Files         117      118       +1     
  Lines        6501     6500       -1     
==========================================
+ Hits         4769     4819      +50     
+ Misses       1732     1681      -51     
Impacted Files Coverage Δ
components/provider_cldr/src/lib.rs 100.00% <ø> (ø)
...omponents/provider_fs/src/bin/icu4x-cldr-export.rs 0.86% <0.00%> (ø)
components/provider_fs/src/export/aliasing.rs 0.00% <ø> (ø)
...onents/provider_cldr/src/download/cldr_allinone.rs 40.00% <33.33%> (+12.72%) ⬆️
components/provider_cldr/src/cldr_paths.rs 45.94% <52.17%> (+10.23%) ⬆️
components/provider_cldr/src/transform/dates.rs 73.51% <100.00%> (+4.48%) ⬆️
...nents/provider_cldr/src/transform/likelysubtags.rs 84.61% <100.00%> (+8.46%) ⬆️
components/provider_cldr/src/transform/plurals.rs 71.02% <100.00%> (+14.39%) ⬆️
components/locid/src/subtags/language.rs 89.65% <0.00%> (ø)
utils/litemap/src/lib.rs 100.00% <0.00%> (ø)
... and 4 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3be597c...49f24ab. Read the comment docs.

@coveralls
Copy link

coveralls commented Feb 27, 2021

Pull Request Test Coverage Report for Build 8bd36afc24f768f5de1c6a406218a54c72da7f21-PR-513

  • 31 of 109 (28.44%) changed or added relevant lines in 10 files are covered.
  • 5 unchanged lines in 4 files lost coverage.
  • Overall coverage increased (+0.2%) to 72.926%

Changes Missing Coverage Covered Lines Changed/Added Lines %
components/provider_fs/src/bin/icu4x-cldr-export.rs 0 1 0.0%
resources/testdata/src/metadata.rs 0 1 0.0%
components/provider_cldr/src/download/cldr_allinone.rs 2 6 33.33%
components/provider_cldr/src/cldr_paths.rs 12 23 52.17%
resources/testdata/src/bin/icu4x-gen-testdata.rs 0 61 0.0%
Files with Coverage Reduction New Missed Lines %
resources/testdata/src/bin/icu4x-gen-testdata.rs 1 0.81%
resources/testdata/src/metadata.rs 1 45.45%
resources/testdata/src/test_data_provider.rs 1 60.0%
components/provider_cldr/src/download/cldr_allinone.rs 2 40.0%
Totals Coverage Status
Change from base Build d1e746357f79f9fcf2a63e50db15f365ab9b063b: 0.2%
Covered Lines: 4940
Relevant Lines: 6774

💛 - Coveralls

Copy link
Member

@zbraniecki zbraniecki left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm!

let json_str = std::fs::read_to_string("tests/testdata/likelySubtags.json").unwrap();
let provider = LikelySubtagsProvider::try_from(json_str.as_str()).unwrap();
let cldr_paths = crate::cldr_paths::for_test();
let provider = LikelySubtagsProvider::try_from(&cldr_paths as &dyn CldrPaths).unwrap();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you need as here? I think it should auto-coerce, no?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems Rust doesn't auto-convert concrete references like &CldrPathsAllInOne to dynamic references like &dyn CldrPaths. I want to change the function to take &impl (CldrPaths + ?Sized) instead of &dyn CldrPaths, but I can't do that in TryFrom; I tried removing TryFrom but that broke a bunch of stuff. I intend to follow up in the next PR, in the context of #442, to fix this.

@@ -113,6 +126,10 @@ clap = { version = "2.33", optional = true }
icu_provider_cldr = { version = "0.1", path = "../../components/provider_cldr", optional = true, features = ["download"] }
log = { version = "0.4", optional = true }
simple_logger = { version = "1.11", optional = true }
globwalk = { version = "0.8", optional = true }
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know it's testdata, but glob crate is maintained by the Rust core team and has no dependencies. Could you use it?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The README for globwalk lists 4 reasons why it exists, and 3 of the 4 apply to me:

  1. I need {} syntax for locales
  2. I need to give a list of multiple selectors, gitignore-style
  3. Support starting from a specified directory, not just current working directory rust-lang/glob#54, a fundamentally important feature request that has been open for almost 5 years
  4. Also, I like the API of globwalk better

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, not only is this testdata, but this is an optional dependency for testdata that is downloaded only when gen-testdata is run. So the impact of this dependency is very minimal, to the extent that I didn't consider it when deciding to add it.

@sffc sffc merged commit c308b40 into unicode-org:master Mar 2, 2021
@sffc sffc deleted the cldrjson branch March 2, 2021 03:38
nordzilla pushed a commit to nordzilla/icu4x that referenced this pull request Mar 4, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants