Reading from R-Instat into Buckets: Code

[lilyclements]

Here is a bit of a glimpse in the code to read from R-Instat into the google buckets

So if you firstly run in R-Instat your start of rains, end of rains, seasonal length, crop success probabilities, etc. Then you can reformat these into the format used in python and the app (i.e., same variable names and types).

Here:

• Prepare > Data Reshape > Column Summaries are used to generate the annual and monthly temperatures for min, max, mean. These create columns in ghana_by_station_year (and creates ghana_by_station_year_month_abbr for monthly temperature summaries.
• Prepare > Data Reshape > Column Summaries are used to generate the annual total rainfall. This adds a column in ghana_by_station_year which is used in the annual_summaries
• Go to "Climatic > Prepare > ..." for the start of rains, end of rains, end of season, seasonal length, number of rainy days in a year (Transform), total rainfall in a season summaries (Climatic summary). All bits to populate ghana_by_station_year which is used in the annual_summaries summary.
• Climatic > PICSA > Crops... generates the crop_prop and crop_def data frames

So here's the code if you create these and wish to reformat them. In this code, we are using the names that came out of this using the ghana_2_stations data stored in R-Instat

# reformat them for all four
crop_prop <- data_book$get_data_frame("crop_prop")
crop_prop <- reformat_crop_success(crop_prop, station = "station", "rain_total", "plant_day",
                                   "plant_length", "prop_success")

crop_def <- data_book$get_data_frame("crop_def")
crop_def <- reformat_season_start(crop_def, "station", "year", "plant_day", "plant_day_cond")

annual_temp <- data_book$get_data_frame("ghana_by_station_year")
annual_temp <- reformat_temperature_summaries(annual_temp, "station", "year", NULL,
                                              "mean_min_temperature", "min_min_temperature", "max_min_temperature",
                                              "mean_max_temperature", "min_max_temperature", "max_max_temperature")

monthly_temp <- data_book$get_data_frame("ghana_by_station_year_month_abbr")
monthly_temp <- reformat_temperature_summaries(monthly_temp, "station", "year", "month_abbr",
                                               "mean_min_temperature", "min_min_temperature", "max_min_temperature",
                                               "mean_max_temperature", "min_max_temperature", "max_max_temperature")

annual_rain <- data_book$get_data_frame("ghana_by_station_year")
annual_rain <- reformat_annual_summaries(data = annual_rain,
                                         "station", "year", start_rains_doy_col = "start_rain",
                                         end_rains_doy_col = "end_rains",
                                         annual_rain_col = "sum_rainfall",     
                                         n_rain_col = "sum_Rainday")

Then we can run the function to export our definitions and summaries into R-Instat.
Hopefully the documentation explains these options in details but in brief:

• Our first four arguments are different data frames names in the data book where the definitions are found
• the next five arguments are different variables. This is useful in creating the definitions so that we are checkign for the correct bits
• summaries states which summaries to calculate (not yet ready for extremes)
• station_id and country are always given and for purposes of saving
• anything ending _data is the data itself. In our case, we've reformatted these using our reformat functions above.

export_r_instat_to_bucket(data = "ghana",
                          data_by_year = "ghana_by_station_year",
                          data_by_year_month = "ghana_by_station_year_month_abbr",
                          crop_data_name = "crop_def",
                          rain = "rainfall",
                          tmin = "min_temperature",
                          tmax = "max_temperature",
                          year = "year",
                          month = "month_abbr",
                          summaries = c("annual_rainfall", "annual_temperature", "monthly_temperature", "extremes", "crop_success", "start_season"),
                          station_id = "test_3",
                          country = "zm",
                          include_summary_data = TRUE,
                          annual_rainfall_data = annual_rain,
                          annual_temperature_data = annual_temp,
                          monthly_temperature_data = monthly_temp,
                          crop_success_data = crop_prop,
                          season_start_data = crop_def)

As always, any feedback would be great. If others could test this out too then that would be very helpful.

[lilyclements]

@jkmusyoka @rdstern here is some data to test this out.

So the following is a script to run all of this:

library(epicsadata)
library(epicsawrap)

# Initialising R (e.g Loading R packages)
setwd(dir="C:/Program Files/R-Instat/0.7.16/static/InstatObject/R")

source(file="Rsetup.R")

data_book <- DataBook$new()

# Setting display options (e.g Number of significant digits)
options(digits=4, show.signif.stars=FALSE, dplyr.summarise.inform=FALSE)

# Code generated by the dialog, Import Dataset
new_RDS <- readRDS(file="C:/Users/lclem/OneDrive/Documents/ghana_example_epicsa.RDS")
data_book$import_RDS(data_RDS=new_RDS)

rm(new_RDS)


# Ran in the dialog - reformatting the data that is called so that we have the correct columns and names of columns for the API
crop_prop <- data_book$get_data_frame("crop_prop")
crop_prop <- reformat_crop_success(crop_prop, station = "station", "rain_total", "plant_day",
                                   "plant_length", "prop_success")

crop_def <- data_book$get_data_frame("crop_def")
crop_def <- reformat_season_start(crop_def, "station", "year", "plant_day", "plant_day_cond")

annual_temp <- data_book$get_data_frame("ghana_by_station_year")
annual_temp <- reformat_temperature_summaries(annual_temp, "station", "year", NULL,
                                              "mean_min_temperature", "min_min_temperature", "max_min_temperature",
                                              "mean_max_temperature", "min_max_temperature", "max_max_temperature")

annual_rain <- data_book$get_data_frame("ghana_by_station_year")
annual_rain <- reformat_annual_summaries(data = annual_rain,
                                         "station", "year", start_rains_doy_col = "start_rain",
                                         end_rains_doy_col = "end_rains")

# Main dialog function - exporting to buckets
[ghana_example_epicsa.zip](https://github.com/IDEMSInternational/epicsawrap/files/15148210/ghana_example_epicsa.zip)

export_r_instat_to_bucket(data = "ghana",
                          data_by_year = "ghana_by_station_year",
                          crop_data_name = "crop_def",
                          rain = "rainfall",
                          tmin = "min_temperature",
                          tmax = "max_temperature",
                          year = "year",
                          month = "month_abbr",
                          summaries = c("annual_rainfall", "annual_temperature", "extremes", "crop_success", "start_season"),
                          station_id = "test_3",
                          country = "zm",
                          include_summary_data = TRUE,
                          annual_rainfall_data = annual_rain,
                          annual_temperature_data = annual_temp,
                          crop_success_data = crop_prop,
                          season_start_data = crop_def)

To download the packages:

devtools::install_github("IDEMSInternational/epicsadata")
devtools::install_github("IDEMSInternational/epicsawrap")

[jkmusyoka]

@lilyclements
I ran the code but I got this message at the end which I think implies that I do not have permissions to write into the google bucket?

Can this be resolved just for testing purposes?

[lilyclements]

@jkmusyoka yes, this means you need the authentication key. @Vitalis95 is in the process of setting this up on the dialog, but I'll send you the key via email for now.

Before running the R code, just run:

epicsawrap::gcs_auth_file("path/to/filename.json")

[jkmusyoka]

@lilyclements Thanks. the code worked ok and the export was successful.

Quick question - Is there a way to check what has been exported into the bucket?

[lilyclements]

Yes, does this link here work for you? https://console.cloud.google.com/storage/browser?project=e-picsa

I cannot remember now how it was all set up for me. Perhaps a combination of that and the json file I shared with you securely.

In case that does not work, @ChrisMarsh82 do you have the powers (or know who has the powers) to give James access to the bucket?

[jkmusyoka]

@lilyclements thanks for the link but it doesn't work for me. Perhaps @ChrisMarsh82 has to give me permissions to access the bucket

[ChrisMarsh82]

@jkmusyoka please could you try again. You should have permissions now

[jkmusyoka]

@ChrisMarsh82 thanks, I now have access to the bucket. But I I can't find the exported file(s) in the bucket despite getting a successful export message in R. This is what I see when I open the bucket.

There is no new file since Jan 2024. Am I looking at the right place?

[lilyclements]

@jkmusyoka good question.

1. So you ran country = "zm", which means you need to go under the "zambia_data" file.
2. In there you can find the definition are in the "definitions/" folder, and summaries are in the "summaries/" folder.
3. For summaries, you can see the data for the different summaries:
   

e.g.

here we have "annual_rainfall_summaries_test_3_.rds" for the annual rainfall summaries file.
(test_3 because you ran station_id = "test_3" in the R code. You can change this and it would change the saved file name)

[jkmusyoka]

@lilyclements. I couldn't find a way of including a second definition of "start of rains" for the same station in the summaries file that I uploaded into the bucket. Is this possible in the current set up or could be added if not there? Sometimes this is needed.

[lilyclements]

@jkmusyoka this is really useful to know is a requirement. I spoke with @rdstern about it today, and I will implement it in. I will let you know how I get on! Thank you for this

[jkmusyoka]

@lilyclements. The uploading into the google buckets works fine with the R script. I am now waiting to test the same process with the new R-Instat dialogues.

An observation - I have noticed that the names of the files uploaded into the google bucket have a extension that looks like a combination of date and time added to the station_id i.e. annual_rainfall_summaries_zambia_eastern.20240510181254.rds and similarly for the other related files.

So I went ahead to test the APIs using this data but I got an error 500 which is not very informative. So I am not sure where the problem is but kept wondering whether the names could be an issue. This is because only test_1 works fine with the API. Test_1 seems to be the only file in the zambia summary folder without the funny extension to its name. Your thoughts?

[lilyclements]

@jkmusyoka thanks for sharing. Yes, you are correct we have a date/timestamp added to the names when uploaded. This was requested by @ChrisMarsh82 in #49 if that helps!

I'm not sure why you have received error 500. Would you mind sharing the R code which caused this and I can dig into it a bit?

That's an interesting observation about test_1. Perhaps then is your suggestion that we put the names as "_" before the timestamp not a "."? If so, I definitely agree, so we have
"annual_rainfall_summaries_zambia_eastern_20240510181254.rds" instead of "annual_rainfall_summaries_zambia_eastern.20240510181254.rds"

[jkmusyoka]

@lilyclements and @ChrisMarsh82 I am getting the following server response error 500 with the "Get Annual Summaries" API for station_id: zambia_eastern.

Am not sure what this means and whether this is an error in the R code or with API

By the way, the only station that does not return an error with the API is test_1 which is the only station without timestamp on its respective filenames.

[lilyclements]

Great, can you try manually changing the name to "_ .RDS" and seeing if it still gives an error?

e.g., change the file name in the bucket to:

annual_rainfall_summaries_zambia_eastern_20240510181254.rds

[jkmusyoka]

@lilyclements I still get the same error 500 but with a bit more information. See image

[lilyclements]

Ok, what are you setting as country and station_id when you run it in the API? I think it should be that country = "zm", and station_id = "zambia_eastern"

[jkmusyoka]

See I have the correct setting for country and station_id

[lilyclements]

Thanks. @jkmusyoka I have tried calling your data into R and found the error. It was occurring because I had previously tested with multiple files in the bucket, but here we only had one. I have fixed the bug in PR here.

Can you update your epicsadata package, and then this should work (note, I find sometimes I need to restart R if I've just made the package change. Perhaps give it ten minutes, then restart R to see)

To update:

devtools::install_github("IDEMSInternational/epicsadata")

[jkmusyoka]

Thanks @lilyclements. But I am still getting the same error 500 with the API after updating and running the script again. Not sure why. The error message is the same. I have now saved the data as zambia_eastern_2. Could you call the data from R to see if you get the same error or not?

[lilyclements]

Interesting. This is working for me. Can you try closing and reopening R, and running this line

devtools::install_github(“IDEMSInternational/epicsadata”, force = TRUE)

the force = TRUE says to install it even if it thinks it’s got the most recent version. Sometimes it just says it has got the most recent, but hasn’t.

Can you see if it works in R for you? To test in R

library(epicsawrap)
country <- "zm"
station_id <- "zambia_eastern"
annual_rainfall_summaries(country, station_id)

(edited to correct epicsawrap package name - originally wrote rpicsawrap!)

[lilyclements]

@jkmusyoka if you update your epicsadata branch again, then you can now use country = "zm_test"

[lilyclements]

@ChrisMarsh82 I added in yesterday in our meeting the ability to have "zm_test" - was your last package update before then? In any case, can you try updating the package again? Sometimes I find that R just ... lies and pretends it's updated:

devtools::install_github("IDEMSInternational/epicsadata", force = TRUE)

You'll know if it's updated if your get_bucket_name function has the option of "zm_test" in it - so if you run:

get_bucket_name

It'll give the R function output, and should say:

  if (length(country) > 1) stop("'country' must be length 1.")
  country <- match.arg(country)
  if (country == "mw") return("malawi_data")
  else if (country == "zm") return("zambia_data")
  else if (country == "zm_test") return("zambia_test_data")

Thanks!

And by default, R gives version 0.0.0.9000. Do you have a suggested version we should give at this stage?

[ChrisMarsh82]

This looks like it is an issue with my version as dont have 'get_bucket_name' function in either epicsadata or epicsawrap. I have 2 library paths defined and I think they may be conflicting each other.

As for the version I would go for 0.0.1 but I don't think it matters too much as long as it is incremented each time.

[lilyclements]

Try epicsadata:::get_bucket_name. Three colons (:::) works if you don't export the function in the R package. I think that might be the case here? (This shouldn't affect the API)

[ChrisMarsh82]

can use get_bucket_name but still have an issue with summaries

> epicsadata:::get_bucket_name
function (country = c("mw", "zm", "zm_test")) 
{
    if (length(country) > 1) 
        stop("'country' must be length 1.")
    country <- match.arg(country)
    if (country == "mw") 
        return("malawi_data")
    else if (country == "zm") 
        return("zambia_data")
    else if (country == "zm_test") 
        return("zambia_test_data")
}
<bytecode: 0x000001f264c1eec8>
<environment: namespace:epicsadata>
> epicsawrap::annual_rainfall_summaries("zm_test","test_1",c("annual_rain"),FALSE)
Error in match.arg(country) : 'arg' should be one of “mw”, “zm”

[lilyclements]

test_1 has been removed from the metadata so this wouldn't work. Can you try with

epicsawrap::annual_rainfall_summaries("zm_test","zambia_eastern_4",c("annual_rain"),FALSE)

If that gives the same error, then can you try updating your epicsadata package?

devtools::install_github("IDEMSInternational/epicsadata")

[lilyclements]

Changes to the R Code -

If you update rpicsa, epicsadata, and epicsawrap, then there are new additions to consider.

1. No longer read in tmin and tmax
2. Added definitions_id variable
3. You now have to put in the variable names of the following columns:
   #' start_rains_column The name of the start of rains column in the data.
   #' start_rains_status_column The name of the start of rains status column in the data.
   #' end_rains_column The name of the end of rains column in the data.
   #' end_season_column The name of the end of season column in the data.
   #' seasonal_length_column The name of the seasonal length column in the data.
   #' min_tmin_column The name of the minimum of minimum temperature column in the data.
   #' max_tmin_column The name of the maximum of minimum temperature column in the data.
   #' mean_tmin_column The name of the mean of minimum temperature column in the data.
   #' min_tmax_column The name of the minimum of maximum temperature column in the data.
   #' max_tmax_column The name of the maximum of maximum temperature column in the data.
   #' mean_tmax_column The name of the mean of maximum temperature column in the data.
   (There are default values, "start_rain", etc - whatever the default in R-Instat is for them)

export_r_instat_to_bucket(data = "ghana",
                          data_by_year = "ghana_by_station_year",
                          crop_data_name = "crop_def",
                          rain = "rainfall",
                          year = "year",
                          month = "month_abbr",
                          summaries = c("annual_rainfall", "annual_temperature", "extremes", "crop_success", "start_season"),
                          station_id = "test_3",
                          definitions_id = "1111",
                          country = "zm_test",
                          include_summary_data = TRUE,
                          annual_rainfall_data = annual_rain,
                          annual_temperature_data = annual_temp,
                          crop_success_data = crop_prop,
                          season_start_data = crop_def,
                          min_tmax_column = "min_tmax",
                          max_tmax_column = "max_tmax",
                          mean_tmax_column = "mean_tmax",
                          min_tmin_column = "min_tmin",
                          max_tmin_column = "max_tmin",
                          mean_tmin_column = "mean_tmin")

This is not yet reflected in the R-Instat dialog.

In the dialog, TODO:

1. Remove tmin and tmax options
2. Add in new boxes to fill in the column names of start_rains_status_column
3. Amend R code in R-Instat:

For start_rains_column, end_rains_column, end_season_column, seasonal_length_column - we already read these in in the sub-dialog for the reformat_annual_summaries function.
We want to amend the R code for these to be read as parameters in export_r_instat_to_bucket as well as reformat_annual_summaries

For min_tmax_column, max_tmax_column, mean_tmax_column, min_tmin_column, max_tmin_column, mean_tmin_column - we already read these in in the sub-dialog for the reformat_temperature_summaries function.
We want to amend the R code for these to be read as parameters in export_r_instat_to_bucket as well as reformat_temperature_summaries

(The name when that variable was created is that of the name of the definition we call to construct it. We call the column name here to get the correct definitions. Problem may arise if they have changed variable names. I should look into fixing that).

[lilyclements]

Changes outlined in R-Instat in #9018

[Vitalis95]

@lilyclements , we are encountering an error when we use Malawi dataset. Here is the code;

epicsawrap::gcs_auth_file(filename="C:/Users/User/Downloads/epicsa_token.json")

data_RDS_by_station_s_year <- data_book$get_data_frame(data_name="data_RDS_by_station_s_year")

annual_rain <- epicsawrap::reformat_annual_summaries(data=data_RDS_by_station_s_year, station_col="station", year_col="s_year", start_rains_doy_col="start_rain", start_rains_date_col="start_rain_date", end_rains_doy_col="end_rains", end_rains_date_col="end_rains_date", seasonal_rain_col="seasonal_rain_days", season_length_col="length", annual_rain_col="annual_rain_days")

data_RDS_by_station_s_year_month_abbr <- data_book$get_data_frame(data_name="data_RDS_by_station_s_year_month_abbr")
data_RDS_by_station_s_year_month_abbr<-na.omit(data_RDS_by_station_s_year_month_abbr)
annual_temp <- epicsawrap::reformat_temperature_summaries(data=data_RDS_by_station_s_year_month_abbr, station_col="station", year_col="s_year", mean_tmin_col="mean_tmin", min_tmin_col="min_tmin", max_tmin_col="max_tmin", mean_tmax_col="mean_tmax", min_tmax_col="min_tmax", max_tmax_col="max_tmax")

crop_prop <- data_book$get_data_frame(data_name="crop_prop")

crop_prop <- epicsawrap::reformat_crop_success(data=crop_prop, station_col="station", total_rain_col="rain_total", plant_day_col="plant_day", plant_length_col="plant_length", prop_success_col="prop_success")

crop_def <- data_book$get_data_frame(data_name="crop_def")

crop_def <- epicsawrap::reformat_season_start(data=crop_def, station_col="station", year_col="s_year", plant_day_col="plant_day", plant_day_cond_col="plant_day_cond")

data_RDS_by_station_s_year_month_abbr <- data_book$get_data_frame(data_name="data_RDS_by_station_s_year_month_abbr")

monthly_temp <- epicsawrap::reformat_temperature_summaries(data=data_RDS_by_station_s_year_month_abbr, station_col="station", year_col="s_year", month_col="month_abbr", mean_tmin_col="mean_tmin", min_tmin_col="min_tmin", max_tmin_col="max_tmin", mean_tmax_col="mean_tmax", min_tmax_col="min_tmax", max_tmax_col="max_tmax")

epicsawrap::export_r_instat_to_bucket(summaries=c("annual_rainfall", "annual_temperature", "crop_success", "monthly_temperature", "start_season"), annual_rainfall_data=annual_rain, data_by_year="data_RDS_by_station_s_year", mean_tmin_column="mean_tmin", min_tmin_column="min_tmin", max_tmin_column="max_tmin", mean_tmax_column="mean_tmax", min_tmax_column="min_tmax", max_tmax_column="max_tmax", start_rains_column="start_rain", data_by_year_month="data_RDS_by_station_s_year_month_abbr", annual_temperature_data=annual_temp, crop_data="crop_def", end_rains_column="end_rains", data="data_RDS", rain="rain", crop_success_data=crop_prop, season_start_data=crop_def, monthly_temperature_data=monthly_temp, year="s_year", month="month_abbr", seasonal_length_column="length", include_summary_data=TRUE, station_id="004", definitions_id="mala_9", country="ml_test")

This is the error we are getting
It looks like the function is expecting a boolean result (TRUE or FALSE), but it encounter a missing value

[jkmusyoka]

@lilyclements I get this same error when I try to upload summaries from Zambia Eastern by station.

And this is how I populated the dialogue

I will send the data by email as I can't attach an rds file here

[lilyclements]

I have fixed the bug you have reported - you can update and get the changes in devtools::install_github("IDEMSInternational/epicsadata")

Another (small) issue I spotted -

annual_rain <- epicsawrap::reformat_annual_summaries(data=data_RDS_by_station_s_year, station_col="station", year_col="s_year", start_rains_doy_col="start_rain", start_rains_date_col="start_rain_date", end_rains_doy_col="end_rains", end_rains_date_col="end_rains_date", seasonal_rain_col="seasonal_rain_days", season_length_col="length", annual_rain_col="annual_rain_days")

In here you are calling start_rains_date_col, end_rains_date_col, seasonal_rain_col, season_length_col, and annual_rain_col. The variables don't exist in the data set you provided. Also the annual_rain_col should give the rainfall amount not the number of rainy days