PR #530 follow up #533

jinningwang · 2024-04-14T13:11:07Z

Fill dae data from loaded csv file by a helper function _csv_data_to_dae()

Related to #526

…fter init()

jinningwang · 2024-04-14T14:07:30Z

andes/routines/tds.py

+            self.from_csv = system.options.get("from_csv")
+        if self.from_csv is not None:
+            self.data_csv = self._load_csv(self.from_csv)
+


_load_csv() after init(), to ensure smooth init when from_csv is given in TDS.run()

What was the issue with the previous order?

The issue is when CSV is given at "andes.load(from_csv='xx.csv')", TDS.init() run into error for the incompatible data size. It is because _load_csv() is called after TDS.init() in TDS.run().

We can forget this as the introduced helper function TDS._csv_output() addressed it.

jinningwang · 2024-04-14T14:08:30Z

andes/routines/tds.py

+                    xyidx = system.Output.xidx + [yidx+system.dae.n for yidx in system.Output.yidx]
+                    t = data[:, 0]
+                    _xy[:, xyidx] = data[:, 1:]
+                    data = np.hstack((t[:, np.newaxis], _xy))


padding data with zeros in _load_csv() to ensure only one-time execution.

cuihantao · 2024-04-14T15:03:44Z

andes/routines/tds.py

+            self.from_csv = system.options.get("from_csv")
+        if self.from_csv is not None:
+            self.data_csv = self._load_csv(self.from_csv)
+


What was the issue with the previous order?

cuihantao · 2024-04-14T15:07:36Z

andes/routines/tds.py

                    logger.warning("CSV data contains fewer variables than required.")
                    logger.warning("Check if the CSV data file is generated with selected output.")
+                else:
+                    # reconstruct data with padding zeros
+                    _xy = np.zeros((data.shape[0], system.dae.n + system.dae.m))


It might work, but it seems to waste a lot of memory.

If the system has 100 total variables, memory for all of them will be preallocated for each step. If the Output only has three variables, then only three out of 100 will be used.

Why not just store the indices? Do a one-time read-in of the CSV data. At each step, index into dae.x and dae.y with the stored indices and overwrite them with the CSV data.

Improved by adding a helper function to fit the recorded data in TDS._csv_step().

cuihantao · 2024-04-16T21:30:59Z

andes/routines/tds.py

@@ -1108,3 +1090,22 @@ def check_criteria(self):
        res = deltadelta(self.system.dae.x[self.system.SynGen.delta_addr],
                         self.config.ddelta_limit)
        return res
+
+    def _csv_output(self):


Consider renaming this for clarity. _csv_output reads like outputting to a CSV file.

Consider _csv_data_to_dae or similar.

Do you want to add a test case under tests by the way?

The changed lines are covered by following tests. Do you mean a test on the helper function itself?

andes/tests/test_output.py

Lines 78 to 129 in 7f56cc9

def test_from_csv(self):

"""

Test from_csv when loading selected output from csv file.

"""

case = andes.get_case("5bus/pjm5bus.json")

ss = andes.load(case,

no_output=True,

setup=False,

default_config=True)

ss.add("Output", {"model": "Bus", "varname": "v"})

ss.setup()

ss.PFlow.run()

ss.TDS.config.tf = 0.1

ss.TDS.run()

ss.TDS.load_plotter()

ss.TDS.plt.export_csv("pjm5bus_selec_out.csv")

# Test assign CSV in TDS.run()

ss2 = andes.load(case,

no_output=True,

setup=False,

default_config=True)

ss2.add("Output", {"model": "Bus", "varname": "v"})

ss2.setup()

ss2.PFlow.run()

ss2.TDS.run(from_csv="pjm5bus_selec_out.csv")

self.assertTrue(ss2.TDS.converged)

# Test assign CSV in andes.load()

ss3 = andes.load(case,

no_output=True,

setup=False,

default_config=True,

from_csv="pjm5bus_selec_out.csv")

ss3.add("Output", {"model": "Bus", "varname": "v"})

ss3.setup()

ss3.PFlow.run()

ss3.TDS.run()

self.assertTrue(ss3.TDS.converged)

os.remove("pjm5bus_selec_out.csv")

cuihantao · 2024-04-16T21:37:03Z

Oh the test looks good! Get Outlook for iOS<https://aka.ms/o0ukef>

________________________________ From: Jinning Wang ***@***.***> Sent: Tuesday, April 16, 2024 4:36:11 PM To: CURENT/andes ***@***.***> Cc: Hantao Cui ***@***.***>; Review requested ***@***.***> Subject: Re: [CURENT/andes] PR #530 follow up (PR #533) @jinningwang commented on this pull request.

________________________________ In andes/routines/tds.py<#533 (comment)>:

@@ -1108,3 +1090,22 @@ def check_criteria(self):

res = deltadelta(self.system.dae.x[self.system.SynGen.delta_addr], self.config.ddelta_limit) return res + + def _csv_output(self): The changed lines are covered by following tests. Do you mean a test on the helper function itself? https://github.com/CURENT/andes/blob/7f56cc9ea4c16d9d58f5b440984f7ee2e6fd1018/tests/test_output.py#L78-L129 — Reply to this email directly, view it on GitHub<#533 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABSNZA2KNQFKU5PRWU67W3DY5WKUXAVCNFSM6AAAAABGGFCRLSVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDAMBUGYZTSMRVGY>. You are receiving this because your review was requested.Message ID: ***@***.***>

cuihantao · 2024-04-16T23:22:31Z

Jinning, thanks a lot!

jinningwang added 3 commits April 14, 2024 08:16

Add note to _load_csv

669818d

Move CSV data completetion in _load_csv; In run(), move _load_csv() a…

e4bf4e4

…fter init()

Update release-notes

febb68c

jinningwang requested a review from cuihantao April 14, 2024 14:04

jinningwang commented Apr 14, 2024

View reviewed changes

cuihantao requested changes Apr 15, 2024

View reviewed changes

Improve _csv_step() by adding a helper function

6792561

cuihantao requested changes Apr 16, 2024

View reviewed changes

jinningwang added 2 commits April 16, 2024 17:38

Rename TDS._csv_output() as TDS._csv_data_to_dae() for clarity

e0a703a

Minor fix

65e10e0

cuihantao approved these changes Apr 16, 2024

View reviewed changes

cuihantao merged commit 6e178a2 into CURENT:develop Apr 16, 2024
3 checks passed

jinningwang deleted the ops branch October 5, 2024 18:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PR #530 follow up #533

PR #530 follow up #533

jinningwang commented Apr 14, 2024 •

edited

Loading

jinningwang Apr 14, 2024

cuihantao Apr 14, 2024

jinningwang Apr 16, 2024

jinningwang Apr 16, 2024

jinningwang Apr 14, 2024

cuihantao Apr 14, 2024

cuihantao Apr 14, 2024

cuihantao Apr 14, 2024

jinningwang Apr 16, 2024

cuihantao Apr 16, 2024

cuihantao Apr 16, 2024

jinningwang Apr 16, 2024

cuihantao commented Apr 16, 2024 via email

cuihantao commented Apr 16, 2024

	def test_from_csv(self):
	"""
	Test from_csv when loading selected output from csv file.
	"""
	case = andes.get_case("5bus/pjm5bus.json")
	ss = andes.load(case,
	no_output=True,
	setup=False,
	default_config=True)

	ss.add("Output", {"model": "Bus", "varname": "v"})

	ss.setup()

	ss.PFlow.run()
	ss.TDS.config.tf = 0.1
	ss.TDS.run()
	ss.TDS.load_plotter()

	ss.TDS.plt.export_csv("pjm5bus_selec_out.csv")

	# Test assign CSV in TDS.run()
	ss2 = andes.load(case,
	no_output=True,
	setup=False,
	default_config=True)

	ss2.add("Output", {"model": "Bus", "varname": "v"})

	ss2.setup()

	ss2.PFlow.run()
	ss2.TDS.run(from_csv="pjm5bus_selec_out.csv")

	self.assertTrue(ss2.TDS.converged)

	# Test assign CSV in andes.load()
	ss3 = andes.load(case,
	no_output=True,
	setup=False,
	default_config=True,
	from_csv="pjm5bus_selec_out.csv")
	ss3.add("Output", {"model": "Bus", "varname": "v"})

	ss3.setup()

	ss3.PFlow.run()
	ss3.TDS.run()

	self.assertTrue(ss3.TDS.converged)

	os.remove("pjm5bus_selec_out.csv")

PR #530 follow up #533

PR #530 follow up #533

Conversation

jinningwang commented Apr 14, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cuihantao commented Apr 16, 2024 via email

cuihantao commented Apr 16, 2024

jinningwang commented Apr 14, 2024 •

edited

Loading