Skip to content

Commit

Permalink
update docs and patch settings/tables 🐞 (#637)
Browse files Browse the repository at this point in the history
  • Loading branch information
@jacksund
jacksund authored Mar 26, 2024
1 parent e5b0f87 commit b7d0e27
Show file tree
Hide file tree
Showing 9 changed files with 2,982 additions and 2,912 deletions.
10 changes: 0 additions & 10 deletions .gitattributes
View file

This file was deleted.

979 changes: 498 additions & 481 deletions docs/change_log.md
View file

Large diffs are not rendered by default.

358 changes: 198 additions & 160 deletions docs/getting_started/custom_tables_and_apps/create_a_custom_app.md
View file

Large diffs are not rendered by default.

271 changes: 137 additions & 134 deletions docs/getting_started/custom_tables_and_apps/create_a_custom_table.md
View file
Original file line number Diff line number Diff line change
@@ -1,135 +1,138 @@
# Creating Custom Database Tables

-------------------------------------------------------------------------------

## Building Custom Tables

1. Review the tutorial on [python inheritance](../../database/intro_to_python_inheritance/) (from the `How tables are built` tutorial in the `Database` section). This topic is important to understanding when building new tables.

2. In your new app, open the file `example_app/models.py` and you will see two example tables defined for you. This is all you need to build tables! Note, how we also use `Structure` and `Calculation` to add a bunch of columns to in the 2nd table:
``` python
class MyCustomTable1(DatabaseTable):
custom_column_01 = table_column.IntegerField()
custom_column_02 = table_column.FloatField()

class MyCustomTable2(Structure, Calculation):
input_01 = table_column.FloatField()
input_02 = table_column.BooleanField()
output_01 = table_column.FloatField(null=True, blank=True)
output_02 = table_column.BooleanField(null=True, blank=True)
```

!!! tip
We only demonstrate `Structure` and `Calculation` in the examples shown, but there are many more `base_data_types` you can utilize. Make sure to review our guides in the [`simmate.database`](/full_guides/database/overview/) module for more information.

!!! info
In Django (which Simmate uses under the hood), a `DatabaseTable` is referred to as a `Model`. Therefore, a model and table can be considered the same. As we're using Django, the file name `models.py` must remain as-is. That's where Django searches for your custom database tables.

3. These examples will build tables with the following columns:
```
# TABLE 1
# TABLE 2
```

4. Note... These tables are not actually in your database yet. You can check your tables in DBeaver to confirm this. Next, we will actually "apply" them to our database.

-------------------------------------------------------------------------------

## Applying Changes to the Database

1. Open the `example_app/models.py` file and review the example tables.

2. Ensure the Simmate configuration includes your new app:
``` bash
simmate config show --user-only
```

3. Add these tables to the database by updating it:
``` bash
simmate database update
```

4. Verify the output of your `update` command. You should see that your new app and tables have been added.
``` bash
Migrations for 'example_app':
example_app/migrations/0001_initial.py
- Create model MyCustomTable1
- Create model MyCustomTable2
```

!!! danger
All changes are tracked as "migrations" and stored in an extra folder within your app. Typically, you can just continue to let these files build because migrations are designed to be committed to, and distributed as part of, your codebase. Make sure to read & understand Django's [official guides](https://docs.djangoproject.com/en/5.0/topics/migrations/) on how to interact with these files.

5. Check DBeaver and you should see new tables such as `example_app_mycustomtable1` and `example_app_mycustomtable2`

6. In a separate python script, let's ensure we can view the new tables in your database:
``` python
from simmate.database import connect

from example_app.models import MyCustomTable1, MyCustomTable2

# both tables should output 0 because we haven't added data yet
n1 = MyCustomTable1.objects.count()
n2 = MyCustomTable2.objects.count()
```

7. You now have your tables in the database!

!!! note
If you ever change your tables (such as adding or removing a column), you will need to re-update your database. Therefore, whenever you modify the `models.py` file, make sure to run `simmate database update` for your changes to take effect in your database.

-------------------------------------------------------------------------------

## Adding Data to Your Tables

### Basic Population

For basic tables, you can set each column manually:

``` python
from simmate.database import connect
from example_app.models import MyCustomTable1

new_entry = MyCustomTable1(
custom_column_01=123,
custom_column_02=3.14,
)
new_entry.save()
```

Once saved, try viewing your new row in DBeaver!

### Advanced population

For more complex tables that include Simmate data types, we automatically populate data using the `from_toolkit` method:

``` python
from simmate.database import connect
from simmate.toolkit import Structure

from example_app.models import MyCustomTable2

nacl = Structure.from_file("NaCl.cif")

new_entry = MyCustomTable2.from_toolkit(structure=nacl)
new_entry.save()
```

Once saved, try viewing your new row in DBeaver. You will see many columns were filled out automatically. Some columns are still empty though (such as those from a `Calculation`). We will learn how to populate these tables in the next tutorial on `Adding app workflows`

!!! tip
Manually populating your table with data is often unnecessary. Instead, you can connect your database to a workflow, which will automatically fill it with data. We'll cover this in a later step.

-------------------------------------------------------------------------------

## Searching Your New Data

You can filter results just like any other table. If you need a refresher, refer back to the earlier database tutorial or refer to the Full Guides for more info:

``` python
df = MyCustomTable2.objects.filter(nsites__lte=10).to_dataframe()
```

# Creating Custom Database Tables

-------------------------------------------------------------------------------

## Building Custom Tables

1. Review the tutorial on [python inheritance](../../database/intro_to_python_inheritance/) (from the `How tables are built` tutorial in the `Database` section). This topic is important to understanding when building new tables.

2. In your new app, open the file `example_app/models.py` and you will see two example tables defined for you. This is all you need to build tables! Note, how we also use `Structure` and `Calculation` to add a bunch of columns to in the 2nd table:
``` python
class MyCustomTable1(DatabaseTable):
custom_column_01 = table_column.IntegerField()
custom_column_02 = table_column.FloatField()

class MyCustomTable2(Structure, Calculation):
input_01 = table_column.FloatField()
input_02 = table_column.BooleanField()
output_01 = table_column.FloatField(null=True, blank=True)
output_02 = table_column.BooleanField(null=True, blank=True)
```

!!! tip
We only demonstrate `Structure` and `Calculation` in the examples shown, but there are many more `base_data_types` you can utilize. Make sure to review our guides in the [`simmate.database`](/full_guides/database/overview/) module for more information.

!!! info
In Django (which Simmate uses under the hood), a `DatabaseTable` is referred to as a `Model`. Therefore, a model and table can be considered the same. As we're using Django, the file name `models.py` must remain as-is. That's where Django searches for your custom database tables.

3. These examples will build tables with the following columns:
```
# TABLE 1
# TABLE 2
```

4. Note... These tables are not actually in your database yet. You can check your tables in DBeaver to confirm this. Next, we will actually "apply" them to our database.

-------------------------------------------------------------------------------

## Applying Changes to the Database

1. Open the `example_app/models.py` file and review the example tables.

2. Ensure the Simmate configuration includes your new app:
``` bash
simmate config show --user-only
```

3. Add these tables to the database by updating it:
``` bash
simmate database update
```

!!! note
The `update` command attempts to add any changes to your current database. If you wish to start from a clean and empty database, you can use `reset` instead.

4. Verify the output of your `update` command. You should see that your new app and tables have been added.
``` bash
Migrations for 'example_app':
example_app/migrations/0001_initial.py
- Create model MyCustomTable1
- Create model MyCustomTable2
```

!!! danger
All changes are tracked as "migrations" and stored in an extra folder within your app. Typically, you can just continue to let these files build because migrations are designed to be committed to, and distributed as part of, your codebase. Make sure to read & understand Django's [official guides](https://docs.djangoproject.com/en/5.0/topics/migrations/) on how to interact with these files.

5. Check DBeaver and you should see new tables such as `example_app_mycustomtable1` and `example_app_mycustomtable2`

6. In a separate python script, let's ensure we can view the new tables in your database:
``` python
from simmate.database import connect

from example_app.models import MyCustomTable1, MyCustomTable2

# both tables should output 0 because we haven't added data yet
n1 = MyCustomTable1.objects.count()
n2 = MyCustomTable2.objects.count()
```

7. You now have your tables in the database!

!!! note
If you ever change your tables (such as adding or removing a column), you will need to re-update your database. Therefore, whenever you modify the `models.py` file, make sure to run `simmate database update` for your changes to take effect in your database.

-------------------------------------------------------------------------------

## Adding Data to Your Tables

### Basic Population

For basic tables, you can set each column manually:

``` python
from simmate.database import connect
from example_app.models import MyCustomTable1

new_entry = MyCustomTable1(
custom_column_01=123,
custom_column_02=3.14,
)
new_entry.save()
```

Once saved, try viewing your new row in DBeaver!

### Advanced population

For more complex tables that include Simmate data types, we automatically populate data using the `from_toolkit` method:

``` python
from simmate.database import connect
from simmate.toolkit import Structure

from example_app.models import MyCustomTable2

nacl = Structure.from_file("NaCl.cif")

new_entry = MyCustomTable2.from_toolkit(structure=nacl)
new_entry.save()
```

Once saved, try viewing your new row in DBeaver. You will see many columns were filled out automatically. Some columns are still empty though (such as those from a `Calculation`). We will learn how to populate these tables in the next tutorial on `Adding app workflows`

!!! tip
Manually populating your table with data is often unnecessary. Instead, you can connect your database to a workflow, which will automatically fill it with data. We'll cover this in a later step.

-------------------------------------------------------------------------------

## Searching Your New Data

You can filter results just like any other table. If you need a refresher, refer back to the earlier database tutorial or refer to the Full Guides for more info:

``` python
df = MyCustomTable2.objects.filter(nsites__lte=10).to_dataframe()
```

-------------------------------------------------------------------------------
114 changes: 57 additions & 57 deletions docs/getting_started/database/dbeaver.md
View file
Original file line number Diff line number Diff line change
@@ -1,57 +1,57 @@

# Viewing data with DBeaver

!!! note
DBeaver is 100% optional, but without it, you may struggle to understand what your database is & what it even looks like!

----------------------------------------------------------------------

## Why DBeaver?

Just as we use Excel for reading and analyzing `.xlsx` or `.csv` files, DBeaver is our go-to program for working with data from SQL databases.

This includes managing files like `.sqlite3`, which is the format our Simmate database is currently in. In more advanced cases, we can even use DBeaver to connect to cloud-hosted databases, such as Postgres, which we will cover in a later tutorial.

Ultimately, DBeaver will let us view data as if we were in Excel:

![Table in DBeaver](https://dbeaver.com/wp-content/uploads/2022/03/screen_data_editor.png)

----------------------------------------------------------------------

## Install DBeaver

Download install [DBeaver Lite](https://dbeaver.io/download/). It is free, and you do not need to make an account.

DBeaver is just a regular desktop app, so open it up once installed. You'll see a variety of components when you first open it (but they will be empty at first):

![DBeaver interface](https://dbeaver.com/wp-content/uploads/wikidocs_cache/images/ug/appwindow-with-markup.png)

----------------------------------------------------------------------

## Connect to the Simmate Database

1. Select the "New Database Connection" button. This button is in the toolbar at the top of the page and looks like this: ![new db button](https://dbeaver.com/wp-content/uploads/wikidocs_cache/images/ug/new-connection-wizard-button.png)

2. In the pop-up, you'll see a TON of database types that you can connect to. Search for "SQLite", select it, and then hit "Next". Its icon looks like this: ![new db button](https://www.sqlite.org/images/sqlite370_banner.gif){ width="50" }

3. Keep the `Connect by:` setting as `Host`. Then click the `Open ...` button and find your SQLite file. Remember, this file is located at `~/simmate/my_env-database.sqlite3`. Once selected, you can hit `Test Connection ...` (and install any necessary drivers if prompted). Hit `Finish` when everything looks good!

4. You'll now see your Simmate database in the `Database Navigator`! (left side panel)

----------------------------------------------------------------------

## Viewing a Table's Data

1. Find your database in `Database Navigator` and `Tables` folder and see everything that Simmate built for you.
2. Scroll down to the `workflows_staticenergy` table and double click to open it in the `Data Editor` (your main panel).
3. You'll see three tabs at the top of the `Data Editor`: `Properties`, `Data`, and `ER Diagram`. Currently, we are looking at the `Properties` tab, which shows us a list of all the columns this table has. To view the actual table and data, select the `Data` tab.
4. If you followed all tutorials, you should see some data from your workflow runs! If you accidentally reset your database, the table will be empty.
5. Take a look at what information was stored from your calculation(s). There are a TON of columns available, so click around / sort by column / fiddle with your data. In the next tutorial, we will learn about why there are so many columns and where they come from.

!!! tip
If you'd like, you can export the table to a `.csv` file that you can then open in Excel. There is an `Export data` button at the bottom of your `Data` view.

!!! danger
DBeaver let's you click on & update values within your table. Avoid doing this! You can corrupt and/or misrepresent your data this way. We recommend using DBeaver as if you have read-only permissions to your database.

----------------------------------------------------------------------

# Viewing data with DBeaver

!!! note
DBeaver is 100% optional, but without it, you may struggle to understand what your database is & what it even looks like!

----------------------------------------------------------------------

## Why DBeaver?

Just as we use Excel for reading and analyzing `.xlsx` or `.csv` files, DBeaver is our go-to program for working with data from SQL databases.

This includes managing files like `.sqlite3`, which is the format our Simmate database is currently in. In more advanced cases, we can even use DBeaver to connect to cloud-hosted databases, such as Postgres, which we will cover in a later tutorial.

Ultimately, DBeaver will let us view data as if we were in Excel:

![Table in DBeaver](https://dbeaver.com/wp-content/uploads/2022/03/screen_data_editor.png)

----------------------------------------------------------------------

## Install DBeaver

Download install [DBeaver Lite](https://dbeaver.io/download/). It is free, and you do not need to make an account.

DBeaver is just a regular desktop app, so open it up once installed. You'll see a variety of components when you first open it (but they will be empty at first):

![DBeaver interface](https://dbeaver.com/wp-content/uploads/wikidocs_cache/images/ug/appwindow-with-markup.png)

----------------------------------------------------------------------

## Connect to the Simmate Database

1. Select the "New Database Connection" button. This button is in the toolbar at the top of the page and looks like this: ![new db button](https://dbeaver.com/wp-content/uploads/wikidocs_cache/images/ug/new-connection-wizard-button.png)

2. In the pop-up, you'll see a TON of database types that you can connect to. Search for "SQLite", select it, and then hit "Next". Its icon looks like this: ![new db button](https://www.sqlite.org/images/sqlite370_banner.gif){ width="50" }

3. Keep the `Connect by:` setting as `Host`. Then click the `Open ...` button and find your SQLite file. Remember, this file is located at `~/simmate/my_env-database.sqlite3`. Once selected, you can hit `Test Connection ...` (and install any necessary drivers if prompted). Hit `Finish` when everything looks good!

4. You'll now see your Simmate database in the `Database Navigator`! (left side panel)

----------------------------------------------------------------------

## Viewing a Table's Data

1. Find your database in `Database Navigator` and `Tables` folder and see everything that Simmate built for you.
2. Scroll down to the `workflows_staticenergy` table and double click to open it in the `Data Editor` (your main panel).
3. You'll see three tabs at the top of the `Data Editor`: `Properties`, `Data`, and `ER Diagram`. Currently, we are looking at the `Properties` tab, which shows us a list of all the columns this table has. To view the actual table and data, select the `Data` tab.
4. If you followed all tutorials, you should see some data from your workflow runs! If you accidentally reset your database, the table will be empty.
5. Take a look at what information was stored from your calculation(s). There are a TON of columns available, so click around / sort by column / fiddle with your data. In the next tutorial, we will learn about why there are so many columns and where they come from.

!!! tip
If you'd like, you can export the table to a `.csv` file that you can then open in Excel. There is an `Export data` button at the bottom of your `Data` view.

!!! danger
DBeaver let's you click on & update values within your table. Avoid doing this! You can corrupt and/or misrepresent your data this way. We recommend using DBeaver as if you have read-only permissions to your database.

----------------------------------------------------------------------
Loading

0 comments on commit b7d0e27

Please sign in to comment.