Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Memory only databases option (stats , queries etc) #5992

Open
3 tasks done
finBitorg opened this issue Jul 10, 2023 · 10 comments
Open
3 tasks done

Memory only databases option (stats , queries etc) #5992

finBitorg opened this issue Jul 10, 2023 · 10 comments

Comments

@finBitorg
Copy link

Prerequisites

  • I have checked the Wiki and Discussions and found no answer

  • I have searched other issues and found no duplicates

  • I want to request a feature or enhancement and not ask a question

The problem

Adguard is writing a ton of queries to disk and this is often causes wear on SDCard with Raspberry pi.

Proposed solution

please add a configuration setting to keep the query cache and query stats "in memory only" since you already have a max cache size option anyway this should limit memory use to whatever user wants. I am totally ok with loosing the stats on occasional reboot as opposed to having the Sdcards fail

Alternatives considered and additional information

trying to set overlay FS on raspberrypi but this will not allow for adguard updates and uses more memory then an option to just store query data and cache in memory with a specified size limit

@ainar-g
Copy link
Contributor

ainar-g commented Jul 10, 2023

For query logs, there is already the querylog.file_enabled option. I think we can add something similar for stats as well.

@finBitorg
Copy link
Author

That would be great; also if possible that be exposed as a checkbox in Web UI setting section.

@nunu6689
Copy link

Any update on this?
AGH killed my 2 sdcard. I give up on historical stats. A few days stats and queries while running is enough. I disabled stats for now.

adguard pushed a commit that referenced this issue Feb 12, 2024
Updates #5992.

Squashed commit of the following:

commit 39d3df7
Author: Stanislav Chzhen <[email protected]>
Date:   Mon Feb 12 15:25:22 2024 +0300

    all: fix typo

commit 21e03e4
Author: Stanislav Chzhen <[email protected]>
Date:   Mon Feb 12 15:22:35 2024 +0300

    all: upd docs

commit 1118006
Merge: ac5fd8d 7f83707
Author: Stanislav Chzhen <[email protected]>
Date:   Mon Feb 12 14:57:35 2024 +0300

    Merge branch 'master' into 5992-stats-qlog-custom-dir

commit ac5fd8d
Author: Stanislav Chzhen <[email protected]>
Date:   Wed Feb 7 14:55:51 2024 +0300

    all: upd chlog

commit fe00652
Merge: 21ad1ec 56b9808
Author: Stanislav Chzhen <[email protected]>
Date:   Wed Feb 7 14:53:24 2024 +0300

    Merge branch 'master' into 5992-stats-qlog-custom-dir

commit 21ad1ec
Author: Stanislav Chzhen <[email protected]>
Date:   Wed Feb 7 14:52:12 2024 +0300

    home: imp docs

commit 739b158
Author: Stanislav Chzhen <[email protected]>
Date:   Mon Feb 5 20:15:50 2024 +0300

    home: add stats qlog custom dir
@schzhn
Copy link
Member

schzhn commented Feb 15, 2024

Since v0.108.0-b.53, you can specify tmpfs directory to store query log and statistics using the new querylog.dir_path and statistics.dir_path properties in the configuration file.

@nunu6689
Copy link

Since v0.108.0-b.53, you can specify tmpfs directory to store query log and statistics using the new querylog.dir_path and statistics.dir_path properties in the configuration file.

That's good.
If I specify a directory path would AGH still store the stats in ram as well? That would double the ram usage then. For small ram like a router it does matter.
Since the query logs has in memory only option with query_logs.file_enabled why not make it for the stats also?

@ElTopo
Copy link

ElTopo commented Mar 18, 2024

Any update on this? AGH killed my 2 sdcard. I give up on historical stats. A few days stats and queries while running is enough. I disabled stats for now.

Although this feature request (i.e., writing constantly changes to tmpfs) helps your SD card a lot, you should consider using overlayroot of Raspberry Pi OS (sudo raspi-config / Performance Options / Overlay File System), after you configured your Pi properly and enabled overlayroot, all changes (were to your SD card) will be written in RAM (and discarded when you reboot the system), your SD card will be fully read-only and no bytes will be written to it, and your Pi will survive even when it loses power. It should work with other processes that write too much to your SD card.

@ainar-g
Copy link
Contributor

ainar-g commented Mar 20, 2024

@ElTopo, as @schzhn mentioned, v0.107.46 includes the ability to change statistics and querylog directories without changing the entire workdir, which should allow using e.g. tmpfs for that.

Any real in-memory statistics storage would have to wait either until v0.108.0 and #2290 or etcd-io/bbolt#227.

@tipuraneo
Copy link

I updated to v0.107.46 but could not find the option – neither in the gui nor in https://github.com/AdguardTeam/AdGuardHome/wiki/Configuration
Where is the option located?

@jwgn
Copy link

jwgn commented Mar 21, 2024

The option is dir_path and it is mentioned in the documentation.

For example, this is my current configuration:

querylog:
  dir_path: ""
  ignored: []
  interval: 168h
  size_memory: 1000
  enabled: true
  file_enabled: false
statistics:
  dir_path: /tmp
  ignored: []
  interval: 720h
  enabled: true

In my case, /tmp is tmpfs. You can also use /dev/shm if your /tmp is on disk.

@ElTopo
Copy link

ElTopo commented Mar 22, 2024

I found the easiest way to write log/db in memory is:

sudo systemctl stop AdGuardHome.service
cd AdGuardHome
rm -rf data
ln -sf /dev/shm data
sudo systemctl start AdGuardHome.service

/dev/shm is typically created using 50% of memory. you can create your own tmpfs in /etc/fstab if you want more, for example
tmpfs /mnt/mycache tmpfs nodev,nosuid,relatime,size=80%,mode=0777 0 0
then use /mnt/mycache as data directory.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

7 participants