Out of memory

[unclefab]

I'm on v 1.77.
When I try to import my settings back up file it stops like in the middle of the process and gives me an java.lang.OutOfMemory error.

[M66B]

Out of memory errors cannot be fixes, they are what they are.

[unclefab]

Ok, but I didn't have this problem with previous versions, actually until 1.75...

[M66B]

I guess you have set more restrictions since then.

[unclefab]

Mmmmh, I see... So I will investigate, thanks for the info!!!

[pogovault]

I have same issue with import.
I have 16GB HTC One S with very moderate amount of apps: 199 in total - 55 user apps and 144 system apps, as reported by Titanium Backup. I have around 680MB of free internal storage and around 520MB of free RAM in everyday use.
Currently, my XPrivacy xml file size is 303KB (yes, I have a lot of restrictions active). There are 3123 entries for all these apps - an average 15 per app. And each restriction added in XPrivacy app make my file bigger even if I don't change anything.
I've tried to reinstall XPrivacy and Xposed, also tried to kill as many apps as possible to free more RAM but nothing helps. My workaround is to Titanium Backup to restore XPrivacy, but others may not have backup.
I understand this is not strictly XPRivacy problem, but you might look at this issue. Users who have devices with less memory and/or more apps should be affected as soon as they start do dig deeper and apply more restrictions.
I don't know anything about Android development, but have some programming background. My suggestions could be an educated guess at bwest, but here are two ideas:

• Instead of loading complete xml file in memory during the import, load the file in chunks of manageable size.
• Apply reverse logic on xml entries: all permissions and functions should be consider restricted (checked) by default and xml file will contain only unchecked (not restricted) items.

[M66B]

I will see how I can improve this.
Hopefully I don't have to write my own XML parser and can the built-in parser be tweaked.

[M66B]

I will switch to the SAX parser to save memory.
http://developer.android.com/reference/javax/xml/parsers/SAXParser.html

[M66B]

Could you please test if this version works okay for you: http://d-h.st/Eyb
(if you have a file manager installed, there is a bonus ;-)

[M66B]

See also #327

[pogovault]

Same error, but work longer than before.

Marcel Bokhorst [email protected] wrote:

Could you please test if this version works okay for you:
http://d-h.st/Eyb
(if you have a file manager installed, there is a bonus ;-)

---

Reply to this email directly or view it on GitHub:
#329 (comment)

Sent from my Android phone with K-9 Mail. Please excuse my brevity.

[M66B]

There is little more I can do than I already did, sorry.

[GhostlyCrowd]

Maybe change xprivacy to use a sql database. and export the data base as such.

May solve the memory errors and if you locate the database in /data like pdroid it will keep nosy and malicious apps out of it.

[M66B]

That would not be compatible across devices.

[GhostlyCrowd]

Doesnt just about every messaging app use SQL for its database? android includes sql as well as default. and 2.3+ allow the use of the WAL sql database system. It would probably improve the performance alot if this can be done.

Can you explain more why it would not be corss-device compatible? I dont understand why it wouldn't considering my above statements. As for putting it in /data your app will need rood access. but every one who uses exposed framework has to have root anyways. so no issues there.

[M66B]

There are different versions of SQLite database, depending on Android version.

[GhostlyCrowd]

well, you can address it the same way other devs do could you not? You could even include your own version of sql binary. alot of backup apps and tweaking apps do this already.

[M66B]

Too much work that I rather spent on something else (for example restricting Android 4.3)

[GhostlyCrowd]

Fair enough. But If this goes un solved, the inability to import ones settings is going to detour some users and pro users.

I don't have the issue "yet"

But keep in the back of your mind that this issue is fixable with my suggestion of moving to a sql database, and eventually people will expect it to be. I would make this "low prio", not "will not fix" in my opinion.

And again just to mention it would probably improve performance, which i know you have been aiming for.

[M66B]

@GhostlyCrowd
An SQLite database doesn't necessarily use less memory.
The situation with the new XML parser has greatly been improved.
The memory footprint of the new parser is magnitudes lower that the old one.

@pogovault which device are you using and how much free memory do you have?

[GhostlyCrowd]

Also the answer to this stack overflow question hits the nail on the head for the issues your having

http://stackoverflow.com/questions/5213550/raw-resources-versus-sqlite-database/5213569#5213569

easier and faster to retrieve sort and filter data and it has none of the limitations you are getting with SAX or the previous xml parser.

As xprivacy grows so will the things its filtering, sorting and what not. I still suggest you consider this.

As devices become more powerful it will not be uncommon for a user to have 300-600 apps. So what then?

if users are having issues importing settings for 200-300 apps now. thi9nk about the future.

[pogovault]

HTC One S, have around 500mb of RAM free.

Marcel Bokhorst [email protected] wrote:

@GhostlyCrowd
An SQLite database doesn't necessarily use less memory.
The situation with the new XML parser has greatly been improved.
The memory footprint of the new parser is magnitudes lower that the old
one.

@pogovault which device are you using and how much free memory do you
have?

---

Reply to this email directly or view it on GitHub:
#329 (comment)

Sent from my Android phone with K-9 Mail. Please excuse my brevity.

[GhostlyCrowd]

Here is something i have been able to replicate with xprivacy as well.

I wanted to filter all my apps, including system. I know this breaks android and you boot loop but the purpose to check "All" was so i could then sort and unchecked what i needed to unblock. much faster this way.

So check "All" xprivacy will get an out of memory error and fail to check anything. Check "All" again it will check everything and then xprivacy starts to freeze and lock up. Android asks if you wish to wait for xprivacy and you can, or some times it just blatantly crashes.

I am running cm with about 213 apps including system.

Using the all option just kills xprivacy performance. even with SAX

[M66B]

This doesn't say anything about memory usage when reading/writing XML files (XPrivacy doesn't manipulate the XML files in any other way). XML is the standard to export/import data. Some people edit the exported file manually.

Filtering/sorting has nothing to do with this.

Again: I don't expect problems with the new XML parser, even with a lot of apps, unless the device is already low on memory. The XML parse process as it is now uses hardly any memory.

"... the minimum memory required for a SAX parser is proportional to the maximum depth of the XML file ... and the maximum data involved in a single XML event ..."
https://en.wikipedia.org/wiki/Simple_API_for_XML
The depth of XML of the XPrivacy XML is the minimum you can get: one and the data involved is almost nothing.

[M66B]

'All' does nothing with the XML file.

[GhostlyCrowd]

then how is the database kept of what you have checked i thought it was all xml? The app keeps a list of your settings i assumed if it wasn't sql it was xml since thats what import and xport use

[M66B]

It uses shared preferences.
http://developer.android.com/reference/android/content/SharedPreferences.html

[GhostlyCrowd]

thanks for the clarification, i assumed the fc and freezing was all related to the xml issue. which is why i have not bothered to report it.

[GhostlyCrowd]

I see that in shared_prefs all the apps settings are still xml, they are not affected then by the import xport xml memory issue?

I dunno then, i was hoping switching to sql would increase the performance as well as fix the import export for the users.

[cobaltdr]

My comments were actually for 1.7.16. I'm trying 1.7.15 now. What app can I use to check memory use by apps? First import on 1.7.15 completely froze my phone part of the way through the procedure.

[M66B]

Version 1.7.18 might solve your problem.

[cobaltdr]

Uh oh. No.

Never had this happen with Xprivacy before.

Import causes reboot then bootloops, had to flash remover.

[cobaltdr]

I had 1.7.17 before that happened. 1.7.17 only gives out of memory. 1.7.18 bootloops. Had to flash remover. Reinstalling 1.7.17 still bootlooped. Had to flash remover again. 1.7.16 now allows phone to boot.

[M66B]

I see no change which can cause boot loops.
Can you give me a logcat of the boot loop?

[cobaltdr]

That will require adb, I have no setup where I can run adb unfortunately. Reinstalling 1.7.17 after the bootloops have stopped now re allows the phone to boot with 1.7.17. I will see if I can get a logcat of 1.7.18 causing the initial crash.

[cobaltdr]

After 1.7.18 bootloops, flashing the remover recovers the system. However, every subsequent install of Xprivacy of any version down to 1.7.15 carries on causing the bootloop, until after installation of the apk you attempt to run Xprivacy. It complains that it is not enabled in Xposed, and when you close that dialog, it is refreshing its app list. Usage data and settings appear preserved. Rebooting after that gives a working phone and Xprivacy.

I am not on my laptop at the moment, so I cannot give you a logcat, but I will mail you them as soon as I get a chance. 1.7.18 is a no-no for me. Had never had any bootloops before.

[M66B]

There is already a 1.7.19

[cobaltdr]

Haha. I'm almost scared to upgrade now. I really appreciate the effort you're putting into this app, and hope more people realise this and contribute.

[cobaltdr]

1.7.19 causes a crash and reboot when importing, but no bootloop. Phone appears to take a bit longer to boot. However, 1.7.19 is really, really slick. Nice interface. Usage data is very well integrated. I will try to get a catalog of this when I get a chance. By tomorrow hopefully. Oh and the app list loads faster, did you do something to that?

[cobaltdr]

1.7.20 just fails, notification freezes. Most of the apps I have taking up memory are system apps.

[cobaltdr]

1.7.22 still has the out of memory issue, but only after trying to export. Doesn't the dalvik vm deal with memory management? Before import I have at least 100MB free.

[GhostlyCrowd]

You've probably said before but whats your devices specs for ram and storage. also

adb shell "cat /system/build.prop | grep heap"

see what your heaps are.

May help give M66B a better idea of your memory circumstances.

[cobaltdr]

Galaxy S3 GT-I9300 with 1GB RAM.

dalvik.vm.heapstartsize=8m
dalvik.vm.heapgrowthlimit=64m
dalvik.vm.heapsize=128m

[GhostlyCrowd]

do this

adb shell "setprop dalvik.vm.heapsize 256m"

dont reboot and try seeing if you can make xprivacy run out of memory.

my heap is 512mb but i have 2gb of ram. 128 seems low.

[GhostlyCrowd]

i edited it so make sure u refresh my commands were incorrect

[GhostlyCrowd]

also adb shell "setprop dalvik.vm.heapgrowthlimit 128m"

[M66B]

@cobaltdr as @GhostlyCrowd suggest you should really look into your configuration.
There is little I can do anymore seen from the perspective of XPrivacy.
Anything that could be improved is improved now.
As soon as Dev-Host allows me, I will upload version 1.7.23, which runs the privacy provider in a separate process.
This version will be available as beta version in the Play store in a few hours.

[M66B]

Version 1.7.23: http://d-h.st/f0D

[cobaltdr]

Set it to heapsize to 128m to decrease the frequency at which the Touchwiz UI was being killed off by the system.

Have adopted your values, but still crashes. I wonder why I'm the only one getting this.

[M66B]

@cobaltdr maybe you should try another ROM

[cobaltdr]

@M66B: thanks, I appreciate what you say. I'm not entirely sure how to modify my setup. I only have about 5-6 processes taking up memory other than stock Samsung bloat. Hehe. Killing stuff to free memory doesn't appear to help. I might try to split the XML file to see if that does anything.

[cobaltdr]

@M66B yeah, I've thought of that. I've been putting it off. It's a real shame Samsung have bloat. With their backing they could have produced more efficient solutions.

[cobaltdr]

Dammit. 1.7.23 bootloops again when importing. I will try to get logcats to post, but I need some sleep, been working all night.

[M66B]

With a bit of luck this version will solve the out of memory problem:
http://forum.xda-developers.com/showpost.php?p=44078895&postcount=1956

[cobaltdr]

Took about 5 minutes, but 1.7.27 is the first version that has allowed me to import settings without problems.

[pogovault]

I also managed to import settings with 1.7.27 after long, long time. First attempt was disaster - need reboot and restore XPrivacy from backup. For second attempt I kill every app I could and deleted all XPrivacy data before import. So, it is better now, but I'm now afraid to install any new app or change any restrictions.
Another thing: after importing, all accounts settings ("Select accounts to allow") are gone. I was under impression that export/import also preserve that info. Is this a bug or feature?

[M66B]

Accounts and contacts were not exported until so far.
The just released version export accounts and contacts, but only imports them when it is the same device.

[cobaltdr]

People need to be aware that Xprivacy is exporting account and contact details to a text file. Is the data encrypted? They might otherwise inadvertently be distributing private data themselves.

[M66B]

You should know by now that I pay attention to every litlle detail ... ;-)