Translation unit cache is never cleared

[cameronwhite]

When indexing a project of mine, the memory usage of clang-tags-server grows to over 4GB after parsing about 100 files - the main cause of the memory usage seems to be the caching of translation units.

The simplest fix would be to just disable the caching of translation units while indexing, but I think a better fix would be to limit the cache to a maximum number of translation units and/or a maximum memory size so that this problem doesn't occur for clang-tags complete as well.

What do you think?

[ffevotte]

Yes, this is a major issue. I also encountered it a few weeks ago, when I started using clang-tags on a big project at work (around 200 translation units, ~1000 files in total).

Until now, I just used the following quick and dirty patch to limit cache size to only one translation unit.

diff --git a/clangTags/cache.hxx b/clangTags/cache.hxx
index 08e37a7..9b1741f 100644
--- a/clangTags/cache.hxx
+++ b/clangTags/cache.hxx
@@ -20,6 +20,7 @@ public:
     auto it = tu_.find (fileName);
     if (it == tu_.end()) {
       LibClang::TranslationUnit tu = index_.parse (clArgs);
+      tu_.erase (tu_.begin(), tu_.end());
       return tu_.insert (std::make_pair (fileName, tu)).first->second;
     } else {
       it->second.reparse();

But like you, I think a better solution would be to limit the cache size to a maximum size, discarding oldest TUs. While I more or less see how to implement a limit to the number of TUs, I have absolutely no idea how to limit the memory size (or even how to measure the memory consumption of a given TU)

[ffevotte]

By the way, which platform do you use clang-tags on ?

I've been working lately on a feature to have clang-tags-server reindex the project in a background thread, using Linux' inotify system to detect changes in the filesystem. But I'm afraid publishing it now since it is linux-specific.

If you are using Mac OS X or a non-linux kernel, I think I should consider using a higher-level file alteration monitoring system or try to provide another implementation using kqueue or fsevents...

[cameronwhite]

I think it might be possible to get the memory usage of a translation unit (http://clang.llvm.org/doxygen/group__CINDEX__TRANSLATION__UNIT.html), so I'll see what I can do with that.

I'm using Linux, but I'd imagine that other people would probably find OS X support useful.

[ffevotte]

Very nice! Thanks.

I'm currently in the process of cleaning the inotify branch. I've made the dependency on inotify optional, so that Linux users can have the index automatically updated, but others can still manually request an update like before.

Although the inotify branch introduces lots of structural changes, it almost leaves LibClang++ and the cache untouched, so I don't think we'll run into too many integration problems.