How to Age-Out (Purge Old) Data?

[nickwallen]

Is there a mechanism to purge or age-out old data with OpenTAXII? How do I ensure that the data store doesn't grow without bounds?

[traut]

@nickwallen default SQL backend does not support any clean-up procedures at the moment.

What would be your requirements here?

[nickwallen]

I am looking at integrating OpenTAXII as a source of threat intel data for the default deployment of Apache Metron. In most cases, I envision Metron users leveraging their own existing aggregator platform that would be used instead. Although if none exists, I am hoping that OpenTAXII can fill that role.

I have some logic that will likely be added to Metron that deploys OpenTAXII and can sync with some external 3rd party feeds. If I setup a mechanism that continually syncs external threat intel feeds, I need to ensure that the data store size doesn't grow without bound. That's what I need.

[code040]

Are you planning of running this using the SQLite database? Then all you can do is implement a cleanup routine inside the persistence layer (and possible in the manager), which runs periodically any cleanup task. As @traut mentioned, this is currently not the case.

[traut]

@nickwallen another option would be to implement another OpenTAXII CLI command to clean-up DB. Something as simple as "delete content blocks older than X in collections A and B". Host admin can use this inside scheduled cron jobs // @code040

[nickwallen]

Yes, I like that idea. Ideally, it would work the same no matter what persistence API (sqlite, mysql, etc) is used.