Adoption

[rshkv]

Palantir is seeing great perf improvements in benchmarks and we’re considering adopting Velox (with Gluten). We’re trying to gauge the risk from behavior changes and how conservative we should be as result.

I was wondering if you could share who besides Meta adopted Velox at scale.

[kgpai]

cc: @pedroerp @aditi-pandit @majetideepak

[aditi-pandit]

@rshkv : Are you using Presto, Spark or some other engine with Velox via Gluten ?

Curious about this since you are asking about behavior change. As Velox is used by query engines, the behavior change is usually wrt the engine being modified.

[rshkv]

We're going to use Spark -> Gluten -> Velox.

So this risk for us are inadvertent behavior changes between Spark and Velox (i.e., Velox's functions for Spark).

[pedroerp]

@rshkv great to hear Palantir is interested in Velox (via Gluten).

From a behavior change perspective, the vision is making Velox to be a drop-in replacement for Java/Scala based execution (in both Spark and Presto, at least), meaning that using Velox should be completely transparent to users. Velox provides an exact bug-by-bug implementation of Presto functions and types, for example, and the same is being implemented for Spark.

I was wondering if you could share who besides Meta adopted Velox at scale.

Other than Meta, at least IBM/Ahana and ByteDance (through Presto/Prestissimo) have announced using it internally; Voltron Data is using through their recently announced Theseus engine. Several companies have been actively engaged and helping develop Gluten (Pinterest, Alibaba, Tencent, Microsoft, and others, led by Intel), but I'm not aware of any production usage as of today.

[rshkv]

Thank you for the answer, Pedro. I get the vision. We have a huge Spark install base that's barely getting faster and we won't be able to rewrite. So something faster, with bug-for-bug parity, is very exciting.

We're cautious in our roll-out (so far we worry more about perf regressions and adapting memory settings, less Velox correctness). But that is already more adoption than I presumed. I hope that list will grow and Palantir be a part of it.