Make NumFDs not allocate excessively, implement throttling, and disable Prometheus process collector

[richardartoul]

What this PR does / why we need it:
Recent changes to M3DB mean that it uses many more F.Ds than before. This can result in counting the number of F.Ds allocating excessively and wasting a ton of CPU resources.

This P.R prevents counting the F.Ds from allocating excessively and implements self-throttling so that even though counting F.Ds takes longer, it will not cause spikes in CPU usage.

• Make NumFDs() not allocate excessively
• Make NumFDs() throttle itself
• Turn off FD counting in the Prometheus reporter

Also verified with start_m3 that metrics continue to be emitted

[codecov]

Codecov Report

❗ No coverage uploaded for pull request base (master@f0505d6). Click here to learn what that means.
The diff coverage is 58.8%.

@@           Coverage Diff            @@
##             master   #1633   +/-   ##
========================================
  Coverage          ?   72.1%           
========================================
  Files             ?     958           
  Lines             ?   79587           
  Branches          ?       0           
========================================
  Hits              ?   57405           
  Misses            ?   18445           
  Partials          ?    3737

Flag	Coverage Δ
#aggregator	69.3% <ø> (?)
#cluster	67.6% <ø> (?)
#collector	48% <ø> (?)
#dbnode	70.4% <ø> (?)
#m3em	67.9% <ø> (?)
#m3ninx	71% <ø> (?)
#m3nsch	51.1% <ø> (?)
#metrics	100% <ø> (?)
#msg	74.3% <ø> (?)
#query	49.7% <ø> (?)
#x	82.5% <58.8%> (?)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f0505d6...5d1ae0f. Read the comment docs.

[Haijuncao]

@richardartoul

It seems that we report num file descriptors during instrumentation and emit it from dbnode as application level stats.

I am wondering if we need to emit FD count/stat from application level, FD stat per process can be queried from os.

[richardartoul]

@Haijuncao Yeah its really helpful since the instrument library gets loaded into a lot of our services. I.E we don't need any other software running on the host to get this information which can be really important for debugging internally and in O.S.S

It also makes it really easy to know which process it is.

[robskillington]

Don't you also need NumFDsReference (or whatever that gets renamed to) from the non-linux source file? Otherwise it won't be able to build on non-linux since it will be missing.

[richardartoul]

ended up not exporting it

[robskillington]

LGTM