Dump prof files from AdvancedProfiler

[clumsy]

Description & Motivation

Currently AdvancedProfiler only outputs textual summary where the rows are sorted by cumulative time. Unfortunately this does not give a full picture about the structure and duration of call stacks involved.

Pitch

Raw cProfiler dumps provide more information to help identify bottlenecks than just module names and line numbers. E.g. using https://jiffyclub.github.io/snakeviz one can see the hierarchy of the call stacks and sort/drill-down using an interactive interface.

Luckily there's not much needed to make this available. AdvancedProfiler already collects profiling data we can easily dump using dump_stats():

    @override
    def summary(self) -> str:
        recorded_stats = {}
        for action_name, pr in self.profiled_actions.items():
		 	####### dumping prof file ########
            if self.filename and self.dirpath:
                import os
                import tempfile
                from fsspec.core import url_to_fs
                from fsspec.implementations.local import AbstractFileSystem

                filepath = os.path.join(self.dirpath, self._prepare_filename())
                def get_filesystem(path, **kwargs) -> AbstractFileSystem:
                    fs, _ = url_to_fs(str(path), **kwargs)
                    return fs
                with tempfile.TemporaryDirectory(prefix="test", suffix="test", dir=os.getcwd()) as tmp_dir:
                    # alas dump_stats() expects a file path it can open()
                    tmp_file = os.path.join(tmp_dir, "tmp.prof")
                    pr.dump_stats(tmp_file)
                    fs = get_filesystem(filepath)
                    fs.mkdirs(self.dirpath, exist_ok=True)
                    fs.copy(tmp_file, filepath + ".prof")
			###############################
            s = io.StringIO()
            ps = pstats.Stats(pr, stream=s).strip_dirs().sort_stats("cumulative")
            ps.print_stats(self.line_count_restriction)
            recorded_stats[action_name] = s.getvalue()
        return self._stats_to_str(recorded_stats)

I'm willing to contribute this but would like to clarify the best configuration and injection options. Thanks!

Alternatives

Write a custom callback and do the profiling from there - thus not using the one provided by PyTorch Lightning.

Additional context

Related to #7424

cc @Borda @carmocca

[clumsy]

I'm happy to address any feedback, @Borda @carmocca @awaelchli
Thanks!