.fdt file still exist when disable _source

[lexlee327]

Elasticsearch version (bin/elasticsearch --version):
7.11

Plugins installed: []
no

JVM version (java -version):
jdk 12

OS version (uname -a if on a Unix-like system):
mac

Description of the problem including expected versus actual behavior:
When I disable the _source, It is expected lower storage usage, but it didn't.
After set _source: { enabled: false} , Elasticsearch doesn't return _source in get/search request, but it still keep the .fdt file in segment
Elasticsearch are wildly use in OLAP case, when we just need the analysis report without show the documents, we could get rid of the stored field

Steps to reproduce:
1, add sample data:

2, reindex it with _source disable

3, new index return without _source

4, new index with the same size

5. have the .fdt file
   

[markharwood]

When I disable the _source, It is expected lower storage usage, but it didn't.

Thanks for the report. One thing that looks to be missing is the file size comparison between the source enabled and source disabled indices. Could you provide that? Otherwise all you may be showing is that there is some residual use of stored values (possibly not related to source).

[DaveCTurner]

I think we need to know exactly what version this is about too, the OP says 7.x but this is not at all helpful.

In some versions we temporarily retain the source for replica recovery (and CCR) even if _source is disabled. It gets removed by a later merge once it's no longer needed.

[ywelsch]

Relates #41628

[elasticmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[lexlee327]

When I disable the _source, It is expected lower storage usage, but it didn't.

Thanks for the report. One thing that looks to be missing is the file size comparison between the source enabled and source disabled indices. Could you provide that? Otherwise all you may be showing is that there is some residual use of stored values (possibly not related to source).

I reindex the sample data with the name source_enable and source_disable, you could see that, the source_disable one even bigger.
I think the key issue is that, it still generate the .fdt file in segment

[lexlee327]

I think we need to know exactly what version this is about too, the OP says 7.x but this is not at all helpful.

In some versions we temporarily retain the source for replica recovery (and CCR) even if _source is disabled. It gets removed by a later merge once it's no longer needed.

I tested it in 7.9.0, 7.10.0, 7.11.0

[lexlee327]

just like @DaveCTurner said, I have to disable soft_delete

    "settings": {
      "number_of_replicas" : "0",
      "soft_deletes": {
        "enabled": false
      }
    }

then, the .fdt file is the proper size:

[DaveCTurner]

just like @DaveCTurner said, I have to disable soft_delete

I said nothing about disabling soft deletes, and I certainly don't recommend that either.

What I did say was that the source is only kept while it's needed for recovery or CCR, and it's removed by a later merge. It's therefore the expected behaviour for there to be stored fields in some segments: there's only a bug here if we are retaining stored fields in merges even though there's no reason to retain it, and so far we haven't seen any evidence of that.

[DaveCTurner]

No further response after a few weeks so I'm closing this. If you can find and share evidence of a bug (see my previous message) then we can reopen this.