does not index content #1303

sarce666 · 2021-11-09T15:49:19Z

sarce666
Nov 9, 2021

hello

i trying to index data to elasticloud and worksoace
here is the conf :

name: "feeds"
fs:
  url: "/feeds"
  update_rate: "15m"
  json_support: false
  filename_as_id: false
  add_filesize: true
  remove_deleted: true
  add_as_inner_object: false
  store_source: false
  index_content: true
  attributes_support: false
  raw_metadata: true
  xml_support: false
  index_folders: true
  lang_detect: false
  continue_on_error: true
  ocr:
    language: "eng"
    enabled: true
    pdf_strategy: "ocr_and_text"
  follow_symlinks: false
elasticsearch:
  username: ""
  password: ""
  nodes:
  - cloud_id: 
workplace_search:
  access_token: ""
  key: ""
  server: ""
  bulk_size: 1000
  flush_interval: "5s"
  byte_size: "10mb"

it works fine but in elasticsearch i dont have the content ( which i have into workspace )

when launch fscrwaler it found the pdf to index
15:39:35,340 DEBUG [f.p.e.c.f.FsParserAbstract] [/florent/2021-09-28--Invoice_202110.pdf] can be indexed: [true] 15:39:35,340 DEBUG [f.p.e.c.f.FsParserAbstract] - file: /florent/2021-09-28--Invoice_202110.pdf

but it index it with those fields : Available fields
9
_id
_index
score
_type
file.content_type
file.filename
path.real
path.root
path.virtual

and i have not content

t

but in workplace it is indexed with content

thanks for your help and bravo for fscrawler

dadoonet · 2021-11-09T17:27:45Z

dadoonet
Nov 9, 2021
Maintainer

Hey 👋🏼

Could you edit your post and format code/settings/logs with <> icon or markdown?

1 reply

sarce666 Nov 10, 2021
Author

sorry

i just did it thanks

dadoonet · 2021-11-09T17:27:47Z

dadoonet
Nov 9, 2021
Maintainer

Hey 👋🏼

Could you edit your post and format code/settings/logs with <> icon or markdown?

0 replies

sarce666 · 2021-11-10T09:14:09Z

sarce666
Nov 10, 2021
Author

i might have found the issue
in my conf file i try to index it with elastic cloud and workspace
it looks like if it put both it only index it in workspace, if i remove workspace it index perfectly the elastic
then should i create 2 different files ?

1 reply

dadoonet Nov 10, 2021
Maintainer

No. One single file.

If you want to index in workplace search, you need to set the elasticsearch part and the workplace search part.

I'll post in a moment what I'm exactly using right now for this. Stay tuned.

dadoonet · 2021-11-10T10:55:26Z

dadoonet
Nov 10, 2021
Maintainer

Here is the typical settings you could use for Workplace Search on elastic cloud:

---
name: "test"
fs:
  url: "/path/to/dir"
elasticsearch:
  username: "elastic"
  password: "PASSWORD"
  nodes:
  - cloud_id: "CLOUD_ID"
workplace_search:
  server: "WORKPLACE_SEARCH_URL"

1 reply

dadoonet Nov 10, 2021
Maintainer

So try something like:

name: "feeds"
fs:
  url: "/feeds"
elasticsearch:
  username: "elastic"
  password: "PASSWORD"
  nodes:
  - cloud_id:  "CLOUD_ID"
workplace_search:
  server: "WORKPLACE_SEARCH_URL"

sarce666 · 2021-11-10T12:55:40Z

sarce666
Nov 10, 2021
Author

ok
well this exactly what i have in my conf
when i do sudo bin/fscrawler analysis-feeds --trace --restart
it does not seems to take in effect my change in the conf , do i have to remove the _default folder

8 replies

sarce666 Nov 10, 2021
Author

ok so i deleted the index
i delete _default , i restart with sudo bin/fscrawler analysis-feeds --trace
i see a lot of file indexed

13:15:38,362 DEBUG [f.p.e.c.f.f.FsCrawlerUtil] computeVirtualPathName(/home/data/feeds, /home/data/feeds/Intel471/CVE Weaponization/20210107-Intel471-CVEWeaponizationReport.pdf) = /Intel471/CVE Weaponization/20210107-Intel471-CVEWeaponizationReport.pdf
13:15:38,377 DEBUG [f.p.e.c.f.s.FsCrawlerDocumentServiceElasticsearchImpl] Indexing analysis-feeds/5266296060e16dfefc47c4e3d89b666?pipeline=null
13:15:38,392 DEBUG [f.p.e.c.f.f.FsCrawlerUtil] computeVirtualPathName(/home/data/feeds, /home/data/feeds/Intel471/CVE Weaponization/20210205-Intel471-CVEWeaponizationReport.pdf) = /Intel471/CVE Weaponization/20210205-Intel471-CVEWeaponizationReport.pdf
13:15:38,394 DEBUG [f.p.e.c.f.f.FsCrawlerUtil] directory = [false], filename = [/Intel471/CVE Weaponization/20210205-Intel471-CVEWeaponizationReport.pdf], includes = [null], excludes = [[*/~*]]
13:15:38,398 DEBUG [f.p.e.c.f.f.FsCrawlerUtil] filename = [/Intel471/CVE Weaponization/20210205-Intel471-CVEWeaponizationReport.pdf], excludes = [[*/~*]]
13:15:38,402 DEBUG [f.p.e.c.f.f.FsCrawlerUtil] filename = [/Intel471/CVE Weaponization/20210205-Intel471-CVEWeaponizationReport.pdf], includes = [null]
13:15:38,402 DEBUG [f.p.e.c.f.FsParserAbstract] [/Intel471/CVE Weaponization/20210205-Intel471-CVEWeaponizationReport.pdf] can be indexed: [true]
13:15:38,403 DEBUG [f.p.e.c.f.FsParserAbstract]   - file: /Intel471/CVE Weaponization/20210205-Intel471-CVEWeaponizationReport.pdf
13:15:38,404 DEBUG [f.p.e.c.f.FsParserAbstract] fetching content from [/home/data/feeds/Intel471/CVE Weaponization],[20210205-Intel471-CVEWeaponizationReport.pdf]
13:15:38,405 DEBUG [f.p.e.c.f.f.FsCrawlerUtil] computeVirtualPathName(/home/data/feeds, /home/data/feeds/Intel471/CVE Weaponization/20210205-Intel471-CVEWeaponizationReport.pdf) = /Intel471/CVE Weaponization/20210205-Intel471-CVEWeaponizationReport.pdf
13:17:14,268 DEBUG [f.p.e.c.f.f.FsCrawlerUtil] computeVirtualPathName(/home/data/feeds, /home/data/feeds/Intel471/CVE Weaponization/20210205-Intel471-CVEWeaponizationReport.pdf) = /Intel471/CVE Weaponization/20210205-Intel471-CVEWeaponizationReport.pdf
13:17:14,272 DEBUG [f.p.e.c.f.s.FsCrawlerDocumentServiceElasticsearchImpl] Indexing analysis-feeds/6073538b73ac44122b1fbf4ff86e7140?pipeline=null
13:17:14,278 DEBUG [f.p.e.c.f.f.FsCrawlerUtil] computeVirtualPathName(/home/data/feeds, /home/data/feeds/Intel471/CVE Weaponization/20210401-Intel471-CVEWeaponizationReport.pdf) = /Intel471/CVE Weaponization/20210401-Intel471-CVEWeaponizationReport.pdf
13:17:14,279 DEBUG [f.p.e.c.f.f.FsCrawlerUtil] directory = [false], filename = [/Intel471/CVE Weaponization/20210401-Intel471-CVEWeaponizationReport.pdf], includes = [null], excludes = [[*/~*]]
13:17:14,279 DEBUG [f.p.e.c.f.f.FsCrawlerUtil] filename = [/Intel471/CVE Weaponization/20210401-Intel471-CVEWeaponizationReport.pdf], excludes = [[*/~*]]
13:17:14,280 DEBUG [f.p.e.c.f.f.FsCrawlerUtil] filename = [/Intel471/CVE Weaponization/20210401-Intel471-CVEWeaponizationReport.pdf], includes = [null]
13:17:14,280 DEBUG [f.p.e.c.f.FsParserAbstract] [/Intel471/CVE Weaponization/20210401-Intel471-CVEWeaponizationReport.pdf] can be indexed: [true]
13:17:14,281 DEBUG [f.p.e.c.f.FsParserAbstract]   - file: /Intel471/CVE Weaponization/20210401-Intel471-CVEWeaponizationReport.pdf
13:17:14,309 DEBUG [f.p.e.c.f.FsParserAbstract] fetching content from [/home/data/feeds/Intel471/CVE Weaponization],[20210401-Intel471-CVEWeaponizationReport.pdf]
13:17:14,310 DEBUG [f.p.e.c.f.f.FsCrawlerUtil] computeVirtualPathName(/home/data/feeds, /home/data/feeds/Intel471/CVE Weaponization/20210401-Intel471-CVEWeaponizationReport.pdf) = /Intel471/CVE Weaponization/20210401-Intel471-CVEWeaponizationReport.pdf
13:17:14,487 WARN  [o.a.p.p.f.PDTrueTypeFont] Using fallback font 'LiberationSans' for 'TimesNewRomanPSMT'
13:17:26,226 WARN  [o.a.p.p.f.PDTrueTypeFont] Using fallback font 'LiberationSans' for 'ArialMT'
13:19:19,927 DEBUG [f.p.e.c.f.f.FsCrawlerUtil] computeVirtualPathName(/home/data/feeds, /home/data/feeds/Intel471/CVE Weaponization/20210401-Intel471-CVEWeaponizationReport.pdf) = /Intel471/CVE Weaponization/20210401-Intel471-CVEWeaponizationReport.pdf
13:19:19,931 DEBUG [f.p.e.c.f.s.FsCrawlerDocumentServiceElasticsearchImpl] Indexing analysis-feeds/d57babe8a897daf8d39359f31862c9?pipeline=null
13:19:19,936 DEBUG [f.p.e.c.f.f.FsCrawlerUtil] computeVirtualPathName(/home/data/feeds, /home/data/feeds/Intel471/CVE Weaponization/20210504-Intel471-CVEWeaponizationReport.pdf) = /Intel471/CVE Weaponization/20210504-Intel471-CVEWeaponizationReport.pdf
13:19:19,939 DEBUG [f.p.e.c.f.f.FsCrawlerUtil] directory = [false], filename = [/Intel471/CVE Weaponization/20210504-Intel471-CVEWeaponizationReport.pdf], includes = [null], excludes = [[*/~*]]
13:19:19,940 DEBUG [f.p.e.c.f.f.FsCrawlerUtil] filename = [/Intel471/CVE Weaponization/20210504-Intel471-CVEWeaponizationReport.pdf], excludes = [[*/~*]]
13:19:19,941 DEBUG [f.p.e.c.f.f.FsCrawlerUtil] filename = [/Intel471/CVE Weaponization/20210504-Intel471-CVEWeaponizationReport.pdf], includes = [null]
13:19:19,941 DEBUG [f.p.e.c.f.FsParserAbstract] [/Intel471/CVE Weaponization/20210504-Intel471-CVEWeaponizationReport.pdf] can be indexed: [true]
13:19:19,941 DEBUG [f.p.e.c.f.FsParserAbstract]   - file: /Intel471/CVE Weaponization/20210504-Intel471-CVEWeaponizationReport.pdf
13:19:19,966 DEBUG [f.p.e.c.f.FsParserAbstract] fetching content from [/home/data/feeds/Intel471/CVE Weaponization],[20210504-Intel471-CVEWeaponizationReport.pdf]
13:19:19,967 DEBUG [f.p.e.c.f.f.FsCrawlerUtil] computeVirtualPathName(/home/data/feeds, /home/data/feeds/Intel471/CVE Weaponization/20210504-Intel471-CVEWeaponizationReport.pdf) = /Intel471/CVE Weaponization/20210504-Intel471-CVEWeaponizationReport.pdf
13:21:56,124 DEBUG [f.p.e.c.f.f.FsCrawlerUtil] computeVirtualPathName(/home/data/feeds, /home/data/feeds/Intel471/CVE Weaponization/20210504-Intel471-CVEWeaponizationReport.pdf) = /Intel471/CVE Weaponization/20210504-Intel471-CVEWeaponizationReport.pdf
13:21:56,127 DEBUG [f.p.e.c.f.s.FsCrawlerDocumentServiceElasticsearchImpl] Indexing analysis-feeds/6e8846c08b3d2a3b7d71174f8d442267?pipeline=null
13:21:56,130 DEBUG [f.p.e.c.f.f.FsCrawlerUtil] computeVirtualPathName(/home/data/feeds, /home/data/feeds/Intel471/CVE Weaponization/20210601-Intel471-CVEWeaponizationReport.pdf) = /Intel471/CVE Weaponization/20210601-Intel471-CVEWeaponizationReport.pdf
13:21:56,130 DEBUG [f.p.e.c.f.f.FsCrawlerUtil] directory = [false], filename = [/Intel471/CVE Weaponization/20210601-Intel471-CVEWeaponizationReport.pdf], includes = [null], excludes = [[*/~*]]
13:21:56,130 DEBUG [f.p.e.c.f.f.FsCrawlerUtil] filename = [/Intel471/CVE Weaponization/20210601-Intel471-CVEWeaponizationReport.pdf], excludes = [[*/~*]]
13:21:56,131 DEBUG [f.p.e.c.f.f.FsCrawlerUtil] filename = [/Intel471/CVE Weaponization/20210601-Intel471-CVEWeaponizationReport.pdf], includes = [null]
13:21:56,131 DEBUG [f.p.e.c.f.FsParserAbstract] [/Intel471/CVE Weaponization/20210601-Intel471-CVEWeaponizationReport.pdf] can be indexed: [true]
13:21:56,131 DEBUG [f.p.e.c.f.FsParserAbstract]   - file: /Intel471/CVE Weaponization/20210601-Intel471-CVEWeaponizationReport.pdf
13:21:56,178 DEBUG [f.p.e.c.f.FsParserAbstract] fetching content from [/home/data/feeds/Intel471/CVE Weaponization],[20210601-Intel471-CVEWeaponizationReport.pdf]
13:21:56,179 DEBUG [f.p.e.c.f.f.FsCrawlerUtil] computeVirtualPathName(/home/data/feeds, /home/data/feeds/Intel471/CVE Weaponization/20210601-Intel471-CVEWeaponizationReport.pdf) = /Intel471/CVE Weaponization/20210601-Intel471-CVEWeaponizationReport.pdf

but still got index with 4 count

dadoonet Nov 10, 2021
Maintainer

What is the output of:

GET /_cat/indices/feeds*?v

sarce666 Nov 10, 2021
Author

sorry
not sure to understand ?
where should i type this , i work on elastci cloud

dadoonet Nov 10, 2021
Maintainer

In the Kibana dev console

sarce666 Nov 10, 2021
Author

i got this health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
but
actually it works , i sorry , my index increase , it just take a lof of time because i use OCR + tesseract

i think i know why it does not work workplace and elastci in the same file , whern i launch workspace it stop the process one it index
so it does not take the time to index in elastic. ?

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

does not index content #1303

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 5 comments 11 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

does not index content #1303

sarce666 Nov 9, 2021

Replies: 5 comments · 11 replies

dadoonet Nov 9, 2021 Maintainer

sarce666 Nov 10, 2021 Author

dadoonet Nov 9, 2021 Maintainer

sarce666 Nov 10, 2021 Author

dadoonet Nov 10, 2021 Maintainer

dadoonet Nov 10, 2021 Maintainer

dadoonet Nov 10, 2021 Maintainer

sarce666 Nov 10, 2021 Author

sarce666 Nov 10, 2021 Author

dadoonet Nov 10, 2021 Maintainer

sarce666 Nov 10, 2021 Author

dadoonet Nov 10, 2021 Maintainer

sarce666 Nov 10, 2021 Author

sarce666
Nov 9, 2021

Replies: 5 comments 11 replies

dadoonet
Nov 9, 2021
Maintainer

sarce666 Nov 10, 2021
Author

dadoonet
Nov 9, 2021
Maintainer

sarce666
Nov 10, 2021
Author

dadoonet Nov 10, 2021
Maintainer

dadoonet
Nov 10, 2021
Maintainer

dadoonet Nov 10, 2021
Maintainer

sarce666
Nov 10, 2021
Author

sarce666 Nov 10, 2021
Author

dadoonet Nov 10, 2021
Maintainer

sarce666 Nov 10, 2021
Author

dadoonet Nov 10, 2021
Maintainer

sarce666 Nov 10, 2021
Author