FTS reinstallation is broken #2326

enoch85 · 2022-05-16T19:43:05Z

Steps To Reproduce

Install FTS
Reinstall it

Expected Result

Everything should ve removed and it should reinstall 100% fresh

Actual Result

Reinstalling Full Text Search...
TRUNCATE TABLE
reset.
fulltextsearch disabled
fulltextsearch 24.0.0 removed
fulltextsearch_elasticsearch disabled
fulltextsearch_elasticsearch 24.0.0 removed
files_fulltextsearch disabled
files_fulltextsearch 24.0.0 removed
Press any key to continue. Press CTRL+C to abort
084f2f53c9af
084f2f53c9af
Total reclaimed space: 0B
Press any key to continue. Press CTRL+C to abort
Error response from daemon: remove fts_os-data: volume is in use - [3109e58ee9c15715911d1410b8ed8affeba522c7ac5905d66e185f0931644cba]
Reinstalling Full Text Search...
13
RAM for FullTextSearch OK! (3.83 GB)
CPU for FullTextSearch OK! (2)
Installing fulltextsearch...
vm.max_map_count=512000
Max map count already set, skipping...
Using default tag: latest
latest: Pulling from opensearchproject/opensearch
Digest: sha256:5acd42d84d0dd4910c8a98ff097b0e473525f5489db6d6a363e408c557083107
Status: Image is up to date for opensearchproject/opensearch:latest
docker.io/opensearchproject/opensearch:latest
/var/scripts/fulltextsearch.sh: line 106: /opt/opensearch/opensearch.yml: Is a directory
/var/scripts/fulltextsearch.sh: line 143: /opt/opensearch/internal_users.yml: Is a directory
/var/scripts/fulltextsearch.sh: line 157: /opt/opensearch/roles_mapping.yml: Is a directory
Can't open "root-ca.pem" for writing, Is a directory
403779DB3C7F0000:error:80000015:system library:BIO_new_file:Is a directory:../crypto/bio/bss_file.c:67:calling fopen(root-ca.pem, w)
403779DB3C7F0000:error:10080002:BIO routines:BIO_new_file:system lib:../crypto/bio/bss_file.c:77:
pkcs8: Can't open "admin-key.pem" for writing, Is a directory
Could not read private key from admin-key.pem
Can't open "admin.csr" for reading, No such file or directory
40B74CD6007F0000:error:80000002:system library:BIO_new_file:No such file or directory:../crypto/bio/bss_file.c:67:calling fopen(admin.csr, r)
40B74CD6007F0000:error:10000080:BIO routines:BIO_new_file:no such file:../crypto/bio/bss_file.c:75:
Unable to load certificate request input
pkcs8: Can't open "node-key.pem" for writing, Is a directory
Could not read private key from node-key.pem
Can't open "node.csr" for reading, No such file or directory
40978DF8057F0000:error:80000002:system library:BIO_new_file:No such file or directory:../crypto/bio/bss_file.c:67:calling fopen(node.csr, r)
40978DF8057F0000:error:10000080:BIO routines:BIO_new_file:no such file:../crypto/bio/bss_file.c:75:
Unable to load certificate request input
Certificate request self-signature ok
subject=C = CA, ST = NEXTCLOUD, L = VM, O = OPENSEARCH, OU = FTS, CN = CLIENT
Could not read CA certificate from root-ca.pem
Unable to load CA certificate
rm: cannot remove 'admin.csr': No such file or directory
rm: cannot remove 'node.csr': No such file or directory
mv: cannot stat 'client.pem': No such file or directory
Starting fts_os-node ... error

ERROR: for fts_os-node  Cannot start service fts_os-node: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/opt/opensearch/node-key.pem" to rootfs at "/usr/share/opensearch/config/node-key.pem": mount /opt/opensearch/node-key.pem:/usr/share/opensearch/config/node-key.pem (via /proc/self/fd/6), flags: 0x5000: not a directory: unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type

ERROR: for fts_os-node  Cannot start service fts_os-node: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/opt/opensearch/node-key.pem" to rootfs at "/usr/share/opensearch/config/node-key.pem": mount /opt/opensearch/node-key.pem:/usr/share/opensearch/config/node-key.pem (via /proc/self/fd/6), flags: 0x5000: not a directory: unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type
ERROR: Encountered errors while bringing up the project.

Screenshots, Videos, or Pastebins

@Ark74

I also wonder why we are using a demo installer?

Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
Clustername: fts_os-cluster
Clusterstate: GREEN
Number of nodes: 1
Number of data nodes: 1
.opendistro_security index already exists, so we do not need to create one.
Populate config from /usr/share/opensearch/plugins/opensearch-security/tools
Force type: internalusers
Will update '_doc/internalusers' with ../securityconfig/internal_users.yml 
   SUCC: Configuration for 'internalusers' created or updated
Done with success
-> Installing ingest-attachment
-> Downloading ingest-attachment from opensearch
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@     WARNING: plugin requires additional permissions     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
* java.lang.RuntimePermission accessClassInPackage.sun.java2d.cmm.kcms
* java.lang.RuntimePermission accessDeclaredMembers
* java.lang.RuntimePermission getClassLoader
* java.lang.reflect.ReflectPermission suppressAccessChecks
* java.security.SecurityPermission createAccessControlContext
* java.security.SecurityPermission insertProvider
* java.security.SecurityPermission putProviderProperty.BC
See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
for descriptions of what these permissions allow and the associated risks.
-> Installed ingest-attachment with folder name ingest-attachment

Enabling execution of install_demo_configuration.sh for OpenSearch Security Plugin
OpenSearch Security Demo Installer
** Warning: Do not use on production or public reachable systems **
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Basedir: /usr/share/opensearch

Additional Context

No response

Build Version

latest (24.0.0)

Environment

By using the scripts

Environment Details

No response

The text was updated successfully, but these errors were encountered:

enoch85 · 2022-05-16T19:43:42Z

@Ark74 Please have a look at this, I have no clue (or time). :)

Ark74 · 2022-05-17T02:37:08Z

Hello! Sorry I was out the whole day, I'll be taking a look shortly. Cheers! El 16/05/22 a las 14:43, Daniel Hansson escribió:

…

@Ark74 <https://github.com/Ark74> Please have a look at this, I have no clue (or time). :) — Reply to this email directly, view it on GitHub <#2326 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABYZQJSHDYPSCEJ2OJNSW7LVKKQPVANCNFSM5WCRXGYQ>. You are receiving this because you were mentioned.Message ID: ***@***.***>

Ark74 · 2022-05-17T16:14:39Z

I guess the first problem is here, this volume is not been stopped and deleted correctly, as it is still running.

Error response from daemon: remove fts_os-data: volume is in use - [3109e58ee9c15715911d1410b8ed8affeba522c7ac5905d66e185f0931644cba]

Ark74 · 2022-05-17T23:38:31Z

@enoch85 fixed!
My bad a one-time-use container was keeping behind on exit status.
Also added a proper way to stop docker-compose items.
Cheers!

Ark74 · 2022-05-17T23:47:55Z

Enabling execution of install_demo_configuration.sh for OpenSearch Security Plugin
OpenSearch Security Demo Installer
** Warning: Do not use on production or public reachable systems **
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

This is just when starting the default container we rewrite it, that's the reason to use docker-compose instead of docker images as the resulting container is a custom one, all security gets rewritten and changes applied here:

vm/apps/fulltextsearch.sh

Lines 240 to 251 in e3721d4

    
           # Make sure password setup is enforced. 
        
           docker-compose exec fts_os-node \ 
        
               bash -c "cd \ 
        
                        plugins/opensearch-security/tools/ && \ 
        
                        bash securityadmin.sh -f \ 
        
                        ../securityconfig/internal_users.yml \ 
        
                        -t internalusers \ 
        
                        -icl \ 
        
                        -nhnv \ 
        
                        -cacert ../../../config/root-ca.pem \ 
        
                        -cert ../../../config/admin.pem \ 
        
                        -key ../../../config/admin-key.pem"

# Make sure password setup is enforced.
docker-compose exec fts_os-node \
    bash -c "cd \
             plugins/opensearch-security/tools/ && \
             bash securityadmin.sh -f \
             ../securityconfig/internal_users.yml \
             -t internalusers \
             -icl \
             -nhnv \
             -cacert ../../../config/root-ca.pem \
             -cert ../../../config/admin.pem \
             -key ../../../config/admin-key.pem"

and here:

vm/apps/fulltextsearch.sh

Lines 198 to 205 in e3721d4

    
                 - $OPNSDIR/root-ca.pem:/usr/share/opensearch/config/root-ca.pem 
        
                 - $OPNSDIR/node.pem:/usr/share/opensearch/config/node.pem 
        
                 - $OPNSDIR/node-key.pem:/usr/share/opensearch/config/node-key.pem 
        
                 - $OPNSDIR/admin.pem:/usr/share/opensearch/config/admin.pem 
        
                 - $OPNSDIR/admin-key.pem:/usr/share/opensearch/config/admin-key.pem 
        
                 - $OPNSDIR/opensearch.yml:/usr/share/opensearch/config/opensearch.yml 
        
                 - $OPNSDIR/internal_users.yml:/usr/share/opensearch/plugins/opensearch-security/securityconfig/internal_users.yml 
        
                 - $OPNSDIR/roles_mapping.yml:/usr/share/opensearch/plugins/opensearch-security/securityconfig/roles_mapping.yml

      - $OPNSDIR/root-ca.pem:/usr/share/opensearch/config/root-ca.pem
      - $OPNSDIR/node.pem:/usr/share/opensearch/config/node.pem
      - $OPNSDIR/node-key.pem:/usr/share/opensearch/config/node-key.pem
      - $OPNSDIR/admin.pem:/usr/share/opensearch/config/admin.pem
      - $OPNSDIR/admin-key.pem:/usr/share/opensearch/config/admin-key.pem
      - $OPNSDIR/opensearch.yml:/usr/share/opensearch/config/opensearch.yml
      - $OPNSDIR/internal_users.yml:/usr/share/opensearch/plugins/opensearch-security/securityconfig/internal_users.yml
      - $OPNSDIR/roles_mapping.yml:/usr/share/opensearch/plugins/opensearch-security/securityconfig/roles_mapping.yml

So no demo at all.

enoch85 · 2022-05-18T04:35:04Z

Thanks @Ark74! As always, good work.

vigorio · 2022-05-18T11:27:45Z

Dear enoch85 and Ark74,

this is what I am getting after reinstalling. Even purging docker doesn't help.

ERROR: The Compose file './docker-compose.yml' is invalid because:
volumes.fts_os-data value Additional properties are not allowed ('name' was unexpected)

When I remove the name:fts_os-data from /opt/opensearch/docker-compose.yml I get the node installed but then I get this after executing sudo -u www-data php /var/www/nextcloud/occ fulltextsearch:index

An unhandled exception has been thrown:
Error: Call to a member function getUID() on null in /var/www/nextcloud/apps/files_fulltextsearch/lib/Service/FilesService.php:449
Stack trace:
#0 /var/www/nextcloud/apps/files_fulltextsearch/lib/Service/FilesService.php(421): OCA\Files_FullTextSearch\Service\FilesService->generateFilesDocumentFromFile()
#1 /var/www/nextcloud/apps/files_fulltextsearch/lib/Service/FilesService.php(318): OCA\Files_FullTextSearch\Service\FilesService->generateFilesDocumentFromParent()
#2 /var/www/nextcloud/apps/files_fulltextsearch/lib/Provider/FilesProvider.php(269): OCA\Files_FullTextSearch\Service\FilesService->getFilesFromUser()
#3 /var/www/nextcloud/apps/fulltextsearch/lib/Service/IndexService.php(183): OCA\Files_FullTextSearch\Provider\FilesProvider->generateIndexableDocuments()
#4 /var/www/nextcloud/apps/fulltextsearch/lib/Command/Index.php(416): OCA\FullTextSearch\Service\IndexService->indexProviderContentFromUser()
#5 /var/www/nextcloud/apps/fulltextsearch/lib/Command/Index.php(279): OCA\FullTextSearch\Command\Index->indexProvider()
#6 /var/www/nextcloud/3rdparty/symfony/console/Command/Command.php(255): OCA\FullTextSearch\Command\Index->execute()
#7 /var/www/nextcloud/core/Command/Base.php(168): Symfony\Component\Console\Command\Command->run()
#8 /var/www/nextcloud/3rdparty/symfony/console/Application.php(1009): OC\Core\Command\Base->run()
#9 /var/www/nextcloud/3rdparty/symfony/console/Application.php(273): Symfony\Component\Console\Application->doRunCommand()
#10 /var/www/nextcloud/3rdparty/symfony/console/Application.php(149): Symfony\Component\Console\Application->doRun()
#11 /var/www/nextcloud/lib/private/Console/Application.php(211): Symfony\Component\Console\Application->run()
#12 /var/www/nextcloud/console.php(99): OC\Console\Application->run()
#13 /var/www/nextcloud/occ(11): require_once('/var/www/nextcl...')

enoch85 · 2022-05-18T11:42:27Z

Baah, I should have tested first.

Ark74 · 2022-05-18T12:17:37Z

@vigorio what ubuntu version are you running?

vigorio · 2022-05-18T12:22:50Z

@Ark74 20.04.4 LTS

Ark74 · 2022-05-18T12:53:50Z

I see, please run fulltextsearch.sh with that change (#2329), and select: "Uninstall fulltextsearch", then install again.
That should be enough, you can test "Reinstall" later on.
Regards.

enoch85 · 2022-05-18T12:56:10Z

The final fix is now merged.

@vigorio Please confirm that it works to both reinstall, and uninstall. Thanks!

Ark74 · 2022-05-18T12:58:45Z

His current state seems to be failing, it should be better to first Uninstall then Install, and test the Reinstall afterwards.

vigorio · 2022-05-18T13:39:42Z

@Ark74 I have tested reinstall, uninstall and install and even purging the whole docker instance with images config files etc and I still get the same error…

Ark74 · 2022-05-18T13:52:28Z

@vigorio this issue was opened for a docker issue.
The error you are getting is not related to docker, seems to be an issue with the PHP app.

I would invite fill another bug with a step by step details on how to reproduce the error.
Regards.

enoch85 · 2022-05-18T14:26:48Z

@vigorio Please open another issue.

Thanks!

vigorio · 2022-05-18T15:27:41Z

Issue has been resolved. Seems like my installation has been pulling the old script?! Everything works as expected. Thank you!

Ark74 · 2022-05-18T15:37:04Z

That make sense, thanks for reporting back.
Cheers.

enoch85 added the bug label May 16, 2022

enoch85 assigned Ark74 May 16, 2022

Ark74 mentioned this issue May 17, 2022

Fix fts reinstall #2328

Merged

enoch85 closed this as completed May 18, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

FTS reinstallation is broken #2326

FTS reinstallation is broken #2326

enoch85 commented May 16, 2022 •

edited

Loading

enoch85 commented May 16, 2022

Ark74 commented May 17, 2022 via email

Ark74 commented May 17, 2022

Ark74 commented May 17, 2022

Ark74 commented May 17, 2022

enoch85 commented May 18, 2022

vigorio commented May 18, 2022

enoch85 commented May 18, 2022

Ark74 commented May 18, 2022

vigorio commented May 18, 2022

Ark74 commented May 18, 2022

enoch85 commented May 18, 2022

Ark74 commented May 18, 2022

vigorio commented May 18, 2022 •

edited

Loading

Ark74 commented May 18, 2022 •

edited

Loading

enoch85 commented May 18, 2022

vigorio commented May 18, 2022

Ark74 commented May 18, 2022

FTS reinstallation is broken #2326

FTS reinstallation is broken #2326

Comments

enoch85 commented May 16, 2022 • edited Loading

Steps To Reproduce

Expected Result

Actual Result

Screenshots, Videos, or Pastebins

Additional Context

Build Version

Environment

Environment Details

enoch85 commented May 16, 2022

Ark74 commented May 17, 2022 via email

Ark74 commented May 17, 2022

Ark74 commented May 17, 2022

Ark74 commented May 17, 2022

enoch85 commented May 18, 2022

vigorio commented May 18, 2022

enoch85 commented May 18, 2022

Ark74 commented May 18, 2022

vigorio commented May 18, 2022

Ark74 commented May 18, 2022

enoch85 commented May 18, 2022

Ark74 commented May 18, 2022

vigorio commented May 18, 2022 • edited Loading

Ark74 commented May 18, 2022 • edited Loading

enoch85 commented May 18, 2022

vigorio commented May 18, 2022

Ark74 commented May 18, 2022

enoch85 commented May 16, 2022 •

edited

Loading

vigorio commented May 18, 2022 •

edited

Loading

Ark74 commented May 18, 2022 •

edited

Loading