Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

File 381191 could not be scanned with GData VaaS because there was a temporary upstream server error #158

Open
aakerbeere opened this issue Nov 8, 2024 · 23 comments · May be fixed by #169
Assignees

Comments

@aakerbeere
Copy link

After having set up VaaS it worked as expected for some time. At a given moment it stopped with logging errors as seen in post title. I found somewhere in the documentation in this case to restart the server.
I don't want to frequently do this only because connection is lost.

@lennartdohmann lennartdohmann self-assigned this Nov 19, 2024
@lennartdohmann
Copy link
Contributor

lennartdohmann commented Nov 19, 2024

Hi @aakerbeere , thanks for reporting your problem, I will look into it.

I can't recreate the problem though. Could you send me a detailed log message from the Nextcloud log reader, on the context menu and then Show details as shown on the screenshot? That would help a lot.
2024-11-19_11-32

It would also be very helpful to know:

  • In which environment is your Nextcloud instance running?
  • Which version are you using? (Antivirus app and Nextcloud server)
  • How often does the problem occur and after how long?

The more details you can give me, the more likely it is for me to see your problem as well.

Thanks again for reporting it!

@aakerbeere
Copy link
Author

aakerbeere commented Nov 20, 2024

thx @lennartdohmann for spending time.

After posting i disabled VAAS. After enabling it takes few minutes for the error to come up again and now is recorded approx. every 5 min.

Raw data:
{"reqId":"NtLeP0uMAn4dLclGzIbT","level":3,"time":"2024-11-20T22:01:48+01:00","remoteAddr":"","user":"--","app":"gdatavaas","method":"","url":"--","message":"File 381191 could not be scanned with GData VaaS because there was a temporary upstream server error","userAgent":"--","version":"30.0.2.2","data":{"app":"gdatavaas"},"id":"673e4e553c85a"}

Nextcloud: 30.0.2.2 running on:
Raspberry Pi 4 Model B Rev 1.4, 8GB running Debian 12.8 (bookworm)
G DATA Antivirus, Version 30.1.2 (originating from app store)

cheers

@unglaublicherdude
Copy link
Member

thx @lennartdohmann for spending time.

After posting i disabled VAAS. After enabling it takes few minutes for the error to come up again and now is recorded approx. every 5 min.

Raw data: {"reqId":"NtLeP0uMAn4dLclGzIbT","level":3,"time":"2024-11-20T22:01:48+01:00","remoteAddr":"","user":"--","app":"gdatavaas","method":"","url":"--","message":"File 381191 could not be scanned with GData VaaS because there was a temporary upstream server error","userAgent":"--","version":"30.0.2.2","data":{"app":"gdatavaas"},"id":"673e4e553c85a"}

Nextcloud: 30.0.2.2 running on: Raspberry Pi 4 Model B Rev 1.4, 8GB running Debian 12.8 (bookworm) G DATA Antivirus, Version 30.1.2 (originating from app store)

cheers

Can you send us the Sha256-Checksum of that file?

@aakerbeere
Copy link
Author

thx @unglaublicherdude

The object "381191" can only be recognized at file system level represented by "/var/www/nextcloud/data/appdata_oc97vfin04v4/preview/c/e/b/5/9/a/9/381191" and is a empty folder. I don't know how it got there, what it is and if it does something.
At user level it is not recognized.
So no Sha256-Checksum available.

@unglaublicherdude
Copy link
Member

thx @unglaublicherdude

The object "381191" can only be recognized at file system level represented by "/var/www/nextcloud/data/appdata_oc97vfin04v4/preview/c/e/b/5/9/a/9/381191" and is a empty folder. I don't know how it got there, what it is and if it does something. At user level it is not recognized. So no Sha256-Checksum available.

This is weird as it should not scan folders. Is this possibly a symbolic link in your directory structure?

Can you send us the output of a

ls -lha /var/www/nextcloud/data/appdata_oc97vfin04v4/preview/c/e/b/5/9/a/9/381191
stat /var/www/nextcloud/data/appdata_oc97vfin04v4/preview/c/e/b/5/9/a/9/381191
file /var/www/nextcloud/data/appdata_oc97vfin04v4/preview/c/e/b/5/9/a/9/381191

@aakerbeere
Copy link
Author

thx @unglaublicherdude
Have a look:

ls -lha /var/www/nextcloud/data/appdata_oc97vfin04v4/preview/c/e/b/5/9/a/9/381191
total 8.0K
drwxr-xr-x 2 www-data www-data 4.0K Nov 28 13:57 .
drwxr-xr-x 3 www-data www-data 4.0K May 10  2024 ..
stat /var/www/nextcloud/data/appdata_oc97vfin04v4/preview/c/e/b/5/9/a/9/381191
  File: /var/www/nextcloud/data/appdata_oc97vfin04v4/preview/c/e/b/5/9/a/9/381191
  Size: 4096      	Blocks: 8          IO Block: 4096   directory
Device: 8,2	Inode: 37363850    Links: 2
Access: (0755/drwxr-xr-x)  Uid: (   33/www-data)   Gid: (   33/www-data)
Access: 2024-12-10 10:35:12.113449924 +0100
Modify: 2024-11-28 13:57:45.525450963 +0100
Change: 2024-12-10 10:35:12.113449924 +0100
 Birth: 2024-12-10 10:35:12.113449924 +0100
file /var/www/nextcloud/data/appdata_oc97vfin04v4/preview/c/e/b/5/9/a/9/381191
/var/www/nextcloud/data/appdata_oc97vfin04v4/preview/c/e/b/5/9/a/9/381191: directory

@unglaublicherdude
Copy link
Member

This is strange, because we specifically select the items from the _files-table, that are not directories. Are you familiar with SQL? We would like you to run a Query to get some further data for this issue.

The first would be this:

SELECT
  fileid,
  (SELECT
     mimetype 
   FROM oc_mimetypes 
   WHERE oc_filecache.mimetype = oc_mimetypes.id) as mimetype
FROM oc_filecache 
WHERE fileid = 381191 \G

The table prefix (in this example the default oc_) might differ for you,

The result should be something like this:

*************************** 1. row ***************************
  fileid: 168
mimetype: httpd/unix-directory
1 row in set (0.000 sec)

So we can see, what the actual mimetype your nextcloud filecache is.

In our query we already have this part to omit directories, but the strings might differ.

@aakerbeere
Copy link
Author

thx @unglaublicherdude

The command "\G" at the end of the "WHERE"-clause is considered invalid (PG 17.2) and without it, the statement returns nothing.
"oc_mimetypes" and "oc_filecache" are available.

Also maybe syntax error (don't know, statements bracketing)

cheers

@unglaublicherdude
Copy link
Member

OK. Does the exact error still occur or has the ID in the error changed?

@aakerbeere
Copy link
Author

I just enabled VaaS again and the error comes up after about 5min. But the file-ID has changed.

Result is the same (Invalid command for "\G", empty output without "\G":

SELECT
  fileid,
  (SELECT
     mimetype 
   FROM oc_mimetypes 
   WHERE oc_filecache.mimetype = oc_mimetypes.id) as mimetype
FROM oc_filecache 
WHERE fileid = 911571 \G

@aakerbeere
Copy link
Author

I actually recognize the file ID to be different on every item.

@aakerbeere
Copy link
Author

Multiple errors are actually listed for different files:

Unexpected error while scanning file with id 912236: {"type":"https://tools.ietf.org/html/rfc9110#section-15.6.1","title":"An error occurred while processing your request.","status":500,"detail":"Could not get ScanResult","traceId":"00-a44450dd3de6068104fa5ed3d1973788-e321eea3da6bb9bb-00"}

@unglaublicherdude
Copy link
Member

What's weird is, that you also cannot find the files with these ids in the database, because we basically get all the ids from the database. Does your Nextcloud instance have highly fluctuating data? So maybe the file is already gone, when the actual scan is done, because there is a time in between the query and the actual scan depending on where the fileid is in the list and how long the scans before that fileid actually take. But to avoid having this error in the logs, we can at least introduce some checks before the scan so you don't get that error anymore.

@unglaublicherdude unglaublicherdude linked a pull request Dec 20, 2024 that will close this issue
@aakerbeere
Copy link
Author

No higly fluctuating data. I am the one and only user :)

Since one hour, no more errors of this type did show up.

If the scan continues after the error, i don't mind about it. I believed it stopped because VaaS UI does not update anymore the output of "Files scanned: 45761 / 66239". But this is probably because all the work has already been done.

@unglaublicherdude
Copy link
Member

Just do be sure that we don't just don't know about a mimetype that is in your DB, can you provide the output of the following query?

SELECT * FROM oc_mimetypes

@aakerbeere
Copy link
Author

Returns nothing. Doing something wrong?

sudo -i -u postgres psql
\c nextcloud
SELECT * FROM oc_mimetypes

@unglaublicherdude
Copy link
Member

I don't know how psql works with multiple command but i would suggest selecting the db with the command line parameter --dbname and then send only the SELECT command.

sudo -i -u postgres psql --dbname=nextcloud --command='SELECT * FROM oc_mimetypes'

@aakerbeere
Copy link
Author

sry.
Have a look:

 id  |                                   mimetype                                    
-----+-------------------------------------------------------------------------------
   1 | httpd
   2 | httpd/unix-directory
   3 | application
   4 | application/json
   5 | application/pdf
   6 | text
   7 | text/markdown
   8 | application/vnd.openxmlformats-officedocument.wordprocessingml.document
   9 | image
  10 | image/png
  11 | application/vnd.oasis.opendocument.text
  12 | application/vnd.oasis.opendocument.presentation
  13 | application/octet-stream
  14 | application/vnd.oasis.opendocument.spreadsheet
  15 | video
  16 | video/mp4
  17 | image/jpeg
  18 | application/javascript
  19 | application/x-gzip
  20 | text/css
  21 | image/svg+xml
  22 | application/gpx+xml
  23 | text/vcard
  24 | application/zip
  25 | text/csv
  26 | image/heic
  27 | video/quicktime
  28 | text/calendar
  29 | application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
  30 | text/plain
  31 | application/vnd.oasis.opendocument.text-template
  32 | application/vnd.oasis.opendocument.spreadsheet-template
  33 | application/vnd.oasis.opendocument.presentation-template
  34 | application/vnd.android.package-archive
  35 | application/xml
  36 | application/vnd.ms-excel.sheet.macroEnabled.12
  37 | application/x-ms-dos-executable
  38 | application/vnd.oasis.opendocument.graphics
  39 | application/vnd.ms-excel
  40 | application/msword
  41 | application/vnd.google-earth.kml+xml
  42 | application/vnd.google-earth.kmz
  43 | image/bmp
  44 | application/vnd.ms-powerpoint
  45 | image/gif
  46 | audio
  47 | audio/mpeg
  48 | video/x-ms-wmv
  49 | text/html
  50 | application/x-msdos-program
  51 | text/x-shellscript
  52 | application/cmd
  53 | application/internet-shortcut
  54 | text/rtf
  55 | application/vnd.openxmlformats-officedocument.wordprocessingml.template
  56 | application/vnd.ms-excel.template.macroEnabled.12
  57 | application/msaccess
  58 | application/x-msi
  59 | application/font-sfnt
  60 | application/x-bzip2
  61 | application/vnd.visio
  62 | application/x-bin
  63 | image/tiff
  64 | application/msonenote
  65 | application/vnd.ms-word.document.macroEnabled.12
  66 | application/vnd.openxmlformats-officedocument.presentationml.presentation
  67 | application/vnd.openxmlformats-officedocument.spreadsheetml.template
  68 | text/x-c
  69 | text/x-c++src
  70 | text/x-h
  71 | application/postscript
  72 | application/x-perl
  73 | application/x-php
  74 | text/x-python
  75 | application/x-tex
  76 | application/x-iwork-keynote-sffkey
  77 | application/x-gimp
  78 | application/x-deb
  79 | application/vnd.oasis.opendocument.text-flat-xml
  80 | application/vnd.oasis.opendocument.spreadsheet-flat-xml
  81 | application/vnd.oasis.opendocument.graphics-flat-xml
  82 | application/vnd.oasis.opendocument.presentation-flat-xml
  83 | application/vnd.openxmlformats-officedocument.wordprocessingml.document.oform
  84 | application/vnd.openxmlformats-officedocument.wordprocessingml.document.docxf
  85 | text/asciidoc
  86 | application/gzip
  87 | image/x-icon
  88 | image/webp
  89 | image/emf
  90 | message/rfc822
  91 | application/vnd.ms-outlook
  92 | application/x-blender
  93 | image/heif
  94 | video/MP2T
  95 | video/x-msvideo
  96 | video/ogg
  97 | video/mpeg
  98 | video/webm
  99 | video/x-matroska
 100 | audio/mp4
 101 | audio/ogg
 102 | audio/vnd.wav
 103 | audio/flac
 104 | audio/aac
 105 | text/x-rst
 106 | application/vnd.oasis.opendocument.graphics-template
 107 | application/vnd.excalidraw+json
 108 | video/3gpp
 109 | application/x-7z-compressed
 110 | application/illustrator
 111 | image/x-dcraw
 112 | image/bpg
 113 | application/comicbook+7z
 114 | application/comicbook+ace
 115 | application/comicbook+rar
 116 | application/comicbook+tar
 117 | application/comicbook+truecrypt
 118 | application/comicbook+zip
 119 | application/coreldraw
 120 | application/java
 121 | application/x-cbr
 122 | video/dv
 123 | application/vnd.ms-fontobject
 124 | application/epub+zip
 125 | application/x-fictionbook+xml
 126 | video/x-flv
 127 | text/impress
 128 | text/x-java-source
 129 | image/jp2
 130 | application/x-kdbx
 131 | application/km
 132 | application/x-krita
 133 | text/x-ldif
 134 | application/vnd.lotus-wordpro
 135 | video/mp2t
 136 | audio/mpegurl
 137 | audio/m4b
 138 | application/x-freemind
 139 | application/x-mobipocket-ebook
 140 | application/x-iwork-numbers-sffnumbers
 141 | application/vnd.oasis.opendocument.formula
 142 | text/org
 143 | application/x-iwork-pages-sffpages
 144 | application/x-font
 145 | audio/x-scpls
 146 | application/vnd.ms-powerpoint.template.macroEnabled.12
 147 | application/vnd.openxmlformats-officedocument.presentationml.template
 148 | application/vnd.ms-powerpoint.addin.macroEnabled.12
 149 | application/vnd.ms-powerpoint.slideshow.macroEnabled.12
 150 | application/vnd.openxmlformats-officedocument.presentationml.slideshow
 151 | application/vnd.ms-powerpoint.presentation.macroEnabled.12
 152 | application/x-photoshop
 153 | application/x-rar-compressed
 154 | text/reveal
 155 | application/rss+xml
 156 | application/sgf
 157 | image/sgi
 158 | application/x-shockwave-flash
 159 | application/x-compressed
 160 | application/x-tar
 161 | application/vnd.garmin.tcx+xml
 162 | image/tga
 163 | video/dvd
 164 | application/vnd.ms-visio.drawing.macroEnabled.12
 165 | application/vnd.ms-visio.drawing
 166 | application/vnd.ms-visio.stencil.macroEnabled.12
 167 | application/vnd.ms-visio.stencil
 168 | application/vnd.ms-visio.template.macroEnabled.12
 169 | application/vnd.ms-visio.template
 170 | audio/wav
 171 | audio/webm
 172 | application/font-woff
 173 | application/vnd.wordperfect
 174 | application/vnd.ms-excel.addin.macroEnabled.12
 175 | application/vnd.ms-excel.sheet.binary.macroEnabled.12
 176 | application/vnd.xmind.workbook
 177 | application/yaml
 178 | message
(178 Zeilen)

@unglaublicherdude
Copy link
Member

Ok. Nice. That is definetly alot more than we have in our database with just the base installation. We will look into them. Might it be possible that you also run the other query with the fileid again?

@aakerbeere
Copy link
Author

With the last corresponding error showing up few minutes before last midnight i get

sudo -i -u postgres psql --dbname=nextcloud --command='SELECT fileid, (SELECT mimetype FROM oc_mimetypes WHERE oc_filecache.mimetype = oc_mimetypes.id) as mimetype FROM oc_filecache WHERE fileid = 912629 \G'
ERROR:  syntax error at or near "\"
ZEILE 1: ...s.id) as mimetype FROM oc_filecache WHERE fileid = 912629 \G
                                                                      ^
sudo -i -u postgres psql --dbname=nextcloud --command='SELECT fileid, (SELECT mimetype FROM oc_mimetypes WHERE oc_filecache.mimetype = oc_mimetypes.id) as mimetype FROM oc_filecache WHERE fileid = 912629'
 fileid |         mimetype         
--------+--------------------------
 912629 | application/octet-stream
(1 Zeile)

@unglaublicherdude
Copy link
Member

So this should definetly be a file by this Filetype. Is it the same for the file that you found to actually be a directory?

@aakerbeere
Copy link
Author

sudo -i -u postgres psql --dbname=nextcloud --command='SELECT fileid, (SELECT mimetype FROM oc_mimetypes WHERE oc_filecache.mimetype = oc_mimetypes.id) as mimetype FROM oc_filecache WHERE fileid = 381191'
 fileid |         mimetype         
--------+--------------------------
 381191 | application/octet-stream
(1 Zeile)

@unglaublicherdude
Copy link
Member

Ok. Thank you for all the effort. So it seems that both ids are saved as files in your Nextcloud database. That's a good thing because that means our query IS correct. I let that information cook the next days and come back to you when I have an idea how to proceed with that information.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants