Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Internal Server Error since v1.1.0 #527

Closed
basti-nis opened this issue Sep 26, 2019 · 22 comments · Fixed by voxpupuli/puppet-puppetboard#343
Closed

Internal Server Error since v1.1.0 #527

basti-nis opened this issue Sep 26, 2019 · 22 comments · Fixed by voxpupuli/puppet-puppetboard#343

Comments

@basti-nis
Copy link

Hi,

since i upgraded to v1.1.0 i've got an Internal Server Error in "Overview" & "Nodes".
All other categories seems to work, so an connection problem shouldn't be.

Complete Error MSG:

Internal Server Error

This error usually occurs because:

We were unable to reach PuppetDB;
The query to be executed was malformed resulting in an incorrectly encoded request.

Please have a look at the log output for further information.

Anybody else has that problem?
Any further information needed?

PuppetDB Version = 6.6.0
Puppetboard = 1.1.0
@bastelfreak
Copy link
Member

Hi, I did a quick test with the Dockerfile in this repo and puppetdb 6.6.0 on centos, I'm not able to reproduce the error. Did you try to downgrade puppetboard/pypuppetdb?

@basti-nis
Copy link
Author

Hi,

I did not try to downgrade back to v1.0.0 of puppetboard yet, this was my last option i wanted to do.

I see some errors in Apache2, maybe they can help a little bit:
[remote 10.131.0.161:58778] INFO:puppetboard.utils:PuppetDB Version 6.6.0 [remote 10.131.0.161:58778] ERROR:flask.app:Exception on / [GET] [remote 10.131.0.161:58778] Traceback (most recent call last): [remote 10.131.0.161:58778] File "/srv/puppetboard/virtenv-puppetboard/lib/python2.7/site-packages/flask/app.py", line 2292, in wsgi_app [remote 10.131.0.161:58778] response = self.full_dispatch_request() [remote 10.131.0.161:58778] File "/srv/puppetboard/virtenv-puppetboard/lib/python2.7/site-packages/flask/app.py", line 1815, in full_dispatch_request [remote 10.131.0.161:58778] rv = self.handle_user_exception(e) [remote 10.131.0.161:58778] File "/srv/puppetboard/virtenv-puppetboard/lib/python2.7/site-packages/flask/app.py", line 1718, in handle_user_exception [remote 10.131.0.161:58778] reraise(exc_type, exc_value, tb) [remote 10.131.0.161:58778] File "/srv/puppetboard/virtenv-puppetboard/lib/python2.7/site-packages/flask/app.py", line 1813, in full_dispatch_request [remote 10.131.0.161:58778] rv = self.dispatch_request() [remote 10.131.0.161:58778] File "/srv/puppetboard/virtenv-puppetboard/lib/python2.7/site-packages/flask/app.py", line 1799, in dispatch_request [remote 10.131.0.161:58778] return self.view_functions[rule.endpoint](**req.view_args) [remote 10.131.0.161:58778] File "/srv/puppetboard/puppetboard/puppetboard/app.py", line 178, in index [remote 10.131.0.161:58778] for node in nodes: [remote 10.131.0.161:58778] File "/srv/puppetboard/virtenv-puppetboard/lib/python2.7/site-packages/pypuppetdb/api.py", line 406, in nodes [remote 10.131.0.161:58778] nodes = self._query('nodes', **kwargs) [remote 10.131.0.161:58778] TypeError: _query() got an unexpected keyword argument 'with_event_numbers'

@vchepkov
Copy link
Contributor

try downgrading puppetdb to 6.5 There are some connections issues with 6.6

@basti-nis
Copy link
Author

hm, i've only upgraded the puppetboard.
Puppetboard v1.0.0 and PuppetDB 6.6 worked before.

There musst be a little problem to be solved, i'll take a look into it

@tparkercbn
Copy link
Contributor

tparkercbn commented Sep 30, 2019 via email

@othalla
Copy link
Contributor

othalla commented Oct 1, 2019

I think the problem comes from pypuppetdb dep

@basti-nis
Copy link
Author

basti-nis commented Oct 1, 2019 via email •
edited
Loading

@basti-nis
Copy link
Author

I completely rebuild the vm, with same puppetcode as before and now it works like a charm!

@tparkercbn
Copy link
Contributor

tparkercbn commented Oct 4, 2019 via email

@vchepkov
Copy link
Contributor

vchepkov commented Oct 4, 2019

in my experience, I had to rollback to v1.0.0 both, puppetboard and pypuppetdb to make it work again

@othalla
Copy link
Contributor

othalla commented Oct 31, 2019

I completely rebuild the vm, with same puppetcode as before and now it works like a charm!

What do you exactly mean by "same puppetcode as before" downgrading pypuppetdb?

@nmaludy
Copy link
Member

nmaludy commented Nov 1, 2019

We had a similar problem, but it was giving us the following stacktrace:

mod_wsgi (pid=64283): Exception         occurred processing WSGI script '/srv/puppetboard/puppetboard/wsgi.py'.
Traceback (most recent call last):
  File "/srv/puppetboard/puppetboard/   wsgi.py", line 15, in <module> 
    from puppetboard.app import app as  application
  File "/srv/puppetboard/puppetboard/   puppetboard/app.py", line 5, in <module>
    from urllib.parse import unquote,   unquote_plus, quote_plus
ImportError: No module named parse

This was also solved by downgrading to v1.0.0. Looks like v1.1.0 is only working with Python 3+

@basti-nis
Copy link
Author

Sorry for the delayed answer to you guys!!

@othalla
I have an puppet manifest for the VM, which installs the Puppetboard with Apache.
I've used the puppetboard puppet module from Vox Pupuli.
"https://github.com/voxpupuli/puppet-puppetboard"

Version: "version": "5.0.1-rc0"

I've just deleted the VM and rebuild it with my puppetcode.
After that, it worked for me.
No downgrade or something like that.

@tparkercbn
How exactly can i help you?

@basti-nis basti-nis mentioned this issue Nov 15, 2019
Closed
@rismoney
Copy link

rismoney commented Nov 16, 2019 •
edited
Loading

deleted (unrelated issue)

@mvilain
Copy link

mvilain commented Feb 1, 2020

This is still broken. I followed the puppet forge instructions for setting up puppetdb, apache, and puppetboard from the provided modules and I've not been able to get it working after a week of weiling at it. I'd "roll my own" by editing the module but that totally defeats the purpose.

What's the solution going forward?

@bastelfreak
Copy link
Member

@mvilain which changes did/would you do to the module? Can you provide them as a patch?

@basti-nis
Copy link
Author

basti-nis commented Feb 3, 2020 •
edited by kenyon
Loading

Today i've upgraded to v2.0.0!

Now i've got an Internal Server Error again.
After i rebuild my VM now with an Debian Buster image, the failure is still there.

I've investigated some more time into the error message:

[wsgi:error] [pid 456:tid 140332278073088] Traceback (most recent call last):
[wsgi:error] [pid 456:tid 140332278073088]   File "/srv/puppetboard/puppetboard/wsgi.py", line 15, in <module>
[wsgi:error] [pid 456:tid 140332278073088]     from puppetboard.app import app as application
[wsgi:error] [pid 456:tid 140332278073088]   File "/srv/puppetboard/puppetboard/puppetboard/app.py", line 5, in <module>
[wsgi:error] [pid 456:tid 140332278073088]     from urllib.parse import unquote, unquote_plus, quote_plus
[wsgi:error] [pid 456:tid 140332278073088] ImportError: No module named parse

Ok, seems there is a problem with python.
Maybe something with python2 and python3?

Tried it with:
apt-get install libapache2-mod-wsgi-py3 -y

Now we stumble in line 8, seems the package won't fix it.

Next message:

[wsgi:error] [pid 1485:tid 140029063599872] Traceback (most recent call last):
[wsgi:error] [pid 1485:tid 140029063599872]   File "/srv/puppetboard/puppetboard/wsgi.py", line 8, in <module>
[wsgi:error] [pid 1485:tid 140029063599872]     execfile(activate_this, dict(__file__=activate_this))
[wsgi:error] [pid 1485:tid 140029063599872] NameError: name 'execfile' is not defined

Back to libapache2-mod-wsgi:
apt-get install libapache2-mod-wsgi -y

2to3-2.7 "/srv/puppetboard/puppetboard/puppetboard/app.py"

Shows:

RefactoringTool: Skipping optional fixer: buffer
RefactoringTool: Skipping optional fixer: idioms
RefactoringTool: Skipping optional fixer: set_literal
RefactoringTool: Skipping optional fixer: ws_comma
RefactoringTool: Refactored /srv/puppetboard/puppetboard/wsgi.py
--- /srv/puppetboard/puppetboard/wsgi.py        (original)
+++ /srv/puppetboard/puppetboard/wsgi.py        (refactored)
@@ -1,11 +1,11 @@
-from __future__ import absolute_import
+

 import os
 import sys

 os.environ['PUPPETBOARD_SETTINGS'] = '/srv/puppetboard/puppetboard/settings.py'
 activate_this = '/srv/puppetboard/virtenv-puppetboard/bin/activate_this.py'
-execfile(activate_this, dict(__file__=activate_this))
+exec(compile(open(activate_this, "rb").read(), activate_this, 'exec'), dict(__file__=activate_this))

 me = os.path.dirname(os.path.abspath(__file__))
 # Add us to the PYTHONPATH/sys.path if we're not on it
RefactoringTool: Files that need to be modified:
RefactoringTool: /srv/puppetboard/puppetboard/wsgi.py

I use the Puppet Module "puppet-puppetboard" in version 6.0.0 to build the VM with following code:

  $ssl_dir = $::settings::ssldir
  $puppetboard_certname = $::fqdn

  class { 'apache': }
  class { 'apache::mod::wsgi': }
  class { 'puppetboard':
    groups              => 'root',
    enable_catalog      => false,
    manage_git          => true,
    manage_virtualenv   => true,
    default_environment => '*',
    manage_selinux      => false,
    puppetdb_host       => 'puppetdb-01.*******',
    puppetdb_port       => 8081,
    puppetdb_key        => "${ssl_dir}/private_keys/${puppetboard_certname}.pem",
    puppetdb_ssl_verify => "${ssl_dir}/certs/ca.pem",
    puppetdb_cert       => "${ssl_dir}/certs/${puppetboard_certname}.pem",
  }
  class { 'puppetboard::apache::vhost':
    vhost_name               => 'puppetboard.*********',
    port                     => 80,
    custom_apache_parameters => {
      serveraliases => ['puppetboard.***********']
    }
  }

Any suggestions?

@mvilain
Copy link

mvilain commented Feb 3, 2020 via email •
edited
Loading

@basti-nis
Copy link
Author

basti-nis commented Feb 4, 2020 •
edited by kenyon
Loading

I'm using a new solution with docker which is working like a charm.
Maybe it's interesting for the others and it could be an example for the documentation. (@bastelfreak ?)

Known issue:

  • SSL problem: parts of the website are not secure, that's why i commented 'ssl_redirect' out in the nginx::resource::server part. (redirect to http favicon)

You need following puppet modules:

  • puppet/nginx
  • puppetlabs/docker
  • puppetlabs/vcsrepo

Example-Puppet-Class:

class my_puppetboard {
  $git_repo_dir = '/srv/puppetboard'
  $puppetboard_certname = $::fqdn
  $puppetdb_host = '#######'
  $ssl_dir = $::settings::ssldir
  $ssl_crt = '/etc/ssl/certs/#######.crt'
  $ssl_key = '/etc/ssl/private/#######.key'
  $docker_port = 9080
  $servername = 'puppetboard.#######.'

  file { $git_repo_dir:
    ensure => directory,
    mode   => '0755',
  }

  vcsrepo { $git_repo_dir:
    ensure   => present,
    provider => git,
    source   => 'git://github.com/voxpupuli/puppetboard.git',
  }

  include 'docker'

  docker::image { 'puppetboard':
    docker_file => "${git_repo_dir}/Dockerfile",
    docker_dir  => $git_repo_dir,
  }

  docker::run { 'puppetboard':
    image   => 'puppetboard',
    ports   => ["${docker_port}:80"],
    volumes => ["${ssl_dir}:${ssl_dir}"],
    env     => [
        "PUPPETDB_HOST=${puppetdb_host}",
        'PUPPETDB_PORT=8081',
        "PUPPETDB_SSL_VERIFY=${ssl_dir}/certs/ca.pem",
        "PUPPETDB_KEY=${ssl_dir}/private_keys/${puppetboard_certname}.pem",
        "PUPPETDB_CERT=${ssl_dir}/certs/${puppetboard_certname}.pem",
        "INVENTORY_FACTS='Hostname,fqdn, IP Address,ipaddress'",
        'ENABLE_CATALOG=True',
        "GRAPH_FACTS='architecture,puppetversion,osfamily'",
      ],
  }

  file { $ssl_crt:
    ensure => 'present',
    source => "puppet:///modules/${module_name}/ssl/#######..crt"
  }
  file { $ssl_key:
    ensure => 'present',
    source => "puppet:///modules/${module_name}/ssl/#######.key"
  }
  exec { 'create custom dhparam file':
    command => '/usr/bin/openssl dhparam -dsaparam -out /etc/ssl/dhparams.pem 2048',
    creates => '/etc/ssl/dhparams.pem',
  }

class { 'nginx':
    confd_purge             => true,
    server_purge            => true,
    manage_repo             => false,
    worker_processes        => 4,
    worker_connections      => 8192,
    worker_rlimit_nofile    => 250000,
    names_hash_bucket_size  => 128,
    names_hash_max_size     => 1024,
    client_body_buffer_size => '128k',
    events_use              => 'epoll',
    multi_accept            => 'on',
    server_tokens           => 'off',
    gzip                    => 'on',
    require                 => [
      Exec['apt_update'],
      Docker::Run['puppetboard'],
    ],
  }

  nginx::resource::upstream { 'docker-puppetboard':
    ensure  => present,
    members => {
      "puppetboard:${docker_port}" => {
        server       => 'localhost',
        port         => $docker_port,
        fail_timeout => '10s',
      },
    },
  }

  nginx::resource::server { $servername:
    http2                 => 'on',
    index_files           => ['index.php'],
    ssl                   => true,
    # ssl_redirect          => true,
    ssl_cert              => $ssl_crt,
    ssl_key               => $ssl_key,
    ssl_dhparam           => '/etc/ssl/dhparams.pem',
    ssl_protocols         => 'TLSv1 TLSv1.1 TLSv1.2',
    proxy                 => 'http://docker-puppetboard',
    proxy_buffering       => 'off',
    proxy_connect_timeout => '300',
    proxy_read_timeout    => '300',
    proxy_http_version    => '1.1',
  }
}

Hope i can help.
And maybe anybody has a solution for the ssl issue?

Greetings

@gdubicki
Copy link
Contributor

gdubicki commented Mar 2, 2020 •
edited by kenyon
Loading

Same here with puppetboard v2.0.0 and puppet/puppetboard v5.0.0:

mod_wsgi (pid=28932): Target WSGI script '/srv/puppetboard/puppetboard/wsgi.py' cannot be loaded as Python module.
mod_wsgi (pid=28932): Exception occurred processing WSGI script '/srv/puppetboard/puppetboard/wsgi.py'.
Traceback (most recent call last):
  File "/srv/puppetboard/puppetboard/wsgi.py", line 15, in <module>
    from puppetboard.app import app as application
  File "/srv/puppetboard/puppetboard/puppetboard/app.py", line 5, in <module>
    from urllib.parse import unquote, unquote_plus, quote_plus
ImportError: No module named parse

I am using Centos 7 with:

# python --version
Python 2.7.5
# python3 --version
Python 3.6.8

@vchepkov
Copy link
Contributor

I think it has been fixed in v2.2.1

@natehouk
Copy link

natehouk commented May 12, 2020 •
edited by kenyon
Loading

Python 3 is now required.

This is still a problem for the puppet/puppetboard module which defaults to Python 2. For anyone that has ended up here due to the error:

ImportError: No module named parse

and that is using the Puppet Forge module puppet/puppetboard (https://forge.puppet.com/puppet/puppetboard), then you need to add the following to your class declaration to enforce that Python 3 is used:

class { 'puppetboard':
  # use python3 when setting up the virtualenv for puppetboard
  virtualenv_version => '3',
  # specify other parameters here
}

Note that the documentation for the module is incorrect, and the correct parameter to use above is virtualenv_version, not virtualenv_python.

You'll also need to make sure you are using the WSGI Python 3 interpreter, like follows:

class { 'apache': }
class { 'apache::mod::wsgi':
  package_name           => 'libapache2-mod-wsgi-py3',
  mod_path               => '/usr/lib/apache2/modules/mod_wsgi.so',
  wsgi_application_group => 'puppet',
}

*This info is current as of puppet/puppetboard module version 6.1.0

@mike406 mike406 mentioned this issue Jul 7, 2020
Closed
@kenyon kenyon mentioned this issue Nov 6, 2021
Merged
mergwyn added a commit to mergwyn/control-repo that referenced this issue Dec 25, 2021
@mergwyn
Workaround for internal server error
471123f 
As per voxpupuli/puppetboard#527
This was referenced Mar 29, 2022
Closed
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Always use python3 wsgi apache module on Debian kenyon/puppet-puppetboard
10 participants
@bastelfreak @gdubicki @natehouk @vchepkov @rismoney @nmaludy @mvilain @tparkercbn @basti-nis @othalla