Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pcs cluster auth does not work #109

Closed
eileon opened this issue Oct 7, 2016 · 8 comments
Closed

pcs cluster auth does not work #109

eileon opened this issue Oct 7, 2016 · 8 comments

Comments

@eileon
Copy link

eileon commented Oct 7, 2016 •
edited
Loading

This makes it is impossible to create any cluster basically.

[root@archie corosync]# pcs  --debug cluster auth archie archicomble -u hacluster --force
Password: 
Running: /usr/bin/ruby -I/usr/lib/pcsd/ /usr/lib/pcsd/pcsd-cli.rb auth
--Debug Input Start--
{"local": false, "nodes": ["archie", "archicomble"], "password": "mypassword", "username": "hacluster", "force": true}
--Debug Input End--
Return Value: 0
--Debug Output Start--
{
  "status": "ok",
  "data": {
    "auth_responses": {
      "archie": {
        "status": "noresponse"
      },
      "archicomble": {
        "status": "noresponse"
      }
    },
    "sync_successful": true,
    "sync_nodes_err": [
    ],
    "sync_responses": {
    }
  },
  "log": [
    "I, [2016-10-07T09:42:14.104798 #10722]  INFO -- : PCSD Debugging enabled\n",
    "D, [2016-10-07T09:42:14.104907 #10722] DEBUG -- : Did not detect RHEL 6\n",
    "I, [2016-10-07T09:42:14.104974 #10722]  INFO -- : Running: /usr/sbin/corosync-cmapctl totem.cluster_name\n",
    "I, [2016-10-07T09:42:14.105049 #10722]  INFO -- : CIB USER: hacluster, groups: \n",
    "D, [2016-10-07T09:42:14.117997 #10722] DEBUG -- : [\"totem.cluster_name (str) = archiclust\\n\"]\n",
    "D, [2016-10-07T09:42:14.118156 #10722] DEBUG -- : []\n",
    "D, [2016-10-07T09:42:14.118216 #10722] DEBUG -- : Duration: 0.012948105s\n",
    "I, [2016-10-07T09:42:14.118338 #10722]  INFO -- : Return Value: 0\n"
  ]
}
--Debug Output End--

**Error: Unable to communicate with** archie
**Error: Unable to communicate with** archicomble

Without --force option

[root@archie corosync]# pcs  --debug cluster auth archie archicomble -u hacluster
Password: 
Running: /usr/bin/ruby -I/usr/lib/pcsd/ /usr/lib/pcsd/pcsd-cli.rb auth
--Debug Input Start--
{"local": false, "nodes": ["archicomble", "archie"], "force": false, "username": "hacluster", "password": "mypassword"}
--Debug Input End--
Return Value: 0
--Debug Output Start--
{
  "status": "ok",
  "data": {
    "auth_responses": {
      "archicomble": {
        "status": "noresponse"
      },
      "archie": {
        "status": "noresponse"
      }
    },
    "sync_successful": true,
    "sync_nodes_err": [
    ],
    "sync_responses": {
    }
  },
  "log": [
    "I, [2016-10-07T09:58:18.938807 #10963]  INFO -- : PCSD Debugging enabled\n",
    "D, [2016-10-07T09:58:18.938913 #10963] DEBUG -- : Did not detect RHEL 6\n",
    "I, [2016-10-07T09:58:18.938980 #10963]  INFO -- : Running: /usr/sbin/corosync-cmapctl totem.cluster_name\n",
    "I, [2016-10-07T09:58:18.939029 #10963]  INFO -- : CIB USER: hacluster, groups: \n",
    "D, [2016-10-07T09:58:18.952007 #10963] DEBUG -- : [\"totem.cluster_name (str) = archiclust\\n\"]\n",
    "D, [2016-10-07T09:58:18.952163 #10963] DEBUG -- : []\n",
    "D, [2016-10-07T09:58:18.952220 #10963] DEBUG -- : Duration: 0.012951644s\n",
    "I, [2016-10-07T09:58:18.952347 #10963]  INFO -- : Return Value: 0\n",
    "I, [2016-10-07T09:58:18.953243 #10963]  INFO -- : SRWT Node: archicomble Request: check_auth\n",
    "E, [2016-10-07T09:58:18.953352 #10963] ERROR -- : **Unable to connect to node archicomble, no token available\n",**
    "I, [2016-10-07T09:58:18.953553 #10963]  INFO -- : SRWT Node: archie Request: check_auth\n",
    "E, [2016-10-07T09:58:18.953600 #10963] ERROR -- : **Unable to connect to node archie, no token available\n"**
  ]
}
--Debug Output End--

**Error: Unable to communicate with** archicomble
**Error: Unable to communicate with** archie

In the last report (without --force option) we can notice : no token available
There is no tokens file in /var/lib/pcsd
@tomjelinek
Copy link
Member

Thank you for reporting the issue. I am not able to reproduce it however, so I would like to know more about your setup.
What pcs version do you have? What is the output of pcs status pcsd archie archicomble --debug? Based on totem.cluster_name (str) = archiclust I think a cluster is running on your nodes. Is that right? If so, can you try authenticating again with the cluster stopped? What is the content of your /etc/corosync/corosync.conf or /etc/cluster/cluster.conf?

@eileon
Copy link
Author

eileon commented Oct 7, 2016 •
edited
Loading

Thank you for your reply.

[root@archie ~]# pcs --version
0.9.154

Installed directly from GitHub with compilation of both pcs and pcsd

[root@archie ~]# pcs status pcsd archie archicomble --debug
Running: /usr/sbin/corosync -v

Finished running: /usr/sbin/corosync -v
Return value: 0
--Debug Stdout Start--
Corosync Cluster Engine, version '2.3.5'
Copyright (c) 2006-2009 Red Hat, Inc.

--Debug Stdout End--
--Debug Stderr Start--

--Debug Stderr End--

Running: /usr/sbin/crm_node -l
Return Value: 1
--Debug Output Start--
--Debug Output End--

Running: /usr/sbin/corosync-cmapctl -b nodelist.node
Return Value: 1
--Debug Output Start--
Failed to initialize the cmap API. Error CS_ERR_LIBRARY
--Debug Output End--

Running: /usr/bin/ruby -I/usr/lib/pcsd/ /usr/lib/pcsd/pcsd-cli.rb read_tokens
Running: /usr/bin/ruby -I/usr/lib/pcsd/ /usr/lib/pcsd/pcsd-cli.rb read_tokens
--Debug Input Start--
{}
--Debug Input End--
--Debug Input Start--
{}
--Debug Input End--
Return Value: 0
--Debug Output Start--
{
  "status": "ok",
  "data": {
  },
  "log": [
    "I, [2016-10-07T14:21:08.222963 #15277]  INFO -- : PCSD Debugging enabled\n",
    "D, [2016-10-07T14:21:08.223070 #15277] DEBUG -- : Did not detect RHEL 6\n",
    "I, [2016-10-07T14:21:08.223140 #15277]  INFO -- : Running: /usr/sbin/corosync-cmapctl totem.cluster_name\n",
    "I, [2016-10-07T14:21:08.223188 #15277]  INFO -- : CIB USER: hacluster, groups: \n",
    "D, [2016-10-07T14:21:08.229005 #15277] DEBUG -- : []\n",
    "D, [2016-10-07T14:21:08.229139 #15277] DEBUG -- : [\"Failed to initialize the cmap API. Error CS_ERR_LIBRARY\\n\"]\n",
    "D, [2016-10-07T14:21:08.229219 #15277] DEBUG -- : Duration: 0.005793061s\n",
    "I, [2016-10-07T14:21:08.229371 #15277]  INFO -- : Return Value: 1\n"
  ]
}
--Debug Output End--

Sending HTTP Request to: https://archicomble:2224/remote/check_auth
Data: None
Response Code: 401
--Debug Response Start--
{"notauthorized":"true"}--Debug Response End--
  archicomble: Unable to authenticate
Return Value: 0
--Debug Output Start--
{
  "status": "ok",
  "data": {
  },
  "log": [
    "I, [2016-10-07T14:21:08.353884 #15278]  INFO -- : PCSD Debugging enabled\n",
    "D, [2016-10-07T14:21:08.354005 #15278] DEBUG -- : Did not detect RHEL 6\n",
    "I, [2016-10-07T14:21:08.354075 #15278]  INFO -- : Running: /usr/sbin/corosync-cmapctl totem.cluster_name\n",
    "I, [2016-10-07T14:21:08.354124 #15278]  INFO -- : CIB USER: hacluster, groups: \n",
    "D, [2016-10-07T14:21:08.361200 #15278] DEBUG -- : []\n",
    "D, [2016-10-07T14:21:08.361774 #15278] DEBUG -- : [\"Failed to initialize the cmap API. Error CS_ERR_LIBRARY\\n\"]\n",
    "D, [2016-10-07T14:21:08.361883 #15278] DEBUG -- : Duration: 0.006909293s\n",
    "I, [2016-10-07T14:21:08.362037 #15278]  INFO -- : Return Value: 1\n"
  ]
}
--Debug Output End--

Sending HTTP Request to: https://archie:2224/remote/check_auth
Data: None
Response Reason: [Errno 111] Connection refused
  archie: Offline

I try to run a cluster archiclust with two nodes archie and archicomble

corosync.conf :

[root@archie corosync]# cat corosync.conf
totem {
        version: 2
    cluster_name: archiclust
        # Disable encryption
        secauth:        off      
        interface {
                member {
                    memberaddr: 192.168.1.2
                }
                member {
                    memberaddr: 192.168.1.3
                }
                ringnumber: 0
                bindnetaddr: 192.168.1.100
                broadcast: yes
                mcastport: 5405
        }
nodelist {
        node {
                ring0_addr: 192.168.1.2
                name: archie
                nodeid: 1
        }
        node {
                ring0_addr: 192.168.1.3
                name: archicomble
                nodeid: 2
        }
}
logging {
        to_stderr: no
        to_logfile: yes
        logfile: /var/log/cluster/corosync.log
        to_syslog: yes
        syslog_facility: daemon
        debug: off
        timestamp: on
        logger_subsys {
            subsys: QUORUM
            debug: off
        }
}

quorum {
        provider: corosync_votequorum
        two_node: 1
}
}

I have problems with corosync.conf too because
transport: udpu
makes service corosync does not run
and/or leaving quorum, nodelist and logging out of totem brackets makes corosync does not run

[root@archie corosync]# systemctl stop corosync.service
[root@archie corosync]# systemctl stop pacemaker.service

Stopping the same on other node archicomble. Same result as previously :

[root@archie corosync]# pcs  --debug cluster auth archie archicomble -u hacluster
Password: 
Running: /usr/bin/ruby -I/usr/lib/pcsd/ /usr/lib/pcsd/pcsd-cli.rb auth
--Debug Input Start--
{"username": "hacluster", "force": false, "local": false, "password": "mypassword", "nodes": ["archie", "archicomble"]}
--Debug Input End--
Return Value: 0
--Debug Output Start--
{
  "status": "ok",
  "data": {
    "auth_responses": {
      "archie": {
        "status": "noresponse"
      },
      "archicomble": {
        "status": "noresponse"
      }
    },
    "sync_successful": true,
    "sync_nodes_err": [
    ],
    "sync_responses": {
    }
  },
  "log": [
    "I, [2016-10-07T14:35:10.556368 #15484]  INFO -- : PCSD Debugging enabled\n",
    "D, [2016-10-07T14:35:10.556476 #15484] DEBUG -- : Did not detect RHEL 6\n",
    "I, [2016-10-07T14:35:10.556544 #15484]  INFO -- : Running: /usr/sbin/corosync-cmapctl totem.cluster_name\n",
    "I, [2016-10-07T14:35:10.556594 #15484]  INFO -- : CIB USER: hacluster, groups: \n",
    "D, [2016-10-07T14:35:10.562219 #15484] DEBUG -- : []\n",
    "D, [2016-10-07T14:35:10.562348 #15484] DEBUG -- : [\"Failed to initialize the cmap API. Error CS_ERR_LIBRARY\\n\"]\n",
    "D, [2016-10-07T14:35:10.562425 #15484] DEBUG -- : Duration: 0.005602347s\n",
    "I, [2016-10-07T14:35:10.562540 #15484]  INFO -- : Return Value: 1\n",
    "I, [2016-10-07T14:35:10.563814 #15484]  INFO -- : SRWT Node: archicomble Request: check_auth\n",
    "E, [2016-10-07T14:35:10.563920 #15484] ERROR -- : Unable to connect to node archicomble, no token available\n",
    "I, [2016-10-07T14:35:10.564123 #15484]  INFO -- : SRWT Node: archie Request: check_auth\n",
    "E, [2016-10-07T14:35:10.564290 #15484] ERROR -- : Unable to connect to node archie, no token available\n",
    "I, [2016-10-07T14:35:10.567586 #15484]  INFO -- : No response from: archie request: /auth, exception: Failed to open TCP connection to archie:2224 (Connection refused - connect(2) for \"archie\" port 2224)\n"
  ]
}
--Debug Output End--

Error: Unable to communicate with archie
Error: Unable to communicate with archicomble

Humm sorry last line points that pcsd is not running on archie

`[root@archie corosync]# systemctl start pcsd.service
`

Same as previously with pacemaker and corosync stopped on both nodes :

[root@archie corosync]#  pcs  --debug cluster auth archie archicomble -u hacluster
Password: 
Running: /usr/bin/ruby -I/usr/lib/pcsd/ /usr/lib/pcsd/pcsd-cli.rb auth
--Debug Input Start--
{"nodes": ["archie", "archicomble"], "force": false, "username": "hacluster", "local": false, "password": "mypassword"}
--Debug Input End--
Return Value: 0
--Debug Output Start--
{
  "status": "ok",
  "data": {
    "auth_responses": {
      "archicomble": {
        "status": "noresponse"
      },
      "archie": {
        "status": "noresponse"
      }
    },
    "sync_successful": true,
    "sync_nodes_err": [
    ],
    "sync_responses": {
    }
  },
  "log": [
    "I, [2016-10-07T15:00:53.862179 #15596]  INFO -- : PCSD Debugging enabled\n",
    "D, [2016-10-07T15:00:53.862286 #15596] DEBUG -- : Did not detect RHEL 6\n",
    "I, [2016-10-07T15:00:53.862353 #15596]  INFO -- : Running: /usr/sbin/corosync-cmapctl totem.cluster_name\n",
    "I, [2016-10-07T15:00:53.862400 #15596]  INFO -- : CIB USER: hacluster, groups: \n",
    "D, [2016-10-07T15:00:53.867691 #15596] DEBUG -- : []\n",
    "D, [2016-10-07T15:00:53.867806 #15596] DEBUG -- : [\"Failed to initialize the cmap API. Error CS_ERR_LIBRARY\\n\"]\n",
    "D, [2016-10-07T15:00:53.867882 #15596] DEBUG -- : Duration: 0.005267721s\n",
    "I, [2016-10-07T15:00:53.867986 #15596]  INFO -- : Return Value: 1\n",
    "I, [2016-10-07T15:00:53.869325 #15596]  INFO -- : SRWT Node: archicomble Request: check_auth\n",
    "E, [2016-10-07T15:00:53.869435 #15596] ERROR -- : Unable to connect to node archicomble, no token available\n",
    "I, [2016-10-07T15:00:53.869638 #15596]  INFO -- : SRWT Node: archie Request: check_auth\n",
    "E, [2016-10-07T15:00:53.869685 #15596] ERROR -- : Unable to connect to node archie, no token available\n"
  ]
}
--Debug Output End--

Error: Unable to communicate with archie
Error: Unable to communicate with archicomble

@eileon
Copy link
Author

eileon commented Oct 7, 2016 •
edited
Loading

Any idea ?

[root@archie pcsd]# gem list

*** LOCAL GEMS ***

backports (3.6.8)
bigdecimal (1.2.8)
bundler (1.13.2)
did_you_mean (1.0.0)
io-console (0.4.5)
json (1.8.3)
minitest (5.8.3)
multi_json (1.12.1)
net-telnet (0.1.1)
open4 (1.3.4)
orderedhash (0.0.6)
power_assert (0.2.6)
psych (2.0.17)
rack (1.6.4)
rack-protection (1.5.3)
rack-test (0.6.3)
rake (10.4.2)
rdoc (4.2.1)
rpam (1.0.1)
sinatra (1.4.7)
sinatra-contrib (1.4.7)
test-unit (3.1.5)
tilt (2.0.5)

Are gems missing ?

@eileon
Copy link
Author

eileon commented Oct 7, 2016 •
edited
Loading

Corosync.conf changed to :

[root@archie corosync]# cat corosync.conf
totem {
        version: 2
    cluster_name: archiclust
        # Disable encryption
        secauth:        off
    transport: udpu
}
nodelist {
        node {
                ring0_addr: 192.168.1.2
                name: archie
                nodeid: 1
        }
        node {
                ring0_addr: 192.168.1.3
                name: archicomble
                nodeid: 2
        }
}

logging {
        to_stderr: no
        to_logfile: yes
        logfile: /var/log/cluster/corosync.log
        to_syslog: yes
        syslog_facility: daemon
        debug: off
        timestamp: on
        logger_subsys {
            subsys: QUORUM
            debug: off
        }
}

quorum {
        provider: corosync_votequorum
        two_node: 1
}

and this makes corosync run on archie. Set the same on archicomble.

corosync stopped on archie and archicomble

corosync.conf same on both archie and archicomble and like above but still same error message with

[root@archie corosync]# pcs --debug cluster auth archie archicomble -u hacluster

@tomjelinek
Copy link
Member

I'm sorry I'm still unable to reproduce this.

It looks like your pcsd is not running or is crashing upon processing the auth request. I think so because of:

"auth_responses": {
  "archicomble": {
    "status": "noresponse"
  },
  "archie": {
    "status": "noresponse"
  }
},

and No response from: archie request: /auth, exception: Failed to open TCP connection to archie:2224 (Connection refused - connect(2) for \"archie\" port 2224).

Please make sure that:

  • corosync and pacemaker are not running on any of your nodes
  • there's no /etc/corosync/corosync.conf file on any of your nodes
  • there's no file in /var/lib/pcsd/ directory on any of your nodes
  • /var/log/pcsd/pcsd.log is empty on all your nodes

This is to reduce possibility of other errors. You normally don't need to do that but it simplifies debugging.

It looks like you're missing rubygem rpam-ruby19 (1.2.1). This is essential for authentication to work.

Once all of the above is fixed, start pcsd and make sure it is running on all your nodes. Try running pcs --debug cluster auth archie archicomble again. If it doesn't work, post the output and /var/log/pcsd/pcsd.log from both nodes.

You can ignore "Unable to connect to node archicomble, no token available" messages when running pcs cluster auth without --force flag. Pcsd simply tries if the node is authenticated. If it isn't, it proceeds with the actual authentication.

@tomjelinek tomjelinek mentioned this issue Oct 13, 2016
Closed
@eileon
Copy link
Author

eileon commented Oct 14, 2016 •
edited
Loading

I did what you said and still no success :

  • corosync stopped
  • pacemaker stopped

And also :

  • ucarp stopped
  • corosync-notifyd stopped
  • No more corosync.conf
  • Nothing in /var/lib/pcsd and nothing in /var/log/pcsd (Doing that, permissions in /var/lib/pcsd went from 700 to 755)

Result of the command :

[root@archie pcsd]# pcs --debug cluster auth archie archicomble
Running: /usr/bin/ruby -I/usr/lib/pcsd/ /usr/lib/pcsd/pcsd-cli.rb read_tokens
--Debug Input Start--
{}
--Debug Input End--
Return Value: 0
--Debug Output Start--
{
  "status": "ok",
  "data": {
  },
  "log": [
    "I, [2016-10-14T11:37:44.226441 #3144]  INFO -- : PCSD Debugging enabled\n",
    "D, [2016-10-14T11:37:44.226547 #3144] DEBUG -- : Did not detect RHEL 6\n",
    "I, [2016-10-14T11:37:44.226615 #3144]  INFO -- : Running: /usr/sbin/corosync-cmapctl totem.cluster_name\n",
    "I, [2016-10-14T11:37:44.226663 #3144]  INFO -- : CIB USER: hacluster, groups: \n",
    "D, [2016-10-14T11:37:44.232766 #3144] DEBUG -- : []\n",
    "D, [2016-10-14T11:37:44.232889 #3144] DEBUG -- : [\"Failed to initialize the cmap API. Error CS_ERR_LIBRARY\\n\"]\n",
    "D, [2016-10-14T11:37:44.232966 #3144] DEBUG -- : Duration: 0.006079916s\n",
    "I, [2016-10-14T11:37:44.233081 #3144]  INFO -- : Return Value: 1\n",
    "W, [2016-10-14T11:37:44.233186 #3144]  WARN -- : Cannot read config 'corosync.conf' from '/etc/corosync/corosync.conf': No such file\n",
    "W, [2016-10-14T11:37:44.233305 #3144]  WARN -- : Cannot read config 'corosync.conf' from '/etc/corosync/corosync.conf': No such file or directory - /etc/corosync/corosync.conf\n"
  ]
}
--Debug Output End--

Sending HTTP Request to: https://archicomble:2224/remote/check_auth
Data: None
Response Reason: [Errno 110] Connection timed out
Username: hacluster
Password: 
Running: /usr/bin/ruby -I/usr/lib/pcsd/ /usr/lib/pcsd/pcsd-cli.rb auth
--Debug Input Start--
{"local": false, "username": "hacluster", "password": "mypassword", "force": false, "nodes": ["archicomble", "archie"]}
--Debug Input End--
Return Value: 0
--Debug Output Start--
{
  "status": "ok",
  "data": {
    "auth_responses": {
      "archie": {
        "status": "noresponse"
      },
      "archicomble": {
        "status": "noresponse"
      }
    },
    "sync_successful": true,
    "sync_nodes_err": [

    ],
    "sync_responses": {
    }
  },
  "log": [
    "I, [2016-10-14T11:40:08.494412 #3172]  INFO -- : PCSD Debugging enabled\n",
    "D, [2016-10-14T11:40:08.494518 #3172] DEBUG -- : Did not detect RHEL 6\n",
    "I, [2016-10-14T11:40:08.494585 #3172]  INFO -- : Running: /usr/sbin/corosync-cmapctl totem.cluster_name\n",
    "I, [2016-10-14T11:40:08.494633 #3172]  INFO -- : CIB USER: hacluster, groups: \n",
    "D, [2016-10-14T11:40:08.500921 #3172] DEBUG -- : []\n",
    "D, [2016-10-14T11:40:08.501052 #3172] DEBUG -- : [\"Failed to initialize the cmap API. Error CS_ERR_LIBRARY\\n\"]\n",
    "D, [2016-10-14T11:40:08.501135 #3172] DEBUG -- : Duration: 0.0062624s\n",
    "I, [2016-10-14T11:40:08.501286 #3172]  INFO -- : Return Value: 1\n",
    "W, [2016-10-14T11:40:08.501392 #3172]  WARN -- : Cannot read config 'corosync.conf' from '/etc/corosync/corosync.conf': No such file\n",
    "W, [2016-10-14T11:40:08.501519 #3172]  WARN -- : Cannot read config 'corosync.conf' from '/etc/corosync/corosync.conf': No such file or directory - /etc/corosync/corosync.conf\n",
    "I, [2016-10-14T11:40:08.502251 #3172]  INFO -- : SRWT Node: archicomble Request: check_auth\n",
    "E, [2016-10-14T11:40:08.502359 #3172] ERROR -- : Unable to connect to node archicomble, no token available\n",
    "I, [2016-10-14T11:40:08.502587 #3172]  INFO -- : SRWT Node: archie Request: check_auth\n",
    "E, [2016-10-14T11:40:08.502637 #3172] ERROR -- : Unable to connect to node archie, no token available\n",
    "I, [2016-10-14T11:41:08.506348 #3172]  INFO -- : No response from: archicomble request: /auth, exception: execution expired\n"
  ]
}
--Debug Output End--

Error: Unable to communicate with archicomble
Error: Unable to communicate with archie

Contents of pcsd.log on archie :


I, [2016-10-14T11:40:08.616097 #3131]  INFO -- : CIB USER: hacluster, groups: 
I, [2016-10-14T11:40:08.638019 #3131]  INFO -- : Return Value: 1
W, [2016-10-14T11:40:08.638182 #3131]  WARN -- : Cannot read config 'corosync.conf' from '/etc/corosync/corosync.conf': No such file
W, [2016-10-14T11:40:08.638290 #3131]  WARN -- : Cannot read config 'corosync.conf' from '/etc/corosync/corosync.conf': No such file or directory - /etc/corosync/corosync.conf
I, [2016-10-14T11:40:08.639011 #3131]  INFO -- : Attempting login by 'hacluster'
2016-10-14 11:40:08 - NoMethodError - private method `auth' called for Rpam:Module:
    /usr/lib/pcsd/auth.rb:21:in `validUser'
    /usr/lib/pcsd/remote.rb:1314:in `auth'
    /usr/lib/pcsd/remote.rb:131:in `call'
    /usr/lib/pcsd/remote.rb:131:in `remote'
    /usr/lib/pcsd/pcsd.rb:220:in `block in <top (required)>'
    /usr/lib/ruby/gems/2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1611:in `call'
    /usr/lib/ruby/gems/2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1611:in `block in compile!'
    /usr/lib/ruby/gems/2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:975:in `block (3 levels) in route!'
    /usr/lib/ruby/gems/2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:994:in `route_eval'
    /usr/lib/ruby/gems/2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:975:in `block (2 levels) in route!'
    /usr/lib/ruby/gems/2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1015:in `block in process_route'
    /usr/lib/ruby/gems/2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1013:in `catch'
    /usr/lib/ruby/gems/2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1013:in `process_route'
    /usr/lib/ruby/gems/2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:973:in `block in route!'
    /usr/lib/ruby/gems/2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:972:in `each'
    /usr/lib/ruby/gems/2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:972:in `route!'
    /usr/lib/ruby/gems/2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1085:in `block in dispatch!'
    /usr/lib/ruby/gems/2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in `block in invoke'
    /usr/lib/ruby/gems/2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in `catch'
    /usr/lib/ruby/gems/2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in `invoke'
    /usr/lib/ruby/gems/2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1082:in `dispatch!'
    /usr/lib/ruby/gems/2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:907:in `block in call!'
    /usr/lib/ruby/gems/2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in `block in invoke'
    /usr/lib/ruby/gems/2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in `catch'
    /usr/lib/ruby/gems/2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in `invoke'
    /usr/lib/ruby/gems/2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:907:in `call!'
    /usr/lib/ruby/gems/2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:895:in `call'
    /usr/lib/ruby/gems/2.3.0/gems/rack-1.6.4/lib/rack/session/abstract/id.rb:225:in `context'
    /usr/lib/ruby/gems/2.3.0/gems/rack-1.6.4/lib/rack/session/abstract/id.rb:220:in `call'
    /usr/lib/pcsd/session.rb:13:in `call'
    /usr/lib/ruby/gems/2.3.0/gems/rack-1.6.4/lib/rack/commonlogger.rb:33:in `call'
    /usr/lib/ruby/gems/2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:219:in `call'
    /usr/lib/ruby/gems/2.3.0/gems/rack-protection-1.5.3/lib/rack/protection/xss_header.rb:18:in `call'
    /usr/lib/ruby/gems/2.3.0/gems/rack-protection-1.5.3/lib/rack/protection/path_traversal.rb:16:in `call'
    /usr/lib/ruby/gems/2.3.0/gems/rack-protection-1.5.3/lib/rack/protection/json_csrf.rb:18:in `call'
    /usr/lib/ruby/gems/2.3.0/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
    /usr/lib/ruby/gems/2.3.0/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
    /usr/lib/ruby/gems/2.3.0/gems/rack-protection-1.5.3/lib/rack/protection/frame_options.rb:31:in `call'
    /usr/lib/ruby/gems/2.3.0/gems/rack-1.6.4/lib/rack/logger.rb:15:in `call'
    /usr/lib/ruby/gems/2.3.0/gems/rack-1.6.4/lib/rack/commonlogger.rb:33:in `call'
    /usr/lib/ruby/gems/2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:219:in `call'
    /usr/lib/ruby/gems/2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:212:in `call'
    /usr/lib/ruby/gems/2.3.0/gems/rack-1.6.4/lib/rack/head.rb:13:in `call'
    /usr/lib/ruby/gems/2.3.0/gems/rack-1.6.4/lib/rack/methodoverride.rb:22:in `call'
    /usr/lib/ruby/gems/2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:182:in `call'
    /usr/lib/ruby/gems/2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:2013:in `call'
    /usr/lib/ruby/gems/2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1487:in `block in call'
    /usr/lib/ruby/gems/2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1787:in `synchronize'
    /usr/lib/ruby/gems/2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1487:in `call'
    /usr/lib/ruby/gems/2.3.0/gems/rack-1.6.4/lib/rack/handler/webrick.rb:88:in `service'
    /usr/lib/ruby/2.3.0/webrick/httpserver.rb:140:in `service'
    /usr/lib/ruby/2.3.0/webrick/httpserver.rb:96:in `run'
    /usr/lib/ruby/2.3.0/webrick/server.rb:296:in `block in start_thread'
::1 - - [14/Oct/2016:11:40:08 +0200] "POST /remote/auth HTTP/1.1" 500 30 0.0493
::1 - - [14/Oct/2016:11:40:08 +0200] "POST /remote/auth HTTP/1.1" 500 30 0.0496
::1 - - [14/Oct/2016:11:40:08 CEST] "POST /remote/auth HTTP/1.1" 500 30
- -> /remote/auth
I, [2016-10-14T11:40:10.961527 #3131]  INFO -- : Running: /usr/sbin/corosync-cmapctl totem.cluster_name
I, [2016-10-14T11:40:10.961676 #3131]  INFO -- : CIB USER: hacluster, groups: 
I, [2016-10-14T11:40:10.984058 #3131]  INFO -- : Return Value: 1
W, [2016-10-14T11:40:10.984224 #3131]  WARN -- : Cannot read config 'corosync.conf' from '/etc/corosync/corosync.conf': No such file
W, [2016-10-14T11:40:10.984324 #3131]  WARN -- : Cannot read config 'corosync.conf' from '/etc/corosync/corosync.conf': No such file or directory - /etc/corosync/corosync.conf

Maybe a problem ?

Contents of pcsd.log on archicomble :

Nothing special. Seems not have been contacted.

My gem list of the moment :

[root@archie pcsd]# gem list

*** LOCAL GEMS ***

addressable (2.4.0)
backports (3.6.8)
bigdecimal (1.2.8)
bundler (1.13.2)
colored (1.2)
did_you_mean (1.0.0)
domain_name (0.5.20160826)
eventmachine (1.2.0.1)
facets (3.1.0)
ffi2-generators (0.1.1)
fileutils (0.7)
highline (1.7.8)
http (2.0.3)
http-cookie (1.0.3)
http-form_data (1.0.1)
http_parser.rb (0.6.0)
io-console (0.4.5)
json (1.8.3)
minitest (5.8.3)
monitor (0.1.3)
monkey-lib (0.5.4)
multi_json (1.12.1)
net-telnet (0.1.1)
open4 (1.3.4)
orderedhash (0.0.6)
power_assert (0.2.6)
psych (2.0.17)
quality_extensions (1.4.0)
rack (1.6.4)
rack-protection (1.5.3)
rack-test (0.6.3)
rake (10.4.2)
rdoc (4.2.1)
rmagick (2.16.0)
rpam (1.0.1)
rpam-ruby19 (1.2.1)
rubysl-fcntl (2.0.4)
sinatra (1.4.7)
sinatra-contrib (1.4.7)
sinatra-sugar (0.5.1)
test-unit (3.1.5)
thor (0.19.1)
tilt (2.0.5)
unf (0.1.4)
unf_ext (0.0.7.2)
unroller (1.0.0)

I have rpam-ruby19 (1.2.1)
`

@tomjelinek
Copy link
Member

Can you try it again with this patch e5347d3 included?

@eileon
Copy link
Author

eileon commented Oct 14, 2016

Patch seems to work.

Setting it on archicomble made me see that rpam-ruby19 (1.2.1) was not installed. It was installed on archie but not on archicomble. (With this patch pcsd could not be restarted on archicomble)

Result of the command on archie :

[root@archie pcsd]# pcs --debug cluster auth archie archicomble
Running: /usr/bin/ruby -I/usr/lib/pcsd/ /usr/lib/pcsd/pcsd-cli.rb read_tokens
--Debug Input Start--
{}
--Debug Input End--
Return Value: 0
--Debug Output Start--
{
  "status": "ok",
  "data": {
  },
  "log": [
    "I, [2016-10-14T14:55:59.469982 #4323]  INFO -- : PCSD Debugging enabled\n",
    "D, [2016-10-14T14:55:59.470091 #4323] DEBUG -- : Did not detect RHEL 6\n",
    "I, [2016-10-14T14:55:59.470158 #4323]  INFO -- : Running: /usr/sbin/corosync-cmapctl totem.cluster_name\n",
    "I, [2016-10-14T14:55:59.470206 #4323]  INFO -- : CIB USER: hacluster, groups: \n",
    "D, [2016-10-14T14:55:59.476013 #4323] DEBUG -- : []\n",
    "D, [2016-10-14T14:55:59.476137 #4323] DEBUG -- : [\"Failed to initialize the cmap API. Error CS_ERR_LIBRARY\\n\"]\n",
    "D, [2016-10-14T14:55:59.476214 #4323] DEBUG -- : Duration: 0.005784438s\n",
    "I, [2016-10-14T14:55:59.476327 #4323]  INFO -- : Return Value: 1\n",
    "W, [2016-10-14T14:55:59.476427 #4323]  WARN -- : Cannot read config 'corosync.conf' from '/etc/corosync/corosync.conf': No such file\n",
    "W, [2016-10-14T14:55:59.476546 #4323]  WARN -- : Cannot read config 'corosync.conf' from '/etc/corosync/corosync.conf': No such file or directory - /etc/corosync/corosync.conf\n"
  ]
}
--Debug Output End--

Sending HTTP Request to: https://archicomble:2224/remote/check_auth
Data: None
Response Reason: [Errno 110] Connection timed out
Username: hacluster
Password: 
Running: /usr/bin/ruby -I/usr/lib/pcsd/ /usr/lib/pcsd/pcsd-cli.rb auth
--Debug Input Start--
{"force": false, "nodes": ["archicomble", "archie"], "password": "mypassword", "username": "hacluster", "local": false}
--Debug Input End--
Return Value: 0
--Debug Output Start--
{
  "status": "ok",
  "data": {
    "auth_responses": {
      "archie": {
        "status": "noresponse"
      },
      "archicomble": {
        "status": "noresponse"
      }
    },
    "sync_successful": true,
    "sync_nodes_err": [

    ],
    "sync_responses": {
    }
  },
  "log": [
    "I, [2016-10-14T14:58:23.211333 #4338]  INFO -- : PCSD Debugging enabled\n",
    "D, [2016-10-14T14:58:23.211440 #4338] DEBUG -- : Did not detect RHEL 6\n",
    "I, [2016-10-14T14:58:23.211506 #4338]  INFO -- : Running: /usr/sbin/corosync-cmapctl totem.cluster_name\n",
    "I, [2016-10-14T14:58:23.211554 #4338]  INFO -- : CIB USER: hacluster, groups: \n",
    "D, [2016-10-14T14:58:23.217576 #4338] DEBUG -- : []\n",
    "D, [2016-10-14T14:58:23.217698 #4338] DEBUG -- : [\"Failed to initialize the cmap API. Error CS_ERR_LIBRARY\\n\"]\n",
    "D, [2016-10-14T14:58:23.217774 #4338] DEBUG -- : Duration: 0.005997222s\n",
    "I, [2016-10-14T14:58:23.217933 #4338]  INFO -- : Return Value: 1\n",
    "W, [2016-10-14T14:58:23.218035 #4338]  WARN -- : Cannot read config 'corosync.conf' from '/etc/corosync/corosync.conf': No such file\n",
    "W, [2016-10-14T14:58:23.218152 #4338]  WARN -- : Cannot read config 'corosync.conf' from '/etc/corosync/corosync.conf': No such file or directory - /etc/corosync/corosync.conf\n",
    "I, [2016-10-14T14:58:23.218850 #4338]  INFO -- : SRWT Node: archicomble Request: check_auth\n",
    "E, [2016-10-14T14:58:23.218950 #4338] ERROR -- : Unable to connect to node archicomble, no token available\n",
    "I, [2016-10-14T14:58:23.219166 #4338]  INFO -- : SRWT Node: archie Request: check_auth\n",
    "E, [2016-10-14T14:58:23.219216 #4338] ERROR -- : Unable to connect to node archie, no token available\n",
    "I, [2016-10-14T14:59:23.221882 #4338]  INFO -- : No response from: archicomble request: /auth, exception: execution expired\n",
    "I, [2016-10-14T14:59:23.222121 #4338]  INFO -- : Running: /usr/sbin/pcs status nodes corosync\n",
    "I, [2016-10-14T14:59:23.222314 #4338]  INFO -- : CIB USER: hacluster, groups: \n",
    "D, [2016-10-14T14:59:24.081593 #4338] DEBUG -- : []\n",
    "D, [2016-10-14T14:59:24.081732 #4338] DEBUG -- : [\"Error: Unable to read /etc/corosync/corosync.conf: No such file or directory\\n\"]\n",
    "D, [2016-10-14T14:59:24.081820 #4338] DEBUG -- : Duration: 0.859295915s\n",
    "I, [2016-10-14T14:59:24.082019 #4338]  INFO -- : Return Value: 1\n",
    "I, [2016-10-14T14:59:24.082967 #4338]  INFO -- : Saved config 'tokens' version 1 e3edf31e7cabefe3f4034907d1dfe12525829643 to '/var/lib/pcsd/tokens'\n",
    "I, [2016-10-14T14:59:54.147407 #4338]  INFO -- : No response from: archie request: /auth, exception: Net::ReadTimeout\n",
    "I, [2016-10-14T15:00:24.084775 #4338]  INFO -- : No response from: archicomble request: /auth, exception: execution expired\n"
  ]
}
--Debug Output End--

Error: Unable to communicate with archicomble
Error: Unable to communicate with archie

So i can see tokens have been written now 👍
"I, [2016-10-14T14:59:24.082967 #4338] INFO -- : Saved config 'tokens' version 1 e3edf31e7cabefe3f4034907d1dfe12525829643 to '/var/lib/pcsd/tokens'\n",
and

[root@archie pcsd]# ls
pcsd.cookiesecret  pcsd.crt  pcsd.key  pcs_users.conf  tokens
[root@archie pcsd]# pwd
/var/lib/pcsd

Effectively i have :

rpam (1.0.1)
rpam-ruby19 (1.2.1)

both installed.

Thank you for time and for help !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@eileon @tomjelinek