Wednesday, 2019-10-02

« Tuesday, 2019-10-01 Index Thursday, 2019-10-03 »
*** tetsuro has joined #openstack-placement00:18
*** tetsuro has quit IRC00:20
*** irclogbot_0 has quit IRC02:09
*** irclogbot_3 has joined #openstack-placement02:12
*** tetsuro has joined #openstack-placement02:31
*** tetsuro has quit IRC02:33
*** takashin has joined #openstack-placement04:37
*** sean-k-mooney has quit IRC04:40
*** sean-k-mooney has joined #openstack-placement04:40
*** ttsiouts has joined #openstack-placement06:39
*** ttsiouts has quit IRC07:26
*** ttsiouts has joined #openstack-placement07:26
*** rastos2 has quit IRC07:30
*** rastos2 has joined #openstack-placement07:30
*** ttsiouts has quit IRC07:31
*** tetsuro has joined #openstack-placement07:41
*** tetsuro has quit IRC07:46
*** ttsiouts has joined #openstack-placement07:56
*** e0ne has joined #openstack-placement07:57
*** takashin has left #openstack-placement08:00
*** ttsiouts has quit IRC08:12
*** ttsiouts has joined #openstack-placement08:12
*** ttsiouts has quit IRC08:17
*** ttsiouts has joined #openstack-placement08:18
*** tssurya_ has joined #openstack-placement08:19
*** e0ne has quit IRC08:20
*** tetsuro has joined #openstack-placement09:03
*** tetsuro has quit IRC09:08
*** tssurya_ is now known as tssurya09:19
*** e0ne has joined #openstack-placement09:24
*** tetsuro has joined #openstack-placement09:39
*** tetsuro has quit IRC09:45
*** tetsuro has joined #openstack-placement10:13
*** tetsuro has quit IRC10:18
*** ttsiouts has quit IRC10:32
*** ttsiouts has joined #openstack-placement10:33
*** ttsiouts has quit IRC10:37
*** tetsuro has joined #openstack-placement10:48
*** e0ne has quit IRC10:51
*** tetsuro has quit IRC10:53
*** cdent has joined #openstack-placement10:55
*** cdent has quit IRC11:03
*** openstackstatus has quit IRC11:09
*** ttsiouts has joined #openstack-placement11:14
*** artom has joined #openstack-placement11:47
*** mriedem has joined #openstack-placement13:08
*** e0ne has joined #openstack-placement13:13
*** mriedem has quit IRC14:10
*** mriedem has joined #openstack-placement14:10
*** openstackstatus has joined #openstack-placement14:28
*** ChanServ sets mode: +v openstackstatus14:28
*** rastos3 has joined #openstack-placement14:28
rastos3anybody here for the office hours?15:07
rastos3I'm trying to follow https://docs.openstack.org/install-guide/openstack-services.html#minimal-deployment-for-stein and I got stuck at verifying installation 'placement' service where the step #3 at https://docs.openstack.org/placement/stein/install/verify.html fails with message that just says: 'errors'15:10
*** cdent has joined #openstack-placement15:12
gibirastos3: what is the exact commnad you run?15:13
rastos3gibi: openstack --os-placement-api-version 1.2 resource class list --sort-column name15:13
rastos3the apache log says:15:14
rastos3 WARNING keystonemiddleware.auth_token [-] Identity response: {"error":{"code":401,"message":"The request you have made requires authentication.","title":"Unauthorized"}}15:14
rastos32019-10-02 17:06:24.241708 : keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-ea00e7c0-5166-4d5c-bb99-3a54c1f5d06d)\x1b[00m15:14
gibirastos3: could you run openstack --debug <rest of your command> ? and paste the output some pastebin?15:14
gibirastos3: do you have proper user / pass set up? can you say $ opentack endpoint list?15:15
cdentrastos3: did you do the '. admin-openrc'  (or whatever the equivalent is in your environment)?15:15
rastos3https://pastebin.com/saWMYE9w15:16
rastos3I believe that I do have the username and passwords correct.15:16
rastos3I can do "token issue" command and I get a reasonabe output from that15:16
cdentrastos3: this part "The server is currently unavailable. Please try again at a later time.<br /><br />\nThe Keystone service is temporarily unavailable.\n\n", "code": "503 Service Unavailable""15:17
rastos3cdent: yes - see the pastebin. the same happens for "admin" user as well as "placement" user (I created placement-openrc with the right OS_ variable values)15:17
cdentsuggests that placement isn't configured to talk to keystone prorperly15:17
cdentor may have a problem reaching it15:18
cdentif you do 'curl http://controller:8778/ ' do you get placement version info (it shouldn't need auth)15:18
rastos3root@controller:~# curl http://controller:8778/15:19
rastos3{"versions": [{"id": "v1.0", "max_version": "1.31", "min_version": "1.0", "status": "CURRENT", "links": [{"rel": "self", "href": ""}]}]}15:19
* gibi let cdent handle rastos3's problem15:20
cdentyeah, sorry for jumping in gibi, but it was another case of distraction from other less interesting stuff15:20
rastos3I think I'm not the only one with the problem. see http://lists.openstack.org/pipermail/openstack/2017-March/018880.html and  https://ask.openstack.org/en/question/123187/release-stein-placement-service-credentials-do-not-work/15:21
gibicdent: no worries I'm glad that I'm not alone15:21
cdentrastos3: are you seeing similar errors in keystone?15:23
rastos3cdent: I'm new to all this. Where should I see that?15:23
cdentin something like /var/log/keystone/keystone-wsgi-public.log15:24
cdentbut it depends on how you did your install. Are you using packages, from source, osa, ?15:24
rastos32019-10-02 17:14:55.013 36 WARNING keystone.auth.core [req-f53a6c95-4ca3-41e8-b622-d91e53b4e068 - - - - -] Could not find domain: default.: keystone.exception.DomainNotFound: Could not find domain: default.15:24
rastos32019-10-02 17:14:55.017 36 WARNING keystone.server.flask.application [req-f53a6c95-4ca3-41e8-b622-d91e53b4e068 - - - - -] Authorization failed. The request you have made requires authentication. from 172.17.0.3: keystone.exception.Unauthorized: The request you have made requires authentication.15:24
rastos3can it be that the domain name is case sensitive?15:25
rastos3I think I have "Default" on some places and "default" on others15:25
rastos3the "domain list" returns:15:26
rastos3root@controller:~# openstack domain list15:26
rastos3+----------------------------------+------------+---------+--------------------+15:26
rastos3| ID                               | Name       | Enabled | Description        |15:26
rastos3+----------------------------------+------------+---------+--------------------+15:26
rastos3| default                          | Default    | True    | The default domain |15:26
rastos3| 9a54d3588ad047bba35343d19dee4cca | amc_domain | True    | Automarker Domain  |15:26
rastos3+----------------------------------+------------+---------+--------------------+15:26
efriedseems like osc can talk to keystone, but placement can't talk to keystone. This suggests that something may be amiss with your placement.conf's [keystone_authtoken] section.15:26
rastos3I have appended this to the end of the installed keystone.conf:15:28
rastos3[keystone_authtoken]15:28
rastos3auth_url = http://controller:5000/v315:28
rastos3memcached_servers = controller:1121115:28
rastos3auth_type = password15:28
rastos3project_domain_name = default15:28
rastos3user_domain_name = default15:28
rastos3project_name = service15:28
rastos3username = placement15:28
rastos3password = PLACEMENT_PASS15:28
rastos3uh. sorry. meant to say " ... to the end of installed placement.conf"15:28
cdentsorry, got called away, catching up15:32
cdentefried: yes, that's what I think do15:32
efriedrastos3: you had that in your placement.conf originally, or you're trying that now?15:33
cdentthat keystone_authoken chunk needs to be in placement.conf not keystone.conf and you need whatever the real password is where PLACEMENT_PASS is (unless15:33
efried...and I'm not sure whether you can use the domain ID (default) rather than the name (Default) as {project|user}_domain_name15:34
rastos3I'm actually doing all this in a docker container and the hostname in that container is indeed controller, the password for placement user is indeed PLACEMENT_PASS and the pasted [keystone_authtoken] section was and is in placement.conf15:35
cdentrastos3: is keystone in the same container as placement?15:36
cdentor are you doing some kind of port mapping amongst containers?15:37
cdentwhat you might do is exec /bin/bash into the placement container and see if you can reach http://controller:5000/15:37
rastos3it is all in the same container15:37
rastos3I can15:37
rastos3see curl comand above15:37
cdentthat was 8778, can you get to port 5000?15:38
*** ttsiouts has quit IRC15:39
rastos3root@controller:~# curl http://controller:500015:40
rastos3{"versions": {"values": [{"id": "v3.12", "status": "stable", "updated": "2019-01-22T00:00:00Z", "links": [{"rel": "self", "href": "http://controller:5000/v3/"}], "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}]}]}}15:40
*** ttsiouts has joined #openstack-placement15:40
cdentthank15:41
cdents15:41
* cdent thinks15:41
cdentefried: as I recall keystonemiddleware is a bit cagey with the 503. It's a catchall that doesn't accurately align with what 503 really means15:43
*** ttsiouts has quit IRC15:44
rastos3I think that the log message from /var/log/keystone/keystone-wsgi-public.log should be pretty narrowing it down.15:44
cdentrastos3: I'd try changing the case of default/Default and see what happens15:44
efriedcan we try s/default/Default/... yeah15:44
cdentjinx15:44
rastos3root@controller:~# openstack --os-placement-api-version 1.2 --os-domain-id Default resource class list --sort-column name15:45
rastos3The request you have made requires authentication. (HTTP 401) (Request-ID: req-da04f60b-6ee4-4087-8d2b-8345d8f4ea43)15:45
rastos3root@controller:~# openstack --os-placement-api-version 1.2 --os-domain-id default resource class list --sort-column name15:45
rastos3'errors'15:45
rastos3mhm. in the admin-openrc I have:15:46
rastos3export OS_PROJECT_DOMAIN_NAME=Default15:46
rastos3export OS_USER_DOMAIN_NAME=Default15:46
cdentrastos3: no, in placement.conf15:46
rastos3ok. just a sec ...15:46
cdentand then restart the service15:46
rastos3wow!15:48
rastos3root@controller:~# openstack --os-placement-api-version 1.2  resource class list --sort-column name15:48
rastos3+----------------------------+15:48
rastos3| name                       |15:48
rastos3+----------------------------+15:48
rastos3| DISK_GB                    |15:48
rastos3| IPV4_ADDRESS               |15:48
rastos3| MEMORY_MB                  |15:48
rastos3| NET_BW_EGR_KILOBIT_PER_SEC |15:48
rastos3| NET_BW_IGR_KILOBIT_PER_SEC |15:48
rastos3| NUMA_CORE                  |15:48
rastos3| NUMA_MEMORY_MB             |15:48
rastos3| NUMA_SOCKET                |15:48
rastos3| NUMA_THREAD                |15:48
rastos3| PCI_DEVICE                 |15:48
efriedwoot15:48
rastos3root@controller:~# grep -i domain /etc/placement/placement.conf15:48
rastos3#logging_user_identity_format = %(user)s %(tenant)s %(domain)s %(user_domain)s %(project_domain)s15:48
rastos3project_domain_name = Default15:48
rastos3user_domain_name = Default15:48
rastos3thanks guys. thanks a lot15:48
gibigood job cdent, efried :)15:49
rastos3I guess the documentation https://docs.openstack.org/placement/stein/install/install-ubuntu.html should be fixed to reflect that.15:50
cdentrastos3: can you recall which piece of document had it wrong15:50
cdentjinxu again :)(15:50
rastos3( or the case sensitivity check in code)15:50
cdenthmm. I wonder where the domain is being created, because I suspect those instructions are correct some of the time...15:52
rastos3the domain comes from https://docs.openstack.org/keystone/stein/install/keystone-install-ubuntu.html15:53
rastos3I think.15:54
rastos3probably from "keystone-manage bootstrap  ..." command. In the subsequent pages it is already expected to exist.15:54
openstackgerritBalazs Gibizer proposed openstack/placement master: Refine concurrent_update error code  https://review.opendev.org/68619415:54
rastos3gibi, cdent, efried: I appreciate your help, I'll have to disconnect now and try to continue with "nova" tomorrow.15:57
rastos3bye15:57
efriedo/15:57
gibirastos3: bye15:57
cdentgood luck15:57
rastos3thanks15:57
*** rastos3 has quit IRC15:57
cdentgibi, efried : https://storyboard.openstack.org/#!/story/200665216:01
efriedack16:01
cdenti wanted to get that captured, mostly because this is about the extent of my available attention until friday16:01
gibiefried, cdent: I can push a doc patch quickly16:07
cdentgibi: we'll likely want to confirm that "Default" is the default everywhere16:17
*** cdent has quit IRC16:18
gibicdent: I quickly checked the different config guides and all the tree (ubunut, obs, rdo) has the domain created with name Default and with id default16:18
gibiso the confusion is between id and name16:19
openstackgerritBalazs Gibizer proposed openstack/placement master: Fix domain name in install doc  https://review.opendev.org/68621416:42
gibicdent, efried: ^^16:42
*** e0ne has quit IRC16:55
*** tssurya has quit IRC16:56
*** kevko_ has joined #openstack-placement17:17
kevko_hello guys , can someone advice me why placement is logging CRITICAL keystonemiddleware.auth_token [req-e33c4601-9398-4940-b52d-51e7cdcb8b78 - - - - -] Unable to validate token: Unable to establish connection to https://127.0.0.1:35357: HTTPSConnectionPool(host='127.0.0.1', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f16ee6ee860>: Failed to establi17:17
kevko_sh a new connection: [Errno 111] Connection refused')): keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to https://127.0.0.1:35357: HTTPSConnectionPool(host='127.0.0.1', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f16ee6ee860>: Failed to establish a new connection: [Errno 111] Connection refused'))           even17:17
kevko_if I have correct url configured ? thanks17:17
kevko_and also , if it is hardcoded in a code ...why 35357 ? public and admin endpoint is the same few versions past ..17:19
*** kevko_ has quit IRC17:46
*** rastos3 has joined #openstack-placement18:04
*** e0ne has joined #openstack-placement18:34
*** e0ne has quit IRC18:40
*** e0ne has joined #openstack-placement21:49
*** takashin has joined #openstack-placement21:51
*** rastos3 has quit IRC21:56
mriedemefried: you're placement stable core, can you hit this https://review.opendev.org/#/c/684392/ to unblock the rest of that series?22:12
efried...22:12
efriedmriedem: done. But are you not a placement stable core?22:14
mriedemit's my patch...22:14
efriedI thought there was an implicit +2 somewhere in [Chris's +1, your +2, and smcginnis's +2]22:15
efriedanyway, done.22:15
openstackgerritMerged openstack/osc-placement stable/train: Be explicit about auth type in functional tests  https://review.opendev.org/68439222:25
openstackgerritMerged openstack/osc-placement stable/train: Update .gitreview for stable/train  https://review.opendev.org/68349422:26
*** mriedem has quit IRC22:31
*** e0ne has quit IRC22:35
openstackgerritMerged openstack/osc-placement stable/train: Update TOX/UPPER_CONSTRAINTS_FILE for stable/train  https://review.opendev.org/68349523:05
« Tuesday, 2019-10-01 Index Thursday, 2019-10-03 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!