Wednesday, 2018-03-14

*** savvas has joined #openstack-ansible00:00
*** savvas has quit IRC00:04
*** chyka has quit IRC00:21
openstackgerritMerged openstack/openstack-ansible-os_nova master: ansible-role-requirements: Add common systemd_service role  https://review.openstack.org/55258100:24
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-os_glance master: Convert systemd mount to use a common role  https://review.openstack.org/55218500:33
cloudnullif folks get a change, https://review.openstack.org/#/c/552047 unblocks master00:34
cloudnullill be around tonight to update if needed.00:35
openstackgerritMerged openstack/openstack-ansible-tests stable/pike: Add tests for APTRepositoryCacheUpdateRule  https://review.openstack.org/55202500:35
cloudnull**get a chance00:35
*** Leo_m has quit IRC00:48
*** osnaya has joined #openstack-ansible00:50
osnaya@cloudnull @odyssey4me Just did a dry run with --syntax-check... it shows up one old network range which I don't want. I updated the user_config.yml but the --syntax-check inventory file shows the old entry. How do I clean up and re-run the --syntax-check? so it will update the changes?00:52
*** odyssey4me has quit IRC00:53
*** odyssey4me has joined #openstack-ansible00:53
osnaya@odyssey4me Just did a dry run with --syntax-check... it shows up one old network range which I don't want. I updated the user_config.yml but the --syntax-check inventory file shows the old entry. How do I clean up and re-run the --syntax-check? so it will update the changes?00:55
*** jbadiapa_ has quit IRC01:00
*** kstev1 has quit IRC01:02
osnaya@odyssey4me deploying on Ubuntu Host... openstack_user_config.yml and env.d is set for metal... After the dry run with --syntax-check observed the inventory.json output... all other places it shows up infra1 host (container empty).... But 2 container entries still show up (infra1_galera_container### and infra1_utility_container#### and Ansible is trying to reach these 2 containers..... Any ideas  on what config files I sho01:04
*** kstev has joined #openstack-ansible01:05
osnaya@odyssey4me checked openstack_user_config.yml already and shared-infra_hosts shows up infra1 host ip correctly (which I believe is meant for galera, memcached, rabbit and utility.... then why are the 2 container entries for galera and utility still show up... ?01:07
*** kstev has quit IRC01:11
*** jbadiapa_ has joined #openstack-ansible01:16
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible master: Converge distro interfaces to systemd-networkd  https://review.openstack.org/55272101:23
*** osnaya has quit IRC01:26
openstackgerritMerged openstack/openstack-ansible-os_heat master: ansible-role-requirements: Add common systemd_service role  https://review.openstack.org/55259401:50
prometheanfiremhayden: aa5ro has not switched over to dst02:01
openstackgerritJimmy McCrory proposed openstack/openstack-ansible-os_keystone master: Avoid conflicting delegation with authorized_keys  https://review.openstack.org/55271102:03
*** nurdie has quit IRC02:16
*** nurdie has joined #openstack-ansible02:16
*** flemingo_ has quit IRC02:25
*** savvas has joined #openstack-ansible02:34
*** savvas has quit IRC02:45
*** savvas has joined #openstack-ansible02:45
*** esberglu has quit IRC03:32
*** savvas has quit IRC03:41
*** vnogin has joined #openstack-ansible03:46
*** savvas has joined #openstack-ansible03:50
*** vnogin has quit IRC03:51
*** savvas has quit IRC03:54
*** gkadam has joined #openstack-ansible03:59
*** coolj has quit IRC04:00
*** coolj has joined #openstack-ansible04:02
*** poopcat has quit IRC04:04
*** esberglu has joined #openstack-ansible04:05
*** udesale has joined #openstack-ansible04:05
*** esberglu has quit IRC04:09
*** ianychoi_ has joined #openstack-ansible04:12
*** ianychoi has quit IRC04:15
*** dave-mccowan has quit IRC04:17
openstackgerritTaseer Ahmed proposed openstack/openstack-ansible master: Integrate Congress with OSA.  https://review.openstack.org/50397104:26
*** chyka has joined #openstack-ansible04:38
*** karthi has joined #openstack-ansible04:39
*** chyka has quit IRC04:43
*** esberglu has joined #openstack-ansible04:59
*** esberglu has quit IRC05:03
*** karthi has quit IRC05:04
*** karthi has joined #openstack-ansible05:05
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible master: Converge distro interfaces to systemd-networkd  https://review.openstack.org/55272105:05
*** karthi has quit IRC05:09
*** savvas has joined #openstack-ansible05:11
*** savvas has quit IRC05:16
*** karthi has joined #openstack-ansible05:22
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible master: Converge distro interfaces to systemd-networkd  https://review.openstack.org/55272105:27
*** savvas has joined #openstack-ansible05:30
*** OpenStackDude has joined #openstack-ansible05:34
*** esberglu has joined #openstack-ansible05:34
*** savvas has quit IRC05:34
*** esberglu has quit IRC05:38
OpenStackDudeIs it possible to use OVS bridges as the base "br-mgmt" bridge that connects to the LXC containers?  OSA is throwing a error when attempting to connect a container interface to the bridge.  It is using brctl commands on the OVS bridge which fails: https://ask.openstack.org/en/question/113180/osa-fails-to-add-lxd-container-interfaces-to-ovs-bridge/05:41
openstackgerritMerged openstack/openstack-ansible-galera_client master: zuul: Add more SUSE jobs  https://review.openstack.org/55251105:48
openstackgerritMerged openstack/openstack-ansible-galera_client master: Updated from global requirements  https://review.openstack.org/55219605:48
*** masber has quit IRC05:53
*** OpenStackDude has quit IRC06:00
*** karthi has quit IRC06:03
openstackgerritMerged openstack/openstack-ansible-galera_server master: tasks: galera_upgrade_pre: Fix conditional for apt sources  https://review.openstack.org/55253606:03
*** savvas has joined #openstack-ansible06:06
openstackgerritMerged openstack/openstack-ansible-galera_server master: tasks: galera_post_install: Do not secure the DB on upgrades  https://review.openstack.org/55263906:07
*** savvas has quit IRC06:10
*** karthi has joined #openstack-ansible06:11
*** karthi has quit IRC06:15
*** karthi has joined #openstack-ansible06:18
openstackgerritTaseer Ahmed proposed openstack/openstack-ansible master: Integrate Congress with OSA.  https://review.openstack.org/50397106:18
Taseerevrardjp: please let me know if I am going in the right direction about adding the tempest API ^06:22
*** esberglu has joined #openstack-ansible06:28
*** esberglu has quit IRC06:32
*** jbadiapa_ has quit IRC06:33
*** radeks has joined #openstack-ansible06:38
*** yifei1 has joined #openstack-ansible06:43
*** ianychoi__ has joined #openstack-ansible06:43
*** yifei has quit IRC06:45
*** yifei1 is now known as yifei06:45
*** ianychoi_ has quit IRC06:46
*** jbadiapa_ has joined #openstack-ansible06:48
*** gkadam has quit IRC06:50
*** ianychoi__ is now known as ianychoi06:51
*** savvas has joined #openstack-ansible07:01
*** pcaruana has joined #openstack-ansible07:04
*** vnogin has joined #openstack-ansible07:05
*** Fritzmen has quit IRC07:05
*** savvas has quit IRC07:05
*** DimGR has quit IRC07:14
*** karthi has quit IRC07:15
*** karthi has joined #openstack-ansible07:15
*** sar has joined #openstack-ansible07:19
*** savvas has joined #openstack-ansible07:19
*** DimGR has joined #openstack-ansible07:20
*** savvas has quit IRC07:23
*** Fritzmen has joined #openstack-ansible07:24
*** masber has joined #openstack-ansible07:32
*** armaan has quit IRC07:36
*** armaan has joined #openstack-ansible07:36
*** savvas has joined #openstack-ansible07:37
*** vnogin has quit IRC07:38
*** karthi has quit IRC07:38
*** karthi has joined #openstack-ansible07:38
*** gkadam has joined #openstack-ansible07:40
*** savvas has quit IRC07:42
*** chyka has joined #openstack-ansible07:42
*** vnogin has joined #openstack-ansible07:44
*** chyka has quit IRC07:47
evrardjpTaseer: that looks great07:51
evrardjpI guess the test basic ops would only work if there is many more components, so I'd keep that commented out for now, and have an extra line to test only the API (if there is such test)07:52
*** vnogin has quit IRC07:56
evrardjpodyssey4me: why was https://review.openstack.org/#/c/552486/ abandonned?08:02
Taseerevrardjp: the api tests are empty08:02
evrardjpTaseer: go go complain on the congress side then08:03
*** epalper has joined #openstack-ansible08:03
Taseerso until api tests are written, this patch would be in the pipeline ?08:03
*** shardy has joined #openstack-ansible08:03
*** epalper has joined #openstack-ansible08:03
Taseerand are all other jobs failing with the same error => http://logs.openstack.org/71/503971/38/check/openstack-ansible-deploy-aio_lxc-ubuntu-xenial/5a961df/job-output.txt.gz#_2018-03-14_07_38_42_70430108:04
Taseerthe keystone container is timing out08:04
evrardjpLet's focus on your thing -- if congress doesn't have API tests, we have no choice than to add functional tests on a more complete system08:05
evrardjpso your overrides are good08:05
evrardjpwe just need to add more components08:05
evrardjpfor the keystone issue, I am querying what's going on08:05
*** ThomasS has joined #openstack-ansible08:06
Taseerokay08:06
*** karthi has quit IRC08:10
openstackgerritMerged openstack/openstack-ansible-os_neutron master: Simplify SELinux check  https://review.openstack.org/55021008:10
evrardjpI am very confused what's going on here. hwoarang are you there?08:11
hwoarangevrardjp: depends...08:11
fdegirevrardjp: hwoarang: https://bugs.launchpad.net/openstack-ansible/+bug/175312508:11
openstackLaunchpad bug 1753125 in openstack-ansible "os_cinder : Ensure cinder api is available timeout" [Undecided,New]08:11
fdegirevrardjp: hwoarang: the original submitter set bug to invalid but we still see it with queens rc2 versions with ha deployment08:12
fdegirsorry for just jumping in...08:12
fdegirevrardjp: hwoarang: just to add that those shas work with no ha deployments08:13
*** savvas has joined #openstack-ansible08:14
*** karthi has joined #openstack-ansible08:16
*** vnogin has joined #openstack-ansible08:17
evrardjphwoarang: we can update the patch, for sure, but I think this is not a good idea.08:18
evrardjpI think we need to have a proper spec up08:18
*** savvas has quit IRC08:18
evrardjpthis replacement of systemd is fine for me, but there is no spec that I am aware of that describes what we want to achieve08:18
evrardjpthe systemd_mount and system_service roles become fundamental to our deploy, yet we have no written evidence of why this happened08:19
evrardjpif human mistake of removal of this repo, nothing can deploy anymore.08:20
hwoarangi dont know what's the deal with the new systemd roles to be honest you need cloudnull for that. we can spli tthis patch into two bits again. one that simply fixes the problem in a dirty way and another one that does it properly08:20
evrardjphwoarang: a patch that does what it says in the current commit message seems reasonable.08:21
hwoarangok08:21
evrardjpI re-enabled the revert of the keystone patch08:21
hwoarangi think it's best to submit a new patch08:21
hwoarang(for the lxc thing)08:21
hwoarangi will do it08:21
evrardjpthere are times we can merge things fast, but changes in the structure of the deploy should be documented.08:21
evrardjphwoarang: you mean for the disk space thing?08:22
hwoarangyes i agree except this is not my patch08:22
hwoarangyes the disk space thing08:22
evrardjpyeah okay08:23
evrardjpI am fine with a new patch.08:23
evrardjpIt is your patch though, it's written owner: Markos :)08:23
hwoarangwell that's a mistake i only wrote 3 lines of it08:23
evrardjphaha08:23
evrardjpthe usual thing :)08:24
evrardjpI guess we thought doing it good08:24
evrardjpI just don't think it's good enough for merging08:24
evrardjpI am sorry08:24
openstackgerritMerged openstack/openstack-ansible-os_tempest master: update for changes needed to pass refstack  https://review.openstack.org/55020708:26
hwoaranghmm i noticed that when you pass variables to roles lke foo: "{{ foobar }}" the foobar variable is evaluated when the role is executed and not when you actually call it08:27
hwoaranginteresting feature...08:27
hwoaranglike08:27
hwoarangroles:08:27
hwoarang   - role: bar08:27
hwoarang     foo: "{{ foobar }}"08:27
hwoarangfoo is being passed as "{{ foobar }}" to role08:27
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible master: playbooks: containers-lxc-host: Set correct disk size limit for containers  https://review.openstack.org/55283608:29
*** vnogin has quit IRC08:30
PTOevrardjp: Regarding bug repport #1750841 (missing /opt) - Im running ubuntu, but the reason for the folder is missing is because the installation screwed up completely and I did a "rm -rf /opt"08:32
*** electrofelix has joined #openstack-ansible08:33
*** karthi has quit IRC08:33
evrardjpPTO: haha08:35
evrardjpthat makes sense08:35
evrardjphwoarang: I am not sure what you mean, yes we are passing the variable to the role ?08:35
PTOevrardjp: When I deleted /opt, then everything failed. It tooke me some time to figure out it was because the /opt folder was missing08:36
evrardjpnot sure what you meant by "evaluated when the role is executed" vs "not when you actually call it"08:36
hwoarangevrardjp: i mean assuming fobar=1, then you pass foo="{{ foobar }}" instead of foo=108:36
evrardjpbecause that's the same for me08:36
hwoarangthe evalutation of foobar happens in the role itself, not when you call the role from the playbook08:36
evrardjphwoarang: oh yeah08:36
openstackgerritMerged openstack/openstack-ansible-os_nova master: Simplify SELinux check  https://review.openstack.org/55020208:37
hwoarangsneaky08:37
evrardjpI guess it all depends on what's foobar in your case08:37
evrardjpbut yeah08:37
*** ygl has joined #openstack-ansible08:37
yglhi all08:38
yglwhen I run the playbook openstack-ansible setup-hosts.yml, it is saying no hosts matched08:38
evrardjphwoarang: do you expect a regression?08:38
yglwhat could be the  problem ?08:38
evrardjpbecause the scope itself wouldn't change08:38
yglcan anyone help me please08:39
evrardjpygl: your inventory is most likely the cause08:39
hwoarangevrardjp: no i was just curious08:40
yglevrardjp: should I regenerate it ?08:40
evrardjpygl: you should double check what you wrote. Linting and all that jazz.08:40
evrardjpygl: forgetting a group maybe08:40
evrardjpor erroneous spaces08:40
yglevrardjp: I have create a user_config file and listed right ips for this hosts.08:41
Taseerevrardjp: how much more work still needs to be done for congress ? I don't understand when you imply "testing the system as whole"08:41
evrardjpTaseer: well, because congress upstream don't have api testing...08:42
evrardjpTaseer: you have no choice than to do real tempest testing.08:42
evrardjpso you'd need to add all the involved components08:42
evrardjpso I guess neutron, glance, nova?08:42
evrardjp(yes we are returning to what you did earlier, but now we have an explanation for the commit message :p )08:43
Taseereven cinder ?08:43
yglevrardjp: the yaml files is ok08:43
evrardjpTaseer: not sure cinder is necessary08:43
evrardjpI guess it all depends on congress08:43
*** vnogin has joined #openstack-ansible08:44
Taseerevrardjp: okay, have you figured out the keystone thingy ? so I can update the patch and trigger zuul ?08:44
evrardjpTaseer: we don't have consensus on the approach :)08:46
evrardjpEither I let a patch pass -- which I don't find great-- and we're unblocked, or we do things the right way08:46
evrardjpI prefer doing it the right way08:47
evrardjpppl will complain.08:47
Taseerevrardjp: so I will wait...08:47
evrardjpTaseer: don't wait08:47
evrardjpcheck what's needed, add the necessary bits, modify your commit message08:48
*** vnogin has quit IRC08:48
evrardjphelp on the review of the keystone issue08:48
evrardjpthat's how we get things done08:48
Taseersure :)08:48
evrardjpygl: post them08:49
evrardjpcode don't lie08:49
evrardjp:D08:49
yglevrardjp: my openstack_hostnames_ips.yml file is empty08:49
evrardjpI will have a new tv show about that.08:49
evrardjp:D08:49
evrardjpygl: my point precisely!08:49
yglevrardjp: do u want me to post the user_config ?08:50
evrardjpopenstack_user_config and conf.d (if any)08:50
yglevrardjp: ok08:50
*** savvas has joined #openstack-ansible08:50
yglevrardjp: http://paste.openstack.org/show/700877/08:51
*** savvas has quit IRC08:55
yglevrardjp: found anything ?08:58
evrardjpygl: yes08:58
evrardjpall your hosts are over indented08:58
evrardjpthey are in global overrides08:58
evrardjpinstead of being their own group definition08:58
evrardjpunindent shared-infra_hosts and all.08:58
yglevrardjp: oh ok, i will do it and check now08:59
evrardjpwhen I mean and all, I guess you understand08:59
evrardjpygl: sure08:59
yglevrardjp: i have removed spaces for all the hosts. still the same error09:01
yglevrardjp: this is the new one http://paste.openstack.org/show/700878/09:02
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible-lxc_hosts master: tasks: Ensure machinectl quota value is reasonable  https://review.openstack.org/55284209:03
*** karthi has joined #openstack-ansible09:03
evrardjpwrong indent again ygl09:06
evrardjpgroupname: <NL, 2 spaces> hostname <NL, 2 spaces> ip:09:06
evrardjpodyssey4me: ping me when you're here09:07
evrardjpplease:)09:07
yglevrardjp: oh ok , it is very strict09:07
evrardjpofc it's yaml09:07
evrardjp:D09:08
*** savvas has joined #openstack-ansible09:08
*** holser__ has joined #openstack-ansible09:09
openstackgerritTaseer Ahmed proposed openstack/openstack-ansible master: Integrate Congress with OSA.  https://review.openstack.org/50397109:10
yglevrardjp: ah goodness me. it succeeded now. thanks a lot :)09:10
*** savvas has quit IRC09:13
*** admin0 has joined #openstack-ansible09:13
*** karthi has quit IRC09:24
*** vnogin has joined #openstack-ansible09:24
*** karthi has joined #openstack-ansible09:24
*** ygl has quit IRC09:26
*** savvas has joined #openstack-ansible09:27
*** udesale_ has joined #openstack-ansible09:27
*** udesale__ has joined #openstack-ansible09:29
*** udesale has quit IRC09:29
*** savvas has quit IRC09:31
*** udesale_ has quit IRC09:32
*** A1ve5 has quit IRC09:35
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-os_searchlight stable/queens: Point to Queens instead of master  https://review.openstack.org/54486209:35
admin0morning09:39
evrardjpmorning09:40
*** holser__ has quit IRC09:42
*** vnogin has quit IRC09:43
*** karthi has quit IRC09:43
*** holser__ has joined #openstack-ansible09:43
admin0where do i see how many more days/bugs/tests remaininng before queens get out09:44
*** savvas has joined #openstack-ansible09:45
*** savvas has quit IRC09:50
*** Sha0000 has joined #openstack-ansible09:56
*** Sha0000 has quit IRC10:00
*** savvas has joined #openstack-ansible10:03
*** savvas has quit IRC10:08
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible-galera_server stable/queens: tasks: galera_upgrade_pre: Fix conditional for apt sources  https://review.openstack.org/55286210:10
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible-galera_server stable/queens: tasks: galera_post_install: Do not secure the DB on upgrades  https://review.openstack.org/55286410:11
*** fix has joined #openstack-ansible10:13
*** A1ve5 has joined #openstack-ansible10:13
*** karthi has joined #openstack-ansible10:14
*** fix has quit IRC10:16
*** karthi has quit IRC10:22
*** karthi has joined #openstack-ansible10:23
evrardjpadmin0: it should be out soon10:24
evrardjprc2 was tagged and it will be used for release in the next day or so.10:24
evrardjpwe'll ofc still have bugs, all software has.10:24
evrardjpOkk I've put up the patches for the inclusion of systemd roles (mount and service) into OSA.10:27
evrardjpinto infra/governance10:28
evrardjpwhen these will merge we can continue on stuff like that.10:28
evrardjpin the meantime, we should be independant10:28
*** karthi has quit IRC10:29
*** karthi has joined #openstack-ansible10:32
*** vnogin has joined #openstack-ansible10:33
admin0evrardjp, thanks .. i will have a demo cluster up today to test .. will test with rc2 and report bugs10:35
*** vnogin has quit IRC10:38
*** esberglu has joined #openstack-ansible10:41
*** esberglu has quit IRC10:45
*** savvas has joined #openstack-ansible10:50
*** savvas has quit IRC10:54
*** esberglu has joined #openstack-ansible10:58
*** esberglu has quit IRC11:03
*** ianychoi has quit IRC11:07
*** ianychoi has joined #openstack-ansible11:13
*** ansmith has quit IRC11:17
*** armaan has quit IRC11:25
*** karthi has quit IRC11:28
*** chyka has joined #openstack-ansible11:30
odyssey4meevrardjp I abandoned https://review.openstack.org/552486 because it kept failng on the upgrade test. So I decided to fail forward, rather than back out.11:33
*** chyka has quit IRC11:34
evrardjpodyssey4me: oh11:34
evrardjpwhy does it fail on upgrade?11:35
*** karthi has joined #openstack-ansible11:36
evrardjpthis morning it failed due to a problem in name resolution11:36
evrardjpI submitted the patches for the inclusion. We don't need to revert the change in a-r-r, we'll just adapt them11:36
odyssey4meevrardjp it was failing during the queens deployment of keystone for some reason which I couldn't figure out11:37
evrardjpit seems to have passed11:37
evrardjpcheck at the restore: https://review.openstack.org/#/c/552486/11:37
odyssey4menot consistently11:37
evrardjpoh ok11:37
evrardjpgood to know11:37
evrardjpI see we haven't triaged stuff yesterday, if there are enough ppl today maybe we can do it today11:38
*** karthi has quit IRC11:38
*** Sha000000 has joined #openstack-ansible11:41
odyssey4mehwoarang evrardjp so the disk space issue comes down to a real disk space issue11:41
evrardjpOSA queens is released!11:42
odyssey4mein our tests, we get a rax performance 1-8 node ~ 2/3 of the time - that flavor has a small root disk and a large ephemeral additional disk11:42
evrardjp#thanks all the OSA community for making Queens happen!11:42
openstackstatusevrardjp: Added your thanks to Thanks page (https://wiki.openstack.org/wiki/Thanks)11:42
hwoarangawesome :) now we need to get more stuff in ;p11:43
odyssey4mein the bootstrap we recognise an extra disk is available, and partition it - but that extra partition isn't used for anything - the btrfs partition is on the root disk11:43
evrardjphwoarang: ofc :)11:43
evrardjpodyssey4me: that's more like it!11:43
odyssey4me:) well done to all for queens11:43
evrardjpI would have loved having at the same time removing newton and adding queens.11:44
evrardjpBut newton is not done yet.11:44
*** udesale__ has quit IRC11:45
*** ygl has joined #openstack-ansible11:52
yglcan anyone tell me where I can find the logs of my ansible playbook execution ?11:52
*** Sha000000 has quit IRC11:52
*** armaan has joined #openstack-ansible11:53
yglevrardjp: where can I find the logs of my ansible playbook execution11:53
*** vnogin has joined #openstack-ansible11:53
odyssey4meygl it outputs to your stdout, but also by default to /openstack/log/ansible-logging/ansible.log or whatever you've set ANSIBLE_LOG_PATH to11:56
yglodyssey4me: thanks for your information11:58
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible-os_cinder master: tasks: Do not configure backends and API at the same time  https://review.openstack.org/55288912:01
odyssey4meevrardjp hwoarang ok, so given the above information - how do you wish to proceed? want me to put together a patch that solves the disk space issue directly, instead of revamping the bootstrap role as done in https://review.openstack.org/#/c/552047/ ?12:05
evrardjpspotz: I like the idea of office hours. Let's discuss this.12:05
*** armaan has quit IRC12:05
evrardjpodyssey4me: I think hwoarang has a patch, or maybe I am wrong?12:05
evrardjpnew patch*12:05
hwoarangodyssey4me: i have submitted two patches12:05
odyssey4methat patch doesn't solve anything12:05
hwoarangso it's not a quota thing?12:06
*** armaan has joined #openstack-ansible12:06
odyssey4menope, it actually is a lack of disk space12:06
hwoarangok12:06
evrardjpodyssey4me: you can patch then12:06
evrardjpit seems those patches have their own purpose12:06
*** ygl has quit IRC12:07
odyssey4meI'm not saying that the patches to prevent a quota larger than the real disk doesn't have value - if that is actually an issue, we should cater to preventing that.12:07
evrardjpodyssey4me: the minimum required (not bringing the whole world with you) would be nice12:07
odyssey4meIn this case though, our failure, is actually due to a disk space issue.12:07
evrardjpodyssey4me: so the partition thing.12:07
evrardjpOk12:07
odyssey4melemme fire up an exact same flavor as the gate uses and figure out a minimal patch12:08
evrardjpodyssey4me: I don't mind having a bigger patch, but its scope should be clear, and explicit.12:08
evrardjpno hidden things :)12:08
hwoarangmaybe it's just that the commit messages needs to explain things a little bit more because it's too short right now12:09
evrardjphwoarang: it pulls at the same time the other repos and all that jazz12:10
*** armaan has quit IRC12:10
hwoarangyep12:10
evrardjpI think it's reasonable to say we can fix that without the need of a new role, not that the role isn't useful.12:10
evrardjpmore speed less haste12:11
evrardjplogan-: could you ping me when you're there? I'd like to discuss https://bugs.launchpad.net/openstack-ansible/+bug/175293212:11
openstackLaunchpad bug 1752932 in openstack-ansible "hostvars breaks on Queens" [Undecided,New] - Assigned to Jean-Philippe Evrard (jean-philippe-evrard)12:11
*** ygl has joined #openstack-ansible12:15
yglevrardjp: when I execute the infrastructure playbook , it failed saying this12:16
yglevrardjp: msg": "SSH Error: data could not be sent to remote host \"172.29.236.2\". Make sure this host can be reached over ssh12:16
yglevrardjp: it is reachable however from the deploy host12:17
evrardjpygl: a log would be necessary. Could you also run healthcheck-hosts playbook (if running master)12:18
yglevrardjp: what is the exact command to run that ?12:20
*** chyka has joined #openstack-ansible12:23
yglevrardjp: this is the log http://paste.openstack.org/show/700913/12:23
*** armaan has joined #openstack-ansible12:24
*** armaan has quit IRC12:24
PTOIs there a clever idea behind the way the /etc/hosts file generation is implemented in: https://github.com/openstack/openstack-ansible-openstack_hosts/blob/master/tasks/openstack_update_hosts_file.yml12:24
evrardjpPTO there are... and ppl can't use dns12:25
evrardjpsome ppl*12:25
odyssey4mePTO you can use the unbound service instead if you like by adding the appropriate hosts into a host group where you want the service to run12:25
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible-os_cinder master: tasks: Do not configure backends and API at the same time  https://review.openstack.org/55288912:26
*** chyka has quit IRC12:27
openstackgerritMerged openstack/openstack-ansible-os_keystone master: Revert "Convert role to use a common systemd init role"  https://review.openstack.org/55248612:27
yglevrardjp: did u get a chance to view the logs I have pasted12:27
PTOevrardjp: I would suggest that the template script openstack-host-hostfile-setup.sh.j2 would set a marker which outline a part in the host file which is ansible managed. And then replce that part if the script is re-run. Otherwise the hosts file can become inconsistent with duplicate entities - or even inconsistent12:28
*** armaan_ has joined #openstack-ansible12:29
*** hw_wutianwei has joined #openstack-ansible12:29
evrardjpygl: no I didn't get the chance to look at it12:32
yglevrardjp: I am running the setuphosts yml again to check if I could see any errors there12:32
*** dave-mccowan has joined #openstack-ansible12:32
evrardjpygl: looks like a network issue. Double check your ips are correct, and that the deploy node can reach those nodes.12:33
*** vnogin has quit IRC12:33
yglevrardjp: it is indded reaching those nodes12:34
*** sawblade6 has joined #openstack-ansible12:36
*** vnogin has joined #openstack-ansible12:38
*** karthi has joined #openstack-ansible12:45
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: bootstrap-host: Prepare disk for machinectl storage  https://review.openstack.org/55289312:45
odyssey4meevrardjp hwoarang ^ I think that should do it - testing on two profiles of machine now to verify12:46
*** odyssey4me has quit IRC12:47
*** odyssey4me has joined #openstack-ansible12:47
yglevrardjp: it is failing at this point again12:47
yglevrardjp: is it a problem of pip packages or python version issue ?12:47
yglevrardjp: this is the latest log  http://paste.openstack.org/show/700915/12:48
yglevrardjp: this is happening when I run openstack-ansible setup-hosts.yml12:49
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible-os_cinder master: tasks: Do not configure backends and API at the same time  https://review.openstack.org/55288912:50
*** armaan_ has quit IRC12:53
*** armaan has joined #openstack-ansible12:53
yglodyssey4me: hi, I need your help eith my issue please12:54
hwoarangodyssey4me: thanks it looks quite clean12:55
odyssey4meygl it would seem that the issue is lack of ssh connection, as the error says quite clearly12:56
hwoarangbut the loopback thing also happens in lxc_hosts role. i wonder if we duplicate stuff there12:56
yglodyssey4me: but the nodes are perfectly reachable form the deployment node12:56
*** ansmith has joined #openstack-ansible12:56
openstackgerritMerged openstack/openstack-ansible-os_nova master: Rearrange cell mapping tasks  https://review.openstack.org/54707212:57
yglodyssey4me: what else could be the issue ?12:58
yglodyssey4me: it is able to reach the nodes as root user12:58
odyssey4meygl I don't know, and am not in a position to dig into it for you at this time.12:59
*** SerenaFeng has joined #openstack-ansible12:59
yglevrardjp: can you help me if you dont mind12:59
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_nova stable/queens: Rearrange cell mapping tasks  https://review.openstack.org/55289813:01
logan-ygl: run osa with -vvvvv and you can see the exact ssh command that ansible is using. that command is failing. from your osa playbooks directory you could 'ansible -vvvvv hosts -m ping' and debug from there13:02
logan-evrardjp: o/13:02
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_nova stable/pike: Rearrange cell mapping tasks  https://review.openstack.org/55289913:04
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_nova stable/ocata: Rearrange cell mapping tasks  https://review.openstack.org/55290013:04
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-os_keystone master: Only try to stop services that exist  https://review.openstack.org/55101013:05
ygllogan-: how to execute this exactly13:05
evrardjplogan-: finishing something quick then I can have a look at our host vars thingy13:06
ygllogan-: i can excute this command successfully13:06
ygllogan-: ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=5 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o ServerAliveInterval=64 -o ServerAliveCountMax=1024 -o Compression=no -o TCPKeepAlive=yes -o VerifyHo13:06
ygllogan-: it is connecting, but the playbook setup-hosts is failing after this line13:07
logan-ygl: can you pastebin the full setup-hosts output while running with -vvvvv13:07
logan-evrardjp: sure np, going to get coffee13:08
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: bootstrap-host: Prepare disk for machinectl storage  https://review.openstack.org/55289313:08
ygllogan-: this is the log http://paste.openstack.org/show/700917/13:10
ygllogan-: r u able to see my log in the pastebin ?13:12
logan-ygl: yes but it is not really helpful... none of the connection debugging is there that would get outputted to the console with -vvvvv13:15
ygllogan-: just a sec I can get that13:16
ygllogan-: you can check this http://paste.openstack.org/show/700918/13:16
logan-how are you executing the playbook? what is the exact command string? are you running with sudo?13:17
*** savvas has joined #openstack-ansible13:19
ygllogan-: I am using this  "root@demo-deploy:/opt/openstack-ansible/playbooks# openstack-ansible -vvvv  setup-hosts.yml"13:19
*** vnogin has quit IRC13:20
logan-what's the output of http://paste.openstack.org/show/700919/ ?13:21
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible-os_cinder master: tasks: Do not check the load balancer uri for API status  https://review.openstack.org/55288913:22
ygllogan-: http://paste.openstack.org/show/700921/13:22
*** savvas has quit IRC13:24
evrardjpno keys?13:24
evrardjpygl: make sure your connection is fine with keys.13:24
yglevrardjp: yes it is fine13:24
evrardjpI haven't looked in details13:25
logan-looks like it succeeds there, very strange13:25
ygllogan-: found anything ?13:25
ygllogan-: yes13:25
ygllogan-: does it have to do with any pip packages or version mismatches as such13:26
evrardjpand ansible connection doesn't work ?13:26
evrardjpansible -m ping ?13:26
logan-instead of 'echo TESTING' try 'python --version'13:26
yglevrardjp: what is the exact ping command to use ?13:26
evrardjpansible -m ping <youransibleinventoryhostname>13:26
evrardjpyou need: ssh + python-json on the host to work13:27
evrardjpshould be explicit if that doesnt work and ssh work, then you know the issue.13:27
odyssey4meygl not sure I understand why you think it has to do with python packages. Have you done something out of the ordinary to make you think that?13:27
yglevrardjp: i see no output for "ansible -m ping 172.29.236.1" . it is showing warnings thats all13:27
evrardjpygl: not the IP13:27
yglodyssey4me:not so13:27
evrardjpthe ansible inventory hostname13:28
evrardjpI guess for your case it would infra113:28
evrardjpor compute1 or compute2 or log113:28
yglevrardjp: in that case also it is showing warnings but  no output13:28
evrardjpthat's not possible13:29
evrardjpit should show failure or ok13:29
yglevrardjp: so we have to resolve hostnames for those ips ?13:29
evrardjpand maybe extra warnings13:29
evrardjpno13:29
evrardjpbut it wouldn't hurt13:29
yglevrardjp: it si showing only warnings13:29
evrardjpit looks like http://paste.openstack.org/show/700915/  you have failure on gathering facts13:29
yglevrardjp: yes13:30
evrardjpcheck you have python installed on hosts.13:30
yglyers it is13:30
yglpython 2.713:30
evrardjpplease paste the output of logan- 's command13:30
yglevrardjp: u mean python version ?13:31
evrardjpAHAHAHAH13:31
evrardjpI think I know.13:31
evrardjpcould you show your inventory again?13:31
yglevrardjp: how to do it ?13:31
evrardjpI think you wrapped your ip with double quotes.13:31
*** karthi has quit IRC13:31
evrardjpwell13:31
evrardjpmaybe13:31
evrardjpI don't know13:31
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_keystone master: Add memcache setup to shibboleth2.xml  https://review.openstack.org/55042713:31
evrardjpygl: first execute logan- 's command :)13:32
yglevrardjp: which command ?13:32
evrardjpthe one with echo TESTING, replace echo TESTING with python --version13:32
logan-ie http://paste.openstack.org/show/700923/13:33
yglevrardjp: http://paste.openstack.org/show/700922/13:33
odyssey4mehmm, this looks like a good idea to me, but I'm not trusting my thought process - evrardjp can you look ? https://review.openstack.org/55271113:33
ygllogan-: http://paste.openstack.org/show/700922/13:33
yglevrardjp: logan- I am testing this on vms hosted on openstack. I enabled all tcp ports in the security groups. any more ports to open ?13:35
yglevrardjp: logan- since it is saying it failed to transfer ssh data in particular13:36
yglevrardjp: logan- but I enable port 22 however13:36
yglevrardjp: logan- any thoughts13:37
logan-everything seems fine w/ connection path. could you pastebin your openstack_user_config file13:39
ygllogan-: http://paste.openstack.org/show/700924/13:39
evrardjpodyssey4me: will have a look right away13:40
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: bootstrap-host: Prepare disk for machinectl storage  https://review.openstack.org/55289313:41
*** SerenaFeng has quit IRC13:41
ygllogan-: do I have generate inventory before executing the playbooks ?13:42
*** vnogin has joined #openstack-ansible13:44
ygllogan-: found anything ?13:44
logan-no ansible will execute osa's dynamic inventory automatically to generate the inventory.13:46
logan-dont see any obvious issues in there13:46
ygllogan-: no13:46
*** vnogin has quit IRC13:48
ygllogan-: any luck ?13:52
ygllogan-: one more thing. will ansible fail if I execute playbboks from a screen ?13:53
*** savvas has joined #openstack-ansible13:53
ygllogan-: i dont think so13:54
openstackgerritMatthew Thode proposed openstack/openstack-ansible-os_tempest stable/queens: update for changes needed to pass refstack  https://review.openstack.org/55290513:55
evrardjpjmccrory: commented on https://review.openstack.org/#/c/552711/213:55
openstackgerritMatthew Thode proposed openstack/openstack-ansible-os_tempest stable/pike: update for changes needed to pass refstack  https://review.openstack.org/55290613:55
logan-only if your screen breaks your control path to the nodes.... but since you appear to be running the ansible commands alongside the ssh commands where you verified a valid control path that probably isn't the issue13:55
prometheanfire^ since that was originally tested on pike13:55
ygllogan-: hmm13:55
*** savvas has quit IRC13:58
ygllogan-: still it is failing14:00
*** esberglu has joined #openstack-ansible14:00
openstackgerritLogan V proposed openstack/openstack-ansible master: Stop running get-ansible-role-requirements with -vvv  https://review.openstack.org/55290814:03
*** Sha000000 has joined #openstack-ansible14:03
cloudnullmornings14:04
yglcloudnull: Hi14:04
evrardjplogan-: it's time for my hot beverage now, then I guess we are good to go :)14:05
*** Sha000000 has quit IRC14:05
evrardjpmorning cloudnull14:05
evrardjpcloudnull: https://review.openstack.org/#/c/552861/14:05
ygllogan-: what does it do exactly at gathering facts stage ?14:06
cloudnullevrardjp: want to pull in https://github.com/cloudnull/ansible-systemd_networkd too ?14:07
*** savvas has joined #openstack-ansible14:07
odyssey4megood call14:08
mnasercan i get a lil bit of love for https://review.openstack.org/#/c/551718/ :(14:08
mnaseri need to backport all the way to stable/pike14:08
evrardjpcloudnull: oh yes why not14:08
evrardjpwe have three patches to edit then it should be quick14:09
mnaseris the linters job fixed btw?14:09
evrardjpmnaser: right away14:09
cloudnullmnaser: looks like -% is all that changed?14:09
*** gokhan_ has joined #openstack-ansible14:09
evrardjpmnaser: we should probably add a test in config_template to be more lenient14:10
mnasercloudnull: yes, but without that, if neutron_lbaasv2 is True, there is 2 more spaces added14:10
evrardjpthat's probably due to the change we added to allow comments14:10
mnasercloudnull: and then the first line of config becomes '  # General, applies to all host groups' rather than '# General, applies to all host groups'14:10
odyssey4memnaser so if there are blank line breaks before [DEFAULT] then something falls apart?14:10
cloudnullah. mnaser lgtm :)14:11
mnaseri think the issue is the fact there are two spaces before the comment before the firs tsection14:11
evrardjpin the commit message it's said spaces and #14:11
cloudnulland everyone else it seems :)14:11
mnaseralso, it isnt breaking in config_template also14:11
odyssey4meoh, so it's not the line breaks - it's that there are spaces before the comment #14:11
mnaserwhen i hacked around it manually, neutron wouldnt start because ConfigParser fails to parse it14:11
mnaserso its more of a configparser issue14:11
evrardjpmnaser: because empty line with comments?14:11
mnaserevrardjp: empty line with leading space with comment, before the first section14:12
evrardjpmnaser: it's worth adding as a test14:12
odyssey4meyeah, sounds like something better solved in config_template14:12
evrardjpbut we should probably have the same thing as template: a validate clause14:12
evrardjpwhich we do not, because we leverage copy module, not template, behind the scenes.14:13
*** SerenaFeng has joined #openstack-ansible14:13
mnaserbut even then i guess config_template would have to be smart enough not to render things with leading spaces too14:13
mnaserit should be fairly easy to replicate.. just use a config_template with '  # foo\n[DEFAULT]' with overrides and itll fail14:13
evrardjpmnaser: at least having something that allows that14:13
*** karthi has joined #openstack-ansible14:14
evrardjplike if you allow comments, allow empty lines with comments, and be smart with it14:14
evrardjpmnaser: yes indeed14:14
evrardjpcould you at least file a bug this way we won't forget?14:14
evrardjpI can tackle it in the future when config_template is in its own repo14:14
mnasersounds good14:15
*** savvas has quit IRC14:15
evrardjpthere is no urgency right now I'd say, with that fix included14:15
mnaserall bugs go under osa?14:15
evrardjpyeah14:15
*** electrofelix has quit IRC14:16
gokhan_hi folks, in my environment my internet connection has been down about 8 hours. after that I tried to create an instance with self service network. but I didn't reach the instance with floating ip. I also tried create new network and router and then create instance with new network. I see that I can reach instance  which is created with new network. the problem seem about routing/router. What is your thoughts about this problem and how can I solv14:17
gokhan_e this ?14:17
odyssey4mecloudnull evrardjp I have a suggestion for https://review.openstack.org/552047 - given that it was entirely rewritten, abandon that and cloudnull recreate it, but focus it on switching the bootstrap to use the new roles. Base it on https://review.openstack.org/552893 which, if we get it merged, we can backport all the way to pike - whereas the new roles really should not be used in the stable branches.14:17
mnaserhttps://bugs.launchpad.net/openstack-ansible/+bug/175582114:17
openstackLaunchpad bug 1755821 in openstack-ansible "config_template fails to parse template if it contains a comment with leading spaces" [Undecided,New]14:17
mnaservoila14:17
ygllogan-: found any thing ?14:18
evrardjpmnaser: thanks!14:19
evrardjpodyssey4me: completely agreed.14:19
logan-ygl: nope.. im not sure what the issue is sorry.14:19
logan-evrardjp: need to go afk but i posted tons of additional output on https://bugs.launchpad.net/openstack-ansible/+bug/175293214:20
openstackLaunchpad bug 1752932 in openstack-ansible "hostvars breaks on Queens" [Undecided,New] - Assigned to Jean-Philippe Evrard (jean-philippe-evrard)14:20
*** ygl has quit IRC14:20
logan-evrardjp: the last 2 pastes allow you to completely recreate the environment in either py2/py3 with just a few commands and a few minutes to sit through bootstrap-ansible14:20
odyssey4mejmccrory please validate that https://review.openstack.org/#/q/I4f7328e1aff79eb94f278999b97fbad8f44c6a3e are all right - they're backports of your patch to master14:21
evrardjpodyssey4me: cloudnull you can deal with the co-author if need be, but I think this is cleaner approach.14:21
logan-dang just noticed paste.openstack.org truncated the output14:22
evrardjplogan-: thanks.14:23
evrardjpcloudnull: we shouldn't add a new deliverable into stable branches, that's a nightmare later. We can now though.14:23
odyssey4meevrardjp with thanks to jrosser we save just a little time for new builds with https://review.openstack.org/55101014:24
evrardjpI'd just prefer if we keep the two separate14:24
cloudnullodyssey4me: ++14:25
evrardjpodyssey4me: that is nice14:26
evrardjplet me check if that interface is stable on the ansible side14:26
odyssey4mecloudnull happened upon the same and pushed up a similar patch yesterday, but jrosser got there first ;)14:26
* cloudnull abandoned mine14:27
* jrosser waves o/14:27
cloudnullanyone want to give this a nudge forward https://review.openstack.org/#/c/552643/114:28
evrardjpjrosser: you love your systemd do you?14:28
evrardjp:)14:28
odyssey4mecloudnull that's not necessary any more - the systemd role inclusion was reverted14:29
evrardjpwow we are depending on systemd being stable for this to not fail.14:29
odyssey4meevrardjp We are depending on systemd, whether we like it or not.14:30
cloudnullwe've been depending on systemd whether we like it or not :)14:30
evrardjpodyssey4me: yeah, I am fine with depending on systemd. Depending on a systemd stable interface with is relatively not user related is more tricky14:31
evrardjpdepending on a systemd interface (that's not user related) being stable* is another thing.14:31
evrardjpI think we are good14:31
evrardjpwe'll see14:31
evrardjpchances are high it will stay as is.14:31
Bico_FinoMorning.14:32
cloudnullo/ Bico_Fino14:32
Bico_FinoWhat's the best way to inject an entry on /etc/hosts for every container?14:32
evrardjpmorning Bico_Fino14:32
cloudnullodyssey4me: so do i revert the revert w/ those changes?14:33
cloudnullor just abandon that effort for now14:34
evrardjpcloudnull: I planned to revert the revert14:35
evrardjpjust take your time, and use the new repo14:35
evrardjpcloudnull: why did you abandon this? https://review.openstack.org/#/c/552643/114:37
evrardjpoh it all gonna be in your role maybe?14:37
cloudnullno.14:37
cloudnullit was a dependent patch14:37
*** savvas has joined #openstack-ansible14:37
cloudnulland in merge conflict after the revert14:38
cloudnullso i'll just incorporate those changes into the main one14:38
cloudnullit was an unforseen issue when using py3 in the integrated gate.14:39
evrardjpit's a good patch14:40
evrardjpneeds to be backported to Q I guess14:40
cloudnullonly if we backport the systemd things14:41
cloudnull** systemd_role things14:41
evrardjpoh.14:41
evrardjpOk.14:41
evrardjpthen it won't be possible :(14:42
*** savvas has quit IRC14:42
evrardjpManual stuff I guess.14:42
*** sar has quit IRC14:43
*** marwel has joined #openstack-ansible14:43
odyssey4meevrardjp the revert of the revert will have to include that patch, otherwise it won't work - so perhaps better to propose a whole new combined patch14:44
openstackgerritMerged openstack/openstack-ansible-galera_server master: Updated from global requirements  https://review.openstack.org/55219714:45
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-os_keystone master: Convert role to use a common systemd init role  https://review.openstack.org/55291414:48
cloudnullthere's the new patch. I'll update it and the other ones I had in flight when the roles are imported14:48
odyssey4mecloudnull perhaps mark them all -w then while you continue working through the roles?14:49
cloudnulldone.14:56
cloudnullshould we revert all of the service role inclusions too ?14:58
cloudnullor will we just make a new set of prs once the roles are in the os namespace ?14:58
*** chyka has joined #openstack-ansible14:59
evrardjpodyssey4me: yes you're right.14:59
evrardjpcloudnull just a new set of PRs, it's not hurting to just download stuff15:00
*** Sha000000 has joined #openstack-ansible15:00
odyssey4mecloudnull once the roles are imported I can revise them - I don't think they're doing any harm right now15:00
evrardjpfor role tests15:00
evrardjpyeah no harm right now on each of the repos' tests.15:00
*** Sha000000 has quit IRC15:01
*** vnogin has joined #openstack-ansible15:03
cloudnullto that end, will we need to pull in the following roles into the OS namesapce http://paste.openstack.org/show/700951 - I assume this comment applies to all of our non-os namespace roles http://eavesdrop.openstack.org/irclogs/%23openstack-ansible/latest.log.html#t2018-03-14T08:20:18 ?15:05
*** bitshifter has joined #openstack-ansible15:05
*** SerenaFeng has quit IRC15:06
bitshifter@odyssey4me Do you know where i might be able to find out what variables are required to be defined in user_variables.yml for the desginate install? I assume i can scrap most of them aside from things i want to specify15:11
openstackgerritMerged openstack/openstack-ansible-os_searchlight stable/queens: Point to Queens instead of master  https://review.openstack.org/54486215:11
bitshifteris it purely just the 4 entries listed here? https://docs.openstack.org/openstack-ansible-os_designate/latest/15:11
evrardjpcloudnull: the idea was to: re-use what we can for infra. We can re-use systemd things, but it's a dependency to all openstack components, so the question is asked: How will you do releasing if we depend on your thing?15:19
evrardjpon your paste, there is only one that I think should be included15:19
evrardjphttps://github.com/logan2211/ansible-haproxy-endpoints15:19
evrardjpthat one is used in all our roles as a mandatory patch15:19
evrardjppath*15:19
evrardjpthe rest is skippable15:19
evrardjpcloudnull: what's your point though? How do you plan to use this in your namespace, if we don't include it?15:20
evrardjpDo you want us to rely on this, and you'll tag appropriately?15:20
evrardjpYou see all this discussion should not have happened on the channel, but in a spec instead15:21
evrardjpto keep an easy record15:21
odyssey4mebitshifter it shouldn't need anything if you're using the integrated build - the passwords are already in user_secrets.yml - you'll only need to populate the config entries to inform designate where your dns service is15:23
evrardjpcloudnull: should I write the spec for you?15:24
openstackgerritMerged openstack/openstack-ansible-tests stable/queens: Use OpenStack-Infra mirrors for tests  https://review.openstack.org/54821615:26
bitshifter@odyssey4me - You're talking about pools.yml right?15:29
openstackgerritMerged openstack/openstack-ansible-os_keystone master: Only try to stop services that exist  https://review.openstack.org/55101015:29
odyssey4mebitshifter it's been a very long time since I've looked at it - and I'm in a conf call right now... my advice is not to over think it :) try, fail, fix, repeat ;)15:30
bitshifterall good. Thanks for the advice thus far :)15:31
*** Sha000000 has joined #openstack-ansible15:32
*** esberglu has quit IRC15:33
openstackgerritMerged openstack/openstack-ansible-galera_server stable/queens: tasks: galera_post_install: Do not secure the DB on upgrades  https://review.openstack.org/55286415:34
openstackgerritMerged openstack/openstack-ansible-galera_server stable/queens: tasks: galera_upgrade_pre: Fix conditional for apt sources  https://review.openstack.org/55286215:34
openstackgerritMerged openstack/openstack-ansible-tests stable/queens: test-prepare-host: Switch package state to 'present' on openSUSE  https://review.openstack.org/55192315:38
openstackgerritMerged openstack/openstack-ansible-specs master: Add spec for centralized nginx  https://review.openstack.org/52481215:41
*** joseg has joined #openstack-ansible15:42
*** joseg has left #openstack-ansible15:42
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_nova stable/queens: Rearrange cell mapping tasks  https://review.openstack.org/55289815:42
odyssey4meevrardjp updated per your request15:43
*** joseg has joined #openstack-ansible15:43
evrardjpthanks15:43
evrardjpcloudnull: I'd like we discuss this with everyone15:44
*** throwsb1 has joined #openstack-ansible15:46
*** esberglu has joined #openstack-ansible15:48
odyssey4mejmccrory PS2 uses include ;)15:48
jmccroryhah yeah saw the update right after i commented of course15:48
jmccrorythink it's just weird 2.5 behavior, should be fine with queens and back15:49
evrardjpjmccrory: mmm what do you mean there?15:50
*** ThomasS has quit IRC15:50
cloudnullevrardjp: in this mornings scroll back it was said that we have to stop work on the systemd include role work because the roles were not in the OS namespace and is risky. However IMHO these roles are no more risky than the other external roles we're carrying all of which have the same risk, should they be deleted or messed with all deployments would stop dead in their tracks.15:50
cloudnullI'm fine with them being in the OS namespace, in fact I'd prefer that.15:51
evrardjpok15:51
evrardjpcloudnull: you have to remember we talked about including many of those roles into OSA in the past :)15:51
cloudnullbut the various blocks, workarounds, and reverts seem like we're not applying the same logic everywhere15:51
evrardjpwe are not applying the same logic everywhere, that's true15:52
evrardjpinputs are different15:52
cloudnullthey are not15:52
cloudnullthe inputs are the same .15:52
evrardjpthat's where we disagree :)15:53
evrardjpfor deploying nova you don't care about keepalived.15:53
*** Sha000000 has quit IRC15:53
cloudnullif you were to delete that role all deployments of osa would break15:53
evrardjpfor deploy a complete AIO you don't care about keepalived, resolvconf role15:53
cloudnullregardless if keepalived was even used15:54
evrardjpcloudnull: that's true. But an empty repo as a replacement would still "pass"15:54
cloudnullor haproxy for that matter15:54
evrardjpit wouldn't do the same thing.15:54
evrardjphaproxy-endpoints-management is, as I said, the only one that's very risky15:54
evrardjpI'd like to see what we can do there15:55
cloudnullansible-sshd15:55
cloudnullno aio would build15:55
evrardjpthat's only for aio15:55
jmccroryevrardjp with 2.5, using an include or include_tasks there wasn't delegating correctly. some combination of assigning the delegated host through a variable and using a loop maybe?15:55
evrardjpand we can probably skip it too15:55
evrardjpjmccrory: I guess that doesn't matter for queens?15:56
jmccrorydon't think so15:56
evrardjpok15:56
evrardjpwe need to care about what's happening there in 2.515:56
evrardjpin master I meant15:56
admin0out of curiosity, what is a 2.5 ?15:57
evrardjpadmin0: it's ansible 2.5 we are talking about15:57
evrardjpout soon15:57
admin0i thought 3 was the latest :)15:57
evrardjpno15:58
evrardjpnice try!15:58
evrardjpcloudnull: what are the repos we need to bring? did you name them all systemd- on your github?16:00
*** SerenaFeng has joined #openstack-ansible16:01
cloudnullyup16:02
cloudnullansible-systemd_*16:02
hwoarangjmccrory: could you please have a look on https://review.openstack.org/552889 ?16:03
*** PTO has quit IRC16:03
*** flemingo has joined #openstack-ansible16:04
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-memcached_server stable/newton: Add handler when systemd is used  https://review.openstack.org/55294216:04
jmccroryhwoarang lgtm16:05
evrardjpcloudnull: ok next question: for me we should add that into our deliverable, but I'd like this to be re-used by other ppl.16:07
hwoarangthank you jmccrory16:07
evrardjpcloudnull: Do you think this should follow openstack branching?16:07
evrardjpor not?16:07
cloudnullno, it should not follow openstack branching .16:07
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible master: inventory: all: Switch package state to 'present' on openSUSE  https://review.openstack.org/55294416:07
evrardjpok16:08
evrardjpso independant releasing?16:08
evrardjpsounds good16:08
cloudnull++16:08
*** masber has quit IRC16:10
evrardjpI suppose in the future you can even include your library into that, if you want to use more modules for managing these things.16:12
evrardjpif we want*16:12
evrardjplogan-: cloudnull has raised a valid concern here, what about the haproxy-endpoints-manage?16:14
cloudnullI really am OK with those roles being in a separate place. they're no different than any project dependency, like pip16:15
cloudnullmy frustration is that external by someone else seems to be ok, external by me is cause for a work stopage16:16
evrardjpcloudnull: I will not stand you being a victim of my brain! I will defend your honor!16:17
*** radeks has quit IRC16:18
*** flemingo_ has joined #openstack-ansible16:18
evrardjpcloudnull: don't worry if I am like that, I just think that commit wasn't good enough for merging -- It might be only the commit message, it might be the inclusion in the same patch, but anyway, reading the patch commit message and what it did was two different things16:19
evrardjpthat's not normal16:19
evrardjpnow the inclusion is more inclusion into common roles, outside OSA namespacing, and that's a good opportunity to share. If we leave it into a separate github thing, it will never have the visibility this can have16:20
*** udesale has joined #openstack-ansible16:20
*** flemingo has quit IRC16:21
cloudnullTotally agree. however the import of those roles into the os namespace could happen in parallel no in serial.16:21
*** exodusftw has quit IRC16:22
evrardjpI guess all I needed to know is "what's going on, where are we headed with that"16:22
evrardjpit's now clearer16:23
evrardjpI am confused about the order too16:23
evrardjpI was confused about the order16:23
evrardjpI'd have thought PoC > spec > import role > modify roles to use the imported roles16:24
odyssey4mepersonally, my only hassle was that it merged in keystone and broke everything else - given that we're early in the cycle, and the intent to import the role into the namespace was clear, I had no objections to it going ahead as-is16:25
*** exodusftw has joined #openstack-ansible16:25
evrardjpI didn't see the value of adding more patches to now move to a new location and stuff16:25
*** udesale has quit IRC16:26
evrardjpalso adding the role into the namespace sooner rather than later brings more visibility into the process, respecting the idea of the 4 opens.16:26
*** karthi has quit IRC16:27
evrardjpI guess it's all done now16:27
cloudnullits done now. I'll wait for the roles to hit the os namespace and continue marching on16:27
*** masber has joined #openstack-ansible16:29
*** exodusftw has quit IRC16:30
*** SerenaFe_ has joined #openstack-ansible16:33
*** exodusftw has joined #openstack-ansible16:33
spotzevrardjp: I'm debugging something but around if you need me16:34
*** karthi has joined #openstack-ansible16:35
*** SerenaFeng has quit IRC16:35
evrardjpjust wanted to discuss office hours16:38
*** vnogin has quit IRC16:40
*** vnogin has joined #openstack-ansible16:42
*** vnogin has quit IRC16:43
*** karthi has quit IRC16:44
*** masber has quit IRC16:44
spotzevrardjp: okie16:47
evrardjpspotz: so what are the responsibilities/duties/organizational aspects of this?16:47
evrardjpPros/cons16:47
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-memcached_server stable/newton: Add handler when systemd is used  https://review.openstack.org/55294216:49
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible master: zuul: Make openSUSE a voting job  https://review.openstack.org/55297716:49
spotzevrardjp: It seems the origin of it was to help squash bugs faster by having folks available to answer questions and help new contributers. Now we're actually really good at helping folks, so in our case it might just be hey folks are available between these hours as we have atleast 3-5 folks who work on OSA full time?16:50
*** masber has joined #openstack-ansible16:51
odyssey4mespotz evrardjp most who join us in channel find that we're available to help most times :)16:52
odyssey4meI guess instituting office hours would have value in designating a time where we try to not be busy with other things and try to be ready to help anyone with any questions16:52
*** SerenaFe_ has quit IRC16:52
spotzodyssey4me evrardjp Yeah I think except for during the holidays and maybe late night we have folks about16:53
odyssey4meit all depends on how well we're sleeping ;)16:53
spotzAnd maybe its just during holidays we communicate folks will be out but you can find someone between 4-5 or something16:53
Taseerevrardjp: the congress tempest test failed. Do I need to make it pass ?16:54
*** radeks has joined #openstack-ansible16:54
*** armaan has quit IRC16:59
evrardjpTaseer: what's the cause?17:02
evrardjpspotz: odyssey4me so my understanding is that it's not really needed ?17:02
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_glance master: [WIP] Use a common python build role for source builds  https://review.openstack.org/55134417:03
spotzevrardjp odyssey4me - Or needed during holidays?17:03
*** jamesdenton has joined #openstack-ansible17:04
Taseerevrardjp: https://hastebin.com/ajilabawag.tex17:04
*** gkadam has quit IRC17:05
openstackgerritMerged openstack/openstack-ansible-os_keystone master: Avoid conflicting delegation with authorized_keys  https://review.openstack.org/55271117:05
*** flemingo has joined #openstack-ansible17:06
*** radeks has quit IRC17:06
*** radeks has joined #openstack-ansible17:07
evrardjpspotz: not sure to understand the message17:07
odyssey4meevrardjp spotz If it is instituted, then it should be done with the explicit purpose of ensuring that newcomers and other projects are aware that this is a good time to interact. It will need to be a time when the PTL and some supporting contributors are always ready and able to help. We would, during that time, set aside whatever we're busy with and specifically be ready to answer questions.17:08
spotzevrardjp: Basically if everyone or most everyone is going to be out communicate to the lists17:08
*** radeks has quit IRC17:08
odyssey4meConfident contributors, and even some newcomers come into the channel and ask any time - but in this case it would be a specific time where we make ourselves available and are not distracted.17:08
*** radek_ has joined #openstack-ansible17:09
*** radek_ has quit IRC17:10
*** flemingo has quit IRC17:10
*** flemingo_ has quit IRC17:10
*** radeks has joined #openstack-ansible17:10
odyssey4meI'm not advocating that we do it - I'm just suggesting that if we do, we should scope it properly.17:10
evrardjpodyssey4me: understood. It's all about setting expectations. We don't want to set office hours and have no one in it.17:13
spotzevrardjp odyssey4me correct17:13
odyssey4meyep, we also don't want it to be like any other time in the channel17:13
*** armaan has joined #openstack-ansible17:13
openstackgerritJimmy McCrory proposed openstack/openstack-ansible-os_keystone stable/queens: Avoid conflicting delegation with authorized_keys  https://review.openstack.org/55298317:13
evrardjpit could change the face of our channel too, here it's very welcoming. If we say: Could you come back when it's office hours, it kinda creates a frozen barrier between people17:13
spotzIt'd be interesting to see if some folks show up during bug triage because they think that'ss the only time to find us17:13
spotzevrardjp: I think if folks are here we should ever do that unless it's just me and I can't answer:)17:14
spotzever=never17:14
odyssey4meevrardjp spotz yeah, quite honestly I think we're welcoming already, and very available - I'm not sure setting office hours adds much value17:14
evrardjpspotz: no what we see the most is ppl asking question, anytime, and during bug triage, we just ask to delay the question a little... And we generally answer even during triage :)17:14
odyssey4meif anything, I would think that perhaps ML notification of decisions taken or things to be communicated might be something worthwhile...17:15
evrardjpyeah basically throwing a list of discussions and decisions would be nice. I am planning to do this manually but it could even be a bot doing it.17:17
*** pcaruana has quit IRC17:18
evrardjpor we are just very open with the startmeeting.17:18
evrardjp:)17:18
odyssey4mewell now, using a bot with a wiki page to record things could be something to do... like infra has a bot to record events/actions17:20
odyssey4methat said, I'm not sure we have all that many of those things going on17:20
odyssey4meat this point I'd far rather see some sort of tooling for bug trends17:21
evrardjpyeah that's better indeed.17:21
evrardjpwell I can summarize this for you. Up to the right!17:21
evrardjp(open bugs)17:21
odyssey4meit'd be super-nice to be able to see the flow of bugs - incoming, throughput, etc17:21
hwoarangyep^ +117:22
evrardjpyup I had asked bernardo to have that. So basically we can use something Ian did in the past.17:22
*** epalper has quit IRC17:22
evrardjpI haven't the code yet, but I can maybe ask17:22
odyssey4meI found this, but it's likely dated and unmaintained: https://github.com/Mirantis/launchpad-reports-summary17:23
admin0i think bugs should be addressed in revere date order in the triage :)  the one who reported first has to keep waiting because the time is up till it reaches his part17:23
evrardjpadmin0: the triage is checking newest bugs17:24
evrardjpit's not about addressing them17:24
evrardjpfixing bugs is another story, and is probably why we have the triage in the first place17:25
* odyssey4me goes to infra to discuss :)17:25
evrardjpbut I agree, we sometimes lack time to triage everything17:25
evrardjpodyssey4me: great :)17:25
admin0yes what i have seen is we do this weekly .. so the triage of the bug reported 1 hour after the triage never gets his day because next week in the next triage, we address it from the latest one17:25
odyssey4meadmin0 yep, all we can do generally is try to help triage the bug... fixing it is up to the one who is pained the most by the bug17:26
admin0so if you have to address something, open it like 1 hr before the triage and it gets the first attention :D17:26
admin0pro tip of the day :D17:26
evrardjpadmin0: haha, you can also be part of the triage and tell you'd like to discuss a bug17:27
evrardjpwe are open17:27
admin0i do that, but wait in line :)17:27
evrardjp:)17:28
admin0well, general remark .. sometimes i feel if we should ( when we have time ) pre-triage the triage and give some importance .. when the list is big.. something does not get addressed17:28
evrardjpadmin0: I 100% agree there.17:28
evrardjpwe shouldn't wait for the triage to triage17:28
spotzUnless it's one of the cores though folks shouldn't self-triage cause you know all my stuff is #1:)17:29
evrardjpit's not like a patch to merge code I guess, I'd be enclined for ppl to self-triage, but it would be weird to self-triage to high/critical and not fix it yourself17:30
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-memcached_server stable/newton: Add handler when systemd is used  https://review.openstack.org/55294217:31
evrardjpcloudnull: on the nick of time, there is a EOL request that was sent yesterday.17:31
evrardjplet's merge this real quick :)17:32
cloudnullif its possible17:32
cloudnullI told my org that it might not be17:32
evrardjpwell it makes sense to merge it17:32
evrardjplet's just see how it goes with EOLing17:32
cloudnullso if we can make it go, great! else meg.17:32
cloudnull**meh17:32
evrardjpI have to speak with tonyb tomorrow17:33
evrardjpthanks for the insights spotz admin0 odyssey4me17:34
spotzThat's what I'm here for:)17:34
odyssey4mecloudnull evrardjp if it merges, the sha will always exist - even if it's not at the eol tag :p17:36
jrosserwhere does container_networks get built? https://github.com/openstack/openstack-ansible-lxc_container_create/blob/stable/pike/tasks/lxc_container_config.yml#L12617:36
cloudnullodyssey4me: ++ :)17:37
cloudnulljrosser: that happens within the dynamic inventory17:37
cloudnullif you crack open the openstack-inventory.json file the entry "container_networks" is under the hostname17:38
*** hamza21 has joined #openstack-ansible17:41
jrossercloudnull: ah ok i see that. if ive adjusted container_mtu in my openstack_user_config.yml, how should a propagate that change through?17:42
cloudnullin the json you should see the mtu value.17:42
cloudnullonce you run the dynamic-inventory.py script, or any playbook run17:43
odyssey4mejrosser I'm guessing you need https://review.openstack.org/528357 merged?17:43
jrosserodyssey4me: oh right! that might explain a thing or two17:44
odyssey4methat looks like it fixes the issue you're seeing17:44
jrosseryeah i've tried to bump my network node container_mtu up from 1500 and it's not making it beyong provider_networks17:45
evrardjpjrosser: oh good catch on this patch17:47
evrardjpI always forget it17:47
openstackgerritJimmy McCrory proposed openstack/openstack-ansible master: [TEST] Update Ansible to 2.5.0rc2  https://review.openstack.org/50181417:48
evrardjpand it's an important one for upgrades  :'(17:48
jrosserfeels like i'm on a bit of a roll tripping over these atm17:49
*** bitshifter has quit IRC17:49
openstackgerritMerged openstack/openstack-ansible stable/queens: scripts: scripts-library.sh: Fix dstat background process command  https://review.openstack.org/55200017:51
*** sar has joined #openstack-ansible17:52
jrosseralso, if i delete a container, blow away its facts, rebuilt it with --limit then the rebuild fails17:53
jrosserwithout the --limit it is ok and runs through17:53
odyssey4mejrosser include lxc_hosts in the limit17:54
odyssey4methe container build needs facts from its host17:54
jrosseris it ok to just include the actual host for that container?17:54
jrosserif i know which it is17:54
odyssey4mesure17:55
armaanlogan-: Hello! We are upgrading ocata to pike and the ceph client role tried to install ceph-mon, ceph-mgr in the compute node?17:56
openstackgerritAndy McCrae proposed openstack/openstack-ansible-memcached_server stable/newton: Add handler when systemd is used  https://review.openstack.org/55294218:02
openstackgerritMerged openstack/openstack-ansible-os_nova stable/pike: Rearrange cell mapping tasks  https://review.openstack.org/55289918:03
*** poopcat has joined #openstack-ansible18:04
openstackgerritMerged openstack/openstack-ansible master: bootstrap-host: Prepare disk for machinectl storage  https://review.openstack.org/55289318:09
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/queens: bootstrap-host: Prepare disk for machinectl storage  https://review.openstack.org/55301018:15
evrardjpandymccr_: listen doesn't work in 2.2?18:16
andymccr_nope :(18:16
odyssey4menewton is 2.1, listen was introduced in 2.2 IIRC18:16
andymccr_^ this18:16
andymccr_it basically just errors out saying you cant do it, but the other solution is fine. since the tasks are mutually exclusive18:17
odyssey4meevrardjp cloudnull pike uses overlayfs, so no need to port https://review.openstack.org/#/q/Icadbcfd420bbe7c14a04606790cab09a1dce03ce back to it - it's master/queens only18:17
odyssey4mefor master, next step is to evolve to using the systemd/networkd roles - we leave queens alone18:18
evrardjpodyssey4me: andymccr_ haha my bad18:19
andymccr_meh nah i thought it would work. keeping track of all the changes in ansible going back to newton is not something id write down as a strength of mine :D18:20
odyssey4mesomething else I learned is that 2.2 didn't use action plugins as much as 2.3 - so working on ocata sucks a bit too18:20
odyssey4meso yeah, trying to track many ansible minors is quite a pain18:21
*** radeks has quit IRC18:24
odyssey4meI'm out for the night - cheerio all!18:25
*** holser__ has quit IRC18:27
*** flemingo_ has joined #openstack-ansible18:28
evrardjpodyssey4me: have a good night18:29
*** Sha0000 has joined #openstack-ansible18:30
*** yifei has quit IRC18:36
*** yifei has joined #openstack-ansible18:36
*** shardy has quit IRC18:48
openstackgerritMerged openstack/openstack-ansible-memcached_server stable/newton: Add handler when systemd is used  https://review.openstack.org/55294218:49
*** idlemind has joined #openstack-ansible19:14
idlemindhttp://pimpmylog.com/ ... would it be silly to add that to the stack (optionally) to make rsyslog viewing a little more pleasurable (at least in development clouds)19:15
idlemindor what that be something we'd be responsible to do outside of osa19:15
*** esberglu has quit IRC19:20
*** esberglu has joined #openstack-ansible19:23
*** savvas_ has joined #openstack-ansible19:25
openstackgerritMerged openstack/openstack-ansible-os_keystone stable/queens: Avoid conflicting delegation with authorized_keys  https://review.openstack.org/55298319:26
*** savvas_ has quit IRC19:29
*** savvas_ has joined #openstack-ansible19:39
*** flemingo_ has quit IRC19:42
*** savvas_ has quit IRC19:43
*** savvas_ has joined #openstack-ansible19:48
*** savvas_ has quit IRC19:56
*** savvas_ has joined #openstack-ansible20:04
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible master: Add container journal linking  https://review.openstack.org/54451820:10
*** savvas_ has quit IRC20:14
*** hw_wutianwei has quit IRC20:31
*** hw_wutianwei has joined #openstack-ansible20:32
*** ansmith has quit IRC20:36
*** hamza21 has quit IRC20:37
*** d3n14l has joined #openstack-ansible20:39
openstackgerritMerged openstack/openstack-ansible-os_nova stable/ocata: Rearrange cell mapping tasks  https://review.openstack.org/55290020:41
*** idlemind_ has joined #openstack-ansible20:44
*** idlemind_ has quit IRC20:44
*** idlemind has quit IRC20:44
*** esberglu has quit IRC20:52
*** flemingo has joined #openstack-ansible20:53
*** _jbadiapa has joined #openstack-ansible20:54
*** esberglu_ has joined #openstack-ansible20:55
*** esberglu_ has quit IRC20:56
*** esberglu_ has joined #openstack-ansible20:56
*** jbadiapa_ has quit IRC20:57
*** throwsb1 has quit IRC21:00
*** savvas_ has joined #openstack-ansible21:01
*** TxGirlGeek has joined #openstack-ansible21:02
*** TxGirlGeek has quit IRC21:05
*** savvas_ has quit IRC21:06
*** TxGirlGeek has joined #openstack-ansible21:07
*** TxGirlGeek has quit IRC21:09
*** sar has quit IRC21:09
*** jra has joined #openstack-ansible21:14
jraSo we've built a dev deployment on pike from OSA, and things mostly work - except: API requests will intermittently time out to the requestor, causing widespread "slowness" and sometimes outright failures in the clients (Horizon and CLI).21:17
jraWhat's odd here is that in the logs from the OpenStack API components, the requests are seen, and successfully returned within sub-second time21:17
*** geb has quit IRC21:17
jraBut the client never seems to get the response21:18
jraThe only notable item in the logs is a recurring "SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected)" message that the failed requests seem to trigger21:18
*** TxGirlGeek has joined #openstack-ansible21:19
jraWhich I take to mean that the API service tried to return data, but its connection to its client (which in the OSA-deployed case is HAproxy) went away21:19
jraI fully acknowledge this may not necessarily be in any way related to OSA, but I'm just fresh out of ideas at this point. Anybody have any suggestions?21:20
*** geb has joined #openstack-ansible21:21
admin0jra21:22
admin0disable 2 haproxy/keepalive21:22
admin0keep only 1 running21:22
admin0see if the issue still comes21:22
admin0if its multi node21:22
jraadmin0: it is multinode; I'll give that a try21:22
*** TxGirlGeek has quit IRC21:25
*** TxGirlGeek has joined #openstack-ansible21:25
*** dave-mccowan has quit IRC21:26
jraadmin0 Well, with only one of the two nodes up the problem persists; with only the other of the two nodes up, the logs are flooded with "can't connect to DB" and "connection refused" messages21:30
jraso it sure seems like something's not well in HAproxy land21:30
*** savvas has joined #openstack-ansible21:31
*** dave-mccowan has joined #openstack-ansible21:31
admin0did the keepalive ip shifted properly to the one funcitonal ?21:33
*** ansmith has joined #openstack-ansible21:33
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible master: Convert role to use a common systemd role(s)  https://review.openstack.org/55204721:34
*** savvas has quit IRC21:35
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible master: Convert role to use a common systemd role(s)  https://review.openstack.org/55204721:37
evrardjpcloudnull: is this in?21:38
evrardjpthe newton thing21:38
*** esberglu_ has quit IRC21:38
evrardjptony sent ayay21:39
evrardjpwoops21:39
evrardjpyay it's included21:39
evrardjplet's phase out those things!21:39
cloudnull++21:39
cloudnulltag it all eol21:39
cloudnullevrardjp: no the roles are not in quite yet21:40
cloudnullat least not that I've seen21:40
cloudnullhowever I'm just marching along with the changes and will rebase on the integrated roles when possible21:40
evrardjpyeah fine for me21:40
jraadmin0 it looks like it, yes21:44
jraadmin0 the connection refused errors only lasted a few seconds, so I guess that was just failover?21:45
admin0yep21:45
jraunfortunately the SIGPIPE issue persists, regardless of node21:45
admin0can u do the operation from the util node and stil get this error ?21:45
evrardjpor I think it's fine21:46
evrardjpI'm off to bed now21:46
evrardjpttyl everyone!21:46
cloudnulllate evrardjp21:46
cloudnull**later21:46
admin0see ya21:47
d3n14lHaving a hard time debugging neutron plugins. Is there any known issue with loading neutron plugin drivers? Getting stevedore.named: Coudl not load … for lbaas and vpnaas on a fresh pike install.21:47
admin0d3n14l, how is it named ?21:48
admin0the driver names ?21:48
cloudnulld3n14l: is that something you're setting ? or from a stock install ?21:48
admin0i have lbaas as: neutron_lbaas.services.loadbalancer.plugin.LoadBalancerPluginv221:49
d3n14lvpn_device_driver = neutron_vpnaas.services.vpn.device_drivers.strongswan_ipsec.StrongSwanDriver21:49
admin0firewall_v2 does not work .. not sure what the correct thing is for firewall21:49
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible master: Converge distro interfaces to systemd-networkd  https://review.openstack.org/55272121:49
d3n14lFor lbaas i have neutron_lbaas.drivers.haproxy.namespace_driver.HaproxyNSDriver21:50
jrossercloudnull: do you have a moment for a question about provider_networks?21:51
cloudnulld3n14l: I think that would be "neutron_lbaas.services.loadbalancer.plugin.LoadBalancerPluginv2"21:52
cloudnulli thought neutron removed the NSDriver in pike ?21:52
*** esberglu has joined #openstack-ansible21:52
cloudnulljrosser: sure21:52
*** savvas has joined #openstack-ansible21:53
jrossercool, i was looking at this https://docs.openstack.org/project-deploy-guide/openstack-ansible/queens/app-config-pod.html21:53
jrosserits reasonably similar to my deployment. the provider_networks section of the user config is very verbose21:54
jrosseri was wondering if more sections are needed if you added further pods that only consisted of compute nodes, i.e there are no container hosts in them21:54
cloudnullyes. that config (https://github.com/openstack/openstack-ansible/blob/master/etc/openstack_deploy/openstack_user_config.yml.pod.example) provided by jmccrory21:54
cloudnulljrosser: I wouldn't think so21:55
cloudnullthe compute hosts are just a name and IP which have little, if anything to do with the provider_networks21:55
jrosserclearly some of it is very much related to wiring the containers, but i was concerned that there might be ml2 driver bits and pieces there21:55
jrosseri.e which is the provider networks mappings21:56
cloudnullI guess you'd have to have something that plumbed the neutron agent into the relevent network on the host21:56
cloudnulltypically done via host_bind override21:56
*** Matias has quit IRC21:57
cloudnullbut even that, i'd imagine, would be relatively uniform across the pods21:57
*** savvas has quit IRC21:57
cloudnullhttps://github.com/openstack/openstack-ansible/blob/master/etc/openstack_deploy/openstack_user_config.yml.pod.example#L182-L19021:58
jrosseryes indeed. perhaps all the compute nodes can be targetted with reference_group and just the host bind set21:58
d3n14lcloudnull: sorry - i was meaning the device_driver setting in lbaas_agent.ini21:58
d3n14lThats triggering the stevedore.named error.21:59
*** Matias has joined #openstack-ansible21:59
cloudnulljrosser: that might make some sense22:00
jrossercloudnull: yeah, for me its potentially very simple as the pods are compute only22:00
jrosserthe container hosts all have their own little world which is seperate22:01
jrosserand are done that way becasue the networking to the container hosts is done more resiliently than to the compute hosts22:02
cloudnullthat makes a lot sense in this case.22:02
cloudnullmaybe there should be a general purpose compute network group?22:02
cloudnullmaybe a child to the neutron_lxb/ovs agent groups ?22:03
*** chyka has quit IRC22:03
*** Matias has quit IRC22:04
jrosseri've got 3 groups to consider, all the computes, all the network nodes, all the various infra22:05
jrosserand they're all uniform within each group, but vary between the groups22:05
cloudnulld3n14l: looking at https://docs.openstack.org/ocata/config-reference/networking/lbaas.html22:10
cloudnullit looks like the device driver is set correctly22:10
cloudnullsadly it looks like the lbaasv2 agent is no longer documented in pike +22:11
cloudnullthey only reference octavia22:11
cloudnulljrosser: seems the most sensible approach is to get the env.d files in the right spot for your pod config22:13
cloudnullim not quite sure how best to attack this otherwise22:14
jraadmin0 Yes, my Rally instance that's flagging these issues is in the utility container22:14
jrossercloudnull: i'll take a dig at this tomorrow22:17
cloudnullsorry i dont have a good trick up my sleeve for that at the moment22:18
cloudnullhowever i can see this as being something we should generally just have / document22:18
jrosserwhat am i missing with reference_group? why can't that do it?22:18
cloudnullreference_group ?22:19
cloudnullI could just be confused too22:19
jrosserlike here https://github.com/openstack/openstack-ansible/blob/master/etc/openstack_deploy/openstack_user_config.yml.pod.example#L5222:19
jrosserso long as i organise the lower part of user_config right then the grouping might just fall out there22:21
*** TxGirlGeek has quit IRC22:21
cloudnullyea. it would seem so22:23
cloudnullhttps://github.com/openstack/openstack-ansible/blob/master/osa_toolkit/generate.py#L55422:23
cloudnulllooks like that just creates pseudo groups but keeps the core mapping the same.22:23
cloudnullTIL22:23
jrosserthat seems to support a hierarchy out of the box https://github.com/openstack/openstack-ansible/blob/master/etc/openstack_deploy/openstack_user_config.yml.pod.example#L47222:23
*** TxGirlGeek has joined #openstack-ansible22:32
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible master: Converge distro interfaces to systemd-networkd  https://review.openstack.org/55272122:37
*** dave-mccowan has quit IRC22:47
*** Sha0000 has quit IRC22:48
*** masber has quit IRC22:57
DimGRhi22:59
*** masber has joined #openstack-ansible23:00
*** sawblade6 has quit IRC23:01
*** d3n14l has quit IRC23:02
cloudnullo/ DimGR23:03
jraokay, so I tracked down the root of my timing-out Rally tests, but it's only opened up a new mystery to solve23:08
jraThe tests that succeed are made against API targets referencing the internal LB address, which makes sense since they're being run from inside a container23:09
jrabut the tests that fail are made against API targets referencing the external LB address, and they all time out horrifically23:09
jraAm I wrong in thinking that they should all target the internal LB address?23:10
jraAnd if I'm not wrong, does that mean I've misconfigured something at the user yml file level?23:11
cloudnulljra: I would suspect that the rally tests _should_ use the internal address of the cluster instead of the external23:11
*** armaan has quit IRC23:11
cloudnulli've little knowledge of the internals of rally23:12
jracloudnull: I get the sense from the tempest debug log that it's using the information it obtains from successful requests to canonicalize further requests23:12
cloudnullbut I would suspect that there's something in config that can repoint it to use only the internal23:12
jraIE, it asks the API where its targets are, sees the external address in the response, and uses it23:12
cloudnullif it's using the keystone service catalog it should be able to see internal, admin, public23:13
cloudnulland then go from there23:13
jraso Rally can address multiple deployments, and it stores what it knows about them in the DB23:16
jradoing 'rally deployment show <id>' shows that it is targeting the internal address on :5000 for auth, and it has an endpoint_type attribute set to "internal"23:17
jraseems like the application is well aware of the internal/external split23:18
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible master: common-tasks: Container Start/Stop conditions  https://review.openstack.org/55309423:22
jmccrorycloudnull is this not needed anymore? https://review.openstack.org/#/c/552643/23:24
*** hw_wutianwei has quit IRC23:24
*** hw_wutianwei has joined #openstack-ansible23:24
cloudnulljmccrory: nope, the change it relied on was reverted23:24
cloudnullI put that change in https://review.openstack.org/#/c/55291423:25
cloudnullwhich is waiting on the infra importing the systemd roles we're trying to use23:25
jmccroryah ok23:25
cloudnulljra: are you cable to capture the auth request and a single test run to see it's respecting that setting ?23:25
cloudnullmaybe it's simply using public no matter the setting?23:26
jraI've got it running just a single test right now, the setup and teardown steps seem to work, but the actual test (getting the details of a flavor) fails with this timeout23:26
jraI can pastebin the tempest logs if you'd like to see 'em?23:28
*** TxGirlGeek has quit IRC23:37
*** TxGirlGeek has joined #openstack-ansible23:41
cloudnulljrs so im not sure23:42
cloudnulllooking around i dont see anything that stands out saying internal urls are broken23:42
cloudnulland there's not much on the bug tracker23:42
cloudnullhttps://bugs.launchpad.net/rally23:42
*** savvas has joined #openstack-ansible23:42
cloudnulla long time ago there was this https://bugs.launchpad.net/rally/+bug/141676023:43
openstackLaunchpad bug 1416760 in Rally "Rally does not work for "internal" endpoint types" [Medium,Invalid]23:43
*** Matias has joined #openstack-ansible23:43
cloudnullbut that was fixed here https://review.openstack.org/#/c/157988/23:43
*** TxGirlGeek has quit IRC23:43
cloudnullso IDK .23:43
*** TxGirlGeek has joined #openstack-ansible23:44
cloudnullif you could paste some logs I'd be happy to look at them23:44
jracloudnull: okay, I actually think I've got the answer23:46
jraSo Rally dynamically configures Tempest when you create it as a verifier23:46
jraand it uses a default config which specifies "publicURL" for all services except identity23:47
*** savvas has quit IRC23:47
jraif I override that when reconfiguring Tempest, I should be able to specify internal23:47
*** esberglu has quit IRC23:50
*** hamzy has quit IRC23:56
*** hamzy has joined #openstack-ansible23:58

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!