Thursday, 2019-02-21

openstackgerritRedHat RDO CI proposed openstack/tripleo-heat-templates stable/queens: GATE CHECK for TripleO  https://review.openstack.org/56722400:00
*** agopi has joined #tripleo00:00
openstackgerritRedHat RDO CI proposed openstack/tripleo-heat-templates master: GATE CHECK for TripleO  https://review.openstack.org/60429800:00
openstackgerritRedHat RDO CI proposed openstack/tripleo-heat-templates stable/rocky: GATE CHECK for TripleO  https://review.openstack.org/60429300:00
*** jcoufal has quit IRC00:03
*** itlinux has joined #tripleo00:05
*** wolverineav has quit IRC00:05
*** itlinux has quit IRC00:06
*** wolverineav has joined #tripleo00:06
openstackgerritDan Sneddon proposed openstack/tripleo-heat-templates master: Backup /etc/os-net-config/config.json when os-net-config runs  https://review.openstack.org/63830000:07
*** wolverineav has quit IRC00:11
*** agopi has quit IRC00:12
*** tosky has quit IRC00:19
larsksstevebaker: sent you an email; I think your original theory was correct.00:21
stevebakerlarsks: ok, I think you've kicked off some needed changes anyway00:23
stevebakerlarsks: all the environment files are merged in the order they are specified on the command line, so if you want to be certain of the value of NeutronMechanismDrivers then set it in a custom environment file that you specify last00:25
larsksRight, yes, that's what I meant by "...if ironic.yaml is included after ..."00:26
*** wolverineav has joined #tripleo00:37
*** wolverineav has quit IRC00:42
*** mjturek has joined #tripleo00:45
*** rascasoft has joined #tripleo00:46
openstackgerritSagi Shnaidman proposed openstack/tripleo-quickstart master: WIP: fix freeipa role for quickstart  https://review.openstack.org/63830600:49
*** rascasoft has quit IRC00:50
*** mjturek has quit IRC00:51
*** slaweq has quit IRC00:52
*** thrash is now known as thrash|g0ne00:54
dtrainorwhen assigning / using ironic ports, is it important to list the MAC addresses of *all* the ports, or just the one(s) you intend on using?  I can't figure out why these nodes are stuck in the 'enroll' state after introspection and all I'm able to conclude at this point is because introspection finds additional interfaces (ports) with other mac addresses than what I've listed when creating the nodes00:59
openstackgerritMerged openstack/tripleo-common master: Enable placement-api health check  https://review.openstack.org/63444601:09
dsneddondtrainor, I usually only list the PXE boot MAC address, and let introspection figure out the rest.01:15
dtrainorgotcha.  that's all i have defined, the ctlplane address one and the same01:17
dtrainor2019-02-20 19:52:07.819 1 DEBUG ironic_inspector.plugins.standard [req-a1e42b16-5e8f-4941-9151-29866a83f507 - - - - -] [node: MAC d4:3d:7e:bc:ad:02] No BMC address provided in introspection data, assuming virtual environment before_processing /usr/lib/python2.7/site-packages/ironic_inspector/plugins/standard.py:24501:18
dtrainori see that one....01:18
dtrainorbut I have no need for pm_address, this is using the snmp driver01:18
*** rbrady_ has quit IRC01:24
*** wolverineav has joined #tripleo01:24
*** agopi has joined #tripleo01:26
*** wolverineav has quit IRC01:28
*** tzumainn has quit IRC01:36
*** rascasoft has joined #tripleo01:36
*** dciabrin has quit IRC01:37
*** dciabrin has joined #tripleo01:38
*** rascasoft has quit IRC01:42
openstackgerritSteve Baker proposed openstack/tripleo-common master: Allow tripleo-admin user to have a keypair  https://review.openstack.org/63831601:44
*** sdake has quit IRC01:44
*** rh-jelabarre has quit IRC01:44
*** sdake has joined #tripleo01:46
*** wolverineav has joined #tripleo01:50
*** _fragatina has quit IRC02:12
*** sai_p_ has quit IRC02:14
*** _fragatina has joined #tripleo02:16
*** _fragatina has quit IRC02:17
*** _fragatina has joined #tripleo02:17
*** wolverineav has quit IRC02:20
*** mschuppert has quit IRC02:21
*** wolverineav has joined #tripleo02:21
*** _fragatina has quit IRC02:21
*** wolverineav has quit IRC02:22
*** wolverineav has joined #tripleo02:26
*** wolverineav has quit IRC02:31
openstackgerritSteve Baker proposed openstack/tripleo-heat-templates master: WIP run mistral_executor unprivileged  https://review.openstack.org/63681302:41
openstackgerritSteve Baker proposed openstack/tripleo-heat-templates master: Copy tripleo-admin key to mistral home directory  https://review.openstack.org/63832202:42
openstackgerritSteve Baker proposed openstack/tripleo-common master: Set the private key for undercloud tripleo-admin  https://review.openstack.org/63832302:43
openstackgerritSteve Baker proposed openstack/tripleo-heat-templates master: Run mistral_executor unprivileged  https://review.openstack.org/63681302:45
*** sdake has quit IRC02:48
*** sdake has joined #tripleo02:49
*** psachin has joined #tripleo02:53
*** sdake has quit IRC02:57
*** sdake has joined #tripleo02:59
*** jaganathan has joined #tripleo03:00
*** wolverineav has joined #tripleo03:06
*** wolverineav has quit IRC03:10
*** apetrich has quit IRC03:14
*** saneax has joined #tripleo03:25
*** skramaja has joined #tripleo03:29
*** spsurya has joined #tripleo03:35
*** ramishra has joined #tripleo03:43
*** wolverineav has joined #tripleo03:48
*** udesale has joined #tripleo03:51
*** rascasoft has joined #tripleo04:06
*** saneax has quit IRC04:11
*** rascasoft has quit IRC04:18
*** wolverineav has quit IRC04:29
*** _fragatina has joined #tripleo04:33
*** _fragatina has quit IRC04:37
*** _fragatina has joined #tripleo04:37
*** jbadiapa has quit IRC04:38
*** jbadiapa has joined #tripleo04:38
*** moshele has joined #tripleo04:39
*** moshele has quit IRC04:45
*** sdake has quit IRC04:50
*** sdake has joined #tripleo04:54
openstackgerritPranali Deore proposed openstack/tripleo-heat-templates stable/queens: Adding support of glance cinder store settings  https://review.openstack.org/63833704:59
*** pdeore has joined #tripleo04:59
*** jtomasek has joined #tripleo05:09
*** sdake has quit IRC05:14
*** sdake has joined #tripleo05:15
*** shyamb has joined #tripleo05:19
*** moshele has joined #tripleo05:23
openstackgerritMerged openstack/tripleo-common master: Append qemu-img to Mistral executor package list  https://review.openstack.org/63751505:33
*** _fragatina has quit IRC05:38
*** ratailor has joined #tripleo05:40
*** saneax has joined #tripleo05:45
*** sdake has quit IRC05:48
*** agopi has quit IRC05:49
*** dmacpher has quit IRC05:50
*** sdake has joined #tripleo05:51
*** cfontain has joined #tripleo05:53
*** wolverineav has joined #tripleo05:55
*** cfontain has quit IRC05:57
*** wolverineav has quit IRC05:59
*** dmacpher has joined #tripleo05:59
*** kmrchdn is now known as chandankumar06:01
*** sdake has quit IRC06:03
*** sdake has joined #tripleo06:04
*** spsurya has quit IRC06:25
*** sdake has quit IRC06:37
*** quiquell|off is now known as quiquell06:50
*** shyamb has quit IRC06:55
*** rascasoft has joined #tripleo07:05
*** shyamb has joined #tripleo07:06
*** jfrancoa has joined #tripleo07:07
*** slaweq has joined #tripleo07:08
*** rascasoft has quit IRC07:09
*** rascasoft has joined #tripleo07:09
*** psachin has quit IRC07:10
*** egonzalez has quit IRC07:10
*** egonzalez has joined #tripleo07:11
*** rcernin has quit IRC07:13
*** spsurya has joined #tripleo07:17
*** wolverineav has joined #tripleo07:18
*** kopecmartin|off is now known as kopecmartin07:19
*** pcaruana has joined #tripleo07:19
*** mschuppert has joined #tripleo07:20
*** agurenko has joined #tripleo07:23
*** ksambor has joined #tripleo07:27
*** quiquell is now known as quiquiell|brb07:30
*** psachin has joined #tripleo07:33
*** apetrich has joined #tripleo07:33
*** cfontain has joined #tripleo07:33
*** shyamb has quit IRC07:37
openstackgerritJose Luis Franco proposed openstack/tripleo-heat-templates master: Remove tasks that stop and disable Horizon services.  https://review.openstack.org/63813407:39
*** jaganathan has quit IRC07:43
*** rlandy|bbl is now known as rlandy07:44
*** paramite has joined #tripleo07:47
*** wolverineav has quit IRC07:52
*** avivgt has joined #tripleo07:53
*** yprokule has joined #tripleo07:54
*** morazi has joined #tripleo07:56
*** aufi has joined #tripleo07:57
*** quiquiell|brb is now known as quiquell07:59
*** jaosorior has quit IRC08:08
*** florianf has joined #tripleo08:15
*** ccamacho has quit IRC08:18
*** tkajinam has quit IRC08:23
*** jpich has joined #tripleo08:23
*** tosky has joined #tripleo08:26
openstackgerritMarios Andreou proposed openstack-infra/tripleo-ci master: Temporary disable scenarios 1+4 standalone for ceph bluestore  https://review.openstack.org/63836308:26
*** dsneddon has quit IRC08:26
mariosfultonj: ^ we can do that unless someone suggests better way ^08:26
mariosfultonj: & make https://review.openstack.org/#/c/637196/ depends-on https://review.openstack.org/63836308:27
*** cylopez has joined #tripleo08:27
mariosfultonj: (added to topic we talk later when you're in https://review.openstack.org/#/q/topic:bluestore i'l also post the revert in a sec and add there)08:27
*** wolverineav has joined #tripleo08:28
*** jpena|off is now known as jpena08:29
*** bogdando has joined #tripleo08:30
openstackgerritMarios Andreou proposed openstack-infra/tripleo-ci master: Temporary disable scenarios 1+4 standalone for ceph bluestore  https://review.openstack.org/63836308:33
*** agurenko has quit IRC08:35
*** agurenko has joined #tripleo08:36
*** hberaud|gone is now known as hberaud08:36
*** saneax has quit IRC08:37
openstackgerritMarios Andreou proposed openstack-infra/tripleo-ci master: Temporary disable scenarios 1+4 standalone for ceph bluestore  https://review.openstack.org/63836308:37
openstackgerritMarios Andreou proposed openstack-infra/tripleo-ci master: Revert "Temporary disable scenarios 1+4 standalone for ceph bluestore"  https://review.openstack.org/63836408:37
mariosfultonj: ^08:38
*** bogdando has left #tripleo08:39
*** bogdando has joined #tripleo08:39
openstackgerritKamil Sambor proposed openstack/puppet-tripleo master: Add OVN to enabled provider drivers in Octavia  https://review.openstack.org/63814108:51
*** dtantsur|afk is now known as dtantsur\08:53
*** dtantsur\ is now known as dtantsur08:53
*** chem has quit IRC08:56
*** shyamb has joined #tripleo08:58
*** gfidente has joined #tripleo08:59
*** dsneddon has joined #tripleo09:00
*** wolverineav has quit IRC09:00
ramishrad0ugal: Hi! around?09:03
d0ugalramishra: I am :)09:03
ramishrad0ugal: I was looking at https://github.com/openstack/tripleo-common/commit/5798748cb5f0e182f8f3066e70f7989555f0674209:04
d0ugalI roughly remember that :)09:04
ramishraTrying to understand what it's supposed to do, nothing in the commit message09:04
ramishraLooks like when adding that parameter and validating the template, user-environment is not available09:05
d0ugalramishra: Sure. So simply it sets the RootStackName parameter in heat. So that the stack knows its own name? I think09:05
ramishraso if there is customer service added to roles_data and the mapping in user-env, the template validation would fail09:06
d0ugalramishra: iirc I added it for jaosorior, but I forget why it was needed.09:06
ramishrayeah, nothing in the commit message;)09:06
d0ugalIndeed, that is unlike me. So I'm not sure why09:06
d0ugalI think maybe it was added in a rush09:07
ramishraI was wondering if we can just drop it as it's causing regressions09:07
ramishrabut need to know what it was trying to fix09:07
d0ugalramishra: Causing regressions where? That commit is quite old.09:07
bogdandohi folks. I wonder why http://logs.openstack.org/61/635161/4/check/tripleo-ci-centos-7-standalone/98e6c4f/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz fails on rabbit: 2019-02-20 17:48:02 |         "Warning: Facter: Could not retrieve fact='rabbitmq_plugins_dirs', resolution='<anonymous>': undefined method `[]' for nil:NilClass",09:07
openstackgerritNatal Ngétal proposed openstack/python-tripleoclient master: Add support of extra variables.  https://review.openstack.org/63753009:07
d0ugalramishra: it doesn't look like it is used anywhere... http://codesearch.openstack.org/?q=RootStackName&i=nope&files=&repos=09:08
bogdandodciabrin: could it be some master regression?..09:08
bogdandonot sure https://review.openstack.org/#/c/635161/ could affect rabbit09:08
ramishrad0ugal: regression when adding custom services in roles_data09:08
*** holser_ has joined #tripleo09:08
bogdandotho it did to mysql it seems09:08
ramishrawith the mapping in an user environment09:08
d0ugalramishra: I would check with jaosorior if you can. He hopefully remembers is09:09
dciabrinbogdando, sorry i didn't follow. what's the pb?09:09
*** ccamacho has joined #tripleo09:09
bogdandodciabrin: ^^ http://logs.openstack.org/61/635161/4/check/tripleo-ci-centos-7-standalone/98e6c4f/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz fails on rabbit: 2019-02-20 17:48:02 |         "Warning: Facter: Could not retrieve fact='rabbitmq_plugins_dirs', resolution='<anonymous>': undefined method `[]' for nil:NilClass",09:09
openstackgerritBogdan Dobrelya proposed openstack/tripleo-heat-templates master: Apply stop period for paunch-managed stateful svcs  https://review.openstack.org/63516109:09
*** dsneddon has quit IRC09:09
ramishrad0ugal: Ok will check, does not seem to around atm09:09
bogdandojust rebased, may be that helps09:09
*** florianf has quit IRC09:10
d0ugalramishra: http://eavesdrop.openstack.org/irclogs/%23tripleo/%23tripleo.2017-11-22.log.html#t2017-11-22T12:40:5309:10
dciabrinbogdando, hmm this is a warning though09:10
bogdandoindeed09:10
bogdandoso it should be to my mysqld changes, thanks09:10
dciabrinbogdando, pb is in mysql09:10
bogdandoyea09:10
bogdandojust was asking for rabbit part09:10
*** ccamacho has quit IRC09:10
*** ccamacho has joined #tripleo09:11
dciabrinbogdando, the puppet module can't contact the db, so i assume it's missing /root/.my.cnf09:11
d0ugalramishra: I suspect it was needed but isn't used now. So you might want to try reverting my patch and see what happens in CI09:12
dciabrinbogdando, hmmm wait it's erroring in mysql_init_tasks? weird09:12
*** janki has joined #tripleo09:12
bogdandodciabrin: indeed09:12
bogdandowill try the repro script09:12
dciabrinack ping me if you have an env to look at. normally mysql_init_tasks doesn't barely anything, i'm surprised it's trying to change user credentials here09:12
ramishrad0ugal: I did not get from that conversation why it was needed though, yeah I'll revert it09:12
d0ugalramishra: Yeah, I don't really understand either :)09:13
*** chem has joined #tripleo09:13
ramishrad0ugal: plan update is also broken.. so I'll see if we can have a better solution like not validating/flattening params before plan actually has the user-environment09:15
ramishraor something like that09:15
d0ugalIt has been a long time since I looked at those workflows.09:21
*** suuuper has joined #tripleo09:26
*** derekh has joined #tripleo09:31
*** florianf has joined #tripleo09:32
*** bogdando has quit IRC09:32
*** moguimar has joined #tripleo09:33
openstackgerritGael Chamoulaud proposed openstack/tripleo-validations stable/rocky: Update generate_validations_doc.py script  https://review.openstack.org/63837309:35
openstackgerritLukas Bezdicka proposed openstack/tripleo-heat-templates stable/rocky: FFWD: Introduce workaround for neutron cisco plugin  https://review.openstack.org/61948409:39
*** shyamb has quit IRC09:41
*** bogdando has joined #tripleo09:45
openstackgerritJakub Libosvar proposed openstack/tripleo-heat-templates master: ml2-ansible: Set api_max_retries when net-ansible is used  https://review.openstack.org/63811909:46
openstackgerritMartin Schuppert proposed openstack/tripleo-docs master: [Documentation] Document procedure to manage additional nova cells v2  https://review.openstack.org/63628609:47
*** shyamb has joined #tripleo09:50
*** panda|off is now known as panda09:54
openstackgerritMartin André proposed openstack/tripleo-common master: image-serve: redirect root to /v2  https://review.openstack.org/63740409:56
openstackgerritMartin Schuppert proposed openstack/tripleo-heat-templates master: Add functionality to manage KSM on compute nodes per role  https://review.openstack.org/63102109:57
*** wolverineav has joined #tripleo09:57
*** jaosorior has joined #tripleo09:57
*** d4c07 has joined #tripleo10:00
jaosoriorsshnaidm: hey, the error you saw is only in rocky?10:01
*** iurygregory has quit IRC10:01
d4c07I am trying to get network isolation working on tripleo queens (centos) but cannot access the horizon dashboard. I have asked a question on ask.openstack but no replies as yet.10:01
d4c07Anyone able to help if i provide additional information?10:01
jaosoriormschuppert: could you take a look at this error in rocky http://logs.rdoproject.org/68/638068/1/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039/401ef12/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz ?10:02
jaosoriord4c07: How are you trying to access horizon? what version are you running?10:02
d0ugaljaosorior: How much do you remember about the RootStackName parameter? ramishra was asking about it10:03
jaosoriord0ugal: I don't remember that parameter at all :D10:03
mschuppertjaosorior: checking10:03
jaosoriord0ugal: it doesn't appear in t-h-t even10:03
jaosoriord0ugal: not in rocky, nor in queens10:04
d0ugaljaosorior: haha, well, I added it to tripleo-common for you :)10:04
d4c07Trying to access remotely via web browser. Without network isolation i was able to connect via ssh tunnnl through the undercloud node. I have followed the tripleo instructions for queens. Sorry this is fairly new to me.10:04
d0ugaljaosorior: http://eavesdrop.openstack.org/irclogs/%23tripleo/%23tripleo.2017-11-22.log.html#t2017-11-22T12:40:5310:04
jaosoriord0ugal: daaaaamn10:05
d4c07I am also not able to ping the external IP assigned from the undercloud node. Strangly i see the default route is via the undercloud which i didnt think should be the case with network isolation10:05
d0ugaljaosorior: https://github.com/openstack/tripleo-common/blob/master/workbooks/plan_management.yaml#L13610:05
jaosoriord0ugal: damn, I don't remember10:05
d0ugaljaosorior: it only exists there, I don't think it is used10:05
d0ugalbut ramishra hit a problem with it, so I think they are going to try reverting it10:06
jaosoriord0ugal: what's the issue?10:06
d0ugaljaosorior: I don't fully understand, but this is what ramishra said earlier:10:07
d0ugalhttp://eavesdrop.openstack.org/irclogs/%23tripleo/%23tripleo.2019-02-21.log.html#t2019-02-21T09:06:0910:07
* d0ugal never uses eavesdrop and then uses it twice in one day10:07
openstackgerritDougal Matthews proposed openstack/tripleo-common master: Revert "Add the RootStackName parameter to the plan"  https://review.openstack.org/63837710:08
ramishrajaosorior, d0ugal https://bugzilla.redhat.com/show_bug.cgi?id=1677230, though there are too many useless comments in it10:08
openstackbugzilla.redhat.com bug 1677230 in rhosp-director "Error validating plan with custom service ERROR: The Resource Type (OS::TripleO::Services::Dummy) could not be found." [Medium,Assigned] - Assigned to cjeanner10:08
d0ugalramishra: FYI https://review.openstack.org/#/c/638377/10:08
jaosoriord0ugal: I guess we can remove it. Maybe it was to be used at some point and in the end we used something else10:09
openstackgerritDmitry Tantsur proposed openstack/tripleo-common master: nova-less-deploy: implement deploy_roles workflow  https://review.openstack.org/63719110:09
jaosoriorI don't recall really10:09
mschuppertjaosorior: 2019-02-20 15:50:19 |         "Error: Execution of '/usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg20190220-16-1mdtg5o -c' returned 1: [ALERT] 050/154803 (332) : parsing [/etc/haproxy/haproxy.cfg20190220-16-1mdtg5o:28] : 'bind 172.17.0.16:9292' : unable to load SSL private key from PEM file '/etc/pki/tls/certs/haproxy/overcloud-haproxy-internal_api.pem'.",10:09
jaosoriormschuppert: oho!10:09
mschuppertjaosorior: wasn't that something we recently have sen in master?10:09
jaosoriormschuppert: yeah10:10
jaosoriormschuppert: so we're probably just missing a backport10:10
d0ugaljaosorior: k, thanks. I'll see how my revert patch does in CI10:10
d4c07jaosorior: Trying to access remotely via web browser. Without network isolation i was able to connect via ssh tunnnl through the undercloud node. I have followed the tripleo instructions for queens. Sorry this is fairly new to me.10:10
jaosoriord4c07: sure, are you using the  external IP to connect?10:11
d4c07jaosorior: I am also not able to ping the external IP assigned from the undercloud node. Strangly i see the default route is via the undercloud which i didnt think should be the case with network isolation10:11
d4c07jaosorior: yes that is correct10:11
jaosoriord4c07: that is to be expected. the undercloud usually doesn't have a route to the external network. You need to add that yourself10:11
jaosoriord4c07: what are you using to deploy? are you deploying manually? or using quickstart?10:12
jaosoriormschuppert: I'm trying to look at the logs but my internet is SLOOOOwwww.... X_x10:12
d4c07jaosorior: manually using tripleo instructions. One undercloud node, one controller and one compute.10:13
d4c07jaosorior: using single nic with vlans for isolation.10:15
mschuppertjaosorior: it seems the certs were never requested. also novajoin was not run on overcloud10:15
jaosoriorsshnaidm: ^^10:15
openstackgerritDougal Matthews proposed openstack/tripleo-common master: Revert "Add the RootStackName parameter to the plan"  https://review.openstack.org/63837710:16
jaosoriormschuppert: I think I know what one of the issues is.10:17
mschuppertjaosorior: 8b69c6b584ea5a9c81e5e7880e6de22613421cc1 ?10:17
openstackgerritDougal Matthews proposed openstack/tripleo-common master: Revert "Add the RootStackName parameter to the plan"  https://review.openstack.org/63837710:17
jaosoriormschuppert: exactly10:17
jaosoriormschuppert: that's just one of the issues though10:17
jaosoriorif the nodes didn't enroll to FreeIPA... that's something else10:17
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates stable/rocky: Add CertmongerUser role to OVB defaults  https://review.openstack.org/63837810:19
jaosoriorsshnaidm, mschuppert ^^10:19
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates stable/queens: Add CertmongerUser role to OVB defaults  https://review.openstack.org/63837910:19
shyambHi10:21
shyambOn my env, introspection is failing for overcloud node10:21
jaosoriormschuppert: seems that novajoin did run. As the nodes did enroll to FreeIPA http://logs.rdoproject.org/68/638068/1/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039/401ef12/logs/overcloud-controller-0/var/log/ipaclient-install.log.txt.gz10:21
shyambI can it can get dhcp ip10:21
shyambbut it's saying nothing to boot; no such file or directory10:21
mschuppertjaosorior: ok, I don't we usually have a novajoin log dir?10:22
shyambjaosorior: any comments?10:22
*** spsurya has quit IRC10:22
jaosoriorshyamb: I'm not very acquainted with that part of the deployment to be honest.10:22
jaosoriordtantsur: ^^10:22
jaosoriormschuppert: that directory is not on the undercloud?10:22
shyambjaosorior: ok10:22
mschuppertjaosorior: it is :)10:23
dtantsurshyamb: https://docs.openstack.org/ironic-inspector/latest/user/troubleshooting.html#troubleshooting-pxe-boot10:23
* jaosorior still having connectivity problems10:23
mschuppertjaosorior: well not in containers/novajoin10:24
jaosoriormschuppert: rocky already had a containerized undercloud, right?10:25
mschuppertyes10:25
jaosoriormaybe novajoin is only logging to stdout10:26
jaosoriorso the logs might be in the journal10:26
jaosoriortrying to check, but this internet's slowness is killing me10:26
jaosoriormaybe I gotta move to anther café10:26
*** moshele has quit IRC10:27
openstackgerritLukas Bezdicka proposed openstack/tripleo-heat-templates stable/rocky: Upgrades: Ensure idempotency of pacemaker services  https://review.openstack.org/63838410:29
*** wolverineav has quit IRC10:30
*** dciabrin has quit IRC10:31
*** dciabrin has joined #tripleo10:31
mschuppertjaosorior: looks like novajoin was started ok.10:32
*** mschuppert is now known as mschuppert|afk10:35
openstackgerritSagi Shnaidman proposed openstack/tripleo-quickstart master: Fix freeipa role for quickstart  https://review.openstack.org/63830610:36
sshnaidmjaosorior, ^^10:36
shyambdtantsur: Thanks10:37
shyamblooking into it10:37
jaosoriorsshnaidm: nice! thanks!10:38
jaosoriorI don't have my machine right now to try it out. Were you able ot?10:38
jaosorior* to10:38
jaosoriormschuppert|afk: cool! did you see that from /var/log/messages?10:38
*** psachin has quit IRC10:38
*** pdeore has quit IRC10:39
openstackgerritJakub Libosvar proposed openstack/tripleo-heat-templates master: ml2-ansible: Set api_max_retries when net-ansible is used  https://review.openstack.org/63811910:39
*** agurenko has quit IRC10:39
*** d4c07 has quit IRC10:41
*** agurenko has joined #tripleo10:42
*** psachin has joined #tripleo10:42
*** mschuppert|afk is now known as mschuppert10:43
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-docs master: Add service template sections walkthough (part 3)  https://review.openstack.org/63838810:45
*** shyamb has quit IRC10:46
mschuppertjaosorior: yes from /var/log/journal10:48
openstackgerritSorin Sbarnea proposed openstack-infra/tripleo-ci master: DNM: Add py3/f28 support to tripleo-container-build-base  https://review.openstack.org/63616010:49
openstackgerritSorin Sbarnea proposed openstack-infra/tripleo-ci master: DNM: Add py3/f28 support to tripleo-container-build-base  https://review.openstack.org/63616010:50
jaosoriormschuppert: nice nice!10:52
*** jaosorior has quit IRC10:55
*** d4c07 has joined #tripleo11:02
openstackgerritSorin Sbarnea proposed openstack-infra/tripleo-ci master: DNM: Add py3/f28 support to tripleo-container-build-base  https://review.openstack.org/63616011:02
*** moshele has joined #tripleo11:04
openstackgerritDmitry Tantsur proposed openstack/tripleo-common master: nova-less-deploy: implement deploy_roles workflow  https://review.openstack.org/63719111:06
*** jaosorior has joined #tripleo11:15
*** shyamb has joined #tripleo11:15
openstackgerritDmitry Tantsur proposed openstack/tripleo-common master: nova-less-deploy: implement deploy_roles workflow  https://review.openstack.org/63719111:16
jaosoriormschuppert: now I have better internet :D11:16
jaosoriordtantsur: how's that going? ^^ (nova-less-deploy)11:16
dtantsurjaosorior: moving along. unfortunately, I'm quite far from being full-time on it. But I think this patch brings us quite close to something one can play with.11:17
jaosoriordtantsur: got it, let me check out the patch.11:20
openstackgerritLuigi Toscano proposed openstack/tripleo-heat-templates master: Set the NTP server on Sahara too  https://review.openstack.org/63296411:21
jaosoriordtantsur: I may lack a little context here. But what is a "baremetal_deployment file"?11:22
*** psachin has quit IRC11:22
dtantsurjaosorior: http://specs.openstack.org/openstack/tripleo-specs/specs/stein/nova-less-deploy.html#baremetal-deployment-yaml11:22
jaosoriorgotta re-read the spec11:22
jaosorioralready forgot a lot of it11:22
jaosoriorthanks for the link11:22
*** pbandark has joined #tripleo11:24
dtantsurI had to re-read it myself recently :) I'm doing too many things in parallel..11:24
dtantsurjaosorior: also the notes at https://etherpad.openstack.org/p/tripleo-nova-less-deploy are still relevant11:25
jaosoriorack11:25
*** udesale has quit IRC11:26
*** wolverineav has joined #tripleo11:27
openstackgerritNatal Ngétal proposed openstack/python-tripleoclient master: Move the code.  https://review.openstack.org/63839911:30
*** raildo has joined #tripleo11:33
jaosoriordtantsur: that's not being tested in CI yet, right?11:34
dtantsurjaosorior: nope. I haven't got to a full-flow deployment with this code yet :(11:35
jaosorioralright11:35
dtantsurI should probably start putting some quickstart bits in, but $priorities...11:35
*** moshele has quit IRC11:37
dtantsuractually, I can probably try a deployment with deployed-server. expect a lot of stupid questions :)11:37
*** moshele has joined #tripleo11:38
*** ratailor has quit IRC11:42
shyambHi11:47
shyambhttps://bugzilla.redhat.com/show_bug.cgi?id=162933111:47
openstackbugzilla.redhat.com bug 1629331 in openstack-tripleo "docker container ironic_inspector_dnsmasq continuously restarts" [High,New] - Assigned to jslagle11:47
shyambinspector_dnsmasq container is not starting11:54
shyambthrowing address_already in use11:55
shyambI think it is conflicting with neutron's dnsmasq11:55
shyambHow can I solve this/=?11:55
shyambWe need both containers to be up11:55
*** jpena is now known as jpena|lunch11:56
*** pbandark has quit IRC11:57
openstackgerritMerged openstack-infra/tripleo-ci master: Remove the container cli override for standalone scenario 1-3  https://review.openstack.org/63821211:59
*** dciabrin_ has joined #tripleo12:00
*** holser_ is now known as holser|lunch12:00
*** wolverineav has quit IRC12:00
openstackgerritSorin Sbarnea proposed openstack/tripleo-repos master: Add fedora support to tripleo-repos  https://review.openstack.org/63721212:01
*** dciabrin has quit IRC12:01
*** sdake has joined #tripleo12:02
shyambowalsh: Hi12:04
openstackgerritSorin Sbarnea proposed openstack-infra/tripleo-ci master: DNM: Add py3/f28 support to tripleo-container-build-base  https://review.openstack.org/63616012:05
shyambI am blocked on osp14 due to https://bugzilla.redhat.com/show_bug.cgi?id=162933112:05
openstackbugzilla.redhat.com bug 1629331 in openstack-tripleo "docker container ironic_inspector_dnsmasq continuously restarts" [High,New] - Assigned to jslagle12:05
*** quiquell is now known as quiquell|lunch12:05
shyambI see this issue is still in NEW state12:05
shyambDo we have any workaround?12:05
*** jfrancoa has quit IRC12:17
EmilienMquiquell|lunch: https://review.openstack.org/#/c/638266/12:29
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: Podman support in haproxy-public-tls-inject  https://review.openstack.org/63824612:31
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: Podman support in haproxy-public-tls-inject  https://review.openstack.org/63824612:32
*** jfrancoa has joined #tripleo12:32
jaosoriorhjensas: around?12:33
*** hberaud is now known as hberaud|lunch12:39
*** saneax has joined #tripleo12:40
*** shyamb has quit IRC12:41
*** ansmith has quit IRC12:43
* dtantsur tries deploying openstack on metalsmith-deployed nodes12:44
openstackgerritMerged openstack/tripleo-common master: drop python-crypto changes from overrides  https://review.openstack.org/63446512:44
openstackgerritMerged openstack/tripleo-heat-templates stable/queens: FFWD: Introduce workaround for neutron cisco plugin  https://review.openstack.org/61948612:44
dtantsurjaosorior: what SSH key do we use *first* when doing config-download deployment with pre-deployed servers? just the one in ~/.ssh/id_*?12:46
dtantsurslagle: if you know ^^?12:46
jaosoriorgood question. I'm not sure.12:47
dtantsurslagle: also re yesterday's discussion: I'm a bit confused by http://paste.openstack.org/show/745607/. Am I doing something wrong with HostnameFormat?12:48
dtantsurjaosorior: I'm thinking of making tripleoclient pass ~/.ssh/id_rsa.pub to the new workflow, so that we can SSH initially12:50
*** pbandark has joined #tripleo12:50
dtantsurbut I'm confused by the message I pasted above re HostnameFormat...12:50
dtantsurshould I really use HostnameFormatDefault instead? I think no, it seems to be a part of roles12:51
*** galaxyyahoo has quit IRC12:52
*** moshele has quit IRC12:52
*** galaxyyahoo has joined #tripleo12:52
jaosoriordtantsur: don't use the Default12:53
jaosoriorthat's set by the roles after they're rendered12:53
dtantsurjaosorior: I got an answer two my first question in the output:12:54
dtantsurUsing ssh user heat-admin for initial connection.12:54
dtantsurUsing ssh key at /home/stack/.ssh/id_rsa for initial connection.12:54
slagledtantsur: we use a generated key for the tripleo-admin user to do the config-download deployment. we inject the public side of that key with tripleoclient using whatever private key is specified by --overcloud-ssh-key12:54
dtantsurokay, that makes sense12:55
slagleit defaults to ~/.ssh/id_rsa12:55
*** dprince has joined #tripleo12:55
slaglethat way we never have to pass the actual private key to mistral12:55
slaglemistral only stores the generated private/public12:55
dtantsurright, yeah. now it all clicks in my mind12:55
* dtantsur has forgotten to provide the public key, and now his deployment hangs12:56
dtantsurslagle: now, my workflow generates an environment file wiht http://paste.openstack.org/show/745608/, but the deployment complains with http://paste.openstack.org/show/745607/12:57
dtantsurdo you know what goes wrong?12:57
*** wolverineav has joined #tripleo12:58
slagledtantsur: what are your roles?12:59
slagledtantsur: should probably be "ControllerDeployedServerHostnameFormat" if using the deployed-server roles12:59
dtantsurslagle: using http://tripleo.org/install/advanced_deployment/deployed_server.html#deployment-command literally with the generated environment file added in the end.12:59
slagledtantsur: then the HostnameFormat parameters need to match the role names13:00
slagleControllerDeployedServerHostnameFormat and ComputeDeployedServerHostnameFormat13:00
dtantsuroh, I see. hmmmm13:01
dtantsurjaosorior: probably needs fixing in my patch ^^^13:01
jaosoriordtantsur: ack13:01
slagledtantsur: you can use environments/split-stack-consistent-hostname-format.yaml13:02
jaosoriorsshnaidm: https://review.openstack.org/#/c/638379/ failed because the DnsServers was switched from being FreeIPA, to being the google ones.13:02
jaosoriorsshnaidm: that's for queens13:02
dtantsurslagle: next question (I know, sorry!): how does it determine which IP address to connect to for each node?13:02
slagledtantsur: DeployedServerPortMap13:03
dtantsurslagle: okay, so it's pretty much mandatory, right? the documentation reads as if it was optional..13:03
jaosoriorsshnaidm: which is strange... cause DnsServers was set correct (as a parameter), yet resolv.conf has 8.8.8.8 and 8.8.4.413:03
slagledtantsur: it is optional13:04
slagledtantsur: DeployedServerPortMap is for telling Heat what the existing ctlplane IP's are13:04
dtantsurslagle: oh wait, I seem to get it. it actually creating a neutron port. can I prevent it from touching neutron?13:05
slagledtantsur: if you wanted to instead use a new port with dhcp or something, you don't need to use DeployedServerPortMap13:05
dtantsurslagle: I have ports already (created by metalsmith)13:05
dtantsuris it possible to reuse them?13:05
slagledtantsur: then DeployedServerPortMap is what you want to use13:05
slagledtantsur: just set the IP in that parameter value. new ports won't be created13:06
*** trown|outtypewww is now known as trown13:06
slagleport resources won't be created for the ctlplane ip's at all in that case13:06
*** rh-jelabarre has joined #tripleo13:06
slagleexcept for the vip13:06
openstackgerritNatal Ngétal proposed openstack/python-tripleoclient master: Move the code.  https://review.openstack.org/63839913:06
dtantsurslagle: (looking at the example in the docs) do I need "subnets"/"network" keys as well? or just "fixed_ips"?13:08
slagledtantsur: all of it13:09
dtantsurokay. will need to talk to neutron to figure out, I guess13:09
*** quiquell|lunch is now known as quiquell13:10
openstackgerritJose Luis Franco proposed openstack/tripleo-heat-templates stable/queens: [Queens only] Ensure redhat-subscription is installed in overcloud nodes.  https://review.openstack.org/63841513:10
dtantsurmany thanks, slagle, you helped me a lot!13:10
slagledtantsur: yw! let me know how it goes13:11
*** toure|gone is now known as toure13:12
*** jcoufal has joined #tripleo13:13
quiquellEmilienM: is going to retry all the layers ?13:13
*** jpena|lunch is now known as jpena13:15
*** thrash|g0ne is now known as thrash13:16
openstackgerritJakub Libosvar proposed openstack/tripleo-heat-templates master: ml2-ansible: Switch to default Neutron backend  https://review.openstack.org/63841613:17
*** sdake has quit IRC13:23
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci master: WIP - Add fedora-28 support to container-build job on fedora-28 with Buildah  https://review.openstack.org/63599113:25
*** sdake has joined #tripleo13:25
*** holser|lunch is now known as holser_13:26
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci master: kolla-build: remove quotes in namespace config  https://review.openstack.org/63841913:26
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci master: WIP - Add fedora-28 support to container-build job on fedora-28 with Buildah  https://review.openstack.org/63599113:26
openstackgerritJakub Libosvar proposed openstack/tripleo-heat-templates master: ml2-ansible: Switch to default Neutron backend  https://review.openstack.org/63841613:28
EmilienMquiquell: yes, and I think it's safer at this point13:28
quiquellEmilienM: ack at least we can stop somewhere13:29
quiquellEmilienM: do you workflow it ?13:30
*** wolverineav has quit IRC13:30
EmilienMquiquell: I think there is a check-rdo comment13:30
EmilienMah no13:31
EmilienMdone13:31
jaosoriorsshnaidm: uhm... sucks... https://review.openstack.org/#/c/638378/ passed in the deployment, but timed out.13:33
*** moshele has joined #tripleo13:34
dtantsurslagle: I'm back :) how is <short hostname> generated in http://tripleo.org/install/advanced_deployment/deployed_server.html#network-configuration ?13:34
*** shyamb has joined #tripleo13:35
dtantsurand why not use full hostnames?13:36
*** mcornea has joined #tripleo13:36
*** sdake has quit IRC13:36
*** moshele has quit IRC13:37
*** sdake has joined #tripleo13:38
slagledtantsur: short hostname isn't really generated.13:40
slagledtantsur: it's the actual short hostname that is already set on the host13:40
*** rlandy has joined #tripleo13:40
dtantsurI'm trying to think what it is in my context..13:40
slagledtantsur: we have to use HostnameMap to tell Heat the mapping between the Heat generated server resource name and the actual host short hostname13:41
dtantsurah! so short names are the names in the right side of HostnameMap?13:41
slagleDeployedServerPortMap just happens to use keys that contain the short hostname because that's how the existing network interfaces work in the templates13:41
slagledtantsur: yes13:41
dtantsursuper, that's what I needed to know13:42
slagleHostnameMap is a mapping from the Heat assigned name to the actual short hostname13:42
*** hberaud|lunch is now known as hberaud13:42
*** shyamb has quit IRC13:44
openstackgerritCarlos Camacho proposed openstack/python-tripleoclient master: Add new CLI option openstack tripleo validate list  https://review.openstack.org/63658113:45
*** artom has quit IRC13:50
openstackgerritJakub Libosvar proposed openstack/tripleo-heat-templates master: ml2-ansible: Set api_max_retries when net-ansible is used  https://review.openstack.org/63811913:52
*** ansmith has joined #tripleo13:55
slagle#startmeeting TripleO Edge Squad Meeting14:01
openstackMeeting started Thu Feb 21 14:01:35 2019 UTC and is due to finish in 60 minutes.  The chair is slagle. Information about MeetBot at http://wiki.debian.org/MeetBot.14:01
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.14:01
slagleping slagle, csatari, jaosorior, owalsh, fultonj, gfidente, hjensas, jtomasek, bogdando, dtantsur, rbrady, d0ugal, toure, abishop14:01
*** openstack changes topic to " (Meeting topic: TripleO Edge Squad Meeting)"14:01
slagle#info remove or update your nick from the Meeting Template on the etherpad if you want (or don't want) to be ping'd for the start of the meeting14:01
openstackThe meeting name has been set to 'tripleo_edge_squad_meeting'14:01
bogdandoo/14:01
slagle#link https://etherpad.openstack.org/p/tripleo-edge-squad-status14:01
slagleAnyone can use the #link, #action, #help, #idea, #agreed, and #info commands, not just the moderatorǃ14:01
slaglehe edge'ers. who's around14:02
dtantsur\o14:02
slagle*hi14:02
fultonjo/14:02
owalsho/14:02
openstackgerritCarlos Camacho proposed openstack/python-tripleoclient master: Add new CLI option openstack tripleo validate run  https://review.openstack.org/63693014:02
slagle#topic Agenda14:02
*** openstack changes topic to "Agenda (Meeting topic: TripleO Edge Squad Meeting)"14:02
slagle* Review past action items14:02
slagle* Goals/action items for the week14:03
slagle#topic Review past action items14:03
*** openstack changes topic to "Review past action items (Meeting topic: TripleO Edge Squad Meeting)"14:03
slagleACTION: fultonj get glance container on compute node in split control plane ci job to talk to mysql on controller node14:03
fultonjhttp://logs.openstack.org/88/615988/16/check/tripleo-ci-centos-7-split-controlplane-standalone/8b9c4e5/logs/subnode-2/home/zuul/standalone_deploy.log.txt.gz#_2019-02-21_01_19_2014:03
fultonji have identified the problem...14:03
fultonjThe failure [0] occurred when the compute tried to reach 192.168.24.1 because it connected to it's own 192.168.24.1 (which is not listening on port 3306), and not the controller's 192.168.24.1.14:04
fultonjso both nodes have a 192.168.24.1 address14:04
fultonjso the edge compute node is trying to reach mysql on itself14:04
fultonjboth nodes also have a good 192.168.24.2 and 24.3 address14:05
fultonjthe compute can connect to the controller's 192.168.24.214:05
fultonjcan Maria on the controller listen on 0.0.0.0 or 192.168.24.2 instead of 192.168.24.1?14:05
fultonjso i need to find a way to get maria on the controller to listen on another IP14:06
slaglewell we probably need to assign different IP's14:06
slagleright14:06
slagledo these nodes have just the 1 nic?14:07
jaosorioro/14:07
fultonjhttp://logs.openstack.org/88/615988/15/check/tripleo-ci-centos-7-split-controlplane-standalone/f976ed8/logs/undercloud/var/log/extra/network.txt.gz14:07
fultonjeth0 br-ex br-ctlplane14:07
fultonjslagle: i have IPs i can use14:07
fultonjsplitcontrol plane job assigns the desired IPs 24.2 controller and 24.3 edge (that's ok i think)14:08
fultonji'll just tweak the end points on the compute to reach 24.2 (not 24.1)14:08
*** jaosorior has quit IRC14:08
fultonjjust need to get all services listening on 24.2 on controller14:08
fultonjunless you think i should take a diff approach?14:09
*** vinaykns has joined #tripleo14:09
slaglewe'll probably need to fix the duplicate ip's, as other issues will arise14:09
slaglecan we just tell the compute to use different IP's, then it should be ok to connect to 24.1?14:09
fultonjslagle: i'll try that14:10
fultonji'm doing the undercloud pre set up so that's probably how it got there14:11
fultonji could just disable the 24.1 as a workaround to see if that will work14:11
fultonj#ACTION fultonj to fix network on splitcontrol plane ci job14:11
owalsharen't 24.2 and 24.3 on the wrong network?14:11
d4c07hi - after deploying the overcloud with network isolation should the controller node have the default gateway of the router of the external network? For some reason it looks like the default route is out via the undercloud14:12
fultonjd4c07: we're in a meeting ATM, perhaps we can talk later14:12
d4c07sorry just noticed that.14:12
fultonjd4c07: np14:12
fultonjowalsh: 24.{2,3} are what is normally used in standalone as per docs14:13
fultonjit's just a convention14:13
fultonjbut i don't see anyting wrong with it14:13
owalshfultonj: but they are on br-ex14:13
fultonjhttps://docs.openstack.org/tripleo-docs/latest/install/containers_deployment/standalone.html14:13
fultonjstep 614:13
*** sdake has quit IRC14:14
fultonji'm not using network isoation here, just one network for this job to reproduce scenario from doc14:14
slaglethe vxlan tunnel is setup between the nodes on br-ex. i assume you can still reach 24.1 (br-ctlplane) with that setup14:15
slagle*from the compute node14:15
slagleguess we'll see14:15
fultonjslagle: yes, if only the compute node didn't also have that IP14:15
fultonjslagle: so if i tell the compute node to not have that IP i think things should work14:15
fultonji'll find that out today14:16
*** panda is now known as panda|ruck14:16
slagleyea so, i'd fix that first, and then see what happens. it will be easier to debug that way at least14:16
fultonjack14:16
owalshack14:16
slagleACTION: : land https://review.openstack.org/#/c/631489 and https://review.openstack.org/#/c/63208914:16
slaglelooks like https://review.openstack.org/#/c/631489/ just needs another +214:16
slaglethen we can land these14:17
fultonji'll review it today14:17
slaglei also plan to add docs for the dcn deployment with ceph14:17
slaglefultonj: thanks!14:17
slagleACTION: slagle/fultonj to review multiple compute+ceph deployment14:17
slaglewe did that :)14:17
slagleowalsh: fultonj : thanks for taking a look14:17
fultonjslagle: yeah it works14:18
slaglefultonj: to answer your question about cinder-volume, i did mean to deploy that because I used the DistributedHCI role14:18
slaglei think the longer term plan is also deploy etcd there to manage cinder-volume A/A14:18
slagleso we need to add etcd to the DistributedComputeHCI role14:19
fultonjhttps://docs.openstack.org/tripleo-docs/latest/install/containers_deployment/standalone.html#example-2-nodes-2-nic-using-remote-compute-with-tenant-and-provider-networks14:19
fultonjhas an example of using cinder for standalone ^14:19
fultonje.g. CinderRbdAvailabilityZone: edge114:19
slagleok, i don't think i had that configured14:20
fultonjslagle: with only one controller you should be ok without etcd as a first step14:20
fultonjthough i agree we should get it14:20
slagledoes that need to match the nova AZ? or does it not matter14:20
fultonjit should match14:20
fultonjhmmmm14:20
fultonji made it match not sure if you want to support non-matching, might be possible14:21
*** tzumainn has joined #tripleo14:21
* fultonj not sure about if it matters or not14:21
slaglei'll make it match just for convention sake at this point14:21
openstackgerritLukas Bezdicka proposed openstack/tripleo-heat-templates master: Add environment enabling memcached for cache and authtoken  https://review.openstack.org/63842414:21
openstackgerritLukas Bezdicka proposed openstack/tripleo-heat-templates master: Use dogpile.cache.memory by default  https://review.openstack.org/63842514:21
fultonj+114:21
slagle#action slagle to doc DCN deployment with ceph and cinder-volume config14:21
*** udesale has joined #tripleo14:22
slagle#topic Goals/action items for the week14:22
*** openstack changes topic to "Goals/action items for the week (Meeting topic: TripleO Edge Squad Meeting)"14:22
slagle#action slagle look at deploying etcd for cinder-volume14:22
slagleanything else to capture for the upcoming week?14:22
openstackgerritLukas Bezdicka proposed openstack/tripleo-heat-templates master: Add environment enabling memcached for cache and authtoken  https://review.openstack.org/63450514:23
openstackgerritLukas Bezdicka proposed openstack/tripleo-heat-templates master: Use dogpile.cache.memory by default  https://review.openstack.org/63842514:23
paramiteGuys, can we please merge https://review.openstack.org/#/c/638143/ to unblock gate?14:23
owalshfultonj, slagle: AFAIK the AZ need to match14:23
slagleowalsh: ack, will make it so14:24
slaglethanks folks14:25
slagle#endmeeting14:25
*** openstack changes topic to "CI Status: Should be green, recheck if needed | https://docs.openstack.org/tripleo-docs/latest/"14:25
openstackMeeting ended Thu Feb 21 14:25:06 2019 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)14:25
openstackMinutes:        http://eavesdrop.openstack.org/meetings/tripleo_edge_squad_meeting/2019/tripleo_edge_squad_meeting.2019-02-21-14.01.html14:25
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/tripleo_edge_squad_meeting/2019/tripleo_edge_squad_meeting.2019-02-21-14.01.txt14:25
openstackLog:            http://eavesdrop.openstack.org/meetings/tripleo_edge_squad_meeting/2019/tripleo_edge_squad_meeting.2019-02-21-14.01.log.html14:25
owalshslagle, fultonj: with the AZ, IIUC we set cross_az_attach=False then computes in nova AZ1 can only attach volumes from cinder AZ1, etc...14:27
fultonjowalsh: sounds familiar14:27
owalshalthough I guess in theory we could not have matching AZs and set cross_az_attach=True but that would encourage silly things14:27
slagleowalsh: yea i think that's what we want14:28
fultonji figured why would anyone not want to use the same AZ in cinder as in Nova14:28
*** wolverineav has joined #tripleo14:28
slaglethe non-silly things I mean14:28
slaglei'll try to make these changes in the environment I deployed14:28
slagleand see what happens14:28
owalshfultonj: I guess they don't necessarily align, depends on the architecture14:29
fultonjowalsh: we're probably safe in planning and recommending that they be the same14:29
owalshyup, definitely for DCN14:29
*** jaosorior has joined #tripleo14:30
*** sdake has joined #tripleo14:31
openstackgerritCarlos Camacho proposed openstack/python-tripleoclient master: Add unit tests for running and listing validations from the CLI  https://review.openstack.org/63842614:32
*** hjensas has quit IRC14:39
*** ramishra has quit IRC14:40
openstackgerritSorin Sbarnea proposed openstack-infra/tripleo-ci master: Make build-containers job definition reusable  https://review.openstack.org/63810814:43
openstackgerritSorin Sbarnea proposed openstack-infra/tripleo-ci master: DNM: Add py3/f28 support to tripleo-container-build-base  https://review.openstack.org/63616014:43
*** mjturek has joined #tripleo14:51
*** sdake has quit IRC14:51
*** ekultails has joined #tripleo14:53
d4c07hi - after deploying the overcloud with network isolation should the controller node have the default gateway of the router of the external network? For some reason it looks like the default route is out via the undercloud14:55
d4c07Any one able to possible give me some pointers?14:55
openstackgerritCarlos Camacho proposed openstack/python-tripleoclient master: Add unit tests for running and listing validations from the CLI  https://review.openstack.org/63842614:56
openstackgerritDmitry Tantsur proposed openstack/tripleo-common master: nova-less-deploy: implement deploy_roles workflow  https://review.openstack.org/63719114:59
larsksd4c07: the default gw for the external network will be found inside the qrouter namespace associated with your external routers.15:01
*** dsneddon has joined #tripleo15:01
*** wolverineav has quit IRC15:01
larsks...if by "external" you mean the network used for floating ips.15:01
openstackgerritEmilien Macchi proposed openstack/tripleo-common master: DNM - add --debug to buildah  https://review.openstack.org/63843315:01
*** sdake has joined #tripleo15:01
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci master: WIP - Add fedora-28 support to container-build job on fedora-28 with Buildah  https://review.openstack.org/63599115:02
larsksd4c07: if you mean the network used for public api/horizon access, then yes, that should be the default route on the controllers.15:02
jaosoriorlarsks: I think he means the public network that's used to access horizon.15:02
larsksjaosorior: we have unfortunately overloaded the terms "external" and "public" :)15:03
d4c07larsks: yes this is for the horizon dashboard. Without network isolation i can access horizon via a tunnel through undercloud. But cannot access on "external" once deployed with isolation15:03
*** skramaja has quit IRC15:04
larsksd4c07: you can set the external route in your network configuration.15:04
larsksHere's an example from a deployment I've worked on recently: https://github.com/CCI-MOC/rhosp-director-config/blob/master/templates/network/config/controller.yaml#L15615:04
d4c07larsks: everything i look at config wish seems correct and can see references to external router ip but not set after successful deployment.15:04
larsksYou can see there that we're explicitly setting the default route to the value of the ExternalInterfaceDefaultRoute parameter (defined in an environment file that we feed to the deploy)15:05
d4c07larsks: thanks i will take a look.15:06
d4c07Some of this config was configured using the process-templates.py but from what i can see the IPs and VLANs all look correct.15:07
*** artom has joined #tripleo15:07
d4c07larsks: I am pretty new to this but learning slowly. Thanks for the pointer i will see how i get on.15:07
*** jaosorior has quit IRC15:08
*** saneax has quit IRC15:08
*** dsneddon has quit IRC15:11
d4c07larks: hmm yeah my config looks pretty similar and the ExternalInterfaceDefaultRoute is set to the external gateway IP.15:11
*** dsneddon has joined #tripleo15:12
*** hjensas has joined #tripleo15:12
d4c07i can see a route-vlan448 file is created with correct IP (172.17.221.1). But route -n shows gateway of 192.168.24.115:14
*** gfidente has quit IRC15:14
larsksTake a lookat  /etc/os-net-config/config.json on your controllers. Does that look correct?15:14
*** Vorrtex has joined #tripleo15:15
*** dsneddon has quit IRC15:18
d4c07 It has two routes in there ip_netmask: "172.17.221.24" routes default true next_hop "172.17.221.1" then there is br-ex routes ip_netmask: 169.254.169.254/32 next_hop 192.168.24.115:18
openstackgerritDan Macpherson proposed openstack/ansible-role-openstack-operations master: Redis backup, restore, and validation tasks  https://review.openstack.org/63746715:18
d4c07*sorry cannot copy direct as my company do not allow access to IRC - so on another device :).15:18
*** dsneddon has joined #tripleo15:19
*** holser_ has quit IRC15:20
d4c07larsks: i have just created paste.openstack.org/show/74562615:21
*** jistr is now known as jistr|afk15:22
* larsks takes a look15:22
*** dsneddon has quit IRC15:24
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: Remove redundant logs collection for non-ovb  https://review.openstack.org/63843815:24
*** dsneddon has joined #tripleo15:25
*** holser_ has joined #tripleo15:26
larsksd4c07: I'm a little confused by that config.json, because it looks as if everything is defined as members on br-ex. For example, the config I pointed you at earlier maps to this config.json: https://termbin.com/o92k15:30
*** ramishra has joined #tripleo15:30
openstackgerritRabi Mishra proposed openstack/tripleo-heat-templates master: Don't look for primary_role ips in AllNodesValidationConfig  https://review.openstack.org/63844015:31
ramishrahjensas: ^^15:31
larsksI'm not an expert reading the os-net-config stuff, though; somebody else here mightg be able to give you better pointers.15:31
*** artom has quit IRC15:31
*** jistr|afk is now known as jistr15:32
*** dxiri has joined #tripleo15:32
hjensasramishra: thanks, looking at your mail.15:34
hjensasramishra: ok, I wonder if there is more. ControllerIpListMap contain all 'null' values in one of the reproducers.15:36
*** apetrich has quit IRC15:36
hjensasramishra: other roles Compute1IpListMap , Compute2IpListMap have a lot of data.15:36
ramishrahjensas: from all the errors reported till now, this seems to be reason to me15:37
d4c07larsks: thanks for looking. Yeah maybe I have done something wrong creating the network files. Although i used the network_data.yaml method and ran the process-templates.py to generate them.15:37
ramishraIt's just looking for networks for Controller in all these roles for the networks enabled for that role15:37
*** artom has joined #tripleo15:38
d4c07larsks: maybe if i try to create them manually using something similar to your repo. It seems a lot of the templates now are jinja2 which is different from the last time i looked at this 2/3 years ago.15:39
ramishrahjensas: Do you see any other traceback other than the one I mentioned in the bug?15:39
larsksd4c07: note that my repo is for the Queens release; things may look different in more recent releases.15:40
*** apetrich has joined #tripleo15:41
ramishrahjensas: late for me, will catchup tomorrow, but I think this patch would fix the current issue15:41
d4c07larsks: i am actually using queens too.15:42
hjensasramishra: tracebacks are all similar, Controller <some_network>_ip_address.15:42
hjensasramishra: I will try to get sasha to test you patch. It looks very promising!15:43
ramishrahjensas: ack15:45
*** jaosorior has joined #tripleo15:46
*** jobewan has joined #tripleo15:47
*** yprokule_ has joined #tripleo15:47
*** yprokule has quit IRC15:47
*** yprokule_ is now known as yprokule15:47
*** sdake has quit IRC15:48
openstackgerritBogdan Dobrelya proposed openstack/tripleo-heat-templates master: Apply stop period for paunch-managed stateful svcs  https://review.openstack.org/63516115:50
*** sdake has joined #tripleo15:50
openstackgerritNatal Ngétal proposed openstack/python-tripleoclient master: Add support of extra variables.  https://review.openstack.org/63753015:52
*** janki has quit IRC15:53
*** tosky has quit IRC15:53
*** agurenko has quit IRC15:53
*** aufi has quit IRC15:54
*** tosky has joined #tripleo15:54
dtantsurslagle: (me again) do I have to use https://github.com/openstack/tripleo-heat-templates/blob/master/deployed-server/deployed-server-bootstrap-centos.sh if the image I'm deploying is overcloud-full?15:54
openstackgerritCarlos Goncalves proposed openstack/tripleo-common stable/rocky: Append qemu-img to Mistral executor package list  https://review.openstack.org/63844615:55
slagledtantsur: maybe not. i can't say for certain w/o checking if all those changes are included within overcloud-full15:56
dtantsurI guess I'll figure it out rather soon15:56
slagledtantsur: i do know that those are the minimum set of changes to work on top of a vanilla centos cloud image15:57
slaglewhich is what i typically test with15:57
dtantsurright15:57
fultonjslagle bogdando there were some spelling mistakes in this docpatch. details are in the review: https://review.openstack.org/#/c/631489/2..3/doc/source/install/advanced_deployment/distributed_compute_node.rst15:57
fultonjlet me know if you'd like me to add the proposed the changes myself15:57
bogdandoWe could amend in a follow up perhaps15:58
bogdandoas there is a patch on top seems ready for merging15:58
bogdandofultonj: did I miss something? let me check...15:58
*** wolverineav has joined #tripleo15:58
*** moguimar has quit IRC15:59
*** moguimar has joined #tripleo15:59
openstackgerritRedHat RDO CI proposed openstack/tripleo-heat-templates stable/queens: GATE CHECK for TripleO  https://review.openstack.org/56722416:00
*** moguimar has quit IRC16:01
*** moguimar has joined #tripleo16:02
*** moshele has joined #tripleo16:04
*** ekultails has quit IRC16:04
*** jobewan has quit IRC16:05
*** ramishra has quit IRC16:05
*** rfolco is now known as rfolco|rover16:07
openstackgerritMerged openstack/tripleo-heat-templates stable/rocky: FFWD: Introduce workaround for neutron cisco plugin  https://review.openstack.org/61948416:08
openstackgerritMerged openstack/tripleo-heat-templates master: Create deployment/deprecated directory  https://review.openstack.org/63789016:08
openstackgerritMerged openstack/tripleo-heat-templates master: Deprecate Docker  https://review.openstack.org/63791516:08
openstackgerritMerged openstack/tripleo-common master: image-serve: redirect root to /v2  https://review.openstack.org/63740416:08
*** udesale has quit IRC16:11
dtantsurslagle: rings any bells? DeploymentError: Cannot find any hosts on 'overcloud' in network 'ctlplane'16:12
dtantsurthis is after wiring DeployedServerPortMap properly16:12
*** moshele has quit IRC16:12
*** quiquell is now known as quiquell|off16:13
*** artom has quit IRC16:14
*** hamzy has quit IRC16:15
dtantsurget_role_net_ip_map returns nothing?16:16
*** ekultails has joined #tripleo16:19
openstackgerritJohn Fulton proposed openstack/tripleo-quickstart-extras master: WIP/DNM: Add split-controlplane-standalone.yml playbook  https://review.openstack.org/61736816:20
openstackgerritJohn Fulton proposed openstack/tripleo-quickstart master: WIP/DNM: Add featureset 59 for split controlplane  https://review.openstack.org/61598716:22
openstackgerritJohn Fulton proposed openstack-infra/tripleo-ci master: WIP/DNM: Add split-controlplane standalone non-voting job definition  https://review.openstack.org/61598816:22
openstackgerritEmilien Macchi proposed openstack/python-tripleoclient master: container image build: introduce --use-buildah  https://review.openstack.org/63432216:23
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci master: Add fedora-28 support to container-build job on fedora-28 with Buildah  https://review.openstack.org/63599116:23
*** sai_p_ has joined #tripleo16:26
slagledtantsur: not really. what layer of the onion did that come from? heat, mistral, client?16:27
dtantsurslagle: it's in tripleoclient. I inserted a few print's to get a better idea.16:28
* dtantsur waits for the heat stack to be created16:28
slagledtantsur: yea i was wondering if it might be from the enable_ssh_admin function16:29
dtantsurslagle: it's surely from somewhere around. it worked correctly before I added DeployedServerPortMap though.16:29
mariosfultonj: panda|ruck rfolco|rover wdyt about disable scen1/4 standalone @ https://review.openstack.org/638363 & revert in https://review.openstack.org/638364 cc weshay16:29
mariosfultonj: panda|ruck rfolco|rover weshay fultonj patch depends on the disable thing, then the revert depends on the last of fultonj patches to merge which fix the job again16:30
mariospanda|ruck: rfolco|rover weshay is there a better way16:30
fultonjmarios: thanks for that16:31
panda|ruckmarios: isnt' enough to mark the job as non-voting ?16:31
*** wolverineav has quit IRC16:31
mariospanda|ruck: yeah and remove from gate16:31
slagledtantsur: that could likely mean that it didn't find any matches for the short hostnames in the DeployedServerPortMan value, so it assigned no ip's16:31
jaosoriormarios: why does ceph bluestore need the disabling of those scenarios?16:31
mariospanda|ruck: cos non voting in gate no16:31
panda|ruckmarios: maybe not very efficient I realize ..16:31
fultonjjaosorior: bluestore requires a change in tripleo ci and tht16:32
fultonjuntil both lands things will be broken16:32
mariosjaosorior: see commit message https://review.openstack.org/#/c/638363/16:32
mariosjaosorior: 2 patches, tht and tripleo-ci. either merges breaks the jobs16:32
mariosjaosorior: so we need both for green jobs again16:32
jaosoriormarios: understood16:32
fultonjbluestore requires the disk be prepared differently (tripleo-ci prepares the disk) and tht changes the scenario to bluestore16:32
dtantsurslagle: probably. does this look sane? http://paste.openstack.org/show/745629/16:32
*** kopecmartin is now known as kopecmartin|off16:33
slagledtantsur: no :)16:34
slaglefixed_ips should be a list of dicts16:34
slagleeg, ip_address: "1.1.1.1/24"16:35
slagledtantsur: check the example again at https://docs.openstack.org/tripleo-docs/latest/install/advanced_deployment/deployed_server.html#network-configuration16:35
dtantsurdaaaaamn. that was stupid :)16:35
*** ksambor has quit IRC16:35
slagleit's that way b/c the structure has to match the attribute structure of the OS::Neutron::Port resource so that it's a pluggable replacement16:36
openstackgerritBrent Eagles proposed openstack/puppet-tripleo stable/rocky: Change haproxy mode to http for octavia  https://review.openstack.org/63845616:36
*** florianf has quit IRC16:36
openstackgerritBrent Eagles proposed openstack/puppet-tripleo stable/queens: Change haproxy mode to http for octavia  https://review.openstack.org/63845716:36
mariosfultonj: well lets give it till tomorrow see if there is any other way suggested/comments otherwise we'll make it so :) or do it today anyway maybe ad the depends on your patch so is clearer?16:37
dtantsurslagle: right, yeah. I noticed that bit and immediately forgot about it16:37
mariosfultonj: and we can go for friday merge16:37
fultonjmarios: sounds good to me, thank you16:37
mariosfultonj: with input from panda|ruck and rfolco|rover on state of jobs so we hopefully get a smooth chain16:37
fultonji will add the changes you suggested w/ depends-on16:37
mariosfultonj: cool16:37
mariosjaosorior: fyi ^ thanks for looking here16:38
mariosjaosorior: hopefully minimal impact if we get the chain right and things aren't blocked on merge16:39
panda|ruckmarios: how did you know about bluesomething ?16:39
fultonjpanda|ruck: bluesomething?16:39
mariospanda|ruck: heh, just from code review and fultonj ping about the ceph-loop role16:40
*** hamzy has joined #tripleo16:40
*** suuuper has quit IRC16:40
*** hjensas has quit IRC16:40
jaosoriormarios: ack16:40
panda|ruckfultonj: bluething  ? :)16:40
*** jobewan has joined #tripleo16:40
fultonjhttps://ceph.com/community/new-luminous-bluestore16:41
mariospanda|ruck: it was fultonj suggestion in fact to do the temp disable see https://review.openstack.org/#/c/637196/3//COMMIT_MSG@11 discussion16:41
*** hberaud is now known as hberaud|school-r16:41
fultonjso if anything goes wrong you can blame me :)16:43
*** ekultails has quit IRC16:43
openstackgerritDmitry Tantsur proposed openstack/tripleo-common master: nova-less-deploy: implement deploy_roles workflow  https://review.openstack.org/63719116:43
mariosfultonj: exactly!16:43
mariosi mean unless gfidente is around16:43
mariosbut he isn't16:43
mariosso...16:43
mariossorry fultonj16:43
marios:/16:43
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: Podman support in haproxy-public-tls-inject  https://review.openstack.org/63824616:43
fultonjwell, i'm trying to be his backup in more than once sense marios ;)16:44
fultonjhe's not here ATM so you can blame me16:44
*** yprokule has quit IRC16:45
*** ekultails has joined #tripleo16:45
*** hberaud|school-r is now known as hberaud16:47
openstackgerritJohn Fulton proposed openstack-infra/tripleo-ci master: WIP/DNM: Add split-controlplane standalone non-voting job definition  https://review.openstack.org/61598816:49
*** jfrancoa has quit IRC16:51
openstackgerritJohn Fulton proposed openstack/tripleo-heat-templates master: Switch scenario00{1,4}-standalone to Ceph bluestore  https://review.openstack.org/54768216:53
*** khyr0n has joined #tripleo16:53
fultonjmarios: ^ I think i got the commit changes you requested ^16:53
*** khyr0n has quit IRC16:54
*** khyr0n has joined #tripleo16:55
mariosfultonj: ack tomorrow morning16:55
mariosfultonj: will check on reviews and see comments etc by then16:56
*** d4c07 has quit IRC16:56
*** gfidente has joined #tripleo16:57
fultonjgfidente: fyi https://review.openstack.org/#/c/63836416:58
dtantsurslagle: that worked, you rock! :)16:58
fultonjplanning to land tomorrow16:58
slagledtantsur: :-)16:58
gfidentefultonj ack voting16:59
fultonjgfidente: so temporarily disable 001/004 to land both tripleo-ci + tht chnage and then renable them16:59
gfidenteyeah16:59
dtantsurslagle: small issue: we do not abort immediately if we cannot upload the generated SSH keys (e.g. when we cannot log into the deployed servers)17:00
*** vinaykns has quit IRC17:01
openstackgerritJohn Fulton proposed openstack-infra/tripleo-ci master: WIP/DNM: Add split-controlplane standalone non-voting job definition  https://review.openstack.org/61598817:01
slagledtantsur: ok. we should. i know we've disallowed password auth, so ssh should fail quickly17:02
dtantsuryeah, but we proceed for some reason and fail later in ansible17:02
slagleok. sounds like a new case17:02
openstackgerritJohn Fulton proposed openstack-infra/tripleo-ci master: WIP/DNM: Add split-controlplane standalone non-voting job definition  https://review.openstack.org/61598817:03
*** _fragatina has joined #tripleo17:03
openstackgerritJohn Fulton proposed openstack-infra/tripleo-ci master: Add LVM option for ceph-loop-device role  https://review.openstack.org/63719617:04
*** vinaykns has joined #tripleo17:06
*** jpich has quit IRC17:06
*** pcaruana has quit IRC17:07
fultonjmarios: can we actually have a tripleo-ci patch depends-on a tripleo-ci patch?17:07
fultonji think we need to combine https://review.openstack.org/#/c/637196 and https://review.openstack.org/#/c/638363 into the same patch17:08
vinayknsmwhahaha: need review for https://review.openstack.org/#/c/600843/17:08
fultonjand then not have a revert but a new patch17:08
vinayknsmwhahaha: not sure why CI is failing..?17:08
fultonjmarios: gfidente ^ is that right because you can't have a dpeends-on to the same project; both are changes to openstack-infra/tripleo-ci17:09
*** _fragatina has quit IRC17:09
gfidentefultonj yeah we can't they should just be stacked17:09
*** _fragatina has joined #tripleo17:09
fultonjgfidente: yeah i could stack too so they're too reviews17:10
fultonjtwo* reiviews17:10
fultonj(gfidente taught me that)17:11
fultonjmarios: if you could ack that would be good, i'll goahead with the change either way17:12
*** bogdando has quit IRC17:12
openstackgerritJohn Fulton proposed openstack-infra/tripleo-ci master: Add LVM option for ceph-loop-device role  https://review.openstack.org/63719617:15
*** moguimar has quit IRC17:17
*** thrash is now known as thrash|biab17:19
openstackgerritJill Rouleau proposed openstack/tripleo-heat-templates master: flatten gnocchi services config  https://review.openstack.org/63846417:20
*** cylopez has quit IRC17:21
*** morazi has quit IRC17:22
fultonjgfidente: i stacked them. care to vote? https://review.openstack.org/#/c/637196/ https://review.openstack.org/#/c/547682/17:22
*** hberaud is now known as hberaud|gone17:23
dtrainorI'm using a downstream rocky with the SNMP power driver and after what appears to be an otherwise successful introspection, my nodes are stuck in 'enroll' state.  I went through the troubleshooting guide, I double checked my node port definitions, and they seem fine.  The guide suggests wrong IPMI information being used, but the node does power up / down as expected during introspection.17:23
*** wolverineav has joined #tripleo17:26
*** dtantsur is now known as dtantsur|afk17:29
*** dxiri has quit IRC17:35
*** ccamacho has quit IRC17:36
*** trown is now known as trown|lunch17:38
*** artom has joined #tripleo17:39
dtrainorthis might be a potential clue but I don't know where it points me:  DEBUG nova.virt.ironic.driver [req-efb3355a-7389-4527-89bb-029c8fb2acdd - - - - -] Node 70e313ca-6d26-4c2e-8c32-b0c07519d59b is not ready for a deployment, reporting resources as reserved for it. Node's provision state is enroll, power state is power off and maintenance is False. update_provider_tree /usr/lib/python2.7/site-packages/nova/virt/ironic/driver.py:80217:41
*** moshele has joined #tripleo17:41
*** zzzeek has quit IRC17:44
*** moshele has quit IRC17:47
*** panda|ruck is now known as panda|ruck|off17:48
*** wolverineav has quit IRC17:48
*** zzzeek has joined #tripleo17:49
*** shardy has quit IRC17:49
*** sdake has quit IRC18:05
*** sdake has joined #tripleo18:05
*** hjensas has joined #tripleo18:06
*** wolverineav has joined #tripleo18:06
openstackgerritNagasai Vinaykumar Kapalavai proposed openstack/instack-undercloud stable/ocata: Install python-panko client  https://review.openstack.org/62016618:09
*** thrash|biab is now known as thrash18:12
openstackgerritDmitry Tantsur proposed openstack/tripleo-common master: nova-less-deploy: implement deploy_roles workflow  https://review.openstack.org/63719118:23
*** jpena is now known as jpena|off18:29
openstackgerritwes hayutin proposed openstack/tripleo-common master: Run keypair creating as root  https://review.openstack.org/63684418:30
openstackgerritEmilien Macchi proposed openstack/tripleo-common stable/queens: Ansible json error callback plugin  https://review.openstack.org/61912918:30
*** moshele has joined #tripleo18:36
openstackgerritMerged openstack/tripleo-heat-templates master: Handle upper and lower case system uuids  https://review.openstack.org/63798418:36
openstackgerritMerged openstack/tripleo-heat-templates stable/queens: Add missing RoleParameters and ServiceNames  https://review.openstack.org/63816418:36
openstackgerritLuke Short proposed openstack/tripleo-common stable/queens: Ansible json error callback plugin  https://review.openstack.org/61912918:38
*** moshele has quit IRC18:39
*** morazi has joined #tripleo18:48
mwhahahavinaykns: rdo has issues, i'll check if it's related18:50
vinayknsmwhahaha: thx..!!18:50
*** trown|lunch is now known as trown18:52
*** wolverineav has quit IRC18:54
openstackgerritMerged openstack/tripleo-heat-templates stable/rocky: Add missing RoleParameters and ServiceNames  https://review.openstack.org/63816318:55
openstackgerritMerged openstack/tripleo-heat-templates master: Deprecate xinetd service management  https://review.openstack.org/63703518:55
*** holser_ has quit IRC18:55
*** gbarros has quit IRC18:57
*** moshele has joined #tripleo19:02
*** mcornea has quit IRC19:02
*** mcornea has joined #tripleo19:02
jidarhey guys, is ceph-ansible or ceph in general just broken on UPDATE ?19:04
fultonjjidar: no19:04
jidarI can not, for some reason get a deploy to UPDATE without it halting at Step219:04
jidarfultonj: :(19:04
fultonjjidar: did you debug it to see what ceph-ansible was doing?19:05
fultonja stack update should re-run the ceph-ansible playbook19:05
jidarhttps://gist.github.com/d3611a4a7a2efb41a1b6c51138218f0f19:05
jidarfultonj: ceph finishes without error19:05
jidarbut after it finishes, step2 never finishes up19:06
*** khyr0n has quit IRC19:06
fultonjjidar: ceph-ansible finishes fine but then heat times out waiting for it to ack?19:06
fultonjfor ceph-asnible to say "i'm done now"19:06
fultonjjidar: so your ceph-ansible logs confirm the ceph-ansible update completed?19:07
jidarhttps://gist.github.com/85f77b41dd056666ec2a888ee5ef6e6a19:07
jidaryes ^19:07
fultonjjidar: good19:07
*** khyr0n has joined #tripleo19:07
fultonjso your issue is mistral heat ack'ing to heat that step2 is done and it can move on19:08
jidarI've _tried_ for hours to figure out what mistral is doing19:08
jidarthe best I get is a workflow saying "it's working" and then "it failed" with about that much detail19:08
fultonjhttps://bugzilla.redhat.com/show_bug.cgi?id=156586119:08
openstackbugzilla.redhat.com bug 1565861 in ceph-ansible "enable_ssh_admin task unable to signal undercloud due to SSL handshake error" [Low,Closed: notabug] - Assigned to johfulto19:08
fultonjjidar: see comment #6 to see if these symptoms apply to you19:09
fultonjhttps://review.openstack.org/#/c/588216/19:10
fultonjjidar: if that's your issue see ^19:12
jidarI'm using the default generate_service_certificate = false (its commented) in my undercloud.conf and I'm not including injest-trust-anchor.yaml in my deploy.sh19:12
fultonjjidar: so your undercloud.conf has '# generate_service_certificate = false'19:13
fultonjplease confirm ^19:13
fultonjbecause the default is true, you have to set it to false if you don't want it19:13
fultonjthe default became true in 1319:13
fultonjaka queens19:13
jidaroh sorry, its commented out https://gist.github.com/44656b8ffba6235f4f0d87dac311974819:14
jidarand this is queens19:14
fultonjjidar: that explains it19:14
fultonjuncomment and re-run your undercloud install19:14
jidartesting!19:15
fultonjor get a working trust anchor19:15
jidarunlikely for this customer19:15
fultonjhttps://review.openstack.org/#/c/588216/19:15
fultonjas per my comment on aug9 ^ i advocated that the defaults be eaiser19:15
jidar100% agree with you there hah19:16
*** wolverineav has joined #tripleo19:18
*** wolverineav has quit IRC19:19
openstackgerritAlex Schultz proposed openstack/tripleo-heat-templates stable/rocky: Handle upper and lower case system uuids  https://review.openstack.org/63849119:25
*** wolverineav has joined #tripleo19:28
openstackgerritAlex Schultz proposed openstack/tripleo-heat-templates stable/queens: Handle upper and lower case system uuids  https://review.openstack.org/63849419:29
*** moshele has quit IRC19:31
*** ccamacho has joined #tripleo19:31
*** wolverineav has quit IRC20:01
openstackgerritRonelle Landy proposed openstack-infra/tripleo-ci master: Add base job for TripleO CI baremetal jobs  https://review.openstack.org/63676320:04
openstackgerritBrent Eagles proposed openstack/tripleo-common master: Use Octavia's project for lb security group ops  https://review.openstack.org/63428620:08
jidarfultonj: no dice, it still hung after doing a undercloud install, and trying to re-deploy. I'll note I don't see the SSL errors proposed in the bug report anywhere in the controller os-collect-config logs, and the url it's using to post to is http. I'll also note that I see: ceph_base_ansible_workflow | tripleo.overcloud.workflow_tasks.step2, for every execution that's failed with no "updated at" time20:08
jidarqq though, do I need to blow away the overcloud after doing the undercloud install again?20:08
jidarcaus I'm about too anyway at this point :(20:09
openstackgerritBrent Eagles proposed openstack/tripleo-common master: Set selinux contexts on ansible generated configuration  https://review.openstack.org/63661020:14
*** dprince has quit IRC20:14
fultonjtry rebuilding uc + oc20:15
*** _fragatina has quit IRC20:20
openstackgerritLuke Short proposed openstack/tripleo-heat-templates stable/queens: Tag step plays  https://review.openstack.org/63850320:29
openstackgerritMerged openstack/python-tripleoclient master: Deprecate docker config options  https://review.openstack.org/63824320:29
*** wolverineav has joined #tripleo20:33
*** dprince has joined #tripleo20:36
*** wolverineav has quit IRC20:38
openstackgerritLuke Short proposed openstack/tripleo-heat-templates stable/queens: Remove "when failed" from debug task names  https://review.openstack.org/63850720:45
*** gfidente is now known as gfidente|afk20:52
openstackgerritJill Rouleau proposed openstack/tripleo-heat-templates master: flatten gnocchi services config  https://review.openstack.org/63846421:02
*** wolverineav has joined #tripleo21:06
*** wolverineav has quit IRC21:09
*** wolverineav has joined #tripleo21:10
*** rcernin has joined #tripleo21:11
*** dsneddon has quit IRC21:11
larsksHey folks, I have two tripleo standalone deployments on which something appears to be killing openvswitch (details @ http://lists.openstack.org/pipermail/openstack-discuss/2019-February/002954.html).  Has anyone else seen this?21:23
*** nkinder has joined #tripleo21:27
*** brault has joined #tripleo21:28
*** nkinder has quit IRC21:32
*** dsneddon has joined #tripleo21:32
*** wolverineav has quit IRC21:34
*** slaweq has quit IRC21:35
*** ansmith has quit IRC21:36
*** wolverineav has joined #tripleo21:39
*** wolverineav has quit IRC21:40
*** wolverineav has joined #tripleo21:40
*** wolverineav has quit IRC21:41
*** wolverineav has joined #tripleo21:41
*** artom has quit IRC21:42
*** slaweq has joined #tripleo21:48
*** Vorrtex has quit IRC21:54
openstackgerritEmilien Macchi proposed openstack/tripleo-common master: DNM - add --debug to buildah  https://review.openstack.org/63843321:57
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci master: Add fedora-28 support to container-build job on fedora-28 with Buildah  https://review.openstack.org/63599121:57
*** karthiks has quit IRC22:00
openstackgerritMichele Baldessari proposed openstack/tripleo-heat-templates master: WIP Do not bind mount the ovs_db folder  https://review.openstack.org/63851822:06
*** trown is now known as trown|outtypewww22:07
openstackgerritNagasai Vinaykumar Kapalavai proposed openstack/instack-undercloud stable/pike: Install python-panko client  https://review.openstack.org/63851922:10
*** ansmith has joined #tripleo22:11
openstackgerritAde Lee proposed openstack/tripleo-heat-templates stable/rocky: Move ipa enrollment to host_prep_tasks  https://review.openstack.org/63852122:15
*** jcoufal has quit IRC22:19
*** khyr0n has quit IRC22:20
*** hamzy has quit IRC22:21
*** khyr0n has joined #tripleo22:22
*** artom has joined #tripleo22:26
*** _fragatina has joined #tripleo22:26
*** sdake has quit IRC22:27
*** cfontain has quit IRC22:28
*** cfontain has joined #tripleo22:29
openstackgerritAde Lee proposed openstack/tripleo-heat-templates stable/queens: Move ipa enrollment to host_prep_tasks  https://review.openstack.org/63852322:30
openstackgerritMerged openstack/tripleo-heat-templates master: Add specific upgrade hiera data file.  https://review.openstack.org/63102522:30
openstackgerritMerged openstack/tripleo-quickstart master: Fix freeipa role for quickstart  https://review.openstack.org/63830622:30
*** wolverineav has quit IRC22:30
*** wolverineav has joined #tripleo22:31
*** cfontain has quit IRC22:33
*** wolverineav has quit IRC22:36
*** _fragatina_ has joined #tripleo22:37
*** jtomasek has quit IRC22:37
tonybHas anyone seen the config download phase stop/exit without an error?22:38
*** _fragatina has quit IRC22:40
*** gfidente|afk has quit IRC22:40
*** _fragatina has joined #tripleo22:42
*** slaweq has quit IRC22:43
*** _fragatina_ has quit IRC22:45
openstackgerritMerged openstack/tripleo-common master: image_uploader: conditionally import docker  https://review.openstack.org/63823822:48
openstackgerritMerged openstack/tripleo-common master: kolla_builder: don't build template mapping when not needed  https://review.openstack.org/63765222:48
openstackgerritMerged openstack/tripleo-quickstart-extras master: Run dstat prior deploying to standalone  https://review.openstack.org/63815922:48
openstackgerritMerged openstack/tripleo-quickstart-extras master: Re-enable neutron-tempest-plugin scenario tests  https://review.openstack.org/63515222:49
openstackgerritMerged openstack-infra/tripleo-ci master: kolla-build: remove quotes in namespace config  https://review.openstack.org/63841922:49
*** wolverineav has joined #tripleo22:49
openstackgerritMerged openstack/tripleo-quickstart-extras master: ensure pip is updated in reproducer  https://review.openstack.org/63801622:51
openstackgerritMerged openstack/tripleo-quickstart-extras master: standalone: enable debug on container image build  https://review.openstack.org/63823422:51
*** rh-jelabarre has quit IRC22:52
*** slaweq has joined #tripleo22:56
*** slaweq has quit IRC23:00
*** tkajinam has joined #tripleo23:01
sshnaidmweshay, rlandy https://review.openstack.org/#/c/63843823:03
openstackgerritCarlos Goncalves proposed openstack/diskimage-builder master: [wip] rhel8 beta support  https://review.openstack.org/62313723:05
*** jobewan has quit IRC23:07
*** rascasoft has quit IRC23:10
*** cfontain has joined #tripleo23:10
*** slaweq has joined #tripleo23:11
*** tosky has quit IRC23:11
*** percevalbot has joined #tripleo23:11
*** sdake has joined #tripleo23:12
*** _fragatina has quit IRC23:15
*** _fragatina has joined #tripleo23:15
*** slaweq has quit IRC23:16
rlandylooking into it23:17
*** _fragatina has quit IRC23:20
*** _fragatina has joined #tripleo23:23
*** sdake has quit IRC23:33
*** rh-jelabarre has joined #tripleo23:35
*** sdake has joined #tripleo23:38
*** raildo has quit IRC23:39
*** cfontain has quit IRC23:44
*** sdake has quit IRC23:46
*** sdake has joined #tripleo23:47
*** rh-jelabarre has quit IRC23:48
*** sdake has quit IRC23:51
*** sdake_ has joined #tripleo23:52
*** wolverineav has quit IRC23:57
*** ekultails has quit IRC23:57

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!