Thursday, 2017-06-22

openstackgerritDan Sneddon proposed openstack/tripleo-heat-templates master: Create NIC templates for Neutron Networker composable role  https://review.openstack.org/40039000:00
openstackgerritDan Sneddon proposed openstack/tripleo-heat-templates master: Fix descriptions on bonding templates  https://review.openstack.org/42867200:09
*** limao has quit IRC00:17
openstackgerritGabriele Cerami proposed openstack/tripleo-quickstart master: WIP: enable container periodic job  https://review.openstack.org/47574300:32
openstackgerritGabriele Cerami proposed openstack/tripleo-quickstart-extras master: WIP: enable containers periodic job  https://review.openstack.org/47574500:32
*** mdnadeem has joined #tripleo00:39
openstackgerritDan Sneddon proposed openstack/tripleo-heat-templates master: Convert role templates to consume roles_data map  https://review.openstack.org/46985300:59
openstackgerritDan Sneddon proposed openstack/tripleo-heat-templates master: Convert role templates to consume roles_data map  https://review.openstack.org/46985300:59
openstackgerritDan Sneddon proposed openstack/tripleo-heat-templates master: Convert role templates to consume roles_data map  https://review.openstack.org/46985301:04
*** achadha has quit IRC01:12
*** fzdarsky_ has joined #tripleo01:13
openstackgerritDan Sneddon proposed openstack/tripleo-heat-templates master: Convert role templates to consume roles_data map  https://review.openstack.org/46985301:14
*** achadha has joined #tripleo01:16
*** achadha has quit IRC01:17
*** fzdarsky has quit IRC01:17
openstackgerritJamie Lennox proposed openstack/diskimage-builder master: Use the dib python to do cleanup  https://review.openstack.org/47630201:20
*** rbrady has quit IRC01:30
*** rbrady has joined #tripleo01:32
*** limao has joined #tripleo01:35
*** chlong has joined #tripleo01:43
*** dmacpher has quit IRC01:47
*** pliu has quit IRC01:47
*** myoung has quit IRC01:47
*** dmacpher has joined #tripleo01:47
*** cschwede has quit IRC01:47
*** jschlueter has quit IRC01:47
*** lhinds has quit IRC01:47
*** rnoriega has quit IRC01:47
*** adarazs has quit IRC01:48
*** faceman has quit IRC01:49
*** marios has quit IRC01:49
*** MrRoot has quit IRC01:50
*** achadha has joined #tripleo01:52
*** limao has quit IRC01:52
*** limao has joined #tripleo01:54
*** achadha has quit IRC01:56
openstackgerritSteve Baker proposed openstack/python-tripleoclient master: Add list options to overcloud container image build  https://review.openstack.org/47115701:58
*** ansmith has quit IRC01:59
*** yamahata_ has quit IRC02:01
*** gbarros has joined #tripleo02:02
*** limao has quit IRC02:03
*** limao has joined #tripleo02:04
*** chlong has quit IRC02:04
*** apetrich has quit IRC02:06
*** MrRoot has joined #tripleo02:07
*** apetrich has joined #tripleo02:11
*** eck` is now known as eck`gone02:12
*** gbarros has quit IRC02:19
*** chlong has joined #tripleo02:36
*** gbarros has joined #tripleo02:38
*** limao has quit IRC03:02
*** limao has joined #tripleo03:02
*** rlandy|bbl is now known as rlandy03:04
*** rlandy has quit IRC03:04
*** psahoo has joined #tripleo03:12
*** limao has quit IRC03:23
*** limao has joined #tripleo03:23
*** udesale has joined #tripleo03:33
*** limao has quit IRC03:44
*** limao has joined #tripleo03:44
*** rwsu has joined #tripleo03:49
*** atheurer has quit IRC03:54
*** yamahata_ has joined #tripleo03:59
*** chlong has quit IRC04:02
*** rhallisey has quit IRC04:06
*** axisys has quit IRC04:07
*** gkadam has joined #tripleo04:09
*** limao has quit IRC04:11
*** gbarros has quit IRC04:15
*** chlong has joined #tripleo04:15
*** achadha has joined #tripleo04:17
*** links has joined #tripleo04:21
*** achadha has quit IRC04:22
openstackgerritIan Wienand proposed openstack/diskimage-builder master: Additional mount order unit test  https://review.openstack.org/47634004:28
*** MrRoot has quit IRC04:37
*** ykarel has joined #tripleo04:39
*** ykarel_ has joined #tripleo04:44
*** ykarel has quit IRC04:46
*** janki has joined #tripleo04:50
*** ykarel_ has quit IRC04:56
*** ykarel_ has joined #tripleo04:57
*** limao has joined #tripleo05:09
*** limao has quit IRC05:09
*** limao has joined #tripleo05:09
*** jaganathan has quit IRC05:10
*** jaganathan has joined #tripleo05:10
*** agurenko has joined #tripleo05:14
*** skramaja has joined #tripleo05:15
*** rwsu has quit IRC05:19
*** jbadiapa_ has quit IRC05:19
*** jaganathan has quit IRC05:21
*** axisys has joined #tripleo05:24
*** marios has joined #tripleo05:26
*** jprovazn has joined #tripleo05:28
*** ykarel_ has quit IRC05:34
*** ykarel_ has joined #tripleo05:35
*** prateek has joined #tripleo05:36
*** achadha has joined #tripleo05:36
*** limao_ has joined #tripleo05:37
*** limao has quit IRC05:37
*** dmacpher has quit IRC05:38
*** achadha has quit IRC05:40
*** pgadiya has joined #tripleo05:43
*** masco has joined #tripleo05:47
*** dsariel has quit IRC05:49
*** limao_ has quit IRC05:49
*** limao has joined #tripleo05:49
*** dmacpher has joined #tripleo05:51
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo master: Always start httpd at the same time  https://review.openstack.org/47611905:53
*** ccamacho has quit IRC05:56
*** achadha has joined #tripleo05:57
*** achadha has quit IRC06:00
*** achadha has joined #tripleo06:00
*** yprokule has joined #tripleo06:01
*** dpawar has joined #tripleo06:01
*** fzdarsky_ has quit IRC06:03
*** limao has quit IRC06:06
*** dmacpher has quit IRC06:06
*** dmacpher has joined #tripleo06:07
*** limao has joined #tripleo06:07
*** limao_ has joined #tripleo06:14
*** limao has quit IRC06:15
*** limao_ has quit IRC06:20
*** limao has joined #tripleo06:21
*** limao_ has joined #tripleo06:22
*** achadha has quit IRC06:22
*** prateek_ has joined #tripleo06:23
*** limao has quit IRC06:25
*** prateek has quit IRC06:25
*** rcernin has joined #tripleo06:30
*** artom has joined #tripleo06:30
*** mrunge_ is now known as mrunge06:34
*** limao has joined #tripleo06:35
*** limao_ has quit IRC06:35
*** dsariel has joined #tripleo06:36
*** rcernin has quit IRC06:37
*** pliu has joined #tripleo06:40
*** marios has quit IRC06:42
*** marios has joined #tripleo06:42
*** marios has quit IRC06:43
*** marios has joined #tripleo06:44
*** prateek__ has joined #tripleo06:44
*** jaganathan has joined #tripleo06:46
*** ffiore has joined #tripleo06:46
*** prateek_ has quit IRC06:46
*** mcornea has joined #tripleo06:48
*** dbecker has joined #tripleo06:51
*** rcernin has joined #tripleo06:51
*** jbadiapa_ has joined #tripleo06:56
*** brault has joined #tripleo06:56
*** jaganathan has quit IRC06:57
*** jaganathan has joined #tripleo06:57
*** dbecker has quit IRC07:00
*** limao has quit IRC07:02
*** tesseract has joined #tripleo07:02
*** limao has joined #tripleo07:02
*** limao has quit IRC07:04
*** limao has joined #tripleo07:04
openstackgerritDnyaneshwar proposed openstack/puppet-tripleo master: Add new profile for the Veritas HyperScale's cinder backend.  https://review.openstack.org/47576207:05
*** pcaruana has joined #tripleo07:06
*** bogdando has joined #tripleo07:07
*** gkadam is now known as gkadam-afk07:08
openstackgerritXinliang Liu proposed openstack/diskimage-builder master: Fix mkfs use wrong label option for vfat  https://review.openstack.org/47638207:10
*** jlinkes has joined #tripleo07:12
openstackgerritDnyaneshwar proposed openstack/puppet-tripleo master: Add new profile for the Veritas HyperScale's cinder backend.  https://review.openstack.org/47576207:12
*** f13o has joined #tripleo07:14
*** yprokule_ has joined #tripleo07:18
*** sid1 has joined #tripleo07:19
openstackgerritDnyaneshwar proposed openstack/puppet-tripleo master: Add new profile for the Veritas HyperScale's cinder backend.  https://review.openstack.org/47576207:19
*** yprokule has quit IRC07:21
*** yprokule_ is now known as yprokule07:21
*** sid1 has quit IRC07:21
*** jpich has joined #tripleo07:22
*** aufi has joined #tripleo07:25
*** ebarrera has joined #tripleo07:27
*** ccamacho has joined #tripleo07:28
*** gfidente has joined #tripleo07:31
*** gfidente has quit IRC07:31
*** gfidente has joined #tripleo07:31
*** mrchao has joined #tripleo07:33
*** dparkes has joined #tripleo07:33
*** dtantsur|afk is now known as dtantsur07:35
*** egonzalez has joined #tripleo07:38
openstackgerritSagi Shnaidman proposed openstack/tripleo-quickstart-extras master: Generate ARA statistics  https://review.openstack.org/47639207:38
*** apetrich has quit IRC07:39
*** amoralej|off is now known as amoralej07:40
*** jpena|off is now known as jpena07:42
*** prateek__ is now known as prateek07:45
*** oidgar has joined #tripleo07:46
*** anshul has joined #tripleo07:48
*** sid1 has joined #tripleo07:48
bogdandosshnaidm: wrt07:51
bogdando> ovb-ha-oooq job running both in gates and periodic jobs07:51
bogdandodo we have similar for containers?07:51
sshnaidmbogdando, no, containers run only on patches07:51
sshnaidmbogdando, no problem to make periodic, but no much sense too..07:52
*** bnemec has quit IRC07:52
sshnaidmbogdando, usually gate jobs are not shown in health07:52
bogdandosshnaidm: not sure. Devs wouldn't like to monitor gates, but periodic - could be. The latter is like ALARM ring for all07:52
bogdandoso would be nice to have a guard ringing it (rss feeds for all)07:53
sshnaidmbogdando, if devs don't monitor broken gates, they definitely will ignore periodic too :)07:53
bogdandothere is also cryptic scenario1-4 jobs07:53
bogdandosshnaidm: not really, the periodic checks the master or stable. While gates just make noise for non related patches, right?07:54
sshnaidmbogdando, some gate jobs are shown in health though, we try to check with mtreinish why it happens and why there are no ovb gate jobs07:54
bogdandowho is interested in listening that noise? I bet - no one07:54
sshnaidmbogdando, periodic jobs checks not master but consistent repo and promote it07:55
bogdandobut this not alwaus true for periodic jobs07:55
bogdandosshnaidm: well, do we need to have periodic jobs for master then?07:55
sshnaidmbogdando, no, we have gates for this07:55
bogdandoum07:55
sshnaidmbogdando, gate jobs check master every minute07:56
bogdandoisn't check / gate run against *patches* ??07:56
sshnaidmbogdando, yep07:56
bogdandosshnaidm: oh, I didn't know that!07:56
sshnaidmbogdando, in the patches I mean07:56
bogdandosshnaidm: that's not exactly that contributors would like to see I suppose07:57
bogdandoanyone who wants to help would first start monitor that merged and fails periodic jobs, like regressions introduced07:57
bogdandoIMO\07:57
sshnaidmbogdando, anybody who is looking at http://status-tripleoci.rhcloud.com/ and see all containers job fail forever could suspect that something is wrong with it07:57
bogdandoactually, not really. that may be 99% failure of some random patch not passing07:58
bogdandofor 99% cases. And it is very opaque to distinguish it vs regressions in master07:58
sshnaidmbogdando, all patches in all repos not passing? not realistic07:58
sshnaidmbogdando, what is the problem you try to solve?07:59
bogdandojust shared my feedback. If there were rss feeds for periodic ovb/multinode containers jobs, I'd subscribe to that and inspect failures. But much less likely - for random patches failing07:59
*** zzzeek has quit IRC08:00
bogdandothe problem to solve is to attract contributors to guarding master against regressions08:00
bogdandofor*08:00
*** nyechiel_ has joined #tripleo08:00
bogdandoas a new comer, I can see there are many jobs in many repos, making noise. Nothing is clear08:00
*** zzzeek has joined #tripleo08:00
bogdandoregressions can't be seen easily08:00
bogdandoEmilienM: what do you think on that topic?..08:01
sshnaidmbogdando, ok, so we have lp bug crying here in channel every our about failing gate, we have escalations and trello cards and mails.. next step will be visit homes of devs with leaving post there..08:01
*** suuuper has joined #tripleo08:01
bogdandosshnaidm: the latter is rss :)08:02
bogdandoif only one could simply subscribe to things08:02
sshnaidmI think there are really a lot of communication channels where people can know about failure and adding another one is not really critical08:02
*** fzdarsky_ has joined #tripleo08:02
bogdandonot doing manual inspection of "alll read" reports08:02
bogdando"all red"08:02
sshnaidmthe fact that regressions are not solved doesn't mean people don't know about them08:03
bogdandosshnaidm: the use case - containers job is non voting now, so it's easy to miss a bad thing get into the master.08:03
bogdandoalarms , escalations, trello boards contain only 99% of noise wrt that use case specifically08:04
bogdandoand everyone who cares, will have to go through that noise and numerous red jobs (completely unrelated) each time08:04
bogdandowhat's a pain08:04
sshnaidmbogdando, no, escalations are done for critical problems only, no any noise there08:04
bogdandookay, so my case is seems non critical08:05
bogdandonon voting, after all =)08:05
bogdandobut still may be an area of interest of a team08:05
sshnaidmbogdando, containers job is escalated and cared by mandre, jistr08:05
bogdandoyeah, and I'm trying to make things easier to detect and alarm everyone who is interested, via rss08:05
bogdandosay, some periodic job is failing, this simple08:06
bogdandonot the gate08:06
sshnaidmbogdando, ok, but I don't think the problem is with alarming, from my experience I can say that nobody cares about periodic jobs08:06
hjensasOn topic of CI, I would like to see more jobs moved out of non-voting. Just this week I have bug/1699026, and c/437544/ full of issues. 1st is a regression, 2nd is probably du to a change in requirments merged monday.08:07
bogdandosshnaidm: Well sometimes "no one cares" == "no one understands and gives up"08:07
sshnaidmbogdando, not in this case08:07
sshnaidmbogdando, again, when gate job is broken it's alarmed in a lot of communication channels08:07
bogdandothe lack of info or than there is too much of the noice unrelated08:07
bogdandothen*08:07
sshnaidmbogdando, sorry, I don't understand what we are arguing about, if people want to check job they check, if not - no any rss,mail, etc will force them to do it08:08
bogdandoI think I failed to explain the case. Sorry for that. Perhaps someone else could do that better08:08
*** bnemec has joined #tripleo08:08
openstackgerritMerged openstack/tripleo-docs master: Add docs around updating undercloud ssh key  https://review.openstack.org/47627508:08
bogdandosshnaidm: I wanted to check why my deployment fails. It was cuz of containers images outdated08:09
sshnaidmhjensas, which jobs exactly?08:09
bogdandoif there were a periodic job checkin, I'd subscribe and get the root cause much earlier08:09
hjensasgate-instack-undercloud-docs-ubuntu-xenialFAILURE in 43s08:09
hjensasgate-instack-undercloud-pep8-ubuntu-xenialFAILURE in 29s08:09
hjensasgate-instack-undercloud-python27-ubuntu-xenialFAILURE in 1m 50s08:09
hjensasgate-instack-undercloud-python35FAILURE in 41s08:09
hjensassshnaidm: so for c/437544 - or instack-undercloud master, the tox stuff is broke.08:10
sshnaidmbogdando, if you present on this channel or subscribed to lp bugs or any internal maillist - you would know about this immediately08:10
bogdandoso the case is "people want to check, but there is too much of unrelated things, or missing check points at all"08:10
bogdandosshnaidm: the last two cases with outdated containers I've discovered on my own08:11
sshnaidmhjensas, hmm.. not sure I understand, these jobs actually vote, aren't they?08:11
bogdandothere were no alarms08:11
bogdandoI'm not sure why08:11
hjensassshnaidm: yes, these jobs vote. The other regression is the gate-tripleo-ci-centos-7-3nodes-multinode-nv job.08:11
bogdandoI believe we're not arguing, but I'm failing to explain my case :)08:12
sshnaidmbogdando, https://bugs.launchpad.net/tripleo/+bug/169906208:12
openstackLaunchpad bug 1699062 in tripleo "Containers CI broken by out of date images" [Critical,Fix released]08:12
bogdandoand that we really need periodic jobs for ovb-ha/multinode for containers ran against master08:12
hjensassshnaidm: so Issue 1. the regression would have been caught earlier if 3nodes-multinode was voting. :)08:12
bogdandosshnaidm: that was a two days after I was hit by the issue08:13
*** yprokule_ has joined #tripleo08:13
sshnaidmhjensas, what does it mean? gate-tripleo-ci-centos-7-3nodes-multinode-nv was broken because of code change?08:14
hjensassshnaidm: no composable roles in tripleo was broken.08:14
bogdandoweshay: could you share your thoughts on that topic please?08:14
sshnaidmbogdando, if people don't fill bugs, that's another issue.. how rss would help here?08:14
bogdandoweshay: "we really need periodic jobs for ovb-ha/multinode for containers ran against master" ?08:14
sshnaidmhjensas, and it was checked only in gate-tripleo-ci-centos-7-3nodes-multinode-nv?08:14
bogdandosshnaidm: rss would notify everyone subscribed, there is a regression. New one.08:15
bogdandosee, a job now started failing08:15
hjensassshnaidm: I think soo. (I am not very familiar with exactly what is tested in what jobs)08:15
bogdandoand not the one that is kept being red for weeks by now =)08:15
bogdandothe idea is to detect and notify the periodc job state changes08:16
bogdandonot keep ringing days after08:16
*** yprokule has quit IRC08:16
*** yprokule_ is now known as yprokule08:16
mcorneatherve: jistr hi, good morning! I have a reproducing environment with a timed out upgrade. Do you still need it?08:17
hjensassshnaidm: the other issue. Voting jobs start failing for instack-undercloud with no code change. I belive this is because of commit 053d3eaf771d29372497c186dac672edde03e328 in requirements. (A periodic job on master would have detected this?)08:17
*** snecklifter has joined #tripleo08:17
thervemcornea, Yes please!08:17
bogdandosshnaidm: basically, an alarm is good only when it starts, a minute after that it's being shut down or silenced or ignored08:18
sshnaidmhjensas, yeah, it happens, and periodic can't do nothing with it, it could be broken also by any other involved project - nova,ironic,neutron, etc08:19
bogdandoand I hoped to get a tool for early notification, via rss, and periodic jobs08:20
hjensassshnaidm: yes any involved project could break it. But we don't detect it before someone tries to push a change.08:20
bogdandowith a particular scope only08:20
bogdandolike those "ovb-ha/multinode for containers ran against master"08:20
sshnaidmbogdando, when I want to see why my job fails and if it's known issue, I firstly look at status page and if there are long list of failing jobs with the same error - most likely it's the same bug, then it's better to look in lp if it's opened already, in channel - if there are alarms, and if not - you won the opportunity to fill the bug first.08:21
bogdandosshnaidm: yeah, that's clear. Just a long list and many actions08:21
bogdandoI hoped to bring shortcuts by subject areas08:22
sshnaidmhjensas, we work on this now, we have a few projects that run our jobs and now encourage people from related fields to watch them, like networking folks to watch tripleo jobs on neutron etc08:22
bogdandoscoped jobs and notifications, individual, for subscribers08:22
bogdandoand focused on regressions, not patches08:22
bogdandothere are too much of status pages and cryptic jobs... https://status-tripleoci.rhcloud.com/ and http://tripleo.org/cistatus.html and http://status.openstack.org/openstack-health/#/g/project/openstack-infra~2Ftripleo-ci08:24
sshnaidmbogdando, I personally think that problem with long time failing jobs is not related to their visibility, but I can be wrong of course08:24
bogdandosshnaidm: long time failing jobs is another story08:24
sshnaidmbogdando, it's not too much, it's only 3 :)08:24
bogdandoI'm tryin to improve the very first failure vuz of a newly discovered regression case08:25
bogdandosshnaidm: and each has tons of jobs :)08:25
bogdandoto be inspected cross repos for potential long running regressions08:25
sshnaidmbogdando, now you try to solve some other problem08:26
bogdandothat's the same one, just trying to explain better :D08:26
*** limao has quit IRC08:27
bogdandoestablishing early and scoped alarms via subscriptions to scoped jobs, that's the case08:27
*** limao has joined #tripleo08:27
bogdandoscoped periodic jobs for master or stable branches08:27
bogdandoonce again please, not jobs checking patches08:27
bogdando> that problem with long time failing jobs is not related to their visibility08:29
bogdandoright. But visibility issue makes them becoming long time failing!08:29
bogdandoand scoped by deployment cases periodic jobs would help to address that visibility issue, that's my point08:30
*** sid1 has quit IRC08:30
*** sid1 has joined #tripleo08:31
bogdandoso there are only ovb-ha, and nonha-multinode-scenario1-4 jobs. That seems not enough to cover all cases, at least for containers sshnaidm08:31
bogdandoI meant periodic ones08:31
sshnaidmbogdando, current periodic jobs are for promotion mostly08:31
bogdandosshnaidm: that's also an issue, the scope of those jobs is not transparent for contributors08:32
bogdandoI know there is a matrix but...08:32
sshnaidmbogdando, what do you mean?08:32
bogdandolet's make it complete for non promotion cases as well08:32
bogdandoHA, containers, upgrades08:32
bogdandoneeds to be issues on periodic basis against masters08:33
*** lucas-afk is now known as lucasagomes08:33
bogdandoissued08:33
bogdandoand put into the test matrix08:33
sshnaidmbogdando, to make periodic master jobs for voting doesn't make any sense08:33
bogdandoso folks from teams could subscribe with rss or the like08:33
bogdandosshnaidm: for ealry alarming only08:33
bogdandothe state green to red changes08:34
*** b00tcat has joined #tripleo08:34
sshnaidmbogdando, still no any sense, broken gates are alerted immediately08:34
*** yamahata_ has quit IRC08:34
bogdandoif all is green, and one starts red, it's the way simple to inspect and fix than later, with other reds overlapping08:34
*** b00tcat_ has quit IRC08:34
sshnaidmbogdando, you can see in status page when exactly job started failing, it's much more helpful than "yesterday"08:35
bogdandosshnaidm: gates alerts come for patches08:35
bogdandoand contain unrelated noise08:35
*** fzdarsky_ has quit IRC08:35
sshnaidmbogdando, I don't understand which noise you are talking08:35
*** ckyriakidou has joined #tripleo08:35
sshnaidmbogdando, if gate job is broken, no any patch could be submitted, people start to complain immediately08:36
bogdandosshnaidm: right, I think I understood your point now08:36
bogdandoso then we should sed the history with s/periodic/gate/g :)08:37
bogdandosshnaidm: could you point me please containers gate jobs, ovb-ha/multinode to subscribe?08:37
bogdandocouldn't locate them on openstack-health page, for setting rss08:38
*** derekh has joined #tripleo08:38
sshnaidmbogdando, only *periodic* jobs are in health page08:39
sshnaidm<sshnaidm> bogdando, some gate jobs are shown in health though, we try to check with mtreinish why it happens and why there are no ovb gate jobs08:39
*** dmk0202 has joined #tripleo08:40
bogdandosshnaidm: ok, thanks08:40
*** gkadam-afk is now known as gkadam08:40
*** tosky has joined #tripleo08:40
*** fzdarsky has joined #tripleo08:43
*** zoli|gone is now known as zoli|wfh08:44
sshnaidmmandre, ping08:44
*** marios has quit IRC08:44
*** marios has joined #tripleo08:45
openstackgerritCarlos Camacho proposed openstack/tripleo-heat-templates master: Enable the HA deployment by default  https://review.openstack.org/35906008:47
*** f13o has quit IRC08:47
mandresshnaidm: hey08:50
sshnaidmmandre, hi, just curious if you can share your solution to last containers problem - how to build/update them and what was required to fix08:51
sshnaidmmandre, I hardly understand the process of updating containers08:52
*** milan has joined #tripleo08:52
mandresshnaidm: are you talking about the out of sync container images issue?08:54
sshnaidmmandre, I think so, the last problem of containers job08:55
*** cmyster has quit IRC08:55
*** cmyster has joined #tripleo08:56
*** adarazs has joined #tripleo08:56
mandresshnaidm: yeah, so if we're talking about the same thing, that was caused by the containers images used in CI being slightly too old, and the cinder container was at a different DB migration than the cinder running on the host08:57
mandresshnaidm: the solution here was to update the containers in the tripleoupstream dockerhub registry so new images are picked up in CI08:58
*** ckyriakidou has quit IRC08:59
mandresshnaidm: right now, we're still building/pushing images manually, either dprince or myself usually08:59
mandrebut we're talking about using the images built in the RDO pipeline09:00
sshnaidmmandre, so you build them with kolla and push them manually to dockerhub?09:00
mandresshnaidm: that's right, until now, but it's about to change09:00
*** limao_ has joined #tripleo09:00
*** limao has quit IRC09:02
*** f13o has joined #tripleo09:02
sshnaidmmandre, and how often will it built on rdo?09:03
mandresshnaidm: rdo builds them in the promotion pipeline, https://ci.centos.org/view/rdo/view/promotion-pipeline/job/rdo_trunk-promote-master-current-tripleo/09:06
mandresshnaidm: and as of now, they're uploaded with https://ci.centos.org/job/rdo-tripleo-containers-promote/09:06
mandresshnaidm: but I think the promotion is going to change a little bit, CC dmsimard panda09:07
openstackgerritRaoul Scarazzini proposed openstack/tripleo-docs master: Add multi-virtual undercloud guide  https://review.openstack.org/47422509:08
*** limao has joined #tripleo09:08
*** athomas has joined #tripleo09:10
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: DONT REVIEW: prepare subunit fail  https://review.openstack.org/47642709:10
*** limao_ has quit IRC09:11
*** dbecker has joined #tripleo09:11
*** ckyriakidou has joined #tripleo09:13
*** salmankhan has joined #tripleo09:13
*** b00tcat has quit IRC09:17
*** b00tcat has joined #tripleo09:17
*** limao has quit IRC09:19
*** limao_ has joined #tripleo09:19
*** ckyriakidou has quit IRC09:24
*** dtantsur is now known as dtantsur|bbl09:25
*** athomas has quit IRC09:25
*** limao has joined #tripleo09:26
*** akrivoka has joined #tripleo09:26
*** limao_ has quit IRC09:27
*** dbecker has quit IRC09:29
*** apetrich has joined #tripleo09:29
*** limao has quit IRC09:29
*** cylopez has joined #tripleo09:40
*** dmk0202 has quit IRC09:40
*** mdnadeem has quit IRC09:45
openstackgerritAbhishek Kane proposed openstack/tripleo-heat-templates master: Add composable service for the Veritas HyperScale's cinder backend.  https://review.openstack.org/47576509:51
*** karim has joined #tripleo09:54
*** karim is now known as Guest4614809:55
*** Guest46148 is now known as karimb09:55
openstackgerritSagi Shnaidman proposed openstack/tripleo-quickstart-extras master: Generate ARA statistics  https://review.openstack.org/47639209:57
*** r2bit__ is now known as r2bit09:57
*** dbecker has joined #tripleo10:01
*** jlinkes has quit IRC10:02
*** SlickNik has quit IRC10:03
pandasshnaidm: a periodic job will build containers, test the deployment with them, then upload to rdo registry. For the gate part, which requires an update on all containers involved, we'll build a container with a mirror of the delorean hash, then launch a script that updates all the containers using the mirror.10:03
*** rhallisey has joined #tripleo10:04
*** rhallisey has quit IRC10:04
*** milan has quit IRC10:04
*** SlickNik has joined #tripleo10:04
*** rhallisey has joined #tripleo10:05
openstackgerritCarlos Camacho proposed openstack/tripleo-quickstart-extras master: Enable the HA deployment by default  https://review.openstack.org/47644810:06
*** mdnadeem has joined #tripleo10:06
sshnaidmpanda, can't you update containers just with existing dlrn repo? why to create a special container with mirror?10:07
*** jlinkes has joined #tripleo10:09
*** f13o has quit IRC10:13
pandasshnaidm: During the build all the container will have to access the repo to install the packages, for the gate, in some cases we may need to update 30 containers. All of them will have to yum update the packages from the repo. The mirror container will probably be already in the undercloud image, will have to update only once, at start, then everything else will use it. We hope it will be faster to get them10:15
pandalocally10:15
openstackgerritFlavio Percoco proposed openstack/tripleo-heat-templates master: WIP: Deploy kubernetes using TripleO on the overcloud  https://review.openstack.org/47175910:15
*** flepied has joined #tripleo10:16
openstackgerritGabriele Cerami proposed openstack-infra/tripleo-ci master: WIP: containers periodic test  https://review.openstack.org/47574710:17
*** chem has quit IRC10:17
*** sai has quit IRC10:18
*** rook has quit IRC10:18
*** sid1 has quit IRC10:18
openstackgerritCarlos Camacho proposed openstack/tripleo-quickstart-extras master: Enable the HA deployment by default  https://review.openstack.org/47644810:21
*** sid1 has joined #tripleo10:22
*** sai has joined #tripleo10:23
*** chem has joined #tripleo10:24
openstackgerritMerged openstack/tripleo-heat-templates master: Remove trailing whitespace  https://review.openstack.org/47442010:24
openstackgerritCarlos Camacho proposed openstack/tripleo-quickstart-extras master: Enable the HA deployment by default  https://review.openstack.org/47644810:25
openstackgerritCarlos Camacho proposed openstack/tripleo-quickstart-extras master: Enable the HA deployment by default  https://review.openstack.org/47644810:25
oidgarhi, after deploying overcloud with 1xCompute and 1xController (each with only eth0), br-ex does not exist on the compute node. any idea how to create it?10:26
*** rook has joined #tripleo10:27
*** rook is now known as Guest25010:28
*** f13o has joined #tripleo10:28
pandaoidgar: IIRC br-ex should be only on the network node, which in this case is the controller10:29
sshnaidmpanda, adarazs, other cores, can you please review adding multinode to promotion? https://review.openstack.org/#/c/475197/ https://review.openstack.org/47520010:30
*** mdnadeem has quit IRC10:33
*** dtantsur|bbl is now known as dtantsur10:34
oidgarpanda: wierd, I'm getting this message at the compute node from neutron-openvswitch-agent.log10:35
oidgarpanda: 2017-06-22 10:18:37.876 7866 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-05680b79-0e6d-4fdd-be4a-eacf199158a3 - - - - -] Bridge br-ex for physical network datacentre does not exist. Agent terminated!10:35
openstackgerritCarlos Camacho proposed openstack/tripleo-heat-templates master: Enable the HA deployment by default  https://review.openstack.org/35906010:35
sshnaidmadarazs, EmilienM, panda this is for logstash, please re-review https://review.openstack.org/#/c/473095/10:38
dtantsurhi folks! looking for a +W on https://review.openstack.org/#/c/473456/ and https://review.openstack.org/#/c/473457/, both have 2x +210:38
dtantsurthis should preferrably land early to account for potential (though unlikely) regressions10:39
sshnaidmoidgar, with containers?10:39
*** limao has joined #tripleo10:41
pandasshnaidm: why are you grepping on ^20 ? what's that ? a timestamp ?10:41
sshnaidmpanda, yes10:41
sshnaidmwill work until 209910:42
pandawe want only logs from the 2000 and not 3000 or we are indexing only timestamped lines ? why ?10:43
sshnaidmpanda, logstash doesn't eat lines without timestamps10:43
adarazssshnaidm: probably we should have something more than ^20, there could be lines starting with twenty and not being timestamps :)10:43
pandagourmand logstash10:44
sshnaidmadarazs, my perfectionism stops here :)10:44
oidgarsshnaidm: yes10:45
openstackgerritSaravanan KR proposed openstack/tripleo-common master: Derive Params (part 2): Get services and feature list  https://review.openstack.org/45647110:45
sshnaidmoidgar, I think it's known issue. With network isolation?10:46
oidgarsshnaidm: I think so10:46
*** cylopez has quit IRC10:46
sshnaidmoidgar, https://bugs.launchpad.net/tripleo/+bug/168955610:47
openstackLaunchpad bug 1689556 in tripleo "CI: containers overcloud can't start vm when deployed with multinic network isolation" [High,Triaged]10:47
*** flepied has quit IRC10:48
*** milan has joined #tripleo10:49
hjensasbnemec: around? In instack-undercloud, tox fails with "upper-constraints.txt Could not satisfy constraints for 'instack-undercloud': installation from path or url cannot be constrained to a version". I guess this started after instack-undercloud was added to global-requirements.txt? I can't figure out how to fix it.10:50
adarazssshnaidm: added another comment to https://review.openstack.org/47309510:50
thervemcornea, I haven't found the heat issue, but the update failed a little bit after retrying10:50
thervemcornea, " Cannot connect to the Docker daemon. Is the docker daemon running on this host?"10:50
mcorneatherve: hm, that's ok but first time it failed after 4h10:51
sshnaidmadarazs, this file is for logstash server, not for people10:52
sshnaidmadarazs, nobody is supposed to look there and understand :)10:53
*** jlinkes has quit IRC10:53
thervemcornea, Yeah I know. I'm not finding the cause yet10:53
adarazssshnaidm: hmm, I think logstash can be more useful if we can see which files have errors... though I admittedly don't know much about logstash... I understand you can analyze trends and errors with it, so I thought the filename could be useful.10:54
mcorneatherve: would it help if I try to bring it the first update so we can see the timeout in action?10:54
sshnaidmadarazs, and when you find a string in kibana, you can't know which file is this from anyway10:54
sshnaidmadarazs, you won't see filename10:54
adarazssshnaidm: okay /o\ :)10:55
openstackgerritCarlos Camacho proposed openstack/tripleo-quickstart-extras master: Enable the HA deployment by default  https://review.openstack.org/47644810:56
*** katkapilatova has joined #tripleo10:57
sshnaidmadarazs, just try it: http://logstash.openstack.org/#/dashboard/file/logstash.json?query=build_name:%20gate-tripleo-ci-centos-7-ovb-ha-oooq%20AND%20build_status:%20FAILURE%20AND%20message:%20%5C%22Already%20timed%20out%5C%2210:57
sshnaidmadarazs, set period in upper right cornet to "last 30 days"10:57
sshnaidmadarazs, you will see the message itself and file that message appear in it - in our case it will be "logstash.txt" or "console.html"10:58
adarazsyeah, I had to click in the "filename" field. yeah. so... we can't upload it separately, or have a field for the real filename? I guess it's not really important indeed if you only use it to generate graphs to see if some error message reoccurs.11:00
*** cylopez has joined #tripleo11:01
sshnaidmadarazs, maybe it's possible to insert filename to each line of this file, then you will see the whole line and filename will appear there..11:01
adarazsyeah, okay, I guess it's not necessary, or we can do it later...11:01
sshnaidmadarazs, but it seems to be a little different solution, yeah11:01
sshnaidmadarazs, fill the card for it, maybe we'll get back to it once11:01
sshnaidmadarazs, wrt regex, I think we can live with it, considering that all mentioned scripts now are timestamped..11:04
*** jkilpatr has joined #tripleo11:04
sshnaidmit's for safety only, not the main feature11:05
adarazsall right.11:05
openstackgerritSaravanan KR proposed openstack/tripleo-common master: WIP: DPDK Derive parameters workflow  https://review.openstack.org/46898911:07
sshnaidmpanda, sorry, I didn't understand you comment: https://review.openstack.org/#/c/475197/2/scripts/mirror-server/multinode_status.py11:07
sshnaidmpanda, what does mean " check if the metadata is already updated"?11:08
sshnaidmpanda, if I parsed logs once, I don't parse them again ever11:09
*** lucasagomes is now known as lucas-hungry11:10
*** limao has quit IRC11:11
*** flepied has joined #tripleo11:11
akrivokaI'm getting the "Max retries exceeded"  error very often when trying mistral CLI commands in my tripleo dev environment11:16
thervemcornea, Yeah, but it be good to add some debug first11:16
akrivokarestarting mistral-engine or mistral-executor doesn't help11:16
akrivokais there a better workaround for this than rebooting the undercloud vm?11:16
*** adarazs has quit IRC11:17
mcorneatherve: ok, let me get it to the state before upgrade and I'll ping you11:18
thervemcornea, Thanks!11:18
*** dbecker has quit IRC11:18
*** limao has joined #tripleo11:19
openstackgerritJaganathan Palanisamy proposed openstack/tripleo-common master: WIP: DPDK Derive parameters workflow  https://review.openstack.org/46898911:21
*** chem has quit IRC11:29
*** ffiore has quit IRC11:29
*** chem has joined #tripleo11:31
fultonjanyone want to review https://review.openstack.org/#/c/475952 ? thanks11:34
fultonjbogdando: ^ ?11:35
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-heat-templates stable/newton: Keep floating ip reachability during pacemaker migration.  https://review.openstack.org/47496711:35
gfidentefultonj morning there11:35
fultonjhi gfidente11:36
gfidentefultonj /me looking into how to corretly set the subnets for the ceph cluster11:37
gfidentethen moving on with clients11:37
fultonjgfidente: sounds goode11:37
gfidenteI think those are the two things still missing to test this in CIO11:37
gfidente*CI :D11:37
fultonjgfidente: fyi: colonwq's OSD list was empty yesterday (but the mon list was good)11:38
*** salmankhan has quit IRC11:38
fultonjgfidente: yes, clients (containerized and non-containerized) + other params11:38
*** abishop has joined #tripleo11:40
colonwqfultonj, I need to add 467682 to my uc and revert the manual edit. I'll see if that helps the list issue.11:41
fultonjcolonwq: ack11:41
gfidentecolonwq if you git checkout https://review.openstack.org/465066 it'll pull in the two submissions it needs as well11:41
gfidente(instead of chetty-ick)11:42
colonwqgfidente, ack. Thanks.11:42
openstackgerritBogdan Dobrelya proposed openstack/puppet-tripleo master: Split docker options and insecure registry  https://review.openstack.org/47647611:43
dtantsurhjensas: oh that's fun to fix11:43
dtantsurlemme find you a link11:43
bogdandofolks, do you remember the strange issue with insecure registry config change applied, but docker service restart unscheduled by puppet due to augeas error (nothing to change)? so that's to address it ^^11:44
bogdandoso there are only ovb-ha, and nonha-multinode-scenario1-4 jobs. That seems not enough to cover all cases, at least for containers sshnaidm11:44
bogdandoum bad paste, sorry11:44
*** salmankhan has joined #tripleo11:44
bogdandohttps://review.openstack.org/476476 this11:44
dtantsurhjensas: you'll have to cargo-cult something like this https://review.openstack.org/#/c/463679/11:44
hjensasdtantsur: thanks, I saw something about that on mailing list but did'nt think it was the way to do it from the discussion. :) I'll give it a go.11:46
*** amoralej is now known as amoralej|lunch11:47
*** pradk has joined #tripleo11:49
*** rwsu has joined #tripleo11:50
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: Set jobs subunit file as fail until it's overwritten  https://review.openstack.org/47642711:51
openstackgerritHonza Pokorny proposed openstack/tripleo-ui master: Use component objects when comparing for type  https://review.openstack.org/47508911:51
*** udesale has quit IRC11:52
*** masco has quit IRC11:54
*** dsariel has quit IRC11:58
openstackgerritMerged openstack/diskimage-builder master: On suse the python2 dev package is python-devel  https://review.openstack.org/47505112:01
*** ramishra has quit IRC12:03
*** adarazs has joined #tripleo12:03
openstackgerritDamien Ciabrini proposed openstack/puppet-pacemaker master: HA: update bundle creation to be compatible with newest pcs  https://review.openstack.org/47648612:04
*** thrash|g0ne is now known as thrash12:04
*** ramishra has joined #tripleo12:05
*** zoli|wfh is now known as zoli|lunch12:05
openstackgerritCarlos Camacho proposed openstack/tripleo-heat-templates master: Enable the HA deployment by default  https://review.openstack.org/35906012:07
*** rbrady has quit IRC12:08
*** pkovar has joined #tripleo12:09
*** dprince has joined #tripleo12:12
openstackgerritGabriele Cerami proposed openstack/tripleo-quickstart master: WIP: enable container periodic job  https://review.openstack.org/47574312:13
openstackgerritGabriele Cerami proposed openstack-infra/tripleo-ci master: WIP: containers periodic test  https://review.openstack.org/47574712:13
*** trown|outtypewww is now known as trown12:13
*** lucas-hungry is now known as lucasagomes12:16
EmilienMbogdando: what topic exactly? can I have tl;dr?12:17
bogdandoEmilienM: Thoughts on improving visibility of failed gate jobs with rss subscription available from openstack-health web page. Like this example http://health.openstack.org/runs/key/build_name/periodic-tripleo-ci-centos-7-scenario002-multinode-oooq/recent/rss12:19
bogdandoEmilienM: and making those gate jobs more scoped, like containers vs non containers12:20
*** sbrzozow has joined #tripleo12:20
EmilienMbogdando: jobnames already do that, you can know which jobs are container or not12:20
bogdandoEmilienM: I tried to find multinode and ovb-ha gate jobs (or periodic) for containers to subscribe as rss, and failed miserably12:20
EmilienMyeah I know12:21
EmilienMI found that yesterday too12:21
EmilienMand I haven't found why yet12:21
*** limao has quit IRC12:22
*** chem has quit IRC12:22
bogdandoEmilienM, sshnaidm: mentioned that gate jobs can't be accessed for health, or there are subtle caveats12:22
bogdandofrom the o-health*12:22
EmilienMbogdando: I confirm gate jobs are in gate12:22
EmilienMlet me show you12:22
EmilienMbogdando: http://status.openstack.org/openstack-health/#/job/gate-nova-python3512:23
EmilienMgate-nova-python35 is a gate job in Nova, not periodic12:23
bogdandoEmilienM: some of them, right. Not all, not for containers AFAICT12:23
EmilienMyeah something is weird12:24
bogdandoEmilienM: to shrink it very tl;dr all I wanted is some help with "find multinode and ovb-ha gate jobs (or periodic) for containers to subscribe as rss"12:25
bogdandoto bring a shortcut for that flow "when I want to see why my job fails and if it's known issue, I firstly look at status page and if there are long list of failing jobs with the same error - most likely it's the same bug, then it's better to look in lp if it's opened already, in channel - if there are alarms, and if not - you won the opportunity to fill the bug first."12:25
bogdandosshnaidm: ^^12:26
*** chlong has quit IRC12:26
bogdandoand once I get a solution, perhaps making it publicly available for other contributors12:26
bogdandoas rss feeds for a job of a day12:27
bogdandogate/periodic only. to catch regressions via scoped subject area channels12:27
sshnaidmbogdando, if will be possible to subscribe to rss of container gate jobs - go for it, I don't see any problems with it12:27
bogdandosshnaidm: I couldn't find how12:27
*** agopi has joined #tripleo12:28
sshnaidmbogdando, how - to subscribe to rss?12:28
skramajacould someone review https://review.openstack.org/#/c/468390/, which  provides an option to overide plan-environment file?12:28
bogdandosshnaidm: to get the link for that job =)12:28
bogdandofrom openstack health page or other places12:29
openstackgerritHarald Jensås proposed openstack/instack-undercloud master: Properly support upper-constraints  https://review.openstack.org/47649512:29
*** jpena is now known as jpena|lunch12:29
sshnaidmbogdando, I think I've already replied to that :)12:30
jtomasekhonza: I wonder why isn't this being verified by CI, do I miss something? https://review.openstack.org/#/c/462230/1112:31
jtomasekhonza: I don't see it queued in zuul12:31
honzajtomasek: I was wondering the same thing12:31
EmilienMsshnaidm, bogdando: yeah I don't find container job either12:32
EmilienMhttp://status.openstack.org/openstack-health/#/?groupKey=build_name&resolutionKey=day&searchProject=container12:32
*** rlandy has joined #tripleo12:32
sshnaidmEmilienM, as ovb-ha-ooq too12:33
*** pgadiya has quit IRC12:34
*** myoung has joined #tripleo12:34
*** flepied has quit IRC12:36
*** cylopez has quit IRC12:37
jtomasekhonza: do you mind if I update the PlanCard slightly (in subsequent patch) to look something like this? http://storage6.static.itmages.com/i/17/0622/h_1498135141_7063804_1596d189f5.png12:40
honzajtomasek: nope :)12:40
jtomasekhonza: ok12:41
honzajtomasek: lol if you think i have design skills :)12:41
jtomasek:)12:41
mcorneatherve: ok, I'm ready to trigger the upgrade attempt. I set debug to true in heat.conf. Anything else I should prepare before?12:41
thervemcornea, Let me think a little bit. We need a good amount of debug logs12:42
*** dsariel has joined #tripleo12:44
*** limao has joined #tripleo12:45
*** rbowen has joined #tripleo12:45
*** jayg|g0n3 is now known as jayg12:45
*** ffiore has joined #tripleo12:45
*** morazi has joined #tripleo12:46
*** ramishra has quit IRC12:46
*** ramishra has joined #tripleo12:48
EmilienMmandre, jistr, dprince : I see container jobs (both multinode + ovb) are passing today - should we make them voting now?12:51
thervemcornea, OK! I think we should be good to go12:53
mcorneatherve: alright, I'm starting it12:53
EmilienMsshnaidm: hey, how do you get graphs on https://status-tripleoci.rhcloud.com/ ? i don't see them now12:54
yolandahi, i tried to cancel a heat update, using heat-cancel-update command. but now it seems to be forever in rollback_in_process status. How can i recover?12:54
*** gkadam has quit IRC12:55
sshnaidmEmilienM, try to reload, I see them12:55
*** tzumainn has joined #tripleo12:56
*** tzumainn has quit IRC12:57
*** jmelvin has joined #tripleo12:57
openstackgerritwes hayutin proposed openstack/tripleo-quickstart-extras master: configure the undercloud to use nameservers from virthost  https://review.openstack.org/47587812:57
*** tzumainn has joined #tripleo12:57
EmilienMsshnaidm: I had to click on "Load unsafe scripts" in Chrome !12:58
*** limao has quit IRC12:58
sshnaidmEmilienM, wow.. google consider its own framework as unsafe12:59
EmilienMsshnaidm: :D12:59
*** limao has joined #tripleo13:00
EmilienMjistr: also do you know the status of gate-tripleo-ci-centos-7-scenario00X-multinode-oooq-container jobs recently?13:00
*** pradk has quit IRC13:02
*** sid1 has quit IRC13:03
*** rbrady has joined #tripleo13:03
*** rbrady has quit IRC13:03
*** rbrady has joined #tripleo13:03
thervemcornea, OK got the error13:03
*** f13o has quit IRC13:04
mcorneatherve: cool, what was it?13:04
thervemcornea, Invalid condition "server_not_blacklisted"13:04
*** Guest250 is now known as rook13:06
therveI think I got it13:08
*** abhishek has joined #tripleo13:08
*** sid1 has joined #tripleo13:09
*** psahoo has quit IRC13:10
EmilienMmandre, jistr, dprince: https://review.openstack.org/476510 please review13:10
*** athomas has joined #tripleo13:13
EmilienMsshnaidm, bogdando: I tried to ping mtreinish who did openstack-health about the issue we have (yesterday) but got no response so far13:13
EmilienMI'll try again today13:13
EmilienMI'm not sure why this data is not collected13:14
sshnaidmyeah, data is collected very selectively13:14
*** bfournie has joined #tripleo13:15
*** ffiore_ has joined #tripleo13:16
*** ffiore has quit IRC13:16
openstackgerritMerged openstack/instack-undercloud master: Update constraints support  https://review.openstack.org/47621113:17
*** f13o has joined #tripleo13:18
*** pradk has joined #tripleo13:22
EmilienMsshnaidm, bogdando: I think I found it13:23
EmilienMsshnaidm, bogdando: I think OpenStack Health doesn't collect logs from -nv & also third party ci jobs13:24
EmilienMonly gate + periodic13:24
sshnaidmEmilienM, you know.. I think it collects only gate, but not *check* jobs13:24
*** sasha2 has joined #tripleo13:24
EmilienMyes it makes sense now13:25
sshnaidmEmilienM, we don't have ovb jobs in gates13:25
EmilienMI remember what Matt told me a few months (years?) ago13:25
EmilienMright13:25
*** sasha2 has left #tripleo13:25
EmilienMso for containers, we just need to move it in gate ,like I'm doing in https://review.openstack.org/47253613:25
EmilienMbut for OVB, we might need to patch openstack-health13:25
EmilienMand maybe collect check-tripleo13:26
sshnaidmEmilienM, it should be in "gate" pipeline13:26
*** zoli|lunch is now known as zoli|wfh13:26
EmilienMsshnaidm: what should be in gate pipeline?13:27
openstackgerritHarald Jensås proposed openstack/instack-undercloud master: Tripleo routed networks ironic inspector, and Undercloud  https://review.openstack.org/43754413:27
EmilienMI don't think it's possible13:27
sshnaidmEmilienM, any job for o.health13:27
EmilienM(to hav ovb in gate)13:27
sshnaidmEmilienM, exactly13:27
sshnaidmI was asking a month ago about jobs monitoring on infra, there is some graphite server which collects statistics, but I didn't find how to get status and link to logs from there13:29
sshnaidmthere is also gerrit even streamer, not sure if it's helpful too13:29
EmilienMsshnaidm: not having ovb in health is not critical imho13:29
EmilienMsshnaidm: at least for promotion pipeline13:29
sshnaidmEmilienM, just weird that we don't have such interface..13:29
EmilienMfor rss it would have been useful13:30
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: CI test - never merge  https://review.openstack.org/46100013:30
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: CI test - never merge  https://review.openstack.org/46100013:30
thervemcornea, So the source is probably https://review.openstack.org/#/c/475003/13:31
therveBecause of a heat bug I suspect, but that doesn't work13:31
therveEmilienM, ^^^ source of upgrade breakage13:32
*** Goneri has joined #tripleo13:32
*** jpena|lunch is now known as jpena13:32
mwhahahatherve: shall we revert in the mean time then?13:33
thervemwhahaha, That's probably the quickest path yeah.13:33
openstackgerritFlorian Fuchs proposed openstack/tripleo-validations master: Add a validation that checks for available update  https://review.openstack.org/47651713:33
mwhahahatherve: is there an existing bug we can reference?13:33
therveIf there is a heat fix it still needs to be promoted and all13:33
thervemwhahaha, No https://bugs.launchpad.net/tripleo/+bug/1699463 is our source for now13:34
openstackLaunchpad bug 1699463 in tripleo "6 upgrade jobs (cont. and non-cont.) stuck on UpgradeInitDeployment, SoftwareDeployment resources signals get response code 400" [Critical,Triaged]13:34
*** karthiks has quit IRC13:34
openstackgerritAlex Schultz proposed openstack/tripleo-heat-templates master: Revert "Blacklist support for ExtraConfig"  https://review.openstack.org/47651913:35
mwhahahatherve: -^13:35
hjensastherve: would that condition bug hit this as well? https://review.openstack.org/#/c/473817/15/network/ports/ctlplane_vip.yaml13:35
therveOK thanks13:35
*** jlinkes has joined #tripleo13:35
thervehjensas, Possibly, do you hit that in updates?13:36
*** amoralej|lunch is now known as amoralej13:36
dciabrinmandre, shardy ping, if you have some time for looking at https://review.openstack.org/#/c/476486, that would be awesome :)13:36
hjensastherve: gate-tripleo-ci-centos-7-ovb-updates job passed for that change.13:36
thervehjensas, I think it's upgrade? not sure13:36
*** dsariel has quit IRC13:36
*** chem has joined #tripleo13:36
*** liverpooler has joined #tripleo13:37
*** chlong has joined #tripleo13:38
hjensastherve: ok, gate-tripleo-ci-centos-7-multinode-upgrades-nv failed. But not sure if it is the same issue. what do I look for to verify?13:39
thervehjensas, Hold. All upgrades are broken :)13:39
therveSo right now, you wait for master to be fixed and then retry13:39
hjensastherve: :) ok.13:39
EmilienMtherve: in a call, brb13:40
*** limao has quit IRC13:45
weshayjrist, ping.. re: the undercloud install error13:47
weshayjrist, did you get through it?13:48
*** rbrady is now known as rbrady-brb13:48
*** mrchao has quit IRC13:48
*** gbarros has joined #tripleo13:50
*** karthiks has joined #tripleo13:50
jpichhonza: Would it be possible to use a separate queue dedicated to logging, or doesn't really fit?13:50
honzajpich: yes, i think that's a good idea13:50
honzajpich: i have been trying to avoid it because it'll require additional configuration but yeah it seems to make sense given today's developments13:51
adarazsEmilienM, trown, sshnaidm, weshay, hrybacki, panda, rlandy, arxcruz jistr: CI Squad meeting in ~37 minutes, I see a bunch of agenda items already, but add more if you have something: https://etherpad.openstack.org/p/tripleo-ci-squad-meeting -- see you there :)13:53
jpichhonza: Oh, what kind of configuration? I know the CLI uses a bunch of different queues for messages so I thought it'd be lightweight13:53
jpichhonza: also thanks for the tips on how to test code as "prod" :)13:53
*** karimb has quit IRC13:53
honzajpich: tripleo_ui_config.js + puppet-tripleo, and everything else can probably be done with defaults (mistral workflows)13:54
jpichhonza: aaah I see, of course13:54
honzajpich: I should probably add that to our "developer" docs13:54
jpichhonza: Great idea!13:55
*** jokke__ is now known as jokke_13:55
jristweshay: nope13:56
jristweshay: fired it up again with just --master13:56
jristno dice13:56
jristweshay: that said, I can get to the undercloud machine13:56
jristbut nothing is running and it's not sueful13:56
*** limao has joined #tripleo13:56
jristuseful13:56
weshayjrist, can you chunk.io the undercloud_install.log13:57
weshayjrist, I suspect you are hitting dns issues13:57
jristweshay: from the vm?13:57
*** jcoufal has joined #tripleo13:58
jristhmm I think you're right13:58
jrist2017-06-21 18:53:00 | 2017-06-21 18:53:00,936 INFO: 14: curl#7 - "Failed to connect to 2604:1580:fe02:2::10: Network is unreachable"13:58
*** lblanchard has joined #tripleo13:58
*** fzdarsky has quit IRC13:59
weshayjrist, what is your quickstart.sh command?13:59
weshaymay need to add a dns server you know to work w/ -e undercloud_undercloud_nameservers=<dns_server_ip>14:00
jrist[toor@galena ~]$ ./tripleo-quickstart/quickstart.sh --release master --no-clone --clean 127.0.0.2 --ansible-debug or [toor@galena ~]$ ./tripleo-quickstart/quickstart.sh --release master-ci --no-clone --clean 127.0.0.2 --ansible-debug14:00
weshayjrist, google is ratelimiting 8.8.8.8 and 8.8.4.4.14:00
*** fzdarsky has joined #tripleo14:00
jristsadface14:00
*** jbadiapa_ has quit IRC14:01
*** rhallisey has quit IRC14:04
*** rhallisey has joined #tripleo14:05
*** rbrady-brb is now known as rbrady14:07
*** dciabrin has quit IRC14:08
*** sid1 has quit IRC14:08
*** atheurer has joined #tripleo14:08
jristweshay: will try that14:10
*** athomas has quit IRC14:10
*** aufi has quit IRC14:11
jristweshay: can that be comma separated?14:11
jrist -e undercloud_undercloud_nameservers=209.244.0.3,209.244.0.414:12
*** myoung has quit IRC14:12
*** sid1 has joined #tripleo14:13
openstackgerritMerged openstack/tripleo-quickstart-extras master: Create a file for indexing it in logstash  https://review.openstack.org/47309514:15
openstackgerritMerged openstack/puppet-tripleo master: Deploy ironic-api with WSGI  https://review.openstack.org/47345614:15
openstackgerritMerged openstack/tripleo-quickstart-extras master: Add option to copy ready cirros images from directory  https://review.openstack.org/47547914:15
*** jbadiapa_ has joined #tripleo14:16
*** karthiks has quit IRC14:16
sshnaidmEmilienM, fyi: https://review.openstack.org/#/c/476524/14:17
bnemechjensas: Not sure if you saw this, but https://review.openstack.org/#/c/476211/ should fix the instack-undercloud test failures.14:17
*** jbadiapa_ has quit IRC14:18
*** skramaja has quit IRC14:18
*** jbadiapa_ has joined #tripleo14:18
hjensasbnemec: I saw it. Works for me. :)14:20
EmilienMsshnaidm: looking14:20
openstackgerritOliver Walsh proposed openstack/tripleo-heat-templates stable/newton: WIP: ensure nova migration package is installed during minor update  https://review.openstack.org/47653414:22
openstackgerritMerged openstack/tripleo-heat-templates master: Deploy ironic-api with WSGI  https://review.openstack.org/47345714:22
*** dsariel has joined #tripleo14:22
weshayjrist, yes it should and we have a fix on the way https://review.openstack.org/#/q/topic:virthost-nameserver14:22
b00tcatHi all, I realized that I can't boot UEFI-enabled hosts with the default files in /tftpboot - is there special EFI file that I have to download in order to boot the baremetal servers?14:22
*** dciabrin has joined #tripleo14:23
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart master: WIP: change devmode to deploy containerized services by default  https://review.openstack.org/47241214:23
*** mburned has quit IRC14:25
*** mburned has joined #tripleo14:25
*** aufi has joined #tripleo14:29
jprovazndsneddon, hi, ping when you are around14:30
jristweshay: nice, thanks14:32
*** ykarel_ has quit IRC14:32
adarazsEmilienM, hrybacki, jistr: CI squad meeting starting14:32
*** sid2 has joined #tripleo14:33
EmilienMadarazs: on my way14:34
*** ffiore_ has quit IRC14:34
*** sid1 has quit IRC14:35
jristweshay: trying with that DNS command. thanks!14:37
trozetmwhahaha: hey, do you have a minute to talk about that odl patch to lock the step?14:39
mwhahahatrozet: what's up?14:39
trozetmwhahaha: did you have a chance to see my response in https://review.openstack.org/#/c/475819/14:39
mwhahahatrozet: yea, i did. so the problem i would have is that what happens with updates14:40
mwhahahatrozet: as that would still suffer from the same problem14:40
trozetmwhahaha: ODL doesnt support update right now :/14:40
openstackgerritHonza Pokorny proposed openstack/tripleo-ui master: Document how to test a production build  https://review.openstack.org/47653714:40
mwhahahatrozet: right that's a problem14:40
trozetmwhahaha: but im also looking here for your perspective14:40
trozetmwhahaha: like another option is, I could always return the password, so we assume it is always set correctly and modifying the password isnt supported (but that seems like an even worse behavior)14:41
*** janki has quit IRC14:41
trozetmwhahaha: but it would make the whole operation idempotent14:41
mwhahahatrozet: let me poke at the odl code, is there a way to do a connectivity check with the user/password we could do?14:42
*** dprince has quit IRC14:43
trozetmwhahaha: there is.  Take a look at the discussion in the ODL bug.  One option is to curl the ODL REST interface, but that assumes ODL is up, and we have to know the port info.  I suggested in the bug that we just interact with the DB tool directly14:44
mwhahahatrozet: at a minimum of being able to do a user check and we just document/prevent password changes would be best14:44
trozetmwhahaha: yeah we do the user check today14:44
mwhahahatrozet: so if we had that today, it shouldn't cause problems14:44
trozetmwhahaha: ok so until we have a way to check the password, you think the better option is to just prevent password change14:44
*** dhill_ has joined #tripleo14:44
mwhahahatrozet: yea14:44
trozetmwhahaha: ok, I can push that fix to puppet-opendaylight and abandon this one14:45
trozetmwhahaha: thanks for the input!14:45
*** jpena is now known as jpena|brb14:45
*** links has quit IRC14:45
mwhahahatrozet: fyi here's how we do the passwor dcheck bit for keystone, https://github.com/openstack/puppet-keystone/blob/master/lib/puppet/provider/keystone_user/openstack.rb#L10514:46
*** ffiore_ has joined #tripleo14:47
trozetmwhahaha: thanks I will look att his14:47
*** katkapilatova has left #tripleo14:47
openstackgerritMarius Cornea proposed openstack/tripleo-specs master: WIP: A unified tool for upgrading TripleO based deploments  https://review.openstack.org/47653914:48
*** fragatina has joined #tripleo14:49
openstackgerritMarius Cornea proposed openstack/tripleo-specs master: WIP: A unified tool for upgrading TripleO based deploments  https://review.openstack.org/47653914:49
openstackgerritMerged openstack/tripleo-quickstart-extras master: Change default DNS name server  https://review.openstack.org/47433414:50
*** fragatina has quit IRC14:50
dpawarmwhahaha: Hi, we have follwoing reviews https://review.openstack.org/#/c/475765/ , https://review.openstack.org/#/c/475762/7, https://review.openstack.org/#/c/47575814:52
mwhahahadpawar: if they are ready for review, please lift the -1 wip14:53
fultonjd0ugal: do you have a few cycles to review https://review.openstack.org/#/c/475952/ ?14:53
dpawarmwhahaha: i was trying to call these using method mentioned at https://docs.openstack.org/developer/tripleo-docs/advanced_deployment/cinder_netapp.html14:54
dpawarit did not work14:54
mwhahahadpawar: ok i'll take a look in a bit (in meeting)14:54
openstackgerritMarius Cornea proposed openstack/tripleo-specs master: WIP: A unified tool for upgrading TripleO based deploments  https://review.openstack.org/47653914:54
openstackgerritOliver Walsh proposed openstack/tripleo-heat-templates stable/newton: Install openstack-nova-migration on computes during M->N upgrade  https://review.openstack.org/47654214:55
*** marios has quit IRC14:56
dpawarmwhahaha: thanks14:56
openstackgerritOr Idgar proposed openstack/tripleo-heat-templates master: adding --config-dir parameters to neutron containers  https://review.openstack.org/47654415:02
jristweshay: http://paste.openstack.org/show/613406/ - that -e didn't work15:02
*** MrRoot has joined #tripleo15:02
*** jbadiapa_ has quit IRC15:02
openstackgerritCarlos Camacho proposed openstack-infra/tripleo-ci master: Enable the HA deployment by default  https://review.openstack.org/41517915:04
*** dciabrin has quit IRC15:04
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: Bind GlanceApiNetwork on internal_api network  https://review.openstack.org/47654815:05
EmilienMtrozet: ^ fyi15:05
EmilienMtrozet: oops I missed your patch15:06
trozetEmilienM: yeah15:06
EmilienMtrozet: it wasn't in launchpad15:06
trozetEmilienM: hmm weird15:06
mwhahahadpawar: do you have a specific error that you're encountering?  Since you listed a bunch of other deps, you may want to look at http://hardysteven.blogspot.com/2016/08/tripleo-deploy-artifacts-and-puppet.html as ways to get your other puppet module changes in place for testing15:07
trozetEmilienM: weird thing is our tempest tests in OPNFV caught this, but only when we switched to using V3 creds.  I'm not sure why.  I'm not sure which endpoint keystone public auth uri will map to for glance api15:07
*** ebarrera has quit IRC15:07
trozetEmilienM: or how it is different from V2 creds...i need to figure that out15:08
saibnemec: hey, when im scaling up compute nodes, do u know if the nova-compute process on pre-existing computes is bounced?15:08
dtantsurcan someone knowing about containers please jump on the discussion at https://review.openstack.org/#/c/457822/8/docker/services/ironic-inspector.yaml@74 ?15:08
dtantsurI don't quite understand what I should do there..15:08
openstackgerritMartin Mágr proposed openstack/tripleo-heat-templates master: Make collectd log to file  https://review.openstack.org/47580815:09
*** rcernin has quit IRC15:09
EmilienMtrozet: not sure15:09
*** yamahata has joined #tripleo15:10
*** karimb has joined #tripleo15:10
*** pblaho has quit IRC15:10
*** yprokule has quit IRC15:10
*** yprokule_ has joined #tripleo15:11
*** yprokule_ has quit IRC15:11
*** yprokule_ has joined #tripleo15:11
*** yprokule_ is now known as yprokule15:12
EmilienMmcornea: nice catch on https://bugs.launchpad.net/tripleo/+bug/169944315:12
openstackLaunchpad bug 1699443 in tripleo "Ocata -> Pike overcloud upgrade: heat-api services are running both under systemd and docker containers" [High,Triaged]15:12
EmilienMmcornea: we need some ansible playbook to stop the services during the upgrade process15:12
EmilienMmcornea: in THT15:12
*** aufi has quit IRC15:12
*** pchavva has joined #tripleo15:13
*** rbowen has quit IRC15:13
pradkcan i please get some feedback on https://review.openstack.org/#/c/467713/15:16
EmilienMbnemec: if you can add https://review.openstack.org/#/c/469474 do your review list, please15:16
*** dciabrin has joined #tripleo15:17
*** jbadiapa_ has joined #tripleo15:17
openstackgerritCarlos Camacho proposed openstack-infra/tripleo-ci master: Enable the HA deployment by default  https://review.openstack.org/41517915:19
*** f13o has quit IRC15:19
openstackgerritGabriele Cerami proposed openstack/tripleo-quickstart-extras master: WIP: enable containers periodic job  https://review.openstack.org/47574515:20
d0ugalfultonj: sure, looking now15:20
fultonjd0ugal: thanks15:20
jristdtrainor: any thoughts on this dns thing?15:21
jrist[toor@galena ~]$ ./tripleo-quickstart/quickstart.sh --release master-tripleo-ci --no-clone --clean 127.0.0.2 -e undercloud_undercloud_nameservers=209.244.0.315:21
jristthat fails ^15:21
jristeventually it complains cp: cannot stat '/usr/lib64/python2.7/site-packages/_selinux.so': No such file or directory15:21
*** adarazs_ has joined #tripleo15:21
jristUsage: ./tripleo-quickstart/quickstart.sh --install-deps15:21
jristand then just lists out the help15:21
dtrainori have not encountered that one, let me check something15:21
weshayjrist, try w/ just one dns server15:22
weshayjrist, question.. tripleo-ui is moving to a container ya?15:22
jristI did15:22
weshayhrm15:22
jristweshay: no15:22
weshayjrist, can you paste the error you are getting15:22
jristI did15:22
jrist<jrist> weshay: http://paste.openstack.org/show/613406/ - that -e didn't work15:22
dtrainorcan you try specifying the virthost as the last argument?15:23
weshayjrist, postition matters.. -e must come before the virthost 127.0.0.215:23
jristah15:23
jristtake 3 action15:23
*** dciabrin has quit IRC15:23
*** oidgar has quit IRC15:23
weshaythis time w/ more LOVE15:23
jristsheesh15:24
jrist100% love here15:24
weshaygive me EMOTION MAN15:24
jristcan't give it any more!15:24
weshaybah.. SCENE! perfect15:24
jristlol15:24
jristok imma let that run while I go get my teeth drilled on15:24
jristthanks weshay15:24
dtrainorjrist, can you paste the output of:  rpm -qi --queryformat '%{NAME}-%{VERSION}' libselinux-python15:25
dtrainorsorry, guess i'm oldschool... rpm -qa | grep libselinux-python15:25
jristhaha15:25
jristold school15:25
openstackgerritMerged openstack/tripleo-heat-templates master: Test Horizon by default in multinode jobs  https://review.openstack.org/47474215:25
jristlibselinux-python-2.5-6.el7.x86_6415:26
openstackgerritMerged openstack/tripleo-heat-templates master: Remove add_listen: false from Horizon hieradata  https://review.openstack.org/47275615:26
dtrainorthere's an update out there15:26
openstackgerritRyan Brady proposed openstack/tripleo-common master: TEST DO NOT MERGE  https://review.openstack.org/47660015:26
jristorly15:26
dtrainori'm at libselinux-python-2.6-6.fc26.x86_6415:26
dtrainorprob not related but, maybe probably idk.  not familiar enough with oooq yet15:26
jristk15:27
jristit's making it further right now15:27
jristso15:27
jristI'll let you guys know when I get back!15:27
*** jbadiapa_ has quit IRC15:27
*** sid1 has joined #tripleo15:29
openstackgerritCarlos Camacho proposed openstack-infra/tripleo-ci master: Enable the HA deployment by default  https://review.openstack.org/41517915:29
dpawarmwhahaha: i did not get any error. but do not see veritas hyperscale attributes set in cinder.conf15:29
*** jlinkes has quit IRC15:30
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates master: Add network/cidr mapping into ServiceNetMap  https://review.openstack.org/47660215:30
*** akane has joined #tripleo15:31
trozetdprince: can  you please review https://review.openstack.org/#/c/466920/ ?15:31
*** sid2 has quit IRC15:31
openstackgerritDerek Higgins proposed openstack/tripleo-heat-templates master: Add Ironic services to containerized deployment  https://review.openstack.org/47660315:32
mwhahahadpawar: can you provide the logs for the deployment?15:32
EmilienMbogdando: can you help me to figure out why sahara is not containerized in http://logs.openstack.org/00/461000/24/experimental/gate-tripleo-ci-centos-7-scenario003-multinode-oooq-container/db5a3e6/logs/subnode-2/var/log/containers/ ?15:32
*** ccamacho has quit IRC15:33
EmilienMbogdando: it sounds like it's deployed on the host outside a container: http://logs.openstack.org/00/461000/24/experimental/gate-tripleo-ci-centos-7-scenario003-multinode-oooq-container/db5a3e6/logs/subnode-2/var/log/sahara/15:33
EmilienMbogdando: see http://logs.openstack.org/00/461000/24/experimental/gate-tripleo-ci-centos-7-scenario003-multinode-oooq-container/db5a3e6/logs/subnode-2/var/log/host_info.txt.gz15:33
EmilienMbogdando: i'm wondering if the environment used is correct15:34
thervemwhahaha, So we have identified the issue, need some time to have a test and fix it15:34
EmilienMjistr: any idea?15:34
bnemecEmilienM: Reviewed.  That won't actually work, but at least we're getting closer to being able to do the thing that will.15:34
mwhahahatherve: ok15:34
thervemwhahaha, It affects conditions usage in properties during upgrades, FWIW15:35
bogdandoEmilienM: indeed, its configured non-docker https://review.openstack.org/#/c/461000/24/ci/environments/scenario003-multinode.yaml@515:35
bogdandoIIUC15:36
dtantsur(that's me again) mind also checking https://review.openstack.org/#/c/473896/ ? this (and an ocata backport to follow) make sure users don't need to log into controllers and run commands there to run ironic15:36
dtantsurespecially someone knowing about nova ^^^15:36
bogdandoEmilienM: so those from lines 4 to 13 will run on host mode15:37
*** cylopez has joined #tripleo15:37
EmilienMbogdando: ah!15:37
EmilienMbogdando: so I need to change that I guess15:37
bogdandoaye15:37
*** jprovazn has quit IRC15:38
EmilienMbogdando: is there someone working on the multinode scenarios containers?15:38
openstackgerritJohn Trowbridge proposed openstack/tripleo-quickstart master: Bump Ansible to version 2.3  https://review.openstack.org/47660415:39
dpawarmwhahaha: i lost the logs. re-runnig the test. will be able to provide in 20 mins.15:39
bogdandoweshay, ^^ ? do you know the answer by a chance?15:39
openstackgerritJohn Trowbridge proposed openstack/tripleo-quickstart master: Fix Ansible 2.3 bump breaking virt task  https://review.openstack.org/47031215:39
mwhahahadpawar: ok i'm looking for the logs from the node that this runs on to see if the puppet actually gets executed15:40
trownadarazs: there is another patch besides hrybacki's? https://review.openstack.org/#/q/topic:ansible2.315:40
*** dtantsur is now known as dtantsur|afk15:40
weshayEmilienM, as in fixing them?15:40
adarazstrown: I think so, let me find it.15:41
EmilienMweshay: making them working first, and then moving them to the gate. They are in experimental pipeline now15:41
EmilienMweshay: and I don't see progress on making them working15:41
EmilienMweshay: they are critical15:41
weshayEmilienM, aye agree15:41
dpawarmwhahaha: any perticular log file ? or console logs ?15:42
*** pcaruana has quit IRC15:42
mwhahahadpawar: the os-collect-config logs should have the details (ie journalctl -u os-collect-config > file.log)15:43
adarazstrown: this: https://review.openstack.org/#/c/458193/5/roles/libvirt/setup/undercloud/tasks/main.yml15:43
adarazstrown: one is for the undercloud, the other for the supplemental.15:43
hrybackithanks for reminding me about that trown :P15:43
trownadarazs: ah I see, I should have just reverted the revert, it was part of the original patch15:43
trownI will do that instead15:44
adarazstrown: yup. and so there were some problems with "when" conditions not evaluating properly with 2.3 I think, but a full-ci-check should bring those out I think.15:44
*** adarazs has quit IRC15:45
*** adarazs_ is now known as adarazs15:45
openstackgerritJohn Trowbridge proposed openstack/tripleo-quickstart master: Revert "Revert "Bump Ansible to 2.3""  https://review.openstack.org/47660715:46
pandaI don't want to change everyone's backlog, point me at those errors and I'll fix them15:46
pandait is mainly blocking for me15:46
trownpanda: ya I am not commiting to fixing things :P15:46
trownjust putting back up the patch15:46
*** rbowen has joined #tripleo15:46
adarazsI'll try to find what was really wrong.15:46
openstackgerritJohn Trowbridge proposed openstack/tripleo-quickstart master: Fix Ansible 2.3 bump breaking virt task  https://review.openstack.org/47031215:47
pandaalso there was no time to discuss it, but what's our position in pulling external roles to quickstart-extras ?15:47
*** pliu has quit IRC15:47
*** yprokule has quit IRC15:48
openstackgerritMerged openstack/tripleo-heat-templates master: Fix fallback logic in run-os-net-config.sh  https://review.openstack.org/47628015:48
trownpanda: k, https://review.openstack.org/#/c/470312/ will have results just base fixes on top of that :)15:48
*** pliu has joined #tripleo15:48
trownpanda: wdym?15:48
*** yprokule has joined #tripleo15:48
pandaone of the source of problems is ansible-role-kolla-build. I need it for the container periodic and it needs 2.3 to work properly15:48
trownie what are external roles?15:48
trownah15:48
openstackgerritJaganathan Palanisamy proposed openstack/tripleo-common master: DPDK Derive parameters workflow  https://review.openstack.org/46898915:49
pandaalso 2.2 doesn't output at all image building errors, and I'm killing myself in reproducing the upstream ci error in building right now15:49
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates master: [WIP] Deploy Ceph using ceph-ansible via external workflow  https://review.openstack.org/46506615:49
panda"image build failed" that's all in 2.215:49
trownI dont see why we couldnt import the kolla build role to extras... assuming dmsimard is ok with taht15:50
trownor is the question different15:50
pandatrown: no, this is the question15:50
dmsimardI need to double check if it does need 2.315:50
*** bogdando has quit IRC15:51
pandadmsimard: I need 2.3 for other reasons too, like the image build output15:51
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: scenario003: containerize services for CI  https://review.openstack.org/47660915:52
dmsimardtrown, panda: I don't have a strong opinion on where the role lives -- I do think it's something that is generic and can and is used independently of tripleo/tripleo-quickstart however15:52
adarazstrown: I think actually the only other problem was that "ansible_user" that was not populated for localhost in the OVB devmode case. so if you have time please run a devmode.sh --ovb --no-gate --delete-all-stacks with the change too. but I think we fixed that.15:52
dmsimardneed to brb, picking up kid at school15:52
yolandahi, suddenly i cannot deploy OSP8 any more:  Error: Invalid parameter transport on Class[Nova::Migration::Libvirt] at /var/lib/heat-config/heat-config-puppet/e3e75ce9-42cb-4706-b12f-72af33debe56.pp:127 on node overcloud-compute-0.localdomain15:52
yolandais there some known problem?15:52
EmilienMjistr: when you're around, can you maybe quickly look https://review.openstack.org/#/c/476609/ please?15:52
adarazsdmsimard, panda: btw, did you think about making at least a few container builds parallel? it might speed things up (or crash the VM/host. :)15:54
adarazsansible should be able to do that.15:54
*** jaganathan has quit IRC15:54
*** dprince has joined #tripleo15:54
dmsimardadarazs: see my comment in https://review.rdoproject.org/r/717515:54
dmsimardI even blogged about it :) https://dmsimard.com/2017/05/10/limiting-the-amount-of-concurrent-asynchronous-tasks-in-ansible/15:55
adarazsdmsimard: cool stuff! you're one step ahead. and it seems it was far from trivial, heh :)15:57
adarazsdmsimard: congrats for figuring it out.15:57
*** rbowen has quit IRC15:58
openstackgerritAttila Darazs proposed openstack/tripleo-quickstart-extras master: GATE CHECK for quickstart-extras  https://review.openstack.org/47260716:00
*** jpena|brb is now known as jpena16:01
*** egonzalez has quit IRC16:03
*** gbarros has quit IRC16:03
*** nyechiel_ has quit IRC16:08
*** jpich has quit IRC16:08
dpawarmwhahaha: http://paste.openstack.org/show/613411/16:08
*** jschlueter has joined #tripleo16:08
dpawarmwhahaha: "grep -i veritas /var/lib/heat-config/heat-config-puppet/* " do not show anything16:09
jschlueterquestion in openstack overcloud deploy is there a way to specify if it should be enforcing or permissive mode for selinux?16:09
*** karimb has quit IRC16:10
mwhahahadpawar: what was the deploy command line?16:10
*** dparkes has quit IRC16:10
openstackgerritMartin André proposed openstack/tripleo-heat-templates master: [WIP] Copy only generated puppet files into the container  https://review.openstack.org/47615316:11
*** jschluet has joined #tripleo16:12
*** ffiore_ has quit IRC16:12
mwhahahadpawar: that log also seemed to have been truncated, so not sure. it seems to point something not being properly configured in the deploy but i'm not seeing anything at the moment. I have to run but i'll look at the reviews later to see if i can spot anything16:12
*** agurenko has quit IRC16:13
dpawarmwhahaha: http://paste.openstack.org/show/613412/16:13
*** ramishra has quit IRC16:14
openstackgerritwes hayutin proposed openstack/tripleo-quickstart-extras master: Ensure the tripleo-ui-config.js file exits prior to modifying  https://review.openstack.org/47661316:15
mwhahahadpawar: ok i'll take a look when i get back. i don't see anything specific at the moment. i'll let you know16:15
*** oidgar has joined #tripleo16:19
openstackgerritRyan Brady proposed openstack/tripleo-common master: TEST DO NOT MERGE  https://review.openstack.org/47660016:20
dpawarmwhahaha: thanks! i am located in India so would be going offline now. could you please comment on https://review.openstack.org/#/c/475762/7 ?16:22
*** thrash is now known as thrash|biab16:22
*** gbarros has joined #tripleo16:22
weshayEmilienM, ping.. where are the puppet jobs executed from toci?16:27
weshayya.. nvrmind16:28
openstackgerritMerged openstack/tripleo-heat-templates master: Fixes incorrect glance api network  https://review.openstack.org/47618516:29
*** snecklifter has quit IRC16:30
*** dprince has quit IRC16:31
*** yprokule has quit IRC16:31
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates master: Add network/cidr mapping into ServiceNetMap  https://review.openstack.org/47660216:31
*** social has quit IRC16:32
*** jschlueter has quit IRC16:34
*** jschluet is now known as jschlueter16:34
*** social has joined #tripleo16:35
*** dpawar has quit IRC16:38
openstackgerritwes hayutin proposed openstack-infra/tripleo-ci master: update toci to use --environment and --nodes options in quickstart  https://review.openstack.org/47593016:39
openstackgerritGabriele Cerami proposed openstack/tripleo-quickstart-extras master: WIP: enable containers periodic job  https://review.openstack.org/47574516:39
openstackgerritDerek Higgins proposed openstack/tripleo-heat-templates master: Implement scenario006 with Ironic in overcloud  https://review.openstack.org/47480216:41
*** akane has quit IRC16:43
*** fultonj has quit IRC16:45
*** pkovar has quit IRC16:47
*** lucasagomes is now known as lucas-afk16:49
openstackgerritHonza Pokorny proposed openstack/tripleo-ui master: Implement Zaqar logger adapter  https://review.openstack.org/45929416:50
*** myoung has joined #tripleo16:51
openstackgerritMartin André proposed openstack/tripleo-heat-templates master: Copy only generated puppet files into the container  https://review.openstack.org/47615316:52
openstackgerritAssaf Muller proposed openstack/tripleo-quickstart master: Add libselinux-python to bindep requirements  https://review.openstack.org/47662616:52
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates master: Add network/cidr mapping into ServiceNetMap  https://review.openstack.org/47660216:52
*** fultonj has joined #tripleo16:53
*** yamahata has quit IRC17:00
*** derekh has quit IRC17:05
*** ggillies_ has quit IRC17:05
*** trown is now known as trown|lunch17:06
*** ggillies has joined #tripleo17:07
openstackgerritRyan Brady proposed openstack/tripleo-common master: TEST DO NOT MERGE  https://review.openstack.org/47660017:07
*** cylopez has quit IRC17:08
*** agopi has quit IRC17:08
*** dparkes has joined #tripleo17:08
*** akrivoka has quit IRC17:11
*** oidgar has quit IRC17:12
openstackgerritLuigi Toscano proposed openstack/tripleo-heat-templates master: Fix Sahara to use containers in a container scenario  https://review.openstack.org/47663417:12
*** milan has quit IRC17:13
*** tosky has quit IRC17:13
*** pkovar has joined #tripleo17:15
*** jpena is now known as jpena|off17:17
*** sid1 has quit IRC17:17
*** agopi has joined #tripleo17:21
*** mcornea has quit IRC17:22
*** jlinkes has joined #tripleo17:24
*** jlinkes has quit IRC17:24
*** jaganathan has joined #tripleo17:25
*** jbadiapa_ has joined #tripleo17:26
*** pchavva has quit IRC17:28
*** fzdarsky is now known as fzdarsky|afk17:29
*** thrash|biab is now known as thrash17:29
*** anshul has quit IRC17:30
openstackgerritMerged openstack/tripleo-common master: Add tags and skip-tags options to Mistral Ansible actions  https://review.openstack.org/47595217:31
openstackgerritwes hayutin proposed openstack/tripleo-quickstart master: add featureset for containerized undercloud  https://review.openstack.org/47624217:31
sshnaidmtrown|lunch, can you review please in your time https://review.openstack.org/#/c/475197/17:33
*** tosky has joined #tripleo17:33
Sloweruh oh17:33
Slowerdetach: false isn't working..17:33
*** gbarros has quit IRC17:34
openstackgerritIan Main proposed openstack/paunch master: Fix "detach: false"  https://review.openstack.org/47664117:37
openstackgerritJaganathan Palanisamy proposed openstack/tripleo-common master: DPDK and Host derive parameters workflows  https://review.openstack.org/46898917:41
*** yamahata has joined #tripleo17:41
*** pchavva has joined #tripleo17:44
openstackgerritGabriele Cerami proposed openstack/tripleo-quickstart-extras master: WIP: enable containers periodic job  https://review.openstack.org/47574517:45
*** salmankhan has quit IRC17:45
saiping #tripleo how do i exit from the deeploy command if i know the scale up is going to fail (i ctrl+ced from the deploy command and restarted heat-engine). The stack is still stuck in UPDATE_IN_PROGRESS. How do I make it go into ERROR so I can retry the scale up?17:47
*** jaganathan has quit IRC17:49
openstackgerritwes hayutin proposed openstack/tripleo-quickstart master: Add libselinux-python to bindep requirements  https://review.openstack.org/47662617:50
openstackgerritHonza Pokorny proposed openstack/tripleo-common master: Add GUI logging workflows  https://review.openstack.org/46919617:52
*** ykarel has joined #tripleo17:53
openstackgerritJohn Fulton proposed openstack/tripleo-common master: Add Mistral workbook to call ceph-ansible to install Ceph  https://review.openstack.org/46964417:55
*** dprince has joined #tripleo17:59
*** tesseract has quit IRC17:59
*** iranzo has quit IRC17:59
*** pkovar has quit IRC18:01
*** ykarel has quit IRC18:02
bnemecsai: Sorry, missed your ping earlier.  I think nova-compute should only be restarted if there is a configuration change that requires it.18:09
*** gbarros has joined #tripleo18:10
*** gbarros has quit IRC18:10
bnemecsai: Also, I thought restarting heat-engine was supposed to kick a stack into error status, so I'm not sure what else you could try. :-/18:10
saibnemec: np, thank u!18:10
saibnemec: zaneb advised to heat-manage reset_stack_status18:11
bnemecCool18:12
Slowerstevebaker: ping18:13
Slowerdprince: this could make for a lot of troubles.. : https://review.openstack.org/47664118:14
*** zoli|wfh is now known as zoli|gone18:14
*** limao_ has joined #tripleo18:15
*** limao has quit IRC18:15
*** limao_ has quit IRC18:17
*** limao has joined #tripleo18:17
EmilienMweshay: did you find the answer?18:17
weshayEmilienM, the answer is 718:19
fultonjcan https://review.openstack.org/#/c/475728 get restored?18:19
fultonjd0ugal: flaper87 ^ ?18:19
openstackgerritwes hayutin proposed openstack-infra/tripleo-ci master: update toci to use --environment and --nodes options in quickstart  https://review.openstack.org/47593018:21
weshayEmilienM, ya.. I found it.. thanks18:22
*** vpickard is now known as vpickard_18:22
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart-extras master: Use templated network-environment file for OVB  https://review.openstack.org/47478118:22
*** sshnaidm is now known as sshnaidm|off18:24
*** fzdarsky|afk is now known as fzdarsky18:25
openstackgerritIan Main proposed openstack/paunch master: WIP: Add debug tools to paunch  https://review.openstack.org/47665418:30
*** paramite has quit IRC18:31
*** ckyriakidou has joined #tripleo18:35
*** fzdarsky is now known as fzdarsky|afk18:38
*** jprovazn has joined #tripleo18:42
*** jmelvin has quit IRC18:42
jristdtrainor: so18:42
jristundercloud install still failed18:42
dtrainorok - can you ssh to the undercloud and get /home/stack/undercloud_install.log for info?18:43
dtrainorjrist, #oooq ?18:43
jristum18:43
jristaren't we moving away from that? :)18:43
dtrainoroooquickstart18:44
jristI'm trying to figure out a quick/easy way to upload this undercloud_install.log18:44
jristdtrainor: it is oooq, but aren't we moving away from the #oooq channel?18:44
dtrainoryou can prob just tail the last 1000 lines or so18:44
dtrainoroh, i do not know18:44
dtrainormaybe i got it wrong, sorry18:44
dtrainorhttps://docs.openstack.org/developer/tripleo-quickstart/accessing-undercloud.html#access-via-the-cli18:44
*** trown|lunch is now known as trown18:45
jristmight still be the DNS issue18:46
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates master: Add network/cidr mapping into ServiceNetMap  https://review.openstack.org/47660218:46
jristdtrainor: http://paste.openstack.org/show/613430/18:46
jristI am having no luck today18:46
dtrainorhmm you sure those nameserver variables stuck?18:47
dtrainorand, the values correct?18:47
dtrainorthat might be an internal url that doesn't resolve on a normal nameserver idk18:47
jristno I'm not and no I'm not sure :)18:48
jristthat's my nameservers for my house18:49
jristso I know they work18:49
jristand I only used one18:49
jristsince the comma wasn't working18:49
*** limao has quit IRC18:49
dtrainori just curl'd the url from my undercloud which uses 8.8.8.8 and it resolves, and gives me data, so the url seems to be valid18:49
jristya18:51
jristI'm thinking the one time it failed with the comma18:51
jristit was the virthost not being at the end18:51
jristso I'll try one more time with comma18:51
jristso it has two DNS servers18:51
dtrainorgive me the nameservers again?  what are they?18:52
jrist209.244.0.3, 209.244.0.418:52
dtrainorit's possible that your isp nameservers aren't recursive for clients not on your isp's network18:53
jristh18:53
jristnah18:53
jristthat's the nameserver on the router18:53
jristeverything picks those up18:53
dtrainortheory debunked, just tested18:53
dtrainorright, your router is on your isp's network and by proxy everything behind it also is18:54
dtrainorare you on the undercloud right now?18:57
*** pkovar has joined #tripleo18:57
jristno it is installing18:58
*** pkovar has quit IRC18:58
*** suuuper has quit IRC18:59
*** artom has quit IRC19:02
dtrainorcool19:02
dtrainormaybe try curl'ing that url once you get an install and get more info from there19:03
jriston the install task now19:03
*** amoralej is now known as amoralej|off19:04
*** artom has joined #tripleo19:10
openstackgerritwes hayutin proposed openstack/tripleo-quickstart master: add a support check for the openstack version  https://review.openstack.org/45395419:14
openstackgerritGabriele Cerami proposed openstack-infra/tripleo-ci master: WIP: containers periodic test  https://review.openstack.org/47574719:15
openstackgerritSandhya Dasu proposed openstack/tripleo-heat-templates master: cisco nexus: keep OVS on the Compute  https://review.openstack.org/47623819:16
openstackgerritwes hayutin proposed openstack/tripleo-quickstart master: remove outdated playbook, tripleo-roles.yml  https://review.openstack.org/46986419:16
EmilienMdprince: hey, do you know anything about tripleo CI / containers / scenarios status? I did https://review.openstack.org/#/c/476609/ to enable Sahara & Mistral in container jobs, it works, but it doesn't pass Upgrade jobs (the one in containers) - can we go ahead and merge it?19:18
EmilienMdprince: I want to make container jobs in scenarios voting ASAP19:18
jrist:(19:18
toskyEmilienM: uh, sorry, I didn't check and created a (partial) duplicate in https://review.openstack.org/#/c/476634/19:19
dprinceEmilienM: looking...19:19
toskyI guess I can abandon it19:19
EmilienMtosky: thanks, yeah please abandon, I'll do all scenarios, in one19:20
toskyack19:20
toskyI was not sure about the status of the other services19:20
dprinceEmilienM: commented, you might try enabling them independently to get more info about why it fails19:21
openstackgerritTim Rozet proposed openstack/tripleo-heat-templates stable/ocata: Fixes incorrect glance api network  https://review.openstack.org/47666519:21
toskyuh, should I restore the patch?19:21
tosky:)19:21
openstackgerritPradeep Kilambi proposed openstack/tripleo-quickstart master: DNM: TESTING TELEMETRY IN OVB HA  https://review.openstack.org/47666619:22
*** dpivonka has joined #tripleo19:22
dprinceEmilienM: havent' looked at them specifically, but I'd be more familiar with the Mistral side of things19:22
EmilienMdprince: what is failing?19:22
EmilienMdprince: the patch works, see gate-tripleo-ci-centos-7-scenario003-multinode-oooq-container19:23
dprinceEmilienM: sorry, I thought you said the upgrade parts were failing?19:23
EmilienMdprince: it does, but it's because the upgrade process is broken at this stage19:23
EmilienMdprince: Bogdan has sent a lot of bugs in Launchpad about it19:23
EmilienMthe biggest problem is that services are not stopped during the upgrade to containers19:23
EmilienMbut I would like to make progress and at least have gate-tripleo-ci-centos-7-scenario003-multinode-oooq-container green now19:24
EmilienMand iterate on upgrades later19:24
dprinceEmilienM: those should be easy fixes, likely just slipped through the cracks19:24
dprinceas the upgrades architecture came after some of these I think19:24
EmilienMcool19:24
EmilienMdprince: if you +2, we can make progress on the CI, thanks19:25
dprinceEmilienM: +2, your patch looks fine w/ me19:25
*** dprince has left #tripleo19:25
*** dprince has quit IRC19:25
EmilienMthanks19:25
*** dprince has joined #tripleo19:26
*** florianf has quit IRC19:27
*** trown is now known as trown|brb19:28
*** MrRoot has quit IRC19:30
openstackgerritCarlos Camacho proposed openstack-infra/tripleo-ci master: Enable the HA deployment by default  https://review.openstack.org/41517919:34
*** trown|brb is now known as trown19:35
jristdtrainor: gone yet?19:37
jristdtrainor: cat /etc/resolv.conf19:37
jrist[stack@undercloud ~]$ cat /etc/resolv.conf19:37
jrist; generated by /usr/sbin/dhclient-script19:37
jristnameserver 192.168.23.119:37
*** jprovazn has quit IRC19:38
*** bswartz has quit IRC19:38
dtrainorhmm that's the undercloud's ip19:39
jristisn't that dns?19:41
dtrainoryeah but now i'm wondering what happened to the nameservers you actually specified19:41
dtrainorwhere is this being deployed to?  what is your virthost?19:42
*** pchavva has quit IRC19:43
openstackgerritwes hayutin proposed openstack/tripleo-quickstart-extras master: ensure template_check.stat exists  https://review.openstack.org/47667419:44
jristdtrainor: what do you mean19:44
jristthe virt host is a vm on the box?19:44
dtrainoron which box19:44
*** trozet has quit IRC19:44
openstackgerritwes hayutin proposed openstack/tripleo-quickstart master: Revert "Revert "Bump Ansible to 2.3""  https://review.openstack.org/47660719:45
*** prateek has quit IRC19:48
jristdtrainor: just a machine in my office19:49
dtrainorthat's the ip address that the undercloud takes for itself on the ctrlplane network19:51
dtrainori don't know where the dns servers you specified are being applied19:51
jristdtrainor: that's where DNS server information is needed right?19:51
jristI could try putting the DNS servers in there19:51
jristyeah the nameservers in my /etc/resolv.conf are the ones I need for RH and home19:52
*** jcoufal has quit IRC19:53
*** artom has quit IRC19:55
*** agopi has quit IRC19:58
dtrainorjrist, to be honest at this point i don't have much experience with non-rdocloud oooq19:58
jrist:(19:58
jristweshay: is that the right command to pass for DNS servers?19:58
*** lblanchard has quit IRC19:59
jristweshay: the -e undercloud_undercloud_nameservers=209.244.0.3,209.244.0.419:59
weshayya.. pass it directly after quickstart.sh19:59
weshaybut not sure if a list works19:59
weshayso just try w/ one for now20:00
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: scenario002: containerize services for CI  https://review.openstack.org/47668020:00
*** trozet has joined #tripleo20:00
weshayjrist, if all else fails.. tmate me in20:00
*** jmelvin has joined #tripleo20:00
jristweshay: I tried 120:00
jristweshay: tried 220:00
jristweshay: it doesn't seem to be effecting /etc/resolv.conf20:01
openstackgerritDan Sneddon proposed openstack/tripleo-common master: Pass all network data into network/port templates when rendering  https://review.openstack.org/47668220:01
weshayjrist, you need a gui20:01
weshay:)20:01
weshayjrist, let me ssh in20:01
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci master: Test with bonded public interface  https://review.openstack.org/38556220:01
openstackgerritDan Sneddon proposed openstack/tripleo-heat-templates master: Render isolated network templates using jinja2  https://review.openstack.org/47561220:02
jristweshay: :)20:03
openstackgerritMerged openstack/tripleo-quickstart master: baseos-undercloud release configuration for all supported releases.  https://review.openstack.org/46403320:03
openstackgerritDan Sneddon proposed openstack/tripleo-heat-templates master: Render isolated network templates using jinja2  https://review.openstack.org/47561220:04
openstackgerritDan Sneddon proposed openstack/tripleo-common master: Pass all network data to render network/port templates  https://review.openstack.org/47668220:05
*** dprince has quit IRC20:07
*** jrist has quit IRC20:09
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci master: Test with bonded public interface  https://review.openstack.org/38556220:10
*** ckyriakidou has quit IRC20:11
*** abhishek has quit IRC20:11
EmilienMfultonj: where are docker/ceph services in THT?20:12
EmilienMfultonj: we don't add them because of tripleo-ceph-ansible ?20:12
fultonjEmilienM: gfidente has a patch in review20:12
fultonjyes20:13
fultonjso ceph-ansible configures them20:13
fultonjwe have the image provided by ceph in our registry20:13
EmilienMfultonj: my question is more: don't we have docker/ceph-*.yaml files?20:14
EmilienMlike other services?20:14
EmilienMit means ceph can only be containerized with ceph-ansible?20:14
*** liverpooler has quit IRC20:14
fultonjyes20:14
fultonjthat was how we were going to containerize ceph20:14
*** itlinux_ has joined #tripleo20:15
*** rhallisey has quit IRC20:15
itlinux_hello what's the best to clean the instack.json from the undercloud?20:15
fultonjhttps://review.openstack.org/#/q/topic:bp/tripleo-ceph-ansible20:15
itlinux_TY20:15
EmilienMfultonj: k20:15
fultonjok20:15
gfidenteEmilienM currently this is the experimental tht change https://review.openstack.org/#/c/465066/20:16
gfidentethere are three below it20:16
EmilienMk20:16
*** jrist has joined #tripleo20:16
jristweshay: pm'd you20:18
jristweshay: did you get any of it? heh20:18
weshayjrist, ya.. then you quit :(20:18
EmilienMgfidente, fultonj: we'll test ceph-ansible on scenario001-containers FYI20:18
EmilienMdon't create a new scenario for that20:18
weshayjrist, last message was "poke"20:18
EmilienMceph was deployed in scenario001, we'll test your stuff in scenario001-containers20:18
gfidenteEmilienM yeah ideally the existing jobs using ceph20:19
gfidentecan be migrated to ceph-ansible20:19
EmilienMgfidente: do you have plans on testing?20:19
EmilienMor do we wait to finish and merge the code?20:19
openstackgerritMerged openstack/tripleo-heat-templates master: Specify file url when deploy-artifacts fails  https://review.openstack.org/47607420:19
jristweshay: lol it was probably the poke20:19
jristweshay: works now20:19
gfidenteEmilienM as soon as we get it working, I think I can try a change in CI with the appropriate depends_on20:19
EmilienMcool20:20
EmilienMthx for the update :)20:20
*** jkilpatr has quit IRC20:20
gfidenteEmilienM yeah sorry have been out for the last three weeks20:20
*** sbrzozow has quit IRC20:22
openstackgerritIan Main proposed openstack/paunch master: Fix "detach: false"  https://review.openstack.org/47664120:23
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: Rename CongressApi to Congress (docker)  https://review.openstack.org/47669220:25
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: scenario001: containerize services for CI  https://review.openstack.org/47669320:25
*** thrash is now known as thrash|bbl20:26
EmilienMgfidente: no worries at all20:27
openstackgerritIan Main proposed openstack/paunch master: WIP: Add debug tools to paunch  https://review.openstack.org/47665420:27
itlinux_hello what's the best to clean the instack.json from the undercloud?20:27
*** rwsu has quit IRC20:28
*** paramite has joined #tripleo20:33
openstackgerritGabriele Cerami proposed openstack-infra/tripleo-ci master: WIP: containers periodic test  https://review.openstack.org/47574720:34
*** jkilpatr has joined #tripleo20:40
EmilienMvkmc: hey, do we have any patch WIP to containerize manila share and ManilaBackendCephFs ?20:40
EmilienMah found it https://review.openstack.org/#/c/471527/20:40
vkmcEmilienM, '20:41
vkmc^20:41
vkmcI'm currently setting up a tripleo env to see what's missing for that20:41
EmilienMok20:41
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: scenario004: containerize services for CI  https://review.openstack.org/47669820:41
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: CI: Enable OVN services for scenario 3  https://review.openstack.org/41283120:43
tbarronhey is it expected that old-fashioned un-containerized overcloud deploy will still work on master-ci ?20:45
tbarronattempting an overcloud-deploy w/o referencing any env file to pull in the docker stuff (just as a first step) I get this failure with deploy commands that used to work:20:47
itlinux_hello I get a time out on this any suggestions ? Started Mistral Workflow tripleo.baremetal.v1.register_or_update. Execution ID: b1f074d3-53b7-4966-a418-909bad40af8920:48
itlinux_Waiting for messages on queue 'a0a4c020-b743-44f0-836c-2c0192e1a57c' with no timeout.20:48
tbarron(undercloud) [stack@undercloud ~]$ cat failed_deployment_list.log20:48
tbarronovercloud.AllNodesDeploySteps.ControllerDeployment_Step2.0:20:48
tbarron  resource_type: OS::Heat::StructuredDeployment20:48
tbarron  physical_resource_id: ca10ba1a-c987-431c-862c-17a3ef4fda4c20:48
tbarron  status: CREATE_FAILED20:48
tbarron  status_reason: |20:48
tbarron    Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 120:48
tbarron  deploy_stdout: |20:48
tbarron    Notice: hiera(): Cannot load backend module_data: cannot load such file -- hiera/backend/module_data_backend20:48
tbarron    Notice: Scope(Class[Tripleo::Firewall::Post]): At this stage, all network traffic is blocked.20:48
tbarron    Notice: Compiled catalog for overcloud-controller-0.localdomain in environment production in 7.06 seconds20:48
tbarron  deploy_stderr: |20:48
tbarron    exception: connect failed20:48
tbarron    Warning: Unknown variable: 'haproxy_stats_bind_certificate'. at /etc/puppet/modules/tripleo/manifests/haproxy.pp:810:620:48
tbarron    Warning: ModuleLoader: module 'haproxy' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules20:48
tbarron       (file & line not available)20:48
tbarron    Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::String. There is further documentation for validate_legacy function in the README. at ["/etc/puppet/modules/haproxy/manifests/init.pp", 131]:["/var/lib/heat-config/heat-config-puppet/9aebed13-7351-401f-a9ed-59eaec731d94.pp", 71]20:48
tbarron       (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation')20:48
tbarron    Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Bool. There is further documentation for validate_legacy function in the README. at ["/etc/puppet/modules/haproxy/manifests/init.pp", 132]:["/var/lib/heat-config/heat-config-puppet/9aebed13-7351-401f-a9ed-59eaec731d94.pp", 71]20:48
tbarron       (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation')20:48
tbarron    Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Hash. There is further documentation for validate_legacy function in the README. at ["/etc/puppet/modules/haproxy/manifests/init.pp", 135]:["/var/lib/heat-config/heat-config-puppet/9aebed13-7351-401f-a9ed-59eaec731d94.pp", 71]20:48
tbarron       (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation')20:48
tbarron    Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Absolute_Path. There is further documentation for validate_legacy function in the README. at ["/etc/puppet/modules/haproxy/manifests/init.pp", 136]:["/var/lib/heat-config/heat-config-puppet/9aebed13-7351-401f-a9ed-59eaec731d94.pp", 71]20:48
tbarron       (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation')20:48
tbarron    Warning: Undefined variable 'ipaddress_eth0';20:48
tbarron       (file & line not available)20:48
tbarron    Warning: Undefined variable 'netmask_eth0';20:49
tbarron       (file & line not available)20:49
tbarron    Warning: Undefined variable 'ipaddress_ovs_system';20:49
tbarron       (file & line not available)20:49
tbarron    Warning: Undefined variable 'netmask_ovs_system';20:49
tbarron       (file & line not available)20:49
tbarron    Warning: notify is a metaparam; this value will inherit to all contained resources in the keepalived::instance definition20:49
tbarron    Warning: This method is deprecated, please use match expressions with Stdlib::Compat::Ipv6 instead. They are described at https://docs.puppet.com/puppet/latest/reference/lang_data_type.html#match-expressions. at ["/etc/puppet/modules/tripleo/manifests/keepalived.pp", 129]:["/var/lib/heat-config/heat-config-puppet/9aebed13-7351-401f-a9ed-59eaec731d94.pp", 86]20:49
itlinux_tbarron: for a log message like that use pastebin!20:49
tbarron       (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation')20:49
tbarron    Warning: This method is deprecated, please use match expressions with Stdlib::Compat::String instead. They are described at https://docs.puppet.com/puppet/latest/reference/lang_data_type.html#match-expressions. at ["/etc/puppet/modules/memcached/manifests/init.pp", 45]:["/etc/puppet/modules/tripleo/manifests/profile/base/memcached.pp", 30]20:49
tbarron       (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation')20:49
tbarron    Warning: ModuleLoader: module 'mysql' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules20:49
tbarron       (file & line not available)20:49
tbarron    Warning: Unknown variable: '::nova::db::mysql_api::setup_cell0'. at /etc/puppet/modules/nova/manifests/db/mysql.pp:53:2820:49
tbarron    Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Array. There is further documentation for validate_legacy function in the README. at ["/etc/puppet/modules/tripleo/manifests/profile/base/nova.pp", 132]:["/etc/puppet/modules/tripleo/manifests/profile/base/nova/api.pp", 68]20:49
tbarron       (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation')20:49
tbarron    Warning: This method is deprecated, please use the stdlib validate_legacy function, with Pattern[]. There is further documentation for validate_legacy function in the README. at ["/etc/puppet/modules/ntp/manifests/init.pp", 68]:["/etc/puppet/modules/tripleo/manifests/profile/base/time/ntp.pp", 29]20:49
tbarron       (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation')20:49
tbarron    Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Numeric. There is further documentation for validate_legacy function in the README. at ["/etc/puppet/modules/ntp/manifests/init.pp", 76]:["/etc/puppet/modules/tripleo/manifests/profile/base/time/ntp.pp", 29]20:49
tbarron       (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation')20:49
tbarron    Warning: ModuleLoader: module 'rabbitmq' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules20:49
tbarron       (file & line not available)20:50
tbarron    Warning: ModuleLoader: module 'redis' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules20:50
tbarron       (file & line not available)20:50
tbarron    Warning: ModuleLoader: module 'ssh' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules20:50
tbarron       (file & line not available)20:50
tbarron    Warning: ModuleLoader: module 'swift' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules20:50
tbarron       (file & line not available)20:50
tbarron    Warning: ModuleLoader: module 'openstacklib' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules20:50
tbarron       (file & line not available)20:50
tbarron    Warning: ModuleLoader: module 'timezone' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules20:50
tbarron       (file & line not available)20:50
tbarron    Warning: Unknown variable: 'listen_options_real'. at /etc/puppet/modules/tripleo/manifests/haproxy/endpoint.pp:152:2520:50
tbarron    Warning: Scope(Haproxy::Config[haproxy]): haproxy: The $merge_options parameter will default to true in the next major release. Please review the documentation regarding the implications.20:50
tbarron    Warning: Could not find resource 'Exec[wait-for-settle]' in parameter 'require'20:50
tbarron       (at /etc/puppet/modules/pacemaker/manifests/property.pp:86)20:50
tbarron    Error: Could not find dependency Exec[wait-for-settle] for Pcmk_property[property-overcloud-controller-0-manila-share-role] at /etc/puppet/modules/pacemaker/manifests/property.pp:7820:50
tbarronsorry, wrong paste buffer20:50
* tbarron is bad20:50
tbarronit was supposed to be: https://paste.fedoraproject.org/paste/dUWCqJ~K9MDLuBZ8iVwdOA20:50
* tbarron hides20:50
tbarronitlinux_: please see my apology above20:50
tbarronitlinux_: i thought I had the pastebin url available from my mouse click but I didn't20:50
openstackgerritJohn Trowbridge proposed openstack/tripleo-quickstart master: WIP: Test the example nic configs we actually ship  https://review.openstack.org/47670420:52
trozetcan some OOO cores please review https://review.openstack.org/#/c/419648/ ?20:53
openstackgerritEmilien Macchi proposed openstack/tripleo-quickstart-extras master: Support composable roles for deploying overcloud  https://review.openstack.org/45571920:53
*** dhill_ has quit IRC20:54
*** prateek has joined #tripleo20:55
openstackgerritJohn Trowbridge proposed openstack/tripleo-heat-templates master: Copy nic configs from tripleo-ci repo  https://review.openstack.org/47670820:59
*** eck`gone is now known as eck`21:00
rcritowalsh, but the service (the nova part of -p nova/{$::fqdn} would need to be a variable21:02
openstackgerritJohn Trowbridge proposed openstack/tripleo-quickstart master: Test the example nic configs we actually ship  https://review.openstack.org/47670421:02
*** fpan has quit IRC21:03
*** fpan has joined #tripleo21:04
*** gbarros has joined #tripleo21:05
*** abishop has quit IRC21:09
openstackgerritIan Main proposed openstack/paunch master: WIP: Add debug tools to paunch  https://review.openstack.org/47665421:10
*** prateek has quit IRC21:13
*** fragatina has joined #tripleo21:20
*** trozet has quit IRC21:20
*** trown is now known as trown|outtypewww21:21
openstackgerritMerged openstack/tripleo-heat-templates master: scenario003: containerize services for CI  https://review.openstack.org/47660921:21
openstackgerritMerged openstack/tripleo-heat-templates master: Revert "Blacklist support for ExtraConfig"  https://review.openstack.org/47651921:22
openstackgerritwes hayutin proposed openstack/tripleo-quickstart-extras master: ensure template_check.stat exists  https://review.openstack.org/47667421:23
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart master: WIP: change devmode to deploy containerized services by default  https://review.openstack.org/47241221:24
itlinux_am I missing something or the quickstart has some missing steps..but failed on me.. does anyone have a blog with the steps to double check them?TY21:25
stevebakermorning21:25
*** cdearborn has joined #tripleo21:29
openstackgerritDan Sneddon proposed openstack/tripleo-heat-templates master: Render isolated network templates using jinja2  https://review.openstack.org/47561221:29
openstackgerritwes hayutin proposed openstack/tripleo-quickstart master: allow for nondefault ssh port on virthost  https://review.openstack.org/47671321:31
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci master: Test with bonded public interface  https://review.openstack.org/38556221:32
openstackgerritMerged openstack/python-tripleoclient master: Make config options required  https://review.openstack.org/47583121:32
openstackgerritwes hayutin proposed openstack/tripleo-quickstart master: allow for nondefault ssh port on virthost  https://review.openstack.org/47671321:34
*** morazi has quit IRC21:42
openstackgerritwes hayutin proposed openstack/tripleo-quickstart master: allow for nondefault ssh port on virthost  https://review.openstack.org/47671321:45
*** fragatina has quit IRC21:47
openstackgerritGabriele Cerami proposed openstack-infra/tripleo-ci master: WIP: containers periodic test  https://review.openstack.org/47574721:52
openstackgerritMerged openstack-infra/tripleo-ci master: Collect workflow lists and execution lists in host_info  https://review.openstack.org/46506521:52
*** fragatina has joined #tripleo21:52
openstackgerritDan Sneddon proposed openstack/tripleo-heat-templates master: Render isolated network templates using jinja2  https://review.openstack.org/47561221:53
openstackgerritDan Sneddon proposed openstack/tripleo-common master: Pass all network data to render network/port templates  https://review.openstack.org/47668221:53
*** gbarros has quit IRC21:53
*** chlong has quit IRC21:57
*** gbarros has joined #tripleo21:57
*** eck` is now known as eck`gone21:58
openstackgerritwes hayutin proposed openstack/tripleo-quickstart master: allow for nondefault ssh port on virthost  https://review.openstack.org/47671322:00
*** dmacpher has quit IRC22:08
*** cdearborn has quit IRC22:08
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates master: Add network/cidr mapping into ServiceNetMap  https://review.openstack.org/47660222:09
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates master: [WIP] Deploy Ceph using ceph-ansible via external workflow  https://review.openstack.org/46506622:10
*** gfidente has quit IRC22:11
*** fragatina has quit IRC22:19
*** jcoufal has joined #tripleo22:20
*** dmacpher has joined #tripleo22:21
itlinux_does anyone have steps on overcloud for a virtualmachine? TY22:22
*** abhishek has joined #tripleo22:24
*** myoung|remote has joined #tripleo22:24
*** myoung|remote has quit IRC22:24
*** hewbrocca_afk has quit IRC22:26
tbarronitlinux_: i am following https://docs.openstack.org/developer/tripleo-quickstart/readme.html#tripleo-quickstart22:27
itlinux_were you successul ? tbarron:22:28
tbarronitlinux_: with ocata and no changes to anything, yes22:28
tbarronitlinux_: on CentOS 7.322:28
*** weshay has quit IRC22:29
itlinux_tbarron: I did PM you22:30
*** markmc has quit IRC22:31
tbarronitlinux_: and I did answer :)22:31
*** jpena|off has quit IRC22:32
*** rasca has quit IRC22:32
itlinux_:) thanks tbarron!22:32
itlinux_tbarron: do you use virtual machines or BM?22:32
*** weshay has joined #tripleo22:33
tbarronitlinux_: that is on a 64GB centos7.3 baremetal22:33
itlinux_for the UC how about the OC?22:33
tbarronitlinux_: but the undercloud and all the overcloud machines are VMs22:33
itlinux_so you are using virsh! KVM?22:33
itlinux_back to PM tbarron: TY22:34
tbarronitlinux_: the quickstart setup in the link above sets that stuff up for you; it is explained there.22:34
*** markmc has joined #tripleo22:35
*** gbarros has quit IRC22:36
*** rasca has joined #tripleo22:36
*** hewbrocca_afk has joined #tripleo22:36
*** jpena|off has joined #tripleo22:37
*** amoralej|off has quit IRC22:37
*** weshay has quit IRC22:39
*** dmanchad has quit IRC22:39
*** weshay has joined #tripleo22:42
*** dmanchad has joined #tripleo22:43
*** jcoufal has quit IRC22:46
*** dmanchad has quit IRC22:50
openstackgerritMerged openstack/tripleo-quickstart-extras master: Create new arg for composable services  https://review.openstack.org/46928622:50
*** jpena|off has quit IRC22:50
openstackgerritHarald Jensås proposed openstack/instack-undercloud master: Tripleo routed networks ironic inspector, and Undercloud  https://review.openstack.org/43754422:51
*** weshay has quit IRC22:51
*** dmanchad has joined #tripleo22:52
*** jpena|off has joined #tripleo22:52
*** weshay has joined #tripleo22:52
*** markmc has quit IRC22:53
*** amoralej has joined #tripleo22:54
itlinux_anyone bout this error ./quickstart.sh: line 479 no /home/user1/ansible_ssh_env.sh: No such file or directory22:57
*** hewbrocca_afk has quit IRC22:57
*** dmanchad has quit IRC22:59
itlinux_found the way.. need to clone the repo down the quickstart doc does not state that..22:59
*** weshay has quit IRC23:00
openstackgerritGabriele Cerami proposed openstack-infra/tripleo-ci master: WIP: containers periodic test  https://review.openstack.org/47574723:02
*** weshay has joined #tripleo23:03
*** rlandy has quit IRC23:03
*** dmanchad has joined #tripleo23:08
*** chlong has joined #tripleo23:10
*** dmacpher has quit IRC23:10
*** sid1 has joined #tripleo23:11
openstackgerritMerged openstack/instack-undercloud master: Move UI package install to base manifest  https://review.openstack.org/47535223:13
*** abhishek has quit IRC23:18
*** dmanchad has quit IRC23:19
*** paramite has quit IRC23:20
*** dmacpher has joined #tripleo23:22
*** sid1 has quit IRC23:23
*** sid1 has joined #tripleo23:26
*** dmanchad has joined #tripleo23:29
*** itlinux_ has quit IRC23:29
*** tosky has quit IRC23:33
*** amoralej has quit IRC23:33
*** paramite has joined #tripleo23:33
*** dmanchad has quit IRC23:34
*** gbarros has joined #tripleo23:35
*** paramite_ has joined #tripleo23:37
*** jpena|off has quit IRC23:37
*** paramite has quit IRC23:40
*** dmanchad has joined #tripleo23:40
*** jpena|off has joined #tripleo23:41
*** dmacpher has quit IRC23:42
*** jpena|off has quit IRC23:50
*** amoralej has joined #tripleo23:50
*** jpena|off has joined #tripleo23:51
dsneddonemilienM: Heads-up, I did a workflow +1 on this change, when I probably should have only give it a +2, since it's part of a multi-patch BP. Hope that is OK.23:52
dsneddonhttps://review.openstack.org/#/c/46768223:52
dsneddonIt didn't have any Depends-on23:53
*** amoralej has quit IRC23:55
*** jpena|off has quit IRC23:55
*** dmacpher has joined #tripleo23:56
*** jpena|off has joined #tripleo23:59

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!