Wednesday, 2017-08-16

mwhahahawe're working on the fix for that00:00
*** brault has quit IRC00:00
pabelangerdsneddon: ya, look at elastic recheck comment00:00
pabelangerthat will point you to bug00:00
dsneddonI see, Docker failed to download. Thanks.00:00
pabelangerhttps://review.openstack.org/491923 and https://review.openstack.org/494021 are the fixes00:01
pabelangermwhahaha: http://logs.openstack.org/32/491832/2/gate/gate-puppet-tripleo-puppet-syntax-4-centos-7/8d16a46/console.html#_2017-08-15_23_56_06_01923500:03
pabelangerjobs also using new rubygems reverse proxy cache00:03
mwhahahayay \o/00:03
pabelangerwon't be much more things to cache00:04
pabelangerBLAM00:04
pabelangersomething just hit docker reverse proxy cache00:04
pabelanger23.253.111.13800:04
pabelangernow 23.253.90.5400:08
*** snecklifter has quit IRC00:09
*** ooolpbot has joined #tripleo00:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION00:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295500:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171053300:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171077300:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171093800:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm)00:10
*** ooolpbot has quit IRC00:10
openstackLaunchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)00:10
openstackLaunchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,Triaged] - Assigned to John Fulton (jfulton-org)00:10
openstackLaunchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to wes hayutin (weshayutin)00:10
*** snecklifter has joined #tripleo00:14
openstackgerritMichael Chapman proposed openstack/tripleo-quickstart-extras master: Add opendaylight to collect-logs  https://review.openstack.org/49404300:15
fultonjmwhahaha: may i get your advice on an issue i am trying to solve?00:15
mwhahahafultonj: i've got a few minutes, what's up?00:16
fultonjmwhahaha: we have two cent repos in scenario001-multinode-oooq-container00:16
fultonjhttp://logs.openstack.org/88/479288/32/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container/2dd0e52/logs/undercloud/etc/yum.repos.d/00:16
fultonjone is enabled and one is disabled00:16
mwhahahaso we just went through the process of disabling all repos except the ones defined in oooq00:17
fultonjas a result of this a job to install ceph-ansible fails with...00:17
fultonjRepository centos-ceph-jewel is listed more than once in the configuration00:17
fultonjyes, i saw that00:17
mwhahahathe disabled ones are the ones coming from the image00:17
mwhahahathe enabled ones are the ones in oooq00:17
fultonjsounds reasonable00:17
fultonjbut they conflict from having two configs so i can't use either one00:18
mwhahahawhat do you mean00:18
fultonjif your /etc/yum.repos.d matches what we set up00:18
fultonjhttp://logs.openstack.org/88/479288/32/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container/2dd0e52/logs/undercloud/etc/yum.repos.d/00:19
fultonje.g. if you start a centos box and populate it as ^00:19
mwhahahaso i think the problem is they are named the same00:19
mwhahahawe should renme centos-ceph-jewel00:19
fultonjthen you cannot install ceph-ansible00:19
mwhahahaor match cases00:19
mwhahahawhere does ceph-ansible come from?00:19
mwhahahaalso what's the error00:19
fultonjmy comments in https://review.openstack.org/#/c/479288/32 have the details00:20
openstackgerritMichael Chapman proposed openstack/tripleo-quickstart-extras master: Add opendaylight to collect-logs  https://review.openstack.org/49404300:20
fultonjmwhahaha: can you see my last comment? in that review?00:20
fultonji now want to add a fix to get this job working00:21
fultonjbut wanted your advice on an approach00:21
pabelangerI noticed that too, repo setup is not using case sensative for repo file, so some get duplicated00:21
mwhahahafix the names in oooq00:21
pabelangerbut 1 is enabled other disabled00:21
mwhahahato match the rpm provide ones00:21
mwhahahastupid cases00:21
pabelanger+100:21
EmilienMpabelanger: back online, so you said docker proxy works now?00:22
pabelangerEmilienM: yes, I see traffic00:23
pabelangerjust waiting for some jobs to pass00:23
*** catintheroof has joined #tripleo00:23
EmilienMok good00:23
mwhahahafultonj: https://github.com/openstack/tripleo-quickstart/blob/master/config/release/tripleo-ci/consistent-master.yml#L38-L7900:23
mwhahahafultonj: make the file names match the rpm versions00:23
EmilienMpabelanger: from https://review.openstack.org/#/c/491923/ ?00:23
pabelangerthen might promote https://review.openstack.org/49192300:23
fultonjSo, CentOS-Ceph-Jewel.repo is on the image and  whatever sets up centos-ceph-jewel.repo should have made its changes to CentOS-Ceph-Jewel.repo ?00:23
pabelangerEmilienM: https://review.openstack.org/494021 is the actual fix00:23
mwhahahafultonj: if we had done that we wouldn't have neede dto disable anything so yea00:24
pabelangerneeded to spin new images00:24
EmilienMpabelanger: right but we need https://review.openstack.org/#/c/491923/ as well for tripleo ci, no?00:24
pabelangerEmilienM: yes, that is in gate pipeline now00:24
openstackgerritMichael Chapman proposed openstack/tripleo-heat-templates master: Add OPNFV scenario environment  https://review.openstack.org/48690500:24
EmilienMin the check I think00:24
fultonjmwhahaha: ok, thanks. I'll set up that fix.00:25
pabelangerEmilienM: and gate... hmm00:25
pabelangerdid somebody recheck00:25
pabelangerwill keep and eye on it00:26
pabelangerEmilienM: Oh, I know why. I promoted 461000,32, to kick off new jobs. zuul pulled it back in00:29
mwhahahafultonj: alternatively change the names of the repos to include quickstart in the front00:29
pabelangerEmilienM: trying to warm up the reverse proxy cache for docker.io00:29
mwhahahafultonj: i think the problem is the way they are getting read since they are duplicated in the name that when yum reads all the files it's merging them together in an odd fashion00:29
EmilienMpabelanger: cool00:30
pabelangerEmilienM: so, fingers crossed this is the last large thing needed for reverse proxies00:30
EmilienMpabelanger: I'll be online and do recheck in case00:31
EmilienMstevebaker as well in case, maybe we can point the patches to recheck if they fail00:31
stevebakerpabelanger, EmilienM: yep, let me know if there is anything I should nurse through the gate00:33
EmilienMstevebaker: https://review.openstack.org/#/c/491923/00:33
pabelangerk, mirror.ord.rax.o.o is primed00:33
pabelangertook 25mins to cache00:33
pabelangerwaiting for next job to hit to see how long it takes ansible to run00:33
stevebakerEmilienM: I'll watch it00:33
fultonjmwhahaha: would we be relying on a side-effect for alphabetical sorting of the repo files? e.g. did foo.repo disable the bar repos but quickstart-foo.repo enabled it only because the enable statement would be evaluated later?00:34
pabelanger3 more merges happening00:34
mwhahahafultonj: no, i mean not for file name but also for repo name00:34
mwhahahafultonj: the problem is that we have [centos-ceph-jewel] created in two different files00:34
mwhahahafultonj: so if we rename it to [quickstart-centos-ceph-jewel] and it i tin quickstart-centos-ceph-jewel.repo we don't have this problem anymore00:35
fultonjmwhahaha: i see, let me try that then in a prposed oooq patch to change the RDO repos in https://github.com/openstack/tripleo-quickstart/blob/master/config/release/tripleo-ci/consistent-master.yml#L38-L7900:36
openstackgerritAlex Schultz proposed openstack/tripleo-specs master: Propose tech-debt policy  https://review.openstack.org/49404400:36
fultonji'll open a bug too00:36
fultonjthanks00:37
mwhahahafultonj: imho this is a difficiency in our repo management in ansible00:37
openstackgerritMerged openstack/tripleo-heat-templates master: Consolidate deployment in major-upgrade-composable-steps  https://review.openstack.org/49349900:39
openstackgerritMerged openstack/tripleo-ui master: Upgrade webpack and webpack-dev-server  https://review.openstack.org/47586600:39
openstackgerritMerged openstack/puppet-tripleo master: Replace enabled languages with excluded languages in UI  https://review.openstack.org/47354200:39
fultonjmwhahaha: I am curious what you mean by that. I get that quickstart enables some respos w/ ansible and what this name conflict is, but is there some general guideline, or lack of, in quickstart or CI?00:40
fultonjfor repos00:41
mwhahahafultonj: we're just using ansible to create files not manage the whole yum configuration00:41
fultonjmwhahaha: i get it00:41
mwhahahafultonj: unlike in puppet where you can purge your entire yum definition and just declare only have these, we're having to do it in a very ackward way in ansible00:41
mwhahahaso the disable all, add new is because of the file management00:42
mwhahahait's not nice :/ but it's the only way without rm -rf /etc/yum.repo.d/* which is not nice for operators00:42
*** limao has joined #tripleo00:43
fultonjso it's an issue w/ ansible's yum_module00:43
fultonjor is it that we should stop using shell commands and only use the yum module00:43
mwhahahaprobably the later00:44
mwhahahalatter00:44
mwhahahabecause we don't use the yum module00:44
mwhahahaeither way the fix right now would be stop duplicating the repo name, long term it would probably be better to fully use ansible rather than shell scripts for system related configs00:45
fultonjyeah, makes sense.00:45
mwhahahafor tripleo stuff it might make sense for shell scripts00:46
mwhahahabut yum configs, not so much00:46
*** cshastri has joined #tripleo00:48
pabelangerwow, 7 incoming merges00:51
pabelangerlike we are doing things right00:51
*** stendulker has joined #tripleo00:52
openstackgerritMerged openstack/tripleo-quickstart master: enable container periodic job  https://review.openstack.org/47574300:56
openstackgerritMerged openstack/tripleo-quickstart master: Use images created by periodic pike job in pike configs  https://review.openstack.org/49220000:57
openstackgerritMerged openstack/tripleo-ui master: MistralApiService, SwiftApiService error handling  https://review.openstack.org/48438200:57
*** tzumainn has quit IRC00:57
openstackgerritMerged openstack/tripleo-ui master: IronicApiService error handling  https://review.openstack.org/48697600:57
openstackgerritMerged openstack/tripleo-ui master: IronicInspectorApiService error handling  https://review.openstack.org/48697700:57
openstackgerritMerged openstack/tripleo-ui master: HeatApiService error handling  https://review.openstack.org/48697800:57
openstackgerritMerged openstack/tripleo-ui master: KeystoneApiService error handling  https://review.openstack.org/48697900:57
pabelangernice00:57
EmilienMyeah00:57
EmilienMthough these jobs don't run a lot of container jobs00:57
EmilienMsorry00:58
EmilienMthese patches*00:58
*** artom has quit IRC00:58
pabelangerYa, still a few more until 49192300:58
pabelanger90mins out if things keep passing00:58
*** dixiaoli has joined #tripleo01:01
*** dixiaoli_ has joined #tripleo01:02
*** dixiaoli has quit IRC01:02
*** ooolpbot has joined #tripleo01:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION01:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295501:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm)01:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171053301:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171077301:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171093801:10
*** ooolpbot has quit IRC01:10
openstackLaunchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)01:10
openstackLaunchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,Triaged] - Assigned to John Fulton (jfulton-org)01:10
openstackLaunchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to wes hayutin (weshayutin)01:10
*** number80 has quit IRC01:10
*** gkadam has quit IRC01:28
*** yamahata has quit IRC01:35
openstackgerritMerged openstack/tripleo-heat-templates master: Convert network templates to be rendered via j2  https://review.openstack.org/49221801:38
openstackgerritMerged openstack/tripleo-heat-templates master: Bind mount tripleo.cnf in transient bootstrap containers  https://review.openstack.org/49296301:38
*** catintheroof has quit IRC01:47
*** gbarros has joined #tripleo01:48
openstackgerritSteve Baker proposed openstack/tripleo-common master: overcloud_containers.yaml.j2 map images to services  https://review.openstack.org/44832802:01
openstackgerritEmilien Macchi proposed openstack/tripleo-quickstart-extras master: upload containers to undercloud in upgrade scenario  https://review.openstack.org/49397202:03
openstackgerritEmilien Macchi proposed openstack/tripleo-quickstart-extras master: Make env variables for multinode into ansible vars  https://review.openstack.org/48393002:04
EmilienMtrown|outtypewww: rebased ^02:04
*** stendulker has quit IRC02:07
*** trozet has quit IRC02:08
*** cdearborn has quit IRC02:09
*** ooolpbot has joined #tripleo02:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION02:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295502:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm)02:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171053302:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171077302:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171093802:10
*** ooolpbot has quit IRC02:10
openstackLaunchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)02:10
*** trozet has joined #tripleo02:10
openstackLaunchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,Triaged] - Assigned to John Fulton (jfulton-org)02:10
openstackLaunchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)02:10
*** bkopilov has quit IRC02:11
*** trozet has quit IRC02:17
*** trozet has joined #tripleo02:20
openstackgerritJohn Fulton proposed openstack/tripleo-quickstart master: Rename centos repositories created by quickstart  https://review.openstack.org/49405602:30
openstackgerritJohn Fulton proposed openstack/tripleo-heat-templates master: Convert scenario001-multinode-containers job to ceph-ansible  https://review.openstack.org/47928802:32
*** yamahata has joined #tripleo02:34
*** trozet has quit IRC02:38
*** ramishra has quit IRC02:44
*** ramishra has joined #tripleo02:46
*** jmelvin has quit IRC02:46
*** catintheroof has joined #tripleo02:50
itlinuxhas anyone seen this one http://paste.openstack.org/show/618483/ ?? TY02:53
*** catintheroof has quit IRC03:03
*** myoung|remote has joined #tripleo03:09
*** ooolpbot has joined #tripleo03:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION03:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295503:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171053303:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171077303:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171093803:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm)03:10
*** ooolpbot has quit IRC03:10
openstackLaunchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)03:10
openstackLaunchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,Triaged] - Assigned to John Fulton (jfulton-org)03:10
openstackLaunchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)03:10
*** myoung|remote has quit IRC03:10
*** eck` is now known as eck`gone03:19
openstackgerritMatt Young proposed openstack/tripleo-quickstart-extras master: Display output from machine provisioning script  https://review.openstack.org/49302303:28
*** stendulker has joined #tripleo03:28
*** bkopilov has joined #tripleo03:30
*** psahoo has joined #tripleo03:39
*** udesale has joined #tripleo03:44
*** links has joined #tripleo03:47
EmilienMstevebaker: https://review.openstack.org/#/c/448328/ reviewed03:48
*** ykarel_ has joined #tripleo03:49
EmilienMmichapma: I was rebasing https://review.openstack.org/#/c/486905/ on top of a THT patch that was supposed to help03:50
EmilienMmichapma: not sure why you rebased on master03:50
EmilienMmwhahaha: I rebased it on https://review.openstack.org/#/c/492027/ before03:50
stevebakerEmilienM: thanks. that change is mostly from a script which parses THT, so I'll dig in to see if it is doing the right thing03:50
EmilienMstevebaker: oh, ok03:51
EmilienMI'm out03:52
*** shreshtha has joined #tripleo03:53
*** ramishra has quit IRC03:55
*** ramishra has joined #tripleo03:57
openstackgerritDan Sneddon proposed openstack/tripleo-heat-templates master: Render IP map and host maps according to network_data.yaml  https://review.openstack.org/49398404:00
*** gbarros has quit IRC04:06
*** ooolpbot has joined #tripleo04:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION04:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295504:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171053304:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171077304:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171093804:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm)04:10
*** ooolpbot has quit IRC04:10
openstackLaunchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)04:10
openstackLaunchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,Triaged] - Assigned to John Fulton (jfulton-org)04:10
openstackLaunchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)04:10
*** dtrainor has quit IRC04:11
*** dtrainor has joined #tripleo04:11
*** brault has joined #tripleo04:16
*** brault has quit IRC04:21
*** limao has quit IRC04:22
*** nyechiel has joined #tripleo04:44
*** brault has joined #tripleo04:46
*** brault has quit IRC04:50
*** limao has joined #tripleo05:01
*** janki has joined #tripleo05:03
*** pdeore has joined #tripleo05:06
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo master: Remove extra keystone admin haproxy listen and allow TLS  https://review.openstack.org/49393705:07
*** ooolpbot has joined #tripleo05:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION05:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295505:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171053305:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171077305:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171093805:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm)05:10
*** ooolpbot has quit IRC05:10
openstackLaunchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)05:10
openstackLaunchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,Triaged] - Assigned to John Fulton (jfulton-org)05:10
openstackLaunchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)05:10
*** anshul has joined #tripleo05:19
*** rbrady has quit IRC05:22
*** rbrady has joined #tripleo05:22
*** rbrady has joined #tripleo05:22
*** pdeore is now known as pdeore|afk05:23
*** yprokule has joined #tripleo05:24
*** mdnadeem has joined #tripleo05:27
*** udesale__ has joined #tripleo05:35
*** udesale has quit IRC05:35
*** mdnadeem has quit IRC05:37
*** agurenko has joined #tripleo05:41
*** brault has joined #tripleo05:44
*** pdeore|afk is now known as pdeore05:44
*** pcaruana has joined #tripleo05:45
*** masco has joined #tripleo05:46
*** brault has quit IRC05:48
*** mdnadeem has joined #tripleo05:49
*** pgadiya has joined #tripleo05:50
*** iranzo has joined #tripleo05:56
*** marios has joined #tripleo06:00
openstackgerritAttila Darazs proposed openstack/tripleo-quickstart-extras master: GATE CHECK for quickstart-extras  https://review.openstack.org/47260706:00
*** dpawar has joined #tripleo06:00
*** iranzo1 has joined #tripleo06:01
*** iranzo has quit IRC06:01
*** dsariel has joined #tripleo06:02
*** iranzo1 has quit IRC06:03
*** iranzo has joined #tripleo06:03
*** iranzo has joined #tripleo06:03
*** jbadiapa has joined #tripleo06:04
*** brault has joined #tripleo06:05
*** iranzo has quit IRC06:09
*** brault has quit IRC06:10
*** ooolpbot has joined #tripleo06:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION06:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295506:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171053306:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171077306:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171093806:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm)06:10
*** ooolpbot has quit IRC06:10
openstackLaunchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)06:10
openstackLaunchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,Triaged] - Assigned to John Fulton (jfulton-org)06:10
openstackLaunchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)06:10
*** skramaja has joined #tripleo06:12
*** iranzo has joined #tripleo06:15
*** iranzo has quit IRC06:15
*** iranzo has joined #tripleo06:15
*** jfrancoa has joined #tripleo06:20
*** udesale has joined #tripleo06:23
*** udesale__ has quit IRC06:23
*** dciabrin has joined #tripleo06:25
*** iranzo has quit IRC06:28
*** iranzo has joined #tripleo06:28
*** iranzo has joined #tripleo06:28
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo master: Certmonger: Make postsave command configurable  https://review.openstack.org/49408506:29
*** limao has quit IRC06:29
*** limao has joined #tripleo06:29
*** bogdando has joined #tripleo06:31
openstackgerritItzik Brown proposed openstack/tripleo-heat-templates master: Adding a Missing configuration to support QoS in ODL Closes-Bug: 1708131  https://review.openstack.org/49202706:31
openstackbug 1708131 in tripleo "AttributeError: 'NoneType' object has no attribute 'get_policies'" [High,In progress] https://launchpad.net/bugs/1708131 - Assigned to Emilien Macchi (emilienm)06:31
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo master: Certmonger: Make postsave command configurable  https://review.openstack.org/49408506:33
*** udesale has quit IRC06:35
skramajad0ugal: could you take a look at https://review.openstack.org/#/c/491753/?06:36
*** udesale has joined #tripleo06:36
*** florianf has joined #tripleo06:37
*** cylopez has joined #tripleo06:39
*** aufi has joined #tripleo06:40
*** paramite has joined #tripleo06:41
*** jaganathan has joined #tripleo06:42
openstackgerritMerged openstack/tripleo-quickstart master: Use undercloud docker proxy  https://review.openstack.org/49192306:42
openstackgerritMerged openstack/tripleo-heat-templates master: Don't unregister on system/resource delete  https://review.openstack.org/49297006:43
*** stendulker has quit IRC06:44
openstackgerritJiri Tomasek proposed openstack/tripleo-ui master: Update updatePlan action  https://review.openstack.org/48919606:46
*** pgadiya has quit IRC06:54
*** jprovazn has joined #tripleo06:55
*** rcernin has joined #tripleo06:57
honzaI'm running into nokogiri build issues in the puppet syntax job, is that a known thing?  http://logs.openstack.org/08/469608/8/check/gate-instack-undercloud-puppet-syntax-4-centos-7/a3ccc18/console.html#_2017-08-16_04_24_27_92815107:05
*** pgadiya has joined #tripleo07:07
*** ooolpbot has joined #tripleo07:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION07:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295507:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171053307:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171077307:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171093807:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm)07:10
*** ooolpbot has quit IRC07:10
openstackLaunchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)07:10
openstackLaunchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,Triaged] - Assigned to John Fulton (jfulton-org)07:10
openstackLaunchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)07:10
*** shardy_afk is now known as shardy07:10
*** ebarrera has joined #tripleo07:13
honzajtomasek: do we have a bug for the CORS issue with PATCH in ironic?  which services are affected?  is there a repro case?07:13
apetrichhonza, https://review.openstack.org/#/c/469608/ is passing and ready for reviews :)07:14
apetricha lot to merge to get that in though07:14
honzaapetrich: passing?  what about that weird nokogiri issue?07:15
* apetrich needs coffee.. 07:15
apetrichhonza, sorry07:15
honzaapetrich: no, no, maybe that's fine07:15
*** rcernin has quit IRC07:16
apetrichhonza, oh that's the coverage that has been acting up07:16
*** number80 has joined #tripleo07:17
shardyHi all, can I get a second reviewer for https://review.openstack.org/#/c/490839 please, thanks!07:18
*** ccamacho has joined #tripleo07:18
apetrichhonza, sorry I was looking into the wrong review. I WILL GET MORE COFFEE. bbiam07:18
*** jlinkes has joined #tripleo07:18
honzaapetrich: i linked directly to the error above07:19
*** jpena|off is now known as jpena07:19
shardycoffee++ :)07:19
*** rcernin has joined #tripleo07:19
honzaif that helps with the uncaffeinated state07:19
* apetrich curses libxml for the first time this month07:20
*** brault has joined #tripleo07:25
bogdandofolks, do we need this https://review.openstack.org/#/c/492178/ ? I can use it from my fork as well07:26
bogdandoI hoped it helps to tshoot failed CI jobs' logs07:27
bogdandoplease try it, if you do :)07:27
*** jpich has joined #tripleo07:28
jtomasekhonza: I don't see it in launchpad, so it has not been filed yet, I can do so07:29
bogdandoplease review the logrotate for contnainers' logs topic https://review.openstack.org/#/q/topic:bug/170091207:29
honzajtomasek: please and thank you :)07:29
shardybogdando: I added derekh to the getthelogs one, as AFAICS he's the only person who's ever touched that script before07:29
apetrichback07:30
bogdandoshardy: hm, I hoped more ppl are used to grep through the CI jobs logs07:30
bogdandobut thanks shardy! :)07:30
shardybogdando: maybe, but it'd be good to get derek to take a look anyway I think07:30
shardyI'll try to review later, but tbh I don't use that script locally07:30
bogdandoshardy: how do you normally inspect logs of failed jobs?07:31
bogdandoI need more feedback from folks, the script may help with that task07:31
shardybogdando: just using the web ui and knowledge of which files to look at07:31
bogdandoyeah, right. That works :)07:31
shardyI can see that grep would be useful sometimes, but I've only occasionally needed to do that07:32
*** shardy is now known as shardy_afk07:32
bogdandoshardy: it really starts to be needed, then debugging distributed (HA) issues ;)07:32
bogdandoand sorted in time07:32
bogdandoso that's the only reason I started to download logs then apply filters like I track here https://github.com/bogdando/fuel-log-parse07:33
bogdandootherwise, it's hard to get a full view of pacemaker cluster distributed events07:33
bogdandobut for non HA jobs, it's ok to just look into web ui, agree07:34
bogdandos/then/when debugging*07:34
d0ugalskramaja: sure07:37
jtomasekhonza: https://bugs.launchpad.net/tripleo/+bug/171106107:37
openstackLaunchpad bug 1711061 in tripleo "ironic cors don't allow PATCH method" [High,Triaged]07:37
skramajathanks d0ugal07:37
honzajtomasek: excellent bug report, thanks07:37
*** oidgar has joined #tripleo07:40
openstackgerritBogdan Dobrelya proposed openstack/tripleo-heat-templates master: Add logrotate with crond service  https://review.openstack.org/49005007:40
*** brault has quit IRC07:40
*** egonzalez has joined #tripleo07:41
openstackgerritBogdan Dobrelya proposed openstack/puppet-tripleo master: Add logrotate-crond configuration  https://review.openstack.org/49004807:43
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Make swift's endpoint type configurable for gnocchi storage  https://review.openstack.org/49410307:47
openstackgerritBogdan Dobrelya proposed openstack/tripleo-heat-templates master: Persist containerized services httpd logs  https://review.openstack.org/48995507:48
*** brault has joined #tripleo07:49
bogdandothis one https://review.openstack.org/#/c/489955 was ready for merge, please take care of it once it passes CI07:54
openstackgerritBogdan Dobrelya proposed openstack/tripleo-heat-templates master: Add logrotate with crond service  https://review.openstack.org/49005007:55
oidgarhi everybody, I'm still struggling to deploy overcloud with brand new undercloud server07:59
oidgarwhen I'm running "openstack overcloud deploy --templates" it fails on "overcloud.AllNodesDeploySteps.ComputeDeployment_Step4.0"07:59
oidgarI'm getting in addition "[overcloud]: CREATE_FAILED  Create timed out"08:00
jaosorioroidgar: what's the output of openstack stack failures list --long overcloud   ?08:00
oidgarjaosorior: nothing08:00
oidgarstdout and stderr are empty08:00
jaosorioruhm... so it did fail on the timeout08:00
jaosorioroidgar: I guess the only way to check the controller nodes' logs and check what's taking so long08:01
openstackgerritJose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Heat]  https://review.openstack.org/49139908:01
openstackgerritJose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Aodh]  https://review.openstack.org/49141108:01
openstackgerritJose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Mistral]  https://review.openstack.org/49347108:01
openstackgerritJose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Ceilometer]  https://review.openstack.org/49039208:01
openstackgerritJose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Neutron]  https://review.openstack.org/48995708:01
openstackgerritJose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Gnocchi]  https://review.openstack.org/49143308:01
*** dparkes has joined #tripleo08:01
oidgarjaosorior: which logs do you recommend to check?08:01
oidgarjaosorior: and I think the problem is with the compute node08:01
oidgarjaosorior: also I get "Permission denied" when trying to connect with heat-admin user... but I changed the ssh key before deploying overcloud so the question is if this can affect08:04
openstackgerritBogdan Dobrelya proposed openstack/tripleo-quickstart master: Add support for openstack inventory provider  https://review.openstack.org/46370108:04
jaosorioroidgar: I'm honestly not sure if that would be a problem. Maybe?08:05
*** lucas-afk is now known as lucasagomes08:06
jaosoriorHey guys, could someone check this puppet-tripleo change out https://review.openstack.org/#/c/493937/ ?08:06
oidgarjaosorior: I guess I'll try to reinstall again (in the 100th time this week) the undercloud vm to make sure it's sterile...08:06
jaosoriorAlso, any reviews for this series is appreciated https://review.openstack.org/#/q/topic:bp/tls-via-certmonger-containers+status:open08:06
mariosjfrancoa: o/ hey did you drop api accidentally here https://review.openstack.org/#/c/491433/708:07
michapmaEmilienM: I didn't realise you'd rebased on something in flight - I only meant to add the depends-on to get the ODL log08:07
oidgarjaosorior: thanks for the help anyway, let's see if it work after this small change08:08
jfrancoamarios: o/ I did understand from your comment that it was not necessary (remove this?)08:08
mariosjfrancoa: oh i meant the highlighted bit https://review.openstack.org/#/c/491433/6/docker/services/gnocchi-api.yaml ("post upgrade")08:09
*** ooolpbot has joined #tripleo08:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION08:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295508:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm)08:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171053308:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171077308:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171093808:10
openstackLaunchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)08:10
*** ooolpbot has quit IRC08:10
openstackLaunchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,Triaged] - Assigned to John Fulton (jfulton-org)08:10
openstackLaunchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)08:10
*** derekh has joined #tripleo08:10
mariosjfrancoa: sorry for that, i do think it still makes sense to have a check it *is* under httpd before stopping there.  Its just that the 'post upgrade' part was confusing (I think it was copied from the puppet/services/ one possibly?) since the 'upgrade to httpd gnochi api ' happened last cycle08:10
openstackgerritJanki Chhatbar proposed openstack/tripleo-heat-templates master: Add param to configure snat mechanism  https://review.openstack.org/49386108:11
jfrancoamarios: ups... :-S yes, I took it from the cinder-api.yaml service. I want to wait for CI results anyway, because this was the only commit from bug/1704389 which got a -1 and I was wondering if that step could be the problem08:12
jfrancoamarios: I will add the step back once I check the CI jobs08:12
mariosjfrancoa: thanks,  added comment at https://review.openstack.org/#/c/491433/6 as well08:13
jfrancoamarios: good, thanks!08:14
*** jlinkes has quit IRC08:14
mariosjfrancoa: added another note at https://review.openstack.org/#/c/491433/6/docker/services/gnocchi-api.yaml :) when you get a chance thanks08:17
*** shardy_afk is now known as shardy08:17
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Containarise Barbican API  https://review.openstack.org/48145108:20
*** mcornea has joined #tripleo08:21
openstackgerritMichael Chapman proposed openstack/tripleo-heat-templates master: Adding a Missing configuration to support QoS in ODL Closes-Bug: 1708131  https://review.openstack.org/49202708:21
openstackbug 1708131 in tripleo "AttributeError: 'NoneType' object has no attribute 'get_policies'" [High,In progress] https://launchpad.net/bugs/1708131 - Assigned to Itzik Brown (itzikb1)08:21
openstackgerritMichael Chapman proposed openstack/tripleo-heat-templates master: Add OPNFV scenario environment  https://review.openstack.org/48690508:21
*** gfidente has joined #tripleo08:23
*** gfidente has quit IRC08:23
*** gfidente has joined #tripleo08:23
openstackgerritDougal Matthews proposed openstack/tripleo-common master: Automatically retry introspection for failing nodes  https://review.openstack.org/46291608:24
*** achadha has joined #tripleo08:24
openstackgerritHonza Pokorny proposed openstack/tripleo-ui master: Don't log app state in Zaqar  https://review.openstack.org/49386308:24
*** ramishra has quit IRC08:24
*** jlinkes has joined #tripleo08:25
*** amoralej|off is now known as amoralej08:25
achadhaany changes with pacemaker/corosync in 7.4? I see deployments fail coz corosync-cluster doesn't form? Started seeing this only after moving to 7.4 (everything worked fine in 7.3)08:25
*** ramishra has joined #tripleo08:27
*** chem has joined #tripleo08:27
openstackgerritItzik Brown proposed openstack/tripleo-heat-templates master: Adding a Missing configuration to support QoS in ODL  https://review.openstack.org/49202708:29
openstackgerritMerged openstack/tripleo-validations master: Fix `when` statement in deployment image validation  https://review.openstack.org/48297908:34
*** dixiaoli has joined #tripleo08:34
*** dixiaoli has quit IRC08:36
*** dixiaoli_ has quit IRC08:38
*** ykarel_ is now known as ykarel|lunch08:48
*** jbadiapa_ has joined #tripleo08:50
*** jbadiapa has quit IRC08:51
openstackgerritOpenStack Proposal Bot proposed openstack/tripleo-ui master: Imported Translations from Zanata  https://review.openstack.org/49412708:53
*** salmankhan has joined #tripleo08:57
*** hewbrocca_afk is now known as hewbrocca08:58
*** psahoo has quit IRC08:59
*** stendulker has joined #tripleo09:05
*** psahoo has joined #tripleo09:05
*** lucasagomes is now known as lucas-brb09:08
openstackgerritJiri Tomasek proposed openstack/tripleo-ui master: Fix plan export URL  https://review.openstack.org/48816409:09
openstackgerritJiri Tomasek proposed openstack/tripleo-ui master: App config and I18n selectors refactor  https://review.openstack.org/49414109:09
*** ooolpbot has joined #tripleo09:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION09:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295509:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171053309:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171077309:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171093809:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm)09:10
*** ooolpbot has quit IRC09:10
openstackLaunchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)09:10
openstackLaunchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,Triaged] - Assigned to John Fulton (jfulton-org)09:10
openstackLaunchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)09:10
*** milan has joined #tripleo09:13
*** dixiaoli has joined #tripleo09:14
openstackgerritDougal Matthews proposed openstack/python-tripleoclient master: Allow users to specify the Introspection batch size  https://review.openstack.org/45122809:15
openstackgerritDougal Matthews proposed openstack/python-tripleoclient master: Allow users to specify the Introspection concurrency  https://review.openstack.org/45122809:18
socialmarios: re updates and --update-plan-only vs update_noop.yaml09:18
mariossocial: o/09:21
*** limao has quit IRC09:22
jaosoriorbandini: got a chance to check this out https://review.openstack.org/#/c/493561/ ?09:24
openstackgerritChandan Kumar proposed openstack/tripleo-quickstart-extras master: Switch to overcloudrc.v3 for running tempest  https://review.openstack.org/49303009:25
socialmarios: so our issue is that update-plan-only didn't update any parameters and the update_tasks didn't get created in heat so it's kinda useless?09:25
mariossocial: in a call right now (join us?)09:25
socialmarios: on call with lyan09:26
*** tosky has joined #tripleo09:26
social*lee09:26
mariossocial: ack09:31
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates master: Render IP map and host maps according to network_data.yaml  https://review.openstack.org/49398409:34
*** achadha has quit IRC09:35
*** deadnull has joined #tripleo09:36
openstackgerritJose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Heat]  https://review.openstack.org/49139909:39
openstackgerritJose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Aodh]  https://review.openstack.org/49141109:39
openstackgerritJose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Mistral]  https://review.openstack.org/49347109:39
openstackgerritJose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Gnocchi]  https://review.openstack.org/49143309:39
*** deadnull has quit IRC09:41
jfrancoamarios: does this make sense? https://review.openstack.org/#/c/491399/8/docker/services/heat-api.yaml09:41
openstackgerritMerged openstack/diskimage-builder master: Increase timeout for removal  https://review.openstack.org/49302609:45
*** dixiaoli has quit IRC09:45
*** ykarel|lunch is now known as ykarel09:46
*** cshastri has quit IRC09:47
*** dixiaoli has joined #tripleo09:48
sshnaidmjaosorior, hi09:49
sshnaidmjaosorior, I prepare white/exclude lists for tripleo jobs logs and I'd like to ensure I include all required logs from IPA job09:50
sshnaidmjaosorior, sorry, not logs - /etc configs09:50
sshnaidmjaosorior, I've added /etc/ipa, /etc/sssd. Something else..? Do you need /etc/gss /etc/gssproxy, /etc/gnupg ?09:51
*** akrivoka has joined #tripleo09:51
jaosoriorsshnaidm: not really09:55
jaosoriorbut /etc/ipa and /etc/sssd are good :)09:55
sshnaidmjaosorior, and all /etc/krb5.* ?09:55
jaosoriorsshnaidm: oh, that would be good too. Not the keytab file though09:56
sshnaidmjaosorior, ok09:57
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: DONT REVIEW: test undercloud containers  https://review.openstack.org/49415110:01
*** snecklifter is now known as snecklifter|away10:01
*** dixiaoli has quit IRC10:03
openstackgerritDmitry Tantsur proposed openstack/tripleo-common master: Set resource_class=baremetal for newly enrolled nodes  https://review.openstack.org/49394310:05
mariosjfrancoa: yes it lgtm but agree on checking running10:06
openstackgerritDmitry Tantsur proposed openstack/instack-undercloud master: Switch to scheduling based on resource classes  https://review.openstack.org/49085110:06
*** dtantsur|afk is now known as dtantsur10:06
dtantsurbnemec: the ordering should be good now ^^^10:07
skramajadsneddon: i think the templates for dpdk shoul be separate given the fact that there should not be mixing of ovs and ovs_user bridges. i have left a comment on https://review.openstack.org/#/c/474533/, ptal10:07
*** ooolpbot has joined #tripleo10:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION10:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295510:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171053310:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171077310:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171093810:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm)10:10
*** ooolpbot has quit IRC10:10
openstackLaunchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)10:10
openstackLaunchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,Triaged] - Assigned to John Fulton (jfulton-org)10:10
openstackLaunchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)10:10
*** dsariel has quit IRC10:10
*** dtrainor has quit IRC10:11
*** cylopez has quit IRC10:16
chemmarios: could we land https://review.openstack.org/#/c/491799/ ?10:16
*** lucas-brb is now known as lucasagomes10:18
marioschem: looking10:20
*** yamahata has quit IRC10:21
chemmarios: thanks (and this one for the road as well https://review.openstack.org/#/c/493825/ ...)10:22
marioschem: done, i guess it needs to go to stable/ocata too (from the bug). i was initially confused thought it was clone for bug https://bugs.launchpad.net/tripleo/+bug/1706951 but it is newton.. ocata in this case10:23
openstackLaunchpad bug 1706951 in tripleo "Ocata to Pike upgrade fails when cinder-volume runs on host because cinder-manage db sync runs when galera is unavailable" [Critical,In progress] - Assigned to Marios Andreou (marios-b)10:23
*** snecklifter|away is now known as snecklifter10:24
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: Create a whitelist for /etc configs  https://review.openstack.org/49397310:25
chemmarios: yeap10:25
marioschem: k voted on the second one too, just wondering is there a case where we don't have polling but have compute?10:27
*** chandankumar is now known as chkumar|travel10:27
chemmarios: legit question10:28
chemlyarwood: hi, regarding https://review.openstack.org/#/c/493825/ are we sure that polling is always there or do we need to check ?10:29
*** udesale has quit IRC10:29
*** jbadiapa_ has quit IRC10:30
*** jbadiapa_ has joined #tripleo10:31
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: Exclude list for logs collection  https://review.openstack.org/49402210:32
*** atoth has quit IRC10:33
lyarwoodchem: looking10:34
gfidenteflaper87 EmilienM I am unable to reproduce the image upload issue in my env, that is, it works for me10:35
lyarwoodchem: I think it's the other way around, polling should always be there now and compute might not be iirc10:35
lyarwoodchem: I'll add a comment with links10:36
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: Create a whitelist for /etc configs  https://review.openstack.org/49397310:37
openstackgerritMartin AndrĂ© proposed openstack/tripleo-heat-templates master: Remove unused docker-centos-tripleoupstream.yaml  https://review.openstack.org/49415810:39
*** florianf has quit IRC10:42
chemlyarwood: thanks :)10:44
lyarwoodchem: done, so yeah we need to check both really10:44
lyarwoodchem: we moved to polling in Pike, so anyone moving from BM to containers (a useful test case) would hit this10:45
openstackgerritMartin AndrĂ© proposed openstack/tripleo-heat-templates master: Stop hardcoding host's config volume path  https://review.openstack.org/49416310:47
chemlyarwood: oki, cool, and thanks marios, he's the one who spotted it :)10:48
lyarwoodmarios++10:48
mariosthanks chem lyarwood10:48
*** stendulker has quit IRC10:49
*** dparkes has quit IRC10:49
*** bkopilov has quit IRC10:50
gfidenteflaper87 you around?10:52
gfidenteflaper87 I think I need your skills here10:52
gfidente :D10:52
openstackgerritMarios Andreou proposed openstack/tripleo-heat-templates master: Adds PostUpgradeConfigStepsDeployment to drive post config ansible  https://review.openstack.org/49387810:53
mariosshardy: had a go at making it a variable instead of overriding tags when you get another chance thanks ^^10:53
openstackgerritLee Yarwood proposed openstack/tripleo-heat-templates master: docker: Stop all active ceilometer services during compute upgrade  https://review.openstack.org/49382510:53
*** jkilpatr has quit IRC10:54
shardymarios: hey, thanks - I wasn't blocking on it btw, just wanted to have the discussion10:54
shardymarios: seems like the tags are proving a headache for running outside heat, so I was thinking we'd move away from them in favor of vars10:55
shardywill check it out10:55
mariosshardy: i kinda like it this way and it wasn't too hard to change so I thought i'd post it for discussion.10:55
shardymarios: ack, thanks, will review10:55
mariosshardy: thank you10:55
shardyI still kinda wonder if we should just sed all the things and convert t-h-t to variables instead of tags10:55
mariosshardy: it is a bit risk though i mean given everything else going on10:55
shardymarios: yeah, I guess so10:55
mariosshardy: minor update, update non controller, then a list of blocking bugs like the cinder db sync that this change is addressing10:56
shardymaybe we can find a way to transition to vars in early queens tho10:56
openstackgerritLee Yarwood proposed openstack/tripleo-heat-templates master: docker: Stop all active ceilometer services during compute upgrade  https://review.openstack.org/49382510:56
*** florianf has joined #tripleo10:57
*** jpena is now known as jpena|lunch11:05
trown|outtypewwwEmilienM: that rebase is incorrect... you undid part of my change... can we please stop merging conflicting patches ahead of the 3nodes patches with no tripleo-ci core +2's on them?11:07
chemlyarwood: maybe we should add the check only on step2, unless I miss something ?11:08
*** ooolpbot has joined #tripleo11:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION11:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295511:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm)11:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171053311:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171077311:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171093811:10
*** ooolpbot has quit IRC11:10
openstackLaunchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)11:10
*** lucasagomes is now known as lucas-hungry11:10
openstackLaunchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,Triaged] - Assigned to John Fulton (jfulton-org)11:10
openstackLaunchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)11:10
openstackgerritLee Yarwood proposed openstack/tripleo-heat-templates master: docker: Stop all active ceilometer services during compute upgrade  https://review.openstack.org/49382511:10
lyarwoodchem: ack yeaaaah, not sure why I left it as common, thanks11:10
gfidentejprovazn is it possible the problem you found in https://review.openstack.org/482496 is https://bugs.launchpad.net/tripleo/+bug/1710773 ?11:17
openstackLaunchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,Triaged] - Assigned to John Fulton (jfulton-org)11:17
* jprovazn looks11:18
jprovazngfidente: thanks for pointing this out, I'm trying to reproduce this since this morning11:20
gfidentejprovazn yeah except that I was unable to11:20
gfidentethat config works for me11:20
jprovazngfidente: well, my first thought is that it's not directly caused by containerized glance11:20
jprovazngfidente: because I'm hitting this in CI on scenario001-multinode-oooq too11:20
*** dsariel has joined #tripleo11:20
gfidentethat is using ceph too11:20
gfidentesee the bug title11:21
jprovazngfidente: but not containerized, right?11:21
gfidenteah non-containerized right11:21
jprovazngfidente: yea, but what I did actualyl find11:21
gfidenteI think it says writes are currently waiting for active PGs11:22
jprovazngfidente: if I omitted pooldefaultsize to set it to 1 (and left 3), then stack deployed but ceph cluster was unhappy and tempest was failing with this issue - uploading iamge to glance11:22
gfidenteright yes11:22
gfidentewas about to post just that change to try it11:22
gfidentechecking the scenario envs :D11:22
jprovazngfidente: cool11:23
jprovazngfidente: I was looking at them, but it seems that defaultsize is 1 for 00111:23
gfidentePGs are degraded because of size if it's not 111:23
jprovazngfidente: but it might be some other env file overwrites the ci/environments/scenario001-multinode.yaml11:23
jprovazngfidente: I failed to find any logs from overcloud, I guess we don't copy them, do we?11:24
gfidenteright though not override the same param I hope11:24
gfidenteI see it set in scenario004 as well11:26
*** jbadiapa_ has quit IRC11:27
gfidentejprovazn http://logs.openstack.org/29/490129/4/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container/1d6c57a/logs/subnode-2/etc/ceph/ceph.conf.txt.gz pool_default_size is indeed wrong11:28
*** jkilpatr has joined #tripleo11:29
*** atoth has joined #tripleo11:29
jprovazngfidente: I wonder where does it sneak in?11:30
gfidenteyeah11:30
openstackgerritHonza Pokorny proposed openstack/tripleo-ui master: App config and I18n selectors refactor  https://review.openstack.org/49414111:30
*** brault has quit IRC11:31
jprovazngfidente: hm, though in my patch where I hit glance error too, I can see that pool_size is set to 1 - http://logs.openstack.org/96/482496/7/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq/3753922/logs/subnode-2/11:35
gfidenteis it the same issue then? :D11:35
gfidentejprovazn not the same it seems http://logs.openstack.org/96/482496/7/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq/3753922/logs/subnode-2/var/log/glance/api.log.txt.gz#_2017-08-15_12_24_39_53611:36
gfidentejprovazn oh indeed we set it to 1 only in the non-containers11:38
jprovazngfidente: that would explain it11:38
openstackgerritMarios Andreou proposed openstack/instack-undercloud master: Adds execution of post-upgrade validations on undercloud upgrade  https://review.openstack.org/47457811:39
gfidentejprovazn not the issue you're seeing though11:41
jprovazngfidente: ack11:41
*** pgadiya has quit IRC11:41
*** jbadiapa_ has joined #tripleo11:42
*** shreshtha has quit IRC11:43
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates master: Set default OSD pool size to 1 in scenario 001/004 containers  https://review.openstack.org/49417611:43
*** rhallisey has joined #tripleo11:43
gfidentejprovazn so that one started now but it's weird, looks as if something is happening too early, like the problem you had with the key ownership11:43
gfidentewhat caused that?11:43
*** pkovar has joined #tripleo11:43
jprovazngfidente: you mean what caused http://logs.openstack.org/96/482496/7/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq/3753922/logs/subnode-2/var/log/glance/api.log.txt.gz#_2017-08-15_12_24_39_536 ?11:44
*** ansmith has quit IRC11:53
*** cylopez has joined #tripleo11:55
*** brault has joined #tripleo11:56
*** artom has joined #tripleo11:56
*** thrash|g0ne is now known as thrash11:58
*** lucas-hungry is now known as lucasagomes11:59
gfidentejprovazn yeah we didn't change when the pools are created, maybe the ceph logs have more details about that too12:00
EmilienMhello12:00
*** morazi has joined #tripleo12:00
shardy'morning EmilienM12:00
jprovazngfidente: I didn't find anything suspect in ceph logs so far, I'll see if I can reproduce with the same 001 env and w/o including manila service (it passes locally with it)12:01
*** trown|outtypewww is now known as trown12:04
*** pdeore has quit IRC12:05
*** dtrainor has joined #tripleo12:05
*** rlandy has joined #tripleo12:05
*** abishop has joined #tripleo12:06
*** jcoufal has joined #tripleo12:10
*** ooolpbot has joined #tripleo12:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION12:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295512:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm)12:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171053312:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171077312:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171093812:10
*** ooolpbot has quit IRC12:10
openstackLaunchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)12:10
openstackLaunchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,In progress] - Assigned to Giulio Fidente (gfidente)12:10
openstackLaunchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)12:10
gfidenteEmilienM I think we have a fix for the scenario jobs12:11
EmilienMyeah I saw, cool12:11
gfidenteregarding the upgrade CI instead, I think it currently does not test ceph12:11
EmilienMfinger crossed12:11
EmilienMthanks btw!12:11
gfidenteI would like to add that too instead12:11
gfidentewdyt?12:11
fultonjgfidente: i reproduced the problem, with glance getting the 503, set the new pool size and now I don't get the 503, so i think your fix is good.12:12
gfidentefultonj hey morning there12:12
EmilienMgfidente: any help in upgrades is very welcome so I think: yes please12:12
fultonjAlso, https://review.openstack.org/#/c/479288 Convert scenario001-multinode-containers job to ceph-ansible is now green12:13
fultonjhey gfidente12:13
gfidentefultonj right that was great12:13
openstackgerritJohn Trowbridge proposed openstack/tripleo-quickstart-extras master: Make env variables for multinode into ansible vars  https://review.openstack.org/48393012:13
gfidenteI am not sure what is happening in oooq with the repos12:13
gfidentetrown can probably help reviewing that12:13
EmilienMgfidente: see comment https://review.openstack.org/#/c/47928812:14
fultonjgfidente: the repos problem is explained in https://bugs.launchpad.net/tripleo/+bug/171100612:14
openstackLaunchpad bug 1711006 in tripleo "tripleo-quickstart creating redundant repositories" [High,In progress] - Assigned to John Fulton (jfulton-org)12:14
fultonji dropped in a proposed fix. i then made the ci proposal depend on it12:14
fultonjbasically, we had redundant repos conflicting which prevented ceph-ansible from getting installed12:14
*** dprince has joined #tripleo12:15
gfidentefultonj yeah I understand the issue I think, not sure if case is something we should fix in oooq12:15
fultonjmwhahaha suggested remaming them which seemed to solve the issue12:15
gfidentefultonj seems very messy to me12:15
gfidentebut I might not know the reasons why12:16
rcerninshardy: we found yesterday the issue why the heat was timing out, there was 3,7s clock offset12:16
fultonjgfidente: it was working and then a change was introduced, to support mirrors, which broke it12:16
EmilienMfultonj, gfidente : another option would be to remove them from the nodepool image12:16
EmilienMbut we already disable them, so it's clean as well12:16
gfidentefultonj have a link to change which added support for mirrors?12:17
fultonjgfidente: yes12:17
fultonjhttps://github.com/openstack/tripleo-quickstart/blob/master/config/release/tripleo-ci/consistent-master.yml#L38-L7912:17
EmilienMfultonj: your patch lgtm12:17
fultonjsec12:17
fultonjthanks12:17
EmilienMI checked, all repos use mirrors like before12:17
shardyrcernin: aha, and that broke the RPC?12:18
shardyrcernin: was the ntp service not enabled on the nodes?12:18
fultonjhttps://github.com/openstack/tripleo-quickstart/commit/84e0168e49fce2f3d79559866ebea8f7050eef5112:18
fultonjgfidente: that's the link you asked for ^12:18
rcerninshardy: it was not. there is a bug with chrony and ntpd currently https://bugzilla.redhat.com/show_bug.cgi?id=141375412:19
openstackbugzilla.redhat.com bug 1413754 in rhosp-director-images "Chrony and ntpd are present in overcloud-full images" [Medium,Closed: errata] - Assigned to emacchi12:19
shardyrcernin: ah, I see, well good to hear you tracked down the problem :)12:19
rcerninshardy: we did, but now trying to understand why that 3,7s actually matters so much.12:19
fultonjalex suggested quickstart, in the long run, use the yum module and not shell commands to manage repos so it's less messy12:19
shardyrcernin: do you have tls enabled?12:20
*** dpawar has quit IRC12:21
*** dpawar has joined #tripleo12:22
rcerninshardy: no12:22
*** sbrzozow has joined #tripleo12:23
gfidenteEmilienM fultonj yeah but I don't understand the attempt to reuse same names12:23
gfidentein lowercase12:23
gfidentethough now I understand better :D12:23
gfidentethanks :D12:23
numansEmilienM, can you please see if this fine with you - https://review.openstack.org/#/c/468860/. Once this scenario007 container job is merged in project-config - https://review.openstack.org/#/c/490622/, I will update the other corresponding patches.12:23
EmilienMnumans: sure, will look asap12:23
numansEmilienM, thanks.12:24
EmilienMnumans: are you sure you want the job voting?12:24
*** dpawar has quit IRC12:26
numansEmilienM, since scenario007 baremetal job is voting, I submitted the patch same way12:26
EmilienMgfidente: any comment on https://review.openstack.org/#/c/479288 ?12:26
EmilienMnumans: it was non voting first12:27
EmilienMnumans: and we made it voting afterward12:27
numansEmilienM, oh ok. i missed that part. i will respin another patch then. thanks12:27
*** bkopilov has joined #tripleo12:28
EmilienMnumans: we documented the bits btw: http://specs.openstack.org/openstack/tripleo-specs/specs/policy/adding-ci-jobs.html12:28
numansEmilienM, thanks.12:29
*** jpena|lunch is now known as jpena12:29
EmilienMsshnaidm: https://status-tripleoci.rhcloud.com/ is down12:30
EmilienMweshay: ^ did you guys changed something?12:31
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates master: Render IP map and host maps according to network_data.yaml  https://review.openstack.org/49398412:31
sshnaidmEmilienM, http://cistatus.tripleo.org/12:31
sshnaidmEmilienM, there were redirect, will check what happenes12:32
EmilienMsshnaidm: or just change the html12:32
EmilienMsshnaidm: wow, new layout?12:32
sshnaidmEmilienM, yeah, a few improvements12:33
*** skramaja has quit IRC12:33
abishopshardy: https://review.openstack.org/480587 has one +2 but needs another. could you take a look?12:33
shardysshnaidm: nice work on the cistatus :)12:34
shardyabishop: ack will do12:34
weshayEmilienM, http://cistatus.tripleo.org/12:34
shardyabishop: the blueprint link in the comment seems to be broken, can you pls link it, and does it have a FFE?12:35
sshnaidmshardy, EmilienM if you're interested, take a look: https://docs.google.com/document/d/15CwUUDi4Oca95IuY9s2gme03Z8COvJqmvp7uKdEKeU8/edit#12:35
shardyabishop: I know this work has been ongoing for a while, but I can't see it targetted to rc1?12:35
abishopshardy: hmm, I'll need to check on that12:36
shardyhttps://launchpad.net/tripleo/+milestone/pike-rc112:36
EmilienMsshnaidm: yeah, nice doc12:36
honzaapetrich: are you working on the nokogiri/ruby/libxml issue?  should i?12:37
openstackgerritJames Slagle proposed openstack/tripleo-heat-templates master: Separate config_volume for ringbuilder  https://review.openstack.org/49400812:38
*** psahoo has quit IRC12:38
apetrichhonza, sorry I got distracted with other reviews. If you have the cycles, yes. Also I don't have much idea on how to go about that12:39
honzaapetrich: I think I have even less :)12:39
lvdombrkrhello guys, as i understood by default 1 compute and 1 control node will be deployed, how can i deploy compute and controller node all in one?12:39
shardylvdombrkr: you can deploy with an environment file similar to this:12:40
shardyhttps://github.com/openstack/tripleo-heat-templates/blob/master/ci/environments/multinode.yaml#L17..L6212:40
shardyhttps://hardysteven.blogspot.co.uk/2016/08/tripleo-composable-services-101.html12:40
ccamachoshardy quick question have you seen this error lately ?   "/usr/bin/docker-current: Error while pulling image: Get https://index.docker.io/v1/repositories/tripleoupstream/centos-binary-iscsid/images: dial tcp: lookup index.docker.io on 192.168.24.5:53: server misbehaving.",12:40
shardylvdombrkr: that has some background, and shows how to disable the compute node12:40
shardylvdombrkr: you can then build up ControllerServices based on what services you want12:41
apetrichhonza, no worries I can give it a go. Do you know if that was working before?12:41
shardylvdombrkr: note there's no dependency mechanisim yet, so you need to know which services depend on which others12:41
shardyor figure that out while testing ;)12:41
honzaapetrich: This is the first time I have seen that error, here or anywhere.  AFAIK that job was passing earlier on that patch.12:41
apetrichhonza, oh that's true12:42
honzaapetrich: If the CI pipeline wasn't so backed up right now, I'd just give it a cheeky recheck...12:42
shardylvdombrkr: there is also https://docs.openstack.org/tripleo-docs/latest/install/advanced_deployment/composable_services.html but I notice that has a mistake, it should just say ControllerServices not CustomControllerServices, and you can specify an environment file instead of a custom roles_data.yaml12:43
shardyI'll push an update to fix that later12:43
shardyccamacho: No but if it's in CI there's been a bunch of issues related to caching the docker images I think12:44
ccamachoshardy mmm nope, thats from a fresh master install with containers :(12:44
*** psahoo has joined #tripleo12:44
shardyccamacho: ah, sorry I'm working on composable networking this week, so not tested latest w/containers12:45
shardyI'd probably try pushing everything to a local registry and using that instead12:45
ccamachoshardy thanks! Ill do that12:45
shardyccamacho: FWIW I don't use the undercloud registry, I have a local box configured like this:12:46
shardyhttp://paste.openstack.org/show/618523/12:46
shardyccamacho: that works as a pull-through cache so you don't actually need to push anything12:46
EmilienMshardy: just saw your comment on https://bugs.launchpad.net/tripleo/+bug/171093812:47
openstackLaunchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)12:47
EmilienMshardy: do you suggest that CI runs a different workflow that customers would run?12:47
ccamachoshardy ack thanks!12:47
shardyEmilienM: I think it is a workflow customers would run if they have a local registry of their own12:47
*** jlabarre has joined #tripleo12:47
shardyEmilienM: so I'm just saying not all container jobs necessarily have to use the undercloud registry12:47
shardyparticularly if it saves time and fixes the upgrades job?12:48
shardye.g do we waste time pulling to the undercloud, then again from undercloud to subnode?12:48
EmilienMshardy: yes I would support the idea12:48
openstackgerritJose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Heat]  https://review.openstack.org/49139912:48
openstackgerritJose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Aodh]  https://review.openstack.org/49141112:48
openstackgerritJose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Mistral]  https://review.openstack.org/49347112:48
openstackgerritJose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Gnocchi]  https://review.openstack.org/49143312:48
*** lucasagomes is now known as lucas-brb12:48
*** catintheroof has joined #tripleo12:49
shardyEmilienM: I'm not quite clear how the infra caching works, but regardless it seems like we're doing a double download which could be avoided for single-node CI tests12:49
EmilienMI agree12:49
*** jaganathan has quit IRC12:49
shardyEmilienM: did you manage to chat with stevebaker?  I wondered if that was what he had in mind12:50
shardythen you'd just do the prepare CLI and not the upload12:50
EmilienMshardy: what we had in mind is to download/upload containers that are used only and not everything12:50
EmilienMhe has a patch to start this work, a sec12:50
*** atoth has quit IRC12:50
EmilienMshardy: https://review.openstack.org/#/c/448328/12:50
shardyEmilienM: yeah, so I'm suggesting we just do the prepare step to generate the environment12:51
shardywhich will only contain the images we need, and point to the infra proxy12:51
EmilienMshardy: can we do that already now? I guess yes12:51
shardyEmilienM: yes, you just switch the IP/port to point somewhere else instead of the undercloud when doing the prepare12:52
shardyI already do that locally as does dprince I think12:52
*** jmelvin has joined #tripleo12:52
lvdombrkrshardy: what will happen if i have only one node in instackenv.json and i will run just default: openstack overcloud deploy --templates12:53
lvdombrkr?12:54
shardylvdombrkr: it will fail because by default we deploy two nodes12:54
*** tzumainn has joined #tripleo12:55
dprinceshardy: yes, it works great12:55
shardyEmilienM: I would guess we just change local_docker_registry_host to docker_registry_host here:12:55
shardyhttps://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/overcloud-prep-containers/templates/overcloud-prep-containers.sh.j2#L4312:55
EmilienMshardy: to the regional AFS mirror I guess?12:55
shardyEmilienM: I guess it'd be like this:12:56
shardyhttps://github.com/openstack/tripleo-quickstart/commit/8a78e84f173743a5f2042f86854c8b0302dcd4f512:56
shardysshnaidm: ^^ can we just point the overcloud nodes at undercloud_docker_registry_mirror, e.g is it a docker-registry instance?12:57
EmilienMshardy: we might need to change https://review.openstack.org/#/c/493972/4/roles/overcloud-upgrade/templates/major-upgrade-overcloud-containers.sh.j2 as well, not sure we need to do the upload then12:57
shardyEmilienM: sorry, yeah maybe to start with we only change it there12:57
shardyand see if we can get the upgrade job working12:57
jtomasekhonza: are you working on rebasing https://review.openstack.org/#/c/473933/ ?12:58
shardyor add a boolean to allow pull-from-proxy or not12:58
jtomasekhonza: I can do it if you want12:58
honzajtomasek: rebasing?  why?12:58
honzajtomasek: I was hoping it would merge as is.12:59
EmilienMshardy: ok I'll do that, thx12:59
jtomasekhonza: the actions error handling is broken there as the error handling patches merged. It is a simple update12:59
lvdombrkrshardy: to be shore that there is no discomunication,my goal is to in one node combaine controller and compute12:59
honzajtomasek: I see.  I'll do that right now.13:00
jtomasekhonza: ok, cool13:00
*** oidgar_ has joined #tripleo13:00
shardylvdombrkr: yup, if you look at the links I provided, you can see you need an environment file which modifies the default ControllerServices (to add the compute services to the Controller node)13:01
shardylvdombrkr: and ComputeCount: 0 to disable deploying any compute node13:01
shardylvdombrkr: you can customize the ControllerServices list to suit your requirements, including adding the compute bits to the one "controller" node13:01
*** trozet has joined #tripleo13:02
EmilienMshardy: we alraedy have containerized_overcloud_upgrade_pull_images boolean13:02
shardylvdombrkr: the multinode environment I linked does exactly what you describe every commit in our CI, but there's some additional config in there you probably don't need13:02
EmilienMI think we could just re-use that13:02
*** oidgar has quit IRC13:02
EmilienMshardy: do we need to execute these bits in your idea? https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/overcloud-upgrade/templates/major-upgrade-overcloud-containers.sh.j2#L16-L2913:03
trozetEmilienM: good morning.  With that odl scenario do you know if it is using a br-ex bridge?13:06
*** dsariel has quit IRC13:06
shardyEmilienM: I think we'd need only one prepare, which points at undercloud_docker_registry_mirror instead of local_docker_registry_host13:06
shardybe good to get sshnaidm to check that first tho13:06
*** liverpooler has joined #tripleo13:06
*** atoth has joined #tripleo13:07
sshnaidmshardy, sorry, can you elaborate?13:07
shardyEmilienM: this prepare workflow has been changing a bit lately so may need to check locally - I've not tested it for a few days13:07
shardysshnaidm: we're figuring out how to skip pulling images to the undercloud docker registry13:07
shardysshnaidm: and instead, generate an environment file with overcloud images prepare which points directly to the infra proxy13:08
shardye.g the subnode pulls directly from the undercloud docker proxy from NODEPOOL_DOCKER_REGISTRY_PROXY13:08
*** shreshtha has joined #tripleo13:08
sshnaidmshardy, I see13:08
shardysshnaidm: the idea is it saves some time, for a single node test we don't really need to proxy the images twice13:08
shardyand we need to find some time to fix the upgrade job13:09
*** brault has quit IRC13:09
sshnaidmshardy, so to skip pull-and-push step?13:09
shardysshnaidm: yup13:09
EmilienMyes13:09
*** brault has joined #tripleo13:09
EmilienMI was looking at setting containerized_overcloud_upgrade_pull_images to False but it won't work13:09
shardyEmilienM: are you set up to test this locally or would you like me to try and etherpad the steps?13:09
EmilienMwe need to write the DockerInsecureRegistryAddress anyway13:09
EmilienMshardy: I don't have the setup right now13:10
shardyEmilienM: Ok I'll try it now, sec13:10
*** ooolpbot has joined #tripleo13:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION13:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295513:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm)13:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171053313:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171077313:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171093813:10
*** ooolpbot has quit IRC13:10
openstackLaunchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)13:10
*** oidgar_ is now known as oidgar13:10
openstackLaunchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,In progress] - Assigned to Giulio Fidente (gfidente)13:10
openstackLaunchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)13:10
openstackgerritDmitry Tantsur proposed openstack/instack-undercloud master: Switch to scheduling based on resource classes  https://review.openstack.org/49085113:10
gfidenteEmilienM you might I think I know where the ceph logs are :D13:11
gfidenteEmilienM I think in the containers only right now :(13:11
sshnaidmshardy, I'll try to make this step conditional then, and using afs mirror instead of local_docker_registry_host ?13:11
EmilienMgfidente: we need to fix that13:11
gfidenteEmilienM yeah I see what you mean13:11
*** ansmith has joined #tripleo13:12
*** psahoo has quit IRC13:12
gfidenteEmilienM I can probably mount /var/log/ceph in the containers13:12
gfidenteand that should do it13:13
gfidenteor put it in /containers/ ?13:13
EmilienMgfidente: like we had before /var/log/ceph probably13:14
openstackgerritHonza Pokorny proposed openstack/tripleo-ui master: Download logs interface  https://review.openstack.org/47393313:15
honzajtomasek: ^^^ this new error handling is *awesome*13:15
lvdombrkrshardy: thank you now its getting clear. but there is no ready tampletes which i can use without any corrections ? cause there is greate chance to failed if creating tamplete wihout big bacground13:15
shardygfidente: the logs for most other containers not managed by pacemaker are under /var/log/containers/foo13:16
gfidenteshardy ack13:16
*** cdearborn_ has joined #tripleo13:16
EmilienMoh yes right13:16
shardylvdombrkr: you could cut/paste the multinode one as a starting point, sec let me do it for you13:16
abishopshardy: not sure what's going on, because blueprint is here: https://blueprints.launchpad.net/tripleo/+spec/tripleo-derive-parameters13:16
gfidenteEmilienM shardy fine thanks I'll update it quickly13:17
lvdombrkrshardy: i will be grateful13:17
EmilienMgfidente: thanks13:17
abishopshardy: it was marked implemented in m3, and we're just polishing and cleaning up TODOs13:17
EmilienMgfidente: next step, I'll enable pingtest and see how it works13:17
shardylvdombrkr: something like http://paste.openstack.org/show/618524/13:17
gfidenteEmilienM pingtest should work with both puppet-ceph and ceph-ansible13:17
gfidentenet the known issue with glance13:18
*** chem has quit IRC13:18
shardylvdombrkr: you can customize ControllerServices depending on your requirements, we don't have a fully documented all in one example though unfortunately13:18
openstackgerritAlex Schultz proposed openstack/tripleo-specs master: Propose tech-debt policy  https://review.openstack.org/49404413:19
EmilienMgfidente: I won't believe you until I see it in CI ;-)13:19
shardyabishop: Ok no problem, is this the final patch, or do we need a bug to track the remaining patches?13:19
pabelangermorning13:19
gfidenteEmilienM true and also, the logs change, should I put it into the scenario001 conversion submission13:19
gfidenteor better to split it into its own given scope?13:19
*** pradk has quit IRC13:19
EmilienMgfidente: wherever you like13:19
*** chem has joined #tripleo13:20
EmilienMgfidente: at this stage :D13:20
pabelangerEmilienM: mwhahaha: weshay: This is the next things we need to solve: http://logs.openstack.org/08/487408/9/gate/gate-tripleo-ci-centos-7-scenario004-multinode-oooq/7744844/logs/delorean_logs/b1/df/b1df493d43ffc59b67f9ea0e0bf4c0bca4279fe0_dev/rpmbuild.log.txt.gz13:20
gfidentebetter to use less submissions probably13:20
pabelangergate just reset because DLRN failed to download from buildlogs.centos.org13:20
abishopshardy: I think there are a couple more small items, for example https://review.openstack.org/48923913:20
pabelangeranybody know the status of having DLRM / mock use our AFS mirrors?13:20
lvdombrkrshardy: greate, thank you!!!13:20
abishopshardy: would it be better to file lp bugs for each item like this?13:21
shardyabishop: or we should perhaps reopen the blueprint if it's still in progress?13:21
EmilienMpabelanger: you better ask jpena or amoralej on this one, tbh13:21
EmilienMgfidente: yes I support that13:21
shardyabishop: if the remaining bits can be considered bugs in the implementation landed then sure, we can raise some bugs13:22
weshaypabelanger, ah.. to add the mirror variables to the rpm build?13:22
pabelangerweshay: ya, mock.cfg files need to be updated13:23
abishopshardy: I think lp bugs are better fit, so I'll make sure lp bugs are opened and resubmit the reviews13:23
shardyabishop: Ok thanks - sounds like we can't get these gating atm anyway, let me know when you've pushed and we can fast-approve when the gate is better13:23
shardyat least the bugs will help us track the patches for the rc releases13:23
abishopshardy: will do, thx13:23
weshayadarazs, fyi ^ see pabelanger comments here13:25
weshaypabelanger, that is probably a hot issue we need to fix right now?13:25
*** ramishra has quit IRC13:25
weshaypabelanger, adarazs I'll open a bug13:26
weshayugh13:26
*** dsariel has joined #tripleo13:26
pabelangerweshay: ya, sooner the fix, the less resets we get in the gate, and merge code faster13:27
weshayaye13:27
adarazs weshay: thanks, please create a bug for it. are we talking about messing with mock.cfg when quickstart gates build packages?13:27
*** ramishra has joined #tripleo13:28
*** janki has quit IRC13:28
jtomasekhonza: :)13:28
*** ecerquei_ has joined #tripleo13:28
sshnaidmwho can check what failed with zaqar subscription here? http://logs.openstack.org/50/480950/32/check/gate-tripleo-ci-centos-7-scenario003-multinode-oooq-puppet/f602d77/logs/undercloud/var/log/zaqar/zaqar-server.log.txt.gz#_2017-08-16_11_09_04_00413:28
*** janki has joined #tripleo13:28
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: scenario002/container: run Barbican non-containerized  https://review.openstack.org/49373413:28
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: scenario002/multinode: do not run containerized Zaqar  https://review.openstack.org/49400513:29
openstackgerritEmilien Macchi proposed openstack/tripleo-quickstart master: Enable pingtest on scenarios container jobs  https://review.openstack.org/49012913:29
EmilienMgfidente, fultonj ^ testing pingtest on container jobs with ceph NOT containerized yet13:29
openstackgerritDougal Matthews proposed openstack/tripleo-common master: Automatically retry introspection for failing nodes  https://review.openstack.org/46291613:30
EmilienMgfidente: i'll test containerized ceph (with pingtest) once we have logs13:30
gfidenteEmilienM non-containerized needs the glance fix13:30
*** oidgar_ has joined #tripleo13:31
EmilienMgfidente: which one?13:31
*** chlong_ has joined #tripleo13:31
gfidenteEmilienM https://review.openstack.org/49417613:31
EmilienMgfidente: correct, I added it in Depends-On :D13:31
EmilienMso I'll tell you how it works13:31
EmilienMwith pingtest13:31
gfidentegreat thanks13:32
gfidente++13:32
*** ansmith has quit IRC13:32
derekh EmilienM can you take another look at this if you have a chance https://review.openstack.org/#/c/485261/13:32
*** oidgar has quit IRC13:33
EmilienMderekh: where the creation of a server is tested?13:34
EmilienMderekh: this patch regress scenario004 in the sense it's not testing server creation anymore13:34
EmilienMhttp://logs.openstack.org/61/485261/5/check/gate-tripleo-ci-centos-7-scenario004-multinode-oooq/7dabdfc/logs/undercloud/home/jenkins/overcloud_validate.log.txt.gz#_2017-08-08_14_37_1213:34
EmilienMscenario004 is a ceph + rgw use case (used by glance) - so we don't know if that use case works anymore13:34
derekhEmilienM: there isn't one, yet,  I've  asked in my email if this was ok for now? and then I'd add one13:35
*** jlabarre has quit IRC13:35
EmilienMderekh: I wouldn't not do it *afterward* - it would be a regression in scenario004 testing in the meantime13:35
derekhEmilienM: ok so I need to add that first13:36
*** ykarel is now known as ykarel|afk13:36
*** lblanchard has joined #tripleo13:38
*** jlabarre has joined #tripleo13:38
*** lblanchard has quit IRC13:39
openstackgerritNuman Siddique proposed openstack/puppet-tripleo master: Add OVN DBs bundle support for pacemaker HA  https://review.openstack.org/48437313:39
*** ykarel|afk has quit IRC13:40
openstackgerritNuman Siddique proposed openstack/tripleo-heat-templates master: Support OVN DBs docker HA using pacemaker bundle  https://review.openstack.org/48437613:41
pabelangerweshay: adarazs bug 171067813:41
openstackbug 1710678 in tripleo "DLRN not configured to use regional mirrors" [High,Triaged] https://launchpad.net/bugs/1710678 - Assigned to Alfredo Moralejo (amoralej)13:41
amoralejpabelanger, EmilienM, implementing ^ is not trivial, i need some time for it, we have a couple of ideas to do it without breaking DLRN but i need to do some tests13:43
pabelangeramoralej: do you have your thoughts posted some place? Maybe we can have a few people itterate on this13:44
amoralejpabelanger, not yet, i will post in the LP13:44
dalvarezbeagles, ping :)13:46
*** gbarros has joined #tripleo13:46
beaglesdalvarez, pong!13:46
dalvarezops 1sc :)13:47
openstackgerritAdriano Petrich proposed openstack/instack-undercloud master: Add an hourly cron trigger for tripleo-ui logging  https://review.openstack.org/46960813:47
pabelangeramoralej: k, for now, we could symlink /etc/mock/default.cfg to a new openstack-infra spedific config and see what happens.13:49
*** lblanchard has joined #tripleo13:49
mwhahahapabelanger: EmilienM, looks like we didn't quite get all the mirror.centos.org references, http://logs.openstack.org/39/494039/2/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq-puppet/2adf7d9/console.html#_2017-08-15_23_48_44_50110913:50
pabelangermwhahaha: EmilienM: Yes ^ is because tripleo-ci is running yum commands before mirrors are properly setup13:51
EmilienMmwhahaha: probably in tripleo-ci repo13:51
pabelangerso, that needs to be fixed too13:51
*** lucas-brb is now known as lucasagomes13:53
*** dparkes has joined #tripleo13:55
openstackgerritEmilien Macchi proposed openstack/tripleo-quickstart-extras master: container/upgrades: unconditionally create containers-default-parameters.yaml  https://review.openstack.org/49421413:55
openstackgerritEmilien Macchi proposed openstack/tripleo-quickstart master: DNM - fs011: disable container pull/push  https://review.openstack.org/49421513:55
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: Upgrade CI test - never merge  https://review.openstack.org/46100013:57
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: Upgrade CI test - never merge  https://review.openstack.org/46100013:57
EmilienMsshnaidm: ^ trying something13:58
EmilienMpabelanger, mwhahaha : ok I'm looking into that now13:58
*** chlong_ has quit IRC13:59
*** ansmith has joined #tripleo14:00
*** udesale has joined #tripleo14:01
*** jprovazn has quit IRC14:01
*** agurenko has quit IRC14:03
*** masco has quit IRC14:03
jaosoriorbnemec: could you check this out https://review.openstack.org/#/c/493937/14:06
*** ebarrera has quit IRC14:07
*** ebarrera has joined #tripleo14:07
*** nyechiel has quit IRC14:09
*** oidgar_ is now known as oidgar14:09
*** dtantsur is now known as dtantsur|bbl14:10
*** dsariel has quit IRC14:10
*** ooolpbot has joined #tripleo14:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION14:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295514:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171053314:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm)14:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171077314:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171093814:10
*** ooolpbot has quit IRC14:10
openstackLaunchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)14:10
openstackLaunchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,In progress] - Assigned to Giulio Fidente (gfidente)14:10
openstackLaunchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)14:10
*** ykarel|afk has joined #tripleo14:11
*** ecerquei_ has quit IRC14:12
*** atoth has quit IRC14:12
*** ecerquei has joined #tripleo14:12
EmilienMsshnaidm: I'm trying to see where we should deploy qemu-img instead of here http://logs.openstack.org/39/494039/2/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq-puppet/2adf7d9/console.html#_2017-08-15_23_48_43_96640514:19
EmilienMsshnaidm: so we could take benefit from the regional mirrors14:19
EmilienMsshnaidm: it sounds like we need qemu-img in the multinode preparation, do you know a place in quickstart where we could do it *after* having the repos in place? or is it too early?14:20
*** janki has quit IRC14:22
*** atoth has joined #tripleo14:25
*** mdnadeem has quit IRC14:26
amoralejpabelanger, there is no mock.cfg, dlrn creates one on the fly14:26
amoralejthat's the main problem14:26
sshnaidmEmilienM, sorry, to do what?14:26
sshnaidmEmilienM, iirc it's needed to create a fake image, right?14:26
EmilienMsshnaidm: yeah, but we want to download qemu-img from the mirror not from centos.org14:27
EmilienMsshnaidm: trying to find the best place to put this code14:27
openstackgerritMerged openstack/python-tripleoclient master: Add --set argument to prepare command  https://review.openstack.org/48908314:27
sshnaidmEmilienM, I see, checking..14:27
pabelangerEmilienM: weshay: mwhahaha: I think we need to add debug: true to /etc/docker/daemon.json, wanted to see why it is possible to get 404 client error: not found14:29
weshaypabelanger, k.. not sure if that is possible via the undercloud variable, maybe one could hack that in.. or we could restore my patches for debug purposes14:30
weshayI can do the latter14:30
weshayif that would help14:30
*** jmelvin has quit IRC14:30
pabelangerweshay: ya, I think collecting debug logs from docker is the next step14:30
weshayk.. will need a few minutes.. mtgs mtgs mtgs14:31
weshaymtg-stack14:31
*** links has quit IRC14:32
*** jmelvin has joined #tripleo14:32
*** dsariel has joined #tripleo14:36
lvdombrkrshardy: i get error after openstack overcloud deploy --templates one.yaml14:37
lvdombrkrConfiguration has 1 errors, fix them before proceeding.14:37
lvdombrkrNot enough nodes - available: 1, requested: 214:37
openstackgerritHonza Pokorny proposed openstack/instack-undercloud master: Add PATCH to list of allowed methods for Ironic  https://review.openstack.org/49422814:37
openstackgerritAlex Schultz proposed openstack/tripleo-specs master: Propose tech-debt policy  https://review.openstack.org/49404414:41
*** pchavva has joined #tripleo14:42
sshnaidmEmilienM, I'm not sure in which stage we need this fake 1G image for multinode. If it's just before overcloud install, then I'll move this step to oooq roles and we're good. If it's required to bootstrap nodes, then we need to look how to hack it with tripleo.sh. The bootstrapping is done still with tripleo.sh --bootstrap-nodes :(14:42
sshnaidmslagle, ^^14:42
shardylvdombrkr: try penstack overcloud deploy --templates -e one.yaml14:43
shardyassuming one.yaml has the environment file with parameter_defaults in it14:43
shardyopenstack overcloud deploy ...14:43
gfidentetrown can you check https://review.openstack.org/#/c/494056/ ? has enough votes but I think you want to be aware of that?14:44
slaglesshnaidm: what are you asking?14:44
*** pradk has joined #tripleo14:44
sshnaidmslagle, when we actually need the fake image in multinode job?14:45
slaglesshnaidm: if you don't want it done with tripleo.sh, then change it14:45
*** gkadam has joined #tripleo14:45
sshnaidmslagle, when the fake image is started to being used14:45
slaglesshnaidm: i dont know if oooq uses that or not14:45
trowngfidente: thanks14:45
trowngfidente: looks good and importantly does not conflict with anything :P14:46
sshnaidmslagle, not related to oooq, I'm asking for multinode deployment generally14:46
gfidentetrown cool I wasn't sure about the naming choice14:46
lvdombrkrshardy: same error14:46
trowngfidente: ya seems fine to me14:46
gfidentefultonj thanks ^^14:47
fultonjcool14:47
pabelangerHmm 2 failures in mirror.sto2.citycloud.o.o for docker.io14:47
lvdombrkrlvdombrkr: i just copy/paste list of services what you give me to one.yaml14:47
*** agurenko has joined #tripleo14:48
sshnaidmEmilienM, either I can put it here: https://github.com/openstack-infra/tripleo-ci/blob/master/scripts/tripleo.sh#L1436 if repo-setup of tirpleo.sh uses mirrors14:48
EmilienMsshnaidm: yes we could do that14:48
pradkjaosorior, Hi, can i abandon this https://review.openstack.org/#/c/486709/14:49
pradkjaosorior, seems like things are working fine without this now?14:49
slaglesshnaidm: that's what i was trying to answer since multinode is generally done with oooq. i dont know if oooq needs it or not14:49
slaglesshnaidm: i believe there are still a few multinode jobs not using oooq. you could remove it and see if those pass14:50
sshnaidmslagle, ok, will try14:50
sshnaidmEmilienM, ^^14:50
EmilienMslagle: afik the only job that doesn't use oooq now is 3nodes but it's being fixed by trown14:51
EmilienMslagle: oh and the upgrade job on stable/ocata14:51
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: DONT REVIEW: test removing fake image from multinode  https://review.openstack.org/49423314:53
*** vkhanna1 has joined #tripleo14:53
*** jprovazn has joined #tripleo14:54
*** snecklifter has quit IRC14:55
EmilienMsshnaidm: can you test a patch in THT stable/ocata as well please? the upgrade job doesn't use quickstart14:56
openstackgerritSagi Shnaidman proposed openstack/tripleo-quickstart-extras master: DONT REVIEW: test without fake image  https://review.openstack.org/49423514:56
sshnaidmEmilienM, ok, after master14:56
*** vkhanna1 has quit IRC14:57
lvdombrkrshardy: any ideas?14:57
*** sri_ has joined #tripleo14:57
lvdombrkrshardy: i belive that im not first with that setup14:57
lvdombrkr)))14:57
*** snecklifter has joined #tripleo14:58
sshnaidmEmilienM, undercloud-containers doesn't use ooq too14:58
EmilienMindeed14:58
*** dtantsur|bbl is now known as dtantsur14:58
EmilienMpabelanger: I could add debug option with weshay after our mtg14:59
pabelangerthanks14:59
*** ykarel|afk is now known as ykarel14:59
openstackgerritDougal Matthews proposed openstack/tripleo-common master: Automatically retry introspection for failing nodes  https://review.openstack.org/46291615:01
adarazsamoralej, EmilienM: I see this is assigned to Alfredo: https://bugs.launchpad.net/tripleo/+bug/1710678 -- did you start  working on it?15:01
*** _milan_ has joined #tripleo15:01
openstackLaunchpad bug 1710678 in tripleo "DLRN not configured to use regional mirrors" [High,Triaged] - Assigned to Alfredo Moralejo (amoralej)15:01
d0ugalthrash: https://review.openstack.org/#/c/462916/27..28/workbooks/baremetal.yaml15:01
*** ecerquei has quit IRC15:01
d0ugalthrash: does that look good?15:01
*** ecerquei_ has joined #tripleo15:01
thrashd0ugal: the message can now just use $.introspection_attempt.. No need to add 115:02
thrashright?15:02
*** dsariel has quit IRC15:02
thrashor... Does it start at 0?15:03
d0ugalthrash: it starts at 0, I think that might be the real bug.15:03
d0ugalI forget why I done that, and my comment doesn't help :)15:03
mhenkelslagle: Is there a patch for https://bugzilla.redhat.com/show_bug.cgi?id=1412352 ?15:04
openstackbugzilla.redhat.com bug 1412352 in diskimage-builder "rhel-registration script broken with satellite 6.2.5+" [Medium,Assigned] - Assigned to jslagle15:04
thrashd0ugal: hehe... solve it as you like.. I think leaving it as '+ 1' is probably the easiest, if not the most readable15:04
*** oidgar has quit IRC15:04
*** milan has quit IRC15:04
thrashd0ugal: if you start it at 1, you'll need to make sure your inequalities are correct15:04
*** sshnaidm is now known as sshnaidm|afk15:05
slaglemhenkel: no15:05
mhenkelslagle: any plans?15:06
openstackgerritDougal Matthews proposed openstack/tripleo-common master: Automatically retry introspection for failing nodes  https://review.openstack.org/46291615:07
*** rcernin has quit IRC15:07
weshayamoralej, adarazs please sync up re: https://bugs.launchpad.net/tripleo/+bug/171067815:08
openstackLaunchpad bug 1710678 in tripleo "DLRN not configured to use regional mirrors" [High,Triaged] - Assigned to Alfredo Moralejo (amoralej)15:08
*** aufi has quit IRC15:08
adarazsweshay: I just asked him if he's working on it a minute ago15:08
adarazsno answer yet though15:08
slaglemhenkel: not specifically. it's not a wide spread issue apparently. so it must be something environmental, e.g. not a high priority at the moment15:08
lvdombrkrguys, anybode bofore deplot overcloud with compute and controller in one node?15:09
mhenkelslagle: as noted in the bug, the issue is introduced by satellite >6.1 (I tested on 6.2). So, all customers using that version or beyond will face the issue.15:10
*** ooolpbot has joined #tripleo15:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION15:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295515:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171053315:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm)15:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171077315:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171093815:10
*** ooolpbot has quit IRC15:10
openstackLaunchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)15:10
openstackLaunchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,In progress] - Assigned to Giulio Fidente (gfidente)15:10
openstackLaunchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)15:10
*** ebarrera has quit IRC15:10
*** cylopez has quit IRC15:12
openstackgerritVineet Paul proposed openstack/tripleo-heat-templates master: Drop extraconfig for nova-nuage  https://review.openstack.org/49423915:13
dalvarezbeagles, in order to create the ini file for the metadata agent, i copied the puppet-neutron/lib/puppet/provider/neutron_metadata_agent_config to puppet-neutron/lib/puppet/provider/ovn_metadata_agent_config15:14
dalvarezand also added this line to puppet-neutron/manifests/deps.pp: Anchor['neutron::config::begin'] -> Ovn_metadata_agent_config<||> ~> Anchor['neutron::config::end']15:14
dalvarezDoes it make sense?15:14
amoralejadarazs, i'll ping you after mtg15:14
adarazsamoralej: great. no idea how to modify the centos mirror it uses for now :)15:15
adarazsamoralej: I mean apart from replacing stuff in rdo.yml before the build.15:16
lvdombrkr guys, anybode bofore deplot overcloud with compute and controller in one node?15:16
openstackgerritMarios Andreou proposed openstack/tripleo-heat-templates master: Adds PostUpgradeConfigStepsDeployment to drive post config ansible  https://review.openstack.org/49387815:17
adarazslvdombrkr: I'm not sure it's possible for now with tripleo. though I remember people trying to get that working. and maybe there are some experimental versions somewhere.15:18
beaglesdalvarez, on call at the moment, I'll ping you when done15:18
dalvarezbeagles, sure no worries, thanks a lot!15:18
akrivokajtomasek: the new node wrapping workflow needs to include register, introspect and provide - correct?15:19
lvdombrkradarazs: my goal is olso just test env.15:19
openstackgerritJohn Fulton proposed openstack/tripleo-heat-templates master: Add support for installing Ceph MDS via ceph-ansible  https://review.openstack.org/47942615:20
akrivokajtomasek: what about the manage nodes operation? it is not referenced in the user docs (https://docs.openstack.org/tripleo-docs/latest/install/basic_deployment/basic_deployment_ui.html#manage-nodes) - when would this operation be used in the workflow? does it need to get included in the wrapping workflow?15:20
shardyadarazs: I think lvdombrkr has an undercloud and wants to deploy a single node overcloud, but perhaps I'm mistaken15:21
adarazsoh, okay :)15:21
adarazsI misunderstood.15:21
slaglemhenkel: if not using rhel registration during the actual image build, then i dont think katello-ca-consumer will get installed until the actual overcloud deployment15:22
slaglemhenkel: at that point, the hostname ought to be set by cloud-init15:22
slaglemhenkel: so, are we talking about the case where you're building rhel images with dib and registering the image build?15:22
lvdombrkrshardy: you catch my idea15:23
lvdombrkrshardy: so any soulutions?15:23
shardylvdombrkr: I already linked everything needed to do it, but I'll try to create some more step by step instructions later15:24
shardythe solution is to set the ComputeCount to 0 and specify the ControllerServices needed15:24
shardyas I said earlier15:24
*** gkadam is now known as gkadam-afk15:24
shardysorry if I didn't explain it clearly enough, probably we need some better docs15:24
*** udesale has quit IRC15:25
openstackgerritHonza Pokorny proposed openstack/tripleo-ui master: Download logs interface  https://review.openstack.org/47393315:25
lvdombrkrshardy: i excatlly do that, i creata file one.yaml and copy a list which you prepare for me15:26
lvdombrkrthats all as i understood?15:26
shardylvdombrkr: yes, I'll have to test it later if it didn't work for you15:27
*** shardy is now known as shardy_afk15:27
lvdombrkrshardy_afk: but you tested such a setup before?15:28
*** yprokule has quit IRC15:29
trozetmichapma: hi15:29
openstackgerritMerged openstack/tripleo-heat-templates master: Render VIPs dynamically based on network_data.yaml  https://review.openstack.org/48626015:30
*** gkadam-afk has quit IRC15:31
*** gkadam has joined #tripleo15:32
*** gbarros_ has joined #tripleo15:32
*** links has joined #tripleo15:32
*** gbarros has quit IRC15:33
akrivokajtomasek: seems like manage is used to put nodes back from available to manageable... so this operation does not need be a part of register -> introspect -> provide wrapping workflow, IIUC15:33
*** mdnadeem has joined #tripleo15:37
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates master: Add params needed for the ceph-ansible switch to containers playbook  https://review.openstack.org/49424615:41
*** brault has quit IRC15:41
*** yolanda has quit IRC15:44
*** yolanda has joined #tripleo15:44
*** ccamacho has quit IRC15:44
*** brault has joined #tripleo15:45
*** ccamacho has joined #tripleo15:46
*** yamahata has joined #tripleo15:47
michapmatrozet: hey15:47
trozetmichapma: hey are you adding those logging items to the scenario somehow?15:48
michapmatrozet: I thought I did, https://review.openstack.org/#/c/494043/15:48
michapmatrozet: but I don't think it worked.15:48
trozetmichapma: oh i didnt see this patch15:49
*** links has quit IRC15:49
michapmatrozet: I only put it up today15:49
*** jfrancoa has quit IRC15:50
trozetmichapma: we also need all those ovs outputs, is that also possible?15:50
*** gbarros_ has quit IRC15:50
michapmatrozet: there is another part of oooq that I think runs some diagnostics so we could maybe add it there15:51
trozetmichapma: ok cool15:51
*** marios has quit IRC15:51
openstackgerritDmitry Tantsur proposed openstack/instack-undercloud master: Switch to scheduling based on resource classes  https://review.openstack.org/49085115:54
openstackgerritDougal Matthews proposed openstack/python-tripleoclient master: Remove printing the introspection status  https://review.openstack.org/49424915:56
michapmatrozet: ah, yep it might belong in here: https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/collect-logs/tasks/collect.yml15:56
michapmatrown: would that be a suitable location for some ODL diagnostics? only run when ODL is active of course.15:56
*** pcaruana has quit IRC15:57
openstackgerritDougal Matthews proposed openstack/python-tripleoclient master: Allow users to specify the Introspection concurrency  https://review.openstack.org/45122815:59
trownmichapma: ya16:00
michapmatrown: I might make an odl-collect and include that when the package is there so it doesn't get cluttered16:00
trownmichapma: ya that makes sense16:01
michapmatrown: I can see us wanting a bunch of these for different plugins and whatnot16:01
trownmichapma: ya the collect-logs role is even used by some other teams without using quickstart16:01
*** gbarros has joined #tripleo16:02
openstackgerritDougal Matthews proposed openstack/python-tripleoclient master: Let users customise the node timeout for introspection  https://review.openstack.org/49425416:05
*** ykarel is now known as ykarel|afk16:07
*** bogdando has quit IRC16:09
*** ooolpbot has joined #tripleo16:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION16:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295516:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm)16:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171053316:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171077316:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171093816:10
*** ooolpbot has quit IRC16:10
openstackLaunchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)16:10
openstackLaunchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,In progress] - Assigned to Giulio Fidente (gfidente)16:10
openstackLaunchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)16:10
*** ramishra has quit IRC16:13
*** trown is now known as trown|lunch16:13
*** achadha has joined #tripleo16:14
*** anshul has quit IRC16:14
openstackgerritMichael Chapman proposed openstack/tripleo-quickstart-extras master: Add opendaylight to collect-logs  https://review.openstack.org/49404316:16
*** jlinkes has quit IRC16:18
michapmatrozet: I updated the patch, so the diags should be in an odl subdir in here for each run: http://logs.openstack.org/05/486905/21/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/df4e8e5/logs/subnode-2/var/log/extra/16:19
*** mcornea has quit IRC16:19
openstackgerritMichael Chapman proposed openstack/tripleo-heat-templates master: Add OPNFV scenario environment  https://review.openstack.org/48690516:21
*** jmelvin has quit IRC16:21
*** itlinux has quit IRC16:22
owalshlhinds: you still here?16:23
openstackgerritAttila Darazs proposed openstack/tripleo-quickstart-extras master: Use CENTOS_MIRROR for DLRN builds if available  https://review.openstack.org/49426216:25
*** stendulker has joined #tripleo16:26
honzaapetrich: Green across the board, well done :)16:28
*** nyechiel has joined #tripleo16:29
*** ccamacho has quit IRC16:32
openstackgerritOliver Walsh proposed openstack/tripleo-heat-templates stable/newton: SSHD Service extensions  https://review.openstack.org/49426316:32
*** florianf has quit IRC16:32
*** snecklifter has quit IRC16:35
*** yamahata has quit IRC16:36
openstackgerritOliver Walsh proposed openstack/tripleo-heat-templates stable/newton: SSHD Service extensions  https://review.openstack.org/49426316:40
*** iranzo has quit IRC16:43
*** _milan_ has quit IRC16:46
*** rcernin has joined #tripleo16:49
openstackgerritEmilien Macchi proposed openstack/tripleo-quickstart master: Enable pingtest on containers-multinode-upgrade  https://review.openstack.org/49426816:50
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: Upgrade CI test - never merge  https://review.openstack.org/46100016:50
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: Upgrade CI test - never merge  https://review.openstack.org/46100016:50
openstackgerritBen Nemec proposed openstack/tripleo-docs master: Update SSL docs to refer to the generated environments  https://review.openstack.org/49426916:50
EmilienMweshay: with pingtest now, let's see16:50
owalshEmilienM: Any idea why we use --validation-errors-nonfatal in the gate? https://review.openstack.org/492152 got through with the wrong heat template version16:51
weshayall eyes on zuul :)16:51
*** jpich has quit IRC16:51
bnemecowalsh: It's used because otherwise the Ironic node count validation will fail the deployment.16:52
weshayEmilienM, if I update https://review.openstack.org/#/c/494268/ will that affect the current test?16:53
bnemecowalsh: That wouldn't have been caught by the pre-deploy validations though, would it?16:53
*** egonzalez has quit IRC16:53
EmilienMweshay: what do you want to update?16:53
weshayEmilienM, the doc16:53
owalshbnemec: yea, did when I deployed newton on my local env16:54
EmilienMweshay: I'll have to make recheck16:54
*** lucasagomes is now known as lucas-afk16:54
weshaymeh.. I'll do it in another patch16:54
EmilienMweshay: ah, let me -1 it myself16:54
weshayEmilienM, oh jebus.. I'll just put a patch on top of it16:55
EmilienM:)16:55
weshayEmilienM, you're an idiot16:55
bnemecowalsh: Hmm.  Maybe we should add that to the pep8 check: https://github.com/openstack/tripleo-heat-templates/blob/master/tools/yaml-validate.py#L43116:55
EmilienMweshay: HR !16:55
weshayya.. go cry to HR.. THIS IS FREENODE16:56
EmilienMI'll call HR16:56
mwhahahastatus confirmed16:56
owalshbnemec: ahh... no... resource is set to None so it wouldn't have been included in the pre-deploy validation16:56
owalshbnemec: but my cherry-pick in https://review.openstack.org/494263 enables the service by default so I found it in my local env16:57
EmilienMowalsh: I have no idea but it's sad :)16:57
*** rhallisey has quit IRC16:57
EmilienMowalsh: probably because we want to ignore some problems we have to fix16:57
bnemecowalsh: Yeah, that makes sense.16:57
bnemecEmilienM: <bnemec> owalsh: It's used because otherwise the Ironic node count validation will fail the deployment.16:57
EmilienMgfidente, fultonj : I got pingtest working with ceph-ansible16:58
*** dprince has quit IRC16:58
fultonjEmilienM: that's great!16:58
EmilienMgfidente, fultonj: err, but ceph isn't containerized yet nevermind16:58
gfidenteso with puppet-ceph16:58
owalshbnemec: sounds like something for pep8 alright16:58
EmilienMgfidente, fultonj : yes16:58
EmilienMgfidente: https://review.openstack.org/#/c/494176/ helped16:58
gfidentestill good, when migrating to ceph-ansible it'll have to stay green16:58
fultonjyeah16:59
gfidenteright so I think we can merge pingtest regardless16:59
EmilienMscenario004 is still red though, let me show you logs16:59
*** jmelvin has joined #tripleo16:59
EmilienMhttp://logs.openstack.org/29/490129/8/check/gate-tripleo-ci-centos-7-scenario004-multinode-oooq-container/fb47733/logs/undercloud/home/jenkins/overcloud_validate.log.txt.gz#_2017-08-16_15_37_2116:59
EmilienMso everything is containerized except ceph ^16:59
*** strigazi_OFF is now known as strigazi17:00
*** rhallisey has joined #tripleo17:02
openstackgerritVineet Paul proposed openstack/tripleo-heat-templates master: Drop extraconfig for nova-nuage  https://review.openstack.org/49423917:02
*** rhallisey has joined #tripleo17:02
*** shardy_afk is now known as shardy17:03
openstackgerritwes hayutin proposed openstack/tripleo-quickstart master: update fs doc, pingtest for fs011  https://review.openstack.org/49427417:04
weshayEmilienM, ^17:04
adarazsamoralej, weshay: I had my first jab at https://bugs.launchpad.net/tripleo/+bug/1710678 -- but I think I didn't find the right place to override the CentOS mirrors. let me know if you know where to do it.17:05
openstackLaunchpad bug 1710678 in tripleo "DLRN not configured to use regional mirrors" [High,In progress] - Assigned to Attila Darazs (adarazs)17:05
weshayk17:06
adarazsI commented on the bug. and made a change that didn't seem to work on my local test.17:06
adarazsgotta log off for today.17:06
*** derekh has quit IRC17:06
amoralejadarazs, that will not help17:06
adarazsamoralej: I noticed. :)17:06
amoralejadarazs, let me explain how dlrn works in the LP17:06
adarazsamoralej: thanks, please do. I'll pick it up tomorrow17:07
adarazsor you can take over, if you have free cycles. :)17:07
*** gbarros has quit IRC17:07
*** salmankhan has quit IRC17:09
EmilienMweshay: thx17:09
*** ooolpbot has joined #tripleo17:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION17:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295517:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm)17:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171053317:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171077317:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171093817:10
*** ooolpbot has quit IRC17:10
openstackLaunchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)17:10
openstackLaunchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,In progress] - Assigned to Giulio Fidente (gfidente)17:10
openstackLaunchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)17:10
*** karthiks has quit IRC17:10
*** hewbrocca is now known as hewbrocca_afk17:10
gfidenteEmilienM fultonj the ceph logs can probably be capture from within the ceph containers17:11
gfidenteI see there are commands in tripleo-ci/scripts/get_docker_logs.sh17:11
gfidentewhere we might add what we need17:11
*** salmankhan has joined #tripleo17:11
gfidentebut I am going AFK for a while, will be back later, if you manage to get anything out add on review so I +2 :D17:12
fultonjok, thanks gfidente i will look at that17:12
gfidentecool, tty17:12
*** gfidente is now known as gfidente|afk17:12
*** ansmith has quit IRC17:13
*** yamahata has joined #tripleo17:15
*** salmankhan has quit IRC17:15
openstackgerritJames Slagle proposed openstack/tripleo-heat-templates master: Separate config_volume for ringbuilder  https://review.openstack.org/49400817:16
*** dtantsur is now known as dtantsur|afk17:19
jtomasekakrivoka: sorry, I am back now17:19
jtomasekakrivoka: correct, the wrapping workflow shoudl do register > introspect > provide17:19
jtomasekakrivoka: also it would be nice if we could optionally decide which workflows will be run (make introspect and provide optional) so we can let user choose via checkbox17:19
*** pkovar has quit IRC17:19
*** ykarel|afk has quit IRC17:21
amoralejadarazs, i just commented in https://bugs.launchpad.net/tripleo/+bug/1710678 let me know if it's clear enough17:21
openstackLaunchpad bug 1710678 in tripleo "DLRN not configured to use regional mirrors" [High,In progress] - Assigned to Attila Darazs (adarazs)17:21
*** dougbtv_ has joined #tripleo17:21
amoralejtricky part is generate custom.cfg17:21
*** ansmith has joined #tripleo17:22
*** gbarros has joined #tripleo17:23
*** shardy has quit IRC17:23
*** dougbtv has quit IRC17:24
openstackgerritBen Nemec proposed openstack/tripleo-docs master: Document relation between self-signed certs and parameters  https://review.openstack.org/49427817:25
*** gkadam has quit IRC17:25
openstackgerritEmilien Macchi proposed openstack/tripleo-quickstart master: Enable pingtest on 2 scenarios container jobs  https://review.openstack.org/49012917:26
*** dprince has joined #tripleo17:26
EmilienMweshay: ^ ok this patch should work (until we fix the rest), so we can have pingtest on 2 container jobs from now17:26
EmilienMweshay: little progress :)17:27
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: README: Fix CI coverage layout  https://review.openstack.org/49427917:28
weshayEmilienM++17:29
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: Create a whitelist for /etc configs  https://review.openstack.org/49397317:33
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: Exclude list for logs collection  https://review.openstack.org/49402217:35
*** dsneddon is now known as dsneddon_pto17:36
*** achadha has quit IRC17:36
*** rlandy is now known as rlandy|brb17:37
*** achadha has joined #tripleo17:38
*** itlinux has joined #tripleo17:39
*** agurenko has quit IRC17:40
*** jtomasek is now known as jtomasek|afk17:41
*** tosky has quit IRC17:41
weshayneed reviews https://review.openstack.org/#/c/487598/17:41
weshayhrm..17:42
weshay: [Errno 256] No more mirrors to try.", "2017-08-16 17:14:36 | http://mirror.ord.rax.openstack.org:8080/buildlogs.centos/centos/7/cloud/x86_64/openstack-pike/repodata/repomd.xml: [Errno -1] Error importing repomd.xml for delorean-pike-testing: Damaged repomd.xml file"], "warnings": []}17:43
*** achadha has quit IRC17:43
weshaypabelanger, ^17:43
weshayhttp://logs.openstack.org/00/461000/36/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq/b33e3ab/console.html#_2017-08-16_17_14_36_35008517:43
pabelangerweshay: we are going to land https://review.openstack.org/494265/17:44
pabelangeris this might be related17:44
pabelangerweshay: tl;dr repdata from buildlogs.cdn.centos.org is likely out of date17:44
weshayk17:45
weshayEmilienM, fyi.. ^ your pingtest jobs hit a bump, unrelated to your work17:48
EmilienMok17:48
EmilienMyou wanna make me cry right?17:48
*** stendulker has quit IRC17:50
*** ebarrera has joined #tripleo17:51
*** anshul has joined #tripleo17:51
*** trown|lunch is now known as trown17:52
*** milan has joined #tripleo17:53
*** achadha has joined #tripleo17:54
*** gbarros has quit IRC17:55
*** achadha has quit IRC17:55
*** strigazi is now known as strigazi_OFF17:55
*** achadha has joined #tripleo17:55
openstackgerritEmilien Macchi proposed openstack/tripleo-quickstart master: fs010: enable tempest  https://review.openstack.org/49428417:56
EmilienMweshay: early try of running tempest again on container-multinode17:56
EmilienMweshay: just to see what would block us17:57
weshaySlower, ^17:58
*** achadha_ has joined #tripleo17:58
weshaygetting tempest running against the container jobs upstream is something that is a priority atm17:58
*** achadha_ has quit IRC17:59
*** achadha has quit IRC17:59
Slowerweshay: ok cool.  I've worked with tempest before too18:01
weshaysshnaidm|afk, are you on?18:01
sshnaidm|afkweshay, yep18:01
weshaysshnaidm|afk, do you recall the issues we had w/ using ovb + net-iso + containers?18:01
*** rlandy|brb is now known as rlandy18:02
openstackgerritSagi Shnaidman proposed openstack/tripleo-quickstart-extras master: WIP: move creating fake image to oooq extras  https://review.openstack.org/49423518:02
*** achadha has joined #tripleo18:02
weshayI'm at a loss here, and I was going to try running the upstream ovb playbook w/ the ./devmode.sh path18:02
EmilienMarxcruz: hey, you still around?18:02
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: WIP: test creating fake image in oooq extras  https://review.openstack.org/49423318:02
sshnaidm|afkweshay, network isolation didn't work with containers18:03
weshaysshnaidm|afk, do you have time to chat about that18:03
EmilienMarxcruz: I think I found out why scenario001 doesn't run Telemetry autoscaling scenario test, it's beacuse "panko" isn't enabled in tempest.conf. Can we fix that?18:03
weshaysshnaidm|afk, you tried multinic right?18:03
sshnaidm|afkweshay, I think both singlenic, but singlenic anyway wouldn't work with upstream ci18:03
sshnaidm|afkweshay, so mostly multinic, yes18:03
weshaysshnaidm|afk, right... but single-nic should work w/ ovb in devmode18:04
weshaymaybe I'll go that route again18:04
weshaysshnaidm|afk, Slower ah.. so the upstream container jobs also have net-iso turned off18:05
weshayhttps://github.com/openstack/tripleo-quickstart/blob/master/config/general_config/featureset016.yml#L518:05
weshayrlandy, ^18:05
*** nyechiel has quit IRC18:05
weshayso the only confirmed network-isolation + containers is single-nic-vlan18:06
weshayor none18:06
weshayok.. I think that's our answer.. and something we need to figure out18:06
sshnaidm|afkweshay, maybe it's worth to try multinic again..18:06
weshayEmilienM, dprince, Slower are there any notes on using network isolation w/ containerized deployments?18:07
Slowerok18:07
rlandyweshay: I have one successful deploy with net iso18:07
weshayrlandy, on ovb?18:07
weshaywhich net-iso?18:07
Slowerweshay: not that I know of18:07
weshaythis may be a problem gentleman18:07
*** tosky has joined #tripleo18:07
weshaymen18:07
dprinceweshay: should be the same notes as BM actually18:07
weshayand ladies18:07
weshayok18:07
Slowerdprince: yeah18:08
dprinceweshay: we use --net=host which mostly covers us18:08
Slowerit shouldn't be any different really18:08
weshayrlandy, what worked for you.. libvirt + single-nic is known good.. we know that much18:08
dprinceweshay: anything in particular you are hitting issues w/18:08
weshaydprince, I'll put some information together.. and send it to you guys18:09
weshaydprince, Slower just in full disclosure afaict.. all the container jobs upstream have net-iso turned off.. we have one job in rdo that runs w/ single-nic18:09
*** pcaruana has joined #tripleo18:09
trozetmichapma: thanks18:10
weshayrlandy, ?18:10
*** ooolpbot has joined #tripleo18:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION18:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295518:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm)18:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171053318:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171067818:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171077318:10
rlandyweshay: this was net iso - no pacemaker - https://thirdparty.logs.rdoproject.org/jenkins-tq-gate-devmode-master-ovb-rdocloud-public-bond-168/undercloud/home/stack/. Successful deploy but validate failed18:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171093818:10
*** ooolpbot has quit IRC18:10
openstackLaunchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)18:10
Slowerhmm, so I rebooted my compute node and now I can't even ssh into it18:10
openstackLaunchpad bug 1710678 in tripleo "DLRN not configured to use regional mirrors" [Critical,In progress] - Assigned to Attila Darazs (adarazs)18:10
openstackLaunchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,In progress] - Assigned to Giulio Fidente (gfidente)18:10
openstackLaunchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)18:10
sshnaidm|afkweshay, https://bugs.launchpad.net/tripleo/+bug/168955618:10
openstackLaunchpad bug 1689556 in tripleo "CI: containers overcloud can't start vm when deployed with multinic network isolation" [High,In progress]18:10
weshayrlandy, ok.. that is great news18:11
weshayrlandy, looks like the overcloud had no dns18:11
weshay2017-08-15 23:12:30 | curl: (6) Could not resolve host: download.cirros-cloud.net; Name or service not known18:11
weshayrlandy, can you share that config so that Slower and I can poke at it18:11
rlandyweshay: my runs since then are failing before I get to deploy -18:11
weshaydoh18:11
rlandyon prep containerized deployment18:12
*** rhallisey has quit IRC18:13
Slowerbleh18:13
weshayrlandy, k.. point me at the config18:13
rlandythat job was my last shot at running through the devmode workflow18:14
rlandy$WORKSPACE/tripleo-quickstart/devmode.sh --ovb --no-gate --delete-all-stacks -CD18:14
weshayoh18:14
rlandysince jkilpatr said all he did was set containerized_overcloud to be true,  I tried that again18:15
rlandythe featuresets use deploy_composable_scenario: true18:16
jkilpatrwell it does work, just did it today, validations don't but the deploy runs all the way through18:16
*** nyechiel has joined #tripleo18:16
rlandyare we supposed to be running with deploy_composable_scenario?18:16
rlandyweshay: ^^?18:16
Slowerrlandy: this is with the default-to-containers patch right?18:16
rlandySlower: yes18:16
Slowerok18:17
*** rhallisey has joined #tripleo18:17
SlowerI ran it yesterday and got some strange error18:17
rlandySlower: I did two extra things ...18:17
Slowerno attribute 'rc' or something18:17
rlandy-e undercloud_flavor=m1.xlarge  -e enable_pacemaker=false18:18
SlowerI posted it in the patch actually18:18
Slowerrlandy: ok cool18:18
* rlandy checks patch18:18
Slowerrlandy: I just started it again 10 mins ago18:18
rlandySlower: the -CD avoids the message on containers18:20
rlandybut sshnaidm|afk also objected to that18:20
rlandywe wanted to warn people that they were getting a containerized overcloud18:21
*** mcornea has joined #tripleo18:24
weshayrlandy, this is in a patch? <rlandy> -e undercloud_flavor=m1.xlarge  -e enable_pacemaker=false18:24
rlandyweshay: no - just testing with it atm18:25
weshayrlandy, k.. good idea18:25
rlandyand you mentioned that pacemaker should make no diff18:25
rlandyweshay: we need to rethink the prompts for deploying with containers18:25
weshayya.. it certainly won't deploy containerized ha18:25
rlandyinitially we agreed to prompt the user to agree to a containerized deployment18:26
weshayyou.. I think we can make it default if release = pike or master18:26
rlandyboth Slower and sshnaidm|afk are objecting to that18:26
rlandyit is18:26
rlandybut we still stop and tell people18:26
weshayyes18:27
rlandyweshay: so - is there agreement now that we no longer stop and tell users18:27
openstackgerritEmilien Macchi proposed openstack/tripleo-quickstart master: Switch scenario003 to run Tempest  https://review.openstack.org/49429018:27
rlandyyou can avoid the questions with -CD or -ND options18:28
*** mcornea has quit IRC18:31
*** rhallisey has quit IRC18:34
openstackgerritTim Rozet proposed openstack/tripleo-heat-templates master: Disables QoS with OpenDaylight deployments  https://review.openstack.org/49202718:34
pabelangerweshay: okay, seeing Error importing repomd.xml for delorean-pike-testing: Damaged repomd.xml file in more then 1 place now. Mirrors should be updated shortly, but is the repo file actually valid?18:34
openstackgerritKeith Schincke proposed openstack/tripleo-heat-templates master: Add support for deploying RGW with ceph-ansible  https://review.openstack.org/48080418:34
*** salmankhan has joined #tripleo18:34
openstackgerritOliver Walsh proposed openstack/tripleo-heat-templates master: Stricter heat template version validation  https://review.openstack.org/49429218:37
openstackgerritEmilien Macchi proposed openstack/tripleo-quickstart master: Switch scenario007 to run Tempest  https://review.openstack.org/49429318:38
EmilienMweshay: so, scenario001, 002 and 004 have some blockers that you already know. 003 and 007 are under testing. That's the progress done on "running tempest instead of pingtest"18:39
pabelangeramoralej: is the repo file for delorean-pike-testing corrupt?18:41
pabelangeramoralej: or just badly out of sync because of 49426518:42
amoralejpabelanger, it's just out of sync18:43
amoralejonly jobs requiring last changes in the repo are failing18:43
*** gbarros has joined #tripleo18:43
arxcruzEmilienM: working on that, deploying a local machine to test18:44
openstackgerritEmilien Macchi proposed openstack/tripleo-quickstart master: Switch scenario002 to run Tempest  https://review.openstack.org/49110218:44
EmilienMarxcruz: ok thx18:44
pabelangeramoralej: okay, thanks. mirrors should be updating in 15-30mins18:44
amoralejnice, thx for your help18:45
rlandySlower: for Pike and later, the default is containerized overcloud and containerized undercloud?18:46
rlandyie: what is the expected undercloud status wrt containers18:46
*** amoralej is now known as amoralej|off18:47
*** nyechiel has quit IRC18:47
openstackgerritEmilien Macchi proposed openstack/tripleo-quickstart master: Switch scenario004 to run Tempest  https://review.openstack.org/49111318:47
openstackgerritVictoria Martinez de la Cruz proposed openstack/tripleo-heat-templates master: Containerize Manila Share for HA  https://review.openstack.org/48268018:48
openstackgerritTim Rozet proposed openstack/tripleo-heat-templates master: Add OPNFV scenario environment  https://review.openstack.org/48690518:49
openstackgerritBen Nemec proposed openstack/tripleo-docs master: Document relation between self-signed certs and parameters  https://review.openstack.org/49427818:52
openstackgerritBen Nemec proposed openstack/tripleo-docs master: Update SSL docs to refer to the generated environments  https://review.openstack.org/49426918:52
*** anshul has quit IRC18:55
*** _milan_ has joined #tripleo18:55
Slowerrlandy: there's no plan to containerize undercloud atm18:57
*** milan has quit IRC18:57
*** _milan_ has quit IRC18:57
Slowerrlandy: several of us devs would like to just to simplify things18:57
Slowerrlandy: I mean to really switch to it18:58
Slowerright now it's just a developmen tool18:58
rlandySlower: thanks - will make the options explicit to containerized_overcloud for now18:58
Slowerrlandy: perfect18:58
Slowerrlandy: mostly we just don't have the time this cycle to deal with containerizing undercloud18:59
Slowerso don't worry about testing it18:59
SlowerI think there may be one job that checks it or something, not sure18:59
*** _milan_ has joined #tripleo18:59
Slowerlarsks: was it you that had the super nifty paste cmd?18:59
*** mdnadeem has quit IRC19:00
larsksSlower: possibly! curl -T <anything> -s chunk.io19:00
larsksOr, for stdin, <anything> | curl -T- -s chunk.io19:00
Slowerlarsks: that's it, thanks!19:00
openstackgerritMerged openstack/tripleo-docs master: add how to check gate with tripleo to the contrib doc  https://review.openstack.org/48759819:01
*** gbarros has quit IRC19:07
*** ooolpbot has joined #tripleo19:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION19:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295519:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171067819:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171077319:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171093819:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm)19:10
*** ooolpbot has quit IRC19:10
openstackLaunchpad bug 1710678 in tripleo "DLRN not configured to use regional mirrors" [Critical,In progress] - Assigned to Attila Darazs (adarazs)19:10
openstackLaunchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,In progress] - Assigned to Giulio Fidente (gfidente)19:10
openstackLaunchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)19:10
weshayweshay19:10
weshayheh19:10
trowntrown19:10
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart master: Change devmode to deploy containerized services by default  https://review.openstack.org/47241219:11
EmilienMweshay: thoughts on https://review.openstack.org/#/c/494039/ ?19:12
pabelangerEmilienM: yes please19:15
EmilienMpabelanger: I already voted :D19:15
trownEmilienM: we know the ssh fix is in the latest package?19:18
trownEmilienM: aiui the original issue was intermittent, so a single CI run doesnt really show us whether it is still present19:18
mwhahahathe bz was closed out 2 weeks ago and current version upstream is 6.6.1p1-3519:19
mwhahahameaning we're installing an older version19:20
pabelangertrown: right, if these is still an issue however, we should work on it in openstack-infra, so all centos-7 jobs benefit.  I recomment revert and setup elastic-recheck query if we need to track it19:24
trownk, into the queue it goes19:25
trownwhen it comes out... nobody knows19:25
trownI will see myself out19:25
mwhahahayou don't get to leave19:25
trownhehe19:25
EmilienMlol19:25
*** pchavva has quit IRC19:26
*** trown is now known as trown|brb19:29
*** rhallisey has joined #tripleo19:30
*** salmankhan has quit IRC19:35
*** trown|brb is now known as trown19:39
trown11023019:39
weshaySlower, https://github.com/openstack/tripleo-quickstart-extras/tree/master/roles/build-test-packages19:41
pabelangerweshay: amoralej|off: mirrors have been updated to first hit buildlogs.centos.org now19:41
*** jprovazn has quit IRC19:43
openstackgerritMerged openstack/tripleo-heat-templates master: Add NeutronOverlayIPVersion parameter to neutron-plugins-ml2 service  https://review.openstack.org/46616219:46
*** rcernin has quit IRC19:55
*** homeski has joined #tripleo19:56
*** jpena is now known as jpena|off19:58
*** sbrzozow has quit IRC19:59
*** _milan_ has quit IRC20:01
*** milan has joined #tripleo20:01
*** homeski has quit IRC20:02
*** homeski has joined #tripleo20:03
*** ooolpbot has joined #tripleo20:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION20:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295520:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171067820:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171077320:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171093820:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm)20:10
*** ooolpbot has quit IRC20:10
openstackLaunchpad bug 1710678 in tripleo "DLRN not configured to use regional mirrors" [Critical,In progress] - Assigned to Attila Darazs (adarazs)20:10
openstackLaunchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,In progress] - Assigned to Giulio Fidente (gfidente)20:10
openstackLaunchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)20:10
*** trozet has quit IRC20:11
*** akrivoka has quit IRC20:11
*** liverpooler has quit IRC20:12
openstackgerritMerged openstack/tripleo-common master: Provide a list of unused user env parameters in plan  https://review.openstack.org/49175320:17
*** dprince has quit IRC20:18
*** homeski_ has joined #tripleo20:18
openstackgerritMerged openstack/tripleo-heat-templates stable/newton: Fix rpms being installed via DeployArtifactURLs  https://review.openstack.org/49386720:20
*** ansmith has quit IRC20:20
*** homeski has quit IRC20:20
*** gbarros has joined #tripleo20:21
openstackgerritJohn Fulton proposed openstack/tripleo-docs master: Update Pike storage documentation for ceph-ansible  https://review.openstack.org/48715520:23
*** akrivoka has joined #tripleo20:23
openstackgerritArx Cruz proposed openstack/tripleo-quickstart master: Configure the set of tests to execute in scenario001  https://review.openstack.org/49037620:25
*** trozet has joined #tripleo20:26
*** trown is now known as trown|outtypewww20:28
*** achadha has quit IRC20:29
openstackgerritEmilien Macchi proposed openstack/tripleo-quickstart master: Switch scenario007 to run Tempest  https://review.openstack.org/49429320:29
*** achadha has joined #tripleo20:30
*** achadha has quit IRC20:31
EmilienMpabelanger, mwhahaha: do you remember if we mirror cirros image somewhere? (I'm not joking)20:31
EmilienMhttp://logs.openstack.org/84/494284/1/check/gate-tripleo-ci-centos-7-containers-multinode/b45c603/logs/undercloud/home/jenkins/tempest_output.log.txt.gz#_2017-08-16_19_30_4320:31
mwhahahayea20:31
mwhahahait's local on the img20:31
*** achadha has joined #tripleo20:32
mwhahahain a folder20:32
mwhahahaEmilienM: https://review.openstack.org/#/c/475479/\20:32
EmilienMok thxw20:33
*** achadha_ has joined #tripleo20:34
openstackgerritEmilien Macchi proposed openstack/tripleo-quickstart master: fs010: enable tempest  https://review.openstack.org/49428420:35
EmilienMmwhahaha: mhh, I'm wondering about the failure though20:36
*** jcoufal_ has joined #tripleo20:36
EmilienMmwhahaha: sounds like the image is uploaded in config_tempest20:36
EmilienMnot in oooq20:36
mwhahahaEmilienM: right we probably need to be configuring the source in tempest20:37
mwhahahapreviously oooq did it for pingtest20:37
*** homeski has joined #tripleo20:37
*** achadha has quit IRC20:38
EmilienMmwhahaha: indeed, it's not http://logs.openstack.org/84/494284/1/check/gate-tripleo-ci-centos-7-containers-multinode/b45c603/logs/undercloud/home/jenkins/tempest/etc/tempest.conf.txt.gz20:38
*** achadha_ has quit IRC20:39
openstackgerritEmilien Macchi proposed openstack/tripleo-quickstart master: Enable pingtest on containers-multinode-upgrade  https://review.openstack.org/49426820:39
*** jcoufal has quit IRC20:39
*** homeski_ has quit IRC20:40
*** trozet has quit IRC20:40
*** pcaruana has quit IRC20:41
*** jlabarre has quit IRC20:47
weshayrlandy, I think moving to xlarge will make a difference in the container deployment20:47
*** jlabarre has joined #tripleo20:47
weshay16gb memory is the requirement20:48
rlandyweshay: ok - will add that to another review20:48
weshayrlandy, is your job still running20:48
rlandyweshay: it just failed deploy20:48
weshayugh20:48
* weshay takes a look20:48
rlandycontacting Slower to either debug or move on to script testing20:49
*** brault has quit IRC20:49
rlandySlower: ^^ got  a failed deployment this time20:49
rlandyhttps://thirdparty.logs.rdoproject.org/jenkins-tq-gate-devmode-master-ovb-rdocloud-public-bond-172/undercloud/home/stack/20:50
rlandywhich we can either debug or remove to work on the script20:50
weshayrlandy, Slower   "status_reason": "Deployment cancelled.",20:50
*** ecerquei_ has quit IRC20:50
rlandyout of time?20:51
rlandy2017-08-16 20:18:29 | 2017-08-16 20:17:56Z [overcloud]: CREATE_FAILED  Create timed out20:51
rlandycould be that step 3 stalled20:52
*** jcoufal_ has quit IRC20:52
rlandyweshay: can rerun overcloud deploy with a longer timeout20:52
*** dciabrin has quit IRC20:52
rlandyworth the try?20:53
weshaysure..20:53
*** trozet has joined #tripleo20:53
*** ebarrera has quit IRC20:53
rlandyok - trying deploy again20:54
*** atoth has quit IRC20:54
*** rhallisey has quit IRC20:54
radezslagle: I'm going to jump to #triple-o my question is not so ironic related I odn't think20:57
radezthe two messages were:20:57
radez(heat-config) [ERROR] Skipping group os-apply-config with no hook script None20:57
radezos-collect-config[2023]: No hosts in Heat, nothing written.20:57
radezdo either of those indicate bad things? my hosts are not running puppet when they come up and I'm trying to figure out why20:58
*** achadha has joined #tripleo20:59
*** _milan_ has joined #tripleo21:03
*** milan has quit IRC21:03
*** dciabrin has joined #tripleo21:05
Slowerrlandy: I've seen that too.. the cancelled thing21:06
rlandySlower: I am trying the deploy again with a longer timeout21:06
rlandydid you see the canceled deploy only on ovb for elsewhere?21:06
Sloweronly on ovb21:07
Slowermaybe a resource limit thing?21:07
rlandydefault timeout of 90 mins21:07
rlandybut it should not take that long21:08
SlowerI'm getting a failure in fatal: [undercloud]: FAILED! => {"changed": true, "cmd": "\"/home/stack\"/validate-overcloud-ipmi-connection.sh"21:08
*** shreshtha_ has joined #tripleo21:10
*** ooolpbot has joined #tripleo21:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION21:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295521:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm)21:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171067821:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171077321:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171093821:10
*** ooolpbot has quit IRC21:10
openstackLaunchpad bug 1710678 in tripleo "DLRN not configured to use regional mirrors" [Critical,In progress] - Assigned to Attila Darazs (adarazs)21:10
openstackLaunchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,In progress] - Assigned to Giulio Fidente (gfidente)21:10
openstackLaunchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)21:10
pabelangerweshay: where did we land on the daemon.json debug:true again? We need your patches?21:10
rlandySlower: I don't see the ipmi connection failure21:11
weshayah geez, ya.. let me do that now21:11
weshaybefore it slips my mind again, sorry21:11
pabelangersure, np.21:11
openstackgerritMerged openstack/tripleo-heat-templates master: Add ServiceData to hidden params  https://review.openstack.org/49324421:11
sshnaidm|afkpabelanger, just fyi http://paste.openstack.org/show/618576/21:12
*** lblanchard has quit IRC21:12
*** shreshtha has quit IRC21:12
sshnaidm|afkpabelanger, can it be fixed? happened in two jobs21:12
pabelangersshnaidm|afk: yes, that should be fixed https://review.openstack.org/494265/21:13
pabelangersshnaidm|afk: if it happens again, please ping me21:13
pabelangerissue is repodata on buildlogs.cdn.centos.org was stale21:13
sshnaidm|afkpabelanger, greate21:13
pabelangernow we get it directly from buildlogs.centos.org21:13
openstackgerritMerged openstack/tripleo-ui master: Don't log app state in Zaqar  https://review.openstack.org/49386321:17
openstackgerritwes hayutin proposed openstack/tripleo-quickstart-extras master: Use AFS mirrors to download containers instead of docker.io  https://review.openstack.org/49372821:17
weshaypabelanger, ok.. check my patches here.. https://review.openstack.org/#/q/topic:bug/171053321:18
weshayhttps://review.openstack.org/#/c/493728/7/roles/overcloud-prep-containers/templates/docker_daemon.json.j221:18
*** abishop has quit IRC21:19
pabelangerthanks I'll look shortly21:20
*** catintheroof has quit IRC21:21
*** itlinux has quit IRC21:21
*** eck`gone is now known as eck`21:26
openstackgerritMerged openstack/tripleo-ui master: Fix plan export URL  https://review.openstack.org/48816421:27
slagleradez: it has to do with you having some Heat deployments that are of type "group: os-apply-config", yet in your images, you do not have that hook script installed (it's deprecated)21:32
slagleradez: sounds like you may be mixing old templates with newer images21:32
*** jlabarre has quit IRC21:33
*** nyechiel has joined #tripleo21:35
*** cylopez has joined #tripleo21:36
*** cylopez has quit IRC21:36
*** achadha has quit IRC21:39
*** achadha has joined #tripleo21:43
*** achadha has quit IRC21:44
*** achadha has joined #tripleo21:45
*** thrash is now known as thrash|g0ne21:48
*** jmelvin has quit IRC21:49
*** gfidente|afk is now known as gfidente21:50
EmilienMgfidente: we are close :D21:54
gfidenteEmilienM hey yeah I think it failed before getting to the real thing21:55
gfidentewe'll see21:55
gfidentebut it's terrible21:55
gfidentethat we're late21:56
gfidenteI know21:56
gfidenteEmilienM actually, do you know if for upgrade patches we're using a special branch topic, always referring to a LP bug, creating as many LP bugs?21:57
EmilienMgfidente: not atfik21:59
EmilienMafik21:59
EmilienMgfidente: i'm having issue to test upgrades locally right now with scenarios but i keep trying22:00
EmilienMgfidente: have we done some tests on upgrading ocata to pike with ceph-ansible on pike already?22:00
gfidenteyeah was talking about that with mcornea earlier today about that22:01
gfidentewe did some22:01
gfidenteand we need patches22:01
gfidenteone of which I was working on right now22:01
EmilienMdo we know if upgrades to containers doesn't blow up the cloud?22:01
EmilienMhas someone tried that?22:01
gfidentethe playbook existed in ceph-ansible22:01
*** achadha_ has joined #tripleo22:01
gfidentefrom a while22:01
gfidentebut for me it didn't work yet22:01
gfidenteI suppose some versions of it worked22:02
gfidenteand it looks like there is CI in ceph for it22:02
gfidentehttps://github.com/ceph/ceph-ansible/issues/175822:02
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: Convert scenario001-multinode-containers job to ceph-ansible  https://review.openstack.org/47928822:04
weshayrlandy, Slower we may need to check if we have a proxy setup in rdo cloud re: containers22:04
EmilienMgfidente: I continue the tests ^ and try to see if pingtest pass22:05
*** achadha has quit IRC22:05
gfidenteEmilienM so the existing pingtest submission is testing puppet-ceph22:05
gfidenteI think if that passes and we merge it22:05
*** achadha_ has quit IRC22:05
gfidentethen the change above will also only pass if pingtest passes22:06
gfidentewhich sounds fine to me22:06
EmilienMgfidente: yeah I have another patch coming22:06
gfidentefor which change?22:07
*** cdearborn_ has quit IRC22:07
gfidenteI mean, to change what22:07
*** _milan_ has quit IRC22:07
*** _milan_ has joined #tripleo22:08
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: Convert scenario001-multinode-containers job to ceph-ansible  https://review.openstack.org/47928822:09
*** jkilpatr has quit IRC22:09
openstackgerritEmilien Macchi proposed openstack/tripleo-quickstart master: Enable pingtest on 2 scenarios container jobs  https://review.openstack.org/49012922:09
*** ooolpbot has joined #tripleo22:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION22:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295522:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm)22:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171067822:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171077322:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171093822:10
*** ooolpbot has quit IRC22:10
openstackLaunchpad bug 1710678 in tripleo "DLRN not configured to use regional mirrors" [Critical,In progress] - Assigned to Attila Darazs (adarazs)22:10
openstackLaunchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,In progress] - Assigned to Giulio Fidente (gfidente)22:10
openstackLaunchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)22:10
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: Convert scenario001-multinode-containers job to ceph-ansible  https://review.openstack.org/47928822:10
EmilienMgfidente: ok it's a bit a spaguetti but it should test pingtest now on ^22:10
EmilienMlet's see how it goes22:10
gfidenteI don't see differences, the rebase yes but not changes22:11
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: Upgrade CI test - never merge  https://review.openstack.org/46100022:12
EmilienMgfidente: no change, just a spaguetti on depends-on :D22:13
EmilienMgfidente: I didn't touch your code22:13
EmilienMgfidente: I'm just trying to test things now22:13
*** achadha has joined #tripleo22:13
gfidenteEmilienM whatever22:14
gfidentetrusted22:14
* gfidente looking at the upgrade thing :D22:14
EmilienM:)22:14
openstackgerritEmilien Macchi proposed openstack/tripleo-quickstart master: Switch scenario003 to run Tempest  https://review.openstack.org/49429022:17
*** homeski has quit IRC22:17
EmilienMgfidente: right now, this is my blocker for testing upgrades in CI22:20
EmilienMhttp://logs.openstack.org/00/461000/36/check/gate-tripleo-ci-centos-7-scenario004-multinode-oooq-container-upgrades-nv/defe4a7/logs/undercloud/home/jenkins/overcloud_deploy.log.txt.gz#_2017-08-16_18_20_1222:20
EmilienMit looks like it takes pike environment with THT on ocata22:20
EmilienMdocker/services/manila-api.yaml doesn't exist on ocata22:20
*** nyechiel has quit IRC22:21
EmilienMmhh wait, I found something weird though22:21
EmilienMhttps://github.com/openstack/tripleo-heat-templates/blob/stable/ocata/ci/environments/scenario004-multinode-containers.yaml#L1722:21
EmilienMwhy do we do this22:21
gfidentebecause the service is disabled by default22:22
EmilienMso22:22
EmilienMit's weird because I manually backported https://review.openstack.org/#/c/489874/ in the RPM22:22
EmilienMbut it sounds like it didn't work22:22
EmilienMok so I'm reverting the custom patch in the spec file: https://review.rdoproject.org/r/#/c/857522:24
EmilienMand i'll try to merge https://review.openstack.org/#/c/489874/22:24
EmilienMeven if we have to temporarly disable voting on the upgrade jobs just the time to have this one22:24
gfidenterebase?22:26
EmilienMgfidente: where?22:26
gfidente874b6fce-9383-4586-ae02-c1386907035222:26
gfidentesorry https://review.openstack.org/#/c/489874/722:26
EmilienMgfidente: no need to rebase22:27
gfidenteah the merge conflict is in delorean22:27
EmilienMit's HEAD already22:27
EmilienMyeah22:27
*** pradk has quit IRC22:27
EmilienMI'm going to follow that plan ^22:27
*** achadha has quit IRC22:33
*** achadha has joined #tripleo22:34
openstackgerritSteve Baker proposed openstack/tripleo-common master: overcloud_containers.yaml.j2 map images to services  https://review.openstack.org/44832822:35
*** achadha_ has joined #tripleo22:36
*** akrivoka has quit IRC22:37
pabelangerEmilienM: weshay: mwhahaha: pretty sure the job timeout issues we are now seeing on some tripleo jobs are because of DLRN http://logs.openstack.org/19/494219/1/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq-puppet/5230441/console.html#_2017-08-16_14_42_15_518071 alone is trying for 30mins to build-test-packages22:39
EmilienMouch22:39
*** achadha has quit IRC22:39
pabelangerIt would be helpfully to collect the DLRN logs there22:39
weshaygeez22:39
pabelangerand see what is going one22:39
pabelangeron*22:40
weshaypabelanger, EmilienM I know adarazs and amoralej|off were looking at it22:40
pabelangerhowever, I think, because this is infracloud, and DLRN still hitting upstream mirrors just don't have bandwidth to download everything22:40
pabelangerso, really need it using AFS mirrors22:40
*** achadha_ has quit IRC22:41
pabelangeraside from repodata issue today with buildlogs, every time I've seen a gate reset is because job timed out. And I am pretty sure, if we audit these failures, going to point to DLRN22:41
pabelangerweshay: EmilienM: telnet://15.184.66.45:19885 that job is stuck on DLRN22:42
pabelangersame task22:42
weshayNEWMAN22:42
pabelangerbuild-test-packages22:42
pabelangerabout to time out after 2.5 hours22:42
pabelangerwhich will reset gate22:42
pabelangerweshay: EmilienM: And if that didn't timeout, we'd likely have about 10 merges happing22:43
weshayjebus22:43
pabelangerhappening*22:44
pabelangerbecause everything behind that 1 job, has passed, or about to pass22:44
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates master: Swith to the appropriate ceph-ansible playbook on upgrade  https://review.openstack.org/49433622:45
pabelangerokay, I just put a hold on 10478149, want to see what it is doing22:45
weshayI wonder if delorean was using mirrors previously in ci22:46
weshayin tripleo.sh22:46
pabelangerHmm22:46
pabelangerweshay: in quickstart do you overwrite ~root/authorized_keys?22:47
pabelangercause I can't ssh into that node22:47
weshayno22:47
pabelangerk, I'll have to hit it from a zuul-launcher22:47
pabelanger1 sec22:47
weshaythat would be a shitty trick22:47
*** jkilpatr has joined #tripleo22:48
*** brault has joined #tripleo22:49
pabelangerya, something overwrote ~/root/.ssh/authorized_keys22:50
pabelangerI'll figure that out later22:50
weshayhere is the old build https://github.com/openstack-infra/tripleo-ci/blob/master/scripts/tripleo.sh#L436-L52722:51
* weshay compares it to the oooq22:51
pabelangerit hung doing a git clone22:52
weshay<sigh>22:53
* weshay updates the bug22:53
pabelangerlooks like maybe cloning from github.com?22:53
pabelangerI guess for package -dist?22:53
*** brault has quit IRC22:54
pabelanger{{ artg_rdoinfo_repo_url }} currently looking to see what that would be22:55
pabelangerhttps://github.com/redhat-openstack/rdoinfo ?22:55
pabelangerk, looks like a config file needed for DLRN22:56
*** achadha has joined #tripleo22:58
weshayupdated https://bugs.launchpad.net/tripleo/+bug/171067822:59
openstackLaunchpad bug 1710678 in tripleo "DLRN not configured to use regional mirrors" [Critical,In progress] - Assigned to Attila Darazs (adarazs)22:59
pabelangerweshay: so, if it is failing to clone github.com, that is a different issue. Also, we should not be cloning from github.com, its pretty unreliable23:01
weshaysure.. just going through it23:01
*** salmankhan has joined #tripleo23:01
*** dsariel has joined #tripleo23:02
pabelangerweshay: that info should maybe be on trunk.rdoproject.org, going to ask in #rdo now23:03
weshaypabelanger, git should be http://git.openstack.org23:03
weshay?23:03
weshayya23:03
weshaynvrmind23:04
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci master: Copy Ceph logs from running containers  https://review.openstack.org/49434023:05
*** achadha_ has joined #tripleo23:05
pabelangerweshay: it could be git.o.o but we'd have to mirror that repo into our infra23:06
pabelangerweshay: possible to do, but we'd need to talk about it first in #openstack-infra23:06
*** achadha has quit IRC23:08
*** achadha_ has quit IRC23:09
stevebakergfidente: hi, this has finally landed https://review.openstack.org/#/c/489083/23:09
*** ooolpbot has joined #tripleo23:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION23:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295523:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171067823:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171077323:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171093823:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm)23:10
*** ooolpbot has quit IRC23:10
openstackLaunchpad bug 1710678 in tripleo "DLRN not configured to use regional mirrors" [Critical,In progress] - Assigned to Attila Darazs (adarazs)23:10
*** _milan_ has quit IRC23:10
openstackLaunchpad bug 1710773 in tripleo "scenario001 and 004 fails when Glance with rbd backend is containerized but not Ceph" [Critical,In progress] - Assigned to Giulio Fidente (gfidente)23:10
openstackLaunchpad bug 1710938 in tripleo "Upgrades from Ocata to Pike (containerized) missing container upload step" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)23:10
*** _milan_ has joined #tripleo23:11
gfidentestevebaker yeah I tried to use that and the general set mechanism works fine23:12
*** achadha has joined #tripleo23:12
gfidentebut then I think panda pointed out in RDO they use the --prefix argument to set the registry address23:12
gfidenteand we get the ceph images prepended with that too23:12
gfidentewhen they actually want only the osp images to be prepended23:13
gfidenteplus the naming choice for the ceph images upstream vs downstream is so different that it was hard even to set the variables appropriately23:14
gfidentebasically upstream the version is all set in the tag name23:14
*** achadha_ has joined #tripleo23:16
stevebakergfidente: ok, could you raise a bug which documents all the upstream and downstream image name conventions so we can figure something out which works?23:16
*** achadha has quit IRC23:16
radezslagle: thx for the insight, I'll take a look at my packages and build and see if I've got something mixed up, I'm intending to be deploying Ocata23:17
*** achadha_ has quit IRC23:20
*** tosky has quit IRC23:33
*** achadha_ has joined #tripleo23:41
*** achadha_ has quit IRC23:46
*** salmankhan has quit IRC23:58

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!