Friday, 2018-06-15

openstackgerritRedHat RDO CI proposed openstack/tripleo-heat-templates stable/pike: GATE CHECK for TripleO  https://review.openstack.org/56428500:01
*** psahoo has joined #tripleo00:23
*** rlandy is now known as rlandy|afk00:29
openstackgerritJohn Fulton proposed openstack/tripleo-heat-templates master: WIP/DNM: Do not grant caps if metrics pool name is empty  https://review.openstack.org/57557100:30
*** mcornea has quit IRC00:32
*** pradk has joined #tripleo00:37
openstackgerritJohn Fulton proposed openstack/tripleo-heat-templates master: WIP/DNM: Do not grant caps if metrics pool name is empty  https://review.openstack.org/57557100:40
*** limao has joined #tripleo00:53
*** thrash|g0ne has quit IRC01:04
*** Petersingh has joined #tripleo01:10
*** thrash has joined #tripleo01:12
*** thrash has joined #tripleo01:12
*** lifeless_ has quit IRC01:13
*** lifeless has joined #tripleo01:13
*** lifeless_ has joined #tripleo01:24
openstackgerritLars Kellogg-Stedman proposed openstack/puppet-tripleo master: [WIP] enable support for openidc federation in keystone  https://review.openstack.org/57485101:24
openstackgerritLars Kellogg-Stedman proposed openstack/tripleo-heat-templates master: [WIP] enable support for openidc federation in keystone  https://review.openstack.org/57485501:24
*** lifeless has quit IRC01:25
*** psahoo has quit IRC01:28
openstackgerritLars Kellogg-Stedman proposed openstack/puppet-tripleo stable/pike: install python-neutron-lbaas before starting neutron  https://review.openstack.org/57550001:33
*** rlandy|afk is now known as rlandy01:36
*** lifeless has joined #tripleo01:40
*** zaneb has quit IRC01:40
*** Petersingh is now known as Petersingh|afk01:40
*** lifeless_ has quit IRC01:41
*** zaneb has joined #tripleo01:42
*** psahoo has joined #tripleo01:42
*** pmannidi has joined #tripleo01:46
*** Petersingh|afk is now known as Petersingh01:48
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: undercloud: manage mysql user + revoke grant during upgrade  https://review.openstack.org/57518601:48
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: undercloud: manage mysql user + revoke grant during upgrade  https://review.openstack.org/57518601:48
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: Check container health as part of the deploy  https://review.openstack.org/56915301:48
*** lifeless has quit IRC01:54
*** dxiri has joined #tripleo01:55
*** lifeless has joined #tripleo02:00
*** dxiri has quit IRC02:00
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates stable/queens: Delete not-used services-docker files  https://review.openstack.org/56781002:08
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates stable/queens: Move remnants of environments/services-docker  https://review.openstack.org/56781102:08
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates stable/queens: Switch remaining env files to containers defaults  https://review.openstack.org/56781302:08
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates stable/queens: Support containerized DVR in compute role  https://review.openstack.org/57319902:09
*** Petersingh is now known as Petersingh|afk02:12
*** ayoung has joined #tripleo02:20
*** dxiri has joined #tripleo02:20
*** dxiri has quit IRC02:25
openstackgerritSteve Baker proposed openstack/tripleo-heat-templates master: Allow a containerized mistral-executor to access docker  https://review.openstack.org/57490502:29
openstackgerritSteve Baker proposed openstack/puppet-tripleo master: Use user and group instead of ensure_resource  https://review.openstack.org/57558302:32
*** Petersingh|afk is now known as Petersingh02:33
weshay|ruckEmilienM, https://review.openstack.org/#/c/572830/02:33
*** brault has joined #tripleo02:34
EmilienMweshay|ruck: your patch needs more work02:34
EmilienMsee comments02:34
weshay|ruckk02:34
*** brault has quit IRC02:34
EmilienMweshay|ruck: you did like we run standalone on pike lol02:34
EmilienMweshay|ruck: this patch fixes standalone, please review02:37
EmilienMhttps://review.openstack.org/#/c/575564/02:37
EmilienMweshay|ruck, stevebaker and https://review.openstack.org/#/c/575563/ please02:37
EmilienM(see zuul, it's passing)02:37
stevebakerEmilienM: lgtm02:38
EmilienMweshay|ruck: my tempest patch does nothing, validate-tempest is not executed. Any clue?02:39
EmilienManything to hack in tripleo-ci repo?02:39
weshay|ruckEmilienM, I'll get tempest going w/ chandan and arx man02:40
weshay|ruckdon't worry about it02:40
openstackgerritwes hayutin proposed openstack/tripleo-quickstart master: use fs config to assign standalone role, tests  https://review.openstack.org/57283002:45
openstackgerritSteve Baker proposed openstack/tripleo-heat-templates master: Allow a containerized mistral-executor to access docker  https://review.openstack.org/57490502:49
EmilienMstevebaker: anything we need to review?02:49
*** agopi has quit IRC02:50
*** agopi has joined #tripleo02:50
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: skydive external_deploy_task migration  https://review.openstack.org/56127402:51
stevebakerEmilienM: no, I need to write a custom fact in ruby, no hurry on my series until that is working02:51
EmilienMok02:51
stevebakerEmilienM: see the TODO. Do you think this is best done as a custom fact? https://review.openstack.org/#/c/575583/1/manifests/profile/base/mistral/executor.pp02:52
EmilienMlooking02:52
stevebakerEmilienM: I need to stat /var/run/docker.sock02:52
EmilienMstevebaker: omg02:53
EmilienMthat sounds tricky02:53
EmilienMI mean doable but dangerous02:53
EmilienMbut I have nothing better in mind right now02:53
stevebakerEmilienM: me neither, other than some messy explicit discover gid on the host and pass it in as a parameter02:59
openstackgerritwes hayutin proposed openstack/tripleo-quickstart-extras master: WIP: add tempest to standalone  https://review.openstack.org/57558803:00
weshay|ruckEmilienM, ^03:00
EmilienMoh that03:00
weshay|ruckEmilienM, we need some rc file03:00
EmilienMyeah03:00
EmilienMI know03:00
EmilienMthis thing is done for undercloud03:01
EmilienMlet me see what it takes for standalone03:01
openstackgerritwes hayutin proposed openstack-infra/tripleo-ci master: Streamline variables passed in different environments  https://review.openstack.org/57381903:02
weshay|ruckhttps://review.openstack.org/#/c/573819/03:02
*** sai_p has quit IRC03:04
EmilienMweshay|ruck: I talked with safchain today (Skydive dev) and I suggested to create a CI job for it03:06
EmilienMthey need keystone + skydive03:06
EmilienMso another great use case for standalone in CI03:06
EmilienMI said he could talk to you03:06
EmilienMwhile I'm on PTO03:07
*** agopi has quit IRC03:07
*** dxiri has joined #tripleo03:08
weshay|ruckk03:09
EmilienMstevebaker: hey re: https://review.openstack.org/#/c/575552/03:10
EmilienMstevebaker: I just tested it03:10
EmilienMwith skydive03:10
EmilienMand it solved my problems, thanks again03:10
EmilienMso I didn't have to run any prepare command :D03:11
EmilienMjust "openstack tripleo deploy" and my patch ^03:11
EmilienMso discovery works fine, no bug03:11
*** dxiri has quit IRC03:13
*** ramishra has joined #tripleo03:13
stevebakerEmilienM: cool, tripleo deploy should just work03:14
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: skydive external_deploy_task migration  https://review.openstack.org/56127403:15
*** dxiri has joined #tripleo03:17
*** d0ugal has quit IRC03:17
*** d0ugal_ has joined #tripleo03:17
*** dsneddon has quit IRC03:23
*** dsneddon has joined #tripleo03:24
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: standalone: execute undercloud_post to have stackrc  https://review.openstack.org/57559703:25
EmilienMweshay|ruck: ^ that should do the job for now03:25
weshay|ruckrock03:25
EmilienMweshay|ruck: I suspect alex will want to rename the env or something so we don't have "undercloud"03:27
EmilienMbut the idea is here03:27
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: standalone: execute undercloud_post to have stackrc  https://review.openstack.org/57559703:27
openstackgerritwes hayutin proposed openstack/tripleo-quickstart-extras master: standalone: switch default NeutronPublicInterface to br-ex  https://review.openstack.org/57556403:27
weshay|ruckwhat03:27
*** rlandy has quit IRC03:27
EmilienMweshay|ruck: why did you do that?03:28
EmilienMplease remove the depends-on03:28
EmilienMon https://review.openstack.org/#/c/575564/03:28
EmilienMit's useless and the patch was in gate :(03:28
openstackgerritwes hayutin proposed openstack/tripleo-quickstart-extras master: standalone: switch default NeutronPublicInterface to br-ex  https://review.openstack.org/57556403:28
*** psachin has joined #tripleo03:28
openstackgerritwes hayutin proposed openstack/tripleo-quickstart master: use fs config to assign standalone role, tests  https://review.openstack.org/57283003:30
*** tzumainn has joined #tripleo03:31
EmilienMI've reset the gate to save time, a job was failing anyway, we saved 2h03:38
*** skramaja has joined #tripleo03:38
*** agopi has joined #tripleo03:42
*** Haresh has joined #tripleo03:45
*** limao has quit IRC03:48
*** Petersingh is now known as Petersingh|afk03:48
*** salmankhan has quit IRC03:49
*** yamahata has joined #tripleo03:56
*** limao has joined #tripleo03:57
*** ykarel_ has joined #tripleo04:08
*** haleyb has quit IRC04:08
*** haleyb has joined #tripleo04:09
*** shreshtha has joined #tripleo04:16
*** Petersingh|afk is now known as Petersingh04:17
*** jaganathan has joined #tripleo04:18
*** gouthamr has quit IRC04:22
*** stevebaker has quit IRC04:22
*** dmellado has quit IRC04:23
*** tzumainn has quit IRC04:43
*** moshele has joined #tripleo04:44
*** moshele has quit IRC04:47
openstackgerritIan Wienand proposed openstack/diskimage-builder master: Don't run setfiles on /boot/efi  https://review.openstack.org/57531504:54
*** links has joined #tripleo04:55
*** ratailor has joined #tripleo04:56
*** slaweq has joined #tripleo05:00
*** yprokule has joined #tripleo05:04
*** ratailor has quit IRC05:05
*** pdeore__ has joined #tripleo05:08
*** janki has joined #tripleo05:13
*** dxiri has quit IRC05:15
*** thrash has quit IRC05:16
*** pcaruana has quit IRC05:18
*** thrash has joined #tripleo05:20
*** thrash has joined #tripleo05:20
*** pmannidi_ has joined #tripleo05:20
*** leanderthal has quit IRC05:23
*** leanderthal has joined #tripleo05:25
*** saneax has joined #tripleo05:26
*** tcw has quit IRC05:27
*** tcw has joined #tripleo05:27
*** leanderthal has quit IRC05:30
*** leanderthal has joined #tripleo05:36
*** leanderthal has quit IRC05:41
*** leanderthal has joined #tripleo05:43
*** dparkes has quit IRC05:49
*** ykarel__ has joined #tripleo05:50
*** ykarel_ has quit IRC05:53
*** morazi has quit IRC05:57
*** dparkes has joined #tripleo06:01
*** pgadiya has joined #tripleo06:02
*** mschuppert has joined #tripleo06:04
bandinianyone else not reaching gerrit@rdo?06:05
*** mhenkel_ has joined #tripleo06:05
*** mhenkel_ has quit IRC06:05
*** dparkes has quit IRC06:06
*** mhenkel_ has joined #tripleo06:06
*** jfrancoa has joined #tripleo06:06
bandiniyeah it's unreachable from a bunch of networks06:07
*** dparkes has joined #tripleo06:07
ramishrabandini: I was told on #rdo that rdo-cloud is experiencing some network issue06:08
bandiniramishra: ah ok thanks. so folks are aware06:09
*** Petersingh is now known as Petersingh|afk06:09
*** paramite has quit IRC06:10
bandinianyone around to review this queens cherry-pick ? https://review.openstack.org/#/c/575525/06:10
*** dparkes has quit IRC06:13
*** dmsimard has quit IRC06:13
openstackgerritMerged openstack/tripleo-common master: Add dbus-python to overcloud-full image  https://review.openstack.org/57467406:14
*** cylopez has joined #tripleo06:21
*** cylopez has left #tripleo06:21
*** Petersingh|afk is now known as Petersingh06:24
*** agopi has quit IRC06:24
openstackgerritLee Yarwood proposed openstack/tripleo-heat-templates master: WIP nova: Use nova-status upgrade check during upgrade  https://review.openstack.org/57562306:26
*** dmsimard has joined #tripleo06:27
*** gkadam has joined #tripleo06:38
*** jfrancoa has quit IRC06:38
*** holser__ has joined #tripleo06:39
openstackgerrityolanda.robla proposed openstack/tripleo-heat-templates master: Adds support to configure enabled and default bios interfaces  https://review.openstack.org/57546206:41
*** kopecmartin has joined #tripleo06:42
openstackgerritMerged openstack/tripleo-heat-templates master: Uses new format for CephX keys caps  https://review.openstack.org/57466406:43
*** pcaruana has joined #tripleo06:44
*** pkovar has joined #tripleo06:45
*** ffiore has joined #tripleo06:47
*** ffiore has quit IRC06:49
*** ffiore has joined #tripleo06:50
*** slaweq has quit IRC06:50
*** cmurphy has joined #tripleo06:51
*** ykarel_ has joined #tripleo06:52
*** dparkes has joined #tripleo06:53
*** ykarel__ has quit IRC06:55
*** zul has quit IRC06:56
*** zul has joined #tripleo06:56
*** slaweq has joined #tripleo07:01
*** pmannidi has quit IRC07:03
*** dmellado has joined #tripleo07:04
*** brault has joined #tripleo07:07
*** ykarel_ is now known as ykarel07:07
*** rcernin has quit IRC07:08
*** ratailor has joined #tripleo07:10
*** ramishra has quit IRC07:13
*** tesseract has joined #tripleo07:17
*** ramishra has joined #tripleo07:20
*** bogdando has joined #tripleo07:20
bogdandojaosorior, mwhahaha, stevebaker: hi PTAL https://review.openstack.org/#/c/572404/807:20
*** tosky has joined #tripleo07:28
bandinibogdando: morning sir, clean cherry pick to queens of the logrotate thing if you have a min https://review.openstack.org/#/c/575525/ ;)07:29
*** chem` has joined #tripleo07:30
*** Petersingh is now known as Petersingh|lunch07:32
*** amoralej|off is now known as amoralej07:32
*** florianf has joined #tripleo07:33
*** chem has quit IRC07:33
bogdandobandini: done, thanks!07:33
bandinibogdando: merci!07:34
*** limao has quit IRC07:35
*** jfrancoa has joined #tripleo07:35
*** ksambor has quit IRC07:37
*** ksambor has joined #tripleo07:37
*** radeks has joined #tripleo07:38
openstackgerritMerged openstack/diskimage-builder master: Add iscsi-boot element  https://review.openstack.org/51149407:38
*** gfidente has joined #tripleo07:40
*** suuuper has joined #tripleo07:41
*** dmellado has quit IRC07:42
*** dmellado has joined #tripleo07:42
silehthttps://images.rdoproject.org/ looks dead07:47
dmelladoif it's hosted on rdocloud07:47
dmelladohouston we've an issue atm07:47
dmelladoxD07:47
*** jpena|off is now known as jpena07:47
*** ramishra has quit IRC07:49
*** ramishra has joined #tripleo07:51
*** ccamacho has joined #tripleo07:51
*** jbadiapa_ is now known as jbadiapa07:54
openstackgerritBogdan Dobrelya proposed openstack/tripleo-heat-templates master: Log memcached to stdout/err only  https://review.openstack.org/57567607:55
*** chem`` has joined #tripleo07:55
*** chem` has quit IRC07:57
*** verdurin has quit IRC07:59
openstackgerritBogdan Dobrelya proposed openstack/tripleo-heat-templates master: Log memcached to stdout/err only  https://review.openstack.org/57567608:00
*** verdurin has joined #tripleo08:00
*** tzumainn has joined #tripleo08:00
openstackgerritFlavio Percoco proposed openstack/tripleo-heat-templates master: Update for openshift 3.9  https://review.openstack.org/57423308:00
openstackgerritFlavio Percoco proposed openstack/tripleo-heat-templates master: Run dispatch script for all unmanaged devices  https://review.openstack.org/57567908:00
*** d0ugal_ has quit IRC08:04
*** d0ugal has joined #tripleo08:04
*** jpich has joined #tripleo08:06
openstackgerritJiri Stransky proposed openstack/tripleo-heat-templates stable/queens: Updates: run host_prep_tasks between update tasks and deployment tasks  https://review.openstack.org/57568008:07
*** radeks has quit IRC08:16
*** jfrancoa has quit IRC08:16
*** tzumainn has quit IRC08:18
openstackgerritArx Cruz proposed openstack/tripleo-quickstart-extras master: DNM - Testing swift integration  https://review.openstack.org/57568308:19
*** threestrands has quit IRC08:20
flaper87bogdando: do you know Sergii's irc nick?08:22
flaper87:D08:22
*** dparkes has quit IRC08:24
bogdandoflaper87, holser__:  ^^08:24
holser__here08:24
bogdandoa perfection hunter ;)08:24
holser__what's up flaper87 ?08:25
flaper87holser__: https://review.openstack.org/#/c/575679/ :)08:25
*** janki has quit IRC08:25
flaper87holser__: just saw your comment and replied but I thought I'd ping you on IRC too08:25
holser__yeah - it's nitpick08:25
*** dparkes has joined #tripleo08:25
holser__will +1 after comment08:25
flaper87holser__: no worries, just wanted to mention that it's prob not worth to -1 on nitpicks.08:25
flaper87that said, thanks a bunch for the suggestion08:25
flaper87I'm honestly not an awk expert and it's good to learn something new08:26
holser__well, the problem I did a lot of bash for very loaded machines08:26
holser__the problem was forking a new process08:26
bogdandoless pipes is goodness08:26
flaper87indeed08:26
bogdandobut indeed, does not worth -108:26
holser__so | | | | - 4 forks08:26
*** suuuper has quit IRC08:27
flaper87fully agree! just not a huge deal in this case ^^08:27
bogdandothat's why I said much better but not perfect haha08:27
holser__flaper87 done08:27
jistrweshay|ruck: re https://review.openstack.org/#/c/574417 -- according to what i see in logs (on the other patch i mentioned) the job is doing the upgrade properly08:28
*** suuuper has joined #tripleo08:29
flaper87holser__: you rock, thanks08:29
bogdandothere is a funny topic btw http://lists.openstack.org/pipermail/openstack-dev/2018-May/130802.html , so I opened the patch https://review.openstack.org/#/c/570940/ and the first thing I see there is https://review.openstack.org/#/c/570940/2/reference/principles.rst@55 :D08:30
bogdandoalthough, Eric didn't make it -1 :)08:31
bogdandosorry, it is Friday yet...08:32
*** Petersingh|lunch is now known as Petersingh08:34
*** derekh has joined #tripleo08:36
openstackgerritYurii Prokulevych proposed openstack/tripleo-upgrade master: Create L3 start/stop scripts for minor updates.  https://review.openstack.org/56576008:37
holser__chem`` bogdando08:39
*** chem`` is now known as chem08:39
chembogdando: hey, I've commented on https://review.openstack.org/#/c/575676/2, I think it's worth a new review, WDYT ?08:39
openstackgerritMerged openstack/diskimage-builder master: Don't run setfiles on /boot/efi  https://review.openstack.org/57531508:42
bogdandochem: sounds fair, though could we make it a follow up?08:43
bogdandochem: would you go for it? :)08:44
chembogdando: well, would make it more work to get it in the product ... but oki, I'll do a followup08:44
bogdandothanks!08:44
* chem doing it now08:44
*** ykarel is now known as ykarel|lunch08:47
*** akrivoka has joined #tripleo08:48
*** panda|off is now known as panda08:54
*** lifeless has quit IRC08:57
*** social has joined #tripleo08:57
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-heat-templates master: Activate memcached debug only when using MemcachedDebug param.  https://review.openstack.org/57569308:59
chembogdando: holser__ mwhahaha ^09:00
holser__+109:00
*** gouthamr has joined #tripleo09:02
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-heat-templates master: Activate memcached debug only when using MemcachedDebug param.  https://review.openstack.org/57569309:04
*** ratailor_ has joined #tripleo09:06
holser__chem +109:08
holser__I thought about upper/lower case of true09:08
holser__but it's not a big deal, but imho heat should support regexp :D09:10
*** ratailor has quit IRC09:10
*** lifeless has joined #tripleo09:10
openstackgerritBogdan Dobrelya proposed openstack/python-tripleoclient master: Add undercloud admin VIP to insec registries  https://review.openstack.org/57569709:10
*** jtomasek_ has joined #tripleo09:12
*** jtomasek has quit IRC09:12
*** quiquell|off is now known as quiquell09:14
*** janki has joined #tripleo09:16
*** quiquell is now known as quiquell|rover09:18
openstackgerritBogdan Dobrelya proposed openstack/python-tripleoclient master: Add undercloud admin VIP to insec registries  https://review.openstack.org/57569709:19
openstackgerritSagi Shnaidman proposed openstack/tripleo-common stable/queens: DNM: test queens patch  https://review.openstack.org/57524409:22
*** ykarel|lunch is now known as ykarel09:25
*** stevebaker has joined #tripleo09:26
openstackgerritHarald Jensås proposed openstack/tripleo-heat-templates master: Get ctlplane interface config from the subnet  https://review.openstack.org/56954709:29
openstackgerritMartin André proposed openstack/tripleo-common master: Fix ansible tmp dir to mistral working directory  https://review.openstack.org/57570309:30
quiquell|roverEmilienM: Queues are high again, maybe the containerized activation is affecting it09:33
*** zoli is now known as zoli|lunch09:35
openstackgerritNguyen Van Trung proposed openstack/diskimage-builder master: Add iscsi-boot element for CentOS images  https://review.openstack.org/54270809:37
*** psachin has quit IRC09:38
*** lifeless has quit IRC09:38
openstackgerritNguyen Van Trung proposed openstack/diskimage-builder master: Add iscsi-boot element for CentOS images  https://review.openstack.org/54270809:40
*** lifeless has joined #tripleo09:45
bogdandostevebaker, EmilienM: could you folks please comment https://trello.com/c/osszhcjk/79-autogen-container-image-params#comment-5b1e83e0a6d2883b78a0fc98 ?09:46
*** leanderthal has quit IRC09:48
*** Petersingh has quit IRC09:52
*** Petersingh has joined #tripleo09:52
*** Petersingh has quit IRC09:53
*** Petersingh has joined #tripleo09:54
*** Petersingh is now known as Petersingh|afk10:00
openstackgerritDan Sneddon proposed openstack/os-net-config master: [WIP] Add Policy Based Routing Support to os-net-config  https://review.openstack.org/57571210:01
quiquell|roverAny cores for a +2 +1w here ?10:01
quiquell|roverhttps://review.openstack.org/#/c/575407/10:01
quiquell|roverAffects collecting logs10:01
*** psahoo has quit IRC10:03
*** Petersingh|afk is now known as Petersingh10:03
*** Petersingh is now known as Petersingh|afk10:05
*** dtantsur|afk is now known as dtantsur10:06
*** rcernin has joined #tripleo10:07
openstackgerritDan Sneddon proposed openstack/os-net-config master: [WIP] Add Policy Based Routing Support to os-net-config  https://review.openstack.org/57571210:08
*** leanderthal has joined #tripleo10:09
*** Petersingh|afk is now known as Petersingh10:10
dtantsurmorning folks, can someone please W+1 https://review.openstack.org/#/c/541565/ ?10:10
*** paramite has joined #tripleo10:12
*** Petersingh has quit IRC10:13
openstackgerritMerged openstack/tripleo-heat-templates master: Disable memcached's cachedump  https://review.openstack.org/53898710:13
mschuppertfultonj: hi, could you +2 https://review.openstack.org/#/c/575380/ ci passed10:20
*** lifeless_ has joined #tripleo10:22
openstackgerritMerged openstack/tripleo-heat-templates master: Add environments/net-noop.yaml  https://review.openstack.org/55007210:23
openstackgerritMerged openstack/tripleo-heat-templates stable/queens: Ensure WorkflowSteps resource is enabled in ceph-upgrade-prepare  https://review.openstack.org/57205310:23
openstackgerritMerged openstack/tripleo-common master: Write to /etc/ssh with super user privileges  https://review.openstack.org/57472110:23
*** lifeless has quit IRC10:23
*** lifeless_ has quit IRC10:39
*** lifeless has joined #tripleo10:41
*** dtantsur is now known as dtantsur|afk10:45
*** pdeore has joined #tripleo10:49
*** zoli|lunch is now known as zoli10:50
*** brault has quit IRC10:50
*** gfidente has quit IRC10:50
*** pdeore__ has quit IRC10:51
*** lifeless_ has joined #tripleo10:55
*** lifeless has quit IRC10:56
*** pdeore has quit IRC10:58
*** raildo has joined #tripleo10:59
*** lifeless has joined #tripleo11:00
*** lifeless_ has quit IRC11:01
*** pdeore has joined #tripleo11:02
*** yolanda has quit IRC11:03
*** lifeless has quit IRC11:05
*** yolanda has joined #tripleo11:06
*** gfidente has joined #tripleo11:08
*** lifeless has joined #tripleo11:10
*** jpena is now known as jpena|lunch11:12
*** lifeless_ has joined #tripleo11:15
*** lifeless has quit IRC11:15
fultonjmschuppert: +2'd :)11:17
*** shreshtha has quit IRC11:25
*** brault has joined #tripleo11:30
openstackgerritMerged openstack/tripleo-heat-templates stable/queens: Fix update/upgrade or config change for ODL  https://review.openstack.org/57403911:31
mschuppertfultonj thx!11:34
fultonjmschuppert: np11:35
openstackgerritMerged openstack/tripleo-quickstart-extras master: Disable update of OpenShift and k8s containers  https://review.openstack.org/57363011:36
openstackgerritMerged openstack/tripleo-heat-templates master: Map OpenShift resources to services  https://review.openstack.org/57322511:36
openstackgerritMerged openstack/tripleo-heat-templates master: Do not fail on non existent ansible group  https://review.openstack.org/57418711:36
openstackgerritMerged openstack/tripleo-quickstart master: Define a 'baremetal' file for nodes config  https://review.openstack.org/55346511:36
*** rodolof has joined #tripleo11:38
*** anande has joined #tripleo11:38
*** jpena|lunch is now known as jpena|off11:38
*** rh-jelabarre has joined #tripleo11:41
*** rh-jelabarre has quit IRC11:43
*** rh-jelabarre has joined #tripleo11:44
*** cylopez has joined #tripleo11:46
*** rh-jelabarre has quit IRC11:46
*** rh-jelabarre has joined #tripleo11:46
*** rh-jelabarre has quit IRC11:47
*** rh-jelabarre has joined #tripleo11:48
*** rh-jelabarre has quit IRC11:48
*** vpickard_ is now known as vpickard11:49
*** pdeore has quit IRC11:52
*** vpickard is now known as vpickard_11:53
*** morazi has joined #tripleo11:54
weshay|ruckEmilienM, https://review.openstack.org/#/c/571613/ :(11:58
weshay|ruckbogdando, stevebaker ^11:59
weshay|rucksecond time it died in the gate like that11:59
bogdandoweshay|ruck: the same issue with overloading registry mirrors?12:00
* weshay|ruck looking12:00
bogdandoweshay|ruck, EmilienM, stevebaker: I have a patch in work to improve exp backoff logic for pull and push12:00
bogdandowe keep those too short in busy12:00
bogdandoI'll push it for review shortly12:00
bogdandoshort and busy*12:01
weshay|ruckya.. same thing http://logs.openstack.org/13/571613/18/gate/tripleo-ci-centos-7-scenario001-multinode-oooq-container/298ee13/logs/undercloud/home/zuul/overcloud_prep_containers.log.txt.gz12:01
*** edmondsw has joined #tripleo12:01
openstackgerritMerged openstack/tripleo-quickstart-extras master: Add upgrade and update logs to logstash indexing  https://review.openstack.org/57089612:03
openstackgerritMerged openstack/tripleo-quickstart-extras master: Run openstack overcloud failures when failures  https://review.openstack.org/57189712:03
openstackgerritMerged openstack/tripleo-heat-templates master: Upgrades: Refactor package removal to step3  https://review.openstack.org/57143112:03
openstackgerritMerged openstack/tripleo-quickstart master: Revert "Revert "fs050: enable upgrade cleanup""  https://review.openstack.org/57150012:03
openstackgerritMerged openstack/tripleo-heat-templates master: undercloud: add missing services for upgrade cleanup  https://review.openstack.org/57028312:03
openstackgerritMerged openstack/tripleo-heat-templates master: Rename Dell EMC XTREMIO iSCSI environment file  https://review.openstack.org/57023012:03
openstackgerritMerged openstack/puppet-tripleo stable/pike: Conditionally Enable Nuage VRS  https://review.openstack.org/57263112:03
openstackgerritMerged openstack/python-tripleoclient master: Add XClarity in the list of supported hw types  https://review.openstack.org/54157012:03
openstackgerritMerged openstack/tripleo-heat-templates master: Set Ceph pools rule_name to replicated_rule  https://review.openstack.org/57434112:03
openstackgerritMerged openstack/puppet-tripleo master: Make sure remotes are fully up before proceeding  https://review.openstack.org/56957812:03
weshay|ruckquiquell|rover, ^ good sign :)12:03
weshay|rucklots o merge12:04
*** jpena|off is now known as jpena12:05
*** moshele has joined #tripleo12:06
*** trown|outtypewww is now known as trown12:09
*** sanjay__u has quit IRC12:13
*** artom has quit IRC12:14
*** skramaja has quit IRC12:17
*** cdearborn has joined #tripleo12:18
*** rlandy has joined #tripleo12:19
*** rh-jelabarre has joined #tripleo12:20
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates stable/queens: Set Ceph pools rule_name to replicated_rule  https://review.openstack.org/57572112:21
*** moshele has quit IRC12:23
*** ratailor_ has quit IRC12:23
beaglesbandini: does this ring any bells with you http://paste.openstack.org/show/723516/ ?12:25
beaglesbandini: context - I'm running 'openstack overcloud update run --nodes Controller' using master12:26
beaglesbandini: I think this is the last controller node being updated and it is bombing there (heartbreak!)12:26
*** lblanchard has joined #tripleo12:27
*** liverpooler has joined #tripleo12:28
openstackgerritChristian Schwede proposed openstack/tripleo-heat-templates master: Fix Swift storage device setup  https://review.openstack.org/57572212:31
*** tcw1 has joined #tripleo12:31
*** tcw has quit IRC12:31
*** jaganathan has quit IRC12:32
*** leanderthal has quit IRC12:34
mhenkel_hi all,12:36
mhenkel_anyone ever had an issue where the workflow creation silently stops with the the only line 'passwords' ?12:36
d0ugalmhenkel_: What CLI command?12:37
mhenkel_http://paste.openstack.org/show/723545/12:37
d0ugalmhenkel_: I suspect you are getting a KeyError exception12:37
mhenkel_this happened between two openstack deploy runs12:37
mhenkel_I stopped one with CTRL+C12:38
d0ugalmhenkel_: Right - it is a limitation of python-tripleoclient/python-openstackclient12:38
mhenkel_anything I can do without re-installing the undercloud?12:38
d0ugalbasically there is a keyerror in tripleoclient and by default openstackclient prints str(exception) - which when it is a keyerror is just the name of the missing key :/12:38
*** ykarel has quit IRC12:39
*** ykarel has joined #tripleo12:39
d0ugalmhenkel_: I am not sure what the root cause is.12:39
mhenkel_ok12:39
d0ugalI think there may have been a bug about it12:39
* d0ugal digs12:40
d0ugalmhenkel_: if you run the command again with "--debug" at the end you will get more information and the full Python traceback12:40
mhenkel_unfortunately the search parameters for this are quiet wide12:40
d0ugalIndeed12:41
mhenkel_http://paste.openstack.org/show/723546/12:41
mhenkel_this is very clear12:41
d0ugalOkay, so my guess about KeyError was correct :)12:41
d0ugalmhenkel_: so it seems, for some reason the passwords are missing from your plan.12:42
d0ugalI think this is the similar, but different root cause, bug I was thinking of: https://bugs.launchpad.net/tripleo/+bug/173470412:43
openstackLaunchpad bug 1734704 in tripleo "Tripleoclient failures don't produce enough information" [Medium,In progress] - Assigned to Dougal Matthews (d0ugal)12:43
mhenkel_checking12:43
d0ugalSimilar issue in tripleoclient12:43
*** abishop has joined #tripleo12:44
d0ugalmhenkel_: did you disable password generation?12:45
d0ugalThere is probably a way to trigger the password generation to make sure they exist...12:45
* d0ugal checks12:45
*** rlandy_ has joined #tripleo12:46
d0ugalStrange, I can't connect to my undercloud. So I can't check easily :/12:46
*** pradk has quit IRC12:46
*** lifeless_ has quit IRC12:46
openstackgerritJiri Tomasek proposed openstack/tripleo-ui master: [WIP] Add Network Topology diagram  https://review.openstack.org/56912512:46
*** dmellado_ has joined #tripleo12:46
*** gouthamr has quit IRC12:46
*** stevebaker has quit IRC12:47
*** rlandy_ has quit IRC12:48
*** rlandy has quit IRC12:48
*** rlandy_ has joined #tripleo12:48
*** dmellado has quit IRC12:49
*** rlandy_ is now known as rlandy12:49
*** myoung|off is now known as myoung12:49
mhenkel_d0gual: can the plan just be re-created?12:51
*** dougbtv_ has joined #tripleo12:53
d0ugalmhenkel_: Yeah, with the CLI commands "openstack overcloud plan delete" and "openstack overcloud plan create"12:53
mhenkel_thx12:54
*** anande has quit IRC12:58
*** rcernin has quit IRC13:02
beaglesdoes anyone know of any place in updates, upgrades, or even just deploying where something might cause a restart of all running containers on a host?13:03
*** pradk has joined #tripleo13:04
*** erlon_ has joined #tripleo13:05
*** paramite has quit IRC13:05
beaglesEmilienM: mwhahaha: bogdando: ccamacho: dciabrin_ ^13:06
beaglesI mean a blanket restart, not a paunch restart because a config changed etc13:07
beagles(the containers I'm concerned with aren't paunch managed at any rate)13:07
mhenkel_d0ugal: looks like plan re-creation helped. It's at least progressing further now13:08
mhenkel_thanks!13:08
*** ooolpbot has joined #tripleo13:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION13:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/177713013:10
*** ooolpbot has quit IRC13:10
openstackLaunchpad bug 1777130 in tripleo "RDO cloud is down" [Critical,New] - Assigned to Quique Llorente (quiquell)13:10
dciabrin_beagles, in t-h-t puppet/services/docker.yaml use to docker stop all container before a docker package update. it's moved to ansible starting I2f5bc355d3c952e2daf9ec7442850e24bed275e113:11
dciabrin_beagles, not sure where it is now, or if it still stops containers13:12
beaglesdciabrin_: thanks I'll check13:12
d0ugalmhenkel_: np13:13
openstackgerritHonza Pokorny proposed openstack/tempest-tripleo-ui master: Add basic project structure  https://review.openstack.org/57573013:14
erlon_@all, folks, I'm trying to build a virtual environment using RH director. I have director installed but I'm not being able to get a heat configuration that can bring my overcloud up. Does anyone has have done this before and have the templates+plus CLI command to share?13:14
beaglesdciabrin_: strictly speaking we shouldn't really need to do that anymore - correct?13:14
dciabrin_beagles, correct, now that the docker bug (hanging on restart) has been fixed, we shouldn't need it anymore13:15
erlon_I tried using single_nic_vlans.yaml13:15
*** tzumainn has joined #tripleo13:16
*** tzumainn has quit IRC13:17
*** tzumainn has joined #tripleo13:17
*** rh-jelabarre has quit IRC13:18
*** amoralej is now known as amoralej|lunch13:22
*** rh-jelabarre has joined #tripleo13:25
*** mcornea has joined #tripleo13:27
*** suuuper has quit IRC13:27
*** links has quit IRC13:27
*** ykarel is now known as ykarel|away13:31
beaglesdciabrin_: I think that is what is causing my issue, thanks13:32
dciabrin_beagles, ouch :)13:32
beaglesdciabrin_: indeed, but at least we know now :)13:33
beaglesor at least think we do13:33
*** Haresh has quit IRC13:34
*** mmethot has quit IRC13:36
openstackgerritMartin Schuppert proposed openstack/tripleo-heat-templates master: Introduces NovaComputeCpuSharedSet parameter  https://review.openstack.org/57544913:38
*** pblaho has quit IRC13:39
*** ykarel|away has quit IRC13:40
beaglesdciabrin_: what do you think the chances of us dropping the "stop all containers" steps from rocky and queens would be?13:42
dciabrin_beagles, tough question. the problem is that docker ships in RHEL IIRC, so if the RHOS channel is updated before we have the fixed docker in the env, minor update will fail, for another reason.13:44
openstackgerritMarius Cornea proposed openstack/tripleo-upgrade master: Add split-stack environment support.  https://review.openstack.org/57277813:44
beaglesEmilienM: mwhahaha: that's a question for you as well ^13:44
dciabrin_beagles, does the problem only happen in OSP13? in which case, we might be covered by major upgrade if it enforces a RHEL upgrade before proceeding13:44
beaglesdciabrin_: ugh yeah13:44
dciabrin_marios, ^^13:44
EmilienMbeagles: good question for upgrade folks13:44
EmilienM(I haven't been much involved on that area, tbh)13:45
beaglesdciabrin_: yeah, I think we are talking queens/13 as this is the only affect neutron and it's agents at the moment13:45
beaglesEmilienM: ack13:45
openstackgerritEmilien Macchi proposed openstack/tripleo-quickstart-extras master: Use the in-built image prepare for undercloud install  https://review.openstack.org/57161313:46
openstackgerritEmilien Macchi proposed openstack/tripleo-quickstart-extras master: Use the in-built image prepare for undercloud install  https://review.openstack.org/57161313:46
openstackgerritEmilien Macchi proposed openstack/tripleo-quickstart-extras master: Use role tripleo-modify-image to update containers  https://review.openstack.org/57163113:46
openstackgerritEmilien Macchi proposed openstack/tripleo-quickstart-extras master: Use role tripleo-modify-image to update containers  https://review.openstack.org/57163113:46
EmilienMgate is 11h13:47
EmilienMI think we should stop approving things until it's resolved13:47
*** dansmith is now known as superdan13:47
*** toure|gone is now known as toure13:47
beaglesack13:48
beagles"impending weekend gate syndrome"13:48
*** Haresh has joined #tripleo13:49
*** quiquell|rover is now known as quique|rover|lch13:49
openstackgerritBogdan Dobrelya proposed openstack/tripleo-common master: Improve exp backoff for images pull/push/rpm-upd  https://review.openstack.org/57574213:49
EmilienMbandini: ++ thanks for your feedback on storyboard, the search is not great I agree13:49
EmilienMweshay|ruck: is someone actively looking at why the gate is 11h long?13:49
bogdandoweshay|ruck, StevenK, EmilienM: https://review.openstack.org/#/c/575742/ I hope that improves the containers situation for CI infra13:50
bogdandoStevenK: oh, I meant Steve Baker, sorry13:51
StevenKbogdando: Heh heh, no worries13:51
*** saneax has quit IRC13:52
EmilienMbogdando: do you have an example of logs in CI that indicated it was that?13:53
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-heat-templates master: Log memcached to stdout/err only  https://review.openstack.org/57567613:53
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-heat-templates master: Activate memcached debug only when using MemcachedDebug param.  https://review.openstack.org/57569313:53
EmilienMbogdando: without bug report or example of logs, it's pure guessing now13:53
*** dparkes has quit IRC13:54
chembogdando: fyi I rebased memcached https://review.openstack.org/#/c/57567613:55
bogdandoEmilienM: I didn't found a bug report forhttp://lists.openstack.org/pipermail/openstack-dev/2018-June/131482.html13:56
bogdandothat falls into that context13:56
weshay|ruckEmilienM, I started to look this morning.. the containers patch failed in gate agian13:56
weshay|ruckagain13:56
bogdandoyeah, wanted to post that link13:56
weshay|ruckI'll free up in a bit, I need quique|rover|lch to cover that now13:57
bogdandoEmilienM: 3:01:30 PM GMT+3 - weshay|ruck: ya.. same thing http://logs.openstack.org/13/571613/18/gate/tripleo-ci-centos-7-scenario001-multinode-oooq-container/298ee13/logs/undercloud/home/zuul/overcloud_prep_containers.log.txt.gz13:57
bogdandosee the log time stamps for failures13:57
bogdandoand we need to compare it between all of the failed multinode jobs13:57
bogdandoin that run13:57
bogdandoor other runs taken the same time prolly13:58
bogdandoI think that data leaves in logstash...13:58
EmilienMbogdando: I'm not sure the retry will really help here13:58
bogdandobut have not idea how to query it13:58
bogdandocannot prove that strictly EmilienM13:59
bogdandoformally*13:59
bogdandoI have no data to operate13:59
mariosdciabrin_: beagles so if the question is do we expect rhel updated before the major upgrade the answer is yes (minor update delivers that and we expect operator to do minor before major)13:59
*** ratailor has joined #tripleo14:00
bogdandobut I think exponential backoff shifting all that DoS messy CI jobs load into a 3 min intervals randomly, might alleviate14:00
bogdandoand multi-process workers in each of the jobs as well14:00
bogdandoI designed those intervals be interleaved with random fallbacks with fixed and longer ones14:01
*** lblanchard has quit IRC14:02
*** ksambor has quit IRC14:03
*** pradk has quit IRC14:03
*** etingof has quit IRC14:05
bogdandoEmilienM: If I had not mistaken, the backoff for each of the worker will be distributed like (rnd(6s), rnd(18s)), 15s, (rnd(6s), rnd(18s)), 45s,  (rnd(6s), rnd(18s)) <= 132s.14:05
bogdandonot sure with the very last round14:06
bogdandoif it would ever take a place14:06
bogdandoif we do not want too much wait for image workflows, we are free to experiment like we want for CI only containers update script14:07
*** yamahata has quit IRC14:07
EmilienMbogdando: https://review.openstack.org/#/c/575564/14:08
EmilienMbogdando: this is how the containeirzed undercloud is configured with br-ex now14:08
EmilienMbogdando: eth1 was wrong14:08
EmilienMbogdando: quickstart creates br-ex though14:08
*** rodolof has quit IRC14:08
bogdandowell, EmilienM I just searched all code and found it literally everyther14:08
bogdandoand nothing that uses br-ex14:09
*** rodolof has joined #tripleo14:09
bogdandohttp://codesearch.openstack.org/?q=NeutronPublicInterface&i=nope&files=&repos=14:09
bogdandoso I'm not sure14:09
alee__d0ugal, rbrady , toure  - made changes to https://review.openstack.org/#/c/574782/ as requested14:09
EmilienMbogdando: the job is working now14:10
EmilienMbogdando: it's actually fixing the job, please remove your -114:10
*** ooolpbot has joined #tripleo14:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION14:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/177713014:10
openstackLaunchpad bug 1777130 in tripleo "RDO cloud is down" [Critical,New] - Assigned to Quique Llorente (quiquell)14:10
*** ooolpbot has quit IRC14:10
EmilienMbandini: nice spot for the mirror problem14:11
EmilienMhttp://logs.openstack.org/29/575129/1/check/tripleo-ci-centos-7-containerized-undercloud-upgrades/bf6cb55/logs/undercloud/home/zuul/undercloud_upgrade.log.txt.gz#_2018-06-14_14_54_3014:11
EmilienMException: Error inspecting image: docker://docker.io/tripleomaster/centos-binary-iscsid:current-tripleo14:11
EmilienM2018-06-14 14:54:30 | time="2018-06-14T14:54:30Z" level=fatal msg="Error determining repository tags: Get https://registry-1.docker.io/v2/tripleomaster/centos-binary-iscsid/tags/list: EOF"14:11
EmilienMI would rather investigate that problem14:11
bandiniEmilienM: :)14:12
EmilienMmwhahaha, bandini : I think it happens before we configure docker14:12
EmilienMso the mirror isn't configured yet, let me check14:12
mwhahahaEmilienM: negative it's configured14:12
mwhahahaEmilienM: it's a different set of software14:12
EmilienMmhh weird, docker is configured http://logs.openstack.org/29/575129/1/check/tripleo-ci-centos-7-containerized-undercloud-upgrades/bf6cb55/logs/undercloud/etc/docker/daemon.json.txt.gz14:12
mwhahahaso i don't think it honors the docker config14:12
* mwhahaha files a bug14:12
EmilienMok14:12
ratailorDoes anyone have any suggestion about this comment here. https://review.openstack.org/#/c/575623/1/docker/services/nova-api.yaml14:13
openstackgerritFlorian Fuchs proposed openstack/tripleo-validations master: [WIP]Add a OpenShift of OpenStack HW validation  https://review.openstack.org/57574914:13
openstackgerritFlorian Fuchs proposed openstack/tripleo-validations master: [WIP]Add a OpenShift on OpenStack HW validation  https://review.openstack.org/57574914:13
ratailorEmilienM, jaosorior ^^14:13
mwhahahaEmilienM, bandini, bogdando: https://bugs.launchpad.net/tripleo/+bug/177714314:14
openstackLaunchpad bug 1777143 in tripleo "container tag discovery does not use a mirror" [High,Triaged]14:14
mwhahahai think we can switch the mirror by configuring the namespace correctly in oooq14:14
* mwhahaha isn't 100% on that though14:15
bogdandoEmilienM: I removed -1, could you please elaborate how undercloud already has br-ex configured for NeutronPublicInterface?14:15
mwhahahabogdando: in CI we add it in the vxlan script14:15
mwhahahabogdando: see bottom of http://logs.openstack.org/29/575129/1/check/tripleo-ci-centos-7-containerized-undercloud-upgrades/bf6cb55/logs/undercloud/home/zuul/vxlan_networking.sh.log.txt.gz#_2018-06-14_14_16_4714:16
bogdandomwhahaha: I cannot see how this relates to heat's NeutronPublicInterface14:17
mwhahahabogdando: so that's what we end up putting the local-ip on14:17
mwhahahabogdando: this is just matching what we already do for the undercloud14:18
mwhahahain CI14:18
mwhahahathis is purely an upstream CI configuration14:18
*** etingof has joined #tripleo14:18
mwhahahabogdando: see local_interface in http://logs.openstack.org/29/575129/1/check/tripleo-ci-centos-7-containerized-undercloud-upgrades/bf6cb55/logs/undercloud/home/zuul/undercloud.conf.txt.gz14:18
EmilienMmwhahaha: there is "docker_registry_host" in quickstart14:19
mwhahahaEmilienM: yea but i don't know if skopeo honors that stuff14:19
mwhahahabogdando: so in undercloud local_interface -> NuetronPublicInterface14:19
mwhahahabogdando: that's our network config mapping crud14:19
EmilienMand it's used by "openstack overcloud container image prepare"14:19
mwhahahabogdando: https://github.com/openstack/python-tripleoclient/blob/master/tripleoclient/v1/undercloud_config.py#L6714:20
*** pradk has joined #tripleo14:20
mwhahahaEmilienM: but we don't use it when we do the automagic tag discovery in containerized undercloud i bet14:20
bogdandomwhahaha: oh, right! those 'local_interface': 'NeutronPublicInterface' magic mappings14:21
*** lblanchard has joined #tripleo14:21
*** florianf has quit IRC14:21
*** florianf has joined #tripleo14:22
*** mvenesio has joined #tripleo14:22
EmilienMmwhahaha: what worries me is I don't see docker_registry_host overridden with our mirror except for rdo cloud jobs14:22
*** amoralej|lunch is now known as amoralej14:23
EmilienMgrep docker_registry_host in tripleo-ci repo14:23
*** bdodd_ is now known as bdodd14:24
EmilienMso the problem is probably in both "tripleo image container prepare" and the old container image prepare as well14:24
EmilienMbandini spotted the one when deploying the undercloud, so the new command14:24
*** quique|rover|lch is now known as quiquell|rover14:24
EmilienMbut for the second one I suspect something in quickstart14:24
*** rodolof has quit IRC14:24
bogdandomwhahaha, EmilienM: here is that magic mapping http://logs.openstack.org/29/575129/1/check/tripleo-ci-centos-7-containerized-undercloud-upgrades/bf6cb55/logs/undercloud/home/zuul/tripleo-heat-installer-templates/undercloud_parameters.yaml.txt.gz . I completely forgot about that mappings, thanks for refreshing my mind :D14:27
openstackgerritRonelle Landy proposed openstack-infra/tripleo-ci master: Add dry run option to toci_quickstart - playbook order in tact  https://review.openstack.org/57069414:27
* EmilienM brb 30m14:28
mwhahahabogdando: yea so for standalone we're just generating that file by hand for now until we can create a standalone.conf or some sort14:28
openstackgerritJohn Trowbridge proposed openstack-infra/tripleo-ci master: Remove zuul-cloner from buildimage job  https://review.openstack.org/57195114:28
openstackgerritJohn Trowbridge proposed openstack-infra/tripleo-ci master: Remove zuul-cloner from tripleo-ci job  https://review.openstack.org/57199514:28
openstackgerritJohn Trowbridge proposed openstack-infra/tripleo-ci master: Refactor Zuul job using base jobs to consolidate the configuration  https://review.openstack.org/57202714:28
openstackgerritJohn Trowbridge proposed openstack-infra/tripleo-ci master: Add TRIPLEO_CI_ROOT variable  https://review.openstack.org/57412214:28
openstackgerritJohn Trowbridge proposed openstack-infra/tripleo-ci master: WIP: bootstrap a native zuulv3 tripleo-ci job  https://review.openstack.org/57575214:28
openstackgerritBrent Eagles proposed openstack/ansible-role-container-registry master: Do not stop all running containers on docker update  https://review.openstack.org/57575614:29
openstackgerritCarlos Goncalves proposed openstack/puppet-tripleo stable/pike: Install python-neutron-lbaas before starting neutron  https://review.openstack.org/57550014:32
openstackgerritBrent Eagles proposed openstack/tripleo-heat-templates stable/queens: Do not stop all running containers on docker update  https://review.openstack.org/57575814:33
*** yprokule has quit IRC14:39
larsksmwhahaha: so, in the horizon service template, we're setting horizon::keystone_url to the internal endpoint...but this is used for browser redirects when using web sso, so it should actually be the public endpoint.  Should I just submit a patch to change that? Or is there a way to paramaterize which endpoint we use (nested get_param)?14:40
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates stable/queens: Fix error in the role-specific parameter merging in rhsm service  https://review.openstack.org/57576114:40
larsks(https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/services/horizon.yaml#L124)14:40
mwhahahalarsks: i'd assume there's a KeystonePublic but you should check with the keystone folks on that14:41
*** ramishra has quit IRC14:41
mwhahahalarsks: yea there's a KeystonePublic so if it's supposed to be the public end point just submit a patch to change it14:41
larsksmwhahaha: I have, and they agree we should use the public endpoint there.  But should I replace KeystoneInternal with KeystonePublic, or should I replace it with something like {get_param: KeystoneEndpointName}?14:41
larsks(so that it can be configured conveniently at deploy time)14:41
larsks(and introduce a new parameter, obviously)14:42
mwhahahalarsks: no KeystonePUblic14:42
larsksAwesome.14:42
mwhahahalarsks: because that comes from the endpoint map14:42
larsksThanks.14:42
*** rpioso|afk is now known as rpioso14:42
larsksOh, right, but I was asking if we could parameterize which endpointmap key we're requesting.  E.g., KeystoneEndpointName would be a parameter that could be KeystoneInternal or KeystonePublic, and then we wwould get that from EndpointMap.14:42
mwhahahaoh not sure about that14:43
mwhahahaunless we're already doing that elsewhere i don't think we should14:43
larsksOkay. I'll look.14:44
mwhahahaIf you were going to add a param it would be a condition between internal/external that we could then use to switch the get param bits. but i think that's over thinking it unless there's a reason14:44
mwhahahayou may want to ask jaosorior14:44
openstackgerritLars Kellogg-Stedman proposed openstack/tripleo-heat-templates master: [WIP] enable support for openidc federation in keystone  https://review.openstack.org/57485514:46
openstackgerritLars Kellogg-Stedman proposed openstack/tripleo-heat-templates master: use keystone public endpoint in horizon  https://review.openstack.org/57576214:46
*** ratailor has quit IRC14:46
larsksmwhahaha: jaosorior: https://review.openstack.org/#/c/575762/14:48
openstackgerritBogdan Dobrelya proposed openstack/tripleo-common master: Improve exp backoff for images pull/push/rpm-upd  https://review.openstack.org/57574214:49
weshay|rucktime="2018-06-15T14:35:19Z" level=fatal msg="pinging docker registry returned: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 127.0.0.1:53: no such host"14:53
weshay|ruckEmilienM, this is odd14:53
weshay|ruckgate failure14:53
weshay|ruckguess just a dns fail14:53
quiquell|roverweshay|ruck: http://logs.openstack.org/13/571613/18/gate/tripleo-ci-centos-7-scenario007-multinode-oooq-container/9a52ece/logs/undercloud/home/zuul/overcloud_prep_containers.log.txt.gz14:54
quiquell|roversomething about not finding RPMs14:54
*** myoung is now known as myoung|bbl14:54
quiquell|roverERROR: ERROR running rpm query in container:14:55
*** dparkes has joined #tripleo14:57
weshay|ruckEmilienM, do you have a sec?15:00
weshay|ruckor bogdando or sbaker15:01
EmilienMyes I'm here now15:01
*** pcaruana has quit IRC15:01
weshay|ruckEmilienM, k.. need your eyes.. it looks like container update is failing to find the container in the undercloud registry when updating15:02
weshay|ruckhave a log up need eyes15:02
weshay|ruckmy bluejeans channeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee15:02
weshay|ruckeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeel15:02
weshay|ruckugh.. bluejeans lag15:02
*** lblanchard has quit IRC15:04
weshay|ruckEmilienM, I suspect the diff in containers between the containerized undercloud and whats required by the overcloud?15:07
weshay|ruckhttp://logs.openstack.org/13/571613/18/gate/tripleo-ci-centos-7-scenario007-multinode-oooq-container/9a52ece/logs/undercloud/home/zuul/overcloud_prep_containers.log.txt.gz15:07
weshay|ruckmeh.. this is not a containerized undercloud nvrmind15:08
quiquell|roverweshay|ruck: Have to drop15:08
openstackgerritBogdan Dobrelya proposed openstack/tripleo-common master: Improve exp backoff for images pull/push/rpm-upd  https://review.openstack.org/57574215:08
bogdandoweshay|ruck: LD15:09
*** quiquell|rover is now known as quiquell|off15:09
bogdandofor the moment I started to believe that might be the case15:09
weshay|ruckbogdando, so weird https://review.openstack.org/#/c/571613/ works in check but fails in gate over and over15:10
*** ooolpbot has joined #tripleo15:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION15:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/177713015:10
*** ooolpbot has quit IRC15:10
openstackLaunchpad bug 1777130 in tripleo "RDO cloud is down" [Critical,Triaged] - Assigned to Quique Llorente (quiquell)15:10
*** weshay|ruck changes topic to "Welcome to Rocky. CI status: RED - rdo-cloud is down, long upstream zuul queue | http://tripleo.org/ | https://docs.openstack.org/tripleo-docs/latest"15:10
bogdandoweshay|ruck: are node pool vms are the same for check vs gate?15:11
EmilienMyes15:11
bogdandomay be the latter cannot access registry mirrors?15:11
weshay|ruckbogdando, they could be running on diff providers between check and gate15:11
bogdandoor poorly located, slow etc15:11
bogdandomeh, kolla images really need to be included into those vms15:11
weshay|ruckthe error is that it can't access the container from the undercloud during container-check15:11
EmilienMbogdando: it doesn't help, we modify them15:12
bogdandothe delta might be less at least for containers pull15:12
bogdandonot for rpm updates, indeed...15:12
weshay|ruckTrying to pull repository 192.168.24.1:8787/tripleomaster/centos-binary-iscsid ...15:12
weshay|ruck2018-06-15 08:45:23 | Pulling repository 192.168.24.1:8787/tripleomaster/centos-binary-iscsid15:12
weshay|ruck2018-06-15 08:45:23 | /usr/bin/docker-current: Error: image tripleomaster/centos-binary-iscsid:39d396e0ae0d982d63cbcadef57e75e84a052911_033c532c not found.15:12
* weshay|ruck bringing up a recreate now15:12
*** gouthamr has joined #tripleo15:19
bogdandoPTAL https://review.openstack.org/#/c/572404/15:23
bogdandohttps://review.openstack.org/#/c/575676/15:24
*** mschuppert has quit IRC15:29
bogdandoweshay|ruck, EmilienM: that's weird it pulls from the local registry w/o pushing it there from the upstream source15:29
*** dxiri has joined #tripleo15:30
bogdandoit seems15:30
bogdandoopenstack tripleo container image prepare didn't pull it15:30
bogdandoor was it cleaned up?15:34
larsksEmilienM: what's the correct way of setting multi-valued keystone configuration settings in puppet (ones that should result in multiple lines each with the same config key)?15:35
larsks(I tried keystone_config { 'trusted_dashboard': value => $a_list_variable }, and that didn't work...)15:37
bogdandoweshay|ruck, EmilienM: http://logs.openstack.org/13/571613/18/gate/tripleo-ci-centos-7-scenario007-multinode-oooq-container/9a52ece/logs/undercloud/var/log/extra/docker/docker_allinfo.log.txt.gz shows 0 images in the local cache, I think that's expected for cleanup. So we need to figure out why the image is not in local registry15:37
*** gouthamr has quit IRC15:38
larsksOr mwhahaha ^^^^ if you have any idea.15:38
*** gouthamr has joined #tripleo15:39
EmilienMbogdando: http://logs.openstack.org/13/571613/18/gate/tripleo-ci-centos-7-scenario007-multinode-oooq-container/9a52ece/logs/undercloud/home/zuul/overcloud_prep_containers.log.txt.gz15:39
EmilienMit's because the container image prepare failed15:39
*** rbrady is now known as rbrady-afk15:39
EmilienMand the undercloud isn't containerized so no containers are in local registry15:39
mwhahahalarsks: the provider has to support those (it may not)15:39
mwhahahalarsks: also, i hate those types of params15:39
larsksYeah, me too.15:39
EmilienMmulti params is the worst thing they created for operators wrt config management15:40
mwhahahawe did add support in the puppet providers and i think some of the neutron ones support it, the keystone ones may not15:40
mwhahahaso it would have to be updated to15:40
larsksmwhahaha: it looks like they just use puppet-openstacklib?15:41
mwhahahayea but there's a specific pattern for it15:41
* mwhahaha doesn't remember it's been a few yuears15:41
mwhahahai think we even added it in for lbaas or some crap like that15:41
* larsks looks at puppet-neutron15:42
EmilienMi think we added it to puppet keystone15:42
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates master: Update CephX client.openstack keyring to use 'profile rbd'  https://review.openstack.org/57577615:43
mwhahahanova has it15:43
larsksmwhahaha: in lib/puppet/provider/nova_config/openstackconfig.rb?15:44
EmilienMwell, can't find it, probably not then15:44
mwhahahahttps://review.openstack.org/#/c/329991/15:44
mwhahahayou have to use the openstackconfig version of the provider15:44
mwhahahanot ini_setting15:44
EmilienMhttps://review.openstack.org/#/c/492172/15:45
mwhahahait's complicated15:45
bogdandoEmilienM: in that log, I can see the container-check command invoked before any failures15:45
bogdandohow was it expected to work w/o containers pulled and pushed into local registry?15:45
larsksmwhahaha: okay, thanks.15:46
larsksI'll poke at it.15:46
bogdandoEmilienM: or do you mean there is a silent failure after http://logs.openstack.org/13/571613/18/gate/tripleo-ci-centos-7-scenario007-multinode-oooq-container/9a52ece/logs/undercloud/home/zuul/overcloud_prep_containers.log.txt.gz#_2018-06-15_08_39_07 ?15:46
bogdandoit had been doing pull/push for a 6 min, right? didn't make it?15:47
bogdandowe need to improve logging :)15:47
openstackgerritBob Fournier proposed openstack/tripleo-docs master: Document how to use a composable network with Ironic in Overcloud  https://review.openstack.org/57043715:50
EmilienMbogdando: right, that's one of the problems we need to look15:52
EmilienMbut I think there is more15:52
EmilienMI wish we had a list of reasons why gate job are failing now15:52
EmilienMis the CI team can provide that?15:52
EmilienMweshay|ruck: ^ do some work please15:52
bogdandoEmilienM: hm, there is a tones of ' HTTP response to HTTPS client' in underlcoud journald15:52
EmilienMweshay|ruck: :P15:53
weshay|ruckEmilienM, interesting.. openstack-infra is saying the proxy cache is not working correctly15:53
* EmilienM trolling it's friday !!15:53
bogdando\o/15:53
bogdandoplease do the needful? :D15:53
EmilienMbogdando has a good point with the update issues15:53
*** radeks has joined #tripleo15:53
EmilienMbut we also need to figure out https://bugs.launchpad.net/tripleo/+bug/177714315:53
openstackLaunchpad bug 1777143 in tripleo "container tag discovery does not use a mirror" [High,Triaged]15:53
weshay|ruckoh man15:54
weshay|ruckthe jokes15:54
weshay|ruckthe jokes15:54
EmilienMweshay|ruck: can you provide links of jobs that failed in the gate during the last 10 hours15:55
EmilienMweshay|ruck: let's look at the error, and file a bug for each of them15:55
weshay|ruckEmilienM, when rdo cloud comes back15:55
EmilienMcome on don't tell me we rely on rdo cloud to tell what gate job is failing15:55
EmilienMI quit15:55
weshay|ruckhttp://38.145.34.131:3000/d/pgdr_WVmk/ruck-rover?orgId=115:56
openstackgerritJohn Trowbridge proposed openstack-infra/tripleo-ci master: WIP: bootstrap a native zuulv3 tripleo-ci job  https://review.openstack.org/57575215:56
weshay|ruckis down.. that has the list15:56
EmilienMdamn15:56
EmilienMtrying to use http://zuul.openstack.org/builds.html15:56
EmilienMbut it's damn slow15:56
weshay|ruckEmilienM, read through openstack-infra as well15:58
EmilienMI'm sure we can find it via logstash15:58
EmilienMlet me build a query to find out errors in gate15:58
weshay|ruckEmilienM, if we do use the proxy.. it's just a direct pass through15:58
weshay|ruckand they are trying to fix that15:58
EmilienMah so if docker.io decides to not respond the proxy is useless15:59
EmilienMwhy do we use a proxy then15:59
bogdandointeresting16:00
bogdandohow did you figure that out EmilienM?16:00
bogdandothat behavior of mirrors?16:00
EmilienMbogdando: join #openstack-infra16:00
bogdandoah, ok16:00
*** dmellado_ has quit IRC16:02
bogdandoEmilienM: aside of those HTTP vs HTTPS errors, there is a craptone of 'Attempting next endpoint for pull after error: manifest unknown: manifest unknown'16:02
bogdandoand others like Not continuing with pull after error: Error: image tripleomaster/centos-binary-heat-api:39d396e0ae0d982d63cbcadef57e75e84a052911_033c532c not found16:02
bogdandothose are failing into those 6 silent minutes16:03
bogdandoEmilienM: my bad, they are later16:03
bogdandocome after 08:45:0916:04
bogdandoso we really have nothing to inspect for those 6 mins it seems16:04
EmilienMI have a list of failures in the gate :16:05
EmilienMgoo.gl/kbdMH716:05
*** ykarel|away has joined #tripleo16:05
EmilienMweshay|ruck: ^16:05
weshay|rucklooking at it now16:06
weshay|ruckthat is a very fine point16:06
EmilienMI'm trying to build a better query but you got the idea16:07
weshay|ruckI have one16:07
*** ffiore has quit IRC16:07
EmilienMshare it like : build_queue: ^gate$ AND build_name: *tripleo-ci* AND build_status: FAILURE16:08
*** tcw1 has quit IRC16:08
EmilienMso it's pretty clear when we purged the gate16:08
EmilienMand when it came back unstable16:08
EmilienMyou can see the revert between the 2 picks16:08
EmilienMand the revert didn't help that much it seems16:09
EmilienMthe queue is 13h now16:09
*** ooolpbot has joined #tripleo16:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION16:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/177713016:10
*** ooolpbot has quit IRC16:10
openstackLaunchpad bug 1777130 in tripleo "RDO cloud is down" [Critical,Triaged] - Assigned to Quique Llorente (quiquell)16:10
EmilienMhttp://zuul.openstack.org/builds.html is also great to see job health16:10
EmilienMweshay|ruck: ^ we should use more of existing infra tools sometimes16:11
EmilienMinstead of creating new things16:11
EmilienMnew things are good, but we need to also use existing things16:11
EmilienManyway, I'm building up a list of failures seen in logstash16:11
EmilienMhttp://etherpad.openstack.org/p/tripleo-gate-issues-june-201816:11
*** bogdando has quit IRC16:13
EmilienMthe only thing with http://zuul.openstack.org/builds.html is I can't find how to use regex16:13
EmilienMI guess we just can't16:13
weshay|ruckEmilienM, re: building tools, if the team has to go to 10 different web sites to get clarity of the status16:14
weshay|ruckthat is a problem, and that is what we are resolving16:14
EmilienMsure but your stuff is not available now16:15
EmilienManyway16:15
EmilienMcan we get help from CI folks please16:15
*** tcw has joined #tripleo16:16
openstackgerritRaoul Scarazzini proposed openstack/tripleo-ha-utils master: WIP - Introduce simulate failure task  https://review.openstack.org/57578416:16
*** ayoung has quit IRC16:18
EmilienMthe query isn't good16:23
EmilienMI have successful jobs in the list of results16:23
*** holser__ has quit IRC16:25
*** ayoung has joined #tripleo16:25
*** jpena is now known as jpena|off16:27
*** cylopez has quit IRC16:28
*** jpena|off is now known as jpena16:29
*** jpena is now known as jpena|away16:29
*** panda is now known as panda|off16:30
EmilienManother issue in gate:16:30
EmilienMhttp://logs.openstack.org/84/573184/9/gate/tripleo-ci-centos-7-scenario007-multinode-oooq-container/886f6e3/job-output.txt.gz#_2018-06-15_14_32_37_61151116:30
EmilienMbuilding python-networking-ovn rpm this time16:30
mwhahaha:(16:30
EmilienMjust read logstash it's magic16:31
EmilienMit tells you where shit broke16:31
mwhahahaovn packaging is bad16:31
mwhahahagot new files16:31
EmilienMfiling a bug for this one16:31
mwhahahaansible'd16:31
mwhahahahttp://logs.openstack.org/84/573184/9/gate/tripleo-ci-centos-7-scenario007-multinode-oooq-container/886f6e3/logs/delorean_logs/cc/17/cc17d0a2eaa87d51f313e8f42bf59a3b25579e1f_dev/rpmbuild.log.txt.gz16:31
mwhahahawhy the hell is tripleo update role going in neutroning-ovn-migration16:32
mwhahahadalvarez: https://review.openstack.org/#/c/574211/ -^16:33
EmilienMnew alert: https://bugs.launchpad.net/tripleo/+bug/177716816:33
openstackLaunchpad bug 1777168 in tripleo "Failures to build python-networking-ovn" [Critical,Triaged]16:33
EmilienMwe probably want to make the job non voting now if nobody is looking at it quickly16:33
mwhahahayes plz16:38
*** jpich has quit IRC16:39
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates stable/pike: Set Ceph pools rule_name to replicated_rule  https://review.openstack.org/57578916:40
*** derekh has quit IRC16:41
honzaEmilienM: i'm working on the tempest plugin for tripleo-ui, and i'm wondering where to plug it in --- do you know, off the top of your head, if there is a ci job that has both an undercloud and tempest available?16:42
*** mhenkel_ has quit IRC16:43
*** gkadam has quit IRC16:43
*** mhenkel_ has joined #tripleo16:43
*** ykarel|away has quit IRC16:43
EmilienMhonza: fs02716:46
EmilienMI'm going to purge the gate (again)16:46
EmilienMso we can land the patch weshay|ruck will propose to disable voting on scenario00716:46
openstackgerritwes hayutin proposed openstack-infra/tripleo-ci master: scen007 is failing in the gate  https://review.openstack.org/57579016:47
*** mhenkel_ has quit IRC16:47
*** hamzy_ has joined #tripleo16:48
EmilienMhttp://logs.openstack.org/81/568781/4/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/631a562/logs/ara_oooq/16:50
EmilienMweshay|ruck: ^16:50
*** gouthamr has quit IRC16:50
*** gouthamr has joined #tripleo16:50
*** hamzy has quit IRC16:51
dalvarezmwhahaha: those roles are to migrate from ML2/OVS to OVN16:51
dalvarezmwhahaha: i proposed the patch to RDO16:51
dalvarezbut revoew.rdoproject.org seems down16:52
mwhahahadalvarez: hmm ok seems odd to have something named tripleo in that repo and not as part of the upgrade bits for us16:53
dalvarezmwhahaha: it's the migration from ml2ovs to ovn from tripleo16:54
dalvarezi proposed the patch to fix the packaging a few days back but we didnt have all the upstream bits merged in networking-ovn16:54
dalvareznow they got merged but the package is broken and review.rdoproject is down :S16:54
mwhahahadalvarez: yea we probably should have added conditionals to the packaging so it worked in both cases before merging that16:55
mwhahahanow it's completely hosed the packaging for that16:55
mwhahahaand rdocloud is having problem so yea16:55
dalvarezyeah :s didn't know about those conditionals16:55
EmilienMmnaser: just by curiosity, is there anything happening in your cloud right now?16:55
EmilienMlike networking issues or something16:55
mwhahahawhen adding new directories you can just  if [ -d ... ] on the d for inclusion16:56
EmilienMwe are seeing that package downloads in our tripleo ci jobs took more time than usual in your cloud16:56
dalvarezmwhahaha: but in the spec it'll see the new files and it'll fail no?16:56
mwhahahadalvarez: depends on what we're doing there's some workarounds to get it in16:57
*** zoli is now known as zoli|gone16:57
*** zoli|gone is now known as zoli16:57
dalvarezmwhahaha: ack, i checked with amoralej to do this but wasn't aware of any workaround, i will double check with him for next times16:57
mwhahahadalvarez: https://github.com/rdo-packages/tripleo-common-distgit/blob/rpm-master/openstack-tripleo-common.spec#L11716:57
mwhahahadalvarez: you pre-create the dir and add it to the list16:58
dalvarezmwhahaha: gotcha, and same for entry points? this is adding console scripts too16:58
dalvarezlike for example touch <file> ?16:58
mwhahahadalvarez: yea16:58
dalvarezack16:58
mwhahahathen you could clean them up after16:58
mwhahahathat would prevent this type of failure16:59
dalvarezgot it16:59
mwhahahawe'll just disable scenario007 for now16:59
*** gfidente has quit IRC16:59
dalvarezyeah, good one... wasn't aware, hope rdocloud is back again and we can fix it ASAP, the dist-git patch passed CI last week16:59
dalvarezack16:59
openstackgerritwes hayutin proposed openstack-infra/tripleo-ci master: scen007 is failing in the gate  https://review.openstack.org/57579016:59
mwhahahaand hope it didn't break anything else :/16:59
*** tesseract has quit IRC17:01
amoralejdalvarez, mwhahaha what's the problem?17:03
mwhahahaamoralej: networking-ovn packaging is broken and rdocloud is dead17:03
dalvarezamoralej: that the upstream patches got merged in networking-ovn and we can't merge the changes in the spec so ovn jobs are broken17:03
amoralejahh17:03
EmilienMgate is purged17:03
amoraleji see17:03
openstackgerritwes hayutin proposed openstack-infra/tripleo-ci master: scen007 is failing in the gate  https://review.openstack.org/57579017:05
openstackgerritwes hayutin proposed openstack-infra/tripleo-ci master: scen007 is failing in the gate  https://review.openstack.org/57579017:10
*** ooolpbot has joined #tripleo17:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION17:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/177713017:10
openstackLaunchpad bug 1777130 in tripleo "RDO cloud is down" [Critical,Triaged] - Assigned to Quique Llorente (quiquell)17:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/177716817:10
*** ooolpbot has quit IRC17:10
openstackLaunchpad bug 1777168 in tripleo "Failures to build python-networking-ovn" [Critical,Triaged]17:10
openstackgerritwes hayutin proposed openstack/puppet-tripleo master: scen007 is failing in the gate  https://review.openstack.org/57579517:11
*** amoralej is now known as amoralej|off17:14
*** kopecmartin has quit IRC17:14
openstackgerritwes hayutin proposed openstack/tripleo-common master: scen007 is failing in the gate  https://review.openstack.org/57579617:15
openstackgerritwes hayutin proposed openstack/tripleo-heat-templates master: scen007 is failing in the gate  https://review.openstack.org/57579817:18
dalvarezcan anybody take a look at this pls? https://review.openstack.org/#/c/570942/ it's only missing workflow17:20
dalvarezbeagles: ^ :?17:20
mwhahahadalvarez: scenario007 is broken17:20
mwhahahawe cannot merge that right now17:20
dalvarezmwhahaha: ops true! damn sorry17:20
*** radeks has quit IRC17:21
openstackgerritwes hayutin proposed openstack/tripleo-quickstart master: scen007 is failing in the gate  https://review.openstack.org/57579917:22
EmilienMI've purged the check as well17:23
EmilienMso we can merge the infra patch that fix the proxy17:23
*** pradk has quit IRC17:23
EmilienM@all please do not recheck or +A17:24
*** ccamacho has quit IRC17:24
EmilienMI've abandoned a bunch of jobs from the check, i'll restore them shortly17:25
EmilienMI wanted to let the infra patch to land quickly17:25
*** pradk has joined #tripleo17:25
*** gouthamr_ has joined #tripleo17:27
*** gouthamr has quit IRC17:27
*** gouthamr_ is now known as gouthamr17:27
openstackgerritwes hayutin proposed openstack/tripleo-quickstart-extras master: WIP: use the tq ansible.cfg for the undercloud install  https://review.openstack.org/57553517:33
*** pkovar has quit IRC17:33
*** janki has quit IRC17:34
*** pradk has quit IRC17:35
*** bugzy_ is now known as bugzy17:40
*** cdearborn has quit IRC17:43
*** gkadam has joined #tripleo17:44
*** artom has joined #tripleo17:44
*** leanderthal has joined #tripleo17:47
*** jpena|away is now known as jpena17:47
*** chandankumar is now known as chkumar|pto18:01
*** stevebaker has joined #tripleo18:01
*** ooolpbot has joined #tripleo18:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION18:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/177713018:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/177716818:10
*** ooolpbot has quit IRC18:10
openstackLaunchpad bug 1777130 in tripleo "RDO cloud is down" [Critical,Triaged] - Assigned to Quique Llorente (quiquell)18:10
openstackLaunchpad bug 1777168 in tripleo "Failures to build python-networking-ovn" [Critical,Triaged]18:10
*** agopi has joined #tripleo18:11
EmilienMhttps://review.openstack.org/#/c/575787 landed, the proxy should be in place soon18:12
*** jpena is now known as jpena|off18:13
*** cdearborn has joined #tripleo18:17
*** florianf has quit IRC18:26
*** rmascena has joined #tripleo18:27
*** raildo has quit IRC18:30
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci master: scen007 is failing in the gate  https://review.openstack.org/57579018:30
EmilienMweshay|ruck: it was failing in check18:30
EmilienMI moved it to check18:30
EmilienMerr, it was failing in gate, I moved it to check18:30
*** rlandy_ has joined #tripleo18:31
*** rlandy_ has quit IRC18:31
*** rlandy_ has joined #tripleo18:31
*** rlandy has quit IRC18:33
*** gkadam_ has joined #tripleo18:33
*** rlandy_ is now known as rlandy18:36
*** gkadam has quit IRC18:37
beaglesEmilienM: are we still holding off on +A stuff because of queue?18:37
*** leanderthal has quit IRC18:37
beagleshe asks as he goes to see what the state of the queue is18:37
EmilienMbeagles: yeah please don't approve anything18:38
EmilienMyou can use check18:38
beaglesEmilienM: gotcha18:38
EmilienMbut as long as we haven't seen successful gate jobs, we hold off18:39
* beagles nods18:39
*** artom has quit IRC18:41
*** Goneri has joined #tripleo18:43
*** matbu has quit IRC18:47
*** radeks has joined #tripleo18:52
mnaserEmilienM: nothing to note in specific, do you have urls that you're seeing download dslowly18:56
mnaserEmilienM: if you have some, i can dig in and figure out whats going on18:57
*** radeks_ has joined #tripleo18:59
*** radeks has quit IRC19:01
*** ooolpbot has joined #tripleo19:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION19:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/177713019:10
openstackLaunchpad bug 1777130 in tripleo "RDO cloud is down" [Critical,Triaged] - Assigned to Quique Llorente (quiquell)19:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/177716819:10
*** ooolpbot has quit IRC19:10
openstackLaunchpad bug 1777168 in tripleo "Failures to build python-networking-ovn" [Critical,Triaged]19:10
openstackgerritJohn Fulton proposed openstack/tripleo-heat-templates master: Do not grant caps if metrics pool name is empty  https://review.openstack.org/57557119:11
*** pradk has joined #tripleo19:11
openstackgerritBen Nemec proposed openstack/tripleo-heat-templates master: Make BIND listen address configurable  https://review.openstack.org/57582219:11
openstackgerritBen Nemec proposed openstack/tripleo-heat-templates master: Make BIND /var dir persistent  https://review.openstack.org/57582319:11
*** dougbtv_ has quit IRC19:14
*** rlandy has quit IRC19:15
*** sai_p has joined #tripleo19:18
*** rlandy has joined #tripleo19:18
*** weshay|ruck changes topic to "Welcome to Rocky. CI status: RED - rdo-cloud is back, zuul queue cleared| http://tripleo.org/ | https://docs.openstack.org/tripleo-docs/latest"19:19
weshay|ruckdsneddon, if you are avail to poke at https://bugs.launchpad.net/tripleo/+bug/1777168 RDO is back and we can get a spec update in19:22
openstackLaunchpad bug 1777168 in tripleo "Failures to build python-networking-ovn" [Critical,Triaged]19:22
dsneddonweshay|ruck, I'll take a look19:26
dsneddonweshay|ruck, https://review.openstack.org/#/c/575799 looks good to me, should I merge it?19:28
dsneddonweshay|ruck, Nevermind, Emilien already did...19:29
weshay|ruckah ha19:30
weshay|ruckthanks dsneddon /me looks19:30
*** Goneri has quit IRC19:31
openstackgerritSai Ram Peesapati proposed openstack/puppet-tripleo stable/ocata: Conditionally Enable Nuage VRS  https://review.openstack.org/57582619:31
openstackgerritSai Ram Peesapati proposed openstack/puppet-tripleo stable/newton: Conditionally Enable Nuage VRS  https://review.openstack.org/57582719:31
weshay|ruckdsneddon, /me looking for a patch on distgit for http://logs.openstack.org/84/573184/9/gate/tripleo-ci-centos-7-scenario007-multinode-oooq-container/886f6e3/logs/delorean_logs/cc/17/cc17d0a2eaa87d51f313e8f42bf59a3b25579e1f_dev/rpmbuild.log.txt.gz19:33
weshay|ruckhttps://review.rdoproject.org/r/#/q/project:openstack/networking-ovn-distgit19:34
*** lifeless has joined #tripleo19:34
*** mvenesio has quit IRC19:36
*** holser__ has joined #tripleo19:37
weshay|ruckhrm.. I see networking-ovn-migration/playbooks in https://github.com/rdo-packages/networking-ovn-distgit/blob/rpm-master/python-networking-ovn.spec19:38
weshay|rucktrown, what error did you see  the build?19:38
weshay|rucksetup.cfg?19:39
*** rlandy_ has joined #tripleo19:39
*** Goneri has joined #tripleo19:39
trownweshay|ruck: hmm let me find it19:39
*** rlandy has quit IRC19:41
jristreview begging https://review.openstack.org/#/c/49893619:41
weshay|rucktrown, hrm.. https://github.com/rdo-packages/networking-ovn-distgit/blob/rpm-master/python-networking-ovn.spec#L151-L15519:42
weshay|ruckis not in19:42
weshay|ruckhttp://logs.openstack.org/84/573184/9/gate/tripleo-ci-centos-7-scenario007-multinode-oooq-container/886f6e3/logs/delorean_logs/cc/17/cc17d0a2eaa87d51f313e8f42bf59a3b25579e1f_dev/rpmbuild.log.txt.gz19:42
trownweshay|ruck: it is in http://logs.openstack.org/84/573184/9/gate/tripleo-ci-centos-7-scenario007-multinode-oooq-container/886f6e3/logs/delorean_logs/cc/17/cc17d0a2eaa87d51f313e8f42bf59a3b25579e1f_dev/rpmbuild.log.txt.gz ... but maybe that pulled older distgit?19:42
* weshay|ruck looks at new log19:42
weshay|rucktrown, /me doesn't see %files migration-tool19:44
weshay|rucktrown, we're not consistent19:44
*** radeks_ has quit IRC19:45
trownI dont think that should matter for a package built in a job19:45
openstackgerritDan Sneddon proposed openstack/os-net-config master: [WIP] Add Policy Based Routing Support to os-net-config  https://review.openstack.org/57571219:45
weshay|rucktrown, oh that is probably true19:46
* weshay|ruck wonder why it has a old copy of the spec19:46
trownweshay|ruck: only thing I can think of is that github mirror of rdoproject was not up to date because of cloud outage... maybe it will resolve now?19:53
weshay|ruckya infra!19:54
weshay|ruckI mean.. yay infra!19:54
*** holser___ has joined #tripleo20:01
*** lifeless has quit IRC20:02
*** lifeless has joined #tripleo20:03
*** dxiri has quit IRC20:05
*** holser__ has quit IRC20:05
*** dxiri has joined #tripleo20:08
*** ooolpbot has joined #tripleo20:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION20:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/177716820:10
*** ooolpbot has quit IRC20:10
openstackLaunchpad bug 1777168 in tripleo "Failures to build python-networking-ovn" [Critical,Triaged]20:10
*** social has quit IRC20:13
openstackgerritwes hayutin proposed openstack/tripleo-quickstart-extras master: WIP: use the tq ansible.cfg for the undercloud install  https://review.openstack.org/57553520:14
openstackgerritMerged openstack/tripleo-heat-templates master: scen007 is failing in the gate  https://review.openstack.org/57579820:18
*** owalsh has quit IRC20:20
*** owalsh has joined #tripleo20:20
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart master: DNM - Test change  https://review.openstack.org/57583720:21
openstackgerritEmilien Macchi proposed openstack/tripleo-quickstart-extras master: Use the in-built image prepare for undercloud install  https://review.openstack.org/57161320:24
openstackgerritEmilien Macchi proposed openstack/tripleo-quickstart-extras master: Switch to workflow driven image prepare  https://review.openstack.org/57347620:24
openstackgerritEmilien Macchi proposed openstack/tripleo-quickstart-extras master: Use role tripleo-modify-image to update containers  https://review.openstack.org/57163120:24
openstackgerritEmilien Macchi proposed openstack/tripleo-quickstart-extras master: Switch to workflow driven image prepare  https://review.openstack.org/57347620:24
EmilienMweshay|ruck: let's try a revert of https://review.openstack.org/#/c/575798/ and a patch that kick off scenario00720:26
EmilienMsee if it works with the patch in the distgit20:26
weshay|ruckk20:26
*** tcw has quit IRC20:30
weshay|ruckEmilienM, this is all we need to test https://review.openstack.org/#/c/575843/20:31
fultonjdoes anyone know anything about /opt/stack/new/tripleo-ci/test-environments/worker-config.yaml in rdo CI?20:31
fultonjhttps://logs.rdoproject.org/73/570473/2/openstack-check/gate-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-pike/Zdba5acc45f3940fe8bebbe2bf00883b4/undercloud/home/jenkins/overcloud-deploy.sh20:32
fultonj^ context20:32
fultonjit's being based in oooq's featureset024 but i can't see it so i don't know what it does20:33
fultonjs/based/passed20:33
*** tcw has joined #tripleo20:33
fultonjthis is an OVB job20:33
EmilienMweshay|ruck: oh wait20:34
EmilienMit's pointless20:34
weshay|ruckfultonj, am I blind, don't see it in fs2420:34
EmilienMah nevermind20:34
EmilienMweshay|ruck: nevermind what I said. Thanks it's good20:35
fultonjweshay|ruck: https://review.openstack.org/#/c/570469/20:35
fultonjin rdo ci20:35
weshay|ruckfultonj, worker-config.yaml is not in that dep20:36
weshay|ruckis that the issue?20:36
weshay|ruckwhy is it getting passed20:37
fultonjweshay|ruck: i am testing a change to fs24 by updating tht20:37
fultonjhttps://review.openstack.org/#/c/570473/20:37
weshay|rucktoci_gate_test-orig.sh:374:    OVERCLOUD_DEPLOY_ARGS="$OVERCLOUD_DEPLOY_ARGS -e $TRIPLEO_ROOT/tripleo-ci/test-environments/worker-config.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/low-memory-usage.yaml"20:37
fultonjweshay|ruck: so that source caused this overcloud-deploy command https://logs.rdoproject.org/73/570473/2/openstack-check/gate-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-pike/Zdba5acc45f3940fe8bebbe2bf00883b4/undercloud/home/jenkins/overcloud-deploy.sh20:38
weshay|ruckya.. I see it there20:39
fultonjweshay|ruck: is worker-config.yaml somewhere i can read it so i can see what it might be doing?20:39
weshay|ruckafaict, it should not be20:39
weshay|ruckhttps://github.com/openstack-infra/tripleo-ci/blob/master/test-environments/worker-config.yaml20:40
fultonjahhh, it's in infra20:40
fultonjthanks weshay|ruck20:40
mwhahahaweshay|ruck: did we ever fix tripleo-docs so it runs stuff?20:40
weshay|ruckah here it is http://git.openstack.org/cgit/openstack-infra/tripleo-ci/tree/toci-quickstart/config/testenv/multinode.yml#n720:40
weshay|ruckheh..20:41
weshay|ruckit's been in there for a year20:41
weshay|ruckhttps://github.com/openstack-infra/tripleo-ci/commit/ab7914d28de5cc79269d3839de10f6017e4c8c1320:42
weshay|ruckpile on ci day20:42
weshay|ruckpi day20:42
weshay|ruckor just a day20:42
mwhahahaaka day ending in y20:42
mwhahahanow get back to work20:42
fultonji'm trying to make fs24 use ceph-ansible but the ceph-storage nodes don't have docker running.20:45
weshay|ruckmwhahaha, so looking, it's missing zuul config entirely20:45
weshay|ruckafaict20:45
openstackgerritMerged openstack/tripleo-common master: scen007 is failing in the gate  https://review.openstack.org/57579620:46
openstackgerritPradeep Kilambi proposed openstack/puppet-tripleo master: Add cors class to gnocchi profile  https://review.openstack.org/57584520:46
mwhahahaweshay|ruck: yea, i'm poking at it as well20:46
fultonji'm looking for evidence that there's a custom role20:46
openstackgerritPradeep Kilambi proposed openstack/puppet-tripleo master: Include cors to gnocchi profile  https://review.openstack.org/57584520:47
weshay|ruckmwhahaha, docs for openstack/tripleo-quickstart is working w/o any specfic config20:47
mwhahahaweshay|ruck: it's likely that project-config is bad20:48
openstackgerritwes hayutin proposed openstack/tripleo-quickstart master: DNM test doc  https://review.openstack.org/57584620:48
*** lifeless has quit IRC20:49
mwhahahaweshay|ruck: i think it's the files stuff in project-config20:49
mwhahahaweshay|ruck: we're limiting stuff when i don't think we should be20:49
openstackgerritwes hayutin proposed openstack/tripleo-docs master: DNM testing doc  https://review.openstack.org/57584820:49
*** lifeless has joined #tripleo20:50
mwhahahaweshay|ruck: i think it's https://review.openstack.org/57584920:51
weshay|ruckmwhahaha, I blame u20:52
mwhahahapffft it was like that when i got here20:52
* mwhahaha blames everyone else20:52
*** mhenkel_ has joined #tripleo20:52
openstackgerritMarius Cornea proposed openstack/tripleo-upgrade stable/queens: Fix docker images validation script  https://review.openstack.org/57546120:52
weshay|ruckthat's what we need is a docker validation script20:53
mwhahahaso salty20:54
*** rmascena has quit IRC20:54
openstackgerritwes hayutin proposed openstack/tripleo-docs master: DNM testing doc  https://review.openstack.org/57584820:56
fultonjweshay|ruck: another way to ask my question is, why do I need to start docker manually with a preboot script to run ceph containers when they should already be included in the role? https://review.openstack.org/#/c/570473/2/ci/environments/dev_block_devices.sh20:57
weshay|ruckmwhahaha, job kicked https://review.openstack.org/#/c/575848/220:57
weshay|ruckhrm.. then died20:58
mwhahahayea i think it's the files20:58
mwhahahaso unless you touch one of the ones that was listed, no jobs20:58
weshay|ruckya20:58
weshay|ruckI have a dep on your change though20:58
EmilienMnice, an opportunity to blame mwhahaha for once20:58
EmilienMhe always blames me :(20:59
mwhahahafultonj: the docker bits should be covered by a the docker service on the role20:59
weshay|ruckfultonj, hrm.. you should not have to start docker like that20:59
weshay|ruckfor shiz20:59
fultonjmwhahaha: exactly!20:59
mwhahahafultonj: can you point me to what you're looking at?20:59
fultonjmwhahaha: so i thought maybe the service was removed from the ceph storage role21:00
fultonjhttps://logs.rdoproject.org/73/570473/2/openstack-check/gate-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-pike/Zdba5acc45f3940fe8bebbe2bf00883b4/undercloud/home/jenkins/overcloud-deploy.sh21:00
openstackgerritEmilien Macchi proposed openstack/tripleo-quickstart master: Revert "Revert "Switch scenario00{1,2,3,4,7} to containerized undercloud""  https://review.openstack.org/57533021:00
fultonj^ so i looked at how the deployment was done to see if there's "-e no-docker-for-ceph.yaml" or something like it21:01
fultonjtwo reviews21:01
mwhahahafultonj: you know that's for pike right?21:01
fultonjhttps://review.openstack.org/#/q/topic:bug/1772034+(status:open+OR+status:merged)21:01
fultonjmwhahaha: yes21:01
mwhahahaso you're trying to switch it in pike?21:01
fultonji want to change the pike job to ceph-ansible21:02
mwhahahawe didn't support ceph-ansible in pike did we?21:02
fultonjwe did21:02
fultonjso that i can test an upgrade of ceph w/ 3 mons to queens21:02
fultonjfs24 is one of the few jobs w/ 3 controllers21:02
mwhahahain ovb we use the regular roles21:02
fultonjyeah, AFAICT it's using regular roles21:03
fultonjand no role changing things done via a -e21:03
fultonjceph-ansible was introduced in pike21:03
fultonjfs24, the rdo job w/ 3 mons to test upgrades from pike to queens still uses puppet-ceph21:04
mwhahahafultonj: so docker should be managed via https://github.com/openstack/tripleo-heat-templates/blob/stable/pike/roles_data.yaml#L288 but it might be an ordering thing21:04
fultonjso i'm trying to switch it21:04
fultonjinteresting...21:04
mwhahahathough docker should be up in step121:04
mwhahahamy thought would be that the external workflow gets kicked off before the node gets anything run but that seems od21:05
weshay|ruckEmilienM, so the most recent container runs in prep look good21:05
fultonjthe external workflow waits until step221:05
weshay|ruckupdating a few rpms.. working etc21:05
*** gkadam_ has quit IRC21:05
mwhahahafultonj: so why doesn't the deploy fail?21:07
mwhahahafultonj: it seems to just fail on the validation21:07
fultonjyou mean at step1 if it fails to bring up docker?21:07
mwhahahafultonj: no i mean ceph ansible fails with docker?21:07
mwhahahafultonj: shouldn't the deployment fail if that was the case?21:07
mwhahahabut the deployment is successful21:07
fultonjthe deply used to fail until i added a hack to start it manually21:08
fultonjthen evertyting is fine21:08
mwhahahaplz don't add a hack, make it hard to debug :D21:08
fultonjmwhahaha: i only added a hack to prove my point that the root cause is docker not running21:08
fultonjthe hack will go21:08
fultonjline 2221:08
fultonjhttps://review.openstack.org/#/c/570473/2/ci/environments/dev_block_devices.sh21:08
fultonjeasy to see it fail21:09
fultonji can remove line 22 and we can see it fail the old way21:09
fultonji was just proving it was tripleo not starting docker, as opposed to ceph-ansible having some other issue21:10
*** ooolpbot has joined #tripleo21:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION21:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/177716821:10
*** ooolpbot has quit IRC21:10
openstackLaunchpad bug 1777168 in tripleo "Failures to build python-networking-ovn" [Critical,Triaged]21:10
*** brault has quit IRC21:10
mwhahahafultonj: so do you have link to the logs for a failure w/o the hack?21:10
fultonjmwhahaha: i think results of ps1 (without the hack) would be gone after ps2 was tested21:11
mwhahahabecause the docker was started on it's own https://logs.rdoproject.org/73/570473/2/openstack-check/gate-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-pike/Zdba5acc45f3940fe8bebbe2bf00883b4/overcloud-cephstorage-0/var/log/journal.txt.gz#_May_30_22_12_4121:11
*** pradk has quit IRC21:11
fultonji will make ps3 without the hack so we can get some logs21:11
fultonjmwhahaha: sound like ok next step?21:11
mwhahahalet me look at teh ps1 logs21:11
fultonjahhh21:11
fultonjhttps://logs.rdoproject.org/73/570473/1/openstack-check/gate-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-pike/Zb2bfcd4b6e9d49d9aa733634eeb1c9cd/undercloud/var/log/mistral/ceph-install-workflow.log.txt.gz21:12
fultonjwe do have ps1 logs21:12
openstackgerritBen Nemec proposed openstack/tripleo-heat-templates master: Fix DNS firewall rules  https://review.openstack.org/57514621:12
openstackgerritBen Nemec proposed openstack/tripleo-heat-templates master: Make BIND listen address configurable  https://review.openstack.org/57582221:12
openstackgerritBen Nemec proposed openstack/tripleo-heat-templates master: Make BIND /var dir persistent  https://review.openstack.org/57582321:12
mwhahahafultonj: can't connect doesn't mean it's not running21:12
mwhahahafultonj: could be a docker permissions thing21:12
fultonjbut starting it as root made it run...21:13
* fultonj clicking around https://logs.rdoproject.org/73/570473/1/openstack-check/gate-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-pike/Zb2bfcd4b6e9d49d9aa733634eeb1c9cd/overcloud-cephstorage-0/var/log/extra/?C=N;O=A 21:14
*** agopi has quit IRC21:14
*** agopi has joined #tripleo21:15
mwhahahafultonj: so this is ovb, docker should already be installed21:15
*** rfolco has quit IRC21:15
fultonjconfirmed it's isntalled21:15
fultonje.g. in rpm-list.txt.gz21:15
*** yamahata has joined #tripleo21:16
mwhahahafultonj: so step1 ran21:16
fultonjyou'd think i would see it in pstree if it were running21:16
mwhahahahttps://logs.rdoproject.org/73/570473/1/openstack-check/gate-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-pike/Zb2bfcd4b6e9d49d9aa733634eeb1c9cd/overcloud-cephstorage-0/var/log/journal.txt.gz#_May_24_23_57_1721:17
mwhahahathat's the start of puppet step121:17
mwhahahalet me see where that is supposed to start21:17
*** slaweq has quit IRC21:18
*** slaweq has joined #tripleo21:19
mwhahahafultonj: so yea i don't see a reference to Service[docker] anywhere21:19
mwhahahathat seems odd21:19
*** holser___ has quit IRC21:21
*** jtomasek_ has quit IRC21:21
* fultonj nods21:21
mwhahahathough i don't see where it gets started on the controller either21:23
*** slaweq has quit IRC21:24
fultonjthat's a good point21:24
fultonjone thing odd about this job is that it's doing a '-e puppet-pacemaker.yaml'21:24
*** akrivoka has quit IRC21:24
fultonjseems like a blast from the past that shouldn't be in pike, unless I am misremembering21:24
mwhahahathats normal21:24
fultonjis it?21:24
mwhahahayea it's an ovb job21:25
mwhahahawe ran ha on those21:25
fultonjshouldn't this job have docker-ha too?21:25
mwhahahafultonj: uh, i don't think job is containerized21:26
fultonjright!21:26
fultonjhttps://logs.rdoproject.org/73/570473/2/openstack-check/gate-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-pike/Zdba5acc45f3940fe8bebbe2bf00883b4/undercloud/home/jenkins/overcloud-deploy.sh21:26
fultonjno -e docker.yaml!21:26
fultonjomg21:26
fultonjdahhhhhhhhhh21:27
fultonjthank you mwhahaha21:27
fultonjnow i know what to do21:27
fultonjcontainerize it as i was going to21:27
mwhahahaalso -e docker-ha.yaml21:27
fultonjexactly21:27
mwhahahaHEH21:27
fultonjwe want our pike CI to be containerized right...21:28
fultonjthis job21:28
mwhahahasome of it, yea21:28
mwhahahabecause we were not completely containerized upstream21:28
mwhahahaso only some of the jobs were switched over21:28
mwhahahafultonj: so we usualy don't cut over a fs to containers i thought we created new ones21:29
mwhahahafultonj: i'd have to defer back to EmilienM as to wtf we were doing around that21:29
*** edmondsw has quit IRC21:29
fultonjhmmmm21:29
openstackgerritMerged openstack/tripleo-quickstart master: scen007 is failing in the gate  https://review.openstack.org/57579921:30
*** ipsecguy has quit IRC21:33
*** ipsecguy has joined #tripleo21:34
fultonjmwhahaha: thanks for helping me figure this out.  i will continue on this and get input from EmilienM before it merges by adding him to the review21:34
* fultonj stepping out for now21:34
EmilienMweshay|ruck: is your dashboard up again?21:44
EmilienMI want to check the failures21:44
weshay|ruckyes21:44
EmilienMfultonj: sorry what's the patch? I can look now21:45
EmilienMhttp://38.145.34.131:3000/d/pgdr_WVmk/ruck-rover?orgId=121:45
weshay|ruckEmilienM, everything merged.. I thought the same thing21:45
EmilienMI have it21:45
EmilienMweshay|ruck: ^21:45
weshay|ruckstill at 13 fails21:45
weshay|ruckit didn't send all the merge messages to this channel21:46
EmilienMweshay|ruck: 13 fails?21:46
EmilienMweshay|ruck: we need to count from when infra pushed the new proxy21:46
weshay|ruckEmilienM, 021:47
EmilienM0 is a good number so far :D21:48
EmilienMI think there is a new problem coming up21:51
EmilienMhttp://logs.openstack.org/27/572027/3/check/tripleo-ci-centos-7-containerized-undercloud-upgrades/10d0022/logs/undercloud/home/zuul/undercloud_upgrade.log.txt.gz#_2018-06-15_19_43_1921:51
EmilienMI can't believe it wasn't catch by https://review.openstack.org/#/c/571500/21:52
EmilienMit's hitting the gate as well21:52
EmilienMI'm about to propose a revert and a fix21:52
*** dbecker has quit IRC21:53
*** mcornea has quit IRC21:53
openstackgerritEmilien Macchi proposed openstack/tripleo-quickstart master: Revert "Revert "Revert "fs050: enable upgrade cleanup"""  https://review.openstack.org/57585521:54
weshay|ruckwhen did their patch merge ?21:54
openstackgerritEmilien Macchi proposed openstack/tripleo-quickstart master: Revert "Revert "Revert "fs050: enable upgrade cleanup"""  https://review.openstack.org/57585521:54
weshay|ruckand get pushed in21:54
EmilienMweshay|ruck: which one21:54
weshay|ruck2hr ago?21:54
EmilienMweshay|ruck: oh infra?21:54
weshay|ruckya21:54
EmilienMlike 2.5h ago yea21:54
EmilienMweshay|ruck, mwhahaha ^ can I get quick +A on my revert21:55
EmilienMweshay|ruck: when you +A I'll ask infra to move it to gate directly21:56
EmilienMI see a lot of failures in check with that job21:56
*** dmacpher has quit IRC21:57
EmilienMI'll go ahead and approve it21:57
weshay|ruckEmilienM, this one? https://review.openstack.org/#/c/575330/21:58
EmilienMweshay|ruck: it's fine, it was https://review.openstack.org/57585521:59
EmilienMyou got it21:59
openstackgerritAlex Schultz proposed openstack/tripleo-docs master: Document single nic configuration for standalone  https://review.openstack.org/57585922:00
mwhahahaEmilienM: -^22:00
*** agopi has quit IRC22:00
mwhahahasome day we'll get the tripleo-docs jobs back and we can merge that22:00
EmilienMnice22:00
*** lifeless has quit IRC22:01
openstackgerritAlex Schultz proposed openstack/tripleo-docs master: Document single nic configuration for standalone  https://review.openstack.org/57585922:02
*** owalsh has quit IRC22:07
openstackgerritAlex Schultz proposed openstack/tripleo-heat-templates master: Update standalone role for a cloud  https://review.openstack.org/57586222:08
*** owalsh has joined #tripleo22:09
*** ooolpbot has joined #tripleo22:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION22:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/177716822:10
*** ooolpbot has quit IRC22:10
openstackLaunchpad bug 1777168 in tripleo "Failures to build python-networking-ovn" [Critical,Triaged]22:10
mwhahahaanyone remember the openstack command to attach a volume to a server :D22:10
* mwhahaha can never remember these things22:10
mwhahahaor is this one of those things that you still have to use the nova command for22:11
mwhahahanm openstack serfver add volume22:11
*** slaweq has joined #tripleo22:11
EmilienMoh I think I found the cont/uc upgrade issue22:11
EmilienMmwhahaha: use horizon :P22:11
mwhahahaPFFFFT22:12
mwhahaha(but it does work, i just didn't want to setup sshuttle)22:12
EmilienMmwhahaha: could you spawn and ping a vm using horizon at the end?22:12
mwhahahayea22:12
EmilienMniceeeee22:12
mwhahahawith that doc update and that role patch -^ it's an openstack cloud in a box in ~35mins22:13
mwhahahanova/neutron/cinder/glance/swift22:13
mwhahahaall working22:13
mwhahahano ceph tho22:13
* mwhahaha shuns ceph22:13
EmilienMpretty awesome22:13
mwhahahaSHUUUUUUUNNNNN22:13
EmilienMwe can pretty much switch scenarios on that once we have tempest working22:13
* EmilienM hides22:13
mwhahahaconfig file first22:14
EmilienMthe immediate benefit is runtime in CI22:14
mwhahahawe need to ditch that doc bit22:14
mwhahahait's ugly22:14
*** rlandy_ is now known as rlandy22:15
mwhahahacinder worked22:16
mwhahahawooo22:16
mwhahahai think i'm going to give up for teh week22:16
EmilienMwait22:18
mwhahahai will not wait22:18
EmilienMdon't let me here22:18
mwhahahaIT'S MY LIFE WOOOO22:18
mwhahahawell i'm going to send an email first22:18
EmilienMok go now22:18
EmilienMthat's too much22:18
EmilienMyeah send an email22:18
EmilienMpeople can start using it imho22:18
EmilienMi've got the skydive folks poking at it today as well22:19
*** owalsh has quit IRC22:19
*** slaweq has quit IRC22:19
*** owalsh has joined #tripleo22:19
EmilienMoh I know why the upgrade job is failing22:20
EmilienMit's because we aren't consistent22:20
mwhahaha:(22:20
EmilienMtherefore we need https://github.com/openstack/tripleo-heat-templates/commit/029f500feedd01c12a2e9b24d4bab8f2a6c8beed22:20
*** cdearborn has quit IRC22:20
EmilienMlet's merge the revert anyway22:21
EmilienMand let's see why we aren't consistent22:21
EmilienMhttps://git.openstack.org/cgit/openstack/python-cloudkittyclient/commit/?id=d070f6a68cddf51c57e77107f1b823a8f75770ba22:22
EmilienMlol22:22
EmilienMthat kind of patch makes me sad22:22
EmilienM113 files changed, 4231 insertions, 8399 deletions22:22
EmilienMpeople need to learn what iteration means :D22:23
mwhahahaiterwhat?22:23
EmilienMamoralej|off, jpena|off here by any chance?22:24
EmilienMI'll take a stab otherwise22:24
*** yamahata has quit IRC22:24
*** mhenkel_ has quit IRC22:29
mwhahahaEmilienM: did you restore all the rest of the stuff or do we need your upgrade revert first22:38
EmilienMmwhahaha: I'll take care of it22:39
mwhahahak22:39
EmilienMyeah we need my revert22:39
EmilienMsorry for that22:39
*** matbu has joined #tripleo22:39
EmilienMcoz we aren't consistent22:39
EmilienMI'm working on it22:39
EmilienM(cloudkittyclient)22:39
*** mhenkel_ has joined #tripleo22:41
*** mhenkel_ has quit IRC22:45
*** lifeless has joined #tripleo22:49
*** Goneri has quit IRC22:50
*** edmondsw has joined #tripleo22:50
*** edmondsw has quit IRC22:55
*** ooolpbot has joined #tripleo23:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION23:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/177716823:10
*** ooolpbot has quit IRC23:10
openstackLaunchpad bug 1777168 in tripleo "Failures to build python-networking-ovn" [Critical,Triaged]23:10
*** slaweq has joined #tripleo23:11
*** hamzy_ has quit IRC23:15
*** slaweq has quit IRC23:16
*** owalsh_ has joined #tripleo23:16
*** owalsh has quit IRC23:17
*** owalsh has joined #tripleo23:17
*** owalsh has quit IRC23:18
*** owalsh_ is now known as owalsh23:19
*** hamzy has joined #tripleo23:25
*** rpioso is now known as rpioso|afk23:36
*** ipsecguy has quit IRC23:39
*** tosky has quit IRC23:41
*** tahini has quit IRC23:54
*** tahini has joined #tripleo23:57
*** agopi has joined #tripleo23:58

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!