Friday, 2016-08-12

*** zhurong has quit IRC00:01
*** ddieterly has quit IRC00:03
kevinbentontonyb: hey, can you take another look at https://review.openstack.org/#/c/351081/ . had to resolve merge conflict with oslo.contexrt00:03
*** testing1234 has quit IRC00:03
*** kragniz1 has joined #openstack-infra00:03
tonybkevinbenton: looking ...00:04
*** itsuugo has quit IRC00:04
tonybkevinbenton: +2'd now for testing ;P00:06
*** spzala has quit IRC00:06
kevinbentontonyb: so long for just a little change :)00:07
*** spzala has joined #openstack-infra00:07
*** sflanigan has joined #openstack-infra00:10
*** sflanigan has joined #openstack-infra00:10
*** spzala has quit IRC00:11
*** SumitNaiksatam has quit IRC00:15
*** baoli has joined #openstack-infra00:17
openstackgerritPaul Belanger proposed openstack-infra/project-config: Use local AFS mirrors to build tripleoclient images  https://review.openstack.org/35445100:17
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Change deprecated 'warn' to 'warning' method  https://review.openstack.org/35445600:20
*** Swami has quit IRC00:20
openstackgerritMathieu Gagné proposed openstack-infra/grafyaml: Add datasource option to more elements  https://review.openstack.org/30918100:20
*** xarses_ has joined #openstack-infra00:21
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Change deprecated 'warn' to 'warning' method  https://review.openstack.org/35445600:21
*** gouthamr has joined #openstack-infra00:23
*** Julien-zte has joined #openstack-infra00:23
*** xarses has quit IRC00:24
*** amitgandhinz has joined #openstack-infra00:25
*** fguillot_ has quit IRC00:25
*** spzala has joined #openstack-infra00:26
*** fguillot_ has joined #openstack-infra00:27
*** karthik__ has joined #openstack-infra00:27
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Change deprecated 'warn' to 'warning' method  https://review.openstack.org/35445600:27
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Change deprecated 'warn' to 'warning' method  https://review.openstack.org/35445600:29
*** _sarob has quit IRC00:30
*** amitgandhinz has quit IRC00:31
*** ddieterly has joined #openstack-infra00:32
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Change deprecated 'warn' to 'warning' method  https://review.openstack.org/35445600:33
*** adrian_otto has quit IRC00:34
*** rajinir has quit IRC00:35
*** maishsk_ has joined #openstack-infra00:35
*** maishsk has quit IRC00:36
*** maishsk_ is now known as maishsk00:36
openstackgerritMerged openstack-infra/project-config: Add libs to project list for shade libs test  https://review.openstack.org/35438000:36
*** kzaitsev_mb has quit IRC00:37
*** hamzy has joined #openstack-infra00:38
openstackgerritMerged openstack-infra/gerritbot: Add gerrit stream over MQTT support to gerritbot  https://review.openstack.org/35065100:39
tonybkevinbenton: Yeah it arrived during a period of chnage for the requirement team00:40
*** karthik__ has quit IRC00:40
*** raunak has quit IRC00:40
*** karthik__ has joined #openstack-infra00:42
*** pvaneck has quit IRC00:44
*** tphummel has quit IRC00:44
*** sdake has joined #openstack-infra00:46
*** fguillot_ has quit IRC00:50
*** fguillot_ has joined #openstack-infra00:50
*** karthi___ has joined #openstack-infra00:53
*** davidlenwell has quit IRC00:53
*** karthik__ has quit IRC00:54
*** ddieterly has quit IRC00:56
*** psilvad has quit IRC01:01
*** rfolco has quit IRC01:02
*** karthi___ has quit IRC01:03
*** zhurong has joined #openstack-infra01:05
*** raunak has joined #openstack-infra01:05
*** davidlenwell has joined #openstack-infra01:05
*** gouthamr_ has joined #openstack-infra01:06
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Change deprecated 'warn' to 'warning' method  https://review.openstack.org/35445601:09
*** gouthamr has quit IRC01:10
*** tonytan4ever has joined #openstack-infra01:11
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Change deprecated 'warn' to 'warning' method  https://review.openstack.org/35445601:11
*** baoli has quit IRC01:13
*** ddieterly has joined #openstack-infra01:14
*** gouthamr_ is now known as gouthamr01:14
*** Apoorva has quit IRC01:16
*** tonytan4ever has quit IRC01:16
*** nwkarsten has joined #openstack-infra01:16
openstackgerritChangcheng Intel proposed openstack-infra/jenkins-job-builder: add compress-log option to compress log  https://review.openstack.org/35413801:16
*** rpodolyaka_ has joined #openstack-infra01:18
*** esberglu has quit IRC01:18
*** yanyanhu has joined #openstack-infra01:18
*** Julien-zte has quit IRC01:19
openstackgerritHenry Gessau proposed openstack-infra/project-config: Use python-db-jobs for networking-sfc  https://review.openstack.org/35435801:19
*** nwkarste_ has joined #openstack-infra01:19
*** Apoorva has joined #openstack-infra01:20
*** nwkarsten has quit IRC01:20
*** nwkarsten has joined #openstack-infra01:21
*** rpodolyaka has quit IRC01:21
*** rpodolyaka_ is now known as rpodolyaka01:21
*** Julien-zte has joined #openstack-infra01:22
*** fguillot_ has quit IRC01:22
*** Apoorva has quit IRC01:23
*** nwkarste_ has quit IRC01:23
*** Apoorva has joined #openstack-infra01:23
*** rhallisey has joined #openstack-infra01:24
*** nwkarsten has quit IRC01:25
*** tqtran has quit IRC01:27
*** amitgandhinz has joined #openstack-infra01:27
*** baoli has joined #openstack-infra01:29
*** adrian_otto has joined #openstack-infra01:32
*** amitgandhinz has quit IRC01:32
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Add version-number wrapper functionality  https://review.openstack.org/23898401:41
openstackgerritMerged openstack-dev/pbr: Fix typo in the index.rst  https://review.openstack.org/35388901:42
*** adrian_otto has quit IRC01:43
*** sdake has quit IRC02:01
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: Add temporary pin to last known good current tripleo repo  https://review.openstack.org/35448102:01
*** rhallisey has quit IRC02:01
*** yamahata has quit IRC02:02
*** PollyZ has joined #openstack-infra02:04
*** nwkarsten has joined #openstack-infra02:04
*** yamahata has joined #openstack-infra02:07
openstackgerritMerged openstack-infra/zuul: Correct default value for hold-following-changes  https://review.openstack.org/35072502:09
*** sdake has joined #openstack-infra02:13
*** ddieterly has quit IRC02:14
*** Apoorva_ has joined #openstack-infra02:14
*** Apoorva_ has quit IRC02:15
*** raunak has quit IRC02:15
*** gongysh has joined #openstack-infra02:17
*** gyee has quit IRC02:17
*** Apoorva has quit IRC02:17
*** tphummel has joined #openstack-infra02:19
*** gildub has joined #openstack-infra02:19
*** woodster_ has quit IRC02:19
openstackgerritMerged openstack-infra/zuul: Tests: Improve support in fake builds  https://review.openstack.org/35072602:20
openstackgerritMerged openstack-infra/zuul: Tests: ensure fake builds recorded in order  https://review.openstack.org/35189702:22
*** aimeeu__ has joined #openstack-infra02:22
openstackgerritEmilien Macchi proposed openstack-infra/project-config: puppet/tripleo: restore ovb stable jobs  https://review.openstack.org/35417202:22
openstackgerritMerged openstack-infra/zuul: Restore test_parallel_changes  https://review.openstack.org/35072702:22
openstackgerritMerged openstack-infra/zuul: Remove unused Inheritable class  https://review.openstack.org/35072802:22
openstackgerritMerged openstack-infra/zuul: Tests: remove FakeWorker  https://review.openstack.org/35073002:23
*** aimeeu has quit IRC02:25
*** aimeeu__ is now known as aimeeu02:25
*** sarob has joined #openstack-infra02:27
*** amitgandhinz has joined #openstack-infra02:28
*** ddieterly has joined #openstack-infra02:29
*** itsuugo has joined #openstack-infra02:31
*** sarob has quit IRC02:32
*** amitgandhinz has quit IRC02:33
*** bswartz has quit IRC02:33
*** sdake has quit IRC02:34
*** bswartz has joined #openstack-infra02:37
*** shashank_hegde has quit IRC02:38
*** roxanaghe has joined #openstack-infra02:38
*** ddieterly has quit IRC02:45
*** roxanaghe has quit IRC02:46
*** sflanigan is now known as sflanigan_afk02:49
*** ayoung has joined #openstack-infra02:49
*** sdake has joined #openstack-infra02:54
*** betherly has quit IRC02:55
*** zhiyan has quit IRC02:55
*** mwhahaha has quit IRC02:55
*** tdurakov has quit IRC02:56
*** ameade has quit IRC02:56
*** jraim has quit IRC02:56
*** kien-ha has quit IRC02:59
*** betherly has joined #openstack-infra02:59
*** mwhahaha has joined #openstack-infra02:59
*** tdurakov has joined #openstack-infra03:00
*** amotoki has joined #openstack-infra03:02
*** senk_ has joined #openstack-infra03:02
*** zhiyan has joined #openstack-infra03:03
*** ameade has joined #openstack-infra03:03
*** jraim has joined #openstack-infra03:03
openstackgerritMatthew Treinish proposed openstack-infra/system-config: Add firehose.o.o to cacti  https://review.openstack.org/35448903:05
*** sflanigan_afk is now known as sflanigan03:08
*** PollyZ has quit IRC03:10
*** amotoki has quit IRC03:11
*** PollyZ has joined #openstack-infra03:11
*** raunak has joined #openstack-infra03:13
*** yamahata has quit IRC03:14
*** yamamoto has joined #openstack-infra03:17
openstackgerritMatthew Treinish proposed openstack-infra/elastic-recheck: Fix template links for split uncategorized  https://review.openstack.org/35449103:19
*** kushal has quit IRC03:20
*** amotoki has joined #openstack-infra03:22
*** adrian_otto has joined #openstack-infra03:24
*** sdake has quit IRC03:24
dmsimardNice, spam on launchpad. Is that new ? https://bugs.launchpad.net/nova/+bug/161188803:26
openstackLaunchpad bug 1611888 in OpenStack Compute (nova) "1844-887-9236 Quickbooks Activation Support Phone Number" [Undecided,New]03:26
*** amitgandhinz has joined #openstack-infra03:29
pleia2dmsimard: unfortunately it's been happening for some time, it's frequently the same ones who are hitting the wikis (all using lp auth)03:29
dmsimardAh, just noticed it for the first time I guess03:29
pleia2I think you can report the spam in #launchpad and they'll clean it up03:30
*** senk_ has quit IRC03:31
*** amotoki has quit IRC03:31
openstackgerritAdrian Turjak proposed openstack-infra/project-config: new project - StackTask  https://review.openstack.org/35381803:31
*** amitgandhinz has quit IRC03:33
*** links has joined #openstack-infra03:33
*** spzala has quit IRC03:35
*** gouthamr has quit IRC03:35
*** amotoki has joined #openstack-infra03:35
*** baoli has quit IRC03:43
*** ayoung has quit IRC03:45
*** raunak has quit IRC03:47
openstackgerritBrad Behle proposed openstack-infra/project-config: Rename ovn-native-l3 gate job to ovn-native-services  https://review.openstack.org/35267603:48
*** ramishra_ has joined #openstack-infra03:49
*** ramishra has quit IRC03:49
*** raunak has joined #openstack-infra03:49
*** kushal has joined #openstack-infra03:51
*** netsin has quit IRC03:53
*** csomerville has quit IRC03:53
*** senk_ has joined #openstack-infra03:54
*** sdake has joined #openstack-infra03:54
*** csomerville has joined #openstack-infra03:54
*** amotoki has quit IRC04:01
*** adrian_otto has quit IRC04:02
*** adrian_otto has joined #openstack-infra04:02
openstackgerritRoopak Parikh proposed openstack-infra/project-config: Adding first version of project mors  https://review.openstack.org/35195604:03
*** Julien-zte has quit IRC04:08
*** Sukhdev has joined #openstack-infra04:09
*** amotoki has joined #openstack-infra04:09
*** amotoki has quit IRC04:10
*** amotoki has joined #openstack-infra04:11
*** PollyZ has quit IRC04:11
*** cody-somerville has joined #openstack-infra04:11
*** adrian_otto has quit IRC04:11
*** tonytan4ever has joined #openstack-infra04:12
*** csomerville has quit IRC04:13
*** psachin has joined #openstack-infra04:15
*** nwkarsten has quit IRC04:16
*** tonytan4ever has quit IRC04:17
*** yamamoto_ has joined #openstack-infra04:18
*** yamamoto_ has quit IRC04:18
*** yamamoto has quit IRC04:18
*** yamamoto has joined #openstack-infra04:19
*** PollyZ has joined #openstack-infra04:20
*** YorikSar has quit IRC04:21
*** yamamoto has quit IRC04:23
*** nwkarsten has joined #openstack-infra04:26
*** gongysh has quit IRC04:26
*** YorikSar has joined #openstack-infra04:28
*** amitgandhinz has joined #openstack-infra04:29
*** nwkarsten has quit IRC04:30
*** nwkarsten has joined #openstack-infra04:32
*** amitgandhinz has quit IRC04:34
*** senk_ has quit IRC04:34
*** gongysh has joined #openstack-infra04:36
*** PollyZ has quit IRC04:41
*** mhayden has quit IRC04:51
*** Sukhdev has quit IRC04:54
*** mhayden has joined #openstack-infra04:54
*** yamamoto has joined #openstack-infra05:04
*** jaosorior has joined #openstack-infra05:04
*** tphummel has quit IRC05:05
*** mhayden has quit IRC05:06
*** AJaeger has joined #openstack-infra05:10
AJaegerinfra-root, internap and rax-ord still have old packages - I'd love to have current ones with the bindep.txt change in (osic, ovh are fine). Could you check image upload to internap and rax-ord and upload new images manually, please?05:11
openstackgerritIan Wienand proposed openstack-infra/system-config: Add a short note on removing mirror from AFS  https://review.openstack.org/35437105:13
*** mriedem has quit IRC05:14
openstackgerritIan Wienand proposed openstack-infra/system-config: Refactor mariadb mirror into 2 repos  https://review.openstack.org/35420305:14
*** moravec has quit IRC05:15
*** moravec has joined #openstack-infra05:16
*** IlyaG has quit IRC05:16
*** netsin has joined #openstack-infra05:19
*** nwkarsten has quit IRC05:20
*** nwkarsten has joined #openstack-infra05:21
*** dkehn_ has quit IRC05:21
*** dkehn has quit IRC05:22
*** nwkarsten has quit IRC05:25
*** dkehn has joined #openstack-infra05:29
*** Julien-zte has joined #openstack-infra05:30
*** amitgandhinz has joined #openstack-infra05:30
openstackgerritJuan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: Include journal in the outputted logs  https://review.openstack.org/35453605:30
*** ggillies has quit IRC05:31
openstackgerritThomas Bechtold proposed openstack-infra/project-config: designate: Add a non-voting job with postgres as DB backend  https://review.openstack.org/35414105:31
*** sdake has quit IRC05:32
*** AJaeger has quit IRC05:32
*** raunak has quit IRC05:33
*** coolsvap has joined #openstack-infra05:34
*** amitgandhinz has quit IRC05:35
*** spzala has joined #openstack-infra05:35
*** jerryz has quit IRC05:38
*** senk_ has joined #openstack-infra05:39
*** spzala has quit IRC05:40
*** amotoki_ has joined #openstack-infra05:40
*** dkehn_ has joined #openstack-infra05:40
*** amotoki has quit IRC05:43
*** IlyaG has joined #openstack-infra05:44
*** r-mibu_ has quit IRC05:46
*** r-mibu has joined #openstack-infra05:46
openstackgerritIan Wienand proposed openstack-infra/system-config: Refactor mariadb mirror into 2 repos  https://review.openstack.org/35420305:46
*** Hal has joined #openstack-infra05:49
ianwAJaeger: hmm, seems like todays builds uploaded?05:49
openstackgerritKrzysztof Szukiełojć proposed openstack-infra/project-config: Add repo for murano-pkg-check. Murano package validator tool.  https://review.openstack.org/35386105:50
*** rcernin has quit IRC05:51
*** tphummel has joined #openstack-infra05:51
*** tphummel has quit IRC06:00
*** IlyaG has quit IRC06:00
*** ggnel_t has quit IRC06:04
*** tqtran has joined #openstack-infra06:04
*** armax has quit IRC06:05
*** tqtran has quit IRC06:08
openstackgerritMerged openstack-infra/project-config: Add gpg signing job for reprepro  https://review.openstack.org/34333606:08
*** shashank_hegde has joined #openstack-infra06:14
*** IlyaG has joined #openstack-infra06:17
*** rcernin has joined #openstack-infra06:21
*** _nadya_ has joined #openstack-infra06:21
*** e0ne has joined #openstack-infra06:25
openstackgerritKrzysztof Szukiełojć proposed openstack-infra/project-config: Add repo for murano-pkg-check. Murano package validator tool.  https://review.openstack.org/35386106:26
*** furlongm has quit IRC06:28
*** pgadiya has joined #openstack-infra06:30
*** ggnel_t has joined #openstack-infra06:30
*** pgadiya has quit IRC06:31
*** amitgandhinz has joined #openstack-infra06:31
*** furlongm has joined #openstack-infra06:32
*** IlyaG has quit IRC06:34
*** rbuzatu has joined #openstack-infra06:34
*** amitgandhinz has quit IRC06:35
*** camunoz has quit IRC06:37
*** IlyaG has joined #openstack-infra06:37
*** aeng has quit IRC06:38
*** e0ne has quit IRC06:40
*** kaisers has left #openstack-infra06:42
*** e0ne has joined #openstack-infra06:43
*** _nadya__ has joined #openstack-infra06:44
*** _nadya_ has quit IRC06:46
*** mrunge has quit IRC06:47
*** IlyaG has quit IRC06:48
*** _nadya__ has quit IRC06:49
*** _nadya_ has joined #openstack-infra06:57
*** IlyaG has joined #openstack-infra06:58
*** tesseract- has joined #openstack-infra06:59
openstackgerritAleksey Zvyagintsev proposed openstack-infra/project-config: Add repo for murano-pkg-check. Murano package validator tool.  https://review.openstack.org/35386107:01
*** _nadya_ has quit IRC07:02
*** IlyaG has quit IRC07:02
*** e0ne has quit IRC07:04
*** florianf has joined #openstack-infra07:06
*** daemontool has joined #openstack-infra07:06
*** oanson has joined #openstack-infra07:07
yolandagood morning07:07
*** martinkopec has joined #openstack-infra07:11
*** e0ne has joined #openstack-infra07:13
openstackgerritJuan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: Include journal in the outputted logs  https://review.openstack.org/35453607:16
*** yamahata has joined #openstack-infra07:20
*** maishsk has quit IRC07:20
*** e0ne has quit IRC07:22
*** AJaeger has joined #openstack-infra07:22
AJaegerianw: are you sure? Maybe it was too early, see http://logs.openstack.org/18/353918/1/check/gate-openstack-manuals-tox-checkniceness-ubuntu-xenial/69343ef/console.html07:23
AJaegerThat one was not checking for bindep.txt07:23
AJaegerand this one is rax-ord from two hours ago: http://logs.openstack.org/41/354141/5/check/gate-project-config-jenkins-project/6804d49/console.html - also not checking for bindep.txt07:24
*** tqtran has joined #openstack-infra07:25
*** maishsk has joined #openstack-infra07:25
*** tqtran_ has joined #openstack-infra07:26
*** vincentll has joined #openstack-infra07:26
*** nwkarsten has joined #openstack-infra07:28
*** AJaeger has quit IRC07:29
*** tqtran has quit IRC07:29
*** amitgandhinz has joined #openstack-infra07:32
*** nwkarsten has quit IRC07:33
*** markvoelker has quit IRC07:34
*** amitgandhinz has quit IRC07:36
*** jamielennox|away has quit IRC07:38
*** yaume has joined #openstack-infra07:43
*** pgadiya has joined #openstack-infra07:44
*** pgadiya has quit IRC07:44
*** amoralej|off is now known as amoralej07:44
*** sarob has joined #openstack-infra07:46
*** ifarkas_afk is now known as ifarkas07:46
*** jamielennox|away has joined #openstack-infra07:47
*** jamielennox|away is now known as jamielennox07:48
*** gildub_ has joined #openstack-infra07:48
*** jpich has joined #openstack-infra07:49
openstackgerritChangcheng Intel proposed openstack-infra/jenkins-job-builder: add compress-log option to compress log  https://review.openstack.org/35413807:49
*** gildub has quit IRC07:49
zhenguoyolanda: morning,  would you please help to look at this https://review.openstack.org/#/c/353575/ when you have time, thanks!07:50
yolandahi zhenguo , going through the review queue now07:50
openstackgerritYanyan Hu proposed openstack-infra/project-config: Enable zaqar for senlin integration test  https://review.openstack.org/35456607:50
*** sarob has quit IRC07:51
yolandaianw, around? i commented on the dib change07:51
ianwyolanda: hey, just for a little.  yeah, i think it might be better to just disable it07:52
*** matthewbodkin has joined #openstack-infra07:52
*** yaume has quit IRC07:53
yolandaand we do for all the operating systems that can be affected, as i see it can be a general need?07:53
*** mrmartin has joined #openstack-infra07:54
*** kushal has quit IRC07:55
*** yamahata has quit IRC07:55
ianwpossibly, i haven't looked at how the others do the cleanup... that was why i thought a generic name might not be as appropriate07:56
*** _nadya_ has joined #openstack-infra07:58
*** tonytan4ever has joined #openstack-infra07:59
*** zzzeek has quit IRC08:00
yolandaok i'll iterate over it today08:00
*** zzzeek has joined #openstack-infra08:01
*** _nadya_ has quit IRC08:02
*** _nadya_ has joined #openstack-infra08:03
*** maishsk has quit IRC08:03
*** tonytan4ever has quit IRC08:03
*** Goneri has joined #openstack-infra08:04
*** tqtran_ has quit IRC08:07
*** matrohon has joined #openstack-infra08:07
openstackgerritMatthew Bodkin proposed openstack-infra/storyboard-webclient: Tweak to the StoryBoard 'about' page Show where to file UX bugs with link Show where to file bugs in the API with link Make obvious what 'task tracking' includes Make clear that the user needs to login to be able to edit Grammar fix  https://review.openstack.org/35457008:08
*** tqtran has joined #openstack-infra08:09
*** shashank_hegde has quit IRC08:10
openstackgerritYanyan Hu proposed openstack-infra/project-config: Enable zaqar for senlin integration test  https://review.openstack.org/35456608:10
*** shardy has joined #openstack-infra08:11
openstackgerritChangcheng Intel proposed openstack-infra/jenkins-job-builder: add compress-log option to compress log  https://review.openstack.org/35413808:12
openstackgerritMerged openstack-infra/project-config: Move gate-tripleo-ci-centos-7-ovb-updates to experimental-tripleo  https://review.openstack.org/35444808:12
*** _nadya__ has joined #openstack-infra08:13
openstackgerritMerged openstack-infra/project-config: Create new project Nimble  https://review.openstack.org/35357508:13
openstackgerritAlexander Evseev proposed openstack-infra/system-config: Move local user definitions to hiera  https://review.openstack.org/35399308:13
*** martinkopec has quit IRC08:13
openstackgerritIan Wienand proposed openstack-infra/system-config: Refactor mariadb mirror into 2 repos  https://review.openstack.org/35420308:14
*** _nadya_ has quit IRC08:15
openstackgerritMerged openstack-infra/puppet-jenkins: Update jenkins repo URL + option to select version  https://review.openstack.org/35408608:15
*** lucas-dinner is now known as lucasagomes08:15
*** mrmartin has quit IRC08:18
openstackgerritMerged openstack-infra/project-config: Switch ironic-multinode job to use multitenant  https://review.openstack.org/35392108:20
openstackgerritMerged openstack-infra/project-config: Get IPA buildimage POST job into healthcheck  https://review.openstack.org/35180608:20
*** oanson has quit IRC08:21
*** tqtran has quit IRC08:22
*** amitkqed has quit IRC08:23
*** yaume has joined #openstack-infra08:24
*** amitkqed has joined #openstack-infra08:24
*** electrofelix has joined #openstack-infra08:28
openstackgerrityolanda.robla proposed openstack/diskimage-builder: Allow to skip kernel cleanup  https://review.openstack.org/35399408:29
yolandaianw ^08:29
*** e0ne has joined #openstack-infra08:30
*** ogelbukh_ has joined #openstack-infra08:31
*** tkelsey has joined #openstack-infra08:31
*** amitgandhinz has joined #openstack-infra08:33
*** mhickey has joined #openstack-infra08:34
*** ogelbukh has quit IRC08:34
*** ogelbukh_ is now known as ogelbukh08:34
*** markvoelker has joined #openstack-infra08:34
*** jamesmcarthur_ has joined #openstack-infra08:35
*** kozhukalov_ has joined #openstack-infra08:35
*** oanson has joined #openstack-infra08:36
*** amitgandhinz has quit IRC08:37
*** jamesmcarthur has quit IRC08:38
*** jamesmcarthur_ is now known as jamesmcarthur08:38
*** Goneri has quit IRC08:38
*** kozhukalov has quit IRC08:38
*** kozhukalov_ is now known as kozhukalov08:38
*** markvoelker has quit IRC08:39
*** rossella_s has joined #openstack-infra08:40
*** tqtran has joined #openstack-infra08:43
*** mrmartin has joined #openstack-infra08:44
*** pahuang has quit IRC08:46
*** ihrachys has joined #openstack-infra08:47
*** mrmartin has quit IRC08:50
*** sflanigan has quit IRC08:50
*** dtantsur|afk is now known as dtantsur08:52
*** d0ugal has quit IRC08:52
*** Goneri has joined #openstack-infra08:53
*** mrmartin has joined #openstack-infra08:58
*** sambetts|afk is now known as sambetts09:04
*** rbuzatu has quit IRC09:07
zhenguoyolanda: thanks for +A my patch, could you please add me to nimble-core and nimble-release!09:07
yolandasure09:07
zhenguoyolanda:  Thanks a lot!09:08
*** tqtran has quit IRC09:08
*** rbuzatu has joined #openstack-infra09:09
yolandadone09:11
*** chem has quit IRC09:15
*** yamamoto has quit IRC09:17
openstackgerritSteven Hardy proposed openstack-infra/tripleo-ci: Try increasing KeystoneWorkers to improve performance  https://review.openstack.org/35460609:17
openstackgerritSam Betts proposed openstack-infra/project-config: Ensure we alway build old Ironic ramdisk  https://review.openstack.org/35460809:20
*** amotoki has joined #openstack-infra09:20
*** chem has joined #openstack-infra09:23
*** amotoki_ has quit IRC09:23
*** rraja has quit IRC09:28
*** _nadya__ has quit IRC09:33
*** amitgandhinz has joined #openstack-infra09:33
*** jpeeler has quit IRC09:35
*** markvoelker has joined #openstack-infra09:36
*** yanyanhu has quit IRC09:36
*** amitgandhinz has quit IRC09:38
*** markvoelker has quit IRC09:40
*** amitkqed has quit IRC09:40
*** amitkqed has joined #openstack-infra09:41
*** amotoki_ has joined #openstack-infra09:41
*** mrunge has joined #openstack-infra09:41
*** martinkopec has joined #openstack-infra09:43
*** amotoki has quit IRC09:44
*** sdague has joined #openstack-infra09:46
*** sarob has joined #openstack-infra09:47
*** matrohon has quit IRC09:48
*** jpeeler has joined #openstack-infra09:50
*** _nadya_ has joined #openstack-infra09:50
openstackgerritAleksey Zvyagintsev proposed openstack-infra/project-config: Add repo for murano-pkg-check. Murano package validator tool.  https://review.openstack.org/35386109:52
*** sarob has quit IRC09:53
*** vincentll has quit IRC09:53
*** TRUMP-PENCE has joined #openstack-infra09:56
*** zhurong has quit IRC10:04
*** tqtran has joined #openstack-infra10:05
*** gongysh has quit IRC10:08
*** tqtran has quit IRC10:09
*** coolsvap is now known as _coolsvap_10:13
*** mhayden has joined #openstack-infra10:13
*** tosky has joined #openstack-infra10:15
*** amotoki_ has quit IRC10:15
*** dtantsur is now known as dtantsur|brb10:20
openstackgerritThomas Bechtold proposed openstack-infra/project-config: designate: Add a non-voting job with postgres as DB backend  https://review.openstack.org/35414110:21
*** acoles_ is now known as acoles10:22
rcarrillocruzpabelanger, fungi : happy to report that with the fixes put in place for bifrost, i'm now doing ipmi commands from baremetal00 to controller00 succesfully10:23
rcarrillocruzprovisioning ongoing10:23
rcarrillocruzlet's see how it goes10:23
rcarrillocruzPROGRESS!10:23
*** _nadya_ has quit IRC10:24
*** amotoki has joined #openstack-infra10:27
*** mhayden has quit IRC10:30
*** amitgandhinz has joined #openstack-infra10:34
*** amitkqed has quit IRC10:35
*** amitkqed has joined #openstack-infra10:36
*** amitgandhinz has quit IRC10:38
*** daemontool has quit IRC10:39
*** daemontool has joined #openstack-infra10:41
*** kzaitsev_mb has joined #openstack-infra10:41
azvyagintsevAJaeger https://review.openstack.org/#/c/353861/  could you please help ? looks like we still miss some logic...10:45
*** daemontool_ has joined #openstack-infra10:47
*** daemontool has quit IRC10:47
openstackgerrityolanda.robla proposed openstack/diskimage-builder: Allow to skip kernel cleanup  https://review.openstack.org/35399410:49
*** otherwiseguy has quit IRC10:51
*** otherwiseguy has joined #openstack-infra10:52
*** kragniz1 has quit IRC10:52
*** kragniz1 has joined #openstack-infra10:54
*** kragniz1 has quit IRC10:59
*** rossella_s has quit IRC10:59
*** _nadya_ has joined #openstack-infra11:07
rcarrillocruzi'm starting afresh on the baremetal00 node11:09
rcarrillocruzprovisioning don't go thru11:10
rcarrillocruzi performed a provision-state deleted , and the node went to a cleaning phase11:10
mordredrcarrillocruz: woot!11:10
rcarrillocruzdespite having ironic.conf cleaning = false11:10
mordredoh11:10
rcarrillocruzbut jroll told me that conf param is deprecated11:10
rcarrillocruzthus11:10
rcarrillocruzi upgraded bifrost11:10
rcarrillocruzBUT11:10
rcarrillocruzironic.conf was not11:10
jrollwell, removed now, but yeah :/11:10
rcarrillocruzi.e. there's cruft around11:10
mordredah - nod11:10
rcarrillocruzso i'll nuke everything11:10
*** kushal has joined #openstack-infra11:13
*** mhayden has joined #openstack-infra11:13
*** weshay has joined #openstack-infra11:14
*** kragniz1 has joined #openstack-infra11:15
*** yolanda has quit IRC11:16
*** sambetts is now known as sambetts|lunch11:16
openstackgerritIvan Udovichenko proposed openstack-infra/project-config: Add new/update existing projects  https://review.openstack.org/34704711:17
*** kragniz1 has quit IRC11:18
*** yolanda has joined #openstack-infra11:18
*** jkilpatr has joined #openstack-infra11:19
*** mhayden has quit IRC11:19
*** kragniz1 has joined #openstack-infra11:19
*** dkehn has quit IRC11:20
*** dkehn_ has quit IRC11:21
pabelangerrcarrillocruz: yay11:25
pabelangerrcarrillocruz: let me know if you have screen running, I wouldn't mind watching11:25
*** dkehn has joined #openstack-infra11:27
*** jaosorior has quit IRC11:29
*** jaosorior has joined #openstack-infra11:30
*** amotoki has quit IRC11:30
*** Julien-zte has quit IRC11:33
pabelangermordred: off to get some coffee, then ready to take on nodepool restart11:33
*** kushal has quit IRC11:34
*** dtantsur|brb is now known as dtantsur11:34
*** dkehn_ has joined #openstack-infra11:34
*** kushal has joined #openstack-infra11:34
openstackgerritMonty Taylor proposed openstack-infra/project-config: Add experimental grenade job with neutronclient from source  https://review.openstack.org/35466411:34
mordredpabelanger: woot!11:34
*** mhayden has joined #openstack-infra11:35
*** amitgandhinz has joined #openstack-infra11:35
*** ldnunes has joined #openstack-infra11:35
*** kragniz is now known as Guest6666611:36
*** kragniz1 is now known as kragniz11:36
*** markvoelker has joined #openstack-infra11:37
mordredpabelanger: shade 1.10.0 is installed on nodepool.o.o, so it's ready to go11:37
pabelangermordred: okay, ready when you are11:38
rcarrillocruzpabelanger: sure, eating now11:38
mordredpabelanger: ready as I'll ever be11:38
zigoAm I right that there's newly added capacity in infra?11:39
rcarrillocruzlet me bring back bifrost to life, i nuked everything 30min ago11:39
zigoI'm not waiting forever anymore ... :P11:39
rcarrillocruzzigo: bluebox was added yeah11:39
zigoNice.11:39
*** amitgandhinz has quit IRC11:39
*** jistr has quit IRC11:39
mordredpabelanger: I'm tailing logs in one window - opening another to do the actual restart .. anything I need to do other than just restart the service these days?11:39
pabelangermordred: ya, that's all I did last time too11:40
mordredgreat11:40
*** dansmith has quit IRC11:40
zigoWhat's the best place to document how to contribute to packaging-deb ?11:40
*** StevenK has quit IRC11:41
*** amotoki has joined #openstack-infra11:41
zigoopenstack-manuals/doc/contributor-guide ?11:41
*** dansmith has joined #openstack-infra11:41
*** dansmith is now known as Guest1395711:42
*** markvoelker has quit IRC11:42
*** StevenK has joined #openstack-infra11:42
pabelangermordred: we should restart nodepool-builder too11:42
rcarrillocruzpabelanger: something off with dns on baremetal0011:43
rcarrillocruzrunning the remote_puppet for infracloud11:43
pabelangerzigo: yes, bluebox adds another 39 nodes for us to use11:43
rcarrillocruzchanges /etc/hosts11:43
rcarrillocruzand then 'facter fqdn' returns 'baremetal00'11:44
rcarrillocruzwhich prevents from fully configuring the node, as there's no match by puppet apply against site.pp11:44
pabelangerrcarrillocruz: Hmm11:44
rcarrillocruzper the logs is something with the unbound stuff11:44
pabelangerrcarrillocruz: possible we depended on search and domain from dhcp?11:45
openstackgerritIvan Udovichenko proposed openstack-infra/project-config: Add new/update existing projects  https://review.openstack.org/34704711:45
rcarrillocruzthing is we also install dnsmasq on this machine, so it's possible it could be interfering?11:45
*** sambetts|lunch is now known as sambetts11:46
rcarrillocruzneed to check the unbound stuff to understand what's going on11:46
mordredpabelanger: I'm going to revert11:46
pabelangermordred: ack11:46
openstackgerritIvan Udovichenko proposed openstack-infra/project-config: Add new/update existing projects  https://review.openstack.org/34704711:46
*** YorikSar has quit IRC11:46
mordredpabelanger: there's a bug - it's not in the thing I was worried about though - so that's good11:47
openstackgerritIvan Udovichenko proposed openstack-infra/project-config: Add new/update existing projects  https://review.openstack.org/34704711:48
*** gouthamr has joined #openstack-infra11:48
*** YorikSar has joined #openstack-infra11:48
*** sarob has joined #openstack-infra11:48
*** mhayden has quit IRC11:50
openstackgerritMonty Taylor proposed openstack-infra/shade: Do not instantiate logging on import  https://review.openstack.org/35467411:51
rcarrillocruzk11:53
*** sarob has quit IRC11:53
rcarrillocruzputting 8.8.8.8 on resolv.conf and fixing /etc/hosts did it, ansible-puppet going thru now, which kicked the install11:53
rcarrillocruzpabelanger: i'll open a screen when it finishes the install11:54
rcarrillocruzin case you wanna see how it goes the enroll and deploy11:54
*** matthewbodkin has quit IRC11:54
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: tripleo-jobs-gerrit.py: update undercloud & multinode job names  https://review.openstack.org/35467911:56
*** YorikSar has quit IRC11:56
mordredpabelanger: tl;dr - we focused on fixing the fip leak so much, we forgot to skip fixing the fip leak on clouds that don't have fips :)11:56
rcarrillocruzbtw, i need to puppet the routing to the iLO network11:56
*** YorikSar has joined #openstack-infra11:56
rcarrillocruzit has to go thru the private interface on baremetal00, going thru public the router is no happy11:57
openstackgerritIvan Udovichenko proposed openstack-infra/project-config: Add new/update existing projects  https://review.openstack.org/34704711:57
rcarrillocruzjroll: running the install playbook i do see the automated_clean param now11:58
rcarrillocruzon ironic.conf that is11:58
jrollrcarrillocruz: awesome11:58
rcarrillocruzthx a bunch!11:58
jrollyou're welcome :)11:58
jrollrcarrillocruz: I'm very excited about getting infra-cloud online, so feel free to bug me when you're having problems11:58
*** gongysh has joined #openstack-infra11:58
rcarrillocruzyup, long overdue to have it back online11:59
rcarrillocruzthx11:59
*** tonytan4ever has joined #openstack-infra12:00
*** lucasagomes is now known as lucas-hungry12:00
*** ccamacho is now known as ccamacho|lunch12:02
pabelangermordred: doh12:02
*** psilvad has joined #openstack-infra12:03
*** kushal has quit IRC12:04
*** tonytan4ever has quit IRC12:05
*** alaski is now known as lascii12:05
openstackgerritEmilien Macchi proposed openstack-infra/project-config: puppet/tripleo: restore ovb stable jobs  https://review.openstack.org/35417212:05
mordredpabelanger: yah12:06
openstackgerritMonty Taylor proposed openstack-infra/shade: Don't supplement floating ip list on clouds without  https://review.openstack.org/35468712:06
mordredpabelanger: there ya go12:06
*** tqtran has joined #openstack-infra12:06
pabelangermordred: ack12:07
mordredrcarrillocruz: you too - if you have a sec - ^^12:08
rcarrillocruzyeah, was looking12:08
*** AJaeger has joined #openstack-infra12:08
rcarrillocruzcode lgtm12:09
mordred\o/12:09
openstackgerritPaul Belanger proposed openstack-infra/system-config: Update shade configuration in nodepool for tripleo  https://review.openstack.org/35428112:09
AJaegerinfra-root, I think we do not have current images on internap and rax-ord (at least). Those still do not use bindep.txt. has anybody time to upload manually the images for those clouds that are outdated, please?12:10
openstackgerritZara proposed openstack-infra/storyboard-webclient: Search for boards and worklists  https://review.openstack.org/34156212:10
AJaegerOr do you want to wait for shade updates and then monitor upload everywhere?12:10
pabelangerAJaeger: working on that12:10
* AJaeger would love to have bindep available to be able to rename bindep.txt everywhere12:10
*** amotoki has quit IRC12:10
AJaegerthanks, pabelanger12:10
*** tqtran has quit IRC12:11
openstackgerritMonty Taylor proposed openstack-infra/project-config: Add experimental grenade job with neutronclient from source  https://review.openstack.org/35466412:11
*** xyang1 has joined #openstack-infra12:14
openstackgerritMerged openstack-infra/system-config: Refactor layout for clouds_layouts.yaml  https://review.openstack.org/35359412:14
AJaegermordred: that will not work ^ your job will only run on stable/liberty and mitaka. Is that really your intention?12:14
*** rhallisey has joined #openstack-infra12:15
*** amotoki has joined #openstack-infra12:16
mordredAJaeger: nope :)12:17
mordredAJaeger: I'm trying to accomplish a job that can show that a patch to neutronclient along with a patch to requirements results in a working grenade12:18
mordred(all for master)12:18
*** sean-k-mooney has joined #openstack-infra12:21
sean-k-mooneyhi quick question is there a way to rerun just one of the many jobs run by jenkins/zuul instead of doing a full recheck?12:22
*** amoralej is now known as amoralej|lunch12:23
*** jistr has joined #openstack-infra12:24
sean-k-mooneythe gate-grenade-dsvm-multinode job failed because it could not ssh into the host vm for https://review.openstack.org/#/c/334048/ so was wondering if i could just trigger gate-grenade-dsvm-multinode again?12:24
openstackgerritMonty Taylor proposed openstack-infra/shade: Do not instantiate logging on import  https://review.openstack.org/35467412:24
sean-k-mooneyill assume the answer is no but it might be nice to add in the future as it would save alot of ci time if i coulud just say recheck "job name" or recheck jenkins if i did not want to also recheck the third party cis12:28
openstackgerritMonty Taylor proposed openstack-infra/shade: Protect cinderclient import  https://review.openstack.org/35469712:29
*** amotoki has quit IRC12:29
*** matthewbodkin has joined #openstack-infra12:30
*** jistr has quit IRC12:31
mordredsean-k-mooney: yah - it's not a thing you can do - and it's actually on purpose ... the concern is that if we had single-job spot rechecks that someone could get flaky code in by just rechecking each job over and over until the job in question turned green12:31
mordredsean-k-mooney: that said - it is a big resource drain to do the full rechecks for sure12:31
*** amotoki has joined #openstack-infra12:31
*** sarob has joined #openstack-infra12:31
AJaegermordred: great, hope you get it all together ;)12:32
*** jistr has joined #openstack-infra12:32
*** gongysh has quit IRC12:33
*** adreznec has quit IRC12:33
sean-k-mooneymordred: i guess a mitagation would be to allow recheck only job groups so recheck jenkin so that the thrid party cis dont have to re run12:33
*** chlong|mtg has quit IRC12:33
*** markvoelker has joined #openstack-infra12:35
sean-k-mooneymordred: or maybe a little more granular so recheck nova (runs gate-nova-*) recheck grenade (runs gate-grenade-*) and recheck tempest (runs gate-tempest-*)12:35
*** adreznec has joined #openstack-infra12:35
*** senk_ has quit IRC12:35
mordredtotally. I had thought that each third-party-ci was supposed to have it's own recheck tag - but I haven't been following that recently12:35
sean-k-mooneymordred: but yes i understand the reasoning12:35
*** amotoki has quit IRC12:35
sean-k-mooneymordred: they do but if you do a recheck i think it recheck them too12:35
*** amitgandhinz has joined #openstack-infra12:36
sean-k-mooneymordred: i chould be wronge though12:36
mordredahhh12:36
mordrednod12:36
*** sarob has quit IRC12:36
*** baoli has joined #openstack-infra12:36
*** spzala has joined #openstack-infra12:36
openstackgerritMerged openstack-infra/storyboard-webclient: Search for boards and worklists  https://review.openstack.org/34156212:37
openstackgerritMerged openstack-infra/system-config: Update shade configuration in nodepool for tripleo  https://review.openstack.org/35428112:37
*** baoli_ has joined #openstack-infra12:37
sean-k-mooneymordred: actully talking to waldek who set up our cis aparently it depends on the third party ci but most seam to run on just rechck also.12:37
*** gildub_ has quit IRC12:38
sean-k-mooneymordred: but ya thats what i taught ill just kick off the full recheck instead.12:38
sean-k-mooneymordred: thanks12:38
*** amitgandhinz has quit IRC12:40
openstackgerritVitaly Gridnev proposed openstack-infra/project-config: don't run smoke tests in sahara grenade  https://review.openstack.org/35470012:40
pabelangermordred: sean-k-mooney: I believe anteaya has more details, but I think the current policy is 3rd CI only support recheck, not recheck acmeCI12:40
*** amitgandhinz has joined #openstack-infra12:40
*** spzala has quit IRC12:41
*** baoli has quit IRC12:41
*** jistr has quit IRC12:41
sean-k-mooneypabelanger: most if not all of the third party cis also have a special recheck command for just them. im not aware of any that dont12:43
*** Na3iL has joined #openstack-infra12:43
sean-k-mooneypabelanger: wether that is requied or not is a seperate thing.12:43
rcarrillocruzeugh12:45
rcarrillocruzinstall playbook fails12:45
*** rlandy has joined #openstack-infra12:45
rcarrillocruzcos it does a git clone twice at the same time12:45
rcarrillocruzon 127.0.0.112:45
rcarrillocruzand localhost12:45
rcarrillocruzhttps://github.com/ansible/ansible/issues/1689612:45
pabelangersean-k-mooney: Right, I think it is more a policy at this point.12:45
mordredrcarrillocruz: oh neat12:46
mordredAJaeger: any chance you know what I'm doing wrong herE: http://logs.openstack.org/52/289052/12/check/gate-shade-dsvm-functional-libs-nv/41c5795/logs/devstacklog.txt.gz#_2016-08-11_22_04_39_28512:46
rcarrillocruzi'll check it out if that's on a minor release already on pip in a bit12:46
rcarrillocruzlater, moving back home12:46
mordredAJaeger: I have python-keystoneclient in PROJECTS - at least I think I do12:46
mordredand the same thing worked for keystoneauth just a little bit before this12:47
sean-k-mooneypabelanger: ya the reason i brought this up actully was not a concern about the thrid party cis it was actully the lenght the main ci takes now that we have swapped to xenial12:47
*** rfolco has joined #openstack-infra12:47
pabelangerrcarrillocruz: eep12:47
sean-k-mooneypabelanger: i understand that is related to a small number of xenial nodes being available currently for ci use12:47
*** amotoki has joined #openstack-infra12:47
AJaegermordred: devstack? I'm not really an expert there - but let me check quickly12:47
Zaragerrit integration with storyboard is so close, arrrarrarararh https://review.openstack.org/#/c/347486/12:48
Zaraahem12:48
pabelangersean-k-mooney: as we move forward, what will change. Once ubuntu-xenial is the dominate image for jobs, it jobs will build sooner12:48
pabelangersean-k-mooney: but then ubuntu-trusty lagging with be the issue12:48
sean-k-mooneypabelanger: :) ya i was assuming it was something like that12:49
AJaegermordred: I'm confused, have a look at http://logs.openstack.org/52/289052/12/check/gate-shade-dsvm-functional-libs-nv/41c5795/logs/reproduce.sh and search for PROJECTS ;(12:50
*** zhurong has joined #openstack-infra12:50
sean-k-mooneypabelanger: on a slightly different topic do you know how project are added to http://ci-watch.tintri.com/ ?12:50
pabelangersean-k-mooney: basically, we've maxed out our cloud resources in openstack-infra.  Adding bluebox back this week, gets us another 39 nodes.  But once infracloud comes online, that should increase our resources greatly12:51
pabelangersean-k-mooney: sorry, no. Not sure how that works12:51
pabelangerha ha12:51
pabelangeris says ask here12:51
openstackgerritIvan Udovichenko proposed openstack-infra/project-config: Add new/update existing projects  https://review.openstack.org/34704712:51
mordredAJaeger: WEIRD!12:51
mordredAJaeger: but thanks - that gives me something to go on ...12:52
mordred(I think)12:52
sean-k-mooneypabelanger: ya kolla does not currently have thirdpary cis. we might add on at some point but i was hoping we could just use it to track the infra jeninks kolla jobs12:52
*** links has quit IRC12:53
sean-k-mooneypabelanger: but no worries12:53
jrollsean-k-mooney: https://git.openstack.org/cgit/openstack-infra/ciwatch/tree/ciwatch.conf.sample12:53
jrollprobably either that or something in puppet?12:53
mordredAJaeger: oh!12:53
mordredAJaeger: wow, I REALLY suck12:53
jrollsean-k-mooney: or ping the people with commits on that repo and ask :)12:53
mordredAJaeger: I already landed the fix for this - the job run is from before the job was fixed12:53
AJaegerAh, so you had issues with timetravel ;)12:54
mordredI did12:54
mordredsilly me12:54
AJaegermordred: happens - no worries....12:54
sean-k-mooneyjroll: oh thanks for the link ill take a look. ya i probaly should have looked for ci watch in openstack infra. just checked openstack-dev and openstack.12:54
jrollsean-k-mooney: I just googled the domain and found an email thread about it: https://openstack.nimeyo.com/56556/openstack-third-party-announcing-watch-third-party-monitoring12:54
*** amoralej|lunch is now known as amoralej12:59
*** zul has quit IRC13:00
*** senk_ has joined #openstack-infra13:00
*** rossella_s has joined #openstack-infra13:00
*** lucas-hungry is now known as lucasagomes13:00
*** tonytan4ever has joined #openstack-infra13:01
*** baoli_ has quit IRC13:02
*** jistr has joined #openstack-infra13:03
*** daemontool_ has quit IRC13:05
*** matt-borland has joined #openstack-infra13:05
*** tonytan4ever has quit IRC13:06
*** zul has joined #openstack-infra13:07
*** psachin has quit IRC13:08
matthewbodkinWhen someone comments on your patch, there is an option to 'fix' the problem which then allows you to edit your work. Can anybody tell me if this actually works when you save it as it looks good on my end but I'm not sure about everybody else's i.e. can other people see that you've fixed it and review it again, and does it then work in the merge? Or would it be best to just make a new commit? Opinions please13:09
AJaegermatthewbodkin: you need to publish the change as well - and then it *is* a new commit13:10
matthewbodkinAhh okay thank you13:11
*** sean-k-mooney has left #openstack-infra13:11
matthewbodkinNow I've seen it... *facepalm*13:11
*** jistr has quit IRC13:13
odyssey4memorning all13:13
*** ccamacho|lunch is now known as ccamacho13:14
*** sdake has joined #openstack-infra13:14
*** edmondsw has joined #openstack-infra13:14
*** jcoufal has joined #openstack-infra13:15
*** spzala has joined #openstack-infra13:16
*** tbarron|gone is now known as tbarron13:16
odyssey4meI believe that it's possible for a test to be able to test cross repository patch combinations. ie I add 'depends-on: <change id>' in one repo and it tests with both patches applied. I know that this needs  zuul-cloner map to tell z-c where to place the repositories... is there anything else needed to make it work?13:16
odyssey4meie is there anything specific in project-config that needs to be done? Perhaps someone can point me to an example repo where this is done?13:16
*** mdrabe has joined #openstack-infra13:17
*** dprince has joined #openstack-infra13:18
*** dprince has quit IRC13:19
*** itisha has joined #openstack-infra13:19
*** dprince has joined #openstack-infra13:20
openstackgerritMonty Taylor proposed openstack-infra/project-config: Remove troveclient and ironicclient from shade libs  https://review.openstack.org/35471813:21
openstackgerritAlexander Evseev proposed openstack-infra/system-config: Move local user definitions to hiera  https://review.openstack.org/35399313:21
*** tonytan4ever has joined #openstack-infra13:22
pabelangerzigo: have you triggered an autobuid in the last hour? if not, can you?13:23
pabelangerauto-backports build13:23
clarkbmordred also worth noing we have tracked major issues back to repeat recheckers more than once13:26
clarkbso its nit a theoretical problwm13:26
rcarrillocruzmordred: do I approve https://review.openstack.org/#/c/354687/ or you want to13:26
rcarrillocruz?13:26
*** yamamoto has joined #openstack-infra13:28
ianychoiHello. I have basic question(s): how can I check current running operating system for translate-dev.o.o and translate.o.o (I think the OS would be trusty, not xenial). If it is trusty, how can we upgrade the OSes to xenial? Could somebody let me know?13:31
*** ajmiller has joined #openstack-infra13:32
*** esberglu has joined #openstack-infra13:33
ianychoihttps://git.openstack.org/cgit/openstack-infra/system-config/tree/manifests/site.pp says both OSes are trusty as comments, but it is difficult for me to find where the OSes are specified.13:34
openstackgerritMatthew Bodkin proposed openstack-infra/storyboard: Fix to slight errors in docs  https://review.openstack.org/35389113:35
*** ajmiller has quit IRC13:35
*** ddieterly has joined #openstack-infra13:39
*** rodrigods has quit IRC13:39
*** rodrigods has joined #openstack-infra13:40
*** mriedem has joined #openstack-infra13:40
AJaegerodyssey4me: you need the other repo checked out from source for this to work...13:41
openstackgerritAdam Coldrick proposed openstack-infra/storyboard: Create timeline events for boards and worklists  https://review.openstack.org/35014613:41
openstackgerritAdam Coldrick proposed openstack-infra/storyboard: Make it possible to get worklist/board timeline events via the API  https://review.openstack.org/35472913:41
openstackgerritAdam Coldrick proposed openstack-infra/storyboard: Send notifications to subscribers for worklists  https://review.openstack.org/35473013:41
AJaegerianychoi: pleia2 is the resident translate expert. Those servers do not run xenial. We first want to migrate all servers away from precise before going to xenial.13:42
*** yamamoto has quit IRC13:43
*** ajmiller has joined #openstack-infra13:44
odyssey4meAJaeger so as I understand it, the project-config needs to be using the zuul-cloner builder - there needs to be a clone map somewhere (in the repo root?) to indicate where things go. Zuul-cloner will use the git repo cache and handle the checkout. What else needs to be in place?13:44
openstackgerritZara proposed openstack-infra/storyboard-webclient: Fix Pagination for Admin Users Page  https://review.openstack.org/35473613:45
*** PollyZ has joined #openstack-infra13:45
ianychoiAJaeger, thanks for kind explanation!13:46
*** sdake has quit IRC13:46
*** toymachine has joined #openstack-infra13:46
toymachineHi, I'm trying to accept the ICLA on review.openstack.ork and I get 'Server Error - Cannot store contact information'13:48
toymachines/ork/org/r13:48
*** Julien-zte has joined #openstack-infra13:48
*** senk_ has quit IRC13:49
*** _ari_ has joined #openstack-infra13:49
*** Guest13957 is now known as dansmith13:50
*** dansmith is now known as Guest3789013:50
*** d0ugal has joined #openstack-infra13:51
*** Guest37890 is now known as dansmith13:51
*** ams__ has joined #openstack-infra13:51
*** ajmiller has quit IRC13:52
*** thorst_ has joined #openstack-infra13:53
*** zhurong has quit IRC13:53
*** rbrndt has joined #openstack-infra13:53
*** e0ne has quit IRC13:54
rcarrillocruznm, Shrews approved13:55
*** e0ne has joined #openstack-infra13:55
*** eharney has joined #openstack-infra13:56
pabelangerinfra-root: I am going to start the process to graceful stop / start zuul-launcher to pickup http://git.openstack.org/cgit/openstack-infra/zuul/commit/?id=17262fd5128e0bb4c462ba88bdf3e084cade9431 for sdague13:56
*** sdake has joined #openstack-infra13:56
sdaguepabelanger: thank you13:56
*** thorst_ has quit IRC13:57
*** dansmith is now known as superdan13:57
zaromorning13:58
mtreinishfungi, pabelanger, pleia2, rcarrillocruz: it looks like the change to elastic-recheck uncategorized https://review.openstack.org/#/c/353630/ isn't working right14:02
mtreinishI'm not seeing either of the new files on status.o.o14:02
mtreinishif you get a sec can you check the logs and see if something is up?14:02
*** ddieterly is now known as ddieterly[away]14:03
openstackgerritVitaly Gridnev proposed openstack-infra/project-config: don't run smoke tests in sahara grenade  https://review.openstack.org/35470014:03
Zarazaro: morning! I'm watching https://review.openstack.org/#/c/347486/ in excitement14:03
openstackgerritVitaly Gridnev proposed openstack-infra/project-config: don't run tempest tests in sahara grenade  https://review.openstack.org/35470014:04
*** pradk has quit IRC14:05
zaroZara: well don't need to watch it.  it's scheduled for today @ 20:00 UTC, http://lists.openstack.org/pipermail/openstack-dev/2016-August/101394.html14:05
*** baoli has joined #openstack-infra14:05
Zarahahaha14:05
*** bin_ has joined #openstack-infra14:06
*** vgridnev has joined #openstack-infra14:06
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: [NO MERGY] Test a fake periodic job  https://review.openstack.org/22978914:06
vgridnevhello team, could you please review the https://review.openstack.org/#/c/354700/ ?14:07
*** xarses_ has quit IRC14:07
*** bnemec is now known as beekneemech14:07
*** pradk has joined #openstack-infra14:07
Zarazaro: I just like seeing it! :D thank you for everything14:08
*** burgerk has joined #openstack-infra14:09
*** hongbin has joined #openstack-infra14:10
pleia2ianychoi: for future reference, we do keep the site.pp updated, so trusty is correct14:13
*** thorst_ has joined #openstack-infra14:14
EmilienMAJaeger: hey, about https://review.openstack.org/#/c/354172 - did yo mean zuul template? if yes I can do it now.14:15
openstackgerritMerged openstack-infra/shade: Don't supplement floating ip list on clouds without  https://review.openstack.org/35468714:15
*** ddieterly[away] is now known as ddieterly14:16
*** _coolsvap_ is now known as coolsvap14:16
*** edtubill has joined #openstack-infra14:19
mordredpabelanger: there are two more patches half-way through the gate after that ^^ - as soon as they land, we can release and restart again14:20
*** ryanpetrello has quit IRC14:21
openstackgerritIvan Udovichenko proposed openstack-infra/project-config: Add new/update existing projects  https://review.openstack.org/34704714:21
openstackgerritMerged openstack-infra/tripleo-ci: tripleo-jobs-gerrit.py: update undercloud & multinode job names  https://review.openstack.org/35467914:22
*** ryanpetrello has joined #openstack-infra14:22
zaroZara: well, friday night for you so might be difficult for you to catch. should be quick update though.14:22
Zara:) I'll still take a look!14:24
Zarathen there's just the per-project stuff...14:24
pabelangermordred: ++14:25
*** raunak has joined #openstack-infra14:26
openstackgerritEmilien Macchi proposed openstack-infra/project-config: puppet/tripleo: restore ovb stable jobs  https://review.openstack.org/35417214:27
EmilienMAJaeger: updated with a template14:27
*** ajmiller has joined #openstack-infra14:27
*** ryanpetrello has quit IRC14:29
*** rbuzatu has quit IRC14:29
*** thiagolib has joined #openstack-infra14:30
*** yamamoto has joined #openstack-infra14:30
*** rbuzatu has joined #openstack-infra14:32
*** ajmiller has quit IRC14:32
*** _ari_ has quit IRC14:32
openstackgerritIvan Udovichenko proposed openstack-infra/project-config: Add new/update existing projects  https://review.openstack.org/34704714:32
openstackgerritSam Yaple proposed openstack-infra/shade: Add update_endpoint()  https://review.openstack.org/28905214:32
*** ryanpetrello has joined #openstack-infra14:33
*** _ari_ has joined #openstack-infra14:33
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Don't deploy net-iso env when we won't use it  https://review.openstack.org/34750314:33
*** karthik__ has joined #openstack-infra14:34
*** valderrv has joined #openstack-infra14:35
*** baoli has quit IRC14:36
*** xarses_ has joined #openstack-infra14:36
*** baoli has joined #openstack-infra14:36
openstackgerritMonty Taylor proposed openstack-infra/shade: Infer nova-net security groups better  https://review.openstack.org/35249814:38
mordredShrews: ^^14:38
*** jistr has joined #openstack-infra14:39
*** senk_ has joined #openstack-infra14:39
Shrewsk14:39
*** PollyZ has quit IRC14:39
pleia2in case you were curious about my whereabouts last week, good conference, and woo apnic :) http://blog.apnic.net/2016/08/12/view-sanog-open-source-specialist/14:39
*** nwkarsten has joined #openstack-infra14:41
*** karthik__ has quit IRC14:41
fungiany system-config-core reviewer feel like approving https://review.openstack.org/347486 before the gerrit restart at 20:00?14:42
krotscheckAJaeger: Did the various bindep dependencies land in time for yesterday's dib build & upload? https://review.openstack.org/#/c/346130/14:42
fungizaro: do we also need a system-config change to update the gerrit build on review.o.o for me to merge during the maintenance?14:43
*** raunak has quit IRC14:43
pleia2fungi: lgtm14:44
*** woodster_ has joined #openstack-infra14:45
openstackgerritZara proposed openstack-infra/storyboard-webclient: 'All Boards and Worklists' Page  https://review.openstack.org/27975314:46
*** zz_dimtruck is now known as dimtruck14:46
*** nwkarsten has quit IRC14:46
Zara\o/ thanks pleia2 and fungi!14:47
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck: Remove query for duplicate bug 1450110  https://review.openstack.org/35476014:47
openstackbug 1450110 in Cinder "boto tests randomly fail in resource_cleanup with "Failed to delete volume"" [High,Fix released] https://launchpad.net/bugs/145011014:47
zarofungi: look for 'gerrit-upgrade' topic14:47
pleia2Zara: I'm excited :)14:47
pabelangerjlk: nibalizer: I see bluebox make the change for cpu overcommit. However, I still only see 63VCPUs per hypervisor.  Is there something I am missing?14:48
*** adrian_otto has joined #openstack-infra14:48
*** _nadya_ has quit IRC14:49
openstackgerritZara proposed openstack-infra/storyboard-webclient: 'All Boards and Worklists' Page  https://review.openstack.org/27975314:50
*** mdrabe has quit IRC14:52
*** dingyichen has quit IRC14:52
Zarapleia2: me toooooo, this was described to me as a 'one day, maybe storyboard will do this' feature when I first started work on it. :D14:53
*** amitgandhinz has quit IRC14:54
toymachinehi, is contact information storage down for doing ICLA's on review.openstack.org?14:54
*** amitgandhinz has joined #openstack-infra14:54
openstackgerritMerged openstack-infra/system-config: Install its-storyboard plugin on review.o.o  https://review.openstack.org/34748614:54
toymachineI'm trying to submit a review, but I keep getting errors that contact information is not present, and when I try to add it, it fails.14:54
anteayatoymachine: have you seen this? https://ask.openstack.org/en/question/3413/icla-contributor-agreement-requires-current-contact-information-error-while-using-git-review/14:55
pleia2Zara: cake all around!14:55
*** _nadya_ has joined #openstack-infra14:55
Zaraall of it!14:56
toymachineanteaya: yep, I've tried submitting the contact info, but when I click "Submit" review.openstack.org gives a storage error14:56
* mordred passes around pieces of the cake he found laying here14:56
openstackgerritMerged openstack-infra/shade: Do not instantiate logging on import  https://review.openstack.org/35467414:56
openstackgerritMerged openstack-infra/shade: Protect cinderclient import  https://review.openstack.org/35469714:56
toymachineanteaya: I'm filling in all submission boxes14:57
mordredjeblair: ^^ your logging problems should now be fixed14:57
anteayatoymachine: can you paste your storage error? paste.openstack.org14:58
*** martinkopec has quit IRC14:58
toymachineanteaya: sure, one sec14:58
*** _nadya_ has quit IRC14:58
*** rajinir has joined #openstack-infra14:58
toymachineanteaya: http://paste.openstack.org/show/556112/14:59
*** mdrabe has joined #openstack-infra14:59
anteayathat is it? that is the entire error?15:00
toymachineanteaya: yep15:00
anteayawhat command are you running to get that error?15:00
toymachinethis is in the web browser on review.openstack.org15:00
toymachineI'm trying to add offline contact info15:00
anteayaso you are here? yes? https://review.openstack.org/#/settings/contact15:02
toymachineanteaya: yes15:02
ianychoipleia2, thanks! According to https://etherpad.openstack.org/p/austin-i18n-workshop it seems that upgrading from trusty to xenial is needed. I have been tracking how many steps are to upgrade Zanata.15:03
anteayaso I just reentered my Country and pressed Save Changes and got Contact information last updated on Aug 12, 2016 at 11:02 AM.15:03
anteayatoymachine: so the server took my information15:03
pleia2ianychoi: yeah, that's correct, since we need the newer version of Java15:03
toymachineanteaya: interesting, am I doing something wrong?15:03
anteayayes15:03
toymachineanteaya: is there someone who can check my account?15:03
pleia2ianychoi: unfortunately things are held up as we try to find time to update our puppet manifests to support Xenial (I had hoped to have more time for this, alas...)15:04
anteayatoymachine: did you follow all the steps: http://docs.openstack.org/infra/manual/developers.html#account-setup15:04
toymachineanteaya: I believe so, but I will go through it again15:04
pleia2hopefully I'll be able to poke at it next week15:04
anteayatoymachine: please read our documentation again about account creation15:04
anteayatoymachine: thank you15:04
openstackgerritMonty Taylor proposed openstack-infra/project-config: Remove python3 jobs from zuul  https://review.openstack.org/35477315:04
anteayatoymachine: emails must match and you have to be the correct foundation member15:04
toymachineanteaya: that fixed it, I had wrong email selected15:05
toymachineanteaya: thank you!15:05
openstackgerritMerged openstack-infra/zuul: Add some documentation on functional testing tools  https://review.openstack.org/35193115:05
*** adrian_otto has quit IRC15:05
ianychoipleia2, I see. Although Jenkins is running with Xenial, the problem is that puppet does not support Xenial.. thanks!15:05
prometheanfirewe install snmp now?15:06
anteayatoymachine: thanks for reporting back, welcome and have fun15:07
openstackgerritMerged openstack-infra/zuul: Stop sharing Gerrit event queues in tests  https://review.openstack.org/35263515:07
anteayapleia2: great blog post15:07
pleia2ianychoi: we aren't running jenkins anymore15:07
pleia2anteaya: thanks :)15:07
anteaya:)15:08
pleia2ianychoi: if you mean our test nodes that are running Xenial, it's true, but they aren't using the same tooling as our long-lived servers15:08
*** armax has joined #openstack-infra15:08
*** adrian_otto has joined #openstack-infra15:08
*** kzaitsev_mb has quit IRC15:08
anteayait seems to odd to see you type we aren't running jenkins anymore15:08
mtreinishpleia2: firehose.o.o is on xenial, but I hand wrote all the policy to only work there (including systemd unit files, etc)15:08
pleia2mtreinish: nods15:09
pleia2exim was giving some errors on xenial last I checked too15:09
prometheanfireare we monitoring nodepool instances with snmp?15:09
pleia2I should do a once through today to see where we are, I have a storyboard story about upgrading this somewhere15:09
openstackgerritMerged openstack-infra/zuul: Update category mapping in tests  https://review.openstack.org/35263615:10
pleia2anteaya: odd?15:10
prometheanfireif we are then it's breaking osa-security role (stig specifies v3 only)15:10
anteayajust unusual15:10
jlkpabelanger: nibalizer: overcommit comes into play with scheduling. It won't _show_ 2x the VCPUs but it will allow you to _consume_ 2x the CPUs.15:10
pleia2anteaya: how so?15:10
anteayait has been part of us for so long15:10
pleia2ah :)15:10
anteayayeah that15:10
*** ayoung has joined #openstack-infra15:10
anteayayou are accurate15:10
anteayajust sounds odd to my ears15:10
jlkpabelanger: nibalizer: the scheduler won't stop scheduling instances to HVs until after 2x the available vCPUs have been consumed.15:10
zigopabelanger: I'm working on deb-auto-backports yes.15:11
zigopabelanger: We're also preparing all git repos, lots of them are comming.15:11
*** xyang1 has quit IRC15:11
*** xyang1 has joined #openstack-infra15:12
ianychoipleia2, anyway thanks a lot for your kind effort! Now I am learning automated tools. Hope that someday I can help such kind of infra jobs..! :)15:12
pabelangerzigo: okay, gpg signing should be working, but want to test it15:12
pleia2ianychoi: we'd be happy to have the help :)15:12
*** kzaitsev_mb has joined #openstack-infra15:12
ianychoipleia2, xD15:13
zigopabelanger: I'll let you know.15:13
fungiprometheanfire: i don't think we need to preinstall snmpd on our single-use nodes, it's just an effect of having them share some configuration management in common with our persistent job nodes and control plane servers. we ought to be able to refactor it to omit that package for single-use nodes15:13
*** mhayden has joined #openstack-infra15:14
*** spzala has quit IRC15:14
*** spzala has joined #openstack-infra15:14
prometheanfirefungi: that sounds good, mhayden might be able to provide more info relating to the failure if you want to modify the config instead15:14
*** eharney has quit IRC15:15
*** eharney_ has joined #openstack-infra15:15
*** jaosorior has quit IRC15:15
*** senk_ has quit IRC15:15
openstackgerritVasyl Saienko proposed openstack-infra/project-config: Add networking-cumulus project  https://review.openstack.org/35478415:15
*** karthik__ has joined #openstack-infra15:15
pabelangerjlk: ah, okay. Even thought we see 63vcpus total, we'll have 126 used15:15
pabelangerif we run that many15:15
jlkpabelanger: that's correct15:16
pabelangerjlk: thanks for explaining15:16
*** nwkarsten has joined #openstack-infra15:16
jlkno prob, welcome to another awkward corner of OpenStack :D15:16
pabelangerindeed15:16
*** eharney_ is now known as eharney15:16
*** karthik__ has quit IRC15:18
*** spzala has quit IRC15:20
*** yamahata has joined #openstack-infra15:20
AJaegerodyssey4me: sorry, was away - for jobs using devstack-gate , devstack prepares the repos and checks them out. For most others, we only check the current repo out. Best to discuss this on a specific example IMHO. Note that I'm not around much today...15:21
pabelangerproject-config-core: 354338 is ready to land, which increased bluebox back to 39 nodes.  I've tested the change already and nodepool is happy15:21
openstackgerritPaul Belanger proposed openstack-infra/project-config: Migrate to gate-{name}-tox-db-{envlist}-{node}  https://review.openstack.org/35261415:21
zigopabelanger: BTW, it looks like the POST job isn't accepting some packages, that's a bit weird.15:22
*** spzala has joined #openstack-infra15:22
pabelangerzigo: log?15:22
prometheanfirefungi: do we need a tracking bug for that?15:22
zigopabelanger: Soon ! :)15:22
zigopabelanger: I'm trying to fix an issue in one of my script first.15:22
AJaegerkrotscheck: images with the bindep.txt change have been build, but seemed that upload did not succeed everywhere. OSIC and OVH are ready, internap and rax-ord had older images when I checked a couple of hours ago.15:23
zigopabelanger: I've made it so it can just download from official jessie-backports, so I don't need to actually maintain.15:23
*** thorst_ has quit IRC15:23
pabelangerzigo: If it is the issue I think it is, because you are trying to import multiple packages using the same version, which reprepro fails on.15:23
*** florianf has quit IRC15:23
*** pcaruana has quit IRC15:23
*** jistr is now known as jistr|call15:23
*** kien-ha has joined #openstack-infra15:24
zigopabelanger: Well, normally, it shouldn't happen, because my script uses madison-lite to check what's installed or not.15:24
mhaydenAJaeger: ah, thanks for the update on the bindep15:24
*** thorst_ has joined #openstack-infra15:24
krotscheckAJaeger: Thanks for checking. I know it's feierabend where you are, is this something we'll have to wait for monday on or can I poke a U.S. infra person to finish it off?15:24
zigopabelanger: Just let me finish to debug...15:24
fungiprometheanfire: nah, just need somebody to suss out where it's coming from and move the entry for the package to a more appropriate place. i'll take a look in a sec15:24
pabelangerjlk: are you able to help debug networking issues on montytaylor-sjc? we see some instances fail to resolve dns when first booted, still trying to figure out why15:24
prometheanfirefungi: cool15:24
pabelangerzigo: ack15:25
jlkpabelanger: not well. Networking isn't my strong suit. What is the DNS settings for the subnet you're booting them on? Is it relying on neutron provided DNS, or an external provider? (we see better results when using an external provider).15:25
pabelangerfungi: jeblair: With the cpu overcommit that bluebox has enabled, I've just proposed 39 nodes, which was our per upgrade count.  Did we want to consider increasing that number more?15:26
*** e0ne has quit IRC15:26
pabelangerjlk: we should be ignoring DNS from neutron and using a local unbound service. But need to confirm that still15:26
*** yamamoto has quit IRC15:26
jlkyeah, that can be set as a subnet attribute15:27
*** eharney_ has joined #openstack-infra15:27
AJaegerkrotscheck: those images are build and uploaded every day. pabelanger mentioned earlier that he's looking at them... Best check status with the team15:27
* zigo loves the telnet thing, thanks to whoever did that15:28
*** thorst_ has quit IRC15:28
*** yamamoto has joined #openstack-infra15:28
*** eharney has quit IRC15:29
kien-haelectrofelix: Hello, you around?15:29
*** raddaoui has joined #openstack-infra15:29
fungipabelanger: i'm more concerned with keeping tabs on job runtimes/timeouts to make sure the overcommit doesn't cause us an issue. since we're the only users in there are our workload is somewhat consistent, we could end up starving ourselves of resources15:30
fungipabelanger: i suppose the more we increase the max there, the sooner we'll know15:30
fungizigo: jeblair wrote the majority of that, to compensate for the loss of console log streaming in from the jenkins webui when we had to drop jenkins15:31
*** florianf has joined #openstack-infra15:31
*** devkulkarni has joined #openstack-infra15:31
fungizigo: the plan is to also have a websockets proxy for it so console-averse people can also pull it up in a browser15:31
jeblairpabelanger: also, we should keep an eye out on distribution.  previously, nova would fill compute nodes sequentially on bluebox.  i don't know if it still will, but if it does, we might see the full 2x overcommit on one compute host before it moves on to the second.15:31
*** thorst_ has joined #openstack-infra15:31
zigofungi: Please don't just break the telnet:// link.15:32
jeblairzigo: oh no, the telnet link is definitely a first-class feature now.  we will keep it (or something very much like it).15:32
AJaegerpabelanger: your change for tox-db is broken, we will run again both a xenial and a trusty job on some branches. Please carefully review all regex and branch conditions. I expect that wherever there is a branch condition, you only need to add either trusty or xenial - but never both.15:33
*** adrian_otto has quit IRC15:33
*** ifarkas is now known as ifarkas_afk15:33
fungiwhat class is higher than first-class? because it's that as far as i'm concerned15:33
openstackgerritMonty Taylor proposed openstack-infra/devstack-gate: Add osc-lib and os-client-config to PROJECTS  https://review.openstack.org/35479515:33
jlkHAPPY BIRTHDAY mordred !!!15:33
mordredjlk: thanks!15:33
fungimordred's secret is out15:33
mordreduhoh15:33
*** ayoung has quit IRC15:33
* AJaeger joins the choir in singing Happy Birthday...15:33
pabelangerfungi: Agreed, we could try, if bad things, revert15:33
pabelangerjlk: Oh, good point15:34
pabelangerjlk: sorry, jeblair ^15:34
pleia2mordred: it's my grandfather's 80th today, congrats on not being that old yet!15:34
fungior as i always like saying, congratulations on making it another year without dying yet15:34
jlkyou're only as old as the number of kids you yell at to get off your lawn in any given day.15:34
mordredjlk: I'm so old15:34
pabelangerAJaeger: Ya, thanks. Going to review that again15:34
pleia2haha15:34
mordredpleia2: yay I'm not 80!!!15:34
zigopabelanger: http://mirror.regionone.bluebox-sjc1.openstack.org/debian/dists/jessie-backports/main/source/Sources.gz doesn't seem to exist ... :/15:34
*** thorst__ has joined #openstack-infra15:35
jlkI'm there too.15:35
jlkand thanks to Pokemon Go, I _literally_ have to yell at kids to get off my lawn.15:35
zigopabelanger: I need it.15:35
pabelangerzigo: will have to check our reprepo configuration15:35
anteayamordred: oh yeah it is your birthday too today15:35
pleia2jlk: be nice, there's an Eevee there15:35
anteayahappy birthday15:35
anteayafungi: and happy birthday to your grandfather too15:35
anteayaAJaeger: are you available to weigh in on https://review.openstack.org/#/c/354402/15:36
fungianteaya: that was pleia2's g'padre i think15:36
AJaegerfungi, regarding bindep.txt: We have some clouds with updated images but not all. I can send out my announcemnt today and rename other-requirements, do it after my vacation, let you do it,... What is your preference?15:36
pleia2yeah15:36
anteayaoh sorry15:36
mordredjlk: additional nice thing about living in dallas - I have yet to be annoyed by any people playing pokemon go15:36
pleia2:)15:36
*** thorst_ has quit IRC15:36
anteayapleia2: happy birthday to your grandfather15:36
openstackgerritZane Bitter proposed openstack-infra/git-review: Clarify that submitting multiple commits is OK  https://review.openstack.org/35188815:36
anteayayour nicks aren't even the same colour15:36
jlkmordred: just wait until the 2 weeks a year people won't die of heat death from being outside for 5 minutes.15:37
anteayaI have no excuse15:37
fungii haven't needed to yell at any kids to get off my lawn. i mainly just have to deal with vacationers thinking it's socially acceptable to cut through my yard to get to/from the public boat ramp, or occasionally not understanding that they shouldn't be fishing from my dock15:37
pleia2I am not even wearing a hawaiian shirt today15:37
*** rbuzatu has quit IRC15:37
*** karthik__ has joined #openstack-infra15:37
anteayapleia2: how unusual15:37
* fungi is wearing a hawaiian shirt today. not that anyone should be surprised by that15:37
anteayafungi: do they eventually understand?15:37
Zarahappy birthday to everyone with a birthday today or any other day! :D15:37
mordred\o/15:37
AJaegeranteaya: regarding 354402, I would not veto it, the team understands the risks. It's not my preference as you know...15:37
anteayafungi: I am not surprised by that15:37
anteayaAJaeger: yes, I know15:38
anteayaAJaeger: mostly I wanted you to see it15:38
anteayaI also am not vetoing it15:38
fungiAJaeger: i can try to manually update images, though i'm already somewhat oversubscribed for the day15:38
anteayathough would prefer not voting if just in check15:38
AJaegeris there any other infra-root that can update images, so that we have bindep.txt handling everywhere?15:39
fungiinlaws are coming to stay for the night on short notice. luckily i already scheduled and announced two important maintenance windows i have to work this evening ;)15:39
jlkfunny that15:39
anteayafungi: well played15:39
*** sdake_ has joined #openstack-infra15:39
AJaegerpabelanger: you mentioned earlier that you work on images - was that part of it?15:39
anteayajlk: they positioned a pokemon on your lawn?15:39
jlkanteaya: so my wife tells me15:39
anteayajlk: wow15:40
anteayaI just, I don't even15:40
*** thorst__ has quit IRC15:40
pabelangerAJaeger: mordred is planning on restart nodepool again, once that is done, and working, I'm going to upload images15:40
jlkbut, lets not let facts get in the way of a good joke15:40
zigopabelanger: Oh, also, I used to maintain an arm64 backport of Jessie, for that, I have a VM within the OpenStack of linaro.org. Do you think it'd be possible to hook ourselves to it, to build in arm64 instead of amd64 ?15:40
anteayajlk: by all means15:40
zigopabelanger: Otherwise, I probably can trigger a build on the Jenkins which I installed, but that's really suboptimal, as I'd have to maintain my Jenkins configuration.15:40
fungijlk: you should erect an obvious and exaggeratedly oversized cage trap in that spot15:41
AJaegerpabelanger: thanks. Please prioritze internap and rax-ord since those have oldest images.15:41
jlklol15:41
AJaegerfungi, pabelanger : Then I'll prepare my announcement (will blog it again) and my patches...15:41
*** sdake has quit IRC15:41
*** karthik__ has quit IRC15:41
yolandamordred, pabelanger , i have a pair of changes waiting for review for several time, can you take a look? https://review.openstack.org/327707, https://review.openstack.org/34534915:42
*** jistr|call is now known as jistr15:42
anteayajlk: I support fungi's suggestion15:42
mordredyolanda: oh! virtual packages and require is a new thing to me15:43
mordredyolanda: that does the same thing as the if defined thing?15:43
mordredyolanda: does it protect if something else doesn't use virtual package?15:44
yolandamordred , it installs the packages on the realize. But it doesn't cause duplication problems, that was the main problem with puppet-openstackci as well15:44
krotscheckAJaeger: Ok, so once the images land we're safe to land that doc patch?15:44
yolandayes, as far as i read, it protects from real package declaration duplicates15:44
AJaegerkrotscheck: yes15:45
*** jpich has quit IRC15:45
jlkI actually want to put a plush pokemon out there, IN a cage15:45
yolandaactually i found a typo, a remainder of ensure_packages15:46
devkulkarniHi infra folks, I have a question regarding how to setup a devstack gate to replicate upstream failure.. let me explain the situation. I am trying to setup gate environment to debug failure of tempest-dsvm-lxc-ubuntu-trusty experimental gate on this patch https://review.openstack.org/#/c/354062/1. Devstack early logs (http://logs.openstack.org/62/354062/1/experimental/gate-tempest-dsvm-lxc-ubuntu-trusty/f772966/logs/devst15:46
*** kzaitsev_mb has quit IRC15:47
*** matthewbodkin has quit IRC15:47
openstackgerrityolanda.robla proposed openstack-infra/puppet-jeepyb: Require packages before installing jeepyb  https://review.openstack.org/32770715:47
devkulkarni I have access to rackspace nodes, and ubuntu-trusty on them has 3.13.0-93-generic. looking at https://github.com/openstack-infra/project-config/blob/master/nodepool/nodepool.yaml#L49 there are 8 different options for nodes which can be used for spinning up ubuntu-trusty node15:47
fungidevkulkarni: have you seen the readme file in teh devstack-gate repo? it has instructions on simulating devstack-gate jobs we run15:47
*** _nadya_ has joined #openstack-infra15:47
*** rcernin has quit IRC15:47
devkulkarniI was wondering a) if it is possible to control which node gets used for running a test, b) if it is possible to know the lowest kernel version in these 8 nodes15:47
*** _nadya_ has quit IRC15:48
fungidevkulkarni: all our ubuntu-trusty nodes are booted from one common image we upload into glance in all of our providers15:48
devkulkarnifungi: oh I see.. hmm, so that means when I spun up rax cloud server, I probably chose a different image.. hmm15:49
*** ddieterly is now known as ddieterly[away]15:49
*** slaweq__ has quit IRC15:49
fungidevkulkarni: we also have image build logs for them published at http://nodepool.openstack.org/ and a script at http://git.openstack.org/cgit/openstack-infra/project-config/tree/tools/build-image.sh you could use to build the same images yourself15:49
*** mdrabe has quit IRC15:49
devkulkarnifungi: oh cool15:49
*** mdrabe has joined #openstack-infra15:50
fungidevkulkarni: in theory just upgrading distro packages on the server instance you've booted and then rebooting it should get you the same kernel version, but i'm checking what that currently comes up with15:50
devkulkarnifungi: one more question.. when an image is uploaded to glance for rackspace, what name does it get?15:51
*** senk_ has joined #openstack-infra15:51
*** senk_ has quit IRC15:51
*** weshay is now known as weshay_brb15:51
openstackgerritMiles Gould proposed openstack/diskimage-builder: Allow installing proliant tools from an OS package  https://review.openstack.org/34421415:51
fungidevkulkarni: looks like our ubuntu-trusty images currently boot with 3.13.0-93-generic from the linux-image-3.13.0-93-generic (3.13.0-93.140) deb15:52
*** mdrabe has quit IRC15:52
*** mdrabe has joined #openstack-infra15:52
pabelangerzigo: Oh, I know why. We don't mirror source packages today from upstream. cc mordred15:52
fungidevkulkarni: amd64 arch15:52
pabelangerzigo: mordred: I think the reason was nobody needed them, until now15:53
mordredyup15:53
*** thorst_ has joined #openstack-infra15:53
mordredthat was the reason - we did not use them for anything, and they take up space15:53
fungidevkulkarni: the images we upload are only usable by our tenant, so probably not much help to you, but they look to us like | 102e2659-7661-4d79-9e14-5b9e9c7fada3 | template-ubuntu-trusty-1470104577  | active |15:54
pabelangermordred: are we interested to enable them for deb-opentack now?15:54
*** hockeynut has joined #openstack-infra15:54
*** senk_ has joined #openstack-infra15:54
devkulkarnifungi: I see15:55
mordredpabelanger: I mean, I don't have any personal uses for them ... but if zigo needs them then I could imagine that to be a good reason15:55
*** thorst_ has quit IRC15:55
*** ddieterly[away] is now known as ddieterly15:55
*** thorst_ has joined #openstack-infra15:56
zigomordred: pabelanger: I am comparing version of packages using the Sources.gz, so I do need it.15:56
zigoI also need it to do the backports...15:56
mordredpabelanger: we're going to need to do that carefully, as it'll be a very large vos release the first time15:57
pabelangermordred: agree. I don't plan on doing that work today15:58
devkulkarnifungi: little confused about one thing then.. for my patch, devstack-early logs show 3.13.0-92-generic.. this is for ovh-gra1. http://logs.openstack.org/62/354062/1/experimental/gate-tempest-dsvm-lxc-ubuntu-trusty/f772966/logs/devstack-early.txt.gz#_2016-08-11_14_04_24_466 .. it is the case that newer images have not been uploaded to ovh yet?15:58
pabelangerzigo: mordred: let me get a patch up and we what other infra-root say. zigo it is going to take a bit to land this, can you use upstream mirrors for now?15:58
openstackgerritJames E. Blair proposed openstack-infra/zuul: Rename test_connection to test_connections  https://review.openstack.org/35371215:58
openstackgerritJames E. Blair proposed openstack-infra/zuul: Stub out aborting jobs in ansible launcher  https://review.openstack.org/35377815:58
openstackgerritJames E. Blair proposed openstack-infra/zuul: Re-enable test_failed_changes  https://review.openstack.org/35441915:58
openstackgerritJames E. Blair proposed openstack-infra/zuul: Fix addFailTest  https://review.openstack.org/35263715:58
openstackgerritJames E. Blair proposed openstack-infra/zuul: Re-enable test_connection tests  https://review.openstack.org/35194915:58
openstackgerritJames E. Blair proposed openstack-infra/zuul: Tests: update references from worker to launch_server  https://review.openstack.org/35376815:58
openstackgerritJames E. Blair proposed openstack-infra/zuul: Tests: add an extra debug line when stalled  https://review.openstack.org/35376915:58
openstackgerritMerged openstack-infra/puppet-httpd: Fix selinux problems on vhost  https://review.openstack.org/34534915:59
*** krtaylor has quit IRC15:59
*** Apoorva has joined #openstack-infra15:59
zigopabelanger: Yeah, I'm doing it already (ie: httpredir.debian.org), though it sometimes fail ...15:59
*** sdake_ has quit IRC15:59
zigo(dak push, etc.)15:59
fungidevkulkarni: sometimes image uploads do fail, yes, so we may not be booting exactly the same vintage of image in all places. it eventually corrects itself, but some providers are more failure-prone in the face of very large (in our case ~5gib) glance uploads than others15:59
openstackgerritAleksey Zvyagintsev proposed openstack-infra/project-config: Add repo for murano-pkg-check. Murano package validator tool.  https://review.openstack.org/35386115:59
*** weshay_brb is now known as weshay15:59
pabelangerzigo: understood. Assuming no objections we can likey push the update next week16:00
zigoOk.16:00
*** eharney_ is now known as eharney16:00
*** vgridnev has quit IRC16:00
fungidevkulkarni: pabelanger is, i believe, working on making sure we get some of those updated with manual reupload attempts now, so probably will be better later today16:00
devkulkarnifungi: ok, understood.16:00
*** IlyaG has joined #openstack-infra16:01
devkulkarnifungi: oh I see. in that case, all the providers should have same image later today/tomorrow16:01
pabelangerfungi: devkulkarni: yes, we're aiming to restart nodepool shortly. Once that is done, I'll requeue image uploads16:01
anteayajlk: I would like a photo should you act on that16:01
fungii suppose we could consider a nodepool option to not switch an image to active in any provider until it's able to successfully upload it to all providers, though that might mean we update images far less frequently as the number of providers we have continues to grow16:01
openstackgerritJames E. Blair proposed openstack-infra/zuul: Rename test_connection to test_connections  https://review.openstack.org/35371216:01
openstackgerritJames E. Blair proposed openstack-infra/zuul: Stub out aborting jobs in ansible launcher  https://review.openstack.org/35377816:01
openstackgerritJames E. Blair proposed openstack-infra/zuul: Re-enable test_failed_changes  https://review.openstack.org/35441916:01
openstackgerritJames E. Blair proposed openstack-infra/zuul: Tests: update references from worker to launch_server  https://review.openstack.org/35376816:01
openstackgerritJames E. Blair proposed openstack-infra/zuul: Tests: add an extra debug line when stalled  https://review.openstack.org/35376916:01
*** roxanaghe has joined #openstack-infra16:01
fungithough it would probably make for a more consistent experience16:02
devkulkarnithanks pabelanger!! that will be very helpful..16:02
fungiwe'd also likely have to defer deletion of older images in that scenario, so could end up piling up a lot of unused images in providers as a result16:02
fungianyway, worth pondering16:03
jeblairinfra-root: who created logstash-worker-test.openstack.org?16:03
*** adrian_otto has joined #openstack-infra16:03
pabelangerI did not16:03
*** yamahata has quit IRC16:03
funginot i, afaik16:03
mordredI did not16:03
pleia2not I16:03
devkulkarniI have related question fungi/pabelanger.. is there a way to trigger gate run on a specific provider's node in situations when the images are not yet synced up on all the providers yet?16:03
fungithough beer could have claimed those braincells at some point16:03
fungidevkulkarni: no16:04
*** roxanaghe has quit IRC16:04
devkulkarnifungi: I see16:04
fungidevkulkarni: there is a very roundabout way to do it by having different image names in some providers and specifying nodes booted from those in job definitions (we did that once upon a time) but it's hard to manage and causes a less efficient overall usage of our aggregate quota16:05
jeblairalso, we lose entire clouds all the time16:05
fungibasically, fewer common images used across as many jobs as possible gives us the best throughput for the quotas we have available16:05
devkulkarnifungi: sure, I can imagine it to be not a very straight forward way16:06
*** TRUMP-PENCE has quit IRC16:07
*** tqtran has joined #openstack-infra16:08
openstackgerritPaul Belanger proposed openstack-infra/system-config: Also mirror source packages for debian / ubuntu  https://review.openstack.org/35481816:09
pabelangermordred: zigo: ^should be the change for source packages16:09
*** raunak has joined #openstack-infra16:09
*** cody-somerville has quit IRC16:10
*** PollyZ has joined #openstack-infra16:10
*** cody-somerville has joined #openstack-infra16:10
zigoEasy enough !16:10
*** TRUMP-PENCE has joined #openstack-infra16:10
*** annegentle has joined #openstack-infra16:10
pabelangereffort is downloading and releasing packages to AFS16:11
mordredyah16:11
pabelangerthat will take some time16:11
*** oanson has quit IRC16:11
mordredyup16:12
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Move snmpd module from template class to server  https://review.openstack.org/35481916:12
*** rbuzatu has joined #openstack-infra16:12
fungiprometheanfire: ^16:12
*** tqtran has quit IRC16:13
fungithough i still posit that any job running a security analyzer against our job nodes is suffering from a serious design flaw16:13
fungiwe don't treat them as secure/hardened systems, since we can't16:13
*** jerryz has joined #openstack-infra16:13
devkulkarnifungi: one more question.. earlier you mentioned that all trusty images "boot with 3.13.0-93-generic from the linux-image-3.13.0-93-generic (3.13.0-93.140) deb". where can I find similar information for rest of the platforms (debian, fedora, etc.).. (I trying to figure out options to test nova lxc driver)16:13
mordredinfra-root: could I get a coupla folks to look at https://review.openstack.org/354795 - we're trying to work our way out of an occ related gate issue that happened on monday, and getting occ into devstack-able places is needed16:13
fungidevkulkarni: the nodepool image build logs i linked earlier16:14
cloudnullpabelanger mordred: so we have a dual stack solution for IPv6/4 in the OSIC cloud1 env. so long as infra can ssh to the VM on IPv6 there "shouldn't" be much needed on the infra side to make it go.16:14
devkulkarnifungi: ok16:14
devkulkarniwill check in there16:14
mordredwoot16:14
cloudnullwould it be possible to have infra test a few jobs using the new network ?16:14
fungidevkulkarni: or we could add a uname -a check to a common informational builder or have ansible collect that when launching jobs or something16:14
cloudnullbefore making the major switch ?16:14
*** xarses_ is now known as xarses16:15
prometheanfirefungi: true, he ( mhayden ) was just going to disable that check on gate16:15
openstackgerritMonty Taylor proposed openstack-infra/project-config: Add experimental grenade job with neutronclient from source  https://review.openstack.org/35466416:16
mordredAJaeger: ^^ I believe that fixes your comments from earlier16:17
mordredanteaya: might you feel like gazing upon a project-config patch?16:17
*** bkero is now known as bkero-pto16:18
*** florianf has quit IRC16:18
pabelangercloudnull: Oooh16:19
mhaydenfungi: the goal certainly isn't to analyze the security of the gate image ;)16:19
cloudnullpabelanger:  the new network is GATEWAY_NET_V616:20
*** clenimar_ has joined #openstack-infra16:20
jeblairmordred: what's an osc-lib?16:20
cloudnullit'll need sec groups to be configured to allow for ingress ipv6 but that should go ?16:20
pabelangersounds like a job for ansible-role-cloud-launcher!16:20
mordredjeblair: a portion of python-openstackclient has been split out into a library so that projects implementing osc plugins can just depend on osc-lib instead of all of osc16:20
jeblairmordred: are we replacing amqp with open sound control (yes please!)16:21
mordredjeblair: I wish!16:21
mordredjeblair: osc is the abbreviation for "python-openstackclient"16:21
pabelangercloudnull: aside from a new provider in nodepool, not sure how to tests a few jobs16:21
cloudnullok16:21
pabelangerI man, we could add it16:21
*** tphummel has joined #openstack-infra16:22
pabelangerYa, that's going to be some work16:22
mordredjeblair: thanks16:22
*** csomerville has joined #openstack-infra16:23
mordredjeblair: you may also have an opinion on https://review.openstack.org/#/c/354773/16:23
cloudnullis there a manual change we could make for a few min to force jobs to use the new network config, which could be reverted in the event that we've overlooked something terrible? I'd like to not slow the gate down any more due to an oversight .16:24
*** oanson has joined #openstack-infra16:24
openstackgerritPaul Belanger proposed openstack-infra/project-config: Switch osic-cloud1 to IPv6  https://review.openstack.org/35482716:25
pabelangercloudnull: ^ Likely easier just to make the change to ipv6 in nodepool, and roll back if issues16:25
mordredyah16:26
*** cody-somerville has quit IRC16:26
mordrednodepool is good about re-reading its config16:26
*** clenimar_ has quit IRC16:26
*** yamahata has joined #openstack-infra16:26
pabelangerwe could manually test it on nodepool.o.o first, before landing the final change16:26
pabelangerput nodepool.o.o into emergency for a few minutes16:26
anteayamordred: you have xenial in one file and trusty in another on 35466416:26
anteayamordred: I predict zuul will be unhappy16:27
pabelangermordred: mind approving: https://review.openstack.org/#/c/354338/16:27
nibalizerpabelanger: jlk as you may have seen on the infra list, they got us 2x cpu overcommit on the bluebox cloud16:27
openstackgerritAleksey Zvyagintsev proposed openstack-infra/project-config: Add repo for murano-pkg-check. Murano package validator tool.  https://review.openstack.org/35386116:27
*** roxanaghe has joined #openstack-infra16:27
mordredanteaya: good catch16:27
jlkyup, we had a convo on it this morning16:28
pabelangernibalizer: Ya, confirmed it is working16:28
nibalizerwewt16:28
anteayamordred: I was eating lunch, then I gazed16:28
anteayanibalizer: I don't even know what that is supposed to mean16:28
openstackgerritMonty Taylor proposed openstack-infra/project-config: Add experimental grenade job with neutronclient from source  https://review.openstack.org/35466416:28
anteayawet newts?16:28
fungii need to disappear and have a slightly non-antisocial lunch with the inlaws, but expect to return by 18:00 utc to start prepping for maintenance windows16:28
cloudnullpabelanger: +1 local tests are working great16:29
cloudnullso if that goes then we can majorlly up the quota16:29
mordredwoot!16:29
pabelangercloudnull: yay16:29
mordredwe like it when you majorly up the quota16:29
pabelanger++16:29
anteayafungi: have fun with the inlaws16:30
*** karthik__ has joined #openstack-infra16:30
*** PollyZ has quit IRC16:31
nibalizeranteaya: just a configuration we asked them to do16:31
anteayaah16:31
anteayaI don't know the wewt configuration16:31
nibalizeroh 'wewt' is just slang for 'w00t'16:32
mordredwhich is slang for woot - which is an ironic re-appropriation of early 90s slang16:32
anteayaapparantly we are lacking in slang16:32
anteayathanks for ensuring we have quota for the day16:32
mordredslang is what makes the world go round16:33
*** pt_15 has joined #openstack-infra16:33
anteayaapparently16:33
anteayahas anyone told tim?16:33
anteayathey could save a lot of money16:33
pabelangerfungi: thoughts on https://review.openstack.org/#/c/354827 ? cloudnull says we are ready to make the change, figured we can enqueue it into gate to avoid the delay16:35
*** tesseract- has quit IRC16:35
mordredpabelanger: it lgtm16:36
*** electrofelix has quit IRC16:37
*** PollyZ has joined #openstack-infra16:37
anteayaready for it to go?16:37
anteayafungi is at lunch do you want to wait for him?16:37
nibalizeropenstack could use a jargon file16:39
anteayabut would it be used?16:40
*** dtantsur is now known as dtantsur|afk16:41
*** hpe-hj has joined #openstack-infra16:41
Zarawe have an 'acronyms' wiki page at our office that gets used sometimes (though I think mainly just when people add acronyms they've just found out)16:41
openstackgerritMerged openstack-infra/project-config: Revert "Temporarily reduce bluebox-sjc1 to 34 max-servers"  https://review.openstack.org/35433816:41
pabelangercan nodepool auto hold for - Automatically held after failing gate-shade-dsvm-functional-neutron be removed?16:41
anteayapabelanger: do you want to wait for fungi on 354827?16:44
mordredpabelanger: yah16:46
pabelangerthe pressure!16:46
*** roxanaghe has quit IRC16:46
*** PollyZ_ has joined #openstack-infra16:47
pabelangerI'll go with the flow, if we want to land it without fungi, I'm fine with that.16:47
pabelangerwanted to make sure he was looped in16:47
*** shashank_hegde has joined #openstack-infra16:47
*** PollyZ has quit IRC16:47
*** ddieterly is now known as ddieterly[away]16:48
anteayaI can approve now or wait16:49
*** senk_ has quit IRC16:49
anteayaI'll let you decide pabelanger16:49
pabelangercloudnull: what does your schedule look like?16:50
cloudnullim around today16:50
cloudnullcurrently at the OSA midcycle but around16:50
pabelangeranteaya: mordred: cloudnull: lets wait 30mins for fungi, then proceed16:51
*** kien-ha has quit IRC16:51
anteayapabelanger: very good16:51
*** senk_ has joined #openstack-infra16:52
*** ddieterly[away] is now known as ddieterly16:53
devkulkarnifungi: whenever you get a chance, can you confirm if this analysis is correct: http://paste.openstack.org/show/556139/16:53
*** PollyZ_ has quit IRC16:54
*** ganesan has joined #openstack-infra16:54
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add held nodes to singlestat for nodepool dashboard  https://review.openstack.org/35484316:55
*** lucasagomes is now known as lucas-dinner16:55
*** yamamoto has quit IRC16:57
*** e0ne has joined #openstack-infra16:58
cloudnullpabelanger: IDK if node pool will automatically add the neutron security group rules but we'll need to have ICMP and at a minimum SSH for v6 added to the rule set17:01
mordredah - no, it won't17:01
mordredbut our usual plan with security groups is set them to open all/all17:02
pabelangerya17:02
mordredI'm not sure if we added ipv6 copies of those rules though17:02
pabelangerwe have ingress IPv6/ipv6 any any right now17:02
cloudnullif someone has access http://cdn.pasteraw.com/jklby8oom2fxx78zgz6h220sz4vy6xv17:02
mordredawesome!17:02
*** karthik__ has quit IRC17:02
mordredyah - I think we're covered17:02
cloudnullok17:02
pabelangeryup17:02
cloudnulljust making sure.17:02
pabelangertotally adding osic-cloud1 to cloud-launcher too17:03
mordredstep one in getting new cloud: open all ports17:03
mordredpabelanger: ++17:03
cloudnull++17:03
mordredpabelanger: I'm excited about having the things in the cloud-launcher17:03
mordredpabelanger: I look forward to being able to add the user stuff17:03
pabelangermordred: Ya, worked well17:03
pabelangermordred: I think we want to have 1 cloud yaml file per provider, so we don't have a single massive file17:04
*** yaume has quit IRC17:04
pabelangerthen iterate over the yaml files17:04
mordredpabelanger: I'm not sure I agree17:04
mordredbut you've been actually doing the work :)17:04
*** sambetts is now known as sambetts|afk17:04
*** rbuzatu has quit IRC17:05
openstackgerritZsolt Dudás proposed openstack-infra/jenkins-job-builder: Replaces logging with oslo.log This commit replaces the logging module with the OpenStack standard oslo.log.  https://review.openstack.org/35485017:05
mordredpabelanger: like, the security group rules, keypairs and maybe even base images are a profile we should be able to define once and apply to all clouds17:05
*** klindgren_ is now known as klindgren17:07
pabelangerright17:07
* cloudnull fooding bbl17:07
*** asettle has joined #openstack-infra17:08
*** shashank_hegde has quit IRC17:08
*** tpsilva has joined #openstack-infra17:09
*** woodster_ has quit IRC17:09
*** _nadya_ has joined #openstack-infra17:10
*** rbuzatu has joined #openstack-infra17:11
*** rbuzatu has quit IRC17:11
*** mark-casey has joined #openstack-infra17:12
*** mark-casey has left #openstack-infra17:12
*** ayoung has joined #openstack-infra17:13
openstackgerritPaul Belanger proposed openstack-infra/system-config: Add tripleo-test-cloud configuration for cloud-launcher  https://review.openstack.org/35426817:14
openstackgerritPaul Belanger proposed openstack-infra/system-config: Add IPv6 security group rule to cloud-launcher  https://review.openstack.org/35485117:14
AJaegermonty, your change https://review.openstack.org/354664 looks fine - but I'd like neutron infra liasons to +1 as well. armax, dougwig, mestery : Could either of you review this, please?17:14
AJaegers/monty/mordred/ ^17:16
*** sputnik13 has joined #openstack-infra17:16
*** devkulkarni has quit IRC17:16
*** roxanaghe has joined #openstack-infra17:17
openstackgerritPaul Belanger proposed openstack-infra/system-config: Add osic-cloud1 to cloud-launcher  https://review.openstack.org/35485217:18
*** baoli has quit IRC17:19
pabelangerrcarrillocruz: do you have an example for ansible-role-cloud-launcher for a cloud with different regions?  EG: rackspace17:20
pabelangerrcarrillocruz: I don't think that would work today17:20
*** coolsvap has quit IRC17:21
*** roxanaghe has quit IRC17:23
ganesancan anyone one share me the link to setup 3rd party CI system for Cinder project (FiberChannel)?17:23
*** krtaylor has joined #openstack-infra17:23
*** roxanaghe has joined #openstack-infra17:25
*** pvaneck has joined #openstack-infra17:25
anteayaganesan: have you asked in the #openstack-cinder channel17:26
*** gyee has joined #openstack-infra17:26
anteayaI believe cinder has a wikipage they update for new third party ci systems to follow17:26
anteayasmcginnis: ^^17:26
ganesananteaya, not yet, I will17:27
anteayaganesan: I think getting to know the cinder folks is a great first step for you in being successful in your goal17:27
*** amoralej is now known as amoralej|off17:27
ganesananteaya: thanks17:28
openstackgerritAndreas Jaeger proposed openstack-infra/elastic-recheck: Move other-requirements.txt to bindep.txt  https://review.openstack.org/35485717:29
mordredarmax, dougwig, mestery: my intention is for that job to not stick around terribly long - but to use it to be able to show that https://review.openstack.org/#/c/352594/ works17:29
openstackgerritAndreas Jaeger proposed openstack-infra/gear: Move other-requirements.txt to bindep.txt  https://review.openstack.org/35485817:29
*** ddieterly is now known as ddieterly[away]17:29
* AJaeger starts spamming with bindep renames...17:30
openstackgerritAndreas Jaeger proposed openstack-infra/groups: Move other-requirements.txt to bindep.txt  https://review.openstack.org/35485917:30
openstackgerritAndreas Jaeger proposed openstack-infra/openstackid: Move other-requirements.txt to bindep.txt  https://review.openstack.org/35486017:30
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Move other-requirements.txt to bindep.txt  https://review.openstack.org/35486117:30
*** rbrndt has quit IRC17:30
openstackgerritAndreas Jaeger proposed openstack-infra/puppet-openstackci: Move other-requirements.txt to bindep.txt  https://review.openstack.org/35486217:30
AJaegermordred, armax, dougwig, mestery: The job is experimental - still, I'd like your feedback if you're around...17:30
openstackgerritAndreas Jaeger proposed openstack-infra/release-tools: Move other-requirements.txt to bindep.txt  https://review.openstack.org/35486317:30
openstackgerritAndreas Jaeger proposed openstack-infra/reviewday: Move other-requirements.txt to bindep.txt  https://review.openstack.org/35486417:30
openstackgerritAndreas Jaeger proposed openstack-infra/storyboard: Move other-requirements.txt to bindep.txt  https://review.openstack.org/35486517:31
openstackgerritAndreas Jaeger proposed openstack-infra/storyboard-webclient: Move other-requirements.txt to bindep.txt  https://review.openstack.org/35486617:31
openstackgerritAndreas Jaeger proposed openstack-infra/system-config: Move other-requirements.txt to bindep.txt  https://review.openstack.org/35486817:31
openstackgerritAndreas Jaeger proposed openstack-infra/zuul: Move other-requirements.txt to bindep.txt  https://review.openstack.org/35486917:31
*** e0ne has quit IRC17:31
anteayaganesan: welcome17:33
* anteaya grabs a spam surfboard17:33
*** admcleod_ has quit IRC17:36
*** admcleod has joined #openstack-infra17:38
*** admcleod has joined #openstack-infra17:38
*** hockeynut has quit IRC17:38
*** rossella_s has quit IRC17:39
*** tphummel has quit IRC17:40
*** karthik__ has joined #openstack-infra17:42
zarofungi: wondering if it would be appropriate to enable gerrit ssh.idleTimeout? https://review.openstack.org/Documentation/config-gerrit.html#sshd.idleTimeout17:42
*** e0ne has joined #openstack-infra17:42
*** shashank_hegde has joined #openstack-infra17:43
zaroby default, it's set to never close idle connections.17:43
*** sambetts|afk has quit IRC17:43
*** tphummel has joined #openstack-infra17:43
anteayazaro: I think we could enable that and set leave it at default17:43
anteayathen bring it down when we encourage folks to consume the firehose17:44
anteayathose are my thoughts17:44
anteayaAJaeger: armax says yes, you want to approve?17:44
*** sambetts has joined #openstack-infra17:45
*** vhosakot has joined #openstack-infra17:45
*** ihrachys has quit IRC17:46
*** Swami has joined #openstack-infra17:46
*** acoles is now known as acoles_17:46
*** oanson has quit IRC17:46
*** tonytan4ever has quit IRC17:49
*** mhickey has quit IRC17:50
*** jdennis1 has joined #openstack-infra17:50
*** jdennis has quit IRC17:51
*** sdague has quit IRC17:51
AJaegerdone, anteaya17:51
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: DO NOT MERGE - Periodic test  https://review.openstack.org/34694917:52
*** pblaho has quit IRC17:53
*** devananda has quit IRC17:54
*** krtaylor has quit IRC17:54
*** ganesan has quit IRC17:57
openstackgerritMerged openstack-infra/tripleo-ci: Reinstall python-requests in bootstrap script  https://review.openstack.org/35052117:57
*** oanson has joined #openstack-infra17:59
*** sdague has joined #openstack-infra18:00
*** edtubill has quit IRC18:01
*** TRUMP-PENCE has quit IRC18:02
*** yamamoto has joined #openstack-infra18:02
anteayaAJaeger: thanks18:03
*** TRUMP-PENCE has joined #openstack-infra18:03
pabelangeranteaya: cloudnull: mordred: fungi: I am ready to proceed with 354827 if others are18:06
*** yamamoto has quit IRC18:07
openstackgerritMerged openstack-infra/tripleo-ci: Make $TRIPLEO_ROOT overrideable  https://review.openstack.org/35053918:08
openstackgerritMerged openstack-infra/tripleo-ci: Raise ping test count after multinode setup  https://review.openstack.org/35191618:08
openstackgerritPeter Stachowski proposed openstack-infra/project-config: [trove] Add more nv scenario tests to the check  https://review.openstack.org/35488118:10
* fungi is back, and catching up on scrollback before prepping for 20:00 gerrit maintenance and 21:00 wiki maintenance18:12
fungiso many nick highlights...18:12
openstackgerritMerged openstack-infra/tripleo-ci: Fix resetting /etc/hosts  https://review.openstack.org/35191718:14
Shrews~/b 1118:15
Shrewsgah18:15
*** _nadya_ has quit IRC18:16
*** Na3iL has quit IRC18:16
*** chem is now known as chem|off18:16
anteayapabelanger: let's give fungi a few minutes to catch up on backscroll?18:16
*** baoli has joined #openstack-infra18:17
pabelangeranteaya: agreed18:17
mordredShrews: ~/b 11 indeed18:18
Shrewsmordred: and i meant it18:18
fungiZara: v.e.r.a. (the virtual entity of relevant acronyms) is my go-to for acronym lookup. luckily, the `dict` command-line utility consults it by default. though it's not so good for slang... for that the jargon file/hacker dictionary tends to be a better resource (also consulted by `dict`)18:18
nibalizerfungi: over in -social there is some discussion of an openstack jargon file18:19
nibalizeri hope it happens18:19
nibalizerShrews:18:19
nibalizerer, mistab sorry18:19
openstackgerritGraham Hayes proposed openstack-infra/project-config: Remove grenade from designate stable branches  https://review.openstack.org/35488818:20
mugsiecan I get someone to look at ^ review please - designate's stable gate is currently bocked18:20
*** sdake has joined #openstack-infra18:21
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: DO NOT MERGE - Periodic test  https://review.openstack.org/34694918:22
fungiif devkulkarni shows back up, that list of kernels looks probably accurate to me though i haven't manually confirmed the results18:22
anteayamugsie: ummm, is removing grenade from stable branches in line with stable policy?18:23
cloudnullfungi mordred pabelanger: and others if they're interested -- this is the network setup w/in the OSIC for IPv6 http://cdn.pasteraw.com/73rflczdcnk284u33w99h7d8sk5ho4v -- RE: https://review.openstack.org/#/c/35482718:24
*** asettle has quit IRC18:24
*** asettle has joined #openstack-infra18:25
*** amcguign has joined #openstack-infra18:25
fungizaro: i don't think we've _actually_ seen issues we attribute to a pileup of idle ssh sessions (perhaps because they get closed by our semi-frequent gerrit restarts?), so it doesn't seem urgent to me for now18:25
*** rbuzatu has joined #openstack-infra18:26
*** devananda has joined #openstack-infra18:26
*** adrian_otto has quit IRC18:26
*** amcguign has quit IRC18:26
mugsieanteaya: sorry, I realised I was not clear in the review - I added a comment18:27
mugsiewe dont actually ahve a plugin to test it in mitaka or liberty18:27
mugsiewe added it this cycle18:27
anteayamugsie: well I was actually asking based on the commit title18:27
rcarrillocruzpabelanger: last time i checked, it was a limitation on the ansible openstack modules18:28
fungialso, removing normal grenade jobs (though not grenade-forward) from stable/liberty is necessary since we don't have stable/kilo around any longer18:28
anteayafungi: ack18:28
rcarrillocruzeven though the docs state you can pass a region, that is only used in case you pass an auth dict, not an oscc cloud18:28
fungiso trying to run a pull-up grenade job on liberty just plain won't work18:28
rcarrillocruzworkaround: having an oscc cloud per region18:29
rcarrillocruzi know it's bad, but it works18:29
*** asettle has quit IRC18:29
rcarrillocruzthat's some tech debt i chatted with Shrews at some point, we may want to implement it18:29
Shrewseh?18:29
*** ddieterly[away] has quit IRC18:30
Shrewsoh, the multi-region thing18:30
rcarrillocruzShrews: if you remember, i  mentioned a while ago it would be good we would handle regions on 'cloud' parameter18:30
rcarrillocruzas it's only used on auth dicts18:30
pabelangermordred: had a moment to help debug an issue on puppetmaster?  http://paste.openstack.org/show/556152/18:31
pabelangermordred: we are no longer running puppet on all of our hosts18:31
*** rbuzatu has quit IRC18:31
pabelangermordred: for example, nodepool.o.o18:31
*** sarob has joined #openstack-infra18:31
*** weshay is now known as weshay_mtg18:33
*** senk_ has quit IRC18:34
pabelangermordred: we are running shade 1.10.0 on puppetmaster18:34
mugsieanteaya: fungi thank you for the quick reviews!18:35
mordredpabelanger: k. looking18:35
*** sarob has quit IRC18:35
pabelangermordred: a quick test of reverting seems to fix the problem18:35
anteayamugsie: thanks for explaining18:35
*** sarob has joined #openstack-infra18:35
mordredpabelanger: yah - that'll be the thing we just landed a patch to fix18:35
fungi#status log Added wiki.openstack.org to /etc/ansible/hosts/emergency on puppetmaster.openstack.org in preparation for 21:00 UTC upgrade maintenance18:36
openstackstatusfungi: finished logging18:36
pabelangermordred: okay, great18:36
mordredpabelanger: in about 20 minutes we'll cut a 1.10.118:36
pabelanger++18:36
*** tkelsey has quit IRC18:37
Shrews.1 revisions... a.k.a. the "oopsie" release18:37
openstackgerritMerged openstack-infra/elastic-recheck: Remove query for duplicate bug 1450110  https://review.openstack.org/35476018:37
openstackbug 1450110 in Cinder "boto tests randomly fail in resource_cleanup with "Failed to delete volume"" [High,Fix released] https://launchpad.net/bugs/145011018:37
mordredShrews: ++18:37
fungibrown-bag releases ftw18:38
mordredShrews: maybe if we didn't have to re-implement floating ip metadata tracking client-side ...18:38
* mordred cries18:38
fungimordred: you actually expect openstack to work? for realz?18:38
mordredfungi: ha. hahahaha. BWAHAHAHAHAHAHAHAHA18:39
*** krtaylor has joined #openstack-infra18:39
mordredfungi: I have no expectations of anything18:39
fungiyou haven't just toured the sausage factory, after all18:39
mordredwell. NOW I expect sausage18:39
mordreddo you think we could start a sausage making project in openstack?18:39
fungias long as it can produce a passable andouille, i'm on board18:40
cloudnull++ ^18:40
fungioh, and also kamaboko18:40
mordredfungi: of course, what we'll get is a thing that looks like sausage but also has an arm sticking out of it and probably also a tile roof18:40
* fungi loves him some surimi18:40
*** devkulkarni1 has joined #openstack-infra18:41
cloudnullhahaha18:41
anteayaha ha ha18:41
* anteaya is so glad she is a vegetarian18:41
anteayadoes it blend?18:41
cloudnulllol18:41
*** whoops has quit IRC18:41
*** amotoki has quit IRC18:42
mordredanteaya: the blackberry milkshade I just had contained no sausage, arms or tile roofs18:42
mordredmilkshake18:42
mordredsigh18:42
fungimilkshade!18:42
fungifreshly renamed for the 1.10.1 release18:43
anteayamilkshade sounds awesome18:43
anteayaha ha ha18:43
*** whoops has joined #openstack-infra18:44
mordredI need to write a project called milkshade now18:44
openstackgerritKevin Carter (cloudnull) proposed openstack-infra/project-config: Increase the quota in the OSIC cloud1  https://review.openstack.org/35490018:45
*** sdake has quit IRC18:45
cloudnull^ just figured I'd queue that up assuming all is right with the world18:46
* cloudnull knows that its not, but one can hope18:46
mordredzomg. that's gonna be yes18:46
anteayacloudnull: toss a workflow -1 on that until we are ready for it18:48
openstackgerritMerged openstack-infra/project-config: Switch osic-cloud1 to IPv6  https://review.openstack.org/35482718:48
nibalizermmmm ipv618:48
cloudnull^ hold on to your butts18:48
cloudnullanteaya: done18:49
openstackgerritMerged openstack-infra/nodepool: Include subnodes in updateStats function v2  https://review.openstack.org/35143518:49
cloudnullthanks for the review btw18:49
mordreddude. it's totally going to just work18:49
anteayathanks for the quota18:49
openstackgerritMerged openstack-infra/storyboard-webclient: 'All Boards and Worklists' Page  https://review.openstack.org/27975318:49
*** tonytan4ever has joined #openstack-infra18:50
cloudnullwhen do we expect the first jobs to be queued ?18:50
openstackgerritMerged openstack-infra/shade: Add update_endpoint()  https://review.openstack.org/28905218:50
* cloudnull is monitoring the cloud18:51
mordredpabelanger: I think I see a place in nodepool where this logic may fall apart18:51
mordredbut let's see how it does18:51
mordredif it does fall apart, I think I know how to fix it18:52
anteayacloudnull: as soon as zuul and nodepool know it is there and have some quota for it, not sure how long it will take18:53
pabelangermordred: okay, until we fix shade we cannot access nodepool.o.o from puppetmaster.18:53
pabelangerwhich means ipv6 patch won't land18:53
mordredah. neat18:53
AJaegercloudnull: why -1 on your own change? Better use WIP - that blocks it from merging18:53
openstackgerritPeter Stachowski proposed openstack-infra/project-config: [trove] Add more nv scenario tests  https://review.openstack.org/35488118:53
cloudnullAJaeger: because I pushed the wrong button :)18:54
AJaeger;)18:54
mordredpabelanger: ok. it's not going to work today - I see a bug18:54
*** tonytan4ever has quit IRC18:55
pabelangermordred: okay18:55
mordredpabelanger: I think it's an easy bug to fix - but it's going to take a patch18:56
mgagneanyone knows if there is a way to tell graphite to be more tolerant to missing values? There are holes in my graph and it messes up sumSeries =)18:56
mgagnekeepLastValue =)18:57
*** tosky has quit IRC18:58
AJaegerpabelanger, fungi, we still have jobs in the release queue for 54 hours now - due to waiting for wheel builds...18:59
AJaegerthey're building slowly, so making progress. Still, a long time...19:00
pabelangerAJaeger: yes, we should likely rector those jobs to remove wheel-build on release. And just rely on a periodic job for now19:00
openstackgerritMerged openstack-infra/shade: Infer nova-net security groups better  https://review.openstack.org/35249819:01
pabelangerotherwise, we keep them as they exist today and it will be slow going19:01
*** e0ne has quit IRC19:01
*** fifieldt has quit IRC19:03
AJaegeryeah, might be the best way forward. I'm off in a few minutes so can't help here...19:03
fungiAJaeger: yeah, they don't hurt anything. those jobs are basically idempotent (or at least not actually specific to the releases that trigger them). we could just as well run them in a continuous loop completely detached from the release pipeline if we wanted19:03
*** vhosakot has quit IRC19:04
AJaegerfungi, they block right now the periodic job ;(19:04
fungiAJaeger: well, the periodic wheel build jobs are also basically identical19:05
fungiso that's presumably not hurting anything either19:06
EmilienMAJaeger: https://review.openstack.org/#/c/354977/19:06
EmilienMAJaeger: we have 34 repositories in Puppet19:06
EmilienMand stable/liberty and stable/mitaka :)19:06
EmilienMyou might need some help :)19:06
AJaegerEmilienM: you get 34 reviews ;)19:06
EmilienMand a script19:07
*** krtaylor has quit IRC19:07
AJaegerEmilienM: I have a script ;)19:07
AJaegerI haven't done stable branches since we kept the old name for compatibility19:07
AJaegerEmilienM: if you want to backport, go ahead.19:08
*** vkramskikh has quit IRC19:08
*** thiagolib has quit IRC19:08
*** vkramskikh has joined #openstack-infra19:08
EmilienMAJaeger: well, if there is no need, I won't backport it19:08
*** krtaylor has joined #openstack-infra19:08
AJaegerIf you want me to use my script to rename all the branches, I can do so in 10 days - after my vacation...19:08
EmilienMit's lot of CI resources19:08
EmilienMI'm not sure we want to waste it19:08
EmilienMif you keep it backward compatible until liberty & mitaka are gone, then don't backport it19:09
AJaegerEmilienM: Yes, we'll keep that for some time...19:09
AJaegerEmilienM: sorry that I don't have a script for you to auto-review those 34 changes ;)19:10
*** tonytan4ever has joined #openstack-infra19:10
EmilienMthat would be a nice feature in Gertty19:10
anteayayou can batch approve in gertty19:10
EmilienMbeing able to review a whole topic19:10
anteayayou can I believe19:11
EmilienMhow?19:11
AJaegernever done it - topic is "bindep-mv"19:12
anteayamordred: do you know how to batch approve in gertty?19:12
anteayaI do not, just heard about its mystical properties19:12
fungiAJaeger: EmilienM: i anticipate keeping that backward-compatible forever if possible19:12
mordredanteaya: yes19:13
fungithe resulting tech debt is maybe two lines of code19:13
AJaegerfungi, yeah...19:13
EmilienMfungi: ++19:13
mordredanteaya: you use the % to mark changes19:13
mordredanteaya: and then you press r to vote on the things you have marked19:13
mordredanteaya: you must do this in a change list screen19:13
AJaegermordred: does it work for several repos?19:13
EmilienMcan someone approve https://review.openstack.org/#/c/354172/ please?19:13
mordredyes19:13
fungianteaya: if it's not obvious, you do it from a change list (resulting from a query or from browsing a project). the context help calls % a "process mark"19:14
openstackgerritAndreas Jaeger proposed openstack/python-jenkins: Move other-requirements.txt to bindep.txt  https://review.openstack.org/35501219:14
anteayamordred: fungi thank you19:14
AJaegerEmilienM: My bot ^ ;)19:14
anteayaEmilienM: does that help?19:14
EmilienMcool19:14
pabelangerfungi: raddaoui was just telling there might be some additional cloud resources available for openstack-infra.19:14
fungiassuming AJaeger uses a consistent change topic for them, you could easily query by topic19:15
pabelangerfungi: raddaoui: figured I'd pass along the information19:15
mrhillsmanpabelanger raddaoui works with myself and inc019:15
raddaouio/19:15
fungipabelanger: raddaoui: exciting! have you seen http://docs.openstack.org/infra/system-config/contribute-cloud.html yet?19:15
mrhillsmanhe is the primary contact at this point after us getting the networking stuff resolved19:15
anteayaEmilienM: I question your comment on line 1483: https://review.openstack.org/#/c/354172/7/zuul/layout.yaml19:15
fungimrhillsman: oh, got it. there are so many of you19:16
pabelangermrhillsman: Ah, great19:16
raddaouifungi: looking at it ^^19:16
fungipabelanger had said something about raddaoui offering resources from cloud8, which i didn't realize was connected to osic19:16
*** roxanaghe has quit IRC19:16
pabelangerindeed, some confusion on my part19:17
raddaouihaha yeah it is from OSIC19:17
fungiNOW i understand ;)19:17
fungithanks!19:17
mrhillsmanwe want to make sure you all know us :)19:17
*** raunak has quit IRC19:17
*** weshay_mtg is now known as weshay19:17
mrhillsmanand spread the word of how helpful osic is ;)19:17
fungihopefully i'll eventually have an opportunity to encounter all of you in person at our conferences19:18
anteayaEmilienM: I think you want stable/mitaka there, I think that is a copy/paste error19:18
mrhillsmandefinitely19:18
fungi(and if i already have, apologies, i'm terrible at remembering who i've met sometimes)19:18
*** senk_ has joined #openstack-infra19:18
*** fifieldt has joined #openstack-infra19:18
fungii have enough trouble remembering what i had for breakfast this morning19:18
mrhillsmani only know nibalizer19:19
mrhillsmanat least that i know of19:19
fungihe's hard to forget19:19
mrhillsmanlol19:19
mrhillsmanindeed19:19
AJaegerEmilienM: happy reviewing ;) All changes submitted...19:19
fungi(not that i've tried to forget him, but...)19:19
* AJaeger signs off now, will be offline the next 10 days19:20
pabelangerAJaeger: enjoy19:20
anteayaAJaeger: have fun at the sea19:20
*** nwkarsten has quit IRC19:20
AJaegerThanks, pabelanger and anteaya ! I'll do my best, anteaya ;)19:20
*** AJaeger has quit IRC19:20
anteaya:)19:20
*** ddieterly has joined #openstack-infra19:21
*** nwkarsten has joined #openstack-infra19:21
EmilienManteaya: ok I'll fix it in a future patch19:21
anteayasure pabelanger just approved that one19:21
anteayait was small so I didn't leave my concern on the patch19:21
*** rbuzatu has joined #openstack-infra19:22
pabelangerYa, just noticed that ping in #tripleo19:22
pabelangermordred: is your bug fix in shade or nodepool?19:23
*** rajinir has quit IRC19:25
*** nwkarsten has quit IRC19:25
mordredpabelanger: might be both. still working on it19:25
mordredactually, no. I believe we're fine on the nodepool front19:26
pabelangerHmm, I think we just had a network hiccup in bluebox. See grafana:19:26
pabelangerhttp://grafana.openstack.org/dashboard/db/nodepool-blue-box?from=1471026368251&to=147102996825119:26
*** rbuzatu has quit IRC19:26
*** rbuzatu has joined #openstack-infra19:26
pabelangerlooking at a the job failures in status.o.o/zuul, looks to ansible-launcher failed to SSH into the host19:27
pabelanger19:14UTC19:27
*** thorst_ has quit IRC19:27
EmilienMthis rename is going to cost lot of CI jobs19:28
*** senk_ has quit IRC19:28
*** devkulkarni2 has joined #openstack-infra19:28
pabelangergood thing it is the weekend tomorrow19:28
EmilienMyeah but if you submit a patch *now* it won't be tested right away :(19:28
pabelangerEmilienM: your gate failures look to be a bluebox networking issue19:29
*** devkulkarni1 has quit IRC19:29
pabelangertoday has been good, about 4.5hours to run in check19:29
pabelangerbest day this week19:29
EmilienMwell, 493 jobs are in check now19:29
EmilienMok19:29
*** rbrndt has joined #openstack-infra19:30
*** esberglu_ has joined #openstack-infra19:30
*** esberglu has quit IRC19:31
*** esberglu_ is now known as esberglu19:31
openstackgerritMerged openstack-infra/tripleo-ci: Bump baremetal vm memory to 5.5 GB  https://review.openstack.org/35365519:32
openstackgerritPaul Belanger proposed openstack-infra/project-config: Resolve DNS soon in configure_mirror.sh  https://review.openstack.org/35504719:34
*** tkelsey has joined #openstack-infra19:35
*** thorst_ has joined #openstack-infra19:35
fungipabelanger: are those bluebox failures tightly clustered?19:37
*** devkulkarni2 has quit IRC19:37
fungiwould probably be nicer if zuul-launcher requeued when it can't contact the node to start the job19:37
openstackgerritMonty Taylor proposed openstack-infra/shade: Base auto_ip on interface_ip not public_v4  https://review.openstack.org/35504819:37
mordredpabelanger, Shrews: ^^19:37
*** devkulkarni1 has joined #openstack-infra19:38
nibalizerhi19:38
nibalizermrhillsman: whats up?19:38
mordredShrews, pabelanger: this is one of those that the semantics of the patch are way more complex than the 2 line change itself19:38
mordredjeblair: ^^ if you have a sec, I'd appreciate your eyes on that too19:38
nibalizeroh right, yes we met19:38
pabelangerfungi: yes, we seem to have deleted about 20 nodes from what I see, all around the same time19:39
*** tkelsey has quit IRC19:39
mordredShrews, pabelanger, jeblair: for context, interface_ip is set in shade._meta.add_server_interfaces which is called for every server returned by list_servers19:40
*** thorst_ has quit IRC19:40
pabelangerfungi: I think the better fix is to talk with our ansible friends and expose a new setting for tasks to ignore SSH failures for async.  And continue to run, regardless of an SSH failure, until our timeout is reached19:40
*** oanson has quit IRC19:41
mordredpabelanger: I actually have a thought on that ... but my head is too much on the other thing to expand on it19:41
*** devkulkarni1 has quit IRC19:41
mordredpabelanger: but I think it's a thing we can solve zuul-side19:41
*** devkulkarni1 has joined #openstack-infra19:41
pabelangerfungi: actually, we do requeue under certain failure, like -9. Let me check what ansible-playbook returns when we have an SSH failure during a run19:42
pabelangermordred: oh, neat19:42
Shrewsmordred: looking19:42
pabelangermordred: fungi: Actually, we should have something today: https://review.openstack.org/#/c/328341/19:45
pabelangerlet me see why that didn't work19:45
mordredpabelanger: oh indeed19:45
*** elo has quit IRC19:46
pabelanger2016-08-12 19:13:18,904 DEBUG zuul.NodeWorker.centos-7-2-node-bluebox-sjc1-3444563: Ansible exit code: 119:48
pabelanger:(19:48
*** matrohon has joined #openstack-infra19:49
mordredblerg19:49
*** liusheng has quit IRC19:50
mrhillsmannibalizer nothing19:50
*** liusheng has joined #openstack-infra19:50
mrhillsmanfungi was just saying how you made a terrible impression on him19:50
nibalizerhaha19:50
Shrewsmordred: so, that change switches one logic path with another, and i don't see where the interface_ip path shares any of the logic that get_external_public_ipv4() has (for v4 addresses)19:50
pabelangermordred: apparently ansible doesn't yet support AnsibleHostUnreachable: https://github.com/ansible/ansible/blob/devel/bin/ansible#L10619:51
Shrewsmordred: i just have a gut feel that is going to break someone, but i have nothing concrete19:51
pabelangermordred: going to ask in ansible-devel19:51
openstackgerritMerged openstack-dev/cookiecutter: sync latest requirements.txt  https://review.openstack.org/35141719:52
*** Apoorva has quit IRC19:52
mordredShrews: the only ones I can think of where it might be a problem is split-stacks like the deprecated region at dreamhost where you get a public v6 and a private 4 by default. this could result in, in those situations, auto_ip not making a floating ip when you were expecting it to19:52
mordredShrews: although if that happens you should be able to add force_ipv4: True to your clouds.yaml19:52
fungii'm going to go ahead and approve https://review.openstack.org/352546 since the upgrade restart maintenance is scheduled for 5 minutes from now19:54
fungiinfra-root: ^19:54
fungifyi19:54
mordredShrews: it does use the output of the old logic though19:54
mordredShrews: at the end of _get_interface_ip it'll return server['public_v4']19:55
fungiif necessary i'll stop puppet for review.o.o and roll back manually if it won't start or looks suspect once up19:55
mordredShrews: which is set from get_server_external_ipv419:55
*** adrian_otto has joined #openstack-infra19:55
fungishould hopefully be brief enough that we don't need to tear down zuul19:55
mordredShrews: which is what the old code was looking at19:55
Shrewsmordred: oh, well that's true19:56
anteayafungi: ack19:56
*** esberglu_ has joined #openstack-infra19:56
mordredShrews: so I think the easiest way to characterize this is that it's putting a check to see if you've configured a default_network or ipv6 first19:56
mordredand if so, returning those, else returning what it returned before19:56
*** bknudson has left #openstack-infra19:57
Shrews*nod*19:57
*** esberglu has quit IRC19:58
*** esberglu_ is now known as esberglu19:58
mordredShrews: possibly one of the more complex 2 line changes I've done in a while19:59
*** gouthamr has quit IRC19:59
mordredShrews: this one could probably use with a release note19:59
fungizaro: i just remembered... we still have our puppet module set to trigger an offline reindex on upgrade i think? http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/manifests/gerrit.pp#n7519:59
fungizaro: is it safe to just kill the reindex java process?19:59
*** karthik__ has quit IRC19:59
*** esberglu has quit IRC20:00
fungiotherwise i'll find some quick way to bypass it20:00
openstackgerritDean Troyer proposed openstack-infra/project-config: Add lib-forward-testing to os-client-config and osc-lib  https://review.openstack.org/35505220:00
*** karthik__ has joined #openstack-infra20:01
*** devkulkarni1 has quit IRC20:01
*** esberglu has joined #openstack-infra20:01
*** esberglu has quit IRC20:01
*** esberglu has joined #openstack-infra20:02
Shrewsmordred: well, i can't immediately poke more holes in it20:02
openstackgerritMerged openstack-infra/project-config: Remove grenade from designate stable branches  https://review.openstack.org/35488820:03
*** esberglu has quit IRC20:03
*** ddieterly has quit IRC20:04
*** esberglu has joined #openstack-infra20:04
*** esberglu has quit IRC20:04
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Upgrade gerrit on review.o.o  https://review.openstack.org/35254620:04
fungi^ needed a rebase because its parent commit changed. enqueuing now20:04
Zara:D20:05
*** raunak has joined #openstack-infra20:05
*** esberglu has joined #openstack-infra20:05
*** esberglu has quit IRC20:05
pabelangermordred: fungi: okay, our usage of fail task is the issue. that is forcing the exit code to 120:05
pabelangermordred: fungi: So, we can likely fix it, let me do some more testing and get a patch20:06
*** esberglu has joined #openstack-infra20:06
*** sigmavirus is now known as sigmavirus24|awa20:06
*** sigmavirus24|awa is now known as sigmavirus|away20:06
Shrewspabelanger: sorry, missed something... how is a task failure causing exit code of 1 breaking something? that seems like something that's intentional?20:07
pabelangerShrews: https://review.openstack.org/#/c/328341/20:07
pabelangerShrews: basically, if we get a network issue, we could have zuul requeue the job20:07
pabelangerShrews: which worked at some point20:08
Shrewspabelanger: ah, i see20:08
pabelangeradding the fail task, seems to be the reason it stopped working20:08
*** spzala has quit IRC20:08
*** xyang1 has quit IRC20:08
*** spzala has joined #openstack-infra20:08
pabelangerhttps://review.openstack.org/#/c/324726/ was the original issue20:09
*** esberglu_ has joined #openstack-infra20:10
*** esberglu has quit IRC20:10
*** esberglu_ is now known as esberglu20:10
*** kzaitsev_mb has joined #openstack-infra20:10
fungizaro: around? did you see my question about our puppet currently being configured to force an offline reindex when installing a new gerrit build?20:12
*** spzala has quit IRC20:13
*** nwkarsten has joined #openstack-infra20:13
Shrewspabelanger: looking at that, if runAnsiblePlaybook() returns "ret == 0", how was that code ever going to get a return value of 3?20:13
*** inc0 has joined #openstack-infra20:13
fungii'm going to make a quick backup of our current index and hope that i can roll back to it if killing the reindex process screws things up20:14
Shrewspabelanger: in 328341, that is20:14
openstackgerritMerged openstack-infra/system-config: Upgrade gerrit on review.o.o  https://review.openstack.org/35254620:14
pabelangerShrews: maybe that is the issue20:15
pabelangerShrews: ya, that code changed a while back20:16
Shrewspabelanger: it certain seems like *an* issue  :)20:16
Shrewscertainly20:16
pabelangerShrews: we do the check inside runAnsiblePlaybook today20:16
pabelangerShrews: here you can see the issue: http://logs.openstack.org/37/319337/11/gate/gate-tripleo-ci-centos-7-nonha-multinode/f4fb7d7/_zuul_ansible/ansible_log.txt20:17
pabelangernode                       : ok=16   changed=14   unreachable=0    failed=120:17
pabelangerbecause we used fail20:17
openstackgerritMerged openstack-dev/cookiecutter: don't need to omit this as oslo libs are now used  https://review.openstack.org/35025120:17
*** gouthamr has joined #openstack-infra20:17
pabelangerlocal testing shows it would be node                       : ok=16   changed=14   unreachable=1    failed=020:18
Shrewspabelanger: ah, yeah. current code does the right thing20:18
Shrewsi thought i'd seen that issue before20:18
pabelangerI'll have to look more at this next week20:20
rderose_irc_nick120:21
*** xyang1 has joined #openstack-infra20:22
mordredpabelanger, Shrews: I think I should release what we have in shade currently, since shade/ansible are broken for any cloud that does not have floating ips right now20:23
openstackgerritPeter Stachowski proposed openstack-infra/project-config: [trove] Add more nv scenario tests  https://review.openstack.org/35488120:23
mordredand then we can ponder the ipv6 enablement when it's not end of the day on friday20:23
Shrewsmordred: do eet20:24
pabelangersure20:24
pabelangerlet me revert our ipv6 change then20:24
jeblairyou had fun while i lunched20:24
mordredpabelanger: it shouldn't break anything20:24
Shrewsmordred: is EOD friday not a good time to make a change we are unsure of?20:24
mordredwell - wait - no, it will - yes please revert20:24
mordredShrews: of course it is!20:25
zarofungi: are we ready to upgrade already?20:25
mordredjeblair: I want to claim your first attention spot20:25
fungizaro: yes, i announced it for 20:00 today20:25
mordredjeblair: also, welcome back from lunch20:25
jeblairmordred: for 048?  i'm reading it now :)20:25
mordredcool20:25
*** yamahata has quit IRC20:25
*** dprince has quit IRC20:25
fungizaro: the upgrade patch merged about 4 minutes ago, so i'm just waiting for puppet to apply it now20:26
openstackgerritPaul Belanger proposed openstack-infra/project-config: Revert "Switch osic-cloud1 to IPv6"  https://review.openstack.org/35506320:26
pabelangermordred: ^20:26
zarofungi: ohh i thought it was 2 hrs from now.  anyways.  i forgot puppet would offline reindex20:26
cloudnull:'(20:26
zaromaybe we can manually stop the offline reindex?20:26
*** Apoorva has joined #openstack-infra20:27
*** rderose_ is now known as rderose20:27
*** rajinir has joined #openstack-infra20:27
zarofungi: is offline reindex going to happen?20:27
fungizaro: that's what i asked you. is it safe to kill it?20:28
mordredcloudnull: it's not you, it's me20:28
rajinirIs there a way to restart services before running the tests using devstack-gate?20:28
zarofungi: yes, i believe so. i've done it may times before and everything servives20:28
fungizaro: awesome. i'm basically watching the process list and as soon as i see the reindex pop up i'll kill it and start gerrit manually20:28
zarofungi: i guess it's time to remove the reindex thing from puppet.20:29
cloudnullmordred: well when its ready, everything is in place to up the quota to 12820:29
mordredcloudnull: we're very excited about this20:29
fungizaro: agreed, well past time in fact20:30
cloudnulland assuming all is well, i'd love to see us @ 256 > 512 in quick order.20:30
jeblairmordred: would it make sense to add a test to 048 to simulate the current situation and show how 048 fixes it (and makes us more robust in the future)?20:30
fungizaro: ever since the 2.11 upgrade in particular, having gerrit automatically invoke a 4-hour downtime has not really been something we've wanted20:30
fungier, s/gerrit/puppet/20:30
mordredjeblair: yah. I thnk so. I'm also going to write a release note covering it in more detail20:30
mordredjeblair: since I think cramming that into a release today is not a good idea20:31
*** valderrv has quit IRC20:31
zarofungi: on another note, i've been pumping changes into review-dev. slow going though.  up to 70K changes  now.20:32
zarotried the online reindex on that and it took about 1 hr.  you think that's enough time to test things while reindexing?20:32
*** sdake has joined #openstack-infra20:32
*** shardy has quit IRC20:32
*** ihrachys has joined #openstack-infra20:33
fungizaro: definitely20:33
zaroi guess it'll be a good time to stop then.20:34
fungipabelanger: i saw you mention something earlier about ansible/shade problems... should i expect puppet apply to be getting triggered on our servers right now?20:34
fungithis is taking longer than i expected and need to start debugging so the maintenance window doesn't run over20:35
jeblairmordred: cool.  i agree that the current code path adds a fip to an ipv6 only host, and it looks like 048 should fix that.  i'm going to +1 but i think we should wait on a test to merge it.20:35
zarofungi: puppet is taking a long time?20:35
*** tkelsey has joined #openstack-infra20:36
fungizaro: yeah, i expected the upgrade to have triggered by now20:36
fungistarting to dig into our automation logs20:36
pabelangerfungi: not until mordred releases the new version of shade20:36
*** roxanaghe has joined #openstack-infra20:36
fungithis is unfortunate timing, for sure20:37
fungitime to remind myself how to do this manually20:37
jeblairfungi: how are you running it?20:38
fungijeblair: i'm not, yet anyway20:38
fungijeblair: i was waiting for our ansible cron to trigger the upgrade now that the patch has merged20:38
*** _ari_ has quit IRC20:39
jeblairfungi: 2016-08-12 20:32:57,832 p=11157 u=root |  PLAY [review.openstack.org:!disabled] ******************************************20:39
fungimaybe i'll just manually download the warfile and restart gerrit20:39
mordredpabelanger: I have just released 1.11.020:39
jeblair2016-08-12 20:32:57,832 p=11157 u=root |  skipping: no hosts matched20:39
fungijeblair: does that mean review.openstack.org is not disabled (!disabled)?20:39
mordredpabelanger: which should fix puppet20:39
fungioh, perhaps not20:39
fungijeblair: i thought that was related to how we stage git.* updates before review.o.o updates20:40
*** tkelsey has quit IRC20:40
fungiour update dependency ordering to account for manage-projects and create-cgitrepos20:40
jeblairfungi: the no hosts matched which follows that makes me think it's not running there20:40
fungijeblair: yeah, we have a stage where all servers that aren't git.* or review are updated, then a stage where git.* are updated, then a stage where review is updated20:41
fungior something along those lines20:41
fungireview.o.o is definitely not disabled in the usual lists anyway20:41
*** rhallisey has quit IRC20:43
*** jkilpatr has quit IRC20:43
jeblairfungi: i think that if it's not disabled, then ansible doesn't think it exists any more.  the last time it touched review.o.o was 15:33:40,94820:43
mordredroot@puppetmaster:~# ansible review.openstack.org --list-hosts20:43
mordred [WARNING]: provided hosts list is empty, only localhost is available20:43
pabelangerright, this is the shade issue20:43
mordredah20:43
mordredyah20:43
fungijeblair: yeah, this seems possibly related to what pabelanger/mordred/et al are trying to fix for some hours now20:43
pabelangerinternap fails on FIPs I think20:44
mordredyah. which I just relased for20:44
fungii'm just going to try to upgrade gerrit manually for now20:44
fungizaro: will gerrit download the its-storyboard plugin when it gets called with init?20:45
zarono,20:45
zarono, its-storyboard comes from tarballs and puppet handles that seperately.20:46
fungiso i need to download that into the plugins directory?20:46
fungigot it20:46
zaroit should probably already been in review_site/plugins20:46
fungiconfirming20:46
fungizaro: indeed, it is. thanks20:46
zaroi believe the change to install it puts it in review_site/plugins20:46
fungiyep. definitely already present20:46
*** rfolco has quit IRC20:46
*** weshay is now known as weshay_afk20:47
zarojust need to run `java -jar gerrit.war init --batch --no-auto-start -d review_site`20:47
*** inc0 has quit IRC20:47
fungizaro: thanks, that matches what i copied from the puppet manifest. about to start now that i've updated the warfile20:48
fungi#status notice The Gerrit service on review.openstack.org is restarting for a scheduled upgrade, but should return to service momentarily: http://lists.openstack.org/pipermail/openstack-dev/2016-August/101394.html20:48
openstackstatusfungi: sending notice20:48
mordredfungi, pabelanger: shade is updated on puppetmaster20:48
*** freerunner has quit IRC20:49
-openstackstatus- NOTICE: The Gerrit service on review.openstack.org is restarting for a scheduled upgrade, but should return to service momentarily: http://lists.openstack.org/pipermail/openstack-dev/2016-August/101394.html20:49
fungigerrit is coming back up now20:51
openstackstatusfungi: finished sending notice20:51
*** freerunner has joined #openstack-infra20:51
fungii'm getting a webui now20:51
fungiPowered by Gerrit Code Review (2.11.4-22-ge0c0f29)20:52
zaroahh cool20:52
*** oomichi has joined #openstack-infra20:52
fungieverybody with a few spare cycles, please double-check the interface20:52
fungii need to finalize prep for the wiki upgrade maintenance starting in 7 minutes20:52
*** karthik__ has quit IRC20:52
* zaro preps a change with a task20:53
mordredthe interface looks good in general20:53
pabelangermordred: /etc/ansible/hosts/openstack --list is having some issue on puppetmaster, care to look?20:54
mordredpabelanger: I'd love to!20:54
pabelangersadly, I have to head out in a few more minutes20:55
fungi#status alert The Mediawiki service at wiki.openstack.org will be offline from 21:00 UTC until approximately 23:00 UTC for a planned upgrade http://lists.openstack.org/pipermail/openstack-dev/2016-August/101395.html20:55
openstackstatusfungi: sending alert20:55
mordredpabelanger: what sort of issues did you see?20:56
mordredpabelanger: for me it's just taking a while20:56
*** jkilpatr has joined #openstack-infra20:56
pabelangermordred: http://paste.openstack.org/show/556167/20:56
*** raildo has quit IRC20:57
pabelangerthere we go20:57
pabelangerI logged out and back in20:57
-openstackstatus- NOTICE: The Mediawiki service at wiki.openstack.org will be offline from 21:00 UTC until approximately 23:00 UTC for a planned upgrade http://lists.openstack.org/pipermail/openstack-dev/2016-August/101395.html20:57
pabelangernot sure why that fixed it20:57
*** ChanServ changes topic to "The Mediawiki service at wiki.openstack.org will be offline from 21:00 UTC until approximately 23:00 UTC for a planned upgrade http://lists.openstack.org/pipermail/openstack-dev/2016-August/101395.html"20:57
*** gouthamr has quit IRC20:58
mordredpabelanger: thanks. I'm re-running it right now in debug to watch20:58
mordredjust to make sure20:58
pabelangerokay, I'll check back in a bit20:59
mordredShrews: zomg. I'm going to make a config option for ansible inventories to turn off the fetching of flavor extra_specs20:59
fungifyi, this is the rough upgrade plan i'm following for the wiki https://etherpad.openstack.org/p/mediawiki-1.27-upgrade-plan21:00
*** jkilpatr has quit IRC21:00
fungistep 0 was done a couple hours ago before i started working on the gerrit maintenance21:00
openstackstatusfungi: finished sending alert21:01
*** psilvad has quit IRC21:01
zaroahh Gerrit SB integration not working yet because its-storyboard has not been enabled yet.  would love to get an infra-root to help with that.21:02
*** burgerk has quit IRC21:03
fungizaro: is that just a gerrit config update and another restart? if so, someone can probably knock it out this weekend (maybe even me)21:03
zaroit's a change to project.config  directions in commit message of change https://review.openstack.org/#/c/347486/21:04
zarorestart not required21:04
mordredSTAB STAB STAB21:07
sdakehey folks21:07
sdakesomething changed related to the gating - and kolla-kubernetes non-voting gates turned into voting gates21:08
sdakeany idea why?21:08
kevinbentonOMG OMG the javascript to move the CI results to the side isn't working. The sky is falling! :)21:08
sdakei've been trying  to decode the project-config to see what changed21:08
sdakeandd i see alot of xenial changes and whatnot21:08
sdakeand I'm not sure what to change to make the jobs non-voting21:09
*** karthik__ has joined #openstack-infra21:09
fungizaro: any guess if ui dom elements changed that might break references in hideci.js?21:09
*** valderrv has joined #openstack-infra21:09
zarotoggle ci button still works for me21:10
fungizaro: yeah, but the results table is no longer rendering21:10
*** amitgandhinz has quit IRC21:10
*** hockeynut has joined #openstack-infra21:11
openstackgerritMonty Taylor proposed openstack-infra/shade: Deal with clouds that don't have fips betterer  https://review.openstack.org/35508321:11
*** sarob has quit IRC21:11
mordredfungi, Shrews: ^^ that is a smidge of an emergency patch - I have tested that it fixes the issue on puppetmaster21:12
zarofungi: ohh i see.  let me check review-dev to see if there's an ci results to compare21:12
mordredShrews: I will follow up with a patch that adds comprehensive tests for the thing that we missed21:13
mordredShrews: this, btw, is a great example of where the betamax tests will be super helpful21:14
mordredthe combinations of what you get back from clouds is ... vast21:14
*** sarob has joined #openstack-infra21:14
sdakethis commit seems to be in error 7742abdb6cfdd64e903b94a2cecad7396b86bb9821:14
sdakebut it says merge commit, I dont see the original commit to revert21:14
mordredsdake: you got change-id ?21:15
*** ldnunes has quit IRC21:15
sdakemordred the merge commit doesn't have a change id21:15
mordredoh. piddle21:15
sdakeand git log doesn't show the title name in anything except the merge commit which is extremely odd21:15
mordredsdake: you are looking in project-config?21:15
sdakeyes21:16
fungisdake: a merge commit will have two parent commits. the first one will be the former branch tip and the second one should be the change commit21:16
*** rfolco has joined #openstack-infra21:16
phschwartzmordred: Happy Bday man :)21:16
*** yamahata has joined #openstack-infra21:16
mordredphschwartz: thanks! apparently what I do on my birthday is break libraries :)21:16
sdakefungi I hear what your syaing- but I dont see it in the git log21:16
*** rfolco has quit IRC21:16
phschwartzmordred: It is a bday gift to everyone else21:16
mordredsdake: Merge: 61349ec 802366021:17
fungisdake: second like of git show <commit>21:17
fungis/like/line/21:17
sdakefungi thanks21:17
mordred8023660f28fcea1880b1ad015b422ae6dfa64cc6 is the one to revert21:17
sdakemordred thanks21:17
fungi8023660 is https://review.openstack.org/35208921:17
openstackgerritSteven Dake proposed openstack-infra/project-config: Revert "Enable ubuntu-xenial for Kolla"  https://review.openstack.org/35508721:19
fungisdake: sorry about that. someone probably got over-zealous with sed21:19
zarofungi: finally found change with ci result on review-dev and it looks like it's broken there as well.21:19
jeblairsdake: would it be possible to fix that rather than revert?21:19
sdakefungi it was actually Jeffrey4l  - one of our communit y membeers21:19
zarofungi: let me see take a look to see what might have caused the problem21:19
sdakeJeffrey4l ya i guess, but I'm out of steam for the day21:19
sdakejeblair ya I guess,s but I'm out of steam for the day21:20
sdakeI am goign to bed21:20
sdakeour osic cluster owrk happens from 2am-2pm pst21:20
sdake(I am pst)21:20
sdakejeblair and atm, kolla-kubernetes devs completely blocked21:20
sdakebut i've got nothing left for today - need a nap21:20
jeblairsdake: sleep well :)21:21
sdakejeblair thanks :)21:21
*** rbuzatu has quit IRC21:21
*** edmondsw has quit IRC21:22
mgagneso I found that latest grafana accepts config in a different format than prevision version. How should grafyaml handles version difference?21:23
*** sdake has quit IRC21:23
*** lascii is now known as alaski21:23
*** ddieterly has joined #openstack-infra21:24
mgagnelike v2.6 vs v3.021:24
*** esberglu has quit IRC21:24
*** matt-borland has quit IRC21:25
*** oomichi has quit IRC21:28
*** oomichi has joined #openstack-infra21:29
*** ddieterly has quit IRC21:29
*** dimtruck is now known as zz_dimtruck21:29
*** tonytan4ever has quit IRC21:30
*** ddieterly has joined #openstack-infra21:31
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: DO NOT MERGE - Periodic test  https://review.openstack.org/34694921:31
*** sarob has quit IRC21:33
fungii wanted to get a good updated server snapshot before the os upgrade on wiki.o.o, but i have one from a week or so ago which should be good enough to roll back if i copy the database dump i just made along with a tarball of /srv/mediawiki21:35
fungithis snapshot's been running for more than an hour at this point, no idea how long it should take to snapshot a 315G rootfs that's 55G used21:36
fungiwish i'd more closely timed the one i ran a couple weeks ago21:36
*** sarob has joined #openstack-infra21:38
fungianybody happen to know whether server snapshots in rackspace are pinned to the block device state at the point where you start the snapshot, or are working toward the state of the block device at the point where the snapshot finishes writing?21:38
*** esberglu has joined #openstack-infra21:38
zarofungi: only thing i see that may have cause that problem is https://review.openstack.org/#/c/345540/  am investigation to make sure21:38
*** spzala has joined #openstack-infra21:39
*** ihrachys has quit IRC21:39
* Zara has lurked around for gerrity things but is heading to bed nowish. hope the project.config stuff is easy to sort later :)21:39
Zara'night!21:39
Zarathanks again21:39
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: Deploy minimal services in multinode job  https://review.openstack.org/35509721:43
*** spzala has quit IRC21:44
*** sarob has quit IRC21:44
anteayaif sdake wants approval on changes to jobs on kolla in future, that is one way of fixing this21:45
anteayawonder how long it will take before he can figure out they are called jobs21:45
anteayaor at least assuring the same thing doesn't happen again in future21:45
*** valderrv has quit IRC21:45
*** toymachine has quit IRC21:46
*** jistr has quit IRC21:47
*** jistr has joined #openstack-infra21:51
*** xyang1 has quit IRC21:52
openstackgerritMerged openstack-infra/project-config: Add experimental grenade job with neutronclient from source  https://review.openstack.org/35466421:54
*** thorst_ has joined #openstack-infra21:54
*** shashank_hegde has quit IRC21:55
*** mriedem has quit IRC21:59
*** shashank_hegde has joined #openstack-infra22:00
*** pradk has quit IRC22:00
*** nwkarsten has quit IRC22:00
*** jistr has quit IRC22:00
*** nwkarsten has joined #openstack-infra22:01
*** shashank_hegde has quit IRC22:04
*** nwkarsten has quit IRC22:06
*** sdague has quit IRC22:07
*** tpsilva has quit IRC22:08
*** spzala has joined #openstack-infra22:08
*** jistr has joined #openstack-infra22:09
Shrewsmordred: just put it through. not really here atm22:09
*** mdrabe has quit IRC22:11
fungithat describes my state most of the time22:11
zarofungi: ok.  i've confirmed that the rel=nofollow (https://review.openstack.org/#/c/345540/) is the thing that is causing the table thing to not work.22:12
zarofungi: what would you like to do about it atm?  revert or fix?22:12
fungizaro: fix hideci.js22:12
fungiit's probably a simple patch22:12
*** beekneemech is now known as bnemec-pto22:12
fungii expect it's doing a pattern search on the hyperlink reference tag and with the added option that's no longer matching22:13
zarofungi: ok. i'll look into that but i need to stop in about 1 hr.  so might not be fixed until sometime this weekend or monday.22:13
fungizaro: i don't think it's too urgent, but also maybe someone with more javascript gumption will pick it up22:14
fungiyour diagnosis is a huge step toward fixing22:14
fungiso thanks!22:14
zaroohh, that would be swell :)22:14
*** spzala has quit IRC22:17
*** spzala has joined #openstack-infra22:17
*** jistr has quit IRC22:19
*** rbuzatu has joined #openstack-infra22:20
fungioh, geez. i shoulda touched /fastboot before rebooting wiki.o.o... "fsck:/:2%"22:21
*** nwkarsten has joined #openstack-infra22:22
*** spzala has quit IRC22:23
fungi"Press C to cancel all checks in progress" (unless you are using the rackspace virtual console apparently!)22:23
anteaya:(22:24
*** karthik__ has quit IRC22:24
*** karthik__ has joined #openstack-infra22:24
*** tonytan4ever has joined #openstack-infra22:25
fungiat least it's already up to 10% done22:25
*** jistr has joined #openstack-infra22:25
*** rbuzatu has quit IRC22:25
fungipretty fast for a 350gb rootfs that's also still trying to be snapshotted by the provider22:25
fungioh, actually the snapshot finished in the last few minutes22:25
mordredyay!22:26
*** nwkarsten has quit IRC22:27
fungithankfully this fsck has sped up a lot now. almost to 75%22:29
fungithough if i end up having to roll back this upgrade, i'll probably run over the two hours i budgeted at this point22:29
asselinI'm trying to digest http://docs.openstack.org/infra/zuul/zuul.html#jobs skip-if(optional).  all-files-match-any (optional)  " This parameter will be considered matching a change only if all files in a change match at least one of these expressions."22:30
asselinso if there's an extra file in the change that's not matching, then the job will still run?22:30
*** kzaitsev_mb has quit IRC22:31
asselinI would like to run a job only if it matches a file regex, regardless if there are other files in the change22:32
jeblairasselin: correct.  it's so that you can write "don't run devstack if someone submits a change that only touches docs"22:32
jeblairasselin: "files:" should do that22:32
*** kzaitsev_mb has joined #openstack-infra22:33
asselinjeblair, thanks, yes that's what I needed22:33
*** sarob has joined #openstack-infra22:39
*** kzaitsev_mb has quit IRC22:40
*** armax has quit IRC22:40
*** rbrndt has quit IRC22:41
*** devkulkarni2 has joined #openstack-infra22:41
devkulkarni2pabelanger: wondering if the image uploads that you had started have completed?22:43
*** karthik__ has quit IRC22:43
*** adrian_otto has quit IRC22:43
*** annegentle has quit IRC22:43
*** sarob has quit IRC22:43
fungiwiki upgrade complete, but getting a blank response on pages. troubleshooting now22:43
*** sarob has joined #openstack-infra22:45
fungiaha, there was a skin on the upgrade-test server outside the mw slot. copying over now22:45
openstackgerritMerged openstack-infra/tripleo-ci: Don't try to get stack details if the stack doesn't exist  https://review.openstack.org/31933722:48
*** spzala has joined #openstack-infra22:48
*** sarob has quit IRC22:49
fungiand an extension22:49
*** devkulkarni1 has joined #openstack-infra22:49
*** tkelsey has joined #openstack-infra22:50
*** ddieterly has quit IRC22:50
fungihey! wiki's up22:51
fungianybody with free cycles, please browse around https://wiki.openstack.org/ and make sure it's working and you can edit stuff22:51
fungii'll okay the status alert momentarily22:51
anteayaI can log in to the wiki22:52
*** ddieterly has joined #openstack-infra22:52
fungihttps://wiki.openstack.org/wiki/Special:Version says we're on 1.27.022:52
fungii've logged in too22:52
pleia2yep, I'm logged in as well22:53
*** markvoelker has quit IRC22:53
*** devkulkarni2 has quit IRC22:53
fungii can edit a page, though since i'm a wiki admin that doesn't trigger any captcha22:54
pleia2trying with my non-admin user now22:54
anteayawas able to edit https://wiki.openstack.org/wiki/Sprints#Newton_sprints22:54
*** tkelsey has quit IRC22:54
fungithough also the nocaptcha extension is _supposed_ to only fire up a captcha _if_ the javascript-based humanity tests are inconclusive22:54
anteayaI don't think I was admin when I edited22:54
*** apetrich has quit IRC22:54
fungiso you might not see one even then22:54
anteayaI did not see a captcha22:54
*** ddieterly has quit IRC22:55
*** adrian_otto has joined #openstack-infra22:55
pleia2I'm not a robot22:55
fungipleia2: looks like it autocreated your nonadmin account as expected https://wiki.openstack.org/w/index.php?title=Special%3AListUsers&username=&group=&creationSort=1&desc=1&limit=5022:56
pleia2the captcha worked for me22:56
* pleia2 thumbs up22:56
*** adrian_otto has quit IRC22:56
fungii'm calling the sniff test good and clearing our status alert with 3 minutes to spare22:56
*** sarob has joined #openstack-infra22:56
pleia2nice work, fungi22:56
anteayathanks fungi22:57
fungi#status alert ok https://wiki.openstack.org/ is now running Mediawiki 1.27.0; please let us know in #openstack-infra if anything seems wrong22:57
openstackstatusfungi: sending alert22:57
fungigrrr22:57
fungii'll wait for that to finish before it correct it22:57
anteayawhat is the mistake?22:57
fungialert ok should have just been ok22:58
*** adrian_otto has joined #openstack-infra22:58
anteayaah22:58
fungiand interrupting a status alert has brought shame upon my household in the past22:58
anteayaindeed22:58
anteayadon't shame the household22:58
* anteaya goes to find a temple22:58
anteayaI loved in Japan that I could literally find a temple every 200 paces22:58
anteayaI had barely left the temple before there was another one22:59
fungipleia2: most of the credit goes to jpmaxman for getting this working on a snapshot. i just mostly filled in the blanks for his instructions and copied some files around22:59
anteayathank you jpmaxman22:59
fungialso our wikidb is now in trove22:59
-openstackstatus- NOTICE: ok https://wiki.openstack.org/ is now running Mediawiki 1.27.0; please let us know in #openstack-infra if anything seems wrong22:59
*** ChanServ changes topic to "ok https://wiki.openstack.org/ is now running Mediawiki 1.27.0; please let us know in #openstack-infra if anything seems wrong"22:59
fungiover the weekend or early next week i'll move /srv/mediawiki onto a cinder volume too23:00
fungiand then we can start trying out Krenair's awesome puppetry updates23:00
fungion a virgin ubuntu trusty vm23:00
*** adrian_otto has quit IRC23:00
pleia2\o/23:01
* fungi is not comfortable running on the server in this state any longer than absolutely necessary23:01
pleia2agreed23:01
*** xarses has quit IRC23:01
fungibut this also gets us less spam in the interim hopefully, or at least the ability to iterate further on newer spam mitigation extensions in addition to the confirmedit nocaptcha support23:01
fungiin parallel with the puppeting23:02
openstackstatusfungi: finished sending alert23:03
fungi#status ok https://wiki.openstack.org/ is now running Mediawiki 1.27.0; please let us know in #openstack-infra if anything seems wrong23:03
openstackstatusfungi: sending ok23:03
fungithat's what i _meant_ to do23:03
anteayayou household cheers you23:03
anteayayour23:03
jpmaxmanCool. Yeah fungi thank you for all your help and support. Now the real test happens :)23:03
fungidisappearing to help make dinner for inlaws, but will try to keep one eye on irc/mailing lists23:03
anteayafungi: happy inlaws23:04
*** armax has joined #openstack-infra23:04
*** ams__ has quit IRC23:04
fungijpmaxman: per the upgrade plan i e-mailed you, the only things i missed were that you had symlinked to new copies of skins/extensions in /srv/mediawiki outside /srv/mediawiki/slot423:04
fungiwhich were easily divined from the apache error.log23:04
*** ChanServ changes topic to "Discussion of OpenStack Developer and Community Infrastructure | docs http://docs.openstack.org/infra/ | bugs https://storyboard.openstack.org/ | source https://git.openstack.org/cgit/openstack-infra/ | channel logs http://eavesdrop.openstack.org/irclogs/%23openstack-infra/"23:05
-openstackstatus- NOTICE: https://wiki.openstack.org/ is now running Mediawiki 1.27.0; please let us know in #openstack-infra if anything seems wrong23:05
*** esberglu has quit IRC23:07
anteayayour divination tool of choice I see23:07
jpmaxmanAhh cool. Whoops. :)23:08
fungijpmaxman: no worries. the process was pretty straightforward23:08
*** devkulkarni1 has quit IRC23:09
openstackstatusfungi: finished sending ok23:09
fungiand with that, i disappear in a puff of magic smoke23:09
*** sarob has quit IRC23:10
*** sputnik13 has quit IRC23:10
*** amitgandhinz has joined #openstack-infra23:11
*** bin_ has quit IRC23:14
*** amitgandhinz has quit IRC23:16
Krenairfungi, so the current state is... an unpuppetised machine?23:16
*** kzaitsev_mb has joined #openstack-infra23:16
anteayaKrenair: yes23:16
anteayawhich is less than our hope23:17
*** thorst_ has quit IRC23:17
*** adrian_otto has joined #openstack-infra23:18
*** hongbin has quit IRC23:19
*** adrian_otto has quit IRC23:21
*** chem|off has quit IRC23:22
*** kzaitsev_mb has quit IRC23:22
*** sarob has joined #openstack-infra23:22
*** aviau has quit IRC23:23
*** aviau has joined #openstack-infra23:23
*** thorst_ has joined #openstack-infra23:26
*** Goneri has quit IRC23:26
*** xarses has joined #openstack-infra23:26
*** chem|off has joined #openstack-infra23:27
openstackgerritJames E. Blair proposed openstack-infra/nodepool: Shut down gearman client in tests  https://review.openstack.org/35510923:27
*** sarob has quit IRC23:27
*** adrian_otto has joined #openstack-infra23:28
*** rlandy has quit IRC23:29
*** packet_sh_ has quit IRC23:29
*** thorst_ has quit IRC23:30
*** raunak has quit IRC23:32
*** jerryz has quit IRC23:33
*** sdake has joined #openstack-infra23:33
*** hockeynut has quit IRC23:35
pabelangermgagne: Oh, interesting. We still need to upgrade to 3.0 here.  How bad is the config changes?23:36
pabelangerfungi: I too am able to use wiki.o.o23:37
*** roxanaghe has quit IRC23:46
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: DO NOT MERGE - Periodic test  https://review.openstack.org/34694923:51
*** markvoelker has joined #openstack-infra23:54
*** Swami has quit IRC23:55
*** roxanaghe has joined #openstack-infra23:55
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: Deploy minimal services in multinode job  https://review.openstack.org/35509723:55
*** baoli has quit IRC23:56
*** hparekh has quit IRC23:58
*** markvoelker has quit IRC23:59
*** gouthamr has joined #openstack-infra23:59
*** yamahata has quit IRC23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!