Thursday, 2016-08-04

*** rossella_s has quit IRC00:00
*** lezbar has quit IRC00:00
*** lezbar has joined #openstack-infra00:00
*** rossella_s has joined #openstack-infra00:00
*** karthik__ has quit IRC00:00
*** jamielennox|away is now known as jamielennox00:01
*** amitgandhinz has quit IRC00:02
*** EricGonczer_ has joined #openstack-infra00:02
*** mriedem has joined #openstack-infra00:06
*** zul has quit IRC00:07
*** zul has joined #openstack-infra00:09
*** EricGonczer_ has quit IRC00:10
*** r-mibu has quit IRC00:10
*** sputnik13 has quit IRC00:10
*** sdake has quit IRC00:10
*** sputnik13 has joined #openstack-infra00:12
*** baoli has joined #openstack-infra00:13
*** mtanino has quit IRC00:14
*** rbuzatu has joined #openstack-infra00:14
*** rwsu has quit IRC00:14
JayFpabelanger: getting another set of new failures now too
*** sputnik13 has quit IRC00:16
JayFpabelanger: that appears to be well before it runs anything specific to my patch, and I have multiple failures of that nature :x00:17
*** baoli has quit IRC00:18
*** rbuzatu has quit IRC00:18
* JayF leaving for the evening00:19
openstackgerritK Jonathan Harker proposed openstack-infra/puppet-openstack_infra_spec_helper: Pin json_pure gem for ruby1.9 support
*** shashank_hegde has quit IRC00:30
*** tonytan4ever has joined #openstack-infra00:33
*** r-mibu has joined #openstack-infra00:35
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Add support for Growl plugin
*** tonytan4ever has quit IRC00:38
*** Na3iL has joined #openstack-infra00:39
*** xarses has quit IRC00:39
*** thorst has joined #openstack-infra00:45
jlkmordred: so your cluster I believe is back in action, and ready for infra to take it on again.00:47
jlk(I forgot to say something last week)00:47
*** pvaneck has quit IRC00:50
*** gildub has joined #openstack-infra00:50
*** amitgandhinz has joined #openstack-infra00:58
*** xarses has joined #openstack-infra00:59
*** amitgandhinz has quit IRC01:02
*** thorst has quit IRC01:04
*** thorst has joined #openstack-infra01:05
*** zhurong has joined #openstack-infra01:06
*** elo has quit IRC01:07
*** Sukhdev has quit IRC01:12
*** thorst has quit IRC01:13
*** yuanying has quit IRC01:14
*** yuanying has joined #openstack-infra01:14
*** yuanying has quit IRC01:15
*** mmandel has joined #openstack-infra01:15
*** Apoorva_ has joined #openstack-infra01:16
*** zhurong has quit IRC01:16
*** zhurong has joined #openstack-infra01:17
*** yuanying has joined #openstack-infra01:18
*** Apoorva has quit IRC01:18
*** gouthamr has joined #openstack-infra01:19
*** tphummel has joined #openstack-infra01:19
*** mmandel has quit IRC01:19
*** Apoorva_ has quit IRC01:20
*** Na3iL has quit IRC01:20
*** asettle has joined #openstack-infra01:24
*** asettle has quit IRC01:28
*** EricGonczer_ has joined #openstack-infra01:34
*** jamielennox is now known as jamielennox|away01:34
*** weshay has quit IRC01:40
*** gouthamr has quit IRC01:41
*** tqtran_ has quit IRC01:42
*** gouthamr has joined #openstack-infra01:42
mordredjlk: woot!01:43
mordredjlk: tomorrow I will make it do things01:43
mordredjlk: tomorrow I'm also going ot land the change that updates shade to use neutron directly for fips01:43
openstackgerritEmilien Macchi proposed openstack-infra/project-config: puppet: workaround for puppet-ceph
jlkoh shoot, I was supposed to test that out for you01:44
jlkcrap on a stick01:44
mordredjlk: it's ok ... I'll double check it on the bb cloud tomorrow before I release01:44
mordredjlk: there are a few things still in the queue that need to land01:45
jlkI'm going to test it quickly now.01:45
jlkis it on shade master?01:45
jlkokay, I'll clone/install01:45
mordredjlk: create_server auto_ip should work in the way you expect01:45
mordredif it doesn't, something is sad01:45
*** thorst has joined #openstack-infra01:46
*** thorst has quit IRC01:47
*** thorst has joined #openstack-infra01:47
openstackgerritJoshua Harlow proposed openstack-dev/pbr: Expose deb version to match exposing rpm version
jlkmy quick test will just be using os_floating_ip01:48
jlkoh haha, gotta install this on the controller side. haderp.01:48
openstackgerritYAMAMOTO Takashi proposed openstack-infra/project-config: networking-midonet: Update grafana dashboard for stable periodic jobs
*** shashank_hegde has joined #openstack-infra01:55
*** dimtruck is now known as zz_dimtruck01:55
jlkmordred: well the quick test seems to have been okay01:55
*** thorst has quit IRC01:56
*** baoli has joined #openstack-infra01:56
*** dmellado has quit IRC01:57
*** migi has quit IRC01:57
*** ajo has quit IRC01:57
*** migi_ has joined #openstack-infra01:57
*** amitgandhinz has joined #openstack-infra01:59
*** dmellado has joined #openstack-infra02:02
*** ajo has joined #openstack-infra02:03
*** amitgandhinz has quit IRC02:03
*** amotoki has joined #openstack-infra02:10
*** thorst has joined #openstack-infra02:18
*** mriedem has quit IRC02:20
*** sarob has joined #openstack-infra02:23
*** zz_dimtruck is now known as dimtruck02:24
*** markvoelker has joined #openstack-infra02:26
*** baoli has quit IRC02:26
*** sarob has quit IRC02:27
*** thorst has quit IRC02:27
*** thorst has joined #openstack-infra02:28
*** furlongm_ has joined #openstack-infra02:28
*** furlongm has quit IRC02:29
*** dkehn_ has quit IRC02:31
*** dkehn has quit IRC02:32
*** ddieterly has joined #openstack-infra02:33
*** ddieterly has quit IRC02:36
*** zhurong has quit IRC02:36
*** jamielennox|away is now known as jamielennox02:36
*** thorst has quit IRC02:37
*** zhurong has joined #openstack-infra02:38
*** dkehn has joined #openstack-infra02:39
*** dkehn_ has joined #openstack-infra02:43
*** liusheng has quit IRC02:49
*** yuanying has quit IRC02:51
*** sarob has joined #openstack-infra02:51
*** gouthamr has quit IRC02:51
*** yuanying has joined #openstack-infra02:54
*** sarob has quit IRC02:55
*** hichihara has joined #openstack-infra02:56
*** yuanying has quit IRC02:56
*** amitgandhinz has joined #openstack-infra03:00
*** shashank_hegde has quit IRC03:02
*** yamahata has quit IRC03:03
*** sdake has joined #openstack-infra03:04
*** amitgandhinz has quit IRC03:04
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Add support for Cloud Foundry plugin
*** binbincong has joined #openstack-infra03:08
*** yanyanhu has joined #openstack-infra03:11
*** binbincong has quit IRC03:13
*** mrmartin has joined #openstack-infra03:22
*** coolsvap has joined #openstack-infra03:22
*** sdake has quit IRC03:23
*** afred312_ has quit IRC03:29
*** persia has quit IRC03:30
*** persia has joined #openstack-infra03:33
*** shashank_hegde has joined #openstack-infra03:34
*** thorst has joined #openstack-infra03:35
*** thorst has quit IRC03:42
*** markvoelker has quit IRC03:46
*** gongysh has joined #openstack-infra03:47
*** sdake has joined #openstack-infra03:50
*** yuanying has joined #openstack-infra03:51
*** yuanying has quit IRC03:51
*** yuanying has joined #openstack-infra03:52
*** amotoki has quit IRC03:54
*** amotoki has joined #openstack-infra03:55
*** EricGonczer_ has quit IRC03:56
*** rossella_s has quit IRC04:00
*** amitgandhinz has joined #openstack-infra04:00
*** asselin_ has joined #openstack-infra04:00
*** rossella_s has joined #openstack-infra04:00
*** sarob has joined #openstack-infra04:01
*** asselin has quit IRC04:02
*** Sukhdev has joined #openstack-infra04:03
*** gongysh has quit IRC04:04
*** Sukhdev has quit IRC04:04
*** Sukhdev has joined #openstack-infra04:04
*** gongysh has joined #openstack-infra04:05
*** amitgandhinz has quit IRC04:06
*** sarob has quit IRC04:06
*** ayoung has quit IRC04:08
openstackgerritMatthew Treinish proposed openstack-infra/gerritbot: Add gerrit stream over MQTT support to gerritbot
*** julim has quit IRC04:16
*** roxanaghe has joined #openstack-infra04:18
*** shashank_hegde has quit IRC04:21
*** markvoelker has joined #openstack-infra04:22
*** markvoelker has quit IRC04:26
*** links has joined #openstack-infra04:28
*** asselin has joined #openstack-infra04:29
*** kien-ha has quit IRC04:31
*** asselin_ has quit IRC04:31
odyssey4mewe're getting intermittent POST_FAILURES which appear to be due to an auth issue when uploading logs:
odyssey4mejhesketh ^04:33
*** sdake has quit IRC04:37
*** Qiming has quit IRC04:39
*** thorst has joined #openstack-infra04:41
*** tqtran has joined #openstack-infra04:42
*** Qiming has joined #openstack-infra04:44
jheskethodyssey4me: hmm, is your job touching any file perms on the logs per chance?04:45
*** tqtran has quit IRC04:46
odyssey4mejhesketh hmm, lemme check - we had an issue with that some time ago and I thought we had addressed it04:46
*** shashank_hegde has joined #openstack-infra04:46
jheskethodyssey4me: if I were to guess either that log is copied or symlinked into that position and the jenkins user doesn't have the right permission to read it04:47
*** thorst has quit IRC04:47
*** psachin has joined #openstack-infra04:48
odyssey4mehmm, our job runs with sudo - but the directory containing all the data is traversable and all files world readable04:48
odyssey4mejhesketh ok, I see it's the swift logs specifically04:50
odyssey4meheh, ok - those are not world readable04:50
jheskethright, maybe they have different permissions for some reason?04:50
jheskethah there we go04:50
odyssey4methanks - I'll look into what's happening there04:50
jheskethso doesn't need to be world readable, just readable by the user 'jenkins'04:50
jhesketh(although easier to be world readable, particularly if we rename the user to zuul one day - but that'll be a v3 thing which has other changes004:51
*** gildub has quit IRC04:51
odyssey4meyeah, all our other logs are world readable so I think we'll likely just go with that04:51
*** gongysh has quit IRC04:53
*** icey has quit IRC04:57
*** javeriak has joined #openstack-infra04:59
*** afred312 has joined #openstack-infra04:59
*** rbuzatu has joined #openstack-infra05:00
*** amitgandhinz has joined #openstack-infra05:01
*** afred312 has quit IRC05:04
*** javeriak has quit IRC05:04
*** jraju has joined #openstack-infra05:04
*** icey has joined #openstack-infra05:04
*** amitgandhinz has quit IRC05:06
*** links has quit IRC05:06
*** narayrak has joined #openstack-infra05:06
*** shashank_hegde has quit IRC05:07
*** tphummel has quit IRC05:10
*** rcernin has joined #openstack-infra05:16
*** markvoelker has joined #openstack-infra05:16
*** aeng has joined #openstack-infra05:19
openstackgerritMerged openstack/diskimage-builder: Optionally remove portage files
*** markvoelker has quit IRC05:22
*** shashank_hegde has joined #openstack-infra05:22
*** tphummel has joined #openstack-infra05:25
*** roxanaghe has quit IRC05:26
*** aeng has quit IRC05:32
*** Qiming has quit IRC05:32
*** Qiming has joined #openstack-infra05:33
openstackgerritAlexey Weyl proposed openstack-infra/project-config: Vitrage tempests
*** rbuzatu has quit IRC05:35
*** thorst has joined #openstack-infra05:44
*** roxanaghe has joined #openstack-infra05:45
*** roxanaghe has quit IRC05:50
openstackgerritJuan Antonio Osorio Robles proposed openstack/diskimage-builder: Nothing to see here
*** thorst has quit IRC05:52
*** apetrich has quit IRC05:55
openstackgerritBen Kero proposed openstack-infra/project-config: Add channel #diskimage-builder
bkerogreghaynes: ^05:59
*** oanson has joined #openstack-infra06:00
*** afred312 has joined #openstack-infra06:00
*** apetrich has joined #openstack-infra06:00
openstackgerritIan Wienand proposed openstack-infra/project-config: Fix dib PROJECTS path
*** javeriak has joined #openstack-infra06:02
*** amitgandhinz has joined #openstack-infra06:02
*** sarob has joined #openstack-infra06:02
openstackgerritBen Kero proposed openstack-infra/project-config: Add channel #diskimage-builder
ianwpabelanger / jeblair : ^^ see my comment in above (350998), this failure should have bailed out setup_workspace but it didn't?  maybe ansible ignoring it?  need to dig further06:03
*** Srinu has joined #openstack-infra06:04
*** afred312 has quit IRC06:06
*** pt_15 has quit IRC06:06
*** amitgandhinz has quit IRC06:06
*** sarob has quit IRC06:07
*** camunoz has quit IRC06:09
*** gongysh has joined #openstack-infra06:09
*** asettle has joined #openstack-infra06:09
*** jtomasek has quit IRC06:09
*** pcaruana has joined #openstack-infra06:10
*** itisha has quit IRC06:10
*** Ravikiran_K has joined #openstack-infra06:12
*** gongysh has quit IRC06:13
*** rbuzatu has joined #openstack-infra06:13
*** asettle has quit IRC06:14
*** bhavik1 has joined #openstack-infra06:17
*** jaosorior has joined #openstack-infra06:20
openstackgerritBen Kero proposed openstack-infra/project-config: Add channel #diskimage-builder
bkerowomp womp, gerritbot is at its channel limit. It says to ask in here about setting up another gerritbot.06:28
openstackgerritYAMAMOTO Takashi proposed openstack-infra/project-config: grafana: Update midonet unit test jobs after xenial renames
bkeroAnybody know what I'd need to do, or what the status of that is?06:28
bkeroAJaeger: ^06:29
*** gildub has joined #openstack-infra06:30
openstackgerritYAMAMOTO Takashi proposed openstack-infra/project-config: grafana: Update ovn unit test jobs after xenial renames
*** pgadiya has joined #openstack-infra06:32
*** gongysh has joined #openstack-infra06:35
*** Sukhdev has quit IRC06:38
*** karthikp_ has joined #openstack-infra06:38
*** javeriak has quit IRC06:39
*** yamamoto_ has quit IRC06:41
*** tqtran has joined #openstack-infra06:42
*** karthikp_ has quit IRC06:43
*** jlanoux has joined #openstack-infra06:45
*** roxanaghe has joined #openstack-infra06:46
yolandagood morning06:46
*** pahuang has quit IRC06:47
*** tqtran has quit IRC06:47
*** jed56 has joined #openstack-infra06:47
AJaegerbkero: we need to run a second gerritbot. Not sure what needs done exactly. Basically running a second one, figure out how to distribute load over those two so that each stays under the limit of jobs... I cannot give details on that.06:47
bkeroAJaeger: It would be very simple to just have a second yaml file inside gerritbot/06:48
AJaegermorning, yolanda !06:48
bkeroalthough maybe someone could come up with something cleaner06:48
AJaegerbkero: Yeah, that would work...06:48
* bkero not a rooter, so likely can't set up the box, but I can do the puppet work necessary for it06:48
bkeroif we agree that is the correct approach.06:48
AJaegerbkero: Best talk with fungi later today - this would be appreciated!06:49
bkeroSure :)06:49
bkeroI'll talk with him during normal US hours.06:49
*** roxanaghe has quit IRC06:50
*** armax has quit IRC06:51
*** thorst has joined #openstack-infra06:51
AJaegerthanks, bkero !06:53
*** tesseract- has joined #openstack-infra06:53
*** permalac has quit IRC06:54
openstackgerritMerged openstack-infra/project-config: Implement tripleo-ci-centos-7-nonha-multinode-upgrades job
openstackgerritMerged openstack-infra/project-config: Add searchlight api-ref jobs
*** thorst has quit IRC06:57
openstackgerritAlexey Weyl proposed openstack-infra/project-config: Vitrage tempests
*** rbuzatu_ has joined #openstack-infra06:57
*** rbuzatu has quit IRC07:01
*** ihrachys has joined #openstack-infra07:02
*** afred312 has joined #openstack-infra07:02
*** amitgandhinz has joined #openstack-infra07:03
openstackgerritMerged openstack-infra/project-config: Fix dib PROJECTS path
*** javeriak has joined #openstack-infra07:05
*** amitgandhinz has quit IRC07:07
*** afred312 has quit IRC07:08
openstackgerritMerged openstack-infra/project-config: puppet: workaround for puppet-ceph
openstackgerritMerged openstack-infra/project-config: networking-midonet: Update grafana dashboard for stable periodic jobs
*** mikelk has joined #openstack-infra07:10
*** florianf has joined #openstack-infra07:11
*** tphummel has quit IRC07:11
openstackgerritMerged openstack-infra/system-config: Fix emaint sync call
openstackgerritMerged openstack-infra/project-config: Forbid ironic grenade job from building ramdisk
*** mikelk has quit IRC07:14
openstackgerritMerged openstack-infra/project-config: Add OpenStack Infra internal repositories to APT source lists
*** ccamacho has quit IRC07:16
*** gildub has quit IRC07:16
*** yamahata has joined #openstack-infra07:18
*** ccamacho has joined #openstack-infra07:18
*** nmagnezi has joined #openstack-infra07:19
*** witek has left #openstack-infra07:22
openstackgerritMerged openstack-infra/project-config: Move {name}-docs to ubuntu-xenial
openstackgerritMerged openstack-infra/project-config: Make bindep Python 3.5 unit test jobs vote
*** HeOS has joined #openstack-infra07:26
openstackgerritMerged openstack-infra/project-config: Disable services for tempest gate
openstackgerritMerged openstack-infra/project-config: Add common testing repo to OpenStack-Ansible
*** gongysh has quit IRC07:28
*** berendt has quit IRC07:30
*** dimtruck is now known as zz_dimtruck07:32
*** gongysh has joined #openstack-infra07:33
*** openstackgerrit has quit IRC07:33
*** ggnel_t has joined #openstack-infra07:33
*** openstackgerrit has joined #openstack-infra07:33
*** javeriak has quit IRC07:38
*** ihrachys has quit IRC07:40
*** acoles_ is now known as acoles07:45
*** vincentll has joined #openstack-infra07:47
*** shardy has joined #openstack-infra07:48
*** ihrachys has joined #openstack-infra07:48
*** thorst has joined #openstack-infra07:54
*** furlongm_ has quit IRC07:54
*** e0ne has joined #openstack-infra07:55
*** e0ne has quit IRC07:55
*** e0ne has joined #openstack-infra07:56
*** ihrachys has quit IRC07:57
*** markvoelker has joined #openstack-infra07:58
*** ihrachys has joined #openstack-infra07:59
*** gildub has joined #openstack-infra07:59
*** zzzeek has quit IRC08:00
*** rossella_s has quit IRC08:00
*** sflanigan has quit IRC08:00
*** jpich has joined #openstack-infra08:00
*** rossella_s has joined #openstack-infra08:00
openstackgerritOpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml
*** zzzeek has joined #openstack-infra08:01
*** thorst has quit IRC08:02
*** afred312 has joined #openstack-infra08:03
*** markvoelker has quit IRC08:03
*** amitgandhinz has joined #openstack-infra08:03
*** ihrachys has quit IRC08:06
*** amitgandhinz has quit IRC08:08
*** afred312 has quit IRC08:08
*** ihrachys has joined #openstack-infra08:10
openstackgerritMerged openstack-infra/puppet-openstackci: Pinning JJB and Zuul for 3rd party CI
*** jaosorior has quit IRC08:13
*** jaosorior has joined #openstack-infra08:13
*** hichihara has quit IRC08:14
*** e0ne has quit IRC08:14
*** e0ne has joined #openstack-infra08:15
*** Goneri has joined #openstack-infra08:15
*** rbuzatu_ has quit IRC08:16
*** dtantsur|afk is now known as dtantsur08:16
*** shashank_hegde has quit IRC08:17
*** shashank_hegde has joined #openstack-infra08:18
*** shashank_hegde has quit IRC08:19
*** rbuzatu has joined #openstack-infra08:20
*** yamahata has quit IRC08:22
*** dtantsur is now known as dtantsur|bbl08:23
*** lucas-dinner is now known as lucasagomes08:24
*** yaume has joined #openstack-infra08:25
*** yaume has quit IRC08:26
openstackgerritMerged openstack-infra/project-config: Normalize projects.yaml
*** javeriak has joined #openstack-infra08:30
*** roxanaghe has joined #openstack-infra08:34
*** chem has joined #openstack-infra08:35
*** yaume has joined #openstack-infra08:36
*** roxanaghe has quit IRC08:38
*** asettle has joined #openstack-infra08:40
*** javeriak has quit IRC08:41
*** tqtran has joined #openstack-infra08:44
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: Don't run undercloud idempotent on tempest jobs
*** aviau has quit IRC08:47
*** aviau has joined #openstack-infra08:47
*** tqtran has quit IRC08:49
*** apetrich has quit IRC08:51
*** zz_dimtruck is now known as dimtruck08:52
*** Ahharu has joined #openstack-infra08:52
*** mhickey has joined #openstack-infra08:53
*** markvoelker has joined #openstack-infra08:53
Ahharustrating today zuul cloner is not getting the right parameters on our module08:53
Ahharuit just gets this08:53
Ahharu2016-08-04 08:48:58.090984 | + /usr/zuul-env/bin/zuul-cloner -m /tmp/tmp.ZnCQR3pEad08:53
*** jtomasek has joined #openstack-infra08:53
Ahharueven though we have it setup on the spec helper acceptance08:53
*** dingyichen has quit IRC08:54
*** Na3iL has joined #openstack-infra08:55
*** markvoelker has quit IRC08:57
*** yamamoto_ has joined #openstack-infra08:59
*** thorst has joined #openstack-infra09:00
*** dimtruck is now known as zz_dimtruck09:01
*** yamamoto_ has quit IRC09:02
*** e0ne has quit IRC09:03
*** yamamoto has joined #openstack-infra09:03
*** e0ne has joined #openstack-infra09:04
*** afred312 has joined #openstack-infra09:04
*** amitgandhinz has joined #openstack-infra09:04
*** thorst has quit IRC09:06
*** yamamoto has quit IRC09:08
*** afred312 has quit IRC09:08
*** javeriak has joined #openstack-infra09:09
*** amitgandhinz has quit IRC09:09
*** gildub has quit IRC09:12
*** berendt has joined #openstack-infra09:17
*** _degorenko|afk is now known as degorenko09:22
*** dulek has quit IRC09:25
*** dulek has joined #openstack-infra09:27
*** skipp has quit IRC09:27
*** stewie925 has quit IRC09:28
*** eeiden has quit IRC09:28
*** eeiden has joined #openstack-infra09:31
*** markmcd has joined #openstack-infra09:34
*** electrofelix has joined #openstack-infra09:35
AJaegerAhharu: can you point me to job and logfile?09:37
*** apetrich has joined #openstack-infra09:39
Ahharuand puppet swift for example >
AJaegerAhharu: I see the problem, working on a fix...09:41
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Add missing slash to puppet-module-jobs
AJaegerAhharu: ^09:42
AJaegeryolanda, jhesketh, ianw : If you're around, could you review this, please?09:43
Ahharugood !09:43
AJaegerAhharu: Sorry, I missed that in reviewing the original change.09:43
*** gongysh has quit IRC09:44
AJaegerAhharu: with current load of gate: might take two hours to get this merged and then max 1h to be deployed.09:46
*** stewie925 has joined #openstack-infra09:46
*** markvoelker has joined #openstack-infra09:47
Ahharuwell .. i will do a workaround by copying my module with the changes for testing until then09:48
*** permalac has joined #openstack-infra09:49
*** markvoelker has quit IRC09:52
*** skipp has joined #openstack-infra09:53
*** yamamoto has joined #openstack-infra09:53
*** sambetts|afk is now known as sambetts09:55
*** nadya has joined #openstack-infra09:55
*** nadya is now known as _nadya_09:56
openstackgerritYAMAMOTO Takashi proposed openstack-infra/project-config: grafana: Update ovn unit test jobs after xenial renames
*** ams__ has joined #openstack-infra10:02
* AJaeger pushes 351086 through to unbreak puppet10:02
*** sarob has joined #openstack-infra10:03
*** zhurong has quit IRC10:04
*** afred312 has joined #openstack-infra10:04
*** amitgandhinz has joined #openstack-infra10:05
vponomaryovwho knows what have changed to trusty images?10:07
*** sarob has quit IRC10:08
vponomaryovapt-get upgrade falls into endless loop on update of grub that started making dialog10:08
vponomaryovlike this -
*** afred312 has quit IRC10:09
*** amitgandhinz has quit IRC10:10
*** ihrachys has quit IRC10:11
*** furlongm_ has joined #openstack-infra10:12
*** dtantsur|bbl is now known as dtantsur10:12
*** yaume has quit IRC10:13
*** kzaitsev_mb has joined #openstack-infra10:13
openstackgerritMerged openstack-infra/project-config: Add missing slash to puppet-module-jobs
*** roxanaghe has joined #openstack-infra10:22
*** roxanaghe has quit IRC10:26
*** chlong has quit IRC10:28
*** yamamoto has quit IRC10:29
*** yamamoto has joined #openstack-infra10:30
*** jimbaker has quit IRC10:30
*** thorst has joined #openstack-infra10:31
*** amotoki has quit IRC10:31
*** tosky has joined #openstack-infra10:31
*** yamamoto has quit IRC10:32
*** ihrachys has joined #openstack-infra10:33
*** yanyanhu has quit IRC10:33
*** jimbaker has joined #openstack-infra10:34
*** jimbaker has quit IRC10:34
*** jimbaker has joined #openstack-infra10:34
*** thorst has quit IRC10:36
AJaegervponomaryov: was there an upstream update of grub-common ? I suggest to check the history of that package...10:36
AJaegerOr of grub-pc...10:36
*** kzaitsev_mb has quit IRC10:37
*** markvoelker has joined #openstack-infra10:41
vponomaryovAJaeger: tried just to avoid its update like here -
vponomaryovAJaeger: will see whether it helps or not10:41
*** chlong has joined #openstack-infra10:41
*** csomerville has joined #openstack-infra10:43
*** cody-somerville has quit IRC10:43
AJaegerI suggest to inform upstream about this if that fixes the issue.10:44
*** apetrich has quit IRC10:45
*** markvoelker has quit IRC10:46
vponomaryovAJaeger: upstream of what?10:46
vponomaryovAJaeger: grub project upstream?10:46
*** gildub has joined #openstack-infra10:47
openstackgerritSagi Shnaidman proposed openstack-infra/project-config: Return tripleo tempest job to experimental pipe
AJaegervponomaryov: That's an Ubuntu package - os upstream Ubuntu10:48
*** gongysh has joined #openstack-infra10:48
AJaegeryeah, released 4 hours ago ;(10:49
*** Ahharu has quit IRC10:49
*** kaisers has quit IRC10:50
*** kaisers has joined #openstack-infra10:51
*** kaisers has quit IRC10:51
*** kzaitsev_mb has joined #openstack-infra10:53
*** kaisers has joined #openstack-infra10:57
*** ihrachys has quit IRC11:02
sshnaidmjeblair, hi, we talked recently about new gear release, just curious if everybody agreed to this and we can proceed? thanks!11:03
*** thorst has joined #openstack-infra11:03
*** Ravikiran_K has quit IRC11:04
*** afred312 has joined #openstack-infra11:05
*** amitgandhinz has joined #openstack-infra11:06
*** rbuzatu has quit IRC11:06
*** ihrachys has joined #openstack-infra11:06
*** rbuzatu has joined #openstack-infra11:09
*** jlanoux has quit IRC11:09
*** afred312 has quit IRC11:10
*** bhavik1 has quit IRC11:10
*** amitgandhinz has quit IRC11:10
AJaegerEmilienM: your change is still broken, see
*** mat128|afk is now known as mat12811:13
*** Mmike has quit IRC11:16
*** Mmike has joined #openstack-infra11:17
*** Mmike has joined #openstack-infra11:17
*** amotoki has joined #openstack-infra11:17
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Fix  puppet-openstack-integration-checkout
AJaegeryolanda: we overlooked another issue in EmilienM's change ;( WE should have reverted instead - please check that my fix is really good, otherwise I'm for reverting.11:19
yolandaah, i misunderstood the logic there, but yes, that fix makes sense11:21
*** sdague has joined #openstack-infra11:21
*** lucasagomes is now known as lucas-hungry11:21
AJaegeryolanda: so did I ;(11:22
*** yamamoto has joined #openstack-infra11:23
yolandai gave +2, the shame is that it will take long to merge, right?11:23
AJaegeryolanda: last one took less than an hour to merge. Let me self-approve again...11:25
*** chlong has quit IRC11:27
*** chlong has joined #openstack-infra11:28
*** sshnaidm is now known as sshnaidm|lnch11:31
*** ldnunes has joined #openstack-infra11:34
*** kzaitsev_mb has quit IRC11:34
*** ericksonsantos has quit IRC11:34
*** rhallisey has joined #openstack-infra11:34
*** raildo has quit IRC11:34
*** markvoelker has joined #openstack-infra11:35
*** jlanoux has joined #openstack-infra11:35
*** gordc has joined #openstack-infra11:39
*** markvoelker has quit IRC11:39
*** jkilpatr has joined #openstack-infra11:40
*** dulek has quit IRC11:43
*** dulek has joined #openstack-infra11:45
*** acoles is now known as acoles_11:49
*** amotoki has quit IRC11:50
openstackgerritTang Chen proposed openstack-infra/project-config: Export openstack/ironic/ to $PROJECT
EmilienMAJaeger: good morning11:53
EmilienMAJaeger: and thank you :)11:53
AJaegermorning, EmilienM. We were a bit eager merging one of your changes and overlooked a few things, hope that 351127  fixes everything.11:53
*** psilvad has joined #openstack-infra11:54
EmilienMAJaeger: ok I'll watch is closely this morning11:54
AJaegerplease slowly recheck once that is in - we have a long queue and don't want endless rechecks (so, recheck one change and wait for results, please)11:54
*** weshay has joined #openstack-infra11:56
EmilienMAJaeger: sure thing11:56
*** asettle has quit IRC11:57
*** hieulq_ has joined #openstack-infra11:57
*** asettle has joined #openstack-infra11:57
*** lucas-hungry is now known as lucasagomes11:58
*** gouthamr has joined #openstack-infra12:01
*** asettle has quit IRC12:02
*** eharney has quit IRC12:02
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: WIP - Implement undercloud upgrade job - Mitaka -> Newton
*** markvoelker has joined #openstack-infra12:05
*** afred312 has joined #openstack-infra12:06
*** amitgandhinz has joined #openstack-infra12:06
*** zul has quit IRC12:07
*** sshnaidm|lnch is now known as sshnaidm12:08
*** hieulq_ has quit IRC12:09
*** roxanaghe has joined #openstack-infra12:10
*** afred312 has quit IRC12:11
*** amitgandhinz has quit IRC12:11
*** zhurong has joined #openstack-infra12:12
*** apetrich has joined #openstack-infra12:13
*** tpsilva has joined #openstack-infra12:14
*** roxanaghe has quit IRC12:15
*** Goneri has quit IRC12:17
*** baoli has joined #openstack-infra12:20
*** zul has joined #openstack-infra12:20
*** dkranz has joined #openstack-infra12:22
*** eharney has joined #openstack-infra12:25
*** Goneri has joined #openstack-infra12:25
openstackgerritMerged openstack-infra/project-config: Fix  puppet-openstack-integration-checkout
*** rfolco has joined #openstack-infra12:25
*** Srinu has quit IRC12:25
*** kzaitsev_mb has joined #openstack-infra12:25
*** baoli_ has joined #openstack-infra12:25
*** asettle has joined #openstack-infra12:26
*** xyang1 has joined #openstack-infra12:27
*** dprince has joined #openstack-infra12:28
*** baoli has quit IRC12:29
pabelangerAJaeger: Do we want to remove gate-{name}-tox-{envlist} and replace it with gate-{name}-tox-{envlist}-{node}?12:30
AJaegerpabelanger: that'S my understanding.12:31
pabelangerAJaeger: okay, let me do that this morning12:31
AJaegerpabelanger: There have been so far only some steps done in this direction but not the complete move...12:31
*** ramishra has quit IRC12:31
AJaegerpabelanger: thanks!12:31
*** ramishra has joined #openstack-infra12:33
openstackgerritMerged openstack-infra/system-config: Switch the floating ip source to neutron for rh1
*** pradk_ has quit IRC12:35
*** kgiusti has joined #openstack-infra12:41
*** amotoki has joined #openstack-infra12:43
*** tqtran has joined #openstack-infra12:45
*** burgerk has joined #openstack-infra12:48
*** raildo has joined #openstack-infra12:48
EmilienMAJaeger: it seems to work, I continue to check12:48
AJaegerEmilienM: so, first success?12:49
*** psilvad has quit IRC12:50
EmilienMAJaeger: yes :)12:50
*** tqtran has quit IRC12:50
*** dkranz has quit IRC12:50
AJaegeryeah, looks like we're fine now...12:51
*** zz_dimtruck is now known as dimtruck12:53
Zarasomeone posted a story, 'zuul consumes too much memory' and has asked if there are any updates. I'm not sure if people are subscribed to it so I'm acting as storyboardbot just in case.!/story/200064012:53
*** wanghua has joined #openstack-infra12:53
*** javeriak has quit IRC12:54
*** dtantsur is now known as dtantsur|mtg12:54
wanghuaHi all, as I know, zuul merger prepares a tmp repo for zuul cloner, why zuul cloner still need git_base_url? I think zuul cloner can clone the repo from the zuul merger directly. Can anyone help me?12:54
*** psilvad has joined #openstack-infra12:55
*** rlandy has joined #openstack-infra12:56
EmilienMAJaeger: I still have my problem with puppet-ceph though (not critical at this time) but I'm working on it12:56
*** jsavak has joined #openstack-infra12:56
*** gongysh has quit IRC12:56
*** javeriak has joined #openstack-infra12:57
openstackgerritMerged openstack-infra/ciwatch: Remove discover from test-requirements
*** psilvad has quit IRC13:00
*** jpich has quit IRC13:03
*** dimtruck is now known as zz_dimtruck13:03
*** pradk has quit IRC13:04
*** psilvad has joined #openstack-infra13:04
*** afred312 has joined #openstack-infra13:07
*** amitgandhinz has joined #openstack-infra13:07
*** zul has quit IRC13:07
*** mdrabe has joined #openstack-infra13:09
*** sdake has joined #openstack-infra13:11
*** afred312 has quit IRC13:11
*** amitgandhinz has quit IRC13:12
openstackgerritIvan Kolodyazhny proposed openstack-infra/project-config: Disable test_volume_from_snapshot tests for Cinder BDD job
*** markvoelker has quit IRC13:13
*** matt-borland has joined #openstack-infra13:15
*** gildub has quit IRC13:16
*** Liuqing has joined #openstack-infra13:17
*** amitgandhinz has joined #openstack-infra13:17
*** ihrachys has quit IRC13:18
*** zul has joined #openstack-infra13:22
*** acoles_ is now known as acoles13:22
*** timl0101 has quit IRC13:22
*** sdake has quit IRC13:24
*** thiagop has joined #openstack-infra13:25
*** amitgandhinz has quit IRC13:27
*** amitgandhinz has joined #openstack-infra13:28
openstackgerritMonty Taylor proposed openstack-infra/shade: Add ability to upload duplicate images
*** matrohon has joined #openstack-infra13:30
*** julim has joined #openstack-infra13:31
asselinpabelanger, I'm reviewing Is this expected output?
*** _ari_ has joined #openstack-infra13:32
EmilienMAJaeger: ok so I confirm that the problem is still here for puppet-ceph only though13:33
pabelangerasselin: Yup, means you don't have puppet installed locally13:33
EmilienMI'm trying to undercloud why zuul-cloner doesn't checkout the repo at the commit in Depends-On13:33
EmilienMthe patch is here:
*** Jeffrey4l_ has quit IRC13:34
*** esberglu has joined #openstack-infra13:35
*** hieulq_ has joined #openstack-infra13:35
anteayaEmilienM: the job prepared the parent of the ceph backport patch, which is the depends on13:37
*** timl0101 has joined #openstack-infra13:38
EmilienManteaya: c13a4187aa2725d593f6e3f1c752fd1550665465 is the latest patch in stable/mitaka, no the commit id in dependson13:38
EmilienMso the depends-on is not adressed13:38
anteayaEmilienM: ah sorry13:39
EmilienManteaya: no prob13:39
EmilienMthx for looking13:39
*** mriedem has joined #openstack-infra13:39
anteayawelcome, still looking13:40
*** hieulq_ has quit IRC13:41
*** burgerk has quit IRC13:41
asselinpabelanger, what exactly is it looking for? I have puppet installed:
*** acoles is now known as acoles_13:44
*** jpich has joined #openstack-infra13:45
*** EricGonczer_ has joined #openstack-infra13:45
*** aeng has joined #openstack-infra13:46
*** jraju has quit IRC13:46
*** yamamoto has quit IRC13:46
anteayaEmilienM: looks to me like zuul cloner is looking for a stable/hammer branch in puppet-openstack-integration and not finding such a branch:
EmilienManteaya: in fact, we force it to stable/mitaka
*** matrohon has quit IRC13:48
*** yamamoto has joined #openstack-infra13:49
Zarahm, question-- I want to use review-dev so I can see how gerrit-storyboard integration is doing and test things (I think we need to set an additional parameter in some config first, but I'm not sure, and I'd rather be able to check before I ask about things). I think the `git review` command will default to talking to review.o.o. so how would I go about talking to review-dev?13:49
*** javeriak has quit IRC13:49
*** ddieterly has joined #openstack-infra13:50
*** pgadiya has quit IRC13:51
*** markvoelker has joined #openstack-infra13:51
*** javeriak has joined #openstack-infra13:52
mordredZara: for the second thing, the .gitreview file in the git repo points to which will set up the 'gerrit' remote in git13:53
*** mmandel has joined #openstack-infra13:53
*** sdake has joined #openstack-infra13:54
Zaramordred: ahhhh, right, I see13:54
openstackgerritJeremy Stanley proposed openstack-infra/infra-manual: Clarify post job log URLs for merge commits
mordredZara: there are a couple of projects that only exist in review-dev that have their .gitreview set up ... but if you want to use a project that is in both, you can edit the remote entry in .git/config13:55
Zaraokay, thanks. I may just be lazy and clone again so I don't forget where I'm sending things. :)13:56
*** yamamoto has quit IRC13:56
*** rbuzatu_ has joined #openstack-infra13:57
anteayaEmilienM: I'm questioning whether or not it is accepting the force13:57
*** PollyZ has joined #openstack-infra13:58
*** xarses has quit IRC13:58
*** hieulq_ has joined #openstack-infra13:58
*** baoli_ has quit IRC13:59
*** shamail has joined #openstack-infra14:00
*** rbuzatu has quit IRC14:00
*** hieulq_ has quit IRC14:00
*** ddieterly has quit IRC14:01
*** ddieterly has joined #openstack-infra14:01
*** vincentll has quit IRC14:01
*** vincentll has joined #openstack-infra14:01
*** PollyZ has quit IRC14:02
*** edmondsw has quit IRC14:02
*** ZZelle has quit IRC14:03
*** PollyZ has joined #openstack-infra14:03
*** rbuzatu_ has quit IRC14:04
*** rbuzatu has joined #openstack-infra14:05
*** matrohon has joined #openstack-infra14:05
*** matrohon has quit IRC14:05
openstackgerritBrad P. Crochet proposed openstack-infra/tripleo-ci: Use tripleo-build-images for CI
*** pradk has joined #openstack-infra14:06
*** zul has quit IRC14:07
*** zul has joined #openstack-infra14:07
*** rbrndt has joined #openstack-infra14:07
*** afred312 has joined #openstack-infra14:07
*** coolsvap has quit IRC14:11
AJaegerfungi, do you have time today to look at constraints and release/post queue, please? is the review14:11
*** afred312 has quit IRC14:12
*** pcrews has quit IRC14:12
*** apetrich has quit IRC14:12
*** rockyg has joined #openstack-infra14:12
*** pcrews has joined #openstack-infra14:14
EmilienMAJaeger: do you have an idea why zuul-cloner checkout the last commit in the branch and not the commit in depends-on in ?14:14
*** jsavak has quit IRC14:14
*** dtantsur|mtg is now known as dtantsur14:14
AJaegerEmilienM: not right now, that needs some more digging than I can do right now.14:15
AJaegerEmilienM: the --branch parameter might be the problem...14:16
EmilienMAJaeger: ok14:16
*** aeng has quit IRC14:16
fungiEmilienM: the branch names don't seem to match?14:17
fungiyou have a "stable/hammer" change depending on a "stable/mitaka" change14:17
fungizuul-cloner looks for matching branch names14:17
fungii think that will be very hard to work around14:17
*** jaosorior has quit IRC14:17
*** javeriak has quit IRC14:18
EmilienMfungi: yes, our puppet-ceph module has a stable/hammer branch that require to work with stable/mitaka14:18
fungii don't think we have any good solutions to let you integration test that without writing separate integration test tooling14:19
*** apetrich has joined #openstack-infra14:20
fungiusing a stable/mitaka branch on puppet-ceph too would make the current model feasible, as would just using the master branch of puppet-ceph, but i'm guessing those options are incompatible with your release model for that repo14:21
*** aeng has joined #openstack-infra14:21
*** matrohon has joined #openstack-infra14:22
*** PollyZ has quit IRC14:22
*** amotoki has quit IRC14:22
*** PollyZ has joined #openstack-infra14:23
*** signed8bit has joined #openstack-infra14:24
greghaynescinerama: ianw You all ok with ?14:24
*** afred312 has joined #openstack-infra14:24
*** aeng has quit IRC14:25
*** zul has quit IRC14:25
*** zul has joined #openstack-infra14:25
*** gongysh has joined #openstack-infra14:27
Zarahm, actually, are there projects that are in both review-dev and review.o.o? I looked for a dummy project to send commits to to start with, and couldn't find overlap. looked in and couldn't see anything that matched up with a project in review-dev; tried cloning from review-dev, but I got 'server certificate verification failed'. I'm not sure how to add a project to it, if I need to14:27
Zara do that.14:27
*** burgerk has joined #openstack-infra14:28
*** mtanino has joined #openstack-infra14:31
*** zz_dimtruck is now known as dimtruck14:31
*** vincentll has quit IRC14:32
*** gongysh has quit IRC14:32
*** matrohon has quit IRC14:32
anteayaZara: I think you need to be a gerrit admin to be able to add a project to our review-dev14:32
pabelangerasselin: it looks at the local package manager IIRC, how did you install puppet?14:33
anteayacan you get what you need with just being able to add a patch to a current project?14:33
anteayaor do you need to add a project?14:33
fungi(or add a corresponding project to storyboard-dev?)14:33
asselinpabelanger, don't remember...but most likely with install_puppet.sh14:33
*** mdrabe has quit IRC14:34
EmilienMfungi: right, it make things weird, as Ceph & OpenStack don't have the same release cycles14:34
*** javeriak has joined #openstack-infra14:34
Zarajust adding a patch should be enough to begin with, I just couldn't find a way to clone a project to send patches for it. Ideally I'd like a project in there with a corresponding storyboard-dev project.14:34
*** Goneri has quit IRC14:35
Zara(though I think the config to say 'update this storyboard project' is a matter for a future patch, but yeah, that's something I want to find out by having a wander around)14:35
openstackgerritPaul Belanger proposed openstack-infra/project-config: Migate tox to node
pabelangerasselin: dpkg -l | grep puppet14:36
pabelangerasselin: anything listed?14:36
pabelangerasselin: ya, you don't have puppet install14:38
pabelangerthe puppet package14:38
*** asettle has quit IRC14:38
pabelangeryou mind have the binary14:38
pabelangerwhat version of ubuntu are you using?14:38
*** asettle has joined #openstack-infra14:39
*** xarses has joined #openstack-infra14:39
asselinpabelanger, ok thanks for clarifying. I see it here in a test-vm, so my error.
*** mdrabe has joined #openstack-infra14:39
pabelangerasselin: looks like we should be using puppet-common for ubuntu14:39
asselini'm on trusty14:39
pabelangerbut puppet package has a dependency on puppet-common14:40
pabelangerasselin: ya, I would expect it to work in your test-vm14:40
pabelangersince puppet package is listed14:40
asselinok, so just an issue w/ my workstation14:40
asselinso fails correctly14:41
anteayaZara: try some version of git clone ssh:// && cd Project-Foo && git fetch origin refs/meta/config && git checkout FETCH_HEAD14:42
*** jcoufal has joined #openstack-infra14:45
*** Jeffrey4l has joined #openstack-infra14:45
*** mhayden has joined #openstack-infra14:46
mhaydenfungi: are you the right person to ask about bindep releases?14:47
*** Liuqing has quit IRC14:47
fungimhayden: yeah, did i break something with 2.0.1 yesterday?14:47
*** Liuqing has joined #openstack-infra14:48
mhaydenah, i must have missed the release :)14:48
*** matrohon has joined #openstack-infra14:49
fungiit was a minor bugfix release14:49
fungia handful of cosmetic patches and one change allowing _ in package names (which seems to be an rpmism)14:49
mhaydenah, the package name one the patch i was concerned about :)14:49
mhaydenwow that was terrible grammar14:50
mhaydenperhaps i need more coffee14:50
fungiif our image uploads succeed today, we should have 2.0.1 in the bindep virtualenv on them14:50
*** berendt has quit IRC14:50
fungiand then any jobs trying to introduce package names with an underscore should pass14:51
fungier, any changes14:51
mhaydenthat's good news for me -- thanks, fungi14:51
*** Teresita-Warrior has joined #openstack-infra14:51
jeblairEmilienM: something like this approach may work:  (cc: fungi)14:53
*** javeriak has quit IRC14:53
EmilienMthat's what I did :)14:54
*** edmondsw has joined #openstack-infra14:54
fungijeblair: EmilienM: ahh, yep that's similar i think to what gnocchi does for their nonstandard stable branches too14:54
*** vincentll has joined #openstack-infra14:54
fungiit requires maintaining a branch mapping, so not entirely automagic, but better than nothing14:55
*** karthikp_ has joined #openstack-infra14:55
*** annegentle has joined #openstack-infra14:55
fungialso tricky if you're wanting to mix and match branch names in one zuul-cloner call14:55
jeblairEmilienM: cool.  so if you stuck the --project-branch argument on the first zuul-cloner command, you can drop the second one, and it should make Ivan happy too :)14:55
*** vvalderrv has joined #openstack-infra14:55
fungisince ZUUL_BRANCH can only ever have one value per zuul-cloner call14:56
jeblairfungi: right, but --project-branch can have any number14:56
jeblair(you can add multiple --project-branch arguments)14:56
EmilienMoh wouah14:56
EmilienMI can do that?14:56
EmilienMand it will try all of them in a specific order?14:57
fungitrue. you just need to pick one to be your $ZUUL_BRANCH default14:57
jeblairEmilienM: look closely at the last line of the paste
fungiso preferably whichever branch is used by the greatest number of changes14:57
*** yamamoto has joined #openstack-infra14:57
EmilienMjeblair: ok, I didn't know this feature14:58
jeblair(i commented on 350845)14:58
EmilienMjeblair: have you seen this workaround ?
EmilienMjeblair: we also need that to checkout puppet-openstack-integration repo to the right branch but it doesn't checkout the commit in DependsOn14:59
*** annegentle has quit IRC15:00
jeblairyeah, i saw that in the log; i think that's probably inevitable15:00
EmilienMso when we'll patch puppet-ceph stable/hammer, we won't be able to have a Depends-On in puppet-openstack-integration, that's all. Right?15:01
* EmilienM making sure to understand :-)15:01
jeblairi think if you make the mapping bi-directional, and have it in both places, it should work15:02
Zaraanteaya: added ssh keys to review-dev (same ones as on review.o.o, so they should work) and tried that approach for the clone. still getting 'permission denied (publickey)'. I noticed the 'download' link had the username set to 'null', so I tried that, too, but it didn't work.15:02
EmilienMjeblair: I tested it and it didn't work15:02
pabelangerEmilienM: I would think it works, if you updated your zuul-cloner logic within puppet-openstack-integration15:02
anteayaZara: :(15:02
anteayaZara: is your username the same on review and review-dev?15:03
pabelangerEmilienM: what commit are you expecting?15:03
EmilienMpabelanger: I think the problem is not in p-o-i now, see the log15:03
jeblairEmilienM: to clarify:  have the mapping support both 'stable/mitaka' and 'stable/hammer' as input, and put it both in the puppet-openstack-integration-checkout and the main checkout... and do the zuul-cloner stuff like in  i think it should work15:03
EmilienMso there is "Depends-On: I004ee95e6bd97992aeebd08447396e3b8a29e09f" in
openstackgerritMatthew Treinish proposed openstack-infra/system-config: Add docs for firehose.o.o
EmilienMthe parent is c13a4187aa2725d593f6e3f1c752fd1550665465 (and that's what is checked out) but the actual commit I wanted is a83add1a2d401c24ec8f8a419b26d0618620d55015:03
*** acoles_ is now known as acoles15:04
*** rockyg has quit IRC15:04
Zarayes, but hm, for some reason it doesn't associate the ICLA15:04
Zaranot sure if that might cause it or not.15:04
anteayahmmm, review-dev shouldn't block you on the icla15:04
pabelangerEmilienM: right, I think what me and jeblair are saying. in the line you lined, you are not cloning puppet-ceph, you are only cloning puppet-opentack-integration. I believe you have logic in puppet-openstack-integraton to clone your other puppet modules15:04
anteayait might, but if it does it shouldn't15:04
*** Liuqing has quit IRC15:05
EmilienMpabelanger: ok15:05
pabelangerEmilienM: isn't the commit you are looking for?15:05
*** rockyg has joined #openstack-infra15:05
*** Liuqing has joined #openstack-infra15:05
*** yamamoto has quit IRC15:05
fungiZara: anteaya: it depends on what the project is configured for. some of them were set for testing cla integration (and we deploy a dummy foundation api server on review-dev in the form of a cgi that always returns the desired result)15:05
EmilienMpabelanger: it is but not
EmilienMpabelanger: ok I'll give a try, maybe I'll use some help :)15:06
Zaraoh, actually, hang on, it's not my username, I thought that was full name but apparently it's not the same and it wasn't set15:06
fungiZara: anteaya: should just be able to click through the cla agreement and add dummy contact info if you're using one that has cla enforcement set15:06
*** moravec has quit IRC15:07
*** oanson has quit IRC15:07
EmilienMjeblair: iiuc, I need logic on both sides15:07
*** vhosakot has joined #openstack-infra15:07
*** moravec has joined #openstack-infra15:07
Zaraugh, there we go, sorry for noise. thanks.15:07
anteayaZara: fungi works for me15:07
*** akshai has joined #openstack-infra15:07
anteayaZara: ah good, glad you are on your way15:07
*** _nadya_ has quit IRC15:08
anteayafungi: thank you, I didn't know that15:08
*** EricGonczer_ has quit IRC15:08
Zara(yeah, it needed me to manually set the username, but it was displaying 'Zara' in the top-right and as my full name so I thought it was in there.)15:08
*** apetrich has quit IRC15:09
*** rcernin has quit IRC15:09
*** EricGonczer_ has joined #openstack-infra15:09
openstackgerritPaul Belanger proposed openstack-infra/project-config: Replace gate-{name}-tox-{envlist} with gate-{name}-tox-{envlist}-{node}
*** akshai has quit IRC15:12
pabelangerAJaeger: ^ I think got everything in that15:12
pabelangerAJaeger: too longer then expected15:12
*** vincentll has quit IRC15:13
*** psachin has quit IRC15:13
*** mhickey has quit IRC15:13
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Setup constraints for release
AJaegerpabelanger: Thanks! Will review later today15:13
AJaegerfungi, corvus: I've updated 349627 addressing fungi's suggestions. Could you review again, please?15:14
AJaegerpabelanger: only 10 conflicts for your change - that's good ;)15:15
*** e0ne has quit IRC15:15
pabelangerAJaeger: I see some issues, pushing up a new patch15:15
openstackgerritPaul Belanger proposed openstack-infra/project-config: Replace gate-{name}-tox-{envlist} with gate-{name}-tox-{envlist}-{node}
*** ayoung has joined #openstack-infra15:17
AJaegerpabelanger: I just commented on zuul/layout - there're a few places that have branch conditions to run only on master or future stable branches - for those we could use only xenial and no trusty.15:18
EmilienMjeblair: is it --project-branch puppet-ceph=$BRANCH or --project-branch openstack/puppet-ceph=$BRANCH ?15:18
*** yaume has joined #openstack-infra15:18
EmilienMjeblair: also, in <big list of projects>: should I remove puppet-ceph from it? or it's fine to keep it?15:18
AJaegerpabelanger: So, If you want to remove some of these, have a look - note that I wouldn't do it when it means splitting up templates!15:19
AJaegerpabelanger: Will review later again, now cycling home... The general approach is fine!15:19
*** baoli has joined #openstack-infra15:19
fungiAJaeger: thanks, having another look15:19
jeblairEmilienM: it's  --project-branch openstack/puppet-ceph=$BRANCH and keep it in the big list15:19
EmilienMjeblair: ack, thanks a lot for your help15:20
openstackgerritPaul Belanger proposed openstack-infra/project-config: Replace gate-{name}-tox-{envlist} with gate-{name}-tox-{envlist}-{node}
*** baoli_ has joined #openstack-infra15:23
*** baoli has quit IRC15:23
*** matrohon has quit IRC15:24
*** armax has joined #openstack-infra15:24
Zarain the name of testing interesting gerrit things, would anyone like to merge: ? I don't have the privileges.15:25
Zara(I'm double-checking task-statuses change as expected)15:26
*** rwsu has joined #openstack-infra15:26
EmilienMjeblair: I followed your suggestion,
*** karthikp_ has quit IRC15:27
* AJaeger has to get reacquainted with jeblair again and his switching personality ;)15:27
AJaegerjeblair: could you review 349627  as well, please?15:28
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add tripleo-test-cloud grafana dashboard
*** karthikp_ has joined #openstack-infra15:28
*** armax has quit IRC15:29
*** baoli_ has quit IRC15:29
*** timl0101 has quit IRC15:29
*** armax has joined #openstack-infra15:29
*** baoli has joined #openstack-infra15:29
*** timl0101 has joined #openstack-infra15:29
openstackgerritPaul Belanger proposed openstack-infra/project-config: Replace gate-{name}-tox-{envlist} with gate-{name}-tox-{envlist}-{node}
*** ggnel_t has quit IRC15:30
*** zhurong has quit IRC15:30
anteayaZara: you may need to beckon to an infra-root or zaro for that request15:31
*** baoli_ has joined #openstack-infra15:31
fungii thought we had those set to allow anyone to do the submit button15:31
pabelangerjlk: mordred: let me know if I can help bring up bluebox again in nodepool15:31
pabelangerjlk: mordred: excited to have it back!15:32
fungianteaya: Zara: but i can check the acl for that project on review-dev once my current meeting ends15:32
Zarafungi: thank you. :) it's not urgent and I'm happy to try with another project if that works better.15:34
*** baoli has quit IRC15:34
anteayathank you15:34
*** amotoki has joined #openstack-infra15:35
mordredpabelanger: I've updated the admin password information in the passwords file15:35
*** EricGonczer_ has quit IRC15:35
mordredpabelanger: i'm currently working on making a patch/playbook to create and setup all of the things we want in it15:35
EmilienMjeblair: I wonder what we are missing in though15:35
*** EricGonczer_ has joined #openstack-infra15:35
sdaguemirror break - ?15:36
*** ddieterly is now known as ddieterly[away]15:37
*** ddieterly[away] is now known as ddieterly15:37
jeblairAJaeger: lgtm15:37
*** ddieterly is now known as ddieterly[away]15:37
*** EricGonczer_ has quit IRC15:37
jeblairEmilienM: i think that may have everything it needs15:37
EmilienMjeblair: but like I said before, it doesn't work right. I tested it and it checkout the latest commit in stable/mitaka and not the patch in Depends-On15:39
jpmaxmanfungi:  So it seems this nocaptcha thing might be a bug in the current version of mediawiki15:39
dhellmanndoes anyone know if the civs election system has moved URLs? is reporting a 40415:39
jpmaxmanI talked to some people on the mediawiki IRC and filed a bug15:39
EmilienMjeblair: I will try again15:39
jeblairsdague: using wget on that url, i get the correct md5, so probably an error in transport that pip masked15:39
jpmaxmanI'm going to try the latest 1.26 version to see if it is still there15:39
dhellmannnevermind, anteaya found it15:39
jpmaxmancan you drop a fresh dump of the database15:39
jpmaxmanon the wiki-upgrade-test.openstack.org15:40
anteaya is the new url15:40
jpmaxmanserver ?15:40
sdaguejeblair: ok, it's happing with some regularity -
anteayaI'll update the wiki in a few minutes with the new url15:40
*** ddieterly[away] is now known as ddieterly15:40
mordredanteaya: ++15:40
*** matrohon has joined #openstack-infra15:40
anteayamordred: dhellmann wondered if they might be trying to hide from you15:40
jpmaxmanok fungi - might not be around will check in later :D15:40
anteayayou breaker of polls you15:41
sdaguealways in rax-ord it seems15:41
mordredanteaya: it would be a  good strategy on their part I think15:41
*** priteau has joined #openstack-infra15:41
anteayahe he he15:42
*** narayrak has quit IRC15:49
*** shashank_hegde has joined #openstack-infra15:50
fungijpmaxman: i'll look in a bit, in the middle of a meeting right now, sorry15:50
*** yamahata has joined #openstack-infra15:50
mtreinishsdague: was that the region that's had the recent network issues?15:51
sdaguemtreinish: I feel like it always has network issues15:52
jlkwelcome to RAX....15:53
*** karthikp_ has quit IRC15:55
openstackgerritAdam Coldrick proposed openstack-infra/storyboard: Position archived items at the bottom of worklists
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: Complex priorities UI in stories
*** rossella_s has quit IRC16:00
*** matrohon has quit IRC16:00
*** rossella_s has joined #openstack-infra16:00
*** shashank_hegde has quit IRC16:01
*** karthikp_ has joined #openstack-infra16:02
Zaragerrit things work nicely for changing task-status to/from todo and review (can't check 'merged') yet. first reactions suggest people find the comment links a bit noisy; we might want to disable them if people are already getting email notifications from gerrit itself.16:02
Zarahuh, early closing bracket is early16:02
Zarabut \o/, review-dev.o.o is talking nicely to storyboard-dev.o.o16:03
*** Liuqing has quit IRC16:04
Zara(see events timeline changes from today for an example:!/story/6 )16:05
*** yottatsa has joined #openstack-infra16:05
*** Hal has quit IRC16:06
*** osnexus has quit IRC16:06
*** apetrich has joined #openstack-infra16:06
*** fernnest has joined #openstack-infra16:06
*** jlanoux has quit IRC16:07
zaroZara: cross comment will not be enabled on review.o.o16:08
*** baoli_ has quit IRC16:08
*** ddieterly is now known as ddieterly[away]16:09
zaroit's only enabled on review-dev.o.o for testing.  i'm thinking maybe changing setting on review-dev.o.o to same as review.o.o (turn auto comments off)16:10
*** raildo has quit IRC16:10
Zaraohhhh, I missed that it was only enabled there. yeah, that's probably a good idea, so people can get an accurate idea of what it will look like16:11
*** raildo has joined #openstack-infra16:12
openstackgerritMatt Riedemann proposed openstack-infra/system-config: Start indexing ansible_log.txt in logstash
mriedemclarkb: mtreinish: sdague: ^ hope that's not terrible16:12
zarohmm, it doesn't look like puppet is keeping review-dev.o.o up to date.  was wondering if an infra-root can help?16:13
jeblairsshnaidm: i have tagged gear 0.7.0  (cc: fungi, mordred, notmorgan, SpamapS)16:13
*** eharney has quit IRC16:13
jeblair(should show up on pypi in a little while)16:13
sshnaidmjeblair, \o/ great news, thanks!16:13
zarothis change should have re-enabled puppet on review-dev
*** aorourke has joined #openstack-infra16:14
*** baoli has joined #openstack-infra16:14
fungizaro: we also had it listed in the emergency disable file because of the bug we were tracking down that was causing ansible to ignore other groups16:14
*** raildo has quit IRC16:15
fungizaro: i've removed it just now16:15
*** ddieterly[away] is now known as ddieterly16:15
fungishould get hit with puppet on the next pulse16:15
*** mat128 is now known as mat128|lunch16:15
fungimaube in 20-30 minutes16:15
*** seldenr has joined #openstack-infra16:15
zarofungi: cool, thanks!16:15
*** dtantsur is now known as dtantsur|afk16:16
*** nmagnezi has quit IRC16:17
*** matrohon has joined #openstack-infra16:17
*** raildo has joined #openstack-infra16:18
jeblairShrews, mordred, notmorgan: this is an interesting ansible error:
mordredjeblair: I don't lke the word interesting in this context16:19
jeblairi think it may be a behavior change in -- at least, that code was touched recently16:19
mordredjeblair: I agree16:19
jeblairthe underlying issue is that it's two ansible processes racing to create the local tmp dir16:19
mordredjeblair: I think we should poke our friends in #ansible-devel16:20
jeblair(and apparently python's os.makedirs is suceptible to such races!)16:20
jeblairit only happened once in my stack of 5 changes16:20
*** _nadya_ has joined #openstack-infra16:20
*** weshay is now known as weshay_food16:21
mordredjeblair: which two ansible processes is it do you think?16:21
mordredlike, the async thing and then the next task in the list?16:21
jlkwe had to back away from :(16:21
jlkbut for unrelated reasons16:21
jeblairmordred: i think it's probably two parallel test runs16:21
jeblairmordred: but that's just a guess16:21
openstackgerritKhai Do proposed openstack-infra/system-config: Update its-storyboard actions on review-dev.o.o
zaroZara: ^16:22
mordredjeblair: oh - like, that's ~/.ansible/tmp on the launcher host?16:22
Zarazaro: thanks, taking a look now :)16:22
mordredjeblair: I thought we made a change to put that into the launcher temp dir we made already - I guess we did not do that?16:22
mordredjlk: yah - we have two fixes we made in upstream that landed in - so moving to it allowed us to not run patched locally16:23
jeblairmordred: hrm, let me look into some things16:23
*** e0ne has joined #openstack-infra16:23
mordredjlk: it'll be sad panda if we also have to back away16:23
mordredjeblair: (seems that if we didn't do that, doing that should potentially ameliorate this)16:23
jlksomething changed in the way variables are resolved that led to weird scenarios in our playbook tree16:23
SpamapSjeblair: sweet16:23
*** matrohon has quit IRC16:23
jlkfor us, we just had to back away while we do a deeper investigation.16:23
mordredjlk: oh - I think I saw you talking about that16:23
mordredand yah - you have some complex variable overlays going on16:24
Zarazaro: thank you, looks fine to me :)16:24
jeblairmordred: i believe we are running with the zuul-launcher change that sets remote_tmp to something inside of /tmp so that it will not collide what what a job is running.  so i think that we have removed the zuul v2.5 ansible launcher from the equation.16:25
jlk2.1.1.0 also got more strict about some syntax things that previously I think were ignored. So, yay for fixing bugs, and oops on us for having bugs16:26
jeblairmordred: so i suspect that this is just two parallel zuul testr runs trying to create the _local_ ansible tmp dir16:26
SpamapSjlk: hopefully only things that were warnings before?16:26
jlkSpamapS: I think it was silently ignored16:27
jlkit was clearly a syntax issue16:27
*** karthikp_ has quit IRC16:27
jlkwith_items: "{{ groups["controller"] }}"16:27
SpamapSjlk: that's kind of b.s. though16:27
jlkthat was the line that now is a immediate fail16:27
jlkdouble use of " is bad16:28
SpamapSI see why it would fail16:28
SpamapSbut I'm kind of bullish on always being backward compatible16:28
jlkyeah I'm not really upset about this one.16:28
jlkI think if this file were ever hit by previous versions it would have failed too, it just happens that this was in an unused code path16:29
SpamapShow did that ever parse?16:29
SpamapSthat's totally invalid yaml?16:29
*** Apoorva has joined #openstack-infra16:29
SpamapSjlk: you guys need our super awesome yaml file syntax checker. :)16:29
jeblairmordred: this is the apparently new code path:
jlkSpamapS: I think the code path was never parsed before16:30
jlkand now every code path is parsed?16:30
jeblairmordred: i think the actual solution to this is to forward-port the zuulv2.5 ansible tmp path stuff to zuulv316:31
mordredjeblair: ++16:31
*** _nadya_ has quit IRC16:31
mordredjeblair: forward-porting the zuulv2.5 ansible stuff was coming up soon on my TDL16:31
*** Apoorva has quit IRC16:31
jeblairmordred: cool; i can probably recheck through this for now16:31
*** Apoorva has joined #openstack-infra16:32
*** ddieterly is now known as ddieterly[away]16:32
*** kro_focused is now known as krotscheck16:32
jeblairmordred: *maybe* ansible could do: try: os.makedirs() except OSError e: if e.errno==17: pass16:32
jeblairmordred: that seems weird to me since os.makedirs is already supposed to not do that :)16:33
jeblairmordred: i think i'll mention this in -devel in case they want to run with it.  but i wouldn't blame them for not.16:33
*** sandanar has joined #openstack-infra16:33
*** tphummel has joined #openstack-infra16:34
*** shashank_hegde has joined #openstack-infra16:36
*** lucasagomes is now known as lucas|dinner16:39
*** SumitNaiksatam has joined #openstack-infra16:39
*** notmyname has joined #openstack-infra16:40
*** matrohon has joined #openstack-infra16:40
*** pt_15 has joined #openstack-infra16:42
*** gomarivera has joined #openstack-infra16:43
*** weshay_food is now known as weshay16:43
*** e0ne has quit IRC16:43
*** yaume has quit IRC16:44
*** florianf is now known as florianf|biab16:44
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add gate-python-tripleoclient-buildimage jobs
*** tqtran has joined #openstack-infra16:47
*** Swami has joined #openstack-infra16:48
*** Sukhdev has joined #openstack-infra16:50
*** itisha has joined #openstack-infra16:51
*** tqtran has quit IRC16:51
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add gate-python-tripleoclient-buildimage jobs
*** david-lyle has quit IRC16:53
*** david-lyle has joined #openstack-infra16:53
*** ddieterly[away] is now known as ddieterly16:54
*** tosky has quit IRC16:55
*** jtomasek has quit IRC16:55
*** berendt has joined #openstack-infra16:55
*** asettle has quit IRC16:55
*** jpich has quit IRC16:57
jeblairfungi: the release management team's mascot is an anteater (pangolin).  and infra are the ants?16:57
jeblairfungi: er, sorry, security team, not relmgt16:58
* anteaya thinks about jumping ship16:58
jeblair"What if one team's mascot's primary diet is that of a mascot of another team?" is not in the FAQ.16:58
anteayaor did fungi set it up that way?16:59
anteayaha ha ha16:59
anteayait should be16:59
jeblairanteaya: indeed -- how many people are on both teams?16:59
anteayaha ha ha16:59
anteayaa dozen to over 20016:59
* anteaya looks pensively into the distance16:59
jeblairoh right our team is big :)16:59
jlklots of ants in the hill17:00
anteayayeah, so many repos17:00
anteayajlk: you also thinking security is looking good?17:00
jlkoh hell no17:00
anteayaha ha ha17:00
jlktalk about tilting at windmills17:00
anteayaha ha ha17:00
anteayayou are maintaining clouds17:00
*** piet_ has joined #openstack-infra17:01
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add automatic backports
pabelangerzigo: ^ should pass now17:02
jeblairanteaya: what's worse is trying to build a platform on a cloud.  have you ever tried that?  it doesn't work well.17:02
*** sambetts is now known as sambetts|afk17:03
anteayaha ha ha17:03
jeblairjoists fall right through17:03
anteayathat was what I was thinking17:03
*** flaper87 has quit IRC17:03
anteayaso hard to get those nails in place in time17:03
openstackgerritMerged openstack-infra/project-config: grafana: Update ovn unit test jobs after xenial renames
jeblairanteaya, fungi: i have sent a query to heidi :)17:05
anteayajeblair: please share any reply17:08
jeblairwill do17:10
*** sarob has joined #openstack-infra17:11
*** matrohon has quit IRC17:11
*** gomarivera has quit IRC17:11
*** permalac has quit IRC17:11
*** sandanar has quit IRC17:11
*** eharney has joined #openstack-infra17:12
*** _sarob has joined #openstack-infra17:13
fungiyeah, i hadn't thought about that. might be a good excuse to go with the honeycomb which came in second17:14
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Update iLO IP for baremetal00
* rcarrillocruz dances ^17:15
mtreinishfungi: isn't that what refstack picked...17:15
*** sarob has quit IRC17:16
*** signed8bit is now known as signed8bit_Zzz17:16
*** SumitNaiksatam has quit IRC17:16
*** PollyZ has quit IRC17:18
mordredrcarrillocruz: ooh! that makes it look like baremetal00 might exist again!17:19
rcarrillocruzso now consolidation reboot 2.0 begins17:19
rcarrillocruzi need to reboot all servers17:19
rcarrillocruzenter on rescue mode17:19
rcarrillocruzfigure out what server is by comparing it's "old public ip" to our hiera17:20
rcarrillocruzso i can update the right entry on the hiera infracloud.yaml17:20
fungimtreinish: maybe? i have heidijoy a ranked list of 23 options, so she'll mark them off until she reaches one we're able to use17:20
rcarrillocruzit will take time, but hey, our servers are reachable on the iLOs at least17:20
*** shashank_hegde has quit IRC17:20
fungier, s/have/gave/17:20
*** karthikp_ has joined #openstack-infra17:21
*** rbuzatu has quit IRC17:22
*** mat128|lunch is now known as mat12817:23
*** karthikp_ has quit IRC17:24
*** jcoufal_ has joined #openstack-infra17:24
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Enable net-iso for ovb jobs
mordredfungi: I'd like to express my sadness that "pink fairy armadillo" wasn't the winner17:24
*** dprince has quit IRC17:24
*** vhosakot has quit IRC17:25
fungimordred: right after i complain that almost nobody wanted a tanuki17:25
*** vhosakot has joined #openstack-infra17:25
jeblairfungi: refstack got the honeycomb+bee17:25
anteayathat is two17:26
*** yottatsa_ has joined #openstack-infra17:26
*** jcoufal has quit IRC17:26
anteayait is tough enough to not conflict on one17:26
*** daemontool has joined #openstack-infra17:28
anteayamordred: well if someone else has beaver we might be down to the pink fairy armadillo17:28
anteayaI'm nost sure if others have selected the pink fairy armadillo17:28
pabelangerrcarrillocruz: nice17:29
*** yottatsa has quit IRC17:29
*** _nadya_ has joined #openstack-infra17:30
*** karthikp_ has joined #openstack-infra17:30
*** javeriak has joined #openstack-infra17:32
*** degorenko is now known as _degorenko|afk17:32
*** rbuzatu has joined #openstack-infra17:32
*** Na3iL has quit IRC17:36
*** amitgandhinz has quit IRC17:36
*** zul has quit IRC17:36
*** amitgandhinz has joined #openstack-infra17:36
fungijpmaxman: there's now a copy of this morning's production wiki db backup available for you at
AJaegerfungi, - the constraints change - has passed check jobs. Do you want to approve it? Then I'll give a heads up to the release team...17:38
jeblairanteaya: i think i would summarize heidi's reply as: good question, not one they had considered in that context, most projects chose mascots to reflect strengths.  iow, i don't think she's worried about the symbolism.  (i'm not either, fwiw)17:38
*** rbuzatu has quit IRC17:38
Zara(pink fairy armadillo for president)17:38
fungiAJaeger: i'll check with them real quick just to make sure we're not risking major disruption for them17:39
*** shardy has quit IRC17:39
anteayajeblair: thank you for sharing that17:39
AJaegerthanks, fungi17:39
*** rbuzatu has joined #openstack-infra17:39
jeblairbut i think it's good for us to be aware of it and not be surprised.  beyond that, i leave things in fungi's capable hands :)17:39
*** rockyg has quit IRC17:40
*** acoles is now known as acoles_17:41
*** rbuzatu has quit IRC17:41
*** rbuzatu has joined #openstack-infra17:41
*** dprince has joined #openstack-infra17:41
*** SumitNaiksatam has joined #openstack-infra17:42
*** vhosakot has quit IRC17:42
*** HeOS has quit IRC17:42
harlowjafungi u got some time for some historical questions, i'm sorta interested in why zuul got created (vs just jenkins) and the whole drop jenkins --> ansible and how this all happened17:43
harlowjamaybe i'll write about all this in a book someday, ha17:44
fungiharlowja: jeblair can do it more justice than i, though i'm happy to take a stab at it17:44
harlowjaah, either is fine17:44
* harlowja more of just curios17:44
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add gate-python-tripleoclient-buildimage jobs
*** Sukhdev has quit IRC17:45
harlowjaas i start trying to get some internal folks to get more involved with the openstack-jenkins-cloud plugin17:45
*** apetrich has quit IRC17:45
harlowjait seems to be a interesting question, of why doesn't the openstack infra folks also use that (or have created it or ...)17:45
fungiharlowja: it sort of grew out of the limited featureset in the gerrit-trigger plugin for jenkins and our desire to be able to test changes in a serialized context before allowing them to merge17:45
*** apetrich has joined #openstack-infra17:45
harlowjaand that gets into the whole jenkins, zuul, history, ansible. ...17:45
EmilienMjeblair: when you have time, can you +1 this puppet CI patch please?
EmilienMjeblair: just to make sure I'm doing the right way17:45
harlowjafungi do u know if that got better (in jenkins itself?)17:46
harlowja(or in the gerrit plugin in jenkins)17:46
jeblairharlowja: HP heavily sponsored the jclouds plugin on behalf of the openstack project17:46
fungiharlowja: jenkins has just in the last few months grew a concept of "pipelining" which i think might be closer to that17:46
harlowjajeblair right, its interesting to see which that other plugin dropped the jclouds one17:46
* harlowja has been talking (sorta, via issues, ha) with the openstack-cloud-plugin author17:47
jeblairfungi: i don't think jenkins pipeline is like zuul17:47
jeblairharlowja: jenkins and zuul do different things :)17:47
fungiahh, i probably completely misunderstood the point of jenkins pipelines17:47
fungilimitations in the jclouds plugin (which we tried to use for a while) eventually drove the invention of nodepool though17:48
jeblairfungi: it's more like chained jobs17:48
harlowjajeblair jenkins does tasks/jobs in certain orders, so does zuul?17:48
harlowja(really simplified view?) ha17:48
mordredharlowja: it's the combination of different proposed repository states that zuul does that jenkins has no concept of17:48
mordredharlowja: so, thikn depends-on in patches17:49
fungizuul (well, excluding zuul-launcher) doesn't really do any tasks/jobs itself. it's mainly a scheduler17:49
mordredharlowja: and the way nova and keystone and glance and devstack all are interrelated17:49
*** shashank_hegde has joined #openstack-infra17:49
*** zul has joined #openstack-infra17:49
harlowjafungi  kk17:49
*** karthikp_ has quit IRC17:49
jeblairharlowja: zuul is a project gating system, at heart:
mordredit's a completely different basic world view17:49
*** signed8bit_Zzz is now known as signed8bit17:49
*** ddieterly is now known as ddieterly[away]17:49
*** javeriak has quit IRC17:49
harlowjainteresting, i wonder if jenkins could have changed (?) to have shared views, (just theorizing)17:50
*** sputnik13 has joined #openstack-infra17:50
jeblairfungi: yes, the scheduling aspect is key17:50
fungito get a similar model in jenkins would, at a minimum, have required interested parties who didn't mind trying to implement that in java ;)17:51
mordredharlowja: anything is possible, but it would essential be a rewrite of the fundamental concepts of jenkins17:51
*** vhosakot has joined #openstack-infra17:51
harlowjafair, i guess thus zuul17:51
harlowja(due to that desire to not try that rewrite and java)17:52
mordredharlowja: as for drop jenkins for ansible ... by the time we had nodepool creating nodes and attaching them to a set of jenkins masters and had zuul scheduling the running of jobs - jenkins had turned in to essentially a remote execution engine for us17:52
harlowjamordred  right17:52
openstackgerritMerged openstack-infra/project-config: Setup constraints for release
AJaegerIsn't the scaling part also a reason for zuul? We used in the end 8 different jenkins instances and needed to distribute work to them.17:53
mordredAJaeger: yup17:53
*** e0ne has joined #openstack-infra17:53
* harlowja gets confused to as how other people (companies) use jenkisn though17:53
harlowjadoes everyone not use jenkins at scale, ha17:53
mordredharlowja: we have VERY intense scale needs that dwarf what other people tend to need17:53
*** tesseract- has quit IRC17:53
fungiscaling more drove zuul 2.0 and its use of gearman so that we could tie in multiple jenkins masters17:54
mordredharlowja: a single jenkins master for us dies on the altar of internal mutexes17:54
harlowjafair, though it makes me wonder how travis is doing it17:54
harlowja(forget if they are using jenkins)17:54
clarkbnot to mention the crippling slave thread leak bug that requires weekly master restarts17:54
fungizuul 1.0 was just associating jobs with one jenkins master17:54
mordredwell, travis also does their own thing and does not use jenkins17:54
* clarkb goes back ti not being here17:54
mordredclarkb: ++17:54
fungiclarkb: you're terrible at not being here17:54
mordredharlowja: also, last time we compared, which was a while ago, we were a bit larger than travis :)17:55
*** thorst has quit IRC17:55
harlowjabigger and badder17:55
*** thorst has joined #openstack-infra17:55
mordredthey've been growing a lot though, so they may have caught up ... they're also openvz container based rather than vm based, so it's not entirely an apples to apples game17:55
fungialso what's scarier, an eldritch demigod with a taste for devouring whole civilizations, or some guy named "travis"?17:56
fungiooh, anteaya may have a point17:56
anteayawatch out for travis17:56
harlowjau never expect travis17:56
mordredbut also they have the benefit of easy use case - since they dont' to interdependent repos, everything for them is like our check queue17:56
fungihe's like the spanish inquisition?17:56
harlowja(vs u do expect zuul to do bad things)17:56
*** _sarob has quit IRC17:57
anteayayou have your eye on the demigod, and travis? he is just doing whatever17:57
harlowjamordred  fair17:58
*** jkilpatr has quit IRC17:58
harlowjasomeone should write all this history down, ha17:58
*** kzaitsev_mb has quit IRC17:58
jeblairmordred: though, our check queue has interdependent repos these days :)17:58
*** mdrabe_ has joined #openstack-infra17:58
openstackgerritKhai Do proposed openstack-infra/puppet-gerrit: fix its_rules parameterization in template
jeblairso, er, it's like our easy use case a couple years ago17:59
zaroZara: ^17:59
mordredjeblair: (ssh, you're making my simplistic story more copmlex)17:59
mordredmmm. copmlex17:59
mordredjeblair: but yeah17:59
*** berendt has quit IRC18:00
mordredharlowja: in any case, we did spend a decent amount of effort making things work with jenkins in each case before we discarded any of its pieces - which has involved actually writing several jenkins plugins, funding the writing of more we never could use and whatnot18:00
*** mdrabe_ has quit IRC18:00
*** mdrabe_ has joined #openstack-infra18:00
harlowjamordred ya, its interesting to know the trials and attempts and such18:00
mordredharlowja: my biggest concern with writing down a history is that it would be taken as a negative at jenkins, which would not be the intent18:01
*** thorst_ has joined #openstack-infra18:01
fungiharlowja: also the jenkins-jclouds and openstack-jenkins-cloud plugins are more along the lines of nodepool than zuul i think (though that distinction is soon to become somewhat moot)18:01
*** rbrndt has quit IRC18:01
mordredharlowja: but I agree, the trials and attempts are potentially good info for people18:01
harlowjamordred  well maybe just don't frame it like that18:01
*** ccamacho has quit IRC18:01
harlowja(like a negative, vs just not a good fit at the time)18:01
mordredharlowja: yah. that would be quite key18:01
jeblairjenkins is very good at what it does, it has served us very well, and the authors are a pleasure to work with.18:01
jeblairwe just do something fundamentally different18:02
fungialso i think a big driver in all of this is our use case for open infrastructure is not well aligned with the design of jenkins that assumes a lot of access control and running things behind firewalls on internal networks18:02
*** mdrabe has quit IRC18:02
*** julim has quit IRC18:02
phschwartzjeblair: so it looks like we are having a nodepool demand calculation issue and it would be appreciated if you could point me at the right place to look.
harlowjaya, it does make me wonder though if they (the jenkins folks) knew about more our ' trials and attempts' that the paths could align at some point18:02
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add gate-python-tripleoclient-buildimage jobs
phschwartzjeblair: all the jobs queued in that paste are all on the bare-trusty label.18:02
*** jkilpatr has joined #openstack-infra18:02
jeblairharlowja: i gave a talk on zuul at a jenkins user conference18:03
jeblairharlowja: we talk to them a lot in various ways :)18:03
*** thorst has quit IRC18:03
fungiwe talked to them a lot more when we were actually running jenkins, granted18:03
jeblairfungi: true, i have not been as involved *lately* :)18:03
*** rockyg has joined #openstack-infra18:04
mordredjeblair: going to the jenkins uc for us _now_ might be a bit strange18:04
fungiharlowja: situations like were pretty frustrating though up to the end18:04
jeblairi have not submitted a zuulv3 talk, no.18:04
harlowjafungi ha, nice18:05
jeblairphschwartz: have you changed the labels for those jobs?18:05
*** julim has joined #openstack-infra18:05
jeblairphschwartz: if so, nodepool may not be mapping them correctly, to fix, restart the gearman server (which may mean restart zuul)18:05
jeblairphschwartz: (the job-label mapping is made through gearman function names, and those stick around permanently until the gearman server is restarted)18:06
phschwartzjeblair: so the label name hasn't changed in 6 months, all that we have done is add jobs using it18:06
fungiharlowja: we basically ended up with ansible in cron automating rolling restarts of all 8 of our jenkins masters weekly, but even that was about half as often as it needed to be after a while18:07
phschwartzZuul was restarted yesterday and it still has this issue. But I don't mind restarting it again.18:07
jeblairphschwartz: can you paste the output of "echo status|nc $ZUUL_SERVER 4730"18:08
phschwartzjeblair: sec, grabbing it18:09
harlowjafungi  ya, seems to be a pita thread junk18:09
openstackgerritMerged openstack-infra/nodepool: Include subnodes in updateStats function
*** _nadya_ has quit IRC18:09
phschwartzjeblair: It also seems we get image-upload and image-delete that hang around in there till restart of zuul.18:11
jeblairphschwartz: yep.  they're harmless but annoying.18:11
*** tpeoples has quit IRC18:12
*** rbuzatu has quit IRC18:12
phschwartzjeblair: The reason why I am asking for help is one of our jobs is a test deployment with full tempest being run and takes upwards of 2 hours. Right now, we have 8 of them taking up all 8 nodes that were biult because of the min-ready: 818:12
phschwartzjeblair: We have a large capacity, we just can't get it to boot more nodes.18:12
openstackgerritMerged openstack-infra/elastic-recheck: Add query for nova resource tracker/scheduler race
jeblairphschwartz: gimme a minute while i run the demand calculation on the back of a napkin18:16
*** tqtran has joined #openstack-infra18:17
openstackgerritAnita Kuno proposed openstack-infra/infra-specs: Making space for new lead
*** tpsilva has quit IRC18:18
*** PollyZ has joined #openstack-infra18:18
*** tonytan4ever has joined #openstack-infra18:21
*** Sukhdev has joined #openstack-infra18:22
jeblairphschwartz: how many bare-trusty nodes do you have in the each state?  (according to 'nodepool list') ?18:22
*** javeriak has joined #openstack-infra18:22
*** PollyZ has quit IRC18:24
jeblairphschwartz: also, can you give me the "Demand from gearman:" log line that goes along with these log entries?
*** seldenr has left #openstack-infra18:24
phschwartzjeblair: 8 in used state18:25
*** karthikp_ has joined #openstack-infra18:25
*** sarob has joined #openstack-infra18:26
openstackgerritMerged openstack-infra/elastic-recheck: Make the index format string configurable in the conf file.
phschwartzjeblair: there is no demand from gearman line in the logs18:27
openstackgerritMatthew Treinish proposed openstack-infra/germqtt: Add config option to set QOS of published messages
AJaegerfungi, I think 349627 is live now - can you confirm, please?18:27
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: WIP - Implement undercloud upgrade job - Mitaka -> Newton
jeblairphschwartz: what version of nodepool are you running?18:29
fungiAJaeger: thanks was just about to confirm18:30
*** xnox has joined #openstack-infra18:30
phschwartzjeblair: the tag before your major changes18:30
AJaegerfungi, I checked puppetboard and it should be applied but can't actually confirm18:31
jeblairphschwartz: can you confirm that you're running 0.3.0 at commit 3a2ed8a755737f34bf4d9842bb44404cc40ecee5 ?18:31
phschwartzjeblair: when I do a --version it is 0.3.018:32
phschwartzjeblair: we pinned the tag in our puppet18:32
*** whoops has quit IRC18:32
fungiAJaeger: oh right, this is job configuration, so i need to see if i can spot how zuul-launcher logs its reconfiguration events18:33
*** vhosakot has quit IRC18:33
fungiAJaeger: from zl01: 2016-08-04 18:11:04,761 DEBUG zuul.LaunchServer: Reconfiguration complete18:33
*** whoops has joined #openstack-infra18:33
jeblairphschwartz: i think that will always print the latest tag, but not if there are any patches on top of it18:33
fungiAJaeger: so i think we're good18:33
*** vhosakot has joined #openstack-infra18:33
jeblairphschwartz: how about pbr freeze, pip freeze, or checking the HEAD commit in /opt/nodepool18:34
AJaegerfungi, I agree. Let's tell the release team...18:34
*** florianf|biab is now known as florianf18:34
*** karthikp_ has quit IRC18:35
phschwartzjeblair: looking18:35
*** karthikp_ has joined #openstack-infra18:35
*** yottatsa_ has quit IRC18:36
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: WIP - Implement non-ovb overcloud update job - Newton -> Newton
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add controller01.vanilla metadata for better tracking
*** ayoung has quit IRC18:38
phschwartzjeblair: pip freeze shows 0.3.018:38
phschwartzjeblair: pschwar@nodepool:/opt/nodepool$ git status18:38
phschwartzOn branch tag/0.3.018:38
fungiwow, was troubleshooting stackalytics.o.o (had become unresponsive and needed a uswgi restart)... in skimming to make sure it was working, i noticed it says that project-config has more commits during newton than any other repo, 1338 so far edging out nova at 1209 (granted the commits involved are not comparable at all, but still an interesting statement on volume)18:40
jeblairphschwartz: when i run the code from 0.3.0 that gets the demand from gearman on the gear output you gave me, i get:   Demand from gearman: bare-trusty: 2318:41
*** rbrndt has joined #openstack-infra18:42
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute040.vanilla metadata for better tracking
phschwartzjeblair: I grepped the nodepool logs for demand with `grep -i demand` and got no return18:42
AJaegerfungi: looking at it's 1408 for nova and 1388 for project-config18:42
phschwartzjeblair: is there a chance that an old version is running in memory since the last install?18:42
AJaegerLet's merge some more ;)18:42
fungiAJaeger: i wonder why they're giving different results18:43
jeblairphschwartz: absolutely.  did you run something after 0.3.0 before you pinned it to 0.3.0?18:43
jeblairphschwartz: what's the start time for nodepoold in ps?18:43
AJaegerfungi, so do I...18:44
*** florianf has quit IRC18:44
phschwartzjeblair: Jul19 1602:2718:44
phschwartzbut that should be after 0.3.018:44
AJaegerare both using the same calculation for start of newton?18:44
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add gate-python-tripleoclient-buildimage jobs
jeblairphschwartz: yes, but did you pin it yourself, or is it pinned because I289186b7a25755c9258897202f01d278d0cc5a75 landed this morning?18:45
phschwartzjeblair: pin'ed in our system-config that uses puppet-openstackci18:45
jeblairphschwartz: when did that take effect?18:45
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute042.vanilla metadata for better tracking
phschwartzjeblair: Spencer Krum         2016-07-21 11:51:24 -0700 147)     revision                  => '0.3.0',18:46
nibalizeruh oh18:46
phschwartzjeblair: so looks like nibalizer pinned it after that. Let me restart nodepool18:46
nibalizergetting called out18:46
phschwartznibalizer: lol18:47
phschwartznibalizer: jeblair is helping me track down our demand issue18:47
*** Goneri has joined #openstack-infra18:47
fungiAJaeger: they should be using the same configuration entirely. they both get it from the stackalytics git repo18:47
*** ddieterly[away] is now known as ddieterly18:47
phschwartzjeblair: ok, now I see the demand line in the logs18:48
AJaegerstrange, fungi18:48
phschwartzjeblair: 2016-08-04 18:46:22,499 DEBUG nodepool.NodePool:   Demand from gearman: bare-trusty: 718:48
phschwartzjeblair: but the queue has cleared to alomst nothing since earlier.18:48
phschwartzjeblair: I might have to wait for it to get loaded down again18:48
jeblairphschwartz, nibalizer: yeah, you were running a version that had a malfunctioning demand calculation.18:49
phschwartzjeblair: oh, so this was known?18:49
*** yottatsa has joined #openstack-infra18:49
fungi(and later fixed)18:49
jeblairphschwartz: yep, we fixed it on jul 20.18:49
fungiit hit us too, which is how it was discovered18:49
fungiif memory serves18:49
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute002.vanilla metadata for better tracking
phschwartzah, ok. so it should work now. awesome18:50
jeblairfrom jul 11 to jul 20, demand calcs were wrong-ish in various forms.18:50
fungiphschwartz: nibalizer: i'm guessing it got pinned there because of the database schema change disruption?18:50
*** daemontool has quit IRC18:51
openstackgerritKhai Do proposed openstack-infra/system-config: Fix setup of server certificate for its-storyboard
jeblairfungi: the tag they pinned to was before this, but the pin was applied after they started running the buggy version18:51
jeblairso, yeah, everything should be good now :)18:51
*** yottatsa has quit IRC18:52
jeblair(iow, they have reverted to before the bug was introduced)18:52
*** daemontool has joined #openstack-infra18:52
AJaegerpabelanger: I mean: is a file. The sentence reads Check the grafana directory in project-config - and then gives a file URL instead of an URL for the grafana directory18:53
AJaegerpabelanger: but I just noticed it's wrong everywhere, so don't bother...18:54
jeblairwell, it's not *wrong*...18:54
jeblairit's just extra helpful18:54
*** rbuzatu has joined #openstack-infra18:55
AJaegerjeblair: ;)18:55
*** ddieterly is now known as ddieterly[away]18:55
*** pvaneck has joined #openstack-infra18:56
zaronibalizer: what's up with Vinz?  you don't like the new gerrit UI either?18:57
phschwartzzaro: gerrit ui is bad, always has been. lol18:57
zarophschwartz: i'm talking about the new new UI18:58
phschwartzzaro: the new new ui is still grouped in that. lol18:58
mordredzaro: there is a new new ui?18:58
nibalizerzaro: nerp18:59
jeblairi think zaro may be asking about the polygerrit ui18:59
jeblair(aka the new new *new* gerrit ui)18:59
zaroyes, you can try it with this chrom plugin
*** cody-somerville has joined #openstack-infra18:59
zarotry it against gerrit's gerrit at https://gerrit-review.googlesource.com19:00
*** stewie925 has quit IRC19:00
fungianteaya: who was the kaizen that you added as a wiki admin on may 13? they just mass deleted a whole bunch of user pages claiming they were spam19:01
pabelangerAJaeger: Oh, ya. We can fix that in a follow up patch19:01
*** csomerville has quit IRC19:02
*** mdrabe_ has quit IRC19:03
*** mdrabe__ has joined #openstack-infra19:03
openstackgerritRichard Theis proposed openstack-infra/project-config: Add release-notes-jobs to osc-lib
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add tripleo-test-cloud grafana dashboard
*** ccamacho has joined #openstack-infra19:05
AJaegerpabelanger: leave it as is, seeing it everywhere, let's not change the URL - and I'm not creative in a better text...19:06
mordredfungi, pleia2: so - it seems that gitiles does the README rendering people keep asking for:
*** ldnunes has quit IRC19:07
*** fifieldt has quit IRC19:07
mordred(I got sucked down a hole by looking at polygerrit)19:07
openstackgerritMatthew Treinish proposed openstack-infra/puppet-logstash: WIP: Add mqtt output plugin to logstash indexers
mtreinishfungi, clarkb, rcarrillocruz: ^^^ a random thing we could add to firehose19:08
*** Goneri has quit IRC19:08
mtreinishalthough that depends on finishing the tls bits19:08
mtreinishit also won't work because all the logstash docs say run 'bin/plugin' but never say what the cwd is....19:09
rcarrillocruzoh nice...19:09
mtreinishI'm still not sure the best way to generate the certs19:09
*** armax_ has joined #openstack-infra19:11
fungimtreinish: generate the certs. you mean for an ssl/tls server cert?19:11
*** daemontool_ has joined #openstack-infra19:11
AJaegerfungi, for constraints I consider as next steps:19:12
*** armax has quit IRC19:12
*** armax_ is now known as armax19:12
AJaeger1) enable constraints for releasenotes We could use the new zuul-release-git-prep-upper-constraints instead of zuul-git-branch-prep-upper-constraints.19:12
AJaeger2) Rename  zuul-git-branch-prep-upper-constraints to zuul-release-git-prep-upper-constraints.19:12
AJaeger3) Use constraints for translation sync:
mtreinishfungi: yeah19:12
AJaegerfungi, for that I need to update the changes - but first would like to hear whether you agree with the plan19:12
fungimtreinish: i usually just buy them and then expense them to the foundation19:13
*** sdake_ has joined #openstack-infra19:13
mtreinishfungi: I dug through my history and when I setup my moquitto at home I used:
*** javeriak has quit IRC19:13
zaromordred: ohh, i didn't even realize it but looks like there's even single page diff view on new UI.19:14
fungiAJaeger: i don't think we can combine zuul-git-branch-prep-upper-constraints with zuul-release-git-prep-upper-constraints (or did you mean rename it to zuul-release-git-branch-prep-upper-constraints?)19:14
mordredzaro: there is? how do I see it?19:14
*** chem has quit IRC19:14
mordredzaro: I went to with that extensoin installed19:14
zaro'show diffs' link19:14
*** daemontool has quit IRC19:15
*** maishsk has joined #openstack-infra19:15
mordredthat is not obvious to me :)19:15
*** mdrabe__ is now known as mdrabe19:15
mordredbut thank you - that's great!19:15
zarowow, it's fast19:15
AJaegerfungi, I meant "mv", so basically use the new zuul-release-git-prep-upper-constraints whereever currently zuul-git-branch-prep-upper-constraints is used.19:15
*** sdake has quit IRC19:16
*** thorst_ is now known as thorst19:16
*** javeriak has joined #openstack-infra19:16
mordredzaro: that is an improvement19:16
AJaegerand rename it as part of that so that we only have zuul-git-branch-prep-upper-constraints19:16
mordredit would be great if it wasn't tied just to urls19:16
fungiAJaeger: i don't think that's possible. the zuul-git-branch-prep-upper-constraints macro is meant for periodic pipelines where we have to supply a {branch} parameter to tell it what branch to run on19:17
mordredzaro: I have a feeling that there are people in our community who might be interseted in running the browser extension just to get single-page-diff19:17
*** rbuzatu has quit IRC19:17
AJaegerfungi, oooh, indeed.19:17
fungiAJaeger: hence the "branch" in its name19:18
zaromordred: it should work with any gerrit 2.12.19:18
mordredoh. we're not on 2.12 yet19:18
zaroactually is actually 2.1319:18
*** fifieldt has joined #openstack-infra19:18
mordrednibalizer: if you haven't looked at polygerrit through the chrome extension zaro linked above - it's worth looking at to see what might be learned for vinz19:19
AJaegerfungi, yeah ;( I got confused19:19
clarkbdo you still have to run a separate service for it?19:19
clarkbif so that seems like a gerrit api bug19:20
mordredyou do not need to install the chrome extensoin19:21
openstackgerritPaul Belanger proposed openstack-infra/project-config: Replace gate-{name}-tox-{envlist} with gate-{name}-tox-{envlist}-{node}
AJaegerfungi, checked again your comment - it was "agreed on dropping --branch in the non-branch versions of the zuul-cloner wrapper macros. i too suspected they were there as a workaround for older z-c behavior" - and I misremembered ;(19:21
mordredif you go to:
mordredit will enable the polygerrit ui19:21
fungiAJaeger: right, we can't drop --branch from the branch parameterized versions of the macro, but we should be able to stop using it in the ones which don't take a branch parameter19:22
AJaegerlet me propose that change then...19:22
fungiAJaeger: and ideally that would be the only difference i think (or we could have it set ZUUL_BRANCH instead of passing --branch to zuul-cloner)19:22
*** ddieterly[away] is now known as ddieterly19:23
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute021.vanilla metadata for better tracking
nibalizerchorme extesinon?19:24
nibalizerthats funny because vinz lived for a little while as a greasemonkey script19:24
*** thorst has quit IRC19:25
*** ihrachys has joined #openstack-infra19:25
*** rbuzatu has joined #openstack-infra19:25
AJaegerfungi, that will be the case after my change19:26
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Remove unneeded --branch from zuul-cloner
AJaegerfungi, ^19:27
*** thorst has joined #openstack-infra19:28
*** thorst has quit IRC19:29
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute018.vanilla metadata for better tracking
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Enable constraints for releasenotes
AJaegerfungi, would appreciate a review of this as well ^19:31
*** thorst has joined #openstack-infra19:32
*** bhunter71 has quit IRC19:33
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute027.vanilla metadata for better tracking
*** _nadya_ has joined #openstack-infra19:33
openstackgerritMerged openstack-infra/tripleo-ci: Remove pinning of delorean
AJaegerfungi, and once is in, we have everything ready for constraints that I'm aware of19:35
*** javeriak has quit IRC19:35
openstackgerritMerged openstack-infra/tripleo-ci: Add undercloud job in cistatus
*** berendt has joined #openstack-infra19:35
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute028.vanilla metadata for better tracking
*** karthikp_ has quit IRC19:36
*** tonytan_brb has joined #openstack-infra19:36
*** PollyZ has joined #openstack-infra19:36
*** Apoorva has quit IRC19:37
*** burgerk has quit IRC19:37
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute029.vanilla metadata for better tracking
*** baoli has quit IRC19:38
*** tonytan4ever has quit IRC19:38
*** Sukhdev has quit IRC19:39
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add gate-tripleo-quickstart-base-centos-7-nv job
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute020.vanilla metadata for better tracking
*** xnox has quit IRC19:41
*** baoli has joined #openstack-infra19:42
*** karthikp_ has joined #openstack-infra19:44
*** kzaitsev_mb has joined #openstack-infra19:45
*** kzaitsev_mb has quit IRC19:50
*** kzaitsev_mb has joined #openstack-infra19:52
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute026.vanilla metadata for better tracking
*** maishsk has quit IRC19:53
*** rakhmerov has quit IRC19:53
*** vhosakot has quit IRC19:55
openstackgerritRichard Theis proposed openstack-infra/project-config: Add release-notes-jobs to osc-lib
mgagneso I'm trying to use voluptuous, thanks to grafyaml, I found good example of how to use it. Now I'm trying to find a way so one of 2 dict keys needs to be provided, maybe both I suppose but not none.19:55
*** shamail has quit IRC19:56
fungianteaya: nevermind my question from earlier... kaizen on the wiki is apparently shamail on freenode so we've straightened it all out19:56
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute022.vanilla metadata for better tracking
*** maishsk has joined #openstack-infra19:56
*** ayoung has joined #openstack-infra19:56
*** rakhmerov has joined #openstack-infra19:57
*** rbuzatu has quit IRC19:58
*** matrohon has joined #openstack-infra19:58
*** maishsk has quit IRC19:59
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute013.vanilla metadata for better tracking
*** rockyg has quit IRC19:59
pabelangermgagne: you'll need to write your own validation function. Something like: but check if None19:59
*** asettle has joined #openstack-infra20:00
mgagnepabelanger: yea, I'm depth into writing custom validator atm. I guess it will be a matter of detecting the presence of those keys and updating the schema accordingly.20:00
*** eharney has quit IRC20:02
ianwgreghaynes / cinerama : we meeting, or did i get my weeks messed around?20:02
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute010.vanilla metadata for better tracking
greghaynesianw: I think next week20:02
*** maishsk has joined #openstack-infra20:03
*** maishsk has quit IRC20:03
pabelangermgagne: but, v.Any() should be work too20:03
*** mmandel has quit IRC20:03
cineramaianw: we're alternating, i still need to submit the change :P20:04
mgagnepabelanger: oh, I see what you mean, tyvm for the hint!20:04
*** asettle has quit IRC20:04
cineramadib appears to be on summer holidays at the mo :)20:05
*** cody-somerville has quit IRC20:05
cinerama*(summer subject to availability in your region)20:05
*** zul has quit IRC20:05
ianwgreghaynes: for wouldn't it go under #openstack- ?20:06
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute019.vanilla metadata for better tracking
*** rbuzatu has joined #openstack-infra20:06
cineramagreghaynes: oh cool thanks for putting that forward20:07
cineramaalso, i agree20:07
ianwcinerama: yeah, i've been slack on reviews, trying to get f24 working20:07
*** vhosakot has joined #openstack-infra20:08
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add gate-tripleo-quickstart-base-centos-7-nv job
*** gomarivera has joined #openstack-infra20:09
ianwpabelanger: which, i just realised my cunning plan to install a custom kernel from copr might blow up as dib writes out the final image and doesn't pick up my new kernel ... have to see20:09
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute006.vanilla metadata for better tracking
*** Apoorva has joined #openstack-infra20:10
greghaynescinerama: bkero did it, not I :)20:10
*** Apoorva has quit IRC20:10
*** Apoorva has joined #openstack-infra20:10
*** vhosakot has quit IRC20:10
*** gomarivera has left #openstack-infra20:10
greghaynesianw: cinerama I am fine with #openstack-dib, or the proposed one, either way. One issue with #openstack-dib is someone squatted it after we mentioned that channel in our meeting20:11
greghaynesso well need to deal with freenode opers20:11
ianwohh, presumably it's not a person we can ask to leave?20:12
bkerogreghaynes: what did I do now?20:12
*** vvalderrv has left #openstack-infra20:12
greghaynesthe channel patch20:12
greghaynesianw: I asked once and didnt get a reply20:13
bkeroYeah, I did.20:13
greghaynesianw: next step is probably to figure out their email and ask that way, actually20:13
bkeroIt doesn't have gerritbot though because gerritgot has maxed channels20:13
*** baoli has quit IRC20:13
*** markvoelker has quit IRC20:13
*** baoli has joined #openstack-infra20:14
*** rbuzatu has quit IRC20:14
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute000.vanilla metadata for better tracking
pabelangerjeblair: fungi: any objections to restart nodepool to pick up 349119? Updates stats to include subnodes20:16
jeblairgreghaynes: you deal with me for that20:17
jeblairgreghaynes: openstack-infra has the group registration for #openstack-* channels, so we can fix it.20:17
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute001.vanilla metadata for better tracking
*** eharney has joined #openstack-infra20:19
fungipabelanger: i think a bunch of people were hoping to have bindep 2.0.1 from yesterday make it into centos 7 images, so maybe make sure those have all uploaded first?20:19
*** vhosakot has joined #openstack-infra20:19
pabelangerfungi: Ya, I want to let nodepool-builder finish its uploads first too20:20
fungiAJaeger: i'll take a look. can you set the topic on them to something consistent like "constraints"?20:20
jeblairpabelanger: no objection20:20
*** sarob has quit IRC20:20
phschwartzjeblair: fungi: that was defn the issue. I have 34 instances spawned by demand now.20:20
fungibkero: saw your gerritbot discussion earlier. i think "sharding" could be as simple as tweaking the puppet module to run more than one gerritbot, using separate channel lists and freenode accounts. doesn't (i think) need a change to the gerritbot code at least20:21
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute034.vanilla metadata for better tracking
greghaynesjeblair: ah, awesome20:22
pabelangerjeblair: fungi: actually, it looks like DIBs didn't build today.  Looks to be a hung process for disk-image-create is blocking things20:22
jeblairgreghaynes: shall i start the process?20:22
pabelangerif you want to check nodepool.o.o20:22
greghaynesjeblair: That'd be great, thanks20:22
jeblairgreghaynes: will do20:22
*** kzaitsev_mb has quit IRC20:23
pabelangerfungi: jeblair: actually, I might be wrong about that. let me check something20:23
pabelangerHmm, disk-image-create is running but nodepool dib-image-list doesn't show it20:24
greghaynespabelanger: I think that happens when the builder isnt grabbing the job - if it doesnt know to register as a worker for that image name, etc20:24
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute017.vanilla metadata for better tracking
greghaynesI may have foot-gunned that a day ago when adding a new image and forgetting to restart the builder ;)20:25
fungicould the zuul geard restart a couple days ago have caused that?20:25
fungioh, got it20:26
*** HeOS has joined #openstack-infra20:26
fungiyeah, i forgot changes to the list of images needs a builder restart right now20:26
fungii guess we never did get otf reloads going for that20:26
*** rbrndt has quit IRC20:28
*** ihrachys has quit IRC20:29
*** PollyZ has quit IRC20:29
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute039.vanilla metadata for better tracking
pabelangerfungi: okay, so we can restart both nodepool and nodepool-builder I guess20:31
*** e0ne has quit IRC20:31
fungircarrillocruz: remember as you start bringing servers back up that we want to use a separate subdomain zone since rackspace limits the number of records we can have in a single dns zone and we're pushing pretty close to that in already20:31
openstackgerritTimothy R. Chavez proposed openstack-infra/jenkins-job-builder: Add support for the random string parameter
*** e0ne has joined #openstack-infra20:32
timrcI'm not proud of this^^20:32
bkerofungi: I haven't read the puppet module, but I'm pretty sure this should be very simple. I just wanted confirmation that 'gerritbot' and 'gerritbot2' sharding would be okay.20:33
rcarrillocruzi have a long way still to make metadata and ilo ips consolidation20:33
rcarrillocruzi have like...100 and something servers to go20:33
fungircarrillocruz: we have an zone already created for these, and right now only has a baremetal00.vanilla address record in it20:33
timrcrcarrillocruz: If you have another child, I'm going to recommend naming them ilo :)20:34
rcarrillocruzyeah, i think that was nibalizer that added it20:34
*** jkilpatr has quit IRC20:34
rcarrillocruztimrc: lulz20:34
fungircarrillocruz: it probably was (or maybe it was me, my memory is terrible)20:34
fungii know we discussed it at length anyway20:34
rcarrillocruzotoh, i could call him grunt, for *grunt work*20:34
*** Jeffrey4l has quit IRC20:34
*** Jeffrey4l_ has joined #openstack-infra20:35
jeblairgreghaynes: #openstack-dib is all set up; you have ops-level perms with chanserv20:36
fungibkero: sure, making the second conffile, pidfile, logfile, et cetera be gerritbot2 and leaving the current gerritbot instance alone is fine by me. once it's puppeted we can split the current channel list in half and just let people add to which ever one and rebalance them if we ever start to fill up again before we get around to making a gerritbot replacement20:36
bkerofungi: sounds good to me20:37
fungiharlowja: that brings up an interesting irc bot challenge... we're exceeding the freenode channel max on our gerritbot account, curious if errbot has solutions for this other than "run multiple errbots"20:37
harlowjanot quite sure20:38
fungigerritbot is our most popular one, but wouldn't be surprised if meetbot and eventually statusbot experience similar issues20:38
harlowjamy guess is run multiple :-P20:38
*** _ari_ has quit IRC20:39
harlowja is that backend20:39
jpmaxmanok fungi20:39
fungilike, an errbot feature allowing it to connect to multiple accounts under different nicks and somehow separate actions/triggers between them20:39
harlowjaoh maybe, not sure20:39
jpmaxmanthe mediwiki team helped it turned out it was a stupid mis-named key on my part20:39
fungijpmaxman: saw my comment earlier about the database dump being there for you?20:39
jpmaxmanso 1.27 is working20:40
jpmaxmanin the troubleshooting I ended up installing a real cert from letsencrypt20:40
jpmaxmanthe mobile view is the only extension that isn' working20:40
*** hemanthm is now known as hemanthm|afk20:40
jpmaxmanbut I sort of feel like we shouldn't use it anyway20:40
*** jkilpatr has joined #openstack-infra20:40
jpmaxmanas it opens up a login vulnerability20:40
fungii wonder if wikipedia obsoleted that for something else?20:40
jpmaxmanmaybe I can dig into it more for sure20:41
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: test if overcloud update can actually fail
fungiand yeah, i don't want more surface area to have to cover spam mitigation across20:41
jpmaxmanbut for the first time I feel like what we have on the test server *might* be better20:41
jpmaxmanthan what we have now20:41
timrcrcarrillocruz: You working US hours right now?  Awfully late there isn't it?20:41
jpmaxmanso I thought I'd discuss next steps20:41
fungijpmaxman: i assume you had to do an in-place distro upgrade from trusty to xenial?20:41
rcarrillocruzwell, not really20:41
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Update iLO IP for compute016.vanilla
fungijpmaxman: between 1.26 and 1.27?20:42
timrcI guess only 10PMish?20:42
rcarrillocruzit's just that i had today an appointment at the child hospital20:42
rcarrillocruzand got back home at 6PM20:42
*** dimtruck is now known as zz_dimtruck20:42
greghaynesjeblair: wooo ty!20:42
rcarrillocruzso working a bit till my eyes close20:42
*** mikelk has joined #openstack-infra20:42
timrcrcarrillocruz: Oh noes.  Hope things went well at the appt.20:42
jpmaxmanI did an in place from precise to trusty20:42
fungijpmaxman: oh, right, that's what i meant20:42
jpmaxmanwhich was actually required20:42
jpmaxmaneven for 1.2620:43
fungijpmaxman: but between the 1.26 and 1.27 upgrades presumably?20:43
rcarrillocruzit went ok, we have another check next week20:43
fungioh, before 1.26?20:43
jpmaxman1.25 --> 1.26 required trusty20:43
fungigot it20:43
jpmaxmanto get on the right version of php20:43
jpmaxmanthere were work arounds20:43
fungijust trying to get the sequence straight in my head20:43
jpmaxmanbut I figured just push forward20:43
*** pvaneck_ has joined #openstack-infra20:43
*** vhosakot has quit IRC20:43
fungijpmaxman: so do you have a summary of the steps you had to follow, such that we could repeat them on the production deployment?20:43
jpmaxmanso yeah we're running on trusty20:43
jpmaxmanI tried to follow the conventions in place20:44
jpmaxmanso there was slot0 and slot120:44
jpmaxmanwith wiki installs20:44
*** e0ne has quit IRC20:44
fungiyeah, i think mediawiki does red/blue upgrades on their servers, so that's how Ryan_Lane set it up20:44
*** vhosakot has joined #openstack-infra20:44
jpmaxmanI put slot2 --> 1.25 latest, slot3 --> 1.26 latest, slot 4 1.27 latest20:45
openstackgerritMonty Taylor proposed openstack/os-client-config: Revert "Fix precedence for pass-in options"
fungijpmaxman: oh, cool that works20:45
jpmaxmanI then rain the maintenance/update.php20:46
jpmaxmanfor each version20:46
*** pvaneck has quit IRC20:46
jpmaxmanto upgrade the database in steps20:46
fungiand did teh distro upgrade between 1.25 latest and 1.26 latest?20:46
jpmaxmanI moved the extensions and skins directory to /srv/mediawiki20:46
jpmaxmanand put in symbolic links for the ones we used20:47
fungidid you have to do anything with the static file tree it uses for uploaded content or did it just reuse something central?20:47
jpmaxmaninto /srv/mediawiki/slot4/skins /msr/mediawiki/slot4/extensions20:47
jpmaxmanI had to make the cache directory writable20:47
*** tonytan_brb has quit IRC20:48
jpmaxmanI think the uploads go here20:48
jpmaxmaninto /srv/mediawiki/images20:48
jpmaxmanwhich was already aliased and writable20:48
jpmaxmanbut we should probably test - we disabled file uploads20:48
jpmaxmanif we want to reanable we can20:49
thiagolibhow do I set up the cinder-volume active / passive20:49
fungijpmaxman: i'm more worried about existing files still showing up. checking around now20:49
openstackgerritBen Kero proposed openstack-infra/project-config: Add channel #diskimage-builder
jpmaxmanok cool - I can't say I did a deep QA dive so a good idea to poke on i20:49
fungijpmaxman: though yeah, finding out whether the new recaptcha also covers file uploads in 1.27 would be great20:49
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Update iLO IP for compute004.vanilla
fungijpmaxman: mostly i think i just want to look around wiki-upgrade-test and then figure out some time when we can announce the wiki going offline for a while to take a fresh server snapshot (as an insurance policy) and repeat the upgrade steps on wiki.o.o20:50
*** zz_dimtruck is now known as dimtruck20:50
openstackgerritBen Kero proposed openstack-infra/project-config: Add channel #openstack-dib for diskimage-builder
*** vhosakot has quit IRC20:51
pabelangerjeblair: fungi: I've restarted nodepool, however I haven't see an Allocation requests since restarting. Did I miss something?20:51
jpmaxmanI guess in the interest of getting the production wiki functional without spam as quickly as we can I'd probably leave it disabled and then iterate forward20:51
fungijpmaxman: once we have production running 1.27 that gives us a stable target to try and hit with our configuration management work rather than trying to have to deal with multiple mediawiki versions we're not running20:51
jpmaxmanyeah I agree20:51
pabelangerlast node calculation was at 20:33UTC20:51
jpmaxmanI'm familiar enough with it now20:52
jpmaxmanI think we could do a fresh trusty20:52
jpmaxmanif you didn't want to do an in place upgrade20:52
jpmaxmanit's up to you though20:52
fungijpmaxman: oh? i'd be happy to deploy a fresh trusty replacement, sure20:52
fungijpmaxman: that does make me a little more comfortable than having a server upgraded from precise to trusty (i'm less concerned about the in-place upgrade of the mw pieces)20:53
jpmaxmanIt's basically a LAMP stack and copy what's in /srv/mediawiki + Apache config + MySQL20:53
fungijpmaxman: awesome. i have a number of other plans to improve this (move database to trove, static content files to a cinder volume, et cetera) to make redeployments smoother but this is a great first step20:54
jpmaxmanyeah in place MW is somewhat necessary to migrate the db20:54
jpmaxmancool - biggest take away is just stay on top of keeping it updated20:55
*** raildo has quit IRC20:55
*** rbrndt has joined #openstack-infra20:55
fungijpmaxman: yes, agreed. that's the main reason we want to have it under configuration management. the existing deployment was not well documented and nobody'd found the time to dig into it and document it. in our case configuraiton management serves well as a summary documentation of the important parts of the system, but we didn't even have that20:56
*** _nadya_ has quit IRC20:57
*** matrohon has quit IRC20:57
*** gomarivera has joined #openstack-infra20:57
fungijpmaxman: so anyway, this is awesome--thank you so much. let's sync up next week if you're around and figure out what sort of timeframe we should be looking at for an outage to lock down production so people aren't updating it while we upgrade/switch to the replacement server20:59
fungiyou presumably have some feel for how long the db migrations take to complete at each stage20:59
pabelangerfungi: jeblair: nodes are launching now.  Took almost 30mins for nodepool to start doing that after the restart20:59
*** piet_ has quit IRC20:59
jpmaxmanok great - in the man time if anyone wants to poke around on the test server and let me know if there are deal breakers for making that version the production version, it'd be great20:59
fungijpmaxman: absolutely. i'll round up some test subjects21:00
fungithanks again!21:00
jpmaxmanThe db migrations were fast21:00
jpmaxmanless than a minute21:00
jpmaxmanfor each21:00
jeblairpabelanger: wait, why?21:00
fungijpmaxman: oh, then we can probably schedule the cut-over on pretty short notice21:00
*** jkilpatr has quit IRC21:00
jpmaxmanthe majority of time was me troubleshooting ;)21:00
*** rhallisey has quit IRC21:01
pabelangerjeblair: I am still trying to figure that out21:01
jpmaxmanI'd give us an hour or so to be safe21:01
*** psilvad has quit IRC21:01
fungijpmaxman: i agree, more time is better just in case of the unexpected21:01
*** sputnik13 has quit IRC21:02
jpmaxmanok I'm going to call it a night for now - will be around tomorrow if you need anything21:02
*** baoli_ has joined #openstack-infra21:02
notmorganjeblair: *sigh* i just wasted hours wondering why my change was conflicting with everything... git branch -v shows I was on master not feature/zuu;v3 ugh21:03
notmorganjeblair: just sharing, cause i'm sure that's never happened to you or anyone else :P ever21:03
*** baoli has quit IRC21:04
fungijpmaxman: have a good night--thanks again21:05
pabelangerjeblair: I think I man have done something incorrectly and could use some help.  In the past, we would restart nodepool and then clean up any left over nodes from jenkins.  Should I have done something different with zuul-launchers?21:05
*** yamamoto has joined #openstack-infra21:06
jeblairnotmorgan: nevar!  :)21:07
jeblairpabelanger: nothing beyond restarting nodepool should be necessary21:08
pabelangerjeblair: or maybe our demand calculation is out of sync again? We have about 200+ ready ubuntu-trusty nodes21:09
*** gomarivera has quit IRC21:09
*** thorst has quit IRC21:10
jeblairpabelanger: they may be running jobs with nodepool having missed the start event21:11
*** PollyZ has joined #openstack-infra21:12
*** baoli_ has quit IRC21:12
*** baoli has joined #openstack-infra21:12
*** yamamoto has quit IRC21:13
*** gordc has quit IRC21:13
pabelangerjeblair: okay, as new jobs come online they should get picked up? or is zuul-launcher release needed?21:13
jeblairpabelanger: if they are running jobs, nodepool will get the complete event and delete them.  release is only for idle nodes, and there should be none with our current load21:14
pabelangerokay thanks. I'll sit back and watch for a bit to see what happens21:15
*** thorst has joined #openstack-infra21:15
notmorganjeblair: so.. i now have zuul running this test case but it always hits SIGLARM *and* always just spins forever even when i put my capture log hack in21:16
notmorganjeblair: i am unsure where to go next here.21:16
jeblairnotmorgan: i usually run in the foreground with python -m tests.test_scheduler.TestScheduler.test_parallel_changes21:16
*** ddieterly is now known as ddieterly[away]21:17
*** signed8bit is now known as signed8bit_Zzz21:17
*** baoli has quit IRC21:18
*** julim has quit IRC21:18
notmorganjeblair: thnx.21:18
*** julim has joined #openstack-infra21:18
*** baoli has joined #openstack-infra21:18
*** Sukhdev has joined #openstack-infra21:18
*** eharney has quit IRC21:19
*** thorst has quit IRC21:19
*** signed8bit_Zzz is now known as signed8bit21:21
openstackgerritMonty Taylor proposed openstack/os-client-config: Pass the argparse data into to validate_auth
*** julim has quit IRC21:23
thiagolibgeguileo: hi i am trying to set the cinder - volume active / passive , and would like to know what I need to configure in cinder.conf and also how should I set the cinder - volume service to start on both nodes .21:24
pabelangerjeblair: I'm starting to think we have a problem with nodepool.o.o, we have 123 ubuntu-xenial nodes that are ready, with pending jobs in the check and gate queues21:25
*** baoli has quit IRC21:26
*** sarob has joined #openstack-infra21:26
*** mikelk has quit IRC21:26
*** amotoki has quit IRC21:27
fungipabelanger: log into a couple and see if there's anything in ~jenkins/workspace/?21:28
*** priteau has quit IRC21:28
pabelangerfungi: yes, it appears jobs are running on some of them21:29
fungimight be zeromq is letting us down again?21:29
*** yamahata has quit IRC21:30
pabelangerhow can I check that?21:30
*** xnox has joined #openstack-infra21:30
*** lbeliveau has joined #openstack-infra21:30
pabelangerya, a random sample of 5 nodes, show they are all running / ran jobs21:30
*** sarob has quit IRC21:31
*** matt-borland has quit IRC21:31
fungithe nodepool debug log claims to be setting some to used or delete21:31
fungiso maybe it's just some launchers21:31
fungii'm running some quick stats across them21:31
*** pcarver has joined #openstack-infra21:32
*** sputnik13 has joined #openstack-infra21:32
jeblairi see zmq events from all 721:32
fungiready node distribution across launchers looks pretty consistent too21:32
pabelanger3194612 is running a job21:33
pabelangerbut nodepool still thinks it is ready21:33
jeblairpabelanger: if what i suggested happened earlier has happened, i would not expect it to correct itself for about an hour21:33
pabelangerjeblair: okay21:33
jeblairpabelanger: they will have to run and complete jobs before nodepool updates their status21:34
openstackgerritPaul Carver proposed openstack-infra/project-config: Enable api-ref publishing for networking-sfc
fungiyeah, chances are we can just consider the ready node count to be part of the used node count for the next little while21:34
pabelangersure, if both are good with that, I am too21:34
jeblairpabelanger: if you want to confirm this, you could parse the zuul status json and see how many jobs are actually running, and see if that number approximates ready+used21:34
*** xnox has quit IRC21:35
jeblairoh, the gearman graph does tell us that21:35
jeblairit says 266 are running21:36
fungiwhere we were flat at around 500 before the restart21:36
*** dimtruck is now known as zz_dimtruck21:36
jeblairso there may be something else at play21:36
jeblairfungi: i'd probably go with 450 for than number21:37
fungisure, i was looking at the tiny graph where a 10% deviation is tough to spot21:37
jeblairah heh21:37
notmorganjeblair: so I'm feeling really dumb right now.did zuul in v2 just roundrobin the connections to the same gerrit? i'm looking at testConnections and don't see how it used the second connection (civoter) previously since both "approvals" were sent with self.fake_review_gerrit21:39
pabelangerWe restart at 20:33UTC so were an hour in right now21:39
pabelangerjeblair: fungi: I'll have to step away shortly for a family dinner but will be able to follow up once I get back21:39
fungithe other disturbing behavior i see is, running a count of ready nodes reported by nodepool list, the total number across all types isn't really decreasing (hovering between 280 and 285)21:40
fungino, wait, it just took a largeish drop to about 27021:40
funginow it's falling somewhat steadily so maybe it is self-correcting21:41
notmorganoh nvm i see it now21:41
notmorganbrain... not connecting some strings.21:41
openstackgerritMerged openstack-infra/infra-specs: Making space for new lead
pabelangerfungi: ya, used nodes is starting to tick up21:41
stevemarcan we get views to set up release notes setup for osc-lib we're eager to release it soon21:42
pabelangerokay, have to run. Will check back in and see the results21:42
fungithanks pabelanger21:42
jeblairnotmorgan: the approvals go via different connections (cause they are specified in the layout) -- the only thing the test does is create the fake changes, it doesn't leave any approvals in that one21:42
*** ayoung has quit IRC21:43
*** zz_dimtruck is now known as dimtruck21:43
notmorganjeblair: yeah now i just need to figure out why the failure is getting "event.type = data.get('type') with a NoneType object has no attribute.get21:43
*** kzaitsev_mb has joined #openstack-infra21:48
*** ddieterly[away] is now known as ddieterly21:48
*** yamahata has joined #openstack-infra21:49
*** amotoki has joined #openstack-infra21:53
*** esberglu has quit IRC21:54
*** thorst has joined #openstack-infra21:56
*** dprince has quit IRC21:59
*** amitgandhinz has quit IRC21:59
*** mdrabe has quit IRC22:02
*** thorst has quit IRC22:03
asselinrcarrillocruz, you around? I'm having issues with ansible-role-cloud-launcher linters: ERROR! the role 'ansible-role-cloud-launcher' was not found in
*** gouthamr has quit IRC22:05
*** thorst has joined #openstack-infra22:05
*** gouthamr has joined #openstack-infra22:05
*** gouthamr has quit IRC22:05
jeblairi think the disk on is full22:05
*** PollyZ has quit IRC22:05
*** sarob has joined #openstack-infra22:07
*** thorst has quit IRC22:07
*** thorst has joined #openstack-infra22:08
fungiafter looking at df, i am inclined to believe you22:08
fungilooks like logs22:09
fungiit's /var/log/logprocessor/log-client-debug.log*22:10
*** karthikp_ has quit IRC22:10
fungilooks like it may have ceased rotating22:10
fungiyeah, hasn't rotated since july 1222:10
fungiup to then it was rotating daily22:11
*** signed8bit has quit IRC22:11
asselinrcarrillocruz, this rolename is unexpected:
fungiclarkb: i know you're not here, but should i try to preserve the last several weeks of log-client-debug.log?22:13
clarkbfungi no its not really valuable22:14
fungiokay, i'll blow it away and see if it starts logging anything new and alarming22:14
fungii have a feeling something caused it to fail to rotate briefly and then it got too big to rotate22:14
fungii'm rebooting it now since when / fills up you never really know what else can go sideways22:15
fungiwill try to keep tabs on the logs and make sure they go back to rotating normally and not growing crazily22:16
*** karthikp_ has joined #openstack-infra22:16
*** thorst has quit IRC22:16
*** PollyZ has joined #openstack-infra22:17
anteayafungi: glad you got the wiki username sorted out22:18
anteayafungi: I was offline22:18
*** piet_ has joined #openstack-infra22:18
fungino worries. it's all good. thanks for checking in!22:18
anteayashould he not have deleted those pages?22:18
*** jcoufal_ has quit IRC22:19
fungianteaya: well, not without further discussion at least. for example i was surprised to see my user page suddenly deleted. others may have been as well22:19
anteayafungi: fair enough, did you want them reinstated?22:19
fungii took care of it already22:20
anteayahe was uncertain and had been looking for direciton previously22:20
anteayasorry I didn't read all the backscroll22:20
fungii gave him some tips on how i've been checking for spammers, and how to easily block and clean up after them22:20
fungiit was in privmsg, he wasn't in channel22:21
*** mriedem has quit IRC22:21
anteayaah okay thanks for the status update22:21
anteayaI appreciate it22:21
*** xyang1 has quit IRC22:22
asselinrcarrillocruz, weird...I don't know why I need to use ../.. here: Isn't tox -e linters run from the top-level folder of the repo?22:23
*** xnox has joined #openstack-infra22:24
*** ayoung has joined #openstack-infra22:26
openstackgerritAaron Rosen proposed openstack-dev/cookiecutter: sync latest requirements.txt
anteayamorning jhesketh22:27
*** kzaitsev_mb has quit IRC22:27
*** zul has joined #openstack-infra22:28
jeblairfungi, pabelanger: i'm seeing a lot of errors regarding deleting subnodes from the db.  i'm going to assume something is wrong with that patch, revert it, and restart nodepool22:30
*** furlongm_ has quit IRC22:30
*** furlongm_ has joined #openstack-infra22:31
fungijeblair: 349119? yeah, agreed that seems likely22:31
jeblairthe rate of "Exception deleting subnode" errors is much higher than yesterday22:32
openstackgerritJames E. Blair proposed openstack-infra/nodepool: Revert "Include subnodes in updateStats function"
fungialso the ready count has only fallen to ~200 so far22:32
*** rfolco has quit IRC22:32
openstackgerritMerged openstack-infra/nodepool: Revert "Include subnodes in updateStats function"
rhedlindUpdated nodepool to the latest from an ancient verison. Anyone willing to share an example of a working nodepool secure.conf? This one causes nodepool to throw an exception on the dburi line:
asselinrhedlind, are you using puppet-openstackci?22:34
*** PollyZ has quit IRC22:34
*** dimtruck is now known as zz_dimtruck22:34
jeblairrestarted nodepool with the revert in place22:34
*** pradk has quit IRC22:34
*** PollyZ has joined #openstack-infra22:36
fungirhedlind: in production we use dburi=mysql+pymysql://someuser:somepassword@localhost/nodepool22:37
fungirhedlind: obviously with a different username and password22:37
rhedlindasselin: No, the Ci was installed a while back. I got it back to a working state earlier this week. Everything ran great until zuul was killed due to an out of memory issue. After that zuul refused to restart. I suspect it was because the apscheduler was not the right version.22:37
rhedlindupgrading apscheduler caused nodepool to freak out instead, so I decided to upgrade nodepool as well22:38
rhedlindand now I am hitting the issue with secure.conf22:38
*** edmondsw has quit IRC22:40
fungirhedlind: you're likely running into the (not particularly recent) shift from mysql-python to pymysql22:40
asselinrhedlind, and some database schema changes if I recall22:40
asselinrhedlind, secure.conf is here:
asselinrhedlind, but I suggest you try to downgrade your nodepool back to where it was before22:41
asselinrhedlind, try one of the tags:
fungiyeah, and if you do go with a significant jump in nodepool versions you'll likely just end up having to drop your tables and let nodepoold recreate them. and if you do that you'll likely need to manually clean up old images (and floating ips too if you use them)22:43
*** camunoz has joined #openstack-infra22:43
fungiand probably old server instances if the old version od nodepool you were using predates the automatic alien cleanup metadata on instances22:44
rhedlindremoving the ' ' helped! I followed the instructions on that includes ' '22:44
rhedlindI needed to upgrade nodepool so I can run a newer version of apscheduler22:44
asselinrhedlind, if I remember isn't required and the old way still works...22:45
rhedlindasselin: ok22:45
zarofungi: i've added the force online reindex change to 'gerrit-upgrade' topic.
fungizaro: oh, did they finally agree to that solution?22:47
fungithe upstream change you've linked in the commit message goes to a 404 page22:47
zarosurprisingly yes22:47
rhedlindDo the newer versions of nodepool only support building images using disk-image builder?
*** rbrndt has quit IRC22:47
asselinrhedlind, probably...there was discussion about that in the mailing list. 3rd party ci folks are supposed to pin to known versions.22:48
asselinrhedlind, there's also other changes going on for zuulv3 that you'll want to pin to what's listed here ^^22:49
fungirhedlind: asselin:
zarofungi: oops, just fixed it22:50
fungithanks zaro!22:50
zarothat was a feature and a bug fix in one22:50
anteayazaro: have you tested online force reindex yet?22:51
*** sdague has quit IRC22:52
zaroanteaya: i've tested on my local machine22:52
fungii guess if we approve 351418 then we can upgrade review-dev to the new build and exercise it there too22:52
rhedlindasselin: fungi: thanks for your help! I will look into the links above and see if I can straighten this out.22:52
zaroanteaya: fungi has the right plan22:52
*** kien-ha has joined #openstack-infra22:52
fungizaro: that will make our gerrit maintenances _soooo_ much smoother22:54
fungithanks for hacking out a solution upstream would agree to!22:54
*** kzaitsev_mb has joined #openstack-infra22:54
*** kien-ha has quit IRC22:55
zaronp, i think we all are happier with less downtime.22:55
*** kgiusti has left #openstack-infra22:56
zaronow if we can get that memory issue under control...22:56
*** kien-ha has joined #openstack-infra22:56
*** ddieterly is now known as ddieterly[away]22:58
*** ddieterly[away] has quit IRC22:58
openstackgerritCraige McWhirter proposed openstack-infra/infra-specs: Initial commit of the Pholio spec
anteayafungi: zaro ah wonderful22:59
anteayaless downtime ++22:59
anteayazaro: nice work here22:59
anteayayou are so persistent :)22:59
fungiespecially for the upcoming storyboard-its plugin reindexing, but also for teh pending project rename request we've been putting off because we don't want a 4-hour downtime just yet23:00
*** Teresita-Warrior has quit IRC23:00
*** amitgandhinz has joined #openstack-infra23:00
*** kzaitsev_mb has quit IRC23:01
zaroanteaya: yeah, don't give me a reason to come after you. it won't end.. :)23:02
*** sarob has quit IRC23:02
anteayaha ha ha23:02
anteayaI don't think I ever have23:02
*** pahuang has joined #openstack-infra23:02
anteayanor is it on my todo list23:02
*** asettle has joined #openstack-infra23:03
*** ams__ has quit IRC23:04
*** amitgandhinz has quit IRC23:05
*** ayoung has quit IRC23:06
*** asettle has quit IRC23:07
*** Swami has quit IRC23:09
*** kzaitsev_mb has joined #openstack-infra23:13
*** thorst has joined #openstack-infra23:15
pabelangerjeblair: fungi: thanks, I'll dig into the patch tomorrow to see why it would be causing problems with deleting sub nodes.23:17
*** dingyichen has joined #openstack-infra23:18
*** kzaitsev_mb has quit IRC23:19
*** thorst has quit IRC23:22
*** thiagop has quit IRC23:23
fungithough we do still seem to be running a glut of ready nodes23:27
jeblairyeah just starting to look again23:27
jeblairhowever, none of them have a recent age23:28
jeblairso there's a possibility they are all from the buggy time.23:28
*** markvoelker has joined #openstack-infra23:28
*** amotoki has quit IRC23:28
*** PollyZ has quit IRC23:28
pabelangergrep "deleting subnode" /var/log/nodepool/nodepool.log.2016-08-04_08 | wc -l23:29
pabelangerreturned 471 instances23:29
pabelangerI think our exceptions for deleting subnodes might be something other then our recent patch23:29
jeblairi see 2 nodes that have been ready for < 50 minutes.23:29
jeblairpabelanger: deleting subnodes is normal23:29
jeblairpabelanger: Exceptions deleting subnodes is less normal23:30
*** zz_dimtruck is now known as dimtruck23:30
*** jamielennox is now known as jamielennox|away23:30
jeblairi'm going to assume all ready nodes with an age > 1 hour are done and delete them23:31
pabelangerjeblair: right, our exceptions have been high today, 471+ exceptions even before we landed our patch23:31
pabelangergoing to see if I can find a timeframe it started at23:31
fungijeblair: if they happen to actually be running jobs, zuul will presumably reschedule them anyway, right?23:31
jeblairfungi: yeah23:31
fungiseems like a safe measure in that case23:31
*** hichihara has joined #openstack-infra23:32
jeblairpabelanger: ok.  i will just have a lot more confidence in your result if your grep is for " Exception deleting subnode" not "deleting subnode".23:32
fungii was looking at long-running ready nodes as a signal of similar loss of state changes and there are a handful (~30) running for 3+ hours23:32
pabelangerjeblair: agreed23:32
jeblairbefore the restart, i found ready nodes with an age of ~10 minutes which had started running jobs23:33
jeblairso somehow nodepool was missing state changes23:33
fungiactually, up to 43 now running >3 hours23:33
fungiso i tend to concur23:33
*** mrmartin has quit IRC23:34
fungii suggest that in addition to nodes ready for >1h, nodes in any state >3h are likely showing stale states23:34
pabelangerYa, I'm at a lose to explain why that could be23:34
pabelanger first 8 hours of subnode deleting exceptions23:34
jeblairi spot checked one, and z-l claimed to have sent the zmq message, but nodepool did not log receiving it.  yet nodepool did log other messages from the same host around the same time.23:35
pabelangerway up from previous days23:35
jeblairit doesn't make sense, but with so many database errors, i'm not assuming sense at the moment.23:35
fungipabelanger: jeblair: there was also a network incident in dfw today, which could account for some earlier failures (though not the more recent loss of state transitions)23:36
pabelangerfungi: ack23:37
fungi04:20-11:15 cdt, so 09:20-16:15 utc23:37
*** markvoelker_ has joined #openstack-infra23:37
*** berendt has quit IRC23:38
fungithough that looks more like it would have impacted build tasks rather than delete tasks23:38
jeblairokay, there are only 2 ready nodes now, and they are at 56 minutes23:39
jeblairi'll delete them in 10 mins or so.  we'll see if there are any more leaks23:39
*** markvoelker has quit IRC23:41
*** xarses has quit IRC23:41
*** jamielennox|away is now known as jamielennox23:41
fungishall i delete nodes in a used state >3 hours?23:41
jeblairfungi: ++23:42
fungiwe're up to 45 of them now23:42
*** Jeffrey4l_ has quit IRC23:42
fungiand here they go23:43
fungithey too seemed to be of relatively expected type breakdown and evenly distributed across launchers23:43
jeblairpabelanger: i wonder if len(subnodes) takes a long time doing sql23:44
*** sdake has joined #openstack-infra23:44
pabelangerjeblair: Ya, I was thinking that too23:44
jeblairpabelanger: could probably just use the configuration instead of doing that -- say, this node is type 'devstack-trusty-2-node' and that label has one subnode, so add 2.23:45
*** Jeffrey4l has joined #openstack-infra23:45
*** sdake_ has quit IRC23:46
pabelangerjeblair: sure, let me see how easy that would be23:47
zarofungi: where is that db script we use to update project names?23:48
*** kzaitsev_mb has joined #openstack-infra23:51
pabelangerjeblair: I think we could do: states[key] += 1 + self.config.labels[node.label_name].subnodes23:52
pabelangerseems to be the way other functions calculate subnodes counts23:53
*** sflanigan has joined #openstack-infra23:54
*** zhurong has joined #openstack-infra23:55
*** thorst has joined #openstack-infra23:56
*** weshay has quit IRC23:56
*** piet_ has quit IRC23:56
*** thorst has quit IRC23:57
*** thorst has joined #openstack-infra23:57
*** esberglu has joined #openstack-infra23:59
*** zul has quit IRC23:59

Generated by 2.14.0 by Marius Gedminas - find it at!