Tuesday, 2016-05-03

fungioh, _gerrit_ is downloading it, not our puppet module?00:00
clarkboh! huh00:00
jeblairi believe so, because it's happening in 'gerrit-initial-init'00:01
clarkbya I agree, it is erroring inside of that exec, not within a "native" puppet resource00:01
*** doug-fis_ has joined #openstack-infra00:02
jeblairhrm.  our gerrit source code is declaring a dep on 1.5100:02
*** eil397 has quit IRC00:02
*** yamamoto has joined #openstack-infra00:02
jeblairso i don't understand where bcprov-jdk16-144.jar is coming from00:02
jeblaircodesearch to the rescue00:03
*** doug-fish has quit IRC00:03
jeblair(that's not where i was looking, so clearly i don't understand deps here)00:03
jeblairthat's in *master*00:03
jeblairi think we may have just discovered that our puppet module doesn't work with gerrit master00:04
clarkbwhich isn't surprising, but don't we feed in a specific war that isn't master?00:04
openstackgerritVipin Balachandran proposed openstack/requirements: Add defusedxml to requirements  https://review.openstack.org/31185700:04
jeblairoh, er... that's not in master00:06
*** salv-orlando has joined #openstack-infra00:07
clarkbhttp://git.openstack.org/cgit/openstack-infra/gerrit/tree/gerrit-pgm/src/main/resources/com/google/gerrit/pgm/init/libraries.config?h=openstack/2.11.4 is what I think we should be running00:07
*** jerryz has quit IRC00:07
*** doug-fis_ has quit IRC00:07
jeblairthat's in openstack master00:07
jeblairwhich is ancient00:07
jeblair2016-05-02 23:40:39.698 |   Notice: /Stage[main]/Gerrit/Exec[download:http://tarballs.openstack.org/ci/gerrit/gerrit-v2.]/returns: executed successfully00:07
jeblairwhich is what that module is testing00:07
* clarkb looks to see where that comes form00:07
clarkbfound it00:08
clarkbit is in spec/acceptance/fixtures/default.pp00:08
clarkbjeblair: do you want to push a patch to update ^ or should I?00:08
jeblairclarkb: i agree that seems to be the prob (along with the spec acceptance that checks it)00:08
jeblairclarkb: can you please?00:09
*** yamamoto has quit IRC00:09
clarkbcan do, one sec00:09
*** markvoelker has joined #openstack-infra00:09
*** heyongli_ has quit IRC00:09
*** heyongli_ has joined #openstack-infra00:10
openstackgerritClark Boylan proposed openstack-infra/puppet-gerrit: Use current prod war in testing  https://review.openstack.org/31189400:10
clarkbif ^ works we can recheck the cache timeout change after ^ merges00:11
*** Sukhdev has quit IRC00:11
*** ddieterly has joined #openstack-infra00:11
*** salv-orlando has quit IRC00:14
*** salv-orlando has joined #openstack-infra00:14
*** vtapia_ has quit IRC00:14
*** baoli has joined #openstack-infra00:14
*** salv-orlando has quit IRC00:15
jeblairclarkb: i think there are other version strings in the beaker asserts00:16
*** doug-fish has joined #openstack-infra00:16
*** salv-orlando has joined #openstack-infra00:16
*** baoli has quit IRC00:16
clarkbjeblair: git grep 2.11 only shows that one line00:16
*** piet has quit IRC00:18
jeblairclarkb: git grep 2.8 shows more00:18
clarkboh derp I needed to do that00:18
*** piet has joined #openstack-infra00:18
nibalizeri believe i have cleaned all the infracloud dns records00:18
clarkbI have curled review.o.o to get the string that *should* show up in beaker and will update those two asserts00:19
*** heyongli_ has quit IRC00:20
*** heyongli_ has joined #openstack-infra00:20
*** amitgandhinz has joined #openstack-infra00:20
clarkbhrm it still fails before it gets to the asserts00:21
*** vtapia_ has joined #openstack-infra00:22
clarkbjeblair: http://logs.openstack.org/94/311894/1/check/gate-puppet-gerrit-puppet-beaker-rspec-ubuntu-trusty/39c3444/console.html#_2016-05-03_00_19_07_56500:22
*** amitgandhinz has quit IRC00:22
clarkbalso a 40400:22
*** salv-orlando has quit IRC00:22
openstackgerritClark Boylan proposed openstack-infra/puppet-gerrit: Use current prod war in testing  https://review.openstack.org/31189400:23
clarkbthat will fail but it is more complete00:23
*** salv-orlando has joined #openstack-infra00:25
*** psilvad has quit IRC00:25
clarkbour best bet may be to make that job non voting for now00:26
clarkbguessing this also breaks git review tests thugh00:26
jeblairclarkb: yeah, it's looking like they are deleting old releases00:26
clarkbI will make zuul change for this00:27
*** mylu has quit IRC00:28
jeblairtheir mirrors still have it, but not the master site00:28
*** heyongli_ has quit IRC00:30
*** Sukhdev has joined #openstack-infra00:30
openstackgerritClark Boylan proposed openstack-infra/project-config: Make puppet-gerrit beaker rspec non voting  https://review.openstack.org/31189800:30
*** heyongli_ has joined #openstack-infra00:30
jeblair#status log gerrit configuration update blocked on failing beaker tests due to missing bouncycastle releases; job being made nonvoting in https://review.openstack.org/31189800:37
openstackstatusjeblair: finished logging00:37
*** heyongli_ has quit IRC00:40
*** heyongli_ has joined #openstack-infra00:41
*** sigmavirus24 is now known as sigmavirus24_awa00:45
openstackgerritMerged openstack-infra/project-config: Make puppet-gerrit beaker rspec non voting  https://review.openstack.org/31189800:46
nibalizerthey just pulled the jar or something?00:46
clarkbnibalizer: ya they all 40400:46
clarkbI can get the cache timeout change approved and rechecked once zuul reloads its layout00:47
*** fitoduarte has quit IRC00:47
openstackgerritMerged openstack-infra/puppet-openstackci: Initiate nodepool-builder on third_party_ci.rst  https://review.openstack.org/29308700:48
nibalizerclarkb: i can pull  https://repo1.maven.org/maven2/org/bouncycastle/bcprov-jdk15on/1.51/bcprov-jdk15on-1.51.jar00:49
clarkbnibalizer: ya jeblair said the mirrors still have it, but gerrit downloads from bouncycastle directly00:49
clarkbnibalizer: maybe what we need to do is point to maven nexus instead and make a new war00:49
nibalizergerrit the application?00:50
nibalizerbecause we have repo1.maven in our gerrit module00:50
clarkbnibalizer: yup, it has an "init" command which we use and that is what does the downloads00:50
*** fawadkhaliq has quit IRC00:50
clarkbnibalizer: http://git.openstack.org/cgit/openstack-infra/gerrit/tree/gerrit-pgm/src/main/resources/com/google/gerrit/pgm/init/libraries.config?h=openstack/2.11.4 is the file to update00:50
*** heyongli_ has quit IRC00:50
*** piet has quit IRC00:50
*** heyongli_ has joined #openstack-infra00:51
nibalizerok, do you want me to do that?00:51
clarkbnibalizer: if you are up to it, you can use my https://review.openstack.org/311894 change (well update it first) to test a resulting war too00:53
*** kjorgensen has quit IRC00:56
*** kjorgensen has joined #openstack-infra00:58
openstackgerritMerged openstack-infra/puppet-openstackci: Clarify oscc_file_contents instructions  https://review.openstack.org/28201000:58
nibalizerclarkb: oh man I don't really undertand the gerrit pipeline00:59
nibalizerso we push a tag and that builds it and deposits it on tarballs?00:59
*** heyongli_ has quit IRC01:01
openstackgerritMerged openstack-infra/tripleo-ci: tripleo.sh: deploy missing deps to run rdopkg  https://review.openstack.org/31186801:01
clarkbnibalizer: if you push a patch you should get a testing war that is useable01:01
clarkbnibalizer: then if we merge that we can a non testing war01:01
clarkbno tag required due to upstream determining versions for us mostly01:01
nibalizerso patches in review generate tarballs01:02
nibalizeri was confused how i'd work around that01:02
nibalizerbut i don't have to01:02
*** kjorgensen has quit IRC01:02
clarkbnibalizer: and note the branch you push to, its based onthe release version01:02
*** heyongli_ has joined #openstack-infra01:03
*** salv-orlando has quit IRC01:04
nibalizerclarkb:  https://review.openstack.org/#/c/311903/ eh?01:04
clarkbnibalizer: ya that should make you a war at http://tarballs.openstack.org/ci/test/ which you can then plug into my earlier change (don't forget to update the asserts too) to test01:05
*** baoli has joined #openstack-infra01:07
*** thorst has quit IRC01:08
*** thorst has joined #openstack-infra01:09
*** david-lyle has joined #openstack-infra01:09
*** heyongli_ has quit IRC01:10
clarkbnibalizer: ansible looks stuck01:14
clarkbyou just kill the process?01:14
nibalizeri havent killed anything recently01:15
nibalizercurrently jn line for tacos01:15
Clintkill some tacos01:15
* clarkb kills some ansibles01:15
*** yamahata has quit IRC01:15
*** bpokorny_ has joined #openstack-infra01:16
*** Hal has joined #openstack-infra01:16
*** Hal is now known as Guest7300801:17
nibalizerits abboying that it gets stuck01:17
nibalizermaybe we need to wrap ansible in timeout01:17
*** thorst has quit IRC01:17
clarkbnibalizer: ya I think we should consider that, can be really conservative too. 2 hours would've been enough today for example01:17
clarkblogging ended at 2213UTC01:18
*** bpokorny_ has quit IRC01:18
nibalizerpost tacos i will put up a patch01:18
*** bpokorny_ has joined #openstack-infra01:18
*** bpokorny has quit IRC01:19
*** cody-somerville has joined #openstack-infra01:21
*** cody-somerville has quit IRC01:21
*** cody-somerville has joined #openstack-infra01:21
*** zz_dimtruck is now known as dimtruck01:23
*** gildub has joined #openstack-infra01:26
*** ddieterly is now known as ddieterly[away]01:26
*** nelsnelson has joined #openstack-infra01:27
*** Jeffrey4l has joined #openstack-infra01:28
*** banix has quit IRC01:28
fungiclarkb: nibalizer: pabelanger posted a patch earlier to adjust the localhost connections in ansible in an attempt to avoid something related to that, i think01:28
fungior that might have been strictly to address memory consumption01:29
*** jyuso has joined #openstack-infra01:30
*** EricGonczer_ has joined #openstack-infra01:30
*** jerryz has joined #openstack-infra01:32
openstackgerritDan Prince proposed openstack-infra/tripleo-ci: Add planet blog aggregator feeds to website  https://review.openstack.org/31191001:33
openstackgerritHongbin Lu proposed openstack/requirements: Bump upper-constraints on magnumclient to 2.1.0  https://review.openstack.org/31191101:36
*** baoli has quit IRC01:37
*** EricGonczer_ has quit IRC01:38
*** edmondsw has quit IRC01:42
*** bardia has quit IRC01:42
morganmordred: ++01:44
clarkbI have rechecked and approved the maxAge fix01:44
clarkbwhen that merges I can #status01:44
*** baoli has joined #openstack-infra01:45
*** anteaya has quit IRC01:47
clarkbnibalizer: http://tarballs.openstack.org/ci/test/gerrit-v2. want me to update my change with that war to test or you going to do it?01:48
openstackgerritSpencer Krum proposed openstack-infra/system-config: Run ansible-playbook in timeout  https://review.openstack.org/31191301:49
nibalizerclarkb: i was about to01:53
*** itsuugo has quit IRC01:53
nibalizerbut yea if you want to go for it01:53
clarkbok will do01:53
*** EricGonczer_ has joined #openstack-infra01:56
openstackgerritClark Boylan proposed openstack-infra/puppet-gerrit: Use working war in testing  https://review.openstack.org/31189401:57
clarkbnibalizer: ^01:57
*** mriedem has quit IRC01:57
*** doug-fish has quit IRC01:57
*** doug-fish has joined #openstack-infra01:57
*** doug-fis_ has joined #openstack-infra02:00
*** bpokorny_ has quit IRC02:01
*** doug-fish has quit IRC02:01
*** doug-fis_ has quit IRC02:05
clarkbnibalizer: https://repo1.maven.org/maven2/org/bouncycastle/bcprov-jdk15on/1.51/bcprov-jdk15on-1.51.ja is a 404 too :)02:05
clarkbwow I managed to use the gerrit web editor for the new patch02:07
*** doug-fish has joined #openstack-infra02:07
*** rhallisey has quit IRC02:10
nibalizeri can screw up anything!02:11
openstackgerritMerged openstack-infra/puppet-gerrit: Change default login cache length to 7 days  https://review.openstack.org/31188802:12
*** apoorvad has quit IRC02:12
clarkb#status log https://review.openstack.org/311888 has merged and should update the gerrit config to set cache maxAge to 7days. Please double check the config file is up to date on review.o.o before restarting the service.02:14
*** piet has joined #openstack-infra02:14
*** thorst has joined #openstack-infra02:15
openstackgerritClark Boylan proposed openstack-infra/puppet-gerrit: Use working war in testing  https://review.openstack.org/31189402:16
*** zhurong has joined #openstack-infra02:16
*** Sukhdev has quit IRC02:18
*** _ody_ has joined #openstack-infra02:18
*** _ody has quit IRC02:19
*** _amrith_ is now known as amrith02:19
*** jerryz has quit IRC02:19
*** jerryz has joined #openstack-infra02:19
*** sdake has joined #openstack-infra02:20
*** thorst has quit IRC02:22
*** tphummel has quit IRC02:23
clarkbnibalizer: success!02:23
*** madhu_ak has quit IRC02:24
*** yamamoto has joined #openstack-infra02:24
*** ddieterly[away] has quit IRC02:26
*** aarefiev has quit IRC02:26
*** piet has quit IRC02:27
*** aarefiev has joined #openstack-infra02:28
*** piet has joined #openstack-infra02:28
*** ddieterly has joined #openstack-infra02:31
*** ddieterly is now known as ddieterly[away]02:32
*** fedexo has joined #openstack-infra02:33
*** EricGonczer_ has quit IRC02:34
*** hdd has quit IRC02:37
*** yamamoto has quit IRC02:37
*** piet has quit IRC02:42
*** binbincong has joined #openstack-infra02:50
*** yamamoto has joined #openstack-infra02:50
*** kzaitsev_mb has quit IRC02:57
*** baoli has quit IRC03:01
*** amrith is now known as _amrith_03:04
*** tphummel has joined #openstack-infra03:11
*** bpokorny has joined #openstack-infra03:15
*** bpokorny has quit IRC03:15
*** bpokorny has joined #openstack-infra03:15
*** thorst has joined #openstack-infra03:20
*** yamamoto has quit IRC03:26
*** DevBox has joined #openstack-infra03:27
*** thorst has quit IRC03:28
*** asettle has quit IRC03:29
*** asettle has joined #openstack-infra03:30
*** piet has joined #openstack-infra03:32
*** ddieterly[away] has quit IRC03:33
*** Jeffrey4l has quit IRC03:37
*** rfolco has joined #openstack-infra03:49
*** harlowja_at_home has joined #openstack-infra03:49
*** tlian has quit IRC03:50
*** kzaitsev_mb has joined #openstack-infra03:51
*** kzaitsev_mb has quit IRC03:56
*** piet has quit IRC04:01
*** oanson has joined #openstack-infra04:03
*** tqtran has quit IRC04:07
*** docaedo2 has joined #openstack-infra04:07
openstackgerritMerged openstack-infra/project-config: Do not run release notes for openstack-manuals/liberty  https://review.openstack.org/31170104:08
*** doug-fish has quit IRC04:09
*** asettle has quit IRC04:10
*** asettle has joined #openstack-infra04:10
*** kuromagi has quit IRC04:13
*** Jeffrey4l has joined #openstack-infra04:14
*** asettle has quit IRC04:15
*** Clint has quit IRC04:15
*** Jeffrey4l_ has joined #openstack-infra04:15
*** Guest73008 has quit IRC04:15
*** pleia2 has quit IRC04:16
*** waynr has quit IRC04:16
*** Jeffrey4l__ has joined #openstack-infra04:16
*** pleia2 has joined #openstack-infra04:16
*** rfolco has quit IRC04:16
*** links has joined #openstack-infra04:17
*** Jeffrey4l__ has quit IRC04:17
*** Jeffrey4l__ has joined #openstack-infra04:17
*** kuromagi has joined #openstack-infra04:17
*** kzaitsev_mb has joined #openstack-infra04:17
*** markvoelker has quit IRC04:19
*** Jeffrey4l_ has quit IRC04:20
*** Jeffrey4l has quit IRC04:20
*** waynr has joined #openstack-infra04:21
*** Jeffrey4l__ has quit IRC04:22
*** Clint has joined #openstack-infra04:23
*** yamamoto has joined #openstack-infra04:23
*** yamamoto has quit IRC04:23
*** thorst has joined #openstack-infra04:25
*** thorst has quit IRC04:32
*** psachin has joined #openstack-infra04:34
*** kzaitsev_mb has quit IRC04:39
*** kzaitsev_mb has joined #openstack-infra04:40
*** harlowja_at_home has quit IRC04:41
*** kzaitsev_mb has quit IRC04:46
*** oanson has quit IRC04:47
*** asselin has joined #openstack-infra04:48
openstackgerritSergey Vilgelm proposed openstack-dev/pbr: Parse requrements files in extras  https://review.openstack.org/31192904:50
*** asselin__ has quit IRC04:50
*** sree has joined #openstack-infra04:56
*** tphummel has quit IRC04:57
*** kzaitsev_mb has joined #openstack-infra05:01
openstackgerritSergey Vilgelm proposed openstack-dev/pbr: Parse requrements files in extras  https://review.openstack.org/31192905:01
*** abregman has joined #openstack-infra05:02
*** bpokorny has quit IRC05:04
openstackgerritSergey Vilgelm proposed openstack-dev/pbr: Parse requrements files in extras  https://review.openstack.org/31192905:06
*** kzaitsev_mb has quit IRC05:07
openstackgerritSergey Vilgelm proposed openstack-dev/pbr: Parse requrements files in extras  https://review.openstack.org/31192905:08
*** kzaitsev_mb has joined #openstack-infra05:09
*** asettle has joined #openstack-infra05:15
*** Jeffrey4l__ has joined #openstack-infra05:18
*** markvoelker has joined #openstack-infra05:19
*** DevBox has quit IRC05:20
*** yamamoto has joined #openstack-infra05:24
*** markvoelker has quit IRC05:24
*** Jeffrey4l__ has quit IRC05:24
*** thorst has joined #openstack-infra05:30
openstackgerritSergey Vilgelm proposed openstack-dev/pbr: Parse requrements files in extras  https://review.openstack.org/31192905:35
*** sdake has quit IRC05:37
*** thorst has quit IRC05:37
*** sdake has joined #openstack-infra05:38
*** yamamoto has quit IRC05:40
openstackgerritfumihiko kakuma proposed openstack-infra/devstack-gate: Use local.conf instead of localrc in devstack configuration  https://review.openstack.org/31166805:40
*** kzaitsev_mb has quit IRC05:43
*** yolanda has joined #openstack-infra05:44
*** DevBox has joined #openstack-infra05:44
*** yolanda has quit IRC05:48
*** yolanda has joined #openstack-infra05:48
openstackgerritfumihiko kakuma proposed openstack-infra/devstack-gate: Use local.conf instead of localrc in devstack configuration  https://review.openstack.org/31166805:52
*** kzaitsev_mb has joined #openstack-infra05:55
openstackgerritMerged openstack-infra/project-config: Add magnum plugin to keystone2 job in shade  https://review.openstack.org/31161005:55
*** asettle has quit IRC05:58
*** xarses has quit IRC05:59
*** xarses has joined #openstack-infra06:04
*** yamamoto has joined #openstack-infra06:08
*** salv-orlando has joined #openstack-infra06:11
*** kzaitsev_mb has quit IRC06:15
*** rcernin has joined #openstack-infra06:15
*** kzaitsev_mb has joined #openstack-infra06:17
*** kushal has quit IRC06:18
*** sree has quit IRC06:19
*** sree has joined #openstack-infra06:19
*** markvoelker has joined #openstack-infra06:20
tonybA whole bunch of the stable-periodic pipeline just failed with "UNSTABLE" jobs.06:21
tonybis there an infra-root around that canpoke at that?06:22
tonybeg http://lists.openstack.org/pipermail/openstack-stable-maint/2016-May/004275.html06:23
*** yamamoto has quit IRC06:23
*** sree has quit IRC06:23
*** markvoelker has quit IRC06:25
*** yolanda has quit IRC06:27
*** yolanda has joined #openstack-infra06:27
*** gibi has quit IRC06:33
*** Adri2000 has quit IRC06:33
*** gibi has joined #openstack-infra06:34
*** yolanda has quit IRC06:34
*** thorst has joined #openstack-infra06:35
openstackgerritOpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints  https://review.openstack.org/30927306:36
*** kzaitsev_mb has quit IRC06:37
*** ccamacho has joined #openstack-infra06:38
jheskethtonyb: looks like a bunch of jobs are unstable06:39
jheskethlet me poke06:39
tonybjhesketh: thanks.06:39
tonybjhesketh: double thanks as you're not s'posed to be around ATM06:39
* jhesketh makes no promises06:40
*** yolanda has joined #openstack-infra06:40
*** kzaitsev_mb has joined #openstack-infra06:41
*** Adri2000 has joined #openstack-infra06:41
yolanda#status alert Filesystem on logs.openstack.org is broken, we are on the process of repairing it. Please stop checking your jobs until further notice06:42
openstackstatusyolanda: sending alert06:42
*** thorst has quit IRC06:42
*** nbloom has joined #openstack-infra06:42
*** tesseract has joined #openstack-infra06:44
-openstackstatus- NOTICE: Filesystem on logs.openstack.org is broken, we are on the process of repairing it. Please stop checking your jobs until further notice06:44
*** ChanServ changes topic to "Filesystem on logs.openstack.org is broken, we are on the process of repairing it. Please stop checking your jobs until further notice"06:44
*** tesseract is now known as Guest4047906:44
*** daemontool_ has quit IRC06:45
openstackstatusyolanda: finished sending alert06:47
jheskethyolanda: oh, looks like you're working on it06:47
jheskethyolanda: did you unmount it?06:47
* jhesketh was just trying to figure out why it didn't remount readonly06:47
*** Guest40479 has quit IRC06:47
jheskethokay, I'll leave you to it06:47
jheskethI'm around if you need a hand though06:48
yolandai'm running fsck now06:48
jheskethtonyb: ^06:48
*** gibi has quit IRC06:48
jheskethyolanda: don't forget you'll need to do some log shuffling when done as they are currently being written to the static partition06:48
tonybjhesketh, yolanda: Thanks!06:48
yolandajhesketh, care reminding me the rsync command you used last time?06:48
*** tesseract- has joined #openstack-infra06:49
jheskethsigh, it's not in my bash history06:49
*** gibi has joined #openstack-infra06:49
yolandai'll figure...06:50
*** sree has joined #openstack-infra06:52
jheskethyolanda: something like: rsync -vv -aHAXPh --ignore-existing --dry-run backup.logs/ logs/06:52
jheskethcheck the dry-run first06:53
*** gibi has quit IRC06:54
*** gibi has joined #openstack-infra06:55
*** DevBox has quit IRC06:57
*** kzaitsev_mb has quit IRC06:58
*** fedexo has quit IRC06:59
*** salv-orlando has quit IRC07:04
*** arxcruz has joined #openstack-infra07:04
*** rcernin_ has joined #openstack-infra07:04
openstackgerrityolanda.robla proposed openstack-infra/shade: Add baymodel list, get and search call for shade+magnum  https://review.openstack.org/30800907:09
*** dimtruck is now known as zz_dimtruck07:10
*** jed56 has joined #openstack-infra07:11
openstackgerritSamuel Gauthier proposed openstack-infra/project-config: add networking-vpp project to OpenStack  https://review.openstack.org/31194107:12
openstackgerrityolanda.robla proposed openstack-infra/shade: Add initial setup for magnum in shade  https://review.openstack.org/24323907:14
*** haypo has joined #openstack-infra07:15
*** yaume has joined #openstack-infra07:16
*** kaisers1 has joined #openstack-infra07:17
*** markvoelker has joined #openstack-infra07:21
*** zeih has joined #openstack-infra07:22
*** markvoelker has quit IRC07:26
*** zz_dimtruck is now known as dimtruck07:26
haypohi. what's the issue with check jobs logs? https://review.openstack.org/#/c/311612/ was checked 1 day ago, but http://logs.openstack.org/12/311612/1/check/gate-keystone-python27-db/a35964b/ says "File Not Found"07:26
*** nadya has joined #openstack-infra07:27
haypoone or two weeks ago, i read there that it sometimes takes several hours to copy logs. is it still the case?07:27
fricklerhaypo: see the topic ;)07:27
haypofrickler: hum, i see :)07:29
*** asettle has joined #openstack-infra07:30
*** nmagnezi has joined #openstack-infra07:31
*** vincentll has joined #openstack-infra07:32
*** rcernin_ has quit IRC07:33
*** dimtruck is now known as zz_dimtruck07:35
*** rcernin has quit IRC07:36
*** pfallenop has quit IRC07:38
*** nadya has quit IRC07:39
*** thorst has joined #openstack-infra07:40
*** binbincong has quit IRC07:40
*** pfallenop has joined #openstack-infra07:45
*** thorst has quit IRC07:47
*** ggnel_t has joined #openstack-infra07:47
*** ggnel_t2 has joined #openstack-infra07:47
*** chlong has quit IRC07:47
*** ggnel_t has quit IRC07:47
*** ggnel_t2 has quit IRC07:48
*** ggnel_t has joined #openstack-infra07:48
*** shardy has joined #openstack-infra07:50
openstackgerritBalazs Gibizer proposed openstack-infra/irc-meetings: Increasing the frequency of notification meeting  https://review.openstack.org/31194807:52
*** ggnel_t has quit IRC07:53
*** ggnel_t has joined #openstack-infra07:53
*** yuanying has joined #openstack-infra07:55
*** sdake has quit IRC07:56
*** binbincong has joined #openstack-infra07:56
*** spiffxp has joined #openstack-infra07:56
*** maximov_ has joined #openstack-infra07:58
*** larainema_ has joined #openstack-infra07:58
*** scottda has quit IRC07:58
*** yonglihe has quit IRC07:58
*** dham1 has quit IRC07:58
*** fkautz has quit IRC07:58
*** DuncanT has quit IRC07:58
*** boris-42 has quit IRC07:58
*** rguillebert has quit IRC07:58
*** sweston has quit IRC07:58
*** zhiyan has quit IRC07:58
*** jed56 has quit IRC07:58
*** cargonza has quit IRC07:58
*** jaranovich has quit IRC07:58
*** hrybacki has quit IRC07:58
*** mwhahaha has quit IRC07:58
*** NobodyCam has quit IRC07:58
*** asettle has quit IRC07:58
*** docaedo2 has quit IRC07:58
*** jyuso has quit IRC07:58
*** larainema has quit IRC07:58
*** GheRivero has quit IRC07:58
*** vdrok has quit IRC07:58
*** msuriar has quit IRC07:58
*** maximov has quit IRC07:58
*** spiffxp_ has quit IRC07:58
*** bryan_att has quit IRC07:58
*** rackertom has quit IRC07:58
*** timl0101 has quit IRC07:58
*** woodster_ has quit IRC07:58
*** zzzeek has quit IRC07:58
*** hughsaunders has quit IRC07:58
*** zxiiro has quit IRC07:58
*** scottda has joined #openstack-infra07:58
*** sulo has quit IRC07:58
*** vdrok_ has joined #openstack-infra07:58
*** larainema_ is now known as larainema07:58
*** zhiyan_ has joined #openstack-infra07:58
*** NobodyCam_ has joined #openstack-infra07:58
*** cargonza has joined #openstack-infra07:58
*** dham1 has joined #openstack-infra07:58
*** vdrok_ has quit IRC07:58
*** vdrok_ has joined #openstack-infra07:58
*** scottda has quit IRC07:58
*** scottda has joined #openstack-infra07:58
*** dham1 has quit IRC07:58
*** dham1 has joined #openstack-infra07:58
*** cargonza has quit IRC07:58
*** cargonza has joined #openstack-infra07:58
*** asettle has joined #openstack-infra07:58
*** yolanda has quit IRC07:58
*** Guest78717 has quit IRC07:58
*** mmedvede has quit IRC07:58
*** dstanek has quit IRC07:58
*** dtroyer has quit IRC07:58
*** ayoung has quit IRC07:58
*** pblaho has quit IRC07:58
*** d0ugal has quit IRC07:58
*** anthonyper has quit IRC07:58
*** gus has quit IRC07:58
*** nt has quit IRC07:58
*** docaedo has quit IRC07:58
*** auggy has quit IRC07:58
*** yuanying_ has quit IRC07:58
*** lbragstad has quit IRC07:58
*** jlk has quit IRC07:58
*** mrodden has quit IRC07:58
*** jyuso has joined #openstack-infra07:58
*** sulo has joined #openstack-infra07:58
*** maximov_ is now known as maximov07:58
*** pblaho has joined #openstack-infra07:59
*** msuriar has joined #openstack-infra07:59
*** d0ugal has joined #openstack-infra07:59
*** DuncanT_ has joined #openstack-infra07:59
*** BobBall has joined #openstack-infra07:59
*** jistr has joined #openstack-infra07:59
*** zxiiro has joined #openstack-infra07:59
*** dstanek has joined #openstack-infra07:59
*** auggy has joined #openstack-infra07:59
*** sweston has joined #openstack-infra07:59
*** BobBall is now known as Guest8257107:59
*** mrodden has joined #openstack-infra07:59
*** lbragstad has joined #openstack-infra07:59
*** nt has joined #openstack-infra07:59
*** jlk has joined #openstack-infra07:59
*** hrybacki has joined #openstack-infra07:59
*** anthonyper has joined #openstack-infra07:59
*** ayoung has joined #openstack-infra08:00
*** jlk has quit IRC08:00
*** jlk has joined #openstack-infra08:00
*** docaedo has joined #openstack-infra08:00
*** bryan_att has joined #openstack-infra08:00
*** zhiyan_ is now known as zhiyan08:00
*** dtroyer has joined #openstack-infra08:00
*** fkautz has joined #openstack-infra08:00
*** yonglihe has joined #openstack-infra08:00
*** hughsaunders has joined #openstack-infra08:00
*** zzzeek has joined #openstack-infra08:00
*** chlong has joined #openstack-infra08:01
*** DuncanT_ is now known as DuncanT08:02
*** gus has joined #openstack-infra08:02
*** rguillebert has joined #openstack-infra08:02
*** matrohon has joined #openstack-infra08:03
*** yolanda has joined #openstack-infra08:03
*** mwhahaha has joined #openstack-infra08:04
*** woodster_ has joined #openstack-infra08:05
yolandafsck finished, going to move logs and mount again08:05
*** jlanoux has joined #openstack-infra08:05
openstackgerritSamuel Gauthier proposed openstack-infra/project-config: add networking-vpp project to OpenStack  https://review.openstack.org/31194108:05
*** pilgrimstack has joined #openstack-infra08:05
openstackgerritRob Cresswell proposed openstack/requirements: Add horizon-cisco-ui to projects.txt  https://review.openstack.org/30006208:06
*** rackertom has joined #openstack-infra08:08
*** jerryz has quit IRC08:08
*** timl0101 has joined #openstack-infra08:09
*** boris-42 has joined #openstack-infra08:10
*** GheRivero has joined #openstack-infra08:10
*** jaranovich has joined #openstack-infra08:11
*** mmedvede has joined #openstack-infra08:11
*** jed56 has joined #openstack-infra08:11
matrohonhi all,08:12
matrohonIs there something broken on static.openstack.org?08:13
matrohonI'm having some merge failed on doc jobs : http://logs.openstack.org/29/311729/1/gate/gate-networking-bgpvpn-docs/1b11566/console.html#_2016-05-03_07_59_59_52308:13
*** sshnaidm has quit IRC08:14
*** jogo has quit IRC08:16
*** fbo has joined #openstack-infra08:17
*** ifarkas has joined #openstack-infra08:17
matrohonAJaeger, hi08:19
matrohonAJaeger, ^08:19
*** lipeng has joined #openstack-infra08:20
odyssey4mematrohon you may notice the topic of this channel indicates that there are issues in general with the logs - all other channels also got the notice08:21
matrohonodyssey4me, thanks, I didn't look at the topic!08:22
lipengHi, when I sign 'OpenStack Project Individual Contributor License Agreement' on web pages and commit, Server error occured.08:22
yolanda#status ok Logs filesystem has been successfully restored, please recheck your jobs08:22
openstackstatusyolanda: sending ok08:22
lipengCode Review - Error08:22
lipengServer Error08:22
lipengCannot store contact information08:22
lipengCan anyone help me?08:22
*** markvoelker has joined #openstack-infra08:22
*** chem has joined #openstack-infra08:22
AJaegerlipeng: If you already followed the instructions at  http://docs.openstack.org/infra/manual/developers.html#account-setup and still get it, see https://ask.openstack.org/question/56720 for additional tips.08:24
*** ChanServ changes topic to "Discussion of OpenStack Developer and Community Infrastructure | docs http://docs.openstack.org/infra/ | bugs https://storyboard.openstack.org/ | source https://git.openstack.org/cgit/openstack-infra/ | channel logs http://eavesdrop.openstack.org/irclogs/%23openstack-infra/"08:24
-openstackstatus- NOTICE: Logs filesystem has been successfully restored, please recheck your jobs08:24
haypo"NOTICE: Logs filesystem has been successfully restored, please recheck your jobs" good job, thanks ;)08:24
*** jogo has joined #openstack-infra08:25
yolandahi, so logs filesystem is ok again08:25
odyssey4methanks yolanda :)08:26
AJaegermatrohon: see yolanda's message - she repaired the file system. recheck your change...08:26
AJaegeryolanda: Thanks!08:26
yolandaglad to help08:26
openstackgerritJulien Danjou proposed openstack/requirements: Remove Ceilometer from projects  https://review.openstack.org/31198908:26
matrohonAJaeger, yolanda : thanks!08:26
*** markvoelker has quit IRC08:27
jd__thank you infra!08:27
openstackstatusyolanda: finished sending ok08:27
*** keedya has quit IRC08:28
*** hashar has joined #openstack-infra08:29
*** |-paul-| has joined #openstack-infra08:30
odyssey4mehmm, I'm still getting 'UNSTABLE' results from jobs08:30
odyssey4mean example: https://review.openstack.org/31196508:31
yolandaoh, docs-draft crashed as well08:31
yolandai fixed logs08:31
yolandaso i'll check other volumes and repair again08:31
odyssey4meah ok - that makes sense :/08:32
yolandadisk fun today!08:33
yolanda#status notice Filesystem on docs-draft.openstack.org is broken, we are on the process of repairing it. Please stop checking jobs using this filesystem until further notice08:33
openstackstatusyolanda: sending notice08:33
*** kushal has joined #openstack-infra08:34
*** itsuugo has joined #openstack-infra08:34
-openstackstatus- NOTICE: Filesystem on docs-draft.openstack.org is broken, we are on the process of repairing it. Please stop checking jobs using this filesystem until further notice08:35
lennybHi Infra, I faced a strange zuul deadlock #link http://paste.openstack.org/show/495932/  caused by patchset 12 of #link https://review.openstack.org/#/c/30927308:35
*** dmk0202 has joined #openstack-infra08:35
openstackstatusyolanda: finished sending notice08:36
*** dingyichen has quit IRC08:37
sc68calIs infra using storyboard? we have a bug that probably needs some infra help, that was talked about last week- but I want to make sure it doesn't get lost in the shuffle - https://bugs.launchpad.net/devstack/+bug/157058408:38
openstackLaunchpad bug 1570584 in devstack "Docs page broken - running off page" [Undecided,New]08:38
*** derekh has joined #openstack-infra08:39
openstackgerritMerged openstack-infra/irc-meetings: Increasing the frequency of notification meeting  https://review.openstack.org/31194808:40
*** thorst has joined #openstack-infra08:45
yolandaAJaeger, i'll need to revert the shade+magnum again, it's totally weird. It passed several of my non-voting jobs, but now since i merged, it fails on a neutron test :(08:45
yolandathere must be some side effect08:45
openstackgerrityolanda.robla proposed openstack-infra/project-config: Revert "Add magnum plugin to keystone2 job in shade"  https://review.openstack.org/31199208:45
*** jang has joined #openstack-infra08:45
*** haypo has left #openstack-infra08:46
AJaegeryolanda: what fun ;(08:48
yolandayep, going to move to non-voting again, and debug, not sure if that's something new, because the non-voting and voting jobs were identical08:48
yolandaand the failing one is not even a magnum test, is a neutron test08:49
*** thorst has quit IRC08:52
*** e0ne has joined #openstack-infra08:52
odyssey4meit would appear that docs-draft is working again yolanda08:54
odyssey4mewell, tests are passing08:55
*** sileht has quit IRC09:01
*** sileht has joined #openstack-infra09:02
*** electrofelix has joined #openstack-infra09:02
*** dizquierdo has joined #openstack-infra09:03
*** dizquierdo has quit IRC09:04
*** sshnaidm has joined #openstack-infra09:04
*** vgridnev has quit IRC09:05
openstackgerritMerged openstack-infra/project-config: Revert "Add magnum plugin to keystone2 job in shade"  https://review.openstack.org/31199209:05
*** asettle has quit IRC09:05
*** asettle has joined #openstack-infra09:06
*** dizquierdo has joined #openstack-infra09:06
*** markmcd__ has quit IRC09:09
*** markmcd has quit IRC09:09
*** markmcd has joined #openstack-infra09:10
*** markmcd1 has joined #openstack-infra09:11
*** asettle has quit IRC09:11
yolandaodyssey4me, it's in the process of being restored, and while the volume is unmounted, the docs are placed on another volume09:12
*** zz_dimtruck is now known as dimtruck09:12
odyssey4meyolanda ah ok... restored? wow, much brokenness :/09:13
*** skolekonov has joined #openstack-infra09:13
*** skolekonov has quit IRC09:13
lipengAJaeger: I followed your advice and now it is OK, thanks!09:14
*** dizquierdo has quit IRC09:14
*** dizquierdo has joined #openstack-infra09:15
yolanda#status ok Docs-draft filesystem has been restored. Please check your affected jobs again09:19
openstackstatusyolanda: sending ok09:19
openstackgerritSean M. Collins proposed openstack-infra/project-config: Make grenade-dsvm-neutron-multinode voting  https://review.openstack.org/29984309:20
*** oanson has joined #openstack-infra09:20
*** e0ne has quit IRC09:21
*** zeih has quit IRC09:21
-openstackstatus- NOTICE: Docs-draft filesystem has been restored. Please check your affected jobs again09:21
*** dimtruck is now known as zz_dimtruck09:22
*** markvoelker has joined #openstack-infra09:23
*** sheeprine_ is now known as sheeprine09:23
*** sheeprine has joined #openstack-infra09:23
*** jordanP has joined #openstack-infra09:24
*** dizquierdo has quit IRC09:24
openstackstatusyolanda: finished sending ok09:25
*** dizquierdo has joined #openstack-infra09:25
*** oanson has quit IRC09:27
*** markvoelker has quit IRC09:28
*** asettle has joined #openstack-infra09:28
*** dizquierdo has quit IRC09:32
*** sambetts|afk is now known as sambetts09:32
*** dizquierdo has joined #openstack-infra09:34
*** asettle has quit IRC09:39
*** gildub has quit IRC09:41
*** lipeng has quit IRC09:41
*** dizquierdo has quit IRC09:43
*** tosky has joined #openstack-infra09:46
*** ccamacho has left #openstack-infra09:47
*** links has quit IRC09:48
*** thorst has joined #openstack-infra09:50
*** dizquierdo has joined #openstack-infra09:51
*** thorst has quit IRC09:57
frickleryolanda: I'm still seeing logging errors after your o.k. message, like http://logs.openstack.org/18/307718/2/check/gate-nova-docs/301bfd9/console.html , if you could have another look at that?09:57
fricklerah, that is static.openstack.org instead of logs, probably needs fixing, too?09:58
yolandafrickler, i fixed logs and docs-draft. Both should be working now, i tested creating files inside docs-draft and works for me now09:59
yolandacan you recheck again ?10:00
frickleryolanda: can you check static.openstack.org, too, please?10:00
*** jlanoux has quit IRC10:00
*** jlanoux has joined #openstack-infra10:00
yolandafrickler, everything is static.openstack.org10:00
yolandadocs-draft is just a volume on that server10:01
frickleryolanda: hmm, o.k., so I'll just try another recheck, thx10:01
yolandaplease try and let me know10:01
*** rossella_s has joined #openstack-infra10:01
*** kengo_sakai has joined #openstack-infra10:02
*** e0ne has joined #openstack-infra10:02
*** links has joined #openstack-infra10:04
*** links has quit IRC10:07
*** zhurong has quit IRC10:08
frickleryolanda: looking better now, will take another hour though for all of the jobs to finish10:11
yolandafrickler, cool that it works10:11
*** kushal has quit IRC10:13
*** zeih has joined #openstack-infra10:13
*** _amrith_ is now known as amrith10:14
*** links has joined #openstack-infra10:14
*** Guest82571 is now known as BobBall10:23
*** cody-somerville has quit IRC10:23
*** markvoelker has joined #openstack-infra10:24
*** sdague has joined #openstack-infra10:25
*** hieulq has joined #openstack-infra10:27
*** kushal has joined #openstack-infra10:27
*** markvoelker has quit IRC10:28
*** hieulq has quit IRC10:29
*** hieulq has joined #openstack-infra10:29
*** jlanoux_ has joined #openstack-infra10:30
*** rhallisey has joined #openstack-infra10:30
*** jlanoux has quit IRC10:30
*** mikelk has joined #openstack-infra10:31
*** jlanoux has joined #openstack-infra10:32
*** jlanoux_ has quit IRC10:33
*** Kennan has quit IRC10:35
*** hieulq has quit IRC10:35
*** hashar has quit IRC10:35
*** Kennan has joined #openstack-infra10:35
*** hieulq has joined #openstack-infra10:35
*** kushal has quit IRC10:39
*** rossella_s has quit IRC10:39
openstackgerritJianghua Wang proposed openstack-infra/nodepool: Add namespace for nodepool  https://review.openstack.org/31069810:39
*** hieulq has quit IRC10:40
*** jlanoux_ has joined #openstack-infra10:43
*** zeih_ has joined #openstack-infra10:45
*** zeih has quit IRC10:45
*** jlanoux has quit IRC10:46
*** markus_z has joined #openstack-infra10:51
*** pfallenop has quit IRC10:52
*** thorst has joined #openstack-infra10:55
*** Kennan has quit IRC10:58
*** lucasagomes is now known as lucas-hungry10:59
*** thorst has quit IRC11:02
*** amrith is now known as _amrith_11:02
odyssey4mehas anyone else been seeing tox errors for 'baseenv=python2' on a mac ?11:02
odyssey4mesorry, that's actually 'basepython = python2'11:03
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: Update setuptools in Centos for DLRN  https://review.openstack.org/31201611:03
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: Update setuptools in Centos for DLRN  https://review.openstack.org/31201611:06
*** chem has quit IRC11:08
*** kengo_sakai has quit IRC11:09
*** zz_dimtruck is now known as dimtruck11:13
*** Kennan has joined #openstack-infra11:14
*** zeih_ has quit IRC11:17
*** rossella_s has joined #openstack-infra11:18
*** zeih has joined #openstack-infra11:18
*** yolanda has quit IRC11:22
*** dimtruck is now known as zz_dimtruck11:23
*** links has quit IRC11:24
*** markvoelker has joined #openstack-infra11:24
*** yolanda has joined #openstack-infra11:27
*** hashar has joined #openstack-infra11:27
*** markvoelker has quit IRC11:29
*** gordc has joined #openstack-infra11:29
*** kushal has joined #openstack-infra11:31
*** banix has joined #openstack-infra11:34
*** banix has quit IRC11:36
*** Hal has joined #openstack-infra11:36
*** Hal is now known as Guest5957011:37
sdaguehow do I find a shortname for a 3rd party ci in jenkins?11:38
*** rossella_s has quit IRC11:41
*** thorst has joined #openstack-infra11:43
*** rossella_s has joined #openstack-infra11:44
*** Guest59570 has quit IRC11:45
*** dizquierdo has quit IRC11:48
*** ccamacho has joined #openstack-infra11:49
*** kengo_sakai has joined #openstack-infra11:52
*** dprince has joined #openstack-infra11:53
*** lucas-hungry is now known as lucasagomes11:54
*** weshay has joined #openstack-infra11:55
*** zz_ja has quit IRC11:55
*** zeih has quit IRC11:56
*** zeih has joined #openstack-infra11:57
*** baoli has joined #openstack-infra11:57
*** tpsilva has joined #openstack-infra11:57
*** rfolco has joined #openstack-infra12:00
*** kengo_sakai has quit IRC12:01
*** weshay is now known as weshay_mtg12:02
*** ldnunes has joined #openstack-infra12:02
*** links has joined #openstack-infra12:03
*** psilvad has joined #openstack-infra12:04
*** zz_ja has joined #openstack-infra12:04
*** trown|outtypewww is now known as trown12:06
*** kgiusti has joined #openstack-infra12:08
*** yaume has quit IRC12:09
*** yaume has joined #openstack-infra12:09
*** psilvad has quit IRC12:09
*** zz_ja has quit IRC12:13
*** sshnaidm has quit IRC12:14
*** aysyd has joined #openstack-infra12:15
openstackgerrityolanda.robla proposed openstack-infra/project-config: Revert "Revert "Add magnum plugin to keystone2 job in shade""  https://review.openstack.org/31203212:16
*** krtaylor has quit IRC12:18
nbloomHi, I update our(Vitrage) post_test_hook.sh and reran an experimental job that calls it though it doesn't seem to have been updated.. does the update takes time?12:20
*** krtaylor has joined #openstack-infra12:20
*** markvoelker has joined #openstack-infra12:22
toabctlI'm trying to get a new requirement (called "spec_cleaner") into the global requirements. but the test run fails with "No matching distribution found for spec-cleaner>=0.8.2" . on pypi is 0.8.2 available. how often are the pypi mirrors updated?12:22
toabctlthe logs are here: http://logs.openstack.org/68/311468/3/check/gate-requirements-integration-dsvm/4e44e1d/console.html12:22
*** burgerk has joined #openstack-infra12:23
*** zz_ja has joined #openstack-infra12:24
AJaegertoabctl: we had some problems with our pypi mirroring and some have been working last night on it. I haven't checked backscroll from last 18 hours whether it's solved - but fear it takes a little bit longer.12:25
openstackgerritMerged openstack-infra/tripleo-ci: Finished: Update deprecated commands in tripleo.sh (Old heat * calls)  https://review.openstack.org/30903912:25
*** nadya has joined #openstack-infra12:25
*** sshnaidm has joined #openstack-infra12:25
toabctlAJaeger: ok. then I'll way a bit and retrigger the test later. thx12:25
AJaegertoabctl: Normally it should be available within an hour12:25
openstackgerritMerged openstack-infra/puppet-gerrit: Use working war in testing  https://review.openstack.org/31189412:25
AJaegertoabctl: or ask before you leave the office today whether pypi mirror is up to date again - and then recheck12:25
*** nadya has quit IRC12:26
*** nadya has joined #openstack-infra12:27
*** amotoki has joined #openstack-infra12:27
*** amotoki has quit IRC12:27
nbloomnevermind, found my mistake12:27
*** vgridnev has joined #openstack-infra12:28
*** EricGonczer_ has joined #openstack-infra12:29
*** jheroux has joined #openstack-infra12:32
*** _prad_ has joined #openstack-infra12:33
*** _prad_ is now known as pradk_12:33
openstackgerritvinay kumar muddu proposed openstack/diskimage-builder: Included "sudo" for devuser element  https://review.openstack.org/30874712:35
*** julim has joined #openstack-infra12:36
pabelangerAJaeger: do you mind +A https://review.openstack.org/#/c/311721. I've confirmed with tripleo PTL they are good with starting up a new node on tripleo-test-cloud-rh112:36
*** kzaitsev_mb has joined #openstack-infra12:39
*** Hal has joined #openstack-infra12:42
*** Hal is now known as Guest2064312:42
*** javeriak has joined #openstack-infra12:44
*** nadya has quit IRC12:44
electrofelixfungi: hoping you have a few mins to iron how you'd like to handle JJB releases12:47
openstackgerritEmilien Macchi proposed openstack-infra/project-config: Implement periodic job to update Puppet OpenStack constraints  https://review.openstack.org/30137512:47
*** EricGonczer_ has quit IRC12:47
openstackgerritPaul Belanger proposed openstack-infra/system-config: Wrap ansible-playbook in a 30min timeout  https://review.openstack.org/31204512:48
pabelangerfungi: nibalizer: clarkb: mordred: not pretty but fixes us having to manually kill ansible-playbook ever few hours. ^12:49
fungielectrofelix: what commit do you want tagged as 1.5.0? the master branch tip or something older?12:49
pabelangercurrently it is OOM on the final step, almost every time12:49
javeriakfungi, clarkb, jeblair, pleia2 ; i tried creating a branch on my openstack repo using https://wiki.openstack.org/wiki/Fuel/Plugins#Creating_branches ; but i ended up creating a topic one of of master; and now it wont allow backport cherrypicks or any commit; can i please get the branch deleted, and some guidance on how to really create a new one off of master12:49
AJaegerpabelanger: ok, let me review...12:50
pabelangerfungi: if you also have a moment for https://review.openstack.org/#/c/311754/ that will help EmilienM generate tarballs12:50
fungistill waking up, and keeping an eye out for the impending openssl advisory before i know how today's shaping up, but i'll take a look shortly12:51
AJaegerjaveriak: please read http://docs.openstack.org/infra/manual/drivers.html for official documentation12:51
fungii don't think branches need anything particularly special to be able to propose backports to them12:52
javeriakAJaeger thanks; do i still need someone from the infra team to delete the topic one i have currently?12:53
*** piet has joined #openstack-infra12:53
fungijaveriak: not sure what you mean by "topic branch" in this context. can you elaborate on how you created it?12:53
*** xyang1 has joined #openstack-infra12:53
AJaegerjaveriak: we're confused, do you have a link to show us what you did?12:53
javeriakfungi sure; i followed the 'from the web UI' steps here https://wiki.openstack.org/wiki/Fuel/Plugins#Creating_branches12:54
*** Goneri has joined #openstack-infra12:54
AJaegerjaveriak: which repository?12:54
javeriakthis one https://github.com/openstack/fuel-plugin-plumgrid12:54
pabelangerfungi: good to know12:54
*** psilvad has joined #openstack-infra12:55
AJaegerand which "topic branch" is the problem?12:55
javeriakAJaeger the commit i pushed, ended up going into master; the branch is stable/7.012:55
fungijaveriak: that looks like the same way i would have created a branch12:55
AJaegerjaveriak: So, you by accident merged a change into master instead of branch? Do you have a reference to it?12:56
*** psilvad has quit IRC12:56
fungijaveriak: it sounds more like you're expecting git-review to know to propose changes against that branch without telling it to do so?12:56
*** psilvad has joined #openstack-infra12:56
fungibranch is a command-line parameter to git-review, but it can also be inferred from the defaultbranch setting in .gitreview if you want to update it on that branch so you can avoid specifying a branch name every time12:56
javeriakfungi; the commit is here: https://review.openstack.org/#/c/311209/ ; it wouldnt allow a cherrpick and push to the stable/7.0 so i did a new commit; but that ended up getting merged into master12:57
*** kengo_sakai has joined #openstack-infra12:57
AJaegerjaveriak: "Branch master"12:57
javeriakfungi i cloned the new branch specially using git clone -b ; did my changes and then pushed12:57
AJaeger"Topic: stable/7.0" - topic is just a label12:57
AJaegerBranch: master is the important information12:57
fungijaveriak: "pushed" using the `git review` commnd, or something else?12:58
javeriakfungi yes git review12:58
AJaegerjaveriak: as fungi said, you need to tell git review to use the right branch.12:58
AJaegerjaveriak: see http://docs.openstack.org/infra/manual/drivers.html#create-stable-branch on .gitreview12:58
mordredpabelanger: did the localhost/local_connection thing fix anything?12:58
fungijaveriak: either `git review stable/7.0` or update .gitreview on that branch to set defaultbranch=stable/7.012:58
javeriakhmm but its worked before;  i mean for other repo's when i do a git review, it always pushes to whatever branch im working off of locally12:58
AJaegerOr run "git review stable/7.0" to push to 7.012:59
openstackgerritMerged openstack-infra/project-config: Add centos-7 to tripleo cloud  https://review.openstack.org/31172112:59
fungijaveriak: someone has probably updated the .gitreview file on those for you already12:59
*** asettle has joined #openstack-infra12:59
AJaegerjaveriak: there's nothing here for us to do at all. You have all the power to fix this yourself ;)13:00
*** Guest20643 has quit IRC13:00
javeriakoh... so i need to update the git review and thats all..; thanks guys :)13:00
AJaegerjaveriak: fungi and myself can give you a hand, but I do not see anything that needs us13:00
pabelangermordred: I haven't tested / landed the patch yet. So open to suggestions to how to properly test13:00
fungithe .gitreview file, yes13:00
*** kushal has quit IRC13:00
javeriakthey should've put that in the web ui guide page though; managing a repo is tricky :)13:01
rcarrillocruzpabelanger: you aware nibalizer pushed similar patch for ansible OOM problem?13:01
*** kengo_sakai has quit IRC13:01
fungijaveriak: git review is not written by the gerrit developers, so integrating a configuration file for it into the gerrit ui would be strange13:01
pabelangerrcarrillocruz: haha, I am now :)13:01
AJaegerjaveriak: it's documented in the Infra documentation that I mentioned...13:02
pabelangerrcarrillocruz: thanks, abandoned my13:02
rcarrillocruz:-) np13:03
*** tlian has joined #openstack-infra13:04
*** yaume_ has joined #openstack-infra13:04
AJaegerfungi, did you see my mail on openstack-infra with the bindep plan? Can you take the first step, please?13:04
*** weshay_mtg is now known as weshay13:04
*** edmondsw has joined #openstack-infra13:04
fungiAJaeger: yep, replying to it is on my to do pile13:06
fungialso trying to get my last stack of exit code changes redone for bindep, but maybe i'll save that for 1.1.013:06
*** yaume has quit IRC13:07
AJaegerfungi, then I'll wait ;)13:07
*** vgridnev has quit IRC13:07
*** oanson has joined #openstack-infra13:07
electrofelixfungi: I think stick with what we were happy with last week - bea7bdf608a4ae77acbca7689c30f88ab8990c1d13:08
fungiokay, tagging that momentarily13:10
mordredpabelanger: when you say "OOM on the last step" - do you mean the last step in the puppet role?13:11
*** mylu has joined #openstack-infra13:11
electrofelixfungi: as a general process, is the preferred approach to request from one of the infra cores to apply the tag as needed?13:12
*** mylu has quit IRC13:12
pabelangermordred: usually in TASK [puppet : post facts] is when things start breaking13:12
mordredkk. thanks13:12
openstackgerritDan Prince proposed openstack-infra/tripleo-ci: Add planet blog aggregator feeds to website  https://review.openstack.org/31191013:12
electrofelixfungi: or specially to request it from you?13:12
*** MarkAtwood has joined #openstack-infra13:13
fungielectrofelix: for now, yes. in the future we'll likely be granting the current jjb core team further autonomy in that realm13:13
*** inc0 has joined #openstack-infra13:13
*** yaume_ has quit IRC13:14
mordredpabelanger: so - we _do_ read in all of the log file stuff into ram13:14
fungielectrofelix: but 1.5.0 release tag is pushed now, and release jobs are hopefully running for it. happy to tag further releases for you in the future, was just hesitant to do it in the middle of the summit13:14
*** zz_dimtruck is now known as dimtruck13:14
openstackgerritNoam Bloom proposed openstack-infra/project-config: Vitrage - Jenkins  https://review.openstack.org/31205313:14
*** zhurong has joined #openstack-infra13:14
mordredpabelanger: so I think your connection_local thing will help13:15
pabelangermordred: \o/13:15
*** _ari_ has joined #openstack-infra13:16
electrofelixfungi: that's ok, as long as we know what's needed, who to ask, etc. presume ok to ask in the room in general and any infra-core will take care of it13:16
mordredpabelanger, fungi: uhm. can either of you shell in to puppetmaster?13:17
pabelangermordred: I have a shell13:17
mordredpabelanger: dns is resolving a new server for me13:18
fungimordred: i was able to ssh in13:18
*** akshai has joined #openstack-infra13:18
mordredI'm getting The authenticity of host 'puppetmaster.openstack.org (' can't be established.13:18
pabelangerpuppetmaster.openstack.org. 3584 IN A is my dig13:18
mordredall my bad13:18
mordredno - I was in a non-mordred shell on my laptop :)13:19
*** inc0 has quit IRC13:19
mordredso that's good - accounts that are not my actual account in fact cannot log in13:19
*** inc0 has joined #openstack-infra13:19
*** nbloom has quit IRC13:20
*** nbloom has joined #openstack-infra13:20
fungialways a good thing13:21
*** nadya has joined #openstack-infra13:22
mordredfungi, pabelanger: to try pabelanger's patch, we need to enable ansible root logins from localhost13:22
mordredany security concerns with that?13:22
*** sc68cal has quit IRC13:22
fungimordred: as in let root ssh into itself?13:23
*** |-paul-| has quit IRC13:23
fungienable remote root logins in sshd13:23
mordredwell - we currently enable them only from puppetmaster.openstack.org13:23
fungiit's key-based auth only anyway, so no concerns13:23
* mordred makes patch13:24
*** dimtruck is now known as zz_dimtruck13:24
openstackgerritDarragh Bailey proposed openstack-infra/git-review: Limit listing of reviews by branch when specified  https://review.openstack.org/31205613:24
*** nadya has quit IRC13:24
openstackgerritDan Prince proposed openstack-infra/tripleo-ci: Add planet blog aggregator feeds to website  https://review.openstack.org/31191013:24
openstackgerritMerged openstack/diskimage-builder: Remove cloud-initramfs-growroot package  https://review.openstack.org/21543913:24
*** salv-orlando has joined #openstack-infra13:25
*** NobodyCam_ is now known as NobodyCam13:25
*** sc68cal has joined #openstack-infra13:25
*** sc68cal has quit IRC13:27
*** piet has quit IRC13:28
*** eharney has joined #openstack-infra13:28
*** asettle has quit IRC13:28
*** sc68cal has joined #openstack-infra13:28
*** piet has joined #openstack-infra13:28
*** julim has quit IRC13:29
openstackgerritPaul Belanger proposed openstack-infra/project-config: Don't configure a mirror for tripleo-centos-7  https://review.openstack.org/31205813:29
pabelangergood news is centos-7 dib boots in tripleo-test-cloud-rh1 but no AFS mirrors prevent it from coming online.  ^ is a workaround13:30
*** julim has joined #openstack-infra13:32
*** salv-orlando has quit IRC13:32
dimspabelanger : fungi : any ETA for the AFS issues around pypi mirrors? guessing by end of this week?13:34
*** jamesmcarthur has joined #openstack-infra13:34
*** inc0 has quit IRC13:34
*** Hal has joined #openstack-infra13:34
*** ccarmack has joined #openstack-infra13:35
*** inc0 has joined #openstack-infra13:35
*** Hal is now known as Guest4839213:35
pabelangerdims: fungi still running, it was restarted at May02   0:00. I don't know how long, but if a full restore, 2.5 days was my calculation using 300GB at 12/MBps13:36
*** jlanoux_ has quit IRC13:36
fungidims: as far as them still not being updated? i would hope sometime today but i don't know exactly how much data has changed with the pypi file re-org13:36
*** yaume has joined #openstack-infra13:36
*** sc68cal has quit IRC13:36
fungipabelanger's estimate is about as exact as i think we can make13:37
mordredya. I'd guess 2.5 days too13:37
dimsfungi : ack, will ping back late on friday or early monday if things are not back to normal13:37
dimsthanks pabelanger mordred13:37
jeblairif we need it sooner, we can switch to the rw volume13:38
dimsjeblair : i am holding back releases13:39
*** sree has quit IRC13:39
dimsjeblair : so far nothing really critical13:39
*** degorenko has joined #openstack-infra13:39
*** piet has quit IRC13:39
*** sree has joined #openstack-infra13:40
*** jamesmcarthur has quit IRC13:41
mordredpabelanger: so - our fine friends in #ansible-devel tell us that connection: local tasks also run in a subprocess13:41
*** javeriak has quit IRC13:41
mordredso we should still be getting process separation13:42
pabelangermordred: ack13:42
mordredwhich makes me sad panda13:42
phschwartz_mordred: yes connection: local forces a subprocess. This was part of what was also adding to the anisble-puppet run bloat which makes me a sad panda. lol13:43
*** baoli has quit IRC13:44
*** ddieterly has joined #openstack-infra13:44
*** baoli has joined #openstack-infra13:45
*** sree has quit IRC13:45
*** deadnull_ has joined #openstack-infra13:45
openstackgerritBrent Eagles proposed openstack-infra/tripleo-ci: Fail pingtest more quickly  https://review.openstack.org/30848613:45
mordredphschwartz_: I'm confused ... how is local subprocess any different than subprocess to run ssh?13:46
*** anteaya has joined #openstack-infra13:46
* mordred is still on first coffee, likely missing something13:46
*** mriedem has joined #openstack-infra13:46
*** dizquierdo has joined #openstack-infra13:46
*** annegentle has joined #openstack-infra13:46
openstackgerritMonty Taylor proposed openstack-infra/puppet-ssh: Allow root logins from localhost  https://review.openstack.org/31206413:47
openstackgerritMonty Taylor proposed openstack-infra/system-config: Allow root logins from localhost  https://review.openstack.org/31206513:48
mordredpabelanger: ^^ I think those should get ansible able to ssh to itself13:48
*** sc68cal has joined #openstack-infra13:48
pabelangermordred: ack13:50
clarkbmordred it should be cheaper even due to cow13:50
*** jamesmcarthur has joined #openstack-infra13:51
mordredI think I'm asking the wrong question here and/or being unclear13:54
*** pradk has quit IRC13:55
*** x00350071 is now known as xiangxinyong13:56
*** sigmavirus24_awa is now known as sigmavirus2413:56
clarkbI think we should local connection not ssh to self13:56
mordredit sounds like regardless of how heavy of a subprocess it is that we have in the task, it's not task-specific that would cause the OOM, rather it would theoretically be running _anything_ at a similar concurrency for a similar duration13:57
*** matrohon has quit IRC13:57
*** burgerk has quit IRC13:58
mordredand we just happen to be hitting it on our final task13:58
*** oanson has quit IRC13:58
mordredI find that hard to believe, since our playbook and role are actually rather trivial and I can't imagine anyone would be able to use ansible at all if that was the case13:59
anteayasdague: third party ci systems should be listed here: https://review.openstack.org/#/admin/groups/270,members but being listed requires the ci operators to request to be listed so some many not be on the list14:00
anteayasdague: then run a curl command with the email address14:00
anteayacurl -X GET 'https://review.openstack.org/accounts/<email>'14:00
sdagueanteaya: ok, cool, thanks14:00
anteayaif by shortname you mean gerrit username, I am finding some accounts have not created a gerrit username for themselves14:01
anteayaas that is a manual step14:01
*** tonytan4ever has joined #openstack-infra14:01
anteayaand you are welcome14:01
anteayasdague: let me know if you need more14:01
sdagueanteaya: yeh, I was thinking about gerrit username14:01
sdaguejust because it's not clear that I can do certain queries with the long name14:02
anteayaokay, some are not setting that for themselves I have learned14:02
sdagueah, ok14:02
openstackgerritMerged openstack-infra/storyboard: Stop overriding the signal module in terminate()  https://review.openstack.org/30849214:02
fungimordred: is it possible our plays generate an atypical volume of stdout and it's being buffered to memory on the puppetmaster?14:02
anteayathe email is a good backup14:02
sdagueyeh, I think vmware is one of those14:02
sdaguebut 9008 works (which I think is it's internal id)14:02
anteayasdague: also if you do the curl command you get the gerrit account id, you can use the gerrit account id in queries14:02
anteayayeah that sounds like the gerrit account id14:03
*** banix has joined #openstack-infra14:03
anteayaso, awesome14:03
sdaguejust fyi, it's related to this thread - http://lists.openstack.org/pipermail/openstack-dev/2016-May/093753.html14:03
* anteaya reads14:03
mordredfungi: not really - the entirety of the output of /var/lib/puppet/reports is only 488M14:03
mordredfungi: and we only process one log at a time14:03
fungithat includes facter output? i recall we had issues with extremely large facts at one point14:04
fungithough i agree this seems unlikely14:04
*** esker has quit IRC14:04
*** xarses has quit IRC14:04
mordredfungi: I'm playing with a patch right now to skip processing the facts14:04
anteayasdague: nice work14:06
clarkbmordred it does copy a bunch of git repos though perhaps that is all in memory?14:06
*** Jeffrey4l has joined #openstack-infra14:06
clarkbI could see that potentially being different with ssh vs local14:06
mordredclarkb: no - different task14:07
mordredclarkb: this is just focusing on the report facts task, which is where the OOM is apparently happening14:07
anteayaodyssey4me: I have not seen anyone else reporting tox errors for 'basepython = python2' on a mac14:08
clarkbmordred: right but it is python which means we may never get the earlier memory back14:08
mordredclarkb: and the subprocess for the git repo copy should be a different subprocess14:08
*** kzaitsev_mb has quit IRC14:08
mordredclarkb: so we should get it back due to the subprocess going away14:09
odyssey4meanteaya thanks - was just checking as this error was new to me, changing 'python2' to 'python2.7' seems to resolve it14:09
anteayaodyssey4me: thanks for sharing the fix :)14:10
bswartzAJaeger: what is "Needed-By"? I've never heard of this14:10
anteayabswartz: it is a companion to Depends-On14:10
bswartzdoes it work exactly like depends-on, only in reverse?14:10
*** links has quit IRC14:10
anteayabswartz: patch B Depends-On patch A, patch A is Needed-By patch B14:11
*** ddieterly is now known as ddieterly[away]14:11
AJaegerbswartz: it's the counterpart but has no effect, just for documentation14:11
anteayabswartz: mostly it is for reviewers to be able to track the relationship14:11
bswartzAh! okay that help14:11
clarkbmordred and the source side isnt also reading it all to memory?14:11
anteayabswartz: thanks for asking14:11
AJaegerbswartz: and it's documented here http://docs.openstack.org/infra/manual/creators.html#add-new-repository-to-the-governance-repository as part of governance setup14:11
*** lucasagomes is now known as lucas-post-offic14:11
clarkbmordred I am just trying to think how ssh could be different than local14:11
bswartzAJaeger: ty14:11
mordredclarkb: right. I'm having a hard time believe it would be14:12
mordredclarkb: like, I think if we could prove it was we would have found a _giant_ bug in ansible14:12
clarkbmordred: ++14:12
openstackgerritMonty Taylor proposed openstack-infra/ansible-puppet: Stop posting facts  https://review.openstack.org/31210114:13
mordredclarkb: currently thinking about trying that ^^14:13
mordredclarkb: (taking the sledgehammer approach to try to whittle down what is or isn't affecting things)14:14
*** nadya has joined #openstack-infra14:14
*** ddieterly[away] is now known as ddieterly14:15
openstackgerritBen Swartzlander proposed openstack-infra/project-config: Add manila-specs repo  https://review.openstack.org/31185314:20
*** annegentle has quit IRC14:20
*** jamesmcarthur has quit IRC14:20
anteayasc68cal: yes, infra uses storyboard14:21
anteayalennyb: https://review.openstack.org/#/c/309273 hasn't merged yet, so please review the patch and include your paste: http://paste.openstack.org/show/495932/ with your issue so the requirements reviewers can be aware how this is affecting you14:22
*** abregman has quit IRC14:23
*** jlanoux has joined #openstack-infra14:24
*** irdkehn has joined #openstack-infra14:24
anteayasc68cal: and yes, cleaning up old docs from the docs site is an issue that has affected us for some time and to be honest, I don't know what our workflow is for addressing it14:25
nibalizergood morning14:25
*** amitgandhinz has joined #openstack-infra14:25
*** rbrndt has joined #openstack-infra14:25
mordredit's a nibalizer !!!14:26
*** dynarro has joined #openstack-infra14:27
*** arxcruz has quit IRC14:27
anteayamorning nibalizer14:27
*** mylu has joined #openstack-infra14:27
*** zhurong has quit IRC14:27
*** dkehn has joined #openstack-infra14:28
pabelangerclarkb: fungi: as a follow up to our xenial plan, we can flip our infra votes voting first: https://review.openstack.org/#/c/311878/ I confirmed the current experimental jobs are passing14:29
*** dkehn has quit IRC14:29
*** irdkehn is now known as dkehn14:30
*** nmagnezi_ has joined #openstack-infra14:31
*** dkehn_ has joined #openstack-infra14:31
*** nmagnezi has quit IRC14:31
fungipabelanger: yeah, if we're going to start having other projects running voting jobs on it, we should be testing to make sure we don't break it14:32
fungigo for it14:32
pabelangerfungi: my thoughts too14:32
lennybanteaya: are you sure they are related? I can't  see the connection14:34
*** deadnull_ has quit IRC14:34
*** pradk_ is now known as pradk14:37
*** burgerk has joined #openstack-infra14:37
sc68calanteaya: ok. I just don't want it to sink back below the waves ;)14:37
sc68calgive me root and i'll do it ;)14:37
AJaegersc68cal: I'll take care of it now...14:37
openstackgerritMerged openstack-infra/system-config: Add directory resources to puppetdb module  https://review.openstack.org/31163614:38
*** piet has joined #openstack-infra14:38
sc68calAJaeger: ok. sorry to add to your workload14:39
sc68calhas there been discussion about using rsync, with all the flags it needs, including --delete14:39
openstackgerritPaul Belanger proposed openstack/diskimage-builder: DNM - test ubuntu-minimal for precise  https://review.openstack.org/31210414:40
sc68caloh wait ftp. doh14:40
AJaegersc68cal: http://docs.openstack.org/developer/devstack/stack.sh.html is gone - together wit hsome even more dated content...14:40
openstackgerritDan Prince proposed openstack-infra/tripleo-ci: Add planet blog aggregator feeds to website  https://review.openstack.org/31191014:41
openstackgerritMerged openstack-infra/project-config: Move ubuntu-xenial voting for infra jobs  https://review.openstack.org/31187814:41
sc68calAJaeger: ok, thanks :)14:41
*** zeih has quit IRC14:41
fungisc68cal: docs.o.o is (currently) hosted at rackspace cloudsites, not on a normal webserver, so availability of protocols to update it with are limited14:41
openstackgerritRonald Bradford proposed openstack-infra/project-config: Remove kite configuration  https://review.openstack.org/29773814:41
sc68calgotcha. makes sense :)14:41
anteayalennyb: well you posted that they are related14:41
anteayalennyb: I have no idea, my point is that if you experience something as a result of a patchset you need to add a comment on the patchset14:42
anteayaposting to the infra channel will not get the information to the people who need to see it14:42
anteayasc68cal: ha ha ha14:42
anteayaAJaeger: thanks for cleaning it up14:43
anteayasc68cal: I guess the workflow is ping AJaeger :)14:43
AJaegeranteaya: please don't document that workflow ;)14:44
anteayaha ha ha14:44
anteayaha ha ha14:44
*** nadya has quit IRC14:45
anteayaso last night there was a issue with the log server, first mention of it in backscroll was 2016-05-03T06:21:58  <tonyb> A whole bunch of the stable-periodic pipeline just failed with "UNSTABLE" jobs.14:45
*** baoli has quit IRC14:45
AJaegeranteaya: speaking of review: What do you think about https://review.openstack.org/#/c/310929/ ? Those are my first review guidelines14:45
anteayathen yolanda popped into channel and changed the status14:45
AJaegeranteaya: yolanda fixed the filesystem14:45
nibalizerwhat was wrong with the filesystem?14:45
anteayais anyone aware of what the issue _was_ with the log server?14:45
anteayaAJaeger: yes, but what happened that fixing it was required?14:46
AJaegeranteaya: no idea ;(14:46
*** ramishra has quit IRC14:46
anteayayeah the backscroll also gives no clues14:46
lennybanteaya: I see your point. It also looks like a bug in zuul that should be handled. I am wandering if it's only me who sees this behavior. I needed to restart zuul in order to keep my CI running14:46
anteayalennyb: even if it is only you, you need to share your experience with the folks that can do something about it14:47
*** e0ne has quit IRC14:47
anteayalennyb: you might be the canary in the coal mine, you never know14:47
*** mtanino has joined #openstack-infra14:47
anteayalennyb: http://www.wisegeek.org/what-does-it-mean-to-be-a-canary-in-a-coal-mine.htm14:47
fungi[Tue May  3 04:16:51 2016] end_request: I/O error, dev xvde, sector 6064102414:48
*** Keedya has joined #openstack-infra14:48
anteayafungi: thank you14:48
fungi(from dmesg on static.o.o)14:48
fungithe first of many thousands of error messages about tha underlying block device14:48
*** ramishra has joined #openstack-infra14:48
fungier, that14:48
openstackgerritMoshe Levi proposed openstack-infra/irc-meetings: Change sriov meeting frequency to weekly  https://review.openstack.org/31210714:49
*** baoli has joined #openstack-infra14:49
*** esker has joined #openstack-infra14:49
fungimost common cause is network interruption between the nova host and the cinder backend14:49
lennybanteaya: you know, they all dead... eventually :)14:49
*** esker has quit IRC14:49
anteayalennyb: eventually14:49
*** flepied has quit IRC14:49
*** esker has joined #openstack-infra14:49
anteayafungi: ah, do we have any form of recovery?14:50
*** asselin__ has joined #openstack-infra14:50
fungianteaya: yolanda took the filesystem offline and ran a consistency check/repair, then brought it back online14:51
*** fawadkhaliq has joined #openstack-infra14:51
*** asselin has quit IRC14:51
anteayayes, so that is our recovery workflow I guess then14:51
anteayathank you14:51
fungilast mount registered in the log at 09:16:26 utc so presumably that's approximately when it was fixed14:51
* clarkb didnt realize we had rsynced data last time but that likely is the cause of the perms issues we have seen in thr past...14:52
*** doug-fish has joined #openstack-infra14:52
anteayayes, the backscroll confirms14:52
fungirsync should preserve ownership/permissions (if they used the correct options with it anyway)14:53
clarkbwith -a14:53
*** rossella_s has quit IRC14:53
*** baoli has quit IRC14:53
fungiyeah, i didn't look to see how they ran it14:53
clarkbbut its the only thing I can figure and it wasnt a step that was made known previously14:53
*** Keedya has quit IRC14:53
yolandaanteaya, yes, fixed that14:53
yolandaon docs-draft and on logs14:54
anteayayolanda: yes I saw, thank you14:54
anteayayolanda: just had not seen the cause discussed in backscroll14:54
clarkbyolanda: did you write anything to the filesystems ?14:54
clarkbif so can you share how that was done as I syspect this is the source of the odd permissions issues we have seen14:55
yolandaclarkb, i unmounted, ran fsck, and rsynced the content that was on /srv/static while it was unmounted14:55
fungimost common occurrence of that disconnect failure pattern are maintenance or an unplanned outage in rackspace dfw14:55
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: [NO MERGY] Test a fake periodic job  https://review.openstack.org/22978914:55
clarkbyolanda: what was the rsync command14:55
yolandachecking in history14:55
yolanda rsync -vv -aHAXPh --ignore-existing --dry-run /tmp/docs-draft/ /srv/static/docs-draft/14:56
yolandasame for logs, and of course, removing the dry-run14:56
*** mtanino has quit IRC14:57
anteayafungi: ah thank you14:57
clarkbyolanda: do you know if that was the same command used previously when we had the file owner ship issues?14:58
*** kengo_sakai has joined #openstack-infra14:58
*** doug-fis_ has joined #openstack-infra14:58
*** tosky has quit IRC14:58
yolandaclarkb, got the command from jhesketh, is the same that we used last time14:58
*** baoli has joined #openstack-infra14:59
*** tosky has joined #openstack-infra14:59
*** nmagnezi_ has quit IRC14:59
*** xarses has joined #openstack-infra15:00
*** xarses has quit IRC15:00
*** doug-fi__ has joined #openstack-infra15:00
*** xarses has joined #openstack-infra15:00
clarkbok the -a should preserve the ownership bits15:01
*** doug-fish has quit IRC15:01
clarkbif we do find any files owned by root again we should try to trace it back to the recovery. I did not realize we did any write ops to the fs last time when I was trying to debug that ...15:02
*** _ody_ is now known as _ody15:02
*** kengo_sakai has quit IRC15:02
*** doug-fis_ has quit IRC15:02
*** doug-fi__ has quit IRC15:05
openstackgerritTim Laszlo proposed openstack-infra/shade: Add support for host aggregates  https://review.openstack.org/31093715:05
*** doug-fish has joined #openstack-infra15:07
*** chem has joined #openstack-infra15:07
*** mylu has quit IRC15:08
*** hashar is now known as hasharAway15:08
dynarrohey pabelanger, it's doraly navarro, we met at the summit. I got REALLY curious about infra after our conversation and I wanted to know if you could tell me how to start contribute, maybe point me a low-hanging-fruit bug to start with.15:09
mordreddynarro: welcome to the jungle!15:10
dynarromordred, hey there!15:10
*** psachin has quit IRC15:10
*** tesseract- has quit IRC15:12
pabelangerdynarro: excellent! We have an openstack-infra meeting today, see: http://eavesdrop.openstack.org/#Project_Infrastructure_Team_Meeting I highly recommend joining us15:12
pabelangerdynarro: as for getting started, what things interest you15:13
*** r-mibu_ has quit IRC15:13
*** r-mibu has joined #openstack-infra15:13
*** tphummel has joined #openstack-infra15:14
*** jamesmcarthur has joined #openstack-infra15:16
fungidynarro: http://docs.openstack.org/infra/system-config/project.html#contributing15:17
fungithat's our canned answer anyway15:18
dynarrocool! I try to be there. I don't know, I was hoping on an easy software development bug in which I can learn how infra works15:18
*** zz_dimtruck is now known as dimtruck15:19
*** links has joined #openstack-infra15:19
openstackgerrityolanda.robla proposed openstack-infra/project-config: Remove keystone2+magnum job  https://review.openstack.org/31211415:19
dynarrobut as I'm still new to coding I don't mind learning other stuff :)15:20
*** markmcd has quit IRC15:21
*** markmcd1 has quit IRC15:21
anteayasince we are using the status page as a team communication tool now, do we think https://wiki.openstack.org/wiki/Infrastructure_Status should be added to the channel topic?15:21
*** markmcd has joined #openstack-infra15:21
*** markmcd1 has joined #openstack-infra15:21
*** sdake has joined #openstack-infra15:23
*** yaume has quit IRC15:23
dynarrofungi, yeah I was reading that wiki before and I was looking to the storyboard, but I don't know where to start15:24
anteayatonyb: so tacker knows that if someone else takes the slot directly before or after their meeting they don't get to change their meeting time then, correct? https://review.openstack.org/#/c/311789/15:25
*** EricGonczer_ has joined #openstack-infra15:25
anteayatonyb: all the rest of us can deal with utc time, and there is a resolution stating we use it15:25
AJaegerdynarro: start with one repository that sounds interesting and start reviewing - and get an overview with reading http://docs.openstack.org/infra/ . Welcome!15:25
anteayatonyb: http://git.openstack.org/cgit/openstack/governance/tree/resolutions/20150512-utc.rst15:26
*** sdake_ has joined #openstack-infra15:26
dynarroAJaeger, cool, thanks!15:27
pabelangerdynarro: if you are looking to get into puppet, we are working on converting some of our servers from ubuntu-precise to ubuntu-trusty.15:27
*** tonytan_brb has joined #openstack-infra15:27
pabelangerdynarro: We're also trying to bring more mirrors online, like RPM and gems, if you want to learn more about AFS15:27
*** vincentll has quit IRC15:28
*** sdake has quit IRC15:28
mnaserjust curious, was there a block storage issue that's causing those fs issues?15:28
*** vgridnev has joined #openstack-infra15:29
*** jyuso has quit IRC15:29
clarkbzaro: https://review.openstack.org/#/c/311903/2 will build a war that is useable for deployments with puppet again if you are able to review that15:29
*** kmartin has quit IRC15:30
anteayamnaser: hello15:30
*** jyuso has joined #openstack-infra15:30
*** tonytan4ever has quit IRC15:31
anteayamnaser: http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2016-05-03.log.html#t2016-05-03T14:48:0615:31
openstackgerrityolanda.robla proposed openstack-infra/project-config: Move shade+magnum neutron job to non-voting  https://review.openstack.org/31212015:31
fungimnaser: strangely, there was a block storage issue reported impacting two of the three regions we use in that provider, but not the one where the system in question was hosted https://status.rackspace.com/index/viewincidents?group=1115:32
*** kzaitsev_mb has joined #openstack-infra15:32
fungii haven't checked our trouble tickets though15:32
mnaserhi anteaya15:32
mnaserthat's the worst15:32
*** julim has quit IRC15:32
*** dizquierdo has quit IRC15:33
*** julim has joined #openstack-infra15:33
anteayamnaser: oh there are worse things15:33
clarkbzaro then if we get 311903 merged we can update the change to https://review.openstack.org/#/c/311894/ to use the non test uri15:33
anteayamnaser: but sad fs systems are sad making :(15:33
clarkbzaro: but hoping you can confirm if that change looks good to you (and maybe if you know how upstream gerrit is handling this problem)15:34
*** EricGonc_ has joined #openstack-infra15:34
openstackgerritChris Denneen proposed openstack-infra/jenkins-job-builder: gitlab trigger: Support new "trigger-open-merge-request-push" options  https://review.openstack.org/31212115:34
*** hdd has joined #openstack-infra15:34
clarkbnibalizer: did you get around to launching new server(s) yesterday? if so any word on https://review.openstack.org/#/c/311872/ ?15:35
*** nbloom has quit IRC15:35
*** ccarmack has left #openstack-infra15:35
nibalizerno i didn't but i might do that today15:35
nibalizeri mean, maybe15:35
nibalizerthere are quite a few irons in the fire15:35
*** EricGonczer_ has quit IRC15:36
openstackgerritClark Boylan proposed openstack-infra/project-config: Revert "Make puppet-gerrit beaker rspec non voting"  https://review.openstack.org/31212315:36
clarkbnibalizer: kk15:36
*** mtanino has joined #openstack-infra15:36
*** salv-orlando has joined #openstack-infra15:36
clarkbyolanda: pabelanger ^ 312123 is the revert to make puppet-gerrit beaker rspec voting again15:36
pabelangerclarkb: ack15:38
*** piet has quit IRC15:38
openstackgerritSpencer Krum proposed openstack-infra/system-config: Add a storyboard-dev server  https://review.openstack.org/31212415:38
nibalizerclarkb: in theory that is the next server I would be building15:38
yolandaclarkb, approve15:40
*** kushal has joined #openstack-infra15:40
*** jang has quit IRC15:40
*** mylu has joined #openstack-infra15:41
*** lucas-post-offic is now known as lucasagomes15:41
pabelangerany folks proficient with grub?  Running into the following error with ubuntu-precise dib: http://paste.openstack.org/show/495974/15:41
morganpabelanger: looking.15:41
clarkbas a heads up one thing I will be attempting to do in the near future is upgrading elasticsearch to latest version of 1.7.x ES15:42
clarkbif anyone is interested in learning ore about ES I am happy to help them do it instead :)15:42
clarkbhubcap said at the summit that this should make our lives better with the broken shard replication allocation15:43
morganpabelanger: thats a new one.15:43
pabelangerclarkb: I'd be interested in learning more about ES15:43
anteayaclarkb: I'm interested in following along15:44
openstackgerritMerged openstack/requirements: Babel Integration tests  https://review.openstack.org/30980815:44
pabelangermorgan: currently trying to get more output15:44
morganpabelanger: that would be good15:44
clarkbpabelanger: anteaya great, maybe we can aim for end of week, thursday/friday15:45
*** mylu has quit IRC15:45
anteayaclarkb: let's aim for that15:45
*** piet has joined #openstack-infra15:45
*** kmartin has joined #openstack-infra15:45
morganpabelanger: since this is dib, i'm assuming this is local build not pxe or something, right?15:46
pabelangeranteaya: clarkb: ack15:46
anteayapabelanger: great15:47
pabelangermorgan: Right, running it locally from centos-7 host machine15:47
anteayaclarkb: did you want to put an agenda item on today's meeting agenda about selecting a week for the operating system control plane upgrade sprint we agreed to do?15:47
*** eil397 has joined #openstack-infra15:47
*** bpokorny has joined #openstack-infra15:48
*** pradk has quit IRC15:49
*** tonytan_brb has quit IRC15:49
morganpabelanger: the only things that tend to point to that in my experience is where the partitioning scheme didn't work on the img.15:49
clarkbanteaya: ya I can do that shortly15:49
morganpabelanger: but more output probably would help (i can also try and centos7 a machine and try it here as well if you'd like)15:49
anteayaclarkb: awesome, thank you15:49
electrofelixfungi: I've noticed that there hasn't been an upload to pypi of JJB 1.5.0, something not get published correctly?15:50
openstackgerritGhe Rivero proposed openstack-infra/shade: Add quotas support  https://review.openstack.org/28511015:50
fungielectrofelix: i'll take a look15:51
*** annegentle has joined #openstack-infra15:51
openstackgerritMerged openstack-infra/project-config: Revert "Make puppet-gerrit beaker rspec non voting"  https://review.openstack.org/31212315:51
openstackgerritGhe Rivero proposed openstack-infra/shade: Add quotas support  https://review.openstack.org/28511015:51
*** doug-fish has quit IRC15:52
*** sridhar_ram has joined #openstack-infra15:52
*** doug-fish has joined #openstack-infra15:52
*** ggnel_t has quit IRC15:52
*** tonytan4ever has joined #openstack-infra15:52
*** mikelk has quit IRC15:52
*** hasharAway is now known as hashar15:52
openstackgerritMerged openstack-infra/jenkins-job-builder: Update workspace-cleanup to support newer options  https://review.openstack.org/31008415:52
*** tonytan4ever has quit IRC15:53
yolandafungi, electrofelix , that is because our pypi mirror is out of sync15:53
fungiyolanda: no, it's not because of that15:53
electrofelixyolanda: shouldn't it appear in the main pypi repo though?15:53
yolandaah, pypi15:53
fungielectrofelix: yolanda: http://logs.openstack.org/91/91fef0bd90b06f8f0e64319987bb9995e639c857/release/jenkins-job-builder-tarball/759facb/console.html#_2016-05-03_13_17_04_09415:53
*** EricGonc_ has quit IRC15:53
yolandaok pypi directly, not to our mirrors ok15:53
fungilooks like it failed to write the tarball to static.openstack.org15:53
*** doug-fish has quit IRC15:53
pabelangermorgan: Ya, I was thinking that was the issue too. Going to run parted manually and see if I can reproduce the DIB failure15:53
fungiis the tarballs volume also in a bad state?15:54
morganpabelanger: ok sounds good. let me know if you need to poke at a parted command being run / help out, happy to do so.15:54
morganpabelanger: s/need/want me/15:54
*** pradk has joined #openstack-infra15:55
*** inc0 has quit IRC15:56
dynarropabelanger, the actually converting servers from ubuntu-precise to ubuntu-trusty sounds really good, where can I found more info about that?15:58
*** tphummel has quit IRC15:58
*** eil397 has left #openstack-infra15:59
openstackgerrityolanda.robla proposed openstack-infra/project-config: Move shade+magnum neutron job to non-voting  https://review.openstack.org/31212016:01
*** lindsayk_ is now known as lindsayk16:02
*** dmk0202 has quit IRC16:03
*** salv-orlando has quit IRC16:03
*** kjorgensen has joined #openstack-infra16:04
sdake_did anyone in infra team register kolla-kubernetes in pypi?16:04
*** earlephilhower has quit IRC16:05
*** doug-fish has joined #openstack-infra16:05
*** doug-fish has quit IRC16:05
*** gordc has quit IRC16:05
*** kjorgensen has quit IRC16:06
*** doug-fish has joined #openstack-infra16:06
*** kjorgensen has joined #openstack-infra16:06
*** diazjf has joined #openstack-infra16:06
*** keedya has joined #openstack-infra16:06
*** lakshmiS has joined #openstack-infra16:07
*** doug-fish has quit IRC16:08
*** doug-fish has joined #openstack-infra16:08
*** mylu has joined #openstack-infra16:11
*** e0ne has joined #openstack-infra16:11
anteayasdake_: I doubt it, but checking pypi should tell you if the name is registered and the owner if it is16:11
anteayaAJaeger: have you a moment for a repo retirement clarification question?16:12
anteayaAJaeger: my read of http://docs.openstack.org/infra/manual/drivers.html#remove-project-content is that all the files are gone including the dot files, with only the readme remaining16:12
anteayaAJaeger: kite and kiteclient still have dot files remaining, along with the new readme16:12
anteayaAJaeger: what are your thoughts?16:12
*** ifarkas has quit IRC16:13
*** mylu has quit IRC16:13
anteayaAJaeger: 297738 is the patch for reference16:13
*** earlephilhower has joined #openstack-infra16:13
openstackgerritMerged openstack-infra/jenkins-job-builder: Update Google Cloud Storage publisher  https://review.openstack.org/31008016:15
*** roxanaghe has joined #openstack-infra16:15
*** matt-borland has joined #openstack-infra16:16
*** infra-red has joined #openstack-infra16:16
*** mylu has joined #openstack-infra16:16
*** nadya has joined #openstack-infra16:16
*** abregman has joined #openstack-infra16:16
openstackgerritDavanum Srinivas (dims) proposed openstack-infra/project-config: Add a functional job for python-k8sclient  https://review.openstack.org/31214616:17
*** nadya has quit IRC16:17
*** abregman has quit IRC16:17
*** abregman has joined #openstack-infra16:17
*** mylu has quit IRC16:18
openstackgerritEmilien Macchi proposed openstack-infra/project-config: zuul/layout: add missing experimental jobs for Puppet CI  https://review.openstack.org/31214816:18
*** sputnik13 has joined #openstack-infra16:19
*** roxanaghe has quit IRC16:19
pabelangerdynarro: I hope to talk about it today in our meeting16:20
dynarropabelanger, cool then!16:20
*** apoorvad has joined #openstack-infra16:21
*** docaedo2 has joined #openstack-infra16:21
*** apoorvad has quit IRC16:22
*** fawadkhaliq has quit IRC16:22
*** apoorvad has joined #openstack-infra16:22
sdake_anteaya yup we got itsorted out, its a friendly rather then a squatter :)16:22
sdake_I think they were squatting in case of unfriendlies :)16:23
anteayagood news16:23
pabelangerdynarro: here is our etherpad from austin about the topic: https://etherpad.openstack.org/p/newton-infra-distro-upgrade-plans16:23
sdake_anteaya i did the same thing (dot files) on kolla-mesos retiremeenet16:23
sdake_anteaya I can show you how I solved it16:23
dynarropabelanger, thanks16:24
sdake_anteaya of course you will have to add the acls back in to get the readme.rst with the exactgit hash id in16:24
anteayasdake_: thanks for the example, I was looking for one16:26
anteayaright so easier to delete the dot files while the acl files still exist16:27
anteayasdake_: thank you16:27
sdake_anteaya dims actually suggsted it16:27
*** gyee has joined #openstack-infra16:27
sdake_so I dont take credit for it  :)16:27
*** ccamacho is now known as ccamacho|away16:29
anteayasdake_: thank you16:29
AJaegeranteaya: you're right - that should have been all cleaned out in the previous patch. But if we clean this up now, the instructions won't work for the HEAD^1 checkout, so I would leave it. Or do it like sdake_ suggested - feel free t ocomment and WIP the change...16:29
*** pilgrimstack has quit IRC16:30
anteayaAJaeger: thanks16:30
*** javeriak has joined #openstack-infra16:31
AJaegerproject-config cores, could you review a change to use constraints for Babel in our scripts, please? https://review.openstack.org/309809 . The requirements change has merged16:31
*** diazjf has left #openstack-infra16:32
*** annegentle has quit IRC16:33
openstackgerritPaul Belanger proposed openstack-infra/system-config: Remove devstack-centos7 logging for nodepool  https://review.openstack.org/31215816:34
*** doug-fish has quit IRC16:36
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add ceph-deb-hammer.list to both trusty and xenial  https://review.openstack.org/31216116:36
electrofelixwaynr: you happy with https://review.openstack.org/274195?16:36
*** doug-fish has joined #openstack-infra16:37
*** doug-fish has quit IRC16:37
*** johnny has joined #openstack-infra16:38
*** johnny is now known as Guest2184816:39
openstackgerritMerged openstack-infra/jenkins-job-builder: Removing perModuleEmail paramater management  https://review.openstack.org/25998716:39
*** jlanoux has quit IRC16:39
dimsanteaya : AJaeger : hopefully easy one when you get a chance https://review.openstack.org/#/c/312146/ please16:40
anteayadims: if it is voting (and it appears that it is) it goes in the gate pipeline as well16:41
*** bpokorny_ has joined #openstack-infra16:42
openstackgerritDavanum Srinivas (dims) proposed openstack-infra/project-config: Add a functional job for python-k8sclient  https://review.openstack.org/31214616:42
dimsanteaya : ack done16:42
anteayadims: +216:43
anteayaand thank you16:43
*** inc0 has joined #openstack-infra16:43
*** bpokorny_ has quit IRC16:43
*** roxanaghe has joined #openstack-infra16:43
dimsanteaya : thanks for the quick review16:43
anteayatiming frw16:44
*** bpokorny_ has joined #openstack-infra16:44
anteayafor rrrrraaa win was the other acronym16:44
*** bpokorny has quit IRC16:45
*** gordc has joined #openstack-infra16:46
openstackgerritPaul Belanger proposed openstack-infra/system-config: Switch non dib image loggers to INFO  https://review.openstack.org/31217016:48
pabelangerianw: clarkb: ^ Should fix our empty log files for non-dib image builds.16:48
openstackgerritJordan Pittier proposed openstack/requirements: Change urllib3 version to urllib3>=1.15  https://review.openstack.org/31217216:50
openstackgerritSpencer Krum proposed openstack-infra/system-config: Remove infracloud from ansible inventory  https://review.openstack.org/31217316:50
nibalizerclarkb: ^^16:51
anteayanibalizer: can you expand that commit message to say why?16:51
nibalizeranteaya: sure16:51
anteayanibalizer: thanks16:52
*** vhosakot has joined #openstack-infra16:52
anteayahard to stay current with all the things16:52
*** infra-red has quit IRC16:52
openstackgerritSpencer Krum proposed openstack-infra/system-config: Remove infracloud from ansible inventory  https://review.openstack.org/31217316:52
*** madhu_ak has joined #openstack-infra16:52
anteayawonderful, thank you16:53
anteaya<-- going for a walk, expected to be back in about an hour16:54
*** RickA-HP has joined #openstack-infra16:54
*** tlian has quit IRC16:55
openstackgerritRonald Bradford proposed openstack-infra/project-config: Remove kite configuration  https://review.openstack.org/29773816:55
*** doug-fish has joined #openstack-infra16:57
*** akshai has quit IRC16:57
anteayathe bugs have started16:57
*** Jeffrey4l has quit IRC16:58
*** rkukura has joined #openstack-infra16:59
*** akshai has joined #openstack-infra16:59
odyssey4meis there a specific stable maint IRC channel?17:00
*** doug-fish has quit IRC17:00
*** tonytan4ever has joined #openstack-infra17:01
*** hashar has quit IRC17:01
*** hdd has quit IRC17:01
jrollodyssey4me: #openstack-stable17:02
odyssey4meah, thanks jroll17:02
*** pilgrimstack has joined #openstack-infra17:03
*** jistr has quit IRC17:04
*** tlian has joined #openstack-infra17:04
krotscheckWho's on vacation right now?17:05
krotscheckActually, lemme reask that question.17:05
krotscheckIs jeblair on vacation right now?17:05
*** RickA-HP has quit IRC17:05
clarkbkrotscheck: I think he mentioned he was largely afk this week17:05
krotscheckclarkb: Coooooolio.17:05
*** sshnaidm has quit IRC17:06
krotscheckclarkb: I'll ping him next week to finish NPM mirror things.17:06
*** dizquierdo has joined #openstack-infra17:06
*** doug-fish has joined #openstack-infra17:06
*** ericksonsantos has joined #openstack-infra17:06
waynrelectrofelix: looking!17:08
*** vhosakot has quit IRC17:09
*** ams_ has joined #openstack-infra17:09
*** jordanP has quit IRC17:09
waynrelectrofelix: looks like patch is in merge conflict17:09
*** sambetts is now known as sambetts|afk17:09
zxiiroelectrofelix: did we release yet?17:10
*** Guest48392 has quit IRC17:10
mordredkrotscheck: I approve of our new NPM mirror overlords17:10
*** mriedem1 has joined #openstack-infra17:10
*** trown is now known as trown|lunch17:10
krotscheckmordred: It really is the Big Man of the mirrors, isn't it.17:10
*** mriedem has quit IRC17:11
*** Hal has joined #openstack-infra17:11
*** Hal is now known as Guest7052317:11
*** doug-fish has quit IRC17:12
pabelangermorgan: appears to be related to a centos-7 build host; possible missing dependency. Running from ubuntu-trusty, ubuntu-precise worked as expect17:12
zaroclarkb: good new about gerrit memory.17:12
*** earlephilhower has quit IRC17:13
timothyb89fungi, jhesketh, pabelanger, clarkb, mtreinish: if you have a free moment, could you take a look at this revised StackViz patch? https://review.openstack.org/#/c/311768/17:13
*** vhosakot has joined #openstack-infra17:14
*** _amrith_ is now known as amrith17:14
*** earlephilhower has joined #openstack-infra17:15
mtreinishtimothyb89: that's pabelanger's revert from yesterday17:16
*** doug-fish has joined #openstack-infra17:16
pabelangertimothyb89: have you run stackviz across all our images? Personally, I'd like to see some logs attached to the issue just confirming stackviz builds properly17:17
pabelangerjust to make sure we don't break nodepool dibs again17:17
zaroclarkb: had a question for you on already merged change https://review.openstack.org/#/c/31189417:17
*** tphummel has joined #openstack-infra17:17
timothyb89mtreinish: ah, you're right, sorry! this is the correct link: https://review.openstack.org/#/c/311815/17:17
clarkbzaro: ok17:17
clarkbzaro: the version we have deployed is no longer deployable17:18
timothyb89pabelanger: we saw it build correctly save for a (now fixed) race condition on Fedora, at least17:18
clarkbzaro: so no we cannot test with it or even make a new gerrit with it17:18
openstackgerritMerged openstack-infra/storyboard: Don't try to cancel the timer thread if it isn't alive  https://review.openstack.org/30847117:18
clarkbzaro: which is why I have asked you to review the change that updates the gerrit war so that we can potentially switch to it17:18
timothyb89pabelanger: I'd be happy to manually test on the images, but I'm unsure how to do so17:19
clarkbzaro: also curious how upstream gerrit is dealing with this17:19
clarkbzaro: as ideally we would do what they are doining17:19
*** mriedem1 is now known as mriedem17:20
pabelangertimothyb89: my current way is using tools/build-image.sh in openstack-infra/project-config. You'll need to export some variables to setup the distro, and include stackviz element.17:21
pabelangertimothyb89: otherwise, you can use disk-image-create directly17:21
*** bpokorny_ has quit IRC17:21
pabelangertimothyb89: you can then look into nodepool/nodepool.yaml for the 5 OSes we do DIBs for17:22
zaroclarkb: i reviewed the other change, pls take a look.17:22
*** abregman has quit IRC17:22
*** inc0 has quit IRC17:22
*** bpokorny has joined #openstack-infra17:22
openstackgerritMerged openstack-infra/project-config: Add a functional job for python-k8sclient  https://review.openstack.org/31214617:23
*** roxanaghe has quit IRC17:23
timothyb89pabelanger: alright, I'll take a look, thank you!17:23
clarkbzaro: repo2 doesn't have valid ssl17:23
zaroclarkb: I'm still not sure why you think we can't deploy 2.11.4-11 but can deploy 2.11.4-1317:23
clarkbzaro: my preference would be to use repo1 as a result, maybe you can ask why gerrit is using insecure repos?17:23
clarkbzaro: because -11 uses the old url for bouncy castle which is now a 40417:23
clarkbzaro: this causes gerrit init to fail17:23
clarkbzaro: basically the -11 war is DOA17:24
*** roxanaghe has joined #openstack-infra17:24
zaroclarkb: ahh ok i didn't realize the ssl thing17:25
*** weshay has quit IRC17:25
zarobut yeah, makes sense now.17:25
*** weshay has joined #openstack-infra17:26
*** lucasagomes is now known as lucas-dinner17:27
*** salv-orlando has joined #openstack-infra17:27
*** ddieterly is now known as ddieterly[away]17:27
*** e0ne has quit IRC17:31
*** nadya has joined #openstack-infra17:33
*** skolekonov__ has joined #openstack-infra17:33
*** vhosakot has quit IRC17:34
*** serverascode has quit IRC17:34
*** woodster_ has quit IRC17:35
*** sweston has quit IRC17:35
*** zxiiro has quit IRC17:35
*** maximov has quit IRC17:35
*** zhiyan has quit IRC17:35
*** dougwig has quit IRC17:35
*** xgerman has quit IRC17:35
*** lindsayk has quit IRC17:35
*** pvaneck has joined #openstack-infra17:35
*** serverascode has joined #openstack-infra17:35
*** zhiyan has joined #openstack-infra17:37
zaroclarkb: i'm not sure why gerrit chose to use not secure connection.  i'm guessing it's a bug since they use https to download all other maven resources.17:37
*** lindsayk has joined #openstack-infra17:37
*** fmccrthy has quit IRC17:37
*** patrickeast has quit IRC17:37
*** maximov has joined #openstack-infra17:37
*** woodster_ has joined #openstack-infra17:37
*** sweston has joined #openstack-infra17:38
openstackgerritMerged openstack-infra/project-config: Don't configure a mirror for tripleo-centos-7  https://review.openstack.org/31205817:38
*** patrickeast has joined #openstack-infra17:38
electrofelixzxiiro: yes, but there was a problem with the upload to tarballs and the release didn't get pushed to pypi17:39
*** akshai has quit IRC17:39
*** tonytan4ever has quit IRC17:39
*** xgerman has joined #openstack-infra17:39
*** dougwig has joined #openstack-infra17:40
electrofelixwaynr: I must have approved something that caused an issue :p17:40
*** akshai has joined #openstack-infra17:41
*** dynarro has quit IRC17:41
*** fmccrthy has joined #openstack-infra17:41
*** sileht has quit IRC17:42
*** jerryz has joined #openstack-infra17:42
*** zxiiro has joined #openstack-infra17:42
waynrelectrofelix: so i am going to optimistically say that i will set aside some time either tonight or tomorrow night to rebase the 2.0 work onto master branch now that the 1.5 release is underway17:43
openstackgerritJim Rollenhagen proposed openstack-infra/project-config: Add api-ref jobs for ironic  https://review.openstack.org/31218417:44
*** skolekonov__ has quit IRC17:45
openstackgerritDarragh Bailey proposed openstack-infra/jenkins-job-builder: Compatibility support for older maven project plugins  https://review.openstack.org/27419517:46
*** wajdi has joined #openstack-infra17:46
electrofelixwaynr: it's tagged from a commit last week, so we're definitely free to start moving forward with v2 stuff17:46
*** sileht has joined #openstack-infra17:47
*** vhosakot has joined #openstack-infra17:48
*** skolekonov has joined #openstack-infra17:48
zaroclarkb: hopefully well get an answer soon, https://gerrit-review.googlesource.com/#/c/7717917:48
*** dizquierdo has quit IRC17:49
*** skolekonov is now known as skolekonov__17:49
openstackgerritMerged openstack-infra/storyboard-webclient: Remove pointless variable creation  https://review.openstack.org/30640117:51
fungielectrofelix: waynr: i don't think it's jjb's fault. i suspect it's a problem with the tarballs volume on static.openstack.org (perhaps related to the logs and docs-draft volume problems from earlier today?) but i don't have time to look into it yet. dealing with openssl vulnerability updates and then need to prep for the weekly team meeting17:51
fungielectrofelix: waynr: the job log had a java traceback about a write error and ended with an "UNSTABLE" status17:51
mordredwajdi: heya! saw your message in #openstack-ironic ... we definitelky want to include fungi in such a discussion17:52
mordredfungi: wajdi may want to be involved in donating baremetal resources to infra17:52
*** jamesmcarthur has quit IRC17:52
electrofelixfungi: tag is there, so as long as we can find a way to retrigger the release job sometime this week to upload the necessary, I don't think there is any pressure (though maybe avoid doing it Friday...)17:53
*** salv-orlando has quit IRC17:53
fungielectrofelix: it's a simple zuul enqueue-ref command to get the machinery retriggered as soon as we suss out why tarballs.o.o refused the upload17:54
clarkbmordred: as in here is cloud with ironic flavors or here is hardware to make an infra cloud?17:55
*** kzaitsev_mb has quit IRC17:56
mat128clarkb: not sure what was discussed (I wasn't part of it) but my understanding was more like providing a third party CI setup that can actually test against baremetal in a lab17:56
jesusaurfungi: clarkb: jeblair: was there every any discussion about what to do about the zuul tests hitting the 50MB subunit limit?17:57
mordredclarkb: I do not know17:57
fungijesusaur: none to my knowledge17:57
mordredclarkb: I believe the ultimate motivation is to enhance infra's ability to test ironic things - but digging in to the details of what is desired and what is possible is likely useful17:57
*** akshai has quit IRC17:58
*** baoli has quit IRC18:00
*** pilgrimstack1 has joined #openstack-infra18:00
*** inc0 has joined #openstack-infra18:00
*** kzaitsev_mb has joined #openstack-infra18:01
*** baoli has joined #openstack-infra18:01
*** pilgrimstack has quit IRC18:01
*** sshnaidm has joined #openstack-infra18:01
*** akshai has joined #openstack-infra18:01
*** dolphm has left #openstack-infra18:02
*** vhosakot_ has joined #openstack-infra18:03
*** vhosakot_ has quit IRC18:03
*** markvoelker has quit IRC18:04
*** vhosakot_ has joined #openstack-infra18:04
*** annegentle has joined #openstack-infra18:04
*** piet has quit IRC18:04
*** vhosakot has quit IRC18:05
*** markvoelker has joined #openstack-infra18:05
*** tosky has quit IRC18:07
*** inc0 has quit IRC18:07
*** markvoelker_ has joined #openstack-infra18:07
*** markvoelker has quit IRC18:08
*** kushal has quit IRC18:09
*** inc0 has joined #openstack-infra18:10
jesusaurfungi: any preference on where/how I should start that conversation?18:10
*** skolekonov__ has quit IRC18:10
*** amrith is now known as _amrith_18:11
fungijesusaur: if you want to add a "when" to that, probably after people are rested up from last week18:11
fungisome of the potential participants for such conversations aren't really around until next week18:13
*** HeOS has joined #openstack-infra18:13
jesusaurah, ok18:14
* AJaeger has to leave now and won't be around for the IRC meeting18:14
AJaegerOne wish for today: project-config cores, could you review a change to use constraints for Babel in our scripts, please? https://review.openstack.org/309809 . The requirements change has merged...18:15
*** trown|lunch is now known as trown18:15
nibalizerAJaeger: <318:15
*** _amrith_ is now known as amrith18:16
*** Remy has joined #openstack-infra18:18
openstackgerritMerged openstack-infra/project-config: zuul/layout: add missing experimental jobs for Puppet CI  https://review.openstack.org/31214818:19
*** tonytan4ever has joined #openstack-infra18:19
anteayaAJaeger: thank you18:19
*** markus_z has quit IRC18:19
*** Goneri has quit IRC18:23
*** Goneri has joined #openstack-infra18:24
openstackgerritDarragh Bailey proposed openstack-infra/jenkins-job-builder: Test stream no encoding bug  https://review.openstack.org/31219718:24
*** vgridnev has quit IRC18:25
*** electrofelix has quit IRC18:26
*** ddieterly[away] has quit IRC18:27
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Define job-groups for api-ref and api-guide  https://review.openstack.org/31219818:27
*** flwang1 has joined #openstack-infra18:29
flwang1ttx: ping re the tags of zaqar18:29
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add gate-tripleo-ci-centos-7-nonha experimental job  https://review.openstack.org/31220018:31
*** armax_ has joined #openstack-infra18:31
pabelangerEmilienM: ^ experimental tripleo job for puppet-aodh18:31
*** bpokorny_ has joined #openstack-infra18:31
*** bpokorny_ has quit IRC18:31
*** bpokorny has quit IRC18:31
RemyHi! Quick question, does Zuul require a specific version of Gerrit to work? I am using Gerrit 2.11.4-11-ga14450f with the latest revision of Zuul from the repository and I get this error when zuul tries to send the jobs statuses to Gerrit: http://pastebin.com/nd3EsJTy18:31
*** bpokorny has joined #openstack-infra18:32
*** edmondsw has quit IRC18:32
EmilienMpabelanger: trigger: gate?18:33
anteayapabelanger: what does the [OT] tag mean?18:33
EmilienMit's ok for experimental?18:33
clarkbRemy: those are the same versions we use18:33
*** dynarro has joined #openstack-infra18:33
*** armax has quit IRC18:33
*** armax_ is now known as armax18:33
morgananteaya: off topic?18:33
clarkbinfra-root what is test-magnum-devstack? and will you be sad if I delete it?18:33
pabelangerEmilienM: Ya, I left it as gate, since it will replace fedora-22 eventually18:33
EmilienMpabelanger: also, why in ironic project?18:33
pabelangeranteaya: yes, off-topic18:33
anteayaclarkb: I don't know what that is, I have no attachment to it personally18:34
pabelangerEmilienM: there is the single location where jobs are defined18:34
mordredclarkb: I know yolanda was working on getting magnum support going18:34
anteayapabelanger: oh, I didn't consider your thread off topic at all18:34
Remyclarkb: arf, my mistake I pasted the wrong version. I'm using  2.12.2 for Gerrit sorry.18:34
*** ddieterly has joined #openstack-infra18:34
EmilienMpabelanger: ok. +118:34
anteayaRemy: we haven't started testing gerrit 2.12 yet, so yes would be interested in your experience with the 2.12 and zuul interaction18:35
pabelangeranteaya: Ya, I was being cautious as not sure members we interested in speaking at conferences outside of OpenStack summit18:35
pabelangerEmilienM: as a clean up, we could create a job-template specific to tripleo things18:35
anteayaoh okay, caution can be a good thing18:35
anteayaRemy: do you have more build log than that?18:36
anteayaRemy: what was zuul working on when it encountered that error?18:36
EmilienMpabelanger: ++18:36
anteayaRemy: build log/log18:37
Remyanteaya: I do not have the full logs, as they were on my test machine. I could try to repro. Basically what I was just doing was testing that the gate jobs would work fine.18:37
RemyThey did get triggered, and built, but gerrit never received the information back18:38
anteayawe would be interested in full logs of that experience if you could reproduce, thank you18:38
anteayaif zuul and 2.12 are unhappy together we also would like to get to the bottom of the situation18:39
Remyoki doki, I'm going to rebuild my test env, and provide you with more info. Thank you :)18:39
anteayaRemy: thanks so much18:39
ttxflwang1: o/18:40
*** salv-orlando has joined #openstack-infra18:40
flwang1ttx: i just replied your comments on https://review.openstack.org/#/c/310317/118:40
flwang1ttx: but i'm not really sure if i followed18:40
clarkbRemy: anteaya it would also be good if you could manually run such an ssh command to see how it fails18:41
ttxflwang1: it's core reviewers reviews18:41
clarkbits possible you just don't have a verify category and it explodes18:41
flwang1ttx: can you let me know which rule is requiring that? http://governance.openstack.org/reference/tags/team_diverse-affiliation.html#requirements18:41
anteayapabelanger: so pyconau talk submission proposal is closing soon, do you want to start an infra thread to see if anyone has or is going to submit a talk about infra tools?18:41
ttxthe sum of all reviews done by core reviewers against patches submitted to any of the git repositories managed by the team in mitaka is 83% RedHat+CatalystIT18:42
*** d34dh0r53 is now known as m1r4nt15_b0y18:42
Remyclarkb: this I have18:42
*** chem has quit IRC18:42
anteayapabelanger: https://2016.pycon-au.org/programme/call_for_proposals closes the 8th of May18:42
ttxflwang1: ^18:42
*** hdd has joined #openstack-infra18:42
*** marcusvrn_ has quit IRC18:43
flwang1ttx: do you mean the reviews counts reviewed by core reviewers from 2 companies    vs.    reviews by the whole core reviewers team ?18:43
ttxyes, 2 companies represent 83% of all reviews by core developers18:43
flwang1the sum of all reviews done by core reviewers against patches submitted to any of the git repositories managed by the team in mitaka,  i can see your point. but then i question is how to define the 'team'18:44
ttxwhich is hardly surprising given that you represent 80% of active core reviewers18:44
pabelangeranteaya: I just see pleia2 link https://etherpad.openstack.org/p/infra-conferences-2016 on a follow-up email.  What do you think of listing it there?18:44
Remyclarkb: anteaya : here is the result I got from running the SSH command manually: http://pastebin.com/iTGNRkrF18:44
flwang1i don't think it's a 'core team' but a general team, maybe i'm wrong18:44
ttxthe team is the Zaqar project team18:44
anteayapabelanger: even better18:44
rcarrillocruzfolks, not sure if i'll be for infra meeting, have yet to get my son asleep18:44
*** nadya has quit IRC18:45
*** salv-orlando has quit IRC18:45
anteayarcarrillocruz: understood, thanks for the update18:45
ttxflwang1: the rule is "no two organizations should represent >80% of the sum of all reviews done by core reviewers against patches submitted to any of the git repositories managed by the team"18:45
anteayarcarrillocruz: hopefully he will drift off before the end of the meeting :)18:46
clarkbRemy: I think those errors are due to ssh escaping18:46
clarkbRemy: try wrapping the entire gerrit command in ''s18:46
ttxI agree it could be clearer, but that is meant as a core review share -- as exemplified by the testing script18:46
Remyclarkb: ok, I'm going to try that18:46
*** javeriak has quit IRC18:46
Remythank you18:46
flwang1ttx: ok, thanks for the clarification. i will abandon the patch for now18:47
ttx<80% of commits, reviews, core reviews and core reviewers basically18:47
ttxflwang1: note that if you add a new core that may just help18:48
ttxsince the RAX ones are a bit inactive18:48
*** salv-orlando has joined #openstack-infra18:48
*** salv-orlando has quit IRC18:48
flwang1ttx: yep, i see. i'm encouraging the guys from huawei to contribute more18:48
*** EricGonczer_ has joined #openstack-infra18:49
flwang1ttx: again, many thanks18:49
ttxsince you seem to fill everything else (are at 80% on the core reviewer criteria)18:49
*** hdd has quit IRC18:50
*** roxanaghe has quit IRC18:50
*** armax has quit IRC18:51
*** salv-orlando has joined #openstack-infra18:51
yolandaclarkb, it's mine yes, i'm using that to test magnum+shade integration18:51
*** baoli has quit IRC18:52
*** roxanaghe has joined #openstack-infra18:52
*** hashar has joined #openstack-infra18:53
*** dprince has quit IRC18:53
*** hashar has quit IRC18:53
*** hashar has joined #openstack-infra18:53
*** salv-orlando has quit IRC18:54
clarkbyolanda: can you delete it and recreate in the jenkins account instead of the ci account?18:54
clarkbyolanda: the issue is that the ci accounts have their hosts added to ansible inventory and that host causes ansible to sit and wiat for forever18:54
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Define job-groups for api-ref and api-guide  https://review.openstack.org/31219818:56
*** baoli has joined #openstack-infra18:56
anteayajhesketh: morning18:56
*** yolanda has quit IRC18:56
*** kgiusti has quit IRC18:57
*** links has quit IRC18:58
*** doug-fis_ has joined #openstack-infra18:58
*** baoli has quit IRC18:58
*** armax has joined #openstack-infra18:59
pabelangeranteaya: added into about pycon to the etherpad18:59
*** banix has quit IRC18:59
*** doug-fi__ has joined #openstack-infra19:00
anteayapabelanger: great, thank you19:00
*** EricGonczer_ has quit IRC19:00
pleia2meeting time19:00
clarkbugh already?19:00
*** kgiusti has joined #openstack-infra19:02
*** doug-fish has quit IRC19:02
*** doug-fis_ has quit IRC19:03
*** julim has quit IRC19:04
*** doug-fi__ has quit IRC19:04
anteayaclarkb: did you add pick a week for the operating system control plane upgrade sprint?19:04
clarkbno, busy with more pressing things currently19:04
anteayaI understand19:04
anteayaif there is open discussion I'll try to introduce it there19:05
anteayaunless you would like to?19:05
clarkbya thats fine19:05
*** baoli has joined #openstack-infra19:05
anteayagreat, thanks19:05
openstackgerritBen Nemec proposed openstack-infra/project-config: Add stable branch jobs for tripleo-ci  https://review.openstack.org/29739019:06
*** kzaitsev_mb has quit IRC19:06
*** julim has joined #openstack-infra19:07
*** bpokorny_ has joined #openstack-infra19:12
pabelangerianw: mind giving me a quick rundown on break=after-error for diskimage-builder? Is is a simple as export break=after-error ?19:14
*** EricGonczer_ has joined #openstack-infra19:16
*** ddieterly is now known as ddieterly[away]19:16
*** bpokorny has quit IRC19:16
*** vhosakot_ has quit IRC19:16
*** gyee has quit IRC19:17
*** jamesmcarthur has joined #openstack-infra19:20
*** banix has joined #openstack-infra19:20
*** maishsk has joined #openstack-infra19:21
*** ddieterly[away] is now known as ddieterly19:21
openstackgerritMerged openstack-infra/jenkins-job-builder: Compatibility support for older maven project plugins  https://review.openstack.org/27419519:21
*** matrohon has joined #openstack-infra19:23
*** Sukhdev has joined #openstack-infra19:23
timothyb89pabelanger: looks like the stackviz patch is running correctly on Ubuntu and Fedora, log output in a comment here: https://review.openstack.org/#/c/311815/119:25
*** roxanaghe has quit IRC19:28
openstackgerritJohn Trowbridge proposed openstack-infra/tripleo-ci: Fix overcloud pingtest for stable branches  https://review.openstack.org/31221219:30
*** EricGonc_ has joined #openstack-infra19:30
EmilienMI'm not sure infra can help, but let's try: I'm currently testing Ubuntu Xenial images and I'm having SSL troubles: http://logs.openstack.org/30/308530/6/experimental/gate-puppet-openstack-integration-3-scenario002-tempest-ubuntu-xenial/d91c189/console.html#_2016-05-03_12_09_55_464 - see [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)19:30
EmilienMpackages in trusty: http://logs.openstack.org/30/308530/6/check/gate-puppet-openstack-integration-3-scenario003-tempest-ubuntu-trusty/8d287e1/logs/dpkg-l.txt.gz19:31
EmilienMand packages in xenial: http://logs.openstack.org/30/308530/6/experimental/gate-puppet-openstack-integration-3-scenario002-tempest-ubuntu-xenial/d91c189/logs/dpkg-l.txt.gz19:31
anteayaEmilienM: there was an openssl cve today, I'm not sure if that is related or not19:31
clarkbEmilienM: I would check if xenial python requests package honors your cert list19:31
mordredmorgan: ^^19:31
mordredwhat clarkb just said19:31
clarkbbecause upstream requests does not19:31
EmilienManteaya: not related, I'm having this issue for weeks19:31
clarkband if ubuntu stopped patching that then well bad things19:31
morganmordred: oh hi?19:31
mordredthe distro packaged requests usually patch the part of requests that ignores system certs19:31
anteayaEmilienM: okay thanks19:31
EmilienManteaya: thanks!19:32
morganmordred: correct19:32
EmilienMclarkb, morgan: ok I'll check that. Thanks a lot19:32
EmilienMand mordred -> thx19:32
morganEmilienM: there is a env that can be set to point at a real CA bundle.19:32
EmilienMin trustry I have 2.9.1-3~cloud0 and in xenial 2.9.1-319:33
morganEmilienM: i also know they are really looking to honor system-cas properly in... one of the upcoming releases.19:33
morganof requests19:33
morgani wonder if the xenial just ended up unpatched :(19:33
EmilienMI need to download the .deb and compare :-)19:34
EmilienMmorgan: could it be in python-requestsexceptions ? or just python-requests ?19:34
* EmilienM goes find .deb files19:35
morganEmilienM: i would be surprised if it was unpatched.19:35
clarkbya another possibilitiy is the system cert management has hcnaged19:36
*** sballe_ has left #openstack-infra19:36
StevenKDoes http://changelogs.ubuntu.com/changelogs/pool/main/r/requests/requests_2.9.1-3/changelog help?19:36
EmilienMyeah, we use /usr/sbin/update-ca-certificates -f to update ca in trusty19:36
EmilienMStevenK: the thing is trusty & xenial have both the same version, 2.9.1-319:37
morganclarkb: true.19:37
EmilienMStevenK: trusty has 2.9.1-3~cloud0 and xenial 2.9.1-319:38
*** Remy has quit IRC19:38
pabelangertimothyb89: the seem to be truncated19:39
*** ericksonsantos has quit IRC19:39
StevenKEmilienM: Do you have both deb19:39
StevenKEmilienM: Do you have both deb's locally?19:39
EmilienMStevenK: not yet19:39
StevenKI suspect ~cloud0 is from UCA19:40
*** RickA-HP has joined #openstack-infra19:40
RickA-HPFolks. We are looking at adding Tempest tests to the Searchlight service. Is this allowed, having a non-core service adding Tempest tests?19:41
EmilienMmorgan, StevenK: so in xenial package, they still have patches19:41
RickA-HP@AJaeger @anteaya ^^^^^19:41
morganEmilienM: yeah.19:41
StevenKEmilienM: Is there a series file?19:42
*** trey has quit IRC19:42
*** matrohon has quit IRC19:42
*** banix has quit IRC19:42
*** EricGonczer_ has quit IRC19:42
*** ddieterly has quit IRC19:42
*** jerryz has quit IRC19:42
*** gordc has quit IRC19:42
*** asselin__ has quit IRC19:42
*** degorenko has quit IRC19:42
*** MarkAtwood has quit IRC19:42
*** jlk has quit IRC19:42
*** pblaho has quit IRC19:42
*** david-lyle has quit IRC19:42
*** bookwar_ has quit IRC19:42
*** zul has quit IRC19:42
*** dangers_away has quit IRC19:42
*** cfarquhar has quit IRC19:42
*** ajmiller has quit IRC19:42
*** otherwiseguy has quit IRC19:42
*** elo has quit IRC19:42
*** clsacramento has quit IRC19:42
*** trown has quit IRC19:42
*** timothyb89 has quit IRC19:42
*** rvba has quit IRC19:42
*** lyz has quit IRC19:42
*** Douhet has quit IRC19:42
*** Kiall has quit IRC19:42
*** skraynev has quit IRC19:42
*** gmann has quit IRC19:42
*** pcm_ has quit IRC19:42
*** gabriel-bezerra has quit IRC19:42
*** thingee has quit IRC19:42
*** akshai has quit IRC19:42
*** flwang1 has quit IRC19:42
*** mriedem has quit IRC19:42
*** sputnik13 has quit IRC19:42
*** keedya has quit IRC19:42
*** mtanino has quit IRC19:42
*** docaedo2 has quit IRC19:42
*** markmcd1 has quit IRC19:42
*** markmcd has quit IRC19:42
*** r-mibu has quit IRC19:42
*** esker has quit IRC19:42
*** burgerk has quit IRC19:42
*** eharney has quit IRC19:42
*** zz_ja has quit IRC19:42
*** ccamacho|away has quit IRC19:42
*** thorst has quit IRC19:42
*** jogo has quit IRC19:42
*** d0ugal has quit IRC19:42
*** binbincong has quit IRC19:42
*** Clint has quit IRC19:42
*** aarefiev has quit IRC19:42
*** netsin has quit IRC19:42
*** fungi has quit IRC19:42
*** kaisers has quit IRC19:42
*** LinuxJedi has quit IRC19:42
*** hughhalf has quit IRC19:42
*** whoops has quit IRC19:42
*** |GonZo2000| has quit IRC19:42
*** lennyb has quit IRC19:42
*** hrubi has quit IRC19:42
*** bkeller` has quit IRC19:42
*** persia has quit IRC19:42
*** rhe00 has quit IRC19:42
*** bexelbie has quit IRC19:42
*** crinkle has quit IRC19:42
*** uberjay_ has quit IRC19:42
*** Vivek has quit IRC19:42
*** kevinbenton has quit IRC19:42
*** kevinshort has quit IRC19:42
*** jgriffith has quit IRC19:42
*** adreznec has quit IRC19:42
*** michaelxin has quit IRC19:42
*** andreaf has quit IRC19:42
*** markmcclain has quit IRC19:42
*** tobiash has quit IRC19:42
StevenKEmilienM: The existance of a patch is not enough, it also needs to be listed in the series19:42
*** jamesmcarthur has quit IRC19:42
EmilienMStevenK: it's listed19:42
morganEmilienM: hmmmmm.19:42
*** jamesmcarthur has joined #openstack-infra19:43
StevenKI've debdiff'd the two python-requests, and the ony real difference is the Depends line19:43
StevenKSo maybe the issue isn't requests, but the CA bundle19:44
EmilienMxenial: ca-certificates = 20160104ubuntu1 / trusty is 20160104ubuntu0.14.04.119:44
openstackgerritDan Smith proposed openstack-infra/devstack-gate: WIP: Start up nova-api on the subnode during multinode upgrades  https://review.openstack.org/31222119:44
EmilienMthey look similar19:44
StevenKI daresay they are pretty much identical except for changelog entries and maybe their depends19:46
*** ericksonsantos has joined #openstack-infra19:47
EmilienMjamespage: hey, before the netsplit we were talking about SSL issues on xenial19:47
*** yolanda has joined #openstack-infra19:47
EmilienMjamespage: http://logs.openstack.org/30/308530/6/experimental/gate-puppet-openstack-integration-3-scenario002-tempest-ubuntu-xenial/d91c189/console.html#_2016-05-03_12_09_55_46419:47
EmilienMjamespage: we were wondering if something changed in python-request or maybe on the way to deploy ca bundles, right now we use /usr/sbin/update-ca-certificates19:48
*** Swami has joined #openstack-infra19:48
*** baoli has quit IRC19:49
*** phschwartz_ is now known as phschwartz19:49
*** baoli has joined #openstack-infra19:50
morganEmilienM: hm. i'll look at that a little more in depth after food if you haven't found it out19:50
jamespageEmilienM, StevenK: the requests in UCA and Xenial should be the same source package apart from the changlog entry19:50
clarkbEmilienM: the other thing to check is if openstack client is using system requests or pypi requests19:50
*** dmk0202 has joined #openstack-infra19:51
morganclarkb: i expect it's using pypi19:51
*** david-lyle has joined #openstack-infra19:51
*** akshai has joined #openstack-infra19:51
*** mriedem has joined #openstack-infra19:51
*** docaedo2 has joined #openstack-infra19:51
*** sputnik13 has joined #openstack-infra19:51
*** keedya has joined #openstack-infra19:51
*** mtanino has joined #openstack-infra19:51
*** markmcd1 has joined #openstack-infra19:51
*** markmcd has joined #openstack-infra19:51
*** r-mibu has joined #openstack-infra19:51
*** esker has joined #openstack-infra19:51
*** burgerk has joined #openstack-infra19:51
*** eharney has joined #openstack-infra19:51
*** zz_ja has joined #openstack-infra19:51
*** ccamacho|away has joined #openstack-infra19:51
*** thorst has joined #openstack-infra19:51
*** jogo has joined #openstack-infra19:51
*** d0ugal has joined #openstack-infra19:51
*** binbincong has joined #openstack-infra19:51
*** Clint has joined #openstack-infra19:51
*** aarefiev has joined #openstack-infra19:51
*** netsin has joined #openstack-infra19:51
*** fungi has joined #openstack-infra19:51
*** kaisers has joined #openstack-infra19:51
*** LinuxJedi has joined #openstack-infra19:51
*** hughhalf has joined #openstack-infra19:51
*** whoops has joined #openstack-infra19:51
*** |GonZo2000| has joined #openstack-infra19:51
*** lennyb has joined #openstack-infra19:51
*** hrubi has joined #openstack-infra19:51
*** rhe00 has joined #openstack-infra19:51
*** bkeller` has joined #openstack-infra19:51
*** bexelbie has joined #openstack-infra19:51
*** persia has joined #openstack-infra19:51
*** crinkle has joined #openstack-infra19:51
*** uberjay_ has joined #openstack-infra19:51
*** Vivek has joined #openstack-infra19:51
*** kevinbenton has joined #openstack-infra19:51
*** jgriffith has joined #openstack-infra19:51
*** kevinshort has joined #openstack-infra19:51
*** adreznec has joined #openstack-infra19:51
*** michaelxin has joined #openstack-infra19:51
*** andreaf has joined #openstack-infra19:51
*** markmcclain has joined #openstack-infra19:51
*** tobiash has joined #openstack-infra19:51
*** thingee has joined #openstack-infra19:51
*** banix has joined #openstack-infra19:51
*** ddieterly has joined #openstack-infra19:51
*** jerryz has joined #openstack-infra19:51
*** gordc has joined #openstack-infra19:51
*** asselin__ has joined #openstack-infra19:51
*** degorenko has joined #openstack-infra19:51
*** MarkAtwood has joined #openstack-infra19:51
*** jlk has joined #openstack-infra19:51
*** pblaho has joined #openstack-infra19:51
*** bookwar_ has joined #openstack-infra19:51
*** zul has joined #openstack-infra19:51
*** dangers_away has joined #openstack-infra19:51
*** cfarquhar has joined #openstack-infra19:51
*** ajmiller has joined #openstack-infra19:51
*** otherwiseguy has joined #openstack-infra19:51
*** elo has joined #openstack-infra19:51
*** clsacramento has joined #openstack-infra19:51
*** trown has joined #openstack-infra19:51
*** timothyb89 has joined #openstack-infra19:51
*** rvba has joined #openstack-infra19:51
*** lyz has joined #openstack-infra19:51
*** Douhet has joined #openstack-infra19:51
*** Kiall has joined #openstack-infra19:51
*** skraynev has joined #openstack-infra19:51
*** gmann has joined #openstack-infra19:51
*** pcm_ has joined #openstack-infra19:51
*** gabriel-bezerra has joined #openstack-infra19:51
StevenKjamespage: debdiff reports no real changes of substance, at least19:51
jamespageStevenK, the resulting binaries might diff a bit on dependencies...19:52
*** Goneri has quit IRC19:52
jamespageas those are generated to some extent during the package build19:52
*** matrohon has joined #openstack-infra19:53
EmilienMclarkb, morgan: this is not devstack runs, I'm using packaging provided by Ubuntu Xenial19:53
morganEmilienM: oh. so no pip install ?19:53
EmilienMmaybe something does pip install, but not in Puppet CI19:54
EmilienMmaybe during the image build19:54
morganEmilienM: hmm.. then something is wonky with the system CA and how it's being referenced.19:54
*** fbo has quit IRC19:54
morgani just looked through the logs, no pip afaict19:54
EmilienMnot in the puppet logs, indeed, maybe in the xenial image build19:54
EmilienMlet me look dib19:54
jamespageEmilienM, is this a diff between trusty/mitaka and xenial/mitaka?19:55
*** ddieterly is now known as ddieterly[away]19:55
mordredjamespage: yah19:55
clarkbEmilienM: we should confine that to virtualenvs in the image builds19:55
EmilienMjamespage: we don't setup UCA on Xenial, that's all about it19:56
*** trey has joined #openstack-infra19:56
jamespageEmilienM, yes - that correct - no UCA for Xenial yet...19:56
jamespagemitaka is all just in distro19:56
EmilienMeverything else is the same, except the nodepool image obviously19:56
*** yolanda has quit IRC19:56
*** fbo has joined #openstack-infra19:57
mgagnemordred: any plan on supporting bonding in glean?19:58
mordredmgagne: I believe someone has written some initial patches?19:58
mgagnemordred: will do my homework and check review queue19:58
jlvillalgreghaynes: I remember a year or so ago you were talking about a decent priced colo facility in Portland. Do you remember the name of it?19:58
mordredmgagne: but yeah, anything that's in the neutron network-info structure should ultimately be supported in glean19:58
greghaynesjlvillal: opus interactive19:58
jlvillalgreghaynes: Thanks :)19:59
mgagnemordred: I'm about to defenestrate cloud-init19:59
EmilienMI thought defenestrate was french19:59
EmilienMneat, I can also use it in english19:59
jamespageEmilienM, guessing your module a bit here - you're autogenerating self-signed certs and injecting the CA to the host by writing it to /etc/ssl/certs and running update-ca-certificates -f or suchlike?19:59
mordredmgagne: \o/19:59
mtreinishEmilienM: it's a great word :)19:59
mgagnerare: throw (someone) out of a window.20:00
anteayaEmilienM: it is a great word20:00
anteayamtreinish: jinx20:00
*** kgiusti1 has joined #openstack-infra20:00
EmilienMjamespage: exactly!20:00
pleia2fungi: you'll enjoy our parallel universe meeting log, good luck with the copy/pasta interleaving :)20:00
nibalizerthats so funny to me that we had double meetings20:00
anteayafungi: I want to read your ramblings20:00
*** nadya has joined #openstack-infra20:00
EmilienMjamespage: we inject the file like this: https://github.com/openstack/puppet-openstack-integration/blob/master/manifests/cacert.pp20:00
nibalizerscalable infra team right20:00
nibalizerfault tollerabt20:00
nibalizerall the jokes20:01
anteayaha  ha ha20:01
*** kgiusti1 has quit IRC20:01
*** kgiusti1 has joined #openstack-infra20:01
pleia2HA meeting chair20:01
EmilienMjamespage: with the paths: https://github.com/openstack/puppet-openstack-integration/blob/master/manifests/params.pp20:01
fungipleia2: i may just keep your meeting and try to integrate my updates into an e-mail for the list20:01
fungithanks for taking over when i ended up in the quiet universe20:01
pleia2sure thing20:01
anteayafungi: was anyone with you?20:01
* crinkle was20:01
anteayafungi: did you have any company?20:01
anteayacrinkle: nice20:01
*** nadya has quit IRC20:01
*** EricGonc_ has quit IRC20:02
*** nadya has joined #openstack-infra20:02
*** kgiusti has quit IRC20:02
EmilienMjamespage: so I put the crt in /usr/local/share/ca-certificates/puppet_openstack.crt and run /usr/sbin/update-ca-certificates -f20:02
jamespageEmilienM, that's what I would do (and indeed what the openstack charms do as well0...20:03
jamespagestill poking...20:03
*** sridhar_ram1 has joined #openstack-infra20:03
fungianteaya: i was preaching to a room of 40 there for a while. it was probably me, crinkle, persia and 37 other people who weren't watching20:03
anteayaha ha ha20:03
EmilienMI'm going to defenestrate SSL20:03
anteayaso that's how you get to have a quiet meeting20:03
anteayaI've learned that defenstranation is good for starting wars20:03
*** sridhar_ram has quit IRC20:04
* fungi notes to schedule more netsplits in the future20:04
anteayahe he he20:04
pabelangerEmilienM: looking at http://nodepool.openstack.org/dib.ubuntu-xenial.log I don't see that we install python-requests20:04
*** shardy has quit IRC20:04
pabelangerEmilienM: but trusty we do: http://nodepool.openstack.org/dib.ubuntu-trusty.log20:04
*** baoli has quit IRC20:04
pabelangerEmilienM: so a simple test would be to apt-get install python-requests in your run_tests.sh script20:04
pabelangerEmilienM: and see what happens20:04
*** baoli has joined #openstack-infra20:04
*** esker has quit IRC20:05
mtreinishmordred: so on the db stuff and moving off of trove, is there anything I can do to help get things started?20:05
*** rockyg has joined #openstack-infra20:05
pabelangerI should note python-requests-whl is an extra package for trusty, perhaps the dependency has been broken for xenial20:05
EmilienMpabelanger: I can try it, the thing is the result is we have the same version on both trusty & xenial20:05
pabelangerEmilienM: we likely need to ensure it is added to bindep-fallback.txt if we expect it to be installed20:06
*** amitgandhinz has quit IRC20:06
*** Sukhdev has quit IRC20:07
*** doug-fish has joined #openstack-infra20:08
*** doug-fish has quit IRC20:08
pabelangerEmilienM: actually, looks like I am wrong. We are only downloading it, not installing it20:08
pabelangerwith trusty20:08
StevenKpabelanger: python-requests-whl was dropped on purpose during Xenial's cycle20:09
pabelangerStevenK: okay, so that explains it missing from xenial20:10
mordredmtreinish: I actually started poking at it a little yesterday20:10
*** jamesmcarthur has quit IRC20:10
mtreinishmordred: oh, cool20:10
*** amrith is now known as _amrith_20:12
*** esker has joined #openstack-infra20:13
*** hashar has quit IRC20:15
EmilienMdoes anyone tested xenial nodes already?20:16
*** hdd has joined #openstack-infra20:17
mgagneEmilienM: I'm trying to make it respond to ping on our cloud :D20:18
fungiEmilienM: some experimental jobs and, as of a few hours ago, now infra20:18
*** ddieterly[away] is now known as ddieterly20:18
EmilienMfungi: does something test openstack/ssl?20:19
fungiEmilienM: i don't think devstack/tempest is working on xenial yet20:19
clarkbfungi: EmilienM devstack/tempest is working on xenial but they don't ssl anything20:20
EmilienMok so i'm an early tester maybe20:20
EmilienMah, we ssl & ipv620:20
*** ddieterly is now known as ddieterly[away]20:20
fungioh, closer than i thought already20:20
*** ccamacho|away has quit IRC20:21
fungipabelanger: we're preinstalling the python-requests distro package on ubuntu-trusty again?!?20:22
*** amitgandhinz has joined #openstack-infra20:22
pabelangerfungi: no, I was confused20:22
* fungi will pretend not to have seen that. can only focus on troubleshooting one broken thing at a time now20:22
pabelangerfungi: we are only caching it20:22
fungioh, caching. good *whew*20:23
*** ccamacho has joined #openstack-infra20:24
*** banix has quit IRC20:24
fungiwe've hit a brief lull on the crazy openssl emergency upgrades. i'm assuming nobody's started to look into the write errors uploading to tarballs.o.o yet?20:24
*** matrohon has quit IRC20:24
*** kgiusti has joined #openstack-infra20:24
anteayafungi: yay lull20:24
fungii guess not as i'm still the only one logged into it20:25
anteayafungi: I haven't seen anyone say they are looking at tarballs20:25
clarkb#status notice restarting apache on review.openstack.org to pick up security patches. Gerrit web ui may disappear for a short time.20:25
clarkbdid that noop because I am not auth'd with nickserv?20:25
*** fitoduarte has joined #openstack-infra20:26
anteayathe bot is in channel20:26
openstackgerritTim Laszlo proposed openstack-infra/shade: Add support for server groups  https://review.openstack.org/31223720:26
clarkb#status notice restarting apache on review.openstack.org to pick up security patches. Gerrit web ui may disappear for a short time.20:26
openstackstatusclarkb: sending notice20:26
StevenKclarkb: Blah, you should switch to using SSL certs to auth to services20:26
fungilrwxrwxrwx 1 root root 7 Jun 30  2015 /dev/mapper/main-tarballs -> ../dm-220:26
fungi[Tue May  3 04:37:01 2016] EXT4-fs (dm-2): Remounting filesystem read-only20:27
*** kgiusti1 has quit IRC20:27
fungile sigh20:27
-openstackstatus- NOTICE: restarting apache on review.openstack.org to pick up security patches. Gerrit web ui may disappear for a short time.20:27
*** matrohon has joined #openstack-infra20:27
clarkbjpmaxman: hey, we will also need to do the same for apache on openstackid, is it fine to go ahead and just do that?20:27
anteayafungi: so that was static and logging and tarballs then?20:28
openstackstatusclarkb: finished sending notice20:29
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add wget to puppet element  https://review.openstack.org/31223920:29
pabelangerfungi: clarkb: Looks like we need to add wget to the puppet element for dib ^20:29
*** kgiusti has quit IRC20:31
openstackgerritEmilien Macchi proposed openstack-infra/project-config: Implement periodic job to update Puppet OpenStack constraints  https://review.openstack.org/30137520:31
*** annegentle has quit IRC20:31
EmilienMAJaeger: I addressed all your comments, even if you +2-ed, so we're clean20:31
*** e0ne has joined #openstack-infra20:34
jpmaxmanfungi: I think it should be fine let me just double check20:35
clarkbjpmaxman: ya I got confirmnation out of band20:35
*** nadya has quit IRC20:35
clarkbjpmaxman: from smarcet and jimmy20:35
jpmaxmanok cool20:36
clarkbjpmaxman: it is all done now, was a quick one20:36
jpmaxmanI feel secure again20:36
fungi#status log through some careful scripting (which involved apache reconfiguration to stop holding an open file lock) i offlined the tarballs volume on static.openstack.org to repair its filesystem so it could be remounted read-write20:38
openstackstatusfungi: finished logging20:38
*** nadya has joined #openstack-infra20:40
*** ddieterly[away] is now known as ddieterly20:41
*** doug-fish has joined #openstack-infra20:42
*** Sukhdev has joined #openstack-infra20:42
*** bigjools has quit IRC20:43
*** salv-orlando has joined #openstack-infra20:44
*** bigjools has joined #openstack-infra20:45
*** doug-fish has quit IRC20:46
*** tpsilva has quit IRC20:47
*** gyee has joined #openstack-infra20:47
fungiwaynr: looks like electrofelix has dropped off, but now that i've got the tarballs site writeable again i've retriggered the jjb 1.5.0 release jobs and it's updated on pypi now https://pypi.python.org/pypi/jenkins-job-builder20:47
asselin_hey, i'm seeing an issue in our ci where zuul says  build:check-tempest-dsvm-neutron-full   0       0       2, but the zuul status page shows a few jobs with that name queued up. Any idea how to reset the state?20:48
clarkbasselin_: offline and online a node that can run that job or enable disable that job20:49
*** salv-orlando has quit IRC20:50
fungiasselin_: odds are jenkins has gotten itself in a state where it's not picking up jobs20:50
*** vgridnev has joined #openstack-infra20:50
fungiasselin_: oh, thing to check though, it it stuck on the merger work request?20:50
fungii think the pending jobs could would be nonzero unless somehow zuul got disconnected from its gearman subprocess20:51
*** Sukhdev has quit IRC20:51
asselin_taking one offline does update the 3rd number: build:check-tempest-dsvm-neutron-full   0       0       120:52
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add ubuntu-precise dib to nodepool  https://review.openstack.org/31224820:52
asselin_but bringing it back online doesn't help: build:check-tempest-dsvm-neutron-full:devstack-trusty   0       0       220:52
fungiasselin_: i'm betting something has caused your zuul-merger worker not to pick up merger work requests20:53
fungiis the zuul-merger daemon running?20:53
*** Swami has quit IRC20:53
fungido you see a nonzero value in the third column for merger jobs?20:53
fungithat's where your merger workers would be registered in gearman20:54
asselin_merger:update   16      0       120:54
asselin_merger:merge    1959    1       120:54
*** salv-orlando has joined #openstack-infra20:54
*** baoli has quit IRC20:54
fungiyeah, looks like you're backed up on merger work requests for sure20:54
*** lucas-dinner has quit IRC20:54
fungii'd check the debug log for zuul-merger20:55
fungisomething has likely made it not happy20:55
asselin_I see.....20:55
pabelangerI think the vos release for mirror.pypi might be finished20:55
asselin_someone submitted a huge patch chain to a repo, and abandoned it. Zuul merger is still working on it though.20:56
fungidims: ^ you may want to double-check the things20:56
pabelangerI think it was run on afs01.dfw.openstack.org20:56
asselin_I guess restart the zuul merger is the best way to skip that?20:56
pabelangerbut no longer see the process20:56
*** aysyd has quit IRC20:56
fungiwow. there's a fun new way to gum up a zuul20:56
pabelangervos examine mirror.pypi --verbose does show the mirror out of sync still20:57
fungiasselin_: i think restarting the merger won't help, because gearman still holds the queue for it20:57
dimsfungi : hmmm what did i break?20:57
fungiasselin_: you may need to export your changes from the pipelines you care about, restart zuul, and then reenqueue the exported list20:57
asselin_yeah....avoiding that, but may not have a choice.....20:58
*** lucasagomes has joined #openstack-infra20:58
pabelangerfungi: jeblair: I think we need another vos release manually?20:58
fungidims: pabelanger was saying the long-running pypi mirror sync may be done. just giving you a heads up in case you wanted to double check the things you knew were missing20:58
dimsfungi : ah. thanks. will check on it in a bit20:58
dimsthanks for the heads up. pabelanger and fungi20:59
fungidims: well, it sounds like maybe it's still not done after all20:59
dimsno worries20:59
pabelangerYa, we should see what jeblair says20:59
fungipabelanger: there's a root screen session on that server i just joined21:00
fungiRelease failed: VOLSER: Problems encountered in doing the dump !21:00
pabelangerfungi: ya, I haven't checked it yet21:00
*** rockyg has quit IRC21:01
fungii can start it back up again21:01
fungiit ran for 952m39.949s before erroring21:01
*** Sukhdev has joined #openstack-infra21:02
pabelangerTue May  3 13:39:28 2016 1 Volser: ReadVnodes: IH_CREATE: File exists - restore aborted21:02
pabelangerI see that in VolserLog on ord21:02
*** matrohon has quit IRC21:03
fungithe last vos release was started at 21:46:48 utc yesterday, so that's probably 952m40s later21:05
*** trown is now known as trown|outtypewww21:06
*** doug-fish has joined #openstack-infra21:07
*** Sukhdev has quit IRC21:07
*** doug-fis_ has joined #openstack-infra21:08
*** mylu has joined #openstack-infra21:09
fungii see discussions of using salvage in cases like this, but am hesitant and would welcome a jeblair or mordred opinion if one of them is around21:09
*** doug-fi__ has joined #openstack-infra21:09
* mordred is reading scrollback21:09
*** Swami has joined #openstack-infra21:09
mordredfungi: before I try to figure out from context ...21:10
*** mylu has quit IRC21:10
mordredthe zuul problem and the afs problem are distinct, yes?21:10
fungimordred: yep, merely interleaved21:10
*** Sukhdev has joined #openstack-infra21:10
*** psilvad has quit IRC21:11
fungiasselin_ was (is?) having an issue with his zuul server choked on a bunch of abandoned changes. completely unrelated to the afs discussion21:11
*** doug-fish has quit IRC21:11
mordredmy brain was not processing scrollback well21:12
fungithe afs question is how to get past vos release aborting with a "file exists" error on the destination server21:12
mordredyah. I think the best plan at this point is to wait on jeblair21:12
*** doug-fis_ has quit IRC21:13
*** e0ne has quit IRC21:13
fungii'm browsing the salvager manpage but really leery of making the situation worse without knowing what i'm doing21:13
openstackgerritJim Rollenhagen proposed openstack-infra/project-config: Add api-ref jobs for ironic  https://review.openstack.org/31218421:13
waynrfungi: sweet!21:14
*** maishsk_ has joined #openstack-infra21:14
*** maishsk has quit IRC21:15
*** maishsk_ is now known as maishsk21:15
*** mylu has joined #openstack-infra21:15
*** ldnunes has quit IRC21:15
*** thorst has quit IRC21:16
openstackgerritHenrique Truta proposed openstack-infra/project-config: Run identity v3 only job in ironic  https://review.openstack.org/30430721:17
*** vgridnev has quit IRC21:17
*** thorst has joined #openstack-infra21:18
fungithis is most strange. it's been several days since the last error from a job trying to hit git.o.o21:19
fungii wonder if the network issues in rackspace magically vanished?21:20
*** lucasagomes has quit IRC21:20
*** tonytan4ever has quit IRC21:21
*** lucasagomes has joined #openstack-infra21:22
*** thorst has quit IRC21:22
pabelangerGood news everybody. I was able to build ubuntu-precise with our puppet element. Now for a full build with all the elements nodepool needs.21:23
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add ubuntu-precise dib to nodepool  https://review.openstack.org/31224821:23
pabelangercurrent stack21:23
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add ubuntu-precise dib to nodepool  https://review.openstack.org/31224821:24
*** flwang1 has joined #openstack-infra21:25
*** |-paul-| has joined #openstack-infra21:25
*** nadya has quit IRC21:25
*** nadya has joined #openstack-infra21:26
*** lakshmiS has quit IRC21:26
*** asettle has joined #openstack-infra21:27
*** burgerk has quit IRC21:30
*** _ari_ has quit IRC21:30
*** jsavak has joined #openstack-infra21:30
*** julim has quit IRC21:30
*** nadya has quit IRC21:30
mriedemclarkb: looks like we might have some issues with ES shards stuck initializing? http://logstash.openstack.org/elasticsearch/_cluster/health?pretty=true21:31
*** maishsk has quit IRC21:31
*** ccamacho has quit IRC21:31
clarkbmriedem: ya te cluster deadlocks and is unable to go green, plan is to pgrade which i am told makes it better21:31
*** sdake_ is now known as sdake21:33
*** e0ne has joined #openstack-infra21:34
*** pfallenop has joined #openstack-infra21:35
fungii'm going to go find some dinner and transition into a (hopefully) quieter evening mode21:36
*** salv-orlando has quit IRC21:37
*** e0ne has quit IRC21:38
*** baoli has joined #openstack-infra21:39
*** baoli has quit IRC21:39
*** baoli has joined #openstack-infra21:40
*** derekh has quit IRC21:40
*** jheroux has quit IRC21:41
*** lucasagomes has quit IRC21:41
*** baoli_ has joined #openstack-infra21:42
*** inc0 has quit IRC21:42
openstackgerritSpencer Krum proposed openstack-infra/system-config: Add ssl upgrade playbook  https://review.openstack.org/31226121:42
*** sridhar_ram1 has quit IRC21:42
*** lucasagomes has joined #openstack-infra21:44
*** baoli has quit IRC21:45
asselin_fungi, thanks for your help. Restarting zuul merger didn't work, and at the rate of merges & patch chain it woud've taken 8 hours to finish. Restarting zuul worked.21:45
*** Sukhdev has quit IRC21:47
*** ccamacho has joined #openstack-infra21:47
*** ddieterly is now known as ddieterly[away]21:50
tonybanteaya: I haven't said that to them explictly but it seems like a pretty reasonable expectation.21:50
openstackgerritTim Laszlo proposed openstack-infra/shade: Add support for server groups  https://review.openstack.org/31223721:50
tonybanteaya: they *are* using utc it's just that they re-evaluate which UTC time they use every 6 months21:51
*** salv-orlando has joined #openstack-infra21:51
*** amitgandhinz has quit IRC21:52
*** ddieterly[away] is now known as ddieterly21:54
dhellmannhey, folks, dims mentioned earlier today that there's an issue with the pypi mirror so we should hold off on releasing. Can someone give me a brief status update? I'd like to go ahead and announce that we won't do releases this week if it looks like we'll need that time to deal with the issue, just so everyone knows what to expect.21:54
*** ddieterly is now known as ddieterly[away]21:54
openstackgerritSpencer Krum proposed openstack-infra/system-config: Add ssl upgrade playbook  https://review.openstack.org/31226121:55
*** asettle has quit IRC21:58
*** asettle has joined #openstack-infra21:58
*** mriedem has quit IRC21:59
*** kengo_sakai has joined #openstack-infra22:00
*** DevBox has joined #openstack-infra22:03
*** asettle has quit IRC22:03
asselin_i'm running a bandersnatch mirror and for some reason it says it's up to date, but missing oslo.i18n 3.6.0 package.22:04
asselin_Is there a known issue with that?22:04
*** sambetts|afk has quit IRC22:04
*** mylu has quit IRC22:04
*** doug-fi__ has quit IRC22:04
*** kengo_sakai has quit IRC22:05
*** mylu has joined #openstack-infra22:05
asselin_i'm seeing the same behavior on 2 bandersnatch mirrors22:05
*** salv-orlando has quit IRC22:06
*** sambetts has joined #openstack-infra22:07
*** mylu has quit IRC22:09
*** nemish has quit IRC22:10
*** sigmavirus24 is now known as sigmavirus24_awa22:10
*** nemish has joined #openstack-infra22:10
*** tlbr has joined #openstack-infra22:12
*** mylu has joined #openstack-infra22:12
*** baoli_ has quit IRC22:14
pabelangerclarkb: ianw: ubuntu-precise built! https://review.openstack.org/#/c/31224822:14
*** ccamacho has quit IRC22:15
*** salv-orlando has joined #openstack-infra22:15
pabelangerclarkb: ianw: haven't tested image uploads, but figure we can let nodepool try22:15
pabelangerstepping away for a bit22:15
*** ccamacho has joined #openstack-infra22:15
*** ddieterly[away] is now known as ddieterly22:17
*** wajdi has quit IRC22:17
clarkbdhellmann: last I saw afs failed to replicate again and people were wanting to get jeblairs input22:17
clarkbas an alternative we can switch to the rw volume22:18
*** jsavak has quit IRC22:19
*** Guest70523 has quit IRC22:19
*** jsavak has joined #openstack-infra22:20
*** cody-somerville has joined #openstack-infra22:20
*** xyang1 has quit IRC22:23
*** nadya has joined #openstack-infra22:27
*** matt-borland has quit IRC22:27
*** salv-orlando has quit IRC22:28
*** salv-orlando has joined #openstack-infra22:28
*** dimtruck is now known as zz_dimtruck22:29
dhellmannclarkb : ok, I'm comfortable declaring a release hiatus for the week to give you all time to see what's what22:30
*** nadya has quit IRC22:31
clarkbdhellmann: works for me22:33
*** weshay has quit IRC22:34
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Add deadline option for build-timeout plugin  https://review.openstack.org/31227122:35
dhellmannclarkb : done. I'll check back on monday if I don't hear from someone before then.22:37
*** mylu has quit IRC22:38
*** baoli has joined #openstack-infra22:39
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Add deadline option for build-timeout plugin  https://review.openstack.org/31227122:40
openstackgerritDavid Lyle proposed openstack-infra/project-config: Adding tempest-horizon repo and job  https://review.openstack.org/31227222:41
*** salv-orlando has quit IRC22:42
*** ccamacho has quit IRC22:43
*** krotscheck is now known as krotscheck_dcm22:44
*** mylu has joined #openstack-infra22:46
*** mylu has quit IRC22:47
*** baoli has quit IRC22:48
*** baoli has joined #openstack-infra22:49
*** _amrith_ is now known as amrith22:49
*** sdague has quit IRC22:50
*** ddieterly is now known as ddieterly[away]22:51
*** ddieterly[away] is now known as ddieterly22:51
*** |-paul-| has quit IRC22:53
anteayatonyb: okay well it might be time to make it explicit22:53
anteayatonyb: as far as I know they are the only group switching meetings back and forth22:53
*** baoli has quit IRC22:54
openstackgerritStephen Lowrie proposed openstack/os-testr: Add subunit-describe-calls  https://review.openstack.org/31227522:54
*** dmk0202 has quit IRC22:54
anteayatonyb: might be nice to give them some heads up that they are stuck with one slot (which ever one they are in) if someone claims the other22:54
*** sdake has quit IRC22:55
*** sdake has joined #openstack-infra22:55
*** akshai has quit IRC22:57
*** baoli has joined #openstack-infra22:58
*** akshai has joined #openstack-infra22:58
*** ddieterly has quit IRC22:59
*** akshai has quit IRC23:00
*** bpokorny_ has quit IRC23:02
*** dynarro has quit IRC23:02
*** bpokorny has joined #openstack-infra23:02
*** rbrndt has quit IRC23:03
*** vhosakot has joined #openstack-infra23:04
sc`in addition to 29418 for the gerrit event stream, does zuul also need to talk to gerrit over 443?23:05
*** baoli has quit IRC23:06
*** baoli has joined #openstack-infra23:06
mgagnesc`: yes to query git repository and see if change has merged IIRC23:07
sc`i thought as much. trying to debug a fortified local install23:08
mgagneURL is configured with [gerrit]/baseurl23:08
mgagneI once had a bad config and was wondering why Zuul wouldn't mark changes as merged. It was waiting for like 5 minutes before giving up23:09
sc`that's where i am, i think23:10
*** gordc has quit IRC23:11
*** jsavak has quit IRC23:12
*** baoli has quit IRC23:13
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: [NO MERGY] Test a fake periodic job  https://review.openstack.org/22978923:16
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Don't remove the instack.qcow2 image  https://review.openstack.org/31228123:16
*** markvoelker_ has quit IRC23:16
*** vhosakot has quit IRC23:18
*** vhosakot has joined #openstack-infra23:18
*** rhallisey has quit IRC23:21
*** MarkAtwood has quit IRC23:22
*** flepied has joined #openstack-infra23:24
*** mriedem has joined #openstack-infra23:25
*** vhosakot has quit IRC23:25
*** HeOS has quit IRC23:26
*** nadya has joined #openstack-infra23:27
*** vhosakot has joined #openstack-infra23:28
*** nelsnels_ has joined #openstack-infra23:30
*** nelsnelson has quit IRC23:31
*** nadya has quit IRC23:32
*** keedya has quit IRC23:36
*** keedya has joined #openstack-infra23:37
*** rfolco has quit IRC23:38
tonybanteaya: There is more than one group doing the switch every 6 months thing.23:39
*** Swami has quit IRC23:43
*** Guest21848 has quit IRC23:44
*** cody-somerville has quit IRC23:45
*** dingyichen has joined #openstack-infra23:45
anteayatonyb: oh really23:45
tonybanteaya: I agree it goes counter to the norm but I don't think it's a problem that needs to be controlled.  It's really no different (IMO) to groups that need to meet on a monthly schedule rather than weekly23:46
anteayaokay I defer to you on this23:47
*** yamamoto has joined #openstack-infra23:47
anteayabut as a data point I don't have any sympathy for a group that does this if they get upset that a slot gets taken by another group whilst they are switching back and forth23:48
anteayafor me it is more about setting expectations23:48
anteayayou can switch but you accept that someone may take your now empty slot23:48
anteayaif they are aware, then fine23:48
tonybanteaya: I agree.  Any empty slot is fair game23:49
*** asettle has joined #openstack-infra23:49
anteayagreat, thanks23:51
anteayaI appreciate you helping me understand the situation23:51
*** RickA-HP has quit IRC23:53
tonybanteaya: Thanks for keeping an eye out for potential problems.23:57
tonybanteaya: that's not something I've made time to do for a while23:57
anteayait is hard to make time for it, I agree23:57

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!