Tuesday, 2016-09-06

*** dingyichen has joined #openstack-infra00:00
*** sarob has joined #openstack-infra00:01
*** sarob has quit IRC00:05
*** tphummel has quit IRC00:06
ianwpabelanger: firstly ... i can't seem to even get debug output out of the unit-tests00:06
ianwistr fixing that at some point00:06
ianwlooks to be something in oslotest ... some debug logs for some scenarios come out, but not others00:19
*** Julien-zte has joined #openstack-infra00:20
*** roxanaghe has joined #openstack-infra00:22
*** mriedem has joined #openstack-infra00:26
*** mriedem has left #openstack-infra00:26
*** roxanaghe has quit IRC00:27
*** fguillot has joined #openstack-infra00:28
ianwhttp://paste.openstack.org/show/566929/ <- i'm not sure i want to go down the rabbit-hole of why that would capture debug logs from two tests, but none of the others00:31
openstackgerritwes hayutin proposed openstack-infra/tripleo-ci: [DO NOT MERGE] Test a fake periodic job, newton 3  https://review.openstack.org/36580200:36
openstackgerritIan Wienand proposed openstack-infra/glean: Use OS_LOG_CAPTURE to get log output  https://review.openstack.org/36580700:40
*** thorst has joined #openstack-infra00:40
*** thorst has quit IRC00:47
*** thorst has joined #openstack-infra00:47
*** thorst has quit IRC00:47
*** kaisers1 has joined #openstack-infra00:51
*** kaisers has quit IRC00:51
*** amitgandhinz has joined #openstack-infra00:52
*** kaisers has joined #openstack-infra00:52
*** kaisers2 has quit IRC00:52
*** markvoelker has joined #openstack-infra00:53
*** thorst has joined #openstack-infra00:54
*** amitgandhinz has quit IRC00:56
*** thorst has quit IRC01:05
pabelangerianw: feel free to push up any changes you need today to the patch.01:09
ianwpabelanger: i think the intersection of sets is what is actually wanted, just testing it out01:11
*** vsaienko has quit IRC01:12
*** yamahata has joined #openstack-infra01:21
*** yanyanhu has joined #openstack-infra01:23
*** yuanying has joined #openstack-infra01:24
*** yuanying has quit IRC01:24
*** zhurong has joined #openstack-infra01:25
*** rbergeron has quit IRC01:38
*** rbergeron has joined #openstack-infra01:38
*** roxanaghe has joined #openstack-infra01:38
*** roxanaghe has quit IRC01:43
*** raddaoui has quit IRC01:43
*** raddaoui has joined #openstack-infra01:44
*** zshuo has joined #openstack-infra01:44
*** Genek has joined #openstack-infra01:46
*** baoli has joined #openstack-infra01:51
*** amitgandhinz has joined #openstack-infra01:52
*** baoli_ has joined #openstack-infra01:53
*** ijw has joined #openstack-infra01:55
*** akshai has joined #openstack-infra01:56
*** baoli has quit IRC01:56
*** amitgandhinz has quit IRC01:57
*** fguillot has quit IRC02:01
*** ijw has quit IRC02:01
*** fguillot has joined #openstack-infra02:02
*** fguillot has quit IRC02:02
*** baoli_ has quit IRC02:03
*** ijw has joined #openstack-infra02:03
*** ijw has quit IRC02:04
*** ijw has joined #openstack-infra02:04
openstackgerritguo yunxian proposed openstack/os-testr: Update home-page url  https://review.openstack.org/35617902:04
*** thorst has joined #openstack-infra02:05
*** dkehn has joined #openstack-infra02:09
*** ijw has quit IRC02:10
*** thorst has quit IRC02:13
*** armax has quit IRC02:13
openstackgerritIan Wienand proposed openstack-infra/glean: Skip interfaces not in sys_interfaces  https://review.openstack.org/36581902:16
ianwpabelanger: ^ it's kind of different to yours, so I put it up separately to make comparing easier02:17
*** rfolco has quit IRC02:18
tonybI assume the 'proposal' node is runnign trusty, but where is it managed so I can verify that?02:20
*** akshai_ has joined #openstack-infra02:29
*** salv-orl_ has joined #openstack-infra02:29
*** caowei has joined #openstack-infra02:32
*** akshai has quit IRC02:32
*** salv-orlando has quit IRC02:32
*** rhallisey has quit IRC02:33
*** sflanigan is now known as sflanigan_afk02:36
*** ijw has joined #openstack-infra02:36
*** thorst has joined #openstack-infra02:40
*** ijw has quit IRC02:40
*** dkehn_ has joined #openstack-infra02:44
*** amotoki has joined #openstack-infra02:48
*** Genek has quit IRC02:48
*** Genek has joined #openstack-infra02:49
*** hongbin_ has quit IRC02:49
*** hongbin_ has joined #openstack-infra02:50
*** amotoki has quit IRC02:50
*** yuanying has joined #openstack-infra02:52
*** akshai_ has quit IRC02:52
*** markvoelker has quit IRC02:53
*** amitgandhinz has joined #openstack-infra02:53
*** yuanying has quit IRC02:55
*** yuanying has joined #openstack-infra02:57
*** amitgandhinz has quit IRC02:58
*** ijw has joined #openstack-infra03:00
*** yuanying has quit IRC03:00
*** yuanying has joined #openstack-infra03:00
*** sflanigan_afk is now known as sflanigan03:01
prometheanfiretonyb: http://logs.openstack.org/65/365565/1/check/gate-requirements-integration-dsvm-ubuntu-xenial/3381ce2/console.html#_2016-09-05_10_55_44_92060803:02
prometheanfiretonyb: looks like it's running xenial?03:03
prometheanfireat least by name03:03
*** ijw has quit IRC03:05
*** amotoki has joined #openstack-infra03:07
*** roxanaghe has joined #openstack-infra03:09
*** senk has joined #openstack-infra03:10
tonybprometheanfire: right.  I don't follow03:11
tonybprometheanfire: We know the (requirements) gate is busted03:11
*** roxanaghe has quit IRC03:13
*** yuanying has quit IRC03:18
*** amotoki has quit IRC03:18
*** yuanying has joined #openstack-infra03:20
*** gildub has joined #openstack-infra03:33
*** gildub_ has joined #openstack-infra03:33
*** cody-somerville has joined #openstack-infra03:39
*** csomerville has joined #openstack-infra03:41
*** vikrant has joined #openstack-infra03:43
*** ijw has joined #openstack-infra03:44
*** cody-somerville has quit IRC03:45
*** ijw has quit IRC03:48
*** dkehn_ has quit IRC03:49
*** csomerville has quit IRC03:49
*** dkehn_ has joined #openstack-infra03:51
openstackgerritTony Breeds proposed openstack-infra/project-config: Include both Trusty and Xenial gate-requirements-integration-dsvm  https://review.openstack.org/36584003:51
*** coolsvap_ has joined #openstack-infra03:52
*** amitgandhinz has joined #openstack-infra03:54
*** amotoki has joined #openstack-infra03:54
*** thorst has quit IRC03:58
*** thorst has joined #openstack-infra03:58
*** amitgandhinz has quit IRC03:59
*** ijw has joined #openstack-infra04:00
*** yuanying has quit IRC04:00
*** thorst has quit IRC04:00
*** hongbin_ has quit IRC04:01
*** Sukhdev has joined #openstack-infra04:01
*** yuanying has joined #openstack-infra04:02
*** thorst has joined #openstack-infra04:03
*** sdake has joined #openstack-infra04:05
*** watanabe_isao has joined #openstack-infra04:06
*** thorst has quit IRC04:08
*** ijw has quit IRC04:09
*** caowei has quit IRC04:10
*** thorst has joined #openstack-infra04:10
*** caowei has joined #openstack-infra04:13
*** thorst has quit IRC04:15
*** pgadiya has joined #openstack-infra04:17
*** roxanaghe has joined #openstack-infra04:25
*** sdake has quit IRC04:25
*** ijw has joined #openstack-infra04:30
*** roxanaghe has quit IRC04:30
*** abregman has joined #openstack-infra04:37
*** psachin has joined #openstack-infra04:39
*** links has joined #openstack-infra04:39
*** ijw has quit IRC04:40
*** ijw has joined #openstack-infra04:41
*** ijw has quit IRC04:42
*** harlowja_at_home has joined #openstack-infra04:42
*** harlowja_at_home has quit IRC04:47
*** sdake has joined #openstack-infra04:50
*** amitgandhinz has joined #openstack-infra04:55
AJaegerjhesketh, ianw : Could you put this on your low-prio review queue, please? These are a few project-config changes of mine: https://review.openstack.org/365360 https://review.openstack.org/364782 https://review.openstack.org/365275 https://review.openstack.org/365074 https://review.openstack.org/364782 https://review.openstack.org/36489604:55
*** thorst has joined #openstack-infra04:55
AJaegertonyb, prometheanfire: we can revert clarkb's change for requirements if needed. We eventually need to make the change, Newton should be tested on Xenial. I prefer to give clarkb a chance to look at this later today - but if this blocks you, I'll approve a revert.04:57
AJaegerand then we'll do it again in a few days when we have all pieces together...04:57
tonybAJaeger: we have a fix we can do in requirements that *shoudl* work04:57
tonybAJaeger: I'd rather not revert because as you say we need to do this in the future anyway04:58
tonybAJaeger: my change will let us work out what the next steps are04:58
*** adriant has quit IRC04:59
*** thorst has quit IRC04:59
*** amitgandhinz has quit IRC04:59
ianwtonyb: on proposal OS ... best way is probably too look at  https://git.openstack.org/cgit/openstack-infra/system-config/tree/manifests/site.pp#n94505:00
tonybianw: Thanks.05:00
*** sdake has quit IRC05:01
AJaegertonyb: I've seen the change, it looks fine. And I agree, I'd like us to move forward here as well.05:02
tonybAJaeger: now to work out what I did wrong here: https://review.openstack.org/#/c/365840/105:04
*** oanson has joined #openstack-infra05:05
AJaegertonyb: I just commented ;)05:05
AJaegertonyb: the job is already there. You need to change zuul/layout to trigger that it runs on all branches. Right now we have a global branch regex that runs ubuntu-trusty only on liberty and mitaka, so add an override branches condition.05:06
tonybAJaeger: Ahh okay that's what I was missing.  Thanks.05:06
AJaegerianw: thanks for reviewing!05:07
*** thorst has joined #openstack-infra05:08
openstackgerritMerged openstack-infra/project-config: Fix the Zun gate  https://review.openstack.org/36580305:08
*** vikrant has quit IRC05:08
ianwAJaeger: np :)  i'm not sure if you've worked on log-copying in d-g before -- i wouldn't mind some eyes on https://review.openstack.org/#/c/364046/6 which is a refactor of the copy process.  it's been very tangle for a while and it bugs me every time :)05:09
*** vikrant has joined #openstack-infra05:10
*** sdake has joined #openstack-infra05:10
AJaegerianw: I haven't looked at d-g at all - but let me see what you've done...05:10
*** itisha has quit IRC05:11
ianwAJaeger: oh, no worries, don't worry too much05:11
*** ilyashakhat_mobi has joined #openstack-infra05:12
openstackgerritMerged openstack-infra/project-config: Remove some unused jobs from JJB  https://review.openstack.org/36536005:15
openstackgerritAnusha Ramineni proposed openstack-infra/project-config: Congress - Add replicated PE job  https://review.openstack.org/36586205:15
openstackgerritMerged openstack-infra/project-config: oslo.db: Add voting jobs to gate as well  https://review.openstack.org/36527505:15
openstackgerritMerged openstack-infra/project-config: Remove octavia bandit job  https://review.openstack.org/36507405:15
AJaegerianw: it's large ;( One thing I noticed: You remove archive_test_artifact - but don't seem to compress and chmod/chown these files anymore. Is that on purpose?05:16
openstackgerritMerged openstack-infra/project-config: Remove experimental bandit job from anchor  https://review.openstack.org/36489605:17
*** sdake has quit IRC05:17
openstackgerritAnusha Ramineni proposed openstack-infra/project-config: Congress - Add replicated PE job  https://review.openstack.org/36586205:17
*** rossella_s has quit IRC05:18
*** rossella_s has joined #openstack-infra05:19
ianwAJaeger: good point, next review i'll compress it.  the permissions is fixed up with recursive chown's at the end (so is a lot of zipping, but not for .subunit files)05:19
openstackgerritSteve Martinelli proposed openstack-infra/project-config: change post_test_hook location for osc functional tests  https://review.openstack.org/36586505:20
AJaegerianw: ah...05:20
*** jaosorior has joined #openstack-infra05:21
openstackgerritZhenguo Niu proposed openstack-infra/project-config: Add #openstack-nimble to the IRC channels  https://review.openstack.org/36586605:22
openstackgerritSteve Martinelli proposed openstack-infra/project-config: change post_test_hook location for osc functional tests  https://review.openstack.org/36586505:24
*** thorst has quit IRC05:25
*** ijw has joined #openstack-infra05:29
*** thorst has joined #openstack-infra05:29
*** florianf has joined #openstack-infra05:30
*** zzzeek has quit IRC05:30
*** ijw_ has joined #openstack-infra05:31
*** ijw has quit IRC05:32
*** thorst has quit IRC05:33
*** gildub has quit IRC05:34
*** senk has quit IRC05:34
*** gildub_ has quit IRC05:35
*** gildub has joined #openstack-infra05:35
*** ijw_ has quit IRC05:35
*** sarob has joined #openstack-infra05:37
*** sarob has quit IRC05:37
*** ijw has joined #openstack-infra05:38
*** ijw has quit IRC05:40
*** sandanar has joined #openstack-infra05:41
zhenguoAJaeger: hi, regarding https://review.openstack.org/#/c/365866/ , you mean we can't add new channels now?05:41
*** ilyashakhat_mobi has quit IRC05:43
*** ijw has joined #openstack-infra05:43
*** snarwade has quit IRC05:44
*** snarwade has joined #openstack-infra05:44
openstackgerritZhenguo Niu proposed openstack-infra/project-config: Add #openstack-nimble to the IRC channels  https://review.openstack.org/36586605:45
*** e0ne has joined #openstack-infra05:46
*** ijw has quit IRC05:46
*** e0ne has quit IRC05:47
*** ijw has joined #openstack-infra05:47
openstackgerritmathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review  https://review.openstack.org/36485905:48
openstackgerritTony Breeds proposed openstack-infra/project-config: Include both Trusty and Xenial gate-requirements-integration-dsvm  https://review.openstack.org/36584005:52
*** amitgandhinz has joined #openstack-infra05:56
*** abregman has quit IRC05:56
*** ijw has quit IRC05:57
*** abregman has joined #openstack-infra05:57
*** ijw has joined #openstack-infra05:58
*** amitgandhinz has quit IRC06:00
*** ijw has quit IRC06:01
*** rcarrillocruz has quit IRC06:02
*** ijw has joined #openstack-infra06:02
*** esikache1 has joined #openstack-infra06:02
*** ilyashakhat_mobi has joined #openstack-infra06:02
*** ijw has quit IRC06:03
*** esikache1 has quit IRC06:05
*** esikachev has joined #openstack-infra06:05
*** martinkopec has joined #openstack-infra06:06
openstackgerritYanyan Hu proposed openstack-infra/project-config: Fix senlin integration test  https://review.openstack.org/36587506:07
*** ijw has joined #openstack-infra06:10
*** binbincong_ has quit IRC06:12
*** roxanaghe has joined #openstack-infra06:13
*** senk has joined #openstack-infra06:16
fricklerseems gate-tempest-dsvm-neutron-identity-v3-only-full-nv has been broken since the morning of 2016-08-04, has anyone looked into this? if I run the reproduce.sh locally, I do not get the error, so maybe it is related to the trusty images06:18
*** roxanaghe has quit IRC06:18
openstackgerritMerged openstack-infra/devstack-gate: Ironic: Copy all deployment logs  https://review.openstack.org/36286706:18
*** ilyashakhat_mobi has quit IRC06:19
clarkbfrickler: have an example log? how does it fail?06:22
*** esikachev has quit IRC06:24
*** ijw has quit IRC06:25
*** dimtruck is now known as zz_dimtruck06:27
*** ihrachys has joined #openstack-infra06:29
*** claudiub|2 has joined #openstack-infra06:29
*** aeng_ has quit IRC06:29
*** _nadya_ has joined #openstack-infra06:30
*** claudiub has joined #openstack-infra06:31
*** claudiub|2 has quit IRC06:34
*** binbincong_ has joined #openstack-infra06:36
AJaegerzhenguo: correct - see the failure you got.06:36
zhenguoAjaeger: ok, I have removed the gerritbot changes:)06:38
AJaegerzhenguo: we wait for somebody to set up a second gerritbot in our infrastructure before we can accept more.06:39
zhenguoAJaeger: do we have a plan for that or is there somebody working on that?06:40
*** berendt has joined #openstack-infra06:40
AJaegerzhenguo: somebody volunteered but haven't seen followup on it ;(06:43
AJaegerbut if anybody else wants to do it...06:43
*** Hal1 has joined #openstack-infra06:44
*** andreas_s has joined #openstack-infra06:45
*** vsaienko1 has joined #openstack-infra06:45
zhenguoAJaeger: maybe I can try, where can I get some instructions for gerritbot setup?06:45
AJaegerzhenguo: best talk with fungi later today, he might now what happened with the other volunteer06:46
AJaegerzhenguo: there's some part in system-config06:47
AJaegerand there'S a puppet-gerritbot module.06:47
AJaegerWe need to run two instances of gerritbot.06:47
AJaegerzhenguo: there's already https://review.openstack.org/#/c/355588/ - reviewing that one might help...06:48
*** zz_dimtruck is now known as dimtruck06:48
*** pcaruana has joined #openstack-infra06:48
*** gildub has quit IRC06:49
zhenguoAJaeger: ok, will look into that, thanks!06:50
*** gildub has joined #openstack-infra06:50
*** gildub has quit IRC06:51
*** gildub has joined #openstack-infra06:52
*** markvoelker has joined #openstack-infra06:54
vsaienko1AJaeger: could you please help to merge https://review.openstack.org/#/c/365706/ one line patch, to update ironic-multinode job timeout.06:54
*** dkehn_ has quit IRC06:56
*** amitgandhinz has joined #openstack-infra06:56
AJaegervsaienko1: please wait for anteaya to reconsider her -1 first.06:56
*** dimtruck is now known as zz_dimtruck06:58
fricklerclarkb: one of the last successful runs I found: http://logs.openstack.org/04/350804/1/check/gate-tempest-dsvm-neutron-identity-v3-only-full-nv/96f3381/06:58
fricklerclarkb: failing: http://logs.openstack.org/88/350888/1/check/gate-tempest-dsvm-neutron-identity-v3-only-full-nv/b92d026/06:58
*** markvoelker has quit IRC06:59
fricklerclarkb: the failure is in the glance-api getting an auth error while trying to find the swift endpoint06:59
*** tesseract- has joined #openstack-infra07:00
*** ijw has joined #openstack-infra07:00
fricklerclarkb: like here http://logs.openstack.org/67/354367/1/check/gate-tempest-dsvm-neutron-identity-v3-only-full-nv/8cf4a21/logs/screen-g-api.txt.gz#_2016-08-11_21_45_47_22907:00
*** zz_dimtruck is now known as dimtruck07:00
*** ijw has quit IRC07:00
*** ijw has joined #openstack-infra07:00
*** dkehn_ has joined #openstack-infra07:01
*** amitgandhinz has quit IRC07:01
*** esikachev has joined #openstack-infra07:01
*** gildub has quit IRC07:02
*** gildub has joined #openstack-infra07:02
yolandagood morning07:02
openstackgerritMerged openstack-infra/glean: Add check to skip bridge interfaces  https://review.openstack.org/35998207:04
*** Julien-zte has quit IRC07:04
vsaienko1AJaeger: sure, lets wait for anteaya decision, thanks!07:05
*** Sukhdev has quit IRC07:06
*** Sukhdev has joined #openstack-infra07:06
AJaegermorning, yolanda !07:07
AJaegerclarkb: I added you also to https://review.openstack.org/#/c/365840 - another requirements xenial change. PLease review your ;) Tuesday morning07:09
*** matrohon has joined #openstack-infra07:09
*** dimtruck is now known as zz_dimtruck07:10
*** Sukhdev has quit IRC07:11
*** esikachev has quit IRC07:11
*** ihrachys has quit IRC07:12
openstackgerritIan Wienand proposed openstack-infra/devstack-gate: Fix devstack subunit output  https://review.openstack.org/36404507:12
openstackgerritIan Wienand proposed openstack-infra/devstack-gate: Refactor stackviz run  https://review.openstack.org/36456007:12
*** ihrachys has joined #openstack-infra07:12
openstackgerritIan Wienand proposed openstack-infra/devstack-gate: Refactor devstack log copying  https://review.openstack.org/36404607:12
*** rvba has joined #openstack-infra07:14
*** rvba has quit IRC07:14
*** rvba has joined #openstack-infra07:14
*** slaweq__ has joined #openstack-infra07:14
*** ifarkas_afk is now known as ifarkas07:16
*** slaweq_ has quit IRC07:17
*** ihrachys has quit IRC07:23
*** drifterza has joined #openstack-infra07:26
*** binbincong_ has quit IRC07:29
*** jpich has joined #openstack-infra07:30
*** Na3iL has joined #openstack-infra07:32
openstackgerritTony Breeds proposed openstack-infra/project-config: Include both Trusty and Xenial gate-requirements-integration-dsvm  https://review.openstack.org/36584007:33
*** hashar has joined #openstack-infra07:33
*** ijw has quit IRC07:35
*** esikachev has joined #openstack-infra07:35
*** abregman has quit IRC07:35
*** ijw has joined #openstack-infra07:36
*** isaacb has joined #openstack-infra07:36
*** gildub has quit IRC07:38
*** rcarrillocruz has joined #openstack-infra07:39
*** matthewbodkin has joined #openstack-infra07:41
*** matthewbodkin has quit IRC07:41
*** matthewbodkin has joined #openstack-infra07:41
*** mptacekx has joined #openstack-infra07:43
*** yaume has joined #openstack-infra07:45
openstackgerritmathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review  https://review.openstack.org/36485907:49
*** Ravikiran_K has joined #openstack-infra07:50
*** abregman has joined #openstack-infra07:51
*** binbincong_ has joined #openstack-infra07:53
*** daemontool has joined #openstack-infra07:54
*** Julien-zte has joined #openstack-infra07:56
*** amitgandhinz has joined #openstack-infra07:57
*** mikelk has joined #openstack-infra07:59
HeOSHello, infra team! Have you read my letter in openstack-infra ML about adding members to the following group? https://review.openstack.org/#/admin/groups/1555,members07:59
openstackgerritafazekas proposed openstack/os-testr: Make the listbuilder the default strategy  https://review.openstack.org/34945107:59
HeOSCould you please add them there?08:00
*** roxanaghe has joined #openstack-infra08:01
openstackgerritafazekas proposed openstack/os-testr: Allow to specifiy black/white list at the same time  https://review.openstack.org/34947708:01
*** amitgandhinz has quit IRC08:02
*** abregman has quit IRC08:04
openstackgerritafazekas proposed openstack/os-testr: Add --black-regex/-B option  https://review.openstack.org/34950508:05
*** abregman has joined #openstack-infra08:05
*** roxanaghe has quit IRC08:05
openstackgerritafazekas proposed openstack/os-testr: Use dash instead of underscore in args  https://review.openstack.org/34997208:07
*** jistr is now known as jistr|dentist08:07
*** asettle has joined #openstack-infra08:11
*** sflanigan has quit IRC08:11
openstackgerritshizhihui proposed openstack-infra/project-config: Make py35 voting for Horizon  https://review.openstack.org/35912308:12
fricklerHeOS: yesterday was a holiday for most infra-people, I assume they should be able to handle this later today08:12
HeOSfrickler, Ohhh, yes, I forgot about that. Thank you. :)08:12
*** tsufiev_ is now known as tsufiev08:13
*** derekh has joined #openstack-infra08:15
*** lucas-dinner is now known as lucasagomes08:16
openstackgerritafazekas proposed openstack/os-testr: Make the listbuilder the default strategy  https://review.openstack.org/34945108:16
*** ijw has quit IRC08:17
*** isaacb has quit IRC08:19
*** markvoelker has joined #openstack-infra08:19
openstackgerritOpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml  https://review.openstack.org/36593308:22
*** dizquierdo has joined #openstack-infra08:23
*** ihrachys has joined #openstack-infra08:23
*** markvoelker has quit IRC08:24
openstackgerritAnusha Ramineni proposed openstack-infra/project-config: Congress - Add replicated PE job  https://review.openstack.org/36586208:24
*** gongysh has quit IRC08:25
*** gongysh has joined #openstack-infra08:26
*** Jeffrey4l has quit IRC08:28
*** salv-orlando has joined #openstack-infra08:29
*** salv-orl_ has quit IRC08:32
openstackgerritafazekas proposed openstack/os-testr: Allow to specifiy black/white list at the same time  https://review.openstack.org/34947708:33
openstackgerritafazekas proposed openstack/os-testr: Add --black-regex/-B option  https://review.openstack.org/34950508:33
openstackgerritafazekas proposed openstack/os-testr: Use dash instead of underscore in args  https://review.openstack.org/34997208:34
openstackgerritAnusha Ramineni proposed openstack-infra/project-config: Congress - Add replicated PE job  https://review.openstack.org/36586208:35
*** dingyichen has quit IRC08:36
openstackgerritMerged openstack-infra/project-config: Add #openstack-nimble to the IRC channels  https://review.openstack.org/36586608:37
*** watanabe_isao has quit IRC08:37
openstackgerritMerged openstack-infra/project-config: Fix senlin integration test  https://review.openstack.org/36587508:39
*** ijw has joined #openstack-infra08:39
openstackgerritMerged openstack-infra/project-config: npm-docs job is now voting  https://review.openstack.org/36503308:43
openstackgerritMerged openstack-infra/project-config: murano-pkg-check: add coverage job  https://review.openstack.org/36490708:43
openstackgerritMerged openstack-infra/project-config: Use python-db-jobs for Glare  https://review.openstack.org/36437008:43
*** MmikeM is now known as Mmike08:45
*** abregman has quit IRC08:46
zynzelfungi: can you take a look at http://lists.openstack.org/pipermail/openstack-infra/2016-September/004707.html ? :)08:47
*** Mmike has quit IRC08:48
*** Mmike has joined #openstack-infra08:49
*** Mmike has joined #openstack-infra08:49
openstackgerritMerged openstack-infra/project-config: Publish console logs for python-muranoclient job template  https://review.openstack.org/36564808:50
openstackgerritMerged openstack-infra/project-config: Add python-storyboardclient  https://review.openstack.org/36556808:50
*** isaacb has joined #openstack-infra08:52
openstackgerritMerged openstack-infra/project-config: Promote OSA os_keystone nv jobs to voting  https://review.openstack.org/36555708:52
*** m3m0 has left #openstack-infra08:53
openstackgerritMerged openstack-infra/project-config: Change tripleo ha2 job types back too ha  https://review.openstack.org/36340908:58
*** amitgandhinz has joined #openstack-infra08:58
openstackgerritMerged openstack-infra/project-config: Add service-control charm interface  https://review.openstack.org/36359508:58
*** abregman has joined #openstack-infra08:59
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Remove the ha2 JOBTYPE  https://review.openstack.org/36341109:00
*** zz_dimtruck is now known as dimtruck09:01
openstackgerritmathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review  https://review.openstack.org/36485909:01
*** florianf has quit IRC09:02
*** amitgandhinz has quit IRC09:03
*** florianf has joined #openstack-infra09:04
*** ijw has quit IRC09:07
amrithhiya infra, I'm trying to debug an issue with the trove gate. we've had no success on trove or trove-integration and can't reproduce this on our local machines (where things seem to pass). I've pushed up a commit https://review.openstack.org/#/c/365960/ and it is now running on 2001:4800:1ae1:18:f816:3eff:fe51:7092. Would you please add https://gist.github.com/amrith/ad6e47f805d7f83bdf630ad84ff0141a (my public key) to09:10
amrithallow access to that machine.09:10
amrithAJaeger yolanda ^^ g'morning09:10
*** dimtruck is now known as zz_dimtruck09:10
*** nmagnezi has joined #openstack-infra09:13
*** ijw has joined #openstack-infra09:18
*** gongysh has quit IRC09:19
*** sambetts|afk is now known as sambetts09:21
*** gongysh has joined #openstack-infra09:21
*** r-mibu has quit IRC09:23
*** jistr|dentist is now known as jistr09:23
*** drybjed has quit IRC09:24
*** aderyugin has joined #openstack-infra09:26
*** gongysh has quit IRC09:26
*** acoles_ is now known as acoles09:28
*** daemontool has quit IRC09:29
*** drybjed has joined #openstack-infra09:30
*** r-mibu has joined #openstack-infra09:33
*** zshuo has quit IRC09:37
openstackgerritIan Wienand proposed openstack-infra/devstack-gate: Remove tracing since ansible seems to be working  https://review.openstack.org/17839809:40
*** shardy_afk has joined #openstack-infra09:43
*** shardy has joined #openstack-infra09:44
*** sdake has joined #openstack-infra09:45
*** Genek has quit IRC09:46
amrithhiya ... anybody home?09:49
*** roxanaghe has joined #openstack-infra09:49
*** roxanaghe has quit IRC09:54
openstackgerritMerged openstack-infra/project-config: Normalize projects.yaml  https://review.openstack.org/36593309:55
vsaienko1I'm wondered if anteaya around?09:58
*** zhurong has quit IRC09:58
*** dtantsur|afk is now known as dtantsur09:59
*** vsaienko has joined #openstack-infra09:59
*** amitgandhinz has joined #openstack-infra09:59
*** yanyanhu has quit IRC10:01
*** pgadiya has quit IRC10:02
*** amitgandhinz has quit IRC10:03
frickleris there a way to see a list of results for -nv jobs? looks like http://status.openstack.org/openstack-health only has statistics for voting jobs10:03
*** drybjed has quit IRC10:06
*** zz_dimtruck is now known as dimtruck10:06
*** Sukhdev has joined #openstack-infra10:07
*** vsaienko has quit IRC10:08
openstackgerritDmitry Tantsur proposed openstack-infra/project-config: Do not build any ramdisks in Ironic grenade jobs  https://review.openstack.org/36602810:08
dtantsursambetts, jroll, aarefiev ^^^10:08
*** sshnaidm|afk is now known as sshnaidm10:10
dtantsurfolks, please take a look ^^^ this will make our grenade job feel a bit better hopefully10:10
*** pgadiya has joined #openstack-infra10:11
dtantsurAJaeger, yolanda, sorry for personal ping, trying to unbreak the gate before the RC ^^^10:12
*** Julien-zte has quit IRC10:13
*** Sukhdev has quit IRC10:13
*** pgadiya has quit IRC10:15
*** e0ne has joined #openstack-infra10:15
*** pgadiya has joined #openstack-infra10:17
*** drybjed has joined #openstack-infra10:20
*** ijw has quit IRC10:20
openstackgerritOmer Anson proposed openstack-infra/project-config: Add Dragonflow fullstack test for ML2 module  https://review.openstack.org/36603310:22
*** Julien-zte has joined #openstack-infra10:23
*** caowei has quit IRC10:23
vsaienko1infra core: recently new project networking-cumulus was added: https://review.openstack.org/#/c/354784/. Could you please add me to project-core group so I can grant access for other users. Thanks in advance.10:24
*** asettle has quit IRC10:26
*** kaisers2 has joined #openstack-infra10:29
*** kaisers1 has quit IRC10:29
*** kaisers1 has joined #openstack-infra10:30
*** kaisers has quit IRC10:30
*** ramishra has quit IRC10:30
*** ramishra has joined #openstack-infra10:32
*** sdake has quit IRC10:33
*** dizquierdo is now known as dizquierdo_afk10:34
*** pgadiya has quit IRC10:35
*** askb has quit IRC10:40
*** abregman is now known as abregman|brb10:41
*** pblaho has joined #openstack-infra10:43
*** HenryG_ is now known as HenryG10:46
*** baoli has joined #openstack-infra10:46
*** Hal1 has quit IRC10:48
*** zzzeek has joined #openstack-infra10:51
*** pgadiya has joined #openstack-infra10:51
*** abregman|brb is now known as abregman10:52
*** dizquierdo_afk is now known as dizquierdo10:52
*** zzzeek has quit IRC10:56
*** zzzeek has joined #openstack-infra10:57
*** baoli has quit IRC10:57
*** kaisers2 has quit IRC10:58
*** kaisers1 has quit IRC10:59
*** kaisers has joined #openstack-infra10:59
*** kaisers1 has joined #openstack-infra10:59
*** baoli has joined #openstack-infra10:59
*** amitgandhinz has joined #openstack-infra10:59
*** dprince has joined #openstack-infra11:00
*** baoli has quit IRC11:01
openstackgerritMerged openstack-infra/project-config: Do not build any ramdisks in Ironic grenade jobs  https://review.openstack.org/36602811:03
*** amitgandhinz has quit IRC11:04
*** Jeffrey4l has joined #openstack-infra11:05
*** shardy_afk has quit IRC11:11
*** rhallisey has joined #openstack-infra11:13
*** Genek has joined #openstack-infra11:15
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Move the clouds_layouts.yaml file  https://review.openstack.org/36487111:16
*** ldnunes has joined #openstack-infra11:16
*** lucasagomes is now known as lucas-hungry11:22
*** rtheis has joined #openstack-infra11:22
*** dkehn_ has quit IRC11:23
openstackgerritMerged openstack-infra/puppet-infracloud: Refactor infra-cloud-bridge element to support CentOS/RH  https://review.openstack.org/35990911:24
*** e0ne has quit IRC11:25
*** ijw has joined #openstack-infra11:27
*** Hal has joined #openstack-infra11:29
*** amotoki_ has joined #openstack-infra11:29
*** ijw has quit IRC11:29
*** ijw has joined #openstack-infra11:30
*** amotoki has quit IRC11:31
*** dizquierdo has quit IRC11:32
*** gildub has joined #openstack-infra11:35
*** dkehn_ has joined #openstack-infra11:35
*** asettle has joined #openstack-infra11:37
*** binbincong_ has quit IRC11:37
*** roxanaghe has joined #openstack-infra11:37
*** sdague has joined #openstack-infra11:39
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Update infracloud documentation to remove old geo information  https://review.openstack.org/36501811:41
*** jkilpatr has joined #openstack-infra11:41
*** kaisers has quit IRC11:41
*** roxanaghe has quit IRC11:42
*** kaisers has joined #openstack-infra11:42
*** kaisers1 has quit IRC11:43
*** kaisers1 has joined #openstack-infra11:43
*** dprince has quit IRC11:43
openstackgerritMathieu Mitchell proposed openstack-infra/system-config: Introduce TinyCoreLinux mirror  https://review.openstack.org/32598111:43
*** dprince has joined #openstack-infra11:43
openstackgerritMerged openstack-infra/tripleo-ci: Retry stack-delete in case of failure  https://review.openstack.org/35064011:44
*** kaisers2 has joined #openstack-infra11:46
*** kaisers3 has joined #openstack-infra11:47
*** kaisers1 has quit IRC11:48
*** kaisers has quit IRC11:49
*** salv-orl_ has joined #openstack-infra11:52
*** salv-orlando has quit IRC11:52
openstackgerritBrad P. Crochet proposed openstack-infra/tripleo-ci: Add Zaqar to scenario002  https://review.openstack.org/36502611:52
*** asettle has quit IRC11:52
*** abregman has quit IRC11:53
*** abregman has joined #openstack-infra11:54
*** kaisers3 has quit IRC11:55
*** kaisers has joined #openstack-infra11:55
*** asettle has joined #openstack-infra11:58
*** amitgandhinz has joined #openstack-infra12:00
*** daemontool has joined #openstack-infra12:04
*** ijw has quit IRC12:05
*** amitgandhinz has quit IRC12:05
*** trown|outtypewww is now known as trown12:06
*** rfolco has joined #openstack-infra12:09
*** pgadiya has quit IRC12:10
*** daemontool_ has joined #openstack-infra12:10
*** zhurong has joined #openstack-infra12:10
AJaegeramrith: Sorry, I can't help with holding notes - and seems indeed as not many are around yet12:11
*** daemontool has quit IRC12:12
*** edmondsw has joined #openstack-infra12:13
*** gordc has joined #openstack-infra12:15
openstackgerritYaroslav Isakov proposed openstack-infra/project-config: Added networking-avaya project  https://review.openstack.org/36608712:16
*** gildub has quit IRC12:18
openstackgerritYaroslav Isakov proposed openstack-infra/project-config: Added networking-avaya project  https://review.openstack.org/36608712:20
*** oanson has quit IRC12:22
*** kgiusti has joined #openstack-infra12:23
*** Genek has quit IRC12:24
*** psilvad has joined #openstack-infra12:24
*** woodster_ has joined #openstack-infra12:26
*** sandanar_ has joined #openstack-infra12:26
*** clenimar has quit IRC12:26
*** lucas-hungry is now known as lucasagomes12:27
amrithAJaeger, figured out what the problem was (blind squirrel, acorn ...). Fix is in review now. Thanks!12:28
*** pgadiya has joined #openstack-infra12:28
AJaegerfungi, https://review.openstack.org/#/c/364723/ is in "merge conflict" - but if I try rebasing, it tells me that it's already up-to-date. Any bug in gerrit? Anything that needs doing there - or should we wait for next translation push?12:29
*** sandanar has quit IRC12:29
AJaegeramrith: glad to hear12:29
*** thorst has joined #openstack-infra12:30
*** rlandy has joined #openstack-infra12:30
*** Julien-zte has quit IRC12:30
openstackgerritYaroslav Isakov proposed openstack-infra/project-config: Added networking-avaya project  https://review.openstack.org/36608712:31
*** clenimar has joined #openstack-infra12:31
*** sandanar_ has quit IRC12:31
openstackgerritJoe Hakim Rahme proposed openstack-infra/project-config: Adds a new project: nova-tempest-staging  https://review.openstack.org/36609112:36
*** mdrabe has joined #openstack-infra12:37
*** tongli has joined #openstack-infra12:40
*** dmsimard|pto is now known as dmsimard12:43
openstackgerritSean Dague proposed openstack-infra/os-loganalyze: add placement-api to list of understood logs  https://review.openstack.org/36609612:43
openstackgerritYaroslav Isakov proposed openstack-infra/project-config: Added networking-avaya project  https://review.openstack.org/36608712:44
*** baoli has joined #openstack-infra12:44
*** baoli_ has joined #openstack-infra12:45
openstackgerritYaroslav Isakov proposed openstack-infra/project-config: Added networking-avaya project  https://review.openstack.org/36608712:46
openstackgerritYaroslav Isakov proposed openstack-infra/project-config: Added networking-avaya project  https://review.openstack.org/36608712:47
openstackgerritZara proposed openstack-infra/python-storyboardclient: Give user option to bypass verification  https://review.openstack.org/36609912:47
*** baoli has quit IRC12:48
*** furlongm_ is now known as furlongm12:48
*** jaosorior has quit IRC12:48
*** jaosorior has joined #openstack-infra12:49
openstackgerritYaroslav Isakov proposed openstack-infra/project-config: Added networking-avaya project  https://review.openstack.org/36608712:49
*** mmedvede has quit IRC12:49
*** jaosorior has quit IRC12:49
*** pcaruana has quit IRC12:50
*** jaosorior has joined #openstack-infra12:51
*** caowei has joined #openstack-infra12:53
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: Save console logs from all vms to files  https://review.openstack.org/36536912:54
*** dtardivel has joined #openstack-infra12:55
*** mhickey has joined #openstack-infra12:55
openstackgerritZara proposed openstack-infra/python-storyboardclient: Give user option to bypass verification  https://review.openstack.org/36609912:55
*** psilvad has quit IRC12:58
*** amitgandhinz has joined #openstack-infra13:01
*** amitgandhinz has quit IRC13:05
*** Goneri has joined #openstack-infra13:07
*** mbacchi_ is now known as mbacchi13:09
*** mmedvede has joined #openstack-infra13:11
openstackgerritWaldemar Znoinski proposed openstack/diskimage-builder: fix systemd resource deadlock  https://review.openstack.org/36579713:11
*** nmagnezi_ has joined #openstack-infra13:12
*** pgadiya has quit IRC13:13
*** moravec has quit IRC13:14
*** nmagnezi has quit IRC13:14
*** matt-borland has joined #openstack-infra13:17
openstackgerritThierry Carrez proposed openstack-infra/irc-meetings: Fix count_slot_usage.py meeting links  https://review.openstack.org/36612313:17
*** e0ne has joined #openstack-infra13:17
*** akshai has joined #openstack-infra13:18
*** rossella_s has quit IRC13:18
*** rossella_s has joined #openstack-infra13:19
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/irc-meetings: Create Architecture Working Group meeting  https://review.openstack.org/36612713:19
openstackgerritMerged openstack-dev/hacking: Fix issues detected by pycodestyle  https://review.openstack.org/35948713:20
openstackgerritMerged openstack-dev/hacking: Add documentation about off-by-default options  https://review.openstack.org/35942713:20
*** moravec has joined #openstack-infra13:24
*** smarcet has joined #openstack-infra13:25
*** eharney has joined #openstack-infra13:25
smarcetfungi: morning, i am trying to create a signed tag here https://github.com/openstack-infra/openstackid-resources.git13:25
smarcetbut its said that i dont have the rites13:25
*** roxanaghe has joined #openstack-infra13:25
smarcetfungi: from what i am seeing here https://review.openstack.org/Documentation/error-prohibited-by-gerrit.html13:26
*** zhurong has quit IRC13:26
*** ddieterly has joined #openstack-infra13:27
*** dizquierdo has joined #openstack-infra13:28
*** _degorenko|afk is now known as degorenko13:28
*** roxanaghe has quit IRC13:30
openstackgerritmathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review  https://review.openstack.org/36485913:30
openstackgerritMonty Taylor proposed openstack-infra/nodepool: Use interface_ip instead of internal logic  https://review.openstack.org/36044613:31
openstackgerritMonty Taylor proposed openstack-infra/nodepool: Plumb ipv6-preferred into force-ipv4 setting  https://review.openstack.org/36044513:31
openstackgerritMonty Taylor proposed openstack-infra/nodepool: Use MySQL create user statement to silence warning  https://review.openstack.org/36613313:31
*** drawsmcgraw has left #openstack-infra13:31
*** vikrant has quit IRC13:32
AJaegersmarcet: check the ACLs...13:33
*** drawsmcgraw has joined #openstack-infra13:33
AJaegermorning, pabelanger13:33
AJaegersmarcet: https://review.openstack.org/#/admin/projects/openstack-infra/openstackid-resources,access13:33
*** mriedem has joined #openstack-infra13:34
AJaegerAnd release team is only infra - so fungi can do it for you.13:34
*** amitgandhinz has joined #openstack-infra13:34
smarcetAJaegar: well i have those rites on openstackid13:35
*** _nadya_ has quit IRC13:35
smarceti could do the tags13:35
smarceton openstackid13:35
smarcetand need the same on openstackid-resources13:35
AJaegersmarcet: then fungi needs to empower you - and add to that group13:35
*** markvoelker has joined #openstack-infra13:36
smarcetok AJaeger thx for the info :)13:36
*** andymaier has joined #openstack-infra13:37
*** vikrant has joined #openstack-infra13:37
*** dteselkin has quit IRC13:41
*** pblaho has quit IRC13:43
*** pblaho has joined #openstack-infra13:44
*** yamahata has quit IRC13:44
*** pblaho has quit IRC13:44
*** pblaho has joined #openstack-infra13:45
*** esikachev has quit IRC13:46
*** dteselkin has joined #openstack-infra13:46
*** dimtruck is now known as zz_dimtruck13:47
*** vikrant has quit IRC13:47
openstackgerritMonty Taylor proposed openstack-infra/shade: Prevent Negative Caching  https://review.openstack.org/36332113:48
openstackgerritMonty Taylor proposed openstack-infra/shade: Revert "Revert per-resource dogpile.cache work"  https://review.openstack.org/36614313:48
*** xarses has quit IRC13:48
*** _nadya_ has joined #openstack-infra13:50
*** gongysh has joined #openstack-infra13:52
*** mhickey_ has joined #openstack-infra13:55
*** cardeois has joined #openstack-infra13:56
*** mhickey has quit IRC13:57
*** annegentle has joined #openstack-infra13:58
*** admcleod has joined #openstack-infra13:58
*** admcleod_ has quit IRC14:00
*** krtaylor has joined #openstack-infra14:01
*** Julien-zte has joined #openstack-infra14:03
*** smarcet has quit IRC14:03
*** bin_ has joined #openstack-infra14:04
*** thorst has quit IRC14:05
*** thorst has joined #openstack-infra14:05
openstackgerritMerged openstack-infra/project-config: Update ironic-multinode job timeout  https://review.openstack.org/36570614:07
*** rbrndt has joined #openstack-infra14:07
*** xarses has joined #openstack-infra14:09
*** smarcet has joined #openstack-infra14:09
fungitoo bad smarcet dropped off. anyway, if he comes back, i've made openstackid-resources-release have the same members as openstackid-release just now14:09
*** amotoki_ has quit IRC14:09
*** jcoufal has joined #openstack-infra14:09
openstackgerritAlexey Khivin proposed openstack-infra/project-config: Add jenkins-plugins-for-murano project to Openstack  https://review.openstack.org/36615414:10
smarcetfungi: thx a lot :D14:10
smarcethad a glitch on connneciton14:10
openstackgerritHongbin Lu proposed openstack-infra/project-config: Enable a missing Magnum job  https://review.openstack.org/36615514:10
fungioh, you came back!14:11
fungicool, give it another try14:11
lucasagomeshi all, can I please have some eyes on https://review.openstack.org/#/c/364921/ ? It does add a query to elastic-search14:11
jpmaxmanFungi just pinged smarcet14:12
*** esberglu has joined #openstack-infra14:12
fungiAJaeger: it's possible 364723 was in merge conflict 3 days ago but is no longer (perhaps some other conflicting change got reverted?). try a recheck comment14:14
*** thorst has quit IRC14:14
fungiAJaeger: also possible that something happened to one of the zuul mergers to cause it to fail to generate that merge commit14:14
*** inc0 has joined #openstack-infra14:15
*** nmagnezi_ has quit IRC14:15
*** amotoki has joined #openstack-infra14:17
pabelangerfungi: Any objections on bringing cloud8 online with 10 nodes this morning? https://review.openstack.org/#/c/364408/14:17
pabelangercloudnull: day 3, 0 launch node errors in osic-cloud114:18
AJaegerfungi, recheck does not change anything14:19
fungipabelanger: lgtm. looks like we've got images uploading successfully there14:20
fungiAJaeger: are you sure? i don't see another zuul comment there yet14:20
*** xarses has quit IRC14:20
*** gongysh has quit IRC14:20
pabelangerfungi: Ya, we've need uploading them for a few days now14:20
*** hashar is now known as hasharAway14:20
*** links has quit IRC14:21
fungiAJaeger: double-check zuul status and make sure it's not running jobs in the check pipeline right now14:21
AJaegerfungi, indeed it's building in zuul status14:21
*** yaume_ has joined #openstack-infra14:21
fungiin that case, the merge conflict message will disappear when we get new check results14:21
*** yaume has quit IRC14:23
pabelangerclarkb: https://review.openstack.org/#/c/365819/ ianw uploaded a new patch for glean, that does the same thing, just a little nicer I think.14:23
openstackgerritAleksandr Dobdin proposed openstack-infra/project-config: New Project timmy - ansible-like tool  https://review.openstack.org/35983114:23
pabelangermordred: ^ might be interested too, since rackspace is currently broken with glean14:23
mordredpabelanger: blerg14:24
*** senk has quit IRC14:25
mordredpabelanger: so you like ianw's patch betterer?14:25
*** zigo has quit IRC14:25
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: Add ipv6 nic-configs  https://review.openstack.org/36447914:25
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: Add IPv6 network configuration for ipv6 job types  https://review.openstack.org/36367414:25
pabelangermordred: Ya, does same thing. Think his is cleaner14:25
*** roxanaghe has joined #openstack-infra14:26
cloudnullpabelanger: ++ that's awesomre14:27
*** akshai has quit IRC14:27
jkilpatrare the jenkins check/gate jobs supposed to not have libffi anymore? I have builds failing today that didn't before this weekend with no changes.14:28
*** jordanP has joined #openstack-infra14:28
mordredpabelanger: looks good14:29
*** zz_dimtruck is now known as dimtruck14:29
openstackgerritAlexey Khivin proposed openstack-infra/project-config: Add jenkins-plugins-for-murano project to Openstack  https://review.openstack.org/36615414:29
anteayajkilpatr: sounds like one of your dependencies released a new version14:29
*** salv-orlando has joined #openstack-infra14:29
*** yaume_ has quit IRC14:30
*** roxanaghe has quit IRC14:31
sdaguemordred / fungi - can I get a quick review of this os-log-analyzer filter update - https://review.openstack.org/#/c/366096/ ?14:31
anteayajkilpatr: and your jobs have whatever they are told to have14:31
openstackgerritAlexey Khivin proposed openstack-infra/project-config: Add jenkins-plugins-for-murano project to Openstack  https://review.openstack.org/36615414:31
*** zigo has joined #openstack-infra14:32
*** salv-orl_ has quit IRC14:32
*** yar_ has joined #openstack-infra14:33
*** zigo is now known as Guest3914714:33
openstackgerritMerged openstack-infra/glean: Skip interfaces not in sys_interfaces  https://review.openstack.org/36581914:33
mordredpabelanger: need we to cut a point-release of glean?14:33
*** mhickey_ has quit IRC14:33
*** rook has joined #openstack-infra14:33
pabelangermordred: yes, if possible14:34
fungijkilpatr: which job(s) specifically?14:34
mordredpabelanger: tag for 1.6.1 pushed14:34
pabelangermordred: thanks14:35
*** jamesdenton has joined #openstack-infra14:35
*** xarses has joined #openstack-infra14:36
anteayamorning zaro14:37
*** akshai has joined #openstack-infra14:37
openstackgerritMerged openstack-infra/os-loganalyze: add placement-api to list of understood logs  https://review.openstack.org/36609614:37
*** amotoki has quit IRC14:38
*** fguillot has joined #openstack-infra14:38
*** Guest39147 has quit IRC14:39
*** mhickey_ has joined #openstack-infra14:39
*** sarob has joined #openstack-infra14:40
jkilpatrfungi, https://review.openstack.org/#/c/363688/914:40
mtreinishyolanda: thanks for the +A on the lpmqtt patches yesterday. If you get a sec could you populate the gerrit groups: https://review.openstack.org/#/admin/groups/1550,members https://review.openstack.org/#/admin/groups/1551,members https://review.openstack.org/#/admin/groups/1552,members and https://review.openstack.org/#/admin/groups/1553,members14:40
*** amitgandhinz has quit IRC14:41
*** vinaypotluri has joined #openstack-infra14:42
*** zigo_ has joined #openstack-infra14:42
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add libffi-dev to bindep  https://review.openstack.org/36616814:42
yolandahi mtreinish , sure14:42
mtreinishyolanda: cool, thanks14:43
*** krtaylor has quit IRC14:43
openstackgerritmathieu bultel proposed openstack-infra/tripleo-ci: Implement undercloud upgrade job - Mitaka -> Newton  https://review.openstack.org/34699514:43
openstackgerritWitold Bedyk proposed openstack-infra/project-config: Move monasca-log tempest job to check queue  https://review.openstack.org/36568114:44
fungijkilpatr: it looks like it may have previously been relying on a prebuilt wheel for cffi from our mirror, but cffi 1.8.2 was released today and we don't have a new wheel built for it yet so your job is falling back to attempting to build it from the sdist tarball14:44
*** claudiub has quit IRC14:44
*** sdake has joined #openstack-infra14:44
*** sarob has quit IRC14:45
openstackgerritmathieu bultel proposed openstack-infra/tripleo-ci: Implement undercloud upgrade job - Mitaka -> Newton  https://review.openstack.org/34699514:45
fungijkilpatr: you should probably add cffi's build dependencies to http://git.openstack.org/cgit/openstack/browbeat/tree/bindep.txt, which would look something like the libffi-related lines at http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/data/bindep-fallback.txt#n3814:45
jkilpatrfungi, thanks14:45
fungijkilpatr: i expect that would pass in the ci's current state, and would be more correct going forward14:46
*** daemontool_ has quit IRC14:47
*** zigo_ has quit IRC14:47
*** daemontool has joined #openstack-infra14:48
*** zigo_ has joined #openstack-infra14:48
*** mikelk has quit IRC14:49
openstackgerritWitold Bedyk proposed openstack-infra/project-config: Move monasca-log tempest job to check queue  https://review.openstack.org/36568114:49
*** JerryOpenix has joined #openstack-infra14:52
*** _nadya_ has quit IRC14:52
*** annegentle has quit IRC14:53
mtreinishcorvus, clarkb, fungi: so now that lpmqtt is a thing, did we ever come to a conclusion on what to do for the imap for it?14:53
fungimtreinish: i think it comes down to choosing between a dedicated alias address for our infra inbox (in rackspace apps where openstack.org e-mail is hosted), or configuring exim on firehose01 to accept e-mail from lp and then setting up courier or cyrus or something to listen on localhost14:56
clarkbpabelanger: cool looks like that fix for glean merged and we got a new release14:56
fungimtreinish: i'm more in favor of the latter, personally, since we're not entirely in control of the former14:57
*** ifarkas is now known as ifarkas_afk14:57
*** dkehn_ has quit IRC14:57
mtreinishfungi: that's fine, although do we have puppet for exim and the imap server anywhere?14:57
pabelangerclarkb: Yup, and ubuntu-xenial build should pull it in today14:58
*** amitgandhinz has joined #openstack-infra14:58
mtreinishmy personal mail infra is postfix and dovecot (although I used cyrus a long time ago) so I'm not sure how much input I can give on those modules14:59
clarkbAJaeger: catching up on requirements related changes/bugs15:00
clarkbAJaeger: will review though I have a call now15:00
*** pradk has quit IRC15:00
*** piet has joined #openstack-infra15:01
*** rodrigods has quit IRC15:02
*** rodrigods has joined #openstack-infra15:02
*** aviau has quit IRC15:02
*** aviau has joined #openstack-infra15:03
*** yaume has joined #openstack-infra15:03
*** thcipriani|afk is now known as thcipriani15:03
*** AJaeger has quit IRC15:04
clarkbpabelanger: nodepool.image.build.ubuntu-xenial: Successfully installed glean-1.6.0 might need to rebuild to get 1.6.115:06
mordredfungi, mtreinish: I'm also in favor of the second option15:06
pabelangerclarkb: nice15:07
mordredwe have exim modules in puppet already. jeblair and I have been cyrus users for forever - although our existing cyrus is not puppeted ... but I'd imagine getting a cyrus up and going would not be terribly hard15:07
pabelangerclarkb: cool, back down to 1 hour builds15:08
*** annegentle has joined #openstack-infra15:08
pabelangerclarkb: oh, wait. that is from yesterday15:08
clarkbpabelanger: oh that would explain it15:08
mtreinishmordred: ok cool, iirc setting up cyrus wasn't very difficult when I did it before15:08
openstackgerritZara proposed openstack-infra/python-storyboardclient: Give user option to bypass verification  https://review.openstack.org/36609915:09
pabelangerclarkb: we doing ubuntu-trusty now15:09
mtreinishmordred: sounds like you're volunteering to help :)15:09
clarkbkk so need a rebuild just hasn't happened yet :)15:09
*** andreas_s has quit IRC15:09
openstackgerritMatthew Bodkin proposed openstack-infra/storyboard-webclient: Make tags reusable  https://review.openstack.org/36618815:09
*** dkehn_ has joined #openstack-infra15:10
*** gouthamr has joined #openstack-infra15:10
*** amotoki has joined #openstack-infra15:11
*** thorst has joined #openstack-infra15:11
*** _bmjen is now known as bmjen15:11
mordredmtreinish: I am theoretically volunteering ... but I will defer to jeblair for really thinking about it :)15:12
*** gouthamr_ has joined #openstack-infra15:12
mtreinishmordred: heh I only found 1 cyrus module on puppetforge and it doesn't look so complete: https://forge.puppet.com/buzzdeee/cyrus_imapd15:12
mtreinishand it might be openbsd specific15:13
*** spzala has joined #openstack-infra15:13
*** yamahata has joined #openstack-infra15:14
*** gouthamr has quit IRC15:15
*** matt-borland has quit IRC15:17
clarkbpabelanger: mgagne are we ready to increase the mtl01 max-servers today?15:17
pabelangerThink so, working to bring cloud8 online15:18
clarkbif mgagne acks I can approve15:18
*** thorst has quit IRC15:19
*** yaume has quit IRC15:19
*** pradk has joined #openstack-infra15:20
fungimtreinish: it's entirely possible that the debian/ubuntu package for cyrus automatically does what we need without needing to be tweaked15:20
fungiso might not need a separate puppet module15:20
*** zigo_ is now known as zigo15:20
*** AJaeger has joined #openstack-infra15:22
mtreinishfungi: oh heh, yeah that's possible. My only experience running cyrus was with gentoo running on a ps3. (which I do not recommend to anyone)15:22
openstackgerritmathieu bultel proposed openstack-infra/tripleo-ci: Implement undercloud upgrade job - Mitaka -> Newton  https://review.openstack.org/34699515:23
rcarrillocruzthat's awesome mtreinish  :D15:23
*** spzala has quit IRC15:26
mtreinishrcarrillocruz: heh, it makes a terrible mail server. The main cpu is too slow (and the spus were never used by anything) and interpreted languages were just terrible on that old powerpc cpu, it made using spamassassin fun.15:27
*** piet has quit IRC15:27
mtreinishmy favorite part was that acpi didn't really work on it either so it ate >200w at idle15:27
*** gokrokve has joined #openstack-infra15:27
*** mriedem1 has joined #openstack-infra15:27
*** Julien-zte has quit IRC15:28
openstackgerritZara proposed openstack-infra/python-storyboardclient: Give user option to bypass verification  https://review.openstack.org/36609915:29
*** armax has joined #openstack-infra15:29
*** mriedem has quit IRC15:30
*** mptacekx has quit IRC15:30
*** thorst has joined #openstack-infra15:30
rcarrillocruzwell, yeah, the ps3 was an akward machine to code for...15:31
rcarrillocruzgame devs complaining about it all the time15:31
*** piet has joined #openstack-infra15:31
clarkbmtreinish: I ran folding at home on mine during the winter because it kept my room warm15:31
*** M-docaedo_vector has quit IRC15:32
openstackgerritZara proposed openstack-infra/python-storyboardclient: Give user option to bypass verification  https://review.openstack.org/36609915:32
pabelangerclarkb: rcarrillocruz: mind helping https://review.openstack.org/#/c/366168/ land? adds libffi-dev to bindep.txt15:32
mtreinishclarkb: heh, that probably stressed the cpu about the same as spamassassin. (it really didn't like running perl)15:32
pabelangersomething is pulling it in for nodepool gate test15:32
*** caowei has quit IRC15:33
clarkbpabelanger: maybe we should just make sure nodepool has a bindep.txt?15:33
clarkboh wait this is not the fallback its the bindep for project-config itself15:33
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: pingtest: run 'openstack stack failures list' when failure  https://review.openstack.org/36391815:34
*** mhickey__ has joined #openstack-infra15:34
pabelangerAlso, working on mirror.regionone.osic-cloud8.openstack.org again now that networking issue has been sorted out15:35
clarkbhttps://review.openstack.org/#/c/363984/1 is the internap quota increase15:35
*** spzala has joined #openstack-infra15:35
clarkblooks like mgagne acked the change itself. Should I just approve then?15:35
*** Jeffrey4l has quit IRC15:35
pabelangerclarkb: ya, we need to land 366168 first, but works for me15:35
clarkbpabelanger: cool remember that you ened to update the existing dns records for that mirror as currently it points at cloud1 (rather than just adding a new set of records)15:36
openstackgerritmathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review  https://review.openstack.org/36485915:36
*** mhickey_ has quit IRC15:36
pabelangerclarkb: ack15:36
clarkbpabelanger: looks like rcarrillocruz and I +2'd 366168 at the same time. I have since approved it so hopeflly that goes in soon15:36
pabelangercan't SSH into the server right now so need to fix that first15:37
*** martinkopec has quit IRC15:37
pabelangerWhile testing ssh access: [Errno None] Unable to connect to port 22 on
clarkbpabelanger: security groups?15:37
*** tonytan4ever has joined #openstack-infra15:38
*** matbu is now known as matbu|bbl15:38
*** tonytan_brb has joined #openstack-infra15:39
odyssey4mepabelanger hmm, have some sort of filtering things been implemented somewhere between Sep 1 and Sep 4... our non-master branches are now missing functional tests for trusty15:39
odyssey4mepabelanger an example: https://review.openstack.org/#/c/364356/ & https://review.openstack.org/#/c/366203/15:39
*** andymaier has quit IRC15:39
openstackgerritEmilien Macchi proposed openstack-infra/system-config: Added Gem Mirror to Infra  https://review.openstack.org/25361615:40
*** pcaruana has joined #openstack-infra15:42
*** roxanaghe has joined #openstack-infra15:42
pabelangerodyssey4me: could be a side affect of https://review.openstack.org/#/c/363639/15:42
*** tonytan4ever has quit IRC15:42
pabelangerodyssey4me: yes, we need to include stable branches15:43
pabelangerodyssey4me: let me get a patch up15:43
odyssey4mepabelanger we already have a filter that allowed it - so that one has broken ours15:43
odyssey4mewhatever we do - it needs to combine the functional result of both of those15:43
pabelangerodyssey4me: regex likely applied after yours, which removes them15:44
*** sdake has quit IRC15:44
pabelangerlet me see15:44
*** berendt has quit IRC15:45
openstackgerritMerged openstack-infra/project-config: Add libffi-dev to bindep  https://review.openstack.org/36616815:45
odyssey4mepabelanger a quick fix is probably to ensure that your additional rule doesn't act on any 'openstack-ansible' jobs15:46
*** mhickey__ has quit IRC15:46
*** roxanaghe has quit IRC15:47
pabelangerodyssey4me: ^gate-(?!openstack-ansible).*(-dsvm)?-ansible-func-ubuntu-trusty(-nv)?$ should work15:48
*** psachin has quit IRC15:49
odyssey4mepabelanger yep, or alternatively adjust https://github.com/openstack-infra/project-config/blob/master/zuul/layout.yaml#L3084-L3087 to be something like15:49
odyssey4meurgh, my brain is unable to come up with something there15:50
openstackgerritSergei Chipiga proposed openstack-infra/project-config: Add non-voting job for new horizon ui tests  https://review.openstack.org/36622315:50
*** tonytan_brb has quit IRC15:51
odyssey4meso yeah, your suggestion seems fine15:51
odyssey4mepabelanger want me to patch it up?15:51
openstackgerritPaul Belanger proposed openstack-infra/project-config: Ignore openstack-ansible for ansible-func-ubuntu-trusty  https://review.openstack.org/36622515:52
pabelangerodyssey4me: ^\15:52
*** mriedem1 is now known as mriedem15:53
*** tonytan4ever has joined #openstack-infra15:53
*** dkehn__ has joined #openstack-infra15:54
mgagneclarkb: go15:55
*** matrohon has quit IRC15:55
*** gyee has joined #openstack-infra15:55
clarkbmgagne: done :)15:55
openstackgerritHongbin Lu proposed openstack-infra/project-config: Correct the directory path for Zun  https://review.openstack.org/36623115:56
odyssey4mepabelanger yeah, that appears to correctly ignore the job names we're seeing being affected now15:56
*** spzala has quit IRC15:57
pleia2good morning15:59
*** isaacb has quit IRC15:59
*** claudiub has joined #openstack-infra16:00
*** adrian_otto has joined #openstack-infra16:00
openstackgerritZara proposed openstack-infra/python-storyboardclient: Give user option to bypass verification  https://review.openstack.org/36609916:03
*** spzala has joined #openstack-infra16:03
*** gokrokve has quit IRC16:05
*** david-lyle_ has joined #openstack-infra16:05
*** david-lyle_ has quit IRC16:05
*** Swami has joined #openstack-infra16:06
*** abregman has quit IRC16:08
*** spzala has quit IRC16:08
openstackgerritMatthew Bodkin proposed openstack-infra/storyboard-webclient: Make tags reusable  https://review.openstack.org/36618816:09
*** senk has joined #openstack-infra16:09
openstackgerritMerged openstack-infra/project-config: Set max-servers value to 150 for internap-mtl01  https://review.openstack.org/36398416:09
*** dkehn__ has quit IRC16:09
openstackgerritZara proposed openstack-infra/python-storyboardclient: Make dev the default StoryBoard instance  https://review.openstack.org/36623816:10
*** jascott1 has joined #openstack-infra16:12
openstackgerritMerged openstack-infra/project-config: Bring osic-cloud8 online with 10 nodes  https://review.openstack.org/36440816:14
*** derekh is now known as derekh_afk16:14
pabelangerrcarrillocruz: seeing some scheduling errors on infracloud right now, looking to see why16:14
clarkbpabelanger: you should beable to take the insatnce uuid and track it from api to compute16:15
*** M-docaedo_vector has joined #openstack-infra16:15
clarkbat least thats what I do when debugging devstack I am sure its the same thing in a real cloud ;)16:15
rcarrillocruzpabelanger: ack16:16
rcarrillocruzi'm away from home now16:17
rcarrillocruzif it's too serious, disable it, i can look at it later after meeting or during , will be around16:17
pabelangerclarkb: rcarrillocruz: ^ looks like networking issues?16:17
pabelangerseeing a fair bit of ConnectTimeout exceptions16:18
clarkbpabelanger: thats specifically trying to do a networking allocation16:20
*** e0ne has quit IRC16:20
*** smarcet has quit IRC16:20
*** gouthamr_ has quit IRC16:20
*** thorst has quit IRC16:21
*** daemontool has quit IRC16:21
clarkbpabelanger: it may help to grep all the connection timeouts and see if they are all in the same code path. That would point at neutron whereas timeouts across the board might indicate general performacne issues or an underlying network problem fullfilling those requests16:21
*** thorst has joined #openstack-infra16:21
*** rhallisey has quit IRC16:21
*** rhallisey has joined #openstack-infra16:22
pabelangerclarkb: Ya, we've had 20 ConnectionTimeout today just for ports.json. Maybe we need to up our timeout value16:23
*** tphummel has joined #openstack-infra16:23
openstackgerritJoe Hakim Rahme proposed openstack-infra/project-config: Adds a new project: nova-tempest-staging  https://review.openstack.org/36609116:23
clarkbpabelanger: ya if its just that one request having it then likely a tuning item related to that16:23
*** yamahata has quit IRC16:23
clarkbpabelanger: or maybe we need to run more neutron api servers or something16:23
pabelangerclarkb: Right, let me find the value, see what the default is and update. Also going to see how we configured neutron api16:24
*** daemontool has joined #openstack-infra16:25
*** drifterza has quit IRC16:26
pabelangerlooks like timeout=30 by default in nova.conf16:26
*** akshai has quit IRC16:26
clarkbpabelanger: which seems like it should be plenty16:26
clarkbjust off hand 30 seconds to get a port for a VM should be plenty imo16:26
pabelangerchecking neutron now to see if there is an issue16:27
*** dtantsur is now known as dtantsur|afk16:28
*** akshai has joined #openstack-infra16:29
*** tonytan_brb has joined #openstack-infra16:29
*** daemontool has quit IRC16:29
*** dkehn__ has joined #openstack-infra16:29
*** piet has quit IRC16:29
*** Apoorva has joined #openstack-infra16:29
*** SamYaple has quit IRC16:29
*** SamYaple has joined #openstack-infra16:30
*** Sukhdev has joined #openstack-infra16:30
*** thorst has quit IRC16:30
*** matthewbodkin has quit IRC16:30
pabelangerneutron-server.log error http://paste.openstack.org/show/567209/16:30
*** piet has joined #openstack-infra16:31
*** piet has quit IRC16:31
*** stevemar has quit IRC16:31
openstackgerritZara proposed openstack-infra/python-storyboardclient: Make dev the default StoryBoard instance  https://review.openstack.org/36623816:31
*** tonytan4ever has quit IRC16:32
*** gouthamr has joined #openstack-infra16:33
pabelangerclarkb: so, neutron is using sqlite right now.16:33
pabelangerconnection = sqlite:////var/lib/neutron/neutron.sqlite16:33
clarkbpabelanger: that may explain it :)16:33
clarkbsince sqlite can't do concurrent connections iirc. Basically forcing everything to be serialized16:34
pabelangerincoming patch16:34
clarkbpabelanger: keep in mind that the migration from sqlite to mysql is likely not to be a clean one. Probably want to stop running instances there. And start over from scratch with neutron?16:34
clarkbits provider networking so that should mostly just work except we need to make sure we steady state to no instances first16:35
*** jordanP has quit IRC16:35
*** derekh_afk is now known as derekh16:35
pabelangerclarkb: Ya, we should likely disable infracloud for now16:35
*** vhosakot has joined #openstack-infra16:35
*** isaacb has joined #openstack-infra16:36
openstackgerritThomas Goirand proposed openstack-infra/project-config: Add missing deb-* git repo  https://review.openstack.org/36624616:36
*** tonytan_brb is now known as tonytan4ever16:37
pabelangermy mistake16:37
pabelangerseems neutron is installed on the compute nodes16:38
mgagnepabelanger: you checked the agent and not neutron-server?16:38
pabelangerand I was looking at that16:38
zigoSince when Neutron supports SQLite migrations? :)16:38
mgagnepabelanger: yea, agent doesn't have access to database and usually Puppet or whatever won't put database access in there16:38
clarkbpabelanger: only the l2 agent should be on the compute hosts and maybe the metadata proxiy16:38
*** smarcet has joined #openstack-infra16:38
pabelangermgagne: clarkb: Ya, that's what I was looking at16:38
pabelangerassuming l2 agent doesn't need database16:39
clarkb(not sure if we enabled metadat or not)16:39
*** matt-borland has joined #openstack-infra16:39
mordredwe should not have16:39
mordredlast we spoke about it it the plan was to not run it16:39
clarkbpabelanger: if the controller processes are all set up to use mysql the other thing to check is how many processes are we running and are we using pymysql or mysql-python16:41
*** notmars-lounge has quit IRC16:41
*** docaedo3 has quit IRC16:41
clarkbpabelanger: I want to say that with mysql-python you need many more processes as it serializes when it jumps into C. If we use pymysql we can get away with fewer processes (I would prefer pymysql as its what we test with)16:41
*** xyang1 has joined #openstack-infra16:42
pabelanger51 neutron-server processes16:42
pabelanger14 neutron-metadata-agent16:42
pabelanger1 neutron-dhcp-agent16:42
*** _nadya_ has joined #openstack-infra16:42
*** piet has joined #openstack-infra16:42
clarkbok with that number of neutron-servers I am gonna guess we are using mysql-python16:43
*** roxanaghe has joined #openstack-infra16:43
nikhilAJaeger: hey, so this being merged https://review.openstack.org/#/c/363088/ https://review.openstack.org/#/c/363066/ , if you'd like to consider them critical enough for backport to newton please feel free to create a review. we can consider then for a release for stable/newton if the gate is significantly affected.16:43
pabelangerclarkb: python-pymysql is installed16:44
clarkbthoguh since nodepool serializes its requests anyways maybe thats unrelated (to avoid api throttling ndoepool serializes its requests through the provider manager for each provider)16:44
*** lucasagomes is now known as lucas-dinner16:44
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources: Upgrade Laravel Version and ORM Framework  https://review.openstack.org/32230716:46
clarkbpabelanger: maybe share that traceback with neutron devs and see if they have any useful info?16:47
pabelangerclarkb: https://bugs.launchpad.net/neutron/+bug/1615820 is the same traceback we are seeing16:47
openstackLaunchpad bug 1612798 in neutron "duplicate for #1615820 Move db retry logic away from API layer" [Critical,In progress] - Assigned to Kevin Benton (kevinbenton)16:47
clarkboh cool someone else has already reported it16:47
pabelangerya, not sure the same16:47
*** weshay is now known as weshay_bbiab16:47
*** pradk has quit IRC16:47
pabelangerbut recent16:47
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources: Upgrade Laravel Version and ORM Framework  https://review.openstack.org/32230716:47
*** roxanaghe has quit IRC16:47
*** jamesdenton has quit IRC16:48
amrithmtreinish, ping16:50
mtreinishamrith: pong16:50
amrithhiya matt16:50
amrithwanted to see if you had a chance to take a look at the message I'd sent you16:50
amrithI think I've gone over most of the things on your list from a couple of weeks ago16:51
mtreinishamrith: yeah it looks that way, most of the things on that etherpad were from the middle of mitaka16:53
amrithyes, and there may have been some from before as well16:53
amrithlike trove-integration16:53
amrithdare I say redstack16:53
*** Sukhdev has quit IRC16:53
amriththat's not yet done, but on my list for O16:53
mtreinishI haven't really watched for the backwards incompat and api change stuff though. But I believe you if you say you're better about that now16:54
amrithwe're taking a closer look at those now16:54
mtreinishand after the tox cleanup I think the cti stuff is fixed16:54
*** ihrachys has quit IRC16:55
amrithOK, I'm putting together a list, I'll send it by you. some of them may be related to the stuff you mentioned16:55
amrithrelated to CI16:55
*** mminesh has quit IRC16:55
openstackgerritMatthew Treinish proposed openstack-infra/system-config: Add lpmqtt to firehose deployments  https://review.openstack.org/36331316:56
mtreinishamrith: ok, I'll keep an eye out for it16:57
*** sdake has joined #openstack-infra16:57
amriththx matt16:57
*** asettle has quit IRC16:57
mtreinishmordred, fungi: ^^^ I guess we'll need to add the install and setup imap in front of that patch16:58
*** jcoufal_ has joined #openstack-infra16:58
*** hogepodge has quit IRC16:58
*** eranrom_ has joined #openstack-infra16:58
pabelangermrhillsman: osic-cloud8 now online16:59
pabelangerwill show traffic once we start allocating nodes16:59
*** jpich has quit IRC16:59
*** trown is now known as trown|lunch17:00
pabelangerraddaoui: ^17:00
clarkbpabelanger: thats still using cloud1 mirror though right?17:00
*** eranrom has quit IRC17:00
*** kzaitsev_mb has joined #openstack-infra17:00
pabelangerclarkb: yes, still having some network issues launching the mirror17:00
clarkbok I am gonna grab caffeine and breakfast but let me know if you want more eyes on that17:01
*** jcoufal has quit IRC17:01
raddaouinice pabelanger was waiting for it17:02
*** hogepodge has joined #openstack-infra17:03
*** florianf has quit IRC17:04
*** andymccr has quit IRC17:04
mrhillsmanpabelanger awesome to hear17:05
*** bradjones has quit IRC17:06
mrhillsmanpabelanger what issues you having with networking?17:06
*** AnarchyAo has joined #openstack-infra17:06
mrhillsmanare you in talks with raddaoui regarding?17:06
pabelangernot yet, just started an hour ago17:06
*** Hal has quit IRC17:06
*** yamahata has joined #openstack-infra17:07
pabelangerbasically, want to see if we can launch the server you created using the external-v4 network and address17:07
*** andymccr has joined #openstack-infra17:07
mrhillsmanok cool17:07
*** reed has quit IRC17:07
*** reed has joined #openstack-infra17:07
pabelangerright now, we fail to ssh for some reason.17:07
pabelangerUnable to connect to port 22 on
mrhillsmanraddaoui ^17:07
pabelangerbut haven't debug yet17:08
mrhillsmanhe may be on lunch17:09
pabelangerya, no rush17:09
mrhillsmangood deal, i'll be around if needed :)17:09
SamYaplemrhillsman: youre always needed17:09
openstackgerritMerged openstack-infra/openstackid-resources: Upgrade Laravel Version and ORM Framework  https://review.openstack.org/32230717:10
mrhillsmanyou are everywhere17:10
*** vhosakot has quit IRC17:10
*** florianf has joined #openstack-infra17:10
irtermitehe kind of is mrhillsman17:10
clarkb2016-09-06 16:22:07,522 INFO nodepool.image.build.ubuntu-xenial: Successfully installed glean-1.6.1 and that image build finished a few minutes ago17:11
SamYaplei cloned myself17:11
*** armax has quit IRC17:11
mrhillsmani'm b...^17:11
clarkbpabelanger: ^ so if we can just get that into rax we should be good17:11
irtermitepretty sure I just watched SamYaple reply to 4 different chans and 2 different servers at once17:11
SamYapleirtermite: just the channels you are in17:11
mrhillsman"i'm beginning to think he has clones"17:11
*** bradjones has joined #openstack-infra17:11
*** bradjones has quit IRC17:11
*** bradjones has joined #openstack-infra17:11
irtermiteapparently he only signs into chans i am in17:11
mrhillsmani could see him being the stalker type17:12
*** sdake_ has joined #openstack-infra17:12
SamYapleim not arguing but i still feel i need to somehow defend myself17:12
SamYapleall i got17:12
irtermiteif he shows up in the pm between cloudnull and me, I'm closing my laptop and going home17:12
*** isaacb has quit IRC17:12
SamYaplei broke irc!17:13
SamYaplealright that enough off topic17:13
pabelangerclarkb: when you are back, maybe you can help me understand why clouds with less resource (10 max-server: infracloud, osic-cloud8) don't get allocated nodes, where we large capacity still get allocations17:13
* irtermite throws up arms and runs away17:13
clarkbpabelanger: nodepool allocates based on a ratio of quotas17:13
clarkbpabelanger: this way we fill up all of the providers at roughly the same time (in theory)17:14
AJaegernikhil: thanks - I don't think we need to backport them.17:14
pabelangerclarkb: Ah, that explains it17:14
clarkbpabelanger: so when demand is low only those with high quotas get allocations17:14
AJaegernikhil: it wouldn't harm to backport ;)17:15
pabelangerclarkb: Ya, now that internap-mtl01 is 150, it is getting allocated more nodes17:15
*** sdake has quit IRC17:15
nikhilAJaeger: ack... so may be see later :)17:15
raddaouipabelanger: still unable to connect to
anteayaclarkb: pabelanger should that algorithm be tweaked?17:15
*** pradk has joined #openstack-infra17:16
clarkbyay updated openstackclient can talk to neutron now17:16
clarkbanteaya: no I think thats probably still a desireable behavior17:16
pabelangerraddaoui: let me try again17:16
anteayaI mean we _do_ value all our host providers, not just those with the largest numbers17:16
anteayawould be great if our algorithm supported that value17:16
anteayamrhillsman: what do you mean ha?17:16
clarkbanteaya: pabelanger one reason for it is to avoid being our own noisy neighbors unless we have to17:16
anteayaclarkb: okay that is fine17:17
anteayaI just want to ensure _all_ our cloud providers feel valued17:17
clarkbbut also it means that we don't completely throttle the small contributor 24/7 while we have plenty of extra capacity in larger cloud17:17
clarkbin theory the cost is spread evenly17:17
anteayasince we need them all and are grateful for their donations17:17
* anteaya is grateful17:17
pabelangerraddaoui: creating server now17:17
*** vhosakot has joined #openstack-infra17:17
anteayaclarkb: okay that is fine17:18
anteayaclarkb: thanks for taking the time to share your thoughts with me17:18
*** rossella_s has quit IRC17:18
mrhillsmandon't mind me anteaya17:18
anteayamrhillsman: as you wish17:18
*** dizquierdo has quit IRC17:18
clarkbpabelanger: rcarrillocruz one thing I have noticed about our security group rules now that openstackclient is working. We actually never want the group to group rules. They perform poorly in the database and at least one cloud has called me at 3am to delete the rules17:19
* anteaya can ignore many folks and doubly so at their request17:19
pabelangerclarkb: anteaya: Ya, it is only noticeable right now because we have 2 providers (infracloud-vanilla, osic-cloud8) with 10 max-servers. Once we bring them up to capacity, I think things will work as expected17:19
*** rossella_s has joined #openstack-infra17:19
clarkbpabelanger: rcarrillocruz so we will want to add that to the clouds management playbook17:19
*** ddieterly is now known as ddieterly[away]17:19
pabelangerraddaoui: Oh, new expection17:20
raddaouipabelanger: I think it is because of the image you're using, try with the other image it will work fine17:20
*** asselin_ has joined #openstack-infra17:20
pabelangerraddaoui: http://paste.openstack.org/show/567232/17:20
anteayapabelanger: oh, my interpretation from clarkb's explanation was that things already were working as expected17:21
*** Benj_ has joined #openstack-infra17:21
clarkbpabelanger: rcarrillocruz both osic-cloud8 and infracloud need to have their group to group security group rules deleted17:21
pabelangeranteaya: Yes, agreed17:21
anteayapabelanger: okay thanks17:22
pabelangerraddaoui: sure, let me try with that image. But the image we are using now is just the ubuntu-cloud image17:22
openstackgerritOmer Anson proposed openstack-infra/project-config: Add Dragonflow fullstack test for ML2 module  https://review.openstack.org/36603317:23
pabelangerclarkb: okay, did we create them with cloud-launcher or was that done manually?17:24
clarkbpabelanger: its part of neutrons defaults which you "can't" change so as a user you have to edit them17:24
clarkbpabelanger: so the cloud launcher just needs to know how to remove them17:24
pabelangerclarkb: okay17:25
clarkb(you actually can change them and mgagne does for internap but its not supported as part of openstack proper aiui)17:25
*** stevemar has joined #openstack-infra17:26
*** armax has joined #openstack-infra17:26
mgagneclarkb: we don't provide group to group rules in our defaults for reasons mentioned above.17:26
*** mriedem1 has joined #openstack-infra17:26
mgagneI can double check and make sure they aren't present in mtl0117:26
*** thorst has joined #openstack-infra17:26
clarkbmgagne: yup I think its teh right thing to do but we can't rely on every cloud deciding that neutrons defaults are wrong and should be hacked to fix them :)17:26
clarkbmgagne: I already checked at at least in our test account they aren't there17:27
mgagnesure, I think it's a good idea to enforce your rules and purge the unmanaged ones17:27
clarkbpabelanger: the problem is that any time you add a new instance or delete an instance in a security group you have to update all of the other instances in that group17:28
*** mriedem has quit IRC17:28
*** e0ne has joined #openstack-infra17:28
*** sdake_ has quit IRC17:28
clarkbpabelanger: and since nodepool add deletes in such a tight loop that can put an unnecessary strain on the control plane of the cloud. Its redundant too since we already allow all traffic regardless17:28
*** sdake has joined #openstack-infra17:29
*** akshai has quit IRC17:30
*** tqtran has joined #openstack-infra17:31
clarkbmrhillsman: fyi got "HttpException: Gateway Timeout, <html><body><h1>504 Gateway Time-out</h1>" running openstack security group rule show 2fe7489d-3eaf-4833-b6ca-214497188018 against osic-cloud8 a couple minutes ago17:31
*** aspiers has quit IRC17:31
pabelangerya same17:31
*** zz_ja is now known as zz_zz_ja17:32
*** tonytan4ever has quit IRC17:33
openstackgerritMerged openstack-infra/project-config: Include both Trusty and Xenial gate-requirements-integration-dsvm  https://review.openstack.org/36584017:33
*** jheroux has joined #openstack-infra17:34
clarkbprometheanfire: tonyb ^17:34
prometheanfireyep, saw that17:34
prometheanfirethe oslo-db bump will probably be the first to test that17:34
prometheanfireif it's ever released...17:35
*** thorst has quit IRC17:35
*** vhosakot has quit IRC17:36
*** jaosorior has quit IRC17:36
*** e0ne has quit IRC17:36
*** acoles is now known as acoles_17:38
*** harlowja has joined #openstack-infra17:38
*** kzaitsev_mb has quit IRC17:38
*** amotoki has quit IRC17:39
*** sarob has joined #openstack-infra17:39
*** thiagolib has joined #openstack-infra17:40
*** Ravikiran_K has quit IRC17:40
fungiokay, i'm back. my isp seems to be offline, but thankfully i have a backup connection through my mobile phone provider17:40
*** florianf has quit IRC17:40
clarkbfungi: did the hurricane turn around and come back to NC?17:41
funginah, but this _is_ probably related to post-storm utility work17:41
fungiwe were lucky. some other neighborhoods out here are only just getting power back17:41
fungiand a number of roads are out, which makes it harder for the utility trucks to do stuff quickly17:42
anteayafungi: I'm glad you fared well17:42
anteayafungi: indeed17:42
*** spzala has joined #openstack-infra17:43
*** tonytan4ever has joined #openstack-infra17:44
*** sarob has quit IRC17:44
fungicorvus: i've put https://review.openstack.org/361463 on the agenda as a proposed spec superseding its counterpart from last week. let me know if i should not have17:44
*** vhosakot has joined #openstack-infra17:44
fungiproposed spec adjustment anyway17:44
*** notnownikki has joined #openstack-infra17:47
*** oomichi- is now known as oomichi17:49
*** chem has quit IRC17:49
*** sdake has quit IRC17:49
*** chem has joined #openstack-infra17:49
anteayaI was just reading that agenda, was wondering what context was for superseded, thank you17:51
anteayawell actually I was just outside watching a red squirrel bury cedar cones, and missing jog017:52
fungiyeah, that spec change has replaced the one we discussed in last week's meeting (if you look at the old change it's in an abandoned state referencing the new change linked in our agenda for this week)17:52
anteayabut before that I was looking at the agenda17:52
anteayafungi: ah thank you17:52
*** mriedem1 is now known as mriedem17:53
*** florianf has joined #openstack-infra17:54
*** jamesdenton has joined #openstack-infra17:55
*** sdake has joined #openstack-infra17:56
*** rderose_ is now known as rderose17:56
*** akshai has joined #openstack-infra17:57
*** spzala has quit IRC17:57
*** e0ne has joined #openstack-infra17:58
*** bknudson has joined #openstack-infra18:00
*** spzala has joined #openstack-infra18:00
*** thiagolib has quit IRC18:00
*** persia has quit IRC18:01
*** thiagolib has joined #openstack-infra18:01
*** mrjk has quit IRC18:01
*** maishsk has joined #openstack-infra18:02
*** dkehn__ has quit IRC18:03
*** dkehn_ has quit IRC18:03
*** maishsk has quit IRC18:03
*** dkehn has quit IRC18:03
*** piet has quit IRC18:04
*** spzala has quit IRC18:04
*** crinkle_ is now known as crinkle18:07
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck: Add query for cells v1 snapshot race bug 1620761  https://review.openstack.org/36627718:08
openstackbug 1620761 in OpenStack Compute (nova) "test_create_second_image_when_first_image_is_being_saved intermittently times out in teardown in cells v1 job" [Undecided,Confirmed] https://launchpad.net/bugs/162076118:08
mrhillsmanclarkb check now18:08
clarkbmrhillsman: just ran the command 3 times and it seems happy each time18:08
*** dkehn has joined #openstack-infra18:09
*** oanson has joined #openstack-infra18:10
*** _nadya_ has quit IRC18:10
openstackgerritOmer Anson proposed openstack-infra/project-config: Add Dragonflow fullstack test for ML2 module  https://review.openstack.org/36603318:11
mordredodyssey4me: I've made some progress on the role import topic18:11
*** trown|lunch is now known as trown18:12
mordredodyssey4me: turns out with two patches there is support for triggering an import that mostly works18:12
*** mrjk has joined #openstack-infra18:13
raddaouipabelanger: try again and see if you can ssh18:13
*** Na3iL has quit IRC18:13
clarkbfungi: pabelanger I am going to do a little digging into the size of log dirs for various jobs to see if we can determine why we filled up our log server recently18:14
*** dkehn__ has joined #openstack-infra18:15
*** dkehn_ has joined #openstack-infra18:16
anteayaclarkb: is that /srv/static/log? http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=717&rra_id=all18:17
*** degorenko is now known as _degorenko|afk18:17
clarkbanteaya: yes, we had to reduce our retention period in order to make space there again18:18
*** pradk has quit IRC18:18
clarkband it seemed to happen relatively quickly like all of a sudden we had way more log data being stored18:18
fungiclarkb: thanks! in the short term, we've manually "expired" logs older than 2 months. which is unfortunate but was better than leaving things broken18:18
* clarkb discovers the magic is wget -e robots=off18:19
anteayayay magic18:19
*** ddieterly[away] has quit IRC18:19
anteayamy read on the graph is the rate changed near the beginning of June18:20
fungiclarkb: i don't think it was super sudden. we began curving upward around may-june18:20
anteayaand then hit the ceiling in August18:20
anteayafungi: I concur18:20
openstackgerritChris Krelle proposed openstack-infra/glean: Adjust the way we wait for interfaces to become available  https://review.openstack.org/35947118:20
clarkbfungi: in that graph we shot up 2TB in 2 weeks18:20
fungiand yeah, august was when things broke and i expired logs older than 2 months which only freed up about 10%18:21
fungier, logs older than 3 months18:21
fungilast week is when we then had to knock that down to 2 months18:21
clarkbbut yes looks like things generally started growing in an upward trajectory around june18:21
clarkbheh our help footer for the logs totally breaks recursive downloading of logs. So many 404s18:21
openstackgerritMike Fedosin proposed openstack-infra/project-config: Add py35 and pypi jobs to Glare and its client  https://review.openstack.org/36628118:22
fungialso my config management change to drop retention down to 3 months didn't work because we still have a dead copy of the log compression and expiration script in system-config but are now using the copy in openstackci18:22
anteayaclarkb: ha ha ha18:22
fungiit's on my list to remove the dead copy and update the live copy before we run out of space again18:23
*** berendt has joined #openstack-infra18:23
anteayaI also didn't know we have a repo in the infra namespace that is not an infra repo18:24
anteayaI learn something new everyday18:24
anteayaI think we as a group are just trying to test tonyb18:24
anteayaand see how many hats he can wear at the same time18:24
anteayadhellmann: ^^18:25
*** spzala has joined #openstack-infra18:25
anteayafungi: can we get a kick in -meeting please?18:25
clarkb88MB just for neutron logs might help explain it18:25
anteayaor any other channel op?18:26
dolphmabusive user in #openstack-meeting: antipsychiatry18:26
clarkbespecially since we run more neutron now18:26
lbragstadanteaya ++ thanks18:26
anteayacan we ban that user please?18:26
clarkbalso we have 46MB of libvirt logs each run so between neutron and libvirt thats 120 MB of logs18:26
anteayaI have seen them previously, same nick18:26
anteayathank you18:26
*** oomichi has quit IRC18:26
*** oomichi has joined #openstack-infra18:27
*** senk has quit IRC18:27
*** shardy has quit IRC18:27
*** rhallisey has quit IRC18:28
*** spzala has quit IRC18:28
anteayaclarkb: yes that may explain it18:28
*** spzala has joined #openstack-infra18:28
*** oanson has quit IRC18:29
anteayathe move to neutron as devstack default happened this release though I can't recall the timing18:29
mordredclarkb: did the neutrons start logging more things?18:29
mordredor is it just the neutron-by-default you think18:29
mordredanteaya: I cannot remember the timing either18:29
mriedemi'm seeing several failures with "error: command 'x86_64-linux-gnu-gcc' failed with exit status 1" on different nodes, is that known?18:29
anteayaarmax: dougwig did the neutrons start logging more things?18:29
armaxanteaya: what do you mean?18:29
clarkbmordred: I think maybe its a combo? also not sure why we need all of that libvirt logging seems excessive18:30
anteayaarmax: our log storage filled up faster than expected18:30
anteayaarmax: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=717&rra_id=all18:30
mordredclarkb: I think I remember someone coming in and chatting about saving the libvirt logging18:30
mordredclarkb: something something debugging something something18:30
anteayaarmax: clarkb is looking into why that is18:30
mriedemthe libvirt logging was needed for live migration debug18:30
*** rhallisey has joined #openstack-infra18:30
anteayaarmax: we moved to running neutron as a devstack default18:30
clarkbmriedem: this isn't a live migration job though18:30
clarkbmriedem: I have no issues turning it on where its needed but it apepars to be on everywhere18:30
anteayaarmax: clarkb notes taht neutron logs are large18:30
mriedemclarkb: yeah i know18:31
anteayaarmax: we are wondering if neutron logs have always been large and we didn't notice18:31
*** roxanagh_ has joined #openstack-infra18:31
anteayaarmax: or if they have increased of late18:31
anteayaarmax: and hello, nice to see you :)18:31
armaxanteaya: not that I am aware, we tend to keep an eye on it to make sure we don’t log trash18:31
armaxbut one thing to consider is that server side we one big log18:31
armaxwhereas other projects are smaller but multiple log files18:31
anteayaarmax: okay thank you18:32
armaxlike nova has api, sched, ctrl, cpu18:32
*** thorst has joined #openstack-infra18:32
clarkboh hrm my downloads may be inflating the logs18:32
clarkbin which case it would be slightly flawed numbers checking18:32
*** pvaneck has joined #openstack-infra18:32
armaxanteaya: I’ll go over to see if there’s some particular bad offender18:32
*** jamesdenton has quit IRC18:32
armaxand trim it down if we can18:32
anteayaarmax: thanks we appreciate you taking a look18:33
clarkbya its inflating them18:33
clarkbthats annoying. Let me rerun with better infoa18:33
clarkbthe expanded numbers are still reatively interesting to each other18:33
fungiif it were me, i'd just ssh into static.o.o and check files as they exist on its filesystem, since you have shell access18:35
*** roxanagh_ has quit IRC18:35
clarkbya but I like doing it the other way to show its possible and not unreasonably difficult18:35
*** pradk has joined #openstack-infra18:35
fungifair enough. it's a worthwhile exercise18:35
fungii just wouldn't want to try to mirror all ~12tb18:36
anteayathat would take a while18:36
anteayaexpecially using your phone as your isp connection18:36
fungiindeed. this rather unfortunate outage is preventing me from easily building the last batch of contributor invites for barcelona18:37
*** asettle has joined #openstack-infra18:37
*** piet has joined #openstack-infra18:37
anteayado you think you will have service tomorrow?18:37
fungiso that may have to wait for tomorrow depending on how soon they get things back up and running again18:37
anteayahopefully by tomorrow then18:37
lbragstadanteaya fungi we seem to have the same use from the #openstack-meeting channel spamming #openstack-keystone now, too18:39
*** thorst has quit IRC18:39
anteayalbragstad: sigh18:40
fungilbragstad: yeah, i added them to the akick list for that channel now too, though they only /join'ed long enough to spam you and then /part'ed again immediately after18:41
lbragstadfungi ah - got it.. thank you!18:41
anteayafungi: do we have a way of blacklisting a user like this one from all our channels?18:41
*** oomichi has quit IRC18:42
*** asettle has quit IRC18:42
*** baoli has joined #openstack-infra18:42
fungianteaya: not easily. it's something we could probably add to accessbot but the delay involved in that may make it less attractive18:42
*** baoli_ has quit IRC18:42
openstackgerritJay Faulkner proposed openstack-infra/project-config: Publish install guide for Ironic  https://review.openstack.org/36629018:44
*** asettle has joined #openstack-infra18:45
anteayafungi: understood18:45
*** asettle has quit IRC18:46
clarkbfungi: oh ya definitely don't want to try and get all 12TB, but a small representative portion of the logs should be easy enough18:46
*** priteau has joined #openstack-infra18:48
*** oomichi has joined #openstack-infra18:48
sshnaidmyolanda, pabelanger could you please enable rh2? https://review.openstack.org/35316318:49
pabelangersshnaidm: I'd like to land https://review.openstack.org/#/c/357308/ first so we can remove the private infrastructure from nodepool logs18:50
*** weshay_bbiab is now known as weshay18:51
pabelangersshnaidm: bascially, 3 projects in tripleo-test-cloud-rh2, openstackci (afs mirror), openstackzuul (nodepool), tripleo (for private infrastruture).18:52
*** oomichi has quit IRC18:52
sshnaidmpabelanger, you will not see the results though (or failures), ovb rh1 cloud is disabled18:52
*** oomichi has joined #openstack-infra18:52
mordredinfra-root, Shrews: we're still running nodepool on a git checkout - I'm going to tag a shade release unless there is a rancorous outcry complete with wailing and gnashing of teeth18:52
pabelangersshnaidm: there is no ovb for tripleo-test-cloud-rh2?18:52
* fungi musters only a very limited quantity of gnashing18:53
*** mwhahaha has quit IRC18:53
sshnaidmpabelanger, now it doesn't run, bnemec have this patch for enabling it, we're without any ovb now18:53
sshnaidmpabelanger, rh2 is out of configs and rh1 suffers from heat problem and is disabled18:54
*** mwhahaha has joined #openstack-infra18:54
pabelangersshnaidm: even so, the issue is the private infrastructure in the nodepool project. If we can create openstackzuul for tripleo-test-cloud-rh2, I think that is all we need today to bring it back online18:54
sshnaidmpabelanger, I think it's safer to land your patch on working cloud so we can see any problems right after18:54
Shrewsmordred: 2d70f8b?18:55
mordredShrews: yah18:55
mordredShrews: as 1.12.0 - yah?18:55
sshnaidmpabelanger, sorry, what do you mean by "create openstackzuul"?18:55
Shrewsgeez. how did we get so many changes in since the last tag?18:56
Shrewsmordred: yeah18:56
openstackgerritMerged openstack-infra/system-config: launch-node.py: restart interface  https://review.openstack.org/35941618:56
openstackgerritMerged openstack-infra/system-config: launch-node.py: save key when failing early  https://review.openstack.org/35941718:57
clarkbfungi: and I have discovered the issue with doing it without root. If you set accept-encoding to gzip then it can't read the index.html files bceause they are gzipped :)18:57
mordredShrews: because evil18:57
mordredShrews: that's how18:57
fungiclarkb: ouch18:57
clarkbin any case it looks like things are actaully getting gzipped serverside which is good. Once inflated libvirt and neutron account for the majority of the disk use from what I can tell18:57
pabelangersshnaidm: this is the current server list in the openstack-nodepool project: http://paste.openstack.org/show/567251/18:57
*** ddieterly has joined #openstack-infra18:58
clarkbtrying to see if the compression ratio preserves that on the compressed side of things18:58
pabelangersshnaidm: Which is causing warnings to appear in nodepool.o.o.  If you create a new project, openstackzuul, which shared network access to the openstack-nodepool project, we'd update nodepool.o.o to use that18:58
clarkbarmax: if you inflate all of the screen-q-* services logs you get almost 100MB of logs for neutron on the normal neutron dsvm tempest full job fwiw. I am not sure if that is considered normal or not18:59
rcarrillocruzpabelanger, clarkb : so, neutron sqlite18:59
rcarrillocruzis there anything in review?18:59
pabelangerrcarrillocruz: nope, my fail18:59
rcarrillocruzotherwise i can poke at it tomorrow18:59
pabelangerrcarrillocruz: got confused by compute nodes18:59
armaxclarkb: what about other server projects?18:59
rcarrillocruzso you found the issue ?18:59
armaxwhat’s the target you feel comfortable with?18:59
rcarrillocruzclarkb: yeah, i remember that from mid cycle19:00
fungiit's that (weekly infra team meeting) time again! find us in #openstack-meeting for the next hour19:00
rcarrillocruzwe had a playbook for the default secgroup rules19:00
rcarrillocruzi think we can fix that by putting pertinent absent os_security_group_rule on the cloud launcher19:00
mordredrcarrillocruz: that should be a thing cloud-launcher should be able to express, yeah?19:00
anteayameeting time19:00
rcarrillocruzmordred: yup19:01
clarkbarmax: I am not sure I have a target, just pointing out that once uncompressed you have a lot more logs than the other services. This may mean that your logs are too chatty or maybe its the appropriate amount19:01
clarkbarmax: mostly just sharing data at this point.19:01
clarkbarmax: but if you consider that a cloud running for half an hour generates 100MB of logs just for neutron translate that to production running for months and you may not be the happiest ops person19:02
mordredShrews: ok. shade 1.12.0 relased19:02
mordredShrews: I, for one, am looking forward to whatever carnage we create with that19:02
*** jascott1 has quit IRC19:04
*** jascott1 has joined #openstack-infra19:04
openstackgerritMonty Taylor proposed openstack-infra/nodepool: Update shade depend to 1.12.0  https://review.openstack.org/36629419:05
armaxclarkb: understood, most clouds don’t run in debug mode though19:05
clarkbarmax: I have heard the opposite :)19:05
clarkbarmax: most ops people I talk to say they always run with debug on because its the only way to debug issues19:06
armaxclarkb: let me look if there’s any fat that can be trimmed19:06
armaxclarkb: knowing about the size of other projects’ logs would be good though, just to have a sense of how things compare to one another19:08
*** e0ne has quit IRC19:08
*** akshai has quit IRC19:09
*** hasharAway is now known as hashar19:09
clarkbarmax: ya I am rerunning my crawl to get that comparative data19:09
clarkbarmax: I can post a paste when I have it put together19:09
*** Sukhdev has joined #openstack-infra19:09
*** pcaruana has quit IRC19:11
*** Guest78236 is now known as melwitt19:14
openstackgerritMerged openstack-infra/infra-specs: Zuulv3: drop variable interpolation and add nodesets  https://review.openstack.org/36146319:14
*** mriedem has quit IRC19:16
*** dahoo has joined #openstack-infra19:18
armaxclarkb: thanks19:20
*** amitgandhinz has quit IRC19:20
*** pradk- has joined #openstack-infra19:20
*** amitgandhinz has joined #openstack-infra19:20
*** pradk has quit IRC19:21
dhellmannmordred : did you see that the shade tarball job failed? it looks like a missing dev package for ffi -- http://logs.openstack.org/b1/b1f35b4bce91ae1da5070cd3fc0063e6632486db/release/shade-tarball/3c56972/console.html19:21
openstackgerritJeremy Stanley proposed openstack-infra/infra-specs: Prioritize Docs Publishing via AFS  https://review.openstack.org/36630319:21
*** pradk- is now known as pradk19:22
rcarrillocruzdhellmann: i believe pabelanger pushed a change earlier to include libffi-dev19:22
dhellmannrcarrillocruz : ok, cool. I saw the failure email and wanted to make sure someone else had noticed it.19:23
pabelangerrcarrillocruz: dhellmann: Oh, that was for project-config. We likely need to update shade now too19:23
mordredpabelanger: oh goodie19:24
mordreddhellmann: thanks!19:24
*** tesseract- has quit IRC19:25
pabelangerguess we don't have wheels for cffi19:25
openstackgerritMonty Taylor proposed openstack-infra/shade: Add libffi-dev to bindep.txt  https://review.openstack.org/36630419:25
mordredpabelanger: yah? ^^19:25
*** akshai has joined #openstack-infra19:25
*** maishsk has joined #openstack-infra19:26
pabelangeronly see cff-1.7.0 right now19:26
*** persia has joined #openstack-infra19:27
pabelangerlooks like 1.8.2 was release today19:27
*** maishsk has quit IRC19:27
*** ilyashakhat_mobi has joined #openstack-infra19:27
*** kzaitsev_mb has joined #openstack-infra19:27
*** schang_ has joined #openstack-infra19:28
*** amitgandhinz has quit IRC19:30
*** schang_ has quit IRC19:30
openstackgerritMatthew Treinish proposed openstack-infra/system-config: Add lpmqtt to firehose deployments  https://review.openstack.org/36331319:30
clarkbarmax: http://paste.openstack.org/show/567259/ from one sample of the neutron tempest dsvm full job19:30
*** amitgandhinz has joined #openstack-infra19:30
clarkbarmax: also that was from the gate not check to avoid broken change causing weirdness19:30
*** roxanagh_ has joined #openstack-infra19:32
*** coolsvap_ has quit IRC19:32
mtreinisharmax, clarkb: I gave up trying to fix that right after I started with: https://review.openstack.org/#/c/330848/ (although I'm curious how many -1s I can get)19:33
*** maishsk has joined #openstack-infra19:33
*** claudiub has quit IRC19:34
*** Apoorva has quit IRC19:34
*** roxanagh_ has quit IRC19:36
*** thorst has joined #openstack-infra19:36
*** maishsk_ has joined #openstack-infra19:37
*** maishsk_ has quit IRC19:37
*** maishsk has quit IRC19:38
*** maishsk has joined #openstack-infra19:39
*** ilyashakhat_mobi has quit IRC19:39
*** maishsk has quit IRC19:39
*** rcernin has joined #openstack-infra19:41
rcarrillocruzso pabelanger19:41
rcarrillocruzred herring the sqlite19:41
rcarrillocruzit was other thing you were looking19:41
rcarrillocruzwe don't know what those timeouts are coming from, yeah?19:41
pabelangerrcarrillocruz: Ya, I confused which server was running neutron-server19:41
rcarrillocruzor we have some suspicion already19:41
pabelangernot sure right now. I did see an error in neutron logs, but didn't match up the timestamps19:42
openstackgerritMatthew Treinish proposed openstack-infra/system-config: Add lpmqtt to firehose deployments  https://review.openstack.org/36331319:42
pabelangerI'll keep poking at the logs but clarkb suggested reaching out to neutron team, see if they have any suggestions19:43
rcarrillocruzi wip'd the bump change19:43
rcarrillocruzpabelanger: and sorry, catching up since i was out, the default secgroup thing you haven't poked at it or have you?19:44
rcarrillocruzis to start looking at it tomorrow in my morning19:44
*** thorst has quit IRC19:45
pabelangerrcarrillocruz: not yet. I wasn't sure the syntax we should be using19:45
pabelangerrcarrillocruz: Oh, should we remove infracloud mirror server from cloud-launcher right now?  Since we manually provisioned it19:46
*** ilyashakhat_mobi has joined #openstack-infra19:46
*** sdake has quit IRC19:46
*** smarcet has quit IRC19:46
rcarrillocruzeither way works, os_server is pseudo-idempotent, it won't recreate the server if there's already a server with that name19:47
rcarrillocruzi'll tackle os_server refactor this week19:47
*** weshay is now known as weshay_mtg19:47
rcarrillocruzi'm hoping to have the pre/post actions on the launcher draft based off it for the mid cycle19:47
armaxmtreinish: problem is, if we remove it, what happens if you need it?19:48
pabelangerrcarrillocruz: okay, cool19:48
*** rcernin has quit IRC19:48
*** pradk has quit IRC19:49
mtreinisharmax: so you log a prettified json blob (which can be quite big sometimes)  ~600 times during a dsvm run in the off chance you need it once and it's useful for debug19:50
*** ilyashakhat_mobi has quit IRC19:50
mtreinishit's all about striking a balance, and that definitely feels like it's too far past the line19:50
armaxmtreinish: what’s the proverb? it rains when you don’t have an umbrella?19:50
*** andrey-mp has joined #openstack-infra19:54
andrey-mpHi, can anyone from core check this simple patch? https://review.openstack.org/#/c/357636/19:54
*** tqtran has quit IRC19:55
*** eharney has quit IRC19:56
*** devananda is now known as devananda|travel19:57
*** AnarchyAo has quit IRC19:58
*** asettle has joined #openstack-infra19:58
*** asettle has quit IRC19:58
*** davidlenwell has quit IRC19:59
*** florianf has quit IRC20:00
clarkbfungi: I poked at the puppet/ansible/kolla logs to see if maybe their deployments had runaway with log generation and collection but they all do much less than devstack does20:00
zaropleia2: browser load in itself isn't going to be a good test for gerrit.20:01
clarkbbut they don't reliably compress all of their files like devstack does20:01
*** amitgandhinz has quit IRC20:01
pleia2zaro: yeah, it would have to do certain actions20:01
zaropleia2: the load has to be from the data20:01
* pleia2 nods20:01
*** amitgandhinz has joined #openstack-infra20:01
pleia2well, I'll make some time to figure out what's out there and touch base with you to see if anything makes sense20:01
*** gyee has quit IRC20:01
clarkbprobably worthwhile asking them to update their jobs to deal with compressed logs just because its a simple thing to do20:01
zaropleia2: sounds good20:02
*** ilyashakhat_mobi has joined #openstack-infra20:02
fungiclarkb: i think you need to look at a sample of job logs from at least a week ago20:02
fungito factor out the delayed compression20:02
openstackgerritThomas Goirand proposed openstack-infra/project-config: Add missing deb repo 1/5  https://review.openstack.org/36631820:03
*** sdake has joined #openstack-infra20:03
*** AnarchyAo has joined #openstack-infra20:04
wznoinskhi infra20:04
wznoinskssh review.openstack.org -p 2941820:04
andrey-mpclarkb: fungi: pleia2: please check this one-line changeset... https://review.openstack.org/#/c/357636/20:04
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Don't send metrics to graphite in failed jobs  https://review.openstack.org/36511420:05
pleia2andrey-mp: we can't merge that while meetings are going on20:05
pleia2it restarts the bot, breaks in progress meetings :\20:05
wznoinskgit review doesn't work over ssh to me and this is what I have when done manually, normally I saw 'non-interactive logons not permitted' AFAIR http://paste.openstack.org/show/567264/,20:05
andrey-mppleia2: ok, I've got it20:06
clarkbfungi: ya I am saying we shouldnt have to wait for the delayed compression. they should compress upfront like devstack gate does20:06
pleia2andrey-mp: I'll keep it on my radar for when we have a quite time meeting-wise20:06
pleia2quiet too20:06
andrey-mppleia2: thank you!20:06
pleia2sure thing20:06
fungiclarkb: i agree on that, but it's unlikely to be the cause of our longer-term consumption which is the better thing to get a handle on20:06
*** andrey-mp has left #openstack-infra20:06
mordredwznoinsk: put the -p 29418 before the host20:07
clarkbfungi: yup agreed20:07
mordredwznoinsk: ssh -p 29418 review.openstack.org - the command in your paste is saying "please run thecommnad '-p 29418'" on review.openstack.org20:07
*** eharney has joined #openstack-infra20:08
mordredwznoinsk: nevermind. I lied20:08
wznoinskmordred: didn't think it would make a diff, it didn't20:08
openstackgerritIan Wienand proposed openstack-infra/glean: Refactor side-effect functions  https://review.openstack.org/32022520:08
openstackgerritIan Wienand proposed openstack-infra/glean: Patch glean.cmd.open for unit testing  https://review.openstack.org/32022420:08
openstackgerritIan Wienand proposed openstack-infra/glean: Use mock patching for most functions  https://review.openstack.org/31846420:08
mordredyah. I just suck today20:08
openstackgerritIan Wienand proposed openstack-infra/glean: Use OS_LOG_CAPTURE to get log output  https://review.openstack.org/36580720:08
openstackgerritIan Wienand proposed openstack-infra/glean: Add selinux context manager for writing files  https://review.openstack.org/30435720:08
wznoinskmordred: yeah, but a good guess it would maybe try to run '-p 29418' as a command ;-)20:08
ianwmordred / pabelanger: ^^ i feel like that series is in pretty good shape ... and i'd like to not have to rebase again :)20:09
*** piet has quit IRC20:09
clarkbwznoinsk: mordred that looks like your ssh executable is a bash script that isnt working20:10
clarkblook at what ssh actually is on that machine20:10
zigofungi: Would you have time to check on this? https://review.openstack.org/36547720:11
zigoWe can't build oslo.cache without (ie: we need to update dogpile.cache, but can't because of missing tag...)20:11
*** rockyg has joined #openstack-infra20:11
fungizigo: i commented in channel earlier, but you don't want to push upstream tags or upstream commits yourself. this is a use case for our track-upstream option20:11
*** davidlenwell has joined #openstack-infra20:12
fungiwe do this with our fork of the gerrit repo, for example. upstream commits are replicated into separate branches and upstream tags are also replicated. then you can just propose a merge for review from those into your packaging branch20:13
*** edmondsw has quit IRC20:13
zigofungi: In many cases, I need to re-tag upstream tags. For example, dogpile.cache has rel_0_6_2, and we need 0.6.2 instead. Many upstream have the silly habbit to prefix with a "v". Also, for the more openstack-ish case, we need to rename things like as 4.0.0_b2, so that we then replace _ by ~.20:14
openstackgerritArmando Migliaccio proposed openstack-infra/reviewday: Make Neutron script pull all Stadium projects  https://review.openstack.org/36143220:14
wznoinskclarkb: mordred yeah, it looks like something around my NIS configuration on the machine20:14
fungizigo: in those cases, you would sign the tag, right? i guess what you have in 365477 makes sense. earlier i was reacting more to the request to be able to re-push upstream's unsigned tags into the repo yourself20:15
zigofungi: If Gerrit mandates signing tags, then I would, but in many cases, it's not necessary, because you can just add a new tag to an existing one, and then it keeps the original signature.20:16
fungiwznoinsk: i bet whatever wrapper you have around the ssh executable is looking for what it thinks is the end of command-line options, and is more strict about parameter ordering than openssh is20:16
*** kzaitsev_mb has quit IRC20:16
*** dtardivel has quit IRC20:17
pabelangerclarkb: have a moment to look at nodepool-builder logs? I'm not sure any images are getting uploaded ATM. Last timestamp: 2016-09-06 18:13:31,560 INFO nodepool.builder: Uploading dib image id: 1014 from /opt/nodepool_dib/1014.qcow2 in ovh-bhs120:17
*** piet has joined #openstack-infra20:17
zigofungi: For example, in the deb-nova repo, look at the tag 13.0.0_b3, and you'll see Doug's pgp signature on that tag, even though he only signed
clarkbpabelanger: I think its just slow20:18
zigofungi: So, will gerrit mandate signing?20:18
clarkbpabelanger: you can strace -fp builderpid to see if its writing data20:18
fungizigo: we're generally strict about only allowing users to have pushSignedTag permission, which allows signed tags but not plain tags or lightweight tags20:19
fungistill cloning deb-nova to look at your example. i've had to switch to a mobile connection because my isp is offline today20:19
pabelangerclarkb: okay, I see write happening20:20
fungi(and since it's a clone of nova, it takes a while)20:20
zigofungi: I can show you on a more lightweight repo ! :)20:20
zigofungi: Any service with a beta release is fine ...20:20
sdakepabelanger isthere a way to override bindep.txt20:21
sdakeI see in the script it checks for existence of this file20:21
zigofungi: Tag 3.0.0_b2 in mistral for example.20:21
sdake(from the zuul output)20:21
fungizigo: what you're demonstrating there is what's called a "lightweight" tag, that's basically a symlink. presumably you didn't use -a or -s when you ran `git tag`20:22
clarkbsdake: if you dont provide your own bindep.txt we use a fallback so just pr9vide your own file20:22
sdakeclarkb just check into the repo then?20:22
sdakeclarkb cool i'll give that a go20:22
sdakeclarkb thanks :)20:22
sdakeclarkb bindep installing all kinds of stuff we dont need20:23
zigofungi: So what do you suggest I do then?20:23
fungisdake: jobs using the "install-distro-packages" builder will use any bindep.txt (or other-requirments.txt) found in the repo for the change that triggered the job, and if there isn't one, it'll use our fallback package list20:23
*** mlavalle has joined #openstack-infra20:23
*** kgiusti has left #openstack-infra20:23
*** srobert has joined #openstack-infra20:23
mlavallemtreinish: ping20:23
fungizigo: we strongly recommend against lightweight (and annotated) tags, in favor of signed tags20:23
clarkbsdake: yes the idea is to give you control while installing what we have always installed for you if you dont have a bindep.txt20:24
mtreinishmlavalle: pong20:24
fungizigo: lightweight tags are better thought of as "aliases"20:24
zigofungi: Then I'm fine with signed tags. packaging core group will do.20:24
fungizigo: they don't always refer to other tags. they can refer to any got object20:24
sdakeclarkb sounds god i'll give it a go - since it appears our jobs ar einstall-distro-packages20:24
fungier, git object20:24
zigogit it20:25
zigoer, got20:25
mlavallemtreinish: hi.... I have a question about logstash / kibana..... Does it capture LOG.debug level messages?20:25
*** ilyashakhat_mobi has quit IRC20:25
zigofungi: But I need https://review.openstack.org/#/c/365477/ to do signed tags, no?20:25
zigoOr is there a hidden way that I didn't see?20:26
mtreinishmlavalle: no20:26
pabelangerclarkb: looks like nodepool is trying to finish uploading an image to ovh-bhs1. But not sure why we haven't started the other clouds. I wonder if it has to do with my upload attempt this morning. Which is still running20:26
mlavallemtreinish: yean that was what we were suspecting..... Thanks :-)20:26
mlavallemtreinish: see you in Barcelona20:26
mtreinishmlavalle: it's too much data and of questionable value. If it's something we need to track it should be important enough for a higher log level20:26
clarkbpabelanger: we have 16 uploaf yhreads and many more uploads20:26
mtreinishmlavalle: heh, yep :)20:26
clarkbpabelanger: so all 16 are likely busy at the moment20:26
clarkbmlavalle: mtreinish ya that is an important thing to remwmber if it is important it shouldnt be debug20:27
pabelangerclarkb: Ya, I would expect that too. But logs and cacti.o.o show different20:28
mlavalleclarkb, mtreinish: agree, will work aroud it :-) Thanks again20:28
pabelangerno network traffic for 3 hours20:28
fungizigo: yes, you do. i'm sorry, i'm trying really hard to follow the tc meeting right now and conversations with half a dozen other people are making that hard. can we discuss it later? or can you discuss it with someone who's available right now?20:28
clarkbpabelanger: it may be waitibg on a qcow2 to raw conversion in ovh20:28
zigofungi: Sure, ping me when you can.20:28
clarkbpabelanger: or waiting for raxto import to glancefrom swift20:28
clarkbneither ofthose will need network traffic20:28
pabelangerwe've only uploaded to osic-cloud1, infracloud-vanilla and ovh-bhs120:29
pabelangerand osic-cloud1 and infracloud-vanilla are online20:29
pabelangerI think my image-upload task is blocking the rest of the pipe20:30
*** salv-orl_ has joined #openstack-infra20:30
clarkbpabelanger: maybe?20:33
*** salv-orlando has quit IRC20:33
clarkbI see what you are saying thought we did cloud1 then vanilla then the first image for ovh20:33
*** thorst has joined #openstack-infra20:33
clarkband cloud1 and vanilla are done20:33
wznoinskfungi: mordred: clarkb: it was something as stupid as "/bin/bash" line on its own in my ~/.cshrc and it was clashing with ProxyCommand /usr/bin/nc ... in my ssh_config20:34
clarkbpabelanger: it also looks like we may have done those earlier uploads in a serialized fashion20:34
*** tqtran has joined #openstack-infra20:34
clarkbpabelanger: you might try running a manual upload command and see if that job gets picked up right away20:35
clarkbpabelanger: I am going to check the gear status20:35
openstackgerritThomas Goirand proposed openstack-infra/project-config: Add missing deb repo 1/5  https://review.openstack.org/36631820:36
*** akscram has quit IRC20:36
*** ddieterly is now known as ddieterly[away]20:37
clarkbpabelanger: according to gear status we have all 16 threads in use20:37
clarkbpabelanger: yup it looks like we are still uploading yesterdays images20:37
*** akscram has joined #openstack-infra20:37
pabelangerwelp, that might explain it20:38
clarkbbut I am trying to get more info to confirm that20:38
pabelangerand we likely didn't abort them before doing todays20:38
clarkbimage-upload:1001 1 1 0 is one gear status20:39
clarkbthats a weird one becuse its saying we are uploading it with 1 worker but we don't haev any workers that can run it currently20:39
*** thorst has quit IRC20:39
*** jascott1 has quit IRC20:40
clarkbya we are supposedly using wto workers to upload 1013 which is yesterday's xenial image20:40
*** jascott1 has joined #openstack-infra20:40
* clarkb digs in logs to see if he can find which cloud that is to20:41
*** amitgandhinz has quit IRC20:41
*** Goneri has quit IRC20:41
*** amitgandhinz has joined #openstack-infra20:42
clarkblooks like rax-dfw and ovh-bhs120:43
pabelangerI see that now20:43
pabelangernodepool-builder.log.2016-09-04_09:2016-09-04 16:58:40,155 INFO nodepool.builder: Uploading dib image id: 1001 from /opt/nodepool_dib/1001.qcow2 in ovh-bhs120:43
pabelanger2 days ago20:43
clarkbwe may want to consider switching ovh back to raw to see if that helps20:44
*** pradk has joined #openstack-infra20:44
clarkbbut yes this is our new scaling issue I think20:44
clarkbmultiple builders may also help if its jsut a bandwidth issue but I doubt thats the only issue20:44
pabelangerclarkb: do you want to revert raw for just ovh-bhs1 or both?20:45
clarkbI would do both20:45
clarkbthen we can keep an eye on upload time over the next couple days to see if its still bad20:45
*** weshay_mtg is now known as weshay20:47
openstackgerritPaul Belanger proposed openstack-infra/project-config: Revert "Force image-type qcow2 for OVH"  https://review.openstack.org/36633320:47
pabelangerworks for me20:47
*** roxanagh_ has joined #openstack-infra20:47
matt-borlandHm, I had to fix something in freezer-web-ui, but I seem to have an invocation error (clearly not related to my change) in several of the gate jobs.20:47
*** amitgandhinz has quit IRC20:47
pabelangeronce that lands, let restart nodepool-builder so rest of images get upload as raw20:47
openstackgerritMerged openstack-infra/shade: Add libffi-dev to bindep.txt  https://review.openstack.org/36630420:48
*** amitgandhinz has joined #openstack-infra20:48
clarkbmatt-borland: always link to example logs :)20:48
matt-borlandclarkb, my pleasure! http://logs.openstack.org/58/366258/3/check/gate-freezer-web-ui-python27-ubuntu-xenial/c10042b/console.html20:48
clarkbpabelanger: hrm that doesn't need to be image-type: raw?20:48
clarkbpabelanger: I am assuming that the assumption is that we look it up in oscc but I don't knwo if the builder/dib will do that20:48
pabelangerclarkb: I believe that is the default from os-cloud-config, let me double check20:48
clarkbmatt-borland: you are using py.test which isn't a thing we currently support so not really able to help20:49
clarkbmatt-borland: I'd have to go digging and relearn a bunch of new tools20:49
matt-borlandyou're using 'you' a little liberally, but sure!20:49
pabelangererr, os-client-config20:49
clarkbmatt-borland: that project's test infrastructure20:49
matt-borlandclarkb, thank you20:50
pabelangerclarkb: http://git.openstack.org/cgit/openstack/os-client-config/tree/os_client_config/vendors/ovh.json#n1320:50
clarkbmatt-borland: basically we have a setup for collecting ocverage and running unittests its not that20:50
matt-borlandI'l take it up with whoever owns that project20:50
clarkbpabelanger: right but how does dib know to use that?20:50
clarkbpabelanger: I am going to guess nodepool looks it up and passes it along, just need to double check on that really quick20:50
clarkbyup found it.20:51
pabelangerclarkb: That is a good question, I assume nodepool does the right thing. But we can force it to raw20:51
*** roxanagh_ has quit IRC20:52
clarkbmatt-borland: it looks like py.test decided to run no tests for some reason and that was correctly determined to be a failure situation20:52
clarkbpabelanger: nodepool does the right thing. Where it sets image-type in the config it defaults to what oscc says is the format20:52
*** jascott1 has quit IRC20:52
matt-borlandclarkb, I'll take a look at it, just wondered if there was an 'aha!' moment20:52
zaroclarkb: was wondering whether you want to A+ this one for next Gerrit restart?20:52
*** jascott1 has joined #openstack-infra20:53
clarkbzaro: maybe we can split it so that we can do review-dev now make sure that build works then do review.o.o? Since I think what is on review-dev isn't exactly the same as proposed there?20:54
openstackgerritThomas Goirand proposed openstack-infra/project-config: Add missing deb repo 2/5  https://review.openstack.org/36633720:54
clarkbmtreinish: the way i would do it is create a central devstack change with a bunch of depends on. Iterate on that until you can get that change to run with python3 enabled by default and no depends on. Then add a new job that is "run against python3" or just stop running with python220:58
clarkb(likely need to continue running with python2 also hence the new job)20:58
*** Hal1 has joined #openstack-infra20:58
clarkbmtreinish: but thats a really low cost from the CI/infrastructure side that allows devs to iterate quickly on making things work20:58
mtreinishclarkb: sure there is a simple path forward, my point was it's about someone owning doing that test patch and wrangling people to look at results and figure out failures20:59
zaroclarkb: actually you can't approve yet. there's a bug that will kill gerrit on javamelody updates.  we'll need to do it at the time of a gerrit restart.20:59
mtreinishwhich if there isn't someone driving it just doesn't happen20:59
clarkbthis is the same approach I would like to see with tls and devstack except we shouldn't make a new job for tls we should just run tls everywhere always20:59
*** ddieterly[away] is now known as ddieterly20:59
clarkbmtreinish: oh ok it definirely read more like "its too hard to juggle this in gerrit"?21:00
fungiclarkb: mtreinish: any idea if our py3k wheel mirror is in better shape now?21:00
zaroclarkb: btw, i did manually install on review-dev and it did work fine. i'll mention in commit message.21:00
clarkbfungi: I didn't realize there was something wrong with it21:00
clarkbzaro: ok just make sure you WIP the change for review.o.o and leave a note about that bug21:01
clarkbzaro: so that we update it correctly21:01
mtreinishclarkb: no it wasn't a tooling issue. I just brought up d-g and devstack as examples to show that there isn't a clear owner of the effort21:01
*** mriedem has joined #openstack-infra21:01
*** raildo has quit IRC21:01
mtreinishfungi: no idea21:01
fungiclarkb: well, we weren't building one for a while, then it was added but it made the wheel build job run too long and hit our timeout so it got disabled again. there was also some concern that we were unnecessarily re-building universal wheels in each pass21:02
*** sdake has quit IRC21:02
mtreinishfungi: I wasn't aware there was an issue with the py3 mirror before21:02
*** trown is now known as trown|outtypewww21:02
clarkbfungi: huh I completely missed all that. FWIW the way our wheel mirroring works we have to at least copy all of the universal wheels21:02
fungilooks like we do independent jobs for python2 and python3 wheels now21:02
clarkbsince we bucket by the unique tuple of distro,release,cpuarch21:03
clarkbanother optimization may be to only make wheels for things not already wheeled in pypi21:03
clarkbthough those are very cheap aiui since we just grab the wheel for pypi then copy into our mirror21:03
fungii'm wondering if adding our wheel mirror to the index list for the python3 wheel build jobs would make sense21:04
clarkber that tuple has a 4th section, python version21:04
fungiso that it would reuse the universal wheels we built with the python2 pass21:04
*** derekh has quit IRC21:05
clarkbfungi: yes, I think that would sovle that particular issue since pip should be smart enough to not use the python2 only wheels there21:05
mordredthat souds non-crazy to me21:05
*** jkilpatr has quit IRC21:05
clarkbso the trickle down effect is pypi -> python2 wheels -> python3 wheels21:05
*** sambetts is now known as sambetts|afk21:06
fungiclarkb: challenge i guess would be to make it use the universal wheels already built under python 2 but not the previously build python 3 wheels21:06
clarkbmtreinish: also it might just take someone pushing that change getting it to break, then responding to the goal ml post and saying "heres the thing who is going to work on it now"21:07
fungier, previously built21:07
*** nt has quit IRC21:07
mtreinishclarkb: that already was done21:07
* mtreinish digs up the patch21:07
mtreinishwell for py3421:07
clarkbmtreinish: was it advertised? I certainly never heard about it21:07
mtreinishthat I don't know21:08
clarkbfungi: I think that also isn't an issue bceause using old python3 wheels is probably file as long as the package version didn't change and new version will get rebuilt21:08
mtreinishclarkb: https://review.openstack.org/#/c/331224/21:08
clarkbmtreinish: ya I would advertise that directly and also don't do it the way its done there and use depends on21:08
fungiclarkb: one thing not reusing the previous wheels gets us is builds against newer dependency libraries21:09
mtreinishwell it's dated at this point21:09
fungianteaya: catching up on comments at the end of the tc meeting, but in gerrit a "change" has one or more "patch sets" (not change sets)21:09
mtreinishclarkb: I also have no idea who the submitter was21:09
clarkbfungi: ya but in theory those are relatively stable on our distros right?21:09
clarkbmtreinish: ya that patch is completely subverting how this should actually be done21:09
clarkbyou want error on clone = true and use depends on21:09
mtreinishclarkb: sure, I didn't actually look at tbh. I just saw it linked in a tempest py3 fix at one point21:10
fungiclarkb: relatively. i guess you really only care about things like soversion bumps/symbol changes which _shouldn't_ be the case on lts distros at least21:10
mtreinishand I've seen patches doing the same thing in the past21:10
clarkbfungi: yup21:10
fungithough there are exceptions21:10
clarkbfungi: though it wouldn't surprise me if it happens every once in a while21:10
fungiindeed, and it does21:10
mtreinishclarkb: but that's my point is someone needs to volunteer to drive it. Otherwise it kinda just sits there21:10
fungiclarkb: just not often21:10
mtreinishand it's not like you or I have the bandwidth to deal with it21:11
clarkbmtreinish: yes but I think thats how you get volunteers21:11
*** mmedvede has quit IRC21:11
fungiso maybe infrequent enough for us to ignore until it bites us21:11
clarkbmtreinish: people don't know where to start but if you hand them a template for the work that needs to be done that gets them halfway there I bet you would have at least a couple volunteers21:11
*** mmedvede has joined #openstack-infra21:11
clarkband it looks like 2 months ago that got really close21:11
zigomordred: We need the PTL's +2: https://review.openstack.org/36631821:12
fungii need to run out and get something to eat, then come back and see if my isp has recovered so i can try to generate this last batch of summit invites21:12
clarkbgrenade isn't going to work due to previous release and centos doesn't have python321:12
mtreinishclarkb: right but it assumes someone is going to push that template. I'm saying I don't want to be the one doing that, it needs someone to get things started21:12
mtreinishclarkb: it's step 0 vs step 121:12
*** rhallisey has quit IRC21:12
clarkbI mean its 5 minutes of work if you want I can update taht change and send an email21:12
*** xarses has quit IRC21:14
openstackgerritThomas Goirand proposed openstack-infra/project-config: Add missing deb repo 2/5  https://review.openstack.org/36633721:14
*** inc0 has quit IRC21:15
*** ldnunes has quit IRC21:15
*** dprince has quit IRC21:16
openstackgerritThomas Goirand proposed openstack-infra/project-config: Add missing deb repo 1/5  https://review.openstack.org/36631821:16
*** bin_ has quit IRC21:17
*** tonytan_brb has joined #openstack-infra21:17
openstackgerritKhai Do proposed openstack-infra/system-config: upgrade Gerrit's javamelody plugin on review-dev.o.o  https://review.openstack.org/36634821:17
openstackgerritThomas Goirand proposed openstack-infra/project-config: Add missing deb repo 2/5  https://review.openstack.org/36633721:18
*** matt-borland has quit IRC21:18
*** rossella_s has quit IRC21:18
*** rossella_s has joined #openstack-infra21:19
*** tonytan4ever has quit IRC21:19
openstackgerritMerged openstack/diskimage-builder: Revert "Revert "Pre-install pip/virtualenv packages""  https://review.openstack.org/34193421:21
*** Apoorva has joined #openstack-infra21:21
openstackgerritKhai Do proposed openstack-infra/system-config: upgrade Gerrit's javamelody plugin on review-dev.o.o  https://review.openstack.org/36634821:22
*** dimtruck is now known as zz_dimtruck21:23
openstackgerritKhai Do proposed openstack-infra/system-config: upgrade Gerrit's javamelody plugin on review.o.o  https://review.openstack.org/36442421:24
mordredShrews, infra-root: the earlier release of shade failed because of missing libffi so the tarball couldn't get built so it couldn't get released21:25
mordredthat is now fixed21:25
mordredso I'm going ot tag/push 1.12.1 so that we have a release that exists21:25
openstackgerritMerged openstack-infra/project-config: Revert "Force image-type qcow2 for OVH"  https://review.openstack.org/36633321:25
clarkbmordred: ok21:25
zaroclarkb: ^ done, split em up for you21:25
clarkbzaro: tyty don't forget to WIP the review.o.o one with a comment about merging it during planned outage21:26
zaroWIPed, warning is in the commit message21:27
clarkbzaro: perfect thanks21:27
*** notnownikki has quit IRC21:28
openstackgerritAlexey Khivin proposed openstack-infra/project-config: Add jenkins-plugins-for-murano project to Openstack  https://review.openstack.org/36615421:28
openstackgerritAlexey Khivin proposed openstack-infra/project-config: Add jenkins-plugins-for-murano project to Openstack  https://review.openstack.org/36615421:29
*** jcoufal_ has quit IRC21:29
pabelangerclarkb: restarted nodepool-builder to pickup qcow2 revert. Raw image uploads will start up tomorrow21:29
pabelangeralready looking better21:30
*** ddieterly is now known as ddieterly[away]21:30
clarkbpabelanger: ok, probably want to upload rax xenial images manually now?21:30
pabelangerclarkb: they are still in the queue actually21:31
pabelangerstill 51 images to upload21:31
clarkboh cool for some reason I thought it dropped all of them when it restarted21:31
pabelangerI'll check back here in a bit, need to run some errands.21:31
openstackgerritThomas Goirand proposed openstack-infra/project-config: Add missing deb repo 3/5  https://review.openstack.org/36635221:31
*** rfolco has quit IRC21:32
clarkbfungi: I have confirmed that the weekly compression run seems to be working for console logs and is compressing those. Now to check if its catching the puppet/kolla/ansible logs too21:32
*** dizquierdo has joined #openstack-infra21:33
*** rtheis has quit IRC21:33
clarkbfungi: it looks like it skips over all the *.log files http://logs.openstack.org/31/325631/8/check/gate-kolla-dsvm-deploy-centos-source/6953633/logs/kolla/nova/21:34
*** srobert has quit IRC21:34
clarkbfungi: maybe we should add them to the find name patterns?21:34
*** rhallisey has joined #openstack-infra21:34
zigomordred: I *really* need your PTL's approval: https://review.openstack.org/36631821:35
zigo(sorry to insist...)21:36
*** ddieterly[away] is now known as ddieterly21:36
zigoI got 5 like that, probably they will merge conflict...21:37
zigoThen if I did right, all repo will be there.21:37
*** thorst has joined #openstack-infra21:37
*** adrian_otto has quit IRC21:38
mordredzigo: fine by me - next time, fwiw, it's easier if you put the same topic for each patch - that way it's easy to find the sibling patches21:38
mordredoh - you did. great21:38
zigomordred: As I wrote, I expect merge conflicts, since it's touching close enough lines...21:39
zigomordred: So let's process them one by one.21:39
zigoThanks btw.21:39
mordredzigo: you've got a test failure on the governance patch21:39
clarkbalso I have caved and am using du on the logs server21:39
clarkbsince nothing really bad was showing up via web crawling21:39
*** annegentle has quit IRC21:39
zigothe Add deb-python-{keystoneauth1,requestsexceptions} ?21:40
mordredzigo: yah - I don't know if that's important or not - but I left a +1 there21:41
clarkbhttps://review.openstack.org/#/c/361500/ is using 1.8GB to store all of its logs for check and gate21:41
mordredclarkb: that's impressive21:41
clarkbthere is 29MB of coverage data for every neutron coverage run21:43
clarkbwe don't appear to compress any of it21:43
clarkb(probably because thats output of a tool and it won't host properly if not compressed21:43
*** pradk has quit IRC21:43
*** adriant has joined #openstack-infra21:44
clarkbdevstack+tempset runs seem to need about 100MB per run on the high end for multinode and ~80MB on the low end21:45
clarkbwith some variance in the middle21:45
*** thorst has quit IRC21:45
*** kzaitsev_mb has joined #openstack-infra21:47
clarkbjackpot http://logs.openstack.org/01/365601/3/check/gate-openstack-ansible-openstack-ansible-aio-ubuntu-xenial-nv/ea577a3/ used 1.1GB of disk just for that one job21:47
clarkbnow run it a couple times per patchset then have more than one patchset21:47
openstackgerritThomas Goirand proposed openstack-infra/project-config: Add missing deb repo 2/5  https://review.openstack.org/36633721:48
clarkbodyssey4me: prometheanfire ^ is that you all? Can we cut back on the amount of logs being grabbed there. Thats about 10x what a devstack run needs21:48
prometheanfireclarkb: hi?21:49
clarkbits half a gig of uncompressed syslog and user.log21:49
clarkbprometheanfire: you hack on the openstack ansible things right?21:49
prometheanfirenot too much :(21:49
clarkbsorry thats each half a gig for user.log and half a gig for syslog21:49
clarkbprometheanfire: ah ok21:49
prometheanfirethat is a ton21:50
clarkbya so I think we need to do two things here. 1) make sure people are precompressing before upload in their jobs like devstack does but also not trying to store all the data ever :)21:50
clarkbhttp://logs.openstack.org/01/365601/3/check/gate-openstack-ansible-openstack-ansible-aio-ubuntu-xenial-nv/ea577a3/logs/host/ both files live there21:51
*** mrjk has quit IRC21:51
* clarkb continues scanning the filesystem to find other places we can trim21:51
*** Sukhdev has quit IRC21:51
*** stevemar has quit IRC21:52
clarkbha a keystone change with 27 patchsets uses less disk than the openstack ansible change does with 2 tested patchsetse21:52
mordredclarkb: odyssey4me is the right human for openstack-ansible21:54
clarkbI am also only look at the 36XXXX series changes otherwise du would be too slow21:55
*** Sukhdev has joined #openstack-infra21:55
clarkbI figure better to look at recent things as far as how to improve for the future21:55
*** mrjk has joined #openstack-infra21:56
clarkbmordred: I will have to dig up their bug tracker and file a bug21:56
clarkbneutron changes are regularkly showing up in the higher end up disk usage though. Related to realtive size of neutron logs and number of jobs they run per patchset21:57
clarkbthey are the distributed scale out disk users and openstack-ansible is the scale up disk user in 2 files21:57
*** Sukhdev has quit IRC21:59
openstackgerritMonty Taylor proposed openstack-infra/nodepool: Use interface_ip instead of internal logic  https://review.openstack.org/36044622:00
*** askb has joined #openstack-infra22:00
*** rhallisey has quit IRC22:00
*** ddieterly is now known as ddieterly[away]22:00
mordredclarkb: I just assume odyssey4me will see things in scrollback :)22:00
anteayafungi: ah thank you, sorry I mis-communicated that, I was sure it was change sets22:01
*** zz_dimtruck is now known as dimtruck22:02
mordredianw: in answer to your question from the mysql user creation patch - python shows mysql warnings by default, but the mysql command line does not - so I doubt we're seeing the warning there22:02
*** annegentle has joined #openstack-infra22:02
ianwmordred: ahh, that would explain why nobody ever noticed22:04
*** cap_ has joined #openstack-infra22:05
*** asettle has joined #openstack-infra22:05
*** gyee has joined #openstack-infra22:05
*** sarob has joined #openstack-infra22:05
mordredyah - I do think we can update it just for completeness22:05
*** asettle has quit IRC22:06
*** asettle has joined #openstack-infra22:07
*** cap_ has quit IRC22:07
*** Sukhdev has joined #openstack-infra22:09
clarkbaccording to logstash the last 7 days we ran those ansible jobs ~100 times22:09
clarkbso every 7 days of ansible jobs is worth about 100GB22:09
mordredianw: oh bother. it's new syntax as of mysql 5.7.6 - so we really can't do the project config macro until we're on xenial22:09
*** hashar has quit IRC22:10
*** Goneri has joined #openstack-infra22:10
clarkbassuming that trend holds and we have actually run those jobs at that rate for the last 2 months that would account for about 5TB of our disk use?22:11
clarkbsomeone double check on my maths22:11
mordredianw: I set -W on the nodepool change so that I can poke further22:11
ianwmordred: heh, ok.  let the yak shaving begin :)22:11
mordredianw: /me cries22:11
*** ddieterly[away] is now known as ddieterly22:12
*** baoli has quit IRC22:12
mordredclarkb: if one week is 100GB, then 2 months would be 800G not 5TB22:12
mordredclarkb: unless I'm missing a math22:12
clarkbmordred: ya I derped the maths :)22:13
mordredclarkb: if _one_ day is 100GB, then I would agree22:13
*** baoli has joined #openstack-infra22:13
clarkbya thats how I derped it22:13
clarkbmultiplied by 722:13
clarkbinstead of divide22:13
mordredsometimes just multiplying by 7 is the right choice22:13
clarkbok so thats still a nontrivial amount of our log usage22:13
clarkbwe are at ~9TB now and almost 1TB is potentially just openstack ansible jobs22:13
*** kzaitsev_mb has quit IRC22:14
clarkbbasically to even get close to per change disk usage of openstack ansible I need to find neutron changes with a bunch of patchsets that have each run the tests over and over22:14
clarkbhttps://review.openstack.org/#/c/361508/ is a tempest chagne using about half the disk of 363307 which is an osa change. osa has half as many patchsets as tempest22:15
*** esberglu has quit IRC22:15
* clarkb continues to run du to see if there are any other outliers22:16
*** fguillot_ has joined #openstack-infra22:16
*** priteau has quit IRC22:16
mtreinishclarkb: how many jobs does osa run? Tempest runs a lot22:16
clarkbmtreinish: 4 in check and 3 in gate. Its far fewer. But at 1.1GB per job...22:17
clarkbmtreinish: basically they add up very quickly in osa land whereas with tempest/neutron its a lot of jobs that slowly accumulate22:17
*** xyang1 has quit IRC22:17
*** ddieterly is now known as ddieterly[away]22:17
clarkband the vast majority of other changes seem to come in under 100MB total22:18
* clarkb looks at older changes just to make sure its not because these changes are so new that we see this behavior22:18
*** eharney has quit IRC22:18
mtreinishclarkb: ah, ok. Yeah tempest has like ~24 check and ~13 gate22:21
*** esberglu has joined #openstack-infra22:21
*** mdrabe has quit IRC22:22
clarkbI think we can likely improve both. osa is easy because its basically 2 fiels that need to be dealt with but we should also get them (and kolla and puppet et al) to compress their logs22:22
*** vhosakot has quit IRC22:22
*** priteau has joined #openstack-infra22:22
*** piet has quit IRC22:22
clarkbthen for the dsvm jobs I think its just a matter of keeping the log files useful and sane which has sort of been an ongoing project for the last few years22:22
mtreinishclarkb: well not really the last year or 2. That effort kinda died off unfortunately22:23
mtreinishit's the same fight over and over again22:23
*** stevemar has joined #openstack-infra22:24
*** ddieterly[away] is now known as ddieterly22:24
*** esberglu has quit IRC22:26
*** stevemar has quit IRC22:26
ianwclarkb: i'll have to look at ansible, i remember we went through getting good logs with puppet and came up with http://git.openstack.org/cgit/openstack/puppet-openstack-integration/tree/copy_logs.sh22:26
*** priteau has quit IRC22:26
*** stevemar has joined #openstack-infra22:26
*** Swanson has joined #openstack-infra22:27
*** Goneri has quit IRC22:29
ianwhttp://logs.openstack.org/01/365601/3/check/gate-openstack-ansible-openstack-ansible-aio-ubuntu-xenial-nv/ea577a3/logs/host/syslog ... 506mb?22:29
*** Hal1 has quit IRC22:29
clarkbianw: the puppet logs are also not coming through all compressed like they should. We should problably update that script to also gzip .logs22:29
clarkbianw: yes and userl.log is about that big too22:30
clarkbianw: so between two files we basically hit 1GB22:30
*** sarob has quit IRC22:30
*** stevemar has quit IRC22:31
ianwclarkb: audispd: ... it's like putting in about 3 lines for every syscall22:31
clarkbianw: seems consistent across xenial and trusty too so not something release specific22:32
clarkbalso syslog for dsvm jobs is only 14MB uncompressed22:32
ianwi wonder ... i had to fix installing auditd on fedora-minimal to stop log spew22:32
*** krtaylor has joined #openstack-infra22:32
clarkbhttps://review.openstack.org/#/c/352610/ a nova change with 4 patchsets hit 12GB my biggest change so far22:33
* clarkb digs into that one22:33
*** kzaitsev_mb has joined #openstack-infra22:34
*** dimtruck is now known as zz_dimtruck22:34
*** berendt has quit IRC22:35
clarkbhrm that ran the neutron multinode test multiple times on ps4 to the tune of about 1GB each22:35
clarkboh tahts the 1GB keystone logs22:35
ianwhttps://access.redhat.com/solutions/2380591 -> "Turn off audispd syslog plugin in the /etc/audisp/plugins.d/syslog.conf file:"22:35
clarkbfungi: ^ was keystone addressed for that?22:35
*** sarob has joined #openstack-infra22:35
*** roxanagh_ has joined #openstack-infra22:35
mtreinishclarkb: yeah that change was reverted22:36
*** rajinir has joined #openstack-infra22:36
clarkbmtreinish: ok so probably we are still hurting from that but the underlying issue is corrected22:36
mtreinishclarkb: https://review.openstack.org/#/c/356010/22:36
ianwclarkb: and reading http://docs.openstack.org/developer/openstack-ansible-security/controls-cat1.html -> V-38471: The system must forward audit records to the syslog service.22:37
*** vhosakot has joined #openstack-infra22:37
clarkbmtreinish: and doesn't look like the original was backported so we don't need revert backports. Cool /me will ignore that for now22:38
SwansonPushed up a patch https://review.openstack.org/361453 Trying to use it with dev stack via CINDER_BRANCH=refs/changes/53/361453/7 and getting a "couldn't find remote ref" fatal. Anyone seeing this?22:39
SwansonPrevious ref still works.22:39
*** yamahata has quit IRC22:40
*** roxanagh_ has quit IRC22:40
clarkbSwanson: what remote are you using?22:40
*** fguillot_ has quit IRC22:40
*** fguillot_ has joined #openstack-infra22:41
Swansonclarkb, looks like git://git.openstack.org/openstack/cinder.git22:42
*** ddieterly has quit IRC22:43
*** thorst has joined #openstack-infra22:43
*** rockyg has quit IRC22:44
clarkbSwanson: looks like there may be a backlog syncing to the mirrors22:44
Swansonclarkb, I'll give it a while and try again. Thanks!22:45
*** annegentle has quit IRC22:46
Swansonclarkb, synced already. Thanks!22:46
*** nt has joined #openstack-infra22:46
clarkbSwanson: I am not sure what caused it but looks like its currently getting through it22:46
clarkbwent from ~580 tasks to 27 in the last couple minutes22:47
*** zz_dimtruck is now known as dimtruck22:47
SwansonGlad it is working now.22:47
*** thorst has quit IRC22:49
*** thorst has joined #openstack-infra22:49
*** yamahata has joined #openstack-infra22:49
*** mlavalle has left #openstack-infra22:52
*** rbrndt has quit IRC22:53
*** akshai has quit IRC22:53
*** asettle has quit IRC22:54
clarkbok from what I can tell digging around the log server the two issues we have as far as crazy disk utilization go are the osa jobs with their two half gig files and the leftovers from the apache for keystone logs being 1GB each22:54
clarkbwhen a normal dsvm run clocks in at most 100MB those that run at 1GB add up quickly22:55
*** rlandy is now known as rlandy|bbl22:55
*** asettle has joined #openstack-infra22:57
*** asettle has quit IRC22:57
*** vhosakot has quit IRC22:57
EmilienMI'm probably late in the party but anyone aware about pep8 job failures ? http://logs.openstack.org/87/366287/3/check/gate-python-tripleoclient-pep8-ubuntu-xenial/6bd1710/console.html#_2016-09-06_21_01_26_59938222:58
EmilienMfatal error: ffi.h: No such file or directory22:58
*** thiagolib has quit IRC22:58
clarkbEmilienM: you need to add libffi-dev to your bindep.txt22:58
openstackgerritSergey Reshetnyak proposed openstack-infra/project-config: Add py3 jobs for fuel-ccp-entrypoint repo  https://review.openstack.org/36641222:58
*** baoli has quit IRC22:58
clarkbEmilienM: problem is related to some transient dep along the way somewhere that now depends on libffi-dev at build time22:58
clarkbEmilienM: I didn't bother to look up which dep it was but project-config and shade were both affected22:59
EmilienMclarkb: thx22:59
clarkbEmilienM: I think the package is libffi-dev for dpkg and libffi-devel for rpm22:59
pabelangerEmilienM: clarkb: or wait until we build a cffi wheel23:00
pabelangerbut think we are capped right now23:00
clarkbpabelanger: you should still reflect what you need to build from scratch imo23:00
pabelangerclarkb: agreed23:00
pabelangerjust explains how it worked until today23:00
*** spzala has quit IRC23:05
clarkbianw: odyssey4me https://bugs.launchpad.net/openstack-ansible/+bug/1620849 there is a bug that can be used to fix this23:06
openstackLaunchpad bug 1620849 in openstack-ansible "openstack ansible test jobs are logging excessive amounts of data" [Undecided,New]23:06
*** krtaylor has quit IRC23:06
*** cardeois has quit IRC23:07
*** fguillot_ has quit IRC23:07
*** dizquierdo has quit IRC23:07
*** stevemar has joined #openstack-infra23:07
*** fguillot_ has joined #openstack-infra23:08
*** asettle has joined #openstack-infra23:09
*** asettle has quit IRC23:09
*** sdake has joined #openstack-infra23:10
fungiclarkb: EmilienM: the libffi surprise is that cffi released a new version on pypi today so lots of jobs found no prebuilt wheel, fell back on trying to build from the sdist tarball, and then discovered that those projects hadn't put libffi-dev packages in their bindep.txt23:12
EmilienMfungi: right23:12
clarkbfungi: ah gotcha23:12
fungibasically, our wheel mirrors and out bindep rollout happened in parallel... the two hadn't taken one another into account23:13
fungiso bindep implementation assumed projects with missing build deps would fail self-testing bindep.txt additions, but then we added a wheel mirror while that was going on so we no longer actually test that jobs can build all their dependency sdists consistently23:13
fungi(because doing that takes lots of extra time, so we pre-build wheels elsewhere)23:14
*** adrian_otto has joined #openstack-infra23:15
pabelangerso, a test we could add is, if bindep.txt is modified, remove wheel mirrors from node, then try to build a wheel. If passes, great, if fails then you are missing a dependency23:16
clarkbor just always have a job that doesn't use the wheel mirror23:16
fungii'm not a fan of the "if bindep.txt is modified" part23:16
fungijobs dependent on which files have changed make me nervous23:17
fungiit's too easy to fail to run them when you need them23:17
clarkbespecially when we are due for a bindep.txt filename change ina bout 3 months23:17
pabelangerRight, it could run every patch23:17
pabelangerI said could23:17
*** hongbin has quit IRC23:17
fungibut yeah, a job that doesn't use wheel mirrors would be a sane addition. it's a question of whether there's an existing job that's a good candidate for it23:18
pabelangerhopefully once zuulv3 lands, we can rework out static wheel builders23:18
fungigiven that our "pep8" macro already forces an sdist of the project, that might be a good one to overload23:18
pabelangerYa, that could be a good fit23:19
clarkbya I like that one because its a simple job to run locally and its relatively quick23:19
*** markvoelker has quit IRC23:19
clarkbthat way its easier to reproduce the "why does thsi random job fail" fails locally23:19
fungiright, so the overhead from the job payload is small, making it likely to still complete when or before unit tests do23:19
pabelangeris there a way not to download wheels with pip?23:19
pabelangerwhile keeping our mirror?23:19
clarkbpabelanger: --no-use-wheel23:20
pabelangergoing to test it out23:20
clarkboh thats deprecated should use --no-binary instead23:20
clarkbbut looks like they do roughly the same thing23:20
openstackgerritSergey Reshetnyak proposed openstack-infra/project-config: Add py3 jobs for fuel-ccp-entrypoint repo  https://review.openstack.org/36641223:20
*** jamesdenton has joined #openstack-infra23:21
*** xarses has joined #openstack-infra23:22
clarkbfungi: wiki-dev01.openstack.org is sending mail that pabelanger's MX is rejecting beacuse we lack DNS records for it?23:22
*** akshai has joined #openstack-infra23:22
fungithat's one i'm deleting and re-launching so wasn't planning to put in dns until it's working23:23
clarkbfungi: gotcha23:23
fungiright now fussing over puppet errors from vcsrepo23:23
fungiwas hoping to continue on it today, but that hasn't happened23:23
*** gildub has joined #openstack-infra23:24
clarkbno problem, just wasn't sure if taht was an oversight23:24
mordredclarkb, pabelanger, fungi: I like the idea of the pep8 job skipping the wheel mirror - I think it's a good hack23:25
*** jamesdenton has quit IRC23:26
*** mriedem has quit IRC23:30
*** mriedem has joined #openstack-infra23:32
*** sarob has quit IRC23:32
*** dimtruck is now known as zz_dimtruck23:32
openstackgerritMerged openstack/diskimage-builder: fix systemd resource deadlock  https://review.openstack.org/36579723:33
*** akshai has quit IRC23:33
*** bapalm has quit IRC23:35
pabelangermordred: fungi: clarkb: Ya, just tested locally and appears to work.23:37
mordredpabelanger: woot23:38
*** dingyichen has joined #openstack-infra23:38
*** Sukhdev has quit IRC23:40
*** bapalm has joined #openstack-infra23:41
*** fguillot_ has quit IRC23:42
*** sarob has joined #openstack-infra23:46
*** zz_dimtruck is now known as dimtruck23:46
clarkblooks like tripleo jobs are copying /var/log twice...23:48
clarkbonce in a tarball and again not in a tarball23:48
clarkbpabelanger: ^ maybe you can poke someon about that? gate-tripleo-ci-centos-7-ovb-ha-liberty is job where I noticed it not sure if it affects newer jobs too23:49
*** Jeffrey4l has joined #openstack-infra23:49
pabelangerclarkb: sure23:49
dstufftDoes openstack support 2.623:50
dstufftPython 2.6 that is23:50
clarkbdstufft: no, its no longer a tested platform23:50
fungidstufft: i almost chimed in on the distutils-sig thread about that23:50
* clarkb wonders whats going on23:51
clarkbpabelanger: tyty23:51
dstufftUnless a pip core speaks out I'm planning to kill pip's support for it in pip 10 (with a red warning in pip 9)23:51
fungiclarkb: you should subscribe to teh ml ;)23:51
dstufftwanted to warn y'all if you supported 2.6 still23:51
clarkbdstufft: ah23:51
fungiwe dropped support for python 2.6 when we could successfully test things on rhel 723:51
clarkbI know lifeless wanted to make sure pbr was still tested with it but I don't think anyone did the work for that23:52
dstufftonly ~3% of downloads from PyPI using a modern pip (6+) are via 2.6, so I don't feel like we should care about it anymore.23:52
clarkband at this point if pip is dropping support I think its mostly moot23:52
fungiwe supported python 2.6 while it was still the distro-provided default python in rhel (while 6 was current)23:52
*** sarob has quit IRC23:52
dstufftalso I really want to stop thinking about 2.623:52
dstufftso there's that :D23:52
*** sflanigan has joined #openstack-infra23:52
*** sflanigan has joined #openstack-infra23:52
fungionce rhel 7 was out with python 2.7 default, and we had a release cycle to switch over, and then our supported stable branches that released back before we tested on rhel 7 finally reached eol, we stopped building rhel 6 servers to test changes on23:53
fungiand at that point, we had no existing means nor incentive to support python 2.623:54
fungiand i felt we were on the conservative side of dropping 2.6 support23:54
clarkbfungi: I already have a hard enoug htime keeping up with all of our mailing lists23:54
*** Jeffrey4l has quit IRC23:55
dstufftI figured y'all were probably not to worried about it anymore, but also breaking openstack tends to mean I get yelled at on multiple fronts so :V23:55
dstufft"yelled at"23:55
* fungi tties really hard not to yell at dstufft23:55
dstufft(not actually yelled at, I just have an odd sense of humor)23:56
*** ddieterly has joined #openstack-infra23:56
fungiyou do awesome work on pip/virtualenv/pypi/warehouse. i am always awed23:56
dstufftfungi: heh, thanks :)23:57
*** markvoelker has joined #openstack-infra23:57
*** fguillot_ has joined #openstack-infra23:59
clarkbI do have to say pypi/pip today compared to 2, 3 years ago is like night and day23:59
clarkbno longer do I fuss over setting up my mirrors. tl;dr yay things are much better today23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!