Thursday, 2016-09-01

*** baoli has quit IRC		00:00
openstackgerrit	Merged openstack-infra/shade: Revert per-resource dogpile.cache work https://review.openstack.org/362901	00:04
*** signed8bit_Zzz is now known as signed8bit		00:08
ianw	I've declared bankruptcy on scrollback today, so not sure I'm much help :)	00:08
clarkb	ianw: uh tl;dr is funny networking in rackspace to the git mirror. Lots of connections being killed under haproxy both on the front and backends. So I filed a ticket after much debugging	00:09
clarkb	ianw: networking-vpp clogged up the tubes again and has been asked to wait until dhellmann is done with release things before continuing	00:09
ianw	clarkb: ahh, so that was the issue. jhesketh and I were looking yesterday afternoon, that weird situation where nothing seems to be wrong but something is :)	00:10
clarkb	ianw: osic-cloud8 has a weird floating ip setup for our mirror so to start we are just going to use the mirror in cloud1 (by creating dns records for cloud8 that point at cloud1)	00:10
clarkb	ianw: ya its a fun one whatever it is	00:10
clarkb	nodepool is now running the shade fixes for dual stack networking	00:10
jhesketh	clarkb: good to know, thanks for digging :-)	00:11
clarkb	and we have enabled infracloud and internap mtl01	00:11
clarkb	I think at least some of these things got status logged	00:11
clarkb	I am going to restart the nodepool builder now so that it can learn how to talk to osic-cloud8 and upload images there	00:12
ianw	clarkb: cool, thanks for the update :)	00:14
*** katyafervent_awa has quit IRC		00:17
*** rakhmerov has quit IRC		00:18
clarkb	cloud8 is still set to max servers 0 so should just upload images then idle	00:18
*** Hal1 has quit IRC		00:19
*** Sukhdev has joined #openstack-infra		00:20
*** baoli has joined #openstack-infra		00:21
*** signed8bit is now known as signed8bit_Zzz		00:23
*** rakhmerov has joined #openstack-infra		00:23
*** katyafervent_awa has joined #openstack-infra		00:25
*** watanabe_isao has joined #openstack-infra		00:25
*** sputnik13 has quit IRC		00:27
ianw	clarkb: btw you were right about growroot on f24 -> https://bugzilla.redhat.com/show_bug.cgi?id=1371761 : return status of sfdisk changed	00:27
openstack	bugzilla.redhat.com bug 1371761 in util-linux "sfdisk return code breaks growpart" [Unspecified,Post] - Assigned to kzak	00:27
bkero	ianw: i filed a bug/ml post on util-linux like a year ago for that	00:28
*** sflanigan has joined #openstack-infra		00:28
bkero	That didn't get fixed in the release? sigh	00:28
*** Julien-zte has joined #openstack-infra		00:29
ianw	bkero: not sure if it's the same issue? this one went in https://git.kernel.org/cgit/utils/util-linux/util-linux.git/commit/disk-utils/sfdisk.c?id=14f644f386a1708483ed446e983c0976e3976a9d	00:30
ianw	i'm really not sure how we're the only people who noticed	00:30
*** baoli has quit IRC		00:30
bkero	There's probably a kludge in libguestfs that works around this	00:31
*** jamesdenton has joined #openstack-infra		00:31
clarkb	ianw: it seems like we have a fairly consistent set of failure modes now. init + cloud-init/glean fail to network, growroot fails to growroot, random new firewall software ignores the rules I tell it, new package manager is cranky	00:31
clarkb	rinse and repeat :)	00:31
bkero	Looks to be an adjacent issue. Mine was that the ioctl for sfdisk was failing because loopback device	00:31
openstackgerrit	Merged openstack-infra/system-config: Log gear at debug level on nodepoold https://review.openstack.org/362455	00:32
ianw	clarkb: yeah. on my todo list is to check that growroot worked in our setup scripts. the problem was that RAX worked, as there was just enough space. and when i was running experimental with just one node, i didn't get out to other providers	00:32
ianw	but once things started getting heterogeneous ...	00:33
*** signed8bit_Zzz is now known as signed8bit		00:33
ianw	but, pabelanger we should still see why out-of-disk lead ansible to hang around for it's full 3 hour timeout	00:34
*** Hal1 has joined #openstack-infra		00:34
clarkb	ianw: if I am going to guess its doing a write that blocks despite not enough disk and just never returns?	00:35
*** kzaitsev_mb has joined #openstack-infra		00:35
clarkb	hrm though that should ENOSPC	00:35
*** baoli has joined #openstack-infra		00:35
*** thorst has joined #openstack-infra		00:35
*** zhurong has joined #openstack-infra		00:36
*** pvaneck has quit IRC		00:36
*** tphummel has joined #openstack-infra		00:37
ianw	clarkb: it was something slightly weirder in that console.html showed everything finishing up, but there was an ansible copy process that got stuck. i'll have to dig out the logs, i should make a note for posterity	00:37
*** kzaitsev_mb has quit IRC		00:40
ianw	ahh, looking at the logs, it's zuul_runner that's behaving odd in this case. that makes more sense, being our custom bit	00:42
*** thcipriani is now known as thcipriani\|afk		00:43
*** signed8bit has quit IRC		00:44
pabelanger	ianw: ya, I haven't had a chance to dig into fedora24 yet	00:44
pabelanger	clarkb: cool, osic-cloud8 images already uploaded. We can work on launching servers tomorrow	00:46
ianw	pabelanger: so i remember later -> http://paste.openstack.org/show/565462/ . maybe something in the bg causing ssh exit to hang?	00:46
pabelanger	ianw: is it possible git clone is failing?	00:47
pabelanger	ianw: because we had the same issue today in tripleo-test-cloud-rh1	00:47
pabelanger	ianw: not failing, hanging	00:47
ianw	pabelanger: possibly ... http://logs.openstack.org/12/363212/1/check/gate-tempest-dsvm-platform-fedora24-nv/fc07025/logs/ the workspace-setup-new is 0 bytes. i'm guessing that means whatever output never got flushed to it, rather than it never ran	00:48
ianw	so what's acutally going on ... shrug?	00:48
*** mtanino has quit IRC		00:49
*** ociuhandu has joined #openstack-infra		00:49
pabelanger	ianw: also, I don't think it is an SSH issue, because ansible async will poll the server every 10 seconds, and usually ansible will bark is SSH connection fails	00:49
pabelanger	ianw: ya, this looks like what I seen in tripleo-test-cloud-rh1 today, if you get into that node, I suspect you'll see hung git clone process	00:50
pabelanger	from devstack-gate	00:50
*** gyee has quit IRC		00:51
*** vinaypotluri has quit IRC		00:52
*** Goneri has joined #openstack-infra		00:54
*** Sukhdev has quit IRC		00:55
*** sarob has quit IRC		00:55
ianw	mtreinish: if around, have some questions on https://review.openstack.org/#/c/234447/	01:01
*** spzala has quit IRC		01:02
ianw	particularly what devstack.subunit it's trying to pick up	01:02
*** kzaitsev_mb has joined #openstack-infra		01:06
*** ociuhandu has quit IRC		01:08
*** priteau has joined #openstack-infra		01:11
*** Sukhdev has joined #openstack-infra		01:12
*** Sukhdev has quit IRC		01:13
*** priteau has quit IRC		01:16
openstackgerrit	Paul Belanger proposed openstack-infra/project-config: Revert "Revert "Disable rax-iad due to launch failure rate"" https://review.openstack.org/364012	01:16
pabelanger	ianw: ^if you don't mind a +A, I forgot to disable that earlier today. Were still having issues in rax-iad	01:17
pabelanger	I was able to reproduce the issue manually, so we can keep the region offline until we know the fix	01:18
*** rossella_s has quit IRC		01:18
*** chlong has joined #openstack-infra		01:18
*** shashank_hegde has quit IRC		01:18
ianw	pabelanger: LGTM	01:18
*** rossella_s has joined #openstack-infra		01:19
*** tqtran has quit IRC		01:19
*** aeng has quit IRC		01:20
pabelanger	ianw: thanks	01:20
ianw	who is responsible for stackviz?	01:20
pabelanger	ianw: I think timothyb89	01:26
*** yanyanhu has joined #openstack-infra		01:27
ianw	pabelanger / timothyb89: cool ... well i'm not sure what it thinks it's doing during grenade runs, but i'm pretty sure it's not doing it	01:27
*** baoli_ has joined #openstack-infra		01:28
*** baoli has quit IRC		01:28
*** baoli has joined #openstack-infra		01:30
*** esp has quit IRC		01:32
*** baoli_ has quit IRC		01:33
*** salv-orlando has joined #openstack-infra		01:34
*** Benj_ has joined #openstack-infra		01:34
openstackgerrit	Merged openstack-infra/project-config: Revert "Revert "Disable rax-iad due to launch failure rate"" https://review.openstack.org/364012	01:36
*** changzhi has joined #openstack-infra		01:37
*** spzala has joined #openstack-infra		01:38
*** Benj_ has quit IRC		01:41
*** hockeynut has quit IRC		01:43
*** spzala has quit IRC		01:43
*** salv-orlando has quit IRC		01:44
*** spzala has joined #openstack-infra		01:45
*** sarob has joined #openstack-infra		01:49
*** esp has joined #openstack-infra		01:58
*** woodster_ has quit IRC		01:59
*** changzhi has quit IRC		02:00
*** sarob has quit IRC		02:00
*** itisha has quit IRC		02:00
*** ijw has quit IRC		02:01
*** zshuo has joined #openstack-infra		02:02
*** Apoorva has quit IRC		02:03
*** changzhi has joined #openstack-infra		02:04
*** spzala has quit IRC		02:05
*** apetrich has quit IRC		02:05
*** esp has quit IRC		02:05
*** sdake has quit IRC		02:06
*** apetrich has joined #openstack-infra		02:07
*** esberglu has joined #openstack-infra		02:09
*** gildub has quit IRC		02:10
*** cody-somerville has joined #openstack-infra		02:13
*** yamamoto_ has joined #openstack-infra		02:14
*** hichihara has joined #openstack-infra		02:14
timothyb89	ianw: what's the issue?	02:17
ianw	timothyb89: have a look at 2016-08-31 07:32:32.699 in http://logs.openstack.org/26/363326/3/check/gate-grenade-dsvm-neutron-ubuntu-trusty/05856df/logs/devstack-gate-cleanup-host.txt	02:17
ianw	timothyb: i'm working on a refactor of bits of this anyway, it's incredibly hard to understand IMO	02:18
*** tqtran has joined #openstack-infra		02:18
*** thorst has quit IRC		02:18
*** thorst has joined #openstack-infra		02:19
timothyb89	ianw: ah, hmm. I thought I had the paths for grenade set correctly but I guess not :)	02:20
*** nstolyarenko has joined #openstack-infra		02:21
timothyb89	ianw: I'll make sure to take a look at the devstack-gate bits when I'm back in the office tomorrow	02:21
*** tqtran has quit IRC		02:22
*** fguillot has quit IRC		02:23
*** nstolyarenko has quit IRC		02:25
*** thorst has quit IRC		02:28
*** edtubill has joined #openstack-infra		02:28
*** dimtruck is now known as zz_dimtruck		02:30
openstackgerrit	Changcheng Intel proposed openstack-infra/jenkins-job-builder: update base_email_ext to adapt Email-ext plugin https://review.openstack.org/355139	02:30
*** Goneri has quit IRC		02:35
*** vhosakot has joined #openstack-infra		02:37
*** mriedem has quit IRC		02:39
*** jamielennox\|away is now known as jamielennox		02:41
*** salv-orlando has joined #openstack-infra		02:43
*** aeng has joined #openstack-infra		02:44
*** armax has quit IRC		02:45
*** zz_dimtruck is now known as dimtruck		02:47
*** vhosakot has quit IRC		02:48
*** Sukhdev has joined #openstack-infra		02:48
*** gouthamr has quit IRC		02:49
*** salv-orlando has quit IRC		02:50
*** amotoki has joined #openstack-infra		02:51
*** salv-orlando has joined #openstack-infra		02:52
*** salv-orlando has quit IRC		02:57
*** dimtruck is now known as zz_dimtruck		03:02
*** esp has joined #openstack-infra		03:03
*** docaedo has quit IRC		03:03
*** armax has joined #openstack-infra		03:03
*** cody-somerville has quit IRC		03:05
*** amotoki has quit IRC		03:08
*** armax has quit IRC		03:08
*** cody-somerville has joined #openstack-infra		03:11
*** docaedo has joined #openstack-infra		03:11
*** esberglu has quit IRC		03:12
*** vinaypotluri has joined #openstack-infra		03:14
*** cody-somerville has quit IRC		03:16
*** yamahata has quit IRC		03:17
*** amotoki has joined #openstack-infra		03:21
*** tphummel has quit IRC		03:23
*** zz_dimtruck is now known as dimtruck		03:23
*** thorst has joined #openstack-infra		03:26
*** amotoki has quit IRC		03:30
amrith	did something just hiccup in zuul?	03:32
amrith	I had a recheck running on a review https://review.openstack.org/#/c/363654/	03:32
amrith	and it seems to have vanished without a trace	03:32
*** thorst has quit IRC		03:33
*** shashank_hegde has joined #openstack-infra		03:33
*** chem\|off has quit IRC		03:34
clarkb	its there... the recheck was after you asked?	03:34
clarkb	I am confused	03:34
clarkb	the reverify 3.5 hours ago reported. then you just rechecked and its queued	03:35
*** amotoki has joined #openstack-infra		03:35
clarkb	from ehat I can see its all working as expected	03:35
openstackgerrit	Ian Wienand proposed openstack-infra/devstack-gate: Fix devstack subunit output https://review.openstack.org/364045	03:35
openstackgerrit	Ian Wienand proposed openstack-infra/devstack-gate: [WIP] Refactor devstack log copying https://review.openstack.org/364046	03:35
*** vikrant has joined #openstack-infra		03:36
ianw	timothyb89: ^ i'm thinking something like this. rolling stackviz into the devstack processing part of 364046 would probably remove the confusion	03:36
timothyb89	ianw: oh, cool, that would be much better	03:38
amrith	clarkb I see it now	03:39
*** sarob has joined #openstack-infra		03:39
amrith	after I refreshed my screen a couple of times	03:39
amrith	I just requested a recheck	03:39
amrith	something weird	03:39
amrith	for sure	03:39
amrith	petr requested the reverify at 8:06	03:39
amrith	it failed at 9:50	03:39
clarkb	and that ran and reported fine	03:39
amrith	but at 11:30 I saw nothing	03:40
amrith	just the previous gate	03:40
clarkb	then you did a recheck and it worked fine	03:40
clarkb	you didntcomment untill:33	03:40
amrith	and on zuul nothing	03:40
amrith	correct	03:40
amrith	I was refreshing a couple of times	03:40
clarkb	yes and it was in zuul at that point	03:40
amrith	on my browser	03:40
clarkb	so worked fine	03:40
clarkb	you were saying ut wasnt queued before you commented	03:40
amrith	from about 10:30 to 11:30, I saw nothing in review.openstack.org	03:40
clarkb	I think maybe you just didnt get the comment to poat in gerrit as quickly as you thought	03:41
amrith	which is when I posted the question here on IRC :)	03:41
amrith	well it is running again	03:41
clarkb	what woukd there have been to see?	03:41
amrith	that it failed at 9:50pm	03:41
amrith	all I was seeing was the previous success	03:42
amrith	so, check passed and it went to gate. and failed	03:42
amrith	petr reverified	03:42
amrith	then it failed in check	03:42
amrith	that failure was at 9:50pm	03:42
clarkb	yes and from what I see thats all rwcorded properly	03:42
clarkb	then you rechecked and it restarted jobs like asked	03:43
amrith	yes, except from about 10:30 to 11:30 I've been refreshing my screen, and checking zuul	03:43
amrith	and saw nothing :)	03:43
amrith	My guess is that after I posted the recheck	03:43
amrith	something happened	03:43
clarkb	I wasnt able to check that but when I did it worked.any chance you have a proxy being overzealous with caching?	03:43
amrith	not that I know of, I'm sitting at home	03:44
amrith	comcast doesn't typically cache this stuff, I don't think	03:44
openstackgerrit	Changcheng Intel proposed openstack-infra/jenkins-job-builder: [Don't Merge]update base_email_ext to adapt Email-ext plugin https://review.openstack.org/355139	03:44
amrith	I don't use a proxy at home for sure	03:44
amrith	but, zuul appears to be quite a memory hog	03:44
amrith	zuul.openstack.org, the site that is	03:44
amrith	it makes firefox limp	03:44
clarkb	but zuul isnt what we arw talking about we are talking about comments in gerrit	03:45
amrith	viewed through a browser that maybe the culprit	03:45
clarkb	and yes comcast shouldnt proxy that for you	03:45
*** Srinu has joined #openstack-infra		03:46
amrith	hmmm	03:46
clarkb	maybe you had toggle ci toggled?	03:46
clarkb	thats actually lrobably the most likely cause	03:46
amrith	toggle CI shouldn't impact the middle of the screen	03:46
amrith	which shows the results	03:47
amrith	it only shows the stuff in the history	03:47
clarkb	that I dont kniw. its a mess of hacky js to parse the gerrit	03:47
clarkb	I tend to rely in the actual comments	03:47
amrith	below the CR +2's, Verified and Workflow is the jenkins check and gate output	03:47
amrith	and that didn't refresh	03:47
amrith	I didn't even look at the history	03:47
clarkb	also if you were looking at an old patchset that also affects the table	03:47
*** akshai_ has quit IRC		03:48
clarkb	it wont update like you expect	03:48
clarkb	iirc	03:48
amrith	this section: http://picpaste.com/Capture-HJj23Ccv.JPG	03:48
amrith	circled in the image	03:49
amrith	oh, an f5 won't update that section?	03:49
clarkb	if you are on an old patchset it gets weird I think	03:49
amrith	that I did not know. maybe I should just navigate away from the review and back	03:49
amrith	that could be	03:49
amrith	I could, maybe have been on previous patch set	03:50
clarkb	if you refresh on thr latest patchset it shoukd be fine with a hardrefresh at least	03:50
amrith	don't recall	03:50
amrith	good to know	03:50
amrith	in future just navigate away and come back	03:50
clarkb	I dont know enough about the js details to know if a soft refresh is enough	03:50
amrith	it was 10:30, nothing much has worked today	03:50
amrith	someone said something about mercury going retrograde and causing all the computrons to spin in the wrong direciton	03:50
Srinu	hi	03:51
amrith	our stable branches (both) just died inexplicably in the past couple of days; just realized it	03:51
Srinu	anyone face this issue. http://paste.openstack.org/show/565606/	03:51
Srinu	please help me.	03:51
amrith	but, on the plus side, the helical inclined plane worked and pulled the cork out of the bottle just fine	03:51
amrith	hi Srinu	03:51
Srinu	amrith: hi. did you saw my question	03:52
clarkb	Srinu: that log points you at thr other log files for specificsyou will need to look at them to determine what is happening	03:52
clarkb	do you have a link to the job logs?	03:52
amrith	yes Srinu .. as clarkb says, the error messages are pointing you to the right log file	03:52
amrith	what's the review #?	03:52
*** salv-orlando has joined #openstack-infra		03:53
amrith	did he go away?	03:53
Srinu	clarkb,amrith: while the running the block storage tests cases it is killed and then copying logs	03:54
amrith	was this in the gate/CI	03:54
amrith	or on your local machine?	03:54
amrith	looks like CI/gate to me	03:54
Srinu	amrith: ci	03:55
amrith	ok, what's the review #	03:55
amrith	review.openstack.org/#/c/.....	03:55
amrith	or as clarkb said, the link to the file where you got the stuff that you put in paste	03:56
*** salv-orlando has quit IRC		03:56
amrith	clarkb, what tz are you in?	03:56
clarkb	pst	03:56
amrith	ah	03:56
amrith	wondered if you were a night owl too; it is getting close to being tomorrow now.	03:57
Srinu	amrith,clarkb: please check this. http://paste.openstack.org/show/565607/	03:57
* jlvillal thinks clarkb should be off work by now :)		03:57
amrith	Srinu, that won't help	03:57
amrith	what's the review #	03:57
amrith	or a link to the logs	03:57
amrith	just a URL would do ...	03:57
amrith	or is this a private CI?	03:57
amrith	which we can't get to?	03:58
clarkb	amrith screaming toddlers keeping me awake	03:58
openstackgerrit	Ian Wienand proposed openstack-infra/devstack-gate: Update bashate to 0.5.0 https://review.openstack.org/236815	03:58
Srinu	amrith: it is not a patch. this error is coming in jenkins ci(private).	03:58
EmilienM	hi, I'm waiting for this review to release tripleo newton-3: https://review.openstack.org/#/c/363897/	03:58
tristanC	ttx: can you please confirm the proposed schedule for upcoming election: http://docs-draft.openstack.org/53/335253/3/check/gate-election-docs-ubuntu-xenial/b8bd2a9//doc/build/html/ (rendered from https://review.openstack.org/#/c/335253/2/events.yaml) ?	03:58
amrith	clarkb, it is quiet here, almost midnight	03:58
EmilienM	if you're project-config core, please look at this patch when you can	03:58
amrith	sorry Srinu can't tell from that error; something took too long, someone got angry and killed it. there are lots of books and movies with the same story.	03:59
clarkb	Srinu: I think that means you timed out	03:59
amrith	some test had a 600s timeout	03:59
clarkb	Srinu: try increasing the timeout or make it run faster	03:59
amrith	and your test didn't run in time ...	03:59
amrith	game over	03:59
Srinu	amrith.clarkb: thank you	03:59
*** rlandy has quit IRC		03:59
*** amotoki has quit IRC		04:01
*** Srinu has quit IRC		04:02
*** cody-somerville has joined #openstack-infra		04:05
*** spzala has joined #openstack-infra		04:05
*** jamielennox is now known as jamielennox\|away		04:05
openstackgerrit	Merged openstack-infra/project-config: tripleo-ui: add missing jobs for release management https://review.openstack.org/363897	04:07
amrith	hmm, clarkb is there a simple way to look at the history of a particular job in the CI? project=openstack/trove-integration, job=gate-trove-functional-dsvm-mysql-mitaka. I looked in openstack-health (status.openstack.org/openstack-health) but it seems to only have data through 8/19.	04:08
*** Sukhdev has quit IRC		04:09
*** mcarden has quit IRC		04:09
clarkb	amrith: the three places are graphite.openstack.org the health dashboard and elasticsearch	04:09
*** changzhi has quit IRC		04:09
*** alexey_weyl has joined #openstack-infra		04:09
amrith	ok, thx. let me look at the other two	04:09
*** alexey_weyl has left #openstack-infra		04:09
*** vikrant has quit IRC		04:09
*** alexey_weyl has joined #openstack-infra		04:09
alexey_weyl	Hi Guys,	04:09
alexey_weyl	Please approve this change	04:10
alexey_weyl	https://review.openstack.org/#/c/363905/	04:10
*** vikrant has joined #openstack-infra		04:10
*** spzala has quit IRC		04:10
*** yamahata has joined #openstack-infra		04:12
amrith	wow! this graphite is cool stuff	04:13
*** mcarden has joined #openstack-infra		04:13
clarkb	amrith: there is also grafana.openstack.org that is a different frontend to the same data	04:14
clarkb	we have a grafyaml config somehwere that you can write out cof igs for dashboard in	04:14
amrith	wow, awesome	04:15
amrith	didn't know I could do this ...	04:15
amrith	does one have to sign in with lp credentials or some other?	04:16
openstackgerrit	Merged openstack-infra/project-config: move tripleo scenario jobs to check pipeline, non-voting https://review.openstack.org/363629	04:16
*** amotoki has joined #openstack-infra		04:16
clarkb	no its all public data	04:16
clarkb	read only	04:16
openstackgerrit	Ian Wienand proposed openstack-infra/devstack-gate: Fix devstack subunit output https://review.openstack.org/364045	04:16
openstackgerrit	Ian Wienand proposed openstack-infra/devstack-gate: [WIP] Refactor devstack log copying https://review.openstack.org/364046	04:16
amrith	ok	04:17
alexey_weyl	hello,	04:18
alexey_weyl	Can you please check this change:	04:18
alexey_weyl	https://review.openstack.org/#/c/363905/	04:18
amrith	g'night clarkb .. I just pushed a change to revert to the last point where stable passed, will see what tomorrow (crap: today) brings.	04:19
*** alexey_weyl has quit IRC		04:20
*** cody-somerville has quit IRC		04:20
*** dimtruck is now known as zz_dimtruck		04:22
*** yamamoto_ has quit IRC		04:23
*** baoli has quit IRC		04:24
*** edtubill has quit IRC		04:24
*** esp has quit IRC		04:25
*** shashank_hegde has quit IRC		04:26
openstackgerrit	Merged openstack-infra/project-config: Vitrage tempests https://review.openstack.org/363905	04:27
openstackgerrit	Ian Wienand proposed openstack-infra/project-config: Run bashate test over devstack-gate too https://review.openstack.org/236819	04:28
*** thorst has joined #openstack-infra		04:30
*** links has joined #openstack-infra		04:32
*** asselin__ has joined #openstack-infra		04:34
*** krtaylor has quit IRC		04:34
*** rajinir has quit IRC		04:35
*** jraim has quit IRC		04:36
*** maximov has quit IRC		04:36
*** jraim has joined #openstack-infra		04:36
*** maximov has joined #openstack-infra		04:37
*** esp has joined #openstack-infra		04:37
openstackgerrit	Masayuki Igawa proposed openstack-infra/irc-meetings: Update the QA team meeting chair https://review.openstack.org/364065	04:37
*** thorst has quit IRC		04:37
*** Illumitardi has joined #openstack-infra		04:37
*** asselin_ has quit IRC		04:37
*** edwarnicke has quit IRC		04:37
*** amit213 has quit IRC		04:37
*** Ng has quit IRC		04:38
*** NobodyCam has quit IRC		04:38
*** scottda has quit IRC		04:38
*** ogelbukh has quit IRC		04:38
*** zhiyan has quit IRC		04:38
*** nikhil has quit IRC		04:38
*** samueldmq has quit IRC		04:38
*** betherly has quit IRC		04:38
*** fmccrthy has quit IRC		04:38
*** vdrok has quit IRC		04:38
openstackgerrit	Merged openstack/diskimage-builder: Explain difference between two envvars https://review.openstack.org/345935	04:38
*** yamamoto_ has joined #openstack-infra		04:39
*** zz_ja has quit IRC		04:39
*** shashank_hegde has joined #openstack-infra		04:39
*** senk has joined #openstack-infra		04:39
*** scottda has joined #openstack-infra		04:40
*** amit213 has joined #openstack-infra		04:40
*** cody-somerville has joined #openstack-infra		04:40
*** psachin has joined #openstack-infra		04:40
*** nikhil has joined #openstack-infra		04:40
*** zhiyan has joined #openstack-infra		04:40
*** vdrok has joined #openstack-infra		04:40
*** betherly has joined #openstack-infra		04:40
*** fmccrthy has joined #openstack-infra		04:41
*** edwarnicke has joined #openstack-infra		04:41
*** ogelbukh has joined #openstack-infra		04:41
*** NobodyCam has joined #openstack-infra		04:41
openstackgerrit	Ian Wienand proposed openstack/diskimage-builder: Disabled IPv6 privacy extensions https://review.openstack.org/356456	04:41
*** Ng has joined #openstack-infra		04:42
*** zz_dimtruck is now known as dimtruck		04:42
*** yamahata has quit IRC		04:42
*** samueldmq has joined #openstack-infra		04:42
*** dtroyer has quit IRC		04:44
*** dtroyer has joined #openstack-infra		04:44
*** zz_ja has joined #openstack-infra		04:45
*** esp has quit IRC		04:45
*** pt_15 has quit IRC		04:46
*** jbryce has quit IRC		04:46
*** sarob has quit IRC		04:48
*** Illumitardi has quit IRC		04:48
*** yamamoto_ has quit IRC		04:49
*** jbryce has joined #openstack-infra		04:52
*** maishsk has joined #openstack-infra		04:54
*** yamahata has joined #openstack-infra		04:54
*** roxanaghe has quit IRC		04:56
*** pcaruana has quit IRC		04:57
*** maishsk has quit IRC		04:58
*** kzaitsev_mb has quit IRC		04:59
*** pgadiya has joined #openstack-infra		05:02
*** salv-orlando has joined #openstack-infra		05:03
*** ggherdov- has quit IRC		05:03
*** dougwig has quit IRC		05:03
*** asingh has quit IRC		05:03
*** ediardo has quit IRC		05:03
*** hrybacki has quit IRC		05:03
*** agordeev has quit IRC		05:03
*** warpc has quit IRC		05:03
*** rockstar has quit IRC		05:03
*** cargonza has quit IRC		05:04
*** kun_huang has quit IRC		05:04
*** jamielennox\|away is now known as jamielennox		05:04
*** ediardo has joined #openstack-infra		05:04
*** asingh has joined #openstack-infra		05:04
*** ggherdov- has joined #openstack-infra		05:04
*** dham1 has quit IRC		05:04
*** hrybacki has joined #openstack-infra		05:04
*** agordeev has joined #openstack-infra		05:05
*** warpc has joined #openstack-infra		05:05
*** alexey_weyl has joined #openstack-infra		05:05
*** rockstar has joined #openstack-infra		05:05
*** cargonza has joined #openstack-infra		05:06
*** alexey_weyl has quit IRC		05:06
*** dimtruck is now known as zz_dimtruck		05:06
*** dham1 has joined #openstack-infra		05:06
*** harlowja_at_home has joined #openstack-infra		05:06
*** dougwig has joined #openstack-infra		05:07
*** kun_huang has joined #openstack-infra		05:07
*** mgkwill has quit IRC		05:07
*** mgkwill has joined #openstack-infra		05:08
*** pgadiya has quit IRC		05:09
openstackgerrit	Merged openstack-infra/project-config: import the dependencies needed for creating stable branches https://review.openstack.org/362418	05:10
*** salv-orlando has quit IRC		05:10
openstackgerrit	Merged openstack-infra/project-config: Change ACLs for fuel-qa project https://review.openstack.org/359704	05:10
*** roxanaghe has joined #openstack-infra		05:11
*** jaranovich has quit IRC		05:13
*** dirk has quit IRC		05:13
*** rackertom has quit IRC		05:13
*** petems has quit IRC		05:13
*** puiterwijk\|cld has quit IRC		05:13
*** teran has quit IRC		05:13
*** madhuvishy has quit IRC		05:13
*** andrewbogott has quit IRC		05:13
AJaeger	project-config cores, could you review the storyboard publishing change so that Zara has documents, please? https://review.openstack.org/363795	05:13
*** salv-orlando has joined #openstack-infra		05:14
*** jaranovich has joined #openstack-infra		05:14
*** madhuvishy has joined #openstack-infra		05:14
*** dirk has joined #openstack-infra		05:14
openstackgerrit	Merged openstack/diskimage-builder: add no_proxy when debootstrap trying to use proxy https://review.openstack.org/345830	05:14
openstackgerrit	Merged openstack/diskimage-builder: Update GRUB_MKCONFIG for detecting what's installed https://review.openstack.org/358588	05:14
*** rackertom has joined #openstack-infra		05:14
*** puiterwijk\|cld has joined #openstack-infra		05:15
*** teran has joined #openstack-infra		05:15
*** andrewbogott has joined #openstack-infra		05:15
*** petems has joined #openstack-infra		05:15
openstackgerrit	Ian Wienand proposed openstack-infra/devstack-gate: [WIP] Refactor devstack log copying https://review.openstack.org/364046	05:17
*** senk has quit IRC		05:19
*** nwkarsten has quit IRC		05:21
*** nwkarsten has joined #openstack-infra		05:22
*** pgadiya has joined #openstack-infra		05:22
*** sdake has joined #openstack-infra		05:23
openstackgerrit	Merged openstack-infra/os-loganalyze: Test high-precision timestamps in console log https://review.openstack.org/331898	05:23
*** nwkarsten has quit IRC		05:26
*** roxanaghe has quit IRC		05:26
openstackgerrit	Andreas Jaeger proposed openstack-infra/project-config: Update api-jobs https://review.openstack.org/364076	05:27
*** shashank_hegde has quit IRC		05:30
*** netsin has quit IRC		05:30
*** roxanaghe has joined #openstack-infra		05:31
*** yamamoto_ has joined #openstack-infra		05:34
*** javeriak has joined #openstack-infra		05:34
*** javeriak has quit IRC		05:34
*** javeriak has joined #openstack-infra		05:35
*** thorst has joined #openstack-infra		05:35
openstackgerrit	Ian Wienand proposed openstack/diskimage-builder: Add IMAGE_ELEMENT_YAML https://review.openstack.org/335265	05:36
openstackgerrit	Ian Wienand proposed openstack/diskimage-builder: Making element overriding explicit https://review.openstack.org/334785	05:36
openstackgerrit	Ian Wienand proposed openstack/diskimage-builder: Convert pkg-map and svc-map copies to explicit variables https://review.openstack.org/335308	05:36
*** roxanaghe has quit IRC		05:38
*** asselin_ has joined #openstack-infra		05:38
*** asselin__ has quit IRC		05:40
AJaeger	ianw, jhesketh, could you review https://review.openstack.org/364076 as well, please? I need some debugging logs for a change...	05:42
*** ihrachys has joined #openstack-infra		05:42
jhesketh	AJaeger: looking	05:42
AJaeger	thanks	05:42
*** thorst has quit IRC		05:42
*** ilyashakhat has joined #openstack-infra		05:42
*** zz_dimtruck is now known as dimtruck		05:42
*** Genek has quit IRC		05:43
jhesketh	AJaeger: +w, but we should consider reverting that once you're done (not sure how much noise it'll add to the log storage)	05:44
*** Genek has joined #openstack-infra		05:44
openstackgerrit	Merged openstack/diskimage-builder: Add tests for building *-minimal images https://review.openstack.org/181162	05:44
*** Genek has quit IRC		05:44
AJaeger	jhesketh: we have that on all tox based jobs - so I would leave it in unless I hear of a different policy...	05:44
*** Genek has joined #openstack-infra		05:44
*** cody-somerville has quit IRC		05:45
jhesketh	that's true, I wonder why these were missing..	05:45
*** Genek has quit IRC		05:45
*** Genek has joined #openstack-infra		05:45
AJaeger	jhesketh: see also I8ff8bef0aebdfd1a1e3d2e42a9972e9fb438cd0b and Id2ab952a7b1d34e659916bc38b41be48dd4e34f5	05:45
AJaeger	So, we added some more recently - but forgot these IMHO.	05:45
jhesketh	righto	05:46
*** Genek has quit IRC		05:46
*** Genek has joined #openstack-infra		05:46
AJaeger	bbl	05:46
*** Genek has quit IRC		05:47
openstackgerrit	Kien Ha proposed openstack-infra/jenkins-job-builder: Remove fail_required parameter from convert xml https://review.openstack.org/346140	05:47
*** Genek has joined #openstack-infra		05:47
*** Genek has quit IRC		05:47
openstackgerrit	Merged openstack-infra/project-config: Update storyboard publishing https://review.openstack.org/363795	05:47
*** Genek has joined #openstack-infra		05:48
*** Genek has quit IRC		05:48
*** adriant has quit IRC		05:49
openstackgerrit	Merged openstack-infra/project-config: Update api-jobs https://review.openstack.org/364076	05:51
*** harlowja_at_home has quit IRC		05:51
*** Genek has joined #openstack-infra		05:52
*** binbincong has quit IRC		05:52
*** yamahata has quit IRC		05:52
*** shashank_hegde has joined #openstack-infra		05:54
*** ilyashakhat has quit IRC		05:56
*** Genek has quit IRC		05:57
*** nstolyarenko has joined #openstack-infra		05:57
*** florianf has joined #openstack-infra		05:58
*** edmondsw has joined #openstack-infra		05:59
*** edmondsw has quit IRC		06:00
*** Sukhdev has joined #openstack-infra		06:01
*** nstolyarenko has quit IRC		06:02
*** binbincong has joined #openstack-infra		06:04
openstackgerrit	Ian Wienand proposed openstack-infra/project-config: Switch centos7 job to neutron https://review.openstack.org/362608	06:04
*** coolsvap has joined #openstack-infra		06:05
*** abregman has quit IRC		06:07
*** dteselkin_ has quit IRC		06:08
*** rcernin has quit IRC		06:08
*** katyafervent_awa has quit IRC		06:08
*** asilenkov has quit IRC		06:08
*** salv-orl_ has joined #openstack-infra		06:08
*** kzaitsev_ws has quit IRC		06:08
*** igormarnat has quit IRC		06:09
*** pkarikh has quit IRC		06:09
*** mikelk has joined #openstack-infra		06:09
*** rakhmerov has quit IRC		06:10
*** tsufiev has quit IRC		06:10
*** penguinolog has quit IRC		06:10
*** ilyashakhat has joined #openstack-infra		06:10
*** dteselkin has joined #openstack-infra		06:10
openstackgerrit	Merged openstack/diskimage-builder: Disabled IPv6 privacy extensions https://review.openstack.org/356456	06:11
*** salv-orlando has quit IRC		06:11
*** hashar has joined #openstack-infra		06:11
*** penguinolog has joined #openstack-infra		06:12
*** pkarikh has joined #openstack-infra		06:12
*** hashar is now known as hasharAway		06:12
*** _nadya_ has joined #openstack-infra		06:12
*** rakhmerov has joined #openstack-infra		06:13
*** katyafervent_awa has joined #openstack-infra		06:14
*** tsufiev has joined #openstack-infra		06:15
*** kzaitsev_ws has joined #openstack-infra		06:15
*** asilenkov has joined #openstack-infra		06:16
*** binbincong has quit IRC		06:16
*** igormarnat has joined #openstack-infra		06:16
odyssey4me	jhesketh if you have a moment to review https://review.openstack.org/363640 & https://review.openstack.org/363783 I'd appreciate it	06:16
*** lock__ has joined #openstack-infra		06:18
*** oanson has joined #openstack-infra		06:18
*** Jeffrey4l__ is now known as Jeffrey4l		06:19
*** dulek_ has joined #openstack-infra		06:20
*** mnencia_ has joined #openstack-infra		06:20
*** taron_ has joined #openstack-infra		06:20
*** jlk` has joined #openstack-infra		06:21
*** ljjjustin has quit IRC		06:21
*** dulek has quit IRC		06:21
*** mnencia has quit IRC		06:21
*** taron has quit IRC		06:21
*** hichihar_ has joined #openstack-infra		06:21
*** jlk has quit IRC		06:21
*** vponomaryov has quit IRC		06:21
*** dulek_ is now known as dulek		06:21
*** mnencia_ is now known as mnencia		06:21
*** ljjjustin has joined #openstack-infra		06:21
*** hichihara has quit IRC		06:22
*** lock_ has quit IRC		06:22
*** gnuoy has quit IRC		06:22
*** gnuoy has joined #openstack-infra		06:22
*** vponomaryov has joined #openstack-infra		06:22
*** hichihar_ has quit IRC		06:22
*** hichihara has joined #openstack-infra		06:23
*** kzaitsev_mb has joined #openstack-infra		06:25
*** javeriak_ has joined #openstack-infra		06:26
*** binbincong has joined #openstack-infra		06:28
*** javeriak has quit IRC		06:29
openstackgerrit	Merged openstack/diskimage-builder: Generate and use upper-constraints for ironic-agent https://review.openstack.org/360547	06:29
openstackgerrit	Merged openstack/diskimage-builder: Update portage only if needed https://review.openstack.org/358592	06:29
openstackgerrit	Merged openstack/diskimage-builder: Clarify OVERWRITE_OLD_IMAGE docs https://review.openstack.org/355607	06:29
openstackgerrit	Merged openstack/diskimage-builder: Add blurb about communication to docs landing page https://review.openstack.org/355533	06:29
*** pcaruana has joined #openstack-infra		06:30
*** kzaitsev_mb has quit IRC		06:31
rcarrillocruz	weeee	06:31
rcarrillocruz	infracloud ran jobs during the night:	06:32
rcarrillocruz	http://logs.openstack.org/63/363863/2/check/gate-ironic-pep8-ubuntu-xenial/e3700fb/console.html	06:32
rcarrillocruz	http://logs.openstack.org/periodic-stable/periodic-octavia-docs-mitaka/51235bc/console.html	06:32
rcarrillocruz	MORNING!	06:32
jhesketh	odyssey4me: looking	06:33
*** ihrachys has quit IRC		06:33
*** senk has joined #openstack-infra		06:35
jhesketh	odyssey4me: do you feel like iterating on a nit on 363640	06:36
odyssey4me	jhesketh can do	06:36
jhesketh	odyssey4me: if it's too nitty though, I'm happy to +W	06:36
odyssey4me	jhesketh hmm, perhaps I need more coffee - not sure what you mean there	06:38
*** javeriak has joined #openstack-infra		06:38
*** javeria__ has joined #openstack-infra		06:39
*** javeriak_ has quit IRC		06:39
jhesketh	odyssey4me: you've put the job in projects.yaml twice where the only difference is the suffix -nv right?	06:39
jhesketh	but the voting version isn't currently used in layout.yaml.. so why not just define it the one time and then when you're ready to make the job voting just remove the job-suffix?	06:40
*** dteselkin has quit IRC		06:40
odyssey4me	jhesketh yeah, oh I suppose I could just use one - assuming they all work right	06:40
odyssey4me	it might end up with a split based on platform	06:40
*** thorst has joined #openstack-infra		06:40
jhesketh	odyssey4me: ah, I see... you might make some voting before others..	06:41
*** asilenkov has quit IRC		06:41
*** rcernin has joined #openstack-infra		06:41
odyssey4me	jhesketh I'll be doing a folllow up pretty soon to move some of those jobs to voting and can clean it up then	06:41
jhesketh	odyssey4me: I'm fine with how it is, just need to remember to remove the non-voting duplicate once they are all ovitng	06:41
*** igormarnat has quit IRC		06:41
jhesketh	okay, sounds good to me	06:41
*** tsufiev has quit IRC		06:41
*** pkarikh has quit IRC		06:41
*** penguinolog has quit IRC		06:41
*** kzaitsev_ws has quit IRC		06:41
*** rakhmerov has quit IRC		06:42
*** katyafervent_awa has quit IRC		06:42
jhesketh	odyssey4me: +w	06:42
*** javeriak has quit IRC		06:42
*** Sukhdev has quit IRC		06:42
odyssey4me	thanks jhesketh	06:42
odyssey4me	I have a job ready to test them all - I'll patch again later this morning to move them to voting	06:43
jhesketh	shiny	06:44
*** rcernin has quit IRC		06:45
*** rcernin has joined #openstack-infra		06:45
*** javeria__ has quit IRC		06:46
*** Genek has joined #openstack-infra		06:46
*** kaisers_ has joined #openstack-infra		06:46
*** javeriak has joined #openstack-infra		06:46
*** thorst has quit IRC		06:47
openstackgerrit	Merged openstack-infra/project-config: Move unsuccessful non-voting OSA jobs to experimental https://review.openstack.org/363783	06:47
*** _nadya_ has quit IRC		06:47
*** javeriak has quit IRC		06:48
*** pkarikh has joined #openstack-infra		06:48
*** martinkopec has joined #openstack-infra		06:48
*** javeriak has joined #openstack-infra		06:48
*** rakhmerov has joined #openstack-infra		06:49
*** penguinolog has joined #openstack-infra		06:49
*** ilyashakhat has quit IRC		06:50
*** salv-orl_ has quit IRC		06:50
*** katyafervent_awa has joined #openstack-infra		06:51
*** dteselkin has joined #openstack-infra		06:53
*** tsufiev has joined #openstack-infra		06:53
*** asilenkov has joined #openstack-infra		06:54
*** igormarnat has joined #openstack-infra		06:54
openstackgerrit	Merged openstack-infra/project-config: Add OSA keystone uwsgi functional tests https://review.openstack.org/363640	06:54
*** hferenc has quit IRC		06:55
*** kzaitsev_ws has joined #openstack-infra		06:58
*** javeriak_ has joined #openstack-infra		06:59
*** javeria__ has joined #openstack-infra		07:00
*** _nadya_ has joined #openstack-infra		07:00
*** javeriak has quit IRC		07:00
*** oanson has quit IRC		07:00
*** _nadya_ has quit IRC		07:00
*** claudiub has joined #openstack-infra		07:01
*** vinaypotluri has quit IRC		07:02
*** tesseract- has joined #openstack-infra		07:03
*** rwsu has joined #openstack-infra		07:03
*** esikachev has joined #openstack-infra		07:03
*** netsin has joined #openstack-infra		07:03
*** javeriak_ has quit IRC		07:04
*** aeng has quit IRC		07:04
*** andreas_s has joined #openstack-infra		07:05
*** spzala has joined #openstack-infra		07:06
*** yolanda has joined #openstack-infra		07:06
*** abregman has joined #openstack-infra		07:06
*** Na3iL has joined #openstack-infra		07:08
*** drifterza has joined #openstack-infra		07:08
openstackgerrit	Ricardo Carrillo Cruz proposed openstack-infra/project-config: Bump Infracloud vanilla max-servers to 50 https://review.openstack.org/364101	07:09
*** javeriak has joined #openstack-infra		07:10
*** spzala has quit IRC		07:10
*** mptacekx has joined #openstack-infra		07:11
openstackgerrit	Tony Breeds proposed openstack-infra/irc-meetings: Update the QA team meeting chair https://review.openstack.org/364065	07:12
*** javeria__ has quit IRC		07:13
AJaeger	rcarrillocruz: woot! Great to see the progress on infracloud!	07:13
rcarrillocruz	\o/	07:13
mptacekx	Hi, I am contributing to intel-nfv-ci, currently we are facing an issue with unstable connectivity to OVH file server (ftp.cluster011.ovh.net,213.186.33.208), there are some drop-outs fr e.g. 15mins nearly every second hr producing unstable results. Is it a known issue ? Thanks	07:14
AJaeger	rcarrillocruz: will you be around to babysit the cloud? Then let's ask others to +2A 364101	07:14
*** sshnaidm\|afk is now known as sshnaidm		07:14
*** javeriak has quit IRC		07:14
*** javeriak has joined #openstack-infra		07:15
AJaeger	rcarrillocruz: btw. http://grafana.openstack.org/dashboard/db/nodepool-infra-cloud shows vanilla and west. Is west really correct as name?	07:17
openstackgerrit	Peter Zhurba proposed openstack-infra/project-config: Add repo for openstack/puppet-glare. https://review.openstack.org/362950	07:17
*** jamielennox is now known as jamielennox\|away		07:17
*** yaume has joined #openstack-infra		07:18
*** shashank_hegde has quit IRC		07:19
*** bexelbie has quit IRC		07:20
*** matbu\|afk is now known as matbu		07:20
*** taron_ has quit IRC		07:21
*** nstolyarenko has joined #openstack-infra		07:21
*** dimtruck is now known as zz_dimtruck		07:22
*** jlvillal has quit IRC		07:22
*** taron has joined #openstack-infra		07:22
*** bexelbie has joined #openstack-infra		07:23
*** nwkarsten has joined #openstack-infra		07:23
*** jlvillal has joined #openstack-infra		07:23
AJaeger	zigo: please check https://review.openstack.org/356795	07:24
*** jpich has joined #openstack-infra		07:25
*** kzaitsev_mb has joined #openstack-infra		07:27
*** ifarkas_afk is now known as ifarkas		07:28
*** nstolyarenko has quit IRC		07:28
*** nwkarsten has quit IRC		07:29
rcarrillocruz	AJaeger: it is not, that is something that should be corrected	07:29
rcarrillocruz	west = vanilla	07:29
rcarrillocruz	east = chocolate	07:29
rcarrillocruz	AJaeger: yeah, let's not land 364101 for now	07:29
rcarrillocruz	i'm off for a couple hours	07:30
rcarrillocruz	i better wip it, thx	07:30
*** Genek has quit IRC		07:31
rcarrillocruz	i also need to double-check we won't hit quota issues	07:31
*** apetrich has quit IRC		07:31
AJaeger	rcarrillocruz: ok	07:31
*** abregman_ has joined #openstack-infra		07:32
rcarrillocruz	AJaeger: actually, i see vanilla/chocolate names on project-config/grafana	07:32
*** abregman has quit IRC		07:32
rcarrillocruz	where do you see that, in the UI of Grafana?	07:33
AJaeger	rcarrillocruz: see the link - in the UI	07:33
rcarrillocruz	ah, yeah, the combolist	07:33
*** apetrich has joined #openstack-infra		07:33
AJaeger	historic data?	07:33
rcarrillocruz	probably it needs a restart or something	07:33
AJaeger	Ah, ok...	07:33
rcarrillocruz	i'll check with pabelanger later	07:33
AJaeger	thanks	07:33
AJaeger	yeah, it's not in project-config - checked there now.	07:34
*** dizquierdo has joined #openstack-infra		07:34
*** javeriak has quit IRC		07:35
openstackgerrit	Yuval Brik proposed openstack-infra/project-config: Rename Smaug to Karbor https://review.openstack.org/353304	07:35
ttx	tristanC: Election schedule looks good to me. Note that there is no Requirements team in the PTL list (but that may be ok since there was a very recent election for that ?)	07:36
*** mptacekxx has joined #openstack-infra		07:38
*** mptacekx has quit IRC		07:40
openstackgerrit	Martin André proposed openstack-infra/tripleo-ci: [WIP] Fix scp command with IPv6 addresses https://review.openstack.org/363545	07:41
*** ihrachys has joined #openstack-infra		07:41
mptacekxx	?	07:41
skraynev	AJaeger: hi. sorry for the interruption again. could you please add me to the groups https://review.openstack.org/#/admin/groups/1547,members https://review.openstack.org/#/admin/groups/1546,members https://review.openstack.org/#/admin/groups/1548,members according patch https://review.openstack.org/#/c/357745/	07:41
*** mptacekxx has quit IRC		07:42
*** mptacekxx has joined #openstack-infra		07:43
*** sdake has quit IRC		07:43
*** mptacekxx has quit IRC		07:43
*** sdake has joined #openstack-infra		07:44
*** caowei has quit IRC		07:44
*** javeriak has joined #openstack-infra		07:44
*** thorst has joined #openstack-infra		07:45
*** mptacekx has joined #openstack-infra		07:45
tristanC	ttx: thanks! indeed it's missing Requirements, I've added in the last patchset	07:46
ttx	tristanC: I take it you're not in Quebec right now	07:46
*** esikachev has quit IRC		07:47
*** bexelbie has quit IRC		07:50
*** bexelbie has joined #openstack-infra		07:50
*** matthewbodkin has joined #openstack-infra		07:51
*** salv-orlando has joined #openstack-infra		07:51
*** thorst has quit IRC		07:52
*** javeriak has quit IRC		07:54
*** ihrachys has quit IRC		07:55
*** yfried has joined #openstack-infra		07:55
yfried	Hi, looking for help with shade dependencies	07:55
yfried	shade requires os-client-config>=1.20.0 and python-magnumclient>=2.1.0	07:56
yfried	python-magnumclient: os-client-config!=1.19.0,!=1.19.1,!=1.20.0,!=1.20.1,!=1.21.0,>=1.13.1	07:56
*** shardy has joined #openstack-infra		07:57
*** dtantsur\|afk is now known as dtantsur		07:58
*** esikachev has joined #openstack-infra		07:58
*** zzzeek has quit IRC		08:00
*** salv-orlando has quit IRC		08:00
*** ljjjustin has quit IRC		08:00
*** zzzeek has joined #openstack-infra		08:00
*** dchen has joined #openstack-infra		08:02
*** _nadya_ has joined #openstack-infra		08:03
*** dingyichen has quit IRC		08:04
*** jaosorior has joined #openstack-infra		08:04
*** sflanigan has quit IRC		08:05
openstackgerrit	Merged openstack-infra/project-config: Clone sources of ironic-inspector and ironic-inspector-client https://review.openstack.org/363922	08:07
*** _nadya_ has quit IRC		08:08
*** nmagnezi has joined #openstack-infra		08:09
*** oanson has joined #openstack-infra		08:11
*** matrohon has joined #openstack-infra		08:12
dlahn	good morning	08:15
openstackgerrit	Ian Wienand proposed openstack-infra/devstack-gate: Fix devstack subunit output https://review.openstack.org/364045	08:16
openstackgerrit	Ian Wienand proposed openstack-infra/devstack-gate: [WIP] Refactor devstack log copying https://review.openstack.org/364046	08:16
skraynev	sorry for the interruption again. could you please add me to the groups https://review.openstack.org/#/admin/groups/1547,members https://review.openstack.org/#/admin/groups/1546,members https://review.openstack.org/#/admin/groups/1548,members according patch https://review.openstack.org/#/c/357745/	08:16
AJaeger	skraynev: I can't, I don't have those permissions. I hope somebody reads backscroll and gets back to you. If not: ask when more admins are around - or ask on the infra mailing list.	08:19
skraynev	AJaeger: ok. got it. thank you for advise	08:19
*** shashank_hegde has joined #openstack-infra		08:22
*** kushal has joined #openstack-infra		08:23
*** asettle has joined #openstack-infra		08:26
openstackgerrit	Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Shorten job names for OSA os_keystone repo https://review.openstack.org/364141	08:26
odyssey4me	project config cores - an urgent review of the above would be appreciated	08:26
*** esikachev has quit IRC		08:27
openstackgerrit	Ian Wienand proposed openstack-infra/devstack-gate: Update bashate to 0.5.0 https://review.openstack.org/236815	08:27
AJaeger	odyssey4me: +2A	08:27
odyssey4me	thanks AJaeger	08:28
openstackgerrit	Sagi Shnaidman proposed openstack-infra/tripleo-ci: POC: WIP: oooq undercloud install https://review.openstack.org/358919	08:28
ianw	that problem keeps hitting us	08:28
*** kzaitsev_mb has quit IRC		08:28
openstackgerrit	Martin André proposed openstack-infra/tripleo-ci: Fix scp command with IPv6 addresses https://review.openstack.org/363545	08:29
*** shashank_hegde has quit IRC		08:29
openstackgerrit	Andrey Nikitin proposed openstack-infra/project-config: Add one new Fuel's plugin https://review.openstack.org/362002	08:30
*** dchen has quit IRC		08:31
*** r-mibu has quit IRC		08:33
*** priteau has joined #openstack-infra		08:37
*** hasharAway is now known as hashar		08:37
*** esikachev has joined #openstack-infra		08:38
*** r-mibu has joined #openstack-infra		08:38
*** lucas-dinner is now known as lucasagomes		08:38
*** Julien-zte has quit IRC		08:40
*** berendt has joined #openstack-infra		08:41
*** e0ne has joined #openstack-infra		08:44
*** bethwhite_ has joined #openstack-infra		08:44
openstackgerrit	Evgeny Antyshev proposed openstack-infra/nodepool: Add missing month argument https://review.openstack.org/364151	08:45
*** hichihara has quit IRC		08:45
openstackgerrit	Ricardo Carrillo Cruz proposed openstack-infra/system-config: Change west/east naming for vanilla/chocolate on set quotas playbook https://review.openstack.org/364153	08:47
rcarrillocruz	jhesketh: if you're still around, do you mind reviewing ^ ?	08:48
rcarrillocruz	i need to bump quotas on the zuul projects, prior to bumping max-servers on nodepool	08:48
*** dizquierdo has quit IRC		08:48
rcarrillocruz	and i rather do it via playbook, instead with a one-off command that is not documented/tracked anywhere	08:48
rcarrillocruz	or ianw ^	08:49
*** thorst has joined #openstack-infra		08:50
ttx	project-config core reviewers: could you please approve https://review.openstack.org/#/c/363895/ so that automatic tagging of libraries is fixed ?	08:50
openstackgerrit	Sagi Shnaidman proposed openstack-infra/tripleo-ci: POC: WIP: oooq undercloud install https://review.openstack.org/358919	08:50
ttx	AJaeger already +2ed it	08:51
*** javeriak has joined #openstack-infra		08:51
ttx	(and we have a truckload of libs to process today so that would help)	08:51
rcarrillocruz	ttx: approved	08:52
openstackgerrit	Merged openstack-infra/project-config: Shorten job names for OSA os_keystone repo https://review.openstack.org/364141	08:52
ttx	rcarrillocruz: awesome, thx	08:52
*** sc`_ has joined #openstack-infra		08:53
*** dcwangmit01_ has joined #openstack-infra		08:53
*** mbacchi_ has joined #openstack-infra		08:54
*** hwoarang_ has joined #openstack-infra		08:54
*** derekh has joined #openstack-infra		08:54
*** jhesketh_ has joined #openstack-infra		08:54
*** HenryG_ has joined #openstack-infra		08:54
openstackgerrit	Ricardo Carrillo Cruz proposed openstack-infra/project-config: Bump Infracloud vanilla max-servers to 50 https://review.openstack.org/364101	08:54
*** amotoki_ has joined #openstack-infra		08:55
openstackgerrit	Pawel Koniszewski proposed openstack-infra/project-config: Start LM tests with grenade to test rolling upgrades https://review.openstack.org/364162	08:55
*** jeblair_ has joined #openstack-infra		08:55
*** Ravikiran_K has joined #openstack-infra		08:56
*** jlvillal_ has joined #openstack-infra		08:57
*** toabctl_ has joined #openstack-infra		08:57
*** thorst has quit IRC		08:57
*** dtardivel has joined #openstack-infra		08:57
*** shardy has quit IRC		08:58
*** mptacekx has quit IRC		08:58
*** jlvillal has quit IRC		08:58
*** drifterza has quit IRC		08:58
*** dulek has quit IRC		08:58
*** amotoki has quit IRC		08:58
*** vikrant has quit IRC		08:58
*** skipp has quit IRC		08:58
*** afred312 has quit IRC		08:58
*** M-docaedo_vector has quit IRC		08:58
*** _degorenko\|afk has quit IRC		08:58
*** lennyb has quit IRC		08:58
*** spiffxp has quit IRC		08:58
*** wfoster has quit IRC		08:58
*** jhesketh has quit IRC		08:58
*** darrenc has quit IRC		08:58
*** mat128 has quit IRC		08:58
*** SotK has quit IRC		08:58
*** dcwangmit01 has quit IRC		08:58
*** bapalm has quit IRC		08:58
*** sc` has quit IRC		08:58
*** HenryG has quit IRC		08:58
*** anteaya has quit IRC		08:58
*** sambetts\|afk has quit IRC		08:58
*** stevebaker has quit IRC		08:58
*** jeblair has quit IRC		08:58
*** michaelxin has quit IRC		08:58
*** whoops has quit IRC		08:58
*** mbacchi has quit IRC		08:58
*** antonym has quit IRC		08:58
*** Anticimex has quit IRC		08:58
*** eliqiao has quit IRC		08:58
*** hwoarang has quit IRC		08:58
*** toabctl has quit IRC		08:58
*** aderyugin has quit IRC		08:58
*** andymccr has quit IRC		08:58
*** jgrassler has quit IRC		08:58
*** toabctl_ is now known as toabctl		08:58
*** sc`_ is now known as sc`		08:58
*** salv-orlando has joined #openstack-infra		08:58
*** bapalm has joined #openstack-infra		08:59
*** mat128 has joined #openstack-infra		08:59
*** andymccr has joined #openstack-infra		08:59
*** toabctl has quit IRC		08:59
*** Anticimex has joined #openstack-infra		09:00
*** antonym has joined #openstack-infra		09:00
*** toabctl has joined #openstack-infra		09:00
*** whoops has joined #openstack-infra		09:00
*** jlvillal_ is now known as jlvillal		09:00
*** _nadya_ has joined #openstack-infra		09:01
*** jlvillal is now known as Guest77956		09:01
*** esikachev has quit IRC		09:01
*** salv-orlando has quit IRC		09:01
*** tkelsey has joined #openstack-infra		09:01
*** shardy has joined #openstack-infra		09:03
*** mptacekx has joined #openstack-infra		09:03
*** drifterza has joined #openstack-infra		09:03
*** dulek has joined #openstack-infra		09:03
*** vikrant has joined #openstack-infra		09:03
*** afred312 has joined #openstack-infra		09:03
*** M-docaedo_vector has joined #openstack-infra		09:03
*** _degorenko\|afk has joined #openstack-infra		09:03
*** lennyb has joined #openstack-infra		09:03
*** spiffxp has joined #openstack-infra		09:03
*** wfoster has joined #openstack-infra		09:03
*** darrenc has joined #openstack-infra		09:03
*** SotK has joined #openstack-infra		09:03
*** anteaya has joined #openstack-infra		09:03
*** sambetts\|afk has joined #openstack-infra		09:03
*** stevebaker has joined #openstack-infra		09:03
*** eliqiao has joined #openstack-infra		09:03
*** aderyugin has joined #openstack-infra		09:03
*** skipp has joined #openstack-infra		09:03
*** javeriak has quit IRC		09:03
*** mhickey has joined #openstack-infra		09:03
openstackgerrit	Merged openstack-infra/project-config: be more careful using setuptools commands in release script https://review.openstack.org/363895	09:04
*** jgrassler has joined #openstack-infra		09:05
*** salv-orlando has joined #openstack-infra		09:05
*** tkelsey has quit IRC		09:05
*** asettle has quit IRC		09:08
*** abregman_ has quit IRC		09:08
openstackgerrit	Adam Coldrick proposed openstack-infra/storyboard: Send notifications to subscribers for worklists https://review.openstack.org/354730	09:09
openstackgerrit	Adam Coldrick proposed openstack-infra/storyboard: Make it possible to get worklist/board timeline events via the API https://review.openstack.org/354729	09:09
openstackgerrit	Adam Coldrick proposed openstack-infra/storyboard: Don't allow users to subscribe to private worklists they can't see https://review.openstack.org/363776	09:10
*** salv-orlando has quit IRC		09:10
*** Genek has joined #openstack-infra		09:10
*** asettle has joined #openstack-infra		09:12
openstackgerrit	mathieu bultel proposed openstack-infra/tripleo-ci: WIP - Implement overcloud upgrade job - Mitaka -> Newton https://review.openstack.org/323750	09:16
*** HenryG_ is now known as HenryG		09:17
*** rossella_s has quit IRC		09:18
*** rossella_s has joined #openstack-infra		09:19
*** netsin has quit IRC		09:21
*** sambetts\|afk is now known as sambetts		09:23
*** abregman_ has joined #openstack-infra		09:23
*** mrunge has quit IRC		09:25
*** ihrachys has joined #openstack-infra		09:25
*** Genek has quit IRC		09:28
*** Genek has joined #openstack-infra		09:28
*** HenryG has quit IRC		09:29
*** HenryG has joined #openstack-infra		09:29
openstackgerrit	Vladyslav Drok proposed openstack-infra/project-config: Add Q_USE_PROVIDERNET_FOR_PUBLIC in multitenant job https://review.openstack.org/364187	09:30
*** hwoarang_ is now known as hwoarang		09:31
*** netsin has joined #openstack-infra		09:32
*** Genek has quit IRC		09:32
*** mrunge has joined #openstack-infra		09:36
*** flepied has quit IRC		09:36
*** flepied1 has joined #openstack-infra		09:36
*** darvon has quit IRC		09:38
*** darvon has joined #openstack-infra		09:40
*** zshuo has quit IRC		09:41
openstackgerrit	Alexander Evseev proposed openstack-infra/puppet-zuul: Replace upstream module by Mirantis' one https://review.openstack.org/364194	09:42
rcarrillocruz	sweet, after DC folks magic I'm recovering compute005	09:43
rcarrillocruz	one less to fix	09:43
*** _degorenko\|afk is now known as degorenko		09:43
*** chem\|off has joined #openstack-infra		09:43
*** chem\|off is now known as chem		09:43
*** salv-orlando has joined #openstack-infra		09:45
*** daemontool has joined #openstack-infra		09:45
*** sdague has joined #openstack-infra		09:46
*** chem has quit IRC		09:49
*** chem has joined #openstack-infra		09:49
openstackgerrit	Yuriy Taraday proposed openstack-infra/project-config: Turn off Puppet jobs on old fuel-octane branches https://review.openstack.org/364199	09:50
openstackgerrit	Pawel Koniszewski proposed openstack-infra/project-config: LM tests with grenade to test rolling upgrades https://review.openstack.org/364162	09:51
*** yanyanhu has quit IRC		09:52
YorikSar	Any infra cores around? I've forgot to disable job for old branches before making it voting: https://review.openstack.org/364199 - it blocks octane team, please review.	09:53
openstackgerrit	Merged openstack-infra/tripleo-ci: Fix scp command with IPv6 addresses https://review.openstack.org/363545	09:55
*** matthewbodkin has quit IRC		09:55
*** thorst has joined #openstack-infra		09:55
*** dizquierdo has joined #openstack-infra		09:58
*** flepied1 has quit IRC		10:00
*** dizquierdo has quit IRC		10:00
*** flepied has joined #openstack-infra		10:00
*** Thelo_ has joined #openstack-infra		10:01
Thelo_	l	10:01
*** sdake has quit IRC		10:01
*** kzaitsev_mb has joined #openstack-infra		10:02
*** HeOS has quit IRC		10:02
*** yamamoto_ has quit IRC		10:02
*** thorst has quit IRC		10:03
*** zhurong has quit IRC		10:03
*** HeOS has joined #openstack-infra		10:04
*** asettle has quit IRC		10:05
*** javeriak has joined #openstack-infra		10:08
*** matthewbodkin has joined #openstack-infra		10:13
rcarrillocruz	thx ianw	10:15
*** asettle has joined #openstack-infra		10:16
HeOS	Hello, infra-team! Please repeat review of the following request: https://review.openstack.org/#/c/362002/.	10:19
*** jhesketh_ is now known as jhesketh		10:20
*** tqtran has joined #openstack-infra		10:20
*** dizquierdo has joined #openstack-infra		10:22
openstackgerrit	Merged openstack-infra/project-config: Turn off Puppet jobs on old fuel-octane branches https://review.openstack.org/364199	10:22
openstackgerrit	Merged openstack-infra/system-config: Change west/east naming for vanilla/chocolate on set quotas playbook https://review.openstack.org/364153	10:24
*** pilgrimstack has quit IRC		10:24
*** tqtran has quit IRC		10:25
*** pilgrimstack has joined #openstack-infra		10:27
*** nwkarsten has joined #openstack-infra		10:27
*** pilgrimstack has quit IRC		10:31
*** nwkarsten has quit IRC		10:32
*** yolanda has quit IRC		10:32
*** nstolyarenko has joined #openstack-infra		10:32
openstackgerrit	Derek Higgins proposed openstack-infra/tripleo-ci: Install tripleo-admin ssh keys on CI nodes https://review.openstack.org/363630	10:33
openstackgerrit	Bartosz Kupidura proposed openstack-infra/puppet-apps_site: [wip] Glare support for app-catalog https://review.openstack.org/359029	10:35
*** sigmavirus\|awa is now known as sigmavirus		10:37
*** Genek has joined #openstack-infra		10:39
openstackgerrit	Masayuki Igawa proposed openstack/gertty: Add size column to change list view https://review.openstack.org/364224	10:40
*** Qiming has quit IRC		10:40
*** pilgrimstack has joined #openstack-infra		10:41
*** Genek has quit IRC		10:43
*** yamamoto has joined #openstack-infra		10:43
*** esikachev has joined #openstack-infra		10:44
*** Qiming has joined #openstack-infra		10:45
openstackgerrit	Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Promote working os_keystone nv jobs to voting https://review.openstack.org/364226	10:47
openstackgerrit	Giulio Fidente proposed openstack-infra/tripleo-ci: [NO MERGE] Test Ceph RadosGW as replacement for Swift https://review.openstack.org/364227	10:48
*** Genek has joined #openstack-infra		10:48
*** asettle has quit IRC		10:48
*** dtantsur is now known as dtantsur\|lunch		10:49
*** gongysh has quit IRC		10:49
openstackgerrit	Bartosz Kupidura proposed openstack-infra/puppet-apps_site: [wip] Glare support for app-catalog https://review.openstack.org/359029	10:49
openstackgerrit	Jim Rollenhagen proposed openstack-infra/elastic-recheck: Add query for bug 1619232 https://review.openstack.org/364233	10:50
openstack	bug 1619232 in Ironic "Heartbeat()'s race condition: InvalidState: Can not transition from state 'deploying' on event 'resume' (no defined transition)" [High,New] https://launchpad.net/bugs/1619232 - Assigned to Lucas Alvares Gomes (lucasagomes)	10:50
*** yamamoto has quit IRC		10:52
wznoinsk	hi infra	10:52
wznoinsk	do you have any troubles with OVH of any kind today?	10:52
*** Ravikiran_K has quit IRC		10:52
*** esikachev has quit IRC		10:56
*** Genek has quit IRC		10:57
*** sskripnick has joined #openstack-infra		10:57
sskripnick	hi all	10:57
sskripnick	fatal: unable to access 'http://git.openstack.org/openstack/rally.git/': Failed to connect to git.openstack.org port 80: Connection refused	10:57
*** Genek has joined #openstack-infra		10:57
sskripnick	rally-ci got banned? -_-	10:57
*** salv-orlando has quit IRC		10:58
*** yamamoto has joined #openstack-infra		11:00
openstackgerrit	Merged openstack-infra/elastic-recheck: Add query for bug 1619232 https://review.openstack.org/364233	11:00
openstack	bug 1619232 in Ironic "Heartbeat()'s race condition: InvalidState: Can not transition from state 'deploying' on event 'resume' (no defined transition)" [High,New] https://launchpad.net/bugs/1619232 - Assigned to Lucas Alvares Gomes (lucasagomes)	11:00
*** thorst has joined #openstack-infra		11:00
AJaeger	sskripnick: there were some network problems that clarkb and others debugged yesterday - in the Rackspace datacenter. I don't know whether those are resolved. Is this a recent result? Is it reproduceable?	11:00
*** Genek has quit IRC		11:01
sdague	we've got a ton of multinode issues	11:03
sdague	where the subnode is not responding	11:04
*** rhallisey has joined #openstack-infra		11:05
sskripnick	AJaeger: like 15% attems are successful	11:05
openstackgerrit	Bartosz Kupidura proposed openstack-infra/puppet-apps_site: [wip] Glare support for app-catalog https://review.openstack.org/359029	11:06
AJaeger	sskripnick: that's sad ;( Best talk later with rest of team...	11:06
*** dtroyer has left #openstack-infra		11:06
sskripnick	AJaeger: got it. thanks	11:06
*** thorst has quit IRC		11:08
openstackgerrit	Csaba Henk proposed openstack-infra/project-config: remove manila's glusterfs xenial jobs https://review.openstack.org/359167	11:09
sdague	http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Killed%20timeout%20-s%209%5C%22%20AND%20build_name%3Amultinode	11:10
openstackgerrit	Merged openstack-infra/project-config: Promote working os_keystone nv jobs to voting https://review.openstack.org/364226	11:14
*** javeriak has quit IRC		11:17
*** rtheis has joined #openstack-infra		11:18
sdague	any infra-root up?	11:18
*** esikache1 has joined #openstack-infra		11:19
*** ramishra_ has joined #openstack-infra		11:20
*** ramishra has quit IRC		11:21
*** jkilpatr has joined #openstack-infra		11:21
*** kushal has quit IRC		11:24
rcarrillocruz	i am	11:25
rcarrillocruz	what's up	11:25
*** thorst has joined #openstack-infra		11:26
openstackgerrit	Sagi Shnaidman proposed openstack-infra/tripleo-ci: POC: WIP: oooq undercloud install https://review.openstack.org/358919	11:26
pkoniszewski	hello infra, is there a way to add grenade to existing job? I mean, I'm trying to work on live migration job with grenade, tried this approach but it did not work at all - https://review.openstack.org/#/c/364162/2/jenkins/jobs/devstack-gate.yaml	11:26
*** ldnunes has joined #openstack-infra		11:27
sdague	rcarrillocruz: something is really weird with multinode	11:28
*** nmagnezi has quit IRC		11:28
sdague	the subnode failed to connect rate has gone way up	11:28
sdague	http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Killed%20timeout%20-s%209%5C%22%20AND%20build_name%3Amultinode	11:28
*** nmagnezi has joined #openstack-infra		11:28
sdague	rcarrillocruz: it looks like this command hangs for 2 hours - http://logs.openstack.org/61/363061/8/gate/gate-tempest-dsvm-multinode-live-migration/6abf7c7/console.html#_2016-09-01_04_07_26_396005	11:30
sdague	and the subnode logs just stop a few minutes in	11:32
rcarrillocruz	output is not superuseful	11:32
rcarrillocruz	let me see	11:33
sdague	yeh	11:33
sdague	so this is definitely a spike as of 6 hours ago	11:33
*** lucasagomes is now known as lucas-hungry		11:33
*** Genek has joined #openstack-infra		11:34
sdague	it's not isolated to any particular provider	11:34
sdague	which is weird	11:35
tdurakov	sdague: I was wrong, it affects all multinode jobs	11:35
*** dprince has joined #openstack-infra		11:36
*** Hal1 has quit IRC		11:36
*** Hal1 has joined #openstack-infra		11:36
tdurakov	and there is a spike for last 24 hours	11:36
*** Hal2 has joined #openstack-infra		11:37
sdague	yeh	11:38
sdague	rcarrillocruz: any idea if there is an ansible option that would let us fail a bit faster if we lose the subnode?	11:38
*** asettle has joined #openstack-infra		11:39
sdague	this is super weird	11:41
rcarrillocruz	you could put the task as async and poll for it	11:41
rcarrillocruz	http://docs.ansible.com/ansible/playbooks_async.html	11:41
*** baoli has joined #openstack-infra		11:41
rcarrillocruz	i'm on the subnode	11:41
rcarrillocruz	on a 2-node i just held	11:41
*** rfolco has joined #openstack-infra		11:41
*** Hal1 has quit IRC		11:41
rcarrillocruz	and i don't see any logging or anything pointing to a lock or doing much stuff	11:41
sdague	rcarrillocruz: well, it's not all of them that are dying for sure	11:41
*** baoli_ has joined #openstack-infra		11:42
rcarrillocruz	and the node is pretty much idle, from top output	11:42
*** larainema has quit IRC		11:42
rcarrillocruz	guess i got a 'good' subnode	11:42
sdague	the last thing that runs is - https://github.com/openstack-dev/devstack/blob/08b1475c2a9923a202b2e5112e03c96aa1f9a5c0/stack.sh#L1386	11:42
sdague	or at least the last thing that logs	11:42
sdague	rcarrillocruz: oh, I have a node that I know is borked	11:43
*** dizquierdo has quit IRC		11:43
sdague	telnet://2001:4800:1ae1:18:f816:3eff:febf:faba:19885	11:43
rcarrillocruz	eugh, i have no ipv6	11:44
sdague	the subnode there is in trouble	11:44
rcarrillocruz	i was checking ubuntu-xenial-2-node-rax-ord-4034028 , but i don't see anything wrong with it, again, it may be this one is just fine	11:44
sdague	yeh, let me see if I can find another trouble node that's not ipv6	11:44
rcarrillocruz	++	11:45
sdague	telnet://172.99.124.27:19885	11:45
sdague	I think that one is in the same trouble	11:45
*** baoli has quit IRC		11:46
rcarrillocruz	k	11:46
sdague	I wonder if something is weird with openstackclient 3.2.0	11:47
openstackgerrit	Zara proposed openstack-infra/python-storyboardclient: Add basic docs for python client https://review.openstack.org/362878	11:47
sdague	that spews something that breaks the ssh connection	11:47
*** markusry has joined #openstack-infra		11:48
*** gongysh has joined #openstack-infra		11:49
*** sarob has joined #openstack-infra		11:50
openstackgerrit	Adam Coldrick proposed openstack-infra/storyboard: Send notifications to subscribers for worklists https://review.openstack.org/354730	11:50
openstackgerrit	Adam Coldrick proposed openstack-infra/storyboard: Make it possible to get worklist/board timeline events via the API https://review.openstack.org/354729	11:50
openstackgerrit	Adam Coldrick proposed openstack-infra/storyboard: Don't allow users to subscribe to private worklists they can't see https://review.openstack.org/363776	11:50
openstackgerrit	Adam Coldrick proposed openstack-infra/storyboard: Limit PyMySQL to < 0.7.7 https://review.openstack.org/364250	11:50
Zara	uh oh, another limit	11:51
rcarrillocruz	sdague: i don't see anything suspicious on the subnode	11:52
rcarrillocruz	checked stack logs	11:52
rcarrillocruz	syslog	11:52
rcarrillocruz	and nothing pointing to a problem	11:52
sdague	is openstack complete still running?	11:52
sdague	can you give me ssh to the subnode?	11:53
rcarrillocruz	http://paste.openstack.org/show/565679/	11:53
sdague	so it is still running	11:53
rcarrillocruz	sure, give me your pubkey , you ssh to the node and from there you enter the subnode?	11:53
*** sarob has quit IRC		11:54
*** larainema has joined #openstack-infra		11:55
sdague	rcarrillocruz: http://paste.openstack.org/show/565680/	11:55
rcarrillocruz	sdague: put your key on zuul user, you should be able to sudo su afterwards	11:56
rcarrillocruz	let me know if you can get in	11:56
sdague	ssh zuul@172.99.124.27 ?	11:57
rcarrillocruz	subnode is 10.210.128.136	11:57
sdague	that fails me	11:57
sdague	Permission denied (publickey).	11:57
rcarrillocruz	172.99.124.27	11:57
sdague	yeh, no dice	11:58
*** yaume_ has joined #openstack-infra		11:59
sdague	any thoughts about why my key is no good there?	11:59
*** rfolco has quit IRC		11:59
*** rfolco has joined #openstack-infra		12:00
pabelanger	morning	12:00
rcarrillocruz	try know sdague	12:00
rcarrillocruz	i think it's due to the paste	12:00
sdague	nope	12:00
rcarrillocruz	it wrapped the lines	12:01
sdague	right	12:01
*** yaume has quit IRC		12:01
rcarrillocruz	pabelanger: https://review.openstack.org/#/c/364101/	12:01
rcarrillocruz	infracloud ran tests fine last night	12:01
rcarrillocruz	let's bump it a little further, then we can make the max count for cores/mem quota	12:02
amrith	sdague, I got the public key. it is useless without the private key, would you paste that too please :) PM is fine ...	12:02
rcarrillocruz	(morning!)	12:02
sdague	pabelanger / rcarrillocruz - can we please not change any more cloud things until we figure out why multinode is busted?	12:02
amrith	I would love to do just the same thing, we're having some odd test failures and I'd love to ssh to the host running the tests and look around	12:02
sdague	rcarrillocruz: ok, so still not working, what do you need from me to move forward?	12:03
rcarrillocruz	i'm not sure sdague , i pasted 3 times on the zuul authorized_keys user	12:04
*** dtantsur\|lunch is now known as dtantsur		12:04
rcarrillocruz	still doesn't work for you?	12:04
sdague	it does not	12:04
amrith	rcarrillocruz, for review https://review.openstack.org/364018, and job gate-trove-functional-dsvm-mysql-mitaka which I just triggered, I'd like to ssh into the test instance. would that be possible?	12:04
amrith	the logs are at telnet://158.69.94.249:19885	12:05
*** tpsilva has joined #openstack-infra		12:05
sdague	rcarrillocruz: and a ton of things are recheck grinding that need to land for the release	12:05
*** psilvad has joined #openstack-infra		12:05
sdague	because of the multinode fail spike	12:06
rcarrillocruz	rcarrillocruz: try now?	12:06
rcarrillocruz	i copied line by line	12:06
*** jaosorior has quit IRC		12:06
sdague	nope	12:06
sdague	rcarrillocruz: no, take all the line breaks out of it	12:06
pabelanger	sdague: rcarrillocruz: Ya, I'd like to investigate the launch failures in infracloud-vanilla and internap-mtl01 this morning before we bring more nodes online. There were a few of them over night	12:06
*** jaosorior has joined #openstack-infra		12:06
pabelanger	rcarrillocruz: sdague: Also want to make sure we run at least 10 nodes in both region for a few hours to confirm things are stable too	12:06
sdague	pabelanger: sure, it's just we've got this whole other subnode fail spike	12:07
amrith	rcarrillocruz, if you could do that, here's my public key. https://gist.github.com/amrith/ad6e47f805d7f83bdf630ad84ff0141a	12:07
sdague	amrith: that's typically not done unless it's an infra level fail	12:07
pabelanger	sdague: agreed, we need to confirm jobs are stable too	12:07
amrith	sdague, the issue I'm having is likely that	12:07
amrith	what I have is a guest being spun up	12:07
amrith	which can't seem to talk back to the controller	12:07
amrith	trove guest (vm within the infra machine)	12:08
amrith	and I have no way to debug if it can't get back on the network to the controller (the machine infra spins up)	12:08
amrith	hence the request	12:08
rcarrillocruz	sdague: are you really sshing to 172.99.124.27 ? i don't see ssh attemps on auth.log	12:08
amrith	we've tried to instrument with code to have the guest ship back logs	12:08
amrith	but for all we can tell, it can't seem to get any data back	12:08
sdague	rcarrillocruz: ssh -i ~/.ssh/id_dsa zuul@172.99.124.27	12:08
*** pradk has joined #openstack-infra		12:09
sdague	oh, your ssh doesn't allow that	12:09
sdague	debug1: Skipping ssh-dss key /home/sdague/.ssh/id_dsa - not in PubkeyAcceptedKeyTypes	12:09
rcarrillocruz	:-)	12:10
rcarrillocruz	can you generate a dummy one with rsa real quick	12:10
rcarrillocruz	?	12:10
sdague	ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCkRpGApH9L2PDo+zgwWkuaQ3k5MmPejXu+eWUhvXln5ieAGW7F4dOptBZ+RifJSqFPHD2czwtC7KpNo	12:10
sdague	Rwzm+aU= sdague@ribos	12:10
sdague	?	12:10
sdague	how about that one	12:10
amrith	use mind sdague :)	12:10
amrith	s/mind/mine/	12:10
rcarrillocruz	sdague: mind doing a paste.openstack, the line breaks and i fear i't going to fial for you	12:11
rcarrillocruz	fail too	12:11
*** yamamoto has quit IRC		12:11
*** psilvad has quit IRC		12:11
sdague	http://paste.openstack.org/show/565685/	12:11
rcarrillocruz	try?	12:12
sdague	nope	12:12
* amrith waits		12:12
pabelanger	rcarrillocruz: going to look into this failure from this morning http://paste.openstack.org/show/565684/ Looks like some issues scheduling nodes	12:12
rcarrillocruz	now?	12:12
rcarrillocruz	it's the line wrap i guess	12:12
*** yamamoto has joined #openstack-infra		12:13
sdague	yep	12:13
sdague	ok, I'm in	12:13
rcarrillocruz	\o/	12:13
sdague	ok, how do I get to the subnode?	12:13
rcarrillocruz	the subnode is:	12:13
sdague	also, are these things held, so they won't go away on me?	12:13
rcarrillocruz	10.210.128.136	12:13
rcarrillocruz	yeah, i held	12:13
amrith	rcarrillocruz, may I request ssh access into the machine running the job gate-trove-functional-dsvm-mysql-mitaka in review https://review.openstack.org/364018	12:14
sdague	rcarrillocruz: what key do I use to get there?	12:14
rcarrillocruz	sudo su	12:14
rcarrillocruz	and from there you can ssh in	12:14
rcarrillocruz	amrith: which test	12:15
rcarrillocruz	?	12:15
rcarrillocruz	ah	12:15
rcarrillocruz	i read now	12:15
amrith	no worries, sorry for interrupting you and sdague :)	12:15
amrith	but I'm at my wits end after trying to debug this all night	12:15
amrith	and happened on your conversation with sdague	12:15
amrith	what we're facing is best seen here:	12:16
*** yamamoto has quit IRC		12:16
amrith	https://review.openstack.org/#/q/project:openstack/trove-integration	12:16
amrith	after https://review.openstack.org/355492	12:16
amrith	no job has successfully passed the tests gate-trove-functional-dsvm-mysql-mitaka andgate-trove-functional-dsvm-mysql-liberty	12:17
amrith	I tried to revert the last change, that failed too	12:17
amrith	the issue is that the controller node that is spun up hosts a guest VM for the trove db that gets tested	12:17
sdague	ok, I think it's the auth prompt	12:17
rcarrillocruz	amrith: try as 'zuul'	12:17
amrith	ok, one second	12:17
*** rfolco has quit IRC		12:17
sdague	I think openstackclient is hanging in a password hold	12:18
rcarrillocruz	++	12:18
rcarrillocruz	anything else ?	12:18
rcarrillocruz	i was about to go lunch	12:18
amrith	got it, thanks	12:18
*** rfolco has joined #openstack-infra		12:18
rcarrillocruz	np	12:18
mordred	sdague: it should only prompt for password if it doesn't have a value for PASSWORD ... any chance you can tell how it was executed?	12:19
*** sskripnick has left #openstack-infra		12:20
*** oanson has quit IRC		12:20
*** lucas-hungry is now known as lucasagomes		12:21
sdague	mordred: it's on the subnodes, don't fully know	12:21
mordred	sdague: zomg. it's the tab complete thing	12:21
sdague	https://review.openstack.org/364262	12:21
sdague	mordred: yep, it is	12:21
mordred	stack 1462 20148 0 10:27 pts/0 00:00:00 /usr/bin/python /usr/local/bin/openstack complete	12:21
sdague	oh, I know that bit	12:21
sdague	anyway, I think we need to block that release and sort it out later	12:22
sdague	right now this is causing a huge fail on multinode jobs	12:22
mordred	it's an easy devstack fix	12:22
mordred	if you know where it's being executed	12:22
openstackgerrit	mathieu bultel proposed openstack-infra/tripleo-ci: WIP - Implement overcloud upgrade job - Mitaka -> Newton https://review.openstack.org/323750	12:22
sdague	easy how?	12:22
mordred	openstack --os-cloud=devstack complete	12:22
mordred	will not hang	12:22
mordred	I'm sure the _real_ fix in osc will take more work	12:22
sdague	ok, so what you are saying is everyone using osc anywhere now has to update their scripts using it otherwise their stuff hangs indefinitely?	12:23
mordred	no	12:23
mordred	I'm saying that anyone anywhere who is running openstack complete, just that command, without auth	12:23
mordred	will have to update their scripts	12:23
mordred	the number of people who are doin gthat are likely to be done	12:23
mordred	none	12:23
sdague	that's really the only thing that breaks here?	12:23
mordred	it's a hack - but it'll fix the gate without needing to land an exclusion pin	12:24
mordred	yah	12:24
mordred	all the _other_ openstack commands already specify cloud becaues they're trying to interact with a cloud	12:24
mordred	so they'll hav ea password in clouds.yaml	12:24
sdague	mordred: ok, but this means that --help is going to hang for people the same way, right?	12:24
mordred	lemme try	12:24
mordred	nope	12:25
mordred	help works	12:25
*** yolanda has joined #openstack-infra		12:25
sdague	you sure, did you reset your env?	12:25
sdague	ok, help works	12:25
mordred	sdague: https://review.openstack.org/364266	12:25
sdague	openstack ... with no args, prompts for password before you get to the REPL	12:25
mordred	yah. I would expect it to if there is no OS_PASSWORD env var set	12:26
*** esberglu has joined #openstack-infra		12:26
mordred	this, btw, is amongst the reasons why I have consistently argued that the password prompt is a bad ui choice	12:26
sdague	mordred: ok, I'll take the devstack work around if you register a critical bug somewhere to fix it for real	12:26
mordred	and should not exist	12:26
mordred	yah	12:26
sdague	mordred: ok, what was the change that made this?	12:26
*** yamamoto has joined #openstack-infra		12:26
sdague	mordred: and I 100% agree with you that prompts for password are terrible	12:26
mordred	there was an osc change recently that changed how password prompt on missing password workded	12:27
mordred	we added a hook point in os-client-config for it, and osc started using that hook point	12:27
openstackgerrit	Paul Belanger proposed openstack-infra/puppet-user: Add user to adm group too https://review.openstack.org/364268	12:27
mordred	I'm guessing osc just missed a use case	12:27
sdague	ok, show me the bug, I want to update the code to reference it	12:28
mordred	I have not yet filed one	12:28
mordred	that's next on my list	12:28
sdague	ok, please do that now :)	12:28
*** kgiusti has joined #openstack-infra		12:28
mordred	sdague: how is this not breaking normal devstack runs?	12:28
sdague	don't know, it only seems to trigger on the subnode	12:29
sdague	and only some of the time	12:29
mordred	that makes no sense to me at all	12:29
sdague	me either	12:29
*** javeriak has joined #openstack-infra		12:29
*** raildo has joined #openstack-infra		12:30
sdague	well, on the subnode might make sense if we don't give the subnode the right osc vars	12:31
sdague	because we don't actually run any other osc commands there	12:31
*** mdrabe has joined #openstack-infra		12:31
mordred	sdague: https://bugs.launchpad.net/python-openstackclient/+bug/1619274	12:31
openstack	Launchpad bug 1619274 in python-openstackclient "openstack complete triggers password prompt" [Undecided,New]	12:31
sdague	so we just never setup localrc so that any real commands would work	12:31
mordred	sdague: ah - that does make sense then	12:31
sdague	mordred: your fix assumes a clouds.yaml exists, right?	12:33
sdague	oh, I guess this is late enough, it should	12:34
*** cody-somerville has joined #openstack-infra		12:34
pabelanger	rcarrillocruz: mordred: looks like one of our compute nodes in infracloud-vanilla is having some ConnectTimeout issues: http://paste.openstack.org/show/565687/	12:34
*** zhurong has joined #openstack-infra		12:34
rcarrillocruz	Thx, I will look in a bit	12:35
mordred	sdague: yah. I mean, I tested the fix on the subnode	12:35
pabelanger	rcarrillocruz: mordred: I guess we don't have unbound setup? nameserver 8.8.8.8	12:35
*** Hal2 has quit IRC		12:36
*** gongysh has quit IRC		12:36
*** Genek has quit IRC		12:36
pabelanger	ya, looks like we are hard coding DNS to google in our network interface config	12:37
sdague	mordred / rcarrillocruz / pabelanger could we get a gate enqueue and promote on https://review.openstack.org/#/c/364266 ?	12:37
*** Hal1 has joined #openstack-infra		12:37
mordred	on it	12:37
mordred	sdague: done	12:39
*** cardeois has joined #openstack-infra		12:41
*** Hal1 has quit IRC		12:42
*** Hal1 has joined #openstack-infra		12:44
*** cody-somerville has quit IRC		12:45
amrith	rcarrillocruz, in what linux user name will devstack get run on the test instance?	12:45
amrith	is it zuul or jenkins?	12:45
openstackgerrit	Paul Belanger proposed openstack-infra/system-config: Install unbound on infracloud servers https://review.openstack.org/364282	12:49
pabelanger	mordred: rcarrillocruz: ^ add unbound to infracloud controller and compute nodes	12:49
*** Hal1 has quit IRC		12:49
*** Hal1 has joined #openstack-infra		12:50
mordred	yolanda: ^^ you're using infracloud puppet not for infra ... will that break you?	12:50
*** rlandy has joined #openstack-infra		12:50
mordred	(I think it won't - but just want to check)	12:50
*** markvoelker has joined #openstack-infra		12:50
*** nwkarsten has joined #openstack-infra		12:50
*** links has quit IRC		12:51
*** gouthamr has joined #openstack-infra		12:51
*** Goneri has joined #openstack-infra		12:54
*** abregman_ has quit IRC		12:54
rcarrillocruz	back	12:56
* rcarrillocruz reads backscroll		12:56
openstackgerrit	Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Add non-voting multi-region functional test to OSA os_swift https://review.openstack.org/364286	12:56
rcarrillocruz	oh nice	12:56
rcarrillocruz	sdague: so the strace stuck on the read syscall was due to a pass prompt?	12:57
sdague	rcarrillocruz: yes	12:57
sdague	osc goes into a blocking prompt with no timeout	12:57
rcarrillocruz	eugh	12:57
*** yamamoto has quit IRC		12:57
rcarrillocruz	pabelanger: cool, i was just aware about unbound on baremetal00	12:58
rcarrillocruz	approved the change	12:58
rcarrillocruz	amrith: let me check , not sure if it was changed from jenkins to zuul	12:58
rcarrillocruz	a job log may show	12:58
amrith	rcarrillocruz,	12:59
rcarrillocruz	amrith: the tests puts the stuff on /home/jenkins/workspace, so I assume it's jenkins	12:59
amrith	I found it	12:59
amrith	it is jenkins	12:59
rcarrillocruz	++	12:59
amrith	but I couldn't (initially) get into the trove guest	12:59
*** spzala has joined #openstack-infra		12:59
amrith	because it was using a key I dind't expect	12:59
amrith	looks like cloud-init is pooched	12:59
amrith	I can see what is failing	12:59
mordred	woot!	12:59
amrith	have to figure out why	12:59
amrith	I've seen this before	12:59
amrith	once	13:00
amrith	and entered a bug on cloud-init	13:00
pabelanger	rcarrillocruz: ya, I didn't change baremetal00. But good to know it is running unbound	13:00
amrith	and I was given a promise that this problem wouldn't hit u14.04	13:00
mordred	you could always shift to glean instead of cloud-init ...	13:00
amrith	only 15.04	13:00
amrith	and 16.04	13:00
amrith	but this is now on 14.04	13:00
rcarrillocruz	stack iirc	13:00
rcarrillocruz	sdague: ^ ?	13:00
rcarrillocruz	oh sorry, i was scrolled back, responded to the linux username question on devstack	13:00
amrith	https://bugs.launchpad.net/bugs/1602373	13:01
openstack	Launchpad bug 1602373 in cloud-init (Ubuntu Xenial) "cloud-init doesn't always land files that one expects" [High,Fix released]	13:01
rcarrillocruz	pabelanger: so the intent for putting unbound is to hit the dns cache to avoid http://paste.openstack.org/show/565687/ , or you pasted it to me for another root cause problem?	13:01
*** drawsmcgraw has left #openstack-infra		13:01
amrith	hi monty	13:02
*** sshnaidm is now known as sshnaidm\|mtg		13:02
amrith	we're using (at the whim of) whatever nova uses under the covers, we just ask to inject a file. how it gets there (USPS, Fedex, UPS, ...) we don't know	13:02
mordred	right	13:02
mordred	I suggest not using file injection at all	13:02
*** andreykurilin has left #openstack-infra		13:02
*** andreykurilin has joined #openstack-infra		13:03
mordred	just put an ssh key on the guest and use ssh to put files where you need them - ansible is a good tool for doing that if an ssh key is there, but obviously you can also script that in a ton of different ways	13:03
*** mbacchi_ is now known as mbacchi		13:03
mordred	I mean, that's not a thing you're going to change today obviously	13:04
pabelanger	rcarrillocruz: Ya, that was the reason we failed to launch a node with nodepool	13:04
rcarrillocruz	k	13:04
mordred	but one of the things we've learned here in infra is to keep the amount of stuff that happens in the nova-metadata to cloud-init/glean interaction to a bare minimum	13:04
amrith	mordred, the issue is configurations that are injected at runtime	13:04
mordred	yah	13:04
amrith	yes, we could get those too	13:04
amrith	in a bootstrapping manner	13:04
mordred	I'm not saying don't inject them	13:04
sdague	mordred: your patch failed on a subnode issue that is unlogged	13:04
mordred	just don't use nova metadata to do so	13:04
rcarrillocruz	i'm glad is just that and not some net issues on HPE side :D	13:04
mordred	sdague: YAY!	13:04
sdague	http://logs.openstack.org/66/364266/2/gate/gate-grenade-dsvm-neutron-multinode/63498bd/	13:04
amrith	the issue here is that the injection isn't working	13:04
amrith	I think I've figured out why	13:05
amrith	bbiab	13:05
pabelanger	rcarrillocruz: Figured, we should setup unbound first and see if we can reproduce, since google will throttle DNS requests	13:05
sdague	there are no subnode logs there	13:05
*** binbincong has quit IRC		13:05
AJaeger	mordred: yolanda is on FTO	13:05
*** jamesden_ has joined #openstack-infra		13:05
rcarrillocruz	yup	13:05
mordred	sdague: I believe my complete answer is "awlkjevcalskencalserifnasldkcasdf"	13:05
sdague	mordred: it's a super weird fail	13:06
mordred	sdague: although please imagine me banging my head against something whilst saying that	13:06
sdague	can you re promote	13:06
*** skraynev_ has joined #openstack-infra		13:06
mordred	yah. one sec	13:06
sdague	this never even got to your code, I have no idea why zuul just shot the node in the head	13:06
*** vikrant has quit IRC		13:06
sdague	http://logs.openstack.org/66/364266/2/gate/gate-grenade-dsvm-neutron-multinode/63498bd/console.html	13:06
*** drawsmcgraw has joined #openstack-infra		13:06
sdague	it starts at 12:41	13:07
pabelanger	mordred: jeblair_ : rcarrillocruz: So we need to remove HPE from spamhaus: http://paste.openstack.org/show/565695/	13:07
*** salv-orlando has joined #openstack-infra		13:07
*** markvoelker has quit IRC		13:07
sdague	170 minute timeout set at 12:42	13:07
pabelanger	cause I am getting flooded with Mail Delivery System emails	13:07
sdague	grenade starts at 12:50	13:07
sdague	at 12:54 zuul says - job failed	13:07
rcarrillocruz	pabelanger: i have to admit i have no idea how to do that, we fill a form or something?	13:07
pabelanger	rcarrillocruz: Ya, I think we just request it from the website	13:08
pabelanger	but I haven't done it before	13:08
sdague	and the last thing grenade is doing is downloading a pip	13:08
sdague	http://logs.openstack.org/66/364266/2/gate/gate-grenade-dsvm-neutron-multinode/63498bd/console.html	13:08
sdague	2016-09-01 12:54:54.323 \| Downloading http://mirror.regionone.bluebox-sjc1.openstack.org/wheel/ubuntu-14.04-x86_64/xattr/xattr-0.8.0-cp27-none-linux_x86_64.whl	13:08
*** Julien-zte has joined #openstack-infra		13:08
mordred	wow: 15.184.0.0/17 is listed on the Policy Block List (PBL)	13:09
rcarrillocruz	yup	13:09
rcarrillocruz	https://www.spamhaus.org/query/ip/15.184.64.7	13:09
* rcarrillocruz clicks the button "begin the removal process'		13:10
mordred	oh - because those servers are in an HPE address range but we're not sending email through the HPE mail servers	13:10
*** abregman_ has joined #openstack-infra		13:10
mordred	https://www.spamhaus.org/pbl/query/PBL1525955	13:10
mordred	rcarrillocruz: ++	13:11
sdague	2016-09-01 12:54:50,502 p=21911 u=zuul \| fatal: [node]: FAILED! => {"failed": true, "msg": "Failed to connect to the host via ssh."}	13:11
*** caowei has joined #openstack-infra		13:11
amrith	rcarrillocruz, thx much. I'm done with my test instance for now. I'm going to figure out how to fix it and see what I can do. many many thx (and a great coincidence happening upon your conversations with sdague)	13:12
rcarrillocruz	yw	13:12
mordred	sdague: we're having all the luck this morning	13:12
rcarrillocruz	hmm, the form doesn't seem to allow a CIDR	13:12
*** salv-orlando has quit IRC		13:13
*** abregman_ is now known as abregman\|mtg		13:13
sdague	mordred: yeh, unfortunately I don't even know why zuul was doing that	13:13
sdague	it doesn't make any sense	13:13
sdague	it should have been heads down waiting for grenade run to return	13:14
*** akshai has joined #openstack-infra		13:14
sdague	which was chugging along	13:14
sdague	but then it made another ssh call in parallel, which died	13:14
sdague	and it blew up the job	13:14
AJaeger	sdague: could you review later a change to project-config: https://review.openstack.org/#/c/364162 - I tend to -1 but would like your review	13:14
*** amotoki_ has quit IRC		13:14
mordred	sdague: that's how async jobs in ansible work	13:15
mordred	sdague: it spawns the job ,and then periodically checks on it	13:15
mordred	to make sure it hasn't blown up	13:15
sdague	mordred: ok... how often does it false negative there?	13:15
*** matt-borland has joined #openstack-infra		13:15
sdague	because from the logs, there was nothing wrong	13:15
mordred	not a ton - there were some problems like a month ago but those got sorted	13:15
mordred	we're just extra lucky this morning	13:16
sdague	but it was downloading packages, so it might have saturated the link	13:16
sdague	what's the timeout on that check?	13:16
*** gongysh has joined #openstack-infra		13:16
sdague	and is it tunable?	13:16
pabelanger	yes	13:17
pabelanger	I believe we default to 10	13:17
rcarrillocruz	pabelanger: i'm not a mail master, but would it work if we relayed the mail from those machines to a HPE mail server	13:17
rcarrillocruz	?	13:17
rcarrillocruz	cos i'm reading spamhaus	13:17
sdague	pabelanger: could we up that to 30 or 60?	13:17
rcarrillocruz	and it seems to remove an entire CDR block we have to have an account or something on spamhaus	13:17
*** moravec has quit IRC		13:17
rcarrillocruz	otherwise is ip by ip, one by one	13:17
rcarrillocruz	:S	13:17
sdague	I can definitely imagine a 10 second timeout might wedge if we're going full bore on the network	13:18
pabelanger	rcarrillocruz: that or we setup our own relay	13:18
rcarrillocruz	i vote the latter	13:18
rcarrillocruz	putting it on the bifrost machinee or something	13:18
*** rossella_s has quit IRC		13:18
rcarrillocruz	is not going to take much cpu	13:18
*** cardeois_ has joined #openstack-infra		13:18
openstackgerrit	Merged openstack-infra/shade: Allow str for ip_version param in create_subnet https://review.openstack.org/363846	13:18
pabelanger	sdague: I'm sure we could bump it, I cannot think of a reason not to	13:19
*** rossella_s has joined #openstack-infra		13:19
sdague	pabelanger: can you point me to where it is specified?	13:19
sdague	I'll submit a patch	13:19
*** pgadiya has quit IRC		13:19
*** larsks has left #openstack-infra		13:19
mordred	sdague: line 1106 in zuul/launcher/ansiblelaunchserver.py	13:20
*** yamahata has joined #openstack-infra		13:20
pabelanger	http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/launcher/ansiblelaunchserver.py#n1259	13:20
*** moravec has joined #openstack-infra		13:20
pabelanger	http://docs.ansible.com/ansible/intro_configuration.html#timeout	13:20
pabelanger	is the setting	13:20
mordred	pabelanger: ah - we are talking about different timeouts	13:21
pabelanger	Oh	13:21
mordred	sdague: listen to pabelanger - he's righter than me	13:21
sdague	well, is it that, or the watchdog?	13:21
pabelanger	actually	13:21
*** dizquierdo has joined #openstack-infra		13:22
pabelanger	I think we need to update ssh_args	13:22
*** cardeois has quit IRC		13:22
sdague	https://github.com/openstack-infra/zuul/blob/80fe50f484a2a2617c6645442336d458aeed59b5/zuul/launcher/ansiblelaunchserver.py#L95 ?	13:22
pabelanger	give me a second	13:22
sdague	pabelanger: ok	13:22
rcarrillocruz	sdague: are you done with the held node from earlier	13:22
rcarrillocruz	?	13:22
sdague	rcarrillocruz: yes	13:22
openstackgerrit	Merged openstack-infra/project-config: Add deb-bandit and deb-python-hacking repo https://review.openstack.org/362096	13:23
*** markvoelker has joined #openstack-infra		13:23
mordred	sdague: btw - next week I plan on hacking in this area of zuul	13:23
sdague	mordred: ok, cool	13:23
pabelanger	mordred: sdague: Ya, I think http://docs.ansible.com/ansible/intro_configuration.html#timeout is the right setting that we need to change	13:24
sdague	pabelanger: ok, where would that be set?	13:24
pabelanger	http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/launcher/ansiblelaunchserver.py#n1259	13:24
pabelanger	that is our ansible.cfg per test run	13:24
mordred	pabelanger: I think we should also investigate why controlpersist isn't in effect here	13:24
pabelanger	mordred: maybe we need to bump the value? we only default to 60	13:26
*** berendt has quit IRC		13:26
*** asettle is now known as asettle-afk		13:26
*** yamamoto has joined #openstack-infra		13:26
pabelanger	mordred: we also poll pretty aggressively, every 5 seconds. Maybe we should bump that to 60 seconds	13:27
*** yamamoto has quit IRC		13:27
*** markvoelker has quit IRC		13:28
zigo	mordred: pabelanger: fungi: Hi, I'm back. Sorry that I was absent since Friday, but unfortunately, my father died, and I went to see familly and such. Now I'm back at work, even though I not really in such a mood... Anyway, did you guys have time to discuss about giving us merge ACLs within packaging-deb?	13:28
openstackgerrit	Sean Dague proposed openstack-infra/zuul: bump timeout on ssh commands to 30s https://review.openstack.org/364309	13:28
*** eharney has joined #openstack-infra		13:29
pabelanger	mordred: or some percentage of the value of timeout	13:30
pabelanger	zigo: let me find the log	13:30
*** yamamoto has joined #openstack-infra		13:30
*** amotoki has joined #openstack-infra		13:31
pabelanger	zigo: http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2016-08-30.log.html#t2016-08-30T16:07:42	13:31
zigo	pabelanger: Cheers.	13:31
pabelanger	zigo: that was the discussion on the subject.	13:31
*** yamamoto has quit IRC		13:32
*** yamamoto has joined #openstack-infra		13:33
*** roxanaghe has joined #openstack-infra		13:33
*** yamamoto has quit IRC		13:34
openstackgerrit	Merged openstack-infra/system-config: Install unbound on infracloud servers https://review.openstack.org/364282	13:34
*** szaher_ has quit IRC		13:34
AJaeger	zigo: please check https://review.openstack.org/356795 - I'd like to see that repo fixed.	13:34
*** yamamoto has joined #openstack-infra		13:34
*** yamamoto has quit IRC		13:34
*** szaher has joined #openstack-infra		13:34
pkoniszewski	AJaeger: thanks for pointing this out, I wasn't aware of it	13:37
*** roxanaghe has quit IRC		13:37
pkoniszewski	AJaeger: so what's the best way to work on such gate? propose new job to experimental queue?	13:38
AJaeger	pkoniszewski: yes, an experimental job is the only way to do this...	13:39
sileht	sdague, hi, do you have sometimes to discus about this https://review.openstack.org/#/c/335479/ ? it seems you perhaps have a better solution.	13:40
AJaeger	pkoniszewski: next time, it would also help if you explain in the commit message what you want to test ;)	13:41
pkoniszewski	AJaeger: okay, I will, and I need to start work on experimental job then, thanks a lot! :)	13:42
*** rajinir has joined #openstack-infra		13:43
*** lock__ is now known as ianychoi		13:43
*** amotoki has quit IRC		13:43
*** gordc has joined #openstack-infra		13:44
*** sshnaidm\|mtg is now known as sshnaidm		13:44
sdague	sileht: I definitely don't have the brain bw to handle that atm	13:45
openstackgerrit	Peter Zhurba proposed openstack-infra/project-config: Add repo for openstack/puppet-glare. https://review.openstack.org/362950	13:45
*** mriedem has joined #openstack-infra		13:46
sileht	sdague, ok no problem, can I re-ask tomorrow ? or do you prefer to come back to me when you have the time ?	13:46
sdague	sileht: this has to be post freeze once the release has quieted down	13:47
*** asettle-afk has quit IRC		13:47
sdague	honestly, this probably is better to be done in a design summit session	13:47
sdague	because it sounds like there is a broader range of testing that's desired	13:47
*** asettle-afk has joined #openstack-infra		13:48
openstackgerrit	Paul Belanger proposed openstack-infra/nodepool: Retry ssh connect when EOFError is received https://review.openstack.org/364322	13:48
openstackgerrit	Sagi Shnaidman proposed openstack-infra/tripleo-ci: TEST: DONT RECHECK: periodic jobs https://review.openstack.org/359215	13:48
sileht	sdague, its for projects that have intermediate release and want to test upgrade with previous synced release	13:49
*** openstackgerrit has quit IRC		13:49
pabelanger	jeblair_: please let me know what you think about 364322^, we are seeing random EOFError in nodepool debug logs and think we should retry our SSH connection over aborting.	13:49
*** openstackgerrit has joined #openstack-infra		13:49
*** amotoki has joined #openstack-infra		13:50
*** sarob has joined #openstack-infra		13:51
*** spiette has joined #openstack-infra		13:52
*** asettle-afk has quit IRC		13:52
*** spiette has left #openstack-infra		13:52
*** drifterza has quit IRC		13:54
sdague	sileht: right, so here is the thing, it's a bunch of new complexity, for a small number of projects. So I'm honestly not all that keen on the approach.	13:54
*** Guest77956 is now known as jlvillal		13:55
*** jamesden_ has quit IRC		13:55
rcarrillocruz	pabelanger: i'm going to add a param to 'images' on the cloud launcher, not tied to the os_image module for pulling images from the internet. Meaning, if we have an image with a field "download_file_url", the launcher pulls it to /tmp and uploads it to glance	13:55
*** rodrigods has quit IRC		13:55
*** rodrigods has joined #openstack-infra		13:55
*** sarob has quit IRC		13:55
rcarrillocruz	even if that's not a glance v2 feature, that's super useful to have, we could automate the initial images for our cloud providers to bring up the mirrors, etc	13:56
rcarrillocruz	without having to do it manually	13:56
*** jamesden_ has joined #openstack-infra		13:56
mordred	rcarrillocruz: ++	13:56
mordred	rcarrillocruz: I think that's a GREAT feature	13:56
*** xyang1 has joined #openstack-infra		13:56
rcarrillocruz	i'll show you later mordred , spinning my dsvm local instance (enough infracloud for me , i wanna code!)	13:57
*** nwkarsten has quit IRC		13:57
*** dtroyer has joined #openstack-infra		13:57
*** inc0 has joined #openstack-infra		13:57
*** eharney has quit IRC		13:58
*** salv-orlando has joined #openstack-infra		13:59
*** amotoki has quit IRC		13:59
*** eharney has joined #openstack-infra		14:00
*** zz_dimtruck is now known as dimtruck		14:00
*** yamamoto has joined #openstack-infra		14:00
*** ddieterly has joined #openstack-infra		14:01
*** jheroux has joined #openstack-infra		14:02
*** oanson has joined #openstack-infra		14:03
*** hongbin has joined #openstack-infra		14:03
*** cardeois has joined #openstack-infra		14:03
*** asettle-afk has joined #openstack-infra		14:03
*** jaosorior has quit IRC		14:04
*** asettle-afk is now known as asettle		14:04
YorikSar	ianw, rcarrillocruz, AJaeger: Thanks a lot for your swift review.	14:05
rcarrillocruz	np	14:05
*** cardeois_ has quit IRC		14:05
*** kushal has joined #openstack-infra		14:06
*** matthewbodkin has quit IRC		14:06
*** fguillot has joined #openstack-infra		14:06
*** rbrndt has joined #openstack-infra		14:06
*** matthewbodkin has joined #openstack-infra		14:06
pabelanger	ack	14:06
*** pilgrimstack has quit IRC		14:07
*** xarses has quit IRC		14:08
*** javeriak has quit IRC		14:08
rcarrillocruz	doh, spacex falcon 9 has exploded :/	14:09
openstackgerrit	Sagi Shnaidman proposed openstack-infra/tripleo-ci: Use proxy on OVB environemnt https://review.openstack.org/364335	14:09
*** fguillot has quit IRC		14:10
*** claudiub has quit IRC		14:10
*** sandanar has joined #openstack-infra		14:11
pabelanger	rcarrillocruz: unbound live on infracloud	14:11
pabelanger	DNS working too	14:11
rcarrillocruz	awesome!	14:12
pabelanger	Hmm	14:13
pabelanger	SSHException: No existing session	14:13
pabelanger	paramiko raises that from time to time	14:13
*** Genek has joined #openstack-infra		14:14
*** skraynev_ has quit IRC		14:14
*** amotoki has joined #openstack-infra		14:14
*** pilgrimstack has joined #openstack-infra		14:15
*** tongli has joined #openstack-infra		14:15
*** itisha has joined #openstack-infra		14:15
*** pt_15 has joined #openstack-infra		14:17
openstackgerrit	Rob Cresswell proposed openstack-infra/project-config: Add Ironic UI gerritbot to #openstack-ironic https://review.openstack.org/364347	14:17
openstackgerrit	Derek Higgins proposed openstack-infra/tripleo-ci: Nothing to see here https://review.openstack.org/111011	14:18
sdague	oh neat when did the web console hit zuul?	14:18
sdague	oh, it didn't, just something changed with the way chrome launched it here	14:18
openstackgerrit	Merged openstack-infra/tripleo-ci: Sync the config of the ha JOBTYPE with ha2 https://review.openstack.org/362974	14:20
*** sputnik13 has joined #openstack-infra		14:20
*** coolsvap has quit IRC		14:22
*** skraynev_ has joined #openstack-infra		14:22
*** tqtran has joined #openstack-infra		14:22
*** hsammic has joined #openstack-infra		14:22
openstackgerrit	Paul Belanger proposed openstack-infra/project-config: Revert "Revert "Increase rh1 max-servers to 75"" https://review.openstack.org/364351	14:23
*** mtanino has joined #openstack-infra		14:24
pabelanger	mordred: clarkb: ^ Now that nodepool was restarted and shade upgraded, I think we can increase the nodes in tripleo-test-cloud-rh1 again.	14:24
pabelanger	FIP issues appears to be minimal right no	14:24
pabelanger	now*	14:24
*** mikelk has quit IRC		14:24
*** watanabe_isao has quit IRC		14:25
pabelanger	actually	14:25
pabelanger	mordred: this seems new: http://paste.openstack.org/show/565719/	14:25
*** tqtran has quit IRC		14:26
*** fguillot has joined #openstack-infra		14:27
*** bogdando has quit IRC		14:27
pabelanger	http://paste.openstack.org/show/565720/ is the debug from shade	14:28
*** vhosakot has joined #openstack-infra		14:28
*** skraynev_ has quit IRC		14:31
*** signed8bit has joined #openstack-infra		14:32
*** jamesden_ has quit IRC		14:32
sdague	mordred: patch finally merged	14:33
*** gongysh has quit IRC		14:33
*** dimtruck is now known as zz_dimtruck		14:33
*** zz_dimtruck is now known as dimtruck		14:33
*** ddieterly is now known as ddieterly[away]		14:34
*** ivyyy has quit IRC		14:35
mordred	sdague: woot	14:36
*** eharney has quit IRC		14:36
mordred	pabelanger: looking	14:36
*** jheroux has quit IRC		14:37
openstackgerrit	Sagi Shnaidman proposed openstack-infra/tripleo-ci: TEST: DONT RECHECK: periodic jobs https://review.openstack.org/359215	14:39
*** abregman\|mtg has quit IRC		14:41
clarkb	mordred: pabelanger the last time we had ssh issues like that control persist was fine we just broke ipv6. Maybe we are affecting the test instamce with some job side effect?	14:41
*** oanson has quit IRC		14:41
mordred	clarkb: nod	14:41
mordred	pabelanger: I'm seeing nodes be created there - let's consider that one an outlier for the moment	14:42
*** xarses has joined #openstack-infra		14:42
*** jheroux has joined #openstack-infra		14:42
mordred	also - we _definitely_ need to add floating ip batching	14:42
mordred	we're slamming the hell out of the neutron api there	14:42
*** caowei has quit IRC		14:42
*** dimtruck is now known as zz_dimtruck		14:43
pabelanger	clarkb: Oh, didn't know that. Have a review handy that shows the fix?	14:45
jpich	Is there a story behind the Dell StorageCenter CI leaving a -1 on every patch in the Gerrit sandbox? Getting a 3rd party build failure notification before you even get the "Welcome, new contributor!" message is a bit rough :-) - https://review.openstack.org/#/q/project:openstack-dev/sandbox+status:open	14:45
clarkb	pabelanger: this was the accept ra thing in devstack	14:45
pabelanger	clarkb: okay	14:45
pabelanger	clarkb: I don't know the full back story on the failures, just that sdague wanted to increase the SSH timeout value	14:46
*** kien-ha has joined #openstack-infra		14:46
zigo	pabelanger: The packaging repo needs to hold a copy of the code from upstream for multiple reasons. Do I need to explain?	14:47
*** ggnel_t has joined #openstack-infra		14:47
clarkb	pabelanger: right I was just suggesting the last time we had problems with that fiddling with timeout values would have no effect. And that we shouldnt ignore the possibility of another break neyworking scenario in the jobs	14:47
*** tonytan4ever has joined #openstack-infra		14:47
pabelanger	clarkb: agreed, good to know	14:48
pabelanger	zigo: you might want to sync up with clarkb and fungi. I didn't really give input on that topic.	14:48
zigo	ok	14:49
zigo	clarkb: fungi: Do you have time to discuss this?	14:49
pabelanger	however, I am interested in the discussions	14:49
*** jcoufal has joined #openstack-infra		14:50
clarkb	zigo I am mostly here now. The outcome seemed to be we could just ensure $repo is present to the right version when needed	14:50
clarkb	rather than force pushing into the deb repo	14:50
zigo	clarkb: I don't agree with this outcome, that's the point! :)	14:51
clarkb	ok	14:51
zigo	clarkb: First off, the repo for upstream get EOL very early.	14:51
zigo	I don't want this to happen in the packaging trees.	14:51
zigo	We need to cover the life of Debian Stable for Newton for example.	14:51
clarkb	we dont delete repos	14:52
zigo	But branches are EOLed.	14:52
clarkb	the tags dont	14:52
clarkb	nor should any sha1 go away	14:52
zigo	Anyway, the git-buildpackage workflow IS to import upstream tag/branch/tarballs into the packaging branch.	14:52
*** kien-ha has quit IRC		14:53
mordred	yes, I concur with that	14:53
zigo	There are ways to make it not do that, but then it's going to be over complicated for no reasons.	14:53
zigo	For example, managing quilt patches will be horrible.	14:53
sdague	clarkb: this is not breaking ipv6	14:53
*** Thelo_ has quit IRC		14:54
*** abregman\|mtg has joined #openstack-infra		14:54
sdague	http://logs.openstack.org/66/364266/2/gate/gate-grenade-dsvm-neutron-multinode/63498bd/console.html	14:54
*** vtapia has quit IRC		14:54
*** Sukhdev has joined #openstack-infra		14:54
fungi	zigo: back now. i understand your _manual_ workflow involves merging a copy of nova commits into the pkg-nova repo, and your automated jobs can do precisely the same thing. that doesn't mean that the pkg-nova repo in _gerrit_ needs to carry a complete copy of nova commits	14:54
sdague	clarkb: the last bit in the subnode log is downloading pip packages	14:54
mordred	fungi: I disagree a bit	14:54
mordred	fungi: because the gate isn't the only place where the packaging repo might be used to create packages	14:54
mordred	so it should not assume the encompasing environment that is our gate	14:55
sdague	in the middle of that download, zuul ansible shoots it in the head	14:55
clarkb	if this is a reqyirement you should be a branch on the nova repo	14:55
zigo	Right. Before an upload to Debian, I'll just git clone and rebuild ...	14:55
fungi	mordred: sure. and the manual process (add a remote for the nova repo, pull from it into an upstream branch, merge) is still a valid workflow in that case, right?	14:55
*** sshnaidm has quit IRC		14:55
*** mhickey has quit IRC		14:55
mordred	fungi: that's a severe degredation in funcationality over normal git-buldpackage workflow	14:55
clarkb	sdague: I didnt say it broke ipv6 just consider the nodes can break themselves	14:55
sdague	my theory is that 10s may not be enough time if the node is flat out pulling packages	14:55
*** kgiusti has left #openstack-infra		14:56
*** ddieterly[away] is now known as ddieterly		14:56
sdague	clarkb: sure	14:56
mordred	fungi: like, gbp exists and is used heavily - I don't think we should break it	14:56
mordred	HOWEVER	14:56
openstackgerrit	Sergey Skripnick proposed openstack-infra/project-config: Add python-db-jobs to Glare https://review.openstack.org/364370	14:56
*** yfried has quit IRC		14:56
mordred	zigo: I'd like to make a suggestion that might be a compromise	14:56
sdague	however, where this is, I don't think that's what happened	14:56
fungi	i'm just concerned that now not only do we have hundreds of separate repos for all packaged software, we have additional copies of all commits for all packaged software	14:56
sdague	clarkb: if you have another thought from those logs, please go and look	14:56
mordred	fungi: right. I don't think we need hundreds of commits in the upstream branch	14:56
mordred	"zigo \| Anyway, the git-buildpackage workflow IS to import upstream tag/branch/tarballs into the packaging branch."	14:57
mordred	tag/branch is one of the options	14:57
mordred	the other option is upstream tarballs, which get imported in a single commit and then tagged	14:57
mordred	that is in line with how gerrit works	14:57
*** krtaylor has joined #openstack-infra		14:57
mordred	and sohuld not require tons of infra-side automation ... unfortunately, it would be a non-trivial rework of the existing packaging repos I fear	14:58
zigo	mordred: Even if you use tarballs, it imports the code in the packaging branch.	14:58
*** kdas_ has joined #openstack-infra		14:58
clarkb	there is somethi g a little funny about hating vendored code	14:58
mordred	zigo: it does	14:58
clarkb	then requiring it	14:58
mordred	zigo: but it's not 100 commits - it's one commit	14:58
mordred	zigo: whic hmeans you could submit that commit to gerrit using git review like normal	14:58
mordred	then land it	14:58
fungi	also we've come full circle from "you should avoid putting the debian package metadata in your upstream source tree" to "you should put a copy of the upstream source tree in the repo where you have your debian packaging metadata"	14:58
*** edtubill has joined #openstack-infra		14:58
*** zhurong has quit IRC		14:58
mordred	fungi: it's not the same thing at all	14:58
zigo	mordred: The point is also to be able to use any commit upstream (and not just a branch or a tag) and package that.	14:59
mordred	putting debian package metadata in upstream is problematic because of sequencing issues	14:59
zigo	Could you explain what the problem is? Do we lack resources?	14:59
mordred	if you want to release version 1.2 of an upstream software, what state should the packaging be in in upstream tag 1.2	14:59
zigo	Is it just too big?	14:59
fungi	i understand the reasons, i'm asking that we separate _process_ (steps to build a package) from revision control (data we store for building a package)	14:59
mordred	it necessarily cannot be the packaging that knows how to handle 1.2, because 1.2 doesn't exist until 1.2 exists	14:59
*** _nadya_ has quit IRC		15:00
zigo	fungi: If we get an ACL to merge tags, it is the same process, it goes under the Gerrit code review.	15:00
zigo	I tried, the only issue is the missing ACL.	15:00
zigo	(at least, it looks like it)	15:00
*** vtapia has joined #openstack-infra		15:00
*** sshnaidm has joined #openstack-infra		15:00
fungi	all the data we need to build a debian package of nova exists in a combination of the nova repo and the pkg-deb repo. the steps to build the package can include combining those. the combined result doesn't have to get pushed back into the revision control history though, does it?	15:00
zigo	fungi: If we don't, then it becomes a way harder to do many things.	15:01
*** kushal has quit IRC		15:01
zigo	Like adding a quilt patch, or building before uploading to Debian.	15:01
fungi	i don't understand. what prevents you from adding a git remote for the nova repo, pulling from it into an unstream branch and then doing a git merge?	15:02
mordred	I feel like we're trying ot fight the current established best practice of debian packaging (use gbp with upstream sources included in the packaging repo) in the context of openstack infra, which seems like a very strange thing for us to be in the business of doing	15:02
fungi	it doesn't seem like we're fighting it at all	15:02
*** kushal has joined #openstack-infra		15:02
clarkb	mordred: do you at least agreeforce pushing noca into a different repo is a bad idea?	15:03
*** pcaruana has quit IRC		15:03
fungi	i'm saying it's a step which can be performed locally, the same as it can be performed in our ci. why does the result (a duplicative, generated result) need to be pushed back into revision control history?	15:03
*** eharney has joined #openstack-infra		15:03
mordred	clarkb: I believe we sohuld be importing tarballs using gbp pristine-tarball	15:03
openstackgerrit	Sagi Shnaidman proposed openstack-infra/tripleo-ci: TEST: DONT RECHECK: periodic jobs https://review.openstack.org/359215	15:03
mordred	fungi: but it's a manual step which is not how gbp works	15:03
*** kien-ha has joined #openstack-infra		15:03
fungi	you don'y manually invoke gbp?	15:03
mordred	if it was a normal operational mode of gbp, then sure	15:03
*** annegentle has joined #openstack-infra		15:04
mordred	yes - but you run git-buildpackage and it does all the thigs	15:04
fungi	it seems like any manual process involves _at_least_ one manual step	15:04
mordred	yes. the one step of running git-buildpackage	15:04
openstackgerrit	Lukas Bednar proposed openstack-infra/jenkins-job-builder: Builders: Add ansible-playbook builder https://review.openstack.org/322243	15:04
pabelanger	zigo: mordred: What is the objection to pulling in the upstream git repo or tarball at build time? When I was rolling packages with git-buildpackages, I would use uscan to fetch them before git-buildpackage. Obviously skipping using the upstream-branch step	15:04
fungi	before running gbp, you probably _also_ have a manual step of cloning the repo you're going to run it in, yeah?	15:05
mordred	if for openstack packages it's clone the repo, now do this other stuff to get another repo added to this repo, then run git-buildpackage, then we've subverted the power of the tool	15:05
*** ggnel_t has quit IRC		15:05
*** kien-ha has quit IRC		15:05
mordred	the steps are "clone the packaging repo, cd to the packaging repo, run git-buildpackage" ... what I'm saying is that if we inject more manual steps into that, then we're doing weird things	15:05
zigo	fungi: pabelanger: 1/ It makes it very hard to manage quilt patch 2/ it's not the usual git-buildpackage workflow, so it will confuse contributors 3/ It will be a lot of pain points for no reasons maintaining scripts which we otherwise wouldn't need to write	15:05
*** kdas_ has quit IRC		15:05
zigo	fungi: pabelanger: What's the reason for NOT doing it?	15:06
pabelanger	mordred: zigo: I would use overlay, IIFC	15:06
*** thcipriani\|afk is now known as thcipriani		15:06
pabelanger	zigo: eg: https://github.com/pabelanger/jenkins-job-builder-deb/blob/master/debian/gbp.conf	15:06
mordred	pabelanger: right. with overlay you have to maintain local patches in quilt	15:06
clarkb	zigo: the reason for not pushing all of nova into nova deb is nova code has a home it does not need two homes. its goimg to double the size of our test image cahes and so forth	15:06
*** rcernin has quit IRC		15:06
pabelanger	yes	15:06
fungi	i guess my point is that i've seen plenty of different gbp workflows. some maintainers import tarballs, some pull from upstream git repos, some use quilt, some use single-patch, some push copies of upstream code back into public revision history and some don't...	15:06
mordred	that is very differnt than using git	15:06
mordred	fungi: right.	15:07
clarkb	you are aslo asking to bypass every control on herrit	15:07
pabelanger	mordred: zigo: yes, that is what I would do, quilt	15:07
zigo	pabelanger: Last time we discussed using this in the Python module team, absolutely everyone but a single person liked using overlay.	15:07
mordred	right	15:07
mordred	so ...	15:07
openstackgerrit	Jeffrey Zhang proposed openstack-infra/project-config: Increasing the timeout value for kolla jobs https://review.openstack.org/364377	15:07
mordred	I do not think we should completely rework the 175 packaging repos that have a multi-year packaging history already	15:07
pabelanger	zigo: hehe, ya, I prefer overlay too	15:07
mordred	to use quilt instead of git	15:07
fungi	i'm requesting flexibility in choosing a workflow that meshes well with the situation where upstream and packaging repositories are already in the same system and avoid keeping duplicate copies	15:07
fungi	mordred: well, they use quilt _and_ git afaik	15:08
pabelanger	mordred: Ya, now I understand the issue.	15:08
*** kien-ha has joined #openstack-infra		15:08
fungi	quilt packages carried in teh debian/patches dir, committed into the packaging branch	15:08
*** mptacekx has quit IRC		15:08
mordred	fungi: right. but there is no need to introduce quilt into an existing ecosystem that does not already use quilt	15:08
pabelanger	FWIW: I didn't mind quilt, but I am not the one doing the packaging	15:08
mordred	that is one way these _could_ have been done	15:08
mordred	but it's not the way they were	15:08
fungi	well, i wasn't specifically suggesting to use quilt. i was giving it as an example (and i thought zigo already carried debian/patches in his packages but i'll pull a source package and double-check that)	15:09
zigo	I do use quilt, I'm not sure what fungi suggests here...	15:09
mordred	zigo: wait - you're already using quilt?	15:10
mordred	wtf	15:10
mordred	why are we having this argument then?	15:10
clarkb	anyways my biggest concern is we dont bypass gerrit by allowing force pushes and we dont doubt the size of our test images by literally copy.pasting all our repos	15:10
zigo	mordred: I don't understand ! :)	15:10
clarkb	s/doubt/double/	15:10
*** yamamoto has quit IRC		15:10
*** zz_dimtruck is now known as dimtruck		15:11
*** mhickey has joined #openstack-infra		15:11
mordred	zigo: importing upstream sources into the packaging repo is a thing you do as an alternative to using quilt to manage local changes	15:11
zigo	mordred: No.	15:11
mordred	zigo: if you're using quilt, then just pointing at release tarballs should be fine	15:11
zigo	I use quilt patches to make sure I keep Debian specific patches separated.	15:11
zigo	I don't see any relationship between using quilt and the current gbp workflow.	15:12
pabelanger	Right, that is how I understood it too.	15:12
mordred	right. in a git workflow you can have each of those debian specific patches in the debian packaging branch and gbp will create the patch overlay and packaging build - you don't need the quilt, you have git	15:12
pabelanger	clarkb: Right, https://github.com/openstack/deb-python-os-client-config/tree/debian/newton	15:12
*** kien-ha has quit IRC		15:12
mordred	however, that's getting into the weeds	15:12
zigo	Nop! :)	15:12
zigo	That's not what it is about.	15:13
pabelanger	is basically pytnon-os-client-config repo, with debian folder added	15:13
zigo	At least, that's not how I use quilt.	15:13
pabelanger	clarkb: so, a lot of duplicated code	15:13
zigo	mordred: What you're talking about is more like if we were using git-dpm or gbp pq.	15:13
zigo	pabelanger is right.	15:13
pabelanger	I always thought we we're just talking about the debian folder for the packaging repos	15:13
fungi	the question being floated is, for example, whether we should import a complete copy of nova into the pkg-nova repo	15:14
zigo	We're talking about having upstream code/tag within the debian packaging branch, yes.	15:14
*** pblaho has quit IRC		15:14
pabelanger	like: https://github.com/openstack-infra/zuul-packaging/tree/debian/sid	15:14
zigo	Which is what 99% of package maintainers do.	15:14
mordred	yah. if we're keeping patches in quilt, I see no point in having the upstream sources in the repo. I _do_ argue in favor of upstream sources in the repo so that quilt can be avoided	15:14
clarkb	pabelanger: no this is literally copying all history an potentially making more than one copy one for each branch (not sure how its organized)	15:15
fungi	and my position, which i've yet to see clear evidence indicating it's not viable, is that pulling nova source into the pkg-nova repo could happen at job runtime rather than being a manual process the results of which are pushed back into the repo in gerrit	15:15
zigo	Then how do you actually manage to run dpkg-source --commit ?	15:15
pabelanger	clarkb: looking at https://github.com/openstack/deb-python-os-client-config and https://github.com/openstack/os-client-config it is slightly confusing which one is upstream, since both master branches are the same	15:16
openstackgerrit	Merged openstack-infra/jenkins-job-builder: Add support for Parameterized Scheduler Plugin https://review.openstack.org/353165	15:17
zigo	fungi: I'm not saying it's not possible to do it, I'm saying it's a WAY more complicated to do what you're describing.	15:17
zigo	fungi: First, you got to design the code to do the pull / push from here and there.	15:17
*** Thelo_ has joined #openstack-infra		15:17
*** sandanar has quit IRC		15:17
*** dtantsur is now known as dtantsur\|afk		15:17
fungi	zigo: we actually do a lot of tat in other jobs anyway	15:17
zigo	fungi: Then, when designing a quilt patch, you need to do many things so that you can finally type dpkg-source --commit	15:17
pabelanger	I'm not sure it is way more complicated, but it does delay the packaging process	15:17
*** sputnik13 has quit IRC		15:18
zigo	fungi: Then, we'd have to do more work to upload to debian as well, instead of just git clone, build, upload.	15:18
zigo	I've given my argumentation about why I prefer the current workflow, I still don't get why you prefer not using it.	15:18
fungi	i'm suggesting that when developing locally you can pull from the nova repo while writing your quilt patch, but only push the quilt patch back into the pkg-nova repo rather than merging a copy of all of nova into the pkg-nova repo	15:18
*** Thelo_ has quit IRC		15:18
zigo	fungi: Yes, that's more manual operations !	15:19
zigo	:)	15:19
*** krotscheck has joined #openstack-infra		15:19
*** mlavalle has joined #openstack-infra		15:19
zigo	To me, it just looks like you're saying "whaaaaat? We'll have 2 copies?". Is there any other point of argumentation besides that?	15:19
fungi	and is avoiding that one manual operation worth keeping an entire extra copy of all upstream commits to all packaged projects in gerrit?	15:19
zigo	Also, what would be the workflow if I want to package commit 28af82dec ?	15:20
mlavalle	hi is there anybody here who can help me with a logstash question?	15:20
*** sputnik13 has joined #openstack-infra		15:21
zigo	The version should be something like 1:2.3.4+2016.09.01.git.28af82dec-5+13~bpo8+1	15:21
zigo	Do you suggest that we add some logic to extract the git sha256, and pull the relevant things in the packaging branch, in the build script?	15:21
zigo	This looks very hackish to me.	15:22
clarkb	zigo: I listed my two reasons why I don't like it	15:22
pabelanger	zigo: right, we'd have to do that	15:22
zigo	clarkb: I was on the phone at the same time, I'm sorry, I probably missed them.	15:22
zigo	(now I'm all focussed...)	15:22
pabelanger	zigo: and we do today, some jobs will use zuul-cloner to fetch specific commits. This could be used in your build process with the Depends-On field	15:23
*** jcoufal_ has joined #openstack-infra		15:23
*** sdake has joined #openstack-infra		15:23
*** sputnik13 has quit IRC		15:23
*** Sukhdev has quit IRC		15:24
clarkb	zigo: I don't want multiple copies of every repo floating around as it puts pressure on already large test images which we can't reliably update. I also don't want a git process that bypasses every control in Gerrit	15:24
*** kien-ha has joined #openstack-infra		15:24
pabelanger	rcarrillocruz: over an hour and no launch failures in infracloud	15:25
rcarrillocruz	woot!	15:25
*** jcoufal has quit IRC		15:25
zigo	clarkb: There's no such thing as "bypasses every control in gerrit", if I understand correctly.	15:25
pabelanger	rcarrillocruz: our time to ready is still a little high, but we could still be distributing images	15:26
clarkb	zigo: what you want is to be able to force push arbitrary commits into your repos	15:26
zigo	clarkb: I believe a git merge -X <tag-name> would just end up in a normal CR.	15:26
clarkb	zigo: that is bypassing the controls we have put in place	15:26
clarkb	oh you are wanting to push a merge commit? ok thats better. The previous discussion said you all needed to push the tags straight in	15:26
zigo	clarkb: That isn't what it seemed when I tried to "git review" a merge commit.	15:26
*** armax has joined #openstack-infra		15:26
*** jordanP has joined #openstack-infra		15:26
*** esp has joined #openstack-infra		15:26
clarkb	but its still creates the other problem of doubling (or worse) the size of all the repos in gerrit	15:27
clarkb	mordred: zigo if we do that I will suggest we not cache any of the packaging repos on our test images	15:27
zigo	clarkb: Basically, I want to be able to do: git merge -X theirs <TAG_NAME> .... some packaging change .... git commit -a --amend && git review	15:27
clarkb	because it just doesn't scale that way	15:27
*** mhickey has quit IRC		15:27
zigo	clarkb: That, you guys decide, that wouldn't be me.	15:28
zigo	If you think it's better to remove the cache, that's an -infra team decision.	15:28
*** Thelo_ has joined #openstack-infra		15:28
clarkb	zigo: not that its better just that I am only interested in carrying one copy of the data in the space constrainted environment	15:28
zigo	I've been doing git clone on each build for a very long time on my jenkins, so that's ok to me.	15:28
*** ifarkas is now known as ifarkas_afk		15:28
*** vinaypotluri has joined #openstack-infra		15:29
zigo	clarkb: We're talking about around 1GB of data here.	15:29
*** sputnik13 has joined #openstack-infra		15:29
zigo	Do we really need to save THAT much?	15:29
clarkb	zigo: its not just 1GB	15:30
clarkb	its 1GB in nova then 1GB in pkg nova	15:30
clarkb	then in six months is 1.5GB and 1.5GB	15:30
zigo	Nop, that's the total for all of the Git in all of my packaging.	15:30
clarkb	(illustrative only not real numbers)	15:30
clarkb	zigo: yes trying to illustrate the growth pattern here though	15:30
zigo	Nova is huge, other packages are mostly very small.	15:30
clarkb	and as mroe and more projects happen and you package more and more repos...	15:30
clarkb	the git cache on our images is huge right now and accounts for most of the disk use of those	15:31
*** rcernin has joined #openstack-infra		15:31
pabelanger	I think the issue is, we are proposing another method of doing it, which is different then the workflow you are accustomed too. And the potential amount of work needed to switch workflows	15:31
zigo	For example, a non-bare repo of deb-python-debtcollector is not even 1MB with all the git history.	15:31
zigo	pabelanger: That's one big issue too. And I'm supposed to have all of OpenStack Newton ready this week too ...	15:31
*** senk has quit IRC		15:31
pabelanger	We could use openstack-infra/zuul-packaging as a POC, because I always envisioned using git-buildpackage with overlay to produce its package builds.	15:32
*** annegentle has quit IRC		15:32
zigo	Could we agree to discuss this later, ie when I'm not in the rush of a release?	15:32
*** sputnik13 has quit IRC		15:32
pabelanger	zigo: yes, this would impact that deadline. As we need to workout the process	15:32
clarkb	we turned off the du'ing due to the xenail build issue so I can't easily just look at logs for a disk size number	15:32
clarkb	but over 7GB I think	15:33
zigo	I still hope to release Newton with infra, but if we decide to switch to another workflow, I give up for this release.	15:33
*** senk has joined #openstack-infra		15:33
*** javeriak has joined #openstack-infra		15:33
zigo	I don't want to risk having a bad OpenStack release for Debian Stretch (frozen at the end of the year).	15:34
openstackgerrit	Peter Zhurba proposed openstack-infra/project-config: Add repo for openstack/puppet-glare. https://review.openstack.org/362950	15:34
zigo	(and Mitaka is not an option as Horizon/Mitaka isn't Django 1.10 compatible)	15:34
*** markusry has quit IRC		15:35
*** Hal1 has quit IRC		15:35
*** andreas_s has quit IRC		15:35
pabelanger	Not sure I follow, your going to upload the builds to debian from debian-openstack repo?	15:36
*** dizquierdo has quit IRC		15:36
rcarrillocruz	pabelanger, mordred : https://review.openstack.org/#/c/364397/	15:36
rcarrillocruz	i still need to put a test for it, but you get the idea	15:37
rcarrillocruz	clarkb: we were also talking about ^ these days	15:37
zigo	pabelanger: I'm saying that I wont have the necessary time to change everything (workflow, build scripts, etc) given the time constraints of the Newton release.	15:37
clarkb	pabelanger: zigo openstack/ is 6.7 GB	15:37
clarkb	so if packaging deb is 1GB you represent more than 1/7th of our entire disk use	15:37
zigo	clarkb: This includes stuff I don't need.	15:37
clarkb	zigo: yes I am comparing you to the whole	15:38
zigo	ok	15:38
clarkb	and that will only grow as the rest of openstack/ grows because you are a copy of a significant chunk of it	15:38
pabelanger	zigo: right, you want to use same build script for both repos. I was confused you wanted to drive package builds from openstack into debian some how.	15:38
* zigo tries now to have actual real numbers		15:38
*** rhallisey has quit IRC		15:39
openstackgerrit	mathieu bultel proposed openstack-infra/tripleo-ci: WIP - Implement overcloud upgrade job - Mitaka -> Newton https://review.openstack.org/323750	15:39
*** mhickey has joined #openstack-infra		15:39
clarkb	I am totally happy to cache those repos and host them. I just don't think they should include entire copies of the other repos we host	15:39
pabelanger	clarkb: Ya, that was my thought on using overlay option for git-buildpackage	15:39
pabelanger	otherwise, why not just include debian folder into nova?	15:40
pabelanger	rcarrillocruz: seems right, can we add test for devstack?	15:41
pabelanger	rcarrillocruz: Oh, I see you are working on that	15:41
zigo	clarkb: You do realize that this also includes 3rd party python modules?	15:41
rcarrillocruz	yep, follow up patch	15:41
zigo	(so that the reasoning doesn't apply for them...)	15:41
clarkb	we also created some of the repos multiple times in gerrit	15:41
*** jheroux has quit IRC		15:42
*** Swami has joined #openstack-infra		15:42
clarkb	so my ask is just that we bit a bit more careful and not spam gerrit with a bunch of unnecessary copies of things. AIUI this is possible	15:42
zigo	Currently, in alioth.debian.org all the deb-* are 1.5 GB bare git repos.	15:42
*** salv-orlando has quit IRC		15:43
zigo	clarkb: Would it be possible to install the cache only in the Jessie image?	15:43
openstackgerrit	Merged openstack-infra/jenkins-job-builder: Fix disabled always returning true https://review.openstack.org/357491	15:44
*** roxanaghe has joined #openstack-infra		15:44
clarkb	zigo: ya there is potential tuning we can do around that. However we have in the past tried to treat them all the same so that you don't have to consider such differences when moving among distros. Focus should be on the distro differences	15:44
clarkb	I think we shoul stick to that goal as much as possible.	15:44
*** markusry has joined #openstack-infra		15:44
zigo	Makes sense.	15:45
pleia2	good morning	15:45
*** esp has left #openstack-infra		15:46
*** kaisers_ has quit IRC		15:47
openstackgerrit	Paul Belanger proposed openstack-infra/project-config: Bring osic-cloud8 online with 10 nodes https://review.openstack.org/364408	15:49
pabelanger	clarkb: rcarrillocruz: fungi: brings osic-cloud8 online^	15:49
pabelanger	so far both internap-mtl01 and infracloud-vanilla look good	15:49
rcarrillocruz	+2	15:50
rcarrillocruz	has the multinode issue being solved	15:50
rcarrillocruz	sdake:	15:50
rcarrillocruz	^	15:50
rcarrillocruz	erm	15:50
rcarrillocruz	sdague: ^	15:50
rcarrillocruz	sorry mr dake :-)	15:50
*** matrohon has quit IRC		15:50
pabelanger	yes, sdague should have included you	15:50
sdague	rcarrillocruz: yes	15:50
rcarrillocruz	so, we good to bump now	15:50
rcarrillocruz	?	15:50
rcarrillocruz	i have a change to bump infracloud to 50	15:51
rcarrillocruz	and i increased quota this morning on the zuul project	15:51
sdague	so, probably, I'd just ramp slowly and make sure to watch zuul like a hawk to look for new fails	15:51
*** markvoelker has joined #openstack-infra		15:51
sdague	what would be super cool is if nodes could be brought into a test run holding pen where some changes are just duped onto them	15:52
sdague	so we could get burn in without blowing up real work	15:52
pabelanger	rcarrillocruz: So, I want to see what is needed to start getting mcollective going on infracloud. Or some other method for statsd	15:52
pabelanger	sdague: Ya, I've often thought of that too	15:53
rcarrillocruz	sdague: so now it's on 10 nodes, my change is about bumping to 50. I'm cool bumping to less than that	15:53
pabelanger	would be a good tool to help debug clouds without affecting production	15:53
rcarrillocruz	indeed	15:53
sdague	because this only test live thing is just rough if people don't stay on top of job fails for the hours after	15:54
sdague	I seem to be meat nagios for that a bunch	15:54
*** slashme has left #openstack-infra		15:54
rcarrillocruz	pabelanger: i'm confused, statsd is metrics but my understanding is mcollective is puppet orchestration ?	15:54
pabelanger	rcarrillocruz: haha, collectd	15:55
pabelanger	my mistake	15:55
jeblair_	please don't run collectd on just some of our systems	15:55
jeblair_	if you want to do that, replace cacti completely, and everywhere, and make sure that you get data that is correct like cacti	15:55
pabelanger	I thought that was the tooling for choice from the talks at our last midcycle?	15:55
jeblair_	we don't need two systems graphing memory in parallel	15:56
rcarrillocruz	pabelanger , jeblair_ : that seems like a good topic for either mid cycle or summit	15:56
rcarrillocruz	i agree we should stick to one	15:56
pabelanger	Ya, if we can use the same across everything, that makes things easier	15:57
*** _nadya_ has joined #openstack-infra		15:57
jeblair_	rcarrillocruz: i don't think anyone is opposed to it. i'm just saying it's a lot of hard work, and you have to understand what every graph is measuring to be sure to get the data right. our cacti install had a lot of work put into it to make sure it's correct.	15:58
jeblair_	rcarrillocruz: i mean, maybe we should talk about it so i can say this to everyone	15:58
rcarrillocruz	indeed	15:58
*** gyee has joined #openstack-infra		15:58
ihrachys	depends-on fails for me lately. somehow zuul forgets to pick up a patch for merge after its dependency lands its repo	15:58
jeblair_	because this is the second time in the past few days i've given this little speech :)	15:58
*** roxanaghe has quit IRC		15:58
rcarrillocruz	heh	15:58
ihrachys	has anyone noticed?	15:58
openstackgerrit	Doug Hellmann proposed openstack-infra/project-config: do not fail if there are no constraint updates to push https://review.openstack.org/364417	15:58
sdague	ihrachys: which patches?	15:58
pabelanger	I'm sure its been me twice asking about statsd too	15:58
jeblair_	pabelanger: in the mean time, we do run cacti everywhere, including infra cloud	15:58
ihrachys	sdague: just nudged https://review.openstack.org/#/c/325208/	15:58
rcarrillocruz	fwiw, i added all compute hosts to cacti	15:59
*** kushal has quit IRC		15:59
ihrachys	sdague: there were others, but I can't immediately come up with links because I lost context.	15:59
jeblair_	rcarrillocruz: cool, i'll update the trees later	15:59
*** javeriak_ has joined #openstack-infra		15:59
*** matthewbodkin has quit IRC		16:00
sdague	ihrachys: python-neutronclient isn't in a shared pipeline with neutron	16:00
ihrachys	sdague: another one was https://review.openstack.org/#/c/362772/	16:00
sdague	so I think that's expected	16:00
sdague	they have to share a pipeline for them to auto process together	16:00
sdague	ihrachys: yep, that would be the same thing	16:00
sdague	same reason it happens for project-config changes	16:00
dhellmann	infra folks, it would be good to have some quick attention on https://review.openstack.org/364417 to cut down the false failures we're seeing in release jobs today	16:00
*** javeriak has quit IRC		16:00
ihrachys	sdague: oh I see. not that it's the best user experience because honestly I don't know by heart pipeline setups :)	16:00
pabelanger	jeblair_: Yes. Maybe what I should be asking is how to get data out of cacti to be rendered with grafana.	16:01
jeblair_	ihrachys: if the two projects are not tested together from source, then the depends-on has a slightly different meaning. it means "this patch can't be enqueued until the other patch lands".	16:02
zaro	morning	16:02
ihrachys	jeblair_: yeah, but even when the first lands, the second does not get anywhere near merge queue	16:02
jeblair_	ihrachys: (in this case, the depends-on can't be used to incorporate a pending change from another repo)	16:02
ihrachys	jeblair_: so I need to W+0/W+1 it again to get the job done	16:02
ihrachys	jeblair_: you mean by client patch has not really fetched server side?	16:03
jeblair_	ihrachys: yes, that's true. you could always leave the depends-on header off in that case.	16:03
jeblair_	ihrachys: correct	16:03
sdague	jeblair_: would it be hard to look for those an enqueue them on patch merge?	16:04
ihrachys	jeblair_: gotcha. it sucks and is confusing, but I guess I just had too many hopes in the magic :)	16:04
jeblair_	ihrachys: well, if it did then the pipeline config is wrong	16:04
*** claudiub has joined #openstack-infra		16:04
sdague	I've definitely seen it trip people up	16:04
jeblair_	ihrachys: if it actually did pull in changes, then maybe it should be in the same shared queue	16:04
ihrachys	jeblair_: I guess they were passing fine independently, that's why I haven't spotted anything on check queue	16:04
jeblair_	sdague: i think that would be possible	16:05
*** andrewbogott has quit IRC		16:05
*** andrewbogott has joined #openstack-infra		16:05
sdague	jeblair_: I think that's the thing that people are expecting, then get confused when it doesn't happen.	16:05
ihrachys	jeblair_: ok, for example neutronclient gates on neutron code via its functional job. wouldn't it mean that they should have had the same pipeline?	16:05
*** kien-ha has quit IRC		16:06
sdague	ihrachys: only if they share a job name	16:06
*** jlk` is now known as jlk		16:06
*** jlk has quit IRC		16:06
*** jlk has joined #openstack-infra		16:06
jeblair_	i guess in this case, neutron doesn't gate on neutronclient	16:06
sdague	jeblair_: correct	16:06
sdague	which is intentional	16:06
openstackgerrit	Zara proposed openstack-infra/infra-manual: Document commit message syntax for StoryBoard https://review.openstack.org/360668	16:07
*** nstolyarenko has quit IRC		16:07
openstackgerrit	Khai Do proposed openstack-infra/system-config: upgrade Gerrit's javamelody plugin https://review.openstack.org/364424	16:08
zaro	fungi, clarkb: for tomorrow ^	16:08
jesusaur	infra-root: when you get a chance, I'd like your opinions on https://review.openstack.org/363969 and the output at http://logs.openstack.org/69/363969/1/check/gate-project-config-layout/2386513/console.html#_2016-08-31_23_11_54_465294	16:08
*** mwhahaha has quit IRC		16:10
*** shashank_hegde has joined #openstack-infra		16:11
*** ijw has joined #openstack-infra		16:11
mgagne	pabelanger, clarkb: ready for internap-mtl01	16:11
sdague	any other project-config core folks want to do a quick review here - https://review.openstack.org/#/c/363937 ?	16:11
*** yamamoto has joined #openstack-infra		16:11
openstackgerrit	Merged openstack-infra/project-config: [Rally] Make py35 job voting https://review.openstack.org/361994	16:12
sdake	rcarrillocruz no need to call me mr :)	16:12
*** salv-orlando has joined #openstack-infra		16:12
rcarrillocruz	;-)	16:12
*** jordanP has quit IRC		16:12
openstackgerrit	David Lyle proposed openstack-infra/project-config: Add craton-dashboard repository (Horizon Plugin) https://review.openstack.org/354274	16:13
fungi	sdague: in theory we could do something like that with a nodepool-dev/zuul-dev environment. set them up with a limited project list and a check pipeline and only add quota for new providers to nodepool-dev initially	16:14
fungi	(the new provider incubation corral)	16:15
sdague	fungi: that would be kind of nice	16:15
fungi	it would be a nontrivial thing to create and maintain, but it's probably worth entertaining	16:15
sdague	because the onboarding of new providers is great for capacity, but tends to make fail spikes	16:15
*** yamamoto has quit IRC		16:15
sdague	so is rough during crunch time	16:16
clarkb	I mean	16:17
rcarrillocruz	fungi: so, i assume we'll get chocolate on newton , when we discuss at the mid-cycle no?	16:17
rcarrillocruz	when is newton due?	16:17
*** annegentle has joined #openstack-infra		16:17
clarkb	have ahve fail spikes in long lived clouds all the time too	16:17
clarkb	I think its more "cluod" than "new cloud"	16:17
clarkb	(like the osic ipv6 thing wasn't a new cloud issue, just cloud issue)	16:18
pabelanger	there is a lot of overhead for nodepool-dev too, image uploads for example	16:18
fungi	rcarrillocruz: worth bookmarking https://releases.openstack.org/newton/schedule.html	16:18
clarkb	that said I think its a reasonable thing to do I just don't want the expectation to be we will never have a cloud issue again	16:18
rcarrillocruz	thanks	16:18
*** kien-ha has joined #openstack-infra		16:19
sdague	clarkb: sure	16:19
mordred	clarkb: to be fair though ... the ipv6 was a cloud config change that we knew about and was _like_ adding a new cloud	16:19
mordred	or could have been seen that way - although I will admit I did not think of it that way at the time	16:19
*** hashar is now known as hasharAway		16:19
sdague	but we had to drop an internap region, and infra cloud in the last 2 weeks	16:19
clarkb	mordred: I didn't (and I still don't really getting a new IP address is not liek getting a new cloud imo)	16:19
fungi	rcarrillocruz: so the infra sprint is the week before final release candidates for newton	16:20
sdague	all of which could be better servered with a holding pen	16:20
mordred	clarkb: totally	16:20
sdague	so those could live debug through issues	16:20
sdague	and once got to a good pass rate, get added to the good pool	16:20
clarkb	sdague: yup I agree. I have always tried to run tests myself on the instances before bringing them into the fold not sure if we managed to do that this time. And having a tool to do that would avoid ah uman needing to think about it. So definitely a reasonable thing to do	16:20
fungi	sdague: clarkb: mordred: though i think having a proving ground shadow ci for new providers in some ways simplifies testing them out when we have to determine appropriate flavors and whatnot	16:21
fungi	er, what clarkb just said basically	16:21
*** e0ne has quit IRC		16:21
sdague	fungi: yep	16:21
*** lucasagomes is now known as lucas-dinner		16:21
*** senk has quit IRC		16:21
sdague	I am well aware we will always have fails :)	16:22
fungi	we hadn't seriously considered it in the past because adding a new provider happened once every year or two	16:22
*** ramishra_ has quit IRC		16:22
fungi	now it seems to be something closer to a monthly occurrence	16:22
rcarrillocruz	heh, yeah	16:22
fungi	what an awesome problem to have, btw	16:23
pabelanger	rcarrillocruz: fungi: Maybe Newton RC1?	16:23
*** ramishra has joined #openstack-infra		16:23
fungi	pabelanger: i'm not opposed to running infra-cloud on bleeding-edge prerelease code as long as it doesn't get in the way of being able to use it. having early feedback to the community on new releases is great, but i think our #1 goal should be making sure we're able to run _something_ and use it consistently	16:24
*** coolsvap has joined #openstack-infra		16:24
rcarrillocruz	wfm, although i think that bringing up all servers, iron out issues, get back to DC with tickets etc, it could very well get past one-two weeks, just in time for the newton release	16:24
fungi	if sticking with mitaka helps increase our chances of keeping it up and running i'm much more in favor of that	16:25
*** krtaylor has quit IRC		16:25
pabelanger	rcarrillocruz: What is stopping us from launching infra-chocolate now?	16:25
fungi	and then consider upgrading to newton once newton is releasedc	16:25
*** sarob has joined #openstack-infra		16:25
fungi	pabelanger: mostly that there's still work to finish on vanilla, aiui	16:26
rcarrillocruz	i need to double check the inventory of things	16:26
*** berendt has joined #openstack-infra		16:26
rcarrillocruz	what took a long time was to find out what those servers were	16:26
rcarrillocruz	cos in our inventory we had ilO ips	16:26
pabelanger	rcarrillocruz: fungi: okay, assign me a task! I'm eager to help	16:26
pabelanger	:)	16:26
rcarrillocruz	no serial numbers	16:26
rcarrillocruz	no nothing	16:26
* mtreinish wants ice cream now		16:26
fungi	indeed	16:26
rcarrillocruz	but the DC folks they refer them by serial number	16:27
rcarrillocruz	so i had to go one by one	16:27
rcarrillocruz	guessing by macs	16:27
rcarrillocruz	what was what	16:27
rcarrillocruz	registering the racks the machine were put	16:27
rcarrillocruz	etc	16:27
*** martinkopec has quit IRC		16:27
rcarrillocruz	essetnially, cross-checking two different sources of truth	16:27
rcarrillocruz	but sure, we can start doing provisioning on chocolate	16:27
*** martinkopec has joined #openstack-infra		16:27
rcarrillocruz	as a amtter of ract, i plan to use those machines to do the live demo i promised on the meeting a couple weeks ago	16:28
rcarrillocruz	how to enroll with bifrost	16:28
rcarrillocruz	deploy	16:28
rcarrillocruz	delete	16:28
rcarrillocruz	redeploy	16:28
rcarrillocruz	etc	16:28
*** martinkopec has quit IRC		16:28
fungi	yeah, and doing a demo with software you already have deployed previously increases the chances that it's a viable demonstration rather than getting bogged down in whatever isn't quite right for newton yet	16:29
pabelanger	okay, is there a list of infracloud-vanilla that needs finishing up?	16:29
rcarrillocruz	yeah	16:29
rcarrillocruz	pabelanger: none	16:29
rcarrillocruz	all that can be deployed, are deployed	16:29
rcarrillocruz	we have 3 machines with issues	16:29
pabelanger	rcarrillocruz: everything is online?	16:29
rcarrillocruz	that i have tickets for	16:29
*** jpich has quit IRC		16:29
rcarrillocruz	everything that is ok, is online	16:29
pabelanger	nice	16:29
rcarrillocruz	check oepnstack-dev mailing list, i did a summary of the machines	16:29
fungi	and i guess the new ansible wheel is churning correctly for them, they're in cacti now, et cetera?	16:29
rcarrillocruz	sec	16:29
rcarrillocruz	i'll link	16:30
rcarrillocruz	http://lists.openstack.org/pipermail/openstack-dev/2016-September/102707.html	16:30
jeblair_	fungi: is there a wiki-dev01.openstack.org server without a forward dns record?	16:30
*** jeblair_ is now known as jeblair		16:30
*** sarob has quit IRC		16:30
rcarrillocruz	fungi: they are all in cacti	16:30
fungi	jeblair_: it's the result of a launch-node.py --keep i'm trying to work through vcsrepo errors for	16:31
fungi	jeblair: i'll be deleting it shortly and relaunching	16:31
jeblair	fungi: ok. pabelanger is not the only one to get NDRs -- i just got one for him for that server :)	16:31
fungi	jeblair: cute	16:31
rcarrillocruz	i was hoping to get ALL fixed today, the lab said 'all is good', but out of 4 with issues just one i brought it back to life today	16:31
rcarrillocruz	so we're down to 3 now with issues	16:32
pabelanger	ya, compute016.vanilla.ic.openstack.org looks to be down right now	16:32
jeblair	pabelanger, rcarrillocruz: i'm happy to help click through the spamhaus records for all the infra-cloud ips, or if we want to set up a smarthost and just do one or two, that should work too.	16:32
rcarrillocruz	pabelanger: yeah, that one has a HD broken	16:32
fungi	rcarrillocruz: pabelanger: do we have any good numbers on relative job runtimes and nondeterministic failures (if any) in our nodepool project in infra-cloud yet?	16:32
*** zul has quit IRC		16:32
rcarrillocruz	fungi: i think it may be too soon, just one day of real workload	16:33
rcarrillocruz	i really want to see dsvm runs when there are more noisy neighbours in the computes	16:33
*** Thelo_ has quit IRC		16:33
jeblair	fungi: the 2 graphs at the bottom say 'no datapoints' http://grafana.openstack.org/dashboard/db/nodepool-infra-cloud	16:33
rcarrillocruz	cos in my initial test, a nova tempest full run took the same as osic, but you know, the VM has the entire compute for itself	16:33
fungi	looking in grafana myself now, yes	16:33
fungi	looks like it's all building and deleting?	16:34
fungi	oh, i guess we have some in use	16:34
*** berendt has quit IRC		16:34
pabelanger	yes	16:34
rcarrillocruz	jeblair: in regards to the spamhaus thing, yeah, i think having a smarthost would be good	16:34
*** weshay is now known as weshay_lunch		16:35
pabelanger	fungi: jeblair: So, I'd like some feedback on: https://review.openstack.org/#/c/364322/ The current launch failures in infracloud are related to EOFError from paramiko. Not sure if that is the right approach	16:35
*** sambetts is now known as sambetts\|afk		16:35
pabelanger	fungi: jeblair: the other failure was related to DNS, we were just using google DNS, so we added unbound to infracloud this morning	16:36
jeblair	pabelanger: i'm suspicious of 364622	16:37
jeblair	pabelanger: does it happen elsewhere, or just omfra?	16:37
jeblair	pabelanger: and what's really going on? i mean, it opened a connection, but then it was closed?	16:38
pabelanger	jeblair: Ya, looks like randomly in rax, ovh, internap and bluebox too	16:38
openstackgerrit	Emilien Macchi proposed openstack-infra/project-config: Revert "tripleo-ui: add missing jobs for release management" https://review.openstack.org/364452	16:38
fungi	what's the failure rate from that? frequent enough we could attempt to recreate it with openssh?	16:38
fungi	and if it's a problem for paramiko, is it also going to happen to ansible?	16:39
pabelanger	http://paste.openstack.org/show/565742/ EOFError from today	16:39
fungi	does ansible (or maybe zuul-launcher) already have a similar workaround?	16:39
mordred	fungi: ansible uses openssh, not paramiko	16:40
*** Apoorva has joined #openstack-infra		16:40
*** yamahata has quit IRC		16:40
*** ijw has quit IRC		16:40
fungi	mordred: sure, which is why i asked whether there's any hope of us finding the underlying cause by testing with openssh	16:40
pabelanger	jeblair: Yes, I am not sure what the issue is right now. It looks to be on the first ssh_connect attempt from nodepool. And because we don't retry, the server is deleted and we move on	16:40
mordred	fungi: good point	16:40
jeblair	right, but i think what's being gotten at here is that some of the choices nodepool makes are about preventing bad hosts from making it to zuul	16:40
*** cardeois has quit IRC		16:40
*** daemontool has quit IRC		16:40
jeblair	so is this really a situation where we want to help more things get through	16:41
fungi	agreed, right now _assuming_ this eoferror indicates a problem node, then retry-spamming it into service could be detrimental to whatever job runs on it down the line	16:41
jeblair	this is why i asked what's really going on	16:42
*** woodster_ has joined #openstack-infra		16:42
jeblair	cause the other errors in there all have explanations for why we should ignore them (user not created yet, ssh not started yet, etc)	16:42
pabelanger	right, I cannot answer that. I was hoping retry would better expose the actually issue	16:42
fungi	yep, figuring out what is causing the eoferror paramiko is raising might help us figure out whether it's safe to press nodes exhibiting this behavior into service	16:42
fungi	rather than papering over the failure with retries	16:43
pabelanger	http://paste.openstack.org/show/565743/ is the traceback of the failure	16:43
rcarrillocruz	k folks, gotta run to catch my son	16:43
rcarrillocruz	later	16:43
fungi	is there a way to get paramiko to provide more detail on the failure mode?	16:43
fungi	aha, thanks	16:43
rcarrillocruz	https://review.openstack.org/#/c/364397/ mordred , pabelanger , it passed tests	16:43
rcarrillocruz	later	16:43
pabelanger	fungi: I believe we could enabled debug logs for that	16:44
*** akshai has quit IRC		16:45
fungi	logging.getLogger('paramiko.transport').setLevel(logging.DEB‌UG)	16:45
fungi	apparently	16:45
*** zul has joined #openstack-infra		16:45
*** trown is now known as trown\|lunch		16:45
*** yamamoto has joined #openstack-infra		16:46
openstackgerrit	Emilien Macchi proposed openstack-infra/project-config: tripleo-ui: use nodejs4-publish-to-npm template https://review.openstack.org/364455	16:47
jeblair	fungi: yeah, though looking at the code, i'm not sure we'd learn much from that	16:47
*** jamesdenton has quit IRC		16:47
*** akshai has joined #openstack-infra		16:47
fungi	https://github.com/paramiko/paramiko/issues/520 maybe?	16:47
*** ilyashakhat_mobi has joined #openstack-infra		16:48
*** asettle has quit IRC		16:49
pabelanger	jeblair: was there talk of moving away from paramiko in nodepool?	16:49
*** timello has quit IRC		16:49
jeblair	pabelanger: i don't recall?	16:49
pabelanger	okay	16:49
*** asettle has joined #openstack-infra		16:49
*** ilyashakhat_mobi has quit IRC		16:49
*** rhallisey has joined #openstack-infra		16:50
*** sarob has joined #openstack-infra		16:50
*** sputnik13_ has joined #openstack-infra		16:50
*** mhickey has quit IRC		16:50
*** drifterza has joined #openstack-infra		16:50
*** yamamoto has quit IRC		16:50
*** timello has joined #openstack-infra		16:50
Shrews	pabelanger: mordred: rcarrillocruz: fyi, ansible testing WG meeting happening in 9 min. i keep forgetting	16:51
pabelanger	Shrews: Thanks	16:51
*** pradk has quit IRC		16:51
*** amotoki has quit IRC		16:51
*** ilyashakhat_mobi has joined #openstack-infra		16:52
jeblair	pabelanger: i think if you want to chase this down, that's fine -- i would recommend you create a new log message for it though so you can track it	16:52
openstackgerrit	Merged openstack-infra/storyboard: Limit PyMySQL to < 0.7.7 https://review.openstack.org/364250	16:53
fungi	worth noting, we're on paramiko 1.17.2 for nodepool.o.o	16:53
pabelanger	jeblair: okay, I'll update the patch shortly	16:53
jeblair	pabelanger: and log the exception, so we know where it's coming from. that way if we see multiple eoferrors from the same host, we'll know if they're all from the same spot	16:53
pabelanger	jeblair: will do	16:53
*** asettle has quit IRC		16:54
fungi	ugh. their repo has tags for 1.17.2 and v1.17.2	16:54
*** sarob has quit IRC		16:54
fungi	context: https://github.com/paramiko/paramiko/blob/1.17.2/paramiko/transport.py#L492	16:55
fungi	(that file looks to be the same under the v1.17.2 tag as well)	16:56
*** derekh has quit IRC		16:56
*** drifterza has quit IRC		16:56
*** Thelo_ has joined #openstack-infra		16:58
*** markvoelker has quit IRC		16:58
pabelanger	fungi: ya, looks like we don't even get to the authentication step	16:58
*** cardeois has joined #openstack-infra		16:59
*** esikache1 has quit IRC		16:59
pabelanger	does ssh-server restart after keys are generated?	16:59
*** shashank_hegde has quit IRC		16:59
fungi	pabelanger: so what's interesting about the failures... none seem to be in osic	16:59
fungi	even though osic is now by far the bulk of our volume	17:00
clarkb	pabelanger: no it doesn't start at all until keys are generated	17:00
*** tesseract- has quit IRC		17:01
fungi	no rax-iad either, but i could chalk that up to lack of a statistically significant sample	17:01
fungi	however, if this were consistent across providers, i would expect to see lots in osic too	17:01
*** yamahata has joined #openstack-infra		17:01
pabelanger	rax-iad is currently disabled however	17:01
fungi	hah, that explains that one then	17:01
*** kgiusti has joined #openstack-infra		17:01
pabelanger	Ya, but no osic in any logs on disk	17:01
pabelanger	would should include ipv4	17:02
pabelanger	oh, maybe not	17:02
fungi	right, i think we're only connecting from nodepool to osic nodes via ipv6 now	17:03
pabelanger	2016-08-22 is last log, I think we had ipv6 by then	17:03
*** kdas_ has joined #openstack-infra		17:03
fungi	so the question is whether this is a v4-only issue, or an issue than manifests dirrerently (and raises a different error condition) under v6	17:03
fungi	or an issue that doesn't affect osic for some other reasons unrelated to ipv4 vs ipv6	17:04
*** jerryz has joined #openstack-infra		17:04
*** abregman has joined #openstack-infra		17:04
*** abregman\|mtg has quit IRC		17:06
*** edtubill has quit IRC		17:06
*** mwhahaha has joined #openstack-infra		17:07
*** tqtran has joined #openstack-infra		17:07
*** kdas_ is now known as kushal		17:08
*** kushal has quit IRC		17:08
*** kushal has joined #openstack-infra		17:08
*** nstolyarenko has joined #openstack-infra		17:08
*** ilyashakhat_mobi has quit IRC		17:08
*** krtaylor has joined #openstack-infra		17:08
*** weshay_lunch is now known as weshay		17:08
*** tqtran has quit IRC		17:12
*** ilyashakhat_mobi has joined #openstack-infra		17:12
*** tonytan4ever has quit IRC		17:16
mgagne	pabelanger: let me know when you are ready to enable more in internap-mtl01	17:16
*** HeOS has quit IRC		17:17
pabelanger	mgagne: I think we are trying to determine how the cloud is preforming right now. Average test runs, if anything is failing, etc	17:18
mgagne	alright, fine with me	17:18
clarkb	pabelanger: are any of thosefailures on precise or trusty? if not maybe its related to systemd socket activation and the service not starting fast enough for paramiko?	17:18
*** rossella_s has quit IRC		17:18
mgagne	is there any dashboard/link I can read?	17:18
clarkb	pabelanger: basically with socket activation systemd is going to listen on port22 and accept connections for ssh before sshd is ready. Then when sshd is ready it will hand over control of the socket	17:19
pabelanger	clarkb: let me extra the node type	17:19
*** rossella_s has joined #openstack-infra		17:19
clarkb	so wondering if maybe tcp handshake happens then we have a long pause long enough to make ssh unhappy	17:19
pabelanger	mgagne: mostly looking in logstash.o.o and nodepool logs atm	17:20
*** ilyashakhat_mobi has quit IRC		17:21
*** nstolyarenko has quit IRC		17:22
*** pradk has joined #openstack-infra		17:22
clarkb	hrm glance wants to remove -2 perms from their core group?	17:22
fungi	where did we get to with making nodepool logs public? i think we said we'd be comfortable with it once we finished the migration to shade?	17:23
fungi	and that's done now, afaik	17:23
pabelanger	clarkb: seems ubuntu-xenial and debian-jessie: http://paste.openstack.org/show/565751/	17:24
pabelanger	so you are on to something	17:24
clarkb	I think we are still maybe waiting for swift bits to use ksa? I don't recall if that was required for the password sanitizing	17:24
clarkb	though now all the swift stuff happens in the nodepool builder we could just not serve those logs	17:24
clarkb	fungi: ^	17:24
fungi	ahh, right	17:24
fungi	especially easy if we move the builder daemon to a separate server	17:25
*** niska has quit IRC		17:26
pabelanger	clarkb: ya, so that goes back to my question about ssh server being restart. It sounds like what you are describing with systemd could cause issues with the socket	17:27
pabelanger	if we connect early enough	17:27
*** ddieterly is now known as ddieterly[away]		17:27
pabelanger	So, once https://review.openstack.org/#/c/364322/ is cleaned up, we should see a failure or 2, and then eventually ssh connection	17:28
nikhil	heya.. there're a few glance patches in gate that seem stuck verifying? https://review.openstack.org/363838 https://review.openstack.org/363870 https://review.openstack.org/354332	17:28
nikhil	based on the status on zuul	17:29
clarkb	In theory it should be fine because systemd just holds the fd but maybe there is a timeout or something more aggressive in paramiko	17:29
openstackgerrit	James E. Blair proposed openstack-infra/puppet-nodepool: Enable mod_proxy when proxying status commands https://review.openstack.org/364478	17:29
nikhil	all of them are really really important for us to tag newton-3 today	17:29
*** senk has joined #openstack-infra		17:29
pabelanger	another option, could be to delay our ssh connections per cloud, with some sort of configuration option.	17:29
jeblair	clarkb, pabelanger, fungi: ^ i just did that manually on the nodepool server	17:29
nikhil	any help/pointer would be super useful!	17:29
clarkb	nikhil: have you pulled up teh console logs/	17:29
jeblair	clarkb, pabelanger, fungi, mgagne: http://nodepool.openstack.org/image-list works now	17:29
jeblair	as does http://nodepool.openstack.org/dib-image-list	17:29
clarkb	nikhil: says its still running tempest	17:30
mgagne	that's one interesting output	17:30
*** ramishra has quit IRC		17:30
pabelanger	jeblair: excellent	17:30
AJaeger	fnikil 363838 is still running...	17:30
*** dteselkin has quit IRC		17:30
jeblair	sdague: you may find http://nodepool.openstack.org/image-list useful	17:30
*** igormarnat has quit IRC		17:30
AJaeger	nikhil: sorry for typo ^	17:30
nikhil	AJaeger: clarkb : that's been running for along time	17:30
nikhil	and the other two seem done	17:30
AJaeger	nikhil: gate-tempest-dsvm-neutron-full-ubuntu-xenial is running - did you see that?	17:30
*** rakhmerov has quit IRC		17:30
*** ramishra has joined #openstack-infra		17:30
fungi	2016-09-01 17:30:47.614538 \| {1} tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_stop_start [165.255744s] ... ok	17:30
nikhil	AJaeger: yeah, waiting for it for last 1-1.5 hrs	17:31
*** kzaitsev_ws has quit IRC		17:31
*** kzaitsev_ws has joined #openstack-infra		17:31
fungi	the node seems to be in rax-ord so we can't blame slowness in infra-cloud or anything for this one	17:31
*** tsufiev has quit IRC		17:31
*** katyafervent_awa has quit IRC		17:31
*** penguinolog has quit IRC		17:32
nikhil	what's the best path forward? (recheck won't work) so, wait fora bit more and bug folks then :)	17:32
*** javeriak_ has quit IRC		17:32
*** e0ne has joined #openstack-infra		17:32
*** tsufiev has joined #openstack-infra		17:32
*** akshai has quit IRC		17:33
clarkb	nikhil: you'll have to debug why the job is slow	17:33
clarkb	it is still doing stuff though seems like	17:33
nikhil	clarkb: how do I get on this telnet link telnet://23.253.23.189:19885 ?	17:33
fungi	looks like devstack setup took from 15:17:41 to 15:56:38	17:33
clarkb	nikhil: use telnet or nc to that ip address and port	17:33
*** katyafervent_awa has joined #openstack-infra		17:33
nikhil	(that's what I get when I hover on that gate link)	17:33
*** tphummel has joined #openstack-infra		17:34
clarkb	nikhil: if you are more adventurous there are ways to have your browser do things automatically but I haven't bothered	17:34
*** igormarnat has joined #openstack-infra		17:34
*** cardeois_ has joined #openstack-infra		17:34
*** rakhmerov has joined #openstack-infra		17:35
fungi	looks like it's just slow. for example the gap from when test_server_connectivity_rebuild reported and test_server_connectivity_resize reported was on the order of 3 minutes	17:35
AJaeger	nikhil: telnet 23.253.23.189 19885 (or use nc with same arguments)	17:35
nikhil	clarkb: gotcha, will use telnet	17:35
*** ijw_ has joined #openstack-infra		17:35
nikhil	AJaeger: ty , just hopped on :)	17:35
*** kaisers_ has joined #openstack-infra		17:35
fungi	i don't see any tests that have actually failed in that log (yet)	17:36
*** javeriak has joined #openstack-infra		17:36
fungi	but it's entirely possible this job will continue until it reaches the job timeout and gets killed	17:36
*** dteselkin has joined #openstack-infra		17:36
AJaeger	sdague: for https://review.openstack.org/#/c/363937 we should ask the Neutron folks, shouldn't we? armax, dougwig , please review	17:36
* armax looks		17:36
nikhil	fungi: yeah, it's a change in config & tests are not expected to fail	17:37
nikhil	fungi: ok	17:37
clarkb	I think those tests are the ones that tend to run at the end	17:37
clarkb	but I haven't actually looked at the sorting recently	17:37
AJaeger	Zara: did you merge anything yet on python-storyboardclient? Any dummy commit to get content published?	17:37
openstackgerrit	Zara proposed openstack-infra/python-storyboardclient: Add due_dates https://review.openstack.org/345995	17:37
*** cardeois has quit IRC		17:38
*** abregman is now known as abregman\|mtg		17:38
openstackgerrit	Ben Nemec proposed openstack-infra/tripleo-ci: Add ipv6 nic-configs https://review.openstack.org/364479	17:38
AJaeger	do we want to merge the internap-mtl01 to 150 increase now? https://review.openstack.org/363984 ?	17:38
Zara	Ajaeger: it's not merged yet but there's a patch over here: https://review.openstack.org/#/c/362878/ for docs	17:38
fungi	the same job that ran on the two glance changes behind the slow one only took around an hour to complete, so there's probably something terribly wrong with the node this job is running on	17:38
clarkb	jeblair: I went ahead and just approved the mod proxy change since its already been applied by hand	17:38
jeblair	clarkb: thx	17:39
fungi	system load on that node is around 4	17:39
*** javeriak_ has joined #openstack-infra		17:40
*** dkehn has quit IRC		17:40
openstackgerrit	Paul Belanger proposed openstack-infra/nodepool: Retry ssh connect when EOFError is received https://review.openstack.org/364322	17:40
*** dkehn_ has quit IRC		17:40
nikhil	fungi: saw it succeed in 339 secs!	17:40
*** kaisers_ has quit IRC		17:40
fungi	not an extreme amount of memory pressure	17:41
pabelanger	AJaeger: still confirming if the cloud is ready for more nodes. Should know more in the next little bit	17:41
clarkb	fungi: how many bogomips >_>	17:41
*** javeriak has quit IRC		17:41
AJaeger	pabelanger: Mathieu Gagné commented "We're ready" on the review	17:42
fungi	clarkb: around 5200 on all 8 processors listed in cpuinfo	17:42
clarkb	pabelanger: is the current concern the ssh eof error?	17:42
mat128	mgagne: ^	17:42
*** dkehn has joined #openstack-infra		17:42
*** rbrndt has quit IRC		17:42
clarkb	fungi: thats more than my local machine!	17:42
mgagne	AJaeger: pabelanger said " I think we are trying to determine how the cloud is preforming right now. Average test runs, if anything is failing, etc"	17:42
pabelanger	clarkb: for internap-mtl01? Just wanted to confirm job times are in line with what we expect, I haven't actually checked that yet.	17:43
fungi	clarkb: i think the eoferror seems to be consistent across providers (sans osic) so probably not a concern for ramping up	17:43
clarkb	pabelanger: gotcha	17:43
clarkb	pabelanger: oh I remember things re the EOF error	17:43
clarkb	pabelanger: launch node ran into that too when running the restart command	17:43
AJaeger	mgagne: ah, thanks!	17:43
clarkb	pabelanger: but in that case its killing the services really fast. So maybe there is a restart of the service happenign that I didn't expect that is closing very earlyconnections	17:44
AJaeger	project-config cores, could you review https://review.openstack.org/#/c/364417 for the release team, please?	17:44
fungi	nikhil: well, it finally finished and the two behind it merged so looks like we're in the clear for now but should keep an eye out for more of the same	17:44
clarkb	pabelanger: you can actually reproduce that pretty easily by sshing into a host with systemd and running `reboot`	17:44
nikhil	fungi: thanks much!	17:44
*** nstolyarenko has joined #openstack-infra		17:44
pabelanger	clarkb: sdague: AJaeger: Looking at grafana for internap-mtl01, tempest tests are inline with nyj01. So, if everbody is on board, we can up the capacity I think	17:44
mordred	nikhil: o hai	17:44
nikhil	fungi: clarkb AJaeger : I noticed they merged after we'd a chat here. you guys have magic vision to make things work just by looking at'em!!	17:45
fungi	zuul's going to be busy for a while processing the dozen or so merges that piled up behind that glance change	17:45
nikhil	mordred: \o	17:45
*** jtomasek is now known as jtomasek\|afk		17:45
fungi	queue lengths are already starting to rise (events is up to 1k)	17:46
AJaeger	nikhil: we're all part of the magic team ;)	17:46
mordred	nikhil: rcarrillocruz and I were talking about image import yesterday ... and I just wanted to confirm	17:46
clarkb	pabelanger: so yes, I am suspecting some systemd behavior we may be tickling there	17:46
mordred	nikhil: import_from_url is not a thing in v2? or it's a thing but only with tasks?	17:46
fungi	clarkb: was there a systemd vs non-systemd split on the ssh failures then?	17:46
*** SumitNaiksatam has joined #openstack-infra		17:47
mordred	nikhil: (my understanding is that I should not consider import_from to be a thing anymore, yeah?)	17:47
clarkb	fungi: pabelanger said its just jessie and xenial	17:47
mrhillsman	mordred fungi pabelanger clarkb for osic cloud8 the best solution is the dns suggestion from yesterday	17:47
fungi	clarkb: in that case, yes, sounds highly likely	17:47
pabelanger	clarkb: I think we are restart SSH for some reason	17:47
pabelanger	getting a logs now	17:47
mrhillsman	attaching public address to the VM directly looks like it is going to require quite a bit of work	17:48
clarkb	pabelanger: though in theory restarting sshd doesn't kill existing connections... at least it didn't with upstart maybe this is new and exciting bugs	17:48
pabelanger	clarkb: http://paste.openstack.org/show/565759/ that is from an random osic ubuntu-xenail server	17:48
mrhillsman	i have to discuss with network folks since cloud8 is setup differently than cloud1	17:48
fungi	mrhillsman: it's an okay short-term solution, but long term it means that if we lose cloud1 for some reason then cloud8 will also effectively be dead to us	17:48
mrhillsman	understood	17:48
mrhillsman	makes total sense	17:49
nikhil	mordred: import from url won't be a generic one. things will be rather predefined http://specs.openstack.org/openstack/glance-specs/specs/mitaka/approved/image-import/image-import-refactor.html#api-changes	17:49
AJaeger	armax: is https://review.openstack.org/#/c/363937 fine?	17:49
pabelanger	clarkb: I think it maybe glean that kicks off the stop / start under systemd	17:50
clarkb	pabelanger: aha it does a stop start after the reload	17:50
*** cardeois_ is now known as cardeois		17:50
clarkb	pabelanger: ya I Think that would explain ti then as I have seen the same behavior with launch node on `reboot`	17:50
*** psachin has quit IRC		17:50
nikhil	mordred: as per that value discovery call, and then the info about the what to provide is here http://specs.openstack.org/openstack/glance-specs/specs/mitaka/approved/image-import/image-import-refactor.html#format-discovery	17:50
nikhil	mordred: which says, give me the container name if swift local or give me a stream of data is glance-direct	17:50
pabelanger	So, maybe we should see how to make glean run before networking is started	17:50
clarkb	pabelanger: we may be able to just edit glean to say before sshd	17:51
mordred	nikhil: awesome. thanks. super helpful	17:51
nikhil	mordred: so, once import refactor merges, that will be the case :)	17:51
pabelanger	clarkb: right	17:51
*** ijw_ has quit IRC		17:51
pabelanger	clarkb: let me get a new server up and play with it	17:51
clarkb	pabelanger: kk	17:51
armax	looking	17:52
*** e0ne has quit IRC		17:53
*** dkehn_ has joined #openstack-infra		17:54
*** esikache1 has joined #openstack-infra		17:55
jeblair	sdague, pabelanger: re https://review.openstack.org/364309 i share clarkb and mordred's suspicion that there might be something more subtle at play. ansible uses a persistent ssh connection, so the time/effort required to open a new channel should be greatly diminished. 10 seconds seems more than ample. it's also the case that we saw this sort of thing with jenkins too, which had a different approach to connectivity. having said ...	17:55
jeblair	... that, i don't have any other suggestions for how to make progress on this. i do think if we decide to do it, we need to check on it after a few days in production and evaluate if it's made a difference.	17:55
*** shashank_hegde has joined #openstack-infra		17:55
jeblair	(also, it's worth noting that zuul automatically restarts jobs that hit that problem)	17:55
openstackgerrit	Merged openstack-infra/devstack-gate: remove old tests https://review.openstack.org/360761	17:56
pabelanger	I still want to make ansible async ignore ssh failures and keep trying until the timeout is reached. Or some ignore ssh failure limit	17:57
dhellmann	hey, folks, I think I have some logic wrong in the script that tries to propose upper constraint changes when we release libraries.	17:57
dhellmann	it gets the branching for the requirements repo wrong	17:57
jeblair	pabelanger: yeah, i think that would be a nice improvement	17:57
*** nstolyarenko has quit IRC		17:57
dhellmann	in this log, it should be trying to check out the stable/mitaka branch to propose the commit there, but it doesn't find origin/stable/mitaka: http://logs.openstack.org/7a/7ad1cd3a0da7a95fb7c14cf9eeb9ae683c247efa/release-post/tag-releases/82a4a38/console.html#_2016-09-01_15_55_56_931716	17:57
*** harlowja_ has joined #openstack-infra		17:58
*** niska has joined #openstack-infra		17:58
AJaeger	thanks, armax !	17:59
openstackgerrit	Merged openstack-infra/puppet-nodepool: Enable mod_proxy when proxying status commands https://review.openstack.org/364478	17:59
openstackgerrit	Merged openstack-infra/storyboard: Don't allow users to subscribe to private worklists they can't see https://review.openstack.org/363776	17:59
pabelanger	jeblair: I also think, we maybe be too aggressive with 5s polling for zuul_runner. Maybe we should bump that to every 60 seconds or a percentage (5%) of the timeout value	18:00
*** harlowja has quit IRC		18:00
openstackgerrit	Zara proposed openstack-infra/python-storyboardclient: Add basic docs for python client https://review.openstack.org/362878	18:00
mordred	pabelanger: problem with 60s poll	18:00
clarkb	dhellmann: I think that may be a subtle git behavior where it can't identify a unique thing called that because there may be a file or other item with the same name?	18:00
*** ihrachys has quit IRC		18:00
mordred	pabelanger: is that then there is a 60s lag between a job finishing and ansible knowing that	18:00
dhellmann	clarkb : maybe? I'm pretty sure there is no file called origin/stable/mitaka though?	18:01
mordred	pabelanger, jeblair: in the ansible zuul work I want to do next week - which involves forward porting 2.5 to 3 - I also want to investigate our own action plugin	18:01
clarkb	dhellmann: right above where you linked you can see where it says stable/mitaka -> origin/stable/mitaka and its up to date	18:01
*** e0ne has joined #openstack-infra		18:01
mordred	and that action plugin should be able to be MUCH smarter about how async happens	18:01
mordred	because we know our intent	18:01
* clarkb tries to reproduce locally		18:01
dhellmann	clarkb : I've run into similar issues where the local branch name is not what I expect because of something about how we clone repos that I don't understand.	18:01
jeblair	mordred: ah, neat.	18:01
mordred	whereas the async module from ansible has to be more generic, which means it can't respond	18:01
dhellmann	clarkb : maybe I should change the script to just look for stable/mitaka and not origin/stable/mitaka? it used to be that the shorter name wouldn't exist locally, though	18:02
Shrews	mordred: that's exciting	18:02
jeblair	pabelanger: well, some zuul_runner things are very fast; i'd hate to have to wait 60 seconds for an 'echo' statement	18:02
clarkb	dhellmann: it works locally :/	18:02
dhellmann	clarkb : yeah	18:02
pabelanger	mordred: exciting	18:02
pabelanger	jeblair: ya, down side	18:02
fungi	dhellmann: the non-remote branch name will generally only exist locally in the clone if the job has requested checking out that branch by name	18:03
sdague	jeblair: zuul did not restart this job	18:03
pabelanger	if only we could have linear increasing polling	18:03
jeblair	sdague: i did not see an error report for it	18:03
dhellmann	fungi : ok. I'm looking for the full remote name and not finding it as origin/stable/mitaka. would it have a different name for some reason?	18:03
fungi	dhellmann: by default, a normal clone only gets the default branch indicated in the remote HEAD (or an alternate branch if you specified one explicitly)	18:04
mordred	oh. crap. I was going to fix a bug for jeblair today	18:04
*** _nadya_ has quit IRC		18:04
sdague	jeblair: we force repromoted it because it was the critical patch to fix multinode	18:04
mordred	jeblair: do you remember what the bug was that I was going to track down today?	18:04
fungi	dhellmann: you also won't have an origin/stable/mitaka until after a remote update	18:04
*** degorenko is now known as _degorenko\|afk		18:04
fungi	pretty sure	18:04
fungi	testing now	18:04
sdague	it had failed that job, reset the gate, taken everything off of it	18:04
sdague	but we needed that patch	18:04
dhellmann	fungi : this is from the release.sh script doing its own call to git clone to check out the requirements repository	18:04
dhellmann	fungi : and then it does "git fetch -v --tags" and the output from that includes a bunch of branch names, including stable/mitaka and origin/stable/mitaka	18:05
AJaeger	Zara: you're too fast for me - I explained how to use references...	18:05
openstackgerrit	Emilien Macchi proposed openstack-infra/project-config: tripleo-ui: use nodejs4-publish-to-npm template https://review.openstack.org/364455	18:05
jeblair	sdague: http://logs.openstack.org/66/364266/2/gate/gate-grenade-dsvm-neutron-multinode/63498bd/console.html this one?	18:05
dhellmann	unfortunately that's all in a temporary directory so it's no longer there to examine	18:05
fungi	dhellmann: yeah, just confirmed, if i `git clone ...` the keystone repo, and then cd into it, `git branch -v` only lists "master"	18:05
sdague	jeblair: yes	18:05
*** claudiub has quit IRC		18:05
clarkb	fungi: ya but the job does git fetch -v --tags first which seems to populate the things	18:06
openstackgerrit	Gabriele Cerami proposed openstack-infra/tripleo-ci: Add IPv6 network configuration for ipv6 job types https://review.openstack.org/363674	18:06
*** trown\|lunch is now known as trown		18:06
clarkb	fungi: at least when I do the same for requriments taht git show works locally	18:06
dhellmann	fungi , clarkb : I can add a "git remote update" if you think that would help, but I thought the fetch was more or less doing that?	18:06
fungi	clarkb: well, except git branch -v still only lists my local master after i do that	18:06
fungi	hrm, maybe i'm using git branch incorrectly	18:07
clarkb	fungi: see git branch -a	18:07
fungi	i wanted git branch -a	18:07
Zara	AJaeger: ahaha, thanks. I'll fix it. :)	18:07
fungi	yep	18:07
*** kzaitsev_ws has quit IRC		18:07
fungi	so even immediately after a git clone, git branch -a actually has all the remote branches for the remote i cloned from	18:07
jeblair	pabelanger, mordred, sdague: hrm. ansible did exit with exit code 1, not 3.	18:08
*** salv-orl_ has joined #openstack-infra		18:08
fungi	but no local branches except the one referenced by HEAD	18:08
dhellmann	clarkb , fungi : http://paste.openstack.org/show/565765/	18:08
AJaeger	Zara: and sorry for wrong guideance on ``code``	18:08
pabelanger	jeblair: yes, because failed=1	18:08
*** sshnaidm is now known as sshnaidm\|afk		18:08
clarkb	dhellmann: I think I know what the issue is	18:08
jeblair	pabelanger: why did that end up as a failure?	18:09
clarkb	dhellmann: its a git repo in a git repo	18:09
*** Na3iL has quit IRC		18:09
*** tqtran has joined #openstack-infra		18:09
clarkb	dhellmann: can you confirm that is how the job is setting the tree up? It seems that way from the log	18:09
jeblair	pabelanger: any chance it's because of the block/rescue thing?	18:09
dhellmann	clarkb : oh! could be	18:09
dhellmann	it does do that, yes	18:09
fungi	ahh, right, git has some funky behaviors around git inside git	18:10
clarkb	dhellmann: so I think the ambiguousness is git not knowing which repo's mitaka to show?	18:10
dhellmann	that could well be	18:10
dhellmann	it should be using our fancy tmp dir stuff, let me see why that isn't	18:10
dhellmann	clarkb : it does seem to be using a tmpdir: http://logs.openstack.org/7a/7ad1cd3a0da7a95fb7c14cf9eeb9ae683c247efa/release-post/tag-releases/82a4a38/console.html#_2016-09-01_15_55_54_084809	18:10
dhellmann	oh, except that temporary directory is inside the workspace	18:11
pabelanger	jeblair: I don't fully understand why. When I last tried to debug this, I considered our usage of the failed task the issue, but it might also be possible that something is async task is doing it	18:11
dhellmann	which is a git repo	18:11
*** maishsk has joined #openstack-infra		18:11
*** salv-orlando has quit IRC		18:11
clarkb	dhellmann: ya	18:11
dhellmann	ok, let me see if I can fix that	18:11
pabelanger	jeblair: but when we get unreachable=1, we still called the fail task, which leads me to think using fail is not the problem	18:11
* clarkb does a quick test		18:12
AJaeger	python experts, could you lend me a hand, please? See https://review.openstack.org/#/c/362436 - and my comment from 5:30 this morning. Why is pbr freeze showing openstackdocstheme==1.5.0 instead of something like openstackdocstheme==1.5.1.dev2 ?	18:12
jeblair	pabelanger: i'm looking for the cases where we actually get exit=3	18:12
*** waht has joined #openstack-infra		18:12
fungi	dhellmann: separate (but related) note... we did eventually get /usr/zuul-env/bin/zuul-cloner onto the signing node so that script can likely be simplified again when you're ready to hack on that	18:13
jeblair	pabelanger: and it looks like they are ones where we get an ssh error in zuul_runner, but then we also get an ssh error in the zuul_log in the rescue block....	18:13
dhellmann	fungi : ok, cool, I'll put that on the ocata list	18:13
clarkb	hrm at least my local git knows how to handle that (I am running really new git though)	18:13
AJaeger	fungi, dhellmann : But we don't have a local cache, correct?	18:13
fungi	AJaeger: correct	18:13
jeblair	pabelanger: that makes me more suspicious that the block/rescue thing is converting connection errors from inside the block into failures, unless there is also a connection failure in the rescue.	18:14
jeblair	mordred, Shrews: ^	18:14
*** senk has quit IRC		18:14
openstackgerrit	Merged openstack-infra/project-config: Add Ironic UI gerritbot to #openstack-ironic https://review.openstack.org/364347	18:15
fungi	AJaeger: it would help if we had tox logs collected from that job	18:15
pabelanger	https://github.com/ansible/ansible/blob/bd68c324cebce599ff07d6fd90c36a224581e065/lib/ansible/plugins/connection/ssh.py#L603	18:15
pabelanger	seems to imply a problem with ssh client, 255	18:15
*** Jeffrey4l has quit IRC		18:15
AJaeger	fungi, we have - see the recheck result	18:16
*** ijw has joined #openstack-infra		18:16
fungi	AJaeger: i don't immediately see any indication of you using edit-constraints in that job	18:16
clarkb	AJaeger: I remember reviewing that change I Think :)	18:16
fungi	AJaeger: so you're probably running into http://git.openstack.org/cgit/openstack/requirements/tree/upper-constraints.txt#n196 forcing you to the constrained version?	18:16
jeblair	pabelanger, Shrews: i'd like to back-burner the ssh failures for a moment and focus on the exit code.	18:16
*** senk has joined #openstack-infra		18:16
*** tonytan4ever has joined #openstack-infra		18:16
AJaeger	fungi, http://logs.openstack.org/36/362436/4/check/gate-openstackdocstheme-api-ref/22ac35d/	18:16
pabelanger	ack	18:16
*** awayne has quit IRC		18:17
AJaeger	fungi, but I'm not using constraints in that tox.ini	18:17
*** tonytan4ever has quit IRC		18:17
jeblair	pabelanger, mordred, Shrews: i'm going to see if i can synthetically create an ssh error, and then put it in a block/rescue and see if it gets converted into a failure	18:17
AJaeger	fungi, I agree, if that repo would use constraints, then we would need edit-constraints...	18:18
openstackgerrit	Doug Hellmann proposed openstack-infra/project-config: put release temporary directories under $TMPDIR https://review.openstack.org/364489	18:18
openstackgerrit	Zara proposed openstack-infra/python-storyboardclient: Add basic docs for python client https://review.openstack.org/362878	18:18
*** openstackgerrit has quit IRC		18:18
fungi	AJaeger: yeah, it's not constraints... i see it	18:18
fungi	AJaeger: http://logs.openstack.org/36/362436/4/check/gate-openstackdocstheme-api-ref/22ac35d/tox/api-ref-1.log.txt	18:18
mrhillsman	fungi mordred clarkb pabelanger - spoke with network folks and it is possible but will take some time	18:18
AJaeger	fungi, clarkb , I run tox -e api-ref locally and got "openstackdocstheme==1.5.1.dev2 # git sha 670fbd8" in the freeze...	18:18
dhellmann	clarkb : I need to test ^^ locally but I don't have a lib that's going to trigger a stable branch requirements update	18:18
*** openstackgerrit has joined #openstack-infra		18:18
fungi	AJaeger: Collecting openstackdocstheme>=1.4.0 (from os-api-ref>=1.0.0->-r /home/jenkins/workspace/gate-openstackdocstheme-api-ref/test-requirements.txt (line 12))	18:18
mrhillsman	roll with short-term and i will respond when long-term has been implemented	18:19
clarkb	dhellmann: you should be able to just invent one and remove it from your repo when done	18:19
fungi	dhellmann: i think you uploaded right when openstackgerrit was restarting for a config update	18:19
*** nstolyarenko has joined #openstack-infra		18:19
*** senk has quit IRC		18:19
clarkb	dhellmann: maybe use a temp repo so you don't risk mixing it up with the real world	18:19
fungi	dhellmann: oh, you mean 364489	18:19
fungi	i didn't scroll back far enough, sorry	18:20
*** pvaneck has joined #openstack-infra		18:20
*** maishsk has quit IRC		18:20
AJaeger	fungi, so os-api-ref forces the downgrade? ;(	18:20
mordred	mrhillsman: awesome! thnaks!	18:20
AJaeger	fungi, how can I avoid that?	18:20
dhellmann	clarkb , fungi : I tested by re-releasing muranoclient and it did not work. I'll keep tweaking locally	18:21
*** maishsk has joined #openstack-infra		18:21
jeblair	mordred: it was shade caching images and flavors	18:21
fungi	AJaeger: when test-requirements gets installed by tox, os-api-ref>=1.0.0 gets installed depending on openstackdocstheme>=1.4.0 which triggers a download from pypi	18:21
fungi	AJaeger: looks like pip install -U may be at fault?	18:21
fungi	i honestly can't remember what our position is now on whether install_command should be pip install with -U or without	18:22
fungi	http://git.openstack.org/cgit/openstack/openstackdocstheme/tree/tox.ini#n7	18:22
* AJaeger removes -U and will test that		18:23
openstackgerrit	Emilien Macchi proposed openstack-infra/tripleo-ci: pingtest: run 'openstack stack failures list' when failure https://review.openstack.org/363918	18:23
AJaeger	thanks, fungi	18:23
dhellmann	clarkb, fungi : I found it. I was being bone-headed.	18:23
*** piet has joined #openstack-infra		18:24
fungi	AJaeger: if we want to consider our current cookiecutter templates to be our official advice, then http://git.openstack.org/cgit/openstack-dev/cookiecutter/tree/%7b%7bcookiecutter.repo_name%7d%7d/tox.ini#n8 does not add -U	18:24
fungi	AJaeger: and you're right to be confused, this is a lot of voodoo	18:25
AJaeger	a grep over openstack namespace shows 400 repos with -U and 300 without ;(	18:25
AJaeger	thanks, fungi	18:25
AJaeger	I pushed https://review.openstack.org/364492 and will check it tomorrow.	18:25
clarkb	the erason we had/have -U is pre constraints you needed it to get the requirements updated if things otherwise fit into the reqs ranges	18:25
clarkb	with constraints you will install the constraints version regardless every single time	18:25
clarkb	The small gap without constraints and without -U is setuptools I think. But updating it during the main install run doesn't actually use the new version of setuptools so thats mostly a noop	18:26
AJaeger	ah. Perhaps I should use constraints instead ;)	18:26
*** mtanino__ has joined #openstack-infra		18:26
fungi	well, also we had it to deal with the bad old days where lots of projects were using system site-packages in their tox virtualenvs and we had nodes with a bunch of crufty old distro-packaged python libs installed on them	18:26
AJaeger	thanks, clarkb	18:26
openstackgerrit	Monty Taylor proposed openstack/os-client-config: Split auth plugin loading into its own method https://review.openstack.org/364493	18:27
openstackgerrit	Monty Taylor proposed openstack/os-client-config: Remove validate_auth_ksc https://review.openstack.org/364494	18:27
fungi	and so without -U we'd end up using ancient versions of dependencies from the distro packages	18:27
*** mtanino has quit IRC		18:27
*** abregman\|mtg is now known as abregman		18:28
* AJaeger congratulates Zara on absolving the RST master class ;)		18:28
mordred	jeblair: thank you!	18:28
dhellmann	fungi : since the issue is in my cloning function, I'm going to switch it to use zuul-cloner	18:29
AJaeger	fungi, it did not help, see http://logs.openstack.org/92/364492/1/check/gate-openstackdocstheme-api-ref/72a739d/console.html	18:29
AJaeger	http://logs.openstack.org/92/364492/1/check/gate-openstackdocstheme-api-ref/72a739d/tox/api-ref-2.log.txt - is there a version screwup?	18:30
fungi	AJaeger: yeah, i see it definitely still pulling in an openstackdocstheme 1.5.0 wheel	18:31
*** mtanino has joined #openstack-infra		18:31
*** harlowja_ has quit IRC		18:31
*** harlowja has joined #openstack-infra		18:32
*** _nadya_ has joined #openstack-infra		18:32
*** mtanino__ has quit IRC		18:33
AJaeger	yep ;/	18:33
openstackgerrit	Doug Hellmann proposed openstack-infra/project-config: fix branch handling in clone_repo https://review.openstack.org/364497	18:34
*** nstolyarenko has quit IRC		18:34
*** ddieterly[away] is now known as ddieterly		18:34
dhellmann	fungi, clarkb : ok, that should do it ^^	18:34
fungi	AJaeger: weird, it does look like it later installs the git checkout into the venv claiming to be 1.5.1.dev2 but then immediately after that reports "Successfully installed openstackdocstheme-1.5.0"	18:34
*** _nadya_ has quit IRC		18:34
*** ijw has quit IRC		18:35
Zara	AJaeger: =D thanks!	18:35
*** Thelo_ has quit IRC		18:35
* AJaeger tries now constraints...		18:35
fungi	AJaeger: to use constraints, you'll almost certainly need to add an edit-constraints call since this is a repo listed in upper-constraints.txt	18:36
AJaeger	fungi,done already - https://review.openstack.org/364499	18:36
*** akshai has joined #openstack-infra		18:37
*** nstolyarenko has joined #openstack-infra		18:37
fungi	AJaeger: i wonder if you just need tox.skipdist=True like at http://git.openstack.org/cgit/openstack-dev/cookiecutter/tree/%7b%7bcookiecutter.repo_name%7d%7d/tox.ini#n4	18:38
fungi	iirc there are some odd interactions between skipdist and usedevelop	18:38
*** annegentle has quit IRC		18:39
fungi	mordred or dhellmann probably remember more clearly	18:39
*** shardy is now known as shardy_afk		18:40
dhellmann	I think we usually set both of those, but I think just because building the dist is a waste of time if you're not going to install from it	18:40
mordred	I do not - but I do know that the general intent is to set both	18:41
mordred	yah	18:41
AJaeger	ok, I'll try - thanks.	18:41
mordred	setting both is good for speed when running tox. it's bad for verifying that sdist works, but the tradeoff in the past seemed ot be that devs screaming about tox taking too long trumped the things we might learn from building the sdist every time	18:41
clarkb	and then peopl started using symlinks	18:42
fungi	something definitely seems to be causing tox to force it to the (cached?) wheel rather than the git checkout	18:42
*** salv-orl_ has quit IRC		18:42
*** nstolyarenko has quit IRC		18:43
fungi	mordred: the very end of the tox log here is especially confusing... http://logs.openstack.org/92/364492/1/check/gate-openstackdocstheme-api-ref/72a739d/tox/api-ref-2.log.txt	18:43
*** annegentle has joined #openstack-infra		18:43
mordred	fungi: WOW	18:44
jeblair	mordred, pabelanger, Shrews: i have confirmed with synthetic testing that it's neither the block/rescue, nor zuul_runner that's causing the ssh errors to be failures (exit code 1) rather than unreachable errors (exit 3). it seems to be a behavior of the async module -- if it can't connect at the start, it's 'unreachable'. if it can't connect for one of it's poll checks, it's a failure.	18:44
*** kzaitsev_ws has joined #openstack-infra		18:45
fungi	mordred: if you back up to api-ref-1.log.txt you'll see it previously pulled in a wheel of 1.5.0 because there's a circular (test) dependency of openstackdocstheme on itself (via os-api-ref)	18:45
fungi	so there will be a 1.5.0 wheel in the cache at that point	18:46
mordred	fungi: but why would that matter :(	18:46
fungi	mordred: i don't think it should	18:46
*** Thelo_ has joined #openstack-infra		18:46
fungi	i'm just stretching for odd corner cases that might be exposing a bug we don't normally see	18:46
mordred	fungi: oh - totally - sorry, it was a rhetorical head-against-desk question	18:47
fungi	yeah, no idea whether that has anything to do with the problem, but obviously pip shouldn't be resolving setup_requires for 1.5.1.dev2 and then end by claiming to have installed 1.5.0	18:48
*** annegent_ has joined #openstack-infra		18:49
*** david-lyle has quit IRC		18:49
AJaeger	still wrong ;( http://logs.openstack.org/92/364492/2/check/gate-openstackdocstheme-pep8-ubuntu-xenial/66f399e/	18:49
fungi	er, install_requires i guess	18:49
*** ijw has joined #openstack-infra		18:49
*** hasharAway is now known as hashar		18:49
fungi	AJaeger: was that after switching to constraints, or adding skipdist=true?	18:50
mordred	OH	18:50
AJaeger	Oh, I might have a typo...	18:50
mordred	constraints	18:50
AJaeger	skipdist or skipsdist?	18:50
* AJaeger has two changes ;)		18:50
fungi	mordred: no constraints at play in the log i linked for you	18:50
*** david-lyle has joined #openstack-infra		18:50
mordred	if you do "pip install -c upper-constraints.txt -e . "	18:50
mordred	oh	18:50
mordred	darn	18:50
openstackgerrit	Monty Taylor proposed openstack-infra/shade: Batch calls to list_floating_ips https://review.openstack.org/364508	18:51
jeblair	pabelanger, sdague: i believe we might be able to see a little more of what ansible is doing with the ssh connections and perhaps ascertain the likelihood of a connection timeout increase being effective if we run with verbose logs for a bit. i will enable that on one of the launchers and see if we can catch an error before we run out of disk.	18:51
AJaeger	fungi, skipsdist is it, isn't it?	18:51
openstackgerrit	Monty Taylor proposed openstack-infra/nodepool: Unsubvert image and flavor caching https://review.openstack.org/364510	18:51
AJaeger	that's what cookiecutter uses	18:51
fungi	mordred: you can see at the top of the log it only ran `pip install -e .`	18:51
mordred	fungi: yah. that's excessively weird	18:51
AJaeger	I'm talking about https://review.openstack.org/364492	18:51
mordred	fungi: I kind of want to invoke our friendly pip human	18:52
*** annegentle has quit IRC		18:52
fungi	AJaeger: sorry, yes, seems it's "skipsdist = True" and i never noticed	18:52
AJaeger	no worries, fungi	18:52
jeblair	pabelanger: actually, i'm going to do it on all the launchers to try to catch this faster	18:53
*** waht has quit IRC		18:54
*** ddieterly has quit IRC		18:55
*** abregman has quit IRC		18:56
*** mriedem has quit IRC		18:56
*** asettle has joined #openstack-infra		18:56
pabelanger	jeblair: okay	18:56
*** akshai has quit IRC		18:57
AJaeger	using constraints: http://logs.openstack.org/99/364499/3/check/gate-openstackdocstheme-pep8-ubuntu-xenial/367d855/	18:57
AJaeger	"openstackdocstheme==1.5.1.dev3 # git sha f3782e1"	18:58
AJaeger	that looks finally ok...	18:58
fungi	AJaeger: i wonder if we have some sort of strange versioning going on. checkout the tip of master and run `tox -re venv pip list`	18:58
AJaeger	will do	18:58
*** kien-ha has quit IRC		18:59
fungi	git describe says i've got a commit newer than 1.5.0 but pip list is saying "openstackdocstheme (1.5.0, /home/fungi/work/openstack/openstack/openstackdocstheme/.tox/venv/lib/python2.7/site-packages)"	18:59
AJaeger	fungi, http://paste.openstack.org/show/565858/	18:59
fungi	AJaeger: was that with constraints or no?	18:59
AJaeger	that was master branch	19:00
fungi	weird. that's not at all what i'm getting	19:00
AJaeger	clean tree	19:00
*** yaume_ has quit IRC		19:00
AJaeger	stranger and stranger ;(	19:00
* AJaeger double checks the tree		19:00
AJaeger	fungi, you have an old tree - we released 1.5.1	19:01
fungi	AJaeger: http://paste.openstack.org/show/565860	19:01
*** rbrndt has joined #openstack-infra		19:01
AJaeger	wrong, other project. 1.5.0 is last tag.	19:02
fungi	AJaeger: strangely, remote update isn't picking it up for me	19:02
AJaeger	Sorry, mixed up releases	19:02
AJaeger	you have pip 8.1.2, I use 7.1.2	19:02
*** kushal has quit IRC		19:03
clarkb	its noon already? I haven't even gotten through email :/ where did the day go	19:03
fungi	AJaeger: yeah, i'm using tox 2.3.1 and virtualenv 15.0.3	19:04
pabelanger	clarkb: removing After=network.target from glean, openssh-server just reload, not stop / start	19:04
pabelanger	clarkb: I am going to build an image and see if things still work	19:04
clarkb	pabelanger: I think it can still happen after if you don't have an explicit before	19:04
clarkb	pabelanger: I think it might be better to have an explicit Before sshd	19:04
AJaeger	our test results show pip 8.1.2 and tox 2.3.1 in the actual job	19:05
*** ddieterly has joined #openstack-infra		19:05
clarkb	pabelanger: buit good to know we can manipulate glean's unit file to have it not stop start	19:05
pabelanger	clarkb: sure, we can do that too	19:05
pabelanger	clarkb: also removing After=network.target fixes the dependency cycle that systemd complains about	19:05
fungi	AJaeger: right, i'm starting to wonder if this is a regression in virtualenv/pip/tox somewhere	19:05
clarkb	pabelanger: huh, wasn't that what we had to add to the urandom fixer unit file to make ti work?	19:06
AJaeger	it works with constraints - but I didn't want to enable those here ;)	19:06
pabelanger	clarkb: FWIW: we do have Before=network-pre.target, which should protect before ssh starts	19:06
fungi	AJaeger: yeah, i think there's a bug... i'm going to try to bisect a few tools	19:06
pabelanger	clarkb: yes, and now I thinking more about it, it is also wrong. So we need to patch both	19:06
AJaeger	fungi, thanks a lot!	19:07
fungi	AJaeger: downgrading to virtualenv 14.0.2 gets me the (presumably proper) behavior you're seeing locally	19:08
clarkb	pabelanger: aha gotcha	19:08
AJaeger	cool, fungi.	19:08
fungi	AJaeger: and 15.0.0 has the broken behavior	19:08
AJaeger	So, wouldn't this cause quite some havoc in the gate? We could test wrong things here ;(	19:09
openstackgerrit	Paul Belanger proposed openstack-infra/glean: Remove After=network.target dependency cycle https://review.openstack.org/364516	19:09
pabelanger	clarkb: ^	19:09
pabelanger	like I said, going to do a quick build and test the image	19:10
clarkb	pabelanger: cool I will try to test a local build of that too	19:10
AJaeger	fungi, thanks a lot for your great help with this! clarkb, mordred, others: Thanks as well!	19:10
clarkb	pabelanger: do you know if there is an easy way to make simple-init in dib pull that version of glean?	19:10
pabelanger	clarkb: not sure, haven't tried	19:10
AJaeger	fungi, will you file a bug? Should we block that virtualenv version?	19:10
mordred	clarkb: uhm ... I think so	19:11
AJaeger	Sorry, I have to leave in a few minutes and call it a day ;(	19:11
clarkb	hrm except my lcoal virsh thing appares to have been broken by an update \o/	19:11
clarkb	qemu-bridge-helper where have you gone	19:11
mordred	greghaynes: ^^ how do we build with dib using simple-init from local source dir?	19:11
*** ilyashakhat_mobi has joined #openstack-infra		19:12
greghaynes	export DIB_REPOLOCATION_glean=/path/to/glean	19:12
mordred	clarkb: see - greghaynes continues to be magical pony	19:12
openstackgerrit	Paul Belanger proposed openstack-infra/project-config: Remove After=network.target from initialize-urandom service https://review.openstack.org/364517	19:12
*** gyee has quit IRC		19:13
pabelanger	clarkb: and urandom fix^	19:13
mordred	pabelanger: nice catch, btw	19:13
clarkb	pabelanger: cool I am going to attempt to get my local virt setup working again so I can test an image with both of those things in it	19:13
greghaynes	mordred: clarkb one caveat is I think that will grab master of that repo, so you might also want to set DIB_REPOREF_glean=some_ref	19:13
*** harlowja_ has joined #openstack-infra		19:13
pabelanger	mordred: Ya, finally getting to optimize our nodepool launches.	19:14
*** sarob has joined #openstack-infra		19:14
*** sarob has quit IRC		19:15
*** tonytan4ever has joined #openstack-infra		19:16
* AJaeger waves good bye		19:16
*** tonytan4ever has quit IRC		19:16
*** harlowja has quit IRC		19:17
*** eggshell has joined #openstack-infra		19:18
fungi	have a good evening AJaeger	19:18
openstackgerrit	greghaynes proposed openstack-infra/irc-meetings: Add diskimage-builder meeting agenda https://review.openstack.org/364519	19:19
openstackgerrit	Monty Taylor proposed openstack-infra/nodepool: Unsubvert image and flavor caching https://review.openstack.org/364510	19:19
*** sarob has joined #openstack-infra		19:21
*** waht has joined #openstack-infra		19:21
rcarrillocruz	Shrews: sigh, my wife had split shift today , could not attend ansible testing meeting	19:22
rcarrillocruz	i'll check chat logs	19:22
*** _sarob has joined #openstack-infra		19:22
rcarrillocruz	nikhil: that's awesome news, we'll keep an eye to implement that in shade + ansible when the refactor happens :-)	19:22
sdague	jeblair: cool	19:22
sdague	jeblair: it would also be good if the ansible logs got into elastic search	19:23
sdague	to help look for patterns	19:23
*** salv-orlando has joined #openstack-infra		19:23
nikhil	rcarrillocruz: aye	19:25
*** spzala has quit IRC		19:25
*** sarob has quit IRC		19:25
*** spzala has joined #openstack-infra		19:26
fungi	urgh, something in one of the virtualenv 14.0.x/pip 6.0.x versions horked up my wheel cache	19:29
fungi	this makes bisection decidedly more complicated	19:29
fungi	er, pip 8.0.x i mean	19:30
*** ijw has quit IRC		19:31
*** ijw has joined #openstack-infra		19:32
fungi	okay, so the problem seems to have crept in between virtualenv 14.0.5 and 14.0.6	19:32
*** annegent_ has quit IRC		19:32
fungi	the release history for 14.0.6 includes "Upgrade setuptools to 20.0" and "Upgrade wheel to 0.29.0"	19:34
*** _nadya_ has joined #openstack-infra		19:35
fungi	so we can consider this as probably either a regression between setuptools 19.6.2 and 20.0 or wheel 0.26.0 and 0.29.0	19:36
*** tphummel has quit IRC		19:37
*** ijw has quit IRC		19:37
*** sdague has quit IRC		19:37
*** _nadya_ has quit IRC		19:40
*** ddieterly is now known as ddieterly[away]		19:41
*** vhosakot has quit IRC		19:41
fungi	though interestingly, virtualenv 14.0.6 causes me to end up with setuptools 26.1.1 in the resulting venv	19:42
*** vhosakot has joined #openstack-infra		19:42
fungi	scarily, the official documentation for setuptools only has up through 25.1.3 in their included changelog	19:43
*** nstolyarenko has joined #openstack-infra		19:43
fungi	oh, good, the CHANGES.rst in their git repo is up to date at least	19:44
*** tongli has quit IRC		19:44
mat128	I have the answer to that question	19:45
mat128	fungi: ^	19:45
mat128	fungi: for some reason, virtualenv (trying to find it) installs a "bundled" pip with no version cap	19:45
mat128	and if you have a newer version in your wheel cache, it's going to be used	19:45
mat128	leading to confusion and virtualenv not acting as it's supposed	19:45
mat128	I had filed a bug report, trying to find it	19:46
*** asettle has quit IRC		19:46
fungi	mat128: yeah, that's why the claimed version of setuptools bundled in virtualenv is irrelevant, agreed	19:46
zigo	clarkb: fungi: pabelanger: Can we wrap up the discussion we just had a few hours ago?	19:47
zigo	If I understand correctly, the major concern is that each image will eat up to 1.5 GB of cache data, which will globally slow down infra. Is this the only problem, or is there anything else?	19:47
zigo	Also, since things are the way they are right now, and that Newton release is approaching (technically for me, it's already released as b3, and I'm already late), can we delay switching to overlay mode for after Newton?	19:47
zigo	Last, can we decide that we will use the overlay mode only for when upstream is OpenStack?	19:47
mat128	fungi: https://github.com/pypa/virtualenv/blob/bdef7328d47f18ecf9d1df23e33ec5a039f41048/virtualenv.py#L934	19:47
*** asettle has joined #openstack-infra		19:47
mat128	if that line was changed to pip==VERSION	19:47
mat128	it'd work correctly	19:47
fungi	mat128: so unfortunately this means the problem is somewhere between pip 8.0.2 and 8.1.2 or setuptools 19.6.2 and 26.1.1 or wheel 0.26.0 and 0.29.0	19:48
mat128	in the meantime, flushing your wheel cache seems like the only way out	19:48
*** nstolyarenko has quit IRC		19:48
*** asettle has quit IRC		19:48
*** mriedem has joined #openstack-infra		19:49
pabelanger	clarkb: okay, I think I have a minimal DIB working with glean and our urandom element, just confirming now	19:49
fungi	zigo: i'm a little worried that we didn't anticipate the nature of the repos you were importing, and that they already include a fair amount of git history from the corresponding upstream project repos. cleaning that up is likely to be complicated	19:50
clarkb	pabelanger: ok I just got my virsh issue sorted out	19:50
zigo	fungi: That's the case indeed.	19:50
fungi	e.g. http://git.openstack.org/cgit/openstack/deb-nova/tree/?h=debian%2Fnewton looks like it's not just the debian directory but rather an entire nova source code tree	19:50
zigo	fungi: We already imported everything.	19:50
clarkb	pabelanger: but gonna try to get the buiod started then go eat lunch	19:51
zigo	fungi: Also, I'm really not sure how I will do for keeping the packaging history and changing the hosted format ... :/	19:51
zigo	fungi: I really need answers for my above concerns ASAP. Time is running... :/	19:52
fungi	zigo: yeah, i'm inclined at this point to just go along with what you want because the bulk of the damage is already done and i don't have a good answer for how to go back and fix it	19:52
zigo	fungi: So, would you agree that we delay the transition for after the Newton release?	19:53
fungi	which would probably involve git filter-branch to trim out everything except the debian subtree and forcve-pushing teh result back over the existing repos	19:53
zigo	(if we decide to do so...)	19:53
fungi	zigo: yeah, in the near term merging a few more upstream tags isn't going to make the existing situation considerably worse	19:53
openstackgerrit	David Shrewsbury proposed openstack-infra/nodepool: Replace watch thread with periodic thread https://review.openstack.org/363217	19:53
zigo	fungi: In such case, how can I get the ACL to do merge commits?	19:54
*** Goneri has quit IRC		19:54
openstackgerrit	David Shrewsbury proposed openstack-infra/nodepool: Replace watch thread with periodic thread https://review.openstack.org/363217	19:54
fungi	zigo: we have it documented at http://docs.openstack.org/infra/manual/drivers.html#merge-commits	19:55
*** jcoufal_ has quit IRC		19:56
*** abregman has joined #openstack-infra		19:57
*** asettle has joined #openstack-infra		19:59
*** asettle has quit IRC		19:59
rcarrillocruz	pabelanger: oh, we have cirros or some other image in our mirrors?	20:00
rcarrillocruz	or which file you refer to	20:00
rcarrillocruz	?	20:00
clarkb	pabelanger: ok my build is started	20:01
*** _nadya_ has joined #openstack-infra		20:01
clarkb	will have to see how that goes	20:01
pabelanger	rcarrillocruz: ~/cache/files is what I was referring too	20:01
pabelanger	clarkb: rebuilding, I messed up the glean variable	20:01
clarkb	gonna grab lunch while its going	20:01
clarkb	pabelanger: what vars do you think you need? I used ehat greghaynes said	20:02
openstackgerrit	Sagi Shnaidman proposed openstack-infra/tripleo-ci: Use centos CDN repository with periodic jobs https://review.openstack.org/364534	20:02
pabelanger	clarkb: Ya, I did use it, I had a typo in my path	20:02
*** piet has quit IRC		20:02
clarkb	ah	20:02
openstackgerrit	Guillaume Espanel proposed openstack-infra/project-config: Create puppet-cloudkitty repository https://review.openstack.org/364535	20:03
*** florianf has quit IRC		20:03
fungi	confirmed the pip cache is definitely playing a part in this behavior. testing with various versions i got it into a state where older virtualenv releases that had been working for me started exhibiting the problem behavior until i blew away ~/.cache/pip	20:04
*** ijw has joined #openstack-infra		20:04
mat128	fungi: I tried very hard to find the bug report I remember submitting, but can't find anything	20:04
*** ddieterly[away] is now known as ddieterly		20:04
mat128	fungi: must have had bad dreams about virtualenv and pip.. can't find any trace of my experiment either :(	20:05
*** annegentle has joined #openstack-infra		20:05
mat128	fungi: if you find a way to disable the wheel cache (env variable or smt), that could help as well	20:05
fungi	mat128: i'm less concerned with that to be honest. the current and more troubling issue is that starting with virtualenv 14.0.6 we seem to be installing cached wheels of downloaded releases when we ask it to pip install the current checked out source tree	20:05
mat128	fungi: are we using explicit versions?	20:06
fungi	i think it's something after wheel 0.26.0 but i'm still narrowing it down	20:06
*** derekh has joined #openstack-infra		20:06
openstackgerrit	Thomas Goirand proposed openstack-infra/project-config: Add merge commit ACL for packaging-deb https://review.openstack.org/364537	20:06
zigo	fungi: Like this? ^	20:06
mat128	fungi: can you reproduce it easily?	20:06
fungi	mat128: explicit versions of what?	20:06
mat128	fungi: pip install package==1.0.0	20:06
mat128	or constraints	20:06
fungi	mat128: absolutely, i'm whittling down versions of things in the toolchain to narrow down whether it's virtualenv, pip, wheel or setuptools at fault	20:06
zigo	(minus the english mistakes in the commit header... :( )	20:06
fungi	mat128: no constraints in use in this case	20:07
mat128	fungi: so let me try to understand the issue: you have software==1.1 in your wheel cache, you issue pip install software==1.0 and 1.1 gets installed?	20:07
fungi	mat128: the short version is demonstrated in http://logs.openstack.org/92/364492/1/check/gate-openstackdocstheme-api-ref/72a739d/tox/api-ref-1.log.txt and http://logs.openstack.org/92/364492/1/check/gate-openstackdocstheme-api-ref/72a739d/tox/api-ref-2.log.txt	20:08
*** _nadya_ has quit IRC		20:08
mat128	ouch	20:08
fungi	mat128: it does a pip install -r test-requirements.txt where one of the packages listed there is a circular dependency back on the project being tested and so pulls down the packaged release, then does a pip install -e . and ends up i think installing the cached wheel instead of the current source tree	20:09
*** ijw has quit IRC		20:09
mat128	what about ``pip install -U -e .``	20:09
mat128	?	20:09
fungi	not sure yet, i'm less concerned with workarounds and more with bisecting to a specific version in the toolchain that introduces this regression	20:10
fungi	it's just a little fiddly since some of teh parts are vendored so there's some chicken-and-egg problems pinning them independently	20:11
*** tonytan4ever has joined #openstack-infra		20:11
*** coolsvap has quit IRC		20:12
*** flepied has quit IRC		20:12
fungi	short answer for a reproducer is to clone the openstack/openstackdocstheme repo and then `tox -er venv pip list` and look at the version of openstackdocstheme it reports installed	20:12
fungi	comparing virtualenv 14.0.5 and 14.0.6 behavior (or 14.0.5 and latest release for that matter)	20:12
*** gyee has joined #openstack-infra		20:13
mat128	openstackdocstheme (1.5.0, /Users/mmitchell/projects/openstackdocstheme/.tox/venv/lib/python2.7/site-packages)	20:13
rcarrillocruz	pabelanger: oh nice, did not know we had cirros at cache/files, thanks!	20:13
fungi	as noted, i've had to introduce a `rm -rf ~/.cache/pip` into the loop because of the wheel cache from other versions influencing the test result	20:14
*** kzaitsev_mb has quit IRC		20:14
openstackgerrit	James E. Blair proposed openstack-infra/infra-specs: Zuulv3: drop variable interpolation and add nodesets https://review.openstack.org/361463	20:14
mat128	fungi: pip 8.1.2, virtualenv 15.0.2	20:15
fungi	i think i've mostly ruled it out as a regression in wheel. seems more like setuptools but i'm not sure where in its history this crops up yet	20:15
mat128	oh, virtualenv 14	20:15
*** maishsk has quit IRC		20:15
jeblair	fungi: ^ fixed the merge conflict on that, should be ready for merging when fires are out. or maybe just smoldering.	20:15
*** fguillot has quit IRC		20:16
fungi	jeblair: oh, cool!	20:16
*** tonytan_brb has joined #openstack-infra		20:16
ianw	has the idea of zuul-cloner being a separate thing to zuul been covered before?	20:16
*** maishsk has joined #openstack-infra		20:16
fungi	ianw: we've talked about breaking out the cli tools, but the arguments for doing so have been fairly shallow	20:16
*** tonytan4ever has quit IRC		20:17
fungi	mat128: getting closer... i think it's around setuptools 25	20:17
*** esikache1 has quit IRC		20:18
*** spzala has quit IRC		20:19
*** markusry has quit IRC		20:20
fungi	mat128: bingo... put setuptools==24.3.1 in requirements.txt and compare against 25.0.0	20:20
*** flepied has joined #openstack-infra		20:21
*** kzaitsev_mb has joined #openstack-infra		20:21
fungi	i wonder if SETUPTOOLS_SYS_PATH_TECHNIQUE=rewrite changes this	20:21
mat128	fungi: confirmed here too	20:22
mat128	http://paste.openstack.org/show/565887/	20:22
mat128	fungi: yes, that fixes it	20:23
mat128	http://paste.openstack.org/show/565888/	20:23
fungi	ugh	20:23
fungi	"This project hopes that that few if any environments find it necessary to retain the old behavior, and intends to drop support for it altogether in a future release. Please report any relevant concerns in the ticket for this change."	20:23
fungi	i guess we have one :/	20:24
mat128	installing a project from source doesnt seem like a rare case	20:24
fungi	indeed	20:24
fungi	this is probably a pip -e case	20:24
fungi	i'm going to fiddle a little more to see if usedevelop=False changes things	20:24
mat128	fungi: look at the paths	20:25
openstackgerrit	Monty Taylor proposed openstack-infra/shade: Batch calls to list_floating_ips https://review.openstack.org/364508	20:25
mat128	http://paste.openstack.org/show/565887/	20:25
mat128	fungi: one is from site-packages (as you said, pulled via dep resolution)	20:25
fungi	yep	20:25
*** dprince has quit IRC		20:26
*** ijw has joined #openstack-infra		20:26
clarkb	pabelanger: my local.instance never dhcped	20:26
fungi	okay, so usedevelop=false does indeed cause the problem to go away	20:26
fungi	which means setuptools 25.0.0 basically broke editable installs	20:27
*** pvaneck has quit IRC		20:27
clarkb	pabelanger: so rebuilding to set apassword to logij on console	20:27
*** nmagnezi_ has joined #openstack-infra		20:27
*** dtardivel has quit IRC		20:27
clarkb	pabelanger: I think thr ordering for glean might be more subtle	20:27
fungi	mat128: i'll update the bug mentioned in the changelog and see where that gets us	20:27
mat128	fungi: yeah, probably our best bet	20:28
*** ddieterly is now known as ddieterly[away]		20:28
*** eggshell has quit IRC		20:28
mat128	fungi: and the environemnt variable is our temporary workaround	20:28
pabelanger	clarkb: yup	20:28
*** nmagnezi has quit IRC		20:28
fungi	https://github.com/pypa/setuptools/issues/674 for the record	20:28
fungi	mat128: i'll come up with a reproducer that doesn't rely on tox, now that i know what the problem is	20:29
*** e0ne has quit IRC		20:30
*** gordc has quit IRC		20:30
*** ddieterly[away] is now known as ddieterly		20:30
*** itisha has quit IRC		20:30
*** ijw has quit IRC		20:31
lifeless	fungi: yay quagmire	20:32
lifeless	fungi: lets all switch to rust :)	20:32
mordred	lifeless: if only	20:32
prometheanfire	someone mind looking at this for the release process? https://review.openstack.org/#/c/364497	20:32
prometheanfire	it'd help with getting the bot submitted reviews for target branches working for requirements	20:33
openstackgerrit	David Shrewsbury proposed openstack-infra/nodepool: Replace watch thread with periodic thread https://review.openstack.org/363217	20:34
*** gouthamr has quit IRC		20:34
openstackgerrit	Doug Hellmann proposed openstack-infra/project-config: do not error if a branch already exists https://review.openstack.org/364544	20:36
fungi	lifeless: indeed. who needs editable installs anyway? ;)	20:37
*** annegentle has quit IRC		20:39
*** yfried has joined #openstack-infra		20:40
*** ijw has joined #openstack-infra		20:41
mrhillsman	any thoughts on the time it will take to get some workload on cloud8?	20:41
ianw	lifeless: if around ... http://docs.openstack.org/developer/pbr/#environment-markers <- is it possible to query how you were installed with a various environments? or is the idea you would just do an import and catch exceptions?	20:43
openstackgerrit	Thomas Goirand proposed openstack-infra/project-config: Add merge commit ACL for packaging-deb https://review.openstack.org/364537	20:44
*** kgiusti has left #openstack-infra		20:44
*** ijw has quit IRC		20:46
*** derekh has quit IRC		20:47
*** annegentle has joined #openstack-infra		20:47
*** piet has joined #openstack-infra		20:49
openstackgerrit	Monty Taylor proposed openstack-infra/shade: Fix up image and flavor by name in create_server https://review.openstack.org/364548	20:49
*** maishsk has quit IRC		20:50
mordred	armax, dtroyer: neutronclient released. new os-client-config released - the gate does not seem to have completely broken ... so I think we can consider that good!	20:51
ianw	timothyb89: around? I wouldn't mind rolling stackviz cleanup into my devstack-gate cleanup-refactor, just to get it all done. want to pick your brain on what you think is supposed to be happening under regular & grenade	20:51
*** maishsk has joined #openstack-infra		20:51
armax	mordred: someone in the neutron channel said something got belly up	20:52
timothyb89	ianw: sure, happy to help	20:52
*** ijw has joined #openstack-infra		20:52
mordred	armax: oh no	20:52
*** pvaneck has joined #openstack-infra		20:52
ianw	timothyb89: so see http://logs.openstack.org/46/364046/4/check/gate-grenade-dsvm-neutron-ubuntu-trusty/f527fe1/logs/ ... that has old & new and each has their tempest run	20:53
*** baoli_ has quit IRC		20:53
ianw	timothyb89: do you expect that both would have stackviz output too?	20:53
mordred	jeblair: so ... interesting edge case for your brainhole wrt zuul	20:53
mordred	prometheanfire: ^^ (about to mention the neutron thing to jeblair)	20:54
prometheanfire	mordred: we are handling it in -release	20:54
timothyb89	ianw: ideally if a *.subunit exists, stackviz should too	20:54
mordred	prometheanfire: well, there is a thing here which is, I think, potentially worth considering on the zuul side for future development ...	20:54
prometheanfire	ah	20:55
mordred	jeblair: release automation released a new version of python-neutronclient (yay!) and then submitted a patch to global requirements bumping the constraints for the release	20:55
mordred	jeblair: the tests for that change ran before the release artifact had made it to the mirrors	20:55
clarkb	pabelanger: ok on my local instance I don't see glean or ssh ever start	20:56
mordred	jeblair: I'm mentioning it because it sounds similar to me to one of the things we've heard from distro folks - sometimes they'd like to run a job in response to an artifact being uploaded, not to a git commit	20:56
clarkb	pabelanger: now to determine if it is pebcak during build	20:56
mordred	jeblair: and this seems like a specific case of that for if/when we get around to musing about such a thing	20:56
ianw	timothyb89: so if you look at _copy_devstack_logs in -> https://review.openstack.org/#/c/364046/4/functions.sh . if stackviz is installed, can it use the "tempest.subunit" file in there, like the subunit2html file does?	20:57
ianw	timothyb89: would i just "cat tempest.subunit" \| stackviz ... ?	20:58
timothyb89	ianw: there are a few extra steps, but that would work	20:59
timothyb89	ianw: though specifically on http://logs.openstack.org/26/363326/3/check/gate-grenade-dsvm-neutron-ubuntu-trusty/05856df/logs/devstack-gate-cleanup-host.txt it looks like a permissions issue of some sort?	21:00
*** Gibi is now known as gibi		21:00
*** nmagnezi_ has quit IRC		21:01
pabelanger	clarkb: can you see glean in systemctl?	21:01
*** ddieterly is now known as ddieterly[away]		21:01
clarkb	pabelanger: no	21:01
*** raildo has quit IRC		21:01
clarkb	pabelanger: but it is installed	21:01
pabelanger	Hmm	21:01
clarkb	the command glean is installed I mean	21:01
pabelanger	just bringing my image online now	21:01
clarkb	and its definitely not configuring eth0 which means no dhcp	21:02
pabelanger	looks like glean failed to start for me too	21:02
pabelanger	same with initialize-urandom	21:02
ianw	timothyb89: there is that ... i think it's confusion between when grenade uses "new" v the top-level	21:03
*** jkilpatr has quit IRC		21:03
pabelanger	clarkb: you likely see Ordering cycle found, skipping Network	21:03
*** ddieterly[away] is now known as ddieterly		21:03
ianw	timothyb89: how big is "stackviz"? if we have old/stackviz and new/stackviz are we duplicating a lot?	21:03
jeblair	mordred: maybe the constraints update shouldn't come from the release tag job; maybe it should happen in response to the artifact upload job?	21:03
ianw	timothyb89: can you have one "stackviz" dir and put two lots of data in?	21:04
jeblair	mordred: (in other words, look at it as not being the reponsibility of the release pipeline to update constraints; look at it as the responsibility of the project to update constraints when it's released (regardless of how the release is triggered).	21:04
*** berendt has joined #openstack-infra		21:04
clarkb	pabelanger: no its more like glean just isn't enabled at all for some reason	21:04
clarkb	pabelanger: systemctl status glean says Loaded: not-found (Reason: No such file or directory)	21:06
timothyb89	ianw: stackviz can accept multiple files in one directory, yes	21:06
clarkb	pabelanger: but the unit file is in /usr/lib/systemd/system	21:06
pabelanger	clarkb: it will be glean@eth0.service I think	21:06
timothyb89	ianw: something like, `stackviz-export -f old/testrepository.subunit -f new/testrepository.subunit $log_path/stackviz/data` would work I think	21:07
mordred	jeblair: I believe that is what the release team is going to do	21:07
*** trown is now known as trown\|outtypewww		21:07
clarkb	pabelanger: there is glean@.service	21:08
clarkb	pabelanger: is your instance working?	21:08
ianw	timothyb89: great, i'll try that. will it work if i do the exports separately, or do they have to be done from the one stackviz-export call? making two calls would be easier and fit into the existing fn	21:08
*** yfried has quit IRC		21:08
pabelanger	clarkb: still figuring that out	21:08
mordred	jeblair: I was more just bringing it up as a local example of a thing people who are not in openstack have expressed wanting to be able to do	21:09
timothyb89	ianw: right now it would need 1 call, but we can always just make 2 copies like we do now if that's easier	21:10
*** rvasilets___ has joined #openstack-infra		21:10
*** eharney has quit IRC		21:10
clarkb	pabelanger: hrm I still have an After=network.target in that file so I may not have gotten glean/simple-init to install properly	21:10
timothyb89	ianw: actually, 2 separate may still be the best option so dstat is included ... the export CLI only accepts 1 dstat input right now	21:11
ianw	timothyb89: ok, cool ... let me get something together and we can see how it looks	21:12
timothyb89	ianw: sounds good!	21:12
*** kaisers_ has joined #openstack-infra		21:14
pabelanger	clarkb: ya, same. glean was installed by pip for me	21:15
openstackgerrit	Ilya Shakhat proposed openstack-infra/project-config: Rename os-failures to os-faults https://review.openstack.org/364556	21:15
*** ilyashakhat_mobi has quit IRC		21:15
*** ldnunes has quit IRC		21:16
clarkb	pabelanger: we need to set DIB_INSTALLTYPE_simple_init=repo	21:17
clarkb	pabelanger: so I rebuilding with that. But I don't know why its otherwise broken for me	21:18
clarkb	pabelanger: did glean the service run for you?	21:18
*** kaisers_ has quit IRC		21:18
*** rossella_s has quit IRC		21:18
fungi	mat128: lifeless: https://github.com/pypa/setuptools/issues/674	21:18
pabelanger	clarkb: didn't come up for me, I don't know why	21:18
clarkb	pabelanger: same problem likely then :)	21:18
fungi	dstufft: ^ that issue is probably of interest to you as well, as pip maintainer	21:18
clarkb	pabelanger: I used devuser to create a user with password then logged in on console	21:18
*** rossella_s has joined #openstack-infra		21:19
pabelanger	clarkb: ya, going to do the same	21:19
*** aeng has joined #openstack-infra		21:20
*** tphummel has joined #openstack-infra		21:21
fungi	oh, and now i find https://github.com/pypa/setuptools/issues/729 and https://github.com/pypa/setuptools/issues/447 basically already cover this	21:21
openstackgerrit	Matthew Thode proposed openstack-infra/project-config: Pause before submitting the requirements review https://review.openstack.org/364559	21:23
clarkb	pabelanger: manually running `sudo systemctl start glean@ens3` worked	21:24
* prometheanfire likes glean		21:24
*** rcernin has quit IRC		21:24
pabelanger	clarkb: okay, lets hope this DIB works	21:25
clarkb	now to figure out why it didn't fire on boot	21:25
openstackgerrit	Ian Wienand proposed openstack-infra/devstack-gate: Refactor stackviz run https://review.openstack.org/364560	21:25
pabelanger	clarkb: did you rebuild?	21:25
ianw	timothyb89: ^ that's the idea split out	21:25
clarkb	pabelanger: also there isn't au nit file for each individual interface instead the glean@ unit file is a template that takes the interface dev name as an argument	21:25
fungi	mat128: AJaeger: so in summary, https://github.com/pypa/pip/pull/3904 has already merged to fix the behavior in pip, and will likely be in pip 8.1.3 or whatever the next release is	21:25
clarkb	pabelanger: still rebuilding. But also debugging on my old broken host	21:25
*** Goneri has joined #openstack-infra		21:26
*** thorst has quit IRC		21:26
fungi	mat128: AJaeger: oh, not yet merged, i misread	21:26
clarkb	pabelanger: oh there was an ordering dependency why did that not show up in dmesg?	21:26
*** thorst has joined #openstack-infra		21:27
clarkb	pabelanger: apparently I have to ask journalctl for logs that actually matter :(	21:27
timothyb89	ianw: that looks a lot better! small issue, though, the '--end' flag copied over is actually an old typo, that should be '--env'	21:27
*** matt-borland has quit IRC		21:27
clarkb	pabelanger: so its possible the fix might actuall fix this	21:27
pabelanger	clarkb: Ya, that is the original ordering issue	21:27
pabelanger	yup	21:27
pabelanger	for some reason, the magic make it work today	21:28
clarkb	I should just alias dmesg to journalctl -xe	21:28
timothyb89	ianw: apparently that branch was never followed since it should have been spitting out errors for months... whoops	21:28
*** jerryz has quit IRC		21:28
openstackgerrit	Ian Wienand proposed openstack-infra/devstack-gate: Refactor stackviz run https://review.openstack.org/364560	21:29
*** rtheis has quit IRC		21:29
clarkb	pabelanger: I am to the qemu img convert so hopefully will have answersoon	21:29
*** pt_15 has quit IRC		21:29
ianw	timothyb89: heh, it was really really hard to follow. i think too much "organic" growth	21:29
timothyb89	ianw: that, and probably some poor testing on my part... I guess that explains why dstat hasn't been showing up, though	21:30
*** abregman has quit IRC		21:31
*** thorst has quit IRC		21:31
*** claudiub has joined #openstack-infra		21:31
*** abregman has joined #openstack-infra		21:31
*** ddieterly has quit IRC		21:31
openstackgerrit	Monty Taylor proposed openstack-infra/shade: Fix up image and flavor by name in create_server https://review.openstack.org/364548	21:32
timothyb89	ianw: for what it's worth, I'm hoping to remove the stackviz-export step entirely in the near future, hopefully just the single `sudo cp -r ...` will be required	21:32
*** gouthamr has joined #openstack-infra		21:32
clarkb	pabelanger: ok glean worked but initialize-urandom failed due to no such file or directory. I don't know which file or directory yet	21:33
*** abregman has quit IRC		21:33
pabelanger	clarkb: oh, maybe we are missing haveged	21:36
clarkb	pabelanger: oh!	21:36
clarkb	yes	21:36
pabelanger	I think that is a dependency	21:36
clarkb	its not part of the initialize-urandom element	21:36
* clarkb builds another image		21:36
pabelanger	ya, we can added it as a pkg-map	21:37
*** _sarob has quit IRC		21:37
clarkb	I just added it to infra package needs really quick totest but ya would be better in initialize-urandom	21:38
pabelanger	err, package-install.yaml	21:38
pabelanger	that works	21:38
*** shardy_afk has quit IRC		21:38
pabelanger	rebuild started.... again	21:38
clarkb	:) isn't it fun how fixing things like this ends up being	21:39
fungi	mordred: dhellmann: so... revisiting this apparently intentional behavior change in setuptools path ordering will result in future as-of-yet-unidentified modifications in pip's behavior around editable installs, can you remind me why some projects use them for tox in the first place?	21:39
pabelanger	I'm hoping at the mid-cycle we can talk about DIB elements for a bit	21:39
*** shardy_afk has joined #openstack-infra		21:40
clarkb	pabelanger: part of the problem here is the normal builds take forever and are massive due to all the caching so I try to avoid that and just build with ubuntu-minimal simple-init growroot devuser infra-package-needs (for ssh) and initialize-urandom	21:40
clarkb	but then we find where we have undeclared deps between elements	21:40
clarkb	but also undocumented flags in simple-init that need setting aren't fun either	21:41
clarkb	I should push a patch to dib now to fix that	21:41
*** vhosakot has quit IRC		21:41
pabelanger	Ya, don't want to get into it now. But there is some frustrations around it. I am sure we could make it better, with a little programming	21:42
*** adriant has joined #openstack-infra		21:42
mordred	fungi: because it shortens the iteration cycle	21:42
pabelanger	okay, will be back shortly, need to walk down to store for something	21:42
*** hashar has quit IRC		21:42
*** priteau has quit IRC		21:42
mordred	fungi: for things like nova, needing install to run after each edit before running tox can be a significant delay	21:42
mordred	fungi: the run_tests.sh that nova used before we went tox did editable installs into a virtualenv rather quickly, and the devs were annoyed by how long tox took	21:43
*** gouthamr_ has joined #openstack-infra		21:44
fungi	mordred: got it. and specifically they want to do it with tox	21:44
*** ijw has quit IRC		21:44
fungi	so we can't just take usedevelop out. and the SETUPTOOLS_SYS_PATH_TECHNIQUE envvar is targeted for future removal from setuptools	21:44
*** adriant has quit IRC		21:45
mordred	fungi: so - should I go read the bug in question?	21:45
mordred	fungi: like, why have they decided to break -e ?	21:45
fungi	it doesn't look like setuptools is interested in changing this back to the old behavior, and potential fixes have been proposed to pip but don't seem to have garnered much interest from pip maintainers	21:46
fungi	mordred: https://github.com/pypa/setuptools/issues/674 explains the reason for the behavior change	21:46
openstackgerrit	Clark Boylan proposed openstack/diskimage-builder: Document source glean installs in simple-init https://review.openstack.org/364568	21:46
mordred	fungi: cool	21:46
clarkb	greghaynes: ianw ^	21:46
*** gouthamr has quit IRC		21:47
cinerama	clarkb: thanks!	21:47
mordred	fungi: it seems like what we _really_ want to happen in tox	21:47
fungi	mordred: so anyway, i think we have to consider the possibility that usedevelop=true in tox.ini is simply unsafe (and certainly currently broken with latest pip/setuptools)	21:47
mordred	fungi: is the sequence of "pip install -e . ; pip install -r test-requirements.txt"	21:48
fungi	mordred: other way around	21:48
mordred	fungi: since that should get the -e install to satisfy the transitive/circular dep	21:48
*** Thelo_ has quit IRC		21:48
fungi	mordred: basically tox runs your install_command first, then installs the local tree	21:48
*** inc0 has quit IRC		21:49
fungi	so it's in some cases getting a packaged version of whatever we're testing pulled in, and then does the editable install of the local source tree after	21:49
clarkb	pbr has this behavior	21:49
clarkb	we had a workaround in it involving put . in the deps list or something	21:50
clarkb	(since pbr bootstraps itself)	21:50
*** yamahata has quit IRC		21:50
mordred	yah	21:51
mordred	deps = .	21:51
mordred	-r{toxinidir}/test-requirements.txt	21:51
ianw	clarkb: lg, it might be helpful to give an example of using a upstream review and getting the ref from there. i've manually talked people through that a couple of times (should have taken the time to document it :)	21:52
*** ddieterly has joined #openstack-infra		21:53
*** cardeois has quit IRC		21:56
clarkb	pabelanger: http://paste.openstack.org/show/565895/ looks ok to me it still reloads (and seems to start with a "I shut down ok" message) but I think thats all fine	21:56
clarkb	ianw: oh thats a good idea ya I can make it more verbose	21:56
*** thorst has joined #openstack-infra		21:57
*** Goneri has quit IRC		21:57
*** berendt has quit IRC		21:58
clarkb	pabelanger: also initialize-urandom and glean both ran before the ssh things started	21:58
clarkb	(I also really don't like that its ssh not sshd such muscle memory)	21:59
*** adriant has joined #openstack-infra		21:59
clarkb	pabelanger: you good with me approving the glean change, then we need a release before we can approve the urandom one	22:00
*** thorst has quit IRC		22:01
*** yamahata has joined #openstack-infra		22:02
openstackgerrit	Clark Boylan proposed openstack/diskimage-builder: Document source glean installs in simple-init https://review.openstack.org/364568	22:03
clarkb	ianw: ^ is that better?	22:03
*** ijw has joined #openstack-infra		22:04
*** fguillot has joined #openstack-infra		22:05
*** javeriak has joined #openstack-infra		22:06
*** Julien-zte has quit IRC		22:07
*** rlandy is now known as rlandy\|bbl		22:07
ianw	clarkb: cool. you can even do it directly from the review, but people can probably figure that out	22:08
ianw	i mean upstream git. the hardest part is finding the pull-down in the top-right corner of gerrit ui	22:08
*** javeriak_ has quit IRC		22:09
*** Julien-zte has joined #openstack-infra		22:09
clarkb	ianw: I hope my example also shows how to test local edits by explaining it that way	22:09
clarkb	eg before pushing to gerrit	22:10
rcarrillocruz	pabelanger: mind reviewing https://review.openstack.org/#/c/364397/ pls? it pulls from file now	22:10
*** Swami has quit IRC		22:11
*** rlandy\|bbl is now known as rlandy		22:11
*** rlandy is now known as rlandy\|bbl		22:12
openstackgerrit	Merged openstack-infra/shade: Batch calls to list_floating_ips https://review.openstack.org/364508	22:12
*** vhosakot has joined #openstack-infra		22:13
*** esberglu has quit IRC		22:14
*** tphummel has quit IRC		22:14
*** xyang1 has quit IRC		22:14
*** krotscheck has quit IRC		22:15
*** krotscheck has joined #openstack-infra		22:15
clarkb	mordred: what are your thoughts on making a glean release nowish?	22:15
clarkb	mordred: we will need to coordinate that and the fix for initialize-urandom because if we don't then boot doesn't work	22:16
*** ilyashakhat_mobi has joined #openstack-infra		22:16
clarkb	mordred: glean can't bring up interfaces properly if we run the current glean against the initialize-urandom fix	22:16
clarkb	its good fun	22:16
*** ddieterly is now known as ddieterly[away]		22:17
*** javeriak_ has joined #openstack-infra		22:17
*** javeriak has quit IRC		22:17
*** Thelo_ has joined #openstack-infra		22:17
*** Thelo_ has quit IRC		22:19
*** tphummel has joined #openstack-infra		22:19
*** spzala has joined #openstack-infra		22:20
*** piet has quit IRC		22:20
*** ilyashakhat_mobi has quit IRC		22:21
mordred	clarkb: oh lovely	22:21
mordred	clarkb: well, I am in support of releasing glean anytime you think is good	22:21
openstackgerrit	Ian Wienand proposed openstack-infra/devstack-gate: Refactor stackviz run https://review.openstack.org/364560	22:22
clarkb	mordred: I want pabelanger to confirm his test build functioned then I think we can get both things in and glean released	22:22
mordred	++	22:22
clarkb	dtroyer: stevemar did that osc neutron https thing get merged and released yet? I can't find the bug anymore and I fail at googling	22:22
ianw	i don't want to mess things up, but i have a bunch of glean stuff just sitting -> https://review.openstack.org/#/q/status:open+project:openstack-infra/glean+branch:master+topic:mock	22:22
pabelanger	clarkb: back	22:23
ianw	mostly test cleanups	22:23
mordred	ianw: two of those are merge-conflict now	22:23
mordred	:(	22:23
pabelanger	let me test quickly	22:23
pabelanger	build is done	22:23
jhesketh	Morning	22:23
*** mdrabe has quit IRC		22:24
openstackgerrit	Ian Wienand proposed openstack-infra/glean: Refactor side-effect functions https://review.openstack.org/320225	22:24
openstackgerrit	Ian Wienand proposed openstack-infra/glean: Patch glean.cmd.open for unit testing https://review.openstack.org/320224	22:24
openstackgerrit	Ian Wienand proposed openstack-infra/glean: Use mock patching for most functions https://review.openstack.org/318464	22:24
openstackgerrit	Ian Wienand proposed openstack-infra/glean: Add selinux context manager for writing files https://review.openstack.org/304357	22:24
*** spzala has quit IRC		22:24
mordred	clarkb: you have opinons on https://review.openstack.org/#/c/318464 ?	22:24
clarkb	mordred: no I suffer mock where it exists :)	22:24
ianw	mordred: ^ hmm, rebase seemed happy ...	22:25
clarkb	mock is the python3 happy one right?	22:25
clarkb	I guess that would be the only thing I would want is to make sure we don't regress like mox	22:25
*** adriant has quit IRC		22:25
mordred	yah. mock is the python3 happy one	22:25
*** yolanda has quit IRC		22:26
*** ramishra has quit IRC		22:27
pabelanger	clarkb: blarg, my build actually failed.	22:28
pabelanger	clarkb: I'm happy to use your tests	22:28
pabelanger	while I figure out why my dib failed	22:28
*** ramishra has joined #openstack-infra		22:29
clarkb	pabelanger: ok, so shoudl I approve the glean chagne then and ask modrred for a release then we can approve the project-config change?	22:30
pabelanger	clarkb: Ya, lets do that	22:30
*** Thelo_ has joined #openstack-infra		22:30
clarkb	ok glean change approved	22:31
dtroyer	clarkb: that was in the release of the SDK that just got put into g-r (0.9.5 IIRC)	22:31
*** Thelo_ has quit IRC		22:32
rcarrillocruz	thx	22:32
dtroyer	I haven't tested it myself yet though	22:32
openstackgerrit	Merged openstack-infra/glean: Remove After=network.target dependency cycle https://review.openstack.org/364516	22:34
clarkb	mordred: ^ you want to do the honors of a release?	22:35
clarkb	or maybe get some of ianw's in first?	22:36
clarkb	if they are all tests not sure how urgent they are	22:36
clarkb	ianw: any of them make snse to try and get into a release if it happens nowish?	22:36
*** ddieterly[away] is now known as ddieterly		22:38
ianw	i don't think it's super urgent. the selinux one was to help with, well, selinux. the others were test-cleanups that feel out of trying to test it	22:39
ianw	i'm just catching up ... are you sure it's the After? as described in https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/ that's mostly a shutdown thing	22:39
*** Thelo_ has joined #openstack-infra		22:40
ianw	"network.target has very little meaning during start-up ... It's primary purpose is for ordering things properly at shutdown:"	22:40
*** shardy_afk has quit IRC		22:40
*** Thelo_ has quit IRC		22:41
*** ddieterly is now known as ddieterly[away]		22:42
*** sarob has joined #openstack-infra		22:43
*** tonytan_brb has quit IRC		22:43
openstackgerrit	Monty Taylor proposed openstack-infra/nodepool: Unsubvert image and flavor caching https://review.openstack.org/364510	22:44
clarkb	ianw: I know that those two changes to remove the After result in no more ordering cycle	22:45
clarkb	ianw: and it prevents ssh from being stop started	22:45
pabelanger	right	22:45
mordred	clarkb: let's go ahead and release ... there are a LOT of changes unreleased - I hestitate to land more given the debug cycle	22:45
*** adriant has joined #openstack-infra		22:45
clarkb	ianw: there were two interrelated problems. The ordering cycle which just made things messy and ssh was being started then stopped then started	22:46
clarkb	ianw: so if connections came in during the first start they would get killed when the service was stopped	22:46
mordred	clarkb: 1.6.0 ?	22:47
*** sarob has quit IRC		22:47
clarkb	mordred: uh the change I know of should be backaard compat and it just changes a bug so even a point would be fine	22:47
clarkb	but not sure about all the other changes	22:47
*** signed8bit is now known as signed8bit_Zzz		22:48
jeblair	pabelanger, mordred, clarkb, and i guess someone can tell sdague if they see him: here's what i've noticed about those ansible ssh connection failures: 1) they seem vaguely time correlated. they seem to come in batches, and the batches happen on all the zuul launchers. 2) i suspect they may be hitting a small number of jobs; related: they seem to disproportionately happen on multinode jobs. 3) the ssh failure happens 25 seconds ...	22:48
jeblair	... after the last poll. our polling interval is 5 seconds, and there is a 10 second ssh connection timeout. i can't account for the other 10 seconds.	22:48
jeblair	there is no additional information from the async module about the ssh error, even with -vvv. (this is perhaps not surprising at this point)	22:49
mordred	clarkb: http://paste.openstack.org/show/565904/	22:49
mordred	clarkb: that's the changelog entries since the last release	22:49
jeblair	i'm turning off verbose mode on the launchers	22:50
mordred	clarkb: I think bonding additions make it 1.6.0	22:50
clarkb	mordred: ah ya that would be a new feature	22:50
mordred	ok. Im pushing the tag	22:51
mordred	and done	22:51
mordred	may god have mercy on our souls	22:51
clarkb	mordred: https://review.openstack.org/#/c/364517/1 is the corresponding one now that that is done	22:52
jeblair	pretty sure that ship has sailed	22:52
*** Swami has joined #openstack-infra		22:52
clarkb	and yes lets hope new images work :)	22:52
mordred	jeblair: good point	22:52
clarkb	good thing we won't get those until after dhellmann is done	22:52
*** Swami_ has joined #openstack-infra		22:52
mordred	clarkb: that sohuld be safe to land now, yeah?	22:52
mordred	since glean will exist next time an image is built	22:53
clarkb	mordred: ya	22:53
mordred	k. done	22:53
*** gouthamr_ has quit IRC		22:53
fungi	woah! https://gerrit-review.googlesource.com/85233	22:54
fungi	glad to see there will finally be a fix for that issue	22:54
pleia2	nice	22:55
mordred	neat!	22:55
*** ijw has quit IRC		22:56
*** javeriak_ has quit IRC		22:57
clarkb	mordred: pabelanger the good thing is if that breaks xenial/centos/jessie it won't break any jobs	22:57
clarkb	since it will just prevent us from sshing in to unbooted hosts	22:57
clarkb	but I was able to boot locally and dhcp so I am happy	22:58
*** sarob has joined #openstack-infra		22:58
pabelanger	ya	22:59
mordred	clarkb: if you have a sec ... https://review.openstack.org/#/c/362813 is part of me working through the suboptimal caching in nodepool right now	22:59
*** annegentle has quit IRC		23:01
openstackgerrit	Merged openstack-infra/project-config: Remove After=network.target from initialize-urandom service https://review.openstack.org/364517	23:01
*** kaisers_ has joined #openstack-infra		23:02
*** hongbin has quit IRC		23:03
*** sarob has quit IRC		23:03
*** ijw has joined #openstack-infra		23:04
*** ddieterly[away] has quit IRC		23:05
*** dimtruck is now known as zz_dimtruck		23:05
*** zz_dimtruck is now known as dimtruck		23:05
pabelanger	clarkb: do we want to kick off an image-build tonight for ubuntu-xenial?	23:06
*** kaisers_ has quit IRC		23:07
openstackgerrit	Clark Boylan proposed openstack-infra/project-config: Run requirements/pbr integration on xenial https://review.openstack.org/364588	23:08
*** rbrndt has quit IRC		23:08
clarkb	mordred: tonyb ^ comments on that welcome. I think it should be relatively safe to do as soon as dhellmann is done making milestone 3 things happen	23:08
clarkb	pabelanger: we could. Then upload to osic real quick and see if it works	23:09
clarkb	pabelanger: probably a decent idea. Just make sure that glean hits pypi first	23:09
*** ijw has quit IRC		23:09
*** gouthamr has joined #openstack-infra		23:10
*** salv-orlando has quit IRC		23:10
openstackgerrit	Monty Taylor proposed openstack/os-client-config: Add ability to configure Session constructor https://review.openstack.org/364589	23:10
dhellmann	clarkb : the plan is to finish tagging by 1400 UTC tomorrow when the release team meeting starts, then branch after that for the libs that don't have branches, then some time later in the day tomorrow there's gerrit downtime IIRC. I can let you know here when we're done with the branches	23:10
*** markvoelker has joined #openstack-infra		23:10
clarkb	dhellmann: sounds good thanks	23:11
clarkb	mordred: question about that caching change. We have two different cache settings in the one clouds.yaml	23:11
mordred	clarkb: in the ansible one?	23:11
clarkb	mordred: yes	23:11
mordred	clarkb: yah - one is the per-resource cache setting - which is really the batch/poll setting	23:11
clarkb	looks like one is for the inventory cache?	23:11
mordred	clarkb: the other is "how long should an entire copy of the inventory be kept around"	23:12
clarkb	and it won't conflate the two?	23:12
mordred	nope. they're completely different things	23:12
clarkb	ok approved	23:12
mordred	if there is a valid inventory cache, ansible won't execute any shade calls at all	23:12
clarkb	ah gotcha	23:12
clarkb	but while building an inventory it will use the other cache settings	23:12
mordred	yah	23:13
mordred	in the fullness of time, I want to get the other cache stuff good enough that the giant-hammer inventory cache is not really a thing	23:13
mordred	but we're not there yet	23:13
pabelanger	clarkb: okay, once nodepool updates, I'll kick off the build	23:13
jeblair	clarkb, pabelanger, mordred: this is the number of counts of ansible ssh failures per job over the past month: http://paste.openstack.org/show/565906/	23:13
jeblair	we'd probably need to normalize that by number of jobs run to really pick up a pattern	23:14
mordred	jeblair: wow: 3939 gate-tempest-dsvm-neutron-full-ubuntu-xenial	23:14
*** gouthamr_ has joined #openstack-infra		23:14
clarkb	mordred: thats the ipv6 issue	23:14
*** gouthamr has quit IRC		23:14
mordred	yah	23:14
jeblair	it is?	23:14
clarkb	jeblair: ya it only affected jobs using neutron and it was every job using neutron basically	23:14
jeblair	and that showed up as the problem i'm looking at?	23:15
clarkb	jeblair: you should be able to tell reasonably well if it drops off in the last week or so	23:15
*** shashank_hegde has quit IRC		23:15
*** dimtruck is now known as zz_dimtruck		23:15
clarkb	jeblair: it showed up as ssh connect failed yes	23:15
*** xarses has quit IRC		23:15
jeblair	clarkb: without setting unreachable?	23:15
ianw	timothyb89: so i guess running it for "old" and "new" doesn't quite work to show separate outputs -> http://logs.openstack.org/60/364560/3/check/gate-grenade-dsvm-neutron-ubuntu-trusty/adeae81/logs/stackviz/#/stdin	23:15
clarkb	jeblair: I am not sure what that means	23:16
pabelanger	I think ipv6 was different, because those jobs were requeue in zuul	23:16
clarkb	jeblair: pabelanger I think if you compare the wkeely numbers for the last week and the 3 weeks before those jobs should fall off as being that bad	23:17
clarkb	the ironic jobs all use neutron too (ironic has never supported nova net)	23:17
*** rvasilets___ has quit IRC		23:17
openstackgerrit	Merged openstack-infra/system-config: Add floating-ip cache expiration time settings https://review.openstack.org/362813	23:18
jeblair	clarkb: i'm not convinced that my methodology is not already excluding those	23:19
openstackgerrit	Doug Hellmann proposed openstack-infra/project-config: fix networking-hyperv release acls https://review.openstack.org/364591	23:19
jeblair	clarkb: but i don't want to spend any more days on this than i already am	23:19
jeblair	clarkb: so i will just re-run it for the last week	23:19
openstackgerrit	Derek Higgins proposed openstack-infra/tripleo-ci: Remove support for legacy rh1 https://review.openstack.org/347918	23:19
clarkb	ok I think we would probably have noticed with freeze if neutron was failing that hard. We certainly noticed it when ipv6 was broken	23:19
jeblair	clarkb: most of these are being re-queued by zuul	23:20
clarkb	yes that was the behavior we saw with ipv6. The job would run then timeout then ansible would rerun it.	23:20
clarkb	the impact was in slowing down the throughput of the queues by a lot	23:20
jeblair	clarkb: these aren't timeouts	23:20
pabelanger	clarkb: mordred: BTW, I noticed this gem while chasing the ssh-server stop / start issue: http://paste.openstack.org/show/565907/	23:21
jeblair	they are ssh connection errors	23:21
pabelanger	not sure now to fix that atm	23:21
clarkb	jeblair: yes	23:21
clarkb	jeblair: with ipv6 the existing connections had to timeout then we would get the connection error	23:21
clarkb	we saw both things together because the hosts jsut became unroutable	23:21
clarkb	so tcp thinks its still connected for X amount of time before it gives up and then connection fails	23:22
jeblair	clarkb: okay, that's similar enough that we'd have to get into the weeds to figure out if i'm picking it up. so i'll still just do the last week.	23:23
*** tpsilva has quit IRC		23:24
jeblair	so just starting with aug 26	23:25
mordred	pabelanger: wow	23:25
mordred	pabelanger: uhm ... what is that?	23:25
clarkb	pabelanger: mordred looks like a host that has run neutron	23:26
mordred	yah	23:26
jeblair	clarkb, pabelanger, mordred: http://paste.openstack.org/show/565908/	23:26
mordred	and that glean just picked up all of the things	23:26
clarkb	and glean is gonna try to configure all those interfaces	23:26
mordred	yah	23:26
mordred	and fail	23:26
mordred	jeblair: oh that's much better	23:26
clarkb	ya thats more like I would expect with multinode being more common for whatever reason	23:27
*** annegentle has joined #openstack-infra		23:27
jeblair	here's the full list sorted by time: http://paste.openstack.org/show/565909/	23:29
*** ilyashakhat_mobi has joined #openstack-infra		23:29
pabelanger	mordred: Ya, a neutron job. spotted it in passing	23:29
jeblair	you can see them grouping i was talking about	23:29
mordred	pabelanger: oh - wow - is glean getting triggered by udev/systemd every time neutron creates an interface???	23:30
clarkb	mordred: yes	23:30
pabelanger	mordred: i think so	23:30
mordred	that's amaze	23:30
clarkb	I think its fine if noisy	23:31
mordred	https://review.openstack.org/#/c/364510/ clarkb, jeblair: last piece of inefficient nodepool ... it has passed http://logs.openstack.org/10/364510/3/check/gate-dsvm-nodepool-src-shade/67d715e/ already, which contains the shade fix that enables it (it failed a previous run of that, so the success shows the shade fixed fix it)	23:31
jeblair	13 jobs failed that way within a second of each other at 21:07 today	23:31
mordred	jeblair: it's interesting to me that it clusters like that ...	23:31
mordred	jeblair: but that also makes me think "cloud network hiccup"	23:31
jeblair	mordred: yes, though it is across all of our launchers, and across clouds	23:31
mordred	hrm	23:31
jeblair	so it's at least a datacenter-scale hiccup	23:32
clarkb	one commonality seems to be the job/project?	23:32
clarkb	like 12:54 today a bunch of multinodes fail	23:32
jeblair	clarkb: yeah, let me regenerate this with project + change info as well	23:32
openstackgerrit	Merged openstack-infra/shade: Fix up image and flavor by name in create_server https://review.openstack.org/364548	23:32
clarkb	there is a clump of cinder fails too	23:32
pabelanger	clarkb: ubuntu-xenial DIB started	23:33
*** chlong has quit IRC		23:33
*** ilyashakhat_mobi has quit IRC		23:34
*** sflanigan has joined #openstack-infra		23:34
ianw	how do we feel about devstack-gate making symlinks in the log directory	23:34
*** gongysh has joined #openstack-infra		23:35
clarkb	ianw: is this related to the subunit thing?	23:37
*** vhosakot has quit IRC		23:37
clarkb	ianw: its probably fine though we should be careful not to copy them with the logs	23:37
clarkb	er rather we should follow the links when copying	23:37
ianw	clarkb: yeah, splitting stackviz between "old" and "new" runs in grenade output, but not duplicating the ~3mb of js that's driving it	23:38
clarkb	ianw: I just had to symlink /usr/libexec/qemu-bridge-helper to /usr/lib/qemu-bridge-helper because virsh would not honor the path I set in /etc/libvirt/qemu.conf	23:39
clarkb	so if symlinks can solve this problem too that seems fine with me	23:39
ianw	heh, apparently libncurses split itself in two between fedora 23 & 24, so the other day debugging the sfdisk stuff i symlinked half of the new ncurses back to a libncurses.so which had just enough symbols to make it work. symlinks can do anything :)	23:41
mordred	hahahaha	23:43
pabelanger	clarkb: still didn't get my local DIB working, you were able to SSh into your version?	23:43
clarkb	pabelanger: yup	23:43
pabelanger	okay	23:43
pabelanger	well, giving up now	23:43
pabelanger	since nodepool is building it	23:43
*** Genek has quit IRC		23:43
*** xarses has joined #openstack-infra		23:44
mordred	clarkb, jeblair: woot. the change finished testing. if you look at http://logs.openstack.org/10/364510/3/check/gate-dsvm-nodepool/3db0c11/logs/screen-nodepool.txt.gz you can see the issue that the shade patch we landed fixed. then in the logs for the passing job all is happy	23:46
*** zhurong has joined #openstack-infra		23:48
jeblair	i will be so happy when we have afs on the launchers. i'm doing all kinds of copying files around between them right now and it's silly	23:49
*** thcipriani is now known as thcipriani\|afk		23:49
mordred	jeblair: ++	23:50
*** Julien-zte has quit IRC		23:50
jeblair	i guess i should poke at https://review.openstack.org/305477	23:50
*** markvoelker has quit IRC		23:51
jeblair	clarkb, mordred, pabelanger: http://paste.openstack.org/show/565911/	23:53
*** dingyichen has joined #openstack-infra		23:54
jeblair	it looks like the groupings are not simply "all the jobs for a change"	23:54
jeblair	that seems pretty spread out too	23:54
jeblair	are they all bluebox+osic?	23:55
*** waht has quit IRC		23:56
jeblair	no, but mostly.	23:56
jeblair	mostly osic makes sense. mostly bluebox does not.	23:56
*** markvoelker has joined #openstack-infra		23:56
jeblair	i wonder if we're looking at a real-time map of internet routing issues :)	23:57
mordred	jeblair: we tend to expose many issues	23:58
mordred	jeblair: why not internet routing issues?	23:58
*** zhurong has quit IRC		23:59
jeblair	the project list is pretty broad too.	23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!