Thursday, 2016-03-31

jeblair	all of the jeblair	00:00
clarkb	pabelanger: oh did I misundertsand how this works?	00:00
clarkb	pabelanger: did it start the job after the restart? I may have derped and can stop my process which should remove the delete request	00:00
clarkb	ah yup Iderped will remove the delete request, it wasnt a stale build it was just a waiting build	00:01
pabelanger	clarkb: we had a ubuntu-xenail job in the queue, but nodepool-builder didn't know the elements	00:01
pabelanger	so I think when you restarted, it was happy again	00:01
clarkb	pabelanger: yup	00:01
*** akshai has joined #openstack-infra		00:01
clarkb	and now that my client that is waiting on a delete job to show up is gone the delete job request should be gone too and everything should be happy	00:01
*** mylu has quit IRC		00:02
clarkb	mordred: jeblair where in that stack should a reviewer start?	00:02
*** stevemar has quit IRC		00:02
mordred	clarkb: so ... good question - one sec	00:02
mordred	clarkb: the shade patches	00:03
*** jamesmcarthur has joined #openstack-infra		00:03
openstackgerrit	sebastian marcet proposed openstack-infra/openstackid-resources: Fix on summit event null https://review.openstack.org/299684	00:03
mordred	clarkb: actually, the non-shade patches are the next iteration of happy	00:03
clarkb	mordred: https://review.openstack.org/#/c/299645/ is failing	00:04
jeblair	so shade patches get us going again; the nodepool (and occ) patches ar for later	00:04
openstackgerrit	Merged openstack-infra/openstackid-resources: Fix on summit event null https://review.openstack.org/299684	00:04
pabelanger	clarkb: so, I am hitting: https://github.com/openstack-dev/devstack/blob/master/stack.sh#L188	00:04
clarkb	looks like jeblairs patches too	00:04
jeblair	clarkb: i rechecked 645 -- it hit a pathologically slow vexxhost node	00:04
clarkb	pabelanger: set FORCE=yes in localrc apparently	00:04
openstackgerrit	Monty Taylor proposed openstack-infra/nodepool: Pass extended network information in to occ/shade https://review.openstack.org/299679	00:04
jeblair	was not a legit failure	00:04
clarkb	pabelanger: you can add declare -x DEVSTACK_LOCAL_CONFIG='FORCE=yes'	00:05
clarkb	pabelanger: I think that will get you past that	00:05
pabelanger	ok	00:05
pabelanger	testing	00:05
*** Sukhdev has quit IRC		00:06
*** fawadkhaliq has quit IRC		00:06
*** eil397 has quit IRC		00:06
*** mylu has joined #openstack-infra		00:07
*** fawadkhaliq has joined #openstack-infra		00:07
*** esker has joined #openstack-infra		00:08
*** dimtruck is now known as zz_dimtruck		00:08
*** salv-orl_ has quit IRC		00:09
*** akshai_ has joined #openstack-infra		00:09
fungi	mordred: is there supposed to be a way to get detailed puppet stdout from ansible launch node? here's a snippet of the failure i get http://paste.openstack.org/show/492518/	00:09
fungi	it's decidedly terse and unhelpful	00:10
pabelanger	clarkb: better, looks to be running now	00:10
clarkb	fungi: yes, check syslog on the machine	00:10
*** amitgandhinz has joined #openstack-infra		00:11
fungi	clarkb: followup question... how do i make it not automatically delete the node on failure?	00:11
mordred	fungi: --keep	00:11
clarkb	fungi: there is a flag ya that	00:11
fungi	aha	00:11
fungi	i missed that	00:11
fungi	thanks!	00:11
mordred	fungi, clarkb: rcarrillocruz has written some good stuff that I think we can use for ansible-launch-node, btw	00:11
mordred	fungi, clarkb: I'll send some infos about it soon	00:11
cody-somerville	\o/	00:12
*** pvaneck has quit IRC		00:12
openstackgerrit	James E. Blair proposed openstack-infra/shade: Cache ports like servers https://review.openstack.org/299671	00:12
mordred	cody-somerville: yah man. it's good. you should read the blog post	00:12
openstackgerrit	James E. Blair proposed openstack-infra/shade: Remove conditional blocking on server list https://review.openstack.org/299672	00:12
*** esker has quit IRC		00:12
mordred	cody-somerville: http://rcarrillocruz.com/deploying-multiple-openstack-clouds-with-ansible-in-a-data-driven-fashion/	00:13
* jeblair is preparing to welcome our ansible node launching overlords		00:13
*** akshai has quit IRC		00:13
*** fawadkhaliq has quit IRC		00:13
clarkb	mordred: this is what was talked about at the sprint ya?	00:13
cody-somerville	Saweet.	00:13
*** ZZelle has quit IRC		00:14
mordred	clarkb: yah	00:15
mordred	clarkb: I'm going to poke at it in anger in the next couple of days	00:15
mordred	but looking at it it seems to meet our needs	00:15
jeblair	i'm not going to be in the room for that	00:15
mordred	jeblair: that's what you think	00:15
*** esker has joined #openstack-infra		00:15
*** markvoelker has joined #openstack-infra		00:15
*** amitgandhinz has quit IRC		00:16
jeblair	clarkb: my shade change passes local unit tests now	00:16
*** fawadkhaliq has joined #openstack-infra		00:16
mordred	clarkb: also, https://review.openstack.org/#/c/298432/	00:17
*** nwkarsten has joined #openstack-infra		00:17
*** gildub has joined #openstack-infra		00:17
*** sridhar_ram has quit IRC		00:18
clarkb	mordred: ^ should eb safe to merge before any of the othre change smerge ya?	00:19
clarkb	will just be ignored?	00:19
*** nwkarsten has quit IRC		00:21
*** fawadkhaliq has quit IRC		00:21
pabelanger	Hmm, ubuntu-xenial failed to find: openjdk-7-jdk... odd	00:22
fungi	E: Unable to locate package php5-cli	00:22
pabelanger	currently building again	00:22
*** markvoelker_ has joined #openstack-infra		00:22
clarkb	pabelanger: maybe it is on 8?	00:22
fungi	except i was just able to apt-get install it on that machine	00:22
*** yamada-h has joined #openstack-infra		00:22
clarkb	pabelanger: yup	00:22
clarkb	for that you will liekly need to edit devstack	00:23
clarkb	in any case if the fails are in devstack now thats probably a good improvement from our end	00:23
*** mriedem has joined #openstack-infra		00:23
pabelanger	ii openjdk-7-jdk:amd64 7u95-2.6.4-1 amd64 OpenJDK Development Kit (JDK)	00:23
pabelanger	that is from my local dib	00:23
pabelanger	so, unless something changes yesterday	00:23
*** davideag_ has quit IRC		00:23
*** fawadkhaliq has joined #openstack-infra		00:23
clarkb	oh looks like it has both	00:24
clarkb	odd I dunno then	00:24
pabelanger	hoping something wrong with mirrir	00:24
mordred	clarkb: yes	00:24
pabelanger	mirror*	00:24
*** annegentle has quit IRC		00:24
clarkb	mordred: reading jeblairs change it seems to use a hardcoded value not something from oscc?	00:24
mordred	the port one?	00:25
clarkb	I probably need to read all the changes to understand the whole set	00:25
clarkb	mordred: yes	00:25
jeblair	it's a default; it gets the value from oscc if its set	00:25
mordred	what jeblair said	00:25
*** markvoelker has quit IRC		00:25
jeblair	clarkb: line 228	00:25
mordred	clarkb: the port cache is modeled after how the list_servers cache works	00:25
mordred	so it's doing the same logic, if looking at the other one helps	00:25
clarkb	https://review.openstack.org/#/c/299671/2/shade/openstackcloud.py line 197 why not set it there instead?	00:25
jeblair	clarkb: that's the dogpile cache path	00:26
jeblair	which we won't hit	00:26
clarkb	ah we are just using a buit in cache thing like nodepool which is the else case?	00:27
mordred	yes	00:27
*** yamada-h has quit IRC		00:27
*** _amrith_ is now known as amrith		00:28
pabelanger	clarkb: Ah, new release today. Likely still syncing to mirrors	00:29
*** mtanino has quit IRC		00:29
sc`	does anyone know what might cause glance to throw a 403 when uploading an image? it's in an integration gate, so i can only really see that glance is throwing the 403, not what it's doing or why	00:31
*** jamesmcarthur has quit IRC		00:31
sc`	it only happens on ubuntu, i should note. centos is fine	00:31
clarkb	sc`: the gate jobs should have service logs	00:32
clarkb	including those for glance	00:32
sc`	if they're there, i'm unable to find them. just a console log	00:32
clarkb	mordred: jeblair how does cache invalidation work here? doesn't look like making a new port updates the cache	00:32
clarkb	sc`: what job?	00:32
clarkb	or link to the logs	00:32
*** sridhar_ram has joined #openstack-infra		00:33
*** Swami has quit IRC		00:33
*** Sukhdev has joined #openstack-infra		00:33
clarkb	sc`: sounds like a misconfigured job if it isn't getting service logs on an integration test	00:33
*** Sukhdev has quit IRC		00:33
*** Qiming has quit IRC		00:34
sc`	https://jenkins02.openstack.org/job/gate-ubuntu-trusty-cookbook-openstack-common-chef-rake-integration-nv/9/console is one of the failed builds. it only seems to nab the console log	00:34
*** rossella_s has quit IRC		00:34
*** sdake has quit IRC		00:34
jeblair	clarkb, mordred: correct, it does not invalidate the cache. i wasn't focused on that path since we aren't creating ports. we could add that; it's essentially an optimization (the cache will timeout).	00:35
*** rossella_s has joined #openstack-infra		00:35
clarkb	yup, so you will wnat to look into how the other integration tests deal with this. Since this is for chef cookbooks I am assuming you don't run the services like devstack and manage the logs directly instead they should log to normal logging locations?	00:35
fungi	i cannot for the life of me figure out why this server won't bootstrap. puppet claims apt-get failed to find the php5-cli and php5-gmp packages, but on that same server i can apt-get install them just dandy	00:35
openstackgerrit	Monty Taylor proposed openstack-infra/nodepool: Start using dogpile caching in devstack tests https://review.openstack.org/299690	00:35
clarkb	if that is the case the puppet module folk have a genericish log grabber script that should work if you use semi standard log locations	00:36
clarkb	jeblair: we do create ports though	00:36
pabelanger	fungi: I am having issues installing packages too	00:36
clarkb	jeblair: every time we create a server	00:36
openstackgerrit	Monty Taylor proposed openstack-infra/nodepool: Start using dogpile caching in devstack tests https://review.openstack.org/299690	00:36
pabelanger	fungi: think the mirrors are funky atm	00:36
sc`	fwiw, centos doesn't have service logs either. i suspect an artifact of the recent work to extract logs from a chef-built openstack	00:36
*** sdake has joined #openstack-infra		00:36
jeblair	fungi: any chance it has something to do with our changing the apt repos?	00:36
fungi	pabelanger: i've launched two servers and it complained about the exact same packages each time	00:36
sc`	so i'll have to dig into that before i can figure out why glance is even erroring out	00:36
fungi	jeblair: i suppose it's possible this is racing that	00:36
pabelanger	fungi: Ya, I am failing on openjdk-7-jdk currently	00:37
jeblair	clarkb: we don't explicitly create a port, but yes, one (or more) is implicitly created when we create servers. i added a loop in there so when we're looking for a port to add an ip to a new server, we check at least twice to give the cache a chance to update.	00:37
pabelanger	sc`: Take a look at the work EmilienM did with the puppet integration jobs. They have a good script for collecting logs	00:37
clarkb	sc`: copy-puppet-logs is the thing puppet uses	00:37
clarkb	sc`: its mostly not puppet specific though so you can probably adapt it to your jobs	00:38
*** Jeffrey4l_ has quit IRC		00:38
clarkb	fungi: jeblair launch node shouldn't use our apt mirrors though	00:38
sc`	some stuff has already been merged that's supposed to do that... but clearly it is not :D	00:38
clarkb	should use rax's iirc	00:38
sc`	i'll poke at it some more until my eyes say no more	00:38
fungi	clarkb: we override that in our manifest	00:38
clarkb	fungi: oh to use our mirrors?	00:38
fungi	clarkb: to use us.archive.ubuntu.com and, more importantly, security.ubuntu.com	00:39
clarkb	sc`: looking at the console log you may just be missing the part of the thing that tells jenkins to copy the files	00:39
jeblair	and not "give me a kernel from the future repo" :)	00:39
*** mylu has quit IRC		00:39
fungi	clarkb: so it's possible puppet is trying to install packages from rackspace mirrors before we fix our sources	00:39
clarkb	sc`: the jjb macro for that seems to be 'devstack-logs'	00:39
clarkb	fungi: that would not surprise me	00:40
fungi	though the log indicates the package installation is happening after we fix the sources list	00:41
sc`	i'll take a look at that. see if i can adapt something to our own things. was kind hoping there was a way to avoid the scenic route, but that's fine	00:41
*** gildub has quit IRC		00:41
*** achanda has quit IRC		00:41
clarkb	fungi: it may not have updated	00:41
*** gildub has joined #openstack-infra		00:41
clarkb	fungi: er update the apt-get update cahce things	00:41
fungi	Mar 31 00:30:45 openstackid-dev puppet-user[31408]: (/Stage[main]/Openstack_project::Template/File[/etc/apt/sources.list.d/openstack-infra.list]/ensure) defined content as '{md5}0b159a1823cdad18e31cbce892f5d40f'	00:42
fungi	Mar 31 00:30:47 openstackid-dev puppet-user[31408]: (/Stage[main]/Openstackid/Package[php5-gmp]/ensure) change from purged to present failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install php5-gmp' returned 100: Reading package lists...	00:42
*** nadya has joined #openstack-infra		00:42
fungi	yeah, maybe it doesn't apt-get update?	00:42
pabelanger	basically some thing here with jenkins::slave	00:43
pabelanger	except openjdk-7-jdk	00:43
*** annegentle has joined #openstack-infra		00:45
*** mylu has joined #openstack-infra		00:45
fungi	pabelanger: clarkb: do we need a before directive or something at http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/manifests/template.pp#n193	00:46
fungi	or is there a way to atomically add an exec to those?	00:46
*** nadya has quit IRC		00:47
clarkb	I think you can before => Package meta class thing	00:47
*** tiswanso has joined #openstack-infra		00:47
*** tiswanso has quit IRC		00:47
*** tiswanso has joined #openstack-infra		00:48
pabelanger	ya, I think we are doing apt-get update too late in the process	00:48
pabelanger	2016-03-30 17:56:08,500 INFO nodepool.image.build.ubuntu-trusty: Notice: /Stage[main]/Apt::Update/Exec[apt_update]: Triggered 'refresh' from 3 events	00:48
pabelanger	that is from ubunt-trusty	00:48
pabelanger	but way after we install stuff	00:49
pabelanger	it should be the first thing we do	00:49
pabelanger	http://nodepool.openstack.org/dib.ubuntu-trusty.log	00:49
jeblair	yeah, this sounds like a whoops that we wouldn't notice except in these edge cases	00:49
*** annegentle has quit IRC		00:49
sc`	ok. i think i see why i don't have service logs	00:49
sc`	missing devstack-logs in a couple places	00:50
sc`	it got updated in one place, but not everywhere	00:51
sc`	however. i do have it for integration, and i don't have service logs	00:51
sc`	https://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/chef-jobs.yaml seems to confirm it	00:52
sc`	my integration template has both console-log and devstack-logs as publishers	00:52
clarkb	mordred: you going to figure out why https://review.openstack.org/#/c/299679/2 exploded?	00:52
sc`	that was merged back at the beginning of March	00:52
openstackgerrit	Matthew Thode proposed openstack/diskimage-builder: Add Gentoo to the dhcp-all-interfaces element https://review.openstack.org/299694	00:53
pabelanger	wish we had a boostrap stage in puppet, then we could do our package manager updates then	00:54
clarkb	sc`: all that publisher does is copy files if they exist, if they do not exist you don't get any files	00:54
prometheanfire	not sure how people will like how I'm doing this, but it's the same method I used in https://github.com/prometheanfire/gentoo-cloud-prep	00:54
clarkb	sc`: I would double check that you are copying the files to the appropriate location(s)	00:54
mordred	clarkb: that needs an OCC release	00:54
mordred	clarkb: it's not urgent for now	00:54
prometheanfire	I'll wait for ianw to -1 it :P	00:54
pabelanger	will have to dive into it tomorrow and figure out how to force apt to run first in main	00:54
pabelanger	without an insane chain	00:54
clarkb	mordred: that is the fix for tripleo cloud isnt it?	00:54
mordred	clarkb: it's the _Real_ fix. we have a workaround patch in	00:55
clarkb	mordred: where is that?	00:55
clarkb	is that what we are running in production now?	00:55
jeblair	clarkb: no, production is manually rolled back to pre-shade	00:55
openstackgerrit	Merged openstack-infra/nodepool: Remove unused functions https://review.openstack.org/299678	00:55
sc`	clarkb: is $WORKSPACE/logs the appropriate place?	00:56
*** esker has quit IRC		00:56
jeblair	though we just landed a patch, so it's about to be post-shade again :)	00:56
clarkb	sc`: yes that looks correct	00:56
sc`	then yes we should be	00:56
clarkb	jeblair: woops I approved that patch	00:56
clarkb	sc`: can you show me where you are copying the logs to that path?	00:57
*** gyee has quit IRC		00:57
*** esker has joined #openstack-infra		00:57
*** stevemar has joined #openstack-infra		00:58
jeblair	clarkb: np, just something to be aware of while we're in the sketchy on-again-off-again shade zone. hopefully we can land these updates to shade and restart and stick with it; of course if it drags out, we can just revert shade and try again later	00:58
sc`	clarkb: https://github.com/openstack/openstack-chef-repo/blob/master/Rakefile#L153-L160 is the meat of it. down at the bottom it issues a _save_logs with an arg of log_dir	00:59
sc`	so perhaps it's not picking up the env variable properly?	00:59
clarkb	sc`: or it isn't running when your glance thing fails first	01:00
clarkb	sc`: its odd to me that the log grabbing isn't the last thing you do	01:01
sc`	perhaps. _save_logs might never get a chance to fire for ubuntu due to the initial chef run failing, i believe	01:01
clarkb	and I do not know if an exception can trap and bypass that	01:01
clarkb	_run_nova_tests presumably wants to have its logs recorded too	01:01
sc`	someone else wrote this. i've been wrapping everything in conditionals to make it work for both centos and ubuntu	01:01
*** jamielennox is now known as jamielennox\|away		01:01
sc`	centos works without issue, and ubuntu fails at uploading to glance. the main problem with being able to see is i don't have logs	01:02
clarkb	yes so fix the logs	01:02
clarkb	I think your issue is how you are collecting the logs 1) it isn't the last thing you do and 2) it may be skipped on error	01:02
*** esker has quit IRC		01:02
sc`	perhaps so	01:02
*** stevemar has quit IRC		01:03
*** shashank_hegde has quit IRC		01:03
sc`	i don't know what was in the author's head at the time, so at best we can speculate	01:03
openstackgerrit	Paul Belanger proposed openstack-infra/system-config: [WIP] Add bootstrap stage to puppet https://review.openstack.org/299697	01:03
clarkb	if it were me I would stop reinventing the wheel and just reuse what everything else does	01:03
pabelanger	fungi: ^ will test that tomorrow, but that should give us the ability to force apt to run first	01:04
clarkb	possibly rename copy_puppet_logs to copy_system_logs, add in the chef specific stuff and use that	01:04
sc`	chef is a different kind of wheel, so... yeah	01:04
pabelanger	clarkb: devstack-gate still running, will post the results in the morning	01:04
clarkb	pabelanger: does that mean devstack got past the packge woes?	01:04
clarkb	pabelanger: is it running tempest?	01:05
pabelanger	clarkb: yes	01:05
sc`	i mean, i see your point, but it _is_ a slightly differently shaped wheel	01:05
pabelanger	tempest running.... slowly	01:05
clarkb	the xenial dib build appears to be failing in a loop	01:05
fungi	pabelanger: i'm excited, though in the meantime tonight i either need to wait for the mirror situation to clear up or put together an uglier temporary workaround	01:05
clarkb	sc`: I am not sure why or how it could be so different when openstack is pretty consistent on these platforms	01:05
pabelanger	clarkb: Yup, that is the openjdk missing package	01:05
clarkb	sc`: the chef logs itself may be differnetb but nova and keystone and glance are writing to the same files	01:05
clarkb	pabelanger: ah	01:05
pabelanger	fungi: right, waiting for the mirror to clean up too	01:06
sc`	it shouldn't be so vastly different, i'm in agreement	01:06
fungi	i'm booting another right now to see if it's improved	01:06
*** tphummel has quit IRC		01:06
sc`	making integration work shouldn't be a contortion of code	01:06
*** mylu has quit IRC		01:07
*** mylu has joined #openstack-infra		01:08
*** aysyd has quit IRC		01:08
*** esker_ has joined #openstack-infra		01:10
clarkb	ianw: re debian are you saying that while they use systemd they have not fully bought into it and are just using pieces of it?	01:10
*** jamesmcarthur has joined #openstack-infra		01:11
clarkb	ianw: I think we can worry about htat if/when it becomes a problem but I worry that if debian is doing that then maybe xenial is too?	01:11
clarkb	pabelanger: ^on your xenial box is journalctl all functional and happy for logging?	01:11
*** amitgandhinz has joined #openstack-infra		01:11
*** bpokorny_ has joined #openstack-infra		01:12
*** Goneri has quit IRC		01:12
openstackgerrit	Merged openstack-infra/shade: Workaround multiple private network ports https://review.openstack.org/299645	01:12
*** apoorvad has quit IRC		01:13
pabelanger	clarkb: I didn't test your patch. But journalctl is working	01:13
clarkb	pabelanger: if journalctl works then my patch should be fine	01:14
*** fawadkhaliq has quit IRC		01:14
clarkb	the owrry is in cases where journalctl is installed but non functional	01:14
*** fawadkhaliq has joined #openstack-infra		01:14
*** esker_ has quit IRC		01:14
*** bpokorny has quit IRC		01:15
*** EricGonc_ has joined #openstack-infra		01:16
*** bpokorny_ has quit IRC		01:16
*** jamesmcarthur has quit IRC		01:16
*** amitgandhinz has quit IRC		01:17
ianw	clarkb: yes, people have worried about systems like that when doing similar stuff in devstack. personally, i think make a choice and stick with it	01:18
*** EricGonczer_ has quit IRC		01:19
clarkb	ya if a system does that then they can figure out how to get their syslog :)	01:19
*** resker has joined #openstack-infra		01:20
*** fawadkhaliq has quit IRC		01:20
*** smarcet has quit IRC		01:22
openstackgerrit	Paul Belanger proposed openstack-infra/system-config: [WIP] Add bootstrap stage to puppet https://review.openstack.org/299697	01:22
sc`	interesting. moved _save_logs down in the for loop to the end and it doesn't try to kick off _setup_tempest anymore before trying and failing to scp logs. not sure what to make of that	01:22
*** fawadkhaliq has joined #openstack-infra		01:23
sc`	no, my eyes are crossing. that's the failure from _setup_tempest	01:23
openstackgerrit	Merged openstack-infra/shade: Re-allow list of networks for FIP assignment https://review.openstack.org/298789	01:23
openstackgerrit	James E. Blair proposed openstack-infra/nodepool: Set cache parameters like we do in production https://review.openstack.org/299702	01:23
*** jamesmcarthur has joined #openstack-infra		01:23
*** fawadkhaliq has quit IRC		01:24
*** resker has quit IRC		01:24
*** esker has joined #openstack-infra		01:24
*** annegentle has joined #openstack-infra		01:24
jeblair	fungi: what about adding the apt-get update ?	01:24
openstackgerrit	Monty Taylor proposed openstack-infra/nodepool: Start using dogpile caching in devstack tests https://review.openstack.org/299690	01:25
*** sridhar_ram has quit IRC		01:25
*** fawadkhaliq has joined #openstack-infra		01:25
*** resker has joined #openstack-infra		01:26
jeblair	clarkb: ^ i set 702 to depend-on my shade port caching patch, so that code path should be executed in 702 (so we can get test results)	01:26
fungi	jeblair: yeah, adding apt-get update will in theory work, i just need to continue figuring out the syntax for making sure that happens before any apt-get install operations (e.g. before any package resources are ensured)	01:27
*** fawadkhaliq has quit IRC		01:27
jeblair	fungi: ok cool. was worried that idea got abandoned (it sounded good to me)	01:27
*** zhurong has joined #openstack-infra		01:27
fungi	not abandoned, just magical puppet fantasy land stuff	01:28
fungi	so learning in progress	01:28
*** esker has quit IRC		01:29
*** annegentle has quit IRC		01:29
openstackgerrit	Paul Belanger proposed openstack-infra/system-config: [WIP] Add bootstrap stage to puppet https://review.openstack.org/299697	01:29
fungi	the recommended solution does indeed appear to be pabelanger's patch there	01:30
fungi	(e.g., ensure ordering by using stages)	01:30
fungi	though there's also this...	01:31
fungi	Exec["apt-update"] -> Package <\| \|>	01:31
openstackgerrit	Matthew Thode proposed openstack/diskimage-builder: Add Gentoo to the dhcp-all-interfaces element https://review.openstack.org/299694	01:31
*** resker has quit IRC		01:31
*** jamesmcarthur has quit IRC		01:31
fungi	which is some bizarre syntax to be sure	01:31
pabelanger	fungi: right, that could work too	01:31
*** sputnik13 has quit IRC		01:31
fungi	apparently that will cause the exec to fire on every run though	01:32
pabelanger	okay, calling it a night. eyes on fire	01:32
pabelanger	will hack more on stages tomorrow	01:32
fungi	night pabelanger	01:32
*** jamesmcarthur has joined #openstack-infra		01:32
fungi	and thanks again!	01:33
pabelanger	np	01:33
*** sputnik13 has joined #openstack-infra		01:33
*** baoli has joined #openstack-infra		01:37
*** esker_ has joined #openstack-infra		01:37
*** sputnik13 has quit IRC		01:38
*** baoli_ has joined #openstack-infra		01:39
*** stevemar has joined #openstack-infra		01:39
*** EricGonc_ has quit IRC		01:41
*** jamesmcarthur has quit IRC		01:41
*** achanda has joined #openstack-infra		01:41
*** esker_ has quit IRC		01:42
*** baoli_ has quit IRC		01:42
*** baoli has quit IRC		01:42
*** Daisy has joined #openstack-infra		01:43
*** achanda has quit IRC		01:46
*** Qiming has joined #openstack-infra		01:47
*** esker has joined #openstack-infra		01:47
*** kzaitsev_mb has quit IRC		01:47
*** jamesmcarthur has joined #openstack-infra		01:48
*** sdake_ has joined #openstack-infra		01:49
*** resker has joined #openstack-infra		01:50
*** sdake has quit IRC		01:50
*** nwkarsten has joined #openstack-infra		01:50
*** Daisy has quit IRC		01:51
*** Daisy has joined #openstack-infra		01:51
*** hichihara has joined #openstack-infra		01:51
*** esker has quit IRC		01:51
*** esker has joined #openstack-infra		01:53
*** tqtran has joined #openstack-infra		01:53
*** nwkarsten has quit IRC		01:55
*** esker_ has joined #openstack-infra		01:55
*** resker has quit IRC		01:55
*** jamesmcarthur has quit IRC		01:55
*** Daisy has quit IRC		01:55
*** Daisy has joined #openstack-infra		01:55
*** achanda has joined #openstack-infra		01:56
*** esker has quit IRC		01:57
*** tqtran has quit IRC		01:57
*** jamesmcarthur has joined #openstack-infra		01:59
*** esker_ has quit IRC		01:59
*** Daisy has quit IRC		02:00
*** mtanino has joined #openstack-infra		02:00
*** Daisy has joined #openstack-infra		02:00
*** Daisy has quit IRC		02:03
*** Daisy has joined #openstack-infra		02:03
*** zz_dimtruck is now known as dimtruck		02:04
*** tphummel has joined #openstack-infra		02:06
*** SumitNaiksatam has joined #openstack-infra		02:09
*** nwkarsten has joined #openstack-infra		02:11
*** mylu has quit IRC		02:11
sc`	with the logs saving at the bottom of the loop, my centos gate works and properly copies logs, but ubuntu does not copy logs upon failure	02:11
*** amitgandhinz has joined #openstack-infra		02:12
clarkb	sc`: it is probably short circuiting when things fail earlier	02:12
clarkb	I am not super familiar with ruby/rake but you may need a try: finally: type setup	02:13
*** thorst has quit IRC		02:13
sc`	mmm... perhaps. it is tired and i am late, however	02:13
*** thorst has joined #openstack-infra		02:13
*** jerryz has quit IRC		02:14
*** Goneri has joined #openstack-infra		02:14
*** esker_ has joined #openstack-infra		02:15
*** jamesmcarthur has quit IRC		02:16
*** amitgandhinz has quit IRC		02:17
*** Jeffrey4l has joined #openstack-infra		02:17
*** yolanda has quit IRC		02:18
*** jamesmcarthur has joined #openstack-infra		02:18
*** yamamoto_ has joined #openstack-infra		02:19
*** esker_ has quit IRC		02:20
anteaya	yonglihe: hi	02:20
yonglihe	anteaya: hi,	02:21
*** nwkarsten has quit IRC		02:21
*** shashank_hegde has joined #openstack-infra		02:21
*** salv-orlando has joined #openstack-infra		02:21
anteaya	yonglihe: do you know if any of teh intel ci accounts comment on nova?	02:22
*** nwkarsten has joined #openstack-infra		02:22
*** thorst has quit IRC		02:22
anteaya	yonglihe: I saw your reply to the mailing list thread	02:23
*** Goneri has quit IRC		02:23
anteaya	yonglihe: the nova ptl is trying to understand if any of the intel ci accounts are testing nova	02:23
anteaya	yonglihe: who would know the answer to that question?	02:23
yonglihe	i talked to him, and i know the SRIOV/PCI	02:24
yonglihe	and we escalate the question internally, so we get the answer quickly i think.	02:24
anteaya	yonglihe: wonderful, thank you	02:25
yonglihe	welcome, anteaya	02:25
*** mriedem has quit IRC		02:25
*** xyang1 has quit IRC		02:26
*** nwkarsten has quit IRC		02:26
*** esker has joined #openstack-infra		02:27
*** resker has joined #openstack-infra		02:29
*** markvoelker_ has quit IRC		02:31
*** salv-orlando has quit IRC		02:32
*** esker has quit IRC		02:32
*** markvoelker_ has joined #openstack-infra		02:32
*** esker has joined #openstack-infra		02:32
*** bpokorny has joined #openstack-infra		02:32
*** jamesmcarthur has quit IRC		02:34
*** resker has quit IRC		02:34
*** markvoel_ has joined #openstack-infra		02:36
*** mylu has joined #openstack-infra		02:36
*** asselin has joined #openstack-infra		02:36
*** salv-orlando has joined #openstack-infra		02:36
*** esker has quit IRC		02:37
*** jamesmcarthur has joined #openstack-infra		02:38
*** asselin__ has quit IRC		02:38
*** yamahata has quit IRC		02:38
*** markvoelker_ has quit IRC		02:40
*** resker has joined #openstack-infra		02:41
*** jamesmcarthur_ has joined #openstack-infra		02:43
*** kzaitsev_mb has joined #openstack-infra		02:43
*** jamesmcarthur has quit IRC		02:43
*** jamesmcarthur_ is now known as jamesmcarthur		02:43
*** mylu has quit IRC		02:44
*** mylu has joined #openstack-infra		02:44
*** esker___ has joined #openstack-infra		02:45
*** resker has quit IRC		02:46
*** resker has joined #openstack-infra		02:47
*** kzaitsev_mb has quit IRC		02:48
*** mylu_ has joined #openstack-infra		02:48
*** mylu has quit IRC		02:49
*** esker___ has quit IRC		02:50
*** resker has quit IRC		02:51
*** jamesmcarthur has quit IRC		02:52
*** salv-orlando has quit IRC		02:52
*** amrith is now known as _amrith_		02:53
*** smarcet has joined #openstack-infra		02:53
*** tqtran has joined #openstack-infra		02:54
*** esker_ has joined #openstack-infra		02:54
*** esker__ has joined #openstack-infra		02:56
*** dingyichen has joined #openstack-infra		02:57
*** tqtran has quit IRC		02:58
*** resker has joined #openstack-infra		02:59
openstackgerrit	sebastian marcet proposed openstack-infra/openstackid-resources: Fix on schedule null events https://review.openstack.org/299725	02:59
*** esker_ has quit IRC		03:00
openstackgerrit	Merged openstack-infra/openstackid-resources: Fix on schedule null events https://review.openstack.org/299725	03:00
*** esker__ has quit IRC		03:00
*** markvoelker has joined #openstack-infra		03:01
*** bpokorny has quit IRC		03:01
*** esker has joined #openstack-infra		03:02
*** fbo has quit IRC		03:02
*** achanda has quit IRC		03:03
*** markvoelker_ has joined #openstack-infra		03:04
*** resker has quit IRC		03:04
*** markvoel_ has quit IRC		03:04
*** resker has joined #openstack-infra		03:05
*** fbo has joined #openstack-infra		03:06
*** esker has quit IRC		03:06
*** hichihara has quit IRC		03:06
*** sdake has joined #openstack-infra		03:07
*** fawadkhaliq has joined #openstack-infra		03:08
*** markvoelker has quit IRC		03:08
*** sdake_ has quit IRC		03:10
*** resker has quit IRC		03:10
*** apoorvad has joined #openstack-infra		03:11
*** amitgandhinz has joined #openstack-infra		03:13
*** kzaitsev_mb has joined #openstack-infra		03:13
*** esker has joined #openstack-infra		03:14
*** amitgandhinz has quit IRC		03:18
*** kzaitsev_mb has quit IRC		03:18
*** esker has quit IRC		03:18
*** apoorvad has quit IRC		03:19
*** thorst has joined #openstack-infra		03:19
fungi	smarcet: since it looks like you're still awake, the new openstackid-dev running 14.04 is showing some strange output at https://openstackid-dev.openstack.org/	03:20
*** yuanying has quit IRC		03:21
fungi	smarcet: like the proxy isn't happening or something	03:21
fungi	i'm heading to sleep shortly, but can look at it more tomorrow	03:21
smarcet	fungi: yes looking at it :) tnx	03:22
smarcet	fungi: have a good nite !	03:22
fungi	you too, thanks!	03:22
*** sputnik13 has joined #openstack-infra		03:22
*** pkoniszewski has quit IRC		03:24
*** yamada-h has joined #openstack-infra		03:24
*** simonmcc_ has joined #openstack-infra		03:25
*** fawadkhaliq has quit IRC		03:26
*** bryan_att_ has joined #openstack-infra		03:26
*** kshort_away has joined #openstack-infra		03:26
*** mgkwill_ has joined #openstack-infra		03:26
*** Daviey_ has joined #openstack-infra		03:26
*** fawadkhaliq has joined #openstack-infra		03:26
*** warpc_ has joined #openstack-infra		03:26
*** ajo_ has joined #openstack-infra		03:27
*** jeblair_ has joined #openstack-infra		03:27
*** thorst has quit IRC		03:27
*** waynr1 has joined #openstack-infra		03:27
*** cbader_ has joined #openstack-infra		03:28
*** aspiers_ has joined #openstack-infra		03:28
*** jraim_ has joined #openstack-infra		03:29
*** yamada-h has quit IRC		03:29
*** xnox_ has joined #openstack-infra		03:29
*** arif-ali_ has joined #openstack-infra		03:30
*** krotscheck has joined #openstack-infra		03:30
*** wfoster_ has joined #openstack-infra		03:30
*** zaro has quit IRC		03:30
*** zxiiro_ has joined #openstack-infra		03:30
*** hamzy_ has joined #openstack-infra		03:31
*** mwhahaha_ has joined #openstack-infra		03:31
*** ogelbukh_ has joined #openstack-infra		03:31
*** dmellado_ has joined #openstack-infra		03:31
*** scottda_ has joined #openstack-infra		03:31
*** pkoniszewski has joined #openstack-infra		03:31
*** zaro has joined #openstack-infra		03:31
*** alkhodos has quit IRC		03:31
*** fawadkhaliq has quit IRC		03:32
*** rpodolyaka_ has joined #openstack-infra		03:32
*** fawadkhaliq has joined #openstack-infra		03:32
*** simonmcc has quit IRC		03:32
*** bryan_att has quit IRC		03:32
*** aspiers has quit IRC		03:32
*** mgkwill has quit IRC		03:32
*** kevinshort has quit IRC		03:32
*** kshort_away is now known as kevinshort		03:32
*** Daviey has quit IRC		03:32
*** warpc has quit IRC		03:32
*** xnox has quit IRC		03:32
*** dougwig has quit IRC		03:32
*** jeblair has quit IRC		03:32
*** ajo has quit IRC		03:32
*** fkautz has quit IRC		03:32
*** wfoster has quit IRC		03:32
*** madorn has quit IRC		03:32
*** waynr has quit IRC		03:32
*** cbader has quit IRC		03:32
*** ogelbukh has quit IRC		03:32
*** asettle has quit IRC		03:32
*** jraim has quit IRC		03:32
*** mwhahaha has quit IRC		03:32
*** hamzy has quit IRC		03:32
*** xnox_ is now known as xnox		03:32
*** zxiiro has quit IRC		03:32
*** krotscheck_dcm has quit IRC		03:32
*** scottda has quit IRC		03:32
*** dmellado has quit IRC		03:32
*** arif-ali has quit IRC		03:32
*** andymccr has quit IRC		03:32
*** cargonza has quit IRC		03:32
*** arif-ali_ is now known as arif-ali		03:32
*** onovy_ has joined #openstack-infra		03:33
*** asettle has joined #openstack-infra		03:33
*** aorourke_ has joined #openstack-infra		03:33
*** warpc_ is now known as warpc		03:33
*** peterstac_ has joined #openstack-infra		03:33
*** ogelbukh_ is now known as ogelbukh		03:33
*** mgkwill_ is now known as mgkwill		03:33
*** peterstac has quit IRC		03:33
*** fmccrthy has quit IRC		03:33
*** rvba has quit IRC		03:33
*** rpodolyaka has quit IRC		03:33
*** andrewbogott has quit IRC		03:33
*** serverascode has quit IRC		03:33
*** onovy has quit IRC		03:33
*** maximov_ has quit IRC		03:33
*** sshnaidm has quit IRC		03:33
*** lindsayk has quit IRC		03:33
*** electrical has quit IRC		03:33
*** mancdaz has quit IRC		03:33
*** aorourke has quit IRC		03:33
*** adreznec has quit IRC		03:33
*** fbo has quit IRC		03:33
*** Adri2000 has quit IRC		03:33
*** Hunner has quit IRC		03:33
*** mancdaz has joined #openstack-infra		03:33
*** onovy_ is now known as onovy		03:33
*** fbo_ has joined #openstack-infra		03:33
*** peterstac_ is now known as peterstac		03:33
*** simonmcc_ is now known as simonmcc		03:33
*** bryan_att_ is now known as bryan_att		03:33
*** Adri2000 has joined #openstack-infra		03:33
*** Adri2000 has quit IRC		03:33
*** Adri2000 has joined #openstack-infra		03:33
*** andymccr has joined #openstack-infra		03:33
*** madorn has joined #openstack-infra		03:33
*** rpodolyaka_ is now known as rpodolyaka		03:34
*** rvba has joined #openstack-infra		03:34
*** adreznec has joined #openstack-infra		03:34
*** fmccrthy has joined #openstack-infra		03:34
*** cargonza has joined #openstack-infra		03:34
*** jraim_ is now known as jraim		03:34
*** rvba has quit IRC		03:34
*** rvba has joined #openstack-infra		03:34
*** Jeffrey4l_ has joined #openstack-infra		03:34
*** waynr1 has quit IRC		03:34
*** mwhahaha_ is now known as mwhahaha		03:34
*** waynr has joined #openstack-infra		03:34
*** scottda_ is now known as scottda		03:34
*** sshnaidm has joined #openstack-infra		03:34
*** zxiiro_ is now known as zxiiro		03:35
*** Hunner has joined #openstack-infra		03:36
*** Hunner has quit IRC		03:36
*** Hunner has joined #openstack-infra		03:36
*** esker has joined #openstack-infra		03:36
*** maximov has joined #openstack-infra		03:36
*** Jeffrey4l has quit IRC		03:37
*** electrical has joined #openstack-infra		03:37
*** andrewbogott has joined #openstack-infra		03:38
*** serverascode has joined #openstack-infra		03:38
*** fkautz has joined #openstack-infra		03:39
*** dougwig has joined #openstack-infra		03:39
openstackgerrit	sebastian marcet proposed openstack-infra/puppet-openstackid: Fix Vhost (Apache2.4) https://review.openstack.org/299732	03:40
*** esker has quit IRC		03:40
*** bpokorny has joined #openstack-infra		03:40
*** esker has joined #openstack-infra		03:40
smarcet	fungi: if u are still awake https://review.openstack.org/#/c/299732/1 fix the issue	03:41
smarcet	fungi: nite!	03:41
*** nwkarsten has joined #openstack-infra		03:42
*** nwkarsten has quit IRC		03:43
*** smarcet has quit IRC		03:43
*** nwkarsten has joined #openstack-infra		03:43
*** links has joined #openstack-infra		03:44
*** esker has quit IRC		03:45
*** nwkarsten has quit IRC		03:48
*** ramishra has quit IRC		03:49
*** esker has joined #openstack-infra		03:50
*** ramishra has joined #openstack-infra		03:51
*** salv-orlando has joined #openstack-infra		03:52
*** salv-orlando has quit IRC		03:55
*** esker has quit IRC		03:55
openstackgerrit	sebastian marcet proposed openstack-infra/puppet-openstackid: Fix Vhost (Apache2.4) https://review.openstack.org/299732	03:55
*** smarcet has joined #openstack-infra		03:57
*** smarcet has quit IRC		03:57
*** mylu_ has quit IRC		03:58
*** fawadkhaliq has quit IRC		03:58
*** yamamoto_ has quit IRC		03:59
*** fawadkhaliq has joined #openstack-infra		03:59
*** sputnik13_ has joined #openstack-infra		04:00
*** fawadkhaliq has quit IRC		04:00
*** fawadkhaliq has joined #openstack-infra		04:01
*** fawadkhaliq has quit IRC		04:02
*** fawadkhaliq has joined #openstack-infra		04:02
*** achanda has joined #openstack-infra		04:04
*** yuanying has joined #openstack-infra		04:07
*** bpokorny has quit IRC		04:08
*** achanda has quit IRC		04:09
*** achanda has joined #openstack-infra		04:09
*** baoli has joined #openstack-infra		04:10
*** eil397 has joined #openstack-infra		04:13
*** amitgandhinz has joined #openstack-infra		04:14
*** kzaitsev_mb has joined #openstack-infra		04:14
*** baoli has quit IRC		04:14
*** salv-orlando has joined #openstack-infra		04:16
*** mylu has joined #openstack-infra		04:17
*** amitgandhinz has quit IRC		04:18
*** eil397 has quit IRC		04:19
*** kzaitsev_mb has quit IRC		04:21
*** markvoelker_ has quit IRC		04:21
*** harlowja_at_home has joined #openstack-infra		04:23
prometheanfire	ianw: before I was using diskimage-builder to build gentoo images it's what I used for network setup for openstack guests	04:23
prometheanfire	if that makes you feel better :P	04:23
*** salv-orlando has quit IRC		04:23
*** bpokorny has joined #openstack-infra		04:24
*** sridhar_ram has joined #openstack-infra		04:25
*** sdake_ has joined #openstack-infra		04:28
*** sridhar_ram1 has joined #openstack-infra		04:28
*** fawadkhaliq has quit IRC		04:29
*** fawadkhaliq has joined #openstack-infra		04:30
*** sdake has quit IRC		04:30
*** esker has joined #openstack-infra		04:30
*** sridhar_ram has quit IRC		04:30
*** baoli has joined #openstack-infra		04:31
*** dimtruck is now known as zz_dimtruck		04:32
*** mylu has quit IRC		04:34
*** Jeffrey4l_ has quit IRC		04:34
*** esker has quit IRC		04:35
*** fedexo has joined #openstack-infra		04:36
*** baoli has quit IRC		04:36
*** maishsk has quit IRC		04:38
*** yamada-h has joined #openstack-infra		04:39
*** yamamoto_ has joined #openstack-infra		04:39
*** camunoz has joined #openstack-infra		04:40
ianw	prometheanfire: i believe you	04:40
*** mylu has joined #openstack-infra		04:42
*** nadya has joined #openstack-infra		04:43
prometheanfire	if it helps even more it's listed here https://github.com/prometheanfire/gentoo-cloud-prep/blob/master/02-catalyst-that-shit.sh#L70	04:47
*** bpokorny has quit IRC		04:47
prometheanfire	and here https://github.com/prometheanfire/gentoo-cloud-prep/blob/master/files/prep.sh#L32-L35	04:47
*** IlyaG has quit IRC		04:47
*** yamahata has joined #openstack-infra		04:48
openstackgerrit	Merged openstack-infra/project-config: midonet: Make tempest jobs voting https://review.openstack.org/298142	04:56
*** abregman has joined #openstack-infra		04:58
*** rcernin has joined #openstack-infra		04:59
*** harlowja_at_home has quit IRC		05:05
*** lindsayk has joined #openstack-infra		05:05
*** yamada-h has quit IRC		05:08
*** yamada-h has joined #openstack-infra		05:09
*** nikhil has quit IRC		05:13
*** yamada-h has quit IRC		05:13
*** sputnik13_ has quit IRC		05:13
*** nikhil has joined #openstack-infra		05:14
*** sdake_ is now known as sdake		05:14
*** jaosorior has joined #openstack-infra		05:14
*** amitgandhinz has joined #openstack-infra		05:14
*** fawadkhaliq has quit IRC		05:15
*** fawadkhaliq has joined #openstack-infra		05:16
*** kzaitsev_mb has joined #openstack-infra		05:18
*** amitgandhinz has quit IRC		05:19
*** nadya has quit IRC		05:20
*** salv-orlando has joined #openstack-infra		05:23
*** kzaitsev_mb has quit IRC		05:23
sc`	clarkb: thanks for reminding me about begin / rescue / end. finally got some service logs. if only the error i finally have wasn't so nonspecific :)	05:23
*** jerryz has joined #openstack-infra		05:25
*** sridhar_ram1 has quit IRC		05:26
*** Sukhdev has joined #openstack-infra		05:26
*** salv-orlando has quit IRC		05:28
*** sputnik13_ has joined #openstack-infra		05:29
*** annegentle has joined #openstack-infra		05:29
*** mtanino has quit IRC		05:31
*** sputnik13_ has quit IRC		05:32
*** baoli has joined #openstack-infra		05:32
*** thorst has joined #openstack-infra		05:32
openstackgerrit	Merged openstack-infra/tripleo-ci: Use Fedora 23 atomic in container gate https://review.openstack.org/293108	05:33
*** esker has joined #openstack-infra		05:33
*** sputnik13_ has joined #openstack-infra		05:33
*** annegentle has quit IRC		05:34
*** baoli has quit IRC		05:37
*** thorst has quit IRC		05:37
*** fawadkhaliq has quit IRC		05:37
*** yamada-h has joined #openstack-infra		05:39
*** Jeffrey4l_ has joined #openstack-infra		05:39
*** mylu has quit IRC		05:39
*** fawadkhaliq has joined #openstack-infra		05:39
*** sputnik13_ has quit IRC		05:40
*** esker has quit IRC		05:42
*** esikache1 has joined #openstack-infra		05:46
*** pcaruana has quit IRC		05:49
*** alexey_weyl has joined #openstack-infra		05:50
alexey_weyl	Hello,	05:50
*** mtanino has joined #openstack-infra		05:50
*** esikache1 has quit IRC		05:50
alexey_weyl	I have an issue with jenkins in gerrit.	05:50
alexey_weyl	I have unskipped my test in the Vitrage project, and it passes tox, but zuul returns an error for py27.	05:51
alexey_weyl	The error message doesn't have any useful information about the error, and I would like to ask you what can I do about it?	05:52
alexey_weyl	You can find it here:	05:52
alexey_weyl	https://review.openstack.org/#/c/299408/	05:52
alexey_weyl	http://logs.openstack.org/08/299408/1/check/gate-vitrage-python27/97c7a1a/	05:52
*** oanson has joined #openstack-infra		05:53
alexey_weyl	p.s. I am starting another thread in test, and I think that what causes it, but still my tox passes the test and it looks fine.	05:53
*** Jeffrey4l_ has quit IRC		05:54
*** k4n0 has joined #openstack-infra		05:54
*** k4n0 has quit IRC		05:54
*** e0ne has joined #openstack-infra		05:56
*** ihrachys has joined #openstack-infra		05:57
*** Sukhdev has quit IRC		05:57
*** flepied has joined #openstack-infra		05:58
*** Swami has joined #openstack-infra		05:59
*** fawadkhaliq has quit IRC		05:59
*** fawadkhaliq has joined #openstack-infra		06:00
*** torandu has quit IRC		06:02
*** mtanino has quit IRC		06:03
*** aeng has quit IRC		06:03
*** torandu has joined #openstack-infra		06:04
*** fawadkhaliq has quit IRC		06:04
*** ramishra has quit IRC		06:05
*** binbincong has quit IRC		06:06
*** ramishra has joined #openstack-infra		06:07
*** esikache1 has joined #openstack-infra		06:07
AJaeger	alexey_weyl: "Build timed out (after 50 minutes). Marking the build as failed."	06:07
*** fawadkhaliq has joined #openstack-infra		06:08
AJaeger	that can be a problem of the node or some indeterminism in your tests/repo	06:08
*** roxanaghe has joined #openstack-infra		06:08
*** nadya has joined #openstack-infra		06:08
*** ihrachys has quit IRC		06:11
*** amitgandhinz has joined #openstack-infra		06:15
*** woodster_ has joined #openstack-infra		06:18
*** fawadkhaliq has quit IRC		06:19
*** kzaitsev_mb has joined #openstack-infra		06:19
*** fawadkhaliq has joined #openstack-infra		06:19
*** roxanaghe has quit IRC		06:19
*** binbincong has joined #openstack-infra		06:19
*** amitgandhinz has quit IRC		06:20
*** aeng has joined #openstack-infra		06:21
*** mikelk has joined #openstack-infra		06:21
*** daemontool has joined #openstack-infra		06:22
*** yamahata has quit IRC		06:23
*** mohankumar has joined #openstack-infra		06:23
*** e0ne has quit IRC		06:24
*** Swami has quit IRC		06:24
*** kzaitsev_mb has quit IRC		06:25
openstackgerrit	Sachi King proposed openstack-dev/pbr: Sort 'pbr freeze' output https://review.openstack.org/299766	06:26
AJaeger	Nakato: nice! ^ Thanks!	06:28
*** tphummel has quit IRC		06:28
openstackgerrit	OpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/299769	06:28
*** nadya has quit IRC		06:28
*** asettle has quit IRC		06:28
*** mikelk has quit IRC		06:29
*** annegentle has joined #openstack-infra		06:30
alexey_weyl	AJaeger: What kind of indeterminsm? Tox passed this test every time for 20-30 times.	06:31
alexey_weyl	The only thing I can think of is that the thread that I am opening is not being started, which can cause the indeterminsm. What can I do to check if the thread is running?	06:32
*** fedexo has quit IRC		06:32
openstackgerrit	Mehdi Abaakouk (sileht) proposed openstack/requirements: Bump gnocchiclient version to 2.2.0 https://review.openstack.org/299770	06:33
*** nadya has joined #openstack-infra		06:33
*** esikache1 has quit IRC		06:33
*** thorst has joined #openstack-infra		06:34
*** mikelk has joined #openstack-infra		06:34
openstackgerrit	OpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/288849	06:35
*** annegentle has quit IRC		06:35
*** nadya has quit IRC		06:35
*** nadya has joined #openstack-infra		06:35
Nakato	AJaeger: Thanks. :)	06:36
*** sdake_ has joined #openstack-infra		06:37
*** sdake has quit IRC		06:40
*** thorst has quit IRC		06:42
*** kzaitsev_mb has joined #openstack-infra		06:43
*** sdake has joined #openstack-infra		06:45
*** yamada-h_ has joined #openstack-infra		06:45
*** abregman_ has joined #openstack-infra		06:45
*** aeng has quit IRC		06:46
AJaeger	alexey_weyl: Just looking at the timeout, there are these two explanations for me - I didn't check code.	06:46
*** abregman has quit IRC		06:46
AJaeger	alexey_weyl: if it's reproduceable in the gate, one way is to hold a node and access it. We can also check which cloud this runs in and see whether there is a corrolation if that happens more often...	06:48
*** salv-orlando has joined #openstack-infra		06:48
*** sdake_ has quit IRC		06:48
*** yamada-h has quit IRC		06:48
alexey_weyl	What I am trying to understand is how can I check or print log in the ZUUL \ Jenkins environment	06:48
*** fawadkhaliq has quit IRC		06:48
alexey_weyl	Yes, it occurs all the time in the gate, and never in my devstack environment	06:48
*** fawadkhaliq has joined #openstack-infra		06:49
*** tesseract has joined #openstack-infra		06:49
AJaeger	alexey_weyl: wait for yolanda to show up - or later another infra-root (during US time) and ask them to hold a node for you. Then you can ssh into such a node and investigate...	06:49
*** tesseract is now known as Guest11037		06:49
AJaeger	But perhaps somebody else has a better idea, you might want to ask in #openstack-qa as well.	06:50
*** baoli has joined #openstack-infra		06:50
alexey_weyl	sound great! Thank you very much	06:50
*** salv-orlando has quit IRC		06:51
openstackgerrit	Shamail Tahir proposed openstack-infra/irc-meetings: Adding AsiaPac Regional PWG Meeting https://review.openstack.org/299776	06:51
*** esikache1 has joined #openstack-infra		06:52
*** shashank_hegde has quit IRC		06:53
*** alexey_weyl has quit IRC		06:54
*** baoli has quit IRC		06:55
openstackgerrit	Merged openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/299769	06:56
*** esikache1 has quit IRC		06:56
*** ihrachys has joined #openstack-infra		06:58
*** binbincong has quit IRC		07:03
openstackgerrit	Vasyl Saienko proposed openstack-infra/devstack-gate: DO NOT REVIEW https://review.openstack.org/299273	07:03
*** daemontool has quit IRC		07:04
*** abregman_ is now known as abregman		07:05
*** torandu has quit IRC		07:06
*** Daisy_ has joined #openstack-infra		07:07
*** yamada-h_ has quit IRC		07:08
*** yamada-h has joined #openstack-infra		07:09
*** akaszuba has joined #openstack-infra		07:10
*** Daisy has quit IRC		07:10
*** \|-paul-\| has joined #openstack-infra		07:10
*** maishsk has joined #openstack-infra		07:11
*** Daisy has joined #openstack-infra		07:11
*** yamada-h has quit IRC		07:11
*** Daisy__ has joined #openstack-infra		07:12
openstackgerrit	Derek Higgins proposed openstack-infra/tripleo-ci: Fix image caching logic https://review.openstack.org/299537	07:12
openstackgerrit	Derek Higgins proposed openstack-infra/tripleo-ci: Pre install packages on the instack image https://review.openstack.org/299538	07:12
*** Daisy___ has joined #openstack-infra		07:12
*** zeih has joined #openstack-infra		07:13
*** Daisy_ has quit IRC		07:14
*** torandu has joined #openstack-infra		07:14
*** e0ne has joined #openstack-infra		07:15
*** Daisy has quit IRC		07:16
*** Daisy___ has quit IRC		07:16
*** amitgandhinz has joined #openstack-infra		07:16
*** Daisy__ has quit IRC		07:16
*** Daisy has joined #openstack-infra		07:16
*** mrmartin has joined #openstack-infra		07:16
*** binbincong has joined #openstack-infra		07:17
*** Daisy_ has joined #openstack-infra		07:18
*** e0ne has quit IRC		07:18
*** Daisy has quit IRC		07:18
*** Daisy has joined #openstack-infra		07:20
*** amitgandhinz has quit IRC		07:20
*** esikache1 has joined #openstack-infra		07:21
*** fhubik has joined #openstack-infra		07:22
*** esikache1 has quit IRC		07:23
*** esikachev has joined #openstack-infra		07:23
*** Daisy_ has quit IRC		07:23
*** nmagnezi has joined #openstack-infra		07:24
*** Daisy has quit IRC		07:24
*** jlanoux has joined #openstack-infra		07:26
*** jlanoux_ has joined #openstack-infra		07:28
*** jlanoux has quit IRC		07:28
*** arxcruz has joined #openstack-infra		07:30
*** annegentle has joined #openstack-infra		07:31
*** mrmartin has quit IRC		07:32
*** stevemar has quit IRC		07:33
*** annegentle has quit IRC		07:35
*** kzaitsev_mb has quit IRC		07:38
openstackgerrit	Sagi Shnaidman proposed openstack-infra/tripleo-ci: Run tempest tests with non-HA jobs https://review.openstack.org/297038	07:39
*** jtomasek has joined #openstack-infra		07:40
*** thorst has joined #openstack-infra		07:41
*** pcaruana has joined #openstack-infra		07:41
*** thorst has quit IRC		07:47
*** permalac has joined #openstack-infra		07:48
*** shardy has joined #openstack-infra		07:48
*** armax_ has joined #openstack-infra		07:49
*** hashar has joined #openstack-infra		07:49
*** jerryz has quit IRC		07:51
*** annegentle has joined #openstack-infra		07:52
*** armax has quit IRC		07:52
*** armax_ is now known as armax		07:52
*** ihrachys has quit IRC		07:53
*** jistr has joined #openstack-infra		07:53
*** yaume has joined #openstack-infra		07:53
*** matrohon has joined #openstack-infra		07:54
*** dizquierdo has joined #openstack-infra		07:56
*** annegentle has quit IRC		07:56
*** sflanigan has quit IRC		07:57
*** fawadkhaliq has quit IRC		07:59
*** jordanP has joined #openstack-infra		08:01
*** fawadkhaliq has joined #openstack-infra		08:01
openstackgerrit	Luigi Toscano proposed openstack-infra/project-config: WIP Revert "[sahara-tests] create and merge, disable tests" https://review.openstack.org/299652	08:03
openstackgerrit	Luigi Toscano proposed openstack-infra/project-config: [sahara-tests] create and merge, disable tests https://review.openstack.org/299407	08:03
*** fawadkhaliq has quit IRC		08:03
*** fawadkhaliq has joined #openstack-infra		08:04
*** permalac has quit IRC		08:06
*** salv-orlando has joined #openstack-infra		08:07
*** sshnaidm has quit IRC		08:09
*** salv-orlando has quit IRC		08:09
*** zhurong has quit IRC		08:11
*** zhurong has joined #openstack-infra		08:11
*** briancli1e has quit IRC		08:12
*** kzaitsev_mb has joined #openstack-infra		08:13
*** briancline has joined #openstack-infra		08:13
*** Daisy has joined #openstack-infra		08:14
*** Daisy has quit IRC		08:15
*** jcoufal has joined #openstack-infra		08:15
*** Daisy has joined #openstack-infra		08:15
*** sdake_ has joined #openstack-infra		08:17
*** amitgandhinz has joined #openstack-infra		08:17
*** ifarkas has joined #openstack-infra		08:17
*** salv-orlando has joined #openstack-infra		08:18
*** sdake has quit IRC		08:19
*** maishsk has quit IRC		08:20
*** amitgandhinz has quit IRC		08:21
*** sdake has joined #openstack-infra		08:22
*** salv-orlando has quit IRC		08:22
*** sdake_ has quit IRC		08:24
*** esikachev has quit IRC		08:25
*** permalac has joined #openstack-infra		08:25
*** woodster_ has quit IRC		08:27
*** Daisy_ has joined #openstack-infra		08:27
*** e0ne has joined #openstack-infra		08:28
*** e0ne has quit IRC		08:29
*** ihrachys has joined #openstack-infra		08:30
*** Daisy has quit IRC		08:30
*** achanda has quit IRC		08:30
*** Daisy_ has quit IRC		08:31
*** sfinucan has joined #openstack-infra		08:33
*** stevemar has joined #openstack-infra		08:33
*** achanda has joined #openstack-infra		08:34
*** rossella_s has quit IRC		08:34
*** e0ne has joined #openstack-infra		08:34
*** rossella_s has joined #openstack-infra		08:35
*** e0ne has quit IRC		08:35
*** akshai_ has quit IRC		08:37
*** cody-somerville has quit IRC		08:37
*** fawadkhaliq has quit IRC		08:37
*** stevemar has quit IRC		08:39
*** fawadkhaliq has joined #openstack-infra		08:40
openstackgerrit	Vincent Françoise proposed openstack-infra/project-config: Watcher multinode - disable plugin in subnode https://review.openstack.org/299809	08:41
*** daemontool has joined #openstack-infra		08:41
*** esikachev has joined #openstack-infra		08:42
*** gildub has quit IRC		08:43
*** kushal has quit IRC		08:44
*** yamada-h has joined #openstack-infra		08:44
*** thorst has joined #openstack-infra		08:45
*** daemontool has quit IRC		08:48
*** achanda has quit IRC		08:48
*** achanda has joined #openstack-infra		08:49
*** yamada-h has quit IRC		08:50
*** baoli has joined #openstack-infra		08:51
*** zeih has quit IRC		08:51
*** thorst has quit IRC		08:52
*** cody-somerville has joined #openstack-infra		08:52
*** achanda has quit IRC		08:52
*** baoli has quit IRC		08:55
*** daemontool has joined #openstack-infra		08:58
*** achanda has joined #openstack-infra		08:59
*** dizquierdo has quit IRC		09:00
*** abregman has quit IRC		09:00
*** Daisy has joined #openstack-infra		09:00
*** Daisy_ has joined #openstack-infra		09:01
*** Daisy_ has quit IRC		09:03
*** fawadkhaliq has quit IRC		09:05
*** sshnaidm has joined #openstack-infra		09:05
*** Daisy has quit IRC		09:06
*** torandu has quit IRC		09:06
*** torandu has joined #openstack-infra		09:07
*** rlandy has quit IRC		09:09
*** dtantsur\|afk is now known as dtantsur		09:09
*** fawadkhaliq has joined #openstack-infra		09:09
*** asettle has joined #openstack-infra		09:10
*** asettle has quit IRC		09:10
*** Daisy has joined #openstack-infra		09:10
*** ihrachys has quit IRC		09:11
*** Daisy has quit IRC		09:11
*** dtardivel has joined #openstack-infra		09:14
*** fhorvath has joined #openstack-infra		09:14
*** esikachev has quit IRC		09:15
*** abregman has joined #openstack-infra		09:15
*** zeih has joined #openstack-infra		09:16
*** [1]Thelo has joined #openstack-infra		09:16
*** amitgandhinz has joined #openstack-infra		09:17
*** mrmartin has joined #openstack-infra		09:18
*** derekh has joined #openstack-infra		09:19
*** eliqiao has quit IRC		09:19
*** Thelo has quit IRC		09:19
*** [1]Thelo is now known as Thelo		09:19
openstackgerrit	Anton Arefiev proposed openstack-infra/project-config: Add Ironic Inspector basic tempest job https://review.openstack.org/299821	09:22
*** amitgandhinz has quit IRC		09:22
*** salv-orlando has joined #openstack-infra		09:23
*** esikachev has joined #openstack-infra		09:24
openstackgerrit	Rossella Sblendido proposed openstack-infra/reviewday: Add a link for Neutron Gerrit dashboard https://review.openstack.org/298779	09:25
*** fawadkhaliq has quit IRC		09:25
*** placko_ has joined #openstack-infra		09:25
*** electrofelix has joined #openstack-infra		09:26
*** achanda has quit IRC		09:26
*** Jeffrey4l_ has joined #openstack-infra		09:26
openstackgerrit	Aparna proposed openstack/diskimage-builder: Support to add self-signed certificate in ironic-agent https://review.openstack.org/295203	09:26
*** fawadkhaliq has joined #openstack-infra		09:26
*** Daisy has joined #openstack-infra		09:29
Zara	anteaya: I'm around now	09:30
*** e0ne has joined #openstack-infra		09:30
*** sdake_ has joined #openstack-infra		09:32
*** sdake has quit IRC		09:32
*** sdake has joined #openstack-infra		09:33
*** skolekonov\|away is now known as skolekonov		09:34
*** exploreshaifali has joined #openstack-infra		09:36
*** mrmartin has quit IRC		09:36
*** sdake_ has quit IRC		09:36
*** asettle has joined #openstack-infra		09:37
*** e0ne has quit IRC		09:37
*** salv-orlando has quit IRC		09:39
*** Daisy has quit IRC		09:40
*** Daisy has joined #openstack-infra		09:40
*** fawadkhaliq has quit IRC		09:41
*** e0ne has joined #openstack-infra		09:42
*** fawadkhaliq has joined #openstack-infra		09:42
*** daemontool has quit IRC		09:43
*** daemontool has joined #openstack-infra		09:44
*** fawadkhaliq has quit IRC		09:44
*** Daisy has quit IRC		09:44
*** tosky has joined #openstack-infra		09:45
*** nadya has quit IRC		09:45
*** _degorenko\|afk is now known as degorenko		09:47
*** thorst has joined #openstack-infra		09:50
*** fawadkhaliq has joined #openstack-infra		09:50
*** nmagnezi has quit IRC		09:50
*** Qiming has quit IRC		09:52
openstackgerrit	Sean M. Collins proposed openstack-infra/project-config: Make grenade-dsvm-neutron-multinode voting https://review.openstack.org/299843	09:54
*** cody-somerville has quit IRC		09:54
*** thorst has quit IRC		09:56
*** kzaitsev_mb has quit IRC		09:56
openstackgerrit	Sean M. Collins proposed openstack-infra/project-config: Move gate-grenade-dsvm-neutron-dvr-multinode into check queue https://review.openstack.org/299844	09:56
*** gnuoy_ has joined #openstack-infra		09:59
*** gnuoy_ has quit IRC		09:59
*** gnuoy_ has joined #openstack-infra		09:59
*** esikachev has quit IRC		10:00
*** cdent has joined #openstack-infra		10:01
*** eliqiao has joined #openstack-infra		10:01
*** kzaitsev_mb has joined #openstack-infra		10:02
*** verdurin has quit IRC		10:03
*** sdake has quit IRC		10:03
*** sdague has joined #openstack-infra		10:03
*** verdurin has joined #openstack-infra		10:04
*** nmagnezi has joined #openstack-infra		10:04
*** kzaitsev_mb has quit IRC		10:07
*** electrofelix has quit IRC		10:08
*** electrofelix has joined #openstack-infra		10:09
*** fawadkhaliq has quit IRC		10:09
*** ihrachys has joined #openstack-infra		10:10
*** fawadkhaliq has joined #openstack-infra		10:10
*** gnuoy_ has quit IRC		10:12
*** electrofelix has quit IRC		10:13
*** electrofelix has joined #openstack-infra		10:13
*** e0ne has quit IRC		10:15
openstackgerrit	Derek Higgins proposed openstack-infra/tripleo-ci: ssh less https://review.openstack.org/295973	10:16
*** e0ne has joined #openstack-infra		10:17
*** fawadkhaliq has quit IRC		10:18
*** amitgandhinz has joined #openstack-infra		10:19
*** fawadkhaliq has joined #openstack-infra		10:19
*** zeih has quit IRC		10:19
*** e0ne has quit IRC		10:20
*** e0ne has joined #openstack-infra		10:21
*** zeih has joined #openstack-infra		10:22
*** e0ne has quit IRC		10:22
*** e0ne has joined #openstack-infra		10:23
*** amitgandhinz has quit IRC		10:23
*** e0ne has quit IRC		10:24
*** fawadkhaliq has quit IRC		10:25
*** skraynev has quit IRC		10:27
*** skraynev has joined #openstack-infra		10:27
*** ociuhandu has quit IRC		10:30
*** electrofelix has quit IRC		10:31
*** electrofelix has joined #openstack-infra		10:31
*** zeih has quit IRC		10:33
*** asettle has quit IRC		10:34
bookwar	hi, is it possible to disable e-mails for a certain gerrit user? We have third-party ci user to comment on reviews and it gets notifications for all changes in all those reviews	10:34
*** asettle has joined #openstack-infra		10:35
*** stevemar has joined #openstack-infra		10:35
*** dizquierdo has joined #openstack-infra		10:37
*** fawadkhaliq has joined #openstack-infra		10:39
*** stevemar has quit IRC		10:39
csaba	AJaeger: can you please help me out with a strange issue about glusterfs manila jobs (gate-manila-tempest-dsvm-glusterfs, gate-manila-tempest-dsvm-glusterfs-native)? They are ending up in UNSTABLE because scp-ing the log files fails. This can be observed with any recent manila change, eg. https://review.openstack.org/#/c/299322/	10:44
*** permalac has quit IRC		10:44
*** yamamoto_ has quit IRC		10:47
*** Qiming has joined #openstack-infra		10:49
*** fawadkhaliq has quit IRC		10:49
*** tpsilva has joined #openstack-infra		10:51
*** vgridnev has joined #openstack-infra		10:51
*** salv-orlando has joined #openstack-infra		10:51
*** doug-fish has joined #openstack-infra		10:51
*** baoli has joined #openstack-infra		10:52
*** dmellado_ is now known as dmellado		10:52
*** thorst has joined #openstack-infra		10:54
openstackgerrit	Anton Arefiev proposed openstack-infra/project-config: Add Ironic Inspector basic tempest job https://review.openstack.org/299821	10:55
*** baoli has quit IRC		10:57
*** esker has joined #openstack-infra		10:57
*** esikachev has joined #openstack-infra		10:58
*** resker has joined #openstack-infra		10:58
*** abregman is now known as abregman\|brb		11:01
*** ociuhandu has joined #openstack-infra		11:01
*** esker_ has joined #openstack-infra		11:01
*** thorst has quit IRC		11:01
*** esker has quit IRC		11:01
esikachev	hi! infra-cores need merge https://review.openstack.org/#/c/299407/	11:02
*** esker has joined #openstack-infra		11:02
*** resker has quit IRC		11:02
openstackgerrit	Beth Elwell proposed openstack-infra/storyboard-webclient: Added comments to dashboard.html https://review.openstack.org/299856	11:03
*** resker has joined #openstack-infra		11:03
*** cdent has left #openstack-infra		11:05
*** esker_ has quit IRC		11:06
*** esker has quit IRC		11:07
*** esker has joined #openstack-infra		11:08
*** resker has quit IRC		11:08
*** resker has joined #openstack-infra		11:10
*** _amrith_ is now known as amrith		11:11
*** smarcet has joined #openstack-infra		11:12
*** esker has quit IRC		11:12
*** fhorvath has quit IRC		11:12
*** esker_ has joined #openstack-infra		11:13
*** kgiusti has left #openstack-infra		11:13
*** esker__ has joined #openstack-infra		11:14
*** cdent has joined #openstack-infra		11:14
*** resker has quit IRC		11:15
*** resker has joined #openstack-infra		11:15
*** esker___ has joined #openstack-infra		11:16
*** e0ne has joined #openstack-infra		11:17
*** kzaitsev_mb has joined #openstack-infra		11:17
*** esker_ has quit IRC		11:17
*** julim has quit IRC		11:18
*** esker has joined #openstack-infra		11:18
*** e0ne has quit IRC		11:18
*** esker__ has quit IRC		11:19
openstackgerrit	Merged openstack-infra/tripleo-ci: Clarify location of large output log files https://review.openstack.org/296767	11:19
*** esker_ has joined #openstack-infra		11:19
*** amitgandhinz has joined #openstack-infra		11:19
*** resker has quit IRC		11:20
*** resker has joined #openstack-infra		11:20
*** esker___ has quit IRC		11:21
*** dkehn has quit IRC		11:21
*** abregman\|brb is now known as abregman		11:21
*** dkehn has joined #openstack-infra		11:21
*** julim_ has joined #openstack-infra		11:22
*** dstanek has quit IRC		11:22
*** nibalize1 has joined #openstack-infra		11:22
*** jlvillal_ has joined #openstack-infra		11:22
*** adreznec has quit IRC		11:22
*** rkukura has quit IRC		11:22
*** nibalizer has quit IRC		11:22
*** _ody has quit IRC		11:22
*** redrobot has quit IRC		11:22
*** andymccr has quit IRC		11:22
*** fabo has quit IRC		11:22
*** jlvillal has quit IRC		11:22
*** hashar has quit IRC		11:22
*** bigjools has quit IRC		11:22
*** dhellmann has quit IRC		11:22
*** _ody_ has joined #openstack-infra		11:22
*** fabo_ has joined #openstack-infra		11:22
*** electrichead has joined #openstack-infra		11:22
*** dhellmann_ has joined #openstack-infra		11:22
*** andymccr_ has joined #openstack-infra		11:23
*** esker has quit IRC		11:23
openstackgerrit	Derek Higgins proposed openstack-infra/tripleo-ci: ssh less https://review.openstack.org/295973	11:23
*** adreznec has joined #openstack-infra		11:23
*** e0ne has joined #openstack-infra		11:23
*** dstanek has joined #openstack-infra		11:23
*** andymccr_ is now known as andymccr		11:23
*** dizquierdo has quit IRC		11:23
*** jlvillal_ is now known as jlvillal		11:23
*** bigjools has joined #openstack-infra		11:24
*** bigjools has quit IRC		11:24
*** bigjools has joined #openstack-infra		11:24
*** amitgandhinz has quit IRC		11:24
*** esker_ has quit IRC		11:24
*** jlvillal is now known as Guest87578		11:24
openstackgerrit	Vasyl Saienko proposed openstack-infra/project-config: Increase OS_TEST_TIMEOUT for ironic-multitenant https://review.openstack.org/298569	11:24
*** claudiub has joined #openstack-infra		11:25
*** hashar has joined #openstack-infra		11:25
*** annegentle has joined #openstack-infra		11:25
*** resker has quit IRC		11:25
*** e0ne has quit IRC		11:26
*** dhellmann_ is now known as dhellmann		11:26
*** resker has joined #openstack-infra		11:26
openstackgerrit	Anton Arefiev proposed openstack-infra/project-config: Add Ironic Inspector basic tempest job https://review.openstack.org/299821	11:28
*** esker has joined #openstack-infra		11:29
*** annegentle has quit IRC		11:29
*** esker_ has joined #openstack-infra		11:31
*** jiggalanator has joined #openstack-infra		11:31
*** resker has quit IRC		11:32
*** resker has joined #openstack-infra		11:32
*** esker has quit IRC		11:33
*** mrmartin has joined #openstack-infra		11:34
*** netsin has quit IRC		11:34
*** esker has joined #openstack-infra		11:34
*** e0ne has joined #openstack-infra		11:35
*** esker___ has joined #openstack-infra		11:35
*** esker_ has quit IRC		11:35
*** resker has quit IRC		11:37
*** resker has joined #openstack-infra		11:37
*** nmagnezi has quit IRC		11:38
*** esker has quit IRC		11:39
*** cdent has left #openstack-infra		11:39
*** esker has joined #openstack-infra		11:39
*** esker___ has quit IRC		11:40
*** esker_ has joined #openstack-infra		11:40
*** thorst has joined #openstack-infra		11:41
*** e0ne has quit IRC		11:41
*** esker__ has joined #openstack-infra		11:42
*** resker has quit IRC		11:42
*** ldnunes has joined #openstack-infra		11:42
*** gordc has joined #openstack-infra		11:43
*** esker has quit IRC		11:44
*** esker_ has quit IRC		11:45
*** esker has joined #openstack-infra		11:46
*** esker__ has quit IRC		11:47
*** resker has joined #openstack-infra		11:47
*** yamamoto_ has joined #openstack-infra		11:48
*** nmagnezi has joined #openstack-infra		11:48
*** esker_ has joined #openstack-infra		11:48
openstackgerrit	Sagi Shnaidman proposed openstack-infra/tripleo-ci: Run tempest tests with non-HA jobs https://review.openstack.org/297038	11:49
*** esker has quit IRC		11:51
*** flepied has quit IRC		11:51
*** resker has quit IRC		11:52
*** esker_ has quit IRC		11:53
*** rfolco has joined #openstack-infra		11:53
*** weshay has joined #openstack-infra		11:53
AJaeger	csaba: this might be related to a change from our Jenkins update a week (or two?) ago.	11:53
AJaeger	csaba: I see "(Permission denied)" - old Jenkins ignored files it cannot access, new one is unhappy as you notice	11:53
AJaeger	csaba: So, we need to find a way to change permissions on these files - or not upload them. Better talk with the rest of the team on best solution. I think clarkb has been looking in some of this.	11:54
*** flepied has joined #openstack-infra		11:56
*** dizquierdo has joined #openstack-infra		11:58
*** esker has joined #openstack-infra		11:58
*** e0ne has joined #openstack-infra		11:59
*** asettle has quit IRC		11:59
*** e0ne has quit IRC		12:00
*** resker has joined #openstack-infra		12:00
*** annegentle has joined #openstack-infra		12:00
*** esker_ has joined #openstack-infra		12:02
*** esker has quit IRC		12:03
*** kgiusti has joined #openstack-infra		12:04
*** \|-paul-\| has quit IRC		12:05
*** esker has joined #openstack-infra		12:05
*** annegentle has quit IRC		12:05
*** resker has quit IRC		12:05
*** resker has joined #openstack-infra		12:06
*** esker_ has quit IRC		12:06
*** nadya has joined #openstack-infra		12:07
*** yamamoto_ has quit IRC		12:07
*** esker_ has joined #openstack-infra		12:08
*** esker has quit IRC		12:09
*** aysyd has joined #openstack-infra		12:09
*** esker has joined #openstack-infra		12:10
*** resker has quit IRC		12:10
*** ayoung has quit IRC		12:10
*** sdague_ has joined #openstack-infra		12:11
*** e0ne has joined #openstack-infra		12:11
*** aysyd has quit IRC		12:11
*** sfinucan has quit IRC		12:11
*** dprince has joined #openstack-infra		12:12
*** EricGonczer_ has joined #openstack-infra		12:12
*** e0ne has quit IRC		12:13
pabelanger	morning	12:13
*** esker_ has quit IRC		12:13
AJaeger	morning, pabelanger !	12:14
*** resker has joined #openstack-infra		12:14
*** esker has quit IRC		12:14
*** esker has joined #openstack-infra		12:15
*** aysyd has joined #openstack-infra		12:16
openstackgerrit	Jim Rollenhagen proposed openstack-infra/project-config: Ironic: always install tempest plugin from master https://review.openstack.org/299888	12:16
*** esker_ has joined #openstack-infra		12:16
*** pradk has joined #openstack-infra		12:16
*** esker__ has joined #openstack-infra		12:17
*** \|-paul-\| has joined #openstack-infra		12:17
*** sfinucan has joined #openstack-infra		12:18
*** yamamoto_ has joined #openstack-infra		12:18
*** resker has quit IRC		12:18
*** resker has joined #openstack-infra		12:19
*** esker has quit IRC		12:19
*** amitgandhinz has joined #openstack-infra		12:20
*** esker has joined #openstack-infra		12:20
*** esker_ has quit IRC		12:21
*** annegentle has joined #openstack-infra		12:21
*** esker__ has quit IRC		12:22
*** resker has quit IRC		12:24
openstackgerrit	Andreas Jaeger proposed openstack-infra/project-config: Fix releasenotes translation https://review.openstack.org/299892	12:25
AJaeger	project-config cores, this change is needed to fix syncing of translations for openstack-manuals, please approve ^	12:25
*** resker has joined #openstack-infra		12:25
*** amitgandhinz has quit IRC		12:25
*** esker has quit IRC		12:25
*** annegentle has quit IRC		12:26
*** baoli has joined #openstack-infra		12:27
*** baoli has quit IRC		12:27
*** nadya has quit IRC		12:27
*** baoli has joined #openstack-infra		12:27
*** esker_ has joined #openstack-infra		12:28
*** zeih has joined #openstack-infra		12:29
*** esker__ has joined #openstack-infra		12:29
*** fabo_ is now known as fabo		12:29
*** nadya has joined #openstack-infra		12:30
*** resker has quit IRC		12:30
*** moravec has quit IRC		12:30
*** esikachev has quit IRC		12:30
*** annegentle has joined #openstack-infra		12:32
*** esker_ has quit IRC		12:33
*** resker has joined #openstack-infra		12:33
*** baoli_ has joined #openstack-infra		12:34
*** esker__ has quit IRC		12:34
*** esker_ has joined #openstack-infra		12:34
*** rossella_s has quit IRC		12:34
*** \|-paul-\| has quit IRC		12:35
*** rossella_s has joined #openstack-infra		12:35
*** stevemar has joined #openstack-infra		12:36
*** annegentle has quit IRC		12:37
*** _ari_ has joined #openstack-infra		12:37
*** baoli has quit IRC		12:37
*** esker has joined #openstack-infra		12:37
*** resker has quit IRC		12:38
*** _ody_ has quit IRC		12:38
*** _ody has joined #openstack-infra		12:38
*** resker has joined #openstack-infra		12:39
*** daemontool_ has joined #openstack-infra		12:39
openstackgerrit	Jan Hruban proposed openstack-infra/zuul: Support GitHub PR webhooks https://review.openstack.org/163117	12:39
openstackgerrit	Jan Hruban proposed openstack-infra/zuul: Merge pull requests from github reporter https://review.openstack.org/243250	12:39
openstackgerrit	Jan Hruban proposed openstack-infra/zuul: Fix job hierarchy bug. https://review.openstack.org/192457	12:39
openstackgerrit	Jan Hruban proposed openstack-infra/zuul: Encapsulate determining the event purpose https://review.openstack.org/247487	12:39
openstackgerrit	Jan Hruban proposed openstack-infra/zuul: support github pull reqeust labels https://review.openstack.org/247421	12:39
openstackgerrit	Jan Hruban proposed openstack-infra/zuul: Better merge message for GitHub pull reqeusts https://review.openstack.org/280667	12:39
openstackgerrit	Jan Hruban proposed openstack-infra/zuul: Add 'push' and 'tag' github webhook events. https://review.openstack.org/191207	12:39
openstackgerrit	Jan Hruban proposed openstack-infra/zuul: Add 'pr-comment' github webhook event https://review.openstack.org/239203	12:39
openstackgerrit	Jan Hruban proposed openstack-infra/zuul: Support for dependent pipelines with github https://review.openstack.org/247500	12:39
openstackgerrit	Jan Hruban proposed openstack-infra/zuul: Configurable SSH access to GitHub https://review.openstack.org/239138	12:39
openstackgerrit	Jan Hruban proposed openstack-infra/zuul: GitHub file matching support https://review.openstack.org/292376	12:39
openstackgerrit	Jan Hruban proposed openstack-infra/zuul: Allow github trigger to match on branches/refs https://review.openstack.org/258448	12:39
openstackgerrit	Jan Hruban proposed openstack-infra/zuul: Log GitHub API rate limit https://review.openstack.org/292377	12:39
openstackgerrit	Jan Hruban proposed openstack-infra/zuul: Allow list values in template parameters. https://review.openstack.org/191208	12:39
openstackgerrit	Jan Hruban proposed openstack-infra/zuul: Add basic Github Zuul Reporter. https://review.openstack.org/191312	12:39
openstackgerrit	Jan Hruban proposed openstack-infra/zuul: Support for github commit status https://review.openstack.org/239303	12:39
*** esker_ has quit IRC		12:39
*** burgerk has joined #openstack-infra		12:39
*** esker_ has joined #openstack-infra		12:40
*** stevemar has quit IRC		12:40
*** esker__ has joined #openstack-infra		12:41
*** esker___ has joined #openstack-infra		12:42
smarcet	fungi: morning . i have this patch ready to go https://review.openstack.org/#/c/299732/	12:42
*** esker has quit IRC		12:42
*** dkehn has quit IRC		12:42
*** daemontool has quit IRC		12:42
*** esker has joined #openstack-infra		12:43
*** annegentle has joined #openstack-infra		12:43
*** dkehn has joined #openstack-infra		12:43
*** resker has quit IRC		12:43
*** resker has joined #openstack-infra		12:44
*** exploreshaifali has quit IRC		12:44
*** esker_ has quit IRC		12:45
*** ZZelle has joined #openstack-infra		12:45
anteaya	Zara: morning	12:45
*** esker__ has quit IRC		12:45
*** esker___ has quit IRC		12:46
*** esker_ has joined #openstack-infra		12:46
anteaya	Zara: I won't be going away after all, my basement flooded in the night within 5 hours so I have to cancel my vacation and take care of the house	12:46
anteaya	Zara: so cheer me up with great questions please and I'll do my best to answer them	12:46
*** esker has quit IRC		12:47
*** zhurong has quit IRC		12:48
*** gtmanfred has quit IRC		12:48
*** resker has quit IRC		12:48
AJaeger	anteaya: That's sad to hear ;( I had hoped the flooding was over...	12:48
anteaya	AJaeger: no it just started, this is the third wave	12:49
*** daemontool__ has joined #openstack-infra		12:49
*** rlandy has joined #openstack-infra		12:49
openstackgerrit	Paul Belanger proposed openstack-infra/system-config: [WIP] Add bootstrap stage to puppet https://review.openstack.org/299697	12:49
anteaya	yeah I'm really disappointed, was looking forward to seeing Germany again	12:50
anteaya	I haven't told the folks I was going to viist yet	12:50
*** e0ne has joined #openstack-infra		12:50
anteaya	I hope they didn't make large plans	12:50
*** dmellado is now known as dmellado\|lunch		12:50
pabelanger	clarkb: devstack-gate results: http://paste.openstack.org/show/492570/	12:51
Zara	anteaya: Oh no! :0	12:51
*** e0ne has quit IRC		12:51
*** tiswanso has quit IRC		12:52
anteaya	yeah, I cleaned out the sump at 3am and figured I was good, but at 8:30am water was all over, no way I can leave and ask someone else to stay up all night dealing with waater while i go away	12:52
*** daemontool_ has quit IRC		12:52
*** tiswanso has joined #openstack-infra		12:52
AJaeger	anteaya: Third? ;(	12:52
Zara	;_;	12:52
Zara	I'm not sure I can promise any questions that'll be good enough to make up for that...	12:53
AJaeger	pabelanger, could you review https://review.openstack.org/299892 , please?	12:53
anteaya	AJaeger: yeah it started raining yesterday been raining all night, it was slowing down before that	12:53
anteaya	Zara: just you company is lovely thank you	12:54
anteaya	Zara: so storyboard and gerrit, what do you understand currently?	12:54
*** daemontool__ has quit IRC		12:55
*** gtmanfred has joined #openstack-infra		12:55
AJaeger	anteaya: Next weeks weather forecast for GErmany isn't too good ;) But really sad that you have to cancel :(	12:55
*** exploreshaifali has joined #openstack-infra		12:56
*** sdague_ has quit IRC		12:57
AJaeger	bbl	12:57
*** edmondsw has joined #openstack-infra		12:57
*** thiagop has joined #openstack-infra		12:58
*** links has quit IRC		12:58
openstackgerrit	Paul Belanger proposed openstack-infra/system-config: [WIP] Add bootstrap stage to puppet https://review.openstack.org/299697	12:58
anteaya	AJaeger: yeah, thanks	12:59
*** amrith is now known as _amrith_		12:59
*** jsavak has joined #openstack-infra		12:59
annegentle	anteaya: ohh that sounds awful to deal with plus missing vacay!	13:00
*** daemontool__ has joined #openstack-infra		13:00
*** daemontool__ is now known as daemontool		13:00
*** achanda has joined #openstack-infra		13:00
*** Goneri has joined #openstack-infra		13:01
*** Daisy has joined #openstack-infra		13:01
openstackgerrit	Marton Kiss proposed openstack-infra/system-config: Add missing mysql client to groups-dev and groups https://review.openstack.org/299917	13:02
*** esikachev has joined #openstack-infra		13:03
Zara	anteaya: almost nothing... I'm looking generally in the field of getting storyboard talking to other things, review tools, etc, and gerrit is the most promising.	13:03
*** denisra has quit IRC		13:04
anteaya	annegentle: it is a semi-annual event, so the flooding is just a whole lot of loss of sleep and mess but used to it, mostly so sad I have to cancel and disppoint the folks that had asked me to go to munich	13:04
anteaya	they are such lovely people and was looking forward to seeing them again	13:04
anteaya	annegentle: thanks for understanding	13:05
Zara	anteaya: so I've looked a little at the python-storyboardclient, and I think the next step is probably looking at the launchpad integration to see how that works, and how mcuh can be copied	13:05
annegentle	anteaya: definitely have sympathy for ya	13:05
anteaya	Zara: that would be a far assessment	13:05
anteaya	annegentle: thanks, I appreciate it	13:05
openstackgerrit	Anton Arefiev proposed openstack-infra/project-config: Add Ironic Inspector basic tempest job https://review.openstack.org/299821	13:05
*** yolanda has joined #openstack-infra		13:05
*** jaosorior has quit IRC		13:05
*** achanda has quit IRC		13:06
*** Daisy has quit IRC		13:06
*** jaosorior has joined #openstack-infra		13:06
anteaya	Zara: I think that looking at the launchpad gerrit bits for how launchpad is able to pick up commit message triggers about bugs would be very helpful	13:06
anteaya	Zara: gerrit has a rest api as well as the ability to communicate via ssh	13:06
anteaya	Zara: I'm supportive of using the rest api	13:07
*** torandu has quit IRC		13:08
*** zul has quit IRC		13:08
Zara	anteaya: okay, I'll start there. In general, I don't have much python, and haven't done anything much in this field (working with rest apis and so on), so any tips on where to start, or good tutorials, etc, is appreciated. I think I have a fuzzy handle on the theory of it but have a lot of gaps in my knowledge.	13:08
openstackgerrit	Paul Belanger proposed openstack-infra/system-config: [WIP] Add bootstrap stage to puppet https://review.openstack.org/299697	13:08
anteaya	sorry I think I want Biennial not semi-annual	13:08
anteaya	Zara: understood, me too	13:08
anteaya	my best tip for python would be get to know dhellmann if you don't already :)	13:09
Zara	hehe, and I normally pester SotK but he's off holidaying.	13:09
*** torandu has joined #openstack-infra		13:10
*** akshai has joined #openstack-infra		13:10
anteaya	yup	13:11
*** rfolco has quit IRC		13:11
openstackgerrit	Paul Belanger proposed openstack-infra/system-config: [WIP] Add bootstrap stage to puppet https://review.openstack.org/299697	13:11
*** yamahata has joined #openstack-infra		13:14
*** baoli_ has quit IRC		13:16
openstackgerrit	Anton Arefiev proposed openstack-infra/project-config: Add Ironic Inspector basic tempest job https://review.openstack.org/299821	13:17
*** salv-orlando has quit IRC		13:18
*** rfolco has joined #openstack-infra		13:19
*** salv-orlando has joined #openstack-infra		13:19
*** exploreshaifali has quit IRC		13:19
*** mtanino has joined #openstack-infra		13:20
*** electrofelix has quit IRC		13:21
*** electrofelix has joined #openstack-infra		13:21
*** electrofelix has quit IRC		13:21
*** Goneri has quit IRC		13:22
*** salv-orl_ has joined #openstack-infra		13:24
*** nwkarsten has joined #openstack-infra		13:25
*** salv-orlando has quit IRC		13:26
*** zeih has quit IRC		13:26
*** amitgandhinz has joined #openstack-infra		13:26
openstackgerrit	John Trowbridge proposed openstack-infra/project-config: Add tripleo-quickstart project to tripleo team https://review.openstack.org/299932	13:27
*** zul has joined #openstack-infra		13:27
*** dprince has quit IRC		13:28
openstackgerrit	John Trowbridge proposed openstack-infra/project-config: Add tripleo-quickstart project to tripleo team https://review.openstack.org/299932	13:28
*** dprince has joined #openstack-infra		13:29
*** akshai has quit IRC		13:29
pabelanger	fungi: So, I have the bootstrap stage working in puppet. However, we need to have a discussion if we want to move forward with it. Since it will require updates to other puppet modules using apt	13:29
pabelanger	fungi: it might just be easier to have DIB run apt-get update again before we run puppet	13:30
*** akshai has joined #openstack-infra		13:30
*** knikolla has joined #openstack-infra		13:32
esikachev	yolanda: hi! can you merge https://review.openstack.org/#/c/299407/ please	13:32
*** e0ne has joined #openstack-infra		13:32
*** tqtran has joined #openstack-infra		13:34
yolanda	hi esikachev, i'm not really around today, but will take a look at the review queue in short	13:35
esikachev	yolanda: thanks, this small change	13:36
*** stevemar has joined #openstack-infra		13:36
*** Goneri has joined #openstack-infra		13:37
*** mriedem has joined #openstack-infra		13:37
yolanda	esikachev, why commenting the jobs instead of removing those? is that something very very temporary?	13:37
*** salv-orl_ has quit IRC		13:38
esikachev	yolanda: yes, is very-very temporary change (see patches with same topic)	13:38
*** tqtran has quit IRC		13:39
yolanda	ok approved	13:39
*** tonytan4ever has joined #openstack-infra		13:39
*** abregman has quit IRC		13:39
esikachev	thanks!	13:39
*** moravec has joined #openstack-infra		13:40
*** mylu has joined #openstack-infra		13:40
tosky	yolanda: thanks (I'm the author); the commit has a reference to a (long-ish) email which explain the process	13:41
*** stevemar has quit IRC		13:41
*** abregman has joined #openstack-infra		13:41
*** salv-orlando has joined #openstack-infra		13:43
*** burgerk has quit IRC		13:44
*** permalac has joined #openstack-infra		13:45
*** cdent has joined #openstack-infra		13:46
mordred	infra-root, Shrews: when the two shade patches in the gate queue land I'm going to cut another shade release and we'll be ready to try shade-nodepool again	13:46
mordred	yolanda, pabelanger: https://review.openstack.org/#/c/299702 if you get a sec	13:47
*** regXboi has joined #openstack-infra		13:47
dmsimard	Where can I see the output of devstack gate for a particular job ? Trying to look if fix_disk_layout from functions.sh is doing some wrong stuff again	13:48
*** _amrith_ is now known as amrith		13:49
Shrews	mordred: i have concerns about jeblair_'s port cache	13:49
mordred	Shrews: yeah?	13:49
Shrews	mordred: if we create or delete a port, would we not want that to show up immediately in the cache? it isn't invalidated in the create/delete apis	13:50
mtreinish	dmsimard: for fix_disk_layout that'll be in the setup workspace log: http://logs.openstack.org/06/283806/8/check/gate-tempest-dsvm-ironic-agent_ssh-nv/a91a361/logs/devstack-gate-setup-workspace-new.txt.gz#_2016-02-27_15_18_42_820	13:50
*** ayoung has joined #openstack-infra		13:50
*** cbader_ has quit IRC		13:50
mtreinish	that was just a log I had open, it's kinda old though	13:50
*** baoli has joined #openstack-infra		13:50
openstackgerrit	Merged openstack-infra/project-config: [sahara-tests] create and merge, disable tests https://review.openstack.org/299407	13:50
*** cdent has left #openstack-infra		13:50
*** cbader_ has joined #openstack-infra		13:51
mordred	Shrews: yah, so the code that polls for has a timeout double the time of the server cache time	13:51
*** xyang1 has joined #openstack-infra		13:52
dmsimard	mtreinish: hmm, does a job need to do something special to get that log there ? I don't have it picked up in our logs http://logs.openstack.org/19/299719/1/check/gate-packstack-integration-scenario003-tempest-dsvm-centos7/90813ab/logs/	13:52
mordred	Shrews: so that the code in _get_free_fixed_port won't return until it's had a chance to get it	13:52
openstackgerrit	Paul Belanger proposed openstack-infra/puppet-jenkins: Ubuntu xenial supports openjdk-8-jdk https://review.openstack.org/299948	13:52
*** e0ne has quit IRC		13:52
*** salv-orlando has quit IRC		13:52
*** mtanino has quit IRC		13:52
Shrews	mordred: oh, i see that now. that's a curious way to handle that	13:53
*** Daisy has joined #openstack-infra		13:53
pabelanger	clarkb: fungi: So, apparently openjdk-7 was deprecated at the last moment for xenial ^	13:53
*** permalac has quit IRC		13:53
mtreinish	dmsimard: does packstack use devstack-gate? it doesn't run devstack so I wouldn't think it would	13:53
openstackgerrit	Derek Higgins proposed openstack-infra/tripleo-ci: Use cached instack qcow image for all ci tests https://review.openstack.org/299537	13:53
openstackgerrit	Derek Higgins proposed openstack-infra/tripleo-ci: Pre install packages on the instack image https://review.openstack.org/299538	13:53
*** maishsk has joined #openstack-infra		13:53
mordred	Shrews: if you're using the batch-the-list-operations version of caching and touching ports directly it's a thing you'd have to do... but this is a thing that really shouldn't be needed by many people that are not doing nodepool levels of parallel operations	13:53
openstackgerrit	Paul Belanger proposed openstack-infra/puppet-jenkins: Ubuntu xenial supports openjdk-8-jdk https://review.openstack.org/299948	13:53
Shrews	mordred: if we're waiting for the cache to invalidate, why not just go ahead and make it so?	13:53
mordred	Shrews: I'm going to take a node to write up some docs on it	13:53
mordred	Shrews: hrm	13:54
*** e0ne has joined #openstack-infra		13:54
mordred	Shrews: still coffeeing - lemme think about that for a sec	13:54
dmsimard	mtreinish: it uses dsvm so I guess it would ?	13:54
Shrews	mordred: just not seeing the advantage of waiting	13:54
*** dmellado\|lunch is now known as dmellado		13:54
mordred	(there was a reason yesterday)	13:54
dmsimard	mtreinish: https://github.com/openstack-infra/project-config/blob/master/jenkins/jobs/packstack.yaml#L3 (devstack-{ostype})	13:55
*** e0ne has quit IRC		13:56
mordred	Shrews: I can't come up with a good reason ... probably would be good to dig in and see if we can prove out the active invalidation approach as a follow up? (landing it this way then imrpoving it shouldn't be an interface break, yeah?)	13:56
*** mriedem has left #openstack-infra		13:56
*** mriedem has joined #openstack-infra		13:57
mtreinish	dmsimard: right, it uses a devstack node type. But where is devstack-gate called?	13:57
dmsimard	mtreinish: does it need to be called explicitely ? (I'm a noob with this, thanks for bearing with me)	13:57
*** mohankumar has quit IRC		13:57
*** zaro has quit IRC		13:58
mtreinish	dmsimard: look at a regular devstack-gate job defined in the devstack-gate.yaml you'll see that's what's actually run by the job	13:58
mordred	Shrews: I think the invalidation would be "acquire port lock in blocking mode, zero out port list" yeah?	13:58
*** zaro has joined #openstack-infra		13:58
pabelanger	mordred: done	13:58
Shrews	mordred: yes, the same way i just modified the network cache	14:00
mordred	Shrews: oh - also, we're not creating the port	14:00
Shrews	mordred: but if you want to put it in now, that's fine. it could be improved i think	14:00
mordred	Shrews: it's a thing that happens when create server happens - so we'd have to invalidate the cache in the server loop too	14:00
*** zhurong_ has joined #openstack-infra		14:00
*** Hal has joined #openstack-infra		14:01
dmsimard	mtreinish: hmm, they all look to actually run devstack ? All I really want is that fix_disk_layout from functions.sh (https://github.com/openstack-infra/devstack-gate/blob/master/functions.sh#L333-L394) runs on the VMs packstack jobs run on.	14:01
*** kushal has joined #openstack-infra		14:01
mordred	Shrews: I agree on it being able to be improved - I just have a hunch we're going to find a few edge cases that might take a while to make sure we're solid on	14:01
*** Hal is now known as Guest67601		14:01
*** eharney has joined #openstack-infra		14:02
Shrews	mordred: ah, i see. i think my argument about invalidating in create/delete still holds though	14:02
dmsimard	mtreinish: apparently I might have wrongly assumed that ran on every VM that was provided by nodepool..	14:02
*** kushal has quit IRC		14:02
dmsimard	as a pre-setup before handing the VM out	14:02
*** salv-orlando has joined #openstack-infra		14:02
mordred	Shrews: totally	14:02
Shrews	mordred: to be clear, in create_port/delete_port	14:02
*** achanda has joined #openstack-infra		14:02
mordred	++	14:03
*** cdent_ has joined #openstack-infra		14:03
*** electrofelix has joined #openstack-infra		14:03
*** electrofelix has quit IRC		14:03
pabelanger	mordred: do you mind helping with https://review.openstack.org/#/c/299948/ fixed ubuntu-xenial builds	14:04
*** cdent_ has quit IRC		14:04
pabelanger	yolanda: ^ if you are still around	14:04
mordred	pabelanger: looking	14:04
*** cdent has joined #openstack-infra		14:04
mordred	pabelanger: and jenkins works on java 8 I'm guessing?	14:05
*** sigmavirus24_awa is now known as sigmavirus24		14:05
pabelanger	mordred: yup. tested it a few days ago	14:05
*** dprince has quit IRC		14:06
*** eharney has quit IRC		14:06
*** dprince has joined #openstack-infra		14:07
*** placko_ has quit IRC		14:07
*** sdague has quit IRC		14:08
*** achanda has quit IRC		14:08
*** sdague has joined #openstack-infra		14:08
openstackgerrit	Derek Higgins proposed openstack-infra/tripleo-ci: Use cached instack qcow image for all ci tests https://review.openstack.org/299537	14:09
openstackgerrit	Derek Higgins proposed openstack-infra/tripleo-ci: Pre install packages on the instack image https://review.openstack.org/299538	14:09
*** abregman has quit IRC		14:09
*** cdent has quit IRC		14:09
fungi	pabelanger: having dib run apt-get update doesn't solve the race i have, which is that puppet is updating teh sources list and then tries to install packages without running apt-get update in between. if we can ensure that the sources list and ensuing apt-get update exec occur before it ever tries to install any packages then that should fix it. from what i was reading, run stages were the preferred	14:09
fungi	solution to deal with that	14:09
*** akshai has quit IRC		14:10
pabelanger	fungi: Ya, with my old manifests, I always did package updates in a different stage then main. Worked well	14:10
pabelanger	If we are going that route, we should talk about it more, since this is the first time adding a new stage to puppet.	14:11
pabelanger	crinkle and nibalize1 likely have suggestions too	14:11
*** akshai has joined #openstack-infra		14:11
*** rfolco has quit IRC		14:12
openstackgerrit	Merged openstack-infra/shade: Cache ports like servers https://review.openstack.org/299671	14:12
openstackgerrit	Derek Higgins proposed openstack-infra/tripleo-ci: Fix image caching logic https://review.openstack.org/299537	14:14
openstackgerrit	Derek Higgins proposed openstack-infra/tripleo-ci: Pre install packages on the instack image https://review.openstack.org/299538	14:14
*** sigmavirus24 is now known as sigmavirus24_awa		14:14
sdague	pabelanger: could we get a couple xenial nodes put into the pool and an experimental devstack with it?	14:14
*** sigmavirus24_awa is now known as sigmavirus24		14:15
pabelanger	sdague: ubuntu-xenial still not online yet. We need to land https://review.openstack.org/#/c/299948/ first	14:15
sdague	pabelanger: ok, gotcha	14:15
pabelanger	sdague: once we do, I don't see a ready why not.	14:15
pabelanger	except for official support from -infra won't happen until xenial is official released :D	14:16
*** cbader has joined #openstack-infra		14:16
dmsimard	clarkb: oi, more fix_disk_layout woes. I wrongly assumed that function ran by default on devstack-* nodes before being handed out to jobs. Should I just call it explicitely for Packstack jobs or is there another way ?	14:16
pabelanger	Doing my best to keep it online in the beta stage	14:16
sdague	sure, but an experimental devstack job actually means we can address issues early	14:16
* mordred hands pabelanger a box of kittens		14:16
mordred	sdague: ++	14:16
*** Guest67601 has quit IRC		14:16
*** cbader has quit IRC		14:17
pabelanger	sdague: Ya, devstack-gate actually ran on my ubuntu-xenial dib. Which is good news	14:17
*** cbader_ has quit IRC		14:17
*** cbader has joined #openstack-infra		14:17
*** ajmiller has joined #openstack-infra		14:17
*** kushal has joined #openstack-infra		14:18
*** esker_ has quit IRC		14:18
openstackgerrit	Paul Belanger proposed openstack-infra/project-config: Add centos-7 to bindep-fallback https://review.openstack.org/299988	14:18
*** burgerk has joined #openstack-infra		14:18
*** cdent has joined #openstack-infra		14:19
crinkle	pabelanger: fungi oh this was happening on infra-cloud too, but we solved it by just not removing the default sources list :(	14:19
openstackgerrit	Raildo Mascena proposed openstack-infra/project-config: Changing gate on devstack identity v3 only voting https://review.openstack.org/292894	14:19
*** cdent has quit IRC		14:20
crinkle	pabelanger: fungi we could have an exec { 'apt-get update': } with a before relationship to Package<\|\|> and a require on the apt source file	14:20
mordred	crinkle: that sounds sane	14:21
crinkle	i don't really understand stages so i'm hesitant to recommend it, in my mind it's a workaround to something that in theory puppet should handle well	14:21
*** hogepodge has joined #openstack-infra		14:21
*** eharney has joined #openstack-infra		14:21
fungi	crinkle: yeah, leaving the old sources behind is probably not terrible since the goal was to pick up newer packages from official ubuntu locations when rackspace is lagging behind (for example, they commit the unpardonable sin of mirroring the ubuntu security repo and then setting that as teh only source for security updates in their images)	14:22
pabelanger	I'm fine with crinkles suggestion. I've just done stages in the past, and don't want to push it if people don't think we need them. Also, switching to stages will be more work	14:23
fungi	crinkle: i was also looking at the Package<\|\|> trick, but from what i was reading it seemed like it was going to exec an apt-get update every time we apply regardless of whether we're installing packages... i guess that's the only way for it to know there are new packages anyway though?	14:23
fungi	oh, though we don't rely on puppet to upgrade our packages, that's what we have unattended upgrades set up for	14:24
*** cdent has joined #openstack-infra		14:24
crinkle	fungi: it wouldn't need to apt-get update every time	14:24
csaba	clarkb: ping	14:24
crinkle	it could refreshonly on updating the apt source file	14:24
fungi	aha, right you are	14:24
fungi	i'll see if i can get that passing tests. i had it halfway written up before i passed out last night	14:25
crinkle	cool	14:25
fungi	thanks!	14:25
crinkle	np :)	14:25
csaba	AJaeger: thanks.	14:26
*** rfolco has joined #openstack-infra		14:26
csaba	clarkb: I have some permission issues with uploading log files, AJaeger suggested to talk to you. (Please see http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2016-03-31.log.html#t2016-03-31T10:44:04 and http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2016-03-31.log.html#t2016-03-31T11:53:36 for details)	14:27
*** zaro has quit IRC		14:28
openstackgerrit	Sagi Shnaidman proposed openstack-infra/tripleo-ci: Run tempest tests with periodic jobs https://review.openstack.org/297038	14:28
*** dingyichen has quit IRC		14:28
*** links has joined #openstack-infra		14:29
*** e0ne has joined #openstack-infra		14:29
*** e0ne has quit IRC		14:29
*** mohankumar has joined #openstack-infra		14:31
*** zaro has joined #openstack-infra		14:31
*** esikachev has quit IRC		14:33
*** mizar has joined #openstack-infra		14:34
*** mizar is now known as flongo		14:34
*** Jeffrey4l_ has quit IRC		14:35
*** Jeffrey4l_ has joined #openstack-infra		14:35
openstackgerrit	Carlos Camacho proposed openstack-infra/tripleo-ci: Added validation to tripleo.sh --overcloud-deploy option. https://review.openstack.org/296362	14:35
*** oanson has quit IRC		14:36
Zara	anteaya: btw, I'm going to end up approaching this updating-from-gerrit stuff with my usual thinking-out-loud, so currently doing that in #storyboard, can in here or someplace else, but concerned it'll be a bit spammy.	14:37
Zara	they're used to me waffling away to myself in there	14:37
anteaya	I'lll join you in storyboard	14:37
anteaya	feel free to waffling away	14:38
*** ajmiller has quit IRC		14:39
*** ajmiller has joined #openstack-infra		14:41
openstackgerrit	Jeremy Stanley proposed openstack-infra/system-config: Always apt-get update after replacing sources https://review.openstack.org/300005	14:43
fungi	crinkle: that ^ what you were suggesting?	14:43
fungi	pabelanger: ^	14:43
*** esker has joined #openstack-infra		14:43
sdague	we're getting some really weird and slow vexxhost nodes - http://logs.openstack.org/85/262585/28/check/gate-nova-python27-db/080afb3/console.html	14:45
*** e0ne has joined #openstack-infra		14:45
sdague	that took an hour to not start running unit tests	14:45
fungi	sdague: yeah, i'm trying to catch and hold one. i have several log examples	14:45
sdague	fungi: it looks like the whole thing is on molasses	14:46
fungi	sdague: i have a feeling it's (lack of) write caching on some of their hosts, which makes apt/dpkg painfully slow	14:46
sdague	the timing for package installs	14:46
*** jlanoux_ has quit IRC		14:46
sdague	right, io constraints probably	14:46
fungi	yeah, debian package operations are meticulous about syncing between writes to avoid broken systems, and without write caching that turns into pain	14:47
crinkle	fungi: yep that looks right	14:47
*** andrewbogott has quit IRC		14:47
*** andrewbogott has joined #openstack-infra		14:47
*** esker has quit IRC		14:48
fungi	sdague: anyway, i want to catch and hold an impacted instance, but second best thing would be to just get a list of some affected hostnames so i can turn them into nova instance uuids to pass along to mnaser	14:48
mrmartin	hi	14:48
*** links has quit IRC		14:48
mnaser	i'll have a look in a few	14:48
AJaeger	sdague, fungi, anteaya : Could either of you review https://review.openstack.org/299892 to fix release notes translations, please? pabelanger already gave a +2.	14:49
*** SumitNaiksatam has quit IRC		14:49
mrmartin	yolanda, the groups-dev is ok on the trusty instance, may I ask you to switch the dns and drop the old one? (it is the dev!)	14:49
*** SumitNaiksatam has joined #openstack-infra		14:49
mnaser	we're making some adjustments to the scheduler, the whole "compute node with most free memory = best host to place instances on" does not prove to be correct	14:49
*** links has joined #openstack-infra		14:49
*** lyarwood has quit IRC		14:49
fungi	also, the ticket we opened with rackspace about network issues in ord is going nowhere. we're down a couple hundred quota until we can turn rax-ord back on but i'm not comfortable doing that until we have more details on whether the network issues will come right back when we do	14:50
pabelanger	fungi: 2 comments	14:50
mnaser	but impacted instance uuids can help confirm the theory of this	14:50
*** zaro has quit IRC		14:50
fungi	mnaser: yep, will be working on coming up with some, or better still holding an instance that's impacted so we can experiment further	14:50
*** SumitNaiksatam has quit IRC		14:51
mnaser	fungi: if the behaviour is common, it could be the block storage seeing some problems therefore it's really slow to respond too	14:51
*** esikachev has joined #openstack-infra		14:51
mnaser	we found out that the journal drives we were using were less-than-ideal so they're all being replaced, but the price of this = recovery of cluster which can cause io slowdowns	14:51
*** jaosorior_ has joined #openstack-infra		14:52
fungi	mnaser: yep, that would also account for it. seems to be disk i/o that's especially slow	14:52
*** zaro has joined #openstack-infra		14:52
mnaser	so the mirror instance might be stuggling (that might be easy to check)	14:52
fungi	particularly write operations though i think	14:52
mnaser	disk i/o to block storage (for the mirror instance?) or local?	14:52
mnaser	yes, writes have been bad mainly because the ceph journal drive got punished with writes, so we're replacing them with much higher durability drives	14:52
fungi	mnaser: oh, the mirror doesn't seem to be affected but i'll look closer	14:52
*** jaosorior has quit IRC		14:53
fungi	i was thinking it looked like slow i/o on the rootfs of teh worker instances, but it was hard to tell	14:53
mnaser	could also be. we have a customer that came has be pounding our CPUs so we've been playing with the scheduler	14:53
mnaser	more specifically the utilization based scheduler	14:53
fungi	if it were i/o struggling on the mirror block device i'd have expected high iowait but that server is basically idle	14:54
*** xnox has quit IRC		14:54
*** SumitNaiksatam has joined #openstack-infra		14:54
mnaser	so that means that it's the local instance store, thanks fungi	14:54
yolanda	hi mrmartin, going to do it	14:55
fungi	i ocattionally see iowait pop up to 0.1% of a processor on the mirror for a single poll and then drop back to 0.0% again	14:55
fungi	er, occasionally	14:56
mrmartin	yolanda, thank you, and can you make me a similar instance for groups.o.o also?	14:56
mnaser	yeah that shouldn't be the reason behind it	14:56
yolanda	yep	14:56
*** sdake has joined #openstack-infra		14:57
mmedvede	hi, can any puppet/infra/nodepool core take a look at https://review.openstack.org/#/c/276964/ ? It needs +A	14:57
*** FallenPegasus has joined #openstack-infra		14:57
fungi	so anyway, for rackspace, i think we're going to need to add `traceroute6 git.openstack.org` in our jobs and then inch ord back up a little bit in nodepool to see if we can catch one with network issues	14:58
mmedvede	pabelanger: hi, were you able get a tempest run on xenial VM? any problems with volume snapshots?	14:58
*** jsavak has quit IRC		14:59
*** sshnaidm has quit IRC		14:59
*** xnox has joined #openstack-infra		14:59
*** jsavak has joined #openstack-infra		15:00
pabelanger	mmedvede: {1} tempest.api.volume.admin.test_snapshots_actions.SnapshotsActionsV1Test.test_snapshot_force_delete_when_snapshot_is_creating [273.344596s] ... FAILED	15:00
pabelanger	but devstack-gate timed out before the rest finished	15:00
*** kushal has quit IRC		15:00
*** mtanino has joined #openstack-infra		15:00
mmedvede	pabelanger: ok, that might be confirming the same problem I have seen	15:01
mmedvede	pabelanger: lvm2 snapshots are slow	15:01
*** e0ne has quit IRC		15:01
*** maishsk has quit IRC		15:01
pabelanger	mmedvede: We should have ubuntu-xenial uploaded today, then we can setup an experimental job for people to debug with	15:02
mmedvede	pabelanger: for me most of snapshots tests where unstable/timing out. I found the workaround, described in https://bugs.launchpad.net/ubuntu/+source/lvm2/+bug/1561228	15:03
openstack	Launchpad bug 1561228 in lvm2 (Ubuntu) "dmevent can not open shared object file libdevmapper-event-lvm2snapshot.so leads to lvcreate/lvremove being slow" [Undecided,New]	15:03
*** IlyaG has joined #openstack-infra		15:04
yolanda	mrmartin, dns record updated, going to launch the replacement for groups	15:04
pabelanger	mmedvede: Ya, here was the error: Details: (SnapshotsActionsV1Test:test_snapshot_force_delete_when_snapshot_is_creating) Failed to delete volume-snapshot 2cdcf5dc-27c0-44ef-903c-1049790e4d1f within the required time (196 s)	15:04
yolanda	it will take a time for dns to be updated, once it's done i'll remove the old groups-dev	15:05
*** zaro has quit IRC		15:05
mmedvede	pabelanger: that is exactly it. There is a script in the bug to reproduce the problem	15:05
mmedvede	pabelanger: if you have access to syslog, it also would have errors in it	15:06
*** electrichead is now known as redrobot		15:06
*** stevemar has joined #openstack-infra		15:06
*** zaro has joined #openstack-infra		15:06
*** fhubik has quit IRC		15:07
*** jistr has quit IRC		15:07
*** esker has joined #openstack-infra		15:07
*** mtanino has quit IRC		15:09
*** jerryz has joined #openstack-infra		15:09
*** jistr has joined #openstack-infra		15:09
*** mrmartin has quit IRC		15:09
*** mtanino has joined #openstack-infra		15:11
*** salv-orlando has quit IRC		15:11
*** esker has quit IRC		15:12
*** nmagnezi has quit IRC		15:12
*** andymaier has joined #openstack-infra		15:12
openstackgerrit	Jeremy Stanley proposed openstack-infra/project-config: Traceroute to git.o.o in net-info macro https://review.openstack.org/300021	15:13
fungi	infra-root: that's ^ intended to help us help rackspace once we turn ord back up a bit, but it's going to affect a ton of job configs so will need some manual jjb updates once merged	15:13
anteaya	AJaeger: done	15:14
*** piet has joined #openstack-infra		15:14
smarcet	fungi: morning , seems that https://review.openstack.org/#/c/299732/ timeouted ?	15:14
pabelanger	fungi: okay, I am offline in 45mins for the next few hours, but should be able to help monitor once back online	15:15
fungi	smarcet: indeed, looks also affected by the slowness we've been seeing in vexxhost. i'll look up the uuids for those workers and let them know (and i'll reapprove the change in the meantime)	15:16
*** vgridnev has quit IRC		15:16
*** mohankumar has quit IRC		15:16
*** esker has joined #openstack-infra		15:16
*** zz_dimtruck is now known as dimtruck		15:16
smarcet	fungi: tnx a lot :)	15:17
*** jlanoux has joined #openstack-infra		15:17
*** kdas_ has joined #openstack-infra		15:18
*** links has quit IRC		15:19
*** armax_ has joined #openstack-infra		15:22
openstackgerrit	Merged openstack-infra/project-config: Fix releasenotes translation https://review.openstack.org/299892	15:22
fungi	mnaser: here's three affected uuids to start you out: 273e30d6-19d4-4b56-9a91-43146fada083 8a44c948-d30f-40c3-a024-a48fd4b15fea f3feed0e-a6ba-4503-bc86-28c35672f96a	15:22
fungi	one's the job sdague mentioned earlier, the other two we just hit testing another change	15:23
*** rkukura has joined #openstack-infra		15:23
*** kdas_ has quit IRC		15:23
*** jaosorior_ has quit IRC		15:24
*** bpokorny has joined #openstack-infra		15:24
*** ayoung has quit IRC		15:24
*** Qiming has quit IRC		15:24
*** armax has quit IRC		15:25
*** armax_ is now known as armax		15:25
*** ayoung has joined #openstack-infra		15:25
*** arxcruz has quit IRC		15:25
*** esker has quit IRC		15:26
openstackgerrit	Merged openstack-infra/devstack-gate: Use journald when it is present https://review.openstack.org/299682	15:26
openstackgerrit	Merged openstack-infra/puppet-openstackid: Fix Vhost (Apache2.4) https://review.openstack.org/299732	15:26
fungi	smarcet: ^ much better. passed and merged this time	15:26
smarcet	cool thx	15:27
*** vgridnev has joined #openstack-infra		15:27
*** bpokorny has quit IRC		15:28
*** jistr has quit IRC		15:28
*** zaro has quit IRC		15:28
*** zhurong_ has quit IRC		15:28
*** jistr has joined #openstack-infra		15:28
*** tiswanso has quit IRC		15:29
*** bpokorny has joined #openstack-infra		15:30
*** zaro has joined #openstack-infra		15:30
fungi	smarcet: i'm keeping tabs on when the next puppet apply pulse hits the server, then we can see if there's anything else missing	15:31
fungi	should hopefully be within the next 15 minutes	15:31
smarcet	fungi: cool	15:31
smarcet	i think that would be all, hopefully	15:31
*** ams_ has joined #openstack-infra		15:31
*** jcoufal_ has joined #openstack-infra		15:33
openstackgerrit	Jeremy Stanley proposed openstack-infra/system-config: Always apt-get update after replacing sources https://review.openstack.org/300005	15:33
*** mohankumar has joined #openstack-infra		15:34
*** dimtruck is now known as zz_dimtruck		15:34
*** jcoufal has quit IRC		15:35
*** FallenPegasus has quit IRC		15:35
yolanda	mordred, i'm having issues on launch node	15:36
yolanda	bootstrap server fails , although it creates the server, stops after that	15:36
*** SumitNaiksatam has quit IRC		15:37
*** markvoelker has joined #openstack-infra		15:37
*** kdas_ has joined #openstack-infra		15:37
yolanda	mm, actually is puppet so it may be a problem with the manifest	15:37
*** e0ne has joined #openstack-infra		15:37
*** tosky has quit IRC		15:38
fungi	yolanda: i believe you're running into the same thing i was, which prompted 300005	15:38
*** sshnaidm has joined #openstack-infra		15:38
*** kdas_ has quit IRC		15:38
*** sigmavirus24 is now known as sigmavirus24_awa		15:38
fungi	yolanda: add the --keep option when you run launch-node.py so that it won't delete the instance, then ssh into it and look in /var/log/syslog for puppet apply failures	15:39
*** jerryz has quit IRC		15:39
fungi	yolanda: if it's failing on an attempt to install some package(s) then that's what we're working on trying to fix	15:39
yolanda	yep, i have it with keep, going to check	15:39
*** e0ne has quit IRC		15:40
mnaser	fungi: thank you, this is noted	15:40
*** markvoelker has quit IRC		15:40
*** sigmavirus24_awa is now known as sigmavirus24		15:40
yolanda	fungi i see a drush site-install failed error	15:40
*** torandu has quit IRC		15:40
*** markvoelker_ has joined #openstack-infra		15:41
*** sigmavirus24 is now known as sigmavirus24_awa		15:41
fungi	yolanda: another possibility then is it's hitting a timeout? i know a fresh groups deployment takes... a while	15:41
*** sigmavirus24_awa is now known as sigmavirus24		15:41
yolanda	going to run it with a limit	15:42
fungi	and also the output from it last i checked was so verbose syslog stopped accepting entries	15:42
fungi	so the only way to diagnose was to try to run the same exec myself manually	15:42
*** kushal has joined #openstack-infra		15:42
*** torandu has joined #openstack-infra		15:42
fungi	and redirect the output to a file for easier inspection (and to be able to hand off to mrmartin as well)	15:43
yolanda	fungi http://paste.openstack.org/show/492609/	15:43
*** e0ne has joined #openstack-infra		15:44
yolanda	looks as a problem with the manifest, so i'll check with mrmartin	15:44
yolanda	i actually thing i saw the same error on groups-dev	15:44
*** Swami has joined #openstack-infra		15:44
smarcet	fungi: seems to be up https://openstackid-dev.openstack.org !!! :)	15:46
*** dtantsur is now known as dtantsur\|afk		15:46
*** thiagop is now known as thiagop-afk		15:46
fungi	yolanda: looks like it was the `drush dsd-init @groups` command that failed (returned nonzero)	15:46
*** e0ne has quit IRC		15:46
fungi	smarcet: yep, i just noticed it updated ~6 minutes ago	15:46
smarcet	fungi: cool tnx for all your help on this :)	15:47
*** e0ne has joined #openstack-infra		15:47
fungi	smarcet: so anyway, next steps are to confirm your load tests there are returning the desired results, and then we can plan for a replacement build of the production server and try for a fast cut-over	15:47
smarcet	fungi: i think so	15:48
*** bpokorny has quit IRC		15:48
fungi	that'll require another system-config patch to set the production redis version and also adjust the disablement of the production server to be by uuid instead of by name so we can successfully build the replacement with the same name	15:48
*** bpokorny has joined #openstack-infra		15:48
fungi	also i'll need to see if there's more cleanup we can do in that tenant so i'll have enough quota to temporarily have the new 15gb server at the same time we're still using the old one	15:49
smarcet	fungi: roger that	15:50
*** markvoelker_ has quit IRC		15:50
*** markvoelker has joined #openstack-infra		15:51
openstackgerrit	Merged openstack-dev/hacking: Update the assertRaises example to be more modern https://review.openstack.org/284097	15:52
*** zhurong has joined #openstack-infra		15:53
*** kushal has quit IRC		15:53
*** eharney has quit IRC		15:53
*** zz_dimtruck is now known as dimtruck		15:54
*** Guest87578 is now known as jlvillal		15:55
*** hashar has quit IRC		15:55
smarcet	fungi: i am testing i would need to propose a patch to tweak a little connection settings	15:55
*** markvoelker has quit IRC		15:55
*** jaypipes has joined #openstack-infra		15:55
openstackgerrit	Merged openstack-infra/project-config: Traceroute to git.o.o in net-info macro https://review.openstack.org/300021	15:56
*** markvoelker has joined #openstack-infra		15:56
fungi	smarcet: that's fine--i expected the values may still need some adjusting based on your subsequent testing. those are easy to review	15:56
smarcet	fungi: ok cool tnx	15:56
*** baoli_ has joined #openstack-infra		15:56
*** markvoelker_ has joined #openstack-infra		15:57
*** baoli has quit IRC		15:57
*** e0ne has quit IRC		15:58
robcresswell	Hello. How do I go about getting a project to sync with global-reqs automagically?	15:58
clarkb	robcresswell: add your project name to the projects list in the requirements repo and add the requirements sync jenkins jobs to your project	15:58
robcresswell	clarkb: Straightforward. Whats the order, assuming we need a Depends-On?	15:59
*** esker has joined #openstack-infra		15:59
fungi	requirements check jobs (the requirements sync is a job on the requirements repo controlled via the projects.txt file there)	15:59
clarkb	yup that	15:59
*** kzaitsev_mb has quit IRC		16:00
*** tphummel has joined #openstack-infra		16:00
*** jlanoux_ has joined #openstack-infra		16:00
*** eharney has joined #openstack-infra		16:00
clarkb	csaba: the log file explains the issue http://logs.openstack.org/22/299322/2/check/gate-manila-tempest-dsvm-glusterfs/e7a146b/console.html#_2016-03-31_06_04_55_761	16:00
*** esker has quit IRC		16:00
clarkb	csaba: you will need to correct that in your jobs	16:00
*** esker has joined #openstack-infra		16:00
*** angdraug has joined #openstack-infra		16:00
*** jlanoux has quit IRC		16:00
*** markvoelker has quit IRC		16:01
zxiiro	Any Zuul core-reviewers around who can merge my 2 documentation patches? They've been sitting around for awhile now.... https://review.openstack.org/271868 https://review.openstack.org/271872	16:01
robcresswell	Oh I see, so its just a single patch?	16:01
yolanda	fungi about https://review.openstack.org/#/c/300005/2/modules/openstack_project/manifests/template.pp ... we are normally not using the arrow syntax for dependencies, i actually remember having -1 on some of my patches for that	16:01
*** sdake has quit IRC		16:01
clarkb	robcresswell: no its two separate changes. But the jobs go in first then the project list update controls the jobs	16:01
yolanda	is something we want to continue enforcing or are we ok with arrow deps?	16:01
fungi	yolanda: it's magic symbols to me. suggest a more standard syntax and i'm happy to update the change	16:02
crinkle	yolanda: you can't not use the arrow syntax when using collectors (the <\|\|>)	16:02
robcresswell	clarkb: Gotcha. Thanks.	16:02
*** bmjen has quit IRC		16:03
*** Hunner has quit IRC		16:03
fungi	right, and in this case we need a collector to be able to force the exec to run before any package resource is satisfied, aiui	16:03
crinkle	right	16:03
*** Hunner has joined #openstack-infra		16:03
*** bmjen has joined #openstack-infra		16:03
*** Hunner has quit IRC		16:03
*** Hunner has joined #openstack-infra		16:03
fungi	i assume -> acts similar to a before parameter in this case?	16:03
yolanda	ok so then there is no way to do with requires?	16:04
crinkle	fungi: yes	16:04
clarkb	fungi: crinkle yolanda that said did we confirm we need an explicit exec or is it sufficient to write that file before installing any packages?	16:04
*** Daisy has quit IRC		16:04
*** kzaitsev_mb has joined #openstack-infra		16:04
*** abregman has joined #openstack-infra		16:04
fungi	yolanda: i think to make it work with a requires statement, we'd have to add requires to every single package resource we declare	16:04
*** jsavak has quit IRC		16:04
*** Daisy has joined #openstack-infra		16:04
*** esker has quit IRC		16:04
*** Jeffrey4l_ has quit IRC		16:04
*** Jeffrey4l_ has joined #openstack-infra		16:05
yolanda	we don't want that :) i was thinking there was some alternative with requires or before syntax, as we normally use	16:05
fungi	clarkb: i looked at the puppet stdout order and the sequence was 1. updates sources list, 2. tries to install packages and some are not found, 3. after that failure, i am able to manually install the same packages	16:05
fungi	so something somewhere was doing an apt-get update, but not echoed to the log and not early enough to solve the package install race apparently	16:05
*** achanda has joined #openstack-infra		16:05
*** SumitNaiksatam has joined #openstack-infra		16:06
clarkb	I see, in that case do we want to make that a refreshonly or just unconditionally update?	16:06
fungi	clarkb: i only saw it on bootstrapping, it was eventually consistent, so i think only running it if we modify the sources list should be sufficient	16:06
*** [HeOS] is now known as HeOS		16:07
openstackgerrit	Graham Hayes proposed openstack-infra/project-config: Add designate stable/mitaka to gerritbot https://review.openstack.org/300058	16:07
crinkle	clarkb: it's only really a problem once in a deployment, between when /etc/apt/sources.list is removed and apt-get update is run	16:07
*** asettle has joined #openstack-infra		16:07
crinkle	so i think refreshonly is sufficient	16:07
*** vgridnev has quit IRC		16:07
clarkb	I see	16:07
*** esker has joined #openstack-infra		16:07
clarkb	its the remove add that we are sorting out not just the add, makes sense	16:07
*** zhurong has quit IRC		16:07
*** vgridnev has joined #openstack-infra		16:08
*** salv-orlando has joined #openstack-infra		16:08
clarkb	do we need to explicitly run the purge before the apt-get update as ewll?	16:08
*** vgridnev has quit IRC		16:08
*** thorst is now known as thorst_afk		16:08
*** pcaruana has quit IRC		16:08
*** mohankumar has quit IRC		16:09
*** Daisy has quit IRC		16:09
crinkle	i think the plabs apt module figures that out for us, let me check	16:09
*** esker has quit IRC		16:09
openstackgerrit	Rob Cresswell proposed openstack-infra/project-config: Add requirements job to horizon-cisco-ui https://review.openstack.org/300061	16:10
*** sridhar_ram has joined #openstack-infra		16:10
*** maishsk has joined #openstack-infra		16:10
*** esker has joined #openstack-infra		16:10
openstackgerrit	Rob Cresswell proposed openstack/requirements: Add horizon-cisco-ui to projects.txt https://review.openstack.org/300062	16:11
*** e0ne has joined #openstack-infra		16:11
*** achanda has quit IRC		16:11
openstackgerrit	Shamail Tahir proposed openstack-infra/irc-meetings: Adding AsiaPac Regional PWG Meeting https://review.openstack.org/299776	16:11
*** asettle has quit IRC		16:12
*** Jeffrey4l_ has quit IRC		16:12
*** rfolco has quit IRC		16:12
openstackgerrit	Rob Cresswell proposed openstack-infra/project-config: Add translation-jobs to horizon-cisco-ui https://review.openstack.org/300065	16:14
crinkle	clarkb: the purge will happen before the apt-get update https://github.com/puppetlabs/puppetlabs-apt/blob/master/manifests/init.pp#L108	16:14
crinkle	and we could maybe reuse that apt::update class instead of the exec if we wanted	16:14
*** kushal has joined #openstack-infra		16:14
clarkb	that seems reasonable	16:15
clarkb	delete the Exec and change out the resource on the left side of the ->	16:15
clarkb	and make sure the file is before that	16:15
*** jtomasek has quit IRC		16:16
*** nmagnezi has joined #openstack-infra		16:16
*** fawadkhaliq has joined #openstack-infra		16:19
*** shardy has quit IRC		16:19
*** kzaitsev_mb has quit IRC		16:20
*** kzaitsev_mb has joined #openstack-infra		16:21
*** shardy has joined #openstack-infra		16:21
clarkb	pabelanger: looks like xenial builds are still failing, also we will be cutting it close on disk space on nodepool I think	16:22
*** cdent has quit IRC		16:22
clarkb	so will want to keep an eye on that	16:22
*** cdent has joined #openstack-infra		16:23
clarkb	looks like it is still mad about openjdk 7	16:23
*** sdake has joined #openstack-infra		16:24
*** sdake has quit IRC		16:24
*** mikelk has quit IRC		16:24
clarkb	I cleaned out one extra copy of an image in /opt I made for mounting and editing purposes	16:24
*** sdake has joined #openstack-infra		16:24
clarkb	I don't think there is anymore throw away data on that volume now. Its all dib cache or dib images	16:25
*** sdake has quit IRC		16:28
*** sdake has joined #openstack-infra		16:29
*** apoorvad has joined #openstack-infra		16:29
*** yamahata has quit IRC		16:30
*** sdake has quit IRC		16:30
*** jsavak has joined #openstack-infra		16:30
*** davideagnello has joined #openstack-infra		16:30
*** LaT0rtue has joined #openstack-infra		16:31
*** esker has quit IRC		16:31
dmsimard	clarkb: did you come across my question earlier about running fix_disk_layout ?	16:31
*** e0ne has quit IRC		16:32
clarkb	dmsimard: I did not, is it in scrollback?	16:32
*** esker has joined #openstack-infra		16:32
*** nadya has quit IRC		16:32
clarkb	ah yup its part of the job itself	16:32
dmsimard	clarkb: yeah, but basically it seems like I wrongly assumed fix_disk_layout ran on essentially everything	16:32
clarkb	we avoid the overhead where it isn't necessary (like unittests)	16:32
clarkb	probably the simplest thing is to call the function if you need it	16:33
dmsimard	so should I clone devstack-gate and just call fix_disk_layout before starting my testsor something like that ?	16:33
dmsimard	is devstack gate implicitly cloned somewhere already ?	16:33
clarkb	we could also put it in the jenkins scripts instead of devstack-gate so that you don't need to hvae devstack gate cloned	16:33
clarkb	nothing is implicitly cloned. The slaves do have a repo cache on them and if you use zuul-cloner it will take advantage of that. The devstack-gate jobs do this for the devstack-gate setup	16:34
openstackgerrit	John L. Villalovos proposed openstack-infra/project-config: Ironic grenade jobs do not work until after Mitaka https://review.openstack.org/300082	16:34
clarkb	I think it even has a macro you can reuse	16:34
*** jerryz has joined #openstack-infra		16:34
*** rossella_s has quit IRC		16:34
*** mylu has quit IRC		16:34
dmsimard	clarkb: I guess https://github.com/openstack-infra/project-config/blob/master/jenkins/jobs/macros.yaml#L573	16:35
clarkb	dmsimard: yup	16:35
*** rossella_s has joined #openstack-infra		16:35
*** esker has quit IRC		16:36
*** fawadkhaliq has quit IRC		16:36
openstackgerrit	John Trowbridge proposed openstack-infra/project-config: Add tripleo-quickstart project to tripleo team https://review.openstack.org/299932	16:37
*** fawadkhaliq has joined #openstack-infra		16:37
dmsimard	clarkb: okay I'll try something along these lines. Thanks for the help.	16:39
*** sfinucan has quit IRC		16:39
*** cdent has quit IRC		16:41
*** fawadkhaliq has quit IRC		16:41
*** trown has joined #openstack-infra		16:41
*** fawadkhaliq has joined #openstack-infra		16:41
*** pvaneck has joined #openstack-infra		16:42
*** matrohon has quit IRC		16:42
trown	can I ignore the -1 Jenkins on https://review.openstack.org/#/c/300084/ or is there something I need to do for that?	16:43
*** cdent has joined #openstack-infra		16:43
clarkb	trown: "This change depends on a change that failed to merge"	16:44
clarkb	you will likely need to address that if you wish to get it merged	16:44
jroll	trown: recheck it, you'll get that if you update the dependent patch while CI is running (which you did)	16:44
AJaeger	trown: add a "recheck" once the other change has passed testing	16:44
trown	ah perfect, thanks!	16:45
*** e0ne has joined #openstack-infra		16:47
*** piet has quit IRC		16:48
*** sdake has joined #openstack-infra		16:48
*** maishsk has quit IRC		16:48
*** yaume has quit IRC		16:49
*** e0ne has quit IRC		16:50
*** dizquierdo has quit IRC		16:51
*** resker has joined #openstack-infra		16:51
*** maishsk has joined #openstack-infra		16:52
clarkb	pabelanger: ianw: do we have a rough plan for getting off of devstack-trusty yet? I think I have seen some experimental jobs for centos-7 floating around	16:52
openstackgerrit	Rob Cresswell proposed openstack-infra/project-config: Add translation-jobs to horizon-cisco-ui https://review.openstack.org/300065	16:53
*** nwkarsten has quit IRC		16:54
*** esker_ has joined #openstack-infra		16:54
*** nwkarsten has joined #openstack-infra		16:54
*** piet has joined #openstack-infra		16:55
*** flongo has quit IRC		16:55
*** resker has quit IRC		16:56
clarkb	fungi: crinkle so what do you think? should we update that change to use the update resource that already exists? or should I go ahead and approve it as is?	16:57
*** thorst_afk is now known as thorst		16:57
clarkb	I think my preference is to avoid the exec if we can but its not strong enough to not merge as is	16:57
*** rpodolyaka is now known as rpodolyaka_afk		16:57
crinkle	i have a slight preference for using the existing apt::update class	16:57
crinkle	fungi: ^	16:58
fungi	clarkb: i'm open to either solution. avoided using the apt module for managing the sources list since it didn't really have the features we needed there, but for calling update it seems perfectly well suited	16:58
fungi	i'll update the change shortly	16:58
clarkb	sounds good thanks	16:58
*** esker_ has quit IRC		16:58
*** nwkarsten has quit IRC		16:58
*** mylu has joined #openstack-infra		16:59
openstackgerrit	sebastian marcet proposed openstack-infra/puppet-openstackid: Tweaked server connection settings https://review.openstack.org/300097	16:59
*** nadya has joined #openstack-infra		16:59
*** nwkarsten has joined #openstack-infra		16:59
*** jistr has quit IRC		17:00
*** nadya has quit IRC		17:01
*** rbrndt has joined #openstack-infra		17:03
openstackgerrit	Merged openstack-infra/release-tools: script to propose final releases for a series https://review.openstack.org/299551	17:03
*** yamamoto_ has quit IRC		17:03
*** yamamoto_ has joined #openstack-infra		17:04
*** jlanoux_ has quit IRC		17:04
*** Guest11037 has quit IRC		17:05
*** tosky has joined #openstack-infra		17:05
openstackgerrit	Derek Higgins proposed openstack-infra/tripleo-ci: Fix image caching logic https://review.openstack.org/299537	17:05
openstackgerrit	Derek Higgins proposed openstack-infra/tripleo-ci: Pre install packages on the instack image https://review.openstack.org/299538	17:05
AJaeger	clarkb, pleia2 : Could you review rossella_s's reviewday/neutron change again, please? https://review.openstack.org/#/c/298779	17:05
*** yamahata has joined #openstack-infra		17:05
*** ZZelle has quit IRC		17:05
*** ZZelle has joined #openstack-infra		17:06
*** rcernin has quit IRC		17:06
*** tqtran has joined #openstack-infra		17:06
*** derekh has quit IRC		17:06
rossella_s	AJaeger, thanks a lot for pushing this forward :)	17:07
*** esikachev has quit IRC		17:08
*** ihrachys has quit IRC		17:08
*** tqtran has quit IRC		17:10
*** angdraug has quit IRC		17:10
mordred	infra-root, Shrews: tagging shade 1.7.0 unless there are outcries of derision	17:13
*** lezbar has quit IRC		17:13
fungi	mordred: outcries only of elation from me	17:13
clarkb	we'll just throw groceries if it breaks	17:13
fungi	and wet goats	17:13
*** woodster_ has joined #openstack-infra		17:14
*** achanda has joined #openstack-infra		17:15
*** rajinir has quit IRC		17:15
*** IlyaG has quit IRC		17:16
*** kzaitsev_mb has quit IRC		17:16
*** sigmavirus24 is now known as sigmavirus24_awa		17:17
anteaya	can I use them to sop up water from my basement before you toss them?	17:20
pabelanger	clarkb: ianw: Yup, I am moving from devstack-centos7 to centos-7 now.	17:20
pabelanger	clarkb: will create a topic	17:21
pabelanger	clarkb: also, landed a fix for xenial today, checking the status now	17:21
clarkb	rossella_s: commented, one thing about pathing that I don't expect to work quite right	17:21
clarkb	pabelanger: I think xenial is waiting for the daily builds to finish then it will start again	17:22
pabelanger	clarkb: Yup, it should work this time	17:22
clarkb	ok	17:22
*** sputnik13_ has joined #openstack-infra		17:22
rossella_s	clarkb, thanks!	17:23
AJaeger	pabelanger: are there any changes you need reviews for?	17:23
pabelanger	clarkb: AJaeger: https://review.openstack.org/#/q/topic:centos-7 for migration from devstack-centos7	17:23
pabelanger	going to add reset of ansible jobs today too	17:23
mordred	fungi, clarkb: done	17:23
Shrews	mordred: you don't want the server list change in?	17:24
*** sputnik13net has joined #openstack-infra		17:24
Shrews	mordred: looks like it just needs a recheck	17:24
mordred	BLASTIT	17:24
Shrews	yay 1.8!!!	17:25
fungi	mmmmmmore tags!	17:25
mordred	sorry. my query to check that everything had been landed was broke	17:25
*** daemontool has quit IRC		17:25
mordred	Shrews: it can be a 1.7.1 - it's just fixing a thundering herd	17:25
AJaeger	clarkb: do you want to review some of pabelanger's changes? Those three have a +2 from me already...	17:25
Shrews	yay 1.7.1!	17:25
mordred	all hail 1.7.1	17:26
openstackgerrit	Khai Do proposed openstack-infra/system-config: Replace the Gerrit icon for Workflow -1 vote. https://review.openstack.org/165228	17:26
clarkb	AJaeger: yup I will queue those up	17:26
mordred	clarkb, fungi: I just sent one of my regular emails to the mailing list reminding people to not think that floating ips are the same thing as public ips	17:26
clarkb	mordred: yup and +1 to changing devstack's defaults even if the "public" range in devstack is unroutable in most devstacks	17:26
mordred	yup	17:27
fungi	flublic ips	17:27
*** sputnik13 has quit IRC		17:27
mordred	clarkb: yah - people need to start grokjking th edual model, since it is the defacto standard way opentsack works	17:27
*** fawadkhaliq has quit IRC		17:28
openstackgerrit	sebastian marcet proposed openstack-infra/puppet-openstackid: Tweaked server connection settings https://review.openstack.org/300097	17:29
*** rbrndt has quit IRC		17:30
*** fawadkhaliq has joined #openstack-infra		17:30
*** kzaitsev_mb has joined #openstack-infra		17:30
timrc	clarkb: In TX we have "BBQ resorts" (or will, soon)... http://austin.eater.com/2016/3/30/11332762/texas-chainsaw-massacre-barbecue-resort-bastrop	17:30
mordred	++	17:31
clarkb	pabelanger: what is windmill	17:31
rossella_s	clarkb, sorry I don't get your comment...what do you mean exactly regarding relying on path resolution? I don't know exactly how all this is installed on the infra server..can you maybe explain me please, thanks!	17:31
openstackgerrit	Paul Belanger proposed openstack-infra/project-config: Switch ansible-role-functional-jobs-centos7 to centos-7 https://review.openstack.org/300104	17:31
clarkb	rossella_s: src/gerrit-dash-creator is where you have cloned that repo to	17:31
zaro	crinkle, yolanda : would you be interested in getting puppet code highlighting in gerrit? https://review.openstack.org/#/c/260324/	17:31
clarkb	rossella_s: but if you don't clone the repo and just pip install it that is not where it will end up	17:31
clarkb	rossella_s: you can confirm this by running the pip install for it in a virtuaelnv and see how it gets laid on in the virtualenv	17:31
pabelanger	clarkb: ansible playbook to test some roles I have upstream. I used to deploy / test my CI changes	17:32
*** yonglihe has quit IRC		17:32
clarkb	pabelanger: ok just making sure we eren't changing things under someone that would be surprised	17:32
mnaser	fungi: made some changes to the scheduler, new instances should perform better as the importance of compute nodes with less CPU contention was increased	17:32
pabelanger	clarkb: Ya, I've tested it yesterday and already using bindep for it	17:32
mnaser	existing instances might still struggle but new ones should (hopefully) perform much better	17:32
*** tiswanso has joined #openstack-infra		17:33
fungi	mnaser: we turn them over pretty rapidly so should see an improvement shortly if so--thanks!	17:33
fungi	mnaser: did that seem to be the case for the uuids i gave you?	17:33
AJaeger	clarkb: looking at puppet-reviewday, do we pip install at all? I know too little about puppet but wonder how this gets installed...	17:34
*** akaszuba has quit IRC		17:34
clarkb	AJaeger: I would assume so, but let me double check	17:34
mnaser	fungi: yeah, they were on compute nodes that had a lot of free memory but heavy cpu contention (those are some older compute nodes which was being selected)	17:34
anteaya	so storyboard folks need to create a gerrit account for their bot to interact with gerrit and storyboard	17:35
*** esker has joined #openstack-infra		17:35
crinkle	zaro: ooh neat	17:35
anteaya	to naming options are Codethink StoryBoard Bot or just StoryBoard Bot	17:35
anteaya	I'm thinking StoryBoard Bot would work	17:35
clarkb	AJaeger: you are correct it apperas we do not	17:35
anteaya	any objections?	17:35
clarkb	AJaeger: but maybe we should	17:35
*** jsavak has quit IRC		17:35
AJaeger	fungi, pabelanger I tried to add an other-requirements file to swift following the nova change - looking at cschwede's comment, I'm a bit confused. Could either of you check this, please? https://review.openstack.org/#/c/298313/5	17:36
*** jsavak has joined #openstack-infra		17:36
rossella_s	clarkb, I ran the pip install on the host, not in a virtual env, anyway according to the doc of pip it will be placed in <venv path>/src/	17:36
fungi	anteaya: Zara: no objection from me on "StoryBoard Bot"	17:36
anteaya	fungi: thank you	17:36
AJaeger	clarkb: I wonder how the dependency like gerrit-dash-creator will get installed if we don't run "pip install"	17:36
*** sdague has quit IRC		17:36
clarkb	AJaeger: it won't :) so we will need ot address that	17:37
*** sridhar_ram is now known as sridhar_ram_afk		17:37
anteaya	Zara: give it another few minutes to let anyone else share an opinion if they have one	17:37
clarkb	rossella_s: ya it should be all relative to the base python install	17:37
clarkb	rossella_s: it is possible that pkg_resources will give you that rooted path now that I think about it	17:38
clarkb	rossella_s: since pkg_resources knows about the things that are installed	17:38
openstackgerrit	Merged openstack-infra/project-config: Add centos-7 to bindep-fallback https://review.openstack.org/299988	17:38
*** dimtruck is now known as zz_dimtruck		17:38
*** sdague has joined #openstack-infra		17:38
mugsie	I am seeing some weird errors in gate right now - its looking for traceroute & traceroute6 but they are not installed	17:39
mugsie	is this a known issue?	17:39
*** esker has quit IRC		17:39
mugsie	http://logs.openstack.org/84/296784/1/gate/gate-designate-dsvm-powerdns/c4ca379/console.html	17:39
rossella_s	clarkb, you mean querying pkg_resources from the python code to get the path?	17:39
pabelanger	AJaeger: So, until we add fedora / centos-7 gate checks to other-requirements.txt I see adding support for them as a best effort.	17:39
*** tqtran has joined #openstack-infra		17:39
openstackgerrit	Merged openstack-infra/project-config: Add experimental centos-7 puppet-apply https://review.openstack.org/293409	17:39
clarkb	rossella_s: yup	17:39
pabelanger	AJaeger: I'd rather land ubuntu support, and figure out add a bindep job, and follow up patch for fedora support	17:39
*** dims_ has quit IRC		17:39
AJaeger	pabelanger: agreed. The other issue is the double "tox -e bindep" invocation.	17:40
fungi	mugsie: uhoh, we need to revert that. i didn't even think about the possibility	17:40
clarkb	mugsie: I direct yout to fungi :)	17:40
mugsie	:)	17:40
pabelanger	AJaeger: which double tox?	17:40
pabelanger	the comments are 2 different OSes, showing different dependencies	17:40
*** dims has joined #openstack-infra		17:40
openstackgerrit	Jeremy Stanley proposed openstack-infra/project-config: Revert "Traceroute to git.o.o in net-info macro" https://review.openstack.org/300111	17:41
*** zz_dimtruck is now known as dimtruck		17:41
fungi	i'm directly submitting that now	17:41
openstackgerrit	Merged openstack-infra/project-config: Move windmill to centos-7 dib https://review.openstack.org/299597	17:41
AJaeger	pabelanger: "So, the error on the first bindep run keeps me from adding a +2 on this yet..."	17:41
mugsie	fungi: great, thanks	17:41
fungi	and then i'll manually jjb everything	17:41
AJaeger	pabelanger: "Interestingly, this was only when running "tox -e bindep" the first time, ie when creating the venv. The second run returned the following outputs"	17:41
pabelanger	AJaeger: well, that is tox that is failing to compile python libs right?	17:41
AJaeger	pabelanger: yes	17:42
pabelanger	So, tox -edocs would fail too	17:42
AJaeger	pabelanger: indeed.	17:42
pabelanger	basically, gcc is a dependency of our tox	17:42
openstackgerrit	Merged openstack-infra/project-config: Revert "Traceroute to git.o.o in net-info macro" https://review.openstack.org/300111	17:42
fungi	jjb'ing commencing now	17:42
*** mylu has quit IRC		17:42
pabelanger	AJaeger: So, for that, you need to bootstrap your Os to run tox and tox dependency. For me it is usually, python-pip, gcc	17:43
mnaser	fungi: is there a way to see average times for a specific commonly ran job?	17:43
pabelanger	otherwise, tox doesn't work	17:43
AJaeger	pabelanger: So, is adding "usedevelop = False" a good recommendation? I doubt it.	17:43
mnaser	i'd like to see if the average run time for a commonly run job is running faster overall	17:43
AJaeger	pabelanger: could you comment on the review, please? Thanks for thinking this through, yes, this is valid.	17:43
pabelanger	AJaeger: Ya, not sure about that. I would have somebody else confirm that	17:43
pabelanger	AJaeger: sure	17:43
*** nwkarsten has quit IRC		17:44
*** skolekonov is now known as skolekonov\|away		17:44
Shrews	mordred: oh, wow. the server list change is failing all over the place now	17:44
fungi	Shrews: that may have been me	17:44
*** nwkarsten has joined #openstack-infra		17:44
Shrews	traceroute: not found	17:44
Shrews	fungi: you broke tracing the routes???	17:44
fungi	yep, i'm undoing it as we speak	17:44
fungi	i did indeed	17:45
*** shardy has quit IRC		17:45
openstackgerrit	Chris Dent proposed openstack/requirements: Add microversion-parse to requirements https://review.openstack.org/300113	17:46
*** fawadkhaliq has quit IRC		17:46
AJaeger	fungi, so, first adding the packages to build images, then building new images, then merging again?	17:46
*** fawadkhaliq has joined #openstack-infra		17:46
dmsimard	clarkb: I've been trying different ways of running the fix disk layout but I feel your idea to just take that out of functions.sh and into a jenkins slave script by itself is the right one	17:47
fungi	#status notice Job failures for missing traceroute packages are in the process of being fixed now, ETA 30 minutes to effectiveness for new jobs	17:47
openstackstatus	fungi: sending notice	17:47
*** fawadkhaliq has quit IRC		17:47
*** jsavak has quit IRC		17:47
*** IlyaG has joined #openstack-infra		17:47
clarkb	dmsimard: I wouldn't delete it from devstack-gate to start, first step I would copy it then migrate devstack-gate away from its old thing	17:47
*** jsavak has joined #openstack-infra		17:47
clarkb	or possibly leave it in devstack-gate too I dunno	17:47
*** fawadkhaliq has joined #openstack-infra		17:47
dmsimard	clarkb: I planned on doing a depends-on, but I can do that too	17:47
fungi	AJaeger: yeah, i think we probably want to add whatever provides traceroute and traceroute6 to our infra-package-needs element	17:47
*** fawadkhaliq has quit IRC		17:47
pabelanger	Ah, what happen with traceroute patch?	17:48
fungi	or perhaps in puppet	17:48
pabelanger	err, oh	17:48
*** degorenko is now known as _degorenko\|afk		17:48
*** fawadkhaliq has joined #openstack-infra		17:48
-openstackstatus- NOTICE: Job failures for missing traceroute packages are in the process of being fixed now, ETA 30 minutes to effectiveness for new jobs		17:48
fungi	minimal images being _minimal_ again	17:48
pabelanger	AJaeger: comment left	17:48
*** fawadkhaliq has quit IRC		17:48
AJaeger	thanks, pabelanger	17:48
fungi	i'm kicking myself for not confirming those were installed	17:49
*** nwkarsten has quit IRC		17:49
pabelanger	Doh	17:49
*** fawadkhaliq has joined #openstack-infra		17:49
*** jaosorior has joined #openstack-infra		17:49
openstackstatus	fungi: finished sending notice	17:49
*** fawadkhaliq has quit IRC		17:50
*** mriedem has quit IRC		17:50
*** shashank_hegde has joined #openstack-infra		17:50
*** vgridnev has joined #openstack-infra		17:50
*** fawadkhaliq has joined #openstack-infra		17:50
openstackgerrit	Paul Belanger proposed openstack-infra/project-config: Migrate chef jobs to centos-7 https://review.openstack.org/300114	17:50
*** mriedem has joined #openstack-infra		17:51
AJaeger	fungi, see http://packages.ubuntu.com/trusty/amd64/traceroute/filelist	17:51
*** mrmartin has joined #openstack-infra		17:51
AJaeger	I just wonder why it's /usr/sbin/tcptraceroute.db	17:51
AJaeger	I mean /usr/bin/traceroute.db - still why .db?	17:53
pleia2	it's just how it is, always been that way as far as I can recall	17:53
*** netsin has joined #openstack-infra		17:53
AJaeger	pleia2: so, traceroute.db instead of traceroute?	17:53
*** gyee has joined #openstack-infra		17:53
pleia2	AJaeger: that's tcptraceroute	17:54
pleia2	regular traceroute is still in /usr/bin	17:54
pleia2	no .db	17:54
openstackgerrit	Paul Belanger proposed openstack-infra/project-config: Move gate-solum-devstack-dsvm-centos7 to centos-7 https://review.openstack.org/300117	17:54
AJaeger	pleia2: looking at the filelist, it's not	17:54
clarkb	maybe an install script makes a symlink? I can check my local install	17:55
*** sridhar_ram_afk is now known as sridhar_ram		17:55
fungi	which executable provides the traceroute command is handled by the alternatives system (at least on debian)	17:55
clarkb	lrwxrwxrwx 1 root root 33 Aug 13 2015 /usr/sbin/traceroute -> /etc/alternatives/traceroute.sbin	17:55
clarkb	and lrwxrwxrwx 1 root root 22 Aug 13 2015 /etc/alternatives/traceroute.sbin -> /usr/bin/traceroute.db	17:55
clarkb	so ya	17:55
AJaeger	fungi, clarkb: Ah, thanks!	17:56
openstackgerrit	Paul Belanger proposed openstack-infra/project-config: Move gate-tempest-dsvm-docker-centos7 to centos-7 https://review.openstack.org/300119	17:56
fungi	you can't have a /usr/bin/traceroute binary _and_ a symlink at the same path, so the binary has to be renamed	17:56
fungi	i think the .db was for "debian" but i don't really know	17:56
pleia2	ah yes, thanks fungi	17:56
AJaeger	fungi: yeah, missed the alternatives	17:57
openstackgerrit	David Moreau Simard proposed openstack-infra/project-config: Add fix_disk_layout script and builder https://review.openstack.org/300122	17:57
openstackgerrit	David Moreau Simard proposed openstack-infra/project-config: Ensure VMs used in Packstack jobs have consistent swap configuration https://review.openstack.org/300123	17:57
dmsimard	clarkb: ^	17:57
*** esikachev has joined #openstack-infra		17:58
openstackgerrit	Doug Hellmann proposed openstack-infra/project-config: upload both sdist and wheel on pre-release https://review.openstack.org/300124	17:58
openstackgerrit	David Moreau Simard proposed openstack-infra/project-config: Add fix_disk_layout script and builder https://review.openstack.org/300122	17:59
dmsimard	I nitpicked myself	17:59
clarkb	traceroute is an odd thing to worry about alternatives for	17:59
openstackgerrit	Paul Belanger proposed openstack-infra/project-config: Move periodic-package-afs-centos7 to centos-7 https://review.openstack.org/300126	17:59
clarkb	how many traceroutes do you need?	17:59
fungi	well, there's bsd traceroute, tcptraceroute, i think hping2 might also provide a traceroute tool?	18:00
fungi	quite a few floating around, probably more i'm forgetting	18:00
Clint	iputils	18:00
fungi	yep, that	18:00
dmsimard	mtr > traceroute :)	18:00
pleia2	heh	18:00
*** jaosorior has quit IRC		18:01
pleia2	I just went to the movies with the author of tcptraceroute, I should ask him	18:02
fungi	jenkins03 updated in a mere 19 minutes. that was fast	18:03
*** zara_the_lemur__ has joined #openstack-infra		18:04
mnaser	completetly unrelated but	18:04
mnaser	jenkins css is a bit messed up :)	18:05
fungi	yeah, we know	18:05
mnaser	the background that says Jenkins CI is under the actaul logo, maybe a missing display: none;	18:05
*** kzaitsev_mb has quit IRC		18:05
fungi	i think they tweaked something about their css/dom in the version we just upgraded to	18:05
*** kzaitsev_mb has joined #openstack-infra		18:06
*** fawadk has joined #openstack-infra		18:06
*** jsavak has quit IRC		18:06
*** fawadkhaliq has quit IRC		18:07
zara_the_lemur__	hi, so it's late in the day for me, but we've been talking (again!) about updating storyboard task statuses from gerrit. so some questions	18:07
*** jsavak has joined #openstack-infra		18:07
fungi	lemur is such a fun word	18:07
zara_the_lemur__	fungi: :D I'm named after my long tail of underscores	18:08
clarkb	fungi: for some reason that made me think of velure which put zapp brannigans voice in my head	18:08
fungi	there's an awesome lemur habitat and research facility i used to go to a lot back when i lived closer	18:08
zara_the_lemur__	(the questions are 1) if we make a user acct for a storyboard bot in launchpad, what's the best email to use for it? 2) are we okay to call it 'StoryBoard Bot'? )	18:09
pleia2	fungi: aw, that's awesome	18:09
*** fawadk has quit IRC		18:09
fungi	pleia2: if you're ever around durham, nc you should go	18:09
pleia2	fungi: noted :)	18:09
clarkb	zara_the_lemur__: we already have an account for that in launchpad	18:09
clarkb	zara_the_lemur__: gerrig	18:09
*** gyee has quit IRC		18:10
clarkb	or openstack-hudson	18:10
mnaser	clarkb: we've made updates/changes to the ceph cluster so these glance image upload issues should be gone (so if you see any failures agian, please do let me know!)	18:10
clarkb	mnaser: will do thank you	18:10
*** SumitNaiksatam has quit IRC		18:10
clarkb	mnaser: also not sure if you saw that nova bug about the config drive and cinder conflict over the device name, nova is asking if you cna provide the version of libvirt in use there	18:10
pabelanger	clarkb: ubuntu-xenial build started	18:10
fungi	clarkb: zara_the_lemur__: it's https://launchpad.net/~hudson-openstack	18:10
mnaser	clarkb: oh i did not, not sure im subscribed to that bug, would you have a link?	18:11
clarkb	mnaser: https://bugs.launchpad.net/bugs/1558807	18:11
openstack	Launchpad bug 1558807 in OpenStack Compute (nova) "Volume attached at different path than reported by nova/cinder when using config drive" [Undecided,Incomplete]	18:11
trown	AJaeger: I got both present and future PTL sign off on https://review.openstack.org/#/c/299932 :)	18:11
*** kzaitsev_mb has quit IRC		18:11
fungi	trown: the future is now	18:11
*** Sukhdev has joined #openstack-infra		18:12
trown	and also the future	18:12
fungi	this is true	18:12
AJaeger	trown: then I have to +2 ;)	18:12
trown	\o/ thanks	18:13
*** devananda has quit IRC		18:14
mnaser	clarkb: replied, thanks	18:14
*** ajmiller has quit IRC		18:14
*** gyee has joined #openstack-infra		18:14
smarcet	fungi: when u get some time, please review https://review.openstack.org/#/c/300097/ :)	18:14
*** tqtran has quit IRC		18:14
*** ajmiller has joined #openstack-infra		18:14
zara_the_lemur__	clarkb: fungi: I'm confused; is that capable of updating storyboard task statuses somehow? I think I thought that was for updating launchpad based on a bug's status in another task tracker? or maybe I'm wrong again?	18:15
AJaeger	mordred, fungi, clarkb: Could you review https://review.openstack.org/#/c/300124/ and explain why for pre-releases we only upload wheels but no tarballs to pypi? This seems to be the case since two years now.	18:15
*** Sukhdev has quit IRC		18:15
fungi	smarcet: excellent--i was waiting for test results to come back on that second patchset and got sidetracked. approved now so we should see that on the server shortly	18:15
smarcet	ok cool tnx	18:15
AJaeger	dhellmann: https://review.openstack.org/#/c/56760/ has you as committer and mordred as initial author ;)	18:15
mordred	AJaeger: it has to do with supportin old pip	18:15
mnaser	Are the respnose times on this page https://jenkins01.openstack.org/computer/ (and other jenkins) graphed somewhere? I'd like to see if we can trend the response times for the nodes deployed on our side to see if the scheduler changes we did have improved things	18:15
clarkb	zara_the_lemur__: it should be able to authenticate to an openid consumer and perform actions	18:16
*** dizquierdo has joined #openstack-infra		18:16
zara_the_lemur__	clarkb: ohh, I see, sorry.	18:16
AJaeger	mordred: is it safe now to change it - or should we -2 300124 ?	18:16
fungi	AJaeger: it's because versions of pip old enough to know that prereleases are not to be installed without explicitly requesting them are also old enough to know how to find and install wheels by default	18:16
mordred	AJaeger: old pip that doesn't grok pre-release versions also doesn't grok wheels	18:16
mordred	I think it's safe now personally	18:16
mordred	becuase we just flat do not support versions of pip that old anymore	18:16
clarkb	mnaser: the openstack health dashboard keeps track of job runtimes	18:16
clarkb	I think	18:17
clarkb	let me see if I can find that	18:17
*** baoli_ has quit IRC		18:17
fungi	yeah, it's mainly an issue for people wanting to use default distro-provided pip in ubuntu precise, centos 6, et cetera	18:17
mordred	we have job runtimes in graphite	18:17
* mordred looks at his private graphs real quick		18:17
clarkb	yes grafana.openstack.org/dashboard/db/nodepool-vexxhost shows two of the jobs but not a real aggregate	18:17
zara_the_lemur__	thanks for the help, all :)	18:17
dhellmann	mordred , AJaeger : if we don't want pre-release sdists we should probably add a comment in the layout file. We hit this with trove-dashboard, and the release team needs to sort out whether UI plugins are "libraries" or something else, and if they should even be doing release candidates like this, but it wasn't obvious why we had separate job definitions there	18:17
mordred	dhellmann: I think we should just release all the things now	18:18
mordred	dhellmann: we waited long enough	18:18
dhellmann	mordred : wfm. AJaeger, I should update the comment in my new patch, let me do that.	18:18
*** FallenPegasus has joined #openstack-infra		18:18
AJaeger	dhellmann: and remove the colon	18:18
*** jcoufal_ has quit IRC		18:19
dhellmann	AJaeger : well spotted, yes	18:19
*** sigmavirus24_awa is now known as sigmavirus24		18:20
AJaeger	clarkb, fungi, mordred: Nakato has updated the change to use zuul-cloner in post pipeline. It's quite a change in usage IMHO so once you have some spare time, please review whether the changed semantics are what we need - and the code as well: https://review.openstack.org/293194	18:20
mnaser	yeah, the nodepool grafana just shows those 2 jobs which don't seem to run all that often on our side	18:20
*** zul has quit IRC		18:20
clarkb	mnaser: ya so you could take those graphs and add additional ones for all the jobs whihc I assume is what mordred has hidden away	18:20
pabelanger	mordred: still planning on nodepool-shade upgrade today?	18:21
clarkb	mnaser: http://status.openstack.org/openstack-health/#/g/node_provider/ymq-1?groupKey=node_provider&resolutionKey=hour&duration=P7D&end=2016-03-31T18:17:10.558Z doesn't appear to have the runtimes there	18:21
clarkb	so doing like what mordred has may be best (and maybe we add that to grafana)	18:22
*** zul has joined #openstack-infra		18:22
openstackgerrit	Doug Hellmann proposed openstack-infra/project-config: upload both sdist and wheel on pre-release https://review.openstack.org/300124	18:22
dhellmann	AJaeger, mordred : ^^	18:22
*** sigmavirus24 is now known as sigmavirus24_awa		18:22
fungi	AJaeger: on 298313 for swift, it might make sense to wait for https://review.openstack.org/200672 to merge to nova first. would like to get lifeless to review that too and see if he has suggestions for how we should be better leveraging bindep (maybe with a "test" profile for some packages or something?). that was a minimal "here's what i think it looks like" patch	18:23
*** sigmavirus24_awa is now known as sigmavirus24		18:24
fungi	sufficient to pass tests, maybe still needs polish before we make it a recommendation	18:24
*** bpokorny has quit IRC		18:24
*** dims_ has joined #openstack-infra		18:24
*** yolanda has quit IRC		18:24
*** dims has quit IRC		18:24
AJaeger	fungi, ok. let's point both reviews out to lifeless then ;)	18:25
AJaeger	thanks, fungi	18:25
openstackgerrit	Merged openstack-infra/puppet-openstackid: Tweaked server connection settings https://review.openstack.org/300097	18:25
*** rfolco has joined #openstack-infra		18:25
fungi	i think the concerns raised by cschwede on 298313 may also apply to 200672	18:25
*** salv-orlando has quit IRC		18:26
fungi	as it looks like i missed that we need to add usedevelop = False there too so as to prevent trying to do a develop install of nova	18:26
*** FallenPegasus is now known as MarkAtwood		18:26
AJaeger	fungi: yep	18:26
fungi	when i add that, it's way faster	18:26
clarkb	pabelanger: I think xenial failed, trying to figure out where things are in this log	18:27
fungi	waaaaay faster	18:27
*** devananda has joined #openstack-infra		18:27
fungi	updating now	18:27
*** jiggalanator has quit IRC		18:27
fungi	cschwede: thanks for spotting that!	18:27
*** mrmartin has quit IRC		18:27
*** yolanda has joined #openstack-infra		18:27
pabelanger	clarkb: still using openjdk-7	18:28
clarkb	pabelanger: or maybe I am misreading logs? so much information to process	18:28
pabelanger	hmm	18:28
clarkb	ah yup there it is	18:28
pabelanger	checking logs why	18:28
*** nwkarsten has joined #openstack-infra		18:29
*** nwkarsten has quit IRC		18:29
clarkb	fungi: wait develop should be faster	18:29
*** fawadkhaliq has joined #openstack-infra		18:29
fungi	clarkb: why?	18:29
clarkb	odd that it slows stuff down	18:29
mordred	I agree with clarkb	18:29
mordred	develop avoids the need to make a tarball	18:29
clarkb	fungi: because it doens't make an sdist it just symlinks some stuff around	18:29
fungi	clarkb: usedevelop = True causes it to pip install the repo	18:29
mordred	and just makes an egg-link in site-packages	18:29
*** nwkarsten has joined #openstack-infra		18:29
clarkb	fungi: tox should always pip install the repo	18:30
mordred	++	18:30
fungi	oh, hrm. double-checking	18:30
clarkb	not using develop is fine, its just odd that that would make stuff faster	18:30
mordred	might want to make sure the skipsdist = True is in there too	18:30
AJaeger	pabelanger: question on your comment at https://review.openstack.org/#/c/300126/1/jenkins/jobs/infra.yaml	18:30
mordred	otherwise you'd buidling that sdist AND then doing a develop install	18:30
mordred	fungi: ^^	18:31
AJaeger	pabelanger: I doubt there's a other-requirments file that we could add the content to - or can we?	18:31
fungi	clarkb: mordred: it most decidedly does _not_ install nova	18:31
fungi	if i add usedevelop = False	18:31
fungi	according to .tox/bindep/bin/pip list	18:31
mordred	fungi: what thing are we looking at so that I can look at?	18:31
fungi	mordred: just a sec and let me push up this new patchset	18:32
*** zul has quit IRC		18:32
AJaeger	mordred: https://review.openstack.org/298313 and https://review.openstack.org/200672	18:32
AJaeger	mordred: fungi is updated 200672	18:32
clarkb	mnaser: I might be blind but I don't see any comment on https://bugs.launchpad.net/nova/+bug/1558807	18:32
openstack	Launchpad bug 1558807 in OpenStack Compute (nova) "Volume attached at different path than reported by nova/cinder when using config drive" [Undecided,Incomplete]	18:32
*** zul has joined #openstack-infra		18:32
pabelanger	clarkb: odd: https://review.openstack.org/#/c/299948/ didn't merge	18:33
fungi	AJaeger: mordred: clarkb: https://review.openstack.org/200672 is now commented and updated accordingly	18:33
AJaeger	pabelanger: what is ubuntu-centos7? ;)	18:34
pabelanger	AJaeger: .trolo a fail by me :)	18:34
fungi	sounds both scary and awesome	18:34
mordred	an awesome new thing	18:34
mordred	sort of like Ubuntu Windows 10	18:34
pabelanger	zing!	18:34
pleia2	hah	18:34
*** kushal has quit IRC		18:35
pabelanger	AJaeger: which review, I can fix	18:35
clarkb	mordred: wow	18:36
AJaeger	pabelanger: https://review.openstack.org/300117	18:37
openstackgerrit	Paul Belanger proposed openstack-infra/project-config: Move periodic-package-afs-centos7 to centos-7 https://review.openstack.org/300126	18:37
mordred	fungi: OH	18:38
mordred	fungi: I get it now	18:38
openstackgerrit	Paul Belanger proposed openstack-infra/project-config: Move gate-solum-devstack-dsvm-centos7 to centos-7 https://review.openstack.org/300117	18:38
*** esikachev has quit IRC		18:38
mordred	fungi: with skipsdist AND usedevelop=False then nova does not get installed, which is what you are, in fact, looking for here	18:38
*** cdent has quit IRC		18:38
pabelanger	AJaeger: both fixed	18:39
*** piet has quit IRC		18:39
fungi	mordred: should i explicitly skipdist in that env just to be extra safe future-proofing and for the benefit of people cargo-culting examples?	18:39
mordred	skipsdist is a global not an env setting	18:39
fungi	oh, okay	18:39
mordred	a comment might be useful though	18:39
fungi	definitely	18:39
*** piet has joined #openstack-infra		18:39
AJaeger	mordred: do you want to +2 dhellmann's pre-release change ? https://review.openstack.org/#/c/300124	18:40
*** bpokorny has joined #openstack-infra		18:40
*** annegentle has quit IRC		18:40
lifeless	fungi: +1 on a test profile	18:40
*** bpokorny has quit IRC		18:40
fungi	"note that this needs tox.skipdist = True to be effective"	18:40
*** zara_the_lemur__ has quit IRC		18:40
lifeless	fungi: I shall look in more detail later today - I have meetingish things end to end this morning though	18:40
*** annegentle has joined #openstack-infra		18:40
fungi	lifeless: more hoping you have a concrete example of what that would look like, and what we should be doing to make sure we include the default profile for the platform in addition to the test profile in our scripts	18:41
*** bpokorny has joined #openstack-infra		18:41
fungi	lifeless: so any time you have to play with it is much appreciated!	18:41
fungi	lifeless: extra context can be found in the script we use to run bindep: http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/scripts/install-distro-packages.sh	18:41
fungi	likely we'd need to improve that for specific additional profiles	18:42
fungi	i mostly am wanting to make sure we're following your vision for how the tool should work and evolve as it was your idea to begin with	18:43
anteaya	channel ops meeting-alt and meeting-3 need a topic refresh	18:43
fungi	anteaya: awesome, on it	18:43
anteaya	thanks	18:43
fungi	done!	18:43
lifeless	fungi: ack thanks; have tabs open for later	18:44
fungi	thanks!	18:45
jroll	AJaeger: https://review.openstack.org/300146 for ironic upper-constraints use	18:45
jroll	if you'd like a look	18:45
anteaya	fungi: thank you	18:46
fungi	you're welcome	18:46
pabelanger	fungi: what was that package again to improve apt-get performance, but at the cost of corruption?	18:46
fungi	pabelanger: there are several options, but eatmydata was one	18:47
pabelanger	Yes, that was it	18:47
pabelanger	vexxhost is pretty slow ATM: https://jenkins03.openstack.org/job/gate-puppet-jenkins-puppet-syntax/29/console	18:47
clarkb	its likely we won't see much speedups with eatmydata	18:47
fungi	i've been pondering whether we should play with it in some of our jobs, though there's the possibility it could have other side effects	18:47
clarkb	I have tested that in the past and I think most/all our clouds already do unsafe disk io for the VMs	18:47
fungi	pabelanger: slow still? mnaser though he fixed it	18:47
clarkb	pabelanger: fungi how old is the VM that is running on?	18:48
openstackgerrit	Merged openstack-infra/project-config: Add tripleo-quickstart project to tripleo team https://review.openstack.org/299932	18:48
pabelanger	fungi: Oh, missed that	18:48
fungi	clarkb: that's what i'm checking	18:48
pabelanger	clarkb: let me check	18:48
clarkb	I suppose its possible vexxhost isn't doing unsafe io and that is why it is slower	18:48
fungi	and getting a uuid for it	18:48
pabelanger	1a050c99-7d4c-4e45-bbbc-da917382ad86	18:49
pabelanger	only 11:12 mins old	18:49
fungi	thanks pabelanger!	18:49
fungi	mnaser: ^	18:49
*** kushal has joined #openstack-infra		18:49
AJaeger	jroll: LGTM.	18:50
jroll	thanks	18:50
mnaser	thanks for the uuid pabelanger	18:51
mnaser	this is strange, ~2% iowait on the host node, ~70% idle cpu on the host node	18:51
pabelanger	mnaser: hack the gibson	18:51
mnaser	pabelanger: how possible is it to not let jenkins kill that machine and get access to it?	18:52
pabelanger	mnaser: fungi can put a hold	18:52
clarkb	pabelanger: you can too	18:52
mnaser	the IO is fine on the actual compute node.	18:53
pabelanger	clarkb: not sure syntax :(	18:53
mnaser	%Cpu(s): 27.3 us, 3.7 sy, 0.0 ni, 66.7 id, 2.1 wa, 0.0 hi, 0.2 si, 0.0 st	18:53
*** cody-somerville has joined #openstack-infra		18:53
clarkb	pabelanger: sudo -H -u nodepool nodepool hold 9258156	18:53
pabelanger	clarkb: done, thanks	18:53
mnaser	now the 2nd request which i think is a bit harder, would it be possible to get access to the instance?	18:53
clarkb	mnaser: yup	18:53
mnaser	i want to do some basic io check (im curious to know if it could be a network issue?)	18:53
pabelanger	just need a public SSH key	18:54
mnaser	sure, 1 secon	18:54
mnaser	pabelanger: http://paste.openstack.org/show/492650/	18:54
pabelanger	mnaser: ssh jenkins@162.253.54.210	18:55
mnaser	merci	18:55
*** tqtran has joined #openstack-infra		18:57
clarkb	pabelanger: then when mnaser is done with the host you can s/hold/delete/ (assuming jenkins is also done with it)	18:57
pabelanger	clarkb: sure	18:57
mnaser	oh got iostat looks awful on that node	18:58
AJaeger	fungi: to make it perfect ;) shouldn't it be "further relies on tox.skip..."?	18:58
fungi	_on_ yes. sigh thanks!	18:59
pleia2	mnaser: btw created re: jenkins css https://storyboard.openstack.org/#!/story/2000534	18:59
pleia2	^^ and everyone else, I added low-hanging-fruit tag, because tags are a thing now!	18:59
AJaeger	pabelanger's centos-7 changes look fine to me - anybody else wants to review some more? https://review.openstack.org/#/q/topic:centos-7	18:59
mnaser	iostat on compute node has w_await of around 3ms, inside the virtual machine around ~150ms	18:59
clarkb	see I am not getting email about that, not surethat the project group subscription is sufficient, will have to play with it more	18:59
clarkb	also tags	19:00
*** jordanP has quit IRC		19:00
clarkb	pleia2: wow thats an old plugin	19:01
pleia2	clarkb: yep	19:01
pleia2	we likely need someone to find us a new one to use or something	19:02
pleia2	it's a very good newcomer task	19:02
fungi	mnaser: that definitely sounds suboptimal by a couple orders of magnitude	19:02
mnaser	yeah.. but I have no idea why. :(	19:02
mnaser	so im investigating to see what's going on	19:02
*** MarkAtwood has quit IRC		19:02
fungi	mnaser: feel free to do anything you like to/in/with that instance. we don't need it for anything, just let us know when you're done so we can delete it	19:02
mnaser	there's no cpu contention on the hypervisor (~75% idle cpu), almost no iowait on the hypervisor, but it's struggling inside the VM only	19:03
clarkb	mnaser: we had io issues with infracloud and SpamapS thought it was related to the kvm disk caching	19:03
*** openstackgerrit has quit IRC		19:03
mnaser	we use file=none so no caching	19:03
*** openstackgerrit has joined #openstack-infra		19:04
*** maishsk has quit IRC		19:04
mnaser	SSDs are fast enough, + the raid controllers we use usally have ~2GB of DDR3 memory write-back cache	19:04
SpamapS	mnaser: no hypervisor<->VFS layer caching, but do you have BBWC?	19:04
mnaser	so overall we've noticed faster performance by taking this path	19:04
SpamapS	that's why	19:04
mnaser	yes	19:04
SpamapS	and SSD, oh yeah	19:04
*** annegentle has quit IRC		19:04
mnaser	SpamapS: compute node has almost 0 iowait and ~2-3ms w_await	19:05
openstackgerrit	Khai Do proposed openstack-infra/puppet-gerrit: Allow puppet to configure Gerrit's gitHttpUrl option https://review.openstack.org/300160	19:05
SpamapS	no reason at all to mess with VFS's idiosyncracies if you have a few GB of BBWC and SSD	19:05
mnaser	the VMs inside are sitting at 150ms w_await ..	19:05
*** annegentle has joined #openstack-infra		19:05
mnaser	40 core not, ~70% idle cpu (across all 40 cores)	19:05
mnaser	oh this is interesting	19:06
mnaser	avgqu-sz is almost stuck at 1	19:06
mnaser	as if it's literally limited at 1	19:06
*** matt-borland has joined #openstack-infra		19:07
*** austin81 has joined #openstack-infra		19:07
*** maishsk has joined #openstack-infra		19:07
*** cdent has joined #openstack-infra		19:10
*** Sukhdev has joined #openstack-infra		19:10
*** yamamoto_ has quit IRC		19:10
mugsie	fungi: how long should it take for jenkins to catch up and remove the traceroutes?	19:13
pleia2	fungi: before brain gets stale, transferring my zuul restart notes into proper docs in the system-config zuul.rst, unless there's a reason I shouldn't	19:13
clarkb	pleia2: go for it, you might want to add a section about clearing out the used nodes in nodepool too, or someone can add that on top of your change	19:13
pleia2	clarkb: right! I will make a quick comment about it, and perhaps someone else can flesh it out more	19:14
anteaya	pleia2: awesome	19:14
*** annegentle has quit IRC		19:14
*** ociuhandu has quit IRC		19:15
openstackgerrit	Austin Clark proposed openstack-infra/project-config: Create stackviz element and script in nodepool (WIP) https://review.openstack.org/279317	19:15
clarkb	email so much email	19:16
fungi	mugsie: that should have taken effect at least 30 minutes ago. are you still seeing failures?	19:17
*** [1]Thelo has joined #openstack-infra		19:17
fungi	pleia2: sounds great	19:17
mugsie	yeah - https://jenkins06.openstack.org/job/gate-designate-dsvm-bind9/622/console	19:17
mugsie	started 15 mins ago	19:17
*** Thelo has quit IRC		19:19
*** [1]Thelo is now known as Thelo		19:19
mugsie	fungi: https://jenkins06.openstack.org/job/gate-shade-dsvm-functional-neutron/262/console started a min ago, it is hitting it as well.	19:19
*** rbrndt has joined #openstack-infra		19:20
*** LaT0rtue has quit IRC		19:21
clarkb	we turned off rax-ord right?	19:21
clarkb	and that was what the traceroute change was going to help debug?	19:22
AJaeger	clarkb: yes	19:22
fungi	clarkb: yes and yes	19:22
pabelanger	Ya, ord is still disabled I believe	19:22
fungi	mugsie: that is disconcerting if me pulling project-config and manually rerunning jenkins-jobs update on our jenkins masters didn't update the job configs	19:22
clarkb	fungi: is rax unabe to do their own tracerouting between networks they own?	19:23
clarkb	or are we just trying to get as much data as possible so that we can point back ath the lcoud?	19:23
fungi	clarkb: apparently. a list of affected uuids means nothing when it comes to them troubleshooting their network	19:23
clarkb	wow ok	19:23
sdague	http://logs.openstack.org/35/299535/7/gate/gate-grenade-dsvm/ade68d4/console.html	19:23
fungi	they asked us if we could collect some traceroutes if we end up deciding to use ord again	19:23
openstackgerrit	Merged openstack-infra/puppet-jenkins: Ubuntu xenial supports openjdk-8-jdk https://review.openstack.org/299948	19:23
sdague	also, apparently traceroute... not in the osic image	19:23
bnemec	tripleo is still hitting the traceroute problem as of a few minutes ago too: http://logs.openstack.org/42/300142/1/check-tripleo/gate-tripleo-ci-f22-nonha/382ada6/console.html	19:23
*** annegentle has joined #openstack-infra		19:24
fungi	sdague: it's not in our dib images period (we upload the same image everywhere)	19:24
mugsie	nearly make me miss hpcloud :)	19:24
sdague	fungi: um, ok, well it's very early used in devstack-gate	19:24
sdague	so... fun hard fail?	19:24
sdague	I just had 3 jobs insta fail on that here - https://review.openstack.org/#/c/299535/7	19:25
sdague	http://logs.openstack.org/35/299535/7/gate/gate-grenade-dsvm-multinode/5ea58a3/console.html	19:25
sdague	http://logs.openstack.org/35/299535/7/gate/gate-tempest-dsvm-full/a95f209/console.html	19:25
sdague	http://logs.openstack.org/35/299535/7/gate/gate-grenade-dsvm/ade68d4/console.html	19:25
pabelanger	fungi: do you have a check up to add traceroute into dib elements?	19:25
fungi	sdague: yes, we reverted it a while ago and i personally applied the reverted configuration on all our jenkins masters. i'm trying to figure out why jjb updates aren't apparently effective	19:25
fungi	pabelanger: no, i haven't decided yet if that should be dib or puppet. i'd like to get the original fallout behind us before i even think about trying again	19:26
pabelanger	fungi: ack	19:26
fungi	what gives? it's definitely still in that job config on jenkins06	19:26
*** jamielennox\|away is now known as jamielennox		19:26
fungi	i a good minute after it merged, i did `cd /etc/project-config ; sudo git pull --ff-only`	19:27
fungi	and then `time sudo -H jenkins-jobs update --delete-old /etc/jenkins_jobs/config`	19:27
clarkb	fungi: jjb uses /etc/jenkins_jobs	19:27
fungi	and it completed, with no errors	19:27
clarkb	so that pull is not sufficient	19:27
fungi	ugh	19:27
clarkb	need puppet to run first	19:28
fungi	what am i missing that normally moves the files into /etc/jenkins_jobs?	19:28
*** dtardivel has quit IRC		19:28
clarkb	puppet	19:28
fungi	i guess it's puppeted by now. i'll double-check that	19:28
fungi	seems to be	19:30
*** sileht has quit IRC		19:30
fungi	updating everywhere _again_	19:30
pabelanger	clarkb: okay, made it passed know error, but apparently openjdk is hard coded in puppet-jenkins, fixing	19:30
*** apoorvad has quit IRC		19:30
clarkb	pabelanger: meaning https://review.openstack.org/#/c/299948/2 was insufficient?	19:31
pabelanger	clarkb: right, I didn't see the openjdk-7 stuff below	19:31
pabelanger	and gate didn't catch it because of missing ubuntu-xenial image	19:31
*** piet has quit IRC		19:32
*** vgridnev has quit IRC		19:32
*** nwkarsten has quit IRC		19:32
*** piet has joined #openstack-infra		19:33
*** ociuhandu has joined #openstack-infra		19:37
*** maishsk has quit IRC		19:39
*** krtaylor has quit IRC		19:40
*** binbincong has quit IRC		19:41
openstackgerrit	Paul Belanger proposed openstack-infra/puppet-jenkins: Don't update-java-alternatives on ubuntu-xenial https://review.openstack.org/300186	19:41
pabelanger	clarkb: okay, going to test that now ^	19:41
*** binbincong has joined #openstack-infra		19:41
smarcet	fungi: i ran several load tests against dev openstackid and now handles 2k concurrent connections :)	19:42
fungi	smarcet: that's excellent! well double of what you were shooting for	19:42
pabelanger	clarkb: that could can likely be removed in the future, as it looks to only be added because of bare-* nodes with had openjdk 6 installed	19:42
smarcet	fungi: however , jpmaxman and jimmy wants to wait till apple approves the mobile app	19:42
smarcet	fungi: and then move to production	19:43
*** ociuhandu has quit IRC		19:43
clarkb	pabelanger: yup I think so	19:43
fungi	smarcet: just let me know and i'm happy to work on upgrading the production server whenever you're all redy	19:43
*** sdake_ has joined #openstack-infra		19:43
fungi	ready	19:43
clarkb	pabelanger: once bare-precise is gone?	19:43
pabelanger	clarkb: Yup, keep forgetting about that little guy	19:44
matt-borland	doug-fish, I wanted to solve the OPENSTACK_IMAGE_BACKEND issue you pointed out in Image Edit, but had a problem.	19:44
*** sdake has quit IRC		19:44
smarcet	fungi: sure, perhaps we could do more improvements on the meantime to puppet script, i would like to put a watch dog for redis like ps-watcher or something	19:44
pabelanger	okay, new build going	19:44
*** ociuhandu has joined #openstack-infra		19:44
pabelanger	will test with local nodepool too	19:44
* matt-borland was in the wrong room :(		19:44
smarcet	fungi: and try to tweak a little redis config too	19:44
openstackgerrit	Khai Do proposed openstack-infra/system-config: Change download command to point to git.o.o https://review.openstack.org/300187	19:45
fungi	smarcet: okay, though definitely want to be able to reenable puppet on production soon, which at this point we'll only be able to do when we can replace it with a 14.04 server	19:45
*** mrmartin has joined #openstack-infra		19:45
smarcet	fungi: i think that next week we would have some news from apple	19:46
smarcet	and then we could move forward on that	19:46
*** dizquierdo has quit IRC		19:46
*** Swanson has joined #openstack-infra		19:46
fungi	smarcet: okay, sounds good	19:46
fungi	thanks!	19:46
*** mrmartin has quit IRC		19:47
*** dizquierdo has joined #openstack-infra		19:48
fungi	mugsie: sdague: the couple of jenkins masters which aren't done updating with the revert have been set to not start any new jobs until they're fixed, so should be safe to recheck things now	19:50
*** SumitNaiksatam has joined #openstack-infra		19:50
pabelanger	mnaser: let me know once you are done with the slave we held	19:50
mnaser	pabelanger: sorry, will do, i'm at loss at what's going on	19:51
*** tonytan4ever has quit IRC		19:52
fungi	okay, jenkins01 and jenkins02 updates have completed so i've taken them back out of prepare for shutdown	19:53
fungi	#status notice Any jobs which erroneously failed on missing traceroute packages should be safe to recheck now	19:53
openstackstatus	fungi: sending notice	19:53
pabelanger	mnaser: no rush, just a reminder	19:54
*** nwkarsten has joined #openstack-infra		19:54
*** nwkarsten has quit IRC		19:54
-openstackstatus- NOTICE: Any jobs which erroneously failed on missing traceroute packages should be safe to recheck now		19:54
*** ihrachys has joined #openstack-infra		19:54
mugsie	fungi: thanks!	19:55
*** nwkarsten has joined #openstack-infra		19:55
fungi	mugsie: no thanks necessary. just cleaning up the mess i made for 3k developers	19:55
mugsie	it happens :)	19:56
openstackstatus	fungi: finished sending notice	19:56
*** sdake_ has quit IRC		19:56
*** dizquierdo has quit IRC		19:57
*** briancline has quit IRC		19:58
*** mylu has joined #openstack-infra		19:58
*** briancline has joined #openstack-infra		19:58
*** Cormite has joined #openstack-infra		20:00
*** tphummel has quit IRC		20:00
*** esikachev has joined #openstack-infra		20:01
pabelanger	clarkb: actually, I have no idea how to get diskimage-builder to pick up that puppet-jenkins commit. We don't have code exposed to change puppet modules	20:01
pabelanger	unless I quickly hack modules.env to use that sha1	20:02
clarkb	pabelanger: yes I think you have to hack modules.env	20:02
*** mrmartin has joined #openstack-infra		20:02
clarkb	and there is code in our dib elements to use a specific system config ref	20:02
*** MarkAtwood has joined #openstack-infra		20:02
pabelanger	yup	20:03
openstackgerrit	Andrew Woodward proposed openstack-infra/project-config: Move puppet-ceph under puppet-openstack ACL https://review.openstack.org/300192	20:04
openstackgerrit	Paul Belanger proposed openstack-infra/system-config: DNM - testing ubuntu-xenial https://review.openstack.org/300193	20:04
*** esikachev has quit IRC		20:06
*** amrith is now known as _amrith_		20:06
*** salv-orlando has joined #openstack-infra		20:08
*** Sukhdev has quit IRC		20:09
*** briancline has quit IRC		20:10
*** yamamoto_ has joined #openstack-infra		20:11
openstackgerrit	Jeremy Stanley proposed openstack-infra/system-config: Always apt-get update after replacing sources https://review.openstack.org/300005	20:12
*** sridhar_ram1 has joined #openstack-infra		20:14
*** SumitNaiksatam has quit IRC		20:15
*** sridhar_ram has quit IRC		20:15
*** baoli has joined #openstack-infra		20:15
*** sileht has joined #openstack-infra		20:15
*** baoli_ has joined #openstack-infra		20:15
*** dprince has quit IRC		20:16
openstackgerrit	Doug Wiegley proposed openstack-infra/devstack-gate: Fail the test if the pre test hook fails https://review.openstack.org/292175	20:17
*** briancline has joined #openstack-infra		20:17
*** SumitNaiksatam has joined #openstack-infra		20:17
mnaser	pabelanger can i shutdown and start the instance or will that likely destroy it?	20:18
*** kgiusti has left #openstack-infra		20:18
clarkb	mnaser: that should be fine	20:19
mnaser	alright, thanks!	20:19
*** baoli has quit IRC		20:19
*** yamamoto_ has quit IRC		20:20
*** abregman is now known as abregman\|afk		20:20
fungi	yeah, reboots work fine in this situation	20:22
fungi	we've basically told nodepool hands-off with that instance until we say otherwise	20:22
*** piet has quit IRC		20:23
*** tphummel has joined #openstack-infra		20:23
mnaser	fungi: the instance can be killed, we've found an oddity which is worked around.	20:23
*** piet has joined #openstack-infra		20:23
fungi	mnaser: oh? great!	20:23
mnaser	if it happens again (it could, on another compute node, but not as likely), we'll check	20:23
fungi	i'll let pabelanger handle switching it to delete at his convenience	20:23
mnaser	but if we get another slowdown that spawns on another compute node, then we know that it was the reason for the problem	20:24
fungi	mnaser: thanks for investigating--well keep an eye out	20:24
mnaser	whats the best way to keep an eye on this?	20:24
fungi	mnaser: unfortunately we mostly find out about it when someone notices a job run too long and/or hit a configured timeout. which can sometimes be for legitimate reasons too. we don't directly track it	20:24
clarkb	you can query logstash for timed out jobs in vexxhost	20:25
clarkb	let me get a query together for that	20:25
mnaser	that would be useful keep up	20:25
fungi	yep, that would yield _likely_ (but not guaranteed) recurrences	20:25
pabelanger	mnaser: fungi okay, deleted	20:25
fungi	would still need manual vetting to see if they're the general everything-is-slow symptom, or more like an indefinite hang on some operation	20:26
*** briancline has quit IRC		20:26
fungi	the former is what we'd been witnessing sometimes in vexxhost recently	20:26
fungi	teh latter can be due to a variety of external causes	20:26
*** fawadkhaliq has quit IRC		20:27
mnaser	yep, that makes sense	20:27
clarkb	mnaser: message:"Build timed out" AND node_provider:"ymq-1" AND filename:"console.html"	20:27
*** fawadkhaliq has joined #openstack-infra		20:27
clarkb	mnaser: you can query that on logstash.openstack.org using kibana or hit the elasticsearch api directly at http://logstash.openstack.org/elasticsearch as the api root	20:27
mnaser	build_node in there is super useful	20:28
*** krtaylor has joined #openstack-infra		20:28
*** ihrachys has quit IRC		20:28
clarkb	ya that will give you a specific instance which we can track back via nodepool/nova	20:28
*** apoorvad has joined #openstack-infra		20:28
mnaser	so we can just query the db for the uuid (but build_node_uuid would be nice too :D)	20:28
*** sridhar_ram1 has quit IRC		20:29
clarkb	mnaser: unfortunately the instance uuids are not exposed to anything but nodepool so hard to pass that along automatically	20:29
clarkb	at least I think it isn't passed along anywhere	20:29
fungi	right, the nova instance uuid unfortunately isn't something passed through at the job layer	20:29
mnaser	fungi gr	20:29
mnaser	actually more a grrr towards us	20:30
fungi	i end up tracking it back in nodepool's debug log	20:30
mnaser	the build timed out in logstash for the last 7 days	20:30
mnaser	i think it will be quite easy to see when its fied	20:30
clarkb	mnaser: and in theory ouy could us that info to see if there were other affected hypervisors	20:30
*** piet has quit IRC		20:31
mnaser	yeah the name is fine, we can get the uuid easily after	20:31
fungi	yeah, at least those build_node names are the corresponding nova instance names\	20:31
fungi	so it's something to key from	20:31
mnaser	unless you randomize the instance name before deleting it #nextgensecurity	20:31
fungi	heh	20:31
*** bpokorny has quit IRC		20:32
*** stevemar has quit IRC		20:33
clarkb	grun timeout of 0 apepars to have made it into our images ok	20:33
clarkb	*grub	20:33
*** stevemar has joined #openstack-infra		20:33
clarkb	so everything should be 10 seconds faster	20:34
clarkb	in aggregate thats pretty awesome	20:34
*** bpokorny has joined #openstack-infra		20:34
*** rossella_s has quit IRC		20:34
*** rossella_s has joined #openstack-infra		20:35
*** nwkarsten has quit IRC		20:35
*** kzaitsev_mb has joined #openstack-infra		20:36
*** moravec has quit IRC		20:38
pabelanger	okay, how do we checkout a review.o.o ref with modules.env?	20:38
clarkb	pabelanger: it should take any name it can fetch/checkout. I now realize the hard part is the fetch	20:39
pabelanger	Ya, not sure how to do what with the existing code	20:40
clarkb	I think the "easy" solution is put it on not gerrit, then make a branch or tag for it	20:40
clarkb	and point modules.env at that	20:40
pabelanger	ya, will have to do that	20:40
clarkb	can be local to your build host or on github or whatever	20:41
*** asettle has joined #openstack-infra		20:42
*** fawadkhaliq has quit IRC		20:42
openstackgerrit	Paul Belanger proposed openstack-infra/system-config: DNM - testing ubuntu-xenial https://review.openstack.org/300193	20:43
*** ayoung has quit IRC		20:43
*** briancline has joined #openstack-infra		20:43
pabelanger	clarkb: okay, testing again.	20:44
pabelanger	would be nice to have zuul-clone support for our elements :)	20:44
clarkb	pabelanger: one of the reasons its designed this way is to minimize bootstrap deps	20:45
clarkb	pabelanger: you need bash and git	20:45
*** fawadkhaliq has joined #openstack-infra		20:45
pabelanger	true	20:46
pabelanger	however, I think zuul is installed before we run puppet	20:46
clarkb	pabelanger: it shouldn't be	20:46
clarkb	and its not installed on any of our real servers	20:46
clarkb	except for zuul and zuul mergers	20:46
ianw	clarkb: i'm pretty convinced that centos-7 runs in the same amount of time as the image-based builds, so we can swap out the devstack jobs. this is just based on min() of runs, because variability is quite high	20:47
clarkb	part of that was when I attempted to use librarian and broke everything due to its crazy idea of deps	20:47
pabelanger	clarkb: Ah, it was pip I seen	20:47
clarkb	ianw: ya	20:47
fungi	clarkb: pabelanger: well, with dib we install it into /usr/zuul-env so that we have access to zuul-cloner on our images	20:47
clarkb	fungi: ya but our puppet runs without dib	20:48
pabelanger	fungi: Ya. I think it is done after puppet, but ya	20:48
clarkb	basically its doable	20:48
fungi	anyway, i've broken enough things for one day, so am going to buzz off and find early dinner	20:48
ianw	we know something about everything failing with "traceroute not found" right?	20:48
clarkb	we just need to be careful that it works for more than slave builds	20:48
clarkb	ianw: yes	20:48
fungi	ianw: it shouldn't be any longer	20:48
fungi	is it still?	20:48
*** smarcet has quit IRC		20:48
*** \|-paul-\| has joined #openstack-infra		20:48
pabelanger	ianw: clarkb: here are the current reviews open to move to centos-7: https://review.openstack.org/#/q/status:open+project:openstack-infra/project-config+branch:master+topic:centos-7	20:48
pabelanger	leaving puppet until the last	20:48
pabelanger	likely do kolla in the next day	20:49
ianw	fungi: probably not, doing rechecks. sorry, just coming online here	20:49
fungi	jobs should have been completely updated with the revert by about 19:50 utc	20:49
fungi	so for about an hour	20:49
fungi	back later	20:49
*** rcarrillocruz has left #openstack-infra		20:50
*** rcarrillocruz has joined #openstack-infra		20:50
*** jsavak has quit IRC		20:53
mgagne	could someone review those changes? They are changes to Gerrit UI https://review.openstack.org/#/q/status:open+project:openstack-infra/system-config+branch:master+topic:gerrit-ui	20:53
mgagne	I've been using them with tampermonkey for a couple of months without problem	20:54
*** IlyaG has quit IRC		20:54
ianw	mgagne: maybe drop a screenshot on imgur, might make it easier for reviewers to visualise?	20:55
mgagne	there are some already	20:55
*** rfolco has quit IRC		20:56
mgagne	except for max-width, can add this one	20:56
*** stevemar_ has joined #openstack-infra		20:56
*** knikolla has quit IRC		20:56
*** stevemar has quit IRC		20:56
ianw	heh, that was the one i looked at :)	21:00
*** edmondsw has quit IRC		21:00
mgagne	done	21:00
*** mylu has quit IRC		21:00
*** asettle has quit IRC		21:01
*** asettle has joined #openstack-infra		21:01
*** sputnik13_ has quit IRC		21:01
clarkb	mnaser: http://logs.openstack.org/80/299980/1/gate/gate-puppet-openstack-integration-3-scenario003-tempest-ubuntu-trusty/15db1b1/console.html is possibly another slow node that happened ~12 minutes ago	21:01
*** aysyd has quit IRC		21:02
*** sputnik13 has joined #openstack-infra		21:03
openstackgerrit	David Moreau Simard proposed openstack-infra/project-config: Ensure VMs used in Packstack jobs have consistent swap configuration https://review.openstack.org/300123	21:04
*** Goneri has quit IRC		21:04
*** Cormite has quit IRC		21:07
*** thorst has quit IRC		21:07
*** trown is now known as trown\|outtypewww		21:07
*** thorst has joined #openstack-infra		21:08
*** cdent has quit IRC		21:08
openstackgerrit	Ricardo Carrillo Cruz proposed openstack-infra/shade: Follow name_or_id pattern on domain operations https://review.openstack.org/288715	21:10
*** mylu has joined #openstack-infra		21:12
*** abregman\|afk has quit IRC		21:12
*** thorst has quit IRC		21:12
*** thorst has joined #openstack-infra		21:14
*** piet has joined #openstack-infra		21:14
*** ldnunes has quit IRC		21:15
rcarrillocruz	mordred, fungi, clarkb, jeblair_: i'm off a couple days, be back on Sunday. Let me know what you find on the ansible launch role. I'll continue adding some enhancements I have in mind on Monday and potentially prepare another blogpost with an example of a deploy with similar config to what you folks do in the various providers	21:15
anteaya	rcarrillocruz: have fun	21:16
rcarrillocruz	thx anteaya :-)	21:16
anteaya	welcome :)	21:16
*** aeng has joined #openstack-infra		21:16
*** baoli_ has quit IRC		21:17
*** Sukhdev has joined #openstack-infra		21:18
*** mrmartin has quit IRC		21:18
*** thiagop-afk has quit IRC		21:18
*** piet has quit IRC		21:18
*** eharney has quit IRC		21:18
*** thorst has quit IRC		21:19
*** achanda has quit IRC		21:22
*** mylu has quit IRC		21:23
*** baoli has joined #openstack-infra		21:24
openstackgerrit	Merged openstack-infra/project-config: Switch ansible-role-functional-jobs-centos7 to centos-7 https://review.openstack.org/300104	21:26
openstackgerrit	Ian Wienand proposed openstack-infra/project-config: Switch devstack job to centos-minimal https://review.openstack.org/300220	21:26
*** matt-borland has quit IRC		21:27
ianw	pabelanger: ^ maybe we should do that before switching other dsvm jobs	21:27
*** mylu has joined #openstack-infra		21:27
ianw	pabelanger: also, did you skip the anvil jobs, or just not got to yet?	21:27
Swanson	So. traceroute stuff still broken?	21:28
*** kzaitsev_mb has quit IRC		21:28
ianw	harlowja: ^	21:28
*** knikolla has joined #openstack-infra		21:28
ianw	Swanson: should not be	21:28
*** amitgandhinz has quit IRC		21:29
harlowja	whats up	21:29
harlowja	u guys can switch over the anvil jobs	21:29
harlowja	i need to work on that stuff sometime soon anyway	21:29
*** sdake has joined #openstack-infra		21:30
Swanson	I just rechecked a patch and got traceroute6 and traceroute not found.	21:31
*** imcsk8 is now known as imcsk8\|afk		21:31
ianw	Swanson: yeah, me too. i wonder if jjb is a bit stuck updating things	21:31
Swanson	ianw, so long as people know it is still broken (and I saw the message saying it wasn't an hour or two ago so maybe they don't) I'm happy.	21:32
ianw	harlowja: cool, yeah i don't think it will affect them. you can quote me on that :)	21:32
harlowja	ianw all good :)	21:32
*** asselin_ has quit IRC		21:32
*** EricGonc_ has joined #openstack-infra		21:32
openstackgerrit	Merged openstack-infra/project-config: Migrate chef jobs to centos-7 https://review.openstack.org/300114	21:33
openstackgerrit	Elizabeth K. Joseph proposed openstack-infra/system-config: Add Zuul restart documentation https://review.openstack.org/300223	21:34
*** esker has joined #openstack-infra		21:34
pabelanger	ianw: WFM	21:35
pabelanger	ianw: Ya, haven't done it yet	21:35
*** mriedem has quit IRC		21:35
*** nwkarsten has joined #openstack-infra		21:36
*** esker has quit IRC		21:36
openstackgerrit	Ricardo Carrillo Cruz proposed openstack-infra/shade: Follow name_or_id pattern on domain operations https://review.openstack.org/288715	21:36
*** nwkarsten has quit IRC		21:36
*** EricGonczer_ has quit IRC		21:36
*** esker has joined #openstack-infra		21:37
*** dasm has joined #openstack-infra		21:38
*** fawadkhaliq has quit IRC		21:38
*** imcsk8\|afk is now known as imcsk8		21:39
*** fawadkhaliq has joined #openstack-infra		21:39
*** _ari_ has quit IRC		21:40
*** fawadkhaliq has quit IRC		21:40
*** fawadkhaliq has joined #openstack-infra		21:41
*** _ari_ has joined #openstack-infra		21:42
openstackgerrit	Paul Belanger proposed openstack-infra/system-config: DNM - testing ubuntu-xenial https://review.openstack.org/300193	21:42
*** fawadkhaliq has quit IRC		21:44
*** fawadkhaliq has joined #openstack-infra		21:44
*** _ari_ has quit IRC		21:44
*** asselin_ has joined #openstack-infra		21:45
*** ybathia has joined #openstack-infra		21:45
*** gyee has quit IRC		21:46
*** tojuvone_ has joined #openstack-infra		21:47
*** angdraug has joined #openstack-infra		21:48
*** gyee has joined #openstack-infra		21:48
*** \|-paul-\| has quit IRC		21:49
*** burgerk has quit IRC		21:49
*** tojuvone has quit IRC		21:51
openstackgerrit	Paul Belanger proposed openstack-infra/puppet-jenkins: Don't update-java-alternatives on ubuntu-xenial https://review.openstack.org/300186	21:53
*** dasm has left #openstack-infra		21:53
pabelanger	clarkb: crinkle: I think that might be the better approach ^	21:54
*** denisra has joined #openstack-infra		21:55
*** mylu has quit IRC		21:56
*** sigmavirus24 is now known as sigmavirus24_awa		21:56
*** krtaylor has quit IRC		21:57
crinkle	pabelanger: i liked the first approach better, i don't get the new one	21:57
*** yolanda has quit IRC		21:59
clarkb	pabelanger: ya the new one will still try to update alternative on xenial	21:59
*** xyang1 has quit IRC		22:01
*** amotoki has joined #openstack-infra		22:01
*** nmagnezi has quit IRC		22:02
rcarrillocruz	hmm	22:06
rcarrillocruz	folks	22:06
rcarrillocruz	you seeing tracerout6 missing package errors on tests	22:06
rcarrillocruz	?	22:06
rcarrillocruz	https://jenkins07.openstack.org/job/gate-shade-dsvm-functional-nova/276/console	22:06
clarkb	rcarrillocruz: yes was already removed	22:07
*** yolanda has joined #openstack-infra		22:07
clarkb	basically rax asked us for traceroute data so we added it to the jobs but minimal images don't have traceroute	22:07
clarkb	things went boom so we removed it from the jobs	22:07
ianw	i wonder if it's jenkins07 ... that's where i saw a recent failure	22:07
ianw	https://jenkins07.openstack.org/job/gate-tempest-dsvm-neutron-full/26238/console	22:07
rcarrillocruz	so recheck ?	22:08
clarkb	rcarrillocruz: yes	22:08
rcarrillocruz	cool thx	22:08
clarkb	ianw: I think fungi had to redo the cleanup	22:09
Swanson	I did a recheck after it was claimed to be fixed and it failed again.	22:09
*** fbo_ has quit IRC		22:09
clarkb	Swanson: after it was claimed to be fixed the first time or the second time?	22:09
* clarkb logs in to see the config of that job		22:10
Swanson	clarkb, 2016-03-31 21:24:44.059 \| /tmp/hudson6254995475921326830.sh: 23: /tmp/hudson6254995475921326830.sh: traceroute6: not found	22:10
*** Goneri has joined #openstack-infra		22:10
Swanson	If that helps	22:10
*** icey has quit IRC		22:10
*** fbo has joined #openstack-infra		22:10
clarkb	huh its still there I guess fungi didnt manage to get it reconfigured on 07	22:11
clarkb	I will run that now	22:11
*** resker has joined #openstack-infra		22:11
ianw	yeah, everything i'm seeing fail in devstack queue is jenkins07	22:12
*** resker has quit IRC		22:12
clarkb	jjb is running there now	22:12
clarkb	I will also put it in shutdown mode	22:12
*** esker has quit IRC		22:13
*** icey has joined #openstack-infra		22:13
clarkb	it will stop running new jobs now	22:13
Swanson	recheck then?	22:13
*** esker has joined #openstack-infra		22:13
*** nwkarsten has joined #openstack-infra		22:13
*** nwkarsten has quit IRC		22:14
mnaser	does anyone know if ubuntu vms will prefer ipv6 over ipv4 (or at least the build machines?)	22:14
clarkb	yes now a recheck should be fine	22:14
clarkb	mnaser: yes they should	22:14
mnaser	hmm	22:14
mnaser	im trying to find other builds of the same type	22:14
*** nwkarsten has joined #openstack-infra		22:14
mnaser	and how long the git clone process has taken	22:14
clarkb	mnaser: the software running in them at least	22:14
clarkb	yes git definitely will	22:14
pabelanger	clarkb: crinkle: The issue is, I think we need a conditional anchor, not to be present when we do xenial... otherwise, the update-alt will be called	22:14
mnaser	i think there has been issues with ipv6 at rackspace?	22:15
clarkb	pabelanger: won't it get mad the anchor doesn't exist?	22:15
mnaser	so if it tries to contact git.openstack.org via ipv6	22:15
craige	o/	22:15
pabelanger	clarkb: that is what I am testing, but I can wrap it with and if statement again too	22:15
*** thorst has joined #openstack-infra		22:16
openstackgerrit	Paul Belanger proposed openstack-infra/puppet-jenkins: Don't update-java-alternatives on ubuntu-xenial https://review.openstack.org/300186	22:17
*** tpsilva has quit IRC		22:17
*** erlon has quit IRC		22:17
clarkb	07 claims no jobs need updating so its got corrupt cache, we can fix that	22:18
*** annegentle has quit IRC		22:18
Swanson	See you in hell, 07! Thanks for looking into this!	22:19
*** annegentle has joined #openstack-infra		22:20
rcarrillocruz	++	22:20
crinkle	pabelanger: i don't really think the anchor matters, the update-java-alternatives exec gets called unconditionally, it just happens after the anchor	22:21
crinkle	it's requiring the anchor, not subscribing to it	22:21
ianw	no jenkins07, i expect you to die!	22:25
*** andymaier has quit IRC		22:25
*** pasquier-s has quit IRC		22:26
*** sridhar_ram has joined #openstack-infra		22:27
*** bpokorny_ has joined #openstack-infra		22:28
*** bpokorny_ has quit IRC		22:30
*** bpokorny_ has joined #openstack-infra		22:30
*** dimtruck is now known as zz_dimtruck		22:31
*** bpokorny has quit IRC		22:32
openstackgerrit	Merged openstack-infra/irc-meetings: Adding AsiaPac Regional PWG Meeting https://review.openstack.org/299776	22:33
*** regXboi has quit IRC		22:33
pabelanger	crinkle: Ya, still running a test but for what ever reason, it was still calling the update-java exec, regardless of my conditional logic. Will report back in 10mins	22:35
*** tphummel has quit IRC		22:36
*** esker has quit IRC		22:36
openstackgerrit	Paul Belanger proposed openstack-infra/puppet-jenkins: Don't update-java-alternatives on ubuntu-xenial https://review.openstack.org/300186	22:37
clarkb	we are up to gate-hacking now	22:37
clarkb	(it is alphabetical	22:37
*** thorst has quit IRC		22:37
*** tphummel has joined #openstack-infra		22:38
*** baoli has quit IRC		22:38
pabelanger	Ya, this openjdk is frustrating.... must be on 12th DIB build	22:38
*** rossella_s has quit IRC		22:38
*** ybathia has quit IRC		22:38
clarkb	just thinking out loud, the if else around the anchor shouldn't be necessary	22:40
clarkb	its just that the anchor is only used on the exec side if on older ubuntu	22:41
*** annegentle has quit IRC		22:41
*** annegentle has joined #openstack-infra		22:42
pabelanger	clarkb: okay, testing again	22:44
*** bpokorny_ has quit IRC		22:44
ianw	pabelanger: now you see why f23 took so long :)	22:44
pabelanger	ianw: no doubt	22:44
*** bpokorny has joined #openstack-infra		22:44
openstackgerrit	Paul Belanger proposed openstack-infra/puppet-jenkins: Don't update-java-alternatives on ubuntu-xenial https://review.openstack.org/300186	22:44
pabelanger	I think the issue is the before => Anchor syntax	22:45
*** ayoung has joined #openstack-infra		22:45
pabelanger	how can that be true, if it is conditional	22:45
clarkb	an anchor is just a noop reasource to affect ordering	22:46
clarkb	so before anchor should be fine	22:46
*** nwkarste_ has joined #openstack-infra		22:47
openstackgerrit	Thomas Goirand proposed openstack-infra/project-config: Add a job to build openstack-pkg-tools https://review.openstack.org/294022	22:47
*** nwkarste_ has quit IRC		22:47
zigo	mordred: Around? I still didn't see you reviewing this patch, and it's been opened for 2 weeks.	22:48
zigo	mordred: I wonder if you really want to do something with this PTL hat that you've asked for...	22:49
*** nwkarsten has quit IRC		22:49
*** sdague has quit IRC		22:50
*** akshai has quit IRC		22:51
*** tsg has joined #openstack-infra		22:51
*** annegentle has quit IRC		22:53
*** sdake has quit IRC		22:57
crinkle	clarkb: http://logs.openstack.org/31/300131/2/check/gate-tempest-dsvm-full/379c52e/console.html failed on the traceroute thing just a few minutes ago, not sure if this was jenkins07 or a different issue?	22:57
clarkb	if it started before I put jenkins07 in shutdown mode it could be that one	22:58
clarkb	but if it failed a few minutes ago then its like a different server :(	22:58
clarkb	I will look for it	22:58
crinkle	2016-03-31 21:43:34.302 \| Started by user anonymous	22:58
openstackgerrit	Paul Belanger proposed openstack-infra/system-config: DNM - testing ubuntu-xenial https://review.openstack.org/300193	22:59
clarkb	ya you have to look at each of hte masters because we don't record the master in the job but maybe we can/should	22:59
clarkb	crinkle: https://jenkins07.openstack.org/job/gate-tempest-dsvm-full/31842/console it was on 07	23:00
crinkle	clarkb: okay, so i'll recheck	23:01
clarkb	yup should be fine, I haven't seen any recent fails for that specific job on the other masters	23:01
*** akshai has joined #openstack-infra		23:02
*** austin81 has left #openstack-infra		23:03
*** akshai_ has joined #openstack-infra		23:03
*** ams_ has quit IRC		23:03
*** sputnik13 has quit IRC		23:05
*** akshai has quit IRC		23:06
pabelanger	finally	23:08
pabelanger	okay, https://review.openstack.org/#/c/300186/ appears to work if I manually run it	23:09
clarkb	pabelanger: got it to work?	23:09
pabelanger	just waiting for DIB to finish	23:09
*** sdake has joined #openstack-infra		23:09
pabelanger	my issue was a typo in my modules.env file and becuase install_modules.sh doesn't have set -e it was not bombing on error	23:09
clarkb	it has my +2	23:09
clarkb	07 update is into the periodic jobs	23:09
pabelanger	let me confirm with this nodepool dib first	23:09
pabelanger	getting a drink first	23:10
clarkb	pabelanger: ok, we should get that merged if the rest of the build is happy though	23:10
clarkb	so that we stop looping around and failing	23:10
*** knikolla has quit IRC		23:10
*** annegentle has joined #openstack-infra		23:10
pabelanger	clarkb: Yup, in about 5mins here	23:11
*** salv-orlando has quit IRC		23:11
*** rbrndt has quit IRC		23:11
openstackgerrit	Jay Pipes proposed openstack/gertty: py3k: Prevent stacktrace opening links in browser https://review.openstack.org/300247	23:12
*** sdake_ has joined #openstack-infra		23:12
pabelanger	ianw: my only concern about 300126, if we remove install-distro-packages, it will almost be an empty slave.	23:13
*** sdake has quit IRC		23:13
*** amotoki has quit IRC		23:13
pabelanger	ianw: but I can remove it	23:13
pabelanger	clarkb: okay, lets merge it. This dib is slow	23:14
ianw	pabelanger: i don't think it matters much, but i think the dependencies are all in the job	23:14
clarkb	pabelanger: feel free to self approve you have my +2	23:15
pabelanger	clarkb: ok	23:15
pabelanger	ianw: okay, removingt	23:15
clarkb	I am going to approve ianw's centos-7 dsvm job switch as soon as 07 is done updating its jobs	23:15
openstackgerrit	Paul Belanger proposed openstack-infra/project-config: Move periodic-package-afs-centos7 to centos-7 https://review.openstack.org/300126	23:15
pabelanger	clarkb: ack	23:15
pabelanger	we should be able to do puppet jobs over the weekend	23:16
pabelanger	since it will be a big JJB update	23:16
pabelanger	everything else we can finish off tomorrow	23:16
clarkb	07 is done udpating so I have put it back into the rotation	23:19
clarkb	I have also approved the centos-7 dsvm change	23:19
*** ybathia has joined #openstack-infra		23:19
*** nwkarsten has joined #openstack-infra		23:19
pabelanger	we can likely do https://review.openstack.org/#/c/300119/ and https://review.openstack.org/#/c/300117/ too for centos-7	23:22
* clarkb looks		23:23
clarkb	I mostly want to be around to field problmes if we break solum or nova-docker	23:24
* pabelanger nods		23:24
*** nwkarsten has quit IRC		23:24
clarkb	I figure ianw will be around to debug his change	23:24
clarkb	etc	23:24
*** ayoung has quit IRC		23:26
*** annegentle has quit IRC		23:26
*** pradk has quit IRC		23:26
clarkb	pabelanger: you going to approve your puppet change for xenial?	23:27
pabelanger	clarkb: ya, I can do that now	23:27
*** aeng has quit IRC		23:27
*** ybathia has quit IRC		23:27
ianw	yeah, we should do dsvm stuff in regular hours	23:27
openstackgerrit	Merged openstack-infra/project-config: Switch devstack job to centos-minimal https://review.openstack.org/300220	23:28
*** niska has quit IRC		23:30
lifeless	fungi: ugh, current release of bindep has clamped pbr dep :(	23:31
lifeless	Collecting pbr!=0.7,<1.0,>=0.6 (from bindep)	23:31
mgagne	I should have started using shade a long time ago :O	23:31
*** gordc has quit IRC		23:32
*** tosky has quit IRC		23:32
*** sridhar_ram has quit IRC		23:34
*** tphummel has quit IRC		23:34
*** tphummel_ has joined #openstack-infra		23:35
*** niska has joined #openstack-infra		23:35
openstackgerrit	Merged openstack-infra/puppet-jenkins: Don't update-java-alternatives on ubuntu-xenial https://review.openstack.org/300186	23:35
*** mriedem has joined #openstack-infra		23:36
*** thorst has joined #openstack-infra		23:38
*** bpokorny_ has joined #openstack-infra		23:41
*** bpokorny has quit IRC		23:41
*** bpokorny_ has quit IRC		23:41
*** bpokorny has joined #openstack-infra		23:42
*** sdake_ has quit IRC		23:43
lifeless	fungi: AJaeger: feedback on those changes given	23:43
*** aeng has joined #openstack-infra		23:43
*** thorst has quit IRC		23:46
pabelanger	2016-03-31 23:50:39,232 INFO nodepool.image.build.ubuntu-xenial: Notice: Finished catalog run in 161.85 seconds	23:50
pabelanger	\o/	23:51
pleia2	pabelanger: exciting!	23:51
pabelanger	pleia2: Indeed! Also, I would like to file my complaint with ubuntu for removing openkjdk-7 after I did all my work to make it work with our manifests	23:52
pleia2	complaint filed ;)	23:53
pabelanger	:D	23:53
*** knikolla has joined #openstack-infra		23:53
openstackgerrit	Merged openstack/diskimage-builder: ironic-python-agent should use console output https://review.openstack.org/295501	23:54
pleia2	it's in main, so I suspect they didn't want to drag support for it along in another LTS (I don't blame them)	23:54
* pabelanger nods		23:55
pabelanger	just surprising they waited this long to remove it	23:55
pleia2	indeed	23:55
*** SumitNaiksatam has quit IRC		23:55
clarkb	right after beta 2 or something	23:55
*** tsg has quit IRC		23:55
pleia2	they also swapped out the gui software installer quite late for something that barely works, so	23:56
*** moravec has joined #openstack-infra		23:56
clarkb	ooh fun	23:56
clarkb	I have already decided that I won't be upgrding and will be tumbleweeding at some point soon	23:56
pleia2	yeah, switched to gnome-software from the rickety old ubuntu software center written in python2	23:56
clarkb	at least on the laptop which doens't have a giant nvidia card that needs to run	23:57
pabelanger	Ah, I heard that	23:57
*** ajmiller has quit IRC		23:58
*** Qiming has joined #openstack-infra		23:58
anteaya	I have no problem tumbleweeding myself, as long as I am able	23:59
clarkb	mostly I like the idea of a continuously updated and gated distro	23:59
clarkb	since thats sort of what we do here for openstack	23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!