Thursday, 2015-02-19

fungihey whaddyaknow, ironic py27 passes and runs db migrations for mysql and postgresql00:01
fungii guess i _could_ just drop the db_name param from the mysql-db and pgsql-db macros in that case00:01
fungisimplifies things nicely00:02
fungialso tested neutron and cinder. they check out without needing anything additional beyond what i install for nova00:02
fungii expect ceilo to be a little on the special side tho00:02
fungirunning it now00:03
fungii should probably try some random clients too00:03
fungithough those don't need the db00:03
fungibut would be a good exercise of teh package set00:03
clarkbya clients shouldn't need the db, just lxml00:03
*** yamamoto has joined #openstack-infra00:04
*** camunoz has quit IRC00:05
*** dangers is now known as dangers_away00:07
fungiyep, ceilo wants mongod00:08
clarkbwe don't install that for them do we?00:08
clarkbapparently we do00:08
clarkbalso zookeeper is there for tooz/taskflow00:08
fungifor their unit tests, looks like we preinstall mongodb-server00:08
*** patrickeast has quit IRC00:10
openstackgerritgreghaynes proposed openstack-infra/project-config: Respect NODEPOOL_SSH_KEY in prepare-node for dib
*** stevemar has quit IRC00:16
fungiokay, mongodb-server got ceilo passing00:17
mordredThis is exciting00:18
jheskethclarkb: so I think we need to restart zuul to get the update swift timeout... is this somethign we want to do? Alternatively I should be able to increase the timeout on a per-job basis in the layout00:18
clarkbjhesketh: ya we probably should00:19
*** markvoelker has quit IRC00:19
jheskethclarkb: as in should restart or do the more graceful layout.yaml?00:19
clarkbjhesketh: sorry, should restart zuul00:19
clarkbjogo: I think that centos7 glusterfs job is broken00:20
fungii guess the tooz unit tests opportunistically check for zookeeper?00:20
clarkbjogo: at least on that tempest change, I am watching a couple others now00:20
clarkbfungi: yes iirc00:20
*** stevemar has joined #openstack-infra00:22
*** ddieterly has joined #openstack-infra00:22
*** camunoz has joined #openstack-infra00:22
openstackgerritMerged openstack-infra/jenkins-job-builder: xUnit publisher: add test-time-margin extra config option
clarkbharlowja_: ^ you would know care to full us in00:22
clarkbharlowja_: 00:20:12      fungi | i guess the tooz unit tests opportunistically check for zookeeper?00:23
harlowja_tooz/taskflow do at least00:24
*** dkliban is now known as dkliban_afk00:24
clarkbthanks (we are working on doing package installs at job runtime rather than during image build time, so that we can reduce the number of images we carry and better document/manage system deps)00:25
fungiharlowja_: thanks, i just managed to confirm that for myself too00:25
harlowja_redis also00:25
harlowja_and probably memcache (at least by tooz)00:26
clarkbwe would cache the packages on the images so installs should be pretty quick00:26
*** ddieterly has quit IRC00:26
*** sarob has quit IRC00:27
harlowja_does 'test-time-margin' affect these kind of probings (aka the tests try to connect to redis, and timeout if can't get there...)00:27
* harlowja_ pretty sure that the fails pretty quick, but just want to check00:27
fungicool, i'll check those too. thanks!00:28
openstackgerritLucas Dutra Nunes proposed openstack-infra/jenkins-job-builder: xUnit publisher: add option to skip if there are no tests files
openstackgerritMerged openstack-infra/jenkins-job-builder: add support for EnvInject mask password params option
clarkbharlowja_: I don't understand the test-time-margin question00:29
clarkbharlowja_: where is test-time-margin from?00:29
harlowja_nm, probably a bad up pointer, lol00:29
clarkbwe don't use xunit00:30
harlowja_ya, nm, my confusion00:30
clarkbthats just a feature that JJB supports but not something we use. that plugin has a global lock in the jenkins master so each job is serially processed one after the other00:31
clarkbharlowja_: create a major backup for jobs that run quickly when they bump up against jobs that run slowly00:31
* harlowja_ thought that was connected to this discussion, but i think its not, ha00:31
*** rmcall has joined #openstack-infra00:31
*** stevemar has quit IRC00:32
*** bhuvan has quit IRC00:32
*** zul has quit IRC00:32
fungiharlowja_: yeah, this is just working on figuring out how to make unit and static analysis jobs run successfully (and completely) on the same relatively raw servers where we also run devstack-based jobs00:32
*** melwitt has quit IRC00:33
openstackgerritMerged openstack-infra/jenkins-job-builder: Add 'last-completed' to copyartifact 'which-build' values
*** tmcpeak has quit IRC00:33
*** ghostpl_ has joined #openstack-infra00:33
fungiharlowja_: interesting, so with zookeeperd, redis-server and memcached installed, the tooz py27 job still says Ran 295...skips=24300:33
clarkbfungi: maybe the opportunistic stuff isn't working, should cross check against previous jobs on bare-*00:34
harlowja_probably missed out on a few more oppurinities fungi00:34
fungiyep, doing00:34
harlowja_kazoo (zookeeper), memcached, redis should all proceed00:34
clarkbharlowja_: well we only care about maintaining those that are tested today00:34
clarkband if none are tested today then this gets simpler :)00:35
*** tmcpeak has joined #openstack-infra00:35
harlowja_fair enough, ha00:35
fungiour motto is "no regressions"00:35
fungi(or as few as possible anyway)00:35
clarkbshould be simpleish to fix once we move too since this makes it a lot more direct in the job(s)00:35
fungieven moreso if i do the bindep integration i want next00:36
clarkbjhesketh: did you see my comment on 98306?00:36
fungiahh, yep, same quantity of tests run and skipped on a bare-trusty node too, so this matches anyway00:36
clarkbyup confirmed the glusterfs centos7 job is borked00:36
clarkbthe nodes just seem to stop talking to jenkins after a bit then zuul restarts the jobs00:37
clarkbI am going to push up a change to stop running that test00:37
jheskethclarkb: nope, that's an old one... I'll need to refresh my state but will reply shortly :-)00:37
*** ghostpl_ has quit IRC00:38
*** dprince has quit IRC00:38
openstackgerritClark Boylan proposed openstack-infra/project-config: Remove the tempest glusterfs job
clarkbjogo: fungi ^00:39
clarkbjhesketh: ya I sat down with fresh eyes and had new questions :)00:39
clarkband the author of that test isn't in here anymore00:40
*** david-lyle is now known as david-lyle_afk00:41
fungiclarkb: who was shepherding that job?00:41
jheskethclarkb: you mean since the first patchset, or you have more since your latest comment (and I should wait before replying)00:41
*** bdpayne has quit IRC00:42
*** gordc has quit IRC00:42
clarkbjhesketh since the land before time. my comment thwre is current00:42
*** zul has joined #openstack-infra00:42
*** tmcpeak has quit IRC00:43
*** arosen1 has quit IRC00:45
clarkbfungi: thats an example of a job that just dropped off00:46
funginice. my bet is on network reconfiguration00:46
clarkbya but gluster shouldn't do any of that but its a long time since I played with gluster00:46
fungithough i suppose it could be flat out panicking the instance00:47
clarkbit is in the object storage tests I don't know if glusterfs is being used as a backend for swift there but ya it may be causing the fs to go belly up00:47
clarkband kernel panic00:47
*** tkelsey has quit IRC00:49
*** ddieterly has joined #openstack-infra00:51
*** hyakuhei has quit IRC00:51
*** markvoelker has joined #openstack-infra00:52
*** ddieterl_ has joined #openstack-infra00:52
*** ddieterly has quit IRC00:53
*** pcrews has quit IRC00:55
ianwclarkb: i guess on reflection i want to walk "providers->images" not "labels->provider" since the logger is setup logging.getLogger("" % (,
clarkbianw: yup00:55
clarkbianw: I think one of my comments addresses that (and using a set as there may be duplicate entries in that list)00:56
*** dannywilson has quit IRC00:56
*** nelsnelson has joined #openstack-infra00:56
*** amitgandhinz has joined #openstack-infra00:56
openstackgerritJoshua Hesketh proposed openstack-infra/nodepool: Remove the need for zmq in proper demand calcs.
jheskethclarkb: ^ updated the commit message to help a little (and commented)00:57
*** otter768 has joined #openstack-infra00:57
ianwclarkb: don't think there are duplicates in provider->images['name'] ?00:57
clarkbjhesketh: thank you00:57
*** ZZelle_ has quit IRC00:57
*** sdake__ has joined #openstack-infra00:58
*** bknudson has quit IRC00:58
clarkbianw: well you want to map label -> image -> provider to only use images that are in use otherwise its just noise (though maybe that won't hurt here)00:58
clarkbianw: and you can have mutiple entries for image -> provider from multiple labels. one example is the case of our 2 node envs they use the same images as our single node envs00:58
*** markvoelker has quit IRC00:59
ianwclarkb: right, but i think really the simplest thing to do is just create loggers for provider.image['name'] .  even if min-ready is 0 it's still there as a image build01:00
*** amitgandhinz has quit IRC01:00
*** sdake_ has quit IRC01:01
clarkbianw: not talking about min ready here, we don't build images if they are not associated to a label regardless of the min ready01:01
*** otter768 has quit IRC01:01
*** sigmavirus24 is now known as sigmavirus24_awa01:02
clarkbsince nodepool only acts on images that end up attached to jenkins with a label01:02
clarkbbut you are probably right, that since its just logging config not a runtime optimization we can ignore that01:02
ianwi even called the fn _get_providers_and_images so i'm not sure what happened in between.  too much staring at yaml01:04
*** yamamoto has quit IRC01:05
*** vishy has quit IRC01:07
*** vishy has joined #openstack-infra01:08
*** patrickeast has joined #openstack-infra01:10
jeblairoh, i'm an hour late, but happy utc lunar new year01:12
*** sdake__ has quit IRC01:13
openstackgerritGhanshyam Mann proposed openstack-infra/project-config: Add non-voting job for Nova V2.1 API with neutron
openstackgerritJoe Gordon proposed openstack-infra/devstack-gate: Set custom cpu_model for live_migrate
*** krotscheck has quit IRC01:18
*** MarkAtwood has joined #openstack-infra01:19
*** sputnik13 has quit IRC01:21
asselinjeblair, happy lunar year :)01:22
*** zz_dimtruck is now known as dimtruck01:23
openstackgerritIan Wienand proposed openstack-infra/nodepool: Add generate-log-config command
*** mayurig has joined #openstack-infra01:25
*** ivar-laz_ has quit IRC01:25
*** mayurig has quit IRC01:25
*** mtanino has quit IRC01:25
*** ivar-lazzaro has joined #openstack-infra01:26
*** ivar-lazzaro has quit IRC01:28
*** markvoelker has joined #openstack-infra01:29
*** ivar-lazzaro has joined #openstack-infra01:29
*** nelsnelson has quit IRC01:29
*** doug-fish has quit IRC01:31
*** EmilienM is now known as EmilienM|afk01:36
*** stevemar has joined #openstack-infra01:37
*** ivar-lazzaro has quit IRC01:40
*** crc32 has joined #openstack-infra01:41
*** ivar-lazzaro has joined #openstack-infra01:41
clarkbzaro: I just saw that space jam dunk01:41
*** kgiusti has left #openstack-infra01:43
*** tiswanso has joined #openstack-infra01:47
*** tiswanso has quit IRC01:49
*** tiswanso has joined #openstack-infra01:50
anteayazaro: and to you01:53
* anteaya is a sheep01:53
anteayazaro: what are you?01:53
*** patrickeast has quit IRC01:54
*** devlaps has quit IRC01:55
*** claudiub has joined #openstack-infra02:00
*** krotscheck has joined #openstack-infra02:03
*** crc32 has quit IRC02:03
*** vilobhmm has quit IRC02:03
*** krtaylor has joined #openstack-infra02:08
openstackgerritK Jonathan Harker proposed openstack-infra/nodepool: Reset the client object before creating the image
*** krotscheck has quit IRC02:12
*** pcrews has joined #openstack-infra02:13
*** krotscheck has joined #openstack-infra02:13
anteayamy goodness, two weeks of eating:
*** arosen1 has joined #openstack-infra02:19
arosen1Hi I was wondering if anyone could tell me why my CI is going to and here:
arosen1In my nodepool configurations I don't see any more references to pypi.dfw02:20
clarkbarosen1: depending on how old your images are puppet sets up pypi.dfw as a default02:21
clarkbI don't know why is also used yet02:21
*** ghostpl_ has joined #openstack-infra02:22
clarkbarosen1: look in /etc/pip.conf $HOME/.pip/pip.conf02:22
arosen1clarkb:  the only place I have it set is: scripts/${NODEPOOL_PYPI_MIRROR:-}02:22
openstackgerritMatthew Treinish proposed openstack-infra/project-config: Fix if check in neutron lib forward testing definition
*** FallenPegasus has joined #openstack-infra02:23
arosen1clarkb:  i should look on one of the jenkin slaves?02:23
mtreinishjhesketh, clarkb, fungi, anteaya: ^^^ That should be super simple02:23
clarkbarosen1: yes02:23
clarkbmtreinish: I have used my discretion to fast approve02:24
mtreinishclarkb: cool, thanks02:24
arosen1clarkb:  ah okay i see: [global]02:24
arosen1index-url = there02:24
*** MarkAtwood has quit IRC02:25
*** FallenPegasus is now known as MarkAtwood02:25
arosen1clarkb:  seems weird because:
clarkbarosen1: images up to date?02:26
arosen1clarkb:  hrm one thing i'm wondering is why  i only have one dib image here :
*** ghostpl_ has quit IRC02:27
arosen1clarkb:  when I added this new provider it didn't create a dib image just a normal image though it's the same config as the other provider02:27
clarkbarosen1: ya the idea is you can use a single dib image02:28
arosen1clarkb:  okay let me delete the dib image and have it rebuild it and see if that fixes this.02:29
clarkbarosen1: this way you don' do a couple hundred image builds a day like us :) also you get better consistency across providers02:29
*** armax has quit IRC02:30
arosen1clarkb:  ah okay i see so you don't build a new dib image for multiple providers.02:31
clarkbnope, it does mean a few things get a little weird but overall its been a big win02:31
arosen1i went to update the image but i created a new provider to do so and i guess it pushed the same image.02:31
arosen1I added a new provider only because there is this gluster bug in our cloud02:32
clarkbya it will upload the same qcow202:32
arosen1cool thanks hopefully this will fix it.02:32
*** achanda has quit IRC02:33
*** otter768 has joined #openstack-infra02:38
*** salv-orlando has quit IRC02:40
openstackgerritMerged openstack-infra/project-config: Fix if check in neutron lib forward testing definition
*** achanda has joined #openstack-infra02:42
*** koolhead17 has joined #openstack-infra02:45
*** achanda has quit IRC02:46
arosen1clarkb:  quick question. Rebuilding the image im seeing this error: INFO [ERROR] /tmp/image.ZsWcsDJr/mnt/opt/git/openstack-dev/devstack/stackrc:665 Could not determine host ip address. See local.conf for suggestions on setting HOST_IP.02:50
clarkbarosen1: I think there is a patch up to devstack to avoid that problem02:50
clarkbarosen1: otherwise check the HOST_IP stuff that devstack is looking for02:51
arosen1clarkb:  how do i set that for nodepool dib?02:51
clarkbarosen1: it will depend on what DEVSTACK is looking for02:51
clarkber devstack02:51
*** dims_ has quit IRC02:52
clarkbits just a chroot so its your build node's interfaces that it iwll be looking at02:52
arosen1clarkb:  sorry i don't follow. This error is showing in the nodepool log.02:52
*** mriedem has joined #openstack-infra02:53
clarkbarosen1: yes, one of the things nodepool does is ask devstack for a list of files to cache. When it does that devstack expects to be able to determine your ip address02:53
*** cody-somerville has joined #openstack-infra02:53
clarkbarosen1: this is unnecessary for the functionality so there is a patch to devstack to stop requiring that, but as a workaround you just need to have devstack be able to determine the host's ip address02:53
arosen1clarkb: okay let me see want i need to do to make that happen.02:54
*** rkukura has quit IRC02:54
*** rkukura has joined #openstack-infra02:55
*** hyakuhei has joined #openstack-infra03:02
*** achanda has joined #openstack-infra03:02
openstackgerritAdam Gandelman proposed openstack-infra/elastic-recheck: Add query for bug 1423427
openstackbug 1423427 in OpenStack Compute (nova) "os-baremetal-node ext relies on presence of specific ironic node properties " [Undecided,New]
*** spzala has joined #openstack-infra03:10
*** harlowja_ is now known as harlowja_away03:15
*** amitgandhinz has joined #openstack-infra03:15
arosen1clarkb: do you know why you guys don't hit this bug ?03:18
clarkbdevstack can figure out what our hosts ip is03:19
*** rmcall has quit IRC03:21
*** nelsnelson has joined #openstack-infra03:36
*** unicell1 has quit IRC03:37
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck: Cap irc < 11.1 due to ImportError
*** salv-orlando has joined #openstack-infra03:41
*** achanda has quit IRC03:45
arosen1clarkb:  ah i figured it out. It's because my host_ip is in the range of as $FIXED_IPS03:47
arosen1clarkb:  i'm guessing there is no way to inject my own localrc or setting HOST_IP through nodepool.03:48
*** yamamoto has joined #openstack-infra03:48
arosen1i'd rather not have to re ip the box unless i don't have to .03:48
openstackgerritMerged openstack-infra/elastic-recheck: Cap irc < 11.1 due to ImportError
*** Vivek_ has quit IRC03:49
*** spzala has quit IRC03:50
*** achanda has joined #openstack-infra03:52
*** dims__ has joined #openstack-infra03:52
*** koolhead17 has quit IRC03:52
*** yamahata has quit IRC03:54
openstackgerritAdam Gandelman proposed openstack-infra/elastic-recheck: Add query for bug 1423427
openstackbug 1423427 in OpenStack Compute (nova) "os-baremetal-node ext relies on presence of specific ironic node properties " [Undecided,New]
*** dims__ has quit IRC03:57
*** hyakuhei has left #openstack-infra03:58
*** tkelsey has joined #openstack-infra03:59
*** amotoki has joined #openstack-infra04:04
*** koolhead17 has joined #openstack-infra04:09
*** koolhead17 has joined #openstack-infra04:09
*** ghostpl_ has joined #openstack-infra04:11
*** otter768 has quit IRC04:11
*** teran has quit IRC04:13
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck: Uncap some test requirements
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck: Uncap some test requirements
*** ghostpl_ has quit IRC04:16
*** achanda has quit IRC04:16
mordredclarkb: gahhhhhh!!!!!!!04:16
*** claudiub has quit IRC04:19
*** yamamoto has quit IRC04:20
arosen1clarkb:  doh i figured it out. I just needed to export HOST_IP="1..." in the process that  nodepoold was running to get it to use that value instead.04:22
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck: Add query for bug 1423427
openstackbug 1423427 in OpenStack Compute (nova) "os-baremetal-node ext relies on presence of specific ironic node properties " [Undecided,New]
*** ivar-laz_ has joined #openstack-infra04:25
*** coolsvap_ is now known as coolsvap04:28
*** btran has quit IRC04:28
*** ivar-lazzaro has quit IRC04:29
*** nelsnelson has quit IRC04:30
*** ivar-laz_ has quit IRC04:30
*** salv-orlando has quit IRC04:31
openstackgerritMerged openstack-infra/elastic-recheck: Add query for bug 1423427
openstackbug 1423427 in OpenStack Compute (nova) "os-baremetal-node ext relies on presence of specific ironic node properties " [High,Confirmed]
*** koolhead17 has quit IRC04:36
*** changbl has joined #openstack-infra04:38
mordredclarkb: +++++++++04:40
*** woodster_ has quit IRC04:40
mordred(We miss you in the bar)04:40
clarkbso I hear04:41
clarkbmordred I appointed crinkle my proxy04:41
clarkband instructed on proper cloud init stance04:41
*** markvoelker has quit IRC04:42
*** miqui has joined #openstack-infra04:42
*** markvoelker has joined #openstack-infra04:42
*** markvoelker has quit IRC04:46
mordredclarkb: I may be coming around .. You haven't taken the right stance on a different topic04:47
*** dimtruck is now known as zz_dimtruck04:47
clarkbwhats the other thing04:47
clarkbthe ansible stuff? I think I have come around. facter does the right thing04:51
*** BharatK has joined #openstack-infra04:52
*** amitgandhinz has quit IRC04:52
*** miqui has quit IRC04:52
*** miqui has joined #openstack-infra04:53
mordredclarkb: I'm not telling, I'm assuming we are in complete agreement04:56
*** bhunter71 has quit IRC04:58
*** asselin_ has joined #openstack-infra05:00
*** camunoz has quit IRC05:01
*** vigneshvar has joined #openstack-infra05:02
*** asselin_ has quit IRC05:02
*** mriedem has quit IRC05:05
*** coolsvap is now known as coolsvap_05:05
*** yamamoto has joined #openstack-infra05:07
*** garyh has quit IRC05:09
*** mriedem has joined #openstack-infra05:10
*** mriedem has quit IRC05:11
*** koolhead17 has joined #openstack-infra05:11
*** markvoelker has joined #openstack-infra05:13
*** camunoz has joined #openstack-infra05:14
*** coolsvap_ is now known as coolsvap05:15
*** achanda has joined #openstack-infra05:16
*** tkelsey has quit IRC05:17
*** markvoelker has quit IRC05:17
*** achanda has quit IRC05:18
*** achanda has joined #openstack-infra05:18
*** Sukhdev has joined #openstack-infra05:19
*** yamahata has joined #openstack-infra05:19
dstufftmordred: what dates are you in philly again05:19
dstufftI forget05:20
dstufftmarch something05:20
mordreddstufft: march 9-1005:20
dstufftmordred: cool thanks05:23
*** koolhead_ has joined #openstack-infra05:24
*** koolhead17 has quit IRC05:27
*** mfink_ has joined #openstack-infra05:28
*** ParsectiX has quit IRC05:30
*** ParsectiX has joined #openstack-infra05:31
openstackgerritKhai Do proposed stackforge/gertty: Disable InsecureRequestWarning
*** gyee has quit IRC05:36
*** vigneshvar has quit IRC05:36
*** vigneshvar has joined #openstack-infra05:37
*** sandywalsh has joined #openstack-infra05:37
*** sandywalsh_ has quit IRC05:40
*** amotoki_ has joined #openstack-infra05:44
*** achanda has quit IRC05:45
openstackgerritIgor Belikov proposed openstack-infra/jenkins-job-builder: Add 'readable-message' parameter to gerrit trigger
*** tkelsey has joined #openstack-infra05:47
*** amotoki has quit IRC05:47
*** zigo has quit IRC05:54
*** zigo has joined #openstack-infra05:55
*** achanda has joined #openstack-infra05:56
*** ParsectiX has quit IRC05:58
*** ParsectiX has joined #openstack-infra05:58
*** ghostpl_ has joined #openstack-infra05:59
*** woodster_ has joined #openstack-infra06:00
*** r1chardj0n3s is now known as r1chardj0n3s_afk06:01
*** ghostpl_ has quit IRC06:04
*** hamzy has quit IRC06:10
*** cnesa1 has quit IRC06:10
*** garyh has joined #openstack-infra06:10
*** otter768 has joined #openstack-infra06:12
*** Sukhdev has quit IRC06:12
*** markvoelker has joined #openstack-infra06:13
*** hamzy has joined #openstack-infra06:14
*** mfink_ has quit IRC06:15
*** salv-orlando has joined #openstack-infra06:15
*** otter768 has quit IRC06:17
*** mfink_ has joined #openstack-infra06:17
*** markvoelker has quit IRC06:18
*** Sukhdev has joined #openstack-infra06:21
*** tkelsey has quit IRC06:22
*** signed8bit has joined #openstack-infra06:22
*** signed8bit_ZZZzz has quit IRC06:23
*** belmoreira has joined #openstack-infra06:26
*** miqui has quit IRC06:26
openstackgerritfumihiko kakuma proposed openstack-infra/devstack-gate: Allow for git base override
openstackgerritfumihiko kakuma proposed openstack-infra/devstack-gate: post_test_hook is not executed when the gate test returns an error
*** dannywilson has joined #openstack-infra06:34
*** camunoz has quit IRC06:38
*** dannywilson has quit IRC06:41
*** dannywilson has joined #openstack-infra06:42
*** ildikov has quit IRC06:47
*** tkelsey has joined #openstack-infra06:49
*** yamahata has quit IRC06:51
*** yamahata has joined #openstack-infra06:52
openstackgerritMerged openstack-infra/jenkins-job-builder: xUnit publisher: add option to skip if there are no tests files
*** harlowja_at_home has joined #openstack-infra06:52
*** tkelsey has quit IRC07:03
*** mrunge has joined #openstack-infra07:03
*** mrunge has quit IRC07:03
*** mrunge has joined #openstack-infra07:03
*** Sukhdev has quit IRC07:04
*** mrmartin has joined #openstack-infra07:06
*** afazekas has joined #openstack-infra07:08
*** hdd has joined #openstack-infra07:08
*** Ryan_Lane has joined #openstack-infra07:11
*** harlowja_at_home has quit IRC07:13
*** markvoelker has joined #openstack-infra07:14
*** Sukhdev has joined #openstack-infra07:16
*** doobi-sham-95717 has quit IRC07:17
*** chlong has quit IRC07:19
*** markvoelker has quit IRC07:19
*** arosen1 has quit IRC07:20
*** MarkAtwood has quit IRC07:20
*** MarkAtwood has joined #openstack-infra07:21
*** Ryan_Lane has quit IRC07:22
*** salv-orlando has quit IRC07:26
*** ildikov has joined #openstack-infra07:29
*** dims__ has joined #openstack-infra07:30
*** zz_dimtruck is now known as dimtruck07:31
*** ZZelle has quit IRC07:32
*** ZZelle has joined #openstack-infra07:33
*** Sukhdev has quit IRC07:34
*** dims__ has quit IRC07:35
*** hdd has quit IRC07:37
*** cipcosma has joined #openstack-infra07:39
*** sputnik13 has joined #openstack-infra07:48
*** ghostpl_ has joined #openstack-infra07:48
*** sdake_ has joined #openstack-infra07:49
*** cinerama has quit IRC07:49
*** ghostpl_ has quit IRC07:53
*** vponomaryov has quit IRC07:53
*** vponomaryov has joined #openstack-infra07:54
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Bandit is a stackforge project
*** jgallard__ has joined #openstack-infra07:55
openstackgerritMerged openstack-infra/project-config: VPNaaS: make functional test voting
*** harlowja_away has quit IRC08:03
*** dimtruck is now known as zz_dimtruck08:07
*** doude has quit IRC08:08
*** arxcruz has joined #openstack-infra08:08
*** cnesa has joined #openstack-infra08:09
*** sergsh has joined #openstack-infra08:09
*** doude has joined #openstack-infra08:13
*** otter768 has joined #openstack-infra08:13
*** markvoelker has joined #openstack-infra08:15
*** otter768 has quit IRC08:17
*** Ala has joined #openstack-infra08:19
*** markvoelker has quit IRC08:20
*** mpavone has joined #openstack-infra08:21
*** stevemar has quit IRC08:22
*** HeOS has quit IRC08:23
*** scheuran has joined #openstack-infra08:23
*** yamamoto has quit IRC08:24
*** jcoufal has joined #openstack-infra08:25
*** salv-orlando has joined #openstack-infra08:25
*** achanda has quit IRC08:29
*** andreykurilin_ has joined #openstack-infra08:29
*** achanda has joined #openstack-infra08:29
*** sdake__ has joined #openstack-infra08:29
*** achanda has quit IRC08:30
*** woodster_ has quit IRC08:30
*** sdake_ has quit IRC08:33
*** koolhead_ has quit IRC08:35
*** luqas___ has quit IRC08:36
*** dtantsur|afk is now known as dtantsur08:41
openstackgerritMerged openstack-infra/jenkins-job-builder: Add support for 'All Nodes For Label Factory'
*** jistr has joined #openstack-infra08:42
*** sergsh has quit IRC08:43
*** hamzy has quit IRC08:43
*** hamzy has joined #openstack-infra08:44
*** zz_avozza is now known as zz_zz_avozza08:44
*** sergsh has joined #openstack-infra08:45
*** luqas___ has joined #openstack-infra08:47
*** cinerama has joined #openstack-infra08:48
*** markus_z has joined #openstack-infra08:48
*** MaxV has joined #openstack-infra08:50
*** deepakcs has joined #openstack-infra08:52
deepakcsclarkb, ping08:52
*** salv-orlando has quit IRC08:52
*** salv-orlando has joined #openstack-infra08:52
AJaegerdeepakcs, I expect clarkb to sleep...08:52
*** teran has joined #openstack-infra08:52
deepakcsAJaeger, sure :) I wanted to see if someone has more details on
deepakcsAJaeger, you seem to +2 it.. myself and BharatK are working on the glusterfs job. We are not clear on what 'partway' in the commit msg mean ?08:53
AJaegerdeepakcs, just check the experimental queue at
deepakcsAJaeger, ok.. checking now08:53
*** teran_ has joined #openstack-infra08:53
AJaegerthere'S a job running for 40+ hours - and that because of glusterfs08:54
deepakcsAJaeger, is it bcos of 'check experimental' in ?08:55
* deepakcs still trying to understand why it would take 40+ hours. local tempest runs go fine08:56
AJaegerdeepakcs, also scrollback or read the public logs for details08:56
*** jp_at_hp has joined #openstack-infra08:56
*** teran has quit IRC08:57
AJaegerdeepakcs, as said in the commit message: This comes from "check experimental" and somehow the glusterfs job stops in the middle and then gets restarted - in an endless loop08:57
AJaegerWe can't have that ;( Please talk with clarkb or fungi at US time again on how to fix this.08:57
deepakcsAJaeger, i read the logs, it says its failign in the object tests08:58
*** andreykurilin_ has quit IRC08:58
AJaegerdeepakcs, I can't help further here...08:58
deepakcsAJaeger, any idea how to add a job to run tempest volume tests only ?08:59
AJaegerdeepakcs, there're ways to limit what tempest tests are run, check what other jobs do - in project-config/jenkins/jobs/*09:00
deepakcsAJaeger, thanks, will do09:00
*** xyang1 has quit IRC09:03
*** jlibosva has joined #openstack-infra09:03
*** jcoufal_ has joined #openstack-infra09:05
AJaegerdeepakcs, the top most experimental job just got into the next iteration ;(09:07
AJaegerand I now approved that change to get it stopped09:07
AJaegerdeepakcs, once you're sure everything is working as it should, we can enable the test again09:08
*** jcoufal has quit IRC09:08
deepakcsAJaeger, you mean is running for 40+ hours. I am still trying to understand how is that related to glusterfs job09:08
AJaegerdeepakcs, look at
AJaeger141424 is the first change09:08
AJaegerIf you monitor the glusterfs job, you see it'S the only job that has not passed09:09
AJaegerAnd if you look at it over time, you see that it runs for a long time - and then starts again, as just now09:09
*** dizquierdo has joined #openstack-infra09:10
AJaegerI can't help you debug why this happens - but what happens is that the glusterfs job suddenly dies and then zuul starts it again and again and again - now since 40 hours.09:10
deepakcsAJaeger, i see 141424 change, but what do I need to do to monitor glusterfs job ? The zuul links points me back to the patch09:11
AJaegerdeepakcs, check the link of the glusterfs job, it's currently:
AJaegerOr check one of the others - each glusterfs job is a link09:11
*** mpaolino has joined #openstack-infra09:11
*** ghostpl_ has joined #openstack-infra09:12
*** vigneshvar has quit IRC09:12
deepakcsAJaeger, clicking on check-tempest-dsvm-full-glusterfs-centos7 #36: None  gets me to the same page again :)09:13
openstackgerritMerged openstack-infra/project-config: Remove the tempest glusterfs job
deepakcslet me check the console link09:13
AJaegerdeepakcs, yes, in jenkins you need to click on console output09:13
*** vigneshvar has joined #openstack-infra09:14
deepakcsAJaeger, thanks, looking at that.. new to jenkins/infra... so trying to make my way to the error. I still haven't been able to see the actual error on the node that causes zuul to think its gone and hence restartsit09:14
AJaegerdeepakcs, there are more experimental glusterfs jobs running if you need more data ;)09:15
deepakcsAJaeger, yeah i am still a bit lost on what exactly is happening inside the job. Shall i just start from the jenkins dashboard to get a list of all experimental glustefs jobs ?09:16
AJaegerdeepakcs, you might need to discuss with clarkb and fungi later today (US time - so UTC 15:00 or later).09:16
deepakcsAJaeger, ok will do09:16
AJaegerstart from the zuul page - we have several jenkins running and zuul is the best way to guide you09:16
*** markvoelker has joined #openstack-infra09:16
*** skolekonov has joined #openstack-infra09:18
*** dannywilson has quit IRC09:18
*** derekh has joined #openstack-infra09:19
*** markvoelker has quit IRC09:21
*** ildikov has quit IRC09:27
*** salv-orlando has quit IRC09:29
*** yamamoto has joined #openstack-infra09:31
*** salv-orlando has joined #openstack-infra09:33
*** ssam2 has joined #openstack-infra09:34
*** HeOS has joined #openstack-infra09:35
*** sdake_ has joined #openstack-infra09:36
*** yamahata has quit IRC09:39
*** ildikov has joined #openstack-infra09:40
*** sdake__ has quit IRC09:40
*** mpaolino has quit IRC09:43
*** viktors|afk is now known as viktors09:47
*** yamamoto has quit IRC09:50
*** zz_zz_avozza is now known as avozza09:52
*** hashar has joined #openstack-infra09:54
mescanefhi there, ihve created a patch for adding new repository on stackforge for fuel plugin (
mescanefwould like to be added to the core-group09:55
mescanefplease add me :-)09:55
*** mpaolino has joined #openstack-infra09:56
*** yamamoto has joined #openstack-infra10:00
AJaegermescanef, which repository exactly and what's your email?10:01
mescanefAJaeger: fuel-plugin-contrail10:02
mescanefAJaeger: mmatuszkowiak@mirantis.com10:02
mescanef"stackforge/fuel-plugin-contrail" ;-)10:02
AJaegermescanef, Let's ask fungi or clarkb to do this later today (US time). They normally scroll back and should see the above lines.10:03
mescanef:-)) ok TY10:03
*** teran_ has quit IRC10:08
*** yamamoto has quit IRC10:10
*** hashar_ has joined #openstack-infra10:13
*** amotoki_ has quit IRC10:14
*** otter768 has joined #openstack-infra10:14
*** hashar has quit IRC10:17
*** markvoelker has joined #openstack-infra10:17
*** amotoki has joined #openstack-infra10:18
cody-somervilleHey folks.10:18
cody-somervilleAre you seeing timeouts on sshing into nodes in hp cloud on spawn?10:19
*** otter768 has quit IRC10:19
*** dtantsur is now known as dtantsur|bbl10:21
*** ihrachyshka has joined #openstack-infra10:21
*** koolhead17 has joined #openstack-infra10:21
*** e0ne has joined #openstack-infra10:21
*** yamamoto has joined #openstack-infra10:21
*** markvoelker has quit IRC10:22
*** briancline has quit IRC10:22
*** hogepodge has quit IRC10:22
*** briancline has joined #openstack-infra10:23
*** teran has joined #openstack-infra10:24
*** hogepodge has joined #openstack-infra10:24
*** teran has quit IRC10:28
*** yamamoto has quit IRC10:31
*** nikil89_ has joined #openstack-infra10:35
*** nikil89_ is now known as naggappan10:35
*** rlandy has joined #openstack-infra10:38
*** e0ne is now known as e0ne_10:39
*** e0ne_ is now known as e0ne10:39
*** yamamoto has joined #openstack-infra10:39
*** ldnunes has joined #openstack-infra10:40
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Updates training guides jobs
*** rfolco has joined #openstack-infra10:42
*** jcoufal_ has quit IRC10:45
*** jcoufal has joined #openstack-infra10:46
*** yfried_ is now known as yfried|afk10:46
*** pelix has joined #openstack-infra10:46
*** yamamoto has quit IRC10:47
*** yfried|afk is now known as yfried_10:48
*** dims__ has joined #openstack-infra10:50
*** hashar_ has quit IRC10:52
*** e0ne is now known as e0ne_10:53
*** e0ne_ is now known as e0ne10:54
*** dims__ has quit IRC10:54
*** teran has joined #openstack-infra11:03
*** yfried_ is now known as yfried|afk11:04
*** mpaolino has quit IRC11:05
*** avozza is now known as zz_avozza11:07
openstackgerritMerged openstack-infra/elastic-recheck: Uncap some test requirements
*** salv-orlando has quit IRC11:11
*** salv-orlando has joined #openstack-infra11:12
openstackgerrityolanda.robla proposed openstack-infra/shade: Add hasExtension method to check cloud capabilities
*** hashar has joined #openstack-infra11:14
*** koolhead17 has quit IRC11:14
*** zz_avozza is now known as avozza11:16
openstackgerritSergey Lukjanov proposed openstack-infra/project-config: Clone using git.o.o instead of directly from gerrit
*** kashyap has left #openstack-infra11:17
*** markvoelker has joined #openstack-infra11:18
*** mpaolino has joined #openstack-infra11:19
*** hichihara has quit IRC11:19
*** jedimike has joined #openstack-infra11:20
*** mpaolino has quit IRC11:21
*** markvoelker has quit IRC11:23
openstackgerrityolanda.robla proposed openstack-infra/infra-specs: Spec for nodepool image update retries
*** yfried|afk is now known as yfried_11:24
*** salv-orlando has quit IRC11:31
*** aysyd has joined #openstack-infra11:31
*** salv-orlando has joined #openstack-infra11:31
openstackgerrityolanda.robla proposed openstack-infra/infra-specs: Add nodepool image testing spec.
*** e0ne is now known as e0ne_11:34
*** deepakcs has quit IRC11:38
*** pc_m has joined #openstack-infra11:40
*** hashar_ has joined #openstack-infra11:40
*** chlong has joined #openstack-infra11:41
*** hashar has quit IRC11:43
*** nicodemos has joined #openstack-infra11:44
*** vigneshvar has quit IRC11:49
*** marcusvrn1 has quit IRC11:49
*** enikanorov_ has quit IRC11:49
*** enikanorov has joined #openstack-infra11:50
*** yamamoto has joined #openstack-infra11:50
*** marcusvrn has joined #openstack-infra11:50
*** alexpilotti has joined #openstack-infra11:51
*** marcusvrn1 has joined #openstack-infra11:52
*** yamamoto has quit IRC11:55
*** marcusvrn has quit IRC11:56
*** dtantsur|bbl is now known as dtantsur11:56
*** adalba has quit IRC11:59
*** adalba has joined #openstack-infra11:59
*** agentle_ has quit IRC12:06
*** koolhead17 has joined #openstack-infra12:08
*** oomichi has joined #openstack-infra12:08
*** oomichi has quit IRC12:09
openstackgerritMerged openstack-infra/jenkins-job-builder: Add 'readable-message' parameter to gerrit trigger
*** mwagner_lap has quit IRC12:10
*** e0ne_ is now known as e0ne12:10
*** otter768 has joined #openstack-infra12:15
*** ihrachyshka has quit IRC12:16
*** yamamoto has joined #openstack-infra12:16
*** dannywilson has joined #openstack-infra12:19
*** markvoelker has joined #openstack-infra12:19
*** otter768 has quit IRC12:20
*** yfried_ is now known as yfried|afk12:21
*** woodster_ has joined #openstack-infra12:22
*** dannywilson has quit IRC12:23
*** markvoelker has quit IRC12:24
*** hashar_ has quit IRC12:28
*** doude has quit IRC12:30
*** salv-orlando has quit IRC12:31
*** hashar has joined #openstack-infra12:31
*** salv-orlando has joined #openstack-infra12:31
*** koolhead17 has quit IRC12:33
*** doude has joined #openstack-infra12:33
*** salv-orlando has quit IRC12:40
*** cdent has joined #openstack-infra12:40
*** salv-orlando has joined #openstack-infra12:41
*** avozza is now known as zz_avozza12:42
*** psedlak has quit IRC12:43
*** amotoki has quit IRC12:43
*** dims__ has joined #openstack-infra12:44
*** psedlak has joined #openstack-infra12:45
*** yfried|afk is now known as yfried_12:45
*** salv-orlando has quit IRC12:49
*** tiswanso has quit IRC12:53
*** dkranz has quit IRC12:54
*** spredzy|afk is now known as spredzy12:57
*** Hal has joined #openstack-infra12:58
*** EmilienM|afk is now known as EmilienM12:58
*** kashyap has joined #openstack-infra12:58
*** Hal is now known as Guest4322412:59
*** marcusvrn has joined #openstack-infra13:01
*** yfried_ is now known as yfried|afk13:02
*** markvoelker has joined #openstack-infra13:02
kashyaprfolco, Redirecting discussion from #openstack-dev,13:03
*** marcusvrn1 has quit IRC13:04
kashyapYeah - for minimal installs - I can't recommend enough `virt-builder`, I personally think it is a lot more secure and convenient than 'disk-image-builder'13:04
kashyapHere's some convenient CLI on usage -
*** alexpilotti has quit IRC13:05
*** sputnik13 has quit IRC13:05
rfolcokashyap, hmm perhaps its a good time for moving to f21 ?13:06
kashyaprfolco, Sure. I've been testing/developing (and DevStack too) on F21 for about 6 months already13:06
kashyaprfolco, Especially, for a server/minimal-core style stuff, it should not be much disruptive at all13:07
*** vigneshvar has joined #openstack-infra13:08
openstackgerritNikita Konovalov proposed openstack-infra/storyboard: Session per request
rfolcokashyap, nice, the reason for rebuilding my base image is that f20 updates repo is broken for perl... and I had pre-cached many packages from the updates repo so I can't downgrade all them now...are your aware of that problem ?13:09
*** zz_avozza is now known as avozza13:10
*** koolhead17 has joined #openstack-infra13:11
*** koolhead17 has joined #openstack-infra13:11
kashyaprfolco, 1 sec, having a parallel discussion elsewhere on #openstack-nova13:11
*** BharatK has quit IRC13:12
rfolcokashyap, sure :)13:13
BobBallDoes Zuul have this magic way of finding changesets that it should check but it hasn't voted on?  (i.e. missed changesets)... My external CI seems to have queued up a whole bunch of old jobs from openstack-dev/sandbox without there being resubmits of the jobs or comments13:16
openstackgerritMerged openstack-infra/release-tools: Skip tag and push if tag exists
BobBalloh - ignore me... it was a "depends on" chain. Ouch.13:19
*** mbacchi has joined #openstack-infra13:21
*** Guest43224 has quit IRC13:21
*** yfried|afk is now known as yfried_13:22
*** cnesa has quit IRC13:23
*** dkliban_afk is now known as dkliban13:23
*** cnesa has joined #openstack-infra13:24
*** bswartz has quit IRC13:27
*** cnesa has quit IRC13:28
*** kgiusti has joined #openstack-infra13:29
openstackgerritMerged openstack-infra/storyboard: Removed unsupported python versions from setup.cfg
*** yfried_ is now known as yfried|afk13:32
*** nicodemos has quit IRC13:34
*** jistr_ has joined #openstack-infra13:37
kashyaprfolco, I don't know about any Perl broken updates13:37
*** ociuhandu has joined #openstack-infra13:37
rfolcokashyap, not sure if it affects ppc64 only...13:38
*** jistr has quit IRC13:39
*** markvoelker_ has joined #openstack-infra13:42
*** markvoelker has quit IRC13:42
*** yfried|afk is now known as yfried_13:45
*** Hal has joined #openstack-infra13:45
*** Hal is now known as Guest2155513:46
*** koolhead17 has quit IRC13:46
openstackgerritMerged openstack-infra/storyboard-webclient: Updated library versions.
*** hogepodge has quit IRC13:47
openstackgerritMerged openstack-infra/storyboard-webclient: Added root state
openstackgerritThierry Carrez proposed openstack-infra/python-storyboardclient: Improved exception handling
openstackgerritMerged openstack-infra/storyboard-webclient: Session checks resolution against session info service.
*** briancline has quit IRC13:48
*** jistr_ is now known as jistr13:48
*** briancline has joined #openstack-infra13:49
*** hogepodge has joined #openstack-infra13:49
*** salv-orlando has joined #openstack-infra13:50
*** yamamoto has quit IRC13:51
openstackgerritMerged openstack-infra/python-storyboardclient: Python34 Fixes.
*** sigmavirus24_awa is now known as sigmavirus2413:52
openstackgerritMerged openstack-infra/python-storyboardclient: Removed py33, pypy from python-storyboardclient
*** ihrachyshka has joined #openstack-infra13:52
openstackgerritAleksey Ripinen proposed openstack-infra/storyboard: Split auth in api and change user_tokens
*** mkerrin has quit IRC13:55
*** jklare has quit IRC13:56
*** hashar has quit IRC13:57
*** BharatK has joined #openstack-infra13:57
*** jklare has joined #openstack-infra13:58
*** claudiub has joined #openstack-infra13:59
*** bhunter71 has joined #openstack-infra14:00
openstackgerritAleksey Ripinen proposed openstack-infra/storyboard: Split auth in api and change user_tokens
*** bswartz has joined #openstack-infra14:01
*** jklare has quit IRC14:02
BobBallHow do I specify credentials for zuul-swift-upload-logs?  Seems I need to set environment variables like a HMAC but I can't find in the infra repos where this is created14:02
*** sdake_ has quit IRC14:03
*** jklare has joined #openstack-infra14:03
*** vigneshvar has quit IRC14:04
*** mpaolino has joined #openstack-infra14:04
*** pblaho has quit IRC14:07
*** BharatK has quit IRC14:08
*** naggappan has quit IRC14:08
*** dkranz has joined #openstack-infra14:09
*** weshay has joined #openstack-infra14:11
*** koolhead17 has joined #openstack-infra14:11
*** ddieterl_ has quit IRC14:12
*** xyang1 has joined #openstack-infra14:15
*** otter768 has joined #openstack-infra14:16
*** dustins has joined #openstack-infra14:20
*** BharatK has joined #openstack-infra14:20
*** otter768 has quit IRC14:20
anteayawznoinsk: are you in the nova meeting?14:23
wznoinskI am14:23
anteayaokay I hadn't seen your hand up, so I thought I would check14:23
*** mriedem has joined #openstack-infra14:27
*** gulic has joined #openstack-infra14:27
*** ildikov has quit IRC14:28
*** woodster_ has quit IRC14:30
openstackgerritMax Rasskazov proposed openstack-infra/jenkins-job-builder: Add support of multiple credentials for ssh-agent-credentials wrapper
openstackgerritMerged openstack/requirements: Remove configobj
*** EmilienM is now known as EmilienM|afk14:32
openstackgerritMerged openstack/requirements: remove python-openstackclient
*** hogepodge has quit IRC14:36
*** briancline has quit IRC14:37
*** markvoelker_ has quit IRC14:38
*** briancline has joined #openstack-infra14:38
*** hogepodge has joined #openstack-infra14:39
*** salv-orlando has quit IRC14:39
*** markvoelker has joined #openstack-infra14:39
*** sabeen has joined #openstack-infra14:39
*** yfried_ is now known as yfried|afk14:40
*** salv-orlando has joined #openstack-infra14:40
*** yfried|afk is now known as yfried_14:40
*** woodster_ has joined #openstack-infra14:41
*** koolhead17 has quit IRC14:42
*** tiswanso has joined #openstack-infra14:42
*** dustins has quit IRC14:43
*** bswartz has quit IRC14:43
*** marcusvrn1 has joined #openstack-infra14:45
*** marcusvrn has quit IRC14:46
*** marcusvrn has joined #openstack-infra14:46
anteayawznoinsk: were you going to introduce yourself?14:47
*** marcusvrn has quit IRC14:47
*** marcusvrn has joined #openstack-infra14:47
*** zz_dimtruck is now known as dimtruck14:49
*** marcusvrn1 has quit IRC14:49
*** deepakcs has joined #openstack-infra14:51
*** MaxV has quit IRC14:51
deepakcsclarkb, fungi ping14:51
anteayadeepakcs: best to state what you need14:52
anteayaothers may be available to help14:52
deepakcsanteaya, sure14:52
deepakcsclarkb, fungi reg:
*** ildikov has joined #openstack-infra14:52
deepakcsanteaya, ^^14:52
anteayait is merged14:53
anteayaso what do you need?14:53
deepakcsanteaya, :) yeah, it is removing the glusterfs tempest job, bcos in recent past the job was getting restarted by zuul14:53
anteayathat is fine14:53
*** wenlock has joined #openstack-infra14:53
anteayawhat you do need?14:53
deepakcsanteaya, i need that job back so that i can claim my CI to be working. I don't see those failures in the local setup14:54
anteayasee right there we have a problem14:54
deepakcsanteaya, as you can see, the job is getting killed bcos of some java hudson excp, i have no idea why thats happening14:54
anteayasince I am not interersted in helping you to _claim_ your CI is working14:55
*** yfried_ is now known as yfried|afk14:55
anteayaI will however support you in getting your CI working14:55
anteayacan you see the difference?14:55
deepakcsanteaya, yeah the latter :) i see it14:55
*** mwagner_lap has joined #openstack-infra14:55
deepakcsI did some research14:55
anteayaso you are asking how to get your CI working?14:55
deepakcsanteaya, and it looks like most of the glusterfs tempest jobs are failing with java excp14:55
*** sabeen has quit IRC14:55
anteayathat is good information to have14:56
deepakcsanteaya, which looks like zuul <-> jenkins issue ? ( I am not a expert in infra)14:56
deepakcsanteaya, and some of the jobs are failing with "Build timeout"14:56
anteayahave you a stacktrace of the exceptions?14:56
anteayathat might be a place to begin14:56
deepakcsanteaya, the link above, scroll to end, it has the java trace14:57
*** carl_baldwin has joined #openstack-infra14:57
*** yamahata has joined #openstack-infra14:57
*** ddieterly has joined #openstack-infra14:58
anteaya2015-02-19 08:40:36.696 | FATAL: hudson.remoting.RequestAbortedException: Unexpected termination of the channel14:58
deepakcsanteaya, yes and i have no idea why thats happening. so need help from infra folks14:59
*** sabeen has joined #openstack-infra14:59
*** bswartz has joined #openstack-infra14:59
anteayaso are you asking for information on what that exception means?14:59
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck: Uncap irc, block on 11.1
deepakcsanteaya, well it says IO Excp and below the original excp seems to be EOF, but not sure why or how is it related to glusterfs devstack job15:00
deepakcsanteaya, wondering in infra has seen such things before for any other job/scenario from where we can get some clues15:00
anteayawell since it occurs due to the addition of your job are you willing to consider that it might in fact be a result of your job and start there?15:00
deepakcsanteaya, there are around 30+ glustefs experimental jobs that have failed and they don't fail at the same testcase, its random, so can't tie it to the testcase running at failure too15:01
*** amitgandhinz has joined #openstack-infra15:03
sdagueso did someone look at the melody stats?15:04
*** dimtruck is now known as zz_dimtruck15:04
*** dustins has joined #openstack-infra15:04
*** dannywilson has joined #openstack-infra15:05
*** yfried|afk is now known as yfried_15:05
anteayaI think melody stats were mentioned the other day15:06
anteayaI haven't seen them mentioned today15:06
openstackgerrityolanda.robla proposed openstack-infra/nodepool: WIP: Not even close to finished - use shade
sdagueright, gerrit is quite slow again15:07
sdagueit would be nice to see if there is a thing to be done to make it not so, as it's definitely impacting review time15:08
*** dannywilson has quit IRC15:09
openstackgerritMax Rasskazov proposed openstack-infra/jenkins-job-builder: Add support of multiple credentials for ssh-agent-credentials wrapper
BobBalldeepakcs: I suspect that error means the node lost SSH connectivity ?15:10
deepakcsBobBall, ok, so does it mean something in the job is causing it to lose SSH connectivity ?15:11
openstackgerritMerged openstack-infra/elastic-recheck: Uncap irc, block on 11.1
deepakcsBobBall, anteaya couple of days back the job was running fine, with 2 testcases failing, it was never broken, until now15:11
BobBallIf it's reproducible, perhaps, yes.  Otherwise might be some random cloudy thing that caused that node to die15:11
AJaegeranteaya, sdague: I did some hands-on help with the training-guides team, could you review my updated patch this week, please?15:12
deepakcsBobBall, most of the jobs there are aborted due to this java excp15:12
deepakcsBobBall, and few due to build timeout error15:12
fungideepakcs: so, there are two prime possibilities for what was happening with that job15:12
openstackgerritMax Rasskazov proposed openstack-infra/jenkins-job-builder: Add support of multiple credentials for ssh-agent-credentials wrapper
deepakcsfungi, ah, u r there :)15:12
*** nelsnelson has joined #openstack-infra15:13
fungideepakcs: either something is making network configuration changes (routing, interface reconfiguration, firewall rules, et cetera) which are causing the jenkins master to no longer be able to reach teh worker, or something is causing a kernel panic (maybe something filesystem-related?)15:13
fungideepakcs: can you attempt to recreate the situation on virtual machines of your own to identify the problem?15:14
*** nelsnelson has quit IRC15:14
deepakcsfungi, we don't touch neutron / networking, its a cinder-glusterfs job. Do u remember where/when u saw the kernel panic.. most of the failed jobs I saw either had java excp or build timeout as the reason for abortion15:14
*** MaxV has joined #openstack-infra15:15
fungideepakcs: we see nothing. we just stop getting a response from the vm15:15
*** nelsnelson has joined #openstack-infra15:15
*** mpaolino has quit IRC15:15
*** koolhead17 has joined #openstack-infra15:15
deepakcsfungi, BharatK runs the tempest volume only and full tests in VM locally, and they are running fine.15:15
fungideepakcs: those java exceptions are the jenkins master saying "i am no longer getting a response from the remote worker so i'm giving up now"15:15
*** cdent has quit IRC15:15
*** cdent has joined #openstack-infra15:16
deepakcsfungi, 2 days back the job was running fine (~2 hours to complete as its a full test), but were failing 2 testcases, we added a change to allow nova policy for online snap, post which we are seeing this15:16
fungideepakcs: another possibility is an out-of-memory condition which is causing the kernel to kill lots of processes on the worker and also happens to blow out the jenkins slave agent in the process... is it possible this job would require more than 8gb ram?15:16
deepakcsfungi, with the same change on a local VM, we don't see that issue15:16
*** hyakuhei has joined #openstack-infra15:17
anteayaAJaeger: slides will be posted to docs after the patch merges?15:17
deepakcsBharatK, how much is ur local VM RAM where u test the tempest ?15:17
*** dims__ has quit IRC15:17
AJaegeranteaya, slides will not be posted at the moment for master15:17
BharatKdeepakcs, locally I am using 4GB RAM15:17
AJaegerThey're in complete reorganization mode and have nothing to publish from master for now...15:18
anteayaAJaeger: ah okay just created in the check and gate jobs, okay15:18
deepakcsfungi, since it says EOF excp (if u follow the caused by in java excp) i was wondering if its running out of disk space ?15:18
fungiso that's probably not the issue then15:18
anteayaAJaeger: ah okay thanks15:18
*** dims__ has joined #openstack-infra15:18
AJaegeranteaya, correct15:18
fungideepakcs: do you need more than 20gb disk?15:18
deepakcsBharatK, locally whats ur disk size ?15:18
BharatKfungi, in the local environment I am using 40 GB15:18
clarkbalso note we run the "full" tempest run not a subset15:19
BharatKfungi, while job is running I have seen the 34GB free space in "df -h" output15:19
deepakcsBharatK, fungi  but the max we do is create a loop device of 8G using the disk file to be used as a brick for glusterfs storage, so 20G should be sufficient15:19
clarkbso you may not see this locally if running a subset15:19
deepakcsclarkb, hi15:19
fungijenkins will automatically offline the worker if its root filesystem fills up, but it also often misreports a full filesystem as the reason for offlining when in actuality it loses contact with the worker, so that tends to be a false error15:19
fungideepakcs: do you create it in /opt or somewhere else? we do have hundreds of gb in /opt generally but / can be as small as 20gb15:20
BharatKclarkb, In local environment we ran full tempest15:20
deepakcsBharatK, u did run "full" tempest too, and it ran fine, right ?15:20
BharatKdeepakcs, just one hour back I ran full tempest without any issues.15:20
deepakcsBharatK, can u answer fungi 's q on where we create the 8G disk file ?15:21
deepakcsfungi, looking at the console logs, before the tests start, df -h shows 32G as free space for root15:21
clarkbBobBall you put your swift creds in zuul's ini file then zuul generates the job specific creds15:21
deepakcsfungi,  /dev/xvda1       40G  5.9G   32G  16% /15:21
BharatKfungi, we are creating loopback device in /var/lib/glance15:22
BobBalloh brilliant clarkb *looks* thanks.15:22
fungideepakcs: yeah, some have 40 and some 20 depending on what provider they end up running in (we have virtual machines from a couple different service providers)15:22
BharatKfungi, /var/lib/glusterfs/15:22
clarkbBobBall those creds are scoped to the container and path and have anexpiry iirc15:22
deepakcsfungi, i don't see /opt as a diff mnt in the df -h15:23
deepakcsso /opt is part of / itself15:23
clarkbthat df likely runs before the mount then15:23
clarkbsee the setup host log15:23
fungiwe might want to see if we can fix the ordering there at some point15:23
clarkbwe carve out the ephemeral drive and mount swap and /opt15:24
deepakcsclarkb, ok, setup host log can be seen only if the job completed gracefully, right ?15:24
*** timcline has joined #openstack-infra15:24
clarkbdeepakcs oh right because the node goes away in this case15:24
fungigood point, we won't have other logs/files on the aborted or timed out jobs15:24
deepakcsclarkb, yes :)15:24
BobBallclarkb: Yes - I saw that.  Now looking at the zuul config file it seems easy enough.  Will test and see if/where I get errors...15:25
fungiclarkb: any idea if we were we seeing this happen in both hpcloud and rax?15:25
deepakcsclarkb, fungi since we are not able to recreate this in local VM is it possible to somehow enable this job, but not come in the queue for others, so that we can debug with some ideas ?15:25
clarkbfungi no I didnt get that far. it was ever job running yesterday though so I would be suprised if it wasnt both15:27
openstackgerritAleksey Ripinen proposed openstack-infra/storyboard: Split auth in api and change user_tokens
fungideepakcs: i can try to rerun the job by hand as long as i know which provider i'm likely to trigger the failure scenario in15:27
deepakcsclarkb, fungi Other than the java excp , the only other error i saw was "build timeout / aborted" - what does that signify ? jenkins server timedout connecting to slave ?15:27
clarkbdeepakcs I think in both cases that is what happens15:28
*** koolhead17 has quit IRC15:28
deepakcsclarkb, fungi just wondering if there is any clue hidden in there when both the diff types of aborts are taken into consideration ?15:28
clarkbjenkins just doesnt express it the same depending on how it notices15:28
*** mtanino has joined #openstack-infra15:29
fungideepakcs: i suspect i'll need to manually run the same steps the job does, and then when i (likely) lose access to the virtual machine, try to log in through an out-of-band console to see what's gone wrong15:29
*** yamamoto has joined #openstack-infra15:29
deepakcsclarkb, fungi by provider u mean the cloud provider (hpcloud Vs rax) ?15:29
deepakcsclarkb, and where in the console log i can figure that ?15:29
clarkbat the beginning, it is encoded into the hostname15:29
*** tim_o has quit IRC15:30
sdagueclarkb: so, was there any resolution from the melody graphs on gerrit the other day? Gerrit still seems like it's dragging quite a bit15:30
fungii suspect anyone here running a lenovo laptop is installing linux anyway, but this sets a disturbing precedent:
fungii wonder what hardware backdoors they're also installing in those15:31
clarkbsdague no I think it likely need to be restarted again, but jeblair wasnt convinced? should see if the graph helped swing him over to restarting15:31
deepakcsclarkb, 2015-02-19 06:14:56.918 | Available disk space on this host:15:31
deepakcs2015-02-19 06:14:56.920 |     Filesystem      Size  Used Avail Use% Mounted on15:31
deepakcs2015-02-19 06:14:56.920 |     /dev/xvda1       40G  5.9G   32G  16% /15:31
deepakcs2015-02-19 06:14:56.920 |     devtmpfs        3.7G     0  3.7G   0% /dev15:31
deepakcs2015-02-19 06:14:56.920 |     tmpfs           3.7G     0  3.7G   0% /dev/shm15:31
deepakcs2015-02-19 06:14:56.920 |     tmpfs           3.7G  8.3M  3.7G   1% /run15:31
deepakcs2015-02-19 06:14:56.921 |     tmpfs           3.7G     0  3.7G   0% /sys/fs/cgroup15:31
deepakcsOops.. sorry!15:31
clarkbfungi ya never booted windows at all :)15:31
deepakcsso its rax provider15:31
clarkbdepoakcs please use a paste service15:31
deepakcsclarkb, that was by mistake15:32
*** AJaeger has quit IRC15:32
sdagueso I saw you send him a graph, but then the conversation stopped. Anyway, lots of "people nagios" are reporting in that it's slow, including me.15:32
deepakcsclarkb, fungi is it a bad idea to add the job again and we do some debug (Like limit the tests to volume only, reduce the size of loop device etc )?15:33
fungideepakcs: i can grab a centos7 machine in rax-iad and try to recreate the failure15:33
mugsieanteaya: ping?15:33
fungiclarkb: when did we get confirmation that those centos7 workers were actually operational?15:33
*** achanda has joined #openstack-infra15:33
deepakcsclarkb, fungi or add it to a different expt pipeline so that it doesn't affect others ?15:33
clarkbfungi a while back, just devstack/temlest failed on them but the nodes themselves were ok15:34
fungioh, got it. so as long as they're not running devstack/tempest on centos7 their job should work i guess?15:34
deepakcsfungi,  i just randonly checked 4-5 jobs and all had 'rax' in the hostname15:34
fungibecause it sort of sounds like they're trying to set up devstack and run tempest against it on centos7 there15:35
clarkbfungi it is a tempest job but the other centos7 jobs dont die like this they fail normally15:35
*** frickler has joined #openstack-infra15:35
deepakcsfungi, yes thats what we are doing there15:35
fungioh, got it. so other jobs failing on centos7 weren't tanking the entire worker15:35
*** avozza is now known as zz_avozza15:36
fungii'll hold a ready devstack-centos7 worker in rax and see if i can recreate the failure by hand15:36
openstackgerritMax Rasskazov proposed openstack-infra/jenkins-job-builder: Fix broken link on documentation
deepakcsfungi, let me or BharatK know if u need any help15:37
*** yamamoto has quit IRC15:37
*** zz_dimtruck is now known as dimtruck15:37
deepakcsfungi, clarkb Do we in parallel also start looking at a F20/f21 based gate job, if centos7 has issues ?15:37
*** ddieterly has quit IRC15:38
anteayamugsie: how can I help?15:38
*** yamamoto has joined #openstack-infra15:39
anteayawhat about it?15:39
clarkbdeepakcs no I think we figure this one out15:39
mugsieis there a problem allowing us to have the third party ci?15:39
anteayado you need to have third party ci?15:40
mugsiewe don;t have a massive immediate need, but I want to try running a few other job against our patches15:40
anteayaI'm feeling is is a fashion statement rather than something folks need15:40
deepakcsclarkb, fungi thanks in advance :) Do we shar eour email IDs with you so that you can keep us in the loop (given the TZ diff, we will be asleep soon) :)15:40
mugsieand we will eventually need it (in the next cycle or so)15:40
anteayamugsie: you can run a few jobs against your patches15:40
anteayaand you can have the ci systems comment on your patches15:40
mugsiebut not vote15:40
anteayaall without needing a gerrit group15:40
fungimugsie: you need to test things against designate that involve special hardware or non-free software? curious why you can't just run the jobs in our system15:41
*** achanda has quit IRC15:41
*** EmilienM|afk is now known as EmilienM15:41
mugsiei am not sure if they are usful, its a sonar job15:41
openstackgerritJames Graf proposed openstack-infra/jenkins-job-builder: Add support for Job DSL plugin for Groovy-based job and view creation and management
fungimugsie: that'15:42
fungis coverage right?15:42
mugsieI want to see if having something that votes helps us with code complexity15:42
mugsiecode complexity / syntax etc15:42
fungiahh, code complexity15:42
deepakcsclarkb, fungi ^^15:42
mugsieit can take code coverage as well15:42
mugsiebut its mainly a static analyis tool15:42
*** skolekonov has quit IRC15:43
mugsiewe will need to start testing backend that talk to APIs / comercial software as well, and 3rd party testing is must for that15:43
fungimugsie: a commercial/proprietary tool?15:43
mugsienope, there is a GPL version15:43
clarkbthere are free/open tools to do this15:43
*** abhirc has joined #openstack-infra15:44
fungimugsie: because you can probably just install and run it on our workers without needing to set up a separate entire system for it15:44
*** Sukhdev has joined #openstack-infra15:44
fungideepakcs: if you can't leave your irc client connected while you sleep, then i guess start a brief mailing list thread and one of us will follow up to it with what we find15:44
deepakcsfungi, ok, will start a thread on -infra, hope that is fine ?15:45
clarkbyes that would be good15:45
fungideepakcs: i would say start one on instead15:45
fungibut infra will work if you'd rather15:45
clarkbor there15:45
mugsiefungi: yeah. but if left as a long running thing it gives interesting trends etc15:45
clarkbactually ya -dev will get more eyes on it and maybe other will recognize the behavior15:46
deepakcsi thought infra was more appropriate for this15:46
fungimugsie: oh, so it's not a testing tool, it's a trending tool?15:46
fungideepakcs: infra is more appropriate for topics which are not of interest to openstack, but you're testing openstack with glusterfs, right?15:46
mugsieit can do both. Its can give an incremental report, and then keep track of results over time15:46
deepakcsfungi, yes15:46
deepakcsfungi, i can start on -dev too, not a issue15:47
*** mrunge has quit IRC15:47
*** cipcosma has quit IRC15:48
fungimugsie: out of curiosity, what trending is of interest across _proposed_ changes? wouldn't you want to trend the state of the codebase itself?15:48
mugsiefungi: yeah, was going to trigger the trending jobs on post-merge15:49
fungimugsie: in that case as anteaya says you don't need a -ci group for that. it's only to be able to control what accounts can leave code verification votes on proposed changes15:49
fungimugsie: any account can listen to the event stream, which is all you need to be able to see the change-merged events15:50
*** hashar has joined #openstack-infra15:50
mugsieyeah, and as i said earlier - we have some backends that aren't fully tested currently, and will need this as well15:51
mugsie(commercial API driven dns vendors etc)15:51
*** jaypipes has quit IRC15:51
mugsiebut that req is not immediate15:51
fungiahh, and for that yes you would need something like this. we wouldn't be able to interface with your commercial vendor dns server devices15:51
*** imcsk8 has quit IRC15:51
*** yamamoto has quit IRC15:51
*** imcsk8 has joined #openstack-infra15:52
*** ddieterly has joined #openstack-infra15:52
* fungi assumes things like infoblox, bluecoat, et cetera15:52
mugsieso, shoiuld I back out the reuest until we need it asap? I figured getting it in early means its one less thjing to worry about15:52
mugsieyup, and Dyn, and as few others15:52
anteayamtreinish: can you indicate you are okay with this new non-voting job being added to tempest:
fungiit's fine if you forsee a need, just be wary not to run things externally which you could instead run upstream15:54
anteayamugsie: if you recall my question was why do you need it15:54
anteayamugsie: if you have a need that is fine15:54
mtreinishanteaya: sure I'll take a look15:55
anteayabut many folks feel they have to have something because other projects do and then don't know what to do with it15:55
anteayamtreinish: thanks15:55
*** bogdando has quit IRC15:55
*** ddieterl_ has joined #openstack-infra15:55
*** ddieterly has quit IRC15:56
*** yolanda has quit IRC15:57
*** bcourt has joined #openstack-infra15:57
*** Sukhdev has quit IRC15:58
*** yolanda has joined #openstack-infra16:00
*** ParsectiX has quit IRC16:00
*** ParsectiX has joined #openstack-infra16:01
*** hyakuhei has quit IRC16:01
*** jaypipes has joined #openstack-infra16:01
*** armax has joined #openstack-infra16:02
*** jlibosva has quit IRC16:04
*** annegentle has joined #openstack-infra16:04
*** david-lyle_afk is now known as david-lyle16:04
fungimescanef: i've added you as the initial member of,members sorry for the wait16:05
*** denis_makogon has quit IRC16:05
*** afazekas has quit IRC16:05
*** jlibosva has joined #openstack-infra16:06
mescaneffungi: thank you16:06
openstackgerritMatthew Treinish proposed openstack-infra/devstack-gate: DO NOT MERGE COPY ALL OF .TESTREPOSITORY to log server
mtreinishanteaya: lgtm16:07
mugsieanteaya: yeap, I understand that16:07
*** tmcpeak has joined #openstack-infra16:07
anteayamtreinish: thanks16:07
anteayamugsie: great, so can you add a reply as a comment on the patch16:07
mugsieit just seems everytime we go to get something that is standard for other projects we get push back16:07
anteayaso others can read it too?16:07
mugsieanteaya: doing it now16:07
anteayamugsie: you believe having a -ci group is standard for other projects?16:08
mugsieno, but the ability to have it is standard16:08
anteayathat is a perception that is a shame if that is what you believe16:08
*** hashar_ has joined #openstack-infra16:08
anteayayou have the ability to have it16:08
anteayaI simply asked a question16:08
anteayapart of my reviewing process is to ask questions16:08
*** arxcruz has quit IRC16:09
fungimugsie: other projects that have it don't have it just because they're able to have it. they have it because they have ci systems leaving comments on their changes and want to be able to control whether those can vote16:09
*** hashar has quit IRC16:10
fungimugsie: we simply ask that you don't speculatively request support for features you might exercise in the future without some certainty of that, because otherwise it adds unnecessary workload on our team, which is already stretched extremely thin (hundreds and hundreds of outstanding changes/support requests we have to prioritize)16:11
fungimugsie: so it's not just your project. pretty much any project which requests something should expect to have to justify their request16:12
*** marun has joined #openstack-infra16:12
*** deepakcs has quit IRC16:12
openstackgerritMatthew Treinish proposed openstack-infra/devstack-gate: DO NOT MERGE COPY ALL OF .TESTREPOSITORY to log server
*** cody-somerville has quit IRC16:13
*** sergsh has quit IRC16:13
*** stevemar has joined #openstack-infra16:14
anteayafungi clarkb any problem with raising a stable job timeout from 40 to 90? python26:
*** otter768 has joined #openstack-infra16:17
clarkbpcrews: ^ any chance we can bump that timeout a little less and still get things to go through?16:17
clarkbanteaya: my only concern is that that is a huge jump and may hide a larger underlying issue if 90 minutes is really necessary16:17
*** dannywilson has joined #openstack-infra16:19
*** dangers_away is now known as dangers16:19
*** dannywilson has quit IRC16:19
pcrewsclarkb: sure - just shot for a high value to ensure it would pass (or if it doesn't run in 90 min, it needs more tlc imo) - do you have an opinion / recommendation for time?16:19
anteayaclarkb: thanks16:19
clarkbpcrews: no immediate opinion, but usually if jobs start timing out over time we are straddling the boundary and don't need a large jump. maybe just bump to 50-60 minutes?16:20
*** otter768 has quit IRC16:21
fungipcrews: the general rule of thumb is that if you allow that job to run for 90 minutes instead of only 40 then it will _very_soon_ start taking 90 minutes instead of only 4016:21
pcrewsfungi: ack and noted ;)16:22
fungibecause the only pushback on whether tests are poorly-written and take too long is when they start causing the job to time out16:22
pcrewsclarkb: ack - omw to office then will submit a new patch16:22
*** marcusvrn1 has joined #openstack-infra16:22
*** zz_avozza is now known as avozza16:23
*** dannywilson has joined #openstack-infra16:23
fungiclarkb: the experimental glusterfs job has gotten as far as starting tempest now. still up and running at the moment16:23
anteayapcrews: ack, I 1'd for now16:23
fungino signs yet of resource exhaustion or disconnection16:23
*** med_` is now known as medberry16:24
*** ddieterl_ has quit IRC16:24
*** tmcpeak has quit IRC16:24
*** ssam2 has quit IRC16:24
*** medberry is now known as medberry216:24
clarkbfungi: ya it seems to go under during object storage tests16:24
clarkbfungi: so hopefully you will see something soon :)16:24
*** marcusvrn has quit IRC16:24
*** krtaylor has quit IRC16:25
fungiclarkb: running on if you get curious16:25
sdaguepcrews: do you have any idea why this regressed?16:25
*** ddieterly has joined #openstack-infra16:26
fungipcrews: also, note that we have some problem on centos6 nodes where sometimes git remote actions take an indefinite amount of time and cause jobs to time out. if it's hitting that, then upping the timeout won't help at all16:26
*** pcrews has quit IRC16:27
clarkboh right this is the python26 job16:27
fungii've so far been entirely unable to recreate the condition to be able to nail down a root cause, but seemed to start right after the most recent big security fix scramble16:27
mordredclarkb: I remembered why I did the puppet ansible thign in python rather than pure ansible yaml16:28
mordredclarkb: it's because there is a need to parse the stdout string frmo puppet because it double-uses 2 return codes for different things16:29
clarkbmordred: I want to cry right now16:29
mordredclarkb: and although the primitives for respoding to error codes are good - once you get to expressing string parsing logic in yaml16:29
mordredyou know16:29
mordredit's just crazypants16:29
*** rcarrillocruz has quit IRC16:29
mordredsorry it took me so long to remember the reasoning that led to that16:29
clarkbmordred: thats ok, maybe we should tack it into a docstring/comment for the python file that says "No really there are reasons and they are:"16:30
*** spredzy is now known as spredzy|afk16:30
mordredI will do that - as soon as I write the patch I owe you :)16:30
openstackgerritMerged openstack-infra/project-config: Remove retired repos from channel notification
*** rcarrillocruz has joined #openstack-infra16:32
*** ssam2 has joined #openstack-infra16:32
*** hashar_ has quit IRC16:32
*** openstackgerrit has quit IRC16:36
*** openstackgerrit has joined #openstack-infra16:36
*** jlibosva has quit IRC16:37
*** cnesa has joined #openstack-infra16:37
*** annegentle has quit IRC16:39
*** yamahata has quit IRC16:39
BobBallSo apparently I'm missing something obvious... Trying to add a new parameter I added swift_authurl to the class block mimicking other parameters but I get an error saying the parameter is invalid.  I'd appreciate help on two levels - 1) why is it invalid, 2) where is this line 114 it's complaining about :)
*** annegentle has joined #openstack-infra16:40
clarkbBobBall: I think the os_ext_testing puppet class doesn't take that parameter16:41
BobBallI added it at
clarkbBobBall: so likely need to update that puppet to pass through swift_authurl similarly to
BobBallI may have needed to add it somewhere else too, but that's the bit I don't understand currently...16:42
BobBallyes, that's exactly what I tried to do :/16:43
*** mpavone has quit IRC16:43
clarkbBobBall: that appears to be the right file, is it updated in your puppet path?16:43
krotscheckSo, when using storyboard, can y'all keep an eye on that page-not-loading-when-direct-link issue you were seeing? A patch that went in recently is supposed to fix that.16:43
*** ddieterly has quit IRC16:43
clarkbkrotscheck: noted, will try to track if I notice it16:43
krotscheckclarkb: Thanks.16:43
BobBallYup ... root@jenkins:~/os-ext-testing/puppet# grep -r swift_authurl /root/os-ext-testing/puppet/os-ext-testing/puppet/modules: /root/os-ext-testing/puppet/os-ext-testing/puppet/modules/logging/manifests/master.pp:  $swift_authurl = '',16:44
*** yfried_ is now known as yfried|afk16:45
*** ddieterly has joined #openstack-infra16:45
BobBallhang on...16:47
* BobBall is an idiot.16:47
BobBallThanks :)16:47
*** dmsimard_away is now known as dmsimard16:47
*** krtaylor has joined #openstack-infra16:47
clarkbBobBall: oh what was it? I had traced it down through the zuul manifest too and it seemed to be there16:48
clarkbthe only odd thing I noticed was the line number if complains about doesn't seem right based on your github link16:48
*** mestery has quit IRC16:49
clarkbkrotscheck: fwiw I just ran through a couple things that normally triggered it for me and it was fine (open bug, login, get bug back instead of blank page)16:50
*** hyakuhei has joined #openstack-infra16:50
BobBallI had a nested version of the repository which was messing things up16:50
BobBall"os-ext-testing/puppet/os-ext-testing/puppet" from above16:50
BobBallI need a rubber duck.16:50
*** koolhead17 has joined #openstack-infra16:50
*** tkelsey has joined #openstack-infra16:51
*** salv-orlando has quit IRC16:52
*** yamamoto has joined #openstack-infra16:52
*** scheuran has quit IRC16:52
fungigit clean -dfx16:53
openstackgerrityolanda.robla proposed openstack-infra/shade: Add hasExtension method to check cloud capabilities
*** gondoi is now known as zz_gondoi16:55
*** zz_gondoi is now known as gondoi16:57
*** yamamoto has quit IRC16:57
openstackgerritMerged openstack-infra/project-config: Python34 Jobs for StoryBoard are now voting.
*** coolsvap is now known as coolsvap_16:57
*** dprince has joined #openstack-infra16:58
*** tiswanso has quit IRC16:58
openstackgerritAndrew Lazarev proposed openstack/requirements: Bump sahara client version
*** annegentle has quit IRC17:01
*** jlibosva has joined #openstack-infra17:02
*** carl_baldwin has quit IRC17:02
*** ihrachyshka has quit IRC17:03
ctlaughAre there plans / another recommendation to move away from using scp-plugin?  No new release since 2011?17:03
openstackgerritpatrick-crews proposed openstack-infra/project-config: Increase py26 timeouts for nova-stable
zaroctlaugh: we use master17:04
krotscheckclarkb: Awesome, thanks17:05
ctlaughzaro: master, as in manually built it from master?17:05
anteayakrotscheck: ack17:05
zaroctlaugh: yes.17:05
*** ParsectiX has quit IRC17:05
clarkbctlaugh: zaro: and we are moving to using swift uploads instead17:06
clarkball of the infra jobs use swift uploaded logs exclusively now17:06
* anteaya hands BobBall a rubber duck17:06
*** ParsectiX has joined #openstack-infra17:06
clarkbor a vast majority at least17:06
ctlaughzaro: yeah, I did that too... just doesn't feel right relying on it.17:06
*** tmcpeak has joined #openstack-infra17:06
ctlaughclarkb: any more details/pointers to info on how that works?17:07
*** tmcpeak has quit IRC17:07
*** tmcpeak1 has joined #openstack-infra17:07
clarkbctlaugh: zuul hands out upload urls to jobs that are scoped to specific locations and have an expiry, then the job has a script that uploads files with that info17:07
fungithe glusterfs job is _still_ going17:07
clarkbfungi: is tempest still running new tests or has it hung?17:08
*** dtantsur is now known as dtantsur|afk17:08
fungiclarkb: still going through tests17:08
ctlaughclarkb: thank you!17:08
BobBallthanks anteaya!17:09
clarkbfungi: it is much slower than the non glusterfs job so this may be normal17:09
fungiyeah, the devstack setup phase took 20+ minutes (i didn't wall-clock it but a while)17:10
anteayaBobBall: let me know if it helps17:10
*** yamahata has joined #openstack-infra17:10
* anteaya has to leave soon for errands which will take most of the afternoon17:11
*** markus_z has quit IRC17:12
fungiclarkb: btw i'm testing on master tip of everything... is it possible the experimental job loop you saw was specific to a particular change?17:12
clarkbfungi: I thought so initially so I made sure to check all of the glusterfs runs that were going and every one did it17:13
*** teran_ has joined #openstack-infra17:13
*** afazekas has joined #openstack-infra17:13
clarkbfungi: so I think it is unlikely related to a specific unmerged change17:13
clarkbthere were ~5 running across 4 projects iirc17:13
fungiahh, yeah then must have been endemic to the job17:14
fungiright now it's on tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_update_server_set_empty_name17:14
*** teran has quit IRC17:14
*** ihrachyshka has joined #openstack-infra17:15
fungii've got free and df looping once a second in a separate ssh session to the worker, and am also trying to maintain an active virtual console session to it in case there's a kernel panic17:16
*** pradk has joined #openstack-infra17:19
*** Ala has quit IRC17:20
*** teran has joined #openstack-infra17:20
*** cdent has quit IRC17:21
*** pcrews has joined #openstack-infra17:21
fungiokay, it's on object_storage tests now17:21
fungiso hopefully rsn17:22
clarkbfungi: this is where I saw the others stop17:22
*** teran__ has joined #openstack-infra17:22
fungiit's running very close to 8gb active ram, about to dip into swap i think17:23
krotscheckI'm pondering removing the apache2 vhost from puppet-storyboard, because it presumes to know how you want to set things up.17:23
krotscheckAny opinions?17:23
*** teran_ has quit IRC17:23
clarkbkrotscheck: what do you have in mind to replace it with?17:23
fungikrotscheck: we could move that into system-config's openstack_project module instead17:23
clarkboh I see17:24
clarkbya you could decouple storyboard deployment from deploment of a proxy for storyboard17:24
fungibut i sort of like the idea that installing the module gives you a working _something_ you can then proceed to extend as needed17:24
clarkbor maybe ship a manifest that you can include that uses some relatively sane defaults but otherwise doesn't force you into using it17:24
fungiyeah, i think move the vhost handling into a separate optional class?17:24
*** cdent has joined #openstack-infra17:24
clarkbfungi: ya17:24
*** teran has quit IRC17:24
krotscheckYeah, what fungi just said makes the most sense to me.17:25
*** jcoufal has quit IRC17:25
fungithen a deployer can choose to use it or substitute their own17:25
krotscheckThat also neatly solves my http/https problem.17:25
*** bknudson has joined #openstack-infra17:26
*** cdent has quit IRC17:26
fungiclarkb: actually the example failure got to volume snapshot and instance resize tests17:26
*** teran_ has joined #openstack-infra17:27
*** teran__ has quit IRC17:27
*** jistr has quit IRC17:28
*** MaxV has quit IRC17:30
*** teran__ has joined #openstack-infra17:30
*** dansmith has quit IRC17:30
fungikrotscheck: though if you want to make http vs https configurable, have a look at my proposed puppet-zuul change (not quite the same, it allows you to enable/disable https in addition to http, but the mechanisms you need to switch that stuff are similar)17:30
*** dansmith has joined #openstack-infra17:30
BobBallclarkb: So I've got the values set in zuul/zuul.conf now (, I've restarted zuul and rebuild the jobs but I'm getting Environment variable 'SWIFT_logs_LOGSERVER_PREFIX' not found17:31
BobBallWhere should I be looking to debug why zuul isn't setting this up like you expect?17:31
*** bdpayne has joined #openstack-infra17:31
fungiand now it's past the cli and scenario tests and into the thirdparty boto tests17:31
clarkbBobBall: oh I think you ma actually need to enable it in layout.yaml per job as well (or with a regex that matches everything)17:31
*** teran_ has quit IRC17:32
*** jlibosva has quit IRC17:32
clarkbBobBall: sorry I forgot about that17:32
BobBallahhh perfect! Many thanks again :)17:32
* BobBall is getting there...17:33
fungiclarkb: hrm, i guess tempest test order is nondeterministic so trying to track progress there is likely not tractable17:33
clarkbfungi: ya17:33
*** teran has joined #openstack-infra17:33
*** teran__ has quit IRC17:34
*** unicell has joined #openstack-infra17:34
fungibeen running for ~95 minutes now17:34
*** derekh has quit IRC17:34
*** Shrews has quit IRC17:34
*** patrickeast has joined #openstack-infra17:34
mtreinishfungi: heh, well normally you can since the preseeding is a crapshoot on whether it does anything or not17:35
jogoanyone else having issues with gerrit's responsiveness?17:35
mtreinishthen it's pseudo alphabetical order17:35
fungimtreinish: nice!17:35
*** ldnunes has quit IRC17:35
*** krotscheck has quit IRC17:36
clarkbanteaya: on 157330 would you like to see those changes enough to -1 instead of +2? otherwise I think I will approve it17:36
mtreinishfungi: do you have a console output somewhere?17:38
*** ihrachyshka has quit IRC17:38
*** hdd has joined #openstack-infra17:38
fungimtreinish: in my ssh session17:38
mtreinishah, ok17:38
zaroclarkb: i don't see how asselin change ( helps with getting '/p/' in review-dev.o.o URL to work.  would you be able to enlighten me?17:39
*** krotscheck has joined #openstack-infra17:39
sigmavirus24Is anyone else noticing that review.o.o is sluggish today?17:40
fungisigmavirus24: sdague was noticing it17:40
*** teran_ has joined #openstack-infra17:40
mtreinishfungi: fwiw, 95 min is a very long time (if you're running in parallel)17:40
BobBallOK - one more question... When running a job kicked by zuul, jenkins does offline my node, but nodepool doesn't delete it.  It doesn't seem to go into the Used state either, just sits there "Ready".  Any ideas?17:40
zarosigmavirus24: me too17:40
clarkbzaro: by default /p is served by gerrit itself so the normal proxying to gerrit makes /p work. However for us we want /p to bypass gerrit and hit the git repo replica on /var/lib/git. This requirse running a git cgi script so apache needs to be told that it is ok to do that17:40
* sigmavirus24 wonders if we'll ever find the memory leak causing this17:40
fungimtreinish: this is the glusterfs job we turned off last night17:40
clarkbzaro: for 155208, should we build 2.8 and 2.9 instead of just 2.9?17:41
*** teran has quit IRC17:41
*** Shrews has joined #openstack-infra17:41
fungisigmavirus24: no signs of packet loss or excessive latency between your system and review.o.o?17:41
*** bhuvan has joined #openstack-infra17:41
sigmavirus24fungi: everything else seems to be just fine I haven't done a lot of investigation yet though17:41
clarkbfungi: no I am pretty sure its related to gerrits JVM heap size17:41
clarkbfungi: let me link the melody graph again17:41
*** carl_baldwin has joined #openstack-infra17:42
*** gondoi is now known as zz_gondoi17:42
fungilog-start-timestamp.txt says i started running it at 16:01:57 utc17:43
*** markus_z has joined #openstack-infra17:43
zarofungi: can you check the gerrit queue to see if there are errors?17:43
fungii want it to hurry up and break so i can go get lunch17:43
fungizaro: 107 tasks, lots of pending push/upload17:44
*** ParsectiX has quit IRC17:44
*** ParsectiX has joined #openstack-infra17:44
* StevenK peers at zuul17:44
clarkbI think the whole thing gums up as it runs into memory pressure17:44
*** timcline has quit IRC17:44
clarkb is the week graph17:44
clarkbmean use has been in steady increase since monday17:45
*** timcline has joined #openstack-infra17:45
*** unicell has quit IRC17:46
*** unicell has joined #openstack-infra17:46
*** markus_z has quit IRC17:46
krotscheckHrm, this puppet module is starting to look like it needs a more well developed conf manifest :/17:46
*** markus_z has joined #openstack-infra17:47
zaroclarkb: good point about keeping the javamelody 2.8 build.  but i don't see any development on it after the upgrade.17:47
zarofungi: that doesn't seem too bad.  are those mostly from replication?17:48
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/shade: Revamp README file
clarkbzaro: I am more thinking to that we have it if necessary (rollback etc)17:49
zaroclarkb: also wouldn't it get a little unwiedly if we start adding more plugins?17:49
clarkbzaro: then we remove 2.8 when we are happily on 2.917:49
clarkbzaro: I don't think so17:49
fungizaro: the git push ones are (that's about half of them) but not sure if the git-upload-pack tasks (roughly the other half) are replication as well17:49
*** BobBall is now known as BobBall_AWOL17:49
*** timcline has quit IRC17:50
fungithough now the queue has dropped to 53 tasks so it does seem to be working through them17:50
clarkb that graph is a lot more telling17:50
clarkbzaro: just instantiate the jobs multiple times with different parameters17:50
openstackgerritMichael Krotscheck proposed openstack-infra/puppet-storyboard: Added new OAuth Parameters
openstackgerritMichael Krotscheck proposed openstack-infra/puppet-storyboard: Added a Vagrant module and manifest for easier development.
zaroclarkb: i don't see why you would need it for rollback since the built version is still gonna be on tarballs17:51
fungiclarkb: score! no kernel panic (yet) but the console is now unresponsive to carriage returns and the df i had running in a loop is hanging17:51
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/shade: Revamp README file
krotschecknibalizer: I'm trying to find some docs on best practice for module development - like, how to break out all the various config options into something shared.17:52
krotschecknibalizer: Any pointers?17:52
clarkbfungi: I have never been this excited for brokeness before :)17:52
clarkbzaro: its more a matter of, we haev it today, so it is probably useful, therefore why remove it until we don't need it anymore17:52
fungiclarkb: unfortunately, i have no additional detail, so this doesn't help a bunch17:52
*** HeOS has quit IRC17:52
*** salv-orlando has joined #openstack-infra17:52
clarkbfungi: at least it is reproduceable :)17:53
zaroclarkb: ok, i have no arguement against that except that it adds code where not really necessary and requires slightly more maintenance going forward.17:53
fungiclarkb: the ip address still responds to icmp echo but attempting to open a tcp socket to port 22 times out17:54
zaroclarkb: i'm happy to update the change if you feel we should keep it around though.17:54
fungiclarkb: and both my existing ssh sessions seem to have hung17:54
clarkbfungi: anything on the console? eg panic?17:54
mkodererhi folks, could someone add steveg and me to telcowg-usecases-core?17:54
funginope, though it's back to responding to carriage returns ...very...slowly17:55
*** SumitNaiksatam has joined #openstack-infra17:55
nibalizerkrotscheck:  eh not really17:55
clarkbmkoderer: sure, give me a minute17:55
nibalizerkrotscheck: you're in seattle tho17:55
mkodererclarkb: thx :)17:55
nibalizerso sit with crinkle17:55
fungiclarkb: it's too bad i can't send sysrq through rackspace's virtual console17:55
*** Ryan_Lane has joined #openstack-infra17:56
clarkbmkoderer: you are added but there is no 'steveg' account17:56
clarkbmkoderer: you should be able to add whomever you like at this point though17:56
mkodererclarkb: ah great :)17:56
fungiclarkb: i'll have to try again to recreate this after lunch and leave myself a console backdoor open. this vm isn't likely to give us much else useful :/17:57
clarkbfungi: ok thanks for the help17:57
mkodererclarkb: ok worked thx again17:57
openstackgerritMichael Krotscheck proposed openstack-infra/puppet-storyboard: Added a Vagrant module and manifest for easier development.
openstackgerritMichael Krotscheck proposed openstack-infra/puppet-storyboard: Added new OAuth Parameters
zarofungi: are you referring to ssh hang to review.o.o?17:57
fungii lacked the foresight to create a working password login so i could do more than just play with the virtual console login prompt17:57
clarkbzaro: no17:57
fungizaro: no, completely unrelated problem we've been trying to identify all morning17:58
clarkbzaro: the review.o.o slowness appears to be a memory leak in gerrit17:58
clarkbzaro: over time we use up the entire 8GB heap and things grind to a halt17:58
fungiokay, i'm going out to the burrito bar. back in a while17:58
krotscheckfungi: I hope that's a bar made out of burrito's17:59
fungithankfully the weather has finally subsided to where we can leave the house again17:59
zaroclarkb: nothing's changed for a while. so memory leak must be caused by something right?17:59
fungikrotscheck: it's a burrito made out of bars. steel bars17:59
StevenKfungi: Sounds rather hard to digest, really.17:59
* krotscheck passes fungi the tums.17:59
clarkbzaro: this has been going on for a while too17:59
zaroclarkb: any info from javamelody?17:59
fungiStevenK: it's an australian recipe18:00
StevenKWoo, zuul queue length is now >1,00018:00
clarkbzaro: yes javamelody is how I have narrowed it down to memory pressure but haven't been able to narrow it from there18:00
* fungi will bbiaw18:00
StevenKfungi: Surely Australians would be using vegemite jars and beer cans?18:00
clarkbalso cpu pegs but I think that is because the garbage collector is doing more work18:00
*** bswartz has quit IRC18:01
*** ivar-lazzaro has joined #openstack-infra18:03
clarkbzaro: also restarting the service corrects the problem18:04
*** bswartz has joined #openstack-infra18:04
*** AJaeger has joined #openstack-infra18:05
*** timcline has joined #openstack-infra18:07
*** olaph has joined #openstack-infra18:07
*** e0ne has quit IRC18:08
*** avozza is now known as zz_avozza18:09
openstackgerritAndreas Jaeger proposed openstack/requirements: Require openstack-doc-tools 0.22
*** harlowja has joined #openstack-infra18:10
*** jgallard__ has quit IRC18:10
*** doug-fish has joined #openstack-infra18:10
*** belmoreira has quit IRC18:11
*** sdake_ has joined #openstack-infra18:12
clarkbjavamelody reports that we are spending 10% of our time garbage collecitng now18:13
clarkband that is a recent change18:13
clarkbwill link that graph for people that have access shortly18:13
asselinclarkb,^^ " Forbidden access" for me18:15
*** teran has joined #openstack-infra18:15
*** teran has quit IRC18:15
olaphsame here18:15
clarkbasselin: ya melody is restricted to gerrit admins because it does a few things that are potentially scary18:15
*** teran has joined #openstack-infra18:15
clarkbthough I can probably screenshot the graph and share that18:15
*** ParsectiX has quit IRC18:16
*** ParsectiX has joined #openstack-infra18:16
*** sdake_ has quit IRC18:17
*** otter768 has joined #openstack-infra18:18
*** teran_ has quit IRC18:18
*** otter768 has quit IRC18:22
clarkbzaro: olaph asselin ^18:22
clarkbzaro: any other ideas on how to debug this? I am gessing we will need a dump of the heap to figure out where the memory is being used18:25
zaroclarkb: yeah, dump might help18:27
*** andreaf_ has joined #openstack-infra18:29
*** prad has quit IRC18:30
clarkbgreghaynes: commented on your ssh key in dib change. needs small puppet fix18:31
*** achanda has joined #openstack-infra18:31
*** pradk is now known as prad18:31
*** cnesa1 has joined #openstack-infra18:35
*** cnesa has quit IRC18:35
*** sputnik13 has joined #openstack-infra18:35
bcourtthis might be a silly question, how does nodepool validate that an image built correctly?  Is it checking the return code of the prepare node script?18:37
clarkbbcourt: it has two bits that do that. First is image building where it checks the return code of the prepare node script. Second is the ready node script which it runs post boot and also checks the return code on18:39
clarkbbcourt: so if nodeool cannot ssh into a node and subsequently run whatever ready script it has successfully that node will be deleted and a new one booted18:39
bcourtclarkb, thanks18:40
*** EmilienM is now known as EmilienM|afk18:40
*** hdd has quit IRC18:41
*** ashleighfarnham has quit IRC18:41
*** dizquierdo has quit IRC18:42
*** ashleighfarnham has joined #openstack-infra18:42
greghaynesclarkb: im guessing puppet doesnt mind a trailing end comma?18:42
greghaynesbecause I also have that going on18:42
clarkbgreghaynes: it does not (at least not in puppet3 it did way back when)18:42
openstackgerritgreghaynes proposed openstack-infra/project-config: Respect NODEPOOL_SSH_KEY in prepare-node for dib
*** melwitt has joined #openstack-infra18:43
*** BharatK has quit IRC18:43
nibalizerogod greghaynes is writing poopit?18:44
* nibalizer watches for other signs of the apocalypes18:44
*** Sukhdev has joined #openstack-infra18:47
*** sdake_ has joined #openstack-infra18:47
tchaypojeblair: i *finally* managed to invoke gertty’s search functionality18:47
greghaynesnibalizer: most certainly not18:48
tchayponibalizer: have you looked at the weather map of the eastern seaboard?18:49
tchaypomy hometown had an earthquake last week and is expecting a hurricane tomorrow18:49
*** teran has quit IRC18:50
*** MaxV has joined #openstack-infra18:50
dstufftsdague: do you just ant a commit message like "stevedore depends on this but doesn't declare it, so let's declare it" ?18:52
*** ashleighfarnham has quit IRC18:53
openstackgerritDan Prince proposed openstack-infra/project-config: Remove check job from tripleo-puppet-elements
sdaguedstufft: sure18:54
*** ssam2 has quit IRC18:54
*** yolanda has quit IRC18:57
openstackgerritJames Polley proposed stackforge/gertty: Expand sample keymaps to ameliorate OSX features
*** salv-orlando has quit IRC18:57
*** yolanda has joined #openstack-infra18:57
openstackgerritDonald Stufft proposed openstack/requirements: Add setuptools to global requirements
dstufftsdague: ^^18:58
*** dims_ has joined #openstack-infra18:59
dstufftdhellmann: ^^18:59
openstackgerritDan Prince proposed openstack-infra/project-config: Update to use
*** dims__ has quit IRC19:02
openstackgerritJames Polley proposed stackforge/gertty: Print keymap command names correctly
*** bswartz has quit IRC19:03
*** yfried|afk has quit IRC19:04
*** e0ne has joined #openstack-infra19:06
*** ghostpl_ has quit IRC19:06
*** r-daneel has joined #openstack-infra19:10
*** achanda has quit IRC19:11
*** mestery has joined #openstack-infra19:12
*** TravT_ has joined #openstack-infra19:12
*** Guest21555 has quit IRC19:14
*** Hal has joined #openstack-infra19:14
*** Hal is now known as Guest4210219:15
zaroclarkb: Just a guess, it might be sshd causing memory leak, similar to ,  dump might narrow it down.19:15
*** dustins has quit IRC19:15
TravT_Hello infra: if anybody has some time to look at a couple of requirements patches we have up. Would appreciate it.19:17
TravT_we're running a "virtual sprint" in horizon19:17
TravT_where everybody comes together for several hours on code reviews19:17
TravT_and the patches that need a couple of requirements are coming up.19:18
*** MarkAtwood has quit IRC19:18
*** r-daneel has quit IRC19:19
zaroclarkb: Gerrit 2.8 is using sshd ver 0.11.0, Gerrit 2.9 reverted to using ver 0.9.019:19
*** Guest42102 has quit IRC19:19
*** sarob has joined #openstack-infra19:19
AJaegerTravT_: infra is just the fallback for requirements, see here for list of cores:,access19:19
TravT_thanks, also figured dropping a note in here might help.19:20
*** HeOS has joined #openstack-infra19:23
*** MaxV has quit IRC19:23
*** markus_z has quit IRC19:23
*** bswartz has joined #openstack-infra19:24
*** dims_ has quit IRC19:26
greghayneshuh, my loal nodepool dib image is failing to build because the 90-venv-swift-logs install.d needs virtualenv but this isnt installed19:26
greghaynesdo we not have an element that does this?19:26
openstackgerritMichael Krotscheck proposed openstack-infra/puppet-storyboard: Added a Vagrant module and manifest for easier development.
asselinanyone seen this nodepool exception deleting a node?
*** vigneshvar has joined #openstack-infra19:28
otherwiseguySo, all neutron patches seem to be failing on gate-neutron-pylint all of a sudden.19:29
fungiotherwiseguy: new version of pylint maybe?19:29
*** tiswanso has joined #openstack-infra19:30
*** dustins has joined #openstack-infra19:30
fungihrm, no new pylint for a month19:30
fungiclarkb: interesting update, i left my hung ssh sessions going while lunching. on returning i found they'd recovered19:30
*** abhirc has quit IRC19:31
fungiclarkb: devstack-gate _did_ eventually successfully timeout and kill tempest19:31
otherwiseguyLooks like a change in novaclient or something maybe.19:31
* otherwiseguy investigates19:31
*** BobBall_AWOL is now known as BobBall19:32
fungiclarkb: oom19:32
fungiclarkb: Killed process 6448 (mysqld) total-vm:4797460kB, anon-rss:228456kB, file-rss:0kB19:32
*** salv-orlando has joined #openstack-infra19:32
fungiotherwiseguy: i have to wonder why neutron pylint jobs would be checking novaclient source19:34
openstackgerritJames Graf proposed openstack-infra/jenkins-job-builder: Add support for Job DSL plugin for Groovy-based job and view creation and management
*** achanda has joined #openstack-infra19:34
otherwiseguyneutron has a notifier that uses novaclient19:34
BobBallWhat should be setting nodepool nodes to USED?  Zuul successfully runs a job on a node, but it's not transitioning to USED then to deleting - presumably because the onStarted messages aren't sent and/or received, but I don't see where they are sent from and why they might not be being sent.19:35
*** mpavone has joined #openstack-infra19:35
fungiotherwiseguy: oh, is pylint failing in some way other than pointing out lint-specific issues?19:35
otherwiseguyNo name 'client' in module 'novaclient.v1_119:35
fungilike hitting an import failure or something?19:35
fungiyeah, that seems likely then19:35
otherwiseguyNo name 'contrib' in module 'novaclient.v1_119:35
fungiso the pylint run isn't so much failing ans completely breaking19:36
otherwiseguyfrom lines:19:36
otherwiseguy18 import novaclient.v1_1.client as nclient19:36
otherwiseguy 19 from novaclient.v1_1.contrib import server_external_events19:36
fungiprobably was a novaclient release today19:36
fungiyep, see and -dev ml thread19:37
fungi2.21.0 release today19:37
otherwiseguywhy would the v1_1 stuff change with a new release?19:37
* otherwiseguy sighs19:37
otherwiseguySo close to getting everything merged. No more 20-patch-long chains for me. :p19:38
*** krtaylor has quit IRC19:38
*** tiswanso has quit IRC19:38
fungiotherwiseguy: i would suggest asking the novaclient devs for details (though a quick hack through git diff 2.20.0..2.21.0 and some selective blaming would get you a commit message and possibly bug/spec link with details)19:38
*** vigneshvar has quit IRC19:38
*** tiswanso has joined #openstack-infra19:39
*** TravT_ has left #openstack-infra19:39
otherwiseguy"Rename v1_1 to v2" for a patch name sounds ominous.19:40
fungiif that was a public api, then leaving v1_1 aliased to that would have been nice19:41
*** mrunge has joined #openstack-infra19:41
otherwiseguyfungi: yes, yes it would have been.19:41
fungiotherwiseguy: might be a quick patch and 2.21.1 brown-bag release away19:42
fungii suggest following up to the release announcement on the -dev ml19:42
openstackgerritDan Prince proposed openstack-infra/project-config: Actually test tripleo-puppet-elements
*** pelix has quit IRC19:44
*** mestery has quit IRC19:44
fungialso, mikal ^ if you're around, see otherwiseguy's shiny new brokenness19:44
clarkbfungi interesting19:46
clarkbI am running errands for sick wife should be able to help debug in a while19:46
fungiclarkb: take your time. it will keep19:46
clarkbBobBall nodepool listens to the jenkins zmq event publisher19:46
BobBall*sigh* somehow that had been disabled.  That'll be the issue then.  Thanks again clarkb.19:47
fungiBobBall: sounds like you need to take away someone's mouse finger19:48
mordredfungi: wait - what?19:48
mordredthey renamed v1_1 to v2?19:48
fungimordred: what the what what?19:48
* mordred reading scrollback19:48
fungimordred: yeah, but not in anything widely used, just you know novaclient19:48
fungiand apparently at least some feature of neutron (master at a minimum) was importing from it19:49
fungii guess not in devstack-deployed scenarios, so only tripped on neutron's pylint job19:50
fungihence not noticed until a new novaclient release happened19:50
*** erlon has quit IRC19:51
dhellmanndstufft: +2a19:52
mordredfungi: well, also - I USE THAT CODE PATH19:52
fungimordred: not any more you don't19:52
mordredmikal, sdague, jogo: ^^^ I would like to complain about your breakage of library19:52
fungi(well, until they fix it hopefully)19:53
otherwiseguyfungi: is it possible to temporarily disable neutron's pylint test voting?19:54
mordredmikal, sdague, jogo: ^^^ it turns out that infra uses that code path19:54
*** yamamoto has joined #openstack-infra19:55
mordredand I find removing it from a widely used library yuck19:55
otherwiseguymordred: it looked like they just modify sys.modules directly to alias it, which pylint doesn't see?19:55
fungiotherwiseguy: yeah, you could override it in neutron's tox.ini to just always return true, or submit a patch to openstack-infra/project-config to override it to voting:false in zuul/layout.yaml19:55
mordredotherwiseguy: oh, how clever19:56
mordredotherwiseguy: sigh19:58
*** yamamoto has quit IRC19:59
openstackgerritTerry Wilson proposed openstack-infra/project-config: Disable voting on pylint because of novaclient breakage
otherwiseguyfungi: ^ like that?19:59
* otherwiseguy has no idea what he is doing20:00
fungiit's entirely possible that this could also be considered a on pylint's part20:00
mordredok - I will scream less - I have tested that the code paths work20:00
mordredalthough now they throw deprecation warnings20:01
mordredit would be great if someone had an example of the "right" way20:01
*** dims__ has joined #openstack-infra20:01
fungiotherwiseguy: also further down in the openstack/neutron entry for the projects list take it out of the gate pipeline20:01
*** bswartz has quit IRC20:01
*** crc32 has joined #openstack-infra20:02
fungiotherwiseguy: this way it runs non-voting in the check pipeline and not at all in the gate pipeline20:02
openstackgerritTerry Wilson proposed openstack-infra/project-config: Disable voting on pylint because of novaclient breakage
*** r1chardj0n3s_afk is now known as r1chardj0n3s20:03
openstackgerritMonty Taylor proposed openstack-infra/shade: Import from v2 instead of v1_1
vipulquestion about global-requirements.. is there a reason why python-openstackclient isn't there?  noticed most of the other clients exist20:05
fungivipul: what other projects use python-openstackclient?20:05
otherwiseguymordred: looks like
*** dims__ has quit IRC20:05
fungivipul: that is, what is it a requirement of?20:05
otherwiseguymordred: I'm guessing get_client_class()20:05
*** crc32 has quit IRC20:06
vipulfungi: also...
openstackgerritMonty Taylor proposed openstack-infra/nodepool: Change references from v1_1 to v2
mordredotherwiseguy: awsome20:07
fungivipul: weird. what do they use openstackclient for? just avoiding depending on the python sdk (novaclient, neutronclient, et cetera)?20:07
*** r1chardj0n3s has left #openstack-infra20:07
vipulfungi: they are implemented as 'plugins' to python-openstackclient rather than introducing their own shell20:08
*** dprince has quit IRC20:08
fungivipul: that seems like it gets the dependency backwards then... you're saying they don't use openstackclient, but openstackclient uses them?20:09
mordredno - they're right20:09
mordredif you want to install cueclient20:09
mordredyou install it20:09
*** ghostpl_ has joined #openstack-infra20:09
mordredand it installs an _additional_ optional plugin for python-openstackclient20:09
*** MaxV has joined #openstack-infra20:09
fungioh, so it's still invoked ac cueclient, not as a plugin to openstackclient20:09
mordredthat is then consumed by running openstackclient20:09
mordredyou run openstackclient20:09
vipulessentially it allows you to run 'openstack <your plugin command>'20:09
mordredbut if you install the cueclient plugin, you get more commands in openstackclient20:09
mordredso openstackclient doesn't depend on it20:09
mordredbut cueclient is useless without openstackclient20:10
fungii mean, the same could be said of any optional dependency i guess20:10
mordredit's like "apt-get install firefox-plugin-lightning" which you'd expect to wind up also installing firefox if it's not installed20:10
vipulmordred: great example !20:11
fungiyep, that analogy works20:12
vipulshall i submit a patch? fungi20:12
fungiso this goes back to the "should we include dependencies only requested by stackforge projects in the openstack projects global requirements list?" debate20:13
mordredfungi: I honestly think python-openstackclient is fair game for requirements20:13
fungivipul: feel free to submit a patch. i don't generally vote on requirements patches though so my opinion should be taken with a grain of salt20:13
mordredwithout having to go down the rabbithole20:13
vipulshould stackforge projects be allowed to plugin to devstack?20:13
*** ghostpl_ has quit IRC20:13
*** mestery has joined #openstack-infra20:13
*** ghostpl_ has joined #openstack-infra20:13
vipulthis is where this dependency is needed.. when setting up devstack20:13
vipulfungi: will do20:14
*** aysyd has quit IRC20:14
fungivipul: we have soft requirements where unofficial projects can set a flag that lets them include requirements which aren't in the official global requirements list20:14
fungiin devstack-based jobs20:14
vipulsomething i can set in my local.conf?20:15
fungiit basically turns off devstack's complaint about using a package which isn't in global requirements20:15
* fungi gets the details20:15
*** krtaylor has joined #openstack-infra20:16
*** dprince has joined #openstack-infra20:16
fungilooking for an example job definition to point you at now20:16
vipulfungi: awesome i'll try that.. thx for the help20:16
*** otter768 has joined #openstack-infra20:19
fungiyeah, REQUIREMENTS_MODE defaults to "strict" in devstack-gate but can be overridden from the calling environment, and then passes that into devstack which also defaults to "strict" but bypasses that extra requirements check if it's "soft" instead20:19
fungihowever i don't see any current jobs passing that into devstack-gate, in theory it's an implemented and working feature20:19
*** tkelsey has quit IRC20:20
*** rfolco has quit IRC20:22
*** ociuhandu has quit IRC20:23
*** otter768 has quit IRC20:23
*** abhirc has joined #openstack-infra20:25
*** achanda has quit IRC20:28
openstackgerritMerged openstack-infra/project-config: Disable voting on pylint because of novaclient breakage
AJaegerotherwiseguy: once pylint passes again, please propose a revert of the patch20:29
otherwiseguyAJaeger: will do. Also have a patch that should adapt to the new novaclient change pending in the neutron queue.20:29
AJaegerotherwiseguy: great!20:30
otherwiseguyI haven't run even basic testing on it, and have no idea if I did things right because I pretty much just wrote and submitted to see what would happen.20:30
otherwiseguy(the neutron way?) /js20:30
otherwiseguyer, 's20:30
* otherwiseguy gives up20:31
*** ihrachyshka has joined #openstack-infra20:32
*** achanda has joined #openstack-infra20:32
jogomordred: looking into it20:37
*** MaxV has quit IRC20:37
*** hdd has joined #openstack-infra20:37
otherwiseguyweird, disconnected right as I sent that message. Guest79?20:39
*** tkelsey has joined #openstack-infra20:42
*** bswartz has joined #openstack-infra20:42
openstackgerritmbitard proposed openstack-infra/jenkins-job-builder: add slack notification support, largely inspired by hipchat module
*** ddieterly has quit IRC20:44
*** Sukhdev has quit IRC20:46
*** e0ne has quit IRC20:47
jogomordred: we didn't break anything novaclient.v1_1 still works,cm20:47
*** jedimike has quit IRC20:47
jogomordred: its just pylint isn't smart20:48
fungijogo: yeah, it turns out it just wasn't backward-compatible in a way which pylint could follow20:48
mikalmordred: I find the tone of your conversation intriguing20:48
clarkbok mostly back now20:48
jogofungi: silly pylint20:48
*** ZZelle_ has joined #openstack-infra20:48
fungijogo: i like to anagram it as "pintly"20:49
*** ddieterly has joined #openstack-infra20:49
*** arosen has quit IRC20:50
*** yamahata has quit IRC20:51
fungijogo: a cursory review of novaclient commit subject lines since the prior release turned up a change which looked like it was moving v1_1 to v2 which would have explained the pylint error, but closer examination revealed that it was still aliased to the new name and pylint was merely a mendicant20:51
*** mpavone has quit IRC20:52
*** w_ has joined #openstack-infra20:52
*** dkranz has quit IRC20:52
*** gyee has joined #openstack-infra20:53
mikaljogo: but we still managed to make it sound like the nova team is incompetent, just to keep it real20:53
mordredmikal: I'd like to sell you some real estate20:53
*** sdake_ has quit IRC20:54
clarkbhrm if I were to make a heap dump of gerrit I doubt I could share it because secrets20:54
clarkbalso I doubt that I will have the patience to dig through it to find a memory leak20:55
fungiclarkb: yeah we'd probably need to spend quite a lot of time to filter than down and sanitize/redact things20:55
mriedemoops, disregard20:56
clarkbso I am leaning towards just restarting it, but can figure out jmap if someone really wants that heap data and its safe to share it with them20:56
clarkbfungi: jeblair mordred jhesketh SergeyLukjanov pleia2 ^20:56
*** sdake_ has joined #openstack-infra20:56
fungiclarkb: i wouldn't know what to do with it, but if it's easy to dump and stash for now then we can always trot it out later to answer more directed questions if there's a gerritite willing to help us troubleshoot post-mortem20:57
clarkbwait people willingly use pylint?20:57
clarkbfungi: well I think it will make an 8GB binary file20:57
fungiclarkb: yes, neutron (recentlyish i believe) added it back20:57
clarkbfungi: which would then have to be sanitized20:58
*** arosen has joined #openstack-infra20:58
*** ZZelle__ has joined #openstack-infra20:58
*** ZZelle__ has quit IRC20:58
clarkband jmap isn't in my path on review.o.o20:58
clarkbzaro: ^ is jmap what I should be using? the internet says it is but I am a noob at jvm debugging tools20:59
fungiclarkb: if memory serves the reasoning was that pyflakes' static analysis didn't cover some of the problems deeper inspection via pylint pointed out20:59
clarkbfungi: ya but pylint also points out a lot of things that aren't issues20:59
clarkbits false positive rate is incredibly high20:59
BobBallfungi: Did you get to the bottom of the EOF issue from earlier?  I'm hitting it too on my Ubuntu nodes - testing with Xen.  During the devstack setup script it dies20:59
dansmithclarkb: yeah I was thinking "holy crap, voting pylint in a gate?"20:59
fungiand also to achieve that deeper analysis it has to do things like import modules outside of normal python mechanisms21:00
fungihence... this21:00
fungiBobBall: eof problem what now?21:00
*** crc32 has joined #openstack-infra21:00
anteayaclarkb: nah, you can go ahead on
BobBallfungi: SSH session just dying causing an exception log in jenkins21:01
clarkbanteaya: thanks21:01
BobBallfungi: it was deepakcs problem with the glusterfs jobs21:01
fungiBobBall: for glusterfs?21:01
fungiBobBall: the held worker i ran it manually on eventually recovered, though the kernel oom killer did shoot mysqld in the head along the way21:01
openstackgerritmbitard proposed openstack-infra/jenkins-job-builder: add slack notification support, largely inspired by hipchat module
BobBallHmmmz - interesting...21:02
*** sdake_ has quit IRC21:02
otherwiseguyclarkb: yeah, neutron gates on pylint...after disabling at least 50 tests.21:02
BobBallAlso interesting I just noticed that nodepool failed to kill the nodes that seem to be dying... saying they don't exist.  Which sound highly fatal and interesting to me...21:02
*** sdake_ has joined #openstack-infra21:03
clarkbsupposedly jmap comes from the jdk, we must only install jre. I am confirming that. There is an option to have signal 3 create a heap dump which I may enable post restart for next time this happens21:03
fungii started devstack-gate at 16:01:57 and the kernel killed mysqld at 18:47:08 so that may have been unrelated breakage after the fact21:03
BobBallah no - I misread the log - it was the node in jenkins that had disappeared21:03
*** unicell has quit IRC21:04
*** unicell has joined #openstack-infra21:04
* BobBall holds a node to try and get some debug info out21:04
clarkbya we install the openjdk-7-jre-headless package so no jmap. My vote is for not doing a heap dump today, but add the flag to the jvm that will do a heap dump on signal 3 when we restart gerrit21:04
clarkbI will write the puppet change for that now21:04
fungiahh, nope, timing works out. mysqld oom'd about 4 seconds before the devstack-gate-cleanup-host.txt log starts21:05
BobBallAh - yes.  I bet mine is an OOM too.21:06
jogomordred: btw did you ever re-fix the pbr/setuptools issue with creating binaries21:06
BobBall*fixes dom0 RAM*...21:06
*** achanda has quit IRC21:06
BobBallI wonder if we should give the OOMkiller some hints not to kill SSH21:06
*** sabeen2 has joined #openstack-infra21:06
jogomordred: a release with  in it that is21:07
BobBallbut perhaps to kill the d-g-wrap script first which should let us extract logs perhaps?21:07
mordredNo. Thanks for reminder21:07
clarkbBobBall: at one point we did use cgroups for that. not sure if those are still in use21:07
*** sabeen has quit IRC21:07
clarkbBobBall: basically set a lower memory threshhold for the safe processes allowing ssh to survive21:08
*** mpavone has joined #openstack-infra21:08
*** sdake_ has quit IRC21:09
*** doude has quit IRC21:10
*** doude has joined #openstack-infra21:10
openstackgerritMerged openstack-infra/project-config: Clone using git.o.o instead of directly from gerrit
*** mpavone has quit IRC21:13
*** abhirc has quit IRC21:13
openstackgerritClark Boylan proposed openstack-infra/puppet-gerrit: Collect Gerrit heap dumps on signal 3
clarkbzaro: fungi ^ something like that21:14
clarkbI should probably test that on review dev first though21:14
clarkbzaro: is it ok for me to use review-dev for some testing of that?21:15
AJaegerclarkb, fungi: If either of you have some spare review cycles, I would like to unblock the training-guides folks:
AJaegercould you review it, please?21:15
clarkbAJaeger: sure21:16
AJaegerThere's on +2 already - and I did the last change, so don't want to +2 myself ;)21:16
AJaegerthanks, clarkb!21:16
openstackgerritKhai Do proposed openstack-infra/project-config: Add a job for Gerrit javamelody plugin ver 2.9
*** abhirc has joined #openstack-infra21:17
jogowee cells broke the gate21:18
*** markmcclain has quit IRC21:18
openstackgerritJoe Gordon proposed openstack-infra/project-config: Stop making devstack-dsvm-cells voting
jogomriedem alaski: ^21:19
jogofungi clarkb: ^ that should unwedge the world21:20
*** mbacchi has quit IRC21:20
clarkbhahahahahahahaha I figured out why my puppet env is borken21:20
clarkbI fetched puppet-gerrit into system-config21:21
clarkbsuch shame21:21
clarkbjogo: why not revert the nova change that broke cells?21:21
jogoclarkb: we don't know what it is yet21:21
jogoplus that job should really go bye bte21:21
jogoplus that job should really go bye bye21:21
*** timcline has quit IRC21:21
*** ghostpl_ has quit IRC21:22
*** mbacchi has joined #openstack-infra21:22
clarkbthen delete the job don't make it non voting...21:22
openstackgerritJoe Gordon proposed openstack-infra/project-config: Stop making devstack-dsvm-cells voting
openstackgerritMatt Riedemann proposed openstack-infra/project-config: Stop making devstack-dsvm-cells voting
clarkbzaro: ok I don't hear any nays to me restarting review-dev with those jvm options21:23
clarkbzaro: I am going to restart it shortly if I don't hear otherwise21:23
alaskijogo: why do you want to remove that job?  It works as a good smoke test for cells21:23
jogoalaski: first things first21:23
jogoalaski: lets make it non-voting to unwedge everything21:23
jogothen we can have the to remove or not discussion21:24
*** achanda has joined #openstack-infra21:24
*** signed8bit has quit IRC21:24
*** signed8bit has joined #openstack-infra21:24
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck: Add query for cells bug 1423695
openstackbug 1423695 in OpenStack Compute (nova) "gate-devstack-dsvm-cells fails attaching volume" [Undecided,New]
*** AJaeger has quit IRC21:25
openstackgerritMerged openstack-infra/project-config: Updates training guides jobs
*** mrmartin has quit IRC21:26
* clarkb restarts review-dev21:27
pleia2oof, between conference and related events, and the time change, I've been totally not here all week21:27
anteayapleia2: how goes the long sleeves in hot weather set?21:28
*** mbacchi has quit IRC21:28
pleia2anteaya: they have amazing air conditioning, and it's a dry heat, so when we do go outside it's ok21:28
*** ddieterly has quit IRC21:28
anteayahow is the food?21:28
*** ddieterly has joined #openstack-infra21:28
clarkbwoo my change totally makes gerrit not start :)21:29
pleia2very indian-influenced, so it's good21:29
clarkbzaro: any idea why?21:29
jogoandreaf: ping, see my comment on
anteayapleia2: sounds yummy21:29
anteayapleia2: when do you fly again?21:29
pleia2anteaya: it is :)21:29
*** dims__ has joined #openstack-infra21:29
pleia2my first flight is in 4 hours21:29
anteayayou are in an airport then, yes?21:29
anteayahow are the dates and figs there?21:30
pleia2nah, just wrapping up at my hotel, heading over in an hour21:30
pleia2dates and figs are good!21:30
anteayaI bet21:30
anteayathat would be what I would consider a highlight21:30
pleia2and they drink a lot of juice here21:30
anteayado they21:30
anteayawhat kind?21:30
pleia2mango, orange, pineapple21:30
anteayasounds tasty21:30
pleia2it's a dry country, so most places where we'd have alcohol are replaced with juice, no complaints from me :)21:31
*** MaxV has joined #openstack-infra21:31
anteayaI have no problem with that21:31
*** mbacchi has joined #openstack-infra21:31
pleia2anyway, I shall finish packing and will be back next week to catch up on things21:31
anteayasafe flights21:31
pleia2thanks :)21:32
*** morganfainberg is now known as needscoffee21:32
nibalizerpleia2: don't forget your charger this time21:33
jogoclarkb: can you take a second look at
jogoas this is hurting a lot of projects21:35
clarkbjogo: sure, ajaegers -1 applies still21:35
*** sabeen2 has quit IRC21:35
jogoclarkb: kk will update21:36
openstackgerritJoe Gordon proposed openstack-infra/project-config: Stop making devstack-dsvm-cells voting
*** BobBall is now known as BobBall_AWOL21:39
openstackgerritMerged openstack-infra/elastic-recheck: Add query for cells bug 1423695
openstackbug 1423695 in OpenStack Compute (nova) "gate-devstack-dsvm-cells fails attaching volume" [Undecided,New]
*** achanda has quit IRC21:40
*** doude has quit IRC21:40
*** doude has joined #openstack-infra21:40
clarkbjogo: almost one small change21:40
*** ivar-laz_ has joined #openstack-infra21:41
openstackgerritJoe Gordon proposed openstack-infra/project-config: Stop making devstack-dsvm-cells voting
anteayasdague: any objection to this change merging? the stable python26 jobs going from 40 to 60:
*** bswartz has quit IRC21:42
clarkbjogo: +221:43
*** ivar-lazzaro has quit IRC21:43
openstackgerritDan Prince proposed openstack-infra/project-config: Remove wasteful tripleo workspace-cache cloning
*** ihrachyshka has quit IRC21:44
jogoclarkb: thanks!21:44
*** andreykurilin_ has joined #openstack-infra21:45
*** ivar-laz_ has quit IRC21:46
*** achanda has joined #openstack-infra21:46
*** ivar-lazzaro has joined #openstack-infra21:46
*** sarob has quit IRC21:46
*** ivar-lazzaro has quit IRC21:47
*** jp_at_hp has quit IRC21:48
*** ivar-lazzaro has joined #openstack-infra21:48
clarkbjhesketh: I would totally try to make zuul things happen but I am wrestling java and gerrit21:48
jheskethclarkb: no worries, all good :-). Maybe there's something I could help with?21:49
clarkbjhesketh: it appears that gerrit 2.8 has a memory leak. I have discovered that there are in theory options to the jvm that will cause it to dump a heap dump on signal 321:50
clarkbbut applying those options to review-dev makes gerrit daemon not start21:50
clarkbmy current plan is to figure out how to run gerrit with those options so that after we restart to kill the current leak we have debugging options fr next time21:50
anteayajhesketh: morning21:51
*** dkranz has joined #openstack-infra21:52
*** wenlock has quit IRC21:53
*** jaypipes has quit IRC21:53
jheskethclarkb: okay. I don't know much about java but if there is something specific I can do let me know :-).21:53
*** oomichi has joined #openstack-infra21:54
clarkbjhesketh: will do, maybe I should recheck the dg jobs right now and see if you can get in there with the 2 hour window?21:54
clarkbjhesketh: since its relatively low overhead for me to start a job and hold a node21:54
clarkbok HeapDumpOnCtrlBreak is an unrecognized option21:55
*** wenlock has joined #openstack-infra21:55
clarkbapparently its an old option and no longer supported21:56
jheskethclarkb: up to you. Maybe keep your current state and I'll deal with email and other stuff instead of distracting you21:56
clarkbjhesketh: well I think I am about to give up on this. I think our only other option is to install the jdk instead of jre to get jmap21:57
clarkbfungi: what do you think about just restarting gerrit now? then we can worry about switching to a jdk install later?21:59
clarkbjeblair: mordred jhesketh ^21:59
fungiclarkb: i'm fine with it. need help?22:00
fungii'm mostly just tweaking and rerunning random non-devstack jobs on held devstack nodes to improve my baseline for the image convergence22:00
clarkbfungi: I shouldn't just want to make sure that others are on board. Will restart it now22:01
*** zz_gondoi is now known as gondoi22:01
clarkbfungi: maybe do a statusbot alert?22:02
*** patrickeast_ has joined #openstack-infra22:02
anteayagerrit is down for me22:02
greghayneshrmmm, gerrit 50322:02
anteayadid you restart?22:02
clarkbyou guys crack me up22:02
greghaynesIm glad we can entertain :)22:02
*** dannywil_ has joined #openstack-infra22:02
anteayawhy do a status bot alert then22:03
anteayaafter the restart22:03
clarkbanteaya: because I just restarted it22:03
*** MaxV has quit IRC22:03
anteayastatus that 503? that was me22:03
fungiit's working for me! ;)22:03
clarkbanteaya: I am confused22:03
greghaynesYes, it has returned22:03
clarkbanyways /me digs up statusbot syntax22:04
*** patrickeast has quit IRC22:04
*** patrickeast_ is now known as patrickeast22:04
*** dannywilson has quit IRC22:04
anteayawell usually for me a status bot is effective before the restart22:04
*** marun has quit IRC22:04
anteayanot sure of the value after the restart22:04
clarkbanteaya: to let people know why it happened so that I don't get a flood of "is gerrit down?" questions22:05
clarkblooks like #status notice Gerrit was restarted to correct memory pressure due to what appears to be a memory leak22:06
clarkbhow does that sound?22:06
anteayaI can live with that22:06
fungiclarkb: lgtm22:06
fungiand yeah, you want notice so that it's a non-stateful message with no topic editing22:07
*** carl_baldwin has quit IRC22:07
clarkb#status notice Gerrit was restarted to correct memory pressure due to what appears to be a memory leak22:07
*** vponomaryov has quit IRC22:07
*** enikanorov_ has joined #openstack-infra22:07
clarkbno statusbot?22:07
*** viktors has quit IRC22:08
*** rlandy has quit IRC22:08
clarkbor is it #notice? this bot doesn't have docs and can never remember what to execute /me should fix this22:08
fungino, openstackstatis is mia apparently22:09
*** enikanorov has quit IRC22:09
fungier, openstackstatus22:09
*** carl_baldwin has joined #openstack-infra22:09
fungialso it's been long enough now i'd say just fix statusbot and no bother sending the notice ;)22:09
openstackgerritSpencer Krum proposed openstack-infra/infra-specs: Spec to refactor puppet code in openstack_project
jheskethclarkb: restarting the dev Gerrit or live?22:10
anteayaand done22:11
*** vponomaryov has joined #openstack-infra22:11
jheskethCool :-)22:11
openstackgerritMerged openstack-infra/project-config: Remove check job from tripleo-puppet-elements
clarkbfungi: statusbot is running on eavesdrop I guess it just left this channel? restart the whole daemon?22:11
clarkbseems like you tpically kick that bot22:11
*** bswartz has joined #openstack-infra22:12
*** viktors has joined #openstack-infra22:12
openstackgerritJoe Gordon proposed openstack-infra/devstack-gate: Skip volume devstack exercise
fungii just use the initscript to restart it22:12
fungion eavesdrop22:12
*** mrunge has quit IRC22:13
openstackgerritgreghaynes proposed openstack-infra/project-config: Allow ability to prevent static dns resolver
fungi2015-02-07 22:50:33     <--     openstackstatus ( has quit (Ping timeout: 245 seconds)22:13
fungiso it's been gone... a while22:13
anteayait should send postcards22:13
clarkbfungi: kk doing that now22:13
anteayaI try to when I am gone that long22:13
fungieven my vacation next week won't rival that bot's time off22:14
anteayafungi: where are you next week?22:14
anteayaor would you prefer not to say?22:14
*** koolhead17 has quit IRC22:14
fungiget away from some of the ice and wind here22:14
anteayabeach sand sun22:14
anteayagood call22:14
openstackgerritgreghaynes proposed openstack-infra/project-config: Allow ability to prevent static dns resolver
anteayashark's teeth?22:14
clarkbfungi: you can see the drop in memory use post restart thee22:14
*** openstackstatus has joined #openstack-infra22:15
*** ChanServ sets mode: +v openstackstatus22:15
anteayai remember collecting shark's teeth in florida as a kid22:15
fungiclarkb: yep, classic leak (or cache, but if that's a cache it's a terrible one)22:15
fungianteaya: yeah, they have those. also sharks22:15
anteayaokay maybe leave the sharks alone?22:15
clarkbfungi: not just the teeth? :P22:15
*** dims__ has quit IRC22:15
fungiclarkb: oh, they have both22:16
*** tmcpeak1 has quit IRC22:16
*** sabeen has joined #openstack-infra22:16
clarkbok  Ithink gerrit is a happier camper now22:16
clarkbjhesketh: ok where were we on swift logs?22:16
*** hdd has quit IRC22:17
clarkbjhesketh: I'm thinking maybe we just change the timeout on the jobs for now? that avoids a zuul restart and the gate is thrashing so I would like to avoid making that worse22:18
openstackgerritJamie Lennox proposed openstack/requirements: Update version of neutronclient
jheskethclarkb: okay, fair enough, give me a minute22:19
*** otter768 has joined #openstack-infra22:19
jogoclarkb: an alternate unbreak the world patch
clarkbjogo: looks like they are narrowing in on an actual fix though22:20
*** tmcpeak has joined #openstack-infra22:20
jogoclarkb: hopefully. at this point its a race22:20
jogoclarkb: ideally we get a fix out first. but just in case22:21
anteayajogo: I can't weigh in on this patch until dansmith and alaski get behind it:
jogothe clock is ticking on a half dozen projects are blocked22:22
anteayaor at least change their -1's to 0's22:22
clarkbthey are actively debugging so I am happy to let them do that and probably remove the job only from ceilometer22:22
clarkb(based on what dansmith said)22:22
dansmithclarkb: thanks22:22
jogoanteaya: there is an alternate proposal22:22
*** ghostpl_ has joined #openstack-infra22:22
dansmithclarkb: from what we can tell, it's an actual novaclient bug that we can't hit any other way than this exercise22:22
dansmithclarkb: just happens that we only run those on the cells job22:22
clarkbdansmith: neat22:22
dansmithclarkb: so leaving it is the right thing to do for the moment, IMHO22:22
openstackgerritMerged openstack-infra/project-config: Add non-voting job for Nova V2.1 API with neutron
openstackgerritJoshua Hesketh proposed openstack-infra/project-config: Update swift upload expiry for all jobs
jheskethclarkb: ^22:24
*** otter768 has quit IRC22:24
jogodansmith: well whoever gets there first the fix or this skip passes check
clarkbjhesketh: +2, maybe we can get fungi behind that too then I can go about triggering and holding nodes and all that22:24
*** ociuhandu has joined #openstack-infra22:25
*** oomichi has quit IRC22:26
*** signed8bit has quit IRC22:26
*** ghostpl_ has quit IRC22:27
*** signed8bit has joined #openstack-infra22:28
openstackgerritMerged openstack-infra/jenkins-job-builder: Fix broken link on documentation
*** mbacchi has quit IRC22:31
harlowjajeblair dhellmann was i think wanting to change the owner of i think u can do that right?22:31
* harlowja not sure why i set that as the owner (guess i did, ha)22:31
clarkbharlowja: what should it be?22:32
harlowjaclarkb i think 'Oslo Drivers'22:33
*** mriedem has quit IRC22:33
clarkbharlowja: done22:33
*** ghostpl_ has joined #openstack-infra22:35
openstackgerritMerged openstack-infra/project-config: Enable non-voting keystoneclient functional tests
openstackgerritJoshua Harlow proposed openstack/requirements: Add openstack/debtcollector to projects.txt
*** medberry2 is now known as med_22:37
*** med_ has quit IRC22:37
*** med_ has joined #openstack-infra22:37
*** signed8bit has quit IRC22:38
openstackgerritDouglas Mendizábal proposed openstack-infra/project-config: Add devstack gate to barbicanclient
openstackgerritMerged openstack-infra/project-config: Add pypi jobs for stackforge/networking-cisco
*** pc_m has quit IRC22:43
openstackgerritMerged openstack-infra/project-config: Update swift upload expiry for all jobs
anteayaclarkb: am I reading your comment correctly? the value that currently is 115 should be 125, yes?
*** Alexandra_ has joined #openstack-infra22:45
*** hyakuhei has quit IRC22:45
hogepodgefungi irc operator error22:46
zaroclarkb: did you get review-dev working with special VM args yet?22:46
*** MarkAtwood has joined #openstack-infra22:47
*** camunoz has joined #openstack-infra22:47
clarkbzaro no they arent supported so I reverted22:47
openstackgerritVipul Sabhaya proposed openstack/requirements: Add python-openstackclient to global-requirements
zaroclarkb: not supported on jvm version?22:48
clarkbanteaya yes22:48
*** hyakuhei has joined #openstack-infra22:48
clarkbzaro ya according to my testing java didnt like those args22:48
zaroclarkb: where did you find those args?22:50
ekarlsowhat's up with stackforge projects now being enforced with globla requirements ?22:50
fungihogepodge: i consider defcore/refstack concerns to be equally on-topic in #-qa, so either was fine by me ;)22:50
anteayaclarkb: thank you, I will comment22:50
fungiekarlso: what do you mean?22:51
clarkbzaro on a blog that took me to oracle docs22:51
*** thingee has joined #openstack-infra22:51
clarkbbut we dont use oracle jvm and I think the args were removed in favor of jmap22:51
fungiekarlso: stackforge projects can both be global requirements and can have global requirements forced on them, but whether they can request additions to the global requirements list is still debated i think22:52
*** kgiusti has left #openstack-infra22:52
*** hyakuhei has quit IRC22:54
*** jgrimm is now known as zz_jgrimm22:55
clarkbfungi can you review jheskeths change above?22:56
*** hyakuhei has joined #openstack-infra22:56
clarkbwill help the swift log upload performance debugging22:56
*** yamamoto has joined #openstack-infra22:57
*** jedimike has joined #openstack-infra22:58
*** claudiub has quit IRC22:58
*** andreaf_ has quit IRC23:00
fungiclarkb: the one that already merged, or something else?23:00
clarkboh it merged ignore me23:01
clarkbtoo much going on23:01
*** marun has joined #openstack-infra23:01
fungiyeah, i approved it a while back23:01
fungiso, um, done! ;)23:01
anteayaha ha ha23:01
*** yamamoto has quit IRC23:02
openstackgerritDouglas Mendizábal proposed openstack-infra/project-config: Add devstack-gate job to barbicanclient
openstackgerritDouglas Mendizábal proposed openstack-infra/project-config: Add a devstack-gate job to barbicanclient's tests
*** dannywil_ has quit IRC23:04
*** abhirc has quit IRC23:04
ekarlsofungi: funky that bit, cause it didn't use to be like that23:05
fungiekarlso: which bit?23:06
*** marun has quit IRC23:06
ekarlsothe hard requirement enforcement :p23:06
clarkbits always been that way...23:06
ekarlsoit use to work like a few days or weeks back23:06
ekarlsorly with devstack ?23:06
clarkbekarlso: I think we are talking past each other on what "it" is23:06
fungiekarlso: yeah, i'm not following23:07
clarkbglobal requirements has always been applied to all projects in its project list23:07
clarkband requirements have been enforced23:07
*** hichihara has joined #openstack-infra23:07
ekarlso 'python-openstackclient' is not in global-requirements.txt23:07
*** sarob has joined #openstack-infra23:07
*** crc32 has quit IRC23:07
ekarlsowhen python-cueclient is depending on it23:07
fungiekarlso: why would it be?23:07
ekarlsothat used to work before :/23:07
dtroyerit was just removed in the last day or so.23:07
ekarlsodtroyer: rly, why ?23:08
dtroyerbecause nothing 'should' depend on it23:08
clarkbjhesketh: I just rechecked a change and will hold a couple nodes from that23:08
clarkbjhesketh: should runwith the 4 hour expiry23:08
dtroyerapparently a couple of clients with OSC plugins do though.  I'd argue that they should not so that as it is really optional23:08
ekarlsodtroyer: isn't openstackclient the way to go for cli forwards kinda ?23:08
jheskethclarkb: thanks :-)23:09
fungiekarlso: if python-cueclient is a stackforge project, it can use the REQUIREMENTS_MODE=soft option to devstack-gate/devstack allowing it to install additional requirements which aren't in the global requirements list23:09
dtroyerif you just want the congressclient libs, you would otherwise have to install OSC and _all_ of its dependencies, which may not be what you want23:09
fungidtroyer: the counterargument from congressclient and cueclient is "i want to pip install this thing that's a plugin to openstackclient, so it should automatically install openstackclient for me when i do"23:10
dtroyerIOW, OSC and the various client lins with OSC plugins should not have each other in their requirements as they are both useful without the other23:10
clarkbjhesketh: do you think I/we should add better logging to the threadpull change? we should be able to use python's logging lib to get synchronized log writes across the threads23:10
*** MarkAtwood has quit IRC23:11
needscoffeemordred, you know if I stick with this name instead of morganfainberg, no more tab-doh,wrongperson23:11
*** bkero has quit IRC23:12
needscoffeefor "mor-<tab>"23:12
*** MarkAtwood has joined #openstack-infra23:12
clarkbfungi: does nodepool look funny to you?23:12
dtroyerekarlso, fungi: I'm not going to get in their way from putting OSC into their requirements.txt, but I think it the tradeoff of having to install two pacakges with pi vs being required to install 23 deps is a better user experience23:12
clarkbfungi: the graph indicates we aren't building nodes. I am tailing logs now23:12
fungiclarkb: no, looks like zuul stopped running jobs though?23:12
ekarlsodtroyer: +123:12
openstackgerritJoshua Harlow proposed openstack-infra/devstack-gate: Add debtcollector to
fungiclarkb: i think nodepool "caught up" now that zuul seems to have ceased starting new jobs23:13
*** Guest936 has joined #openstack-infra23:13
clarkboh good point23:13
clarkbjhesketh: so is it possible that config change broke teh zuul?23:13
*** Guest936 is now known as annegentle23:13
anteayafungi: do you consider this patch mergable or are you still working on it?
clarkbAttributeError: 'int' object has no attribute 'format'23:14
jheskethclarkb: possibly not a bad idea, although any errors should raise up to the main thread23:14
fungianteaya: it's mergeable, though if nobody approves it i may extend it to test more broadly. otherwise i'll extend with new changes23:14
*** hichihara has quit IRC23:14
anteayavery good23:15
clarkbjhesketh: I wonder if that string should be quoted23:15
jheskethclarkb: :-(23:15
clarkbjhesketh: let me try that really quick by hand23:15
jheskethclarkb: should we revert?23:15
anteayaneedscoffee: yes easier for tab complete, keep this one23:15
needscoffeeanteaya, haha23:15
anteaya#vote or23:15
needscoffeeanteaya, troooooooollll ;)23:15
*** yamamoto has joined #openstack-infra23:15
*** prad has quit IRC23:16
clarkbjhesketh: no I think quotes fixes it23:16
anteayayou opened the door23:16
fungijhesketh: clarkb: yep, timing and the error message indicate that it is indeed that23:16
clarkbjhesketh: so just '14400' unless the swift stuff needs it as an int23:16
jheskethclarkb: looking at there is int(time() + settings['expiry']), so I'm not sure if a string will screw that up23:16
jheskethor if the layout stuff reads it in as an int23:16
fungiyeah, i think without wrapping like int(settings['expiry']) that will asplode23:17
clarkboh then ya we should revert23:17
jheskethclarkb: I'll revert the change to layout23:17
clarkbso I think a better way to handle this is where the exception is thrown coerce to a string first23:18
clarkbI will write that change23:18
openstackgerritJoshua Hesketh proposed openstack-infra/project-config: Revert "Update swift upload expiry for all jobs"
jheskethclarkb: ^23:19
jheskethclarkb: okay so you're going to look into fixing zuul? Otherwise I can23:19
*** crc32 has joined #openstack-infra23:19
*** mjturek1 has quit IRC23:19
clarkbya I think I see a simple fix already23:19
fungii'm mildly surprised the layout parser doesn't hate that23:19
fungiseems like it would be opinionated about what datatype it's getting there23:20
*** weshay has quit IRC23:20
clarkbI think the layout may expect an int23:20
openstackgerritMichael Krotscheck proposed openstack-infra/puppet-storyboard: Parameterized working directories.
openstackgerritMichael Krotscheck proposed openstack-infra/puppet-storyboard: Added storyboard working_dir config option to correct block.
openstackgerritMichael Krotscheck proposed openstack-infra/puppet-storyboard: Moved all module parameters into one manifest.
fungiwell, except doesn't the voluptuous check run on config reloading too?23:21
fungiso i expect you'd have seen it complain when you manually edited the file too23:21
jheskethyeah the layout expects an int23:21
fungii need to spend some time rereading that i guess23:22
jheskethit's more that the code tries to substitute things on all variables rather than just the strings23:22
*** carl_baldwin has quit IRC23:22
*** gondoi is now known as zz_gondoi23:23
clarkbya but I think coersion to string is what we want anyways since they end up as env vars right?23:23
*** yamamoto has quit IRC23:24
clarkbanyways waiting for tox to finish to avoid any embarassing format issues but will push up what I think is the simple fix23:24
jheskethclarkb: I think for the expiry we want to coerce it at the hmac23:24
jheskethotherwise adding time.time() to it won't work23:24
anteayafungi: what is the reason the new job is experimental in nova?
jheskethsee lib/ line 15123:24
anteayaas opposed to an infra repo23:25
fungianteaya: because it's a nova job?23:25
clarkbjhesketh: I see23:25
fungianteaya: selecting nova as probably one of the more complex/needy projects from a unit testing perspective (exercises database configuration on the worker as well as some system package expectations)23:26
anteayafungi: but that is just the title of it, right? there is nothing else that is specific to nova, is there?23:26
*** bhuvan has quit IRC23:26
anteayafungi: oh okay, yes that makes sense23:26
jheskethclarkb: although expires is substituted into the string with %s which is surprising23:26
anteayaI just wasn't seeing anything nova specific23:26
jheskethif you're fixing that up, it should probably be %d23:26
fungianteaya: it checks out nova source code and runs tox -e py27 basically23:26
anteayait just installs packages and sets up the dbs23:27
jheskethclarkb: let me know if you'd prefer I made the changes23:27
clarkbjhesketh: %s basically just does str() on the thing first iirc23:27
jogoclarkb: so we broke icehouse which means d-g is wedged too23:27
jogoclarkb: so your welcome :)23:27
jheskethclarkb: ah, okay23:27
fungianteaya: the next step (whether in that same change or in a new change) will be to extend that so i can add it as several experimental jobs to basically all projects23:27
clarkbjhesketh: so I think we don't coerce in launchers/gearman then, instead maybe just only do the format() call if v is a string type?23:27
clarkbjhesketh: since there shouldn't be a reason for expiry to have that format called on it23:28
jheskethclarkb: that was my thinking23:28
anteayafungi: okay yes in the experimental-workers.yaml23:28
fungianteaya: but nova provides a good initial proving ground that the mechanics of the macros work (i've already tested the scripts they contain so more worried about making sure parameter substitution and worker node label selection work correctly)23:28
clarkbjhesketh: ok I have things open should have this done soon23:28
anteayafungi: right that is what I was expecting, hence my question about nova23:28
anteayaokay thanks23:28
anteayaI understand now23:29
krotschecknibalizer: So, working on puppet-storyboard, I'm trying to figure out a good way to notify an optional module. Have you seen something like that before?23:29
anteayaI'll wait until -nova has fixed their fire23:29
anteayathen bug them23:29
anteayathe look close23:29
fungiokay, so having built a _full_ package install list from our thick slave class manifest, installing those (mostly locally cached) at job run time adds ~1m11s. once i add bindep hooks, projects can bypass that by providing a more specific bindep manifest to just install the packages they really need23:29
*** dannywilson has joined #openstack-infra23:30
fungibut once i extend it to cover more than just ubuntu-trusty and also get the same package lists unioned to the lists we're pre-caching via nodepool prep scripts/dib elements, we'll be almost there23:31
nibalizerkrotscheck: the optional thing can subscribe to the required thing23:31
krotschecknibalizer: BUT THAT WOULD MAKE SENSE.23:31
krotschecknibalizer: :)23:31
krotscheckSo clearly I can't do it.23:31
openstackgerritClark Boylan proposed openstack-infra/zuul: Only format swift string parameters
clarkbjhesketh: ^ something like that?23:31
krotschecknibalizer: How would I then tell service['http'] (from an extrnal module) to subscribe to something that I'm doing inside of mine?23:32
*** yamamoto has joined #openstack-infra23:32
anteayafungi: makes sense23:32
clarkboh wait the fix won't hrm23:32
openstackgerritMiro Hroncok proposed openstack-infra/project-config: Add Python 3 template for openstack/heat-cfntools
clarkbfungi: jhesketh so zuul stopped running jobs so our fix won't merge :) I will stop puppet on zuul and apply by hand then reenable puppet once it goes through23:33
greghaynesclarkb: do we care about python3 in that location?23:33
clarkbgreghaynes: no zuul isn't py3k friendly23:34
jheskethclarkb: if zuul stopped running jobs won't it need a manual restart?23:34
clarkbjhesketh: well its "stopped" basically was in a tight loop on that error, removing the error allowed it to continue23:35
clarkbjobs are starting now23:35
clarkbbut will keep an eye on it just in case it does need a restart for some reason23:35
jheskethoh right, puppet kept putting the bad config back in23:35
clarkbjhesketh: right23:35
jheskethokay, sounds good then23:36
clarkbso puppet is disabled, I removed the line, and HUP'd the service23:36
clarkbonce your change gets in I will reenable puppet and we should be good23:36
openstackgerritDouglas Mendizábal proposed openstack-infra/project-config: Add a devstack-gate job to barbicanclient's tests
clarkbDeficit: devstack-trusty: 0 (start: 293 min: 293 ready: 312 capacity: 8) looks much better on the nodepool side23:36
clarkbjhesketh: I think this would happen with the global change as well23:37
clarkbso good thing we caught it just with the simple config change :)23:37
openstackgerritBrant Knudson proposed openstack-infra/project-config: Add bandit jobs
*** thingee has quit IRC23:38
*** salv-orlando has quit IRC23:39
*** bdpayne has quit IRC23:39
clarkbjogo: ok now that I have caught up. we intend on making the icehouse and juno jobs for dg non voting23:39
clarkbjogo: pleia2 was working on that I thought I had reviewed the change too23:39
jogoclarkb: cool, I spoke too soon -- neutron broke it not nova23:39
jogoclarkb: a better upper limit is needed in icehouse for neutronclient23:40
clarkbjogo: also it would be good if you could expand in your email. What you linked does not explain anything as far as how these things are related23:40
clarkbpython-keystone is the issue23:40
*** andreykurilin_ has quit IRC23:40
jogoclarkb: yeah I should explain but I am sick of cleaning up after people23:40
*** salv-orlando has joined #openstack-infra23:40
clarkbjogo: but no one else knows how to clean up bceause there is no info...23:41
clarkbjogo: also this sort of makes my point about capping vs pinning :)23:41
jheskethclarkb: actually I'm not sure it would because the defaults are looked up in lib/ and aren't formatted23:41
clarkbwe aren't any better so meh23:41
*** amitgandhinz has quit IRC23:41
clarkbjhesketh: but those don't appear to be the defaults from the ini file?23:42
* clarkb double checks23:42
jheskethclarkb: line 149 it fetches from the ini file23:42
clarkboh nevermind you are right23:42
*** carl_baldwin has joined #openstack-infra23:43
jogothey broke semver23:43
fungii guess the good semver solution would be to revert that patch, make a new micro point release, then restore it and have a version bump?23:44
clarkbjogo: I don't think they did23:44
clarkbjogo: requiring a different version of keystoneclient isn't a backward incompatible change23:44
jogoclarkb: wild dependency jumps23:44
clarkbits only becuse you have decided to pin everthing :)23:45
clarkbbut I don't think that makes it a backward incompat change. Its just not compatible with how you ahve chosen to do requirements23:45
*** carl_baldwin has quit IRC23:45
jogoclarkb: actually no. it was from this Id5a53eaf3da86932999d62e51d07f7015b0ab53a23:46
*** tmcpeak has quit IRC23:47
fungii tried to embark on a similar solution for folsom/grizzly and didn't get nearly this far because we didn't have good requirements sync across projects, but could forsee even then that trying to do that given how pypi handles dep resolution was going to be fraught with peril23:47
clarkbjogo: I don't follow the leap there either23:47
*** dimtruck is now known as zz_dimtruck23:47
jogoclarkb: that is where OSC was pinned. which is what is breaking23:47
clarkbright but its breaking because ou have pinned keystone clietn and neutronclient did a release that didn't respect that pin23:48
clarkbwhich would be fine if the pins in general weren't in place23:48
clarkbanyways this is why I have been wary of the pins. I think it will lead to just as much work and pain23:49
*** markvoelker has quit IRC23:49
*** hyakuhei has quit IRC23:49
openstackgerritMichael Krotscheck proposed openstack-infra/puppet-storyboard: Apache Config is now an optional module.
clarkbcaps on the other hand give you enough flexibility to work around with things when they shift within reason23:49
jogoclarkb: well neutron client has a cap <2.423:49
jogonot a pin23:49
jogosame with OSC23:49
krotschecknibalizer: ^^ is the relevant code. I'm not certain where I can fit the subscription though. Wouldn't mind your feedback23:50
jogoits not a hard '=='23:50
clarkbjogo: what about keystoneclient?23:50
clarkbkeystoneclient is the thing at fault here23:50
jogoclarkb: I think its safe to say there is no easy answer for any of this23:50
clarkbjogo: yes that is what I have said :)23:50
tchaypoah, this is still going, excellent23:50
clarkbjogo: which is why I have had a preference to just deal with status quo since its all bad and least effort23:50
jogoclarkb: capped,uncapped,pinned or other23:50
*** Alexandra_ has quit IRC23:51
clarkbbut am open to trying other things, we just have to realize there is pain in every direction we turn23:51
jogoclarkb: this was a a status quo break. my cap all stable/juno didn't impact this23:51
fungiwhen i started to try, i concluded that it's really not particularly tractable even with requirements sync (which we thankfully now have) unless someone has available time to help get a smart dep solver into pip23:52
clarkbjogo: you didn't also cap/pin icehouse?23:52
clarkbjogo: I thought you did both23:52
*** dustins has quit IRC23:53
lifelessfungi: harlowja wrote one, does it have to be /in/ pip ?23:53
jogoclarkb: nope. I didn't realize it was voting on anything besides stable/icehouse and juno.23:53
*** mriedem has joined #openstack-infra23:53
jogoclarkb: forgot about devstack-gate23:53
*** yamamoto has quit IRC23:53
harlowjai start writing one, its still sorta WIP :-P23:53
fungilifeless: i suppose not. i guess we can replace pip23:53
*** bknudson has quit IRC23:53
harlowja has a few idea(s)23:54
fungiharlowja: my understanding is it just turns pip into a downloader and handles the dep resolution on its own?23:54
fungisounds reasonable, though i can already see the ensuing shitstorm from "here's how you pip install openstack projects _now_"23:55
harlowjathe current code in that repo uses pip for downloading, expands the full dependency graph, then creates 'requirement' trees (where each path in that tree is one solution for a requirement)23:55
clarkbjhesketh: ssh jenkins@ for and ssh jenkins@ for they are running anwys so may as well try to use them?23:55
*** oomichi has joined #openstack-infra23:55
*** tkelsey has quit IRC23:55
harlowjafungi ya, anythings possible :-P23:55
harlowja (the main routine); feel free to hack on that as much as u want23:56
clarkbhonestly the right place to fix this *is* pip23:56
clarkbbecause you will install other python things too23:56
fungii mean, we're already partway there with declaring that we just don't support easy_install, so saying you can't use straight pip to install this stuff either is only likely to alienate the remaining parts of the python community we haven't already23:56
jheskethclarkb: awesome, thanks!23:56
lifelessfungi: we don't need to change anyone elses behaviour to fix the gate23:56
*** abhirc has joined #openstack-infra23:56
*** ddieterly has quit IRC23:57
fungilifeless: only those people who wish to have test results similar to ours, or installations similar to what we tested23:57
clarkblifeless: you do if ou expect to map your tests onto what people are doing real world23:57
fungiyou're right about that23:57
clarkbya what fungi just said23:57
fungibut honestly, pip doesn't guarantee much of that to start with23:57
jheskethclarkb: I have to wait for the job to finish to get the build params23:57
harlowjau can all join!topic/pypa-dev/YqgoE1ZLpqg :-P23:58
clarkbjhesketh: and to have it generate all the artifacts23:58
harlowja'Dependency solver (working group?)' (my post)23:58
*** dannywilson has quit IRC23:58
dstufftoh goody23:58
dstufftwhat did pipd o now23:58
fungiharlowja: i applaud this effort ;)23:58
fungidstufft: it convinced people to start pitching in on the lack of a smart dep solver23:58
fungidstufft: positive stuff23:58
clarkblooks like matthew has outlined exactly what needs to be done23:59
openstackgerritKhai Do proposed openstack-infra/project-config: Refactor Gerrit plugin build jobs
harlowjaya, and i think its also fine to have a POC somewhere that someone can get to work (whether thats pippin or not)23:59
harlowjai've got the full dep graph23:59
clarkbharlowja: POC doesn't hurt but I don't think we should use it for the reasons fungi described23:59
lifelesspip doesn't guarantee same as what we have anyway :)23:59

Generated by 2.14.0 by Marius Gedminas - find it at!