Tuesday, 2014-06-10

anteayamattoliverau: yeah, thought it was a really small deer for the longest time, the colour was right but the size was wrong00:00
anteayaand the behaviour was all wrong for a deer00:00
zarojhesketh: anyways don't list to me, we should just wait to hear from jeblair. i wouldn't be surprised if he waves his wand and POOF!00:00
anteayaI could see its chest but not its head until it moved00:00
devanandaadam_g: fwiw, there's an E-R query for 1326754 but it doesn't seem to be commenting on any failures in ironic's queues00:01
mattoliverauanteaya: amazing, you must've been in the right place at the right time!00:01
jheskethzaro: yeah, it would be quite a re-write, but yeah, curious to see what jeblair knows00:02
adam_gdevananda, hmm. has the ironic gate been getting any E-R comments? i added one a while ago and still haven't seen it comment, but i'm not sure the bug exists anymore (the NodeLocked thing)00:03
devanandaadam_g: no, none at all. I pointed that out to clarkb earlier but didn't get any traction00:03
devanandaadam_g: i've searched my inbox for all comments on all ironic patch sets in the last 7 days and didn't see a single one from E-R00:04
anteayamattoliverau: yeah, I happened upon it, I think we both were surprised but the wolf was really curious and not afraid, it held its ground00:04
anteayait was about 80 metres away00:04
anteayasecond time I ever saw a wolf00:05
adam_gdevananda, weird, its certainly being found. its tracking @ http://status.openstack.org/elastic-recheck/ but not commenting.00:06
mattoliverauanteaya: scary, I've never seen one outsde a zoo, there isn't many over this way :P00:07
anteayadevananda: there are still some unclassified failures for ironic here: http://status.openstack.org/elastic-recheck/data/uncategorized.html00:07
anteayamattoliverau: I wasn't scared until after it moved off00:07
anteayaI knew it was curious and had a feeling it was going to try to circle behind me00:07
anteayadogs and wolves do that00:07
anteayathe circle behind you and follow you00:07
*** lakshmiS has quit IRC00:08
anteayamattoliverau: how was your weekend?00:09
mattoliverauanteaya: umm, <sarcasm> that isn't scary at all </sarcasm> .. and I bet also don't run or they'll chase you!00:09
*** lcostantino has quit IRC00:09
anteayaI turned and walked toward the house and found my father, who came out with me and we walked to where the wolf had been00:10
clarkbdevananda: its not a ironic specific problem...00:10
clarkbdevananda: we are working onit00:10
anteayano, that is bears, for wolves, run00:10
clarkbdevananda: there just isn't anything to do for ironic right now00:10
fungimattoliverau: but you get dingoes instead00:10
anteayayou just stand there, they will attack anyway if they are going to attack00:10
*** zhiyan_ is now known as zhiyan00:10
mattoliverauanteaya: mine was great, the wife and I spend the long weekend in Tasmania, so we explored Hobart, drove to a move historic town, drove up mount wellington and met up with jhesketh and his lovely wife for dinner!00:10
mattoliveraufungi: yeah, but they only eat babies :P00:11
*** reed has quit IRC00:11
anteayamattoliverau: sounds like an awesome weekend00:11
clarkbfungi: I am going to start spinnnig up those new nodes shortly00:11
fungiclarkb: sounds good. i'm mostly not around fwiw00:11
mattoliverauanteaya: it was, it's a beautiful part of the world :) jhesketh is a lucky man to work from there!00:11
devanandaclarkb: what projects does E-R watch and post comments on? is there a defined list somewhere?00:12
anteayajhesketh: I hear you are a lucky man00:12
devanandaclarkb: recheckwatchbot.yaml?00:12
jheskethanteaya: Indeed I am. mattoliverau is learning to visit more often ;-)00:13
anteayadevananda: http://git.openstack.org/cgit/openstack-infra/elastic-recheck/tree/queries00:13
clarkbdevananda: it listens to everything00:13
anteayadevananda: those are the queries for all the bugs e-r knows about00:13
clarkbfungi: should be fine, just adding nodes00:13
anteayadevananda: if there is a query there, e-r knows about it, if not, it doesn't00:13
devanandaanteaya: yes, and there are several in there that match ironic bugs, but we're not seeing *any* comemnts from E-R *ever*, and i'm trying to figure out why00:13
*** mmaglana has quit IRC00:13
anteayajhesketh: :D00:13
clarkbdevananda: because we are behind00:13
devanandaclarkb: i get that. sorry if i'm just being dense, but why would e-r comment on nova but not ironic?00:14
devanandaif it's that far behind that that's preventing it from commenting00:14
clarkbdevananda: it isn't commenting on nova00:15
devanandaclarkb: it is ...00:15
clarkband if it does it just pure luck00:15
*** melwitt has quit IRC00:15
clarkbnova has more changes by an order of magnitude00:15
clarkbtherefore chances of things being indexed for random nova change are higher00:15
clarkbarg is_openstack_project isn't actually checking if things are an openstack project00:17
clarkbjogo: mtreinish ^00:18
clarkbdevananda: return "tempest-dsvm-full" in self.comment maybe part of the problem too00:18
devanandayep, that looks likely00:19
jerryzfungi clarkb: i have a nodepool that doesn't mark nodes as used if a job runs on them. does jenkins support two nodepool subscribe to zmq publisher?00:19
jesusaurusclarkb: what does is_openstack_project do?00:20
openstackgerritClark Boylan proposed a change to openstack-infra/config: Add four new logstash-worker nodes.  https://review.openstack.org/9892500:20
clarkbfungi: jhesketh SergeyLukjanov ^ please don't merge that (I will do it when I have DNS records added) but reviews would be great00:20
clarkbjesusaurus: it determines if e-r should check bugs for that project00:20
devanandajesusaurus: http://git.openstack.org/cgit/openstack-infra/elastic-recheck/tree/elastic_recheck/elasticRecheck.py#n31200:21
clarkbjerryz: nodepool can listen to two jenkins00:21
devanandaclarkb: thanks. I think this is the reason.00:21
clarkbjesusaurus: we have 800:21
clarkber jerryz ^00:21
clarkbdevananda: well right now it wouldn'00:21
clarkbt work regardless00:21
clarkbdevananda: I am adding capacity now to combat backlog00:21
jheskethclarkb: noted00:21
devanandaclarkb: that's fair. but it never worked, even during periods with no backlog00:22
*** weshay has joined #openstack-infra00:22
devanandaclarkb: filing a bug so this gets fixed at some point00:22
jerryzclarkb: yeah, but can one jenkins support two nodepools?00:22
*** gokrokve has quit IRC00:24
clarkbjerryz: oh for that probably not00:26
clarkbjerryz: actually maybe00:26
clarkbsince nodepool should just ignore cases where it doesn't knwo about a node00:26
*** matsuhashi has joined #openstack-infra00:27
clarkbsimilar to hwo we have long lived slaves00:27
clarkbnodepool should treat things the same if they came from another nodepool00:27
jerryzclarkb: ok00:27
jerryzclarkb: then my issue with nodepool not marking node as used is not caused by this00:27
clarkbprobably not00:28
clarkbis your nodepool getting the build started event from zmq?00:28
clarkbit should log that in the debug log00:28
jerryzclarkb: i will take a look. if no log, i will trigger a job00:29
devanandajogo, mtreinish: https://bugs.launchpad.net/openstack-ci/+bug/132831800:29
uvirtbotLaunchpad bug 1328318 in openstack-ci "elastic-recheck not matching ironic test runs" [Undecided,New]00:29
openstackgerritA change was merged to openstack/requirements: Add bash8 to global-requirements  https://review.openstack.org/9762500:31
*** jaypipes has quit IRC00:32
*** ArxCruz_ has joined #openstack-infra00:32
*** ArxCruz has quit IRC00:33
jerryzclarkb: i didn't see any log saying build started00:36
*** chuckC has quit IRC00:37
clarkbjerryz: so I would check your zmq event publish connectivity00:37
jerryzclarkb: netstat shows there is connection00:37
clarkbmake sure the nodepool is talking to it and that that job is configured to emit events or that the jenkins server is configured to emit them globally00:37
*** marcoemorais has quit IRC00:38
*** marcoemorais has joined #openstack-infra00:38
*** mmaglana has joined #openstack-infra00:43
jerryzclarkb: how to restart zmq publisher, just  restart jenkins? i see there are 40 connections with CLOSE_WAIT status left by a nodepool for test, which i already stopped.00:43
clarkbjerryz: yes though you might be able to get away with changing the port in the global config, running a job, then changing the port back00:45
*** aconrad_ has quit IRC00:45
clarkbI believe that will create new objects in the jvm00:45
devanandaanteaya: regarding http://status.openstack.org/elastic-recheck/data/uncategorized.html#gate-ironic-python2600:49
devanandaanteaya: my perhaps flawed understanding is that logstash doesn't index py* jobs and therefor E-R queries for them aren't helpful. Am I wrong?00:49
clarkbdevananda: it does index the console logs for those jobs00:50
*** chuckC has joined #openstack-infra00:50
devanandaclarkb: interesting. my logstash queries aren't matching them ... i'll keep trying00:50
*** yamahata has quit IRC00:52
clarkbfungi: jhesketh SergeyLukjanov ok dns records just created00:54
clarkbI am going to self approve now00:55
jheskethclarkb: noted00:55
* jhesketh waits for the world to break00:55
clarkbjhesketh: :P it should not cause anything to break00:55
clarkbworst case we have derpy indexers and don't index things00:55
jheskethheh, all good, just teasing00:56
clarkbnow back to why qemu-nbd and mount are not doing what I expect them to00:56
clarkbI just want to mount a partition in a qcow2 :(00:56
clarkbhrm maybe my qcow2 isn't good00:57
StevenKclarkb: What does file say?00:58
clarkbfile reports the correct thing00:58
clarkbbut qemu-nbd -c /dev/nbd1 pathto.qcow2 && mount /dev/nbd1 /someplace fails demanding I specify a fs type00:58
clarkbif I specify ext* it barfs00:58
clarkband fdisk on /dev/nbd1 throws up00:59
StevenKclarkb: /dev/nbd1p<num>00:59
clarkbStevenK: that doesn't exist even after a partprobe00:59
StevenKclarkb: Just to show it does work (for me at least): http://paste.openstack.org/show/83462/01:00
clarkbyou are trusty yes?01:00
StevenKThis is Saucy01:00
clarkbok this is precise01:00
clarkbI mean things should still work01:00
clarkbbut maybe nodepool will need to be upgraded too...01:01
openstackgerritDevananda van der Veen proposed a change to openstack-infra/elastic-recheck: Broaden 1326312 to match py26/py27 failures  https://review.openstack.org/9892901:01
StevenKclarkb: Give me a few, I shall try on a Precise server I have around01:01
clarkblol I use nbd0 and that gives me a p101:01
clarkbguessing someting has used nbd1 and hates me01:02
devanandaafter a bug fix lands upstream, how long should we wait to remove the e-r query?01:02
StevenKOh, what -c did you use for qemu-nbd?01:02
clarkbStevenK: first time was /nbd101:02
clarkber /dev/nbd101:02
devanandaanteaya: ^ should categorize all of those ironic py26/27 failures -- judging by the timestamps and looking at a few, I think they're all the same bug01:02
clarkbbut then I switched to nbd0 and happy now01:03
clarkbdevananda: usually we wait ~2 weeks to see te full period of indexed logs not return any hits01:03
StevenKclarkb: Strange, you might be right that something is using nbd101:03
devanandaclarkb: ack01:04
anteayadevananda: this was going to be my suggestion: message:"FAIL: ironic.nova.tests.scheduler.test_ironic_host_manager.IronicHostManagerChangedNodesTestCase.test_get_all_host_states" AND tags:"console" AND project:"openstack/ironic" AND build_queue:"gate"01:04
*** michchap has joined #openstack-infra01:04
openstackgerritA change was merged to openstack-infra/config: Add four new logstash-worker nodes.  https://review.openstack.org/9892501:06
*** aconrad_ has joined #openstack-infra01:07
jerryzclarkb: after i restarted jenkins, zmq publisher works again01:09
*** arnaud has joined #openstack-infra01:10
anteayadevananda: and I agree they are all the same bug, it seems to have occured on June 4th and then been fixed01:10
jerryzclarkb: wonder why tcp sockets piled up and didn't get closed01:10
openstackgerritDevananda van der Veen proposed a change to openstack-infra/elastic-recheck: Broaden 1286635  https://review.openstack.org/9893101:13
devanandaanteaya: that ^ should start catching a few more of the network failure related issues01:14
clarkbjerryz: probably a zmq bug01:15
*** gema has quit IRC01:17
*** jhesketh has quit IRC01:17
jerryzclarkb: though i haven't been able to reproduce it. maybe because my nodepool for test got restarted constantly01:18
*** sweston has joined #openstack-infra01:20
*** yaguang has joined #openstack-infra01:20
anteayadevananda: ah okay, I get results from that query01:22
*** ramashri has quit IRC01:22
anteayaI don't know enough to review queries yet so let's see what the reviewers say01:22
*** gema has joined #openstack-infra01:25
mikalanteaya: do you understand what Sean Dague means by "work unit testing" in his email about nova-network?01:25
mikalSorry, work unit logging01:25
*** godp1301 has joined #openstack-infra01:25
*** marcoemorais has quit IRC01:26
openstackgerritNikhil Manchanda proposed a change to openstack-infra/config: Fixed KEY_DIR for the trove-buildimage-mysql job  https://review.openstack.org/9893301:26
clarkbmikal: I think he is saying that distinct things like create ip attach ip etc aren't logged01:26
clarkbso it isn't traceable01:26
clarkbmikal: though I don;t actually know what a nova network unit of work is01:27
*** godp1301 has quit IRC01:27
clarkbthink of it is a trace though01:27
*** adalbas has joined #openstack-infra01:27
*** adalbas has quit IRC01:27
*** godp1301 has joined #openstack-infra01:27
*** marun has quit IRC01:28
anteayamikal: I'm blanking on context can I get a link to the email?01:28
*** aconrad_ has quit IRC01:28
anteayamikal: ah okay01:28
anteayamikal: I think what is happening is that nova-network is lacking logging for unit tests01:29
anteayamikal: so when nova-network unit tests fail, there is no logging, or not enough logging to debug the unit test failures for nova-network01:29
anteayawhich is understandable since it was frozen and everyone expected it to be deprecated01:29
anteayabut now we are using it in our testing and nova-network unit test failures that can't be debugged is allowing gate races to creep in01:30
fungianteaya: not unit tests, but under operation of the server. when tempest tests try to create too many instances in devstack it stops being able to reach some of them reliably. i think it got narrowed down to the default network size in nova being a 256 so when tempest tried to make more than that many instances...01:30
anteayaoh and what clark said01:30
anteayagreat, I'm glad you know more fungi01:31
mtreinishmikal: yeah look at the ssh timeout bugs we've been trying to get to the bottom of for the last few days01:31
*** arnaud has quit IRC01:31
anteayayes at one point, sdague mentioned he thought nova-network was leaking01:31
anteayaor might be01:31
mtreinishit feels like a floating ip issue but the n-net logs don't have enough details about what happening so we can't figure out where it went wrong01:31
jesusauruscan i get some reviews on https://review.openstack.org/#/c/98561/ and https://review.openstack.org/#/c/98549/01:32
*** Ryan_Lane1 has quit IRC01:33
SlickNikfungi / clarkb: https://review.openstack.org/98933 => small fix for a mistake I made while setting up the trove-image-build job. Would appreciate a look when you guys get a chance. Thanks!01:33
*** nosnos has joined #openstack-infra01:33
jerryzclarkb: also after jenkins restart, the first build event will be lost because the connection is not there yet, after this event, the connection is established and works from then on01:35
*** godp1301 has quit IRC01:35
mikalOk, so if I went through the driver interface and added info logs to each of the nova-network events, that would help?01:35
*** nati_uen_ has quit IRC01:36
ianwjesusaurus: maybe it's obvious for the people you really want to review this, but the change doesn't describe *why* you want to do that...01:36
jesusaurusianw: which change are you referring to?01:37
*** yamahata has joined #openstack-infra01:37
ianwjesusaurus: https://review.openstack.org/#/c/98561/01:37
*** jhesketh has joined #openstack-infra01:37
*** rickerc has quit IRC01:38
mtreinishmikal: it'd probably be a pretty good start. Definitely better than what's there now.01:38
*** rkukura has quit IRC01:38
*** prad has quit IRC01:38
mikalmtreinish: yeah, it might indicate where we need more specific stuff.01:39
mikalmtreinish: ok, I will take a look at that in a minute, it shouldn't be very painful to do01:39
clarkbmordred: it looks like we can't easily do the db and user setup without having the server being declared in the same puppet process01:39
clarkbmordred: we could rip out the entire DB stack into slave_db and not just the users and db stuff01:40
clarkbmordred: so I think I am leaning that direction. we *should* be able to tell puppet to only listen on socket files and not do tcp with the DBs then have a second post install action update that for final image build01:41
*** penguinRaider has quit IRC01:41
harlowjaso a quick question for someone with more experience with testr than me, if i want to say see all the log output that a single cinder tests creates (assume the test does not fail) using testr is there anyway to have testr (or cinder) absorb all the log files and hide them away (somewhere?)01:42
clarkbharlowja: yes they should do that today in the subunit log file01:43
harlowjaseems like it squirrels away the logs (or cinder just sends them all the /dev/null), both could be possible01:43
clarkbharlowja: look in .testrepository/$NUMBEROFTESTRUN01:43
harlowjakk, ya, so i get a brief overview there in those files01:43
clarkbsdague: fungi jhesketh ok 4 new logstash workers are up01:43
mtreinishclarkb: does cinder have the logging fixture add the logs to the subunit stream?01:43
jheskethnice stuff clarkb01:43
clarkbwe have 59 worker processes (short one but I need to do dinenr and stuff so will debug tomorrow)01:43
harlowjaso there is the following, https://github.com/openstack/cinder/blob/master/cinder/test.py#L12401:44
harlowjabut fake logger seems to just absorb them all01:44
clarkbfungi: jhesketh sdague I think we may also want to restart the logstash gearman client at some point because the geard there seems "slow" I think it may have leaked01:44
clarkbmtreinish: it should01:44
clarkbharlowja: yes it should write them to .testrepository/$NUMBEROFTESTRUN01:44
clarkbwhich is the subunit log01:44
mtreinishclarkb, harlowja: it doesn't look like it does01:45
harlowjaya, its very skimpy on the logs in those files01:45
harlowjalet me see if i can find what nova is doing, and see if that works for this case01:46
mtreinishharlowja: you need something like: http://git.openstack.org/cgit/openstack/tempest/tree/tempest/test.py#n29401:46
mtreinishor you could start using oslotest which has that in the base test class01:46
harlowjamtreinish ya, nova is doing something different than that, https://github.com/openstack/nova/blob/master/nova/test.py#L25401:46
harlowjalet me try one of these, see what can showup01:47
mtreinishharlowja: that's because of game nova's playing to catching sytax errors in the log messages01:47
*** weshay has quit IRC01:47
harlowjathose nova people and there games :-P01:48
* harlowja not sure if i'd buy that game at toys'r'us01:49
mtreinishharlowja: it's to ensure that all the log messages at any level get compiled in the unit tests, but only those at info or above get added to the subunit stream by the fixture01:50
clarkbmordred: ok I am trying it that way. I am moving all stuff into slave_db.pp then normal puppet runs run that code and in_chroot doesn't. This allows us to explicitly run it from post-install.d01:53
*** rickerc has joined #openstack-infra01:53
*** rkukura has joined #openstack-infra01:53
clarkbmordred: we'll see if this works. I am also hacking skip-networking int mysql conf.d into things before puppet runs01:53
clarkband will have to do something similar to postgres01:53
*** gokrokve has joined #openstack-infra01:56
openstackgerritK Jonathan Harker proposed a change to openstack-infra/config: Fix nodepool instance hostnames  https://review.openstack.org/9856101:56
harlowjamtreinish got it to work, thx. much nicer01:57
* harlowja wonders how the cinder folks are doing without this 01:57
clarkbharlowja: you should propose that change to cinder01:57
*** sweston has quit IRC02:01
*** timrc-afk is now known as timrc02:03
*** rickerc has quit IRC02:07
*** amcrn has quit IRC02:08
*** rkukura has quit IRC02:08
*** sweston has joined #openstack-infra02:09
openstackgerritA change was merged to openstack-infra/config: Fixed KEY_DIR for the trove-buildimage-mysql job  https://review.openstack.org/9893302:09
*** rkukura has joined #openstack-infra02:10
*** sweston has quit IRC02:13
*** rickerc has joined #openstack-infra02:13
*** zz_gondoi is now known as gondoi02:14
*** gokrokve has quit IRC02:18
*** trinaths has joined #openstack-infra02:19
*** reed has joined #openstack-infra02:19
*** nati_ueno has joined #openstack-infra02:23
harlowjamtreinish https://review.openstack.org/#/c/98937/ (if u want to comment)02:23
*** ramashri has joined #openstack-infra02:24
*** penguinRaider has joined #openstack-infra02:24
*** [1]trinaths has joined #openstack-infra02:25
*** reed has quit IRC02:25
*** trinaths has quit IRC02:28
*** [1]trinaths is now known as trinaths02:28
*** doug-fish has left #openstack-infra02:30
*** arnaud has joined #openstack-infra02:33
*** godp1301 has joined #openstack-infra02:34
*** trinaths has quit IRC02:36
*** godp1301 has quit IRC02:37
lifelessare we allowed to +A things now?02:38
lifelesssdague: btw you might like to put #tripleo in your channels02:38
*** penguinRaider has quit IRC02:41
*** ramashri has quit IRC02:45
*** primeministerp has quit IRC02:46
*** rwsu has quit IRC02:52
mordred_phoneclarkb: nod02:59
*** signed8bit has quit IRC03:00
*** praneshp has quit IRC03:03
clarkbit doesnt work03:03
clarkbeven in post install no services :(03:03
*** crc32 has quit IRC03:05
*** nati_ueno has quit IRC03:10
*** david-lyle has joined #openstack-infra03:11
*** lbragstad has quit IRC03:11
*** prad has joined #openstack-infra03:12
*** baohua has joined #openstack-infra03:17
*** zhiyan is now known as zhiyan_03:20
*** zhiyan_ is now known as zhiyan03:26
boris-42clarkb mordred  hm I am facing strange issue with global update requirements03:27
boris-42clarkb mordred  https://review.openstack.org/#/c/96833/4/test-requirements.txt it says that sphinx is not ok here03:28
boris-42clarkb mordred  http://logs.openstack.org/33/96833/4/check/gate-rally-requirements/8e66a49/console.html03:28
boris-42and in global requirements https://github.com/openstack/requirements/blob/master/global-requirements.txt#L16903:28
fungiboris-42: https://review.openstack.org/97893 will fix it, but i think we've just been short on other core reviewers so i'll go ahead and approve it03:35
boris-42fungi ah ok, it's not rally issue in my project huh03:35
fungiboris-42: no, the job itself was enforcing in the wrong place03:36
*** talluri has joined #openstack-infra03:37
*** arnaud has quit IRC03:42
*** nosnos has quit IRC03:43
boris-42fungi thanks03:43
*** CaptTofu_ has joined #openstack-infra03:50
*** pcrews_ has quit IRC03:52
*** Longgeek has joined #openstack-infra03:52
*** dims_ has quit IRC03:53
openstackgerritA change was merged to openstack-infra/config: Revert "Checks if requirements overlap between head and branch reqs"  https://review.openstack.org/9789303:53
*** basha has joined #openstack-infra03:54
clarkbwait now all those broken sphinx reqs can get through03:55
clarkbI didnt realize that was the "fix"03:55
boris-42clarkb are they broken?03:55
boris-42clarkb oh03:55
clarkbthey dont overlap03:56
*** basha has quit IRC03:57
*** Longgeek has quit IRC03:57
fungiclarkb: the breakage was felt on the front end when the reqs change went in, which was where that was supposed to have gotten enforced03:57
clarkbsure but at least it prevented broken03:58
fungithat change also needs to be reverted03:58
* fungi checks to see if anyone also did that part03:59
*** jamespage has quit IRC04:00
*** gnuoy has quit IRC04:00
*** jamespage has joined #openstack-infra04:00
*** jamespage has joined #openstack-infra04:00
*** gnuoy has joined #openstack-infra04:00
*** sweston has joined #openstack-infra04:00
adam_ganyone for a quickie? seems  ironic's devstack job no longer likes to run on HP.  https://review.openstack.org/#/c/98886/04:01
fungii think i'll implement dhellmann's suggestion from https://review.openstack.org/88631 which apparently got overlooked when the change was approved04:02
*** MarkAtwood has joined #openstack-infra04:02
*** MarkAtwood has quit IRC04:02
*** basha has joined #openstack-infra04:07
openstackgerritJeremy Stanley proposed a change to openstack/requirements: Fix sphinx requirement to add overlap  https://review.openstack.org/9894704:07
fungiclarkb: that ^ ?04:08
*** basha has quit IRC04:08
fungii couldn't find any indication of an existing change along those lines. though also possible that it's closing the barn door after the horse has left04:09
notmynamefungi: what was the usb microscope you used at the summit keysigning? is it good enough eg to look at stuff in pond water or at cheek cells?04:09
funginotmyname: that one i brought because it was low enough magnification i could use it for macro range. i have a much higher magnification one which will. hold on and i'll get you the model number04:10
notmynamefungi: cool, thanks!04:10
clarkbfungi: +2 thanks04:10
funginotmyname: the celestron model i brought with me does 150x and is good enough for examining solder joints for surface-mount electronics. the better one i have goes to 220x magnification, which i got at http://www.adafruit.com/products/636 (a steal at us$80)04:17
*** om has joined #openstack-infra04:17
notmynamefungi: thanks :-)04:18
funginotmyname: though keep in mind that you need ~500x to see, say, human blood cells04:18
notmynamefungi: got it. I'm looking for something for my kids (who am I kidding? I think it will be really cool)04:19
fungiat that price it's a great starter. i can stream off it at 640x480 using the uvc/v4l drivers in debian no problem04:20
fungii haven't tried the snapshot function though. might require other drivers04:21
fungithough uvccapture may just work on it04:21
*** lcheng has joined #openstack-infra04:21
*** yfried has quit IRC04:21
*** nosnos has joined #openstack-infra04:24
*** gokrokve has joined #openstack-infra04:25
*** om has quit IRC04:25
fungii'm tempted to upgrade to http://www.adafruit.com/products/1061 (doesn't support the higher-res stills, but goes to 300x magnification)04:29
*** penguinRaider has joined #openstack-infra04:31
*** david-lyle has quit IRC04:33
*** CaptTofu_ has quit IRC04:34
*** CaptTofu_ has joined #openstack-infra04:34
*** marcoemorais has joined #openstack-infra04:34
StevenKDamn it, I left Zuul Status open on my laptop and it OOM-killed Firefox04:36
*** marcoemorais1 has joined #openstack-infra04:36
fungiStevenK: yeah, does the same to me if i leave it overnight04:37
*** gokrokve has quit IRC04:37
fungii blame javascript04:37
fungifor all the world's troubles04:37
clarkbhuh. zuul is pretty well behaved for me04:38
*** lbragstad has joined #openstack-infra04:38
clarkbgerrit gives it a hard time though04:38
StevenKI think Firefox allocates new RAM everytime it gets more JSON, and doesn't free anything ever04:38
*** marcoemorais has quit IRC04:39
*** om has joined #openstack-infra04:39
*** CaptTofu_ has quit IRC04:39
lifelessStevenK: buy more ram04:41
fungithe firefox, it hungers04:41
fungiit must be fed with ram sacrifices04:42
StevenKlifeless: And install it into the laptop how? Gaffa tape it to the side?04:42
StevenKIt has 16GiB, it physically doesn't support more.04:42
StevenKI left Zuul open in Firefox while I went into the office on Friday, and got home ~ 8 hours later too:04:43
StevenK3610 steven    23   3 19.0g  17g  28m R  69.5 73.5 163:42.52 firefox04:43
*** trinaths has joined #openstack-infra04:43
StevenK(On my desktop, which has 24GiB)04:43
*** rlandy has joined #openstack-infra04:45
*** basha has joined #openstack-infra04:46
*** dims_ has joined #openstack-infra04:46
*** dims_ has quit IRC04:50
*** ildikov has quit IRC04:51
*** ArxCruz_ has quit IRC04:52
lifelessStevenK: beowulf04:53
openstackgerritYuriy Taraday proposed a change to openstack-infra/git-review: Ignore newline in bp/bug search in commit message  https://review.openstack.org/9832404:53
StevenKlifeless: If Firefox needs a beowulf to not OOM, I have missed a few memos04:54
openstackgerritA change was merged to openstack-infra/config: Add devstack-f20 to hpcloud  https://review.openstack.org/9802404:55
*** om has quit IRC04:55
lifelesswouldn't help anyhow, beowulf is process based IIRC04:57
*** gokrokve has joined #openstack-infra04:58
fungiyeah, beowulf clusters usually used a distributed scheduler like mosix or message passing between remote processes with pvm or mpich04:58
*** gokrokve has quit IRC04:58
* fungi has no idea why he remembers this after ~15 years04:59
*** harlowja is now known as harlowja_away04:59
* fungi still has an "extreme linux" install cd somewhere in his lab... held onto it for reasons of nostalgia05:00
*** wenlock has joined #openstack-infra05:00
clarkbugh mpi05:01
clarkbI remember teaching grad students how to compile their mpi05:01
fungimpi was a lot more tightly-coupled than pvm, for sure05:01
*** arnaud has joined #openstack-infra05:02
*** om has joined #openstack-infra05:10
*** lbragstad has quit IRC05:13
*** ildikov has joined #openstack-infra05:15
*** greghaynes has joined #openstack-infra05:15
*** lbragstad has joined #openstack-infra05:15
*** wenlock has quit IRC05:17
*** doude has quit IRC05:22
*** yfried has joined #openstack-infra05:24
*** rlandy_ has joined #openstack-infra05:24
*** rlandy has quit IRC05:26
*** devkulkarni has joined #openstack-infra05:28
*** gyee has quit IRC05:38
*** arnaud__ has joined #openstack-infra05:46
*** dims_ has joined #openstack-infra05:47
*** TheSheep has joined #openstack-infra05:47
*** TheSheep has quit IRC05:48
*** Longgeek has joined #openstack-infra05:49
*** Longgeek has quit IRC05:49
*** rdopiera has joined #openstack-infra05:50
*** rdopiera has quit IRC05:50
*** Longgeek has joined #openstack-infra05:50
*** rdopieralski has joined #openstack-infra05:51
*** rdopieralski has quit IRC05:51
*** Longgeek has quit IRC05:51
*** rdopieralski has joined #openstack-infra05:51
*** Longgeek has joined #openstack-infra05:51
*** dims_ has quit IRC05:51
*** gondoi is now known as zz_gondoi05:52
*** adrian_otto has joined #openstack-infra05:57
*** wenlock has joined #openstack-infra06:02
*** Longgeek has quit IRC06:04
*** Longgeek has joined #openstack-infra06:04
*** devkulkarni has quit IRC06:04
*** Longgeek has quit IRC06:04
*** devkulkarni has joined #openstack-infra06:05
*** Longgeek has joined #openstack-infra06:05
*** Longgeek has quit IRC06:06
*** Longgeek_ has joined #openstack-infra06:06
*** devkulkarni has quit IRC06:07
*** arnaud__ has quit IRC06:07
*** arnaud has quit IRC06:08
*** _nadya_ has joined #openstack-infra06:10
*** CaptTofu_ has joined #openstack-infra06:14
*** cody-somerville has joined #openstack-infra06:17
*** cody-somerville has quit IRC06:17
*** cody-somerville has joined #openstack-infra06:17
*** dims_ has joined #openstack-infra06:17
*** arnaud has joined #openstack-infra06:17
*** andreykurilin_ has joined #openstack-infra06:18
*** arnaud__ has joined #openstack-infra06:19
*** arnaud__ has quit IRC06:19
*** CaptTofu_ has quit IRC06:19
*** mkoderer has joined #openstack-infra06:21
*** dims_ has quit IRC06:22
*** arnaud__ has joined #openstack-infra06:22
*** arnaud has quit IRC06:22
*** arnaud__ has quit IRC06:22
*** arnaud has joined #openstack-infra06:23
*** arnaud has quit IRC06:23
*** e0ne has joined #openstack-infra06:27
*** rlandy_ has quit IRC06:30
rdopieralskiHi, I need some help with https://review.openstack.org/#/c/95716/ -- specifically, how would I go about creating a new team for it?06:32
*** Ryan_Lane has joined #openstack-infra06:35
*** rlandy_ has joined #openstack-infra06:37
*** Ryan_Lane1 has joined #openstack-infra06:41
*** Ryan_Lane has quit IRC06:41
*** wenlock has quit IRC06:44
*** YorikSar has quit IRC06:46
*** lbragstad has quit IRC06:49
*** lbragstad has joined #openstack-infra06:50
*** afazekas is now known as __afazekas06:51
*** rcarrill` has quit IRC06:52
*** e0ne has quit IRC06:54
*** adrian_otto has quit IRC06:55
*** cody-somerville has quit IRC06:55
*** e0ne has joined #openstack-infra06:55
*** rcarrillocruz has joined #openstack-infra06:57
*** e0ne has quit IRC06:57
*** salv-orlando has quit IRC06:58
*** jcoufal has joined #openstack-infra06:59
*** achuprin has quit IRC06:59
*** adrian_otto has joined #openstack-infra06:59
*** nosnos has quit IRC06:59
*** matsuhashi has quit IRC07:01
*** matsuhas_ has joined #openstack-infra07:01
*** basha has quit IRC07:02
*** basha has joined #openstack-infra07:03
*** nosnos has joined #openstack-infra07:05
*** jpich has joined #openstack-infra07:06
*** flaper87|afk is now known as flaper8707:06
*** pblaho has joined #openstack-infra07:06
*** cody-somerville has joined #openstack-infra07:09
*** achuprin has joined #openstack-infra07:11
mattoliverauNight all07:12
*** basha has quit IRC07:16
*** dims_ has joined #openstack-infra07:17
openstackgerritA change was merged to openstack-infra/git-review: Disable proxies for tests that clone over http  https://review.openstack.org/9689307:18
SergeyLukjanovrdopieralski, all groups will be created automatically and then you'll need to request init group members additions07:21
*** dims_ has quit IRC07:22
*** rlandy__ has joined #openstack-infra07:22
*** rlandy_ has quit IRC07:22
*** bogdando has quit IRC07:23
openstackgerritA change was merged to openstack-infra/git-review: Keep track of gerrit.war and golden_site versions  https://review.openstack.org/9681807:23
*** tkelsey has joined #openstack-infra07:24
*** bogdando has joined #openstack-infra07:24
*** Hal_ has joined #openstack-infra07:26
*** bauzas has joined #openstack-infra07:26
rdopieralskiSergeyLukjanov: how would I do that?07:27
*** _nadya_ has quit IRC07:28
*** andreaf has quit IRC07:31
*** dizquierdo has joined #openstack-infra07:32
*** mwagner_lap has quit IRC07:34
*** rlandy__ has quit IRC07:35
*** dizquierdo has quit IRC07:36
*** mwagner_lap has joined #openstack-infra07:36
*** ominakov has joined #openstack-infra07:40
*** jlibosva has joined #openstack-infra07:42
SergeyLukjanovrdopieralski, you can ask here after the repo adding change will be merged07:44
SergeyLukjanovand someone from infra root team will add init team members07:44
SergeyLukjanovneed to go, brb in 1h07:44
rdopieralskiSergeyLukjanov: thank you07:45
*** Ryan_Lane1 has quit IRC07:46
*** thomasbiege has joined #openstack-infra07:47
*** rlandy__ has joined #openstack-infra07:48
*** e0ne has joined #openstack-infra07:49
*** dizquierdo has joined #openstack-infra07:49
*** sweston has quit IRC07:54
*** afazekas has joined #openstack-infra07:55
*** vponomaryov has joined #openstack-infra07:55
openstackgerrityolanda.robla proposed a change to openstack-infra/nodepool: Build images using diskimage-builder  https://review.openstack.org/4648207:56
*** hashar has joined #openstack-infra07:56
*** andreaf has joined #openstack-infra07:58
*** ihrachyshka has joined #openstack-infra07:59
*** jistr has joined #openstack-infra08:00
*** CaptTofu_ has joined #openstack-infra08:03
*** andreykurilin_ has quit IRC08:03
*** e0ne has quit IRC08:03
openstackgerritValeriy Ponomaryov proposed a change to openstack-infra/config: Set env vars explicitly for manila jobs  https://review.openstack.org/9774708:07
*** bauzas has quit IRC08:07
*** CaptTofu_ has quit IRC08:08
*** basha has joined #openstack-infra08:10
*** e0ne_ has joined #openstack-infra08:12
*** IvanBerezovskiy has joined #openstack-infra08:12
openstackgerritA change was merged to openstack-infra/git-review: Restrict tests SSH auth to only the provided key  https://review.openstack.org/9689408:12
*** srenatus has joined #openstack-infra08:12
srenatusi can haz https://review.openstack.org/#/c/98084/ without creating another lame cat meme pic? :}08:13
*** roeyc has joined #openstack-infra08:15
*** sweston has joined #openstack-infra08:15
*** derekh_ has joined #openstack-infra08:17
*** dims_ has joined #openstack-infra08:18
*** dims_ has quit IRC08:22
*** marcoemorais1 has quit IRC08:23
*** habib__ has joined #openstack-infra08:23
*** _nadya_ has joined #openstack-infra08:25
*** sweston has quit IRC08:35
*** doude has joined #openstack-infra08:36
*** matsuhas_ has quit IRC08:38
*** nosnos has quit IRC08:40
*** pelix has joined #openstack-infra08:41
*** om has quit IRC08:41
*** lbragstad has quit IRC08:43
*** nosnos has joined #openstack-infra08:43
*** lbragstad has joined #openstack-infra08:43
*** nosnos has quit IRC08:44
*** adrian_otto has quit IRC08:44
*** matsuhashi has joined #openstack-infra08:44
*** mmaglana has quit IRC08:54
*** om has joined #openstack-infra08:54
*** jcoufal has quit IRC08:55
*** adrian_otto has joined #openstack-infra08:55
*** jcoufal has joined #openstack-infra08:56
*** alkari has quit IRC08:58
*** thomasbiege1 has joined #openstack-infra09:00
*** habib___ has joined #openstack-infra09:01
*** sweston has joined #openstack-infra09:02
*** adrian_otto has quit IRC09:03
*** nosnos has joined #openstack-infra09:03
*** habib__ has quit IRC09:04
*** thomasbiege has quit IRC09:04
*** jgallard has joined #openstack-infra09:05
*** habib___ has quit IRC09:06
*** habib___ has joined #openstack-infra09:07
*** habib___ has quit IRC09:08
*** habib___ has joined #openstack-infra09:10
*** ihrachyshka has quit IRC09:10
*** isviridov_ is now known as isviridov09:12
*** ociuhandu has quit IRC09:15
*** zhiyan is now known as zhiyan_09:18
*** dims_ has joined #openstack-infra09:19
*** sweston has quit IRC09:24
*** habib___ has quit IRC09:24
*** dims_ has quit IRC09:25
*** habib___ has joined #openstack-infra09:30
*** baohua has quit IRC09:34
*** mugsie has quit IRC09:40
*** mugsie has joined #openstack-infra09:42
*** weshay has joined #openstack-infra09:43
*** rlandy_ has joined #openstack-infra09:43
srenatusalright. I see no other options.  http://i.imgur.com/aGkA8DJ.jpg >> https://review.openstack.org/#/c/98084/09:45
*** maxbit has quit IRC09:46
*** rlandy__ has quit IRC09:46
*** mugsie has quit IRC09:46
*** CaptTofu_ has joined #openstack-infra09:51
*** ihrachyshka has joined #openstack-infra09:53
*** roeyc has quit IRC09:54
*** CaptTofu_ has quit IRC09:56
*** salv-orlando has joined #openstack-infra10:00
*** Alexei_987 has quit IRC10:02
*** ociuhandu has joined #openstack-infra10:06
*** jgallard has quit IRC10:07
*** deepakcs has joined #openstack-infra10:13
deepakcsHi, has anyone here faced issues with devstack hosted on F20 ?10:13
deepakcsI heard that f20 was recently added to openstack-infra, hence the Q10:13
openstackgerritA change was merged to openstack/requirements: Updated zake now that zake 0.0.20 is released  https://review.openstack.org/9315610:14
*** om has quit IRC10:14
deepakcsOn my local F20 VM + devstack, Nova instance always gets stuck in 'spawning' state10:14
openstackgerritRoger Luethi proposed a change to openstack-infra/git-review: Fix a typo in HACKING.rst  https://review.openstack.org/9899810:15
deepakcsso was wondering if anyone here has seen such issue on F2010:15
*** rlandy__ has joined #openstack-infra10:20
*** dims_ has joined #openstack-infra10:21
*** rlandy_ has quit IRC10:22
sdaguedeepakcs: it's still not running regularly10:26
*** e0ne_ has quit IRC10:26
*** e0ne has joined #openstack-infra10:27
*** dims_ has quit IRC10:27
deepakcssdague, ok, but when it runs, u dont see any issues ?10:28
*** habib___ has quit IRC10:29
deepakcssdague, I assume u folks run devstack on F20 as part of CI, rite ?10:29
sdaguedeepakcs: we do not10:29
deepakcssdague, I am facing issues with F20+devstack, hence :)10:29
deepakcssdague, sorry, 'we do not' response was for "not seeign any issues" ?10:30
sdaguewe do not run it in ci10:30
sdagueianw has been working on getting it in, but it's not10:30
*** hashar has quit IRC10:32
deepakcssdague, ok, does that mean not many in community would be running F20+devstack combo ?10:32
deepakcssdague, i am curious to know whether its something that is related to my setup or there is a real issue with F20 and devstakc10:32
deepakcsianw, ^^10:33
*** pblaho has quit IRC10:33
openstackgerritA change was merged to openstack-infra/zuul: Use except x as y instead of except x, y  https://review.openstack.org/9661510:34
openstackgerritA change was merged to openstack-infra/zuul: replace dict.iteritems() with six.iteritems(dict)  https://review.openstack.org/9844010:38
*** om has joined #openstack-infra10:39
*** roeyc has joined #openstack-infra10:45
roeycHi, Devstack setup is failing as part of our 3rd party CI, the problem is with installation of the pbr library.10:47
roeychoped to get some help here..10:47
roeycI added more details here: http://paste.openstack.org/show/83508/10:47
*** rcarrillocruz has quit IRC10:48
*** rlandy_ has joined #openstack-infra10:48
roeyccan anyone assist please?10:48
openstackgerritA change was merged to openstack-infra/zuul: Use import from six.moves to import the ConfigParser module  https://review.openstack.org/9859710:48
*** rlandy__ has quit IRC10:49
openstackgerritA change was merged to openstack-infra/zuul: Use import from six.moves to import the queue module  https://review.openstack.org/9859510:50
*** lcostantino has joined #openstack-infra10:50
openstackgerritA change was merged to openstack-infra/nodepool: Use import from six.moves to import the queue module  https://review.openstack.org/9859610:51
*** rcarrillocruz has joined #openstack-infra10:52
*** Ryan_Lane has joined #openstack-infra10:54
*** mmaglana has joined #openstack-infra10:55
*** rlandy_ has quit IRC10:57
*** rlandy has joined #openstack-infra10:57
*** ekarlso has quit IRC10:57
*** ekarlso has joined #openstack-infra10:58
*** mmaglana has quit IRC11:00
*** om has quit IRC11:02
*** Ryan_Lane has quit IRC11:03
*** simonmcc has quit IRC11:04
*** bogdando has quit IRC11:04
*** bogdando has joined #openstack-infra11:04
*** simonmcc has joined #openstack-infra11:04
*** cyeoh has quit IRC11:06
*** talluri has quit IRC11:09
*** talluri has joined #openstack-infra11:09
*** cyeoh has joined #openstack-infra11:10
*** yaguang has quit IRC11:10
*** talluri has quit IRC11:11
*** talluri has joined #openstack-infra11:11
*** talluri has quit IRC11:16
*** om has joined #openstack-infra11:20
*** dims_ has joined #openstack-infra11:23
openstackgerritAlexandre Levine proposed a change to openstack-infra/config: Initial commit for the AWS EC2 and VPC API standalone service project  https://review.openstack.org/9779511:24
*** _nadya_ has quit IRC11:26
*** mugsie has joined #openstack-infra11:27
*** salv-orlando has quit IRC11:27
*** om has quit IRC11:29
*** alexpilotti has joined #openstack-infra11:29
*** salv-orlando has joined #openstack-infra11:32
*** yamahata has quit IRC11:32
*** trinaths has quit IRC11:35
*** miqui has quit IRC11:38
openstackgerritPh. Marek proposed a change to openstack/requirements: Add dbus library, needed for the DRBD cinder driver integration.  https://review.openstack.org/9901311:39
*** CaptTofu_ has joined #openstack-infra11:39
*** yfried has quit IRC11:40
*** CaptTofu_ has quit IRC11:44
*** om has joined #openstack-infra11:44
openstackgerritSean Dague proposed a change to openstack-infra/devstack-gate: don't update apt repos  https://review.openstack.org/9901611:45
*** CaptTofu_ has joined #openstack-infra11:47
*** hashar has joined #openstack-infra11:49
*** ArxCruz has joined #openstack-infra11:52
*** rlandy has quit IRC11:56
openstackgerritafazekas proposed a change to openstack-infra/elastic-recheck: Add query for 1328500  https://review.openstack.org/9902011:57
*** _nadya_ has joined #openstack-infra11:57
*** adalbas has joined #openstack-infra12:04
openstackgerritAnne Gentle proposed a change to openstack-infra/config: Add separate project for training-guides  https://review.openstack.org/9633412:04
*** radez_g0n3 is now known as radez12:05
*** YorikSar has joined #openstack-infra12:06
*** afazekas has quit IRC12:09
*** aysyd has joined #openstack-infra12:09
*** yfried has joined #openstack-infra12:10
*** mbacchi has joined #openstack-infra12:16
*** afazekas has joined #openstack-infra12:24
*** dkranz has joined #openstack-infra12:24
*** lcostantino has quit IRC12:25
*** Longgeek_ has quit IRC12:25
openstackgerritA change was merged to openstack/requirements: Update openstack-doc-tools to at least 0.15  https://review.openstack.org/9587912:25
*** Longgeek has joined #openstack-infra12:26
*** mwagner_lap has quit IRC12:26
*** CaptTofu_ has quit IRC12:26
*** CaptTofu_ has joined #openstack-infra12:27
*** dprince has joined #openstack-infra12:28
*** lcostantino has joined #openstack-infra12:28
*** dhellmann has quit IRC12:28
*** yaguang has joined #openstack-infra12:29
*** yaguang has quit IRC12:30
*** yaguang has joined #openstack-infra12:30
*** oomichi_ has joined #openstack-infra12:31
*** CaptTofu_ has quit IRC12:31
anteayahi roeyc I found this: https://bugs.launchpad.net/swift/+bug/121728812:32
uvirtbotLaunchpad bug 1217288 in swift "Swift installation crashes" [High,Triaged]12:32
anteayaroeyc: which it appears suggests a workaround for the moment12:32
*** bknudson has quit IRC12:34
anteayaroeyc: the suggested workaround is `sudo pip install -r requirements.txt`12:35
roeycanteaya: thanks, but it doesn't seem to work here.12:37
anteayamordred is our pbr expert so unless someone else wants to step in, you might be waiting for him to appear12:38
*** deepakcs has quit IRC12:39
openstackgerritA change was merged to openstack-infra/config: Adjust ironic tempest regex for new tests  https://review.openstack.org/9783012:39
roeycalright then, thanks again12:39
openstackgerritThierry Carrez proposed a change to openstack-infra/release-tools: Add script for new-style milestone publication  https://review.openstack.org/9812312:42
*** heyongli has joined #openstack-infra12:42
*** palar has joined #openstack-infra12:43
*** heyongli has quit IRC12:43
openstackgerritThierry Carrez proposed a change to openstack-infra/release-tools: Add script for new-style milestone publication  https://review.openstack.org/9812312:43
*** heyongli has joined #openstack-infra12:43
*** pdmars has joined #openstack-infra12:44
*** heyongli has quit IRC12:44
*** pdmars has quit IRC12:44
*** pdmars has joined #openstack-infra12:45
*** pblaho has joined #openstack-infra12:45
roeycanteaya: problem was with the setuptools package, re-installing it and it looks fine now12:45
*** lbragstad has quit IRC12:45
openstackgerritThierry Carrez proposed a change to openstack-infra/release-tools: Add script for new-style milestone publication  https://review.openstack.org/9812312:46
*** yhe_ has joined #openstack-infra12:46
openstackgerritIlya Sviridov proposed a change to openstack-infra/config: Added docs-on-rtfd for MagnetoDB  https://review.openstack.org/9903912:47
*** palar has quit IRC12:47
*** palar has joined #openstack-infra12:48
*** dkranz has quit IRC12:48
openstackgerritPhilip Marc Schwartz proposed a change to openstack-infra/config: Creation of vinz project in the openstack-infra scheme.  https://review.openstack.org/9395312:49
anteayaroeyc: awesome, thanks for letting me know12:49
anteayasdague: isn't line 502 missing a %s? https://review.openstack.org/#/c/99002/1/nova/network/manager.py12:49
anteayaand line 56712:50
anteayamaybe this is a part of python strings I don't understand12:50
*** yhe_ has quit IRC12:50
sdagueanteaya: no, becuase our logger knows how to log instance_id separately12:51
anteayathat was the piece I was missing, thansk12:51
*** bknudson has joined #openstack-infra12:52
*** HenryG has quit IRC12:53
*** alexpilotti_ has joined #openstack-infra12:53
sdaguethere are a bunch of special fields past by name that we pull out - https://github.com/openstack/nova/blob/master/nova/openstack/common/log.py if you want all the details12:53
*** afazekas has quit IRC12:53
*** alexpilotti has quit IRC12:54
*** alexpilotti_ is now known as alexpilotti12:54
* anteaya looks12:54
sdaguethe logic is a little squirely, but a combination of process() and format() will give you the feel12:54
*** timrc is now known as timrc-afk12:55
openstackgerritAnne Gentle proposed a change to openstack-infra/config: Add separate project for training-guides  https://review.openstack.org/9633412:55
*** rlandy has joined #openstack-infra12:56
*** mmaglana has joined #openstack-infra12:56
openstackgerritIvan Kolodyazhny proposed a change to openstack-infra/config: Make 'puppetmaster_host' configurable'  https://review.openstack.org/9904112:57
*** changbl has quit IRC12:57
*** dims_ has quit IRC12:59
*** skolekonov has joined #openstack-infra13:00
sdaguemordred: so this is fun - https://review.openstack.org/#/c/99016/ looks like the local apt caches are only good on 50% of our nodes?13:00
*** mmaglana has quit IRC13:01
*** miqui has joined #openstack-infra13:05
*** yamahata has joined #openstack-infra13:05
*** amotoki_ has joined #openstack-infra13:06
*** pblaho` has joined #openstack-infra13:06
*** godp1301 has joined #openstack-infra13:06
*** julim has joined #openstack-infra13:06
*** rlandy has quit IRC13:07
*** godp1301 has left #openstack-infra13:07
*** afazekas has joined #openstack-infra13:07
*** basha has quit IRC13:08
*** pblaho has quit IRC13:08
*** mfer has joined #openstack-infra13:16
*** mriedem has joined #openstack-infra13:16
*** lbragstad has joined #openstack-infra13:17
anteayaI got distracted by the email list13:18
anteayafolks who make tools that abstract away messiness need to identify the criteria for the decisions the abstraction is making13:18
anteayafound process() yeah what it is doing with context and instance make more sense now13:20
*** alexpilotti_ has joined #openstack-infra13:21
*** alexpilotti has quit IRC13:22
*** alexpilotti_ is now known as alexpilotti13:22
pelixin git-review, is there a reason for the tests to care about having the correct ssh hosts key?13:22
pelixas opposed to using StrictHostKeyChecking=no for the tests?13:22
pelixSeems to be causing a lot of failures in testing at the moment13:22
*** rlandy has joined #openstack-infra13:22
*** bcrochet has quit IRC13:23
*** bcrochet has joined #openstack-infra13:25
*** lbragstad1 has joined #openstack-infra13:26
*** trinaths has joined #openstack-infra13:26
anteayapelix: what failures in testing?13:27
*** smarcet has joined #openstack-infra13:27
anteayathe host key has changed once since I have been with openstack (1.5 years)13:27
anteayaif the host key settings are accurate, there is no reason for tests to keep failing on host key checking13:27
*** basha has joined #openstack-infra13:27
*** [1]trinaths has joined #openstack-infra13:28
*** lbragstad has quit IRC13:28
*** lbragstad has joined #openstack-infra13:29
*** lbragstad1 has quit IRC13:30
*** dims_ has joined #openstack-infra13:30
*** gokrokve has joined #openstack-infra13:31
*** trinaths has quit IRC13:31
*** [1]trinaths is now known as trinaths13:31
*** jgrimm has joined #openstack-infra13:34
openstackgerritIvan Kolodyazhny proposed a change to openstack-infra/config: Make 'puppetmaster_host' configurable'  https://review.openstack.org/9904113:34
openstackgerritIvan Kolodyazhny proposed a change to openstack-infra/config: Make 'puppetmaster_host' configurable  https://review.openstack.org/9904113:35
*** salv-orlando has quit IRC13:35
*** salv-orlando has joined #openstack-infra13:36
pelixanteaya: take a look at https://review.openstack.org/#/c/96894/13:36
devanandasdague: g'morning! not sure you saw my ping last night13:36
pelixanteaya: it's not testing against the openstack gerrit instance, I'm talking about the unit tests in git review13:36
sdaguedevananda: probably not13:37
devanandasdague: i'd like to land several patches for ironic today. will be working with the core team to rebase/reorganize/reprioritize our queue, since it was completely stalled for a week13:37
devanandasdague: and we've got a few crit and many high pri bug fixes in flight now13:37
devanandasdague: i know the gate is still fragile... so i'm checking before we approve much more. is this ok?13:38
*** pblaho` has quit IRC13:38
sdaguedevananda: sure, it will just be slow13:38
devanandasdague: understood. thanks13:39
sdaguealso it looks like ironic is racing on it's own tests quite a bit13:39
devanandasdague: we're getting a lot of failures from network issues13:39
devanandaespecially Bug 132675413:39
uvirtbotLaunchpad bug 1326754 in tripleo "Ubuntu CI failures  : Unable to locate package ubuntu-cloud-keyring" [High,In progress] https://launchpad.net/bugs/132675413:39
*** CaptTofu_ has joined #openstack-infra13:39
*** matsuhashi has quit IRC13:39
*** devkulkarni has joined #openstack-infra13:40
*** hdd_ has joined #openstack-infra13:40
devanandasdague: are you seeing some other races in ironic?13:40
*** devkulkarni has quit IRC13:40
sdaguehonestly, I can't remember13:40
sdaguesorry, kind of burnt out on bugs13:40
*** devkulkarni has joined #openstack-infra13:40
*** eharney has joined #openstack-infra13:41
*** Longgeek has quit IRC13:42
*** Longgeek has joined #openstack-infra13:42
*** CaptTofu_ has quit IRC13:44
*** dkliban_afk is now known as dkliban13:45
*** signed8bit has joined #openstack-infra13:45
*** lbragstad has quit IRC13:48
*** lbragstad has joined #openstack-infra13:48
*** nosnos has quit IRC13:50
*** vtapia has joined #openstack-infra13:50
anteayapelix: okay so that is a patch13:50
*** _nadya_ has quit IRC13:50
anteayapelix: connect me to where you are seeing failures13:50
*** vtapia has left #openstack-infra13:50
*** homeless has joined #openstack-infra13:50
*** basha has quit IRC13:51
pelixplenty of others occurring, it seems to be a frequent problem that is just being worked around by commenting 'recheck no bug' in the reviews13:52
anteayapelix: okay now I am with you: key_verify failed for server_host_key13:57
anteayafungi hopefully will be around soon13:57
anteayaunless he has had to hide in a closet or work in his car today13:57
pelixanteaya: seems like there is some odd interaction between the ssh client and server13:57
*** lbragstad1 has joined #openstack-infra13:57
*** lbragstad has quit IRC13:58
anteayapelix: yes, now that I see what you are seeing, I am in agreement with you13:58
pelixI'll try a workaround that may work better by setting HostKeyAlgorithms=ssh-rsa on the client side, and give it about 10-11 runs locally to see if I still hit the problem13:59
pelixif that doesn't work I'll look at disabling the stricthostkey checking14:01
*** om has quit IRC14:03
*** jcoufal has quit IRC14:03
fungianteaya: yeah, i've been around, just catching up on bugs/e-mail/reviews14:05
anteayafungi: figured as much14:05
anteayaah the silence while it lasts14:05
fungipelix: the git-review tests are racy--nobody's had time to fix them14:06
fungione theory is adding bouncy castle instead of using gerrit's built-in crypto primitives may alleviate it14:06
*** rlandy has quit IRC14:06
openstackgerritChristian Berendt proposed a change to openstack-infra/elastic-recheck: Prettified all HTML files  https://review.openstack.org/9844514:07
fungifor a while we were rechecking those failures with the number of the bug which requested adding tests, since it's technically not complete if they're not stable14:07
fungibut i've temporarily forgotten which bug number that is14:07
*** CaptTofu_ has joined #openstack-infra14:08
fungihowever it's fundamentally a shortcoming of the current integration test framework for git-review, rather than any one particular test being at fault14:08
fungiwe've at least added a lot of additional debugging output and logs to it, which has enabled us to rule out a lot of possible causes14:09
*** basha has joined #openstack-infra14:10
boris-42fungi hi14:11
fungiboris-42: hi, what's up?14:11
boris-42fungi I have some issues with new project, recently added14:11
boris-42fungi I am not core in it =)14:11
fungiboris-42: what's the project name? i'll add you14:11
boris-42fungi https://review.openstack.org/#/c/97778/14:11
fungiit's a manual process we have to perform after the project is created14:12
boris-42fungi hm14:12
boris-42fungi even if group exist?14:12
boris-42fungi cause I would like rally-core group to be maintainers of project14:13
fungiboris-42: oh, fun... looks like the acl didn't get applied. checking for typos in the review now14:13
fungii don't see anything obviously mistyped which would have caused the acl not to apply. checking logs from the manage-projects run now14:15
*** oomichi_ has quit IRC14:16
*** basha has quit IRC14:16
*** lbragstad1 has quit IRC14:16
*** devkulkarni has quit IRC14:16
pelixfungi: I believe I have some fixes from them14:16
*** zz_gondoi is now known as gondoi14:17
*** _nadya_ has joined #openstack-infra14:17
pelixfungi: for git-review I suspect that it's a combination of missing method calls to setup the username correctly and Gerrit not handling the full list of keyalgothrims correctly14:17
fungipelix: yes, i think you do14:17
fungipelix: sounds plausible14:17
boris-42fungi thanks14:18
pelixfungi: stressing my system using --concurrency=8 and 20 iterations at the moment to confirm :)14:18
anteayasdague: I'm considering proposing a talk entitled "how to read the logs"14:20
*** dhellmann has joined #openstack-infra14:20
*** pvo has joined #openstack-infra14:20
anteayamind you if I did that I would need to learn how to read the grenade logs14:20
sdagueanteaya: ++14:21
openstackgerritYuriy Taraday proposed a change to openstack-infra/git-review: Retry command on "hash mismatch" error in tests  https://review.openstack.org/9835414:21
anteayaokay, I will propose14:22
anteayatime to stop assuming people know how to read our logs and teach them14:22
openstackgerritSean Dague proposed a change to openstack-infra/elastic-recheck: add a tool for determining unaccounted for rechecks  https://review.openstack.org/9462414:23
*** miqui has quit IRC14:23
YorikSarzzelle, fungi: ^ - I've given up waiting for that error to reproduce, so that would be the final patchset14:23
*** miqui has joined #openstack-infra14:23
*** miqui has quit IRC14:23
sdagueanteaya: realistically I tried to do a little of that with the apache footer14:23
*** miqui has joined #openstack-infra14:23
YorikSarzzelle, fungi: I mean my CR to git-review https://review.openstack.org/9835414:23
sdaguehowever, it could be done better14:23
*** rdopieralski has quit IRC14:24
*** isviridov is now known as isviridov|away14:24
anteayasdague: the documentation you provide is great14:24
anteayasdague: mostly I'm casting about for a topic to propose so I can play in the water parks in brisbane14:24
anteayamikal mattoliverau and jhesketh promised to take me to water14:24
fungiboris-42: it looks from the logs like manage-projects is completely broken again. i have to get ready for a meeting (and am unfortunately in meetings most of the day) but hopefully someone will have time to investigate further and fix it14:25
*** mwagner_lap has joined #openstack-infra14:25
anteayaso while I'm there, forcing myself to learn the logs better and teach others is a win14:25
boris-42fungi ok will wait14:25
boris-42fungi thanks for help14:25
anteayafungi: can you link to the logs that shows manage-projects is broken?14:25
*** adrian_otto has joined #openstack-infra14:26
fungianteaya: i'm not sure it will help. when that change merged puppet claimed that it though every acl file had changed and then got unspecified errors trying to apply them all14:26
fungii don't have time to dig deeper into other logs (for example in the gerrit end of the connection) to see what the issue might be14:27
* anteaya nods14:27
anteayaenjoy your meeting14:27
fungiand by every acl file i mean every acl file in our gerrit14:27
anteayapabelanger: ohai14:27
pabelangerSo, finally have some time to dig into https://review.openstack.org/#/c/83980/ again (delete surplus ready nodes in nodepool).  Wanted to see how best to handle it now14:27
fungibut we haven't recently made any changes to jeepyb14:27
anteayaevery acl file in gerrit?14:27
belliottcan i get a 2nd core review on https://review.openstack.org/#/c/96570/ please?  it's just a stackforge repo request14:28
*** dkranz has joined #openstack-infra14:28
* anteaya looks at jeebyb logs14:28
anteayagit log that is14:28
anteaya9 days ago we merged this: http://git.openstack.org/cgit/openstack-infra/jeepyb/commit/?id=35bdef857a3b4ccdfe48b76b934a5d664d941b4b14:29
anteayasdague: would that change how manage projects edits acl files?14:29
*** jistr has quit IRC14:29
*** adrian_otto has quit IRC14:29
openstackgerritA change was merged to openstack/requirements: Add hacking 0.9.0 series  https://review.openstack.org/9882414:30
*** hashar has quit IRC14:31
*** rlandy has joined #openstack-infra14:31
*** jistr has joined #openstack-infra14:31
sdagueanteaya: yeh, and there might have been a change by fungi on top of it14:31
anteayasdague: hmmmm14:32
*** thomasbiege1 has quit IRC14:34
*** thomasbiege has joined #openstack-infra14:35
*** jaypipes has joined #openstack-infra14:37
*** pcrews_ has joined #openstack-infra14:38
*** jaypipes has quit IRC14:39
*** jaypipes has joined #openstack-infra14:39
*** jgrimm has quit IRC14:40
*** jaypipes has quit IRC14:41
openstackgerritBen Nemec proposed a change to openstack-infra/reviewstats: Encode to utf-8 before printing  https://review.openstack.org/9910414:41
*** jaypipes has joined #openstack-infra14:41
fungianteaya: sdague: yeah https://review.openstack.org/94684 but it's not gotten much reviewing yet14:42
fungiso not approved14:42
*** jaypipes has quit IRC14:44
*** yaguang has quit IRC14:45
*** jgallard has joined #openstack-infra14:45
*** jaypipes has joined #openstack-infra14:45
*** thedodd has joined #openstack-infra14:45
fungigate-tempest-dsvm-virtual-ironic seems to be an unhappy camper in the gate at the moment14:46
*** ominakov has quit IRC14:47
mordred_phonesdague: spectacular14:50
openstackgerritA change was merged to openstack-infra/git-review: Fix a typo in HACKING.rst  https://review.openstack.org/9899814:50
*** ominakov has joined #openstack-infra14:54
*** reed has joined #openstack-infra14:54
*** ominakov has quit IRC14:54
*** wenlock has joined #openstack-infra14:54
openstackgerritDzmitry Horbach proposed a change to openstack-infra/jenkins-job-builder: support Managed Script Plugin https://wiki.jenkins-ci.org/display/JENKINS/Managed+Script+Plugin  https://review.openstack.org/9910914:56
*** afazekas has quit IRC14:56
*** isviridov|away is now known as isviridov14:56
openstackgerritA change was merged to openstack-infra/config: Add md5 checksum to groups tarball publisher  https://review.openstack.org/9723814:56
*** mmaglana has joined #openstack-infra14:57
*** otherwiseguy has joined #openstack-infra14:57
fungimordred_phone: manage-projects needs love if you get near a real computer today... see review.o.o:/var/log/syslog.1 +/12:50:4814:57
*** roeyc has quit IRC14:58
* fungi will try to poke at it too after this first meeting wraps up14:58
*** maxbit has joined #openstack-infra14:58
*** james_li has joined #openstack-infra14:58
*** gokrokve has quit IRC14:59
*** palar_ has joined #openstack-infra15:00
*** gokrokve has joined #openstack-infra15:00
*** jcoufal has joined #openstack-infra15:01
*** mmaglana has quit IRC15:01
*** hashar has joined #openstack-infra15:02
anteayafungi: +1'd trusting merging will happen after current brokeness is addressed15:02
*** jcoufal has quit IRC15:02
sdaguefungi: it does occur to me that some unit tests on manage-projects would probably help. It's about 20th on my todo list :)15:03
*** palar has quit IRC15:03
*** jcoufal has joined #openstack-infra15:03
*** ihrachyshka has quit IRC15:04
*** penguinRaider has quit IRC15:04
*** HenryG has joined #openstack-infra15:04
*** palar_ has quit IRC15:05
*** wenlock has quit IRC15:07
*** ildikov_ has joined #openstack-infra15:08
*** marun has joined #openstack-infra15:08
*** ildikov has quit IRC15:09
*** skolekonov has quit IRC15:10
openstackgerritIlya Sviridov proposed a change to openstack-infra/config: Added docs-on-rtfd for MagnetoDB  https://review.openstack.org/9903915:11
*** mmaglana has joined #openstack-infra15:12
*** jistr has quit IRC15:12
*** jistr has joined #openstack-infra15:12
*** jpeeler has quit IRC15:13
*** praneshp has joined #openstack-infra15:13
*** ominakov has joined #openstack-infra15:14
KiallIs it just me, or is https://review.openstack.org/ failing to load? Stalls at "Loading Gerrit Code Review ..."15:15
*** jcoufal has quit IRC15:15
sdaguejust you15:15
KiallI had a useragent switcher turned on to pretend I was using IE7 -_-15:15
mordredfungi: excellent15:16
sdagueyou should be punished much for that15:16
Kiallsdague: lol - HP should be punished for having things that say "Only Internet Explorer 7 and Internet Explorer 8 are supported"15:16
boris-42^ hehe15:17
*** dprince has quit IRC15:17
*** sarob has joined #openstack-infra15:17
*** markwash has joined #openstack-infra15:18
openstackgerrityolanda.robla proposed a change to openstack-infra/storyboard-webclient: Added angular-momentjs to dependencies  https://review.openstack.org/9671315:18
anteayaKiall: oh yes, do share that message15:18
*** Ryan_Lane has joined #openstack-infra15:19
*** dangers_away is now known as dangers15:19
phschwartzKiall: Working for me.15:21
Kiallphschwartz: apologies, was my own stupidity ;)15:21
openstackgerrityolanda.robla proposed a change to openstack-infra/storyboard-webclient: Added angular-momentjs to dependencies  https://review.openstack.org/9671315:21
phschwartzKiall: It happens to us all some times. Me more then others especially yesterday15:21
*** atiwari has joined #openstack-infra15:24
*** jcoufal has joined #openstack-infra15:24
*** annegentle_ has joined #openstack-infra15:26
*** jgrimm has joined #openstack-infra15:28
*** changbl has joined #openstack-infra15:29
*** viktors is now known as viktors|afk15:30
mriedemhmm, check-swift-dsvm-functional is voting...but doesn't seem to be checked by e-r?15:31
*** gyee has joined #openstack-infra15:31
*** thedodd has quit IRC15:32
*** thedodd has joined #openstack-infra15:34
mriedemtrying to find anything in logstash that hits in screen-key.txt seems to be very sad atm15:36
mriedemnot sure why that would be15:36
*** trinaths has quit IRC15:36
sdaguemriedem: we're many hours behind in indexing15:38
*** Longgeek has quit IRC15:39
mriedemok, there is that keystone IANA address bug that i hit, but e-r status page shows we haven't hit that in 2 weeks15:39
mriedemso let's just remember to not delete that one...15:39
openstackgerritIlya Sviridov proposed a change to openstack-infra/config: Added new python-magnetodbclient project to Stackforge  https://review.openstack.org/9105015:39
mriedemplus, god forbid i have to actually dig into logs and cross check launchpad for bugs, right?15:39
*** jpeeler has joined #openstack-infra15:40
sc68calHey, sorry to blast random questions, but for *-specs, the rendered HTML output, are there plans to take those rendered HTML files and place them in a known URL? I was thinking it'd be useful to be able to link to them in developer docs15:40
sdaguefungi / mordred / SergeyLukjanov: https://review.openstack.org/#/c/98871/15:41
sc68calfor example, specs for Neutron, we could link to them in the devref doc for neutron (http://docs.openstack.org/developer/neutron/devref/)15:41
mordredfungi: investigating manage-projets now15:41
mordredsdague: looking15:41
bknudsonis there more to be done for https://review.openstack.org/#/c/97893/ ?  Does Jenkins need a nudge to get it to pick up the change?15:41
sdaguewill reduce overall d-g load, and brings us back to defaults15:41
*** basha has joined #openstack-infra15:42
mordredsdague: +215:42
*** changbl has quit IRC15:44
fungisc68cal: there are plans for a specs.openstack.org webpage where they will be organized and published. creating the site itself is easy and we can reuse the existing job templates for publishing them, but we also need someone to write a job to generate/update an overall index for the site15:44
*** doug-fish has joined #openstack-infra15:45
mordredfungi: ok. SO - the problem seems to be ...15:45
sc68calfungi: awesome15:45
mordredfungi: for some reason15:45
*** adrian_otto has joined #openstack-infra15:45
mordredfungi: that even though the git diff --quiet is returning 015:45
mordredfungi: we're still trying to commit15:46
mordred        if not copy_acl_config(project, repo_path, acl_config):15:46
mordred            # nothing was copied, so we're done15:46
mordred            return15:46
mordredis failing15:46
fungioh, funzies15:53
fungiand i wonder why puppet claims they've all changed (says it has a different hash for each of them)15:53
mordredfungi: oh, you're going to LOVE this...15:53
*** trinaths has joined #openstack-infra15:54
* fungi braces himself15:54
*** hdd_ has quit IRC15:55
*** tkelsey has quit IRC15:55
mordredfungi: the boolean logic is reversed15:55
*** Hal_ has quit IRC15:56
fungiexit code coerced into a python boolean type again?15:56
*** IvanBerezovskiy has left #openstack-infra15:57
fungithere really ought to be a standard method to all the subprocess calls for that situation, so people get used to using it instead of parsing != 015:57
openstackgerritMonty Taylor proposed a change to openstack-infra/jeepyb: Swap the boolean logic on copy_acl_config  https://review.openstack.org/9913615:58
mordredfungi: ^^15:58
mordredfungi: that's tested, btw - my brain still hurts thinking about it - so I just changed it in place to track what was happening15:58
*** basha has quit IRC15:58
*** markwash has quit IRC15:58
*** rdopieralski has joined #openstack-infra15:59
fungimordred: ohh, that happened in the recent exceptions-for-errors improvement i'm betting, which is why we hadn't spotted it until now15:59
*** julim has quit IRC15:59
fungiso it's been broken less than a week at least16:00
mordredfungi: yah16:00
mordredfungi: and it's an honest mistake - I stared at it for ages16:01
*** wenlock has joined #openstack-infra16:01
*** terryw has joined #openstack-infra16:01
fungii reviewed the change too and it didn't jump out at me. then re-reviewed it a little while ago when i noticed this, since it was the most recent change... still wasn't obvious16:01
*** otherwiseguy has quit IRC16:01
phschwartzI laughed at that one. That is a tough one to catch too.16:02
fungiit is a persistent thorn in the side of python scripts which call out to shells16:02
*** julim has joined #openstack-infra16:02
morganfainbergmriedem, the IANA bug is a long way off from being 100% solved. but we've reduced the impact as much as reasonable + documenting.16:02
mordredfungi: we could always start using the sh module16:03
morganfainbergmriedem, as aFYI of where we sit with it16:03
fungimordred: i'm always tempeted by that module, and then i seem to fond reasons not to use it but afterward can't recall why16:03
phschwartzfungi: I have always had the same thing crop up. I forced myself to use it and deal with the issues and seem to have liked it when I have used it.16:05
*** otherwiseguy has joined #openstack-infra16:06
*** mbacchi has quit IRC16:06
*** jgrimm has quit IRC16:07
*** dizquierdo has quit IRC16:07
mordredfungi: last time I poked at it there was one bug we needed fixed and I sent in a PR16:07
*** markwash has joined #openstack-infra16:07
*** aconrad has joined #openstack-infra16:08
*** marcoemorais has joined #openstack-infra16:08
mordredlet me look and see if it's fixed16:10
*** terryw has quit IRC16:10
anteayamorning pleia216:10
*** annegentle_ has quit IRC16:10
phschwartzmordred: I started moving my playbooks to use the nova_compute module instead of the Rax module that way I can upstream my playbook. I have also moved my dns work to a role for dns that uses the rax_dns module.16:13
*** annegent_ has joined #openstack-infra16:14
phschwartzBut that will wait. I have been working this morning on lending a hand with reviews. :)16:14
* mordred hands phschwartz a puppy dog16:14
*** mmaglana has quit IRC16:15
phschwartzmordred: now I have to stop and clean my laptops keyboard and screen as my coffee just decided of its own accord to relocate from my mouth.16:15
*** arnaud has joined #openstack-infra16:18
fungidstufft: finally got around to opening https://bitbucket.org/pypa/wheel/issue/116/packages-with-only-filesdata_files-get16:19
*** jaypipes has quit IRC16:20
*** adrian_otto has quit IRC16:20
fungiphschwartz: thank you for reviewing! (and thanks to everyone who is reviewing, and shame on me for not reviewing enough)16:21
phschwartzHey, I promised that I was going to get more involved.16:21
*** jgallard has quit IRC16:22
*** om has joined #openstack-infra16:22
*** e0ne has quit IRC16:22
phschwartzsdague: When you have some time, I am interested in here what your doing with devstack-vagrant :)16:23
*** adrian_otto has joined #openstack-infra16:23
*** thomasbiege1 has joined #openstack-infra16:25
sdaguephschwartz: sure, there was a whole thread on the mailing list about the fact that there were a bunch of these out there16:25
rdopieralskia question -- launchpad allegedly has some poll/voting functionality -- is there a way for me to set up a poll for horizon's new name?16:25
sdagueso was trying to consolidate down16:25
sdagueto a shared stackforge repo16:26
sdaguethough that's currently blocked on cla question16:26
phschwartzhmm, interesting that I haven't noticed it at all.16:26
*** dprince has joined #openstack-infra16:26
sdaguephschwartz: it was a couple months ago on -dev16:26
*** [1]trinaths has joined #openstack-infra16:26
dstufftfungi: awesome16:26
*** thomasbiege has quit IRC16:26
phschwartzBut then I now realize for the past week I am not getting mail from the lists.16:26
fungidstufft: digging in now but i think it's probably a trivial patch to is_pure()16:26
*** marcoemorais has quit IRC16:27
phschwartzfungi: mordred: Any chance one of you can see if the server for the lists is getting bounces from rackspace.com again?16:27
*** marcoemorais has joined #openstack-infra16:27
phschwartzsdague: I will have to look at the digests, ty16:27
*** denis_makogon_ has joined #openstack-infra16:27
*** trinaths has quit IRC16:27
*** [1]trinaths is now known as trinaths16:27
fungirdopieralski: that or a civs poll. were you wanting to use that to narrow down the top candidates before sending them to the foundation for vetting/research?16:27
sdaguephschwartz: http://lists.openstack.org/pipermail/openstack-dev/2014-April/032495.html16:27
phschwartzsdague: ty16:28
fungiphschwartz: checkingh16:28
rdopieralskifungi: yes, pretty much16:28
rdopieralskifungi: do you know how I would set it up?16:28
*** marcoemorais has quit IRC16:28
rdopieralskifungi: preferably, we would like horizon contributors to be able to vote16:28
*** marcoemorais has joined #openstack-infra16:29
fungirdopieralski: no, though ttx might have suggestions (and opinions on whether lp or civs is the better option there)16:29
*** marcoemorais has quit IRC16:29
*** james_li has quit IRC16:29
rdopieralskittx: can you help? :)16:29
*** ihrachyshka has joined #openstack-infra16:29
*** marcoemorais has joined #openstack-infra16:29
*** jgrimm has joined #openstack-infra16:29
ttxrdopieralski: actually polls i nLP are set up at the group level16:30
*** thomasbiege has joined #openstack-infra16:30
*** thomasbiege1 has quit IRC16:30
ttxrdopieralski: so depending on who you want to allow to vote, it may not work for you16:31
ttxfor public polls, you should use something like surveymonkey16:31
mordredor civs16:31
ttxfor clear electorate polls, you should use CIVS16:31
*** thomasbiege has quit IRC16:31
mordredrdopieralski: horizon is getting renamed?16:32
ttxfor pools over a Launchpad group (but who uses those those days) you can use LP polls16:32
fungimordred: half of horizon is getting renamed ;)16:32
rdopieralskimordred: it's getting split, and one part of it is getting renamed16:32
* ttx needs to run16:32
rdopieralskittx: any hint on how to find the polls on lanunchpad?16:33
*** jaypipes has joined #openstack-infra16:33
rdopieralskianybody? :)16:33
ttxrdopieralski: pools in Launchpad are done at a group level16:33
rdopieralskiso there is Horizon Developers group that looks right16:33
ttxrdopieralski: there is no public poll in LP16:33
openstackgerritA change was merged to openstack-infra/git-review: Ignore newline in bp/bug search in commit message  https://review.openstack.org/9832416:33
ttxrdopieralski: I doubt that it's current though16:33
fungiphschwartz: going back to late last week, the only rackspace.com rejects the lists server has logged are these... 2014-06-07 07:25:00 H=localhost (lists.openstack.org) [] F=<mailman-loop@lists.openstack.org> rejected RCPT <amesserl@rackspace.com>: Sender verify failed16:34
ttxrdopieralski: but let's use that as an example16:34
fungiphschwartz: any idea what the date/time on the last message you received was?16:34
*** annegent_ is now known as annegentle_16:34
ttxPoll creation there would be at https://launchpad.net/~horizon/+newpoll16:34
rdopieralskiah, not allowed, I see, I will neet to ask our ptl16:34
ttxYou probably need to be a team admin to be able to do that16:35
phschwartzfungi: 06-04 5:59pm est16:35
*** mriedem has quit IRC16:35
fungiphschwartz: oh, that long. i'll dig back further then16:35
ttxrdopieralski: and the team admins are way out of date, which gives you a hint as to whether the group is actually up to date16:35
phschwartzfungi: if you don't see anything it might be on this side.16:36
ttxteam admin is Devin Carlen16:36
rdopieralskittx: ok, I will use civs then, thank you16:36
*** adrian_otto has quit IRC16:36
*** markmcclain has joined #openstack-infra16:36
ttxwho has not been touching anything openstack since 201216:36
ttxrdopieralski: sounds like a better bet, yes. Also let you express preferences16:37
*** gondoi is now known as zz_gondoi16:37
ttxrather than arbitrary pick16:37
* ttx really runs now16:37
rdopieralskithank you!16:37
mordredttx: should we delete those teams?16:39
*** lakshmiS has joined #openstack-infra16:40
*** isviridov is now known as isviridov|away16:40
*** gokrokve_ has joined #openstack-infra16:40
*** aconrad has quit IRC16:41
*** mmaglana has joined #openstack-infra16:41
*** gokrokve has quit IRC16:43
fungiphschwartz: i show deliveries to you continually after that time, if you need message ids/next hops to track down in your receiving mta16:43
phschwartzfungi: no, I should need it. As long as they are coming it, the trashing of them by the spam filters should be easy to trace. ty16:44
fungiphschwartz: yw16:44
*** hcc has joined #openstack-infra16:45
*** hcc has quit IRC16:45
openstackgerritA change was merged to openstack-infra/jeepyb: Swap the boolean logic on copy_acl_config  https://review.openstack.org/9913616:46
*** ramashri has joined #openstack-infra16:46
phschwartzfungi: And fixed. The spam filter was getting me. I just received about 40 messages.16:48
*** mwagner_lap has quit IRC16:48
*** dkranz has quit IRC16:49
*** mriedem has joined #openstack-infra16:49
*** trinaths has quit IRC16:49
*** basha has joined #openstack-infra16:50
*** changbl has joined #openstack-infra16:50
phschwartzok, time to take a break and clear my head. That is 4 hours of reviewing16:51
*** wenlock has quit IRC16:52
*** hashar has quit IRC16:54
*** jcoufal has quit IRC16:55
*** melwitt has joined #openstack-infra16:56
*** _nadya__ has joined #openstack-infra16:56
*** james_li has joined #openstack-infra16:57
jogoI thought the proposla bot doesn't push out new hacking changes?16:59
Alex_Gaynorjogo: why wouldn't it?16:59
jogobecause you can't just bump hacking versions on there own17:00
*** denis_makogon_ is now known as denis_makogon17:00
jogoyou have to update the ignore list or fix things etc17:00
*** _nadya_ has quit IRC17:00
jogohacking is not backwards compatable accross 0.x.0 releases17:00
jogoin the sense that new rules are added17:00
Alex_GaynorPersonally I think it's fine, this is probably the best way to notify people they need to update their code17:01
jogoalso having issues with the requirments sync17:01
jogobut the patch that is proposed cannot be landed17:01
jogoalso g-r has a differnt version then what projects should have17:01
*** julim has quit IRC17:03
*** weshay has quit IRC17:04
*** weshay has joined #openstack-infra17:04
*** jpich has quit IRC17:04
jogoclarkb: o/17:05
*** doug-fish has quit IRC17:05
*** thomasbiege has joined #openstack-infra17:05
*** yjiang51 has joined #openstack-infra17:05
openstackgerritAndrew Melton proposed a change to openstack/requirements: Add IDMapShift for Nova  https://review.openstack.org/9888817:05
jogowho has been working on the requirements bot? and checks17:06
*** aconrad has joined #openstack-infra17:06
*** harlowja_away is now known as harlowja17:08
*** melwitt has quit IRC17:13
*** derekh_ has quit IRC17:15
*** zzelle_ has joined #openstack-infra17:15
clarkbjogo: what do you mean by that?17:16
fungijogo: i think the ideal solution there is that if a project sees proposed new hacking bumps breaking for them, they should first update their tox.ini to exclude the new things they need excluded and once that's merged the reqs proposal will magically pass17:16
openstackgerritSean Roberts proposed a change to openstack-infra/config: Add separate project for training-guides  https://review.openstack.org/9633417:16
*** basha has quit IRC17:17
mordredjogo: yeah - what fungi said. the system is working as designed17:17
mordredjogo: it's the same with any other requirement17:17
jogofungi: but the version limits in global requirements aren't what projects should sue17:17
mordredsometimes, to be able to land the req change, you have to git review -d it, fix the code, and re-upload17:17
mordredjogo: the version limits in global requirements should be what projects should use17:17
fungithey *could* just amend the proposal, but as we've seen in the past new proposal updates are frequent enough that they may not get through review before it overwrites them aith another update17:18
jogobut that would be a non overalaping change17:18
jogogo from hacking 0.8.x to 0.9.x17:18
jogoso global-reqs has support both 0.8 and 0.9 so that 0.8 stays in the mirror etc17:19
*** jgrimm has quit IRC17:19
*** HenryG has quit IRC17:19
*** melwitt has joined #openstack-infra17:19
fungijogo: i think overlap only matters for the global-reqs list. sometimes projects lag behind global-reqs long enough that non-overlapping version updates in a project's reqs is unavoidable17:19
zaropelix: did you do anything besides rebase this? https://review.openstack.org/#/c/4878317:20
fungithe overlap check we wanted to install was to avoid jobs running in the gate demanding newer hacking than we had updated into our mirror yet17:20
*** HenryG has joined #openstack-infra17:20
jogofungi: what about the mirror?17:20
*** otherwiseguy has quit IRC17:20
fungior s/hacking/whatever/17:20
fungijogo: what about the mirror?17:20
jogoolder versions will stay around?17:20
fungijogo: yes, indefinitely17:20
fungiwe don't purge things from our mirror when global-reqs no longer refers to them17:21
fungithey stick around forever17:21
pelixzaro: some rework around the tests/base.py due to another merged change17:21
pelixzaro: but besides that, it's just a rebase so that it'll merge cleanly17:22
*** rdopieralski has quit IRC17:23
openstackgerritJoe Gordon proposed a change to openstack/requirements: Update hacking version to align with what projects should use  https://review.openstack.org/9917217:24
jogofungi: in that case ^ should fix things17:24
*** doug-fish has joined #openstack-infra17:25
*** wenlock has joined #openstack-infra17:25
*** sabari has quit IRC17:25
*** markmcclain has quit IRC17:25
*** om has quit IRC17:26
*** wenlock_ has joined #openstack-infra17:26
*** dims_ has quit IRC17:28
*** markmcclain has joined #openstack-infra17:29
*** wenlock has quit IRC17:29
fungijogo: yep, as long as the earlier change to set it to hacking>=0.8.0,<0.10,!=0.9.0 already merged, that should be safe17:29
jogoat one point the requirments bot didn't push out new hacking versions I thought17:29
*** wenlock_ has quit IRC17:30
fungiwhich it seems to have17:30
*** wenlock has joined #openstack-infra17:30
fungijogo: i have no idea what would have prevented it from suggesting a sync with the hacking version in global requirements, other than a bug (the proposal bot was broken for extended periods of time last cycle)17:30
*** trinaths has joined #openstack-infra17:32
*** penguinRaider has joined #openstack-infra17:33
mriedemis there a config script somewhere that tells jenkins which files to archive for builds?17:33
*** ociuhandu has quit IRC17:33
mriedemnot seeing grenade status/*.failure logs archived17:33
jogofungi: hrm anyway looks like this should solve things17:33
*** aconrad has quit IRC17:34
*** wenlock has quit IRC17:36
openstackgerritElizabeth K. Joseph proposed a change to openstack-infra/publications: Minor updates, new gerrit and zuul screenshots  https://review.openstack.org/9917317:37
*** denis_makogon has quit IRC17:37
*** amotoki_ has quit IRC17:37
pleia2image diffs ftw!17:37
fungimriedem: that gets enumerated in openstack-infra/devstack-gate functions.sh17:37
zaropelix: do you think we need a 'yaml_loader' section or can include_path just get put under job_builder section in jenkins.ini?17:38
pelixprevious reviews complained when I had it that way :)17:38
pelixbut also it probably is a better separation in that other options could in theory be added to control how the yaml loading is performed which is entirely separate to how the jobs themselves are parsed/generated or uploaded17:39
*** trinaths has quit IRC17:39
*** wenlock has joined #openstack-infra17:39
*** duran has joined #openstack-infra17:40
pelixif we put the include_path under the job_builder section, what's the argument for separating any of the other options?17:40
*** jistr has quit IRC17:40
pelixeither way doesn't really bother me, as long as it seems consistent and is acceptable17:41
*** markwash_ has joined #openstack-infra17:41
*** dims_ has joined #openstack-infra17:42
*** thedodd has quit IRC17:42
jesusaurusCan I get some reviews on https://review.openstack.org/#/c/98549/ and https://review.openstack.org/#/c/98561/ ?  I would really like to fix the hostname issue on the hpcloud bare-precise images17:42
*** markwash has quit IRC17:43
*** markwash_ is now known as markwash17:43
zarogood point.  i'm asking because i just reviewed the recursive option. the proposal for that one is in job_builder.  i'm thinking these should be in same section?17:43
*** _nadya__ has quit IRC17:45
*** e0ne has joined #openstack-infra17:47
zaropelix: hmm, i can't seem to find any comments in the change discussing the jenkins.ini sections.17:47
*** wenlock has quit IRC17:48
pelixhmm, maybe I'm confusing it with another review?17:48
pelixor possibly just asked in here at the time17:48
*** sballe_ has quit IRC17:50
*** zns has joined #openstack-infra17:51
*** ildikov_ has quit IRC17:53
*** wenlock has joined #openstack-infra17:54
*** ildikov_ has joined #openstack-infra17:54
*** alkari has joined #openstack-infra17:56
*** ihrachyshka has quit IRC17:57
*** andreykurilin_ has joined #openstack-infra17:57
openstackgerritA change was merged to openstack-infra/elastic-recheck: Add fingerprint fur bug 1268725  https://review.openstack.org/9891317:57
uvirtbotLaunchpad bug 1268725 in openstack-ci "ConnectionError: HTTPConnectionPool(host='pypi.openstack.org', port=80): Max retries exceeded" [Medium,Triaged] https://launchpad.net/bugs/126872517:57
openstackgerritA change was merged to openstack-infra/elastic-recheck: Remove duplicate query for 1326820 and 1326855  https://review.openstack.org/9891917:57
devanandahi! any chance to get another infra core to review/approve this -- https://review.openstack.org/#/c/98886/ -- it's causing about half of ironic's tests to fail in teh gate right now17:58
devanandasee the E-R query for bug 132675417:58
uvirtbotLaunchpad bug 1326754 in tripleo "Ubuntu CI failures  : Unable to locate package ubuntu-cloud-keyring" [High,In progress] https://launchpad.net/bugs/132675417:58
devanandathat fix looks like it should help a lot, at least afaict17:59
*** e0ne has quit IRC17:59
fungijd__: the reply on https://github.com/smontanaro/pylockfile/issues/11 makes it sound like we should consider offering to absorb pylockfile (like we've done for sqlalchemy-migrate) or look at replacing use of it with a different module17:59
zaropelix: discussion about jenkins sections was in this change: https://review.openstack.org/#/c/5208018:00
clarkbdevananda: done18:01
*** jlibosva has quit IRC18:01
devanandaclarkb: tyvm!18:01
zaropelix: PS #1218:01
clarkbdevananda: you should see that in ~25 minutes or so18:01
clarkbdevananda: assuming it merges shortly18:01
pelixzaro: looking18:01
devanandaclarkb: fantastic.18:01
*** rwsu has joined #openstack-infra18:02
pelixzaro: guess since it's not encapsulated within the localyaml class, and is instead controlled by what is passed to that class, it makes sense to have it under the job_builder section to start with18:02
mordreddevananda: everyhting about that patch makes me sad18:02
mordreddevananda: why, in the name of GOD does a-a-r try to install ubuntu-cloud-keyring at run time?18:03
mordredshouldn't that just be a depend of a-a-r?18:03
zaropelix: i think i agree, doesn't seem like there's a need for seperate section in this case.18:03
mordredzul: ^^ I blame you18:03
*** jaypipes has quit IRC18:03
devanandaclarkb: fyi, we've already gotten 7 bug fixes landed today, with another ~10 on the way18:03
zulmordred:  what did i do or didnt do?18:03
pelixzaro: I was thinking about adding a safe_load option at some stage in case people wanted to explicitly disable within their configuration the possibility of yaml instantiating python objects. Does that follow on well enough?18:03
lifelessclarkb: https://review.openstack.org/#/c/98043/ <- any red flags on that? Its blocking us getting much better logs out of things18:04
clarkbsdague: question about 99016, doesn't devstack minimally do updates now? has that been insufficient?18:04
clarkblifeless: what does logs/** look like?18:05
*** mrodden has joined #openstack-infra18:05
clarkblifeless: is it logs/new/nova-api.txt and so on?18:05
*** ildikov_ has quit IRC18:06
*** changbl has quit IRC18:06
sdagueclarkb: the reason for trying 99016 is that we've got a pretty big failure rate due to apt mirrors being down18:06
clarkbsdague: even after the devstack change?18:06
zaropelix: you mean as another option in job_builer?18:06
pelixzaro: yeap, one that makes the yaml loading go through the safe loader instead of the default18:07
clarkbsdague: and why are we reverting the ceilometer poll interval change?18:07
sdagueclarkb: the devstack change is inbound, this is sort of orthonginal, but based on what I see in the devstack logs, I doubt the retry is going to help18:07
devanandamordred: where should d-g be pulling that instead?18:07
clarkbsdague: well importantly it only updates once18:07
sdaguemore interestingly, 50% of our nodepool images don't have an actual apt cache18:08
clarkbsdague: which I think is more important than the retry. Once we have a good update we don't update again18:08
sdagueclarkb: if it works18:08
openstackgerritA change was merged to openstack-infra/config: Update apt cache before installing UCA keyring  https://review.openstack.org/9888618:08
sdaguehonestly, I really doubt the blip was that quick18:08
sdagueit's not like they die on 1 update file18:08
clarkbsdague: yeah that was my concern too18:09
lifelessclarkb: seed_logs/nova-compute.log, seed_logs/nova-conductor.log, seed_logs/rabbitmq-server.log etc https://review.openstack.org/#/c/97947/2/toci_devtest.sh is the change that sets that up18:09
sdaguethey typically fail all of them18:09
clarkblifeless: thanks18:09
lifelessclarkb: the ubuntu logs aren't quite as pretty18:09
lifelessclarkb: the problem we're trying to solve is that we need N where N may be large copies of nova-conductor.log18:09
fungiclarkb: sdague: yeah, i pointed that out as well. i think it solves some classes of failure (and does reduce the number of times we connect) but won't solve the main ones we've seen recently i think18:09
devanandahuh, i just started getting 500 from github.com18:09
zaropelix: that makes sense to me.  can you tell me what the use case would be for safe load?18:09
lifelessand of mysql-server.log18:09
lifelessclarkb: so we can I guess prefix-mangle each file, if that will work better18:10
lifelessclarkb: but a directory structure just seems more intuitive18:10
anteayapeople being idiots18:10
sdagueclarkb: on the ceilometer revert, I'd like to take the polling back to the defaults18:10
clarkbyup works for me. just wanted to make sure we werent doing something like logs/var/log/upstart/foo18:10
lifelessclarkb: so we may do that on Ubuntu18:10
clarkbsdague: oh right I thought we had to set the value but the default is fine18:11
clarkbsdague: just needed to wire that up in my brain. thanks18:11
lifelessclarkb: we have two hierarchies there - the /mnt/state/var/log and the root /var/log one18:11
clarkblifeless: oh, please don't :P18:11
pelixzaro: protection from code injection security issues18:11
lifelessclarkb: the common path is /18:11
johnthetubaguyhey I had noticed lots of failed builds in Rackspace IAD for the node pool builds, noticed from our side, that should all be sorted now, I hope18:11
clarkblifeless: or at least make the prefix minimal18:11
fungijohnthetubaguy: thanks!18:11
lifelessclarkb: is that a UI concern, or system resources/performance?18:12
*** arnaud has quit IRC18:12
fungijohnthetubaguy: we've sort of become numb to instance boot errors in providers since they come and go so often any more18:12
clarkblifeless: a little of both. dir cleanup with find is a bit clunky right now so that deeper the tree the longer it takes to free those inodes18:12
fungijohnthetubaguy: unless they get so severe we're mostly out of resources to run things18:12
pelixzaro: you can use yaml to create python objects, the default yaml.load is only supposed to be used with trusted yaml code, while the yaml.safe_load is what should be used if you are accepting code from other sources18:12
clarkblifeless: and as a user digging into those deeper trees is a bit annoying18:12
clarkblifeless: eventually we will be all swifted then it wont matter18:13
johnthetubaguyfungi: yeah, that sucks, I kinda surprised things kept working given how many errors were happening, anyways should be sorted now18:13
clarkbsdague: fungi: was there any reason to keep the ceilometer poll interval? eg tests that needed to run? I think there are some proposed changes up now for that.18:13
ArxCruzIs there a command to upgrade nodepool db? latest upgrade create a new column in database, and nodepool is failing because this wasn't created18:13
sdaguewent through that with the ceilo team18:14
sdaguealmost all the proposed changes now don't need it18:14
zaropelix: ahh, thanks for the security lesson.  that would be great!18:14
clarkbArxCruz: you can either drop the entire DB and restart nodepool to recreate it (you will have to do manual cleanup of reosurces). Or you can add the column to the node table18:14
ArxCruzclarkb: okay, thanks :)18:14
sdagueit will also reduce the ceilometer API traffic by a factor of 4018:15
sdaguewhich I think is probably a better place to hang for now18:15
clarkbsdague: ++ approving18:15
*** wenlock has quit IRC18:15
lifelessclarkb: so, AIUI the choice we have is a) flatten the namespace ourselves, or allow for a hierarchy and iterate the test-script code towards a better shape18:15
*** aconrad has joined #openstack-infra18:15
clarkblifeless: right. d-g does flattening of the namespace for the devstack logs18:16
*** nikhil___ has joined #openstack-infra18:16
lifelessclarkb: but right now we bundle all the logs into one tar, which doesn't get indexed by logstash AIUI18:16
lifelessclarkb: and having lots (lots! for scale tests, coming soon to a crazy pants person near you)18:16
lifelessclarkb: of flattened logs terrifies me18:16
nikhil___hey, I'm looking to do a recheck on a MP and was wondering if the gates were looking okay or do you want us to wait a bit more?18:16
clarkblifeless: you have to do manual flatting of some sort across different nodes18:17
*** Alexei_987 has joined #openstack-infra18:17
phschwartzwho is the best person to reach out to with questions about the xenserver ci gating?18:17
lifelessclarkb: also searching for 'nova-conductor.log with foo in it' in logstash might be tricky if nova-conductor.log is seed_nova-conductor.log, undercloud-$uuid_nova-conductor.log etc18:17
clarkbphschwartz: BobBall is probably the best person18:18
clarkblifeless: no it should be seed/nova-conductor18:18
*** __afazekas is now known as afazekas18:18
*** adrian_otto has joined #openstack-infra18:18
johnthetubaguyphschwartz: thats me or BobBall18:18
anteayaphschwartz: BobBall18:18
lifelessclarkb: can only do that if we can turn the preservation of hierarchy on18:18
clarkblifeless: we grok basename correcntly18:18
johnthetubaguyI am trying to help BobBall with that at the moment18:18
anteayajohnthetubaguy: are you xenserver ci too?, awesome18:18
lifelessclarkb: I was investigating the consequences of doing flattening ourselves.18:18
clarkblifeless: I am saying flatten the hierarchy18:19
anteayanice to meet you, johnthetubaguy18:19
lifelessclarkb: now I'm confused18:19
johnthetubaguyanteaya: kinda, yeah, trying to get some help for Bob18:19
clarkbor whatever18:19
*** _nadya_ has joined #openstack-infra18:19
clarkbas long as basename is correct things should work18:19
anteayajohnthetubaguy: glad to hear it, Bob is awesome and its great you are helping him18:19
lifelessclarkb: sure, but thats lost when the scp flattens it18:19
phschwartzclarkb: anteaya: johnthetubaguy: ty18:19
clarkblifeless: right, I am on board with scp keeping the hierarchy18:19
lifelessclarkb: ok18:20
*** james_li has quit IRC18:20
lifelessclarkb: so now you're talking about what we should do to make the impact on storage cleanup and so forth tolerable18:20
johnthetubaguyanteaya: I am over at Rackspace in the cloud servers team, nova blueprint czar and such like, but starting to help Bob out, and trying to find someone with more time than me to help more!18:20
clarkblifeless: yup18:20
lifelessclarkb: I think we can do seed/mnt/nova-conductorlog and seed/root/nova-conductor.log18:20
clarkbsomething like that would be perfect thanks18:20
openstackgerritA change was merged to openstack-infra/elastic-recheck: Refine queries for tripleo jobs  https://review.openstack.org/9809618:20
lifelessclarkb: because we may have a name conflict between /var/log and /mnt/state/var/log18:21
lifelesswe shouldn't but it can happy18:21
lifelessclarkb: ok, so we'll get right on that; do you want to wait scp wise for that to be in place, or can we land the scp patch to get Fedora jobs sorted ?18:21
*** james_li has joined #openstack-infra18:22
clarkblifeless: I approved scp patch I think we are ok with having it in while you get sorted18:22
lifelesspcrews_: around ?18:23
*** rlandy has quit IRC18:23
pcrews_lifeless: si! what's up?18:23
*** melwitt has quit IRC18:24
*** melwitt has joined #openstack-infra18:24
lifelesspcrews_: tripleo + CI + thought you might like a pointer at something topical and gentle introduction18:24
lifelessI'm just filing a bug to capture state; gimme a sec18:25
*** dangers is now known as dangers_away18:25
clarkblifeless: fwiw we had to put the inode heavy doc builds on a different filesystem hence my slight concern here18:25
*** harlowja is now known as harlowja_away18:25
pcrews_lifeless: sounds great :)18:25
*** dims_ has quit IRC18:25
lifelessclarkb: yup. right now ubuntu is tarred up18:25
lifelessclarkb: so its when we stop tarring that we have to get it right18:26
lifelessjogo: around? stopping tarring *is* needed for logstashing, right ?18:26
*** gokrokve_ has quit IRC18:26
clarkblifeless: jogo yes18:27
lifelessok great18:27
* lifeless finishes filing ze bug18:27
anteayajohnthetubaguy: good to know your list, thanks, I hope you find that special someone, good luck in the search :D18:27
johnthetubaguyanteaya: hehe, thanks :)18:27
phschwartzare we currently logging info from instances that end up in ERROR state from node-pool? really best thing would be if we have logs that we can scrape to get uuid's18:28
*** melwitt has quit IRC18:30
johnthetubaguyphschwartz: are you looking at the rax-IAD errors that were happening before? (that were probably hurting the xenserver-ci too) or is this just a general thing?18:30
phschwartzrax-iad and rax-dfw errors18:30
lifelessclarkb: do we need to compress ourselves or will scp do that for us ?18:32
*** ildikov has joined #openstack-infra18:33
*** jgrimm has joined #openstack-infra18:33
*** adrian_otto has quit IRC18:33
fungilifeless: compressing before copy helps. compression in-transit is a waste since we have a mop-up cron job on the logs.o.o server which tries to find and compress any which weren't compressed prior to collection18:34
sarobim getting fail on check-projects-yaml-alphabetized https://review.openstack.org/#/c/96334/18:34
sarobany hints on how i can fix to pass?18:34
*** nati_ueno has joined #openstack-infra18:35
*** rlandy has joined #openstack-infra18:35
mordredphschwartz: yes, we are currently logging info18:35
fungiopenstack/training-guides needs to be added to review.projects.yaml in alphabetic order with respect to the other entries in that file18:36
fungisarob: ^18:36
phschwartzmordred: is there any way I can get access to the logs to scrub uuid's for error state instances for say a 1 week periood? or would there be a way I could be provided them.18:36
anteayafungi: I have tidied up the agenda for today: https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting_.2810_Jun_2014.2918:36
clarkblifeless: you need to compress yourselves18:37
anteayafungi: do you want me to keep last week's agenda or remove it18:37
openstackgerritA change was merged to openstack-infra/config: Keep the hierarchy of tripleo files copied to log server  https://review.openstack.org/9804318:37
anteayafungi: we tend to remove them18:37
fungisarob: between openstack/tempest and openstack/tripleo-heat-templates18:37
mordredphschwartz: sure. I can ship you a week of logs ... one sec18:37
anteayafungi: I am assuming you are chairing18:37
phschwartzmordred: awesome, ty18:37
fungianteaya: i usually remove entries i know don't need to be covered and leave topics which i think are being revisited18:37
anteayafungi: okay I will leave for you to clean up then18:38
lifelesspcrews_: whats your LP uid ?18:38
anteayaI can remove dups though18:38
openstackgerritBen Nemec proposed a change to openstack-infra/reviewstats: Oslo project updates  https://review.openstack.org/9918418:38
mordredphschwartz: don't thank me yet - you haven't seen the logs :)18:38
pcrews_lifeless: patrick-crews18:38
fungianteaya: oh, neat, when did we start separating last and next meeting agendas rather than just carrying forward? i clearly haven't been paying close attention18:38
*** srenatus has quit IRC18:38
*** srenatus has joined #openstack-infra18:39
anteayareed did that18:39
lifelessah, not a driver, can't assign to you18:39
lifelesspcrews_: grab https://bugs.launchpad.net/openstack-ci/+bug/1328645 :)18:39
uvirtbotLaunchpad bug 1328645 in openstack-ci "ubuntu tripleo-ci jobs are not logstash indexable" [Undecided,New]18:39
reedI deny18:39
anteayafungi: jeblair doesn't tend to, so I am cleaning them18:39
phschwartzmordred: I have scrapped web logs from a hosting companies control panels that services 290k customers with almost 195,000,000 requests per day. So don't worry, you won't scare me for that long ;)18:39
anteayareed: just that every project agenda has a different style18:39
lifelessok, I need to make C's lunch before the tripleo meeting, bbs18:39
* mordred tries harder to scare phschwartz18:39
anteayareed: you went with a style qa uses18:39
* phschwartz 18:40
anteayareed: that's all, no problem, just figuring out what fungi wants to chair today's meeting18:40
anteayafungi: did you want to talk about git-review this week?18:40
reedthanks for the explanation anteaya18:40
pcrews_lifeless: 'tis grabbed18:40
anteayaianw: more on fedora/centos7 this week?18:40
clarkbphschwartz: we do have a few billion things indexed in elasticsearch18:40
anteayareed: np :D18:40
clarkb~6 or so18:40
fungianteaya: nah, i think we covered what we needed to on that18:41
mordredphschwartz: http://nodepool.openstack.org/nodepool-log-week.tgz18:41
anteayafungi: very good18:41
mordredphschwartz: it's actually about 10 days18:41
mordredphschwartz: because the glob was easier18:41
phschwartzmordred: that would work :)18:41
clarkbsdague: on the indexing front adding 4 new hosts seems to have helped18:42
anteayafungi: I'll stick ianw in there anyway and we can see what he wants to say18:42
clarkbsdague: I think based on the current numbers if we can just keep up we should stay caught up18:42
mordredphschwartz: btw - there is no leading dir on that tarball18:42
clarkbsdague: but I may need to add a few more nodes18:42
mordredphschwartz: so untar it carefully18:42
clarkbmordred: boo!18:42
phschwartzmordred: 310m compressed. not bad.18:42
phschwartzmordred: ty for the warning18:42
anteayafungi: cleaned up: https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting18:43
fungianteaya: me too. hopefully we didn't step on one another's edits18:43
anteayaoh okay18:43
anteayawiki didn't complain for me18:43
anteayadid I ruin yours18:44
funginor me. good work ;)18:44
*** ilyashak_ has joined #openstack-infra18:44
mordredclarkb, fungi: should we change the nodepool apache config so that it just serves all of /var/log/nodepool? I'm pretty sure those logs are safe/clean18:45
*** ilyashak_ has quit IRC18:45
*** ilyashak_ has joined #openstack-infra18:45
*** ilyashak_ has quit IRC18:46
*** thedodd has joined #openstack-infra18:46
*** gokrokve has joined #openstack-infra18:46
fungimordred: is it possible for tracebacks to leak provider credentials from novaclient in our debug logs?18:48
*** annegentle_ has quit IRC18:48
fungils: cannot open directory: Permission denied18:48
*** pelix has quit IRC18:48
mordredfungi: I do not believe so - I do not believe we've ever seen that18:50
mordredjogo: ^^ opinions?18:50
clarkbmtreinish: comment on 9816218:50
mtreinishclarkb: oh I should abandon that we decided to keep the fail logs in the console just put them at the end instead of inline18:51
jogomordred: tracebacks won't leak credentials since they don't dump variables. but the nova console lgos do18:51
clarkbmtreinish: that works thanks18:51
jogoerr novaclient debug logs do18:51
clarkbsdague: in 97965 can we just echo "nprocs: $(nproc)"18:52
mordredjogo: I don't think we do novaclient debug in our debug logs18:52
fungimordred: i guess the question is whether anyone is likely to accidentally switch novaclient into debug mode in production to track down a bug18:52
clarkbsdague: /proc/cpuinfo is fairly verbose with lots of irrelevant info that the hypervisor lies to us about18:52
*** hashar has joined #openstack-infra18:52
*** olaph has joined #openstack-infra18:52
fungimordred: though if we don't capture it, then i guess that's safe18:52
mordredfungi: nod18:52
jogosdague: ^18:53
*** e0ne has joined #openstack-infra18:55
*** pvo has quit IRC18:55
*** rlandy has quit IRC18:56
*** markmcclain has quit IRC18:56
*** rfolco has quit IRC18:57
*** dkliban is now known as dkliban_afk18:57
*** rfolco has joined #openstack-infra18:57
*** talluri has joined #openstack-infra18:57
*** nati_ueno has quit IRC18:57
*** marun is now known as marun_afk18:59
ianwanteaya: no, not this week19:00
fungimeeting time... -> #openstack-meeting19:01
*** spiffxp_ has quit IRC19:01
*** SlickN1k has quit IRC19:01
clarkbjesusaurus: I reviewed 9856119:02
*** bknudson has quit IRC19:02
*** arnaud has joined #openstack-infra19:03
*** zz_gondoi is now known as gondoi19:03
*** harlowja_away is now known as harlowja19:03
openstackgerritEmilien Macchi proposed a change to openstack-infra/config: ceilometer: enable gate-grenade-dsvm-forward  https://review.openstack.org/9743019:03
openstackgerritEmilien Macchi proposed a change to openstack-infra/config: ceilometer: enable gate-grenade-dsvm-forward  https://review.openstack.org/9743019:04
*** thedodd has quit IRC19:04
*** thedodd has joined #openstack-infra19:05
lifelessclarkb: mordred says you're hitting a wall with dib19:06
lifelessclarkb: if you want to brainstorm (here, or voice or whatever, I'd be delighted to help)19:06
clarkblifeless: yes the service things are wonky19:06
clarkblifeless: can do after meeting and lunch19:06
*** praneshp_ has joined #openstack-infra19:06
lifelessclarkb: reach out to me whenever19:07
*** sarob has quit IRC19:07
*** rdopieralski has joined #openstack-infra19:07
mordredlifeless: tl;dr we need to run database commands at image build time. we're also worried that lennart, being lennart, is about to tie systemd versions to kernel versions which will make building modern fedora on ubuntu lts problematic19:07
*** praneshp has quit IRC19:08
*** praneshp_ is now known as praneshp19:08
lifelessmy opinion of lennart is unprintable19:08
lifelessI saw that particular thread19:09
openstackgerritA change was merged to openstack-infra/config: StoryBoard module bounces apache on update  https://review.openstack.org/9851219:09
lifelessUbuntu had a similar issue with upstart in chroots at one point19:09
lifelessI can't believe fedora folk won't notice and lose their shit19:09
*** nati_ueno has joined #openstack-infra19:09
*** dims_ has joined #openstack-infra19:10
*** marcoemorais has quit IRC19:11
*** marcoemorais has joined #openstack-infra19:11
*** marcoemorais has quit IRC19:12
clarkbright I think there are some fundamental issues using lightweight separation of resources when doing cross platform image builds19:12
clarkbdib users that have a consistent platform they deploy on will have no issues19:12
*** marcoemorais has joined #openstack-infra19:12
clarkbbut we have to support 2 ubuntu LTS versions, 2 centos versions, and at least one fedora version19:12
lifelessso we can add cross-platform 'gate' tests to tripleo-ci19:14
openstackgerritJoshua Harlow proposed a change to openstack/requirements: Updated taskflow now that 0.3.x is released  https://review.openstack.org/9918819:14
*** dangers_away is now known as dangers19:16
fungiperhaps pottering wants to deprecate chroots19:16
lifelesspcrews_: ok so I'm back if you want to talk about that bugs details, or you can follow your nose and ask me/derekh/spamaps/etc if you get lost19:16
lifelessfungi: and containers... no19:16
fungilifeless: heh, right19:16
lifelessI think their answer will be 'build on a latest kernel and its all fine'19:17
lifelesse.g. you can build old stuff on new, but not new stuff on old.19:17
harlowjaqq, is it supported to do oslo development releases yet, taskflow just released 0.3.21 (prior 0.3 push failed) and would like to make it so that version numbering starts from 0.4.0.a0 (or similar), is all that is needed is that i tag the current master and the rest will happen automatically (with regards to versioning)19:17
lifelessI'm not endorsing this argument, just observing that it is the obvious solution when one side of an API cares about backwards compat and the other doesn't.19:17
* fungi rather likes being able to develop more recent debian in chroots on stable debian, but maybe that's greybeardthink19:17
*** _nadya_ has quit IRC19:17
pcrews_lifeless: ack - been reading up on it and making sure I grok it.  Have to tend to my 4-legged master quickly, but will likely have some questions when I dive back into things.19:18
*** signed8bit has quit IRC19:18
fungiharlowja: it should--and prereleases should now generate only wheels, so that older pip won't upgrade to them19:18
harlowjafungi cool19:19
harlowjaso i'll try this, we have one patch going through gating, i guess once that merges, i can tag that commit with 0.4.0.a0 and then further patches will be in the 0.4.0a? series19:19
lifelessharlowja: btw see me reply on the pbr semver spec?19:20
harlowjalifeless saw it, ya its all cool, and understandable :)19:20
harlowjagood to match with a use-case so u know it works out19:20
harlowjajust after that gets goign i'll probably come in with rpm support :-P19:20
harlowjalifeless https://review.openstack.org/#/c/95234/ (for example)19:21
harlowjawhich is similar to yours ;)19:21
harlowjasound good? :)19:21
lifelessharlowja: I was thinking it would be good to get the rpm use case into the current spec19:22
*** ociuhandu has joined #openstack-infra19:22
lifelessharlowja: if you can give the high level description ....19:22
harlowjalifeless sure19:22
*** amotoki has quit IRC19:22
phschwartzmordred: would you mind if I opened a review against node-pool and changed the format of the logging for LaunchStatusException? Right now it spits out uuid, node id, and status. Would make tracing easier if it also had the provider.19:22
*** duran has quit IRC19:22
*** signed8bit has joined #openstack-infra19:22
harlowjalifeless let me write something up, shall i email it to u? or put a diff somewhere19:22
*** gyee has quit IRC19:23
lifelessput it as a review comment in the spec, where you'd like it added, and I'll incorporate it when I tweak for dhellmann's latest feedback19:23
openstackgerritIvan Kolodyazhny proposed a change to openstack-infra/config: Make 'jenkins-job-builder' repo configurable  https://review.openstack.org/9919019:23
harlowjalifeless k19:23
lifelessfungi: you're such a greybeard :P19:25
*** afazekas has quit IRC19:26
*** sarob has joined #openstack-infra19:27
*** sweston has joined #openstack-infra19:27
*** sweston has quit IRC19:30
*** Sukhdev has joined #openstack-infra19:31
*** amcrn has joined #openstack-infra19:31
fungilifeless: i'm staving off grey hairs, but not so good at growing beards unfortunately19:32
*** pdmars_ has joined #openstack-infra19:33
*** ramashri has quit IRC19:35
*** pdmars has quit IRC19:35
openstackgerritIvan Kolodyazhny proposed a change to openstack-infra/config: Make 'jenkins-job-builder' repo configurable  https://review.openstack.org/9919019:36
*** arnaud has quit IRC19:38
*** Sukhdev has quit IRC19:40
*** bknudson has joined #openstack-infra19:40
*** sarob has quit IRC19:42
openstackgerritA change was merged to openstack-infra/config: Added 'docs' job for manila project  https://review.openstack.org/9846519:43
*** talluri has quit IRC19:44
openstackgerritA change was merged to openstack-infra/config: except ImportError when importing cPickle on Python3  https://review.openstack.org/9836719:46
*** thedodd has quit IRC19:47
jerryzlifeless: see my comments on your spec. both nodepool and jenkins are working with jump host now19:48
lifelessdhellmann: avoiding roundtrips - I've replied to your comment in the spec19:48
lifelessjerryz: awesome!19:48
*** doude has quit IRC19:48
*** thedodd has joined #openstack-infra19:50
*** gondoi is now known as zz_gondoi19:50
openstackgerritAnne Gentle proposed a change to openstack-infra/config: Add separate project for training-guides  https://review.openstack.org/9633419:51
*** zz_gondoi is now known as gondoi19:51
*** e0ne has quit IRC19:51
lifelessjerryz: I will review later today19:52
*** e0ne has joined #openstack-infra19:52
openstackgerritA change was merged to openstack-infra/config: Use the correct name for apache  https://review.openstack.org/9799419:52
lifelesscody-somerville: / mordred: you may find the no-public-ip spec stuff from jerryz and me interesting too19:52
*** signed8bit has quit IRC19:53
*** signed8bit has joined #openstack-infra19:54
*** marun_afk is now known as marun19:55
jesusauruslifeless: jerryz: is there a plan for how to segregate ip space for the jump hosts?19:56
*** e0ne has quit IRC19:57
mordredphschwartz: sounds good to me - clarkb, fungi ^^ see phschwartz above - any thoughts?19:57
*** marcoemorais has quit IRC19:57
*** marcoemorais has joined #openstack-infra19:58
lifelessjesusaurus: what do you mean?19:58
dhellmannlifeless: ack, I'll read up after the TC & release meetings19:58
jesusauruslifeless: managing a single pool of slaves behind one jump host is easy, but what happens when a second provider is added that also wants to use a jump host?19:59
clarkbmordred: phschwartz shouldn't be a problem20:00
clarkbit isn't hard to grep for that info though you just have to do a "join"20:00
jesusauruslifeless: the best way i can see to manage that is to have one ProxyCommand under Host 10.0.* and another under 10.1.*20:00
*** rdopieralski has quit IRC20:00
jesusauruslifeless: which means that we should have some plan for how to allocate 10.0.* and 10.1.*20:00
*** jaypipes has joined #openstack-infra20:00
openstackgerritA change was merged to openstack-infra/config: Update launch script and instructions  https://review.openstack.org/9784120:01
lifelessjesusaurus: separate jump host20:01
clarkbanteaya: fungi: ok that isn't clear from the etherpad20:01
anteayayes, well I was hoping I would have more time20:02
lifelessjesusaurus: I'm not sure why the IP range matters at all20:02
anteayabut reed's topics were important20:02
lifelessjesusaurus: can you explain ?20:02
clarkbanteaya: gotcha20:02
clarkbanteaya: in any case I am happy with the guidelines as long as we don't require existing accounts to change usernames20:02
*** signed8b_ has joined #openstack-infra20:02
*** james_li has quit IRC20:02
*** markmcclain has joined #openstack-infra20:02
anteayaclarkb: so as it stands I am hoping we can change the full name as fungi said and have new usernames and full names match20:02
anteayaclarkb: kk, thanks20:03
fungiphschwartz: sounds good to me. we've added it other places20:03
anteayaI am also hoping to convince fungi to let me set up a list of people willing to voluntarily change their usernames20:03
*** signed8bit has quit IRC20:03
anteayabut that is new info20:03
clarkbanteaya: that would be fine as well20:03
anteayaand I might be crushed by fires20:03
clarkbanteaya: as we should avoid surprises and confusion goign that route20:03
anteayasince that might require a window with no fires20:04
clarkbbtu ya you might not get any volunteers20:04
fungiyeah, it's just that their ability to use the account will be broken until they update their systems to match what they're changed to in gerrit20:04
anteayaso I'm okay to ask for volunteers for that and not force anyone?20:04
anteayaor worry about folks I can't find20:04
jesusauruslifeless: if you want to use ProxyCommand in .ssh/config to transparently configure the jump host, then the ProxyCommand will either apply to all ssh connections, or will be tied to a Host in .ssh/config and if you tie the jump host to a Host entry, then you either have a unique entry for every target host that goes through a jump host (and nodepool would need a way to dynamically modify that), you20:05
jesusauruscan * against a dns entry, or you can * against an IP range20:05
*** changbl has joined #openstack-infra20:06
clarkbwe will probably do something like zuuls ssh command wrapper (I haven't looked at any actual implementations though)20:06
jesusauruslifeless: or am I completely confused about what you are trying to do with .ssh/config?20:06
clarkbok I need to grab lunch now20:07
jesusaurusclarkb: how is zuul's ssh wrapper configured?20:07
clarkbback soonish to talk about dib more and things20:07
clarkbjesusaurus: its a bit git specific but it is just a shell script zuul spits out so that git + ssh does the correct thing20:07
*** e0ne has joined #openstack-infra20:08
clarkbbut you could probably do something similar with ssh itself20:08
jesusaurushuh, i'll take a look at that soonish20:08
clarkbwhere ssh-wrapper is actually ssh -o ProxyCommand20:08
*** e0ne has quit IRC20:10
sdaguemordred: we definitely do neutronclient20:10
*** e0ne has joined #openstack-infra20:10
*** pcrews_ has quit IRC20:11
*** nati_ueno has quit IRC20:11
lifelessjesusaurus: you can match on the hostname20:11
mordredsdague: ENOCONTEXT20:11
*** pcrews has joined #openstack-infra20:11
lifelessjesusaurus: rather than sshing to the IP20:11
lifelessjesusaurus: we can map the provider20:11
*** pcrews_ has joined #openstack-infra20:11
jesusaurusoh, right, it doesnt need to be a dns entry20:12
*** pcrews has quit IRC20:12
clarkbmordred: I think logs for nodepool20:13
clarkbok really need lunch now20:13
mordred"have gotten it to work by parsing nodepool.yaml, reading parameters like proxyhost, proxyuser, proxykey and joining them into a proxycommand used by paramiko.ProxyCommand."20:13
mordred^^ this == yes20:13
*** e0ne has quit IRC20:13
* mordred does not want this directly in a .ssh/config file - but if it's in nodepool.yaml and we can tell paramiko to do the right hting, then AWESOME20:13
mordredso, basically, jerryz++20:14
*** duran has joined #openstack-infra20:15
*** denis_makogon has joined #openstack-infra20:15
sdagueclarkb: so on the cpu thing, I'd kind of like to understand the differences in runs, even if they are lying to us20:16
*** homeless has quit IRC20:16
sdaguebecause those lies will impact the performance of c libs20:16
sdagueand it's information we can't ever get again if we don't dump it with the run20:17
sdagueespecially when we see one node actually time out20:17
*** homeless has joined #openstack-infra20:17
sdaguelike take > 2hrs on a tempest run without errors20:17
reedis there an API call to figure out which email addresses have signed the iCLA on gerrit?20:17
fungireed: yes, i can get it for you after the tc and project meetings, unless someone else wants to dig up the member query api call and the icla group name20:18
*** homeless has quit IRC20:19
*** nati_ueno has joined #openstack-infra20:19
harlowjalifeless pbr-versioning comments added20:19
reedfungi, i found the Group UUID, if that's all is needed then it's easy :)20:19
*** miqui has quit IRC20:20
*** homeless has joined #openstack-infra20:20
fungireed: you can query by group name too, pretty sure20:20
harlowjadprince yt, u might/probably are interested in https://review.openstack.org/#/c/96608/ (be nice to have a better rpm expert than me there, haha)20:20
reedfungi, indeed, I'm just failing at authenticating via curl right now :)20:20
*** e0ne has joined #openstack-infra20:21
fungireed: yeah, there is also the ssh api... https://review.openstack.org/Documentation/cmd-ls-members.html#_examples20:22
fungireed: i haven't played around with the rest api much yet20:22
reedfungi, thanks20:22
fungilooks like the rest api may want you to first query for the group uuid and then use it to query for the member list after all20:23
fungireed: to authenticate to the rest api, you need to go to your account settings in gerrit and click a button to generate a reusable http password, which you can then pass for authentication in your calls20:23
*** afazekas has joined #openstack-infra20:24
*** gondoi is now known as zz_gondoi20:25
*** dmsimard has joined #openstack-infra20:26
*** zz_gondoi is now known as gondoi20:27
*** arnaud has joined #openstack-infra20:27
*** devkulkarni has joined #openstack-infra20:31
*** arunkant has quit IRC20:33
*** kmartin has quit IRC20:33
*** AaronGr has quit IRC20:33
*** nijaba has quit IRC20:33
*** plars-away has quit IRC20:33
*** jeremyb has quit IRC20:33
*** pfalleno1 has quit IRC20:33
*** BadCub has quit IRC20:33
*** enikanorov_ has quit IRC20:33
*** esmute has quit IRC20:33
*** dguerri has quit IRC20:33
*** phschwartz has quit IRC20:33
*** timrc-afk has quit IRC20:33
*** ogelbukh has quit IRC20:33
*** ryanpetrello has quit IRC20:33
*** Sukhdev has joined #openstack-infra20:33
*** esmute has joined #openstack-infra20:33
*** pfallenop has joined #openstack-infra20:33
*** dguerri has joined #openstack-infra20:33
*** BadCub has joined #openstack-infra20:33
*** phschwartz has joined #openstack-infra20:33
*** jeremyb has joined #openstack-infra20:33
*** kmartin has joined #openstack-infra20:33
*** kmartin has quit IRC20:33
*** kmartin has joined #openstack-infra20:33
*** arunkant has joined #openstack-infra20:33
*** timrc-afk has joined #openstack-infra20:33
*** ryanpetrello has joined #openstack-infra20:33
*** AaronGr has joined #openstack-infra20:33
*** timrc-afk is now known as timrc20:33
*** melwitt has joined #openstack-infra20:34
*** nijaba has joined #openstack-infra20:34
*** ogelbukh has joined #openstack-infra20:34
*** devkulkarni has quit IRC20:35
*** devkulkarni has joined #openstack-infra20:35
*** plars has joined #openstack-infra20:35
lifelessharlowja: so in rpm20:37
*** dhellman_ has joined #openstack-infra20:37
lifelessharlowja: how do you sort something *before* a release ?20:37
*** thuc has joined #openstack-infra20:37
*** sarob has joined #openstack-infra20:38
dprincelifeless: latest release number plus the git commit number in the release20:38
lifelessharlowja: ~ says 'sort before' -20:38
lifelessdprince: that sorts before ?20:38
*** dmsimard has left #openstack-infra20:38
lifelesse.g. let me give a concrete example, we have three versions that should sort A < B < C20:39
lifelessA = 1.2.0 'dev3'20:39
lifelessB = 1.2.0 'dev4'20:39
*** gilliard_ has joined #openstack-infra20:39
lifelessC = 1.2.020:39
*** plomakin_ has joined #openstack-infra20:39
*** doude has joined #openstack-infra20:41
*** dhellmann_ has joined #openstack-infra20:41
*** zns has quit IRC20:42
*** sarob has quit IRC20:43
fungi~ was still a debianism last i recall20:43
mesteryInfra folks: I've surveyed the current state of Neutron 3rd party CI systems, and needless to say, it's not very good.20:43
phschwartzThe error state builds in graphite contain in the count errors that are not related to the actual instance building, but the bootstrap of it also correct?20:43
mesteryanteaya and I are wokring on making this better.20:43
dprincelifeless: Yeah I suppose those would need to be special case. Our pre-release tagging is sort not good in this regard20:44
mesteryI have an ehterpad which lists the current state of CI systems for Neutron here: https://etherpad.openstack.org/p/ZLp9Ow3tNq20:44
mesteryAnd here is how these will get better: https://wiki.openstack.org/wiki/NeutronThirdPartyTesting20:44
mesteryFYI, as once I send an email with deadlines to fix this, folks may bounce in here for help.20:44
fungilifeless: what i've seen for rpm is backversioning like 1.1.99really1.2.0dev320:44
mesteryI'll be around to help as well, but wanted to share this info with eveyrone here.20:44
*** HenryG has quit IRC20:45
lifelessdprince: we're trying to design it in the pbr semver spec - see oslo-specs; I added you as a reviewer last week:)20:45
dprincelifeless: I see that now, will have a look20:46
mordredphschwartz: that's right. "error" for us means anything that did not result in an instance that works20:46
mordredphschwartz: we do most of our bootstrap in image build - so there isn't a ton post-boot20:47
fungiit's useful to distinguish between "nodepool error state" and "nova error state" there20:47
mordredphschwartz: but we do go in and check for things like outbound network connectivity working etc20:47
anteayamestery: let's ensure there is a link to http://ci.openstack.org/third_party.html in the email that goes out20:47
mesteryanteaya: OK, will do, and thanks!20:48
*** dhellmann_ has quit IRC20:48
funginova error state gets spotted by nodepool right away, while something like not eventually becoming reachable takes a while for nodepool to figure out and give up20:48
*** dhellmann_ has joined #openstack-infra20:48
*** sabari has joined #openstack-infra20:48
*** gyee has joined #openstack-infra20:49
phschwartzmordred: I will have to look more at the logs to see if I can scrap that info out of them as well, but so far I have all the instances that threw the exception of LaunchStatusException with the instance in ERROR state.20:49
phschwartz~/src/openstack/logs ? cat rax-iad | wc -l20:49
phschwartz   7223020:49
fungithough we *have* i believe also seen nodes transition from building to error to ready in the past as well20:49
phschwartzouch is all I can say.20:49
mordredphschwartz: wow20:49
phschwartzThat is in the 9 days of logs you gave me20:49
mordredphschwartz: so, one of the things I was noting to pvo earlier20:49
*** sballe has joined #openstack-infra20:49
mordredis that if a cloud is erroring, we actually wind up REALLY hammering it20:49
mordredbecause we don't get a node, so we ask for a new one quickly20:49
mordredit made me think that we should put in a backoff threshold20:50
phschwartzexactly and since I am working towards using our public cloud for our internal infra setup, it is defn a worry20:50
mordredlike, if 50% of the requests in the last X period of time are errors, chill out for a bit with that cloud20:50
anteayamestery: can we also have a link to the third-party meeting as well?20:50
mordredbecause the increased requests could actually be making it _harder_ to get a good node20:50
mordredfungi, clarkb ^^20:50
lifelessdprince: so if using .99 is the right way in RPM world, we can do that20:51
fungimordred: tcp backoff20:51
*** gyee has quit IRC20:51
lifelessdprince: my goal isn't to be perfect, its to meet the broad requirements at the top in a systematic fashion20:51
*** thuc has quit IRC20:51
fungimordred: it's a good algorithm to reuse there i think, and we can tweak the scaling on it as we see fit20:51
*** thuc has joined #openstack-infra20:52
fungisomething along the lines of truncated binary exponential backoff maybe20:53
mordredfungi: ++20:53
*** gyee has joined #openstack-infra20:53
*** bknudson has quit IRC20:54
*** amcrn has quit IRC20:54
*** olaph has quit IRC20:54
*** mmaglana has quit IRC20:54
*** dhellmann has quit IRC20:54
*** bogdando has quit IRC20:54
*** achuprin has quit IRC20:54
*** michchap has quit IRC20:54
*** masayukig has quit IRC20:54
*** bradm has quit IRC20:54
*** mattymo has quit IRC20:54
*** plomakin has quit IRC20:54
*** openstackgerrit has quit IRC20:54
*** gilliard has quit IRC20:54
*** dhellmann_ is now known as dhellmann20:54
fungiwhat i just heard is that phschwartz just volunteered to write the patch because we're hammering his cloud ;)20:54
*** e0ne has quit IRC20:54
*** adrian_otto has joined #openstack-infra20:55
mordredfungi: :)20:55
phschwartzfungi: I can do that. Shouldn't be that hard of an algorithm exponentially increase the wait time to launch more instances till it is actually passing. Key would be the ratio of wait to retry.20:55
fungiyeah, i expect there will be some tuning20:55
*** olaph has joined #openstack-infra20:56
fungibut it will probably also help make nodepool more sane too, because it goes into some real fits when, for example, quotas cause its requests to get rejected20:56
*** nati_ueno has quit IRC20:56
*** dhellmann has quit IRC20:56
*** bogdando has joined #openstack-infra20:56
fungiand it starts rapidly looping them even though there's no hope20:56
*** dhellman_ is now known as dhellmann20:56
*** thuc has quit IRC20:56
phschwartzI have noticed that it falls over bad if it gets pigeon holed into a cell and that cell is over commited.20:57
*** mattymo has joined #openstack-infra20:58
mesteryanteaya: I will add that link as well, good idea!20:59
anteayamestery: thanks20:59
*** enikanorov_ has joined #openstack-infra20:59
fungiphschwartz: yeah, better distribution from the provider's scheduler would be preferable i think20:59
*** ArxCruz has quit IRC20:59
*** e0ne has joined #openstack-infra21:00
mordredfungi, clarkb ^^ SO - it seems rax-iad has neutron now21:00
mordredso some of the thigns we did with hp 1.1 may need doing with rax-iad21:01
mordreddo we remember them all?21:01
mordredone of the issues was hitting fixed ip limits, yeah?21:01
fungii believe so21:02
fungibasically creating separate networks so that we don't approach 256 nodes in a /2421:02
mordredphschwartz: you can see here: http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/templates/nodepool/nodepool.yaml.erb#n34721:02
*** gondoi is now known as zz_gondoi21:02
mordredphschwartz: that we're tying a config to a particular network on our side21:03
*** pdmars_ has quit IRC21:03
fungibut part of that may have had to do with hpcloud's default choices21:03
mordredand we made five networks within the b region there21:03
*** zz_gondoi is now known as gondoi21:03
*** adrian_otto has quit IRC21:03
*** adrian_otto1 has joined #openstack-infra21:03
mordredfungi: it may do - but if we're seeing similar rampant failures in iad, then applying similar us-side mitigations might be friendly21:03
fungimordred: sure enough21:04
*** gondoi is now known as zz_gondoi21:04
*** nati_ueno has joined #openstack-infra21:05
bodepddo you guys use heat templates to deploy out infrastrcuture?21:05
*** CaptTofu_ has quit IRC21:06
*** mwagner_lap has joined #openstack-infra21:06
*** hashar has quit IRC21:06
fungibodepd: i think we want to, but we need standalone heat21:07
bodepdfungi: why?21:07
fungibodepd: or rather, it's been suggested that we may want to eat that dogfood if we're hungry21:07
*** CaptTofu_ has joined #openstack-infra21:07
bodepdfungi: oh cool, I didn't know what existined21:08
* bodepd researches21:08
*** bknudson has joined #openstack-infra21:08
*** amcrn has joined #openstack-infra21:08
*** mmaglana has joined #openstack-infra21:08
*** achuprin has joined #openstack-infra21:08
*** michchap has joined #openstack-infra21:08
*** masayukig has joined #openstack-infra21:08
*** bradm has joined #openstack-infra21:08
*** openstackgerrit has joined #openstack-infra21:08
fungibodepd: why would we use heat templates, or why would we need standalone heat?21:08
phschwartzmordred: actually, I am not sure at all that it is neutron causing the issue.21:08
phschwartzNeutron is active in all our regions21:08
bodepdfungi: let me read a bit. I was more of the 'what the heck is it'21:09
fungibodepd: the standalone functionality need stems from the providers donating resources to us not offering heat services21:09
mordredphschwartz: oh - neat21:09
phschwartzand less then 300 error state in 9 days between ord and dfw21:09
*** marcoemorais has quit IRC21:09
mordredbodepd: we don't right now, even with standalone heat, for a couple of reasons21:09
*** marcoemorais has joined #openstack-infra21:09
bodepdfungi: my heat research is based on the fact that ansible is gpl :)21:09
*** adrian_otto has joined #openstack-infra21:09
*** zz_gondoi is now known as gondoi21:10
*** adrian_otto1 has quit IRC21:10
mordredbodepd: one is multi-cloud -although our main infra control plane is ina  single region, so that's not a rEAL problem21:10
mordredbut the second is that heat's current response to failure is to just give up21:10
bodepdmordred: gotcha.21:10
mordredthere are folks working on that21:10
mordredbut it's not ready for us yet21:10
*** rfolco has quit IRC21:10
mordredbodepd: is the GPL part giving you problems somewhere?21:10
phschwartzbrb, heading to the store21:10
bodepdmordred: it just doesn't align well with contract terms present or future21:11
mordredbodepd: hrm. the fact that ansible is gpl shouldn't really matter if you're not forking it21:12
mordredbodepd: ansible $things won't be gpl themselves21:12
mordredbut, you know - I'd LOVE to use heat as the thing that deploys our things21:12
*** adrian_otto1 has joined #openstack-infra21:12
*** sweston has joined #openstack-infra21:12
mordredonce it can handle our brand of crazy21:13
bodepdpeople are just gpl wary for toolchains. It's not all based on logic.21:13
mordredbodepd: they should go look at the license of ls21:13
*** doude has quit IRC21:13
mordredactually, tbh - given our current rate of adding new control plane services, and now taht heat has "adopt-stack" it probably is worthwhile looking at21:13
mordredif we update the stack and it doesn't work, it _does_ just stop, which is what we'd want it to do21:14
bodepdI just need a way to make the base infra self building21:14
mordredand then we can go remedy21:14
mordredbodepd: tell me about it21:14
*** adrian_otto has quit IRC21:14
*** doude has joined #openstack-infra21:14
bodepdI have to choose tools,make them work, etc... :)21:14
*** shivharis has joined #openstack-infra21:15
*** nati_ueno has quit IRC21:15
shivharisfungi: ping21:15
fungishivharis: what's up?21:16
shivharisfungi: hi how are you?21:16
shivharisfungi:how do i look up an account on review.openstack.org if I know the username21:17
shivharise.g. brocade_jenkins21:17
*** dprince has quit IRC21:17
clarkbok back from lunch21:17
reedfungi, I believe the list of members of ICLA group is only visible to admins21:17
reedfungi, I'm talking about gerrit API21:17
clarkbmordred: yes we need to bump a ton of quotas (this is arguably why I find neutron in public cloud by default a bit annoying. It allows me to do amazing things but with a very high overhead when I just want node + public IP)21:18
fungireed: oh, you may be right. i think i need to add an account to the owners group for it to enable that, but we can add that fairly easily21:18
clarkbmordred: specifically we need to bump network quota, subnet quota, fixed IP quota, floating ip quota, and router quota21:18
mordredphschwartz: ^^21:18
mordredclarkb: it's possible that the rax folks took care of that already21:18
fungishivharis: what do you mean by "look up an account"?21:19
fungishivharis: what do you need to know about the account?21:19
reedfungi, let me know21:19
shivharisi wanted to change the name of the account, not the username21:19
clarkbsdague: ok I can live with that21:20
shivharisfungi: possible?21:20
fungishivharis: i believe we have to do that manually. if this is in response to anteaya's plan for renaming third-party ci accounts consistently, that's going to be done for you. you won't need to do anything21:20
clarkbmordred: fungi: agreed a tcp like back off would probably work well21:20
*** markmc has quit IRC21:20
clarkbmordred: fungi we can do that simply be tweaking the rate factor in a manager itself21:20
shivharisfungi: yes it is in prep for that. I'll wait as u suggest21:21
clarkbmordred: fungi: phschwartz I think that change should be relatively straightforward since all of that code is failry well self contained21:21
clarkblifeless: is now still a good time to talk dib?21:22
lifelesslet me wrap up this OSRB call first21:22
shivharisfungi: sorry one more question. How can I trigger a build with updating a patch. I mean not just 'jenkins' but all CI?21:22
shivharisfungi: or a specific CI21:23
*** alkari has quit IRC21:23
*** gondoi is now known as zz_gondoi21:23
clarkblifeless: I think we have a few options available to us. We can use dib, boot resulting image in cloud, perform DB and other state steps, then snapshot that. We can just perform post boot in all VMs we boot. We can dump puppet for this stuff and do it all in native dib. Or we can continue to use nodepool to build images as it does today21:24
clarkbwe would like to be able to build images once and use them in all clouds. we want to be able to provide $notinfrapeople a way to build their own images. and I think we want a better story for CI of the images themselves21:25
clarkbI would put those goals as the top 3. mordred did I miss anything?21:25
*** adrian_otto1 is now known as adrian_otto21:25
*** andreykurilin_ has quit IRC21:25
*** jaypipes has quit IRC21:26
anteayashivharis: you don't have to wait for anything, you don't have to do anything21:27
fungishivharis: i don't believe we currently have any enforced standard behavior there, so the comments needed to recheck various systems' votes currently varies i think21:27
*** zz_gondoi is now known as gondoi21:27
anteayashivharis: and by the way you and sweston need to spend some time talking with each other21:27
harlowjalifeless so u can modify the release number to use dotted components, for example21:27
*** e0ne has quit IRC21:27
harlowjanled-2.52-6.fc13.x86_64.rpm is newer than nled-2.52-6.fc12.x86_64.rpm (the release number here is 6.fc12 and 6.fc13 accordingly)21:28
lifelessharlowja: I have a sketch coming in a sec21:28
russellbso, commit message updated on a patch, and it kept my +1 ... expected behavior?21:28
lifelessharlowja: using 999921:28
*** changbl has quit IRC21:28
swestonanteaya: shivharis hello21:28
harlowjalifeless k21:28
shivharissweston: hi21:28
anteayasweston: so yeah, you and shivharis need to sit down and have a chat21:28
swestonshivharis:  We have a meeting to discuss the details tomorrow.   I will include you on the meeting invite.21:28
shivharissweston: thx21:29
*** jaypipes has joined #openstack-infra21:29
swestonshivharis: :-)21:29
*** changbl has joined #openstack-infra21:29
clarkblifeless: the first and last options fail at the simple for $notinfraperson to make image. the second and third options better fit the goals but with a much higher labor or cloud resource cost21:29
clarkbso looking for alternatives that might better meet goals21:29
*** adalbas has quit IRC21:29
*** devkulkarni has quit IRC21:31
*** pdmars has joined #openstack-infra21:31
*** amcrn has quit IRC21:31
*** mfer has quit IRC21:32
*** flaper87 is now known as flaper87|afk21:32
anteayaso tomorrow I am away most of the day running errands21:33
anteayaand on Thursday I am voting, so gone a chunk of the day21:33
anteayamost likely morning21:33
*** gondoi is now known as zz_gondoi21:34
*** nati_ueno has joined #openstack-infra21:35
fungioh, i meant to mention it at the end of the meeting today but we were pressed for time as usual... i'll be offline/unreachable for long periods next week while i'm moving, and will probably miss next tuesday's meeting, but jeblair should be back by then21:36
*** pdmars has quit IRC21:36
clarkbooh moving21:37
clarkbhave fun21:37
*** jaypipes has quit IRC21:37
lifelessharlowja: dhellmann: spec updated21:37
anteayafungi: yes I hope moving goes well21:37
lifelessclarkb: ok I'm with you21:37
fungiclarkb: anteaya: thanks!21:38
*** signed8b_ has quit IRC21:38
*** sarob has joined #openstack-infra21:38
lifelessclarkb: I need an etherpad to track this21:38
*** adrian_otto has quit IRC21:39
lifelessclarkb: is there one?21:39
shivharisfungi, anteaya: thanks21:39
clarkblifeless: not yet but I can braindump into one really quickly21:39
clarkbjogo: can you see my comment on 98154 and see if that makes sense?21:39
*** zz_gondoi is now known as gondoi21:39
lifelessclarkb: my head just lost 'last option' in the context of a big wall of text :)21:40
*** sweston has quit IRC21:40
lifelessmordred: have you had a chance to poke at 94107 ?21:41
*** adrian_otto has joined #openstack-infra21:41
sdaguefungi: so one last thing if I can bug you about. The gerrit option change :)21:42
sdaguebecause I'm daily getting confused people ping me about why I -1ed their patch.21:42
*** sarob has quit IRC21:43
fungisdague: the reset votes on commit message change setting?21:43
sdaguefungi: yes please21:43
lifelessclarkb: highlight me when pad exists :)21:43
*** adrian_otto1 has joined #openstack-infra21:44
sdaguerussellb: currently expected, but I think wrong behavior, and the subject of me pinging fungi atm21:44
fungisdague: i'll see if i can slip that into gerrit and not break stuff... though i should also send a change for the gerrit deployment documentation taking that setting out as well21:44
sdaguefungi: it would be hugely appreciated21:44
*** signed8bit has joined #openstack-infra21:44
russellbsdague: cool thanks21:44
sdaguethe old behavior is something we kind of relied on21:44
*** CaptTofu_ has quit IRC21:45
*** CaptTofu_ has joined #openstack-infra21:46
*** adrian_otto1 has quit IRC21:46
*** adrian_otto has quit IRC21:46
clarkblifeless: mordred https://etherpad.openstack.org/p/nodepool-dib-integration\21:47
clarkber s/\\//21:47
*** mriedem has quit IRC21:47
clarkbsdague: huh I didn't realize that was the issue21:49
clarkbgerrit continues to have the info in it...21:49
*** ramashri has joined #openstack-infra21:50
*** CaptTofu_ has quit IRC21:50
*** lcostantino has quit IRC21:51
*** dims__ has joined #openstack-infra21:52
*** thuc has joined #openstack-infra21:52
openstackgerritMichael Krotscheck proposed a change to openstack-infra/storyboard-webclient: New StoryBoard Logo  https://review.openstack.org/9921721:53
*** UtahDave has joined #openstack-infra21:53
*** dims_ has quit IRC21:54
*** dhellmann has quit IRC21:55
*** ramashri_ has joined #openstack-infra21:55
*** ramashri has quit IRC21:55
*** dhellmann has joined #openstack-infra21:55
sdagueclarkb: yeh, but not reseting your vote by something that was fixable with a commit message change, ends up badly breaking review flow21:58
*** signed8bit has quit IRC21:59
*** kimj has joined #openstack-infra21:59
*** kimj has quit IRC21:59
*** kimj has joined #openstack-infra21:59
lifelessclarkb: so we make one schema for all projects? Or we make N schemas on the node image for the N projects?22:00
clarkblifeless: we make 2 schemas because ironic was special22:00
lifelessclarkb: I just want to get my head around whats done today - feel free to point me at the config code22:00
clarkblifeless: sure one sec22:00
*** changbl has quit IRC22:00
*** duran has quit IRC22:00
*** thuc has quit IRC22:01
*** kimj has quit IRC22:01
clarkblifeless: it starts at https://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/manifests/thick_slave.pp#n10422:01
*** kimj has joined #openstack-infra22:01
clarkblifeless: the install bits are not a problem. but eg https://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/manifests/thick_slave.pp#n150 is22:01
*** shivharis has quit IRC22:01
*** kimj has quit IRC22:02
*** rcarrillocruz has quit IRC22:02
clarkblifeless: basically we need a "root" db user without giving tests root on the machine22:02
lifelessclarkb: ok so we don't do the schema22:02
clarkbright we don't fill it in22:02
lifelesswe just ensure there's a usable DB service22:02
*** kimj has joined #openstack-infra22:03
lifelessclarkb: how long does that take - it looks like a fraction of a second worth of work22:03
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Don't copy review scores on commit message updates  https://review.openstack.org/9921922:03
fungirussellb: sdague: ^ already applied in gerrit22:03
clarkblifeless: well it is puppet... that is the bulk of our puppet runs22:04
*** rcarrillocruz has joined #openstack-infra22:04
clarkblifeless: easily several seconds22:04
sdaguefungi: thanks!22:04
clarkblifeless: which may be puppet being slow22:04
clarkbthe underlying puppet types do a lot of checks and things to make sure that they do the correct thing22:05
lifelessyep, and they have to because they are meant to run on live in-use systems22:05
lifelesswith admins changing stuff around them22:05
fungiclarkb: we should probably update gerrit.rst to reflect the draft disabling change you pushed to the all-projects acl too?22:05
clarkbfungi: ah yup22:05
fungii can add it22:05
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Prevent Draft patchsets from being pushed  https://review.openstack.org/9922022:07
fungiclarkb: ^22:07
*** marcoemorais has quit IRC22:08
clarkbfungi: approved22:09
fungireed: so it occurs to me that we have some options to weigh on queryability of the icla group members list... we can either make that group visible to everyone, or we can add a separate owner group for it which grants control to specific accounts (but would also grant those specific accounts the ability to add and remove users from it as well)22:09
fungizaro: ^ am i missing any alternatives you're aware of there?22:09
reedfungi, we need something read only...22:10
reedlet's see... is that detail public already?22:10
reedor do we have any concern about privacy for it?22:10
reedI think we need to discuss this publicly on a mlist22:11
*** aconrad has quit IRC22:11
fungireed: we can potentially add something more convoluted, but if there are no privacy concerns over the list of which gerrit accounts have signed the icla then i'd prefer we just make the group visible to all registered users22:11
lifelessclarkb: back on the etherpad22:11
clarkblifeless: yup22:11
fungireed: it's probably a slim superset of the list of e-mail addresses in official openstack git repositories after all, so there's likely little privacy concern in my mind22:12
fungi(i mean, we autogenerate authors files in all official projects which are literally the list of names and e-mail addresses of the people who committed to them, no need to even scrape commit logs for that info)22:13
*** smarcet has quit IRC22:13
reedfungi, makes sense to me... i'd like to have a second opinion22:14
zarofungi: there's another option, but probably same as latter.  i think there's a gerrit plugin that allows giving users direct access to groups.22:14
*** nati_ueno has quit IRC22:14
clarkbfungi: reed I agree that shouldn't be secret info as all contributions are public22:14
reedi know we consider email addresses of committers as public knowledge and that we have been keeping a public list of ICLA signatories until recently, on the wiki22:14
zarofungi: but easier just to make them visible to all.  +1 for that option.22:15
reedi'm convinced22:15
openstackgerritA change was merged to openstack-infra/config: Don't copy review scores on commit message updates  https://review.openstack.org/9921922:15
reedclarkb, fungi: you're right, who signed the iCLA is not a secret22:15
openstackgerritA change was merged to openstack-infra/config: Prevent Draft patchsets from being pushed  https://review.openstack.org/9922022:15
*** gondoi is now known as zz_gondoi22:16
reedsame with the Government CLA22:16
fungireed: i'll fix them right now22:17
*** thuc has joined #openstack-infra22:17
*** thuc_ has joined #openstack-infra22:19
*** denis_makogon has quit IRC22:19
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Don't deny visibility of ICLA group to its members  https://review.openstack.org/9922322:21
fungireed: clarkb: zaro: i'm having to do that ^ to accomodate it22:21
clarkbfungi: +222:21
*** bknudson has quit IRC22:22
*** thuc has quit IRC22:22
reedok, let's have that reviewed by Mark Collier22:22
reedCYA :)22:23
fungireed: will do22:23
harlowjalifeless if u get anytime, your knowledge might be useful on https://review.openstack.org/#/c/98946/22:23
fungireed: you want to ping him on that? if he's +1 i'll push the change to the gerrit acl accomodating it22:23
lifelessharlowja: what project is that in ?22:24
openstackgerritSean Dague proposed a change to openstack-infra/elastic-recheck: add a tool for determining unaccounted for rechecks  https://review.openstack.org/9462422:24
fungisometimes sparkycollier is in here, but apparently not at the moment22:24
* fungi switches to evening tasks now22:24
harlowjalifeless taskflow :)22:24
reedfungi, pinged22:25
reedfungi, thanks22:25
lifelessharlowja: (using gertty, and haven't gotten around to a 'jump to review' feature)22:25
harlowjalifeless gotcha, np, i gotta work on my gertty skillz22:26
lifelessmmm shiny, ubuntu are going to do an image based server thing next cycle22:26
harlowjalifeless oh, so my misunderstanding, that spec is in oslo-specs (the project its for is taskflow)22:27
clarkblifeless: ^O22:28
clarkbthat should allow you to punch in a change number22:29
*** Sukhdev has quit IRC22:29
*** Sukhdev has joined #openstack-infra22:29
lifelessclarkb: ah! so that wants to be in the top level f1 text22:29
clarkbindeed that was apparently not added22:31
*** nati_ueno has joined #openstack-infra22:32
*** zns has joined #openstack-infra22:33
morganfainbergwow, we're going to hit 100k reviews through gerrit soonish.22:36
morganfainbergwell proposed reviews.22:36
zarofungi, clarkb i think visibility is a gerrit system config setting?22:36
lifelessclarkb: so some more options up about nodepool22:37
lifelessclarkb: I have a call in 23m22:37
zaroclarkb, fungi : https://gerrit-review.googlesource.com/Documentation/config-gerrit.html#accounts22:37
lifelessharlowja: can subgraphs not run today ?22:38
*** zns_ has joined #openstack-infra22:38
harlowjalifeless they do, just not conditionally22:38
*** zns has quit IRC22:38
*** penguinRaider has quit IRC22:38
*** sarob has joined #openstack-infra22:38
lifelessharlowja: if they don't run, is it an error ?22:38
clarkblifeless: looking. I do have my image running in qemu now though. very slow as epxected22:38
clarkblifeless: waiting for my ssh connection to give me a prompt22:38
*** signed8bit has joined #openstack-infra22:38
lifelessclarkb: do-be-do-be-do22:38
harlowjalifeless just means its the workflow isn't complete, not neccasarily error22:38
harlowja*just means that the22:39
lifelessok so specifically, you cannot have a complete workflow unless all nodes of the dag have executed22:39
harlowjalifeless correct22:39
clarkblifeless: according to the console output we are still booting though...22:39
*** doude has quit IRC22:39
harlowjalifeless i'm working on a update to that to see how the usage would look, we have a way to ignore parts of the workflow and still be complete (so i think we can do that easily)22:40
anteayamorning jhesketh22:41
lifelessharlowja: I'm going to sketch some thoughts, probably using the wrong words22:41
lifelessharlowja: but hopefully useful22:41
harlowjalifeless thats fair :)22:41
harlowjayour brainpower useful no matter ;)22:41
clarkbfungi: anteaya: should we go ahead and address the infra specs initial commit coments now and push that through?22:42
clarkbI am happy to address them while I wait on things to boot22:42
anteayaclarkb: I'm for that22:42
clarkblifeless: cloud-init boot finished at Tue, 10 Jun 2014 22:42:13 +0000. Up 673.35 seconds22:42
clarkblifeless: not really quick :/22:42
anteayateh comment you had was /** for createing the docs22:42
*** sarob has quit IRC22:42
clarkbanteaya: correct, though I think I like fungi's idea of explicitly calling out implemented22:43
anteayaclarkb: I don't think adding a second asterisk is that wild of a change22:43
*** kimj has quit IRC22:43
*** thedodd has quit IRC22:43
anteayaclarkb: I'm meh on that, if you like it put it in22:45
*** thuc has joined #openstack-infra22:45
anteayaclarkb: if jim hates it he will take it out22:45
*** thuc_ has quit IRC22:45
anteayaand we will have specs in review at the same time22:45
anteayathough none in implemented/ for a while22:45
openstackgerritAlex Gaynor proposed a change to openstack-infra/config: Added a puppet module for livegrep.  https://review.openstack.org/9687722:46
*** zns has joined #openstack-infra22:46
*** ArxCruz has joined #openstack-infra22:46
*** penguinRaider has joined #openstack-infra22:47
*** zns_ has quit IRC22:49
clarkbanteaya: well they fix the same problem22:49
*** devkulkarni has joined #openstack-infra22:50
anteayado they?22:51
anteayaI guess I don't know enough about what /** does22:51
clarkbanteaya: actually I am leaning back towards my original comment22:52
clarkbwell maybe not22:52
clarkblifeless: Notice: Compiled catalog for precise30gbtest.openstacklocal in environment production in 128.17 seconds22:52
lifelessthats tolerable I guess?22:53
lifelessharlowja: ok, when gertty syncs (it has 750 pending!) you'll get some feedback :)22:53
harlowjaharlowja thx much22:54
harlowja*oops, lifeless ^22:54
clarkblifeless: well 11 minutes to boot, 2 minutes just to compile catalog then you have to apply it22:54
lifelessclarkb: heh. But its async right, once a day ?22:54
clarkblifeless: I am running puppet under time so should have a rough guestimate for total time22:54
clarkblifeless: ya I think once a day is what we would go for22:55
*** homeless has quit IRC22:55
mattoliverauGood Morning22:55
clarkbmordred: fungi re pbr versioning in setup.py I thought we did want it to be versioned22:55
clarkbwe just use the big range22:56
anteayamorning mattoliverau22:56
openstackgerritMichael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Including UX Feedback on menu and nav.  https://review.openstack.org/9920922:56
anteayaclarkb: yeah I think your comment is a gentle change22:56
anteayaclarkb: I think adding implemented is a design change22:57
anteayaI'm fine with /** and merge22:57
anteayathen we have time to add implemented before anything is actually implemented22:57
*** CaptTofu_ has joined #openstack-infra23:00
*** thuc has quit IRC23:03
*** thuc has joined #openstack-infra23:03
*** CaptTofu_ has quit IRC23:04
*** bogdando has quit IRC23:04
*** radez is now known as radez_g0n323:04
mordred_phoneclarkb: no. never version pbr in setup.py23:05
mordred_phonenever ever ever23:05
clarkbmordred_phone: huh23:06
clarkbmordred_phone: you should probably fix that everywhere then23:06
mordred_phonethere is no remediation for setup_requires failure23:06
mordred_phoneI did a while back23:06
*** dims__ has quit IRC23:06
clarkbNotice: Finished catalog run in 732.66 seconds23:07
clarkbso we are talking about ~half an hour to do the qemu post dib stuff23:07
*** bogdando has joined #openstack-infra23:07
*** zns has quit IRC23:08
*** dims__ has joined #openstack-infra23:08
clarkbanteaya: ok, so I did it the other way, I will wait and see what fungi says23:10
clarkbanteaya: they really do the same thing23:10
clarkbanteaya: they publish the contents of imeplemented/*. One happens to do it in a nicer way than the other imo23:10
*** nati_ueno has quit IRC23:10
anteayaclarkb: okay23:11
anteayaI trust you23:11
*** doug-fish has left #openstack-infra23:12
*** rfolco has joined #openstack-infra23:13
openstackgerritAndrew Melton proposed a change to openstack/requirements: Add IDMapShift for Nova  https://review.openstack.org/9888823:13
*** zns has joined #openstack-infra23:13
*** UtahDave has quit IRC23:14
SlickNikHey guys.23:14
SlickNikSo we have a couple of patches to trove which are updates from the global requirements.23:14
SlickNikBut they are failing the requirements gate.23:15
SlickNikexample: https://review.openstack.org/#/c/99095/23:15
SlickNikAny idea why this might be happening, or if it is a known issue (before I try and dig deeper)?23:16
*** atiwari has quit IRC23:16
clarkbSlickNik: that may be a new issue with the exclusion23:16
clarkbSlickNik: there is an existing problem where we don't apply that test to the requirements changes themselves23:16
harlowjalifeless good ideas & comments, think we were thinking on the same wave-length (similar changes i was also writing up, haha)23:17
harlowja*scary*, ha23:17
*** UtahDave has joined #openstack-infra23:17
*** zns has quit IRC23:18
SlickNikclarkb: Thanks for the info!23:19
*** gokrokve has quit IRC23:19
*** pcrews_ has quit IRC23:20
*** pcrews has joined #openstack-infra23:20
*** reed_ has joined #openstack-infra23:21
*** reed has quit IRC23:22
clarkbanteaya: fungi I am testing those changes now real quick (want to make sure we can still build sphinx after the pbr and sphinx changes23:23
clarkbanteaya: and of course it finds a bug with the implemented stuff...23:23
clarkbanteaya: going to switch to /** now23:24
clarkbanteaya: (you need to have a file in there otherwise sphinx goes warning no files and everything fails)23:24
anteayahelpful sphinx23:24
anteayanow I know23:24
openstackgerritClark Boylan proposed a change to openstack-infra/infra-specs: Initial commit  https://review.openstack.org/9444023:25
clarkbanteaya: mordred_phone jhesketh SergeyLukjanov fungi ^23:25
clarkbthe diff should be relatively small23:25
*** reed_ has quit IRC23:27
anteayaclarkb: and that patch passed?23:27
devkulkarniSlickNick, clarkb: we are seeing similar error in Solum as well https://review.openstack.org/#/c/99103/ .. based on your exchange I am not clear what actions if any we need to take23:27
clarkbanteaya: yes tox -redocs worked with what I pushed23:29
anteayaclarkb: and line 8 was the only change? https://review.openstack.org/#/c/94440/4/doc/source/index.rst23:29
clarkbdevkulkarni: SlickNik I believe that specific check was reverted last night23:29
anteayaI wish we had difs between patchsets23:29
clarkbanteaya: no, I fixed pbr and added fungi's security section23:30
clarkbanteaya: you do have diffs between patchsets don't you?23:30
*** praneshp has quit IRC23:30
clarkbanteaya: diff between patchsets works for me23:30
devkulkarniclarkb: thanks for the info. so I assume nothing needs to be done from our sides. once the check gets reapplied the review should pass23:30
anteayahow do I get that view?23:30
clarkbanteaya: are you using gerrity or gertty?23:30
clarkbdevkulkarni: yes I think that is the case23:31
openstackgerritAngus Salkeld proposed a change to openstack-infra/config: Add a solum-spec repo to stackforge  https://review.openstack.org/9823723:31
clarkbanteaya: * gerrit or gertty?23:31
*** olaph has quit IRC23:31
anteayaI don't have gertty working reliably yet23:31
devkulkarniclarkb: thanks23:32
sdagueclarkb: so we've seemed to level off on logstash catching up23:32
clarkbanteaya: just change the base to 3 on that change23:32
sdagueit remains 25k events behind23:32
anteayak and then?23:32
clarkbsdague: yeah I think we have the properish number of nodes they just aren't fast enough to bring it down while busy23:32
clarkbanteaya: click on diff23:32
anteayaoh you changed 4 files23:32
anteayacool, thanks23:32
clarkbanteaya: you can also do it in the diff page by picking a patchset on the left hand side23:33
*** praneshp has joined #openstack-infra23:33
sdagueclarkb: bummer :(23:33
*** devkulkarni has quit IRC23:33
anteayaclarkb: ah okay23:33
anteayaso yeah, +1 to your patch23:34
clarkbanteaya: thanks23:34
anteayalet's have some specs23:34
anteayasince the world isn't on fire atm23:34
anteayabut the gate got long again23:35
sdagueanteaya: it's still on fire, it's just not zomg fire23:35
anteaya27 patches added in the last two hours23:36
anteayasdague: did the nova-network logging patch get +A'd yet?23:36
anteayasdague: that was a priorty for the fire, yeah?23:36
sdagueno, it had some issues, I don't think mikal spun a new rev yet23:36
anteayaah okay23:36
anteayajhesketh: is mikal around yet?23:37
jheskethanteaya: I think he's around, but he's very busy23:38
anteayajhesketh: k thanks23:38
*** sarob has joined #openstack-infra23:38
anteayasdague: was there anything else on your list of help quench the fire?23:38
anteayasdague: been sort of pulled off on other things today23:38
*** thuc has quit IRC23:39
*** reed_ has joined #openstack-infra23:39
*** thuc has joined #openstack-infra23:39
sdaguehonestly, my tank is empty there. Today I mostly was trying to get my 25 outstanding patches to a passing state23:39
sdagueand then discovering that 5 - 10% of our logs (by bytes) are keystone auth tokens23:40
morganfainbergsdague, gah, really? sorry :(23:41
clarkbsdague: ya those things are pretty annoying23:41
clarkbsdague: I think they cause the most problems with the crm114 to23:41
sdague40 MB per run (uncompressed)23:41
sdaguemorganfainberg: it's mostly actually the fault that we log them23:41
morganfainbergsdague, right.23:41
*** ArxCruz has quit IRC23:41
morganfainbergsdague, still bleh.23:41
sdaguewhich is actually, arguably, a security issue23:41
anteayasdague: understood23:42
morganfainbergsdague, yeah, shouldn't log tokens...23:42
sdagueI suspect that's going to be the list discussion tomorrow, as there is odd resistance to patches like this - https://review.openstack.org/#/c/98443/23:42
sdagueso we're going to just have to agree as a community that logging auth tokens is terrible, and stop :)23:43
*** sarob has quit IRC23:43
morganfainbergsdague, i'll confirm we're not logging those in the session object23:43
morganfainbergsdague, so perhaps when people move to keystone client session... they'll just "disappear"23:43
morganfainbergif there really is tha tmuch resistence23:43
sdaguewell, they log it in requests as well23:43
*** thuc has quit IRC23:43
anteayaclarkb: check my sanity here: https://review.openstack.org/#/c/91050/423:44
clarkbsdague: right this is one of the things that came up in the user sessions at the summit23:44
anteayaclarkb: but base to patchset 323:44
morganfainbergsdague, the session object abstracts all that out.23:44
anteayaclarkb: what do you see?23:44
sdagueclarkb: right, I was in there23:44
morganfainbergsdague, keystoneclient would manage that part not "novaclient" for example.23:44
morganfainberg(needed for auth plugins)23:44
clarkbanteaya: usually that indicates a rebase, so the diff includes things from other changes that merged23:44
sdaguemorganfainberg: it would manage req to nova api?23:44
anteayaoh okay, thanks23:44
clarkbanteaya: this is one of the issues with gerrit's diffing. You can use git review -m instead23:44
sdaguemorganfainberg: I'm confused23:44
morganfainbergsdague, keystoneclient has a session object we're trying to convert everyone over to using23:45
sdaguemorganfainberg: sure23:45
anteayaclarkb: I'll just move ahead without diffs between patchsets23:45
morganfainbergsdague, that handles all forms of auth...in fact, could be passed between client objects23:45
morganfainbergsdague, in _theory_ we could prevent the logging of the token via conversion to the session object23:45
sdaguehow does that impact the req_logging in novaclient23:45
morganfainbergsdague, if the req doesn't know the token, how can it log it, the session would handle that bit, session holds the token23:46
morganfainbergsdague, i think that is the plan at least.23:46
sdagueyeh, what's the eta on that?23:46
sdaguethat seems like it would take a while23:46
morganfainbergsdague, look at the masssssive number of reviews jamie lennox has open.23:46
*** yamahata has quit IRC23:46
sdagueI already have a massive number of reviews to look at :)23:46
morganfainberga number of them are converting to the session object23:47
*** mmaglana has quit IRC23:47
morganfainbergi'll poke into these and see if we're still logging the token with the session23:47
morganfainbergbut i think we can make that part of the conversion (make it some other really out of the way logging if it's realllly desired) instead of part of debug23:47
sdagueyeh, that would be good23:47
*** thuc has joined #openstack-infra23:48
anteayamattoliverau: you might enjoy a peek at the meeting log from last week: http://eavesdrop.openstack.org/meetings/infra/2014/infra.2014-06-03-19.02.log.html23:48
morganfainbergas i said in -qa channel, i agree, logging token is only marginally less bad than logging passwords23:48
sdaguemorganfainberg: ++23:48
anteayamattoliverau: we addressed some consistency issues with the acl files in openstack-infra/config23:48
sdaguenow you just need to help me convince jogo of that :)23:48
*** lcheng has quit IRC23:49
morganfainbergsdague, i'll do my best! :)23:49
morganfainbergsdague, but i think he wasn't opposed as long as we document it clearly that that change is happening.23:49
*** CaptTofu_ has joined #openstack-infra23:50
clarkbbodepd: let me know what you think of my comments in 96336.23:50
sdagueI did that, he still -1ed it23:50
sdagueon the updated patch23:50
morganfainbergsdague, bleh23:51
clarkbbodepd: mostly rm -Rvf is scary and that function is known to work so its nice to stick with it23:51
sdagueok, I think it might be drone flying time23:51
*** gokrokve has joined #openstack-infra23:52
mattoliverauanteaya: I'll take a peek, thanks :)23:53
*** bogdando has quit IRC23:54
openstackgerritAnita Kuno proposed a change to openstack-infra/elastic-recheck: Add query for ceilometer test_notify_alarm bug 1321826  https://review.openstack.org/9814923:55
uvirtbotLaunchpad bug 1321826 in ceilometer "periodic notifier unit test failure" [Medium,In progress] https://launchpad.net/bugs/132182623:55
morganfainbergsdague, i commented on that review just now23:55
morganfainbergsdague, i'll get back to you on the state of the logging w/ the session object tomorrow i think.23:56
*** zzelle_ has quit IRC23:56
*** bogdando has joined #openstack-infra23:57
*** thuc has quit IRC23:57
*** thuc has joined #openstack-infra23:58
*** dangers is now known as dangers_away23:58
openstackgerritClark Boylan proposed a change to openstack-infra/config: Use WatchedFileHandler to avoid copytruncate  https://review.openstack.org/9593523:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!