Monday, 2013-08-19

*** dina_belova has joined #openstack-infra00:11
*** dina_belova has quit IRC00:15
fungipew pew scrollback lasers00:20
*** dkliban has quit IRC00:38
*** jesusaurus has quit IRC00:39
*** jesusaurus has joined #openstack-infra00:44
*** colinmcnamara1 has joined #openstack-infra00:48
*** colinmcnamara has quit IRC00:48
*** erfanian has joined #openstack-infra00:50
*** dina_belova has joined #openstack-infra01:11
*** CaptTofu has joined #openstack-infra01:14
*** dina_belova has quit IRC01:16
*** tianst20 has quit IRC01:29
*** xchu has joined #openstack-infra01:48
*** DennyZhang has joined #openstack-infra01:56
*** yaguang has joined #openstack-infra01:57
*** yaguang has quit IRC02:05
clarkbmordred: fungi: jeblair: it just occurred to me that we could potentially cdn our git repos on git.o.o. to make life better for lifeless and other people with high latency02:09
*** dina_belova has joined #openstack-infra02:12
lifelessclarkb: meh. git is very good for keeping a cache; we've got comprehensive support for caching it in dib02:13
clarkbya git being distributed makes it less painful02:15
lifelessalso unless you had a git server cdn it would be worse02:16
lifelessin big projects when a big pack occurs02:16
lifelesse.g. nova02:16
*** zul has quit IRC02:16
*** dina_belova has quit IRC02:16
*** yaguang has joined #openstack-infra02:19
*** zul has joined #openstack-infra02:20
*** zul has quit IRC02:34
*** CaptTofu has quit IRC02:35
*** CaptTofu has joined #openstack-infra02:36
*** DennyZhang has quit IRC02:38
*** zul has joined #openstack-infra02:42
*** erfanian has quit IRC02:51
*** sgviking has joined #openstack-infra02:58
*** mriedem has quit IRC03:00
*** tian has joined #openstack-infra03:02
*** yaguang has quit IRC03:03
tianhi, my evn : ubuntu 12.04 . when i use cmd "mvn clean generate-sources" , there is a error "Plugin org.apache.maven.plugins:maven-clean-plugin:2.3 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.maven.plugins:maven-clean-plugin:jar:2.3: Could not transfer artifact org.apache.maven.plugins:maven-clean-plugin:pom:2.3 from/to rackspace-research (
tiannt/groups/public/): Connection to refused: Connection timed out -> [Help 1]03:11
tian "03:11
*** dina_belova has joined #openstack-infra03:12
tiananybody know why ?03:12
clarkbtian connection timed out to that rackspace url. looks like network related03:14
*** yaguang has joined #openstack-infra03:16
*** colinmcnamara1 has quit IRC03:16
*** dina_belova has quit IRC03:17
*** yaguang has quit IRC03:27
*** yaguang has joined #openstack-infra03:36
*** dkliban has joined #openstack-infra03:40
*** rcleere has joined #openstack-infra03:49
*** colinmcnamara has joined #openstack-infra03:53
*** colinmcnamara has quit IRC03:55
*** vogxn has joined #openstack-infra03:56
*** Ryan_Lane has joined #openstack-infra03:58
*** Ryan_Lane has quit IRC04:03
*** Ryan_Lane has joined #openstack-infra04:04
*** Ryan_Lane has quit IRC04:04
*** Ryan_Lane has joined #openstack-infra04:04
*** dina_belova has joined #openstack-infra04:13
*** dina_belova has quit IRC04:17
*** UtahDave has joined #openstack-infra04:19
*** Alex_Gaynor has quit IRC04:19
*** mkoderer has quit IRC04:19
*** Alex_Gaynor has joined #openstack-infra04:23
*** dkliban has quit IRC04:28
*** dkliban has joined #openstack-infra04:34
*** nati_ueno has joined #openstack-infra04:36
*** rcleere has quit IRC04:40
*** mberwanger has joined #openstack-infra04:41
*** boris-42 has joined #openstack-infra04:53
*** dkliban has quit IRC05:08
*** mberwanger has quit IRC05:10
*** dina_belova has joined #openstack-infra05:13
*** nati_ueno has quit IRC05:15
*** thomasbiege1 has joined #openstack-infra05:15
*** vogxn has quit IRC05:17
*** thomasbiege1 has quit IRC05:17
*** dina_belova has quit IRC05:18
*** thomasbiege has joined #openstack-infra05:21
*** nati_ueno has joined #openstack-infra05:23
*** nicedice_ has quit IRC05:23
*** nati_ueno has quit IRC05:23
*** thomasbiege has quit IRC05:24
*** SergeyLukjanov has joined #openstack-infra05:41
*** vogxn has joined #openstack-infra05:44
*** dina_belova has joined #openstack-infra05:54
*** ftcjeff has quit IRC06:03
*** ftcjeff has joined #openstack-infra06:04
*** vogxn1 has joined #openstack-infra06:05
*** thomasbiege1 has joined #openstack-infra06:06
*** vogxn has quit IRC06:06
*** nayward has quit IRC06:06
*** thomasbiege1 has quit IRC06:08
*** ftcjeff has quit IRC06:08
*** mkoderer has joined #openstack-infra06:21
*** jjmb has quit IRC06:22
*** dina_belova has quit IRC06:24
*** SergeyLukjanov has quit IRC06:25
*** SergeyLukjanov has joined #openstack-infra06:30
*** kspear has quit IRC06:30
*** kspear has joined #openstack-infra06:32
*** SergeyLukjanov has quit IRC06:38
*** SergeyLukjanov has joined #openstack-infra06:42
*** dina_belova has joined #openstack-infra06:43
*** SergeyLukjanov has quit IRC06:48
*** dina_belova has quit IRC06:57
*** DennyZhang has joined #openstack-infra06:58
*** DennyZhang has quit IRC07:12
*** vogxn has joined #openstack-infra07:15
*** fbo_away is now known as fbo07:16
*** odyssey4me3 has joined #openstack-infra07:16
chmoueljeblair: what i about i put that in stackforge? or is that too small of a project for going there?07:19
*** xchu has quit IRC07:22
*** jjmb has joined #openstack-infra07:23
*** nsaje1 has joined #openstack-infra07:23
*** xBsd has joined #openstack-infra07:24
*** Ryan_Lane has quit IRC07:26
*** xchu has joined #openstack-infra07:39
*** vogxn has quit IRC07:39
*** thomasbiege1 has joined #openstack-infra07:42
*** thomasbiege1 has quit IRC07:43
*** UtahDave has quit IRC07:52
*** jpich has joined #openstack-infra07:53
*** Ryan_Lane has joined #openstack-infra07:57
*** jhesketh has quit IRC08:03
*** Ryan_Lane has quit IRC08:04
*** derekh has joined #openstack-infra08:05
*** dina_belova has joined #openstack-infra08:08
*** nayward has joined #openstack-infra08:11
*** dina_belova has quit IRC08:13
*** mkerrin has joined #openstack-infra08:13
openstackgerritYuriy Taraday proposed a change to openstack-infra/git-review: Add more deterministic port selection for Gerrit.
*** boris-42 has quit IRC08:19
*** psedlak has joined #openstack-infra08:21
*** shardy_afk is now known as shardy08:39
*** tian has quit IRC08:48
*** dina_belova has joined #openstack-infra09:09
*** dina_belova has quit IRC09:13
*** ruhe has joined #openstack-infra09:14
*** morganfainberg is now known as morganfainberg|a09:27
*** moted has quit IRC09:30
*** boris-42 has joined #openstack-infra09:30
*** xBsd has quit IRC09:31
*** xBsd has joined #openstack-infra09:34
*** xchu has quit IRC09:34
*** xchu has joined #openstack-infra09:47
openstackgerritMark McLoughlin proposed a change to openstack/requirements: Allow use of oslo.messaging 1.2.0a9
*** chuckieb|2 has quit IRC09:53
*** Ryan_Lane has joined #openstack-infra10:02
*** xchu has quit IRC10:09
*** dina_belova has joined #openstack-infra10:09
*** Ryan_Lane has quit IRC10:09
*** nayward has quit IRC10:11
*** dina_belova has quit IRC10:13
*** weshay has joined #openstack-infra10:14
*** moted has joined #openstack-infra10:15
*** DennyZhang has joined #openstack-infra10:15
*** ruhe has quit IRC10:16
*** pblaho has joined #openstack-infra10:21
*** DennyZhang has quit IRC10:23
*** yaguang has quit IRC10:24
*** nayward has joined #openstack-infra10:30
*** tian has joined #openstack-infra10:30
*** fbo is now known as fbo_away10:34
openstackgerritA change was merged to openstack/requirements: Allow use of oslo.messaging 1.2.0a9
*** nayward has quit IRC10:42
*** ruhe has joined #openstack-infra10:43
openstackgerritMarton Kiss proposed a change to openstack-infra/config: Add node
*** mikal has quit IRC10:45
*** mikal has joined #openstack-infra10:47
*** chuckieb has joined #openstack-infra10:49
*** psedlak_ has joined #openstack-infra11:03
openstackgerritMark McLoughlin proposed a change to openstack-infra/config: Make oslo.messaging gating on devstack/tempest
*** psedlak has quit IRC11:07
*** xBsd has quit IRC11:08
*** xBsd has joined #openstack-infra11:09
*** vogxn has joined #openstack-infra11:16
*** olaph has quit IRC11:19
*** vogxn has quit IRC11:21
*** olaph has joined #openstack-infra11:26
*** dmakogon__ has joined #openstack-infra11:27
*** Ryan_Lane has joined #openstack-infra11:27
*** GheRivero_CH is now known as GheRivero11:27
*** xBsd has quit IRC11:28
*** dmakogon_ has quit IRC11:31
*** Ryan_Lane has quit IRC11:31
*** fbo_away is now known as fbo11:37
*** thomasbiege1 has joined #openstack-infra11:40
*** weshay has quit IRC11:44
*** cody-somerville has joined #openstack-infra11:44
*** thomasbiege1 has quit IRC11:44
*** thomasbiege1 has joined #openstack-infra11:45
*** weshay has joined #openstack-infra11:50
*** dkliban has joined #openstack-infra11:52
openstackgerritMarton Kiss proposed a change to openstack-infra/config: Add node
*** dina_belova has joined #openstack-infra11:57
*** nsaje1 has quit IRC11:58
*** dina_belova has quit IRC11:58
*** rfolco has joined #openstack-infra11:58
*** weshay has quit IRC11:59
*** nayward has joined #openstack-infra11:59
*** lcestari has joined #openstack-infra12:01
*** apcruz has joined #openstack-infra12:04
*** CaptTofu has quit IRC12:05
*** pcm_ has joined #openstack-infra12:05
*** CaptTofu has joined #openstack-infra12:05
*** jjmb1 has joined #openstack-infra12:07
*** dina_belova has joined #openstack-infra12:07
*** psedlak__ has joined #openstack-infra12:10
*** jjmb has quit IRC12:10
*** dina_belova has quit IRC12:11
*** xBsd has joined #openstack-infra12:11
*** psedlak_ has quit IRC12:14
*** sandywalsh has quit IRC12:17
*** woodspa has joined #openstack-infra12:21
*** beagles has joined #openstack-infra12:24
*** dina_belova has joined #openstack-infra12:26
*** sandywalsh has joined #openstack-infra12:30
*** simonmcc has quit IRC12:31
*** simonmcc has joined #openstack-infra12:31
*** simonmcc has quit IRC12:31
*** nayward has quit IRC12:31
*** simonmcc has joined #openstack-infra12:31
*** simonmcc has quit IRC12:32
*** simonmcc has joined #openstack-infra12:32
*** simonmcc has quit IRC12:32
*** dina_belova has quit IRC12:32
*** simonmcc has joined #openstack-infra12:33
*** nayward has joined #openstack-infra12:35
*** mriedem has joined #openstack-infra12:36
pblahohi guys... I have a question w/r/t voting: false settings in zuul/layout.yaml ...12:37
pblahoI have seen a bunch of jobs with this settings... mainly python33 kind of jobs....12:38
pblahodo these jobs even run or they just do not count on gerrit voting and/or when deciding if it will be merged...12:38
*** thomasbiege1 has quit IRC12:42
mordredchmouel: there are no projects that are too small :)12:44
mordredpblaho: they are run and their result does not count for gerrit voting/deciding12:45
*** dprince has joined #openstack-infra12:46
chmouelmordred: cool I will move it there,12:46
chmouelis there a wiki page handy that explains the process ? (moving a project into stackforge)12:48
*** nayward has quit IRC12:50
*** nayward has joined #openstack-infra12:50
*** xqueralt has joined #openstack-infra12:50
pblahomordred: thank you.... so.. b/c I recently submited a change with python33 and pylint for stackforge/python-tuskarclient and now I discovered that it will not be that easy to be python3 compatible (oslo.config is not - for the package ver. 1.1.1 from pypi)....12:51
*** thomasbiege1 has joined #openstack-infra12:51
pblahoso the best solution is to set python33 job for tuskarclient to voting: false... ?12:51
mordredpblaho: yes. I think so12:51
pblahomordred: ok, thanks... preparing change right now...12:51
*** anteaya has joined #openstack-infra12:51
mordredand then you can slowly work on getting python33 working, and then when it does, we can change it back to voting12:51
*** thomasbiege1 has quit IRC12:52
pblahomrodden: yes... that is the way I want it...12:54
pblahomrodden: sorry...12:54
pblahomordred: ^^12:54
*** markmc has joined #openstack-infra12:55
*** adalbas has joined #openstack-infra12:55
*** xBsd has quit IRC12:55
openstackgerritPetr Blaho proposed a change to openstack-infra/config: Voting: false for py33, pylint for tuskarclient
*** erfanian has joined #openstack-infra13:04
*** xBsd has joined #openstack-infra13:05
*** nayward has quit IRC13:10
xqueralthey, I think tempest tests might be broken for grizzly13:10
xqueraltcould somebody take a look at: ?13:11
uvirtbotLaunchpad bug 1213912 in tempest "keystone cli tests fail in grizzly" [Undecided,New]13:11
xqueraltI've tried to debug it a bit more more but I don't know where the problem might come from13:12
anteayahi xqueralt, tempest tests tend to be the purview of the folks in #openstack-qa, you might get more of a response to your question if you were to ask in that channel13:14
xqueraltanteaya ok, thanks13:15
*** lyz_ has joined #openstack-infra13:17
*** boris-42_ has joined #openstack-infra13:18
*** boris-42 has quit IRC13:20
*** boris-42_ is now known as boris-4213:20
*** lyz has quit IRC13:22
*** stevebaker has quit IRC13:22
*** Guest63475 has quit IRC13:22
*** antonym has quit IRC13:22
*** oubiwann has quit IRC13:22
*** stevebaker has joined #openstack-infra13:23
*** burt has joined #openstack-infra13:26
*** Steely_Dan is now known as dansmith13:27
*** antonym has joined #openstack-infra13:27
openstackgerritJoshua Hesketh proposed a change to openstack-infra/zuul: Separate reporters from triggers
openstackgerritJoshua Hesketh proposed a change to openstack-infra/zuul: Move gerrit specific result actions to only apply to gerrit reporter
openstackgerritJoshua Hesketh proposed a change to openstack-infra/zuul: Add support for emailing results via SMTP
*** erfanian has quit IRC13:29
*** vogxn has joined #openstack-infra13:29
*** oubiwann has joined #openstack-infra13:30
*** lillie has joined #openstack-infra13:32
*** lillie is now known as Guest7581913:32
*** ryanpetrello has joined #openstack-infra13:32
*** psedlak has joined #openstack-infra13:36
* anteaya finishes reading weekend backscroll13:37
*** psedlak__ has quit IRC13:37
anteayaa relatively quiet weekend, I see13:37
anteayawell done, all13:37
*** dina_belova has joined #openstack-infra13:43
*** pabelanger has quit IRC13:43
*** dkliban has quit IRC13:44
*** erfanian has joined #openstack-infra13:45
*** dina_belova has quit IRC13:47
*** ruhe has quit IRC13:49
*** ruhe_ has joined #openstack-infra13:51
*** ruhe_ has quit IRC14:01
*** xBsd has quit IRC14:05
*** psedlak_ has joined #openstack-infra14:05
*** cppcabrera has joined #openstack-infra14:06
*** cppcabrera has left #openstack-infra14:08
*** psedlak has quit IRC14:09
*** annegentle has joined #openstack-infra14:10
*** mikal has quit IRC14:11
*** ruhe has joined #openstack-infra14:11
*** mikal has joined #openstack-infra14:12
*** rustlebee is now known as russellb14:13
*** blamar has joined #openstack-infra14:16
openstackgerritAlex Gaynor proposed a change to openstack-infra/config: Fixed sending notifications to #openstack-marconi
*** rcleere has joined #openstack-infra14:20
*** changbl has quit IRC14:22
*** markmcclain has joined #openstack-infra14:25
*** xqueralt is now known as xqueralt-afk14:28
Alex_GaynorIf anyone's got a few seconds ^ fixes IRC notifications for marconi14:28
*** xqueralt-afk is now known as xqueralt14:28
*** rnirmal has joined #openstack-infra14:29
*** dkliban has joined #openstack-infra14:31
*** ftcjeff has joined #openstack-infra14:32
*** fbo is now known as fbo_away14:33
pblahoanyone who gets a free cycle please see - sets voting for some tuskarclient jobs14:35
*** dkranz has joined #openstack-infra14:36
*** datsun180b has joined #openstack-infra14:40
*** fbo_away is now known as fbo14:40
openstackgerritJason Meridth proposed a change to openstack-dev/hacking: Adds ability to ignore hacking validations with noqa
*** cdevienne has joined #openstack-infra14:44
openstackgerritJason Meridth proposed a change to openstack-dev/hacking: Adds ability to ignore hacking validations with noqa
*** nayward has joined #openstack-infra14:50
openstackgerritA change was merged to openstack-infra/config: Fixed sending notifications to #openstack-marconi
Alex_GaynorDoes that automatically deploy, or does someone need to kick a thing?14:56
jeblairAlex_Gaynor: automagic14:57
Alex_Gaynorjeblair: should I assume from now on that that's true of everything under puppet?14:57
jeblairAlex_Gaynor: pretty much.  :)  we don't automatically restart services that would be particularly disruptive (eg zuul or gerrit), but otherwise, yes as much as possible14:58
*** pabelanger has joined #openstack-infra14:59
openstackgerritPetr Blaho proposed a change to openstack-infra/config: Voting: false for py33, pylint for tuskarclient
*** openstackgerrit has quit IRC15:01
*** mgagne has joined #openstack-infra15:02
*** openstackgerrit has joined #openstack-infra15:02
mgagnethanks to whoever planned and renamed the project/repo for puppet-neutron =)15:03
jeblairmgagne: mordred did that15:03
mordredwhat did I do?15:03
mordredoh! yay!15:03
openstackgerritPetr Blaho proposed a change to openstack-infra/config: Voting: false for py33, pylint for tuskarclient
mgagnemordred: thanks! now we can turn the page and go forward :D15:04
jeblairmordred: i reported bug 1213958 in novaclient15:06
uvirtbotLaunchpad bug 1213958 in python-novaclient "UUID cache is not concurrency-safe" [Undecided,New]
*** yaguang has joined #openstack-infra15:06
*** prad_ has joined #openstack-infra15:07
jeblairbecause that's happening in nodepool15:08
*** ruhe has quit IRC15:09
mordredjeblair: awesome!15:11
*** ruhe has joined #openstack-infra15:11
mordredjog0: ^^15:11
*** psedlak_ has quit IRC15:11
jeblairmordred: i wonder if heat has seen that?  My vague understanding from a quick grep of the source is that heat uses a single globally cached Client (which is the model nodepool is using)15:12
*** nayward has quit IRC15:13
mordredjeblair: that's the model that you're using now to work around the concurrency issues, right?15:13
mordredjeblair: and I'm not sure if it is or not - SpamapS might know15:13
*** nayward has joined #openstack-infra15:14
jeblairmordred: the lingering file-descriptor issues; which I haven't quite tracked down enough to produce a good bug report (weird stuff started happening when I tried inspecting the object graph to find out what was going on)15:14
jeblair(inspecting the objects caused them to clean up.  sigh)15:15
mordredwow. nice15:15
mordredjeblair: btw - I was having a call with a dude at HP today telling him about how all of our stuff works (he was impressed, fwiw)15:16
*** thomasbiege1 has joined #openstack-infra15:16
mordredjeblair: and had a thought that you might have already had...15:16
mordredjeblair: the code coverage jobs that we run in jenkins right now - should we have something that takes their output and spits it into graphite?15:17
mordredjeblair: so that we could have global report page thing15:17
mordred(low priority to work on that, just wanted to say it out loud while the thought was in my head)15:17
jeblairmordred: sure, that would be a good idea i think.  want to file a bug for it?15:18
mordredjeblair: sure thing15:19
*** SergeyLukjanov has joined #openstack-infra15:19
*** changbl has joined #openstack-infra15:19
*** mrodden has quit IRC15:20
*** dina_belova has joined #openstack-infra15:21
*** cdevienne has left #openstack-infra15:21
jeblairmordred: also, we time out waiting for ssh access to hpcloud nodes _a lot_15:21
mordredjog0: wow15:22
mordredjeblair: oh yeah?15:22
mordredlike, the 600 second wait-for-node-to-really-be-ready timeout?15:22
jeblairmordred: 308 times in the 15.5 hours since utc midnight15:22
mordredhow many timeouts in that period?15:23
jeblairmordred: it's a 60 second timeout15:23
jeblairmordred: so it's _after_ the node becomes ACTIVE, it takes more than 60 seconds to establish an ssh connection15:24
Alex_GaynorSo, I want to get marconi running it's tests under pypy. To do this we need some buidlers with pypy! Where do I start to get PyPy installed under those machines (it'll be a PPA + apt-get to start with)?15:24
mordredjeblair: nod15:24
*** ruhe has quit IRC15:24
SpamapSjeblair: re using the globally cached nova client, I believe that is the case yes.15:25
jeblairSpamapS: is heat multithreaded, in particular such that multiple threads may use that single novaclient instance simultaneously?15:25
*** ruhe has joined #openstack-infra15:28
mordredjeblair: I have reported the timeouts to someone at HP and have copied Clark on the email15:28
jeblairAlex_Gaynor: i think we'd want a new slave definition (to keep it completely separate from other python unit tests)15:29
Alex_Gaynorjeblair: don't all the py2.6-3.3 ones run on the same slaves?15:30
jeblairAlex_Gaynor: you can probably pattern it after the work for the py3k slaves15:30
Alex_Gaynorjeblair: okey doke, where in teh repo are those?15:30
jeblairAlex_Gaynor: each version gets its own slave class15:30
Alex_Gaynorah ok, sounds easy enough15:30
* Alex_Gaynor has visions of spending 6 years learning puppet in a monestary in tibet15:30
jeblairAlex_Gaynor: py26 has its own because centos6 is the only os that supports it...15:30
jeblairAlex_Gaynor: py3k has its own because the pip puppet module can't deal with multiple python installations15:31
SpamapSjeblair: no it is not multi-threaded, but it does use co-routines15:31
SpamapSjeblair: so there will likely be multiple nova requests going on at one time15:31
jeblairAlex_Gaynor: and for the moment (i believe mordred is working to change this) we do have to pip install some things globally (tox, and other stuff)15:32
jeblairAlex_Gaynor: (we want those to be packages)15:32
*** mrodden has joined #openstack-infra15:33
jeblairAlex_Gaynor: so anyway, it's possible that pypy could co-exist on the py27 slaves; um, i wouldn't hold my breath though, i'm sure something will break that15:33
mordrednot going to be fixed before I'm back from the desert though15:33
Alex_Gaynorjeblair: cool, where in teh repo should I poke?15:33
jeblairAlex_Gaynor: so i'd 'git grep py3k' and model the work after that.15:33
Alex_Gaynorjeblair: sounds good15:33
jeblairAlex_Gaynor: that even has an apt-repo that's only used for that slave15:33
*** vogxn has quit IRC15:34
HenryGmordred: ping15:34
*** vogxn has joined #openstack-infra15:34
markmcclainclarkb: are you ok with me approving:
HenryGmordred: OK if I take over ?15:37
mordredHenryG: absolutely15:37
*** boris-42 has quit IRC15:37
HenryGmordred: thanks15:38
*** boris-42 has joined #openstack-infra15:39
*** UtahDave has joined #openstack-infra15:41
jeblairi'm going to manually clean up old images/nodes that leaked from the old devstack-gate scripts15:42
mordredjeblair: awesome15:42
openstackgerritA change was merged to openstack/requirements: python3: Add mox3 dependency
mordredjeblair: "15:43
mordredRemember that ACTIVE in Nova just means that the VM boot process has15:43
mordredjeblair: apparently, nova has status codes that are meaningless15:43
jeblairmordred: yeah, so basically it's a machine that's taking more than 60 seconds to boot.15:43
openstackgerritA change was merged to openstack-infra/nodepool: Delay 1 min before deleting node
jeblairmordred: we may not want to run tests on that machine.  :)15:44
*** reed has joined #openstack-infra15:44
mordredjeblair: do you happen to have success rate for that same period?15:44
mordredjeblair: and that's az1 and az2? or all three?15:45
*** blamar_ has joined #openstack-infra15:45
jeblairmordred: from midnight utc to now, fail: 369, success: 215015:45
mordredjeblair: cool. thanks15:45
jeblairso 17%15:45
*** blamar has quit IRC15:45
*** blamar_ is now known as blamar15:45
jeblairlet me see if i can narrow az15:45
jeblairmordred: i'd have to write a script to tie 2 log lines together :(15:46
reedhello folks15:46
jeblairmordred: can do if you want, let's call it a 15 min job.15:47
jeblairreed: welcome back15:47
mordredjeblair: nah. I don't think it's necessary15:47
mordredjeblair: tanks15:47
*** xqueralt has left #openstack-infra15:47
mordredjeblair: but we're not using az3 yet, right?15:48
jeblairmordred: yes we are15:48
*** pblaho has quit IRC15:50
*** ruhe has quit IRC15:53
mordredoh. neat!15:54
*** SergeyLu_ has joined #openstack-infra15:54
jeblairmordred: we are currently at quota on hpcloud15:55
jeblairmordred: check out that graph i sent earlier15:55
reedadding openstack-infra to the activity board... not sure how to deal with forked projects like gerrit though15:56
reedfirst question: is gerrit the only fork there?15:56
*** hub_cap has joined #openstack-infra15:56
pleia2good morning15:56
mordredjeblair: that is a lot of nodes15:56
hub_caphey does the zuul.o.o page define the order in the queue? is it top to bottom on that page? /me is curious15:56
*** SergeyLukjanov has quit IRC15:57
anteayamorning pleia2 morning reed15:57
jeblairmordred: also that really didn't take long.15:57
mordredreed: so - patches that we've made to the fork are activity on openstack15:57
jeblairreed: if you examine changes in gerrit (as opposed to git history), you will have the openstack-related activity15:58
mordredreed: upstream commits will not have gerrit changes associated with them - so basically, if gerrit has a change for a thing, it's openstack activity15:58
mordredyeah - what jeblair said :)15:58
reedmordred, right ... I'll have them investigate15:58
mordredreed: I promise, we're not trying to be difficult when we suggest that looking at git history for openstack is the wrong place for them to look15:58
*** vogxn has quit IRC15:58
anteayahub_cap: my understanding of is that the jobs at the top of the page are first in the queue and the line forms below them15:58
mordredthe picture that gives is just MASSIVELY incomplete15:58
anteayahub_cap: so yes, top to bottom on that page15:59
hub_capok sweet i assumed that anteaya but didnt have any concrete data behind it :)15:59
anteayamakes sense15:59
*** ruhe has joined #openstack-infra15:59
jeblairmordred: i'm going to spin up new long-running slaves.15:59
mordredjeblair: great!16:00
anteayaI have just watched them as the jobs progress and complete16:00
jeblairanteaya: gate jobs have precedence over post jobs which have precedence over check jobs16:00
anteayathere we go, more info - thanks jeblair16:00
*** nayward has quit IRC16:01
jeblairmordred: i was going to spin up 4 precise nodes.  i think we need a lot more than that.  i think i'll add 16 (double the current number)16:03
mordredjeblair: sounds good16:04
reedjeblair,  is something like this line that differentiates openstack commits from other commits in gerrit repo? Change-Id: Ibfbb8bec245ae494a49cb905925fd2c4f34686f216:05
*** thomasbiege1 has quit IRC16:06
mordredreed: no, that line will also be in the commits we got from upstream gerrit16:06
mordredbecause they also use gerrit to manage gerrit16:06
reedohhh, now I re-read jeblair's suggestion and I think I misunderstood it before16:08
*** ruhe has quit IRC16:12
*** hub_cap has left #openstack-infra16:13
reedmordred, jeblair: what other forks are in openstack-infra?16:13
*** SergeyLu_ has quit IRC16:14
mordredreed: openstack-infra/lodgeit openstack-infra/meetbot openstack-infra/puppet-dashboard openstack-infra/puppet-vcsrepo16:14
reedmordred, I understand that github is the wrong place... I wish we could go back in time and not use it at all, not even for mirroring the stuff16:15
mordredreed: you know about now right?16:15
reedmordred, when did that start"16:16
jeblairmordred, reed: i'm not sure we've quite announced it yet... but i think we're about ready to.  seems to be working.16:16
mordredyah. I've been using it anytime I send a link to someone16:16
*** boris-42 has quit IRC16:16
*** vogxn has joined #openstack-infra16:17
jeblairreed: we're about to switch all of our testing, documentation, and any other references in -infra to use it.16:17
jeblairnodes precise17-32 are launching16:18
reedjeblair, sweet16:18
*** cthulhup has joined #openstack-infra16:20
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Change hpcloud max-servers to 93
jeblairclarkb: morning!  you have to see this graph:
clarkbmarkmcclain: yes fine with me16:24
jeblairclarkb: i think we got within 2 servers of _actually_ exhausting our devstack node resources.16:24
mordredjeblair: could you explain the Used state to me? That's "node currently running a devstack test" right?16:24
jeblairmordred: yes16:24
mordredk. just making sure I was reading it right16:24
clarkbtldr a lot of tests :)16:25
*** BobBall is now known as BobBall_Away16:25
clarkbjeblair so 30 nodes seems about right16:25
jeblairclarkb: maybe? i still think it might be able to be lower16:26
*** pabelanger has quit IRC16:26
jeblairclarkb: when i said within 2 of exhaustion, i meant that we were at quota16:26
clarkboh. there did appear to be a small sliver of green at one point16:26
clarkbI think we can live with a little node exhaustion while running 250 tests16:27
jeblairclarkb: but the pool size actually only needs to be big enough so that it doesn't hit 0 between when patches are uploaded and replacement servers are spun up16:27
jeblairclarkb: once we actually hit quota, it doesn't matter what the pool size is, it'll eventually hit 016:27
*** nicedice_ has joined #openstack-infra16:28
jeblairclarkb: so basically, we'd want to set it to some value where it doesn't hit 0 during the times that there is excess capacity.16:28
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Add additional Rackspace region to nodepool config
mordredjeblair: think it's worth trying that? ^^16:30
*** markmc has quit IRC16:30
*** derekh has quit IRC16:31
clarkbmordred I do, but defer to jeblair on actual deciding16:32
mordredme too16:32
jeblairmordred: trying: yes; throwing into production without any testing like that patch does: i'm not excited about that.16:32
jeblairmordred: i'd give it a new node-type, then we can create a job that runs on that node and try it out16:33
mordredI consider jeblair's brain to be absolute on all things16:33
mordredjeblair: ahhhh. see, that's cause you're smarter16:33
* mordred goes to make that change16:33
*** sdake has joined #openstack-infra16:33
*** jpich has quit IRC16:34
* fungi agrees with everything said above, but is headed to the aquarium for a while and will catch up when he gets back16:36
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Try additional Rackspace region
jeblairmordred: how about we just manually trigger the job for now?  if something goes wrong, i'm worried about getting a backlog in the silent queue16:39
Alex_Gaynormordred: is it in the service catalog for the relevant users already? (I  know it's a phased rollout)16:39
jeblairmordred: also needs a new image section under targets16:39
openstackgerritJason Meridth proposed a change to openstack-dev/hacking: Adds ability to ignore hacking validations with noqa
mordredjeblair: nod. so leave it out of zuul altogether?16:40
jeblairmordred: yeah.  and maybe add the env variables like the periodic jobs have to make it easier to trigger16:40
jeblairAlex_Gaynor: i'll try some nova commands manually and see16:41
openstackgerritA change was merged to openstack-infra/config: Change hpcloud max-servers to 93
HenryGI submitted with no dependencies. Now I want to make it depend on
HenryGHow do I do that?16:43
*** vogxn has quit IRC16:43
mordredHenryG: git review -d 35085 ; git review -x 42473 ; git review16:44
jeblairAlex_Gaynor, mordred: yeah, i can nova boot there16:44
Alex_Gaynorjeblair: awesome16:44
*** jfriedly has joined #openstack-infra16:44
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Try additional Rackspace region
mordredjeblair: thanks - this has been a helpful exercise for me to use the nodepool yaml file (I usually learn by using)16:45
*** tjones has joined #openstack-infra16:45
jeblairmordred: lgtm.  (some of my caution is also because i'm worried a little about bitrot; it's been a while since we've run a devstack job on rax)16:46
mordredjeblair: yah. excellent point16:46
*** marun has joined #openstack-infra16:46
mordredit would be super neat if we could start running there again16:46
*** ruhe has joined #openstack-infra16:48
* mordred run to bank - biab16:48
openstackgerritRyan Petrello proposed a change to openstack-infra/config: Add WSME to StackForge.
jeblairclarkb: did you want to review 42677?16:50
clarkbjeblair: sure16:50
clarkbdoing that now16:50
*** afazekas_ has joined #openstack-infra16:51
*** fbo is now known as fbo_away16:53
clarkbjeblair: mordred: any concern with setting the total min ready nodes to 10?16:54
*** ^d has joined #openstack-infra16:55
*** ^d has joined #openstack-infra16:55
*** yaguang has quit IRC16:55
jeblairclarkb: well, none except that i'll probably have to manually clean them up, so maybe we ought to set it to 2-per master.16:56
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Try additional Rackspace region
*** cthulhup has quit IRC16:58
jeblairclarkb: got a few minutes for some jenkins clicking?16:59
clarkbjeblair: sure16:59
clarkbwhat are we clicking?17:00
jeblairclarkb: can you add the odd-numbered precise17-precise31 slaves to jenkins01?17:00
clarkbjeblair: they are already spun up?17:00
jeblairclarkb: did we decide that if we copied a jenkins node, it would immediately start an executor on the original?17:00
jeblairclarkb: yes, they are spun up17:01
clarkbjeblair: yes, copying is bad. I have been creating dumb slaves from scracth and letting my browser autofill stuff17:01
jeblairclarkb: k, thx17:01
jeblairi will do the evens17:01
*** eharney has joined #openstack-infra17:01
*** ruhe has quit IRC17:02
clarkbjeblair: though what are we doing about ssh credentials now?17:02
clarkbusing jenkins (Jenkins SSH Key) ?17:03
jeblairclarkb: yep17:03
mtreinishjeblair, clarkb, fungi, mordred: the testr parallel runs seem to be taking a long time: but the tempest run time is still the same17:03
*** pabelanger has joined #openstack-infra17:03
mtreinishdo you think it was just a devstack change that made it slow, or could something else be going on?17:03
jeblairclarkb: hrm, i think i forgot to add the credential id to nodepool; i'll have to fix that in a bit17:04
jeblairmtreinish: can i get back to you in a few minutes?17:04
mtreinishjeblair: sure np17:05
*** jerryz has joined #openstack-infra17:07
*** Ryan_Lane has joined #openstack-infra17:07
*** jerryz has quit IRC17:07
*** jerryz has joined #openstack-infra17:07
clarkbjeblair: new nodes added to 0117:09
clarkbprecise27 isn't happy for some reason17:09
Alex_Gaynorjeblair: so I'm looking at the slave.pp, I assume this is the right place, basically what I think needs to happen for a pypy builder is just to have the pypy thing installed, and then make the `tox -epypy` variant run, it doesn't really matter what python tox itself is installed with17:09
Alex_Gaynorjeblair: am I off base?17:10
clarkbAlex_Gaynor: that sounds about right to me. With the one catch that tox doesn't always give you a super clean env (we have seen tox running under py27 leak py27 libs into py26 envs)17:10
anteayahmmmm, I was about to change the link in from to but I see cgit doesn't render .rst files as .rst files17:10
anteayais there a cgit plugin to get it to render .rst files as .rst files?17:11
clarkbjeblair: I can't ssh to precise27 either17:11
anteayapleia2: do you know?17:11
*** nati_ueno has joined #openstack-infra17:11
pleia2anteaya: there is not that I'm aware of17:11
pleia2anteaya: looks like someone has done some work though:
anteayahmmm, any thoughts on how we point folks to cgit .rst files and make them happy for clicking?17:12
*** mezzovide has joined #openstack-infra17:12
*** ruhe has joined #openstack-infra17:12
clarkbjeblair: perhaps that slave got caught in the ipv6 ssh nastiness and we deleted it? I am going to chck in the account now17:12
*** dims has joined #openstack-infra17:13
jeblairclarkb: hrm, i wouldn't have thought that launch-node would print the dns info in that case17:13
clarkbjeblair: the node is present and ACTIVE17:14
clarkbjeblair: and the A record for it matches what nova list shows17:14
anteayapleia2: so basically this solution intercepts the server request for .rst files and converts them to .html files, rendering the .html, is that correct?17:14
jeblairclarkb: today would be a great day to have the git.o.o change merged.17:14
jeblairthe load average on review.o.o is 229.17:14
clarkbjeblair: wow17:14
anteayapleia2: do we have an .html file as a sample anywhere in any of our repos?17:15
clarkbjeblair: I will put the git.o.o change at the top of my review list17:15
clarkbjeblair: do you want me to keep looking at precise27 or should we just ignore that for now?17:15
jeblairi'm watching a job on a new node, and it's spending a lot of time cloning17:15
jeblairclarkb: why don't you finish looking into precise27 first17:16
jeblairclarkb: don't spend long on it; if you decide it's a lost cause, let me know and i'll recreate it (i still have a window open and commands handy)17:16
*** colinmcnamara has joined #openstack-infra17:16
*** pentameter has joined #openstack-infra17:16
clarkbjeblair: it unpings, ssh cannot get to it but nova list reports it is active. I could get on the web console, but deleting it and trying again seems simpler17:17
clarkbespecially since we haven't really invested anything in the current node17:17
*** burt has quit IRC17:18
jeblairclarkb: ok will do. i'm about to upload a new patchset of the git change17:18
pleia2anteaya: I'm sure we do, looking17:18
anteayapleia2: if I read this correctly, cgit 0.9.2 has the ability to convert .md/.rst/.txt files converted for output:
Alex_Gaynorclarkb: So is what I've got so far, can you let me know if I'm on the right track?17:19
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Use cgit server instead of github for everything
clarkbAlex_Gaynor: sure, let me review 38177 first really quick17:19
jeblairclarkb: okay, let's review that  now ^17:19
Alex_Gaynorclarkb: sure thing, thanks17:20
*** arosen has left #openstack-infra17:20
*** arezadr has joined #openstack-infra17:20
pleia2anteaya: so I noticed it would do it for some README files, let me dig in a bit17:20
clarkbjeblair: and I am going to review locally as you can really tell gerrit is struggling17:21
*** mezzovide has quit IRC17:21
jeblairclarkb: a lot (most) of the load is apache serving git17:21
clarkbjeblair: I have a feeling we should grow git.o.o if that is the case17:22
anteayapleia2: yeah when I go to the cgit home page it looks like some form of markup is at play:
mordredanteaya, jog0: you know - we might want to make the hacking guidelines a page in sphinx docs for the hacking repo17:22
clarkbjeblair: and probably before we switch everything to it (git.o.o is a 1GB node and gerrit is 32GB, if apache is most of the load ...17:22
anteayamordred: not a bad idea, what do you think jog0?17:22
jeblair1g? uh wow.17:23
*** pcrews has joined #openstack-infra17:23
mordredanteaya, jog0: and then publish to
clarkbjeblair: or spin up a few of them and load balance17:24
jeblairclarkb: we can also scale it horizontally.  so perhaps what we want to do is grow this one to 16 or 32; check out the load, and then add a few smaller ones and then delete the current one.17:24
mordredpleia2, anteaya: and in general, probably if it's a thing we want to point someone to a link to read - we should probably think of it in terms of what sphinx repo it can live in - rather than linking to magical RST things in git repos. I think. I might be wrong17:24
clarkbjeblair: ++17:24
mordredyay. more 3817717:25
*** pblaho has joined #openstack-infra17:26
anteayamordred: fair enough regarding links. I do think that digging into how to get cgit to render .rst files is a good short term use of time, given we have so many of them since we had been going that direction17:26
mordredanteaya: totally17:26
jeblairclarkb: i'll grow the current git.o.o to 30gb?  it might take a while, but that's what it takes to get 8vcpus.17:26
anteayabut introducing a stronger push to creating sphinx docs is a good push to have, I agree17:26
Alex_Gaynorremember for a spihinx thing, it's not just rendering a .rst file, it's the full sphinx system17:26
pleia2anteaya: yeah, so it seems like in this version it's really strictly tied to about pages17:26
mordredwhen we're done with 38177, and we move on to thinking about cloning/pulling from git.o.o instead of from review.o.o - should we also consider replacing the gitweb links in gerrit with git.o.o links?17:27
pleia2not random-rst-files in the repo17:27
anteayapleia2: huh, can you find the url for the cgit/about page? so far I have been unable17:27
jeblairmordred: yes17:27
mordredjeblair: cool17:27
clarkbjeblair: wfm. I think without growing it or scaling horizontally is something we need to do in any case. so waiting is fine17:27
* dhellmann perks up at the mention of rst17:27
dhellmannmordred, anteaya : the code is opensource17:27
pleia2anteaya: not sure what you mean17:28
anteayapleia2: hmmmm, so a stronger push for sphinx it feels like17:28
mordreddhellmann: yes it is17:28
anteayahey dhellmann17:28
*** olaph has quit IRC17:28
*** afazekas is now known as afazekas_zz17:28
jeblairresize in progress17:28
pleia2anteaya: the template for cgit/about or something on our specific install..?17:28
pleia2(we currently don't have a repo about page enabled)17:28
anteayapleia2: when I try to find the source code for the cgit about page I look in the cgit/tree and can not find the file marked about:
pleia2ah, gotcha17:29
dhellmannmordred, anteaya : but Alex_Gaynor is right that if we're just rendering one-off rst files, rst2html from docutils is probably simpler17:29
Alex_Gaynordhellmann: but also insufficient for things that are actually sphinx17:29
dhellmannAlex_Gaynor: yep17:29
*** ruhe has quit IRC17:29
pleia2anteaya: I don't think it's included in the package, up to the admin to set up a root-readme=/var/www/html/about.html17:29
mordredso - I think that it seems like the hacking guidelines really are what the sphinx docs of hacking should be in this particular case17:29
mordredbut I agree that rst2html for simpler/one-off things also seems correct17:30
anteayadhellmann Alex_Gaynor any opinion on whether it is worthwhile to pursue this route as an .rst to .html fix for us?
dhellmannthat makes sense17:30
anteayapleia2: ah, a behind the scenes about file17:30
Alex_Gaynoranteaya: is the usecase our sphinx docs?17:30
anteayaso we wouldn't be going that way17:30
pleia2anteaya: well, we could commit one to /config ourselves, currently dn't yet have the root-readme option enabled in cgitrc17:31
anteayathe thread I pulled that has unraveled is the link in the review guidelines wiki:
dhellmannanteaya: as Alex_Gaynor pointed out, that's going to work for the subset of rst that docutils understands natively, but not any of the sphinx extensions17:31
*** olaph has joined #openstack-infra17:31
anteayait links to the hacking.rst page on github (very pretty):
anteayawe are using cgit and I thought it would be a good idea to change the link17:32
Alex_Gaynorfor HACKING.rst and other simple documents it's probably worth it17:32
jeblairclarkb: precise27 deleted and rebuilding17:32
anteayabut .rst files in cgit render as .txt files unless they are an about page17:32
clarkbjeblair: 38177 lgtm. (waiting on gerrit to post my vote). I do think we may need to manually change the origin remotes on any puppet vcsrepo repos that change, other than that we should be good17:33
anteayaokay so short term, get current .rst files rendering nicely in cgit with this hack:
jeblairclarkb: yeah, we still have that 'set origin' command in the test scripts from a while ago, right?17:34
anteayalong term, create doc files via sphinx for any documentation and link wiki pages to sphinx generated docs17:34
anteayapleia2 Alex_Gaynor dhellmann how do you feel about these two steps?17:34
clarkbjeblair: maybe?17:35
anteayaand mordred17:35
jeblairclarkb, mordred: we are now using our full devstack-gate capacity17:35
mordredjeblair: nice17:35
dhellmannanteaya: makes sense, maybe fallback to plaintext if there is any error rendering?17:36
anteayadhellmann: I like that17:36
Alex_Gaynorclarkb, jeblair: A version with a few more things I think are needed: (is there documentation on how to do this, I'm cargo culting like mad :))17:36
pleia2anteaya: works for me, we'll want to confirm that the .rst thing isn't too hokey - yeah, failback would be nice17:36
clarkbAlex_Gaynor: there is documentation for adding new jobs to zuul and JJB17:37
anteayapleia2: okay I can work on a patch, with your help and in the meantime we can confirm the non-hokeiness of the proposal17:37
anteayaso by the time it is ready to merge we feel it is the right thing to do, yeah?17:37
clarkbAlex_Gaynor: but less so for puppet changes as I guess we assume some familiarity with it. You are on the right track. But our puppet is made up of onion layers like ogres. So you will need to pass include_pypi all the way from openstack_project::slave to jenkins::slave17:38
*** thomasbiege1 has joined #openstack-infra17:38
pleia2anteaya: perfect, once you have a patch I can toss it up on my test instance and throw some things at it17:38
clarkbAlex_Gaynor: I would also give it a default value of false so that we don't have to set it to false for the centos slaves17:38
anteayapleia2: I'll make you a nice target17:38
jeblairclarkb: hrm, no gerrit-git-prep sets the origin to gerrit17:38
jeblairclarkb, mordred: i'll work on a followup that sets origin to git.o.o17:39
clarkbjeblair: ok17:39
mordredjeblair: k.17:39
mordredjeblair: I've got a patch coming up for the gitweb links17:39
clarkbAlex_Gaynor: I would also move the ppa section into the conditional block for debian derivatives in jenkins::slave17:39
pleia2oh btw, I have to run out for a bit around 2 (have a guest in town, said we'd do some lunch adventures)17:43
*** moted has quit IRC17:43
mordredwhen did linus start writing text editors?17:43
mordred ?17:43
*** moted has joined #openstack-infra17:43
clarkbmordred: he has been a long time user of that emacs clone thing iirc17:43
anteayapleia2: I'm busy at 2 as well, so happy running around17:44
*** nayward has joined #openstack-infra17:44
Alex_Gaynorclarkb: Would this be easier if I did it via a WIP patch review?17:45
clarkbAlex_Gaynor: I want inline commenting. The zuul and JJB stuff looks just about there17:45
anteayapleia2: am I in the right neck of the woods for a cgit patch?
Alex_Gaynorclarkb: Ok, so I'll put this into gerrit then17:45
clarkbAlex_Gaynor: ya I think we are getting close enough to start refining in gerrit17:45
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Have gerrit-git-prep use git.o.o
pleia2anteaya: I just updated ! :)17:46
*** thomasbiege2 has joined #openstack-infra17:46
openstackgerritAlex Gaynor proposed a change to openstack-infra/config: Add support for running tox with PyPy and use it for Marconi
jeblairclarkb, mordred: it turns out having our tests use git.o.o is completely separate from mordred's change...17:47
pleia2anteaya: so yes, that dir, and then some in modules/openstack_project/manifests/git.pp and files/git/cgitrc17:47
Alex_Gaynorclarkb: ^17:47
anteayapleia2: fab thanks17:47
*** melwitt has joined #openstack-infra17:47
jeblairclarkb, mordred: one is s/github/ the other is s/review/.17:47
anteayaperfect, I will have a dig17:47
jeblairclarkb, mordred: and devstack-gate needs updating too.17:47
mordredjeblair: yah17:49
openstackgerritJames E. Blair proposed a change to openstack-infra/devstack-gate: Use as origin
*** olaph has quit IRC17:49
jeblairclarkb, mordred: those 2 changes should do it ^17:49
*** thomasbiege1 has quit IRC17:50
mordredfirst lgtm17:50
*** pabelanger has quit IRC17:50
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Make the gitweb links in gerrit point to git.o.o
*** dina_belova has quit IRC17:51
*** afazekas_ has quit IRC17:51
anteayapleia2: 3 links on this page go to github: 2 puppet links and 1 configuration link17:51
jeblairmordred: next time someone asks you how many jenkins slaves we have, you tell them 328.17:51
mordredjeblair: HAHAHAHA. nice17:51
clarkbI think I managed to successfully put a +2 on the first from the command line17:51
clarkbjeblair: is that all?17:52
jeblairand counting17:52
*** olaph has joined #openstack-infra17:52
anteayapleia2: btw, can I edit this page? it looks like a sphinx generated doc page17:52
mordredI have +2'd both17:52
Alex_Gaynorclarkb: I'm in disbelief that I managed to get a maybe working thing on my first try.17:52
pleia2anteaya: yeah, we haven't moved our docs over to using git.o.o yet (there are a couple patches), I need to add this one to patches too17:52
clarkbAlex_Gaynor: I don't think it is quite working, it is close though17:53
pleia2anteaya: it's in config/doc/source17:53
Alex_Gaynorclarkb: hehe17:53
jeblairmordred, clarkb: i think we want the resize to finish before merging my changes (probably yours too)17:53
clarkbjeblair: yes17:53
mordredjeblair: yah17:53
anteayapleia2: ah okay, so I will hold off until your patch goes through, yes?17:53
pleia2anteaya: ah, good, this patch includes git
pleia2so once that lands, our docs will be good :)17:54
mordredjeblair: sometimes I really don't think people understand exactly how many servers it takes to run our little project here17:54
mordredyou'd think to yourself "oh, well, I mean, serving git repos must be easy - it's git"17:54
pleia2anteaya: yeah, probably want to wait on doc edits until 38177 lands17:55
pleia2should be soon though17:55
anteayaokay, it is in the monster do-everything patch, I will review17:55
jeblairum.  git.opo has been resized to 8192, which is weird because i'm pretty sure i told it to do 30g.17:55
*** odyssey4me3 has quit IRC17:55
mordredjeblair: neat! it's great when they resize it to a different size17:55
pleia2oh - we're also not doing any caching at the moment17:56
clarkbmordred: jeblair: should gitweb and cgit be an XOR? or are we fine potenitally running both?17:56
*** pabelanger has joined #openstack-infra17:56
pleia2there are lots of caching options you can work with in the cgitrc:
jeblairi'm going to confirm the resize and then upgrade again from there (otherwise we'll have stale git repos if i revert it)17:56
*** thomasbiege2 has quit IRC17:56
clarkbjeblair: wfm17:57
jeblair(and i don't want to do a full replication push)17:57
pleia2so we should figure out what we want as far as caching options, the default "cache-size=1000" in centos was unpleasant for testing17:57
clarkbpleia2: I was going to play iwth adding some cache stuff17:57
pleia2clarkb: cool17:57
clarkbpleia2: was it the size of the cache or the ttl on the cahce items?17:57
pleia2clarkb: number of cached items, whatever that means :)17:57
pleia2"output entries"17:58
clarkbpleia2: I think we can reduce the ttl of the more dynamic stuff like log listings and so on and increase the cache ttl for static things like sha objects17:58
pleia2clarkb: yeah, like the front page we can probably set cache reset to a couple minutes or more17:58
Alex_Gaynorclarkb: given the lack of confidence that htis 100% works, should I put these builds into non-voting to start with, and if so, where do I do that?17:59
clarkbAlex_Gaynor: I see I voted on the wrong change woot17:59
clarkbAlex_Gaynor: sorry for the confusing that +2 was meant for jeblairs change18:00
Alex_Gaynorclarkb: aaaaah, this addresses my confusion18:00
*** dina_belova has joined #openstack-infra18:00
*** colinmcnamara has quit IRC18:00
clarkbjeblair's change is 42693...18:00
clarkbthis is what I get for trying to change my workflow to not bother review.o.o apache so much18:01
*** boris-42 has joined #openstack-infra18:02
*** thomasbiege has joined #openstack-infra18:04
*** morganfainberg|a is now known as morganfainberg18:04
jeblairmtreinish: looking18:05
clarkbAlex_Gaynor: your review has been udpated18:06
mtreinishjeblair: ok thanks18:06
clarkbtoo much excitement for monday but I guess that is what we get during freeze week18:06
jeblairmtreinish: looking at
jeblairmtreinish: it spent 40 minutes inside the wrap script.  i don't have detailed timestamps for that, but my suspicion is that most of it was spent doing git operations18:07
jeblairmtreinish: which, interestingly enough, is what we're working on speeding up right now.  :)18:08
*** pabelanger has quit IRC18:08
openstackgerritAlex Gaynor proposed a change to openstack-infra/config: Add support for running tox with PyPy and use it for Marconi
mtreinishjeblair: nice18:09
mtreinishok, yeah 40 min is a long time for that, but that adds up well with the 20-30min for tempest in parallel :)18:09
jeblairdtroyer: how does devstack get those nifty bash timestamps?18:11
clarkbAlex_Gaynor: so close. One JJB quirk and one puppet bug to work around then I believe that change is good18:13
jeblairyay we're about to get more slaves :)18:13
Alex_Gaynorclarkb: coolio, will have a new version up in a few minutes, just need to finish writing an eamil18:13
clarkbjeblair: hmm?18:14
jeblairclarkb: pypy slaves :)18:14
jeblairclarkb: 328++18:14
*** thomasbiege has quit IRC18:15
jeblairoh, i think i just found the thing in devstack that does the timestamps.18:15
openstackgerritAlex Gaynor proposed a change to openstack-infra/config: Add support for running tox with PyPy and use it for Marconi
*** jswarren has joined #openstack-infra18:16
jeblairAlex_Gaynor: those should probably be two separate changes, otherwise we'll get a queue backlog while we build the hosts18:17
*** rihnapstor has joined #openstack-infra18:17
jeblairAlex_Gaynor: (which, sadly, at the moment is not automatic, but maybe soon)18:17
*** colinmcnamara has joined #openstack-infra18:18
Alex_Gaynorjeblair: sorry, what's not automatic?18:18
jeblairAlex_Gaynor: creating a new jenkins slave18:18
clarkbjeblair: it isn't a new slave. it is overloading the py3k slaves18:18
jeblairoh, i see18:18
jeblairperhaps i should actually read it18:19
*** colinmcnamara has quit IRC18:19
*** vipul is now known as vipul-away18:21
*** thomasbiege has joined #openstack-infra18:23
openstackgerritAlex Gaynor proposed a change to openstack-infra/config: Add support for running tox with PyPy and use it for Marconi
Alex_Gaynorlifeless: so trying to run any of the testr projects with tox -epypy results in "memoryview has no attribute 'decode'", which is a bit bizzaro to me, since it doesn't on CPython either, there's no traceback, any suggestions on how to debug?18:24
*** thomasbiege has quit IRC18:24
*** SergeyLukjanov has joined #openstack-infra18:28
dtroyerjeblair: context?  in the logs?18:30
jswarrenSeeing several Jenkins failures in seemingly unrelated changes..for nova, keystone and glance.18:31
jeblairdtroyer: yeah, i found it; the "exec 1>" bits18:31
jswarrenErrors running gate-grenade-devstack-vm18:31
jswarrenSeveral instances of "ERROR:root:Could not find any typelib for GnomeKeyring"18:32
dtroyerjeblair: yup.18:32
jswarrenAnd "glance: error: unrecognized arguments: --public"18:33
jswarrenExamples: and
*** lbragstad has joined #openstack-infra18:34
*** olaph has quit IRC18:35
dtroyerjswarren: the —public thing is because they removed the —public option without thinking anyone used it…no backward compatibility I guess?18:35
*** xBsd has joined #openstack-infra18:36
jswarrenHmm.  How could such a change get merged if Jenkins fails?18:36
dtroyerjswarren: the clients are supposed to maintain that as they are supposed to work all the way back to Essex I think is what we agreed to a while back.  Grenade/devstack pulls the current clients as they are not released in the same cycle18:36
dtroyerjswarren: it got merged because trunk devstack doesn't use it18:36
dtroyerbut in grenade both versions use current clients.  we may need to change that, but also, this is catching breakage that shouldn't happen18:37
jeblairdtroyer: did we not run a grenade job for that change?18:37
*** burt has joined #openstack-infra18:37
dtroyerjeblair: I don' know, I haven't gone looking for it.  but in general we don't run grenade on clients do we?18:37
jeblairdtroyer: we should; it is configured for python-glanceclient18:38
*** vipul-away is now known as vipul18:39
dtroyerjeblair: ok, I hadn't looked at them in a while18:39
dtroyerIt's this change:   and no grenade run18:39
jeblairgate-grenade-devstack-vm is there18:40
dtroyerdoh, it is18:40
*** tjones has quit IRC18:40
*** olaph has joined #openstack-infra18:40
dtroyerhmmm…I need to look at this…the base run clearly worked with —public18:41
dtroyerah, glacneclient is installed from pip as part of the prereqs.  devstack stable/grizzly is still the old borked way for deps...18:42
ttxmordred: still playing with the merge-mp-back-to-master on hte swift m-p branch ? Or can I delete it now ?18:42
jeblairdtroyer: weird, it should be installed from source first18:45
dtroyerso chicken-n-egg here…original grenade run uses glanceclient  with —public still present…then test without —public under devstack that doesn't use it…it passes and is merged and now gets used in further tests with stable/grizzly devstack that wants —public.18:45
jeblairdtroyer: as i read it,  it installed for the old run and for the new (i don't know why those are different)18:46
clarkbjeblair: how is the second resize going?18:46
jeblairclarkb: it looks to be done!18:46
dtroyerjeblair: the long-and-short is they broke cli backward compatibility18:46
clarkbcpuinfo and free confirm the new hardware18:46
jeblairdtroyer: yeah, they should probably revert that patch18:47
jeblairdtroyer: but i still think it shouldn't have landed18:47
clarkbI am going to grab lunch shortly, not sure if we want to merge the git.o.o cutover changes now or when we can all help debug18:47
*** olaph has quit IRC18:47
dtroyerjeblair: I see why it did and it's inherent in how grenade is built.18:49
dtroyerdevstack was changed in, frankly that's how it should work anyway using the documented options.  but that's why it got in18:52
* clarkb lunches. feel free to merge the git.o.o switch stuff, but I won't be able to help updating remotes for about an hour18:53
clarkbthough the g-g-p change is probably most useful18:53
*** nayward has quit IRC18:53
*** mrmartin has joined #openstack-infra18:54
jeblairdtroyer: yes, that makes sense.  i think it installed the old version of glanceclient for the grizzly setup (actually, it would be nice if it didn't do that for the clients, but i think we actually need to test that with a different test), and then installed the proposed change for the new test18:55
*** mriedem has quit IRC18:55
jeblairdtroyer: which as you say, didn't exercise that option18:55
*** mriedem has joined #openstack-infra18:56
jeblairdtroyer: (i think the new job we need is 'test proposed changes to clients against stable branch devstacks')18:56
jeblairdtroyer: so i think we need to revert the commit or backport the option change to devstack stable branches, yeah?18:57
mrmartinjeblair: hi, could you help me with this review:
jeblairclarkb: yes, let's do it after lunch18:57
dtroyerjeblair: the only problem there is the false security as theat is far from complete test, it'll only say that devstack jobs are likely to complete…which may be enough18:57
*** ruhe has joined #openstack-infra18:58
jeblairdtroyer: you're right, the proper solution is a tempest environment that runs cli tests with proposed changes to the cli18:58
jeblairmrmartin: sure18:59
openstackgerritRyan Petrello proposed a change to openstack-infra/config: Add WSME to StackForge.
jeblairmrmartin: what's your question?19:00
mrmartinjeblair: I got a failure with puppet-lint: , does it mean, that a warning will result in Failure?19:00
*** olaph has joined #openstack-infra19:00
jeblairmrmartin: yes, our puppet-lint is MEAN.  if you follow that link, and then the console log, you'll see it points out the 3 warnings about => alignment19:01
jeblairmrmartin: it's a bit hard to read, but the lines like "2013-08-19 12:31:03.391 | manifests/site.pp - WARNING: indentation of => is not properly aligned on line 158"19:01
jeblairmrmartin: are pretty much the only ones that matter in that output19:01
mrmartinyeah I see it, and this is not a problem for me, but if you check the previous one:
mrmartinit was going through all files, and displayed the hints for them19:02
jeblairmrmartin: oh, did you not fix all of those between the two patchsets?19:02
mrmartinbut for second run, this just show warning fors manifests/site.pp19:02
mrmartinjeblair, I fixed most of the warnings, but I forgot the site.pp, and I have >80length issues with another file19:03
mgagnejeblair: I think we should redirect stderr to stdout for puppet-lint, this would avoid the output from being mixed up (rake aborted! in between the error output)19:04
mrmartinI got that I need to repair those, it was just strange for me, that the first run checked all the files, and the second one just displayed error for site.pp19:04
mordredyikes. scrollback19:04
SlickNikreed: got a minute?19:04
jeblairmgagne: sounds like a good idea19:04
mgagnejeblair: I'll take care of that19:05
reedSlickNik, not really, heading out for lunch...19:05
*** w_ has joined #openstack-infra19:05
jeblairmrmartin: if there are supposed to be other warnings, i don't know why it wouldn't show up there.  maybe a bug?  it should run on all the files all the time19:05
SlickNikokay, will ping you later after lunch.19:05
dtroyerjeblair: I proposed a glanceclient revert:, let's see what kind of argument we can stir up… ;)19:05
SlickNikreed: okay, bon apetit, will ping you later. :)19:05
*** nati_uen_ has joined #openstack-infra19:06
mrmartinjeblair: ok, I'll repair the site.pp and push the patch again, let's see the result.19:06
*** olaph has quit IRC19:08
*** nati_ueno has quit IRC19:08
*** ruhe has quit IRC19:09
openstackgerritMathieu Gagné proposed a change to openstack-infra/config: Redirect stderr to stdout for puppet-lint
*** olaph has joined #openstack-infra19:11
*** w_ has quit IRC19:11
*** rihnapstor has quit IRC19:14
*** olaph has quit IRC19:19
ekarlso-jenkins is utterly slow today or ?19:21
lifelessAlex_Gaynor: hi19:22
openstackgerritMarton Kiss proposed a change to openstack-infra/config: Add node
*** olaph has joined #openstack-infra19:23
*** apcruz has quit IRC19:27
*** fbo_away is now known as fbo19:27
mordredclarkb, Alex_Gaynor: I +2'd - looks good - but I remember seeing a question about making it silent first in the scrollback19:28
*** pabelanger has joined #openstack-infra19:30
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Add job to propose tag merges
openstackgerritA change was merged to openstack-infra/config: Voting: false for py33, pylint for tuskarclient
clarkbekarlso-: yes it is very slow. we throw more slaves at it which causes gerrit git to slow down... we are in the process of throwing more resources at that problem19:39
clarkbjeblair: lunch was a little quicker than anticipated. I am ready to switch over to git.o.o whenever you are19:40
*** nati_uen_ has quit IRC19:41
*** tjones has joined #openstack-infra19:42
*** renier_ has joined #openstack-infra19:42
*** nati_ueno has joined #openstack-infra19:42
dansmithdtroyer: is there something proposed for the glance breakage?19:43
dtroyerdansmith: yes,  I'm trying to scare up markwash but I think he's at lunch19:43
clarkbdtroyer: jeblair: should we consider kicking everything out of the gate?19:43
dansmithis he going to approve with that all-one-line commit message? :)19:44
clarkbso that if 42706 is good to go it doesn't get delayed? of course keeping everything else out of the gate is probably more trouble than it is worth19:44
dtroyerthat's how the revert-change button worked…I should try to fix it.19:44
dansmiththen maybe it's broken in the original commit?19:44
dtroyerno, that's what I put in the message box19:44
jeblairclarkb: when it's actionable, we can kick things out.19:45
dansmithwell, I just wanted to know if I should be reverifying things yet.. :P19:45
* dansmith sits quietly in the corner19:45
*** renier_ has left #openstack-infra19:46
clarkbpleia2: what was wrong with 1000 cache entries? was it too much?19:46
jeblairclarkb: how about i pick a precise slave, shut down puppet, and manually install the new gerrit-git-prep there to test?19:47
pleia2clarkb: for me in testing, yeah it was too much, but with lots of users it may e ok19:47
clarkbjeblair: sure19:47
dtroyerdamn, that created another review...19:47
clarkbpleia2: were you having memory issues?19:47
pleia2clarkb: no, but I didn't stress test it much19:47
pleia2it's on a hpcloud instance with 8G of ram19:48
pleia2(my test one)19:48
clarkbpleia2: what exactly was wrong with it? I guess I don't understand what issues were seen with it19:49
dtroyerfor the record, the current glanceclient revert review is
pleia2clarkb: what was wrong with caching? it just cached too much, I'd be working on things, make a change to a config, cache takes forever to turn over with one user19:50
pleia2so my pages wouldn't update with config changes, changes to repos19:50
clarkbgotcha, so TTLs need tuning19:50
pleia2by default on centos the only cache thing specifically defined was the cache-size, I didn't play around with the other caching options19:51
*** sandywalsh has quit IRC19:53
*** derekh has joined #openstack-infra19:54
openstackgerritClark Boylan proposed a change to openstack-infra/config: Enable cgit caching.
clarkbpleia2: ^ that lowers the cache TTL for some of the dynamic things that can be cached19:54
clarkbpleia2: looking in cgitrc(5) those two values seemed like they would be the most dynamic for us19:55
pleia2clarkb: wfm, want to toss in one for cache-root-ttl=1 too? (that should cache the front page too)19:55
clarkbpleia2: I can add that too. How often does the front page change?19:55
pleia2clarkb: only when projects are added/removed19:56
pleia2and changed I guess, since it has a minute by minute timestamp19:56
pleia2in the Idle column19:56
jeblairjog0: do you have any idea what that novaclient exception means?19:56
openstackgerritClark Boylan proposed a change to openstack-infra/config: Enable cgit caching.
clarkbpleia2: ^ that lowers the root ttl to one minute19:57
pleia2clarkb: aah, I just realized it defaults to 519:59
clarkbyeah just about everything does but the scan ttl19:59
pleia2clarkb: yeah, I got this mixed up with scan, sorry19:59
clarkbI think we can try these values as they shouldn't affect the change jeblair is testing19:59
clarkbpleia2: you think the scan ttl should be lower too?20:00
pleia2sounds good20:00
pleia2clarkb: no, I think it's good20:00
clarkbI am personally not too worried about what happens when new projects are added as there are enough moving parts that trying to keep everything in sync is a losing battle20:00
*** colinmcnamara has joined #openstack-infra20:00
clarkbkeeping stuff within reasonable bounds is doable though20:00
jeblairclarkb: can you review and approve ?20:01
jeblairclarkb: (i do want your eyes on it, but it seems to work on precise1)20:01
jeblairclarkb: (i tested both parts; they should be able to apply independently and keep working)20:01
clarkbjeblair: done20:02
clarkbjeblair: yeah I actually reviewed it20:02
clarkbmade sure there were no errant GERRIT_SITEs left behind and so on20:02
jeblairclarkb, mordred: the novaclient object for hpcloud-az1 barfed and is now permanently broken20:03
jeblair(in the running nodepool)20:03
clarkbone thing we need to be careful with is that for any of the jobs running on proposal that we don't try to push to a git:// remote20:03
clarkbI checked that the existing jobs don't do that and I think we are good (they use git review)20:03
jeblair(yet another reason for gerrit and origin to be different remotes)20:04
*** thomasbiege has joined #openstack-infra20:04
clarkbjeblair: permanently within the state of the process or even if new objects are created?20:04
jeblairclarkb: well, we never recreate client objects... actually... that does suggest a way to fix it20:04
jeblairclarkb: we could change an authentication value, then change it back20:05
openstackgerritA change was merged to openstack-infra/config: Have gerrit-git-prep use git.o.o
jeblairclarkb: that should give us a new novaclient for az120:05
*** sandywalsh has joined #openstack-infra20:06
jeblairSpamapS, jog0: you may also be interested in bug 121411820:07
uvirtbotLaunchpad bug 1214118 in python-novaclient "ClientException: <attribute 'message' of 'exceptions.BaseException' objects>" [Undecided,New]
jeblairSpamapS, jog0: which we're now seeing in our very heavy novaclient use20:08
clarkbjeblair: is that change worth puppeting or do you just want to ninja it in?20:09
jeblairclarkb: i think it's more of a cowboy change20:09
clarkbjeblair: that hack seems sound to me. We will leak the file descriptors but there aren't too many of them right?20:10
jeblairclarkb: yep20:11
jeblairclarkb: it currently has 100 fds20:11
morganfainbergclarkb: i'm not sure if this should be brought up with infra or glance or devstack: bug 1214094 is causing a ton of failures in granade gate checks (--public parameter was deprecated in glance), at least that appears to be the issue.20:15
uvirtbotLaunchpad bug 1214094 in devstack "glance --public parameter has been deprecated" [Undecided,New]
morganfainberggate/jenkins etc.20:15
clarkbmorganfainberg: dtroyer has proposed a fix20:15
morganfainbergclarkb: awesome.20:16
morganfainbergclarkb: thanks20:16
clarkbmorganfainberg: I think the gate will basically be spinning wheels until that change or something else addressing the problem merges20:16
morganfainbergclarkb: yep.  good to know.20:16
morganfainbergclarkb: at least someone is aware of it / working on it.20:17
morganfainbergi just didn't want it to lay around with no one aware.20:18
clarkbjeblair: did you update the precise27 slave's DNS records?20:18
clarkbmorganfainberg: yup, no problem20:18
jeblairclarkb: negative20:18
* clarkb tries to remember how to use statusbot20:19
clarkbjeblair: is it #status notice somemessage here?20:19
*** xBsd has quit IRC20:19
clarkbjeblair: I can add the DNS records if you like20:19
jeblairclarkb: if you want to broadcast20:19
clarkbjeblair: I figure broadcasting that hte gate is sideways while grenade gets sorted may be helpful20:20
jeblairclarkb: that does not set the topic20:20
jeblairclarkb: alert sets topics20:20
clarkbah, we should probably set the topic too.20:20
clarkbI will use alert20:20
jeblairclarkb: dns updated20:20
clarkb#status alert gate-grenade-devstack-vm is currently failing preventing merges. Proposed fix:
clarkbstatus bot has left us again :)20:22
clarkber :(20:22
*** sandywalsh has quit IRC20:23
*** vipul is now known as vipul-away20:23
*** vipul-away is now known as vipul20:23
morganfainbergpoor status bot.20:24
morganfainbergit just wants to feel loved.20:24
clarkbjeblair: I am going to try starting statusbot with its initscript20:25
jeblairclarkb: hrm, the recreate-client trick didn't work.20:25
clarkbjeblair: could it be an issue with the account?20:26
jeblairclarkb: (i mean, it logged messages saying it recreated it, but the (presumably new) client is still throwing the same errors)20:26
clarkbmordred: you around?20:26
jeblairclarkb: i will try nova deleting something20:26
*** openstackstatus has joined #openstack-infra20:27
clarkbhurray statusbot has returned20:27
clarkb#status alert gate-grenade-devstack-vm is currently failing preventing merges. Proposed fix:
mrmartinjeblair: it was accepted by Jenkins after puppet-lint cleanup.
openstackstatusNOTICE: gate-grenade-devstack-vm is currently failing preventing merges. Proposed fix:
*** ChanServ changes topic to "gate-grenade-devstack-vm is currently failing preventing merges. Proposed fix:"20:27
jeblairclarkb: the nova delete _is_ taking a long time and has not returned20:28
clarkbI wonder if we long into the gui if it will show us a big red banner20:28
* clarkb tries20:28
jeblairclarkb: az1 only if i didn't mention20:29
jeblair# nova delete 215176720:29
jeblairERROR: n/a (HTTP 500)20:29
jeblairi think i understand the novaclient exception now.20:29
jeblairthere is no message.20:29
clarkbgui is taking forever to open az1 compute20:30
clarkbjeblair: that errored too. I think this problem may be independnet of node pool20:31
*** harlowja_ has quit IRC20:31
jeblairclarkb: this is where if someone from hpcloud were in here, we could let them know their cloud was broken.20:31
clarkbjeblair: ya20:31
*** harlowja has joined #openstack-infra20:31
clarkbjeblair: as is I will try walking down the hall to see if anyone can help20:31
*** w_ has joined #openstack-infra20:31
jeblairclarkb: the zuul graph says we launched apprx 600 jobs last hour.  i believe that's the first time we've topped 500?20:33
*** olaph has quit IRC20:33
clarkbjeblair: no, I think we have gone over at least one other time20:35
clarkbbut it doesn't happen often20:35
*** sandywalsh has joined #openstack-infra20:36
*** vipul is now known as vipul-away20:38
*** w_ has quit IRC20:39
clarkbjeblair: load on review.o.o seems to be falling. Have you been keeping an eye on the application of the g-g-p update?20:40
jeblairclarkb: just started checking in on it20:40
*** w_ has joined #openstack-infra20:42
clarkbjeblair: once you are happy with it I would like to try merging the cgit cache change
jeblairclarkb: jenkins-jobs are running on 01 and 0220:43
jeblairclarkb: i'm sure i will need to re-run them20:43
jeblairbecause of the timeout20:43
jeblairclarkb: i'm also checking to make sure the script is updated on all the slaves20:44
clarkbjeblair: cool. should I start checking centos and you precise or odds/evens?20:44
jeblairclarkb: would you mind checking the centos and py3k?20:44
jeblairclarkb: i'll finish up precise20:44
clarkbwill do20:44
*** dkliban has quit IRC20:45
jeblairclarkb: md5sum 5bbb93f4559b947bf1959591a636f12c  /usr/local/jenkins/slave_scripts/gerrit-git-prep.sh20:45
*** vipul-away is now known as vipul20:46
*** w_ has quit IRC20:47
mordredclarkb: sup20:48
jeblairclarkb: precise nodes check out; i'm running jenkins jobs on 01 and 02 now20:49
clarkbjeblair: centos nodes are fine20:49
clarkbprecisepy3k-4 is the only one that I found with a problem. Correcting it now20:50
*** w_ has joined #openstack-infra20:50
jeblairclarkb: wow, jenkins-jobs claimed it had nothing to do.  is it possible that job update run actually completed?20:50
clarkbmordred: HP AZ1 went sideways. I hopped on hipchat and am sort of following along. tl;dr we need to ride it out20:51
jeblairclarkb, pleia2: reviewing cache change now20:51
clarkbmordred: I figured you might have some great insight20:51
mordredclarkb: ah. ok. so, I just started reading scrollback - but the underlying cause is az1 itself going batty?20:51
clarkbmordred: yup20:52
*** thomasbiege has quit IRC20:53
*** thomasbiege has joined #openstack-infra20:53
jeblairclarkb pleia2: change lgtm; what about setting cache-static-ttl to something high, like 15?20:53
clarkbjeblair: py3k-4 is good now20:53
clarkbjeblair: we can do that.20:53
clarkbjeblair: my only concern was using up all of the available cache entries and not cleaning them up quickly20:54
clarkbjeblair: but we can increase 1000 to something else if we notice that happening20:54
jeblairclarkb: ok, let's go with it as-is and see what it looks like20:54
mordredok. I think I'm caught p20:54
jeblairclarkb: aprvd20:54
pleia2I figure we'll modify the cache stuff after some testing20:54
clarkbjeblair: thanks20:54
pleia2thanks jeblair20:54
jeblairpleia2: some what now? :)20:55
pleia2live testing!20:55
jeblairpleia2: yes, we're about to throw a bunch of load at it much sooner than i would have expected.  :)20:55
pleia2go go git20:55
jeblairin fact.  the load average on git.o.o is currently 37.20:55
* mordred wonders if jeblair has been replaced by mordred20:55
pleia2oh boy20:55
clarkbjeblair: it is 66 on review.o.o20:56
mordredoh wow. 37 on git.o.o?20:56
jeblairclarkb: yay we distributed load!20:56
*** w_ has quit IRC20:56
*** olaph has joined #openstack-infra20:56
pleia2mordred: anteaya found this for you and jeblair
*** thomasbiege has quit IRC20:57
*** thomasbiege has joined #openstack-infra20:57
clarkbjeblair: was approved, but with the current state of the gate it will be hours before merging20:57
clarkbjeblair: I think now might be a good time to kill zuul and start this train over20:58
jeblairclarkb: actually, i think it will never merge20:59
jeblairclarkb: i think we need to bypass testing on that one20:59
mordredjeblair: does it need to merge before it can merge?20:59
*** colinmcnamara has quit IRC21:00
clarkbjeblair: fun21:00
jeblairmordred: yep.21:00
mordredand do we need to accelerate backporting the requirements updates to stable/grizzly?21:00
jeblairclarkb, mordred: if you are in agreement, i'll make it so21:00
mordredjeblair: ok. I'm second the vote to force merge it21:00
clarkbjeblair: yes, it appears that the upgrade testing has us in an unhappy place unti lthat goes in21:00
jeblairmordred: that's not what caused this to get by21:01
*** emagana has joined #openstack-infra21:01
clarkbmordred: iirc it has to do with how grenade works21:01
clarkbgrenade is using merged code for the clients21:01
jeblairmordred: the combo that is not tested is "new client lib being used by stable devstack branch"21:01
clarkband not testing proposed code21:01
jeblairclarkb: that's not correct21:02
jeblairclarkb: it's half-correct21:02
jeblairclarkb: it is using master for the setup, and proposed code for the new run21:02
clarkbbut since master is broken we still have half a broken setup21:03
jeblairclarkb: master glanceclient doesn't work with stable openstack21:03
jeblairdevstack in this case21:04
mordredjeblair: ah21:04
*** dkliban has joined #openstack-infra21:04
mordredjeblair: so, in theory, if we get the "gate trunk client vs. stable servers" in it should trap this21:04
jeblairanyway, it could be fixed by using the proposed code for the clients in the grenade setup, or via a 'test proposed client changes against stable branches of servers (and devstack" job.21:04
jeblairmordred: yep21:05
mordredjeblair: might be a good test case, btw...21:05
openstackgerritA change was merged to openstack-infra/config: Enable cgit caching.
jeblair is merged21:06
*** derekh has quit IRC21:07
clarkbcool. once the gate resets itself we should see things start flowing again. We can short circuit that be killing a test running in the change at the top of the queue. I will go ahead and do that I think21:07
jeblairclarkb: the one at the top may merge21:07
clarkbactually. I will wait since the head of the queue is neutron which can merge21:07
clarkbI will do it to the second change21:07
jeblairclarkb: sounds good21:07
*** woodspa has quit IRC21:08
*** colinmcnamara has joined #openstack-infra21:08
clarkball done.21:08
*** olaph has quit IRC21:08
clarkbsorry sacrificial nova change. It was for the greater good21:09
*** olaph has joined #openstack-infra21:09
*** mrmartin has quit IRC21:10
* mordred bows his head in sorry for the nova change21:11
mordredor in sorrow21:11
* mordred blames the dubstep and the boxes covered in playa dust filling his livingroom21:11
*** SergeyLukjanov has quit IRC21:11
*** thomasbiege has quit IRC21:12
*** pabelanger has quit IRC21:13
*** w_ has joined #openstack-infra21:13
*** pentameter has quit IRC21:13
Alex_Gaynorclarkb: so it looks like I got the ppa syntax wrong in teh puppet file, is there an example of this or some docs I should look at?21:14
*** olaph has quit IRC21:14
clarkbAlex_Gaynor: there is an example. let me find it21:14
mordredAlex_Gaynor: modules/salt/manifests/init.pp21:16
openstackgerritAlex Gaynor proposed a change to openstack-infra/config: Add support for running tox with PyPy and use it for Marconi
Alex_Gaynormordred: sigh, was missing a colon21:16
clarkbload on review.o.o is ~17121:16
clarkbpoor gerrit21:17
Alex_Gaynorclarkb: gerrit is a computer, i"m sure it doens't mind21:17
mordredclarkb: load on git.o.o is 0.2521:17
mordredAlex_Gaynor: it's not a computer - it's a virtual computer21:17
Alex_Gaynormordred: sometimes my friends ask me what rackspace does, my answer is "You know how some companies will sell you a computer? We'll rent you a small fraction of a computer."21:17
mordredAlex_Gaynor: nice21:18
mordredI have the hardest time describing what I do to my friends who mostly work in theatre21:18
Alex_Gaynor(not taolly true since we have the dedicated business, but uslally good for a laugh)21:18
*** boris-42 has quit IRC21:18
clarkblooks like it is still git load. jeblair do we need to update zuul to talk to git.o.o as well?a21:18
jeblairclarkb: no, it's the devstack jobs21:18
mordredI don't run a cloud, I don't really even develop the software that runs a cloud - I develop software that helps people develop software that people can use to run computers that let people run computers to run their applications21:19
clarkboh yes, the other significant portion of our load21:19
clarkbjeblair: did the d-g change merge?21:19
jeblairclarkb:  it hasn't been approved21:19
jeblairit just finished it's second round of tests21:19
mordredand failed grenade21:19
jeblairas expected21:19
clarkbjeblair: I think we should force merge that one too if we want it in quick21:19
clarkbotherwise it may get stuck in teh gate (which isn't the end of the world now that grenade should be happy)21:20
jeblairclarkb: i don't think we're that desperate21:20
clarkbI am going to approve it21:20
jeblairclarkb: ok.  the output looks correct21:20
mordredjeblair: did I miss something in the scrollback about why git usage against gerrit suddenly just went apeshit today? did we just hit a crazy usage spike coupled with the grenade failures?21:21
clarkbmordred: thats it21:21
jeblairmordred: we got very good at running lots of jobs; i don't think grenade was a factor21:21
clarkbmordred: the 21st is the last day to propose feature change sto nova and cinder. so everyone is pushing nova changes21:21
clarkbmordred: if you look at the check queue it is like 95% nova21:22
jeblairmordred: i think it's mostly nodepool letting us scale up to the point where we can meet the load for the feature freeze21:22
*** rfolco has quit IRC21:22
jeblair(which was why i have been rushing to get nodepool into place)21:22
*** w_ has quit IRC21:22
jeblairand of course we've found our next scalability bottelneck.  :)21:22
Alex_Gaynormordred: FTR one of the pypy developer's reactions to your job description: "I develop software for people like that"21:23
clarkbpleia2: does apache need to be restarted to pick up the cgit changes?21:23
pleia2clarkb: not in cgitrc21:23
mordredjeblair: nice!21:23
mordredAlex_Gaynor: haha. :)21:23
*** dina_belova has quit IRC21:23
clarkbpleia2: I see it now. I had to access some pages to create cache items21:24
clarkbpleia2: I guess I had let them all time out21:24
jeblairclarkb: earlier i straced a git upload-pcak process. it was very slowly reading all of the refs directories21:24
* pleia2 nods21:24
jeblairthere are about 9000 in nova21:24
jeblairi wonder if we should start packing refs?21:24
clarkbjeblair: doesn't the gc do that for us?21:25
jeblairi'm slightly concerned about that because i believe they are very hard (eg, you can not do it with normal git commands) to unpack.21:25
mordredjeblair: what does packing refs mean/do?21:25
jeblairthe only reason we were able to remove the zuul refs is because they aren't being packed21:25
clarkbmordred: convert the loose object files into pack files which are compressed and contain a bunch of loose files21:25
jeblairmordred: flattens the refs filesystem hieararchy into files with lists of refs21:25
jeblairclarkb: not objects, _refs_21:26
jeblairclarkb: we are packing objects21:26
clarkblooks like only branch heads will automagically unpack (which makes sense as they move)21:27
* mordred wonders what this might do to the refs/for namespace and/or is there a way to filter that from our replication?21:27
jeblairi'm also not sure what would happen if we moved a tag.21:28
jeblairmordred: refs/for is not a real namespace21:28
jeblairmordred: are you thinking of changes?21:28
clarkbgerrit MiTMs refs/for21:28
jeblairthat can be filtered.  i'm not sure we want to filter it though.21:28
clarkbis gerrit's FS ext3?21:29
jeblairi like the idea that the full change history is available in the git repo.21:29
clarkbmaybe we can make stating lots of files less painful by changing the file system?21:29
mordredif cgit is taking a lot of time to walk refs trees - it could be on account of the massive number of refs/changes - it might be interesting to see what the difference in serving a repo with and without them is21:29
mordredjeblair: I agree - I'm not sure we want to filter it either21:29
mordredbut I'm curious as to what the cost of replicating it everywhere is21:30
Alex_GaynorIs there anyone who knows stuff about testr besides lifeless ?21:30
jeblairmordred: i would expect that if they are not there, it would not take long to look at them.21:30
mordredAlex_Gaynor: how deep do you need to know?21:30
clarkbAlex_Gaynor: I end up helping out as much as I can but I definitely don't know as much as lifeless21:30
Alex_Gaynormordred: I need to know why under pypy it prints: 'memoryview' object has no attribute 'decode'21:30
jeblairmordred, clarkb: i'd like to understand the problem more.  git.o.o is so unloaded now, it's hard to dig deeper21:31
clarkbAlex_Gaynor: does memoryview need a tobytes() first?21:32
* clarkb greps for memoryview in testr21:32
Alex_Gaynorclarkb: no clue! I can't imagine what pypy is doing differently than CPython here21:32
mordredAlex_Gaynor, clarkb: subunit uses memoryview21:32
lifelessAlex_Gaynor: subunit/v2.py21:33
lifelessAlex_Gaynor: if safe_hasattr(__builtins__, 'memoryview'):21:33
lifelessAlex_Gaynor:     body = memoryview(packet[-1])21:33
mordredAlex_Gaynor: and around line 45621:33
mordred            return utf8_bytes.decode('utf-8'), length+pos21:33
mordredutf8_bytes is a slide of that body variable lifeless pasted21:34
Alex_Gaynorlifeless: now that you're here! What should I comment out to get the full TB instead of just printing the error?21:34
lifelessAlex_Gaynor: I ponged you a couple hours back :)21:34
Alex_Gaynorlifeless: doh :( sorry I missed it21:34
mordredAlex_Gaynor: my favorite - traceback hiding :)21:34
jeblairclarkb: did you add precise27?21:34
clarkbjeblair: no sorry. I will do that now21:35
jeblairclarkb: no problem; it looks like dns is updated21:35
clarkbjeblair: actually it seems to have added itself21:35
jeblairah of course.  :)21:35
lifelessAlex_Gaynor: export TESTR_PDB=121:35
clarkbsince I never removed the node def in jenkins21:35
clarkbA memoryview supports slicing to expose its data. Taking a single index will return a single element as a str object. Full slicing will result in a subview: I wonder if pypy returns a subview each time21:36
clarkblifeless: Alex_Gaynor ^21:36
*** w_ has joined #openstack-infra21:36
mordredlifeless: how do I run python-subunit's tests?21:36
*** dkranz has quit IRC21:36
Alex_Gaynorclarkb: nope, that seems to work ok21:36
jeblairclarkb, mordred: perhaps when the gate resets in a minute or so, git.o.o will get slammed and we can see it slow down again21:37
lifelessAlex_Gaynor: whats the bigger context here?21:37
mordredjeblair: that would be nice21:37
clarkbjeblair: it won't get slammed by d-g jobs though, just unittests and docs and things21:37
mordredlifeless: Alex_Gaynor is trying to run marconi's tests using pypy21:37
Alex_Gaynorlifeless: running testr under pypy doesn't work21:37
lifelessAlex_Gaynor: like, why is testr running under pypy? (It should work, and I'll happily take bugs about it not working)21:37
Alex_Gaynormordred: actually marconi works, this is novaclient, marconi isn't on testr21:37
lifelessAlex_Gaynor: but you don't need to run testr under pypy to run marconi's tests under testr.21:37
mordredAlex_Gaynor: ah. neat21:37
mordredlifeless: tox21:37
Alex_Gaynorlifeless: well, it all uses the python from tox, so it's just easier to have it be the same python21:38
mordredlifeless: is how we switch python impls21:38
lifelessAlex_Gaynor: testr is a meta-runner, the test processes are a separate process.21:38
jeblairclarkb: everything behind the devstack-gate change will use it.  :)21:38
mordredlifeless: also, on most of our boxes, testr won't exist outside of the virtualenv, which needs to be created with the python that is going to run the tests21:38
jeblairwhich is nothing atm.21:39
annegentledid we switch over where the maven plugin is loaded from? I'm seeing  [INFO] Unable to find resource 'org.apache.maven.plugins:maven-clean-plugin:pom:2.3' in repository (21:39
Alex_Gaynorlifeless: I found the problem, this is making me rage a bit, where's the testr repo, I'll send you a patch which should fix it (tl;dr; the only reason it works under cpython is that your `hashattr(__builtins__, 'memoryview')` check is a bit broken)21:41
clarkbannegentle: no, your job should still be using the Maven3 build env in Jenkins and unless you changed your project or one of your upstreams changed will go to the same place21:41
clarkbannegentle: I am not sure what is going on there yet21:41
Alex_Gaynorlifeless: uh oh, do I have to learn how to use bzr?21:41
annegentleclarkb: yea I'm digging deeper too21:41
lifelessAlex_Gaynor: no, you can just file a bug with a patch21:41
Alex_Gaynorlifeless: can I ask the really dumb question "how do I check this out"21:42
mordredAlex_Gaynor: "bzr branch lp:testrepository ; cd testrepository ; <fix> ; bzr commit ; bzr push lp:~alexgaynor/testrepository/fix-the-bug"21:42
lifelessAlex_Gaynor: bzr branch lp:testrepository - but the bug isn't in testrepository.21:42
lifelessAlex_Gaynor: it's in subunit21:42
Alex_Gaynorlifeless: err, right, sorry21:42
Alex_Gaynorlifeless: where's the subunit repo :)21:42
mordredAlex_Gaynor: "bzr branch lp:subunit ; cd subunit ; <fix> ; bzr commit ; bzr push lp:~alexgaynor/subunit/fix-the-bug"21:43
lifelessAlex_Gaynor: so, I need to cast bug to a str before decoding?21:43
Alex_Gaynorlifeless: yup, I'll also fix your hasattr check :) the only reason this doesn't break on cpython is that hasattr check never retuirns true21:43
Alex_Gaynorlifeless: __builtins__ is sometimes a dict on cpython and sometimes a module, here it happens to be a dict, so of course hasaatrr doesn't work21:43
lifelessAlex_Gaynor: oh, interactive vs in a module?21:44
lifelessAlex_Gaynor: don't bother with a patch, I've got the changes made here21:44
marunlifeless: ping21:44
mordredAlex_Gaynor: (actually, fwiw) "bzr branch lp:subunit ; cd subunit ; <fix> ; bzr commit ; bzr push lp:~king_kilr/subunit/fix-the-bug"21:44
Alex_Gaynorlifeless: something like that, you want `import __bujiltin__` hasattr(__builtin__, 'memoryview')21:44
Alex_Gaynorlifeless: except on python3 the module is named builtin (no dunder)21:44
jeblairclarkb, mordred: i can reproduce the slowness by cloning nova 10 times21:45
mordredjeblair: nice!21:45
mordredfrom git.o.o ?21:45
mordredthat's excellent news21:45
lifelessAlex_Gaynor: I have a few reviews to finish, but I'll push up a subunit release today for you21:45
*** pblaho has quit IRC21:45
Alex_Gaynorlifeless: thanks, you rock! (let me know if you want me to do a once over a on patch)21:45
lifelessAlex_Gaynor: I presume on pypy __builtins__ is always an object ?21:45
Alex_Gaynorlifeless: yup21:45
Alex_Gaynorlifeless: see the note there :)21:46
Alex_Gaynorlifeless: __builtin__ (no s) always works21:46
jeblairecxept i'm getting a bit of packet loss21:46
jeblairwhich makes it difficult21:46
jeblairstrace -p 2344521:48
jeblairmordred: ^21:48
marunlifeless: is parameterization of a fixture created with testresources (e.g. VmFixture(tenant_id)) a good idea?  Is reuse possible?21:48
mordredjeblair: on which server?21:49
mordredjeblair: [mordred@git ~]$ sudo strace -p 2344521:49
mordredattach: ptrace(PTRACE_ATTACH, ...): No such process21:49
jeblairtoo slow21:49
*** colinmcnamara has left #openstack-infra21:49
fungiwow. had to hit pgup 58 times to get to the start of the scrollback. you're a chatty bunch21:49
*** dprince has quit IRC21:49
jeblairmordred: sudo screen21:49
jeblairmordred: sudo screen -x21:50
clarkbmordred: you and fungi both :P21:50
mordredI'm in21:50
clarkbme too21:50
clarkbjeblair: what processes are you stracing? are they upload packs?21:51
*** xBsd has joined #openstack-infra21:51
mordreduhm. so compressing the objects over and over doesn't seem like the best choice?21:51
clarkbmordred: well it is just once compression I think that reorts updates21:52
clarkbjeblair: you might want to -f that strace21:52
fungibtw, you know you're in seattle when you encounter a wayward mark atwood on a pike st sidewalk21:53
clarkbfungi: the fog into sunny weather didn't give it away earlier?21:53
*** rfolco has joined #openstack-infra21:53
Alex_Gaynorlifeless: untested, but
Alex_Gaynorlifeless: (if you let me know how to run the tests I can do that :P)21:54
*** mrodden has quit IRC21:54
*** lbragstad has quit IRC21:55
*** dims has quit IRC21:56
jeblairclarkb: great21:56
clarkbwoo git21:57
jeblairclarkb, pleia2: any idea where relevant logs for that are?21:57
clarkbjeblair: that would be git daemon /me looks21:57
lifelessAlex_Gaynor: configure; make check21:57
jeblairclarkb: we use xinetd, right?21:57
clarkbjeblair: we do21:57
* fungi is reviewing 38177 before he catches up on the rest of scrollback, since it could probably stand to get crammed through asap21:57
clarkbjeblair: syslog21:58
*** ftcjeff has quit IRC21:58
marunwho can help me with testresources?21:58
Alex_Gaynorlifeless: this yak stack sucks, now I have to figure out why configure doesn't work :21:59
*** rfolco has quit IRC22:00
lifelessAlex_Gaynor: hey, you're the one digging:) I mean, I appreciate it but since I need to do the release anyhow, I'll be doing full test runs etc22:01
lifelessmarun: I can, mordred can, I think clarkb can.22:01
Alex_Gaynorlifeless: not your fault  I'm deep down, but the levels of abstraction from the original problem is getting a bit too high for my comfort :)22:01
jeblairclarkb: do you think we should switch back off of git.o.o?  or switch to https instead of git:// +22:01
clarkbjeblair: git-daemon[24478]: error: git-upload-pack died of signal 13 a lot of that in /var/log/messages22:01
clarkbjeblair: worth a shot of switching to https since the mechanism is different22:02
clarkbjeblair: and in theory is very similar to what we have on review.o.o. The major difference being the version of git aiui22:02
jeblairclarkb: aren't we running an old version of git here?22:03
clarkbjeblair: yes, it is slightly older22:03
marunlifeless: is it possible to parameterize fixture creation (e.g. NeutronVM1(tenant_id)) and have the parameter be a factor in reuse?22:03
clarkbjeblair: but still 1.7.122:03
pleia2yeah, the default from centos is a bit older22:03
clarkbjeblair: which is new enough for smart http22:04
*** dmakogon__ has quit IRC22:04
marunlifeless: I'm not clear on how the parameter could/would be taken into account by testresources22:04
clarkbjeblair: I believe smart http made it into git somewhere in 1.6.X22:05
jeblairclarkb: oh, the signal 13's are probably me22:06
jeblairaborting git clones22:06
lifelessmarun: that looks like you want a dependency resource representing the tenant22:06
marun lifeless: ah, so it would have to be defined at the class level22:06
marunlifeless: that's fine.  i was figuring on having to do that for other resources anyway.22:07
clarkbjeblair: nice :)22:07
clarkbjeblair: maybe we should let git:// sitfor a little while longer?22:07
*** burt has quit IRC22:09
*** qs202 has joined #openstack-infra22:09
clarkbjeblair: python26 jobs for nova are spending a minute and 20 seconds in g-g-p which is much quicker than before22:10
clarkbso keeping git:// if possible is a good thing22:11
jeblairclarkb: i'm really confused22:11
*** dims has joined #openstack-infra22:11
jeblairclarkb: those are the log entries from precise23 around the time of that failed test22:11
BobBall_Awaymordred: ping22:11
clarkbjeblair: weird that it is all one host22:12
jeblairclarkb: yeah, and none of those are for oslo.config22:12
mordredBobBall_Away: sup?22:12
clarkbjeblair: network trouble maybe? if git on the precise23 side couldn't tcp handshake with the git daemon things would derp?22:12
mordredclarkb: I was just about to say that22:12
jeblairclarkb: okay, but why did it try to update a bunch of other repos?22:13
BobBall_Awaymordred: VIRTUAL_ENV patch - was going to upload a new version - can be multiple site-packages (one deeply nested) - will $VIRTUAL_ENV/lib/python*/site-packages always work?22:13
clarkbjeblair: perhaps that slave started grabbing jobs really quickly as things failed? /me looks at build history22:13
BobBall_Awaymordred: this is the cause of the current failure in devstack because the horizon sed fails due to a newline caused by multiple site-packages22:14
clarkb doesn't look like that is the case22:14
mordredBobBall_Away: uhm - wow. really there are multiple site packages?22:14
BobBall_Awaysure - but not intentionally...22:14
BobBall_Awaya side effect of a dodgy package ;)22:14
BobBall_Awayand a test suite ;)22:14
clarkbjeblair: actually it ran a glance client test, then oslo.config then neutron22:14
mordredBobBall_Away: what is it we're doing wiht the dir? it was a chown or something, right?22:15
jeblairclarkb: ok, so that neutron error seems to be from the test after the one we're interested in.22:15
* mordred goes to look at patch22:15
clarkbthen glanceclient then cinder22:15
clarkbjeblair: yes22:15
*** qs202 has quit IRC22:15
jeblairclarkb: the previous one was glance22:15
jeblairclarkb: (in syslog)22:15
BobBall_Awaymordred: to allow horizon to run in a venv we need to set HORIZON_VENV_PATH22:15
mordredBobBall_Away: AH - I remember this now22:15
jeblairclarkb: so there are no logs from that.22:16
jeblairclarkb: xinetd connection limit?22:16
BobBall_Awaysorry - set python-path=.../site-packages22:16
clarkbjeblair: yeah I am thinking xinetd may be the culprit22:16
mordredBobBall_Away: can you show me find $VIRTUAL_ENV -type d -name site-packages output?22:16
fungi38177 lgtm, assuming we've decided a) puppet vcsrepo should use https:// instead of git:// and clone urls should consistently omit .git suffixes. i'm ambivalent on both counts. also i guess replacing the github-org parameter in jjb config will be a separate change?22:16
clarkbjeblair: unable to fork off git daemons quick enough22:16
mordredBobBall_Away: (just so I can be smarter)22:16
BobBall_Awaymordred: two lines: /opt/stack/new/.venv/lib/python2.7/site-packages22:16
mordredo m g22:17
mordredyou're kidding me22:17
BobBall_Awaymordred: although it works in my env, that's what the gate sees22:17
BobBall_AwayI wish.22:17
*** mriedem has quit IRC22:17
BobBall_Awaycould restrict the depth with find easily, but I just thought that we might be able to rely on .venv/lib/python*/site-packages22:17
*** qs201 has joined #openstack-infra22:18
BobBall_Away(btw - if you're crafting a new patch, can you also fix lib/nova to execute get_python_bin_prefix rather than using that string? ;) )22:18
mordredBobBall_Away: yeah - -mindepth 3 -maxdepth 3 should work22:18
clarkbjeblair: yeah looks like our xinetd.conf sets some conservative defaults.22:18
mordredBobBall_Away: but also the * should work22:18
mordredBobBall_Away: and go for it22:18
clarkbjeblair: we can just update those numbers for git daemon22:18
BobBall_AwayI'll upload a patch22:19
mordredBobBall_Away: I'm not crafting one :)22:19
*** xBsd has quit IRC22:19
mordredBobBall_Away: oh - and HAHAHA on the get_python_bin_prefix22:19
BobBall_AwayOne will be there before you can type shakespeare's hamlet backwards22:19
*** qs201 has quit IRC22:19
jeblairclarkb: ok; i'm going to see if i can trigger the error; you want to look up some new values?22:19
clarkbjeblair: yup reading man pages now22:19
*** datsun180b has quit IRC22:20
jeblairclarkb: i think i ran into per_source= 1022:20
jeblairclarkb: got a connection reset by peer if i tried to clone more than 10 repos at once22:21
jeblairfrom a single host22:21
clarkbyup that is one of them22:21
jeblairclarkb: that one maybe we ought to leave alone?22:21
jeblairwe don't actually do that anywhere22:21
*** pabelanger has joined #openstack-infra22:23
*** dina_belova has joined #openstack-infra22:24
jeblairclarkb: yeah, i can get the same error if i do > 50 clones from different hosts22:24
*** prad_ has quit IRC22:24
clarkbjeblair: yup and then there is a 10 second timeout22:24
*** changbl has quit IRC22:24
*** jhesketh has joined #openstack-infra22:25
clarkbjeblair: the settings is 'cps' trying to figure out if we need to give it numeric values or if we can say UNLIMITED 022:25
clarkbor I can try bumping to 16k or something high like that22:26
*** rcleere has quit IRC22:28
jeblairclarkb: why don't we say something like 500 10 ?22:28
clarkbjeblair: ok, I think I want to drop the timeout slightly but we can try that22:28
jeblairclarkb: even with a very large gate reset, we shouldn't actually hit that.22:29
*** dina_belova has quit IRC22:29
jeblairi wish we had a real socket queue here...22:29
*** jerryz has quit IRC22:30
jeblairbecause the server does not, at the moment, actually perform that well with a lot of instances running.22:30
jeblairbut i think that's what we'd need to change in order to avoid the error we saw22:31
anteayamordred: I have sent you a pm22:31
anteayaif you are around22:31
openstackgerritClark Boylan proposed a change to openstack-infra/config: Relax cgit xinetd ACLs.
clarkbjeblair: ^22:31
jeblairclarkb: you sure about unlimited?22:32
jeblairis there another way to run a git daemon other than xinetd?22:32
*** eharney has quit IRC22:32
jeblairclarkb: because with the current tuning, the server got rather heavily loaded earlier22:33
*** ryanpetrello has quit IRC22:33
*** dina_belova has joined #openstack-infra22:34
lifelesshey question on the pypi.o.o mirror22:37
lifelessis there a mirror at / or just at /openstack/ ?22:37
*** UtahDave has quit IRC22:37
lifeless for context22:37
jeblairlifeless: the openstack/ mirror is at /openstack; other mirrors may exist later (eg openstack-infra)22:37
lifelessjeblair: ok, but they would be siblings, yeah ?22:37
jeblairlifeless: yes, in fact there is a stub there:
lifelessjeblair: right, it /looks/ like a mirror but it's empty22:38
clarkbjeblair: I am not sure about UNLIMITEd and yes I think we can run it as a normal service if we add an init script (this is how ubuntu's packaging does it)22:38
lifelesswe've got something accidentally using / and breaking22:38
*** chuckieb|2 has joined #openstack-infra22:39
jeblairlifeless: i believe that patch dtrt22:39
*** dina_belova has quit IRC22:39
jeblairclarkb: i doubt running it as its own servic would help either, i don't see anything about socket queue depth in the params22:40
clarkbjeblair: --max-connections=32 is the default for git-daemon22:41
*** chuckieb has quit IRC22:42
clarkbjeblair: I can change the UNLIMITED to 1024 to give us some room to have more connections running concurrently without going overboard22:42
jeblairclarkb: that's just it, i'm not sure 50 is wrong22:42
jeblairclarkb: the server can not handle 50 connections very well22:43
clarkbI see22:43
jeblairclarkb: i think we should switch the tests to https:// and i think someone needs to do some serious thinking and testing about git://22:43
clarkbjeblair: wfm22:43
jeblairclarkb: so we don't just dos ourselves (or invite someone else to do it for us)22:44
clarkbjeblair: I think our jenkins setup if a giant dos machine :)22:46
clarkbgrenade is passing again22:47
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Use in jobs
*** pcm_ has quit IRC22:49
openstackgerritJames E. Blair proposed a change to openstack-infra/devstack-gate: Use as origin
jeblairi think az1 has started responding22:49
jeblairi also think nodepool is leaking nodes, so i'll need to work on that for a bit.22:51
clarkbjeblair: ok22:51
mordredclarkb, jeblair: both the above look great - should I aprv or wait?22:52
clarkbmordred: I think you can approve22:53
mordredclarkb: jeblair got one of them already22:53
jeblairmordred: i could go either way on the other; no rush22:53
jeblairthe one i approved was to change git:// to https, the other was still using review https22:54
mordredclarkb: do we need to abandon this one now?
jeblairmordred, clarkb: i'm seeing more rate limit errors from az1 than i would expect (and more than we're getting for the other azs)22:55
jeblairmordred, clarkb: any chance something changed there?22:55
mordredjeblair: it's possible that they ratched down rate limit to get themselves out of the hole?22:56
mordredjeblair: but I do not know22:56
mordredlemme pop onto IM and see22:56
*** weshay has joined #openstack-infra22:57
lifelessjeblair: mordred: do you want a review from me on the dib using d-g patch? If it's going to be moved forward this week I'll happily do so.22:58
pleia2tbh I didn't put much thought into git daemon while testing - I assumed it was a proven robust thing :\22:58
*** rnirmal has quit IRC22:58
clarkbpleia2: I think it is robust, the problem here is that it is able to quickly dos the server.22:59
pleia2heh, right22:59
*** emagana has quit IRC22:59
*** nati_ueno has quit IRC22:59
clarkbjeblair: pleia2: I am not entirely concinced that apache doens't DOS us in the same manner though22:59
clarkbthe load on review.o.o was ridiculous and almost purely from apache22:59
mordredclarkb: I think eventually we're going to have to horizontally scale those servers23:00
clarkbmordred: I agree23:00
jeblairclarkb: apache has a socket queue, with appropriate tuning you can get reasonable load without immediate rejecting new connections23:00
mordredjeblair: ++23:00
notmynameis saying that the pep8 command couldn't be run, or that the pep8 check failed?23:00
*** nati_ueno has joined #openstack-infra23:00
pleia2scaling git servers will be easy enough I think23:00
mordrednotmyname: command failed23:00
jeblairand yes, we should scale it horizontally, but we'd be in a better position to do so once we get one of them the way we want.23:00
mordrednotmyname: ./ H802  git commit title ('In the object replicator, make the length of certain lines logged configurable') should be under 50 chars23:00
clarkbjeblair: ++23:00
notmynamemordred: ok, I saw that, but "InvocationError" sure sounds like something else. thanks.23:01
clarkbmordred: let it be known that using flake8/hacking to enforce commit message stuff is broken with the way people use git23:01
Alex_Gaynorhmm, so zuul seems to have a 92 event queue, why are those being slow to process23:01
mordrednotmyname: agree23:01
pleia2I need to run out for a bit, will be back in a few hours23:01
mordredclarkb: agree23:01
mordredclarkb: I'm on board with re-thinking how we deal with that23:01
mordredI think the sequencing is wrong23:02
clarkbAlex_Gaynor: I think all of our slaves are being used23:02
clarkbAlex_Gaynor: the gate queue gets priority so having a large number of changes there can starve the check queue23:02
jeblairclarkb: 'event queue'23:02
Alex_Gaynorclarkb: I thought that count was counting things before they even went into gearman?23:03
* notmyname would be in favor of renaming the "pep8" check to "formatting" or something more specific to what is actually being tested23:03
clarkbjeblair: oh wrong queue23:03
mordrednotmyname: I aslo agree with that23:03
jeblairnotmyname: i would accept that patch.  it's your chance to get involved in infra!23:03
mordrednotmyname: in fact, it'sa low-priority TDL for me - but I think given the number of places where tox -epep8 is referenced, it might take a little while to phase in23:03
mordredooh. I like jeblair's suggestion better23:03
notmynamemordred: clearly, sec23:03
clarkbAlex_Gaynor: jeblair: the event queue relies on git in its processing. Git is currently being DDoSed by our lovely slaves.23:04
Alex_Gaynorah, fun times :)23:04
mordrednotmyname: yeah - if we changed the name of the job in jenkins/zuul, that would be an excellent start23:04
jeblairclarkb: yeah, that's probably it.23:04
mordredeventually, I'd love to rename the tox environments too - but I don't think that's as visible23:04
jeblairmordred: i think they should go hand-in-hand.  i'd hate to have that half-finished.23:05
mordredjeblair: yeah. although I think it's going to be a multi-phase thing23:05
jeblairmordred: yep, but no reason to approve the first phase before the second is at least staged.23:05
mordredjeblair: we're going to need to be able to see which env the project has (pep8 or formatting), then start changing all the names of all of the tox envs23:06
jeblairmordred: yep23:06
clarkbjeblair: pleia2 mordred if we put haproxy in front of xinetd/git-daemon we can use its socket queuing and that can evolve into using mutliple hosts to serve teh content23:06
mordrednow THERE is a script folks will LOVE23:06
* mordred submits 400 patches to gerrit in the same 15 minute span ...23:06
clarkbjeblair: pleia2 mordred does that seem like a sane next step as we want to scale out horizontally?23:06
notmynamewhile we're talking of test failures, can anyone tell me why failed?23:06
mordrednotmyname: yes, although it has since been fixed23:07
mordrednotmyname: a backwards incompatible change to glanceclient got merged23:07
notmynamemordred: since 12:40 pm todya?23:07
mordredand we have not yet merged the change to start testing backwards incompatible changes23:07
jeblairclarkb: do we want to do our own load balancing, use a lbaas, or dns round robin?23:07
mordrednotmyname: yes23:07
notmynamemordred: how did that happen? /me doesnt' really want to know, but seems like the think jenkins is supposed to prevent23:07
mordrednotmyname: we're ALMOST ready with testing that trunk clients work with stable releases ...23:07
clarkbjeblair: I am wary of lbaas because everyone uses a different api iirc. mordred would know better23:08
mordredclarkb: I would want to use lbaas23:08
*** mgagne has quit IRC23:08
clarkbjeblair: dns round robin would be simplest but would not help with connection queueing23:08
mordredclarkb: rax and hp are actually using the same API23:08
clarkbmordred: but it is different than what openstack/neutron are doing right?23:08
*** ^d has quit IRC23:08
jeblairmordred: layer 7 or 2/3?23:08
mordredit is - but the plan (from both, aiui) is to move to openstack/neutron23:09
mordredjeblair: it's haproxy aaS23:09
jeblairclarkb: then perhaps we should start looking into that now23:10
clarkbjeblair: ya I have pulled up the rax docs23:10
mordredclarkb: I believe python-libraclient _should_ be able to work with it, since I think they implemented the same api23:11
mordredbut Shrews would know more23:11
Alex_Gaynorclarkb, mordred: If ya'll have a second is fixed up now23:12
mordredjeblair, clarkb: also, given the nature of these - would it make any sense to have a set in rax and a set in hp?23:12
*** eharney has joined #openstack-infra23:12
clarkbmordred: then you have to do "global" load balancing of some sort (which may be as simple as a dns round robin)23:13
clarkbor hot standby23:13
mordredclarkb: yah. or also have global dns round robin _plus_ a local cloud affinity23:13
mordredso that the hpcloud slaves go to hpcloud git.o.o for their stuff or something? I dunno, might be too complex -and network doesn't seem to be the issue23:14
*** weshay has quit IRC23:14
jeblairmordred, clarkb: let's just try to get one system where we don't actually dos ourselves first.23:14
mordredjeblair: ++23:14
Alex_GaynorIt seems to be a low of nature that any sufficiently interesting distributed system will DDoS itself at some point23:15
jeblairmordred, clarkb: i've manually tuned some nodepool values to slow it down a bit in az123:15
mordredjeblair: I love that Alex_Gaynor's patch is actually one of the things we were hoping we'd see from stackforge- someone adds a feature for a stackforge project that quite clearly could wind up being used by openstack projects too23:15
Alex_Gaynormordred: what'd I do?23:16
mordredadding the pypy support23:16
Alex_Gaynoroh, yeah :)23:16
Alex_GaynorOnce this lands I'll probably add it to a few clients as well23:16
jeblairi have to run now23:16
openstackgerritA change was merged to openstack-infra/config: Use in jobs
clarkbmordred: I am not seeing anything about connection queueing in the rax api23:17
clarkbmordred: there are throttles but it is unclear if throttling with NACK or queue23:18
clarkbbut it looks relatively straightforward to set one of these up, and we can do so without impacting git.o.o directly.23:18
* clarkb goes to find pcrews and see if he knows what the throttling will do23:19
clarkbmordred: jeblair
*** sandywalsh has quit IRC23:34
clarkbthat is per source throttling and I don't read it as providing queuing23:34
clarkbsomething like is what we would want in front of git-daemon23:34
*** dina_belova has joined #openstack-infra23:35
mordredclarkb: I believe I agree with you23:35
mordredand also that that is what we'd want23:36
*** changbl has joined #openstack-infra23:36
mordredmaybe just running a local haproxy for connection throttling is the thing we want23:36
mordredotoh, maybe http/apache just solves it for us23:36
clarkbmordred: I think http/apache solves it for git over http. It may not be worth providing git:// as a result23:37
*** tjones has left #openstack-infra23:37
clarkbthough puppetlabs seems to provide a fairly robust haproxy module23:38
*** dina_belova has quit IRC23:40
*** ^d has joined #openstack-infra23:41
*** ^d has joined #openstack-infra23:41
*** eharney has quit IRC23:41
SpamapShm, Is it possible a new quantumclient released today with an api change that might be breaking Heat's unit tests?23:43
SpamapSoh wait23:43
SpamapSI thought we already finished this? Argh.23:43
SpamapSstale pyc's .. doh23:45
* SpamapS invites everyone to ignore the previous 6 lines.23:45
mordredSpamapS: I was already ignoring them23:46
SpamapSmordred: again, but this time with feeling23:46
* mordred ignores the SHIT out of the previous 6 lines23:46
SpamapSthank you23:47
*** sandywalsh has joined #openstack-infra23:49
*** mrodden has joined #openstack-infra23:50
jeblairmordred, clarkb: it may be time for us to chat with or fedora infra people about how to host git at scale.23:54
mordredjeblair: do they host systems that want to clone this much?23:54
mordredjeblair: and yes23:55
mordredI believe it probably is23:55
clarkbjlk: ^23:57
jlkoh hai23:59
jlkI setup Fedora's git server a while back23:59

Generated by 2.14.0 by Marius Gedminas - find it at!