Monday, 2013-07-29

*** clarkb has joined #openstack-infra00:02
clarkband back00:03
*** bnemec has quit IRC00:05
*** bnemec has joined #openstack-infra00:05
*** CaptTofu has quit IRC00:09
*** wu_wenxiang has quit IRC00:12
stevebakermordred: hey, your unfulfilled heat use cases: would you want a single template to launch resources on more than one cloud?00:12
*** sdake has joined #openstack-infra00:13
*** sdake has quit IRC00:13
*** sdake has joined #openstack-infra00:13
clarkbstevebaker: I think that would be ideal as we run stuff across 2 clouds today00:13
lifelessstevebaker: FSVO template00:15
lifelessstevebaker: 'one stack'00:15
stevebakeryes, i should stop saying that00:16
stevebakerclarkb: that could be done with 2 stacks (launched with different credentials). But if resources in one stack need to reference resources in the other, that might be better represented in a single stack00:17
clarkbfungi: you'll be happy to know that my new IRC client is reporting timestamps to me in UTC00:18
clarkbstevebaker: one use case I would like to use is cross provider clusters. jenkins masters, elasticsearch nodes, and so on. We can probably get away with not coordinating across stack barriers but being able to do so would be nice00:19
lifelessyou need floating ips and other manual bits to do cross-heat apis00:20
stevebakerIts all on the roadmap, but the next thing to implement will probably be one heat that can launch one stack into one of a choice of clouds00:22
*** dstufft has quit IRC00:37
*** dstufft has joined #openstack-infra00:38
*** CaptTofu has joined #openstack-infra00:38
*** bnemec has quit IRC00:43
*** bnemec has joined #openstack-infra00:43
*** dims has joined #openstack-infra00:59
*** CaptTofu has quit IRC01:20
*** michchap_ has joined #openstack-infra01:30
*** michchap has quit IRC01:30
*** sdake has quit IRC01:33
*** sdake has joined #openstack-infra01:36
*** sdake has quit IRC01:36
*** sdake has joined #openstack-infra01:36
*** reed has quit IRC01:39
*** dims has quit IRC02:08
*** dguitarbite has joined #openstack-infra02:18
*** woodspa has joined #openstack-infra02:19
*** xchu has joined #openstack-infra02:21
*** woodspa has quit IRC02:27
mordredstevebaker: yes02:34
mordredstevebaker: I mean, it could be managed with multiple stacks02:35
mordredstevebaker: but for the underlying usecase - think of the openstack slave pool02:35
*** yaguang has joined #openstack-infra02:35
mordredstevebaker: we have 2 (hopefully soon 3) clouds each with a couple of different zones and or regions02:35
mordredstevebaker: we want an elastic pool of build resources. ultimately, we want a build slave from any of them02:36
mordredbut for performance reasons, we want to keep a certain number in each region02:36
mordredor zone02:36
mordredso we want to set both region/zone min and maxes, as well as a global thing we want to occur02:36
mordredand all of them are really part of one system02:37
mordredother than a set of scripts that juggle a database of credentials that we have, it's really one application02:37
mordredwith slaves from az3 at hp cloud publishing things to machines in dallas at Rackspace02:37
mordredstevebaker: I'm expecting that the day that heat can do that in rich manner will be a very exciting day02:38
stevebakerright, I'll be keeping that use case in mind02:42
mordredlifeless: 'floating ips and other manual bits' -- can you expand on that?02:44
*** cody-somerville has quit IRC02:45
mordredstevebaker: cool. let me know if it's ever useful for me to babble more02:45
*** mriedem has quit IRC02:50
stevebakermordred: to help with this, what favors do I perform for whom to waive charges on my hpcloud account?02:52
mordredstevebaker: send me an email with your tenant id and username and I'll get it done02:52
clarkbmordred: did you comp our new infra account?02:53
mordredclarkb: funny story - that right there just reminded me I need to do that...02:53
mordredstevebaker: actually: "We need the username and email address for each account along with the domain ID – the domain ID is the Account Number shown on Console->Account->MyAccount."02:54
clarkbmordred: do we know why there are 3 different numbers for each account?02:54
mordredclarkb: there are MANY questions I tend to try to not ask02:55
clarkbhow hard is ot to SELECT * FROM accout_table WHERE email = foo@bar AND uname = foo;02:55
mordredclarkb: you're assuming there is a relational database with user accounts02:57
*** bnemec has quit IRC02:57
*** bnemec has joined #openstack-infra02:57
*** jjmb has quit IRC03:03
*** blamar has joined #openstack-infra03:14
*** michchap_ has quit IRC03:22
*** michchap has joined #openstack-infra03:23
*** michchap_ has joined #openstack-infra03:26
*** michchap has quit IRC03:26
*** nati_ueno has quit IRC03:34
*** vipul is now known as vipul-away03:37
openstackgerritSteve Baker proposed a change to openstack-infra/config: Create new repo to host legacy heat-cfn client.
stevebakermordred: ^^ we've moved some old client code out of the heat core repo. could we host it on stackforge? It will be maintained but is verging on abandonware03:39
mordredstevebaker: sure thing03:39
mordredstevebaker: although that patch hosts it in the openstack/ org03:40
mordredwhich is also fine03:40
mordredsince it is a part of heat03:40
stevebakerok, there might be other parts of that patch that need switching to stackforge03:43
*** krtaylor has joined #openstack-infra03:44
*** jjmb has joined #openstack-infra03:51
*** nati_ueno has joined #openstack-infra03:51
*** jjmb has quit IRC03:52
*** yaguang has quit IRC03:54
*** bnemec has quit IRC04:01
*** bnemec has joined #openstack-infra04:01
*** yaguang has joined #openstack-infra04:08
*** yaguang has quit IRC04:18
*** michchap_ has quit IRC04:23
*** michchap has joined #openstack-infra04:24
lifelessmordred: if you have two heat stacks managing one cluster04:29
lifelessmordred: you have to pass in things like 'the endpoint for puppet is <here>' where <here> is: always reachable, stable even if the instance is redeployed04:30
*** michchap has quit IRC04:30
*** michchap_ has joined #openstack-infra04:30
mordredlifeless: yes. that makes total sense - and is one of the reaons I'd _prefer_ one stack04:31
mordredlifeless: but - that said - we actually keep our long-lived things that matter in a completely different account anyway04:32
*** SergeyLukjanov has joined #openstack-infra04:32
mordredso I imagine that we'll get to "manage the pool and tell it about things that arent' in the pool" long before we get to "manage long running services"04:32
lifelessclarkb: mordred: btw HP support have clsoed the ticket : you should have glance upload in us-east04:37
mordredlifeless: cool. I mean, that'll be in the beta cloud - but exciting none the less04:37
*** yaguang has joined #openstack-infra04:37
*** nati_ueno has quit IRC04:46
*** w_ has joined #openstack-infra04:50
*** olaph has quit IRC04:53
*** yaguang has quit IRC04:58
*** yaguang has joined #openstack-infra04:59
*** Ryan_Lane has quit IRC04:59
*** Ryan_Lane has joined #openstack-infra05:00
*** xchu has quit IRC05:01
*** jhesketh has quit IRC05:05
*** jjmb has joined #openstack-infra05:14
*** emagana has joined #openstack-infra05:15
*** amotoki has joined #openstack-infra05:16
*** nicedice has quit IRC05:25
*** xchu has joined #openstack-infra05:49
*** yaguang has quit IRC05:49
*** yaguang has joined #openstack-infra06:07
*** afazekas is now known as afazekas__06:07
*** SergeyLukjanov has quit IRC06:09
*** dguitarbite has quit IRC06:21
*** jjmb has quit IRC06:21
*** sileht has quit IRC06:32
*** Ryan_Lane has quit IRC06:32
*** sileht has joined #openstack-infra06:33
GheRiveropabelanger: you got it!06:35
*** michchap_ has quit IRC06:35
*** michchap has joined #openstack-infra06:36
*** emagana has quit IRC06:37
*** xchu has quit IRC06:49
*** GheRivero has quit IRC06:50
*** jjmb has joined #openstack-infra06:54
*** ndipanov has joined #openstack-infra06:58
*** xchu has joined #openstack-infra07:02
*** afazekas has joined #openstack-infra07:04
*** psedlak has joined #openstack-infra07:20
*** odyssey4me has joined #openstack-infra07:28
*** odyssey4me has quit IRC07:33
*** SergeyLukjanov has joined #openstack-infra07:36
*** GheRivero has joined #openstack-infra07:37
*** sdake_ has quit IRC07:39
*** sileht has quit IRC07:39
*** odyssey4me has joined #openstack-infra07:42
*** sileht has joined #openstack-infra07:42
*** dina_belova has joined #openstack-infra07:44
*** jpich has joined #openstack-infra07:48
*** hashar has joined #openstack-infra07:49
openstackgerritA change was merged to openstack/requirements: Added cffi as a requirement.
openstackgerritA change was merged to openstack/requirements: update global requires on clients for keystone fix
*** rustlebee has quit IRC08:01
*** russellb has joined #openstack-infra08:08
*** dguitarbite has joined #openstack-infra08:11
*** derekh has joined #openstack-infra08:12
*** BobBall_Away is now known as BobBall08:28
*** dguitarbite has quit IRC08:36
*** odyssey4me has quit IRC08:37
*** dguitarbite has joined #openstack-infra08:40
*** dguitarbite has quit IRC08:47
*** sdake_ has joined #openstack-infra08:48
*** SergeyLukjanov has quit IRC08:48
*** koolhead17 has quit IRC08:52
*** odyssey4me has joined #openstack-infra08:57
*** dina_belova has quit IRC09:06
*** HueJorgan has joined #openstack-infra09:28
*** jjmb has quit IRC09:28
*** isviridov has joined #openstack-infra09:42
*** koolhead17 has joined #openstack-infra09:51
*** isviridov has quit IRC09:55
*** jjmb has joined #openstack-infra09:55
*** jjmb has quit IRC09:57
*** rpodolyaka has joined #openstack-infra10:05
*** jjmb has joined #openstack-infra10:13
*** jjmb has quit IRC10:13
*** dina_belova has joined #openstack-infra10:15
*** SergeyLukjanov has joined #openstack-infra10:16
*** isviridov has joined #openstack-infra10:18
*** hashar has quit IRC10:39
*** hashar has joined #openstack-infra10:39
*** yaguang has quit IRC10:40
*** dina_belova has quit IRC10:44
*** jjmb has joined #openstack-infra10:47
*** isviridov has quit IRC10:47
*** dina_belova has joined #openstack-infra10:48
*** mjfork has joined #openstack-infra10:49
*** HueJorgan has quit IRC10:49
*** dims has joined #openstack-infra10:57
*** dina_belova has quit IRC10:57
*** psedlak_ has joined #openstack-infra10:58
*** dina_belova has joined #openstack-infra10:59
*** psedlak has quit IRC11:01
*** afazekas has quit IRC11:04
*** mjfork has quit IRC11:07
*** derekh has quit IRC11:22
*** afazekas has joined #openstack-infra11:24
*** derekh has joined #openstack-infra11:29
*** derekh has quit IRC11:32
*** lcestari has joined #openstack-infra11:33
*** dina_belova has quit IRC11:36
*** HueJorgan has joined #openstack-infra11:40
*** dina_belova has joined #openstack-infra11:42
*** dina_belova has quit IRC11:50
*** dkranz has quit IRC11:54
*** dina_belova has joined #openstack-infra11:56
*** xchu has quit IRC11:57
*** psedlak__ has joined #openstack-infra12:00
*** ndipanov has quit IRC12:04
*** psedlak_ has quit IRC12:04
*** sdake_ has quit IRC12:07
*** dina_belova has quit IRC12:14
*** dkliban has quit IRC12:17
*** ndipanov has joined #openstack-infra12:19
*** sdake_ has joined #openstack-infra12:19
*** woodspa has joined #openstack-infra12:22
*** rfolco has joined #openstack-infra12:24
*** beagles has joined #openstack-infra12:24
*** ndipanov has quit IRC12:26
*** CaptTofu has joined #openstack-infra12:29
*** dina_belova has joined #openstack-infra12:32
*** leifmadsen has joined #openstack-infra12:34
*** mriedem has joined #openstack-infra12:35
*** cppcabrera has joined #openstack-infra12:46
*** w_ is now known as olaph12:46
*** HueJorgan has quit IRC12:47
*** adalbas has joined #openstack-infra12:47
*** acfleury has joined #openstack-infra12:48
*** dkranz has joined #openstack-infra12:50
*** anteaya has joined #openstack-infra12:59
*** amotoki has quit IRC13:10
*** cppcabrera has left #openstack-infra13:12
*** HueJorgan has joined #openstack-infra13:14
*** CaptTofu has quit IRC13:16
*** rahmu has quit IRC13:17
*** koobs has quit IRC13:18
*** rahmu has joined #openstack-infra13:18
*** afazekas has quit IRC13:21
*** acfleury has quit IRC13:21
*** koobs has joined #openstack-infra13:21
*** jpeeler has joined #openstack-infra13:24
pabelangerGheRivero, thanks13:27
*** krtaylor has quit IRC13:28
*** pabelanger has quit IRC13:30
*** afazekas has joined #openstack-infra13:36
*** akscram has quit IRC13:39
*** afazekas has quit IRC13:39
*** afazekas has joined #openstack-infra13:39
*** dkliban has joined #openstack-infra13:45
*** jjmb has quit IRC13:45
*** dina_belova has quit IRC13:47
*** _TheDodd_ has joined #openstack-infra13:50
*** dina_belova has joined #openstack-infra13:52
*** sandywalsh has joined #openstack-infra13:57
*** pabelanger has joined #openstack-infra14:00
*** michchap has quit IRC14:01
*** michchap has joined #openstack-infra14:02
*** blamar has quit IRC14:02
*** blamar has joined #openstack-infra14:02
*** vijendar has joined #openstack-infra14:03
*** prad_ has joined #openstack-infra14:04
_TheDodd_clarkb, I've put together the etherpad that you requested last week. ( )14:06
_TheDodd_clarkb, let me know if there is anything else that you would like me to add or clarify.14:07
*** pentameter has joined #openstack-infra14:08
*** dina_belova has quit IRC14:10
anteaya_TheDodd_: well done, that looks awesome14:16
anteayaone question14:16
_TheDodd_Thanks, anteaya!14:16
*** krtaylor has joined #openstack-infra14:16
anteayawhat happens if someone continues to use "bug #<bug_number> in their commit?14:17
_TheDodd_It will still work,14:17
anteayacould you identify this case in the etherpad message?14:17
_TheDodd_It will default to the 'Closes-Bug' functionality.14:17
anteayajust to close that door14:17
anteayaah okay that would be handy information to share, I think14:18
_TheDodd_If an unknown prefix is supplied, or some funky header is used, it will not close the bug, however.14:18
anteayaI was missing the closing double quote14:18
_TheDodd_Indeed. I will add it.14:18
anteayagreat thanks14:18
Alex_GaynorDo all the jenkins slaves have libffi-dev installed, or only some of them?14:19
_TheDodd_lol ... late correction on that double quote :)14:19
*** HueJorgan has left #openstack-infra14:19
anteaya_TheDodd_: yeah, pretty late14:19
anteayaAlex_Gaynor: I haven't seen activity from anyone yet who would have the answer to your question14:20
anteayafungi should be around soon14:20
Alex_Gaynoranteaya: ok, /me waits patiently14:20
fungi_TheDodd_: probably some of them may be stuck puppet-wise. i'll check them here momentarily14:21
fungier, that was for Alex_Gaynor, sorry14:21
Alex_Gaynorfungi: stuck meaning they didn't install necessary packages for some eason?14:22
fungiAlex_Gaynor: yeah, puppet installs packages on the slaves for us, but sometimes gets stuck either on a remote git operation or (for some reason) restarting cgroups management daemons14:23
fungiso some slaves may indeed lack that package14:23
fungibut should be cleared up here in a bit once i churn through them14:23
_TheDodd_fungi, is there an actionable item here for me?14:24
*** michchap has quit IRC14:24
_TheDodd_In regards to puppet, that is.14:25
Alex_Gaynorfungi: awesome, thanks14:25
*** sdake has quit IRC14:25
*** sdake has joined #openstack-infra14:26
*** sdake has quit IRC14:26
*** sdake has joined #openstack-infra14:26
fungi_TheDodd_: nope, i merely misread who i was replying to in the channel14:26
*** burt has joined #openstack-infra14:28
_TheDodd_Copy that.14:28
*** markmcclain has joined #openstack-infra14:30
*** adalbas has quit IRC14:31
jeblairi can tell the openstack list migration is complete as i now have several hundred messages in my inbox.  :|14:34
* jeblair updates sieve filters14:34
mordredjeblair: ++14:36
*** jergerber has joined #openstack-infra14:38
fungijeblair: i'm thinking i should just >/dev/null the ones still spewing from lp14:38
mordredfungi: I am going to do that14:39
fungisince i updated my filters not to account for it, thinking posting would be shut down there, they all began landing in my inbox over the weekend. and there were surprisingly many14:40
dhellmann_mordred, sdague : I'm catching up on the grenade thing. Was it caused by the jsonschema version downgrade? If so, is that a "good" thing, or would it be better if stevedore loaded plugins without enforcing their requirements?14:41
*** dhellmann_ is now known as dhellmann14:41
fungior disappointingly many, perhaps14:41
sdaguedhellmann_: I think it was the downgrade. Honestly, I need to fully postmortem14:41
fungiAlex_Gaynor: cleaned up. if your job ran on precise2, precise4 or centos6-12 then puppet hadn't updated those since the 23rd or 24th14:42
dhellmannsdague: I'm torn on whether the plugin import should be worried about dependencies. It seems like a mistake to let the installation downgrade like that, but at the same time the code may work under that configuration and import time seems a weird place to be doing the check.14:42
Alex_Gaynorfungi: I'm honestly not sure where the libffi-dev packages was configured to be installed, is there; is there a way to query where it's supposed to be installed?14:43
*** adalbas has joined #openstack-infra14:43
fungiAlex_Gaynor: probably on all of them? i'll check the puppet manifest14:43
sdaguejeblair: - the fact that it has no lines on the right drives me a little nuts :)14:44
*** datsun180b has joined #openstack-infra14:44
fungiAlex_Gaynor: all the precise unit test slaves should have libffi-dev and all the centos6 slaves libffi-devel14:45
jeblairsdague: oh! interesting.  so i decided to be cutting edge and i used the <progress> element in HTML5.14:46
*** avtar has joined #openstack-infra14:46
jeblairsdague: without any styling, so it's browser-default drawing14:47
Alex_Gaynorfungi: so in theory "devstack-precise-hpcloud" should have it now?14:47
fungiAlex_Gaynor: ahh, no those are not unit test slaves, but devstack integration test slaves. they won't have it until new devstack slave images are updated and new slaves launched from them. let me see when that change merged to ourpuppet config14:48
Alex_Gaynorfungi: ah, doh, sorry I didn't realize they were of different sorts14:49
jeblairsdague: here's what mine looks like in ff/linux/xfce:
sdagueyeh, well not chrome14:50
sdaguechrome / raring14:50
jeblairsdague: adding explicit css styling might fix it14:50
jeblairsdague: ah yeah, i see that in chromium too.14:51
fungiAlex_Gaynor: seems to have merged 10 days ago, so it's pretty well guaranteed to be there but i'll check one just to confirm14:52
jeblairsdague: i wonder if setting the width to a different value would change it (eg, those bars are just set at a fixed interval that does not match the width i forced)14:52
jeblairsdague: (they are set to approximate the width of the words "SUCCESS" and "FAILURE" to reduce visual disruption)14:52
* fungi is *so* digging the new status page improvements14:53
Alex_Gaynorfungi: leads me to believe it's not :)14:53
fungijeblair: in my browser (ff17 basically) the status bars are a good 15% longer than SUCCESS but it could be my font is a little more condensed than anticipated14:54
fungiAlex_Gaynor: taking a look14:54
jeblairfungi: no, i think you're right, i think they are a little longer (however, they are defined in ems, so they should scale with the font assuming one choses a correct value).14:57
*** rcleere has joined #openstack-infra14:57
fungimakes sense14:57
*** fbo_away is now known as fbo14:58
*** fbo is now known as fbo_away14:59
sdaguejeblair: where is this all at? might want to play with the css quick14:59
*** mrodden has quit IRC15:01
fungiAlex_Gaynor: i think we need to modify devstack to include libffi-dev in its debs list15:03
anteayalook at all those pretty progress bars15:03
fungiAlex_Gaynor: confirming and will have a patch proposed forthwith15:04
Alex_Gaynorfungi: ok, was going to try to write one myself, but Id on't have a checkout of config and GH seems down :/15:05
sdaguejeblair: you have any issue with me pulling the css out of the html?15:05
fungiyeah, i'm noticing it's slow. you can clone from instead if you want15:05
Alex_Gaynorfungi: s/devstack/config/ ?15:05
fungiAlex_Gaynor: no, this is goint to be a change to devstack i think15:06
Alex_Gaynorfungi: oh, I assumed the puppet manifests for it were also in config, not so?15:06
fungiAlex_Gaynor: puppet manifests for the devstack slaves are in openstack-infra/config but devstack has lists of packages it should depend on itself. does libffi-dev/libffi-devel fall into that category of things?15:07
Alex_Gaynorah, I guess it's the latter then15:07
*** jpeeler has quit IRC15:08
jeblairsdague: there's so little css, it seemed a waste to have a second request for it.15:08
*** jpeeler has joined #openstack-infra15:08
fungiAlex_Gaynor: not everyone who uses devstack uses our puppet manifests to set u their devstack machines, so we want to be nice to them15:08
sdagueyou mean an 8th request for it15:08
sdaguethere are already a lot of requests in there anyway15:08
jeblairsdague: indeed.  :)15:08
*** adalbas has quit IRC15:08
Alex_Gaynorfungi: hmm, where approximately should I be looking?15:09
sdagueto solve that you should compile and compress them all server side15:09
*** dkranz has quit IRC15:09
jeblairsdague: i was trying to keep it simple all around.  :)15:09
Alex_Gaynorfungi: I guess lib/swift ?15:09
fungiAlex_Gaynor: files/apts/swift15:10
Alex_Gaynorfungi: ah, tahnks15:10
fungiAlex_Gaynor: and also files/rpms/swift15:10
fungiAlex_Gaynor: those list packages which should be installed by devstack on their respective platforms when swift is enabled15:10
jeblairsdague: if you feel strongly it should be in another file, that's fine.  it's just that there's no re-use of those attributes outside of this single page, so i didn't see that it gained anything.15:11
Alex_Gaynorfungi: does look sane?15:11
fungiAlex_Gaynor: yep!15:11
Alex_Gaynorfungi: review req incoming!15:12
* jeblair breaks fast15:12
sdaguejeblair: yeh, inline css is just something I typically try to avoid. Where are you actually setting the progress element?15:12
*** rcleere has quit IRC15:12
*** rcleere has joined #openstack-infra15:12
jeblairsdague: status.js15:13
Alex_Gaynorfungi: thanks so much for your help!15:13
sdaguedoh, I had an old pull15:14
*** mrodden has joined #openstack-infra15:15
fungiAlex_Gaynor: it was my pleasure15:16
openstackgerritSean Dague proposed a change to openstack-infra/config: float the progress bars to the right
sdaguealso, any chance in landing the htmlifier changes today to get that on the main log servers?15:20
*** adalbas has joined #openstack-infra15:24
*** dkranz has joined #openstack-infra15:24
*** dina_belova has joined #openstack-infra15:29
openstackgerritA change was merged to openstack-infra/config: change date link to something more understandable
*** vijendar has quit IRC15:33
*** adalbas has quit IRC15:33
fungisdague: testing it out in a bit15:33
sdaguethe patch after that is what flips it15:33
sdagueI still don't understand why we broke readme15:33
sdagueI can't replicate that locally15:34
fungisdague: yeah, what i meant was i'll be testing the one that flips it on in a bit, before voting on it15:35
*** SergeyLukjanov has quit IRC15:35
sdaguecool, thanks15:35
*** akscram has joined #openstack-infra15:36
*** hashar has quit IRC15:37
Alex_Gaynorhmm, doesn't seem to be responding15:38
Alex_Gaynorerr, maybe it is but exceptionally slowly15:38
fungithat webui tends to be slow15:39
*** SergeyLukjanov has joined #openstack-infra15:39
Alex_GaynorIt was outside of normal bounds of sanity long, but it seems to be back to normal now.15:40
*** _TheDodd_ has quit IRC15:41
fungii wouldn't normally put sanity and the jenkins webui in the same sentence15:41
Alex_Gaynoryes, well, that's fair.15:41
pleia2good morning15:42
fungijenkins doesn't look too bad off for a west-coast north american monday morning...
*** dina_belova has quit IRC15:43
*** _TheDodd_ has joined #openstack-infra15:44
sdaguefungi: hmmm... I seem to have broken something in that last go around with the logs15:45
fungisdague: in what way?15:46
fungiseems fine to me so far, just browsing around...15:46
sdaguein the not working way15:46
fungii'll try to pick some more recent logs in case i've cached them somehow15:47
mordredsdague: I haven't read all the scrollback - but we broke README?15:47
*** cody-somerville has joined #openstack-infra15:47
sdaguemordred: in apache, and I don't know why15:48
sdaguefungi: ok, I found one of the logs which went funny, will need to look more15:48
jeblairmordred: as long as we don't break REAMDE15:48
sdaguemordred: on the plus side, it's back15:48
mordredthat's amazing15:48
*** dina_belova has joined #openstack-infra15:50
*** adalbas has joined #openstack-infra15:50
ttxmordred: folsom is fixed with -- if you want to apply capping to it15:50
ttxmordred: if you come up with a script that does it, i'll add it to the stable maint toolkit15:51
*** bclifford has joined #openstack-infra15:52
*** w_ has joined #openstack-infra15:53
*** pcrews has joined #openstack-infra15:54
*** olaph has quit IRC15:55
*** reed has joined #openstack-infra15:56
*** w_ has quit IRC15:57
fungisdague: ahh, yes, i see that has no readme displayed, and console.html is not console.html.gz15:59
*** adalbas has quit IRC16:00
fungisdague: actually, i do see readme on old-style log directories, from before we added the extra level of depth... i wonder if that's what broke it16:00
* fungi digs back into the readme path pattern16:01
fungiyep! we've got patterns in logs.vhost.erb matching on the number of parent directories in the path16:03
*** datsun180b has quit IRC16:04
*** bearhands is now known as comstud16:04
*** vijendar has joined #openstack-infra16:06
sdagueah, cool16:07
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Add new jenkins masters.
*** sdake has quit IRC16:08
jeblaircan we match on any number of directories?16:08
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Additional path patterns for log readme files.
fungisdague: ^16:09
jeblairfungi: ^16:09
*** pabelanger has quit IRC16:09
fungijeblair: oh, maybe... i wonder if ** will glob on multiple directory levels16:09
sdaguejeblair: we want to match specific numbers, for a reason16:09
jeblairi'd really like it if that worked for, say, periodic jobs too.16:09
*** longlostnickname is now known as cp16net16:09
jeblairsdague: what's the reason?16:10
sdaguewe have different kinds of docs at different levels16:10
sdagueanyway, need food here, back in a few16:10
jeblairsdague: i mean any number of initial directories16:10
*** pabelanger has joined #openstack-infra16:10
*** w_ has joined #openstack-infra16:10
jeblairit doesn't matter what's before "gate-tempest-devstack*", only after.16:11
fungii'll try a variant of it on the logs-dev vhost and see what happens16:11
fungikeep in mind that we can only do what apache's glob matching allows. regular expressions aren't possible for applying readmename and headername16:11
fungiwhich is unfortunate, and surprising, but backed up by the apache docs16:12
jeblairclarkb, mordred, fungi: quick review on ?  I'd like to start spinning those up today.16:13
jeblairclarkb: i would like to defer your puppet suggestions in that change for another time, unless someone knows exactly how to do that right now.16:13
*** nicedice has joined #openstack-infra16:13
*** dina_belova has quit IRC16:13
fungijeblair: sure thing. same for ...i'd like to start spinning those up today too16:14
fungiand that's the only patch blocking production py33 unit test slaves at the moment16:14
mordredjeblair: is the idea that you'll spin up nodes with those16:15
clarkbjeblair: I am ok with defering my suggestions. It will probably require use of puppet-stdlib to do string manipulation in ways that are not pretty16:15
* clarkb updates review16:15
mordredjeblair: and then just do the global jenkins config by hand?16:15
*** adalbas has joined #openstack-infra16:16
mordred(which is fine, just making sure I grok)16:16
jeblairmordred: yes.  i'm deferring management of the global jenkins config as well.16:16
*** sdake has joined #openstack-infra16:17
*** sdake has quit IRC16:17
*** sdake has joined #openstack-infra16:17
jeblairmordred: (i expect to have a copy of the interesting parts of the config as a side effect, so small progress can be made on that task)16:17
openstackgerritA change was merged to openstack-infra/config: Conditionally install Python 3 development libs.
jeblairalso, if we merge this:
jeblairwe can have better validation of layout.yaml16:18
fungijeblair: do you want to defer addition to modules/openstack_project/files/ssl_cert_check/ssldomains for a separate change as well?16:20
fungiif so, lgtm otherwise16:21
*** reed has quit IRC16:21
jeblairfungi: ah, yeah, what should we do about ssl certs?  atm i'm leaning toward self-signed...16:21
openstackgerritTom Fifield proposed a change to openstack-infra/jeepyb: cease docimpact emails, create bugs only
*** w_ is now known as olaph16:21
jeblair(though currently the change just reuses the jenkins.o.o cert)16:21
fungijeblair: that's cool with me. we don't so much direct people to the jenkins webui ay longer16:21
*** jergerber has quit IRC16:21
jeblairok, how about i update it to use self-signed certs that are in hiera (and i'll add them)16:21
fungiand set crazy-long expiration on them16:22
fungii missed that it was reusing the same key contents, which will be not entirely correct. i see it in there now16:23
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Add new jenkins masters.
jeblairthere it is16:24
fungijeblair: one other concern this raises... won't there be contention on archived job artifact paths?16:26
clarkbjeblair: approve at will16:26
clarkbthe note about the periodic jobs is worth thinking about16:26
clarkbjeblair: we could even have a cron job on zuul.o.o insert those jobs into the gearman queue for now16:27
jeblairclarkb: adding timer support to zuul is item 4 on my todo list16:27
*** adalbas has quit IRC16:27
jeblairfungi: well, lets think about that16:27
fungiright now the job sequence numbers are decided by each jenkins master independently16:28
jeblairfungi: .../change/patchset/pipeline/job/number16:28
*** thingee has quit IRC16:28
fungifor triggered jobs we also include the change and patchset number, so unless there'sa recheck16:28
*** emagana has joined #openstack-infra16:28
jeblairwe could s/number/zuul_uuid/16:29
fungiif zuul decided the job number instead of jenkins, it ceases to be an issue16:29
jeblairurl gets long though.16:29
jeblairwe could truncate the uuid.16:29
fungiyeah, something like that16:30
clarkbor list the master16:30
clarkbmaster/slave/some other unique key making value16:30
*** odyssey4me has quit IRC16:31
Alex_GaynorIs there anyway to enumerate hte people who are core reviewers on a given project?16:31
fungiAlex_Gaynor: you can look at
clarkbAlex_Gaynor: and if you don't know which group you need, you can looked under admin -> project and view the project ACLs16:32
*** _TheDodd_ has quit IRC16:32
fungiclarkb: though the acls are redacted if you don't have the right group membership16:32
clarkbI guess that sort of makes sense16:33
funginot everyone sees the same acl contents in the webui. totally depends on whether they're in the groups mentioned in the acl16:33
fungibut we can point people to openstack-infra/config:modules/openstack_project/files/gerrit/acls/ instead16:33
clarkbdo you get the gitweb link regardless? because that will expose everything16:33
clarkbor point people at the source16:34
*** thingee has joined #openstack-infra16:34
jeblairfungi, clarkb: we could use the build start time instead of build number.16:35
jeblairchance of starting two jobs with the same name for the same change in the same pipeline at the same second is very very small.16:36
clarkbjeblair: and since zuul won't queue a single change up in the same pipelines twice that will be unique16:36
clarkbjeblair: the one place where it will break is with the periodic jobs16:36
clarkbbut we can probably live with that until zuul schedules them16:36
fungiagreed. the job would have to finish and get requeued with the same patchset number in less than a second (assuming we're using unixtime or something)16:36
jeblairclarkb: yes, it seems that in practice it would be unique.  advantage is that it's a sequential counter so it's easy to see whch is latest.16:37
clarkbgood point on that over uuids16:37
jeblair(of course, apache puts timestamps in the dir listing, so i guess a hash would be as well)16:37
jeblairwhat order does apache list dirs in?16:37
*** ^demon has joined #openstack-infra16:37
*** ^demon has joined #openstack-infra16:37
jeblairasceding filename order by default16:38
fungilocally configurable, but alpha order by default. which alpha order (c vs. en_us) depends on the default locale for the server i think16:38
jeblairso we could change that to date and use uuids, or leave as alpha and use unix timestamps16:39
fungiif we're not using capital letters in the id, the locale concerns are moot16:39
clarkbuuids would be technically correct I think16:39
clarkbas they uniquely identify each job16:39
*** dkranz has quit IRC16:40
jeblairclarkb: well, mostly uniquely if we short them, but yes.  :)16:40
fungifairly unlikely to collide given the small number of them which will coexist in that parent dir16:40
jeblairi'm concerned about changing the default sort order though16:41
*** vipul-away is now known as vipul16:41
clarkbOne other thing to keep in mind is today we can easily determine the job url with no external info. If you have the change/commits details you can construct the url16:41
fungizuul could keep an integer counter of its own i guess, but that's more effort16:41
fungiand would have to persist across restarts16:41
jeblairfungi: zuul has no state across restarts currently (i expect that to change)16:41
* fungi nods16:42
jeblairbut i'm trying to keep dependencies for this change low.16:42
clarkbthough I guess the actual job numbers are not constructable16:42
clarkbso uuids/dates would be equivalent there16:42
jeblairclarkb: right.  i think the main thing is that you go into that dir, and look for the most recent (or whichever) job.16:42
fungijenkins job number plus zuul change id, concatenated?16:42
jeblairfungi: that still may not sort correctly16:43
fungioh, right, one master may run lots more of a given job than the other16:43
jeblairfungi: 7-uuid could be followed by 3-uuid16:43
*** adalbas has joined #openstack-infra16:43
jeblairlet's just try short uuid, and expect someone looking at the dir index to read the dates.16:44
*** afazekas has quit IRC16:44
*** simonmcc has quit IRC16:44
jeblair(and we can address that later if it's a problem; not many people do that)16:44
*** _TheDodd_ has joined #openstack-infra16:44
*** simonmcc has joined #openstack-infra16:45
*** jpich has quit IRC16:45
fungisounds fine16:45
*** jog0-away is now known as jog016:45
clarkbjeblair: in other news gear appears to leak memory. My current theory is it is related to the way python lists grow and fail at shrinking16:45
fungiif they really care about sort order in the autoindex, there's a column sort link16:45
clarkbjeblair: I need to read more gear but I think using deques to store the queued jobs may help16:46
*** psedlak__ has quit IRC16:46
jeblairclarkb: quite possibly16:46
jeblairfungi: +116:46
clarkbjeblair: note the drop in memory use last night was a result of my restarting the gear server16:47
*** datsun180b has joined #openstack-infra16:47
clarkbwith the new job log path sorted /me AFKs for a bit16:48
*** SergeyLukjanov has quit IRC16:49
*** bnemec has quit IRC16:51
*** rpodolyaka has left #openstack-infra16:51
*** rpodolyaka has quit IRC16:51
*** nati_ueno has joined #openstack-infra16:52
*** datsun180b has quit IRC16:53
*** datsun180b has joined #openstack-infra16:53
*** dkranz has joined #openstack-infra16:53
*** bnemec has joined #openstack-infra16:53
*** adalbas has quit IRC16:54
*** mriedem has quit IRC16:56
annegentleanyone know if we have transclusion enabled on our wiki? My guess is no after testing those steps.16:58
*** sarob has joined #openstack-infra17:01
*** NobodyCam has quit IRC17:02
*** Ryan_Lane has joined #openstack-infra17:05
openstackgerritJames E. Blair proposed a change to openstack-infra/zuul: Pass the Job to the parameter function
openstackgerritJames E. Blair proposed a change to openstack-infra/zuul: Pass the Job to the parameter function
Alex_Gaynorfungi: once the devstack patch is landed, does anything need to be done for that to propogate (I think you mentioned building new images?), or is that automate?17:08
jeblairAlex_Gaynor: it will take effect instantly (even the check job on the patch should use the change)17:08
jeblairAlex_Gaynor: until a new image is built, it will be downloading it from the ubuntu mirror, which will slow the job down a tiny bit17:09
Alex_GaynorAwesome thanks, now I just need to find some reviewers to bribe. (Note: if it makes a difference, I will bake some cookies to bribe you with)17:09
jeblairAlex_Gaynor: but the next time an image is rebuilt, the package will be cached locally and runtimes will be faster17:09
*** pabelanger has quit IRC17:10
*** adalbas has joined #openstack-infra17:10
fungiAlex_Gaynor: right. it was puppet changes which might imply a need to rebuild devstack slave images and burn the old ones. devstack itself does what it's configured for at that point in time17:10
Alex_Gaynoryay for automation17:10
clarkbfungi devstack should install libffi17:11
clarkbthis is why it wasnt added to puppet for d-g slaves17:12
*** SergeyLukjanov has joined #openstack-infra17:12
fungiclarkb: yep. that's why i nudged Alex_Gaynor to add it to devstack17:13
*** reed has joined #openstack-infra17:14
fungiwhen he was first asking whether it was installed on all the slaves, i thought he meant unit test slaves (since you had a change to add that a week or so ago)17:14
fungithen he linked to a tempest-devstack log, and i realized that wasn't what he was talking about17:15
*** dims has quit IRC17:17
fungijeblair: sdague: tried using /**/ to match multiple depths, but apache globs aren't that smart apparently. looking through the docs, i can't see any way to match multiple directory levels with a single glob (only regular expressions)17:17
jeblairfungi: this is really unfortunate, since i'm about to add a path element to a bunch of logs.  and then change them around again.17:18
jeblairfungi: the config is going to be a mess by the time we're done.  :(17:18
fungiagreed, though the old patterns only need to stick around until the logs using them expire and get cleaned out17:19
jeblairfungi: however, eventually we will have exhaustively listed all of the likely possibilities: '*/', '*/*/', ...17:19
*** pcrews has quit IRC17:19
zaroclarkb: got time to  make a release for zmq?17:19
jeblairfungi: so i guess it's a bounded problem.17:19
fungithe alternative is to make ReadmeName point to an ssi and have it decide what to present based on a more programmatic investigation of the path17:20
jeblairfungi: so i guess in addition to your change, you should also add the current spec for periodic jobs, and also one deeper than the current periodic jobs (if you want to future-proof against what i'm doing)17:21
fungithe up side to that option is that we get to wax nostalgic about ssi17:21
jeblairfungi: :) i think the long term is we keep moving in the direction of having the log serving app handle it.17:21
fungithat definitely seems sanest17:21
fungisdague: just to confirm, are the current readmes relevant to periodic jobs as well? (see above)17:22
sdagueyes, they should be17:23
fungiperfect. i'll tack them on and future-proof them17:23
*** dina_belova has joined #openstack-infra17:23
*** cp16net is now known as cp16net|away17:26
*** dims has joined #openstack-infra17:29
*** cp16net|away is now known as cp16net17:31
*** burt has quit IRC17:34
*** dims has quit IRC17:36
*** dims has joined #openstack-infra17:36
*** ladquin has joined #openstack-infra17:37
*** sarob has quit IRC17:37
*** sarob has joined #openstack-infra17:37
*** avtar has quit IRC17:39
*** zul has quit IRC17:41
*** sarob has quit IRC17:41
*** dina_belova has quit IRC17:42
*** rnirmal has joined #openstack-infra17:43
*** datsun180b_ has joined #openstack-infra17:43
*** datsun180b has quit IRC17:45
*** datsun180b_ is now known as datsun180b17:45
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Switch Zuul/JJB to use LOG_PATH
jeblairso it turns out that's not easy.  and there's at least one reason why we'd want to go ahead and do the zuul trigger change before that as well.17:49
clarkbjeblair: the not easy bits being in zuul?17:52
clarkbthe config change seems minor17:52
jeblairclarkb: all over17:52
jeblairclarkb: requires a code change to zuul, and breaks logstash for periodic jobs.17:53
jeblairclarkb: (until zuul has timed triggers)17:53
clarkbjeblair: line 92. Why are we removing the post and release stuff?17:54
jeblairclarkb: LOG_PATH will always be set by zuul for all jobs17:56
clarkbjeblair: the shortref is included in the fields so that it ends up in the logstash events. That way you can trace particular events back to the commit that caused them to run17:58
clarkbjeblair: I think we should keep the short ref in this case even if it isn't necessaryfor construction of the url17:58
clarkbI will leave a review17:59
*** zul has joined #openstack-infra18:00
jeblairclarkb: which jobs get a shortref and store info in logstash?18:01
clarkbjeblair: the post, prerelease and release jobs18:01
*** dkliban is now known as dkliban_afk18:01
jeblairclarkb: logstash picks those up?  i thought it was just devstack jobs?18:01
clarkbjeblair: it picks up everything18:02
jeblairclarkb: ok, well it should probably just use the full sha instead of the short one then.18:02
clarkbfor most jobs that means only the console log ends up in logstash but for many things that is sufficient18:02
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Additional path patterns for log readme files.
clarkb_TheDodd_: your commit message bug automation message looks good to me18:05
jeblair_TheDodd_: me too18:05
clarkbjeblair: fungi: should we approve that change now and send the message after some burn in or just do both?18:05
_TheDodd_clarkb, great! Glad to hear it.18:05
jeblairclarkb: since it's backwards compat, let's burn-in.18:06
fungiclarkb: since it's backward compatible, ... what jeblair just said18:06
clarkbok, I will approve it now18:06
pleia2fungi: if you have a few minutes as some point, can we take a look at the jeepyb install status on git.o.o? I saw it get nudged in puppet hours ago, but it still seems to have dependency issues18:06
fungipleia2: can you point me to a log or paste.o.o the error?18:07
pleia2sure, just a moment18:07
fungipleia2: wondering if it's another case of what i'm trying to fix with
openstackgerritA change was merged to openstack-infra/jeepyb: Improving Gerrit + commit-log automation.
pleia2fungi: log tab18:08
pleia2fungi: ah yes, I saw that commit earlier this weekend and was wondering if that would be the key18:08
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Switch Zuul/JJB to use LOG_PATH
fungipleia2: your log there looks like a github problem18:09
jeblairclarkb: ^ a slightly more comprehensive treatment of the subject which more closely matches zuul behavior.18:09
pleia2fungi: yeah, just realized that18:09
pleia2fungi: even so, I don't think jeepyb is installed properly (have to dig a bit further to when the first patches went in)18:09
fungipleia2: i'll jump on the server and check18:09
clarkbjeblair: lgtm18:10
clarkb_TheDodd_: 36866 should get applied in the next half hour or so assuming jeepyb is installing properly on review.o.o18:10
* jeblair -> afk18:11
clarkbat that point we can test it with a dummy change18:11
pleia2fungi: it should have installed and exec'ed on /home/cgit/projects.yaml to create /etc/cgitrepos - but I don't see evidence of this having happened18:11
_TheDodd_Ok, sounds great.18:11
fungipleia2: /opt/jeepyb is more than a week behind18:11
fungiwhich makes me think something is wrong with that clone of the git repo. i'll play around with it a little18:12
pleia2fungi: thank you18:12
fungipleia2: not sure what's going on there. i was able to remote update and pull into master18:13
fungialso fetch --tags origin is not erroring for me in that clone18:14
* pleia2 checks on vcsrepo details18:14
pleia2right, is just be the standard jeepyb include18:15
pleia2fungi: does /etc/cgitrepos exist?18:15
pleia2wondering if exec didn't trigger because now it thinks /home/cgit/projects.yaml hasn't changed18:16
pleia2this all went much smoother on my test vm :)18:16
clarkbpleia2: I think that a rule about puppet is you will get beneficial ordering on your test machien and the worst possible order in production18:17
fungipleia2: i tried to run the install exec for the jeepyb puppet module, but looks like jeepyb is not entirely python 2.6 compatible18:17
pleia2clarkb: haha18:17
fungicouple things we may need to patch in jeepyb to make it work18:18
pleia2hmm, how did I get this to work18:19
*** pcrews has joined #openstack-infra18:19
fungipleia2: clarkb:
fungithat's a new one on me18:20
pleia2[root@git-vanilla ~]# ls /opt/jeepyb/18:20
pleia2I definitely managed to get it installed on my system18:21
fungipleia2: in /opt/jeepyb (as root) run 'python install'18:21
clarkbfungi: looks like we have somehow registered an atexit function that is a NoneType18:21
clarkbfungi: is install creating that error?18:22
pleia2seems ok18:22
*** UtahDave has joined #openstack-infra18:22
fungiclarkb: it's hard to tell. the logs is missing context, but it's right after "Installing openstackwatch script to /usr/bin"18:22
*** arosen has quit IRC18:23
fungialso pip freeze doesn't show pbr installed18:24
pleia2mine doesn't have pbr18:24
fungiahh, i guess pbr doesn't need to be installed globally since it's not a runtime dependency, just a setup_requires18:26
fungiso ignore that bit of speculation18:26
pleia2and don't have anything weird going on with python, just default 2.6.6-37.el6_418:29
sdaguemordred: so I'm still scratching my head why mtreinish's change didn't cause the grenade fail18:29
sdaguebut then everything after him did18:29
*** sarob has joined #openstack-infra18:33
*** dkliban_afk is now known as dkliban18:36
*** hashar has joined #openstack-infra18:37
*** dina_belova has joined #openstack-infra18:40
*** arosen has joined #openstack-infra18:40
fungiclarkb: pleia2: the plot thickens... running python install a second time doesn't spew those tracebacks any longer18:42
*** sarob has quit IRC18:42
fungii'm going to try uninstalling jeepyb and seeing if it installs cleanly a second time18:42
*** sarob has joined #openstack-infra18:42
*** mgagne has joined #openstack-infra18:43
fungigot it!18:43
fungiif i 'git clean -dfx' before 'python install' then it happens, so it's something running out of one of the other cached eggs i guess18:44
*** vipul is now known as vipul-away18:45
fungiimplying those tracebacks are possibly benign18:45
mordredsdague: can you show me his change?18:45
mordredttx: ping18:47
pleia2fungi: ok, so jeepyb should be installed ok now, does /etc/cgitrepos exist? wondering if we just wait for next puppet run if not18:47
sdaguewith the associated test runs18:47
fungipleia2: since puppet installs that, i'll need to fire a puppet run18:48
Alex_Gaynorsdague: Is there something is missing to get an accept?18:48
sdaguea second +218:49
mordredsdague: and you're telling me that the change revert it _fixes_ grenade?18:49
Alex_Gaynoroh, duh, sorry I'm slow today apparently18:49
sdaguemordred: it was the awesome... and by that, not very awesome18:49
mordredsdague: so, that makes ZERO sense18:50
fungipleia2: puppet runs clean but /etc/cgitrepos still dne18:50
mordredwe don't install anything from requirements.txt yet18:50
sdaguemordred: yes we do18:50
mordredoh. wait18:50
fungipleia2: maybe i need to step the git repo for jeepyb back a commit to trigger the machinery. trying that now18:50
pleia2fungi: ok, so the exec for creating it runs on /home/cgit/projects.yaml being changed, can you confirm projects.yaml exists?18:50
mordredsdague: sorry man. my brain thought that was requirements.txt in openstack/requirements18:50
sdagueif we didn't, everything would be broken all the time :)18:50
mordredsdague: looking again18:50
*** burt has joined #openstack-infra18:51
fungipleia2: yeah, /home/cgit/projects.yaml is there last modified Jul 22 05:5318:51
pleia2fungi: puppet installs /etc/cgitrc but /etc/cgitrepos is generated by jeepyb's script that runs when projects.yaml cnages18:51
fungipleia2: i'll try tweaking that so it gets updated18:51
pleia2fungi: thanks18:51
sdaguemordred: so .... the force on global requirements in devstack, isn't going to work. Because it pops that up before rpm package install18:52
sdagueor deb package install18:52
mordredsdague: can you tell me what the general strategy is for requirements in grenade?18:52
sdagueregardless, it means you don't have devel headers18:52
sdaguemordred: ok on grenade18:52
sdagueit uses pieces of devstack18:52
sdagueso basically goes into the new git source trees, runs develop, and off we go18:53
fungipleia2: sorry!
mordredso - it installs via devstack the first time - then how does it upgrade the requrements?18:54
mordredsdague: also, I disagree with you re: devstack, the install reuqirements code is right before all the develop calls18:54
adalbassdague, from the logs you sen in the mailing list, it looks like we had a race condition with cinder. My guess is that it may have caused when the job downgrades jsonschema to version 1.3.018:54
sdaguemordred: but it assumes *all* services are enabled18:55
mordredsdague: ah. I see what you are saying18:56
fungipleia2: aha!
mordredbecause we ... oh for the love of18:56
fungipleia2: pbr again18:56
pleia2fungi: bah18:56
mordredsdague: ok. I have ideas - let's come back to them18:56
sdaguewell, we correctly only install the packages we need18:56
mordredyeah. no, I'm with you18:56
fungipleia2: so pbr *is* a runtime dependency of (at least parts of) jeepyb but is not being installed globally since it's only in setup_requires18:56
sdagueok, so yes, it does requirements processing twice18:56
mordredlet's go back to grenade, it's more imporant right now - you install grizzly devstack software as expected, yeah?18:56
sdagueonce through on all enabled packages on the first devstack18:57
pleia2fungi: so `python /opt/jeepyb/jeepyb/cmd/` should work fine, but for the wrapper thing we use we need pbr is required18:57
mordredsdague: are you doing pip install -U on those?18:57
sdaguethen you do the install_* portion for the new devstack trees18:57
mordredor just the install_ functions which do develop18:57
pleia2fungi: except in english. s/we use we need pbr is required/we use we need pbr18:57
mordredsdague: sorry, I'm asking dumb questions on purpose here18:57
*** vipul-away is now known as vipul18:58
mordredwanting to make sure I don't miss an assumption18:58
fungipleia2: more or less. i want to pick mordred's brain on the installing from local source, should pbr be globally installed question once he pulls his brain back out of grenade for a bit18:58
pleia2fungi: ok, thanks18:58
sdaguethat's what we do to each tree18:58
openstackgerritClark Boylan proposed a change to openstack-infra/config: Add Wiki upgrade process documentation.
clarkbRyan_Lane: ^ any chance you can review that18:59
clarkbjeblair: fungi mordred ^ also18:59
sdagueadalbas: yeh, we're actually running through the post mortem on that right now18:59
adalbassdague, yes, im following your thoughts here18:59
mordredsdague: ok. I don't think that's likely to have the effects you want19:00
mordredbecause there is no -U involved19:00
mordredwhich means that pip is probably not going to do what you think it is19:00
sdaguemordred: why19:00
sdaguewe only need to bump the requirement if we have to bump the requirement19:00
mordredI do not believe that pip will bump the requirement even in that case19:01
sdaguewe know it downgrades if you are out of bounds19:01
adalbasmordred, in that specific case, if i correctly understood, what happened was a downgrade instead19:01
sdagueso why don't you think it will upgrade if out of bounds?19:01
* mordred may need to make a sequencing test script19:01
clarkbpleia2: meeting tomorrow and bug day right?19:01
fungipleia2: manually pip-installing pbr and d2to1 gets it further, but something also wants pycrypto>=2.1,!=2.4 and i'm getting 'splosions trying to pip install it19:01
sdaguewe still actually downgrade a package as part of normal runs right now19:01
sdagueand upgrade many19:01
fungipleia2: "RuntimeError: autoconf error" does not sound good19:02
pleia2clarkb: oh bother, I forgot about bug day! MarkAtwood is borrowing me to Palo Alto tomorrow morning (will probably miss the meeting too)19:02
pleia2I can prep an etherpad today that everyone can fill out19:02
clarkbpleia2: that would be super19:02
* pleia2 has a lplib script now :)19:02
fungipleia2: heh, "configure: error: no acceptable C compiler found in $PATH"19:02
* fungi boggles at a modern linux with no cc19:03
mordredsdague: oh. ok. I'm wrong.19:03
pleia2fungi: gosh, what is using gcc?19:03
mordredsdague: (sorry, needed to test it myself)19:03
sdaguemordred: yeh -
sdaguethat's in a normal functioning install19:03
fungipleia2: no, there's no c compiler installed by default at all it seems19:03
*** nati_uen_ has joined #openstack-infra19:03
clarkbfungi: ya ubuntu does not install gcc in a default install19:04
pleia2fungi: right, I'm wondering what we're installing that needs it19:04
sdagueeverything except the cinderclient requests bit is an upgrade, that's actually a downgrade19:04
clarkbneither do a bunch of other distros19:04
pleia2clarkb: centos19:04
sdagueit's hard to one line grep for that though, because pip spits out useful information in too many discreet places19:04
*** nati_ueno has quit IRC19:06
sdagueok, back to global requirements on devstack - so for use to do what we want to do with devstack.... we'd need to make the base case act more like devstack gate, right?19:07
sdaguewhere what we actually do is pull all the trees up front19:08
sdaguethen apply all the pip files at once19:08
_TheDodd_clarkb: where do we stand? Has disaster struck?19:08
mordredsdague: actually19:08
mordredsdague: I think the easier thing might be19:08
mordredsdague: clone requirements19:08
clarkb_TheDodd_: I don't know. I haven't tested the change yet. Let me check really quickly if the jeepyb install updated then we can test19:08
fungipleia2: pycrypto wants to compile modules19:09
mordredsdague: then, in the pip_install function19:09
mordredsdague: run the script from requirements19:09
mordredsdague: which will update the local requirements list with the ones in the requirements file19:09
mordredbut we'll still only run the ones that we're configured to run19:09
mordredand it shouldn't be that hard of a patch19:09
mordredand it should also have the benefit of making sure that update script works19:09
sdagueok, 'splain that too me again, I'm slow19:09
clarkb_TheDodd_: looks like we still have old jeepyb on review.o.o.19:10
mordredopenstack/requirements has an script19:10
clarkb_TheDodd_: fungi is currently debugging issues with installing jeepyb on a different host, hopefully he will know how to correct review.o.o's install19:10
mordredwhich will update the requirements in a project to the ones listed in openstack/requirements - but only for the deps the project already uses19:10
mordredso it won't add things - it will only update versions19:10
clarkbfungi: any chance you can poke at review.o.o when git.o.o is happy?19:10
sdagueso in each of the projects, run the update19:10
mordredso, if in the setup_develop script, we first run the update19:10
sdagueok. interesting. let me have a go at that19:11
fungiclarkb: what needs poking?19:11
mordredthat should also let us know that if a requirements change passes the gate19:11
clarkbfungi: jeepyb on review.o.o isnt updating to pull in _TheDodd_'s update bug change19:11
mordredthat automation around using to update the projects _should_ work19:11
clarkbfungi: 38066 should fixthe problem?19:12
clarkbI am starving. I will grab lunch but can assist with more pbr jeepyb after19:12
fungiclarkb: 38066 *does* work around the problem. according to mordred some heavy hacking on pbr internals would be required to actually *fix* the problem19:12
clarkbfungi: wfm19:13
mordredfungi: I'm not even sure that hacking on pbr will get us out of this hole, just because of how the whole system works - but it's definitely more dev time than we probably want to be without an up to date jeepyb19:13
*** sarob has quit IRC19:13
*** sarob_ has joined #openstack-infra19:14
fungiclarkb: pleia2: as for git.o.o, something about the entrypoint for create-cgitrepos wants pycrypto (either directly or indirectly) which we'll either need a suitable version of installed or need a compiler if we want pip to install it for us19:14
pleia2requirements for that script are pretty minimal, just imports os and yaml, so it must be pretty indirect19:16
fungispecifically from pkg_resources import load_entry_point seems to be triggering that19:16
pleia2does look like I did end up installing gcc somewhere along the way on my test system though, browsing history now19:17
pleia2heh, so I probably did run into this too19:18
*** sarob_ has quit IRC19:18
fungifrom whence did you obtain pycrypto? can you tell whether it was via pip or yum?19:18
pleia2[root@git-vanilla config]# pip freeze | grep pycrypto19:18
fungipleia2: so in theory, to force pbr to be installed globally before jeepyb, along with adding the gcc package to the git.pp manifest ought to get us past this19:20
pleia2should we just add the gcc package as a depenency for jeepyb?19:20
pleia2or that19:20
fungii think so, yes19:21
fungipleia2: -rw-r--r--. 1 root root 20322 Jul 29 19:21 /etc/cgitrepos19:21
pleia2woo, is pretty now19:22
pleia2(no repos behind it yet though, so no links work)19:22
fungipleia2: once we iron out these last bits, we probably ought to blow away this server and launch a fresh one just to make sure all the retooling is sufficient to build from scratch like we hope19:22
pleia2fungi: +119:23
*** bnemec has quit IRC19:23
*** bnemec has joined #openstack-infra19:23
pleia2I think jeblair said he got an ssl cert, so I also need to configure that w/ apache19:23
fungibecause i no longer trust that vm to be 100% consistent with what we'll get, following some of the troubleshooting i've done there19:23
pleia2and finish getting the git:// server installed19:23
pleia2oh, and we need to decide how we want to structure the URLs19:24
fungipotential topic for tomorrow's meeting19:24
fungior just take a first stab and we can hash it out in review19:24
openstackgerritSergey Lukjanov proposed a change to openstack-dev/hacking: Import exceptions list is now configurable
pleia2fungi: so I'll probably miss that meeting (have to go down the peninsula tomorrow for some on-site stuff)19:25
fungihow ever will we manage a bug day without you though?19:25
pleia2I'll see where I get today with other things though, maybe I can snag a cube at noon19:25
anteayattx are you about?19:25
pleia2fungi: I am prepping the etherpad this afternoon ;)19:26
fungipleia2: too awesome19:26
sdaguemordred: is there a reason we do pbr before setup_tools?19:30
sdaguein devstack19:30
mordredit should be the other way around19:30
mordredupgrade setuptools, then install pbr, then install real things19:30
sdagueok, cool19:31
sdaguemakes it easier19:31
fungipuppet agent on review.o.o is hung for the past ~5 hours due to github hiccough. fixing19:31
sdaguei think they are being ddosed today19:32
*** swifterdarrell has quit IRC19:32
*** thingee_ has joined #openstack-infra19:32
*** thingee has quit IRC19:33
*** crank has quit IRC19:33
*** jaypipes has quit IRC19:33
*** klrmn has quit IRC19:33
*** swifterdarrell has joined #openstack-infra19:33
*** jaypipes has joined #openstack-infra19:33
*** klrmn has joined #openstack-infra19:33
*** crank has joined #openstack-infra19:34
fungisame for lists.o.o, though it wasn't hung on anything particular, it seemed to be running and enabled but not reporting into dashboard. i've cleanly stopped and started the agent to see if that helps19:35
sdaguemordred: where does pkg_resources come from in a base install?19:36
sdaguenever mind, got it19:37
*** leifmadsen has quit IRC19:38
sdaguescrubbing back a devstack node to bare is *interesting*19:38
*** nati_uen_ has quit IRC19:38
*** leifmadsen has joined #openstack-infra19:38
mordredsdague: :)19:40
mordredsdague: oh - btw- over the weekend, I put together diskimage-builder elements that makes a qcow2 image that looks like a devstack-gate node19:41
*** nati_ueno has joined #openstack-infra19:41
mordredsdague: it'll be a few steps yet before we can base anything off it - but it's totally useful for locally spinning up a vm that's identical to a d-g node19:41
pleia2mordred: nice!19:41 develop not valid for pbr?19:42
pleia2I think I saw that review come in, but didn't take a close look yet19:42
sdaguedamn you mordred for being unstandard.... :)19:42
mordredpleia2: I also tested takeovernode/kexec to make a cloud server look like it19:42
mordredsdague: it very much shoudl work19:42
sdaguemordred: qcow2 d-g sounds cool :)19:42
mordredsdague: I have trunk pbr develop'd on my laptop all the time19:42
mordredsdague: can you paste me your error?19:42
jeblaircurrent ATC count for Icehouse summit: 86019:42
mordredjeblair: WOW19:42
mordredjeblair: we have a lot of devs19:43
sdaguejeblair: nice19:43
sdaguethat the last 12 months?19:43
jeblairthats 2012-09-27 -- new19:43
mordredis anyone going to be ironic enough to throw a party with kegs of icehouse beer?19:43
jeblairthats 2012-09-27 -- now19:43
mordredwe should throw an Ironic™ party where we ironically service Icehouse™ beer19:43
sdaguemordred: you can't get beer that crappy in hong kong :)19:43
* sdague hits mordred with a trout19:43
mordredsdague: we could import it - hell man, I work for HP - I'm sure we have a fleet of boats somewhere19:44
jeblairfungi: and i haven't forgotten that i plan on checking that in and cronning it; however that has bumped a few places down on the list (currently #8)19:44
*** w_ has joined #openstack-infra19:45
*** cp16net is now known as cp16net|away19:46
*** rpodolyaka has joined #openstack-infra19:46
mordredsdague: you moved the setuptools install to before it?19:46
mordredbecause you're showing me a machine witha broken setuptools upgrade19:46
sdagueyeh, hold on, I'll push the change back out onto gerrit19:46
mordredsdague: this is the pip instlaled setuptools gets weird with os installed setuptools19:47
mordredgod I hate every bit of this19:47
sdague"Why OpenStack hates Python™"19:47
*** olaph has quit IRC19:47
mordredit's VERY tempting to embed a fork of setuptools renamed into pbr that does not use or touch system setuptools or distutils19:48
mordredbut instead just does EVERYTHING on its own19:48
mordredbut I will not do this19:48
sdagueSean Dague proposed a change to openstack-dev/devstack: Update projects to global requirements before setup
fungimordred: embed a renamed python interpreter into pbr19:48
sdagueso that's my new version of your patch19:48
sdagueI'm also trying to unwind back to working pip and friends from the os, which is mostly about uninstalling packages19:50
sdagueso if you think there is something I missed, let me know19:50
rpodolyakamordred: could you take a look at when you've got a minute? I'19:51
mordredsdague: I just updated - try that on for size19:52
mordredsdague: funny enough - I had made another devstack patch a little while ago that actually does exactly that- it uninstalls system setuptools after pip installing it19:53
mordredbecause, it turns out, things go horribly horribly wrong19:53
sdagueyou have got to be kidding me....19:54
rpodolyakamordred: I've modified tox.ini of SQLAlchemy-migrate to run tests on every Python/SQLAlchemy combination we claim to support. It's a bit ugly in tox 1.x... Is it the right approach and can we set up gate jobs to run those tests? Thanks!19:54
sdagueI feel like I'm playing out "whose on first"19:54
mordredrpodolyaka: yes and yes19:55
mordredsdague: tell me about it. I cannot WAIT until zul has fixed ubuntu and dprince has fixed redhat for us so that we can stop needing to do this dance19:56
sdagueyeh, so that doesn't work19:56
rpodolyakamordred: ok, thanks a lot! I'll make a patch for openstack-infra/config and add you to the review19:57
sdagueoh, wait one minute19:57
*** rpodolyaka has left #openstack-infra19:57
Alex_Gaynor2.5 hours is the timeout for jenkins jobs right?19:57
*** cybertron_ has joined #openstack-infra19:58
*** cp16net|away is now known as cp16net19:58
*** bnemec has quit IRC19:58
sdaguemordred: ok, that might work19:59
*** derekh has joined #openstack-infra19:59
mordredsdague: it's an evil sledgehammer19:59
sdagueI had to unborkify a lot to get to a cleanish state on things19:59
fungiAlex_Gaynor: it depends entirely on the job20:00
*** vipul has quit IRC20:00
Alex_Gaynorfungi: the devstack ones specifically20:00
*** vipul has joined #openstack-infra20:00
*** melwitt has joined #openstack-infra20:00
sdaguewell it's doing a thing20:00
*** cybertron_ has quit IRC20:01
*** dkliban has quit IRC20:01
mordredsdague: that's a start!20:01
*** bnemec has joined #openstack-infra20:02
fungiAlex_Gaynor: depends on the devstack job in question. see openstack-infra/config in modules/openstack_project/files/jenkins_job_builder/config/devstack-gate.yaml20:02
fungiAlex_Gaynor: some are as little as 1.5 hours, some as long as 320:03
jog0is jenkins acting funny today?
openstackgerritSergey Lukjanov proposed a change to openstack-dev/hacking: Extract configs-related code to the configs helper
openstackgerritSergey Lukjanov proposed a change to openstack-dev/hacking: Import exceptions list is now configurable
sdagueso, now we run into the oslo-config doesn't have releases problem20:05
*** nati_uen_ has joined #openstack-infra20:06
sdaguemordred: your strips out this entirely from nova's requirements.txt20:06
mordredsdague: ah. bugger. bug in update.py20:08
jog0fungi: ^20:08
fungijog0: yeah, i'm trying to spot what happened in that console log20:09
sdaguealso, global requirements really should require a higher version of sqla, but that's another matter20:09
fungijog0: is "AttributeError: 'module' object has no attribute 'DeprecatedOpt'20:09
fungijog0: typical?20:09
jog0fungi: ahh I see20:10
pleia2fungi: ok, so did we want to include package gcc for jeepyb module or just git.pp manifest (having gcc installed is not a bad thing, and putting it in jeepyb module would prevent this woe in the future)20:10
jog0looks like that is related to oslo.config20:10
*** mgagne1 has joined #openstack-infra20:10
*** mgagne1 has quit IRC20:10
*** mgagne1 has joined #openstack-infra20:10
jog0fungi: at least that is my guess based on, the fact that sdague  and mordred are talking about it20:10
fungijog0: seems likely. i have only been paying partial attention to their discourse, hoping for a tl;dr since i'm also trying to work on stuff20:11
fungipleia2: i think so. if the entrypoints use for jeepyb won't work without pycrypto installed and we want to be able to pip install pycrypto, then gcc is needed20:12
pleia2fungi: thanks20:12
*** mgagne has quit IRC20:12
fungiso the jeepyb puppet module seems like the right place20:13
jog0fungi: makes sense.20:13
jog0sdague: is this the bug you guys are working on
sdagueyes, actually20:13
sdagueis that in the gate?20:13
jog0sdague: yes20:13
sdaguewait... how?20:13
openstackgerritTom Fifield proposed a change to openstack-infra/jeepyb: cease docimpact emails, create bugs only
* jog0 shrugs20:14
SpamapSpleia2: -> #openstack-meeting in case you forgot (TripleO)20:14
jog0thats all I know20:14
pleia2SpamapS: right, thanks20:14
sdagueso I know how we've created, but we're off in crazy land doing bad things to devstack20:15
sdaguemordred: so, fyi, this is the diff of all the projects after the gr update -
mordredsdague: AH -crappit. there's a special case we need to do...20:17
*** mgagne1 has quit IRC20:17
*** mgagne has joined #openstack-infra20:17
*** mgagne has quit IRC20:17
*** mgagne has joined #openstack-infra20:17
mordredsdague: OR - we need to limit this just to requirements.txt and ignore test-requirements.txt for now20:18
mordredsdague: otherwise, hacking is going to screw people20:18
sdagueactually -
sdaguemordred: we don't run hacking on devstack-gate nodes for now20:19
sdaguethe second one is easier to read20:20
mordredsdague: yah - I meant for auto-patch submission20:20
sdagueoh, yeh20:20
mordredsdague: I think we may need to start with just syncing requirements.txt20:20
sdaguewell, if we solve the devstack gate, it's one step closer20:20
clarkbok back from lunch20:21
openstackgerritMonty Taylor proposed a change to openstack/requirements: Remove d2to1 from list
sdagueand I think your only issue there is oslo20:22
mordredsdague: btw ^^ - your diff reminded me of that20:22
openstackgerritChris Jones proposed a change to openstack-infra/gitdm: Add to the HP group.
mordreddhellmann: I think we're good to release a new pbr which should unscrew virtualenvwrappers yeah?20:28
*** pabelanger has joined #openstack-infra20:30
jeblairmordred, clarkb, fungi: i just sent email to openstack-dev nominating 4 new jjb reviewers20:31
* fungi skips ahead in his e-mail20:31
* clarkb reads20:31
*** w_ is now known as olaph20:32
jeblair(one of whom i believe is on vacation)20:32
openstackgerritA change was merged to openstack-infra/gitdm: Add to the HP group.
clarkblooks like I haven't gotten it yet20:32
jeblairwell, i did _just_ send it.  :)20:32
*** russellb has quit IRC20:35
openstackgerritJames E. Blair proposed a change to openstack-infra/reviewstats: Use current username as default
jeblair(^ related :)20:36
openstackgerritElizabeth Krumbach Joseph proposed a change to openstack-infra/config: Add gcc package dependency to jeepyb module
*** thingee_ is now known as thingee20:37
openstackgerritClark Boylan proposed a change to openstack-infra/gear: Use deques instead of lists for gear job queues.
clarkbjeblair: ^ completely untested20:37
*** russellb has joined #openstack-infra20:39
*** ladquin has quit IRC20:39
jeblairclarkb: neat :)20:39
*** dkliban has joined #openstack-infra20:41
fungii've not played with those yet. nifty!20:41
reeddo you guys think we can disable the OpenStack list on launchpad? will it keep the archives if deactivated?20:41
clarkbjeblair: fungi: the underlying deque implementation is a double linked list rather than the dynamic sized arrays used for list()20:41
jeblairclarkb: okay, i'll test it for you if you can help me with something:20:42
clarkbjeblair: fungi: that makes it really fast for operating on the ends and memory use should be proportional to the number of elements20:42
clarkbjeblair: sure20:42
jeblairclarkb: i would like to run testr not in a virtualenv.  that is not covered by Zuul's TESTING file.20:42
fungiclarkb: some bits of deque are python 2.7 only (maxlen, reverse, count)20:42
fungiclarkb: though i doubt we're targeting 2.6 at all20:43
clarkbfungi: thats fine, we aren't using them atm20:43
clarkbjeblair: do you want me to test and document that process?20:43
fungieven better20:43
clarkbjeblair: I can do that20:43
jeblairclarkb: sure20:43
jeblairclarkb: it's the only way i've ever tested gear -- 'python develop' it and then run zuul's unit tests outside of a venv.  but then zuul moved to testr and i don't know how to do that.20:44
clarkbjeblair: is the not in virtualenv impotant? or do we just need direct testr running incantions20:44
jeblairrunning the command in tox.ini does not work.20:44
clarkbjeblair: I see20:44
jeblairclarkb: another way of asking my question is "why does 'python testr' not work outside of a venv?"20:45
clarkbjeblair: can you source .tox/py27/bin/activate for zuul, cd to gear, python develop, then testr run --parallel back in the zuul dir?20:45
jeblairclarkb: (since that's what tox is supposed to be running)20:45
jeblairclarkb: i could probably do that.20:45
mordredit sohuld work20:45
mordredjeblair: do you have pbr and testrepository both installed outside of your venv?20:46
clarkbjeblair: and if you do they will need to be new versions20:46
mordredjeblair: pbr is the thing that provides the testr command now20:46
mordredjeblair: also- you can just run "testr run --parallel" which is what I do20:47
jeblairthat is likely the problem.20:47
jeblairmordred: oh, yeah, that just kept saying AttributeError: 'module' object has no attribute 'StringStream'20:47
jeblairover and over20:47
mordredbecause that way you're actually using testr ui and stuff20:47
jeblairclarkb: i will try to muddle through the venv.20:47
clarkbjeblair: ok, let me know how that goes20:48
*** bnemec has quit IRC20:48
mordredjeblair: and honestly, just fixing whatever's weird with your system testrepository install would potentially give you happy making in the actually using testr directly20:49
*** bnemec has joined #openstack-infra20:50
jeblairmordred, clarkb:
mordredjeblair: the testr command's main feature is that it runs testr init in a directory for you if there is not .testrepository dir there - which when we put the shell commands in to tox.ini for that got awkward with warnings about commands not in venvs20:50
mordredjeblair: that's completely expected20:51
mordredjeblair: you have not installed any of the test-requirements.txt files20:51
jeblairi see.  why doesn't 'develop' do that?20:51
mordredbecause setuptools' develop does not install the contents of tests_require20:52
*** amotoki has joined #openstack-infra20:52
clarkbwait what20:52
mordredalthough, now that you bring it up - it might be a nice feature to add20:52
mordredI'd need to think about it for a long time20:52
clarkbthat seems broken20:52
mordredor, maybe, but that's how it works20:52
reedlooks like we can deactivate the old General mailing list on launchpad20:52
mordredthe only setuptools command that installs the contents of tests_require20:53
mordredis python test20:53
mordredwhich itself _should_ work now in latest pbr20:53
reedanybody objects with me deactivating it?20:53
mordredreed: do it20:53
* reed do it20:53
jeblairmordred: i assumed it would work because i thought: "install is to installation requirements as develop is to development requirements"20:53
*** bnemec has quit IRC20:54
jeblairjust giving new-user feedback here.  :)20:54
mordredjeblair: yes. I believe that that makes perfect sense. sadly, sense doesn't come into the picture here nearly as often :)20:54
*** SergeyLukjanov has quit IRC20:54
mordredjeblair: amazingly enough - this isn't even a pbr thing - although I'm totally going to put your feature request on the TDL20:54
mordredjeblair: because I agree with you20:54
fungireed: does that preserve the archive (or did we import a copy of the old archive too)?20:54
reedfungi, too late20:54
reedthe archives are preserved20:55
reedwe haven't imported them yet20:55
jeblairreed: there may be new user (getting started?) directions in the wiki that say join the lp group20:55
*** bnemec has joined #openstack-infra20:55
reedjeblair, ttx edited that yesterday20:56
jeblairreed: awesome.  i'm behind, but still trying to be helpful.  :)20:56
*** nati_uen_ has quit IRC20:56
reedI and Tom are going through the wiki, the man pages in search of references to lists.launchpa.net20:56
jeblair(that's after 'pip install -r test-requirements.txt)20:58
*** nati_uen_ has joined #openstack-infra20:58
mordredjeblair: and this is in a fresh vm?20:58
jeblair(after the 2 develop commands from earlier)20:58
jeblair'python testr' seems to work20:59
reed still there21:01
anteayais there a page that allows me to see the current open meeting slots for -meeting and -meeting-alt?21:01
*** vipul is now known as vipul-away21:01
anteayaI am here:
jeblairclarkb: 'python testr' says: FAILED (id=238, failures=14 (+14))21:01
jeblairclarkb: how do i find out which 14 tests failed?21:01
clarkbjeblair: above that it should've spat out the exceptions for each of the failing tests21:03
clarkbjeblair: you can also run `testr last` to have it spit out that info again21:03
clarkbjeblair: you can also open .testrepository/238 and look in the subunit directly21:03
jeblairclarkb: it spat out the logs, so the exceptions are burieed in thousands of lines of logs21:03
clarkbjeblair: `testr failing`21:04
clarkbjeblair: the --list option to that command may be useful21:04
jeblairclarkb: same thing.  i can see the last one, so i guess i can look at it one at a time and work backwards.21:04
jeblairclarkb: that's it, thanks.21:04
*** vipul-away is now known as vipul21:05
jgriffithsdague: Just noticed something interesting maybe you have some insight...21:08
jgriffithsdague: got a minute?21:08
clarkbRyan_Lane: can you create the NOTES file that I mention in the new wiki documentation when you get a chance?21:12
openstackgerritJames E. Blair proposed a change to openstack-infra/zuul: Use the public gear API to iterate over jobs in the queue
jeblairclarkb: ^ with that change in place, zuul tests pass using your gear change21:12
clarkbjeblair: the other thing we should test before merging my change is whether or not adding 500k jobs of semi substantial size then removing all of them causing the memory use to shrink21:13
clarkband compare that against the list implementation21:13
jeblairclarkb: +121:14
jeblairclarkb: i'll leave that for you, if that's ok21:14
clarkbjeblair: ya thats fine21:14
clarkbI will wip the change21:14
* reed ranting: BI is hard...hard...hard21:15
jeblairthe zuul change should be safe to merge before then, as the getQueue method returns a copy of the queue(s) regardless21:15
openstackgerritKhai Do proposed a change to openstack-infra/config: enable access for openstack-ci-core
clarkbjeblair: I WIPed it. Will work on modeling the resource usage21:16
*** woodspa has quit IRC21:18
*** mindjive1 has joined #openstack-infra21:18
*** gaelL has quit IRC21:18
*** mindjiver has quit IRC21:19
*** gaelL has joined #openstack-infra21:20
dhellmannmordred: yes, please, for a pbr release21:21
openstackgerritJames E. Blair proposed a change to openstack-infra/zuul: Stats: break out job runtimes and counts by result
openstackgerritJames E. Blair proposed a change to openstack-infra/zuul: Pass the Job to the parameter function
clarkbasking a gearman client to submit 300k jobs in a tight look takes some time >_>21:26
jgriffithdkranz: ping21:27
jeblairclarkb: the gearman client is probably going to use ram in a similar way; it maintains a list of associated jobs, so it may need a similar change (can be separate from the server change)21:28
clarkbjeblair: ok21:29
clarkbmy poor laptop is having a fun time blasting all this info over lo right now21:30
jeblairclarkb: (to a lesser extent, so do the workers, but that's scaling to "number of jobs running" not "number of jobs queueing" which is what you're focusing on)21:30
jeblairclarkb: (ditto for the server's list of running jobs)21:31
clarkbyeah running jobs I am not too worried about21:31
jeblairyeah, just mentioning it for completeness.21:31
*** sandywalsh has quit IRC21:31
clarkbit would be cool to have several hundred thousand workers21:32
clarkbwe should build a botnet21:32
dkranzjgriffith: pong21:32
jgriffithdkranz: about
uvirtbotLaunchpad bug 1202254 in cinder "Flakey test: VolumesActionsTest" [High,Confirmed]21:35
jgriffithdkranz: I haven't gotten to root cause of what's failed but I see what's going on21:35
dkranzjgriffith: OK21:35
jgriffithdkranz: we only handle a limited set of cases for delete21:35
jgriffithdkranz: something fails and when teardown tries to delete we hit the unhandled exception21:36
jgriffithdkranz: I'm patching cinder to give more info here (exactly what the state is)21:36
dkranzjgriffith: Sounds good21:36
jgriffithdkranz: and then I'll have a patch for tempest to catch this and handle it better21:36
dkranzjgriffith: Looking forward to it. Let me know if I can help.21:37
jgriffithdkranz: thx21:37
*** ^demon has quit IRC21:37
*** ^demon has joined #openstack-infra21:38
*** ^demon has joined #openstack-infra21:38
Ryan_Laneclarkb: will do21:38
*** bnemec has quit IRC21:40
* fungi will be back after he gets some food in him21:40
*** bnemec has joined #openstack-infra21:41
*** mrodden has quit IRC21:42
*** dina_belova has quit IRC21:44
*** sandywalsh has joined #openstack-infra21:44
*** dkranz has quit IRC21:44
*** cp16net is now known as cp16net|away21:46
*** cp16net|away is now known as cp16net21:47
jeblairmordred: ping21:50
mordredjeblair: pong21:50
jeblairmordred: we have spoken about adding support for multiple triggers to zuul.  do you have code?21:50
* mordred has blurred vision from all of his phone calls21:50
mordredjeblair: not that works21:50
jeblairmordred: i may do some work in that direction in order to support timed jobs21:51
mordredjeblair: I can pastebin what I have so far - but it's probably not _that_ interesting21:51
jeblairmordred: if it's easy, i'll take a look.  i'm happy to build on what you have, or just implement in the direction we discussed otherwise.21:52
mordredthat's what I had so far21:52
mordredjeblair: but I think that's about when I realized I'd need to do something with extending the config, and then the weekend was over21:53
mordredjeblair: so feel free to either use or ignore21:53
jeblairmordred: cool, i'll incorporate that.21:53
clarkbjeblair: does it make sense to have a submit job utility for zuul (like the existing one) that you can simply put in cron?21:59
*** hub_cap has joined #openstack-infra21:59
jog0have a mailman question21:59
hub_caphey what OS are the slaves that you spin up for tox runs? /me thinks they are rpm based?21:59
jog0is there a shorter URL for ?21:59
jog0as that seems a little long for something we want to put in docs21:59
*** nati_ueno has quit IRC22:00
clarkbhub_cap: for everything but python26 jobs it is ubuntu precise. for python26 jobs it is CentOS 6.422:00
jeblairclarkb: jog0 not that i'm aware of22:00
jeblairjog0: not that i'm aware of22:00
hub_capAh ok its py26 thats a'failin for me22:00
hub_capthx clarkb <322:00
jeblairclarkb: i don't really want to configure the periodic jobs that way.  seems pretty hackish.22:00
*** rnirmal has quit IRC22:01
jeblairclarkb: (and it won't solve the logstash issue)22:01
clarkbjeblair: hmm good point22:01
clarkbactually the submitting job could have a LOG_PATH22:02
clarkbI bring it up because using the gear queue in that way makes me happy. Sure zuul can do the magic for you but one of the nice things about gearman is you don't need much magic22:02
jeblairclarkb: what would the log path be?22:03
clarkb/periodic/job_name/$TIMESTAMP or you could generate a UUID to be consistent with the non periodic jobs22:04
jog0jeblair: thanks22:04
*** vipul is now known as vipul-away22:04
jeblairclarkb: do you feel that timed triggers don't belong in zuul?22:05
jeblairi think it makes a lot of sense to be able to describe all of the jobs that we run related to given projects in layout.yaml22:06
clarkbjeblair: I agree and having zuul support timed jobs so that all jobs are described in one place is a good thing22:06
jeblairi think asking people to edit that is much nicer than saying "oh, the periodic jobs?  edit the crontab that gets installed on this host which triggers the jobs behind the scenes"22:07
*** pentameter has quit IRC22:07
clarkbjeblair: but until that feature is added zuul has been a gerrit event based system. So scope is expanding22:07
jeblairclarkb: if i don't have working code by the end of today, i'll do your idea.  :)22:08
*** burt has quit IRC22:08
*** vipul-away is now known as vipul22:08
jeblair(at least, a first cut)22:08
*** woodspa has joined #openstack-infra22:15
*** _TheDodd_ has quit IRC22:16
*** rfolco has quit IRC22:17
*** zehicle has quit IRC22:23
*** derekh has quit IRC22:24
reedjeblair, I'm having trouble extracting only the new ATCs from the new list you sent me22:26
jeblairreed: 1 sec22:26
jeblairreed: i can get you a diff in about 15 mins22:28
Alex_Gaynorfungi: Does devstack-grenade not use the same devstack things?22:29
jeblairreed: and i'll add that to the list of things the automated job should do22:29
*** vijendar has quit IRC22:29
reedjeblair, cool, thanks22:30
reedone reason why I'm confused is that some ATCs have disappeared from the new list... but now that I think about it, it may be normal: their contribution is older than XX months today, wasn't older 1 month ago22:31
reedand some have just changed/added more emails to their identity22:32
jeblairreed: hrm, i don't think that should be the case because i didn't change the start date for the range22:32
*** prad_ has quit IRC22:32
reeduhm ... maybe my diff skills are bad and I'm not seeing something22:33
jeblairreed: 126 added, 17 removed22:33
jeblairreed: let me look at the removed and see why22:33
jeblairreed: (i know one off the top of my head -- a user rename, which is what i'm keying off of because it's the most stable thing in the diff)22:34
*** gyee has joined #openstack-infra22:34
jeblair(later, we should add the gerrit userid in there so it's more stable)22:34
*** bnemec has quit IRC22:34
reedwhat ids are those? launchpads?22:34
jeblairreed: gerrit username22:34
*** bnemec has joined #openstack-infra22:35
*** rcleere has quit IRC22:35
*** w_ has joined #openstack-infra22:36
*** olaph has quit IRC22:38
jeblairreed: i think i made an error running the script and missed a project; re-running.22:39
*** krtaylor has quit IRC22:39
reedjeblair, thanks22:39
*** dina_belova has joined #openstack-infra22:44
*** bnemec has quit IRC22:45
*** bnemec has joined #openstack-infra22:46
*** lcheng has quit IRC22:46
*** lcheng has joined #openstack-infra22:47
*** esp has joined #openstack-infra22:49
*** dina_belova has quit IRC22:49
esphello folks we ran across an issue in trove land recently involving package versioning.22:49
espok for me to give details here?22:50
clarkbAlex_Gaynor: grenade uses devstack as a library but not as an executable22:50
clarkbAlex_Gaynor: so the bits are the same but composed differently22:51
clarkbesp: yup, go for it22:51
espclarkb: thx22:51
Alex_Gaynorclarkb: So earlier today we landed a thing so devstack installs libffi-dev, this doesn't seem to result in it being present in a devstack-grenade test run. Is tehre somethign else that neesd t obe edited?22:51
espso in the file above: _get_revno(git_dir):22:51
esp    """Return the number of commits since the most recent tag.22:51
clarkbAlex_Gaynor: for that I am not sure, I would expect grenade to install dependencies using the devstack bits22:52
espthis creates a packager version like this: trove-2013.2.a5.ge9d559d22:52
clarkbAlex_Gaynor: however that may be different because it is starting with grizzly devstack and going to master22:52
mordredthat is correct22:52
*** sarob has joined #openstack-infra22:53
esphowever the previous package version was: trove-2013.2.a716.<git tag>22:53
mordredwell, no22:53
mordredthe previous veroisn should have been trove-<git tag>22:53
mordredif there was a tag on the version22:54
espk, but do you know where the a<number> comes from?22:54
*** dina_belova has joined #openstack-infra22:54
mordrednumber of commits from most recent tag22:54
mordredthis will not necessarily be monotonically increasing22:55
mordredbecause if you have an H2 tag in the middle of teh process, it will reset the commit count22:55
*** datsun180b has quit IRC22:56
mordredare you generating debs or rpms automatically from the tarball and you're having problems upgrading?22:56
jeblairreed: new total 879; 129 added, 1 removed (due to an account rename)22:56
Alex_Gaynorclarkb: hmm, do you have any idea who'd be a good person to speak with?22:56
reedjeblair, sounds good22:56
clarkbjeblair: I haven't been able to convince myself that the switch to deques helps anything. my test server isn't reclaiming memory either22:56
espah. so that's what happened, we must have set an H2 tag to mark a milestone or something22:56
jeblairreed: turns out we renamed reddwarf to trove since i last ran.22:56
clarkbAlex_Gaynor: dtroyer22:56
mordredesp: yes. that in fact most likely happened22:56
hub_capsweet mordred that makes sense22:56
mordredesp: there is an env var you can set "PBR_VERSION"22:56
mordredwhich will override the version22:56
espmordred: cool22:56
mordredso that if you're trying to do auto-upgrades from debs or something22:56
mordredyou can make it be what you need it to be22:57
Alex_Gaynordtroyer: So we added a package to things to be installed in devstack by adding it to files/apt/swift; however it doesn't seem to be installed in the devstack-grenade builds. WHat do I need to do to make sure it's installed there as well22:57
Alex_Gaynorclarkb: thanks22:57
esphub_cap: nice, was gonna set to 1.5 million22:57
vipulthe recent package in tarballs.o.o is versioned as '.b2' how did it change to 'bXX'22:57
mordredesp, hub_cap: make sense?22:57
hub_capaye mordred22:57
*** emagana has quit IRC22:57
mordredvipul: b2 was part of the tag22:57
espmordred: yeah I think so.  sounds like we just need to set the var to something more predictable.22:57
vipulah ok22:57
mordredvipul: this is related to the very complex system of pre-release versioning we employ22:57
mordredwhich is versioning for a thing we aim to release that will be versioned 2013.222:58
*** woodspa_ has joined #openstack-infra22:58
mordredI promise, there are WAY more moving parts in there than you ever want to know about ...22:58
* mordred head hurts22:58
vipuli can imagine :)22:58
espk, thx for the clarification everyone!22:59
mordredesp: sure thing!22:59
*** dina_belova has quit IRC22:59
mordredclarkb: also, once jeblair does the thing he does, we should also be able to make a github PR trigger22:59
mordredclarkb: which we can then use to run tests on pull requests to pip22:59
* hub_cap is glad we dont need an explaination of mordred's moving parts22:59
mordredclarkb: and inform them that they're about to break openstack22:59
jeblairreed: email sent.23:00
mordredclarkb: so it is a scope increase - but I think gets us down the road of awesome23:01
*** woodspa has quit IRC23:02
clarkbmordred: cool23:02
clarkbfungi: jeepyb on review.o.o looks happy now. did you kick it?23:02
*** emagana has joined #openstack-infra23:04
fungiclarkb: i did not kick jeepyb on review.o.o but i did fix the puppet agent which had hung for a couple days23:05
clarkbwe should consider restarting mysqlds
clarkbfungi: wow I should've totally checked that23:05
fungiclarkb: had to do the same for lists.o.o and groups.o.o23:06
fungiclarkb: as well as a couple of precise slaves and a centos6 slave this morning23:06
fungi(various causes)23:06
*** ladquin has joined #openstack-infra23:07
*** zehicle has joined #openstack-infra23:09
clarkbfungi: so it looks like jeepyb's git repo updated after kicking puppet but to actually upgrade to a new version we need your puppet fix23:10
fungiclarkb: yes, that is consistent with my findings for the past several weeks23:10
fungipuppet agent updates via vcsrepo, but the install exec fails because of a broken pbr installed on the server23:11
fungiupgrading that pbr gets it out of the cycle23:11
clarkbmordred: jeblair can haz review?23:12
*** bnemec has quit IRC23:12
fungigetting it to upgrade pbr automagically as part of the install when a broken pbr is already in the lib path is what we were unable to find a good solution for23:12
*** bnemec has joined #openstack-infra23:13
fungiclarkb: jeblair: lgtm. are we ready to merge that (i'll be around for at least another several hours to keep an eye on it) or do you want to wait?23:14
jeblairfungi: that won't work until the change to zuul is merged and zuul is restarted.  and somehow that's missing a changed file.23:16
jeblairand no one has reviewed the change to zuul23:17
* clarkb reviews23:17
jeblairfungi: i'm terribly sorry, but i'm slightly confused by the puppet jeepyb pbr change23:18
jeblairfungi: i recall that we talked about it23:18
jeblairfungi: but i'm not clear on whether your change is intended to be permanent23:18
*** pcrews has quit IRC23:18
*** emagana has quit IRC23:19
*** emagana has joined #openstack-infra23:20
*** nati_ueno has joined #openstack-infra23:20
fungijeblair: i don't think i have the wherewithall to figure out how to get pbr to cope with an already-installed broken version of itself. the theory we had about it being caused by a cached egg turned out to be false23:22
jeblairfungi: so here's the thing: i understand things are broken, and upgrading pbr fixes them.23:23
mordredjeblair: yes. we explored the other options since that discussion. at the moment, I believe we shoudl do this until further notice23:23
clarkbjeblair: if I approve does that conflict with any of your changes?23:23
jeblairfungi: in that case, i believe we should ssh into the affected hosts and upgrade pbr23:23
mordredoh - hrm. that's an interesting idea...23:23
jeblairbut i'm not comfortable abandoning "python install" as an installation method.  since we use it lots of places.23:24
fungijeblair: i have no qualms with that. 1) i'll want to hunt for other servers where we potentially gotpbr into a simialr state and 2) this doesn't help other people (not us) who may have used out jeepyb puppet module23:24
* mordred believes jeblair is right. the parse error is an error in a version of pbr that's been fixed23:24
* fungi is typing too fast again23:24
jeblairclarkb: should be fine23:25
*** mberwanger has joined #openstack-infra23:25
mordredjeblair: I do not think we have to abandon that method - I think we got a bad version of a thing installed which prevented self upgrading23:25
clarkbjeblair: thanks23:25
fungii was hoping to find an elegant solution to make the puppet module more robust against this (for the sake of others and the future) since pbr seems to have an odd chicken-and-egg usage there23:25
jeblairfungi: i don't want to paper over that kind of problem with pbr23:25
jeblairfungi: we should either help the pbr fix the problems, or stop using pbr for our tools.  :)23:26
fungibut if we're okay with ignoring it and just upgrading pbr manually whenever this happens, it seems unfortunate but maybe not worth spinning our wheels on any longer23:26
jeblairmordred: i favor the first.  :)23:26
clarkbjeblair: hasn't been merged because of this ...23:26
mordredI do not believe there is a way to fix _this_ problem in a seamless manner23:27
jeblairfungi: i'm okay with mordred fixing it whenever this happens, which hopefully will be never again.  :)23:27
mordredI do not believe this is likely to be a type of problem that will come up frequently23:27
* mordred read wrong23:27
mordredI thought that jeblair meant "fix the problem in upstream pbr" by "help the pbr fix the problems"23:27
fungifine by me. i'll abandon that change and just fix the gerrit servers in that case23:27
* mordred goes back to writing a steady stream of emails23:28
fungithough we exposed a similar problem with installing jeepyb via the puppet module on git.o.o which is also fixed by globally installing pbr23:28
fungibecause python install in the jeepyb source tree does not globally install pbr23:28
jeblairmordred: i meant "help the pbr authors fix such problems".  by which i more or less meant yell at mordred to fix pbr.23:28
fungibut jeepyb's entrypoints wrappers do not work without pbr in the lib path23:29
jeblairmordred: i also meant that if this happens again, i'm okay with you sshing into the servers and fixing them, seeing as how we were kind enough to let our servers tell us pbr was broken.23:29
fungiso that also seems like something which will need fixing in pbr if we're going to stop manually installing pbr as a dependency of the jeepyb puppet module23:29
openstackgerritA change was merged to openstack-infra/zuul: Make E123 pep8 clean
jeblairfungi: that makes me sad as well.23:30
jeblairmordred: are you aware of that problem?23:30
fungior possibly just fixing in jeepyb by making pbr a proper runtime dependency?23:30
mordredpbr should be a proper runtime depend, is it not in requirements.txt ?23:31
fungionly setup_requires23:31
mordredoh. that is a bug23:31
mordredit should also be in requirements.txt23:32
fungithis might solve a number of issues in that case23:32
fungii wonder if that could also be causing the issue with it not updating on the gerrit servers?23:32
mordredsorry - never thought to look/ask on that - still would not have fully helped in this case if there was a version which couldn't read the requirments.txt file ...23:32
openstackgerritJeremy Stanley proposed a change to openstack-infra/jeepyb: PBR is a run-time dependency.
*** bnemec has quit IRC23:33
fungiwe'll see how that goes. i'll try it out on review-dev shortly23:33
*** bclifford has left #openstack-infra23:34
funginot getting my hopes up, but should at least solve pleia2's need on git.o.o23:34
* fungi will brb23:34
*** reed has quit IRC23:39
*** dhellmann is now known as dhellmann_23:40
openstackgerritJoe Gordon proposed a change to openstack-dev/hacking: Make H202 work again
*** michchap has joined #openstack-infra23:42
pleia2 coming together for tomorrow23:49
jeblairmordred, clarkb: i have run "tox -epy27"23:51
jeblairmordred, clarkb: it hangs23:51
jeblairmordred, clarkb: do you know how i can try to find out which test is hanging?23:51
mordredfirstly, just to capture it:23:51
mordredpython -m tests.test_scheduler.TestScheduler.test_zuul_refs23:51
mordredwill run that one test23:52
jeblairi don't know which test is hanging?23:52
jeblairi just know that at least one test out of all of them hangs23:52
mordredI know. that was just for the record from last time23:52
mordredpython -m discover23:52
mordredwill run all of the tests in order and not do any log suppression23:52
jeblairmordred: the thing from last time is in TESTING.rst.  thanks for confirming that though.23:53
mordredjust making sure23:53
jeblairmordred: ok, so how do i find out which test is hanging?23:53
mordredpython -m discover23:53
jeblairno i get that23:53
mordredshould run all of the tests in order, printing out test names as it does23:53
mordredpython -m discover --list23:54
mordredwill show you the list of tests23:54
jeblairmordred: what does the output where it prints test names look ilke?23:54
jeblairmordred: i redirected that to a file, and the very first line is a zuul log message.23:55
*** dina_belova has joined #openstack-infra23:55
mordredoh. my bad- one sec23:56
jeblairmordred: presumably "TestScheduler" should show up in that output if it printed a test name, but grepping for that fails23:56
*** esp has left #openstack-infra23:56
jeblair(the output is 15929 lines long)23:56
mordredtry this: python -m discover 2>1 | subunit2pyunit23:57
mordredjeblair: it _will_ supress logs, but you should see test names printing23:57
jeblairmordred: that seems to be working, thanks.23:58
* mordred has an open tdl to put a flag on to testr that will do the equiv23:58
mordredjeblair: oh, and duh - slightly easier to type: 'testr run --subunit | subunit2pyunit'23:59

Generated by 2.14.0 by Marius Gedminas - find it at!