2013-08-20T00:01:14 jlk: our jenkins slaves are good at DDoSing our git server 2013-08-20T00:01:23 jlk: particularly when we point them at git-daemon 2013-08-20T00:01:32 strange. 2013-08-20T00:01:43 but your repos are significantly larger than Fedoras was 2013-08-20T00:01:51 Fedora was thousands of small repos 2013-08-20T00:03:33 Our hits were probably more distributed as well, distributed over time and network capabilities. RHT infrastructure had networking gear in between our servers and the Internet, I don't know what they did for throttling or whatnot 2013-08-20T00:04:09 jlk: did you use xinetd or run git-daemon itself? 2013-08-20T00:04:43 good question! I believe I used whatever was packaged in EPEL 2013-08-20T00:04:58 would have been rhel6 era 2013-08-20T00:08:06 jlk: that's pretty much what we're doing, which ends up being xinetd. so no particular tuning? 2013-08-20T00:08:24 clarkb: any good reason not to pass --events on mysqldump runs? currently cronspamming us about skipping the mysql.event table on each server 2013-08-20T00:09:10 jeblair: not that I remember. 2013-08-20T00:09:46 I think I looked at one time at doing git export to just get the latest bits instead of doing a full clone, or doing shallow clones, on our build server 2013-08-20T00:09:55 because it didn't need any history, just needed the bits 2013-08-20T00:10:06 jeblair: ^^ that's a little bit what I was afraid of - we tend to absolutely slam the cloning infrastructure 2013-08-20T00:10:23 ah, apparently they do use xinetd to throttle it a lot now 2013-08-20T00:10:29 where "it" == anonymous clones 2013-08-20T00:11:22 Clark Boylan proposed a change to openstack-infra/config: Proxy git-daemon with haproxy. https://review.openstack.org/42784 2013-08-20T00:11:24 mordred: not really, we almost never clone 2013-08-20T00:11:36 I can't help myself 2013-08-20T00:12:03 mordred: are you done merging back swift m-p tags to master, or should I keep the m-p branch alive for some more time ? 2013-08-20T00:12:03 that is completely untested but in theory made easy with the puppetlabs module 2013-08-20T00:12:04 ttx: done with it 2013-08-20T00:12:04 mordred: I can delete it now ? 2013-08-20T00:12:04 ttx: also wrote a patch to potentially do it 2013-08-20T00:12:05 ttx: yup 2013-08-20T00:12:12 jlk: aren't all git-daemon clones anonymous? 2013-08-20T00:12:20 mordred: ok, on my way to final cleanup 2013-08-20T00:12:21 ttx: https://review.openstack.org/#/c/41927/ 2013-08-20T00:12:25 well, yes, I'm not sure why I added that bit of data. 2013-08-20T00:12:29 "data" 2013-08-20T00:13:05 clarkb, mordred: http://paste.openstack.org/show/44553/ 2013-08-20T00:13:30 clarkb, mordred: that thread is just sitting there. best i can tell, it's not waiting on a lock. but it is holding one which is blocking everyone else. 2013-08-20T00:13:55 that should be the jjb update that changes the git url. it applied fine on jenkins01 2013-08-20T00:14:09 jeblair: wow. that's stellar 2013-08-20T00:14:40 i'm leaning towards "try to manually kill that thread". any other ideas before i do that? 2013-08-20T00:15:41 jeblair: is it possibly waiting on a locked file? 2013-08-20T00:16:16 *** pcrews has quit IRC 2013-08-20T00:16:20 *** ^demon has joined #openstack-infra 2013-08-20T00:16:20 *** ^demon has joined #openstack-infra 2013-08-20T00:16:48 mordred: there is a corner case in the merge-tags thing 2013-08-20T00:16:51 clarkb: it looks like a runaway regex 2013-08-20T00:17:08 jeblair: I'd chalk that up to "java sucks sometimes" 2013-08-20T00:17:10 ttx: yeah? 2013-08-20T00:17:19 fungi: uh I don't know 2013-08-20T00:17:21 mordred: for stable/* I'm not sure you actually want to merge tags back... do you ? 2013-08-20T00:17:24 * clarkb reads more manpages 2013-08-20T00:17:38 *** nati_ueno has quit IRC 2013-08-20T00:17:42 pleia2: if you are really adventurous I think it would be cool to apply 42784 to your test server if it is still up 2013-08-20T00:18:04 mordred: 'cept gearman-plugin is a few rungs down the stacktrace 2013-08-20T00:18:09 mordred: so it's our fault 2013-08-20T00:18:10 ttx: branch: ^(milestone-proposed).*$ 2013-08-20T00:18:12 mordred: i.e. when we tag 2013.1.3 on stable/grizzly, do we rally want to merge the tags back to havana master ? 2013-08-20T00:18:23 ttx: the job is configured to only run on milestone-proposed 2013-08-20T00:18:40 since that's the only time we ever want to do this 2013-08-20T00:18:51 mordred: at release time we use milestone-proposed too, and turn that into stable/* 2013-08-20T00:18:51 *** ^d has quit IRC 2013-08-20T00:19:04 mordred: thoughts on fungi's --events mysqldump option? 2013-08-20T00:19:10 ttx: but it's milestone-proposed when you make the tag, right? 2013-08-20T00:19:12 mordred: is that table useful or just noise? 2013-08-20T00:19:30 mordred: so we push like, havana-rc2 tags to milestone-proposed while master switched to icehouse 2013-08-20T00:19:33 clarkb: noise. we don't use it 2013-08-20T00:19:40 ttx: yup. that's fine 2013-08-20T00:20:01 mordred: so better to redirect that warning message to /dev/null than to dump the table? 2013-08-20T00:20:03 mordred: ok, just doublechecking 2013-08-20T00:20:21 ttx: we _do_ want the final tag from havana milestone-proposed to be in master, so that the in-flight versions look "sensible" 2013-08-20T00:20:38 but I agree, the following tags that are made on stable/* do not want to be merged to master 2013-08-20T00:21:07 mordred: can that job generate a conflict ? Or is it always successful ? 2013-08-20T00:21:11 ttx: and we're making it always a null-merge, so the merge will never bring changes from m-p to master 2013-08-20T00:21:27 ok, guess that answers my question 2013-08-20T00:21:29 ttx: it's always successful. it's using the merge strategy which says "just keep my version" 2013-08-20T00:21:40 ack 2013-08-20T00:22:23 +1ed 2013-08-20T00:22:24 Is there anythign I could be doing to help with the "ddosing ourselves with git" issue? 2013-08-20T00:23:06 Alex_Gaynor: right now we are switching to using https instead of git:// as apache deals with ddosing ourselves better 2013-08-20T00:23:19 clarkb, mordred: uh, wow, ok, it got unstuck. 2013-08-20T00:23:25 jeblair: wow 2013-08-20T00:23:40 clarkb: "apache deals with ddosing ourselves better", I feel like this encapsulates everything I feel about computering (for better and for worse) :) 2013-08-20T00:23:42 Alex_Gaynor: https://review.openstack.org/42784 is one potential way of moving back to using git:// but it needs testing and probably input from someone that knows haproxy better than me 2013-08-20T00:23:55 clarkb: I can probably ping some HA proxy friends 2013-08-20T00:24:04 Alex_Gaynor: I am semi hoping we can abuse pleia2's test box if it is still around 2013-08-20T00:24:06 seems really strange to make use of https to make things faster... 2013-08-20T00:24:27 IIRC git:// isn't doing any encryption, which /should/ make it an easier process to handle. 2013-08-20T00:24:31 Alex_Gaynor, jlk: basically, git under xinetd has no socket queueing, so you're either under the 50 process limit, or over, in which case you get your connection dropped 2013-08-20T00:24:41 interesting 2013-08-20T00:24:53 Alex_Gaynor, jlk: apache at least will let you separately tune how many things you run, vs how many things you queue 2013-08-20T00:24:58 and if we increase the connection limit we end up hitting cpu and disk hard 2013-08-20T00:25:05 nod 2013-08-20T00:25:10 Is there anything we can point at github? 2013-08-20T00:25:12 so we can set a reasonable number of processes to run at once, and a larger queue 2013-08-20T00:25:14 let them deal with the problem 2013-08-20T00:25:17 Alex_Gaynor: hehehe 2013-08-20T00:25:22 Alex_Gaynor: that's funny 2013-08-20T00:25:24 Alex_Gaynor: that's been our strategy up to this point 2013-08-20T00:25:40 they appear to be moving away from git:// as much as they can 2013-08-20T00:25:47 mordred: Now the only failure with VIRTUAL_ENV is grenade... not sure how to fix it though, since we're explicitely trying to perform an upgrade it sounds like it might be more difficult than I'd hope... 2013-08-20T00:25:58 but that might just be because they can stick all sorts of tracking around http usage that they can't w/ git:// 2013-08-20T00:26:05 BobBall_Away: I think we just may need to do similar work there 2013-08-20T00:26:08 Alex_Gaynor: github still fails quite often, enough for our automagic to notice 2013-08-20T00:26:16 BobBall_Away: or backport some of the changes to devstack stable/grizzly 2013-08-20T00:26:21 BobBall_Away: but that's thrilling! 2013-08-20T00:26:22 mordred: effectively the error seems to be it's running in the venv but things (such as pip) haven't been installed in it 2013-08-20T00:26:27 Alex_Gaynor: (i should say partial strategy -- we haven't used github in tests for a long time, but we still use it for cronjobs, etc) 2013-08-20T00:26:46 BobBall_Away: I'm going to run out fora second, I'll look at grenade when I get back 2013-08-20T00:26:48 very thrilling 2013-08-20T00:26:51 I'm going to bed now 2013-08-20T00:26:53 I think Fedora infrastructure also has multiple front ends for git 2013-08-20T00:26:57 that use a shared FS 2013-08-20T00:26:59 BobBall_Away: thanks for your help! 2013-08-20T00:27:02 it's 1:30am and I've had enough :D 2013-08-20T00:27:02 use a CDN 2013-08-20T00:27:02 ! 2013-08-20T00:27:03 not positive though 2013-08-20T00:27:21 dstufft: doing invalidation on a CDN'd git repo sounds awful 2013-08-20T00:27:31 yikes 2013-08-20T00:27:36 * mordred has a hunch multiple servers is going to wind up being in the cards eventually 2013-08-20T00:27:41 Alex_Gaynor: I dunno sounds like it wouldn't be that bad actually 2013-08-20T00:27:57 dstufft: I'm not aware of any git CDN's 2013-08-20T00:28:13 lifeless: if you run git over HTTP(S) you can just use any HTTP pass-through CDN 2013-08-20T00:28:16 lifeless: the http stuff should CDN just fine 2013-08-20T00:28:19 mordred: yep. i just want it to be multiple good servers. 2013-08-20T00:28:28 Alex_Gaynor: clarkb: yeouch. No. Thanks. 2013-08-20T00:28:34 multiple servers seems easy for read-only support. it's the read/write that's hard with a load balancer 2013-08-20T00:28:45 master/slave git 2013-08-20T00:28:49 jlk: we don't need read/write 2013-08-20T00:28:54 we have a single writ emaster 2013-08-20T00:28:57 which is gerrit 2013-08-20T00:28:58 and I really didn't want there to be two vastly different URLs for read-only clone vs write clone 2013-08-20T00:29:00 jlk: we are in the fortunate position of only needing to consider read-only mirrors here 2013-08-20T00:29:00 which replicates to things 2013-08-20T00:29:02 *** nati_ueno has joined #openstack-infra 2013-08-20T00:29:08 mordred: oh right, that makes things a lot easier for you 2013-08-20T00:29:09 yup 2013-08-20T00:29:13 Alex_Gaynor: clarkb: I presume you are aware of the way plain HTTP with git (and basically all VCS's) works, right ? 2013-08-20T00:29:43 Alex_Gaynor: clarkb: or perhaps I should say, I presume you *aren't* aware, or you wouldn't suggest a CDN be a good fit. 2013-08-20T00:29:49 pretend network latency doesn't exist and just fetch some files ? :V 2013-08-20T00:30:38 dstufft: thats part A of the terror. part B is to either do readv's, or to sporadically download the entire repo all over again, due to the rebalancing of 'pack' operations 2013-08-20T00:30:47 Clark Boylan proposed a change to openstack-infra/config: Make mysql backup crons quiet. https://review.openstack.org/42785 2013-08-20T00:30:57 fungi: mordred ^ that should make mysqldump cronspam less annoying 2013-08-20T00:31:39 you can probably run multiple git slaves and just front it with haproxying proxying streams around, the only hard part would be determining if an incoming stream is read or write, if there's something obvious in the cnnect that lets you know if something is authentcated you can just shove all authenticated at the master and anonymous at the read slaves 2013-08-20T00:31:39 lifeless: if the repo hasn't changed the packs stay the same 2013-08-20T00:31:57 dstufft: all streams are read. :) 2013-08-20T00:31:59 lifeless: and iirc for large repos like nova you end up with several static packs as git leaves old stuff alone 2013-08-20T00:31:59 for us 2013-08-20T00:32:03 dstufft: I don't think we have to worry about writes, everything is a read 2013-08-20T00:32:08 dstufft: only gerrit has write access 2013-08-20T00:32:13 if everything is read then that's even easier 2013-08-20T00:32:47 just use haproxy as a TCP load balancer 2013-08-20T00:33:03 use whatever protocol you want, http, git, ssh, doesn't matter 2013-08-20T00:33:04 dstufft: https://review.openstack.org/42784 2013-08-20T00:33:09 dstufft: that's what clarkb was looking in to earlier 2013-08-20T00:33:36 wtf is a pp file 2013-08-20T00:33:44 dstufft: puppet 2013-08-20T00:33:46 oh 2013-08-20T00:34:04 dstufft, jlk, Alex_Gaynor: so here's the thing -- we spun up a 30g, 8vcpu cloud server for this, and ddosed it with jenkins (it's arguable whether it performed better or worse than the http setup we have on review.o.o) 2013-08-20T00:34:29 that seems really bizarre, unless you're working with huge repos 2013-08-20T00:34:37 you mean the haproxy solution? 2013-08-20T00:34:42 we have a LOT of activity :) 2013-08-20T00:34:44 dstufft: mordred that is a first stab at using haproxy to do queing but it can be grown to handle mutliple servers 2013-08-20T00:34:48 dstufft, jlk, Alex_Gaynor: before we spin up an army of maxsize(rackspacecloudservers) for this, i figure a little thought and testing of the tuning of one server might be in order. 2013-08-20T00:34:50 clarkb: ports => '29418', ? 2013-08-20T00:35:01 jeblair: so, suggest from a friend of mine "instances=32" 2013-08-20T00:35:08 jeblair: for xinetd 2013-08-20T00:35:10 oh you were just shoving a bigger server at it 2013-08-20T00:35:19 I assume this forks 32 processes to handle requests 2013-08-20T00:35:20 clarkb: it tries to accomodate things yes, which makes the behaviour worse, because you get sporadic 'wtf is it doing' when it has to suck down the entire history again. 2013-08-20T00:35:20 dstufft: mordred or maybe we use lbaas to do handle multiple services and keep the local haproxy for queueing 2013-08-20T00:35:23 mordred: does all that activity require a full clone of the repo? 2013-08-20T00:35:38 what does rackspace have for HD's 2013-08-20T00:35:39 *** dina_belova has joined #openstack-infra 2013-08-20T00:35:44 *** rfolco has joined #openstack-infra 2013-08-20T00:35:44 Alex_Gaynor: we currently have the default of 50. 2013-08-20T00:35:58 spinning up more processes won't help if you're IO bound 2013-08-20T00:35:59 mordred: haproxy will listen on 9418 so I stuck gitdaemon on the alternate that gerrit uses 2013-08-20T00:36:12 clarkb: ahhhh 2013-08-20T00:36:20 clarkb: I agree with jeblair - let's see what a local haproxy queue will do to it 2013-08-20T00:36:27 before we start adding in multi-machine lbaas 2013-08-20T00:36:32 mordred: definitely 2013-08-20T00:36:34 but potentially yes 2013-08-20T00:36:48 i think we ought to do some real performance testing too 2013-08-20T00:36:51 where was the bottleneck? 2013-08-20T00:36:53 *** coderanger has joined #openstack-infra 2013-08-20T00:36:58 where we figure out where the bottleneck actually is :) 2013-08-20T00:37:00 Alex_Gaynor: Fine :P 2013-08-20T00:37:10 coderanger knows how haproxy works and junk 2013-08-20T00:37:23 and what kind of throughput we can get under different configurations 2013-08-20T00:37:27 *** mriedem has joined #openstack-infra 2013-08-20T00:37:35 coderanger: tl;dr; too many things trying to get stuff from git == ddosing ourselves 2013-08-20T00:37:43 yeah, curious where the bottleneck is. Disk, or CPU, or network 2013-08-20T00:37:46 coderanger: Alex_Gaynor https://review.openstack.org/#/c/42784/1/modules/cgit/manifests/init.pp is the important file 2013-08-20T00:37:49 I think before you go changing your configs around you should figure out the bottleneck 2013-08-20T00:37:59 So cranking down maxconns won't buffer connections like it says in the review comment, it will just leave the socket in the listen queue 2013-08-20T00:38:00 because that's going to influence what the solution is a lot :V 2013-08-20T00:38:40 So if you are getting backed up, you are just going to end up with the kernel refusing conns 2013-08-20T00:38:40 coderanger: "anything behind that will queue" is what the commit message says. Is that completely wrong? 2013-08-20T00:38:45 ah 2013-08-20T00:38:49 well that doesn't help 2013-08-20T00:39:01 I mean if can smooth out spikes 2013-08-20T00:39:06 *** michchap has joined #openstack-infra 2013-08-20T00:39:11 Up to whatever you max fds is 2013-08-20T00:39:32 coderanger: spikes are the current issue. Our jenkins slaves are a thundering herd 2013-08-20T00:39:42 Do you know the magnitude? 2013-08-20T00:39:45 coderanger: we need a semi deterministic way of making them wait in line if necessary 2013-08-20T00:40:11 #status ok 2013-08-20T00:40:11 *** ChanServ changes topic to "Discussion of OpenStack Developer Infrastructure | docs http://ci.openstack.org | bugs https://launchpad.net/openstack-ci/+milestone/grizzly | https://github.com/openstack-infra/config" 2013-08-20T00:40:12 *** dina_belova has quit IRC 2013-08-20T00:40:31 clarkb: If thats the way you want to go, make sure you set the backlog param in haproxy too :) 2013-08-20T00:40:53 coderanger: absolute worst case is something like ~300 connections all at once based on the number of slaves we have 2013-08-20T00:41:02 + some fudge for random people using it too 2013-08-20T00:41:14 Ahh okay, for 300 conns thats fine as long as you know you can clear them 2013-08-20T00:41:21 Do the slaves retry on failure? 2013-08-20T00:41:35 coderanger: they do not, and that may help a little but not fix the problem 2013-08-20T00:41:35 If so, you can also just set the xinetd instances=32 2013-08-20T00:41:46 or probably do that anyway jut for safety :) 2013-08-20T00:42:38 Any reason to not use Jenkins' "hash" support in the scm config? 2013-08-20T00:42:48 Thats been the default for a while now for exactly this reason 2013-08-20T00:42:56 coderanger: we don't really use the scm plugin for this 2013-08-20T00:43:36 coderanger: because it has been useless for a long time. I believe mordred helped make it better but we tried switching to it and didn't for some other reason 2013-08-20T00:43:36 mordred: jeblair do you remember why we stuck with g-g-p? 2013-08-20T00:43:42 Ahh, manual build kickoff times every slave trying to pull down code? 2013-08-20T00:43:43 clarkb: because it has a nice echo statement 2013-08-20T00:43:50 less work for jenkins to attempt to do 2013-08-20T00:44:02 coderanger: yeah, we 'manually' run 400-600 jobs per hour 2013-08-20T00:44:32 coderanger: obviously it's not manual, but that's the way jenkins sees it; they're triggered by a project gating system hooked up to our code review 2013-08-20T00:44:38 Yahr 2013-08-20T00:44:48 And to be clear, this is on recent-ish linux, right? :) 2013-08-20T00:45:01 coderanger: haproxy or jenkins? 2013-08-20T00:45:04 well, the git server is running on centos6 2013-08-20T00:45:10 haproxy 2013-08-20T00:45:16 (this would do truly bad things on Windows) 2013-08-20T00:45:21 we don't do windows 2013-08-20T00:45:31 Clark Boylan proposed a change to openstack-infra/config: Proxy git-daemon with haproxy. https://review.openstack.org/42784 2013-08-20T00:45:32 using windows would be truly bad things 2013-08-20T00:45:40 ^^ now with backlog 2013-08-20T00:45:42 clarkb: Error: "^" is not a valid command. 2013-08-20T00:45:48 uvirtbot: sssshhh 2013-08-20T00:45:49 clarkb: Error: "sssshhh" is not a valid command. 2013-08-20T00:46:05 clarkb: yes. that looks good 2013-08-20T00:47:35 clarkb: Other thing to check is that no hooks on the git server are using the remote IP for anything (access control, logging?) 2013-08-20T00:47:42 Other than that, sounds like it will do what you want :) 2013-08-20T00:47:54 coderanger: we don't have server side hooks so we should be fine 2013-08-20T00:48:03 I hadn't thought about hooks on a git-daemon pull 2013-08-20T00:48:38 coderanger: cool thanks 2013-08-20T00:49:18 * jeblair runs again 2013-08-20T00:51:05 coderanger: what does the hash option to jenkins scm plugin do? 2013-08-20T00:51:20 * fungi assumes it's hash-based load distribution 2013-08-20T00:51:45 * Alex_Gaynor assumes it reuses the same clone but just fetches that hash 2013-08-20T00:52:05 ooh, you're probably right 2013-08-20T00:52:10 Yeah, the scm plugin uses a cron-style config 2013-08-20T00:52:30 the hash flag just lets you do /N 2013-08-20T00:52:46 Spreads out the thundering heard, but that only helps balance against multiple jobs 2013-08-20T00:52:50 not multiple slaves on the same job 2013-08-20T00:52:57 coderanger: if you want to see shiny graphs and current tests http://status.openstack.org/zuul/ 2013-08-20T00:53:10 * fungi guessed right 2013-08-20T00:53:12 jeez, 600+ outstanding events 2013-08-20T00:53:19 s/events/results/ 2013-08-20T00:53:49 Alex_Gaynor: this is what happens before milestone 3 every single time 2013-08-20T00:53:56 Alex_Gaynor: for grizzly it was particularly painful 2013-08-20T00:54:06 clarkb: ahaha, this is my first milestone I guess 2013-08-20T00:54:12 Alex_Gaynor: if we had the grizzly load today we would've been fine, but you guys keep writing more code :) 2013-08-20T00:54:20 clarkb: sorry? 2013-08-20T00:54:23 :D 2013-08-20T00:57:54 clarkb: these events/results are all bottlenecked on git? 2013-08-20T00:58:35 *** anteaya has quit IRC 2013-08-20T00:58:42 mordred: is the expectation that doing 'pip install -r requirements.txt' will grab everything a service needs? 2013-08-20T00:58:59 mordred: pyudev which neutron wants is not listed in it's requirements.txt. I suspect it's a transitive dependency :( 2013-08-20T00:59:01 Alex_Gaynor: events definitely are. I don't think results are so it is weird to see results so high 2013-08-20T00:59:16 Alex_Gaynor: actually I take that back. results end up merging code in gerrit which would be bottlenecked too 2013-08-20T01:00:37 Alex_Gaynor: events is gerrit events input into zuul. Things like new patchset or new comment. results are results from jenkins 2013-08-20T01:01:36 clarkb: I assume results are serialized, so it's really a head of the line problem? 2013-08-20T01:02:11 Alex_Gaynor: correct 2013-08-20T01:02:25 *** lbragstad has joined #openstack-infra 2013-08-20T01:02:47 comparing cacti graphs for zuul and review.o.o this really seems to be a zuul problem 2013-08-20T01:02:59 mordred: jeblair fungi I think we should merge the change to point d-g at git.o.o 2013-08-20T01:04:25 jeblair: and I wonder if we shouldn't artificially throttle zuul, or at least have the option to 2013-08-20T01:04:36 I feel better when things are slow but under control :) 2013-08-20T01:05:26 clarkb: what? 2013-08-20T01:05:38 jeblair: see the queue lengths on the zuul status page 2013-08-20T01:06:08 was mgagne in here asking about redirects? 2013-08-20T01:06:29 *** beagles has quit IRC 2013-08-20T01:06:29 * bodepd searches logs... 2013-08-20T01:07:11 bodepd: he was at some point last week iirc 2013-08-20T01:08:40 clarkb: what was the verdict? 2013-08-20T01:08:48 clarkb: shoul I open a ticket? 2013-08-20T01:09:11 clarkb: we've got a lot of changes that need to happen, and decision to make based on if that happens 2013-08-20T01:09:16 bodepd: I want to say he made the change and it merged 2013-08-20T01:09:23 bodepd: check in the git log for openstack/config 2013-08-20T01:09:31 the repo does not exist 2013-08-20T01:09:40 er openstack-infra/config 2013-08-20T01:10:21 *** pabelanger has quit IRC 2013-08-20T01:10:38 no, I meant stackforge/puppet-quantum 2013-08-20T01:10:53 oh renames 2013-08-20T01:11:13 he wanted puppet lint file redirects. I thought that is what you were talking about 2013-08-20T01:11:17 mordred: ^ rename question 2013-08-20T01:11:22 sorry.for hte lack of context 2013-08-20T01:11:30 i believe that repo has been renamed 2013-08-20T01:11:44 mordred: basically, a github redict stackforge/puppet-quantum -> stackforge/puppet-neutron 2013-08-20T01:11:49 would be awesome 2013-08-20T01:12:11 I know it's possible to do if you are admin of an account 2013-08-20T01:12:17 bodepd: i'm opposed to that. 2013-08-20T01:12:24 jeblair: ok. 2013-08-20T01:12:38 jeblair: that is what I need to know. (if it is going to happen or not) 2013-08-20T01:12:54 b/c we have lots of code that needs to be updated otherwise 2013-08-20T01:13:09 jeblair: what is the reason against it? 2013-08-20T01:13:11 bodepd: sorry, it's an extremely busy time, we're even shorter staffed then normal, and we need to focus on keeping openstack running 2013-08-20T01:13:12 jeblair: the last log item for processing result events is from 00:25 2013-08-20T01:13:18 *** xchu has joined #openstack-infra 2013-08-20T01:13:24 *** pabelanger has joined #openstack-infra 2013-08-20T01:14:10 clarkb: yeah, i'm trying to figure out what it's doing 2013-08-20T01:15:21 clarkb: oh really, i thought this was the last 2013-08-20T01:15:21 2013-08-20 00:09:35,360 DEBUG zuul.Scheduler: Processing result event 2013-08-20T01:16:33 clarkb: am away from my laptop for a few hours, can do some tests later (my test server is still up) 2013-08-20T01:16:51 pleia2: awesome. That would be helpful as it seems like I am doing 2 other things at the moment 2013-08-20T01:16:59 pleia2: and I think it can wait for tomorrow 2013-08-20T01:17:28 oh you're right 2013-08-20T01:17:29 2013-08-20 00:25:24,949 DEBUG zuul.Scheduler: Processing result event 2013-08-20T01:20:32 jeblair: which is interesting because it is a result that should end up merging code or anything like that 2013-08-20T01:21:03 *** lcestari has quit IRC 2013-08-20T01:23:53 jeblair: but that would trigger pipeline.manager.onBuildCompleted(build) 2013-08-20T01:25:49 jeblair: 42726,2 is in the check queue 2013-08-20T01:25:51 clarkb: any completion event triggers the pipeline processor 2013-08-20T01:26:06 jeblair: it does look like the gate queue is still being processed though? 2013-08-20T01:27:15 it does? 2013-08-20T01:28:14 bodepd: per github redirects, i got the impression from the article on their site that it happens automagically when a repo is moved/renamed. but maybe not 2013-08-20T01:28:16 jeblair: well the existing changes are getting some updates. I think anything going through the global event loop is stuck 2013-08-20T01:28:36 fungi: yes, when a repo is renamed the redirects should be automatic 2013-08-20T01:28:54 jeblair: though it looks like that is happening for check changes too. So status on the changish/eventqueueobject is being updated but the big while true loop is stuck so we don't update much more than that 2013-08-20T01:30:19 jeblair: are we stuck in the while self.processQueue loop in the pipeline manager? 2013-08-20T01:31:12 jeblair: https://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/scheduler.py#n1036 2013-08-20T01:32:06 *** coderanger has left #openstack-infra 2013-08-20T01:32:21 *** Ryan_Lane has quit IRC 2013-08-20T01:33:00 *** mriedem has quit IRC 2013-08-20T01:34:00 jeblair: http://paste.openstack.org/show/44559/ is the last time I see that log message 2013-08-20T01:34:38 clarkb: it recently logged it again 2013-08-20 01:27:07,488 DEBUG zuul.IndependentPipelineManager: Starting queue processor: check 2013-08-20T01:35:05 jeblair: yeah my version of the debug log was out of date 2013-08-20T01:35:28 clarkb: did it move? 2013-08-20T01:35:44 clarkb: istr top of check had no running jobs 2013-08-20T01:36:07 jeblair: yeah looking at the log it seems to have moved 2013-08-20T01:36:11 clarkb: 2013-08-20 01:27:07,148 DEBUG zuul.Scheduler: Run handler sleeping 2013-08-20T01:36:11 2013-08-20 01:27:07,148 DEBUG zuul.Scheduler: Run handler awake 2013-08-20T01:36:17 *** dina_belova has joined #openstack-infra 2013-08-20T01:36:26 clarkb: so basically it just spent 1 hour in one iteration of that loop 2013-08-20T01:36:27 jeblair: http://paste.openstack.org/show/44560/ 2013-08-20T01:36:31 jeblair: yes 2013-08-20T01:37:12 it looks like the queue started to move again? 2013-08-20T01:37:22 at least a little 2013-08-20T01:37:32 Alex_Gaynor: yeah a little 2013-08-20T01:37:58 I need to head home or food will be cold. But I will check back in from there 2013-08-20T01:38:19 jeblair: tail -f /var/log/zuul/debug.log | grep 'zuul.*PipelineManager' is what I am running now to see it move 2013-08-20T01:40:34 is the gerrit-overloaded-slowing-merges-and-result-posting theory still being batted around? with load average ~300 there and cpu pegged flat out, it seems reasonable for that to crawl 2013-08-20T01:41:09 er, ~200 i guess 2013-08-20T01:41:18 *** dina_belova has quit IRC 2013-08-20T01:41:27 everything broke together is a pretty reasonable explanation it seems 2013-08-20T01:41:31 fungi: it's possible; but we didn't see this earlier when we were busier 2013-08-20T01:42:37 mmm, point 2013-08-20T01:42:55 so what changed such that things started moving again? 2013-08-20T01:43:38 (there's still a ton of oustadning events/results) 2013-08-20T01:44:57 Trying to figure out what went wrong in gate-grenade-devstack-vm here: https://review.openstack.org/35085 2013-08-20T01:45:06 Help? 2013-08-20T01:47:03 HenryG: could this be the client backwards compat issue which was causing problems earlier today? have you asked in #openstack-qa? 2013-08-20T01:47:07 *** pcrews has joined #openstack-infra 2013-08-20T01:47:09 yes it is 2013-08-20T01:47:12 *** ftcjeff has joined #openstack-infra 2013-08-20T01:47:18 HenryG: known issue from earlier. should be fixed now 2013-08-20T01:47:38 mordred: fungi: thanks. recheck bug #? 2013-08-20T01:48:00 HenryG: it's at the top of the page here: http://status.openstack.org/rechecks/ 2013-08-20T01:48:25 HenryG: yeah, looking at the console log for that change it looks the same 2013-08-20T01:49:11 so I'm starting to think those queue counts can't possibly be right 2013-08-20T01:49:40 Alex_Gaynor: why? it's been stuck/slow for over an hour 2013-08-20T01:50:06 jeblair: well, there are ~50 patches in tehre right now, how can there be 965 results (is that queue entirely jenkins results/) 2013-08-20T01:53:03 Alex_Gaynor: those are start and stop events for jenkins; something like more than 700 have arrived since the start of the slowness 2013-08-20T01:53:32 so 50 * (say 6 tests per) * 2 still doesn't account for 900? 2013-08-20T01:53:47 and yeah, it does seem from the cacti graphs that cpu/load have fallen dramatically on zuul in the past couple hours 2013-08-20T01:54:49 Random other point: the SCP step for the logs seems to be slower today 2013-08-20T01:55:17 Alex_Gaynor: it's more than 6 jobs per change 2013-08-20T01:55:21 Alex_Gaynor: nova runs 13 2013-08-20T01:55:29 in the check queue 2013-08-20T01:55:36 gah, good point, I guess it does add up 2013-08-20T01:56:35 1k events :( 2013-08-20T01:57:36 *** nati_ueno has quit IRC 2013-08-20T01:58:52 i have attached a debugger. 2013-08-20T01:59:11 i need to get a stack trace, but the last time i tried that with gdb, the old trick i used to use didn't work 2013-08-20T02:00:33 gdb or pdb? 2013-08-20T02:00:36 gdb 2013-08-20T02:00:46 can you attach pdb to a running process? 2013-08-20T02:00:59 attach a gdb, acquire the GIL, use pdb :) 2013-08-20T02:01:16 Alex_Gaynor: do you have instructions for that? 2013-08-20T02:01:22 you'll have to teach me how to do that someday Alex_Gaynor 2013-08-20T02:01:34 if it's a recent gcc there's actually a python embedded that let's you do stuff 2013-08-20T02:01:38 gdb&\ 2013-08-20T02:01:40 gdb*\ 2013-08-20T02:02:05 http://wiki.python.org/moin/DebuggingWithGdb has some details 2013-08-20T02:02:21 Alex_Gaynor: afaict, the 'py-bt' thing is a fedora-ism 2013-08-20T02:02:36 https://fedoraproject.org/wiki/Features/EasierPythonDebugging#New_gdb_commands 2013-08-20T02:02:50 jeblair: it was originalyl developed by a redhat person for fedora, but it's upstream now 2013-08-20T02:03:00 oh. this is on precise 2013-08-20T02:03:02 maybe debian/friends don't compile with the needed flags or something :( 2013-08-20T02:03:17 Alex_Gaynor: i think those are extra gdb commands 2013-08-20T02:04:04 *** rfolco has quit IRC 2013-08-20T02:04:22 ah, they are also in the precise python dbg package 2013-08-20T02:04:49 load average on review.o.o has collapsed too now 2013-08-20T02:05:00 * fungi needs to head out to a dinner reservation. bbl 2013-08-20T02:05:31 I need to head home from the office because at some point it became 7PM, I'll be around more when I'm home 2013-08-20T02:05:46 *** rfolco has joined #openstack-infra 2013-08-20T02:09:01 jeblair: anything else I can be doing now to help? 2013-08-20T02:10:55 clarkb: i'm still unable to get a stacktrace. 'py-bt' just says (unable to read python frame information) for every frame 2013-08-20T02:11:27 clarkb: figuring out how to get a stacktrace from a running python on ubuntu precise is what i'm working on now. any help there would be appreciated 2013-08-20T02:11:53 *** yaguang has joined #openstack-infra 2013-08-20T02:12:08 jeblair: ok 2013-08-20T02:13:39 clarkb: apparently those macros expect to be run with python-dbg, which of course is not how we started zuul 2013-08-20T02:13:40 jeblair: http://www.python.org/~jeremy/weblog/031003.html not quite a stack trace but possibly useful 2013-08-20T02:16:17 *** xBsd has joined #openstack-infra 2013-08-20T02:17:55 jeblair: also http://svn.python.org/projects/python/trunk/Misc/gdbinit 2013-08-20T02:17:57 clarkb: i think the objects have changed since then 2013-08-20T02:18:27 jeblair: that gdbinit comes with a pystack function 2013-08-20T02:20:00 *** ^demon has quit IRC 2013-08-20T02:20:23 clarkb: No symbol "co" in current context. 2013-08-20T02:20:35 clarkb: these all seem to be obsolete. 2013-08-20T02:20:56 :( yeah they are fairly old 2013-08-20T02:21:06 * clarkb finds python2.7 branch 2013-08-20T02:22:35 *** lbragstad has quit IRC 2013-08-20T02:23:13 clarkb: i think it's due to gcc optimizations 2013-08-20T02:23:37 jeblair: http://hg.python.org/cpython/file/c048b211f634/Misc/gdbinit doesn't seem different but I haven't actually diffed them 2013-08-20T02:23:58 jeblair: ah so the symbols just don't exist because gcc 2013-08-20T02:25:33 i wonder if we could even do Alex_Gaynor's pdb trick with the current level of symbol mangling 2013-08-20T02:26:19 jeblair: if you can grab the Gil and use c execute simple string it should be possible 2013-08-20T02:26:55 Alex_Gaynor: that sounds easy but i have no idea how to go about that 2013-08-20T02:27:25 When I'm at a computer and not my phone I'll try to find av reference 2013-08-20T02:30:05 jeblair: I'm here - I do not what what I can do to be helpful to you 2013-08-20T02:30:35 mordred: we need a stacktrace from running zuul 2013-08-20T02:31:20 http://www.jmcneil.net/2012/04/debugging-your-python-with-gdb-ftw/ 2013-08-20T02:31:22 reading this now 2013-08-20T02:32:31 mordred: my understanding of that is that it does not work because of gcc optimizations 2013-08-20T02:32:42 jeblair: yeah. I believe you are correct 2013-08-20T02:33:11 btw - symbol stripping, which debian is obsessed with, has no real noticable benefit most times 2013-08-20T02:33:17 and screws you in times like this 2013-08-20T02:33:44 jeblair: have you installed python-dbg? sometimes deb packages extract the symbols and put them into external files 2013-08-20T02:33:49 thanks debian! 2013-08-20T02:33:51 mordred: yes i have 2013-08-20T02:33:51 and gdb can be told to load them as symbol maps 2013-08-20T02:34:06 let me see if i can get some info on that 2013-08-20T02:34:14 mordred: that made the backtraces look like this: #33 0x0000000000466a42 in PyEval_EvalFrameEx () 2013-08-20T02:34:23 mordred: but still no understanding of arguments or local variables 2013-08-20T02:34:30 *** eharney has joined #openstack-infra 2013-08-20T02:35:10 so "p *co" does nothing 2013-08-20T02:35:30 No symbol "co" in current context. 2013-08-20T02:35:33 awesome 2013-08-20T02:36:42 so, we could call this a wash 2013-08-20T02:36:53 *** dina_belova has joined #openstack-infra 2013-08-20T02:36:53 and restart zuul using the 'python-dbg' binary 2013-08-20T02:36:53 oh - wait 2013-08-20T02:37:03 there's a thing dhellman tweeted about the other day 2013-08-20T02:37:03 *** jfriedly has quit IRC 2013-08-20T02:37:10 this must be why people gentoo 2013-08-20T02:37:26 and if it happens again, we'd be in a better place (no idea what that would do to performance though, since i think it is doing refcount debugging as well) 2013-08-20T02:37:39 mordred: that's exciting; i'm holding for your tweet 2013-08-20T02:37:55 (i'll be really excited if the actual method is less than 140 characters) 2013-08-20T02:38:08 ok. I don't think this is it, but, while I'm looking, look at: https://github.com/albertz/pydbattach 2013-08-20T02:38:18 *** rfolco has quit IRC 2013-08-20T02:38:41 *** dina_belova has quit IRC 2013-08-20T02:38:43 mordred: wilco 2013-08-20T02:40:50 *** mriedem has joined #openstack-infra 2013-08-20T02:41:31 mordred: neat, but it's complicated, and i don't really want to audit it or compile/run it on our server right now 2013-08-20T02:41:48 jeblair: ok. that's the closest thing I can find right now 2013-08-20T02:42:02 I think that call a wash and restart zuul with python-dbg is our best bet 2013-08-20T02:42:33 wfm 2013-08-20T02:42:49 not elegant, but if it keeps things moving... 2013-08-20T02:43:32 *** bingbu has joined #openstack-infra 2013-08-20T02:46:34 okay that's clearly more complicated than it seems 2013-08-20T02:46:36 ImportError: /usr/local/lib/python2.7/dist-packages/Crypto/Util/_counter.so: undefined symbol: Py_InitModule4_64 2013-08-20T02:47:39 ok, so i can just restart it as normal, and add some more debug lines to it i guess. 2013-08-20T02:48:00 maybe add a jenkins style "threadDump" command. won't that just be the best? 2013-08-20T02:48:31 zuul has been restarted. it has no queue. 2013-08-20T02:49:01 *** mriedem has quit IRC 2013-08-20T02:49:06 *** pcrews has quit IRC 2013-08-20T02:49:10 jeblair: that will work too 2013-08-20T02:49:15 well that doesn't sound good 2013-08-20T02:49:21 jeblair: sigh. I believe, now that you mention, to use python-dbg, you will need -dbg versions of all of the c-based python libraries you might have installed 2013-08-20T02:49:37 in addition to the -dbg versions of the c libraries they depend on 2013-08-20T02:50:06 mordred: lets move all our servers to rhel 2013-08-20T02:50:16 jeblair: ok 2013-08-20T02:50:17 jeblair: or gentoo 2013-08-20T02:50:29 jeblair: or gentoo - and we can compile from source ourselves 2013-08-20T02:51:03 mordred: https://bugs.launchpad.net/nova/+bug/937554/comments/13 2013-08-20T02:51:06 Launchpad bug 937554 in nova "Lots of problems with deleting a server immediately after create (dup-of: 934575)" [High,Fix committed] 2013-08-20T02:51:07 Launchpad bug 934575 in nova "notifier endless loops in is_primitive" [Medium,Fix released] 2013-08-20T02:51:15 *** eharney has quit IRC 2013-08-20T02:51:35 * mordred is looking at the debian packaging and cannot figure out why stack information is missing in the normal python 2013-08-20T02:51:44 *** melwitt has quit IRC 2013-08-20T02:51:52 they aren't passing stupid optimizer flags 2013-08-20T02:51:55 handy instructions for building your own python, in a nova bug report no less! 2013-08-20T02:52:09 mordred: " 2013-08-20T02:52:10 #Recompiling python with make "CFLAGS=-g -fno-inline -fno-strict-aliasing" solves this problem. 2013-08-20T02:52:23 mordred: ^ from that bug report; that help? 2013-08-20T02:52:25 ahhhh 2013-08-20T02:52:28 yes 2013-08-20T02:52:29 -fno-inline 2013-08-20T02:52:39 I forgot - python actually has a bunch of stuff defined in header files 2013-08-20T02:52:49 so -O2 is going to wind up inlining the shit out of it 2013-08-20T02:55:16 -O2 includes -finline-small-functions 2013-08-20T02:55:26 -O0, which python-dbg is compiled with, does not 2013-08-20T02:55:44 they're all compiled with -g but then dh_strip puts the symbols into python-dbg 2013-08-20T02:56:48 none of that is helpful here 2013-08-20T03:02:14 *** afazekas_zz is now known as __afazekas_zz 2013-08-20T03:04:10 i have reverified all the changes that were approved and did not have a vrfy-2 2013-08-20T03:04:56 *** rcleere has joined #openstack-infra 2013-08-20T03:05:54 *** markmcclain has quit IRC 2013-08-20T03:06:13 i have had a very long day and am not useful. tomorrow i intend to work on nodepool. if anyone wants to add some more debugging or a threadDump feature to zuul, that would be great; otherwise, i'll get to it later this week 2013-08-20T03:06:30 also, i'm thinking we should have the gearman-plugin stop seding work status packets. 2013-08-20T03:06:35 sending 2013-08-20T03:07:16 *** Ryan_Lane has joined #openstack-infra 2013-08-20T03:07:17 so, are no builds happening right now? 2013-08-20T03:07:18 I can look into zuul threaddumps 2013-08-20T03:07:32 after I propose changed to add mysql backups (that should be quick) 2013-08-20T03:07:36 Alex_Gaynor: i restarted zuul, should be running now 2013-08-20T03:07:46 jeblair: there doesn't appear to be anythign on http://status.openstack.org/zuul/ 2013-08-20T03:07:46 jeblair: are work status packets causing problems? 2013-08-20T03:07:59 Alex_Gaynor: refresh? there is stuff for me 2013-08-20T03:08:00 *** pcrews has joined #openstack-infra 2013-08-20T03:08:01 Alex_Gaynor: you may need to reload it? 2013-08-20T03:08:12 I don't even know. I hate browsers. 2013-08-20T03:08:15 clarkb: no, but we ignore them. just busy work. 2013-08-20T03:12:47 jeblair: oh, for some reason I thought we were using them for status bars - I agree with anything you say 2013-08-20T03:12:59 mordred: that is what I thought they were for too 2013-08-20T03:13:16 and isn't zuul LOST status the result of not getting a status from gearman? 2013-08-20T03:13:43 *** erfanian has quit IRC 2013-08-20T03:13:44 mordred: we could. what we actually do is grab the estimated time from the first one and then calculate it ourselves. 2013-08-20T03:13:52 jeblair: ah. nice 2013-08-20T03:14:18 clarkb: no, it polls gearman to see if the job is still in the queue. that would be a reasonable thing to do though... 2013-08-20T03:14:27 clarkb: it would have helped with the jobs that got stuck in the jenkins queue and never ran 2013-08-20T03:14:49 clarkb: maybe we should keep it and just reduce the logging. 2013-08-20T03:14:58 ++ 2013-08-20T03:15:11 i've seen several jobs lost because of errors like this: https://jenkins02.openstack.org/job/gate-grenade-devstack-vm/2370/console 2013-08-20T03:15:42 i have no idea what's going on there. perhaps a dead slave (nodepool does not have a periodic job to recheck ssh access) 2013-08-20T03:15:46 but it seems to happen a lot for that. 2013-08-20T03:16:22 For all that jobs that were lost when zuul was restarted, are the patch authors notified so they can recheck/reverfiy? 2013-08-20T03:16:53 Alex_Gaynor: no, but I think jeblair indicated he did it for them 2013-08-20T03:16:57 Alex_Gaynor: i reverified the ones that were approved; 2013-08-20T03:17:02 Oh, that's nice of you! 2013-08-20T03:17:02 I have not done rechecks. 2013-08-20T03:17:16 it's hard to get a gerrit query for that. 2013-08-20T03:17:43 Things that don't hvae a current status from jenkins 2013-08-20T03:17:52 gerrit doesn't have an easy way to do that? :( 2013-08-20T03:19:37 -label:Verified<=2 will get you the ones that are completely new - but it's hard to get the ones that may have had a new patchset uploaded since the last time they were check verified 2013-08-20T03:19:55 because we don't clear the verified status on the start of a new check job like we do for the gate 2013-08-20T03:20:52 actually, you'd want -label:Verified<=2 -label:Approved for the first one, to make sure that you're not catching a thing that the gate has cleared the verified vote 2013-08-20T03:21:06 but still, you're still missing a ton there 2013-08-20T03:25:41 *** HenryG has quit IRC 2013-08-20T03:31:54 *** zul has quit IRC 2013-08-20T03:33:02 *** cthulhup has joined #openstack-infra 2013-08-20T03:37:26 *** cthulhup has quit IRC 2013-08-20T03:37:29 *** dina_belova has joined #openstack-infra 2013-08-20T03:42:09 *** dina_belova has quit IRC 2013-08-20T03:42:17 *** afazekas has joined #openstack-infra 2013-08-20T03:49:07 *** boris-42 has joined #openstack-infra 2013-08-20T03:52:48 fungi: I just went through the following process: https://gist.github.com/bodepd/6276932 2013-08-20T03:53:28 fungi: and my redirects worked as expected. I did, however, use github's GUI, and I am not sure what process was used by your team 2013-08-20T03:53:51 *** xBsd has quit IRC 2013-08-20T03:55:12 *** jfriedly has joined #openstack-infra 2013-08-20T03:56:24 *** wenlock has joined #openstack-infra 2013-08-20T03:59:29 *** mberwanger has joined #openstack-infra 2013-08-20T03:59:42 *** yaguang has quit IRC 2013-08-20T04:01:35 *** vogxn has joined #openstack-infra 2013-08-20T04:04:52 *** michchap_ has joined #openstack-infra 2013-08-20T04:08:52 *** michchap has quit IRC 2013-08-20T04:12:09 *** yaguang has joined #openstack-infra 2013-08-20T04:23:36 *** ftcjeff has quit IRC 2013-08-20T04:24:05 *** wenlock has quit IRC 2013-08-20T04:25:10 *** dims has quit IRC 2013-08-20T04:27:07 *** cthulhup has joined #openstack-infra 2013-08-20T04:31:40 *** cthulhup has quit IRC 2013-08-20T04:38:00 *** dina_belova has joined #openstack-infra 2013-08-20T04:38:37 *** mberwanger has quit IRC 2013-08-20T04:42:45 *** dina_belova has quit IRC 2013-08-20T04:47:04 *** xBsd has joined #openstack-infra 2013-08-20T04:53:26 *** reed has quit IRC 2013-08-20T04:59:52 *** yaguang has quit IRC 2013-08-20T05:03:11 *** rcleere has quit IRC 2013-08-20T05:07:13 bodepd: yeah, mordred did the stackforge/puppet-{quantum,neutron} move, but not sure what he did in github land for it. our http://ci.openstack.org/gerrit.html#renaming-a-project recipe suggests "12. Rename the project in GitHub..." so i would assume that's what he did 2013-08-20T05:08:08 *** dmakogon_ has joined #openstack-infra 2013-08-20T05:12:48 *** yaguang has joined #openstack-infra 2013-08-20T05:21:13 *** cthulhup has joined #openstack-infra 2013-08-20T05:24:00 *** SergeyLukjanov has joined #openstack-infra 2013-08-20T05:25:58 *** cthulhup has quit IRC 2013-08-20T05:29:19 *** nicedice_ has quit IRC 2013-08-20T05:34:14 fungi, bodepd I'm pretty sure I just deleted the old project and let the new project be created by manage_projects 2013-08-20T05:38:41 *** dina_belova has joined #openstack-infra 2013-08-20T05:43:42 *** dina_belova has quit IRC 2013-08-20T05:48:23 *** thomasbiege has joined #openstack-infra 2013-08-20T05:55:48 *** DennyZhang has joined #openstack-infra 2013-08-20T05:55:49 *** mikal has quit IRC 2013-08-20T05:59:43 *** thomasbiege1 has joined #openstack-infra 2013-08-20T06:02:23 *** thomasbiege has quit IRC 2013-08-20T06:13:22 *** thomasbiege1 has quit IRC 2013-08-20T06:15:26 *** cthulhup has joined #openstack-infra 2013-08-20T06:17:42 *** thomasbiege has joined #openstack-infra 2013-08-20T06:20:12 *** cthulhup has quit IRC 2013-08-20T06:39:15 *** dina_belova has joined #openstack-infra 2013-08-20T06:43:57 *** dina_belova has quit IRC 2013-08-20T06:44:57 *** tian has quit IRC 2013-08-20T06:47:10 *** nayward has joined #openstack-infra 2013-08-20T06:49:16 *** fbo is now known as fbo_away 2013-08-20T06:50:22 *** SergeyLukjanov has quit IRC 2013-08-20T06:52:59 *** jfriedly has quit IRC 2013-08-20T06:57:52 mordred: :( . I'm trying to reach out to some folks at github to see if they can help us setup those redirects 2013-08-20T06:58:21 mordred: I may need someone with actual credentials to approve it once I get a hold of the right person 2013-08-20T07:00:01 *** michchap has joined #openstack-infra 2013-08-20T07:00:09 *** xchu has quit IRC 2013-08-20T07:02:18 *** michchap_ has quit IRC 2013-08-20T07:09:28 *** cthulhup has joined #openstack-infra 2013-08-20T07:11:43 *** SergeyLukjanov has joined #openstack-infra 2013-08-20T07:12:40 *** xchu has joined #openstack-infra 2013-08-20T07:14:18 *** cthulhup has quit IRC 2013-08-20T07:14:31 *** SergeyLukjanov has quit IRC 2013-08-20T07:26:23 *** ruhe has joined #openstack-infra 2013-08-20T07:29:32 *** pblaho has joined #openstack-infra 2013-08-20T07:34:35 *** boris-42 has quit IRC 2013-08-20T07:38:43 *** SergeyLukjanov has joined #openstack-infra 2013-08-20T07:39:22 *** dina_belova has joined #openstack-infra 2013-08-20T07:39:24 *** michchap has quit IRC 2013-08-20T07:39:58 *** michchap has joined #openstack-infra 2013-08-20T07:41:44 bodepd: Simply contacting Github support had really good turn around on the redirects from puppetlabs/puppetlabs-* to stackforge/puppet-*. 2013-08-20T07:42:17 They manually put them in long before I actually deleted the repositories. 2013-08-20T07:47:33 *** odyssey4me3 has joined #openstack-infra 2013-08-20T07:48:31 The "Approved" label that seems to be a part of each Gerrit project. What is it used for? Gerrit docs don't make mention of it so I assume it is a custom label. 2013-08-20T07:50:53 * odyi also couldn't find it mentioned in any of the OpenStack/Gerrit workflow docs. 2013-08-20T07:52:12 *** michchap has quit IRC 2013-08-20T07:55:43 *** morganfainberg is now known as morganfainberg|a 2013-08-20T07:56:06 *** DennyZhang has quit IRC 2013-08-20T08:00:31 *** SergeyLukjanov has quit IRC 2013-08-20T08:03:29 *** vogxn has quit IRC 2013-08-20T08:03:40 *** cthulhup has joined #openstack-infra 2013-08-20T08:04:00 *** jpich has joined #openstack-infra 2013-08-20T08:06:21 *** derekh has joined #openstack-infra 2013-08-20T08:07:42 *** fbo_away is now known as fbo 2013-08-20T08:08:05 *** cthulhup has quit IRC 2013-08-20T08:09:35 *** xchu has quit IRC 2013-08-20T08:15:55 *** alex_dolby has joined #openstack-infra 2013-08-20T08:16:30 *** jhesketh has quit IRC 2013-08-20T08:17:52 hi guys.. i am running tox -epy26 in python-novaclient compoennt and getting error about pbr version versions 2013-08-20T08:18:15 pbr version in setup.py and requirement.txt has different versions.. 2013-08-20T08:18:20 any pointers? 2013-08-20T08:20:59 *** mkerrin has quit IRC 2013-08-20T08:21:33 *** dina_belova has quit IRC 2013-08-20T08:24:55 *** ladquin has quit IRC 2013-08-20T08:26:16 *** xchu has joined #openstack-infra 2013-08-20T08:27:54 *** psedlak has joined #openstack-infra 2013-08-20T08:27:58 *** SergeyLukjanov has joined #openstack-infra 2013-08-20T08:40:01 *** boris-42 has joined #openstack-infra 2013-08-20T08:57:52 *** cthulhup has joined #openstack-infra 2013-08-20T09:02:02 *** vogxn has joined #openstack-infra 2013-08-20T09:02:24 *** cthulhup has quit IRC 2013-08-20T09:12:01 *** arezadr has quit IRC 2013-08-20T09:22:05 *** dina_belova has joined #openstack-infra 2013-08-20T09:26:33 *** dina_belova has quit IRC 2013-08-20T09:27:06 *** bingbu has quit IRC 2013-08-20T09:32:11 *** SergeyLukjanov has quit IRC 2013-08-20T09:32:13 *** dina_belova has joined #openstack-infra 2013-08-20T09:34:45 *** dina_belova has quit IRC 2013-08-20T09:35:10 *** dina_belova has joined #openstack-infra 2013-08-20T09:43:48 *** yaguang has quit IRC 2013-08-20T09:54:52 *** odyssey4me3 has quit IRC 2013-08-20T10:03:20 *** xchu has quit IRC 2013-08-20T10:05:45 *** odyssey4me3 has joined #openstack-infra 2013-08-20T10:09:11 *** dina_belova has quit IRC 2013-08-20T10:09:22 *** LinuxJedi has quit IRC 2013-08-20T10:12:25 *** ruhe has quit IRC 2013-08-20T10:12:26 *** alexpilotti has joined #openstack-infra 2013-08-20T10:17:02 *** odyssey4me3 has quit IRC 2013-08-20T10:20:02 *** LinuxJedi has joined #openstack-infra 2013-08-20T10:21:31 *** ruhe has joined #openstack-infra 2013-08-20T10:24:10 *** odyssey4me3 has joined #openstack-infra 2013-08-20T10:28:18 *** thomasbiege has quit IRC 2013-08-20T10:37:45 *** SergeyLukjanov has joined #openstack-infra 2013-08-20T10:45:47 *** nayward has quit IRC 2013-08-20T10:49:26 mordred, jeblair : were you looking for https://github.com/dhellmann/smiley/ last night? 2013-08-20T10:52:35 *** mkerrin has joined #openstack-infra 2013-08-20T10:52:47 *** nayward has joined #openstack-infra 2013-08-20T10:56:03 *** markmc has joined #openstack-infra 2013-08-20T11:02:28 *** thomasbiege has joined #openstack-infra 2013-08-20T11:09:40 *** dina_belova has joined #openstack-infra 2013-08-20T11:09:41 After the glanceclient fix yesterday, I reviewed three changes with "recheck no bug" about 12 hours ago. Jenkins has not re-reviewed them yet. Anything else I need to do? 2013-08-20T11:11:40 For example: https://review.openstack.org/#/c/40232/ 2013-08-20T11:12:30 *** SergeyLukjanov has quit IRC 2013-08-20T11:14:07 *** dina_belova has quit IRC 2013-08-20T11:16:34 *** vogxn has quit IRC 2013-08-20T11:17:47 *** lcestari has joined #openstack-infra 2013-08-20T11:18:33 *** vogxn has joined #openstack-infra 2013-08-20T11:19:06 *** zul has joined #openstack-infra 2013-08-20T11:19:50 *** dina_belova has joined #openstack-infra 2013-08-20T11:20:43 *** dims has joined #openstack-infra 2013-08-20T11:24:23 *** dina_belova has quit IRC 2013-08-20T11:29:44 *** nayward has quit IRC 2013-08-20T11:31:36 *** weshay has joined #openstack-infra 2013-08-20T11:31:53 *** vogxn has quit IRC 2013-08-20T11:32:48 *** ruhe has quit IRC 2013-08-20T11:39:54 *** SergeyLukjanov has joined #openstack-infra 2013-08-20T11:41:04 *** nayward has joined #openstack-infra 2013-08-20T11:44:43 *** SergeyLukjanov has quit IRC 2013-08-20T11:46:14 *** zul has quit IRC 2013-08-20T11:46:53 *** pcm_ has joined #openstack-infra 2013-08-20T11:46:53 *** vogxn has joined #openstack-infra 2013-08-20T11:49:38 *** HenryG has joined #openstack-infra 2013-08-20T11:50:06 Julien Danjou proposed a change to openstack/requirements: Add gevent https://review.openstack.org/42871 2013-08-20T11:58:55 *** jjmb1 has quit IRC 2013-08-20T11:59:27 *** afazekas is now known as afazekas_no_irq 2013-08-20T12:02:06 *** yaguang has joined #openstack-infra 2013-08-20T12:06:18 *** ruhe has joined #openstack-infra 2013-08-20T12:07:38 *** rfolco has joined #openstack-infra 2013-08-20T12:09:45 *** alex_dolby has quit IRC 2013-08-20T12:12:06 *** vogxn has quit IRC 2013-08-20T12:18:05 *** apcruz has joined #openstack-infra 2013-08-20T12:19:23 *** mriedem has joined #openstack-infra 2013-08-20T12:20:19 *** dina_belova has joined #openstack-infra 2013-08-20T12:22:57 *** sandywalsh has quit IRC 2013-08-20T12:24:54 *** sandywalsh has joined #openstack-infra 2013-08-20T12:25:21 *** dina_belova has quit IRC 2013-08-20T12:27:31 *** anteaya has joined #openstack-infra 2013-08-20T12:35:52 *** SergeyLukjanov has joined #openstack-infra 2013-08-20T12:36:38 *** ruhe has quit IRC 2013-08-20T12:37:06 *** zul has joined #openstack-infra 2013-08-20T12:38:10 *** dims has quit IRC 2013-08-20T12:39:34 *** dprince has joined #openstack-infra 2013-08-20T12:39:43 *** dkranz has joined #openstack-infra 2013-08-20T12:40:21 *** dims has joined #openstack-infra 2013-08-20T12:43:45 *** dina_belova has joined #openstack-infra 2013-08-20T12:44:37 so im curious why jenkins hasnt been triggered for https://review.openstack.org/#/c/41093/ and https://review.openstack.org/#/c/42789/ 2013-08-20T12:47:02 *** ruhe has joined #openstack-infra 2013-08-20T12:47:20 zul, you know, I think I'm seeing this too with my nova reviews 2013-08-20T12:47:21 * markmc looks 2013-08-20T12:47:54 *** dina_belova has quit IRC 2013-08-20T12:47:59 *** SergeyLukjanov has quit IRC 2013-08-20T12:48:39 zul, ok, not seeing it now - but think I saw zuul missing some submissions yesterday 2013-08-20T12:48:54 hmmm 2013-08-20T12:49:23 is there a way to kick them off again? 2013-08-20T12:51:42 looks like recheck doesn't work, I don't know of another way 2013-08-20T12:51:57 just change the commit message of the first patch and re-submit 2013-08-20T12:55:06 ok 2013-08-20T12:55:43 *** dkranz has quit IRC 2013-08-20T13:00:51 markmc zul there were issues yesterday with jenkins. The best I understand is that jenkins was ddosing our git server and there was much work to bring about a resolution. Reading the logs, I can not definitively point to a solution that was found. What you are seeing _may_ be related. 2013-08-20T13:01:11 ok, thanks 2013-08-20T13:01:16 *** jog0 is now known as jog0-away 2013-08-20T13:01:23 anteaya: cool thanks 2013-08-20T13:01:30 np 2013-08-20T13:01:55 *** mberwanger has joined #openstack-infra 2013-08-20T13:03:53 *** adalbas has quit IRC 2013-08-20T13:08:38 *** kiall has quit IRC 2013-08-20T13:11:02 *** dkliban has quit IRC 2013-08-20T13:12:51 *** changbl has quit IRC 2013-08-20T13:14:38 *** whayutin_ has joined #openstack-infra 2013-08-20T13:16:42 *** weshay has quit IRC 2013-08-20T13:20:39 *** xchu has joined #openstack-infra 2013-08-20T13:23:44 *** w_ has quit IRC 2013-08-20T13:25:03 *** sgviking has quit IRC 2013-08-20T13:25:53 *** sgviking has joined #openstack-infra 2013-08-20T13:26:18 *** sgviking has quit IRC 2013-08-20T13:26:51 *** sgviking has joined #openstack-infra 2013-08-20T13:27:36 *** lbragstad has joined #openstack-infra 2013-08-20T13:27:40 *** HenryG has quit IRC 2013-08-20T13:30:04 *** michchap has joined #openstack-infra 2013-08-20T13:35:07 *** mberwanger has quit IRC 2013-08-20T13:37:41 *** prad_ has joined #openstack-infra 2013-08-20T13:42:50 *** burt has joined #openstack-infra 2013-08-20T13:43:05 *** thomasbiege2 has joined #openstack-infra 2013-08-20T13:44:38 dhellmann: yes 2013-08-20T13:45:30 *** cppcabrera has joined #openstack-infra 2013-08-20T13:46:06 *** thomasbiege has quit IRC 2013-08-20T13:46:14 ttx, mordred, dhellmann, whoever, I'd need https://review.openstack.org/#/c/42871/ quickly to unblock Ceilomeer CI failing 2013-08-20T13:46:24 zul: ^ 2013-08-20T13:46:59 jd__: can you point me to the failing thing? 2013-08-20T13:47:00 jd__: looks like I don't have +2 on requirements 2013-08-20T13:47:17 * mordred wants to understand why our mirror builder isn't picking it up 2013-08-20T13:47:19 mordred: I thought I had, but meh 2013-08-20T13:47:36 mordred: http://logs.openstack.org/46/42846/1/check/gate-ceilometer-python27/caaca73/console.html.gz 2013-08-20T13:47:41 thank you 2013-08-20T13:48:03 mordred: Pymongo does not specify the dependency… 2013-08-20T13:48:04 I can certainly spare the effort 2013-08-20T13:48:11 jd__: o m g 2013-08-20T13:48:17 jd__: SERIOUSLY? 2013-08-20T13:48:19 I hate people 2013-08-20T13:48:24 *** dina_belova has joined #openstack-infra 2013-08-20T13:48:37 I couldn't agree more 2013-08-20T13:48:53 I've opened a ticket upstream https://jira.mongodb.org/browse/PYTHON-558 2013-08-20T13:48:55 aprvd 2013-08-20T13:49:34 mordred: was I supposed to have +2 on requirements ? I forget what we originally said (discovered recently I wasn't subscribed to it) 2013-08-20T13:49:56 ttx: I'm happy to give you +2 on them - makes sense for you to have it 2013-08-20T13:49:57 *** whayutin_ has quit IRC 2013-08-20T13:50:10 +1 :) 2013-08-20T13:51:06 can't remember if I signed up for it or not 2013-08-20T13:52:57 *** dina_belova has quit IRC 2013-08-20T13:53:07 jd__: ok- there is feedback on that bug... 2013-08-20T13:53:10 mordred: let me watch the reviews for some time to see if I actually care enough 2013-08-20T13:53:21 mordred: just saw, I'm responding 2013-08-20T13:53:27 jd__: I did too 2013-08-20T13:53:32 ah 2013-08-20T13:53:33 * jd__ lags 2013-08-20T13:54:15 jd__: "gevent doesn't support python 3 or pypy" -- is there an internal feature of pymongo that you're using that's going to get us in trouble with python 3 and pypy support? 2013-08-20T13:54:28 mordred: no, we use nothing fancy 2013-08-20T13:54:33 k. cool 2013-08-20T13:54:39 I'll be interested to see what's going on here 2013-08-20T13:54:56 that's why I'm surprised we see errors about gevent now that we pull pymongo 2.6 2013-08-20T13:57:28 *** michchap has quit IRC 2013-08-20T13:58:33 *** dina_belova has joined #openstack-infra 2013-08-20T13:59:11 *** ftcjeff has joined #openstack-infra 2013-08-20T14:00:03 Russell Bryant proposed a change to openstack-infra/config: Disable tempest in the cells job temporarily https://review.openstack.org/42898 2013-08-20T14:01:15 *** weshay has joined #openstack-infra 2013-08-20T14:01:48 *** vogxn has joined #openstack-infra 2013-08-20T14:02:01 ah now that talks about greenlet and I'm going to be lost in that again 2013-08-20T14:02:03 * jd__ runs 2013-08-20T14:03:42 *** dina_belova has quit IRC 2013-08-20T14:06:41 Russell Bryant proposed a change to openstack-infra/config: Disable tempest in the cells job https://review.openstack.org/42898 2013-08-20T14:08:08 *** dkliban has joined #openstack-infra 2013-08-20T14:10:23 *** xBsd has quit IRC 2013-08-20T14:10:39 *** xBsd has joined #openstack-infra 2013-08-20T14:12:31 *** xBsd has quit IRC 2013-08-20T14:15:59 jd__: I think we can remove use_greenlets 2013-08-20T14:16:06 jd__: "If you need to use standard Python threads in the same process as Gevent and greenlets" 2013-08-20T14:16:38 indeed, we don't use threads so that should be ok I guess 2013-08-20T14:16:52 are you sure? 2013-08-20T14:17:00 libraries have been known to spawn random threads :) 2013-08-20T14:17:05 *** pabelanger has quit IRC 2013-08-20T14:17:30 now I'm unsure and scared 2013-08-20T14:17:43 markmc: well, let's solve that problem when we come to it for real - keeping the option means we're adding another python3 incompatability 2013-08-20T14:17:54 dhellmann: ping 2013-08-20T14:18:00 jd__: can we try a patch to ceilometer that removes the option? 2013-08-20T14:18:14 jd__, cooperative coroutines mumble mumble ... oh, look over there! 2013-08-20T14:18:33 mordred: sure, it'll take me a sec' 2013-08-20T14:18:40 jd__: sending patch in... 2013-08-20T14:18:49 mordred: cool 2013-08-20T14:20:26 jd__: https://review.openstack.org/42906 2013-08-20T14:20:28 *** changbl has joined #openstack-infra 2013-08-20T14:21:32 mordred: ack, approving, if Jenkins' happy, we'll be too 2013-08-20T14:21:39 great! 2013-08-20T14:21:46 and we'll be able to revert gevent fortunately 2013-08-20T14:21:53 *** xBsd has joined #openstack-infra 2013-08-20T14:21:56 I already blocked that from merging 2013-08-20T14:22:06 and https://jira.mongodb.org/browse/PYTHON-558?focusedCommentId=407277#comment-407277 for anyone who wants to play along 2013-08-20T14:22:26 dhellmann: nevermind 2013-08-20T14:22:40 *** odyssey4me3 has quit IRC 2013-08-20T14:22:42 nice turnaround on that bug report 2013-08-20T14:23:35 dhellmann: I'm reading the mailing list as being in approval of give Alex_Gaynor +2 on requirements... 2013-08-20T14:23:40 dhellmann: shall we make that happen? 2013-08-20T14:28:01 *** beagles has joined #openstack-infra 2013-08-20T14:28:40 *** thomasbiege2 has quit IRC 2013-08-20T14:32:32 *** rcleere has joined #openstack-infra 2013-08-20T14:33:38 *** mrmartin has joined #openstack-infra 2013-08-20T14:35:22 *** gordc has joined #openstack-infra 2013-08-20T14:37:48 *** markmcclain has joined #openstack-infra 2013-08-20T14:38:04 *** ruhe has quit IRC 2013-08-20T14:38:07 hi folks, would anyone happen to know when the cron job runs to update CI mirror? i just made a release for a lib and was wondering when jenkins would pick it up ... or if i could force it to get picked up. 2013-08-20T14:40:07 *** datsun180b has joined #openstack-infra 2013-08-20T14:40:25 *** yaguang has quit IRC 2013-08-20T14:41:24 *** __afazekas_zz has quit IRC 2013-08-20T14:47:13 gordc: it runs after we land requirements changes - which lib? is it a thing that we should raise the min in openstack/requirements for? 2013-08-20T14:47:26 *** odyssey4me4 has joined #openstack-infra 2013-08-20T14:47:39 *** senk has joined #openstack-infra 2013-08-20T14:50:30 *** derekh has quit IRC 2013-08-20T14:50:40 mordred: its for pycadf library (a new lib for audit data) -- i did not include a min since some changes were still being made aruond time it was added 2013-08-20T14:51:37 *** SergeyLukjanov has joined #openstack-infra 2013-08-20T14:57:30 *** dina_belova has joined #openstack-infra 2013-08-20T14:57:40 *** david-lyle has quit IRC 2013-08-20T14:58:19 *** cthulhup has joined #openstack-infra 2013-08-20T15:01:54 *** sandywalsh has quit IRC 2013-08-20T15:03:56 *** ryanpetrello has joined #openstack-infra 2013-08-20T15:04:23 *** wu_wenxiang has joined #openstack-infra 2013-08-20T15:04:51 gordc: hey, i noticed that this didn't automatically change the status/assignee of the bug in launchpad: https://review.openstack.org/#/c/42904/ 2013-08-20T15:05:10 was going to ask dhellmann if the pycadf project is hooked up to launchpad via gerrit for status changes 2013-08-20T15:06:21 mriedem: it probably isn't hooked up correctly. i created the launchpad project so good chance i mucked it up :) 2013-08-20T15:06:24 I find some commit didn't start check for a long time, for example: https://review.openstack.org/#/c/38963/ and https://review.openstack.org/#/c/42794/ 2013-08-20T15:08:19 *** ruhe has joined #openstack-infra 2013-08-20T15:08:37 *** sridevi has joined #openstack-infra 2013-08-20T15:08:57 *** xchu has quit IRC 2013-08-20T15:09:24 wu_wenxiang: leave a comment with "recheck no bug"; we had to restart zuul yesterday and it lost its queue 2013-08-20T15:09:30 Hi could someone help me with https://review.openstack.org/#/c/34801/ 2013-08-20T15:10:13 I see "ERROR:root:Could not find any typelib for GnomeKeyring" failures 2013-08-20T15:12:04 *** ^d has joined #openstack-infra 2013-08-20T15:12:04 *** ^d has joined #openstack-infra 2013-08-20T15:12:14 *** xBsd has quit IRC 2013-08-20T15:12:21 jeblair: Thanks 2013-08-20T15:13:43 *** SlickNik has quit IRC 2013-08-20T15:13:51 *** vogxn has quit IRC 2013-08-20T15:14:41 *** SlickNik has joined #openstack-infra 2013-08-20T15:15:59 *** pabelanger has joined #openstack-infra 2013-08-20T15:16:40 *** wu_wenxiang has quit IRC 2013-08-20T15:17:04 *** david-lyle has joined #openstack-infra 2013-08-20T15:17:13 *** sandywalsh has joined #openstack-infra 2013-08-20T15:17:41 jeblair: Can I bug you to take a peek at this review? https://review.openstack.org/#/c/42685/2 2013-08-20T15:19:28 *** UtahDave has joined #openstack-infra 2013-08-20T15:19:58 or clarkb for that matter 2013-08-20T15:20:38 *** ruhe has quit IRC 2013-08-20T15:21:07 ryanpetrello: i'm hacking on a fix for a production problem we've been having right now, but i will make it a point to review it today if the rest of the team hasn't taken care of it 2013-08-20T15:21:19 thanks 2013-08-20T15:21:39 this obviously takes a back seat :) 2013-08-20T15:22:50 *** ruhe has joined #openstack-infra 2013-08-20T15:23:17 gordon chung proposed a change to openstack/requirements: assign a min version to pycadf https://review.openstack.org/42923 2013-08-20T15:23:29 *** reed has joined #openstack-infra 2013-08-20T15:24:10 *** dina_belova has quit IRC 2013-08-20T15:24:55 *** sridevi has quit IRC 2013-08-20T15:30:27 ryanpetrello: done 2013-08-20T15:30:32 jeblair: Monty approved it, thanks 2013-08-20T15:30:38 jeblair: anything I can do to help you? 2013-08-20T15:30:41 (thanks) 2013-08-20T15:36:41 A change was merged to openstack-infra/config: Add WSME to StackForge. https://review.openstack.org/42685 2013-08-20T15:39:17 *** nayward has quit IRC 2013-08-20T15:42:14 *** afazekas_no_irq is now known as afazekas 2013-08-20T15:42:16 *** thomasbiege has joined #openstack-infra 2013-08-20T15:43:52 *** vogxn has joined #openstack-infra 2013-08-20T15:45:42 jeblair: seems we have no core members on stackforge/pyghmi we did before the rename 2013-08-20T15:46:03 *** rnirmal has joined #openstack-infra 2013-08-20T15:47:28 *** zehicle_at_dell has joined #openstack-infra 2013-08-20T15:49:48 NobodyCam: looking 2013-08-20T15:50:12 thnank you mordred :) 2013-08-20T15:51:31 Monty Taylor proposed a change to openstack-infra/config: Rename python-impi acl file to pyghmi https://review.openstack.org/42932 2013-08-20T15:51:44 w00t 2013-08-20T15:51:47 NobodyCam: should be fixed soon 2013-08-20T15:51:50 *** changbl has quit IRC 2013-08-20T15:51:52 :) TY 2013-08-20T15:52:09 mordred: shouldn't you be burning things about now? 2013-08-20T15:52:25 NobodyCam: soon 2013-08-20T15:52:28 :) 2013-08-20T15:53:21 *** mrodden has quit IRC 2013-08-20T15:54:53 *** davidhadas has quit IRC 2013-08-20T15:55:15 *** ruhe has quit IRC 2013-08-20T15:58:49 A change was merged to openstack-infra/config: Rename python-impi acl file to pyghmi https://review.openstack.org/42932 2013-08-20T15:59:28 *** xBsd has joined #openstack-infra 2013-08-20T16:00:17 morning 2013-08-20T16:01:04 good morning clarkb 2013-08-20T16:01:17 mordred jeblair: which production issue? 2013-08-20T16:01:29 clarkb: I'm assuming the thing from yesterday 2013-08-20T16:02:06 *** sridevi has joined #openstack-infra 2013-08-20T16:02:19 clarkb: if you have a second, a ton of these: https://review.openstack.org/#/q/watchedby:mordred%2540inaugust.com+-label:CodeReview%253C%253D-1+-label:Verified%253C%253D-1+-label:Approved%253E%253D1++-status:workinprogress+-status:draft+-is:starred+-owner:mordred%2540inaugust.com,n,z 2013-08-20T16:02:28 mordred: which one :) it was like a horrible train wreck 2013-08-20T16:02:38 clarkb: could use a second +2 and are trivial changes 2013-08-20T16:03:12 *** sridevi has quit IRC 2013-08-20T16:03:14 mordred ok I have a couple things I want to fix while I am thinking of them but can look at those after 2013-08-20T16:03:45 clarkb: k. they're not important, but most of them are simple enough to be 'while drinking first cup of coffee' fodder 2013-08-20T16:03:49 mordred jeblair what do you think of adding something like celery.contrib.rdb to zuul for stack traces and remote pdb 2013-08-20T16:04:02 oy 2013-08-20T16:04:11 something about using celery in a project that uses gear seems weird 2013-08-20T16:04:13 I would simplify and vendor it 2013-08-20T16:04:41 *** mrodden has joined #openstack-infra 2013-08-20T16:04:54 mordred forget it is celery :) but their contrib.rdb module seems relatively decent and they have tests for it 2013-08-20T16:05:41 neat 2013-08-20T16:05:51 *** gyee has joined #openstack-infra 2013-08-20T16:05:53 then why not just requirements celery? 2013-08-20T16:06:42 we could do that too... seems heavy for something like a contrib module. I could go either way vendor or require 2013-08-20T16:06:45 mordred: should that merge have fixed us? 2013-08-20T16:07:19 NobodyCam: it'll take a minute 2013-08-20T16:07:27 ahh ok :) TY 2013-08-20T16:07:50 NobodyCam: we have to wait for the git pull cron followed by the puppet agent - so it could be as long as 30 minutes from merge 2013-08-20T16:08:16 *** jfriedly has joined #openstack-infra 2013-08-20T16:08:44 *** gordc has left #openstack-infra 2013-08-20T16:09:04 clarkb: also, your haproxy patch has 3 +2's : https://review.openstack.org/#/c/42784/ so I think whenever you want to land that and ride shotgun, you know, whatever 2013-08-20T16:11:51 *** odyssey4me4 has quit IRC 2013-08-20T16:11:53 mordred, clarkb: i am reworking nodepool (as i mentioned yesterday) 2013-08-20T16:12:14 *** pabelanger has quit IRC 2013-08-20T16:13:09 clarkb: the celery thing is heavyweight. i don't think we need a full remote debugger, we just need better logging, and the ability to get a stacktrace if something is stuck... 2013-08-20T16:13:13 jeblair: It needs an update. because the proxy is a single source we need to bump xinetd limits... i will propose that shortly 2013-08-20T16:13:32 *** thomasbiege has quit IRC 2013-08-20T16:13:43 testing 42784 here now 2013-08-20T16:13:52 clarkb: and that's just for a desparate situation -- in reality we should always be able to figure out what's going on from logs. this is perhaps the first time we've been unable to do that with zuul. :( 2013-08-20T16:14:06 jeblair: ok, I figured remote debugger would give us that and more, but can just log stacktraces as a start 2013-08-20T16:16:09 clarkb: there are some errors for 42784, investigating and drafting up comment now 2013-08-20T16:16:54 pleia2 thanks. /me -> office 2013-08-20T16:22:24 Russell Bryant proposed a change to openstack-infra/config: Disable tempest in the cells job https://review.openstack.org/42898 2013-08-20T16:22:49 *** boris-42 has quit IRC 2013-08-20T16:24:40 *** dina_belova has joined #openstack-infra 2013-08-20T16:26:11 mordred: how long does it generally take for merged openstack-infra/config projects to show up in github.com/stackforge ? 2013-08-20T16:26:47 ryanpetrello: usually quicker than this - let me look 2013-08-20T16:26:50 thx 2013-08-20T16:27:43 Monty Taylor proposed a change to openstack-infra/config: Make the gitweb links in gerrit point to git.o.o https://review.openstack.org/42694 2013-08-20T16:27:46 *** pabelanger has joined #openstack-infra 2013-08-20T16:29:17 *** markmc has quit IRC 2013-08-20T16:32:54 pleia2 try stopping xinetd first. It has port 9418 2013-08-20T16:33:53 or rather kick it to pick up the new config 2013-08-20T16:34:13 *** nicedice_ has joined #openstack-infra 2013-08-20T16:34:23 clarkb: ah, yeah! it didn't pick up the new config, restarting it then starting haproxy is fine 2013-08-20T16:36:23 *** xBsd has quit IRC 2013-08-20T16:36:41 *** psedlak has quit IRC 2013-08-20T16:37:02 cool I eill encode into puppet 2013-08-20T16:38:10 *** adalbas has joined #openstack-infra 2013-08-20T16:41:49 *** dina_belova has quit IRC 2013-08-20T16:42:08 *** pycabrera has joined #openstack-infra 2013-08-20T16:42:33 *** nati_ueno has joined #openstack-infra 2013-08-20T16:42:35 *** kgriffs has joined #openstack-infra 2013-08-20T16:43:05 *** nati_ueno has joined #openstack-infra 2013-08-20T16:43:06 *** pblaho has quit IRC 2013-08-20T16:43:07 having some trouble getting it to clone with haproxy enabled, browsing logs 2013-08-20T16:43:12 hey guys, Kurt here from the Marconi team. We'd like to enable logging and/or meetbot for #openstack-marconi - what's the recommended way to do this? 2013-08-20T16:43:48 host it ourselves, or is there a shared bot? 2013-08-20T16:43:50 *** cppcabrera has quit IRC 2013-08-20T16:43:57 *** pycabrera is now known as cppcabrera 2013-08-20T16:44:07 kgriffs: there is a shared bot, hang on, I'll grab a recent review as an example 2013-08-20T16:44:14 modules/gerritbot/files/gerritbot_channel_config.yaml 2013-08-20T16:44:24 kgriffs: I think that's it. ^^ 2013-08-20T16:44:24 *** alexpilotti has quit IRC 2013-08-20T16:44:41 kgriffs: https://review.openstack.org/#/c/41512/ 2013-08-20T16:44:45 pleia2: mine's not so recent, but https://review.openstack.org/#/c/21696/ 2013-08-20T16:44:46 heh 2013-08-20T16:44:48 for logging it's modules/openstack_project/manifests/eavesdrop.pp 2013-08-20T16:44:52 not gerritbot 2013-08-20T16:45:03 cool, thanks! 2013-08-20T16:45:29 gerritbot is the one that tells you updates in reviews merges and things :) 2013-08-20T16:45:30 actually, I think we are in gerritbot 2013-08-20T16:45:52 yup, we have gerritbot running as of yesterday. :D 2013-08-20T16:45:54 mordred: that seemed to work, thx :) 2013-08-20T16:45:59 kgriffs: once it's in eavesdrop you get logs up on http://eavesdrop.openstack.org/ 2013-08-20T16:46:14 I noticed, however that one of the groups was created - https://review.openstack.org/#/admin/groups/202,members - while the other, wsme-ptl, wasn't 2013-08-20T16:46:17 mordred: I am, too. I was going to wait the number of days specified in https://wiki.openstack.org/wiki/Governance/Approved/CoreDevProcess but I don't have 2013-08-20T16:46:41 pleia2: excellent 2013-08-20T16:48:16 mordred: added Alex_Gaynor to requirements-core group in gerrit 2013-08-20T16:48:28 Clark Boylan proposed a change to openstack-infra/config: Proxy git-daemon with haproxy. https://review.openstack.org/42784 2013-08-20T16:48:56 pleia2: ^ slightly updated. You may want to try those settings as the xinetd ACLs are slightly relaxed to be more friendly to haproxy 2013-08-20T16:49:13 clarkb: great, thanks 2013-08-20T16:49:32 Monty Taylor proposed a change to openstack-dev/pbr: Rework run_shell_command https://review.openstack.org/42337 2013-08-20T16:49:32 Monty Taylor proposed a change to openstack-dev/pbr: Use wheel by default https://review.openstack.org/41255 2013-08-20T16:51:41 ryanpetrello: you are now in wsme-core, so you should be able to add other people 2013-08-20T16:51:45 as you see fix 2013-08-20T16:51:46 fit 2013-08-20T16:51:58 ryanpetrello: poking wsme-ptl 2013-08-20T16:52:07 *** SlickNik has quit IRC 2013-08-20T16:52:07 awesome, and *thank you* 2013-08-20T16:52:31 NobodyCam: you should be set 2013-08-20T16:52:44 *** SlickNik has joined #openstack-infra 2013-08-20T16:52:47 ryanpetrello: I'm excited to have wsme moved in! 2013-08-20T16:52:51 *** alexpilotti has joined #openstack-infra 2013-08-20T16:53:28 mordred: cdevienne is looking forward to having more contributors :-) 2013-08-20T16:53:32 mordred: Thank you !! 2013-08-20T16:53:34 dhellmann: :) 2013-08-20T16:53:47 *** kgriffs has left #openstack-infra 2013-08-20T16:54:09 *** afazekas has quit IRC 2013-08-20T16:54:24 dhellmann: while you're here, could I get a second +2 on https://review.openstack.org/#/c/42515/ ? I have another patch that's wanting it and I'm trying to clear as much of my outstanding niggly stuff before I am out today 2013-08-20T16:54:41 mordred: sure, looking now 2013-08-20T16:54:56 dhellmann: (there's two other in requirements that could probably use love as well) 2013-08-20T16:56:11 mordred: I've got a standup in 3 minutes, but after that can look at anything you'd like reviewed 2013-08-20T16:57:00 pleia2: anything I can do to help testing/debug git-daemon? 2013-08-20T16:57:07 Alejandro Cabrera proposed a change to openstack-infra/config: feat: add marconi channel to eavesdrop https://review.openstack.org/42956 2013-08-20T17:00:21 clarkb: the patch helps us stop losing the puppet lottery (xinetd should have to look at the file it's subscribed to first before haproxy stuff happens) but still unable to clone from git:// with it enabled, looking for haproxy related logs now 2013-08-20T17:00:32 *** ladquin has joined #openstack-infra 2013-08-20T17:01:23 *** fbo is now known as fbo_away 2013-08-20T17:04:54 *** jerryz has joined #openstack-infra 2013-08-20T17:04:59 *** morganfainberg|a is now known as morganfainberg 2013-08-20T17:08:16 gosh, looking for issues with git specifically is a fun google-fu problem 2013-08-20T17:08:31 *** dprince has quit IRC 2013-08-20T17:09:06 Alex_Gaynor: ping 2013-08-20T17:14:16 pleia2: is it like googling for Go? 2013-08-20T17:14:39 yeah, and screen(1) :) 2013-08-20T17:14:57 might be an issue with my test isntance though, it doesn't have a fqdn for one 2013-08-20T17:15:50 i cannot, for the life of me, figure out how to adjust bugtask metadata for git-review on bug 1179008 (trying to set it to fix-committed for example). tried repeatedly over the past few days and every time i get a launchpad "timeout error..." ideas? 2013-08-20T17:15:52 Clark Boylan proposed a change to openstack-infra/zuul: SIGUSR2 logs stack traces for active threads. https://review.openstack.org/42959 2013-08-20T17:15:52 Launchpad bug 1179008 in python-neutronclient "rename requires files to standard names" [Medium,In progress] https://launchpad.net/bugs/1179008 2013-08-20T17:15:59 and ipaddress might show up weird on hpcloud (the local address the machine thinks it has in `ip addr` is not the public address 2013-08-20T17:16:04 * pleia2 manual tweaks 2013-08-20T17:16:49 *** vipul is now known as vipul-away 2013-08-20T17:17:07 pleia2: hi there 2013-08-20T17:17:16 jeblair: ^ 42959 is a bit of a WIP but I figured I would get that out sooner than later. I am still working on testing it (is the easiest way to do that to write a unittest?) 2013-08-20T17:18:47 fungi: it times out for me too. Maybe we attached too many projects to that bug? 2013-08-20T17:19:32 koolhead17: hey, hope you're enjoying your stay in SF :) 2013-08-20T17:22:11 *** vipul-away is now known as vipul 2013-08-20T17:23:00 pleia2: i am. :) 2013-08-20T17:23:01 clarkb: lgtm; you might need to actually run it in order to test it. also, i think there is something messed up with signals and using the internal gear server. 2013-08-20T17:23:01 lets catch up sometime over weekend 2013-08-20T17:23:07 jeblair: oh lovely 2013-08-20T17:23:13 * koolhead17 waves jeblair mordred clarkb & everyone :D 2013-08-20T17:23:26 hey koolhead17 - enjoyin SF? 2013-08-20T17:24:02 yes sir. its great 2013-08-20T17:24:10 :) 2013-08-20T17:25:36 i might be in seattle for a day 2013-08-20T17:25:56 koolhead17: one day is not enough for seattle :P 2013-08-20T17:25:57 morganfainberg: pong 2013-08-20T17:26:12 clarkb: i know :( 2013-08-20T17:26:21 jeblair: is there still a dev zuul that I can use to test within a running system? 2013-08-20T17:26:35 clarkb: won`t mind coming to portland for beer for few hr though. :D 2013-08-20T17:26:38 Alex_Gaynor: hi there 2013-08-20T17:27:00 Alex_Gaynor: hey, wanted to follow up with you regarding: https://review.openstack.org/#/c/42455/ (since you, in theory could bump up to a +2 now, btw, gratz on core for requirements) 2013-08-20T17:27:34 clarkb: so netstat tells me git daemon isn't even running when not on the default port, so trying to fix that now 2013-08-20T17:27:39 jeblair: I have at least one small updated to that. I realized that a reconfigure will also reconfigure logging so I am just going to get the logger each time I need to dump stack traces 2013-08-20T17:27:59 Alex_Gaynor: see if there was any outstanding concerns, since thats thenext blocker for my caching stuff in keystone. 2013-08-20T17:28:13 pleia2: doesn't xinetd fork git-daemon's on demand as connections come in? 2013-08-20T17:28:51 *** vogxn has quit IRC 2013-08-20T17:29:04 however xinetd should be listening on that port 2013-08-20T17:29:04 morganfainberg: I don't think there are any outstanding concerns, but I'll have to give it a once over before +2ing :) I'll come around in a few minutes to it 2013-08-20T17:29:12 *** pcm_ has quit IRC 2013-08-20T17:29:22 clarkb: yeah, but it still should have: :::9418 :::* LISTEN 10606/xinetd 2013-08-20T17:29:28 as fungi says :) 2013-08-20T17:29:30 Alex_Gaynor: thanks! i appreciate it :) 2013-08-20T17:29:41 pleia2: haproxy will be 9418, xinetd on 29418 2013-08-20T17:30:04 right, haproxy shows up on 9418 and no xinetd at all 2013-08-20T17:30:19 can't get it to listen on 29418 2013-08-20T17:31:14 weird 2013-08-20T17:31:52 * pleia2 confirms it's not selinux 2013-08-20T17:32:06 *** pcm_ has joined #openstack-infra 2013-08-20T17:34:45 *** SergeyLukjanov has quit IRC 2013-08-20T17:35:35 jeblair: woot, I wrote a small script that sits in a while loop with that signal handler configured and it seems to work 2013-08-20T17:35:47 jeblair: much easier testing that way than getting a complete zuul running 2013-08-20T17:35:54 *** cthulhup has quit IRC 2013-08-20T17:36:53 Aug 20 17:36:39 git-vanilla xinetd[10709]: Service git expects port 9418, not 29418 2013-08-20T17:36:55 heh 2013-08-20T17:37:13 dear xinetd, do it anyway 2013-08-20T17:37:28 *** mgagne has joined #openstack-infra 2013-08-20T17:37:28 *** mgagne has quit IRC 2013-08-20T17:37:28 *** mgagne has joined #openstack-infra 2013-08-20T17:42:28 Anita Kuno proposed a change to openstack-dev/hacking: Testing how .html files are rendered by cgit. https://review.openstack.org/42961 2013-08-20T17:42:47 *** zul has quit IRC 2013-08-20T17:46:20 Alex_Gaynor: looks like dhellmann got to it before you. thanks :) 2013-08-20T17:46:56 morganfainberg: okey doke, sorry bout that, I'm writing some scripts to setup swift for some benchmarkming :) 2013-08-20T17:47:22 Alex_Gaynor: not a problem man, was just following up with people today about it. thanks again! 2013-08-20T17:48:07 Anita Kuno proposed a change to openstack-dev/hacking: Testing how .html files are rendered by cgit https://review.openstack.org/42961 2013-08-20T17:48:41 pleia2: maybe we should consider running it as a stand alone daemon? 2013-08-20T17:48:51 pleia2: and rely on haproxy to do the DDoS protection 2013-08-20T17:50:10 clarkb: so it looks like xinetd uses /etc/services to determine where it should bind stuff, by patching /etc/services I got it to work, but this seems sub-optimal 2013-08-20T17:51:22 (commented out the 9418 git lines, added ones for 29418) 2013-08-20T17:52:21 *** dina_belova has joined #openstack-infra 2013-08-20T17:52:43 pleia2: so cloning works now? its a start :) 2013-08-20T17:53:32 yeah! This is with haproxy running: git clone git://15.185.127.146/openstack-infra/config.git 2013-08-20T17:54:07 browsing git-daemon docs, the /etc/services thing may actually be more git daemon and less xinetd 2013-08-20T17:55:12 so maybe we do need to change /etc/services 2013-08-20T17:57:14 *** dina_belova has quit IRC 2013-08-20T17:58:00 ok 2013-08-20T17:58:07 pleia2: to add to your list of things to do, here is a patch consisting of an .html file I generated with rst2html: https://review.openstack.org/#/c/42961/ 2013-08-20T17:58:22 let me know how it looks 2013-08-20T17:59:00 pleia2: that seems hacky though 2013-08-20T17:59:32 *** cppcabrera is now known as cppcabrera_afk 2013-08-20T18:00:43 clarkb: yeah, so if we run it stand alone without --inetd we should be able to specify an alternate --port 2013-08-20T18:01:01 pleia2: I like that better 2013-08-20T18:03:11 I am not sure of the best way to do this, as "the centos way" is using xinetd to run services that don't have specific init scripts, git is just a command line "git daemon..." 2013-08-20T18:03:35 pleia2: ubuntu's git daemon package comes with an init script. we could vendor it for centos 2013-08-20T18:03:59 I am sure that the red hat folk in the channel want to beat me after saying that 2013-08-20T18:04:13 hehe 2013-08-20T18:04:51 so we'd just drop it in /etc/init.d/ ? I am really unfamiliar with rh init system stuff 2013-08-20T18:05:10 (well, after tweaking it to work properly, of course) 2013-08-20T18:07:00 Clark Boylan proposed a change to openstack-infra/zuul: SIGUSR2 logs stack traces for active threads. https://review.openstack.org/42959 2013-08-20T18:07:12 jeblair: ^ that comes with a test. Let me know what you think 2013-08-20T18:07:55 pleia2: yes, dropping it in /etc/init.d/ and having puppet ensure the service is enabled should be sufficient 2013-08-20T18:08:18 assuming that the debian/ubuntu script doesn't have a bunch of debianisms in it that centos won't like 2013-08-20T18:09:14 *** changbl has joined #openstack-infra 2013-08-20T18:09:21 clarkb: looking now, it does - hard coded paths, /etc/default references, might actually be worth rewriting 2013-08-20T18:10:21 there are useful things I can pull from it though, hacking away 2013-08-20T18:12:25 anteaya: ok, I'll have a look in a little bit 2013-08-20T18:13:42 or use one someone already wrote http://robescriva.com/blog/2009/01/13/git-daemon-init-scripts-on-centos-52/ 2013-08-20T18:13:57 k thanks 2013-08-20T18:14:07 * pleia2 frowns at no license 2013-08-20T18:14:51 ah, easy enough to write own 2013-08-20T18:16:59 pleia2: let me know if there is anything I can do to help 2013-08-20T18:17:11 I half feel like I threw my crazy haproxy idea over the wall >_> 2013-08-20T18:17:19 *** cthulhup has joined #openstack-infra 2013-08-20T18:17:22 was not my intention :) 2013-08-20T18:17:38 *** zul has joined #openstack-infra 2013-08-20T18:18:03 no worries, it mostly worked, certainly didn't anticipate it being so cranky about non-standard ports, it shouldn't be like this :) 2013-08-20T18:23:25 *** xBsd has joined #openstack-infra 2013-08-20T18:23:27 *** melwitt has joined #openstack-infra 2013-08-20T18:24:23 *** cthulhup has quit IRC 2013-08-20T18:25:00 pleia2: jeblair: mordred: Worth noting that the g-g-p times with https://git.o.o seem to be better than when against review.o.o on centos unittest slaves 2013-08-20T18:25:13 so maybe we should stop worrying too much about git:// 2013-08-20T18:27:45 hmm, maybe there is a way I can edit the server_args line to support port 2013-08-20T18:30:52 *** vipul is now known as vipul-away 2013-08-20T18:30:53 *** vipul-away is now known as vipul 2013-08-20T18:31:27 not so much 2013-08-20T18:32:16 need a staging server for activity.openstack.org 2013-08-20T18:34:11 clarkb: maybe, seems unlikely that if we point everything at https that there will be enough load on git:// to cause problems 2013-08-20T18:35:43 *** arezadr has joined #openstack-infra 2013-08-20T18:39:02 reed: do you want to write the puppet (we can point you to some docs), or do you want someone else to do it? 2013-08-20T18:39:30 jeblair, send me the puppet stuff, I'd like to learn 2013-08-20T18:39:46 (is that a good answer or what?) 2013-08-20T18:40:24 *** markmcclain has quit IRC 2013-08-20T18:40:26 reed: http://ci.openstack.org/sysadmin.html#adding-a-new-server is a good start :) 2013-08-20T18:40:37 are we waiting for anything specific for this patch: https://review.openstack.org/#/c/38177/ Use cgit server instead of github for everything There is quite the lineup of green +'s on it 2013-08-20T18:40:52 reed: it's the most perfect answer ever. :) 2013-08-20T18:41:14 * reed admires his most perfect answer ever, sipping coffee 2013-08-20T18:41:14 reed: http://ci.openstack.org/sysadmin.html#adding-a-new-server 2013-08-20T18:41:25 reed: you should actually start at the top of that doc 2013-08-20T18:41:31 anteaya: still working to tune the git server before we throw everything at it 2013-08-20T18:41:36 reed: it has background info, and also instructions on how to test 2013-08-20T18:41:44 pleia2: ah, okay 2013-08-20T18:41:44 reed: but the section i pointed to has the actual steps 2013-08-20T18:42:09 reed: and somewhere, there's mrmartin's change to add his staging server 2013-08-20T18:42:17 looking 2013-08-20T18:42:37 jeblair, oh, right... I can copy that too 2013-08-20T18:42:43 reed: https://review.openstack.org/#/c/42608/ 2013-08-20T18:42:57 sweet 2013-08-20T18:43:05 *** SergeyLukjanov has joined #openstack-infra 2013-08-20T18:43:37 reed, mrmartin: and sorry i haven't reviewed that yet. it is a high priority, after we get some of the operational issues we've been dealing with under control 2013-08-20T18:43:50 *** SergeyLukjanov has quit IRC 2013-08-20T18:43:54 (this week is very busy due to a feature freeze deadline) 2013-08-20T18:44:01 np, mrmartin is on vacation today anyway 2013-08-20T18:44:06 damn feature freeze 2013-08-20T18:45:35 clarkb, pleia2: git-daemon wants us to edit /etc/services to run it on another port? 2013-08-20T18:46:01 *** vipul is now known as vipul-away 2013-08-20T18:46:31 mordred: well, inetd does 2013-08-20T18:47:33 if running it from xinetd or using --inetd on the command line, you can't specify --port because it just does an /etc/services lookup and will only use what's in that file 2013-08-20T18:47:48 I vote that this is broken :) 2013-08-20T18:47:52 but it is what it is 2013-08-20T18:51:31 *** openstack` has joined #openstack-infra 2013-08-20T18:51:31 *** openstack has quit IRC 2013-08-20T18:52:04 *** pabelanger has quit IRC 2013-08-20T18:52:32 *** openstack` is now known as openstack 2013-08-20T18:52:53 *** boris-42 has joined #openstack-infra 2013-08-20T18:53:51 *** afazekas has joined #openstack-infra 2013-08-20T18:55:53 pleia2: it seems like a very poor design 2013-08-20T18:56:55 James E. Blair proposed a change to openstack-infra/nodepool: WIP: provider manager https://review.openstack.org/42973 2013-08-20T18:57:36 Clark Boylan proposed a change to openstack-infra/zuul: SIGUSR2 logs stack traces for active threads. https://review.openstack.org/42959 2013-08-20T18:58:04 mordred, clarkb: ^ that is my solution to the problems with rate limits we saw yesterday ^. i also think it's a bit cleaner and more reliable. 2013-08-20T18:58:27 jeblair: I will review after the meeting 2013-08-20T18:58:45 mordred, clarkb: it needs a little more work, and testing with a real provider instead of my fake one, but it's mostly there and worth a general review 2013-08-20T18:59:10 jeblair: the zuul change should be ready for review as well 2013-08-20T18:59:37 clarkb: thanks 2013-08-20T18:59:48 meeting time! 2013-08-20T18:59:54 almost 2013-08-20T19:01:10 *** AJaeger has joined #openstack-infra 2013-08-20T19:01:27 *** pabelanger has joined #openstack-infra 2013-08-20T19:02:31 *** mriedem1 has joined #openstack-infra 2013-08-20T19:03:25 *** cppcabrera_afk is now known as cppcabrera 2013-08-20T19:03:30 *** mriedem has quit IRC 2013-08-20T19:05:32 Hi infra team, I'd like to have some guidance and help on getting the Basic Install guide build now also for openSUSE - and thus on the docs.openstack.org 2013-08-20T19:06:05 annegentle guided me in https://review.openstack.org/#/c/41777/ to you. 2013-08-20T19:06:09 *** thomasbiege1 has joined #openstack-infra 2013-08-20T19:06:19 AJaeger: we are in our weekly meeting currently, so we may be a bit slow to answer, but will catch up after the meeting 2013-08-20T19:06:47 clarkb, sorry, didn't know. Ok, I'll stay around and let you finish your meeting. Thanks for the quick heads-up. 2013-08-20T19:07:01 *** gyee has quit IRC 2013-08-20T19:07:10 *** vipul-away is now known as vipul 2013-08-20T19:07:35 *** vipul is now known as vipul-away 2013-08-20T19:07:36 *** vipul-away is now known as vipul 2013-08-20T19:07:36 clarkb, btw if I should send an email or use other means, just tell me 2013-08-20T19:08:02 AJaeger: IRc is probably easiest, it will just be maybe an hour before we can really answer your questiosn 2013-08-20T19:08:30 clarkb, ok, thanks 2013-08-20T19:15:37 *** markmcclain has joined #openstack-infra 2013-08-20T19:15:42 So the amount of time between whe a job finishes on jenkins, and when zuul records it as done seems why too large. Are there any known bottlenecks there, and what can be done to improve that? 2013-08-20T19:15:52 *** fbo_away is now known as fbo 2013-08-20T19:17:22 Jenkins review on Gerrit get really readable! Nice 2013-08-20T19:18:59 *** dprince has joined #openstack-infra 2013-08-20T19:19:35 jeblair, pleia2: since the activity-staging server needs to have apache and mysql should I draw inspiration from static.pp for the include::apache and various mods?? 2013-08-20T19:20:49 *** kiall_ has joined #openstack-infra 2013-08-20T19:21:05 reed: yes 2013-08-20T19:21:10 cool 2013-08-20T19:22:08 Alex_Gaynor: link to an example change? 2013-08-20T19:22:21 nati_ueno: thanks! (jeblair did it) 2013-08-20T19:22:36 *** vipul is now known as vipul-away 2013-08-20T19:23:22 jeblair: Thanks! 2013-08-20T19:26:01 *** nati_ueno has quit IRC 2013-08-20T19:26:07 *** jerryz has quit IRC 2013-08-20T19:26:31 *** HenryG has joined #openstack-infra 2013-08-20T19:26:46 *** cthulhup has joined #openstack-infra 2013-08-20T19:27:23 jeblair: just random ones I'm noticing as they happen 2013-08-20T19:30:31 *** nati_ueno has joined #openstack-infra 2013-08-20T19:34:06 *** gordc has joined #openstack-infra 2013-08-20T19:34:48 *** thomasbiege1 has quit IRC 2013-08-20T19:37:27 *** cthulhup has quit IRC 2013-08-20T19:41:15 *** cthulhup has joined #openstack-infra 2013-08-20T19:42:16 *** xBsd has quit IRC 2013-08-20T19:43:24 Alex_Gaynor: don't forget about severed heads; 2013-08-20T19:43:35 *** vipul-away is now known as vipul 2013-08-20T19:43:48 Alex_Gaynor: the head of the queue was just severed because it failed a test, but it's still running its tests and won't report until they are done 2013-08-20T19:43:59 Alex_Gaynor: (scroll to the bottom of the gate queue to see it) 2013-08-20T19:44:47 jeblair: so the case I was looking at was teh top item in the gate queue 2013-08-20T19:44:53 s/queue/pipeline 2013-08-20T19:45:40 btw, i put up this change earlier today to help free up some jenkins resources over the next couple weeks: https://review.openstack.org/#/c/42898/ 2013-08-20T19:47:05 *** zul has quit IRC 2013-08-20T19:49:46 hey guys. 2013-08-20T19:50:32 just wanted to report in that review.openstack.org is being much slower than usual. 2013-08-20T19:51:02 SlickNik: yes, it's being used much more than usual 2013-08-20T19:51:22 SlickNik: yup, it is getting bogged down by all of the testing to test all of your code :) I think we just agreed to merge a change that will hopefully alleviate some of this 2013-08-20T19:51:33 jeblair: do you want to force merge that change or should I just go ahead and do it? 2013-08-20T19:51:42 clarkb: i'll do it 2013-08-20T19:51:56 SlickNik: with the icehouse feature freeze looming, lots of people are trying to submit/review/merge much more code volume than usual 2013-08-20T19:52:33 A change was merged to openstack-infra/devstack-gate: Use git.openstack.org as origin https://review.openstack.org/42693 2013-08-20T19:52:43 jeblair: thanks 2013-08-20T19:52:49 Cool, thanks! Understandable with the FF looming. 2013-08-20T19:52:58 And thanks for being on top of it (as usual). 2013-08-20T19:53:00 Chers. 2013-08-20T19:53:02 Cheers* 2013-08-20T19:53:28 SlickNik: in the mean time you will probably find that using git review -d and the gerrit ssh interface to be a little more responsive 2013-08-20T19:53:58 and do your reviews locally (not sure if you can do inline comments this way, but otherwise it should work) 2013-08-20T19:54:46 Anne Gentle proposed a change to openstack-infra/config: Ensure that the release.path.name is set for the Block Storage https://review.openstack.org/42984 2013-08-20T19:54:47 *** afazekas has quit IRC 2013-08-20T19:54:49 anybody know if there's a generalized sphinx upload hook for pythonhosted.org ? 2013-08-20T19:54:52 clarkb: I'm heading out to lunch in a couple minutes (might run a bit long), will finish up init script upon my return! 2013-08-20T19:54:53 that does e.g., http://pythonhosted.org/an_example_pypi_project/buildanduploadsphinx.html 2013-08-20T19:55:22 similar to what the rtfd hook does, but uploads directly to pythonhosted.org? 2013-08-20T19:55:32 if not, I'd be glad to experiment in writing one, just wanting to make sure it doesn't already exist... 2013-08-20T19:55:33 *** markmc has joined #openstack-infra 2013-08-20T19:55:33 ryanpetrello: we have not made one 2013-08-20T19:56:03 I wonder if doc_upload has the same permissions as how maintainer roles work 2013-08-20T19:56:09 at some point, I'd love to get a good general design/direction around rtfd/pythonhosted/docs.o.o 2013-08-20T19:56:10 i.e., if you're a maintainer, you can upload docs 2013-08-20T19:56:16 dhellmann, annegentle ^^ 2013-08-20T19:56:31 ryanpetrello: also, look at how we do pypi-upload 2013-08-20T19:56:34 ryanpetrello: note we don't use setup.py to upload stuff to pypi because ugh. Instead we have a wrapper around curl to do it so that we don't have to run arbitrary code 2013-08-20T19:56:35 mordred: I've met with Todd Morey in the last couple weeks to try to synch with www for design 2013-08-20T19:56:50 ryanpetrello: i lookd into it briefly 2013-08-20T19:56:53 ryanpetrello: it's probably more directly related to how we'd need to upload docs to pypi 2013-08-20T19:56:55 mordred: Sphinx does work well for dev docs 2013-08-20T19:56:58 ryanpetrello: it can be done by uplodaing a zipfile 2013-08-20T19:56:59 AJaeger: still around? 2013-08-20T19:57:08 clarkb, Yes. 2013-08-20T19:57:14 ryanpetrello: so basically, it would be like the pypi-upload job 2013-08-20T19:57:18 annegentle: main questoin is - which of the three available locations should we automatically upload to? 2013-08-20T19:57:23 annegentle: or - should we upload to all of them? 2013-08-20T19:57:31 mordred: ah 2013-08-20T19:57:32 AJaeger: ok, give me a quick minute to settle back into doing stuff and I will do my best to answer your questions about new doc jobs 2013-08-20T19:57:40 mordred: one place. 2013-08-20T19:57:42 mordred: we're looking at pythonhosted for wsme because that's one of the places it is already using 2013-08-20T19:57:51 *** dina_belova has joined #openstack-infra 2013-08-20T19:57:59 why not as many as you specify via hooks? 2013-08-20T19:58:00 *** SergeyLukjanov has joined #openstack-infra 2013-08-20T19:58:05 my preference is for rtfd.org, because that's what most people are doing for new projects 2013-08-20T19:58:06 if elect pythonhosted vs rtfd 2013-08-20T19:58:10 ryanpetrello: why clutter the internet? :) 2013-08-20T19:58:12 the submission process for those is quite different 2013-08-20T19:58:17 annegentle: +1 2013-08-20T19:58:24 no, I agree 2013-08-20T19:58:30 dhellmann: my issue with rtfd is we need the GA info to make good decisions about docs 2013-08-20T19:58:31 openstack projects should have their docs uploaded to docs.openstack.org 2013-08-20T19:58:34 just staying we should give folks the flexibility to choose 2013-08-20T19:58:35 for openstack stuff, I think we should just host it ourselves 2013-08-20T19:58:36 jeblair: yes 2013-08-20T19:58:39 stackforge projects can do whatever they want 2013-08-20T19:58:42 sure 2013-08-20T19:58:49 and we do give them the flexibility to do that right now. 2013-08-20T19:58:52 annegentle: right, this would just be for third-party or stackforge stuff 2013-08-20T19:58:53 jeblair: sure 2013-08-20T19:58:55 right, Doug and I are mostly referring to stackforge in this context 2013-08-20T19:58:55 dhellmann: ok 2013-08-20T19:59:18 *** ^demon has joined #openstack-infra 2013-08-20T19:59:19 *** ^demon has joined #openstack-infra 2013-08-20T19:59:24 just suggesting that stackforge folks may find a "auto-upload to pythonhosted.org on release" useful 2013-08-20T19:59:29 they currently have this for rtfd 2013-08-20T19:59:32 just considering another option 2013-08-20T19:59:35 yep 2013-08-20T19:59:59 I think we should allow pythonhosted, but encourage rtfd where possible 2013-08-20T20:00:00 ryanpetrello: ok. nice that it happens on upload 2013-08-20T20:00:05 +1 2013-08-20T20:00:08 ++ 2013-08-20T20:00:10 ryanpetrello: but there are good reasons to ci docs 2013-08-20T20:00:39 I'd probably encourage continuous publishing 2013-08-20T20:00:40 AJaeger: we configure all of our jenkins jobs using the Jenkins Job Builder, http://ci.openstack.org/jjb.html 2013-08-20T20:00:41 sure, s/on release/whenever is applicable 2013-08-20T20:00:58 annegentle: good point 2013-08-20T20:01:01 continuous, if it's right for your project/preference 2013-08-20T20:01:17 *** lcestari has quit IRC 2013-08-20T20:01:43 AJaeger: that page is a good starting point for learning how JJB works. With the help of that page you should be able to grab an existing doc job that does something similar to what you want and copy pasta as needed without losing too much understanding of what is going on 2013-08-20T20:01:51 *** ^d has quit IRC 2013-08-20T20:02:02 AJaeger: then the second thing you need to do is tell zuul to run that jenkins job when you need it to be run 2013-08-20T20:02:29 *** mikal has joined #openstack-infra 2013-08-20T20:03:00 AJaeger: https://github.com/openstack-infra/config/blob/master/modules/openstack_project/files/zuul/layout.yaml is where you do that. http://ci.openstack.org/zuul.html has a brief zuul intro and links ot more in depth docs 2013-08-20T20:03:32 AJaeger: so from a super high level your change will have two parts. 1. add job to jenkins with JJB and 2. tell zuul to run new job in layout.yaml 2013-08-20T20:04:22 clarkb: Thanks, I'll check how the current guides are build and see whether I need to duplicate that setup or can somehow hook into it... 2013-08-20T20:05:16 *** zehicle_at_dell has quit IRC 2013-08-20T20:06:22 Clark Boylan proposed a change to openstack-infra/config: Make mysql backup crons quiet. https://review.openstack.org/42785 2013-08-20T20:06:31 jeblair: mordred fungi ^ 2013-08-20T20:06:33 and now time for reviews 2013-08-20T20:07:05 *** mikal has quit IRC 2013-08-20T20:09:46 clarkb: lgtm. i'm popping out for lunch and then i'll try to review a few changes before my next meeting 2013-08-20T20:10:49 AJaeger: feel free to ask questions as they arise. I know I gave the high level info dump and wasn't very specific 2013-08-20T20:12:10 clarkb, that helped a lot - I got the right pointer. I'll propose a change in a few minutes for you to review that I didn't miss anything... 2013-08-20T20:13:37 *** mikal has joined #openstack-infra 2013-08-20T20:15:44 Andreas Jaeger proposed a change to openstack-infra/config: Build Basic Install Guide for openSUSE https://review.openstack.org/42988 2013-08-20T20:16:07 *** dmakogon_ has quit IRC 2013-08-20T20:16:17 clarkb, my feeling is just that I'm missing something. That was too easy ;) 2013-08-20T20:17:25 you guys aware of review.o.o being a slow today? 2013-08-20T20:17:42 vipul: yes, we are DDoSing it with the jenkins slaves 2013-08-20T20:18:02 ooh fun! 2013-08-20T20:18:25 we recently merged a devstack gate change that will point more tests to git.openstack.org which will hopefully alleviate the pressure on review.o.o but we need the currently running tests to flip over before we see 2013-08-20T20:19:10 James E. Blair proposed a change to openstack-infra/nodepool: Add ProviderManager https://review.openstack.org/42973 2013-08-20T20:21:12 vipul: this is the typical pre feature freeze rush that never fails to break something 2013-08-20T20:21:21 vipul: tl;dr you need to write more code during H1 :) 2013-08-20T20:21:22 *** pabelanger has quit IRC 2013-08-20T20:22:08 clarkb: h1 is for recovering from all the hangovers at the summit :D 2013-08-20T20:22:16 *** pcm_ has quit IRC 2013-08-20T20:23:23 *** HenryG has quit IRC 2013-08-20T20:24:37 *** mikal has quit IRC 2013-08-20T20:27:21 i think jenkins02 is experiencing a similar slowness as before; i've got jstack trying to get a thread dump; it is responding, but very slowly, and it has a bunch of offline nodes sitting aroind. 2013-08-20T20:27:59 clarkb, fungi: ^ i uploaded a polished version of the providermanager change; i'm about to start live-testing it 2013-08-20T20:28:14 jeblair: ok, it is next up in my queue. 2013-08-20T20:28:24 clarkb, fungi: i think i will also do something similar to serialize jenkins access, and try to deploy both of those together. 2013-08-20T20:28:49 Clark Boylan proposed a change to openstack-infra/devstack-gate: Replace review.o.o with git.o.o. https://review.openstack.org/42989 2013-08-20T20:28:58 jeblair: ^ I noticed that needed doing 2013-08-20T20:29:12 clarkb: no it doesn't we don't use those anymore 2013-08-20T20:29:32 jeblair: well it needs doing at least for the README 2013-08-20T20:29:58 jeblair: the image building is elsewhere, maybe there should be a clean up d-g commit then do the git stuff on top of it 2013-08-20T20:30:38 clarkb: ok, sure, we can change the readme. i'm pretty sure the image building, whether run manually or nightly, is not causing current performance problems, so i deferred it 2013-08-20T20:31:00 similarly, i have deferred removing those things until there's a replacement 2013-08-20T20:31:13 (for manually running) 2013-08-20T20:31:52 clarkb: but can we at least avoid adding that to the gate queue until it's not busy? 2013-08-20T20:31:56 jeblair: ya 2013-08-20T20:32:02 I will WIP it 2013-08-20T20:32:08 *** pabelanger has joined #openstack-infra 2013-08-20T20:32:14 Jim Branen proposed a change to openstack/requirements: Allow use of hp3parclient 1.1.0. https://review.openstack.org/42991 2013-08-20T20:32:34 russellb: https://jenkins01.openstack.org/job/gate-nova-python26/1366/console seems to be a fairly frequent test failure 2013-08-20T20:33:05 jeblair: FYI ^ I think that has semi broken the gate (only nova runs that test so only nova is affected) 2013-08-20T20:33:49 boris-42: ^^^ 2013-08-20T20:34:02 boris-42: can you help dig into that? since you (and your team) have been working most in that area 2013-08-20T20:35:51 anteaya: when you get a moment, would you look at the scrollback in the meeting channel 2013-08-20T20:36:13 anteaya: and the discussion of setting up a repo that we'll use for voting for TC motions? 2013-08-20T20:36:24 russellb, I think someone from his team submitted a patch 2013-08-20T20:36:26 * markmc digs it up 2013-08-20T20:36:39 mordred: I was following some of that 2013-08-20T20:36:45 *** kiall_ is now known as Kiall 2013-08-20T20:36:51 russellb I am here 2013-08-20T20:36:52 mordred: am I the resource volunteered for duty? 2013-08-20T20:36:53 russellb, it was victor, https://review.openstack.org/#/c/42649/ 2013-08-20T20:36:56 :D 2013-08-20T20:37:15 anteaya: yup 2013-08-20T20:37:20 markmc: nicedice_ 2013-08-20T20:37:21 okey dokey smokey 2013-08-20T20:37:22 err, nice. 2013-08-20T20:37:22 russellb yeah this is already solve 2013-08-20T20:37:25 anteaya: you know, if you want :) 2013-08-20T20:37:31 yeah yeah yeah 2013-08-20T20:37:35 clarkb: looks like we have a patch up for that ... need to get it reviewed/merged though 2013-08-20T20:37:53 so the way I understand it, I go back through the TC meeting logs and pull out past decisions 2013-08-20T20:37:54 russellb: markmc: great. Note that any nova changes approved before that one probably won't merge 2013-08-20T20:38:01 *** mikal has joined #openstack-infra 2013-08-20T20:38:05 and offer them up as patches to the repo 2013-08-20T20:38:10 that I am about to create 2013-08-20T20:38:20 to gather the history 2013-08-20T20:38:24 clarkb, it only happens like 1 in every 5 times from what I've seen 2013-08-20T20:38:42 is that one of the tasks, apart from creating the repo itself 2013-08-20T20:39:00 ttx: what do we want to call this TC decision repo? 2013-08-20T20:39:37 at the very least, I will learn a lot about the history of the TC 2013-08-20T20:42:23 clarkb: that change is approved now 2013-08-20T20:42:30 russellb nice 2013-08-20T20:42:33 russellb thnaks 2013-08-20T20:42:34 =) 2013-08-20T20:42:35 *** SergeyLukjanov has quit IRC 2013-08-20T20:42:44 boris-42: yep, np 2013-08-20T20:43:22 clarkb, mordred: i think jstack is stuck in its deadlock detection. 2013-08-20T20:44:15 jeblair: wow 2013-08-20T20:44:55 *** dina_belova has quit IRC 2013-08-20T20:45:03 *** cthulhup has quit IRC 2013-08-20T20:45:41 *** cthulhup has joined #openstack-infra 2013-08-20T20:46:09 load on git.o.o is ~18 and under 1 on review.o.o 2013-08-20T20:46:16 jeblair: that is an impressive feat 2013-08-20T20:46:24 Russell Bryant proposed a change to openstack-infra/config: Disable tempest in the cells job https://review.openstack.org/42898 2013-08-20T20:46:28 i'm attaching the debugger and will try that way 2013-08-20T20:46:30 clarkb: woot! 2013-08-20T20:46:31 jeblair: I am working my way through the nodepool client manager change right now 2013-08-20T20:47:32 *** cppcabrera is now known as cppcabrera_afk 2013-08-20T20:48:01 mordred: tags for getting pbr with swift...got a few minutes? 2013-08-20T20:48:56 Alex_Gaynor: do you know, is there a way to get a unicode string directly from a memoryview, rather than copy to bytestrnig, then decode to unicode string? 2013-08-20T20:48:56 clarkb, mordred: i think it's slow because there are so many nodes still attached to it (which is true because it is slow) 2013-08-20T20:49:37 mordred: got a few mins? 2013-08-20T20:49:59 lifeless: apparently! codecs.utf_8_codecs(memoryview) seems to wokr (for example) 2013-08-20T20:50:18 *** cthulhup has quit IRC 2013-08-20T20:50:35 Alex_Gaynor: ahha, thanks! 2013-08-20T20:50:37 mordred: i think jenkins02 needs to be stopped, and have all the nodes removed from its config.xml; all related nodes deleted from nova, and then started again. 2013-08-20T20:50:55 jeblair: that is no good. What do you think about an artificial throttle in zuul or nodepool, so that we can at least prevent it from overrunning itself 2013-08-20T20:51:13 Alex_Gaynor: though 2.7's codecs module has no utf_8_codecs attribute 2013-08-20T20:51:39 clarkb: i mentioned that i wanted to serialize access to jenkins, do you want something else? 2013-08-20T20:52:29 lifeless: codecs.utf_8_decode(m) 2013-08-20T20:52:38 jeblair: question on Gerrit comment syntax. I noticed recently that 'SUCCESS' is green.... and 'FAILED' is red. Is that HTML formatting that does that? or some sort of magic gerrit syntax you'd need to use? 2013-08-20T20:52:46 Alex_Gaynor: ahha! cool. 2013-08-20T20:52:55 jeblair: I think serializing access to jenkins is part of the answer, doing more to add a configurable queue length so that anything going over some limit blocks 2013-08-20T20:53:27 clarkb: if we wanted the whole system to be slow, we could have done nothing. it was self limiting earlier. 2013-08-20T20:53:30 clarkb: and still is 2013-08-20T20:53:37 clarkb: the point is to actually be able to run all of the tests we need to run 2013-08-20T20:53:49 clarkb: that's why we're scaling jenkins horizontally and adding more masters 2013-08-20T20:53:55 jeblair: I am not suggesting to make it slow, you can still make the limit arbitrarily high 2013-08-20T20:54:07 clarkb: what are you suggesting then? 2013-08-20T20:54:20 jeblair: but in cases like this we would be much more better off putting a limit on how fast it can be 2013-08-20T20:54:26 clarkb: how fast what? 2013-08-20T20:54:39 jeblair: jobs per hour 2013-08-20T20:54:48 clarkb: are you talking about zuul? 2013-08-20T20:54:50 jeblair: or nodepool concurrent nodes 2013-08-20T20:55:19 jeblair: I am think of zuul and or nodepool. They can both be throttled to take some of the pressure off of jenkins and gerrit 2013-08-20T20:55:42 clarkb: okay, so we just merged a change that will cause tests to not touch gerrit 2013-08-20T20:56:04 *** mikal has quit IRC 2013-08-20T20:56:04 clarkb: zuul accesses gerrit serially when creating its changes 2013-08-20T20:56:17 ideally, we have just done quite a lot to take the pressure off of gerrit 2013-08-20T20:56:25 clarkb: so what pressure on gerrit do you want to relieve? 2013-08-20T20:56:33 anteaya: openstack/governance ideally, though it's a bit overreaching 2013-08-20T20:56:36 jeblair: our major problem today and yesterday appears to be a thundering herd. If we can let them thunder at a tunable pace we should be able to reign in when jenkins runs faster than it shoes can move 2013-08-20T20:56:50 clarkb: i think you are over-generalizing 2013-08-20T20:56:58 but openstack/tech-governance is a mouthful 2013-08-20T20:57:04 ttx: I'm fine with openstack/governance 2013-08-20T20:57:17 jeblair: I am trying to be generic, because next milestone it will be some other DDoS 2013-08-20T20:57:25 and it's not as if we never renamed any project in the past 2013-08-20T20:57:30 do we want it in the openstack/ namespace or the openstack-infra/ namespace do you think, ttx? 2013-08-20T20:57:47 well if one thing is openstack/, that would be it 2013-08-20T20:57:53 very good 2013-08-20T20:57:53 jeblair: and a generic pace enforcment will help us at least keep moving rather than needing emergency fixes to keep going 2013-08-20T20:58:15 * anteaya goes back to looking up docs for creating a new git repo 2013-08-20T20:58:40 clarkb: overgeneralizing a problem does not help provide a solution. how do you write a patch to "don't cause problems"? 2013-08-20T20:58:48 clarkb: your second point 2013-08-20T20:58:54 clarkb: pressure on jenknis 2013-08-20T20:59:04 mordred: your cookiecutter thing looks good -- looks like an automated mordred-goes-to-fix-your-project merge 2013-08-20T20:59:07 clarkb: we have seen that jenkins can run a lot of jobs, and have a lot of slaves 2013-08-20T20:59:15 ttx: I don't have any expectation of any gate or check tests for openstack/governance 2013-08-20T20:59:27 clarkb: but right now, we've seen issues with slaves not being removed from jenkins 2013-08-20T20:59:55 anteaya: we could enforce some common template 2013-08-20T21:00:06 clarkb: i don't know why that is. there may be a bug in the gearman-plugin. the 'thundering herd' of deleted nodes may just be too much contention for that kind of operation. 2013-08-20T21:00:07 anteaya: but not yet maybe 2013-08-20T21:00:09 ttx: got one in mind? 2013-08-20T21:00:14 ttx: very good 2013-08-20T21:00:27 clarkb: and as you observed earlier, jenkins does not do well if you do lots of things at once 2013-08-20T21:00:31 jeblair: ya 2013-08-20T21:00:44 clarkb: so serializing access to adding and removing nodes from jenkins may help with that 2013-08-20T21:00:52 clarkb: at least, we might get a better idea of what is going on 2013-08-20T21:01:15 jeblair: I am all for fixing the specific bottlenecks because I want to be able to do as many operations as possible. But I also think having some way of pull back so that everything doesn't shut down is useful 2013-08-20T21:01:19 clarkb: anyway, you've had some good suggestions, and i'm trying to implement solutions for the problems we've seen based on them 2013-08-20T21:01:34 clarkb: that sounds great. i have no idea what you're talking about though. 2013-08-20T21:02:50 ttx: who do you want as core for openstack/governance? 2013-08-20T21:03:18 jeblair: I am not sure where we would want the control to go (proabably in zuul) but being able to tell it launch at most 300 jobs per hour or some number of jobs per minute/second etc will be useful so that in cases like now we can continue to run jenkins jobs without making the problem worse. 2013-08-20T21:03:36 clarkb: why would we want to do that? what problem does that solve? 2013-08-20T21:03:41 anteaya: that's where it gets tricky. You want +2/-2 for TC members. And APRV for the chair (me) 2013-08-20T21:03:48 * ttx is in a meeting 2013-08-20T21:03:49 jeblair: I also see that as being useful so that it can be tied to a PID loop (or similar) where it automatically increases the limit and decreases it based on job throughput or some other metric 2013-08-20T21:04:03 ttx: okay, sorry more questions later 2013-08-20T21:04:06 jeblair: right now it would potentially give jenkisn a chance to catch back up on its own 2013-08-20T21:04:12 clarkb: catch up with what? 2013-08-20T21:04:15 jeblair: deleting nodes 2013-08-20T21:04:23 clarkb: oh, i don't think that has anything to do with it 2013-08-20T21:04:26 jeblair: or $otheroperation that has slowed to a crawl 2013-08-20T21:04:29 clarkb: it can't delete nodes because it's deleting nodes 2013-08-20T21:04:35 clarkb: not because it's running jobs 2013-08-20T21:04:55 clarkb: there _are_ things we can control to tune this whole system, but we need to tune the right things. 2013-08-20T21:05:06 *** gyee has joined #openstack-infra 2013-08-20T21:05:25 *** pblaho has joined #openstack-infra 2013-08-20T21:05:37 clarkb: if you want to rate-limit starting or stopping jobs, that can be done with zuul and gearman, in how they dispatch jobs 2013-08-20T21:05:59 clarkb: but setting an arbitrary jobs-per-hour limit doesn't address an actual problem. 2013-08-20T21:06:30 jeblair: right, I see it as a tool help implement proper bottleneck fixes 2013-08-20T21:06:42 clarkb: i really don't think it will help 2013-08-20T21:07:00 clarkb: you're creating and tuning a parameter that has nothing to do with the systems that are actually running 2013-08-20T21:07:32 but it is a parameter that influences everything 2013-08-20T21:07:43 clarkb: for instance, it would do nothing to prevent mass simultaneous deletions of nodes, which is an ACTUAL problem 2013-08-20T21:07:49 *** nati_ueno has quit IRC 2013-08-20T21:07:54 (or at least seems to be) 2013-08-20T21:08:26 *** melwitt has quit IRC 2013-08-20T21:08:31 *** melwitt1 has joined #openstack-infra 2013-08-20T21:08:38 just noticed that the zuul status timers don't do hours properly... 2013-08-20T21:08:45 *** nati_ueno has joined #openstack-infra 2013-08-20T21:08:51 jeblair: but it would reduce the number of nodes that would be deleted together 2013-08-20T21:09:04 clarkb: no, the fix that i'm trying to write right now will do that 2013-08-20T21:09:10 clarkb: it will delete only one node from a jenkins at a time 2013-08-20T21:09:18 clarkb: why would you want to try to fix that another way? 2013-08-20T21:09:27 I am not suggesting this as a fix 2013-08-20T21:09:49 clarkb: what are you suggesting? 2013-08-20T21:09:51 you would still want to fix that particular problem with the change you are writing 2013-08-20T21:10:08 jeblair: I am suggesting that we have some way of slowing everything down to usable levels while you write that fix 2013-08-20T21:10:29 *** rfolco has quit IRC 2013-08-20T21:10:52 we are very spiky and the ability to smooth out really big spikes will help in fixing the fallout 2013-08-20T21:11:16 clarkb: the fix i want to write will do that? why don't i just go write that instead of something else that won't fix it? 2013-08-20T21:11:50 because next week or during icehouse freeze we will run into similar yes different problems 2013-08-20T21:14:41 *** cppcabrera_afk is now known as cppcabrera 2013-08-20T21:16:23 *** fbo is now known as fbo_away 2013-08-20T21:17:34 mordred, fungi: ping 2013-08-20T21:17:39 jeblair: pong 2013-08-20T21:17:45 mordred: can you clean up jenkins02? 2013-08-20T21:18:04 jeblair: yes. is there a description of the problem in the scrollback? 2013-08-20T21:18:05 *** vipul is now known as vipul-away 2013-08-20T21:18:07 mordred: yes 2013-08-20T21:18:15 jeblair: great. I will find it 2013-08-20T21:18:26 mordred: thanks 2013-08-20T21:18:58 *** vipul-away is now known as vipul 2013-08-20T21:18:59 ttx: next year, can we move the nova FF one week prior? having me be only partially here due to burningman prep is not fantastic 2013-08-20T21:19:34 jeblair: oh wow. ok. force stop ok yeah? 2013-08-20T21:19:55 mordred: yep 2013-08-20T21:20:16 stopping 2013-08-20T21:20:59 btw - salt-master has cpu pegged on puppetmaster - I'm going to restart it 2013-08-20T21:21:12 mordred: i thought we stopped all the minions? maybe stop the master too. 2013-08-20T21:21:19 great 2013-08-20T21:22:01 we should make a second pass at cleaning up the salt stuff after featurefreeze 2013-08-20T21:22:18 I believe the minions are still going crazy after the ssh thing 2013-08-20T21:22:29 s/ssh/crypto/ 2013-08-20T21:22:40 oh, we didn't stop them? 2013-08-20T21:22:50 *** thomasbiege1 has joined #openstack-infra 2013-08-20T21:23:31 fungi, jeblair, pleia2: let me know if you think it may work https://review.openstack.org/#/c/42998/ 2013-08-20T21:24:18 jeblair: we stopped them by hand, then restarted them then ran the rekey thing in hopes it would make them sane again 2013-08-20T21:24:23 mordred: nex tyear, you shall scream when I show the schedule on the screen 2013-08-20T21:24:34 jeblair: but it didn't we should probably just disable the minion service on the slaves 2013-08-20T21:24:49 ttx: yes, I will 2013-08-20T21:25:02 ttx: I think he did 2013-08-20T21:25:34 clarkb: oh, you're right 2013-08-20T21:25:37 I did 2013-08-20T21:25:54 I believe I mentioned something like "there's going to be a rush and I'm not going to be much help" if the FF is that week 2013-08-20T21:26:08 *** thomasbiege1 has quit IRC 2013-08-20T21:26:14 next year if we separate summit/conf it would happen earlier 2013-08-20T21:26:24 perfect 2013-08-20T21:27:10 mordred: when do you leave for burning man 2013-08-20T21:27:10 ? 2013-08-20T21:27:17 James E. Blair proposed a change to openstack-infra/nodepool: Add ProviderManager https://review.openstack.org/42973 2013-08-20T21:27:28 clarkb: ^ live-tested 2013-08-20T21:27:43 *** prad_ has quit IRC 2013-08-20T21:27:48 Anita Kuno proposed a change to openstack-infra/config: Creating/adding the openstack/governance repository https://review.openstack.org/43002 2013-08-20T21:27:49 clarkb: i'm basically just going to do the same thing for jenkins now. 2013-08-20T21:28:22 jeblair: ok 2013-08-20T21:28:29 jeblair: I have only found one minor issue so far 2013-08-20T21:28:52 jeblair: but it won't cause any bugs 2013-08-20T21:29:09 mordred ^ 2013-08-20T21:29:11 jeblair: I've stopped jenkins02, amd currently working on deleting devstack slaves that were attached to it 2013-08-20T21:29:57 so in addition to this patch (I basically just followed the instructions for stackforge repos) what else to I have to do to create the repo? 2013-08-20T21:30:01 lifeless: first thing in the morning 2013-08-20T21:30:12 do I just create it on my laptop and push it as an empty repo? 2013-08-20T21:30:15 lifeless: too soon 2013-08-20T21:30:38 giving it a .gitreview file 2013-08-20T21:30:40 mordred: ack 2013-08-20T21:33:36 *** alexpilotti has quit IRC 2013-08-20T21:34:23 jeblair: ERROR: n/a (HTTP 400) 2013-08-20T21:34:50 jeblair: is that ^^ a symptom of az1 rate limiting? 2013-08-20T21:35:51 so trying to access the jenkins pages for some of hte running jobs on the zuul status page is resulting in 502s 2013-08-20T21:37:27 mordred: not that i'm aware; i don't see current rate limiting errors from nodepool 2013-08-20T21:37:34 AWESOME 2013-08-20T21:37:39 are there tc meeting logs prior to October 2012? this link has October 2012 through to now but not prior: http://eavesdrop.openstack.org/meetings/tc/ 2013-08-20T21:37:40 Alex_Gaynor: mordred is working on that 2013-08-20T21:37:59 jeblair: okey doke (as always if I can help in some way, let me know) 2013-08-20T21:37:59 jeblair: I'm getting that error a lot from running nova list and nova delete 2013-08-20T21:38:21 btw - ERROR: n/a (HTTP 400) is a TERRIBLE error message 2013-08-20T21:38:46 *** dprince has quit IRC 2013-08-20T21:40:50 mordred: OverLimit: This request was rate-limited. (HTTP 413) 2013-08-20T21:40:57 ok 2013-08-20T21:40:57 mordred: ^ that's what that looks like (and just happened) 2013-08-20T21:41:03 fantastic 2013-08-20T21:41:52 *** boris-42 has quit IRC 2013-08-20T21:42:55 *** cppcabrera has left #openstack-infra 2013-08-20T21:44:05 jeblair: I'm not having much luck in deleting the nodes... how important is that part of the step? 2013-08-20T21:47:18 mordred: i think you can skip it, nodepool should be able to clean up 2013-08-20T21:47:46 mordred: it will be slow about it, which probably isn't a bad thing 2013-08-20T21:47:56 jeblair: ok. then I'm going to delete the node section from config.xml and restart 2013-08-20T21:48:12 mordred: just the devstack nodes 2013-08-20T21:49:52 *** mrmartin has quit IRC 2013-08-20T21:50:19 *** prad_ has joined #openstack-infra 2013-08-20T21:51:28 *** AJaeger has quit IRC 2013-08-20T21:51:32 *** thomasbiege1 has joined #openstack-infra 2013-08-20T21:55:16 jeblair: jenkins02 is starting 2013-08-20T21:55:24 jeblair: and yes - just hte devstack nodes were delete 2013-08-20T21:55:28 *** dina_belova has joined #openstack-infra 2013-08-20T21:55:54 *** weshay has quit IRC 2013-08-20T21:57:28 * fungi is caught up on scrollback from lunch and reviewing gate-performance-improving changes as a first priority 2013-08-20T21:58:08 jeblair: woo finally got through that change 2013-08-20T21:58:44 jeblair: the only major concern I have is with the default timeout used by the manager code 2013-08-20T21:59:21 oh, my lunch was productive, got to talk to a redhat admin who thinks that for our use case running git daemon as a service makes more sense than xinetd anyway since we're using it so much, feel less bad about writing the init script now ;) 2013-08-20T21:59:35 So is this how it works every feature freeze? We fix the latest rounds of bottlenecks ? 2013-08-20T21:59:52 Alex_Gaynor: yes 2013-08-20T22:00:00 *** dina_belova has quit IRC 2013-08-20T22:00:09 *** thomasbiege1 has quit IRC 2013-08-20T22:00:10 pleia2: oh good 2013-08-20T22:00:29 Alex_Gaynor: each time, the feature freeze has been significantly larger than the previous too 2013-08-20T22:00:50 mordred: sure, that was the underlying premise of my statementn, I didn't meean to imply we weren't making progress :) 2013-08-20T22:00:59 Alex_Gaynor: the number of changes that go in the week before feature freeze is not only much greater than the previous feature freeze but much greater than the weeks before it 2013-08-20T22:01:47 *** gyee has quit IRC 2013-08-20T22:02:00 *** markmc has quit IRC 2013-08-20T22:02:33 mordred: I can do the needful this afternoon for the tagging process to get pbr working 2013-08-20T22:02:59 *** rnirmal has quit IRC 2013-08-20T22:03:22 *** mriedem1 has quit IRC 2013-08-20T22:03:55 *** markmcclain has quit IRC 2013-08-20T22:05:06 notmyname: ok. from my side, I believe we can do that 2013-08-20T22:07:51 mordred: here's, IMO, a simple thing I think will make it all work 2013-08-20T22:08:18 *** burt has quit IRC 2013-08-20T22:08:33 ooh. I like simple things 2013-08-20T22:08:42 mordred: we tag today with 1.9.2 and consume that version number (ie we won't ever "release" a 1.9.2). This will let pbr do the right thing and create version numbers that sort properly 2013-08-20T22:08:51 mordred: if we have another minor release, it will be 1.9.3 2013-08-20T22:09:00 mordred: but most likely will be 1.10.0 anyway 2013-08-20T22:09:03 well... we could do that ... 2013-08-20T22:09:13 but it will cause a 1.9.2 to be released to tarballs.o.o 2013-08-20T22:09:31 but I'm ok with that if you are 2013-08-20T22:09:38 Clark Boylan proposed a change to openstack-infra/zuul: SIGUSR2 logs stack traces for active threads. https://review.openstack.org/42959 2013-08-20T22:09:41 mordred: I don't see that as a problem, but do you have an alternate suggestion? 2013-08-20T22:09:44 jeblair: ^ now with documentation 2013-08-20T22:09:56 clarkb: just looked at your comment 2013-08-20T22:10:05 notmyname: tagging 1.9.2-dev - which will not cause a release to be cut 2013-08-20T22:10:17 and will map closely to your current version in tree 2013-08-20T22:10:26 on cleanupServer in providerManager... 2013-08-20T22:10:36 mordred: to quote from clay on the pbr patch "Rather than waiting for imminent merge, we really should get a 1.9.2 tag on the origin repo *now* so the git based versioning works in sane fashion for review. I don't really care about 1.9.2-dev which doesn't parse by distutils.version.StrictVersion *anyway*." 2013-08-20T22:10:42 jeblair: about the timeout value 2013-08-20T22:10:46 clarkb: yeah 2013-08-20T22:11:04 notmyname: ok. I'm sold by that 2013-08-20T22:11:23 clarkb: so the timeout loop is a big loop that runs inside of the thread that is trying to delete the server 2013-08-20T22:11:24 mordred: ya, mostly the last line 2013-08-20T22:11:39 mordred: and if you haven't you should read his full comment on https://review.openstack.org/#/c/28892/ 2013-08-20T22:12:03 mordred: but I think we can go forward with a 1.9.2 tag and then merge the patch 2013-08-20T22:12:04 clarkb: inside of that loop, it puts a task on the queue to get the server, and waits for that to complete 2013-08-20T22:12:13 clarkb: so i don't think anything about the timeout value changes 2013-08-20T22:12:48 clarkb: overall, we still wait, er, an hour for the server to be deleted (in a thread that is pretty much dedicated to trying to delete the server) 2013-08-20T22:13:19 clarkb: but that shouldn't affect anything else, other than every 2 seconds, that thread asks the provider thread to check on the server 2013-08-20T22:13:29 notmyname: reading now 2013-08-20T22:13:56 mordred: so I think that leaves it here: I'll approve/merge the pbr patch when I see the 1.9.2 tag on master upstream 2013-08-20T22:13:56 clarkb: (if a lot of servers are being slow to be deleted, everything else about that provider will be slow too, but i think that's desirable. mostly.) 2013-08-20T22:13:57 *** ^demon has quit IRC 2013-08-20T22:14:11 *** gyee has joined #openstack-infra 2013-08-20T22:14:31 *** ^d has joined #openstack-infra 2013-08-20T22:14:36 *** pblaho has quit IRC 2013-08-20T22:14:40 jeblair: will it not prevent other tasks for running? for some reason I thought it would, but that function is called from outside the manager thread and does the poll loop there 2013-08-20T22:14:42 *** ^d has quit IRC 2013-08-20T22:14:42 *** ^d has joined #openstack-infra 2013-08-20T22:14:50 jeblair: so I think I was concerned about nothing 2013-08-20T22:15:09 The running of the delete task runs in the manager thread which is quick 2013-08-20T22:15:15 jeblair: I will update my vote 2013-08-20T22:15:46 clarkb: exactly, all of those methods just put a task on the manager's queue, running those tasks happens in the dedicated thread, and all the tasks should be simple 1:1 nova api calls 2013-08-20T22:16:38 jeblair: done 2013-08-20T22:18:14 jeblair: pleia2 http://logs.openstack.org/93/42593/4/gate/gate-grenade-devstack-vm/6de9e45/logs/devstack-gate-setup-workspace-new.txt 2013-08-20T22:18:53 ttx: when you are around but not in a meeting, here is my first attempt: https://review.openstack.org/#/c/43002/ 2013-08-20T22:19:08 *** ^d has quit IRC 2013-08-20T22:19:40 jeblair: pleia2: I think that may be replication related 2013-08-20T22:20:05 though I am not sure because I would've expected git to make that more atomic 2013-08-20T22:20:10 notmyname: ok. yes. I tihnk it's a well written comment, and I appreciate the willingness to go along. 2013-08-20T22:20:14 *** dkliban has quit IRC 2013-08-20T22:20:23 notmyname: do you want me to cut a tag? or do you want to do it? 2013-08-20T22:20:42 mordred: I can't make tags for swift (unless that's changed) 2013-08-20T22:21:00 mordred: if I have the perms, I'd be happy to do it 2013-08-20T22:21:48 clarkb: i agree, it wfm locally 2013-08-20T22:21:53 jeblair: pleia2 http://paste.openstack.org/show/44689/ is what I see in the apache log 2013-08-20T22:21:57 ttx: you around? 2013-08-20T22:22:32 clarkb: what a strange error 2013-08-20T22:22:51 jeblair: ya, file exists though and has timestamps from days in the past 2013-08-20T22:22:55 that is odd, it's just ssh that replicates so it shouldn't be doing something like deleting it first (huh, would it?) 2013-08-20T22:22:56 mordred: after midnight in paris right now.. 2013-08-20T22:23:04 notmyname: ok. I'll just do it 2013-08-20T22:23:23 notmyname, mordred: he's not in that timezone 2013-08-20T22:23:28 ah, ok then :-) 2013-08-20T22:23:37 pleia2: I don't expect it to and the mod time on that dir is from the 13th 2013-08-20T22:23:40 mordred: ok. who has permission to push tags? with the change to pbr is that changing? 2013-08-20T22:23:46 jeblair: clarkb: ^ ? 2013-08-20T22:23:58 notmyname: no - it should be still ttx since it's a server project 2013-08-20T22:24:02 ok 2013-08-20T22:24:13 notmyname: the main change is that it won't need to commit to change the version anymore 2013-08-20T22:24:34 notmyname: so the chances of your milestone-proposed brnach being any different than master are _REALLY_ low :) 2013-08-20T22:25:09 Jim Branen proposed a change to openstack/requirements: Allow use of hp3parclient 2.0 https://review.openstack.org/42991 2013-08-20T22:25:12 notmyname: 5c6f0015d56478108a623cf65641a39ea91fc2b5 work for you? 2013-08-20T22:25:52 mordred: confirm. 5c6f0015d56478108a623cf65641a39ea91fc2b5 2013-08-20T22:26:40 *** changbl has quit IRC 2013-08-20T22:26:59 notmyname: done 2013-08-20T22:27:06 mordred: thanks 2013-08-20T22:27:43 mordred: final tests on pbr branch 2013-08-20T22:27:47 rd 2013-08-20T22:29:14 I wonder 2013-08-20T22:29:59 *** lbragstad has quit IRC 2013-08-20T22:33:01 jeblair: pleia2 so apache is allowed to read the pack and idx files directly without talking to the git http thing 2013-08-20T22:33:12 jeblair: pleia2 and that is what appears to have failed 2013-08-20T22:33:40 *** jungleboyj has joined #openstack-infra 2013-08-20T22:34:30 Can anyone answer questions about how the Transifex Translations are being automatically done? 2013-08-20T22:34:42 pleia2: any chance selinux is involved? 2013-08-20T22:34:48 jungleboyj: yes I can, whats up? 2013-08-20T22:35:00 clarkb: Awesome. Thank you! 2013-08-20T22:35:30 *** jhesketh has joined #openstack-infra 2013-08-20T22:35:50 clarkb: good question, it shouldn't since everything in /var/lib/git should have the right selinux magic to serve it up to httpd 2013-08-20T22:36:34 clarkb: but this is getting quite far out of my git expertise to understand what is happening git-wise (pack and idx files?) 2013-08-20T22:36:58 pleia2: in .git/objects/pack 2013-08-20T22:37:08 clarkb: I am working on Cinder and noticed that we had some english strings that were coming our wrong. When I look at the .po files for en_US I see that it has a msgstr defined that is either incomplete or all together wrong. Trying to figure out the right way to fix that. I had gone through and removed all the msgstr s (msgstr="") since it doesn't make sense to translate English to English but now I see the latest 2013-08-20T22:37:37 jungleboyj: can you defined "coming out wrong" ? 2013-08-20T22:37:49 pleia2: the pack files contain a bunch of object files all compressed together, I believe the idx files tell git where to look in that compressed blob for specific objects 2013-08-20T22:38:06 pleia2: that particular file has been in place since the 13th though 2013-08-20T22:38:17 clarkb: I see, so that doesn't sound to me like anything strange that selinux would have a problem with inside /var/lib/git/ 2013-08-20T22:38:41 jungleboyj: can you link to a particular example in a proposed change? 2013-08-20T22:39:04 jungleboyj: and I think the way i8ln works it does make sense to translate English to English depending on the locale :) 2013-08-20T22:39:39 mordred: I had the string _("Failure creating image %s. Error %s", vol_id, error) or something like that. In the .po the msgstr for that was just "Failure creating image" and that was all that was printed to the logs. 2013-08-20T22:39:53 bad translator, no cookie 2013-08-20T22:40:25 *** apcruz has quit IRC 2013-08-20T22:40:32 *** sandywalsh has quit IRC 2013-08-20T22:40:55 * clarkb updates cinder repo 2013-08-20T22:40:58 *** shardy is now known as shardy_afk 2013-08-20T22:41:41 pleia2: the normal permissions all look fine. I don't know why else apache would fail to see a dir 2013-08-20T22:42:09 *** nijaba has quit IRC 2013-08-20T22:42:16 With JJB, has anyone had the great idea to use parameterized jobs in job-group? 2013-08-20T22:42:39 clarkb: Here is the specific example: https://review.openstack.org/#/c/40948/2/cinder/locale/en_US/LC_MESSAGES/cinder.po Line 583 2013-08-20T22:43:29 clarkb: /var/log/audit.log is where selinux logs violations, so you can look there 2013-08-20T22:43:46 pleia2: thanks 2013-08-20T22:43:51 msgid "Failed to copy image to volume: %(reason)s" 2013-08-20T22:43:51 msgstr "Failed to copy image to volume" 2013-08-20T22:45:31 jungleboyj: we treat transifex as the source of truth for those msgstrs 2013-08-20T22:46:13 jungleboyj: the old string there may have been a casualty of babel doing a fuzzy translation and not understanding the %(reasons) I am not actually sure there 2013-08-20T22:46:37 clarkb: Ok, well, in the case of Cinder the msgstrs are incomplete or wrong. Need to figure out how to fix it. Saw the same thing in other projects too. 2013-08-20T22:46:45 jungleboyj: but for patchset 1 the removal of the msgstr would've come from transifex or the update_catalog that we run prior to updating from transifex 2013-08-20T22:47:21 jungleboyj: yeah, things were wrong at one point because babel allows fuzzy translations by default, we have since disabled that. Let me get you a link to the script that proposes these chagnes 2013-08-20T22:47:53 jungleboyj: i have seen translations from the "c" source language to en get extremely stale because nobody is checking them for some projects, so eventually the source strings grow different numbers of format string parameters than the obsolete en versions which should normally be identical 2013-08-20T22:48:12 jungleboyj: https://github.com/openstack-infra/config/blob/master/modules/jenkins/files/slave_scripts/propose_translation_update.sh 2013-08-20T22:49:19 jungleboyj: https://github.com/openstack-infra/config/blob/master/modules/jenkins/files/slave_scripts/propose_translation_update.sh#L46-L55 is the most relevant section. I wonder if this is fallout from when we didn't prevent fuzzy matches 2013-08-20T22:49:27 jungleboyj: i did a fairly massive pass through nova some months back to clean up english translations (which basically resulted in me duplicating the source strings) 2013-08-20T22:50:00 i'm not familiar with what the impact from fuzzy matches might be though 2013-08-20T22:50:29 jungleboyj: from git blame http://paste.openstack.org/show/44691/ that was long enough ago to be when fuzzy matching was allowed so I think that is the issue 2013-08-20T22:51:08 *** mikal has joined #openstack-infra 2013-08-20T22:51:11 fungi: jungleboyj: we may want to reseed them all with non fuzzy strings based on what is in transifex to get past the cruft that babel let through initially 2013-08-20T22:52:06 *** mikal has quit IRC 2013-08-20T22:52:53 *** prad_ has quit IRC 2013-08-20T22:53:32 i take it there's no way to identify a fuzzy vs. non-fuzzy translation of a string solely from the pofile 2013-08-20T22:53:37 *** sandywalsh has joined #openstack-infra 2013-08-20T22:53:41 mordred: patch merged (merging) and email sent to ML 2013-08-20T22:53:48 notmyname: woot! 2013-08-20T22:53:49 mordred: thanks for your help on it 2013-08-20T22:54:11 notmyname: thanks for yours! I believe pbr is much better today than it was originally due to addressing your concerns 2013-08-20T22:54:17 *** nijaba has joined #openstack-infra 2013-08-20T22:54:32 fungi: there is the # fuzzy comment, but I think babel may not remove those when it has a non fuzzy translation 2013-08-20T22:54:41 fungi: which makes it a little painful to work with 2013-08-20T22:54:46 clarkb: So, let me make sure that I understand. There are some old en translations that didn't happen properly because fuzzy matching was allowed. 2013-08-20T22:55:01 *** ftcjeff has quit IRC 2013-08-20T22:55:08 *** markmcclain has joined #openstack-infra 2013-08-20T22:55:10 mordred: in my email I said, "If you have any issues, just ask Monty. Preferably after 10pm on Tuesdays" ;-) 2013-08-20T22:55:11 *** michchap has joined #openstack-infra 2013-08-20T22:55:40 clarkb: speaking of i18n, we should get swift on the transifex bandwagon - they already use babel and everything 2013-08-20T22:55:49 clarkb: right. unless we actually expect un-fuzzed translations to result in the #fuzzy comment also getting removed, no way to tell just from the translated string itself 2013-08-20T22:55:58 clarkb: and their translations are in top level like I sort of want everyone else's to be :) 2013-08-20T22:56:17 notmyname: I look forward to those questions :) 2013-08-20T22:56:44 jungleboyj: correct 2013-08-20T22:56:49 clarkb: If that is the case, how can I get fixes for those strings that got fuzzed up. 2013-08-20T22:57:22 jungleboyj: you can translate them in transifex, or I think it is still possible to propose a patch that fixes them, but that may not be the case. I will have to double check that 2013-08-20T22:57:58 Elizabeth Krumbach Joseph proposed a change to openstack-infra/config: Swap git daemon in xinetd for service https://review.openstack.org/43012 2013-08-20T22:58:04 *** mkirk_ has quit IRC 2013-08-20T22:58:22 clarkb: Forgive all the noob questions. How do I translate them in transifex? 2013-08-20T22:58:43 jungleboyj: https://github.com/openstack-infra/config/blob/master/modules/jenkins/files/slave_scripts/upstream_translation_update.sh#L42-L53 we still push local git contents back to transifex so you can propose a fix in git if you like 2013-08-20T22:58:44 *** mkirk_ has joined #openstack-infra 2013-08-20T22:59:25 jungleboyj: I have actually never done it :) but I believe you log into https://transifex.com find the cinder project and then you can either update strings in your browser or use the tx tool 2013-08-20T22:59:40 *** gordc has left #openstack-infra 2013-08-20T22:59:48 clarkb: Ok. 2013-08-20T23:00:32 clarkb: FYI, the pot file doesn't have any msgstrs defined in it. Will changing the pos make a difference? 2013-08-20T23:00:49 the pot file is a template, it should not have any msgstrs in it 2013-08-20T23:00:58 the .po files contain the actual translations 2013-08-20T23:01:09 *** rcleere has quit IRC 2013-08-20T23:01:12 Elizabeth Krumbach Joseph proposed a change to openstack-infra/config: Swap git daemon in xinetd for service https://review.openstack.org/43012 2013-08-20T23:01:53 clarkb: That is what I thought. So, I would need to actually put the changes in the POs. 2013-08-20T23:02:09 *** sgviking has quit IRC 2013-08-20T23:02:18 *** dkliban has joined #openstack-infra 2013-08-20T23:02:29 jeblair: pleia2 mordred https://jenkins01.openstack.org/job/gate-neutron-pep8/434/console ugh. I think centos and ubuntu must be sufficiently different that this doesn't work quite right. Or something replication related 2013-08-20T23:02:33 jungleboyj: yup 2013-08-20T23:03:07 clarkb: Once I do that, is there something I need to do to get a new transifex import to happen? 2013-08-20T23:03:07 *** jpich has quit IRC 2013-08-20T23:03:08 jungleboyj: using transifex's tx tool you can get pull the pos and push them back to transifex if you want to use their workflow 2013-08-20T23:03:22 jungleboyj: we import from transifex once a day per project 2013-08-20T23:03:31 so you don't need anything special it should just happen 2013-08-20T23:04:05 clarkb: Ok, and you don't recommend clearing out all the english msgstrs ? Just fix the ones that are wrong? 2013-08-20T23:04:24 jungleboyj: right. as en_US is different than C 2013-08-20T23:04:32 clarkb: yeah, three differences: replication over ssh, operating system, git version 2013-08-20T23:04:37 and different than en_UK and so on 2013-08-20T23:04:56 clarkb: Ok. Thank you so much for the help! 2013-08-20T23:05:09 clarkb: I think it's a rewrite problem! pulling that file from /cgit works, but not the direct git.openstack.org/openstack/neutron/... location 2013-08-20T23:05:21 pleia2: interesting 2013-08-20T23:05:21 Mathieu Gagné proposed a change to openstack-infra/jenkins-job-builder: Job-specific subst. in a job group's job list https://review.openstack.org/43013 2013-08-20T23:06:35 *** mrodden has quit IRC 2013-08-20T23:07:08 pleia2: /cgit will be served by cgit though right? 2013-08-20T23:07:14 pleia2: so possibly completely different processes 2013-08-20T23:07:23 clarkb: right 2013-08-20T23:07:45 but at least the files do exist and are servable by apache somewhere 2013-08-20T23:08:05 might be right about git version weirdness 2013-08-20T23:08:47 clarkb: maybe check if that file exists on disk? 2013-08-20T23:08:56 cgit is serving it 2013-08-20T23:09:02 pleia2: could be cached 2013-08-20T23:09:05 ah 2013-08-20T23:09:17 pleia2: if it exists on disk and apache does not serve it, it's as you say, a rewrite problem 2013-08-20T23:09:27 pleia2: if not, we're back to where we were 2013-08-20T23:09:38 jeblair: the files do exist on disk, at least the ones that I have seen 2013-08-20T23:09:44 s/seen/looked at/ 2013-08-20T23:09:51 *** sgviking has joined #openstack-infra 2013-08-20T23:10:33 clarkb: does openstack/neutron/objects/pack/pack-de6d5d31c8684408cf90392a88fb0176b4ca8f01.idx ? 2013-08-20T23:10:53 https://github.com/openstack-infra/config/blob/master/modules/cgit/templates/git.vhost.erb#L19-L30 for those follwoing along. 2013-08-20T23:10:55 jeblair: checking 2013-08-20T23:11:34 jeblair: yes -r--r--r--. 1 cgit cgit 4488 Aug 20 06:18 pack-de6d5d31c8684408cf90392a88fb0176b4ca8f01.idx 2013-08-20T23:12:10 pleia2: sounds like you're on to something 2013-08-20T23:12:25 jeblair: pleia2 does the RewriteRule and ScriptAlias conflict? 2013-08-20T23:12:58 hmm 2013-08-20T23:13:11 oh you know 2013-08-20T23:13:30 *** jerryz has joined #openstack-infra 2013-08-20T23:13:31 actually no that can't be it 2013-08-20T23:14:17 the regex for pack|idx seems right 2013-08-20T23:14:31 pleia2: yeah that comes straight from the git http man page iirc 2013-08-20T23:15:20 *** dims has quit IRC 2013-08-20T23:18:10 *** ken1ohmichi has joined #openstack-infra 2013-08-20T23:20:54 *** ryanpetrello has quit IRC 2013-08-20T23:21:17 James E. Blair proposed a change to openstack-infra/nodepool: Add JenkinsManager https://review.openstack.org/43014 2013-08-20T23:21:17 James E. Blair proposed a change to openstack-infra/nodepool: Add an ssh check periodic task https://review.openstack.org/43015 2013-08-20T23:21:18 James E. Blair proposed a change to openstack-infra/nodepool: Change credentials-id parameter in config file https://review.openstack.org/43016 2013-08-20T23:21:18 James E. Blair proposed a change to openstack-infra/nodepool: Reduce timeout when waiting for server deletion https://review.openstack.org/43017 2013-08-20T23:21:19 James E. Blair proposed a change to openstack-infra/nodepool: Add ProviderManager https://review.openstack.org/42973 2013-08-20T23:21:38 which repo should I clone to test? I was able to clone stackforge/puppet-glance and openstack/python-heatclient without problem 2013-08-20T23:21:57 mgagne: neutron and nova appear to currently be failing fairly frequently according to the logs 2013-08-20T23:22:50 clarkb: is it therefore an intermittent issue? 2013-08-20T23:23:01 clarkb: so can it get to some pack-de6d5d31c8684408cf90392a88fb0176b4ca8f01.idx files? 2013-08-20T23:23:07 mgagne: yes, it seems to be intermittent 2013-08-20T23:23:15 er, .idx files 2013-08-20T23:23:16 pleia2: I am not sure yet, actually let me try getting that file direclty 2013-08-20T23:23:31 pleia2: mgagne: this may in part depend on the local state of your repo 2013-08-20T23:24:30 clarkb: I'm cloning from scratch, are tests fetching and checking out a specific ref instead? 2013-08-20T23:25:14 mgagne: tests will clone if the repo doesn't already exist otherwise they will do a remote update to fetch what they are missing 2013-08-20T23:25:29 pleia2: directly fetching one of those neutron files with wget fails. This must've been what you tested before 2013-08-20T23:25:39 pleia2: for whatever reason I thought you tested with a git clone which does work 2013-08-20T23:26:12 clarkb: I just tested via web browser 2013-08-20T23:27:38 pleia2: looking at the vhost cgit will serve anything not under .*/objects because ScriptAlias / /usr/libexec/git-core/git-http-backend/ will never be used as we rewrite / to /cgit 2013-08-20T23:28:40 pleia2: oh but we rewrite ^/$ to /cgit so anything like /openstack/foo should go to git-http-backend right? 2013-08-20T23:28:42 clarkb: yeah, I think those rewrite things are not for cgit 2013-08-20T23:28:55 *** mrodden has joined #openstack-infra 2013-08-20T23:29:05 clarkb: I think they are just for git-http-backend 2013-08-20T23:29:32 fungi added them in a change to support git-http-backend 2013-08-20T23:29:54 *** changbl has joined #openstack-infra 2013-08-20T23:30:49 *** dims has joined #openstack-infra 2013-08-20T23:31:41 *** HenryG has joined #openstack-infra 2013-08-20T23:32:08 clarkb: ^ the new stack of nodepool changes is in production 2013-08-20T23:32:08 yup 2013-08-20T23:32:54 clarkb: (i did reduce that timeout, btw, because i think it was ridiculously large) 2013-08-20T23:33:05 from an hour to...? 2013-08-20T23:33:07 *** ken1ohmichi has quit IRC 2013-08-20T23:33:08 10 mins 2013-08-20T23:33:15 * fungi nods. sounds sane 2013-08-20T23:33:24 which is just, well, large. :) 2013-08-20T23:33:36 s/ridiculously// 2013-08-20T23:33:43 clarkb: confirmed, I don't have any of the pack rewrite rules in my test instance and I can download packs via cgit (hi fungi!) 2013-08-20T23:34:50 pleia2: I think it may be an selinux thing 2013-08-20T23:35:12 pleia2: httpd itself will access the git files when they hit the AliasMatches 2013-08-20T23:35:13 * fungi retries to grok where the ^/$ rewrite could conflict at all with the git-http-backend cgi scriptalias 2013-08-20T23:35:21 but httpd runs under a different selinux type 2013-08-20T23:35:33 I am very quickly learning about selinux types so that I can test 2013-08-20T23:35:41 selinux would show that error 2013-08-20T23:35:46 clarkb: look in audit.olg 2013-08-20T23:35:47 log 2013-08-20T23:36:08 audit.log was a pain to look at ... 2013-08-20T23:36:14 hah 2013-08-20T23:36:21 can grep for git probably 2013-08-20T23:36:23 but I think I just get annoyed when there are no timestamps. I will look again 2013-08-20T23:37:44 clarkb: well, there are timestamps, you just need to learn to read unixtime directly ;) 2013-08-20T23:38:51 I don't see any AVC messages in audit.log 2013-08-20T23:38:55 clarkb: I think it has to do with the way packs are generated. Could be that they are generated on-the-fly and there is contention issues on git.o.o due to the high volume of clone, fetch, etc. 2013-08-20T23:39:00 clarkb: https://www.kernel.org/pub//software/scm/git/docs/git-update-server-info.html 2013-08-20T23:39:46 mgagne: it seems to know where the files are though, it just can't get them 2013-08-20T23:41:29 clarkb: a curl returns the file? Could it be caching issue? Or is it a timing issue, by the time you test the existence of the file, it got generated. Trying to figure out what have been tried/tested. 2013-08-20T23:42:14 *** rfolco has joined #openstack-infra 2013-08-20T23:42:34 mgagne: wgetting the file that was failed to fetch on a jenkins slave fails, but the file is on disk and has been there for at leasthours 2013-08-20T23:42:56 mgagne: https://jenkins01.openstack.org/job/gate-neutron-pep8/434/console has a list of things that can't be fetched 2013-08-20T23:43:13 mgagne: however changing the root of the url to /cgit you are able to get the file 2013-08-20T23:43:36 mgagne: so it is only when apache attempts direct access via https://github.com/openstack-infra/config/blob/master/modules/cgit/templates/git.vhost.erb#L28-L29 that it fails 2013-08-20T23:44:06 further evidence the scriptalias is not working: the actual apache error log message says "File does not exist: /var/lib/git/openstack/neutron" 2013-08-20T23:44:12 and that _doesn't_ exist 2013-08-20T23:44:21 because it's /var/lib/git/openstack/neutron.git 2013-08-20T23:44:44 so presumably the scriptalias directive to use the smart http server would normally translate that, 2013-08-20T23:44:57 oh that may be it 2013-08-20T23:45:00 oh wow, right 2013-08-20T23:45:03 but it's not, so apache is just trying to serve a simple file 2013-08-20T23:46:07 https://git.openstack.org/openstack/neutron.git/objects/pack/pack-8dd2daf4e48bc336b39e06bcb5612bdc2c7bec7c.idx works! 2013-08-20T23:46:23 nice one jeblair 2013-08-20T23:46:38 but looking at that, i think we're trying to get apache to just serve the files 2013-08-20T23:47:02 it looks like the aliasmatch directives are intended to take precedence, and then scriptalias catches the rest 2013-08-20T23:47:25 any idea why i'm seeing this in my tox runs? http://paste.openstack.org/show/44692/ 2013-08-20T23:47:37 cannot import setuptools 2013-08-20T23:47:44 jeblair: the config comes from https://www.kernel.org/pub/software/scm/git/docs/git-http-backend.html 2013-08-20T23:47:49 but it actually installs setuptools 1.0 above... 2013-08-20T23:47:56 clarkb: yeah, and it's the same as on review 2013-08-20T23:48:07 *** mriedem has joined #openstack-infra 2013-08-20T23:48:38 clarkb: what if the git smart http server is providing the wrong urls? 2013-08-20T23:48:45 (git version difference) 2013-08-20T23:49:09 jeblair: could be 2013-08-20T23:49:13 GIT_PROJECT_ROOT has a trailing slash 2013-08-20T23:49:18 could it be? 2013-08-20T23:49:44 mrodden: the uninstall of distribute that happens first is causing the problem I htink 2013-08-20T23:49:53 doc doesn't show/use trailing slash 2013-08-20T23:50:17 mrodden: try updating tox? 2013-08-20T23:50:21 mgagne: perhaps, maybe if it has a trailing slash it does assume neutron/ and won't expand to neutron.git/ 2013-08-20T23:50:55 clarkb: ok i'm on 1.4 2013-08-20T23:50:58 1.4.2 i think 2013-08-20T23:51:13 there is a trailing slash on review.o.o, but I can go ahead and update it git.o.o and restart apache to check 2013-08-20T23:51:15 wow they have 1.6.0 out now... 2013-08-20T23:51:28 mrodden: there has been a lot of churn around setuptools and distribute merging 2013-08-20T23:51:34 mrodden: so there are a bunch of updates from tools 2013-08-20T23:51:43 well, we have trailing / on GIT_PROJECT_ROOT for the gerrit servers and zuul in fact 2013-08-20T23:51:53 *** UtahDave has quit IRC 2013-08-20T23:51:59 crazy 2013-08-20T23:52:04 Joshua Hesketh proposed a change to openstack-infra/zuul: Move gerrit specific result actions under reporter https://review.openstack.org/42644 2013-08-20T23:52:05 Joshua Hesketh proposed a change to openstack-infra/zuul: Add support for emailing results via SMTP https://review.openstack.org/42645 2013-08-20T23:52:05 Joshua Hesketh proposed a change to openstack-infra/zuul: Separate reporters from triggers https://review.openstack.org/42643 2013-08-20T23:52:32 fungi: yeah but this is the only server with this version of git 2013-08-20T23:52:41 anyways restarting apache now 2013-08-20T23:53:34 didn't help 2013-08-20T23:53:40 nope :\ 2013-08-20T23:54:02 uh, so there are very few references to pack files in the gerrit logs 2013-08-20T23:54:15 jeblair: maybe it isn't working there either? 2013-08-20T23:54:44 clarkb: oh - interesting 2013-08-20T23:54:46 some of them are to '.git' dirs, and they work, some omit '.git' and are 404s 2013-08-20T23:55:07 same thing here 2013-08-20T23:55:11 by very few, i mean 1 client this week. 2013-08-20T23:55:54 warning hack: what if we just symlink openstack/foo to openstack/foo.git on disk? 2013-08-20T23:55:58 and handle both cases? 2013-08-20T23:56:39 clarkb: it hurts, but if we do we can do it in the jeepyb script 2013-08-20T23:57:28 clarkb: maybe to stop the bleeding? but we really should figure out the problem. 2013-08-20T23:57:43 jeblair: I agree 2013-08-20T23:57:53 let me add a neutron symlink then try grabbing that idx file again 2013-08-20T23:58:02 that will at least tell us if this is the only problem 2013-08-20T23:58:07 * pleia2 nods 2013-08-20T23:58:10 (i don't think we should add it to jeepyb, (unless we decide it's the actual solution) we'll never fix it) 2013-08-20T23:58:17 jeblair: ah, ok 2013-08-20T23:59:36 mordred: i forgot a step earlier: set the nodes to deleted in nodepool 2013-08-20T23:59:45 i'll do that now