Tuesday, 2013-08-13

ttxmaybe it's just a minority, but I've got lots of reports since ~August 500:00
jeblairttx: do you have some specific examples?00:02
jeblairttx: (i imagine you are correct; it'll be easier to figure out what's wrong with some specific cases)00:02
ttxjeblair: I'll forward you my weekly report. usually those are all human errors that I try to fix. This time there were lots of "failed to mark it so I did it".00:03
*** lifeless has joined #openstack-infra00:03
ttxjeblair: sent. You shall find a few examples in that list... for cinder nova...00:04
ttx(this script just looks at when a human sets FixCommitted or FixReleased)00:05
ttxThere are a few weird occurences in there where the script mentioned the commit but did NOT set TO fixCommitted.00:05
ttxanyway, I plan to investigate one of those days... just checking if you heard similar reports.00:06
*** UtahDave has quit IRC00:07
*** fbo is now known as fbo_away00:09
*** sarob_ has joined #openstack-infra00:13
ttxjeblair: looking at recent changes they look alright. Maybe it was just a transient thing00:14
fungittx: the ones i've seen weren't using the closes-bug header in the commit message and used wording other than the handful of backward-compatible patterns (things like "this fixes the issues in bug #XXXXXX"00:14
*** erfanian has joined #openstack-infra00:15
fungithe fallback solution in the newer pattern match for it assumes the few things it didn't know for sure meant "closes-bug" should be interpreted as "related-bug"00:15
ttxfungi: I suspect we had transitional things where the review link was posted before the transition and the fixcommitted failes to show00:15
fungittx: agreed00:15
ttxwe should be good now00:16
fungittx: we're a little over a week with the new version, so stuff from before that is much more likely to not close bugs when merged00:16
openstackgerritAlessandro Pilotti proposed a change to openstack-dev/pbr: Fixes issue with command escaping on Windows  https://review.openstack.org/4128300:17
*** sarob has quit IRC00:17
fungithough if they had "fixes bug #whatever" on a line by itself (which i think was the old documented example) it still gets picked up as closes-bug from what i've seen00:17
fungiand there were a few permutations of that which also got included for backwards-compat00:18
*** sarob_ has quit IRC00:18
*** pcrews has joined #openstack-infra00:18
*** emagana has quit IRC00:25
*** dina_belova has joined #openstack-infra00:29
*** lcestari has joined #openstack-infra00:30
*** lcestari has quit IRC00:30
*** dina_belova has quit IRC00:31
*** dkliban has quit IRC00:33
pabelangerI guess tarballs.openstack.org  does not fall under the realm of -infra?00:36
*** jhesketh has quit IRC00:36
*** dkliban has joined #openstack-infra00:36
pabelangerwas wanting to see some things in puppet, but looks like the host is not managed00:36
*** jhesketh has joined #openstack-infra00:36
jeblairpabelanger: it does; we just haven't gotten around to rsyncinc the content to static.o.o00:36
*** nati_ueno has quit IRC00:37
pabelangerjeblair, Ah, I see00:38
pabelangercool, thanks00:38
*** nati_ueno has joined #openstack-infra00:40
*** pcm__ has quit IRC00:41
*** rfolco has joined #openstack-infra00:42
*** rfolco has quit IRC00:42
*** nati_ueno has quit IRC00:48
*** nati_ueno has joined #openstack-infra00:49
*** nati_uen_ has joined #openstack-infra00:51
*** nati_ueno has quit IRC00:53
fungieek, tarballs.o.o is still on 10.04 lts even00:55
*** mriedem has joined #openstack-infra00:55
openstackgerritJames E. Blair proposed a change to openstack-infra/gearman-plugin: Rework starting/stopping executors  https://review.openstack.org/4148800:56
jeblairfungi: yes, we should move it.00:57
jeblairclarkb, zaro: ^ that should take care of the problem zaro pointed out.00:57
jeblairoh, right unit tests...00:57
openstackgerritTom Fifield proposed a change to openstack-infra/config: Add build job for Japanese Operations Guide  https://review.openstack.org/3984600:59
openstackgerritTom Fifield proposed a change to openstack-infra/config: Add build job for Japanese Operations Guide  https://review.openstack.org/3984601:02
*** ^d has quit IRC01:07
*** yaguang has joined #openstack-infra01:08
*** jerryz has quit IRC01:12
openstackgerritJames E. Blair proposed a change to openstack-infra/gearman-plugin: Rework starting/stopping executors  https://review.openstack.org/4148801:13
*** erfanian has quit IRC01:18
*** vipul is now known as vipul-away01:26
*** dina_belova has joined #openstack-infra01:29
*** dina_belova has quit IRC01:34
*** yaguang has quit IRC01:35
*** vipul-away is now known as vipul01:41
*** mjfork has quit IRC01:45
*** markmcclain has quit IRC01:48
*** yaguang has joined #openstack-infra01:54
*** mrodden1 has quit IRC01:55
*** mrodden has joined #openstack-infra01:56
*** changbl has joined #openstack-infra01:57
*** adalbas has quit IRC02:02
*** anteaya has quit IRC02:03
*** mriedem has quit IRC02:09
openstackgerritbenley proposed a change to openstack-infra/jenkins-job-builder: Add support for parameter filters in copyartifact  https://review.openstack.org/4158202:10
*** nati_uen_ has quit IRC02:11
clarkbgot distracted for far too long02:11
clarkbjog0: still around? re the py3k thing. The print statement in the example is print ('foo') + 'some string' which is not compat02:12
fungiwhereas print ('foo' + 'some string') would be fine02:16
clarkbya, and I am not sure that is something that hacking should try to catch as it is a runtime error02:16
clarkbbut meh02:16
fungii'm similarly on the fence about hacking acting as a proxy for a python 3.x interpreter02:19
fungiwe now have the ability to just try and parse the entire project with python 3.3 and catch waaaay more than something as simple as print statements02:20
jog0clarkb fungi: no I am not around02:28
fungijog0: awesome, neither am i ;)02:28
jog0but yeah now that we have py33 running I am inclined to say just do it that way02:28
*** dina_belova has joined #openstack-infra02:30
*** melwitt has quit IRC02:34
*** dina_belova has quit IRC02:35
*** yaguang has quit IRC02:40
jgriffithsdague: around?02:55
fungijgriffith: i believe he has declared himself officially vacationing as of some hours ago now02:57
jgriffithfungi: ahh... good for him :)03:01
funginot so good for the rest of us though03:02
fungis'okay... i'll be doing the same soon03:02
jgriffithfungi: Ummm... I wasn't informed of this?03:02
jgriffithfungi: sorry, but I think that's going to be terribly inconvenient03:03
funginext you're going to tell me you need me to come into work on saturday03:04
jgriffithfungi: haha.. no need to tell you, you always do anyway03:04
jgriffithfungi: I've just come to expect it :)03:05
jgriffithfungi: anything fun/exciting planned or just some down time?03:05
jgriffithfungi:  or none of jgriffith 's biz03:06
fungishowing my girlfriend around seattle for a week since she's never been, then getting on a boat up to alaska and back for a week after that03:06
jgriffithWow!  That sounds pretty darn good!03:06
* jgriffith is a bit jealous03:06
fungiplanning to try to be useful while in seattle, though probably not as useful as i'd like03:06
jgriffithfungi: take a real vacation!03:07
fungibut the following week i'll be (un)lucky if i can find an internet anywhere03:07
jgriffithfungi: well, here's to hoping you don't find internet access03:07
fungianyway, it'll be a cool way to escape the 100f heat here in nb03:07
fungi(not nb)03:08
jgriffithhot AND humid03:08
fungiyeah, it'll be nice to get away from03:08
jgriffithindeed... hope you enjoy and hope that you don't find any internet access03:08
jgriffithWhen are you heading off?03:09
fungiplane to seattle this coming saturday, then getting on the boat the next friday03:10
jgriffithnice.. I'l be sure NOT to ping you between those dates :)03:10
fungieh, it's okay. if i really can't help you i just won't answer ;)03:10
jgriffithfungi: haha.. fair enough :)03:10
*** yaguang has joined #openstack-infra03:25
*** dina_belova has joined #openstack-infra03:30
openstackgerritJeremy Stanley proposed a change to openstack-infra/jenkins-job-builder: Document escaping braces in template literals  https://review.openstack.org/4097303:31
*** dina_belova has quit IRC03:35
*** tianst has joined #openstack-infra03:40
*** jfriedly has quit IRC03:42
openstackgerritJeremy Stanley proposed a change to openstack-infra/git-review: Wrap long lines  https://review.openstack.org/4080003:50
openstackgerritJeremy Stanley proposed a change to openstack-infra/git-review: Wrap long lines  https://review.openstack.org/4080003:51
*** SergeyLukjanov has joined #openstack-infra04:05
*** mberwanger has quit IRC04:05
*** changbl has quit IRC04:05
*** changbl has joined #openstack-infra04:06
openstackgerritJeremy Stanley proposed a change to openstack-infra/git-review: Migrate to pbr.  https://review.openstack.org/3548604:07
*** tianst has quit IRC04:07
openstackgerritJeremy Stanley proposed a change to openstack-infra/git-review: Implement integration tests.  https://review.openstack.org/3510404:13
openstackgerritJeremy Stanley proposed a change to openstack-infra/git-review: Implement integration tests.  https://review.openstack.org/3510404:20
*** locke105 has quit IRC04:29
openstackgerritA change was merged to openstack-infra/git-review: Pin hacking <0.6  https://review.openstack.org/4079904:30
*** dina_belova has joined #openstack-infra04:31
*** locke105 has joined #openstack-infra04:31
openstackgerritJeremy Stanley proposed a change to openstack-infra/git-review: Implement integration tests.  https://review.openstack.org/3510404:35
*** dina_belova has quit IRC04:35
*** dina_belova has joined #openstack-infra04:42
*** dina_belova has quit IRC04:48
*** nati_ueno has joined #openstack-infra04:52
*** nati_ueno has quit IRC05:02
*** nati_ueno has joined #openstack-infra05:02
*** ruhe has joined #openstack-infra05:08
openstackgerritA change was merged to openstack-dev/hacking: Reword H404 description to match check  https://review.openstack.org/4124005:10
*** vogxn has joined #openstack-infra05:14
*** SergeyLukjanov has quit IRC05:18
*** nayward has joined #openstack-infra05:21
openstackgerritSergey Kolekonov proposed a change to openstack-infra/jenkins-job-builder: Existing plugins improvements  https://review.openstack.org/4137505:22
*** SergeyLukjanov has joined #openstack-infra05:30
*** SergeyLukjanov has quit IRC05:31
*** nicedice_ has quit IRC05:43
*** sarob has joined #openstack-infra05:46
*** dina_belova has joined #openstack-infra05:49
*** nati_uen_ has joined #openstack-infra05:50
*** dina_belova has quit IRC05:53
*** nati_ueno has quit IRC05:54
*** sarob has quit IRC06:03
*** sarob has joined #openstack-infra06:03
*** morganfainberg is now known as morganfainberg_a06:04
*** sarob has quit IRC06:07
*** ruhe has quit IRC06:12
*** ruhe has joined #openstack-infra06:14
*** ruhe has quit IRC06:18
*** odyssey4me has joined #openstack-infra06:25
*** lyz has quit IRC06:49
*** lyz has joined #openstack-infra06:50
*** nati_uen_ has quit IRC07:03
*** nati_ueno has joined #openstack-infra07:04
*** morganfainberg_a is now known as morganfainberg07:07
*** dnavale has quit IRC07:09
*** sarob has joined #openstack-infra07:14
*** sarob has quit IRC07:18
*** vogxn has quit IRC07:33
*** ruhe has joined #openstack-infra07:35
*** dguitarbite has joined #openstack-infra07:35
*** fbo_away is now known as fbo07:35
*** psedlak has joined #openstack-infra07:37
*** nati_uen_ has joined #openstack-infra07:52
*** nati_ueno has quit IRC07:55
*** afazekas has joined #openstack-infra07:55
*** afazekas has quit IRC07:55
*** afazekas has joined #openstack-infra07:55
*** vogxn has joined #openstack-infra07:57
*** boris-42 has joined #openstack-infra08:08
*** jpich has joined #openstack-infra08:09
*** zul has quit IRC08:13
*** yaguang has quit IRC08:16
*** yaguang has joined #openstack-infra08:28
*** dguitarbite has quit IRC08:29
*** markmc has joined #openstack-infra08:37
*** zehicle has quit IRC08:45
*** vogxn has quit IRC08:53
*** yaguang has quit IRC09:00
*** koobs has quit IRC09:07
*** koobs has joined #openstack-infra09:07
*** vogxn has joined #openstack-infra09:08
*** yaguang has joined #openstack-infra09:17
*** mrmartin has joined #openstack-infra09:24
*** SergeyLukjanov has joined #openstack-infra09:49
openstackgerritSerg Melikyan proposed a change to openstack-infra/config: Added murano-common project  https://review.openstack.org/4163409:50
chmouelmordred: is it expected? http://pastie.org/pastes/8232222/text?key=vvw9rh55njzzvqtvoji5rg09:57
chmouelmordred: running tox -epy27 on nova try to install stuff as root09:57
chmouelmordred: running tox -epy27 on nova try to install stuff in /usr/local09:57
*** dina_belova has joined #openstack-infra09:59
*** ruhe has quit IRC10:01
*** koolhead11 has quit IRC10:08
*** nayward has quit IRC10:11
*** nayward has joined #openstack-infra10:13
*** vogxn has quit IRC10:22
*** mrodden1 has joined #openstack-infra10:31
*** nati_ueno has joined #openstack-infra10:32
*** cp16net_ has joined #openstack-infra10:33
*** tianst20 has joined #openstack-infra10:34
*** comstud_ has joined #openstack-infra10:35
*** lifeless_ has joined #openstack-infra10:36
*** samalba_ has joined #openstack-infra10:39
*** SergeyLukjanov has quit IRC10:40
*** nati_uen_ has quit IRC10:40
*** mrodden has quit IRC10:40
*** lifeless has quit IRC10:40
*** ladquin has quit IRC10:40
*** sdake_ has quit IRC10:40
*** tian has quit IRC10:40
*** cp16net has quit IRC10:40
*** ianw has quit IRC10:40
*** yamahata has quit IRC10:40
*** dansmith has quit IRC10:40
*** harlowja has quit IRC10:40
*** Vivek has quit IRC10:40
*** samalba has quit IRC10:40
*** akscram has quit IRC10:40
*** maurosr has quit IRC10:40
*** comstud has quit IRC10:40
*** uvirtbot has quit IRC10:40
*** cp16net_ is now known as cp16net10:40
*** samalba_ is now known as samalba10:40
*** ianw has joined #openstack-infra10:45
*** ruhe has joined #openstack-infra10:45
*** sdake_ has joined #openstack-infra10:45
*** sdake_ has joined #openstack-infra10:45
*** uvirtbot has joined #openstack-infra10:46
*** ladquin has joined #openstack-infra10:46
*** dansmith has joined #openstack-infra10:46
*** Vivek has joined #openstack-infra10:46
*** akscram has joined #openstack-infra10:46
*** maurosr has joined #openstack-infra10:46
*** yamahata has joined #openstack-infra10:46
*** SergeyLukjanov has joined #openstack-infra10:46
*** yaguang has quit IRC10:52
*** ruhe has quit IRC10:53
openstackgerritSerg Melikyan proposed a change to openstack-infra/config: Added murano-common project  https://review.openstack.org/4163410:55
*** fbo is now known as fbo_away10:56
*** vogxn has joined #openstack-infra10:56
*** zehicle has joined #openstack-infra10:57
*** mjfork has joined #openstack-infra10:58
*** pcm_ has joined #openstack-infra11:00
*** vogxn has quit IRC11:00
openstackgerritMarton Kiss proposed a change to openstack-infra/config: Add groups to gerrit  https://review.openstack.org/4164311:02
*** SergeyLukjanov has quit IRC11:02
openstackgerritZane Bitter proposed a change to openstack-infra/reviewstats: Update Heat core team  https://review.openstack.org/4164411:06
openstackgerritZane Bitter proposed a change to openstack-infra/reviewstats: Make reviewers output line up  https://review.openstack.org/4164511:06
*** lcestari has joined #openstack-infra11:21
*** dina_belova has quit IRC11:21
openstackgerritSerg Melikyan proposed a change to openstack-infra/config: Fix ACL for Murano projects  https://review.openstack.org/4165011:21
*** dina_belova has joined #openstack-infra11:24
*** ruhe has joined #openstack-infra11:25
*** woodspa has joined #openstack-infra11:25
*** adalbas has joined #openstack-infra11:26
*** vogxn has joined #openstack-infra11:34
*** SergeyLukjanov has joined #openstack-infra11:37
*** fbo_away is now known as fbo11:43
*** ArxCruz has joined #openstack-infra11:47
*** dina_belova has quit IRC11:57
Alex_GaynorOooh, I like the styling of the completness bars on zuul status11:58
*** rfolco has joined #openstack-infra11:59
*** yaguang has joined #openstack-infra11:59
*** dina_belova has joined #openstack-infra12:01
*** dina_belova has quit IRC12:05
openstackgerritwill soula proposed a change to openstack-infra/jenkins-job-builder: Adding support for the Warnings plugin  https://review.openstack.org/4062112:08
*** zul has joined #openstack-infra12:11
*** mriedem has joined #openstack-infra12:13
openstackgerritRyan Petrello proposed a change to openstack-infra/config: Add pecan project.  https://review.openstack.org/4155212:16
*** Ryan_Lane has joined #openstack-infra12:17
*** weshay has joined #openstack-infra12:19
*** mberwanger has joined #openstack-infra12:20
openstackgerritwill soula proposed a change to openstack-infra/jenkins-job-builder: Adding support for the Warnings plugin  https://review.openstack.org/4062112:22
*** ruhe has quit IRC12:22
openstackgerritRyan Petrello proposed a change to openstack-infra/config: Add pecan project.  https://review.openstack.org/4155212:22
*** dina_belova has joined #openstack-infra12:24
*** dnavale has joined #openstack-infra12:24
*** sandywalsh has quit IRC12:25
openstackgerritwill soula proposed a change to openstack-infra/jenkins-job-builder: Adding support for the Warnings plugin  https://review.openstack.org/4062112:33
*** dina_belova has quit IRC12:34
*** dina_belova has joined #openstack-infra12:34
*** psedlak has quit IRC12:35
*** zul has quit IRC12:37
*** sandywalsh has joined #openstack-infra12:39
*** zul has joined #openstack-infra12:40
*** afazekas has quit IRC12:41
*** prad_ has joined #openstack-infra12:56
*** zul has quit IRC13:02
*** prad_ has quit IRC13:03
*** mberwanger has quit IRC13:03
*** zul has joined #openstack-infra13:04
*** anteaya has joined #openstack-infra13:05
*** vogxn has quit IRC13:05
*** dprince has joined #openstack-infra13:07
*** ruhe has joined #openstack-infra13:10
*** psedlak has joined #openstack-infra13:12
*** zul has quit IRC13:15
*** prad_ has joined #openstack-infra13:16
*** Ryan_Lane1 has joined #openstack-infra13:17
*** Ryan_Lane has quit IRC13:17
*** burt has joined #openstack-infra13:18
*** dina_belova has quit IRC13:19
*** dina_belova has joined #openstack-infra13:22
*** ruhe has quit IRC13:28
*** boris-42 has quit IRC13:29
*** boris-42 has joined #openstack-infra13:30
*** changbl has quit IRC13:34
*** pentameter has joined #openstack-infra13:35
*** jpeeler has joined #openstack-infra13:37
*** afazekas has joined #openstack-infra13:37
*** jpeeler has quit IRC13:37
*** jpeeler has joined #openstack-infra13:37
*** anteaya has quit IRC13:41
*** anteaya has joined #openstack-infra13:41
*** nati_ueno has quit IRC13:45
*** ruhe has joined #openstack-infra13:51
*** dkliban has quit IRC13:57
*** Ryan_Lane1 has quit IRC14:00
*** markmcclain has joined #openstack-infra14:00
*** blamar has joined #openstack-infra14:02
*** zul has joined #openstack-infra14:02
mordredchmouel: no. it should not try to install stuff in /usr/local - that would be a bug if it's doing that14:02
mordredchmouel: but that paste looks like the local sdist command in /opt/stack/nova is trying to run in a directory where devstack has already run python setup.py develop - so the egg-info dir is now owned by root14:03
mordredchmouel: I thought I put a patch in to devstack already to chown those directories back once it's done14:03
fungimordred: couple quick pbrish questions to bounce off you when you have a sec...14:04
*** ruhe has quit IRC14:04
mordredfungi: shoot14:06
fungii was toying with using git to do tarball validation by checking out the ref on which the tarball is based and then untarring it into the working tree and having git report changed files14:07
fungitested with nova and it *almost* works perfectly, with a couple packaging-related differences14:07
fungione is that PKG-INFO appears in the tarball. should it be safe to start adding that to everyone's .gitignore?14:07
*** fifieldt has quit IRC14:08
fungithe other is that something in the sdist is making whitespace-only changes to setup.cfg... what would you guess is the culprit, is it necessary, and if so do we want a hacking rule to check that people are formatting their setup.cfg correctly?14:09
fungiand before you ask why i care, i was pondering the fact that we build our release artifacts on untrusted jenkins slaves (because we run code from those repos to generate them), but if we get around to setting up some signing infrastructure it would be great to have a trusted slave test that the tarballs actually haven't been tampered with14:10
*** ruhe has joined #openstack-infra14:11
openstackgerritMark McLoughlin proposed a change to openstack/requirements: Allow use of oslo.messaging 1.2.0a3  https://review.openstack.org/4169614:11
*** rnirmal has joined #openstack-infra14:12
chmouelmordred: I think I haven't done more than a stack.sh and python setup.py develop in nova, nothing as root, so perhaps it's in devstack14:13
mordredchmouel: yeah - devstack does sudo python setup.py develop14:13
mordredchmouel: (this is another thing that would be made easier by having a global venv)14:14
*** dina_belova has quit IRC14:14
chmouelmordred: yeah so is there technichal impediment to global venv?14:14
mordredchmouel: speaking of which - https://review.openstack.org/#/c/40534/ is a patch for that which fails when it gets to swift14:15
mordredchmouel: basically, getting the patch to work is step one :)14:15
*** yolanda has joined #openstack-infra14:15
chmouelmordred: :)14:15
mordredfungi: yes. add PKG-INFO to everyone's gitignore14:15
*** cppcabrera has joined #openstack-infra14:16
mordredI believe setuptools 'helpfully' makes changes to setup.cfg (remind me to see if I can stub out the part of setuptools that does that in pbr)14:16
mordredfungi: but if you can figure out what it is that setuptools wants to do differently in our setup.cfg, then sure, a hacking rule sounds sensible14:17
mordredand that seems like a sensible thing to do - especially when you consider that we do have signatures on the git tags too14:17
fungimordred: great, i'll open a pbr bug about the "helpful" setup.cfg normalization and, if someone deems fixing it is impossible/unreasonable then i'll write a hacking rule14:17
chmouelmordred: looking at it, do you have by any chance the stack trace? digging  into the logs is like doing the same jump as that guy http://openstackreactions.enovance.com/wp-content/uploads/2013/08/6721699501.gif14:17
yolandahi, i'm doing a test deployment with zuul, and having this error: ERROR zuul.Gearman: Job <gear.Job 0x7f37f005af50 handle: None name: build:gate-noop unique: 7fc91b2bb38047a8831f13f83a8dbfb1> is not registered with Gearman14:19
fungimordred: right, eventual workflow i imagine is that the trusted upload slave has a list of keys of known signers so it can validate the tag, then retrieves and compares the tarball to make sure it's not tampered with, then generates a signature of the tarball with a dedicated archive signing key, then uploads the tarball and detached signature to pypi and wherever else we like14:19
yolandado i need to configure gearman in some way, do i need to configure the jobs?14:19
mordredchmouel: not yet - I haven't been able to sort out where it's unhappy yet - I'll see if I can narrow it down and send you more specific thoughts14:19
chmouelmordred: thanks let me know if i can be any help14:20
mordredfungi: yah14:20
fungiyolanda: i believe you configure the jenkins-gearman plugin once installed in jenkins to tell it where to find zuul, at least14:20
yolandafungi, i did it, installed jenkins with gearman plugin, and i configured that properly14:21
yolandaand i have a job in jenkins called gate-noop, not sure if that's right14:21
fungiyolanda: lemme refresh my memory of our zuul config to see if it needs to know to expect gearman connections from specific addresses14:22
*** dnavale has quit IRC14:22
yolandafungi, i configured zuul to setup gearman address like that14:22
yolandagearman is correctly listening in that port14:23
openstackgerritMatthew Treinish proposed a change to openstack-infra/config: Add testr parallel run to gate pipeline nonvoting.  https://review.openstack.org/4169814:24
fungiyolanda: i believe that tells zuul where to find the gearman server (we simply set ours to letting it know to connect to itself)14:24
* fungi reviews the zuul config documentation14:25
yolandafungi, we have our gearman server in a separate instance14:27
*** datsun180b has joined #openstack-infra14:27
fungiyolanda: current doc at http://ci.openstack.org/zuul/zuul.html#configuration suggests that server is the dns name or ip address of the gearman server to which you wish to connect, and then there's a separate port option if you want it to use something other than the default gearman service port number14:27
fungiand i guess if you're using an external gearman server, then you probably also want to make sure you don't set gearman_server.start in the zuul.conf14:28
*** rcleere has quit IRC14:28
fungithough i don't think it would break anything to have it running, it wouldn't be needed'14:30
*** cppcabrera has left #openstack-infra14:30
yolandafungi, i'll update that in config file14:30
fungiand presumably your jenkins-gearman plugin is configured to connect to the same gearman server as you told zuul to connect to14:30
yolandagearman_server is set to start=false, but server and port were wrong14:30
yolandaand yes, jenkins-gearman-plugin is connecting to the same gearman-server14:31
fungiyolanda: start=false is the default, according to the documentation, so you could also just leave the entry out of the config entirely14:31
fungiwhatever your preference14:31
yolandafungi, so that should be enough? do we need to define the jobs in jenkins, for example the gate-noop?14:33
fungiyolanda: yes, jenkins needs to know what the job should do. we use a separate tool called jenkins-job-builder to create/update jobs on our jenkins masters from a controlled set of templatized yaml14:34
*** mrmartin has quit IRC14:35
yolandafungi, i configured manually a gate-noop job that simply does an echo "SUCCESS" but it doesn't seem to be called14:35
fungibut the no-op job we have just tells a slave to run /bin/true (so that we at least exercise that a worker got the job and ran something)14:36
anteayachmouel: I see you are speaking in gifs, I have to say I like gif-speak better than acronyms14:37
yolandafungi, so even that gate-noop isn't running then14:37
fungiyolanda: it's probably time to start peering into the zuul logs then to see whether it got a trigger it associates with that job, whether the job got queued, et cetera14:38
yolandafungi, now that we have the infrastructure more clear, we'll retry the change14:39
fungijeblair may also have some more specific troubleshooting suggestions14:39
fungiyou're welcome14:39
fungiand also, i'm very excited to see yet another organization starting to try out and possibly use zuul14:40
jeblairyolanda: try "telnet 4730" and then type "status"14:40
fungithere are only a few so far14:40
jeblairyolanda: that should list the functions defined on the gearman server14:40
*** changbl has joined #openstack-infra14:41
*** mrmartin has joined #openstack-infra14:41
yolandaso that gate-noop is defined14:41
jeblairyolanda: then it sounds like a problem with zuul connecting with the gearman server; have you restarted since fixing the configuration?14:43
*** dkliban has joined #openstack-infra14:43
yolandajeblair, we will test a new patchset now14:44
*** dina_belova has joined #openstack-infra14:44
yolandahttp:// : SUCCESS in 0s14:46
yolandabetter now14:46
jeblairyolanda: yay!14:47
*** ruhe has quit IRC14:47
yolandajeblair, is there any minimum requirement for gearman-server? i tried yesterday with old version from precise package and even wasn't triggering error messages. I manually built gearman-server now with the latest tarball and we saw some progress14:48
Bobbadoes Zuul pick up a change that has been moved from draft to published without any other changes?14:48
fungiBobba: you need to leave a comment of "recheck no bug" on the published patchset14:49
fungiBobba: the version of gerrit we're running does not emit any events related to draft changes, not even an event to signal that a draft has been published14:49
Bobbafair enough14:49
jeblairyolanda: it actually needs (works best with) the 'gear' server (which is what it runs if you tell it to use the 'internal' server, or you can run it separately with geard').  there are two reasons:14:50
Bobbaohhhhhh okay14:50
fungiBobba: subsequent patchsets, if uploaded normally (not as drafts) will be tested as usual14:50
jeblairyolanda: cancel queue items -- that should be in the latest tarball14:50
jeblair(of the gearman server)14:50
Bobbathanks fungi14:50
fungiBobba: we also recommend using the "work in progress" button to indicate a patch is not ready for review, instead of drafts (because they are broken in a number of ways)14:51
jeblairyolanda: and strict fifo ordering of the queue -- if you use gearman-server and enable round-robin, it should be mostly okay, but it still won't quite run jobs in the order zuul tells it to14:51
yolandajeblair, so you don't use gearman-job-server?14:52
jeblairyolanda: so basically -- the latest gearman-server tarball should mostly work (but is untested) and should be fine if you don't have much contention for slaves, but if you do, the jobs may run in a less-optimal server14:52
Bobbafungi: can git review upload a WIP change?  I was uploading them as draft14:52
Bobbafungi: or is the recommendation to upload then press the button ;)14:52
jeblairyolanda: no, we use geard; it's included in python-gear -- it's a simple server that just does enough for zuul.14:52
chmouelis anyone knows how i can have the equivalent of -s in testr?14:52
chmouelanteaya: :)14:52
jeblairyolanda: i would like to patch gearman-job-server to support fifo queues, but have not had time14:53
yolandajeblair, good to know, thx14:53
fungiBobba: we have a wishlist bug open against git-review to put a wip option in, but since we're hopefully soon upgrading to a newer gerrit, we're heavily reworking how wip works anyway and would need to change the implementation in git-review to match14:53
Bobbaunderstood.  Thanks fungi14:54
fungiBobba: there is also a gerrit ssh api call which can set wip status, if you want to alias that command. i'll dig up the link to the documentation14:54
Bobbanah - I can do it manually the few times I do it114:54
mordredfungi: there _used_ to be a gitreview feature14:55
mordredit got lost in the undo of the git-review refactoring14:55
*** rcleere has joined #openstack-infra14:56
anteayachmouel: what funtionality are you looking for from -s? (I don't know the context)14:56
mordredbut I agree - waiting for new WIP would be better14:56
chmouelanteaya: have testr show the prints14:56
mordredchmouel: so, all of the prints are going into a detail record and sould be in the output if the test _fails_14:57
mordredhowever ... there's a couple of things you can do (and we're working on getting this better served by the ui)14:57
chmouelmordred: nice thanks.14:58
mordredchmouel: if you want to run just a single test and not capture any of the output:14:58
mordredchmouel: python -m testtools.run python.path.to.test.function14:58
chmouelperfect that works14:59
*** mrodden1 has quit IRC14:59
*** mrmartin has quit IRC15:04
openstackgerritMonty Taylor proposed a change to openstack-infra/jeepyb: Create repo when we create the cgit config file  https://review.openstack.org/4127915:07
mordredpleia2: fixed your reviews  ^^ thans!15:08
mordredfungi: pleia2 says she's tested that that does what we expect it to15:08
mordredpleia2: did you test (or do we know) if this is going to create the repos with the right owner?15:08
*** _TheDodd_ has joined #openstack-infra15:09
anteayahey _TheDodd_15:10
_TheDodd_Hey, anteaya!15:10
openstackgerritA change was merged to openstack-infra/reviewstats: Update Heat core team  https://review.openstack.org/4164415:10
_TheDodd_How have you been?15:10
anteayagood thanks15:10
anteayabusy as always, and yourself?15:11
fungimordred: too awesome--i meant to review it, so now intend to do so when the call i'm on wraps up15:11
openstackgerritA change was merged to openstack-infra/reviewstats: Add requirements to note that paramiko is needed  https://review.openstack.org/3987415:11
_TheDodd_Great! Busy as well.15:11
*** mrodden has joined #openstack-infra15:11
anteayawhen you get a chance, skim the channel logs15:11
_TheDodd_Any thing wild and crazy going on these days, anteaya?15:12
_TheDodd_Ok, will do.15:12
anteayaI do believe that jeblair has some thoughts about the bug labelling patch15:12
_TheDodd_-infra or ref?15:12
_TheDodd_That's cool.15:12
anteayathought you would want to know15:12
openstackgerritA change was merged to openstack-infra/reviewstats: Make reviewers output line up  https://review.openstack.org/4164515:15
anteaya_TheDodd_: timestamps 2013-08-12T23:55:06 to 2013-08-13T00:18:2815:15
*** hughsaunders has quit IRC15:16
anteaya_TheDodd_: those are UTC timestamps15:18
openstackgerritMonty Taylor proposed a change to openstack-infra/reviewstats: Add TripleO projects  https://review.openstack.org/4170815:22
openstackgerritMonty Taylor proposed a change to openstack-infra/reviewstats: Updated Ironic core team  https://review.openstack.org/4170915:22
jpmelosGuys, is there a way to retrigger jobs in Zuul for patches that are already open in Gerrit?15:22
jpmelosOr manually trigger them. =)15:22
Bobbajpmelos: recheck no bug15:22
jpmelosBobba: I didn't understand.15:23
Bobbajpmelos: add it as a comment to the gerrit review15:23
dstufftmordred: I'm fucking with SAT solvers to try and make a real dep resolver for pip15:23
jpmelosI just need to add that string as a comment in Gerrit?15:24
fungidstufft: the entire python world will worship you if you make that happen15:24
dstufftfungi: current status is trying to understand SAT solvers :V I think I have a small basic example figured out though15:25
mordreddstufft: oh my. that's gota be fun15:25
jpmelosBobba: Didn't seem to work.15:25
anteayajpmelos: can you post the url of the patch in question, please?15:26
openstackgerritA change was merged to openstack-infra/reviewstats: Add TripleO projects  https://review.openstack.org/4170815:26
jeblairmordred, fungi, clarkb: a review on this: https://review.openstack.org/#/c/41565/  would help set the direction of my work today.15:26
mordredjeblair: I will review it right now!15:26
jpmelosanteaya: Unfortuantely, I can't. It's inside the company VPN only.15:26
openstackgerritA change was merged to openstack-infra/reviewstats: Updated Ironic core team  https://review.openstack.org/4170915:26
dstufftmordred: it's.. interesting. as far as I can tell all the SAT solvers are written in things like C or C++, so I'm trying to decipher one of them and write a python implementation.15:27
mordredjpmelos: we have the retrigger functionality expressed in our zuul layout config15:27
Bobbajpmelos: ahhhh - the rules that cause zuul to trigger a new job are in the config file - perhaps they aren't set up in the same way as for OpenStack?15:27
jeblairmordred, fungi, clarkb: https://review.openstack.org/#/c/41517/  is step 2 of "get rid of d-g inprogress jobs"15:27
jpmelosBobba: Haha, OK. I'll take a look at openstack layout and mimic that.15:28
*** ruhe has joined #openstack-infra15:28
mordredjeblair: +2 on nodepool from me15:28
mordredjeblair: and I love the idea of it being its own thing15:28
jeblairmordred: good, i hoped you would.  i'm not sure where the dib elements should end up living; do you have thoughts on that?15:29
jeblairmordred: i expect to put the equivalent scripts in the nodepool repo for now.15:30
anteayajpmelos: ah okay, well for conversation's sake, if you were to add the recheck no bug comment, and I had the url of the patch, I could go to zuul's status page at http://status.openstack.org/zuul and confirm your findings15:30
mordredjeblair: I think the elements can probably live with the scripts until we're happy with how it all works for expediency - but I think that eventually the elements probably just want to be their own repo?15:31
jeblairmordred: ok.  or maybe even a directory in infra/config?  i'm thinking we may accumulate several elements for several different node types15:32
mordredjeblair: yes. I'm imagining we will - and I also thought about infra/config ... although I'd love to continue the thought of breaking that up more so that we could give more fine grained access (like I'd love to give russellb and pabelanger +2 on the asterisk stuff)15:33
mordredjeblair: but my feelsings are 100% mushy on this topic15:33
*** hughsaunders has joined #openstack-infra15:33
russellbmordred: can always just give honorary +2 via +115:34
russellbmordred: oh russellb reviewed it already, so i don't really have to look quite as closely, or whatever15:34
mordredrussellb: your +1 are always honorary +2 :)15:34
russellbsounds like less work that way, heh15:35
russellbnot that I want to stifle ifra improvements!  you guys work magic15:35
jeblairmordred: yeah.  it's a bit hard for me to imagine that we'd end up with core teams dedicated to a single dib template each; and that they would change enough for that to be an issue; maybe infra/config or a single new repo as a starting point and see how it goes.  anyway, that's still like 10 steps ahead of where we are.15:35
mordredjeblair: so many steps!15:35
mordredjeblair: wow. I clearly have not been following your work closely enoguh15:35
mordred"Use the new gearman-plugin single-use feature"15:36
*** vogxn has joined #openstack-infra15:36
jeblairmordred: :) it marks a node offline after the job completes.15:37
mordredjeblair: remember when we wanted the jclouds plugin to be able to do that?15:37
jeblairmordred: it should make things nicer with our current scripts (less load and race conditions with the inprogress jobs), and will also make the nodepool workflow better.15:38
jeblairmordred: good times15:38
mordredyeah. I love the idea. I have +2'd your patches15:39
jeblairmordred: you should subscribe to my newsletter15:39
pleia2good morning15:39
mordredjeblair: can I use that to purchase your goods and services?15:39
anteayapleia2: morning15:42
mordredmorning pleia215:42
pleia2mordred: it's logging in with the cgit user for replication over ssh and that's the owner we want, should be good15:44
mordredpleia2: great!15:47
pleia2and hopefully replication is actually trying to put them in the project.git directory for each project, that's the only thing I'm not 100% on15:48
BobbaAny chance I could get reviews of https://review.openstack.org/#/c/41464/? I (very embarassingly) broke XenServer support in DevStack so getting the fix in quickly would help me save a little face ;)15:48
*** pcrews has quit IRC15:49
*** ruhe has quit IRC15:51
anteayaBobba: is that patch restricted to certain users? because I am unable to view it15:57
openstackgerritJason Meridth proposed a change to openstack-dev/hacking: Adds ability to ignore H231, python3x except compatability check  https://review.openstack.org/4171315:57
*** ruhe has joined #openstack-infra15:58
pleia2mordred: patch looks good but I think it needs a recheck15:59
Bobbavery weird anteaya ...15:59
pleia2anteaya: if you copied the ? at the end of the url too, it won't work16:00
pleia2anteaya: just https://review.openstack.org/#/c/41464/16:00
anteayathat was it, I had just selected the link and clicked it16:00
anteayaremoving the ? at the end shows me the patch, thanks pleia216:00
Bobbaanteaya: Sorry!16:01
pleia2morning clarkb16:01
clarkbjeblair will review shortly16:01
Bobbamy fault :/ of course ?'s are part of a URL!16:01
Bobba :)16:01
anteayaBobba: we both learned something, no worries16:01
*** nicedice_ has joined #openstack-infra16:01
anteayamorning clarkb16:01
jgriffithmordred: ping16:02
mordredjgriffith: it wasn't me!16:03
*** sarob has joined #openstack-infra16:03
jgriffithmordred: haha.. probably not, but maybe you can help :)16:03
jgriffithmordred: http://paste.openstack.org/show/44044/16:03
jgriffithmordred: I've been having fits the past week with installing devstack16:03
jgriffithmordred: thought I had most of it sorted... but now I'm hitting this one16:03
mordredjgriffith: it's been a bit of a nightmare - looking16:03
anteayaBobba: I would like to test this before I vote, do I have time?16:04
mordredjgriffith: is this on an old devstack machine which you may have unstack/restacked a few times?16:04
Bobbaanteaya: of course :)  There's always time for testing16:04
anteayaBobba: k, thanks. Testing.16:05
jgriffithmordred: Not really16:05
jgriffithmordred: So here's the deal16:05
jgriffithmordred: my vagrant fails due to the chmod issue16:05
jgriffithmordred: so I log in do the chmod and run stack.sh again16:05
jgriffithmordred: so it's kinda what you suspect16:05
jgriffithmordred: the re-stacking has been the crux of issues at this point16:05
*** dripton_ is now known as dripton16:06
mordredoh. blerg. well that's unpleasant. try this and tell me if it fixes it, and if so I'll try to add it to devstack logic16:06
mordredwhatI want to know is why the pbr in /opt/stack/pbr is reporting: pbr (/opt/stack/pbr),16:07
jgriffithmordred: FYI, I think I can work around it by 'pip unisntall pbr' and rerun, but not sure it's 100%16:07
jgriffithmordred: lemme look16:07
mordredjgriffith: I'd actually try "cd /opt/stack/pbr ; rm -rf *egg-info; git pull --ff-only ; sudo pip install -e ."16:07
mordredbut more interesting is why you have old pbr in /opt/stack/pbr16:08
jgriffithmordred: k16:08
jgriffithmordred: yeah, that I don't know :(16:08
mordredif this is a new vagrant thing - are you caching repos anywhere?16:08
jgriffithmordred: I thought that might be the issue so I did a nuke on my entire vagrant dir16:08
jgriffithmordred: but it's possible it's pickign something up somewhere else16:09
*** mrodden has quit IRC16:09
* jgriffith has so much crap on his laptop who knows16:09
mordredwhat does the vagrant file thing look like? (I've never gotten vagrant to work- but I can tak e a look and see if I see anything)16:09
*** psedlak has quit IRC16:10
jgriffithmordred: hmmm "You asked me to pull without telling me which branch...."16:11
jgriffithmordred: that's different16:11
mordredoh. fascintating. that means it's not on a branch16:11
mordredgit branch output?16:11
jgriffithmordred: indeed... but it git status says master16:11
*** markmc has quit IRC16:11
mordredwait - why is that in /opt/stack? shouldn't it be in /opt/stack/new?16:12
jgriffithmordred: ha!  sure'nuff16:12
jgriffithmordred: nahh.. not running grenade16:12
mordredoh. k16:12
jgriffithmordred: so you're correct though, "git branch output" shows nada16:12
mordredno - I jus tmeant git branch16:13
mordredlike "show me the output of 'git branch'"16:13
mordredsorry - I'm being too terse16:13
jgriffithsays it's on master16:15
jgriffithnow I've created "output" hehe16:15
* jgriffith is just copy/pasting instructions mindlessly16:16
mordredhehe. it's what we all do around devstack breaks16:16
jgriffithmordred: so I can delete and reclone if that's useful16:16
mordredyeah. try that16:16
*** comstud_ is now known as comstud16:16
mordredand then setup.py develop or pip install -e it16:16
*** mrodden has joined #openstack-infra16:18
jgriffithmordred: well the version appears to be updated at least :)16:18
mordrednow - why was it old before...16:18
jgriffiththat I don't know... looking in vagrant cache16:19
*** blamar has quit IRC16:20
*** hughsaunders has left #openstack-infra16:20
*** sarob has quit IRC16:21
jgriffithmordred: must be vagrant related16:22
anteayajgriffith: what version of vagrant are you using?16:22
jgriffithmordred: talked to thingee and he did a fresh spin up with no issues16:22
jgriffithanteaya: 1.216:22
anteayaI just downloaded 1.2.7 yesterday16:23
anteayaso far so good here16:23
jgriffithanteaya: .7 ???16:23
jgriffithI'm on 1.2.216:23
anteaya'tis new16:23
anteayadon't know if that is it or not16:23
jgriffithanteaya: and you're not seeing any issues?16:23
anteayasince yesterday, not so far16:23
fungidstufft: from what i've seen sat solvers are usually written in low-level languages for performance reasons. they tend to be very resource-intensive when solving for large sets16:23
jgriffithanteaya: did you add the chmod to your vagrant files?16:23
anteayaI didn't add chmod to vagrant files16:24
mordredjgriffith: I'm wondering if it's doing anything smart like looking for git repos on your local machien and pre-copying those in?16:24
anteayajust increased the RAM available16:24
jgriffithmordred: I suspect that may be the case16:24
jgriffithmordred: I'm going to nuke my cache directory again and see what happens, but it's odd I've not encountered this before16:24
jgriffithalthough now i want to try 1.2.7 thanks to anteaya :)16:25
anteayaha ha ha, I'm a bad influence16:25
jgriffithindeed :)16:25
dstufftfungi: yea16:25
dstufftfungi: But that's unacceptable for my use cases :) Needs to be pure python. And luckily the sets aren't going to be very large16:25
anteayahere is my list of instructions I follow for vagrant: https://gist.github.com/anteaya/dedca991f761c48bb26216:26
anteayanot very pretty, so be cautious16:26
fungidstufft: i'm definitely eager to see the resulting implementation, and see what performance you get from it16:26
Bobbagah! Sorry dtroyer ... I hate hard tabs!16:26
jgriffithanteaya: ahhhh, I actually have cook-books to do devstack, that could be my problem :)16:27
BobbaCan't we have a check for them? :) And what sneaky side effects are you thinking of that we might have to look for?16:27
anteayajgriffith: perhaps, would have to see the cook-books16:27
jgriffithmordred: anywho... deleted cache and running again16:27
jgriffithmordred: I'll let ya know what happens16:27
mordredjgriffith: cool. I'll be interested to know!16:28
jgriffithanteaya: https://github.com/j-griffith/vagrant_devstack16:28
*** sarob has joined #openstack-infra16:28
jgriffithanteaya: there's a sub-repo there for the cookbooks16:28
anteayagrand, while devstack is stacking, I shall peruse16:29
anteayajgriffith: are you on OSX?16:30
jgriffithanteaya: nahh... ubuntu16:31
anteayajgriffith: k, did you install vagrant as a gem?16:31
jgriffithanteaya: nope, pkg16:32
*** cppcabrera has joined #openstack-infra16:33
anteayasame as me so far, I installed both virtualbox and vagrant from the .debs16:33
*** cppcabrera has left #openstack-infra16:33
*** sarob has quit IRC16:34
pleia2fungi: if you have some time later, I could use some guidance for getting my slides committed, and we are we planning to have an index at http://docs.openstack.org/infra/publications/ right?16:34
*** sarob has joined #openstack-infra16:34
fungipleia2: there is a script already in place which should generate it, but i think it still may not be 100% working16:34
pleia2fungi: ok, I thought it was something like that, I'll take a look16:34
clarkbdstufft: fungi: too bad z3 isn't licensed in a friendly way and I think it depends on C :/16:35
fungipleia2: the script to generate the index lives in the master branch, the different presentations live in their own branches, and the theory is that when you give a talk you tag the commit on that branch (which may or may not include further changes on top of the previous time it was given)16:36
dstufftclarkb: fungi it's ok I found ministat which is 900 lines of C++, so I'm gonna try and port that to Python :V16:36
fungipleia2: i don't think we've gotten to the point of tagging anything yet though16:37
pleia2fungi: oh, so just standard branching we do with commits anyway, but we'll actually continue using them for each presentation16:37
anteayafwiw I find chef recipes easier to grok than puppet manifests, on first glance16:37
clarkbanteaya: probably because chef recipes are just ruby16:38
*** psedlak has joined #openstack-infra16:38
fungii find ansible playbooks to be the most readable of any config management system i've seen so far16:39
anteayaclarkb: yes, that is probably it, I am finding myself just reading them16:39
clarkbjeblair: https://review.openstack.org/#/c/41517/ LGTM and mordred and fungi. I will let you approve when you are ready16:39
anteayajgriffith: I don't do any of the actions in this file when I set up my vagrant: https://github.com/j-griffith/vagrant-openstack/blob/master/templates/default/hostname.erb16:39
Bobbadprince: ping!16:40
*** Bobba is now known as BobBall16:40
*** ruhe has quit IRC16:41
dprinceBobBall: hi16:41
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Add nodepool project  https://review.openstack.org/4156516:42
BobBallLooking at that smokestack failure... I'm very confused... Is it every run now, or is it only a subset? i.e. does it seem random?16:42
anteayajgriffith: it is my understanding that vagrant sets the permissions inside the vm the way it needs, this cookbook was last edited a year ago, I wonder if vagrant has changed the way it sets permissions since then?16:42
*** gyee has joined #openstack-infra16:42
BobBalldprince: It seems to be sporadic but I can't explain that16:42
BobBalldprince: so I'm hoping I'm reading the status pages wrong or it might be per-host or something like that16:43
*** svarnau has joined #openstack-infra16:43
*** ruhe has joined #openstack-infra16:43
*** jerryz has joined #openstack-infra16:44
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Remove unneeded node label parameters  https://review.openstack.org/4151616:44
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Make all devstack-vm jobs single-use  https://review.openstack.org/4151716:44
*** yaguang has quit IRC16:44
*** HenryG_ has quit IRC16:44
dprinceBobBall: Yeah. I'm perplexed by this one.16:45
*** HenryG has joined #openstack-infra16:45
BobBalldprince: other question - how did you get the SS URL in the email?  Did you write it by hand or is there a magic link on smokestack.openstack.org I've not found yet that I can copy :D16:45
BobBalldprince: oh... and now I see it16:45
BobBallstupid Bob16:45
fungiclarkb: i took a stab at your textwrap suggestion for git-review last night... https://review.openstack.org/4080016:45
anteayajgriffith: here is my current Vagrantfile: https://gist.github.com/anteaya/4c96a59008d26110f94816:45
pleia2fungi: hm, actually, I see, overview is it's own branch and not merged with master like we typically would, how exactly would my commit differ from normal?16:46
*** ^d has joined #openstack-infra16:46
fungipleia2: is your update for the overview presentation?16:46
*** emagana has joined #openstack-infra16:46
fungipleia2: or a different presentation?16:46
anteayajgriffith: these are the only edits I made to the file: https://gist.github.com/anteaya/4c96a59008d26110f948#file-my-vagrantfile-L47-L5316:46
BobBalldprince: so it's all hosts, but only a subset of the nova changes - even though all of them should have that change in...16:47
pleia2fungi: new presentation16:47
dprinceBobBall: as far as I could tell yes.16:47
dprinceBobBall: it has definately occurred on multiple hosts16:48
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Give tag upload permissions for os-apply-config  https://review.openstack.org/4173116:49
fungipleia2: we should come up with a (preferably one-word) short name for it which we can use as a branch name16:49
openstackgerritA change was merged to openstack-infra/config: Add nodepool project  https://review.openstack.org/4156516:49
BobBalldprince: do you restart XAPI on each run?16:49
fungipleia2: and then we'll branch from a commit in your commit history common with the existing repo (likely your parent commit?) and then you can push to that branch16:50
*** ladquin is now known as ladquin_afk16:50
mordredfungi, jeblair: ^^ tag upload permission missing for the os-*-config projects and SpamapS needs to just a release16:50
*** mrodden has quit IRC16:50
*** mrodden1 has joined #openstack-infra16:50
*** ftcjeff has joined #openstack-infra16:50
dprinceBobBall: not that I remember16:51
clarkbfungi: reviewed16:51
dprinceBobBall: should we?16:51
pleia2fungi: hm, that last bit is where I need some help (and to document it)16:51
*** ruhe has quit IRC16:52
fungimordred: does os-apply-config-core expect to grow its membership substantially? should there be a smaller group of people who are familiar with signing and pushing tags to release it?16:52
jgriffithmordred: what version of PBR are you expecting?16:52
BobBalldprince: maybe - but I'm not clear how that'd cause this... thinking that it's been cached somewhere, possibly with an address or something which might account for the seemingly randomness... but then that isn't quite borne out by the code I'm looking at16:52
mordredjgriffith: 0.5.2116:52
mordredjgriffith: is what's in tip of trunk16:53
*** pcrews has joined #openstack-infra16:53
jeblairfungi, mordred: right, we normally use the -ptl group for that.16:53
anteayaBobBall: I'm requesting a rebase16:53
mordredjeblair: should we go ahead and make one of those?16:53
jeblairmordred: yes16:53
mordredk. amending. I will also go learn more things16:53
jeblairmordred: to avoid cleaning up the result of bad tags16:53
jgriffithmordred: so interestingly enough, I nuked my cache and it appears as though that was in fact the issue16:53
fungimordred: can't hurt. it mainly helps us not to have to help people unscrew their screwed-up release mistakes16:53
*** fbo is now known as fbo_away16:53
fungier, what jeblair said16:53
mordredjgriffith: excellent. the cache logic seems unhappy then16:53
BobBallanteaya: Sure - done!16:54
jgriffithmordred: indeed, wonder if it doesn't go past the major number??16:54
jgriffithmordred: anywho... thanks for walking through it with me16:54
jeblairmordred: the members of the -ptl group need to read and understand the instructions in https://wiki.openstack.org/wiki/GerritJenkinsGithub#Tagging_a_Release16:54
mordredSpamapS: ^^16:54
jgriffithmordred: I'll let ya know if I discover anything else interesting on the topic16:54
anteayaBobBall: great, I'm destroying and starting again, will ping with results16:55
mordredjgriffith: my pleasure! and I'm glad it was something we were able to find16:55
*** sarob has quit IRC16:55
jgriffithanteaya: thanks to you as well :)16:55
*** ftcjeff has quit IRC16:56
SpamapSmordred: ok so -ptl isn't just the PTL16:56
anteayaglad to help, jgriffith16:56
SpamapSit is the PTL's peeps?16:56
clarkbjeblair: what is the difference between a computer and a node in jenkins?16:56
jeblairclarkb: beats the hell out of me16:56
SpamapSclarkb: a node may also be filled with puss?16:56
mordredSpamapS: yeah, well, really, the one thign the ptl grou pcan do is make releases16:56
jeblairclarkb: there are words in the api documentation16:56
mordredSpamapS: so, it's the people the ptl feels can handle the extra duties16:56
jeblairclarkb: they mean very little to me16:56
fungiSpamapS: we've used the term "ptl delegates"16:56
SpamapSsounds good to me16:57
jeblairclarkb: http://javadoc.jenkins-ci.org/hudson/model/Computer.html16:57
jeblairclarkb: http://javadoc.jenkins-ci.org/hudson/model/Node.html16:57
mordredok. I'll do it with a ptl group for now, and we'll let you and lifeless duke it out over whether you should be in it or now16:57
*** dkliban is now known as dkliban_afk16:57
fungiSpamapS: so it could be just the ptl, or addtionally anyone to whom the ptl wishes to delegate release duties16:57
clarkbjeblair: thanks16:57
jeblairclarkb: i mean, i understand the sentences there; but if you can actually make heads or tails of why they are so different and yet nearly interchangable, i'd love to be enlightened.  :)16:57
SpamapSmordred: I like my chances.. 15:22 <+lifeless> SpamapS: cut a release?16:58
clarkbseems fairly subtle, ut computers are more tightly related to executors which is what we are dealing with16:58
SpamapShrm, the ? makes it less like an imperative16:58
*** UtahDave has joined #openstack-infra16:58
jeblairclarkb: yeah, but then jenkins goes and asks whether a "node" can run a job.  why did they chose that?  a node without a computer can't run a job.16:58
BobBallanteaya: pls hold on a min - I'm rebasing again - somehow the sneaky tab got back in!16:58
anteayaBobBall: sneaky tab! okay thanks, will stand by16:59
anteayajust bringing up a fresh ubuntu16:59
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Set tag uploading permissions for TripleO  https://review.openstack.org/4173116:59
fungimordred: No distributions at all found for pkginfo (from -r /home/jenkins/workspace/gate-jeepyb-pep8/requirements.txt (line 6))16:59
fungimordred: did we screw up jeepyb somehow and slip it past the gate? http://logs.openstack.org/79/41279/2/check/gate-jeepyb-pep8/585a90d/console.html17:00
mordredjeblair, fungi: ^^ that should be more like it - I also added dib while I was at it17:00
clarkbjeblair: and we still use the node to deal with project labels17:00
BobBallanteaya: Done!17:01
fungimordred: so dib isn't going to use milestone-proposed workflow but just direct release tagging?17:01
mordredfungi: is it? I thought they were going to operate more like the client libs?17:02
anteayaBobBall: great, thanks17:02
*** danger_fo_away is now known as danger_fo17:02
mordredfungi: I'm willing to wait for lifeless to get up and verify17:02
mordredfungi: it's possibly I simple am mistaken17:02
fungimordred: no idea--that's why i'm asking, since you removed their milestone-proposed management in the dib acl17:02
fungijust wanted to make sure that was their plan17:02
mordredfungi: good call.17:03
mordredlifeless_: ping when you get up17:03
openstackgerritJames E. Blair proposed a change to openstack-infra/zuul: Test that custom functions are live-reloaded  https://review.openstack.org/3955117:03
SpamapSdib is going to be more like the client libs AFAIK too17:03
fungimordred: i'm cool with taking SpamapS's word for it in this case, but up to you17:04
*** ruhe has joined #openstack-infra17:05
*** mrodden has joined #openstack-infra17:05
clarkbjeblair: for the test that is broken, is commenting it out preferable to removing it?17:05
openstackgerritRussell Bryant proposed a change to openstack-infra/reviewstats: Fix json syntax errors  https://review.openstack.org/4173617:06
jeblairclarkb: i feel like it would be a useful test -- it would be nice actually if it could be used to validate, say, that this change doesn't completely break assumptions about node labels, etc.17:06
openstackgerritA change was merged to openstack-infra/reviewstats: Fix json syntax errors  https://review.openstack.org/4173617:06
clarkbjeblair: do you intend to try and fix it in a new patchset?17:07
jeblairclarkb: however, the test does break, and yet jenkins appears to operate correctly in reality.17:07
jeblairclarkb: i don't, but perhaps someone who knows more about jenkins testing might.17:07
*** mrodden1 has quit IRC17:07
SpamapSfungi: we can wait 3 hours to see what lifeless has to say.17:08
*** openstackgerrit has quit IRC17:09
*** openstackgerrit has joined #openstack-infra17:09
*** nati_ueno has joined #openstack-infra17:10
clarkbjeblair: I will take a quick look at it and bug zaro as needed17:11
jeblairfungi: re-eval? https://review.openstack.org/#/c/41516/17:15
fungijeblair: yep, was just in the middle of pulling that one back up17:16
jeblairfungi: cool, thx17:16
*** vogxn has quit IRC17:16
*** afazekas has quit IRC17:16
zaroclarkb: http://maven.apache.org/surefire/maven-surefire-plugin/examples/single-test.html17:18
openstackgerritA change was merged to openstack-infra/config: Remove unneeded node label parameters  https://review.openstack.org/4151617:18
jeblair mvn -Dtest=GearmanPluginUtilTest17:18
openstackgerritA change was merged to openstack-infra/config: Make all devstack-vm jobs single-use  https://review.openstack.org/4151717:18
*** BobBall is now known as BobBall_Away17:18
jeblairclarkb, zaro: ^ that's what i was running yesterday17:18
jeblairclarkb, zaro: the problem seems to be that in the test, the slave does not have a computer17:19
fungiclarkb: responded to your comment on https://review.openstack.org/40800 but in short that was intentional to deal with a shortcoming in the wrap() function17:19
*** harlowja has joined #openstack-infra17:22
openstackgerritPaul Belanger proposed a change to openstack-infra/config: Properly load modules.conf  https://review.openstack.org/4174017:23
pabelangerrussellb, ^17:23
pabelangerjust noticed the typo17:23
mordredfungi: I'm also ok with taking SpamapS word17:23
*** dina_belova has quit IRC17:23
russellbpabelanger: so ...17:23
mordredfungi: I'm pretty sure he knows the consequences of attempting to thwart lifeless_17:23
russellbpabelanger: how is it working now?17:23
fungimordred: relentless trolling for the remainder of his mortal existence?17:24
pabelangerrussellb, opposite of what you implemented.  We need to explicitly load modules, out side the default was used.17:24
mordredfungi: well, he gets that anyway17:24
pabelangerthen, once asterisk see a file change, it will auto reload specific modules17:24
*** dina_belova has joined #openstack-infra17:24
pabelangerin this case, if modules.conf.d changes, the asterisk service is stopped , then started again17:24
pabelangerto force a reload of modules17:24
russellbpabelanger: oh have there been other changes i haven't seen?17:25
*** vipul is now known as vipul-away17:25
*** dkliban_afk is now known as dkliban17:25
pabelangerrussellb, I guess, we merged reload support the other day17:25
*** yolanda has quit IRC17:25
russellboh ok, didn't see it17:25
russellbso this is a fix to recent changes17:26
russellbi was just confused since obviously confibridge is loaded now17:26
russellb+1 then17:26
*** vipul-away is now known as vipul17:27
fungimordred: do you have any idea what's going on with pip not finding pkginfo in the tests for https://review.openstack.org/41279 ? i'm going to recheck it because i can't see anything wrong with jeepyb's requirements and am hoping it was just transient pypi.p.o suck17:27
anteayaBobBall_Away: reviewed, devstack installed this time17:29
anteayawas there something else I can do to evaluate the patch? or is a successful installation the goal?17:29
anteayaalready +1'd17:30
openstackgerritPaul Belanger proposed a change to openstack-infra/config: Clean up warning messages for asterisk  https://review.openstack.org/4174317:31
dhellmannhi, folks17:31
pabelangerrussellb, ^ another17:31
*** UtahDave has quit IRC17:32
dhellmannwe're ready to move pecan over to stackforge, and my co-worker has put together a patch. could you take a look, when you have a moment? https://review.openstack.org/#/c/41552/17:32
*** UtahDave has joined #openstack-infra17:32
fungiclarkb: to your question yesterday about reordering the find command line in https://review.openstack.org/41544 , what do you think about adding -O3 to those finds instead?17:34
fungiclarkb: they're long-running enough they would almost certainly benefit from full optimization17:34
*** cyeoh has quit IRC17:35
jeblairdhellmann: lgtm17:35
dhellmannthanks, jeblair17:35
*** cyeoh has joined #openstack-infra17:35
pabelangerclarkb, jeblair: not sure if you wanted to us confbridge for the meeting today, but https://review.openstack.org/#/c/41740/ will need to be merged.  Otherwise, we can manually load the modules17:36
fungidhellmann: is the pecan-core group small enough that you expect everyone in it to participate in tagging releases?17:36
dhellmannfungi: yes, there's just a few of us for now17:36
dhellmannif that grows, we can create a separate release group later, right?17:36
jeblairfungi: drat i keep missing that117:36
fungijeblair: i'm just great at being a parrot, that's all ;)17:37
jeblairdhellmann: so we really really don't like it when a lot of people can tag17:37
dhellmannok, we'll go ahead and change it to use a separate group17:37
jeblairdhellmann: because tagging something incorrectly means we have to go and do lots of manual cleanup17:37
jeblairdhellmann: i don't think we need to insist on it for pecan....17:37
dhellmannfair enough17:37
jeblairdhellmann: but it might still be a good idea; basically, the people who can tag...17:38
fungidhellmann: pecan-ptl can be the group with the project lead and any delegates who are intimately familiar with https://wiki.openstack.org/wiki/GerritJenkinsGithub#Tagging_a_Release17:38
dhellmannwell, we'd like to get some non-dreamhost folks as core at some point, so I can see creating a separate group17:38
dhellmannfungi: so call it "pecan-ptl"?17:38
fungidhellmann: that's been the convention so far for ptl delegate groups entrusted with release tagging17:39
fungidhellmann: so yes, preferably17:39
dhellmannsure, no problem, just double checking17:39
fungii'll update the review with comments to that effect too17:40
openstackgerritDoug Hellmann proposed a change to openstack-infra/config: Add pecan project.  https://review.openstack.org/4155217:40
*** sarob has joined #openstack-infra17:40
*** jfriedly has joined #openstack-infra17:40
dhellmannjeblair, fungi: updated ^^17:41
* fungi cancels his review comment in that case17:41
dhellmannfungi: if you add me to those groups, I can add ryan and show him how to add the other members when he returns from lunch17:43
fungidhellmann: will do. remind me in about 30 minutes if i don't tell you i've done it already ;)17:44
dhellmannfungi: sounds good, thanks!17:44
*** sarob has quit IRC17:46
*** vipul is now known as vipul-away17:46
openstackgerritA change was merged to openstack-infra/config: Add pecan project.  https://review.openstack.org/4155217:46
mordrednew thing learned for today:17:47
mordredgit diff HEAD | flake8 --patch --select=E121,E122,E123,E124,E12617:47
mordredwill test flake8 things on just the stuff you changed17:47
mordredso potentially useful for turning on checking/gating of rules where the codebase is _not_ clean, but you don't want to introduce new breakage17:48
*** nati_ueno has quit IRC17:48
*** dina_belova has quit IRC17:49
*** dina_belova has joined #openstack-infra17:49
* anteaya goes for a walk17:53
*** zul has quit IRC17:55
fungimordred: neat. i assume that only works for things where flake8 does not need context? or does it go digging for more context to do static analysis things as needed?17:55
mordredfungi: I believe it gets some context17:55
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Sort mailing lists in alpha order  https://review.openstack.org/4175617:56
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Add marketing-content mailing list  https://review.openstack.org/4175717:56
mtreinishclarkb, jeblair, fungi: so I'm trying to track the success rate of the parallel testr tempest job vs the serial job using graphite. But I don't see any data in the graphs for the testr runs.17:57
mtreinishis there something that I need to do?17:57
*** sarob has joined #openstack-infra17:59
*** vipul-away is now known as vipul17:59
jeblairmtreinish: do you have a url to a graph you've been working on?18:00
mtreinishjeblair: this is what I have so far just trying to plot the data for the jobs: http://graphite.openstack.org/graphlot/?from=00:00_20130811&until=23:59_20130813&target=stats.zuul.job.gate-tempest-devstack-vm-testr-full&target=stats_counts.zuul.job.gate-tempest-devstack-vm-testr-full18:01
mtreinishnothing fancy yey18:01
*** boris-42 has quit IRC18:02
openstackgerritA change was merged to openstack-infra/jeepyb: Create repo when we create the cgit config file  https://review.openstack.org/4127918:04
*** blamar has joined #openstack-infra18:05
openstackgerritA change was merged to openstack-infra/zuul: Allow url to be updated  https://review.openstack.org/4102018:07
*** ryanpetrello has joined #openstack-infra18:08
jeblairmtreinish: http://tinyurl.com/n5fxmjk18:10
jeblairmtreinish: we moved some of those stats recently; i think i need to restart graphite or carbon for the old paths to disappears18:11
jeblairmtreinish: (i usually use the composer, but you can s/composer/graphlot/ in the url)18:11
mtreinishjeblair: ah, ok cool18:12
mtreinishthat actually looks pretty close to ready for realtime (just need to wait longer to see if the trend holds)18:12
mtreinishwhile I've got you ear can you take a look at: https://review.openstack.org/4169818:13
mtreinishso we can get testr parallel non voting on the gate queue too18:14
*** jhesketh has quit IRC18:17
*** mrmartin has joined #openstack-infra18:17
*** ruhe has quit IRC18:17
*** boris-42 has joined #openstack-infra18:22
jeblairmtreinish: good, that'll get better data18:22
jeblairmtreinish: you'll notice that the graphite paths have the pipeline in them, so with your zuul config patch, you'll be able to just look at gate data18:23
mtreinishjeblair: cool, yeah that'll be good to remove the bad patches on the check queue from the plots18:24
mtreinishjeblair: yeah hopefully the trend will be clean and come friday or early next week we can make the switch.18:24
jeblairmtreinish: !! :)18:25
mtreinisheveryone will like a 50% reduction in tempest gate job runtime18:25
dhellmannfungi: ping18:26
fungidhellmann: yep, i nabbed bug 1211912 but noticed the -ptl group didn't get created so am rerunning manage-projects while i try to sort out what happened there18:26
uvirtbotLaunchpad bug 1211912 in openstack-ci "Create initial gerrit core member for Pecan" [Wishlist,In progress] https://launchpad.net/bugs/121191218:26
fungithanks for the reminder too18:27
dhellmannfungi: ok, cool. I noticed that, too, but wasn't sure if the groups were created by hand and you stopped in the middle. :-)18:27
fungicreated by the hand of a very lazy and easily distracted script, it seems18:27
*** dina_belova has quit IRC18:28
clarkbjeblair: I have a fix for that test, should I update your chage?18:28
clarkbjeblair: or push a new patchset instead?18:28
*** SergeyLukjanov has quit IRC18:28
clarkbalso it is a very simple change and super annoying once you figure it out18:28
clarkbbut setNodeName is only meant to be called when cloning an instance and not when creating a new one18:29
*** ruhe has joined #openstack-infra18:29
ryanpetrellothanks for your help, fungi18:29
ryanpetrellosorry I'm late to the party :)18:29
jeblairclarkb: awesome!  i'm fine with you updating my change (unless it's substantial and you want git credit)18:29
*** ruhe has quit IRC18:29
fungidhellmann: worked that time. want me to put you or ryanpetrello in one or both groups?18:29
clarkbjeblair: it isn't substantial. I am fine with updating your change. will do that now18:29
fungidhellmann: how about i add you and then assign ryanpetrello's bug to you18:30
dhellmannworks for me18:30
*** zul has joined #openstack-infra18:30
ryanpetrellosounds good18:30
openstackgerritA change was merged to openstack-infra/config: Add testr parallel run to gate pipeline nonvoting.  https://review.openstack.org/4169818:31
comstudwho maintains paste.openstack?18:31
fungidhellmann: ryanpetrello: it's up to you folks now. happy hacking18:31
comstudit seems hosered.18:31
openstackgerritClark Boylan proposed a change to openstack-infra/gearman-plugin: Rework starting/stopping executors  https://review.openstack.org/4148818:31
clarkbjeblair: ^ there you go18:32
fungicomstud: we do. checking for degree of hosx0rd now18:32
comstudi, for instance, can't get http://paste.openstack.org/show/44012/ to load18:32
jeblairclarkb, zaro: do you like the change?18:32
jeblairfungi: let me know if you want another set of eyes/hands18:32
comstudtho i guess, just the main page won't load for me, either18:32
dhellmannfungi: looks good, thanks again for your help18:32
fungi"The RSA host key for paste.openstack.org has changed"18:32
fungijeblair: clarkb: i can't remember, that was just rebuilt recently right?18:33
comstudhah, that's not good18:33
clarkbjeblair: yes, I am much happier with the change now that we are not commenting out a test18:33
clarkbfungi: it was to move off of oneiric in may iirc18:33
jeblairfungi: relatively; current key is in my known hosts18:33
fungijeblair: thanks, fc:30:14:d3:60:44:79:4a:70:9a:06:ee:63:6d:95:8e hopefully?18:33
comstudi got a Proxy Error18:33
comstudif i wait long enough18:34
fungiseems the paste.o.o page took a while for me to load but did eventually come up18:34
comstudyeah, it loads now18:34
fungimay have been a brief load spike18:34
*** ryanpetrello has quit IRC18:34
clarkbfungi: it is deep into swap18:35
*** ryanpetrello has joined #openstack-infra18:35
fungiclarkb: drizzled!18:36
fungiapparently drizzle wants more than a gig to itself18:36
jeblairfungi: yes, that matches the fp in my known hosts (as well as the key on the host in /etc)18:36
fungijeblair: yeah, once i logged in and w showed you and clarkb logged in i assumed it was probably safe18:37
*** nati_ueno has joined #openstack-infra18:37
fungimaybe after the meeting i focus on helping canonical patch cacti for precise so we can install a hand-built security update and start looking at graphs again while we wait for someone to sponsor it into the archive18:38
fungibut my guess is slow leak in drizzled18:38
clarkbprobably a reasonable guess18:38
zarojeblair: looking thru it now. almost done.18:39
clarkbLinuxJedi: is around here somewhere maybe he can tell us to upgrade18:39
fungiso paste has probably been chewing through ram gradually for the past couple months18:39
clarkbthe version of drizzle that we are running is pretty ancient18:39
jeblairfungi: jeez.  do we need to start running debian to get timely security updates?  (it's like 10 years ago but backwards)18:39
jeblairclarkb: i think we identified that as a good first thing to move to trove.18:40
jeblairas soon as we have backups worked out.18:40
jeblairwhich mordred is working on18:40
fungijeblair: well, i think cacti is in "multiverse" which is community-supported security fixes only18:40
fungion lts at least18:40
fungijeblair: clarkb any objections to an emergency restart of drizzle on paste.o.o for now?18:41
jeblairfungi: please18:41
clarkbnot from me, but you may need to kick apache afterwards to restart paste as well18:42
clarkbiirc that was an issue when we moved the host18:42
clarkb(it won't reconnect cleanly or something)18:42
*** nayward has quit IRC18:46
* anteaya is back18:48
fungiyeah, i had to shut down drizzle and apache and then kill an old python process for lodgeit which was still hanging around18:48
funginow that it's started back up, memory pressure hasn't really let up much but i can't see what's squatting in it now18:48
*** melwitt has joined #openstack-infra18:48
jeblairmrmartin: hello!18:49
mrmartinjeblair: i have a merge conflict during zuul/layout update. I rebased the dev branch and solved to conflict. Is it enought to include change-id in the commit message for the review process?18:50
koolhead17pleia2, around?18:50
jeblairmrmartin: that's exactly right.18:51
mrmartinjeblair, ok I'll give it a try18:51
zarojeblair: approved.  thanks for that much needed fix.18:51
jeblairzaro: cool, thanks!18:51
openstackgerritA change was merged to openstack-infra/gearman-plugin: Rework starting/stopping executors  https://review.openstack.org/4148818:53
fungijeblair: clarkb: well, paste.o.o seems nice and responsive after restarting drizzle and apache, but we still don't have enough ram on that vm by half. any guess how much ram the previous paste.o.o had?18:54
pleia2koolhead17: hey18:54
fungijeblair: clarkb: wondering if maybe we scaled it down when we replaced it18:54
jeblairfungi: the other host still exists in the openstack account18:55
fungiahh, perfect18:55
jeblairfungi: i read 60% free18:55
jeblairfungi: (most of it is in buffers/cache, which doesn't count)18:56
*** SergeyLukjanov has joined #openstack-infra18:56
fungijeblair: well, yeah but trying to figure out why it's preferring to leave ~.5g swapped out in favor of buffers/cache18:56
funginot detrimental, but a good bit more than is typical for most systems18:57
fungibased on overall percentage18:57
fungiespecially since i restarted everything which should be consuming any substantial amount of memory, so i would have expected that to get freed18:58
*** datsun180b_ has joined #openstack-infra19:00
clarkbmeeting time19:00
openstackgerritMarton Kiss proposed a change to openstack-infra/config: Add groups to gerrit  https://review.openstack.org/4164319:01
*** datsun180b has quit IRC19:01
*** datsun180b_ is now known as datsun180b19:01
fungilooks like the old paste.o.o also only had 1g ram and similar memory usage patterns, but had 2g of swap instead of just 1g19:02
openstackgerritJason Meridth proposed a change to openstack-dev/hacking: Adds ability to ignore H231, python3x except compatability check  https://review.openstack.org/4171319:02
mrmartinjeblair, patch set 2 is done. Basically we want to keep just the custom changes of the Drupal Commons codebase in the git repo, as an installation profile.19:03
mrmartinjeblair: so we can use make drush to build a distribution tarball from this profile, and it pulls down contrib modules from drupal.org site. It have the advantage, that drupal.org provides a lots of check / audit on contrib modules, so we can keep custom codes in a minimal level.19:05
ttxmordred: all clear to play with merge-milestone-proposed-back-to-master on swift 1.9.119:07
ttxI'm holding on removing the branch19:08
dhellmannfungi: we're having a permission issue with pecan-core, could you lend a hand when you're done with the infra team meeting?19:09
fungidhellmann: sure19:09
*** datsun180b has quit IRC19:10
dhellmannfungi: pecan-core members can only +1 on changesets for the repo, such as https://review.openstack.org/#/c/41764/19:10
*** datsun180b has joined #openstack-infra19:10
dhellmannfungi: no rush19:10
fungidhellmann: i see the issue. rerunning manage-projects one more time to get the acl applied properly19:14
fungii'll let you know when it fiishes19:14
dhellmannfungi: thanks!19:14
*** nati_ueno has quit IRC19:15
*** markmc has joined #openstack-infra19:16
*** adalbas has quit IRC19:17
*** gyee has quit IRC19:25
fungidhellmann: should be all set now. try again19:26
dhellmannfungi: yep, looks good19:26
*** vipul is now known as vipul-away19:27
*** cthulhup has joined #openstack-infra19:27
*** odyssey4me has quit IRC19:28
*** dina_belova has joined #openstack-infra19:28
*** adalbas has joined #openstack-infra19:29
*** chuckieb has quit IRC19:30
*** dina_belova has quit IRC19:31
ryanpetrellofungi: for pecan, it looks like we've got an old branch laying around that's related to our old review process19:36
ryanpetrellois there a process for removing that branch?19:36
ryanpetrelloit's not really necessary now that we're using gerrit19:36
fungiryanpetrello: yeah, i can delete it for you19:37
fungijust a moment19:37
ryanpetrellookay, awesome19:37
*** dina_belova has joined #openstack-infra19:38
*** sarob has quit IRC19:40
*** sarob has joined #openstack-infra19:41
*** dina_belova has quit IRC19:43
*** cthulhup has quit IRC19:44
russellbjeblair: pabelanger: for after the meeting since i missed the topic ... we don't *need* a large group to load test.  we could generate calls that randomly occasionally play a sound prompt from another rackspace instance in the same region.  and then just a couple people call in to actually hear any effects to the quality.19:45
pabelangerrussellb, yup, that works too.  I figured people actually wanted to talk to each other19:49
*** dina_belova has joined #openstack-infra19:50
jeblairrussellb: that's a good idea; can you take care of that?19:50
jeblairpabelanger, russellb: we may still want to get at least several people calling in to increase the diversity of inputs a bit.19:51
lifeless_mordred: pong19:52
*** lifeless_ is now known as lifeless19:52
mordredlifeless_: it was a question about tripleo-ptl team and releases19:52
*** ruhe has joined #openstack-infra19:53
*** pcm_ has quit IRC19:53
lifelessmordred: go on19:55
*** cthulhup has joined #openstack-infra19:55
russellbjeblair: yep19:55
mordredlifeless: the normal pattern for projects is that there is a group, $project-ptl which has the acls to push tags19:55
russellbjeblair: and by take care of it, i mean i'll make pabelanger do it19:56
mordredthis can include the ptl and other people the ptl can handle the extra responsibilty aroud the tag/release process19:56
mordredlifeless: because you have multiple projects with delegated core groups, such as os-apply-config-core19:56
mordredwe wanted to check in with you before doing what seemed like the 'right' thing - which was making a tripleo-core and putting you and Spamaps in it19:56
mordredso that he could cut releases of os-*-config19:57
lifelessuhm, me SpamapS and devananda right now; I want to chat with Ng_holiday, who has spent quite a while maintaining projects too19:57
pabelangerrussellb, jeblair: sure, I know how to hack the gibbson19:57
jeblairmrmartin: tripleo-ptl ?19:57
jeblairmordred: tripleo-ptl ?19:57
jeblairmrmartin: sorry19:57
mordredjeblair: yes. sorry19:59
mordredlifeless: tripleo-ptl19:59
mrmartinjeblair, not yet ;)19:59
* mordred needs to afk again20:00
clarkbjeblair: http://paste.openstack.org/show/44068/ is the exception. It looks like fileinput.input is a global object and somehow we are abusing that in a bad way20:01
*** sandywalsh has quit IRC20:01
fungiryanpetrello: deleted your "next" cruft branch from stackforge/pecan as requested, leaving only master20:01
clarkbjeblair: I have a patch that uses distinct objects per file20:02
*** hub_cap has joined #openstack-infra20:02
ryanpetrellofungi: thanks for your help :)20:02
fungiryanpetrello: any time20:03
clarkbI think it may be related to the logstash stuff hitting logs.o.o a lot more than logs-dev which causes race conditions20:03
hub_capdo we have some implicit requirements like pexpect? i dont see it in global-requirements, and when we tried to add it to a review it puked with the message "Duplicate requirement in HEAD: pexpect"20:03
clarkbhub_cap: can you link to the review that is failing?20:04
hub_capclarkb: https://review.openstack.org/#/c/40859/20:04
fungiclarkb: as i mentioned earlier, not sure if it helps but the terribly simple mysqldump script i run from cron on my personal systems is http://paste.openstack.org/show/44069/20:04
lifelessmordred: yes: me, SpamapS, devananda please.20:04
lifelessmordred: thanks for asking!20:04
clarkbfungi: thanks20:04
*** zul has quit IRC20:04
fungiclarkb: doesn't version them or anything (i separately snapshot the filesystem later on)20:05
openstackgerritElizabeth Krumbach Joseph proposed a change to openstack-infra/jeepyb: Add chown of git repo directories  https://review.openstack.org/4178120:05
fungiso probably fairly similar to what we'd want for bup's purposes anyway20:05
pleia2^^ looked for a more pythony way to do it, but it turns out recursive chown in python is not awesome20:06
uvirtbotpleia2: Error: "^" is not a valid command.20:06
openstackgerritClark Boylan proposed a change to openstack-infra/config: Create distinct fileinput objects in htmlifier.  https://review.openstack.org/4178220:06
fungipleia2: yes, you'll see manage_projects.py shells out20:06
clarkbjeblair: ^ potential fix for the log stuff. completely untested20:06
pleia2fungi: ah, indeed it does20:07
zaroclarkb: would you be willing to approve? https://review.openstack.org/#/c/41307/20:07
* devananda waves20:08
*** datsun180b_ has joined #openstack-infra20:08
clarkbhub_cap: weird20:10
jeblairclarkb: yeah, actually what we should probably do is extend the current bup class20:10
hub_capisint it?20:10
* hub_cap is confused (but whats new)20:10
jeblairclarkb: so it takes an argument about whether to run mysqldump; and then just have the cron script run that before bup (if the arg is set)20:10
clarkbjeblair: ok20:10
*** datsun180b has quit IRC20:11
*** datsun180b_ is now known as datsun180b20:11
fungijeblair: clarkb: that actually sounds really good. takes care of in-progress races and minimizes the window between dump and archiving20:12
*** dprince has quit IRC20:12
clarkbhub_cap: its breaking when checking the trove change and not openstack/requirements20:12
jeblairhub_cap: it's in both requirements and test-requirements20:13
*** sandywalsh has joined #openstack-infra20:13
hub_capya somethign about a dup in our file right clarkb?20:13
jeblairhub_cap: if you want to move it to requirements, remove it from test-requirements20:13
hub_capyes test requirements is requirements + any testing stuffs right?20:13
hub_capas in, if we need in prod, put it in requirements and when we test, itll already be there20:14
jeblairhub_cap: right20:14
hub_capkk thx20:14
clarkbjeblair: I don't see it in requirements20:14
jeblairclarkb: https://review.openstack.org/#/c/40859/3/requirements.txt20:14
*** vipul-away is now known as vipul20:14
clarkboh that change is adding it to requirements but it is already in test-requirements?20:14
jeblairclarkb: correct20:14
clarkbalso using expect for anything production like is uhm scary20:15
hub_capheh of course it is :)20:15
hub_captell apt that20:15
clarkbwhy does apt need expect?20:15
jeblairexpect: failure20:15
hub_capwe expect that a package will be installed, so to speak20:15
clarkbcan't you just query apt after to see if it is installed?20:16
hub_capwell yes, but thats easy20:16
* hub_cap files a bug20:16
jog0why can't I sort my review.openstack.org page by date?20:18
jog0also need oen mre review on https://review.openstack.org/#/c/41523/20:20
*** sandywalsh has quit IRC20:20
jog0if any one is jonesing to do a review20:20
fungiclarkb: i've used expect quite happily in production in years past, though prefer the python pexpect module as a saner solution which doesn't require plumbing the depths of my tcl memory20:26
fungiclarkb: to your question yesterday about reordering the find command line in https://review.openstack.org/41544 , what do you think about adding -O3 to those finds instead?20:33
openstackgerritJames E. Blair proposed a change to openstack-infra/zuul: Add start/complete custom functions  https://review.openstack.org/4178520:33
*** sandywalsh has joined #openstack-infra20:33
jeblairfungi, clarkb, mordred: if any of you have a minute, can you take a look at that ^20:34
jeblairfungi, clarkb, mordred: i have some design questions i'd like to bounce around20:34
*** cthulhup has quit IRC20:35
fungiooh, hooks!20:35
*** woodspa has quit IRC20:35
jeblairbasically, i'm not entirely sure putting that into 'configuration' is the right way to go.  otoh, why not?  :)20:36
fungiexim certainly has no fear of turning its configuration into software ;)20:37
*** sgviking has quit IRC20:37
jeblairother ideas i've had include actually making entrypoints and calling real hooks that way -- elegant, but really heavy weight (i mean, then we'd have to install a plugin just to do this...)20:37
fungijeblair: is the thought that this would also get used to update stats?20:37
jeblairor going with functions like i have there, but limiting them so that they are exclusively used to run gearman jobs (with custom parameters)...20:39
*** dina_belova has quit IRC20:39
fungiseems reasonable20:39
*** vipul is now known as vipul-away20:39
jeblairthe thinking there is that i feel like i'm breaking some api barriers by passing the _scheduler_ into the custom functions, because then they could do anything, most of which would be bad.  but then if i do what i just suggested, then it's assuming a gearman launcher, which is weird.20:40
jeblairfungi: i would like to use these to call gearman functions that are implemented by the nodepool worker20:40
jeblairfungi: so this will replace the inprogress/complete jobs in jenkins20:40
*** ruhe has quit IRC20:41
jeblairyet another option would be to put zmq support in zuul, but i actually like gearman for this because i want these jobs to be queued20:41
clarkbfungi what were you using to test the htmlifier stuff re https://review.openstack.org/#/c/41782/ ?20:42
clarkbthough I am happy enough to approve that and test it as is as the current solution is broken20:42
clarkbjeblair: looking20:42
fungii really don't feel like it's a violation of any implied api contract there, as long as the interface between zuul and the custom function modules is clearly defined, but it does of course allow them to circumvent that interface by mucking with the namespace contents directly20:42
jeblairthe advantage of the patchset i uploaded is that it doesn't assume any knowlede about zuul or the environment -- because it lets the custom functions do anything.20:42
jeblair(and you can see that in my test functions, i abused knowledge of internals to do what i needed; which is roughly what we'd be doing in production)20:43
clarkbjeblair: what if the api was in gearman?20:43
fungiclarkb: testing the htmlifier stuff before it was in production was easier because i was just manually applying changes to the dev chost20:44
clarkbjeblair: eg register workers for these jobs if you want them to do work? otherwise zuul will not submit the jobs20:44
fungiclarkb: er, dev vhost20:44
*** ruhe has joined #openstack-infra20:45
*** vipul-away is now known as vipul20:45
*** adalbas has quit IRC20:45
*** ruhe has quit IRC20:46
fungiclarkb: i suppose you could put a renamed copy of that script in a puppet development environment, modify the logs-dev vhost to point its filter at that name, and then apply on static.o.o from that environment to test20:47
jeblairclarkb: that's an interesting idea, though i think in that case, the jobs would probably have to always run for every build (the current patch lets you specify that they should only be run for certain jobs)20:47
openstackgerritElizabeth Krumbach Joseph proposed a change to openstack-infra/jeepyb: Add chown of git repo directories  https://review.openstack.org/4178120:47
jeblairclarkb: that would be fine for us; nodepool will quickly disregard things it doesn't know about20:48
openstackgerritA change was merged to openstack-infra/jenkins-job-builder: Document escaping braces in template literals  https://review.openstack.org/4097320:48
clarkbjeblair: going back to mysql backups. I don't think we should make them too tightly coupled to bup. Instead have the module do the mysqldump to a configurable dir which will be logrotated by the mysql backup module on some configurable interval. Then bup can just backup that dir20:48
jeblairclarkb: okay; why's that?  i suppose it's true that having a definabel resource per database would make it easier when we're using trove.20:49
openstackgerritElizabeth Krumbach Joseph proposed a change to openstack-infra/jeepyb: Add chown of git repo directories  https://review.openstack.org/4178120:49
fungipleia2: any particular reason for the separate chgrp pass instead of just passing '%s:%s'(CGIT_USER, CGIT_GROUP) as a parameter?20:50
clarkbjeblair: if we ever go to something other than bup we won't have to figure out mysqldump again, and we can do it per database20:50
jeblairclarkb: sounds good then20:51
fungipleia2: er, '%s:%s' % (CGIT_USER, CGIT_GROUP)20:51
ryanpetrellofungi: for stackforge projects that already have external docs (e.g., readthedocs), is there a best practice towards building these with releases?20:51
pleia2fungi: could do that, just didn't think of it in time :) kept trying different quoting and then gave up20:51
ryanpetrelloI know the existing infra stuff will publish to openstack.org somewhere, but we probably don't want that20:51
clarkbjeblair: the custom functions to submit jobs that d-g/nodepool can consume is fine with me. I was just offering an alternative to stick with a more gearman specific interface20:52
clarkbfungi: I think I may just approve the fileinput change and if that doesn't help revert the logs.o.o vhost change20:52
fungiryanpetrello: we have a rtfd hook, though i believe we need to know the rtfd id number and your rtfd project needs to allow access from our account20:52
jeblairfungi: i don't think there's an account involved20:52
fungiclarkb: that seems very low-risk. at most people won't be able to view the logs for a few minutes20:52
fungijeblair: oh, no permissions settings needed? i think we have a doc on it anyway, checking20:53
jeblairryanpetrello: https://github.com/openstack-infra/config/blob/master/modules/openstack_project/files/jenkins_job_builder/config/projects.yaml#L7020:54
clarkbthe puppetlabs mysql module has a backup manifest but I think it attempts to be too smart20:54
jeblairryanpetrello: should just need to do the rtfd bits from that section, substituting your own rtfd id20:54
ryanpetrellojeblair: awesome, thanks for the reference :)20:54
*** ArxCruz has quit IRC20:54
fungilooks like maybe we don't have actual documentation contributed for the rtfd hook, so that's something which would probably fit well in the stackforge.rst20:55
jeblairfungi: ++20:55
*** ArxCruz has joined #openstack-infra20:57
clarkbwhat is a good default mysql backup dir?20:57
fungiclarkb: i tend to just use /var/backups20:57
fungiit has historical precedent20:57
*** boris-42 has quit IRC21:00
ryanpetrellojeblair: is there anything special we need to do in RTFD to authorize this (doc publication)?21:01
jeblairryanpetrello: afaik no21:01
*** ArxCruz has quit IRC21:01
*** moted has quit IRC21:02
*** vipul is now known as vipul-away21:02
*** moted has joined #openstack-infra21:02
*** boris-42 has joined #openstack-infra21:02
openstackgerritElizabeth Krumbach Joseph proposed a change to openstack-infra/jeepyb: Add chown of git repo directories  https://review.openstack.org/4178121:02
openstackgerritJames E. Blair proposed a change to openstack-infra/zuul: Add start/complete custom functions  https://review.openstack.org/4178521:04
openstackgerritElizabeth Krumbach Joseph proposed a change to openstack-infra/jeepyb: Add chown of git repo directories  https://review.openstack.org/4178121:05
*** boris-42 has quit IRC21:07
jeblairclarkb, fungi: https://jenkins01.openstack.org/computer/devstack-precise-hpcloud-az1-9345/21:11
jeblair"Offline due to Gearman request"21:11
jeblairyay :)21:11
*** rfolco has quit IRC21:12
*** jpich has quit IRC21:14
openstackgerritJames E. Blair proposed a change to openstack-infra/devstack-gate: Don't relabel devstack-gate nodes  https://review.openstack.org/4178921:15
jeblairfungi, clarkb: step 3 ^ :)21:16
openstackgerritClark Boylan proposed a change to openstack-infra/config: Add new mysql_backup module.  https://review.openstack.org/4179021:16
openstackgerritClark Boylan proposed a change to openstack-infra/config: Switch etherpad_lite backups to mysql_backup.  https://review.openstack.org/4179121:16
clarkbjeblair: fungi ^ that is also completely untested, but it is relatively simple and I included an example of how we could use it21:16
clarkbwanted more feedback before I started applying it to things (I intend on running that second change on etherpad_dev to test)21:17
clarkbfungi: does https://review.openstack.org/#/c/41782/1 look good to you? if so can you approve it? I will babysit and propose the logs.o.o vhost revert if necessary21:18
fungiclarkb: lgtm, done21:19
clarkbfungi: ty21:19
clarkbjeblair: for step three, are you using the custom functions in the cahnge you pointed out earlier?21:19
clarkbjeblair: or is it just that other custom function which sets the build env var?21:20
jeblairclarkb: the second thing21:20
jeblairclarkb: which is happening now (and gearman plugin is taking those nodes offline because of it)21:20
clarkbjeblair: what need do the custom functions in the second thing accomplish that the current setup does not?21:21
openstackgerritA change was merged to openstack-infra/config: Create distinct fileinput objects in htmlifier.  https://review.openstack.org/4178221:21
jeblairclarkb: they are for nodepool -- to trigger its worker (so we can get rid of the devstack-gate jobs entirely)21:21
*** markmc has quit IRC21:23
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Document RTFD hook job template in StackForge doc  https://review.openstack.org/4179321:23
*** ryanpetrello has quit IRC21:24
*** vipul-away is now known as vipul21:25
*** nati_ueno has joined #openstack-infra21:25
openstackgerritClark Boylan proposed a change to openstack-infra/config: Switch etherpad_lite backups to mysql_backup.  https://review.openstack.org/4179121:25
openstackgerritClark Boylan proposed a change to openstack-infra/config: Add new mysql_backup module.  https://review.openstack.org/4179021:25
fungiclarkb: quick eyeballs on https://review.openstack.org/41781 and approve if you're cool with that? once it's in place i'll approve something which touches review_projects.yaml.erb if there is anything ready to go (or i'll fix the minor issue with https://review.openstack.org/41643 and put that through)21:29
clarkbfungi: I just approved it :)21:30
fungitoo cool21:30
* fungi disappears to eat something briefly. brb21:31
clarkbmtreinish: super excited about the testr work and potential switch to it in the gate.21:31
openstackgerritA change was merged to openstack-infra/jeepyb: Add chown of git repo directories  https://review.openstack.org/4178121:32
*** cp16net has left #openstack-infra21:32
*** gyee has joined #openstack-infra21:33
mtreinishclarkb: yeah, hopefully it'll stay stable21:33
*** cthulhup has joined #openstack-infra21:35
*** weshay has quit IRC21:36
*** dina_belova has joined #openstack-infra21:39
clarkbmtreinish: ya, I am all for switching now >_> to help avoid new problems sneaking in21:40
clarkbjeblair: fungi I am able to get logs through the htmlifier on logs.o.o now21:41
fungiclarkb: great!21:41
fungiclarkb: also i replied to your question on https://review.openstack.org/4080021:42
mtreinishclarkb: well I'd like that too and just the fight races when they come up. But, my only concern with doing that is doesn't really have an established track record of working for that long. My fix for the last big race only got merged yesterday.21:43
*** dina_belova has quit IRC21:44
*** cthulhup has quit IRC21:44
clarkbmtreinish: I understand. I am just slightly more adventurous considering I don't have a lot of buckets in H3 >_>21:44
mtreinishclarkb: well feel free to share that opinion on the ML thread, maybe the consensus is that people don't care about the proven stability weighed against raw speed and we can push it through now :)21:45
clarkbfungi: responded :)21:45
clarkbmtreinish: I have sent mail, I guess it hasn't gone through yet21:46
mtreinishclarkb: no, I just fail at looking at new mail notification21:46
*** sgviking has joined #openstack-infra21:47
*** dina_belova has joined #openstack-infra21:49
openstackgerritClark Boylan proposed a change to openstack-infra/config: Switch etherpad_lite backups to mysql_backup.  https://review.openstack.org/4179121:50
fungiclarkb: on https://review.openstack.org/40800 i can switch it back to two print statements but it's not going to alter the output (more specific example added inline)21:51
clarkboh because of the explicit newlines?21:52
clarkbwow I am derping hard.21:52
clarkbfungi: updated my review. thank you for your patience21:53
fungiclarkb: yeah, i just figured while i was collapsing repeated print statements i'd go ahead and clean that up21:53
jeblairclarkb, fungi: maybe the new zuul functions are premature.  maybe nodepool should be a jenkins zmq listener...21:53
jeblairat least, for now...21:53
jeblairclarkb, fungi: mostly because if we restart zuul while jobs are running (which we do), nodepool will miss notifications21:53
*** dina_belova has quit IRC21:54
jeblair(it'll eventually clean them up, but those nodes would stick around for quite a while)21:54
jeblair(if it's a zmq listener, it could miss notifications when _nodepool_ isn't running, but hopefully that won't be very often or for very long)21:56
clarkbjeblair: we can actually work around that second issue as well21:58
clarkbjeblair: zmq pub sub allows us to queue jobs in memory on the sender side if the receiver goes away21:58
clarkbjeblair: however I have avoided doing that as it has memory consideratinos, but totally doable if we decide that is a need21:58
*** rnirmal has quit IRC21:59
jeblairclarkb: yeah, i also don't know where we'd want to put this for non-jenkins workers; perhaps nodepool should keep a zmq listener for jenkins and be a gearman worker for other nodes...21:59
jeblairclarkb: (perhaps even the gearman plugin for jenkins should get the ability to put these jobs on the gearman queue)22:00
jeblairclarkb: but the more i think about it, the more i think that right now we should stick with zmq because it's the best match for current operations, and we should still have the flexibility to change in the future.22:00
jeblair(ie, nodepool can have as many of these inputs as we want)22:00
*** beagles has quit IRC22:01
fungiand what would be the longer-term fit in a post-jenkins world?22:01
jeblairfungi: if we have our own workers, it makes a little more sense for them to put jobs on the gearman queue themselves, so they could trigger these on their own.22:02
*** datsun180b has quit IRC22:03
*** burt has quit IRC22:04
*** dkliban has quit IRC22:04
*** SergeyLu_ has joined #openstack-infra22:04
fungigot it. and at that point the vestigial zmq support just gets phased out/dropped entirely22:05
jeblairyeah.  it's _a_ plan.  :)22:05
openstackgerritMarton Kiss proposed a change to openstack-infra/config: Add groups to gerrit  https://review.openstack.org/4164322:05
*** SergeyLukjanov has quit IRC22:06
*** jerryz has quit IRC22:06
*** beagles has joined #openstack-infra22:06
*** mrmartin has quit IRC22:09
*** SergeyLu_ has quit IRC22:09
*** pentameter has quit IRC22:11
*** sarob has quit IRC22:12
*** sarob has joined #openstack-infra22:12
*** sarob has quit IRC22:17
fungiapproved ^ that, so once it merges i'll keep an eye on git.o.o to see i it populates accordingly22:20
openstackgerritA change was merged to openstack-infra/config: Add groups to gerrit  https://review.openstack.org/4164322:21
* fungi rubs hands together22:22
* pleia2 opens up dashboard22:23
*** mrodden has quit IRC22:23
*** _TheDodd_ has quit IRC22:28
*** changbl has quit IRC22:31
*** dripton has quit IRC22:31
*** mriedem has quit IRC22:32
openstackgerritJames E. Blair proposed a change to openstack-infra/nodepool: WIP: Initial commit  https://review.openstack.org/4180522:33
*** ryanpetrello has joined #openstack-infra22:34
*** dripton has joined #openstack-infra22:35
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Gate nodepool on pep8.  https://review.openstack.org/4180622:35
jeblairclarkb, fungi: ^ aprv 41806 at will22:36
*** beagles has quit IRC22:36
jeblairclarkb, fungi, mordred: now might be a good time to add nodepool to your watched projects list22:36
clarkbjeblair: will do22:37
*** sarob has joined #openstack-infra22:38
pleia2fungi: https://git.openstack.org/cgit/openstack/api-site/22:38
pleia2empty \o/22:38
fungipleia2: yep, and this time the permissions check out22:40
pleia2now hopefully gerrit can properly replicate to it22:40
fungipleia2: i forced full replication via the gerrit ssh api, so let's see if it fills them in22:40
openstackgerritA change was merged to openstack-infra/config: Gate nodepool on pep8.  https://review.openstack.org/4180622:41
*** cthulhup has joined #openstack-infra22:41
jeblair3bdfffc9                       push cgit@git.openstack.org:/var/lib/git/opensta22:41
jeblairthat's so helpful.22:41
*** jhesketh has joined #openstack-infra22:41
fungipleia2: you can watch the count of replication jobs queued for it with 'ssh -p 29418 review.openstack.org gerrit show-queue | grep -c cgit'22:41
fungijeblair: yeah, love the truncation there22:41
pleia2fungi: neat, thanks22:41
*** beagles has joined #openstack-infra22:42
jeblairfungi: it's an openstack project.22:42
fungii guess we can say for certain 3bdfffc9 does not belong to a stackforge project22:42
jeblairit probably belongs to nova22:44
pleia2I sure hope so, it's taking a while22:44
fungistatistically speaking, it probably does belong to nova22:46
*** ladquin_afk has quit IRC22:48
*** cthulhup has quit IRC22:49
*** dina_belova has joined #openstack-infra22:50
clarkbdoes this mean cgit is mostly working now?22:50
jeblairi don't think so...22:53
jeblairthere are still 170 cgit jobs22:53
jeblairperhaps none of them have completed22:54
jeblairthey are the only ones left in the nqueue22:54
*** dina_belova has quit IRC22:55
samalbadoes anyone have the answer of that: https://answers.launchpad.net/devstack/+question/233734 ?22:55
samalbaI have the same issue22:55
samalbaand not super familiar with cinder22:55
*** ladquin_afk has joined #openstack-infra23:00
*** ladquin_afk has joined #openstack-infra23:00
*** svarnau has quit IRC23:05
*** ryanpetrello has quit IRC23:07
fungisamalba: you might have more luck asking in #openstack or #openstack-dev23:08
fungisamalba: the -infra channel is the people running the community infrastructure systems23:08
samalbaI thought it was a devstack issue (like config problem), that's why I asked here23:09
samalbabut I'll try on -dev23:09
pleia2so, looks like replication is still unhappy23:09
pleia2sorted permissions, ssh keys should be ok23:10
fungipleia2: yes, tcpdump on git.o.o does show an established ssh connection from review.o.o, just very, very minimal traffic23:10
jog0mordred: I assume you saw this article http://arstechnica.com/information-technology/2013/08/why-ubuntus-creator-still-invests-his-fortune-in-an-unprofitable-company/23:10
* pleia2 headscratch23:10
fungipleia2: i also checked /var/log/secure and it didn't show any authentication issues23:13
*** sdague has quit IRC23:13
openstackgerritA change was merged to openstack/requirements: Add note to test-requirements saying its local only  https://review.openstack.org/4152423:15
pleia2browsing replication docs to see if there is something I'm missing for ssh23:16
fungipleia2: the gerrit log is showing constant ssh exceptions for "connection reset by peer" (typically that indicates a tcp/rst on an open socket)23:16
fungithough i'm not seeing rst set on any of the packets going back to the gerrit server from git.o.o23:17
jeblairfungi: i believe that's normal gerrit behavior, indicating that an ssh connection is finished.23:18
*** blamar has quit IRC23:18
fungijeblair: that's downright sad23:18
jeblairfungi: it's sad.  it makes the error log very useless.23:18
pleia2doesn't seem like there are any special things in the configuration we need, just the url line should be enough23:19
funginetstat -nt shows four ssh connections established inbound from review.o.o to git.o.o23:20
jeblairfungi: but none of them are running git, correct?23:21
fungijeblair: yeah, no git processes23:22
*** ryanpetrello has joined #openstack-infra23:26
jeblairfungi: how about i kill one of those and we see if gerrit logs anything?23:28
fungijeblair: yeah, i wonder if these are still there from a previous failing replication attempt and all the in progress ones need killing23:29
*** jhesketh has quit IRC23:29
*** jhesketh has joined #openstack-infra23:29
jeblairoh, i was thinking of killing the ssh process on git.o.o, but yeah, actually, let's try removing them from the gerrit queue first23:29
fungiyeah, i thought you meant the gerrit api's kill command23:30
jeblairkilled 3bdfffc923:31
fungioh, in fact i should have paid attention to the timestamps on the ssh processes for the cgit user23:31
fungithey were all from 19:32 utc23:31
jeblairyse they are old aren't they23:31
jeblairwell, they are 1 hour old23:32
fungii make that 4 hours old23:32
*** ryanpetrello has quit IRC23:32
jeblairi am bad at math23:32
fungiit's okay ;)23:32
* fungi is awful at math23:33
fungithe easier the math, the more likely i am to get it wrong23:33
jeblairso i killed the other 3 that were running; the others are still waiting; strangely...23:33
jeblairnothing has changed on git.o.o23:33
fungii'll kill those ssh sessions then23:33
jeblairsounds good23:33
fungiand no new ones appearing (yet)23:34
jeblairok, next how about i kill all the waiting gerrit processes23:34
clarkbAlex_Gaynor: I would check with zul if cffi is an actual problem. It might be cloud archived if you need it23:34
fungijeblair: please do. at least the ones for cgit@git anyway23:34
*** ^d has quit IRC23:35
jeblairfungi: done23:36
fungithe start times on the ssh processes were from when i tried to trigger a full replication run during the infra meeting, back when repo permissions were wrong, so maybe it was just confused23:37
jeblairfungi: why don't you trigger a single-project replication now ?23:37
fungiand yeah, no new ssh sessions appearing yet. i'll try to replicate something small like gear23:37
fungi3c1b62a7 23:38:08.922          push git@github.com:openstack-infra/gear.git23:38
jeblairi wonder if it's permanently stuck wrt git.o.o23:38
jeblairfungi: how about we restart gerrit23:39
fungicould be, though that replication job vanished from the queue23:39
*** mriedem has joined #openstack-infra23:39
fungibut yeah, /var/lib/git/openstack-infra/gear.git/refs/heads/ is still empty23:39
jeblairfungi: yeah, but that was for github; i never saw one for git.o.o23:39
fungiright, i had tunnel vision there23:40
fungirestarting gerrit23:40
fungiit's up again now23:41
fungiand it has "Replicate All Projects" in its queue at the moment23:41
*** rcleere has quit IRC23:42
pleia2it's going!23:42
pleia2https://git.openstack.org/cgit/openstack-dev/openstack-qa/ :D23:42
fungigit processes!23:42
fungii see them in ps23:42
fungiwe (i?) clearly confused gerrit earlier23:42
jeblair[2013-08-13 23:42:31,240] ERROR com.google.gerrit.server.git.PushReplication : Cannot replicate to git@github.com:openstack-infra/meetbot.git23:42
fungithat's... exciting23:43
jeblair(there are some other ones that are excpected (melange, pypi-mirror), but that one i think is an actual problem)23:43
* fungi looks at its settings on github23:43
pleia2heading out to the train station to get to a meetup in mt view, will check in on the train23:44
jeblairpleia2: cgit seems to be working :)23:44
fungipleia2: great work!23:44
pleia2yes, I'm very pleased!23:44
pleia2and looks like it's all done syncing23:45
clarkbwe should s/https/git/ in the gerrit git prep script soonish (I think we need zuul to check that replication to git.o.o has happened first)23:45
pleia2good to know that it's quick23:45
jeblairexcept for nova23:45
pleia2git clone from git:// works too, so hooray for git-daemon23:45
jeblairclarkb: well, erm...23:46
fungii don't see anything out of the ordinary with openstack-infra/meetbot.git on github23:46
pleia2hah, yeah, nova23:46
jeblairclarkb: it should be able to fetch most things from git.o.o, then fetch changes from zuul23:46
clarkbjeblair: I have given my email address to someone that can give me the email address of the people to ask about whether or not it is safe to use that hpcloud account23:46
clarkbjeblair: that works too23:46
funginova finished finally, at least23:47
jeblairclarkb: it shouldn't need to check that replication has happened, because zuul has refs for critical things23:47
clarkbjeblair: gotcha, we just need to get enough of a seed there that we don't spend a bunch of time with https23:47
jeblairclarkb: we should also, very very soon, change our vcsrepo refs in puppet  :)23:50
clarkbpleia2: do you want to make the first pass at updating infra related things?23:50
*** dina_belova has joined #openstack-infra23:50
fungithe tip of master on github and gerrit for meetbot match shas, so i don't think there's anything wrong with the repo there23:51
jeblairfungi: weird23:51
fungiand the project config for it on github matches the way we have our other projects set from what i can tell23:51
clarkbmaybe some other underlying problem? maybe git fsck it?23:52
fungimmm, maybe. fscking23:53
*** dina_belova has quit IRC23:53
fungifscks clean on my checkout. is it safe to git fsck the copy in ~gerrit on review.o.o?23:53
fungier, in ~gerrit223:54
jeblairfungi: yes -- however, there is a nightly repack already23:54
fungii'm going to suspect a transient, passing issue on github23:54
jeblairso it might detect or correct errors.23:54
jeblairfungi: that seems reasonable for now.23:54
clarkbfungi: would not be surprised if that was the issue23:55
fungianyway, fsck runs clean on gerrit's copy as well23:55
fungiso really, no sign of anything broken, and this would not be the first time github randomly rejected something we tried which normally works23:56
fungiif the repo for it starts getting behind on github, then i'll worry23:56
*** ryanpetrello has joined #openstack-infra23:58
clarkbjeblair: why does the graphite graph that you gave mtreinish earlier break if I s/check/gate/ ?23:59
jeblairclarkb: no data in gate yet?23:59
jeblairdid we merge that change?23:59
fungii think i approved it23:59
* fungi looks23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!