Monday, 2013-10-14

fungijeblair: sdague: first i forced replication, then noticed that several of the git servers had no head at all (similar to what we've seen sometimes on zuul) so... now what did i do?00:03
* fungi checks his shell history00:03
sdaguefungi: so we think some of the jenkins slaves had stale bad trees00:03
sdaguethat were unrecoverable00:03
fungiahh, that seems likely00:04
sdagueso hopefully it was really just that one slave00:04
fungibecause yeah, what i did after forcing replication was noticed not missing heads but entire missing repos because puppet was failing to complete/update jeepyb because setuptools/pip was broken due to trying to upgrade itself, so i embarked on fixing that on all the git servers00:05
fungiwhat is what spurred mordred on to the new changes in review/post on the -infra ml which i still need to respond to00:06
fungier, which is what00:07
fungianyway, since reverifying sdague's failing change worked, i assumed i'd fixed whatever the issue was. guess not :/00:07
*** fifieldt_ has joined #openstack-infra00:24
*** sarob has joined #openstack-infra00:25
*** senk has joined #openstack-infra00:26
*** sarob has quit IRC00:29
*** david_h has joined #openstack-infra00:33
*** mriedem has joined #openstack-infra00:36
*** david_h has quit IRC00:52
*** dcramer_ has joined #openstack-infra00:59
*** sarob has joined #openstack-infra01:09
*** yaguang has joined #openstack-infra01:12
*** sarob has quit IRC01:14
*** che-arne has quit IRC01:22
openstackgerritQiu Yu proposed a change to openstack-infra/pypi-mirror: Print help message and exit if no config file by default
*** guohliu has joined #openstack-infra01:51
*** jhesketh has quit IRC01:53
*** jhesketh has joined #openstack-infra01:54
*** sileht has quit IRC02:05
*** sdake has joined #openstack-infra02:08
*** sileht has joined #openstack-infra02:10
*** dims has joined #openstack-infra02:25
*** markmcclain has joined #openstack-infra02:34
*** rcleere has joined #openstack-infra02:35
*** rcleere has quit IRC02:37
*** dhouck_ has joined #openstack-infra02:49
*** jtzl has quit IRC02:57
*** jtzl has joined #openstack-infra02:57
*** david_h has joined #openstack-infra03:00
*** dims has quit IRC03:03
*** harlowja_at_home has joined #openstack-infra03:05
harlowja_at_homedoes anyone know if the etherpad switchover worked out? one of my etherpads seems totally busted.03:05
harlowja_at_home"Error: Attempt to load undefined module. in (line 1)"03:05
clarkbharlowja_at_home: yes, our testing indicated it was fine. You may need to hard refresh to pull in new js03:06
clarkbharlowja_at_home: do you have a link to the etherpad?03:06
harlowja_at_homelet me try on a different browser that shouldn't have any cache03:06
harlowja_at_homeah, seems like u are right, maybe just have to clear my cache or something03:07
clarkbit loads fine for me. try ctrl + f5 for chromium03:07
clarkbharlowja_at_home: yes, this is a known issue with the upgrade and was noted in the announcement03:08
harlowja_at_homethx clarkb03:08
harlowja_at_homeall better, woot03:10
*** D30 has joined #openstack-infra03:10
*** harlowja_at_home has quit IRC03:11
*** senk has quit IRC03:14
*** markmcclain has quit IRC03:16
*** markmcclain has joined #openstack-infra03:17
*** dhouck_ has quit IRC03:21
*** xchu has joined #openstack-infra03:23
*** david_h has quit IRC03:26
*** nati_ueno has joined #openstack-infra03:35
*** mriedem has quit IRC03:40
*** kong has left #openstack-infra03:50
openstackgerritZhongyue Luo proposed a change to openstack/requirements: Adds byteformat
fifieldt_clarkb, are you able to take another look at ? I updated one more path04:13
clarkbfifieldt_: done (found one extra set of things that need updating04:15
fifieldt_actually, that's not true :)04:15
fifieldt_hang on04:15
fifieldt_half true :D04:15
fifieldt_the PDF name still follows the old format04:16
fifieldt_but dammit, there's still one04:16
clarkbimo you should fix the pdf name04:16
clarkbits confusing to have so many idfferently named things04:16
fifieldt_oh, wtf - I forgot to commit the previous patch that fixced 4304:17
fifieldt_clarkb, yes, that's in plan - but it needs to be done to everything04:17
openstackgerritTom Fifield proposed a change to openstack-infra/config: Fix Doc Location for Transifex
fifieldt_so much cleanup still to do clarkb, it'04:18
fifieldt_driving me nuts :)04:18
clarkbfifieldt_: is line 63 wrong?04:18
fifieldt_that one wasn't04:21
openstackgerritTom Fifield proposed a change to openstack-infra/config: Fix Doc Location for Transifex
fifieldt_but the other file was04:21
fifieldt_it's so damn confusion - the pom.xml file stuff04:21
fifieldt_there are about 4 different variables in there that do the same thing04:21
fifieldt_with slightly different values04:21
* fifieldt_ rage04:21
fifieldt_apologies for the shoddy patching, clarkb04:21
clarkbnp +2 from me04:22
clarkbas someone that doesn't write docs professionally I have no idea how docbook is a tool of choice04:22
fifieldt_docbook itself isn't a problem04:22
fifieldt_it's maven04:22
clarkbI mean markdown, rst, tex, html, etc would all be simpler and have better renderers04:23
clarkbfifieldt_: right, but docbook forces you to use maven to render/compile stuff04:23
*** xchu has quit IRC04:23
fifieldt_for me, docbook has two killer features: xi:include, and conditional formatting (os="ubuntu")04:23
fifieldt_other than that, don't care04:23
fifieldt_it doesn't get in my way04:24
fifieldt_personally I don't find markdown, rst, tex or html any simpler :)04:24
fifieldt_but maybe I'm crazy04:24
clarkbreally? I can edit all of those in vim and not feel insane :)04:24
* fifieldt_ uses vim to edit docbook04:24
clarkbwhich is what I am basing my "simpler" judgement on. editing docbook in vim makes me feel crazy04:25
fifieldt_oh, strange - I've never had an issue :)04:25
fifieldt_probably again proving my insanity04:25
fifieldt_I think one of the problems people had was that openstack was their first exposure to docbook, and the docs had sooooo much cruft and crappy layout04:26
clarkbthat could be part of it. bitrot is hard to deal with anywhere04:27
* fifieldt_ is currently screwing over his havana test vm every which way trying to write the network install docs :)04:29
fifieldt_things are so much simpler in puppet ;)04:29
*** xchu has joined #openstack-infra04:36
*** nicedice has quit IRC04:40
*** yaguang has quit IRC04:41
*** jcoufal has joined #openstack-infra04:43
*** yaguang has joined #openstack-infra04:46
*** Canaimero-e64b8 has joined #openstack-infra04:46
Canaimero-e64b8 mm,04:46
*** Canaimero-e64b8 has quit IRC04:47
*** dmakogon_ has joined #openstack-infra05:13
D30hi all, good day.. may i know if this is the right room to ask regarding openstack related issue?05:15
*** __afazekas is now known as afazekas05:15
fifieldt_D30, this room is for talking about the infrastructure powering the openstack project05:16
fifieldt_if you have an issue with running openstack05:16
fifieldt_you should try #openstack or #openstack-10105:16
fifieldt_or http://ask.openstack.org05:16
D30ahh okay fifieldt_  thank you for pointing that out..05:16
*** dmakogon_ has quit IRC05:25
*** dafter has joined #openstack-infra05:34
*** dafter has quit IRC05:40
*** dafter has joined #openstack-infra05:41
*** dafter has quit IRC05:45
*** yaguang has quit IRC05:48
*** SergeyLukjanov has joined #openstack-infra05:48
*** jecarey has quit IRC05:56
*** yaguang has joined #openstack-infra06:03
*** michchap has quit IRC06:23
*** oubiwann is now known as oubiwann-lambda06:27
*** SergeyLukjanov is now known as _SergeyLukjanov06:29
*** SergeyLukjanov has joined #openstack-infra06:30
*** beagles has joined #openstack-infra06:33
*** dkehn has quit IRC06:41
*** dkehn has joined #openstack-infra06:42
*** odyssey4me has joined #openstack-infra06:47
*** DinaBelova has joined #openstack-infra07:00
*** dizquierdo has joined #openstack-infra07:07
*** pblaho has joined #openstack-infra07:26
*** DinaBelova has quit IRC07:30
bauzasmorning all07:30
bauzasany chance someone could help me on a Jenkins issue ?07:30
bauzasmy own tox -r -epy27 is fine07:31
bauzasbut Jenkins is building a venv having both oslo-config and oslo.config07:31
bauzasas a result, it doesn't work as oslo-config is too old07:31
bauzasI haven't asked for oslo-config, only asking oslo.config>1.2.007:32
bauzasany idea why Jenkins is pulling oslo-config ?07:32
bauzasand how can I fix it ?07:32
*** markmc has joined #openstack-infra07:33
*** DinaBelova has joined #openstack-infra07:34
*** afazekas is now known as __afazekas07:35
*** osanchez has joined #openstack-infra07:41
*** dafter has joined #openstack-infra07:46
*** bauzas_ has joined #openstack-infra07:52
*** thomasbiege has joined #openstack-infra07:53
*** thomasbiege has quit IRC07:54
*** afazekas has joined #openstack-infra07:57
*** nati_ueno has quit IRC08:02
*** nati_ueno has joined #openstack-infra08:02
*** bauzas has quit IRC08:04
*** yaguang has quit IRC08:05
*** bauzas_ has quit IRC08:05
*** nati_ueno has quit IRC08:07
*** hashar has joined #openstack-infra08:14
*** yassine has joined #openstack-infra08:14
*** yaguang has joined #openstack-infra08:17
*** dizquierdo has quit IRC08:19
*** derekh has joined #openstack-infra08:20
*** SergeyLukjanov has quit IRC08:20
*** SergeyLukjanov has joined #openstack-infra08:21
*** jpich has joined #openstack-infra08:22
*** mihgen has quit IRC08:24
*** aspiers has joined #openstack-infra08:25
*** SergeyLukjanov has quit IRC08:25
*** yaguang has quit IRC08:28
*** rakhmerov has joined #openstack-infra08:33
*** matsuhashi has joined #openstack-infra08:43
*** qba73 has joined #openstack-infra08:46
*** yaguang has joined #openstack-infra08:46
*** qba73 has left #openstack-infra08:47
*** fbo_away is now known as fbo08:48
*** amotoki has joined #openstack-infra08:50
*** amotoki has quit IRC08:51
*** amotoki has joined #openstack-infra08:51
*** mihgen_ has joined #openstack-infra08:58
*** hashar has quit IRC09:02
*** dizquierdo has joined #openstack-infra09:04
openstackgerritPetr Blaho proposed a change to openstack-infra/config: Remove python33 jobs for python-tuskarclient
*** bauzas has joined #openstack-infra09:12
*** hashar has joined #openstack-infra09:18
*** hashar_ has joined #openstack-infra09:23
*** hashar has quit IRC09:24
*** hashar_ is now known as hashar09:24
rakhmerovHi, we've sent a change request to add a new project (Mistral): . Now we've got a question about acl. Could someone find a couple of minutes to discuss that more detailed? Thanks in advance09:25
*** salv-orlando has quit IRC09:39
*** salv-orlando has joined #openstack-infra09:41
*** DinaBelova has quit IRC09:48
*** jtzl has quit IRC09:53
*** wannatest has quit IRC09:53
*** michchap has joined #openstack-infra09:54
*** xchu has quit IRC09:56
*** salv-orlando has quit IRC09:56
*** salv-orlando has joined #openstack-infra10:05
*** pcm_ has joined #openstack-infra10:06
*** pcm_ has joined #openstack-infra10:07
*** erfanian has quit IRC10:11
*** erfanian has joined #openstack-infra10:11
*** DinaBelova has joined #openstack-infra10:15
openstackgerritA change was merged to openstack-infra/jenkins-job-builder: Add optional option to copyartifact plugin
*** branen has quit IRC10:18
*** salv-orlando has quit IRC10:19
*** openstackstatus has quit IRC10:20
*** salv-orlando has joined #openstack-infra10:21
ekarlso < how can I re-trigger a build of it ?10:27
*** salv-orlando_ has joined #openstack-infra10:28
*** salv-orlando has quit IRC10:29
*** salv-orlando_ is now known as salv-orlando10:29
*** salv-orlando_ has joined #openstack-infra10:32
*** salv-orlando has quit IRC10:34
*** salv-orlando_ is now known as salv-orlando10:34
*** DinaBelova has quit IRC10:39
*** boris-42 has quit IRC10:42
*** guohliu has quit IRC10:42
*** michchap has quit IRC10:45
*** michchap has joined #openstack-infra10:48
*** mkerrin has joined #openstack-infra10:51
*** salv-orlando has quit IRC10:53
*** salv-orlando has joined #openstack-infra10:56
*** qba73_ has joined #openstack-infra11:00
openstackgerritSean Dague proposed a change to openstack-infra/config: switch over to os-loganalyze on logs.o.o
*** dafter has quit IRC11:04
*** rickerc has quit IRC11:05
*** michchap has quit IRC11:06
*** yaguang has quit IRC11:07
*** dims has joined #openstack-infra11:08
openstackgerritAntoine Musso proposed a change to openstack-infra/jenkins-job-builder: missing parameter error are no more meaningfull
*** hashar has quit IRC11:09
*** D30 has quit IRC11:09
*** hashar has joined #openstack-infra11:12
*** jhesketh__ has quit IRC11:18
*** fifieldt_ has quit IRC11:21
*** odyssey4me has quit IRC11:26
*** jcoufal has quit IRC11:35
BobBallAwaylifeless / fungi: Euither of you (or anyone) know if it's possible to boot in HP cloud with virtio drivers blacklisted / missing?  A quick test failed to bring up networking - but I could easily be missing something...11:38
*** lcestari has joined #openstack-infra11:39
*** BobBallAway is now known as BobBall11:41
openstackgerritafazekas proposed a change to openstack-infra/elastic-recheck: Add query for 1239637
*** rakhmerov has quit IRC11:49
*** adalbas has joined #openstack-infra11:49
*** rakhmerov has joined #openstack-infra11:53
*** hashar has quit IRC11:59
*** hashar has joined #openstack-infra12:03
*** dafter has joined #openstack-infra12:06
*** bauzas has quit IRC12:07
*** boris-42 has joined #openstack-infra12:12
*** dafter has quit IRC12:12
*** beagles has quit IRC12:12
*** bauzas has joined #openstack-infra12:12
*** beagles has joined #openstack-infra12:14
*** tvb|afk has joined #openstack-infra12:14
*** tvb|afk has quit IRC12:14
*** tvb|afk has joined #openstack-infra12:14
*** matsuhashi has quit IRC12:18
*** matsuhashi has joined #openstack-infra12:18
*** matsuhashi has quit IRC12:22
*** rakhmerov has left #openstack-infra12:24
*** bauzas has quit IRC12:24
*** dprince has joined #openstack-infra12:24
*** ArxCruz has joined #openstack-infra12:27
*** jcoufal has joined #openstack-infra12:28
*** weshay has joined #openstack-infra12:36
fungiBobBall: can't say i've ever tried. how did you go about it (lkm blacklist file, bootloader configuration, renaming the module on disk)?12:38
BobBall/etc/modules.d blacklist12:39
BobBallBlacklist seemed to work in that it didn't load the modules, but networking didn't come up so it couldn't access cloudinit12:40
BobBallI thought it should have been possible from my knowledge of KVM (and that we've also seen VMs running without virtio drivers outside of HP cloud)12:40
sdagueBobBall: I'd expect the way the device model is setup by the libvirt driver, your only option is virtio12:40
*** jaypipes has joined #openstack-infra12:40
sdagueBobBall: but remember the nova libvirt driver is pretty explicit on what it sets up12:41
*** tvb|afk has quit IRC12:41
BobBallI've not played with the nova libvirt driver at all... admittedly this was outside of openstack we had it running12:41
sdaguethere was some metadata tags about device models added to glance images at some point, but I don't know of HP cloud is up to that level12:41
jcoufalhey everybody, is here somebody who was dealing with customization of askbot for
BobBallAnother angle blocked...12:41
sdagueBobBall: is the only possible way around it that I know, but again, I have no idea if HP is on code level that supports that12:43
sdagueit came in late grizzly or early havana IIRC12:43
BobBallohhh thanks sdague12:43
sdaguethat's going to require you getting metadata onto your glance image12:44
BobBallI imagine that's possible... :)12:44
*** dafter has joined #openstack-infra12:44
*** dafter has quit IRC12:44
*** dafter has joined #openstack-infra12:44
*** krtaylor has joined #openstack-infra12:45
BobBallyay it is12:45
openstackgerritA change was merged to openstack-infra/config: Fix Doc Location for Transifex
BobBallprobably :P12:45
sdaguewell, it's at least something to explore12:45
BobBallvery much so, thanks.12:45
*** julim has joined #openstack-infra12:46
*** dims has quit IRC12:48
*** johnthetubaguy has joined #openstack-infra12:49
*** dims has joined #openstack-infra12:51
*** jcoufal has quit IRC12:51
*** dcramer_ has quit IRC12:55
fungiekarlso: looking at 49850 i assume you got an answer to your question earlier (recheck no bug or recheck bug somenumber)12:57
mordredfungi: how's your nodepool grokking?12:59
fungimordred: increasing okay in absence of documentation i guess (rtfs!) but still not as solid as i'd like, why?13:00
mordredfungi: so, I spun up a nodepool on my laptop so that I could test the tripleo cloud without just continuing to retry landing config changes on infra13:00
mordredfungi: here is a snippet of local config:
fungiyeah, i caught some of that yesterday. didn't work completely i guess?13:01
mordredDEBUG:nodepool.NodePool:No current image for tripleo-test-cloud on tripleo-precise13:01
mordredI get that13:01
mordredwhich, may mean I have something configured backwards, or that the message is backwards13:01
mordredin either case - nodepool image-list shows that I do have a tripleo-precise image13:01
fungiare you getting that when it's trying to launch a fresh vm to configure as a template for the pool?13:02
* fungi needs to trace where that error can originate13:02
mordredINFO:nodepool.NodePool:Need to launch 2 tripleo-precise nodes for fake-jenkins on tripleo-test-cloud13:02
mordredso, it wants to launch 2 vms13:02
mordredbut thinks it doesn't have any available images, best I can tell13:02
ttxmordred: what share of the US is expected to observe Columbus day ?13:02
*** thomasm has joined #openstack-infra13:02
mordredttx: I do not expect many people to have it off13:02
fungittx: banks13:02
mordredit's not a real holiday13:02
*** afazekas has quit IRC13:03
ttxmordred: ah, great13:03
fungibankers love to take random holidays13:03
ttxfungi: "we do because we can and f* you"13:03
mordredfungi: oh - I'm sorry - I have an image in 'building'13:03
fungittx: absolutely. it's like you've been to a bank before13:03
mordredfungi: lemme poke a little more13:03
fungimordred: okay. i was thinking that might be benign since it's only debug level13:04
ttxfungi: bankers in france also happen to have the best holiday packages of all.13:04
fungiamazing how that works13:04
mordredO M G13:05
*** senk has joined #openstack-infra13:05
mordredI really really really hate things when they don't respond to ctrl-c all the way13:05
*** freyes has quit IRC13:05
mordredbut I'm really impatient too13:05
sdaguefungi: when you get a chance -,n,z the first one should be a no brainer, the second one will probably need baby sitting when landing.13:06
ekarlsocan I bother anyone to approve and
fungisdague: awesome! i'll have a look13:06
sdaguegiven you and clarkb are the folks I've most worked with on the log servers, I figured I'd hit you up13:06
*** ryanpetrello has joined #openstack-infra13:08
*** mestery_ has joined #openstack-infra13:09
ekarlsoHmm, what's up with it failing to download taskflow from here?
*** rustlebee is now known as russellb13:10
*** pblaho has quit IRC13:11
*** osanchez has quit IRC13:12
*** gaelL has quit IRC13:12
*** OlivierSanchez has joined #openstack-infra13:12
*** mestery has quit IRC13:12
*** gaelL has joined #openstack-infra13:13
fungiekarlso: where is it failing to download taskflow there? i don't see it13:15
*** matty_dubs|gone is now known as matty_dubs13:17
* olaph does not respond to ctrl-c all the way13:17
openstackgerritMehdi Abaakouk proposed a change to openstack-infra/jenkins-job-builder: Allow use template variables in dictionnary key
*** mriedem has joined #openstack-infra13:20
*** jpeeler has quit IRC13:21
*** clayg has quit IRC13:21
*** jpeeler has joined #openstack-infra13:21
*** clayg has joined #openstack-infra13:21
*** rakhmerov has joined #openstack-infra13:23
* mordred realy really hates olaph13:23
Shrewsolaph only responds to beatings about his head13:25
fungiand beer13:25
*** afazekas has joined #openstack-infra13:25
fungiShrews: olaph: bier, ja?13:25
olaphja wohl!13:26
mordredfungi, Shrews, olaph: we're getting bier next week, ja?13:26
Shrewsewww, beer, gross13:26
fungithough the evening after the openstack meetup next monday isn't an option for me. visitors in town early in the week13:26
Shrewsmordred: you staying downtown during ATO?13:27
Shrewsolaph, fungi: you guys going to ATO?13:27
fungiwhat's an ato?13:27
Shrewsall things open conference13:28
olaphstill gotta ask my boss...13:28
fungihuh. how do i miss these announcements?13:28
*** bauzas has joined #openstack-infra13:28
mordredShrews: I'm actually staying in Durham13:28
mordredShrews: (well, closer to RTP, but whatev)13:28
mordredfungi: I get in Wed evening, will be sticking around through the weekend because my 20 year high school reunion is saturday13:29
* fungi is reading the ato manifesto now13:29
mordredolaph: the hotel here in buenos aires has the best power strip ever in the desk drawer13:29
fungimordred: i'm absolutely certain we can work out a time late next week or that weekend (i'll be free pretty much the whole time)13:29
mordred(people should be able to view that without a facebook account)13:30
mordredhandles agentina, us and eu power plugs13:30
mordredI'm thinking about getting a couple and starting to bring them to conferences13:31
fungioh, too nifty13:31
mordredit won't help folks from the uk - but DUDE, those plugs are WAY too big13:31
yolandahi, i'm trying to setup jenkins-job-builder, but i have an issue with authentication. Is jenkins-job-builder prepared to work with SSO?13:31
yolandamy admin user is logged through SSO, so i'm having problems setting the credentials13:32
mordredyolanda: so, there is a trick13:32
mordredyolanda: fungi might remember the steps better than me - but if you log in to jenkins via sso as the user you want to be the api user13:32
mordredyolanda: then in the users.xml file on the filesystem (I thnk) there is an api key13:33
mordredit's a terrible interface13:33
fungiahh, yeah. i haven't had to generate one in a while but that matches my recollection13:33
yolandamordred, but .. what about if i need to configure the jobs before admin ever logs?13:33
mordredyolanda: I do not believe we have solved that problem yet (thank you for working on solving our initial setup problems though!)13:34
mordredwe just do that part of the jenkins install by hand13:34
yolandamordred, bad news... :(13:34
fungiyolanda: this is probably straying into what might better be a "what's possible?" conversation in #jenkins13:34
* mordred has REALLY been meaning to do more work on scripting the initial creation of the jenkins13:35
fungibecause you're likely going to have bootstrapping problems if you want to automatically/programatically interact as an admin with a jenkins which has just been set up and nobody has ever logged into13:35
mordredit honestly should not be terrible to do13:35
mordredfungi: I _think_ everything could completely be done by generating a config.xml and then starting jenkins13:35
mordredthere is completely nothing that's not configured through that file13:35
fungiprobably, similar to bootstrapping gerrit access, you'll need to inject some initial credentials, yeah13:36
yolandamaybe generating a fake user xml file...13:36
mordredindeed - but I believe the jenkins bootstrap might be saner13:36
fungiyeah, no db involved13:36
yolandaif we have the name of the user, we could create an entry in the users dir, makes sense?13:36
*** lascii is now known as alaski13:36
mordredyolanda: if you look in users/$adminusername/config.xml - you'll see an apiToken entry13:37
fungimaking a temporary password-authenticated user might be simplest to bootstrap that, but you'd still need to be able to accurately recreate the password hashing algorithm jenkins uses13:37
mordredthat's the thing you need for jjb13:37
fungioh, or if the api token is in there in the clear, that could work13:37
mordredyup. that's where it comes from13:37
fungiis there nothing else magical about the api token? is it just a random key blob?13:38
* mordred is not sure - guessing random key blob13:38
mordredand then you need an entry in   <authorizationStrategy class="">13:38
mordredin config.xml13:38
mordredwhich references that user13:38
mordredseriously. we should puppet these files - OH - I remember why we donm't do that now13:39
yolandawe have an admin user there, but when i setup the SSO strategy for security, admin becomes unusable13:39
mordredyolanda: ah - we have an SSO account for our jjb user13:39
mordredand we just logged in to jenkins with it once, which made jenkins create the api key, and then we copied it out13:40
yolandamordred, yes, we have one... but i was wondering on how to reuse the old initial admin account, doesn't look possible, right?13:40
mordredI don't believe you can once you've turned on SSO13:40
yolandafor example in gerrit you can have users comming from SSO, but you can create users that are non-interactive13:40
mordredyeah. gerrit is smarter about this13:40
mordredjenkins doesn't seem to understand users from multiple auth sources13:41
*** mestery_ is now known as mestery13:41
fungilooks like our api token is a 384-bit keyblob base64-encoded13:42
funginot sure if jenkins cares whether it can base64 decode it13:42
*** michchap has joined #openstack-infra13:42
yolandaso the trick should be how to generate a fake api token ... if admin has logged once i just could grab it, but it not i need to generate a fake one13:43
bauzasfungi: thanks for your comment on
bauzasfungi: I found that the pypi mirror is different in betwen my own tox env and Jenkins one13:44
bauzasfungi: Jenkins is using
fungiyolanda: dd if=/dev/random bs=48 count=1 | base6413:44
bauzasfungi: while I'm using
*** qba73 has joined #openstack-infra13:44
yolandafungi, thx!13:44
openstackgerritRussell Bryant proposed a change to openstack-infra/reviewstats: Show total age stats last
openstackgerritA change was merged to openstack-infra/reviewstats: Show total age stats last
bauzasfungi: so, I'll swith from to and check what is pulling oslo-config13:45
fungibauzas: yes, i think for unofficial projects our tests will use and then fall back on if the dependency can't be satisfied from the first13:45
bauzasfungi: well, that's not cool at all :(13:46
bauzasany chance to fix this for Climate ?13:46
bauzas(a stackforge project)13:46
*** qba73_ has quit IRC13:47
fungibauzas: well, that allows projects to depend on unreleased openstack libraries while under development. i'd rather just fix whatever is causing the issue. what was the difference in your pip freeze besides oslo.config and oslo-config?13:49
bauzasI have to dig further13:50
*** michchap has quit IRC13:51
*** SergeyLukjanov has joined #openstack-infra13:51
fungiyou should be able to cut and paste the pip freeze output from jenkins and from your local run, use sort to make sure they're in the same order (may not even be necessary) and then just run diff on them13:52
bauzasyup, just done that actually13:52
fungii do that sort of comparison all the time when we see issues which might be introduced by new releases of dependencies13:53
bauzassounds like my nova egg is the faulty one13:53
*** thedodd has joined #openstack-infra13:53
bauzaspasting int13:53
fungiyou're depending on a master branch tarball of openstack/nova from right? that should depend on oslo.config normally13:54
* fungi checks to see if he might just be insane13:54
fungiyeah, nova's master branch has oslo.config>=1.2.0 in requirements.txt13:55
bauzasargh, connection timed out while pasting the diff13:57
bauzasbasically, there was a 41c4313:57
bauzas< nova==2014.1.a265.gfd2492313:57
bauzas> nova==2013.1.g313:57
*** DinaBelova has joined #openstack-infra13:57
bauzaswe're pulling from a tarball13:58
bauzaswhen using, we got a Grizzly release13:59
fungioh, i wonder if your nova>=master line is the issue...13:59
fungimordred: ^13:59
mordredyes. the nova>=master line is a problem13:59
* mordred needs to think for a second about how to do that right ...14:00
mordredspeaking of ...14:00
fungii think that will tell pip to pull in the most recent prerelease version it can find from pypi14:00
bauzaswell, speak to jd__ then :D14:00
mordredttx: I think we might have reached the point where we might need to consider actually releasing nova and friends to pypi on tags14:00
mordredttx: we now have _multiple_ projects in our own ecosystem which are trying to consume nova in their test-requirements.txt in order to test themsleves against it14:01
mordredwhich seems fair14:01
mordredbut we're making them do crazy tarball urls to accomplish it14:01
bauzasmordred: we actually need to test our own Nova scheduler filtrer14:01
*** prad_ has joined #openstack-infra14:01
mordredttx: I'm wondering if we shouldn't start publishing the actual releases to pypi - and then, since we're going to add the code to be able to publish pre-releases to pypi but only as wheels so that pip >=1.4 will do the right thing for oslo.config - we should be able to use the same tooling for server projects too14:02
mordredttx: thoughts - does that thought make you want to kill me?14:02
*** pblaho has joined #openstack-infra14:03
*** sdake_ has joined #openstack-infra14:03
*** sdake_ has quit IRC14:03
*** sdake_ has joined #openstack-infra14:03
*** rwsu has joined #openstack-infra14:03
fungibauzas: so this makes sense. on our test infrastructure the dep on the tarball url gets satisfied, then the dep on >=master gets checked and the master tarball is updated with the most recent nova version on our pypi mirror (which is a grizzly milestone). when not using our mirror the >=master fails to download anything else because there is no nova released to real pypi14:04
bauzasfungi: yup14:04
mordredfungi: why is our mirror picking up a nova?14:04
fungimordred: because we have one on there from circa grizzly g314:04
mordredfungi: ah - hysterical raisins you mean14:04
mordredmakes sense to me now14:04
bauzasfungi: mordred: I'm actually getting a nova package14:05
bauzasfungi: nova==2014.1.a265.gfd2492314:05
fungibauzas: yeah that's getting built from the tarball pip retrieves. i think if the nova>=master line were omitted it would cease to bet replaced with the old grizzly one14:06
fungier, cease to be replaced14:06
bauzasfungi: mmm, you're right14:06
fungibauzas: you can try that by amending your patch to take that nova>=master line out of test-requirements.txt and see if the tests pass14:07
bauzaswe actually only need to test against Nova for the filter14:07
*** prad__ has joined #openstack-infra14:07
*** jecarey has joined #openstack-infra14:07
bauzasfungi: yup, doing it right now14:07
*** prad_ has quit IRC14:07
*** afazekas has quit IRC14:08
bauzasI'll rebuild the tox env for sanity checking before submitting to Gerrit14:09
*** pblaho has quit IRC14:09
mordredjeblair: when you get up - nodepool is assuming that the cloud account has a secgroup with 22 open14:10
bauzasfungi: btw, I guess that I could test the Jenkins behaviour by providing the Openstack pypi mirror when building thru tox ?14:10
mordredjeblair: this is apparently not a thing that can be counted on14:10
ttxmordred: hmm14:10
mordredbauzas: if you just put the openstack mirror in your pip config, your local tox will behave as if it's on our system14:10
fungibauzas: yes, what we do is basically
yolandafungi, mordred, i checked token from the xml file but it should be encrypted in some way. Token is different from the one shown in the UI to the one in the xml file. Using the one in XML doesn't work, but the one from the UI does14:11
mordredyolanda: wow, really?14:11
mordredyolanda: Im guessing that perhaps jenkins has changed behavior but ours is still generated/stored using old behavior14:11
bauzasmordred: fungi: thanks14:11
yolandai'm with jenkins 1.53 version14:12
ttxmordred: I'd rather not do it automatically this time around. We can certainly upload something manually for havana if needed and then discuss at summit what to do for icehouse ?14:12
mordredttx: yes14:12
mordredttx: totally don't want to to automatically this roud - it's too close14:12
fungiyolanda: yes, so my guess is that this is a sha384 hash base64 encoded14:12
mordredttx: but I thnk it might make some of the things people want to do have a more defined set of semantics14:12
*** SergeyLukjanov is now known as _SergeyLukjanov14:13
fungiyolanda: can you confirm that what it generates is 64 printable ascii text characters in length?14:13
fungi(in the config file)14:13
ttxmordred: I want to wrap that discussion with the "where to publish our tarballs" storyboard/tarballs.o.o/pypi discussion we need to have anyway14:13
mordredjeblair: do you think the 'right' thing to do is add code to nodepool to detect secgroup extension, then if it's there, ensure there is a port 22 rule?14:13
*** dcramer_ has joined #openstack-infra14:13
mordredttx: oh - I didn't realize that was a discussion topic14:13
mordredbut if you think it is, then we should _definitely_ have that discussion :)14:14
ttxmordred: there is no reason why we should duplicate LP "tarball downloading" feature if we can have something clean at tarballs.o.o and/or pypi14:14
fungiyolanda: that's what i see in ours anyway. when i base64 decode it i get 48 seemingly-random bytes, and the only common hash function which makes output of that length is sha38414:14
mordredttx: +1000014:14
yolandafungi, it looks like base64 encoded14:14
ttxmordred: the only reason we use LP for that is that it gives you convenient sig upload14:14
mordredttx: I do not think that we need to have tarball downloading in storyboard. at most, I could see having a pointer from a release in storyboard to a pypi/tarball.o.o location14:14
mordredttx: yah14:15
bauzasfungi: that doesn't work, as I'm not pulling the nova tarball14:15
mordredttx: well, I think that pypi has sig uploading that we could figure out how to use14:15
ttxmordred: in the case of storyboard, less is more14:15
fungiyolanda: unfortunately, if they're using sha384 then that's probably only the *last* step in their algorithm, which is almost certainly doing multiple rounds with key stretching of some variety14:15
mordredttx: have you put in a topic for this?14:15
mordredttx: ++ yes14:15
mordredttx: I do not think storyboard wants to be launchpad14:15
mordredI think it wants to be exactly what we need out of it14:16
ttxmordred: we could wrap that discussion into the "release cycle" one... Or just make another one14:16
ttxmordred: actuially would make sense as another one14:16
mordredlet's make anotuher one14:16
mordredI think it's a different topic14:16
ttxmordred: where we'd also discuss stremalining the client lib release process14:16
yolandafungi, doesn't like as good news then14:16
annegentle_ttx: when you have a minute, I have some questions re: release timing14:16
yolandadoesn't look as good news14:16
mordredttx: what's the short story on that one?14:16
fungiyolanda: yes, it's probably something you'd need to read the jenkins source to figure out, or chat with the devs on #jenkins14:17
ttxmordred: I'd like people go through the same steps as far as announcing is concerned14:17
ttxand/or changelogging14:17
mordredah. nod14:17
*** rakhmerov1 has joined #openstack-infra14:17
*** rakhmerov has quit IRC14:17
ttxso it's a genberal discussion on the future of release artifacts in general14:17
ttxmordred: i'll let you file it14:17
ttxyou can push to "release management" topic. If you keep "storyboard" in infra, I have one slot left14:18
*** rakhmerov1 has quit IRC14:18
yolandafungi, i'll try to talk with jenkins, i'll leave that as a manual step now, that people login with admin user and set the api token manually14:18
*** rakhmerov has joined #openstack-infra14:18
*** senk has quit IRC14:18
fungibauzas: you get an error about the -f http:// line in test-requirements.txt or you get an error about an unsatisfied dependency on nova?14:18
bauzasatm, I can't test my filter due to the nova master tarball missing14:18
bauzasfungi: I'm getting an ImportError as nova is missing14:19
mordredttx: submitted14:19
bauzaswhen running the tests14:19
mordredttx: I included one more question ...14:19
*** gaelL has quit IRC14:19
ttxmordred: more question, more better14:19
mordredttx: which is "if we base things around releasing to pypi, what happens if we had a non-python thing"14:19
ttxmordred: ++14:20
mordredttx: which, while not in scope right now - infra does release a few different non-python things14:20
*** gaelL has joined #openstack-infra14:20
mordredandI'd like to be able to track that in storyboard14:20
mordredat the very least14:20
fungifor that matter, there are tripleo projects which include python boilerplate only so they can build release tarballs, but otherwise contain no python at all14:20
*** beagles has quit IRC14:21
*** beagles has joined #openstack-infra14:21
fungiwhich does allow us to release them to pypi, but that's still a little odd14:21
*** alcabrera has joined #openstack-infra14:21
mordredfungi: ooh!14:21
mordredI seem to be building nodepool things now on the tripleo grizzly cloud14:21
fungimordred: congrats!14:22
mordredNg, SpamapS, lifeless ^^14:22
*** SergeyLukjanov has joined #openstack-infra14:22
mordredthree patches that are currently pending need to land14:22
mordredand the security-group has to be added14:22
mordred(although that is now done by hand)14:22
mordredbut I'm getting puppet executions happening14:22 /Stage[main]/Openstack_project::Base/File[/etc/puppet/puppet.conf]/mode: mode changed '0644' to '0444'14:22
mordredwhich means we're probably past the "nodepool interacts weirdly with this cloud" stage14:23
bauzasI do understand the problem, I'm just looking for a workaround14:23
mordredremove the -f14:23
mordredjust put the url14:24
mordredsee if tha tworks14:24
bauzaswhat is doing the -f flag ? (sorry for that question if obvious)14:24
Ngmordred: \o/14:24
rakhmerovmordred: hey, sorry for being a bother :) could you pls tell some details on you comment in ? We'll fix that once you let us know14:25
mordredbauzas: in your file - each line is basically an argument/param to pip14:25
mordredrakhmerov: looking14:25
bauzasmordred: uh oh :)14:25
*** dcramer_ has quit IRC14:26
mordredrakhmerov: you have a section for refs/heads/release-* which is a different pattern than other openstack-y things14:26
mordredthat's fine - I just wanted to check in with you and make sure that was on purpose and was going to do what you thought it was going to do14:26
*** tstevenson has joined #openstack-infra14:26
fungibauzas: "if you want to install from a tarball or zip file with a direct link, you don’t need -f option, you just need to pass the absolute url"14:26
rakhmerovwell, yeah. I thought it shouldn't be necessary a 100% match to other projects' structure14:27
rakhmerovmordred: yes, that's what we wanted14:27
mordredrakhmerov: great!14:27
rakhmerov:) thx14:27
bauzasfungi: mordred: thanks, we actually copied from ceilometer :)14:27
*** gaelL has quit IRC14:27
mordredbauzas: well, they're slightly different - but I think that the weirdness here is an indication that we need to do something different14:28
*** erfanian has quit IRC14:29
fungimordred: is -e needed to parse the egg comment at the end of the url?14:29
mordredfungi: no14:30
mordredfoo/bin/pip install ''14:30
mordredworks fine14:30
*** gaelL has joined #openstack-infra14:30
*** CaptTofu has joined #openstack-infra14:30
*** gaelL has quit IRC14:30
bauzasmordred: fungi: thanks a lot, removing the #egg tag plus the -f is enough for working :)14:31
bauzaslast step before submitting, I'll change my pip mirror to openstack14:31
bauzasand check if still ok14:31
*** tstevenson has quit IRC14:33
*** dafter has quit IRC14:33
*** dafter has joined #openstack-infra14:33
*** dafter has joined #openstack-infra14:33
*** gaelL has joined #openstack-infra14:35
*** markmcclain has joined #openstack-infra14:36
*** michchap has joined #openstack-infra14:37
*** tvb|afk has joined #openstack-infra14:38
*** tvb|afk has quit IRC14:38
*** tvb|afk has joined #openstack-infra14:38
*** dafter has quit IRC14:38
*** swann has joined #openstack-infra14:40
*** gaelL has quit IRC14:41
*** gaelL has joined #openstack-infra14:42
*** datsun180b has joined #openstack-infra14:42
*** wenlock has joined #openstack-infra14:44
*** michchap has quit IRC14:45
*** pentameter has joined #openstack-infra14:46
*** rcleere has joined #openstack-infra14:47
*** rnirmal has joined #openstack-infra14:48
*** zojoncj has joined #openstack-infra14:49
*** tvb|afk has quit IRC14:50
*** senk has joined #openstack-infra14:51
*** jcoufal has joined #openstack-infra14:54
*** flaper87|afk is now known as flaper8714:57
openstackgerritA change was merged to openstack-infra/config: add os-loganalyze to #openstack-infra
*** yassine has quit IRC15:03
*** dcramer_ has joined #openstack-infra15:04
*** senk has quit IRC15:04
openstackgerritDavid Shrewsbury proposed a change to openstack-infra/gear: Add SSL error checking
sdaguefungi: did you have an opinion on the switch over to os-loganalyze as the wsgi filter?15:10
fungisdague: just now getting around to reading it15:10
jeblairmordred: haha.  re: secgroup -- you could rephrase that as "nodepool is assuming the cloud provides vms with a working connection to the internet".  :)15:10
*** mrodden has quit IRC15:10
fungiwe just have a secgroup which is wide open and then filter with iptables on the individual servers there, right?15:11
jeblairmordred: to seriously think about your question -- the secgroup thing isn't something that needs to be done for each host, so it's a little strange to put it in nodepool15:11
jeblairfungi: yep15:11
mordredjeblair: fair15:11
mordredjeblair: so we should just consider that potentially as part of setting up the cloud accout itself15:12
mordred"note to self, configure secgroup on new cloud account"15:12
fungialso, i believe the last time we tried to auto-update security groups with the level of churn in that tenant we brought hpcloud to its knees and then clarkb got called in the middle of the night15:12
mordredjeblair: I'm much closer - I got it running puppet on the nodes, but then something got unhappy and I wasn't looking and the scrollback was gone from y buffer by the time I looked15:13
jeblairmordred: we could have it check something like that at startup, but i'm currently leaning toward what you just said; partly for simplicity and bitrot reasons.15:13
jeblairmordred: you could set up a logging config that goes to console and file15:14
mordredjeblair: I'm about to send a mailing list message to tripleo and infra about "how I set up a nodepool and pointed it at this cloud for testing"15:14
mordredjeblair: I could indeed - but I'm pretty fast and loose with this setup15:14
mordredI'm thkning >file & tail -f file15:14
jeblairmordred: cool, could it be a howto doc for nodepool?15:14
mordredjeblair: might be - I think there are a couple of steps which a bit wonky and specific15:15
mordredjeblair: but certainly could be the start of one15:15
*** yassine has joined #openstack-infra15:16
mordredjeblair: if I wasn't running - would nodepool still try to make images and spin up the min amount?15:16
mordredjeblair: also, this:15:18
mordred                            self.log.debug("No current image for %s on %s"15:18
mordred                                           % (,
*** yassine has quit IRC15:19
*** yassine has joined #openstack-infra15:19
jeblairmordred: yes it would15:20
mordredis giving me DEBUG:nodepool.NodePool:No current image for tripleo-test-cloud on tripleo-precise15:21
mordredwhich scans weirdly in english15:21
jeblairmordred: i believe the interpolation is backwards15:21
*** senk has joined #openstack-infra15:21
openstackgerritMonty Taylor proposed a change to openstack-infra/nodepool: Change image debug interpolation order
*** mrodden has joined #openstack-infra15:23
*** beagles has quit IRC15:27
*** alexpilotti has joined #openstack-infra15:29
bauzasfungi: I'm still stuck with my Jenkins issue15:30
bauzasI provided the workaround you mentioned about nova15:30
bauzasand now the whole gates are failing...15:30
bauzasfungi: error in setup command: 'tests_require' must be a string or list of strings containing valid project/version requirement specifiers15:31
openstackgerritMonty Taylor proposed a change to openstack-infra/nodepool: Add gear to requirements
annegentle_ttx: have any time to chat about docs release?15:33
jcoufalclarkb: hey Clark, is it you who put together customizations for
jcoufalclarkb: mostly skin, layout, etc15:36
fungijcoufal: the askbot developers did all that work for us, i think. reed would know when he's around15:36
fungibauzas: i'm posing around in other projects to find working examples15:36
annegentle_ttx: mostly want to know exact timing and mechanism for cutting a stable/havana branch for openstack-manuals (anyone else in -infra know?)15:37
bauzasfungi: oki doki, that's good point15:37
fungijcoufal: keep in mind that we basically contracted the askbot devs to set up, configure and manage that for us15:37
jcoufalfungi: I know, I know, just basic deployment is pretty terrible, so I wanted to just take the skin and layout customization from already existing openstack solution15:38
jcoufalwill ask reed when he's around15:39
jcoufalfungi: thanks15:39
jcoufalfungi: awesome15:40
fungijcoufal: those *may* be the droids you're looking for15:40
fungibauzas: ceilo uses
fungi(not #egg=nova-master)15:41
fungibauzas: could that be the issue?15:41
* fungi double checks whether they still do that or if his checkout is stale15:41
fungibauzas: yep, they still do it that way...
*** rfolco has joined #openstack-infra15:42
*** dafter has joined #openstack-infra15:43
*** amotoki has quit IRC15:43
bauzasfungi: lookign15:44
bauzasfungi: ok, putting #egg=nova at the end15:45
ttxannegentle_: yes15:45
bauzasand testing15:45
*** UtahDave has joined #openstack-infra15:45
annegentle_ttx: ok, in the past I just gave a SHA to either clarkb or fungi and he made a stable/relname from it, does that sound right? What time of day does that need to be done?15:46
fungiannegentle_: that sounds right, and i can do it whenever you need as long as that sha is already merged15:47
*** flaper87 is now known as flaper87|afk15:47
ttxannegentle_: we can do it anytime between now and Thursday15:47
ttxannegentle_: (we did oslo-incubator already, for example)15:48
clarkbjcoufal: fungi is correct, I didn't do any customization. I had only tried to deploy it with mysql15:48
jcoufalclarkb: ok, thanks15:49
ttxannegentle_: sorry, got distracted15:49
clarkbfungi: mordred: yeah security groups are bad15:49
ttxannegentle_: I can probably do it for you as well15:50
mordredclarkb: well, yeah15:50
fungimordred: Shrews: olaph: i went ahead and registered for ato since early bird pricing is cheap and ends on wednesday. looks like fun15:50
*** guohliu has joined #openstack-infra15:50
fungii may not attend all of it, but at that price it's not like i have to squeeze every ounce from the conference15:50
ttxannegentle_: so the moment when you are done (hopefully on Thursday or before) give me a SHA and i'll create stable/havana for you (or infra will)15:50
clarkbthere are still people wandering around wiht power tools. I wonder if I should've taken this call at home...15:51
annegentle_ttx: okay sounds good15:51
ttxannegentle_: no precise time -- you can actually cut now if that's how you roll :)15:51
annegentle_ttx: I'm working on a patch that I'll rebase15:51
annegentle_ttx: but it takes something like 2 days to get a sitemap.xml, heh15:51
ttxannegentle_: huh15:51
annegentle_ttx: crazy I know15:52
*** hashar has quit IRC15:52
fungimordred: are you the one giving the openstack talk at ato? or do you know who is?15:53
fungitheir site didn't seem to say15:53
fungiclarkb: (local conference)15:54
mordredfungi: atwood and I both are giving one15:54
fungimordred: oh, great!15:54
annegentle_clarkb: or fungi: Do you have a minute to check into why DocImpact flags aren't working any more? Looks like since this went in: we haven't seen DocImpact15:55
fungiannegentle_: i'll check gerrit's logs15:55
fungiannegentle_: do you have an example change which should have opened a bug and didn't?15:55
openstackgerritSean Dague proposed a change to openstack-infra/config: switch over to os-loganalyze on logs.o.o
sdaguefungi: ok, should be fixed now15:56
*** oubiwann has joined #openstack-infra15:57
*** qba73 has quit IRC15:57
annegentle_fungi: yeah I think should have?15:57
*** jcoufal has quit IRC15:59
*** markmc has quit IRC16:03
fungiannegentle_: oh, i wonder if we're not calling the impact script from the change-merged hook16:03
* fungi checks16:03
*** beekneemech is now known as bnemec16:05
*** qba73 has joined #openstack-infra16:05
*** alcabrera has quit IRC16:06
fungiyep, only on patchset-created, looks like16:06
fungii'll propose a fix16:06
fungiannegentle_: the problem was actually introduced in the previous patch,
*** gyee has joined #openstack-infra16:07
fungiwe were never running the script on that event16:07
*** dafter has quit IRC16:08
*** reed has joined #openstack-infra16:09
*** alcabrera has joined #openstack-infra16:09
*** yaguang has joined #openstack-infra16:09
*** markmcclain has quit IRC16:11
*** mihgen_ has quit IRC16:11
*** yassine has quit IRC16:13
*** DinaBelova has quit IRC16:14
*** boris-42 has quit IRC16:15
Shrewsfungi: awesome. c ya there16:16
*** david-lyle_ has joined #openstack-infra16:17
*** matty_dubs is now known as matty_dubs|lunch16:17
annegentle_fungi: oh whew, glad you found it. Too bad we lost a month of docimpact. sigh.16:18
annegentle_fungi: ok maybe just 3 weeks. argh. any way to know how many we missed?16:19
jeblairannegentle_, fungi: i think you could use a gerrit search16:19
annegentle_jeblair: oh I wondered, what is the search incantation?16:19
annegentle_jeblair: (I don't know how to search within commit messages)16:19
jeblairmessage:DocImpact status:merged16:20
annegentle_jeblair: thanks, looks like only a dozen or so, whew16:21
annegentle_oh crap it hasn't been working since Sept 12?16:21
annegentle_more like 2 dozen16:21
jeblairannegentle_: yeah, i think it goes onto the next page then for a total of 2716:22
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Move docimpact from created hook to merged
fungiannegentle_: ^16:23
*** senk has quit IRC16:24
*** fbo is now known as fbo_away16:26
*** yaguang has quit IRC16:27
*** branen has joined #openstack-infra16:28
annegentle_fungi: maybe I don't understand, but didn't we stop sending emails as notification and instead directly log a doc bug?16:28
clarkbthat was the intent, but it had a bug16:29
annegentle_fungi: clarkb oh ok, cuz Sept 13 was the last time our mailing list had a report in it16:29
fungiannegentle_: yeah, in my opinion it got changed in the wrong place, or at least not enough of the right places16:30
annegentle_fungi: clarkb: so will we see mails again?16:30
fungiannegentle_: no, the change to only do e-mails wasn't what broke it16:30
annegentle_fungi: ok16:30
fungiannegentle_: as i said earlier, the change before that one to only update on change-merged events is what broke it16:30
fungiannegentle_: because the script was looking for a condition which would never occur (we weren't calling it on change-merged events)16:31
*** SergeyLukjanov has quit IRC16:31
*** ryanpetrello has quit IRC16:31
annegentle_fungi: ok then I'll +1 (but can't really test it)16:31
annegentle_fungi: ok16:31
*** ryanpetrello has joined #openstack-infra16:31
annegentle_clarkb: ah yeah removing the dest-address would've helped me read it :)16:32
*** ryanpetrello has quit IRC16:32
fungiannegentle_: a simpler approach to only opening the bug on change-merged would have been the patch i just posted, which stopped calling the script on patchset-created events and started calling it on change-merged instead16:32
*** SergeyLukjanov has joined #openstack-infra16:32
*** ryanpetrello has joined #openstack-infra16:32
*** SergeyLukjanov has quit IRC16:32
annegentle_fungi: calling it on changed-merged is our desired state16:32
fungiannegentle_: good, since that's what my patch does16:33
annegentle_fungi: well, then WOOHOO! :)16:33
fungiheh ;)16:33
annegentle_fungi: are we all loopy this week or what? :)16:33
* fungi is loopy every week16:34
*** rakhmerov has quit IRC16:34
annegentle_for once I get to go to a summit with the release done16:34
fungithat slightly increases the chances that you might be able to keep up with mordred's liver while we're there16:34
clarkbannegentle_: I bet that is a nice feeling16:34
*** moted has joined #openstack-infra16:35
*** OlivierSanchez has quit IRC16:35
annegentle_fungi: no chance of that, I'm a lightweight drinker :)16:35
annegentle_clarkb: yeah I'm giddy with anticipation. Now if only the install guide were done. gaaaahh16:36
fungianybody know if there's a gerrit query syntax which would allow me to find changes that i specifically voted a certain way on (for example, changes where i voted <=-1)?16:36
fungitrying to update my review workflow with a step to look for reviews where people replied to my comments without uploading a new patchset16:37
jeblairfungi: i don't see it16:38
fungididn't think so. best i can do is filter for watched changes with a negative vote by any reviewer16:38
fungithat'll have to do i guess16:38
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Add a new ssh key for jeblair
*** mriedem has quit IRC16:39
jeblairfungi, clarkb, mordred: ^ that's a key i'm going to use on a device while traveling16:39
*** zaro0508 has joined #openstack-infra16:40
*** dizquierdo has quit IRC16:41
*** osanchez has joined #openstack-infra16:41
*** aspiers has quit IRC16:43
*** rakhmerov has joined #openstack-infra16:43
*** rfolco has quit IRC16:44
annegentle_Ok, so we have 27 more DocImpact bugs since Sept 12 that weren't picked up by the automation.,n,z finds them all16:51
annegentle_er sorry, wrong channel :)16:51
*** markmcclain has joined #openstack-infra16:52
*** beagles has joined #openstack-infra16:56
*** SergeyLukjanov has joined #openstack-infra16:56
*** nicedice has joined #openstack-infra16:56
wenlockwhats clark's last name?16:57
wenlockmiss type16:57
sdaguefungi: no, you can't do that level of detailed query16:59
*** SergeyLukjanov is now known as _SergeyLukjanov17:00
sdaguesadly... once we get to newer gerrit we've got a rest api though, right?17:00
fungisdague: a pity :(17:00
clarkbthere has been work upstream to use a lucene indexes for more complex queries17:00
clarkbbut I think they found that has problems scaling (indexes are expensive)17:00
fungithat would rock17:00
clarkbnot sure if they sorted the problems they ran into with projects that have a lot of history17:00
fungiand yeah, i imagine something on the order of what we have now for elasticsearch infrastructure to make that feasible17:01
pleia2good morning17:01
clarkbpleia2: morning17:01
clarkbsdague: how are those etherpad style headings working out?17:01
clarkbsdague: :) we finally have those for you. sorry about the wait17:01
sdagueclarkb: good, though I haven't done much more in etherpad since they went live17:01
sdaguethis week though, I need to start planning the qa sessions out17:02
openstackgerritSean Dague proposed a change to openstack-infra/config: switch over to os-loganalyze on logs.o.o
sdaguefungi: commit message fixed17:03
sdagueclarkb: no worries on the wait, it's all good17:03
*** ben_duyujie has joined #openstack-infra17:04
*** derekh has quit IRC17:04
sdagueclarkb: if you want to check that out, the current os-loganalyze tree should fix all the missing log issues17:05
*** guohliu has quit IRC17:05
fungisdague: you're improving the negativity of my review stats. thanks! ;)17:05
clarkbsdague: awesome. I will poke at that as soon as I have finished settling into this office move17:06
sdagueI was +Aing a bunch of stuff in myself this weekend to get to functioning state, I'll stop doing that now to get eyes on it as it's going into prod at some point. So you might want to look at the os-loganalyze history just to familiarize with changes as well17:06
clarkbsdague: ok17:06
*** SergeyLukjanov has joined #openstack-infra17:07
*** matty_dubs|lunch is now known as matty_dubs17:08
*** _david_ has joined #openstack-infra17:12
* fungi pops out for a quick errand and will brb17:13
openstackgerritJoe Gordon proposed a change to openstack-infra/reviewstats: Make openreviews generate quartiles instead of just median.
jog0russellb: ^17:17
*** nati_ueno has joined #openstack-infra17:21
*** thomasbiege has joined #openstack-infra17:22
russellbjog0: cool17:23
openstackgerritA change was merged to openstack-infra/reviewstats: Make openreviews generate quartiles instead of just median.
russellbjog0: regenerating stats now17:24
*** thomasbiege has quit IRC17:24
jog0russellb: neato thanks17:24
*** vipul is now known as vipul-away17:28
russellbjog0: updated17:30
*** senk has joined #openstack-infra17:30
clarkbsdague: does os-loganalyze handle syslog and swift too?17:32
*** vipul-away is now known as vipul17:33
clarkbsdague: looks like ti skips them maybe? (this is fine, trying to look at history like you mentioed)17:33
sdagueclarkb: yep, so now it has a stricter check of what it supports -
sdaguebased on this regex on files -
*** markmcclain has quit IRC17:34
*** dprince has quit IRC17:34
sdagueso the passthrough filter will just ignore the level param if the file name isn't known to be supported17:35
sdagueit will still do timestamp linking in the html though if it can17:35
sdagueso console.html.gz will have timestamp links with this17:36
clarkbI see17:37
*** SergeyLukjanov has quit IRC17:37
*** markmcclain has joined #openstack-infra17:37
*** DennyZhang has joined #openstack-infra17:37
*** rakhmerov has left #openstack-infra17:38
*** jerryz has joined #openstack-infra17:38
*** sarob has joined #openstack-infra17:45
*** markmcclain has quit IRC17:48
*** markmcclain has joined #openstack-infra17:48
clarkbsdague: the config change to use os-loganalzyse lgtm. I am not able to babysit right now or I would approve. Hopefully can babysit after lunch17:49
*** blamar has quit IRC17:50
*** oubiwann_ has joined #openstack-infra17:51
sdagueclarkb: sounds good17:52
sdaguejust let me know17:52
*** ben_duyujie has quit IRC17:54
*** oubiwann has quit IRC17:54
hub_capso now that elastic recheck is telling me that i hit a particular bug, do i get to blindly reverify bug XXX and not feel bad?17:55
*** miqui has joined #openstack-infra17:55
clarkbhub_cap: I think we are hoping that some human interaction will still happen (this is why we aren't auto rechecking for you)17:56
clarkbhub_cap: but we should be giving you good enough bread crumbs most of the time17:56
hub_capclarkb: kk thx17:56
* hub_cap blindly clicks review (i kid i kid)17:56
*** DennyZhang has quit IRC17:57
openstackgerritMatthew Treinish proposed a change to openstack-infra/config: Increase timeout for neutron jobs
*** pblaho has joined #openstack-infra17:59
*** senk has quit IRC18:01
*** boris-42 has joined #openstack-infra18:02
clarkb etherpad.o.o is holding up18:03
*** sarob has quit IRC18:05
*** lcestari has quit IRC18:07
*** qba73 has quit IRC18:07
*** senk has joined #openstack-infra18:08
*** mriedem has joined #openstack-infra18:09
ekarlsodoes one need to register a package before it can be uploaded to pypi ? (python-libraclient)18:11
clarkbekarlso: yes, and you need to make the openstackci account a maintainer18:11
*** johnthetubaguy has quit IRC18:11
clarkbekarlso: eg
*** senk has quit IRC18:12
openstackgerritMatthew Treinish proposed a change to openstack-infra/elastic-recheck: Convert test_classifier to be a unit test
*** senk has joined #openstack-infra18:12
mtreinishclarkb: want to enable gating unit tests for elastic-recheck?:
clarkbmtreinish: looking. for do we expect the tests to take more than an hour? I ask because now that we have parallel testing with testr for the other jobs it might cause a bit of a revolt if we go back to >1hour test times :)18:15
sdagueclarkb: neutron tests take > 1 hr18:16
mtreinishclarkb: it's only for neutron which isn't parallel yet18:16
sdaguebecause they can't run parallel yet18:16
mtreinishwe can drop it back down once we get parallel working on neutron18:16
*** nicedice_ has joined #openstack-infra18:16
*** pblaho has quit IRC18:17
openstackgerritJoe Gordon proposed a change to openstack-infra/elastic-recheck: Remove query for bug 1227915
uvirtbotLaunchpad bug 1227915 in cinder "logical volume "volume-x" already exists in volume group "y" (dup-of: 1223469)" [High,Confirmed]
uvirtbotLaunchpad bug 1223469 in cinder "Volume create tgtadmin fails with error code 22" [High,Fix released]
mtreinishjog0: nice you got a triple ^^^18:19
jog0mtreinish: !18:20
*** nicedice has quit IRC18:20
openstackgerritA change was merged to openstack-infra/config: Add unit tests to gating jobs on elastic-recheck
openstackgerritA change was merged to openstack-infra/elastic-recheck: Add query for 1239637
*** sarob has joined #openstack-infra18:21
openstackgerritA change was merged to openstack-infra/elastic-recheck: Remove query for bug 1227915
uvirtbotLaunchpad bug 1227915 in cinder "logical volume "volume-x" already exists in volume group "y" (dup-of: 1223469)" [High,Confirmed]
uvirtbotLaunchpad bug 1223469 in cinder "Volume create tgtadmin fails with error code 22" [High,Fix released]
*** senk has quit IRC18:23
lifelessmoted: it takes a few seconds for floating ip associations to take effect under neutron18:33
lifelessmordred: ^18:33
lifelessmoted: sorry18:33
motedlifeless: np18:33
*** annegentle_ has quit IRC18:33
*** annegentle has joined #openstack-infra18:34
*** senk has joined #openstack-infra18:34
*** jerryz has quit IRC18:37
mordredlifeless: awesome18:38
mordredlifeless: is there any way to check to see if they're actualy in effect?18:38
clarkbfungi: I have dug through the jenkins changelog and bug tracker but haven't found anything like what we are experiencing. We may want to try and collect more info then submit a bug ourselves18:38
*** dhouck_ has joined #openstack-infra18:38
lifelessmordred: no, you need to know if the neutron agent on the network node (which is where the NAT rules live) has done it's thing18:38
fungiclarkb: the load issue? is it happening again?18:39
clarkbfungi: it isn't happening now18:39
mordredand I suppose it's too much to ask for the cloud ot actually know that status and report it18:39
jeblairlifeless: you can't query it for state like you can a server?18:39
clarkbfungi: but looking at graphs it seems to steadily ramp up (we have a ways to go before it falls over)18:39
mordredlifeless: have any of the people designing this ever tried to use a cloud?18:39
clarkbfungi: also comparing to jenkins01, 02 seems so much busier18:39
lifelessBobBall: I haven't looked into what drivers HPCloud uses at all18:39
lifelessBobBall: sorry18:40
mordredhow do I tell if I'm still waiting for the floating ip to take effect, or if something has failed?18:40
mordredjeblair:     client.floating_ips.delete(self.args['ip_id'])18:40
mordredKeyError: 'ip_id'18:40
lifelessjeblair: not as far as I know18:40
mordredjeblair: I'm hitting that on teardown - any idea where I should look?18:40
*** jerryz has joined #openstack-infra18:40
mordredlifeless: line 27: TRIPLEO_ROOT: unbound variable18:41
lifelessmordred: I suspect all of them have run a cloud in devstack style18:41
mordredlifeless: this is where I'm getting in nodepool + tripeo18:41
lifelessmordred: and are used to grvelling in log files / being patient18:41
lifelessmordred: rather than treating things as 'if I were a user, how would I ...'18:41
jeblairmordred:         return self.submitTask(DeleteFloatingIPTask(floating_ip=ip_id))18:42
lifelessmordred: source refresh-env $wherever_you_put_your_tripleo_root18:42
jeblairmordred: that should probably be ip_id=ip_id18:42
fungiclarkb: i think jeblair's nodepool changes may solve some of the load disparity between 01 and 02?18:42
pleia2lifeless, mordred - looking at how toci does some of this waiting stuff might help, since that's all automated18:42
mordredjeblair: ok. I'll go look for htat line - it didn't look lke id_id existed in the current context18:42
jeblairfungi: i don't think i've fixed that yet.18:42
mordredpleia2: we do that a lot in our current stuff like nodepool18:43
pleia2ah, ok18:43
jerryzlifeless: is there any progress on this bug:
mordredpleia2: the problem is - in production, when we aska cloud we don't control to do something and then we have to just poll over and over again until it works18:43
uvirtbotLaunchpad bug 1217722 in openstack-ci "'rake' or 'rake lint' blows up" [Undecided,New]18:43
fungijeblair: oh, right, you were still philosophizing over that this weekend18:43
mordredpleia2: it's really tough to know when something's actually failed18:43
pleia2mordred: yeah18:43
* mordred stabs humans18:43
pleia2I see18:43
jeblairyeah, the wait-for-resource paradigm is standardized in nodepool18:43
jeblairmordred, pleia2: but it assumes that there's actually some kind of success condition18:44
jeblairwhich doesn't seem to be the case here?18:44
mordredjeblair: I think the success condition is "your IP works"18:44
mordredlifeless: btw - the actual error was that we didn't have a secgroup defined18:44
pleia2I could do with some catching up for a couple of days I was gone, what's the status of the tripleo nodepool thing I wrote last week and had to be disabled?18:44
mordredpleia2: I'm working on debugging it right now18:45
pleia2mordred: ok, great18:45
mordredpleia2: see email I just sent to mailing list...18:45
jeblairmordred: have we seen this problem yet?18:45
mordredpleia2: but the tl;dr is that spinning up a local nodepool with the config for tripleo is a needed thing18:45
lifelessmordred: oh, well not having a sensible default security group18:45
pleia2mordred: oh, haven't caught up on -dev yet :)18:45
lifelessmordred: or a custom one - yearh, that 'll fail18:45
mordredpleia2: so that we can track down it working before we re-proposed to infra18:46
jeblairmordred: nodepool currently waits for an ip to be reported as associated with a server18:46
lifelessjerryz: no idea18:46
lifelessjerryz: I just reported it :>18:46
mordredjeblair: yah. I do not think we've seen the problem lifeless is saying might be there in the wild18:46
lifelessjerryz: when I was told to use it to see if a patch was good.18:46
jerryzjerryz: just curious how precise slaves work with that.18:47
jeblairmordred: ok, let's see if it's a problem then.  perhaps it's covered by the existing check.18:47
jerryzlifeless: did you do it locally or it ran on precise slaves?18:47
fungijerryz: i think you have to gem install puppetlabs_spec_helper (we may already do that on our slaves)18:48
lifelessfungi: jeblair: I have an unmirrored revision on git.o.o18:48
lifelessjerryz: locally18:48
openstackgerritMonty Taylor proposed a change to openstack-infra/nodepool: Make ip_id arg match calling and consuming
jerryzlifeless: thanks18:49
mordredjerryz: agree18:49
lifelessjeblair: de1dc315539f0c69c30c9101d1db93735bc78c26 is in gerrit as merged, but not present in when I pull it from
mordredlifeless, pleia2: in any case, I believe I'm down to nodepool works with grizzly cloud now (once we land the changes I've submitted, plus jeblair's plus pabelanger's)18:50
mordredlifeless, pleia2: and the tripleo script themselves are now unhappy18:50
mordredwhich is great18:50
lifelessgit.o.o is resolving to for me18:50
pleia2mordred: yeah, so right now in this first step I don't think we want any real tripleo stuff running18:50
*** mindjive1 is now known as mindjiver18:50
pleia2mordred: we just want it to run a simple shell script as POC and I'm working on actually running tripleo18:50
fungijerryz: yeah, we install it on the slaves...
pleia2mordred: see iterations 1 & 2
mordredpleia2: totally. but the script in openstack-infra/config that is used to make the image18:52
mordredpleia2: for tripleo nodes18:52
mordredpleia2: is unhappy and fails18:52
pleia2mordred: oh! I see18:52
mordredwhen the image creation process runs it18:52
mordredpleia2: yah! we're to the good stuff :)18:52
fungilifeless: i'll have a look on the git server farm. maybe one of them has that repo in a bad state18:52
mordredpleia2: lemme send a follow up to that email about the nodepool patches that need to be applied or land for it to work18:52
pleia2mordred: cool, thanks18:52
lifelessfungi: tripleo-image-elements18:52
lifelesspleia2: pull-tools is failing18:54
lifelesspleia2: perhaps using refresh-env would make sense18:54
fungilifeless: i see that commit on all four of the git servers18:55
lifelessgit log origin/master | head -n118:55
lifelesscommit 98d3195d2d18851686652af81acbafee2a52984418:55
mordredjeblair, pleia2, clarkb: these are the patches I've been running with locally in order to make things work:
lifelessgit remote show -n origin18:56
lifeless* remote origin18:56
lifeless  Fetch URL:
clarkblifeless: did you fetch origin/master ?18:56
lifelessclarkb: yes18:56
jeblairfungi, lifeless: i cloned using the cgit urls on all 4 servers and the commit only shows up in the clone from 0318:56
lifelessgit fetch origin18:56
*** hashar has joined #openstack-infra18:56
fungilifeless: does git show 98d3195d2d18851686652af81acbafee2a529844 work? it worked for me in the bare repo on all 4 servrers18:57
lifelessfungi: it does for me but that is because I fetched it from gerrit18:57
lifelessfungi: see jeblair's comment18:57
clarkbso maybe the ref for master didn't get updated?18:57
jerryzfungi: thanks!18:57
fungilifeless: oh, sorry, i was looking at the other ref you pasted, de1dc315539f0c69c30c9101d1db93735bc78c2618:57
lifelessfungi: de1dc315539f0c69c30c9101d1db93735bc78c26 is on top of 98d3195d2d18851686652af81acbafee2a52984418:58
lifelessfungi: 98 is what I get when I fetch, de is what I should18:58
jeblairlifeless, fungi: i repeated that using the git:// protocol, and it's the same -- only on git0318:58
pleia2mordred: great, thanks18:58
fungiweird. it's definitely in the repos themselves18:58
*** oubiwann_ is now known as oubiwann18:58
fungii wonder why pull doesn't find it18:58
lifelessfungi: what does git log master | head -n1 get you?18:59
*** senk has quit IRC18:59
*** SergeyLukjanov has joined #openstack-infra18:59
jeblairi see no related errors in the gerrit log18:59
jeblairfungi, lifeless: ^18:59
jeblairfungi, lifeless: i have to run now, sorry18:59
*** alcabrera has quit IRC19:00
fungilifeless: 98d3195d2d18851686652af81acbafee2a529844 on git02 and git04, 2b2aa7cb887a9cce08f7dc0d2a81520bcd283a3a on 01 and de1dc315539f0c69c30c9101d1db93735bc78c26 on 0319:00
fungiso clearly replication is not keeping some of those in sync19:01
fungitrying to figure out why19:02
lifelessfungi: those are head refs?19:02
clarkbfungi: it is syncing the actual contents but not updating the master ref?19:03
fungilifeless: yes, git log master | head -n119:03
*** jpich has left #openstack-infra19:03
lifelessfungi: so yeah, this is borken :)19:03
fungiclarkb: it's a possibility, though i'm not seeing that behavior on repos i've been using19:03
*** david_h has joined #openstack-infra19:03
fungilifeless: yes indeed19:04
lifelessI only spotted this because we had a monkey patch in place in prod19:04
lifelessand I went to undo it now the commit had landed19:04
lifelessand ... things that make you go hmmm19:05
fungii'm going to see if there's something coincident about the vintage of those stale head refs19:06
*** nati_uen_ has joined #openstack-infra19:07
fungihuh, all from the past few days19:10
*** nati_ueno has quit IRC19:10
fungifrustrating. gerrit show-queue showed nothing going on, but gerrit replicate openstack/tripleo-image-elements caused them all to sync up19:13
fungihunting around now for any similar bugs filed on gerrit19:14
*** che-arne has joined #openstack-infra19:18
*** luhrs1 has joined #openstack-infra19:18
*** luhrs1 has quit IRC19:19
lifelessfungi: it's flipping back and forth ? :)19:20
lifelessfungi: wonder if this is related to the 'X showed unmerged but was actually merged' thing19:21
*** _david_ has quit IRC19:21
fungiit's possible, though that was a while back right?19:21
*** julim has quit IRC19:22
*** aspiers has joined #openstack-infra19:22
fungiwow, 24 counts of pushreplication errors in the gerrit error log from today19:23
fungii hate the way java apps like to fill up their logs with multiline stack traces19:23
*** DennyZhang has joined #openstack-infra19:25
*** eharney has joined #openstack-infra19:27
fungimost recent one for tripleo-image-elements was on thursday...
*** dcramer_ has quit IRC19:29
funginot sure if gerrit expects to recover gracefully or not, but it's odd that the later commits did get replicated, just not the state of the head ref19:30
*** sarob has quit IRC19:30
*** sarob has joined #openstack-infra19:31
fungialso, head refs got updated on all of the git servers for a couple days after that error was logged, which makes me think it's unrelated19:31
sdaguefungi: I should point you to our openstack logs some time :)19:32
sdaguereed: multiline stack traces19:32
fungisdague: no need ;)19:32
fungisdague: in a debug log they're not so bad, but elsewhere... annoying19:33
*** nati_uen_ has quit IRC19:34
*** nati_ueno has joined #openstack-infra19:34
fungi!msg/repo-discuss/Y71yLwhC6hk/fex3cEgLcvEJ is interesting (circa 2.2 though)19:35
*** sarob has quit IRC19:36
*** atiwari has joined #openstack-infra19:36
*** hashar has quit IRC19:38
*** jaypipes has quit IRC19:41
*** beagles has quit IRC19:45
*** blamar has joined #openstack-infra19:45
*** dmakogon has joined #openstack-infra19:46
*** SergeyLukjanov is now known as _SergeyLukjanov19:46
*** _SergeyLukjanov has quit IRC19:46
reedsdague, not sure I follow :)19:46
*** hashar has joined #openstack-infra19:48
sdaguereed: oh, that was supposed to be 're:'20:00
reednow it makes more sense :)20:00
clarkbjeblair: mordred: fungi: I am going to approve did anyone else want to review it before I do that?20:01
clarkbthis is the change that uses the new os-loganalyze project20:01
mordredlove it20:02
*** marun has joined #openstack-infra20:02
clarkbok approved20:02
*** lcestari has joined #openstack-infra20:02
fungiahh, yep. already reviewed that one anyway20:03
*** qba73 has joined #openstack-infra20:04
openstackgerritA change was merged to openstack-infra/config: switch over to os-loganalyze on logs.o.o
clarkbthe other thing I am going to do today is get bup backups running on new etherpad.openstack.org20:04
clarkbbefore I do that I want to sort out what I should do with the old backups on old first thought is to tar them up and stash them on new but then they do not get rotated out (I could manually remove them in a month)20:05
clarkbany opinions on that?20:05
clarkbor we could just let them live on the old server and die with the old server when it is removed20:06
fungii'm not too concerned about them sitting on the new server collecting dust until someone thinks to delete them, but leaving the old server up for a month can't hurt either20:06
*** dcramer_ has joined #openstack-infra20:07
*** qba73 has quit IRC20:09
*** SergeyLukjanov has joined #openstack-infra20:13
openstackgerritA change was merged to openstack-infra/nodepool: Change image debug interpolation order
clarkbfungi: re Gerrit repo replication are you seeing the failure to lock exceptions in the error log?20:17
*** DennyZhang has quit IRC20:17
fungiclarkb: i hadn't spotted one, no20:19
*** david_h has quit IRC20:19
fungii'll look again20:19
*** blamar has quit IRC20:20
*** nati_uen_ has joined #openstack-infra20:23
*** dizquierdo has joined #openstack-infra20:25
*** nati_ueno has quit IRC20:26
mtreinishfungi, mordred, jeblair: can one of you nudge through? It's a prereq for enabling more tests on neutron.(I need it to confirm my exclude list) The current list is looking like a full tempest run -10 tests.20:29
jog0mordred: why 3 change-ids
mordredjog0: bug in script :(20:33
mtreinishmordred: did clarkb's fix for that work?20:33
jog0mordred: -1 for you20:34
clarkbmtreinish: I think it should work but we haven't updated requirements since20:34
clarkbwe might also want to abandon the existing derped changes20:34
mtreinishclarkb: you'd have to login as Jenkins to do that right?20:36
mtreinishthe tempest one won't pass jenkins right now so you can just pull that one out if you want to test20:36
mtreinish(or wait until thurs for it to autoexpire)20:36
openstackgerritA change was merged to openstack-infra/config: Increase timeout for neutron jobs
mtreinishfungi: thanks for the +A, if you want to watch the slow pseudo full neutron tempest runs:
mtreinishit's got a fun regex20:38
clarkbmtreinish: Gerrit admins can abandon them. I haven't done that yet, but I think I will if no one disagrees it should be done20:38
*** vipul is now known as vipul-away20:39
mtreinishclarkb: I guess being an admin has it's advantages. :)20:39
clarkbmtreinish: there are a few :)20:39
clarkbmtreinish: sdague: with os-loganalyze in place we are indexing keystone, swift and syslog again so you can use those files in queries20:40
clarkbjog0: ^20:40
mtreinishclarkb: you can go ahead an abandon it, since build_sphinx doesn't work20:40
mtreinishclarkb: ok cool20:40
jog0clarkb: awesome20:41
clarkbmordred: is abandoning the list of changes here with mutliple change ids something that hsould be done?,n,z20:42
*** sarob has joined #openstack-infra20:42
*** sarob has quit IRC20:46
mordredclarkb: sure - except I would like to actually sync to global requirements before final release - but the cat is probably out of the box now20:47
*** SergeyLukjanov has quit IRC20:47
clarkbmordred: I think syncing is a good thing before release. I just think those change are a bit messy20:47
clarkbmordred: newly generated ones should be fine. maybe we should try manually triggering the job after the existing changes are abandoned?20:47
mordredclarkb: totes. I'm just saying I think most projects aren't expecting additional changes now20:47
clarkbmordred: up to you (we could just go with the slightly dirty changes)20:47
mordredI could go either way20:48
*** SergeyLukjanov has joined #openstack-infra20:48
hub_capmordred: lifeless clarkb image caching question. I think it migth make sense for me to run the 10*cache scripts that are in dib (or have a --cache-image flag or something in dib) for this caching. im not sure just curling the images is the thing to do because dib does a _lot_ more than caching the image in the 10* step and we can bypass it all if we run it ahead of time and pass --offline to dib in the actual jobs20:48
mtreinishclarkb, sdague: found a small issue with os-loganalyze:
clarkbmtreinish: that is awesome :)20:49
clarkbsdague: maybe we shouldn't touch the .html files for now?20:49
clarkbhub_cap: right, iirc actual building of some stuff needs to happen or there will be many cache misses20:49
hub_capclarkb: ok good. i think originally there was thought that the 10* script was too much _stuff_ to run20:50
sdaguemtreinish: awesome :)20:51
sdagueok, I'll provide a fix in a few for that20:51
*** nati_uen_ has quit IRC20:52
lifelesssorry, on a call, will drop in later20:52
*** nati_ueno has joined #openstack-infra20:53
hub_caplifeless: no worries we can chat later20:53
*** yamahata has quit IRC20:53
*** yamahata has joined #openstack-infra20:54
openstackgerritClark Boylan proposed a change to openstack-infra/config: Backup etherpad.o.o with bup.
*** michchap has joined #openstack-infra20:55
*** michchap has quit IRC20:56
clarkbmordred: I won't abandon any changed yet then20:57
clarkbmordred: I don't think there is a rush to change anyhting right now so we can mull it over20:57
*** michchap has joined #openstack-infra20:59
*** emagana has joined #openstack-infra20:59
*** emagana has quit IRC20:59
*** emagana has joined #openstack-infra21:00
*** amotoki has joined #openstack-infra21:01
*** HenryG has joined #openstack-infra21:01
*** dmakogon_ has joined #openstack-infra21:02
clarkbfungi: may interest you as you have kicked the irc bots a bit21:03
*** dmakogon has quit IRC21:05
*** adalbas has quit IRC21:05
*** julim has joined #openstack-infra21:06
openstackgerritA change was merged to openstack-infra/gerritbot: Reconnect to IRC on send error
*** miqui has quit IRC21:10
*** sarob has joined #openstack-infra21:11
*** lcestari has quit IRC21:13
*** alexpilotti has quit IRC21:13
*** vipul-away is now known as vipul21:14
sdaguemordred: ... pbr error in keystone? -
*** sarob has quit IRC21:17
*** sarob has joined #openstack-infra21:18
openstackgerritSean Dague proposed a change to openstack-infra/config: only run the console.html logs through os-loganalyzer
sdagueclarkb: that should fix the testr issue21:20
sdaguebut keep things for the console.html logs21:20
clarkbsdague: thanks21:21
mordredsdague: why do yo uthink it's pbr?21:21
mordredI see it21:21
*** esheffield has joined #openstack-infra21:21
*** sarob has quit IRC21:22
pleia2mordred: I don't know how you travel so much, I just went across the country last night and I'm beat today :)21:23
esheffieldclarkb: it was suggested that I ping you about a problem I've encountered21:25
clarkbesheffield: ok21:25
clarkbpleia2: if you get mordred to share his secret you should share it with the rest of us21:25
esheffieldclarkb: I have a patch failing in jenkins21:26
esheffieldclarkb: but it passes locally21:26
esheffieldclarkb: the patch is
esheffieldclarkb: it seems to be the same as bug which was marked invalid at some point21:27
uvirtbotLaunchpad bug 1180670 in nova "XenAPIVMTestCase.test_instance_snapshot_fails_with_no_primary_vdi sometimes fails" [Medium,Invalid]21:27
mordredsdague: why the heck is that only doing that in that one patch?21:27
mordredpleia2: you just learn to leave in pain constantly :)21:27
pleia2mordred: haha21:27
clarkbesheffield: how old is the parent commit of that patch? I wonder if maybe rebasing on latest master would pull in the fix noted in that bug report21:29
*** aspiers has quit IRC21:29
mordredsdague: omg. so -21:29
esheffieldclarkb: I just rebased it this past friday21:29
clarkbesheffield: in general test failures like that are a result of inter test conflicts. We use testr to run tests in parallel, testr does this by partitioning the set of tests then running each partition in its own process. This partitioning results in different test orders across runs which can cause conflicts21:30
clarkbesheffield: typical conflicts arise when test A requires test B to run first or if test A requires test B to not run first21:30
mordredsdague: we're running on the pbr repo21:30
mordredsdague: which is copying in a that setup_requires on pbr21:30
mordredsdague: in devstack21:31
mordredhow has this eer worked?21:31
esheffieldclarkb: interesting - is there a good approach for fixing that or just have to retry the tests later and hope for a different ordering21:31
clarkbesheffield: to hunt those down load into testr as the most recent test run, then do `testr run --analyze-isolation` this causes testr to do test bisection looking for conflicts21:31
esheffieldclarkb: also, that test worked on friday after the rebase, but I had a different failure which I pushed up today and encountered this problem21:32
*** fifieldt_ has joined #openstack-infra21:32
clarkbesheffield: you may need to convert subunit_log.txt to subunit v2 before testr loading the file. `subunit-1to2 < subunit_log.txt > v2.txt && testr load v2.txt && testr run --analyze-isolation`21:32
sdaguemordred: interesting.... ?21:32
esheffieldclarkb: ok, will try that. thanks21:32
clarkbesheffield: right the test order is not deterministic. So if a test breaks we may not see it consistently break (which is both a good and a bad thing, good that it fails, bad that it doesn't fail 100% of the time)21:33
*** hashar has quit IRC21:33
esheffieldclarkb: thanks for the suggestions - I'll try them out and see if I can find anything!21:34
*** mriedem has quit IRC21:34
clarkbesheffield: good luck. let me know if you have more testr/unittest framework questions21:34
*** thomasm has quit IRC21:35
openstackgerritMonty Taylor proposed a change to openstack/requirements: Only sync things that have the bullet
mordredsdague: ^^21:36
*** michchap has quit IRC21:38
*** michchap has joined #openstack-infra21:38
*** matty_dubs is now known as matty_dubs|gone21:41
*** aspiers has joined #openstack-infra21:42
jog0mordred:  NOVA_RAX_AUTH
*** jhesketh has quit IRC21:44
jog0I think we should kill that block of code but don't think that will go over so well21:44
mordredjog0: O M G I thought I deleted that 6 months ago21:45
mordredjog0: I don't care. it's completely inappropriate. I raised a stink about it a while ago21:45
mordredchmouel and I wrote a set of auth plugin drivers21:45
mordredyou don't need it to connect to rax anymore21:45
* mordred is angry21:45
jog0thats not what the random doc I Found on teh Internets said21:46
mordredit's wrong21:46
mordredyou only need that if you want to use rax's proprietary api key extension21:46
*** vipul is now known as vipul-away21:46
mordredwhich is not our just to support21:46
hub_capquick run, mordred is turning gren21:46
hub_capand his pants are ripping21:46
mordredif you use username/password, it works without us carrying code21:46
mordredjog0: oh - so that whole system is deprecated I thought21:47
mordredthe auth_system21:47
jog0mordred: want to push out this patch?21:47
mordredwell, I mean, I'm the maintainer of the rax auth plugin :)21:47
jog0chmouel's comment implies it is deprecated21:47
mordredit is21:48
*** dhouck_ has quit IRC21:48
jog0mordred: heh so who better to remove it21:48
sdaguemordred: unit tests?21:48
jeblairthat code looks like it's a backwards-compat shim for the env variable, so that it uses the plugin21:48
* jog0 likes seeing code get killed21:48
jeblairmordred, jog0: ^21:48
jeblairmordred, jog0: (so it's deprecating the env var, not deprecating the plugin)21:49
jeblairmordred, jog0: (which i agree does not seem to be necessary, though tbh, does provide something you can't get by using a password)21:49
sdaguemordred: also, is there a bug for that?21:49
mordredjeblair: last I talked to chmoel, I thought we were ditching the whole auth plugin approach21:49
mordredchmouel. not chmoel21:50
clarkbjeblair: I think you will like that change21:50
jog0mordred: we should get rid of auth plugins21:50
jeblairmordred: wfm, i don't have any input there; just trying to 'help' clarify the conversation.  :)21:50
*** hemna is now known as hemnafk21:50
sdaguethe pbr thing, so I can recheck with it21:50
jog0OpenStaack uses keystone for auth not 3rd party plugin21:50
mordredjeblair: :)21:50
*** vipul-away is now known as vipul21:50
mordredsdague: nope. I just wrote the patch - filing bugs is too hard21:50
mordredsdague: you want me ot file a bug real quick?21:51
jeblairclarkb: i love that change21:51
jog0jeblair: I am trying to find the actual rackspace auth code it may not be in trunk21:51
jeblairjog0: i believe it's in the plugin21:51
sdaguemordred: if you think it's fixed, I'm fine, I'll just recheck no bug it21:51
mordredI think it should be21:51
mordredI mean- if my patch above clear, and it still doesn't work, there is _another_ bug we'll need to track down21:52
chmoueljog0: i think we can't remove it just yet21:52
mordredchmouel: why not?21:52
jog0chmouel: :(21:52
mordredchmouel: it's not needed21:52
chmouelsome peps use it for other stuff than rax or hp21:52
mordredoh - the plugin system?21:52
mordredthey do?21:52
chmoueli have a review somewhere which was explained21:53
hub_capif you file a  bug for that RAX_AUTH_JUNK, plz file a bug in trove/cinder too jog021:53
jog0chmouel: what about just NOVA_RAX_AUTH21:53
jog0hub_cap: :)21:53
* jog0 files a bug21:53
chmouelthe env variable compa?21:53
mordredchmouel: we were discussing making novaclient consume keystoneclient21:53
mordredchmouel: instead of doing its own thing21:53
chmouelmordred: oh yeah but first we wanted to bring all the functionalities21:54
chmouelthat novaclient had21:54
chmouelwhich was the auth system21:54
chmoueli have a review somewhere that moves it to keystoneclient21:54
chmouelbut i think jamie lennox was doing ina  different way21:54
chmouelthere is a review for that21:54
mordredof course21:54
jog0hub_cap: gahhh21:54
chmoueli'm on a plane atm so i can't check21:54
mordredchmouel: :)21:54
jog0hub_cap: so you either need the plugin auth supprt or we nuke plugin auth all together21:55
chmouelbut will come back to you tomo with the stats of all of this21:55
clarkbjeblair: mordred: you both +2'd 51703 at about the same time, should I +A it?21:55
mordredclarkb: yes21:55
* fungi is going to grab some dinner and then brb21:55
openstackgerritA change was merged to openstack-infra/nodepool: Fix another wrong use of server_id
openstackgerritA change was merged to openstack-infra/nodepool: Add gear to requirements
hub_capjog0: yes... its everywhere (god why isint this code in some common place)21:55
jog0hub_cap: yeah ... sigh21:55
openstackgerritA change was merged to openstack-infra/nodepool: Make ip_id arg match calling and consuming
openstackgerritA change was merged to openstack-infra/nodepool: Add the ability to pass pool for AddPublicIP
openstackgerritA change was merged to openstack-infra/nodepool: Don't use max_overflow with sqlite
chmouelmordred, jog0:
chmouelso yeah there isa ctually three people trying to move the auth system mechanism to keystoneclient :)21:56
jog0chmouel: nice I still don't think clients should support plugin auth21:56
jog0but that is me ignoring reality21:57
chmoueljog0: i think there is some use cases for security that may be needed21:57
jog0chmouel: then we fix keystone21:57
openstackgerritA change was merged to openstack-infra/config: Backup etherpad.o.o with bup.
jog0mordred: want to start a fun ML thread?21:57
chmoueljog0: i had one started some time ago about it21:57
mordredthe auth system plugin is helping people run non-interoperable clouds21:57
chmouellet me check21:57
* mordred helped write it - so hopefully it's not terrible if I trash it21:58
hub_capis present life mordred shaking his fist at past life mordred21:58
clarkbhub_cap: yes, we should get popcorn21:58
uvirtbotLaunchpad bug 966329 in python-novaclient "RAX-specific auth in novaclient" [Low,Fix released]21:58
jog0mordred:  haha21:59
jog0mordred: looks like you didn't actually dothat one21:59
chmoueljog0, mordred:
chmouelthe story was brian removed it22:00
chmouelinfra needed it at that time22:00
chmoueland rax as well22:00
chmouelso we kind of made it in a plugged way22:00
chmouelbut sill not nice22:00
mordredsince then, we've stoped needing it22:01
jog0"IMHO, as long as keystone allows for external authentication (as it does),22:01
jog0the auth plugin system on the client side should exist."22:01
mordredclarkb, jeblair: if you're bored, and its dependency chain are fun22:01
*** sarob has joined #openstack-infra22:01
mordredI quasi-agree with that22:02
mordredalthough I'm pretty sure that the code to implement it sohuld go in keystone client22:02
jog0mordred: for sure22:02
chmouelmordred: yep22:02
mordredI also think that the auth extension system is broken and misdesigned22:02
chmouelyeah totally22:02
mordredbecause it does not allow for discoverability of what auth plugin one needs to use to connect22:02
mordredyou have to know before you connect what auth plugin to use22:03
mordredinstead, like the other registries, you should be able to ask your cloud "hey man, what plugin do I need to talk to you"22:03
clarkbmordred: re 51403. Part of the reason that code exists is that tox's depend on virtualenv doesn't give us new virtualenvs22:03
mordredand it should say "dude, you need x509 auth"22:03
mordredclarkb: no?22:03
*** jpeeler has quit IRC22:03
clarkbmordred: and you want to keep virtualenv up to date so that it syncs up with setuptools22:03
mordredclarkb: I believe that has changed22:03
clarkbmordred: let me look at tox22:03
mordredclarkb: current tox should pull in current virtualenv - because it depends on pip 1.4 being present22:04
mordredin the virtualenv22:04
chmouelI think i have a version of cinderclient that moves it to use keystoneclient22:04
mordredtox actually uses pip 1.4-only command lines22:04
mordredchmouel: neat22:05
chmouelnot sure why i didn't go on with that and submitted for review
jog0mordred chmouel: this is why this belongs in keystoneclient
clarkbmordred:  install_requires = ['virtualenv>=1.9.1', 'py>=1.4.15', ] so we will get at least 1.9.122:05
mordredclarkb: yup. which is what we need22:06
clarkbmordred: but nothing newer once installed. With that puppet resource we keep things in sync and up to date22:06
clarkbwithout it we may drift22:06
*** jhesketh has joined #openstack-infra22:06
*** jhesketh__ has joined #openstack-infra22:06
mordredclarkb: I believe we are now at a point where that is fine22:06
mordredclarkb: because 1.9 is post distribute/setuptools re-merge22:07
clarkbmordred: until the next round of crazy setuptools ness22:07
mordredand post pip1.422:07
mordredif setuptools goes crazy again, I will kill them22:07
*** dizquierdo has left #openstack-infra22:07
clarkbmordred: I think I am more worried about drift22:07
mordredI'm less, actually22:07
clarkbmordred: old slaves will have 1.9.1, new slaves will have 1.9.5 and others 2.0 an dso on22:07
mordredI'm more worried, tbh, that we drift virtualenv up without tox22:07
mordredlemme think about that for a sec22:08
mordredjog0: ++22:09
jog0mordred hub_cap:
uvirtbotLaunchpad bug 966329 in python-novaclient "RAX-specific auth in novaclient" [Low,Fix released]22:09
jog0I brought that one back22:09
mordredclarkb: ok. I'm willing to stew on that one longer22:09
mordredclarkb: how about the two in the dep chain before it22:09
hub_cap<3 jog022:09
clarkbmordred: fwiw I don't feel too strongly about it. If it becomes a problem we upgrade virtualenv again22:10
clarkbor pin the version22:10
mordredclarkb: yeah22:10
clarkbjust trying to make sure we understand why it did exist in that state though22:10
mordredI was mostly trying to clean up extra invocations of pip in our puppet22:10
mordredso that we could see the places we were actually needing to directly have puppet manage it22:10
mordredclarkb: btw - I think that our dance around python2 and python3 pip may be obsolete22:11
mordredclarkb: but I need to go do some testing22:11
mordredclarkb: now that we can do the tox 1.6 use_develop - the pyhton _inside_ of the virtualenv should always be the python used for all things22:12
mordredunless there was another issue I'm forgetting22:12
jog0mordred chmouel:
*** syerrapragada has joined #openstack-infra22:12
clarkbmordred: there is the slight issues of trying to isntall stuff globally for both22:12
clarkbmordred: but that isn't a unittest slave problem so much as a server problem22:13
mordredclarkb: right - but why do we need to do that?22:13
clarkbmordred: initially I wanted it so that infra could be leading hte way in python3 support22:13
* mordred was trying to remember what it was we wanted to accomlish when we wanted to do that22:13
clarkbmordred: the logstash stuff was all python3 at one point22:13
mordredyah. I remember that22:13
clarkbbut daeling with that is a bit derpy for other reasons22:13
mordredsuch as being told we should never install things globally with pip :)22:14
clarkbmordred: I have been thinking about that, and I think it is wrong22:14
clarkbif your package management system can't coexist with other package managers then it is broken22:15
clarkbcabal and maven seem to deal with it alrgith (with their pseudo virtualenv things)22:15
mordredso, the current state is that it is broken then22:15
clarkbI don't think it is a global vs venv problem. I see it more of a `pip install foo` should just work22:15
mordredI agree. but it doesn't22:15
mordredpython and the distros have an intractible disagreement about where things should install22:15
mordredand who should own them22:15
mordredand how it should work22:16
clarkbthe globalness is an implementation detail22:16
mordredwith each side thinking the other is dead wrong22:16
clarkbmordred: right22:16
mordredclarkb: so I agree with you in principle22:16
*** datsun180b has quit IRC22:16
mordredbut in practice, it turns out that global installation of software via pip mixed with global installation of software via package managers is unsupported and undefined22:16
mordredthat this is the case is sad22:17
*** CaptTofu has quit IRC22:17
mordredand makes me want to ditch python and move to java or c++22:17
clarkbmordred: right so pip should instead do a thing where it installs to /usr/local/pip (or /opt or wherever outside of apt and yums pervue)22:17
*** CaptTofu has joined #openstack-infra22:17
clarkbthen deal with paths22:17
mordredwell... funny story22:17
clarkbpip sort of does this on debian22:17
mordredit MOSTLY works on debian22:17
mordredbut python is mad at debian22:17
clarkbbut still thinks it should nuke globally installed things22:18
mordredfor doing this22:18
*** gyee has quit IRC22:18
clarkbits "global" but doesn't interfere22:18
clarkbthis is good for dedup of data on multiuser systems and sanity22:18
mordredwell... you know22:19
*** psedlak has quit IRC22:19
clarkbI think we sometimes forget that these are multiuser systems22:19
mordredyou could always make a global virtualenv, say /opt/python, put 'source /opt/python/bin/activate' into /etc/profile22:19
mordredand go from there22:19
mordredclarkb: I think they are becomming less and less multi-user with the advent of clouds and containers22:20
mordredseparation is happening via containers or entire special-purpose VMs22:20
mordredof course, what was really awesome was the add and attach commands back in jeblair's and my AFS past :)22:21
mordrednow _there_ was a user-space way of dealing with this problem22:21
clarkbI think containers and VMs address a particular subset of needs. There is still a lot of reasons for mutliuser friendliness22:21
clarkbhell I run multiple desktops on my desktop at home concurrently22:22
mordredwell, but you're smart22:22
*** arosen has quit IRC22:23
*** vipul is now known as vipul-away22:23
*** vipul-away is now known as vipul22:24
clarkbthat said, I don't think thin clients are going to happen. They tried and tried and we don't seem to be headed that direction (I think the cheapness of actual usable hardware killed thin clients)22:25
*** psedlak has joined #openstack-infra22:26
clarkbmordred: is GLOBAL_REQUIREMENTS a string in now?22:28
*** syerrapragada has left #openstack-infra22:29
pleia2can this review get nudged along?
*** prad__ has quit IRC22:29
pleia2(very reviewed, just no +A)22:30
pleia2and useful ;)22:30
openstackgerritEdward Raigosa proposed a change to openstack-infra/config:     - remove syntax issues with pip3.rb     - remove syntax issues with python2/3.pp so that class names are pip:python2/3     - change python pp file requirements on bootstrap so we use a define     - added require for install_pip2 to occur after install
*** Protux has quit IRC22:31
*** gyee has joined #openstack-infra22:31
*** dcramer_ has quit IRC22:31
*** Protux has joined #openstack-infra22:32
*** marun has quit IRC22:33
mordredwenlock: if I was you - I'd just squash the two commits and push up over top of mine22:34
*** Protux has quit IRC22:34
mordredwenlock: (we don't consider that sort of thing rude in these parts)22:34
jog0where is the HTML?22:34
wenlockahh , i wasn't sure22:34
jog0clarkb: ^22:34
wenlockcan i actually do that after i've commited?22:34
jog0 works22:34
mordredwenlock: also - two things - a) congrats on your first patch! and b) next time you do produce a new commit, try to keep the first line to under 60 chars22:35
wenlockok, np22:35
*** mrodden has quit IRC22:35
reeddamn, jog0 that page is killing my firefox22:35
mordredwenlock: totally! for now, do 'git rebase -i origin/master' and then select s instead of pick on the second one22:35
mordredthen, when you squash them, make sure that the Change-Id from the first commit is the only change-id line in the commit message and that it's the last line22:36
mordredthen do a git review again22:36
* jog0 is glad he uses chrome22:36
reedjog0, don't be glad to use chrome, google is evil22:36
*** CaptTofu has quit IRC22:36
clarkbjog0: sdague broke it, fix is in review22:36
*** CaptTofu has joined #openstack-infra22:36
clarkbjog0: mordred jog0 fungi
clarkbpleia2: looking22:37
clarkbpleia2: I think we were hoping fungi would review it. I will go ahead and approve it since we have test snow22:38
*** thedodd has quit IRC22:38
mordredclarkb: I approved22:38
clarkbmordred: danke22:38
openstackgerritA change was merged to openstack-infra/config: only run the console.html logs through os-loganalyzer
openstackgerritJames E. Blair proposed a change to openstack-infra/nodepool: Consider existing nodes when allocating to a target
*** thomasm has joined #openstack-infra22:43
*** dmakogon_ has left #openstack-infra22:44
jeblairclarkb, fungi, mordred: ^ that should even out the load on jenkins01 and 0222:45
*** rnirmal has quit IRC22:45
*** alexpilotti has joined #openstack-infra22:45
*** SergeyLukjanov has quit IRC22:46
sdagueclarkb: that change go through yet?22:46
clarkbsdague: doesn't look like it22:48
clarkbI think it will get aplpied in ~4 minutes22:48
sdagueso, new crazy idea...22:48
sdaguehow would you propose I should set up the test to run this - ?22:49
jeblairclarkb, fungi: i just put that nodepool change into prod; current levels are jenkins01=20, jenkins02=71.  we'll see how it levels out.22:49
sdaguethis is basic syntax checking for bash for devstack22:49
clarkbsdague: I would do something similar to the existing pep8 jobs but make it not so pythony22:50
clarkbjeblair: I am working my way through that change22:50
sdagueclarkb: ok, so just run them on the unit test nodes?22:50
clarkbsdague: ya22:51
*** SergeyLukjanov has joined #openstack-infra22:53
*** pentameter has quit IRC22:53
jeblairclarkb, fungi:*.*%29&*.*%2922:54
*** branen has quit IRC22:54
jeblairclarkb, fungi: and we're already at 48 and 6622:55
clarkbsdague: yeah, in this case you don't need the special pythonstuff you can probably just embed it in the JJB configs without it looking nasty22:55
clarkbjeblair: the change looked good. I did have one comment but it doesn't need to be addressed22:55
*** sarob_ has joined #openstack-infra22:56
*** mrodden has joined #openstack-infra22:58
openstackgerritEdward Raigosa proposed a change to openstack-infra/config: Make pip install from upstream better
*** sarob has quit IRC22:59
*** senk has joined #openstack-infra23:00
*** david-lyle_ has quit IRC23:00
*** branen has joined #openstack-infra23:01
*** sarob_ has quit IRC23:01
openstackgerritSean Dague proposed a change to openstack-infra/config: add bash8 to devstack tests
*** sarob has joined #openstack-infra23:02
sdagueclarkb: so.... that's part of this, but I'm a little lost on where to add the layout definitions correctly23:02
*** zojoncj has quit IRC23:02
clarkbsdague: misc.yaml for JJB and under openstack/devstack in zuul/layout.yaml is where I would put it23:03
*** sarob has quit IRC23:04
*** sarob has joined #openstack-infra23:05
*** thomasm has quit IRC23:06
clarkbjog0: sdague: testr html files are good now23:07
jog0clarkb sdague: thanks23:09
openstackgerritSean Dague proposed a change to openstack-infra/config: add bash8 to devstack tests
*** oubiwann has quit IRC23:09
sdagueclarkb: ok, still not sure I got everything, but maybe it's close23:09
*** pcm_ has quit IRC23:09
*** amotoki has quit IRC23:10
openstackgerritJoe Gordon proposed a change to openstack-infra/elastic-recheck: Add query for bug 1239856
uvirtbotLaunchpad bug 1239856 in glance ""TimeoutException: Request timed out" on tempest.api.compute.images.test_list_image_filters.ListImageFiltersTestXML " [Undecided,New]
*** nati_uen_ has joined #openstack-infra23:18
fungieek scrollback23:20
SpamapS .. there's a fail in there because six is not >= 1.4.1 but the change I'm testing adds '>=1.4.1' to python-keystoneclient's requirements.txt .. is there just lag in the devstack gate nodes getting updated to the latest global reqs?23:20
jeblairSpamapS: can you link to the line you're talking about?23:21
*** nati_ueno has quit IRC23:22
*** jecarey has quit IRC23:22
clarkbSpamapS: jeblair: six==1.4.1 was installed23:23
*** nati_uen_ has quit IRC23:23
mordred2013-10-14 22:40:12.351 | 2013-10-14 22:40:12 Downloading/unpacking six (from oslo.config)23:23
mordred2013-10-14 22:40:12.357 | 2013-10-14 22:40:12   Downloading six-1.4.1.tar.gz23:23
mordred2013-10-14 22:40:12.360 | 2013-10-14 22:40:12   Storing download in cache at /var/cache/pip/
*** eharney has quit IRC23:23
jeblairsdague: the console logs seem to be escaping html -- so there's stuff like "<pre>" and "<a href=''>" in there23:23
clarkbjeblair: sdague: yes jenkins escapes the html for us when we write it out as html23:24
mordredjeblair: there was a processing bug that we've APRV'd a change for - but I don't think it's rolled out yet? clarkb ?23:24
clarkbit went out, it was for the non console.html files23:25
mordredah. gotcha23:25
SpamapSjeblair: ^^ re the change that is failing23:26
jeblairSpamapS: i read that in the logs, thanks.  :)23:26
jeblairclarkb: jenkins doesn't escape the html23:26
jeblairclarkb: looking at the file on disk, it looks like perfectly normal html23:27
clarkbjeblair: I thought it did23:27
clarkbbecause it is embedding it in <pre> tags. jeblair is there a particular console.html that is having problems/23:27
jeblairclarkb: the one SpamapS linked23:27
SpamapSjeblair: oh sorry I read your request wrong23:27
clarkboh I see23:27
clarkbjeblair: yeah that is a bug I think23:28
SpamapS2013-10-14 22:48:10.193 |   File "/opt/stack/new/python-keystoneclient/keystoneclient/", line 26, in &lt;module>23:28
SpamapS2013-10-14 22:48:10.193 |     from six.moves.urllib import parse as urlparse23:28
SpamapS2013-10-14 22:48:10.194 | ImportError: No module named urllib23:28
jeblairSpamapS: (and everyone else) you can link to a line using the anchor on the timestamp:
SpamapSOh, neato23:29
clarkbjeblair: sdague: we have nested <pre>s I think that is the problem23:29
SpamapSdid not even notice they were clickable23:29
*** nati_ueno has joined #openstack-infra23:29
jeblairclarkb: not to mention  &lt;module>23:29
*** senk has quit IRC23:30
clarkbSpamapS: the import is wrong23:31
*** senk has joined #openstack-infra23:31
jeblairclarkb: yeah, the log analyzer shouldn't <pre> an html file.23:31
clarkbjeblair: yup23:32
openstackgerritA change was merged to openstack-infra/nodepool: Consider existing nodes when allocating to a target
clarkbSpamapS: hrm in a venv I can import that line23:33
SpamapSclarkb: yeah I can't get that import to fail as long as six == 1.4.123:34
clarkbI wonder, it is only failing in the context of the tempest tests and tempest runs with tox23:35
clarkbperhaps site pacakges isn't doing what we expect there23:35
SpamapSyes it is failing in that context23:35
openstackgerritA change was merged to openstack-infra/git-review: Prefer .gitconfig username
*** atiwari has quit IRC23:37
clarkbtempest six<1.4.023:37
clarkbsitepackage=True must be smart enough to find versions it needs if that don't exist system wide (it didn't do this before)23:37
clarkbSpamapS: I think we can blame ceilometer for that. They pinned six, then we made them unpin because it caused all sorts of trouble23:38
clarkbSpamapS: so I think fixing that in tempest will fix it for you23:38
clarkbupper bounds on six are bad23:38
* SpamapS was just trying to throw 30 minutes on the pile of py3 porting effort23:39
SpamapSgetting dizzy having tumbled down this rabbit hole for a while now23:39
SpamapSclarkb: thanks for the analysis23:39
clarkbSpamapS: np, and yeah py3k is a hole you jump in and you quickly find that the bottom isn't where you expected it to be23:40
SpamapSusually I smack my head on the eventlet rock early on23:40
sdaguejeblair: yeh, I'll build an exclusion for that tomorrow23:41
jeblairsdague: cool, thanks23:41
fungii fell down a py3k hole 'round ought-six i reckon. haven't managed to climb back up yet23:42
sdagueback to cooking dinner....23:42
clarkbI am freezing at my desk (going to walk home and find warmth)23:42
fungiaught-six i means to say23:42
openstackgerritEdward Raigosa proposed a change to openstack-infra/config: Make pip install from upstream better
sdagueI might get to it later tonight, it's a simple enough change. Will be good to put unit tests in place for it as well23:43
SpamapSheh, two of my sons can at least say "well, I 'as born in aught-nine" or "I 'member you 'us born in aught-three"23:44
* SpamapS channeling his inner okie23:44
*** SergeyLukjanov is now known as _SergeyLukjanov23:53
*** _SergeyLukjanov has quit IRC23:53
openstackgerritEdward Raigosa proposed a change to openstack-infra/config: Make pip install from upstream better
*** SergeyLukjanov has joined #openstack-infra23:54
*** SergeyLukjanov has quit IRC23:54
*** CaptTofu has quit IRC23:55
*** CaptTofu has joined #openstack-infra23:56
*** yamahata has quit IRC23:56
openstackgerritDan Nguyen proposed a change to openstack/requirements: Add flufl.password to requirements

Generated by 2.14.0 by Marius Gedminas - find it at!