Tuesday, 2014-08-12

*** daya_k has joined #openstack-infra00:00
mordredclarkb: oh - btw ...00:01
mordredspeaking of upstreams00:01
mordredthe patches I made against ansible and that are sitting in that review in the config repo have all landed upstream00:01
clarkbor as a hack, we can add an option to allauth to do the wrong thing with openid and get username from the openid provider00:01
openstackgerritRussell Bryant proposed a change to openstack-infra/config: add experimental job for nova with f20 virt-preview  https://review.openstack.org/11335000:01
clarkbmordred: cool00:01
*** sdake_ has joined #openstack-infra00:02
*** ianw has quit IRC00:02
*** dkliban is now known as dkliban_afk00:03
*** ianw has joined #openstack-infra00:03
nibalizermuahahahha upstreaming is super fun00:03
*** esker has quit IRC00:03
* anteaya looks for the muahahahha repo00:04
*** e0ne has quit IRC00:06
anteayaokay just as a reality check, we have some jobs in post sitting at 7 hours00:08
*** MarkAtwood has quit IRC00:08
anteayanow I might have missed an important piece explaning this since i was out for a chunk of the day00:08
pleia2yep, node starving00:08
pleia2(ran out)00:09
anteayabut if someone can scrutinize I think we need some scrutiny00:09
anteayaso early fungi had a theory something might be going straight from build to delete00:09
anteayaanyone with root access able to do some splunking?00:09
*** e0ne has joined #openstack-infra00:10
pleia2anteaya: see clarkb's comments about 20 minutes ago00:10
*** rloo_afk has left #openstack-infra00:10
fungiwell, there are about 45 rackspace nodes and 6 hpcloud nodes stuck in error+deleting right now, though in the past few days i've noticed rax is periodically cleaning them up (this may be phschwartz's script in action)00:10
*** camunoz has joined #openstack-infra00:10
clarkbyup and hpcloud appears to have a high error rate00:11
fungihowever that's not really enough to make a huge difference in what we're seeing00:11
fungithe error rate however, would00:11
clarkbfungi: the difference is hpcloud is high error rate too00:11
*** otherwiseguy has joined #openstack-infra00:11
clarkbthey don't have anything about it on https://community.hpcloud.com/status so maybe I should look closer00:11
clarkbwill be a nice break from staring at pootle allauth code00:12
*** flwang_ has quit IRC00:13
*** e0ne has quit IRC00:13
anteayaclarkb: ha ha ha00:13
anteayawell there's that00:13
fungioh yeah... http://paste.openstack.org/show/93595/00:14
clarkbmordred: pleia2: I just tried removing django.contrib.auth.backends.ModelBackend from authentication backends and that didn't remove local auth00:15
fungiwe burned 6485 nodes on launch errors in hpcloud yesterday, according to the logs00:15
fungiand it's continuing in the current freshly-rotated log as well00:16
clarkbmordred: pleia2: I wonder if horizon devs would be of help /me context switches to nodepool now00:16
fungi117 in the past 16 minutes00:16
anteayafungi: go us00:16
*** funzo has quit IRC00:17
*** e0ne has joined #openstack-infra00:17
mordredfungi: what sort of errors are we getting? timeouts? or hard errors?00:17
*** dims_ has joined #openstack-infra00:18
mordredkrotscheck, clarkb: I'm starting to hack on launchpad import00:18
clarkbfungi: looks like it is failing to allocate floating ips00:18
*** nelsnelson has quit IRC00:18
anteayalooks like something changed round about 18 hours ago with hp cloud00:18
clarkbfungi: we could be leaking them again /me looks00:18
jogoso sanity check: https://review.openstack.org/#/admin/groups/188,members should list the QA ptl in groups?00:18
clarkbmordred: ^ novaclient throws exceptions when doing floating ips00:19
mordredI do not think there is a _great_ way to maintain multi-project bug+task info ... I'll come back to taht00:19
*** daya_k has quit IRC00:19
mordredclarkb: are we not reaping floating ips again?00:19
anteayajogo: no idea, that is up to the grenade ptl00:19
clarkbmordred: looking00:19
mordredrememer when we had that problem a while back?00:19
jogoanteaya: grenade is in QA00:19
jogoand there is no QA ptl only tempest ptl https://review.openstack.org/#/admin/groups/355,members00:19
anteayajogo: so I'd go with yes on that00:19
* mordred thinks we should try to re-use floating ips if they exist on the account00:19
jogofungi mordred: thoughts00:20
anteayamtreinish isn't listed there and should be00:20
fungimordred: http://paste.openstack.org/show/93596/00:20
mordredin fact, I wrote the code to do that already in the ansible patches - I could backport to nodepool ...00:20
jogois it easy to rename tempest-ptl to QA ptl?00:20
jogoor something like that00:20
fungimordred: clarkb: could be fip leaking00:20
clarkbmordred: I still disagree on that00:20
mordredclarkb: ok. let's argue about taht later00:20
clarkbmordred: the scale back is hard00:20
clarkbbut ya looks like leaky floating ips00:21
anteayajogo: or add tempest-ptl to grenade-core group00:21
anteayajogo: basically it is up to mtreinish00:21
mordredclarkb: agree re: leaky - you want to get them or youwant me to?00:21
fungilast time this came up, tteggel gave us a quick-n-dirty script we could use to find and remove disassociated floating ips00:22
*** nelsnelson has joined #openstack-infra00:22
jogoanteaya: and the naming tempest-ptl in grenade isn't an issue?00:22
mtreinishanteaya, jogo: it's that way now mostly because I haven't been an active reviewer on grenade historically00:22
clarkbmordred: I am trying to remember how you got htem last time?00:22
*** mkerrin has quit IRC00:22
anteayahe can decide and tell us what he wants and we refer to the email and make it so00:22
anteayajogo: not if mtreinish is happy00:22
jogomtreinish: and that there is no unified review team in qa00:22
anteayamtreinish: so what if any renames do you want in gerrit groups00:23
*** shashankhegde has quit IRC00:23
mordredclarkb: nova floating-ip-list | grep ' \- ' | awk '{print $2}'00:23
mtreinishjogo: yeah I guess that's the other half :)00:23
mordredso I'm going to run : nova floating-ip-list | grep ' \- ' | awk '{print $2}' | xargs -n1 nova floating-ip-delete00:23
mordredany objections?00:23
anteayajogo mtreinish so have a qa meeting item and discuss how you want teh groups to be set up00:23
clarkbmordred: iirc it is faster to use neutron00:23
clarkbmordred: but no objection here00:23
mordredoh - right. why was it faster?00:24
fungimordred: that will get just floating ips which aren't associated with an instance, i take it?00:24
anteayamtreinish jogo then tell us and we will move things around to reflect how you want it to be00:24
mtreinishanteaya: hmm, I'll have to think of a structure that makes sense00:24
jogomtreinish: maybe it would be best with the other grenade cores around00:24
jogoanteaya: thank you00:24
anteayamtreinish: very good00:24
mtreinishanteaya: I'm not sure it warrants a meeting item00:24
clarkbmordred: because it doesn't proxy through nova which is more hops or some such00:24
anteayajogo mtreinish np00:24
mordredone sec00:24
clarkbmordred: except neutron output is less useful00:24
clarkband the command name is different00:24
anteayamtreinish: however you want to to have some air time00:24
mtreinishjogo: yeah I don't want to fiat this without dtroyer or sdague's input00:24
*** xchu has joined #openstack-infra00:24
*** melwitt has quit IRC00:24
clarkbkrotscheck: if the UX folks want a list of grievances they should hang out here :)00:24
jogomtreinish: hehe00:24
mtreinishbecause this will be the same discussion for devstack probably pretty soon00:25
fungimordred: if so, i'm all for it. i think tteggel's script used neutronclient and novaclient to map them up and then generate a list of which ones were orphaned, but it was relatively nontrivial to do that way from python00:25
*** gyee has quit IRC00:25
mordredneutron floatingip-list | grep -v 10.0 | awk '{print $2}'00:25
krotscheckmordred: Thanks00:25
fungi(nontrivial compared to a one-liner at any rate)00:25
krotscheckclarkb: context?00:25
*** dimtruck is now known as zz_dimtruck00:25
mordredkrotscheck: the UX folks want to know what it's like to use openstack ...00:26
clarkbkrotscheck: `nova floating-ip-list` and `neutron floatingip-list` do the same thing with two different command names and two different output formats00:26
mordredkrotscheck: we use it and fail constantly00:26
clarkbkrotscheck: its just terrible UX00:26
clarkbmordred: you need to chomp the header00:26
clarkbmordred: or I guess you can just let that floating ip delate of 'id' fail00:26
mordredmeh. it'll just fail :) - that was my plan :)00:27
clarkbmordred: sounds good00:27
krotscheckOh man, I wonder how many have thought about commandline UX.00:27
clarkbmordred: can you add that to the nodepool page on ci.openstack.org?00:27
clarkbkrotscheck: is there another UX?00:27
krotscheckclarkb: Yes, there’s also hipster UX, but you clearly haven’t heard of it yet.00:27
fungikrotscheck: i think that's the one they're always talking about on reddit? with the handlebar mustache?00:28
krotscheckfungi: reddit? That’s way too mainstream.00:29
clarkbfungi: on hacker news00:29
krotscheckSince everyone’s feeling punchy, how do y’all feel about a puppet CR for storyboard? https://review.openstack.org/#/c/98474/00:29
clarkbkrotscheck: I am not thrilled about rabbitmq ;)00:30
krotscheckclarkb: Too mainstream for you Portlandians? :)00:31
clarkbkrotscheck: no, its super heavy weight for local queues and its logs are bad00:31
clarkbkrotscheck: also if you try to HA it split brains are lossy00:32
fungiclarkb: also, your detailed exit codes nodepool prep fix merged, if you wanted to test a manual image update00:32
*** armax has left #openstack-infra00:32
clarkbthe split brain handling is basically "Oh I am joining a larger side of the split let me delete all of my data'00:32
*** wenlock has quit IRC00:32
clarkbfungi: thanks00:32
*** mkerrin has joined #openstack-infra00:33
fungiclarkb: also bug 1355167 is perplexing, if you have any ideas00:33
uvirtbotLaunchpad bug 1355167 in openstack-ci "run-pep8.sh usage now being output" [Critical,In progress] https://launchpad.net/bugs/135516700:33
*** e0ne has quit IRC00:33
*** vermarah_ has quit IRC00:34
*** gokrokve has quit IRC00:35
clarkbfungi: weird00:35
openstackgerritA change was merged to openstack-infra/jenkins-job-builder: add include/exclude regions/users in svn scm (doc and tests)  https://review.openstack.org/10815200:35
fungiclarkb: it's like there are occasionally slaves getting booted from old snapshots in rax-dfw or something00:36
mattoliverauMaybe delete any old snapshots in DFW just to be safe00:37
mattoliverauold images I mean00:37
fungimattoliverau: oh, good idea... i should see if there are older ones nodepool doesn't have indexed00:37
fungimayhaps dfw is just handing us a random old snapshot instead of the one we're asking for on some compute nodes... bad local cache maybe? could that really happen?00:38
clarkbkrotscheck: commented. let me know what you think of those things00:38
phschwartzfungi: I have pinged the admins to take a look to push through cleaning them faster00:38
krotscheckclarkb: Thanks00:38
*** ZZelle has quit IRC00:38
phschwartzfungi: Most likely won't be till tomorrow that they get them all cleaned00:39
mattoliverauwell its run_pep8.. we should be seeing alot more if it was a bad current image.00:39
mattoliverauso maybe00:39
fungiphschwartz: appreciated. i don't think it's currently crippling us the way our floating ip leak in hpcloud is00:39
clarkbfungi: did run-pep8 regress via some other change maybe? I woudl expect a lot of failures if that was the case00:39
fungiclarkb: nope, i checked that first thing00:39
clarkbfungi: cool00:40
fungiclarkb: i even logged into an instance booted from the snapshots there and they had the right (current) version of the script00:40
fungiit's too rare to be all pep8 jobs in rax-dfw, but it's *only* happening in dfw, which is pretty bizarre00:40
krotscheckclarkb: I think I agree with you on the password thing everywhere except for init.pp - I want that file to be as dead-simple to use as possible, i.e. require as little configuration as possible.00:41
fungiand it's been going on since friday, seems00:41
krotscheckclarkb: I’ll figure out the admin thing and get back to you.00:41
clarkbkrotscheck: ok00:42
clarkbkrotscheck: we can do the passwords in a follow up if the admin and erlang things don't need changes00:42
clarkbkrotscheck: since that is how mysql is done I am not super worried about it, but would be good to clean up at some point00:42
*** SumitNaiksatam has quit IRC00:43
openstackgerritSteve Kowalik proposed a change to openstack-infra/config: Add python3-jobs to os-refresh-config  https://review.openstack.org/11249300:44
mordredfungi: all of the floating ips are cleared now00:45
clarkboh also apparently the precise image on hpcloud changed? jesusaurus did your patch merge yet?00:46
*** marun has quit IRC00:46
*** eharney has joined #openstack-infra00:46
*** marun has joined #openstack-infra00:47
jesusaurusclarkb: lemme check00:47
jesusaurusclarkb: https://review.openstack.org/#/c/112923/00:48
jesusaurusnot yet00:48
*** otter768 has joined #openstack-infra00:49
fungimordred: i'll keep an eye out and see if we continue getting any similar tracebacks. hopefully our velocity will pack back up so we can burn through the current pile00:49
mordredkrotscheck, clarkb: so - I don't think there is any way to maintain story/task associations if we import projects at different times00:49
mordredor at least, not a GOOD way00:49
krotscheckmordred: details?00:50
fungimordred: i think if we import infra projects as a lump, that will probably be mostly fine. when it comes to more general openstack projects, i agree they probably have to all happen in one shot00:50
mordredwell, we don't have a place in the storyboard db to store external ids00:50
krotscheckmordred: Why not make the DB ID in storyboard == the ID from launchpad? I seem to recall that we weren’t about to run into ID conflict problems...00:51
mordredkrotscheck: so, if we import, say, openstack-ci project, which has several bugs that have an openstack-ci task and tasks for other projects like nova00:51
clarkbjesusaurus: woo /me grabs gpg file to confirm new image name00:51
mordredand we make a story per bug with one task in each for openstack-ci00:51
mordredwhen we come back and import nova, I don't know that there is a 'good' way to let nova know to attach its task to an existing story00:51
fungiunless our import methodology is far smarter than i'm envisioning, i don't see how we deal with a bug which affects task series for nova, glance and cinder for example, if those projects' bugs are imported at different times. changes could continue to the bug in lp between when nova and cinder got imported, for example00:52
*** ramashri has quit IRC00:52
mordredwell, I think we have to do all of openstack in a lump00:52
mordredit's mainly how we deal with us00:52
mordredand I'm mostly thinking "ignor it"00:52
fungiyeah, that was my feeling too00:52
mordredor - we could keep a little local db hash thing00:53
mordredthat keeps the mapping for next time00:53
fungiright, for infra bugs, i think we pick one or two fairly disconnected projects to import first and live with any separation of tasks which might occur. then do config and other more active (bug-wise) infra projects in one go00:54
fungiwhere the cross-project tasks are more likely to be present in greater numbers00:55
fungiprobably the biggest schism will be between the openstack-ci bugs and devstack/tempest00:56
fungithough we do generally see a lot of openstack-ci+some_server_project bugs too00:56
*** ominakov has joined #openstack-infra00:56
jesusaurusclarkb: youre a better man than i, i just did a `nova list`00:56
jesusauruser, image-list00:57
clarkbjesusaurus: fungi has us checking :)00:57
fungijesusaurus: canonical publishes those signed lists for a reason--may as well not let them feel like it's a wasted effort right? ;)00:57
jesusaurusfungi: fair enough :)00:58
*** mkerrin has quit IRC00:58
*** bdpayne has quit IRC00:58
clarkbfungi: one of my image builds has fialed in puppet due to a pip derp00:59
clarkbfungi: so we should know real soon now if my thing to catch that works00:59
fungithe fact that hpcloud doesn't actually indicate who the partners providing their partner images are makes me a little paranoid. "some dude in the waterfront park provided these images. we think they may be safe"00:59
*** yaguang has joined #openstack-infra00:59
* jesusaurus imagines some back-alley cloud images sold out of a trenchcoat01:00
fungiadmittedly it's far better than before when they just rolled their own and you had to hope they were basically standardized builds01:00
fungibut i appreciate that there's a trust chain now01:00
* anteaya runs away to find some dinner01:01
*** SumitNaiksatam has joined #openstack-infra01:02
*** amcrn has quit IRC01:02
clarkbjesusaurus: fungi ok image file signature checked and verified01:03
clarkbit seems happy but I am not in that chain of trust so ya01:03
*** packet has joined #openstack-infra01:03
clarkbfungi: any chance yo uwant to review https://review.openstack.org/#/c/112923/ before it gets approved?01:03
fungisure--it's probably still hours out from enqueuing anyway01:04
fungilgtm, +1w01:05
clarkbthank you01:06
clarkbfungi: puppet return code checking worked \o/01:06
fungiclarkb: awesome! one more major class of troubleshooting annoyance down!01:07
fungii will definitely not miss the "puppet failed to set our image up, but we're going to use it anyway" issues01:07
gusWho does elastic-recheck reviews?  https://review.openstack.org/#/c/110531/ has sat for a week and a half. Unsure when and whom I should prompt...01:07
clarkbgus: usually jogo mtreinish sdague and a few others. you can always check the gerrit group for membership to see who cores are01:08
*** unicell has joined #openstack-infra01:10
*** packet has quit IRC01:10
gusaha, thanks.01:10
jogogus: usually I try to review them sooner then that01:10
*** eharney has quit IRC01:10
gusjogo: you commented earlier on https://review.openstack.org/#/c/110531/ - does my update address your concerns?01:11
jogogus: just read your response01:12
jogoso the issue here is: we try to keep the queries fairly tight01:12
gusright, I understand I'm probably doing something unusual here.01:12
jogoso that we don't hide other issues01:12
gusThe issue occurs at such a low level and across all projects, that it might surface anywhere.01:13
jogoso we should explicitly say which files (via tags) these line should be in01:13
jogogus: do you really think its a single issue?01:13
jogoif so thats is a different story01:13
*** aysyd has quit IRC01:13
gusThere is an issue with eventlet + mysqldb (the default mysql driver used by sqlalchemy)01:13
jogobut a prelinary look shows only neutro nhits this line (nova works around)01:13
clarkbmordred: pleia2: ok I am going to afk now before I die of heat stroke01:14
clarkbmordred: pleia2: I am curious to see what jeblair thinks of the all auth situation tomorrow01:14
clarkbpleia2: is that on the meeting agenda?01:14
gusyeah, neutron hits this often because they do lots of things inside transactions.01:14
gusbut _anything_ that yields the eventlet inside a transaction may trigger this (eg: writing an entry to a logfile).01:14
jogogus: so this query is a no go actually01:14
jogogus: not becaause it is not a bug01:15
jogogus: but because you hit passing jobs over half the time01:15
gusoh, yeah that sounds bad.01:15
jogogus: in other words this query isn't a good fingerprint of a bug causing a job to fail01:15
jogogus: its a real bug and oen we should fix, but not one that consistently causes jobs to fail01:15
jogoso its easy for this to hide another bug01:16
jogogus: I recomend adding this fingerprint to the bug itself in a comment01:16
jogoso you can track its progress and whatnot01:16
jogobut I don't think the query belongs in e-r unfortunately01:17
gus(and even in the nova catch-and-retry case, my understanding is that it would make the nova process freeze for ~30s)01:17
krotscheckmordred: I’m mostly in the “ignore it” camp as well, unless we want to lay the foundation for upstream task federation and build in origin/remoteId.01:17
*** mkerrin has joined #openstack-infra01:17
jogogus: even if it does, if you use conductors its less of a big deal01:17
jogogus: still bad01:17
gusjogo: ack. I'll drop the e-r query.01:17
jogogus: as a single conductor may freeze for 30 seconds but you should have many01:17
jogogus: thanks for digging into this bug, I would love to see it fixed01:18
gusIt's so hard to get cross-project consensus on what is a simple change :(01:18
jogosigh yeah01:18
*** harlowja_at_home has quit IRC01:18
gusI'm basically making zero progress because (I think) no-one realises it's a common underlying problem that occurs often.01:18
jogogus: what is your current blocker?01:19
gusinstead they debug and whack each specific instances of it.01:19
gusI need mysqlconnector installed on the jenkins environment so we can actually run tests with it.01:19
jogogus: ahh01:20
jogowell this is the room to sort that out in01:20
fungigus: what was the latest update on getting it published to pypi.python.org? i thought i heard the author agreed to start doing that?01:20
gusyep, I was told that by an oracle employee too, but then haven't heard anything since ...  I01:21
fungimordred: you talked to geert (or someobody) about that too right?01:21
gus'll follow up with him.01:21
mordredsudo apt-get install python-mysql.connector01:21
*** xianghuihui has quit IRC01:21
gusmordred: yeah.  it's so simple to fix :(01:21
pleia2clarkb: I think jeblair will still be gone during our meeting tomorrow01:21
mordredthis can all be done in devstack and doesn't need anything from infra01:21
fungimordred: oh, for devstack-based jobs right. i think the other issues were raised about needing it in python 2.6 and 2.7 unit test jobs01:22
mordredare we actually running in to eventlet/mysql issues in unittests?01:22
mordredif it's needed in unittest jobs, then it'sa  non-starter until it's on pypi.01:22
mordredif it's not needed in unittest jobs (which I do not believe it is)01:23
gusI _think_ we aren't.01:23
gusbecause the only unittests I'm aware of that (may) hit mysql are oslo.db's "opportunistic" tests.01:23
fungiit seems unlikely to me, but i don't recall why it was requested unless the proposal was to stop using python-mysqldb at all01:23
mordredthen all we need to do is put ina  path to devstack which installs python-mysql.connnector from apt and then changes the sqlalchemy string01:23
gus.. and oslo.db runs no tests with eventlet as well.01:23
mordredgus: so, I suggest just making a patch to devstack01:24
fungiyeah, i'm with mordred on this... if the only place you need it is in devstack, then... make devstack bring it in from the distro01:24
mordredI, myself, see no problems with unittests using MySQLdb and devstack using python-mysql.connector01:24
gusI have an oslo.db test that reproduces the problem.  That obviously can't go in until then.01:25
gusbut that's a lower priority than getting devstack tests running.01:25
openstackgerritA change was merged to openstack-infra/config: Update and normalize OpenStackID dev DB creds  https://review.openstack.org/11334501:26
fungistable jobs up to icehouse run on precise and would get 0.3.2, while master jobs get 1.1.6 on trusty. presumably that won't pose a problem?01:26
*** xianghui has joined #openstack-infra01:26
*** jyuso has joined #openstack-infra01:27
gusHere's the devstack change:  https://review.openstack.org/#/c/105209/01:27
gusI think sdague's latest comments mean we won't actually be testing mysqlconnector in gate.01:28
*** SumitNaiksatam has quit IRC01:28
mordredgus: I read those differently01:28
mordredI think he's saying that this needs to be behind an option flag01:28
mordredthat defaults to mysqldb01:28
mordredthen, we can set up a job that runs with the flag turned to mysqlconnector01:28
nibalizerkrotscheck: left one comment on your patch01:29
mordredand, after a while, we can decide to change the default01:29
gusah ok.  and then an additional test run that  uses mysqlconnector. ack.01:29
nibalizeri'm +1 on it, just one question i'd like to see answered01:29
*** gokrokve has joined #openstack-infra01:29
*** SumitNaiksatam has joined #openstack-infra01:29
gusuntil then we'll continue to have ~frequent neutron gate failures, just for full disclosure.01:29
mordredbut make it so that devstack knows how to do both01:29
mordredbut it's a fairly large change to make, and even in the face of pain, we've learned that not going through stepwise with data can get us in worse trouble later01:30
*** esker has joined #openstack-infra01:31
*** xianghui has quit IRC01:34
gusmordred: yeah agreed.  In this case however even if the unknown future locks up occasionally then it will still be no worse than the current situation :/01:34
*** xianghuihui has joined #openstack-infra01:34
gusI'll poke Ihar to progress his devstack change.01:34
gusthanks all.01:34
*** SumitNaiksatam has quit IRC01:36
*** yamahata has joined #openstack-infra01:37
*** mkerrin has quit IRC01:37
openstackgerritAishwarya Thangappa proposed a change to openstack-infra/storyboard: Added a subscription_events table & subscription hander  https://review.openstack.org/11301601:38
*** SumitNaiksatam has joined #openstack-infra01:38
*** marun has quit IRC01:38
*** marun has joined #openstack-infra01:40
*** SumitNaiksatam has quit IRC01:40
*** nosnos has joined #openstack-infra01:43
*** nosnos has quit IRC01:45
*** nosnos has joined #openstack-infra01:45
mroddenso what happened with the keyston 2014.1.2 tag...01:46
*** nosnos has quit IRC01:46
clarkbwhy cant we use the other native python lib?01:46
*** nosnos has joined #openstack-infra01:47
clarkbmordred gus ^01:48
mordredclarkb: what other native python lib?01:49
fungiclarkb: because we can't have nice^H^H^H^Hpy3k things01:49
mrodden(what is ^H)01:49
clarkbits native so can be monkey patched01:49
clarkbits MIT licensed iirc01:50
clarkbstrives for mysql-python compat and has a cython implementation01:50
mordredclarkb: I have no clue01:50
*** dims_ has quit IRC01:50
mordredgus: ^^ ?01:50
gusso the issue is eventlet monkeypatching.01:51
gusC libraries  obviously have a much harder time of that.01:51
fungimrodden: it got initially pushed for (then cached as) the commit on the tip of master rather than stable/icehouse, so there's a 2014.1.2.1 now01:51
mordredyah. pymysql is pure python01:51
gusoh, sorry you meant another python library.01:51
gusyeah we could.01:52
gusI went for mysqlconnector since its the "official" mysql python client.  Presumably others would work too.01:52
mroddenfungi: a lot of my jenkins jobs internally are lodged because they re-tagged and its non-fast-forward :(01:52
mroddenbut it sounds like i just need to clean the old tag out01:52
fungimrodden: yeah, that was a spur of the moment attempt at fixing it, which for obvious reasons in hindsight was not a terribly great idea01:54
mroddenno biggie, as long as its just one project01:54
mordredgus: I like geert and all, but 'official' probably doesn't do as much for us as "released on pypi" and "developed somewhere that we can send patches"01:54
fungigus: it was also suggested that pymysql already supports py3k01:55
clarkband pypy01:55
fungiwhich is a nice step in the right direction there too01:55
gusyeah sure.  At the moment, the fear/blockers prevent us switching to pymsql equally.01:56
mroddenooo pypy01:56
gusso it doesn't really matter exactly which alternative we moving to.01:56
fungieven if the places we use it today might not see immediate py3k/pypy compliance overall01:56
mordredgus: I actually think there are less blockers - infra being opposed to the other one because the files are not published is much harder to deal with than a few devs being afraid of change :)01:56
gus.. the speed tests in the blueprint will need to be repeated.01:57
*** mkerrin has joined #openstack-infra01:57
gusmordred: ack.01:57
mordredbtw - I think at this point I'd happily trade a little raw speed for some overall stability01:57
gusI wasn't aware of the importance of pypi issues when I embarked on this particular path ;)01:58
fungiin infra, we get devs over their fear of jumping by pushing them out of the plane01:58
gusmordred: yeah, I'm all for correctness first too.  It's been an interesting learning experience trying to address all the different objections raised to what is at it's core a 15 character change to a connection URL ;)02:00
mordredgus: welcome to my life :)02:00
mordredgus: the competing requirements up in here can kill a person :)02:00
*** weshay has quit IRC02:01
gusand *everyone* wants to be involved in each decision.02:01
mroddenis this a change for all projects connection strings or just one?02:01
mordredmrodden: all02:01
mroddenthat'll be tough02:02
gusdepends who you ask.  we've had both versions of "-1 do the other thing"02:02
*** dims_ has joined #openstack-infra02:02
mordredmrodden: clearly do not listen to me02:02
clarkbno no it should be the same imo02:02
clarkbotherwise now our matrix is even bigger02:02
clarkband it needs to be smaller02:03
mroddenmordred: i dont follow...02:03
mordredclarkb: I agree - but it does not surprise me that someone else disagrees02:03
clarkbif the intent is better gating we use one02:03
mordredmrodden: I was talking like I had an answer - I was clearly only a dude with an opinion02:03
mroddenoh ok02:03
mordredclarkb: I think the people who are advocating for one project at a time or whatever may find themselves crushed under the mighty power of the Gate02:04
mroddeni would think if using a pure mysql driver allows us to get rid of the _retry_on_deadlock crap then it would be a win02:09
mroddenpure python*02:09
*** Ryan_Lane has quit IRC02:13
*** gokrokve has quit IRC02:14
*** flwang_ has joined #openstack-infra02:14
*** cody-somerville has joined #openstack-infra02:16
*** flwang_ has quit IRC02:18
*** bdpayne has joined #openstack-infra02:20
*** homeless_ has quit IRC02:22
*** bhuvan has quit IRC02:23
*** mfer has quit IRC02:30
*** adalbas has quit IRC02:33
*** dims_ has quit IRC02:39
*** dims_ has joined #openstack-infra02:40
*** cody-somerville has quit IRC02:41
*** david-lyle has joined #openstack-infra02:44
*** dims_ has quit IRC02:44
*** sdake_ has quit IRC02:47
*** pcrews has quit IRC02:49
chmouelmordred: i think the swift functional tests run on a normal devstack (with others openstack services enabled)02:50
openstackgerritSteve Martinelli proposed a change to openstack/requirements: Add a lower limit to oslosphinx  https://review.openstack.org/11342302:55
*** yanghe has joined #openstack-infra03:00
*** dims_ has joined #openstack-infra03:00
*** yanghe has left #openstack-infra03:00
*** esker has quit IRC03:01
*** dims_ has quit IRC03:01
*** yanghe has joined #openstack-infra03:01
*** dims_ has joined #openstack-infra03:02
*** laxity_ has quit IRC03:02
*** laxity_ has joined #openstack-infra03:03
*** jgrimm has joined #openstack-infra03:05
*** lttrl has quit IRC03:06
*** dims_ has quit IRC03:06
*** mkerrin has quit IRC03:06
*** funzo has joined #openstack-infra03:10
*** daya_k has joined #openstack-infra03:10
*** ArxCruz has quit IRC03:12
*** jyuso has quit IRC03:12
*** jyuso has joined #openstack-infra03:12
openstackgerritA change was merged to openstack-infra/config: HPCloud has deprecated another 12.04 image  https://review.openstack.org/11292303:18
*** nosnos has quit IRC03:22
*** yamamot__ has joined #openstack-infra03:26
*** AlexF_ has joined #openstack-infra03:27
*** yamamoto has quit IRC03:29
*** sdake_ has joined #openstack-infra03:29
*** dims_ has joined #openstack-infra03:32
*** rwsu has quit IRC03:38
*** xchu has quit IRC03:38
*** SumitNaiksatam has joined #openstack-infra03:38
*** dims_ has quit IRC03:39
mordredchmouel: nod03:40
openstackgerritSteve Kowalik proposed a change to openstack-infra/config: Add python3-jobs to os-refresh-config  https://review.openstack.org/11249303:41
*** crc32 has quit IRC03:41
*** mkerrin has joined #openstack-infra03:42
*** KimJ has quit IRC03:42
mordredStevenK: ^^ woot03:44
mordredStevenK: also, I'm running a launchpadlib program against your website and it's slow03:45
*** otter768 has quit IRC03:45
mordredStevenK: https://gist.github.com/emonty/4ca365d4bcd284ab927403:45
*** sdake_ has quit IRC03:49
lifelessmordred: so did you review my patch?03:49
*** mkerrin has quit IRC03:49
*** mkerrin has joined #openstack-infra03:50
*** chandankumar has joined #openstack-infra03:55
*** rfolco has quit IRC03:58
StevenKmordred: "My" website? :-)04:01
openstackgerritA change was merged to openstack/requirements: Remove unneeded definitions of Python Source Code Encoding  https://review.openstack.org/11091204:02
*** david-lyle has quit IRC04:08
*** david-lyle has joined #openstack-infra04:08
*** garyh has quit IRC04:09
*** david-lyle has quit IRC04:13
*** HenryG is now known as HenryG_afk04:14
*** flwang_ has joined #openstack-infra04:15
*** marun has quit IRC04:17
*** flwang_ has quit IRC04:19
*** chandankumar has quit IRC04:20
*** nosnos has joined #openstack-infra04:25
*** yfried has quit IRC04:29
*** AlexF_ has quit IRC04:30
StevenKmordred: One suggestion. Set bug = task.bug and then use that, rather than task.bug.owner and so on04:33
StevenKmordred: But it is always going to slow, because lots of data, REST and LP04:34
*** MarkAtwood has joined #openstack-infra04:38
*** garyh has joined #openstack-infra04:40
*** praneshp has quit IRC04:47
*** praneshp_ has joined #openstack-infra04:47
anteayayay done all the acl file change patchsets and new-project patchsets I could find, that weren't wip04:53
*** ArxCruz has joined #openstack-infra04:56
*** ildikov has quit IRC05:03
*** camunoz_ has joined #openstack-infra05:05
*** camunoz_ has quit IRC05:06
*** camunoz__ has joined #openstack-infra05:06
*** bcrochet has quit IRC05:06
*** jamielen- has joined #openstack-infra05:07
*** Longgeek has joined #openstack-infra05:07
*** ianw has quit IRC05:08
*** ianw has joined #openstack-infra05:08
*** jamielennox has quit IRC05:08
*** camunoz has quit IRC05:08
*** bcrochet has joined #openstack-infra05:08
*** maxbit_ has quit IRC05:08
*** camunoz has joined #openstack-infra05:09
*** shashankhegde has joined #openstack-infra05:09
*** camunoz__ has quit IRC05:10
*** enikanorov has quit IRC05:10
*** enikanorov has joined #openstack-infra05:11
*** gema has joined #openstack-infra05:12
*** asettle has quit IRC05:12
*** asettle has joined #openstack-infra05:13
*** jaypipes has joined #openstack-infra05:14
*** gema_ has quit IRC05:15
*** jamielen- is now known as jamielennox05:15
*** chandankumar has joined #openstack-infra05:16
openstackgerritA change was merged to openstack-infra/zuul: Bump Babel requirement to >=1.0  https://review.openstack.org/11257405:16
*** chandankumar has quit IRC05:17
*** chandankumar has joined #openstack-infra05:17
*** gema_ has joined #openstack-infra05:17
*** gema has quit IRC05:18
*** gema_ has quit IRC05:22
*** gema has joined #openstack-infra05:22
*** otherwiseguy has quit IRC05:22
*** dims_ has joined #openstack-infra05:23
*** otherwiseguy has joined #openstack-infra05:23
*** AlexF_ has joined #openstack-infra05:25
*** gema has quit IRC05:27
*** gema has joined #openstack-infra05:27
*** dims_ has quit IRC05:29
*** gema_ has joined #openstack-infra05:32
*** bdpayne has quit IRC05:33
*** gema has quit IRC05:35
*** bdpayne has joined #openstack-infra05:35
*** k4n0 has joined #openstack-infra05:37
*** gema has joined #openstack-infra05:37
*** gema_ has quit IRC05:38
openstackgerritA change was merged to openstack-infra/devstack-gate: address bashate style violations  https://review.openstack.org/10126805:40
*** ianw has quit IRC05:42
*** ianw has joined #openstack-infra05:42
*** camunoz__ has joined #openstack-infra05:42
*** ianw has quit IRC05:43
*** ianw has joined #openstack-infra05:43
*** ianw has quit IRC05:43
*** jamielen| has joined #openstack-infra05:44
*** ianw has joined #openstack-infra05:44
*** ianw has quit IRC05:44
*** jamielen| has quit IRC05:44
*** yfried has joined #openstack-infra05:44
*** ianw has joined #openstack-infra05:44
*** bcrochet has quit IRC05:44
*** ianw has quit IRC05:45
*** ianw has joined #openstack-infra05:45
*** jamielen| has joined #openstack-infra05:45
*** jamielen| has quit IRC05:45
*** ianw has quit IRC05:46
*** jamielennox has quit IRC05:46
*** ianw has joined #openstack-infra05:46
*** camunoz has quit IRC05:46
*** camunoz has joined #openstack-infra05:46
*** jamielennox has joined #openstack-infra05:47
*** bcrochet has joined #openstack-infra05:48
*** ianw has quit IRC05:50
*** ianw has joined #openstack-infra05:51
openstackgerritFrançois Magimel proposed a change to openstack-infra/config: Added python-cloudkittyclient to stackforge projects  https://review.openstack.org/11250705:52
*** otherwiseguy has quit IRC05:53
*** dtantsur|afk is now known as dtantsur05:54
*** ildikov has joined #openstack-infra05:56
*** bdpayne has quit IRC05:58
*** e0ne has joined #openstack-infra05:58
openstackgerritIlia Meerovich proposed a change to openstack-infra/jenkins-job-builder: Implements wrapper for ownership plugin. This wrapper sets "Inject ownership variables into environment"  https://review.openstack.org/11334805:59
*** e0ne has quit IRC06:00
*** camunoz_ has joined #openstack-infra06:00
*** ianw has quit IRC06:01
*** abhishekk has joined #openstack-infra06:01
*** kashyap has joined #openstack-infra06:01
*** camunoz__ has joined #openstack-infra06:01
*** jamielen- has joined #openstack-infra06:01
*** reed has quit IRC06:01
*** reed has joined #openstack-infra06:02
*** loki184 has joined #openstack-infra06:02
*** e0ne has joined #openstack-infra06:02
*** bcrochet has quit IRC06:02
*** dkliban_afk has quit IRC06:03
*** portante has quit IRC06:03
*** markmc has joined #openstack-infra06:03
*** camunoz has quit IRC06:04
*** jamielennox has quit IRC06:05
*** camunoz_ has quit IRC06:05
*** portante has joined #openstack-infra06:06
*** bcrochet has joined #openstack-infra06:06
*** dkliban_afk has joined #openstack-infra06:06
*** SumitNaiksatam has quit IRC06:07
*** evgenyf has joined #openstack-infra06:07
*** shashankhegde has quit IRC06:08
*** SumitNaiksatam has joined #openstack-infra06:08
*** yamamoto has joined #openstack-infra06:09
*** yamamo___ has joined #openstack-infra06:10
*** _nadya_ has joined #openstack-infra06:10
*** e0ne has quit IRC06:11
*** yamamot__ has quit IRC06:12
*** _nadya_ has quit IRC06:14
*** yamamoto has quit IRC06:14
*** AlexF_ has quit IRC06:14
*** ArxCruz has quit IRC06:15
*** AlexF_ has joined #openstack-infra06:15
*** flwang_ has joined #openstack-infra06:16
*** _nadya_ has joined #openstack-infra06:16
*** e0ne has joined #openstack-infra06:16
*** oomichi has quit IRC06:16
*** e0ne has quit IRC06:17
*** Longgeek has quit IRC06:19
*** flwang_ has quit IRC06:20
*** Longgeek has joined #openstack-infra06:21
*** camunoz__ has quit IRC06:25
*** yanghe has quit IRC06:27
*** ildikov has quit IRC06:27
*** yamamo___ has quit IRC06:28
openstackgerritlokesh s proposed a change to openstack-infra/config: Adding windows disk-image builder project  https://review.openstack.org/11056206:29
*** yanghe has joined #openstack-infra06:30
*** linkid has joined #openstack-infra06:30
*** flaper87|afk is now known as flaper8706:33
*** p5ntangle has joined #openstack-infra06:33
*** stevemar has quit IRC06:34
*** e0ne has joined #openstack-infra06:35
*** camunoz__ has joined #openstack-infra06:38
*** e0ne has quit IRC06:40
*** ildikov has joined #openstack-infra06:40
*** e0ne has joined #openstack-infra06:44
*** e0ne has quit IRC06:46
*** denis_makogon_ has joined #openstack-infra06:50
*** e0ne has joined #openstack-infra06:51
*** linkid has left #openstack-infra06:51
*** andreykurilin_ has joined #openstack-infra06:51
*** AlexF_ has quit IRC06:51
*** unicell has quit IRC06:51
*** unicell has joined #openstack-infra06:52
*** abhishekk has quit IRC06:54
*** abhishekk has joined #openstack-infra06:54
*** e0ne has quit IRC06:56
*** yamamoto has joined #openstack-infra06:58
openstackgerritKanagaraj Manickam proposed a change to openstack-infra/config: namos - openstack infra device and service manager  https://review.openstack.org/10949406:59
*** MarkAtwood has quit IRC06:59
*** e0ne has joined #openstack-infra07:00
*** e0ne has quit IRC07:04
*** asettle has quit IRC07:05
*** _nadya_ has quit IRC07:06
*** trinaths has joined #openstack-infra07:07
*** camunoz__ has quit IRC07:07
*** e0ne has joined #openstack-infra07:08
*** yamamoto has quit IRC07:10
*** e0ne has quit IRC07:10
*** rdopieralski has joined #openstack-infra07:11
*** e0ne has joined #openstack-infra07:14
*** rcarrillocruz has joined #openstack-infra07:14
*** rcarrill` has quit IRC07:16
*** e0ne has quit IRC07:16
*** e0ne has joined #openstack-infra07:19
*** jcoufal has joined #openstack-infra07:20
*** jgallard has joined #openstack-infra07:20
*** e0ne has quit IRC07:22
*** jgallard has quit IRC07:23
*** jgallard has joined #openstack-infra07:23
*** e0ne has joined #openstack-infra07:26
*** sballe has joined #openstack-infra07:27
*** salv-orlando has quit IRC07:27
*** pblaho has joined #openstack-infra07:28
*** yamamoto has joined #openstack-infra07:28
*** e0ne has quit IRC07:29
*** afazekas has joined #openstack-infra07:31
*** yamamoto has quit IRC07:31
*** Guest35108 has joined #openstack-infra07:34
*** e0ne has joined #openstack-infra07:34
*** _nadya_ has joined #openstack-infra07:35
*** jistr has joined #openstack-infra07:36
*** e0ne has quit IRC07:36
*** andreykurilin_ has quit IRC07:36
*** linkid has joined #openstack-infra07:36
*** trinaths has quit IRC07:38
*** e0ne has joined #openstack-infra07:40
*** trinaths has joined #openstack-infra07:42
*** e0ne has quit IRC07:42
*** e0ne has joined #openstack-infra07:46
*** e0ne has quit IRC07:48
*** AlexF_ has joined #openstack-infra07:49
*** mrda is now known as mrda-away07:49
*** denis_makogon_ has quit IRC07:49
*** praneshp_ has quit IRC07:51
*** flwang_ has joined #openstack-infra07:51
*** e0ne has joined #openstack-infra07:52
*** praneshp has joined #openstack-infra07:52
*** jpich has joined #openstack-infra07:53
*** s0nea has quit IRC07:54
*** Krast has joined #openstack-infra07:55
*** e0ne has quit IRC07:58
*** fifieldt__ is now known as fifieldt07:59
*** Krast has quit IRC08:02
*** yamamoto has joined #openstack-infra08:02
*** e0ne has joined #openstack-infra08:02
*** yanghe has quit IRC08:03
*** gema has quit IRC08:04
*** AlexF_ has quit IRC08:04
*** gema has joined #openstack-infra08:04
*** dosaboy_ has joined #openstack-infra08:05
*** e0ne has quit IRC08:06
*** yamamoto has quit IRC08:06
*** Krast has joined #openstack-infra08:06
*** e0ne has joined #openstack-infra08:10
*** e0ne has quit IRC08:13
*** jlibosva has joined #openstack-infra08:14
*** derekh has joined #openstack-infra08:15
*** ihrachyshka has joined #openstack-infra08:16
*** e0ne has joined #openstack-infra08:17
*** zehicle_at_dell has joined #openstack-infra08:18
*** praneshp has quit IRC08:18
*** yaguang has quit IRC08:19
*** yaguang has joined #openstack-infra08:19
*** ihrachyshka has quit IRC08:20
*** ihrachyshka has joined #openstack-infra08:21
*** andreaf has quit IRC08:23
*** andreaf_ has joined #openstack-infra08:23
*** jamielen- is now known as jamielennox|away08:24
*** yanghe has joined #openstack-infra08:25
*** dims_ has joined #openstack-infra08:26
*** amuller has joined #openstack-infra08:27
*** yfried is now known as yfried_afk08:27
*** AlexF_ has joined #openstack-infra08:27
*** yamamoto has joined #openstack-infra08:28
*** p5ntangle has quit IRC08:29
*** p5ntangle has joined #openstack-infra08:29
*** yanghe has quit IRC08:29
*** dims_ has quit IRC08:31
*** gema has quit IRC08:33
*** yamamoto has quit IRC08:33
*** gema has joined #openstack-infra08:33
*** p5ntangle has quit IRC08:34
*** e0ne has quit IRC08:35
*** abhishekk has quit IRC08:36
*** loki184 has quit IRC08:36
*** dosaboy_ has quit IRC08:36
*** gema has quit IRC08:36
*** abhishekk has joined #openstack-infra08:37
*** andreaf has joined #openstack-infra08:37
*** e0ne has joined #openstack-infra08:39
*** Krast has quit IRC08:40
*** andreaf_ has quit IRC08:40
*** Krast has joined #openstack-infra08:40
*** AlexF_ has quit IRC08:41
*** kashyap has quit IRC08:41
*** andreaf_ has joined #openstack-infra08:43
*** gema has joined #openstack-infra08:45
*** andreaf has quit IRC08:45
*** dtantsur is now known as dtantsur|lunch08:45
*** dosaboy_ has joined #openstack-infra08:46
*** dosaboy_ is now known as dosaboy-alt08:46
*** penguinRaider has quit IRC08:46
*** abhishekk has quit IRC08:46
*** dosaboy-alt has quit IRC08:47
*** abhishekk has joined #openstack-infra08:47
*** zehicle_at_dell has quit IRC08:48
*** zehicle_at_dell has joined #openstack-infra08:48
*** penguinRaider has joined #openstack-infra08:49
*** e0ne has quit IRC08:52
openstackgerritA change was merged to openstack-infra/storyboard: Add task_id to Timeline Event info  https://review.openstack.org/10675908:55
*** AlexF_ has joined #openstack-infra08:55
*** e0ne has joined #openstack-infra08:57
*** e0ne has quit IRC08:59
openstackgerritTimur Sufiev proposed a change to openstack-infra/config: Create merlin repo  https://review.openstack.org/11032509:03
*** e0ne has joined #openstack-infra09:03
*** AlexF_ has quit IRC09:07
*** mkerrin has quit IRC09:07
*** e0ne has quit IRC09:07
*** AlexF_ has joined #openstack-infra09:08
*** jyuso has quit IRC09:08
*** e0ne has joined #openstack-infra09:13
*** laxity has joined #openstack-infra09:15
*** e0ne has quit IRC09:15
*** laxity_ has quit IRC09:15
*** zehicle_at_dell has quit IRC09:17
*** niedbalski has quit IRC09:18
openstackgerritA change was merged to openstack-infra/config: sahara-ptl group should be able to push 2014.1.*  https://review.openstack.org/11327409:19
*** e0ne has joined #openstack-infra09:19
*** dosaboy has quit IRC09:20
*** wolsen has quit IRC09:20
openstackgerritMike Heald proposed a change to openstack-infra/nodepool: Add stdout and stderr to exception when ready script fails  https://review.openstack.org/11328709:22
*** dosaboy has joined #openstack-infra09:23
*** niedbalski has joined #openstack-infra09:24
*** e0ne has quit IRC09:24
*** wolsen has joined #openstack-infra09:25
*** e0ne has joined #openstack-infra09:26
*** evgenyf has quit IRC09:27
*** yamamoto has joined #openstack-infra09:29
*** e0ne has quit IRC09:30
*** AlexF_ has quit IRC09:32
*** evgenyf has joined #openstack-infra09:32
*** yamamoto has quit IRC09:34
openstackgerritIlya Shakhat proposed a change to openstack-infra/config: Run stackalytics in infra  https://review.openstack.org/9865609:34
*** e0ne has joined #openstack-infra09:34
*** loki184 has joined #openstack-infra09:36
openstackgerritafazekas proposed a change to openstack-infra/config: Do not fail the install_puppet.sh on 2th run  https://review.openstack.org/10500209:36
*** AlexF_ has joined #openstack-infra09:36
*** pelix has joined #openstack-infra09:38
*** e0ne has quit IRC09:39
*** andreaf has joined #openstack-infra09:40
*** andreaf has quit IRC09:40
*** andreaf has joined #openstack-infra09:41
*** andreaf_ has quit IRC09:42
*** e0ne has joined #openstack-infra09:44
DinaBelovagood morning, infra folks! do you know if centos-6 image was updated on the gate? in ceilo we have failures with py26 jobs - like this one http://logs.openstack.org/71/112771/2/check/gate-ceilometer-python26/d6a7161/console.html - "Could not find mongod command"...09:44
DinaBelovaSergeyLukjanov - may you take a look on this moment? ^^09:45
DinaBelovawe have blocked gate ATM09:45
*** andreaf has quit IRC09:46
*** erw has quit IRC09:46
*** serverascode has quit IRC09:47
DinaBelovajhesketh, may you looks as well sir? ^^09:47
*** zhiyan has quit IRC09:47
*** simonmcc_ has quit IRC09:48
*** serverascode has joined #openstack-infra09:48
*** Alec-A2 has quit IRC09:49
*** willroberts has quit IRC09:49
*** ctracey has quit IRC09:49
*** Ng has quit IRC09:49
*** skath has quit IRC09:50
*** sweston has quit IRC09:50
openstackgerritMike Heald proposed a change to openstack-infra/nodepool: Add stdout and stderr to exception when ready script fails  https://review.openstack.org/11328709:50
*** marcusvrn has quit IRC09:51
*** jraim has quit IRC09:51
*** Alec-A2 has joined #openstack-infra09:51
*** erw has joined #openstack-infra09:52
*** zhiyan has joined #openstack-infra09:52
*** nimrodsun has quit IRC09:52
*** willroberts has joined #openstack-infra09:52
*** simonmcc_ has joined #openstack-infra09:53
*** Alex_Gaynor has quit IRC09:53
*** serverascode has quit IRC09:53
*** ctracey has joined #openstack-infra09:53
*** erlon has quit IRC09:53
*** sweston has joined #openstack-infra09:54
*** skath has joined #openstack-infra09:54
*** jraim has joined #openstack-infra09:54
*** marcusvrn has joined #openstack-infra09:54
*** unicell has quit IRC09:55
*** flaviof_zzz is now known as flaviof09:55
*** zhiyan has quit IRC09:57
*** erw has quit IRC09:57
*** nimrodsun has joined #openstack-infra09:57
*** rlandy has joined #openstack-infra09:58
*** Alex_Gaynor has joined #openstack-infra09:59
*** Guest35108 has quit IRC09:59
*** AlexF_ has quit IRC09:59
*** erlon has joined #openstack-infra09:59
*** zhiyan has joined #openstack-infra09:59
*** serverascode has joined #openstack-infra09:59
*** Ng has joined #openstack-infra10:00
*** erw has joined #openstack-infra10:01
*** andreaf has joined #openstack-infra10:04
*** e0ne has quit IRC10:06
DinaBelovadear infra folks, please pay attention to the problem I've listed above ^^ Ceilometer gate is actually blocked and we don't know the reason10:08
*** yaguang has quit IRC10:10
*** mkerrin has joined #openstack-infra10:11
*** alexpilotti has joined #openstack-infra10:12
*** jgallard has quit IRC10:12
*** ianw has joined #openstack-infra10:14
openstackgerritRadomir Dopieralski proposed a change to openstack/requirements: Update xstatic-jquery-ui for Horizon  https://review.openstack.org/11318410:15
*** dims_ has joined #openstack-infra10:15
d0ugalWe currently CI against sqlite for Tuskar, I'd like to include a job for mysql. Is this the right place to ask? Where do I start? Is there a doc I should read?10:19
d0ugalMy Google fu is failing me thus far.10:19
*** dims_ has quit IRC10:20
*** AlexF_ has joined #openstack-infra10:21
*** yaguang has joined #openstack-infra10:25
*** yamamoto has joined #openstack-infra10:29
*** p5ntangle has joined #openstack-infra10:30
*** AlexF_ has quit IRC10:31
*** yamamoto has quit IRC10:33
*** AlexF_ has joined #openstack-infra10:34
*** andreaf_ has joined #openstack-infra10:35
*** yfried_afk is now known as yfried10:35
*** andreaf has quit IRC10:37
*** rfolco has joined #openstack-infra10:37
*** daya_k_ has joined #openstack-infra10:38
*** daya_k has quit IRC10:38
*** daya_k_ is now known as daya_k10:39
*** daya_k has quit IRC10:43
*** andreaf_ has quit IRC10:43
*** sballe has quit IRC10:46
*** Krast has quit IRC10:50
*** ZZelle has joined #openstack-infra10:52
*** jedimike has quit IRC10:56
*** reed has quit IRC10:58
*** loki184 has quit IRC11:00
*** andreaf has joined #openstack-infra11:01
*** jedimike has joined #openstack-infra11:01
*** unicell has joined #openstack-infra11:04
*** andreaf_ has joined #openstack-infra11:04
*** andreaf has quit IRC11:08
*** yamamoto has joined #openstack-infra11:10
*** Longgeek has quit IRC11:10
*** Longgeek has joined #openstack-infra11:10
*** yamahata has quit IRC11:11
*** dims_ has joined #openstack-infra11:12
*** Longgeek has quit IRC11:14
*** yaguang has quit IRC11:15
*** zehicle_at_dell has joined #openstack-infra11:18
openstackgerritMike Heald proposed a change to openstack-infra/nodepool: Add stdout and stderr to exception when ready script fails  https://review.openstack.org/11328711:18
*** ArxCruz has joined #openstack-infra11:20
*** hashar has joined #openstack-infra11:20
*** zehicle_at_dell has quit IRC11:22
*** flaviof is now known as flaviof_zzz11:23
*** dims_ is now known as dims11:28
*** willroberts has quit IRC11:33
*** jamielennox|away has quit IRC11:34
*** Longgeek has joined #openstack-infra11:34
*** willroberts has joined #openstack-infra11:35
*** weshay has joined #openstack-infra11:36
*** jamielennox|away has joined #openstack-infra11:37
*** dims has quit IRC11:37
*** p5ntangle has quit IRC11:38
*** p5ntangle has joined #openstack-infra11:38
*** AlexF_ has quit IRC11:40
*** p5ntangle has quit IRC11:41
*** p5ntangle has joined #openstack-infra11:41
*** dtantsur|lunch is now known as dtantsur11:44
*** andreaf_ has quit IRC11:45
*** andreaf_ has joined #openstack-infra11:45
*** p5ntangle has quit IRC11:50
*** mbacchi has joined #openstack-infra11:50
*** p5ntangle has joined #openstack-infra11:50
*** flwang_ has quit IRC11:58
*** s0nea has joined #openstack-infra11:59
*** AlexF_ has joined #openstack-infra12:04
*** jgallard has joined #openstack-infra12:06
*** ominakov has quit IRC12:07
*** prad has joined #openstack-infra12:07
*** trinaths has quit IRC12:10
*** dprince has joined #openstack-infra12:10
*** salv-orlando has joined #openstack-infra12:10
*** p5ntangle has quit IRC12:14
*** adalbas has joined #openstack-infra12:16
*** AlexF_ has quit IRC12:17
SergeyLukjanovfungi, clarkb, mordred, yeah, it was me (re non-commited change to hierra)12:21
fungiSergeyLukjanov: no worries--i forgot the first few times to12:22
SergeyLukjanovfungi, /me going to look how it should looks like12:23
fungiSergeyLukjanov: we basically all commit changes as root and then just sign one-liner commit messages12:23
*** dkranz has joined #openstack-infra12:24
fungiSergeyLukjanov: it's mainly so that we know who changed what roughly when, to make it easy to undo later if needed12:24
*** ominakov has joined #openstack-infra12:24
SergeyLukjanovfungi, yeah, I see git history12:24
SergeyLukjanovfungi, will do it next time ;)12:25
*** ihrachyshka_ has joined #openstack-infra12:25
openstackgerritPeter Belanyi proposed a change to openstack-infra/config: Change selenium and jshint to voting for tuskar-ui  https://review.openstack.org/11351012:25
*** aysyd has joined #openstack-infra12:28
*** ihrachyshka has quit IRC12:28
*** yamamoto has quit IRC12:31
openstackgerritFlavio Percoco proposed a change to openstack-infra/config: Rename Marconi to Zaqar  https://review.openstack.org/11124412:33
*** maxbit has joined #openstack-infra12:41
*** miqui has joined #openstack-infra12:44
*** ominakov has quit IRC12:46
*** AlexF_ has joined #openstack-infra12:48
*** dims has joined #openstack-infra12:48
*** dims has quit IRC12:49
*** dims has joined #openstack-infra12:49
*** eglynn has joined #openstack-infra12:50
*** dims is now known as Guest7519112:50
eglynngreeting gentle-people!12:50
eglynnFYI a major snafu in Fedora-land has killed the ceilometer py26 gate for the last day12:53
eglynnmongodb packages were retired from EPEL6 by mistake (couchdb was the intended victim)12:53
*** k4n0 has quit IRC12:53
*** vhoward has left #openstack-infra12:53
eglynnfor details see ...12:54
eglynnthe bad commit has been reverted and the version re-tagged, pbrady is actively chasing +1s12:54
eglynnbut it may take a while for the package restore to propogate thru' the suasage machine ... in the meantime, ceilo gating is blocked12:55
* eglynn finally cuts to the chase ...12:55
*** jistr has quit IRC12:56
eglynnas a stop-gap, would it be possible to install the RPM direct from the Fedora package build system (koji) onto the centos-6 jenkins slaves?12:56
eglynne.g. yum install -y http://kojipkgs.fedoraproject.org/packages/mongodb/2.4.6/1.el6/x86_64/mongodb-server-2.4.6-1.el6.x86_64.rpm12:56
*** jistr has joined #openstack-infra12:57
*** Guest35108 has joined #openstack-infra12:57
*** heyongli has joined #openstack-infra12:57
eglynnfungi: ^^^ any thoughts on the above question about working around the breakage in EPEL6?12:58
*** AlexF_ has quit IRC13:00
*** jcoufal has quit IRC13:00
eglynnSergeyLukjanov: ^^^ you might know about the feasibility of the EPEL6 work-around proposed above?13:01
*** e0ne has joined #openstack-infra13:01
*** HenryG_afk is now known as HenryG13:02
fungieglynn: i'd rather we didn't work around breakage if a fix is already percolating. by the time we get the change tested and merged to do that, it will probably no longer be necessary13:02
*** gentlesaurus is now known as annegentle13:02
eglynnfungi: OK, I'll try to getting a better estimate on when the fix will propogate through13:03
*** lttrl has joined #openstack-infra13:03
eglynnfungi: the first indication was "a few days" which is obviously waaay too long13:03
*** jcoufal has joined #openstack-infra13:03
fungieglynn: keep in mind that everyone in the world using that is probably broken, which likely includes many/most of our downstream users, so having it impact our development process similarly seems minor by comparison13:04
*** julim has joined #openstack-infra13:04
*** pblaho has quit IRC13:04
*** juzuluag has joined #openstack-infra13:05
*** pblaho has joined #openstack-infra13:05
*** dustins has joined #openstack-infra13:05
eglynnfungi: always look on the bright side! ... as a wise Python once said/sang ;)13:06
fungithough i do have to say, in a perverse sort of way, it's hilarious that the epel maintainers would confuse mongodb with couchdb ;)13:06
*** ominakov has joined #openstack-infra13:06
*** julim has quit IRC13:06
fungiso, yeah, if they think it's going to be days before epel consumers will have access to mongodb again then we can try to shoehorn in a workaround. if they expect it to be back in sometime today on the other hand, that's likely to just be wasted effort on our part13:07
eglynnfungi: well, one DB is much the same as another, right! ... they all just store stuff :)13:07
eglynnfungi: cool, got it ... I'll try to get a better fix on a realistic ETA13:08
*** bknudson has quit IRC13:09
*** AlexF_ has joined #openstack-infra13:10
*** flaper87 is now known as flaper87|afk13:10
*** dkranz has quit IRC13:11
eglynnfungi: ildikov just had a eureka moment, how about we set ceilo/py26 as non-voting as the stop-gap?13:12
eglynnfungi: (... seeing as we still get the mongo coverage via py27 on Trusty)13:12
*** nosnos has quit IRC13:13
*** shashankhegde has joined #openstack-infra13:13
ildikovfungi: eglynn: also assuming that this issue will be fixed in more a shorter than longer amount of time, so most prolly we will not ruin the whole python 2.6 world of Ceilometer in that time frame if we choose this option13:14
*** chat6062 has joined #openstack-infra13:15
chat6062hey all.  an i ask here a question about Jenkins job builder?13:16
eglynnfungi: from the horse's mouth ... "I currently do not know when the next push will occur, but I guess within the next 24 hours, since everyone should be back from Flock now."13:16
*** jgallard has quit IRC13:17
*** mrmartin has joined #openstack-infra13:20
*** achuprin has quit IRC13:20
fungieglynn: seems fine, i'll propose the config change for you real quick13:20
eglynnfungi: excellent thank you sir!13:21
fungichat6062: you can ask... most of the developers who wrote it hang out here, as well as a lot of users13:21
*** changbl has quit IRC13:21
*** AlexF_ has quit IRC13:21
fungichat6062: though depending on the specifics of your question, it may be a few hours before people with the right knowledge to answer definitively are awake and paying attention13:22
chat6062cool. well, when i run Jenkins-jobs script with the test flag there is no output. what can cause this?13:23
*** julim has joined #openstack-infra13:24
*** MarkAtwood has joined #openstack-infra13:25
*** pcrews has joined #openstack-infra13:26
*** ptoohill has quit IRC13:28
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Temporarily ignore Ceilometer Python 2.6 job  https://review.openstack.org/11352113:28
fungieglynn: so there13:28
eglynnfungi: give that man a chocolate fish! :)13:29
eglynnfungi: thank you sir!13:29
*** mjturek has joined #openstack-infra13:29
*** evgenyf has quit IRC13:29
fungithat will ignore the job result once it merges, but you probably want to consider disabling the mongodb-specific tests in your testsuite instead and having us revert the non-voting change instead if it persists13:29
eglynncool, will do13:30
* eglynn hopes for a quick resolution to the EPEL6 snafu13:30
*** jgallard has joined #openstack-infra13:30
fungieglynn: also, are mongodb tests running under trusty for you now?13:30
*** bknudson has joined #openstack-infra13:30
*** AlexF_ has joined #openstack-infra13:31
eglynnfungi: yes, on ceilo/py27 and also via an experimental Tempest job13:31
eglynn(we want to upgrade that experimental job to full voting status, see ... http://lists.openstack.org/pipermail/openstack-dev/2014-August/042505.html )13:31
fungieglynn: okay, so that probably wouldn't be the end of the world if you had to stop testing it on centos6 for a while, since you have mongodb coverage voting on trusty13:32
*** yamamoto has joined #openstack-infra13:32
funginot ideal of course, but not terrible13:32
eglynnfungi: yeap, exactly, thanks!13:32
*** esker has joined #openstack-infra13:32
mjturekhey all, question. so posting 'recheck' as a comment (even as a substring in the comment) causes a Jenkins recheck. Is this intended? For example if I post 'recheck hyper-v' it will also recheck Jenkins13:33
*** ptoohill has joined #openstack-infra13:33
alexpilottimjturek: we just repalced the Hyepr-V check w recheck-hyper-v because of this13:33
mjturekalexpilotti, I'll need to confirm but I think 'recheck-power-kvm' also triggered Jenkins for me.13:35
*** wenlock has joined #openstack-infra13:35
*** rcarrill` has joined #openstack-infra13:35
*** yamamoto has quit IRC13:36
*** dkranz has joined #openstack-infra13:37
*** ociuhandu has joined #openstack-infra13:37
*** rcarrillocruz has quit IRC13:38
*** ptoohill has quit IRC13:39
*** maxbit_ has joined #openstack-infra13:40
alexpilottiociuhandu: ^^13:40
fungimjturek: yep, intended. we are phasing out bug tracking via recheck comments (because elastic-recheck patterns are far more reliable) and the first phases of that are to support "\nre(check|verify)( .*)?" to rerun jobs in upstream ci13:40
*** reaper has joined #openstack-infra13:40
fungier, actually, the regex is currently "(?i)^(Patch Set [0-9]+:)?( [\w\\+-]*)*(\n\n)?\s*(recheck|reverify)"13:41
*** reaper has quit IRC13:41
fungiso yes, even "recheck-power-kvm" will do the same13:41
ociuhandufungi: would it be ok if we update the trigger for Hyper-V CI to “hyper-v-recheck” for instance?13:41
*** mriedem has joined #openstack-infra13:42
*** maxbit has quit IRC13:42
*** esker has quit IRC13:42
*** homeless has joined #openstack-infra13:42
fungiociuhandu: you could make it whatever you want. i'm not convinced that there's anything wrong with retriggering openstack project ci jobs when you also retrigger third-party jobs13:42
*** mfer has joined #openstack-infra13:42
ociuhandufungi: I know there’s no established syntax for 3rd party CIs and it’s been under discussion if there should be any but I think it would be helpful for the community to find a way to trigger only a particular 3rd party CI13:43
mjturekfungi, thanks!!13:43
fungisee bug 1355480 and http://lists.openstack.org/pipermail/openstack-infra/2014-August/001681.html for discussion13:43
uvirtbotLaunchpad bug 1355480 in openstack-ci "recheck/reverify comment_filter is too loose" [Undecided,Opinion] https://launchpad.net/bugs/135548013:43
*** MarkAtwood has quit IRC13:43
ociuhandufungi: triggering Jenkins on any recheck will just increase the load on the CI, for patches that were already successful13:43
fungiociuhandu: i personally think that's dangerous, but am open to other ideas13:44
*** stevemar has joined #openstack-infra13:44
fungiociuhandu: and trust me, people rechecking because of a third-party ci failure are not going to make a measurable impact on our resource consumption13:44
*** chat6062 has left #openstack-infra13:45
*** ptoohill has joined #openstack-infra13:45
*** e0ne has quit IRC13:48
*** jistr has quit IRC13:49
fungiSergeyLukjanov: if you're around, care to review https://review.openstack.org/113521 (i think we can safely approve it if you're +2 on it)13:49
*** jistr has joined #openstack-infra13:50
*** pblaho is now known as pblaho-meeting13:50
*** pblaho-meeting is now known as pblaho_13:50
fungiSergeyLukjanov: it's currently blocking ceilometer changes13:50
SergeyLukjanovfungi, /me looking13:50
*** pblaho has joined #openstack-infra13:50
*** pblaho is now known as pblaho-meeting13:50
*** marun has joined #openstack-infra13:51
SergeyLukjanovfungi, approved13:51
SergeyLukjanoveglynn, DinaBelova, ^^13:52
SergeyLukjanovfungi, do you remember my attempt to grant myself permissions to push 2014.1.* tags to sahara?13:52
SergeyLukjanovfungi, it's not working13:52
*** maxbit_ has quit IRC13:53
*** flaviof_zzz is now known as flaviof13:53
*** evgenyf has joined #openstack-infra13:53
SergeyLukjanovfungi, looks like th last rule isn't enough :( https://review.openstack.org/#/admin/projects/openstack/sahara,access13:54
*** p5ntangle has joined #openstack-infra13:55
*** pdmars has joined #openstack-infra13:55
*** pdmars has quit IRC13:55
fungiSergeyLukjanov: hrm, that's disappointing13:55
*** pdmars has joined #openstack-infra13:56
fungiSergeyLukjanov: i don't think we've ever tried to do that before (restricting/allowing particular tags by pattern)13:56
fungii wonder if gerrit is merely incapable, or if it has to be implemented in a different manner13:56
eglynnSergeyLukjanov: thank you sir!13:57
openstackgerritDoug Hellmann proposed a change to openstack-dev/pbr: add docs env to tox  https://review.openstack.org/11352913:57
*** julim has quit IRC13:58
openstackgerritA change was merged to openstack-infra/config: Temporarily ignore Ceilometer Python 2.6 job  https://review.openstack.org/11352113:59
*** heyongli has quit IRC13:59
*** flwang_ has joined #openstack-infra13:59
*** doug-fish has joined #openstack-infra13:59
*** harlowja_at_home has joined #openstack-infra13:59
SergeyLukjanovfungi, yeah, I'm trying to find docs or it too14:00
*** julim has joined #openstack-infra14:01
*** flwang_ has quit IRC14:03
openstackgerritJulien Danjou proposed a change to openstack-infra/config: Import pylockfile  https://review.openstack.org/10191114:04
*** afazekas has quit IRC14:05
*** dkranz has quit IRC14:05
*** wenlock has quit IRC14:06
*** hashar has quit IRC14:08
*** gargola has joined #openstack-infra14:08
*** raildo has quit IRC14:09
*** flaper87|afk is now known as flaper8714:09
*** armax has joined #openstack-infra14:10
*** zz_dimtruck is now known as dimtruck14:10
*** raildo has joined #openstack-infra14:11
stevemarhey infra folks, looks like ceilometer builds are failing py26: http://lists.openstack.org/pipermail/openstack-dev/2014-August/042704.html and a bug has been opened: https://bugs.launchpad.net/ceilometer/+bug/135559614:12
uvirtbotLaunchpad bug 1355596 in ceilometer "mongod command not found" [Undecided,New]14:12
*** maxbit has joined #openstack-infra14:12
openstackgerritChristian Berendt proposed a change to openstack-infra/config: Check requirements of stackforge/gerrit-dash-creator  https://review.openstack.org/11276814:13
fungistevemar: see https://review.openstack.org/11352114:13
fungistevemar: not a bug in ceilometer or our infrastructure. epel accidentally deleted mongodb14:14
stevemarfungi, excellent!14:14
stevemarfungi, d'oh14:14
*** hdd has joined #openstack-infra14:14
*** jistr has quit IRC14:14
*** otherwiseguy has joined #openstack-infra14:15
fungistevemar: that job is no longer voting while epel gets a chance to restore those packages and/or ceilometer devs disable mongodb tests in the python 2.6 job14:15
*** jistr has joined #openstack-infra14:15
*** p5ntangl_ has joined #openstack-infra14:16
stevemarfungi, yeah, that makes sense, i hope mongodb comes back soon14:16
*** p5ntangle has quit IRC14:19
openstackgerritJulien Danjou proposed a change to openstack-infra/config: Import pylockfile  https://review.openstack.org/10191114:19
*** hdd has quit IRC14:20
*** afazekas has joined #openstack-infra14:20
yolandamordred, jeblair, did you have any chance to review nodepool changes for dib?14:20
*** dkranz has joined #openstack-infra14:21
*** jistr has quit IRC14:21
*** zz_gondoi is now known as gondoi14:21
*** jistr has joined #openstack-infra14:21
alexpilottimjturek: confirmed, we now opted for “check hyper-v”, but it’d be great to have a standard wau to do this :-)14:21
*** e0ne has joined #openstack-infra14:22
*** abhishekk has quit IRC14:23
*** pblaho-meeting is now known as pblaho14:24
*** nelsnelson has quit IRC14:24
*** jgallard has quit IRC14:25
*** pblaho has quit IRC14:26
*** david-lyle has joined #openstack-infra14:26
*** pblaho has joined #openstack-infra14:27
*** wenlock has joined #openstack-infra14:28
mjturekalexpilotti, cool! it would be nice14:29
*** nelsnelson has joined #openstack-infra14:30
*** ildikov has quit IRC14:30
*** p5ntangl_ has quit IRC14:31
*** p5ntangle has joined #openstack-infra14:31
*** yamamoto has joined #openstack-infra14:32
*** salv-orlando has quit IRC14:33
SergeyLukjanovfungi, could you please grant me god permissions on review-dev?14:33
SergeyLukjanovfungi, is it possible to create projects on it?14:34
*** dangers_away is now known as dangers14:35
fungiSergeyLukjanov: yeah, you can do it through the webui or via review_dev.projects.yaml i think14:35
fungiSergeyLukjanov: what's your account on review-dev? i can't find you to add to the group14:35
SergeyLukjanovfungi, slukjanov14:35
fungiSergeyLukjanov: you're in the administrators group now14:36
SergeyLukjanovfungi, thank you!14:36
fungiyou're welcome, and good luck!14:36
*** yamamoto has quit IRC14:37
SergeyLukjanovfungi, heh, gerrit said "Invalid Name: refs/tags/2014.2.*" :)14:38
SergeyLukjanovfungi, refs/tags/* is ok, but refs/tags/2014.1* isn't working14:39
SergeyLukjanovfungi, that's funny14:39
fungiyeah, not sure what the limits on gerrit acl ref patterns are14:40
*** datsun180b has joined #openstack-infra14:43
openstackgerritDevananda van der Veen proposed a change to openstack/requirements: Set a minimum version for python-ironicclient  https://review.openstack.org/11354214:43
*** markmcclain has joined #openstack-infra14:44
*** funzo has quit IRC14:47
*** funzo has joined #openstack-infra14:48
*** esker has joined #openstack-infra14:49
*** gokrokve has joined #openstack-infra14:50
openstackgerritDevananda van der Veen proposed a change to openstack/requirements: Set a minimum version for python-ironicclient  https://review.openstack.org/11354214:52
*** AlexF_ has quit IRC14:52
*** Guest75191 is now known as dims14:54
*** ihrachyshka_ has quit IRC14:55
*** dmsimard_away is now known as dmsimard14:55
fungijraim: ttx: should https://review.openstack.org/84811 have included python-kiteclient too?14:57
*** AlexF_ has joined #openstack-infra14:57
fungitrying to figure out the scope of the stackforge->openstack transition for the kite projects14:57
david-lyleI'd like to actually use https://launchpad.net/~horizon-bugs to manage Horizon bugs, can someone in Openstack Infra add me as an adminstrator?14:59
david-lyleI know jeblair has been able to make such changes in the past15:00
*** markmcclain1 has joined #openstack-infra15:00
ttxfungi: if there is such thing, certainly15:00
fungidavid-lyle: i can15:00
*** markmcclain has quit IRC15:01
*** jlibosva has quit IRC15:01
david-lylefungi: thanks!15:01
fungidavid-lyle: are you also david-lyle on lp?15:02
david-lyleI'm uncomplicated15:02
*** james_li has joined #openstack-infra15:02
fungidavid-lyle: done15:02
david-lylefungi: thank you15:02
*** maxbit has quit IRC15:03
*** rcarrillocruz has joined #openstack-infra15:03
fungiyou're welcome15:03
*** rcarrill` has quit IRC15:04
*** radez_g0n3 is now known as radez15:04
SergeyLukjanovfungi, I found the solution - such regex should be prefixed with ^15:05
SergeyLukjanovfungi, it works on review-dev15:05
SergeyLukjanovfungi, /me creating CR to fix15:05
*** maxbit has joined #openstack-infra15:07
*** UtahDave has joined #openstack-infra15:07
*** UtahDave has left #openstack-infra15:07
*** Ajaeger has joined #openstack-infra15:08
*** dkranz has quit IRC15:08
*** afazekas has quit IRC15:08
openstackgerritSergey Lukjanov proposed a change to openstack-infra/config: Fix sahara tag acl to make it working  https://review.openstack.org/11355215:09
SergeyLukjanovfungi, ^^15:09
SergeyLukjanovfungi, I think gerrit is trying to parse refs string and not always could understand that it's a real regexp15:10
SergeyLukjanovfungi, and supports only one simple wildcards otherwise15:11
SergeyLukjanovfungi, could you please fast approve it when have some time to unblock sahara release?15:11
fungiSergeyLukjanov: done. amusing that adding a (seemingly unimportant) ^ anchor makes all the difference15:12
SergeyLukjanovfungi, I think it's some kind of parsing magic :(15:13
SergeyLukjanovfungi, thank you15:13
*** amuller_ has joined #openstack-infra15:13
*** todd_dsm has joined #openstack-infra15:13
Ajaegerfungi: Is my change to merge the propose-requirements scripts fine now or do you have further suggestions for improvement? Could you have a look, please? https://review.openstack.org/11134415:14
*** amuller has quit IRC15:15
*** MarkAtwood has joined #openstack-infra15:16
fungiAjaeger: i have it up in a tab hoping to get to it after the current meeting i'm in15:17
openstackgerritSteve Martinelli proposed a change to openstack-dev/pbr: cleanup tox.ini  https://review.openstack.org/11355315:18
YorikSarfungi, mordred, jhesketh: Hi. Have you happen to see my thread on ML about local branches support for git-review? I'd like to know your overall opinion on this as you're those who might -2 this change if I do it within git-review. In such case I'd start working on a tool for this outside git-review.15:18
AjaegerThanks! Just have a few minutes while waiting for my plane and wanted to chat with you if the time is fine. But let's followup the usual process and if I have questions, I'll come back...15:18
YorikSarfungi, mordred, jhesketh: http://lists.openstack.org/pipermail/openstack-dev/2014-August/041872.html15:19
fungiYorikSar: it's not a feature i would use, and my main concerns over it would be adding complexity to the tool and possibly encouraging confusing/dangerous workflow choices for new contributors, but i've mostly just been watching the thread to see what other users of the tool think15:20
*** dkranz has joined #openstack-infra15:21
YorikSarfungi: Yes, the concern that such feature can be used as "autosquash" we'd like to avoid for newcommers has been raised on the thread...15:21
*** afazekas has joined #openstack-infra15:21
fungiin my opinion git-review is, first and foremost, a means of simplifying the contribution process for developers who are new to git and gerrit, far more than encoding arbitrary workflows for the benefit of seasoned devs15:22
YorikSarfungi: Doesn't hiding it behind config option solve that?15:22
YorikSarfungi: Ok, I see...15:22
phschwartz YorikSar: No it doesn't as the commit still has to be pushed to a repo other then what was cloned for it to appear in gerrit, so git-review makes it easier, by handling that for you.15:23
fungipossibly, but i want to see the conversation play out between people who are developers and new contributor mentors, since their opinions are what i'm more interested in15:23
YorikSarfungi: I thought it'd a right place to simplify life of seasoned dev as well :)15:23
fungimy opinion is, to a great extent, secondary. i will support consensus among the community if it makes sense15:24
*** changbl has joined #openstack-infra15:24
YorikSarfungi: Unfortunatelly that thread had died off after strong opposition provided by Ben...15:25
phschwartzfungi: I have talked about this exact thing with a group of other devs before, and their thoughts were the same as yours.15:25
phschwartzIt would be interesting to see the discussion to a greater extent15:26
YorikSarphschwartz: I'm not sure I understand what repos you're talking about...15:26
*** skolekonov has quit IRC15:26
*** eharney has joined #openstack-infra15:26
AjaegerSergeyLukjanov: could you recheck https://review.openstack.org/113080, please? Do you propose a different solution?15:26
Ajaegerright now the requirements-proposal job fails completely.15:26
Ajaeger(I mean the post job)15:26
openstackgerritafazekas proposed a change to openstack-infra/elastic-recheck: Add query for fixed ip shortage with n-net  https://review.openstack.org/11255815:26
phschwartzrepos is a generic term. I was just reiterating why git-review is used. in the repos It refered to the base clone off of git.openstack.org or github.com and the gerrit internal repo that you push to when you do a git-review15:27
*** amotoki has joined #openstack-infra15:27
YorikSarphschwartz: My proposal is to support workflow a bit different from 'git review' - fix issues - amend/rebase - 'git review'15:28
*** Ajaeger has quit IRC15:28
*** alkari has joined #openstack-infra15:29
YorikSarphschwartz: And it will still work for repos cloned from elsewhere.15:29
phschwartzYorikSar: I have read your proposal, and don't agree that it solves that as to how gerrit uses patchsets. I personally feel it will cause more greif then help for new developers.15:30
fungiYorikSar: it was also unclear to me how you use your local branch to distill/refactor your various historical edits into a series of dependent changes on a topic branch before pushing to gerrit (rather than the trivial use case of squashing them to one commit before pushing)15:31
YorikSarphschwartz: By 'solve that' I meant that new devs will have to explicitely turn new behavior on to use that. And they'll have to accept that they understand that their local view of the repo will differ a bit from what Gerrit sees.15:31
phschwartzAnd to note, if that is how you like to work, you can do it. gerrit only requires that you send on patch with a change-id to gerrit. Nothing is stopping you from doing it locally that way.15:31
*** todd_dsm has quit IRC15:32
*** yamamoto has joined #openstack-infra15:32
fungiYorikSar: do the changes with change-ids in their commit messages act as sequential checkpoints in your local history?15:32
phschwartzYorikSar: Which is the exact reason why I would personally be agaisnt it being in there. If there is an issue, it will make it harder for the dev using it to troubleshoot it with help from other users as they will see different things in their repo (makes rebase/merge conflicts harder to help a developer fix)15:33
*** todd_dsm has joined #openstack-infra15:33
YorikSarfungi: For every change request in upstream I'd like to have a separate local branch. Tip point of that branch will be uploaded as a new patchset when you run git-review. If your branch has merge commits in it, that patchset will depend either on master (last commit merged from master to local branch) or another change request (last local branch merged into this branch).15:34
*** afazekas has quit IRC15:35
*** rwsu has joined #openstack-infra15:35
YorikSarphschwartz: Well... For seasoned Git user, handling merge conflicts is not a huge issue. And those conflicts will be the same in uploaded patchset as in your local branch since they will reference the same tree object.15:37
*** yamamoto has quit IRC15:37
openstackgerritSpencer Krum proposed a change to openstack-infra/config: Bump postgres module version  https://review.openstack.org/11356115:38
phschwartzYorikSar: Actually I am not refereing to local merge conflicts, but when the zuul merge has a conflict. The conflict tree from zuul merge will be diferent as it will see the mainline source tree and not the localbranch tree for the developer using this.15:38
openstackgerritSalvatore Orlando proposed a change to openstack-infra/config: Add neutron full job to integrated gate  https://review.openstack.org/11356215:38
*** cdub has joined #openstack-infra15:38
fungiYorikSar: also, git rerere may be able to solve some of your "rebase repetition" pain. give it a try if you haven't...15:39
YorikSarphschwartz: If Zuul fails with merge conflict, you can merge master into your local branch, fix conflicts and run git-review once again.15:39
*** dtantsur is now known as dtantsur|afk15:40
YorikSarfungi: I don't find rebase painful... I just don't like that it doesn't keep history of what I changed during rebase...15:41
fungidepends on what you mean by "keep history"15:41
fungigit rerere will record your resolutions so they can be reapplied15:42
YorikSarfungi: But with Gerrit interactive rebase becomes the only way to fix any commit in patch series but the last one.15:43
fungiat least for me, that's the only history i care about from my rebases... "how did i solve this conflict so i can repeat it?"15:43
YorikSarfungi: So it's not merge conflicts I want to record. It's just usual edits.15:43
fungii also don't see what's wrong with interactive rebase. i use it for tons of things besides just rebasing a change on another branch. you can use it to edit changes in your series, reorder them, insert or delete them, et cetera15:44
fungii suppose it depends on how you view git. it was designed as a machine for developing patches (for eventual posting on the lkml) and as such it's well adapted to building a series ("history") of changes for proposing to the kernel15:45
fungiit's less well adapted to keeping tons of revision cruft around15:46
*** hdd has joined #openstack-infra15:46
fungithough it has grown features for dealing with it15:46
*** shashankhegde has quit IRC15:48
YorikSarfungi: But for Linux, once you've published pull request to ML, you start to carefully keep published commits instead of rebasing them around which is the case when you use Gertit.15:49
YorikSarfungi: So instead of "fixing that issue in 3rd commit of 10" you fix the issue and upload new commit.15:50
YorikSarfungi: But for Gerrit you do 'rebase -i' and your change gets lost in the pile of changes rebase does automatically...15:51
fungitrue, though the lkml also wasn't well adapted to code review, and thus stacking new edits on top of your public tree made it easier for reviewers to catch up on your edits. gerrit on the other hand provides different mechanisms to accomplish that15:52
fungihowever, i also don't keep any local copy of my commits. i only use my local clone to prepare one or more patches to push to gerrit, then delete the local topic branch and forget about it, pulling back from gerrit again if it needs to be modified15:53
mordredYorikSar: also, you said the key "For seasoned Git user"15:53
mordredwe do _not_ have seasoned git users15:53
*** jgallard has joined #openstack-infra15:53
*** ociuhandu has quit IRC15:53
YorikSarfungi: I mean how would you undo change between N-10th patchset and N-9th in the change request that's in the middle of your series of change requests.15:53
*** ramashri has joined #openstack-infra15:53
fungiYorikSar: git review can check both out from gerrit quite easily15:54
mordredYorikSar: I pull the tip of the series in gerrit, do a git rebase -i - edit the patch in the middle, and then re-push15:54
phschwartzmordred: I think that is the exact flow most of us use15:54
*** reed has joined #openstack-infra15:54
*** emagana has joined #openstack-infra15:55
fungii think the question there was about undoing earlier edits to a particular patchset in a programmatic fashion, rather than altering a change in the missle of a proposed series15:55
*** ihrachyshka has joined #openstack-infra15:55
*** annegent_ has joined #openstack-infra15:55
openstackgerritMatt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for py26/centos6 node setup bug 1355907  https://review.openstack.org/11356715:55
uvirtbotLaunchpad bug 1355907 in openstack-ci ""sudo: yum: command not found" in py26 jobs (bare-centos6-rax* nodes)" [Undecided,New] https://launchpad.net/bugs/135590715:56
YorikSarmordred: I just repeated (almost) what fungi said :) But from some replies to ML, it seems that we have people in OpenStack that do almost the same workflow. So I doubt we don't have people that know Git well in our community...15:56
fungibut regardless, git-review lets you fetch specific patchsets from a proposed change15:56
phschwartzfungi: correct and with some bash pipes you can generate it out as a patch file that you can roll back from a later patch using git am15:57
YorikSarmordred: Yes, you can _change_ that middle commit, but you can't _undo_ that change later, for example.15:57
YorikSarI mean, you can, but you can't use simple git revert for that because that change is not in a separate commit.15:57
*** hdd has quit IRC15:58
*** hdd has joined #openstack-infra15:58
YorikSarphschwartz: Reverting to explicit patch storage is what I want to prevent using Git to its potential.15:58
stevemarbnemec, ping15:58
bnemecstevemar: pong15:59
stevemarbnemec, patch https://review.openstack.org/#/c/113553/1/tox.ini , just to be sure, you wanted line 4 to be: envlist = py33,py26,py27,pypy,pep8 ?15:59
*** Guest35108 has quit IRC16:00
*** flwang_ has joined #openstack-infra16:00
*** jcoufal has quit IRC16:00
phschwartzYorikSar: I am not sure that most developers that are experienced with gerrit workflows would agree16:00
stevemarbnemec, also, thoughts on adding docs to that list?16:00
*** arnaud has joined #openstack-infra16:00
fungiYorikSar: it sounds like what would be actually useful is a git branch in gerrit representing the patchsets of each change over the course of its development. chances are that would simplify their interdiff and other things as well16:00
phschwartzYorikSar: yes, outside of using gerrit I would use the exact flow your talking about, but gerrit makes it so you don't have to manage history to that point locally.16:01
bnemecstevemar: Oh, yeah we should probably do that.16:01
phschwartzfungi: if it was on the gerrit side, I agree, but YorikSar had reiterated that he wants it as a local branch on the developers system16:01
stevemarbnemec, so it should read: envlist = py33,py26,py27,pypy,pep8,docs16:01
bnemecstevemar: Yep16:02
* clarkb dives into the discussion. nothing prevents that workflow with gerrit today. are we purely arguing if it should be codified by git review?16:02
stevemari don't see the point in having py33 first, but probably my env.16:02
marcusvrnHi all, I'm trying to install the CI system and after I setup the Jenkins master (using the script provided by http://www.joinfu.com/) the zuul/debug.log returns "SSHException: not a valid DSA private key file". Does anyone know what is happening?16:03
*** jgallard has quit IRC16:03
YorikSarfungi: Unfortunately storing these dependencies on Gerrit side is not really possible. You'll end up having different parents for different patchsets and that won't be the full history in the end anyway.16:03
clarkbI personally think git review should contain the simplest of tools to do the simplest of workflows and was opposed to git review -x16:03
clarkbwhich I have never used16:03
openstackgerritSteve Martinelli proposed a change to openstack-dev/pbr: cleanup tox.ini  https://review.openstack.org/11355316:03
bnemecstevemar: There's some sort of weird thing with one of our test libs where it creates a db, and if the py2 version runs before the py3 version the db is incompatible.16:04
*** juzuluag has quit IRC16:04
clarkbmarcusvrn: I would check your ssh private key file for correctness and do sone manuak ssh testing16:04
*** flwang_ has quit IRC16:04
bnemecstevemar: Basically you only see it if you run tox on a system with both py2 and py3 and the py2 version runs first.16:04
stevemarbnemec, i trust ya, tossed it up ^16:04
fungiYorikSar: well, it will be the full history of the patchsets which were proposed, as long as you do it for all changes consistently16:05
YorikSarfungi: I've described the process of reverting from almost what Gerrit has (with intermediate commits that don't get uploaded) to branch-based history in one of emails...16:05
bnemecstevemar: Thanks!16:05
stevemarnp at all bnemec16:05
* bnemec exits stage left before he gets pulled into the git-review discussion again16:05
*** ihrachyshka has quit IRC16:06
YorikSarfungi: http://lists.openstack.org/pipermail/openstack-dev/2014-August/042168.html16:06
dimsack. i've seen py2 then py3 causing issues16:07
fungiYorikSar: yep, i read it when you posted it16:07
*** hdd has quit IRC16:07
YorikSarfungi: So you don't get all history uploaded to Gerrit. Some intermediate commits get lost and they can still be valuable for bisecting...16:08
fungiyou get all the history of what *was* uploaded to gerrit, is what i was trying to say16:08
fungii don't personally care about multiple adjustments i made which i never pushed to gerrit, nor can i imagine wanting to bisect between them (particularly not if they're muddied by constantly pulling from master into my topic branch)16:09
openstackgerritMatt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for infra setup bug 1354829  https://review.openstack.org/11357516:09
uvirtbotLaunchpad bug 1354829 in tempest ""sudo: 3 incorrect password attempts" in host setup" [Undecided,Invalid] https://launchpad.net/bugs/135482916:09
*** kashyap has joined #openstack-infra16:10
phschwartzIs there something that would purposefully remove glance.hacking from the installed code when you do a setup.py install?16:10
openstackgerritMatt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for infra setup bug 1354829  https://review.openstack.org/11357516:10
uvirtbotLaunchpad bug 1354829 in tempest ""sudo: 3 incorrect password attempts" in host setup" [Undecided,Invalid] https://launchpad.net/bugs/135482916:10
*** vhoward has joined #openstack-infra16:10
fungimriedem: is 1354829 actually a bug?16:11
mriedemfungi: by what definition?16:11
*** vermarah_ has joined #openstack-infra16:11
YorikSarfungi: Well... Have you ever had to maintain long change request chains (> 4) for a long time (> 3 months)?16:11
mriedemfungi: it's a thing that failed jobs16:11
mriedemfungi: 193 hits in 7 days, check and gate16:11
clarkbphschwartz what is glance.hacking?16:11
fungimriedem: as opposed to correct behavior, refusing to run sudo'd commands for users who weren't configured to be allowed to do so16:12
YorikSarfungi: I guess that experience is what left me wanting to have some smarter tool to keep track of all intermediate changes...16:12
mriedemfungi: it's a failure that's not related to the actual change being tested,16:12
mriedemfungi: so not sure what you're asking16:12
clarkbphschwartz but yes it may be excluded by manifest.in or similar if not needed to be installed16:13
phschwartzclarkb: source path in the glance code base. It is loading by hacking from the tox.ini for glance. I installed a venv and can't run the hacking tests on it as it fails to import it.16:13
*** yfried_ has joined #openstack-infra16:13
fungiYorikSar: i have in fact, but having a history of how i dealt with earlier rebases would not have helped me keep up with changes on master16:13
phschwartzclarkb: doesn't look to be excluded in manifest.in16:13
*** mpaolino has joined #openstack-infra16:14
*** yfried has quit IRC16:14
YorikSarfungi: Well, even merges from master would have been recorded (they essentially replace rebases).16:14
fungimriedem: just not sure whether it's suited to an e-r query since it's a very common failure mode which is usually triggered by people trying to add sudo calls into jobs where they're not allowed, or forgetting to adjust the jobs to allow them16:14
clarkbmriedem we dont allow sudo in many cases.I think fungi is saying sudo failing is probably intentional16:14
phschwartzwait, no graft for it in manifest.in so it is not installed as part of it16:14
clarkbmriedem the actual bug is the test that needs sudo when it cant sudo16:14
fungimriedem: so they could in fact end up in the gate as well given that changes to teh config repo (where job configuration resides) is not co-gated nor self-testing16:15
mriedemfungi: clarkb: but this isn't in a test...16:15
*** evgenyf has quit IRC16:15
*** amuller__ has joined #openstack-infra16:15
clarkbyou said it was in the gate16:15
clarkbI use test in the large sense here16:15
mriedemyes, check and gate, multiple jobs16:15
mriedemit's in the devstack setup16:16
*** yfried__ has joined #openstack-infra16:16
mriedemsudo yum install -y libffi-devel16:16
mriedemsudo: no tty present and no askpass program specified16:16
clarkboh that. can we just stop using fedora?16:16
mriedemi would not be the person to ask16:17
mriedembut given we're talking about testing latest libvirt, possibly with a fedora job, i'd think not16:17
clarkbfedora has decided you must be at a tty to root16:17
clarkbwhich is wrong (tm)16:17
fungimriedem: is it possible these were non-voting jobs in the gate pipeline?16:18
mriedemfungi: yeah, i didn't recognize some of the names16:18
mriedemhowever, https://bugs.launchpad.net/neutron/+bug/135482916:18
uvirtbotLaunchpad bug 1354829 in tempest ""sudo: 3 incorrect password attempts" in host setup" [Undecided,Invalid]16:18
mriedemthe guy reported the failure originally against a py26 job16:18
*** arnaud has quit IRC16:18
clarkbpy26 should fail16:18
*** loki184 has joined #openstack-infra16:19
clarkbit not failing would be a bug16:19
fungiright, if it was a sudo failure reported against a unit test job, then that was the fault of the change being proposed16:19
*** amuller_ has quit IRC16:19
mriedemhow so?16:20
clarkbmriedem unittest slaves do not allow jobs to sudo16:20
fungibut as far as "hits in the gate," we probably want to confirm whether e-r is actually counting non-voting changes in the gate pipeline, and possible ways to make it not16:20
*** yfried_ has quit IRC16:20
clarkbbecause running `tox` is unpriveleged and should never require root16:20
mriedemfungi: we don't have anything in logstash to filter on voting/non-voting as far as i know16:20
mriedemclarkb: fungi: i guess i'm at a loss to how joe developer understands that a unit test host setup failures is the fault of this change https://review.openstack.org/#/c/110186/1/neutron/plugins/ofagent/agent/ofa_neutron_agent.py16:21
mriedemwhich is a docstring change16:21
fungimriedem: yeah, we're going to end up with a number of false gate failure hits in that case... we try to limit the number of non-voting jobs run in the gate pipeline, but there are certainly some from time to time16:21
clarkbmriedem its neutrons fault16:21
clarkbthey dont properly mock in their tests16:21
clarkbthe tests then run sudo. the tests fail16:22
*** yfried__ has quit IRC16:22
clarkbneutron is basically the reason we explicitly enforce sudo = failure in unittests16:22
*** shashankhegde has joined #openstack-infra16:22
clarkbthere should be a note in the test log that says use of sudo is not allowed and explains this16:23
mriedemso http://goo.gl/mRtcVn16:23
fungii suspect that the particular failure for that change was actually because someone approved a poorly-thought-out alteration to gate-neutron-python2616:23
mriedemthere are also devstack/tempest/grenade jobs in there16:23
fungiit looks like there was briefly a "sudo yum install -y libffi-devel" in the job builders after revoke-sudo16:24
fungichecking config history now to confirm16:24
clarkbfungi: mordreds fix for that wasnt correct? that is possible16:25
fungiyep, caused by https://review.openstack.org/112972 which was only that way briefly i think16:25
*** jistr has quit IRC16:25
flaper87hey guys, any idea when this will be merged? https://review.openstack.org/#/c/111244/ NOt sure if you already discussed it or are planning to. :)16:26
asselin__clarkb, fungi, would it be safe to +3 this change soon? https://review.openstack.org/#/c/96551/16:26
fungiflaper87: as part of the (not yet scheduled) gerrit outage to do the renaming16:26
fungiflaper87: it's on our agenda to discuss in the infra team meeting today16:26
flaper87fungi: awesome. thanks for the heads up. I'll read the logs :)16:27
*** rcarrillocruz has quit IRC16:27
*** ildikov has joined #openstack-infra16:27
fungiflaper87: if you're not goung to be around for the meeting, do you have any particular suggestions for dates/times which would be (in)convenient for your developer community?16:27
clarkbasselin__ ya I can do it now16:28
*** pballand has joined #openstack-infra16:28
fungiflaper87: noting that all their git remotes will break in their local clones and need updating to the new url16:28
clarkbactually because I am paranoid let me noop apply somewhere first16:29
asselin__clarkb, thanks I'll take out my workaround and test16:29
*** rcarrillocruz has joined #openstack-infra16:29
*** hogepodge has quit IRC16:29
*** pblaho has quit IRC16:31
*** yamamoto has joined #openstack-infra16:32
flaper87fungi: the sooner we do it, the better. We've already renamed the project internally (imports, files and docs)16:32
clarkbmriedem: note the yum -y is done there because yum returns success even when it fails when 1) -y is used and 2) multiple packages are being installed16:32
fungiflaper87: noted. thanks!16:33
flaper87fungi: np, thank you guys.16:33
mordredkrotscheck, ttx: what do we want to do about assignees and bug creators who do not yet have storyboard users16:33
clarkbmriedem so our puppet stuff succeeds but really failed. to "fix" this the jobs double tap that package install16:33
mriedemclarkb: as long as one package is installed it returns success right? i heard about that 'feature' recently16:33
clarkbmriedem yup16:33
mordredkrotscheck: if I create a User object without an openid string associated with it, how many things break?16:33
krotscheckmordred: Let me check.16:34
clarkbmordred: could you prepopulate users?16:34
*** shashankhegde has quit IRC16:35
clarkbmordred lp gives you enough info to make it not terrible right?16:35
mordrednot the openid16:35
mordredbecause they STILL wrongly think that they are write about that16:35
clarkbyou can look that up like account sync did right?16:35
krotscheckmordred: Lots of things would break.16:35
mordredkrotscheck: kk16:35
mordredclarkb: I REALLY don't want to do that16:35
clarkbthats understandable :)16:36
krotscheckmordred: We have a uniqueness constraint on email address in the users table, and when resolving an openID auth request we query by openid, not by email.16:36
krotscheckSo you can create a user fine, but without an openID we’ll end up trying to create duplicate users.16:36
*** chandankumar has quit IRC16:36
krotscheckAnd that will fail because of the uniqueness constraint.16:36
*** rpodolyaka is now known as rpodolyaka_afk16:36
*** AlexF_ has quit IRC16:37
*** yamamoto has quit IRC16:37
fungiclarkb: mriedem: !!!16:37
openstackgerritMatt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for test_postgresql_opportunistically bug 1355929  https://review.openstack.org/11358116:37
uvirtbotLaunchpad bug 1355929 in nova "test_postgresql_opportunistically fails in stable/havana due to: "ERROR:  source database "template1" is being accessed by other users"" [Undecided,New] https://launchpad.net/bugs/135592916:37
fungiclarkb: mriedem: actually, look at the build_node value for all those failures16:37
mriedemyeah, no yum16:38
fungiclarkb: mriedem: strangely, this may be related to the run-pep8.sh failures16:38
clarkbfungi dfw only too?16:38
fungimriedem: the relevant part is that they're *all* in rax-dfw16:38
openstackgerritA change was merged to openstack-infra/config: Fix sahara tag acl to make it working  https://review.openstack.org/11355216:38
mriedemwhat is dfw?16:38
fungimriedem: the airport code for dallas/fort worth16:38
*** _nadya_ has quit IRC16:38
fungimriedem: a region in rackspace cloud16:39
mriedemthen i blame tony romo16:39
fungii think we have something seriously odd with instances booted in rax-dfw16:39
mriedemi could restrict the query to those nodes16:39
mriedembut i'd have to use wildcards16:40
fungii'm not sure we can wildcard in that field16:40
fungiclarkb: ^ ?16:40
mriedemi think we can16:40
clarkbfungi you can16:40
clarkbso *-dfw-* or whatever16:40
mriedemclarkb: but should i?16:40
mriedemi.e. perf?16:40
mriedemor doesn't matter in this case16:40
*** derekh has quit IRC16:41
clarkbit matters. I think we are ok doing it if necessary16:41
*** bdpayne has joined #openstack-infra16:41
mriedemok, i'll add it16:41
clarkbproblem before was a lot of queries didnt need it16:41
mriedemclarkb: fungi: ok this is also yum not found on rax-dfw jobs https://bugs.launchpad.net/openstack-ci/+bug/135590716:42
uvirtbotLaunchpad bug 1355907 in openstack-ci ""sudo: yum: command not found" in py26 jobs (bare-centos6-rax* nodes)" [Undecided,New]16:42
*** markmcclain1 has quit IRC16:42
mriedemthose are bare-centos6-rax-dfw nodes so i thought yum was more appropriate16:42
*** hogepodge has joined #openstack-infra16:43
fungimriedem: so this may be the same underlying cause as bug 135516716:44
uvirtbotLaunchpad bug 1355167 in openstack-ci "run-pep8.sh usage now being output" [Critical,In progress] https://launchpad.net/bugs/135516716:44
openstackgerritA change was merged to openstack-infra/elastic-recheck: Add query for fixed ip shortage with n-net  https://review.openstack.org/11255816:44
fungiphschwartz: are you aware of any snapshot booting weirdness specifically in dfw?16:45
clarkbdoes nova show show us the image booted from?16:45
clarkb(trying to sort out how we might even debug this)16:45
fungiclarkb: not once the node is deleted i don't think... nodepool's debug log might mention it16:45
*** mpaolino has quit IRC16:45
*** dims has quit IRC16:46
*** dims has joined #openstack-infra16:47
*** dims is now known as Guest2710416:47
*** harlowja_at_home has quit IRC16:47
clarkbwe could probably just nova show all dfw nodes every few minutes16:47
clarkbof spscific types and look for deltas16:48
*** e0ne has quit IRC16:48
openstackgerritMichael Krotscheck proposed a change to openstack-infra/storyboard: Added subscription API  https://review.openstack.org/10851616:49
*** harlowja_at_home has joined #openstack-infra16:49
*** penguinRaider has quit IRC16:50
openstackgerritMatt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for infra setup bug 1354829  https://review.openstack.org/11357516:50
uvirtbotLaunchpad bug 1354829 in tempest ""sudo: 3 incorrect password attempts" in host setup" [Undecided,Invalid] https://launchpad.net/bugs/135482916:50
*** Guest27104 has quit IRC16:51
*** penguinRaider has joined #openstack-infra16:51
mordredkrotscheck, clarkb: I'm a bit stumped at to how to handle this ...16:54
mordredkrotscheck: stories and tasks don't allow not having a creator16:55
*** sandywalsh_ has quit IRC16:56
*** SumitNaiksatam has quit IRC16:56
*** aviau has joined #openstack-infra16:56
aviauHello! Can someone help me with my rtfd hook? it doesn't seem to work.16:56
clarkbaviau it uses the project name so make sure it matched on the rtfd side16:57
aviauI have it setup in : modules/openstack_project/files/jenkins_job_builder/config/projects.yaml (Project surveil)16:57
*** rdopieralski has quit IRC16:57
aviauclarkb, And my project is here: https://readthedocs.org/projects/surveil/16:57
clarkbmordred: Can we create as a system user then allow individuals to adopt?16:58
*** ihrachyshka has joined #openstack-infra16:58
clarkbmordred: I really think we may need user syncage though :(16:58
*** amuller__ has quit IRC16:58
*** amuller__ has joined #openstack-infra16:59
vishymordred: happy birthday! Stop working and go get drunk16:59
aviauClabbe, The project names look like they match but nothing updated.17:00
*** jpich has quit IRC17:00
mordredvishy: done!17:00
mordredvishy: and thanks17:01
marcusvrnclarkb: The problem was with my private key...thx! But now...the CI returns this error : "This change was unable to be automatically merged with the current state of the repository. Please rebase your change and upload a new patchset."17:01
openstackgerritBen Nemec proposed a change to openstack/requirements: Add oslo.serialization to projects.txt  https://review.openstack.org/11299317:01
clarkbaviau: next step is to look at jobs logs. http://logs.openstack.org/firsttwoofsha1/sha117:01
marcusvrnclarkb: I'm using the sandbox testing environment17:01
clarkbmarcusvrn make sure the zuul merger is running17:01
*** arnaud has joined #openstack-infra17:01
*** praneshp has joined #openstack-infra17:01
clarkbvishy mordred wait. we like it when mordred works on stuff17:02
clarkbI will accept drunk mordred work though :)17:02
mordredkrotscheck: ok. what about if there is no email address, but we _do_ have the openid?17:02
*** mestery has quit IRC17:02
aviauclarkb, How can I find the sha?17:02
*** mestery has joined #openstack-infra17:03
*** shashankhegde has joined #openstack-infra17:04
aviauclarkb, I found it.17:04
phschwartzfungi: I haven't heard of one. I will check17:05
aviauclarkb, I had forgotten to add the job in modules/openstack_project/files/zuul/layout.yaml17:06
*** sandywalsh has joined #openstack-infra17:06
*** mrmartin has quit IRC17:06
*** ianw has quit IRC17:08
krotscheckmordred: Does Gerrit have the openid?17:08
openstackgerritNicholas Randon proposed a change to openstack-infra/config: Add firewall traffic on port 27410 to 27510.  https://review.openstack.org/10873717:08
krotscheckmordred: i.e. can we do a batch user import from Gerrit first?17:08
mordredkrotscheck: hrm17:09
openstackgerritA change was merged to openstack-infra/elastic-recheck: Add query for py26/centos6 node setup bug 1355907  https://review.openstack.org/11356717:09
uvirtbotLaunchpad bug 1355907 in openstack-ci ""sudo: yum: command not found" in py26 jobs (bare-centos6-rax* nodes)" [Undecided,New] https://launchpad.net/bugs/135590717:09
*** praneshp has quit IRC17:10
*** james_li_ has joined #openstack-infra17:10
openstackgerritAlexandre Viau proposed a change to openstack-infra/config: Added rtfd template to Surveil  https://review.openstack.org/11358717:10
krotscheckmordred: I was just thinking that the authoritative source of “who’s involved in openstack” is the CLA, which is managed through Gerrit, yes?17:10
fungikrotscheck: well, not necessarily (not all projects require a cla, also not all users of gerrit are submitting changes which is the only activity a cla is enforced on)17:12
krotscheckfungi: Well, doh.17:12
*** eglynn has quit IRC17:12
*** james_li has quit IRC17:12
*** james_li_ is now known as james_li17:12
*** SumitNaiksatam has joined #openstack-infra17:12
fungikrotscheck: oh, though yes if you mean more broadly all gerrit users rather than just those who have agreed to a cla, probably closer17:12
mordredhowever, it probably gets us darned close17:13
krotscheckWhat does launchpad use to link openID and their user accounts?17:13
marcusvrnclarkb: the zuul merger is running... :(17:14
mordredwell, launchpad consumes the ubuntu SSO system17:14
mordredkrotscheck: the problem is, we don't necessarily have API access to all of the data we need17:14
mordredto get a person's openid identifier, we actually have to go make a fake openid consumer request to the login page and pull the identifier17:15
*** krtaylor has quit IRC17:15
mordredbut then that can be wrong, because the ubuntu sso people havea  very strange idea of how openid works17:15
mordredso if a person has ever merged an account, it's possible that requesting the ID from the IDP for a user will give you a _different_ openid string from what the finalization of the login process will return17:16
mordredand then also, users are allowed to hide their email address17:16
fungimordred: well, there *is* the lp api's get_user_by_openid method17:16
mordredand there is NO way for us to scrape that information without their involved17:16
krotscheckmordred: because that makes perfect sense.17:16
mordredfungi: right. but it's the wrong way17:16
*** praneshp has joined #openstack-infra17:16
fungimordred: agreed17:16
mordredso, more and more I'm leaning towards krotscheck suggestion of just slurping from teh gerrit db17:17
*** andreaf_ has quit IRC17:17
mordredI'm sure there will be bugs somewhere with a user who does not have a gerrit account ... but maybe we handle those as exceptions17:17
krotscheckmordred: I don’t suppose we can just ask canonical for a DB dump?17:17
fungiyeah, lp doesn't have a get_openid_by_user method17:17
mordredwell, from their pov, email is privileged, so they can't give it to us if the user objects17:18
fungialso, most of the people who could have gotten it for us have been hired away to work on openstack now ;)17:18
krotscheckI don’t think we need that though.17:18
*** harlowja_at_home has quit IRC17:18
krotscheckWe need the userId that’s associated as a task author, and the openid associated with that userid, no?17:18
fungiso they're a bit strapped just keeping things running there17:18
mordredkrotscheck: but to make a User object, I need an email17:19
mordredwhich is a unique key17:19
mordredkrotscheck: unless we wanted to make email nullable17:19
*** Ryan_Lane has joined #openstack-infra17:20
krotscheckmordred: fff… well, yeah. That, or we dump in an autogenerated email address. That’s overwritten the first time the user logs in anyway.17:20
*** andreaf has joined #openstack-infra17:20
krotscheck(User name and email address is auto-updated from openid on every login)17:20
fungii think smarcet and mrmartin might have some similar needs for bootstrapping the openstackid.org backend17:21
*** rcarrill` has joined #openstack-infra17:23
* anteaya looks at the agenda for today's meeting: https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting17:23
anteayajesusaurus nibalizer do you still need Puppet module Split out on today's infra agenda?17:24
fungii don't know if jeblair planned to be around today for the meeting, but if not i'm happy to chair it17:24
nibalizeruh we can talk about it17:24
* anteaya nods17:24
anteayanibalizer: great it can stay17:24
*** rcarrillocruz has quit IRC17:24
nibalizerim lacking a 'next step' short of 'get more consensus' so yea lets put it on the agenda17:25
* anteaya removes the git-upstream item, it has been discussed17:25
clarkbmarcusvrn: check the merger logs then17:25
clarkbmarcusvrn: should be in /var/log/zuul/merger*.log17:25
anteayanibalizer: very good17:25
clarkbnibalizer: anteaya wait I thought we had that last meeting?17:26
anteayaI'll remove the third party item from the agenda, I am following up with folks in the third party space and they are aware that coordinating with nibalizer and jesusaurus is a good idea17:26
*** bhuvan has joined #openstack-infra17:26
clarkbbasically it sounded like we needed a guinea pig project to do it on17:26
anteayaclarkb: we did, I am removing the item17:26
clarkbdo it, figure out how to make it go better then do it for all the things17:26
anteayaso we don't have to discuss it again today17:26
anteayaclarkb: right17:26
*** mmaglana has joined #openstack-infra17:26
nibalizeroh we're gonna go ahead with one?17:27
nibalizerkrotscheck: lets do storyboard!17:27
*** james_li has quit IRC17:27
clarkbnibalizer: I thought that is what we agreed on17:27
fungiclearly it was a popular choice17:27
clarkbpleia2: I am going to try forward proting jeblair's pootle SSO change to 2.6.0-rc1 now17:27
krotscheckI’m confused, what are we doing with storyboard?17:27
clarkbpleia2: hopefully that keeps us moving17:27
nibalizerkrotscheck: split it out into its own module, that then is consumed by infra/config17:28
anteayanibalizer: your agenda item stays, the third party agenda item goes17:28
pleia2clarkb: thanks17:28
anteayanibalizer: I'm directing third party folks to track you and your work17:28
anteayanibalizer: and review and help17:28
krotschecknibalizer: Oh, neat!17:28
clarkbasselin__: but first I am going to apply that change somewhere as a noop then hopefully merge it17:28
pleia2clarkb: learned that zanata does support openid too (their live version has it), fiddling with it on my own demo server now, I think they have out of date instructions again :\17:28
krotschecknibalizer: Does that mean I have to move my patches to other places?17:29
clarkbpleia2: :/17:29
asselin__clarkb, sure17:29
anteayaclarkb: the trusty and tox agenda items don't need air time this week, do they?17:29
*** kmartin has quit IRC17:29
nibalizerkrotscheck: so what pending patches do you have?17:29
nibalizerjust the rabbit one or are there more?17:29
krotscheckThere’s one more, sec...17:29
nibalizeri thought your larger refactor landed17:30
krotschecknibalizer: https://review.openstack.org/#/c/112998/17:30
*** kmartin has joined #openstack-infra17:30
krotscheckBut I can move those as long as we can get it done this week.17:30
* krotscheck is doing honeymoon things next week.17:30
anteayakrotscheck: congratulations17:30
krotscheckanteaya: Thanks!17:30
nibalizerkrotscheck++ awesome!17:30
*** gyee has joined #openstack-infra17:31
*** MarkAtwood has quit IRC17:32
*** yamamoto has joined #openstack-infra17:32
clarkbanteaya: nope, got that settled last time. thanks17:32
clarkbanteaya: doing it on the 20th17:32
*** stevemar has quit IRC17:34
*** gyee has quit IRC17:34
clarkbmordred: fungi: reviews on https://review.openstack.org/#/c/111094/ https://review.openstack.org/#/c/111853/ and https://review.openstack.org/#/c/112431/ would be great this week17:35
anteayaclarkb: kk17:35
clarkbmordred: fungi: if we can get that stuff merged before sdague gets back from vacation that will allow him to make ^ work17:35
clarkbalong with adam_g and devananda17:35
*** yfried__ has joined #openstack-infra17:36
*** emagana has quit IRC17:36
*** whoops has joined #openstack-infra17:36
*** yamamoto has quit IRC17:37
*** emagana has joined #openstack-infra17:37
*** esker has quit IRC17:38
mordredkrotscheck: ok. if I can do autogenerated email and it gets updated, then I'll just do that17:39
mordredkrotscheck: that makes it easy17:40
*** rcarrillocruz has joined #openstack-infra17:40
*** salv-orlando has joined #openstack-infra17:40
*** e0ne has joined #openstack-infra17:40
*** rcarrill` has quit IRC17:41
clarkbfungi: mordred ok https://review.openstack.org/#/c/96551/ appears to apply cleaning to es03 and review.o.o17:41
clarkbany opposition to the method of handling unbounnd there before I approve17:41
*** mriedem has quit IRC17:41
mordredclarkb: fine by me17:42
*** eglynn has joined #openstack-infra17:44
fungiclarkb: sorry, was stuffing my face17:45
fungiclarkb: are you planning to rebase it or something?17:46
fungioh, nevermind. i misread one of the needed-by patches as depends-on17:46
clarkbfungi: ya no rebase, just wanted to make sure people are onboard17:47
*** melwitt has joined #openstack-infra17:47
openstackgerritOscar Romero proposed a change to openstack-infra/jeepyb: Fix close_pull_requests from jeepyb  https://review.openstack.org/11359817:47
clarkbalso a thunderstorm is rolling through so this could be fun17:47
*** dims has joined #openstack-infra17:47
clarkbyou know what github needs? git review -d17:48
*** dims is now known as Guest6125417:48
*** flwang_ has joined #openstack-infra17:48
jeblairhello all17:52
dprinceclarkb: hi sir, so I created the ACL here https://review.openstack.org/#/c/112331/3. Will that automatically create the group in gerrit, or is that a manual step?17:52
anteayahello jeblair17:52
*** flwang_ has quit IRC17:52
clarkbjeblair: o/ we have new pootle stories17:52
clarkbdprince: it will create the group17:52
jrollclarkb: github kind of has -d https://gist.github.com/piscisaureus/334224717:53
*** dmsimard is now known as dmsimard_away17:53
dprinceclarkb: cool, thanks17:53
clarkbjroll: oh I fetched the pr I awnted just fine17:53
jeblairclarkb: i caught the gist that their implementation of openid has an impedence mismatch with mine...17:53
clarkbjroll: it just requires jumping through hoops17:53
clarkbjeblair: a couple of them yes17:53
jrollclarkb: yeah, a little annoying17:54
clarkbjeblair: I am working on forward porting your patch to 2.6.0-rc1 but there are a few conflicts I am not sure how to fix yet17:54
marcusvrnclarkb: looking at merger-logs, I found that the problem is when try to execute this command: "git clone -v ssh://hds-ci@review.openstack.org:29418/openstack-dev/sandbox /var/lib/zuul/git/openstack-dev/sandbox"....but looking at /var/lib/zuul/ssh/ the private key is there...17:54
jrolldid any devstack cores show back up today? :)17:54
jeblairclarkb: let me know if i can help.  i remember little of pootle, but the auth stuff is mostly basic django; i have a bit of openid knowledge17:54
marcusvrnclarkb: the returning error is : "Cloning into '/var/lib/zuul/git/openstack-dev/sandbox'...17:55
marcusvrnPermission denied (publickey).17:55
marcusvrnfatal: The remote end hung up unexpectedly17:55
clarkbjeblair: in this case they appear to have removed enough of the auth mechanism and handed it off to allauth that pootle doesn't provide its own login page?17:55
clarkbmarcusvrn: the merger is using ssh and public key auth to clone that repo17:55
clarkbmarcusvrn: but that is failing. make sure you are using the correct key17:55
jeblairclarkb: neat.  does allauth not have an openid single sign on mode?17:56
*** mriedem has joined #openstack-infra17:56
clarkbjeblair: nope, and it doesn't work with pootle as implemented17:56
*** gyee has joined #openstack-infra17:56
jeblairthat's kind of confusing17:56
clarkbjeblair: so basically they rejected your thing, implemented allauth, closed the bug and never tested that it actualy fixed the bug17:57
*** pdmars has quit IRC17:57
jeblairclarkb: neat.  to be fair, i did zero followup on that, mostly because we couldn't decide if we actually wanted to use pootle.17:57
clarkbjeblair: https://github.com/translate/pootle/issues/3244 and https://github.com/translate/pootle/issues/324317:57
marcusvrnclarkb: the private key that need to be there must be the pair of public key sent to the openstack-infra team?17:58
*** vhoward has left #openstack-infra17:58
clarkbjeblair: maybe you can fill in the sso use case on that first issue17:58
clarkbmarcusvrn: yes17:58
clarkbmarcusvrn: also make sure you are using teh correct username for the account17:58
clarkbmarcusvrn: you will get failures like that if usernames don't match17:58
*** dkliban_afk is now known as dkliban17:59
clarkbjeblair: in particular with allauth pootle/apps/pootle_profile/urls.py is deleted in pootle17:59
clarkbjeblair: because allauth deals with it17:59
clarkbjeblair: pootle/templates/user_profile_menu.html is also gone for similar reasons17:59
*** mestery has quit IRC18:00
*** sabeen3 has joined #openstack-infra18:01
jeblairclarkb: http://bazaar.launchpad.net/~django-openid-auth/django-openid-auth/trunk/view/head:/README.txt18:01
jeblairclarkb: that's basically the process i followed in creating that patch18:02
*** sabeen1 has quit IRC18:02
mordredjeblair: did you see that we're going to need to do openid scraping again for lp->storyboard import?18:02
jeblairmordred: nope; that's a one-time thing though, right?18:02
*** amuller__ has quit IRC18:03
jeblairclarkb: i think issue https://github.com/translate/pootle/issues/3244  probably wants the allauth issue taken care of first, yeah?  because otherwise, it's hard to say what things pootle will need to pass through?18:03
clarkbjeblair: no I think pootle can implement the only allow openid bit concurently18:04
*** andreykurilin_ has joined #openstack-infra18:04
clarkbjeblair: further restricting which openid providers are allowed is what is needed in allauth18:04
*** pdmars has joined #openstack-infra18:04
*** abhishekk has joined #openstack-infra18:04
jeblairclarkb: okay, then i think you ought to reply with that because i don't know what settings to say need to be implemented :)18:04
*** e0ne has quit IRC18:04
clarkbjeblair: right now pootle always allows local auth18:05
clarkbat least that was my experience with it18:05
jeblairclarkb: however, i think 3243 might warrant a mention of http://openid.net/specs/openid-simple-registration-extension-1_1-01.html to provide usernames18:05
clarkbjeblair: can you update 3243 with that? I will respond to the other one18:06
zaroclarkb, jeblair, fungi, mordred : is an infra core able to approve puppetize installation of 3rd party plugins? (https://review.openstack.org/#/c/91193).  i'm able to babysit now.18:06
*** jamielennox|away is now known as jamielennox18:06
openstackgerritA change was merged to openstack-infra/storyboard: Updated storyboard documentation  https://review.openstack.org/10749018:06
marcusvrnclarkb: where can I check the username information?18:07
clarkbmarcusvrn: it should have been given to you in the email for the third party account18:08
*** otherwiseguy has quit IRC18:08
clarkbmarcusvrn: and your /etc/zuul/zuul.conf should have an entry for it under the gerrit section18:08
clarkbasselin__: optional unbound is approved18:09
*** linkid has left #openstack-infra18:09
asselin__thanks! :)18:09
krotschecknibalizer: I recall there being a nodeps option in the install_modules script so as to avoid version problems. That seems to have dissapeared… is it ok for me to just assume module dependencies will be run?18:09
jeblairzaro: done18:09
nibalizerkrotscheck: so if you want to have nodeps use the source install situation18:10
nibalizerwell there is a way to install from source now18:10
nibalizerwhich gets nodeps for free18:10
krotscheckAlright, let me consider that. I _think_ deps resolution for upgradeing ot rabbitmq 4.0.0 is ok though...18:11
jogodid the e-r bot die? I don't see any IRC messages of late18:14
jogoand I am gettint reports of it not posting comments18:14
jeblairjogo: checking18:14
jogojeblair: thanks18:15
openstackgerritA change was merged to openstack-infra/config: puppetize installation of gerrit third party plugins  https://review.openstack.org/9119318:15
*** loki184 has quit IRC18:15
jeblairit's still running; it may be netsplit18:15
jeblair2014-08-12 18:15:38,349 DEBUG recheckwatchbot: Console logs not ready for gate-trove-python26 109824,6,ad2472418:16
jogojeblair: thanks, no stacktraces it the logs I presume18:16
jogojeblair: ahh18:16
jogothat explains it18:16
*** dmsimard_away is now known as dmsimard18:16
jogojeblair: its not commenting because its taking too long to get the logs from ES18:16
jeblairnope, es requests are returning quickly18:16
*** zehicle_at_dell has joined #openstack-infra18:17
jeblairjogo: or yes, depending on exactly what your question was :)18:17
jogojeblair: hmm no how long it takes for ES to respond but the logs not getting added in the 15 minute window18:17
jeblairjogo: however, the graph does not suggest a backlog18:18
clarkbits possible that particular log is backed up but there is no consistent backlog18:18
jogoclarkb jeblair: hmm yeah I see recent logs in ES18:18
*** bhuvan_ has joined #openstack-infra18:19
*** krtaylor has joined #openstack-infra18:19
*** marun has quit IRC18:21
*** stevemar has joined #openstack-infra18:21
clarkbjeblair: https://github.com/cboylan/pootle/compare/translate:stable/2.6.0...openid going to see if something as simple as that will work18:21
*** marun has joined #openstack-infra18:21
*** markmcclain has joined #openstack-infra18:21
*** bhuvan has quit IRC18:22
jogoclarkb jeblair: hmm ER doesn't appear to be commenting18:23
jogoI found https://review.openstack.org/#/c/111837/18:23
jogowhich matched18:23
jogomessage:"Failed to commit reservations" AND tags:"screen-n-cond.txt" AND loglevel:"ERROR"18:23
jogobut has no ER comment18:23
jeblairjogo: yeah, it's still waiting for that trove job18:23
jeblairjogo: oh, you expected it to comment because that job would have completed before the trove one?18:24
*** rcarrill` has joined #openstack-infra18:25
*** pelix has quit IRC18:25
jeblairjogo: http://paste.openstack.org/show/93983/18:26
*** hdd has joined #openstack-infra18:26
jeblairjogo: when i run that now, i get a response, but at the time, it did not.18:26
*** rcarrillocruz has quit IRC18:26
clarkbjeblair: 'django_openid_auth.useropenid: 'user' defines a relation with the model 'auth.User', which has been swapped out. Update the relation to point at settings.AUTH_USER_MODEL.' so we will need to patch django_openid_auth too18:28
jeblairclarkb: what's the source of that?18:31
clarkbjeblair: running pootle setup which does db migrations18:31
jogojeblair: hmm18:31
jogojeblair: can you share the last few hours of e-r logs?18:31
*** Longgeek has quit IRC18:32
clarkbjeblair: maybe that is a django version thing?18:32
jeblairclarkb: what version are you using?18:32
*** yamamoto has joined #openstack-infra18:32
*** crc32 has joined #openstack-infra18:32
jeblairclarkb: https://docs.djangoproject.com/en/1.5/topics/auth/customizing/#auth-custom-user18:33
*** crc32 has quit IRC18:33
jeblairclarkb: trunk django-openid-auth supports django 1.6; i think they have just not implemented support for this optional new setting18:33
*** crc32 has joined #openstack-infra18:34
*** e0ne has joined #openstack-infra18:35
mordredjeblair: yes - the sync is one time - although it'll effectively be _everytime_ we run a sync18:36
*** hdd has quit IRC18:36
jeblairmordred: but this is just for importing existing bugs, which we will do once for infra, and once for openstack, right?18:36
*** yamamoto has quit IRC18:36
jeblairmordred: it's not an ongoing operational concern?18:37
mordredso it shouldn't be _terrible_18:37
jeblairmordred: what's the bit of data we need?18:37
mordredjeblair: well, we need a user account to deal with created_by and friends18:37
mordredjeblair: and user accoutns need openids, otherwise there's no way to match them when someone logs in18:38
mordredso we mainly just need the openid url thing18:38
*** gokrokve_ has joined #openstack-infra18:38
*** alexpilotti has quit IRC18:39
jeblairmordred: what was the new api call they added?  openid -> username?18:40
mordredjeblair: yah. exactly the opposite of what we needed18:40
mordredjeblair: it's ok - I've got the 'fetch openid' code working18:40
jeblairjogo: sanitizing the logs18:41
jeblairjogo: (it logs passwords)18:41
*** gokrokve has quit IRC18:41
*** eglynn has quit IRC18:41
*** homeless has quit IRC18:42
*** gokrokve_ has quit IRC18:43
openstackgerritA change was merged to openstack-infra/elastic-recheck: Add query for infra setup bug 1354829  https://review.openstack.org/11357518:43
uvirtbotLaunchpad bug 1354829 in tempest ""sudo: 3 incorrect password attempts" in host setup" [Undecided,Invalid] https://launchpad.net/bugs/135482918:43
jogojeblair: thanks18:45
jeblairjogo: http://status.openstack.org/debug.log.txt18:45
jeblairjogo: let me know when you have a local copy of that so i can rm it18:46
abhishekkdstanek:hi, you there?18:46
jogojeblair: got it18:46
*** hdd has joined #openstack-infra18:46
openstackgerritMatt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for ceilometer py26 bug 1355983  https://review.openstack.org/11361218:47
uvirtbotLaunchpad bug 1355983 in ceilometer "gate-ceilometer-python26 failing with "Could not find mongod command"" [Undecided,New] https://launchpad.net/bugs/135598318:47
jogoaha Traceback (most recent call last):18:48
jogo  File "/usr/local/lib/python2.7/dist-packages/elastic_recheck/bot.py", line 211, in run18:49
jogo    debug=not self.commenting)18:49
jogo  File "/usr/local/lib/python2.7/dist-packages/elastic_recheck/elasticRecheck.py", line 333, in leave_comment18:49
jogo    msg = msgs['found_bug'] % {'bugs': event.bug_list()}18:49
jogofound the bug18:49
jogojeblair: thank you18:49
jeblairjogo: thank you"18:49
*** Ajaeger has joined #openstack-infra18:49
*** abhishekk has quit IRC18:52
clarkbdjango.db.utils.OperationalError: (1170, "BLOB/TEXT column 'claimed_id' used in key specification without a key length") anyone know how to set the foreignkey legnth restriction with django orm?18:52
*** gokrokve has joined #openstack-infra18:52
clarkbalso I think that is an allauth bug...18:52
*** hdd has quit IRC18:52
clarkbbasically allauth isn't quite mysql safe18:52
*** MarkAtwood has joined #openstack-infra18:54
*** ianw has joined #openstack-infra18:58
pleia2yeesh, openid is hard18:59
clarkbmeeting time19:00
*** zehicle_at_dell has quit IRC19:00
*** dimtruck is now known as zz_dimtruck19:00
fungithe openid spec is a bit more like retroactive codification of the original implementation, from what i gather, iterated over with some incremental improvements19:00
pleia2fungi: that seems like the best way to do things ;)19:01
openstackgerritAndreas Jaeger proposed a change to openstack-infra/config: Move openstack-security-notes to attic  https://review.openstack.org/11361419:01
fungiwell, it was developed as an auth api-ish for lj19:01
*** zz_dimtruck is now known as dimtruck19:01
clarkbin this cause its just django plugins only working in the initial lego set they came in I think19:01
fungiif my memory is correct19:01
pleia2huh, I didn't know that19:01
clarkbthere is clearly very little testing of ow they fit together19:02
*** hashar has joined #openstack-infra19:02
fungiworth noting, though, that most early ietf standards fall into the same category ("this is how we implemented protocol in tool foo")19:03
*** reed has quit IRC19:03
stevemarjust a heads up - possible new infra bug: 135599819:04
*** pdmars has quit IRC19:04
fungibug 135599819:05
uvirtbotLaunchpad bug 1355998 in devstack "PostgreSQL's request for a shared memory segment exceeded your kernel's SHMMAX parameter" [Undecided,New] https://launchpad.net/bugs/135599819:05
*** SergeyLu_ has joined #openstack-infra19:05
clarkbsounds like a postgres bug >_>19:05
*** rcarrillocruz has joined #openstack-infra19:05
*** SergeyLu_ is now known as SergeyLukjanov219:05
openstackgerritMichael Krotscheck proposed a change to openstack-infra/config: Create storyboard configuration directory  https://review.openstack.org/11361619:06
*** rcarrill` has quit IRC19:07
zarojeblair: review & review-dev are not automatically updated on merges to config.  I don't remember if that's on purpose or not but i have manually updated review-dev and have verified that the change to automate 3rd party plugin installation did work as expected.19:08
*** rcarrill` has joined #openstack-infra19:08
fungistevemar: search on logstash. looks like it started around 1500-1530 when the nodepool images would have been finishing their daily update. the build_node column indicates that it's only impacting devstack-trusty nodes on hpcloud-b119:09
fungistevemar: so i'll delete that image19:09
fungiclarkb: ^ worth digging into the image update log for that when our meeting concludes19:10
clarkbfungi: well if it is an issue with our kernel settings it will pop up tomorrow too19:10
fungiclarkb: why only on one image in one provider though?19:10
*** annegent_ has quit IRC19:10
fungi(e.g. not impacting hpcloud-b2 through b5)19:10
*** pdmars has joined #openstack-infra19:11
*** dimtruck is now known as zz_dimtruck19:11
clarkbdid the image update there?19:11
*** rcarrillocruz has quit IRC19:11
clarkbor maybe it was a race around kernel updates and when image updates ran19:11
clarkb(I don't have a good answer, but the error message indicates kernel settings)19:12
*** gema_ has joined #openstack-infra19:12
openstackgerritAishwarya Thangappa proposed a change to openstack-infra/storyboard: Added a subscription_events table & subscription hander  https://review.openstack.org/11301619:14
aviauCan someone help me with this error? http://logs.openstack.org/87/113587/1/check/gate-config-layout/7dfb8ee/console.html19:15
*** gema has quit IRC19:15
aviauhere is the changeset: https://review.openstack.org/#/c/113587/19:15
aviauIt looks like the check script is failing?19:16
openstackgerritMichael Krotscheck proposed a change to openstack-infra/storyboard: Added subscription API  https://review.openstack.org/10851619:16
*** otherwiseguy has joined #openstack-infra19:16
clarkbaviau: that isn't quite how you specify a template iirc. grep for template in that file and you should see the diff19:17
openstackgerritJoe Gordon proposed a change to openstack-infra/elastic-recheck: Fix MessageConfig  https://review.openstack.org/11362019:18
aviauclarkb, My bad! =)19:18
*** imcsk8 has joined #openstack-infra19:18
*** mrmartin has joined #openstack-infra19:18
jogojeblair: the fix ^19:19
Ajaegeraviau: use "- name: docs-on-rtfd"19:19
*** andreykurilin_ has quit IRC19:19
imcsk8hello, is there any problem with the jenkins jobs of review.openstack.org?19:19
pleia2imcsk8: what are you having trouble with?19:20
Ajaegerimcsk8: jenkins has quite some work in the queue do - see status.openstack.org/zuul19:20
*** ihrachyshka has quit IRC19:21
openstackgerritAlexandre Viau proposed a change to openstack-infra/config: Added rtfd template to Surveil  https://review.openstack.org/11358719:21
*** stevemar has quit IRC19:22
aviauAjaeger, Done! Thanks19:22
imcsk8pleia2, Ajaeger some of my reviews are taking too long to merge and get checked, i guessed that jenkins was really busy, i just needed to confirm. thanks!19:22
Ajaegeraviau: lgtm19:23
*** rlandy has quit IRC19:23
Ajaegerimcsk8: check on the status page for your review and see where it is in the queue...19:24
pleia2jogo: you in town at all these next couple weeks? I could use some jogo brain picking time :)19:24
*** Ryan_Lane has quit IRC19:24
*** e0ne has quit IRC19:24
*** gokrokve has quit IRC19:26
*** mestery has joined #openstack-infra19:27
openstackgerritMonty Taylor proposed a change to openstack-infra/storyboard: Add loader subcommand to sync from launchpad  https://review.openstack.org/11362419:27
mordredkrotscheck: ^^19:27
openstackgerritA change was merged to openstack-infra/config: Cleanup elasticsearch related config  https://review.openstack.org/11244819:31
*** annegent_ has joined #openstack-infra19:32
*** yamamoto has joined #openstack-infra19:32
*** ramashri has quit IRC19:32
*** stevemar has joined #openstack-infra19:34
krotscheckmordred: So, I’m not entirely certain that that script should live in storyboard.19:35
krotscheckFor now, maybe…? It feels like too-specific a tool though.19:35
mordredI couldn't come up with a GREAT home for it19:35
mordredbut I tend to agree with you19:35
*** SergeyLukjanov2 has quit IRC19:36
nibalizerits kindof a oneshot19:36
*** todd_dsm has quit IRC19:36
nibalizeralso i think i was supposed to write that once, sorry about that19:36
* mordred stabs nibalizer in face19:36
* nibalizer dies19:36
* nibalizer regenrates19:36
fungiwell, it might not be oneshot for other projects who want to run their own storyboard instance and migrate off lp, but the audience is probably pretty limited19:36
nibalizerbut yea after we do the migration we're gonna throw the script away and never test to see if it works ever again right?19:37
openstackgerritMonty Taylor proposed a change to openstack-infra/storyboard: Add loader subcommand to sync from launchpad  https://review.openstack.org/11362419:37
krotschecknibalizer fungi mordred: Well, wait a sec.19:37
*** yamamoto has quit IRC19:37
openstackgerritA change was merged to openstack-infra/config: Make use of unbound optional  https://review.openstack.org/9655119:37
krotscheckOne of the long-term visions is the ability to set up remote federated sources for stories. I.e. if you have a dependency on a project that’s still tracked in launchpad, you can slurp those in and keep track of them. This _could_ just be the first step.19:38
mordredI mean, it would need work and data model additions and stuff - but one could imagine a world in which the script could be slightly more generally useful19:38
krotscheckSo it’d evolve into being a sync mechanism with pluggable upstream providers.19:39
*** _nadya_ has joined #openstack-infra19:39
krotscheckWith, perhaps a “one time only” or a “continuous” flag.19:39
*** ArxCruz has quit IRC19:40
openstackgerritA change was merged to openstack-infra/jenkins-job-builder: Allow `test` command to use conf files  https://review.openstack.org/10562019:41
*** shashankhegde has quit IRC19:41
*** Ryan_Lane has joined #openstack-infra19:41
mordredttx: btw ... please see https://review.openstack.org/11362419:43
*** annegent_ has quit IRC19:43
*** raildo has left #openstack-infra19:43
*** _nadya_ has quit IRC19:44
fungikrotscheck: oh, interesting (and potentially useful for our phased migration plans)19:44
mordredttx: I split it into two pieces, so you can run the first and get it to spit out a json file with the lp data for verification19:44
mordredttx: since you are the kind of this area - I'd love verification from you that you think it does what you think it should do19:44
krotscheckmordred: I’ll look at your thing as soon as I figure out why this rabbit patch is being stupid19:45
ttxmordred: will.. try to look at it. I'm supposedly on vacation starting today until the end of the week19:45
mordredkrotscheck: I blame your barolo19:46
ttxI'm supposed to fix my roof.19:46
mordredttx: sure. I mean, I can also just soldier forward - I think you did the important thing which was look at the data mapping19:46
fungittx: they have professionals who do those sorts of things for you now ;)19:46
krotscheckmordred: Dude. I pulled two bottles of that before the wedding just in case people loved it. And now those are the only two bottles left.19:46
ttxfungi: they just replace my old tiles though. I like my old tiles.19:46
*** e0ne has joined #openstack-infra19:46
krotscheckmordred: I have a whole case of the pinot though.19:46
fungittx: good point19:46
ttxI'll post pictures.19:47
* fungi assumes terracotta tile roof19:47
clarkbfungi: stevemar http://www.postgresql.org/docs/9.1/static/kernel-resources.html postgres says linux defaults are bad. devstack should probably update them19:48
*** david-lyle has quit IRC19:49
*** flwang_ has joined #openstack-infra19:49
*** david-lyle has joined #openstack-infra19:49
*** marun has quit IRC19:50
*** marun has joined #openstack-infra19:50
fungiclarkb: good to know. did they *just* change something wrt that and for some reason it only impacted one image build in one provider?19:51
mordredkrotscheck: both were delicious19:51
*** vhoward has joined #openstack-infra19:51
clarkbfungi: I don't think they did. maybe it has to do with other services using some of the shared memory?19:51
openstackgerritSpencer Krum proposed a change to openstack-infra/config: Send the puppet-dashboard module to the attic  https://review.openstack.org/11362719:51
*** david-ly_ has joined #openstack-infra19:52
fungiclarkb: maybe. i'm hoping when i have time to look at the image update log for that one it'll be some sort of obvious breakage which our prep scripts ignored19:52
clarkbdtroyer: ^ basically the postgres devstack configurator may want to increate the shared memory space on linux to a larger value19:52
*** flwang_ has quit IRC19:53
rcarrill`guys, i got drowned with some stuff at work and haven't had time to push a spec i had discussed at mid-cycle about issue tracker agnostic jeepyb, aka plugin based jeepyb19:53
*** david-lyle has quit IRC19:53
rcarrill`am I on time?19:53
rcarrill`not familiar on the spec procedure, i have time till the feature freeze to push?19:54
*** dkliban is now known as dkliban_brb19:54
clarkbrcarrill`: infra doesn't follow normal feature freeze19:55
clarkbrcarrill`: you are fine19:55
fungircarrill`: we don't so much have an infra feature freexe19:55
fungiwe do sort of slush up a bit around releases to avoid screwing up our change throughput for the other projects19:55
*** unicell has quit IRC19:55
*** SumitNaiksatam has quit IRC19:55
*** esker has joined #openstack-infra19:55
*** resker has joined #openstack-infra19:57
*** gokrokve has joined #openstack-infra19:57
*** AlexF has joined #openstack-infra19:57
*** esker has quit IRC19:57
*** shashankhegde has joined #openstack-infra19:57
*** esker has joined #openstack-infra19:57
stevemarclarkb, fungi cool. thanks for looking into it19:58
jogopleia2: I'm around the first half the week19:59
jogothen off to chicago19:59
dtroyerclarkb: is the shmmax value in the gate images left at the default?19:59
clarkbdtroyer: we don't touch it, it should be the default as set by ubuntu cloud images20:00
pleia2jogo: lunch on monday?20:00
jogopleia2: sounds good20:00
pleia2speaking of which, lunchtime :)20:00
*** e0ne has quit IRC20:00
openstackgerritAndreas Jaeger proposed a change to openstack-infra/config: Move openstack-security-notes to attic  https://review.openstack.org/11361420:01
*** resker has quit IRC20:01
tchaypo#info Last week's meeting notes got misfiled under http://eavesdrop.openstack.org/meetings/tripleo_weekly/ - I'll ping infra to see if we can get them moved20:01
rcarrill`fungi: ok, thx, i 'hope' to be able to push the spec next week20:01
clarkbtchaypo: ww20:01
tchaypoWho could I talk to about getting ^^ moved back with the rest of the tripleo meetings?20:01
clarkbor did you copy pasta?20:02
tchaypoI deliberately pasted, it was in my buffer from being ready to mention in our meeting20:02
*** amotoki has quit IRC20:02
clarkbI see, I can copy pasta the logs. where should they go?20:02
tchaypounder /meetings/tripleo/20:02
*** jedimike has quit IRC20:03
fungitchaypo: i assume someone did #startmeeting tripleo_weekly last week?20:03
clarkbpleia2: jeblair: I wanted to ask in the meeting if we need to chagne tactics for pootle to be ready by the 20th?20:03
tchaypoyep. I'm updating the wiki to clarify that..20:03
*** stevemar has quit IRC20:03
*** MarkAtwood has quit IRC20:04
anteayajeblair: so what is your decision on ci renames?20:05
*** zz_dimtruck is now known as dimtruck20:05
anteayajeblair: I'll support whatever you decide20:05
anteayaI'm just asking by way of a heads up for myself20:05
clarkbtchaypo: done20:06
* fungi grumbles at lack of any obvious error in image update log20:06
tchaypoclarkb: <320:06
clarkbfungi: I really don't think this is an error20:06
fungiclarkb: have a theory on why it would impact only one nodepool provider out of 8?20:07
clarkbfungi: packages20:07
openstackgerritDavanum Srinivas (dims) proposed a change to openstack-infra/elastic-recheck: Add query for Postgres startup failure  https://review.openstack.org/11363420:07
*** pdmars has quit IRC20:08
*** homeless has joined #openstack-infra20:08
clarkbpostgres asked for >32MB of shared memory20:08
clarkbdefault max is 32MB20:08
fungiclarkb: different package versions on hpcloud-b1 than the others you think? got balanced to a different mirror?20:08
clarkbeither new postgres or new kernel package affected the numbers?20:08
openstackgerritMonty Taylor proposed a change to openstack-infra/storyboard: Add loader subcommand to sync from launchpad  https://review.openstack.org/11362420:08
clarkbfungi: ya, or the image builds raced mirror updates20:08
fungii'll run a comparison20:08
openstackgerritMonty Taylor proposed a change to openstack-infra/storyboard: Assocate StoryTag with Story via mapping  https://review.openstack.org/11363520:08
fungisee whether this is a precursor to everywhere breaking tomorrow20:09
mordredkrotscheck: ^^ there's the bug I found in StoryTag20:09
mordredkrotscheck: basically, it was not possible to actually have the same tag on more than one story20:09
*** andreaf has quit IRC20:10
*** bhuvan_ has quit IRC20:10
*** andreaf has joined #openstack-infra20:12
*** rcarrillocruz has joined #openstack-infra20:12
clarkbthere is a lot of purple on our graph but we don't appear to have leaked floating ips like crazy20:14
clarkband the hpcloud error rate is much lower than yesterday20:14
clarkbI am guessing that is related to nova on rax side cleanup then?20:14
*** rcarrill` has quit IRC20:14
clarkblooks like a small chunk of nodes in delete state for a day or so20:15
*** pdmars has joined #openstack-infra20:16
*** MarkAtwood has joined #openstack-infra20:17
mordredclarkb: so why were you opposed to re-using floating-ips again/20:18
clarkbfungi: did you delete the image that failed with postgres shmmax?20:18
*** gema has joined #openstack-infra20:18
clarkbmordred: beacuse the cleanup aspect of it is hard20:18
krotscheckAnyone here familiar with rabbitmq’s permissions structure? I cannot for the sake of me figure out why I can’t connect with a non-admin user.20:18
*** gema_ has quit IRC20:19
clarkbmordred: right now with a 1:1 relationship cleanup is easy20:19
mordredclarkb: can you give me _slightly_ more to go on than that? I'd like to think about whether or not it's hard or impossible20:19
clarkbmordred: sure. So right now we use all 600 ips in our quota, but on the weekend we only need a max of 10020:19
clarkbmordred: nodepool needs to free the 500 it isn't using20:19
mordredclarkb: why?20:20
mordred(just for sake of argument)20:20
clarkbmordred: because they cost money20:20
mordredfair enough20:20
clarkbwe have users of nodepool that explicitly want to use it to use only the resources they need20:20
*** pdmars_ has joined #openstack-infra20:20
ianwfungi: a lot of check jobs seem to be waiting for dsvm-f20 ATM20:21
mordredclarkb: so, what would you think about a brief logic of "if there is a floating ip that is unattached, I should try to use it, and node deletion should delete floating ips"20:21
mordredclarkb: so that if we happen to leak the occasional ip, it gets re-used20:21
mordredclarkb: but we still reap as usual20:21
clarkbmordred: that races the allocation of the ip20:21
clarkbmordred: unfortunately neutron/nova don't really indicate this well to their users20:21
ianwseveral solum jobs20:21
anteayacinder sprint question, if core abandons a patch can the owner unabandon the patch?20:22
*** pdmars has quit IRC20:22
clarkbanteaya: yes I think so, you cn always hve them test it20:22
clarkbianw: we are behind on node alloactions20:22
clarkbianw: and your patch was reverted20:22
homelesshi guys, I was having problems with jeepyb and the close pull request behavior for github, then I found that the root cause was an if statement in the close_pull_requests.py, if its possible, could someone review this one, https://review.openstack.org/#/c/113598/20:22
*** andreaf_ has joined #openstack-infra20:23
*** flaper87 is now known as flaper87|afk20:23
fungiclarkb: i did delete it... we should be running off yesterday's image there at this point20:23
ianwclarkb: yeah, just more f20 use than i was aware of20:23
mordredjogo, mikal: see above conversation about floating ips and nodepool for a real-world example of why I think nova should grow MORE smarts around proxying to neutron20:23
mordredjogo, mikal: specifically, nova boot --gimme-floating-ip should be a thing I should be able to ask my cloud to do20:24
mordredbeause it's REALLY hard to get right as a user20:24
mordredalso, you should see the floating-ip related patches I recently landed upstream in ansible20:24
mordredthey'll make you cry a gbit20:24
jogomordred: ++ good idea20:25
*** lbragstad has joined #openstack-infra20:25
jogosalv-orlando marun: ^20:25
*** andreaf has quit IRC20:26
clarkbfungi: ok, I was going to say we could boot a node off of it and check the shmmax20:26
fungiclarkb: i may still be able to hold one. checking20:26
*** dprince has quit IRC20:26
mordredmarun, salv-orlando, jogo: https://github.com/ansible/ansible/blob/devel/library/cloud/nova_compute#L37020:28
jogomordred: hehe, I am talking to arosen about this in #openstack-neutron20:28
*** amcrn has joined #openstack-infra20:28
fungiclarkb: was booted over an hour ago and has 3355443220:29
pleia2clarkb: re pootle: we could just set it up with boring old logins for now, and if we actually decide to use it we'll keep pushing on openid stuff20:29
clarkbfungi: that looks correct to me20:29
fungichecking now to see if that's booted from the image i deleted or the other20:29
clarkbpleia2: that is an option20:29
jogomordred: so you should join the room and join  the fun20:29
pleia2my zanata demo just has plain logins, and it looks like SSO for us on that will take work too, so we have work there20:30
fungiclarkb: 2014-08-12 19:10:41,532 INFO nodepool.NodeLauncher: Creating server with hostname devstack-trusty-hpcloud-b1-1429306.slave.openstack.org in hpcloud-b1 from image devstack-trusty for node id: 142930620:30
clarkbfungi: is that the old image?20:31
fungiclarkb: auth.log says i deleted the image at 19:30:2420:31
marcusvrnclarkb: now it's working!! the problem was that zuul didn't find the private key on /home/zuul/.ssh/ so it couldn't clone git repo20:31
anteayaclarkb: yes you are correct we just tested: 9223320:31
fungiclarkb: so it should be booted from the old image20:31
*** emagana has quit IRC20:32
clarkbfungi: ok so I think the kernel is doing the correct thing there. it looks a lot like my shmmax and the documented default20:32
*** yamamoto has joined #openstack-infra20:32
clarkbfungi: so signs would point at postgres asking for more all of a sudden?20:32
*** emagana has joined #openstack-infra20:32
* salv-orlando reading scrollback20:32
clarkbfungi: oh wait, devstack isntalls the packages. we cache them but it should udpate the cache if necessary right?20:33
fungiclarkb: the last job to report with that error and get indexed into logstash was at 19:47:00 but i'll wait and see if we grow more of them20:33
*** ramashri has joined #openstack-infra20:33
openstackgerritMichael Krotscheck proposed a change to openstack-infra/config: Added RabbitMQ to StoryBoard  https://review.openstack.org/9847420:33
clarkbfungi: so probably not purely a package issue20:33
krotscheckAnyone want to review this? I promise it’s quick. https://review.openstack.org/#/c/113616/120:33
fungikrotscheck: 0400 for a dir in /etc? full of super secret stuff?20:34
krotscheckfungi: SUPER secret.20:34
krotscheckfungi: It’s my default “hey there’s passwords in here go away” perms20:34
openstackgerritA change was merged to openstack-infra/elastic-recheck: Fix MessageConfig  https://review.openstack.org/11362020:35
clarkbkrotscheck: done20:35
salv-orlandomordred, jogo: a sort of macro then? boot a vm and allocate also a floating IP (for all interfaces by default)20:35
fungikrotscheck: makes sense for the config file, but the config dir?20:35
clarkband I -1'd for that raisin20:35
clarkbfungi: see my omment on the change20:35
*** Ajaeger has quit IRC20:35
fungithe "security guy" is unparanoid enough to vote for 0755 on the dir20:36
clarkbit should be 0500 or 070020:36
openstackgerritMichael Krotscheck proposed a change to openstack-infra/config: Create storyboard configuration directory  https://review.openstack.org/11361620:36
*** julim has quit IRC20:36
clarkb0400 will only work by way of puppet weirdness20:36
*** emagana has quit IRC20:36
fungi0600 or 0400 on the file is expected, but the dir itself doesn't need that treatment20:36
*** yamamoto has quit IRC20:37
jogosalv-orlando: we are talking about this with arosen in -neutron20:37
krotscheckWanna look at it again?20:37
clarkbkrotscheck: yup +220:37
*** crc32 has quit IRC20:38
openstackgerritSean Roberts proposed a change to openstack-infra/config: fixup congress-specs infra configuration  https://review.openstack.org/11242320:38
openstackgerritSergey Lukjanov proposed a change to openstack/requirements: Remove d-g from projects.txt  https://review.openstack.org/11364320:38
clarkbpleia2: I would be fine using the plan login for now. I will go ahead and set up my server that way and give you an admin account20:39
clarkbpleia2: if nothing else it gets us through the demo phase20:39
openstackgerritIan Wienand proposed a change to openstack-infra/nodepool: Log error on over-allocation  https://review.openstack.org/11364420:39
pleia2clarkb: sounds good20:39
*** bhuvan has joined #openstack-infra20:40
*** Guest61254 has quit IRC20:41
krotscheckOk, nibalizer - that stability patch I asked for just got approved, once it gets merged we can land all the things.20:42
*** homeless has quit IRC20:44
*** stevemar has joined #openstack-infra20:44
openstackgerritMichael Krotscheck proposed a change to openstack-infra/config: Added RabbitMQ to StoryBoard  https://review.openstack.org/9847420:45
clarkbpleia2: ok thats done20:45
clarkbpleia2: but it looks like profile still break20:45
clarkbI think that bug may go deeper than just openid setting username to none20:45
*** homeless has joined #openstack-infra20:46
*** rfolco has quit IRC20:46
* pleia2 nods20:46
clarkbpleia2: if you register I think I can make you an admin20:46
*** dims has joined #openstack-infra20:46
pleia2clarkb: ip again?20:46
*** dims is now known as Guest6009220:46
openstackgerritOscar Romero (homeless) proposed a change to openstack-infra/jeepyb: Fix close_pull_requests from jeepyb  https://review.openstack.org/11359820:47
marunmordred, jogo: So the suggestion is that nova and/or neutron should know how to allocate floating ips rather than having to implement the same behavior in every single orchestration mechanism under the sun?20:47
pleia2clarkb: and can you also add how you set up django to the install notes?20:47
mordredmarun: yes20:47
krotscheckmordred: So about tags - is that going to be required for your data import? Because we’re still arguing about the spec.20:47
mordredmarun: we're talking about it in #openstack-neutron, btw20:47
*** dkliban_brb is now known as dkliban20:47
pleia2clarkb: registered as lyz20:47
mordredkrotscheck: oh! well, if we're still arguing about the spec, I can revert to just commenting out actually using them20:47
clarkbpleia2: I didn't have to do anything to setup django, pip installed it20:48
clarkbpleia2: I haven't gotten to using wsgi20:48
pleia2oh, neat :)20:48
ianwclarkb / fungi : is it possible to get nodepool logs into logstash?  totally new to logstash-agent, etc, digging around to see how it works now20:48
clarkbianw: I want to say no :) I have tried to not put system logs nd test logs in the same cluster20:48
clarkbianw: mostly because system logs can be much more sensitive and shouldn't be 100% public imo20:49
jogomarun: converation moved to #openstack-neutron20:49
clarkbianw: but sorting that out with logstash/elasticsearch is weird because they have rudimentary support for ACLs20:49
krotscheckmordred: Actually, nevermind. https://review.openstack.org/#/c/97211/20:49
marunjogo: got it, thanks.20:49
krotscheckmordred: No code has landed for that yet though, so the data might end up having to be migrated.20:50
ianwclarkb: yeah, that is a good policy.  just wondering how i can monitor for https://review.openstack.org/#/c/113644/20:50
mordredkrotscheck: so, I _think_ the patch I did is still in line with that spec20:51
mordredkrotscheck: the tags table probably needs more metadata to deal with system/protected or something ... but the many/many relationship seems like it's still needed?20:51
clarkbianw: maybe graphite and have nodepool bump a counter? that is really not enough info tough :/20:52
clarkbpleia2: arg I may have to update the db to give us superuser powers, this is so weird20:54
clarkbclearly there is a reason I haven't really used django before20:54
pleia2clarkb: hm, also having trouble going to my account page20:55
clarkbpleia2: ya I think that is just generally broken20:55
clarkbpleia2: regression from the allauth move20:55
ianwclarkb: hmm, yeah, knowing it happened would be enough for now20:55
* pleia2 nods20:55
*** rfolco has joined #openstack-infra20:56
clarkbpleia2: I updated the db directly you should be a super user now. can you log out then back in and see if you have an admin tab?20:57
pleia2clarkb: yep, thanks20:57
*** emagana has joined #openstack-infra20:59
*** markmc has quit IRC21:02
*** miqui has quit IRC21:02
*** praneshp has quit IRC21:03
*** praneshp has joined #openstack-infra21:04
*** bhuvan_ has joined #openstack-infra21:04
*** MarkAtwood has quit IRC21:05
*** bhuvan has quit IRC21:07
homelessHi clarkb, regarding https://review.openstack.org/#/c/113598/ (close pull requests) I think we cannot leave the if statement into one line because if the option seciont is not here, the it fails. look at this, http://paste.forj.io/show/305/21:09
*** pdmars_ has quit IRC21:09
clarkbhomeless: if you say 'if 'options' not in section or use options:' that will garuntee that options is there via short circuiting right?21:11
clarkbif it is not there you short circuit. if it is ther you go to the next thing21:11
homelessis what I thought, but as is an "or" its validate both conditions21:11
*** gondoi is now known as zz_gondoi21:12
mordredhomeless: or only touches the second if the first option is false21:12
mordredhomeless: if the first is true, it will not bother with the second side21:13
*** aysyd has quit IRC21:13
*** pblaho has joined #openstack-infra21:13
pleia2clarkb: can you: apt-get install python3-aeidon unzip iso-codes21:13
clarkbpleia2: on the translate host?21:14
pleia2clarkb: yep21:14
clarkbpleia2: we don't python3 there unless I am mistaken21:14
*** weshay has quit IRC21:14
*** ociuhandu has joined #openstack-infra21:14
clarkbpleia2: or is that just a tool?21:14
pleia2clarkb: ah, right, I'll see if something else provides aeidon21:15
mordredhomeless: http://paste.openstack.org/show/9404621:15
pleia2actually, no need for that really, I don't think we do "subtitles"21:15
pleia2unzip and iso-codes we need though21:15
clarkbpleia2: done21:16
*** radez is now known as radez_g0n321:17
openstackgerritAnita Kuno proposed a change to openstack-infra/config: Added a missing single quote  https://review.openstack.org/11365521:17
*** yamamoto has joined #openstack-infra21:17
*** mrda-away is now known as mrda21:18
nibalizerkrotscheck: wooooooot21:19
*** sandywalsh has quit IRC21:19
*** emagana has quit IRC21:20
openstackgerritA change was merged to openstack/requirements: Updated an explanation to the sqla requirement  https://review.openstack.org/11153121:20
krotscheckmordred: Yeah, I think your changes are still in line.21:20
mordredkrotscheck: ossum21:20
*** emagana has joined #openstack-infra21:21
homelessclarkb, mordred, this is my close_pull_requests.py file http://paste.forj.io/show/306/, and this is what I got http://paste.forj.io/show/307/21:22
homelessprobably im missing something but for me that doesnt work21:22
openstackgerritMichael Krotscheck proposed a change to openstack-infra/storyboard: Added subscriber and publisher modules  https://review.openstack.org/10284221:24
openstackgerritMichael Krotscheck proposed a change to openstack-infra/storyboard: Added subscription API  https://review.openstack.org/10851621:24
*** yamamoto has quit IRC21:24
openstackgerritMichael Krotscheck proposed a change to openstack-infra/storyboard: Added a subscription_events table & subscription hander  https://review.openstack.org/11301621:24
*** yfried__ has quit IRC21:25
*** dkranz has quit IRC21:25
*** ominakov has quit IRC21:25
*** mfer has quit IRC21:27
*** Guest60092 is now known as dims_21:27
anteayaso making a patch that makes a change to http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/jenkins_job_builder/config/projects.yaml21:28
anteayaI don't see any format for how to decide where to make my addition21:28
mordredhomeless: you're missing quotes21:28
anteayaso anywhere I feel like it?21:28
mordredhomeless: if 'options' not in section or 'has-pull-requests' in section['options']:21:29
nibalizerhrm how did i do this the first time21:31
homelessomg.. that is right, also on the has-pull-requests, Im sorry mordred21:31
homelessI just copy/paste the proposal from the comment on gerrit21:31
mordredhomeless: no worries! that's why we all talk all the time :)21:31
homelessyup, I will change it right now21:32
*** mrmartin has quit IRC21:32
*** lyxus has quit IRC21:33
*** marun has quit IRC21:33
*** whoops has quit IRC21:34
*** bknudson has quit IRC21:35
*** crc32 has joined #openstack-infra21:36
mordredjeblair: oh - btw - while you were out, my all of my fixes to the nova module to ansible landed21:36
YorikSaranteaya: Can you please lift -1 from https://review.openstack.org/112666 ? Doug has put his +1 there, repo is reviewed and considered good.21:37
openstackgerritAishwarya Thangappa proposed a change to openstack-infra/storyboard: Added a subscription_events table & subscription hander  https://review.openstack.org/11301621:37
openstackgerritOscar Romero (homeless) proposed a change to openstack-infra/jeepyb: Fix close_pull_requests from jeepyb  https://review.openstack.org/11359821:37
jeblairmordred: neat, that even took less than 6 months21:37
anteayaYorikSar: +1, thanks21:38
mordredjeblair: yah!21:38
mordredjeblair: so, once 1.8 comes out, there are several things I can drop from that ansible patch to config I put up the other day21:38
YorikSaranteaya: Thanks :)21:38
jeblairmordred: so ansible set low expectations, and then exceeded them and you were happy21:38
*** ominakov has joined #openstack-infra21:39
jeblair(as far as project workflow goes)21:39
mordredalthought to be fair, I did ping the project lead out of band21:39
mordredafter having had a beer with him21:39
jeblairthis is all useful information!21:40
mordredso it's entirely  possible that the key in any situation is social engineering21:40
jeblairmordred: i have a list of people for you to have beers with21:40
mordredjeblair: consider it done21:40
krotschecknibalizer: Need any help? I have a vested interest in getting a new feature into storyboard-puppet :)21:42
openstackgerritJoe Gordon proposed a change to openstack-infra/devstack-gate: Collect javelin2 log  https://review.openstack.org/11365821:42
openstackgerritAaron Rosen proposed a change to openstack-infra/nodepool: Remove unneeded call to removeFloatingIP in cleanupServer  https://review.openstack.org/11365921:42
anteayaso any comment on whether is is a viable job? http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/jenkins_job_builder/config/hooks.yaml21:43
nibalizerkrotscheck: i dont think what im doing now is easily parallelizable21:44
anteayaI'm making a patch for a stackforge repo, looking for the right job for publishing docs21:44
krotschecknibalizer: You got it. I’ll be standing in the wings if you nee dme.21:45
*** dims_ has quit IRC21:45
*** dims_ has joined #openstack-infra21:45
mordredanteaya: the rtfd hooks are the right way for stackforge stuff21:46
*** hashar has quit IRC21:47
anteayamordred: thanks21:48
openstackgerritA change was merged to openstack-infra/config: Create storyboard configuration directory  https://review.openstack.org/11361621:49
*** flwang_ has joined #openstack-infra21:50
openstackgerritJoe Gordon proposed a change to openstack-infra/config: Add javelin.log to logstash  https://review.openstack.org/11366321:51
jogomtreinish: patch is up in advance21:52
*** armax has quit IRC21:53
*** flwang_ has quit IRC21:54
*** homeless_ has joined #openstack-infra21:55
mtreinishjogo: I gave reviewed the other 221:56
mtreinishwow, I can't type today...21:56
anteayaSergeyLukjanov: are you about?21:57
anteayaSergeyLukjanov: cinder has a question about a hadoop dependency21:57
* anteaya is at the cinder sprint21:57
*** homeless has quit IRC21:58
*** doug-fish has left #openstack-infra21:59
*** homeless_ has quit IRC21:59
*** rcarrill` has joined #openstack-infra21:59
*** yamamoto has joined #openstack-infra21:59
*** homeless has joined #openstack-infra21:59
*** rcarrillocruz has quit IRC22:01
jogomtreinish: thanks22:02
lifelessmordred: https://review.openstack.org/#/c/94108/ - I'm losing hope22:02
*** bknudson1 has joined #openstack-infra22:02
dansmiththe check queue is growing crazy right now22:02
dansmithlike someone just rechecked everything22:02
clarkbdansmith: its node issues22:03
*** amotoki has joined #openstack-infra22:03
dansmithwow, okay22:03
clarkbdansmith: those upstream bugs in downstream clouds are killing us22:03
dansmithso is this just refilling the queue after a restart or something?22:03
clarkbdansmith: no, it is rax leaking nodes, hpcloud leaking floating ips, + a couple other bugs we know about22:04
dansmithso just to be clear,22:04
clarkbdansmith: that decreases our effective quota and results in overall less throughput22:04
dansmithten minutes ago, the check queue was under 100 and in the last three minutes it's gotten to about 30022:05
mordredlifeless: don't lose hope22:05
clarkbdansmith: oh I see, so the 100 is because of what I describe22:05
clarkbthe 300 is interesting22:05
dansmithyeah, the 300 is the "um..." above :)22:05
mordredjesusaurus: this failure confuses me: http://logs.openstack.org/01/113401/1/check/gate-config-puppet-apply-precise/bfa0abb/console.html22:06
mordrednibalizer: ^^22:06
clarkbdansmith: vmware minesweeper is to blame22:06
mordredlifeless: did you see https://review.openstack.org/#/c/113401/22:06
dansmithclarkb: ah, did it just report on 300 things that triggered stale rechecks?22:06
mordredlifeless: ah, you did22:06
jesusaurusmordred: huh, im not sure what's timing out there22:07
mordredjesusaurus: maybe just bad node that ran REALLY slow?22:07
jesusaurusalso, wow, those logs are UGLY22:07
clarkbdansmith: minesweeper just rechecked a million things22:07
mordredthey make me want to die22:07
*** amotoki has quit IRC22:08
mordreddansmith: why did it do that?22:08
dansmithmordred: why are you asking me?22:08
clarkbdansmith: it is tempting to disable their account22:08
clarkbbut we should probably find them and ask why22:08
*** scottda__ has joined #openstack-infra22:08
mordreddansmith: I meant to say...22:08
mordredclarkb: why did it do that?22:08
anteayadansmith isn't vmware22:08
* mordred knows22:08
dansmithhehe, okay :)22:08
anteayaarosen and salv-orlando are22:08
dansmithI was about to be really insulted :P22:08
anteayaso this is scottda__22:08
jogocan we turn off stale recehcks on 3rd party CI system votes22:08
ianwclarkb: http://graphite.openstack.org/render/?target=stats.nodepool.launch.devstack-f20.*.*&rawData=True <- i'm assuming this is likely due to a bug?22:08
anteayahe is at the cinder sprint with me22:08
openstackgerritlifeless proposed a change to openstack-dev/pbr: Teach pbr about post versioned dev versions.  https://review.openstack.org/8044922:08
anteayaand also works on hp public cloud22:08
openstackgerritlifeless proposed a change to openstack-dev/pbr: Raise an error if preversion versions are too low  https://review.openstack.org/10826922:08
openstackgerritlifeless proposed a change to openstack-dev/pbr: Look for and process sem-ver pseudo headers in git  https://review.openstack.org/10827022:08
openstackgerritlifeless proposed a change to openstack-dev/pbr: Introduce a SemanticVersion object  https://review.openstack.org/9410822:09
mordredscottda__: hey there22:09
scottda__I can be a proxy for pinging the HP Public cloud .NOC room...22:09
salv-orlandojogo, dansmith: I was told the stale recheck issue on vmw minesweeper was sorted?22:09
nibalizermordred: o.O22:09
anteayaclarkb mordred what is useful for scottda__ to pass along to help about hp public cloud?22:09
nibalizerrastu/win 1622:09
lifelessmordred: 94108 doesn't need the haveged fix - it needs your and dhellmann's +2's.22:09
scottda__or someone with a proper hipchat account can also do this...22:09
dansmithsalv-orlando: apparently not22:09
lifelessmordred: the one *above* it needs haveged22:09
salv-orlandodansmith: still the same patch?22:09
scottda__but they are very interested if large numbers of NOva instances are stuck in delete.22:09
* anteaya has forgotten her hipchat password :(22:09
dansmithsalv-orlando: no, every patch (talk to clarkb, I don't know anything)22:09
jogosalv-orlando: why is is minesweeper calling its own recheck22:10
jogothat is silly22:10
salv-orlandojogo: I am afraid I do not understand what do you mean by “its own”…22:10
clarkbscottda__: yesterday we were leaking floating ips. still trying to figure out if that is our side of hpcloud side22:10
dansmithwe're almost at 500 and there are 100 things in the backlog22:10
jogosalv-orlando: see https://review.openstack.org/#/c/65452/22:11
dansmithI'm thinking disabling the account is a good plan right now :)22:11
jogodansmith: ++22:11
clarkbsalv-orlando: see https://review.openstack.org/#/c/65474/22:11
clarkbsalv-orlando: it just did that for hundreds of changes22:11
jogothis is funny22:11
jogoand bad22:11
dansmithjogo: funny in a "can't look away" sort of way22:12
scottda__anteaya: https://www.hipchat.com/sign_in has a forgot password feature22:12
anteayascottda__: thanks22:12
jogodansmith: yup, like a crash22:12
anteayaclarkb mordred anything specific for scottda__ ?22:12
jogowhere no one gets hurt22:12
clarkbanteaya: yesterday and day before we were leaking floating ips22:12
clarkbanteaya: thats all i have right now22:12
* jogo decides to not review any more patches for the day22:13
jogoas it will take hours for this to flush out22:13
anteayascottda__: is the fact we were leaking floating ips enough to go on?22:13
dansmithat least hours, methinks22:13
clarkbanteaya: scottda__ looking at the list of nodes to see if we have a lot in delete states22:13
anteayaclarkb: thanks22:13
*** cdub has quit IRC22:13
*** rcarrillocruz has joined #openstack-infra22:14
clarkbanteaya: scottda__ nothing is more than ~15 minutes22:14
scottda__I just asked the NOC if they are seeing issues.22:14
clarkber nothing more than 15 minutes old in a delete state22:14
salv-orlandoclarkb: just like you I’m playing on several tables…  is vmware recheck triggering a jenkins build upstream?22:14
clarkbsalv-orlando: yes22:14
salv-orlandocrap. I’ve spawn a recheck now on the jenkins for neutron patches.22:14
clarkbsalv-orlando: are you all using zuul? if yes you can recheck en mass directly via the zuul client22:15
clarkbsalv-orlando: which is proably a much better way to recheck 500 changes22:15
dansmithbacklog is dropping,22:15
dansmithso I think the storm is ending22:15
russellbsalv-orlando: see the check queue on http://status.openstack.org/zuul/22:15
*** rcarrill` has quit IRC22:16
russellband the zuul job queue graph at the bottom22:16
russellbthat's what it just caused :)22:16
clarkbscottda__: anteaya: there is also a specific upstream bug that we ran into, let me find the bugs22:16
anteayaclarkb: thanks22:17
clarkbscottda__: anteaya https://bugs.launchpad.net/python-neutronclient/+bug/1297309 and https://bugs.launchpad.net/neutron/+bug/1211915 I don't remember which one was hpcloud and which was rax22:17
uvirtbotLaunchpad bug 1297309 in python-neutronclient "Unauthorized: Unknown auth strategy" [Medium,Fix committed]22:17
clarkbbut we hit them one each in each cloud22:17
*** emagana has quit IRC22:17
anteayaclarkb: was that yesterday?22:18
salv-orlandoclarkb, russellb: ok I’ve disabled the job for the neutron part. The owner of the jenkins instances for the nova part have been warned as well and should disable recheck soon22:18
clarkbanteaya: no this was a week ago? I sent mail to the list on the 5th so I think it was the 5th22:18
*** emagana has joined #openstack-infra22:18
anteayakk thanks22:18
anteayawhich list?22:18
clarkbdev list22:18
salv-orlandoclarkb: I’m not the only one with control over the CI system, so I need to check what happened to the recheck monitor22:18
clarkbanteaya: Gate bugs in the real world22:18
anteayasalv-orlando: thank you22:18
anteayaclarkb: thanks22:18
*** lemongrabber has joined #openstack-infra22:20
*** changbl has quit IRC22:21
*** emagana has quit IRC22:22
*** armax has joined #openstack-infra22:22
*** dustins has quit IRC22:23
fungicatching up on scrollback, it *might* make sense to not trigger stale rechecks on comments which carry a verify vote22:23
*** pblaho has quit IRC22:23
*** dmsimard is now known as dmsimard_away22:23
clarkbfungi: I think CIs should trigger their side if the intent is 500 change recheck22:23
clarkbfungi: zuul enqueue --change foo --pipeline bar22:23
dansmithclarkb: you think they scripted a recheck-vmware comment on every change to trigger minesweeper to re-run everything?22:24
clarkbdansmith: ye22:24
dansmithclarkb: ah, okay.. also: yikes22:24
dansmiththat's a bad idea for more than just this reason22:24
fungiclarkb: oh, gotcha, yes did they comment on those changes with no verify to trigger their own ci runs? i didn't look at an example22:25
clarkbfungi: yup https://review.openstack.org/#/c/65474/22:26
jeblairhow about we disable minesweeper22:30
jeblairand restart zuul22:30
jeblairfungi, clarkb, mordred: ?22:31
mordredjeblair: ++22:31
*** p5ntangl_ has joined #openstack-infra22:32
*** rhsu has joined #openstack-infra22:32
anteayasalv-orlando: we have to disable you22:32
fungijeblair: yes22:32
anteayasalv-orlando: please don't take it personally22:32
jeblairfungi: i will do it22:32
fungithanks. i'm approaching time for eveningish activities which will take me away from computers a while22:33
*** emagana has joined #openstack-infra22:33
*** dkranz has joined #openstack-infra22:33
*** cody-somerville has joined #openstack-infra22:33
*** emagana has quit IRC22:33
salv-orlandoanteaya: if you are disabling me I’d take it personally ;) but I think you mean minesweeper?22:34
*** emagana has joined #openstack-infra22:34
fungisalv-orlando: yeah, having it leave recheck comments on 300+ changes to trigger itself is sort of impactful22:34
*** weshay has joined #openstack-infra22:34
fungiand noisy22:35
*** weshay is now known as weshay_afk22:35
anteayasalv-orlando: minesweeper yes, not you22:35
*** p5ntangle has quit IRC22:35
salv-orlandofungi: I’d do the same. I have stopped neutron rechecks. I have no credentials for stopping rechecks from the jenkins instance managing the nova vc driver22:36
salv-orlandoI can only alert the relevant people and phone them to their desks22:36
salv-orlandomore than that I cant I’m afraid22:36
*** jaypipes has quit IRC22:36
fungisalv-orlando: your efforts are appreciated!22:36
jeblairAttributeError: 'NoneType' object has no attribute 'db'22:37
jeblairnodepool is throwing errors after a restart ^22:37
fungiugh, did it lose the sql socket object?22:37
salv-orlandoclarkb fungi anteaya: can you defer disablement of minesweeper by 10 minutes only? I’m talking with the rest of people managing CI. we should disable recheck soon.22:39
fungii don't see any recent changes to the repo which should have caused that as a regression22:39
*** AlexF has quit IRC22:40
jeblairssh review gerrit set-account --inactive 900822:40
jeblairi did that22:40
jeblairit should be reversible with '--active'22:40
*** david-ly_ is now known as david-lyle22:41
fungiyep, used --inactive on another account over the weekend22:41
fungiseems to work great22:41
*** shashankhegde has quit IRC22:42
clarkbjeblair: do we need to debug nodepool?22:42
fungiit's thinking self.config is none...22:42
salv-orlandofungi: I think there is no way to go back active?22:42
fungisalv-orlando: there is, it's a simple api call from our (admin) end22:43
jeblairclarkb, fungi: yeah, i'm not seeing the problem yet22:43
salv-orlandofungi: ha! I actually meant there is no way I can convince you I’ve fixed the issue or at least I’ve disabled the recheck altogether?22:43
jeblairit's supposed to set up the db before the managers, which it has logged that it has done22:44
*** prad has quit IRC22:44
fungisalv-orlando: probably after current latest fire is extinguished22:44
jeblairjust did a thread dump22:45
salv-orlandofungi: cool we’ll talk about this tomorrow them22:45
jeblairit may be stuck waiting to connect to geard22:45
jeblairfor zuul22:45
jeblairyes it seems much happier now22:46
jeblairso apparently the problem is that it does not completely start if it can not connect to gearman22:46
clarkbjeblair: fun, but it recovered once gearman showed up?22:46
fungiwithout needing any additional help?22:47
jeblairfungi: correct22:47
*** arosen has joined #openstack-infra22:47
fungican't complain then22:47
jeblairbtw, the nodepool restart was to facilitate a mass-delete of all workers22:47
fungiright, i figured as much22:48
fungisalv-orlando: is everything nailed down on the minesweeper end? issues identified?22:48
rhsuhi guys, salv-orlando just pinged me about the recheck chaos22:48
fungirhsu: was that an accident, or...?22:48
salv-orlandofungi, rhsu is my counterpart for the nova vc driver22:48
rhsufungi: we had our build queue cleared and we did the rechecks to get all the reviews back onto the queue22:49
anteayarhsu: ah maybe don't do that in future?22:49
rhsufungi: we've done it twice before, but for some reason this time all the CIs got triggered somehow22:49
anteayarhsu: did you manually trigger the rechecks?22:50
anteayathis was a conscious behaviour?22:50
fungirhsu: right. so we really don't want system accounts leaving what basically amount to "noise" comments on reviews. a lot of reviewers subscribe to e-mail from project change comments, so that was effectively a 300+ e-mail blast out to each and every one of them22:50
rhsuanteaya: we had a script go through and post rechecks on the reviews we had lost22:50
anteayarhsu: how did you lose those reviews?22:51
jeblairrhsu: if you are using zuul, there is a command to enqueue changes in to zuul without comments22:51
jeblairrhsu: and you should know that "recheck" followed by anything will now trigger openstack's ci system22:51
morganfainbergbefore i recheck anything is zuul stable/loading in checks?22:51
jeblairmorganfainberg: yep, gtg.22:51
* morganfainberg noticed you guys were working on it and doesn't want to add fuel to a fire22:51
morganfainbergjeblair, cool22:51
fungias to why it also caused us to rerun checks on all of them, that's because we've loosened our recheck comment syntax22:52
rhsuanteaya: we have an issue with our jenkins acting up..22:52
*** shashankhegde has joined #openstack-infra22:52
anteayarhsu: care to elaborate?22:52
*** gokrokve_ has joined #openstack-infra22:52
rhsujeblair: we're not using zuul yet, but good to know22:52
rhsuanteaya: our jenkins is used for many other projects, not just the CI. we have some internal plugins that messed with the build queue. that resulted in us having to clear it.22:53
*** emagana has quit IRC22:53
fungialso worth noting, since our zuul deleted the jenkins user vote on those changes (which were then cancelled in the zuul restart) i think they're not not going to get refreshed on stale recheck comments or change approvals since they have no existing jenkins user vote22:54
anteayarhsu: is it a separate jenkins from the jenkins running the neutron tests?22:54
anteayaor one jenkins for both nova and neutron plus other things?22:54
anteayafungi: /me nods22:55
rhsuanteaya: yes, a different team manages the neutron one22:55
fungiso they'll probably all have to get rechecked manually at some point22:55
*** gokrokve has quit IRC22:56
anteayarhsu: would it make sense to have one jenkins for both nova and neutron tests and a separate jenkins for internal testing?22:56
*** julim has joined #openstack-infra22:56
anteayarhsu: then when you use zuul you probably will have a happier time22:56
clarkbmordred: fungi: jeblair: arosen and I seem to have a good handle on why we can leak floating ips22:57
mordredclarkb: ooh! excellent22:57
anteayaclarkb: yay, what channel?22:57
mordredarosen: thanks!22:57
clarkbanteaya: -neutron22:57
* anteaya switches22:57
*** bdpayne has quit IRC22:57
clarkbbasically our reliance on https://git.openstack.org/cgit/openstack-infra/nodepool/tree/nodepool/provider_manager.py#n504 to get the ips is killing us22:57
clarkbnovaclient delete ip does a disassocaite then a delete ip. This means the disassociate can succeed but the delete fail. We then short circuit and pop out of that code22:58
clarkblater we try again to delete the node, no ips are listed because we disassociated, and we delete the node (note we never deleted the ip)22:58
*** bdpayne has joined #openstack-infra22:59
*** shashankhegde has quit IRC22:59
fungithat would do it23:00
rhsuanteaya: yes, we could do that but it will take time (the CIs were started/maintained by different teams so all the scripts are different)23:00
jeblairclarkb: can we delete without disassociating?23:00
*** julim has quit IRC23:00
clarkbjeblair: yes but only with neutron client23:00
clarkbso we may have to add a new task manager for neutron client and use it instead23:00
fungiotherwise we end up needing to track the ips separately to retry deletion after disassociation23:01
*** bdpayne_ has joined #openstack-infra23:01
*** datsun180b has quit IRC23:03
anteayarhsu: understood23:03
anteayarhsu: also you are welcome to join the third party meeting anytime: https://wiki.openstack.org/wiki/Meetings/ThirdParty23:04
*** bdpayne has quit IRC23:04
anteayarhsu: I think if you work towards steering in that direction you will be eventually happier with the results than if you keep the structure you have23:04
anteayaespecially for being able to use zuul23:04
anteayaI'm off to dinner and dancing23:05
salv-orlandoanteaya: enjoy both things23:05
*** eharney has quit IRC23:06
*** gargola has quit IRC23:06
*** ZZelle has quit IRC23:08
rhsuanteaya: thanks for your comments, we are definitely heading in that direction23:09
rhsuanteaya: see you in the 3rd party meeting23:09
*** scottda__ has quit IRC23:11
*** emagana has joined #openstack-infra23:15
fungiclarkb: i think we booted precise nodes as trusty in b123:15
clarkbhuh? thats cool23:16
clarkbfungi: this related to postgres?23:16
fungior rather built today's devstack-trusty image in b1 using a precise base image23:16
clarkboh was this jesusaurus change? did it chagne a base image it shouldn't have?23:16
* fungi checks the config23:16
*** homeless has quit IRC23:17
clarkbyup thats it23:17
clarkbfungi: I can push fix or you can then the other can approve it23:17
fungijesusaurus: you want to find/submit the fix? then clarkb and i can both review/approve it ;)23:18
fungiahh, yep, thar she blows23:19
jesusaurusyep, just found it23:19
jesusaurusi'll push up a quick patch23:19
fungijust what and where i expected to see it23:19
*** emagana has quit IRC23:20
fungijesusaurus: closes-bug: #135599823:20
nibalizercommit bomb incoming23:21
nibalizereveryone take cover23:21
openstackgerritK Jonathan Harker proposed a change to openstack-infra/config: Accidentally turned a trusty image into precise  https://review.openstack.org/11367823:21
openstackgerritK Jonathan Harker proposed a change to openstack-infra/config: Accidentally turned a trusty image into precise  https://review.openstack.org/11367823:21
clarkbjesusaurus: can you double check that was the only one?23:21
clarkbfungi: also weird that it would crop up like this on precise but not trusty23:22
*** sdake_ has joined #openstack-infra23:22
clarkbfungi: might be a config difference for postgres I guess23:22
fungiclarkb: perhaps23:22
jogoclarkb: random question - what type of testing process do we have around moving to a newer distro?23:22
fungijogo: mostly manual, and some hopeful prayer23:22
jogoclarkb: this is related to the libvirt version_cap flag.23:22
clarkbjogo: we add image to nodepool, then can manually test on those images or do experimental with them23:22
clarkbjogo: the libvirt verson_cap thing has nothing to do with newer distro testing aiui23:23
jesusaurusclarkb: it appears to be the only s/14.04/12.04/23:23
jogoclarkb: well it *may*. "Yes, that's the idea. Rather than having to scramble when the new23:23
clarkbjesusaurus: great23:23
jogodevstack-gate image shows up, we'd be able to work on any issues in the23:23
jogocontext of a patch series to bump the version_cap."23:23
clarkbjogo: I don't understand that sentence without more context23:23
fungijesusaurus: i double-checked it too, and lgtm23:23
jogoclarkb: as in the version_cap thing decouples moving to a newer version of libvirt and enabling new libvirt dependant features23:24
clarkbjogo: I still don't understand :/23:24
clarkbjogo: the libvirt cap change is self testing in the gate23:24
jogoclarkb: so for the record I am trying to understand myself23:24
fungiclarkb: nova has libvirt-version-dependent features apparently so testing with a newer version of libvirt means we suddenly start exercising new code paths23:25
jogoclarkb: so we hit the live snapshot issue when we upgraded to the new libvirt.23:25
*** dangers is now known as dangers_away23:25
clarkboh that23:25
jogofungi: correct, and the version_cap flag is an attempt at controlling when those new code paths are excercised independant of the libvirt version underneath23:25
fungiclarkb: the idea (from what i gathered in the ml thread) would be to put turning that on in the hands of the nova devs23:26
jogoclarkb fungi: so it makes a single transition into a two phase23:26
clarkbjogo: right the version cap would make it self testing23:26
mordredyah. install the new version of libvirt23:26
mordredthen propose a patch exercising new code paths23:26
jogoclarkb fungi: right, so doesn't this go against our model for all other dependencies23:26
mordredexcept without profit23:26
mordrednope. this is actually exactly how _ALL_ of them should work23:27
clarkbjogo: sort of23:27
clarkbjogo: but we acn pin our python reqs23:27
jogoI thought we have exclicitly left upper bounds open to force us to detect and fix issues ASAP23:27
clarkbjogo: we cannot pin libvirt easily23:27
clarkbjogo: we do, but we also have the option of pinning23:27
mordredjogo: to force us to detect and fix issues that peopel would see23:27
clarkbwithout version capping we have no way of pinnin23:27
mordredif you have the dependency version unbound23:27
mordredbut you have in-code feature flags23:27
mordredthen a user upgrading their version of a library will not affect the features in their system23:28
fungii guess part of the struggle here, which is alluded to in the ml thread, is that users who run nova with a newer libvirt end up using features we're not yet testing23:28
mordredit's a much more scalable thing23:28
*** emagana has joined #openstack-infra23:28
mordredfungi: only if they raise the version_cap flag though, right?23:28
fungimordred: i believe so, yes. warranty voided and all that ;)23:28
jogomordred: the version_cap is back in hypothetica lland23:28
dansmiththe other option, of course, is for us to not add code to nova that we know we can't test :)23:29
mordredoh. this isn't a realy thing?23:29
jogomordred: it was23:29
jogofor a week or two23:29
mordredjogo: gotcha23:29
fungidansmith: novel concept!23:29
mordreddansmith: except that this problem is a real problem in the world23:29
mordreddansmith: not just our gate23:29
mordredpeople need to do rolling updates of things23:29
dansmithmordred: because we put code into our tree that we don't test23:29
*** dims_ has quit IRC23:30
fungimordred: right, this is that nova landed features supporting newer libvirt functionality but which we had no platform on which to validate23:30
jogoso I see this version_cap trying to address two different yet related issues: 1) stop us from upgrading to a new version of libvirt and everything breaks in gate23:30
*** dims_ has joined #openstack-infra23:30
jogo2) reduce risk for folks doing upgrades in production by making the libvirt upgrade a  two phase23:30
*** dimtruck is now known as zz_dimtruck23:30
mordredright. I'm on board with waiting until we can test a new feature before landing it23:30
dansmithjogo: both of which go away if we test things we land :)23:31
jogodansmith: yup23:31
mordredbut I think we need jogo's (1) case sorted for us to be able to do that23:31
mordred1 does not go away23:31
mordredwe have to solve 1 to be able to land the changes to test this23:31
dansmithwhy not?23:31
jogodansmith: and TBH the libvirt break the gate for two days isn't bad23:31
mordredbecause howelse do we test an upgrade to libvirt?23:31
jogomordred: we have a good system developed to detect and triage gate bugs23:31
dansmithmordred: I think you're confusing two things23:31
fungimordred: well, the specific case of gate breakage because we're testing new features of nova we didn't test before does go away, i think was the point23:31
mordredbut libvirt _itself_ could behave differently23:32
dansmithmordred: one is that there are bugs in a new libvirt, and one is that there is a new libvirt that suddenly enables code in nova we've not been exercising23:32
jogomordred: for gate upgrades, things besides libvirt cause us huge pains23:32
dansmithmordred: yeah, but just capping some of our code on libvirt versions doesn't really help that23:32
clarkbdansmith: mordred is right23:32
mordreddansmith: yes. the second part is poopy and I agree with your solution to that23:32
clarkbdansmith: it does, we can update libvirt, then push a change to increase the cap23:32
clarkbdansmith: that change becomes self gating with the newer libvirt23:32
mordredbut the sane path to getting a new libvirt available so that we can test thigns that use its new features23:32
clarkbbut without the cap you immediately jump into using new libvirt23:32
jogoclarkb: self gating isn't enough IMHO23:32
jogoif new failure is 1%23:33
clarkbjogo: I agree23:33
dansmithclarkb: I don't get what you're saying23:33
jogoclarkb: and this is a libvirt specific issue23:33
clarkbjogo: but nothing but running the tests 100000 times is goingto catch that23:33
clarkbjogo: so self gating is just another check23:33
*** reed has joined #openstack-infra23:33
clarkbit won't catch everything23:33
fungiclarkb: i believe with the cap in nova (which switches features on/off) you're still running new libvirt at that point, just not new nova features dependent on it23:33
jogowhen really libvirt isn't the only thing we hit on upgrades23:33
clarkbfungi: correct23:33
clarkbfungi: which is what we need to fix the chicken and egg23:33
fungiclarkb: so "things broke because new libvirt has bugs" is still an issue there23:34
dansmithfungi: right23:34
clarkbfungi: yes to a degree, but not the nova broke because untested code paths23:34
clarkbwhich is the actual problem we have run into23:34
jogoclarkb: so I am bringing this up here because I don't think the version_cap really helps with gaet issues in a meaningful way23:34
clarkbjogo: it does for the reason above23:34
dansmithclarkb: the chicken and egg is solved by always testing on a libvirt that supports all the features we have in tree, no?23:34
*** dims_ has quit IRC23:34
openstackgerritA change was merged to openstack-infra/config: Accidentally turned a trusty image into precise  https://review.openstack.org/11367823:34
clarkbdansmith: yes that solves it too but I don't ever expect that to happen23:34
fungiclarkb: yes. i was trying to separate the two types of breakage there, to narrow what we're talking about ;)23:34
mordreddansmith: we need a sane path to propose an update to the version of libvirt we test with23:35
clarkbdansmith: we can say noav should do that but it won't happen23:35
dansmithclarkb: we need to stop landing code we don't test23:35
mordreddansmith: yes23:35
mordredbut that's only half the picture23:35
jogomordred: /libvirt/any dependency/23:35
mordredwe have to enable testing new code23:35
mordredjogo: yes23:35
mordredjogo: but libvirt is special right now23:35
jogowhich is why i don't like a libvirt specific solution23:35
mordredlibvirt is a snowflake23:35
dansmithmordred: I'm confused about the other half I guess23:35
jogomordred: not really, sqlalchemy is too then23:35
clarkbjogo: sqlalchemy isn't23:36
mordredjogo: sqlalchemy has pip pins23:36
clarkbjogo: sqlalchemy is already self gating23:36
mordredlibvirt does not yet for us23:36
jogomordred: oh when did that happen?23:36
mordredin _THIS_CASE_23:36
mordredlibvirt is different from every other python library we use23:36
mordredonce we get to modern libvirt, this will change23:36
mordredbut we have to get there23:36
clarkbjogo: its always been the case23:36
jogoclarkb: oh never mind23:36
clarkbjogo: since 0.7 broke us23:36
clarkbmordred: well we are on trusty libvirt now23:37
dansmithclarkb: I like that we've codified a grudge :)23:37
mordredclarkb: are we installing that from pip?23:37
clarkbmordred: ya23:37
clarkbmordred: but not the C lib23:37
clarkbmordred: just the python lib23:37
clarkbmordred: in theory libvirt-python could bundle teh C lib too23:37
clarkbmordred: but it doesn't23:37
mordredplease no23:37
mordredplease please now23:37
mordredpython is terrible at compiling c23:38
clarkbmordred: I agree, just pointing out the real issue is the C side23:38
clarkbmordred: because that is what enables the features that aren't being tested23:38
clarkbmordred: but yes we made nova stop using site packages23:38
mordredok. so we're back to the thing we've been saying23:38
clarkbmordred: its glorious23:38
mordredwhich is that we need a path to deal with getting to new versions of things23:39
mordredI thought it was more specific because of libvirt, I'm sorry23:39
mordredit's related to ALL c-based libraries23:39
clarkbmordred: it is actually23:39
*** emagana has quit IRC23:39
mordredlibxml could release a new version23:39
mordredhow do we deal with that?23:39
clarkbwell except that the other C based libs haven't been a problem23:39
mordredyes. this is true :)23:39
clarkblibvirt is a perpetual problem23:39
*** emagana has joined #openstack-infra23:39
jogoso to wrap around, yes we need a path t odeal with etting new versions of things. do you think the version_cap for libvirt makes sense with that in mind?23:39
clarkblxml is not23:39
mordredjogo: I think to clarkb's point, because libvirt has show itself to be a problem child, we may have to treat it as such23:40
clarkbmordred: I think that if we want to solve this "correctly" we use gentoo23:40
dansmithI wonder if we're talking about different things,23:40
jogoassuming that nova won't land any new features that don't have at least some testing23:40
mordreddansmith: I'm certain that we are23:40
dansmithbecause the version cap jogo and I are talking about is purely to allow us to land things we're not testing23:40
jogodansmith: well there was some other pretext there but  yeah23:40
clarkbdansmith: there are two things here. 1) test nova features that are nwe and require new libvirt 2) test that new libvirt works with existing nova features23:41
clarkbdansmith: you are correct that your proposal fixes 123:41
clarkbdansmith: if we can get buy in I think your propsal is the way to solve 123:41
dansmithclarkb: what is "my proposal" ?23:41
jogoclarkb: if this flag doesn't exist then 2 isn't a possible code path23:41
clarkbdansmith: no untested features23:41
dansmithclarkb: okay, then yes23:41
*** rfolco has quit IRC23:41
mordredbut that leads us to 2 ...23:41
dansmithclarkb: the version cap doesn't address #2 tho right?23:41
clarkbdansmith: not completely at least not for the gate23:42
clarkbdansmith: but it does help fix it for our "customers"23:42
mordredbecause I dont think waiting for trusty+next-lts is the way to get us moving with more modern libvirt on a regular basis23:42
*** emagana has quit IRC23:42
dansmithclarkb: I don't think it does23:42
clarkbdansmith: it is a way of saying "look we ran libvirt version X.Y 5.6millions times the last 6 months and it is good"23:42
clarkbdansmith: we test libvirt and qemu more than anyone else I bet23:42
jogoclarkb: so I am confused about 2)23:43
*** emagana has joined #openstack-infra23:43
mordredso I wat to know a) how we as infra get newer libvirt onto systems so that nova devs who want to land new features that depend on newer libvirt have a way to test those23:43
dansmithclarkb: right, so what does disabling some piece of code that worked in one version of libvirt when we're running another one?23:43
dansmithbecause things that used to work can break in the future,23:43
*** wenlock has quit IRC23:43
mordredb) how we do that without screwing nova devs23:43
clarkbdansmith: in our experience most versions of libvirt just don't work23:43
clarkbdansmith: regardless of nova feature23:43
dansmithmordred: requiring testing is not screwing nova devs in any way, IMHO :)23:43
clarkbdansmith: like flat out broken23:43
jogotesting that new libvirt works with old  features. the case we are looking at here is if libvirt > X do a, else do b23:43
dansmithclarkb: right, so I don't see how disabling random pieces of nova based on libvirt version helps with that23:44
clarkbdansmith: it doesn't but nova saying this is the version I work on does23:44
dansmithclarkb: you mean just like a ReleaseNotes thing?23:44
jogodansmith: 'break the future save the present'23:44
mordreddansmith: I mean that how can infra add new libvirt to test machines without screwing nova devs23:44
mordreddansmith: :)23:44
dansmithjogo: so we have to go back to 1955 to stop biff?!23:44
clarkbdansmith: that is one way of doing it23:44
dansmithmordred: okay :)23:44
clarkbmordred: it can't23:45
clarkbmordred: I mean we went to trusty and that screwed people23:45
clarkbmordred: and that was the easy case23:45
dansmithclarkb: because such a communication would be helpful indeed, but I don't think random knobs to disable parts of nova helps in that area at all :)23:45
mordredclarkb: right. I'd love to keep treating that  as a problem to try to solve23:45
jogodansmith: ++ I think we should document which libvirt we test in release notes23:45
mordredclarkb: I don't have a solution right now, but that doesn't mean there is no solution23:45
jogobut not a random knob23:45
*** sdake_ has quit IRC23:45
dansmithmordred: I'd like to be testing a newer libvirt than is coming in the next LTS before the LTS is released23:45
dansmithmordred: that doesn't mean the LTS will work of course,23:46
clarkbmordred: sure, I think it is something we make better but not necessarily "solve"23:46
dansmithmordred: but it helps everyone I think23:46
clarkbmordred: we could write libvirt in python23:46
clarkbmordred: or bundle libvirt in libvirt-python23:46
dansmithjogo: sure that's a very helpful thing to consumers23:46
clarkbdansmith: right so we should use gentoo if that is what we are going to do or arch (and I am not trolling that is how we solve that problem)23:46
clarkbif you want to test a rolling release pipeline then us a rolling release23:47
mordredclarkb: please stop saying bundle libvirt in libvirt-python23:47
clarkbmordred: :P23:47
*** emagana has quit IRC23:47
dansmithclarkb: well, I dunno about that, but if you tell me that's the easiest way, I'm not likely to argue :)23:47
mordredclarkb: I also don't think we need to go to gentoo23:47
mordredor arch23:47
clarkbdansmith: its not the easiest way, but that is what you want to test23:47
mordredI think there are plenty of ways to skin this cat that do not involve arch or gentoo23:47
clarkbdansmith: you are stating you want to get ahead of the slower release based distros and check things in a pipelien as they arrive23:47
dansmithmordred: basically "oh god, hell no" ? :P23:48
clarkbmordred: there are, but they all involve essentially running a rolling release distro23:48
jogoso just to clarify before I say the wrong thing on the ML about this: libvirt version_cap doesn't really solve a lot. it may help a little23:48
mordredclarkb: I disagree23:48
clarkbmordred: then how do you test newer libvirt?23:48
dansmithjogo: I totally disagree that it helps this problem *at*all*23:48
mordredclarkb: I think that we can take important elements into our own hands without having to jump 100% to a rolling release distro23:48
mordredthere are like, maybe 5 pieces of the puzzle tht we need more explicit control over23:48
jogodansmith: I am with you, but I want to make sure I get infras thoughts23:49
dansmithmordred: I agree, I think it's not that hard23:49
clarkbmordred: libvirt has a lot of deps23:49
clarkbmordred: for the way we use it at least23:49
clarkbthose dpes have dpes23:49
clarkband you start getting in kernel land23:49
mordredclarkb: they have a very fixed amount of deps23:49
openstackgerritDavid Pursehouse proposed a change to openstack-infra/jenkins-job-builder: SCM: rename `self` arguments to `parser`  https://review.openstack.org/10694223:49
fungifwiw russellb already has some patches up for a start at sucking newer libvirt upstream packages into the distros on which we test (though starting with fedora first i think, as an experiment)23:49
dansmithfungi: clarkb: yes, but that will break at some point as upstream libvirt pulls a new dep that the base distro doesn't  have23:50
dansmithI know it will happen,23:50
dansmithbecause the dep tree is very deep23:50
nibalizerhrm i having an issue with storyborad23:50
clarkbmordred: my concern isn't the small point reelase. my concern is I need to have a new kernel and new systemd and new iptables and new openvswitch23:50
clarkbmordred: and at that point I am not a rolling release distro23:50
nibalizerremote: (W) 32af657: no files changed, was rebased23:50
nibalizerremote: (W) e85550e: no files changed, was rebased23:50
nibalizerremote: (W) No changes between prior commit c6b132f and new commit de8daa23:50
nibalizerwhat does that even mean23:50
fungidansmith: yes, probably. they have to target specific releases and add further packages which we can pin, and there's a risk that if we do that for too many unrelated subsystems we want to test against they will at some point conflict23:51
mordredclarkb: no, you're a rolling release overlay distro23:51
nibalizeroh are we having the systemd argument in here23:51
*** flwang_ has joined #openstack-infra23:51
clarkbnibalizer: no23:51
* nibalizer gets soapbox23:51
fungis/which we can pin/which we can't pin/23:51
mordredclarkb: I don't need to rolling release the entire thing23:51
dansmithfungi: however, if we're pulling from upstream releases, I think that maybe we can pin the libvirt version when we hit that problem, and work it out before we unpin, just like we do for python packages23:51
clarkbmordred: but you are doing the hard bits23:51
mordredclarkb: I need to rolling release the parts that are important to openstack23:51
clarkbmordred: those same bits are the ones that take up all the time23:52
openstackgerritA change was merged to openstack-infra/storyboard-webclient: Formalized Resource Criteria API  https://review.openstack.org/11016023:52
nibalizerthey explictly say they wont test or even consider it a bug when you dont lockstep upgrade systemd and the kernel23:52
clarkbmordred: this is why I think using a rolling release distro is fine23:52
nibalizerwhich to me is grump23:52
fungidansmith: yeah, though we'll need better mechanisms for spotting when our image builds cease producing new images23:52
clarkbmordred: and a better approach23:52
clarkbmordred: because someone else can deal with teh kernel and systemd and openvswitch23:52
dansmithfungi: yeah, but it's manageable I think23:52
mordredclarkb: except that the rolling release distros are all managed like horseshit23:52
mordredand they will break CONSTANTLY23:52
clarkbmordred: they really aren't23:52
mordredso we'll STILL do the work23:52
* fungi sees foreshadowing in this comment23:52
clarkbmordred: especailly with us rebuulding them every day23:52
clarkbmordred: if we tried to keep gerrit alive for 2 years on arch that would be creazy23:53
mordredclarkb: remember back when I used to run the drizzle build farm and we did care about things like c compilation across multiple platforms?23:53
*** dims_ has joined #openstack-infra23:53
dansmithclarkb: I think mordred just challenged you to a year of gentoo-on-your-laptop23:53
clarkbbut building a new image for devstack every day on arch is fine23:53
mordredclarkb: I could not keep the gentoo slaves going for more than a week at any point in time23:53
mordredbecause deps would break23:53
*** andreaf has joined #openstack-infra23:53
mordredin very strange ways23:53
clarkbmordred: right they were not single use right?23:53
mordredit didn't matter23:53
clarkbmordred: our single use mechanism is the thing that makes it work23:53
mordredtheir behavior changed23:53
mordredthe libraries23:53
mordredthe header files23:53
mordredBEHAVIOR changed23:54
mordredI'm not saing the machiens couldn't operate23:54
*** otherwiseguy has quit IRC23:54
mordredI'm saying they have no consistent interface23:54
openstackgerritMichael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Updated story list interface.  https://review.openstack.org/11016523:54
*** rfolco has joined #openstack-infra23:54
clarkbmordred: if that is the worry you won't fixthat with your plan23:54
mordredclarkb: no, it's exactly the same amount of trouble on our side23:54
clarkbmordred: grabbing new kernel which needs new gcc to support new systemd to support new libvirt has the same problem23:54
*** bdpayne_ has quit IRC23:54
mordredexcept in my plan23:54
mordredthe ONLY people making cowboy changes to library versions are US23:54
openstackgerritMichael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Updated project list interface.  https://review.openstack.org/11016923:54
mordrednot some random ass hipster doing thigns in gentoo outside of our gate23:55
clarkbmordred: right that is the problem :)23:55
clarkbI have zero interest in doing that because it is a ton of busy work23:55
mordredwe will have to do all of that work ourselves in either scenario23:55
mordredall of it23:55
mordredwe will not be able to count on gentoo for any of it23:55
mordredwe'll have to check every single thing they do23:55
mordredso we might as well cut them out, since they will be providing no value23:55
*** flwang_ has quit IRC23:56
mordredand only pain23:56
dansmithjust to point out,m23:56
dansmithif we use fedora/rawhide as our rolling release,23:56
*** andreaf_ has quit IRC23:56
dansmithif we point out that fedora is broken with a libvirt,23:56
clarkbdansmith: I want to rip out fedora completely rsn23:56
clarkbdansmith: fedora has the gentoo problem23:56
dansmithI think we'll be able to get the attention of the fedora and libvirt owners23:56
clarkbdansmith: and noen of the benefits of a stable release apparently23:56
dansmithwell, I think you'll be able to get the attention of paid folks, is my point :)23:56
*** bdpayne has joined #openstack-infra23:56
*** andreaf has quit IRC23:56
dansmithI agree that fedora is no more stable23:56
mordredI think the problem with both things is that it's churn from outside of ourselves23:57
*** andreaf has joined #openstack-infra23:57
clarkband with centos7 we can test the things that people interested in fedora want to see23:57
jogomaybe we should change nova's default virt driver to xen ;)23:57
mordredjogo: hehe23:57
mordredjogo: ++23:57
jogoI bet rax would love that23:57
clarkbjogo: have you read the nodepool scripts for xen?23:57
clarkbjogo: its even worse than this situation23:58
jogoclarkb: I have not, link?23:58
clarkbjogo: tehre are heredoc'd bash scripts nested recursively a few layers deep that boot things in various places with restarts and all kinds of crazy23:58
openstackgerritMichael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Point search requests to search endpoints  https://review.openstack.org/10253623:58
mordredclarkb: so, I think what I'm getting at is taht I'd like to give _our_ devs the ability to take the reins of their destiny with some of these things23:59
openstackgerritMichael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Sort stories and projects.  https://review.openstack.org/11242623:59
mordredit's not just that I want to go faster or slower - it's that I think that external distro release decisions about which version of libvirt should exist for us give us no value23:59
openstackgerritK Jonathan Harker proposed a change to openstack-infra/config: Move some jenkinsuser bits to openstack_project  https://review.openstack.org/10688423:59
clarkbmordred: I agree23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!