Friday, 2014-04-18

*** nati_ueno has quit IRC00:00
clarkbhttp://cloud-images.ubuntu.com/releases/14.04/release/ is where I got the image btw00:01
*** praneshp_ has joined #openstack-infra00:02
*** praneshp has quit IRC00:04
*** praneshp_ is now known as praneshp00:04
*** thuc has joined #openstack-infra00:05
jogoclarkb: for now I am just wondering what a default devstack dsvm full run looks like00:06
*** thuc has quit IRC00:06
clarkbjogo: oh I think we are a long way away from that00:06
*** thuc has joined #openstack-infra00:07
clarkbfirst the puppet/nodepool stuff must be made to work00:07
clarkbthen we need jobs00:07
clarkbthen we can try that00:07
clarkbjogo: you can always run it manually though00:07
*** thuc_ has joined #openstack-infra00:08
jogoclarkb: true on both counts. but its nice to have all the logs and whatnot in our fancy system.00:09
*** thuc__ has joined #openstack-infra00:09
jogoanyway its getting cold out here time to go inside for a bit00:09
*** sarob has joined #openstack-infra00:10
*** zns has quit IRC00:12
*** thuc has quit IRC00:12
mordredclarkb: might be time to start accelerating our parallel puppet3 master00:12
*** SumitNaiksatam has quit IRC00:12
mordredclarkb: nibalizer was going to work on that some wasn't he?00:12
*** thuc_ has quit IRC00:12
*** ilives has joined #openstack-infra00:13
clarkbmordred: well our slave stuff should be puppet3 clean coutesy of pleia2 and fedora 2000:13
mordredoh right00:13
clarkbmordred: I think its more likely that we need to modify puppet to handle new and fun special caes00:13
mordredso we might not be too bad puppet-wise on trusty00:13
*** thuc__ has quit IRC00:13
mordredwe also probably want to stop adding a ppa in the launch script00:13
*** thuc has joined #openstack-infra00:14
*** sarob has quit IRC00:14
*** ilives has quit IRC00:17
*** ilives has joined #openstack-infra00:17
*** praneshp has quit IRC00:17
*** nati_ueno has joined #openstack-infra00:17
*** melwitt has quit IRC00:18
*** praneshp has joined #openstack-infra00:19
*** thuc has quit IRC00:19
*** reed has quit IRC00:19
*** gokrokve has joined #openstack-infra00:21
*** matsuhashi has joined #openstack-infra00:23
clarkbfyi https://bugs.launchpad.net/python-glanceclient/+bug/130927200:26
uvirtbotLaunchpad bug 1309272 in python-glanceclient "Required options should be listed as required and error when not provided." [Undecided,New]00:26
clarkbonly one more devstack precise image to build with the get pip and vm image caching change00:30
clarkbit is looking good00:30
*** derekh has quit IRC00:31
sdaguesalv-orlando: awesome great00:33
*** bhuvan has quit IRC00:33
nibalizerthere is a puppet 3 master change proposed00:34
clarkbsdague: ok all devstack-precise* images have been rebuilt00:34
nibalizeri think its failing puppet parser test right now00:34
nibalizeri can look at that later on tonight00:34
clarkbsdague: is the pip thing being tracked in e-r? we should see that fall to zero over the next few hours as nodes are cycled out00:34
sdagueclarkb: great00:34
sdagueyes, we've got it in er00:35
sdagueBug 1299176 - Rackspace network periodically blips and won't let us download get-pip from github00:35
mnaserhow many vm's does the testing infra usually use up to?00:35
uvirtbotLaunchpad bug 1299176 in openstack-ci "Rackspace network periodically blips and won't let us download get-pip from github" [High,Fix released] https://launchpad.net/bugs/129917600:35
clarkbmnaser: the graph at http://status.openstack.org/zuul/ shows you. I think we peak at about 1k due to quotas00:35
anteayaclarkb are we up to 1k now?00:36
anteayacool00:36
clarkbmnaser: you can always copy that image url and edit it to see other data00:36
mnaserand 8gb vms each?00:36
clarkbmnaser: most are 8GB, some are 30GB but enforced to 8GB because we only get enough cpu time from that provider on the large 30GB images00:36
clarkbanteaya: no we appear to be at ~400 right now00:37
mnaserso up to 8tb of memory all just testing openstack00:37
mnaserpretty cool00:37
anteayasorry I meant our upper limit00:39
mordrednibalizer: hey - you know things ...00:39
anteayadidn't know we could go as high as 1k00:39
*** atiwari has quit IRC00:39
anteayayay to our providers00:39
mordrednibalizer: how do I set postgresql::globals::version so that it's set when I call postgresql::server00:40
*** markmcclain has quit IRC00:41
*** mrodden has quit IRC00:41
*** dstanek has quit IRC00:43
*** changbl has joined #openstack-infra00:44
clarkbmordred: I think you class { 'postgresql::globals': version => 'someversion', } class {'postgresql::server': require => Class['postgresql::globals'], }00:44
*** dstanek has joined #openstack-infra00:44
clarkbmordred: that should ensure that the globals version value is set before postgresql::server does anything with it00:45
* clarkb &00:45
sdaguehttps://bugs.launchpad.net/openstack-ci/+bug/1290562 - adrian left at comment at the bottom of that that new setuptools fixes it00:46
uvirtbotLaunchpad bug 1290562 in openstack-ci "failing pypy jobs with "option --single-version-externally-managed not recognized"" [High,In progress]00:46
*** SumitNaiksatam has joined #openstack-infra00:48
openstackgerritSean Dague proposed a change to openstack/requirements: require new setuptools  https://review.openstack.org/8847400:51
sdaguemordred: you feel like pushing +A on this - https://review.openstack.org/#/c/88302/ - jeblair and SergeyLukjanov were +2 previously00:52
sdaguewill prevent us from landing a patch in tempest that breaks icehouse00:52
mordrednibalizer, clarkb: looks like very few patches get us to working trusty :)00:52
mordredsdague: looking00:52
openstackgerritK Jonathan Harker proposed a change to openstack-infra/config: Ruby evaluates empty strings as true  https://review.openstack.org/8847500:53
*** gokrokve_ has joined #openstack-infra00:53
nibalizerwooot00:55
*** gokrokve has quit IRC00:56
openstackgerritA change was merged to openstack-infra/config: add stable icehouse jobs for tempest master  https://review.openstack.org/8830200:57
*** dcramer_ has joined #openstack-infra00:57
*** dcramer_ has joined #openstack-infra00:57
*** timrc is now known as timrc-afk00:58
nibalizermordred: re: postgres, do you have an example where the code is?00:59
*** marcoemorais has quit IRC01:00
sdaguemordred: thanks01:01
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Changes to get nodepool working with trusty  https://review.openstack.org/8847701:01
mordrednibalizer: ^^ look at that - what I did, I think, is ugly01:01
mordredbut seems to physically work01:01
adam_ghow often do puppet changes get applied to the slave images ?01:02
nibalizerclarkb steered you right according to this https://github.com/puppetlabs/puppetlabs-postgresql#class-postgresqlglobals01:02
nibalizeri've never worked with this pattern before01:02
mordredadam_g: when they are created01:04
mordrednibalizer: ossum01:05
nibalizermordred: since this merged, the postgres module should be dealing with quotes correctly now01:05
nibalizerhttps://github.com/puppetlabs/puppetlabs-postgresql/commit/e26036c176870539c26e568c986de6c68be17fec01:05
mordredadam_g: we run puppet once when we are creating the base image from which we make the slave nodes01:05
adam_gmordred, right.. is that a daily/nightly affair?01:05
mordredadam_g: daily-ish01:05
adam_gah, cool01:05
clarkbI just did the ones ironic uses01:06
clarkbyou should be good01:06
clarkbassuming I didnt derp01:07
adam_gclarkb, oh, party01:07
jeblairclarkb: i don't want us to use manual image uploads for anything01:07
clarkbjeblair ok01:07
*** dims has quit IRC01:08
clarkbeven for using current nodepool image building process?01:08
nibalizermordred: also your changes for trusty and my puppet3 branch look similarish01:08
jeblairi think we should get dib working and use that01:08
clarkbk01:08
clarkbjogo: there's your answer01:09
openstackgerritSpencer Krum proposed a change to openstack-infra/config: Add node def for puppet3 master  https://review.openstack.org/8367801:11
jeblairi mean, it shouldn't take long to add dib and glance support01:11
nibalizerjeblair: clarkb mordred ^01:11
nibalizerwill need to rebase against mordreds trusty changes if those land first01:11
nibalizerbut for now it should be green01:11
jeblairclarkb: also, figuring out, eg, how networking works using a bare cloud image in hp and rax is a separate problem01:12
jeblairclarkb: so i think manually uploading an image and understanding that is time well spent01:12
jeblairbut i don't want to incorporate manual image uploads into the current process (as soon as anything about it needs tweaking or is otherwise unusual, we either need to start documenting that process or building our own images, and that would end up being duplicated effort)01:13
*** wchrisj has joined #openstack-infra01:15
clarkbjeblair: ok, so we can test and work stuff out using the image I uploaded but no nodepool or jenkins/zuul testing on that01:15
clarkbseems sane01:15
jeblairclarkb: ++ and that happening in parallel to mordred's dib will get us there faster01:15
jeblairnibalizer: were you going to respond to the ps7 questions?01:16
*** asselin has quit IRC01:16
jeblairclarkb, mordred: i do think we should both try to get gerrit 2.8 on cloud db as well as on trusty01:17
mordredjeblair: ++01:17
*** asselin has joined #openstack-infra01:17
mordredjeblair: I've got 80% patch for nodepool + dib01:17
mordredmaybe I should publish the work in progress...01:17
jeblairmordred: prolly a good idea01:18
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Changes to get nodepool working with trusty  https://review.openstack.org/8847701:18
mordredjeblair, nibalizer: some more fixes ^^01:18
mordredjeblair: so far, running our stuff on a node based off of the bare trusty image has not been to yucky01:18
jeblairmordred: cool01:19
openstackgerritMonty Taylor proposed a change to openstack-infra/nodepool: Build images using diskimage-builder  https://review.openstack.org/4648201:20
mordredjeblair, clarkb ^^01:20
mordredthat does NOT WORK01:20
mordredI got as far as realizing I needed to figure out how to get a glance client01:20
mordredand then I haven't been able to work on it any more01:20
jeblairmordred: do you want to continue working on the dib part and i can start on glance?01:20
*** dims has joined #openstack-infra01:20
mordredjeblair: yah - sure enough01:20
nibalizerjeblair: okay now im getting it01:21
anteayamordred 46482 has a whitespace error01:21
*** wchrisj has quit IRC01:21
anteayathe rest I am trying to follow along on01:21
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Add support for disk-image-builder in nodepool  https://review.openstack.org/8847901:22
mordredanteaya: oh - 46482 is terrible01:22
mordredI mean, it's quite literally half-done01:22
anteayaoh01:22
mordredso is 8847901:22
jeblairmordred: is there more dib work to do?01:22
anteayak01:22
mordredI'm just pushing them up for communication01:22
mordredjeblair: lemme look real quick - I think the dib bits are largely done01:22
jeblair(i intended to volunteer for parallel work, not to try to take it over if you're done with the dib)01:23
mordredyah01:23
mordredyes. step 1 of dib work should be servicable. it's very non-optimal, as it does not contain the desired de-coupling of image and provider01:24
*** nosnos has joined #openstack-infra01:24
mordredbut so we build/upload 4 times - who cares01:24
mordredthe next set of DIB work is the patch to config above01:24
mordredwhich needs to be updated as well01:24
mordredI should maybe make an etherpad...01:24
*** thuc has joined #openstack-infra01:24
openstackgerritSpencer Krum proposed a change to openstack-infra/config: Add node def for puppet3 master  https://review.openstack.org/8367801:25
jeblairmordred: ok; well, i'm not going to work on it right now, so we can regroup later;  happy to pitch in but don't want to step on toes01:25
mordredjeblair: pitching in is totally welcome - even making sure what's there that's mostly completeish other than upload isn't crazy01:26
mordredand I'm about to stop hacking for the night as well :)01:26
*** nati_ueno has quit IRC01:26
*** UtahDave has quit IRC01:27
*** mriedem has joined #openstack-infra01:28
*** thuc has quit IRC01:29
*** dstanek has quit IRC01:34
*** Sukhdev has quit IRC01:34
*** dcramer_ has quit IRC01:34
*** zns has joined #openstack-infra01:34
*** saschpe has quit IRC01:36
*** saschpe has joined #openstack-infra01:38
*** yaguang has joined #openstack-infra01:43
*** dstanek has joined #openstack-infra01:43
mordredjeblair, clarkb: I just had a possibly terrible idea01:44
mordredwhat if run_remote_puppet.sh was actually a Makefile (I know we were carping about this a little bit earlier) -01:45
mordredbut one with a phony target that ran the equiv of the run_ssh function01:45
openstackgerritA change was merged to openstack-infra/elastic-recheck: Add a fingerprint for bug 1309252  https://review.openstack.org/8846201:47
uvirtbotLaunchpad bug 1309252 in tempest "HTTP errors observed in test_load_balancer_basic" [High,Triaged] https://launchpad.net/bugs/130925201:47
mordredsomething like this: http://paste.openstack.org/show/76197/01:49
* mordred was thinking about how to try to generalize the single depenedncy list01:49
*** khyati_ has quit IRC01:49
mordredok. NOW I'm going away01:49
mordredand will eat salmon01:49
anteayaenjoy salmon01:53
greghaynesIts funny, beause there was a puppet slogan for a while about "because ssh and a for loop doesnt cur it"01:56
clarkbprinted on tshirts01:58
* nibalizer has this shirt01:58
greghaynesI think I somehow have that shirt, even01:58
nibalizermordred: i've thought that you do want to have a more general solution to who goes first, but yea dependency resolution is hard01:59
nibalizermakefiles dont seem like a horrible idea01:59
nibalizermaybe fabric, since it is actually for orchestration02:00
nibalizeractually it looks like fabric doesn't do dependency resolution02:02
*** rlandy has quit IRC02:06
*** nati_ueno has joined #openstack-infra02:11
*** aysyd has quit IRC02:13
*** Longgeek has joined #openstack-infra02:14
*** zns has quit IRC02:16
*** rwsu has quit IRC02:22
*** amcrn has quit IRC02:32
*** pcrews has quit IRC02:35
*** nati_ueno has quit IRC02:41
*** nati_ueno has joined #openstack-infra02:43
*** Ryan_Lane has quit IRC02:47
mnasercheck-grenade-dsvm-partial-ncpu hasn't passed for the past ~5 hours02:48
mnaserwhat's usually the thing to do in these cases02:48
*** CaptTofu has quit IRC02:48
*** harlowja is now known as harlowja_away02:49
clarkbmnaser: usually figure out why, then fix it if possible02:49
*** CaptTofu has joined #openstack-infra02:49
clarkbif a fix is not practical we can make the test non voting or stop running it completely02:49
mnaseralrighty, i'll try to dig out the root cause, would like my patch to go in but keeps failing there (but it doesn't touch any part of that)02:49
mnaserhttps://bugs.launchpad.net/nova/+bug/1254890 -- i'll dig in02:50
uvirtbotLaunchpad bug 1254890 in nova ""Timed out waiting for thing ... to become ACTIVE" causes tempest-dsvm-* failures" [High,Confirmed]02:50
*** alexpilotti has quit IRC02:50
mnaserthanks btw, clarkb :)02:52
*** jhesketh has joined #openstack-infra02:52
*** nati_ueno has quit IRC02:53
jheskethAfternoon02:53
*** CaptTofu has quit IRC02:53
mikaljhesketh: heya02:53
clarkbmnaser: oh you know what02:53
*** Longgeek has quit IRC02:54
clarkbso we released icehouse today and started updating the upgrade tests02:54
*** Longgeek has joined #openstack-infra02:54
mnaseraaaah02:54
*** Longgeek has quit IRC02:54
clarkbI bet the partial ncpu isn't doing that right02:54
mnasershould we look into moving it to non voting02:54
*** Longgeek has joined #openstack-infra02:54
clarkbor possibly we have actually broken the ability to upgrade from icehouse to master02:54
clarkbmnaser: ideally no, since this is likely to be a real bug in novas ability to do upgrades in thsi way02:55
mnaserclarkb: let me do a tiny bit of initial digging02:55
mnaserand i'll report back, cause it might be a minor thing.. or not :p02:55
clarkbmnaser: http://logs.openstack.org/07/83207/21/check/check-grenade-dsvm-partial-ncpu/983dbdd/logs/old/screen-n-cpu.txt.gz?level=ERROR maybe?02:56
mnasereek02:57
clarkball of those errors start after the upgrade02:57
clarkbso ya I think this may be a legit fail02:57
clarkbthose milestone branches should be killed >_>02:57
*** coolsvap|afk is now known as coolsvap02:58
*** gokrokve_ has quit IRC02:58
*** gyee has quit IRC02:59
*** gokrokve has joined #openstack-infra03:05
*** matsuhashi has quit IRC03:09
*** matsuhashi has joined #openstack-infra03:09
*** matsuhashi has quit IRC03:09
*** jhesketh has quit IRC03:09
*** gokrokve has quit IRC03:09
mnaserclarkb: it looks like n-cpu is not starting at all post-upgrade03:09
clarkbmnaser: it keeps running03:11
clarkbmnaser: n-cpu is the one service not restarted during that test03:11
mnaserclarkb: i think that might be the issue, because it gets rpc calls that are newer than the one it supports... let me verify03:12
clarkbso you need to look at timestamps in the old/screen-n-cpu.txt file to correlate when that happens03:12
clarkband it looks like upgrade happens around 01:54ish03:12
clarkberrors show up 02:00 ish03:12
mnaseri'm curious if any other service does rpc calls towards n-cpu03:13
clarkbI think cond and sch?03:15
clarkbI am not super familiar with nova's rpc setup though03:15
mnaserclarkb: i think n-sch is making calls to n-cpu which it does not support03:15
mnasermore specifically, n-cpu is getting calls of version 2.37 which it doesn't support03:16
mordredgreghaynes: yah. it would be great if it were better tahn ssh and a for loop :)03:16
clarkbright, but that test exists to specifically make sure it supports it03:16
clarkbmnaser: so I think this is valid, nova messed up somewhere03:16
mordredalso - for the record - they did not have any pacific salmon :(03:16
mordredwe'll be having beef instead03:17
mnaserso basically, you need to restart your n-cpu for upgrades.. or this has to be addressed (latter obviously)03:17
clarkbmordred: now I am going to eat my lox03:17
clarkbmnaser: right, this test exists to make sure that you don't have to restart your ncpu03:17
mnaserhmm03:17
clarkbthe only reason we have this test is ^03:17
clarkbI think icehouse being off on milestone proposed and master being juno with havana being the stable branch allowed something to slip through03:18
clarkbmordred: can we kill milestone proposed03:18
clarkbmordred: seems like it is good for breaking things03:18
*** nosnos has quit IRC03:19
mnaserum03:20
mnaseri think there is an oops here03:20
clarkbyup I agree03:20
*** wenlock has joined #openstack-infra03:20
*** chandan_kumar has joined #openstack-infra03:20
mordredclarkb: I think so - although I think I need to retrigger the merge propose jobs03:20
mnaserclarkb: i think if onud the reason03:20
clarkbI think us only testing havana to master allowed the switch to icehouse to master to fubar03:21
clarkbmordred: no no I mean completely03:21
clarkbmordred: like don't use them ever03:21
clarkbmordred: we should cut and icehouse branch after FF03:21
mordredI'm down with that03:21
mnaserclarkb: https://github.com/openstack/nova/blob/master/nova/compute/rpcapi.py#L24803:21
mordredit's ttx you have to convince03:21
clarkbthen release off of something on that branch 6 weeks later03:21
mordredI've never thought they were a good idea03:21
mnaseri think icehouse should be set and cut off there03:21
mnaserit was never cutoff..03:21
clarkbbut that way we can test this without crazyness03:21
mordredbut - until ttx says otherwise, I will support his special brand of crazy03:22
clarkbmnaser: ugh03:22
openstackgerritA change was merged to openstack-infra/devstack-gate: add run-tests.sh  https://review.openstack.org/8843003:22
clarkbmordred: the problem with the crazy is it breaks CI03:22
mordredyah03:22
clarkbmordred: we stop testing stuff03:22
*** matsuhashi has joined #openstack-infra03:25
*** dstanek has quit IRC03:26
mnaserclarkb: https://github.com/openstack/nova/commit/13cb4e7d3fb7b119f7ee01d8aa3f265d4be2009003:26
mnaseris compute being set to icehouse-compat?03:27
*** matsuhashi has quit IRC03:27
*** matsuhashi has joined #openstack-infra03:28
clarkbI think so03:29
mnaserclarkb: http://logs.openstack.org/24/88324/6/check/check-grenade-dsvm-partial-ncpu/2e18c53/logs/new/screen-n-sch.txt.gz#_2014-04-18_00_44_53_075 -- that should be icehouse-compat .. where can i go to put the fix in?03:29
*** gokrokve has joined #openstack-infra03:29
*** timrc-afk is now known as timrc03:30
clarkbgrenade manages that test03:30
clarkbfix probably belongs there03:30
clarkbbut this is icehouse to juno that is breaking03:30
clarkbnot havana to icehouse iirc03:30
mnaseroh?03:31
mnaserhttps://github.com/openstack-dev/grenade/blob/master/from-havana/upgrade-nova -- shouldn't this be icehouse-compat03:31
*** matsuhashi has quit IRC03:31
*** gokrokve_ has joined #openstack-infra03:31
clarkbno, because from havana is correct03:31
*** matsuhashi has joined #openstack-infra03:31
mnaseroh03:31
mnaserso03:31
clarkbwhat isn't correct is from icehouse03:31
mnasershouldn't there be from-icehouse in there somewehre03:31
clarkbyeah I am guessing that is what is missing03:32
clarkbits kind of a bummer you have to set any config at all03:32
mnaserso i guess grenade needs to be updated03:32
clarkbyup, its possible the changes have been proposed already03:32
clarkbsdague/dtroyer would know03:32
*** gokrokv__ has joined #openstack-infra03:33
*** nati_ueno has joined #openstack-infra03:33
*** matsuhas_ has joined #openstack-infra03:34
*** unicell has joined #openstack-infra03:34
*** gokrokve has quit IRC03:34
mnaseri see.. i guess tomorrow they can address this, until then, no code can be merged03:35
*** dcramer_ has joined #openstack-infra03:35
mnaseror will be able to03:35
*** matsuhashi has quit IRC03:35
*** gokrokve_ has quit IRC03:35
openstackgerritMatt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for grenade icehouse upgrade bug 1309324  https://review.openstack.org/8848703:37
mriedemclarkb: mnaser: ^03:37
uvirtbotLaunchpad bug 1309324 in grenade "check-grenade-dsvm-partial-ncpu* failing since icehouse release due to compat issue" [Undecided,New] https://launchpad.net/bugs/130932403:37
*** lnxnut has quit IRC03:37
*** Longgeek has quit IRC03:37
*** gokrokv__ has quit IRC03:37
*** Longgeek has joined #openstack-infra03:38
*** dcramer_ has quit IRC03:39
*** lnxnut has joined #openstack-infra03:39
*** wchrisj has joined #openstack-infra03:41
*** Longgeek has quit IRC03:41
mriedemdon't see any grenade patches up for that specifically https://review.openstack.org/#/q/status:open+project:openstack-dev/grenade,n,z03:42
mriedemjust one to open stable/icehouse03:42
mnaseryeah.. looks like nova is going to be blocked until that's fixed03:43
clarkbI wonder if we can trick it into doing a havana to master again03:44
clarkbif that works icehouse to master should be possible too03:44
*** nati_ueno has quit IRC03:44
*** masayukig has quit IRC03:45
dtroyerI'm trying to catch up with what is happening here…   what I don't know is what changed since yesterday?03:46
clarkbalso I think part of the issue here is we don't get d-g on the partial n-cpu test03:46
clarkbdtroyer: we started testing icehouse to master03:46
clarkbdtroyer: instead of havana to master03:46
dtroyertoo early for that03:46
*** Longgeek has joined #openstack-infra03:46
clarkbdtroyer: If46d1624e0b35a37d5dfb043edb513fc04b6ea68 is the change03:46
clarkbdtroyer: why is that tool early?03:46
dtroyergrenade has nothing to build an 'old' devstack that is icehouse03:47
dtroyergrenaderc has to have a couple of settings changed and the entire from-icehouse stuff needs to be created03:47
openstackgerritA change was merged to openstack-infra/elastic-recheck: Add query for grenade icehouse upgrade bug 1309324  https://review.openstack.org/8848703:47
uvirtbotLaunchpad bug 1309324 in grenade "check-grenade-dsvm-partial-ncpu* failing since icehouse release due to compat issue" [Undecided,New] https://launchpad.net/bugs/130932403:47
clarkbdtroyer: ok I will propose a revert of If46d1624e0b35a37d5dfb043edb513fc04b6ea6803:48
dtroyerhttps://review.openstack.org/#/c/88421/ is the first step, to nail down the final havana -> icehouse, rhather than master03:48
openstackgerritClark Boylan proposed a change to openstack-infra/devstack-gate: Revert "update grenade branches"  https://review.openstack.org/8848803:49
clarkbthere we go03:49
clarkbmordred: jeblair sdague ^03:49
clarkbSergeyLukjanov: ^03:49
clarkbdtroyer: any idea why only that test fails?03:51
clarkbdtroyer: does grenade have fallback behavior that is reasonable for the other grenade tests?03:51
dtroyerno.  I'm not sure what is different between the tests03:51
clarkbdtroyer: the partial ncpu test does not upgrade n-cpu03:52
clarkbdtroyer: and apparently needs a config option set in nova to make that possible03:53
clarkbthe others do a full upgrade from old to new03:53
mnaseri suspect what's happening is that n-cpu is receiving RPC requests for a version that it doesn't support/understand03:53
clarkbwhich apparently passed03:53
dtroyerah, than that's why…that is what is set in from-havana/upgrade-nova.   we'll need the equivalent for from-icehouse03:53
mnaserso when a new VM gets created, the RPC call sent from n-sch to n-cpu fails because the version is too new03:54
*** matsuhas_ has quit IRC03:54
*** thuc has joined #openstack-infra03:54
dtroyerso with 'upgrade_levels compute icehouse' in nova.conf it shoudl work, right?03:55
mnaserdtroyer: unfortuantely no, i think03:55
mnaserbecause that version level was never defined in icehouse.. which is probably a mistake03:55
clarkbdtroyer: but why do the normal grenade tests work?03:55
clarkbdtroyer: I think my confusion si related to grenade otherwise being happy03:55
*** khyati has joined #openstack-infra03:55
dtroyerwhat I understand ATM is the RPC versions all match03:56
mnaserclarkb: beacuse when you restart everything, everything is running the same rpcapi, so they all understand each other03:56
clarkbmnaser: right, and that is why I think the milestone proposed branches are bad03:56
mnaserdtroyer: this is the issue https://github.com/openstack/nova/blob/master/nova/compute/rpcapi.py#L24803:56
mriedemclarkb: can you put bug 1309324 on that commit message?03:56
mnaserthere was no RPC API version set in stone for icehouse03:56
clarkbmnaser: if we had cut an icehouse branch 6/7 weeks ago this would've been apparent before we released03:56
uvirtbotLaunchpad bug 1309324 in grenade "check-grenade-dsvm-partial-ncpu* failing since icehouse release due to compat issue" [Undecided,New] https://launchpad.net/bugs/130932403:56
clarkbmriedem: sure03:56
mriedemthanks03:56
*** dcramer_ has joined #openstack-infra03:57
openstackgerritClark Boylan proposed a change to openstack-infra/devstack-gate: Revert "update grenade branches"  https://review.openstack.org/8848803:57
*** matsuhashi has joined #openstack-infra03:57
mnaserthe cutoff for icehouse should have happened and a new juno-compat should be been introduced i guess03:57
*** thuc_ has joined #openstack-infra03:58
dtroyerso is it backward?  ie, for the ncpu run the upgrade_levels should not be set (yet)?03:59
dtroyerfor icehouse -> master03:59
clarkbdtroyer: no it should be03:59
clarkbbut it may not fix the issue03:59
clarkbbut it will need to be set either way I think04:00
dtroyerok.   It's late here.  hopefully it'll make sense in the morning.04:00
mnaserrpc versioning isn't fun04:00
clarkbmore importantly I think we should have that conversation with ttx. Milestone proposed shoves things in a corner where it is harder for things to test it04:01
clarkbmust better to be upfront and start the icehouse testing before we haev an icehouse release04:01
dtroyerI do think though that getting the 'from-icehouse' bits into grenade needs to happen first/next04:01
clarkbdtroyer: agreed04:01
*** dcramer_ has quit IRC04:01
clarkbdtroyer: but you shouldn't need much right?04:01
clarkbI mean it works other than this partial ncpu thing04:01
*** thuc has quit IRC04:01
*** Longgeek has quit IRC04:01
mriedemmnaser: i've gotta drop, you're in good hands though :)04:02
dtroyerI'm not familiar offhand with what is in from-havana, but it shouldn't be much.04:02
mriedemthanks for bringing that up!04:02
mnaserheh, take care, mriedem :) no problem04:02
*** mriedem has quit IRC04:03
mnaseri think this should be brought up with russellb and ask him about https://github.com/openstack/nova/commit/13cb4e7d3fb7b119f7ee01d8aa3f265d4be2009004:03
*** zhiyan_ is now known as zhiyan04:04
*** praneshp has quit IRC04:05
*** zns has joined #openstack-infra04:06
openstackgerritClark Boylan proposed a change to openstack-infra/config: Test d-g against partial n-cpu job.  https://review.openstack.org/8849204:06
clarkband ^ should make sure we actually test this before merging04:06
clarkbya we should definitely follow up with nova folks. jogo too04:06
*** zns has quit IRC04:07
*** wchrisj has quit IRC04:08
*** david-lyle has joined #openstack-infra04:08
*** praneshp has joined #openstack-infra04:08
*** zns has joined #openstack-infra04:10
*** nosnos has joined #openstack-infra04:10
*** harlowja_at_home has joined #openstack-infra04:14
*** timrc is now known as timrc-afk04:16
zaroclarkb: why am i not seeing any gearman packets with tcpdump?04:19
clarkbzaro: make sure you are listening to the correct interface04:19
clarkband that your filter is correct04:19
zaromy vm only has one interface, 'eth0'04:20
*** harlowja_at_home has quit IRC04:21
clarkbzaro: it should have an lo as well04:22
zaroclarkb: listening for everyhting on default gearman port 4730 but getting nothing.  command is 'tcpdump -i eth0 port 4730'04:22
clarkband if you are connecting to localhost it should go over the loopback04:22
*** lnxnut has quit IRC04:22
*** harlowja_at_home has joined #openstack-infra04:23
*** zns has quit IRC04:23
*** zns has joined #openstack-infra04:24
zaroclarkb: ok getting something now.  had to set to 'lo' instead of 'eth0'04:25
*** dstanek has joined #openstack-infra04:28
*** sweston has quit IRC04:30
*** sweston has joined #openstack-infra04:30
*** gokrokve has joined #openstack-infra04:32
*** dstanek has quit IRC04:33
*** harlowja_at_home has quit IRC04:35
*** _nadya_ has joined #openstack-infra04:36
*** gokrokve has quit IRC04:37
*** david-lyle has quit IRC04:37
*** coolsvap is now known as coolsvap|afk04:39
zaroclarkb: gear doc says that you need all 3 files, ssl_key, ssl_cert, and ssl_ca.  any pointers on how to generate that ssl_ca file?04:41
*** zhiyan is now known as zhiyan_04:41
*** thuc has joined #openstack-infra04:42
clarkbzaro: yeah use your cert file as the ca cert (this isn't recommended iirc) then you generate a third cert to use as the cert iirc04:42
clarkbI don't know off the top of my head the process for that though04:43
*** blamar has joined #openstack-infra04:43
*** thomasem has quit IRC04:43
*** thuc_ has quit IRC04:45
*** thuc has quit IRC04:46
clarkbzaro: you might be able to use the cert as the ca cert too04:48
mgagnezaro: is it for test purpose?04:49
*** jgrimm has joined #openstack-infra04:50
zaromgagne: yes04:52
*** ildikov has quit IRC04:52
mgagnezaro: would this procedure work? http://gearman.info/gearmand/ssl.html04:52
mgagnezaro: otherwise I think you could use a snakeoil cert04:53
mgagnezaro: mine is located here: /etc/ssl/certs/ssl-cert-snakeoil.pem04:53
*** _nadya_ has quit IRC04:53
*** wenlock has quit IRC04:54
*** praneshp has quit IRC04:54
zaroughh, it's all so confusing.  will try those out.  thanks.04:55
*** ildikov has joined #openstack-infra04:57
*** coolsvap|afk is now known as coolsvap04:58
*** Longgeek has joined #openstack-infra05:02
mgagneclarkb: with gearman, when using SSL, is the server expecting the client to have a client certificate for authentication?05:02
clarkbmgagne: yeah05:03
clarkbpretty sure client auth is performed05:03
clarkbworkers too05:03
mgagneclarkb: ok, so zaro has to find a CA and sign a bunch of certs for his clients unless he can reuse the same cert as CA, server cert and client cert right?05:04
clarkbya05:04
*** coolsvap is now known as coolsvap|afk05:05
*** ildikov has quit IRC05:06
*** Longgeek has quit IRC05:07
*** wenlock has joined #openstack-infra05:12
*** nati_ueno has joined #openstack-infra05:17
*** Longgeek has joined #openstack-infra05:18
*** khyati has quit IRC05:20
*** basha_ has joined #openstack-infra05:21
* zaro very ignorant about keys05:22
zarothis has been the most useful so far http://forum.synology.com/wiki/index.php/How_to_generate_custom_SSL_certificates05:24
*** nati_ueno has quit IRC05:27
mgagne=)05:27
*** nati_ueno has joined #openstack-infra05:29
zaroclarkb: your suggestion about using sert file as ca cert didn't work.  but it could also be that i'm not doing right.05:29
zaroi've hit the end for tonight.  nighty-night05:30
*** gokrokve has joined #openstack-infra05:32
*** jamespage_ has joined #openstack-infra05:36
*** jamespage_ has quit IRC05:37
*** gokrokve has quit IRC05:37
*** zhiyan_ is now known as zhiyan05:42
SergeyLukjanovhttp://alexnisnevich.github.io/untrusted/ very nice game :)05:42
*** Ajaeger has joined #openstack-infra05:45
*** thuc has joined #openstack-infra05:53
*** nati_ueno has quit IRC05:55
*** thuc has quit IRC05:57
*** dcramer_ has joined #openstack-infra05:57
*** basha_ has quit IRC06:00
*** dcramer_ has quit IRC06:02
*** denis_makogon has joined #openstack-infra06:05
*** Ajaeger has quit IRC06:12
*** coolsvap|afk is now known as coolsvap06:18
*** e0ne has joined #openstack-infra06:21
*** nati_ueno has joined #openstack-infra06:32
*** gokrokve has joined #openstack-infra06:32
*** ildikov has joined #openstack-infra06:35
*** zns has quit IRC06:35
*** gokrokve has quit IRC06:36
*** rcarrillocruz has joined #openstack-infra06:40
*** adalbas has quit IRC06:46
*** adalbas has joined #openstack-infra06:50
openstackgerritFlorian Preinstorfer proposed a change to openstack-infra/jenkins-job-builder: Add support for the Jenkins Doxygen publisher plugin.  https://review.openstack.org/8660406:54
*** chandan_kumar has quit IRC06:57
*** flaper87|afk is now known as flaper8707:01
*** wenlock has quit IRC07:04
*** chandan_kumar has joined #openstack-infra07:05
*** matsuhashi has quit IRC07:06
*** matsuhashi has joined #openstack-infra07:08
SergeyLukjanovsdague, ping07:09
SergeyLukjanovsdague, could you please take a look at https://review.openstack.org/#/c/88422/ and https://review.openstack.org/#/c/88423/07:09
SergeyLukjanovsdague, due to the sahara enabled in d-g by default, we should update the job list for sahara07:09
*** jgallard has joined #openstack-infra07:10
ttxmordred, clarkb: we have a session to disucss my special brand of crazy: http://summit.openstack.org/cfp/details/29307:11
SergeyLukjanovttx, morning, very interesting opic07:15
SergeyLukjanovtopic*07:16
SergeyLukjanovttx, is proposed/icehouse a part of it?07:16
SergeyLukjanov(replacement for m-p branch I mean, don't actually remember the name pattern you proposed)07:16
ttxSergeyLukjanov: yes07:16
SergeyLukjanovttx, sahara was enabled by default in gate tonight :)07:17
ttxwin!07:17
openstackgerritThomas Herve proposed a change to openstack-infra/config: Add a new job for heat-templates  https://review.openstack.org/8376107:17
openstackgerritAndreas Jaeger proposed a change to openstack/requirements: Update openstack-doc-tools for Documentation repos  https://review.openstack.org/8804807:18
*** praneshp has joined #openstack-infra07:18
*** jcoufal has joined #openstack-infra07:20
*** andreykurilin_ has joined #openstack-infra07:20
therveSergeyLukjanov,  ^^^updated the heat gate job07:26
SergeyLukjanovttx, here are the last steps - replace custom jobs with common - https://review.openstack.org/#/c/88422/ https://review.openstack.org/#/c/88423/07:27
SergeyLukjanovtherve, lgtm07:28
*** dstanek has joined #openstack-infra07:31
therveThanks!07:31
*** gokrokve has joined #openstack-infra07:32
*** morganfainberg is now known as morganfainberg_Z07:32
*** afazekas has joined #openstack-infra07:33
*** e0ne_ has joined #openstack-infra07:34
*** dstanek has quit IRC07:35
*** e0ne has quit IRC07:37
*** gokrokve has quit IRC07:37
*** hashar has joined #openstack-infra07:37
*** jp_at_hp has joined #openstack-infra07:38
*** nati_ueno has quit IRC07:45
*** andreykurilin_ has quit IRC07:52
*** jpich has joined #openstack-infra07:54
*** dcramer_ has joined #openstack-infra07:59
*** ildikov_ has joined #openstack-infra08:01
*** dcramer_ has quit IRC08:03
*** jcoufal has quit IRC08:03
*** ildikov has quit IRC08:04
*** Ajaeger has joined #openstack-infra08:09
Ajaegergood morning, anybody around to review a requirements patch, please? https://review.openstack.org/#/c/88048/08:10
Ajaegerah, SergeyLukjanov you did already - thanks!08:10
SergeyLukjanovAjaeger, morning, np :)08:11
*** pblaho has joined #openstack-infra08:11
openstackgerritThomas Herve proposed a change to openstack-infra/config: Add a new job for heat-templates  https://review.openstack.org/8376108:19
*** achuprin has quit IRC08:19
*** oomichi has quit IRC08:19
*** achuprin has joined #openstack-infra08:22
*** denis_makogon has quit IRC08:22
*** ihrachyshka has joined #openstack-infra08:25
*** lukego has joined #openstack-infra08:27
*** lukego has quit IRC08:27
*** lukego has joined #openstack-infra08:27
*** jhesketh_ has joined #openstack-infra08:29
*** jhesketh_ has quit IRC08:30
*** jhesketh has joined #openstack-infra08:31
*** dstanek has joined #openstack-infra08:31
openstackgerritThomas Herve proposed a change to openstack-infra/config: Add a new job for heat-templates  https://review.openstack.org/8376108:31
*** gokrokve has joined #openstack-infra08:32
lukegoHoi. re: Tail-f jenkins comments, sorry about that, I had a misconfigured trigger for some hours yesterday while setting up jenkins. afaik the problem stopped around 24h ago. if there’s more trouble or actions needed please shoot me an email using the contact info on the wiki page.08:35
*** rcarrillocruz has quit IRC08:36
*** rcarrillocruz has joined #openstack-infra08:36
*** dstanek has quit IRC08:36
*** gokrokve has quit IRC08:36
*** jlibosva has joined #openstack-infra08:38
*** unicell has quit IRC08:42
Ajaegerlukego: haven't seen it again, thanks08:46
*** reed has joined #openstack-infra08:56
*** alexpilotti has joined #openstack-infra09:03
*** ihrachyshka has quit IRC09:11
*** rcarrillocruz has quit IRC09:21
*** jcoufal has joined #openstack-infra09:23
*** ildikov has joined #openstack-infra09:27
SergeyLukjanovlukego, yup, I've not seen it for last ~24h09:27
*** ildikov_ has quit IRC09:31
*** gokrokve has joined #openstack-infra09:32
*** Ajaeger has quit IRC09:35
*** hogepodge has quit IRC09:36
*** gokrokve has quit IRC09:37
*** hogepodge has joined #openstack-infra09:37
*** bookwar has quit IRC09:43
*** ChanServ sets mode: +o SergeyLukjanov09:44
*** ChanServ sets mode: -o SergeyLukjanov09:44
*** bookwar has joined #openstack-infra09:44
*** ihrachyshka has joined #openstack-infra09:55
*** lukego has quit IRC10:01
*** e0ne_ has quit IRC10:02
*** e0ne has joined #openstack-infra10:02
*** jgallard has quit IRC10:02
*** praneshp has quit IRC10:03
*** e0ne has quit IRC10:06
*** dkehn__ has joined #openstack-infra10:11
*** dkehn_ has quit IRC10:14
*** dkehn has quit IRC10:14
*** dkehn has joined #openstack-infra10:16
*** om has joined #openstack-infra10:18
*** blamar has quit IRC10:21
*** rcarrillocruz has joined #openstack-infra10:27
*** rcarrillocruz has quit IRC10:29
*** rcarrillocruz has joined #openstack-infra10:29
*** gokrokve has joined #openstack-infra10:32
*** gokrokve has quit IRC10:37
*** unicell has joined #openstack-infra10:42
*** e0ne has joined #openstack-infra10:43
sdagueSergeyLukjanov:10:50
openstackgerritSean Dague proposed a change to openstack-infra/config: don't vote on icehouse jobs we don't on master  https://review.openstack.org/8854310:50
sdagueany chance you can push that change in?10:50
*** coolsvap is now known as coolsvap|afk10:50
sdagueturns out we started voting on some neutron jobs that don't pass in tempest with the icehouse gating10:51
sdaguebecause our regex are strict :)10:51
*** om has quit IRC10:54
*** e0ne has quit IRC11:00
*** e0ne has joined #openstack-infra11:00
*** nosnos has quit IRC11:01
*** CaptTofu has joined #openstack-infra11:07
*** om has joined #openstack-infra11:09
*** matsuhashi has quit IRC11:12
*** matsuhashi has joined #openstack-infra11:14
*** jgallard has joined #openstack-infra11:19
SergeyLukjanovsdague, looking11:20
sdaguealso, I should hopefully have the n-cpu partial bit fixed11:21
SergeyLukjanovsdague, approved11:21
sdaguethanks11:22
SergeyLukjanovsdague, I want to push https://review.openstack.org/#/c/88422/ and https://review.openstack.org/#/c/88423/ to add sahara to the integrated queue in gate pipeline and replace custom jobs11:22
sdagueSergeyLukjanov: so why isn't heat-slow in that list?11:23
dimsSergeyLukjanov, sdague, libvirt 1.2.x etc from UCA/precise/icehouse is behaving well. Where should i look for scripts that create/update images for each provider?11:23
*** _nadya_ has joined #openstack-infra11:23
SergeyLukjanovsdague, did you mean check-tempest-dsvm-neutron-heat-slow?11:23
sdagueSergeyLukjanov: yes11:23
sdaguedims: that's a good question, and one that I don't quite know :)11:24
SergeyLukjanovsdague, hm, it's non-voting in check pipeline and doesn't affect sahara11:25
SergeyLukjanovsdague, do you think we should add it?11:25
sdagueI thought sahara was building a heat based provisioning tool11:25
dimssdague, :) np. still looking. i really don't like the fact that we are running on a patched libvirt11:25
openstackgerritA change was merged to openstack-infra/config: don't vote on icehouse jobs we don't on master  https://review.openstack.org/8854311:25
sdagueso it seems like working heat is an important part of the story11:25
SergeyLukjanovsdague, yup, sounds fair, I'll add it in separated commit11:26
*** nosnos has joined #openstack-infra11:26
sdaguecool11:27
*** nosnos has quit IRC11:29
*** om has quit IRC11:29
sdaguebtw, I think sahara wins for fastest project to be put into the integrated gate after release11:30
sdaguewithin 12 hrs of release sahara was in :)11:30
sdaguegood work on that11:30
*** rcarrill` has joined #openstack-infra11:30
SergeyLukjanov:)11:31
*** matsuhashi has quit IRC11:32
*** gokrokve has joined #openstack-infra11:32
*** matsuhashi has joined #openstack-infra11:32
*** rcarrillocruz has quit IRC11:33
*** aglarendil|nb has joined #openstack-infra11:34
openstackgerritA change was merged to openstack-infra/config: Add integrated-gate-sahara jobs  https://review.openstack.org/8842211:34
SergeyLukjanovI hope that we'll have provisioning testing in gate11:35
*** CaptTofu has quit IRC11:35
SergeyLukjanovin summit time11:35
SergeyLukjanovsdague ^^11:35
*** CaptTofu has joined #openstack-infra11:35
*** gokrokve has quit IRC11:37
*** e0ne has quit IRC11:37
*** matsuhashi has quit IRC11:38
*** rediskin has joined #openstack-infra11:38
openstackgerritSergey Lukjanov proposed a change to openstack-infra/config: Add tempest-dsvm-neutron-heat-slow to sahara gate  https://review.openstack.org/8855211:38
openstackgerritSergey Lukjanov proposed a change to openstack-infra/config: Remove obsolete custom Sahara dsvm jobs  https://review.openstack.org/8842311:38
rediskinsdague SergeyLukjanov could you pls take a look? i have a lot of work depends on it https://review.openstack.org/#/c/88272/11:39
openstackgerritVladislav Kuzmin proposed a change to openstack-infra/nodepool: Add layer for work with KVM/QEMU hosts  https://review.openstack.org/8855411:39
openstackgerritVladislav Kuzmin proposed a change to openstack-infra/nodepool: Correction logic for work with new KVM/QEMU provider manager  https://review.openstack.org/8855511:39
*** CaptTofu has quit IRC11:40
*** aglarendil|nb has quit IRC11:40
sdagueSergeyLukjanov: cool11:41
SergeyLukjanovsdague, we're working on fake plugin that'll make us able to test the whole provisioning layer w/o running hadoop on top of it and it'll work good in gate, in fact it's test of sahara-heat integration11:41
sdagueSergeyLukjanov: nice11:42
sdaguethat sounds very cool11:42
SergeyLukjanovsdague, could you please +1 https://review.openstack.org/#/c/88552/&11:42
SergeyLukjanovrediskin, looking11:42
rediskinsdague SergeyLukjanov thanks11:43
*** e0ne has joined #openstack-infra11:43
sdagueSergeyLukjanov: +111:44
SergeyLukjanovsdague, thx11:44
SergeyLukjanovrediskin, nice job name ...-rally-rally11:44
openstackgerritVladislav Kuzmin proposed a change to openstack-infra/nodepool: Correction logic for work with new KVM/QEMU provider manager  https://review.openstack.org/8855511:44
openstackgerritA change was merged to openstack-infra/config: Remove obsolete custom Sahara dsvm jobs  https://review.openstack.org/8842311:46
openstackgerritVictor Sergeyev proposed a change to openstack-infra/config: Import oslo.db repository  https://review.openstack.org/7895511:46
openstackgerritA change was merged to openstack-infra/config: Add tempest-dsvm-neutron-heat-slow to sahara gate  https://review.openstack.org/8855211:52
openstackgerritVladislav Kuzmin proposed a change to openstack-infra/nodepool: Correction logic for work with new KVM/QEMU provider manager  https://review.openstack.org/8855511:54
openstackgerritVladislav Kuzmin proposed a change to openstack-infra/nodepool: Add layer for work with KVM/QEMU hosts  https://review.openstack.org/8855411:54
*** dprince has joined #openstack-infra11:56
*** _nadya_ has quit IRC12:00
openstackgerritSergey Skripnick proposed a change to openstack-infra/config: Add rally-jobs job group  https://review.openstack.org/8827212:00
*** ccorrigan has joined #openstack-infra12:02
ccorrigananyone working on gate-tripleo-image-elements? its currently breaking on docs and python2612:04
*** lnxnut has joined #openstack-infra12:08
openstackgerritVictor Sergeyev proposed a change to openstack-infra/config: Import oslo.db repository  https://review.openstack.org/7895512:08
*** dstanek has joined #openstack-infra12:11
*** pdmars has joined #openstack-infra12:23
*** dstanek has quit IRC12:24
*** pdmars has quit IRC12:24
*** pdmars has joined #openstack-infra12:24
*** gokrokve has joined #openstack-infra12:32
*** CaptTofu has joined #openstack-infra12:34
*** mriedem has joined #openstack-infra12:35
openstackgerritA change was merged to openstack-infra/jenkins-job-builder: Add hacking to requirements  https://review.openstack.org/8667912:36
*** gokrokve has quit IRC12:37
*** dstanek has joined #openstack-infra12:39
*** jhesketh has quit IRC12:42
*** flaper87 is now known as flaper87|afk12:44
*** dcramer_ has joined #openstack-infra12:45
*** dkehn__ is now known as dkehnx12:46
*** rlandy has joined #openstack-infra12:46
*** andreykurilin_ has joined #openstack-infra12:48
*** dcramer_ has quit IRC12:49
*** dcramer_ has joined #openstack-infra12:50
*** aglarendil|nb has joined #openstack-infra12:59
*** mfer has joined #openstack-infra12:59
*** andreykurilin_ has quit IRC13:00
openstackgerritSalvatore Orlando proposed a change to openstack-infra/config: Make neutron full job voting (neutron gate only)  https://review.openstack.org/8828913:02
*** zz_gondoi is now known as gondoi13:03
openstackgerritAntoine Musso proposed a change to openstack-infra/zuul: Add require-approvals to EventFilter representation  https://review.openstack.org/8857013:04
rediskinsdague: could you pls take a look? https://review.openstack.org/#/c/88272/13:06
YorikSarjeblair: Can you take another look at https://review.openstack.org/87872 ? It looks like everybody's finally happy with it.13:07
*** miqui has joined #openstack-infra13:08
*** dims has quit IRC13:09
*** gondoi is now known as zz_gondoi13:11
*** dcramer_ has quit IRC13:13
*** yaguang has quit IRC13:14
*** aglarendil|nb has quit IRC13:18
*** dims has joined #openstack-infra13:22
*** aglarendil has quit IRC13:23
openstackgerritA change was merged to openstack/requirements: Update openstack-doc-tools for Documentation repos  https://review.openstack.org/8804813:25
*** dcramer_ has joined #openstack-infra13:25
*** dcramer_1 has joined #openstack-infra13:25
*** dcramer_ has quit IRC13:26
openstackgerritA change was merged to openstack/requirements: Remove old pep8, pyflakes and flake8 versions  https://review.openstack.org/8675913:26
*** timrc-afk is now known as timrc13:26
*** andreykurilin_ has joined #openstack-infra13:28
*** gokrokve has joined #openstack-infra13:32
salv-orlandosdague: good morning. I read a comment from anteaya regarding https://review.openstack.org/#/c/88289/ and a conflict with https://review.openstack.org/#/c/8830213:33
salv-orlandoshe mentioned a conflict but can't find any - maybe is that about ensuring 4 jobs are executed on stable/icehouse as well?13:34
openstackgerritJulien Danjou proposed a change to openstack/requirements: Upgrade six requirement to 1.6.0  https://review.openstack.org/8857513:36
*** gokrokve has quit IRC13:37
*** bknudson has joined #openstack-infra13:38
*** zns has joined #openstack-infra13:38
*** prad_ has joined #openstack-infra13:38
*** lcostantino has joined #openstack-infra13:39
*** chandan_kumar has quit IRC13:42
*** lukego has joined #openstack-infra13:43
*** sweston has quit IRC13:44
*** thuc_ has joined #openstack-infra13:44
*** signed8bit has joined #openstack-infra13:46
*** wchrisj has joined #openstack-infra13:47
*** maxbit has joined #openstack-infra13:48
sdaguesalv-orlando: yeh, I think in the strictest sense there would be a merge conflict13:50
sdaguebecause they were changing related chunks13:50
sdagueso I think if you just rebase now that the tempest change landed, life should be good13:51
mnasersdague: jenkins failed for the revert review, probably needs a recheck due to a neutron thing.13:52
salv-orlandosdague: I did a rebase and had no conflict. However, I'm thinking that the patch as it is will put us in a situation where we have only non-voting jobs on stable branches.13:52
salv-orlandobecause I'm keeping the full job non voting on stable branches. Especially for havana it will fail most of the times.13:53
sdaguesalv-orlando: so you could leave the smoke job on stable13:55
salv-orlandothat's what I want to do. I think I can edit project.yaml and add something there13:56
salv-orlandowith branch-override set to stable/icehouse13:56
sdaguebasically, don't delete it from the neutron gate, and instead use a branch stanza on name: ^(gate|check)-tempest-dsvm-neutron$13:56
salv-orlandook makes sense13:57
sdaguesalv-orlando: https://github.com/openstack-infra/config/blob/master/modules/openstack_project/files/zuul/layout.yaml#L38113:58
sdaguean example of doing that13:58
sdaguerunning only on stable13:58
salv-orlandosdague: thanks14:00
*** vhoward has joined #openstack-infra14:02
*** blamar has joined #openstack-infra14:02
*** mriedem1 has joined #openstack-infra14:03
*** mriedem has quit IRC14:04
*** achuprin has quit IRC14:04
*** achuprin has joined #openstack-infra14:07
*** andreykurilin_ has quit IRC14:08
*** sw_away is now known as sandywalsh14:10
*** lukego has quit IRC14:18
*** markmcclain has joined #openstack-infra14:20
*** msabramo has quit IRC14:21
* anteaya is awake14:22
anteayachecking email14:22
anteayadeva seems to be out last candidate14:25
anteayaour14:25
*** andreykurilin_ has joined #openstack-infra14:27
SergeyLukjanovanteaya, morning14:27
*** dcramer_1 has quit IRC14:28
*** dcramer_ has joined #openstack-infra14:28
*** sweston has joined #openstack-infra14:29
*** gokrokve has joined #openstack-infra14:32
*** dkliban is now known as dkliban_afk14:33
*** msabramo has joined #openstack-infra14:33
*** gokrokve has quit IRC14:37
*** ilives has quit IRC14:40
*** atiwari has joined #openstack-infra14:42
anteayaSergeyLukjanov: morning14:44
sdagueanteaya: any chance there will be a single wiki page with links to everyone's post for TC?14:45
*** thedodd has joined #openstack-infra14:46
sdagueto make it easy for voters to see all the platforms without hunting the list themselves14:46
*** thuc has joined #openstack-infra14:46
*** prad_ has quit IRC14:47
YorikSarLet's create a Git repo for that. It's so trending today :)14:47
anteayasdague: https://wiki.openstack.org/wiki/TC_Elections_April_2014#Candidates14:48
sdaguesee, you are just awesome :)14:49
anteayasdague: awww, thanks14:49
*** bookwar has quit IRC14:49
anteayaYorikSar: campagin however you like, just make sure it is public for all to access14:49
*** thuc_ has quit IRC14:50
*** bookwar has joined #openstack-infra14:50
*** tjones has joined #openstack-infra14:50
YorikSaranteaya: I'm just kidding :)14:50
*** zns has quit IRC14:51
reedttx, any clue about this ? https://ask.openstack.org/en/question/27463/authors-file-missing-in-the-cinder-repository-to-allow-merging-in-trunk/14:52
ttxreed: looking...14:53
*** jcoufal has quit IRC14:53
*** matrohon has quit IRC14:53
*** wenlock has joined #openstack-infra14:54
*** gokrokve has joined #openstack-infra14:54
ttxreed: so nowadays AUTHORS file is autogenerated14:55
ttxthat said... the one in Cinder 2014.1 tarball looks hopelessly out of date.14:55
ttxmordred: pbr autogenerates it, right ?14:56
reedI guess there is no mandate to do anything like the cinder dev documentation is saying, right?14:56
*** prad_ has joined #openstack-infra14:57
ttxreed: probably a bug in doc. Will answer that question and followup14:57
reedttx, thank you14:57
reedttx, jgriffith told me on #openstack-cinder https://bugs.launchpad.net/sahara/+bug/126177414:59
uvirtbotLaunchpad bug 1261774 in sahara "Remove AUTHORS file from repos" [Medium,Fix released]14:59
ttxreed: sure, but in the cinder case AUTHORS is already removed and that doesn't make it cirrectly autogenerated either15:00
reedthe docs is old, too15:01
openstackgerritNikita Konovalov proposed a change to openstack-infra/storyboard-webclient: TimeLine events support  https://review.openstack.org/8840515:01
openstackgerritNikita Konovalov proposed a change to openstack-infra/storyboard: Added service comments  https://review.openstack.org/8795915:02
*** pcrews has joined #openstack-infra15:02
*** mriedem has joined #openstack-infra15:04
*** zns has joined #openstack-infra15:05
openstackgerritAntoine Musso proposed a change to openstack-infra/git-review: Skip invalid unicode in commit messages  https://review.openstack.org/8282615:05
*** mriedem1 has quit IRC15:08
ttxmordred, jeblair: https://bugs.launchpad.net/pbr/+bug/130958515:09
uvirtbotLaunchpad bug 1309585 in pbr "AUTHORS file generation is borken" [Undecided,New]15:09
ttxreed: answered15:09
reedttx, thanks15:10
*** ihrachyshka has quit IRC15:11
*** thuc_ has joined #openstack-infra15:11
*** thomasem has joined #openstack-infra15:13
*** thomasem has quit IRC15:14
*** thomasem has joined #openstack-infra15:15
*** thuc has quit IRC15:15
*** lcostantino has quit IRC15:15
*** ildikov has quit IRC15:22
*** thuc_ has quit IRC15:22
openstackgerritA change was merged to openstack-infra/config: Add a new job for heat-templates  https://review.openstack.org/8376115:23
anteayago: http://lists.openstack.org/pipermail/openstack-dev/2014-April/033173.html15:23
*** thuc has joined #openstack-infra15:23
ttxanteaya: yay! let the games begin15:24
anteayayay15:24
ttxanteaya: note that there is a typo for Favio's name on the wiki page15:25
anteayaargh15:25
ttxit's Percoco15:25
anteayaflaper87|afk: sorry about the typo in your name in the wikipage15:26
ttxhe is also misnamed on the ballot, but then there is only one Fkavio so damage is limited15:26
anteayaflaper87|afk: I did a straight copy/paste from the wikipage for the poll15:26
anteayayeah, except it is embarassing15:26
*** jgallard has quit IRC15:26
anteayagetting the names spelled properly is important15:26
*** thuc has quit IRC15:27
*** nati_ueno has joined #openstack-infra15:29
YorikSarjeblair: Thanks for +215:29
*** nati_ueno has quit IRC15:29
YorikSarjeblair: How about +A? :)15:30
anteayahttp://lists.openstack.org/pipermail/openstack-dev/2014-April/033176.html15:30
openstackgerritMarc Abramowitz proposed a change to openstack-infra/jenkins-job-builder: Add `browse` command  https://review.openstack.org/8558015:30
jeblairYorikSar: going to give the latest version a little more time first15:31
YorikSarjeblair: Ok. Thanks :)15:31
YorikSarjeblair: I'm thinking about adding a filter to further limit the size of JSON to allow users to look at their CR progress only.15:34
*** david-lyle has joined #openstack-infra15:35
msabramojenkins-job-builder folks: I could use another review for https://review.openstack.org/84778 — I'd love to get this merged soon as it will make it easier to test other changes I hope to work on15:36
openstackgerritSalvatore Orlando proposed a change to openstack-infra/config: Make neutron full job voting (neutron gate only)  https://review.openstack.org/8828915:36
msabramohashar already approved it; need 1 more for merge15:36
msabramo(fingers crossed :-))15:36
hasharmsabramo: lot of folks where attending US Pycon and european folks (at least) have Easter on monday.15:37
hasharmsabramo: so there might be some delay :-D15:37
msabramohashar: yeah, I was at Pycon as well15:37
hasharmsabramo: I wish I had a million dollars to relocate to US and attend all those conferences hehe15:37
msabramohashar: but yeah delay is a fact of life; no big deal15:38
hasharmsabramo: at least I am sure the patch will land eventually15:38
msabramohashar: you don't need a million dollars, you just need a job at an open source company such as Rackspace or Yahoo :-)15:39
*** thuc has joined #openstack-infra15:39
hasharI am probably just finding good excuses to just stay at home so15:40
msabramohashar: all these patches are all intertwined15:40
*** thuc has quit IRC15:40
msabramohashar: it's too much for my small brain to handle :-)15:40
*** pblaho has quit IRC15:40
*** thuc has joined #openstack-infra15:40
*** msabramo1 has joined #openstack-infra15:43
msabramo1hashar: Maybe you can get your company to pay for you to go to Pycon in Montreal next year — good times and you will get to hang out with lots of francophones. I met mgagne and jeblair but not you unfortunately.15:44
ttxjeblair: does the pbr bug minked above ring a bell ? Do you agree that those Authors files are suspiciously empty ?15:45
ttxlinked*15:45
msabramo1I met a lot of francophones - my colleage, Alex Conrad, is French and knows a lot of French Pythonistas from AfPy15:45
*** msabramo has quit IRC15:45
ttxdamn French15:45
anteayathey are all over the place15:45
hasharmsabramo1: I am a contractor for Wikimedia Foundation and we are on a tight budget. So we only travel when it is absolutely necessary.15:46
msabramo1the French seem to love Python15:46
* hashar gives another La Lubie to ttx15:46
* ttx thinks we love Python because we don't have any wandering in our streets15:46
msabramo1hashar: maybe you can get a grant from OpenStack Foundation or PSF15:46
anteayathere are travel grants to attend summits15:48
*** jlibosva has quit IRC15:49
*** e0ne has quit IRC15:50
*** zhiyan is now known as zhiyan_15:50
*** msabramo1 has quit IRC15:50
*** chandan_kumar has joined #openstack-infra15:51
hasharI can most probably show up at the Paris summit beginning of November since that is closely and i got accommodations there hehe15:52
sdaguejeblair: I'm about to pumpkin for the day, however 88488,2 is really important to land. We were missing the partial test on d-g. Until it lands, devstack, tempest, grenade, and nova are blocked from landing code.15:52
sdagueit's top of gate right now15:52
sdaguehowever it was this morning as well, and bounced off a neutron race15:52
sdagueso hopefully in 20 minutes everything is just fine. But if it bounces again, it might be worth a straight ninja merge15:53
zaromorning15:53
*** nati_ueno has joined #openstack-infra15:54
sdaguehttps://review.openstack.org/#/c/88492/ - will close the failing to test parial bug15:54
SpamapShttps://review.openstack.org/#/c/69918/12 ... failing on a docs check.. but.. I'm not sure why tripleo-image-elements has a docs check.. as.. it has.. no docs.15:55
*** jpich has quit IRC15:55
*** msabramo has joined #openstack-infra15:57
mordredttx: I just tried locally and I get the AUTHORS file I would expect15:58
ttxmordred: you agree that "what you expect" is not "what is in the release tarballs", though ?15:59
jeblairhttp://logs.openstack.org/d1/d135dde44eb80d15cf6ec8f9077d5fe410223eff/release/glance-tarball/3586bb0/console.html16:00
jeblairttx, mordred: ^ i think that is the release build, yeah?16:00
anteayamorning zaro16:00
mordredjeblair: did we start doing shallow clones at some point?16:01
ttxjeblair: let me check md516:01
jeblairmordred: i don't see a shallow clone in there16:01
ttxjeblair: yes that's the job that generated the glance tarball16:01
ttxmordred: fwiw the nova AUTHORS is slightly longer, but still nowhere as long as it should16:02
mordredjeblair: me either - I just had a moment of remembering doing shallow clones16:02
ttxso it's not just taking the first 4 names16:02
ttxmaybe some kind of race ?16:02
mordredwhat is in the tarball is not what I get when I run things locally16:03
* mordred poking deeper16:05
mordredthis is extremely strange16:05
*** ildikov has joined #openstack-infra16:06
*** thuc has quit IRC16:07
*** thuc has joined #openstack-infra16:07
jeblairmordred: i reproduced it locally16:09
sdagueSpamapS: https://github.com/openstack-infra/config/commit/2774501523a3bee388e52ba2ba119d1770e74850 - that was added by bnemec16:10
zarojeblair: i've rebased my change on yours https://review.openstack.org/#/c/8802616:10
jeblairmordred: ttx: tried to replicate what the actual build did as closely as possible: http://paste.openstack.org/show/76289/16:11
sdagueok, later folks, have a good weekend16:11
SpamapShrm16:11
SpamapSok so I need a way to make docs a noop16:11
jeblairsdague: you too (ack on that change btw)16:11
*** yfried__ has quit IRC16:12
*** thuc has quit IRC16:12
SpamapSooo16:12
SpamapSmy git has colors in trusty16:12
*** gokrokve has quit IRC16:13
mordredjeblair: ack. I did something similar, and I got a regular authors file16:14
*** Sukhdev has joined #openstack-infra16:15
openstackgerritAntoine Musso proposed a change to openstack-infra/zuul: Support email lists in require-approval  https://review.openstack.org/8861616:15
openstackgerritA change was merged to openstack-infra/jenkins-job-builder: Add support for the Jenkins Doxygen publisher plugin.  https://review.openstack.org/8660416:16
jeblairmordred: btw, i'm confused by "--use-mailmap" which dosen't seem to work with my 'git log'16:16
zigomordred: & sdague: If you missed it, tgt with init script finally got approved in debian backports.16:17
pleia2good morning16:18
*** shakamunyi has joined #openstack-infra16:18
openstackgerritMichael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Enabled functional tests  https://review.openstack.org/8861716:19
*** nati_ueno has quit IRC16:19
mordredjeblair: it works in mine - perhaps there are git versions that don't have it and we're not properly catching that16:19
* ttx signoffs16:20
ttxmordred, jeblair: ping me when you understand what's happening :)16:20
reedgood friday16:20
openstackgerritAntoine Musso proposed a change to openstack-infra/zuul: Support email lists in require-approval  https://review.openstack.org/8861616:20
reedI'm taking off too,16:20
reedit's Easter time here16:20
*** nati_uen_ has joined #openstack-infra16:20
*** reed has quit IRC16:20
mordredjeblair: but you still get an AUTHORS list yeah?16:21
mordredjeblair: or do you get what the release gets?16:21
jeblairmordred: running the commands in that paste, i get an abbreviated authors list.  running "git log --use-mailmap '--format=%aN <%aE>'" i get fatal: unrecognized argument: --use-mailmap16:22
*** thuc has joined #openstack-infra16:22
SpamapShttps://review.openstack.org/#/c/88618/ <-- is this the best way to solve tripleo-image-elements failing gate because the docs job fails, or is there a simple way to exclude that job from that gate?16:22
mordredjeblair: aha!16:22
mordredthen you've found the bug. I'm confused why there is anybody in the list at all in your case16:22
jeblairmordred: me too, which is why i hadn't assumed i had found the bug yet :)16:22
mordredwell, you've _reproduced_ the bug - which is more than I can do16:23
mordredso at least to some degree, precise git vs. trusty git is failing us16:23
jeblairmordred: i've reproduced it on a fresh clone16:24
*** andreykurilin_ has quit IRC16:24
*** hashar has quit IRC16:24
jeblaireg: git clone glance && cd glance && tox -evenv python setup.py sdist16:24
jeblairso yeah, signs are pointing to "this doesn't work on precise"16:24
*** marun_afk has quit IRC16:25
jeblairmordred: were you using trusty in october?16:26
*** dcramer_1 has joined #openstack-infra16:26
jeblairhttp://git.openstack.org/cgit/openstack-dev/pbr/commit/?id=745d615c678b412c89be1cc3615c19d9bd9b983716:27
*** dcramer_ has quit IRC16:29
SpamapShttp://logs.openstack.org/18/88618/1/check/gate-tripleo-image-elements-docs/8442350/console.html ... failing even though 'tox -evenv -- python setup.py build_sphinx' works locally for me....16:31
*** dcramer_1 has quit IRC16:31
mordredjeblair: I was using saucy16:32
openstackgerritA change was merged to openstack-infra/devstack-gate: Revert "update grenade branches"  https://review.openstack.org/8848816:32
jeblairmordred: so _run_git_command _never_ raises an exception on error16:32
*** bknudson has quit IRC16:32
openstackgerritMonty Taylor proposed a change to openstack-dev/pbr: Revert "Let git do mailmap mapping for us"  https://review.openstack.org/8862516:33
*** gyee has joined #openstack-infra16:33
jeblairso pbr pretty much ignores all git errors16:33
mordredjeblair: I feel like there was a specific reason we had it ignore git errors but at the moemnt i can't think of a good reason for that16:34
*** sweston has quit IRC16:34
mnasernow that the above problem has been fixed, can I force a reverify mid-jenkins job to restart, or do I have to wait for it to finish completely first?16:34
jeblairmordred: perhaps the default should be flipped16:34
mordredjeblair: yah16:35
mnasermore specifically, trying to get this in to avoid the unit test race conditions - https://review.openstack.org/#/c/88324/16:35
jeblairmnaser: either wait, or push a new patchset (which will cancel/restart jobs)16:35
mnaserjeblair: thanks, probably won't push again because then i'd have to get approved, etc. again.. guess i'll wait!16:35
*** dangers_away is now known as dangers16:36
*** Longgeek has quit IRC16:36
*** nati_uen_ has quit IRC16:37
SpamapSn/m .. something wonk in there16:37
jeblairttx, mordred: AUTHORS should actually be called "CO-AUTHORS"16:38
*** Ryan_Lane has joined #openstack-infra16:38
openstackgerritA change was merged to openstack-infra/jenkins-job-builder: Make logger available during tests  https://review.openstack.org/8682516:38
*** thuc_ has joined #openstack-infra16:38
jeblairas we found above, the actual authors list wasn't being generated because pbr was using a newer git command line syntax than what is available in precise.  but the part that scans for co-authors didn't use that syntax and so worked.  so the abbreviated list is just the list of co-authors.16:39
jeblairttx, mordred: ^16:39
openstackgerritA change was merged to openstack-infra/jenkins-job-builder: Allow to specify multiple branches per project in Gerrit trigger  https://review.openstack.org/8564816:39
*** comstud is now known as bearhands16:39
openstackgerritMonty Taylor proposed a change to openstack-dev/pbr: Revert "Let git do mailmap mapping for us"  https://review.openstack.org/8862716:40
mordredjeblair: hah. nice. that explains it16:40
mordredjeblair: I updated the commit message on the above commit16:40
jeblairmordred: i +2d16:40
SpamapSwtf.. I have sphinx pinned to <1.2, but Sphinx-1.2b3-py2.7.egg-info is in my venv16:41
*** Ajaeger has joined #openstack-infra16:41
AjaegerHi everybody. The translation proposal jobs are working now - do you want to abandon the old ones that were run with user Jenkins?16:42
openstackgerritA change was merged to openstack-infra/jenkins-job-builder: Support Fitnesse results publisher  https://review.openstack.org/8771116:42
mordredSpamapS: that's <1.216:42
mordredSpamapS: what version of pip are you using?16:42
SpamapSmordred: the feeling of dread I have when you ask that is hard to describe ;)16:43
*** bhuvan has joined #openstack-infra16:43
*** thuc has quit IRC16:43
jeblair"the wrong one"16:43
SpamapSmordred: pip --version shows 1.5.416:43
SpamapStox 1.6.116:44
SpamapSalso have we tried to push the source_dir thing upstream to sphinx?16:44
mordredSpamapS: that version of pip should appropriately not download pre-releases unless you ask for them16:45
SpamapSmordred: is tox for sure using that pip?16:45
SpamapSvenv runtests: commands[0] | pip --version16:47
SpamapSpip 1.5.4 from /home/clint/src/tripleo-image-elements/.tox/venv/local/lib/python2.7/site-packages (python 2.7)16:47
*** jogo is now known as flashgordon16:47
* SpamapS is perplexed16:47
*** khyati has joined #openstack-infra16:47
mordredSpamapS: oh - well... possibly not16:48
mordredyeah - t-i-e's tox.ini looks good16:49
Ajaegermordred, jeblair, clarkb can either of you abandon the old translation patches that were sent by Jenkins?16:51
*** BadCub01 has joined #openstack-infra16:51
jeblairAjaeger: will do16:51
Ajaegerjeblair: use this query: https://review.openstack.org/#/q/status:open++branch:master+topic:transifex/translations,n,z16:51
Ajaegerjeblair: thanks!16:51
jeblairmordred: btw, i see a good deal of hard-coding jenkins addresses, etc in pbr.  i think the change to use proposal bot will affect that16:51
Ajaegerjeblair: and only take the one by Jenkins  ;)16:51
mordredjeblair: yup16:53
Ajaegerjeblair: that was quick - thanks!16:54
*** branen has joined #openstack-infra16:54
jeblairmordred: i think it might match "infra@lists" right now, but clarkb was thinking of changing that16:55
SpamapSso...16:55
mordredjeblair: I will wait for clarkb to tell me things16:55
SpamapSt-i-e's gate is totally broken now.16:55
mordredoh. well then I should care more16:56
SpamapSI ask again, is there a simple way to simply not run the docs part of the unit tests?16:56
mordredyou have docs things in your unittests?16:56
jeblairSpamapS: so it looks like they have released sphinx versions >1.2 that fix the bug16:56
SpamapSwe're running, and failing them16:56
jeblairaccording to mriedem http://bugs.python.org/issue1957016:57
jeblair1.2.1 works16:57
mordredjeblair: should we try to change our sphinx pin to >1.2.1 ?16:57
SpamapSlovely that 1.2b is there, and breaks stuff16:57
jeblairand the merge request for the workaround landed: https://bitbucket.org/birkenfeld/sphinx/pull-request/193/builddoc-shouldnt-fail-on-unicode-paths/diff16:57
jeblairand they are up to 1.2.2 now (unknown if that works)16:57
mordredSpamapS: well, I'd like to dig in to that - because you should not be getting 1.2b16:57
SpamapSyes I can confirm, >=1.2.1 works for me16:57
SpamapSmordred: happy to oblige.16:57
*** alaski is now known as lascii16:58
*** harlowja_away is now known as harlowja16:58
*** gokrokve has joined #openstack-infra16:58
SpamapSmordred: https://review.openstack.org/#/c/88618/16:58
openstackgerritMichael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Containers now left-align  https://review.openstack.org/8863016:59
*** Ryan_Lane has quit IRC16:59
*** markmcclain has quit IRC16:59
flashgordonclarkb: https://review.openstack.org/#/c/88492/ what broke when moving partial ncpu over?17:00
jeblairinstall_command is missing17:00
mordredSpamapS: oh - eek - ok, I want to edit that patch - why are you gating on docs already if you don't have docs?17:00
openstackgerritMonty Taylor proposed a change to openstack/requirements: Increase sphinx pin now that 1.2.1 is out  https://review.openstack.org/8863117:00
*** vponomaryov has left #openstack-infra17:01
boris-42hi all17:01
boris-42could somebody take a look at one more attempt to make rally gates works https://review.openstack.org/#/c/88272/17:01
jeblairmordred: ^ see comment about install_command?17:01
*** nati_ueno has joined #openstack-infra17:01
SpamapSmordred: I don't know. If there is a way to turn off just gating on docs, let's do that.17:01
*** thuc_ has quit IRC17:02
SpamapSmordred: recent commit enabled unit tests https://github.com/openstack-infra/config/commit/2774501523a3bee388e52ba2ba119d1770e7485017:02
mordredjeblair: when you say install_command is missing ...17:02
*** thuc has joined #openstack-infra17:02
mordredoh - in their tox.ini17:02
jeblairyup17:02
mordred*facepalm* thank you17:02
*** gokrokve has quit IRC17:03
openstackgerritMike Perez proposed a change to openstack-dev/hacking: Import Guideline Exception for gettextutils  https://review.openstack.org/8863217:04
*** jaypipes is now known as leakypipes17:04
SpamapSmordred: I don't mind having a stub and conforming. This shouldn't be hard.17:04
mordredSpamapS: I just pushed an update17:05
openstackgerritMike Perez proposed a change to openstack-dev/hacking: Import Guideline Exception for gettextutils  https://review.openstack.org/8863217:05
SpamapSmordred: ugh17:06
SpamapSmordred: so install_command has to be set or you get "evil pip" ?17:06
mordredI think so - it just worked for me locally and I got 1.1.3 in my venv17:06
SpamapSmordred: I too17:07
mordredSpamapS: you should run ci.md through a converter and make it rst - but that's less important here17:07
*** thuc has quit IRC17:07
mordredactually - let me send one update...17:07
*** dripton has quit IRC17:08
*** dripton has joined #openstack-infra17:09
*** ociuhandu has joined #openstack-infra17:11
clarkbaroo?17:17
clarkbI have 4 highlights on one page of sb17:17
*** gokrokve has joined #openstack-infra17:17
*** lakshmisampath has joined #openstack-infra17:18
mordredclarkb: FIX ALL THE THINGS!!!17:18
clarkbmordred: I pushed a few patches last night to do that17:18
mordredto fix them all?17:18
mordredneat17:18
* mordred goes back to bed - clarkb is here17:18
clarkbwell to fix ncpu and d-g's asymetrical gating of that test17:18
clarkbmordred: what thing were you looking for answers about pbr and infra@lists?17:19
lakshmisampathI have pending review on config project. (https://review.openstack.org/#/c/88139/). can someone review it?17:19
*** Ryan_Lane has joined #openstack-infra17:19
krotscheckHey, can I get someone to poke into the storyboard.o.o logs? I’m getting 500 errors on everything17:20
clarkbwow everything really is broken17:20
krotscheckOh, is this a known thing?17:20
clarkbno, probably not17:20
clarkbjust that mordred was asking if I fixed everything17:20
clarkboh ya install command because tox has a really silly default17:21
clarkbjeblair: mordred https://review.openstack.org/#/c/88492/ would be a good thing to get in17:21
clarkbthen we can be sure that we test the grenade branch switches properly before merging17:22
*** nati_ueno has quit IRC17:24
openstackgerritA change was merged to openstack-infra/config: Test d-g against partial n-cpu job.  https://review.openstack.org/8849217:26
SpamapSmordred: oh good.. we _also_ have never run tests on python 2.617:27
*** nati_uen_ has joined #openstack-infra17:28
flashgordonclarkb: can you point me to a failed grenade partial npcu job17:28
clarkbflashgordon: any that ran between about 2pm and 6pm PDT yesterday17:28
*** bknudson has joined #openstack-infra17:28
flashgordonsuch as?17:29
flashgordonclarkb: was hoping you had one already open17:29
* clarkb queries logstash17:29
clarkbI don't have one open17:29
clarkbactually may have one in my sb17:29
mnaserthere is an elastic recheck query for that issue, flashgordon17:29
mnasermriedem wrote it yesterday, me thinks17:30
mnaserhttps://github.com/openstack-infra/elastic-recheck/commit/d779b5293674d3174852a609c9cf09ac53072cf717:30
clarkbflashgordon: http://logs.openstack.org/24/88324/6/check/check-grenade-dsvm-partial-ncpu/2e18c53/logs/new/screen-n-sch.txt.gz#_2014-04-18_00_44_53_075 was the portion of config pointed out in a failing job that may be wrong17:30
clarkbflashgordon: you can see the errors in old/screen-n-cpu.txt17:30
flashgordonohh that makes sense17:31
boris-42clarkb hi17:31
boris-42clarkb could you pls take a look at this patch https://review.openstack.org/#/c/88272/ ?17:31
mnaserflashgordon: i think it's related to this which i'll be reverifying shortly after another race condition makes itway in - https://review.openstack.org/#/c/88494/17:31
mnaserthat fails unit tests17:31
boris-42clarkb seems without it our rally gates doesn't work =(17:32
boris-42dont*17:32
boris-42clarkb thanks17:32
*** sabari has left #openstack-infra17:32
clarkbboris-42: why does that change completely redo the job setup?17:33
*** dstanek has quit IRC17:33
flashgordonmnaser: yup that sounds right, along with https://review.openstack.org/#/c/8854917:33
clarkbboris-42: it is hard to understand what that change is fixing because it is refactoring the jobs too17:33
boris-42clarkb okay we have some strange issues (that environments variable are not stetted)17:34
mnaserflashgordon: right now i'm trying to get https://review.openstack.org/#/c/88324/ (in jenkins atm) to avoid the race condition, then https://review.openstack.org/#/c/88494/ to address that, then that review to test and see if it works after17:34
openstackgerritClint "SpamapS" Byrum proposed a change to openstack-infra/config: Revert "Run unit tests in tripleo-image-elements"  https://review.openstack.org/8863717:34
SpamapSmordred: ^^ reverting turning on the unit tests until we fix them17:34
clarkbboris-42: right, but that change doesn't just set those variables17:34
clarkbboris-42: it completely rewrites the way the jobs are done17:34
boris-42clarkb yep it's the another goal, to be able to run in other projects rally gates17:34
* clarkb is just trying to understand what this achieves17:34
boris-42clarkb e.g. in cinder we would like to run performance gates17:35
boris-42clarkb with specific rally task17:35
clarkbboris-42: you can;t really have performance gates17:35
boris-42clarkb why?17:35
clarkbbceause we get nodes from ~7 different AZs and they all act differently17:35
boris-42clarkb it's not the big deal17:35
clarkbyou can certainly do performance testing17:35
clarkbbut gating requires stability17:35
boris-42clarkb yep that our goal for juno17:36
boris-42clarkb to normalize results in case of different nodes17:36
boris-42clarkb but for now we would like to get any results17:36
jeblairmordred: are you going to try out the ssh manage-projects thing, or should we do manual new-project-friday?17:36
boris-42clarkb e.g. http://pavlovic.me/rally/glance_list.html17:36
boris-42clarkb even without absolute numbers it's clear that something wired is here17:37
*** aconrad has joined #openstack-infra17:37
*** lakshmi has joined #openstack-infra17:38
clarkbok, I think I understand that change now17:38
*** nati_uen_ has quit IRC17:38
boris-42clarkb so for example that graph with glance I got for 5 mins17:38
clarkbit allows you to also have check-rally-dsvm-cinder or check-rally-dsvm-glance17:38
*** lakshmisampath has quit IRC17:39
boris-42clarkb yep it should at least =)17:39
mordredjeblair: I want to do it with the new set of patches17:40
clarkbmordred: I will look at those patches next17:40
jeblairmordred: cool17:40
openstackgerritKhai Do proposed a change to openstack-infra/jenkins-job-builder: `test` command read/write stdin/stdout by default  https://review.openstack.org/8477817:41
clarkbmordred: https://review.openstack.org/#/c/88461/ can haz review?17:42
*** amcrn has joined #openstack-infra17:42
boris-42clarkb thanks17:42
boris-42clarkb I hope this will run finally properly our jobs in rally17:42
clarkbmordred: why are there two pbr revert git mailmap changes?17:42
mordredclarkb: we decided to stay with infra@lists ?17:42
mordredjeblair: did you see my terrible idea from last night: http://paste.openstack.org/show/76197/ ?17:43
clarkbmordred: I think I am missing context here that I didn't pick up on in sb17:43
mordredclarkb: there was a conversation yesterday or the day before about whether openstack-infra@lists.openstack.org was the right email address - I didn't follow the conclusion - was just asking about it17:43
clarkbmordred: ah, I left the address in the external account ids table but removed the preferred email address to disable email17:44
clarkbmordred: that way if people have questions about things they can send them to the list which I think is a good thing17:44
clarkbbut we won't get bot spammed17:44
openstackgerritA change was merged to openstack-infra/config: Add rally-jobs job group  https://review.openstack.org/8827217:45
mordredkk17:45
clarkbpleia2: assuming that the jobs last night didn't fill up the infra@lists queue again17:45
jeblairmordred: heh, cool.  :)  also, i want to look at clarkb's makefile replacement for tox sometime.  :)17:45
clarkbpleia2: any chance you can double check the queue?17:45
pleia2clarkb: thumbs up17:45
clarkbwoot17:45
pleia2yeah, I checked this morning, all is well :)17:45
mordredjeblair: I looked at that with him - we ran in to a hitch with it17:46
mordredwhich is argument passing17:46
clarkbmordred: no arg passing works fine17:46
clarkbyou just do make py27 ARGS='testname'17:46
mordredyou can't do "make py27 only_run_this_test"17:46
mordredright. you have to do ARGS=17:46
jeblairgotta run for a bit17:47
SpamapSWould appreciate this getting approved ASAP: https://review.openstack.org/#/c/88637/ as I suspect it will require a restart or something to actually enact the change and unbreak tripleo-image-elements' gate.17:47
clarkbmordred: did your fixes for the ssh stuff merge?17:48
clarkbmordred: so next change is driving puppet with it change?17:48
mordredclarkb: yup17:48
clarkbSpamapS: why not just fix the tests?17:48
clarkbThe gate for tripleo-image-elements is broken due to this commit. isn't quite true17:49
clarkbthe gate is broken because the tests are broken17:49
openstackgerritA change was merged to openstack-infra/config: Set config prior to git review -s.  https://review.openstack.org/8846117:49
*** EmilienM has quit IRC17:50
*** EmilienM has joined #openstack-infra17:50
*** Hefeweizen has joined #openstack-infra17:50
mordredclarkb: https://review.openstack.org/#/c/80976/ is the next up - then https://review.openstack.org/#/c/8097617:50
clarkbI love that while fungi is on vacation we are just going to completely change how we run puppet17:52
clarkb:)17:52
anteayahe'll love that too17:52
mordredclarkb: :)17:52
mordredclarkb: just wait until I rewrite run_all.sh to be a gearman client17:52
anteayaso what is going to happen with new project friday today?17:52
mordredanteaya: we're going to land the fix to the thing and try it out17:53
anteayais the mordred sshing in place?17:53
mordredyup17:53
anteayawoohoo17:53
mordredbut we have two more patches to land to finish hooking it up17:53
clarkbmordred: ok +2'd the two changes. But haven't approved. I will let you babysit that?17:53
anteaya80976 and ?17:53
mordredclarkb: yup. I'll do them in just a sec17:53
clarkb8734317:53
anteayak17:53
mordredwhen we land this - it means we'll be ceasing to run puppet agent completely17:54
mordredwell - the daemon17:54
mordredI'm excited about that17:54
SpamapSclarkb: We've _never_ run those tests.17:54
SpamapSclarkb: docs, and python26, are both broken17:54
SpamapSclarkb: so we'll have to ask for a manual merge of at least one of those anyway.17:55
clarkbSpamapS: no manual merge should be necessary17:55
SpamapSclarkb: and this is blocking everyone while we do that.17:55
clarkbjust push a change that fixes it17:55
SpamapSfixes both17:55
clarkbyes17:55
SpamapSneither are small changes17:55
mordredclarkb: I think he's saying "landing the change to enforce two gates that they weren't prepared to meet as a mistake"17:55
mordredwas17:55
clarkbmordred: sure17:55
SpamapSI for one value a commit history that consists of single logical changes.17:55
clarkbbut the fix isn't on infras plate imo17:55
clarkbif a projects reqwuests tests then removes them because they don't pass...17:56
SpamapSbnemec should not have requested them17:56
SpamapSI don't know why he did. I have not received word back. I don't think he thought it through.17:56
SpamapSProbably just ran py27 and figured that was enough.17:56
mriedemflashgordon: yeah https://review.openstack.org/#/c/88487/17:57
SpamapSjust now have a lucid box to test py26 on17:57
*** lukego has joined #openstack-infra17:57
anteaya80976 used to have run_all.sh and ensure_puppet.sh which are no longer part of the patch, where did they go?17:57
anteayaare they merged already?17:57
mordredanteaya: I split them out into a patch that's already merged17:57
SpamapSclarkb: btw, I did try to fix the tests.. :-/17:58
flashgordonmriedem: so I guess the good news is we know why things started failinga nd have fixes for many of the issues17:58
clarkbSpamapS: does py27 pass? should we maybe explicitly add that test to keep it from regressing?17:58
*** wchrisj_ has joined #openstack-infra17:58
mriedemflashgordon: yeah, i sent out a note to the dev list before leaving for lunch17:58
SpamapSclarkb: yes. I think we can risk it for a few more days seeing as it has never been gated before.17:59
clarkbflashgordon: so about that (I bet ttx is afk at this point deep in a bottle of fine wine) but I think we should consider dropping milestone proposed branches completely17:59
*** aconrad has quit IRC17:59
mriedemflashgordon: http://lists.openstack.org/pipermail/openstack-dev/2014-April/033182.html17:59
clarkbflashgordon: then we can do a much more coordinated switch to release branches prior to release to ensure that all of the test keep working even after that branching17:59
SpamapSclarkb: simplest thing for everyone: revert the unit test running.. we'll fix.. then un-revert.17:59
clarkbas it is milestone proposed is a temporary hackish way around doing that17:59
flashgordonmriedem: yeah I saw it17:59
anteayazaro: if you feel like chatting about your -1 here, let me know: https://review.openstack.org/#/c/77376/117:59
mordredanteaya: https://review.openstack.org/#/q/status:merged+owner:2+project:openstack-infra/config,n,z the top 5 - except not the Use virtualenv_version patch18:00
mordredanteaya: if you want to look18:00
flashgordonclarkb: areed on both counts18:00
flashgordonagreed on *18:00
anteayamordred: thanks yes was looking for it18:00
clarkbsomething like, update grenade with known gotchas, update d-g with fallback to previous stable branch if current stable branch does not exist yet, create stable branches, fix issues that show up18:00
mordredok guys. I just pulled the trigger18:00
clarkbthen we can sort all of this out well before release18:00
clarkbthen 6 weeks later we tag commits on stable/icehouse and everything has been working for 6 weeks18:01
openstackgerritMarc Abramowitz proposed a change to openstack-infra/jenkins-job-builder: Add `browse` command  https://review.openstack.org/8558018:01
mordredwhen it lands, I'm going to trigger a puppet run everywhere18:01
*** sabari has joined #openstack-infra18:01
flashgordonclarkb: are you documenting all the issues we are hitting? so we can avoid them next time?18:01
mordredand then verify that puppet is no longer running on any of the nodes18:01
clarkbflashgordon: no, there is only one issue18:01
clarkbflashgordon: we have no coordinated way of making the switch to a stable branch ebtween infra, release, and qa18:02
flashgordonright18:02
clarkband a big part of that is it happens all at once rather than as part of the gradual process that milestone proposed does today18:02
*** wchrisj has quit IRC18:02
flashgordonI just counted that as sub items18:02
flashgordonanyway its agood thing reviews die down around this time anyway18:03
openstackgerritA change was merged to openstack-infra/config: Revert "Run unit tests in tripleo-image-elements"  https://review.openstack.org/8863718:03
*** UtahDave has joined #openstack-infra18:03
clarkbflashgordon: my biggest concern is I think this makes it possible to release broken code18:04
clarkbI don't necessarily think that happened here (I could be wrong, not super familiar with nova rpc) but it points out a gap in testing18:04
anteayamordred where did the run_ssh.sh script end up? I am not seeing it in here: http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files18:04
mordredanteaya: it's templated18:05
mordredhttp://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/templates/run_remote_puppet.sh.erb18:05
clarkbmordred: so 2 pbr reverts. what is the difference there?18:05
mordredclarkb: the first was the auto-generated one from gerrit18:05
clarkbcan it be abandoned?18:05
mordredyes18:05
mordredI thought I had - sorry18:06
flashgordonclarkb: how do you think we could have shiped broken code?18:06
clarkbflashgordon: because icehouse was on milestone proposed for the last ~6 weeks. Which meant all of the upgrade testing effectively ignored it18:06
flashgordonclarkb: oh crap18:07
clarkbsimilar to how swift milestone proposed derped tempest18:07
flashgordonclarkb: well I guess we will learn soon18:07
clarkbmilestone proposed is a corner it isn't well dealt with by the CI stuff18:07
clarkbflashgordon: well havana -> icehouse should've been tested18:08
*** dcramer_ has joined #openstack-infra18:08
clarkbflashgordon: but icehouse -> master would not be18:08
anteayamordred: I was wrong, run_ssh isn't a script it is a function within run_remote_puppet.sh.erb, that was where I was confused18:08
clarkbso for CD deployments it could mean hurt18:08
notmynameshouldn't m-p branches be deleted once they are merged back to master?18:08
clarkbnotmyname: yes but that is a 6 week period after feature freeze18:08
notmyname(also, have those jobs been retriggered yet?)18:08
clarkbnotmyname: yes that is how we found the fail18:08
clarkboh those other jobs18:09
flashgordonclarkb: during milestone proposed we tseted havana -< icehouse (Which was master at the time) but not milestone right?18:09
clarkbI don't know about those. mordred was going to do that iirc18:09
clarkbflashgordon: any changes to m-p should be tested as havana -> m-p18:09
SpamapSthanks for the revert18:09
SpamapSnow I just realized we could just make py26 a noop couldn't we?18:09
clarkbflashgordon: but any master changes would have tested havana -> master not m-p to master18:09
mordrednotmyname: retriggering is on my plate for today18:10
clarkbflashgordon: so there is a gap there for CD folks18:10
flashgordonclarkb: yeah. but we could have derped a backport to break milestone proposed too I think18:10
* mordred thinks instead of m-p we shoudl just cut the stable/* branch early18:10
clarkbmordred: yes18:10
notmynamemordred: ok, thanks. in the meanwhile, all patches that land will have the wrong version number calculated by pbr18:10
mordredbut not tag a release on it until it's time to tag a release18:10
mordrednotmyname: yah. sorry about the dep there18:11
*** mrodden has joined #openstack-infra18:11
mordredderp18:11
clarkbmordred: and before we do that, we make sure d-g and grenade are ready for it and test stable/* from commit 118:11
mordredman18:11
mordredI derped typing derp18:11
mordredclarkb: ++18:11
notmyname:-)18:11
clarkbmordred: completely agree. it simplifies testing so much more18:11
notmynamemordred: just a side-effect of the way openstack does versioning. it has upsides, but period after every m-p tag is the downside18:11
mordrednotmyname: yeah - I want to sort out making that period and transition not suck18:12
clarkbmordred: the branching model above cuold help fix it18:12
clarkbbranch stable/* and make a tag that convinces master to do the right thing18:12
clarkband do that at the beginning of the process not end18:12
mordredI do not think it will help18:12
*** thuc has joined #openstack-infra18:13
SpamapSclarkb: heh.. so.. I may have pushed a bit aggressively there. I think we can just noop the py26 tests and then undo that revert. Sorry for the churn. :-/18:13
mordredI thnk it will just mush the fail aroudn like a plate of unwanted mashed potatoes18:13
clarkbSpamapS: meh18:14
clarkbSpamapS: when the tests pass, push an unrevert18:14
SpamapSclarkb: but we can at least land those things as individual commits and _then_ turn on the tests. ;)18:14
clarkbmordred: nice18:14
*** Ajaeger has quit IRC18:14
*** Sukhdev has quit IRC18:15
SpamapSclarkb: do I need to wait for a zuul/jenkins restart before I start to recheck/reverify things?18:16
clarkbSpamapS: you need to wait for puppet to apply the change to zuul, once that happens zuul is ready (it reloads config on the fly). At the bottom of the zuul status page should be a timestamp for when the last reload happened18:16
anteayazaro: thanks18:16
clarkbSpamapS: looks like it just happened so you should be good18:17
anteayabefore this gets buried again, can I request a +A? https://review.openstack.org/#/c/77376/18:17
anteayaa lineup of +1s and +2s18:17
*** thuc has quit IRC18:17
clarkbmordred: https://review.openstack.org/#/c/88627/ reviewed18:19
clarkbanteaya: I think we are all for it, but trying to avoid doing policy decisions in the config repo18:19
clarkbanteaya: this is one of the reason for my proposed summit session. We should be able to talk about items like that and more18:20
openstackgerritMarc Abramowitz proposed a change to openstack-infra/jenkins-job-builder: Refactor delete_jobs  https://review.openstack.org/8864618:20
*** denis_makogon_ has joined #openstack-infra18:21
anteayaclarkb: okay I'll add the link to the review in teh comments section of your summit proposal so it doesn't get lost18:21
anteayais that okay?18:21
clarkbsure18:21
anteayak18:21
mordredbtw - I've found a new and glorious way in which my laptop can go mental on me18:22
clarkbstory time18:22
mordredlast night - google stopped working. just google. I can get anywhere else, but not google websites18:23
mordrednothing shows in logs, nothing seems unhappy18:23
mordredbasically, google won't resolve18:23
mordredrestart networking? no fixy18:23
openstackgerritA change was merged to openstack-infra/config: Drive puppet from the master over ssh  https://review.openstack.org/8097618:23
mordredreboot computer? yup. google is back18:23
mordredW T F18:23
clarkbthis reminds me I should update from saucy to trusty soon so that I can have working vim again18:24
mordredtrusty mostly works18:24
clarkbI updated my htpc last night. There was at least one segfault during the upgrade18:24
mordredrunning puppet everywhere to roll out new change18:25
nibalizerclarkb: i like that the only thing you copied to keep safe was /etc/pulse18:25
nibalizerthat is exactly how I feel18:25
*** dstanek has joined #openstack-infra18:25
anteayamordred: so did you find out what the problem was?18:26
anteayaor have we reached the end of this tale?18:26
mordredthat was the end18:26
mordredit might have been absurdist comedy18:26
openstackgerritMarc Abramowitz proposed a change to openstack-infra/jenkins-job-builder: Refactor delete_jobs  https://review.openstack.org/8864618:26
clarkbnibalizer: all it does is play videos from the internet. Sound is pretty important18:27
clarkbit was my hbo go box for a long time. Then I ran out of hbo go18:27
clarkbthere was a time when I had diablo running on it18:30
clarkbmanually installed wtihout devstack18:30
clarkbdo not recommend18:30
*** andreykurilin_ has joined #openstack-infra18:30
anteayathe hat passing routine18:30
nibalizeralso i hate fighting pulse18:30
anteayafrom godot18:31
clarkbI have to say openstack has come a long way since diablo18:31
nibalizerclarkb: i parsed that as blizzards diablo at first, was confused18:31
clarkbadam_g: how are those firewall rules working?18:34
*** marun has joined #openstack-infra18:34
adam_gclarkb, they are working perfect. thanks.  i have a patch up to make devstack do that itself, so we can drop them from the puppet config later if you'd prefer it18:35
*** sweston has joined #openstack-infra18:35
openstackgerritA change was merged to openstack-dev/hacking: Import Guideline Exception for gettextutils  https://review.openstack.org/8863218:35
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Stop trying to start gerrit  https://review.openstack.org/8865018:36
clarkbadam_g: sounds good. It may actually be a good idea to attempt to make devstack do a completely locked down iptables and only punch necessary holes documenting the purpose of each hole18:37
*** lukego has quit IRC18:37
clarkbadam_g: might make lives easier for people deploying in datacenters18:37
notmynameis the world falling? https://review.openstack.org/#/c/88359/18:37
clarkbbut probably also a lot of work18:37
*** rcarrill` has quit IRC18:37
clarkbnotmyname: ERROR: unknown branch name feature/ec18:37
notmynameclarkb: this hasn't happened before18:38
clarkboh, this is probably a consequence of sdague's config generator?18:38
clarkbsdague: ^18:38
mordredhe's afk18:38
clarkbnotmyname: there is a new thing to formalize what branches support which features and so on18:38
adam_gclarkb, yeah, i was thinking the same thing yesterday.  a devstack-INPUT /w similar defaults to openstack-INPUT.  that would have saved me a bit of time trying pinpoint why i wasns't able reproduce issues i was seeing in the gate18:38
clarkbnotmyname: I'm guessing it didn't accomodate random branches...18:38
* clarkb reads some code18:38
mordredclarkb: or feature branches - yeah - I bet that' it18:38
clarkbyurp that was an oversight18:39
clarkbhttps://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/test-matrix.py#n2418:40
clarkbso how do we want to handle this. treat the feature/* prefix as equivalent to master?18:41
mordredyah. I think that makes the most sense18:41
* clarkb writes a thing18:41
mordredby definition feature branches don't chase stable18:41
clarkbany other prefixes we should care about or can I hard code feature like stable?18:43
*** ociuhandu has quit IRC18:43
mordredI think you can hardcode feature like stable18:43
pleia2running out to lunch18:43
openstackgerritClark Boylan proposed a change to openstack-infra/devstack-gate: Handle feature branches in test-matrix.py  https://review.openstack.org/8865218:45
clarkbhow does that look?18:45
mordredclarkb: +218:46
*** hashar has joined #openstack-infra18:50
zaromordred: how else would gerrit start?18:51
*** denis_makogon_ has quit IRC18:52
mordredzaro: by hand - we tend to want to restart gerrit when we choose18:56
notmynameclarkb: can you retrigger the patch (or let me know) when zuul is ready?18:56
clarkbnotmyname: sure18:58
clarkbthat apparently broke something though :(18:58
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Properly don't fail on success  https://review.openstack.org/8865618:58
clarkbgah I can't python18:58
*** dims is now known as dims-openstack18:58
openstackgerritClark Boylan proposed a change to openstack-infra/devstack-gate: Handle feature branches in test-matrix.py  https://review.openstack.org/8865218:58
mordredclarkb: a fix I'd applied locally yesterday got missed in being submited ^^18:58
clarkbmordred: k18:58
clarkbI'm going to make lunch while tests for 88652 are in flight18:59
*** atiwari has quit IRC18:59
mordredclarkb: ci-backup-rs-ord.openstack.org18:59
mordredclarkb: things can't ssh in to that- do we expect that?19:00
krotscheckSorry, my log got screwed up. Has someone bounced out to storyboard.o.o yet and taken a look at why our API went belly up?19:00
mordredkrotscheck: looking now19:01
mordred[Fri Apr 18 19:00:30 2014] [error] [client 74.202.214.170] ConfigFileParseError: Failed to parse /etc/storyboard/storyboard.conf: at /etc/storyboard/storyboard.conf:3, Section must be started before assignment: None19:01
krotscheckmordred: What the… did something in the puppet run after the token expiration screw up the config file?19:02
*** dcramer_ has quit IRC19:03
mordrednope. the puppet has correctly installed a very evil config file19:03
mordredkrotscheck: check out ./modules/storyboard/templates/storyboard.conf.erb19:03
mordredyou'll find it less exciting than you'd want19:03
krotscheckmordred: Looking now. I’m expecting demons and succubi19:04
*** jamespage_ has joined #openstack-infra19:04
mordredkrotscheck: dude. if you find succubi, let me know...19:05
*** dcramer_ has joined #openstack-infra19:05
clarkbmordred: no we don't expect that19:05
clarkbmordred: we should figure that out19:06
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Put a default section header above the TTL  https://review.openstack.org/8866019:06
mordredclarkb: my personal user accoutn also does not log in there19:06
mordredclarkb: so I'm thinking that puppet has not run there ina  very long time19:06
mordredkrotscheck: ^^19:06
clarkboh ya its manual puppet19:06
clarkbI will give it a shot after this sandwich19:06
mordredclarkb: why is it manual puppet? should we exclude it from the ssh puppet?19:07
anteayaso who has any thoughts on how to improve the search feature in the wiki19:07
mordredkrotscheck: (the puppet patch, not the other thing)19:07
*** ihrachyshka has joined #openstack-infra19:07
anteayaso that like, it actually returns results that are useful19:07
clarkbmordred: because it is the backup host19:07
clarkbmordred: idea is it won't automatically foobar on a puppet run19:07
clarkbdid puppet run against it? we may not have re certed it yet19:08
mordredanteaya: https://www.google.com/#q=useful+things+site:wiki.openstack.org works well?19:08
krotscheckmordred: Noticed, thanks.19:08
mordredclarkb: it gets picked up in the puppet cert list output19:08
clarkbmordred: huh19:08
mordredclarkb: so, we should run puppet on it by hand, but then also run "puppet agent --disable"19:08
clarkbya19:09
mordredclarkb: OR - we should make an "excluded_hosts" array to block running on hosts that puppet knows about but which we know we don't want to run thigns on19:09
mordredor both19:09
clarkbI like both19:10
*** jamespage_ has quit IRC19:10
anteayamordred: yes it does19:10
*** jamespage_ has joined #openstack-infra19:10
clarkbmordred: when you say things can't ssh to it, what in particular can't?19:11
clarkbmordred: eg are backups failing or is it just your ssh script thing?19:11
mordredthe ssh script thing - and my personal user accoutn19:11
openstackgerritA change was merged to openstack-infra/config: Properly don't fail on success  https://review.openstack.org/8865619:13
openstackgerritA change was merged to openstack-infra/config: Put a default section header above the TTL  https://review.openstack.org/8866019:14
clarkbI just realized I should've written a test for the feature test matrix thing... I have written on locally which I will push as a second patch so that we can get the fix through19:17
clarkbnot ready for all this bash stuff to actually have tests19:18
mordred++19:18
*** jamespage_ has quit IRC19:19
*** dims-openstack is now known as dims19:24
*** shakamunyi has quit IRC19:27
*** shakayumi has joined #openstack-infra19:27
*** BadCub01 has quit IRC19:32
*** zns has quit IRC19:32
jgriffithclarkb: Questions on creating cinder-specs19:32
jgriffithclarkb: clarkb cookie-cutter seems a bit overkill19:33
*** mark0n has joined #openstack-infra19:33
jgriffithclarkb: regardless, still need to figure out "creating" the repo on github/openstack to push to19:33
jgriffithjeblair: ^^  Not sure who I should be harassing on this :)19:34
anteayajgriffith: so far I get you want to create cinder-specs repo19:36
anteayaand the cookie-cutter doesn't answer all your questions19:36
dimsjgriffith, there's a specs cookit cutter too - http://git.openstack.org/cgit/openstack-dev/specs-cookiecutter/19:36
jgriffithanteaya: well... yeah, sort of19:36
jgriffithanteaya: so cookie cutter seems overkill19:36
jgriffithanteaya: ie tests etc etc19:36
jgriffithanteaya: I can create the repo that way no problem19:37
dimsjgriffith, see the specs cookie-cutter ^^ it's stripped down19:37
jgriffithanteaya: but creating a remote to push it too is the question :)19:37
jgriffithdims: k19:37
jgriffithdims: OHHH19:37
jgriffithdims: awesome19:37
jgriffithdims: thanks19:37
dhellmannjgriffith: I usually create the remote on my own github account and then import it from there19:37
dhellmannthen I just delete that repo when I'm done19:38
jgriffithdhellmann: makes sense19:38
dimsjgriffith, not my work, better minds than me - https://www.mail-archive.com/openstack-dev@lists.openstack.org/msg20245.html19:38
*** lukego has joined #openstack-infra19:38
dhellmannthe "openstack" project on github is essentially read-only19:38
*** marun_ has joined #openstack-infra19:38
jgriffithdhellmann: that's the part I ws trying to figure out... assumign I could just bug folks here again after I have it in my github :)19:39
*** marun has quit IRC19:39
*** marun_ is now known as marun19:39
dhellmannjgriffith: this might help: https://wiki.openstack.org/wiki/Oslo/CreatingANewLibrary#Importing_the_Repository_into_the_CI_System19:39
anteayajgriffith: this might also help: https://review.openstack.org/#/c/79363/19:39
anteayaand you don't _have_ to create a repo anywhere19:39
anteayaif the sole purpose is to create a new openstack repo19:40
jgriffithYay... all kinds of useful info19:40
jgriffiththanks dims dhellmann and anteaya19:40
anteayawe can start with nothing19:40
anteayanp19:40
anteayaif you use the topic:new-project for your patch, that helps us19:40
*** zns has joined #openstack-infra19:40
anteayathough since new project creation is going back to automated it isn't strictly necessary19:40
anteayathough I do like reviewing all the new-project/new-repo patches at once19:41
anteayahelps me spot missing items19:41
*** amcrn has quit IRC19:44
*** lukego has quit IRC19:45
*** dstanek has quit IRC19:46
*** jamespage_ has joined #openstack-infra19:46
*** jamespage_ has quit IRC19:48
*** jp_at_hp has quit IRC19:50
clarkbI just self approved the test matrix fix19:55
clarkbcheck testing looked good19:55
*** e0ne has joined #openstack-infra19:56
mordredclarkb: ++19:57
mordredclarkb: I'm going to land the revert revert revert patch19:58
clarkbok19:58
*** amcrn has joined #openstack-infra20:02
openstackgerritClark Boylan proposed a change to openstack-infra/devstack-gate: Test matrix generation for feature branches.  https://review.openstack.org/8867120:02
clarkbmordred: for your pbr revert I think you need to revert 9375a768100f16b5493062449d1274899b900cfd at the same time20:04
clarkbmordred: to get the parse mailmap function back20:04
openstackgerritA change was merged to openstack-infra/config: Revert "Revert "Revert "Stop running manage-projects"""  https://review.openstack.org/8734320:04
*** thuc has joined #openstack-infra20:09
*** thuc_ has joined #openstack-infra20:13
*** thuc has quit IRC20:16
*** e0ne has quit IRC20:16
openstackgerritMichael Krotscheck proposed a change to openstack-infra/storyboard: Expose assignee_id to API  https://review.openstack.org/8867620:19
*** chandan_kumar has quit IRC20:21
openstackgerritMarc Abramowitz proposed a change to openstack-infra/jenkins-job-builder: Refactor the `matches` function  https://review.openstack.org/8867820:25
*** marun has quit IRC20:26
openstackgerritMichael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Fixed text display on Task Status dropdown.  https://review.openstack.org/8867920:26
*** e0ne has joined #openstack-infra20:27
*** sweston has quit IRC20:28
*** sweston has joined #openstack-infra20:29
*** e0ne has quit IRC20:30
openstackgerritRandy Coulman proposed a change to openstack-infra/jenkins-job-builder: Support the Store SCM plugin  https://review.openstack.org/8572920:30
openstackgerritMonty Taylor proposed a change to openstack-infra/config: WIP: Use ansible instead of direct ssh calls  https://review.openstack.org/8782120:32
mordredjeblair, clarkb, anteaya: ^^ I forgot to publish the POC script with teh last version of the patch20:33
mordredagain - as I mention in the commit - I don't think that will actually work as is right now - merely a point of comparison20:33
*** imcsk8_lap has quit IRC20:34
*** sweston has quit IRC20:34
*** imcsk8 has quit IRC20:34
*** marun has joined #openstack-infra20:35
*** imcsk8 has joined #openstack-infra20:35
openstackgerritSergey Skripnick proposed a change to openstack-infra/config: Fix envvars in devstack-rally-gate builder  https://review.openstack.org/8868220:36
openstackgerritSergey Skripnick proposed a change to openstack-infra/devstack-gate: Add RUN_EXERCISES to generated localrc  https://review.openstack.org/8868320:36
*** miqui has quit IRC20:38
openstackgerritMonty Taylor proposed a change to openstack-infra/config: WIP: Use ansible instead of direct ssh calls  https://review.openstack.org/8782120:38
rediskincould someone tell why envvars are cleaned here? https://review.openstack.org/8868220:38
rediskinSergeyLukjanov sdague fungi ^^20:38
*** imcsk8 has quit IRC20:39
rediskinin this log you can see how RUN_EXERCISES is exported, but later it dissapeared http://logs.openstack.org/45/86945/3/check/check-rally-dsvm-rally/08f8be0/console.html20:39
*** david_lyle_ has joined #openstack-infra20:40
rediskinhere it exported http://logs.openstack.org/45/86945/3/check/check-rally-dsvm-rally/08f8be0/console.html#_2014-04-18_19_32_47_30020:40
openstackgerritMarc Abramowitz proposed a change to openstack-infra/jenkins-job-builder: Add `browse` command  https://review.openstack.org/8558020:40
rediskinand here is gone http://logs.openstack.org/45/86945/3/check/check-rally-dsvm-rally/08f8be0/console.html#_2014-04-18_19_46_01_33020:40
*** imcsk8 has joined #openstack-infra20:40
* anteaya clicks to read mordred's latest ramblings20:41
*** nati_ueno has joined #openstack-infra20:41
*** dklyle has joined #openstack-infra20:41
clarkbrediskin: they shouldn't be20:42
*** david-lyle has quit IRC20:43
rediskinclarkb: they shouldn't be dissapeared? xD20:44
clarkbcorrect20:44
clarkbis devstack doing something silly?20:44
boris-42clarkb "is devstack doing something silly?" lol=)20:45
openstackgerritMarc Abramowitz proposed a change to openstack-infra/jenkins-job-builder: Add `browse` command  https://review.openstack.org/8558020:45
rediskinclarkb: i thought it was devstack-gate, but i dont found anything20:45
clarkbrediskin: I don't think devstack gate does that20:45
*** david_lyle_ has quit IRC20:45
clarkbas this methods works for lots of other projects20:45
clarkband there are no cases of unset in devstack gate20:45
*** imcsk8 has quit IRC20:45
rediskinclarkb: but as you can see at that log, it is all gone20:45
clarkbrediskin: yes, I agree it is gone20:46
asselinHi, I have check-tempest-dsvm-full basically working in my nodepool infrastructure. I'm looking to add support for cinder certification tests on real HW. There's quite a few files that need to be updated across multiple projects. Since the scripts pull many of these files from git, is there any recommended approach to develop and test local changes?20:46
rediskinpoltergeist20:46
clarkbI suppose it may start a subshell without inheriting the env somewhere20:46
clarkbasselin: yeah, we use the zuul provided ref. devstack gate will check that out for each project20:47
clarkbasselin: so you can create a ref with the same name in each project with the changes you need then feed that as teh zuul ref to devstack gate20:47
clarkbrediskin: I don't think that change will fix anything though20:48
clarkbexporting in the pre test hook vs exporting before d-g starts shouldn't make a difference20:48
clarkbrediskin: has that actually been tested?20:48
rediskinclarkb: it will. it was working in old version of this job20:48
*** hashar is now known as FoobAR30020:48
*** thuc_ has quit IRC20:49
rediskinclarkb: looks like evvars dropping happened before pre_test_hook is called20:49
asselinclarkb, so I make the change locally, and do the git review for each project to push into openstack gerrit? does that create the zuul ref? how to make it the same ref across projects?20:49
*** thuc has joined #openstack-infra20:50
*** harlowja is now known as harlowja_away20:50
clarkbasselin: well zuul creates it for us based on the order or approved jobs20:50
clarkbyou are a bit on your own if trying to do it without zuul20:51
asselinclarkb, so do I use a local version of zuul? or openstack zuul? http://status.openstack.org/zuul/20:52
clarkbasselin: you would need a local instance of zuul20:52
*** e0ne has joined #openstack-infra20:52
*** FoobAR300 is now known as hashar20:53
*** nati_ueno has quit IRC20:53
*** imcsk8_lap has joined #openstack-infra20:53
clarkbrediskin: so if we read d-g, ENABLED_SERVICES isn't used until after the pre_test hook20:53
clarkbI think20:53
*** jamespage_ has joined #openstack-infra20:53
*** hashar is now known as insecte20:54
clarkbrediskin: yup, ENABLED_SERVICES is used in gate_hook which happens after pre_test_hook20:54
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Add some system logging to run_all  https://review.openstack.org/8868520:54
clarkbso the entire env isn't being dropped20:54
mordredclarkb: ^^20:54
clarkbotehrwise enabled services wouldn't work20:54
clarkbbut we have tempest telling us it does work20:54
clarkbrediskin: so I don't think that is the issue. Would be good to understand why this happens20:54
*** thuc has quit IRC20:55
*** insecte is now known as wmf-insecte20:55
*** wmf-insecte is now known as hashar20:56
clarkbrediskin: and gate_hook is what runs the exercises20:56
asselinclarkb, suppose I wanted to make the changes using openstack's infrastructure, what would be the best approach? i.e. add a new environment to tempext tox, and entry in devstack-vm-gate, etc...20:57
*** dklyle has quit IRC20:57
clarkbasselin: I'm not sure there is a way to do that if you need specialized hardware20:58
*** imcsk8_lap has quit IRC20:58
clarkbasselin: we do everything with open source tools on donated cloud resources20:58
*** wchrisj_ has quit IRC20:58
clarkbasselin: anything we can test in that environment we can definitely do what you describe20:58
asselinclarkb, no, I would run the LVM version. then when that works, set it up for read hardware20:58
*** jamespage_ has quit IRC20:59
clarkbasselin: right so anything you can do on our donated cloud resources with open tools you can just edit the existing JJB jobs21:01
clarkbasselin: in some cases that might mean extending existing tests or adding new ones21:01
rediskinclarkb: https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate.sh#L24521:03
rediskinsudo -H drops envvars21:03
*** david-lyle has joined #openstack-infra21:04
rediskinclarkb: so this probably should partially fix my problem https://review.openstack.org/#/c/88683/21:04
clarkbcool so your fix shouldn't fix it21:04
clarkbalso I don't see RALLY_SCENARIOS used by anything in the rally devstack contrib stuff21:04
clarkbis that for future thigns?21:04
asselinclarkb, this is the part I'm getting stuck on conceptually. I need a new jenkins job (config project) that needs updates to tempest project (new test) and devstack-gate project (invoke new test)....21:05
rediskinclarkb: yes, this is for future things21:05
asselinclarkb, so that's 3 separate commits for everything to work21:05
rediskinclarkb: i thought all envvars are available, so now i dont know what to do. i gonna think about it. thanks for help21:06
clarkbrediskin: you can provide your own gate_hook21:06
clarkbrediskin: and when you do that ensure that the variables you need are passed throguh21:07
clarkbthat might require some wheel reinvention though21:07
rediskinclarkb: yea. thanks21:07
clarkbasselin: we try to do these things in a self tested manner. Usually that means get a job in place that is in the experimental pipeline for the projects that you need to touch21:08
openstackgerritMonty Taylor proposed a change to openstack-infra/config: php5-json is provided by php5-common  https://review.openstack.org/8869221:08
*** dcramer_ has quit IRC21:08
clarkbasselin: then you can push your changes to those projects and comment with `check experimental` to run that test. Eventually you should be able to get it to pass at which point it can be made a normal test21:08
*** mfer has quit IRC21:09
asselinclarkb, ok I'll take a look at that thanks21:09
clarkbrediskin: my initial thought is we should move setup_localrc to the functions file so that it can be sourced and called without all that extra stuff21:09
clarkbrediskin: then your custom gate_hook can set up env vars, call that function, then run devstack exercises directly21:10
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Add some system logging to run_all  https://review.openstack.org/8868521:11
openstackgerritMonty Taylor proposed a change to openstack-infra/config: php5-json is provided by php5-common  https://review.openstack.org/8869221:12
mordredclarkb: would you take an eyeball at https://review.openstack.org/88685 ? I'd like to land that so that it's easier to watch the project stuff happen when we land a new project change21:12
mordredand then I'd like to try landing new project changes21:12
clarkbmordred: can you see my comment on patchset 121:13
mordredoh - looking21:13
clarkbmordred: it covers a thing you fixed in ps2 but also one other thing21:13
clarkbI just managed to apply the comment, sorry for the delay21:13
mordredno worries - race condition in commenting :)21:14
clarkbmordred: I think the etherpad logrotate does the thing I describe21:15
clarkbpossibly also the logstash worker log rotates21:16
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Add some system logging to run_all  https://review.openstack.org/8868521:16
mordredclarkb: there yah go21:16
mordredcopytruncate seems to be the truck21:16
mordredtrick21:16
clarkbyup21:16
anteayaI stuck my +1 on it too21:18
anteayathough I only understood about 1/2 of it21:19
*** dprince has quit IRC21:19
openstackgerritMarc Abramowitz proposed a change to openstack-infra/jenkins-job-builder: Add `browse` command  https://review.openstack.org/8558021:20
*** imcsk8 has joined #openstack-infra21:20
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Stop trying to start gerrit  https://review.openstack.org/8865021:21
mordredclarkb: any opposition from me going ahead and landing that - I'd like to see some logs roll by21:21
clarkbmordred: not from me21:21
rediskinclarkb: what about just ask sudo to preserve envvars?21:22
*** _nadya_ has joined #openstack-infra21:22
clarkbrediskin: I don't think we want to do that in the general case21:22
clarkbrediskin: the idea is that devstack gate will configure devstack via localrc21:22
clarkbleakage around that could lead to subtle brokeness in devsatck21:22
*** prad_ has quit IRC21:22
rediskinsad21:22
clarkbmordred: ^ thoughts?21:22
mordredyah. I agree. can the hook write the values into a file?21:23
openstackgerritA change was merged to openstack-infra/config: Add some system logging to run_all  https://review.openstack.org/8868521:24
mordredhehe21:25
mordredroot      5880  5879  0 21:19 ?        00:00:00 bash -c brazenly_run_ssh cacti.openstack.org21:25
mordredthat's just fun to see in the ps output21:26
*** _nadya_ has quit IRC21:26
clarkbbrazenly_run_ssh is a thing I missed21:26
*** imcsk8 has quit IRC21:26
clarkbnow I wish I had managed to review those fixes21:26
clarkbbecause that sounds fun21:26
mordredclarkb: it's "run ssh commands and categorically ignore antyhing that goes wrong"21:27
anteayait explictly returns 021:27
clarkbmordred: is that good for the make sure thinsg run in order and don't fail stuff?21:27
mordredclarkb: there are two functions21:27
clarkbah21:27
mordredclarkb: one of them cares very much about return codes21:27
mordredone does not21:28
clarkbbtw did manage projects do its thing?21:28
mordredhaven't asked it to yet21:28
mordredwell - although, yeah - it should have run recently on review21:28
anteayadoes manage projects have anything to do?21:28
*** jamespage_ has joined #openstack-infra21:28
clarkbI think there was at least one ACL update pending21:28
anteayaok21:28
mordredoh - neat - it should have run by now21:28
clarkbwe just merge those because ordering doesn't matter on an ACL update21:28
anteayathese 3 are still sitting here: https://review.openstack.org/#/q/status:open+project:openstack-infra/config+branch:master+topic:new-project,n,z21:29
mordredclarkb: we could also look and see if gerrit has new refs21:29
anteaya2 are good to go, solum-dashboard needs an edit21:29
anteayaclarkb: right21:29
*** imcsk8 has joined #openstack-infra21:30
mordredoh - manage_projects will not have run21:31
mordredbecause puppet will have applied the changes to the acls files ages ago21:31
mordredso there will be nothign to trigger it21:31
clarkboh right21:31
clarkbmordred: so I was thinking about that too21:32
clarkbmordred: makefiles21:32
clarkboh ya I need to dig up my makefile for jeblair21:32
mordredyup21:32
* clarkb boots other machine21:32
mordredyou know krow managed all of his servers with make files for the last several years21:32
mordredand thought we were crazy for using puppet at all21:33
clarkbwell I don't think puppet is completely crazy, but I do think make has been ignored as a useable tool21:33
clarkbprobably because of what sendmail did with it21:33
*** dcramer_ has joined #openstack-infra21:33
mordrednod21:33
openstackgerritMarc Abramowitz proposed a change to openstack-infra/jenkins-job-builder: Add a "list" command  https://review.openstack.org/8709021:34
*** imcsk8 has quit IRC21:34
mordredclarkb: also - the virtualenv_version facter change helped21:34
mordredso we're down to a smaller number of thigns that report activity all the time21:35
mordredI think it might be close to time to consider fixing those21:35
*** sweston has joined #openstack-infra21:35
rediskinclarkb: btw ./exercise.sh is run after ./stack.sh is done and stack user removed from sudoers21:36
rediskinclarkb: it cant broke too much if we preserve env for ./exercise.sh21:36
mordredroot      5880  5879  0 21:19 ?        00:00:00 bash -c brazenly_run_ssh cacti.openstack.org21:37
mordredgah21:37
mordrednotice: /File[/etc/resolv.conf]/seltype: seltype changed 'file_t' to 'net_conf_t'21:37
clarkbrediskin: well it could mean that devstack itself isn't able to run its exercises21:37
clarkbrediskin: it woudl be possible for devstack to become dependent on devstack gate21:37
mordredclarkb: ^^ on git03 (And ony on git03) we get that every time puppet is run21:37
mordredpleia2: ^^21:37
clarkbmordred: probably related to jeblairs dns cache changes21:38
mordredclarkb: but why only on one server?21:38
clarkbno idea21:38
*** e0ne has quit IRC21:40
*** harlowja_away is now known as harlowja21:41
pleia2yeah, that's a bit weird21:41
pleia2might compare the selinux attributes on /etc/resolv.conf between servers, ie: ls -lZ /etc/resolv.conf21:42
openstackgerritClark Boylan proposed a change to openstack-infra/zuul: Add a simple tox replacement Makefile.  https://review.openstack.org/8869821:42
clarkbjeblair: mordred ^ there is the makefile I have been fiddling with. I have been using it in zuul hence the push to zuul21:43
clarkbit is a bit repetitive and I am sure a make expert can clean it up a little. But make variable evaluation and dependency evaluation happens in such a way that having venvs like tox with make is hairy21:43
clarkbwe might want to consider a different structure but I wanted to keep it somewhat consistent21:44
anteayaI see tabs21:44
anteayado makefiles need tabs?21:44
mordredyup21:45
anteayaokay then21:45
anteayagood work on the tabs21:45
mordredclarkb: figured it out - it was a sequencing thign - you have to have chattr +i set to change the selinux context - but puppet sets it to -i - so something originally got done in the wrong sequence21:45
mordreds/+i/-i/21:45
mordredI have fixed it by hand - subseuqnet runs should be claen21:46
pleia2oh puppet21:46
jeblairwell, immutable files aren't really ideal for use with puppet21:46
*** marun_ has joined #openstack-infra21:46
jeblairit's hard to blame it for that :/21:46
clarkbjeblair: mordred: another neat trick with that makefile is now we can control pip/virtualenv ourselves21:47
clarkbwhich may or may not be a good thing :)21:47
jeblairi'd prefer to shift the blame to the crazy ways our providers manage to find to configure networking :)21:47
*** imcsk8 has joined #openstack-infra21:47
*** imcsk8_lap has joined #openstack-infra21:47
rediskinclarkb: or maybe i can run ./exercises.sh in post_gate_hook without sudo -H21:48
clarkbrediskin: oh, thats not a bad idea. set the vars such that gate_hook ends up being a noop21:48
clarkbrediskin: then rerun in post_gate_hook21:48
clarkbwell not a complete noop you still want devstack to start, but no exercises21:48
mordredclarkb: what do we expect check_elasticsearch_sha1 to do?21:49
clarkbmordred: it ensures that the sha1 of the downloaded package matches the sha1 we expect it to be21:49
mordredright - but to trigger what thing?21:49
clarkbmordred: to make sure we isntall a good package21:49
clarkbso it should only fire when we are reinstalling ES21:49
openstackgerritA change was merged to openstack-infra/devstack-gate: Handle feature branches in test-matrix.py  https://review.openstack.org/8865221:49
*** marun has quit IRC21:49
*** marun_ is now known as marun21:49
mordredgotcha. it fires every time21:49
clarkbnotmyname: ^ I am going to kick your change now21:49
notmynameclarkb: thanks. just saw the email21:51
*** dcramer_ has quit IRC21:54
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Silence successful elasticsearch runs  https://review.openstack.org/8870021:54
clarkbjeblair: let me know what you think about that makefile21:55
clarkbjeblair: but you should be able to git review -d and use make instead of tox locally21:55
*** imcsk8_l1p has joined #openstack-infra21:56
mordredjeblair http://puppetdb.openstack.org/report/review.openstack.org/f95f51a6a8bade4871256a1ac92264705a391cff gerrit puppet is unhappy and it's causing other thigns to not fire21:56
*** pdmars has quit IRC21:56
*** imcsk8_lap has quit IRC21:57
*** imcsk8 has quit IRC21:58
*** marun has quit IRC21:58
*** gokrokve has quit IRC21:58
mordredjeblair: oh - well, just running it a second time seems to have done the trick ::(21:59
*** marun has joined #openstack-infra21:59
*** andreykurilin_ has quit IRC22:00
*** nati_ueno has joined #openstack-infra22:01
*** ihrachyshka has quit IRC22:02
openstackgerritMarc Abramowitz proposed a change to openstack-infra/jenkins-job-builder: `test` command read/write stdin/stdout by default  https://review.openstack.org/8477822:03
*** lukego has joined #openstack-infra22:05
clarkboh! I can have make call make to work around the weird variable expansion and dep resolution22:06
mordredjeblair: when you get a sec - I'd love a quick look at https://review.openstack.org/#/c/88650/ but also at https://review.openstack.org/#/c/88692/ and https://review.openstack.org/#/c/88700/22:06
jeblairmordred: heh, that's teh definition of insanity, but it also appears to be standard operating procedure with puppet.  ;)22:06
mordredjeblair: tell me about it22:06
mordredjeblair: (I'm trying to clean up a few things before I try landing a new project so that there's less stuff to wade through to track success)22:07
openstackgerritSergey Skripnick proposed a change to openstack-infra/config: Run rally ./exercise.sh inside post_gate_hook  https://review.openstack.org/8868222:08
openstackgerritSergey Skripnick proposed a change to openstack-infra/config: Run rally ./exercise.sh inside post_test_hook  https://review.openstack.org/8868222:09
rediskinclarkb: ^^ i hope it gonna work :)22:10
jeblairmordred: so for the first thing, i think it's worth trying to understand why that doesn't work22:10
jeblairthe theory is that gerrit will auto restart when it's upgraded22:10
jeblairand that's been the practice; so what broke it?22:10
*** thedodd has quit IRC22:11
* rediskin went to sleep22:13
*** rediskin has left #openstack-infra22:13
clarkbhrm nope submakes don't really fix the issue of one .testrepository and lots of venvs22:16
clarkboh I know22:16
jeblairmordred: in order, -1, +2, -1 on those changes22:16
mordredjeblair: awesome22:16
jeblairmordred: maybe i missed something, but i think the 3rd one won't run anymore (clarkb you may want to look at https://review.openstack.org/#/c/88700/ again)22:17
mordredjeblair: it's required by the install exec22:17
jeblairmordred: does a require trigger an exec?  i thought only notify or subscribe do22:17
jeblair(if it's refreshonly, that is)22:18
mordredhttp://git.openstack.org/cgit/openstack-infra/config/tree/modules/elasticsearch/manifests/init.pp#n7222:18
mordredhrm. let me check22:18
clarkbjeblair: oh good point. should s/require/subscribe/22:19
*** rlandy has quit IRC22:19
*** lascii is now known as alaski22:22
mordredjeblair: gerrit-initial-init is run every time22:22
*** hashar has quit IRC22:22
*** imcsk8_l1p is now known as imcsk822:22
mordredit subscribes to /home/gerrit2/review_site/bin/gerrit.war22:22
mordredbut doesn't hav e refreshonly on it22:23
jeblairmordred: yep, i have just come to that same conclusion22:23
jeblairmordred: i don't know why it's necessary if gerrit-init is there...22:23
jeblairmordred: i mean, i read the comment, but gerrit-init has a ';' in it, so shouldn't it ignore init script stop errors?22:24
mordredyeah22:24
jeblairmordred: iow, shouldn't gerrit-init be enough, and maybe we can drop gerrit-initial-init?22:24
*** dangers is now known as dangers_away22:24
mordredjeblair: ++ let's try that22:25
mordredjeblair: you know - I think we had it in there originally and didn't care because the java command was a no0op - but that was before we needed to install the plugins too22:26
jeblairmordred: you added initial-init in f774c793ba6d271b8b64430a9925aaafb65668fb22:26
*** nati_ueno has quit IRC22:26
jeblairmordred: oh!22:27
jeblaironlyif and unless are important22:27
jeblairmordred: it's supposed to do one or the other depending on whether the init script exists22:27
mordredah22:27
jeblairmordred: so maybe the right thing is to add refreshonly to initial init22:27
mordred++22:27
jeblairoh wait, those lines disappeared22:28
jeblairyou added them when you added initial init, but they're gone now.  i need git blame for when lines go away :)22:29
mordred:)22:29
*** sweston has quit IRC22:30
jeblair75cee29cd6be6e044488088b50af76be0c38526e22:31
mordredjeblair: nod22:31
clarkbjeblair: git log -p22:33
jeblairhow about this: add refreshonly to initial-init, add the onlyif and unless lines back to both inits, remove the unless from install-core-plugins, add refreshonly to install-core-plugins22:33
clarkbjeblair: then I grep through that. its a bit of a sledghammer on thumb tack approach but works22:33
mordredyup22:34
jeblairclarkb: i used "git log -Sonlyif .../init.pp"22:34
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Stop trying to start gerrit every time  https://review.openstack.org/8865022:35
*** thomasem has quit IRC22:35
jeblairmordred: +222:36
*** mark0n has quit IRC22:36
mordredwhile I've got your attention ...22:37
*** jeblair is now known as corvus22:37
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Silence successful elasticsearch runs  https://review.openstack.org/8870022:38
mordredhow does that one look?22:38
mordredclarkb: I can get second +2 on https://review.openstack.org/88650 ?22:38
mordredcorvus: if we don't watch out, we're going to have a mostly clean puppet run that runs at predictable times22:39
*** corvus is now known as jeblair22:40
*** dcramer_ has joined #openstack-infra22:40
jeblairmordred: then we'll be ready to move to saltible, or whatever the cool new way to run something over ssh is!22:41
mordredhehe22:41
mordredjeblair: I uploaded the missing python script from the ansible patch22:41
clarkbmordred: done22:42
*** alexpilotti has quit IRC22:42
mordredwhich might make it more betterer to actually, yo u know, read22:42
mordredclarkb: ok. last one to bug you on: https://review.openstack.org/#/c/88700/ <-- does that look better to you now?22:43
openstackgerritA change was merged to openstack-infra/config: Stop trying to start gerrit every time  https://review.openstack.org/8865022:46
openstackgerritA change was merged to openstack-infra/config: php5-json is provided by php5-common  https://review.openstack.org/8869222:46
openstackgerritMichael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Story detail view cleanup  https://review.openstack.org/8870722:47
openstackgerritA change was merged to openstack-infra/config: Silence successful elasticsearch runs  https://review.openstack.org/8870022:50
*** msabramo has quit IRC22:51
mordredjeblair: woot! the gerrit change worked22:55
*** david-lyle has quit IRC22:55
openstackgerritMichael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Story detail view cleanup  https://review.openstack.org/8870722:55
*** zns has quit IRC22:56
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Fix typo in resource name  https://review.openstack.org/8870822:59
*** dcramer_ has quit IRC23:01
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Fix two typos in updated elasticsearch module  https://review.openstack.org/8870823:03
mordredself approving that ^^ there's a flat-out typo - the patch above is tested and fixes it23:04
jeblairmordred: woo!23:04
*** msabramo has joined #openstack-infra23:04
mordredjeblair: yah man23:04
mordredjeblair: once that bad boy lands, I'm a gonna land a new project and see if everything works like magic like it's supposed to23:04
*** dcramer_ has joined #openstack-infra23:05
clarkb++23:07
*** msabramo1 has joined #openstack-infra23:08
*** msabramo has quit IRC23:08
clarkbI got nerd sniped by make again23:08
mordredclarkb: it can suck your soul23:08
clarkbI should have a second patchset for a more terse version of the makefile23:08
mordredhaha23:08
mordredclarkb: so, be wary the arms race here23:09
mordredbecause if you do that too much, I'm going to replace all of our puppet infrastructure with a pile of makefiles23:09
mordredand THEN where will we be?23:09
clarkbthe 80s23:09
mordredthe only way forward then will be to install AFS23:09
*** CaptTofu has quit IRC23:10
asselinclarkb, thanks for the tips earlier...through that investigation I ended up finding some existing hooks I can tap into to run cinder cert tests on LVM "backend" which pass in my setup.23:10
mordredand then re-write all of openstack as a set of Makefiles that call a series of small C programs to do thigns23:10
*** CaptTofu has joined #openstack-infra23:10
clarkbasselin: cool23:10
clarkbmordred: oh man23:11
clarkbmordred: openstack will be like NIS23:11
mordredclarkb: except better, because we'd use hesiod instead23:11
asselingiven that my setup is proving to work, I'd like to get some reviews in my patch. https://review.openstack.org/#/c/87384/23:12
clarkbyou know you are smoking the make crack when you break into the functions to generate python versions from target names23:12
*** wchrisj has joined #openstack-infra23:13
mordredclarkb: that's when it's getting good23:14
* nibalizer actually does want to install kerberos and AFS23:14
nibalizerwhich seems to come up a lot on fridays23:14
mordrednibalizer: I do too. I'm REALLY not kidding when I say that23:14
*** CaptTofu has quit IRC23:14
*** dims has quit IRC23:15
nibalizermordred: do we just set it up, or do we try to find a purpose first?23:15
*** dcramer_ has quit IRC23:15
*** nati_ueno has joined #openstack-infra23:17
mordrednibalizer: oh, we have so many purposes23:17
*** lcostantino has joined #openstack-infra23:20
*** dcramer_ has joined #openstack-infra23:28
*** UtahDave has quit IRC23:28
mordredWOW - why are there so many changes in the nova queue23:29
mordredflashgordon: I blame you personally for this ^^23:30
*** sweston has joined #openstack-infra23:31
flashgordonmordred: Sreeram Chiropractor23:34
flashgordonhttps://review.openstack.org/#/c/84455/23:34
flashgordonhe is running recheck-vmware on all thethings23:35
*** sweston has quit IRC23:35
tjonesminesweeper is back up after a very long time....23:35
tjones"Sreeram Chiropractor" hee hee23:36
mordredflashgordon: I hate everyone23:37
mordredflashgordon: I just want to land an infra patch23:37
clarkbmordred: I think this is a good illustration for why it is insane that tempest needs 31 nodes to test one patch23:38
flashgordonclarkb: haha 31 nodes wow23:39
*** dcramer_ has quit IRC23:42
*** dims has joined #openstack-infra23:42
jeblairmordred: the zuul job graph does not look favorable23:43
openstackgerritClark Boylan proposed a change to openstack-infra/zuul: Add a simple tox replacement Makefile.  https://review.openstack.org/8869823:44
clarkbmordred: jeblair ^ that patchset makes the makefile less copy pastay23:44
mordredjeblair: it looks like hipsters found a new foodtruck23:45
clarkbok I need to get ready for portland trip23:46
* clarkb afks23:46
jeblairanyone know who Sreeram Yerrapragada is?  i mean, that's a lot of rechecking.23:46
clarkbbefore I go getting https://review.openstack.org/#/c/88671/ in would be good23:47
*** bhuvan has quit IRC23:47
clarkbjeblair: no clue23:47
*** lcostantino has quit IRC23:47
jeblairclarkb: nothings going in for hours :/23:47
clarkbjeblair: gate does have precedence23:47
mordredgate requires check23:47
clarkbso I think stuff should merge if it managed to check prior to that rush23:47
clarkbmordred: ya 88671 has check23:47
mordred+A23:48
jesusaurusim looking at all the various puppet modules around the jenkins slaves, and its a bit of spaghetti. as far as i can tell, anything with "bare_node" in it's name sets bare=false23:48
mnaserlol.23:49
mnaserdat check queue.23:50
flashgordonI like that spike23:50
mordredjesusaurus: the two words are a bit overused23:50
clarkbyup and I fixed some of the spaghetti recently23:51
clarkbits a lot better than it was23:51
clarkbjesusaurus: the issue there is we have bare == no services because devsatck and bare == service but no crazy devstack stuff23:51
jesusaurusclarkb: wow. i want to refactor some of this, but its hard to tell where to start23:51
mordredjesusaurus: in devstack-gate- "bare" means "not a devstack node" - in puppet, bare means "doesn't have infra creds"23:51
clarkbit got overloaded in a bad way23:51
jesusaurusoh, ugh23:51
clarkbjesusaurus: there are only two entry points. single_use_slave and slave.pp23:51
mordredjesusaurus: AaronGr also wants to refactor some of it - I think because he similarly wanted to figure something out and got confused23:52
clarkbwhich is better than it was before I hacked on it23:52
jesusaurusclarkb: what about puppet apply in prepare_node?23:52
clarkbjesusaurus: that should be single_use_slave23:52
jeblairwe have no plain precise nodes now, so we can actually change the 'bare-precise' node label to 'precise'23:52
mordredjeblair: ++23:52
clarkbjesusaurus: you may be looking at mordreds fork which is probably speghetti23:53
clarkbbut if you look upstream it is much better than befre (but bare is still overloaded terribly)23:53
jesusaurusim looking at origin/master (where my origin is the github mirror)23:53
clarkbjesusaurus: https://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/nodepool/scripts/prepare_node.sh#n45 single_use_slave23:54
jesusaurusah, yep23:54
jesusaurusim really just trying to wrap my head around everywhere that bare gets used in the puppet modules23:55
clarkbjesusaurus: see fc249c341fbc8c3ae7195965316769ac8ee2b78e23:56
jesusaurusjenkins::slave seems to have a good bit of logic around whether thats set, and i want to move a lot of the openstack-infra specific stuff out of ::jenkins and into ::openstack_project23:56
clarkbjesusaurus: and 0f4b0fae644d1c1156e87466d3988d86d6c39a1823:57
jesusaurusah, fc249c341 is enlightening23:57
*** khyati has quit IRC23:58

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!