Friday, 2015-09-04

nibalizero/00:00
*** richm has quit IRC02:32
EmilienMclarkb: so I got AssertionError: False is not true : Timed out waiting for 172.24.5.48 to become reachable02:53
EmilienMI guess I need to manage 172.24.5.48 interface02:53
EmilienMsorry, 172.24.5.48 is in fact the floating IP02:55
EmilienMso I'm missing something now02:55
EmilienMpabelanger: the experimental job for unit test is working as expected, I guess we can move forward to having it replacing our current jobs03:11
*** asahlin has quit IRC03:27
*** asahlin has joined #openstack-sprint03:29
EmilienM_ody: I can create vs_bridge with current puppet-vswitch module, in one run03:58
*** mrmartin has joined #openstack-sprint04:12
EmilienMcrinkle: I got it work with simple puppet code finally -> https://review.openstack.org/#/c/217352/04:57
*** mrmartin has quit IRC05:16
*** mrmartin has joined #openstack-sprint06:45
*** sbadia has joined #openstack-sprint08:48
EmilienMgood morning sprinters!11:49
*** mrmartin has quit IRC11:54
*** rfolco has joined #openstack-sprint12:00
*** cdelatte has quit IRC12:00
*** cdelatte has joined #openstack-sprint12:15
*** cdelatte has quit IRC12:41
*** cdelatte has joined #openstack-sprint12:43
*** mrmartin has joined #openstack-sprint12:55
*** richm has joined #openstack-sprint13:23
*** mestery has joined #openstack-sprint13:24
vinsh_meow.13:24
*** vinsh_ is now known as Vinsh13:24
*** mestery has quit IRC13:26
*** mestery has joined #openstack-sprint13:27
EmilienMso pabelanger managed to make the experimental job that runs all unit tests in a single job13:31
EmilienMit's working as expected, now I'm preparing a patch to infra so we move the experimental job to replace others13:32
Vinshwoah, runs all the puppet version one after the other?13:32
VinshIs that what you mean by all the unit tests?13:32
EmilienMpabelanger: http://whatis.techtarget.com/definition/six-thinking-hats-retrospective13:50
EmilienMVinsh: yeah13:50
EmilienMto save nodepool resources13:50
Vinshheck ya. great13:50
pabelangerVinsh: it is serial right now, but something could be done to make it parallel13:51
*** richm has quit IRC13:51
VinshStill probably not the slowest job of the bunch13:52
pabelangeractually, you could do something like diskimage-builder to get parallel jobs going on a host13:52
pabelangerwould have to play with it13:52
pabelangereither way, that patch will save some nodepool resources13:52
EmilienMcrinkle, nibalizer: I would like your acks on https://review.openstack.org/#/c/220527/14:52
EmilienMjamespage: b2 seems promising14:54
EmilienMwow14:55
EmilienMtempest even run, etc14:55
jamespageEmilienM, yeah - I see errors for cinder volumes attachment and boot from volume - but that may be a charm misconfiguration14:57
jamespageall other tempest smokes pass14:57
EmilienMyeah, we are not at this stage yet14:57
EmilienMour functional testing is not testing it14:57
EmilienMawesome14:57
EmilienMjamespage++14:57
EmilienMjamespage: ironic fails: https://review.openstack.org/#/c/219830/14:58
EmilienMlet me find why14:58
EmilienMjamespage: http://logs.openstack.org/30/219830/2/check/gate-puppet-ironic-puppet-beaker-rspec-dsvm-trusty/6ba23f5/console.html#_2015-09-04_14_28_48_49614:58
jamespageEmilienM, ironic don't appear to be lining up with everyone else for liberty - https://launchpad.net/ironic/liberty/4.0.014:59
jamespageso you still have kilo packaging there - which will be broken atm15:00
jamespage(oh and ironic is also not part of our core testing)15:00
EmilienManteaya: we would like http://docs.openstack.org/developer/puppet-openstack for now15:00
EmilienManteaya: I want to double check with crinkle / nibalizer if we could use http://docs.openstack.org/developer/puppet15:00
anteayaokay thanks15:00
EmilienMand collaborate on content15:00
EmilienMthat would be the best option15:00
EmilienMbut I need their thoughts15:00
EmilienMso for now, http://docs.openstack.org/developer/puppet-openstack15:00
EmilienMit would be a good start15:01
EmilienMand I think we can iterate later15:01
anteayaI will work towards that15:01
anteayathank you15:01
anteayawell it would be tough to change the name later15:02
EmilienManteaya: why?15:02
anteayaas the docs would be published to one site according to index engines15:02
anteayaand then switching would require a redirect15:02
anteayaas people would expect them in the place they expect them15:03
EmilienManteaya: mhh, ok - let's prepare patches now, like if it would be puppet-openstack15:03
EmilienManteaya: and wait for crinkle / nibalizer to see if we can use 'puppet'15:03
anteayaokay15:04
anteayabest if we agree before the repo is created though15:04
anteayaso thanks15:04
EmilienManteaya: in the meantime, you can work on the patch though, we will iterate patchsets if needed15:04
anteayaright15:06
anteayadoing so now15:06
anteayattp://docs.openstack.org/infra/15:08
*** richm has joined #openstack-sprint15:10
crinkleEmilienM: I don't really care about the docs name, /puppet and /puppet-openstack both seem fine15:17
EmilienManteaya: go for /puppet15:33
anteayaokey dokey15:34
EmilienMcrinkle: good news, we can bump all liberty to 'proposed' for ubuntu15:35
richmEmilienM: Are you working on https://review.openstack.org/218059?15:37
EmilienMrichm: no15:37
richmEmilienM: ok - I will then15:37
EmilienMrichm: ack15:37
EmilienMcrinkle: last night I changed the code to use puppet in a very simple way on https://review.openstack.org/#/c/217352/ and it worked - today I try to get the jobs green for both ubuntu & centos15:41
anteayaEmilienM crinkle and others, patch to create repo puppet-openstack-docs with publish jobs intended to publish to docs.openstack.org/developer/puppet15:46
anteayahttps://review.openstack.org/#/c/220555/415:46
EmilienManteaya: this sounds excellent15:48
EmilienManteaya: why experimental?15:48
anteayabecause it is a new job15:48
anteayaand as a reviewer I really really really like to see new jobs in the experimental pipeline first15:48
anteayaso I need to practice what I preach15:49
anteayaI might need a no-op job then for check15:49
anteayaand gate15:49
EmilienManteaya: yeah, but that means we need to trigger it manually, right?15:53
anteayayes15:56
EmilienMsince the job is not affecting other projects, I suggest we go directly in check and make the first commit pass CI15:57
EmilienMinstead of relying on experimental pipeline that requires manual commands to trigger the job15:57
anteayaokay we can do that16:00
anteayacan you put that comment on the patch?16:01
EmilienManteaya: sure thing16:01
EmilienManteaya: done16:02
crinkleEmilienM: what happened to clarkb's suggestion of using the devstack-gate function?16:03
anteayaEmilienM: thanks16:04
anteayacrinkle: are you talking about the docs patch?16:04
EmilienMno16:04
EmilienMcrinkle: I could not make it work last night, I got tempest issues when not managing br-ex and also the function failed to run in our CI16:04
* anteaya moves on16:04
crinkleanteaya: no, https://review.openstack.org/#/c/21735216:05
EmilienMit seems like the function is really working for devstack16:05
EmilienMand I had troubles to debug it16:05
anteayacrinkle: k, thanks16:05
EmilienMmy exec is really straightforward, I think we should move on with it now16:05
EmilienMmoreover, I manage the vswitch with puppet16:05
EmilienMin fact, there is no bug (cc _ody), puppet run is idempotent for vswitch module too16:05
EmilienMI think we can move to 'proposed', results are really good: https://review.openstack.org/#/q/status:open++branch:master+topic:beaker/trusty-proposed,n,z16:09
sbadiahi here!16:12
* EmilienM afk lunch16:17
_odyHunner: I am in the room...when you get onsite.16:31
Hunnernibalizer: Coming in today, or just hanging out?16:55
* EmilienM back17:28
EmilienMpabelanger: can you -1/+1 https://review.openstack.org/#/c/220527/ when you got time?17:32
*** mrmartin has quit IRC17:37
EmilienMwe are going to loose network for 15 min - office outage17:44
HunnerOffice outage in the middle of the workday? That's odd...17:45
HunnerAnd sounds expensive17:45
EmilienMcrinkle: neutron fixed a bug upstream, that means we can enable again LBaaS https://review.openstack.org/#/c/220566/17:45
crinkleyay \o/17:46
EmilienMHunner: yes... one of the reason I love working from home17:46
EmilienMback online18:02
EmilienMcrinkle: in the next cycle, we could even imagine puppet jobs gating (experimental) some other projects, it would be interested18:02
EmilienMinteresting*18:02
EmilienMcrinkle: the tempest patch is passing on trusty now!! except a bug in tempest that I'm investigating with matthew (tempest ptl) - that means we have integration jobs working on both distros soon :-)18:11
nibalizerHunner: hi18:19
Hunnernibalizer: hi18:19
nibalizeryou at puppetlabs?18:19
nibalizerI'll come down if I'm welcome :)18:19
HunnerYeah, for a bit. Probably leaving around 1ish18:19
HunnerAnd lunch at 12ish, so not much time ¯\_(ツ)_/¯18:20
HunnerBut you're welcome :)18:20
HunnerAnd _ody is here18:20
*** mrmartin has joined #openstack-sprint18:23
nibalizerHunner: aight i'll bomb on down18:26
pabelangerhttp://logs.openstack.org/97/220297/15/experimental/gate-puppet-openstack-integration-dsvm-multinode-nv/af03af6/console.html18:30
pabelangerusing ansible to invoke zuul-cloner / r10k on the remote node18:30
pabelangerworks like expected18:30
pabelangerEmilienM: ^18:30
EmilienMexcellent :)18:31
EmilienMcrinkle: can you review https://review.openstack.org/#/c/219818/ please ?19:10
crinkleEmilienM: I thought openstack_extras::repo took care of installing the keyring and apt-get updating? if not it's not working properly19:11
EmilienMcrinkle: openstack_extras::repo runs apt-get update *after* adding the keyring package19:12
EmilienMcrinkle: we need to do that in integ because trying to install a package without apt-get update provides a race condition19:12
EmilienMsometimes it works19:12
EmilienMsometimes not19:12
EmilienMand we don't have the issue in  beaker because beaker runs apt-get update for us19:12
EmilienMwhen preparing the node19:12
crinklehrm okay19:13
crinkleEmilienM: what's the deal with ceilometer and horizon on centos?19:14
EmilienMcrinkle: I see no clean way to tell puppet running it before, then after again.. except adding an exec somewhere19:14
EmilienMcrinkle: https://etherpad.openstack.org/p/puppet-liberty-blocker19:14
EmilienMhttps://bugs.launchpad.net/puppet-ceilometer/+bug/149098619:14
openstackLaunchpad bug 1490986 in puppet-ceilometer "centos: puppet run is not idempotent (collector & notif agents)" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)19:14
EmilienMhorizon is ? RDO horizon folks are on it right now19:14
EmilienMI'm asking to mrunge19:15
EmilienM<mrunge> 3 missing packages19:17
EmilienM<mrunge> jpena already packaged 2 of them19:17
EmilienMcrinkle: ^19:17
EmilienMthat means horizon should be fixed very soon19:17
EmilienMand since it's feature freeze, I guess we can expect RDO packages more stable19:18
_odynibalizer: I am on the 6th floor at my desk.  Was just thinking about getting food.  Hunner is having lunch with someone today.19:21
* _ody didn't see you 19:25
EmilienMI need to jump in my train, probably back online later19:26
*** cdelatte has quit IRC19:49
EmilienMback19:56
richmEmilienM: I don't know if you saw my latest ramblings on #puppet-openstack19:57
richmEmilienM: I've got a big problem with puppet introspection19:57
EmilienMnibalizer, crinkle: do you know why https://review.openstack.org/#/c/219804/ and others do not merge? the patch in Depends-On is merged though, should I trigger something? 'recheck' ?19:58
EmilienMrichm: looking now19:58
richmPuppet::Resource.indirection.find is apparently not working if the resource name contains "::"?19:58
EmilienMwow, I never tested it, maybe _ody and Hunner did19:58
EmilienMrichm: testing19:59
mfischsounds like a bug19:59
EmilienMI hope not19:59
nibalizerEmilienM: Ill look20:11
nibalizerEmilienM: it is running tests in gate20:12
nibalizeryou can see it at http://status.openstack.org/zuul/20:12
*** asselin_ has joined #openstack-sprint20:14
*** asselin has quit IRC20:16
*** EmilienM has quit IRC20:16
*** EmilienM_ has joined #openstack-sprint20:16
*** EmilienM_ is now known as EmilienM20:16
EmilienMnibalizer: I +A to trigger gate20:17
nibalizerEmilienM: yes and its running20:19
EmilienMnibalizer: but I was wondering if I had to do that or if I missed something20:20
nibalizerI can't say what state it is in before the second +a was added20:21
nibalizerbut I think it doesn't matter20:22
nibalizerEmilienM: pabelanger https://review.openstack.org/#/c/219879/ would add nonvoting puppet4 beaker tests20:44
EmilienMnibalizer: so you want to test 3.x also?20:48
*** mrmartin has quit IRC20:48
nibalizerEmilienM: yes20:48
nibalizerI need to for infra for the forseable future20:48
nibalizerpabelanger: if you're around want to chat about https://review.openstack.org/#/c/220527/4/jenkins/jobs/puppet-module-jobs.yaml ?20:49
nibalizerI think branch inspection inside a job is the path to madness20:49
nibalizerWe did it a TON internally at HP and it ended up being super confusing20:49
nibalizeralthough it is worth noting that infra doesn't write or run unit tests so its all good20:50
nibalizerfrom infras perspective20:50
EmilienMnibalizer: reviewed20:51
EmilienMnibalizer: I actually prefer your naming, but in that case we should also rename unit & syntax jobs to be consistent20:51
EmilienMnibalizer: have you seen https://review.openstack.org/220527 ?20:52
EmilienMoh sorry20:52
EmilienMjust got the buffer (in the train atm)20:53
nibalizerhaha20:53
nibalizerare you headed home already?20:53
nibalizersafe travels if so20:53
EmilienMno I have 3 hours of train20:53
EmilienMthanks :)20:53
_ody3 hours? WHY!20:58
*** rfolco has quit IRC21:02
EmilienM_ody: montreal -> quebec21:03
nibalizerEmilienM: sounds like a good reason to get gertty up21:04
EmilienMyeah :)21:04
*** HenryG has quit IRC21:07
*** zaro has quit IRC21:09
EmilienMHunner: WOOT !!!!! https://forge.puppetlabs.com/openstack21:10
*** HenryG has joined #openstack-sprint21:11
nibalizermmm approvals21:11
EmilienMour pupper modules getting approved, awesome21:11
*** zaro has joined #openstack-sprint21:22
pabelangernibalizer: taking it easy now, but don't mind adding it for experimental for now.  If people don't want it there, then Rakefile works, however I don't want to update each module manually, so we're talking about some sort of gem to hold our unit-test layouts.21:38
pabelangerso, can go either way21:38
pabelangerbut would rather deal with JJB over a gem presonally21:38
pabelangerlonger term, getting -infra involved and doing some sort of single node, that launches chroots / containers for this purpose21:39
nibalizerya I definitely believe there is a need in -infra for a lightweight node21:44
nibalizermaybe it is a bunch of containers that nodepool can give out21:44
nibalizerbecause things like lint and syntax don't require the huge nodes that infra usually deals with21:45
pabelangerexactly21:45
nibalizerat which point having separate jobs is the way to go21:46
EmilienMnibalizer: we made it to save nodepool resources21:47
EmilienMprobably we can improve it21:47
clarkbnibalizer: is there a concern with using nodes that are too big?21:48
* clarkb is somewhat confused but in general the way we handle this elsewhere is run tests in parallel per run21:49
clarkbuses all the avialable resources as quickly as possible21:49
nibalizerclarkb: so at this point we are using 16 nodes per puppet-* change21:53
nibalizerI haev a change in flight to increase that by 621:53
nibalizerso we're trying to bring that number down21:53
nibalizerthe goal is to reduce the impact to infra21:53
clarkbI thought it was like 6? lint + all the version unittests + a rspec breaker run or two21:53
nibalizerhttps://review.openstack.org/#/c/219289/21:53
clarkbnibalizer: so first thing I would do is smash lint and syntax together since they are all common static analysis21:54
nibalizerand I have a change up to run beaker on puppet3 and puppet 421:54
nibalizerclarkb: ya paul has a change up for that21:54
nibalizerthats actually what we are discussing21:54
nibalizer https://review.openstack.org/22052721:54
clarkband I would drop the non upgrade case becaues thats a subset of the upgrade case21:54
nibalizerit gets a little complicated21:54
nibalizeroh hrm21:55
nibalizerI'm not sure if that is true but it might be21:55
clarkband finally I would pick puppet versions that exist on distros or other common deployment and not do all of them21:55
clarkbkind of like how we do python21:55
nibalizerya I think doing 3.4 3.x and 4.x is probably sufficient21:55
clarkbI don't know what those versions are but testing every version seems silly now particularly that puppetlabs says we are sember21:55
clarkb*semver21:55
clarkbif that breaks file a bug with puppetlabs move on21:56
nibalizerwell if someone starts using a feature that was added in 3.5 we need to test on 3.4 to know that it breaks 3.4 compat21:56
clarkbthat gets you to like 6 jobs21:56
clarkbnibalizer: then test the earliest 3 version and the earliest 4 version?21:56
clarkbanyways, my point is that rather than getting smart about how we run jobs and complicating the tooling we should instead get smart about what is worth testing and how we test then go from there21:57
nibalizersure that works today21:57
EmilienMnibalizer: yes21:57
EmilienMit would be imho21:57
nibalizerbut why not have infra provide lightweight nodes so that we can run lots of cheap tests without the complexity of chaining them21:58
clarkbthe test platform is somewhat intentionally dumb and we are trying to make it even simpler21:58
nibalizerim not say we do that today21:58
nibalizerbut in the future21:58
clarkbnibalizer: because that makes the system complicated. Not saying we can't do it if there is a real need but I like simple21:58
clarkband right now I honestly don't think that puppet modules are any more special than everything else we test21:59
clarkbwhich we have done effectively with the nodes available21:59
nibalizerI think this problem exists across the board21:59
nibalizerwhere some tests need big nodes and some do not21:59
clarkbbut thats ok21:59
clarkbthere is never a case where running a small test on a big node is problematic21:59
clarkbif you need to restrict resources to keep a test case small that is doable without extra special node types22:00
nibalizersure but if the quota infra has from cloud providers is the limiting factor, more smaller nodes could exist than big nodes right?22:00
nibalizeror is the api rate limiting the slow part?22:00
clarkbnibalizer: we can query graphite, its both to an extent22:01
clarkbnibalizer: when we aren't at quota we really feel the rate limiting (whther by api servers or naturally due to delays in operations)22:01
nibalizerya22:01
clarkbwhen we are at quota we of course can't boot more things22:01
clarkbbut for example in hpcloud we can't boot more nodes even if we made them smaller so thats not a huge win22:02
nibalizerit looks like we got near quota today22:02
nibalizerbut that has stopped22:02
clarkbin rax thats the same too, would require much more tuning in quota22:02
clarkbso you would hvae the same number of nodes they would just be less useful22:02
clarkband your shcedulre would have to be smarter to place jobs on the appropriate node sizes22:02
nibalizerso I think having more small nodes is only valuable when we're near quota22:02
clarkband to know when to boot which type properly22:02
nibalizerclarkb: you can do that with label right?22:03
clarkbnibalizer: except for the fact that we have quota limits other than ram and cpu22:03
nibalizerlabel is trusty-small or trusty-dsvm22:03
clarkbnibalizer: sort of, I mean nodepool does it but it isn't always great at it22:03
nibalizerso a lint check goes on a small22:03
clarkbnibalizer: right the hard part is when you have 100 small jobs, 100 big jobs, and 25 big quota/40small quota22:03
nibalizerand of course, not to be this guy, but four big nodes in rax that just host docker containers wouldn't be affected by cloud api lmits22:03
clarkbhow do you fit into those buckets?22:03
clarkbpretty sure solving these problems is what people do toget PhDs22:04
nibalizerclarkb: ya thats a hard problem22:04
nibalizerbeyond me22:04
nibalizerbut right now we put a 30G node in charge of running puppet-lint22:04
nibalizerwhich seems silly22:04
clarkbso from my perspetive supporting the simple thing is easier for everyone22:04
clarkbnibalizer: its an 8GB 8vcpu node22:04
nibalizeri thought it was 30G then gimped to 822:05
clarkbnibalizer: what the underlying cloud flavor is is an implementation detail, the tests are unaware22:05
clarkbI suppose they could discover that the kernel boot parameters restrict memory, update the kernel boot param, reboot then the test would fail22:08
clarkbbut that would be the only case any specific test is aware of such things22:08
pabelangerYa simple is great, and I love how our -infra works today. But I agree with nibalizer on that point, launching multiple nodes for lint checks is over kill in that instance. I think something could be added, but not sure what that looks like.  What I did today, was a simple first iteration22:13
pabelangerI also agree with clarkb that we could prune the puppet versions to what distros ship today22:14
pabelangertrain time22:15
pabelangerwill be back in -infra22:16
*** pabelanger has left #openstack-sprint22:16
clarkbbut ya as a today note our quota allows the number of hosts we run today. It would not allow more nodes just because we made msaller nodes22:19
clarkbso there is literally no benefit to mixing smaller nodes in22:20
crinkleis there a problem for us on the puppet side that reducing the number of nodes fixes?22:22
crinkleseems like we're trying to help Infra out and Infra is saying they don't need the help22:23
clarkbwell I am saying that using smaller nodes won't help. Using fewer nodes may help particularly when we are at quota22:23
crinkleokay22:23
clarkbmy suggestions were a possible way of doing that without losing test coverage, but I don't know enough about your testing to be sure22:24
nibalizerclarkb: see the patch the paul made22:24
nibalizerwe beed different thibgs to happen based on branches22:24
nibalizerso ita complex22:24
nibalizerclarkb: im not saying i need smaller nodes, im asking why it wont help and i thibk youve explained that to me22:25
nibalizermaybe we run the normal puppet tests in one test and the 'latest' on a separate test22:26
nibalizerthat way we get less nodes used22:26
nibalizerbut also have less complexity22:26
EmilienMmfisch: I got issues to spawn a VM on ubuntu, http://logs.openstack.org/52/217352/56/check/gate-puppet-openstack-integration-dsvm-trusty/b44e53b/logs/nova/nova-compute.txt.gz#_2015-09-04_21_50_18_64322:31
EmilienMhave you seen that before?22:31
EmilienMI might need to adjust something22:31
EmilienMI think dbus needs to be managed22:42
EmilienMthat's a bug in puppet-nova22:46
EmilienMthanks everyone for joining the sprint, mfisch, clarkb, crinkle, clayton, nibalizer, Vinsh, richm, _ody, mwhahaha, Hunner, anteaya - I'll make sure we do a retrospective to see how we can do better next time23:03
EmilienMhave a great week end23:03
*** greghaynes has joined #openstack-sprint23:30
nibalizerwoooo23:50

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!