14:00:46 #startmeeting tripleo 14:00:47 Meeting started Tue Feb 16 14:00:46 2016 UTC and is due to finish in 60 minutes. The chair is dprince. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:00:48 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 14:00:50 The meeting name has been set to 'tripleo' 14:01:08 o/ 14:01:18 whos here for a meeting? 14:01:22 hello 14:01:28 hello 14:01:32 hello 14:02:53 o. 14:03:12 o/ 14:03:17 o/ 14:03:46 #topic agenda 14:03:46 * bugs 14:03:47 * Projects releases or stable backports 14:03:47 * CI 14:03:48 * Specs 14:03:51 * one off agenda items 14:03:54 * open discussion 14:04:06 normal agenda... anything new to add this week speficically? 14:05:08 okay, lets get started then 14:05:12 #topic bugs 14:06:15 Several bugs earlier this week effected or related to CI outages 14:06:22 most have been closed I think 14:07:29 any other bugs to highlight this week? 14:07:55 https://review.openstack.org/#/q/topic:bug/1542486 is still keeping us from moving trunk 14:08:11 although there are now other issues blocking that, one of which is not even triaged yet 14:08:17 o/ 14:09:14 trown: will that undercloud patch pass CI then? should we recheck it? 14:09:36 dprince: it cant pass without removing the puppet-nova pin 14:09:57 dprince: I was using https://review.openstack.org/#/c/278553/ to test it, but I think now we have new issues 14:10:10 s/think/know/ 14:10:14 trown: okay, so could we do this. Update tripleo-ci to cherry pick this solution in, and remove the pin at the same time 14:10:23 trown: then we go back and land your instack undercloud solution 14:11:16 dprince: it is all more tangled than that... we need newer nova too, hence testing with a review that removes delorean pin and puppet pin at the same time 14:11:32 with depends-on on the fixes 14:11:49 trown: sure, lets just do it all in a tripleo-ci patch. I'm fine cherry picking whatever... 14:12:40 trown: also, rather than go direct for 'current' it might be nice to use another stable link to some hash/uuid'd repo 14:12:43 ok, I will put something up once I get the current issue triaged, not much point until we have a patch for that 14:13:00 trown: cool, thanks for the updates 14:13:03 dprince: ya, but which? :) 14:13:13 that is the hard part when it has been so long 14:13:22 trown: how about today, any of them 14:13:30 trown: don't care, just needs to pass 14:13:40 oh, I see, so we dont keep chasing our tail with new breakages 14:13:48 just keep testing against the same thing 14:13:55 trown: yes 14:14:01 dprince: cool will do 14:14:12 cool, lets move on 14:14:14 #topic CI 14:14:27 CI seems to be running fine mostly 14:14:40 master CI looks much better, but there are still a lot of failures on stable/liberty 14:14:48 other than not being able to bump our pin on master 14:15:25 Erm, CI looks stuck right now to me. 14:15:36 shardy: any bugs that need attention for stable CI? 14:15:39 The top of the zuul queue has been waiting for almost 12 hours. 14:15:49 bnemec: oh? 14:16:11 dprince: not sure, I've been out for a couple of days so just catching up, but I assume we're missing some backports for stable that fix the issues 14:16:11 dprince: http://status.openstack.org/zuul/ 14:16:17 Nothing is running AFAICT. 14:16:31 the patches I looked at were failing pingtest for various reasons, e.g nova failing to spawn the VM 14:17:07 bnemec: okay, thanks 14:17:43 okay, it shoulds like we have a CI outage. Would be nice to indicate the number of queued jobs on our tripleo.org CI status page too :/ 14:17:48 I have this in my logs from last night 14:17:49 [20:05:03] slagle: ohai 14:17:49 [20:05:09] slagle: Your CI is broken (see -infra) 14:17:49 [20:05:15] slagle: https://ci-overcloud.rh1.tripleo.org:13000/v2.0 14:17:49 [20:05:28] that refers to localhost which causes nodepool to try and auth against localhost 14:18:15 shardy: slagle fixup up the "overcloud heat isn't available on pingtest" yesterday with a ci review https://review.openstack.org/#/c/279894/ which may have helped somehow (at least it got a green run for the mido stable/liberty heat templates) 14:18:35 s/fixup/fixed 14:19:06 marios: ack, thanks - the patches I saw weren't that issue but nova errors, will check the latest results after that fix tho 14:19:07 yea i was on last night and infra was pinging us about our cloud 14:19:35 i dont have any more details, i tried to login and look, but i can't seem to login to the ci cloud any longer 14:19:52 I will look into this after the meeting 14:20:22 regardless of any issues ci is having today, there will likely be a big backlog of stable/liberty (which couldn't land for a while cos ci) so the load may/not be helping here 14:20:24 slagle: in the future please email derek and myself if infra pokes you about an outage 14:20:41 dprince: well i just happened to be on, and didn't have time to send an email as well 14:20:47 slagle: as for access, perhaps poke will foster 14:20:48 no one else was on 14:21:06 do we have monitoring? 14:21:11 EmilienM: no 14:21:15 EmilienM: we need it 14:21:21 dprince: i'd sync with infra about who to contact about the ci cloud and issues they are having 14:21:38 dprince: and how you'd like for them to do that 14:21:52 slagle: they probably just reach out to those online at the time 14:22:01 i.e. come into #tripleo and mention it 14:22:07 yep 14:22:30 dprince: we can talk about monitoring in a separated topic or offline or ML but I'm interested to help here 14:23:01 EmilienM: sure, keep in mind we want to re-spin our CI cloud entirely. And if that effort includes monitoring then it would be great too 14:23:20 EmilienM: we are using a quite an old release ATM for our CI cloud 14:23:41 anyways, lets move on. We can diagnose the CI outage later I think 14:24:00 #topic Projects releases or stable backports 14:24:12 I have a question 14:24:14 any stable branch updates this week 14:24:24 apetrich: ? 14:24:46 sorry still in CI this bug was closed and reopened https://bugzilla.redhat.com/show_bug.cgi?id=1295914 14:24:46 bugzilla.redhat.com bug 1295914 in rhel-osp-director "rhel-osp-director: Try to deploy oc 7.2 with netiso, from uc 8.0, fails right away with: Resource CREATE failed: resources.Networks: Property error: resources.StorageMgmtNetwork.properties.StorageMgmtNetValueSpecs: Value must be a string" [High,On_qa] - Assigned to hbrock 14:25:05 it is about backwards compatibility for 8 and 7.3 14:25:52 apetrich: looks like you didn't set the storage environment? 14:26:04 dprince: I started a thread on stable branch policy for Mitaka on openstack-dev 14:26:14 shardy: ++, I will reply 14:26:22 would be good to get input there so we can agree the plan and announce our FF deadline etc 14:27:01 +1 thanks for getting that rolling shardy 14:27:06 marios, I can look at it cheers 14:27:47 apetrich: am not sure, but the original description says "--ceph-storage-scale 1" but no storage env specified afaics, and then steps to reproduce doesn't . i should take a closer look 14:29:01 marios, I have a poc job for uploading of 7 overcloud image into a 8 undercloud I can give you access to the machine 14:29:09 marios, we can talk later 14:29:27 okay, well lets save any thoughts for stable barnches for the email thread 14:29:34 ack apetrich 14:29:42 #topic specs 14:30:13 I have a spec for the mistral integration at https://review.openstack.org/#/c/280407 14:31:12 rbrady: nice, thanks for posting that 14:32:36 I'm starting to get some feedback on pluggable services again 14:32:53 https://review.openstack.org/#/c/245804/ 14:33:18 slagle: could you boil down your concerns into a couple sentances that we might discuss here? 14:34:19 slagle: is it primarily the fact that we concatonate the puppet manifests taht you don't like? 14:34:29 sure 14:34:53 i'm not stricitly opposed to the concatenation in and of itself 14:35:01 more about what the implementation implies 14:35:35 so, if i wanted to use our puppet api services, and a 3rd party api service 14:35:48 slagle: the implementation strives to make it so that a "service architect" (i.e. a team focussed on trove, or sahara) can cleanly integrate their service without looking at the rest of the manifest 14:36:01 i'd have to do the 3rd party one in puppet, within the limitation that the puppet manifest would be concatenated with the others 14:36:40 slagle: this spec doesn't change the fact that we use Puppet for our configuration with t-h-t at this point 14:36:46 and i think that's too restrictive of a framework 14:37:09 slagle: I think that is a high bar to set for this pass. And there are consequences too 14:37:33 i dont think we have to solve this in the first pass 14:37:48 wait, I thought the spec was just about decomposing THT manfiests into smaller manifests in puppet-tripleo or in hieradata directly, like glance 14:37:53 but i would like to define the framework/interface that we would eventually like to get to 14:38:20 i dont feel like i know what direction this first step is even going 14:38:46 if it's clear to everyone but me, i'll dig into it a bit more and figure it out 14:38:58 it just seemed non-obvious what the end goal was 14:39:40 slagle: sure. I thought we had more concensus on this from a few weeks ago before devconf (in Brno). Seems like that discussion only confused the issues? 14:39:52 there's soem mention of developing frameworks and interfaces for composability, but no definition of what those are 14:40:16 slagle: the spec oulines 7 steps, plus a pre and post step 14:40:19 slagle: given the goal is 3rd party service integration, ideally would be nice to see an example of what it looks like this way, i.e. what dprince is suggesting with some existing reviews and the spec itself 14:40:39 slagle: that is the interface which would fit into our existing architecture quite nicely 14:41:00 FWIW I demo'ed this in tokyo: https://review.openstack.org/#/c/237370/ 14:41:14 That is glance-api and glance-registry split out into this architecture 14:41:30 dprince: if we focus on just the 7 steps, then i think the interface can be defined as puppet snippets for each step that are all concatenated together and run as one at each step 14:42:02 and personally i think that's too restrictive of a framework 14:42:05 slagle: yes, that is the interface 14:42:41 slagle: the architecture of t-h-t puppet is someone influenced by previous installers like Spinal Stack, etc. 14:43:02 that's wrong 14:43:06 slagle: With DIB tripleo previously we had a single pass per role. os-collect-config runs once 14:43:12 in spinalstack we had a composable composition layer 14:43:15 slagle: and does everything 14:43:19 something that is not in tripleo 14:43:30 EmilienM: ++ I said influenced. please wait just a second 14:43:37 https://github.com/redhat-cip/puppet-openstack-cloud/tree/master/manifests 14:43:37 It's somewhat difficult to fully define a "plugin" API until we decompose the current monolithic model IMO - I see the work proposed by dprince as just the first step down that path 14:43:54 the link I posted is something composable 14:44:00 i guess the use case i'm thinking of is if i had a service written in bash, or soem locally applied ansible playbooks and wanted those executed alongside the puppet ones 14:44:07 With t-h-t we are trying to have a stepwise architecture (not like DIB which was single pass). So step1 executes on all the roles. 14:44:31 and i'd envinsion where i wrote my templates, and just wrote out the softwraedeployments for 7 steps for my custom service 14:44:40 and that integrated cleanly alongside all the puppet stuff 14:44:52 so if i still wanted to run the puppet api services alongside my own, i could 14:45:00 that's the use case i'm thinking of anyway 14:45:41 instead of having to bolt my bash or ansible thing on at the end via ExtraConfig 14:45:46 i'd like to use the service interface 14:46:04 The initial vision was using stepwise execution. Step 1 on all the roles, then a cluster wide validation to make sure it works. Something like this (which we never landed) https://review.openstack.org/#/c/174150/ 14:46:15 Step 2 -> Then a step 2 cluster wide validation 14:46:26 and have it be more or less opaque to the the templates making up the framework what the actual implementation is 14:46:56 slagle: we do have ExtraConfig templates for some things in t-h-t that allow for some pluggablity in what you are asking 14:46:58 as is, the restriction is that it must be puppet, it must play nice when concatenated with all the other steps at the same sequence 14:47:09 slagle: and I think it would be totally reasonable to add to our interfaces in the future 14:47:46 slagle: I'm not restricting any future additions to the heat templates, rather I'm solving a problem with have with the puppet implementation today. Namely that it is monolithic 14:47:55 that won't help much, but I thought it would be useful to share. I wrote a small tool with ansible to compose puppet hieradata and deploy openstack in a composable way https://github.com/EmilienM/ansible-marionette 14:48:00 slagle: if we don't take this first step (of splitting things up) we can't move forwards 14:49:17 slagle: what I'm saying is, lets iterate towards these things. We can refine our interfaces in the years to come. But what I'm trying to solve here is service pluggability in puppet 14:50:30 slagle: So if you look at my validations patch you'll notice it allows you to plug in a shell script between each deployment run. I think something like that is what you are talking about. If not perhaps you could throw up a patch which gives a demonstration of how you'd like to accomidate these extra interfaces in Heat? 14:51:12 Or perhaps we need to take this to the mailing list to discuss further. 14:51:18 i think that would be premature 14:51:30 I wonder if we can address the requirement expressed by slagle via encoding the config group in the per-service mapping: 14:51:32 i'm not trying to come up with patches, merely define what interfaces we are aiming for 14:51:36 https://review.openstack.org/#/c/237370/7/puppet/roles/glance-api.yaml 14:51:49 so that there's agreement on that 14:51:53 e.g rather than just "config_step4: manifest" 14:51:56 dprince: would it make sense to make the steps themselves pluggable, for example have a step4/5 (whichever corresponds to where neutron services are declared) for each neutron service provoider/intergration... we may have some duplication there though 14:52:07 you'd have config_step4 contain e.g a "puppet" key or something 14:52:27 then you have an interface we could potentially extend to also chain e.g scripts or whatever together 14:52:31 dprince: i think i am thinking too much about the problem description in the spec 14:52:44 slagle: sometimes having a patch helps people see your idea, and/or formulate it. That is all I'm saying 14:52:57 where as what (i think) you're saying is that the implementation isn't going to address a lot of that problem description 14:53:37 it sounds like the problem description is "puppet manifests are monolithic" 14:53:44 or should be, rather 14:53:59 slagle: I had a section called "taking it a step further" (to truely composable roles). Would you like me to remove that language? 14:54:27 dprince: i mean change the values of the output in https://review.openstack.org/#/c/236243/18/puppet/controller-config.yaml so we use the corresponding Step5 for example, for given integration 14:55:04 dprince: it's more the previous 2 paragraphs tbh 14:55:30 marios, slagle, shardy: would you be interested in a video conference this week to speak on these issues then? 14:56:01 the 2nd paragraph talks about defining the interface 14:56:09 At this point I'm almost inclined to delete the spec and say we can just use patches (which may better explain my motivations than the spec does. 14:56:16 dprince: sure 14:56:19 and i think the implementation defines an interface i don't feel is flexible/pluggable enough 14:56:37 slagle: it works, and is an iterative step forwards 14:56:39 sure, a call would be fine for me as well 14:56:54 dprince: sorry, client dropped... yeah would be happy to join a call 14:57:23 It's kinda interesting looking at the fuel plugin interface btw - they have covered a lot of these same issues (deployment steps, dependencies between steps, running scripts vs puppet etc etc) 14:57:31 okay, we are almost out of meeting time 14:58:44 thanks everyone 14:59:06 #endmeeting tripleo