19:01:34 #startmeeting tripleo 19:01:35 hello 19:01:35 Meeting started Tue Oct 8 19:01:34 2013 UTC and is due to finish in 60 minutes. The chair is lifeless. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:01:36 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:01:37 hey 19:01:38 The meeting name has been set to 'tripleo' 19:01:42 o/ 19:01:55 hiya 19:01:57 o/ 19:02:04 hi 19:02:09 _o/ 19:02:31 \o/ 19:02:37 #topic agenda 19:02:47 incoming 19:02:49 bugs 19:02:49 reviews 19:02:49 Projects needing releases 19:02:49 CD Cloud status 19:02:52 CI virtualized testing progress 19:02:54 Discuss latest comments on LXC+ISCSI bug: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1226855 19:02:57 Insert one-off agenda items here 19:03:00 Managing blueprints - migrate them all to tripleo? 19:03:02 Tuskar mailing list - what to do with it? 19:03:05 Reviews and +2 on something you fixed up 19:03:07 open discussion 19:03:10 #topic bugs 19:03:12 #link https://bugs.launchpad.net/tripleo/ 19:03:15 #link https://bugs.launchpad.net/diskimage-builder/ 19:03:17 #link https://bugs.launchpad.net/os-refresh-config 19:03:20 #link https://bugs.launchpad.net/os-apply-config 19:03:22 #link https://bugs.launchpad.net/os-collect-config 19:03:25 #link https://bugs.launchpad.net/tuskar 19:03:27 #link https://bugs.launchpad.net/tuskar-ui 19:03:30 #link https://bugs.launchpad.net/python-tuskarclient 19:05:33 we have just one critical bug this week 19:05:33 \o. 19:05:33 https://bugs.launchpad.net/python-tuskarclient 19:05:33 this is fantastic 19:05:34 https://bugs.launchpad.net/python-tuskarclient/+bug/1213882 19:05:34 pylint is deprecated, I've submitted a patch for openstack-info/config to remove it 19:05:34 but python33 is in need of support 19:06:17 yeah so last time we checked with pblaho, it wasn't possible bc of dependencies 19:06:27 but i think it might be time to try again :) 19:06:52 if it's an oslo problem, we should fix it in oslo and get a release made 19:07:13 but when I looked at the failure log, there was non python 33 stuff in python-tuskarclient itself 19:07:15 I think there was problem with some python-*client 19:07:27 yeah, you are right... 19:07:35 but it is not that much of stuff... 19:07:46 so, we have a choice 19:07:51 we can delete the 33 job and say 'not yet' 19:07:57 some time ago I started to clean it but stopped due to deps 19:08:08 or someone can step up and drive it to completion - going into deps as needed 19:09:05 Is someone interested in getting it done? 19:09:34 i'd say try to fix it for py33 and if we hit the wall really hard, then we say 'not yet'. I'd like to have it py33 ready but i don't want to burn Red Hat time on that if it turns out it's too soon. 19:09:47 .. as we have other priorities 19:09:55 fair enough! 19:10:00 so sign me up for investigation and hopeful fix ;) 19:10:03 cool 19:10:11 maybe me.. but... question if we should introduce deps on git commits/tags.... b/c we may need new release of smth 19:10:12 anyone have pet 'high' bugs they want to talk about? 19:10:28 pblaho: if we need a new release of something else in openstack, help that project do it's release. 19:11:08 pblaho: if we need someting outside of openstack, same thing usually :), but we shouldn't dep on git commits/tags - we can't release like that and expect folk to package it 19:11:30 lifeless: yeah... makes sense (git tags...) 19:11:43 lifeless: not sure, how important this is https://bugs.launchpad.net/tuskar/+bug/1236703 , but it would be nice to here what you guys think about this one 19:12:18 rpodolyaka1: I think it's high 19:12:22 rpodolyaka1: I've triaged it 19:12:41 rpodolyaka1: it's the sort of thing we really need a clear story on before expecting users to use it 19:12:49 py33 thing - for interested - my previous work on it https://github.com/petrblaho/python-tuskarclient/tree/python-2to3 19:13:00 ^ little outdated :-) 19:13:15 pblaho: yeah, fwiw most folk use 'six' within OpenStack to do python 3.3 stuff. 19:13:37 lifeless: I believe, we should look into "auth via keystone -> give Tuskar API your token" direction? 19:14:06 lifeless: I think, this is what Nova does e.g. to talk to Glance/Neutron/etc on behalf of a user 19:14:13 rpodolyaka1: I think it's something like this: if tuskar will be taking independent actions, then it should have it's own specific account 19:14:23 rpodolyaka1: if it won't be, then yeah via keystone. 19:14:27 pblaho: ah sorry i rushed into it, i forgot you have it halfway through. We can arrange who's going to tackle it further based on immediate priorities. 19:14:39 rpodolyaka1: it's not clear to me yet which case we're on; 19:14:45 rpodolyaka1: also, remember there may be many overclouds 19:14:53 rpodolyaka1: so it really /cannot/ be in the config file 19:14:53 jistr: no problem at all 19:15:02 lifeless: agree 19:15:04 rpodolyaka1: may want to capture this into the bug log 19:15:17 ok, any other high bugs to discuss? 19:15:59 also, please remember to do bug triage? pretty much everyone can do triage... 19:16:15 ok 19:16:18 #topic reviews 19:16:23 http://russellbryant.net/openstack-stats/tripleo-openreviews.html 19:16:26 http://russellbryant.net/openstack-stats/tripleo-reviewers-30.txt 19:16:29 http://russellbryant.net/openstack-stats/tripleo-reviewers-90.txt 19:16:36 * SpamapS is here now 19:16:57 helps if phone that alerts you to "all the things" is turned on 19:16:58 I want to talk about the health of the project as far as reviews are concerned - not about how much any single reviewer is diong 19:17:43 and here's the key statistic: 19:17:45 Average wait time: 0 days, 8 hours, 20 minutes 19:17:45 Median wait time: 0 days, 6 hours, 7 minutes 19:17:46 Number waiting more than 7 days: 0 19:17:58 Which I think is pretty good. 19:18:12 that reads well. be nice to get comparisons with other projects for context 19:18:21 this is really good, comparing to what we could see in Nova a few months ago 19:18:32 +1, that beats my experiences with Horizon by a good bit but I don't know actual numbers 19:18:32 this is nova today: 19:18:34 Average wait time: 3 days, 23 hours, 12 minutes 19:18:35 Median wait time: 2 days, 11 hours, 39 minutes 19:18:35 Number waiting more than 7 days: 20 19:18:47 http://russellbryant.net/openstack-stats/horizon-openreviews.html for horizon 19:18:52 Average wait time: 15 days, 1 hours, 15 minutes 19:18:52 I'm a bit worried that we're letting too many regressions through 19:18:52 Median wait time: 7 days, 4 hours, 15 minutes 19:18:52 Number waiting more than 7 days: 16 19:18:58 Average wait time: 1 days, 15 hours, 16 minutes Median wait time: 0 days, 21 hours, 4 minutes Number waiting more than 7 days: 0 19:19:02 thats Heat 19:19:10 with about the same number of contributors/developers 19:19:23 s/developers/core-reviewers/ 19:19:31 looks that we have very good numbers 19:19:38 so lets keep this ticking along - good stuff 19:19:50 right now 1/3 of reviews are awaiting a review 19:19:58 but I suspect thats fine 19:20:55 we have a discussion going on on the list about -core reviewers and metrics etc; lets keep that on the list. 19:20:59 anything else for reviews? 19:21:13 Ng: oh regressions: better testing/CI 19:21:37 Ng: defense in depth; review carefully of course, but defense in depth. 19:21:56 lifeless: indeed, I'm also a little less worried than I might be because we're not very mature yet 19:22:00 lifeless: elaborate please 19:22:03 "defense in depth" 19:22:15 ++ 19:22:24 shadower: use multiple different layers to achieve your goal 19:22:54 shadower: the goal of 'less regressions' -> review, + more/better CI, + see if we can detangle the architecture to make review simpler... 19:23:13 right 19:23:19 We're pretty shallow at the moment, with mostly reviewers standing in the way, but toci has been fired up again, and we are aiming at gating of course. 19:23:31 ok, 19:23:35 #topic Projects needing releases 19:24:09 dib, tie, occ 19:24:10 I think 19:25:02 tuskar folk - I don't know this yet - can we do releases of tuskar and python-client? 19:25:08 occ has one commit, but it is somewhat important. :) 19:25:10 I know they aren't finished, but 0.0.1 etc will getit out there 19:25:22 ensure that the 'do releases' codepath works 19:25:28 SpamapS: still waiting for that nova patch afaik (baremetal id vs uuid) 19:25:29 pardon my ignorance; what is occ? 19:25:43 os-collect-config 19:25:48 also, is tuskar-ui something we can release 19:25:58 client is fairly complete if we talk about python bindings, CLI is not 19:26:01 specifically, if we cut a release and someone pip installs, will it do something reasonably sane ? 19:26:02 Ng: thanks; didn't recognize it by its acronym 19:26:15 marios: I'm not sure I understand what you mean. 19:26:29 jistr: yeah, we might want completeness for 1.0.0, but not for 19:26:34 'get a release done' 19:26:40 lifeless: w/r/t pip install ... I agree with jistr that client if ok as of python bindings 19:26:45 matty_dubs: occ, oac and orc are the trifecta of configuration leverage :D 19:26:58 tuskar-ui has some major missing parts, which blocks user from building overcloud 19:27:06 ok. it still makes sense to release it that way, e.g. so that tuskar-ui can depend on a released version and not fetch from github 19:27:12 jcoufal: thats ok, release doesn't indicate completeness 19:27:19 SpamapS: https://review.openstack.org/#/c/48453/ 19:27:20 jcoufal: still? What's missing? 19:27:40 jcoufal: release indicates installability, and not worse that the previous release ;) 19:27:48 shadower: I believe still racks/groups creation 19:27:49 SpamapS: tuskar needs this 19:28:01 lifeless: sounds fair 19:28:03 jistr: +1 for depending on released version and not on git 19:28:17 lifeless: it should be releasable as a lib you install alongside horizon and then configure the latter 19:28:21 ok 19:28:40 depends on tuskar and python-tuskarclien obviously 19:28:41 so - how about we try to release all three tuskar things as 0.0.1 today and see what breaks 19:28:41 marios: ok, it confused me that you addressed that comment to me. :) 19:28:50 and file bugs if it doesn't release properly 19:29:09 lifeless: bug driven development? I alwayswanted to try it :-) 19:29:19 pblaho: bugs to track the failure :) 19:29:36 SpamapS: how do you feel if I ask you to do that? 19:29:51 SpamapS: audit * for releases, JFDI it all ? 19:30:04 lifeless: a combination of happy and nonchalence. :) 19:30:06 lifeless: I believe it might help to get stuff synchronized, I would say let's try it 19:30:25 any one 'releases' business? 19:30:26 what jenkins jobs it will require? releasing to pypi / tarball ... docs 19:30:27 SpamapS: oh sorry mate it was meant for lifeless but screen scrolling too fast 19:30:30 lifeless: +1 19:30:48 pblaho: yeah, we may find the first thing is fixing up the infra config 19:30:59 pblaho: in which case SpamapS will file appropriate bugs in the *tuskar* projects 19:31:20 pblaho: I'm asking SpamapS to poke at this as he's done it for most of the tripleo projects - polished knowledge 19:31:33 cool 19:31:37 ok 19:31:40 #topic CD Cloud status 19:32:00 SpamapS: jog0: I'm out of date as of last night ;P 19:32:09 lifeless: we're deploying continuously 19:32:12 woo 19:32:23 we don't report if it failed 19:32:24 so the two cards under 'current MVP' can be moved to done ? 19:32:29 and we don't test if it is even working 19:32:33 but it "finishes" 19:32:51 lifeless: I noted two reviews that should land so we can do it again from scratch, then I think yes 19:33:06 cool 19:33:14 ok so 19:33:22 should we open the next MVP 19:33:40 or do a bit of improving reliability / decreasing fragility first 19:33:41 SpamapS: will it halt at least? 19:34:00 jog0: it won't halt :) 19:34:01 jog0: no 19:34:04 never 19:34:13 when this train explodes, we just send another train 19:34:24 we have lots of trains 19:34:54 I'm inclined to suggest we open the next MVP up - stateful upgrades 19:35:09 because until that's done, it's kindof hard to suggest people /use/ this 19:35:19 and any improvements we make either reliability or otherwise are wasted 19:35:27 thoughts? 19:35:52 I think we should get tempest and warning about failures done first 19:36:02 https://trello.com/b/0jIoMrdo/tripleo btw for everyone 19:36:03 actually right now the deploy is "finishing" but then we are seeing a permission denied ssh'ing in for the last bits of overcloud setup 19:36:16 this is the kanban board for the cd deployed overcloud the tripleo project is running 19:36:19 been working on tempest Ran 2259 (+1472) tests in 2314.086s (+1605.471s) 19:36:19 FAILED (id=3, failures=190 (-33), skips=51) 19:36:35 lifeless: I don't feel that mvp0 is done until we at least have a way to know if the last deploy worked. 19:36:45 derekh: cool! perhaps push it up even as a WIP so folk can run with it ? 19:36:47 SpamapS: ++ 19:37:09 derekh: with the dense collaboration around features, follow-the-sun handoff is likely to be a thing 19:37:11 lifeless: ok will do 19:37:31 jog0: warning about failures ++ 19:37:38 jog0: I am torn about tempest 19:37:41 jog0: though it's right up there 19:38:05 jog0: how about we log a status to another file 19:38:19 jog0: e.g. '*********** deploy-overcloud failed **********' 19:38:25 with a timestamp 19:38:31 how else d o we know our overcloud is fully operational 19:38:34 then we can tail that file (and put it on anonymous http) 19:38:42 we need an IRC bot! 19:38:53 Ng: toci has one 19:39:10 :) 19:39:18 lifeless: I'm tempted to suggest that we shift responsibility to jenkins from the very nice while; true loop.. just so we can observe the CD process. 19:39:32 Ng: https://github.com/openstack-infra/tripleo-ci/blob/master/toci_functions.sh#L67 19:39:46 lifeless: not as a blocker, but as a control for extra little helper things like "write to a status file" 19:40:08 derekh: haha, wow 19:40:18 derekh: wow.. that is really sofisticated bot... nice 19:40:22 SpamapS: so it's probably a weeks work to get zuul in our environment : we need to write something to sync project metadata from openstack 19:40:26 that is the first IRC bot I've seen written in sh :D 19:40:37 SpamapS: plus the easier bits of install and configure zuul 19:40:57 man 19:41:04 just when I thought I had established "we don't need zuul" 19:41:06 you say we need zuul. 19:41:14 SpamapS: you mentioned jenkins 19:41:31 SpamapS: maybe I misunderstood 19:41:38 I'm just saying replace while; true and writing to a status file and irc botting and ... with jenkins. 19:41:46 ah 19:42:30 I don't see it as a time sink but as a "lets not spend our time pimping out our temporary jenkins/zuul shim" 19:42:54 I'd be very worried about keeping jenkins feed and -secured- 19:42:58 fed 19:43:16 yeah thats fair. 19:43:41 if we had it in it's own node, we could respin the image as a meta-loop 19:44:02 I think it's too big a step right now 19:44:05 anyway, I'll table it 19:44:15 shelve it even 19:44:22 so lets not spin here: tempest is in progress 19:44:31 we can do something super simple to capture status 19:45:10 and derekh's toci bot seems brilliant for this too 19:46:05 More code in toci that CD wants: derekh can we move that to incubator? I'm more and more thinking we need a 'feedbackloop' dedicated project, separate from CI, but I may be wrong 19:46:15 synthesis from above: 19:46:30 - stay on MVP1 one until folk can tell easily whether it broke recently or nto 19:46:42 - must have success/fail 19:46:45 lifeless: moving it ok with me 19:46:45 yeah because actually it is breaking 19:46:53 * mordred walks in an sees weird conversation about running jenkins 19:46:59 racing with sshd starting and cloud-init finishing, I think 19:47:04 ok 19:47:05 CI virtualized testing progress 19:47:09 #topic CI virtualized testing progress 19:47:32 pleia2 is going well with the etherpad we started at the sprint 19:47:46 I think we should just drop the lxc spike 19:47:52 yeah 19:48:15 #topic Managing blueprints - migrate them all to tripleo? 19:48:24 so this is a question I have 19:48:36 we said last week we'd consolidate blueprints in /tripleo 19:48:44 so that we don't have to search 7 projects for them 19:49:00 but perhaps folk would rather we just made sure the name always has 'tripleo' in it and do a search ? 19:49:31 lifeless: that has worked for me in the past 19:49:47 I think that would definitely make sense for tuskar-ui 19:49:47 we had juju-* spread out over many projects, and just searched for that 19:50:22 shadower - any thoughts? 19:50:38 lifeless: I'm personally fine either way 19:50:46 +1 for 'tripleo' prefix in blueprints 19:50:50 ok 19:51:03 making sure 'tripleo' is in the name sounds a bit better to me. It will be still easy enough to list the blueprints related to a particular project 19:51:11 so the consequence of this is that I'll rename them all, and any outstanding reviews with a blueprint topic will need their commit message updated 19:51:23 #action lifeless to update all blueprint names 19:51:30 # Tuskar mailing list - what to do with it? 19:51:38 lifeless, does it have to be name or link? 19:51:40 you mean the [tuskar] tag? 19:51:43 #topic Tuskar mailing list - what to do with it? 19:51:44 in the subject 19:51:54 no, I mean the mailing list at launchpad.net/~tuskar 19:52:04 I asked the LP sysadmins to merge ~tuskar into ~tripleo 19:52:09 lifeless: oh. Kill it -- no one ever used it 19:52:12 they can't because there is a mailing list attached to ~tuskar 19:52:14 ok 19:52:26 lifeless: we were just using openstack-dev 19:52:31 #action lifeless to remove ~tuskar list on LP 19:52:40 #topic Reviews and +2 on something you fixed up 19:52:52 so this came up in the first few days of the CD experiment 19:53:03 when say I start something, and say SpamapS fixes it 19:53:25 can he still +2 it, or should he recuse himself? Or can I +2 it? 19:54:03 lifeless: you mean two people working on the same commit (gerrit change)? 19:54:10 can we leave that as a "I'll know it when I see it" based on how significant the second person's fix is? 19:54:33 Ng: I'd be fine with that, but there was a fear of it being always bad 19:54:36 shadower: yes 19:54:45 lifeless: I am no sure right now but I thing I have seen auto +2 when you fix something after another dev... Am I right? 19:54:45 shadower: person A does git review; person B does a review 19:54:50 lifeless: I would say the one who started it can +2 it but someone else needs to approve it 19:55:12 shadower: and then says 'I will action these items to move this along' 19:55:24 pblaho: i think that happens only for trivial rebases, maybe regardless of who has done the rebase 19:55:25 lifeless: how about we can accept one +2 from the union of folks who have worked on it, but someone uninvolved needs to provide the second +2? 19:55:27 or that was just rebases with previous +2 19:55:27 shadower: and does git review -d ; hack hack hack git commit --amend; git review 19:55:43 Ng++ 19:55:49 Ng this can be assured by +2 on approval 19:55:52 Ng: I like that 19:56:10 yea me too 19:56:16 Ng: though s/uninvolved/didn't write code for it/ 19:56:24 lifeless: yes, that's what I meant to imply 19:56:24 since we're all rather involved... 19:56:27 :) 19:56:37 looks like consensus to me 19:56:42 #action lifeless to document 19:56:46 So approver can't be Committer in any of the patchsets 19:56:46 bah, :P 19:57:12 derekh: unless all core have contributed :) 19:57:19 * lifeless spots failure modes 19:57:24 in other words patchset needs +2 from not-author...? 19:57:32 lifeless: well, there is that 19:57:32 * SpamapS nods that consensus is indeed reached 19:57:42 pblaho: yes 19:57:46 #topic open discussion 19:57:48 if all core contributed 19:57:54 ... 19:58:18 then all core should just review and default rules apply 19:58:30 seems like a rare case. 19:58:30 I'm fairly sure it's a power law dropoff in folk that will pickup a single patch 19:58:33 most 1 19:58:34 some 2 19:58:35 very few 3 19:58:36 etc 19:59:16 yeah.. .. from my (short) exp it seams that fixes are usually little things 20:00:29 it might be nice if the original author +2'd the later fixes, without approving, but I don't think I'd want to require that 20:01:01 ok, we're out of time 20:01:04 #endmeeting