19:01:02 #startmeeting infra 19:01:03 Meeting started Tue Feb 3 19:01:02 2015 UTC and is due to finish in 60 minutes. The chair is jeblair. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:01:04 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:01:06 The meeting name has been set to 'infra' 19:01:16 o/ 19:01:23 #link completely unverified agenda https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting 19:01:31 #link previous meeting http://eavesdrop.openstack.org/meetings/infra/2015/infra.2015-01-27-19.06.html 19:01:46 we can have a completely unverified meeting to go along with it 19:01:59 #topic Actions from last meeting 19:01:59 o/ 19:02:10 i'm going to skip zaro's action item and talk about it later... 19:02:13 but project renames! 19:02:14 Morning 19:02:15 were done 19:02:31 yep 19:02:32 woo! 19:02:41 did anyone write an oslo.version patch for governance? 19:02:43 we attic'd those projects something fierce 19:02:50 i did not, but will now 19:02:51 i saw annegentle wrote one for the api projects 19:02:57 fungi: cool, thx 19:03:21 #topic Priority Efforts (Swift logs) 19:03:27 hi 19:03:52 i think the partial console download problem is solved, yeah? 19:03:56 yup 19:04:10 and we're dogfooding a devstack job now 19:04:25 clarkb found and jhesketh fixed a ux problem with the sorting in the index 19:04:41 has the sort change merged yet? I need to review it if now 19:04:44 s/now/not/ 19:04:57 clarkb: https://review.openstack.org/152346 19:04:57 so next steps are to merge that, wait for image updates, check it out, and then maybe roll out to more jobs? 19:05:02 #link https://review.openstack.org/152346 19:05:04 o/ 19:05:07 clarkb: not merged 19:05:26 jeblair: I think possibly remove old school logging from the dg tempest jobs to rely only on the swift logs 19:05:35 jeblair: and that can happen while we roll out to other jobs too 19:05:35 clarkb: ok 19:05:38 +1 19:06:18 cool, anythig else for swifty logs? 19:06:38 Also I'd like to thank clarkb for all his hard work finding bugs, fixing things and rolling out new images as we've been working on it for the last 6ish months! 19:06:44 ++ 19:06:50 (it feels like we're nearing the end!) 19:07:05 yeah! 19:07:24 #topic Priority Efforts (puppet module split) 19:07:24 jhesketh: no problem, you did the hard stuff 19:07:25 Yep, I think we're very close now. Just switching over and discovering edge cases 19:07:33 speaking of the end... 19:07:37 \o/ 19:07:44 this is the last meeting for this topic! 19:07:56 asselin is on vacation, but says: 19:08:03 (and other assets that aren't logs such as tarballs and docs) 19:08:05 "Sprint DONE, fun & successful! " 19:08:07 it was really great to use Storyboard for task assignments, I think it went really smoothly 19:08:16 "Huge thanks to everyone who helped submit patches, review, and prepare. " 19:08:20 pleia2: agreed 19:08:25 and i agree with him 19:08:29 and pleia2 19:08:36 +1 19:08:51 * mordred is sad he missed it - but was really impressed seeing all the stuff happen 19:08:57 ya its huge 19:09:10 i wonder if it would be worth sending an email to the list describing our experiences with virtual sprints 19:09:22 totally 19:09:24 we've had 2 so far and i think they've been great 19:09:30 I can draft something in an etherpad 19:09:30 and lack of need for videoconferencing ;) 19:09:36 we invited midcycles - maybe we can help invent not-midcycles too 19:09:55 the third-party folks had one too for docs; we could ask them how theirs went 19:09:58 ++ 19:10:06 * pleia2 nods 19:10:07 (also, they probably came up with changes for us to review :) 19:10:18 pleia2: that would be cool, thanks! 19:10:22 yeah, they had a tag for their sprint too 19:10:23 they work really well. Probably good to just have everyone agree to focus on a specific task over a couple days 19:10:35 #action pleia2 draft summary email about virtual sprints 19:10:42 and have an organized way to go about it 19:10:50 i really like the #sprint channel 19:10:58 being able to ignore -infra is very helpful 19:11:33 i'd love it if we could figure out a way to make the gerritbot configuration for that easier 19:11:43 jeblair: i'd really like to hear about the virtual sprints so i can contrast it with my mid-cycle experiences 19:11:45 that's the only thing i thought was missing 19:12:15 Agreed. I think some handover of a sprint between timezones is also quite helpful. I was able to help out a lot more on this one thanks to jeblair bringing me up to speed before logging off for the night 19:12:22 dstanek: ++ 19:12:44 jhesketh: good point. it's probably best to try to plan that out too. 19:12:54 Yep 19:12:55 jhesketh: maybe instead of just signup, we should do signup with hours 19:12:57 dstanek: my #1 favorite thing about virtual sprints... no sitting on airplanes 19:13:12 fungi: ++ 19:13:21 and #2, no staying in hotels 19:13:28 so probably have a "helpful tips" section, which includes having handoffs, clearing your schedule (ignore -infra) and using a gerrit topic for all changes 19:13:29 jeblair: what about "gerritbot give me a sprint channel for X hours" - and it either hands you one or makes a new one 19:13:59 gerritbot make me a sandwich^Wsprint 19:14:00 mordred: well we also have the #openstack-sprint channel logged, so that would be coordinated too 19:14:06 yah 19:14:39 it's just that to put gerritbot in there currently means 2 (possibly substantial) changes to project-config 19:14:52 that might just end up being a wishlist item for gerritbot-next-gen 19:15:00 i think if gerritbot ever gets a rewrite, yeah 19:15:19 oh - yeah - I was thinking a gerritbotv2 feature 19:15:23 ah k 19:15:25 register the list of project names associated with a given sprint (maybe as a regex) 19:15:31 for an etherpad-like "give me a channel" thing 19:15:32 anyway, fodder for a spec 19:15:38 so you get #openstack-sprint-aslekn34 19:15:52 or something 19:16:03 * mordred gesticulates wildly, then falls over 19:16:03 mordred: (chanserv registration mumble mumble) 19:16:07 yeah 19:16:13 anyway 19:16:21 in summary for this topic; w00t 19:16:30 #agreed w00t 19:16:39 #topic Priority Efforts (Nodepool DIB) 19:16:58 the changes for rackspace config-drive have gone live 19:17:06 mmmnodes 19:17:07 so I can respin the dib elements related to that 19:17:19 to consume that and do less crazy with xenstore api nova-agent things 19:17:39 i'm still playing with and redesigning my thoughts on getting database configuration moved from image build time to job run time 19:17:41 I am working on figuring out why we build one devstack-precise/devstack-trusty node then upload each of those twice. Trying to replicate locally using fake providers without much luck so far and have some new logging that I will restart nodepool for after the meeting 19:18:10 so we have a few concurrent things going on here. 19:18:24 cool. I also learned some things from notmyname about internals of swiftclient that I think we can use for more efficient uploads for that case 19:18:36 also trying to think about how database setup fits in with more general convergence between teh devstack and bare image types (and how we should skin package caching/installation) 19:18:52 fungi: I am in favor of said convergence 19:18:55 fungi: ++ 19:19:26 also - can haz shade reviews ... the current set are gnarly enough that I don't really want to add the keypair stuff on top until the current set is landed because _Sanity_ 19:19:47 though the database config work is at this point a prerequisite for being able to try running unit tests on debian/jessie (since we'd need to dib that in hpcloud, they don't provide base images for it like rax does) 19:20:00 fungi: ++ 19:20:09 so we're still a few weeks away 19:20:45 o/ 19:21:11 and debian/jessie interest in turn because it has a python 3.4 which is new enough to not have the same bugs as ubuntu trusty 19:21:22 this is looking really promising though. i feel like we've wrapped our heads around quite a few things 19:22:06 it's getting usable enough to fan out in parallel bug fixing/feature adding efforts now 19:22:12 ++ 19:22:22 ya 19:22:26 rather than collectively figuring out how to make it work at all 19:22:44 it's amazing how much freedom the "build base image from scratch " option gives us in problem solving 19:22:47 yup and other than the duplicate uploads I haven't really seen any issues 19:22:56 we missed one item in an element for a while that jogo fixed 19:23:06 mordred: its so much faster too 19:23:24 once it's done, I think we need to take a pass through our elements and our puppet and probably refactor some stuff 19:23:33 absolutely 19:23:37 because it's a bit rube-goldberg right now - but it's good enough to not touch for the moment 19:23:45 and i expect fungi's work on image consolidation will help prep us for that 19:24:02 at the moment it's more of a facsimile of our snapshot-based build process 19:24:10 translated to dib 19:24:29 so lots of opportuities for simplification and decruftifying 19:24:37 indeed 19:24:50 opportunities too 19:24:54 also its probably worth noting that if anyone wants me to walk them through building a local image I am happy to do that 19:25:07 its super easy at this point and is potentially useful for testing security bugs and the like 19:25:16 though perhaps an opportuity is where opportunity and tuits meet 19:25:28 and... 19:25:29 #topic Priority Efforts (Jobs on trusty) 19:25:43 i think we can probably stop covering this one after this week 19:25:53 we are knocking them out! :) 19:25:54 clarkb: i'd like to learn. any chance you could screencast it? 19:26:06 at least until it comes time to eol icehouse, but that's months out 19:26:08 dstanek: ya I can probably typescript record it 19:26:11 dstanek: that's a great idea 19:26:14 clarkb: ooh, ooh, or live screencast it! 19:26:18 ++ 19:26:31 i still have a handful of cleanup patches to rip out our py3k special snowflakeness 19:26:54 fungi: are they up for review? 19:26:56 #link https://review.openstack.org/151715 19:27:03 (why, yes!) 19:27:08 #link https://review.openstack.org/152215 19:27:21 #link https://review.openstack.org/152217 19:27:27 i think they need to merge in that order 19:27:36 mordred just approved the one which needed to merge before them 19:28:01 so should be safe to lift my -2 on 151715 soon 19:28:10 clarkb: i'd like that typescript as well 19:28:29 zuul cross-repo dependencies would have been _very_ useful for this series 19:28:46 though i'm thrilled that's almost a thing now 19:28:50 (zuul crd changes are up for review) 19:29:03 yep, been reviewing. can't wait 19:29:17 anyway, that's all i have for precise->trusty migration 19:29:26 #topic Priority Efforts 19:29:52 it would be great of folks could review the changes linked above 19:29:58 the meeting log will call them out easily 19:30:11 and next week, i think we are going to have some openings for new priority efforts 19:30:25 ooh 19:30:26 so we can probably nominate some and discuss them then 19:30:33 that'll be great 19:30:45 * clarkb puts an early vote in for zanata 19:31:04 +1 19:31:12 I want to use zanata with groups portal 19:31:14 so +1 19:31:19 yeah, that seems a shoe in :) 19:31:25 yeah, mrmartin and I talked about it some while at fosdem (hi mrmartin!) 19:31:31 we could have cross-repo dependencies between zanata and groups changes ;) 19:31:38 fungi: nicely done 19:31:40 hi, yeah, I can help in this zanata story 19:31:47 * mordred hands fungi a lovely goat 19:31:53 lovely, lovely 19:31:54 #topic Upgrading Gerrit (zaro) 19:32:02 zaro: what's new? 19:32:11 so i've tested the WIP plugin. 19:32:47 it's not looking good for 2.9.x. I've entered a few issues into the gerrit issue tracker for David O. to look at. he's already fixed some. 19:32:56 also zaro wrote up a plan for moving from precise to trusty based on when he migrated review-dev last week 19:33:00 #link https://etherpad.openstack.org/p/review-to-trusty 19:33:10 but it requires changes in gerrit core which is only available on master 19:33:11 we should think about when we want to do that 19:33:37 ohh yes. review-dev is already on trusty 19:33:50 nice 19:34:00 that would be the 1st step towards upgrade actually. 19:34:12 zaro: were there any issues with trusty? or was it pretty straightforward? 19:34:16 i haven't torn down the old review-dev server yet, but will soon if nobody needs anything from it 19:34:51 zaro: that looks like an os upgrade combined with a gerrit upgrade, is that correct? 19:34:52 No issues. Just required new libs. 19:35:04 yes, that is correct. 19:35:25 cool, i think that's probably fine 19:35:26 fungi: I don't need anything fwiw 19:35:32 these are the new libs i'm refering to : https://review.openstack.org/#/c/151368/ 19:35:57 * jeblair looks at https://wiki.openstack.org/wiki/Kilo_Release_Schedule 19:36:08 i guess we could consider refactoring the work into two steps (move to trusty, upgrade to 2.9) with breathing room between to iron out new bugs we find 19:36:10 let's not do it this week. :) 19:36:11 been testing gerrit with zuul as well. looking good so far. 19:36:33 jeblair: NOW. DO IT NOW!!! 19:36:34 been debugging zuul-dev, it's almost fixed. zuul-merger seems to be having issues cloning from repo 19:36:36 but yeah, we're getting close to busytime for the release cycle 19:36:53 it's something to do with the ssh host key 19:37:21 zaro: if it's the error message you pasted earlier, it's the client public key needing to get loaded into gerrit 19:37:31 not the host key 19:37:47 fungi: yah, i think you are correct. jsut haven't had time to do it. 19:38:14 this weekend is probably too short notice. people we like will yell at us if we do it next weekend. 19:38:34 jeblair: there are people we like? 19:38:51 i have travel for several weekends after that so probably won't be able to help during 19:39:14 * fungi double-checks calendar thingy 19:39:48 weekend after next I'm booked - but other than that I'm actually back to the world of being able to help people with things 19:39:51 yeah, i'm around for the next two weekends, then otherwise occupied for the three following 19:40:34 but if i'm not around, i'm not around. feel free to do it without me 19:40:46 I'm busy this weekend and the next, then my schedule is quite clear for a while 19:41:03 there's always another gerrit upgrade on the horizon, after all ;) 19:41:17 okay, let's take this offline for now 19:41:42 #topic (yamamoto) Portability question 19:41:47 and maybe consider not weekend but other off hours? 19:42:11 " I'm not sure if I can attend the meeting, sorry. I added this here as it's suggested that this needs wider audience." 19:42:12 I have two concerns with the portability patches 19:42:17 " It's easier for me to be able to run "tox" tests on non-linux environments. " 19:42:27 "On the other hand, the gain of portability to platforms that OpenStack does not support is questionable." 19:42:32 "My suggestions " 19:42:37 " Do not refrain from using non-portable features when necessary" 19:42:40 " Accept best-effort changes to make it portable, unless they have significant drawbacks" 19:42:46 (end of quote) 19:42:54 #link https://review.openstack.org/#/c/150264/ 19:43:12 is this specific to the bash->sh portability changes? 19:43:12 my major concern initially was there was no accompanying info with why the changes were more protable, but that appears to have been fixed in newer patchsets 19:43:14 k. that actually addresses one of my concerns - which is that _like_ using bash and dont' think that bourne shell compat leads to nice scripts 19:43:15 #link https://review.openstack.org/#/q/owner:zhaoxinyu%2540huawei.com+status:open,n,z 19:43:18 fungi: and find args and so on 19:43:21 k 19:43:39 mordred: that and bash is usually bash when you invoke bash. sh on the other hand is an non portable mess :) 19:43:39 my other being that we don't have non-linux machiens, so I fear these would bitrot almost immediately 19:43:43 clarkb: ++ 19:44:02 er, wrong link there, sorry 19:44:03 freebsd supports bash 19:44:06 that said - I don't have a problem with the nature of most of the changes (other than s,/bin/bash,/bin/sh,) on their merits 19:44:13 so its actually hard to know whether or not the sh you write is properly portable 19:44:14 and OSX too 19:44:33 #link https://review.openstack.org/#/q/status:open+project:openstack-infra/project-config+topic:portability,n,z 19:44:38 this has come up with devstack too, zsh fixes for example 19:44:59 my thinking is if the change is minimal then whatever makes peoples life easier 19:45:27 tweaking devstack to run under zsh is a little more dubious i think 19:45:28 but if it's worse code, then no. for example replacing ${!expand-me} with tricks involving execs etc 19:45:31 clarkb: well, proper sh could be enforced with bashate 19:45:39 ianw: ++ 19:45:42 i mean, what platforms are going to have zsh but no bash? 19:45:49 jesusaurus: bashate doesn't do that 19:46:04 we could always just rewrite all of our shell in perl ... 19:46:07 bash8 --posix 19:46:33 https://review.openstack.org/#/c/150265/2/tools/run-layout.sh 19:46:37 fungi: also, we don't really support zsh in devstack, we've let compat patches in for the openrc file so you can source openrc from zsh 19:46:38 i'm not certain that is not worse ^ 19:46:39 sdague: not currently, but it is already opinionated, so it would become sh-opinionated 19:46:45 s/would/could 19:46:52 sdague: okay, that makes slightly more sense 19:47:05 jesusaurus: -2, it's not the purpose of the tool 19:47:13 so ya I think keep bash but find/mkdtemp etc changes are fine 19:47:27 (the find | xargs is the pattern I usually use so it makes sense to me with slightly less thinking) 19:47:36 though I am super biased to gnu tools after living on solaris I don't mind if other people are making the fixes 19:48:11 you also only get arrays in bash with bash4, which mac does not have 19:48:21 which at least a few tools use 19:48:32 sdague: indeed, the translation jobs do 19:48:34 so, i can sort of see the case that developers hacking on patches for this stuff want to be able to test their proposed changes locally with minimal fuss, and maybe installing bash is effort, but still curious what platforms it's a burden on 19:49:17 "non-linux environments" is a little unspecific 19:49:35 there's nothing linux-specific about bash after all 19:50:03 without regular contributions from folks on non-linux platforms and testing, i don't think we can make any guarantees about this 19:50:14 and "portability" is a slippery slope. i don't want to be reviewing 100 one-line changes to shebang lines and what have you 19:50:37 i do agree that portability patches are generally non-harmful, but we have way too many changes already, and i don't want to review them 19:51:11 These are tests run in the gate - that's where they have to work. 19:51:13 since we will not be able to prevent non-portable things from sneaking in, we would probably be continually receiving patches to fix them 19:51:21 The contributor wanted to run locally a tox -e XXX. 19:51:43 That's something that should work in general - but I agree that we should use some sane defaults. 19:52:04 ... or requirements. 19:52:20 so who is interested in supporting this effort by reviewing these changes? 19:52:35 I could see a general rule that things called by tox could want a /bin/sh shebang line 19:52:38 (and future ones) 19:52:40 since that's a specific bounded usecase 19:53:06 mordred: that just means you can run the script until the next error 19:53:06 mordred: but can it assume a gnu toolchain? 19:53:11 i am fine reviewing changes such as "this script uses bashisms but starts with a /bin/sh shebang line" 19:53:18 but broadly across all the test of our bash- I think it's a no-go 19:53:23 i consider that a bug 19:53:27 fungi: makes sense 19:53:36 assuming "sh" is bash is an obvious portability problem 19:53:44 fungi, agreed 19:53:45 declaring "bash" is not 19:54:19 mordred: I would support an exception for tox called scripts - but right now I think it's not worth the effort. 19:54:31 This will bitrot soon again. 19:55:00 so i think it's worth giving the feedback that there is not a lot of interest from core reviewers to maintain compatability in the scripts themselves 19:55:13 its also worth noting that none of the software this tests is intended to run on OS X for example. We don't use portable python event polling in all places 19:55:21 so it sounds mostly unanimous that without a clear mandate to support running our scripts under constrained environments, we should probably set expectations that we won't be reviewing that sort of improvement 19:55:21 clarkb: true 19:55:30 and so on, so I think that we may have already asserted you need a linux install 19:55:33 yeah, agreed 19:55:53 okay, a few more mins for mrmartin 19:55:56 #topic Askbot migration (mrmartin) 19:56:11 ok, I just like to get a little care regarding this askbot migration 19:56:33 on that note, i should order the replacement https cert for that today. it expires in a few weeks now 19:56:46 fungi: ++ 19:56:56 so I like to ask somebody to dedicate himself to review the patches, but the larges part is that we need to create the new instance 19:57:11 mrmartin: we probably want to land all the patches first 19:57:28 i can iterate on trying to boot the instance once the patches have made it through sufficient review (preferably approval) 19:57:28 once that's done, hopefully spinning up the new instance will be easy 19:57:32 jeblair: right, it was there since december, and was splited up last week 19:57:43 making this a priority effort is probably a good way to stay on top of it 19:57:58 mrmartin: there are some jenkins -1 votes on those 19:58:16 so after instance creation, we need to backup the existing site and restore in new instance 19:58:26 jeblair: yes, due to a puppet-module dependency 19:58:34 need to approve the puppet-solr module first 19:58:52 sounds like a job for the new depends-on code 19:59:02 mrmartin: i can definitely dump/import the data as long as i know where it goes and what should be run to load it 19:59:33 mrmartin: so maybe some minimal documentation around that would be helpful if you haven't written such a thing already 19:59:33 it does sound liek this might want to join zanata as a priority effort next week 19:59:33 so, as it is a high traffic site, I like to see a silent pilot period before we move the dns zones 19:59:44 #link https://review.openstack.org/140043 19:59:50 #link https://review.openstack.org/151912 19:59:51 mrmartin: that seems reasonable 19:59:57 #link https://review.openstack.org/151910 20:00:15 fungi: right, I can write a spec for this similar to module split out 20:00:18 fyi, I'll be mostly out Friday and Monday for some appointments and things (yay for being home) 20:00:19 i think the reviews for that need to happen bottom-up ^ 20:00:27 mrmartin: we can put it in dns under an alternative name if you want, or just expect testers to modify /etc/hosts 20:00:42 I think hosts file modification is more then enough 20:00:44 we're at time. thanks everyone! 20:00:49 #endmeeting