21:00:15 #startmeeting swift 21:00:16 Meeting started Wed Nov 11 21:00:15 2015 UTC and is due to finish in 60 minutes. The chair is notmyname. Information about MeetBot at http://wiki.debian.org/MeetBot. 21:00:17 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 21:00:19 hello, everyone 21:00:22 The meeting name has been set to 'swift' 21:00:27 who's here for the swift meeting? 21:00:28 o/ 21:00:28 |o| 21:00:28 o/ 21:00:30 hi 21:00:31 evening 21:00:31 hello 21:00:32 o/ 21:00:38 heyoh! 21:00:38 ䷣ 21:00:56 did I see acoles join? 21:00:58 * briancline collapses 21:01:01 yo 21:01:04 i am 21:01:25 all right 21:01:35 #link https://wiki.openstack.org/wiki/Meetings/Swift 21:01:42 here 21:01:52 acoles! 21:01:54 several smaller things to go over this week, and maybe a couple of bit things 21:01:54 acoles: thought you were hanging someone? 21:02:06 peluse: ran out of rope 21:02:10 lol 21:02:12 heh 21:02:45 #topic PyECLib update 21:02:54 going from the top... 21:03:23 tsg and clayg and I have been going over the pyeclib changes, with help from lifeless in -infra for global-requirements changes 21:03:30 so here's the status 21:03:34 (and why you care) 21:03:52 there are a few goals 21:03:56 hint: you don't 21:04:00 aww 21:04:09 first is to use later version of pyeclib 21:04:22 second is to drop any hard bundling with jerasure 21:04:29 new bugs better than old bugs - always 21:04:38 awkward late hello 21:04:45 jerasure sux INAL 21:04:49 third is to set it up to move into the openstack namespace so we have more of a community around landing patches 21:05:00 openstack all the things! 21:05:08 openstack this! 21:05:36 meh, in or out of the openstack namespace, it's good to have more people commit access and able to cut a release 21:05:50 and openstack is a convenient place with the logistics already set up 21:05:58 agreed for sure 21:06:16 currently we're pinned to 1.0.7 21:06:33 and we've got a patch (landed yesterday) on master that lets us support >=1.0.7 21:06:51 that's nice. 21:06:55 oh that landed!? 21:07:02 so global-requirements has been bumped to >=1.0.7. next up is for us to land that to mirror g-r 21:07:08 clayg: yeah, I think this morning 21:07:21 tsg: mentioned it an hour ago or so 21:07:59 so once we have the >=1.0.7 patch landed, we'll bump g-r to be >=1.1.1. then we'll bump swift to be >=1.1.1 and also drop the jerasure references 21:08:31 notmyname: I don't think anything else goes in the >=1.0.7 change - we could start doc fixes in saio and example confs to remove jerasure references (asuming you're more on the > side of the >=1.0.7) 21:08:33 the reason for the dance is because of how dependencies are tested in the gate and that we have to have our requirements file be an exact string match for global requirements 21:08:42 basically https://lh3.ggpht.com/QL094vQpTcJHIxF5qVeURS-0-KpLq_ciDUzHnlYlOQ9GSO_Q98AO7b731jFG=w300 21:08:48 clayg: yes 21:08:48 ^ this the part you don't care about 21:08:51 tsg: basically 21:08:56 torgomatic_: basically 21:08:57 if you try to care you'll end up punching walls 21:09:12 it's true. clayg did that yesterday when he started to care about it ;-) 21:09:20 who won? 21:09:35 the boss man who said go get in a quieter room ;-) 21:09:50 good times. 21:10:07 torgomatic_: it's more like this -> http://static.fjcdn.com/gifs/Rude_05b797_2726116.gif 21:10:14 and that's how dependencies get updated 21:10:32 so I thinkw e should be able to get this done today or tomorrow 21:10:42 yay! 21:10:55 notmyname: well someone needs to do the saio doc updates and scour the rest of the code for jerasure 21:10:55 for everyone working on it, thanks for your patience 21:11:03 at a minimum we should get it claned out in the >=1.1.1 patch 21:11:09 rock on... saio docs need to include how to instal isa-l also.... 21:11:09 right. I think that's in tsg's 1.1.1 patch 21:11:10 *cleaned 21:11:12 is tsg doing that? 21:11:14 oh! 21:11:16 tsg: is a boss 21:11:22 LOL 21:11:30 hopefully no one breaks neutron again! 21:11:43 (today or tomorrow that is :)) 21:11:46 clayg: go check this one https://review.openstack.org/#/c/244287/ 21:12:02 tsg: where's your >=1.1.1 patch with the doc updates? 21:12:05 and test updates? 21:12:55 notmyname: let me get it up there later today - hadn't posted it yet (with the slight change of plans yesterday) 21:13:00 ok 21:13:12 any more questions on pyeclib before we move on? 21:13:27 notmyname: the test updates won't be needed anymore after clayg's recent patch 21:13:34 right 21:13:48 notmyname: and we are keeping jerasure in the supported list, correct? 21:13:50 jsut dropping jerasure from docs and unit/__init__.py 21:13:54 notmyname: ok 21:13:56 will do 21:14:06 I will ;) 21:14:28 tsg: drop down to just liberasure. later we can readd jerasure and add isa-l 21:14:33 if we want 21:14:34 tsg: I think when we say requires pyeclib>=1.1.1 we remove jerasure from the prefer list - a list of one is a good list 21:14:41 ya 21:14:52 clayg: ok :) 21:15:01 ok, moving on (lots to cover) 21:15:09 #topic ring change 21:15:13 whoa - notmyname is on a mission 21:15:16 clayg has a good patch up 21:15:21 patch 241571 21:15:22 notmyname: https://review.openstack.org/#/c/241571/ - Put part-replicas where they go 21:15:32 go review it and run your prod rings through ti! 21:15:41 yeah! multiple part-replica placement on same device is bullshit! 21:15:43 clayg: what do we need to do on it? 21:15:53 what do we look for? 21:15:53 * clayg *shrugs* 21:15:57 * clayg *shrugs* 21:16:28 clayg: you've still got a -2 on it 21:16:41 notmyname: oh yeah i need to write words in the ring_overview 21:16:58 oh yeah. it's on my list for tonight to run our latest ring through it 21:17:04 I mean I don't *need* to - those words won't close the bug any faster - but I will cause folks thought that would be good 21:17:21 the first reason the ring stuff was updated was because sometimes you'd get more than one part placed in the exact same drive. that means that replicas lie about your durability and EC just flat-out breaks 21:17:36 so clayg dived in to fix it and came up with this patch 21:17:54 notmyname: i fixed everything [1] else while i was in there! 21:17:56 which is a big change, but AFAIK much more tested and deterministic 21:17:57 1. not everything 21:18:32 so the first concern is that it doesn't make all your existing clusters shuffle data everywhere. that would be bad 21:19:05 secondly, it needs to have a good plan going forward. ie is it supportable/tested code that makes sense 21:19:11 notmyname: yeah - it's acctually pretty agressive about fixing things - so if you give it an old ring from before overload that's all imbalanced it'll just move 50% of your parts around and be like "what?" 21:19:41 by "fixing" things you mean fixing the duplicate assignments 21:19:59 ? 21:20:09 well either duplicates or like "bad" placements - with the replica_plan stuff you can dial into imbaance and dispersion if you can't have both 21:20:19 so I call "fixed" meaning how successfully it filled the replica_plan 21:20:25 ok 21:20:32 the default replica_plan is "as dispersable as balanced" 21:20:40 then you add a bit of overload when things look stupid 21:20:54 or a bunch of overload if you're like super into 507's 21:21:04 the algo doesn't really care anymore 21:21:11 it says there's two poles you can fit anywhere in the middle 21:21:12 dfg: nadeem: it would be great if you could look at the ring patch and put your rings through it 21:21:40 notmyname: ya we're planning on it 21:21:44 notmyname: sure 21:21:46 dfg: nadeem: I think all you'll get out of it is that it'll be like 10-15% slower - you're welcome 21:21:53 clayg: you did a great job explaining all that in tokyo. do you have those slides linked somewhere? 21:22:05 clayg: hey- i'm not the one who does that 21:22:35 https://docs.google.com/a/swiftstack.com/presentation/d/1HdTyMAjpauoCNpwBMyNmv_11bimZScvpRTxI-e0Gmw4/edit?usp=sharing 21:22:48 but like... I'm not sure it's as good with out my commentary 21:22:53 i'm like *pretty* funny 21:23:13 can other people see that link? 21:23:21 clayg: you are probably going to get a bunch of requests 21:23:36 notmyname: i don't know how to computer? 21:23:42 i sent it now :-) 21:24:03 I can work with clayg to get it publicly sharable 21:24:19 between all of us. we'll figure that out :p 21:24:26 :) 21:24:32 please make sure to add clayg's commentaries ;) 21:24:46 for everyone, please take a look at the patch. it's pretty important and promises a better way to do data placement in the cluster. and will take a while to review 21:25:11 ok, moving on 21:25:19 #topic py3 patches 21:25:38 there is now a Plan for the py3 port 21:26:09 it's nothing more complicated than (1) get a voting gate job and (2) add modules to it until we have everything tested on it 21:26:34 so It hink that means we'll have a steady stream of "port XXX to py3" patches for a long time 21:26:58 which is why someone suggested "hey, can we just agree that we can approve a py3 change with only one core +2?" 21:27:03 so what do you think? 21:28:07 is anyone uncomfortable with that? acoles clayg peluse dfg torgomatic_ tdasilva? 21:28:14 eh, I won't do it 21:28:21 seems reasonable.. the gates are more helpful than usual for this sort of thing 21:28:22 won't do what? py3 reviews? 21:28:35 I've never really ported anything to py3 before, so I won't solo-approve a py3 patch because I lack sufficient confidence to do so 21:28:37 wait, py what? ok, yeah sure 21:28:59 yeah, i'm not a big fan either (of one +2 +A) 21:29:08 seriously though, I don't understand. why propose just one +2? 21:29:17 notmyname: idk i'm not sure what these patches are going to look like 21:29:33 notmyname: maybe after the first N I/we might feel differently 21:29:38 the thought was only to speed up the process 21:29:56 and it sounds like current thinking is that we should keep with 2 +2s 21:30:05 I'm completely fine with that :-) 21:30:17 +1 21:30:17 oh geeze, I think there's probably a better way to speed it up. maybe assign some volunteers to review them. i dont think cutting to one +2 sounds right 21:30:22 #agreed py3 patches aren't special. still needs 2 cores to approve 21:30:23 i'm with torgomatic_ 21:30:35 kota, double meaning? heh 21:30:51 ok, moving on 21:31:02 peluse: exactly, I'm with keep two +2 21:31:09 #topic searchlight integration / oslo.messaging 21:31:15 sjmc7: this is a topic you added to the agenda 21:31:26 hey. this one’s under my name ( i work on searchlight) but after a conversation with briancline yesterday 21:31:47 and as a followup to the presentation the IBM guys did at the summit 21:32:04 the short version - to integrate swift with SL we need notifications and oslo messaging is how we’ve been doing it for other services 21:32:40 there’s a patch up on swift for zaqar based notifications - would there be any objection to an optional oslo.messaging notification middleware? 21:32:51 our alternative is something out of tree deployers could install 21:32:51 searchlight == metadata search ?? 21:33:09 ah, yes, sorry 21:33:24 yes, to back up a little - we’d love to index swift metadata for search 21:33:51 sjmc7: with any dependency addition there's the question of what it also brings in transitively. and many oslo projects also tend to bring in others like config, logging, etc. and since we don't currently use those, it's been a barrier to "just drop it in" 21:34:08 but that being said, I'm find with evaluating it like any other dependency addition 21:34:16 yeah, this solves a few concerns i think we had from ibm/softlayer's perspective as well. my two biggest concerns are the auth issues (some mentioned in the review i think) and some scale concerns 21:34:34 yeah, zaqar is nice because it's just like "POST here; you're done" 21:34:37 yes, i can certainly understand dependency concerns 21:35:20 if I understand, oslo.messaging drags in AMQP, so now you need a messaging broker and you need to maintain the broker and blah blah blah 21:35:26 although keystonemiddleware already brings in most of oslo 21:35:28 it's like "surprise! more infrastructure!" 21:35:49 it's pluggable -- amqp, zeromq, and i believe whatever backends kombu supports 21:35:53 yes - we’re not suggesting this would be enabled by default 21:36:01 Is there any advantage to adding the middleware into Swift rather than just having it be it's own thing in Github somewhere? 21:36:08 sjmc7: sure. and like keystone, it would be an optional dependency to bring in. ie not something every deployer must suddenly install to upgrade 21:36:14 right 21:36:16 torgomatic_: to be fair, zaqar is also a bit of infrastructure as well 21:36:21 briancline: true 21:36:25 hurricanerix - that’s an option, certainly 21:36:28 fyi, there's work ongoing to use oslo.messaging to send notifications to zaqar 21:36:39 orly? 21:36:44 the advantage of in-tree is that if the middleware structure changes it doesn’t get ignored 21:36:51 briancline right. no rpc, no rabbitmq 21:36:57 hurricanerix: not sure. but there's a lot of people doing metadata search, so having a common way upstream to feed that isn't a terrible idea 21:37:32 just notifications 21:37:36 yeah, seems a lot of folks have been interested in the search use case for a few years 21:38:00 there's also the use case of user-facing consumable notifications, which is what zaqar primarily serves 21:38:26 briancline: yeah, and there's some pretty interesting use cases there too 21:38:35 there are other conversations about translating oslo.messaigng notifications into zaqar queues 21:38:40 horizon and heat are both interested in that 21:38:49 notmyname: true, but it seems easier to let it be its own thing first, let it mature some and then if tons of people are using it, then look to add it in. (just my opinion though) 21:38:50 there's a thing I wrote that might indirectly help with search; let me go find it 21:38:52 many of us have wanted some sort of notification/message thing in swift for a while 21:39:14 here we go https://review.openstack.org/212824 21:39:38 what was the other feature discussed in vancouver that could benefit from this? was it tiering? or something else? 21:39:57 that would let you find unindexed objects in your cluster and go index them 21:40:10 or, you know, whatever. that's kind of the point. 21:40:11 briancline: tiering. container sync. utilization processing. probably others 21:40:47 torgomatic_: yeah, probably both would be needed, right? 21:40:48 torgomatic_ briancline : if there's any interest in oslo.messaging for just notications and not using amqp/rabbit, you can ping us on #openstack-oslo 21:41:22 notmyname: definitely. but just using a notifier on new PUTs misses a couple things 21:41:27 right 21:41:32 (a) if the other end is down, you miss objects 21:41:55 (b) you don't get notifications for existing objects (like if you add indexing to a cluster with data already in it) 21:42:01 a) is why using oslo is desirable 21:42:01 hence the watcher thingy 21:42:16 so the original question is "are we ok with using oslo.messaging or not" 21:42:19 dims: excellent. i'll pop in soon 21:42:58 personally oslo anything makes me nervous =) 21:43:17 hurricanerix you should get to know us :) we don't bite 21:43:32 we’re ok with this living out of tree, if that’s preferable 21:43:44 to be fair, using swift with keystone pulls in a lot of oslo deps anyway 21:44:26 sjmc7: it's a little hard for me to say at this point just because I'm not familiar with oslo.messaging 21:44:42 ok. well, perhaps we’ll do something out of tree for now, and revisit this in the future 21:45:24 in general, I'm supportive of having basic hooks/functionality for people to build metadata searching for the stuff in swift 21:45:44 i think the middleware would be very simple, so in that sense the hooks exist 21:46:00 do you have an example we could look at? 21:46:09 for middleware? 21:46:13 notmyname i'm not that familiar with how all the dependency stuff works. does having the middleware included mean that the middleware's deps get pulled in, even if you don't use it? or is there a way to have deps on a per middleware basis? 21:46:58 it would be nice if it were done on a per middleware basis, so if all you had was the most basic Swift install, you wouldn 21:47:03 hurricanerix: dependencies are set for the whole project. there isn't currently a way to do dependencies for just one piece of middleware or optional stuff 21:47:06 't need to install a bunch of extra stuff. 21:47:08 sjmc7: yeah 21:47:22 notmyname: yeah, that what I thought 21:47:26 hurricanerix: right. so eg with keystone today, we have keystone middleware, but we don't have keystone in our requiremnts 21:47:28 every project emits notifications through different mechanisms 21:47:41 i'd imagine it could work the same way that the optional keystoneauth middleware works now -- swift has that in-tree, but doesn't have hard dependencies on the keystonemiddleware package 21:47:46 what john said 21:47:52 but all through oslo.messaging, so i don’t have an example that’d be useful, but looking at existing swift MW it looks like it’d be quite simple 21:47:59 and depending on the dependency I think there's a good point to be made that it doesn't need to be in the upstream tree 21:48:08 minimize dependencies is a thing 21:48:10 clayg: right 21:48:16 and middleware packaged outside of swift is a thing too 21:48:22 ok, we can live with that. thanks for the discussion 21:48:50 sjmc7: thanks for bringing it up. I've very interested, personally, in searchlight, so please let us know what's going on there 21:48:56 if it was so *needed* and trivial to do right in middleware there'd already be dozens of these things implemented outside of swift and we could just pick the one we like the best to bring onto stackfordge or w/e 21:49:08 gotta move on. 2 more things to bring up 21:49:19 #topic -infra is dropping py26 testing 21:49:23 :( 21:49:30 I just discovered this today on the ML in another thread 21:50:00 its only been EOL for 2 years ;) 21:50:15 yeah, but centos 6 users are very stubborn 21:50:23 Re: [openstack-dev] [oslo][all] Oslo libraries dropping python 2.6 21:50:23 compatability" 21:50:24 topic is " 21:50:39 * dims waves again :) 21:50:41 so yeah, we still support py26 in swiftclient 21:51:00 that will get harder to do if we arent' running py26 tests 21:51:10 oh, the -infra date in nov 19. so really soon 21:51:40 dropping py26 from the gate is the same as not supporting py26 in practice 21:51:43 so that raises the question of continuing py26 support and if so how to test it 21:51:51 clayg: right 21:51:55 no one will bother to checkout changes and tox -e py26 them 21:51:59 we could do some 3rd party CI 21:52:07 notmyname: oh - that'd be fine 21:52:11 but 3rd party CI doesnt' vote 21:52:29 notmyname: we could probably socialize it and work around bugs pretty quick if it slipped? 21:52:40 unfortunately, I have zero idea how much py26 swiftclient is used 21:52:43 notmyname: anywa - i've been pretty happy with no py2.6 in swift 21:52:57 notmyname: some folks have said that "well it matters more in the client" - but... no it doesn't 21:53:00 yeah, but no pu26 in swift is very different to no py26 for clients 21:53:05 heh 21:53:07 heh 21:53:12 joeljwright: HOW SO!? ;) 21:53:25 there are loads of people using rhel6/centos6/scientific6 21:53:28 joeljwright: do you have some client machines that don't have a good way to get py2.6 21:53:29 all on my26 21:53:33 py26 21:53:52 rhel5 is probably py2.6 - but rhel6 - really? 21:53:57 yep 21:54:00 I don't know about 'good' 21:54:15 the issue is all these systems are supported for years more 21:54:18 we are also still shipping RHGS in rhel6.5 21:54:30 yep 21:54:32 tdasilva: RHGS? 21:54:36 and this is why we can't have nice things 21:54:38 Red Hat Gluster Storage 21:54:45 ah 21:54:51 which ships with swift packages 21:54:59 Python 2.7 was released on July 3rd, 2010 21:55:20 clayg: RHEL6 is supported until something like 2035 21:55:25 i say that like it was a long time ago but I don't really remember what year it is - 2016? do i have to vote next month? 21:55:33 clayg: python 3.0 was released in 2008, but it's still not default... 21:55:33 notmyname: 2020 -- https://en.wikipedia.org/wiki/CentOS#End-of-support_schedule 21:55:57 so the reality is that someone that needs py26 needs to think very hard about some 3rd party py26 tests 21:56:27 clayg: "you didn't blow away your entire deployment and install everything fresh from the very latest version?! wow." 21:56:50 why is rhel6 py2.6 21:57:14 this week, especially joeljwright and tdasilva, please let's think on what we can do there. and raise it again next week 21:57:18 clayg: I honestly don't know :( 21:57:20 (look at the time) 21:57:25 kk 21:57:27 k 21:57:27 last thing, briefly 21:57:39 #topic functional tests with testrepository 21:57:49 patch 214206 21:57:50 notmyname: https://review.openstack.org/#/c/214206/ - Modify functional tests to use testr 21:57:50 well I don't think we can support py2.6 for another *5* years? 21:58:12 moves just functional tests to use testr. this is important for defcore, but it might also get us other cool stuff too 21:58:18 I already package py2.7 for rhel - so does epel and ius I'm sure 21:58:19 clayg: honestly, I'm not sure I want us to try :( 21:58:38 please look over the patch and understand the small changes you'll see when you run functional tests 21:58:44 notmyname: skeptical of the other cool stuff 21:58:45 if it might get us other cool stuff, I might +2 it ;) 21:59:06 clayg: recheck just failing tests, run a test until failure (eg for timing bugs), run tests concurrently 21:59:09 yawn - if it passes the gate and still runs my tests I'll adust - w/e 21:59:17 heh, ok 21:59:21 clayg: I think the "cool stuff" is not having test duplication between Swift repo and Tempest. =) 21:59:36 +1 21:59:40 notmyname: idk, fancy stuff like that scares me - back in my day we used to run our tests by rubbing two asserts together with a stick 21:59:42 hurricanerix: you'll be shocked to learn clayg doesn't really care about tempest ;-) 21:59:58 clayg: lol 22:00:01 hurricanerix: sorry not sorry 22:00:01 notmyname: lol 22:00:08 clayg: sorry, not trying to put words in your mouth 22:00:13 unfortunately, we're out of time 22:00:21 thanks for helping move everything along quickly this week 22:00:36 clayg: well I don't either, I see this as allowing me to continue to not care about it :) 22:00:49 hurricanerix: nice! 22:00:54 looks like we have follow-ups about py26, maybe oslo.messaging at a later time 22:01:07 thanks for working on swift! 22:01:09 #endmeeting