Wednesday, 2019-01-16

*** yamamoto has joined #openstack-meeting00:01
*** a-pugachev has quit IRC00:16
*** yamamoto has quit IRC00:22
*** yamamoto has joined #openstack-meeting00:31
*** yamamoto has quit IRC00:33
*** yamamoto has joined #openstack-meeting00:34
*** bbowen has joined #openstack-meeting00:39
*** yamamoto has quit IRC00:40
*** artom has quit IRC00:41
*** yamamoto has joined #openstack-meeting00:41
*** mattw4 has quit IRC00:45
*** yamamoto has quit IRC00:45
*** yamamoto has joined #openstack-meeting00:47
*** Liang__ has joined #openstack-meeting00:47
*** yamamoto has quit IRC00:55
*** yamamoto has joined #openstack-meeting00:57
*** boxest has quit IRC00:58
*** macza_ has quit IRC01:03
*** whoami-rajat has joined #openstack-meeting01:08
*** mjturek has quit IRC01:12
*** jamesmcarthur has joined #openstack-meeting01:15
*** yamamoto has quit IRC01:18
*** bbowen has quit IRC01:18
*** bbowen has joined #openstack-meeting01:18
*** jamesmcarthur has quit IRC01:20
*** yamamoto has joined #openstack-meeting01:22
*** igordc has quit IRC01:25
*** yamamoto has quit IRC01:28
*** yamamoto has joined #openstack-meeting01:37
*** yamamoto has quit IRC01:42
*** yamamoto has joined #openstack-meeting01:42
*** bobh has joined #openstack-meeting01:48
*** TxGirlGeek has quit IRC01:52
*** hongbin has joined #openstack-meeting02:03
*** yamamoto has quit IRC02:08
*** Liang__ has quit IRC02:09
*** Liang__ has joined #openstack-meeting02:10
*** mriedem has quit IRC02:16
*** bbowen has quit IRC02:21
*** ijw has quit IRC02:23
*** ijw has joined #openstack-meeting02:25
*** ijw has quit IRC02:29
*** iyamahat has quit IRC02:39
*** yamahata has quit IRC02:39
*** ijw has joined #openstack-meeting02:47
*** yamamoto has joined #openstack-meeting02:47
*** mhen has quit IRC02:51
*** yamamoto has quit IRC02:53
*** mhen has joined #openstack-meeting02:54
*** psachin has joined #openstack-meeting02:59
*** enriquetaso has quit IRC03:02
*** ijw has quit IRC03:04
*** lbragstad has quit IRC03:13
*** apetrich has quit IRC03:13
*** rcernin has quit IRC03:46
*** rcernin has joined #openstack-meeting03:48
*** diablo_rojo has quit IRC03:48
*** bobh has quit IRC03:54
*** imsurit_ofc has joined #openstack-meeting03:56
*** bobh has joined #openstack-meeting04:06
*** yamamoto has joined #openstack-meeting04:07
*** yamamoto has quit IRC04:13
*** bobh has quit IRC04:33
*** yamamoto has joined #openstack-meeting04:43
*** bobh has joined #openstack-meeting04:52
*** bobh has quit IRC04:57
*** rcernin has quit IRC05:17
*** rcernin has joined #openstack-meeting05:19
*** radeks has joined #openstack-meeting05:24
*** ekcs has joined #openstack-meeting05:33
*** rcernin has quit IRC05:58
*** imsurit_ofc has quit IRC05:58
*** rcernin has joined #openstack-meeting06:00
*** imsurit_ofc has joined #openstack-meeting06:02
*** bobh has joined #openstack-meeting06:04
*** yamahata has joined #openstack-meeting06:06
*** bobh has quit IRC06:09
*** macza has joined #openstack-meeting06:11
*** macza has quit IRC06:15
*** hongbin has quit IRC06:16
*** liuyulong has joined #openstack-meeting06:22
*** yamamoto has quit IRC06:26
*** armax has quit IRC06:27
*** radeks has quit IRC06:28
*** TxGirlGeek has joined #openstack-meeting06:34
*** sridharg has joined #openstack-meeting06:35
*** radeks has joined #openstack-meeting06:36
*** ekcs has quit IRC06:39
*** TxGirlGeek has quit IRC06:40
*** e0ne has joined #openstack-meeting06:50
*** diablo_rojo has joined #openstack-meeting06:54
diablo_rojo#startmeeting fc_sig06:59
openstackMeeting started Wed Jan 16 06:59:41 2019 UTC and is due to finish in 60 minutes.  The chair is diablo_rojo. Information about MeetBot at http://wiki.debian.org/MeetBot.06:59
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.06:59
*** openstack changes topic to " (Meeting topic: fc_sig)"06:59
openstackThe meeting name has been set to 'fc_sig'06:59
diablo_rojogmann, cmurphy mattoliverau tonyb around?06:59
mattoliverauo/06:59
diablo_rojoOh hello :)07:00
tonyb\o07:00
tonybmattoliverau: how is Germany?07:00
mattoliverauKinda, at a hotel at breakfast.. and still need to make it to the office, so might need to leave early07:00
mattoliverauYeah NUE is really nice.07:00
mattoliverauHotel is in the old town, so beautiful.07:01
tonybmattoliverau: pfft you can't leave!07:01
tonybmattoliverau: cool07:01
mattoliverautonyb: lol07:01
cmurphymorning07:01
diablo_rojoDon't have a ton to discuss this week so it shouldn't take the whole hour I don't think.07:01
mattoliverauIf I was roaming I'd meeting while walking. But don't need to leave for about 30 - 40 mins07:02
* tonyb will *try* to not be a distraction07:02
mattoliverauGood luck :p07:02
diablo_rojo#link https://wiki.openstack.org/wiki/First_Contact_SIG#Meeting_Agenda Meeting Agenda07:02
diablo_rojoWe can probably finish before then07:02
*** Luzi has joined #openstack-meeting07:03
diablo_rojo#topic New COntributor Patches07:03
*** openstack changes topic to "New COntributor Patches (Meeting topic: fc_sig)"07:03
*** yamamoto has joined #openstack-meeting07:03
mattoliverauSo I've been distracted and haven't done my homework this week, :s07:03
tonybNone for me ... noone new loves the projects I can help with07:04
*** rcernin has quit IRC07:04
diablo_rojoI definitely didn't do my homework either.07:04
diablo_rojoI was bad.07:04
mattoliverauThere looks like a swift one from December.. ill go make sure it's getting some love. And I'll try and hit up some others if I get some spare time today.07:05
diablo_rojoThanks mattoliverau07:06
*** lpetrut has joined #openstack-meeting07:06
diablo_rojoOkay well.. I guess we should all try to go look at patches when we get ten spare minutes at some point today/tomorrow07:06
diablo_rojoMoving on...07:06
diablo_rojo#link Ask.o.o Monitoring07:07
diablo_rojooops07:07
diablo_rojolol07:07
diablo_rojo#topic Ask.o.o Monitoring07:07
*** openstack changes topic to "Ask.o.o Monitoring (Meeting topic: fc_sig)"07:07
diablo_rojoThere we go.07:07
diablo_rojoI blame the nap I took that made me more tired.07:07
diablo_rojoAnyone do this homework?07:07
diablo_rojoI think I added another softball question for someone to answer07:08
mattoliverauNo, but can look now :p07:08
*** yamamoto has quit IRC07:08
diablo_rojomattoliverau, after the meeting works too07:08
* tonyb always forgets that homework07:09
diablo_rojoEven though it stays in the agenda ;)07:09
diablo_rojoI suppose it is a completely separate site07:10
tonybdiablo_rojo: Yeah I need to bookmark it so I at least go there daily07:10
mattoliverauOr wedsday arvo tonyb :p07:11
diablo_rojoDaily is pretty steep, but if you can manage it go for it.07:11
tonybmattoliverau: +107:11
diablo_rojoSo I guess moral of the story here is go find my easy question and answer it. Winner gets a beer in Denver from me.07:12
diablo_rojoMoving on?07:12
*** pcaruana has joined #openstack-meeting07:12
mattoliverau\o/07:12
diablo_rojo#topic Sandbox Bot07:12
*** openstack changes topic to "Sandbox Bot (Meeting topic: fc_sig)"07:12
mattoliverauI asnwered the non code contribution question ;)07:12
mattoliverauMan I can't type today07:13
mattoliverauI'll blame the fact that I'm jetlagged and on my phone :p07:13
* tonyb hasn't done anything about the bot :(07:14
tonyb... which is silly because I was looking for fun stuff last week07:14
diablo_rojoHopefully you are looking for something fun for the rest of this week? or next week tonyb? :)07:15
tonybdiablo_rojo: I hope so07:16
tonybdiablo_rojo: probably my Friday morning07:16
diablo_rojoNothing else on this topic then for now?07:17
mattoliverauNope07:17
*** alex_xu has joined #openstack-meeting07:18
diablo_rojoHokay moving on then07:18
diablo_rojo#topic Reqs for Organisations Contributing to OpenStack07:18
*** openstack changes topic to "Reqs for Organisations Contributing to OpenStack (Meeting topic: fc_sig)"07:18
diablo_rojoStill waiting on the board of directions elections to complete before proposing it for an agenda item07:19
mattoliverauKk07:19
diablo_rojoBasically just kept this topic so that I could remind you all to vote in the election.07:19
tonybthey have less than a week left right?07:19
diablo_rojoYeah07:20
* diablo_rojo looks for end date07:20
* tonyb voted!07:20
* mattoliverau voted07:20
diablo_rojoJanuary 18th07:21
diablo_rojoAlright. Anything else here?07:21
diablo_rojo#topic People to Reach Out to07:23
*** openstack changes topic to "People to Reach Out to (Meeting topic: fc_sig)"07:23
* tonyb did not reach out to his person07:23
diablo_rojoLooks like I had a todo, tonyb had a todo, and so did mattoliverau07:23
diablo_rojoI didn't do it yet but I will this week07:24
mattoliverauI emailed the person.. so far no response :(07:24
mattoliverauI emailed pre Christmas. Maybe time for a polite follow up07:24
diablo_rojoI feel like we should make an etherpad or ethercalc to keep track of these people and if we have contacted them, when we did, etc.07:25
diablo_rojoSomeone want to do that?07:25
* tonyb will07:25
tonybWell I'll create the EtherCalc y'all will need to fill it in ;P07:26
diablo_rojo#action tonyb will make an etherpad/calc for tracking people to contact & details of contacting them07:26
diablo_rojoCan do07:26
diablo_rojoAnything else here?07:27
mattoliverauNot from me07:29
tonybnope.  saving myself for open discussion07:29
diablo_rojo#topic Open Discussion07:29
*** openstack changes topic to "Open Discussion (Meeting topic: fc_sig)"07:29
diablo_rojoWRT Denver07:30
diablo_rojomattoliverau, gonna do OUI?07:30
mattoliverauYes!.. well I'll ask SUSE to let me go and take it from there :)07:30
tonybPfft I was going to ask that07:30
* tonyb feels redundant :/07:30
mattoliverautonyb: are you going to do OUI?07:31
*** apetrich has joined #openstack-meeting07:31
tonybmattoliverau: Yup07:31
cmurphyis OUI going to be sat+sun before summit?07:32
tonybmattoliverau: Well I have the travel request in it hasn't been approved yet07:32
diablo_rojotonyb, gotta type faster ;)07:32
diablo_rojocmurphy, yep07:32
diablo_rojoCan you come too? :)07:32
cmurphytentatively07:32
tonyb\o/07:32
cmurphyprobably for the sunday, i may have plans already saturday07:33
tonyb.me will have to find more RedHatters to come07:33
diablo_rojoGonna be a party!07:33
diablo_rojoThe only other thing I have to mention before wrapping up is the CFP is open still07:37
tonybI have one more thing ....07:37
diablo_rojoso if there is anything we wanted to submit, should probably think about that soon07:37
diablo_rojotonyb, what is it?07:38
tonybWhat's the preferred landing page to point people at to find the FC SIG07:38
diablo_rojothe wiki as of right now07:38
tonybI pointed a person at:07:38
tonyb[1] https://governance.openstack.org/tc/reference/help-most-needed.html07:38
diablo_rojoI saw the thing on Docs for SIGS07:38
tonyb[2] https://www.openstack.org/community/07:38
tonyb[3] https://wiki.openstack.org/wiki/First_Contact_SIG07:38
tonybbut the last one doesn't really tell people how to find us07:38
diablo_rojoI suppose we could be more direct.07:39
tonybI'm gladd I didn't miss it07:39
diablo_rojoEmail whoever fits your tz better or jump in #openstack dev and ping us07:39
tonybWe could add a section at the bootom of that wiki page07:39
mattoliverauGood idea07:39
diablo_rojoI think I would put it right under the chairs and members\07:40
tonybOkay, it's a wiki so I'll add something and we can bikshed on it :)07:41
diablo_rojoPerfect :)07:42
diablo_rojoOh, I do have one more thing I guess07:42
diablo_rojoHow much time, if any do we think we want/need at the PTG?07:42
tonybquick mattoliverau's gotta go ;P07:43
* diablo_rojo waits for an answer and then will end meeting right after07:43
tonyb0.5 days + hallway track?07:43
mattoliverauAt least I catch up session, maybe a who are we (intro session).07:43
tonybOr perhaps just the hallway track07:43
mattoliverauContribution and company guide update/next steps?07:44
diablo_rojoOkay I will request a half day then07:44
*** iyamahat has joined #openstack-meeting07:44
*** imsurit_ofc has quit IRC07:45
mattoliverauTho yeah, that could be covered in 0.5 and or hallway07:45
diablo_rojoNothing for the CFP though?07:46
diablo_rojowe can talk about Forum sessions next time07:46
*** radeks has quit IRC07:46
tonybSounds like a plan07:46
gmanno/ sorry missed the meeting.07:47
mattoliverauOh maybe.. I gotta go, I'll think about it and let's chat maybe if we think of something in the oui channel07:47
mattoliverauRe: cfp07:47
diablo_rojoIf you have an idea for the cfp ping me, otherwise I will consider it a no07:47
* gmann will read log07:47
mattoliverauKk07:47
diablo_rojothe CFP closes before our next meeting I think07:48
diablo_rojoso make it an email to all of us if you have something07:48
diablo_rojoAlirght. That's all I had for real this time07:48
diablo_rojoAnyone else?07:49
diablo_rojoGoing once!07:49
diablo_rojoGoing twice!07:49
* tonyb is done07:50
*** rubasov has left #openstack-meeting07:50
diablo_rojoSold!07:51
diablo_rojoThanks everyone :)07:51
diablo_rojo#endmeeting07:51
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"07:51
tonybdiablo_rojo: Thanks!07:51
openstackMeeting ended Wed Jan 16 07:51:21 2019 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)07:51
openstackMinutes:        http://eavesdrop.openstack.org/meetings/fc_sig/2019/fc_sig.2019-01-16-06.59.html07:51
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/fc_sig/2019/fc_sig.2019-01-16-06.59.txt07:51
openstackLog:            http://eavesdrop.openstack.org/meetings/fc_sig/2019/fc_sig.2019-01-16-06.59.log.html07:51
cmurphythanks diablo_rojo07:51
*** radeks has joined #openstack-meeting07:52
*** nnsingh has quit IRC07:58
*** bobh has joined #openstack-meeting08:00
*** iyamahat has quit IRC08:01
*** ralonsoh has joined #openstack-meeting08:02
*** nnsingh has joined #openstack-meeting08:02
*** radeks has quit IRC08:04
*** bobh has quit IRC08:05
*** kopecmartin|off is now known as kopecmartin08:15
*** verdurin has quit IRC08:36
*** verdurin has joined #openstack-meeting08:42
*** e0ne has quit IRC08:45
*** yamamoto has joined #openstack-meeting08:52
*** yamamoto has quit IRC08:56
*** imsurit_ofc has joined #openstack-meeting09:02
*** bobh has joined #openstack-meeting09:12
*** Bhujay has joined #openstack-meeting09:14
*** Bhujay has quit IRC09:15
*** Bhujay has joined #openstack-meeting09:15
*** Bhujay has quit IRC09:16
*** Bhujay has joined #openstack-meeting09:17
*** bobh has quit IRC09:17
*** Bhujay has quit IRC09:18
*** Bhujay has joined #openstack-meeting09:18
*** Bhujay has quit IRC09:19
*** Bhujay has joined #openstack-meeting09:20
*** Bhujay has quit IRC09:21
*** Bhujay has joined #openstack-meeting09:21
*** Bhujay has quit IRC09:22
*** Bhujay has joined #openstack-meeting09:23
*** Bhujay has quit IRC09:24
*** Bhujay has joined #openstack-meeting09:24
*** Bhujay has quit IRC09:25
*** Bhujay has joined #openstack-meeting09:26
*** Bhujay has quit IRC09:27
*** Bhujay has joined #openstack-meeting09:27
*** Bhujay has quit IRC09:28
*** brault has joined #openstack-meeting09:29
*** Bhujay has joined #openstack-meeting09:29
*** Bhujay has quit IRC09:30
*** Bhujay has joined #openstack-meeting09:30
*** Bhujay has quit IRC09:31
*** Bhujay has joined #openstack-meeting09:32
*** Bhujay has quit IRC09:37
*** a-pugachev has joined #openstack-meeting09:41
*** Liang__ has quit IRC09:43
*** imsurit_ofc has quit IRC09:46
*** ttsiouts has joined #openstack-meeting09:51
*** imsurit_ofc has joined #openstack-meeting10:03
*** diablo_rojo has quit IRC10:05
*** Bhujay has joined #openstack-meeting10:06
*** Bhujay has quit IRC10:07
*** Bhujay has joined #openstack-meeting10:07
*** Bhujay has quit IRC10:08
*** Bhujay has joined #openstack-meeting10:09
*** electrofelix has joined #openstack-meeting10:09
*** Bhujay has quit IRC10:10
*** Bhujay has joined #openstack-meeting10:10
*** Bhujay has quit IRC10:11
*** Bhujay has joined #openstack-meeting10:12
*** Bhujay has quit IRC10:13
*** Bhujay has joined #openstack-meeting10:13
*** Bhujay has quit IRC10:14
*** Bhujay has joined #openstack-meeting10:15
*** Bhujay has quit IRC10:16
*** Bhujay has joined #openstack-meeting10:16
*** Xi0s has joined #openstack-meeting10:17
*** Bhujay has quit IRC10:17
*** Bhujay has joined #openstack-meeting10:18
*** priteau has joined #openstack-meeting10:18
*** Bhujay has quit IRC10:19
*** Bhujay has joined #openstack-meeting10:19
*** Bhujay has quit IRC10:20
*** Bhujay has joined #openstack-meeting10:21
*** Bhujay has quit IRC10:22
*** e0ne has joined #openstack-meeting10:22
*** Bhujay has joined #openstack-meeting10:22
*** Bhujay has quit IRC10:23
*** Bhujay has joined #openstack-meeting10:24
*** Bhujay has quit IRC10:25
*** Bhujay has joined #openstack-meeting10:25
*** Bhujay has quit IRC10:26
*** Bhujay has joined #openstack-meeting10:27
*** Bhujay has quit IRC10:28
*** Bhujay has joined #openstack-meeting10:28
*** sridharg has quit IRC10:28
*** sridharg has joined #openstack-meeting10:29
*** Bhujay has quit IRC10:29
*** Bhujay has joined #openstack-meeting10:30
*** Bhujay has quit IRC10:31
*** Bhujay has joined #openstack-meeting10:31
*** Bhujay has quit IRC10:32
*** Bhujay has joined #openstack-meeting10:33
*** Bhujay has quit IRC10:34
*** Bhujay has joined #openstack-meeting10:34
*** Bhujay has quit IRC10:35
*** Bhujay has joined #openstack-meeting10:36
*** Bhujay has quit IRC10:37
*** erlon_ has joined #openstack-meeting10:37
*** Bhujay has joined #openstack-meeting10:37
*** Bhujay has quit IRC10:38
*** Bhujay has joined #openstack-meeting10:39
*** Bhujay has quit IRC10:40
*** Bhujay has joined #openstack-meeting10:40
*** Bhujay has quit IRC10:41
*** Bhujay has joined #openstack-meeting10:42
*** Bhujay has quit IRC10:43
*** Bhujay has joined #openstack-meeting10:43
*** Bhujay has quit IRC10:44
*** Bhujay has joined #openstack-meeting10:45
*** Bhujay has quit IRC10:46
*** Bhujay has joined #openstack-meeting10:46
*** Bhujay has quit IRC10:47
*** Bhujay has joined #openstack-meeting10:48
*** Bhujay has quit IRC10:49
*** Bhujay has joined #openstack-meeting10:49
*** Bhujay has quit IRC10:50
*** nnsingh has quit IRC10:50
*** Bhujay has joined #openstack-meeting10:51
*** Bhujay has quit IRC10:52
*** Bhujay has joined #openstack-meeting10:52
*** Bhujay has quit IRC10:53
*** Bhujay has joined #openstack-meeting10:54
*** Bhujay has quit IRC10:55
*** Bhujay has joined #openstack-meeting10:55
*** Bhujay has quit IRC10:56
*** Bhujay has joined #openstack-meeting10:57
*** oneswig has joined #openstack-meeting10:57
*** Bhujay has quit IRC10:58
*** Bhujay has joined #openstack-meeting10:58
*** Bhujay has quit IRC10:59
*** psachin has quit IRC10:59
*** Bhujay has joined #openstack-meeting11:00
*** janders has joined #openstack-meeting11:00
*** daveholland has joined #openstack-meeting11:00
oneswig#startmeeting scientific-sig11:00
openstackMeeting started Wed Jan 16 11:00:41 2019 UTC and is due to finish in 60 minutes.  The chair is oneswig. Information about MeetBot at http://wiki.debian.org/MeetBot.11:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.11:00
*** openstack changes topic to " (Meeting topic: scientific-sig)"11:00
oneswigahoy11:00
openstackThe meeting name has been set to 'scientific_sig'11:00
*** noggin143_ has joined #openstack-meeting11:00
oneswig#link agenda for today https://wiki.openstack.org/wiki/Scientific_SIG#IRC_Meeting_January_16th_201911:01
*** Bhujay has quit IRC11:01
jandersgday! :)11:01
davehollandmorning11:01
oneswigEvening janders! Morning daveholland11:01
jandersit's so hot here today my roller blinds half melted..11:01
priteauHello.11:01
oneswigHi priteau! :-)11:01
verdurinMorning.11:02
jandersswapped em out for wooden venetian blinds. If these start burning, it's time to move somewhere cooler..11:02
oneswigjanders: I'm finding that hard to relate to... could do with some warmth here!11:02
oneswigHi verdurin, is Callum with you?11:02
jandersWe'd give away 15 degrees celcius any time11:02
jandersmaximum of 41C today11:02
Xi0shi, Callum here11:03
*** Bhujay has joined #openstack-meeting11:03
belmoreirao/11:03
oneswigXi0s: Hi Callum, thanks for coming.11:03
oneswigHi belmoreira!11:04
oneswigOK let's get going11:04
oneswig#topic Denver CFP11:04
*** openstack changes topic to "Denver CFP (Meeting topic: scientific-sig)"11:04
oneswigThere is an HPC/GPU/AI track again11:04
oneswigDeadline for submissions 23rd January, not far off11:04
oneswigThis time the Foundation have asked the community to assist with submission review and feedback11:05
oneswigBlair has kindly offered to respond to anyone wanting a second opinion on a proposal.11:05
oneswigIn IRC channel #scientific-wg, or next week's meeting at 2100 UTC11:06
oneswigHe's not here today as he's in NZ and it's midnight11:06
oneswig#link Open Infra Summit CFP https://www.openstack.org/summit/denver-2019/call-for-presentations/11:06
jandersThat's great. I haven't started yet but definitely aiming to submit something.11:07
noggin143_Does Blair need some help? If so, I'd be happy to help out with a third opinion11:07
oneswigThanks noggin143_! I'm sure that would be appreciated.  Hopefully we'll have some good material to review11:08
oneswigWhile you're here noggin143_...11:08
oneswig#topic OpenStack days at CERN11:09
*** openstack changes topic to "OpenStack days at CERN (Meeting topic: scientific-sig)"11:09
oneswigbelmoreira: noggin143_: I hear you're planning an event!11:09
belmoreirawe will have an OpenStack Days CERN on the 27th of May11:09
belmoreirathe registration page should be available soon11:10
oneswigWill it be on Indico?11:10
belmoreirathe theme of the event will be "Accelerating Science with OpenStack"11:11
belmoreiraat same time we will open the call for abstracts for lighting talks11:11
belmoreiraoneswig: the registration will be on eventbrite, but abstracts submissions will be on indico11:12
oneswigbelmoreira: will you announce on openstack-discuss when the registration page is online?11:13
belmoreiranot sure if openstack-discuss is really for this announcements. But it will be in many places11:14
oneswigOK, thanks belmoreira noggin143_ - will look out for it11:14
belmoreiraI would like to invite all the Scientific SIG to this event11:14
janderswould you consider having remote presenters?11:14
belmoreirajanders: we didn't discuss that... but I'm not considering it11:16
oneswigOK, thanks belmoreira11:16
*** bbowen has joined #openstack-meeting11:17
oneswig#topic Open Infra Days London11:17
noggin143_recordings are also under discussion for later watching11:17
*** openstack changes topic to "Open Infra Days London (Meeting topic: scientific-sig)"11:17
oneswigThe date is set for the London OpenStack event - 1st April11:17
oneswig#link Open Infra Days London https://openinfradays.co.uk/11:17
*** imsurit_ofc has quit IRC11:18
oneswigA scientific presentation track is planned11:18
*** ttsiouts has quit IRC11:18
oneswigI hope we can get lots of participation from SIG members from this region11:18
*** ttsiouts has joined #openstack-meeting11:19
oneswigExact details on scheduling/submission are TBD11:19
oneswigThat's all I have on that currently11:19
oneswig#topic AAI for Medical Research Computing11:20
*** openstack changes topic to "AAI for Medical Research Computing (Meeting topic: scientific-sig)"11:20
oneswigWe have a guest speaker today - Xi0s from Oxford Big Data Institute11:20
oneswigThanks for coming Xi0s11:20
oneswigI saw his talk at CIUK 2018 and found it really informative11:21
Xi0sNo problem, I hope the content will be of some use to folks11:21
oneswig#link Presentation slides https://www.dropbox.com/s/kvszibl5c0ijk13/AAI%20in%20Rescomp.pptx?dl=011:21
oneswigXi0s: could you give a bit of context on what you were wanting to do?11:21
noggin143_thanks, i'll pass it on to our AAI team11:21
Xi0sSure, so we're moving from simple genomics compute into a world of increasingly sensitive data requirements as we expand to support medical sciences at the university, and as such we've been building a lot of new services to support this11:22
oneswigThis line looks key today: "Single user identity and password store across all services"11:23
Xi0sCore to this is a new OpenStack deployment, for which we wanted to implement a proper AAI underneath that could do 2FA for anyone and everyone in an enforceable and audit-able way11:23
*** ttsiouts has quit IRC11:24
Xi0sexactly that, so single identity across the cluster, with a single password and single 2FA token registration, rather than per-service implementations11:24
oneswigWhat was the second factor - a token to a mobile?11:25
Xi0sit depends, mobile token (google authenticator) is easy, but has it's limitations11:25
Xi0sno less, a boss that uses a Nokia 3310 - so we also support physical tokens (Yubikey for example)11:25
noggin143_did you need a special sshd or was the 2FA standard?11:26
Xi0sstandard11:26
Xi0sone of the driving requirements was that we can't afford the pain of distributing custom clients to users11:26
Xi0sso openssh on the clientside supports the multi-factor auth as does the sshd on the server side11:28
oneswigThe second factor is described as optional (slide 13), what happens if it is not supplied?11:28
*** t0rrant has joined #openstack-meeting11:28
Xi0sdepends on the authentication protocol being used, but in summary, when the auth server receives the request, it will check your account as to whether 2FA is required (at the account or host or other level) and then act appropriately based on that11:29
Xi0sbecause sshd<->sssd integration is pretty good, when you authentcate using ssh the client will (more or less) be aware as to whether your 2FA is required or not, and inform the user as to whether they have to specify it or not11:30
oneswigSo you can still have machine-to-machine service accounts which can authenticate in a non-interactive way11:30
Xi0syes, in theory, though we've not seen that in practice yet11:31
oneswigHow does this play with things like Ansible?11:31
Xi0ssupport for ssh-keys and certificate-based auth is in there too11:31
Xi0sin what context? (our deployment of all the actors in the AAI was done by ansible - though we don't use any service provided in this for credential store for ansible)11:32
oneswigIf I wanted to configure instances deployed with 2FA enabled, for example.11:33
oneswig(as a user)11:33
Xi0sinteresting question - in theory if you are deploying 2FA you should apply it to _all_ instances and services you run as an admin, else you have a weak link that could be exploited11:34
davehollandis the 2FA service intended for user-created instances/websites; or did you apply it to the OpenStack API/Horizon too?11:34
Xi0sit will be applied to the API/Horizon, but also available for user-created instances11:35
Xi0sso, because we have a number of auth-proxies, we can more or less support whatever service the user wants to enable, whether its a web-service or desktop login, etc11:35
Xi0showever, this is a manual process, to ensure that you don't end up exposing user credentials to a maliciously crafted service11:36
Xi0s(see: OAuth credentials leak from Google Docs ~2017?)11:36
oneswigWhere is the policy on whether an account requires 2FA stored?  Is it possible to be more selective (eg, using source address)?11:37
Xi0spolicy is done at the central auth server (freeIPA), and in _theory_ you could have step-up authentication either at a service level, host level, or indeed source address (though the latter is not implemented) - however due to limitations on the LDAP authentication protocol and the _current_ intergration with the web-proxy we have to either force on or force off for an account11:38
Xi0swhich, for us, meets the requirement of flagging users who have access to potentially sensitive data - however there is more work to be done on these tools for a more versatile service11:39
Xi0sthe key limitation (for us) is that Keycloak<-->sssd integration doesn't account for password expiry11:39
oneswigYou mention federated accounts - what can you do for those?11:40
Xi0sso I love federation, the idea of not having to deal with user-password enquiries makes me excited - so we do have support for federated identity to a point11:41
Xi0sso, allowing github or facebook authentication and linking those identities to our internal identities so that users can authenticate from an external source11:41
oneswigEGI Checkin?11:41
Xi0syes, for sure11:42
Xi0showever, it would only work for the web-based authentication at the moment11:42
oneswigtrue.  Although I thought Indigo-DataCloud had a solution for the command line - noggin143_?11:42
Xi0swhile we have plans to support non-web federation, there is no fully versed solution for this (other than X509 certs, which I don't think is the future there)11:42
oneswigDo you use levels of assurance in your user/role mappings?11:43
Xi0sldap-facade is also around, but basically there is no clientless solution that does not expose credentials to the "service"11:43
Xi0sLoA is not something we use, because we don't have any way of passing that kind of data to the service as yet11:44
Xi0sand realistically, unless the service can respond dynamically to the LoA there's no benefit to passing it as a number, though storing it is good for internal audit purposes11:44
*** liuyulong has quit IRC11:44
oneswigDo you also manage posix user ids as part of this service?11:44
Xi0syes, the identity service manages all of those within a set range11:45
oneswignice :-)11:45
Xi0sanother challenge we are about to face is how we integrate the userID mapping of a service like Lustre in a sensible fashion11:46
jandersI was about to ask about exactly that...11:46
jandersdo you use manila?11:46
Xi0sdespite multiple meetings on the topic, i still have yet to gain clarity on how best to manage that11:46
Xi0snot something we've explored yet - we're in discussions to have a test-deployment of a 'secure lustre' so we can see how to get things flowing right11:47
verdurinjanders: that's still to be decided11:47
Xi0spersonally, I dont see how kerberos tickets dont just solve the problem out of the box, and why an ID translation service is required11:48
janderscool!11:48
oneswigXi0s: verdurin: have you seen the work of daveholland and team on Lustre/OpenStack integration: http://bit.ly/2stawpN11:48
Xi0sbut that could still be my naivety about lustre still11:48
verdurinoneswig: yes, I've seen several iterations of their talk11:49
oneswigthought so...11:49
verdurinOne of the reasons why we're looking at Lustre now11:49
oneswigDo we have more questions for Xi0s?11:50
oneswigIs your configuration documented in more detail than the presentation?11:50
oneswigHow would you suggest other people replicate your work?11:51
davehollandwhat user feedback have you had so far? (usability, reliability, features, ...)11:51
*** bobh has joined #openstack-meeting11:51
Xi0svery little so far, we're in a horrible transition world right now where we're running the new auth in parallel with our existing auth for the batch HPC, so i dare say if I was to poll now, feedback would be pretty poor11:52
Xi0sif you wanted to replicate, we're not in a position to share a playbook just yet, but hopefully one day that could be of value11:53
davehollandunderstood :) I would be interesting in seeing configuration too11:53
oneswigThanks Xi0s, I hope you'll keep us updated if you do get to publish more information11:53
Xi0sSure thing11:54
oneswig#topic OpenStack Scientific SIG at ISC?11:54
*** openstack changes topic to "OpenStack Scientific SIG at ISC? (Meeting topic: scientific-sig)"11:54
oneswig#link This ISC https://www.isc-hpc.com/11:54
*** yamamoto has joined #openstack-meeting11:55
oneswigFor a couple of years SIG members have successfully run a BoF at Supercomputing in the US11:55
oneswigThere was a suggestion that something similar ought to be done for ISC (in Frankfurt).11:55
oneswigI don't think that can be decided now, (there's about a month before the submission deadline)11:55
*** bobh has quit IRC11:56
oneswigI'm interested to know if any SIG members attend?11:56
jandersI'd give it about 30% probability at the moment. Will see what's the interest in the team though11:56
davehollandWe sometimes send someone, no decision yet for this year though11:57
oneswigI am aware of a few others who usually go.  If there's enough to run a BoF, I'll ask around11:57
verdurinI'm closer to going than I have been.11:58
oneswigverdurin: do you go to ISC?11:58
oneswigah, snap11:58
oneswigOK, I will follow up in 2 weeks11:59
oneswig#topic AOB11:59
*** openstack changes topic to "AOB (Meeting topic: scientific-sig)"11:59
oneswigWe are practically on the hour but anything to raise?11:59
jandersI wonder if you have any observations of RDMA latency fluctutations for small message sizes. Was playing with SRIOV/IB a bit again. Next week?12:00
jandersI should have some decent benchmarks by then12:00
oneswigjanders: I'd be interested to hear more on that.  I have an old blog post on bandwidth but didn't cover latency.12:00
jandersgreat!12:00
jandersI mostly focused on bandwidth till now, too12:00
oneswigLet's get it onto the agenda, thanks janders12:00
jandersok! let's chat more on that next week12:00
jandersthanks guys!12:00
oneswigOK, time to close12:01
oneswig#endmeeting12:01
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"12:01
verdurinBye12:01
openstackMeeting ended Wed Jan 16 12:01:08 2019 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)12:01
*** noggin143_ has quit IRC12:01
openstackMinutes:        http://eavesdrop.openstack.org/meetings/scientific_sig/2019/scientific_sig.2019-01-16-11.00.html12:01
oneswigThanks Xi0s and all12:01
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/scientific_sig/2019/scientific_sig.2019-01-16-11.00.txt12:01
openstackLog:            http://eavesdrop.openstack.org/meetings/scientific_sig/2019/scientific_sig.2019-01-16-11.00.log.html12:01
Xi0sThank you, cya12:01
*** Xi0s has quit IRC12:01
*** oneswig has quit IRC12:03
*** janders has quit IRC12:05
*** Bhujay has quit IRC12:06
*** Bhujay has joined #openstack-meeting12:07
*** ttsiouts has joined #openstack-meeting12:09
*** t0rrant has quit IRC12:12
*** e0ne has quit IRC12:17
*** daveholland has quit IRC12:18
*** raildo has joined #openstack-meeting12:27
*** sridharg has quit IRC12:46
*** sridharg has joined #openstack-meeting12:49
*** electrofelix has quit IRC12:58
*** a-pugachev has quit IRC12:58
*** yamamoto has quit IRC12:59
*** e0ne has joined #openstack-meeting13:04
*** bobh has joined #openstack-meeting13:06
*** iyamahat has joined #openstack-meeting13:18
*** Bhujay has quit IRC13:25
*** bobh has quit IRC13:25
*** Bhujay has joined #openstack-meeting13:25
*** Bhujay has quit IRC13:26
*** Bhujay has joined #openstack-meeting13:27
*** Bhujay has quit IRC13:28
*** Bhujay has joined #openstack-meeting13:28
*** Bhujay has quit IRC13:29
*** Bhujay has joined #openstack-meeting13:30
*** yamamoto has joined #openstack-meeting13:30
*** Bhujay has quit IRC13:31
*** Bhujay has joined #openstack-meeting13:31
*** Bhujay has quit IRC13:32
*** Bhujay has joined #openstack-meeting13:33
*** Bhujay has quit IRC13:34
*** Bhujay has joined #openstack-meeting13:34
*** Bhujay has quit IRC13:35
*** Bhujay has joined #openstack-meeting13:36
*** Bhujay has quit IRC13:37
*** yamamoto has quit IRC13:37
*** Bhujay has joined #openstack-meeting13:37
*** Bhujay has quit IRC13:38
*** Bhujay has joined #openstack-meeting13:39
*** Bhujay has quit IRC13:40
*** Bhujay has joined #openstack-meeting13:40
*** Bhujay has quit IRC13:41
*** Bhujay has joined #openstack-meeting13:42
*** Bhujay has quit IRC13:43
*** Bhujay has joined #openstack-meeting13:43
*** Bhujay has quit IRC13:44
*** Bhujay has joined #openstack-meeting13:45
*** Bhujay has quit IRC13:46
*** liuyulong has joined #openstack-meeting13:46
*** Bhujay has joined #openstack-meeting13:46
*** mpiwowarczy has joined #openstack-meeting14:01
*** armax has joined #openstack-meeting14:01
*** artom has joined #openstack-meeting14:01
*** donghao has joined #openstack-meeting14:03
*** davidsha has joined #openstack-meeting14:10
*** lbragstad has joined #openstack-meeting14:13
*** jamesmcarthur has joined #openstack-meeting14:21
*** ttsiouts has quit IRC14:29
*** mriedem has joined #openstack-meeting14:31
*** ttsiouts has joined #openstack-meeting14:35
*** radeks has joined #openstack-meeting14:39
*** ssbarnea|rover has quit IRC14:40
*** ssbarnea|rover has joined #openstack-meeting14:40
*** eharney has joined #openstack-meeting14:41
*** mpiwowarczy has quit IRC14:47
*** radeks has quit IRC14:49
*** gouthamr has quit IRC14:54
*** stevebaker has quit IRC14:55
*** dmellado has quit IRC14:56
*** r-mibu has joined #openstack-meeting15:00
*** enriquetaso has joined #openstack-meeting15:01
*** yamamoto has joined #openstack-meeting15:04
*** awaugama has joined #openstack-meeting15:06
*** Bhujay has quit IRC15:06
*** yamamoto has quit IRC15:10
*** mjturek has joined #openstack-meeting15:11
*** Luzi has quit IRC15:36
*** lpetrut has quit IRC15:46
*** a-pugachev has joined #openstack-meeting15:51
*** sridharg has quit IRC15:52
*** Liang__ has joined #openstack-meeting15:56
*** ganso has joined #openstack-meeting15:58
*** rosmaita has joined #openstack-meeting15:58
*** Liang__ is now known as LiangFang15:59
*** TxGirlGeek has joined #openstack-meeting15:59
*** yamamoto has joined #openstack-meeting15:59
*** wwriverrat has quit IRC15:59
jungleboyj#startmeeting cinder16:00
openstackMeeting started Wed Jan 16 16:00:39 2019 UTC and is due to finish in 60 minutes.  The chair is jungleboyj. Information about MeetBot at http://wiki.debian.org/MeetBot.16:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:00
*** openstack changes topic to " (Meeting topic: cinder)"16:00
openstackThe meeting name has been set to 'cinder'16:00
LiangFango/16:00
whoami-rajatHi16:00
davidshao/16:00
jungleboyjcourtesy ping:  jungleboyj diablo_rojo, diablo_rojo_phon, rajinir tbarron xyang xyang1 e0ne gouthamr thingee erlon tpsilva ganso patrickeast tommylikehu eharney geguileo smcginnis lhx_ lhx__ aspiers jgriffith moshele hwalsh felipemonteiro lpetrut lseki _alastor_ whoami-rajat yikun rosmaita enriquetaso16:00
geguileohi! o/16:01
yikunhello16:01
e0nehi16:01
gansohello16:01
erlon_hello16:01
*** luizbag has joined #openstack-meeting16:01
*** lseki has joined #openstack-meeting16:01
smcginniso.16:01
jungleboyj@!16:01
_pewp_jungleboyj (^o^)/16:01
xyanghi16:01
rosmaitao/16:02
jungleboyjOk.  Looks like we have good representation so we should get started.16:02
jungleboyj#topic announcements16:03
*** openstack changes topic to "announcements (Meeting topic: cinder)"16:03
*** carlos_silva has joined #openstack-meeting16:03
jungleboyjWe are now past milestone 2 so no more driver merges and we should have all the specs merged.16:03
jungleboyjOne exception is for the encryption spec given that we are still waiting on core review of that I believe.16:03
eharneyhttps://review.openstack.org/#/c/621465/ still has review priority set and is close?16:04
jungleboyjLisaLi commented but we were waiting for eharney to also respond.16:04
*** walshh_ has joined #openstack-meeting16:04
jungleboyjeharney:  That is a separate discussion we will get to later.16:04
eharneyok16:04
jungleboyjeharney:  Can you take a look at the volume encryption spec and see if your concerns have been addressed?16:05
eharneyimage encryption?16:05
jungleboyj#link https://review.openstack.org/#/c/608663/16:05
* eharney is still coming back online from vacation16:05
eharneywill do16:06
jungleboyjeharney:  Yes.16:06
jungleboyjeharney:  Thank you and welcome back.16:06
*** radeks has joined #openstack-meeting16:06
jungleboyjOk, so I think that wraps up announcements16:06
*** woojay has joined #openstack-meeting16:07
jungleboyj#topic interest in attending meet-ups while in Raleigh for the mid-cycle16:07
*** openstack changes topic to "interest in attending meet-ups while in Raleigh for the mid-cycle (Meeting topic: cinder)"16:07
jungleboyjSo, there are a couple of meet-ups happening Monday and Tuesday night during the mid-cycle in RTP.16:08
jungleboyjWanted to gauge interest in attending those as a team while we are there.16:08
jungleboyjI added the links to the meetup info to the planning etherpad.16:09
smcginnisWho is all planning on attending the midcycle in person?16:09
jungleboyj#link https://etherpad.openstack.org/p/cinder-stein-mid-cycle-planning16:09
*** martinkennelly has joined #openstack-meeting16:09
smcginniso/16:09
jungleboyjo/16:09
smcginnisJust going to be jungleboyj and I sitting in a room together?16:10
jungleboyjhemna:  Is planning to attend.16:10
rosmaitai should be there tuesday and wednesday16:10
rosmaita(haven't gotten official approval yet)16:10
smcginnisAwesome16:10
jungleboyjsmcginnis:   He he, we can do that here.16:10
smcginniseharney: Are you going to make it?16:10
jungleboyjeharney:  Is on the list as well as jbernard16:10
eharneyyes16:10
jungleboyj6 Cinder guys sit in a room...16:10
e0ne:)16:11
smcginnisAnyone not on the list so far?16:11
jungleboyjHas anyone tried getting their management on board?16:11
jungleboyjBeuhler ... ... Beuhler ?16:12
rosmaitahow do you mean?16:13
jungleboyjIf anyone else has asked if they can attend.16:13
jungleboyjAnyway, it looks like we have the attendance list.16:14
rosmaitai think it will be an easier sell next time with the PTG no longer existing as an independent event16:14
enriquetaso0/16:14
smcginnisI do hope it's productive enough for there to be a next time.16:14
jungleboyjSo, smcginnis and I are going to try to join the meetup on Monday night.  Would any of you like to join us?  rosmaita eharney hemna jbernard ?16:14
jungleboyjsmcginnis:  ++16:14
*** munimeha1 has joined #openstack-meeting16:14
jungleboyjWe will just have to make the most of it.16:15
rosmaitasorry about the response lag, i would be interested in that as well16:15
jungleboyjIf anyone wants to join us at the meetup, please sign up and add your name to the list.16:15
jungleboyjThen we can coordinate getting there.16:16
jungleboyjrosmaita: smcginnis eharney hemna  Would you guys like to also do the one on Tuesday night or should we do our own thing?16:16
eharneyi haven't looked at any of this stuff16:16
smcginnisThe Tuesday night one looks potentially more interesting to me. But I would also be fine just going out for a beer with folks too.16:17
jungleboyjeharney:  Ok, take a look and update the etherpad.16:17
jungleboyjsmcginnis:  I had that thought too.16:17
jungleboyjrosmaita:  Thoughts?16:17
smcginniswin 2016:18
rosmaitai'm easy, i will do whatever16:18
jungleboyjOk.16:18
*** abishop has joined #openstack-meeting16:18
jungleboyjsmcginnis:  And I will figure it out and update the etherpad.16:18
rosmaitai just want to spend facetime with openstack folks16:18
smcginnis++16:18
jungleboyj++16:18
jungleboyjOk.  So lets tentatively plan to attend the Tuesday one as well.  If we change our minds after Monday night, we can.16:19
jungleboyjOk.  Lets move on then.16:20
rosmaitaworks for me16:20
smcginnisSeems like everyone needs some DeathWish coffee today. :)16:20
jungleboyj:-)16:20
jungleboyj#topic  May can consider leverage hardware accelerator in image conversion before uploading and after downloading16:21
*** openstack changes topic to "May can consider leverage hardware accelerator in image conversion before uploading and after downloading (Meeting topic: cinder)"16:21
jungleboyjLiangFang:  Are you here?16:21
LiangFangyes16:21
jungleboyjThe floor is your's.16:21
LiangFangcurrently the new server platform may containing compression hardware accelerator16:21
jungleboyjRight.16:22
LiangFangso we may can leverage such kind of hardware to do the image conversion16:22
LiangFangin order to do this, we may need to introduce new format of image, such as zip16:22
eharneywhat does the hardware do exactly?16:22
LiangFange.g.16:23
LiangFangcurrently if upload volume to image16:23
LiangFangcpu do the format change, from raw to qcow2, for example16:24
LiangFangat this time, the server cpu is highly used, and the response is slow for other user16:24
*** pcaruana has quit IRC16:25
jungleboyjSo you are thinking that the image conversion could be offloaded to the specialized hardware?16:25
LiangFangyes16:25
LiangFangif the hardware is there, then use it16:25
smcginnisLiangFang: Do you have a proposed change for this? Or just asking if it makes sense to pursue something like this?16:25
eharneybut only for certain image formats, which maybe aren't used currently?16:25
LiangFangsmcginnis: I want to do this, here ask your opinion16:26
jungleboyjIt is interesting and also has impacts to RSD applications.16:26
LiangFangeharney: yes, the format should be some standard compression format, such as zip16:27
smcginnisIn general, taking advantage of FPGAs sounds great. I guess I would have to understand better how that would be implemented though.16:27
jungleboyjsmcginnis: ++16:27
LiangFangI there's no objection here, I will go ahead to prepare spec16:27
LiangFangif16:27
jungleboyjCould picture this being interesting to telco where they are moving images to edge sights.  May have FPGAs there that could then be used to speed image expansion.16:28
jungleboyjLiangFang:  I think there is a lot to consider and a spec would be the best place to start.16:28
LiangFangyes, so I will prepare spec16:28
jungleboyjAnyone have an objection to that approach?16:29
eharneywould need more details to have an objection :)16:29
jungleboyjeharney:  :-)16:29
rosmaitaeharney: ++16:29
jungleboyjOk, so it sounds like it is worth your time to propose the spec with additional details.16:29
jungleboyjLiangFang: ^16:29
LiangFangok ok16:29
jungleboyj#action LiangFang to write a spec proposing the functionality.16:30
jungleboyjAnything else LiangFang16:30
LiangFangnothing more16:30
LiangFangthanks16:30
jungleboyjThank you and thanks for waiting from last week.  Sorry we ran out of time.16:31
*** a-pugachev has quit IRC16:31
LiangFangjungleboyj: np16:31
jungleboyj#topic Gate Failure eventlet.tpool16:31
*** openstack changes topic to "Gate Failure eventlet.tpool (Meeting topic: cinder)"16:31
jungleboyjwhoami-rajat:  Your turn.16:31
jungleboyjwhoami-rajat:  Did we lose you?16:32
whoami-rajatSo recently a lot of gate failures occurred and one of the most common was testing the eventlet.tpool thread list. I've listed my findings about what might have been caused in the bug.16:32
whoami-rajatjungleboyj: no no, just typing :)16:32
jungleboyjOk.  Cool.16:32
whoami-rajatany feedback on the bug is appreciated.16:33
smcginnisBased on the notes in the bug report, sounds like those unit tests should probably just be removed.16:33
*** vishakha has joined #openstack-meeting16:33
e0nelooks line an issues with some mock or cleanUp16:34
jungleboyjYeah.  Is geguileo here to chime in?16:34
jungleboyjHe is our tpool expert.16:34
*** hongbin has joined #openstack-meeting16:34
geguileoI am here16:34
jungleboyjYay!16:35
geguileosorry, I wasn't paying attention  :-(16:35
jungleboyjThoughts?16:35
jungleboyj#link https://bugs.launchpad.net/cinder/+bug/181166316:35
openstackLaunchpad bug 1811663 in Cinder "Gate failure : AssertionError: Lists differ: [] != [<Thread(tpool_thread_0, started daemon 14[1123 chars]56)>]" [Undecided,In progress] - Assigned to Rajat Dhasmana (whoami-rajat)16:35
*** LiangFang has quit IRC16:36
jungleboyjIt appears that removing the test for now is the appropriate first step.16:36
geguileocould be related to https://review.openstack.org/#/c/630971/16:37
geguileoBut I'd have to read more carefully the bug16:37
geguileoSorry, not that one16:37
geguileoXD16:37
geguileohttps://review.openstack.org/#/c/615934/16:37
geguileo^ that one16:37
eharneywe could just modify the test to assert that the pool is at least X big, instead of exactly X, right?16:38
jungleboyj#link https://review.openstack.org/#/c/615934/16:38
jungleboyjeharney:  That could work too.16:38
e0nesmcginnis: I thinks we should fix these tests instead of removing them16:38
smcginniseharney: Not sure if that would work if it was already initialized with a smaller number though.16:38
e0neeharney: we can just mock all tpool calls in these tests16:38
geguileoit could be that we are not clearing the threads between tests...16:39
whoami-rajateharney:  i think it's not caused by the size rather than the list of threads getting started by some other test.16:39
e0negeguileo: +116:39
smcginnisBut if we have concurrent tests running, we would still have this issue even if the threads were cleared, right?16:40
whoami-rajatgeguileo: it would still be a task to find the tests triggering the threads to start as the tpool.execute method exists in multiple places in code.16:40
e0nesmcginnis: do we use threads in tests? we should use processes for concurrent run16:41
davidshasmcginnis: are the threads mocked out in the unit tests?16:41
whoami-rajatsmcginnis:  +1. The async backup tests failing being the example.16:41
*** ekcs has joined #openstack-meeting16:41
smcginnise0ne: Not sure how that is handled in tox.16:41
smcginnisdavidsha: I'm not sure if they are everywhere.16:42
*** macza has joined #openstack-meeting16:42
geguileolooks like somebody removed code from cinder/test.py16:42
geguileoand that broke it16:42
jungleboyjgeguileo:  What was that?16:42
* geguileo probably wrong, I'll look more carefully at the issue16:42
davidshasmcginnis: kk, you could try purging whatever package you're using for threads and see which tests throw the error?16:42
jungleboyjOk, geguileo  So you will work with whoami-rajat on this?16:43
eharneysetUp() in test.py does do a tpool.killall to try to start clean for each run16:43
geguileoeharney: but maybe eventlet has changed the code and now they don't do what they used to?  r:-??16:44
davidshaanything not mocked would show up.16:44
geguileobecause I added that in there when I introduced the tests16:44
jungleboyjOk.  We have other topics we should get to.  If you could help look into this geguileo it would be appreciated.16:45
*** gouthamr has joined #openstack-meeting16:45
e0ne smcginnis: http://paste.openstack.org/show/742778/16:45
jungleboyjCan we take this to the channel for afterwards?16:46
whoami-rajatSure16:46
jungleboyjOk.16:46
jungleboyj#action whoami-rajat to follow up with geguileo for more information.16:47
jungleboyj#topic follow-up on "remove policy checks at DB layer?" from 5 December meeting16:47
*** openstack changes topic to "follow-up on "remove policy checks at DB layer?" from 5 December meeting (Meeting topic: cinder)"16:47
jungleboyjrosmaita: Your turn.16:47
rosmaitathis will be fast16:47
rosmaitathe quick recap is that our DB layer policy checks were making it difficult to do a commonly requested policy configuration (to have a read-only admin)16:47
rosmaitawe decided that since there are so many of these checks currently in there, it would be too destabilizing to mess with them until we have more tests in place16:48
rosmaitaplus, there *is* a way to do this via the policy configuration file, *and* the workaround is valid into all the stable branches (and beyond)16:48
rosmaitaso i wrote up a "howto" for the cinder docs and revised it after i got some feedback from lbragstad16:48
rosmaitait's available for your reviewing pleasure: https://review.openstack.org/#/c/624424/16:48
*** hongbin has quit IRC16:48
rosmaitashould be a pretty quick read, just explains some concepts and then walks through how to configure the policy file16:48
rosmaitathat's all i got16:49
jungleboyj#link https://review.openstack.org/#/c/624424/16:49
jungleboyjrosmaita:  Ok.  Thank you.16:49
rosmaitanp16:49
jungleboyjOk.  Last topic.16:49
*** hongbin has joined #openstack-meeting16:50
jungleboyj#topic RSD driver16:50
*** openstack changes topic to "RSD driver (Meeting topic: cinder)"16:50
jungleboyjSo, eharney asked about this earlier:16:50
jungleboyj#link https://review.openstack.org/#/c/621465/16:50
*** e0ne_ has joined #openstack-meeting16:50
*** e0ne has quit IRC16:50
jungleboyjThe driver has missed the official deadline for inclusion.16:51
jungleboyjThey have continued to work on resolving the issues.16:51
jungleboyjGetting a little closer each day.16:51
jungleboyjWe need to decide how to proceed.16:51
smcginnisStill code issues being found as recently as yesteday.16:51
smcginnisLast third party CI run failed.16:52
jungleboyjsmcginnis:  Yeah, that is concerning.16:52
jungleboyjThey are down to 7 tests failing.16:52
*** ttsiouts has quit IRC16:52
jungleboyj#link http://52.27.155.124/cinder/rsd/621465/31/testr_result.html.gz16:52
smcginnisI would say with the state it's in now and how far we are past the driver freeze, it's missed stein.16:53
smcginnisIt would be one thing if there was a small issue that got resolved in a day or two after the deadline.16:53
davidshaThese are issues in the backend rather than the driver, we've been engaging with the backend team to help resolve these.16:53
smcginnisBut it still doesn't appear ready.16:53
jungleboyjSo these are problems being found in the storage itself?16:54
davidshaYup16:54
jungleboyjeharney: e0ne_  Other core team thoughts here?16:55
davidshaIt's still being worked atm and we're providing the feedback from the CI to the backend team to help resolve it.16:55
jungleboyjIf the backend itself is having issues it seems it needs some more burn-in time and could merge as soon as Train opens if things get resolved.16:56
e0ne_if backend itself is non-stable and deadline is missed I recommend to move it to T release16:56
jungleboyjeharney:  You brought the topic up first.16:56
smcginnisThere's nothing preventing this driver being available out of tree for Stein users, then added in the Stein release.16:57
eharneyyeah, was asking about it because i was just reviewing it and noticed it was about done16:57
jungleboyjsmcginnis:  You mean Train release?16:58
e0ne_smcginnis: +116:58
davidshaWe have the backend team working on a patch we think might address the last 7 issues, it should be redeployed by COB tomorrow.16:58
jungleboyjI think that we should defer acceptance.  The patch will be out there for people to pull it down to use out of tree and we can merge in Train once we have seen the CI running stably for a while.16:59
jungleboyjIt is consistent with what we have done for other drivers in the past.17:00
smcginnisI think the driver will be a great addition. But we are past the deadline and we've enforced that in the past and made other new drivers adhere to it.17:00
jungleboyjsmcginnis: ++17:00
jungleboyjOk.  We are out of time.  If we need to further discuss we can do it in the cinder channel.17:01
jungleboyjThanks to everyone for joining!17:01
jungleboyj#endmeeting17:01
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"17:01
openstackMeeting ended Wed Jan 16 17:01:27 2019 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)17:01
openstackMinutes:        http://eavesdrop.openstack.org/meetings/cinder/2019/cinder.2019-01-16-16.00.html17:01
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/cinder/2019/cinder.2019-01-16-16.00.txt17:01
openstackLog:            http://eavesdrop.openstack.org/meetings/cinder/2019/cinder.2019-01-16-16.00.log.html17:01
*** ijw has joined #openstack-meeting17:05
*** iyamahat has quit IRC17:12
*** e0ne_ has quit IRC17:18
*** igordc has joined #openstack-meeting17:19
*** kopecmartin is now known as kopecmartin|off17:21
*** woojay has left #openstack-meeting17:23
*** ekcs has quit IRC17:24
*** martinkennelly has quit IRC17:25
*** mattw4 has joined #openstack-meeting17:31
*** lseki has left #openstack-meeting17:33
*** luizbag has left #openstack-meeting17:35
*** jamesmcarthur has quit IRC17:45
*** jamesmcarthur has joined #openstack-meeting17:46
*** iyamahat has joined #openstack-meeting17:48
*** davidsha has quit IRC17:49
*** raildo has quit IRC17:54
*** raildo has joined #openstack-meeting17:54
*** jamesmcarthur has quit IRC18:02
*** jamesmcarthur has joined #openstack-meeting18:02
*** iyamahat has quit IRC18:04
*** yamahata has quit IRC18:04
*** jamesmcarthur has quit IRC18:07
*** jamesmcarthur has joined #openstack-meeting18:07
*** priteau has quit IRC18:08
*** jamesmcarthur has quit IRC18:20
*** iyamahat has joined #openstack-meeting18:20
*** ekcs has joined #openstack-meeting18:21
*** jamesmcarthur has joined #openstack-meeting18:25
*** factor has joined #openstack-meeting18:28
*** jamesmcarthur has quit IRC18:30
*** rosmaita has left #openstack-meeting18:35
*** artom has quit IRC18:37
*** yamahata has joined #openstack-meeting18:39
*** jamesmcarthur has joined #openstack-meeting18:39
*** jamesmcarthur has quit IRC18:42
*** macza has quit IRC18:51
*** diablo_rojo has joined #openstack-meeting19:00
*** mattw4 has quit IRC19:02
SotK#startmeeting storyboard19:03
openstackMeeting started Wed Jan 16 19:03:18 2019 UTC and is due to finish in 60 minutes.  The chair is SotK. Information about MeetBot at http://wiki.debian.org/MeetBot.19:03
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.19:03
*** openstack changes topic to " (Meeting topic: storyboard)"19:03
openstackThe meeting name has been set to 'storyboard'19:03
SotK#link https://wiki.openstack.org/wiki/Meetings/StoryBoard#Agenda_for_next_meeting Agenda19:03
*** mattw4 has joined #openstack-meeting19:03
* diablo_rojo waves hello19:03
*** efried has quit IRC19:04
SotK#topic Announcements19:05
*** openstack changes topic to "Announcements (Meeting topic: storyboard)"19:05
SotKoauth 3.0.0 is breaking things and we need to solve that19:06
SotKthanks fungi for starting to look at it19:06
diablo_rojoI think I saw a patch from fungi?19:06
* diablo_rojo goes to double check19:06
SotKyup!19:06
SotKI think its a work in progress though19:06
diablo_rojoFOr now we just pinned the version to be less than 3.0.019:06
fungiyeah, the patch from me is just to be able to recreate the errors so we can work on squashing them19:07
diablo_rojo#link https://review.openstack.org/#/c/631260/ WIP19:07
fungilooks like there are several basic test failures in there, a couple of which might be interrelated19:07
diablo_rojoYeah I think the one I saw in unit tests wasn't the one you were hitting, but it was still related to OAuth19:08
fungione of them is actually19:08
fungithe story i opened quotes the version from the test failure19:08
fungijust for consistency and ease of investigation19:09
diablo_rojoGot a story link?19:09
fungibut it's the same exception i saw in production (well, on a new dev server deployment at least)19:09
fungi#link https://storyboard.openstack.org/#!/story/2004789 InvalidRedirectURIError with oauthlib 3.0.019:10
diablo_rojoThanks!19:10
fungifigured that gives us a better place to collaborate, and possibly to point a github issue/pr at if we determine it's a legit oauthlib regression and not just us doing things wrong19:10
diablo_rojogood thinking19:11
SotKyep, good idea19:11
fungithe good news is that now https://storyboard-dev.openstack.org/ is being served from a server named storyboard-dev01.opendev.org (note the different domain name for the machine) running ubuntu xenial19:13
diablo_rojoThat's super awesome :)19:13
fungionce i figured out the oauthlib problem, that was the last blocker19:13
SotKnice!19:13
fungiso i can now proceed with the database move on storyboard-dev and confirm that's happy19:14
diablo_rojoThat's exciting :)19:14
fungioh, actually we're also getting an error emitted from puppet about characterset collation settings for the db, which i need to dig into first i suppose19:14
fungibut it doesn't seem to be causing any actual service issues19:14
fungimay affect raw query result ordering/sorting from sql19:15
diablo_rojoInteresting19:16
fungii doubt it's an issue with the migrations, probably just with one of the db configuration parameters we're setting in the puppet manifest19:16
*** gouthamr_ has joined #openstack-meeting19:16
fungibut i need to reacquaint myself with the error details since i was focused more on the openid auth problem19:17
diablo_rojoThat's fair. Still making good progress it sounds like :)19:18
*** brault has quit IRC19:18
SotKindeed, thanks for working on all these things :)19:18
diablo_rojoI can try to look into the OAuth failures a little more to see whats up there.19:19
fungithanks, one is just an http response code mismatch between 400 vs 40119:20
diablo_rojoYeah I was gonna start with the easy stuff first :)19:20
fungithat one is likely easy to fix. definitely take a look at the oauthlib changelog though19:20
*** efried has joined #openstack-meeting19:21
fungi#link https://github.com/oauthlib/oauthlib/blob/master/CHANGELOG.rst oauthlib changelog19:21
diablo_rojoYeah that was going to be my starting point. Thanks for the link again!19:21
fungialso the flowchart and explanation in rfc 6749 are a good reference to explain how that stuff is supposed to work19:22
diablo_rojoNoted19:22
*** artom has joined #openstack-meeting19:22
fungi#link https://tools.ietf.org/html/rfc6749#section-4.1 IETF RFC 6749 The OAuth 2.0 Authorization Framework 4.1 Authorization Code Grant19:22
*** dmellado has joined #openstack-meeting19:22
*** artom has quit IRC19:22
*** artom has joined #openstack-meeting19:23
SotKthanks in advance diablo_rojo :)19:24
diablo_rojoLol don't thank me yet ;)19:25
SotK:D19:25
SotK#topic In Progress Work19:25
*** openstack changes topic to "In Progress Work (Meeting topic: storyboard)"19:25
funginote the oauthlib story isn't not urgent, just worth supporting latest oauthlib eventually19:25
fungier, is not urgent19:26
SotKdiablo_rojo: thanks for the reviews and merges19:27
diablo_rojoMy pleasure :)19:27
SotKand sorry I've not done any myself again19:27
* SotK hopes to get to properly thinking about breaking down the attachments work at some point in the next week19:30
*** igordc has quit IRC19:30
diablo_rojoThat would be awesome.19:30
diablo_rojoBut I get that you're busy too19:30
*** dtrainor has quit IRC19:30
*** vishakha has quit IRC19:31
diablo_rojoI'll do another loop through open things to see what I can single core approve.19:32
diablo_rojoI know I have a few patches out there that need attention19:33
diablo_rojoI probably shouldn't approve my own ;)19:33
SotKprobably not :)19:33
* SotK sticks a "DO CODE REVIEW" post-it above his screen19:33
SotKdoes anyone have anything else to discuss or shall we end the meeting?19:36
diablo_rojoNothing else new atm19:37
*** dtrainor has joined #openstack-meeting19:37
funginothing on my end19:38
SotKin that case, thank you both for coming :)19:38
fungithanks for chairing!19:38
SotK#endmeeting19:38
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"19:38
openstackMeeting ended Wed Jan 16 19:38:36 2019 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)19:38
openstackMinutes:        http://eavesdrop.openstack.org/meetings/storyboard/2019/storyboard.2019-01-16-19.03.html19:38
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/storyboard/2019/storyboard.2019-01-16-19.03.txt19:38
openstackLog:            http://eavesdrop.openstack.org/meetings/storyboard/2019/storyboard.2019-01-16-19.03.log.html19:38
diablo_rojoThanks SotK!19:39
*** mriedem has quit IRC19:40
*** carlos_silva has quit IRC19:41
*** mriedem has joined #openstack-meeting19:41
*** jamesmcarthur has joined #openstack-meeting19:43
*** e0ne has joined #openstack-meeting19:46
*** eharney has quit IRC19:47
*** stevebaker has joined #openstack-meeting19:48
*** jamesmcarthur has quit IRC19:48
*** jamesmcarthur has joined #openstack-meeting19:51
*** artom has quit IRC19:54
*** hongbin has quit IRC19:55
*** dklyle has quit IRC19:57
*** dklyle has joined #openstack-meeting19:57
*** hongbin has joined #openstack-meeting19:57
*** mattw4 has quit IRC19:59
*** mattw4 has joined #openstack-meeting20:00
*** erlon_ has quit IRC20:06
*** TxGirlGeek has quit IRC20:06
*** yamamoto has quit IRC20:08
*** TxGirlGeek has joined #openstack-meeting20:10
*** erlon has joined #openstack-meeting20:35
*** whoami-rajat has quit IRC20:37
*** yamamoto has joined #openstack-meeting20:38
*** raildo has quit IRC20:41
*** raildo has joined #openstack-meeting20:41
*** lpetrut has joined #openstack-meeting20:43
*** e0ne has quit IRC20:44
*** e0ne has joined #openstack-meeting20:45
*** TxGirlGeek has quit IRC20:45
*** raildo has quit IRC20:46
*** awaugama has quit IRC20:47
*** e0ne has quit IRC20:48
*** a-pugachev has joined #openstack-meeting20:51
*** eharney has joined #openstack-meeting20:55
*** TxGirlGeek has joined #openstack-meeting20:57
*** m_kazuhiro has joined #openstack-meeting20:57
*** patchbot has joined #openstack-meeting20:58
*** r-mibu has quit IRC21:00
notmynameswift meeting time21:00
notmyname#startmeeting swift21:00
openstackMeeting started Wed Jan 16 21:00:52 2019 UTC and is due to finish in 60 minutes.  The chair is notmyname. Information about MeetBot at http://wiki.debian.org/MeetBot.21:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.21:00
*** openstack changes topic to " (Meeting topic: swift)"21:00
openstackThe meeting name has been set to 'swift'21:00
notmynamewho's here for the swift team meeting?21:00
claygo/21:01
kota_o.21:01
mattoliverauo/21:01
m_kazuhiroo/21:01
rledisezhi o/21:01
*** zaitcev has joined #openstack-meeting21:01
kota_o/21:01
tdasilvahi21:01
notmynamewelcome21:02
notmynametwo quick logistical points to address before we get intot he main meeting topic21:02
notmynamefirst, FYI, I'll be tagging the stable branches soon. I know I've been saying that, but the last patch just landed, and I realized that I need to do changelog updates. so I'll get that done asap (but it may be while I'm on a plane this weekend21:03
mattoliverauwell you'll have a long enough flight :)21:04
clayghahaha!21:04
notmynamesecond, I'll be traveling for the next two weeks (LCA next week and holiday the week after), so for the next two meetings, it's up to the rest of you to figure out if  there is a meeting and who's leading it21:04
mattoliverauif it makes you feel better, I'll be flying much longer this weekend :P21:04
clayg👃 👈21:04
notmynamemattoliverau: it doesn't ;-)21:04
mattoliveraulol21:04
claygHahah21:04
*** igordc has joined #openstack-meeting21:05
notmynameany questions or comments about those two things?21:05
*** lpetrut has quit IRC21:05
notmynameok, no questions, let's move on21:05
notmyname#topic Reconstructor can rebuild to too many primaries21:05
*** openstack changes topic to "Reconstructor can rebuild to too many primaries (Meeting topic: swift)"21:05
notmynamehttps://review.openstack.org/#/c/629056 has been proposed by clayg, and he'd like to discuss it this week21:05
patchbotpatch 629056 - swift - NEED HELP: Rebuild frags for unmounted disks - 3 patch sets21:06
*** jamesmcarthur has quit IRC21:06
notmynameclayg: you've got the floor.21:06
claygit's related to https://bugs.launchpad.net/swift/+bug/151034221:06
openstackLaunchpad bug 1510342 in OpenStack Object Storage (swift) "Reconstructor does not restore a fragment to a handoff node" [Medium,Confirmed]21:06
claygwhich maybe some people have been aware of...21:06
claygbasically our EC design doesn't support fail-in-place (i.e. unmount a primary disk and worry about ring changes later)21:06
claygreplication does support this strategy for dealing with disk failure and it's been pointed out for awhile that difference is maybe annoying or frustrating or confusing for operators...21:07
claygfor awhile it seemed like there was other more important stuff to work on with improving EC - but eventually we fixed that stuff :D21:07
notmyname"we ran out of easier stuff to fix"21:07
claygI got to thinking about it and couldn't really come up with a good excuse NOT to do this - so I started to look at if that was worth doing and decided it might be pretty good!21:07
claygDoes anyone have any questions about the bug or priority before I talk about implementation/design challanges that I'd like some help with before I continue coding on it?21:08
*** radeks has quit IRC21:08
*** jamesmcarthur has joined #openstack-meeting21:08
kota_the challenge sounds so nice21:08
claygok, great!  so assuming we're onboard with maybe it's worth spending some braincells on it...21:09
clayg(rledisez i know you do EC rebuild sorta different?  you may have some different perspective here?)21:09
claygone issue that was interesting is I had to "index" handoff nodes21:10
claygwe don't want two unmounted causing partner nodes to try and rebuild different fragments to the first handoff21:10
notmynamehow'd you do that? enumerate handoffs and assign them frag indexes based on a MOD or something?21:11
claygso instead if node index 0 is unmounted node index -1 and 1 will rebuild to first handoff, index 1 rebuilds to "second" handoff etc21:11
claygnotmyname: exactly right - minor change in ring code - but it worked out fine... biggest issue seems to be finding all our "fake ring" implementations and teaching them to assign index to handoffs21:12
claygthat part is just worth knowing - i think it's fine21:12
notmynamekk21:12
claygmy challange right now is that the "current" design for primary suffix syncing is... bananas21:12
clayggo look t this code please -> https://review.openstack.org/#/c/629056/3/swift/obj/reconstructor.py21:12
patchbotpatch 629056 - swift - NEED HELP: Rebuild frags for unmounted disks - 3 patch sets21:12
zaitcevI'm too scared to look.21:13
claygwhen we tell a primary to check it's left and right hand partners we stuff that in job['sync_to'] - a list of two nodes then we ... extend the list of nodes we check... with ALL the other nodes!?21:13
claygso the idea is that if our partners respond OK, we're done - but if there's *ANY* kind of error (timeout, rejected for concurrency, or even 507) we'll keep going until we do 2 successful syncs ... or run out of primaries to try21:14
claygthat's not right... I think me or paul was just like... we have to do SOMETHING on failure - this can't be "wrong" although it's probably not optimal21:14
claygthen it sat like that for... whatever 3-4 years?21:15
claygNow that I'm changing it so that if our left partner responds unmounted we'll actually sync to a handoff instead - it seems like *just* doing our left and right partner might be sufficient!?21:15
claygor maybe we could also check with maybe one more "partner" like maybe on the other side of the ring?  then we have a total of three nodes checking on every given fragment...21:16
clayg3 is still < ALL OF THEM so I guess I was leaning that way21:16
claygbut I could see arguments being made for all kinds of other ideas21:16
claygso I was hoping we could have some of that debate now - instead of after I wrote it21:16
claygAnyone have an opinons/ideas/concerns/questions (that's kinda all I got)21:16
notmynamegood summary, thanks21:17
notmynameand it sounds like a hard problem21:17
notmynameif the fragment handoff is MOD the fragment index, doesn't that mean that you "only" need to check every N+M nodes, not all of them? eg with 8+4, you only need to check every 12th one (+frag_index)? hmm, but that means handoffs likely wouldn't be found after a rebalance21:17
notmynameclayg: there's a step you described that I could use more detail on21:18
*** priteau has joined #openstack-meeting21:18
claygi don't know that putting frags further out in the handoff chain makes them any more or less likely to be reachable in the handoff chain after a rebalance21:18
notmynameNow that I'm changing it so that if our left partner responds unmounted we'll actually sync to a handoff instead - it seems like *just* doing our left and right partner might be sufficient!?"21:18
notmynamegood point re handoffs. it's randomish rebalancing anyway21:19
notmynameclayg: can you explain the "just doing our left and right partner" part? what is "doing" in this case?21:19
claygyes... the big change is that if we get a 507 specifically when syncing with a partner node we'll rebuild/sync to a handoff (and keep the handoff in-sync as long as it's mirroed primary is unmounted until a ring change)21:19
*** ganso has quit IRC21:20
mattoliverauSounds good.. but to be honest, I'd need to go read the code and think on it. I might be a little jetlagged and may also may have had a few beers with dinner as I forgot about this meeting.21:21
clayg"doing" means making sure that A SET of our left partners frags are rebuild *somewhere* - either on the primary where it belongs, or if it's unmounted we'll actively find another node and rebuild fragments there... that's what enables fail-in-place21:21
*** TxGirlGeek has quit IRC21:21
notmynameso after a rebalance, the handoff frag will attempt to put to the primary, which may still be unmounted. then the primary neighbor will choose a new handoff location which will again be kept in sync until a rebalance? sounds like a risk for "replication bumbs"21:22
claygmattoliverau: right... it'd be nice if we could go over something like this around a white board.  Except I'd not like a TON of work regardless of what we decide - and we could change it later - so I was hoping to have a direction settled in the next week or so and code it up for review with a couple of weeks tops21:22
claygnotmyname: with a fail-in-place strategy you *remove* unmounted drives from the ring before rebalance21:23
claygnotmyname: also, with EC after a rebalance you almost certainly should be in handoffs_only mode (no rebuilds) until you finish shuffling data around21:23
notmynameI like that second point. not sure I agree with the first21:24
claygso... those handoff nodes with the rebuild fragments will *probably* be the guy that ships the frags to their new primary21:24
notmynameyeah, that makes sense21:24
*** priteau has quit IRC21:25
claygnotmyname: there is zero value in having a "dead" drive in the ring... you might leave the failed drive unmounted in the chassis - but having it assigned parts is kinda mad21:25
*** cloudrancher has joined #openstack-meeting21:25
notmynameso with replicas, drive fails and gets 507s internally and swift works around it. if you remove it from the ring, things get faster, but leaving it doesn't stop any durability things. and with EC you're *required* to remove dead drives before rebalance?21:25
*** ralonsoh has quit IRC21:26
claygyeah... so the closest thing i can think of we have to a "durability config option" is handoffs_delete21:26
notmynameif that's true, I'm not sure it's bad, but I just want to make sure I clearly understand :-)21:26
claygso i'm not sure having "reconstructor extra sync nodes" option is a good idea - and I'm leaning towards 3... but it's a bit different than what we're doing now... and I don't think what we're doing now is very smart21:27
*** yamamoto has quit IRC21:27
*** cloudran_ has quit IRC21:27
*** yamamoto has joined #openstack-meeting21:27
clayguhh... no i don't guess you're "required" to remove dead drives... but I guess I hadn't considered throughly what happens if you DON'T because it seemed odd not to21:28
notmynamegut-reaction, it seems like more "special knowledge" that should be more built-in. makes the "how to run swift" question harder (and it's already hard).21:29
claygon replicated - yes any part-replicas still assigned to the unmounted drive (which could change but also might be largely the same) would have to be re-replicated to a new handoff21:29
notmynameor rather, I wonder what sort of automation could be more built in (eg only running handoffs-only for ec policies after a rebalance)21:29
claygthe handoff from the last ring will try to sync back to the primaries... and might fail to reap unless you have handoffs_delete set to two21:30
*** TxGirlGeek has joined #openstack-meeting21:30
rlediseznotmyname: on that last point, we do that by distributing the builder (it's dirty, but efficient): https://review.openstack.org/#/c/389676/21:30
patchbotpatch 389676 - swift - WIP: reconstructor: do not sync recently moved parts - 2 patch sets21:30
claygyeah having to manage your ec rebalance modes is super annoying 😡21:30
*** jamesmcarthur has quit IRC21:30
*** a-pugachev has quit IRC21:31
notmynameI think my questions/comments/thoughts are a bit of a slight distraction from your original question. you're basically asking if the proposed strategy has obvious correctness issues21:31
*** jamesmcarthur has joined #openstack-meeting21:32
*** jamesmcarthur has quit IRC21:32
claygwell, I don't think it has "obvious correctness" issues - I think the existing code is "obviously superfluous" - i was hoping maybe we might agree "yeah that code should change, go ahead" and then if I'm lucky I'd hear "yeah 3 nodes checking on each frag is probably about right" or "shouldn't it work like XYZ"21:33
*** macza has joined #openstack-meeting21:33
notmynameto the correctness question, as long as neighboring fragments are in different durability zones, then it's probably fine. (right?) for example, you don't want a rack to fail and then have a big enough gap in the available frag set so that they gap doesn't get reconstructed21:33
claygso handoffs go out of their way to bounce around failure domains - but I can't say for sure there wouldn't be a bad sequence for any given part of any given ring - we're just not that regimented in how we structure handoff sequences - and skipping around the handoffs via modulo might exacerbate the issue for a given frag_index in a given part21:35
claygbut I definitely don't have a better idea there21:35
notmynamethat makes sense for handoffs. not sure we can do any better there (and choosing something modulo or random likely won't be any better than neighbors)21:35
clayg... and even imagining failure domain distribution is imperfect - having a copy of the frag rebuild in ANY failure domain is better than not having the fragment because you're using fail-in-place and didn't know about lp bug #151034221:35
openstackLaunchpad bug 1510342 in OpenStack Object Storage (swift) "Reconstructor does not restore a fragment to a handoff node" [Medium,Confirmed] https://launchpad.net/bugs/151034221:35
mattoliverauWell, disks will fail, and duribiltiy is most important, so yes I agree once we unmount we should do something. We do it in Replication so we should also do it in EC. So I think it's the right move.21:36
notmynamebut if frags are only checking neighbors, what if primaries 2, 3, and 4 are down? what does 1 check? what does 5 check? does 3 ever get reconstructed?21:36
clayg@notmyname right - I think that thing (broken chains) is what prompted the weird existing behavior21:36
notmynameexisting == check everything?21:37
clayg(yes)21:37
*** a-pugachev has joined #openstack-meeting21:37
clayg3 is in a tough spot because TWO of the nodes that might check on him are down/unmounted otherwise not checking on him21:37
mattoliverauGetting more nodes from a ring tends to hit the same nodes, so left + right, or L + R + middle sounds good. But seeing as get more nodes should get the same, can't we just use them.. and trust the algorithm to keep them in different domains.21:37
clayg... so that lead me to maybe we should have *three* nodes check on each frag (i have a picture in my notebook) 😁21:38
claygnotmyname: also I think you're up-to-speed - you're thinking about the problem I'm worried about - how can I not check everything but still be "good enough"21:38
notmynameclayg: if I understand correctly, then yeah, I like the idea of your 3-way checks. 2 neighbors + one "across" the ring21:38
claygmattoliverau: yeah I think we can just trust the ring rebalance abstraction from this context until/unless we see a bad behavior (and then we should fix the ring - not the code that's expect it to do it's job)21:39
notmynameand eg with 12 fragments, it's likely good enough without being too expensive. probably should do some mathy things to find the exact numbers21:39
claygnotmyname: right mathy things... kota_ you're good at math!?  timburke is too... is he around?21:40
notmynamerledisez: kota_: tdasilva: m_kazuhiro: does all this make sense?21:40
claygs/all/any of/21:40
notmynametimburke got sick kids and is on PTO today :-(21:40
claygwe're doomed21:40
kota_still catching up... now21:41
m_kazuhirome too21:41
mattoliverauanyway may thoughts are, yup we need to do something. Go make it better. It sounds better then what we have.. well actually a pretty good idea. I'll checkout the code tomorrow and see if I get anymore questions, or bad suggestions ;)21:41
rledisezi *think* i get the idea of the 3-way checks, but I need to think more of it. it's way too complex for my timezone :)21:41
notmynameyeah, that went pretty quickly. I know it's hard to follow fast-moving, technical discussions in your non-native language :-)21:41
kota_got stomach ache and back here, long logs :P21:42
mattoliveraurledisez: +1 :P21:42
claygok good - so maybe this was the sanity check - and I'll keep going - folks can think more about it and maybe next week we'll have something in good shape to discuss... that would work ok with my timelines21:43
notmynameclayg: ok, so assuming kota_ and m_kazuhiro and rledisez and mattoliverau will be able to sleep/wake-up and think more about the code and ideas, did you get answers you can use?21:43
notmynameclayg: good :-)21:43
claygnotmyname: 👍21:43
kota_yup21:43
m_kazuhirook21:43
notmynameclayg: thanks for working on this. it's a hard problem that's important to solve, and I'm glad you're on it :-)21:43
mattoliverau+121:43
claygno worries21:43
notmyname#topic s3 versioning21:43
*** openstack changes topic to "s3 versioning (Meeting topic: swift)"21:44
claygzohno, no tim!21:44
notmynameon behalf of tim, I wanted to mention something, but only for "Read this". not to discuss right now21:44
notmyname#link https://gist.github.com/tipabu/4d49516050f6762ce9cf6b2ebb49e54521:44
notmynamebasically, tim is working on getting s3 versioning support. he's run in to a few interesting problems to solve with it. some of which have API implications. so he wrote up that doc to cover the issues and his thoughts on solutions21:45
notmynameI know he'd appreciate people reading over it to understand what's going on so it can be talked about in more detail later21:45
notmyname#topic open discussion21:45
*** openstack changes topic to "open discussion (Meeting topic: swift)"21:45
notmynameany other topics to bring up this week?21:46
notmynamethe combined ptg/summit in denver is currently accepting presentation proposals. the deadline is next week21:46
notmyname#link https://www.openstack.org/summit/denver-2019/call-for-presentations/21:46
*** abishop has quit IRC21:46
notmynamedoesn't sound like anything else, so let's close the meeting21:48
notmynamethanks for coming this week21:48
notmynamethanks for your work on swift21:48
notmyname#endmeeting21:48
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"21:48
openstackMeeting ended Wed Jan 16 21:48:54 2019 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)21:48
openstackMinutes:        http://eavesdrop.openstack.org/meetings/swift/2019/swift.2019-01-16-21.00.html21:48
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/swift/2019/swift.2019-01-16-21.00.txt21:48
openstackLog:            http://eavesdrop.openstack.org/meetings/swift/2019/swift.2019-01-16-21.00.log.html21:48
mattoliveraunight all21:49
*** patchbot has left #openstack-meeting21:49
*** m_kazuhiro has quit IRC21:50
*** gouthamr has quit IRC21:56
*** gouthamr_ is now known as gouthamr21:56
*** jamesmcarthur has joined #openstack-meeting21:58
*** priteau has joined #openstack-meeting22:00
*** priteau has quit IRC22:01
*** jamesmcarthur has quit IRC22:02
*** bbowen has quit IRC22:04
*** mjturek has quit IRC22:05
*** imacdonn has quit IRC22:07
*** imacdonn has joined #openstack-meeting22:07
*** yamamoto has quit IRC22:14
*** bbowen has joined #openstack-meeting22:18
*** ijw has quit IRC22:24
*** yamamoto has joined #openstack-meeting22:29
*** erlon has quit IRC22:29
*** factor has quit IRC22:30
*** munimeha1 has quit IRC22:30
*** yamamoto has quit IRC22:33
*** ijw has joined #openstack-meeting22:35
*** ijw has quit IRC22:40
*** rcernin has joined #openstack-meeting22:45
*** mattw4 has quit IRC22:46
*** ijw has joined #openstack-meeting22:52
*** slaweq has quit IRC23:03
*** TxGirlGeek has quit IRC23:07
*** TxGirlGeek has joined #openstack-meeting23:12
*** zaitcev has left #openstack-meeting23:14
*** slaweq has joined #openstack-meeting23:29
*** efried has quit IRC23:30
*** enriquetaso has quit IRC23:30
*** slaweq has quit IRC23:34
*** ijw has quit IRC23:35
*** efried has joined #openstack-meeting23:43
*** hongbin has quit IRC23:45
*** ttsiouts has joined #openstack-meeting23:50
*** ijw has joined #openstack-meeting23:50
*** a-pugachev has quit IRC23:53
*** ijw has quit IRC23:55
*** ttsiouts has quit IRC23:57
*** devananda has joined #openstack-meeting23:58
*** ttsiouts has joined #openstack-meeting23:58

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!