Thursday, 2019-05-09

*** jamesmcarthur has joined #openstack-meeting-alt00:04
*** jamesmcarthur has quit IRC00:05
*** jamesmcarthur has joined #openstack-meeting-alt00:05
*** lseki has quit IRC00:09
*** ttsiouts has quit IRC00:12
*** yamamoto has quit IRC00:18
*** ijw has joined #openstack-meeting-alt00:19
*** ijw has quit IRC00:27
*** gyee has quit IRC00:38
*** licanwei has quit IRC01:01
*** jamesmcarthur has quit IRC01:12
*** baojg has joined #openstack-meeting-alt01:32
*** yamamoto has joined #openstack-meeting-alt01:41
*** jamesmcarthur has joined #openstack-meeting-alt01:43
*** jamesmcarthur has quit IRC01:44
*** jamesmcarthur_ has joined #openstack-meeting-alt01:44
*** apetrich has quit IRC01:56
*** yamamoto has quit IRC02:03
*** bhavikdbavishi has joined #openstack-meeting-alt02:04
*** ttsiouts has joined #openstack-meeting-alt02:09
*** bhavikdbavishi has quit IRC02:15
*** bhavikdbavishi has joined #openstack-meeting-alt02:20
*** iyamahat has quit IRC02:24
*** yamahata has quit IRC02:26
*** jamesmcarthur_ has quit IRC02:33
*** jamesmcarthur has joined #openstack-meeting-alt02:34
*** yamamoto has joined #openstack-meeting-alt02:39
*** ttsiouts has quit IRC02:42
*** yamamoto has quit IRC02:51
*** yamamoto has joined #openstack-meeting-alt02:54
*** jamesmcarthur has quit IRC03:08
*** bhavikdbavishi has quit IRC03:10
*** jamesmcarthur has joined #openstack-meeting-alt03:11
*** ayoung has joined #openstack-meeting-alt03:27
*** jamesmcarthur has quit IRC03:34
*** ayoung has quit IRC03:42
*** e0ne has joined #openstack-meeting-alt03:47
*** lpetrut has joined #openstack-meeting-alt03:53
*** jamesmcarthur has joined #openstack-meeting-alt04:05
*** lpetrut has quit IRC04:18
*** janki has joined #openstack-meeting-alt04:38
*** ttsiouts has joined #openstack-meeting-alt04:39
*** e0ne has quit IRC04:46
*** ttsiouts has quit IRC05:12
*** radeks has joined #openstack-meeting-alt05:35
*** yamamoto has quit IRC05:42
*** bhavikdbavishi has joined #openstack-meeting-alt05:44
*** yamamoto has joined #openstack-meeting-alt05:46
*** yamamoto has quit IRC05:51
*** vishalmanchanda has joined #openstack-meeting-alt05:58
*** belmoreira has joined #openstack-meeting-alt06:06
*** slaweq has joined #openstack-meeting-alt06:06
*** whoami-rajat has joined #openstack-meeting-alt06:20
*** yamamoto has joined #openstack-meeting-alt06:28
*** yamamoto has quit IRC06:29
*** yamamoto has joined #openstack-meeting-alt06:30
*** kopecmartin|off is now known as kopecmartin06:54
*** apetrich has joined #openstack-meeting-alt06:59
*** tesseract has joined #openstack-meeting-alt07:09
*** ttsiouts has joined #openstack-meeting-alt07:09
*** panda|off is now known as panda07:19
*** rdopiera has joined #openstack-meeting-alt07:24
*** ttsiouts has quit IRC07:42
*** ttsiouts has joined #openstack-meeting-alt07:46
*** belmoreira has quit IRC07:55
*** belmoreira has joined #openstack-meeting-alt07:58
*** e0ne has joined #openstack-meeting-alt08:14
*** e0ne has quit IRC08:17
*** jtomasek has joined #openstack-meeting-alt08:37
*** panda has quit IRC08:55
*** panda has joined #openstack-meeting-alt08:55
*** ttsiouts has quit IRC09:15
*** tssurya has joined #openstack-meeting-alt09:18
*** wxy-xiyuan has quit IRC09:26
*** belmoreira has quit IRC09:33
*** e0ne has joined #openstack-meeting-alt09:38
*** e0ne has quit IRC09:40
*** yamamoto has quit IRC09:45
*** belmoreira has joined #openstack-meeting-alt09:48
*** e0ne has joined #openstack-meeting-alt09:56
*** tssurya has quit IRC09:57
*** e0ne has quit IRC09:59
*** raildo has joined #openstack-meeting-alt10:00
*** yamamoto has joined #openstack-meeting-alt10:06
*** mugsie has quit IRC10:25
*** yamamoto has quit IRC10:27
*** yamamoto has joined #openstack-meeting-alt10:27
*** yamamoto has quit IRC10:28
*** ttsiouts has joined #openstack-meeting-alt10:31
*** yamamoto has joined #openstack-meeting-alt10:32
*** yamamoto has quit IRC10:32
*** yamamoto_ has joined #openstack-meeting-alt10:32
*** yamamoto_ has quit IRC10:33
*** mugsie has joined #openstack-meeting-alt10:35
*** mugsie has quit IRC10:35
*** mugsie has joined #openstack-meeting-alt10:36
*** ttsiouts has quit IRC10:36
*** mugsie has quit IRC10:38
*** mugsie has joined #openstack-meeting-alt10:39
*** tesseract has quit IRC10:40
*** tesseract has joined #openstack-meeting-alt10:41
*** tssurya has joined #openstack-meeting-alt10:42
*** priteau has joined #openstack-meeting-alt10:43
*** tesseract has quit IRC10:45
*** tesseract has joined #openstack-meeting-alt10:45
*** belmoreira has quit IRC10:55
*** belmoreira has joined #openstack-meeting-alt10:56
*** panda is now known as panda|lunch11:02
*** yamamoto has joined #openstack-meeting-alt11:08
*** carloss has joined #openstack-meeting-alt11:09
*** yamamoto has quit IRC11:13
*** belmoreira has quit IRC11:15
*** priteau has quit IRC11:23
*** yamamoto has joined #openstack-meeting-alt11:34
*** belmoreira has joined #openstack-meeting-alt11:48
*** rf0lc0 is now known as rfolco|ruck11:49
*** janki has quit IRC11:54
*** zigo has quit IRC11:59
*** yamamoto has quit IRC12:02
*** belmoreira has quit IRC12:03
*** panda|lunch is now known as panda12:06
*** yamamoto has joined #openstack-meeting-alt12:07
*** baojg has quit IRC12:08
*** iyamahat has joined #openstack-meeting-alt12:15
*** jamesmcarthur has quit IRC12:21
*** jamesmcarthur has joined #openstack-meeting-alt12:21
*** iyamahat has quit IRC12:21
*** zigo has joined #openstack-meeting-alt12:23
*** jamesmcarthur has quit IRC12:30
*** yamamoto has quit IRC12:40
*** belmoreira has joined #openstack-meeting-alt12:49
*** jamesmcarthur has joined #openstack-meeting-alt12:49
*** lseki has joined #openstack-meeting-alt12:51
*** jamesmcarthur has quit IRC13:05
*** yamamoto has joined #openstack-meeting-alt13:06
*** baojg has joined #openstack-meeting-alt13:20
*** ttsiouts has joined #openstack-meeting-alt13:24
*** lbragstad_ has joined #openstack-meeting-alt13:29
*** lbragstad_ is now known as lbragstad13:30
*** tesseract has quit IRC13:35
*** tesseract has joined #openstack-meeting-alt13:35
*** altlogbot_2 has quit IRC13:43
*** altlogbot_2 has joined #openstack-meeting-alt13:45
*** redrobot has quit IRC13:46
*** helenafm has joined #openstack-meeting-alt13:47
*** Guest89536 has joined #openstack-meeting-alt13:49
*** vhari has joined #openstack-meeting-alt13:50
*** Guest89536 is now known as redrobot13:55
*** vishalmanchanda has quit IRC13:57
*** bhavikdbavishi has quit IRC14:02
*** ttsiouts has quit IRC14:08
*** ccamacho has joined #openstack-meeting-alt14:21
*** ttsiouts has joined #openstack-meeting-alt14:23
*** tesseract has quit IRC14:24
*** liuyulong has joined #openstack-meeting-alt14:25
*** ttsiouts has quit IRC14:28
*** yamamoto has quit IRC14:35
*** ganso has joined #openstack-meeting-alt14:44
*** tesseract has joined #openstack-meeting-alt14:50
*** tesseract has quit IRC14:51
*** tesseract has joined #openstack-meeting-alt14:51
*** hongbin has joined #openstack-meeting-alt14:56
*** ttsiouts has joined #openstack-meeting-alt14:59
gouthamr#startmeeting manila15:00
openstackMeeting started Thu May  9 15:00:21 2019 UTC and is due to finish in 60 minutes.  The chair is gouthamr. Information about MeetBot at http://wiki.debian.org/MeetBot.15:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:00
*** openstack changes topic to " (Meeting topic: manila)"15:00
openstackThe meeting name has been set to 'manila'15:00
gouthamrcourtesy ping: xyang toabctl bswartz ganso erlon tpsilva vkmc amito jgrosso15:00
lsekio/15:00
carlossHey :)15:00
gansohello15:00
gouthamrhello o/15:00
*** dviroel has joined #openstack-meeting-alt15:00
gouthamrtbarron is away, and i'm chairing today - my first in a long time so expect some clumsiness :)15:00
*** tzumainn has joined #openstack-meeting-alt15:00
*** tesseract has quit IRC15:01
bswartz.o/15:01
gouthamrwe'll wait a couple of minutes for quorum15:01
vharihi15:01
dviroelo/15:01
amitoo/15:01
bswartzOh good we can trout slap the chair today15:01
bswartz:-)15:01
gouthamr:D15:01
gouthamr#chair bswartz15:01
openstackCurrent chairs: bswartz gouthamr15:01
bswartzDoh!15:01
gouthamrnow we can15:01
gouthamr#topic Announcements15:02
*** openstack changes topic to "Announcements (Meeting topic: manila)"15:02
*** jgrosso has joined #openstack-meeting-alt15:02
gouthamrWelcome back, everyone that traveled to the mile high city15:02
gouthamrand thanks everyone that joined us remote :)15:03
vkmco/15:03
gouthamrwe had a great PTG and Summit; and it felt good to be back to the old times personally15:03
*** tesseract has joined #openstack-meeting-alt15:03
bswartzDid it feel like old times with the updated format?15:04
bswartzThat what I would have hoped for15:04
*** ttsiouts has quit IRC15:04
gouthamryes, felt as packed as BCN, which i recall was the last combined conference (Summit + Design Summit)15:04
gouthamrthe PTG seemed to have a good turn out, comparable to the exclusive event in Denver last cycle, although there were a few new projects represented15:05
gouthamrwe have a lot to follow up from the PTG ourselves - a day and a half seemed productive and discussions were well paced15:06
gouthamrmore on this later..15:06
gouthamrin terms of announcements, I proposed the change we discussed at the PTG: A Parking lot for old unimplemented specs15:07
gouthamr#LINK https://review.opendev.org/#/c/657894/15:07
gouthamrfound a few ones that we might want to move there, please take a look15:08
vkmclike the "parking lot" term15:08
gouthamrif you have a spec there that needs to be moved to the Train release and not the "unimplemented" section, please note it in review15:09
vkmcdo we want to clean up those specs? say, change the expected release and asignee fields15:09
gouthamrvkmc: yes, the act of moving the specs out of the release specific folder should signify the former15:10
gouthamrvkmc: however, i dunno if we should pop out the assignee/s field..15:10
gouthamrvkmc: that is probably a good idea, however - we can use another field that suggests the design author, and leave the assignee field empty15:11
*** tesseract has quit IRC15:11
vkmcgouthamr, yeah, I mean, in the blueprint you can see who proposed what15:11
vkmcso the authorship remains the same15:12
*** yamamoto has joined #openstack-meeting-alt15:12
vkmcbut... having the assignee field filled in might prevent people to take the blueprint and work on it15:12
gouthamrtrue true... if the authors are available to consult, but not code, it might make sense for the new owner to reach out to them15:13
* gouthamr labels himself design consultant, prints new business cards15:13
*** jamesmcarthur has joined #openstack-meeting-alt15:13
vkmc:)15:14
gouthamrso PTAL, everyone, and we'll take these discussions to the review15:14
gouthamr#topic Summit (29 April 29-May 1) and PTG (May 2-4) summary, action items15:15
*** openstack changes topic to "Summit (29 April 29-May 1) and PTG (May 2-4) summary, action items (Meeting topic: manila)"15:15
gouthamr#LINK https://etherpad.openstack.org/p/manila-ptg-train15:15
*** tesseract has joined #openstack-meeting-alt15:16
gouthamralright, on this one, thanks to note takers, we have a decent summary against each topic and action items listed underneath15:16
gouthamrwe did not cover all the topics we had proposed, so we'll schedule those topics in our weekly meetings if the proposers can participate15:17
gouthamrtbarron is working on an upstream summary, and will post one to the ML (openstack-discuss) soon-ish (TM)15:17
*** yamamoto has quit IRC15:17
gouthamrif he doesn't, vkmc and I'll channel him and do it anyway :)15:18
gouthamrthank you NetApp folks (erlon, dviroel, lseki) for getting us AV support15:18
*** tesseract has quit IRC15:18
gouthamrand thank you vkmc for hosting the bluejeans session for our remote attendees and for recording the whole thing15:18
vkmc++15:19
lseki :-)15:19
dviroel:)15:19
gouthamrwe'll plan on posting the recording along with the summary to the mailing list15:19
*** turnerg has joined #openstack-meeting-alt15:19
gouthamrdoes anyone have anything to say about this $topic?15:19
* gouthamr hears crickets, sees smiles, moves on :)15:20
gouthamr#topic Train Schedule15:20
*** openstack changes topic to "Train Schedule (Meeting topic: manila)"15:20
*** tesseract has joined #openstack-meeting-alt15:20
bswartz#pun15:20
*** dtrainor has joined #openstack-meeting-alt15:20
gouthamrwee15:20
gouthamrooh, if you got lucky, you could get your own train whistle at the PTG and blow it on the corridors15:21
gouthamrand attract annoyed faces :)15:21
bswartzHeh15:21
bswartzWhat is "train" a reference to?15:22
gouthamrhe asks ^15:22
bswartzIs it the loud train near the PTG hotel?15:22
gansobswartz: the trains in denver close to the hotel15:22
*** helenafm has left #openstack-meeting-alt15:22
bswartzK15:22
gouthamrtbarron flagged off a review for the manila release schedule here15:23
gouthamr#LINK https://review.opendev.org/#/c/655667/15:23
vkmcI think it was the only release for OpenStack that was named because everybody got nuts with the naming idea on the denver ptg 2018 feedback session15:23
gouthamrit overlays the openstack release schedule with the manila project specific deadlines15:23
gouthamrplease visit that, and provide your reviews15:24
gouthamrthere's also our own tracking wiki15:24
gouthamr#LINK https://wiki.openstack.org/wiki/Manila/TrainCycle15:25
gouthamrit's only beginning to be filled15:25
gouthamrPTAL (heh, GitHub PRs you ruined me)15:25
gouthamrwe'll track our work on that wiki page, so feel free to add your own work items, and status as we proceed through the release15:26
bswartzvkmc: I think people couldn't resist the puns15:26
gouthamrabsolutely, hop on the choochoostack15:27
vkmcbswartz, exactly, that made it special :)15:27
vkmchaha15:27
gouthamrfrom next week, we'll be the fare enforcement officers on this train, and check your statuses for the work on the wiki15:28
gouthamrmoving on..15:28
gouthamr#topic Extend share when clients are connected (gouthamr)15:28
*** openstack changes topic to "Extend share when clients are connected (gouthamr) (Meeting topic: manila)"15:28
gouthamralright, that's me pinging me15:29
gouthamrthis could be a known issue for our veteran contributors who at one point or another worked with/on the generic driver15:29
gouthamr#LINK https://review.opendev.org/#/c/531568/15:29
gouthamr^ is a scenario test - a test case suggested by vponomaryov15:30
bswartzThis is a good one15:30
* gouthamr misses the dude15:30
bswartzThe dude abides15:31
bswartzWhat's the issue here though?15:31
bswartzDoes it not work?15:31
gouthamrso the problem here is the mount becomes stale after issuing an extend API to manila15:31
gouthamrthe mount inside the nova VM*15:32
gouthamrthe test logs from a previous CI run are here:15:32
gouthamr#LINK http://logs.openstack.org/68/531568/26/check/manila-tempest-dsvm-scenario/1ea651d/logs/testr_results.html.gz15:32
gouthamrthe error of interest is "dd: failed to open '/mnt/t3': Stale file handle"15:32
bswartzYeah that would be expected for some drivers15:33
gouthamrnirg22, who is implementing this test case for us is confused, because it works on the other first party drivers - LVM and ZFSOnLinux15:33
bswartzWe were not strict about the expectation on the client side, because we have no way to prevent this kind of problem15:33
bswartzSometimes a resize will require a remount, and sometimes it won't15:34
bswartzWe don't tell the user which situation they're in15:34
bswartzThey just have to know, or learn empircally15:34
gouthamrwont shares from the generic driver always require a remount?15:34
bswartzYes15:34
bswartzBut the user doesn't know which driver they've got15:34
gouthamrtrue15:35
gouthamri wonder if we can use the "online extend" capability in the cinder API15:35
gouthamralthough it comes with its own caveats15:35
bswartzDoes LVM suffer from this at all?15:36
bswartzMaybe the answer is to stop using Generic15:37
*** baojg has quit IRC15:37
gouthamrbswartz: nope, LVM passes the test: http://logs.openstack.org/68/531568/26/check/manila-tempest-minimal-dsvm-lvm/130046f/logs/testr_results.html.gz15:37
gouthamr"manila_tempest_tests.tests.scenario.test_share_extend.TestShareExtendNFS" is the test to look for there ^15:37
gansogouthamr: generic is cinder using LVM underneath, but it happens because here we need to use resize2fs15:38
gansowait a min, LVM also uses resize2fs15:38
gouthamrganso: so the client (generic driver share server) needs to issue a resize2fs?15:39
gouthamrganso: or do you mean this is if we were to use the new online-extension API15:39
gansogouthamr: I'd say it would need to whatever the LVM is doing, because essentially they would work the same in this situation15:39
bswartzNo15:39
gansogouthamr: s/would need to/would need to do15:40
bswartzCinder is not the same as LVM15:40
bswartzThe problem is the block device resize, not the filesystem resize15:40
bswartzLVs can be resized online15:40
gansogouthamr: what would be the caveat of cinder online extend?15:40
gouthamrganso: the limitations around where it can be used - not all cinder backends support extending attached volumes iirc, and not all nova hypervisors support it either (only libvirt driver has support last i looked, but i could be wrong)15:41
gouthamrbswartz: on stopping the use of the generic driver :) - this is the only job where scenario tests are running with DHSS=True15:42
gouthamrbswartz: we did take an AI to run scenario tests in the container job...15:43
gansogouthamr: what if we make it configurable? the main importance of the generic driver is in the gate or POC deployments15:43
bswartzIf we can use cinder online resize we might be able to work around this issue15:43
bswartzBut the deeper question is what guarantees we make to our end users15:43
gouthamrack15:44
gansogouthamr: if a cloud is using the generic driver in production with a backend that wouldn't work with online extend, just disable it and we fallback to the other problem (if there is no other way to solve the problem)15:44
bswartzIf we want to guarantee online resizes, then certain drivers may not be able to fulfill the contract15:44
bswartzIf we don't want to guarantee online resizes, then the test needs to accommodate the drivers that can't, which probably requires communicating some extra information at the API level15:44
gouthamrganso: yes, that would be a good direction to take - unless we want to extract the cinder pools information, and check for the capability atleast from cinder's side and fail faster15:44
gansogouthamr: that is a good idea as well, but the history that I know of online extend capability is a messy one15:45
gansogouthamr: it was enabled by default for everyone15:45
gouthamrbswartz: makes sense, a new capability flag that defaults to True?15:45
gansogouthamr: and then the vendors that do not support it need to disable15:45
bswartzganso: We have to make the first decision before we discuss the mechanism for the second15:46
gansogouthamr, bswartz: Why a capability and not a config option just for the generic driver?15:46
gouthamrganso bswartz: ^ yeah, we don't know what other drivers suffer this problem15:47
bswartzThe question is, can users assume nondisruptive resize or not15:47
bswartzIf no, then how do they find out if it's disruptive or not15:47
bswartzI kinda like assuming nondisruptive15:48
bswartzBut we should figure which drivers that might break15:48
*** priteau has joined #openstack-meeting-alt15:48
gansothe scenario test should point that out15:48
gansoproblem is 3rd parties are not running scenario tests15:49
gouthamr+1 the test case looks good right now, and everyone needs to be running it and testing the various backends we have15:49
gansoso the only way to obtain this information is mailing list?15:49
*** belmoreira has quit IRC15:49
gouthamrganso: yes, sadly, we have one or two AIs about scenario tests from the PTG15:49
gouthamrpart of the problem is our own inconsistent testing in the first party drivers15:50
bswartzWell with this new scenarios test we are de-facto assuming that resizes are nondisruptive15:50
bswartzSo maybe we just merge it, fix (or kill) the generic driver, and see what other drivers fail the scenario test15:50
bswartzBecause the status quo is to not tell users if the resize is disruptive15:51
gouthamri wonder if we can try...except and log a warning saying a remount was required to pass the test15:51
*** kopecmartin is now known as kopecmartin|off15:51
gansogouthamr: what about the user experience then?15:51
bswartzWe can later choose to relax the contstraints by communicating about disruptive resizes if we decide we want to continue to allow them15:51
gansogouthamr: and the user's expectations15:51
bswartzganso: My suggestion leaves things unchanged, user-experience-wise, until we decide to relax things.15:52
gansoLGTM15:52
gouthamrganso: although i will take an AI to bring this discussion on the ML for the wider audience15:53
gouthamri know that NetApp and Dell/EMC want to run our scenario tests15:53
gouthamrwould be nice to have all the other drivers do the same15:53
* gouthamr recalls a bug downstream that vhari pointed out regarding a manila/CIFS scenario test15:54
jgrossogouthamr I want to run your scenario tests also !15:54
*** gyee has joined #openstack-meeting-alt15:54
gouthamrjgrosso++15:55
vkmcjgrosso++15:55
vharitempest_manila_plugin.tests.scenario.test_shares_scenario.SharesScenarioTest.test_scenario_115:55
*** priteau has quit IRC15:56
gouthamrvhari: that one looks like an internal test..15:56
gouthamri'm thinking about this one: https://bugzilla.redhat.com/show_bug.cgi?id=170318515:56
openstackbugzilla.redhat.com bug 1703185 in python-manila-tests-tempest "Tempest manila test fail" [Unspecified,New] - Assigned to rhos-maint15:56
vharihere is the CIFS one: manila_tempest_tests.tests.scenario.test_share_basic_ops.TestShareBasicOpsCIFS)15:57
gouthamrack, ty :)15:57
vhari:D15:57
gouthamralright, we're almost out of time... this was a great discussion, and we have some AIs15:57
*** bnemec has quit IRC15:58
*** priteau has joined #openstack-meeting-alt15:58
gouthamr#action: suggest next steps for the scenario test regarding the generic driver failure, maintain current user experience/expectations15:58
*** ijw has joined #openstack-meeting-alt15:58
*** bnemec has joined #openstack-meeting-alt15:58
jgrossoThank you Goutham I don't have anyone new bugs to discuss today !15:58
jgrosso:)15:59
jgrossobe prepared next week!15:59
gouthamr#action: raise issue on the ML, point to the scenario test - propose a solution to provide a consistent user experience by claiming support (or lack thereof) of online/nondisruptive extensions15:59
gouthamrcool, ty jgrosso15:59
gouthamr#topic Open Discussion15:59
*** openstack changes topic to "Open Discussion (Meeting topic: manila)"15:59
bswartzThx for chairing gouthamr15:59
jgrossogouthamr: welcome15:59
gouthamrwe have 1 minute :)15:59
gouthamranything anyone wants in the meeting logs before we all retire to #openstack-manila?15:59
* bswartz reaches for large trout16:00
gouthamroh ho, we're out of time :)16:00
gouthamr#endmeeting16:00
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"16:00
openstackMeeting ended Thu May  9 16:00:24 2019 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:00
openstackMinutes:        http://eavesdrop.openstack.org/meetings/manila/2019/manila.2019-05-09-15.00.html16:00
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/manila/2019/manila.2019-05-09-15.00.txt16:00
openstackLog:            http://eavesdrop.openstack.org/meetings/manila/2019/manila.2019-05-09-15.00.log.html16:00
*** dviroel has left #openstack-meeting-alt16:00
gouthamrthank you everyone, see you all in #openstack-manila16:00
priteau#startmeeting blazar16:01
* gouthamr reaches for trout shield, activates it, looks around suspiciously16:01
openstackMeeting started Thu May  9 16:01:15 2019 UTC and is due to finish in 60 minutes.  The chair is priteau. Information about MeetBot at http://wiki.debian.org/MeetBot.16:01
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:01
*** openstack changes topic to " (Meeting topic: blazar)"16:01
*** tesseract has quit IRC16:01
openstackThe meeting name has been set to 'blazar'16:01
priteau#topic Roll call16:01
*** openstack changes topic to "Roll call (Meeting topic: blazar)"16:01
priteauHi. This is the first Blazar meeting in the new timezone for Americas.16:03
*** bhavikdbavishi has joined #openstack-meeting-alt16:04
turnergmaybe everybody slept in16:05
priteauHi turnerg. Are you interested in Blazar?16:06
tzumainnhi! I haven't really been involved in blazar, but I read some notes from the ptg about possible future ironic integration, and thought that maybe I'd lurk a bit16:06
turnergYes;  we have some GPU's that we're thinking about how to make available on a fair basis16:06
*** diurnalist has joined #openstack-meeting-alt16:06
*** bhavikdbavishi has quit IRC16:07
turnergBeen thinking about Blazar off & on since 2016;  come from an HPC background where scheduling is a solved problem +/- error16:07
*** bhavikdbavishi has joined #openstack-meeting-alt16:07
turnergFor the time being our cloud is not at capacity but, someday, it's gonna happen16:08
priteauHi tzumainn. I remember your message on the mailing list back in February. You are working on the Hardware leasing with Ironic project, is that right?16:08
priteauHi diurnalist16:09
*** ttsiouts has joined #openstack-meeting-alt16:09
tzumainnpriteau, that's correct! I saw some notes from dtantsur in ironic that mentioned updating ironic with owner/lessee fields, and blazar possibly integrating with that16:09
priteauI am still expecting another person from University of Chicago to join, they're working through NickServ registration right now.16:09
priteauLet's start and they'll read through the logs later ;-)16:12
priteau#topic Presentation of participants16:12
*** openstack changes topic to "Presentation of participants (Meeting topic: blazar)"16:12
*** jakecoll has joined #openstack-meeting-alt16:13
priteauHi jakecoll16:13
jakecollHey, sorry about that16:13
priteauBecause this is the first meeting in this timezone, I thought it would be useful for everyone to present themselves and the OpenStack deployment or product they're working on, if any. And of course why they're interested in Blazar.16:13
priteauI'll start16:13
priteauI am Pierre Riteau, I am the PTL for Blazar in the Train cycle. I have used and developed Blazar since 2015 through the Chameleon testbed: https://www.chameleoncloud.org/16:14
priteauI am not directly involved with Chameleon anymore, but diurnalist and jakecoll so I'll let them talk about it16:15
priteauLet's go alphabetically. diurnalist?16:15
diurnalistI'm Jason Anderson, DevOps lead for Chameleon. Chameleon is a scientific testbed with ~500 bare metal nodes that we provision with Ironic. Users reserve nodes with Blazar, and we've also been working to extend the idea of reservations to VLAN segments and floating IPs. Blazar is a core part of how Chameleon works; we use it to track allocations on the testbed as well.16:17
priteauActually it will be too slow to do this one by one, everyone can present themselves now16:17
priteauThanks diurnalist16:17
priteauAnyone else? jakecoll, turnerg, tzumainn16:19
turnergI'm George Turner, operator with the Jetstream project https://jetstream-cloud.org/  a cloud for science users. We'd like to schedule/allocate GPUs for defined periods of time; something similar for classroom environment. We  have worked with some of the admins on Chameleon16:19
jakecollMy name is Jake Colleran. I'm Jason Anderson/diurnalist's lackey over on Chameleon.16:19
*** rdopiera has quit IRC16:20
tzumainnhi! I'm Tzu-Mainn Chen, and I'm interested in the idea of using Blazar to lease ironic nodes; I saw some PTG discussion on the ironic side about how to accommodate that, so I thought I'd peek in here to see what future plans might be16:20
priteauturnerg: Nice to see some interest from Jetstream. Do you already have GPUs that users keep allocated for too long?16:21
turnergwanna head that rpoblem off before ti happens ;)    we have some gpus on order; they're not GA yet16:22
priteauBlazar allows users to define exactly when they want to allocate specific kind of resources, though it needs to be used with a policy to enforce sharing16:23
priteauI know Chameleon had to tune its policies because GPU users kept reserving them for long periods16:24
*** diurnalist_ has joined #openstack-meeting-alt16:24
diurnalist_I temporarily dropped out; I blame mozilla16:25
priteauBTW if you get disconnected, you can read the logs at http://eavesdrop.openstack.org/meetings/blazar/2019/blazar.2019-05-09-16.01.log.txt16:25
jakecollWorth mentioning, we have had problems with users stacking leases, especially gpus.16:25
diurnalist_priteau: thanks16:25
*** diurnalist has quit IRC16:26
priteaujakecoll: can you share what is the latest policy in use on Chameleon?16:26
*** diurnalist_ is now known as diurnalist16:27
jakecollWe've developed a policy where users cannot have advanced reservations that back up against each other over 2 days. However, we enforce this with our own scripts that operate outside blazar itself.16:27
priteauturnerg: out of the box Blazar doesn't provide any limitations on reservations, so users can reserve GPU nodes even more easily than they can launch instances (which would be subject to quota). Chameleon has developed extensions to limit reservation length, but users were working around that by making multiple advance reservations one after the other.16:28
priteaujakecoll: so if a reservation finishes on a Thursday at noon, the user cannot reserve the same resource until Saturday noon?16:29
*** stephenfin has quit IRC16:30
jakecollpriteau: Yes. Otherwise, the later advanced reservation will be deleted and an email notification sent.16:30
*** diablo_rojo has joined #openstack-meeting-alt16:31
priteauIt would be good to see if some enforcement policies are common enough that they could be integrated in upstream Blazar.16:32
diurnalistWe have had to introduce some pretty wacky policies due to contention over those highly-prized resources. I'm sure there is a better way of expressing the rule. We considered limiting a user/project to having N leases at a given time. However, this could require that users learn some more of the advanced features of reservations, like the fact that one lease can contain many reservations for different resource types16:32
priteauturnerg: How do you envision users would share your GPU resources? Do you have some kind of usage charged against an allocation?16:33
diurnalistpriteau: correct me if I'm wrong, but Blazar currently has no mechanism to actually charge for usage, correct? it just can handle advanced scheduling of time allotments on a resource.16:34
*** jamesmcarthur has quit IRC16:34
diurnalistbecause I've wondered if this is something that has already been discussed, and what changes might be necessary for Chameleon's solution to this to make it upstream.16:34
turnergasorry; dealing with an interrupt16:36
priteaudiurnalist: That's correct. Unfortunately I am not aware of a standard solution in this space. OpenStack has CloudKitty but I think it only computes usage after the fact and can convert it in $$$ to charge a credit card.16:36
*** tssurya has quit IRC16:38
priteauMaybe some reservation usage could be expressed like other quotas. Each OpenStack project could have a quota of "seconds/hours/days of advance reservation time" and Blazar could check against it.16:40
priteauKeystone is working on unified limits which may be relevant for this: https://docs.openstack.org/keystone/queens/admin/identity-unified-limits.html16:41
turnergWe would find it attractive to be able to charge for the tie reserved;  Users have ana llocation annd if they burn it then there's at least a cost.  We can slice & dice the GPUs down to 1/8 of a GPU; we do it by flavors16:43
priteauturnerg: That's what Chameleon does, using a custom allocation backend and extensions to Blazar. Do you already have a similar allocation system in place for your existing cloud?16:44
turnergReserving without using would be very bad. I could see how smart people could figure that one out16:44
*** ganso has left #openstack-meeting-alt16:44
diurnalistturnerg: yes. we found that users would set up scripts to automatically just keep making leases in the future for one node. one enterprising user effectively reserved one node for a few months until we spotted it16:45
turnergWe have no allocation process other then they use their XSEDE allocation, we go through by hand and disable the project16:45
diurnalistturnerg: do XSEDE allocations come in some sort of unit? like a compute hour. or is it more of a "license to compute"16:46
*** irclogbot_2 has quit IRC16:46
*** munimeha1 has joined #openstack-meeting-alt16:46
turnerghours * CPU_core16:46
*** irclogbot_1 has joined #openstack-meeting-alt16:47
priteauAre the XSEDE allocations stored in a custom database or is there some common cluster/grid software that handles this?16:47
turnergone SU = ONE core for one Hour.    We anticipate using an multipler to make the GPU more expensive16:47
turnergcustom16:47
diurnalistturnerg: that is exactly what we do. custom multipliers for some thing. we also allow users to reserve GENI slices. it's a VLAN tag but we have a 10x multiplier on those because we only have a few and they're very useful for SDN experimentation.16:48
turnergthinking out loud, if we had some phantom instance start up on the reserved resource and that had a high value charge per time yet consumned no resources on the host, then the u ser could launch thier instance with did real work on the reserved node but that instance would have a zero caahrge rate.  this way they'd be charged for the reservation without having to modify our current scripts that just log  the time instances are16:50
turnerg running16:50
priteauturnerg: That's fine for charging reservations that are running, but users who make advance reservations may still reserve more than they should.16:51
turnergah; thinking....16:51
priteauIf Blazar was making REST requests to a customisable endpoint on reservation creation / update, expecting to get a simple yes/no answer (with some details, like how much SUs are left compared to how much would be used), would people be motivated to write a small REST service making the link between Blazar and any custom allocation backend?16:52
turnergI guess the quoda above is a start16:52
priteauTime is flying and we're almost finished with the hour.16:54
priteauThis is a very interesting conversation, I think we should continue it either by email or at the next meeting in two weeks16:54
diurnalist:+1:16:54
priteauI just want to talk a bit about Ironic16:55
priteau#topic Summit and PTG feedback16:55
*** openstack changes topic to "Summit and PTG feedback (Meeting topic: blazar)"16:55
priteauI wasn't in Denver but I was told that Dmitry (from Ironic) and Tetsuro (from Blazar and Placement) had good discussions about how to make Blazar and Ironic standalone work.16:56
priteauThere are some details in Dmitry's email to the list: http://lists.openstack.org/pipermail/openstack-discuss/2019-May/005951.html16:57
tzumainnI read a summary from Dmitry that talked a bit about that, and mentioned that someone on the ironic side that would come up with a blueprint detailing what they'd do16:57
tzumainnwould there be something similar forthcoming on the blazar side?16:57
priteauI haven't had the chance to talk with Tetsuro yet, but I am expecting we would have some specs to match.16:58
tzumainnokay, cool! I'd be excited to read that and perhaps offer some feedback16:59
priteauThere's also a plan for Blazar to work with Nova + Ironic, which we're planning to work on during this release cycle.16:59
priteautzumainn: You are specifically interested in Ironic standalone?17:00
tzumainnpriteau, both17:00
tzumainnis there a spec for the nova + ironic work?17:00
priteauNot yet but I'll let you know when it's up for review.17:00
diurnalistI'm also interested in the Nova + Ironic spec17:01
diurnalistI was pretty confused by this note in the summit notes:17:01
diurnalist> #. Blazar creates an allocation in Ironic (not Placement) with the candidate node matching previously picked node and allocation UUID matching the reservation UUID.17:01
*** stephenfin has joined #openstack-meeting-alt17:01
diurnalistso I'm interested in hearing how Placement will fit in to the Nova + Ironic use-case.17:02
*** liuyulong has quit IRC17:03
diurnalistalso, this note: "To avoid partial allocations, Placement could introduce new API to consume the whole resource provider." - pretty sure this is what resource classes are used for, no? you have CUSTOM_BAREMETAL=1, which can only be 0 or 1, so the node is either 100% or 0% allocated. but I know we're well over time now...17:04
diurnalistperhaps something for the mailing list.17:04
priteaudiurnalist: I haven't looked in detail yet at the proposed standalone workflow. I don't really see how placement can fit with selecting nodes for advance reservations.17:04
priteauLet's end here for today. Thanks everyone for joining!17:05
priteau#endmeeting17:05
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"17:05
openstackMeeting ended Thu May  9 17:05:30 2019 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)17:05
openstackMinutes:        http://eavesdrop.openstack.org/meetings/blazar/2019/blazar.2019-05-09-16.01.html17:05
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/blazar/2019/blazar.2019-05-09-16.01.txt17:05
openstackLog:            http://eavesdrop.openstack.org/meetings/blazar/2019/blazar.2019-05-09-16.01.log.html17:05
priteauFeel free to join #openstack-blazar if you want to discuss further.17:05
*** macza has joined #openstack-meeting-alt17:12
*** jakecoll has quit IRC17:14
*** priteau has quit IRC17:16
*** bhavikdbavishi has quit IRC17:19
*** yamamoto has joined #openstack-meeting-alt17:22
*** diurnalist has quit IRC17:29
*** ijw has quit IRC17:37
*** hongbin has quit IRC17:54
*** ijw has joined #openstack-meeting-alt18:00
*** jtomasek has quit IRC18:07
*** yamamoto has quit IRC18:11
*** jtomasek has joined #openstack-meeting-alt18:13
*** jtomasek has quit IRC18:16
*** ttsiouts has quit IRC18:25
*** jamesmcarthur has joined #openstack-meeting-alt18:34
*** igordc has joined #openstack-meeting-alt18:35
*** jaypipes has quit IRC18:37
*** jamesmcarthur has quit IRC18:38
*** ttsiouts has joined #openstack-meeting-alt18:40
*** igordc has quit IRC18:44
*** ttsiouts has quit IRC18:45
*** jamesmcarthur has joined #openstack-meeting-alt18:47
*** igordc has joined #openstack-meeting-alt18:47
*** yamamoto has joined #openstack-meeting-alt18:51
*** iyamahat has joined #openstack-meeting-alt18:51
*** yamamoto has quit IRC19:02
*** ttsiouts has joined #openstack-meeting-alt19:11
*** carloss has left #openstack-meeting-alt19:16
*** jamesmcarthur has quit IRC19:17
*** jamesmcarthur has joined #openstack-meeting-alt19:31
*** jamesmcarthur has quit IRC19:39
*** hongbin has joined #openstack-meeting-alt19:44
*** jamesmcarthur has joined #openstack-meeting-alt19:50
*** vhari has quit IRC19:53
*** e0ne has joined #openstack-meeting-alt20:09
*** e0ne has quit IRC20:10
*** armstrong has joined #openstack-meeting-alt20:12
*** ttsiouts has quit IRC20:14
*** jamesmcarthur has quit IRC20:23
*** ttsiouts has joined #openstack-meeting-alt20:29
*** ttsiouts has quit IRC20:32
*** ttsiouts has joined #openstack-meeting-alt20:33
*** ttsiouts has quit IRC20:47
*** turnerg has quit IRC20:52
*** jamesmcarthur has joined #openstack-meeting-alt20:54
*** jamesmcarthur has quit IRC21:05
*** jamesmcarthur has joined #openstack-meeting-alt21:17
*** jamesmcarthur has quit IRC21:21
*** radeks_ has joined #openstack-meeting-alt21:28
*** ttsiouts has joined #openstack-meeting-alt21:29
*** radeks has quit IRC21:30
*** ttsiouts has quit IRC21:34
*** jgrosso has quit IRC21:40
*** lbragstad has quit IRC21:45
*** diablo_rojo has quit IRC21:54
*** ijw has quit IRC22:02
*** ijw has joined #openstack-meeting-alt22:03
*** hongbin has quit IRC22:04
*** hongbin has joined #openstack-meeting-alt22:05
*** hongbin has quit IRC22:05
*** hongbin has joined #openstack-meeting-alt22:06
*** hongbin has quit IRC22:06
*** hongbin has joined #openstack-meeting-alt22:06
*** ijw has quit IRC22:07
*** hongbin has quit IRC22:08
*** hongbin has joined #openstack-meeting-alt22:08
*** armstrong has quit IRC22:21
*** radeks_ has quit IRC22:25
*** markmcclain has quit IRC22:35
*** markmcclain has joined #openstack-meeting-alt22:37
*** raildo has quit IRC22:41
*** iyamahat has quit IRC22:42
*** ttsiouts has joined #openstack-meeting-alt22:46
*** ttsiouts has quit IRC22:51
*** slaweq has quit IRC22:51
*** munimeha1 has quit IRC22:52
*** macza has quit IRC22:57
*** slaweq has joined #openstack-meeting-alt22:58
*** slaweq has quit IRC23:03
*** slaweq has joined #openstack-meeting-alt23:11
*** slaweq has quit IRC23:16
*** ttsiouts has joined #openstack-meeting-alt23:17
*** ttsiouts has quit IRC23:21
*** whoami-rajat has quit IRC23:34
*** jamesmcarthur has joined #openstack-meeting-alt23:36
*** lseki has quit IRC23:40

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!