Friday, 2017-02-17

*** erlon has quit IRC00:05
*** kaisers_ has joined #openstack-manila00:35
*** gouthamr has joined #openstack-manila00:39
*** kaisers_ has quit IRC00:40
*** catintheroof has quit IRC01:13
*** mtanino has quit IRC01:22
*** gcb has joined #openstack-manila01:35
*** kaisers_ has joined #openstack-manila01:36
*** gcb has quit IRC01:48
*** kaisers_ has quit IRC01:56
*** gcb has joined #openstack-manila02:00
*** nkrinner_afk has quit IRC02:08
*** tuanluong has joined #openstack-manila02:16
gouthamrbswartz: can these test patches go in as well: https://review.openstack.org/#/c/433770/ https://review.openstack.org/#/c/434289/02:17
bswartzgouthamr: I left those out intentionally02:22
bswartzdo you have a reason they need to be part of RC2?02:22
gouthamrbswartz: they don't *have* to be... but why not? might help if someone downstream is running these tests02:23
bswartznext week we can backport them02:23
bswartzI don't see any urgency to it02:23
bswartzone of those has a -1 on it btw02:23
gouthamrbswartz: sure.. the -1 has an explanation :)02:24
* bswartz is closer to solving the mystery to of the slow scenario tests02:24
gouthamr^ found an offending execution?02:25
bswartzI expect to be up late fighting with the gate and rechecking, rechecking, rechecking before all 7 of these patches land02:25
bswartzso in the mean time I'll be hacking on scenario tests02:25
bswartzgouthamr: as best I can tell, the problem is actually that NFS data traffic is getting dropped by shitty neutron networking02:26
bswartzas long as you perform small I/Os, everything is fine02:26
bswartzonce you ramp up to 1 megabyte files like ganso's test has, data packets start going missing02:26
bswartzand the retries don't happen as fast as one would like02:27
bswartzso tinkering with the NFS retry values has paid off bigtime02:27
gouthamroh..02:27
gouthamrdropped packets. twice in two days02:27
bswartzI don't have direct evidence of dropped packets, but the symptoms seem pretty clear02:27
bswartzand so far my cure is working rather well02:28
gouthamrnice.. i was hoping the cure was to start working on manila-test-image to replace the rather heavy VMs we spin up..02:29
bswartzgouthamr: I already do that02:29
gouthamrbswartz: oh!02:29
bswartzdoesn't solve NFS issues02:29
bswartzso I'm working on a full solution02:30
gouthamrbswartz: you're using the server image though?02:30
bswartzclient02:30
bswartzVM bootup is fast like you would expect02:30
bswartzbut I still face timeouts related to NFS slowness02:31
bswartzI thought originally it was SSH02:31
bswartzbut I can't reproduce any SSH issues while reproducing NFS issues is extremely easy02:31
gouthamryou're attempting to log this verbose-ly?02:32
*** ianychoi has quit IRC02:34
gouthamrah, timeo and retrans02:35
*** ianychoi has joined #openstack-manila02:38
*** kaisers_ has joined #openstack-manila02:39
*** kaisers_ has quit IRC02:44
*** nkrinner_afk has joined #openstack-manila03:12
bswartzgouthamr: I'm able to reproduce the issues on my dev system03:32
bswartzI'm playing around with small buffer sizes, soft mounts, udp transport and yes, timeout and retrans03:32
bswartzthe main point is that when packet loss happens we want the client to retry faster rather than waiting03:33
gouthamrbswartz: so do we have a combination that works?03:33
bswartzgouthamr: early tests look promising, but I'm also seeing problems with migrations themselves failing in the scenario tests03:38
bswartzdunno what that's about03:38
bswartzin any case, even if this works we won't be able to roll a proper fix into our gate until all the pieces fall into place03:38
gouthamrany errors in m-data?03:39
bswartzShareDataCopyFailed: Share Data copy failed: Failed to copy contents from instance c538c0e4-5b38-4b07-bdc9-b507934cd3d3 to instance 32598e7c-7bde-483b-b0e2-a2b27ffcdf8403:39
bswartzwtf does that mean?03:40
*** kaisers_ has joined #openstack-manila03:41
bswartzgouthamr: https://paste.fedoraproject.org/paste/LIEu3pAUsvR~JuhWgWNgYV5M1UNdIGYhyRLivL9gydE=03:41
gouthamrthat doesn't look right: /tmp/7c00a2bf-a498-47df-976d-1687e470e7bf/f1/03:43
bswartzwhy not?03:43
gouthamractually, i take that back.. i was wondering why we would mount inside /tmp03:44
bswartzit's a temporary mount03:44
gouthamrhttp://logs.openstack.org/82/435182/1/check/gate-manila-tempest-minimal-dsvm-lvm-ubuntu-xenial/44c70c6/logs/screen-m-dat.txt.gz#_2017-02-17_02_30_42_79203:44
bswartzmakes perfect sense to me03:44
gouthamrsudo didn't make us a sandwich.03:44
bswartzthe more interesting question is why root failed to get access to a local filesystem03:45
bswartzthat should be impossible03:45
bswartzdoes migration use NFS to mount one or both of the shares?03:45
* gouthamr thinks hard for a meme. finally finds: http://i2.kym-cdn.com/photos/images/original/000/913/758/a12.jpg03:45
bswartzthought you were looking for this: https://imgs.xkcd.com/comics/sandwich.png03:46
gouthamrbswartz: that one is where sudo works :P03:46
gouthamrbswartz: yes, no cross-protocol.. both nfs shares03:46
bswartzokay well probably root is getting squashed03:47
bswartz>_<03:47
bswartzthe only way root can get permission denied is if it's an NFS filesystem with squash root (or squash all) enabled03:48
gouthamrwe setup the exports in this case, don't we.. and iirc, we don't squash root..03:48
bswartzdo we take the default or do we explicitly set no_root_squash,no_all_squash?03:48
*** tommylikehu1 has joined #openstack-manila03:48
gouthamrsudo manila-rootwrap /etc/manila/rootwrap.conf exportfs -o rw,no_subtree_check,no_root_squash 50.50.50.50:/opt/stack/data/manila/mnt/share-abac5736-2776-4403-b48a-bd1ad82bea8d execute /usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:35503:49
gouthamrsample ^03:49
*** zhugaoxiao has quit IRC03:49
*** david-lyle has quit IRC03:49
*** zhugaoxiao has joined #openstack-manila03:50
*** david-lyle has joined #openstack-manila03:50
bswartzwell something is squashing root03:50
bswartzis that the source or destination export you're showing?03:51
*** deepbook5broo has joined #openstack-manila03:51
*** deepbook5broo has left #openstack-manila03:51
*** tommylikehu has quit IRC03:51
*** tommylikehu1 is now known as tommylikehu03:51
*** shausy has joined #openstack-manila03:52
gouthamrbswartz: good question.. it's just a random share from a recent log.. but all exports are setup the same in the driver..03:52
bswartzthat's true...03:52
gouthamrhttps://github.com/openstack/manila/blob/d856e8e841f7b494e64a4c8ba5d0b980a0ba082d/manila/share/drivers/helpers.py#L22103:53
gouthamrbswartz: "recheck" :P03:54
*** kaisers_ has quit IRC03:54
bswartzwhat? the job hasn't timed out yet03:54
gouthamri meant, retry the test...03:55
bswartzyou know that line you highlighted only executes for writable access rules03:55
gouthamrohh. you're waiting on https://review.openstack.org/#/c/435182/03:55
bswartzthe source of a migration should be read-only03:56
bswartzread-only but still with no_root_squash03:56
gouthamrare you on ocata/master03:56
bswartzI'm on HEAD from 2 days ago03:56
bswartzmaster/HEAD03:57
bswartzhaven't restacked since tuesday03:57
gouthamrcasting something to readonly will take it through the "recovery" path.. i.e, add_rules/remove_rules will be empty03:57
gouthamroh crap03:57
gouthamri missed the if statement right above the line03:57
bswartzyeah...03:57
bswartzseems like a bug03:57
bswartzganso: ^03:58
* bswartz presumes ganso is sleeping and will see this in scrollback tomorrow morning03:58
gouthamr+1 -> but how is this not a failure all the time?03:58
bswartzgouthamr: if clients don't create any root-owned directories before migrating then this bug won't get tickled03:58
gouthamri don't see anything different in the test that's creating these files/dirs.. i see some file perms set.. but no user changes..04:01
bswartzgouthamr: I'm using a different NFS client than the gate so it's possible the gate is doing something that results in non-root-owned directories04:02
gouthamr        ssh_client.exec_command("sudo mkdir -p /mnt/f1")04:02
gouthamr        ssh_client.exec_command("sudo mkdir -p /mnt/f2")04:02
gouthamr        ssh_client.exec_command("sudo mkdir -p /mnt/f3")04:02
gouthamr        ssh_client.exec_command("sudo mkdir -p /mnt/f4")04:02
gouthamr        ssh_client.exec_command("sudo mkdir -p /mnt/f1/ff1")04:02
bswartzthe gate invokes all those commands as "ubuntu"04:02
* gouthamr sigh ^ didn't mean to do that04:02
gouthamrah.. and the buildroot user is "root"04:02
bswartzmy tests simply run as root, because root is the only user in my image04:03
bswartznot sure if it makes a difference but in this case it seems like something suspicious04:03
gouthamrgood catch!04:03
bswartzit would be nice to get some debug logging to show the state of the filesystem from the client's and server's perspective right before the data copy04:04
gouthamryes, that'd help...04:05
gouthamrbswartz: not sure if ganso wanted this in rc2: https://review.openstack.org/#/c/434462/04:06
bswartzgouthamr: there's not tag or target04:18
bswartzI got by the "ocata-rc2-potential" tag in LP04:19
gouthamrbswartz: i guess our dummy driver uncovered that race condition.. okay with it being backported tomorrow04:19
gouthamrthat failure on the gate: http://logs.openstack.org/57/435057/2/check/gate-manila-tempest-minimal-dsvm-dummy-ubuntu-xenial/edc502a/console.html#_2017-02-16_21_52_55_31507004:20
bswartzit's not a critical bug04:20
gouthamr+104:20
bswartzand it missed the time cutoff04:20
bswartzso it can wait to next week04:20
gouthamryour wait is done04:22
gouthamr:)04:22
bswartzwait for it.....04:22
gouthamrrace condition ^04:22
* bswartz wonders where the gerritbot is04:23
gouthamroh, gerritbot doesn't report on stable branch changes04:23
gouthamrbswartz: maybe we can change that: https://github.com/openstack-infra/project-config/blob/master/gerritbot/channels.yaml#L112904:25
bswartzhttps://review.openstack.org/#/c/43523004:26
bswartzgouthamr: indeed04:26
bswartzI'm more interested in seeing stable branch activity reported than I am in seeing changes related to puppet04:27
bswartzmost other projects seem to be master-only for the bot too though04:27
gouthamryeah.. some do stable/juno. they definitely like to be informed :D04:29
gouthamrthat was quick. Release Ocata-RC2 for Manila -check.04:34
bswartzgouthamr: it merged04:34
bswartzRC2 is done04:34
gouthamrbswartz: day off tomorrow? :P04:34
bswartzgouthamr: yeah04:35
gouthamrbswartz: sokie... like tbarron says, you may take the weekend off too...04:35
bswartzheh04:36
*** ChanServ changes topic to "OpenStack Shared File Systems | Manila | Ocata RC2 is done, please test! | PTG next week Feb 20-24"04:37
*** gouthamr has quit IRC04:38
*** kaisers_ has joined #openstack-manila04:49
*** kaisers_ has quit IRC04:52
*** kaisers_ has joined #openstack-manila04:52
*** kaisers_ has quit IRC04:57
*** faizy has joined #openstack-manila05:32
*** kaisers_ has joined #openstack-manila05:52
*** kaisers_ has quit IRC05:57
*** lpetrut has joined #openstack-manila06:20
*** nkrinner_afk is now known as nkrinner06:31
openstackgerritTommyLike proposed openstack/manila master: Fix: [LVM] Can't extend share when snapshots exist  https://review.openstack.org/43131506:53
*** kaisers_ has joined #openstack-manila06:55
*** kaisers_ has quit IRC07:16
*** jprovazn has joined #openstack-manila07:17
*** lpetrut has quit IRC07:25
*** lpetrut has joined #openstack-manila07:25
*** dsariel has quit IRC07:26
*** lpetrut has quit IRC07:36
*** dsariel has joined #openstack-manila07:39
openstackgerritxurong00037997 proposed openstack/manila master: Remove unused pylintrc  https://review.openstack.org/43528907:54
*** kaisers_ has joined #openstack-manila08:12
*** pcaruana has joined #openstack-manila08:22
*** wiebalck has joined #openstack-manila08:25
*** a-pugachev has joined #openstack-manila08:57
*** gcb has quit IRC08:58
*** akapil has joined #openstack-manila09:05
*** akapil has quit IRC09:05
*** kaisers_ has quit IRC09:10
*** akapil has joined #openstack-manila09:12
*** gcb has joined #openstack-manila09:13
*** lpetrut has joined #openstack-manila09:43
*** ganso has joined #openstack-manila09:57
*** akapil_ has joined #openstack-manila09:59
*** akapil has quit IRC10:02
*** tuanluong has quit IRC10:03
*** faizy has quit IRC10:12
*** shausy has quit IRC10:13
*** faizy has joined #openstack-manila10:13
*** a-pugachev has quit IRC10:25
*** a-pugachev has joined #openstack-manila10:28
*** a-pugachev has quit IRC11:01
*** a-pugachev has joined #openstack-manila11:05
*** kaisers_ has joined #openstack-manila11:06
*** akapil_ has quit IRC11:10
*** rraja has joined #openstack-manila11:18
*** kaisers_ has quit IRC11:31
*** akapil has joined #openstack-manila11:37
*** tommylikehu_ has joined #openstack-manila11:53
openstackgerritMerged openstack/manila master: Fix migration_success before completing  https://review.openstack.org/43446211:56
*** tommylikehu_ has quit IRC11:57
*** gcb has quit IRC12:07
*** tommylikehu_ has joined #openstack-manila12:15
gansobswartz: ping12:18
*** catintheroof has joined #openstack-manila12:23
*** kaisers_ has joined #openstack-manila12:27
*** ociuhandu has joined #openstack-manila12:29
*** faizy has quit IRC12:45
vponomaryovganso: is https://review.openstack.org/#/c/434462/ related to http://logs.openstack.org/57/435057/2/check/gate-manila-tempest-minimal-dsvm-dummy-ubuntu-xenial/edc502a/logs/tempest_2/testr_results.html.gz ?12:45
gansovponomaryov: yes12:47
vponomaryovganso: so, issue is already fixed by this?12:48
gansovponomaryov: yes12:48
vponomaryovganso: thank you )12:48
gansovponomaryov: you're welcome12:48
*** erlon has joined #openstack-manila12:51
*** mtanino has joined #openstack-manila12:59
*** mtanino has quit IRC13:15
*** kaisers_ has quit IRC13:26
*** kaisers_ has joined #openstack-manila13:26
*** xyang1 has joined #openstack-manila13:33
*** faizy has joined #openstack-manila13:41
*** rraja has quit IRC13:48
*** dustins has joined #openstack-manila13:54
*** gouthamr has joined #openstack-manila14:01
*** rraja has joined #openstack-manila14:05
*** eharney has joined #openstack-manila14:06
*** chlong has joined #openstack-manila14:15
*** mtanino has joined #openstack-manila14:28
*** tommylik_ has joined #openstack-manila14:28
bswartzganso: late pong14:29
*** tommylikehu_ has quit IRC14:30
*** tommylik_ has quit IRC14:36
*** nkrinner is now known as nkrinner_afk14:40
gansobswartz: hey Ben14:44
gansobswartz: since you're an expert on LVM, could you please share your opinion on the latest comment in https://review.openstack.org/#/c/433489/ ?14:44
gansobswartz: I believe that, if the share is 10GB, and the user wants to shrink the share to 5GB, even though he may have only 1GB of used space, he may lose data due to fragmentation14:45
gansobswartz: I am not sure LVM is smart enough to get around that14:46
gansobswartz: and btw, I could not reproduce the problem of not having no_root_squash with RO access rules14:47
gansobswartz: I created root owned folders, and using the generic driver (which uses the NFS helper), and the sudo commands could access these folders and files within those folders without any problem14:48
*** porrua has joined #openstack-manila15:00
bswartzganso: hi15:07
bswartzganso: that's not an LVM issue15:08
bswartzLVM will just chop off the blocks from the end when you shrink an LV15:08
gansobswartz: exactly15:08
bswartzand the driver doesn't support shrinking for that reason15:08
bswartznow ext4 can shrink a filesystem to consume less blocks15:09
gansobswartz: but LVM does not care about filesystems15:09
bswartzthe driver does care about filesystems though15:09
gansobswartz: hmmmm15:09
bswartzif we wanted to implement shrink in LVM we could umount, resize the filesystem, then resize the LV, then remount15:10
bswartzthe filesystem resizing could cause data copying, and thus could be slow15:10
bswartzall of our challenges in the LVM driver come from the fact that we have a mounted filesystem on top of the LV, and we have to manage that filesystem15:11
bswartzmany operations seem to require unexporting and/or umounting15:11
bswartzanyways the revert to snapshot flavor of resize is different too15:12
bswartzI'm not even sure LVM does the right thing when you revert to a different size snapshot -- we could have bugs there15:13
bswartzbut assuming it can handle that case, the filesystem wouldn't be a problem because the snapshotted filesystem would be an appropriate size15:13
gansobswartz: hummm cool, we should validate if ext4 is used then15:16
gansobswartz: that's a good -1 for that patch15:16
bswartzganso: man resize2fs15:16
jprovaznbswartz, hi15:21
bswartzjprovazn: sorry I didn't get back to you yesterday15:21
jprovaznbswartz, np15:21
bswartz(01:08:51 PM) jprovazn: bswartz, hi, re: user messages session at PTG - any chance it could fit to any of these time slots? Wed - 9:30-11am or 2:30-4:30pm; Thu - 9:00-11am or 2:30-4:30pm (EST TZ)15:21
jprovaznbswartz, yes - this is still valid preferred time for me15:22
bswartzjprovazn: which topic was it again?15:22
jprovaznbswartz, user messages15:22
*** faizy has quit IRC15:22
bswartzwednesday's agenda is looking pretty full15:23
jprovaznbswartz, IMO it has lower priority in compare to some others, but if there is general interest in discussing this, I'll be more than happy to connect15:23
bswartzThursday morning is probably the better time, to make sure we get to it15:23
jprovaznbswartz, Thu morning works for me15:23
bswartzokay I'll plan on starting that around 10AM15:23
bswartzand I'll put your name on the etherpad15:23
jprovaznbswartz, cool, thanks15:24
bswartzjprovazn:  scheduled15:24
*** faizy has joined #openstack-manila15:32
openstackgerritGoutham Pacha Ravi proposed openstack/manila master: Turkey Turkey Turkey  https://review.openstack.org/43549515:36
gansogouthamr: yummy ^15:38
*** databus23_ has joined #openstack-manila15:39
*** nkrinner_afk has quit IRC15:39
*** nkrinner_afk has joined #openstack-manila15:40
gouthamrganso: lol15:40
*** crushil has joined #openstack-manila15:46
* vponomaryov is thinking "and those people use 'waste of CI resources' as argument in review battles?"15:47
*** rraja has quit IRC15:53
bswartzvponomaryov: do you know if there's a way to tag a patchset so 3rd party CIs will ignore it?15:56
bswartzif not, perhaps there should be15:56
bswartzI bet a solid 10-20% of our patchsets should never ever have 3rd party CI run on them because they're aimed at debugging upstream issues15:57
vponomaryovbswartz: not aware about such, just know that most of 3rd-party CIs start running after +1 from jenkins15:57
vponomaryovbswartz: like NetApp CI does15:58
vponomaryovbswartz: 3rd-party CI maintainers are responsible for it15:58
vponomaryovbswartz: so, it is their interest to have "smart" CI15:58
openstackgerritGoutham Pacha Ravi proposed openstack/manila master: Turkey Turkey Turkey  https://review.openstack.org/43549516:00
gouthamrvponomaryov: i hear you..16:01
vponomaryovgouthamr: so do I ))16:02
gouthamrvponomaryov: however, desperate times need desperate patchsets.16:02
*** Yogi1 has joined #openstack-manila16:03
vponomaryovbswartz: for example, look how many redundant CI runs here: https://review.openstack.org/#/c/435289/16:05
*** lpetrut has quit IRC16:07
openstackgerritRodrigo Barbieri proposed openstack/manila master: Remove revert-to-snapshot unused test option  https://review.openstack.org/42766316:17
*** pcaruana has quit IRC16:21
*** cknight has joined #openstack-manila16:26
*** vponomaryov has left #openstack-manila16:39
*** faizy has quit IRC16:54
*** akapil has quit IRC17:00
*** akapil has joined #openstack-manila17:00
*** dustins has quit IRC17:00
*** akapil has quit IRC17:01
*** wiebalck has quit IRC17:05
*** wiebalck has joined #openstack-manila17:08
*** akapil has joined #openstack-manila17:10
*** crushil has quit IRC17:17
*** akapil_ has joined #openstack-manila17:18
*** akapil_ has quit IRC17:18
*** akapil_ has joined #openstack-manila17:21
*** akapil has quit IRC17:21
*** akapil_ has quit IRC17:25
*** a-pugachev has quit IRC18:03
*** ociuhandu has quit IRC18:20
*** jprovazn has quit IRC18:21
*** dustins has joined #openstack-manila18:24
*** crushil has joined #openstack-manila18:30
-openstackstatus- NOTICE: Restarting gerrit due to performance problems19:03
*** Yogi1 has quit IRC19:27
*** ociuhandu has joined #openstack-manila19:37
*** lpetrut has joined #openstack-manila19:48
*** akapil has joined #openstack-manila20:00
*** akapil has quit IRC20:04
*** crushil has quit IRC20:07
*** crushil has joined #openstack-manila20:38
*** dustins has quit IRC20:45
*** dustins_ has joined #openstack-manila20:45
*** mtanino has quit IRC20:51
openstackgerritTom Barron proposed openstack/manila master: container driver: log network id as network id  https://review.openstack.org/43559721:13
*** crushil has quit IRC21:18
*** crushil has joined #openstack-manila21:21
*** ganso has quit IRC21:25
*** cknight has quit IRC21:53
*** gouthamr has quit IRC22:11
*** crushil has quit IRC22:19
*** dustins_ has quit IRC22:33
*** wiebalck has quit IRC22:34
*** gouthamr has joined #openstack-manila22:34
*** xyang1 has quit IRC23:02
openstackgerritMerged openstack/manila-specs master: Update Share Migration Ocata Improvements Spec  https://review.openstack.org/40714523:08
*** catintheroof has quit IRC23:21
*** catintheroof has joined #openstack-manila23:22
*** lpetrut has quit IRC23:23
*** catintheroof has quit IRC23:26
*** porrua has quit IRC23:41
*** chlong has quit IRC23:46

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!