Saturday, 2016-03-19

*** cknight has quit IRC00:03
*** yangyapeng has quit IRC00:11
openstackgerritMerged openstack/manila: Fix for incorrect LVMMixin exception message  https://review.openstack.org/29448100:12
*** cknight has joined #openstack-manila00:14
*** ChanServ changes topic to "4 bugs left until RC1"00:16
bswartzameade gouthamr: ganso says he found a failure in https://review.openstack.org/#/c/292659/1000:16
*** tpsilva has quit IRC00:17
bswartzIndeed there were several failures in the LVM job but I didn't see the error he referred to00:17
bswartzganso: if you're still around can you point to where you saw that message?00:17
ameadeits in m-shr logs, looks unrelated to me. Do is this a flakey test?00:18
bswartzameade: LVM has a pretty solid record00:19
bswartzI haven't seen it fail in a long time00:19
bswartzis there a backtrace that points to where it happened? can we rule out the snapshot instances patch?00:19
bswartzameade: every patch that's a child of snapshot instances fails LVM00:22
ameadelookin00:22
ameadeyeah same error it seems00:23
bswartzif we merge now we'll have to fix that in RC200:23
bswartzand we'd rather avoid RC200:23
bswartzso let's spend a least a little bit hunting down the root cause00:24
openstackgerritGoutham Pacha Ravi proposed openstack/manila: Add tests to ensure snapshots across replicas  https://review.openstack.org/29311300:24
*** cknight has quit IRC00:25
*** gouthamr_ is now known as gouthamr00:25
bswartzgouthamr_ gouthamr: ^00:26
gouthamrbswartz: yep looking, nick issues with my ill mannered drop offs00:26
ameadebswartz: figured it out00:27
bswartzI wondered how people get names like nick_ nick__ and nick___00:27
ameadeit's def due to instances00:27
bswartzeasy to fix?00:27
ameadeyeah i think so00:27
* bswartz hopes....00:27
ameadeincoming00:27
gouthamrameade: the lvm driver was expecting something from the snapshot model?00:28
gouthamrameade: ugh00:28
ameadewe added the share instance details ot the snapshot00:28
gouthamrshare size00:28
ameadebut didnt says with_share_data00:28
gouthamri didn't refactor the create shapshot method,00:29
bswartzcould other drivers be affected similarly? we've seen some CI failures but ignored most of them due to normal CI instability00:29
gouthamrsnapshot_instance = self.db.share_snapshot_instance_get(00:29
gouthamr            context, snapshot_ref.instance['id'], with_share_data=True00:29
gouthamr        )00:29
gouthamromg00:30
gouthamr:P00:30
gouthamri know what happened there00:30
ameadeyeh00:31
bswartzwhat happened?00:31
gouthamri wonder how that test ever passed00:31
gouthamr(5th stage of denial)00:32
*** abhi has quit IRC00:36
gouthamrbswartz: other drivers seem to use snapshot['size'], the lvm driver is using snapshot['share']['size'] <-- where share's the share instance object and  we weren't setting the share model data in it.00:37
openstackgerritAlex Meade proposed openstack/manila: NetApp cDOT: Handle replicated snapshots  https://review.openstack.org/29324300:37
openstackgerritAlex Meade proposed openstack/manila: Data Replication: Ensure Snapshots across replicas  https://review.openstack.org/29265900:37
gouthamrthanks ameade00:37
ameadetrying to run unittests atm to make sure thats good00:38
ameadelooks like our ci 7mode fell over00:38
ameadelol00:38
ameadefixing that too00:38
gouthamryah that's how we roll on a friday night 8)00:38
*** gouthamr is now known as gouthamr_away_fo00:44
openstackgerritGoutham Pacha Ravi proposed openstack/manila: Add tests to ensure snapshots across replicas  https://review.openstack.org/29311300:45
*** eharney has quit IRC00:48
openstackgerritMerged openstack/python-manilaclient: Revert fix access rules functional tests  https://review.openstack.org/29339201:03
*** esker has quit IRC01:08
*** gouthamr_away_fo is now known as gouthamr01:15
gouthamrbswartz: fyi, ZFS tests are passing on the latest replicated snapshot tests -> https://jenkins01.openstack.org/job/gate-manila-tempest-dsvm-postgres-zfsonlinux/25/console01:24
*** tpsilva has joined #openstack-manila01:26
*** ganso has quit IRC01:29
*** yangyapeng has joined #openstack-manila01:46
*** yangyapeng has quit IRC01:51
openstackgerritGoutham Pacha Ravi proposed openstack/manila: Add tests to ensure snapshots across replicas  https://review.openstack.org/29311301:53
gouthamrbswartz: ping01:54
ameadewhoa didnt realize how late it was01:54
gouthamryeah its pretty late01:55
ameadei had fallen asleep on the couch earlier01:55
gouthamri'm still hopeful of making it to RC1 though :P01:55
*** akshai has joined #openstack-manila02:00
bswartzgouthamr: pong02:18
gouthamrbswartz: hi! we're all good on all three patches02:18
gouthamr4*02:18
bswartzawesome02:19
bswartzwho's awake to do the merge?02:19
bswartzxyang1? markstur?02:19
bswartzdoes zfs require a rebase?02:21
bswartzgouthamr: https://review.openstack.org/#/c/292659 failed LVM -- seemingly due to timeout02:22
gouthamrbswartz: i just posted the log on the latest recheck02:23
gouthamrhttps://jenkins01.openstack.org/job/gate-manila-tempest-minimal-dsvm-lvm/30/console02:23
gouthamr:)02:23
bswartzoh you already rechecked02:23
bswartzI didn't see that02:23
bswartzyes I see it02:23
bswartzgood02:23
bswartzgouthamr: where is the change that shortens the ZFS update interval?02:26
gouthamrhttps://review.openstack.org/#/c/29311302:26
gouthamrbswartz ^02:26
bswartzgouthamr: vponoparyov is going to -1 that when he sees it, lol02:27
bswartzI know he won't like it02:27
gouthamrbswartz: why?02:27
bswartzbut I can't think of a better way02:27
gouthamrbswartz: 60 seconds is perfect for tempest tests that create 1 gig shares :)02:28
bswartzyeah -- but this change will do it for everyone who installs devstack02:28
bswartzwhy did you use DEFAULT and not $BE?02:28
gouthamrbswartz: could it be done in the backend section?02:29
bswartzI don't know02:29
bswartzbut if it should be in DEFAULT then it should be outside the loop02:29
bswartzthe loop iterates once per backend02:29
gouthamri am expecting the option in the default group ..02:29
gouthamrohh02:29
gouthamrsure. will fix that02:29
bswartzand can we wrap it with a "is tempest enabled" or something02:30
bswartzor wait a minute....02:30
bswartzcan we set it to $MANILA_ZFS_REPLICA_STATE_UPDATE_INTERVAL and then set that value in the pre test hook?02:30
bswartzthat would ensure that it's only set to 60 when run inside of devstack gate02:31
bswartzordinary devstack users would get the ordinary value02:31
gouthamrbswartz: yes that seems better than overriding it for unsuspecting users that use devstack/zfs02:32
gouthamrwill do02:32
*** baojg has joined #openstack-manila02:42
openstackgerritGoutham Pacha Ravi proposed openstack/manila: Add tests to ensure snapshots across replicas  https://review.openstack.org/29311302:59
bswartzgouthamr: typo on that patch02:59
bswartzmissed a $ in plugin.sh03:00
openstackgerritGoutham Pacha Ravi proposed openstack/manila: Add tests to ensure snapshots across replicas  https://review.openstack.org/29311303:00
gouthamr:)03:00
gouthamrthanks fixed!03:00
bswartzalso03:01
bswartzmisspelled the option name in settings03:01
bswartzyou must be tired man03:01
bswartzworking so hard03:01
gouthamrsettings as well?03:02
gouthamrargh :P03:02
bswartzin pre_test_hook.sh you might want to include a comment explaining why 6003:03
bswartzthat's all I can see03:04
openstackgerritGoutham Pacha Ravi proposed openstack/manila: Add tests to ensure snapshots across replicas  https://review.openstack.org/29311303:04
gouthamrbswartz: it was a change of mind about the name03:04
bswartzokay this looks good03:04
bswartzwell you're right we could use a non-ZFS specific option and set it outside the ZFS-specific code in plugin.sh03:05
gouthamrbswartz: =) wanted to see if i can make it generic if our CI wants to reuse MANILA_REPLICA_STATE_UPDATE_INTERVAL in their hooks..03:05
bswartzyes we could do that03:05
bswartzwhy did you decide against that?03:05
bswartzit seems better03:05
gouthamrokay, will revert to that then.03:06
gouthamrakerr may thank me03:06
bswartzwhen you do it include a comment where you set it to 60 explaining why03:06
gouthamrsure will do03:07
*** cknight has joined #openstack-manila03:08
openstackgerritGoutham Pacha Ravi proposed openstack/manila: Add tests to ensure snapshots across replicas  https://review.openstack.org/29311303:13
bswartzdoh!03:14
bswartzforgot $ again03:14
gouthamro.O03:15
bswartzdevstack/plugin.sh03:15
bswartzI dont' see any other issues03:15
openstackgerritGoutham Pacha Ravi proposed openstack/manila: Add tests to ensure snapshots across replicas  https://review.openstack.org/29311303:15
gouthamrsorry about that03:15
gouthamr:) fixed03:16
bswartzokay03:16
bswartzfinally this looks good03:16
bswartzthanks for staying up03:16
gouthamrnp! :)03:16
bswartzhopefully vponomaryov sees this tomorrow morning and workflows03:16
gouthamrare these the final patches?03:17
bswartzas far as I'm concerned03:18
bswartzbut we have 2 cores to merge stuff for a reason03:18
bswartzganso caught that LVM issue which was lucky03:18
bswartzvponomaryov might catch something else03:18
gouthamryes..03:18
bswartznothing to do now but sleep....03:19
bswartzget some rest man03:19
gouthamri was so sure i didn't change the existing snapshot behavior.03:19
bswartzthanks for all your hard work and I hope you enjoy your weekend03:19
gouthamrand there had to be one driver that was doing snapshot['share']['size'] :D03:19
gouthamrhey np!03:20
gouthamri'll drop off now. thanks for helping me fix the test patch. I learnt the hook stuff new today!03:20
*** gouthamr has quit IRC03:30
*** tpsilva has quit IRC03:37
*** cknight1 has joined #openstack-manila04:03
*** cknight has quit IRC04:05
*** baojg has quit IRC04:39
*** yangyapeng has joined #openstack-manila04:50
*** yangyapeng has quit IRC04:55
*** baojg has joined #openstack-manila04:59
*** lpetrut has joined #openstack-manila06:18
*** david-lyle has quit IRC06:22
*** david-lyle has joined #openstack-manila06:23
*** ociuhandu has quit IRC06:50
*** ociuhandu has joined #openstack-manila07:13
*** lpetrut has quit IRC07:18
*** lpetrut has joined #openstack-manila07:27
*** lpetrut has quit IRC07:31
*** yangyapeng has joined #openstack-manila08:27
*** baojg has quit IRC08:27
*** baojg has joined #openstack-manila08:29
*** baojg has quit IRC08:31
*** yangyapeng has quit IRC08:31
*** baojg has joined #openstack-manila08:42
openstackgerritMerged openstack/manila: Data Replication: Ensure Snapshots across replicas  https://review.openstack.org/29265909:15
*** cfey has quit IRC10:58
*** cfey has joined #openstack-manila11:01
*** cfey has quit IRC11:10
*** cfey has joined #openstack-manila11:15
*** cfey has quit IRC11:20
*** cfey has joined #openstack-manila11:23
*** cfey has quit IRC11:25
*** cfey has joined #openstack-manila11:28
*** cfey has quit IRC11:31
*** cfey has joined #openstack-manila11:32
*** ociuhandu has quit IRC11:34
*** cfey has quit IRC11:34
openstackgerritMerged openstack/manila: NetApp cDOT: Handle replicated snapshots  https://review.openstack.org/29324311:35
*** cfey has joined #openstack-manila11:37
*** cfey has quit IRC11:50
openstackgerritGoutham Pacha Ravi proposed openstack/manila: Add tests to ensure snapshots across replicas  https://review.openstack.org/29311311:53
*** cfey has joined #openstack-manila11:53
*** lpetrut has joined #openstack-manila11:54
*** cfey has quit IRC11:59
*** cfey has joined #openstack-manila12:02
*** ganso has joined #openstack-manila12:11
gansobswartz, vponomaryov: ping12:12
*** lpetrut has quit IRC12:14
bswartzganso: thank goodness you're here12:27
gansobswartz: good morning12:27
gansobswartz: what's up?12:28
bswartzhttps://review.openstack.org/#/c/293377/ needs workflow12:28
gansobswartz: I suspect RC1 has not been cut, there are 2 changes still to merge12:28
bswartzvponomyov can't do that one12:28
gansobswartz: aren't you waiting on https://review.openstack.org/#/c/293113/ ?12:28
bswartzeverything was passing last night12:29
bswartznot sure what gouthamr is up to this morning12:29
bswartzlooks like simple rebase12:30
gansobswartz: no change is expected on 377?12:30
bswartznope12:30
gansobswartz: ok, will take a last look at it12:30
bswartzafter these 2 merge I'll cut the RC and then we can start looking for quality issues worthy to an RC212:31
gansobswartz: feel free to workflow it12:48
gansobswartz: btw, generic-no-share-servers is failing consistently after something from yesterday has merged12:50
gansobswartz: my patch was the last one that passed12:53
bswartzyou think it's related?12:59
bswartzI noticed it but did not investigate13:00
gansobswartz: https://review.openstack.org/#/c/290511/13:01
gansobswartz: that patch broke generic-no-share-servers CI13:01
bswartzah13:01
bswartzany idea why?13:01
gansobswartz: either devstack script or changes in nova client13:02
bswartzwell it would be nice to fix that before we cut the RC13:03
bswartzif we can get to the bottom of it13:03
gansobswartz: I am looking13:03
bswartzI won't have much time to spend on it today13:03
bswartzmaybe tonight I can13:03
*** yangyapeng has joined #openstack-manila13:03
bswartzif you see what hte problem is go ahead and file the bug in LP so we can target it13:04
*** yangyapeng has quit IRC13:08
gansobswartz: I found the problem, I don't know what is causing it13:12
gansobswartz: instances are taking very long time to boot13:12
gansobswartz: tempest is starting tests before servers are up13:12
gansobswartz: this causes most of the tests to fail13:13
gansobswartz: but then later everything runs fine13:13
bswartzis there a mechanism to make tempest wait for the VM to boot?13:13
bswartzI would have assumed a wait loop already existed13:13
gansobswartz: I assumed as well13:14
gansoI don't understand why that patch broke it13:14
gansobswartz: https://bugs.launchpad.net/manila/+bug/155945613:18
openstackLaunchpad bug 1559456 in Manila "tests run before instance is ready in generic-no-share-servers CI " [Undecided,New]13:18
bswartzI don't want to target it unless we know what exactly needs to be fixed13:18
gansobswartz: I am not familiar with how it worked before, if there was a wait loop13:19
bswartzlet me take a quick look13:19
gansobswartz: vponomaryov probably knows this13:19
bswartzbut then I have to run13:19
bswartzhttps://github.com/openstack/manila/blob/master/contrib/ci/common.sh#L3513:21
bswartzthis?13:21
gansobswartz: nice13:22
bswartzlooks like it waits for "ping" to succeed13:23
bswartzmaybe it needs to wait for SSH to also be up13:23
gansobswartz: yes13:23
bswartzor for some additional services13:23
gansobswartz: on a succeeded run, it took 2 minutes to be SSH'able13:23
gansobswartz: on the failed runs, it took 5 minutes13:23
bswartzuntil socat /dev/null TCP:$HOST:22,connect-timeout=1 2> /dev/null ; do sleep 1 ; done13:24
bswartzthat's how my scripts wait for SSH to be up13:24
*** yangyapeng has joined #openstack-manila13:24
bswartzlet me throw up a fix for this13:25
gansobswartz: replace a ping by that13:25
bswartzk 1 sec13:25
gansobswartz: that could work13:25
*** yangyapeng has quit IRC13:26
openstackgerritBen Swartzlander proposed openstack/manila: In CI, wait for SSH instead of ping  https://review.openstack.org/29491013:29
*** ChanServ changes topic to "2 bugs left until RC1"13:30
bswartzOkay I'll bbl13:30
gansobswartz:13:30
gansobswartz: ok13:30
*** lpetrut has joined #openstack-manila13:38
openstackgerritMerged openstack/manila: Make ZFSonLinux driver handle snapshots of replicated shares properly  https://review.openstack.org/29337713:54
*** ChanServ changes topic to "1 bug left until RC1"13:58
*** lpetrut has quit IRC14:00
openstackgerritMerged openstack/manila: Add tests to ensure snapshots across replicas  https://review.openstack.org/29311314:01
*** lpetrut has joined #openstack-manila14:04
bswartzganso: I see some fails early on https://jenkins01.openstack.org/job/gate-manila-tempest-dsvm-generic-no-share-servers/33/console14:08
gansobswartz: I see them too14:08
gansobswartz: I cannot see anything other than console... waiting for test to finish14:08
bswartzis this how the other failure looked?14:08
bswartzor is this better14:08
gansobswartz: looks exactly the same14:08
bswartzcrud14:08
bswartzso maybe this sleep isn't big enough? https://github.com/openstack/manila/blob/master/contrib/ci/common.sh#L7514:09
gansobswartz: I don't know, if in m-shr I can see that it is still trying to SSH and instance is not ready, then your fix probably did not work14:09
bswartzok so we have to wait for log upload :-/14:10
gansobswartz: yup14:10
*** lpetrut has quit IRC14:11
gansobswartz: your fix probably did not work, I will test on my lab14:17
bswartzwell it's what I use in my testing to wait for servers to be up and it's rock solid14:19
bswartzso either these VMs are missing the socat binary, or it's not SSH that's the problem14:20
*** vponomaryov1 has joined #openstack-manila14:20
vponomaryov1bswartz, ganso: any logs of finished job is available right away14:21
vponomaryov1http://logs.openstack.org/10/294910/1/check/gate-manila-tempest-dsvm-generic-no-share-servers/fc87b69/14:21
gansovponomaryov1: yea, I have look over them14:21
bswartzhttp://paste.openstack.org/show/491183/14:21
bswartz^ This looks suspicious14:21
bswartzIt appears my fix never even got invoked because something went wrong beforehand14:22
vponomaryov1oh, problem is simple ))14:23
vponomaryov1problems appeared after switch to v314:23
vponomaryov1and v2 endpoint not installed14:23
vponomaryov1so, all client calls in post_test_hook fails now14:24
vponomaryov1we need just to impropert env vars properly now in post_test_hook14:24
vponomaryov1s/impropert/import/14:24
vponomaryov1import + proper = impropert ))14:24
gansovponomaryov1: are you able to upload the necessary fix?14:25
vponomaryov1need to find that "impropert" ))14:26
gansovponomaryov1: lol14:26
bswartzvponomaryov1: do you want to push a patch?14:28
bswartzthen ganso and I can merge it and we can be done with RC114:28
bswartzindeed ganso noted that it started failing after keystonev3 merged14:29
vponomaryov1first guess, admin user stopped being related to project we use14:30
vponomaryov1or domain14:30
bswartzvponomaryov1: if you have a fix, use this bug: https://bugs.launchpad.net/manila/+bug/155945614:31
openstackLaunchpad bug 1559456 in Manila "tests run before instance is ready in generic-no-share-servers CI " [High,In progress] - Assigned to Ben Swartzlander (bswartz)14:31
vponomaryov1I saw14:31
bswartzI'm leaving here in a moment but I'll bring a laptop so I can check in after lunch14:31
bswartzcknight1 maybe be around this morning too to help merge something14:33
cknight1bswartz: I'm here for a bit14:34
openstackgerritValeriy Ponomaryov proposed openstack/manila: Fix call of clients in post_test_hook.sh  https://review.openstack.org/29491814:46
gansovponomaryov1: are you saying we need a fix for manilaclient as well?14:50
bswartzvponomaryov1: you sure those exports won't overwrite anything important?14:50
bswartzI might have used a subshell for the wait for drivers init14:50
bswartzso the env of the devstack gate shell doesn't get altered14:51
vponomaryov1ganso: vice versa14:51
vponomaryov1ganso: manilaclient smarter than novaclient14:51
vponomaryov1ganso: ))14:52
gansovponomaryov1: great :D14:52
bswartzwell I'm going to optimistically +2 this before I leave14:52
gansoI am watching the job14:53
vponomaryov1bswartz: debug print of opts rules14:53
gansobswartz: I can workflow it later if it passes14:53
bswartzif anyone sees an EMC person, please let them know the ISILON CI is hosed badly14:53
vponomaryov1PS2 will be uploaded, mistake sneaked in14:56
openstackgerritValeriy Ponomaryov proposed openstack/manila: Fix call of clients in post_test_hook.sh  https://review.openstack.org/29491814:58
bswartzOkay I'll be AFK for a few hours15:01
bswartzI'll try to checkin from laptop after lunch15:02
vponomaryov1fix works - https://jenkins04.openstack.org/job/gate-manila-tempest-dsvm-generic-no-share-servers/27/console15:33
gansovponomaryov1: yup! :D15:42
gansovponomaryov1: waiting for job to be finished15:42
vponomaryov1ganso: fix is proven, no need to wait15:45
gansovponomaryov1: yes, but job must finish before going to the gate15:46
vponomaryov1and? ))15:46
gansovponomaryov1: I really don't like workflowing before jenkins15:47
vponomaryov1ganso: if you workflow it now, you can driver beer already, if not you should keep an eye on job and be sober15:48
vponomaryov1s/driver/drink/15:48
gansovponomaryov1: I got a flu... I can't drink :(15:48
gansovponomaryov1: not today...15:49
vponomaryov1ganso: ok, then you can get some rest earlier ))15:49
vponomaryov1ganso: and I could recheck myself15:49
vponomaryov1ganso: ))15:49
gansovponomaryov1: ok ok you win :P15:50
gansovponomaryov1: I am already "resting". Can't really enjoy my saturday :(15:51
vponomaryov1ganso: but you have good reason to be lazy whole day!15:52
vponomaryov1ganso: to watch some cool TV series15:52
vponomaryov1ganso: for which you haven't time for months15:54
gansovponomaryov1: not as fun as drinking though :\15:54
gansovponomaryov1: going to need a recheck16:04
*** pcaruana has quit IRC16:15
*** pcaruana has joined #openstack-manila16:28
*** ociuhandu has joined #openstack-manila17:07
*** gouthamr has joined #openstack-manila17:35
*** gouthamr has quit IRC17:38
openstackgerritMerged openstack/manila: Fix call of clients in post_test_hook.sh  https://review.openstack.org/29491818:14
*** pcaruana has quit IRC18:18
gansobswartz: ^18:23
*** pcaruana has joined #openstack-manila18:32
*** amit213 has quit IRC19:16
*** amit213 has joined #openstack-manila19:17
*** gouthamr has joined #openstack-manila19:33
*** gouthamr has quit IRC19:39
*** gouthamr has joined #openstack-manila19:39
*** esker has joined #openstack-manila19:59
*** jasonsb has quit IRC20:07
*** ganso has quit IRC20:29
*** vponomaryov1 has left #openstack-manila20:50
*** jasonsb has joined #openstack-manila21:13
*** dalgaaf has joined #openstack-manila21:43
*** gouthamr has quit IRC21:59
-openstackstatus- NOTICE: Gerrit is restarting to increase performance issues22:24
*** cfey has quit IRC22:27
*** cfey has joined #openstack-manila22:30
*** cfey has quit IRC23:21
*** cfey has joined #openstack-manila23:23
*** ChanServ changes topic to "RC1 complete!"23:53

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!