09:01:06 <dikonoor> #startmeeting powervm_driver_meeting
09:01:07 <openstack> Meeting started Wed Jun 19 09:01:06 2019 UTC and is due to finish in 60 minutes.  The chair is dikonoor. Information about MeetBot at http://wiki.debian.org/MeetBot.
09:01:08 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
09:01:10 <openstack> The meeting name has been set to 'powervm_driver_meeting'
09:04:24 <dikonoor> Waiting for members to join.
09:04:33 <MujahidAli> +1
09:05:06 <dikonoor> #topic PowerVM CI
09:05:29 <dikonoor> Actions from last week are these:
09:05:33 <dikonoor> * amansi26
09:05:33 <dikonoor> * amansi26 to open a devstack LP bug for the neutron l3 problem and
09:05:33 <dikonoor> propose changeset for that.
09:05:33 <dikonoor> * amansi26 and mujahidali continuing to debug the cause of tempest
09:05:33 <dikonoor> testcase failure during a manual CI run.
09:06:33 <dikonoor> amansi26: Thanks for joining. The tempest problem that was discussed about last week, I believe is resolved.
09:07:14 <dikonoor> On this devstack defect > https://review.opendev.org/#/c/664797/
09:07:18 <MujahidAli> dikonoor: we are still facing teardown and setupclass failure issues with tempest.
09:07:45 <dikonoor> MujahidAli : Are these due to the connection errors?
09:07:51 <dikonoor> LAst we checked these were.
09:08:11 <amansi26> There might be some connection errors
09:09:17 <dikonoor> We tried adding retry logic to pypowervm/adapter.py> request method but that hasn't helped. amansi26 reported around 52 failures with this fix applied locally.
09:09:47 <dikonoor> amansi26: Did you get a chance to look at the logs for these 52 failures to know what exactly is the cause of failure?
09:09:55 <MujahidAli> amansi26: applied the patch by dikonoor: but no luck
09:10:02 <dikonoor> Are each of these failures caused due to connectionerrors or something else?
09:10:33 <dikonoor> we have to check why this patch didn't work and try something else MujahidAli.
09:10:53 <dikonoor> #action Check why the retry patch failed.
09:12:09 <MujahidAli> amnsi26 collected the n-cpu logs and tried matching the timestamp, unfortunately there was no log at the timestamp of failure. failure happened at 35th second and logs were there only for 31st and 39th second.
09:13:49 <dikonoor> amansi26: we have to talk to chhagarw. She might have some insights
09:14:05 <dikonoor> But this is top priority on our list.
09:14:30 <dikonoor> Did you or MujahidAli get a chance to talk to someone on the devstack changeset?
09:14:52 <dikonoor> https://review.opendev.org/#/c/664797/
09:15:04 <chhagarw> joined
09:15:05 <amansi26> Yes. I checked for the cinder services on the instance they were down
09:15:34 <amansi26> I pinged the issue on the devstack channel(qa)but no reply
09:15:54 <dikonoor> chhagarw:Thanks for joining. Need some inputs from you on a connection error that the PowerVM CI has been running into for some time
09:16:54 <dikonoor> chhagarw: this is a ssp based environment and we run tempest test cases as part of the CI , that runs these testcases against the lastest instance of devstack with PowerVM drivers.
09:17:35 <dikonoor> Each run lasts for 4 hours and we see that we run into a connection error once or max twice during this run
09:18:27 <dikonoor> Note here that the the VM running CI has tempest /devstack /PowerVM driver and pypowervm and the calls to NovaLink happens remotely..meaning pypowervm and pvm components are on different systems.
09:18:34 <dikonoor> chhagarw: SSLError: SSLError for GET https://172.16.0.1:12443/rest/api/uom/SharedStoragePool/116d6f61-e012-39f5-bcb7-50551596f533?group=None: HTTPSConnectionPool(host='172.16.0.1', port=12443): Max retries exceeded with url: /rest/api/uom/SharedStoragePool/116d6f61-e012-39f5-bcb7-50551596f533?group=None (Caused by SSLError(SSLError('bad handshake: ZeroReturnError()',),))
09:18:38 <dikonoor> This is the exact error
09:23:33 <chhagarw> @dikonoor we need to investigate further on this it looks like some Connection Timeout for connection drop error.
09:28:21 <dikonoor> ok @chhagarw.
09:29:05 <dikonoor> amansi26: as discussed , on the devstack stacking error, we have to take devstack without PowerVM driver and see if stacking works there
09:29:31 <dikonoor> If stacking works there, then it's probably not a generic problem.. and could be specific to PowerVM driver.
09:29:37 <dikonoor> Pease check this and get back.
09:30:02 <dikonoor> #action amansi26 to install devstack latest without any PowerVM driver
09:32:18 <dikonoor> #topic Out of tree driver
09:32:49 <dikonoor> There are a bunch of changesets that have gone into nova, that need few changes in our OOT driver.
09:32:55 <dikonoor> I have added the list at https://etherpad.openstack.org/p/powervm-oot-todos
09:33:32 <dikonoor> arunman:ymadhavi:aarora06 need volunteers to prpose changesets
09:33:51 <dikonoor> Precisely these:
09:33:52 <dikonoor> 
09:33:52 <dikonoor> [dikonoor] Changes from following changesets in in-tree drivers is currently out of sync at OOT:
09:33:52 <dikonoor> https://github.com/openstack/nova/commit/aafd873e2176076609ff17d63137dd6f84ddf8d3#diff-98d1a06e0feb7e5ff43a21433afd9f34
09:33:52 <dikonoor> https://github.com/openstack/nova/commit/08f3ae960623c94bdd997cacb3e81f04b4bbba69#diff-46d49d172cae6b899c1be3628dede5d1
09:33:55 <dikonoor> https://github.com/openstack/nova/commit/8545ba2af7476e0884b5e7fb90965bef92d605bc#diff-5dc5ac3c2bf8010333f1752a1e70ce18
09:33:58 <dikonoor> https://github.com/openstack/nova/commit/4afe8ea5a19e236b485da3132d66064017a479ec#diff-c0abaa2f898cb7044ae6f1b8bd0340d9
09:34:01 <dikonoor> 
09:34:03 <dikonoor> [dikonoor] Changesets that need revival:
09:34:05 <dikonoor> https://review.opendev.org/#/c/627355/  <> https://review.opendev.org/#/c/617693/
09:34:07 <dikonoor> https://review.opendev.org/#/c/654979/
09:35:58 <dikonoor> Request all to please take a look.
09:36:07 <dikonoor> So actions we have are :
09:36:20 <dikonoor> #action amansi26 to install devstack without PowerVM driver
09:36:46 <dikonoor> #action chhagarw to take a closer look at the connection errors causing disruption in PowerVM CI
09:37:40 <dikonoor> #action vijayendra_IBM: prabhuraj: ymadahvi:arunman:aarora06 will take a look at the changesets where nova_powervm is out of sync with nova
09:37:55 <dikonoor> I am ending the meeting
09:38:03 <dikonoor> Thanks all for joining
09:38:04 <dikonoor> #endmeeting