Tuesday, 2019-05-28

*** macza has quit IRC00:03
*** Liang__ has joined #openstack-meeting00:14
*** rcernin has quit IRC00:17
*** Liang__ has quit IRC00:17
*** rcernin has joined #openstack-meeting00:18
*** Liang__ has joined #openstack-meeting00:30
*** Liang__ is now known as LiangFangh00:30
*** LiangFangh is now known as LiangFang00:31
*** plutoinmii has joined #openstack-meeting00:39
*** plutoinmii has joined #openstack-meeting00:41
*** rcernin has quit IRC00:49
*** rcernin has joined #openstack-meeting00:50
*** markvoelker has quit IRC01:05
*** rfolco has quit IRC01:06
*** whoami-rajat has joined #openstack-meeting01:06
*** hongbin has joined #openstack-meeting01:10
*** lseki has quit IRC01:11
*** bobh has joined #openstack-meeting01:12
*** ricolin has joined #openstack-meeting01:13
*** bobh has quit IRC01:23
*** plutoinmii has quit IRC01:33
*** baojg has joined #openstack-meeting02:31
*** _alastor_ has joined #openstack-meeting02:43
*** _alastor_ has quit IRC02:47
*** ktsuyuzaki is now known as kota_03:01
*** ykatabam has quit IRC03:05
*** markvoelker has joined #openstack-meeting03:06
*** psachin has joined #openstack-meeting03:21
*** ykatabam has joined #openstack-meeting03:32
*** markvoelker has quit IRC03:40
*** tpatil has joined #openstack-meeting03:45
*** imsurit_ofc has joined #openstack-meeting03:55
samP#startmeeting masakari04:00
openstackMeeting started Tue May 28 04:00:22 2019 UTC and is due to finish in 60 minutes.  The chair is samP. Information about MeetBot at http://wiki.debian.org/MeetBot.04:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.04:00
*** openstack changes topic to " (Meeting topic: masakari)"04:00
openstackThe meeting name has been set to 'masakari'04:00
samPHi all for masakari04:00
tpatilHi04:00
*** tashiromt has joined #openstack-meeting04:00
samPtpatil: Hi04:00
samPSorry for absent last week04:01
samP#topic Critical bugs and patches04:01
*** openstack changes topic to "Critical bugs and patches (Meeting topic: masakari)"04:01
samPAny critical bugs or patches need to discuss?04:01
tpatilNo new bugs reported since Stein release04:02
samPtpatil: thanks.04:02
samPLet's move to Train work items04:03
samP#topic Train Work Items04:03
*** openstack changes topic to "Train Work Items (Meeting topic: masakari)"04:03
*** shilpasd has joined #openstack-meeting04:03
samPPlease share if any updates on Train work items04:03
tpatilReal time rendering of recovery workflow04:05
tpatilone issue fixed in fastener library used in taskflow will be reverted04:05
samPgot it. thanks.04:05
tpatilso we are again back to the square one.04:07
tpatilwithout this fix, we can implement the real time rendering04:07
tpatilI have a hack to proceed on our implementation without fixing this issue in fastener lib04:08
tpatilwe can monkey patch the method from fastener lib in masakari04:08
tpatilThis was the commit that will be reverted in issues 36: https://github.com/harlowja/fasteners/commit/467ed75ee1e9465ebff8b5edf452770befb9391304:08
tpatilif we monkey patch "_fetch_current_thread_functor" method, it might solve our problem04:09
tpatiluntil this issue in fixed properly in fastener library04:09
tpatilWe will give it a try and let you know about it04:09
samPsounds good. thanks04:10
tpatil2. Host monitor support in devstack04:10
tpatilshilpasd: do you have any update?04:11
shilpasdyes, almost we are through except STONITH configuration04:11
tpatilPlease describe the problem04:12
shilpasdDuring STONITH configuration, there we opening CRM interactive console and adding configuration and committing the same04:13
shilpasdYesterday we are able to do so, but now looking into during uninstall, need to remove though04:14
tpatilok so installation of host-monitor is working properly, correct?04:15
shilpasdyes04:15
tpatilWhat problem are you facing during uninstallation?04:16
samPAre you trying to remove CRM configs before uninstall?04:17
shilpasdSTONITH configuration adds resource tag in '/var/lib/pacemaker/cib/cib.xml'04:17
shilpasdsamP: No, we are removing corosync/pacemaker/crmsh and then removing same04:18
samPshilpasd: right.04:18
shilpasdwe tested manually also by removing resource tag, but it still shows entry04:18
tpatilif you stop the services and uninstall packages, will it not solve this problem04:19
shilpasdDuring stack, it ends with error 'WARNING: Resources my-stonith1,my-stonithA violate uniqueness for parameter "hostname": "<nodename>"04:19
shilpasdtpatil: along with remove and purge, suggesting to stop the services?04:20
tpatilDescribe what you are doing during uninstallation step?04:21
tpatilafter installation is complete, do you see host-monitor is running and the cluster status is ok?04:22
*** pcaruana has joined #openstack-meeting04:22
tpatilAll nodes are setup in the cluster?04:22
shilpasdtpatil: as of now manually tried, removing and purging corosync and pacemaker, updating  '/var/lib/pacemaker/cib/cib.xml' to remove STONITH entry and doing stack04:23
shilpasdtpatil: All nodes are setup in cluster, but firstly verifying on Controller only04:23
tpatilmanually means, outside devstack?04:24
shilpasdyes04:24
tpatilhmm04:24
tpatillet's discuss about this issue offline04:24
shilpasdHow to update xml file for remove on <resource> tag that needs to be llok out04:24
shilpasdsure, thanks04:24
samPnot completely understand what is the issue, but you may use 'crm resource cleanup' to clean the resource definitions04:26
samPshilpasd: thanks for working on this04:26
shilpasdsamP: thanks, will try this option and update accordingly04:27
samPshilpasd: np04:27
tpatil#link: https://clusterlabs.org/pacemaker/doc/en-US/Pacemaker/1.1/html-single/Pacemaker_Explained/index.html04:28
tpatilRefer to 2.3.3. Updating the Configuration Without Using XML04:29
shilpasdtpatil: thanks for this link, will go through04:29
tpatil3. Update Documentation04:30
tpatilNot yet started04:30
tpatilNeed to find out what changes are required first04:31
samPtpatil: thanks for updates04:32
samP#topic AOB04:34
*** openstack changes topic to "AOB (Meeting topic: masakari)"04:34
samPany other items to discuss?04:35
tpatilNo04:35
samPThanks. Nothing from my side.04:35
samPIf not other items, we could finish today's meeting here04:36
tpatilsamP: Ok04:37
*** markvoelker has joined #openstack-meeting04:37
samPOK then, Thank you all for attending to meeting04:37
samPPlease use the #openstack-masakari IRC @freenode or ML for further discussions.04:38
samP#endmeeting04:38
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"04:38
openstackMeeting ended Tue May 28 04:38:06 2019 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)04:38
openstackMinutes:        http://eavesdrop.openstack.org/meetings/masakari/2019/masakari.2019-05-28-04.00.html04:38
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/masakari/2019/masakari.2019-05-28-04.00.txt04:38
openstackLog:            http://eavesdrop.openstack.org/meetings/masakari/2019/masakari.2019-05-28-04.00.log.html04:38
*** psachin has quit IRC04:39
*** psachin has joined #openstack-meeting04:40
*** tpatil has quit IRC04:45
*** hongbin has quit IRC04:49
*** janki has joined #openstack-meeting04:51
*** markvoelker has quit IRC05:10
*** jchhatbar has joined #openstack-meeting05:27
*** janki has quit IRC05:30
*** sridharg has joined #openstack-meeting05:43
*** baojg has quit IRC05:49
Tenguhello there05:51
*** lpetrut has joined #openstack-meeting06:01
*** baojg has joined #openstack-meeting06:03
*** kopecmartin|off is now known as kopecmartin06:04
*** e0ne has joined #openstack-meeting06:06
*** markvoelker has joined #openstack-meeting06:07
*** sridharg has quit IRC06:20
*** slaweq has joined #openstack-meeting06:22
*** e0ne has quit IRC06:25
*** tetsuro has joined #openstack-meeting06:25
*** tashiromt has quit IRC06:32
*** sridharg has joined #openstack-meeting06:35
*** sridharg has quit IRC06:37
*** markvoelker has quit IRC06:40
*** macza has joined #openstack-meeting06:45
*** macza has quit IRC06:50
*** takamatsu has joined #openstack-meeting07:07
*** tetsuro has quit IRC07:09
*** jhesketh has quit IRC07:11
*** ricolin_ has joined #openstack-meeting07:12
*** tesseract has joined #openstack-meeting07:12
*** takamatsu has quit IRC07:13
*** ricolin has quit IRC07:15
*** yamamoto has joined #openstack-meeting07:18
*** yamamoto has quit IRC07:25
*** apetrich has joined #openstack-meeting07:29
*** dkushwaha has joined #openstack-meeting07:29
*** jhesketh has joined #openstack-meeting07:29
*** markvoelker has joined #openstack-meeting07:37
*** tpatil has joined #openstack-meeting07:45
*** rossella_s has joined #openstack-meeting07:49
*** tetsuro has joined #openstack-meeting07:51
*** takahashi-tsc has joined #openstack-meeting07:53
dkushwaha#startmeeting tacker08:03
openstackMeeting started Tue May 28 08:03:33 2019 UTC and is due to finish in 60 minutes.  The chair is dkushwaha. Information about MeetBot at http://wiki.debian.org/MeetBot.08:03
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.08:03
*** openstack changes topic to " (Meeting topic: tacker)"08:03
openstackThe meeting name has been set to 'tacker'08:03
dkushwaha#topic Roll Call08:03
*** openstack changes topic to "Roll Call (Meeting topic: tacker)"08:03
dkushwahawho is here for Tacker weekly meeting?08:04
tpatilHi08:04
hyunsikyangHi08:04
hyunsikyangall08:04
takahashi-tscHi08:04
JangwonLeeHi08:04
*** hjwon has joined #openstack-meeting08:05
*** hokeeeeun has joined #openstack-meeting08:05
dkushwahahello all08:05
*** jaewook_oh has joined #openstack-meeting08:05
*** shubham_potale has joined #openstack-meeting08:05
dkushwahaok lets start..08:08
dkushwaha#topic vnf-rolling-upgrade08:08
*** openstack changes topic to "vnf-rolling-upgrade (Meeting topic: tacker)"08:08
dkushwahahyunsikyang, I see you have updated spec08:08
hyunsikyangYes08:09
dkushwahahyunsikyang, Regarding my comment on #6508:09
dkushwahaany update ?08:09
hyunsikyangAh about your question, I amd Tomi replied.08:10
*** markvoelker has quit IRC08:10
dkushwahaActually my concern was: At any point of time if failure occurred on a host, is it meaningful to move only VNFs which created with tosca.policies.tacker.Maintenance? Rather than control should be at VNFM(Tacker deployment) level so whenever maintenance required, policy will be applicable for all VNF on that site.08:11
hyunsikyangAh.08:11
dkushwahaAlthough, with this, I have little doubt that how trigger will work. So I am open for both approach08:12
hyunsikyangAt first, in the case of VNF maintanance.. It's fine.08:12
hyunsikyangBut like your comment, Whatif, host have some problem , all VNF have to move..08:12
hyunsikyangSo about a second case, we think that it can cover by fenix. For example, we make a VNF group and assigned to move it all at a one time for specific group.08:14
*** yamamoto has joined #openstack-meeting08:14
hyunsikyangBut, Can i get your opinion as a view of tacker?08:14
*** akhil_jain has joined #openstack-meeting08:16
dkushwahahyunsikyang, my point is: maintenance should be for a complete deployed site(which comes under Tacker), rather than just keeping it for VNFs08:16
hyunsikyangYes I understood.08:17
dkushwaha* for few VNFs08:17
*** baojg has quit IRC08:18
hyunsikyangabout a second case, we can support it later.. by fenix or by tacker.. we should think about it later.08:18
*** baojg has joined #openstack-meeting08:18
dkushwahahyunsikyang, ok, Thanks, So lets have a clear stand, and move with your approach(VNF specific).08:19
hyunsikyangOK.08:19
dkushwahahyunsikyang, regarding comment on #2108:20
hyunsikyangI also discuss it with fenix again and let you know.08:20
hyunsikyangYEs08:20
dkushwahacould you please update spec with the current scopes and future items08:20
hyunsikyangOK. Did you check my comment?08:21
dkushwahahyunsikyang, yea i checked, and it will be good to mention scopes and action items clearly08:23
hyunsikyangOK. thanks.08:23
dkushwahahyunsikyang, Thanks08:24
dkushwaha#topic VNF packages support08:25
*** openstack changes topic to "VNF packages support (Meeting topic: tacker)"08:25
dkushwahatpatil, any update on this?08:26
*** ralonsoh has joined #openstack-meeting08:26
tpatildkushwaha: We are updating the VNF package specs. Will upload it soon08:27
dkushwahatpatil, ok, thanks08:27
tpatilIn vPTG, we received few comments from you to update Upload workflow and also explain what validation will be carried out.08:28
tpatilWorking on the same. Also ,we have identified few other missing points.08:28
tpatilAfter we address all these points will upload the specs08:28
dkushwahatpatil, nice08:29
*** baojg has quit IRC08:33
*** dkushwaha has quit IRC08:34
*** jaewook_lee has joined #openstack-meeting08:34
hyunsikyanghe has some connection problem.. waiting..him08:35
*** hochanlee has joined #openstack-meeting08:35
hyunsikyangHello jaewook_lee and hochanlee08:36
jaewook_leeNice meet you hyunsikyang~08:36
hochanleeHi08:37
hyunsikyangwe are wating PTL..08:37
*** tetsuro has quit IRC08:38
*** mikakoski has joined #openstack-meeting08:39
hyunsikyangwho is here now?08:39
*** dkushwaha has joined #openstack-meeting08:40
dkushwahaIt seems I lost my connection08:40
hyunsikyanganyway welcome back.08:40
dkushwahahyunsikyang, :D08:41
dkushwaha#topic OpenDiscussion08:41
*** openstack changes topic to "OpenDiscussion (Meeting topic: tacker)"08:41
dkushwahaI have nothing from my side for now08:41
hyunsikyanghttps://review.opendev.org/#/c/659559/08:42
patchbotpatch 659559 - tacker - Kubernetes as VIM in Tacker in tacker-specs - 6 patch sets08:42
hyunsikyangPlease review this typo too.08:42
hyunsikyangwhen i was checking Kubernetes VIM, I found this one too..08:42
dkushwahahyunsikyang, ok, will review it08:43
hyunsikyangIt is not urgent, but, it should be fixed .08:43
hyunsikyangI am done08:44
dkushwahaFolks, I just want to request all, to please help on review side as well.08:45
hyunsikyangOK:) I will08:45
dkushwahahyunsikyang, thanks08:46
dkushwahaIf no one have any other item to discuss, we can close today meeting08:46
hochanleeWe have question about auto healing and scaling for vnffg spec08:46
dkushwahahochanlee, yes please08:47
hochanleeAs we said in mailing list, we want to go on developing those features.08:49
hochanleeshould we use those existing spec? or can we write new spec for those features?08:50
jaewook_leeHi dkuskhwaha! Hochan and me are going to develop the healing and scaling function for VNFFG in pike specifications.08:50
jaewook_leeI wonder whether we need define new spec or not ?08:51
dkushwahahochanlee, jaewook_lee Do you see any challenge with existing spec?08:52
dkushwahaIf no, it is good to go with existing spec08:52
dkushwahaas both spec are already merged08:53
dkushwahahttps://review.opendev.org/#/c/469975/ https://review.opendev.org/#/c/434974/08:53
patchbotpatch 469975 - tacker-specs - Enable scaling function for VNFFG (MERGED) - 10 patch sets08:53
patchbotpatch 434974 - tacker-specs - Add auto-healing function for VNFFG (MERGED) - 13 patch sets08:53
jaewook_leeOkay! we are going to develop as follow the defined specs!08:54
hochanleeOK Thanks we will :)08:54
dkushwahajaewook_lee, hochanlee Thanks for heading-up to implement these features08:55
*** LiangFang has quit IRC08:56
dkushwahaOk, Thanks Folks for joining this meeting.08:56
dkushwahaclosing it now.08:57
*** tpatil has quit IRC08:57
dkushwaha#endmeeting08:57
*** tetsuro has joined #openstack-meeting08:57
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"08:57
openstackMeeting ended Tue May 28 08:57:09 2019 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)08:57
openstackMinutes:        http://eavesdrop.openstack.org/meetings/tacker/2019/tacker.2019-05-28-08.03.html08:57
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/tacker/2019/tacker.2019-05-28-08.03.txt08:57
openstackLog:            http://eavesdrop.openstack.org/meetings/tacker/2019/tacker.2019-05-28-08.03.log.html08:57
*** dkushwaha has left #openstack-meeting08:57
*** takahashi-tsc has left #openstack-meeting08:57
*** jaewook_oh has quit IRC08:57
*** priteau has joined #openstack-meeting08:58
*** e0ne has joined #openstack-meeting09:00
*** hokeeeeun has quit IRC09:00
*** jaewook_lee has quit IRC09:01
*** hjwon has quit IRC09:01
*** markvoelker has joined #openstack-meeting09:07
*** jesusaur has quit IRC09:17
*** boxiang has joined #openstack-meeting09:17
*** jchhatba_ has joined #openstack-meeting09:18
*** jesusaur has joined #openstack-meeting09:20
*** jchhatbar has quit IRC09:21
*** rossella_s has quit IRC09:22
*** yoctozepto has joined #openstack-meeting09:26
*** markvoelker has quit IRC09:40
*** rossella_s has joined #openstack-meeting09:46
*** ricolin_ has quit IRC10:02
*** tetsuro has quit IRC10:26
*** markvoelker has joined #openstack-meeting10:37
*** jaypipes has joined #openstack-meeting10:42
*** bbowen has quit IRC10:43
*** jaypipes has quit IRC10:46
*** mikakoski has quit IRC10:49
*** yamamoto has quit IRC10:50
*** baojg has joined #openstack-meeting10:51
*** jaypipes has joined #openstack-meeting10:58
*** yamamoto has joined #openstack-meeting11:04
*** sridharg has joined #openstack-meeting11:06
*** markvoelker has quit IRC11:11
*** psachin has quit IRC11:14
*** yamamoto has quit IRC11:14
*** yamamoto has joined #openstack-meeting11:20
*** yamamoto has quit IRC11:21
*** njohnston_ has joined #openstack-meeting11:22
*** yamamoto has joined #openstack-meeting11:32
*** ttsiouts has joined #openstack-meeting11:33
*** yamamoto has quit IRC11:34
*** ttsiouts has quit IRC11:38
*** crudo has joined #openstack-meeting11:38
*** macza has joined #openstack-meeting11:39
*** raildo has joined #openstack-meeting11:42
*** macza has quit IRC11:43
*** yamamoto has joined #openstack-meeting11:44
*** imsurit_ofc has quit IRC11:45
*** yamamoto has quit IRC11:48
*** carloss has joined #openstack-meeting11:48
*** psachin has joined #openstack-meeting11:57
*** bbowen has joined #openstack-meeting11:58
*** rfolco has joined #openstack-meeting11:59
*** electrofelix has joined #openstack-meeting12:02
*** markvoelker has joined #openstack-meeting12:06
*** yamamoto has joined #openstack-meeting12:08
*** rf0lc0 has joined #openstack-meeting12:09
*** lbragstad has joined #openstack-meeting12:09
*** rfolco has quit IRC12:10
*** priteau has quit IRC12:13
*** priteau has joined #openstack-meeting12:14
*** mattia_ has quit IRC12:15
*** mattia has joined #openstack-meeting12:15
*** yamamoto has quit IRC12:17
*** haleyb has joined #openstack-meeting12:25
*** _erlon_ has joined #openstack-meeting12:26
*** chason has joined #openstack-meeting12:31
*** rossella_s has quit IRC12:32
*** rf0lc0 has quit IRC12:34
*** rfolco has joined #openstack-meeting12:34
*** rf0lc0 has joined #openstack-meeting12:35
*** boxiang has quit IRC12:36
*** boxiang has joined #openstack-meeting12:37
*** rfolco has quit IRC12:38
*** igordc has joined #openstack-meeting12:59
*** sridharg has quit IRC12:59
*** yamamoto has joined #openstack-meeting13:02
*** sridharg has joined #openstack-meeting13:02
*** baojg has quit IRC13:08
*** jchhatba_ has quit IRC13:08
*** jchhatba_ has joined #openstack-meeting13:08
*** crudo has left #openstack-meeting13:11
*** mriedem has joined #openstack-meeting13:15
*** beekneemech is now known as bnemec13:18
*** priteau has quit IRC13:20
*** yamamoto has quit IRC13:23
*** yamamoto has joined #openstack-meeting13:24
*** psachin has quit IRC13:24
*** _alastor_ has joined #openstack-meeting13:27
*** vishalmanchanda has joined #openstack-meeting13:29
*** rf0lc0 is now known as rfolco13:32
*** bbowen has quit IRC13:34
*** enriquetaso has joined #openstack-meeting13:38
*** DinaBelova has joined #openstack-meeting13:42
*** akhil_jain has quit IRC13:45
*** priteau has joined #openstack-meeting13:46
*** rf0lc0 has joined #openstack-meeting13:46
*** rfolco has quit IRC13:48
*** jchhatba_ has quit IRC13:55
*** rf0lc0 is now known as rfolco13:57
*** shintaro has joined #openstack-meeting13:58
*** wwriverrat has joined #openstack-meeting13:59
*** jrbalderrama has joined #openstack-meeting14:07
*** yamamoto has quit IRC14:11
*** yamamoto has joined #openstack-meeting14:14
*** yamamoto has quit IRC14:14
*** yamamoto has joined #openstack-meeting14:15
*** Liang__ has joined #openstack-meeting14:17
*** yamamoto has quit IRC14:19
*** yamamoto has joined #openstack-meeting14:20
*** yamamoto has quit IRC14:20
*** yamamoto has joined #openstack-meeting14:20
*** yamamoto has quit IRC14:25
*** dtrainor has joined #openstack-meeting14:29
*** lpetrut has quit IRC14:30
*** gagehugo has joined #openstack-meeting14:34
*** dasp has quit IRC14:36
*** shintaro has quit IRC14:38
*** lseki has joined #openstack-meeting14:40
*** Liang__ has quit IRC14:49
*** jrbalderrama has quit IRC14:51
*** baojg has joined #openstack-meeting14:53
*** bbowen has joined #openstack-meeting14:56
*** baojg has quit IRC14:58
*** lpetrut has joined #openstack-meeting14:59
*** dklyle has joined #openstack-meeting14:59
*** cheng1__ has joined #openstack-meeting15:00
*** hemna has joined #openstack-meeting15:01
*** e0ne has quit IRC15:05
*** hemna has quit IRC15:19
*** hemna has joined #openstack-meeting15:21
*** lpetrut has quit IRC15:38
*** macza has joined #openstack-meeting15:41
*** macza has quit IRC15:44
*** macza has joined #openstack-meeting15:45
*** gagehugo has left #openstack-meeting15:56
*** cheng1__ has quit IRC15:57
*** kopecmartin is now known as kopecmartin|off15:58
*** vishalmanchanda has quit IRC15:59
*** mlavalle has joined #openstack-meeting15:59
slaweq#startmeeting neutron_ci16:00
openstackMeeting started Tue May 28 16:00:04 2019 UTC and is due to finish in 60 minutes.  The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot.16:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:00
*** openstack changes topic to " (Meeting topic: neutron_ci)"16:00
njohnston_o/16:00
openstackThe meeting name has been set to 'neutron_ci'16:00
slaweqhi16:00
*** njohnston_ is now known as njohnston16:00
mlavalleo/16:00
ralonsohhi16:01
slaweqok, lets start16:01
slaweqGrafana dashboard: http://grafana.openstack.org/dashboard/db/neutron-failure-rate16:01
slaweqplease open for later :)16:01
slaweq#topic Actions from previous meetings16:01
*** openstack changes topic to "Actions from previous meetings (Meeting topic: neutron_ci)"16:02
slaweq#undo16:02
openstackRemoving item from minutes: #topic Actions from previous meetings16:02
slaweqsorry, I forgot16:02
mlavalleLOL16:02
slaweqagenda for today's meeting16:02
slaweq#link https://etherpad.openstack.org/p/neutron-ci-meetings16:02
slaweqand now lets start16:02
slaweq#topic Actions from previous meetings16:02
*** openstack changes topic to "Actions from previous meetings (Meeting topic: neutron_ci)"16:02
slaweqmlavalle to continue debuging reasons of neutron-tempest-plugin-dvr-multinode-scenario failures16:02
mlavalleI didn't dovote as much time as I wanted but I made some progress16:03
*** wwriverrat has quit IRC16:03
mlavallelooking at many patches, one common test case that fails is test_connectivity_through_2_routers16:03
mlavalleso I filed a bug: https://bugs.launchpad.net/neutron/+bug/183076316:04
openstackLaunchpad bug 1830763 in neutron "Debug neutron-tempest-plugin-dvr-multinode-scenario failures" [High,Confirmed] - Assigned to Miguel Lavalle (minsel)16:04
mlavalleassigned it to myself16:04
slaweqit's test written by me :/16:04
mlavalleand added a Kibana query: http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22test_connectivity_through_2_routers%5C%22%20AND%20build_status:%5C%22FAILURE%5C%22%20AND%20build_branch:%5C%22master%5C%22%20AND%20build_name:%5C%22neutron-tempest-plugin-dvr-multinode-scenario%5C%22%20AND%20project:%5C%22openstack%2Fneutron%5C%2216:04
mlavalleso I will be focusing on this one over the next few days16:05
mlavallethat's all I have to say about this16:05
slaweqok, thx for update mlavalle16:05
slaweqdo You mind if I will assign it as an action for next week to You?16:05
mlavalleplease do16:06
slaweq#action mlavalle to debug neutron-tempest-plugin-dvr-multinode-scenario failures (bug 1830763)16:06
openstackbug 1830763 in neutron "Debug neutron-tempest-plugin-dvr-multinode-scenario failures" [High,Confirmed] https://launchpad.net/bugs/1830763 - Assigned to Miguel Lavalle (minsel)16:06
slaweqthx16:06
*** tesseract has quit IRC16:06
slaweqok, so next one16:06
slaweqmlavalle to talk with nova folks about slow responses for metadata requests16:06
mlavalleso I decided not to16:06
mlavalleanalyzing in detail some logs and the code16:07
mlavalleand after conversation with slaweq, we decided that the problem doesn't seem to be on the Nova side16:07
slaweqyes16:07
slaweqI agree :)16:07
slaweqmlavalle: do You want to explain what You found in logs there?16:09
mlavallecorrelating the code with the logs, we found that there time elpased16:10
*** ricolin_ has joined #openstack-meeting16:10
mlavallebetween sending the request for keypairs to Nova and getting the response was less than 2 secs16:10
mlavallethat's it in a nutshell16:11
slaweqbut from VM PoV there is (probably) more that 10 seconds for this request and that's why it fails16:11
mlavalleyeap16:11
slaweqsome time ago I started some patch to add zuul role to fetch journal log: https://review.opendev.org/#/c/643733/16:12
patchbotpatch 643733 - zuul/zuul-jobs - Add role to fetch journal log from test node - 3 patch sets16:12
slaweqbut I never had time to work on this16:12
slaweqI today respined this patch as it may help with this issue also16:13
slaweqbecause e.g. haproxy logs are in journal log probably16:13
mlavalleyeah16:13
mlavallegood idea16:13
slaweqso I will assign this to myself as an action for next week :)16:13
slaweq#action slaweq to continue work on fetch-journal-log zuul role16:14
slaweqthat way I will force myself to spent some time on it :)16:14
slaweqok, lets move forward16:14
slaweqnext one16:14
slaweqslaweq to reopen bug related to failures of neutron.tests.fullstack.test_l3_agent.TestHAL3Agent.test_ha_router_restart_agents_no_packet_lost16:14
slaweqDone, bug https://bugs.launchpad.net/neutron/+bug/1798475 reopened16:14
openstackLaunchpad bug 1798475 in neutron "Fullstack test test_ha_router_restart_agents_no_packet_lost failing" [High,Confirmed] - Assigned to LIU Yulong (dragon889)16:14
slaweqI send also patch to mark this test as unstable again https://review.opendev.org/#/c/660592/16:15
patchbotpatch 660592 - neutron - Mark fullstack test_ha_router_restart_agents_no_pa... - 1 patch set16:15
slaweqplease check this patch if You will have some time :)16:15
slaweqand the last one was:16:15
slaweqralonsoh to propose patch with additional logging to help debug https://bugs.launchpad.net/neutron/+bug/179955516:15
openstackLaunchpad bug 1799555 in neutron "Fullstack test neutron.tests.fullstack.test_dhcp_agent.TestDhcpAgentHA.test_reschedule_network_on_new_agent timeout" [High,Confirmed] - Assigned to Rodolfo Alonso (rodolfo-alonso-hernandez)16:15
ralonsohhttps://review.opendev.org/#/c/660785/16:16
patchbotpatch 660785 - neutron - Add debug information to AutoScheduler and BaseSch... - 4 patch sets16:16
slaweqthx ralonsoh16:16
slaweqI will review it tonight or tomorrow morning16:16
ralonsohthanks!16:17
mlavalleit's also in my pile16:17
njohnston+116:17
slaweqok16:17
slaweqthat's all from last week16:17
slaweqquestions/comments?16:17
mlavallenone from me16:18
slaweqok, lets move on then16:18
slaweq#topic Stadium projects16:18
*** openstack changes topic to "Stadium projects (Meeting topic: neutron_ci)"16:18
slaweqPython 3 migration16:18
slaweqStadium projects etherpad: https://etherpad.openstack.org/p/neutron_stadium_python3_status16:19
slaweqnjohnston: any update on this?16:19
njohnstonnothing at present, no.16:19
*** gyee has joined #openstack-meeting16:20
slaweqI think that thanks tidwellr we have migrated neutron-dynamic-routing to py316:20
slaweqit's done in https://review.opendev.org/#/c/657409/16:20
patchbotpatch 657409 - neutron-dynamic-routing - Convert CI jobs to python 3 (MERGED) - 8 patch sets16:20
slaweqso I will update etherpad16:21
njohnstonexcellent16:21
mlavalleGreat16:21
slaweqI will try to pick up one of the projects this week if I will have couple of free minutes :)16:22
slaweqok, next thing related to stadium projects16:22
slaweqtempest-plugins migration16:22
slaweqEtherpad: https://etherpad.openstack.org/p/neutron_stadium_move_to_tempest_plugin_repo16:22
slaweqfor bgpvpn we merged first patch https://review.opendev.org/#/c/652991/16:23
patchbotpatch 652991 - neutron-tempest-plugin - Rehome tempest tests from networking-bgpvpn repo (MERGED) - 14 patch sets16:23
slaweqand I have question regarding to this patch16:23
slaweqI unfortunatelly added  neutron-tempest-plugin-bgpvpn-bagpipe job to "neutron-tempest-plugin-jobs" template16:23
slaweqthus it's now run on every neutron patch16:23
slaweqdo we want it like that?16:24
slaweqI think that it shouldn't be in this template, right?16:24
mlavalleI don't think so16:24
slaweqyes, I though that when I today realized that it's running in neutron gate too16:24
njohnstonI agree, I don't think so16:24
slaweqso I will change that16:24
slaweq#action slaweq to remove neutron-tempest-plugin-bgpvpn-bagpipe from "neutron-tempest-plugin-jobs" template16:24
slaweqexcept that, there is also second patch for bgpvpn project: https://review.opendev.org/#/c/657793/16:25
patchbotpatch 657793 - networking-bgpvpn - Rehome tempest tests to neutron-tempest-plugin repo - 1 patch set16:25
slaweqplease review it if You will have some time16:25
slaweqespecially mlavalle as You probably have +2 power in this repo :)16:25
mlavalleI do, indeed16:25
slaweqths16:26
slaweq*thx16:26
njohnstonthe tests for the migration of neutron-fwaas tempest tests are still failing, but I am concerned that is because fwaas is broken for other reasons, as slaweq noticed in the work to move the neutron-fwaas-fullstack job to zuulv3 https://review.opendev.org/64452616:26
patchbotpatch 644526 - neutron-fwaas - Switch neutron-fwaas-fullstack job to zuulv3 syntax - 20 patch sets16:26
slaweqnjohnston: but what I noticed in my patch is that fullstack job in fwaas repo is broken16:27
mlavallenjohnston: I intend to send an email to Sridhar and xgerman16:27
slaweqall other jobs are working fine there16:27
mlavalleo about their involvment with fwaas16:27
njohnstonI don't even have a working email address for xgerman since he left Rackspace16:27
mlavalleafter they respond, we can proceed to send a plea of help the the general ML16:27
mlavalleI have a way to find it16:28
njohnstonok cool16:28
slaweqmlavalle has got his secret PTL's tools to find it :P16:28
njohnstonanyway I will keep digging to see if I can find the root cause of the issue at least16:28
* mlavalle wishes that was the case, LOL16:28
slaweq:)16:29
haleybi have german's email if you need it16:29
mlavallecool16:29
slaweqok, njohnston if You would need any help with this zuul issues, please ping me16:29
slaweqha, so haleyb is this secret PTL's tool :P16:29
haleyb:)16:29
slaweqok, so moving on16:30
slaweqthere is also networking-sfc project16:30
slaweqand first patch is merged https://review.opendev.org/#/c/65301216:30
patchbotpatch 653012 - neutron-tempest-plugin - Migrate networking-sfc tests to neutron-tempest-pl... (MERGED) - 10 patch sets16:30
slaweqbcafarel said that second patch https://review.opendev.org/#/c/653747 is also ready for review16:31
patchbotpatch 653747 - networking-sfc - Complete move of networking-sfc tempest tests to t... - 21 patch sets16:31
slaweqso please add it to Your review list :)16:31
slaweqespecially people who have +2 in this repo16:31
slaweqok16:32
mlavallewhich I do16:32
slaweqany other questions/comments related to stadium projects?16:32
mlavallejust to say I didn't make much proress with vpnaas this week16:33
mlavalleI'll try again over the next few days16:33
slaweqsure, no rush :)16:33
slaweqwe don't have any deadline for this16:33
slaweq(but You can turn on hulk some day if it will take too long ;D)16:34
* mlavalle shudders just thinking about it16:34
slaweqLOL16:35
slaweqok, lets move on16:35
slaweq#topic Grafana16:35
*** openstack changes topic to "Grafana (Meeting topic: neutron_ci)"16:35
slaweq#link http://grafana.openstack.org/dashboard/db/neutron-failure-rate16:35
slaweqI don't see any urgent issues there16:37
mlavallenot too bad16:37
slaweqall looks "as usually"16:37
slaweqand indeed not very bad :)16:37
mlavalleit clearly shows the effect of the long weekend in the US16:37
slaweqwe still have some failures in tempest jobs and in functional/fullstack jobs but nothig very bad16:37
mlavallemostly functional, the way I see it16:38
njohnstonis that what happened to the unit tests graph in the gate queue - disproportionate impact of job failures due to low volume?16:38
slaweqnjohnston: I think so - look that this highest failure rate is when it was run only 3 times16:39
njohnstonmakes sense16:39
slaweqso it could be even some DNM patch with broken tests :)16:39
*** hemna has quit IRC16:39
slaweqlets see in next couple of days how it will be16:39
slaweqok, lets move on then16:40
slaweq#topic fullstack/functional16:40
*** openstack changes topic to "fullstack/functional (Meeting topic: neutron_ci)"16:40
slaweqI was looking at some recent failed jobs16:41
slaweqand I found fullstack failure on neutron.tests.fullstack.test_l3_agent.TestHAL3Agent. test_ha_router_restart_agents_no_packet_lost16:41
slaweqwhich I already propsoed to mark as unstable again16:41
slaweqand bug is reopened for this16:41
slaweqbut also liuyulong|away have patch which may help for this https://review.opendev.org/#/c/660905/16:42
patchbotpatch 660905 - neutron - Set neutron-keepalived-state-change proctitle - 1 patch set16:42
slaweqso I think that we don't need to talk about this failed test too much here16:42
mlavalleok16:43
slaweqregarding to functional tests I found 2 issues16:43
slaweqone in neutron.tests.functional.test_server.TestWsgiServer. test_restart_wsgi_on_sighup_multiple_workers:16:43
slaweqhttp://logs.openstack.org/69/655969/3/check/neutron-functional/2a0533c/testr_results.html.gz16:43
slaweqaccording to logstash it happend twice in last week16:45
slaweqso it's not too much for now16:45
mlavalleyeah16:45
slaweqIMO let's keep an eye on it and we will see how it will be16:45
slaweqwhat do You think?16:45
mlavalleyes16:46
*** hemna has joined #openstack-meeting16:46
slaweqand the second issue, which I found at least twice in last couple of days16:46
slaweqneutron.tests.functional.agent.test_l2_ovs_agent.TestOVSAgent. test_assert_br_phys_patch_port_ofports_dont_change16:46
slaweqhttp://logs.openstack.org/87/658787/6/check/neutron-functional-python27/100ec44/testr_results.html.gz16:47
slaweqhttp://logs.openstack.org/05/660905/1/check/neutron-functional/f48d9de/testr_results.html.gz16:47
slaweqso, this one is also not happening very often for now16:48
slaweqlets also keep an eye on it and we will see how it will be16:48
slaweqok?16:48
mlavallecool16:48
slaweqok16:48
slaweqother than that I think it's good16:48
slaweqregarding tempest/scenario tests the issue which we have is well known (ssh problems) and we already talked about it16:49
mlavalleyeap16:49
slaweqso, I have one last topic for today16:49
slaweq#topic Open discussion16:49
*** armax has quit IRC16:50
*** openstack changes topic to "Open discussion (Meeting topic: neutron_ci)"16:50
slaweqI wanted to ask about one thing here16:50
slaweqrecently I found that we don't have any API/scenario tests for port forwarding in neutron-tempest-plugin repo16:50
mlavalleonly scenario?16:50
slaweqso we have some e.g. functional tests of course but we are missing any end-to-end tests16:51
slaweqso I started adding such tests in neutron-tempest-plugin repo16:51
slaweqbut I wanted to ask if we can do something to avoid such things in future16:51
slaweqI proposed some small update to reviewers guide https://review.opendev.org/66177016:51
patchbotpatch 661770 - neutron - Add short info about tempest API/scenario tests to... - 2 patch sets16:51
slaweqso please check it16:51
slaweqbut maybe there is something else what we can do also16:52
slaweqwhat do You think?16:52
mlavalleNice!16:52
mlavalleThanks!16:52
njohnstonyeah, earlier today I noticed that it doesnt look like we have any kind of testing for vlan trunking that makes sure it works when instances get migrated16:52
slaweqnjohnston: yep, so it's second thing :/16:52
njohnstonit should be a criterion for completion of the feature16:52
njohnstonslaweq++16:53
slaweqnjohnston: I agree, that's why I added this note to reviewers guide16:53
slaweqbut maybe it should be written also somewhere else?16:53
slaweqI don't know TBH :)16:53
*** wwriverrat has joined #openstack-meeting16:54
*** baojg has joined #openstack-meeting16:54
*** wwriverrat has quit IRC16:55
mlavalleI think that's enough16:56
slaweqok, thx mlavalle and njohnston for opinions :)16:56
haleybi had one question16:56
slaweqand we should all remember about this during reviews16:56
slaweqhaleyb: sure16:56
haleybi might have actually thought of an answer, but i'm trying to fix one of the OVN periodic jobs16:57
haleybhttps://review.opendev.org/#/c/661065/16:57
patchbotpatch 661065 - neutron - Fix OVS build issue on Fedora - 1 patch set16:57
haleybbut they only run on the master branch16:57
haleybi was wondering if there was a way to trigger that job on any change16:57
haleybor should i just add it to a test patch in the regular job run16:58
slaweqhaleyb: You can add it to check queue in zuul config file16:58
slaweqand then it will be run on any patch16:58
haleybslaweq: yes, that's what i thought too just a second ago16:58
haleybwe never much look at those periodic jobs16:59
slaweqit's here https://github.com/openstack/networking-ovn/blob/master/zuul.d/project.yaml16:59
*** baojg has quit IRC16:59
slaweqhaleyb: for neutron I'm usually looking before ci meeting if they are not failing too much16:59
slaweqbut except that, never :)16:59
*** mattw4 has joined #openstack-meeting17:00
haleyb:(17:00
slaweqok, it's time to end meeting17:00
haleybtime is up17:00
slaweq#endmeeting17:00
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"17:00
clarkbnot sure if people know but you can use openstack health to subscribe to rss/atom feeds on jobs17:00
mlavalleo/17:00
openstackMeeting ended Tue May 28 17:00:21 2019 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)17:00
slaweqthx for attending17:00
openstackMinutes:        http://eavesdrop.openstack.org/meetings/neutron_ci/2019/neutron_ci.2019-05-28-16.00.html17:00
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/neutron_ci/2019/neutron_ci.2019-05-28-16.00.txt17:00
slaweqo/17:00
openstackLog:            http://eavesdrop.openstack.org/meetings/neutron_ci/2019/neutron_ci.2019-05-28-16.00.log.html17:00
clarkbyou can also use the zuul builds tab to manually poll for updates17:00
slaweqclarkb: I didn't know that17:00
*** ricolin_ has quit IRC17:00
slaweqI will check17:00
slaweqthx17:00
*** mlavalle has left #openstack-meeting17:00
*** sridharg has quit IRC17:05
*** irclogbot_0 has quit IRC17:17
*** irclogbot_1 has joined #openstack-meeting17:18
*** iyamahat has joined #openstack-meeting17:25
*** armax has joined #openstack-meeting17:25
*** igordc has quit IRC17:34
*** yamahata has joined #openstack-meeting17:44
*** electrofelix has quit IRC17:49
*** ttsiouts has joined #openstack-meeting17:49
*** igordc has joined #openstack-meeting17:55
*** igordc has quit IRC17:55
*** igordc has joined #openstack-meeting17:55
*** priteau has quit IRC17:56
*** ttsiouts has quit IRC17:59
*** hongbin has joined #openstack-meeting18:03
*** hemna has quit IRC18:15
*** hemna has joined #openstack-meeting18:16
*** efried has quit IRC18:17
*** efried has joined #openstack-meeting18:18
*** wwriverrat has joined #openstack-meeting18:19
*** wwriverrat has quit IRC18:21
*** wwriverrat has joined #openstack-meeting18:23
*** jesusaur has quit IRC18:30
*** Shrews has joined #openstack-meeting18:39
*** wwriverrat has quit IRC18:41
*** wwriverrat has joined #openstack-meeting18:41
*** wwriverrat has quit IRC18:42
*** wwriverrat has joined #openstack-meeting18:43
*** lpetrut has joined #openstack-meeting18:47
*** wwriverrat has quit IRC18:47
*** lpetrut has quit IRC18:51
*** baojg has joined #openstack-meeting18:55
clarkbAnyone else around for the infra team meeting? we'll get started in a few minutes18:59
mordredclarkb: nothing to see here19:00
*** baojg has quit IRC19:00
cmurphyo/19:00
corvuso/19:00
corvusmrprhm...19:00
ianwo/19:00
clarkb#startmeeting infra19:01
*** patchbot has left #openstack-meeting19:01
openstackMeeting started Tue May 28 19:01:38 2019 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.19:01
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.19:01
*** openstack changes topic to " (Meeting topic: infra)"19:01
openstackThe meeting name has been set to 'infra'19:01
clarkb#link http://lists.openstack.org/pipermail/openstack-infra/2019-May/006391.html The agenda19:01
clarkbSorry for getting that out late. I thought I would have computer time yesterday but the kids just got bikes and insisted on basically riding them all day19:02
fungisoon they can just ride them all day while you get more computer time ;)19:02
fungian investment in your future19:03
clarkbthat is the goal19:03
clarkb#topic Announcements19:03
*** openstack changes topic to "Announcements (Meeting topic: infra)"19:03
clarkbI announce that I have no other announcements19:03
fungiconcise19:03
clarkbAnything worth mentioning that didn't make it onto the agenda?19:03
clarkbsounds like no. Let's move on19:05
clarkb#topic Actions from last meeting19:05
*** openstack changes topic to "Actions from last meeting (Meeting topic: infra)"19:05
clarkb#link http://eavesdrop.openstack.org/meetings/infra/2019/infra.2019-05-21-19.01.txt minutes from last meeting19:05
clarkbNo recorded actions from last meeting19:05
clarkb#topic Specs approval19:05
*** openstack changes topic to "Specs approval (Meeting topic: infra)"19:05
clarkbAs mentioned previously there are a few new specs up since the summit/forum/ptg. If you have a free moment they would likely appreciate your feedback19:06
clarkbbut I don't think anything is at a point where we are ready to approve things19:06
clarkbWith that out of the way lets jump into the recent happenings.19:06
clarkb#topic Priority Efforts19:06
*** openstack changes topic to "Priority Efforts (Meeting topic: infra)"19:06
clarkb#topic Update Config Management19:07
*** openstack changes topic to "Update Config Management (Meeting topic: infra)"19:07
clarkbPuppet 4 upgrades are basically done at this point19:07
cmurphydo we have anything left?19:07
cmurphyis ask still undone?19:07
clarkbthere are three hosts that are on puppet 3. The arm64 builder. The arm64 mirror. And the backup target host19:07
clarkbcmurphy: ask is done19:07
clarkbthe Arm64 nodes are puppet 3 because there is no puppet 4 packaging for arm6419:07
clarkbianw has done a bunch of work to ansible our mirrors nodes so the mirror node is a non issue19:08
clarkbThat mostly leaves us with the arm64 builder and backup target host19:08
clarkbThinking about this more I think the backup target host should be straightforward to ansible so I don't think we need to upgrade puppet on that one19:08
corvusagree19:08
clarkb(in fact we may already effectively ansible it after our base stuff moved into ansible)19:09
corvus(it's mostly base++)19:09
mordredyah19:09
clarkbI think that leaves us with the arm64 nodepool builder to worry about19:09
ianwi can take an action item to get onto that, as i've played quite a bit with bup now for prior volume switches19:09
clarkbianw: thanks19:09
fungiany idea if anyone's tried to rebuild the puppet 4 source packages on arm64?19:09
clarkb#action ianw ansiblify our backup host so that we don't have to upgrade it to puppet 419:09
clarkbfungi: I haven't heard of any trying that19:10
ianwyeah; i guess the distros might have done that, but it seems not19:11
*** tobiash has joined #openstack-meeting19:11
clarkbWe can probably prioritize ansiblifying nodepool instead19:11
corvuswe could probably ansible/containerize nodepool before the rest of the zuul system19:11
clarkbya19:11
ianw(i haven't done a really deep investigation, though .. there may be conversations out there about it)19:11
clarkbthat19:11
corvusit has fewer interactions with the other systems19:11
corvus(ansible/containerizing zuul itself has dependencies on gerrit/opendev/etc, which we are close to resolving, but haven't yet, which makes it more complex)19:12
fungireally just need to containerize the builder for that matter19:12
clarkbI think we can pretty much say puppet 4 is done at this point and we'll paper over corners with more ansible as necessary.19:12
clarkbbig thank you to cmurphy for pushing this work and getting it done19:12
fungibut would presumably do all of nodepool in one coordinated effort19:12
clarkbfungi: I expect the images will be built for all the bits of nodepool so may as well19:13
fungiyes, thanks again cmurphy! this has been a huge help19:13
corvusfungi: yeah, with the nodepool config files, i think doing all of nodepool together would make the most sense19:13
cmurphy\o/19:13
corvuscmurphy++19:13
clarkbOn the ansible and container side of things we should have reliable docker image builds now regardless of ip version19:14
clarkbAnything else to bring up on this topic?19:15
clarkbOh right Zuul Driven CD19:16
clarkblast week we merged the base playbook split out change19:16
clarkbso if you have/had changes that touch base.yaml you'll likely need to rebase and do some minor reorging19:17
clarkbThis should put us in a position to start having zuul trigger some of the other playbooks when things merge (like dns zone files)19:17
corvusianw had made new changes to other systems since then based on the new framework (yay!)19:17
fungialso this has meant instant increase in test coverage for us19:18
corvusas for zuul itself, i think the next step in that system is moving the opendev project creation from ansible into python19:18
clarkbrelated to that I need to figure out where we are on having nodepool build control plane images for us19:18
corvusclarkb: i think nodepool control plane images change ready to approve19:18
corvus+ "is"19:18
corvushttps://review.opendev.org/640044 just needs a +w19:19
clarkbkk so after my backlog of haproxy and project rename stuff I'll look at that next19:19
corvusoh i think https://review.opendev.org/#/c/640027/ also needs a +w; note wrong depends-on host19:19
corvusbut it's also ready to go19:20
clarkb#link https://review.opendev.org/640044 and https://review.opendev.org/#/c/640027/ nodepool built control plane images19:20
clarkbseems like we continue to have slow but steady progress on updating our config management tooling. Thank you everyone for that19:21
corvusonce we have gitea project creation in python, we can clean up the TODO in that base playbook refactor, and then start to have the gerrit-gitea-zuul playbook run out of the run_all loop.19:21
corvus(and then i think we'll be clear to start ansible/containerizing zuul)19:21
mordredI agree with corvus19:22
clarkbwhat is the dependency there? the puppet web that would otherwise need to be untangled?19:23
corvuscan't run that playbook in test mode because it'll take too long19:23
clarkbgotcha19:24
corvusnor could we run it in zuul in prod mode, for that matter, for the same reason19:24
clarkbright takes like 4 hours right now to do it from scratch?19:24
corvus$longtime at least19:24
clarkbThat is probably a good transition point into opendev stuff19:24
clarkb#topic OpenDev19:24
*** openstack changes topic to "OpenDev (Meeting topic: infra)"19:24
clarkbLast week we talked about general next step items19:24
mordredclarkb: when we update the gitea project creation into python - should we add support to projects.yaml for listing old names for projects? otherwise if we create things from scratch in a gitea we'll miss renames/redirects19:25
fungiwhich were mostly "let's finish cleaning up and polishing what's left from the last transition before we embark on new services (unless they're blocking cleanup from those we already switched)19:25
fungi"19:25
mordredfungi: ++19:26
clarkbya thats how I ended up digging into the docker job flakyness. It was making it hard for us to deploy updated gitea  :)19:26
clarkbmordred: ya I think we need to go back and record that somewhere probably19:26
fungii do expect we need some way of tracking what redirects we're maintaining19:26
corvusor do we just want to consider it precious data and back it up?19:27
clarkbcorvus: that would be database backups right?19:27
corvusyeah19:27
fungidoable, though that makes it harder to rebuild a gitea db from scratch if we want to be able to do that19:27
clarkbwe do know how to make those so that is an easy option too probably19:27
corvusi guess the main thing is, do we want the complexity of a system which creates projects only to rename them (possibly multiple times), or the complexity of additional precious data + backups19:27
*** hemna has quit IRC19:27
*** takamatsu has joined #openstack-meeting19:28
fungidepending on the schema, we could even just back up dumps of the redirects table or something like that19:28
mordredI mean - we have to encode renames into a yaml file anyway to do the renames - seems like if we just put that into a file that we cared about and tracked we'd be closer to being able to handle renames as part of normal process - assuming gerrit can ever grow rename ability19:28
corvusmordred: yeah, that's a good point19:28
clarkbmaybe we do both (backups as a backup)19:29
corvusmordred: i suspect, however, that we could end up in a situation where the "replay" won't work, depending on how we implement it19:29
mordredcorvus: indeed19:29
corvuseg, if multiple projects are renamed over each other, the sequence is important19:29
corvusso a naive implementation which just did one at a time could deadlock19:29
fungii'm fond of the belt-n-braces approach too if we want both19:29
mordredcorvus: that is an excellent point19:29
clarkbfungi: ya the more I think about it the more I think proper backups for the db is a good idea regardless of our normal operating procesure19:30
clarkb*procedure19:30
fungiit is true that any encoding would need to match gitea's logic wrt replacing old redirects19:30
*** pcaruana has quit IRC19:30
fungior sould need to at least encode the replay sequence19:30
corvusmordred: at the very least, i do agree we should go aheand and adjust the yaml schema to accomodate that, if it makes sense... i do think that a "previous name" field is probably the best way for us to eventually implement renames19:31
funginot sure we need backups from all 8 servers though. one copy is probably sufficient19:31
clarkbcorvus: ++19:31
corvusthere's probably another way to approach things as well, which would be to maintain a list of "current" previous names for a project19:32
corvuswhich would mean removing previous names from another project if the other project was renamed over the first19:32
corvusdoes that make sense?19:32
mordredcorvus: ++ I actually really like that19:32
clarkbmaybe step 0 is checking in the repos.yaml in ~root/ on bridge.o.o so that we hvae a copy of that floating about19:32
mordredcorvus: because it makes projects.yaml contain the current state of canonical+redirects expeted19:33
mordredexpected19:33
corvusit doesn't match how gitea works right now, so we'd have to think about how to implement it, but it would be the best sort of yaml state representation of the current status.19:33
corvusya that19:33
clarkbcorvus: I like that19:33
corvuslet's mull that over and see if we can come up with a way to do that19:33
mordred++19:33
clarkbsounds good. I can grab a copy of repos.yaml from the great renaming and throw it up on gerrit os that it is somewhere too.19:34
corvusclarkb: ++19:34
corvusand anything we do friday we should record with that19:34
clarkbya can push a followup with fridays repos.yaml content19:34
clarkbmaybe a new dir in system-cofnig called renames for now19:34
mordredyah19:35
clarkber project-config19:35
corvussystem-project19:35
fungimatching teh present mapping data could solve overwrites with a ci job to check for duplicates19:35
clarkbOk we've got 25 minutes left so let's keep moving. Anything else related to OpenDev or are we generally happy with the plan of make things more reliable and pick off other bits as it will help us with reliability19:35
clarkbcorvus: k19:36
clarkbcorvus: (I'm not actually going to make a new repo)19:36
corvus;)19:36
fungii got nothin urgent19:36
clarkb#topic Storyboard19:36
*** openstack changes topic to "Storyboard (Meeting topic: infra)"19:36
clarkbfungi: diablo_rojo_phon: Anything to bring up re storyboard?19:37
diablo_rojo_phonfungi did the swift migration19:37
fungiswift *test* migration19:37
diablo_rojo_phonTest migration19:37
diablo_rojo_phonYes19:37
timburkeYay! Thank you!19:37
fungiwhich surfaced an issue in that we've grown py3k-isms in the lp migration/import tooling, probably best solved by just rebuilding the current deployments on python 319:38
diablo_rojo_phonNot a problem with swift, more with our tools.19:38
corvussetting up the slow query log (or doing something equivalent) is very near the top of my TODO list now (it's below "file expense report")19:38
fungiright, that19:38
* fungi also needs to file expense report19:39
clarkboh hrm I never filed an expense report either19:39
fungiyeah... yeah19:39
corvusunexpectedly productive topic, apparently!19:39
fungieasy to forget19:39
corvusthus the list ;)19:39
fungipile of denver receipts on teh workbench behind me19:39
clarkbcorvus: ya that suddenly jumped up on my list by being added to it :)19:39
clarkbAlright lets move on19:40
clarkb#topic General Topics19:40
*** openstack changes topic to "General Topics (Meeting topic: infra)"19:40
clarkbFriday May 31, 2019 project renames (that is this friday)19:40
clarkb#link https://etherpad.openstack.org/p/project-renames-2019-05-31 Renaming process notes19:40
clarkbI started writing down a plan there19:41
clarkb#link https://review.opendev.org/#/q/topic:project-rename+status:open Changes to reflect project renames19:41
clarkbif you all can review the plan as well as the changes that that query pulls up it will be helpful19:41
fungiand we determined that last rename maintenance (other than the opendev transition) we bypassed zuul to merge the rename patches, so the test errors are expected?19:41
clarkbOne question I had was do we think we need to stack the project-rename changes so that we resolve any conflicts?19:41
clarkbfungi: that is what our etherpad notes say from a previous one yes19:41
fungisequencing them is probably not a terrible idea19:42
clarkbfungi: with notes about doing it before stopping gerrit so that they can replicate too19:42
corvusyeah, if we're force-merging them that's probably a good idea19:42
clarkbk I can take the todo to sequence those changes up19:42
fungi"back in the day" when renames were rampant, we used to sequence them all as a matter of course19:42
fungi(after running into more than one which needed a rebase after another merged, slowing the maintenance)19:42
clarkbThe other item was what time do we want to start? I can send an email out today with a notice of gerrit downtime to our gerrit user mailing lists19:43
clarkbI'm good with anything after 9am local time and can probably do an early start at 8am if people prefer earlier. That is 1500UTC19:43
fungii'm good with pretty much any time19:43
clarkb8am is 1500UTC19:43
fungii have the benefit of living in your future, after all19:43
clarkbmordred: and I think you said you wouldn't be around right? So probably fungi, corvus, myself and maybe frickler/ajaeger?19:44
mordredyah - I'm out thurs/fri19:44
clarkbcorvus: ^ did you have an opinion on start time?19:44
corvusi can be around; earlier is better19:45
clarkbok lets say 1500UTC if earlier is better19:45
fungiwfm, updated my reminder19:45
clarkbI think that covers my todo items for project renaming19:45
clarkbNext up is server upgrades19:46
clarkb#link https://etherpad.openstack.org/p/201808-infra-server-upgrades-and-cleanup19:46
clarkbmordred: did you have an update on status.o.o upgrades?19:46
clarkband fungi did you manage to get that wiki puppet change in?19:46
mordredI put up two cleanup patches19:46
clarkbmordred: can haz links?19:46
mordredhttps://review.opendev.org/#/c/661385/ and19:46
mordredhttps://review.opendev.org/#/c/661399/119:47
clarkb#link https://review.opendev.org/#/c/661385/ https://review.opendev.org/#/c/661399/ status.o.o server upgrade related cleanup changes19:47
fungiyeah, last wiki-dev puppet change merged, i just haven't had time to check back in on it19:47
clarkbcool so progress here too.19:47
corvusone of those should get a review from ttx19:47
mordredI think my next step is just going to be spinning up a new host with the existing stuff on xenial and seeing if it works fine - the only thing htat's really potentially yuck is the libjq-jquery package - but it's probably fine19:47
clarkbmordred: and we can probably grab that from upstream if necessary pretty easily19:48
mordred(potentially yuck because xenial has a different version of jquery - not because there's anything wrong with the package)19:48
mordredclarkb: yah19:48
clarkbianw: fungi had mentioned that he thought you had said you would do static? is that the case or do we need to find someone to work on that?19:48
fungilooks like puppet hasn't been running on wiki-dev01.opendev.org for a little while, so need to dig into why19:48
mordredthere are many things I'd like to change - but trying my best to avoid unnecessary nerd-sniping19:48
corvusmordred: zuul status used to use jquery, does anything current?19:48
clarkbcorvus: https://review.opendev.org/#/c/661385/ that one?19:48
ianwumm, well i haven't been looking at it ... does it sound appealing to anyone else?19:48
fungier, i mean on wiki-dev01.openstack.org19:48
corvusclarkb: yes19:48
mordredclarkb: yes - elastic-recheck does19:49
mordredgah19:49
mordredcorvus: ^^19:49
corvusmordred: thx19:49
clarkbianw: looking at notes more closely I think that you put your name on notes about switching cinder volumes around which may be where that idea came from19:49
fungiianw: i probably just misremembered, sorry19:49
clarkbianw: doesn't have to be you, just wanted to check if it already was :)19:49
fungiclarkb: that may have been what stuck in my mind, yes19:49
mordredcorvus: I think in the future we could improve that by updating the elastic-recheck web stuff to build with webpack in the gate so we just deploy a tarball19:49
clarkbin any case progress being made on a few fronts here. Thank you to everyone that has helped with this.19:50
corvusmordred: yeah, or build a container or something19:50
mordredyeah19:50
mordredbasically - we can make some improvements - but for now just gonna upgrade it as it is and move on19:51
clarkbNext up is the mirror replacements which will gives us tls on the mirrors which should hopefully eventually lead to more relaible docker jobs19:51
clarkbmordred: +119:51
clarkblink https://review.opendev.org/#/c/661187/ switch rax dfw mirror to new opendev mirror.19:51
clarkb#link https://review.opendev.org/#/c/661187/ switch rax dfw mirror to new opendev mirror.19:51
clarkbis the site variables file expected to be treated as jinja as in that change?19:52
clarkbmordred: corvus ^19:52
clarkbthat was my only concern with that approach. Otherwise I think we can go ahead and do that19:52
corvusi think it's "-e" included and the point of evaluation is when ansible loads that, so i think that should work.19:53
clarkbawesome I should stamp a +2 on it and have ianw approve when ready to watch it?19:53
corvusalso i don't think it will require a restart19:53
corvusi went ahead and +w it19:53
*** ralonsoh has quit IRC19:53
ianwdoes it need a restart?19:53
clarkbcorvus: cool19:54
ianwoh, ok, if corvus says no ++ :)19:54
clarkbAnd that takes us to the end of our regularly scheduled agenda19:54
clarkb#topic Open Discussion19:54
*** openstack changes topic to "Open Discussion (Meeting topic: infra)"19:54
clarkbAnything else we may have missed or should otherwise bring up in our last 5 minutes or so?19:54
ianwcorvus (mostly, but anyone) if we could loop back on the dns jobs in https://review.opendev.org/#/q/status:open+topic:check-zone that would be good to just get out of my queue19:55
ianwand stop me messing up any future mirror bringups :)19:55
corvusianw: will do19:55
corvusoh that's easy19:56
corvussomeone should +3 those now :)19:56
fungii already did19:57
clarkbAn early heads up: Mid june I'll be in central oregon digging up rocks so probably won't work much between June 17-20. This happens to be the week our certs expire so I'll work on replacing certs the week prior and checked that fungi will be around the 17-20 to finish up any stragglers19:57
fungiyep, and then i disappear the next week19:57
corvusclarkb: like digging up with a backhoe, or digging up with a paintbrush and dental pick?19:57
ianwand mordred, or anyone ansibly -- https://github.com/ansible/ansible/pull/57038 i noticed from our devel job.  not really sure if it needs another +2 or whatever the ansible version of that is19:57
clarkbcorvus: shovel and pick/hoe. We are going to try our luck at finding geodes/thunder eggs19:58
corvusooh19:58
mordredI will also be out june 17-20 - as well as most of the week after - but then I should be here all of july19:58
corvusi don't have any plans, but maybe i should make some :)19:59
clarkbha19:59
clarkbAnd that takes us to our hour time limit.19:59
clarkbThanks everyone!19:59
clarkb#endmeeting19:59
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"19:59
openstackMeeting ended Tue May 28 19:59:50 2019 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)19:59
openstackMinutes:        http://eavesdrop.openstack.org/meetings/infra/2019/infra.2019-05-28-19.01.html19:59
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/infra/2019/infra.2019-05-28-19.01.txt19:59
openstackLog:            http://eavesdrop.openstack.org/meetings/infra/2019/infra.2019-05-28-19.01.log.html19:59
mordredcorvus: I'll be moving stuff into storage from a truck the first part of that week, then diving in various springs in florida - in case those are the types of activities you just want to copy20:00
corvusmordred: hrm.... i *do* like putting things in storage.20:00
corvusespecially the part where i label all the boxes20:01
mordredcorvus: what about the part before that were you devise the organizational system you'll use to label the boxes20:01
*** hemna has joined #openstack-meeting20:01
*** takamatsu has quit IRC20:02
*** mriedem has quit IRC20:07
*** wwriverrat has joined #openstack-meeting20:14
*** trandles has joined #openstack-meeting20:14
*** wwriverrat has quit IRC20:19
*** dasp has joined #openstack-meeting20:20
*** wwriverrat has joined #openstack-meeting20:29
*** wwriverrat has quit IRC20:30
*** wwriverrat has joined #openstack-meeting20:31
*** wwriverrat has quit IRC20:32
*** Shrews has left #openstack-meeting20:33
*** wwriverrat has joined #openstack-meeting20:35
*** wwriverrat has quit IRC20:36
*** dasp has quit IRC20:38
*** dasp has joined #openstack-meeting20:38
*** njohnston is now known as njohnston|off20:39
*** takamatsu has joined #openstack-meeting20:42
*** boxiang has quit IRC20:48
*** slaweq has quit IRC20:49
*** mriedem has joined #openstack-meeting20:51
*** baojg has joined #openstack-meeting20:56
*** e0ne has joined #openstack-meeting20:58
*** martial has joined #openstack-meeting21:00
martial#start-meeting scientific-sig21:01
martial#startmeeting scientific-sig21:01
openstackMeeting started Tue May 28 21:01:14 2019 UTC and is due to finish in 60 minutes.  The chair is martial. Information about MeetBot at http://wiki.debian.org/MeetBot.21:01
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.21:01
*** openstack changes topic to " (Meeting topic: scientific-sig)"21:01
openstackThe meeting name has been set to 'scientific_sig'21:01
*** baojg has quit IRC21:01
martialGood Day everybody21:01
martiala little meeting without an agenda today21:01
martiala few of our regulars are either off (Memorial day yesterday in the USA) or participated at CERN21:02
martial(their OpenStack days)21:02
martialas such, a micro agenda21:02
martialhere is hoping we get a recap of the CERN presentations next meeting21:03
martiala couple things to share with people:21:06
martial1) ISC is coming soon (Frankfurt, Germany) Jun 16-20. I will be there for the High Performance Container Workshop (HPCW)21:06
martial2) Call for Presentations for the Shanghai summit is open21:06
martial#link https://www.isc-hpc.com/21:07
martial#link https://cfp.openstack.org/21:07
martialFinally, more as an FYI: for people that want more Scientific SIG conversations, we recently created a Slack for the SIG21:08
martial#link https://join.slack.com/t/os-scientific-sig/shared_invite/enQtNjExNTM0OTY1NjY3LThiZTIzOTljMzM0ODIxMjJjMDhlZGIyOGY3YTE4YmYwODQ0NGVlYTIxZjBmMGIzZmZmZjJmMmNiZjZkMGRmNDg21:09
martialGiven that I have ran a monologue for the past few minutes, it will be a short meeting :)21:09
*** dtrainor has quit IRC21:11
*** slaweq has joined #openstack-meeting21:11
*** janders has joined #openstack-meeting21:12
*** e0ne has quit IRC21:12
*** wwriverrat has joined #openstack-meeting21:13
martialWell, thank you for your time; adjourning the short meeting21:16
martial#endmeeting21:16
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"21:16
openstackMeeting ended Tue May 28 21:16:48 2019 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)21:16
openstackMinutes:        http://eavesdrop.openstack.org/meetings/scientific_sig/2019/scientific_sig.2019-05-28-21.01.html21:16
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/scientific_sig/2019/scientific_sig.2019-05-28-21.01.txt21:16
openstackLog:            http://eavesdrop.openstack.org/meetings/scientific_sig/2019/scientific_sig.2019-05-28-21.01.log.html21:16
*** wwriverrat has quit IRC21:18
*** dtrainor has joined #openstack-meeting21:21
*** slaweq has quit IRC21:24
*** raildo has quit IRC21:25
*** janders has quit IRC21:36
*** yamamoto has joined #openstack-meeting21:45
*** yamamoto has quit IRC21:50
*** bbowen has quit IRC21:52
*** bbowen has joined #openstack-meeting21:53
*** bbowen has quit IRC22:02
*** rcernin has quit IRC22:06
*** whoami-rajat has quit IRC22:11
*** slaweq has joined #openstack-meeting22:16
*** absubram has joined #openstack-meeting22:21
*** slaweq has quit IRC22:24
*** mriedem has quit IRC22:29
*** wwriverrat has joined #openstack-meeting22:29
*** baojg has joined #openstack-meeting22:57
*** baojg has quit IRC23:01
*** slaweq has joined #openstack-meeting23:11
*** ykatabam has joined #openstack-meeting23:12
*** _erlon_ has quit IRC23:16
*** carloss has quit IRC23:19
*** lbragstad has quit IRC23:19
*** rcernin has joined #openstack-meeting23:21
*** absubram has quit IRC23:22
*** lbragstad has joined #openstack-meeting23:23
*** slaweq has quit IRC23:24
*** lseki has quit IRC23:31
*** macza has quit IRC23:36
*** jbadiapa has quit IRC23:37
*** jbadiapa has joined #openstack-meeting23:38
*** gyee has quit IRC23:40
*** yamamoto has joined #openstack-meeting23:41
*** takamatsu has quit IRC23:56
*** ykatabam has quit IRC23:58

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!