Tuesday, 2018-03-06

*** rossella_s has joined #openstack-meeting00:00
*** Patifa has quit IRC00:01
*** esberglu has quit IRC00:02
*** gouthamr has quit IRC00:02
*** Patifa has joined #openstack-meeting00:04
*** aeng has quit IRC00:05
*** aeng has joined #openstack-meeting00:05
*** Zames has joined #openstack-meeting00:15
*** esberglu has joined #openstack-meeting00:17
*** Zames has quit IRC00:20
*** HeOS has quit IRC00:21
*** chyka_ has quit IRC00:25
*** chyka has joined #openstack-meeting00:25
*** Zames has joined #openstack-meeting00:25
*** slaweq has joined #openstack-meeting00:25
*** dmacpher__ has joined #openstack-meeting00:26
*** dmacpher_ has quit IRC00:29
*** slaweq has quit IRC00:30
*** fnaval has joined #openstack-meeting00:30
*** Zames has quit IRC00:30
*** amodi has joined #openstack-meeting00:30
*** Zames has joined #openstack-meeting00:30
*** Zames has quit IRC00:33
*** andreas_s has joined #openstack-meeting00:37
*** tovin07_ has joined #openstack-meeting00:37
*** rossella_s has quit IRC00:39
*** yamamoto has quit IRC00:40
*** andreas_s has quit IRC00:41
*** yamamoto has joined #openstack-meeting00:42
*** rossella_s has joined #openstack-meeting00:43
*** TxGirlGeek has joined #openstack-meeting00:45
*** chyka has quit IRC00:47
*** fnaval has quit IRC00:51
*** fnaval has joined #openstack-meeting00:51
*** jamesmcarthur has joined #openstack-meeting00:52
*** vishwana_ has quit IRC00:53
*** oanson has quit IRC00:56
*** jamesmcarthur has quit IRC00:56
*** liyi has joined #openstack-meeting01:00
*** ykatabam1 has quit IRC01:00
*** ykatabam has joined #openstack-meeting01:01
*** Patifa has quit IRC01:03
*** oanson has joined #openstack-meeting01:03
*** Patifa has joined #openstack-meeting01:13
*** Patifa has quit IRC01:14
*** pzchen has joined #openstack-meeting01:17
*** wanghao_ has quit IRC01:17
*** wanghao has joined #openstack-meeting01:17
*** wanghao has quit IRC01:17
*** wanghao has joined #openstack-meeting01:19
*** yamahata has joined #openstack-meeting01:19
*** kevzha01 has joined #openstack-meeting01:20
*** dangtrinhnt has joined #openstack-meeting01:20
*** yangyapeng has joined #openstack-meeting01:21
*** kevzha01 is now known as kevinz01:22
*** ekcs has joined #openstack-meeting01:28
*** Tom-Tom has quit IRC01:33
*** Tom-Tom has joined #openstack-meeting01:34
*** Tom-Tom has quit IRC01:39
*** amodi has quit IRC01:39
*** hyunsikyang has quit IRC01:48
*** amodi has joined #openstack-meeting01:51
*** hyunsikyang has joined #openstack-meeting01:51
*** Zames has joined #openstack-meeting01:52
*** Tom-Tom has joined #openstack-meeting01:53
*** Zames has quit IRC01:55
*** TxGirlGeek has quit IRC01:59
*** mriedem has quit IRC02:03
*** slaweq has joined #openstack-meeting02:04
*** slaweq_ has joined #openstack-meeting02:05
*** slaweq has quit IRC02:08
*** slaweq_ has quit IRC02:10
*** gongysh has joined #openstack-meeting02:10
*** amodi has quit IRC02:12
*** chyka has joined #openstack-meeting02:13
*** markstur has quit IRC02:14
*** slaweq has joined #openstack-meeting02:17
*** chyka has quit IRC02:18
*** slaweq has quit IRC02:22
*** radeks has joined #openstack-meeting02:26
*** bnemec has joined #openstack-meeting02:29
*** slaweq has joined #openstack-meeting02:30
*** kiennt26 has quit IRC02:32
*** radeks has quit IRC02:33
*** slaweq has quit IRC02:34
*** gongysh has quit IRC02:35
*** slaweq has joined #openstack-meeting02:42
*** diman has joined #openstack-meeting02:43
*** bnemec has quit IRC02:45
*** hongbin has joined #openstack-meeting02:46
*** slaweq has quit IRC02:47
*** diman has quit IRC02:47
*** yamamoto has quit IRC02:49
*** yamamoto has joined #openstack-meeting02:49
*** slaweq has joined #openstack-meeting02:55
*** salv-orl_ has joined #openstack-meeting02:58
*** slaweq has quit IRC02:59
*** jamesmcarthur has joined #openstack-meeting03:00
*** salv-orlando has quit IRC03:01
*** bnemec has joined #openstack-meeting03:02
*** ekcs has quit IRC03:03
*** jamesmcarthur has quit IRC03:04
*** zhurong has joined #openstack-meeting03:05
*** bobh has joined #openstack-meeting03:07
*** slaweq has joined #openstack-meeting03:07
*** slaweq has quit IRC03:12
*** bkopilov has quit IRC03:12
*** fnaval has quit IRC03:17
*** harlowja has quit IRC03:18
*** VW has joined #openstack-meeting03:18
*** slaweq has joined #openstack-meeting03:19
*** bobh has quit IRC03:23
*** slaweq has quit IRC03:24
*** bnemec has quit IRC03:26
*** esberglu has quit IRC03:26
*** chyka has joined #openstack-meeting03:28
*** slaweq has joined #openstack-meeting03:32
*** chyka has quit IRC03:33
*** masuberu has quit IRC03:34
*** bobh has joined #openstack-meeting03:34
*** slaweq has quit IRC03:36
*** dmacpher__ has quit IRC03:40
*** slaweq has joined #openstack-meeting03:44
*** rbudden has quit IRC03:47
*** lhx_ has joined #openstack-meeting03:48
*** slaweq has quit IRC03:49
*** psachin has joined #openstack-meeting03:50
*** jdurgin has quit IRC03:50
*** links has joined #openstack-meeting03:51
*** tpatil has joined #openstack-meeting03:56
*** slaweq has joined #openstack-meeting03:57
tpatil /msg tpatil identify03:57
*** tpatil has quit IRC03:58
*** tpatil has joined #openstack-meeting03:59
*** sagara has joined #openstack-meeting03:59
samP#startmeeting masakari04:00
openstackMeeting started Tue Mar  6 04:00:03 2018 UTC and is due to finish in 60 minutes.  The chair is samP. Information about MeetBot at http://wiki.debian.org/MeetBot.04:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.04:00
*** openstack changes topic to " (Meeting topic: masakari)"04:00
openstackThe meeting name has been set to 'masakari'04:00
*** esberglu has joined #openstack-meeting04:00
samPHi all for masakari04:00
tpatilHi04:00
sagaraHi04:00
samPGlad you all came back safely04:01
samPfrom PTG04:01
*** rkmrHonjo has joined #openstack-meeting04:01
*** slaweq has quit IRC04:01
samP#topic High priority items04:01
*** openstack changes topic to "High priority items (Meeting topic: masakari)"04:01
samPAbout Release04:01
samPall 3 projects has been released04:02
sagaraGreat04:02
samPcurrently, python masakari client does not have stable/queens branch04:02
tpatilOnce the masakari-monitors patch is merged, you will need to release it's new version, correct?04:03
samPThe reason is in comment of following patch04:03
*** gyee has quit IRC04:03
samP#link https://review.openstack.org/#/c/547502/04:03
samPtpatil: correct04:04
tpatil#link : https://review.openstack.org/#/c/546492/1004:04
tpatilsamP: Ok, Is there any problem in cutting stable/queens branch for python-masakariclient?04:04
samPtpatil: waited for release. Currently no blockers. I will create it today04:06
tpatilsamP: OK04:06
*** andreas_s has joined #openstack-meeting04:06
tpatilSo only masakari-monitors patch is pending for review against Queens release04:08
samPAnd fix masakari command for python-masakariclient04:09
*** slaweq has joined #openstack-meeting04:09
samPrkmrHonjo: ^^ correct?04:09
*** bobh has quit IRC04:10
tpatilsamP: yes, I remember, that create_connection method issue04:10
samPtpatil: yes.04:10
rkmrHonjosamP: Correct.04:10
*** dmacpher has joined #openstack-meeting04:10
rkmrHonjoThis patch fixes masakari command. #link https://review.openstack.org/#/c/547879/04:11
samPI think we need to fix those 2 problems and release a new version for both pyclient and monitors04:11
*** andreas_s has quit IRC04:11
samPrkmrHonjo: correct04:11
samPPlease do not merge this patch before create stable/queens for python-masakariclient04:13
tpatilsamP: point noted04:13
*** slaweq has quit IRC04:14
samPI will submit required patches for create stable/queens today.04:14
rkmrHonjosamP: I got it.04:14
samPFor openstack/masakari and masakari-monitors, now we can work master for R04:15
samPFor openstack/python-masakariclient, please wait for stable/queens branch04:16
samP#topic Rocky work items04:16
*** openstack changes topic to "Rocky work items (Meeting topic: masakari)"04:16
samPI would like to take some time to discuss about Rocky work items.04:17
samPFirst, I will create a etherpad for Rocky, and fill it with existing items04:18
samPPlease propose any new items to etherpad before next IRC meeting on 11/3/201804:19
tpatilsamP: Sure04:19
rkmrHonjosamP: OK.04:21
*** esberglu has quit IRC04:21
tpatilIn PTG, I met Greg, Abhishek and Adam along with NTT folks04:21
samP#link https://etherpad.openstack.org/p/masakari-rocky-work-items04:21
samP^^ here is the etherpad04:21
*** slaweq has joined #openstack-meeting04:22
samPtpatil: thanks04:22
tpatilGreg is looking forward to get his team's patch merged in Rocky about Intrusive monitoring04:22
samPtpatil: any update from them. Really sorry that I wan not able to be there04:22
tpatilsamP: Not yet04:22
samPtpatil: got it.04:22
tpatilAlso, if someone is available we can start work on nova-host-alerter04:23
*** masber has joined #openstack-meeting04:23
samPrkmrHonjo: Any progress on your side of this work?04:24
samPrkmrHonjo: nova-host-alerter04:24
tpatilI think you are familiar with this openstack resource agents and this new agent will be part of it04:24
samPtpatil: yes. if I remember correctly, rkmrHonjo was stared to work on this.04:25
tpatil#link : https://aspiers.github.io/openstack-day-israel-2017-compute-ha/#/no-fence_evacuate04:26
rkmrHonjoPlease wait..04:26
*** slaweq has quit IRC04:26
*** nsingh has joined #openstack-meeting04:26
*** VW has quit IRC04:26
*** VW has joined #openstack-meeting04:27
samPrkmrHonjo's team was creating a PoC for this. OCF-RA and masakari plugin04:27
*** Tom-Tom has quit IRC04:28
rkmrHonjoExcuse me, does nova-host-alerter means monitor implemented as RA?04:28
*** Tom-Tom has joined #openstack-meeting04:28
tpatilrkmrHonjo: That's correct04:28
samPrkmrHonjo: monitor implement as RA04:28
rkmrHonjoAh, OK. I'm preparing to submit PoC. Sorry for late. I'll submit it in next week.04:29
samPrkmrHonjo: thanks.04:30
samPI will put this item to Rocky work items.04:30
samPThis is bit heavy task. rkmrHonjo: might better to split the tasks.04:31
samPLet's discuss this on Rocky work item discussion.04:31
*** bkopilov has joined #openstack-meeting04:31
rkmrHonjook.04:32
*** VW has quit IRC04:32
samPFor Rocky work item discussion, which method you prefer, IRC or VoIP?04:32
samPVoIP also have chat, VoIP=Zoom04:32
tpatilsamP: Once nova-host-alerter is available, what is the plan for masakari-monitor?04:33
*** Tom-Tom has quit IRC04:33
samPtpatil: we have 2 options,04:34
*** slaweq has joined #openstack-meeting04:34
samP(1) retire hostmonitor part from masakari-monitors in favour of nova-host-alerter04:34
samP(2) keep both, since current masakari-monitors doing it different manner04:35
samPIn future, process monitor could also implement as same architecture with nova-host-alerter04:36
samPHowever, not the instance monitor.04:36
*** epico has joined #openstack-meeting04:37
*** sridharg has joined #openstack-meeting04:37
tpatilsamP: Ok, process monitor is doing nothing as of now. Do we really want it?04:37
samPsince we have new requirements coming in for instance monitor, we do have to maintain it04:38
*** hongbin has quit IRC04:38
tpatilsamP: Ok I got it.04:38
*** slaweq has quit IRC04:39
samPtpatil: about process monitor, agree with current situation. However, it is still valid as a reference driver for process monitoring04:39
samPIn VMHA04:39
samPIn VMHA usecase, process monitoring is one of the requirement, along with host and instance monitoring.04:40
tpatilsamP: Let's come up with some use case and take some recovery action against process notifcations other than simply logging it04:41
samPOur, current process monitor do not do much at the time. But in future, if we can cutormize the recovery action, then we might able to use it in much efficient way04:41
samPtpatil: totally agree.04:42
tpatilsamP: Recovery customization is next big item for Rocky, we have completed the POC. We can talk about it in next meeting04:42
samPtpatil: great work. thanks04:43
rkmrHonjotpatil: great.04:43
samPtpatil: let's look into details in next meeting04:43
tpatilsamP: Sure04:44
rkmrHonjoSorry, I leave this meeting at 13:55.04:44
rkmrHonjo13:55 JST04:44
samPAbout, "Rocky work items" I will send a mail for details about the discussion.04:45
samPrkmrHonjo: NP04:45
samP#topic Masakari Project mascot04:46
*** openstack changes topic to "Masakari Project mascot (Meeting topic: masakari)"04:46
samPAny new ideas?04:46
*** slaweq has joined #openstack-meeting04:46
samPPlease propose if you have any ideas. I would like to decide some thing in next meeting..:)04:47
rkmrHonjoOK. When is the time limit_04:48
rkmrHonjos/_/?\04:48
sagarasamP: OK, I'm also still considering04:48
*** zhurong has quit IRC04:48
samPrkmrHonjo: before 3/1104:48
samPrkmrHonjo: next meeting04:48
rkmrHonjooh. I got it.04:49
samP#topic AOB04:49
*** openstack changes topic to "AOB (Meeting topic: masakari)"04:49
rkmrHonjopy35 of masakari was failed by date time format error. #link04:50
rkmrHonjo#link https://review.openstack.org/#/c/545545/04:50
samPSkipped the Bug/Patch part, cause we have already discussed above about important issues.04:50
samPPlease share, other than that..04:50
rkmrHonjoOh, sorry.04:51
*** slaweq has quit IRC04:51
samPrkmrHonjo: sorry, not about one posted04:51
*** anilvenkata has joined #openstack-meeting04:52
samPrkmrHonjo: your issue is valid04:52
rkmrHonjosamP: ok. Does anyone analyze this py35 error? If "no", I analyze it.04:53
tpatilrkmrHonjo: Please report it as a bug and someone will take up this issue04:53
rkmrHonjotpatil: ok.04:53
samPtpatil: rkmrHonjo: thanks/04:54
*** rkmrHonjo has quit IRC04:55
samPseems like datetime format error,04:55
samPAnyway, will follow the Bug report.04:55
samPLast 5 mins, any other issues to discuss?04:56
tpatilmaybe some Oslo_utils related changes is causing this problem04:56
samPBTW, NTT system-fault-ci masakari-integration-ci is down for maintenance04:57
samPI will be at OpenStack Ops meetup on 3/7-804:57
samP#link https://wiki.openstack.org/wiki/Operations/Meetups/TYO-ops-meetup04:58
samPalmost time..04:58
samPThank you all...04:58
*** slaweq has joined #openstack-meeting04:59
samPPlease use #openstack-masakari @freenode IRC or openstack-dev ML with [masakari] for further discussion04:59
sagaraThank you, bye04:59
samPThanks you again... bye04:59
*** chkumar246 is now known as chandankumar04:59
samP#endmeeting04:59
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"04:59
openstackMeeting ended Tue Mar  6 04:59:39 2018 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)04:59
openstackMinutes:        http://eavesdrop.openstack.org/meetings/masakari/2018/masakari.2018-03-06-04.00.html04:59
tpatilBye04:59
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/masakari/2018/masakari.2018-03-06-04.00.txt04:59
openstackLog:            http://eavesdrop.openstack.org/meetings/masakari/2018/masakari.2018-03-06-04.00.log.html04:59
*** tpatil has quit IRC04:59
*** bnemec has joined #openstack-meeting05:00
*** slaweq has quit IRC05:03
*** slaweq has joined #openstack-meeting05:11
*** harlowja has joined #openstack-meeting05:12
*** chyka has joined #openstack-meeting05:15
*** slaweq has quit IRC05:16
*** sagara has quit IRC05:18
*** chyka has quit IRC05:20
*** slaweq has joined #openstack-meeting05:24
*** markvoelker has quit IRC05:27
*** slaweq has quit IRC05:29
*** Tom-Tom has joined #openstack-meeting05:30
*** bnemec has quit IRC05:32
*** bnemec has joined #openstack-meeting05:35
*** slaweq has joined #openstack-meeting05:36
*** Zames has joined #openstack-meeting05:36
*** sidx64 has joined #openstack-meeting05:37
*** sidx64 has quit IRC05:38
*** sidx64 has joined #openstack-meeting05:40
*** slaweq has quit IRC05:40
*** Zames has quit IRC05:41
*** bnemec has quit IRC05:41
*** ykatabam has quit IRC05:48
*** slaweq has joined #openstack-meeting05:48
*** slaweq has quit IRC05:53
*** haint has quit IRC05:57
*** sidx64 has quit IRC05:57
*** slaweq has joined #openstack-meeting06:01
*** aeng has quit IRC06:03
*** sidx64 has joined #openstack-meeting06:03
*** slaweq has quit IRC06:05
*** ykatabam has joined #openstack-meeting06:07
*** sidx64 has quit IRC06:07
*** haint has joined #openstack-meeting06:07
*** sidx64 has joined #openstack-meeting06:08
*** armax has quit IRC06:13
*** slaweq has joined #openstack-meeting06:13
*** slaweq has quit IRC06:18
*** marios has joined #openstack-meeting06:22
*** e0ne has joined #openstack-meeting06:25
*** slaweq has joined #openstack-meeting06:26
*** markvoelker has joined #openstack-meeting06:27
*** harlowja has quit IRC06:29
*** slaweq has quit IRC06:30
*** chandankumar has quit IRC06:35
*** chandankumar has joined #openstack-meeting06:36
*** sidx64_ has joined #openstack-meeting06:38
*** slaweq has joined #openstack-meeting06:38
*** julim_ has joined #openstack-meeting06:40
*** sidx64 has quit IRC06:40
*** radeks has joined #openstack-meeting06:41
*** julim has quit IRC06:41
*** ykatabam has quit IRC06:41
*** sidx64 has joined #openstack-meeting06:43
*** slaweq has quit IRC06:43
*** sidx64_ has quit IRC06:44
*** andreas_s has joined #openstack-meeting06:48
*** andreas_s has quit IRC06:53
*** HeOS has joined #openstack-meeting06:55
*** ykatabam has joined #openstack-meeting06:58
*** Tom-Tom_ has joined #openstack-meeting07:01
*** rbartal has joined #openstack-meeting07:02
*** Tom-Tom has quit IRC07:03
*** sidx64_ has joined #openstack-meeting07:12
*** sidx64 has quit IRC07:14
*** rcernin has quit IRC07:14
*** sidx64 has joined #openstack-meeting07:15
*** sidx64_ has quit IRC07:17
*** e0ne has quit IRC07:17
*** janki has joined #openstack-meeting07:19
*** andreas_s has joined #openstack-meeting07:25
*** e0ne has joined #openstack-meeting07:27
*** ttsiouts has quit IRC07:28
*** e0ne has quit IRC07:28
*** ttsiouts has joined #openstack-meeting07:28
*** felipemonteiro has joined #openstack-meeting07:29
*** imcsk8_ has quit IRC07:32
*** imcsk8 has joined #openstack-meeting07:33
*** slaweq has joined #openstack-meeting07:40
*** tovin07_ has quit IRC07:48
*** longkb1 has joined #openstack-meeting07:52
*** cloudrancher has joined #openstack-meeting07:53
*** longkb has quit IRC07:54
*** sidx64 has quit IRC08:02
*** trinaths has joined #openstack-meeting08:22
*** slaweq_ has joined #openstack-meeting08:31
*** pzchen has quit IRC08:32
*** tesseract has joined #openstack-meeting08:32
*** slaweq_ has quit IRC08:36
*** pzchen has joined #openstack-meeting08:36
*** felipemonteiro has quit IRC08:39
*** oanson has quit IRC08:41
*** Tom-Tom_ has quit IRC08:42
*** Tom-Tom has joined #openstack-meeting08:44
*** pcaruana has joined #openstack-meeting08:44
*** electrofelix has joined #openstack-meeting08:45
*** zhurong has joined #openstack-meeting08:45
*** alexchadin has joined #openstack-meeting08:46
*** Tom-Tom has quit IRC08:48
*** alexchadin has quit IRC08:49
*** oanson has joined #openstack-meeting08:50
*** tssurya has joined #openstack-meeting08:50
*** alexchadin has joined #openstack-meeting08:50
*** chyka has joined #openstack-meeting08:51
*** Tom-Tom has joined #openstack-meeting08:51
*** psachin has quit IRC08:52
*** pcaruana has quit IRC08:54
*** chyka has quit IRC08:56
*** phil has joined #openstack-meeting09:00
*** e0ne has joined #openstack-meeting09:00
*** phil is now known as Guest6123109:00
*** janki has quit IRC09:04
*** cloudrancher has quit IRC09:06
*** cloudrancher has joined #openstack-meeting09:07
*** masber has quit IRC09:09
*** psachin has joined #openstack-meeting09:09
*** kopecmartin has joined #openstack-meeting09:11
*** cloudrancher has quit IRC09:11
*** cloudran_ has joined #openstack-meeting09:11
*** sidx64 has joined #openstack-meeting09:19
*** Zames has joined #openstack-meeting09:20
*** cloudran_ has quit IRC09:21
*** marios has quit IRC09:23
*** Zames has quit IRC09:23
*** marios has joined #openstack-meeting09:23
*** Zames has joined #openstack-meeting09:29
*** liyi has quit IRC09:30
*** wanghao has quit IRC09:31
*** wanghao has joined #openstack-meeting09:32
*** wanghao has quit IRC09:32
*** slaweq_ has joined #openstack-meeting09:32
*** wanghao has joined #openstack-meeting09:32
*** wanghao has quit IRC09:33
*** wanghao has joined #openstack-meeting09:33
*** wanghao has quit IRC09:34
*** wanghao has joined #openstack-meeting09:34
*** wanghao has quit IRC09:34
*** Tom-Tom has quit IRC09:35
*** wanghao has joined #openstack-meeting09:35
*** wanghao has quit IRC09:35
*** wanghao has joined #openstack-meeting09:35
*** wanghao has quit IRC09:36
*** chenying has joined #openstack-meeting09:36
*** wanghao has joined #openstack-meeting09:36
*** wanghao has quit IRC09:37
*** slaweq_ has quit IRC09:37
*** wanghao has joined #openstack-meeting09:37
*** wanghao has quit IRC09:37
*** wanghao has joined #openstack-meeting09:38
*** sidx64 has quit IRC09:40
chenying#startmeeting karbor09:42
openstackMeeting started Tue Mar  6 09:42:44 2018 UTC and is due to finish in 60 minutes.  The chair is chenying. Information about MeetBot at http://wiki.debian.org/MeetBot.09:42
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.09:42
*** openstack changes topic to " (Meeting topic: karbor)"09:42
openstackThe meeting name has been set to 'karbor'09:42
chenyinghi jiaopengju09:42
*** slaweq_ has joined #openstack-meeting09:43
jiaopengjuhi chenying09:43
*** diman has joined #openstack-meeting09:43
chenyingNow the queen of karbor has been released. Now we can start the rocky version of the karbor project.09:44
*** sidx64 has joined #openstack-meeting09:44
*** dims has quit IRC09:45
chenyingI have created a link for rocky karbor project.09:45
chenyinghttps://ethercalc.openstack.org/karbor-rocky09:45
jiaopengjuThat’s good09:45
*** rcernin has joined #openstack-meeting09:45
chenyingif we have any task of feature which want to introduce to karbor, we can add it to this link.09:46
chenyingDo you have any idea about the rocky version of karbor?09:46
jiaopengjuok, imo, multi node trigger schedule should be enhanced09:47
*** slaweq_ has quit IRC09:48
*** diman has quit IRC09:48
chenyingYes09:48
chenyingthe goal for rocky also need be considered.09:48
*** dims has joined #openstack-meeting09:49
jiaopengjuyes09:49
chenyingWe can add it to this link.09:49
*** d0ugal has quit IRC09:50
chenyingDo you have any other questions?09:51
jiaopengjuno09:52
*** Tom-Tom has joined #openstack-meeting09:52
jiaopengjuI will add my ideas to thie rocky link in few days09:52
chenyingGood idea.09:52
jiaopengjuthie/the09:52
*** slaweq_ has joined #openstack-meeting09:53
jiaopengjuWe should attract more contributors :)09:53
chenyingyes. We can add it to the link.09:54
*** Zames has quit IRC09:56
chenyingSo we can end this meeting. If you have any question, we can discuss it in karbor's irc meeting?09:56
jiaopengjuOK09:56
*** slaweq_ has quit IRC09:57
*** sidx64 has quit IRC09:58
chenying#endmeeting09:59
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"09:59
openstackMeeting ended Tue Mar  6 09:59:13 2018 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)09:59
openstackMinutes:        http://eavesdrop.openstack.org/meetings/karbor/2018/karbor.2018-03-06-09.42.html09:59
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/karbor/2018/karbor.2018-03-06-09.42.txt09:59
openstackLog:            http://eavesdrop.openstack.org/meetings/karbor/2018/karbor.2018-03-06-09.42.log.html09:59
*** slaweq_ has joined #openstack-meeting10:03
*** d0ugal has joined #openstack-meeting10:07
*** slaweq_ has quit IRC10:08
*** janki has joined #openstack-meeting10:08
*** sidx64 has joined #openstack-meeting10:11
*** slaweq_ has joined #openstack-meeting10:13
*** sidx64 has quit IRC10:15
*** slaweq_ has quit IRC10:18
*** sidx64 has joined #openstack-meeting10:19
*** slaweq_ has joined #openstack-meeting10:23
*** sidx64 has quit IRC10:27
*** masber has joined #openstack-meeting10:28
*** slaweq_ has quit IRC10:28
*** claudiub has joined #openstack-meeting10:29
*** ykatabam has quit IRC10:29
*** alexchadin has quit IRC10:30
*** alexchadin has joined #openstack-meeting10:32
*** slaweq_ has joined #openstack-meeting10:33
*** alexchadin has quit IRC10:37
*** alexchadin has joined #openstack-meeting10:38
*** slaweq_ has quit IRC10:38
*** liyi has joined #openstack-meeting10:40
*** sidx64 has joined #openstack-meeting10:40
*** sidx64 has quit IRC10:43
*** slaweq_ has joined #openstack-meeting10:44
*** liyi has quit IRC10:44
*** sidx64 has joined #openstack-meeting10:46
*** slaweq_ has quit IRC10:49
*** jamesmcarthur has joined #openstack-meeting11:01
*** janki has quit IRC11:03
*** janki has joined #openstack-meeting11:03
*** jamesmcarthur has quit IRC11:05
*** sidx64 has quit IRC11:11
*** slaweq_ has joined #openstack-meeting11:14
*** trinaths has quit IRC11:14
*** sidx64 has joined #openstack-meeting11:17
*** sidx64 has quit IRC11:18
*** slaweq_ has quit IRC11:19
*** alexchadin has quit IRC11:21
*** alexchadin has joined #openstack-meeting11:22
*** slaweq_ has joined #openstack-meeting11:24
*** Tom-Tom has quit IRC11:25
*** sidx64 has joined #openstack-meeting11:27
*** slaweq_ has quit IRC11:29
*** Roamer` has joined #openstack-meeting11:30
*** alexchadin has quit IRC11:31
*** alexchadin has joined #openstack-meeting11:33
*** sidx64 has quit IRC11:33
*** slaweq_ has joined #openstack-meeting11:35
*** lhx_ has quit IRC11:35
*** epico has quit IRC11:36
*** yamamoto has quit IRC11:39
*** slaweq_ has quit IRC11:39
*** psachin has quit IRC11:46
*** sidx64 has joined #openstack-meeting11:46
*** bkopilov has quit IRC11:48
*** liyi has joined #openstack-meeting11:48
*** janki has quit IRC11:51
*** liyi has quit IRC11:53
*** radeks has quit IRC11:56
*** Tom-Tom has joined #openstack-meeting12:00
*** sidx64 has quit IRC12:01
*** Tom-Tom_ has joined #openstack-meeting12:02
*** Tom-Tom has quit IRC12:02
*** rbowen has joined #openstack-meeting12:08
*** sidx64 has joined #openstack-meeting12:14
*** slaweq_ has joined #openstack-meeting12:15
*** slaweq_ has quit IRC12:20
*** Sukhdev_ has joined #openstack-meeting12:22
*** rbowen has quit IRC12:22
*** slaweq_ has joined #openstack-meeting12:26
*** sidx64 has quit IRC12:27
*** chyka has joined #openstack-meeting12:27
*** slaweq_ has quit IRC12:30
*** sidx64 has joined #openstack-meeting12:31
*** chyka has quit IRC12:32
*** alexchadin has quit IRC12:33
*** alexchadin has joined #openstack-meeting12:34
*** slaweq_ has joined #openstack-meeting12:36
*** janki has joined #openstack-meeting12:39
*** yamamoto has joined #openstack-meeting12:40
*** slaweq_ has quit IRC12:41
*** yamamoto has quit IRC12:45
*** sidx64 has quit IRC12:46
*** chenyb4 has joined #openstack-meeting12:46
*** slaweq_ has joined #openstack-meeting12:46
*** sidx64 has joined #openstack-meeting12:47
*** sidx64 has quit IRC12:48
*** slaweq_ has quit IRC12:50
*** egallen has joined #openstack-meeting12:51
*** sidx64 has joined #openstack-meeting12:54
*** zhurong_ has joined #openstack-meeting12:54
*** slaweq_ has joined #openstack-meeting12:56
*** liyi has joined #openstack-meeting12:58
*** egallen has quit IRC12:59
*** slaweq_ has quit IRC13:01
*** zhurong has quit IRC13:02
*** liyi has quit IRC13:02
*** bkopilov has joined #openstack-meeting13:03
*** Tom-Tom_ has quit IRC13:03
*** ttsiouts has quit IRC13:05
*** ttsiouts has joined #openstack-meeting13:05
*** slaweq_ has joined #openstack-meeting13:06
*** XueFeng has joined #openstack-meeting13:10
*** alexchadin has quit IRC13:11
*** slaweq_ has quit IRC13:11
*** slaweq_ has joined #openstack-meeting13:17
*** brault has joined #openstack-meeting13:19
*** eharney has joined #openstack-meeting13:20
*** slaweq_ has quit IRC13:21
*** alexchadin has joined #openstack-meeting13:22
*** markvoelker has quit IRC13:24
*** markvoelker has joined #openstack-meeting13:24
*** slaweq_ has joined #openstack-meeting13:27
*** edmondsw has joined #openstack-meeting13:27
*** pchavva has joined #openstack-meeting13:30
*** Tom-Tom has joined #openstack-meeting13:30
*** lhx_ has joined #openstack-meeting13:31
*** rbowen has joined #openstack-meeting13:31
*** slaweq_ has quit IRC13:31
*** sidx64 has quit IRC13:33
*** abalutoiu_ has joined #openstack-meeting13:34
*** abalutoiu__ has joined #openstack-meeting13:36
*** slaweq_ has joined #openstack-meeting13:37
*** abalutoiu has quit IRC13:38
*** ociuhandu has joined #openstack-meeting13:38
*** abalutoiu_ has quit IRC13:39
*** gongysh has joined #openstack-meeting13:40
*** slaweq_ has quit IRC13:41
*** yamamoto has joined #openstack-meeting13:42
*** zhurong_ has quit IRC13:42
*** ociuhandu_ has joined #openstack-meeting13:42
*** ociuhandu has quit IRC13:42
*** rbowen_ has joined #openstack-meeting13:44
*** fnaval has joined #openstack-meeting13:44
*** ociuhandu_ has quit IRC13:45
*** davidsha has joined #openstack-meeting13:46
*** slaweq_ has joined #openstack-meeting13:47
*** yamamoto has quit IRC13:47
*** fnaval has quit IRC13:49
*** VW has joined #openstack-meeting13:50
*** singlethink has joined #openstack-meeting13:51
*** dprince has joined #openstack-meeting13:51
*** slaweq_ has quit IRC13:52
*** esberglu has joined #openstack-meeting13:55
*** Zames has joined #openstack-meeting13:55
*** julim_ has quit IRC13:57
*** bobh has joined #openstack-meeting13:57
*** slaweq_ has joined #openstack-meeting13:57
*** julim has joined #openstack-meeting13:58
*** slaweq_ has quit IRC14:02
*** mmethot has joined #openstack-meeting14:03
*** Zames has quit IRC14:03
*** rbudden has joined #openstack-meeting14:05
*** slaweq_ has joined #openstack-meeting14:07
*** chenyb4 has quit IRC14:08
*** egallen has joined #openstack-meeting14:11
*** slaweq_ has quit IRC14:12
*** electrofelix has quit IRC14:14
*** dustins has joined #openstack-meeting14:16
*** slaweq_ has joined #openstack-meeting14:18
*** slaweq_ has quit IRC14:23
*** fnaval has joined #openstack-meeting14:27
*** sapd__ has joined #openstack-meeting14:27
*** slaweq_ has joined #openstack-meeting14:28
*** sapd_ has quit IRC14:31
*** Sukhdev_ has quit IRC14:31
*** slaweq_ has quit IRC14:32
*** rbowen_ has quit IRC14:35
*** gibi has joined #openstack-meeting14:38
*** slaweq_ has joined #openstack-meeting14:38
*** egallen has quit IRC14:38
*** abalutoiu has joined #openstack-meeting14:40
*** claudiub|2 has joined #openstack-meeting14:40
*** abalutoiu__ has quit IRC14:40
*** claudiub has quit IRC14:41
*** slaweq_ has quit IRC14:42
*** alexchadin has quit IRC14:43
*** mriedem has joined #openstack-meeting14:43
*** yamamoto has joined #openstack-meeting14:43
*** hongbin has joined #openstack-meeting14:44
*** sridharg has quit IRC14:45
*** liyi has joined #openstack-meeting14:47
*** sridharg has joined #openstack-meeting14:48
*** slaweq_ has joined #openstack-meeting14:48
*** gongysh has quit IRC14:49
*** yamamoto has quit IRC14:49
*** liyi has quit IRC14:51
*** slaweq_ has quit IRC14:53
*** slaweq_ has joined #openstack-meeting14:58
*** awaugama has joined #openstack-meeting14:59
*** slaweq_ has quit IRC15:03
*** spiette has joined #openstack-meeting15:03
*** slaweq_ has joined #openstack-meeting15:09
*** amodi has joined #openstack-meeting15:13
*** slaweq_ has quit IRC15:13
*** dprince has quit IRC15:14
*** julim has quit IRC15:18
*** egallen has joined #openstack-meeting15:18
*** slaweq_ has joined #openstack-meeting15:19
*** slaweq has quit IRC15:22
*** slaweq has joined #openstack-meeting15:22
*** slaweq_ has quit IRC15:23
*** links has quit IRC15:24
*** slaweq has quit IRC15:27
*** masber has quit IRC15:28
*** slaweq has joined #openstack-meeting15:29
*** rbartal has quit IRC15:30
*** mmethot has quit IRC15:31
*** mmethot has joined #openstack-meeting15:31
*** egallen has quit IRC15:31
*** mmethot has quit IRC15:32
*** mmethot has joined #openstack-meeting15:33
*** abalutoiu_ has joined #openstack-meeting15:33
*** slaweq has quit IRC15:34
*** claudiub has joined #openstack-meeting15:34
*** abalutoiu__ has joined #openstack-meeting15:34
*** claudiub|2 has quit IRC15:36
*** abalutoiu has quit IRC15:36
*** dprince has joined #openstack-meeting15:36
*** abalutoiu_ has quit IRC15:38
*** abalutoiu_ has joined #openstack-meeting15:38
*** slaweq has joined #openstack-meeting15:39
*** armax has joined #openstack-meeting15:39
*** abalutoiu__ has quit IRC15:42
*** rcernin has quit IRC15:43
*** slaweq has quit IRC15:44
*** yamamoto has joined #openstack-meeting15:45
*** slaweq has joined #openstack-meeting15:49
*** yamamoto has quit IRC15:51
*** Tom-Tom_ has joined #openstack-meeting15:51
*** slaweq has quit IRC15:53
*** slaweq has joined #openstack-meeting15:54
*** Tom-Tom has quit IRC15:54
*** slaweq_ has joined #openstack-meeting15:55
*** jlibosva has joined #openstack-meeting15:57
*** gouthamr has joined #openstack-meeting15:58
*** mlavalle has joined #openstack-meeting15:59
*** jbadiapa has quit IRC15:59
*** julim has joined #openstack-meeting16:00
*** ihrachys has joined #openstack-meeting16:01
ihrachys#startmeeting neutron_ci16:01
openstackMeeting started Tue Mar  6 16:01:40 2018 UTC and is due to finish in 60 minutes.  The chair is ihrachys. Information about MeetBot at http://wiki.debian.org/MeetBot.16:01
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:01
*** openstack changes topic to " (Meeting topic: neutron_ci)"16:01
openstackThe meeting name has been set to 'neutron_ci'16:01
mlavalleo/16:01
slaweqhi16:01
jlibosvao/16:02
ihrachyshi to everyone who have escaped snowpenstack, or who was lucky enough to not go :p16:02
*** brault has quit IRC16:02
ihrachys#topic Actions from prev meeting16:02
*** openstack changes topic to "Actions from prev meeting (Meeting topic: neutron_ci)"16:02
ihrachys"ihrachys to update grafana periodic board with new names"16:03
ihrachysmerged https://review.openstack.org/54766116:03
ihrachys"report test_get_service_by_service_and_host_name failure in periodic -pg- job"16:03
ihrachysouch that one was on me but I forgot to put my name on it and missed16:03
ihrachysmeh, it's already quite old and we will check status of the job later anyway and figure out what to do with it16:04
ihrachys"mlavalle to look into linuxbridge ssh timeout failures"16:04
slaweqI was looking on that one16:04
mlavalleihrachys: I think we gave that one to slaweq16:04
slaweqso I was debugging it for a while and I couldn't reproduce it locally16:05
slaweqso finally I pushed some DNM patches to be able to telnet to remote_pdb when test will fail16:05
slaweqthis didn't give me anything also16:05
slaweqso I talked with infra team to add my ssh key to such node where this tests failed16:06
*** markstur has joined #openstack-meeting16:06
slaweqand I checked there that all is working fine when I logged in16:06
*** jbadiapa has joined #openstack-meeting16:06
ihrachysand you check in European hours I assume. maybe it's genuine cloud load and slow instance boot?16:07
slaweqso I tried one (stupid) simple thing: https://review.openstack.org/#/c/549324/16:07
slaweqihrachys: no, tests on gate was failing many time16:07
slaweq*many times16:07
*** markstur has quit IRC16:08
slaweqafter all it looks for me that it is just problem with too long time for instance boot16:08
*** markstur has joined #openstack-meeting16:08
ihrachysin linuxbridge job, it's a single node doing all things, tempest, controller, compute...16:09
ihrachysin dvr one, compute is separate16:09
slaweqwhen I proposed patch https://review.openstack.org/#/c/549324/ I run gate jobs twice on it and in both cases lb scenario tests catched my additional log16:09
slaweqihrachys: exactly - I then found that lb is single node and dvr is multinode config16:09
slaweqI couldn't find anything else wrong with it :/16:09
slaweqok, I'm done now :)16:10
ihrachysslaweq, when it hits the message, does it recover quickly after that or still takes some time?16:10
slaweqlet me check as I don't remember exactly16:10
slaweqhttp://logs.openstack.org/24/549324/2/check/neutron-tempest-plugin-scenario-linuxbridge/16197fa/logs/tempest.txt.gz#_2018-03-03_09_52_10_36316:11
slaweqhere is example of such log16:11
slaweqit looks that it took about 1 minute more to connect16:12
*** Tom-Tom_ has quit IRC16:12
ihrachysyeah. and what about the other one? is it also aligned to 1 minute?16:12
ihrachysno, seems like there it took 50 seconds16:13
ihrachysso yeah, it looks just like slow instance spawning16:14
*** ssathaye has quit IRC16:14
slaweqit's not exactly same time for all16:14
slaweqit looks like that for me16:14
ihrachysafair we don't have nested virt enabled in gate16:14
ihrachysafair it was a major issue for octavia scenarios where they needed to start amphora instance per balancer16:15
*** jaypipes has quit IRC16:15
slaweqyes, there was an issue but AFAIR problem was with nested virtualization was only in OVH cloud16:15
*** jaypipes has joined #openstack-meeting16:15
*** julim has quit IRC16:16
ihrachysslaweq, it would probably be interesting to see how long it usually takes in dvr runs16:16
ihrachysto compare. is it also close to the limit?16:16
slaweqone thing which I also found is that it takes really long time to boot an instance in such failed case16:17
*** sapd__ has quit IRC16:17
slaweqe.g.: http://logs.openstack.org/71/548071/5/check/neutron-tempest-plugin-scenario-linuxbridge/d792a4a/job-output.txt.gz#_2018-03-02_17_02_34_20067116:17
slaweqfrom instance logs it looks that it is ready after more than 400 seconds16:17
*** julim has joined #openstack-meeting16:17
*** sapd__ has joined #openstack-meeting16:17
slaweqand it was similar in other cases which I checked16:17
*** trinaths has joined #openstack-meeting16:17
slaweqihrachys: I can check such logs in dvr scenario to compare it16:17
ihrachysslaweq, do I interpret it correctly that it stayed in raising interfaces stage for 3min 42s / 8min 25s ?16:19
ihrachys"Starting Raise network interfaces"16:19
*** sapd__ has quit IRC16:19
ihrachyshere http://logs.openstack.org/71/548071/5/check/neutron-tempest-plugin-scenario-linuxbridge/d792a4a/job-output.txt.gz#_2018-03-02_17_02_34_16802516:20
slaweqI don't know exactly16:20
*** sapd__ has joined #openstack-meeting16:20
ihrachyswell it looks like it. in this case, it may be something on our side that blocks the interface up.16:20
ihrachysmaybe dhcp not replying or smth16:20
ihrachysthough eventually it succeeds16:21
slaweqbut please note that first entry is 2min 52s/ 7 min 47s16:21
ihrachysyeah but maybe that's when it starts to log16:21
ihrachysmy interpretation is that - 'the job took long time already, let's start log about it being still in progress'16:22
slaweqbut please check http://logs.openstack.org/71/548071/5/check/neutron-tempest-plugin-scenario-linuxbridge/d792a4a/job-output.txt.gz#_2018-03-02_17_02_34_16261516:22
slaweqit took 166 seconds even here, before this "raise network interface"16:22
ihrachysyeah but I mean, those 3min+ can be decisive for failure16:23
ihrachysthe limit is what, 5 minutes?16:23
*** andreas_s has quit IRC16:23
slaweqsomething like that,16:23
ihrachysmaybe it doesn't sit there for too long in dvr job16:23
slaweqI will check also dhcp logs for it16:23
ihrachys'just' 166 seconds or so16:23
slaweqplease add me action for that for next week :)16:24
ihrachys#action slaweq to check why it takes too long to raise interfaces in linuxbirdge scenarios job, and to compare with dvr16:24
slaweqthx16:24
ihrachys"ihrachys to update grafana boards to include master data only"16:25
ihrachysso I actually looked into it, and I believe it's my misunderstanding16:25
*** kopecmartin has quit IRC16:25
ihrachysthe board explicitly pulls data from .master. queries16:25
ihrachysso there is no issue with mixed data16:26
ihrachys#topic Grafana16:26
ihrachyshttp://grafana.openstack.org/dashboard/db/neutron-failure-rate16:26
*** openstack changes topic to "Grafana (Meeting topic: neutron_ci)"16:26
*** e0ne has quit IRC16:26
ihrachysthe prev time we decided to give functional and other newly stable jobs some time till next meeting and see what proves itself with time and maybe consider some for voting / gating16:27
*** VW_ has joined #openstack-meeting16:27
*** andreas_s has joined #openstack-meeting16:28
jlibosvafunctional is not voting?16:28
ihrachyslooking at functional / fullstack / api board, it's not clear we can deduce much from it16:28
*** kopecmartin has joined #openstack-meeting16:28
ihrachysjlibosva, I believe it's not gating16:28
ihrachysI mean, it's in check queue but not gate queue16:28
jlibosvais it some recent change?16:28
ihrachyseh. now you make me sanity check :)16:28
*** janki has quit IRC16:29
jlibosvaoh, probably it's just not shown in grafana16:29
ihrachysoh actually nevermind, it's voting / gating16:29
jlibosvaI thought for a second something happened and it was pulled back from gate Q :)16:29
jlibosvasorry for disturbance16:29
ihrachys:) yeah I am bad at tracking the real world16:29
*** VW has quit IRC16:30
ihrachysso as for fullstack, I see spikes in the last week but those seem to repeat what -api- job does16:30
ihrachyscurrently at ~25% failure rate16:30
ihrachyswe even have unit tests at 20%16:31
ihrachysnot sure what happened during the last week16:31
ihrachysbut anyway, we'll take a look at fullstack separately16:32
ihrachysscenarios are doing bad too, 50% for both16:32
slaweqfor linuxbridge it's still same issue as we already talked today16:32
ihrachysslaweq, yeah but now failure rate reflected back in dvr. it was rather stable the last time we gathered.16:33
slaweqI see16:33
ihrachysbut yeah, not clear how much of the rate is because of gate breakage that brought all jobs to 100% failure rate two days ago16:33
ihrachysmaybe we should revisit it again later to get a better picture16:34
ihrachys#action ihrachys to check grafana stats several days later when dust settles16:34
mlavalle++16:34
ihrachysone thing that is new on the board is it seems that rally job is totally busted16:34
slaweqyes, but there is issue for that already reported I think16:35
jlibosvait's gonna be fixed soon16:35
*** jamesmcarthur has joined #openstack-meeting16:35
jlibosvahttps://bugs.launchpad.net/rally/+bug/175371316:35
openstackLaunchpad bug 1753713 in Rally "Rally job on neutron CI broken" [Critical,Fix released] - Assigned to Andrey Kurilin (andreykurilin)16:35
jlibosvafix: https://review.openstack.org/#/c/549627/16:35
ihrachysfix merged16:35
ihrachysso we should be back to normal16:35
jlibosvaoh, it merged :)16:35
ihrachysthey don't need to release right?16:36
jlibosva6 minutes ago, nice :)16:36
ihrachysit's from git16:36
jlibosvathat I don't know16:36
ihrachysyeah it's from master: http://logs.openstack.org/22/549822/4/check/neutron-rally-neutron/1ef49e4/logs/devstacklog.txt.gz#_2018-03-06_11_18_33_77416:37
* jlibosva nods16:37
ihrachysperiodics are stable so that's good too16:37
ihrachysok, now to specific jobs16:38
ihrachys#topic Fullstack16:38
*** openstack changes topic to "Fullstack (Meeting topic: neutron_ci)"16:38
ihrachysslaweq, any patches in your pipeline for fullstack?16:38
slaweqno16:38
slaweqI wasn't doing anything with fullstack in last days16:38
ihrachysehm, I have a question16:39
ihrachysI was looking at a random patch16:39
*** trozet has quit IRC16:39
ihrachyslooking for fullstack results16:39
ihrachyshttps://review.openstack.org/#/c/536747/16:39
ihrachysand I don't see any16:39
ihrachysis it just me16:39
jlibosvait's from january16:39
slaweqI don't see fullstack there also16:40
ihrachysah right. and there's:16:40
ihrachysneutron-fullstack finger://ze03.openstack.org/db42044c8be14621bd79843789dac0c8 : POST_FAILURE in 43m 40s (non-voting)16:40
jlibosvatoday I saw fullstack results16:40
ihrachysin comments16:40
ihrachysok nevermind16:40
ihrachysok here's another one that is fresh16:41
ihrachyshttp://logs.openstack.org/83/549283/1/check/neutron-fullstack/cbad08a/logs/testr_results.html.gz16:41
*** kevinbenton has quit IRC16:41
slaweqlooks like some problem with starting agents/processes16:41
*** andreas_s has quit IRC16:42
ihrachysyeah I checked test_ha_router and neutron-server + 4 agents (2 dhcp and 2 ovs) are up. shouldn't there be also l3?16:42
*** Tom-Tom has joined #openstack-meeting16:43
jlibosvathere should16:43
*** kopecmartin has quit IRC16:43
ihrachyshere is where it failed to start agnet: http://logs.openstack.org/83/549283/1/check/neutron-fullstack/cbad08a/logs/dsvm-fullstack-logs/TestHAL3Agent.test_ha_router.txt.gz#_2018-03-06_08_20_19_23516:43
jlibosvaI checked logs now and Halting async process [neutron/tests/fullstack/cmd/l3_agent.py --log-dir /opt/stack/logs/dsvm-fullstack-logs/TestLinuxBridgeConnectivitySameNetwork.test_connectivity_VXLAN-and-l2pop_ --log-file neutron-l3-agent--2018-03-06--08-13-38-219990.log --config-file /tmp/tmpzpOvIu/tmpZEeIKB/neutron.conf --config-file /tmp/tmpzpOvIu/tmpZEeIKB/l3_agent.ini] in response to an error.16:43
*** jamesmcarthur has quit IRC16:43
*** kevinbenton has joined #openstack-meeting16:43
*** jamesmcarthur has joined #openstack-meeting16:43
ihrachysright. but not many details16:44
jlibosvaI can have a look at it16:44
jlibosvaI'll try it locally16:44
ihrachysyeah. maybe first step is to understand how to collect logs in those cases.16:45
ihrachysmaybe we can somehow get the output at least until it starts logging into console16:45
ihrachyseh, sorry into the logfile16:45
slaweqI'm not sure if agent logs something to console in such case16:46
*** JillS has joined #openstack-meeting16:46
ihrachysand another weird thing is, it fails to start the agent, but still waits for nothing for minutes and minutes. it could fail immediately maybe.16:46
slaweqeven if You spawn it locally with same command if You give --log-file then it will not log to console IMO16:46
jlibosvaI might have a patch for it somewhere in abandoned patches, to read stderr and stdout in case process was halted in response to error16:46
slaweqrecently I was checking it with some DNM patches pushed to gerrit and rechecked :)16:47
jlibosvait should still write output to async process buffers16:47
slaweqbut maybe we should have merged some better logging there16:47
*** jamesmcarthur has quit IRC16:47
*** yamamoto has joined #openstack-meeting16:47
*** Tom-Tom has quit IRC16:47
slaweqjlibosva: You're right16:47
*** jamesmcarthur has joined #openstack-meeting16:47
ihrachys#action jlibosva to look into agent startup failure and missing logs in: http://logs.openstack.org/83/549283/1/check/neutron-fullstack/cbad08a/logs/16:48
slaweqsome time ago I was checking it to fix some problem with l3 agent in fullstack tests16:48
ihrachysthere are not that many fullstack failures actually16:49
ihrachys(I am trying to find another one)16:49
slaweqbtw. there is currently only one test marked as unstable_test() in fullstack16:50
slaweqrelated to security groups16:50
slaweqso if there is no so many failures, it's good IMO :)16:50
ihrachysyeah that's good16:51
ihrachysI actually failed to find another fresh one16:51
*** singlethink has quit IRC16:51
ihrachysso let's assume it's good. we will give it a week to prove it on chart since now it's probably distorted by gate instability16:51
ihrachys#topic Scenarios16:52
*** openstack changes topic to "Scenarios (Meeting topic: neutron_ci)"16:52
ihrachyswe know what's happening in linuxbridge, so let's figure out what's the deal with dvr one16:52
*** Tom-Tom has joined #openstack-meeting16:52
*** yamamoto has quit IRC16:53
haleybsomeone said dvr :)16:54
ihrachys(while looking for a failure, I noticed scenarios not triggered in https://review.openstack.org/#/c/550055/)16:54
jlibosvahaleyb: you have higlighting set for dvr? :)16:54
*** imcsk8 has quit IRC16:54
haleybyup, i'm just looking out of corner of eye due to other meeting16:54
ihrachysok found one failure here: http://logs.openstack.org/07/548607/1/check/legacy-tempest-dsvm-neutron-dvr-multinode-scenario/84cacc5/job-output.txt.gz#_2018-03-06_13_42_42_76753516:55
*** kopecmartin has joined #openstack-meeting16:55
ihrachysssh connection timeout it is16:55
ihrachysslaweq, see it took 1min 29s / 6min 22s16:55
ihrachysto raise interfaces there16:55
ihrachysstill a lot16:55
slaweqihrachys: yes, I was just looking for that :)16:56
ihrachysso maybe effectively bumping the ssh timeout for the job would fix that one too16:56
ihrachysthe instance is up for ~249.666088 until it claims full boot16:58
ihrachysit's ~4 minutes16:58
ihrachysthe timeout of 5 minutes would be enough in this case, so maybe it's something else16:58
ihrachysI guess we can revisit that one after we deal with linuxbridge. maybe the fix will be the same.16:59
mlavallefingers crossed16:59
ihrachysok we are out of time for today. thanks for joining.16:59
ihrachys#endmeeting16:59
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"16:59
slaweqthx16:59
openstackMeeting ended Tue Mar  6 16:59:52 2018 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:59
openstackMinutes:        http://eavesdrop.openstack.org/meetings/neutron_ci/2018/neutron_ci.2018-03-06-16.01.html16:59
slaweqbye16:59
mlavalleo/16:59
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/neutron_ci/2018/neutron_ci.2018-03-06-16.01.txt16:59
openstackLog:            http://eavesdrop.openstack.org/meetings/neutron_ci/2018/neutron_ci.2018-03-06-16.01.log.html16:59
jlibosvathanks, bye :)16:59
*** kopecmartin has quit IRC17:00
*** jamesmcarthur has quit IRC17:00
*** jamesmcarthur has joined #openstack-meeting17:01
*** chyka has joined #openstack-meeting17:01
*** singlethink has joined #openstack-meeting17:02
*** chyka_ has joined #openstack-meeting17:05
*** chyka has quit IRC17:06
*** manjeets has quit IRC17:07
*** lhx_ has quit IRC17:07
*** manjeets has joined #openstack-meeting17:08
*** Patifa has joined #openstack-meeting17:10
*** haint_ has joined #openstack-meeting17:11
*** mlavalle has left #openstack-meeting17:12
*** haint has quit IRC17:13
*** marios has quit IRC17:14
*** trinaths has quit IRC17:19
*** radeks has joined #openstack-meeting17:22
*** Guest61231 has quit IRC17:22
*** imcsk8 has joined #openstack-meeting17:31
*** gyee has joined #openstack-meeting17:33
*** pchavva has quit IRC17:37
*** egallen has joined #openstack-meeting17:41
*** longkb has joined #openstack-meeting17:43
*** longkb has quit IRC17:48
*** yamamoto has joined #openstack-meeting17:49
*** jgr has joined #openstack-meeting17:49
*** spilla has joined #openstack-meeting17:50
*** dustins has quit IRC17:51
*** egallen has quit IRC17:52
*** yamamoto has quit IRC17:54
*** liyi has joined #openstack-meeting17:55
*** gouthamr_ has joined #openstack-meeting17:58
*** gouthamr has quit IRC17:58
*** davidsha has quit IRC17:59
lbragstad#startmeeting keystone18:00
openstackMeeting started Tue Mar  6 18:00:02 2018 UTC and is due to finish in 60 minutes.  The chair is lbragstad. Information about MeetBot at http://wiki.debian.org/MeetBot.18:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.18:00
lbragstadping ayoung, breton, cmurphy, dstanek, gagehugo, henrynash, hrybacki, knikolla, lamt, lbragstad, lwanderley, kmalloc, rderose, rodrigods, samueldmq, spilla, aselius, dpar, jdennis, ruan_he18:00
*** liyi has quit IRC18:00
*** openstack changes topic to " (Meeting topic: keystone)"18:00
openstackThe meeting name has been set to 'keystone'18:00
lbragstad#link https://etherpad.openstack.org/p/keystone-weekly-meeting18:00
lbragstadagenda ^18:00
lbragstado/18:00
lamto/18:00
jgrHello18:00
knikollao/18:01
lbragstadwe have a pretty light agenda today - so we'll give it a minute or two for folks to show up[18:02
lbragstadi also expect people to be recovering from travel or even still dealing with it18:02
rodrigodshey18:03
lbragstadwe'll get started at 5 after18:03
lbragstad#topic trello board/rocky roadmap18:05
*** openstack changes topic to "trello board/rocky roadmap (Meeting topic: keystone)"18:05
lbragstadjust a heads up for everyone18:05
lbragstadi've spent the last few days in trello attempting to capture context from last week18:05
lbragstad#link https://trello.com/b/wmyzbFq5/keystone-rocky-roadmap18:05
*** trozet has joined #openstack-meeting18:05
lbragstadmost of the larger initiatives should be captured accordingly18:05
lbragstadalong with some of the one-off action items18:05
lbragstadhrybacki: and i will being going through it early next week, so there might be some changes between now and next weeks meeting18:06
*** bobh has quit IRC18:06
lbragstadfeel free to poke around and let me know if there is anything i missed18:06
lbragstadnext week we'll focus on setting more realistic timelines and finding/updating ownership18:07
lbragstad#topic new meeting time18:07
*** openstack changes topic to "new meeting time (Meeting topic: keystone)"18:07
lbragstadper the retrospective18:07
lbragstadwe're going to move the keystone weekly meeting time18:07
lbragstadi plan to propose that change today18:07
lbragstadwhich might result in a different meeting room depending on availability18:08
lbragstadnext week we'll plan to at least have the meeting 2 hours earlier18:08
lbragstadi'll send a note to the mailing list once the change is proposed18:09
lbragstad#info no policy meeting tomorrow18:09
lbragstadjust a heads up here - but we're going to cut down on the frequency of the policy meetings on wednesday18:09
lbragstadwe *might* just call for them on an as-needed basis18:09
lbragstadbut - don't expect to have one tomorrow18:10
lbragstad#action lbragstad to propose new meeting time for keystone meeting18:10
lbragstad#action lbragstad to send note to mailing list about new meeting time18:10
lbragstad#action lbragstad to send reminder about policy meeting tomorrow18:10
lbragstadthat's about all i had18:10
lbragstad#topic open discussion18:10
*** openstack changes topic to "open discussion (Meeting topic: keystone)"18:10
lbragstadfloor is open if anyone has comments or questions18:10
lbragstadjust FYI - it's WIP, but i plan to have a full summary of the PTG ready by the eod18:11
samueldmqlbragstad: how was the PTG overall?18:12
lbragstadi thought it went really well18:12
lbragstadimo - the big theme was integrating the stuff we did in Queens with other projects18:13
samueldmqcool. integrating is always great18:13
lbragstadand i thought we got plenty of feedback that push things forward from a cross-project perspective18:13
cmurphyo/ i'm sort of here via phone18:13
samueldmqcmurphy: tapping really fast, ahn?18:13
samueldmq:)18:13
samueldmqtyping********818:14
lbragstadis anyone else working on PTG summaries? or is there something you'd specifically like me to reference in mine?18:14
cmurphyi'm eorking on one18:15
knikollasame18:15
cmurphylots of ntes to go through18:15
cmurphynotes18:15
lbragstadcmurphy: knikolla awesome - i was hoping other would do one18:15
lbragstadothers*18:15
*** pchavva has joined #openstack-meeting18:15
lbragstadthere is so much cross project context to go through18:15
knikollayep. writing is the best way to consolidate that info in a way that flows.18:16
clarkbOne thing I was going to ask about (and maybe this is related to the meeting time change for you all) is someone mentioned lsat week tht in your retrospective certain tools (infra hosted and otherwise) have been problematic for some devs? I'd be curious to hear more about that to make sure there isn't something more infra could be doing to help there18:17
* lbragstad finds a link to the retro18:17
lbragstad#link v18:17
lbragstad#link https://trello.com/b/Vo6dRALh/keystone-queens-retrospective18:17
*** HeOS has quit IRC18:18
cmurphydims had insights into that for huawei devs18:18
clarkbthat board is apparently not visible?18:18
lbragstadah - hrybacki is the only admin18:18
lbragstadand it's only team visible18:18
knikollashould probably make that open18:19
cmurphyirc is cmmonly blocked and there was some problrm (technical?) with responding to mailing lists18:19
lbragstad#action lbragstad to check with hrybacki on making the retro public18:20
cmurphywxy could also speak about it but hes probably asleep18:20
lbragstadcmurphy: yeah - IRC was one of the big problems18:20
*** abalutoiu_ has quit IRC18:20
lbragstadiirc18:20
lbragstadtrello was good because apparently they have access to that18:20
*** abalutoiu has joined #openstack-meeting18:20
lbragstadi'm not sure what other dev tools there were18:21
clarkblbragstad: but no one else has access to trello :/18:21
cmurphyi think email was one but i dont remember if it was a technical reason or a social one18:21
lbragstadcmurphy: i think it was social - they can get access to mailing list, but they don't always feel comfortable responding18:22
lbragstadclarkb: yeah :(18:22
lbragstadcmurphy: i think it was this one - v18:22
lbragstad#link https://trello.com/c/QldE5koC/40-is-there-another-type-of-asynch-communication-method-we-can-use-to-communicate-with-apac-contributors18:22
knikollalbragstad: yes.18:23
*** cloudrancher has joined #openstack-meeting18:23
knikolladiscussion was both on technical and social barriers.18:23
knikollawith irc being blocked, and mailing lists being a social barrier.18:23
clarkbdid we get info on where/how irc was being blocked?18:24
knikollahow to encourage discussion with apac contributors, like etherpads wheere they can easily ++, or write a comment without committing to a full mail post.18:24
clarkb(we don't have to solve all this now, but do want to help if infra can help)18:24
knikollaclarkb: irc was blocked at work, and they don't always have computer access at home. (have to leave laptop at work)18:24
lbragstadyeah - most of the might be trying to follow discussions (in another language) on their phone18:24
lbragstads/the/them/18:25
*** tesseract has quit IRC18:27
cmurphyclarkb: in general infra hosted tools like gerrit and etherpad were said to be fine18:29
clarkbgotcha. Let me know when that trello is opened up and I'll take a look at it in more detail18:29
lbragstadclarkb: ++ will do18:29
lbragstadthanks for checking in18:29
lbragstadclarkb: actually - do you know if you plan to follow up on the performance stuff we talked about last week/18:30
lbragstadwhen we were talking about perf testing and dedicated hardware with mtreinish?18:30
lbragstadi remember the discussion wrapping up with talk about an email thread?18:31
clarkbya I actually ended up briefly talking about it with mnaser at the ptg18:32
clarkbHe had some ideas on the hosting side that I hadn't considered18:32
clarkb(mostly how to orgnize that sort of thing)18:32
lbragstadoh - nice18:32
clarkblbragstad: maybe you can start a thread with some of your needs so they are written down and I can recap current infra abilities then we go get mnaser/mtreinish to chime in too18:33
mnaser^ :)18:33
lbragstad++18:33
lbragstadjust to openstack-dev?18:33
clarkbya I think thats fine18:33
lbragstadok - cool18:34
lbragstad#action lbragstad to write down performance testing needs in a note to the dev mailing list18:34
lbragstadanything else we want to cover today?18:35
*** jamesmcarthur has quit IRC18:36
lbragstadcool - well thanks for coming all18:37
*** rarora has quit IRC18:37
lbragstadreminder office hours will be starting in about 20 minutes18:38
*** chyka_ has quit IRC18:38
lbragstad#endmeeting18:38
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"18:38
openstackMeeting ended Tue Mar  6 18:38:05 2018 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)18:38
openstackMinutes:        http://eavesdrop.openstack.org/meetings/keystone/2018/keystone.2018-03-06-18.00.html18:38
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/keystone/2018/keystone.2018-03-06-18.00.txt18:38
openstackLog:            http://eavesdrop.openstack.org/meetings/keystone/2018/keystone.2018-03-06-18.00.log.html18:38
*** chyka has joined #openstack-meeting18:38
*** cebruns__ has quit IRC18:38
*** harlowja has joined #openstack-meeting18:39
*** cebruns_ has joined #openstack-meeting18:39
*** tobiash has joined #openstack-meeting18:48
*** mmethot has quit IRC18:48
*** mmethot has joined #openstack-meeting18:48
*** mmethot has quit IRC18:49
*** tssurya has quit IRC18:50
*** yamamoto has joined #openstack-meeting18:50
*** Shrews has joined #openstack-meeting18:50
*** mmethot has joined #openstack-meeting18:51
*** yamamoto has quit IRC18:54
*** dustins has joined #openstack-meeting18:58
*** gema has joined #openstack-meeting18:58
*** AJaeger has joined #openstack-meeting18:59
*** dustins has quit IRC18:59
*** hrw has joined #openstack-meeting19:00
*** dustins has joined #openstack-meeting19:00
clarkbhello infra19:00
clarkbanyone here for a meeting?19:00
AJaegerhello clarkb19:00
dmsimard\o19:01
clarkb#startmeeting infra19:01
openstackMeeting started Tue Mar  6 19:01:06 2018 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.19:01
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.19:01
*** openstack changes topic to " (Meeting topic: infra)"19:01
openstackThe meeting name has been set to 'infra'19:01
* hrw just in case19:01
ianwo/19:01
tobiasho/19:01
* dmsimard adds a topic to agenda19:01
clarkb#link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting19:01
clarkb#topic Announcements19:01
*** openstack changes topic to "Announcements (Meeting topic: infra)"19:01
*** VW_ has quit IRC19:01
pabelangerhello19:02
*** VW has joined #openstack-meeting19:02
clarkbThe PTG was last week. Despite being snowed in and the conference kicking us out a day early I think things went reasonably well19:02
* dmsimard added topic to agenda19:02
* mordred waves at the nice people19:03
clarkbBut keep in mind that people may still be traveling (I was traveling less than 12 hours ago still) and don't be surprised if planned discussions didn't happen becuase people were trying to find their way home or just find a guiness19:03
* tobiash waves back19:03
hrwI heard that someone got rebooked from Friday to Thursday19:03
fungiworth noting, the _conference_ didn't kick us out a day early, just the venue19:04
fungiconference went on19:04
mordredhrw: wow19:04
clarkbfungi: right19:04
dmsimarda colleague spent 2 days at the airport for nothing :(19:04
clarkbmany of us did still meet thrusday afternoon and friday, but in a far more ad hoc fashion19:04
dmsimardthe airline just kept delaying the flights before eventually cancelling them19:04
clarkbbasically all that to say expect a slower than normal resumption of dev activities19:05
*** liyi has joined #openstack-meeting19:05
* mordred would like to thank all scandinavian people for their role in getting him away from the snowstorm19:05
clarkb#topic Quick PTG Recap19:05
*** openstack changes topic to "Quick PTG Recap (Meeting topic: infra)"19:05
clarkb#link https://etherpad.openstack.org/p/infra-rocky-ptg19:05
clarkbDespite this the Infra team did manage to get through quite a few of its PTG topics19:05
clarkbIf you are curious about how discussions went we kept notes on that etherpad19:06
fungii was less available for infra discussions than i'd hoped19:06
clarkbAlso I think the helproom days went a bit better than in denver. More communication and explicit scheduling along with zuulv3 being a reality now probably the reason for that19:06
mordred++19:06
fungi#action fungi generate rocky cycle artifact signing key19:07
mordredI wasn't in as may helproom hours as I would have otherwise liked, but I felt I was in a good position to help when I was19:07
* fungi didn't get around to going through that with ptg attendees19:07
pabelangerclarkb: agree, the extra comms we did as a team, helped a lot.19:07
clarkbIf there is anything in particular people are interested in happy to talk about that now, but I should also send a recap to the infra list and we can have proper discussion there too19:08
*** julim_ has joined #openstack-meeting19:08
clarkbok we'll take it to the mailing list then19:09
clarkb#topic Actions from last meeting19:09
*** openstack changes topic to "Actions from last meeting (Meeting topic: infra)"19:09
*** armstrong has joined #openstack-meeting19:09
clarkb#link http://eavesdrop.openstack.org/meetings/infra/2018/infra.2018-02-20-19.01.txt Minutes from last meeting19:09
*** beisner has joined #openstack-meeting19:09
clarkb#action clarkb clean up old specs19:09
clarkbI may actually finally get around to that now that the ptg is over19:09
*** liyi has quit IRC19:10
*** julim has quit IRC19:10
clarkb#topic Priority Efforts19:10
*** openstack changes topic to "Priority Efforts (Meeting topic: infra)"19:10
clarkb#topic Zuul v319:10
*** openstack changes topic to "Zuul v3 (Meeting topic: infra)"19:10
clarkbreally quickly before we get to the gerrit/arm/ara stuff is there anything urgent on to go over with zuul?19:11
ianwumm, it's currently in emergency file, want to talk about that?19:11
fungilet's19:12
clarkb++19:12
ianwzuul-web changed and it needs new puppet in https://review.openstack.org/#/c/549608/ to deploy19:12
clarkb(also I don't think zuul01 ever got rebooted to address the systemd sadness from week before ptg)19:12
fungii saw some of the discussion but didn't have time to work out the situation19:12
* mordred is working on fixing the bits that are needed for 549608 to work19:12
fungisomething about zuul-web static files again, right?19:12
*** imcsk8 has quit IRC19:12
AJaegerclarkb: sorry, jumping back to PTG: Is there an update from some of the help sessions as well? I'm especially interested in the handling of jobs and  irrelevant-files19:13
*** imcsk8 has joined #openstack-meeting19:13
ianwyep, location moved and apache config just needs to be updated to19:13
*** armstrong has quit IRC19:13
AJaegerclarkb: can do via email as well19:13
clarkbAJaeger: I unfortunately was not part of that discussion but definitely something we should follow up with corvus and AJaeger on19:13
clarkber19:13
pabelangerwe are at 25GB of ram, so a reboot / restart would help bring that down again (zuul)19:13
mordredthe main issue remaining is that the building/publishing of the javascript content isn't actually working (and is also broken for storyboard, fwiw)19:13
clarkbcorvus and andreaf19:13
pabelangerhttp://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=64792&rra_id=all19:13
ianwfor 549608 two things for review ... doesn't seem we need the documentroot any more?  since we have a .* redirect?19:14
clarkbmordred: even after updating the job it is still broken?19:14
andreafclarkb what's up?19:14
mordredclarkb: yes - there are a few more issues - I just found one more19:14
mordredclarkb: but we're VERY close19:14
ianwand dmsimard backups of status?  do they actually need to be exported via www?19:14
clarkbandreaf: one sec two discussions happening. We'll come back to the thing you can help with in a few19:14
dmsimardianw: I thought they needed to, but it turns out they don't19:14
dmsimardianw: see the confusion in https://review.openstack.org/#/c/536622/19:15
dmsimardianw: moving them outside of www is fine19:15
ianwok, that's good if they can just live in /var/lib/zuul/backups for admins to use, that's easy19:15
fungiyeah, we should just use them directly from the filesystem and not worry about serving copies of them through apache19:15
pabelanger++19:15
pabelangerI've done that a few times19:15
dmsimardI'll convert 536622 into a documentation patch to explain examples of restoring from file://19:15
fungithey only get used by tools run from the command line on the same server anyway19:15
ianwso seems like fixing all that up is in flight, so seems ok to me19:16
mordredfwiw - for everyone's edification (I'll be updating job descripts with this...)19:16
fungiif the dump script doesn't support a file:/// url or something similar today, we should just fix that and not work around it19:16
ianwi'm around all day if i can help19:16
mordredpublish-openstack-javascript-tarball is about making and publishing a source tarball19:16
mordredpublish-openstack-javascript-content is about making and publishing a bundle of the built html/javascript19:17
dmsimardfungi: I didn't know file:// could be used with the dump script so the confusion came from that19:17
fungii don't know that it can either, just suggesting it's software we wrote, we should make it support what we need rather than adding complexity in other places to work around a lack of a feature we haven't implemented19:18
dmsimardIt can, pabelanger has used it before19:18
fungiperfect. problem solved! ;)19:18
clarkbsounds like the general plan then should be something like 1) fix js publishing jobs 2) fix zuul puppet and remove zuul01 from emergency file 3) reboot zuul01 4) update queue saving tooling if necessary to use local backups ?19:18
ianwmordred: note during debugging, i ran the tools/install_javascript.sh thing on zuul01, so i guess it has yarn & npm installed now.  i ran a manual pip reinstall after that to get the setup hooks to build it19:18
ianwthen i realised the puppet needed to change, so that's when i went for the revert solution19:19
ianwi'll clean that up19:19
mordredianw: cool ...19:19
clarkbmordred: ianw ^ have I captured the list of things above reasonably well?19:19
ianw++19:20
mordredclarkb: ++19:20
clarkbanything else zuul related?19:20
fungisounds like no19:21
*** sridharg has quit IRC19:21
clarkb#topic General Topics19:21
*** openstack changes topic to "General Topics (Meeting topic: infra)"19:21
clarkbianw: Github replication woes19:22
ianwahh, yes19:22
ianwmore people than you maybe think seem to like github19:22
*** brault has joined #openstack-meeting19:22
ianw#link http://lists.openstack.org/pipermail/openstack-infra/2018-March/005842.html19:22
fungithey all come crawling out of the knotholes as soon as replication begins to fall behind19:23
ianwis the debugging i've done, which seems to suggest something has changed and the nova-specs corruption is holding up the github replication thread19:23
* dmsimard has a feeling fungi REALLY doesn't like GitHub19:23
ianwAFAICT, mostly it tries to push and raises an exception, and thing move on19:23
clarkbworth noting we had to restart gerrit during the ptg as it got really slow19:23
clarkband I think this started afterthat19:23
pabelangerload was really high on review.o.o too19:23
ianwbut it seems that it can get stuck, not raise an exception, and then things just bunch up19:23
clarkb(but we didn't update gerrit so behavior shouldn't have changed, but github did update their ssh stuff recently)19:23
fungidmsimard: i'm just not a fan of proprietary software/services, really. nothing particularly unique to github19:24
ianwanyway, i've had a proposal to fix it out for a while described in19:24
ianw#link http://lists.openstack.org/pipermail/openstack-dev/2017-June/119166.html19:24
ianwbut is the issue we want to re-index after we do that?19:24
fungiwe should be able to perform reindexing online19:25
dmsimardclarkb: ah yeah, RDO got bitten by GitHub removing deprecated ciphers -- I didn't notice any issues with the upstream gerrit for that though19:25
fungihowever, gerrit replication and other tasks will lag for some 8-12 hours while online reindexing is performed19:25
clarkbif we don't reindex gerrit will still think it has those refs around?19:26
dmsimardianw: that upstream bug hasn't really gotten any attention afaict :( https://bugs.chromium.org/p/gerrit/issues/detail?id=662219:26
ianwclarkb: i'm not sure, seeing as it's all corrupt who knows if it thinks they're there or not?19:26
fungiclarkb: yeah, not sure what might happen with queries for nova-specs changes if we don't reindex19:27
ianwdmsimard: i'm not surprised, i didn't have a replication for it.  i mostly put it there hoping if someone else sees it, we can collaborate19:27
ianwso a) does someone want to check my work (jump into the logs i guess) and make sure they can't see anything else causing this, other than nova-specs?19:28
ianwb) should i copy out the repo for a backup, run the recovery and trigger a reindex soon?19:28
fungiianw: if memory serves, the corruption seems to have possibly coincided with changes/patch sets which were pushed when gerrit was out of memory19:28
dmsimardianw: does that mean the repo on git.o.o is corrupted as well ?19:29
fungiwe had a few events around that timeframe and during the first couple we checked the repos associated with any of the write errors gerrit logged, but that one seems to have occurred overnight for most of the admins and i don't think anyone checked repository integrity before restarting19:29
clarkbfor a) I know that in the past nova-specs was the only repo doing it, I'm not sure if that is still the case19:29
ianwdmsimard: i guess so, but it doesn't get rejected on replication.  i think it's github's implementation that notices19:30
clarkbfor b) it may be the jetlag or just failing at reading comprehension but I'm not sure I understand the the steps you intend to take. Will gerrit be stopped?19:30
fungiyeah, worst case with git.o.o we can simply blow away the contents on the servers for that repo and then explicitly re-replicate it via the api19:31
*** spilla has left #openstack-meeting19:31
ianwi would stop gerrit, copy out the repo for backup, run the steps to get rid of the bad objects, restart gerrit, reindex?19:31
fungiif we're worried there's lingering corruption there after we repair the canonical copy19:31
*** Sukhdev_ has joined #openstack-meeting19:31
clarkbianw: ok, thats what I thought but wasn't sure based on the email. I think that is a resonable approach19:32
clarkbThe total gerrit outage should be reasonably short in that case, but the reindexing may take a while19:32
clarkbMaybe we just quickly ask release team if that will hold them up and then do it today since people are otherwise recovering from fun travel?19:33
dmsimardianw has the special ability of being on the other side of the planet when things tend to be quieter19:33
fungi"outage" insofar as the service being offline will be short, but effects of ongoing reindexing will be lengthy19:33
fungiwe should make sure, for example, that the release team knows not to approve new releases of stuff until reindexing concludes19:33
dmsimardclarkb: yeah I'm mostly concerned about the cycle-with-trailing projects (not sure if I got that tag name right)19:33
ianwit's also unknown if this will still have issues pushing to github, but i think this is the first step in finding out19:33
clarkbdmsimard: ya but at least one of them was complaining about non working github so may be its ok :)19:34
pabelangerwfm19:34
persiaIf the process is 8-12 hours, maybe start around 20:00 UTC, with the intent of having things online by 8:00 UTC (which is almost but not quite near the end of the day for many of the most easterly of folk)19:34
*** eharney has quit IRC19:34
clarkbanother option is to say no github until the weekend then do it late on friday?19:35
dmsimardianw: I'm not saying it's a solution but out of curiosity, do we have the ability to take nova-specs out of the github replication until this is sorted out so the remainder of the projects can sync ?19:35
fungiyeah, that's probably the least impactful timing given the curve we see on our activity graphs19:35
clarkbthat is less good for ianw though because of timezones (I guess could do it "sunday")19:35
fungii need to take a storyboard outage soonish for some database changes as well and was hoping to do that lateish on friday too19:36
persiaclarkb: Unless I miscalculate 20:00+11 = 7:00 (so later than now).  Not ideal for AU, but less bad than many other times.19:36
ianwusually that's ok, but *this* particular weekend i will be moving house and have very uncertain internet situation19:36
clarkbianw: in that case I think maybe we should try and start today and just work with release team19:36
fungiwfm19:36
dmsimardWe can't temporarily take nova-specs out of the replication to let the remainder of the projects sync ?19:37
dmsimardSo that only nova-specs is out of sync19:37
fungioh, worth noting, the release ptl is in an apac tz this week19:37
mordreddmsimard: not in any easy way19:37
fungiso, yeah, will require some careful notification19:37
* andreaf leaving for dinner now19:37
mordreddmsimard: we do a wildcard replication, don't we?19:37
dmsimardmordred: hm, in RDO's implementation (no jeepyb, mind you) we use regexes to control what we replicate (or not)19:38
clarkbya we do wildcard replication19:38
clarkbwe might be able to configure gerrit to exclude nova-specs19:38
clarkbjeepyb is not involved with replication19:38
clarkbreplication is entirely controlled by the gerrit server config19:38
dmsimardyeah I found http://git.openstack.org/cgit/openstack-infra/puppet-gerrit/tree/templates/replication.config.erb19:39
clarkbif release team says it would be a major burden on them to do this work nowish we can work on an exclusion for nova-specs instead as that will just require a short server restart and no reindex19:39
clarkbianw: ^ that work?19:39
fungichanging replication config also requires a gerrit restart, fwiw19:39
ianwi'll chat with tonyb (after breakfast time :), and if there's issues then look at the alternatives19:39
dmsimardfungi: yeah, this would only allow us to delay the reindexing until it it more convenient for us19:39
clarkbok sounds like we have a plan19:40
clarkbShall we move on to the arm64 update?19:40
fungicheck with smcginnis as well, he should hopefully be waking up soon (though will presumably be busy at the ops meetup)19:40
*** portdirect has quit IRC19:41
clarkbfungi: that may mean its an excellent time for the work19:41
fungipotentially19:41
*** mrmartin has quit IRC19:41
ianwokdokie19:41
*** portdirect has joined #openstack-meeting19:42
*** mrmartin has joined #openstack-meeting19:42
*** tomhambleton_ has quit IRC19:42
clarkbNow for some arm 64 updating19:42
*** tomhambleton_ has joined #openstack-meeting19:42
clarkbsounds like we ran a job or jobs?19:42
ianwyes, i forget where the last update was, but we have nodes and they work19:43
ianwcurrently, AJaeger pointed out late my time last night the builder has disconnected or something http://nl01.openstack.org/dib-image-list19:43
ianwi will look into that, but builds are working19:43
gemaexcellent news, I have asked my team to work with jeffrey4l on adding some kolla jobs19:43
persiahttps://review.openstack.org/546466 merged, and ianw started a job to create the ubuntu-ports mirror.  last I checked, it was published over AFS, but the contents aren't accessible yet.19:43
ianwpersia started on some jobs19:44
mordred\o/19:44
dmsimardTIL /dib-image-list is a thing19:44
ianwfirst issue was i'd named the mirror wrong (mirror.cn1 rather than mirror.regionone...)19:44
ianwthat fixed pip19:44
ianwpersia also updated reprepro for ubuntu-ports19:45
ianwi created the volume and started running a sync, but overnight it's run out of quota so i need to look into that19:45
ianwwe may need some more disk on the afs servers, i will check19:45
ianwhopefully today i will push changes to make the mirror setup use ubuntu-ports when appropriate19:46
pabelangerianw: where did regionone come from?19:46
ianwthat's the actual region name in like clouds.yaml config19:46
pabelangerah, I see it http://logs.openstack.org/19/549319/1/check/storyboard-tox-pep8/bb91d84/zuul-info/inventory.yaml19:46
clarkbright we build the name from the nodepool cloud info19:46
ianwbut the horizion is cn1.linaro.cloud and i've been calling it "cn1"19:46
pabelangerkk, so we need to rebuild the mirror? or just update dns19:46
clarkbsince that is what ends up in the ansible inventory19:46
ianwi just modified the cname, for now.  we could rebuild, not sure it matters19:47
*** pchavva has quit IRC19:47
ianwthe problem for jobs atm is that the mirror setup overwrites things so it tries to get at the x86 repo, ergo no packages can install19:48
*** pchavva has joined #openstack-meeting19:48
clarkbfor now its probably not a major thing but could be once new regions are up if they are also called regionone19:48
pabelangermirror.regionone.linaro.o.o is new CNAME?19:48
ianwso once we have that sorted ... i think jobs will work!19:48
mordredif they are also called regionone it will be a problem19:48
gemaclarkb: we can change the name19:48
ianwpabelanger: yep19:48
persiapabelanger: Yes.19:48
pabelangercool19:48
clarkbgema: I think as long as the new regions have distinct names it iwll be fine (don't have to change existing cloud)19:49
pabelangermordred: yah, thinking that too19:49
persiaIt is possible to run a working job now, but the job has to not require any extra packages.19:49
gemaclarkb: ack, no problem, will let niedbalski know19:49
persiaOutstanding items include: wheel mirrors (to speed up jobs), working around some of the other mirrors not having the right architectures, etc.19:50
mordred(to be fair, we *can* deal with two regions named regionone - it'll just mean each one will get their own cloud name in clouds.yaml)19:50
*** yamamoto has joined #openstack-meeting19:50
clarkbsounds like progress and no major hurdles?19:50
* mordred thinks this is super-cool fwiw19:51
ianwthat's it, i think ... thanks to persia who has been super helpful getting things moving!19:51
pabelanger++19:51
mordred++19:51
clarkbthank you to everyone getting this going, I too think this is super cool19:51
persiaclarkb: As we're now in a place where we can start running things, I think we're about to find the hurdles.  Lots of naming assumptions, need to do manual things (like AFS volume creation), etc.19:51
pabelangerlots of excitement at PTG for arm6419:51
clarkbdmsimard had an ara topic as well which we have a little time left over for19:52
dmsimardohai19:52
persiaMy guess is that we're about a month out from reliably running a significant number of jobs (and I'm hoping gema, niedbalski, and others can find more resources for then)19:52
dmsimardI just wanted to mention that https://review.openstack.org/#/q/topic:ara-sqlite-middleware would let us enable ara reports for all jobs without having to generate (or store) HTML19:53
hrwpabelanger: and that's good thing19:53
clarkb#link https://review.openstack.org/#/q/topic:ara-sqlite-middleware19:53
dmsimardIf I didn't screw up anything, that is19:53
clarkbdmsimard: this will drastically cut down on the number of required inodes per job right? allowing us to go back to having ara enabled for all jobs?19:53
clarkb(currently we only generate ara reports on failed jobs)19:53
dmsimardclarkb: yeah, I explain the delta with an example openstack-ansible job here: https://ara.readthedocs.io/en/latest/advanced.html19:54
dmsimardclarkb: tl;dr, one small file instead of thousands of larger files19:54
pabelangersome jobs like ansible and tripleo are still generating their own ara reports, could we update them to use sqlite first too?19:54
pabelangeras another data point of it working19:54
*** liyi has joined #openstack-meeting19:55
clarkbpabelanger: they would have to update their jobs but I would expect that our apache server would do the right thing for them too19:55
dmsimardclarkb: another advantage is that we don't have to generate the HTML (which can take >1 min for *large* runs) and we also don't need to rsync that to the log server19:55
dmsimardpabelanger: yes, so the goal here is to try it on logs-dev.o.o first19:55
pabelangeryah, logs-dev.o.o wfm19:55
dmsimardpabelanger: and technically, ".*/ara-report/ansible.sqlite" should work for any project19:55
dmsimardI mean, logs.o.o/some/tripleo/job/logs/foo/ara-report/ansible.sqlite19:55
fungimy experience suggests that rsync time is dictated more by inode count than block count too19:56
*** yamamoto has quit IRC19:56
clarkbsounds like it will be a good improvement. Please review if you have time :)19:56
persiaI share that experience with rsync19:56
fungiso overall this could be a significant chunk of time savings19:56
clarkb#topic Open Discussion19:56
*** openstack changes topic to "Open Discussion (Meeting topic: infra)"19:56
clarkbReally quickly before our hour is up anything else?19:56
dmsimardIf you're curious, I have a standalone hosted test for the middleware19:56
jlvillalReview request: https://review.openstack.org/#/c/546700/  Make gerritbot install from git master19:57
fungito the earlier discussion of possibly filtering nova-specs out of gh replication, i'm not immediately seeing how to do that from https://gerrit.googlesource.com/plugins/replication/+/stable-2.13/src/main/resources/Documentation/config.md19:57
clarkb#link https://review.openstack.org/#/c/546700/ install gerritbot from git master instead of latest release on pypi19:57
jlvillalAnd there are 4 (of my) patches up for review for gerritbot: https://review.openstack.org/#/c/545607/19:57
*** gema has left #openstack-meeting19:57
jlvillalThanks :)19:57
mtreinishI still could use some help with the subunit2sql check db stuff:19:57
mtreinishbasically just need reviews and someone to help drive setting things up after they merge19:58
* hrw out19:58
*** hrw has left #openstack-meeting19:58
fungii've also been working with zaro_ in #openstack-storyboard to try to determine why story comments via gerrit's its-storyboard plugin have stopped happening... seems like the timing may be related to our 2.13 upgrade19:58
*** salv-orl_ has quit IRC19:58
fungiif anybody has an interest in helping with that. lmk19:58
mtreinishhttps://review.openstack.org/#/q/status:open+topic:subunit2sql-check-data19:58
*** salv-orlando has joined #openstack-meeting19:59
clarkbfungi: reading the gerrit docs really quickly maybe we want to set max retries and a timeout19:59
clarkbfungi: and just set that for the github replication and maybe that will get things moving again19:59
fungioh, and we have a new donor cloud ready to be brought up. if any newer infra-roots want to give that a try, i have the credentials for the accounts and am happy to pass that activity along19:59
*** liyi has quit IRC20:00
mordredfungi: \o/20:00
dmsimardfungi: I think i'd like to sign up for that20:00
ianwmtreinish: i have some familiarity with that, i will take a look20:00
clarkbI'm largely going to be out today. My brain is unworking and I haven't had a proper meal in over a day20:00
fungidmsimard: awesome--i'll get up with you in #openstack-infra later20:00
mtreinishianw: ok cool, thanks20:00
clarkbI'll follow along on irc and help as I can but should be here properly tomorrow20:00
clarkband with that we are out of time20:00
clarkbthank you everyone20:00
clarkb#endmeeting20:00
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"20:00
openstackMeeting ended Tue Mar  6 20:00:54 2018 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)20:00
openstackMinutes:        http://eavesdrop.openstack.org/meetings/infra/2018/infra.2018-03-06-19.01.html20:00
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/infra/2018/infra.2018-03-06-19.01.txt20:00
*** AJaeger has left #openstack-meeting20:00
openstackLog:            http://eavesdrop.openstack.org/meetings/infra/2018/infra.2018-03-06-19.01.log.html20:01
*** tobiash has left #openstack-meeting20:01
*** tssurya has joined #openstack-meeting20:02
*** Shrews has left #openstack-meeting20:03
*** salv-orlando has quit IRC20:03
*** david-lyle has joined #openstack-meeting20:08
*** david-lyle has quit IRC20:15
*** eharney has joined #openstack-meeting20:18
*** jamesmcarthur has joined #openstack-meeting20:26
*** jezebel_ has joined #openstack-meeting20:33
*** racoonmonk has joined #openstack-meeting20:33
*** jlibosva has quit IRC20:34
*** jezebel_ has quit IRC20:36
*** racoonmonk has quit IRC20:37
*** david-lyle has joined #openstack-meeting20:39
*** ihrachys_ has joined #openstack-meeting20:41
*** e0ne has joined #openstack-meeting20:42
*** ihrachys has quit IRC20:43
*** jamesmcarthur has quit IRC20:45
*** powerd has joined #openstack-meeting20:47
*** david-lyle has quit IRC20:52
*** yamamoto has joined #openstack-meeting20:52
*** dprince has quit IRC20:54
*** martial has joined #openstack-meeting20:54
*** Patifa has quit IRC20:56
*** jamesmcarthur has joined #openstack-meeting20:57
*** yamamoto has quit IRC20:57
*** oneswig has joined #openstack-meeting20:58
*** designbybeck has joined #openstack-meeting20:59
*** Patifa has joined #openstack-meeting20:59
*** salv-orlando has joined #openstack-meeting20:59
*** mmethot has quit IRC21:00
oneswig#startmeeting scientific-sig21:00
openstackMeeting started Tue Mar  6 21:00:22 2018 UTC and is due to finish in 60 minutes.  The chair is oneswig. Information about MeetBot at http://wiki.debian.org/MeetBot.21:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.21:00
*** openstack changes topic to " (Meeting topic: scientific-sig)"21:00
openstackThe meeting name has been set to 'scientific_sig'21:00
oneswigahoy there21:00
oneswig#link agenda for today https://wiki.openstack.org/wiki/Scientific_SIG#IRC_Meeting_March_6th_201821:00
oneswig#topic SIG roundup from the PTG21:02
*** openstack changes topic to "SIG roundup from the PTG (Meeting topic: scientific-sig)"21:02
rbuddenhello21:02
oneswigHi Bob21:02
oneswigHow's Bridges?21:02
rbuddenDoing good21:03
*** ircuser-1 has quit IRC21:03
rbuddenKeeping us all busy!21:03
oneswigWe had some discussion around Ironic deploy steps, ramdisk boot and kexec - I think you were interested in this?21:03
rbuddenindeed21:03
*** b1airo has joined #openstack-meeting21:03
oneswigSeems like the deploy steps concept is right for you then :-)21:04
*** salv-orlando has quit IRC21:04
b1airoMorning oneswig21:04
oneswigIt was lucky, we were scheduled at a quiet time when not much was going on21:04
oneswigHey Blair21:04
oneswig#chair b1airo21:04
*** liyi has joined #openstack-meeting21:04
openstackCurrent chairs: b1airo oneswig21:04
*** e0ne has quit IRC21:04
b1airoI'm wrangling the kids to school so one eye on this21:04
oneswigas a result, we had good attendance by a number of key people21:04
rbuddenawesome21:05
rbuddeni’m checking out the etherpad now21:05
oneswigb1airo: I got my kids to school hours ago...21:05
oneswigJulia Kreger seemed particularly comfortable with the idea of supporting ramdisk boot as a proven technique21:06
b1airoBugger, I must have overslept!21:06
oneswigThat was at the tail end of a couple of hours of discussion though.  We had an in-depth update on preemptible instances from ttsiouts at CERN21:07
TheJuliaextremely comfortable given what I've read where it has been done in various deployments21:08
martialHello all21:08
oneswigHi TheJulia!21:08
oneswigthanks for joining21:08
oneswigHi martial21:08
oneswig#chair martial21:08
openstackCurrent chairs: b1airo martial oneswig21:08
*** liyi has quit IRC21:08
*** mmethot has joined #openstack-meeting21:08
oneswigI was just recapping the discussion (although running backwards)21:09
*** armstrong has joined #openstack-meeting21:09
oneswigrbudden: one action we took was to document more clearly our use cases for non-conventional Ironic deployment steps21:09
rbuddensounds good21:10
oneswigTheJulia: can you remind me the best way use cases could be made available for the Ironic team?21:10
TheJuliaoneswig: a new use case to support a new thing, or an existing usecase that we already support?21:11
oneswigNew thing - eg ramdisk boot21:11
oneswigStoryboard / launchpad?21:12
TheJuliaoneswig: Create an bug tagged with [RFE] in the subject on ironic's launchpad21:12
TheJuliaat least, until we migrate to launchpad. I have to find a spare network cable before I can run a test migration to storyboard21:12
oneswigOK, launchpad will work for now21:13
oneswigIf I create a bug and sketch out the need, rbudden can you add details specific to what you'd like for Bridges?  I'll circulate to Pierre and Tim as well21:13
rbuddensure21:14
rbuddenI think Trandles has the largest use case for ramdisk boot21:14
rbuddenwe could use that as well for our 12TB nodes on Bridges, but we only have a handful of them21:14
oneswigSounds like he's up to something interesting21:14
rbuddenI think boot from Cinder Vol would fix us up as well21:14
rbuddenobviously we like kexec to avoid multiple reboots21:15
oneswigrbudden: there was some discussion on multi-attach for large scale cinder volume boot, I think it needs some testing at scale (which we may try in a couple of months)21:16
oneswigrbudden: how long to reboot a node with 12 TB RAM?21:16
*** mmethot has quit IRC21:16
rbuddeni don’t have the exact number off the top of my head, but i recall when we PXE booted it from the showfloor at SC it was at least 30 min :(21:16
rbuddenwe unded up pulling blades to debug to cut back the boot time since it was a demo ;)21:17
TheJuliaoneswig: if you do, we would love to know the details behind any testing since some systems have architectural limits.21:17
rbuddenthat was years ago though, so i’m unsure if there have been improvements to disable things like ram check, etc. at the iLO level21:17
oneswigTheJulia: johnthetubaguy and mgoddard are likely to be leading it - I'm sure they'll keep you updated.  The rough scale is deploying to ~600 ironic nodes.21:18
TheJuliaAwesome, thanks!21:18
oneswigShould be a lot of fun :-)21:19
*** VW has quit IRC21:20
*** VW has joined #openstack-meeting21:20
*** cloudrancher has quit IRC21:20
oneswigThere was a good deal of interesting discussion on preemptible instances, including how they might interact with reservations in Blazar.  I think that was one of the highlights of the session21:21
oneswigThat discussion gained some user input from the Scientific SIG and went on to a Nova group discussion on the Friday afternoon.21:23
oneswigIt was a bit difficult to focus by that time given everyone had just had their flights cancelled but I think the Nova team soldiered on.21:23
*** david-lyle has joined #openstack-meeting21:24
oneswigOne of the nuances was on whether to perform the preempting action (ie, killing an instance) upon the final "NoValidHost" event, or to attempt to do it slightly before then based on (eg) 95% utilisation.21:24
*** VW has quit IRC21:24
oneswigI think the CERN team want the former to get maximum utilisation21:25
oneswigThe latter might feasibly be a role performed by a process like Watcher.21:25
oneswigThere was also some discussion on a new strategy for resolving quotas across nested project hierarchies.21:27
b1airoWould be nice to have that option integrated given how close they are21:27
oneswigb1airo: right - seems like it, although having many concurrent actors could make a complex system chaotic.  Perhaps one strategy will win out.21:28
*** e0ne has joined #openstack-meeting21:29
b1airoFiguring out what 95% is could be a difficult problem in real deployments21:29
oneswigThe quota issue may be resolved in the long term through managing support for quotas through a new Oslo library, tasked with managing resource quotas across a subtree of projects21:29
oneswigThere were some interesting issues raised on how to count resource consumption when (eg) mixing virtualised and bare metal compute, given the custom resource classes of baremetal.21:31
b1airoMy natural instinct is that they should be separate quotas21:31
oneswigb1airo: does it all come back to the placement service in the end? On your previous comment21:32
b1airoI suspect it has to21:32
*** julim_ has quit IRC21:35
oneswigThere was also some interesting discussion in the Ironic sessions on complex deploys - multi-partition, RAID, etc.21:36
oneswigWe also briefly talked about setting BIOS config during deploy steps.  This raises a question on how to undo in cleaning all that was done in deployment.21:37
rbuddeni’m not sure if it currently exists, but a way to plugin certain cleaning steps would be nice21:38
rbuddenspecifically for us it would be for puppet cert cleanup before a redeploy21:38
martial(catching up on the typed text, why were flight canceled?)21:38
oneswigmartial: it snowed a freakish amount for Ireland.21:39
oneswighttps://twitter.com/DublinAirport/status/96936826566226739321:40
*** david-lyle has quit IRC21:41
*** Sukhdev_ has quit IRC21:41
oneswigI got home after ~36 hours, the airport had only just reopened then.  This part of Europe isn't geared to handle weather like that, everything shuts down...21:41
b1airoNo plows lining the runways like in Chicago21:43
TheJuliaThey had plows at the airport... but yeah21:43
*** salv-orlando has joined #openstack-meeting21:43
TheJuliaoneswig: with regards to undo settings applied, I'm fairly sure ironic may only need to undo the boot node, but I've not had much time to think about it.. nor ability to brain after the two day trek home.21:45
oneswigrbudden: I think you can already create custom clean steps, but perhaps you'd need to roll your sleeves up - https://docs.openstack.org/ironic/pike/admin/cleaning.html21:45
TheJuliaoneswig: a distinct use case where we would need to peel things back that is not raid would be appreciated, if your aware of one21:45
rbuddenoneswig: thanks, i’ll check that out. i haven’t played with cleaning much, but always find simple cleanup steps that would be awesome to just automate21:46
oneswigTheJulia: aside from RAID our main use cases are hyperthreading and power profile.21:46
oneswigI guess hyperthreading is the one you'd notice immediately21:46
TheJuliaI think those could all be done upon next deployment if we get deploy steps sorted with the bios interface work21:47
oneswigTheJulia: if it could be done in one hit, that would be good - avoiding another reset...21:48
rbuddenhyperthreading is a good one, we occasionally get requests for this as well21:48
TheJuliaI suspect it would almost be better to always try to assert desired state upfront. The only thing I can really think of is needing special firmware, but that is.... yeah.21:49
oneswigTheJulia: careful what you wish for! :-21:50
TheJuliaI'm sure that would make some operators happy21:50
*** ekcs has joined #openstack-meeting21:51
oneswigIt would mean a comprehensive picture of default settings, to totally define hardware state upon deployment21:51
*** david-lyle has joined #openstack-meeting21:51
*** VW has joined #openstack-meeting21:51
oneswigI think that was all I had on the PTG - TheJulia was there anything the scientific SIG would really like from the Ironic sessions?21:51
b1airoVirtualisation features would be another common toggle21:52
oneswigb1airo: agreed.21:54
*** yamamoto has joined #openstack-meeting21:54
oneswigBTW have you seen this project from Dell - https://github.com/dsp-jetpack/JetPack21:54
oneswigThe missing piece from python-dracclient (NIC config) is found here.21:55
TheJuliaoneswig: I'm still typing up everything. We did briefly discuss firmware management but there are many different ways we can approach that.21:55
oneswigThanks TheJulia, I'll follow that (probably indirectly via Mark and John)21:56
*** VW has quit IRC21:56
oneswigWe are nearly out of time...21:57
oneswig#topic AOB21:57
*** openstack changes topic to "AOB (Meeting topic: scientific-sig)"21:57
oneswigQueens is imminent!21:57
*** jamesmcarthur has quit IRC21:57
oneswigMark did a test deploy to shake out some things in Kolla and Bifrost21:57
oneswigOne other announcement - https://github.com/openstack/kayobe - one step closer21:59
*** VW has joined #openstack-meeting21:59
b1airoI saw mikal praising Bifrost on Twitter :-)21:59
*** yamamoto has quit IRC21:59
oneswigit's the future of deployment! :-)22:00
oneswigOn that happy note, final comments?22:00
b1airoHe'll be on to Kayobe next22:00
martialOur P2302/ORCA meeting is coming soon ( March 20-21) ... details at federatedcloud.eventbrite.com22:00
oneswigWon't we all b1airo :-)22:00
martial(final comments shameless plug ;) )22:00
*** mmethot has joined #openstack-meeting22:01
oneswigthanks martial22:01
oneswiggood reminder!22:01
oneswigOK, we are out of time22:01
oneswig#endmeeting22:01
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"22:01
openstackMeeting ended Tue Mar  6 22:01:21 2018 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)22:01
openstackMinutes:        http://eavesdrop.openstack.org/meetings/scientific_sig/2018/scientific_sig.2018-03-06-21.00.html22:01
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/scientific_sig/2018/scientific_sig.2018-03-06-21.00.txt22:01
openstackLog:            http://eavesdrop.openstack.org/meetings/scientific_sig/2018/scientific_sig.2018-03-06-21.00.log.html22:01
rbuddencya later!22:01
*** akuznetsova has quit IRC22:04
*** akuznetsova has joined #openstack-meeting22:04
*** jamesmcarthur has joined #openstack-meeting22:05
*** devananda has joined #openstack-meeting22:05
*** martial has quit IRC22:08
*** david-lyle has quit IRC22:13
*** ircuser-1 has joined #openstack-meeting22:13
*** esberglu has quit IRC22:13
*** pchavva has quit IRC22:14
*** oneswig has quit IRC22:16
*** rcernin has joined #openstack-meeting22:20
*** claudiub has quit IRC22:20
*** claudiub has joined #openstack-meeting22:21
*** dustins has quit IRC22:23
*** amodi has quit IRC22:24
*** powerd has quit IRC22:25
*** julim has joined #openstack-meeting22:29
*** david-lyle has joined #openstack-meeting22:30
*** radeks has quit IRC22:34
*** ykatabam has joined #openstack-meeting22:35
*** david-lyle has quit IRC22:40
*** andreas_s has joined #openstack-meeting22:41
*** mriedem is now known as mriedem_afk22:41
*** felipemonteiro has joined #openstack-meeting22:42
*** andreas_s has quit IRC22:46
*** masber has joined #openstack-meeting22:50
*** Patifa has quit IRC22:52
*** tssurya has quit IRC22:54
*** masuberu has joined #openstack-meeting22:55
*** yamamoto has joined #openstack-meeting22:56
*** e0ne has quit IRC22:57
*** masber has quit IRC22:59
*** Patifa has joined #openstack-meeting23:01
*** yamamoto has quit IRC23:02
*** yamamoto has joined #openstack-meeting23:06
*** Patifa has quit IRC23:07
*** Patifa has joined #openstack-meeting23:09
*** slaweq has quit IRC23:13
*** slaweq has joined #openstack-meeting23:14
*** slaweq has quit IRC23:18
*** mmethot has quit IRC23:20
*** david-lyle has joined #openstack-meeting23:26
*** felipemonteiro has quit IRC23:27
*** aeng has joined #openstack-meeting23:27
*** singlethink has quit IRC23:35
*** gouthamr_ has quit IRC23:38
*** david-lyle has quit IRC23:42
*** hongbin has quit IRC23:45
*** jamesmcarthur has quit IRC23:46
*** fnaval has quit IRC23:53

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!