Tuesday, 2018-12-11

*** efried has quit IRC00:05
*** macza has quit IRC00:09
*** macza_ has joined #openstack-meeting00:09
*** efried has joined #openstack-meeting00:10
*** slaweq has joined #openstack-meeting00:11
*** efried has quit IRC00:14
*** efried has joined #openstack-meeting00:14
*** efried has quit IRC00:15
*** slaweq has quit IRC00:16
*** tetsuro has joined #openstack-meeting00:25
*** Liang__ has joined #openstack-meeting00:40
*** yamamoto has quit IRC00:49
*** zaneb has quit IRC00:55
*** mmethot has joined #openstack-meeting00:58
*** _alastor_ has joined #openstack-meeting00:59
*** rockyg has quit IRC00:59
*** hongbin has quit IRC01:00
*** zaneb has joined #openstack-meeting01:01
*** rockyg has joined #openstack-meeting01:07
*** rockyg has quit IRC01:14
*** ianychoi has quit IRC01:20
*** _alastor_ has quit IRC01:25
*** bobh has joined #openstack-meeting01:27
*** bobh has quit IRC01:31
*** Liang__ is now known as LiangFang01:33
*** mahatic has quit IRC01:59
*** _alastor_ has joined #openstack-meeting02:01
*** _alastor_ has quit IRC02:06
*** bobh has joined #openstack-meeting02:08
*** bobh has quit IRC02:12
*** tetsuro has quit IRC02:26
*** mhen has quit IRC02:30
*** mhen has joined #openstack-meeting02:33
*** macza_ has quit IRC02:33
*** macza has joined #openstack-meeting02:37
*** macza has quit IRC02:41
*** tetsuro has joined #openstack-meeting02:48
*** psachin has joined #openstack-meeting02:52
*** anteaya has quit IRC03:01
*** apetrich has quit IRC03:15
*** hongbin has joined #openstack-meeting03:15
*** yamahata has quit IRC03:23
*** bzhao__ has joined #openstack-meeting03:28
*** bobh has joined #openstack-meeting03:39
*** mahatic has joined #openstack-meeting03:39
*** iyamahat_ has joined #openstack-meeting03:40
*** bobh has quit IRC03:43
*** iyamahat has quit IRC03:44
*** iyamahat_ has quit IRC03:50
*** tashiromt has joined #openstack-meeting03:52
*** gyee has quit IRC03:53
*** tpatil has joined #openstack-meeting03:55
*** markvoelker has joined #openstack-meeting03:57
samPhi all for masakari04:00
tpatilHi04:00
tashiromtHi04:00
samP#startmeeting masakari04:00
openstackMeeting started Tue Dec 11 04:00:57 2018 UTC and is due to finish in 60 minutes.  The chair is samP. Information about MeetBot at http://wiki.debian.org/MeetBot.04:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.04:00
*** openstack changes topic to " (Meeting topic: masakari)"04:01
openstackThe meeting name has been set to 'masakari'04:01
samPtpatil: hi04:01
samPtashiromt: hi04:01
samPLet's start04:01
*** bobh has joined #openstack-meeting04:01
samP#topic Critical bugs and patches04:01
*** openstack changes topic to "Critical bugs and patches (Meeting topic: masakari)"04:01
samPAny critical bugs or patches need discussion?04:02
*** markvoelker has quit IRC04:02
tpatil#link : https://review.openstack.org/#/c/621535/04:03
tpatilNot a critical bug04:03
tpatilRequest you to please review this patch04:03
samPAh, yes04:03
samPtpatil: sure, I will04:03
tpatilI had reviewed it once, then got some minor nits comments which are addressed in the new PS04:04
tpatilsamP: Thanks04:04
tpatilanother patch that needs review is https://review.openstack.org/#/c/616068/04:04
tpatilalready got one +2 from Sagara04:04
samPtpatil: sorry, I did review this. forgot to approve it..04:05
samPtpatil: done04:05
tpatilsamP: Great, Thanks04:06
*** bobh has quit IRC04:06
samPkengo updated the following 3 patches,04:06
samP#link https://review.openstack.org/#/c/585625/04:07
samP#link https://review.openstack.org/#/c/585596/04:07
samP#link https://review.openstack.org/#/c/585592/04:07
tpatilYes, I have review these patches once04:07
tpatilShilpa is working on the last patch04:07
samPyes, it seems that your comments not addressed in the patch04:08
samPtpatil: About last patch, got it.04:08
tpatilShe haven't uploaded these patches yet on the community Gerrit04:08
tpatilAll patches will be up for review by end of this week04:09
samPtpatil: if Shilpa's patch lands, no need the last patch 585592?04:09
tpatilSorry I have mistaken it with the notification API related changes which Shilpa is currently on04:11
*** zaneb has quit IRC04:11
samPtpatil: np. so, last patch is still valid04:11
*** zaneb has joined #openstack-meeting04:11
tpatilsamP: Yes04:11
samPtpatil: OK then, I will review all 3 of them. Please do review them if you have time04:12
tpatilsamP: sure04:12
samPThere some amount of patches in all repos, about fixing the docs, readme.rst, hacking, and etc.04:15
*** mriedem_away has quit IRC04:15
tpatilYes, I will check these patches and review them at my end04:16
samPI would like to propose a change a patch merge policy for those; that single core reviewer can review and merge them04:17
*** ShilpaSD has joined #openstack-meeting04:17
tpatil+104:17
samPmost of them contain simple changes. However if you feel that it might need a another look, then feel free to left it with your comment04:18
samPthanks.04:18
*** LiangFang has quit IRC04:18
*** Liang__ has joined #openstack-meeting04:19
tpatilsamP: Ok04:19
samPany other patches or bugs/04:19
samPif any please bring them up anytime in this meeting04:20
samP#topic Stein Work Items04:20
*** openstack changes topic to "Stein Work Items (Meeting topic: masakari)"04:20
samPPlease share any updates on Stein Work Items04:20
tpatilRecovery workflow events : Not yet started writing specs. Will work on it soon. The last part of checking whether notification uuid can be linked to the taskflow LogBook is done now.04:22
tpatilSo now I have all bits that is required to push events during execution of recovery workflow. Based on this info, I will start writing up specs now04:23
*** _alastor_ has joined #openstack-meeting04:23
samPtpatil: thanks04:24
tpatil# link : https://review.openstack.org/#/c/47305704:26
tpatilShilpa is working on implementing the last bits for this spec. All patches should be up for the review by end of this week04:26
samPtpatil: Include update's for spec?04:27
tpatilAlso I will request her to propose specs against Stein release04:28
ShilpaSDshilpa: tpatil: yes will do that04:28
samPShilpaSD: Thanks.04:29
*** Liang__ is now known as LiangFang04:29
*** LiangFang has quit IRC04:29
*** Liang__ has joined #openstack-meeting04:30
*** slaweq has joined #openstack-meeting04:30
*** Liang__ is now known as LiangFang04:30
samPAny other updates?04:30
*** jamesmcarthur has joined #openstack-meeting04:31
tpatilNo04:32
samP#topic AOB04:32
*** openstack changes topic to "AOB (Meeting topic: masakari)"04:32
tashiromtNothing from my side and I will check the new policy04:33
samPtpatil: tashiromt: thanks04:33
tpatilhttps://bugs.launchpad.net/masakari/+bug/180361004:33
openstackLaunchpad bug 1803610 in masakari "masakari installl docs are not published" [Undecided,New]04:33
tpatilThis bug is reported recently. I will check it and find out how we can improve installation doc04:34
samPtpatil: Thanks. Bug is very valid. We have some instructions on how to do that for devs but not for production04:35
samPtpatil: Could you please update the bug report with your progress or plans?04:38
tpatilsamP: Ok04:38
samPtpatil: thank you.04:38
samPAbout my personal schedule, I will be out of the country for new year vacation from 2018/12/29 - 2019/01/2004:39
samPDuring that time, I will try connect time to time at my best.04:40
samPI will be there for do the Stein-2 milestone release, which will be on 2019/1/10 week04:41
tpatilI will be on vacation as well from 2018/12/24 -2019/01/0704:41
samPDuring that time best way to contact me is send a email to sam47priya@gmail.com04:42
samPtpatil: got it.04:42
samPI will send these info to ML also.04:43
tashiromtsamP: OK, thanks04:43
samPif nothing else to share, I wold like to end the meeting.04:43
tpatilsamP: No more updates from my end04:43
samPtpatil: tashiromt: thank you04:45
samPThank you all for your time.04:45
samP#endmeeting04:45
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"04:45
openstackMeeting ended Tue Dec 11 04:45:22 2018 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)04:45
openstackMinutes:        http://eavesdrop.openstack.org/meetings/masakari/2018/masakari.2018-12-11-04.00.html04:45
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/masakari/2018/masakari.2018-12-11-04.00.txt04:45
openstackLog:            http://eavesdrop.openstack.org/meetings/masakari/2018/masakari.2018-12-11-04.00.log.html04:45
*** yamamoto has joined #openstack-meeting04:46
*** lpetrut has joined #openstack-meeting04:55
*** hongbin has quit IRC05:07
*** jamesmcarthur has quit IRC05:18
*** jamesmcarthur has joined #openstack-meeting05:18
*** jamesmcarthur has quit IRC05:23
*** lpetrut has quit IRC05:35
*** jamesmcarthur has joined #openstack-meeting05:50
*** _alastor_ has quit IRC05:50
*** tetsuro has quit IRC05:51
*** tetsuro has joined #openstack-meeting05:53
*** sridharg has joined #openstack-meeting06:09
*** _alastor_ has joined #openstack-meeting06:19
*** _alastor_ has quit IRC06:23
*** macza has joined #openstack-meeting06:28
*** macza has quit IRC06:32
*** sdake has quit IRC06:35
*** sdake has joined #openstack-meeting06:40
*** apetrich has joined #openstack-meeting06:40
*** bobh has joined #openstack-meeting06:41
*** rcernin has quit IRC06:43
*** bobh has quit IRC06:46
*** tpatil has quit IRC06:46
*** Luzi has joined #openstack-meeting06:49
*** jamesmcarthur has quit IRC06:50
*** cloudrancher has quit IRC06:53
*** tashiromt has quit IRC07:00
*** e0ne has joined #openstack-meeting07:10
*** e0ne has quit IRC07:12
*** e0ne has joined #openstack-meeting07:13
*** e0ne has quit IRC07:14
*** e0ne has joined #openstack-meeting07:15
*** e0ne has quit IRC07:17
*** bobh has joined #openstack-meeting07:25
*** bobh has quit IRC07:29
*** macza has joined #openstack-meeting07:30
*** macza has quit IRC07:34
*** sunnaichuan has joined #openstack-meeting07:41
*** trident has quit IRC07:42
*** trident has joined #openstack-meeting07:44
*** rossella_s has quit IRC07:46
*** bhagyashris_ has joined #openstack-meeting07:50
*** radeks has joined #openstack-meeting07:53
*** joxyuki has joined #openstack-meeting07:59
*** bobh has joined #openstack-meeting08:01
*** phuoc has joined #openstack-meeting08:02
*** longkb has joined #openstack-meeting08:02
*** sunnaichuan is now known as naichuans08:02
*** rossella_s has joined #openstack-meeting08:03
*** phuoc has left #openstack-meeting08:03
*** bobh has quit IRC08:05
*** bhagyashris_ has quit IRC08:10
*** yamahata has joined #openstack-meeting08:19
*** imacdonn has quit IRC08:22
*** imacdonn has joined #openstack-meeting08:23
*** _alastor_ has joined #openstack-meeting08:25
*** ralonsoh has joined #openstack-meeting08:29
*** _alastor_ has quit IRC08:30
*** tssurya has joined #openstack-meeting08:31
*** priteau has joined #openstack-meeting08:39
*** a-pugachev has joined #openstack-meeting08:40
*** rubasov has left #openstack-meeting08:43
*** bobh has joined #openstack-meeting08:51
*** bobh has quit IRC08:56
*** kopecmartin|off is now known as kopecmartin08:56
*** yamamoto has quit IRC09:01
*** yamamoto has joined #openstack-meeting09:18
*** lpetrut has joined #openstack-meeting09:21
*** yamamoto has quit IRC09:27
*** bobh has joined #openstack-meeting09:30
*** aojea has joined #openstack-meeting09:40
*** shrasool has joined #openstack-meeting09:56
*** samP has quit IRC10:00
*** yamamoto has joined #openstack-meeting10:06
*** samP has joined #openstack-meeting10:09
*** yamamoto has quit IRC10:10
*** e0ne has joined #openstack-meeting10:12
*** rossella_s has quit IRC10:21
*** joxyuki has quit IRC10:22
*** priteau has quit IRC10:24
*** shrasool has quit IRC10:26
*** electrofelix has joined #openstack-meeting10:28
*** LiangFang has quit IRC10:30
*** tetsuro has quit IRC10:33
*** rossella_s has joined #openstack-meeting10:35
*** shrasool has joined #openstack-meeting10:37
*** iyamahat has joined #openstack-meeting10:41
*** yamamoto has joined #openstack-meeting10:45
*** yamamoto has quit IRC10:47
*** sridharg has quit IRC10:54
*** tetsuro has joined #openstack-meeting10:59
*** erlon has joined #openstack-meeting11:07
*** sridharg has joined #openstack-meeting11:09
*** sridharg has quit IRC11:09
*** yamamoto has joined #openstack-meeting11:11
*** yamamoto has quit IRC11:16
*** yamamoto has joined #openstack-meeting11:16
*** rfolco has quit IRC11:18
*** sridharg has joined #openstack-meeting11:20
*** shrasool has quit IRC11:21
*** shrasool has joined #openstack-meeting11:22
*** rfolco has joined #openstack-meeting11:23
*** shrasool has quit IRC11:26
*** rossella_s has quit IRC11:28
*** longkb has quit IRC11:39
*** rossella_s has joined #openstack-meeting11:43
*** sridharg has quit IRC11:47
*** cloudrancher has joined #openstack-meeting11:50
*** sridharg has joined #openstack-meeting12:00
*** erlon has quit IRC12:01
*** cloudrancher has quit IRC12:01
*** macza has joined #openstack-meeting12:02
*** macza has quit IRC12:06
*** erlon has joined #openstack-meeting12:14
*** yamamoto has quit IRC12:23
*** macza has joined #openstack-meeting12:23
*** yamamoto has joined #openstack-meeting12:28
*** macza has quit IRC12:28
*** yamamoto has quit IRC12:32
*** tetsuro has quit IRC12:33
*** tetsuro has joined #openstack-meeting12:33
*** psachin has quit IRC12:35
*** naichuans has quit IRC12:41
*** jamesmcarthur has joined #openstack-meeting12:42
*** jamesmcarthur has quit IRC12:46
*** tssurya has quit IRC12:46
*** e0ne has quit IRC12:47
*** raildo has joined #openstack-meeting12:52
*** tetsuro has quit IRC12:54
*** yamamoto has joined #openstack-meeting13:05
*** jamesmcarthur has joined #openstack-meeting13:15
*** tssurya has joined #openstack-meeting13:21
*** macza has joined #openstack-meeting13:25
*** erlon has quit IRC13:28
*** macza has quit IRC13:29
*** macza has joined #openstack-meeting13:46
*** erlon has joined #openstack-meeting13:46
*** jamesmcarthur has quit IRC13:47
*** jamesmcarthur has joined #openstack-meeting13:48
*** a-pugachev has quit IRC13:49
*** macza has quit IRC13:50
*** rascasoft has joined #openstack-meeting13:52
*** jamesmcarthur has quit IRC14:02
*** jamesmcarthur has joined #openstack-meeting14:03
*** macza has joined #openstack-meeting14:06
*** mriedem has joined #openstack-meeting14:06
*** jamesmcarthur has quit IRC14:07
*** rossella_s has quit IRC14:08
*** rossella_s has joined #openstack-meeting14:09
*** macza has quit IRC14:10
*** psachin has joined #openstack-meeting14:11
*** shrasool has joined #openstack-meeting14:18
*** _alastor_ has joined #openstack-meeting14:24
*** rascasoft has quit IRC14:24
*** rossella_s has quit IRC14:24
*** awaugama has joined #openstack-meeting14:28
*** _alastor_ has quit IRC14:28
*** rossella_s has joined #openstack-meeting14:31
*** rossella_s has quit IRC14:39
*** e0ne has joined #openstack-meeting14:45
*** jawad_axd has joined #openstack-meeting14:46
*** rossella_s has joined #openstack-meeting14:46
*** jawad_axd has quit IRC14:51
*** jawad_axd has joined #openstack-meeting14:52
*** jawad_ax_ has joined #openstack-meeting14:56
*** jawad_axd has quit IRC14:56
*** eharney has joined #openstack-meeting15:00
*** jawad_ax_ has quit IRC15:00
*** markvoelker has joined #openstack-meeting15:00
*** psachin has quit IRC15:05
*** Luzi has quit IRC15:12
*** eharney_ has joined #openstack-meeting15:23
*** eharney has quit IRC15:26
*** eharney_ is now known as eharney15:27
*** ircuser-1 has joined #openstack-meeting15:27
*** munimeha1 has joined #openstack-meeting15:32
*** bobh has quit IRC15:32
*** wwriverrat has quit IRC15:33
*** bobh has joined #openstack-meeting15:38
*** bobh has quit IRC15:40
*** a-pugachev has joined #openstack-meeting15:45
*** markvoelker has quit IRC15:55
*** markvoelker has joined #openstack-meeting15:56
*** bobh has joined #openstack-meeting15:57
*** tpsilva has joined #openstack-meeting15:58
*** a-pugachev_ has joined #openstack-meeting15:58
slaweq#startmeeting neutron_ci16:00
openstackMeeting started Tue Dec 11 16:00:19 2018 UTC and is due to finish in 60 minutes.  The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot.16:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:00
slaweqhi16:00
*** openstack changes topic to " (Meeting topic: neutron_ci)"16:00
openstackThe meeting name has been set to 'neutron_ci'16:00
haleybhi16:00
*** mlavalle has joined #openstack-meeting16:00
*** a-pugachev has quit IRC16:00
*** markvoelker has quit IRC16:01
*** bobh has quit IRC16:01
* haleyb is in another meeting if he gets unresponsive16:01
mlavalleo/16:01
*** a-pugachev_ has quit IRC16:02
slaweqlets wait few more minutes for njohnston hongbin and others16:02
njohnstonwe may also want sean-k-mooney for this one but I don't think he is in channel16:03
*** a-pugachev has joined #openstack-meeting16:03
slaweqok, lets start16:04
slaweq#topic Actions from previous meetings16:04
*** openstack changes topic to "Actions from previous meetings (Meeting topic: neutron_ci)"16:04
slaweqmlavalle to change trunk scenario test and see if that will help with FIP issues16:04
mlavalleI pushed a patch yesterday16:05
mlavallehttps://review.openstack.org/#/c/624271/16:05
bcafarellate hi o/16:05
mlavalleI need to investigate what the results are16:05
slaweqlooks that this didn't solve problem: http://logs.openstack.org/71/624271/1/check/neutron-tempest-plugin-dvr-multinode-scenario/a125b91/testr_results.html.gz16:06
mlavalleyeah16:06
mlavalleI'll still take a closer look16:06
mlavalleand will continue investigating the bug in gneral16:07
slaweqmaybe there is some issue with timeouts there - those tests are using advaced image IIRC so it may be that it's trying ssh too short time16:07
slaweq?16:07
mlavallethe lifecycle scenario doesn't use advanced image16:08
slaweq#action mlavalle will continue debugging trunk tests failures in multinode dvr env16:08
slaweqahh, right - only one of tests is using adv image16:08
mlavalleyou know what on second thought we don't know if it worked16:08
mlavallebecasue the test case that failed was the other one16:09
mlavallenot the lifecycle one16:09
slaweqin this example both tests failed16:09
mlavalleok16:09
mlavalleI'll investigate16:09
*** jamesmcarthur has joined #openstack-meeting16:10
slaweqthx mlavalle16:10
slaweqlets move on16:10
slaweqnjohnston will remove neutron-grenade from neutron ci queues and add comment why definition of job is still needed16:10
njohnstonSo the feedback I got from the QA team is that they would rather we keep neutron-grenade, as they want to keep py2 grenade testing16:11
*** munimeha1 has quit IRC16:11
*** jawad_axd has joined #openstack-meeting16:11
njohnstonthey consider it part of the minimum level of testing needed until we officially stop supporting py216:11
slaweqok16:12
slaweqso we can consider this point from https://etherpad.openstack.org/p/neutron_ci_python3 as done, right?16:12
njohnstonyes16:13
njohnstonI was waiting for us to talk about it in the meeting before marking it16:13
slaweqI just marked it as done in etherpad then16:14
slaweqthx njohnston16:14
njohnstonthanks16:14
slaweqone more question16:14
slaweqis it only py2 based grenade job which QA wants still to have? or should we keep all grenade jobs with py2 too?16:15
slaweqdo You know?16:15
*** jawad_axd has quit IRC16:15
njohnstonThey want grenad eto cover both py2 and py3, so we should have both - the same way we have unit tests for both16:15
slaweqso we should "duplicate" all our grenade jobs then to have py2 and py3 variants for each16:16
slaweqprobably more rechecks but ok :)16:17
mlavalleLOL16:17
njohnstonSorry, I was not specific enough.  I think they want at least one grenade for py3 and py2 each.  I don't think we need a full matrix.16:17
njohnstonSo we should have grenade-py3 and neutron-grenade... but for example neutron-grenade-multinode-dvr could be just on py3 and they would be fine16:18
slaweqok, so we already have neutron-grenade (py2) and grenade-py3 (py3) jobs16:18
slaweqso we can just switch neutron-grenade-dvr-multinode and neutron-grenade-multinode to py3 now?16:19
njohnstonyes.  I proposed a 'grenade-multinode-py3' job in the grenade repo https://review.openstack.org/#/c/622612/16:19
*** hongbin has joined #openstack-meeting16:19
njohnstonI thought that we could use that perhaps, and then it becomes available for other projects16:20
slaweqok, now it's clear16:20
*** bobh has joined #openstack-meeting16:20
slaweqthx njohnston for working on this16:20
njohnstonnp16:20
slaweqok, lets move on then16:20
slaweqslaweq to continue debugging bug 179847516:20
openstackbug 1798475 in neutron "Fullstack test test_ha_router_restart_agents_no_packet_lost failing" [High,Confirmed] https://launchpad.net/bugs/179847516:20
slawequnfortunatelly I didn't have too much time to work on it last week16:20
slaweqI lost few days because of sick leave :/16:21
slaweqI will try to check it this week16:21
slaweq#action slaweq to continue debugging bug 179847516:21
openstackbug 1798475 in neutron "Fullstack test test_ha_router_restart_agents_no_packet_lost failing" [High,Confirmed] https://launchpad.net/bugs/179847516:21
slaweqnext one16:21
slaweqslaweq to continue fixing funtional-py3 tests16:21
mlavallefeeling better now?16:21
hongbino/16:21
slaweqmlavalle: yes, thx. It's much better16:22
slaweqhi hongbin16:22
hongbinslaweq: sorry, a bit late today16:22
slaweqso according to functional py3 tests, I was playing with it a bit during the weekend16:22
slaweqI tried to disable all warnings in python and so on but it still didn't help16:23
slaweqissue is probably caused by capturing stderr, like e.g.: http://logs.openstack.org/83/577383/17/check/neutron-functional/2907d2b/job-output.txt.gz#_2018-12-10_11_06_04_27239616:23
slaweqbut:16:24
slaweq1. I don't know how to get rid of it16:25
slaweq2. I'm not sure if that's good idea to get rid of it because I'm not sure if that comes from test which failed or from test which actually passed16:25
slaweqif anyone has any idea how to fix this issue - feel free to take it :)16:26
njohnstonI am at a loss for what the best course forward is16:26
slaweqif not I will assign it to myself for next week and will try to continue work on it16:26
slaweq#action slaweq to continue fixing funtional-py3 tests16:27
slaweqok, lets move on16:27
slaweqnjohnston to research py3 conversion for neutron grenade multinode jobs16:27
njohnstonI think we covered that before16:27
slaweqI think we alread talked about it :)16:27
slaweqyes, thx njohnston16:27
slaweqso next one16:28
slaweqslaweq to update etherpad with what is already converted to py316:28
slaweqI updated etherpad https://etherpad.openstack.org/p/neutron_ci_python3 today16:28
bcafarelon functional tests, maybe worth sending a ML post, maybe some other projects would have an idea there16:28
bcafarel(strange that it's only us getting hit by this "log limit")16:28
slaweqbcafarel: good idea, I will send email today16:28
slaweqbasiacally we still need to convert most of tempest jobs, grenade, rally and functional16:30
slaweqfor rally I proposed patch https://review.openstack.org/62435816:30
slaweqlets wait for results of CI now16:30
*** sean-k-mooney has joined #openstack-meeting16:30
slaweqso etherpad is updated, if someone wants to help, feel free to propose patches for jobs which are still waiting :)16:31
slaweqok, and the last action was:16:32
slaweqhongbin to report and check failing neutron.tests.fullstack.test_l3_agent.TestHAL3Agent.test_gateway_ip_changed test16:32
hongbini have to postponse this one since i am still figuring out how to setup the envirnoment for testing16:32
njohnstonsean-k-mooney: thanks for joining, we'll talk about CI issues in a moment16:33
sean-k-mooneynjohnston: no worries16:33
slaweqhongbin: ok, ping me if I You will need any help16:33
hongbinslaweq: thanks, will do16:33
slaweqI will assign it as an action for next week, ok?16:34
hongbinsure16:34
slaweq#action hongbin to report and check failing neutron.tests.fullstack.test_l3_agent.TestHAL3Agent.test_gateway_ip_changed test16:34
slaweqok, lets move on then16:34
slaweq#topic Python 316:34
*** openstack changes topic to "Python 3 (Meeting topic: neutron_ci)"16:34
slaweqwe already talked about grenade-jobs16:34
slaweqI only wanted to mention this patch for neutron-rally job: https://review.openstack.org/62435816:35
slaweqand also I sent today patch  https://review.openstack.org/624360 to remove tempest-full job as we have tempest-full-py3 already16:35
slaweqso I think that we don't need both of them16:36
*** e0ne has quit IRC16:36
slaweqanything else You want to talk about njohnston, bcafarel?16:36
njohnstonnope, I think that covers it16:37
bcafarelsame here16:37
slaweqok, so let's move on then16:37
slaweq#topic Grafana16:37
*** openstack changes topic to "Grafana (Meeting topic: neutron_ci)"16:37
slaweqhttp://grafana.openstack.org/dashboard/db/neutron-failure-rate16:37
njohnstonlooks like failure rate on neutron-tempest-iptables_hybrid job has gone from 8% at 0930UTC to 46% at 1620UTC16:38
njohnstonhttp://grafana.openstack.org/d/Hj5IHcSmz/neutron-failure-rate?orgId=1&panelId=18&fullscreen&from=now%2Fd&to=now%2Fd16:38
njohnstonsean-k-mooney was looking into it and how it might be related to pyroute216:38
slaweqnjohnston: yes, and I think that this is what sean-k-mooney has culprit for, right?16:38
njohnstonhttps://bugs.launchpad.net/os-vif/+bug/180794916:39
openstackLaunchpad bug 1807949 in os-vif "os_vif error: [Errno 24] Too many open files" [High,Triaged] - Assigned to sean mooney (sean-k-mooney)16:39
sean-k-mooneyso i that breaking on all build or jsut some16:39
njohnstonjust the neutron-tempest-iptables_hybrid it looks like16:39
slaweqsean-k-mooney: is it this error: http://logs.openstack.org/60/624360/1/check/neutron-tempest-iptables_hybrid/a6a4a0a/logs/screen-n-cpu.txt.gz?level=ERROR#_Dec_11_15_10_54_319285 ?16:41
njohnstonI am wondering if we should make neutron-tempest-iptables_hybrid non-voting while we figure this out, or blacklist this version of os-vif....16:41
haleybwe already had to blacklist 0.12.0...16:42
njohnstonbased on the 24-hour-rolling-average nature of grafana lines I think a rise this rapid means we may have an effective 100% failure rate at the moment16:42
slaweqsean-k-mooney: do You know why it may happen only in this job?16:42
slaweqI don't see any such error e.g. in tempest-full job logs (at least this which I'm checking now)16:43
*** sean-k-mooney has quit IRC16:43
njohnstonI did not see that error in the neutron-tempest-linuxbridge jobs I spot-checked16:43
njohnston(as another datapoint)16:44
slaweqthat is strange for me, the only thing which is "special" for neutron-tempest-iptables_hybrid is iptables_hybrid firewall driver instead of openvswitch driver16:46
slaweqhow this may trigger such error?16:46
mlavalleI think sean-k-mmoney is not on-line anymore16:49
slaweqok, I think that we should check if happens 100% times in this job, if so, we should, as njohnston said, mark this job as non-voting temporary and then try to investigate it16:49
*** sean-k-mooney has joined #openstack-meeting16:49
slaweqdo You agree?16:49
*** macza has joined #openstack-meeting16:49
mlavalleyes16:49
hongbin+116:49
njohnston+116:49
slaweqok, I will check tomorrow morning grafana and will send a patch to set as non-voting this job16:50
njohnstonshould we send something to the ML asking people not to recheck if the failure is in iptables_hybrid?16:50
slaweq#action slaweq to switch neutron-tempest-iptables_hybrid job as non-voting if it will be failing a lot because of bug 180794916:51
openstackbug 1807949 in os-vif "os_vif error: [Errno 24] Too many open files" [High,Triaged] https://launchpad.net/bugs/1807949 - Assigned to sean mooney (sean-k-mooney)16:51
sean-k-mooneyhi sorry got disconnected16:51
*** eharney has quit IRC16:51
slaweqnjohnston: yes, I will send an email16:51
bcafarelI think I just did :/ (though there was a rally timeout too)16:51
sean-k-mooneyill join the neutorn channel after to discuss the pyroute2 issue16:52
slaweqok, so sean-k-mooney - we will mark our job neutron-tempest-iptables_hybrid mark as non-voting if it will be failing 100% times becaise of this issue16:53
sean-k-mooneyok16:53
slaweqso we will have more time to investigate this :)16:53
sean-k-mooneythanks :)16:53
slaweqthx for helping with this :)16:54
slaweqok, lets move on16:54
slaweqtoday I went through our list of issues in https://etherpad.openstack.org/p/neutron-ci-failures16:54
slaweqand I wanted to find 3 which happens most often16:54
slaweqone of problems which hits as the most is still this issue in db migrations in functional tests:16:55
slaweqwhich happens many times16:55
slaweqand which is in my backlog16:55
slaweqbut maybe we should mark those tests as unstable for now?16:56
slaweqwhat do You think?16:56
bcafarelsounds reasonable, I did see this db migration issue a few times recently16:56
mlavalleyeah, I'm ok with that16:56
njohnstonit is a persistent bugaboo yes16:57
slaweqok, I will do that then16:57
mlavallewe will continue trying to fix it, right?16:57
slaweqmlavalle: of course16:57
mlavalleok16:57
slaweqI even have card for it in our trello, I just need some time16:57
mlavalleyeah, if it is getting in the way, let's mark it unstable16:57
slaweq#action slaweq to mark db migration tests as unstable for now16:57
mlavallethanks slaweq16:58
slaweqother issues which I found were:16:58
slaweq1. issues with cinder volume backup timeouts - I will try to ping cinder guys again with it16:58
slaweq2. various issues with FIP connectivity - it's not same test/job always, only common part is that ssh to fip is not working16:59
slaweqif someone wants to debug it more, I can send list of jobs which failed because of that :)16:59
mlavallesend it to me16:59
slaweqmlavalle: ok, thx16:59
slaweqwe have to finish now17:00
slaweqthx for attending guys17:00
slaweq#endmeeting17:00
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"17:00
openstackMeeting ended Tue Dec 11 17:00:15 2018 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)17:00
openstackMinutes:        http://eavesdrop.openstack.org/meetings/neutron_ci/2018/neutron_ci.2018-12-11-16.00.html17:00
slaweqo/17:00
bcafarelo/17:00
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/neutron_ci/2018/neutron_ci.2018-12-11-16.00.txt17:00
mlavalleI hope El Comandante shows some mercy during next week's meeting if I didn't fix it yet by then17:00
openstackLog:            http://eavesdrop.openstack.org/meetings/neutron_ci/2018/neutron_ci.2018-12-11-16.00.log.html17:00
sean-k-mooney /close17:00
*** sean-k-mooney has left #openstack-meeting17:00
slaweqmlavalle: no way :P17:01
mlavalleslaweq: LOL17:01
slaweqmlavalle: or maybe only little mercy because of Christmas :P17:02
mlavallethanks17:02
slaweqmlavalle: but I can't lost my reputation ;)17:02
mlavalleslaweq: that's true, you have to be consistently mean17:02
slaweqmlavalle: LOL17:02
*** rossella_s has quit IRC17:07
*** eharney has joined #openstack-meeting17:07
*** mlavalle has left #openstack-meeting17:07
*** tssurya has quit IRC17:08
*** jamesmcarthur has quit IRC17:13
*** kopecmartin is now known as kopecmartin|off17:16
*** yamamoto has quit IRC17:19
*** gyee has joined #openstack-meeting17:22
*** a-pugachev has quit IRC17:33
*** erus has joined #openstack-meeting17:33
*** ralonsoh has quit IRC17:46
*** lpetrut has quit IRC17:56
*** bobh has quit IRC17:56
*** gyee has quit IRC17:56
*** sridharg has quit IRC17:56
*** yamahata has quit IRC18:00
*** aojea has quit IRC18:01
*** iyamahat has quit IRC18:01
*** a-pugachev has joined #openstack-meeting18:09
*** Swami has joined #openstack-meeting18:09
*** e0ne has joined #openstack-meeting18:10
*** sridharg has joined #openstack-meeting18:13
*** brault has joined #openstack-meeting18:14
*** iyamahat has joined #openstack-meeting18:17
*** brault has quit IRC18:19
*** _alastor_ has joined #openstack-meeting18:22
*** cloudrancher has joined #openstack-meeting18:29
*** Shrews has joined #openstack-meeting18:31
*** iyamahat has quit IRC18:33
*** sridharg has quit IRC18:34
*** cloudrancher has quit IRC18:35
*** yamahata has joined #openstack-meeting18:36
*** rfolco is now known as rfolco_brb18:38
*** erus has quit IRC18:39
*** iyamahat has joined #openstack-meeting18:43
*** shrasool_ has joined #openstack-meeting18:43
*** Adri2000 has quit IRC18:43
*** shrasool has quit IRC18:45
*** shrasool_ is now known as shrasool18:45
*** mjturek has joined #openstack-meeting18:46
*** armax has joined #openstack-meeting18:47
*** Adri2000 has joined #openstack-meeting18:49
*** _alastor_ has quit IRC18:53
*** sridharg has joined #openstack-meeting18:58
*** _alastor_ has joined #openstack-meeting18:59
clarkbanyone here for the ifnra meeting?19:00
clarkbwe'll get started in a minute19:00
corvusoh, sorry, i was looking for the infra meeting19:00
clarkbyou've found it (you'll have to excuse my inability to type)19:01
ianwo/19:01
clarkb#startmeeting infra19:01
openstackMeeting started Tue Dec 11 19:01:37 2018 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.19:01
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.19:01
*** openstack changes topic to " (Meeting topic: infra)"19:01
openstackThe meeting name has been set to 'infra'19:01
gary_perkinsHi! o/19:01
clarkb#link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting19:02
clarkbmy phone reminded me to send the agenda out 24 hours in advance19:02
fungithanks again for curating the agenda and announcing it in advance!19:02
clarkbhopefully that will become useful as I do it regularly19:02
Shrewsyay phones19:02
clarkb#topic Announcements19:03
*** openstack changes topic to "Announcements (Meeting topic: infra)"19:03
clarkbLooking at a calendar Christmas and New Years fall on Tuesdays. I expect that means for many of us that the 18th will be our last meeting of the year and January 8th will be the first of 201919:04
clarkbI've gone ahead and written down that I don't expect we'll have meetings on December 25 or January 1.19:04
fungithose seem pretty likely19:05
fungii could be around for at least one of those, but best if i'm not19:05
clarkbI'll be traveling on the 20th to 28th myself so don't expect me around much for those two tuesdays :)19:05
clarkbtl;dr is everyone should enjoy their time off if they get it19:06
clarkb#topic Actions from last meeting19:06
*** openstack changes topic to "Actions from last meeting (Meeting topic: infra)"19:06
clarkb#link http://eavesdrop.openstack.org/meetings/infra/2018/infra.2018-12-04-19.01.txt minutes from last meeting19:06
clarkb#link https://review.openstack.org/622624 and parent/child changes provide initial opendev.org website content19:07
clarkbI think I've got that to a point where we can hopefully publish something19:07
clarkbhtml and css are not something you would seek me out for so feedback on that or followups to make it better much appreciated19:07
*** bobh has joined #openstack-meeting19:07
clarkb#link https://review.openstack.org/#/q/topic:inner-ara-results ianw and dmsimard still looking for review on nested ara for system-config ci jobs19:08
clarkb#link https://review.openstack.org/#/q/status:open+topic:fedora29 ianw looking for fedora29 (and associated networkmanager support in glean) change review19:08
clarkbthese two items were sort of an implied please review these changes action for the group19:08
ianwplease ... some interesting things are getting stuck behind it now19:08
clarkbI've gone through the stacks and they seem ready. Mostly just need sanity check from another infra-root (but non infra-root reviews/input also appreciated019:09
*** jamesmcarthur has joined #openstack-meeting19:09
clarkb#topic Specs19:09
*** openstack changes topic to "Specs (Meeting topic: infra)"19:10
clarkb#link https://review.openstack.org/623033 OpenDev Repo Hosting Rework Spec19:10
clarkb#link https://review.openstack.org/607377 Storyboard Attachements19:10
clarkb#link https://review.openstack.org/581214 Anomaly Detection in CI Logs19:10
clarkbthese are three infra specs that I expect will end up driving a bunch of work in the new year. As things wind down this year please take a look at them and provide input too19:10
clarkbI don't think any are ready for approval just yet, but eyeballs appreciated19:11
clarkbcorvus: any idea if you expect the opendev spec to be ready for approval soon?19:11
clarkbthe storyboard attachments spec is close.19:12
corvusclarkb: yes...19:12
*** sridharg has quit IRC19:12
* diablo_rojo_phon sneaks in at mention of storyboard from KubeCon19:12
fungidiablo_rojo_phon should mention storyboard at KubeCon19:13
corvusclarkb: mordred said to me "i bet we can run gitea with cephfs in kubernetes" and i've gone down a rabbit hole with that19:13
clarkbdiablo_rojo_phon: mostly just mentioning that the attachments spec appears near ready. Next week will be the last infra meeting so maybe we put it up for approval at that meeting then approve first week of 2019?19:13
corvusclarkb: i'm trying to see if i can drop in a fully-formed suggestion on how we could run gitea in the next revision.  if i fail, i'll just leave it hand-wavey on how we'll implement it.19:13
*** jamesmcarthur has quit IRC19:14
clarkbcorvus: ok19:14
corvusit's certainly runnable in the way we currently run things, so it's not a big deal.19:14
diablo_rojo_phonclarkb: I'll try to get jhesketh 's comments addressed before next meeting so we can get it merged hopefully before then19:14
clarkb#topic Priority Efforts19:14
*** openstack changes topic to "Priority Efforts (Meeting topic: infra)"19:14
fungicorvus: is gitea/other critical for moving forward on gerrit? what stops us from sticking with cgit until we get that sorted?19:14
clarkb#undo19:14
openstackRemoving item from minutes: #topic Priority Efforts19:14
fungisorry!19:14
clarkbfungi: I think its mostly people being allergic to the cgit web ui19:15
fungiwe can also probably just cover it in the review19:15
*** sridharg has joined #openstack-meeting19:15
corvusfungi: i'd like to know the whole canonical url story before we do that move19:15
clarkband ya gitea (or other tools) fit into ^19:15
fungigood point, though we _can_ make cgit coexist with clone urls from smart backend too19:15
corvusfungi: like, "okay, everyone change to git.opendev.org/openstack/nova".  "okay, now everyone change again to opendev.org/openstack/nova"19:15
corvusthat would be bad :)19:15
clarkb#topic Priority Efforts19:16
*** openstack changes topic to "Priority Efforts (Meeting topic: infra)"19:16
clarkb#topic Storyboard19:16
*** openstack changes topic to "Storyboard (Meeting topic: infra)"19:16
corvusit's possible that actual implementation is still separable, but i'd like to know the end state to be certain about that.19:16
clarkb(sorry for moving things along but I think we have a fairly full agenda and those details will likely come out of running it in test and spec review/updates, I do like the idea of knowing more about what it looks like though)19:17
*** yamamoto has joined #openstack-meeting19:17
clarkbdiablo_rojo_phon: is there a database migration etherpad for storyboard yet? I think I was supposed to dig one of the examples for gerrit?19:17
fungieumel8 is getting started on webclient translations19:17
fungi#link https://review.openstack.org/62350819:17
clarkbdiablo_rojo_phon: if you want to create even a stub etherpad url then fungi or myself can start adding info there that will probably help19:17
fungii posted a puppet change to get started on the db stuff last week, yeah. needs another reviewer19:17
fungii doubt we need a more detailed migration plan, after i dug into it19:18
fungi#link https://review.openstack.org/623290 Run a local MySQL service on StoryBoard servers19:18
clarkbfungi: meaning you think we can forego the etherpad? are you planning on driving that migration then ? (I mostly saw it as a way of communicating to $root what needed to be done)19:19
fungihonestly we just need to try it on the -dev server first, but outage while the db is dumped and loaded (it'll be under a minute) and then changing the database hostname in hiera19:19
clarkbgotcha19:19
fungiand then there's a followup change to fix backups afterward19:19
fungiit dawned on me that this is already how we do local and ci testing of storyboard19:20
fungithe puppet module defaults to this particular deployment model already19:20
clarkbsounds like we just need an infra root volunteer then?19:20
fungiwe've been overriding it in system-config so that we could do trove19:20
clarkb(test on -dev, then schedule short outage on production and do the switch tehre)19:21
fungii volunteer to do the rest of it unless someone else is interested, and then happy to help guide as needed19:21
clarkbgreat and thank you19:21
clarkbone thing that may end up being related is storyboard servers are running trusty. Perhaops this is an opportunity to upgrade to xenail (if we have to take an outage anyway)19:21
fungia great idea19:22
fungii think we already test against xenial too (for what testing sb does have anyway), but will double-check that19:22
clarkb#action fungi migrate to on host db server on storyboard instances unless someone else is interested19:22
fungiconfirmed, we puppet test its deployment on xenial already19:23
clarkb#topic Update Config Management19:23
*** openstack changes topic to "Update Config Management (Meeting topic: infra)"19:23
clarkbWe seem to have slowed down a bit around this with the other fires going on (slow gates, zuul updates to try and be more fair around that etc)19:24
clarkbmordred: Any progress on a script to generate a static config dynamically?19:24
fungii was talking to smarcet in #-infra earlier today about php versions and we concluded that this is likely a good time to upgrade the openstackid servers to xenial19:24
*** _alastor_ has quit IRC19:25
clarkbcorvus did end up deploying a new zuul executor with the static inventory in place. One thing that we learned from that is we don't get the post puppet reboot automatically anymore19:25
clarkbwhich for servers that use HWE kernels and/or AFS this may be required19:25
ianwclarkb: i saw a review for that19:25
fungi(though he needs the php version from bionic and is likely going to end up using a semi-official backport ppa of that on xenial for now)19:25
clarkbcorvus: other than that smallish hiccup it seems that working with static inventory isn't a big pain?19:25
corvusclarkb: no, pretty simple19:25
corvusi don't think the lack of a tool is a big problem right now;19:26
ianwclarkb: https://review.openstack.org/#/c/622964/ -- i had on my todo to run it before voting actually19:26
corvusi think a tiny change to launch-node to output the inventory snippet would be nice.19:26
*** electrofelix has quit IRC19:26
corvusor that19:26
clarkb#link https://review.openstack.org/#/c/622964/ dynamic generation of static inventory19:26
fungiyeah, we were chatting about that. soonish we're probably going to want to output a dns zone update snippet too so having that fits19:27
corvusi think both would be nice.19:27
clarkbcorvus: ++19:27
corvusoutput the snippet for a quick delta; script for full reconciliation.19:27
clarkbour ansible and puppetry do seem to be a fair bit more stable now too19:27
fungianybody know what our round-trip time is on a full run these days? still in the 30-45min range?19:28
clarkbhttp://grafana.openstack.org/d/qzQ_v2oiz/bridge-runtime?orgId=1 says just under 25 minutes? that seems low19:29
clarkboh except now that we don't have to do all the jinja2 expansion that may be accurate19:29
fungioh! i missed we were statsd'ing that19:29
clarkbfungi: ianw did the work on that19:29
fungithat's right, now i remember. thanks again ianw!~19:29
fungimost awesome19:30
clarkbany progress on docker things?19:30
clarkbianw: ^19:30
ianwin a word ... no19:30
corvusmordred updated the patch19:30
fungino progress is good progress? ;)19:30
clarkbcorvus: ah ok, I'll have to rereview it then19:30
*** lbragstad has quit IRC19:30
corvusi think the update addressed the review comments about testing iptables19:31
corvusit's now failing tests: https://review.openstack.org/60558519:31
clarkbseems to fail testing but we can dig into that19:31
*** lbragstad has joined #openstack-meeting19:31
clarkb#link https://review.openstack.org/#/c/605585 docker usage prep change. Fails tests currently but we should fix those tests and get this in when ready19:32
clarkbAnything else related to config management updates before we move into the general topics?19:32
clarkbfungi: you did end up helping cmurphy with the kata lists puppet update?19:32
clarkbif that happened we can likely continue rolling up the puppet4 parser list19:32
fungiyes, it merged, nothing changed, we're good there19:33
clarkbcool I'll have to take a look and see if I can help get any more of the future parser changes in19:33
fungimore evidence of cmurphy's awesomeness19:33
clarkb#topic General Topics19:34
*** openstack changes topic to "General Topics (Meeting topic: infra)"19:34
clarkbStarting with OpenDev just a reminder to check out corvus' spec around git hosting for opendev and my website content change that were noted earlier in the meeting19:34
clarkbAlso considering this has become a standing topic and is a big effort I'd like to make this a priority effort (I think this was mentioned before)19:34
clarkbI'll push up changes for that later today, but if you have objections to doing that feel free to let me/us know here or PM/email me19:35
clarkbcorvus: I figure we can use your spec for that data recording on the specs repo side?19:35
*** jawad_axd has joined #openstack-meeting19:35
corvusclarkb: i'm having trouble parsing that19:35
clarkb(since it gives a concrete set of work to prioritize)19:35
corvusclarkb: use my spec for indicating opendev is priority effort?  yes19:36
clarkbcorvus: we list priority efforts as being attached to specs. Rather than write a nebulous opendev spec I like the idea of attaching priority to concrete work items (as in your spec)19:36
clarkbgreat19:36
fungii rather think opendev is what we're doing. specs related to that are priority efforts on their own19:37
fungiand corvus's gerrit/git spec already proposes itself as such19:37
clarkbyup19:37
cmurphyo/19:37
clarkbNext on the list of general topics was I wanted to remind people that upgrading trusty servers is still valuable. I know we've got a few irons in the fire for this between upgrading to xenial on puppet, running ansible only servers as with dns servers, and switching some service to docker19:38
cmurphythis could use a recheck when an infra-root has time to babysit https://review.openstack.org/61565619:38
clarkbcmurphy: noted thanks19:38
*** sridharg has quit IRC19:38
clarkb#link https://etherpad.openstack.org/p/201808-infra-server-upgrades-and-cleanup has a list of servers that need attention if you can grab one or two please update that list19:39
fungiclarkb: thanks. i think as mentioned earlier in the meeting the sb and openstackid servers are good next candidates19:39
clarkbI'm hoping to get to pbx this week19:39
clarkbfungi: yup19:39
fungiso maybe those drop off the list rsn19:39
*** munimeha1 has joined #openstack-meeting19:39
clarkbWe have until roughly april to get off trusty. Xenial will eol in 2021 so even Xenial gives us a bit of breathing room19:39
clarkb(then bionic will have 10 years of support)19:40
*** jawad_axd has quit IRC19:40
clarkbThe last item on the general topics list has to do with how we use github. Specially admin account setup19:41
clarkbour documents have long said we should use a secondary admin only github account, but I don't believe we ever enforced that19:41
clarkbAdditionally gentoo found (the hard way) that two factor auth with github reduces its ability to be a shiny target for not so nice people19:41
clarkb#link https://review.openstack.org/#/c/620702/19:42
clarkb#link https://review.openstack.org/#/c/620703/19:42
clarkbI proposed two changes to kick off some discussion on this. Basically would we be opposed to requiring github 2fa on those accounts and should we enforce that we set up a second admin account or relax that rule given practice?19:42
fungitwo-factor auth saved gentoo from github compromise? or they merely speculate that _if_ they'd required it they wouldn't have had their accounts hacked?19:43
clarkbfungi: the speculation is that 2fa would have prevented the compromise19:43
clarkband they now require it for all of their github accounts19:43
fungii assumed so, your earlier point merely seemed to imply the other19:43
*** jamesmcarthur has joined #openstack-meeting19:43
clarkbI've run with a hardware token second factor auth on github for about a year now and haven't found ti to be particularly troublesome. It helps that much of the day to day interaction with github is via ssh (whcih doesn't 2fa)19:44
clarkbthey support totp whcih you can run generators for on android or via linux command line as well19:44
fungirelated to this, while the hope under new opendevery is that we can stop caring about needing github credentials for openstack and similar orgs, we do still apparently need them to manage the zuul app/widget/thingy19:45
ianwpersonally i barely use it and would be happy with a shared account19:45
clarkbya we might manage fewer repo mirrors, but would still have things like the zuul app install in github19:45
ianwbut, if the key for the 2fa is with the password, it's not much of a 2fa i guess19:45
clarkbianw: thats a good point. I think ssh is limited in the scope of what it can do, but still dangerous :)19:46
corvusssh is arguably 2 factor.  it's at least 1.5.  depends on how you're characterizing threat vectors (turs out "factors" aren't really one-size-fits-all)19:47
corvusi should say, ssh with a password-protected key19:47
corvus(which should go without saying, but...)19:47
*** jamesmcarthur has quit IRC19:47
corvusanyway, that's beside the point :)19:48
ianwif we consider ssh 2fa as commutative to github 2fa then i'd probably vote for the shared account with a setup on bridge to get the totp token19:48
clarkbianw: oh thats another possibility too. Which is that we use a shared account19:48
fungiif someone compromises bridge we've got bigger problems on our hands than github creds anyway19:49
clarkbpossibilities: Keep using personal account, use new second personal admin only accounts, use shared admin account. * apply 2fa19:49
corvusif we want 2fa on the github accounts, i'll definitely need a second github account.19:49
ianwit's just github does a pretty bad job at separating things, i think.  which means if you have openstack in your account it pollutes all your notifications etc if you use it for "real" work19:49
clarkbianw: yup, so a second account may be desireable anyway19:50
ianwi can take an action item to make a shared account, and see if there's a practical way to have the token working on bridge.  if it all looks good, we can then give it permission19:50
clarkbianw: do you have a preference to shared account over second personal account? I guess organizationally its less for each individual to manage19:51
fungii'm happy to drop all privs for my personal gh account i use to submit the occasional pr or file an issue on some random project, and use a shared account on bridge.o.o (perhaps with a 2fa tool installed there)19:51
clarkb#action ianw investigate practicality of shared github account (with possible 2fa)19:51
*** macza has quit IRC19:51
ianwi'm thinking if i'm setting up a second personal account, it gets confusing as to which one's me, and then i also have to manage two github secrets etc19:52
clarkbianw: ya19:52
fungialso, worth noting, we've been generally sloppy about remembering to add infra-root folk to the orgs we have in gh, or remove the ones who retire19:52
ianwconsidering how much i use it (never) shared account seems better imo19:52
clarkbfungi: thats a good point too. I like the idea of shared creds the more we talk about it19:52
clarkbthis seems like a great next step, thank you ianw for volunteering to check it out19:52
*** macza has joined #openstack-meeting19:53
clarkband with that I'll open the floor19:53
clarkb#topic Open Discussion19:53
*** openstack changes topic to "Open Discussion (Meeting topic: infra)"19:53
clarkbcorvus: I saw these at powells the other day https://www.powells.com/book/-65954922024219:55
clarkbif we're left with book shopping things then I should probably end the meeting19:56
corvusha nice!19:56
clarkbThank you everyone. Reminder we'll meet next week then take a two week break for holidays.19:56
clarkbFind us in the infra channel or on the infra mailing list if you have other questions, concerns, thoughts issues, etc19:56
clarkb#endmeeting19:56
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"19:57
openstackMeeting ended Tue Dec 11 19:56:59 2018 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)19:57
openstackMinutes:        http://eavesdrop.openstack.org/meetings/infra/2018/infra.2018-12-11-19.01.html19:57
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/infra/2018/infra.2018-12-11-19.01.txt19:57
openstackLog:            http://eavesdrop.openstack.org/meetings/infra/2018/infra.2018-12-11-19.01.log.html19:57
fungithanks clarkb!19:57
*** Shrews has left #openstack-meeting19:58
*** erlon has quit IRC20:04
*** b1air has joined #openstack-meeting20:05
*** sambetts_ has quit IRC20:14
*** yamamoto has quit IRC20:19
*** mriedem has quit IRC20:22
*** bobh has quit IRC20:23
*** mriedem has joined #openstack-meeting20:24
*** bobh has joined #openstack-meeting20:26
*** a-pugachev has quit IRC20:26
*** a-pugachev has joined #openstack-meeting20:26
*** eharney has quit IRC20:34
*** shrasool has quit IRC20:34
*** shrasool has joined #openstack-meeting20:35
*** jawad_axd has joined #openstack-meeting20:38
*** jawad_axd has quit IRC20:42
*** a-pugachev_ has joined #openstack-meeting20:45
*** janders has joined #openstack-meeting20:47
*** a-pugachev has quit IRC20:47
*** a-pugachev_ is now known as a-pugachev20:47
*** eharney has joined #openstack-meeting20:49
*** bobh has quit IRC20:56
*** rfolco_brb has quit IRC20:57
*** jawad_axd has joined #openstack-meeting20:58
*** martial__ has joined #openstack-meeting21:00
martial__Hello team21:00
jandersg'day21:01
martial__We have a scientific sig meeting scheduled21:01
martial__With an open agenda21:01
martial__#startmeeting Scientific-sig21:01
openstackMeeting started Tue Dec 11 21:01:46 2018 UTC and is due to finish in 60 minutes.  The chair is martial__. Information about MeetBot at http://wiki.debian.org/MeetBot.21:01
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.21:01
*** openstack changes topic to " (Meeting topic: Scientific-sig)"21:01
openstackThe meeting name has been set to 'scientific_sig'21:01
martial__Opening the floor21:02
*** jawad_axd has quit IRC21:03
janderswho's around today? It seems like things are getting quiet moving towards xmas..21:04
martial__it is indeed usually the case21:04
martial__Stig and Blair are not able to join, so I offered to run an open meeting21:04
jandersgreat! :)21:04
martial__if there is a topic you want to cover or info you want to share, please let us know21:05
jandersI don't have a whole lot of stuff to update on - working on training & procurement21:05
jandersone thing I've been thinking of / looking into recently is BeeGFS - OpenStack integration21:05
jandersit seems to me it's a very promising area, but with little work done so far21:06
jandersdo you have any experience or thoughts on this?21:06
martial__I know this has been discussed recently in the channel21:07
martial__let me find the link21:07
martial__BeeGFS [...] met them at SC18. Them and OpenIP seemed interesting to look into, so putting it out here https://www.enterprisetech.com/2018/11/29/move-over-lustre-spectrum-scale-here-comes-beegfs/21:07
jandersthat's right - we (CSIRO) even got a mention :)21:08
martial__not sure if this link was taken from a Scientific SIG conversation21:08
martial__looks like it was from here then :)21:08
martial__not much more from my end on this subject yet21:09
janderswould you see value in going down this path in the future?21:10
jandersfrom my perspective, BeeGFS is set to be Ceph equivalent for High Performance Cloud workloads21:10
jandersit's an opportunity, but whether it translates into success depends on thinkparq's interest and update from people like us21:10
janderss/update/uptake21:11
martial__not sure, Stig mentioned something in ... October about doing some test21:11
martial__Oct 2nd log `We did a little testing on resource footprint with hyperconverged (non-containerised) on bare metal.  I think the containers could work.` `On the extreme bandwidth topic, one of our team has been getting really great numbers from BeeGFS at scale recently.  Servers with NVME and dual OPA exceeding 21GBytes/s for reads.  Yeee-ha`21:12
jandersregarding our BeeGFS work - I can't say too much yet (will be happy to share when it's complete) but we're somewhere around 80% efficiency in terms of BeeGFS being able to utilise the performance of the underlying hardware21:12
jandersI think we're roughly around the same mark - which is not unexpected given we're using very similar hardware21:13
martial__seems pretty good indeed21:14
*** shrasool has quit IRC21:14
jandersthe guys in Cambridge are doing excellent work trailblazing on the topic of ultra-high-performance BeeGFS, we're following closely here at CSIRO :)21:14
jandersI believe there'll be Cambridge-DELL whitepaper on the topic out soon, looking forward to reading it21:15
jandersok - so that's really the only thing I had - other than BeeGFS is all training and procurement21:15
janders80% of my team is away doing a mix of different courses, so not much engineering happening21:16
jandershow are things on your end Martial?21:17
*** bobh has joined #openstack-meeting21:18
martial__doing good, working on a K8s deplpoyment over kubespray on top of OpenStack kolla-ansible21:19
*** bobh has quit IRC21:19
*** bobh has joined #openstack-meeting21:19
jandersI don't think I've looked at kubespray before - quickly checked it out, looks very cool21:20
jandersif it's k8s + ansible I already like it :)21:21
martial__very simple to deploy a k8s21:21
martial__so if you have an OpenStack project you have a private network21:21
martial__and you just deploy a private k8s on that private network21:21
*** bobh has quit IRC21:21
martial__very cool for quick testing21:22
martial__and prototyping21:22
jandersexcellent!21:22
martial__(and yes there is an kubernetes-ansible :) )21:22
janderscan it be made work for larger scale production deployments, or is it focusing specifically on PoC/prototyping?21:22
martial__you can make large deployments with it21:22
jandershttps://github.com/kubernetes/contrib/tree/master/ansible21:23
martial__OpenStack-Kolla+O-K-Ansible is how we are starting to move our OpenStack deplopyments21:23
jandersKubernetes Ansible DEPRECATED: See repo https://github.com/kubernetes-incubator/kubespray21:23
jandersam I looking at the right repo?21:23
jandersthings are moving so quickly in this field it's sometimes hard to keep track21:24
martial__weird, we are using it in prod21:24
jandersfrom this it seems k8s-ansible turned into kubespray - does this sound possible, or am I getting confused? :)21:24
martial__check https://github.com/kubernetes-sigs/kubespray21:24
jandersis openstack-kolla using lxc or docker for the control plane containers?21:26
martial__docker21:26
jandersgreat!21:26
*** shrasool has joined #openstack-meeting21:26
jandersare openstack-kolla and kolla-ansible very different?21:27
jandersor are these different pieces to the same puzzle?21:27
martial__kolla creates the containers21:31
martial__kolla-ansible is for deployment of said containers21:31
martial__(sorry a couple people in the room asking me questions)21:31
*** shrasool has quit IRC21:32
jandersok - so same pieces to the puzzle :)21:33
jandersare you running your control plane containers on docker/standalone or k8s?21:33
janderswe'll likely be standalone in the first iteration of the production system, but it's an interesting one - I wonder how others do it21:34
martial__we do docker standalone21:35
martial__once I have a project, I deploy a k8s using kubespray on created VMs21:35
jandersdo you reqiure multi-tenancy in the k8s layer?21:36
martial__not for the experiment I am running21:36
martial__one project, one user21:36
jandersunderstood21:37
jandersI had some interesting discussions with Red Hat guys in Berlin about multi-tenant k8s/ocp on OpenStack which I need to follow up21:37
jandersRocky brings improvements in this field but IMHO it's still lacking a couple features before being ready to be used in anger21:38
jandersKuryr seems to be finally doing something useful, but it's lacking mapping of OpenStack tenants to k8s tenants AFAIK21:38
jandersso I think taking a focused approach like yours has potential of saving quite a few headaches :)21:39
martial__works for my tailored need indeed :)21:39
martial__seems that we managed to talk about a few topics here today after all :)21:41
martial__anything else to cover?21:41
jandersindeed! :)21:41
jandersno, not really21:42
jandersI've got a meeting at 900AEDT so happy to wrap up early21:42
martial__otherwise I invite us to save 20 minutes ;)21:42
jandersthank you Martial!21:42
jandershave a good day mate21:42
*** eharney has quit IRC21:42
martial__thanks Janders21:42
martial__you as well21:42
martial__#endmeeting21:42
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"21:42
openstackMeeting ended Tue Dec 11 21:42:40 2018 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)21:42
openstackMinutes:        http://eavesdrop.openstack.org/meetings/scientific_sig/2018/scientific_sig.2018-12-11-21.01.html21:42
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/scientific_sig/2018/scientific_sig.2018-12-11-21.01.txt21:42
openstackLog:            http://eavesdrop.openstack.org/meetings/scientific_sig/2018/scientific_sig.2018-12-11-21.01.log.html21:42
*** janders has quit IRC21:42
*** jamesmcarthur has joined #openstack-meeting21:44
*** markvoelker has joined #openstack-meeting21:45
*** e0ne has quit IRC21:47
*** jamesmcarthur_ has joined #openstack-meeting21:48
*** markvoelker has quit IRC21:49
*** jamesmcarthur has quit IRC21:50
*** rcernin has joined #openstack-meeting21:59
*** jawad_axd has joined #openstack-meeting22:00
*** bobh has joined #openstack-meeting22:03
*** jawad_axd has quit IRC22:04
*** EmilienM has quit IRC22:07
*** EmilienM has joined #openstack-meeting22:08
*** b1air has quit IRC22:15
*** radeks has quit IRC22:18
*** awaugama has quit IRC22:21
*** markvoelker has joined #openstack-meeting22:24
*** slaweq has quit IRC22:26
*** jamesmcarthur_ has quit IRC22:33
*** jamesmcarthur has joined #openstack-meeting22:35
*** jamesmcarthur has quit IRC22:35
*** jamesmcarthur has joined #openstack-meeting22:35
*** jamesmcarthur has quit IRC22:36
*** jamesmcarthur has joined #openstack-meeting22:37
*** bobh has quit IRC22:38
*** bobh has joined #openstack-meeting22:38
*** raildo has quit IRC22:39
*** jamesmcarthur has quit IRC22:40
*** bobh has quit IRC22:41
*** _alastor_ has joined #openstack-meeting22:41
*** jamesmcarthur has joined #openstack-meeting22:42
*** slaweq has joined #openstack-meeting22:43
*** bobh has joined #openstack-meeting22:44
*** jamesmcarthur has quit IRC22:46
*** slaweq has quit IRC22:48
*** a-pugachev has quit IRC22:49
*** mjturek has quit IRC22:51
*** shrasool has joined #openstack-meeting22:55
*** yamamoto has joined #openstack-meeting22:55
*** bobh has quit IRC23:05
*** bobh has joined #openstack-meeting23:06
*** a-pugachev has joined #openstack-meeting23:09
*** mjturek has joined #openstack-meeting23:10
*** slaweq has joined #openstack-meeting23:14
*** _alastor_ has quit IRC23:17
*** slaweq has quit IRC23:19
*** jamesmcarthur has joined #openstack-meeting23:22
*** jawad_axd has joined #openstack-meeting23:23
*** jamesmcarthur has quit IRC23:26
*** jawad_axd has quit IRC23:27
*** hongbin has quit IRC23:34
*** hongbin has joined #openstack-meeting23:35
*** slaweq has joined #openstack-meeting23:35
*** slaweq has quit IRC23:40
*** armax has quit IRC23:42
*** jawad_axd has joined #openstack-meeting23:44
*** jawad_axd has quit IRC23:49
*** dklyle has joined #openstack-meeting23:51
*** erlon has joined #openstack-meeting23:56
*** hongbin has quit IRC23:58

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!