Tuesday, 2019-10-15

*** e0ne has joined #openstack-meeting00:05
*** brault has quit IRC00:16
*** jamesmcarthur has quit IRC00:16
*** e0ne has quit IRC00:18
*** brault has joined #openstack-meeting00:18
*** jamesmcarthur has joined #openstack-meeting00:21
*** armax has quit IRC00:26
*** ociuhandu has joined #openstack-meeting00:31
*** e0ne has joined #openstack-meeting00:31
*** ociuhandu has quit IRC00:35
*** njohnston has quit IRC00:35
*** njohnston has joined #openstack-meeting00:36
*** e0ne has quit IRC00:40
*** e0ne has joined #openstack-meeting00:43
*** e0ne has quit IRC00:47
*** e0ne has joined #openstack-meeting00:54
*** e0ne has quit IRC00:58
*** jamesmcarthur has quit IRC00:58
*** jamesmcarthur has joined #openstack-meeting01:03
*** e0ne has joined #openstack-meeting01:05
*** bnemec has joined #openstack-meeting01:12
*** e0ne has quit IRC01:12
*** e0ne has joined #openstack-meeting01:18
*** e0ne has quit IRC01:22
*** bnemec has quit IRC01:25
*** e0ne has joined #openstack-meeting01:28
*** brinzhang_ has joined #openstack-meeting01:30
*** lseki has quit IRC01:30
*** brinzhang has quit IRC01:33
*** jamesmcarthur has quit IRC01:34
*** e0ne has quit IRC01:36
*** jamesmcarthur has joined #openstack-meeting01:41
*** e0ne has joined #openstack-meeting01:42
*** bnemec has joined #openstack-meeting01:42
*** brinzhang has joined #openstack-meeting01:46
*** e0ne has quit IRC01:46
*** brinzhang_ has quit IRC01:49
*** brinzhang_ has joined #openstack-meeting01:51
*** e0ne has joined #openstack-meeting01:51
*** mhen has quit IRC01:54
*** brinzhang has quit IRC01:54
*** Garyx has quit IRC01:55
*** Garyx has joined #openstack-meeting01:56
*** e0ne has quit IRC02:00
*** jamesmcarthur has quit IRC02:02
*** Garyx has quit IRC02:05
*** jamesmcarthur has joined #openstack-meeting02:05
*** e0ne has joined #openstack-meeting02:06
*** Garyx has joined #openstack-meeting02:07
*** apetrich has quit IRC02:09
*** dtrainor has quit IRC02:12
*** e0ne has quit IRC02:14
*** dtrainor has joined #openstack-meeting02:15
*** e0ne has joined #openstack-meeting02:17
*** ianychoi has quit IRC02:19
*** ianychoi has joined #openstack-meeting02:21
*** e0ne has quit IRC02:22
*** e0ne has joined #openstack-meeting02:23
*** e0ne has quit IRC02:27
*** brinzhang has joined #openstack-meeting02:28
*** e0ne has joined #openstack-meeting02:30
*** brinzhang_ has quit IRC02:31
*** markvoelker has joined #openstack-meeting02:32
*** brinzhang has quit IRC02:37
*** brinzhang has joined #openstack-meeting02:38
*** ricolin has joined #openstack-meeting02:40
*** e0ne has quit IRC02:41
*** e0ne has joined #openstack-meeting02:44
*** e0ne has quit IRC02:49
*** markvoelker has quit IRC02:52
*** radeks has joined #openstack-meeting02:52
*** e0ne has joined #openstack-meeting02:55
*** radeks has quit IRC02:58
*** e0ne has quit IRC03:00
*** rbudden has quit IRC03:25
*** jamesmcarthur has quit IRC03:26
*** dviroel_ has quit IRC03:31
*** slaweq has joined #openstack-meeting03:31
*** hongbin has joined #openstack-meeting03:35
*** slaweq has quit IRC03:35
*** radeks has joined #openstack-meeting03:36
*** psachin has joined #openstack-meeting03:37
*** brinzhang_ has joined #openstack-meeting03:53
*** brinzhang has quit IRC03:56
*** tashiromt has joined #openstack-meeting04:01
*** brault has quit IRC04:19
*** hongbin has quit IRC04:22
*** nitinuikey has joined #openstack-meeting04:29
*** larainema has joined #openstack-meeting04:30
*** brinzhang has joined #openstack-meeting04:31
*** brinzhang_ has quit IRC04:34
*** radeks has quit IRC04:48
*** diablo_rojo has quit IRC04:48
*** brinzhang_ has joined #openstack-meeting04:50
*** brinzhang has quit IRC04:53
*** radeks has joined #openstack-meeting04:57
*** brinzhang has joined #openstack-meeting04:59
*** keiko-k has joined #openstack-meeting05:00
*** brinzhang_ has quit IRC05:03
*** Luzi has joined #openstack-meeting05:16
*** hyunsikyang__ has joined #openstack-meeting05:17
*** hyunsikyang has quit IRC05:20
*** hyunsikyang has joined #openstack-meeting05:21
*** hyunsikyang__ has quit IRC05:22
*** brinzhang_ has joined #openstack-meeting05:28
*** e0ne has joined #openstack-meeting05:30
*** brinzhang has quit IRC05:31
*** markvoelker has joined #openstack-meeting05:47
*** takamatsu has quit IRC05:54
*** e0ne has quit IRC05:58
*** links has joined #openstack-meeting06:04
*** brinzhang has joined #openstack-meeting06:07
*** slaweq has joined #openstack-meeting06:09
*** igordc has quit IRC06:10
*** brinzhang_ has quit IRC06:10
*** nitinuikey has quit IRC06:17
*** brinzhang_ has joined #openstack-meeting06:17
*** brinzhang has quit IRC06:20
*** pcaruana has joined #openstack-meeting06:30
*** vesper11- has quit IRC06:33
*** vesper11 has joined #openstack-meeting06:36
*** brinzhang has joined #openstack-meeting06:40
*** brinzhang has quit IRC06:40
*** brinzhang_ has quit IRC06:44
*** trident has quit IRC06:52
*** trident has joined #openstack-meeting06:56
*** Luzi_ has joined #openstack-meeting07:04
*** Luzi has quit IRC07:08
*** jawad_axd has joined #openstack-meeting07:12
*** tesseract has joined #openstack-meeting07:12
*** rcernin has quit IRC07:16
*** e0ne has joined #openstack-meeting07:25
*** apetrich has joined #openstack-meeting07:26
*** ttsiouts has joined #openstack-meeting07:26
*** ralonsoh has joined #openstack-meeting07:28
*** ociuhandu has joined #openstack-meeting07:30
*** ociuhandu has quit IRC07:35
*** ociuhandu has joined #openstack-meeting07:35
*** kopecmartin|off is now known as kopecmartin07:38
*** ociuhandu has quit IRC07:42
*** priteau has joined #openstack-meeting07:53
*** e0ne has quit IRC07:55
*** tashiromt has quit IRC08:00
*** nitinuikey has joined #openstack-meeting08:04
*** tpatil has joined #openstack-meeting08:04
*** rpittau|afk is now known as rpittau08:07
*** ttsiouts has quit IRC08:12
*** ttsiouts has joined #openstack-meeting08:12
*** ociuhandu has joined #openstack-meeting08:13
*** ttsiouts has quit IRC08:17
*** e0ne has joined #openstack-meeting08:17
*** ttsiouts has joined #openstack-meeting08:17
*** radeks has quit IRC08:19
*** ociuhandu has quit IRC08:20
*** takahashi-tsc has joined #openstack-meeting08:22
*** nitinuikey has quit IRC08:25
*** ociuhandu has joined #openstack-meeting08:25
*** e0ne_ has joined #openstack-meeting08:25
*** takamatsu has joined #openstack-meeting08:26
*** e0ne has quit IRC08:26
*** takamatsu has quit IRC08:29
*** jamesmcarthur has joined #openstack-meeting08:30
*** ociuhandu has quit IRC08:31
*** ociuhandu has joined #openstack-meeting08:31
*** jamesmcarthur has quit IRC08:34
*** ociuhandu has quit IRC08:35
*** ociuhandu has joined #openstack-meeting08:35
*** takamatsu has joined #openstack-meeting08:36
*** ttsiouts has quit IRC08:37
*** ttsiouts has joined #openstack-meeting08:37
*** tpatil has quit IRC08:40
*** takamatsu has quit IRC08:41
*** ttsiouts has quit IRC08:42
*** ociuhandu has quit IRC08:42
*** tssurya has joined #openstack-meeting08:45
*** takamatsu has joined #openstack-meeting08:48
*** ttsiouts has joined #openstack-meeting08:49
*** bnemec has quit IRC08:53
*** e0ne_ has quit IRC09:09
*** brinzhang has joined #openstack-meeting09:23
*** ttsiouts has quit IRC09:24
*** ttsiouts has joined #openstack-meeting09:24
*** ttsiouts has quit IRC09:29
*** ociuhandu has joined #openstack-meeting09:47
*** ociuhandu has quit IRC09:53
*** brinzhang_ has joined #openstack-meeting10:04
*** brinzhang has quit IRC10:08
*** Luzi has joined #openstack-meeting10:08
*** Luzi_ has quit IRC10:10
*** ttsiouts has joined #openstack-meeting10:11
*** njohnston_ has joined #openstack-meeting10:17
*** e0ne has joined #openstack-meeting10:20
*** ttsiouts has quit IRC10:27
*** ttsiouts has joined #openstack-meeting10:28
*** ociuhandu has joined #openstack-meeting10:31
*** brinzhang has joined #openstack-meeting10:33
*** ttsiouts has quit IRC10:33
*** markvoelker has quit IRC10:33
*** brinzhang_ has quit IRC10:36
*** ociuhandu has quit IRC10:36
*** njohnston_ has quit IRC10:38
*** bbowen has quit IRC10:47
*** hyunsikyang__ has joined #openstack-meeting10:53
*** hyunsikyang has quit IRC10:56
*** takamatsu has quit IRC10:56
*** njohnston_ has joined #openstack-meeting10:57
*** keiko-k has quit IRC11:00
*** hyunsikyang has joined #openstack-meeting11:01
*** njohnston_ has quit IRC11:02
*** hyunsikyang__ has quit IRC11:06
*** brinzhang_ has joined #openstack-meeting11:07
*** brinzhang has quit IRC11:09
*** takahashi-tsc has quit IRC11:14
*** ttsiouts has joined #openstack-meeting11:18
*** markvoelker has joined #openstack-meeting11:19
*** takamatsu has joined #openstack-meeting11:34
*** brinzhang has joined #openstack-meeting11:38
*** ttsiouts has quit IRC11:38
*** ttsiouts has joined #openstack-meeting11:39
*** markvoelker has quit IRC11:39
*** brinzhang_ has quit IRC11:41
*** ttsiouts has quit IRC11:43
*** ttsiouts has joined #openstack-meeting11:44
*** takamatsu has quit IRC11:51
*** raildo has joined #openstack-meeting12:02
*** bbowen has joined #openstack-meeting12:04
*** brinzhang_ has joined #openstack-meeting12:04
*** brinzhang has quit IRC12:07
*** brinzhang_ has quit IRC12:14
*** takamatsu has joined #openstack-meeting12:14
*** brinzhang_ has joined #openstack-meeting12:14
*** takamatsu has quit IRC12:16
*** larainema has quit IRC12:18
*** takamatsu has joined #openstack-meeting12:21
*** belmoreira has joined #openstack-meeting12:24
*** takamatsu has quit IRC12:29
*** yaawang has quit IRC12:29
*** yaawang_ has joined #openstack-meeting12:29
*** takamatsu has joined #openstack-meeting12:32
*** markvoelker has joined #openstack-meeting12:38
*** nanzha has joined #openstack-meeting12:41
*** brinzhang has joined #openstack-meeting12:49
*** brinzhang_ has quit IRC12:53
*** markvoelker has quit IRC12:57
*** lseki has joined #openstack-meeting12:59
*** panda|off is now known as panda13:01
*** mriedem has joined #openstack-meeting13:01
*** takamatsu has quit IRC13:06
*** takamatsu has joined #openstack-meeting13:10
*** nanzha has quit IRC13:15
*** nanzha has joined #openstack-meeting13:16
*** liuyulong has joined #openstack-meeting13:16
*** brinzhang_ has joined #openstack-meeting13:16
*** enriquetaso has joined #openstack-meeting13:19
*** brinzhang has quit IRC13:19
*** Luzi has quit IRC13:19
*** lpetrut has joined #openstack-meeting13:21
*** brinzhang has joined #openstack-meeting13:31
*** pcaruana has quit IRC13:32
*** brinzhang_ has quit IRC13:34
*** takamatsu has quit IRC13:36
*** bnemec has joined #openstack-meeting13:43
*** jawad_axd has quit IRC13:52
*** jawad_axd has joined #openstack-meeting13:53
*** jamesmcarthur has joined #openstack-meeting13:55
*** jawad_ax_ has joined #openstack-meeting13:57
*** ociuhandu has joined #openstack-meeting13:58
*** jawad_axd has quit IRC13:58
haleybhi14:01
*** brinzhang_ has joined #openstack-meeting14:01
*** jawad_ax_ has quit IRC14:01
haleyb#startmeeting networking-ovn14:03
openstackMeeting started Tue Oct 15 14:03:03 2019 UTC and is due to finish in 60 minutes.  The chair is haleyb. Information about MeetBot at http://wiki.debian.org/MeetBot.14:03
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.14:03
*** openstack changes topic to " (Meeting topic: networking-ovn)"14:03
openstackThe meeting name has been set to 'networking_ovn'14:03
* haleyb will wait for quorum14:03
*** pcaruana has joined #openstack-meeting14:04
*** brinzhang has quit IRC14:04
njohnstono/14:06
njohnstonsorry I am late14:06
haleybnjohnston: hi14:06
njohnstonstill not used to this meeting slot :-)14:06
haleybyou're the second person here, so not late14:06
liuyulongnot the team meeting this week?14:07
njohnstonno, that was yesterday14:07
njohnston17 hours ago I guess14:08
haleybtidwellr: ping14:08
liuyulongOK, for me, it is today, : )14:08
njohnstonliuyulong: ah! :-)14:09
*** jamesmcarthur has quit IRC14:09
haleybnjohnston: this is not shaping up to be a very exciting meeting with just the two of us14:11
njohnstonhaleyb: https://media.giphy.com/media/KFiQXtO3rWxlzpjnrV/giphy.gif14:12
haleybhah14:13
haleyb#topic Announcements14:13
*** openstack changes topic to "Announcements (Meeting topic: networking-ovn)"14:13
haleyb1st announcement this will be a really short meeting14:13
haleyb#topic ML2-OVS-DVR/OVN convergence14:14
*** openstack changes topic to "ML2-OVS-DVR/OVN convergence (Meeting topic: networking-ovn)"14:14
haleybThere has been some progress on https://etherpad.openstack.org/p/ML2-OVS-OVN-Convergence14:14
* haleyb realized these notes are just for posterity14:14
haleybpart of the outcome of going through the work items is a thought on keeping the networking-ovn code in it's own repo14:15
haleybi am looking into that and will talk to others offline for their thoughts14:16
liuyulongI can imagine a bad thing is that if networking-ovn is moving forward too fast to let user overcome the gap, the community and the user group will totally walk on their own separated way.14:17
haleybliuyulong: yes, one of the short-term goal would have to be to close the gap on a few items14:17
njohnstonagreed14:18
liuyulongNo more feedbacks from the real user group, because users are still have large sets of cloud to operate.14:18
*** dklyle has quit IRC14:18
haleybliuyulong: no, the only people involved so far are the distros and neutron team14:19
slaweqhi, sorry for being late14:19
slaweqI again forgot about this meeting14:19
liuyulongno matter linux bridge or ovs-agent OF flows, the ovn logical and flows or data-plane are totally different.14:20
liuyulongmigration looks not so much easily14:20
haleybslaweq: i will have to buy the beer in Shanghai as a meeting reminder14:20
slaweqhaleyb++ beer always works for me :)14:20
haleybliuyulong: thanks for reminding me of migration, that deserves a section in the convergence etherpad.  There is a rough outline of it, and it's possible, but depends on the deployment tools14:22
tidwellrhaleyb: pong14:22
*** markvoelker has joined #openstack-meeting14:23
liuyulongI don't know if someday the neutron agents would be all abandoned. Maybe after that day, every company will have their own cloud services...14:24
haleybtidwellr: hey, OVN meeting, i can follow-up with you offline if you're going to be able to continue working on this14:24
*** electrofelix has joined #openstack-meeting14:25
liuyulongBut, IMO, everything should be evolving。14:25
haleybliuyulong: i can envision a day when the agents are deprecated...14:25
liuyulongOVN is a better choice, for new cloud deployment I may embrace it. : )14:27
*** jamesmcarthur has joined #openstack-meeting14:27
haleybliuyulong: what is the biggest issue you see moving to OVN?  would your customers be worried?14:27
*** brinzhang has joined #openstack-meeting14:28
liuyulonghaleyb, actually, all most everything. The scale, the availibility, developability and maintainability...14:29
tidwellrhaleyb: we can follow up offline14:29
haleybtidwellr: ack14:29
tidwellrhaleyb: I live in interesting times ;)14:30
*** brinzhang has quit IRC14:30
liuyulongtool-chain, monitor, CI/CD...14:30
haleybliuyulong: are there items in the "gap" your customers are using?  just trying to get a different perspective than a distro14:31
*** brinzhang has joined #openstack-meeting14:31
*** brinzhang_ has quit IRC14:31
*** brinzhang has quit IRC14:32
*** brinzhang has joined #openstack-meeting14:32
liuyulonghaleyb, no, we try to deploy a OVN based environment once, but sadly it is not in-production finally.14:33
*** brinzhang has quit IRC14:33
*** ociuhandu has quit IRC14:33
*** brinzhang has joined #openstack-meeting14:34
liuyulonghaleyb, we just tested it few weeks, and almost everything we developed in neutron-agents will be needed to re-implemented one more time...14:34
*** brinzhang has quit IRC14:35
*** brinzhang has joined #openstack-meeting14:35
*** brinzhang has quit IRC14:35
liuyulonghaleyb, no times to achieve that while we are facing neutron its own bugs... : )14:35
haleybliuyulong: so you have your own fork of the agents?  are there things we need to add to the "gaps"?  or is it just the bugs :(14:36
liuyulongWhat's the status IPv6 implementation in OVN?14:36
*** brinzhang has joined #openstack-meeting14:36
*** brinzhang has quit IRC14:37
haleybliuyulong: one item missing is IPv6 PD, but it is being worked on currently14:37
*** brinzhang has joined #openstack-meeting14:38
*** brinzhang has joined #openstack-meeting14:38
liuyulongresource monitor, IP metering, Ports probe, QoS, pps limitation and some product functions.14:40
*** goldenfri has joined #openstack-meeting14:40
liuyulongAnd I don't think it will be accepted by our team, : )14:41
liuyulongshared bandwidth, peer-to-peer connection, l2gw, VPN, firewall...14:42
*** dklyle has joined #openstack-meeting14:43
liuyulongSo, if someday the networking-ovn move-in and become the default, maybe we can create a new repo for neutron agents, yes, a move-out. : )14:44
haleybliuyulong: that's a long list.  so you don't see yourself migrating to OVN in the next few years14:44
liuyulonghaleyb, for new small cloud, we can give a try if we find some best practice someday. Maybe there are tons of config options need to optimize, I guess.14:46
haleybliuyulong: ack, and i can add these items to the etherpad14:48
liuyulonghaleyb, I may say every cloud may have their own SLA assurance, migration can inevitablely cause data-plane down, so it has risk.14:48
haleybliuyulong: right, there are cases where you maybe can't do an in-place update14:49
*** eharney has joined #openstack-meeting14:49
*** enriquetaso has quit IRC14:51
*** armax has joined #openstack-meeting14:51
liuyulongOne more thing, baremetal, IMO some ovsdb/openflow apps can be easily developed in python...14:54
*** enriquetaso has joined #openstack-meeting14:54
haleybok, thanks for the information14:56
liuyulongnp14:56
haleybany other comments from anyone?  have :3 left14:57
haleybok, thanks for the discussion14:59
*** links has quit IRC14:59
haleyb#endmeeting14:59
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"14:59
openstackMeeting ended Tue Oct 15 14:59:28 2019 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)14:59
openstackMinutes:        http://eavesdrop.openstack.org/meetings/networking_ovn/2019/networking_ovn.2019-10-15-14.03.html14:59
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/networking_ovn/2019/networking_ovn.2019-10-15-14.03.txt14:59
openstackLog:            http://eavesdrop.openstack.org/meetings/networking_ovn/2019/networking_ovn.2019-10-15-14.03.log.html14:59
liuyulongbye14:59
*** ociuhandu has joined #openstack-meeting15:07
*** ociuhandu has quit IRC15:10
*** ociuhandu has joined #openstack-meeting15:11
*** gyee has joined #openstack-meeting15:13
*** mriedem has quit IRC15:18
*** mriedem has joined #openstack-meeting15:18
*** ttsiouts has quit IRC15:21
*** ttsiouts has joined #openstack-meeting15:22
*** ttsiouts has quit IRC15:27
*** markvoelker has quit IRC15:28
*** jawad_axd has joined #openstack-meeting15:28
*** diablo_rojo has joined #openstack-meeting15:29
*** jawad_axd has quit IRC15:33
*** igordc has joined #openstack-meeting15:37
*** brinzhang_ has joined #openstack-meeting15:37
*** mattw4 has joined #openstack-meeting15:38
*** brinzhang has quit IRC15:41
*** jawad_axd has joined #openstack-meeting15:49
*** liuyulong has quit IRC15:51
*** belmoreira has quit IRC15:53
*** jawad_axd has quit IRC15:53
slaweq#startmeeting neutron_ci16:00
openstackMeeting started Tue Oct 15 16:00:29 2019 UTC and is due to finish in 60 minutes.  The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot.16:00
slaweqhi16:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:00
*** openstack changes topic to " (Meeting topic: neutron_ci)"16:00
ralonsohhi16:00
openstackThe meeting name has been set to 'neutron_ci'16:00
* slaweq is on last meeting today \o/16:00
njohnstono/16:00
slaweqok, I think we can start as bcafarel will not be available today16:02
slaweqGrafana dashboard: http://grafana.openstack.org/dashboard/db/neutron-failure-rate16:02
slaweqplease open it now16:02
slaweq#topic Actions from previous meetings16:02
*** openstack changes topic to "Actions from previous meetings (Meeting topic: neutron_ci)"16:02
slaweqnjohnston Update the neutron-tempest-plugin dashboard in grafana16:02
njohnstonso the change has been submitted16:03
njohnston#link https://review.opendev.org/68768616:03
*** rpittau is now known as rpittau|afk16:03
njohnstonwaiting for another +216:04
ralonsoh(you have my +1)16:04
slaweqthx njohnston16:04
njohnstonIt is dependent on this change by slaweq https://review.opendev.org/#/c/685214/16:04
slaweqahh, yes I wanted to ask today for review of it16:05
ralonsoh^ this one, IMO, is ready to be merged16:05
slaweqnjohnston: can You?16:05
slaweq:)16:05
njohnstonslaweq: your wish is my command. +W16:05
slaweqLOL16:05
slaweqthx16:05
slaweqok, so lets move on than16:05
slaweqralonsoh to check root cause of ssh issue in https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_9c2/664646/6/check/tempest-ipv6-only/9c2f68f/testr_results.html.gz16:05
ralonsohslaweq, sorry but I didn't find the error16:06
ralonsohI still don't know why the IP is not given correctly to the VM16:06
ralonsohand I don't see any DHCP message16:06
ralonsohis like this VM has no connectivity to the DHCP16:07
slaweqbut it's single node job, rigth?16:07
ralonsohyes16:07
slaweqso both ports (vm and dhcp) should be plugged into br-int16:07
ralonsohyes16:07
ralonsohI can give another try16:08
slaweqthere can be only to reasons IMO - either port wasn't wired properly either firewall rules were not configured in it properly16:08
slaweqnono ralonsoh16:08
ralonsohbut I spent one whole morning on this16:08
slaweqif there is nothing in logs that You can't do anything :)16:08
slaweqdon't waste Your time on it16:08
clarkbwell usually that is an indication we need to improve our logging16:09
slaweqclarkb: yes, I just think about the same16:10
*** jawad_axd has joined #openstack-meeting16:10
clarkbbut ya update logging to cover likely cases then move on and wait for it to hit again (can also update elastic-recheck to track it for you)16:10
ralonsohclarkb, and that's usually the next action16:10
*** hemna_afk is now known as hemna_16:11
slaweqralonsoh: but do You think we can add some additional debug logs to our code to know something more about it?16:11
*** kopecmartin is now known as kopecmartin|off16:11
ralonsohslaweq, I'll check it and if possible, I'll submit a patch16:11
slaweqtbh we should be able to see if ports were configured by ovs agent16:11
slaweqand in fact IMO it was configured otherwise neutron would not report to nova that port is UP16:12
slaweqand nova would not unpause vm, right?16:12
ralonsohslaweq, yes and we see it in the ovs logs16:12
slaweqso IMO more likely this was some issue with openflow rules in br-int16:13
ralonsohslaweq, this is something we maybe should log better in debug/testing mode16:13
slaweqralonsoh: is there any way to maybe log all OF rules from bridges (br-int) every e.g. 1 second?16:14
ralonsohslaweq, uffff be careful16:14
slaweqsomething similar to e.g. dstat: https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_9c2/664646/6/check/tempest-ipv6-only/9c2f68f/controller/logs/screen-dstat.txt.gz16:14
slaweqralonsoh: LOL, I know but that would help us a lot16:14
ralonsohthis could be too much for the logs16:14
*** jawad_axd has quit IRC16:14
ralonsohbut yes, we can al least track the changes16:14
slaweqralonsoh: right, timestamp and what flow rule was added/removed16:15
ralonsohexactly but I need to check if ovs-ofctl allows this kind of monitoring16:15
slaweqdo You think that would this be possible?16:15
*** raildo_ has joined #openstack-meeting16:15
ralonsohI need to check it first16:15
ralonsohlike ovs-vsctl monitor16:15
slaweqralonsoh: exactly16:15
slaweqwill You do this?16:16
ralonsoh(sure16:16
slaweqthx a lot16:16
*** e0ne has quit IRC16:16
*** raildo has quit IRC16:16
slaweq#action ralonsoh to check if there is any possibility to do somethig like ovsdb-monitor for openflows16:16
slaweqthx ralonsoh for taking care of it16:17
slaweqlets move on16:17
slaweqnext was16:17
slaweqslaweq to prepare etherpad to track dropping py27 jobs from ci16:17
slaweqEtherpad is there https://etherpad.openstack.org/p/neutron_drop_python_2_status but it's not ready yet16:17
slaweqI will prepare list of jobs to remove in each repo16:17
slaweqI know we already started it for some repos like networking-ovn and neutron16:18
slaweqthis should be much faster than switch to python 3 :)16:18
njohnstonjust one note on this16:19
slaweqnjohnston: sure16:19
njohnstonthere are some projects out there that are being more aggressive in deprecating python 216:19
njohnstonfor example, by starting the process of removing six16:19
njohnstonthat level of effort should be discussed and planned out especially for projects that get imported by other projects like neutron/neutron-lib16:20
njohnstonso while it may seem tempting I just wanted to note for posterity's sake that such moves should be held off, at this time16:20
slaweqnjohnston: personally I don't think we should do it now16:20
slaweq1. because of backports to stable branches which still support py216:20
njohnstonthe particular project I saw with the six removal was a widely used oslo project, and so everything I said obviously goes double for oslo16:21
slaweqis there any "common goal" to remove such pieces of code during ussuri also?16:21
bnemecThere's discussion underway: https://etherpad.openstack.org/p/drop-python2-support16:22
bnemecOh, you already saw that.16:22
slaweqor dropping py2 support means just that we will not test it in u/s ci?16:22
bnemecNo wait, the etherpad above is neutron-specific. So that tc one is still relevant here.16:22
slaweqbnemec: yes, thx for link16:22
slaweqI will read it later16:23
njohnstonthanks bnemec, that link was my next thing to say :-)16:23
njohnstonso to sum up: the stadium is complicated, deprecating py2 is complicated, so let's do our usual neutron thing and overcommunicate about it all :-)16:24
bnemeclol16:24
slaweqnjohnston: what do You mean by "overcommunicate"? :)16:25
slaweqhmm, I think we lost njohnston now16:28
slaweqmaybe we can continue with other topics than16:28
slaweq#topic Stadium projects16:28
*** openstack changes topic to "Stadium projects (Meeting topic: neutron_ci)"16:28
njohnstonI am here.  Sorry, I thought you were saying that tongue-in-cheek16:29
slaweqahh, no16:29
slaweqsorry I really wanted to know what You mean exactly16:29
njohnstonI just meant that we'll talk about it in all our meetings and coordinate about it verbosely16:30
slaweqisn't preparing some etherpad and syncing weekly about it enough?16:30
slaweqahh, ok16:30
slaweqthx, so that's something which I want to do with this :)16:30
*** jawad_axd has joined #openstack-meeting16:30
*** igordc has quit IRC16:31
* njohnston is done16:31
slaweqregarding stadium projects, I think we can remove python3 migration from this agenda as we did for neutron meeting16:31
njohnstonagreed16:31
slaweqso we still have tempest-plugins migration16:31
slaweqand there is no any news about it16:32
slaweqanything else You want to discuss regarding stadium projects' ci?16:32
slaweqahh, I have one more small thing16:33
slaweqplease review https://review.opendev.org/#/c/685213/ - it's needed for Train and for stadium projects16:33
slaweqif there is nothing else related to stadium, lets move on to the next topic16:34
slaweq#topic Grafana16:34
*** openstack changes topic to "Grafana (Meeting topic: neutron_ci)"16:34
slaweq#link http://grafana.openstack.org/dashboard/db/neutron-failure-rate16:34
*** jawad_axd has quit IRC16:34
slaweqit's a bit strange that we have so few data for gate queue16:35
slaweqdid we merge only few patches last week in neutron really?16:35
ralonsohnot many merged last week16:36
ralonsohmerges16:36
slaweqhmm, it could be like that really16:36
slaweqin check queue still our bigger problem IMO is functional and fullstack job16:37
*** nanzha has quit IRC16:38
njohnstonyep16:38
slaweqI have couple of examples of new failures there16:38
slaweqfrom good things I think that our (voting) scenario jobs works quite good recently16:39
njohnstonagreed16:39
*** tssurya has quit IRC16:40
slaweqok, so lets talk about those functional/fullstack failures16:40
slaweq#topic fullstack/functional16:40
*** openstack changes topic to "fullstack/functional (Meeting topic: neutron_ci)"16:40
slaweqfirst fullstack16:40
*** ociuhandu_ has joined #openstack-meeting16:40
slaweqralonsoh: I saw 2 new failures of our old friend: neutron.tests.fullstack.test_dhcp_agent.TestDhcpAgentHA.test_reschedule_network_on_new_agent16:41
slaweqhttps://2f600b8d6843c7d64afe-bbbb707b755a08f42bfac9929d4d55b4.ssl.cf2.rackcdn.com/688439/3/check/neutron-fullstack/119e702/testr_results.html.gz16:41
slaweqand16:41
slaweqhttps://e5965e413edbbc117465-bfd96e490b07511790a1eb1aa4beb29d.ssl.cf2.rackcdn.com/665467/37/check/neutron-fullstack/4437eb3/testr_results.html.gz16:41
slaweqIIRC You added some extra logs some time ago to debug this when it will happen again, right?16:41
ralonsohslaweq, yes. I'll add this to my pile for tomorrow16:42
slaweqralonsoh: thx, I can help with this if You want16:42
slaweq#action ralonsoh to investigate failed fullstack tests for dhcp agent rescheduling16:43
slaweqok, next one16:43
slaweqneutron.tests.fullstack.test_qos.TestDscpMarkingQoSOvs16:43
slaweqhttps://openstack.fortnebula.com:13808/v1/AUTH_e8fd161dc34c421a979a9e6421f823e9/zuul_opendev_logs_7c2/684457/1/check/neutron-fullstack/7c2c5d8/testr_results.html.gz16:43
slaweqIt seems like there was only one icmp packet sent and test failed: https://openstack.fortnebula.com:13808/v1/AUTH_e8fd161dc34c421a979a9e6421f823e9/zuul_opendev_logs_7c2/684457/1/check/neutron-fullstack/7c2c5d8/controller/logs/dsvm-fullstack-logs/TestDscpMarkingQoSOvs.test_dscp_marking_packets.txt.gz16:44
slaweqso as it is "my" test I will debug this16:44
slaweqok for You?16:44
*** ociuhandu has quit IRC16:44
ralonsohok16:44
njohnston+116:44
slaweq:)16:44
slaweq#action slaweq to investigate failed neutron.tests.fullstack.test_qos.TestDscpMarkingQoSOvs16:44
slaweqnow functional tests16:45
slaweqneutron.tests.functional.agent.l2.extensions.test_ovs_agent_qos_extension.TestOVSAgentQosExtension.test_port_creation_with_dscp_marking(egress)16:45
slaweqhttps://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_692/688439/3/check/neutron-functional/69288a9/testr_results.html.gz16:45
slaweqthis one is interesting IMO16:45
slaweqin test logs: https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_692/688439/3/check/neutron-functional/69288a9/controller/logs/dsvm-functional-logs/neutron.tests.functional.agent.l2.extensions.test_ovs_agent_qos_extension.TestOVSAgentQosExtension.test_port_creation_with_dscp_marking_egress_.txt.gz16:45
slaweqthere is a lot of "EVENT OVSNeutronAgentOSKenApp->ofctl_service GetDatapathRequest send_event" log lines16:45
slaweqat first glance it looks for me like something was looping there infinitely16:45
njohnstonthat's quite unusual16:46
slaweqnjohnston: yep16:46
slaweqanyone wants to investigate that?16:47
ralonsohsorry not now16:47
slaweqif not, I can assign it to myself16:47
ralonsohI have enough backlog16:47
slaweqralonsoh: sure, I know You are overloaded :)16:47
*** ociuhandu_ has quit IRC16:47
slaweqok, I will try to check that16:47
slaweq#action slaweq to check strange "EVENT OVSNeutronAgentOSKenApp->ofctl_service GetDatapathRequest send_event" log lines in neutron.tests.functional.agent.l2.extensions.test_ovs_agent_qos_extension.TestOVSAgentQosExtension.test_port_creation_with_dscp_marking16:48
slaweqok, next one16:48
slaweqthis one might be interesting for njohnston :)16:48
slaweqfailed db migration tests again16:48
slaweqhttps://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_692/688439/3/check/neutron-functional/69288a9/controller/logs/dsvm-functional-logs/neutron.tests.functional.agent.l2.extensions.test_ovs_agent_qos_extension.TestOVSAgentQosExtension.test_port_creation_with_dscp_marking_egress_.txt.gz16:49
*** takamatsu has joined #openstack-meeting16:49
slaweqbut this time it's not due to timeout and slow node16:49
slaweqsorry, wrong link16:49
slaweqargh, I don't have correct link for this one now16:50
slaweqsorry16:50
njohnstonslaweq: np16:50
slaweqif I will find it somewhere, I will ping You16:50
njohnstonslaweq: sure thing16:50
njohnstonslaweq: also as a lead on the ofctl error from the last bug, check out https://bugzilla.redhat.com/show_bug.cgi?id=138237216:50
openstackbugzilla.redhat.com bug 1382372 in openstack-neutron "Selinux is blocking ovs-vswitchd during functional tests" [Urgent,Closed: currentrelease] - Assigned to twilson16:50
njohnstonit's been seen before, as an selinux issue16:51
slaweqnjohnston: but u/s ci is running on ubuntu and I don't think there is selinux enabled there16:51
njohnstondarn bugzilla, bamboozled again16:52
slaweqlol16:53
slaweqok, lets move on to the last one:16:53
slaweqneutron.tests.functional.agent.test_firewall.FirewallTestCase16:54
slaweq    https://08b11cea7395c153ac8e-9514c1b1570a8e9931b2a7d3207ef22f.ssl.cf2.rackcdn.com/684457/1/check/neutron-functional/06e9d0c/testr_results.html.gz16:54
*** dviroel_ has joined #openstack-meeting16:54
ralonsohthe flows were not applied?? maybe16:55
ralonsoh(we need a flow monitor)16:55
slaweqralonsoh: yes16:55
*** diablo_rojo has quit IRC16:55
slaweqotherwise it may be hard to say what happend there16:55
*** diablo_rojo has joined #openstack-meeting16:56
slaweqbut in this case it's not "br-int" only to monitor16:56
ralonsohI know16:56
slaweqeach test has got own bridge16:56
ralonsohevery test case should be able to deploy it's own monitor16:56
*** ociuhandu has joined #openstack-meeting16:56
ralonsohin any bridge16:56
ralonsohI'll put extra effort on this16:56
slaweqmaybe we can do some simple decorator to run tests and in case of failure dump bridges from bridge used in the test somehow?16:57
ralonsohthis could be useful too16:57
njohnstonthat would be very useful16:57
slaweq*dump flows16:57
slaweqanyone wants to do it maybe?16:57
slaweqif not, I can16:57
slaweq:)16:57
ralonsohI can check it16:57
slaweqat least add it to my backlog16:57
slaweqok, thx ralonsoh16:58
ralonsohalong with the flow monitor16:58
slaweq#action ralonsoh to try to log flows at the end of faliled functional test16:58
slaweqralonsoh: ^^ it's just as a reminder, it's not urgent for sure :)16:58
njohnstonI'll also see if I can clear enough time to take a look16:58
slaweqthx njohnston16:58
ralonsohthanks!16:59
slaweqok, I think we have to finish now16:59
slaweqone last thing16:59
slaweqplease add https://review.opendev.org/#/c/685705/ to Your review pile16:59
slaweqthx for attending16:59
slaweqand have a great day/evening :)16:59
slaweqbye16:59
ralonsohbye!16:59
slaweq#endmeeting16:59
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"16:59
openstackMeeting ended Tue Oct 15 16:59:57 2019 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)17:00
openstackMinutes:        http://eavesdrop.openstack.org/meetings/neutron_ci/2019/neutron_ci.2019-10-15-16.00.html17:00
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/neutron_ci/2019/neutron_ci.2019-10-15-16.00.txt17:00
openstackLog:            http://eavesdrop.openstack.org/meetings/neutron_ci/2019/neutron_ci.2019-10-15-16.00.log.html17:00
njohnstonyou too slaweq o/17:00
*** njohnston is now known as njohnston|lunch17:00
*** ociuhandu has quit IRC17:01
*** lpetrut has quit IRC17:01
*** e0ne has joined #openstack-meeting17:02
*** priteau has quit IRC17:03
*** brinzhang has joined #openstack-meeting17:05
*** e0ne has quit IRC17:08
*** armax has quit IRC17:08
*** brinzhang_ has quit IRC17:08
*** takamatsu has quit IRC17:12
*** ricolin has quit IRC17:20
*** dviroel_ is now known as dviroel17:24
*** ralonsoh has quit IRC17:25
*** ociuhandu has joined #openstack-meeting17:31
*** electrofelix has quit IRC17:32
*** ociuhandu has quit IRC17:35
*** psachin has quit IRC17:38
*** e0ne has joined #openstack-meeting17:42
*** eharney has quit IRC17:43
*** armax has joined #openstack-meeting17:44
*** e0ne has quit IRC17:47
*** ociuhandu has joined #openstack-meeting17:48
*** jawad_axd has joined #openstack-meeting17:50
*** ociuhandu has quit IRC17:56
*** brinzhang_ has joined #openstack-meeting18:03
*** diablo_rojo has quit IRC18:06
*** diablo_rojo has joined #openstack-meeting18:06
*** brinzhang has quit IRC18:07
*** markvoelker has joined #openstack-meeting18:07
*** priteau has joined #openstack-meeting18:11
*** jamesmcarthur has quit IRC18:12
*** markvoelker has quit IRC18:12
*** markvoelker has joined #openstack-meeting18:21
*** ociuhandu has joined #openstack-meeting18:26
*** markvoelker has quit IRC18:31
*** eharney has joined #openstack-meeting18:32
*** igordc has joined #openstack-meeting18:36
*** rubasov has quit IRC18:41
*** ociuhandu has quit IRC18:55
*** e0ne has joined #openstack-meeting18:59
clarkbthe infra meeting will begin shortly18:59
*** Shrews has joined #openstack-meeting19:00
ianwo/19:00
*** jamesmcarthur has joined #openstack-meeting19:00
fungihowdy19:01
corvuso/19:01
clarkb#startmeeting infra19:01
openstackMeeting started Tue Oct 15 19:01:16 2019 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.19:01
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.19:01
*** openstack changes topic to " (Meeting topic: infra)"19:01
openstackThe meeting name has been set to 'infra'19:01
clarkb#link http://lists.openstack.org/pipermail/openstack-infra/2019-October/006493.html Our Agenda19:01
clarkb#topic Announcements19:01
*** openstack changes topic to "Announcements (Meeting topic: infra)"19:01
*** markvoelker has joined #openstack-meeting19:01
clarkbThe OpenStack release is happening this week (~tomorrow)19:01
clarkbI believe the only potential need for infra help is requeuing any of the openstack release teams' tagging jobs19:02
clarkbif they happen to timeout (there was a change to increase the timeout on them to avoid that in the first place)19:02
clarkbbut I guess keep an eye out for any issues they run into that we can help with19:02
corvustagging job > 30 min?19:02
clarkbcorvus: ya it serially ran through every project created a tag for it and pushed it to gerrit19:03
clarkband apparently when you do that over several hundred projects you take longer than 30 minutes (this happened with stein)19:03
corvusi see19:03
clarkbthe job won't retag and repush things so it can be requeued and it basically skips ahead to where it left off safely19:03
fungii don't know that it happened with stein19:04
clarkboh it happend at some point19:04
fungiit happened at stein and train rc119:04
clarkbah19:04
fungibecause those also involve branch creation on every project19:04
fungiand pushing new changes to each of them19:04
mordredbranch creation is slightly more expensive I guess?19:04
fungiwith all the associated change proposals it is19:05
mordredah - yeah - that makes sense19:05
fungiwe're just being extra cautious. i plan to be at the computer by 11:30z and the button mashing commences at 12:00z i believe19:05
clarkbI fully endorse everything being done by the time I wake up :)19:06
fungii expect it will be mostly done, including the follow-on jobs triggered by the tag pushes, by 13:00z19:06
*** markvoelker has quit IRC19:06
clarkbother things to consider: we probably want to avoid zuul restarts tomorrow (at least until fungi gives us the all clear)19:06
*** jakecoll has joined #openstack-meeting19:06
fungiunless something goes sideways, and even then that leaves us hours until the osf has the usual press release planned and the associated press embargoes lift19:06
fungiwhich i think is usually around 15:00z or thereabouts19:07
* diablo_rojo sneaks in late19:07
clarkbbut ya I don't anticipate problems. We've fixed or are fixing a bunch of our known infra issues recently and of those I don't think they would really affect the openstack release19:08
fungiin release-related announcements, i also created the ussuri signing key a few weeks ago:19:08
fungi#link https://sks-keyservers.net/pks/lookup?op=vindex&search=0xbba3b1e67a7303dd1769d34595bf2e4d09004514&fingerprint=on OpenStack Infra (Ussuri Cycle) artifact signing key19:08
clarkb(things like NM on red hat distros and the debian buster mirror)19:08
fungi#link https://docs.openstack.org/infra/system-config/signing.html#attestation infra signing key attestation procedure19:09
fungii'll push up changes to switch the train key out for that in a week or so19:09
clarkbAnything else to announce?19:10
* fungi knows of nothing else19:10
clarkb#topic Actions from last meeting19:10
*** openstack changes topic to "Actions from last meeting (Meeting topic: infra)"19:10
clarkb#link http://eavesdrop.openstack.org/meetings/infra/2019/infra.2019-10-08-19.01.txt minutes from last meeting19:11
clarkbI started drafting up a thing for opendev governance19:11
clarkb#link https://etherpad.openstack.org/p/rCF58JvzbF OpenDev governance email draft19:11
clarkbI think it is still a bit rough and I've tried to bring up a few of the things I expect peopel to question/bring up19:11
clarkbin particular how an OpenDev PTL is chosen, whether we need tie breaker rules on the proposed council and whether we want the "else" group to be covered there too19:12
clarkbI think all of that is open for discussion at this point so please feel free to leave comments on the etherpad19:12
*** tbarron_ is now known as tbarron19:13
corvusclarkb: thanks!  after collecting etherpad comments, what do you think is next?19:13
clarkbcorvus: I think draft a final version that we can all be happy with as a group, then take that to the TC as a "look this is what we'd like to do and we'd like your input and go ahead to do this formally"19:14
corvuskk19:14
clarkbI don't expect that to present any problems but would be good to make it all official like19:14
clarkbwe should probably also ask the other groups that will be part of the council for input19:16
clarkbsince we will be asking them to participate19:16
clarkbfungi: ^ that may help answer the question re kata19:16
clarkb"does this even interest you?"19:17
fungimakes sense19:17
clarkbProbably best to dive into this on the etherpad once we've collected more thoughts and opinions19:18
clarkbwe can come back to this at the ned of the meeting if we have time too19:18
clarkb#topic Priority Efforts19:18
*** openstack changes topic to "Priority Efforts (Meeting topic: infra)"19:18
clarkb#topic OpenDev19:18
*** openstack changes topic to "OpenDev (Meeting topic: infra)"19:18
clarkbAny other opendev topics to bring up?19:18
clarkbSounds like no19:19
fungiwhat's our current opendev focus?19:19
clarkbfor me it has become the governance topic19:19
fungiokay, cool. just making sure i know where we are in the transition19:19
clarkbBased on complaints I think our git hosting is fairly stable at this point other than the performance issue on large repos19:20
fungiand i guess the gerrit upgrade19:20
clarkbwhich is a known upstream issue with people working on it19:20
fungiand the zuul log storage switch19:20
corvusmordred's is gerrit upgrade19:20
fungiright, i meant more collectively19:20
corvusand ianw is driving static retirement19:21
fungilike what do we want to complete before we start moving more services into opendev19:21
fungithose all sound like good things to get behind us first19:21
corvusmore services like?19:21
corvuslike adding services, or renaming?19:21
fungietherpad et cetera19:21
fungiwell, it's never been clear to me if something is officially opendev while it's still on an openstack.org domain19:21
corvusi wouldn't put up any blockers to moving anything over at this point19:22
ianw(on static; yes very distracted with glean + centos8 this week, but will get back to updating that spec as discussed last week)19:22
mordredyah- other than available human time19:22
clarkbya I think we can start shifting services like etherpad as soon as people have time to do them19:22
mordredclarkb: ++19:22
fungigot it. i know we were pushing back on that when the focus was still on stabilizing the gerrit/gitea migration19:22
corvusi know i said "let's not move etherpad" in the past, because i think we needed all hands on deck for gerrit migration, but i don't feel like any current effort is like that19:22
clarkbit might make moving services slightly easier if we prioritize the dns proposal implementation19:22
corvusclarkb: ++19:22
fungiawesome. thanks!19:22
fungiand yes, i agree the dns/ssl plan is a high priority for that19:23
clarkbspeaking of the dns proposal I did bring it up with osf staff and heard no objections19:23
clarkbjbryce did say it would be this week or next before he could really read and digest it though19:23
corvusgroovy19:23
funginot that there was really anything to object to19:23
fungiand there's some possible interest in moving the openstack.org domain into our dns hosting, just not sure how much and on what timeline yet19:24
fungibut i think the etherpadded plan is a simple interim solution even so19:24
corvusyeah, it won't produce any roadblocks to future work like that19:24
*** markvoelker has joined #openstack-meeting19:25
fungithanks, that covers my questions. we just hadn't done a "where are we now?" on this in a few months19:26
clarkbTo summarize, mordred is working on improvements to gerrit config management using containers to help enable a big gerrit upgrade, ianw is working on static.openstack.org cleanup, clarkb is working on the governance side of things. Others should feel free to pick up other service "moves" which might be made easier if we focus on the dns proposal first19:26
clarkbhttps://etherpad.openstack.org/p/openstack-org-dns is the dns proposal19:27
clarkb#topic Update Config Management19:27
*** openstack changes topic to "Update Config Management (Meeting topic: infra)"19:27
clarkbmordred: anything new on gerrit container images?19:28
mordredthere's some patches up to update to bazel 1.019:28
mordredwhich are required to land the latest changes19:28
mordredthat opened up a can of worms with puppet - which I think is finally unwound19:28
mordredhttps://review.opendev.org/#/c/688523/ and friends19:29
*** trident has quit IRC19:29
clarkbya that prompted some cleanup of the centos-7 puppet jobs we were running19:30
mordredsince I missed the last few meetings - the current status is that review-dev is running on the container image - so we know that bit works - the next bit is finishing the patch to actually be able to do that via config management19:30
*** bbowen has quit IRC19:31
mordredat some point it might be good to rethink our gerrit server-side hook scripts that use jeepyb to be zuul jobs - as they currently require us to add jeepyb into the gerrit image itself - but it's not urgent19:32
corvus++19:32
mordredalso - we've got some patches up to add master-based images with checks plugin enabled that we can use for zuul testing - and eventually potentially be more ready for rollouts as we go forward19:32
*** trident has joined #openstack-meeting19:33
clarkbThat sounds good. Anyone else have updates related to config management?19:34
clarkbLets move on19:36
clarkb#topic Storyboard19:36
*** openstack changes topic to "Storyboard (Meeting topic: infra)"19:36
clarkbfungi: diablo_rojo anything new to report here?19:36
fungiwe moved the storyboard "weekly" irc meeting to 14:00z thursdays19:36
fungiin hopes we have critical mass for it to be closer to weekly in frequency again19:37
funginothing new from me on the team management automation front yet. i've started fleshing out the module last week but haven't gotten to pushing anything into gerrit19:38
fungifocus has mostly been elsewhere19:38
fungithough in the sb meeting last week we got consensus on the direction i'm taking it19:38
clarkband that direction is using the api to manage group membership in storyboard instead of the database19:40
fungiyep19:41
fungivia ansible19:41
clarkbgreat. Anything else?19:41
funginothing i'm aware of19:41
clarkb#topic General Topics19:42
*** openstack changes topic to "General Topics (Meeting topic: infra)"19:42
clarkbfungi: I noticed there were some wiki changes today, I assume we should all go and review those?19:42
fungii've imported fresh mysqldump and image tree copy from the production wiki onto wiki-dev, and then discovered mediawiki has a number of unpackaged dependencies which need installing via a php tool called "composer"19:42
fungi#link https://review.opendev.org/#/q/project:opendev/puppet-mediawiki+is:open+topic:xenial-upgrades19:42
mordred"neat"19:42
fungiyeah, i see one has a beaker failure, need to dig into it19:43
fungialso i've basically done that stuff manually on wiki-dev just to see what breaks next19:43
fungiit's now giving me no output instead of an error in the browser, so time to go hunting in logs19:44
clarkbprogress at least19:44
fungiyep19:44
mordredfungi: your beaker failed is the thig I just fixed for gerrit19:44
fungioh, neat19:44
fungithanks mordred!19:44
fungiwill look after the meeting19:45
clarkbianw for static.o.o I think we can go ahead and update the spec with the dns proposal changes involved19:45
mordredso once https://review.opendev.org/#/c/688555/ lands (has +A) - we can recheck that patch and it shoiuld be fine19:45
clarkbianw: other than waiting for new patchset to review on that is there anything we can do to help re static.o.o?19:45
ianwclarkb: yes, will do for sure19:46
ianwsorry, at this point i think it's stalled on us getting ducks in a row on spec19:46
ianw /s/on/in/19:46
fungijust needs some duck sauce then19:46
clarkbno worries, just checking19:47
clarkbNext up I wanted to bring up CentOS 8.19:47
clarkbI think ianw managed to track down the network problems we've had on red hat distros (including 8) and those should now be fixed19:47
clarkbplease keep an eye out for odd behavior on centos and fedora on limestone and fortnebula in particular but we should be good now19:47
clarkbzbr and ianw have updates to fix mirror management on centos 8 in zuul-jobs19:48
ianwyep, and last night the configure-mirror changes went in, so devstack can start19:48
mordredwell done ianw!19:48
clarkbI want to say enough of that has now merged that we should be able to run jobs on centos 8 now and have their actual workloads run19:48
ianwergo all the setup roles are working19:48
fungiyay!19:48
clarkbexcellent thank you for confirming19:48
mordredthat was quite the digging for that one19:48
ianwyep, i plan to announce the images to the list, along with some of the caveats about it being python3 only (by default) today19:48
*** tesseract has quit IRC19:49
zbrtomorrow I am going to start enabling new jobs19:49
clarkb++ thank you for getting that done19:49
clarkbalso apparently we've managed to build centos 8 cloud images before upstream so some individuals have come to us looking for them :)19:49
fungihilarious19:50
fungibut also sad19:50
ianwyes, i need to sync with maybe karinbar there ... there are kickstarts, and their infra is building them but not publishing for some reason19:50
zbrianw: not sure if advertising is really needed,... i prefer to slowly add jobs, so we have time to fix the broken bits.19:50
clarkbzbr: I think in the past we've not been aggressive enough in advertising updated distros which leads to peopel dragging their feet and we end up not updating as quickly as possible19:50
ianwzbr: i'd like people to go in with a heads up that they're different wrt to pip/virtualenv setup to all our other images19:50
clarkbI think we should let everyone know that they can (and should) update now that we are happy with it and that will help shake out any additional problems19:51
clarkbLast up on the agenda is the PTG19:52
clarkb#link https://etherpad.openstack.org/p/OpenDev-Shanghai-PTG-2019 Add your PTG topic ideas here19:52
clarkb#link https://www.openstack.org/ptg/#tab_schedule PTG schedule19:52
clarkbI'm told I should hear back on my visa tomorrow. I am cautiously optomistic that there will be good news19:53
clarkb#topic Open Discussion19:54
*** openstack changes topic to "Open Discussion (Meeting topic: infra)"19:54
mordredclarkb: I hope to see you in Shanghai19:54
clarkbmordred: me too :)19:54
clarkbI'll open it up to further discussion on past topics or new items19:54
mordredfwiw - I'm now in Asia until the Summit (in Japan timezone)19:54
*** e0ne has quit IRC19:55
fungithe debian mirror seems to be updated now, though seeing different build failures around the debian-security mirror now19:55
ianwhangin' in the +919:55
clarkbmordred: is there a secret goal of visiting every timezone?19:55
fungii may need to perform some similar reprepro surgery on it19:55
mordredclarkb: not very secret :)19:55
mordredmy intent is to work hours that are non-normal here so I have more overlap with folks - but I'm jetlagged at the moment and have't 100% sorted out what that looks like yet19:56
corvusmordred: cool, thanks for keeping us updated19:57
clarkbOh we are removing opensuse 42.3 images from nodepool19:57
clarkbthank you to ajaeger and corvus for pushing on that19:57
clarkbthere is one devstack change we are waiting on before rechecking the nodeset removal to see if that will go in now19:58
ianw++19:58
ianwf28 should probably be next19:58
clarkband opensuse 15019:58
clarkbbut ya agreed19:58
corvusi'm guessing we should hold off on the git repo cache move until friday?19:58
corvusor at least post-release?19:59
clarkb++19:59
clarkbthat has the potential to have widespread impact probably best to play it safe19:59
corvusi'm basically waiting for clarkb and fungi to tell me it's okay to break the debian-whatever and old suse images...19:59
corvusthe ones that haven't updated in ages19:59
*** markvoelker has quit IRC20:00
clarkband we are at time20:00
corvushttps://review.opendev.org/68293520:00
clarkbthank you everyone. Feel free to continue discussion in #openstack-infra20:00
clarkb#endmeeting20:00
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"20:00
*** brinzhang has joined #openstack-meeting20:00
openstackMeeting ended Tue Oct 15 20:00:26 2019 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)20:00
openstackMinutes:        http://eavesdrop.openstack.org/meetings/infra/2019/infra.2019-10-15-19.01.html20:00
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/infra/2019/infra.2019-10-15-19.01.txt20:00
openstackLog:            http://eavesdrop.openstack.org/meetings/infra/2019/infra.2019-10-15-19.01.log.html20:00
fungithanks clarkb!20:00
clarkbcorvus: I think we can break both of suse and buster old images as soon as release is done (even if fungi and I don't get those sorted before then)20:01
fungiyep20:01
fungii concur20:01
clarkbcorvus: cleanups/fixes are in progress so should be done soon after if not before20:01
corvuscool, thx.  i wipd it for now20:01
*** brinzhang_ has quit IRC20:03
*** b1airo has joined #openstack-meeting20:04
b1airoAnyone about?20:04
*** Shrews has left #openstack-meeting20:07
*** jamesmcarthur has quit IRC20:09
*** jamesmcarthur has joined #openstack-meeting20:19
*** jawad_axd has quit IRC20:21
*** pcaruana has quit IRC20:25
*** rbudden has joined #openstack-meeting20:32
*** pcaruana has joined #openstack-meeting20:34
*** macz has joined #openstack-meeting20:35
*** slaweq has quit IRC20:38
*** jamesmcarthur has quit IRC20:38
*** bbowen has joined #openstack-meeting20:45
*** ociuhandu has joined #openstack-meeting20:46
*** jamesmcarthur has joined #openstack-meeting20:48
*** pcaruana has quit IRC20:51
*** slaweq has joined #openstack-meeting20:54
*** oneswig has joined #openstack-meeting20:55
*** slaweq has quit IRC20:59
oneswig#startmeeting scientific-sig21:00
openstackMeeting started Tue Oct 15 21:00:20 2019 UTC and is due to finish in 60 minutes.  The chair is oneswig. Information about MeetBot at http://wiki.debian.org/MeetBot.21:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.21:00
*** openstack changes topic to " (Meeting topic: scientific-sig)"21:00
openstackThe meeting name has been set to 'scientific_sig'21:00
oneswigahoy21:00
oneswigb1airo: good morning21:00
oneswig#chair b1airo21:00
openstackCurrent chairs: b1airo oneswig21:00
rbuddenhello21:00
*** brinzhang_ has joined #openstack-meeting21:00
oneswigwhat's new?21:00
oneswighey rbudden21:01
b1airoHey hey, good to see you folks21:01
oneswigI updated my desktop to Mac OS Catalina.  Now if I open Mail it hangs, saying it's doing a DB upgrade. So OpenStack!21:02
b1airoThanks to daylight savings I now have a meeting clash here again, so I'll be a little disengaged sorry21:02
b1airoBurn oneswig !21:02
*** martial has joined #openstack-meeting21:03
rbuddeni hear ya, i’m on daycare pickup in a half hour ;)21:03
oneswigAh, too bad b1airo rbudden21:03
b1airo(should have been an apostrophe in there maybe - don't really want oneswig burnt)21:03
oneswigI've got one in bed and the other brushing her teeth.  But she's 12 so she's got the hang of it21:03
*** brinzhang has quit IRC21:03
martialevening Gents21:04
oneswigHey martial21:04
oneswig#chair martial21:04
openstackCurrent chairs: b1airo martial oneswig21:04
b1airoHi martial21:04
martialso Stig, you are going to Shanghai?21:04
oneswigSure am.  At least, I'm going to the Chinese embassy tomorrow to lend them my passport21:04
oneswigIf they give it back in time, I'll be going to Shanghai..21:05
b1airoo_021:05
martialenjoy your trip my friend; I know you will join us at SC1921:05
oneswigLooking forward to it, but keep procrastinating on sorting the travel...21:06
oneswig(for SC)21:06
oneswigDo you have hotels booked?21:06
martialwhich reminds me, the "Container Advisory Panel" want to promote their "Containers and New Orchestration Paradigms for Isolated Environments in HPC (CANOPIE HPC)"21:07
martial#link https://sc19.supercomputing.org/session/?sess=sess13621:07
jakecollHi everyone. Is there going to be a PTG for the Scientific SIG?21:07
martialI do have my hotel booked, and plane and registration done [today is the last day for early bird for SC19 btw]21:07
oneswigHi jakecoll, welcome.  There's a day for it in Shanghai - Thursday I believe21:08
oneswigYou going?21:08
martialand there will be a PTG session as well, I believe21:08
*** trandles has joined #openstack-meeting21:08
jakecollYes, I'll be in Shanghai21:08
martialso PSA: if you are coming to SC19 and have not registered yet: <blink>today is the last day for early bird</blink> :)21:09
jakecollIs there anything on the docket for the PTG as of yet?21:09
oneswigmartial: thanks for the heads up!21:09
martialstig: hotels are scarce by now21:09
oneswigjakecoll: Nothing as yet.  Usually it's really handy to meet with the Ironic team but this time the session overlaps.21:09
oneswigPerhaps it's time to make an etherpad, if it hasn't been auto-generated this time around21:11
oneswig(should check on that)21:11
jakecollI'll keep a look out then21:11
oneswigjakecoll: are there items you'd like to see covered?21:12
jakecollWell, my PI was mostly wondering if the Scientific SIG had lightning talks on the docket or something of the like this year21:13
jakecollWe were wondering if anyone was interested in discussion of traces21:14
martialwe have two sessions this year as well?21:14
martialwhere is the next one btw?21:14
oneswigNot announced yet I believe, but I think they are dropping to 1 summit per year21:16
martialinteresting21:17
oneswigI don't believe we have the SIG session with lightning talks in the schedule this year, but we have a whole day at the PTG to fill, which more than replaces it.21:18
martialthis is good yet a little sad; this is the primary way we bring in new people to the conversation21:18
martialfew come to the PTG21:18
oneswigjakecoll: are you thinking of control plane profiling traces, or application traces?21:18
oneswigmartial: think you're right.21:18
rbuddenoh wow, dropping to a single summit, that’s a bummer21:19
oneswigdon't remember where I heard that, mind...21:19
jakecollI honestly don't know the difference between the two. I work on Chameleon Cloud. It would be our cloud traces.21:20
martialPTG vs Summit?21:21
martialPTG is an event after the Summit and you need another ticket to get to it21:21
martialSummit is the main event with presentations and such21:22
oneswigjakecoll: Is that data about representative cloud usage?  I think I remember hearing about that.21:22
martialOften people go to the Summit, but few stay for the PTG21:22
jakecollyes21:22
martial[I feel like I misunderstood the question :) ]21:22
*** diablo_rojo has quit IRC21:24
oneswigmartial: dumb question, for the BoF participation, do I need to register for the SC Technical program?21:24
diablo_rojo_phonmartial: it's all one registration21:25
diablo_rojo_phonYou don't need separate tickets this time.21:25
martialoh coo, thanks Kendall [how are you?]21:25
martialstig: I think so21:25
diablo_rojo_phonNo problem :) Good!21:25
trandlesoneswig: I missed the BoF last year because I was only on an exhibitor pass :(21:26
trandlesHi all btw, mostly lurking today21:26
martialstig: let me check the "logo" on it21:26
b1airoThanks for the heads up on early bird for SC, martial21:26
oneswighi trandles, excellent lurking so far21:26
oneswigb1airo: I think he's saved us both many dollars...21:27
martial#link https://sc19.supercomputing.org/session/?sess=sess33221:27
martialif you are an IEEE or ACM SIG HPC member you also get a discount (deal alert)21:27
martialso the SC19 says "EX" and "EXH"21:27
b1airoI too have been procrastinating on organising the travel21:27
b1airoYeah HPC SIG pays for itself21:28
trandlesjakecoll: I'm interested in Chameleon cloud traces if you're thinking of publishing. We have another project (not directly cloud-related) that's gathering trace data for cloud and HPC workloads21:28
verdurinIs it worth registering for the tutorials at SC? Oh, and hello.21:29
martialI think the EX and EXH mean that you do not need a "TP" (Technical Program) badge to attend but21:29
martialverdurin: if you want to see the HPC Container crowd, they have their own Wosrkshop on Monday called CANOPIE21:29
oneswigHi verdurin...21:29
oneswigworkshops are different from TP or so I have just read21:29
martialand on Sunday, there is a side event (not part of SC19) from The Next Platform21:30
trandleswe have a poster and paper accepted for CANOPIE so will be there21:30
martial#link https://www.nextplatform.com/hpc-day-with-the-next-platform/21:30
b1airoYeah, I'll most likely be at CANOPIE, was thinking about a submission, but time..21:30
martial(extra cost for The Next Platform event ($40) but pretty cool)21:30
martialtrandles: cool!21:30
trandlesI think it conflicts with the supercluster workshop though, so meh :(21:31
martialb1airo: if you reach out to CJ or Shane Canon or even our now AWS friend Christian, they might help you21:32
martialtrandles: enjoy still21:33
b1airoOh, Docker-Christian had jumped ship?21:33
b1airo*has21:34
trandlesDocker & Christian came to an "amicable split"21:34
martialb1airo: have been on the calls (I have a task to do I see on that post it :( ) and they are open to allow extra CFP21:34
martialb1airo: yes Sir, he is at AWS, and confirmed he was going to be at SC1921:35
trandlesI saw Christian at a CompBioMed meeting in Amsterdam in March...he was winding down then and looking for something else21:35
b1airoThanks martial , I wouldn't have time to do anything suitably thorough I think21:36
b1airoIs this indicative that Docker Inc is not pursuing further HPC support currently..?21:36
martial*cough*it is*cough*21:37
trandlesb1airo: +1 I believe that to be true...21:37
martial(at least not in the CE world)21:37
trandlesIf you believe the news, Docker is looking to simply survive at this point21:37
b1airoYeah, I've read the analyses suggesting they are struggling to convert market share into $$21:38
martialthe other day I was doing an apt-get update and my distro was complaining that the docker release file was now saying it was EE only21:38
martialgot me a little annoyed obviously21:39
oneswigI believe it's no longer distributed with CentOS 8 either.21:39
martialyesterday did the same update and it reverted to CE21:39
*** ociuhandu has quit IRC21:40
oneswigThere was a tentative agenda item but it does seem to have been nipped in the bud in Slack21:41
oneswigI had mistakenly thought b1airo had had his hands on the AMD GPU that supports SR-IOV, but alas it appears not21:41
*** raildo_ has quit IRC21:42
martialor maybe he is not telling :)21:42
oneswigCan anyone else share experiences...?21:43
b1airoI'm struggling to recall, but I thought it looked like AMD were going to a MR-IOV implementation21:43
*** jamesmcarthur has quit IRC21:45
b1airoThough looks like they only talk about sriov now21:45
oneswigMulti-root PCI - is that for active failover?21:46
oneswigSC Registration complete...21:50
oneswigOK - any other business?21:50
martialnothing from this end21:51
trandlesSC will be here before we know it...looking forward to seeing everyone21:52
martial+121:52
oneswigmartial: is DataMachines a member of the Ceph Foundation?  I know you guys do a lot with Ceph21:52
b1airoWondering if anyone has touched any of the swathe of data management products that are popping up in vendor catalogues ...21:53
oneswigb1airo: such as?21:54
b1airoDellEMC has a thing, HPE does too, IBM, then there's others like starfish, irods..21:55
b1airoHPE's is DMF (v7)21:55
b1airoIBM has Spectrum Discover21:56
oneswigNot me. It's a big open question.21:56
*** armax has quit IRC21:57
oneswigClose to time now, any final comments?21:58
b1airoDellEMC has InsightIQ, which is more an admin tool, but I've heard they are combining it with something else to make it more of a full featured data management thing21:58
oneswigjakecoll: be good to hear more about Chameleon traces - in Shanghai21:58
b1airoThat's it from me21:58
oneswigOK thanks y'all21:59
oneswig#endmeeting21:59
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"21:59
openstackMeeting ended Tue Oct 15 21:59:41 2019 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)21:59
openstackMinutes:        http://eavesdrop.openstack.org/meetings/scientific_sig/2019/scientific_sig.2019-10-15-21.00.html21:59
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/scientific_sig/2019/scientific_sig.2019-10-15-21.00.txt21:59
openstackLog:            http://eavesdrop.openstack.org/meetings/scientific_sig/2019/scientific_sig.2019-10-15-21.00.log.html21:59
*** jakecoll has quit IRC21:59
*** markvoelker has joined #openstack-meeting22:00
*** enriquetaso has quit IRC22:01
*** trandles has quit IRC22:01
*** bcm has joined #openstack-meeting22:01
*** oneswig has quit IRC22:03
*** markvoelker has quit IRC22:04
*** jamesmcarthur has joined #openstack-meeting22:13
*** njohnston|lunch is now known as njohnston22:24
*** rcernin has joined #openstack-meeting22:30
*** jamesmcarthur has quit IRC22:31
*** bnemec has quit IRC22:38
*** mattw4 has quit IRC22:45
*** rbudden has quit IRC22:45
*** armax has joined #openstack-meeting23:02
*** verdurin has quit IRC23:11
*** bswartz has joined #openstack-meeting23:16
*** verdurin has joined #openstack-meeting23:19
*** enriquetaso has joined #openstack-meeting23:20
*** macz has quit IRC23:22
*** mriedem has quit IRC23:27
*** jamesmcarthur has joined #openstack-meeting23:35
*** eharney has quit IRC23:41
*** markvoelker has joined #openstack-meeting23:45
*** brinzhang_ has quit IRC23:47
*** brinzhang has joined #openstack-meeting23:48
*** brinzhang_ has joined #openstack-meeting23:51
*** brinzhang has quit IRC23:54
*** markvoelker has quit IRC23:55
*** jamesmcarthur has quit IRC23:59

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!