| *** gyee has quit IRC | 00:01 | |
| *** mlavalle has quit IRC | 00:15 | |
| *** diurnalist has joined #openstack-meeting | 00:17 | |
| *** jmasud has joined #openstack-meeting | 00:20 | |
| *** yasufum has joined #openstack-meeting | 00:22 | |
| *** yasufum has quit IRC | 00:27 | |
| *** yasufum has joined #openstack-meeting | 00:27 | |
| *** armax has quit IRC | 00:33 | |
| *** yamamoto has quit IRC | 00:35 | |
| *** yamamoto has joined #openstack-meeting | 00:35 | |
| *** tetsuro has joined #openstack-meeting | 00:42 | |
| *** moguimar has quit IRC | 00:47 | |
| *** cheng1 has quit IRC | 01:14 | |
| *** Liang__ has joined #openstack-meeting | 01:15 | |
| *** cheng1 has joined #openstack-meeting | 01:17 | |
| *** yasufum has quit IRC | 01:43 | |
| *** yasufum has joined #openstack-meeting | 01:45 | |
| *** armax has joined #openstack-meeting | 01:52 | |
| *** TusharTgite has joined #openstack-meeting | 01:54 | |
| *** ricolin has joined #openstack-meeting | 02:01 | |
| *** Lucas_Gray has quit IRC | 02:12 | |
| *** jmasud has quit IRC | 02:18 | |
| *** jmasud has joined #openstack-meeting | 02:36 | |
| *** yasufum has quit IRC | 02:38 | |
| *** apetrich has quit IRC | 02:42 | |
| *** psachin has joined #openstack-meeting | 02:48 | |
| *** rcernin has quit IRC | 02:49 | |
| *** yasufum has joined #openstack-meeting | 02:52 | |
| *** rcernin has joined #openstack-meeting | 02:55 | |
| *** TusharTgite has quit IRC | 03:03 | |
| *** rcernin has quit IRC | 03:08 | |
| *** yasufum has quit IRC | 03:08 | |
| *** TusharTgite has joined #openstack-meeting | 03:08 | |
| *** jmasud has quit IRC | 03:24 | |
| *** jmasud has joined #openstack-meeting | 03:25 | |
| *** eharney has quit IRC | 03:30 | |
| *** psachin has quit IRC | 03:31 | |
| *** psachin has joined #openstack-meeting | 03:32 | |
| *** armax has quit IRC | 03:40 | |
| *** rcernin has joined #openstack-meeting | 03:50 | |
| *** eharney has joined #openstack-meeting | 03:52 | |
| *** jmasud has quit IRC | 04:05 | |
| *** jmasud has joined #openstack-meeting | 04:06 | |
| *** yamamoto has quit IRC | 04:19 | |
| *** Liang__ has quit IRC | 04:19 | |
| *** Liang__ has joined #openstack-meeting | 04:23 | |
| *** yasufum has joined #openstack-meeting | 04:26 | |
| *** markvoelker has joined #openstack-meeting | 04:27 | |
| *** evrardjp has quit IRC | 04:33 | |
| *** markvoelker has quit IRC | 04:33 | |
| *** evrardjp has joined #openstack-meeting | 04:33 | |
| *** yamamoto has joined #openstack-meeting | 04:35 | |
| *** jmasud has quit IRC | 04:40 | |
| *** vishalmanchanda has joined #openstack-meeting | 04:42 | |
| *** andrebeltrami has quit IRC | 04:49 | |
| *** diablo_rojo has quit IRC | 04:53 | |
| *** jmasud has joined #openstack-meeting | 05:03 | |
| *** diurnalist has quit IRC | 05:10 | |
| *** jmasud has quit IRC | 05:16 | |
| *** yasufum has quit IRC | 05:19 | |
| *** jmasud has joined #openstack-meeting | 05:19 | |
| *** yasufum has joined #openstack-meeting | 05:32 | |
| *** manpreet has joined #openstack-meeting | 05:59 | |
| *** yasufum has quit IRC | 06:03 | |
| *** yasufum has joined #openstack-meeting | 06:06 | |
| *** yasufum has quit IRC | 06:47 | |
| *** ircuser-1 has joined #openstack-meeting | 06:55 | |
| *** rpittau|afk is now known as rpittau | 06:56 | |
| *** slaweq has joined #openstack-meeting | 07:00 | |
| *** yasufum has joined #openstack-meeting | 07:05 | |
| *** diurnalist has joined #openstack-meeting | 07:07 | |
| *** jmasud has quit IRC | 07:08 | |
| *** apetrich has joined #openstack-meeting | 07:11 | |
| *** ttsiouts has joined #openstack-meeting | 07:13 | |
| *** psachin has quit IRC | 07:15 | |
| *** psachin has joined #openstack-meeting | 07:17 | |
| *** dklyle has quit IRC | 07:25 | |
| *** diurnalist has quit IRC | 07:40 | |
| *** rcernin has quit IRC | 07:47 | |
| *** ttsiouts has quit IRC | 07:49 | |
| *** ttsiouts has joined #openstack-meeting | 07:57 | |
| *** ralonsoh has joined #openstack-meeting | 07:57 | |
| *** Liang__ has quit IRC | 08:01 | |
| *** maciejjozefczyk has joined #openstack-meeting | 08:02 | |
| *** Liang__ has joined #openstack-meeting | 08:02 | |
| *** maciejjozefczyk has quit IRC | 08:03 | |
| *** e0ne has joined #openstack-meeting | 08:15 | |
| *** Lucas_Gray has joined #openstack-meeting | 08:24 | |
| *** cheng1 has quit IRC | 08:34 | |
| *** Lucas_Gray has quit IRC | 08:35 | |
| *** cheng1 has joined #openstack-meeting | 08:36 | |
| *** Lucas_Gray has joined #openstack-meeting | 08:36 | |
| *** jmasud has joined #openstack-meeting | 08:38 | |
| *** manpreet has quit IRC | 08:38 | |
| *** cheng1 has quit IRC | 08:44 | |
| *** cheng1 has joined #openstack-meeting | 08:50 | |
| *** priteau has joined #openstack-meeting | 08:50 | |
| *** ociuhandu has quit IRC | 08:51 | |
| *** links has joined #openstack-meeting | 08:55 | |
| *** e0ne has quit IRC | 09:02 | |
| *** e0ne has joined #openstack-meeting | 09:08 | |
| *** apetrich has quit IRC | 09:17 | |
| *** cheng1 has quit IRC | 09:24 | |
| *** apetrich has joined #openstack-meeting | 09:26 | |
| *** yasufum has quit IRC | 09:26 | |
| *** cheng1 has joined #openstack-meeting | 09:26 | |
| *** Liang__ has quit IRC | 09:33 | |
| *** yaawang_ has quit IRC | 09:40 | |
| *** diurnalist has joined #openstack-meeting | 09:40 | |
| *** e0ne has quit IRC | 09:47 | |
| *** e0ne_ has joined #openstack-meeting | 09:47 | |
| *** e0ne_ has quit IRC | 09:51 | |
| *** e0ne has joined #openstack-meeting | 09:52 | |
| *** yamamoto has quit IRC | 10:01 | |
| *** yamamoto has joined #openstack-meeting | 10:02 | |
| *** yamamoto has quit IRC | 10:02 | |
| *** oneswig has joined #openstack-meeting | 10:06 | |
| *** oneswig has left #openstack-meeting | 10:07 | |
| *** oneswig has joined #openstack-meeting | 10:10 | |
| *** diurnalist has quit IRC | 10:10 | |
| *** yaawang_ has joined #openstack-meeting | 10:17 | |
| *** rpittau is now known as rpittau|bbl | 10:18 | |
| *** ttsiouts has quit IRC | 10:22 | |
| *** ttsiouts has joined #openstack-meeting | 10:23 | |
| *** TusharTgite has quit IRC | 10:26 | |
| *** ttsiouts has quit IRC | 10:27 | |
| *** psachin has quit IRC | 10:31 | |
| *** sluna has joined #openstack-meeting | 10:31 | |
| *** oneswig_ has joined #openstack-meeting | 10:38 | |
| *** yamamoto has joined #openstack-meeting | 10:38 | |
| *** jmasud has quit IRC | 10:39 | |
| *** yamamoto has quit IRC | 10:39 | |
| *** yamamoto has joined #openstack-meeting | 10:40 | |
| *** oneswig has quit IRC | 10:41 | |
| *** e0ne has quit IRC | 10:42 | |
| *** e0ne has joined #openstack-meeting | 10:43 | |
| *** e0ne has quit IRC | 10:44 | |
| *** e0ne has joined #openstack-meeting | 10:45 | |
| *** e0ne has quit IRC | 10:46 | |
| *** e0ne has joined #openstack-meeting | 10:46 | |
| *** Lucas_Gray has quit IRC | 10:49 | |
| *** yamamoto has quit IRC | 10:50 | |
| *** noggin143 has joined #openstack-meeting | 10:56 | |
| *** yamamoto has joined #openstack-meeting | 11:00 | |
| oneswig_ | #startmeeting scientific-sig | 11:00 |
|---|---|---|
| openstack | Meeting started Wed Jun 17 11:00:41 2020 UTC and is due to finish in 60 minutes. The chair is oneswig_. Information about MeetBot at http://wiki.debian.org/MeetBot. | 11:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 11:00 |
| *** openstack changes topic to " (Meeting topic: scientific-sig)" | 11:00 | |
| openstack | The meeting name has been set to 'scientific_sig' | 11:00 |
| oneswig_ | hi all | 11:01 |
| oneswig_ | #link agenda for today https://wiki.openstack.org/wiki/Scientific_SIG#IRC_Meeting_June_17th_2020 | 11:01 |
| *** zeestrat has joined #openstack-meeting | 11:01 | |
| oneswig_ | #topic OpenStack and COVID19 workloads | 11:02 |
| *** openstack changes topic to "OpenStack and COVID19 workloads (Meeting topic: scientific-sig)" | 11:02 | |
| *** ttsiouts has joined #openstack-meeting | 11:02 | |
| *** psachin has joined #openstack-meeting | 11:03 | |
| oneswig_ | It seems like a lot of organisations are working to support workloads for COVID, in various forms | 11:03 |
| *** cheng1 has quit IRC | 11:04 | |
| verdurin | Hello - am partially here. | 11:04 |
| noggin143 | https://home.cern/news/news/cern/cern-contributes-computers-combatting-covid-19 | 11:04 |
| noggin143 | Running Folding@HOME and Rosetta@HOME on hardware about to be retired | 11:05 |
| *** cheng1 has joined #openstack-meeting | 11:05 | |
| oneswig_ | Not heard of Rosetta@home - what is that? | 11:05 |
| noggin143 | https://boinc.bakerlab.org/ | 11:06 |
| noggin143 | "With the recent COVID-19 outbreak, R@h has been used to predict the structure of proteins important to the disease as well as to produce new, stable mini-proteins to be used as potential therapeutics and diagnostics, like the one displayed above which is bound to part of the SARS-CoV-2 spike protein." | 11:06 |
| *** ttsiouts has quit IRC | 11:08 | |
| *** Zama8152 has joined #openstack-meeting | 11:08 | |
| noggin143 | pretty simple cloud-init script to start a new VM - https://clouddocs.web.cern.ch/using_openstack/contextualisation.html#install-the-folding-home-client | 11:08 |
| priteau | NSF-funded infrastructures can also accept workloads related to COVID-19, I saw it mentioned through Chameleon: https://www.chameleoncloud.org/blog/2020/03/13/chameleon-use-covid-19-projects/ | 11:08 |
| oneswig_ | Good article on the CERN blog - how many work units have been completed I wonder! | 11:09 |
| noggin143 | Stats at https://stats.foldingathome.org/teams-monthly | 11:09 |
| noggin143 | but it is important to not overload the volunteer organisations, we have some spare CPU time at the moment because the team doing hardware work is only just back on site | 11:10 |
| noggin143 | However, it's important that the core volunteers don't get displaced by this temporary contribution | 11:10 |
| noggin143 | We're also in discussions with the WHO to see how we can help as they are just down the road from us in Geneva | 11:11 |
| *** ociuhandu has joined #openstack-meeting | 11:12 | |
| oneswig_ | verdurin: are you close to the source of research in Oxford? | 11:12 |
| verdurin | oneswig_: yes, there's a lot going on here. | 11:13 |
| noggin143 | verdurin: we're running F@H at the moment but if there is a better application to run, I can put you in touch with the CERN COVID folk | 11:15 |
| oneswig_ | Aside from simulation work, what about the epidemiology, public health, contact tracing etc. | 11:18 |
| verdurin | noggin143: thanks. As usual there is a myriad different applications in use. | 11:18 |
| verdurin | There is also the RECOVERY clinical trial, and the vaccine trial that originates from a couple of buildings down. | 11:20 |
| oneswig_ | I recall from somewhere that protein simulations don't have significant data requirements, does that also apply for your workloads Adam? | 11:20 |
| verdurin | A lot of the workloads I know about are similar to our normal genomic ones, hence they do have significant data requirements. | 11:21 |
| verdurin | There is also demand from data generators e.g. sequencing, proteomics. | 11:23 |
| noggin143 | I guess there are also privacy concerns for some of the applications, which don't apply for the volunteer projects like F@H | 11:23 |
| verdurin | Yes. In some cases various flavours of patient data. | 11:23 |
| *** cheng1 has quit IRC | 11:23 | |
| oneswig_ | All of which make them difficult to spread to other places. | 11:23 |
| oneswig_ | In the SIG session at the PTG there was reference to this tracking project in India - https://www.aarogyasetu.gov.in/ | 11:24 |
| oneswig_ | Prakash are you here? | 11:25 |
| verdurin | One aspect that may be of interest here is provisioning of resources such as RStudio Server on cloud instances, where in the past dedicated nodes were used. | 11:25 |
| *** ttsiouts has joined #openstack-meeting | 11:25 | |
| oneswig_ | verdurin: is that licensed software? That can often be tricky | 11:26 |
| verdurin | It depends. There are different flavours, free and licensed. | 11:26 |
| *** ttsiouts has quit IRC | 11:26 | |
| *** ttsiouts has joined #openstack-meeting | 11:26 | |
| oneswig_ | Is RStudio being used for post-processing and visualisation of batch simulations? | 11:27 |
| verdurin | It's mainly for code development, I believe. | 11:28 |
| verdurin | Very fast-moving area. | 11:28 |
| oneswig_ | I am sure it is. | 11:29 |
| *** cheng1 has joined #openstack-meeting | 11:29 | |
| oneswig_ | I'd be interested to hear how Public Health England's OpenStack systems are being applied for the modelling work they do. | 11:32 |
| oneswig_ | Anything else to raise on this subject before we move on? | 11:34 |
| sluna | Just one more comment: AFAIK RStudio is an IDE for R code development. RStudio Server is useful when you deploy it next to big data and powerful compute so the researcher connects to it through a web browser to do interactive analyses. | 11:35 |
| oneswig_ | Hi sluna, thanks for clarifying. | 11:35 |
| verdurin | The dividing line is a bit muddier, but it's not that important. | 11:36 |
| *** rfolco|rover has joined #openstack-meeting | 11:36 | |
| oneswig_ | We haven't covered the IOT-class issues of tracking populations but I don't think anyone's here who is working on that. | 11:37 |
| Zama8152 | anyone using or recommending elastic search for analyzing data and monitoring movements | 11:39 |
| oneswig_ | Hello Zama8152, welcome :-) | 11:40 |
| oneswig_ | ElasticSearch is very good for indexing and retrieval of JSON-encoded data. | 11:41 |
| Zama8152 | I'd be interested at knowing what better tools to use with regards to tracking population | 11:41 |
| Zama8152 | oneswig_Hi, thanks for the invite.. | 11:41 |
| noggin143 | We use it mainly for structured search like logs but ES is pretty flexible | 11:42 |
| *** tetsuro_ has joined #openstack-meeting | 11:42 | |
| Zama8152 | researchers on my side use elastic search to analyze Covid-19 self-screening data and monitoring movements of citizens in areas of interest to understand the effectiveness and impact of the lockdown, this information is used by the National Department of health in making relevant decisions | 11:44 |
| *** raildo has joined #openstack-meeting | 11:44 | |
| oneswig_ | Zama8152: it's good for data that doesn't always have the same structure - semi-structured perhaps. If your data is always of the same format, you could also consider an SQL database like postgres | 11:44 |
| oneswig_ | How big is the Elastic Search, will it grow to be massive? | 11:46 |
| *** tetsuro has quit IRC | 11:46 | |
| Zama8152 | Yeah the data does'nt have the same structure and progress db is indeed used .. | 11:46 |
| Zama8152 | *postgress | 11:46 |
| oneswig_ | One other thought on ElasticSearch is that performance can be limited by IOPS and IO latency - it will benefit from local SSD storage in your hypervisors, if you have the local disk capacity for it. | 11:49 |
| oneswig_ | Is your group developing everything from scratch? There probably isn't much precedent for this kind of application. | 11:49 |
| noggin143 | we run in VMs with 4 1TB SSDs per server - works OK, you lose some IOPS and rebalancing can take a while | 11:50 |
| noggin143 | old presentation at https://indico.cern.ch/event/717615/contributions/3033517/attachments/1676735/2692320/ES_Security.pdf, now migrating to ES7 | 11:51 |
| *** tetsuro_ has quit IRC | 11:51 | |
| oneswig_ | Is that 250TB of data in ES back in 2018? Must be even more now... | 11:52 |
| noggin143 | the security one is getting pretty big... used for forensics but otherwise, we purge aggressively with archive in HDFS. | 11:54 |
| Zama8152 | currently they are running on 1 vm with 80GB memory and 1TB ssd | 11:54 |
| oneswig_ | We are short on time - final comments on this topic? | 11:55 |
| Zama8152 | They have complained about memory and I am moving them to a vm with 160GB memory.. | 11:55 |
| oneswig_ | Hopefully that will help! | 11:56 |
| Zama8152 | I have to attend another meeting right now.. Thanks for your input.. | 11:56 |
| oneswig_ | Thanks for coming Zama8152 | 11:56 |
| oneswig_ | OK, a couple more events to mention | 11:57 |
| *** tetsuro has joined #openstack-meeting | 11:57 | |
| oneswig_ | #topic online conferences | 11:57 |
| *** openstack changes topic to "online conferences (Meeting topic: scientific-sig)" | 11:57 | |
| oneswig_ | I'm sure there are plenty of others going on | 11:57 |
| oneswig_ | #link High Performance Container Workshop, 16-18 June https://hpcw.github.io/ | 11:57 |
| oneswig_ | Content looks quite interesting but I haven't listened in to any of the sessions so far | 11:58 |
| oneswig_ | #link Virtual ISC https://www.isc-hpc.com/ | 11:58 |
| *** psachin has quit IRC | 11:58 | |
| oneswig_ | I'm not sure how much of ISC will be virtual but at least we don't have to find a hotel in Frankfurt to participate. | 11:59 |
| *** psachin has joined #openstack-meeting | 11:59 | |
| oneswig_ | The OpenDev events are imminent too | 12:00 |
| oneswig_ | The link eludes me alas and we must close | 12:01 |
| oneswig_ | Thanks all | 12:01 |
| oneswig_ | #endmeeting | 12:01 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 12:01 | |
| verdurin | Bye. | 12:01 |
| openstack | Meeting ended Wed Jun 17 12:01:43 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 12:01 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/scientific_sig/2020/scientific_sig.2020-06-17-11.00.html | 12:01 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/scientific_sig/2020/scientific_sig.2020-06-17-11.00.txt | 12:01 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/scientific_sig/2020/scientific_sig.2020-06-17-11.00.log.html | 12:01 |
| sluna | bye! | 12:01 |
| *** Lucas_Gray has joined #openstack-meeting | 12:02 | |
| *** noggin143 has quit IRC | 12:04 | |
| *** rpittau|bbl is now known as rpittau | 12:06 | |
| *** diurnalist has joined #openstack-meeting | 12:07 | |
| *** ttsiouts has quit IRC | 12:16 | |
| *** ttsiouts has joined #openstack-meeting | 12:16 | |
| *** Lucas_Gray has quit IRC | 12:16 | |
| *** Lucas_Gray has joined #openstack-meeting | 12:19 | |
| *** cheng1 has quit IRC | 12:20 | |
| *** ttsiouts has quit IRC | 12:21 | |
| *** yasufum has joined #openstack-meeting | 12:22 | |
| *** cheng1 has joined #openstack-meeting | 12:22 | |
| *** Wryhder has joined #openstack-meeting | 12:23 | |
| *** andrebeltrami has joined #openstack-meeting | 12:24 | |
| *** Lucas_Gray has quit IRC | 12:24 | |
| *** ttsiouts has joined #openstack-meeting | 12:25 | |
| *** yasufum has quit IRC | 12:26 | |
| *** Wryhder has quit IRC | 12:28 | |
| *** Lucas_Gray has joined #openstack-meeting | 12:32 | |
| *** diurnalist has quit IRC | 12:40 | |
| *** slaweq_ has joined #openstack-meeting | 12:47 | |
| *** slaweq has quit IRC | 12:48 | |
| *** Zama8152 has quit IRC | 12:58 | |
| *** cheng1 has quit IRC | 13:01 | |
| *** slaweq_ is now known as slaweq | 13:03 | |
| *** cheng1 has joined #openstack-meeting | 13:06 | |
| *** tetsuro has quit IRC | 13:15 | |
| *** ttsiouts has quit IRC | 13:18 | |
| *** ttsiouts has joined #openstack-meeting | 13:19 | |
| *** ttsiouts has quit IRC | 13:19 | |
| *** ttsiouts has joined #openstack-meeting | 13:19 | |
| *** cheng1 has quit IRC | 13:22 | |
| *** moguimar has joined #openstack-meeting | 13:24 | |
| *** diurnalist has joined #openstack-meeting | 13:33 | |
| *** ZhuXiaoYu has joined #openstack-meeting | 13:36 | |
| *** e0ne has quit IRC | 13:37 | |
| *** e0ne has joined #openstack-meeting | 13:37 | |
| *** TrevorV has joined #openstack-meeting | 13:37 | |
| *** rpittau is now known as rpittau|brb | 13:39 | |
| *** priteau has quit IRC | 13:40 | |
| *** ricolin has quit IRC | 13:40 | |
| *** e0ne has quit IRC | 13:48 | |
| *** e0ne_ has joined #openstack-meeting | 13:48 | |
| *** Liang__ has joined #openstack-meeting | 13:52 | |
| *** Liang__ is now known as LiangFang | 13:53 | |
| *** rpittau|brb is now known as rpittau | 13:53 | |
| *** lpetrut has joined #openstack-meeting | 13:56 | |
| *** priteau has joined #openstack-meeting | 13:56 | |
| *** LiangFang has quit IRC | 13:57 | |
| *** maciejjozefczyk has joined #openstack-meeting | 14:01 | |
| *** ZhuJoseph has joined #openstack-meeting | 14:04 | |
| *** yamamoto has quit IRC | 14:06 | |
| *** ZhuXiaoYu has quit IRC | 14:07 | |
| *** liuyulong has joined #openstack-meeting | 14:07 | |
| liuyulong | #startmeeting neutron_l3 | 14:08 |
| openstack | Meeting started Wed Jun 17 14:08:46 2020 UTC and is due to finish in 60 minutes. The chair is liuyulong. Information about MeetBot at http://wiki.debian.org/MeetBot. | 14:08 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 14:08 |
| *** openstack changes topic to " (Meeting topic: neutron_l3)" | 14:08 | |
| openstack | The meeting name has been set to 'neutron_l3' | 14:08 |
| liuyulong | Sorry, a bit late... | 14:08 |
| liuyulong | slaweq, haleyb, ping | 14:09 |
| slaweq | hi | 14:09 |
| haleyb | hi | 14:09 |
| liuyulong | hi | 14:09 |
| liuyulong | Alright, let's start | 14:10 |
| liuyulong | #topic Announcements | 14:10 |
| *** openstack changes topic to "Announcements (Meeting topic: neutron_l3)" | 14:10 | |
| liuyulong | #link http://lists.openstack.org/pipermail/openstack-discuss/2020-June/015368.html | 14:11 |
| liuyulong | This is the ptg summary from the Virtual PTG. | 14:11 |
| liuyulong | Thanks slaweq for the detailed summary. | 14:12 |
| liuyulong | #link http://kaplonski.pl/images/Virtual_PTG_2020/photo_3.png | 14:13 |
| liuyulong | I saw you handsome guys. | 14:13 |
| liuyulong | #link http://eavesdrop.openstack.org/meetings/networking/2020/networking.2020-06-16-14.00.log.html#l-13 | 14:14 |
| liuyulong | This is the announcements from the team meeting yesterday. | 14:15 |
| liuyulong | We are in Victoria devloping cycle now, so each spec should be moved to Victoria folder. | 14:16 |
| *** ttsiouts has quit IRC | 14:16 | |
| liuyulong | OK, no more from me now. | 14:16 |
| slaweq | :) | 14:17 |
| *** ttsiouts has joined #openstack-meeting | 14:17 | |
| liuyulong | Neutron CI is down, any idea? | 14:17 |
| liuyulong | #link https://bugs.launchpad.net/neutron/+bug/1883601 | 14:18 |
| openstack | Launchpad bug 1883601 in neutron "ovn based neutron gate jobs failing 100% of times" [Critical,In progress] - Assigned to Jakub Libosvar (libosvar) | 14:18 |
| liuyulong | This is new bug, but seems the real problem is not fixed either. | 14:19 |
| liuyulong | OK... | 14:20 |
| liuyulong | #link https://review.opendev.org/#/c/735536/ | 14:20 |
| liuyulong | This is the gatefix | 14:20 |
| liuyulong | Next topic | 14:20 |
| liuyulong | #topic Bugs | 14:20 |
| *** openstack changes topic to "Bugs (Meeting topic: neutron_l3)" | 14:20 | |
| liuyulong | #link http://lists.openstack.org/pipermail/openstack-discuss/2020-June/015178.html | 14:21 |
| liuyulong | #link http://lists.openstack.org/pipermail/openstack-discuss/2020-June/015323.html | 14:21 |
| liuyulong | #link http://lists.openstack.org/pipermail/openstack-discuss/2020-June/015442.html | 14:21 |
| liuyulong | We have a long list.... | 14:21 |
| *** ttsiouts has quit IRC | 14:21 | |
| *** ttsiouts has joined #openstack-meeting | 14:22 | |
| liuyulong | First one | 14:24 |
| liuyulong | #link https://bugs.launchpad.net/neutron/+bug/1880969 | 14:24 |
| openstack | Launchpad bug 1880969 in neutron "Creating FIP takes time" [Low,New] | 14:24 |
| ralonsoh | IMO, the times spent by the server is ok | 14:25 |
| ralonsoh | c#2 of this LP | 14:25 |
| ralonsoh | (only the Neutron server times) | 14:25 |
| liuyulong | ralonsoh, yes, agreed. The HTTP response time from the neutron server log should be considered first. | 14:26 |
| liuyulong | "GET /v2.0/ports?network_id=55c74232-825a-4a4a-b53d-5b4b7aa4ad74&device_owner=network%3Adhcp HTTP/1.1" status: 200 len: 1272 time: 0.0676231 | 14:27 |
| liuyulong | A simple case from my deployment. | 14:28 |
| liuyulong | A pattern for logstash should be useful. | 14:28 |
| liuyulong | #link https://bugs.launchpad.net/neutron/+bug/1880532 | 14:28 |
| openstack | Launchpad bug 1880532 in neutron "[RFE]L3 Router should support ECMP" [Wishlist,New] - Assigned to XiaoYu Zhu (honglan0914) | 14:28 |
| liuyulong | I have reviewed the spec one time. | 14:29 |
| liuyulong | #link https://review.opendev.org/#/c/729532/ | 14:29 |
| slaweq | I have to review this spec too | 14:29 |
| *** rh-jelabarre has joined #openstack-meeting | 14:30 | |
| liuyulong | In general, the final use scenarios looks limited to the loadbalancer. The main point is not in the Neutron side. | 14:31 |
| liuyulong | So let's continue the discussion on the gerrit. | 14:31 |
| slaweq | yes, there are some suggestions that it can be done with existing neutron API IIRC | 14:31 |
| ZhuJoseph | My current plan is to add a new function to extraroutedb.py to handle this requirement. | 14:32 |
| liuyulong | Hi, you are here. | 14:32 |
| liuyulong | "XiaoYu Zhu" it's you? | 14:33 |
| ZhuJoseph | and use api like :/v2.0/routers/27757e09-fb6a-4196-957d-cdce604f087e/remove_ecmps | 14:33 |
| ZhuJoseph | yes | 14:33 |
| ZhuJoseph | I am | 14:33 |
| liuyulong | Welcome | 14:33 |
| *** yamamoto has joined #openstack-meeting | 14:34 | |
| *** yamamoto has quit IRC | 14:34 | |
| *** mlavalle has joined #openstack-meeting | 14:34 | |
| *** yamamoto has joined #openstack-meeting | 14:34 | |
| *** psachin has quit IRC | 14:34 | |
| liuyulong | ZhuJoseph, if there are some existing code or POC, you may submit it in parallel, that could also be useful for the upstream team to understand your real requirement. | 14:36 |
| liuyulong | And do not forget to add the link to the spec. | 14:37 |
| *** links has quit IRC | 14:37 | |
| liuyulong | One more thing, you should move specs/ussuri/l3-router-support-ecmp.rst, to the Virtual folder. | 14:37 |
| liuyulong | s/Victoria | 14:37 |
| ZhuJoseph | ok | 14:37 |
| liuyulong | OK, next | 14:39 |
| liuyulong | #link https://bugs.launchpad.net/neutron/+bug/1881995 | 14:39 |
| openstack | Launchpad bug 1881995 in neutron "Centralized SNAT failover does not recover until "systemctl restart neutron-l3-agent" on transferred node" [Medium,In progress] - Assigned to Ann Taraday (akamyshnikova) | 14:39 |
| liuyulong | We already have some discussion on the LP, and here is a workaround fix: | 14:39 |
| liuyulong | #link https://review.opendev.org/#/c/734070/ | 14:40 |
| *** yamamoto has quit IRC | 14:41 | |
| liuyulong | For the fix, IMO, it partially revert the fix of the original fix of https://review.opendev.org/#/c/692352/ | 14:41 |
| ralonsoh | IMO this is a workaround | 14:41 |
| liuyulong | in some case | 14:41 |
| ralonsoh | but if accepted and does not clash with any other part of the code | 14:41 |
| ralonsoh | I'm ok | 14:41 |
| ralonsoh | you know better this code... | 14:42 |
| liuyulong | The main problem is in the namespace deletion based on my current research. | 14:42 |
| liuyulong | #link https://bugs.launchpad.net/neutron/+bug/1881995/comments/7 | 14:43 |
| openstack | Launchpad bug 1881995 in neutron "Centralized SNAT failover does not recover until "systemctl restart neutron-l3-agent" on transferred node" [Medium,In progress] - Assigned to Ann Taraday (akamyshnikova) | 14:43 |
| liuyulong | #link https://bugs.launchpad.net/neutron/+bug/1881995/comments/8 | 14:43 |
| liuyulong | I will add some log for this issue as a start. | 14:43 |
| *** dklyle has joined #openstack-meeting | 14:44 | |
| ralonsoh | good finding in c#7 | 14:45 |
| liuyulong | ralonsoh, the pyroute2 namespace deleting could be related. I may need your help. : ) | 14:45 |
| ralonsoh | sure | 14:45 |
| ralonsoh | but where is this called? | 14:45 |
| liuyulong | Wait a sec | 14:45 |
| ralonsoh | no no | 14:45 |
| ralonsoh | I mean | 14:45 |
| ralonsoh | in this executing | 14:45 |
| ralonsoh | why the namespace is deleted? | 14:45 |
| *** manuvakery has joined #openstack-meeting | 14:46 | |
| ralonsoh | *execution | 14:46 |
| liuyulong | #link https://github.com/openstack/neutron/blob/master/neutron/agent/linux/ip_lib.py#L705 | 14:46 |
| liuyulong | #link https://github.com/openstack/neutron/blob/master/neutron/agent/linux/ip_lib.py#L906 | 14:46 |
| ralonsoh | yes and the ns is deleted, so that's ok | 14:47 |
| ralonsoh | but why the ns was deleted? | 14:47 |
| liuyulong | And finally, https://github.com/openstack/neutron/blob/master/neutron/privileged/agent/linux/ip_lib.py#L542 | 14:47 |
| liuyulong | the qrouter namespace was not deleted successfully. | 14:48 |
| liuyulong | bug/1881995/comments/7 | 14:48 |
| liuyulong | Or maybe it is concurrent query and deleting. | 14:50 |
| liuyulong | Delete namespace does not have much log now, I will add some. | 14:50 |
| liuyulong | OK, next one | 14:53 |
| liuyulong | #link https://bugs.launchpad.net/neutron/+bug/1882860 | 14:53 |
| openstack | Launchpad bug 1882860 in neutron "after FIP is assigned vm lost network connection" [Undecided,Incomplete] | 14:53 |
| liuyulong | It's a ovn-router related report. | 14:53 |
| liuyulong | Jakub has left a potential fix of the issue and some questions, no response for now. | 14:54 |
| liuyulong | Next | 14:55 |
| liuyulong | #link https://bugs.launchpad.net/neutron/+bug/1883321 | 14:55 |
| openstack | Launchpad bug 1883321 in neutron "Neutron OpenvSwitch DVR - connection problem" [High,New] | 14:55 |
| liuyulong | This is really a complicated issue. | 14:55 |
| liuyulong | As I said in the fix, there are tons of cases for the real deployment, for instance, DVR, DVR + HA, openflow firewall, network node mixed compute services... | 14:57 |
| liuyulong | I have a long list. | 14:57 |
| liuyulong | Let's continue the talk on LP bug. | 14:58 |
| liuyulong | Last one | 14:58 |
| liuyulong | #link https://bugs.launchpad.net/neutron/+bug/1883089 | 14:58 |
| openstack | Launchpad bug 1883089 in neutron "[L3] floating IP failed to bind due to no agent gateway port(fip-ns)" [Medium,In progress] - Assigned to LIU Yulong (dragon889) | 14:58 |
| liuyulong | reported by me | 14:58 |
| liuyulong | I have two patches. | 14:58 |
| liuyulong | #link https://review.opendev.org/#/c/735432/ | 14:59 |
| liuyulong | #link https://review.opendev.org/#/c/735762/ | 14:59 |
| liuyulong | The test case should be simple, just create a fake external network, and create router/network/subnet/VM. | 14:59 |
| liuyulong | Then just see the changes of fip-namespace on hosts and DvrFipGatewayPortAgentBinding in DB. | 15:00 |
| liuyulong | #link https://review.opendev.org/#/c/702547/ | 15:00 |
| liuyulong | IMO, this fix just missed that DVR related clean up action. | 15:01 |
| liuyulong | OK, we are out of time. | 15:01 |
| *** ttsiouts has quit IRC | 15:01 | |
| liuyulong | #endmeeting | 15:01 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 15:01 | |
| openstack | Meeting ended Wed Jun 17 15:01:28 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:01 |
| ralonsoh | bye | 15:01 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/neutron_l3/2020/neutron_l3.2020-06-17-14.08.html | 15:01 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/neutron_l3/2020/neutron_l3.2020-06-17-14.08.txt | 15:01 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/neutron_l3/2020/neutron_l3.2020-06-17-14.08.log.html | 15:01 |
| liuyulong | See you guys online. | 15:01 |
| *** ttsiouts has joined #openstack-meeting | 15:01 | |
| *** ttsiouts has quit IRC | 15:06 | |
| *** Lucas_Gray has quit IRC | 15:12 | |
| *** ttsiouts has joined #openstack-meeting | 15:17 | |
| *** aprice has joined #openstack-meeting | 15:31 | |
| *** ttsiouts has quit IRC | 15:33 | |
| *** ttsiouts has joined #openstack-meeting | 15:33 | |
| *** jiaopengju1 has quit IRC | 15:35 | |
| *** jiaopengju1 has joined #openstack-meeting | 15:36 | |
| *** ttsiouts has quit IRC | 15:38 | |
| *** e0ne_ has quit IRC | 15:40 | |
| *** e0ne has joined #openstack-meeting | 15:46 | |
| *** gyee has joined #openstack-meeting | 15:58 | |
| *** e0ne has quit IRC | 16:01 | |
| *** armax has joined #openstack-meeting | 16:08 | |
| *** rpittau is now known as rpittau|afk | 16:17 | |
| *** jmasud has joined #openstack-meeting | 16:20 | |
| *** ociuhandu_ has joined #openstack-meeting | 16:38 | |
| *** ociuhandu has quit IRC | 16:42 | |
| *** ociuhandu_ has quit IRC | 16:43 | |
| *** ociuhandu has joined #openstack-meeting | 16:47 | |
| *** ociuhandu has quit IRC | 16:51 | |
| *** liuyulong has quit IRC | 16:56 | |
| *** maciejjozefczyk has quit IRC | 16:56 | |
| *** maciejjozefczyk has joined #openstack-meeting | 16:57 | |
| *** diablo_rojo has joined #openstack-meeting | 16:59 | |
| *** lpetrut has quit IRC | 17:02 | |
| *** e0ne has joined #openstack-meeting | 17:03 | |
| *** oneswig_ has quit IRC | 17:18 | |
| *** e0ne has quit IRC | 17:43 | |
| *** jmasud has quit IRC | 17:44 | |
| *** manuvakery has quit IRC | 18:34 | |
| *** jmasud has joined #openstack-meeting | 18:36 | |
| *** jamesmcarthur has joined #openstack-meeting | 18:47 | |
| *** jamesmcarthur has quit IRC | 18:54 | |
| *** jamesmcarthur has joined #openstack-meeting | 18:55 | |
| *** jamesmcarthur_ has joined #openstack-meeting | 18:59 | |
| *** jmasud has quit IRC | 19:00 | |
| *** diurnalist has quit IRC | 19:00 | |
| *** jamesmcarthur has quit IRC | 19:01 | |
| *** jmasud has joined #openstack-meeting | 19:18 | |
| *** ralonsoh has quit IRC | 19:22 | |
| *** diurnalist has joined #openstack-meeting | 19:30 | |
| *** ttsiouts has joined #openstack-meeting | 19:32 | |
| *** jmasud has quit IRC | 19:36 | |
| *** vishalmanchanda has quit IRC | 19:38 | |
| *** jmasud has joined #openstack-meeting | 19:41 | |
| *** ttsiouts has quit IRC | 19:42 | |
| *** ttsiouts has joined #openstack-meeting | 19:43 | |
| *** jamesmcarthur_ has quit IRC | 19:45 | |
| *** jamesmcarthur has joined #openstack-meeting | 19:46 | |
| *** ttsiouts has quit IRC | 19:47 | |
| *** ociuhandu has joined #openstack-meeting | 19:52 | |
| *** zhuxiaoyu_inspur has joined #openstack-meeting | 20:09 | |
| *** ktsuyuzaki has joined #openstack-meeting | 20:11 | |
| *** Lucas_Gray has joined #openstack-meeting | 20:11 | |
| *** zeestrat_ has joined #openstack-meeting | 20:11 | |
| *** ktsuyuzaki is now known as kota__ | 20:12 | |
| *** tobberydberg_ has joined #openstack-meeting | 20:12 | |
| *** mbuil_ has joined #openstack-meeting | 20:13 | |
| *** irclogbot_0 has quit IRC | 20:14 | |
| *** gyee has quit IRC | 20:17 | |
| *** ZhuJoseph has quit IRC | 20:18 | |
| *** tobberydberg has quit IRC | 20:18 | |
| *** mbuil has quit IRC | 20:18 | |
| *** zeestrat has quit IRC | 20:18 | |
| *** masayukig has quit IRC | 20:18 | |
| *** amorin has quit IRC | 20:18 | |
| *** kota_ has quit IRC | 20:18 | |
| *** zeestrat_ is now known as zeestrat | 20:18 | |
| *** amorin has joined #openstack-meeting | 20:20 | |
| *** irclogbot_3 has joined #openstack-meeting | 20:21 | |
| *** jmasud has quit IRC | 20:22 | |
| *** masayukig has joined #openstack-meeting | 20:24 | |
| *** gyee has joined #openstack-meeting | 20:31 | |
| *** jamesmcarthur has quit IRC | 20:31 | |
| *** jamesmcarthur has joined #openstack-meeting | 20:32 | |
| *** jamesmcarthur has quit IRC | 20:37 | |
| *** csatari has quit IRC | 20:39 | |
| *** aprice has quit IRC | 20:40 | |
| *** knikolla has quit IRC | 20:40 | |
| *** knikolla has joined #openstack-meeting | 20:42 | |
| *** patrickeast has quit IRC | 20:43 | |
| *** aprice has joined #openstack-meeting | 20:44 | |
| *** aprice has quit IRC | 20:48 | |
| *** ociuhandu has quit IRC | 20:48 | |
| *** PrinzElvis has quit IRC | 20:49 | |
| *** priteau has quit IRC | 20:49 | |
| *** thgcorrea has quit IRC | 20:50 | |
| *** knikolla has quit IRC | 20:52 | |
| *** ttsiouts has joined #openstack-meeting | 20:52 | |
| *** PrinzElvis has joined #openstack-meeting | 20:53 | |
| *** jmasud has joined #openstack-meeting | 20:53 | |
| *** patrickeast has joined #openstack-meeting | 20:53 | |
| *** ttsiouts has quit IRC | 20:57 | |
| *** PrinzElvis has quit IRC | 20:57 | |
| *** patchbot has joined #openstack-meeting | 20:58 | |
| *** diablo_rojo_phon has joined #openstack-meeting | 20:59 | |
| timburke | #startmeeting swift | 21:00 |
| openstack | Meeting started Wed Jun 17 21:00:03 2020 UTC and is due to finish in 60 minutes. The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot. | 21:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 21:00 |
| *** openstack changes topic to " (Meeting topic: swift)" | 21:00 | |
| openstack | The meeting name has been set to 'swift' | 21:00 |
| timburke | who's here for the swift meeting? | 21:00 |
| *** patrickeast has quit IRC | 21:00 | |
| kota__ | hi | 21:00 |
| *** maciejjozefczyk has quit IRC | 21:00 | |
| *** patrickeast has joined #openstack-meeting | 21:01 | |
| tdasilva | half here | 21:01 |
| rledisez | o/ | 21:01 |
| alecuyer | o/ | 21:01 |
| mattoliverau | o/ | 21:02 |
| timburke | as usual, the agenda's at https://wiki.openstack.org/wiki/Meetings/Swift | 21:02 |
| timburke | first up | 21:02 |
| timburke | #topic gate | 21:02 |
| *** openstack changes topic to "gate (Meeting topic: swift)" | 21:03 | |
| timburke | you may have noticed that nothing was passing the last couple days | 21:03 |
| *** e0ne has joined #openstack-meeting | 21:03 | |
| timburke | i think it's all resolved now, but i wanted to give an overview of the issues | 21:03 |
| timburke | #link http://lists.openstack.org/pipermail/openstack-discuss/2020-June/015432.html | 21:03 |
| timburke | there was an issue with uwsgi that broke our grenade job (along with *everyone else*) | 21:04 |
| *** aprice has joined #openstack-meeting | 21:04 | |
| *** jamesmcarthur has joined #openstack-meeting | 21:04 | |
| timburke | the qa team's been all over it, and the resolution merged last night | 21:04 |
| timburke | then there was another issue with our probe tests (most visibly; also affected the ceph s3 tests and rolling upgrade tests) | 21:05 |
| *** patrickeast has quit IRC | 21:05 | |
| timburke | pretty sure it was the result of pip no longer being available in the base images | 21:06 |
| timburke | #link http://lists.openstack.org/pipermail/openstack-discuss/2020-June/015425.html | 21:06 |
| clayg | o/ | 21:06 |
| timburke | the fix there did require a change to our tooling, but that merged this morning | 21:06 |
| timburke | https://review.opendev.org/#/c/735992 | 21:06 |
| patchbot | patch 735992 - swift - Use ensure-pip role (MERGED) - 5 patch sets | 21:06 |
| *** diurnalist has quit IRC | 21:06 | |
| timburke | i rechecked a bunch of changes about three hours ago, but everything's all backed up so none of those have actually posted new results yet | 21:08 |
| clayg | thanks for fixing the gate timburke ! | 21:08 |
| *** zaitcev has joined #openstack-meeting | 21:08 | |
| timburke | if anyone sees more issues, holler! | 21:08 |
| timburke | #topic memcache and container failures | 21:09 |
| *** openstack changes topic to "memcache and container failures (Meeting topic: swift)" | 21:09 | |
| *** aprice has quit IRC | 21:10 | |
| timburke | so last week i had all replicas of a container get overloaded | 21:10 |
| clayg | yeah that was pretty cool | 21:10 |
| clayg | actually I wasn't there - it SOUNDED cool (after the fact) | 21:11 |
| timburke | which led me to notice that when the proxy hands back a 503 (because we got timeout, timeout, timeout, 404, 404, 404), we go evict memcache | 21:11 |
| timburke | #link https://bugs.launchpad.net/swift/+bug/1883211 | 21:11 |
| openstack | Launchpad bug 1883211 in OpenStack Object Storage (swift) "get_container_info 503s shouldn't try to clear memcache" [Undecided,In progress] | 21:11 |
| *** rfolco|rover has quit IRC | 21:12 | |
| timburke | which meant that once info fell out of cache while there were hundreds of concurrent requests trying to do things in the container, it couldn't *stay in cache* even when some of those HEADs to try to repopulate managed to get back to the proxy | 21:13 |
| *** jamesmcarthur has quit IRC | 21:13 | |
| timburke | i proposed https://review.opendev.org/#/c/735359/ to fix it (basically, follow what the docstring said to do in set_info_cache), but i was wondering if anyone else has seen similar behavior | 21:14 |
| patchbot | patch 735359 - swift - proxy: Stop killing memcache entries on 5xx responses - 4 patch sets | 21:14 |
| clayg | moral of the story: don't let your primaries get overloaded - but when you do! you know... be better swift | 21:15 |
| zaitcev | I haven't but it sounds persuasive. | 21:15 |
| timburke | note that prior to https://review.opendev.org/#/c/667411/ (from about a year ago), we would've been caching a 404 | 21:16 |
| patchbot | patch 667411 - swift - Return 503 when primary containers can't respond (MERGED) - 2 patch sets | 21:16 |
| clayg | I was reluctant to go mucking with such old code; but once I realized we're a few iterations away for untangling all the things that could possibly lead to clients+sharder overwhelming a root db... I loaded it in my head and it makes sense to me | 21:16 |
| timburke | (funny enough, it was definitely the same cluster and quite possibly the same container that prompted that change, too) | 21:17 |
| clayg | I'm not even sure we really *intended* to clear the cache on error - the history of how it evolved reads more like it just happened on accident as the code evolved | 21:17 |
| *** diurnalist has joined #openstack-meeting | 21:17 | |
| clayg | certainly all the primaries being overloaded isn't something that comes up often - it's possible it was just never bad enough (or when it go that bad there was like OTHER thing that were ALSO bad - like... idk... not enough ratelimiting) | 21:18 |
| timburke | yeah, it sure *seemed like* https://review.opendev.org/#/c/30481/ didn't mean to change behavior like that | 21:18 |
| patchbot | patch 30481 - swift - get_info - removes duplicate code (Take 3) (MERGED) - 17 patch sets | 21:18 |
| clayg | anyway - even if I'm wrong and someone thought they had a good reason to flush cache on error... I can't convince myself anymore it's a good idea | 21:18 |
| *** ociuhandu has joined #openstack-meeting | 21:19 | |
| clayg | when the backend service is saying "please back off" - GO HARDER - is rarely going to be the BEST plan 😁 | 21:19 |
| clayg | anyway; we're shipping it - and at least two cores like the change - so it'll probably merge eventually, but it's fairly fresh and we're open to better ideas! | 21:20 |
| zaitcev | The problem is usually the cache being stale. If the error is indicative of the main storage being changed without cache flushed, then cache needs to be flushed. not sure if 503 is such. The 409 seems like a candidate for suspicion. | 21:20 |
| timburke | *nod* i'm not sure that the container server can send back a 409 on GET or HEAD, but good thinking. will check | 21:22 |
| clayg | which 409? timburke the 404 cache is so weird... to think of that as a "remediation" I mean... maybe a client does a PUT and ends up handoffs!? I don't think that behavior was anymore desirable really. | 21:22 |
| clayg | I'm most happy about the tests - it's now defined behavior - we're saying on 503 we don't want to flush the cache | 21:22 |
| clayg | if we change our minds later at least we have tests that can express what we want - and we won't accidently forget to think about it next time we're working in there | 21:23 |
| *** jmasud has quit IRC | 21:23 | |
| clayg | i'm gunna go +A it right now - I'm totally talking myself into it!!! 😁 | 21:23 |
| timburke | lol | 21:23 |
| *** ociuhandu has quit IRC | 21:24 | |
| *** rh-jelabarre has quit IRC | 21:24 | |
| *** aprice has joined #openstack-meeting | 21:24 | |
| timburke | so as clayg mentioned, the trouble seemed to come from the shard stat reporting. fortunately, we've already landed a latch for that | 21:24 |
| clayg | timburke: so you're saying before p 30481 you think we'd leave the cache alone on 503? Or just that was so old ago who KNOWS what would have happened? | 21:24 |
| timburke | unfortunately, we hadn't gotten that fix out to our cluster yet | 21:24 |
| patchbot | https://review.opendev.org/#/c/30481/ - swift - get_info - removes duplicate code (Take 3) (MERGED) - 17 patch sets | 21:24 |
| timburke | clayg, yeah, pretty sure it would've been left alone | 21:25 |
| clayg | ok, so ... mostly just a heads up for folks I guess - the patch is new; but good. If anyone else had noticed the behavior before that'd be cool - but it's ok if not either. | 21:26 |
| timburke | while we were trying to stop those shard stats from reporting, we were sad to see that we couldn't just stop the replication servers to stop the background traffic | 21:26 |
| timburke | #topic replication network and background daemons | 21:26 |
| *** openstack changes topic to "replication network and background daemons (Meeting topic: swift)" | 21:26 | |
| timburke | i wrote up https://launchpad.net/bugs/1883302 and https://review.opendev.org/#/c/735751/ for that particular issue | 21:27 |
| openstack | Launchpad bug 1883302 in OpenStack Object Storage (swift) "container-sharder should send stat updates using replication network" [Undecided,In progress] | 21:27 |
| patchbot | patch 735751 - swift - sharder: Use replication network to send shard ranges - 1 patch set | 21:27 |
| clayg | oh yeah, this one's heavy - timburke wants to go full on | 21:27 |
| alecuyer | clayg: (sorry, lagging), we have not seen it, but I can't say it hasn"t happened either | 21:28 |
| clayg | hrm... I know that p 735751 is slightly more targeted to the bug - but really the issue and the fix are much more pervasive than we realized originally | 21:28 |
| patchbot | https://review.opendev.org/#/c/735751/ - swift - sharder: Use replication network to send shard ranges - 1 patch set | 21:28 |
| clayg | timburke: I'd argue we reword the bug to at least "sharder and reconciler don't always use replication" and attempt to move forward with p 735991 which is bigger but WAY better | 21:29 |
| patchbot | https://review.opendev.org/#/c/735991/ - swift - Add X-Backend-Use-Replication-Network header - 1 patch set | 21:29 |
| *** jmasud has joined #openstack-meeting | 21:29 | |
| timburke | yeah -- so the writes go over replication, but the sharder still does reads over the client-traffic interface -- but it was harder to fix since it uses internal_client for that | 21:29 |
| timburke | it's got me wondering: which interface should our background daemons be using? | 21:30 |
| clayg | it's like a unified way to make all our different little client interfaces use replication networks like the probably all should have been doing forever; but we never had an interface for 'em before | 21:30 |
| *** aprice has quit IRC | 21:30 | |
| mattoliverau | oh yeah interesting. | 21:31 |
| timburke | the way i've got that second patch at the moment, callers have to opt-in to using the replication network. but i wonder if we could/should use it by default | 21:31 |
| clayg | timburke: I think i'd be willing to say any thing besides the proxy connecting to the node[ip] when a [replication_ip] is available is a bug? like not a design choice, or operator choice - a bug | 21:32 |
| mattoliverau | if a direct client or internal client is ever used inline from a customer request then client traffic else replication network. | 21:32 |
| timburke | clayg says we (nvidia nee swiftstack) have at least one utility we've written that *would* want the client-traffic network; i wonder what other people have written and which interface they'd prefer | 21:32 |
| clayg | that's a fairly strong stance, but personally having a separate storage server for background work (that I can turn off when needed) has been a HUGE QOL improvement for me over the years | 21:33 |
| clayg | mattoliverau: I don't think internally we ever use direct/internal client from inside the proxy (i.e. related to a user request) | 21:33 |
| clayg | timburke: do some of the new UPDATE requests use direct client? | 21:34 |
| clayg | "new" - i'm not sure there's anything landed that does that... and IIRC they just call req.get_resp(app)? | 21:34 |
| mattoliverau | yeah, trying to decide if we use it anywhere | 21:34 |
| *** aprice has joined #openstack-meeting | 21:34 | |
| timburke | nope, it's plumbed through the proxy-server app | 21:34 |
| *** PrinzElvis has joined #openstack-meeting | 21:35 | |
| timburke | well, maybe i keep it opt-in on that patch and propose another to change the default while people think through what they've got and what the upgrade impact would be like | 21:35 |
| clayg | so internal-client and "the proxy-server app" are VERY similar - but Tim found a place between InternalClient and the app itself where we can plumb this header through (and then way down near where we make_connection(node, ...) we get to look at headers to pick node[ip] or node[replication_ip] | 21:35 |
| clayg | it's really sort of slick - and sexy because it works uniformly across both interfaces (because both interfaces already take headers and can set backend defaults) | 21:36 |
| zaitcev | Direct client goes straight to replication network sounds unexpected to me. I thought that proxies might not even have that network. | 21:36 |
| clayg | zaitcev: that's good feedback! proxies don't use direct client - but anything "defaulting" to the backend network might be "surprising" to some | 21:37 |
| mattoliverau | a quick grep, yeah no direct client in proxy or middlewares. | 21:38 |
| clayg | and I hadn't considered access/topology - if someone deploys anything that uses either of these interfaces ON a node that can use the replication network, that could be a big surprise 😞 | 21:38 |
| *** csatari has joined #openstack-meeting | 21:39 | |
| mattoliverau | Well for things like the reconsiler and sharder, they are part of the consistency engine, the sharder is just a type of replicatior (in a way). so yeah totally should do it's work over replication network. | 21:40 |
| *** patrickeast has joined #openstack-meeting | 21:40 | |
| *** knikolla has joined #openstack-meeting | 21:40 | |
| clayg | timburke: I would encourage you to drop p 735751 now that p 735991 is on everyone's radar - to me, it's not so much about "fixing ALL THE THINGS" as "fixing it RIGHT" | 21:40 |
| patchbot | https://review.opendev.org/#/c/735751/ - swift - sharder: Use replication network to send shard ranges - 1 patch set | 21:40 |
| patchbot | https://review.opendev.org/#/c/735991/ - swift - Add X-Backend-Use-Replication-Network header - 1 patch set | 21:40 |
| timburke | 👍 thanks for the feedback, everyone! | 21:41 |
| timburke | on to updates | 21:41 |
| timburke | #topic waterfall EC | 21:41 |
| *** openstack changes topic to "waterfall EC (Meeting topic: swift)" | 21:41 | |
| timburke | clayg, how's it going? | 21:41 |
| *** slaweq has quit IRC | 21:41 | |
| clayg | mattoliverau: I'm glad to hear you say that! I think having internal and direct client growing these new interfaces will amek it much easier to get it right out of the gate for new daemons | 21:41 |
| clayg | timburke: a little better-ish, or maybe? | 21:41 |
| *** raildo has quit IRC | 21:42 | |
| clayg | I like the feeder! | 21:42 |
| clayg | https://review.opendev.org/#/c/711342/8 phew - too many links open | 21:42 |
| patchbot | patch 711342 - swift - wip: asyc concurrent ecfragfetcher - 8 patch sets | 21:42 |
| clayg | I'm still waffling about the code duplication | 21:42 |
| clayg | i don't know exactly how to describe the experience of pulling them apart - it's like I'm starting to see the tear lines and I can't help but try and imagine a few abstraction that could MAYBE cut through them 😞 | 21:44 |
| clayg | I mostly try not to think about it while I make waterfall-ec awseome | 21:44 |
| clayg | which it *totally* is | 21:44 |
| clayg | or at least I can see how it will be - once I add a follow to configure the feeder with per-policy settings and the stair-step configuration that alecuyer talked about at the PTG | 21:45 |
| alecuyer | nice! | 21:45 |
| clayg | I'm much more excited about working on that code than wading through the mess of cutting up GETorHEADHandler and ECFragGetter | 21:46 |
| clayg | at some level I want to just leave the messy turd there finish the stuff I care about and then try to re-evaluate when I feel less pressure to FIX THE DAMN BUG | 21:46 |
| clayg | but I sort of know a new priority will come along, and even though I'll probably get up a patch out of pure guilt - it's not obvious to me "here a 1000 line diff that doesn't change anything" is gunna get merged if I'm not complaining about it | 21:47 |
| clayg | ALSO! I need to chat with folks about extra requests for non-durables - or at least... the existing behavior is obviously wrong and the correct behavior is not obvious | 21:47 |
| clayg | I picked something... and it's... better - but what if Y'ALL have an even BETTER idea!!! | 21:48 |
| zaitcev | Little hope of that I'm afraid. | 21:48 |
| zaitcev | Also | 21:49 |
| clayg | i dunno if we can wait til the next PTG to go over it... | 21:49 |
| timburke | should we read what you've done so far to try to get our heads around the problem, or should we sum it now? | 21:49 |
| clayg | I think it's a complex enough change (I'm really trying to SIMPLIFY) that it's worth a read by anyone who can handle it | 21:50 |
| clayg | I've been trying to drop comments around the interesting bits | 21:50 |
| *** jamesmcarthur has joined #openstack-meeting | 21:50 | |
| timburke | we could schedule a video chat if you think something closer to "in person" would be best | 21:50 |
| mattoliverau | etherpad braindump of the current problem, them video chat to talk through it? | 21:51 |
| mattoliverau | plus time to look at code :) | 21:51 |
| alecuyer | yes waiting for the next ptg is too far if clay is working on it *now* ? | 21:51 |
| clayg | for the non-durable extra requests - yeah I would like to high-bandwidth (very helpful at the PTG); I would definitely try and prepare if there was something scheduled. | 21:51 |
| mattoliverau | let's do something then :) We could zoom or jitsi and announce it in channel so anyone can attend. (keeping it open). | 21:52 |
| clayg | ok, well no one is screaming about the code duplication - that gives me some confidence that I've built it up enough no one is going to go to review and be like "WTF is this!? you can't do this!" | 21:52 |
| clayg | so I'll leave the turd there and move on down the line to the follow on configuration stuff (which will be SUPER sexy) | 21:53 |
| clayg | then we're just left with non-durable extra requests - which I can write up ASAP and Tim will help me with a zoom thing | 21:53 |
| timburke | 👍 | 21:54 |
| mattoliverau | clayg: like you said at the PTG code dup is ok, so long as we all know, it's documented, and it make it easier to grok and understand ;) | 21:54 |
| clayg | mattoliverau: ❤️ you guys are the best | 21:54 |
| timburke | all right | 21:54 |
| mattoliverau | thanks for polishing the turd :) | 21:54 |
| *** jamesmcarthur has quit IRC | 21:54 | |
| timburke | sorry rledisez, alecuyer: i forgot to drop losf from the agenda like i'd promised to last week | 21:54 |
| timburke | so | 21:55 |
| timburke | #topic open discussion | 21:55 |
| *** openstack changes topic to "open discussion (Meeting topic: swift)" | 21:55 | |
| timburke | anything else we should talk about in the last five minutes? | 21:55 |
| alecuyer | well I'll just post a link for clay ;) wrt to a PTG question | 21:55 |
| clayg | https://review.opendev.org/#/c/733919/ | 21:55 |
| patchbot | patch 733919 - swift - s3api: Allow CompleteMultipartUpload requests to b... - 3 patch sets | 21:55 |
| alecuyer | https://docs.python.org/3/library/multiprocessing.shared_memory.html | 21:55 |
| clayg | alecuyer: YAS!! | 21:55 |
| alecuyer | 3.8 only tho - but nice interface to use shared memory, switch the ring to use numpy ? | 21:55 |
| clayg | timburke: my complete multi-part retry has been going for 3.5 hours - and it's still working | 21:56 |
| timburke | alecuyer, that also makes me think of something DHE mentioned earlier today... | 21:56 |
| alecuyer | didn't think about it but thought i'd share the link, and sorry if you're all aware of that | 21:56 |
| timburke | clayg, wow! 5 min seems *way* too short then -- maybe it should work indefinitely | 21:56 |
| clayg | dunno 😞 | 21:56 |
| clayg | also i haven't tried abort - or... what was the other calls you were interested in? | 21:56 |
| timburke | abort after complete and complete after abort are the two sequences i'm a little worried about | 21:57 |
| clayg | alecuyer: I remember thinking "oh it's only arrays? pfhfhfhfh" - but now that you mention it - what is the ring except a big array!? 😁 | 21:57 |
| clayg | there is the error limiting stuff 🤔 | 21:58 |
| timburke | zaitcev, thanks for the review on https://review.opendev.org/#/c/734721/ ! | 21:58 |
| patchbot | patch 734721 - swift - py3: (Better) fix percentages in configs - 4 patch sets | 21:58 |
| clayg | abort after complete - so i'm in that state now... but if that works I could try to complete it again too! 🤔 | 21:58 |
| kota__ | error limiting staff on shared memory seems good idea | 21:59 |
| zaitcev | So, are we trying to load rings into SysV shm? | 21:59 |
| zaitcev | I'd be more comfortable with an mmap() of some temp file into which the json or pickle is dumped first. | 21:59 |
| clayg | kota__: yes! alecuyer will figure out how to make it work :P | 21:59 |
| kota__ | it seems it's not only py3.8 but greater and equals to 3.8? | 22:00 |
| kota__ | not yet 3.9 released yet | 22:00 |
| alecuyer | kota__: right | 22:00 |
| kota__ | good | 22:00 |
| timburke | all right, we're about out of time | 22:01 |
| clayg | kota__: yeah and like zaitcev it's maybe not even a full solution on it's own even if we did want to do it >= 3.8 only (which by the time it's done might seem reasonable) | 22:01 |
| timburke | thank you all for coming! i feel like we had some really good discussions today :-) | 22:01 |
| kota__ | clayg: true, got it. | 22:01 |
| timburke | thank you all for coming, and thank you for working on swift! | 22:01 |
| timburke | #endmeeting | 22:02 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 22:02 | |
| openstack | Meeting ended Wed Jun 17 22:01:59 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 22:02 |
| clayg | yeah, long meeting - good stuff - thanks everyone! | 22:02 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/swift/2020/swift.2020-06-17-21.00.html | 22:02 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/swift/2020/swift.2020-06-17-21.00.txt | 22:02 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/swift/2020/swift.2020-06-17-21.00.log.html | 22:02 |
| *** patchbot has left #openstack-meeting | 22:02 | |
| *** zaitcev has left #openstack-meeting | 22:02 | |
| *** e0ne has quit IRC | 22:19 | |
| *** jmasud has quit IRC | 22:23 | |
| *** jmasud has joined #openstack-meeting | 22:23 | |
| *** TrevorV has quit IRC | 22:36 | |
| *** rcernin has joined #openstack-meeting | 22:42 | |
| *** dmacpher_ has quit IRC | 22:47 | |
| *** seongsoocho has joined #openstack-meeting | 22:51 | |
| *** gyee has quit IRC | 22:52 | |
| *** jmasud has quit IRC | 23:03 | |
| *** Lucas_Gray has quit IRC | 23:07 | |
| *** Lucas_Gray has joined #openstack-meeting | 23:22 | |
| *** diurnalist has quit IRC | 23:30 | |
| *** dmacpher has joined #openstack-meeting | 23:42 | |
| *** andrebeltrami has quit IRC | 23:53 | |
| *** dmacpher_ has joined #openstack-meeting | 23:58 | |
| *** dmacpher has quit IRC | 23:59 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
