| *** subbareddy has quit IRC | 00:47 | |
| *** jinyuanliu has quit IRC | 01:04 | |
| *** jinyuanliu has joined #openstack-helm | 01:04 | |
| *** schwicht has joined #openstack-helm | 01:08 | |
| *** jinyuanliu has quit IRC | 01:16 | |
| openstackgerrit | Merged openstack/openstack-helm-infra master: Elasticsearch Disable Curator in Gate & Chart Defaults https://review.opendev.org/c/openstack/openstack-helm-infra/+/779912 | 02:08 |
|---|---|---|
| *** jinyuanliu has joined #openstack-helm | 02:28 | |
| *** rezroo has joined #openstack-helm | 03:50 | |
| *** miniroy has quit IRC | 05:35 | |
| *** rezroo has quit IRC | 07:23 | |
| sgautam | gagehugo We get the error like this `Error: cannot patch "keystone-credential-setup" with kind Job: Job.batch "keystone-credential-setup" is invalid: spec.template:` and ending with `: field is immutable` | 07:30 |
| sgautam | We usually see this error when there is changes in config in values.yaml or if there is any changes in chart templates that are related to template like configmaps or secrets that are used in jobs. So we have used the helm-hook post-install and post-upgrade so that jobs resources are last to be created. | 07:30 |
| sgautam | gagehugo: So, jobs in helm-toolkit, we send the hook values from the chart that is using it to generate template as helm metadata and for the jobs template in chart itself we have added the hook on them. | 07:33 |
| sgautam | First we were facing issue for the keystone, whose patches have been submitted earlier and merged. Now there is no issue for keystone while upgrade. | 07:33 |
| sgautam | We are facing same issue for neutron right now, so we are sending the hook for others chart too along with neutron. | 07:34 |
| sgautam | They have been tested on our local cluster. | 07:35 |
| *** schwicht has quit IRC | 07:39 | |
| jinyuanliu | Now it's midnight | 07:39 |
| sgautam | jinyuanliu: It's afternoon my time. We can discuss later today about this. Thanks for letting me know. | 07:42 |
| *** ricolin has joined #openstack-helm | 08:32 | |
| openstackgerrit | Susanta gautam proposed openstack/openstack-helm master: Added post-install and post-upgrade hook for jobs in magnum https://review.opendev.org/c/openstack/openstack-helm/+/782678 | 09:45 |
| openstackgerrit | Susanta gautam proposed openstack/openstack-helm master: Added post-install and post-upgrade hook for jobs in magnum https://review.opendev.org/c/openstack/openstack-helm/+/782678 | 10:44 |
| openstackgerrit | Susanta gautam proposed openstack/openstack-helm master: Added post-install and post-upgrade hook for jobs in ironic https://review.opendev.org/c/openstack/openstack-helm/+/782695 | 11:23 |
| openstackgerrit | Susanta gautam proposed openstack/openstack-helm master: Added post-install and post-upgrade hook in jobs for barbican https://review.opendev.org/c/openstack/openstack-helm/+/782710 | 12:55 |
| openstackgerrit | Susanta gautam proposed openstack/openstack-helm master: Added post-install and post-upgrade hook on jobs in designate chart https://review.opendev.org/c/openstack/openstack-helm/+/782738 | 13:27 |
| *** dansmith has quit IRC | 13:31 | |
| openstackgerrit | RADHIKA PAI proposed openstack/openstack-helm-infra master: [WIP][Update] Enable Ping in Prometheus blackbox exporter https://review.opendev.org/c/openstack/openstack-helm-infra/+/782740 | 13:32 |
| *** dansmith has joined #openstack-helm | 13:37 | |
| *** miniroy has joined #openstack-helm | 13:55 | |
| *** miniroy has quit IRC | 13:55 | |
| openstackgerrit | Susanta gautam proposed openstack/openstack-helm master: Added post-install and post-upgrade hook for jobs in mistral https://review.opendev.org/c/openstack/openstack-helm/+/782748 | 13:58 |
| openstackgerrit | Steven Fitzpatrick proposed openstack/openstack-helm-infra master: WIP Elasticsearch S3 stuff https://review.opendev.org/c/openstack/openstack-helm-infra/+/780453 | 14:19 |
| openstackgerrit | Susanta gautam proposed openstack/openstack-helm master: Added post-install and post-upgrade hook for Jobs on octavia https://review.opendev.org/c/openstack/openstack-helm/+/782758 | 14:20 |
| sgautam | Hello, Can you help us review this patch? https://review.opendev.org/c/openstack/openstack-helm-infra/+/780027 | 15:30 |
| sgautam | So we wanted to have aws s3 glacier as remote backup options as we wanted to be able to choose between swift or aws s3/minio. We found rclone support both so we made changes in existing backup script. | 15:30 |
| mnaser | anyone run into "Lost connection to MySQL server during query" ? | 16:19 |
| mnaser | we're seeing this happen often, wondering if the nginx ingress is acting up | 16:20 |
| Deaddy | mnaser: if the machine has many cores (in nginx terms this is something like 4+) it might be necessary to limit the worker processes to some number lest the nginx trys to open a lot of connections per core | 16:48 |
| Deaddy | and then usually either restarts all the time or has some error messages with pthread and some 2 somewhere in it | 16:48 |
| Deaddy | not sure about the 2 though :D | 16:49 |
| openstackgerrit | Steven Fitzpatrick proposed openstack/openstack-helm-infra master: WIP Elasticsearch S3 stuff https://review.opendev.org/c/openstack/openstack-helm-infra/+/780453 | 17:07 |
| mnaser | Deaddy: that's pretty useful, these are all systems with MANY controllers -- im not sure if there's a default built-in limit on the mariadb ingress | 17:27 |
| mnaser | i think there is | 17:27 |
| mnaser | 0 restarts, 19d age | 17:28 |
| openstackgerrit | RADHIKA PAI proposed openstack/openstack-helm-infra master: [WIP][Update] Enable icmp module in Prometheus blackbox exporter https://review.opendev.org/c/openstack/openstack-helm-infra/+/782740 | 17:32 |
| openstackgerrit | RADHIKA PAI proposed openstack/openstack-helm-infra master: [WIP][Update] Enable icmp module in Prometheus blackbox exporter https://review.opendev.org/c/openstack/openstack-helm-infra/+/782740 | 17:33 |
| openstackgerrit | RADHIKA PAI proposed openstack/openstack-helm-infra master: [WIP][Update] Enable icmp module in Prometheus blackbox exporter https://review.opendev.org/c/openstack/openstack-helm-infra/+/782740 | 17:34 |
| openstackgerrit | Jeremy Houser proposed openstack/openstack-helm-deployments master: Feature: Add Rabbitmq to Openstack-Helm-Deployments https://review.opendev.org/c/openstack/openstack-helm-deployments/+/765884 | 17:50 |
| openstackgerrit | Nafiz Haider proposed openstack/openstack-helm-infra master: feat(tls): Enable TLS for OpenStack RabbitMQ https://review.opendev.org/c/openstack/openstack-helm-infra/+/770678 | 17:52 |
| Deaddy | I think then one has to check the logs | 17:54 |
| Deaddy | pretty evident if that is the case iirc, you can just grep for pthread | 17:54 |
| openstackgerrit | Sophie Huang proposed openstack/openstack-helm-infra master: Set strict permission on mariadb data dir https://review.opendev.org/c/openstack/openstack-helm-infra/+/782836 | 18:05 |
| openstackgerrit | Sophie Huang proposed openstack/openstack-helm-infra master: Set strict permission on mariadb data dir https://review.opendev.org/c/openstack/openstack-helm-infra/+/782836 | 18:10 |
| openstackgerrit | Gage Hugo proposed openstack/openstack-helm-images master: Add ubuntu bionic mariadb file https://review.opendev.org/c/openstack/openstack-helm-images/+/782843 | 19:05 |
| openstackgerrit | Sangeet Gupta proposed openstack/openstack-helm-infra master: memcahced: Enable TLS for the memcached-exporter https://review.opendev.org/c/openstack/openstack-helm-infra/+/782844 | 19:07 |
| openstackgerrit | Sangeet Gupta proposed openstack/openstack-helm-infra master: memcahced: Enable TLS for the memcached-exporter https://review.opendev.org/c/openstack/openstack-helm-infra/+/782844 | 19:08 |
| openstackgerrit | chinasubbareddy mallavarapu proposed openstack/openstack-helm-infra master: [ceph-provisioners] Update ceph_mon config as per new ceph clients https://review.opendev.org/c/openstack/openstack-helm-infra/+/781162 | 19:19 |
| openstackgerrit | Gage Hugo proposed openstack/openstack-helm master: Move stein to periodic, update to train https://review.opendev.org/c/openstack/openstack-helm/+/782848 | 19:35 |
| openstackgerrit | Arina Stebenkova proposed openstack/openstack-helm-images master: Prevent python2 installation https://review.opendev.org/c/openstack/openstack-helm-images/+/782849 | 19:36 |
| openstackgerrit | Sophie Huang proposed openstack/openstack-helm-infra master: Set strict permission on mariadb data dir https://review.opendev.org/c/openstack/openstack-helm-infra/+/782836 | 19:57 |
| openstackgerrit | Sophie Huang proposed openstack/openstack-helm-infra master: Set strict permission on mariadb data dir https://review.opendev.org/c/openstack/openstack-helm-infra/+/782836 | 20:20 |
| openstackgerrit | Stephen Taylor proposed openstack/openstack-helm-infra master: [ceph-osd] Update directory-based OSD deployment to use python3 https://review.opendev.org/c/openstack/openstack-helm-infra/+/782857 | 21:03 |
| openstackgerrit | Gage Hugo proposed openstack/openstack-helm-images master: Add ubuntu bionic mariadb file https://review.opendev.org/c/openstack/openstack-helm-images/+/782843 | 21:46 |
| mnaser | Deaddy: nothing for pthread :( | 23:20 |
| mnaser | and seeing this across *all* OSH deployed envs | 23:20 |
| mnaser | i've actually forced nginx worker proceses to be 4 instead of auto because we ended up with way too much that would spawn | 23:25 |
| mnaser | gagehugo: fyi wrt to the jobs, you upgrade fine cause i think airship takes care of deleting those jobs :P | 23:25 |
| gagehugo | yeah probably | 23:28 |
| portdirect | mnaser: another reason for this is the health checks sometimes, which can be pretty rude and just 'hang up' | 23:31 |
| mnaser | portdirect: yeah i've seen that, but in this case, no restarts on the pods :( | 23:32 |
| portdirect | And yes - the original thing armada provided was clean up of jobs for upgrades | 23:32 |
| mnaser | the ONLY thing is because we're slowly transitioning from osa to osh | 23:32 |
| mnaser | so it's the *host* talking to a container | 23:32 |
| portdirect | If you have a ps for doing that via helm lcm hooks, would be happy to +2 | 23:32 |
| portdirect | Hmm | 23:32 |
| mnaser | portdirect: they're all sitting in the review queue :) | 23:32 |
| portdirect | Will have a look 1st thing | 23:33 |
| mnaser | 2021-03-24 23:29:21.186 9443 ERROR oslo_db.sqlalchemy.engines [req-b68a49f8-e802-4d93-9645-66824d2d33c8 067b81c25b4d4ee5bcdc243032c6ed25 20ecded9c8b54c83afe7f8e459df0681 - default default] Database connection was found disconnected; reconnecting: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') | 23:33 |
| mnaser | [SQL: SELECT 1] | 23:33 |
| mnaser | portdirect: fyi https://review.opendev.org/q/owner:xuxant02%2540gmail.com+is:open | 23:33 |
| portdirect | Oh, that's the other way round to how I've seen this in the past... | 23:33 |
| portdirect | I misread above | 23:33 |
| mnaser | but yeah, i've seen this issue in all of our environments, and i think i even see it inside our container=>container services | 23:33 |
| portdirect | Yeah, that's not pretty | 23:34 |
| mnaser | im wrong, this seems to only happen in host => container communication | 23:34 |
| portdirect | I presume you surviving for now with retry's, but that's no way to live :( | 23:34 |
| mnaser | yeah -- and it's causingh some 503s for us and occasionally marking some agents as down | 23:35 |
| portdirect | Ok | 23:35 |
| mnaser | (our monitoring deploys a vm every 5 minutes and we saw it significantly get 'worse' after we moved) | 23:35 |
| mnaser | i think this may not be an OSH thing but a calico thing perhaps | 23:35 |
| portdirect | Possibly, I have a few people I can tap and see if they have any ideas | 23:36 |
| portdirect | Are you using iptables or ipvs for kube service ips | 23:36 |
| mnaser | iptables | 23:37 |
| mnaser | would love to switch to ipvs but that's work(tm) | 23:37 |
| mnaser | also, we're running kernels new enough for cilium which i'd love to look into | 23:37 |
| portdirect | And for node to node traffic in calico, is that encapsulated, or direct? | 23:37 |
| mnaser | portdirect: in this case, i think we have a setting of CrossSubnet, and those systems are all in same L2 context | 23:38 |
| portdirect | Ok | 23:38 |
| mnaser | so there's a bunch of 10.244.x.x/26 routes directly to each other host | 23:38 |
| portdirect | Roger | 23:39 |
| portdirect | 503s ain't cool | 23:39 |
| mnaser | yeah, kinda just almost wanna push-through and switch everything to k8s quickly | 23:40 |
| mnaser | but that seems very brute force-y | 23:40 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!