| *** Liang__ has joined #openstack-helm | 00:50 | |
| *** Liang__ is now known as LiangFang | 00:55 | |
| *** irclogbot_1 has quit IRC | 01:44 | |
| *** JangwonLee_ has joined #openstack-helm | 02:15 | |
| *** JangwonLee has quit IRC | 02:18 | |
| openstackgerrit | Rahul Khiyani proposed openstack/openstack-helm-infra master: rabbitmq securityContext https://review.openstack.org/642918 | 02:42 |
|---|---|---|
| *** pgaxatte has joined #openstack-helm | 06:05 | |
| *** Nishant_ has joined #openstack-helm | 06:24 | |
| *** dpawlik has joined #openstack-helm | 06:45 | |
| pgaxatte | hello | 06:50 |
| pgaxatte | i'm reading the docs on how to add authentication for private registries: https://docs.openstack.org/openstack-helm/latest/ko_KR/specs/support-OCI-image-registry-with-authentication-turned-on.html | 06:52 |
| pgaxatte | is this still active? | 06:52 |
| pgaxatte | i mean there 3 work items at the end, are they stil activetely worked on? | 06:53 |
| *** jamesgu has quit IRC | 06:58 | |
| *** nmimi has joined #openstack-helm | 07:13 | |
| *** dimitris_ has joined #openstack-helm | 07:19 | |
| *** alisanhaji has joined #openstack-helm | 07:24 | |
| *** jsuchome has joined #openstack-helm | 07:28 | |
| *** alisanhaji has quit IRC | 07:29 | |
| evrardjp | o/ | 07:37 |
| evrardjp | pgaxatte: I don't know. Maybe ask tomorrow in the meeting? | 07:37 |
| pgaxatte | evrardjp: alright will do, what time are the meetings? | 07:39 |
| evrardjp | pgaxatte: here is an ICS file for your calendar :) | 07:40 |
| evrardjp | http://eavesdrop.openstack.org/#OpenStack-Helm_Team_Meeting | 07:40 |
| evrardjp | tl:dr; 1500 utc | 07:41 |
| pgaxatte | :) | 07:41 |
| pgaxatte | thanks | 07:41 |
| *** witek has joined #openstack-helm | 07:47 | |
| *** itxaka has joined #openstack-helm | 07:49 | |
| *** alisanhaji has joined #openstack-helm | 07:53 | |
| *** nick_kar has joined #openstack-helm | 07:55 | |
| *** happyhemant has joined #openstack-helm | 08:00 | |
| *** roman_g has joined #openstack-helm | 08:15 | |
| *** witek has quit IRC | 09:20 | |
| *** witek has joined #openstack-helm | 09:31 | |
| *** LiangFang has quit IRC | 10:04 | |
| pgaxatte | hello again | 11:59 |
| pgaxatte | we're trying to deploy the multinode scripts on top of Kubernetes 1.14 | 12:00 |
| pgaxatte | has anyone succeeded already? is openstack-helm limited to a certain version of k8s? | 12:00 |
| pgaxatte | we were able to spawn on kubernetes 1.12 | 12:01 |
| portdirect | What errors have you been seeing with 1.14? | 12:01 |
| portdirect | I've not run on that yet, but we test with 1.13 | 12:01 |
| pgaxatte | portdirect: this kind of errors: http://paste.openstack.org/show/749001/ | 12:02 |
| pgaxatte | does it work out of the box on 1.13? | 12:03 |
| portdirect | It should | 12:03 |
| pgaxatte | the error we get looks like something was deprecated or moved out of beta in k8s 1.14 | 12:03 |
| portdirect | It is possible that the storage class schema has changed/been tightened up slightly | 12:04 |
| portdirect | Agreed | 12:04 |
| portdirect | This should be fairly simple to fix, at the very worst we may need a conditional dependent on version of k8s tiller is talking to | 12:05 |
| pgaxatte | portdirect: we're digging in ceph-mon helm charts and we'll report back if we find something | 12:26 |
| portdirect | I think ceph-client will be what's needs attention here at 1st glance | 12:33 |
| *** spiette has quit IRC | 12:40 | |
| pgaxatte | portdirect so far we see that there is an issue when creating the ceph-mon-keyring secret | 12:40 |
| pgaxatte | the lead we are exploring is maybe upgrading the kubectl used in the pod | 12:41 |
| pgaxatte | it's a v1.10.3 maybe it's a bit too old to work with k8s 1.14 | 12:41 |
| *** spiette has joined #openstack-helm | 12:43 | |
| pgaxatte | the problem is coming from there apparently | 12:50 |
| pgaxatte | using the same version of kubectl as in the pod we could not create the secret by hand but we succeeded when using kubectl 1.12 | 12:51 |
| portdirect | Nice! | 12:52 |
| portdirect | If you could make a ps to openstack-helm-images bumping the version of kubectl in the image it would be really appreciated | 12:53 |
| *** parasitid has joined #openstack-helm | 12:53 | |
| pgaxatte | on my way :) | 12:53 |
| *** bh526r has joined #openstack-helm | 12:55 | |
| parasitid | hi ppl | 12:55 |
| parasitid | has anyone ever tried to base an openstack-helm deploymennt based on rook instead of oshi/ceph ? | 12:56 |
| parasitid | is it a good idea or not | 12:56 |
| parasitid | ? | 12:56 |
| portdirect | parasitid, ive done some very simple sanity tests | 12:58 |
| portdirect | both rook and the osh-infra charts provide vanilla ceph | 12:58 |
| portdirect | so it would be perfectly valid to use rook if so desired | 12:59 |
| portdirect | the one thing that would chnage would be needing to treat the ceph cluster as an `external` one - jayahn has a wip ps for how to do this in gerrit | 13:00 |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm-infra master: HTK: Update k8s-entrypoint container macro to add security context https://review.openstack.org/650519 | 13:03 |
| parasitid | oki | 13:07 |
| parasitid | thanks | 13:07 |
| parasitid | could you point me his wip on gerrit ? | 13:08 |
| parasitid | or is it already in the docs how to use an external ceph cluster ? | 13:08 |
| portdirect | parasitid: https://review.openstack.org/#/c/586992/ | 13:09 |
| parasitid | ah | 13:13 |
| parasitid | cheers | 13:13 |
| parasitid | ok | 13:16 |
| parasitid | so ive partially read it | 13:16 |
| parasitid | it seems to me that, according to the https://review.openstack.org/#/c/586992/9/doc/source/install/developer/deploy-with-existing-ceph.rst | 13:17 |
| parasitid | we still have to use some osh ceph tools to do ceph-provisionning | 13:18 |
| parasitid | am i correct ? | 13:18 |
| pgaxatte | portdirect, i'm not used to storyboard but here it is: https://storyboard.openstack.org/#!/story/2005397 | 13:19 |
| portdirect | you shoudl be able to skip the 1st section as i belive rook provides its own storageclass | 13:19 |
| portdirect | lines 53 onwards will be required if you intend to use ceph with cinder/glance/nova | 13:20 |
| parasitid | ok | 13:20 |
| openstackgerrit | diwakar thyagaraj proposed openstack/openstack-helm-infra master: Add Docker default AppArmor profile to Ceph-osd Change apparmor for ceph-osd into gate script Make MAC section of daemonset optional in case nothing is specified in values https://review.openstack.org/647638 | 13:27 |
| openstackgerrit | diwakar thyagaraj proposed openstack/openstack-helm-infra master: Add default AppArmor profile to Fluentbit Make MAC profile annotation optional because nothing is currently defined in values.yaml https://review.openstack.org/647881 | 13:31 |
| *** jsuchome has quit IRC | 13:53 | |
| openstackgerrit | Luna Das proposed openstack/openstack-helm-infra master: Add docker-default apparmor profile for elasticserach. https://review.openstack.org/649990 | 14:02 |
| openstackgerrit | Hemanth Nakkina proposed openstack/openstack-helm-infra master: [WIP] Helm chart for sriov-cni plugin https://review.openstack.org/650908 | 14:04 |
| openstackgerrit | Deepak proposed openstack/openstack-helm-images master: Makefile and Docker file for calicoctl-utility containers https://review.openstack.org/649742 | 14:06 |
| *** Nishant_ has quit IRC | 14:08 | |
| *** dpawlik has quit IRC | 14:13 | |
| *** kranthikirang has joined #openstack-helm | 14:22 | |
| *** michael-beaver has joined #openstack-helm | 14:25 | |
| openstackgerrit | kranthi kiran guttikonda proposed openstack/openstack-helm-infra master: ceph-mon, tenat-ceph storageclass schema https://review.openstack.org/650428 | 14:27 |
| *** jamesgu has joined #openstack-helm | 14:27 | |
| evrardjp | parasitid: I think I would be happy to follow what's going on on that level (rook support work) | 14:33 |
| openstackgerrit | Ian Howell proposed openstack/openstack-helm master: WIP/DNM - Add the upgrade lifecycle step https://review.openstack.org/649189 | 14:34 |
| *** howell has joined #openstack-helm | 14:34 | |
| openstackgerrit | Merged openstack/openstack-helm master: change the way to get tunnel device https://review.openstack.org/643909 | 14:36 |
| openstackgerrit | Randeep Jalli proposed openstack/openstack-helm-infra master: This commit adds docker-default apparmor profile for prometheus-alertmanager Add in prometheus-alertmanager gate script as a script https://review.openstack.org/650373 | 14:39 |
| *** pgaxatte has quit IRC | 14:40 | |
| parasitid | evrardjp: ok | 14:45 |
| parasitid | will try to keep you informed | 14:45 |
| evrardjp | thanks parasitid :D | 14:46 |
| openstackgerrit | Randeep Jalli proposed openstack/openstack-helm-infra master: Add Docker default AppArmor profile to Ceph-osd Change apparmor for ceph-osd into gate script Make MAC section of daemonset optional in case nothing is specified in values https://review.openstack.org/647638 | 14:52 |
| *** lemko has joined #openstack-helm | 14:53 | |
| *** cfriesen has joined #openstack-helm | 14:54 | |
| happyhemant | evrardjp: Hi I actually trying to deploy horizon on openstack but got some strange error in logs. may be you are famillier with this and could help me. https://www.irccloud.com/pastebin/4XD0qqLC/ | 14:55 |
| happyhemant | I dont get it this "no listening sockets available" | 14:56 |
| *** jsuchome has joined #openstack-helm | 14:56 | |
| jsuchome | portdirect: Hi, I think your comments in https://review.openstack.org/#/c/642844/ were addressed ... | 14:57 |
| *** cfriesen has quit IRC | 14:59 | |
| jsuchome | Any reason not to merge this? https://review.openstack.org/#/c/642415/ As it is just addition of new script, it's not affecting upstream at all... | 14:59 |
| *** sthussey has joined #openstack-helm | 15:01 | |
| happyhemant | evrardjp: I also get this error when i apply my horizon manifest file. https://www.irccloud.com/pastebin/do3Gegui/ | 15:31 |
| portdirect | dwalt / mattmceuen ^ any thoughts? | 15:33 |
| *** itlinux_ has quit IRC | 15:33 | |
| openstackgerrit | Itxaka Serrano Garcia proposed openstack/openstack-helm-images master: Add tempest suse image and version ARG https://review.openstack.org/650933 | 15:40 |
| openstackgerrit | Jagan Mohan Kavva proposed openstack/openstack-helm-infra master: Add docker-default (enforce) AppArmor profile to openvswitch https://review.openstack.org/650940 | 15:57 |
| openstackgerrit | Itxaka Serrano Garcia proposed openstack/openstack-helm master: Fix configmap-etc values for tempest https://review.openstack.org/650948 | 16:03 |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm-infra master: Allow multiple containers per daemonset pod https://review.openstack.org/645958 | 16:08 |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm-infra master: WIP: MariaDB: Update backup scripts https://review.openstack.org/650950 | 16:20 |
| *** michaelbeaver has joined #openstack-helm | 16:21 | |
| *** michael-beaver has quit IRC | 16:24 | |
| *** itlinux has joined #openstack-helm | 16:28 | |
| *** witek has quit IRC | 16:29 | |
| openstackgerrit | Merged openstack/openstack-helm-infra master: HTK: Update k8s-entrypoint container macro to add security context https://review.openstack.org/650519 | 16:34 |
| *** unicell has joined #openstack-helm | 16:37 | |
| openstackgerrit | Merged openstack/openstack-helm-infra master: Ceph: fix overriding ceph monitor hosts value https://review.openstack.org/649279 | 16:38 |
| *** michaelbeaver has quit IRC | 16:40 | |
| openstackgerrit | Randeep Jalli proposed openstack/openstack-helm-infra master: Add docker-default apparmor profile for elasticserach. https://review.openstack.org/649990 | 16:42 |
| *** unicell has quit IRC | 16:45 | |
| openstackgerrit | Randeep Jalli proposed openstack/openstack-helm-infra master: This commit adds docker-default apparmor profile for prometheus-alertmanager Add in prometheus-alertmanager gate script as a script https://review.openstack.org/650373 | 16:50 |
| openstackgerrit | Randeep Jalli proposed openstack/openstack-helm-infra master: This commit adds docker-default apparmor profile for prometheus-node-exporter. https://review.openstack.org/650386 | 16:50 |
| openstackgerrit | Randeep Jalli proposed openstack/openstack-helm-infra master: This commit adds docker-default apparmor profile for prometheus-openstack-exporter. https://review.openstack.org/650388 | 16:50 |
| openstackgerrit | Randeep Jalli proposed openstack/openstack-helm-infra master: Add docker-default apparmor profile for prometheus process exporter. https://review.openstack.org/650395 | 16:50 |
| openstackgerrit | Randeep Jalli proposed openstack/openstack-helm-infra master: Add Docker default AppArmor profile to Ceph-osd Change apparmor for ceph-osd into gate script Make MAC section of daemonset optional in case nothing is specified in values https://review.openstack.org/647638 | 16:51 |
| openstackgerrit | Randeep Jalli proposed openstack/openstack-helm-infra master: Add docker-default apparmor profile for elasticserach. https://review.openstack.org/649990 | 16:51 |
| openstackgerrit | Randeep Jalli proposed openstack/openstack-helm-infra master: [WIP]Change gate job to see if running just libvirt in the AppArmor gate passes https://review.openstack.org/650961 | 16:54 |
| openstackgerrit | Randeep Jalli proposed openstack/openstack-helm-infra master: [WIP]Change gate job to see if running just libvirt in the AppArmor gate passes https://review.openstack.org/650961 | 16:55 |
| openstackgerrit | Randeep Jalli proposed openstack/openstack-helm-infra master: Add Docker default AppArmor profile to Ceph-osd Change apparmor for ceph-osd into gate script Make MAC section of daemonset optional in case nothing is specified in values https://review.openstack.org/647638 | 17:08 |
| *** unicell has joined #openstack-helm | 17:17 | |
| openstackgerrit | Merged openstack/openstack-helm master: Add network policy ingress rule to mariadb https://review.openstack.org/638299 | 17:37 |
| openstackgerrit | Randeep Jalli proposed openstack/openstack-helm-infra master: Add Docker default AppArmor profile to Ceph-osd Change apparmor for ceph-osd into gate script Make MAC section of daemonset optional in case nothing is specified in values https://review.openstack.org/647638 | 17:46 |
| openstackgerrit | diwakar thyagaraj proposed openstack/openstack-helm-infra master: Add default AppArmor profile to Fluentbit Make MAC profile annotation optional because nothing is currently defined in values.yaml https://review.openstack.org/647881 | 18:04 |
| openstackgerrit | Merged openstack/openstack-helm-images master: Add rocky release script https://review.openstack.org/642415 | 18:05 |
| openstackgerrit | Randeep Jalli proposed openstack/openstack-helm-infra master: Add Docker default AppArmor profile to Ceph-osd Change apparmor for ceph-osd into gate script Make MAC section of daemonset optional in case nothing is specified in values https://review.openstack.org/647638 | 18:07 |
| openstackgerrit | Luna Das proposed openstack/openstack-helm-infra master: [WIP]Add docker-default apparmor profile for elasticserach. https://review.openstack.org/649990 | 18:13 |
| *** michael-beaver has joined #openstack-helm | 18:15 | |
| *** lemko has quit IRC | 18:15 | |
| openstackgerrit | Randeep Jalli proposed openstack/openstack-helm-infra master: [WIP]Add docker-default apparmor profile for elasticserach. https://review.openstack.org/649990 | 18:16 |
| *** itlinux has quit IRC | 18:19 | |
| *** bh526r has quit IRC | 18:19 | |
| *** bh526r has joined #openstack-helm | 18:20 | |
| *** alanmeadows has quit IRC | 18:20 | |
| *** mattmceuen has quit IRC | 18:20 | |
| *** megheisler has quit IRC | 18:20 | |
| *** lamt has quit IRC | 18:20 | |
| *** adrianreza has quit IRC | 18:20 | |
| *** mattmceuen has joined #openstack-helm | 18:20 | |
| *** adrianreza has joined #openstack-helm | 18:20 | |
| *** alanmeadows has joined #openstack-helm | 18:21 | |
| *** ChanServ sets mode: +o alanmeadows | 18:21 | |
| *** megheisler has joined #openstack-helm | 18:25 | |
| *** lamt has joined #openstack-helm | 18:34 | |
| *** flaviosr_ has joined #openstack-helm | 18:37 | |
| *** flaviosr has quit IRC | 18:38 | |
| *** michael-beaver has quit IRC | 18:39 | |
| openstackgerrit | Gage Hugo proposed openstack/openstack-helm master: Add wait.resources.type to armada manifest https://review.openstack.org/648503 | 18:42 |
| *** michael-beaver has joined #openstack-helm | 18:44 | |
| *** happyhemant has quit IRC | 18:54 | |
| openstackgerrit | Doug Aaser proposed openstack/openstack-helm-infra master: [WIP] Patroni inclusion work for HA Postgres https://review.openstack.org/644388 | 18:55 |
| openstackgerrit | Randeep Jalli proposed openstack/openstack-helm-infra master: Add Docker default AppArmor profile to Ceph-osd Change apparmor for ceph-osd into gate script Make MAC section of daemonset optional in case nothing is specified in values https://review.openstack.org/647638 | 18:57 |
| *** witek has joined #openstack-helm | 19:03 | |
| *** alisanhaji has quit IRC | 19:05 | |
| openstackgerrit | Meghan Heisler proposed openstack/openstack-helm-infra master: Add wait.resource for LMA services to armada manifest https://review.openstack.org/649141 | 19:05 |
| openstackgerrit | Doug Aaser proposed openstack/openstack-helm-infra master: [WIP] Patroni inclusion work for HA Postgres https://review.openstack.org/644388 | 19:06 |
| *** howell has quit IRC | 19:10 | |
| *** jsuchome has quit IRC | 19:10 | |
| *** witek has quit IRC | 19:25 | |
| *** cfriesen has joined #openstack-helm | 19:33 | |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm-infra master: WIP: MariaDB: Update backup scripts https://review.openstack.org/650950 | 19:53 |
| dwalt | portdirect: happyhemant: thanks. Responded to your message in #airshipit | 19:54 |
| cfriesen | portdirect: thought I'd give you a heads-up on something weird I saw. was booting up controller nodes and one of them didn't come up due to disk pressure. The other mariadb nodes were stuck in /tmp/start.py because check_if_cluster_data_is_fresh() kept evaluating to false (because the one node wasn't updating its timestamp) | 20:00 |
| portdirect | this is kinda by design | 20:01 |
| portdirect | as we need to be be able to be sure which node leads the cluster | 20:01 |
| cfriesen | it does mean that if a single node isn't working your whole DB is down | 20:02 |
| cfriesen | on startup, at least | 20:02 |
| portdirect | on a full cluster restart, all three pods need to come up before the cluster will reform | 20:02 |
| *** itxaka has quit IRC | 20:03 | |
| cfriesen | I'm envisioning recovering from a power outage or something...could get stuck with an extended outage until someone comes in and manually fixes it up. | 20:03 |
| portdirect | as it stands today this is correct | 20:03 |
| portdirect | we are working on some methods to mitigate this however | 20:03 |
| cfriesen | ah? | 20:03 |
| portdirect | mostly centered around the need to have some way of moving the pod from a dead node to one that is running | 20:04 |
| portdirect | a 'reaper' if you will | 20:04 |
| portdirect | here there needs to be some sanity checking - eg if the workload is in `nodelost` or `unknown` for x lins and there is no rbd client connected, then it should be safe to get the pod running on another node | 20:05 |
| cfriesen | portdirect: there's a ~5min timeout in kubernetes to prevent you from accessing the backing store volume on a new node | 20:05 |
| portdirect | if we go to the other extreme - we do however expose ourselves to data loss | 20:06 |
| cfriesen | (apparently the only way around the k8s timeout is to delete the node entirely) | 20:06 |
| portdirect | eg - what if the pod thats not come up has received some tx that has not yet been synced to other pods? | 20:06 |
| *** jaypipes has quit IRC | 20:07 | |
| *** jaypipes has joined #openstack-helm | 20:07 | |
| cfriesen | portdirect: yeah, it's a tough call. do we recover automatically but risk some data loss, or extend the outage? | 20:07 |
| openstackgerrit | Gage Hugo proposed openstack/openstack-helm master: Add credential delete hook to keystone chart https://review.openstack.org/624131 | 20:07 |
| portdirect | cfriesen: we went for the latter, as we felt it safest | 20:08 |
| portdirect | and we can shorten it with some mitigations (eg the above) | 20:08 |
| *** jaypipes_ has joined #openstack-helm | 20:08 | |
| portdirect | but if there was a viable suggestion to relax the 'paranoia' here, we would welcome it | 20:08 |
| *** jaypipes has quit IRC | 20:12 | |
| portdirect | cfriesen: other than the issue above - how has your testing been going? | 20:16 |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm-infra master: WIP: MariaDB: Update backup scripts https://review.openstack.org/650950 | 20:16 |
| openstackgerrit | diwakar thyagaraj proposed openstack/openstack-helm-infra master: Add default AppArmor profile to Fluentbit Make MAC profile annotation optional because nothing is currently defined in values.yaml https://review.openstack.org/647881 | 20:17 |
| openstackgerrit | Doug Aaser proposed openstack/openstack-helm-infra master: [WIP] Patroni inclusion work for HA Postgres https://review.openstack.org/644388 | 20:17 |
| cfriesen | portdirect: I think there are a few issues that will be brought up if they haven't been already. | 20:23 |
| portdirect | cfriesen: nice (or not, as the case may be :) ) it would be great to get your findings tracked so we can ensure we address them | 20:25 |
| *** unicell has left #openstack-helm | 20:48 | |
| *** jaypipes_ is now known as jaypipes | 20:51 | |
| openstackgerrit | Meghan Heisler proposed openstack/openstack-helm-infra master: Add egress network policy to LMA services https://review.openstack.org/642555 | 21:07 |
| cfriesen | portdirect: had a thought...what about specifying a timeout in the values file. the system would behave as it does now initially, but any nodes that haven't updated themselves after the timeout expires would have their information wiped from the state configmap | 21:23 |
| portdirect | how would this protect against potential data loss though? | 21:30 |
| portdirect | or would we work on the assumption that if you set/enabled this option that you'd be accepting of the risk? | 21:31 |
| cfriesen | portdirect: right..so basically the operator could say "if the system hasn't come up within 30 minutes" (or whatever the threshold is) then we want to come up anyway even if it means data loss | 21:37 |
| cfriesen | could default to zero, meaning we wait forever | 21:37 |
| portdirect | cfriesen: how many nodes are in a starling-x cluster? | 21:42 |
| portdirect | i wonder if a deployment and single replica may make more sense here? | 21:42 |
| cfriesen | we have one or two controller nodes. the single node case is easy. the dual-node case is trickier because galera doesn't like that scenario. | 21:43 |
| cfriesen | if we have compute nodes we run a garbd instance on a compute node | 21:43 |
| cfriesen | but the two-node-only case is tricky. what we've ended up doing is running a separate script that will bootstrap the database on the "active" controller if things go south. | 21:44 |
| alanmeadows | actually purging PVC data seems a bit drastic | 21:44 |
| cfriesen | alanmeadows: not purging pvc data, just configmap data | 21:45 |
| alanmeadows | or are you saying just config state? | 21:45 |
| alanmeadows | i see | 21:45 |
| cfriesen | portdirect: I tried a deployment with a single replica, but ran into problem if the node the replica was on died. the pod would start up just fine on the running controller, but kubernetes woudln't let the new node access the PV until after a ~5min timeout | 21:46 |
| alanmeadows | in the single replica case, it may just make sense to setup some sort of external reaper/watcher that can both potentially help unlock the PVC and expire the pod on the lost node; in the very specific two-node-only case the right solution may just be not to be using galera and create an Active-Active or Active-Standby mysql chart? | 21:57 |
| alanmeadows | I mean sometimes galera is right, sometimes it isn't.... | 21:58 |
| cfriesen | alanmeadows: are you aware of any mechanism to unlock the PVC to let it be accessed by another node? I couldn't find one. | 22:13 |
| cfriesen | even force-deleting the pod on the "dead" node didn't allow me to access the PV on the running node until the timeout was up | 22:15 |
| cfriesen | the k8s folks seemed to think the only way to speed it up was to delete the kubernetes node and re-provision it once it came back up | 22:16 |
| alanmeadows | in my experience this has been very storage backend specific | 22:25 |
| cfriesen | if we had a PV capable of multi-attach it'd be much simpler, but at the moment we don't | 22:26 |
| alanmeadows | e.g. in older versions of k8s on ceph based volumes, locks were created that you could clear (or had to if things got gummed up). Now it checks for other in use "active" clients prior to allowing an attachment, which in theory should be relatively quick | 22:27 |
| alanmeadows | other backends may have other mechanisms for allowing reattachment | 22:27 |
| alanmeadows | or rather, early reattachment | 22:28 |
| portdirect | cfriesen: re read-write many PVC, you may want to try cephfs? | 22:29 |
| cfriesen | yeah, I think it's on the list to add eventually. are you folks using it? | 22:29 |
| portdirect | Not atm, though its deployed and available with the ceph charts today. | 22:30 |
| * alanmeadows contemplates whether your situation is improved with galera atop cephfs | 22:30 | |
| alanmeadows | ;-) | 22:30 |
| portdirect | Though I really wonder if Alan's suggestion of moving off galera is the right choice here | 22:31 |
| portdirect | As trying to fit a system requiring a quorum of nodes into an even number is always gonna present challenges | 22:32 |
| *** kranthikirang has quit IRC | 22:32 | |
| alanmeadows | certainly makes more sense in making a two node installation a first class citizen | 22:32 |
| cfriesen | I think if we had multi-attach PVs we could use a deployment instead of a statefulset and then force-kill the pod if we know the node it's on has gone down. | 22:32 |
| portdirect | One thing that does trouble me here | 22:33 |
| cfriesen | I know, it's a hack. :) | 22:33 |
| *** michaelbeaver has joined #openstack-helm | 22:33 | |
| portdirect | Is the issue you are seeing occurring on more than just a single node reboot? | 22:33 |
| portdirect | Sorry, on a single node reboot | 22:33 |
| cfriesen | the specific scenario in question is if you have a power outage or something so all nodes are down. then all the nodes come up except for one | 22:34 |
| cfriesen | what'll happen is that all the nodes sit there forever in /tmp/start.py | 22:35 |
| *** michael-beaver has quit IRC | 22:35 | |
| *** michaelbeaver has quit IRC | 22:37 | |
| openstackgerrit | Georg Kunz proposed openstack/openstack-helm-images master: Adding support for DPDK to openvswitch image https://review.openstack.org/650152 | 23:11 |
| openstackgerrit | Dmitrii Kabanov proposed openstack/openstack-helm-images master: [Ceph] Update Ceph repository and key https://review.openstack.org/651028 | 23:29 |
| openstackgerrit | Dmitrii Kabanov proposed openstack/openstack-helm-images master: [Ceph] Update Ceph repository and key https://review.openstack.org/651028 | 23:34 |
| openstackgerrit | Dmitrii Kabanov proposed openstack/openstack-helm-images master: [Ceph] Update Ceph repository and key https://review.openstack.org/651036 | 23:40 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!