| openstackgerrit | Mohammed Naser proposed openstack/openstack-helm master: Add openstack-helm chart publish jobs https://review.opendev.org/762441 | 00:21 |
|---|---|---|
| openstackgerrit | Mohammed Naser proposed openstack/openstack-helm-infra master: Make publish jobs more generic https://review.opendev.org/762442 | 00:23 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-helm master: Add openstack-helm chart publish jobs https://review.opendev.org/762441 | 00:23 |
| *** jhesketh has joined #openstack-helm | 00:49 | |
| *** thiagop has quit IRC | 01:28 | |
| *** evrardjp has quit IRC | 05:33 | |
| *** evrardjp has joined #openstack-helm | 05:33 | |
| *** vsaienk0 has joined #openstack-helm | 06:24 | |
| *** roman_g has joined #openstack-helm | 07:14 | |
| *** hemna has quit IRC | 07:50 | |
| *** hemna has joined #openstack-helm | 07:56 | |
| openstackgerrit | Gayathri Devi proposed openstack/openstack-helm-infra master: Add "TMOUT" parameter in the pods https://review.opendev.org/762322 | 10:57 |
| *** thiagop has joined #openstack-helm | 11:58 | |
| openstackgerrit | Gayathri Devi proposed openstack/openstack-helm master: Add "TMOUT" parameter in the pods https://review.opendev.org/761878 | 13:28 |
| openstackgerrit | Gayathri Devi proposed openstack/openstack-helm-infra master: Add "TMOUT" parameter for pods in the openstack namespace https://review.opendev.org/762322 | 14:21 |
| openstackgerrit | Tin Lam proposed openstack/openstack-helm master: DNM: test https://review.opendev.org/762361 | 15:09 |
| openstackgerrit | Sergey Galkin proposed openstack/openstack-helm master: Rework Rally chart https://review.opendev.org/726998 | 15:10 |
| openstackgerrit | Sergey Galkin proposed openstack/openstack-helm master: Rework Rally chart https://review.opendev.org/726998 | 15:12 |
| *** miniroy has joined #openstack-helm | 15:28 | |
| mnaser | how are y'all doing upgrades in helm right now? | 18:47 |
| mnaser | i'm getting a lot of upgrade fails because of things like the job spec is immutable | 18:47 |
| mnaser | i'd have to delete all of the jobs in order to be able to progress | 18:48 |
| portdirect | so | 18:48 |
| portdirect | we use armada for this | 18:48 |
| *** roman_g has quit IRC | 18:48 | |
| portdirect | its one of the things it lets us do: https://github.com/openstack/openstack-helm/blob/master/tools/deployment/armada/manifests/armada-osh.yaml#L36-L40 | 18:49 |
| mnaser | aaaaaah | 18:49 |
| mnaser | i wonder if helm hooks are powerful enough to do this natively now | 18:49 |
| portdirect | but you could also add a pre-upgrade hook that launches a job itself to delete/cleanup the artifacts | 18:49 |
| portdirect | they are | 18:49 |
| portdirect | not super clean | 18:49 |
| portdirect | but the above aproach is one ive thought about doing several times | 18:50 |
| portdirect | just not had the time to do so | 18:50 |
| mnaser | so pretty much launch something with a service user that has the appropriate credentials to delete all the jobs | 18:50 |
| portdirect | yup | 18:50 |
| mnaser | as a pre-upgrade hook | 18:50 |
| mnaser | i think the step before that might be to properly label all the jobs, as they don't seem to use the 'application' hook, at least for some services | 18:51 |
| portdirect | ++ theyre may be an even cleaner way to do that now, but if so ive not seen it yet | 18:51 |
| portdirect | urg - they should :( | 18:51 |
| mnaser | yeah for keystone i only see fernet/credential rotate | 18:51 |
| portdirect | hmm - the manifests for the cron jobs there should have the labels at the job level | 18:52 |
| portdirect | probably just got missed :( | 18:52 |
| mnaser | portdirect: yeah no worries .. i guess we can dig into this when we get around finishing our conversions to osh | 18:53 |
| mnaser | portdirect: a workaround seems to be like this -- https://github.com/grafana/helm-charts/commit/c69478eca9dcabb80a6d0b2ca2611c2d4cd349d1 | 18:54 |
| mnaser | but that means running them on every single deploy | 18:54 |
| mnaser | which i think might be an acceptable workaround, so post-install it will delete the job if it succeeds | 18:54 |
| openstackgerrit | Tin Lam proposed openstack/openstack-helm master: DNM: test https://review.opendev.org/762361 | 18:56 |
| mnaser | portdirect: filed https://bugs.launchpad.net/openstack-helm/+bug/1904046 and https://bugs.launchpad.net/openstack-helm/+bug/1904047 -- we're a little tight on time these days but once we get around it, we can probably push some efforts on those | 18:59 |
| openstack | Launchpad bug 1904046 in openstack-helm "Jobs missing labels for applications" [Undecided,New] | 18:59 |
| openstack | Launchpad bug 1904047 in openstack-helm "Jobs should be cleaned up after deploys" [Undecided,New] | 18:59 |
| portdirect | mnaser: that would be awesome | 18:59 |
| portdirect | unless of course gagehugo or someone hits them 1st ;) | 19:00 |
| mnaser | aha, yep, it's a pretty straight forward thing, just requires a bunch of.. copy-pasta ;p | 19:00 |
| openstackgerrit | Tin Lam proposed openstack/openstack-helm master: DNM: test https://review.opendev.org/762361 | 19:00 |
| *** miniroy has quit IRC | 19:01 | |
| openstackgerrit | Tin Lam proposed openstack/openstack-helm master: DNM: test https://review.opendev.org/762361 | 19:15 |
| openstackgerrit | Tin Lam proposed openstack/openstack-helm master: DNM: test https://review.opendev.org/762361 | 19:34 |
| openstackgerrit | Tin Lam proposed openstack/openstack-helm master: DNM: test https://review.opendev.org/762361 | 20:02 |
| gagehugo | mnaser: for this ps https://review.opendev.org/#/c/759136/ it seem that the secrets are getting deleted on upgrade even with that hook, and then not recreated at all | 20:17 |
| gagehugo | fernet keys are meh, credential keys can be bad | 20:18 |
| mnaser | gagehugo: ugh, sorry, i tested this locally and i think my fix was adding `helm.sh/hook-delete-policy: hook-failed` | 20:23 |
| mnaser | so it would only delete it if the hook failed which, it's a secret, so it never will | 20:23 |
| mnaser | let me quickly push, sorry | 20:23 |
| gagehugo | interesting | 20:24 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-helm master: keystone: do not delete secrets on deploy https://review.opendev.org/762575 | 20:27 |
| mnaser | gagehugo: ^ sorry for the fact i missed it, i fixed it locally and been all over the place so missed that extra patch. i left an explanation in the comment | 20:27 |
| mnaser | i've deployed many times since and without a problem | 20:27 |
| gagehugo | mnaser: no worries haha, | 20:28 |
| gagehugo | might need to bump the chart version though | 20:31 |
| openstackgerrit | Tin Lam proposed openstack/openstack-helm master: DNM: test https://review.opendev.org/762361 | 20:36 |
| gagehugo | Doesn't seem to work for me locally, the secrets still get deleted when you upgrade | 20:37 |
| mnaser | gagehugo: weird, it does work for me. i wonder if it has to upgrade 'once' to add the new annotation before it stops being wiped? | 20:44 |
| gagehugo | I am upgrading from 0.1.2 to the new one | 20:44 |
| gagehugo | maybe? | 20:44 |
| mnaser | right but maybe you need to upgrade twice, cause the first time it processes the hook in the original behaviour | 20:44 |
| gagehugo | that still causes secret deletion though :/ | 20:49 |
| mnaser | gagehugo: helm 2 or helm 3? | 20:55 |
| gagehugo | 2 :( | 20:55 |
| gagehugo | I think tiller is deleting it | 20:56 |
| mnaser | gagehugo: i think maybe that might be the difference, we use 3 -- perhaps we can find a way to detect the helm version | 21:01 |
| mnaser | i think there is a value | 21:01 |
| openstackgerrit | Tin Lam proposed openstack/openstack-helm master: DNM: test https://review.opendev.org/762361 | 21:17 |
| gagehugo | mnaser: many github pull requests tell me it cannot be done in helm2 unfortunately | 21:28 |
| gagehugo | portdirect: ^ | 21:28 |
| mnaser | gagehugo: darn, well i'm not sure what's the best way forward -- i think y'all still heavily use helm2 but it also just got eol'd | 21:30 |
| gagehugo | not sure, all the fixes were put into 3 | 21:38 |
| openstackgerrit | Gage Hugo proposed openstack/openstack-helm-deployments master: WIP - Add keystone https://review.opendev.org/762586 | 21:55 |
| openstackgerrit | Tin Lam proposed openstack/openstack-helm master: DNM: test https://review.opendev.org/762361 | 22:16 |
| openstackgerrit | Tin Lam proposed openstack/openstack-helm master: DNM: test https://review.opendev.org/762361 | 22:33 |
| openstackgerrit | Tin Lam proposed openstack/openstack-helm master: DNM: test https://review.opendev.org/762361 | 23:20 |
| openstackgerrit | Tin Lam proposed openstack/openstack-helm-images master: feat(os): adds victoria images https://review.opendev.org/758520 | 23:23 |
| openstackgerrit | Tin Lam proposed openstack/openstack-helm-images master: chore(os): removes build of old OpenStack releases https://review.opendev.org/758521 | 23:23 |
| openstackgerrit | Tin Lam proposed openstack/openstack-helm master: DNM: test https://review.opendev.org/762361 | 23:32 |
| openstackgerrit | Tin Lam proposed openstack/openstack-helm master: DNM: test https://review.opendev.org/762361 | 23:42 |
| openstackgerrit | Tin Lam proposed openstack/openstack-helm master: DNM: test https://review.opendev.org/762361 | 23:58 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!