| openstackgerrit | Sophie Huang proposed openstack/openstack-helm-infra master: [WIP] Add support for Cinder external ceph backend https://review.opendev.org/751624 | 01:24 |
|---|---|---|
| openstackgerrit | Sophie Huang proposed openstack/openstack-helm master: [WIP] Add support for Cinder external ceph backend https://review.opendev.org/751625 | 01:39 |
| *** yingjisun has joined #openstack-helm | 02:11 | |
| *** evrardjp has quit IRC | 04:33 | |
| *** evrardjp has joined #openstack-helm | 04:33 | |
| *** yingjisun has quit IRC | 04:40 | |
| *** yingjisun has joined #openstack-helm | 05:47 | |
| *** dasp has quit IRC | 05:48 | |
| *** yingjisun has quit IRC | 10:06 | |
| mnaser | is there a way to handle release differences inside loci? building the libvirt image i'm noticing that bionic and xenial have different images | 12:56 |
| mnaser | s/images/package names/ | 12:57 |
| mnaser | so the bindep.txt that works for one doesnt work for another | 12:57 |
| openstackgerrit | Sophie Huang proposed openstack/openstack-helm master: [WIP] Configuration for external ceph backend https://review.opendev.org/750520 | 13:05 |
| *** irclogbot_0 has quit IRC | 13:19 | |
| *** irclogbot_3 has joined #openstack-helm | 13:25 | |
| *** mnaser has quit IRC | 13:32 | |
| *** mnaser has joined #openstack-helm | 13:32 | |
| *** mnaser has quit IRC | 13:32 | |
| *** mnaser has joined #openstack-helm | 13:32 | |
| *** mfixtex has joined #openstack-helm | 13:47 | |
| *** mfixtex has quit IRC | 13:47 | |
| openstackgerrit | Gage Hugo proposed openstack/openstack-helm master: [WIP] Test multinode gating https://review.opendev.org/751816 | 14:11 |
| *** lamt has joined #openstack-helm | 14:16 | |
| *** mfixtex has joined #openstack-helm | 14:18 | |
| *** mfixtex has quit IRC | 14:45 | |
| openstackgerrit | Merged openstack/openstack-helm-infra master: [update] Node problem detector path for conntrack https://review.opendev.org/751064 | 14:51 |
| *** ianychoi has joined #openstack-helm | 15:24 | |
| openstackgerrit | Prateek Dodda proposed openstack/openstack-helm master: [WIP] Add missing security context to Placement pods/containers https://review.opendev.org/751846 | 16:01 |
| *** dasp has joined #openstack-helm | 16:46 | |
| openstackgerrit | Prateek Dodda proposed openstack/openstack-helm master: Implement missing security context for placement container https://review.opendev.org/751846 | 16:55 |
| *** dasp has quit IRC | 16:58 | |
| *** dasp has joined #openstack-helm | 17:00 | |
| openstackgerrit | Prateek Dodda proposed openstack/openstack-helm master: Implement missing security context for placement container https://review.opendev.org/751846 | 17:12 |
| openstackgerrit | Merged openstack/openstack-helm-images master: Hide the password in nagios error message https://review.opendev.org/748280 | 17:29 |
| openstackgerrit | Prateek Dodda proposed openstack/openstack-helm master: Implement missing security context for placement container https://review.opendev.org/751846 | 17:51 |
| openstackgerrit | Prateek Dodda proposed openstack/openstack-helm master: Implement missing security context for placement container https://review.opendev.org/751846 | 18:03 |
| openstackgerrit | Stephen Taylor proposed openstack/openstack-helm-infra master: [ceph-osd] Don't set CEPH_LVM_PREPARE to 0 for colocated db/wal https://review.opendev.org/751410 | 18:06 |
| openstackgerrit | Dmitrii Kabanov proposed openstack/openstack-helm-infra master: [WIP] Add Ceph CSI plugin https://review.opendev.org/750844 | 18:13 |
| openstackgerrit | Dmitrii Kabanov proposed openstack/openstack-helm-infra master: [WIP] Add Ceph CSI plugin https://review.opendev.org/750844 | 18:23 |
| openstackgerrit | Merged openstack/openstack-helm master: Add force cron jobs test run https://review.opendev.org/751035 | 18:24 |
| openstackgerrit | Chi Lo proposed openstack/openstack-helm-addons master: WIP - Create region retries when Ranger server not available https://review.opendev.org/751309 | 19:03 |
| mnaser | has anyone seen an issue where log lines are duplicated? | 19:25 |
| mnaser | i can confirm it even by seeing /var/log/containers/... | 19:26 |
| portdirect | srwilkers: ^ | 19:26 |
| mnaser | also https://review.opendev.org/#/c/751556/ is a small and simple osh fix | 19:27 |
| mnaser | https://review.opendev.org/#/c/751582/ is a loci patch that helps us when we already have a pypi wheel or trying to use an external one (we cheated, we use opendev's with that patch to speed up requirements image build) | 19:27 |
| mnaser | for https://review.opendev.org/#/c/751580/ -- i'm wondering if there was ever an established pattern for images where one OS release doesn't match another in bindep (libvirt package underwent changes from xenial to bionic..) | 19:28 |
| mnaser | portdirect: would you remember why 2.5 years sago you decided to change listen_addr to 127.0.0.1 from 0.0.0.0 in the libvirt chart? :) https://github.com/openstack/openstack-helm/commit/3f8895b2b7cb96b5690a06aad97f5893ebd113c0#diff-be7832f8cbd1813f5776285e8ce8cba6R47 | 19:36 |
| mnaser | because of that change, live migrations stop working (but it's also a more secure setup too) | 19:37 |
| portdirect | yes - to avoid having to set up auth for it ;) | 19:37 |
| mnaser | portdirect: so you don't do live migrations or use tunneled live migrations then? | 19:37 |
| portdirect | speaking in a work capacity - no | 19:38 |
| mnaser | i've been trying to avoid having to play with setting up libvirt tls | 19:38 |
| mnaser | but i think this might be the nail that forces me to do it :) | 19:38 |
| portdirect | yeah - things are a lot better now than they were | 19:39 |
| mnaser | i wonder if cert-manager might be the thing to use for this trick, or ill have to look into osh's existing infra | 19:39 |
| portdirect | not that osh uses cert-manager to look after certs this should be a lot simpler today than it would have been a couple of years ago | 19:39 |
| mnaser | as a stop gap, maybe tunneled could be the solution | 19:39 |
| portdirect | we had ssh tunnels with a single pre-shared key as a poc, but id go the tls route if looking to do this properly | 19:40 |
| mnaser | i mean given that for cold migration we already setup ssh (i haven't played with that), we could just leverage that for that. i'm unsure at how libvirt works in tls world right now | 19:40 |
| mnaser | sorry, i mean osh in tls world. i see some lines about "certificates" but yeah. | 19:41 |
| portdirect | lamt & gagehugo would be the best to help there | 19:41 |
| srwilkers | Lol portdirect - why you pinging me? | 20:01 |
| portdirect | oh, just its not only me wondering about double logging from container std out ;) | 20:02 |
| portdirect | you got any advice for mnaser ? | 20:02 |
| mnaser | yeah i tried to observe if there was anything causing it and the only thing that was coming to me was perhaps the fact we do an exec | 20:03 |
| srwilkers | azure logging is great | 20:03 |
| srwilkers | There, /advice | 20:03 |
| mnaser | and so exec prints stdout but also the service itself prints to stdout? | 20:03 |
| srwilkers | Jokes aside, I’ve seen it occasionally but never quite found out the root cause | 20:04 |
| portdirect | mnaser: i supect thats the wrong tree | 20:04 |
| portdirect | quite a lot of stuff is done in here for example: https://github.com/openstack/openstack-helm/blob/master/heat/values.yaml#L503-L562 | 20:05 |
| portdirect | for a time, i know things were writing both to stderr and stdout... | 20:05 |
| mnaser | portdirect: i was just looking there actually | 20:05 |
| mnaser | in nova, loggers/keys=root,nova,os.brick | 20:05 |
| mnaser | logger_root handles level WARNING to stdout, logger_nova handles level INFO to stdout too | 20:06 |
| mnaser | WARNING is not repeated but INFO is | 20:07 |
| mnaser | INFO is not* | 20:08 |
| mnaser | https://github.com/openstack/nova/blob/master/etc/nova/logging_sample.conf | 20:09 |
| mnaser | in nova upstream, handlers = null for logger_root | 20:09 |
| mnaser | which i am going to guess is there to stop the double logs.. | 20:09 |
| srwilkers | megheisler / stevthedev | 20:09 |
| srwilkers | can you guys chime in there? you're a little closer to all this than i am at this point | 20:09 |
| srwilkers | mnaser is a cool guy - he dont bite | 20:09 |
| mnaser | so i think maybe its a matter of syncing up the logging config to the nova defaults and we can all have half our log storage space back :) | 20:10 |
| srwilkers | just use the mangodb logging driver and you gucci | 20:11 |
| mnaser | srwilkers: web scale :) | 20:11 |
| srwilkers | :D | 20:11 |
| mnaser | Open10K8S: can you see discussion above and push up a patch to openstack-helm charts to update the values for logging to match the ones upstream? (e.g the ones for nova above?) | 20:11 |
| srwilkers | i'll +1 the hell out of it | 20:12 |
| srwilkers | that's all i can do anymore | 20:12 |
| Open10K8S | mnaser: ok | 20:17 |
| * mnaser hmms out loud | 20:19 | |
| mnaser | https://github.com/openstack/openstack-helm/blob/master/nova/templates/bin/_ssh-start.sh.tpl | 20:19 |
| mnaser | first i removed readonlyfilesystem which should have fixed things but now the issue is | 20:19 |
| mnaser | i (assume) the container runs as 42424 which cannot run ssh-keygen /etc/ssh/... | 20:20 |
| mnaser | seeing "Saving key "/etc/ssh/ssh_host_rsa_key" failed: Permission denied" | 20:20 |
| mnaser | i wonder if this should be `runAsUser: 0` | 20:20 |
| portdirect | mnaser: honestly i dont think that code has been tested in >18 months, and is not gated... | 20:20 |
| mnaser | portdirect: oh yeah, i'm well aware :P | 20:20 |
| mnaser | i'm just wondering what's the best osh-y fix that can be implemented | 20:21 |
| mnaser | as i slowly pick up those patterns | 20:21 |
| srwilkers | just throw up a change and we can discuss it in code review - probably the best way to do it | 20:22 |
| srwilkers | terms like "best osh-y fix" are pretty loaded | 20:22 |
| mnaser | :) fair enough | 20:22 |
| openstackgerrit | Oleksandr Kozachenko proposed openstack/openstack-helm master: Sync logging values with upstream repos https://review.opendev.org/751896 | 20:37 |
| Open10K8S | mnaser: made a PS | 20:38 |
| openstackgerrit | Merged openstack/openstack-helm-infra master: [ceph-osd] Don't set CEPH_LVM_PREPARE to 0 for colocated db/wal https://review.opendev.org/751410 | 20:44 |
| openstackgerrit | Chi Lo proposed openstack/openstack-helm-addons master: WIP - Create region retries when Ranger server not available https://review.opendev.org/751309 | 20:52 |
| mnaser | Open10K8S: i guess now we can wait for CI and see if it's happy with it doing double logs | 20:54 |
| openstackgerrit | Dmitrii Kabanov proposed openstack/openstack-helm-infra master: [WIP] Add Ceph CSI plugin https://review.opendev.org/750844 | 21:01 |
| Open10K8S | mnaser: ok | 21:10 |
| openstackgerrit | Rahul Khiyani proposed openstack/openstack-helm-infra master: Ingress: Configure ingress dhparam secret https://review.opendev.org/751902 | 21:18 |
| openstackgerrit | Merged openstack/openstack-helm-infra master: [ceph-client] Update queries in wait_for_pgs function https://review.opendev.org/751084 | 21:24 |
| openstackgerrit | Rahul Khiyani proposed openstack/openstack-helm-infra master: [WIP]Ingress: Configure ingress dhparam secret https://review.opendev.org/751902 | 21:38 |
| openstackgerrit | Rahul Khiyani proposed openstack/openstack-helm-infra master: [WIP]Ingress: Configure ingress dhparam secret https://review.opendev.org/751902 | 21:49 |
| openstackgerrit | Chi Lo proposed openstack/openstack-helm-addons master: WIP - Create region retries when Ranger server not available https://review.opendev.org/751309 | 21:57 |
| openstackgerrit | Merged openstack/openstack-helm master: neutron: fix default value for tungsten fabric https://review.opendev.org/751556 | 22:06 |
| openstackgerrit | Merged openstack/openstack-helm master: Do not pass extra config files to neutron agents https://review.opendev.org/739430 | 22:06 |
| megheisler | sorry mnaser missed this earlier, I've occasionally seen doubles as well but hadn't figured out the common thread to track down the cause. | 22:12 |
| mnaser | megheisler: see https://review.opendev.org/751896 :) i think that might be it | 22:13 |
| megheisler | great! I'll check it out | 22:14 |
| openstackgerrit | Rahul Khiyani proposed openstack/openstack-helm-infra master: [WIP]Ingress: Configure ingress dhparam secret https://review.opendev.org/751902 | 22:20 |
| *** portdirect has quit IRC | 23:13 | |
| *** portdirect has joined #openstack-helm | 23:13 | |
| *** ChanServ sets mode: +o portdirect | 23:13 | |
| openstackgerrit | Dmitrii Kabanov proposed openstack/openstack-helm-infra master: [WIP] Add Ceph CSI plugin https://review.opendev.org/750844 | 23:41 |
| openstackgerrit | Dmitrii Kabanov proposed openstack/openstack-helm-infra master: [WIP] Add Ceph CSI plugin https://review.opendev.org/750844 | 23:41 |
| *** yingjisun has joined #openstack-helm | 23:47 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!