| *** iniazi has joined #openstack-kolla | 01:58 | |
| *** iniazi_ has quit IRC | 02:00 | |
| *** BjoernT has joined #openstack-kolla | 02:22 | |
| *** BjoernT_ has joined #openstack-kolla | 02:26 | |
| *** BjoernT has quit IRC | 02:26 | |
| *** JamesBenson has joined #openstack-kolla | 02:35 | |
| *** BjoernT_ has quit IRC | 02:42 | |
| *** BjoernT has joined #openstack-kolla | 02:45 | |
| *** BjoernT has quit IRC | 04:06 | |
| *** BjoernT has joined #openstack-kolla | 04:14 | |
| *** JamesBenson has quit IRC | 04:31 | |
| *** BjoernT has quit IRC | 04:46 | |
| *** dave-mccowan has quit IRC | 04:48 | |
| *** Luzi has joined #openstack-kolla | 05:00 | |
| *** skramaja has joined #openstack-kolla | 05:15 | |
| *** dpawlik has joined #openstack-kolla | 06:00 | |
| osmanlicilegi | morning | 06:11 |
|---|---|---|
| openstackgerrit | Dincer Celik proposed openstack/kolla-ansible master: [prometheus] Added support for extra options https://review.opendev.org/682834 | 06:11 |
| *** luksky has joined #openstack-kolla | 06:17 | |
| *** luksky has quit IRC | 06:23 | |
| yoctozepto | morning | 06:33 |
| *** egonzalez has joined #openstack-kolla | 06:38 | |
| *** rpittau|afk is now known as rpittau | 06:40 | |
| *** k_mouza has joined #openstack-kolla | 06:42 | |
| *** k_mouza has quit IRC | 06:55 | |
| *** egonzalez has quit IRC | 07:09 | |
| *** ivve has joined #openstack-kolla | 07:10 | |
| *** Blinkiz has joined #openstack-kolla | 07:18 | |
| *** egonzalez has joined #openstack-kolla | 07:33 | |
| *** pcaruana has joined #openstack-kolla | 07:41 | |
| openstackgerrit | Chason Chan proposed openstack/kolla-ansible stable/stein: Specific the release of kolla-ansible in quickstart guide https://review.opendev.org/683855 | 07:44 |
| *** cah_link has joined #openstack-kolla | 07:50 | |
| openstackgerrit | Chason Chan proposed openstack/kolla-ansible stable/rocky: Specific the release of kolla-ansible in quickstart guide https://review.opendev.org/683856 | 07:50 |
| openstackgerrit | Chason Chan proposed openstack/kolla-ansible stable/rocky: Specific the release of kolla-ansible in quickstart guide https://review.opendev.org/683856 | 07:53 |
| openstackgerrit | Chason Chan proposed openstack/kolla-ansible stable/rocky: Specific the release of kolla-ansible in quickstart guide https://review.opendev.org/683856 | 07:54 |
| openstackgerrit | Chason Chan proposed openstack/kolla-ansible master: Specific the release of kolla-ansible in quickstart guide https://review.opendev.org/683857 | 07:57 |
| openstackgerrit | Chason Chan proposed openstack/kolla-ansible stable/stein: Specific the release of kolla-ansible in quickstart guide https://review.opendev.org/683855 | 07:58 |
| *** gregoryo has joined #openstack-kolla | 08:21 | |
| openstackgerrit | Dincer Celik proposed openstack/kolla-ansible master: [prometheus] Added support for extra options https://review.opendev.org/682834 | 08:25 |
| yoctozepto | osmanlicilegi: wdyt about writing prometheus usage docs for k-a? | 08:29 |
| osmanlicilegi | yoctozepto: well i'm not a prometheus guru but i can create a basic document for deployment and other stuff | 08:39 |
| yoctozepto | osmanlicilegi: would be more than welcome :-) | 08:42 |
| yoctozepto | (considering we have null atm) | 08:42 |
| *** gregoryo has quit IRC | 08:55 | |
| *** gfidente has joined #openstack-kolla | 08:58 | |
| *** dpawlik has quit IRC | 09:04 | |
| *** dpawlik has joined #openstack-kolla | 09:04 | |
| *** k_mouza has joined #openstack-kolla | 09:06 | |
| mnasiadka | morning | 09:09 |
| openstackgerrit | Merged openstack/kolla master: CI: Use docker mirror proxy https://review.opendev.org/683038 | 09:36 |
| *** dpawlik has quit IRC | 09:37 | |
| *** dpawlik has joined #openstack-kolla | 09:38 | |
| mgoddard | morning | 09:40 |
| *** ondrejduchon has joined #openstack-kolla | 09:52 | |
| *** egonzalez has quit IRC | 09:55 | |
| ondrejduchon | mgoddard: Hello, I modified that logging.conf feature similar way as policy file is. Does it make more sense now? https://review.opendev.org/#/c/679812/ Thank you for reply. | 09:56 |
| *** ktibi has joined #openstack-kolla | 09:56 | |
| *** egonzalez has joined #openstack-kolla | 10:08 | |
| mgoddard | hi ondrejduchon, I'll look at it today | 10:12 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: CI: Reinstate use of Docker registry mirror https://review.opendev.org/682846 | 10:14 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Add <project>_install_type for all projects https://review.opendev.org/683367 | 10:23 |
| openstackgerrit | Mark Goddard proposed openstack/kolla stable/queens: sensu-client: install older whois gem https://review.opendev.org/683361 | 10:27 |
| Blinkiz | Hello. I have an unused OpenStack deployed with kolla. I do think Horizon is slow. Looking for example on a empty volumes page and it takes 1.52 seconds to load (TTFB) according to chrome. Is this normal? | 10:42 |
| Blinkiz | I do feel the API is also slow. Always waiting a second or two before showing me the results. | 10:43 |
| egonzalez | Blinkiz, hard to say, could be storage performance on horizon host, networking, DBs, ssl encryption? something specific said by chrome? | 10:43 |
| *** k_mouza has quit IRC | 10:43 | |
| Blinkiz | How do I troubleshoot this? I mean how can I find what is creating this delay? | 10:43 |
| Blinkiz | On the network topology page in Horizon there is a json request ongoing in the background. Every request takes ~750ms. Why? This is a completely empty cluster. What is creating this delay? | 10:45 |
| Blinkiz | If this is by design, then fine.. then I know.. | 10:46 |
| egonzalez | not sure, could be many things | 10:48 |
| egonzalez | whats your network setup, vms or baremetal servers and what capabilities have? docker storage setup, SSL on public APIs or plan http? | 10:49 |
| egonzalez | I mean, shouldn't slow by design | 10:50 |
| osmanlicilegi | Blinkiz: is it rocky? | 10:50 |
| Blinkiz | egonzalez, It is a new cluster with powerful specs. This is not a hardware problem but a software one. Some tuning that is off | 10:50 |
| Blinkiz | osmanlicilegi, Stein | 10:50 |
| Blinkiz | egonzalez, I have tested without haproxy in the way. I still got the same delay. | 10:51 |
| Blinkiz | egonzalez, I get the feeling of same delay by just using openstack cli tools working against the api | 10:51 |
| egonzalez | Blinkiz, cli API calls are slow to internal API? | 10:52 |
| Blinkiz | I do think api calls are slow but I have not measured these | 10:52 |
| Blinkiz | My experience about openstack has just started so am probably missing something in my setup | 10:52 |
| osmanlicilegi | Blinkiz: have you tried capturing and analysing network traffic? | 10:53 |
| osmanlicilegi | maybe there's something wrong with network. | 10:53 |
| egonzalez | mgoddard, are we aware aptira have its openstack version based on kolla? havent seen no one around here from them (aka possibility to reach them to open the doors for patches) https://aptira.com/openstack-container-orchestration-with-kolla-ansible/ | 10:56 |
| Blinkiz | osmanlicilegi, sure, maybe. Do you know if it possible to measure api request somehow? I mean isolate components in OpenStack and see how long they take to answer. | 10:56 |
| egonzalez | Blinkiz, maybe need to increase openstack_service_workers variable if your host have many cpus | 10:57 |
| *** dpawlik has quit IRC | 10:57 | |
| Blinkiz | egonzalez, okay, I look into that. thanks | 10:58 |
| yoctozepto | Blinkiz: the first few requests may feel slow | 10:58 |
| yoctozepto | due to services initializing for you | 10:58 |
| yoctozepto | being the first and only user | 10:58 |
| *** k_mouza has joined #openstack-kolla | 10:58 | |
| egonzalez | Blinkiz, but from my experience, docker storage performance is bad by default | 10:58 |
| Blinkiz | yoctozepto, I always have delay of 500ms to 1.5 seconds, it does not go away. | 10:59 |
| yoctozepto | I see | 10:59 |
| *** dpawlik has joined #openstack-kolla | 11:04 | |
| *** dave-mccowan has joined #openstack-kolla | 11:06 | |
| Blinkiz | About this suggestion about increasing openstack_service_workers. I have googled it and I can not find information about the option. I can not find a default value either in kolla source code. My three control nodes has 20 cores each plus hyperthreading. What is appropriate value for openstack_service_workers? | 11:06 |
| *** dpawlik has quit IRC | 11:09 | |
| Blinkiz | Oh, now I found the default value. openstack_service_workers: "{{ [ansible_processor_vcpus, 5]|min }}" | 11:10 |
| *** dpawlik has joined #openstack-kolla | 11:15 | |
| mgoddard | egonzalez: thanks for the info about aptira | 11:15 |
| egonzalez | Blinkiz, number of processes for service, in your example will use 5 wsgi processes for keystone and other services | 11:16 |
| Blinkiz | egonzalez, Thanks for the explanation. It seem like I have 5 workers on almost all services. | 11:17 |
| yoctozepto | Blinkiz: though remember the more you set, the more initial slowness you will get as each is being initialized for new request | 11:20 |
| Blinkiz | yoctozepto, okay, thanks | 11:21 |
| *** vmixor has joined #openstack-kolla | 11:21 | |
| *** dave-mccowan has quit IRC | 11:21 | |
| *** kplant has joined #openstack-kolla | 11:22 | |
| *** ktibi has quit IRC | 11:24 | |
| *** jbadiapa has quit IRC | 11:30 | |
| *** sean-k-mooney has joined #openstack-kolla | 11:34 | |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: Implement IPv6 support in the control plane https://review.opendev.org/681573 | 11:36 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: Implement IPv6 support in the control plane https://review.opendev.org/681573 | 11:37 |
| Blinkiz | Nice! IPv6! | 11:38 |
| * Blinkiz likes | 11:38 | |
| yoctozepto | yeah, I rebased my patch now ;D | 11:40 |
| yoctozepto | needs some work still | 11:40 |
| Blinkiz | Nice to see that it is worked on. | 11:40 |
| yoctozepto | bad thing is it conflicts with many patches because it changes addressing all over the place | 11:40 |
| yoctozepto | so will probably get more rebases during its lifetime | 11:41 |
| openstackgerrit | Merged openstack/kolla-ansible master: [prometheus] Added support for extra options https://review.opendev.org/682834 | 11:42 |
| egonzalez | yoctozepto, from my experience you may want to split the patch in several, ie: ansible plugin and globals config, then apply to core services, and then to missing services | 11:44 |
| egonzalez | or you will have the patch open with merge conflict for over a year | 11:44 |
| yoctozepto | egonzalez: I am bravely working on it but may look at the split of module/application, thanks | 11:46 |
| yoctozepto | year's not going to happen as we try to push it in the train cycle | 11:46 |
| *** zul has joined #openstack-kolla | 11:46 | |
| Blinkiz | yoctozepto, as a admin user I would like to see support for IPv4 and IPv6 at the same time at the frontend. Meaning haproxy. Internally I would like to run IPv6 only. | 11:47 |
| Blinkiz | So support for multiple IP addresses for haproxy to handle would be a nice feature. Or just one IPv4 and one IPv6. :) | 11:48 |
| yoctozepto | Blinkiz: sure, support for dual stack on the external side is the right approach for sure, now it's mostly about the control plane being ipv6 only | 11:51 |
| yoctozepto | fun thing, ipv6 only on the external side was done already yet not dual stack | 11:52 |
| yoctozepto | see https://review.opendev.org/635528 | 11:52 |
| Blinkiz | Great. I can not control if my users have IPv6 or not so I need to have dual stack on the external side. Internally I control everything so that can be IPv4 or IPv6. | 11:52 |
| egonzalez | yoctozepto, made a quick review of the plugin | 11:57 |
| *** goldyfruit_ has quit IRC | 11:57 | |
| egonzalez | yoctozepto, i think that creating a fact with each config address used may be a good idea in terms of performance on huge deployments | 11:58 |
| *** vmixor has quit IRC | 12:01 | |
| *** mixor has joined #openstack-kolla | 12:01 | |
| yoctozepto | egonzalez: thanks, it may improve the performance a bit indeed, though I doubt it would be noticeable compared to other sources of slowness | 12:06 |
| *** skramaja has quit IRC | 12:15 | |
| *** vmixor has joined #openstack-kolla | 12:18 | |
| *** mixor has quit IRC | 12:19 | |
| *** GonZo2000 has quit IRC | 12:20 | |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: Implement IPv6 support in the control plane https://review.opendev.org/681573 | 12:22 |
| yoctozepto | egonzalez: since you are around: https://review.opendev.org/683680 <- needs your attention | 12:24 |
| kplant | mgoddard: re: https://bugs.launchpad.net/kolla-ansible/+bug/1844851 -- is there a way i can submit a patch specifically for stable/stein? | 12:35 |
| openstack | Launchpad bug 1844851 in kolla-ansible train "[stein] Prometheus OpenStack Exporter Broken" [Medium,Triaged] | 12:35 |
| kplant | looks like master is already 0.2.1 | 12:35 |
| mgoddard | kplant: yes, just checkout the stable/stein branch, make your commit, then hit git review | 12:53 |
| mgoddard | kplant: .gitreview includes the branch name on stable branches | 12:53 |
| kplant | easy enough, thanks. i'll do that this morning | 12:53 |
| mgoddard | kplant: great. could you mark both 1838391 and 1844851 with Closes-Bug | 12:54 |
| kplant | sure | 12:54 |
| egonzalez | yoctozepto, script is only needed from rocky-> stein upgrade to move data from nova_api to placement database | 12:55 |
| *** egonzalez has quit IRC | 12:56 | |
| *** goldyfruit_ has joined #openstack-kolla | 13:01 | |
| *** mloza has joined #openstack-kolla | 13:02 | |
| *** rouk has joined #openstack-kolla | 13:04 | |
| openstackgerrit | Keith Plant proposed openstack/kolla stable/stein: Fix prometheus-openstack-exporter in Stein https://review.opendev.org/683951 | 13:09 |
| *** Luzi has quit IRC | 13:14 | |
| kplant | mgoddard: thanks for fixing that. i mindlessly commited with git and was trying to figure out how to redo it with bazaar so i could tag it properly | 13:17 |
| mgoddard | kplant: doh :) | 13:19 |
| kplant | yeah :-| | 13:19 |
| kplant | is there a way to replicate --fixes with just vanilla git? | 13:19 |
| mgoddard | --fixes? | 13:21 |
| kplant | yeah with bazaar you can commit with something like "--fixes lp:1234567" | 13:22 |
| kplant | and it will relate the patch with the bug | 13:22 |
| *** BjoernT has joined #openstack-kolla | 13:23 | |
| mgoddard | that's quite nice | 13:30 |
| *** goldyfruit___ has joined #openstack-kolla | 13:30 | |
| *** goldyfruit_ has quit IRC | 13:33 | |
| kplant | should i pull that patch back? | 13:35 |
| *** BjoernT has quit IRC | 13:36 | |
| *** dpawlik has quit IRC | 13:37 | |
| *** BjoernT has joined #openstack-kolla | 13:39 | |
| *** egonzalez has joined #openstack-kolla | 13:39 | |
| mgoddard | kplant: what do you mean? | 13:45 |
| *** signed8bit has joined #openstack-kolla | 13:50 | |
| kplant | looks like jorge is recommending using 0.2.1 and to your point, there's already a patch for that in master | 13:52 |
| *** redrobot has quit IRC | 13:53 | |
| *** egonzalez has quit IRC | 13:56 | |
| mgoddard | kplant: yes, if you backport Jorge's patch, you can just abandon yours | 13:58 |
| *** egonzalez has joined #openstack-kolla | 13:58 | |
| yoctozepto | egonzalez: ah, thanks, thought so but was not sure | 13:59 |
| *** JamesBenson has joined #openstack-kolla | 13:59 | |
| *** JamesBenson has quit IRC | 14:01 | |
| *** JamesBenson has joined #openstack-kolla | 14:01 | |
| *** Guest30550 has joined #openstack-kolla | 14:05 | |
| kplant | mgoddard: sure, is there a proper way to do that? | 14:07 |
| yoctozepto | kplant: in gerrit UI use cherrypick button | 14:07 |
| yoctozepto | "Cherry Pick" | 14:07 |
| yoctozepto | and select stable/stein branch | 14:07 |
| mgoddard | kplant: you can do it all through the gerrit UI. https://review.opendev.org/673311 | 14:07 |
| mgoddard | as yoctozepto says | 14:07 |
| mgoddard | and you patch can be abandoned through gerrit also | 14:08 |
| *** Guest30550 is now known as redrobot | 14:08 | |
| *** BjoernT has quit IRC | 14:09 | |
| kplant | appears to be a merge conflict | 14:12 |
| *** BjoernT has joined #openstack-kolla | 14:13 | |
| yoctozepto | kplant: then it's a bit more complicated | 14:13 |
| yoctozepto | git checkout stable/stein | 14:13 |
| yoctozepto | ^ locally | 14:13 |
| yoctozepto | then | 14:13 |
| yoctozepto | git review -X 673311 | 14:13 |
| yoctozepto | resolve the conflicts | 14:13 |
| yoctozepto | then add and commit | 14:14 |
| yoctozepto | and | 14:14 |
| yoctozepto | git review | 14:14 |
| openstackgerrit | Merged openstack/kolla stable/queens: sensu-client: install older whois gem https://review.opendev.org/683361 | 14:14 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Factor out nova-cell role https://review.opendev.org/675659 | 14:15 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Add service-rabbitmq role https://review.opendev.org/683974 | 14:15 |
| kplant | appreciate it | 14:16 |
| kplant | i'll take care of it | 14:16 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Ensure keepalived is restarted during upgrade https://review.opendev.org/683980 | 14:29 |
| openstackgerrit | Keith Plant proposed openstack/kolla stable/stein: [docker/openstack-exporter] Bump to 0.2.1 version https://review.opendev.org/683982 | 14:41 |
| kplant | ^ think i did that right | 14:42 |
| kplant | err, 1s | 14:42 |
| openstackgerrit | Keith Plant proposed openstack/kolla stable/stein: [docker/openstack-exporter] Bump to 0.2.1 version https://review.opendev.org/683982 | 14:43 |
| kplant | forgot to remove the .orig from the mergetool | 14:43 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: Implement IPv6 support in the control plane https://review.opendev.org/681573 | 14:43 |
| *** JamesBenson has quit IRC | 14:46 | |
| yoctozepto | kplant: we'll see ;-) | 14:52 |
| kplant | yeah.. sorry for the noob questions | 14:58 |
| kplant | thanks for the answers though | 14:58 |
| kplant | much appreciated | 14:58 |
| noxoid | thanks for submitting the patch for that kplant | 15:01 |
| *** gfidente has quit IRC | 15:02 | |
| kplant | my pleasure, it's also selfish since i use stable/stein + prometheus in production :] | 15:04 |
| noxoid | same, you beat me to it :P | 15:05 |
| *** pcaruana has quit IRC | 15:05 | |
| *** Blinkiz has quit IRC | 15:06 | |
| yoctozepto | kplant, noxoid: you are welcome to write prometheus docs then, I have already asked osmanlicilegi to do it just today and it would be nice if we finally had something about that since there certainly is some interest in prometheus here around | 15:08 |
| yoctozepto | never done any myself | 15:08 |
| noxoid | may be a bit premature for me, i literally deployed prometheus on friday lol | 15:08 |
| noxoid | got some testing to do and i want to close out my other designate fix MR first | 15:09 |
| egonzalez | noxoid what MR if i can review | 15:10 |
| noxoid | nah its blocked on me, i gotta add a doc note for it: https://review.opendev.org/#/c/682095/ | 15:10 |
| yoctozepto | noxoid: why prometheus anyway? | 15:11 |
| yoctozepto | is it some smashing monitoring hit? | 15:11 |
| noxoid | flipping one variable in kolla made it too easy to not try | 15:11 |
| noxoid | the company i work for is on a custom graphite plugin/telegraf kick which means id probably have to write a bunch of custom tooling | 15:12 |
| yoctozepto | I see | 15:13 |
| noxoid | plus ceph has native prometheus metric exporting | 15:13 |
| noxoid | and since we already have a grafana instance that i can just point at the prometheus server... | 15:13 |
| noxoid | seemed like an easy win | 15:13 |
| *** chrizl has joined #openstack-kolla | 15:17 | |
| *** k_mouza has quit IRC | 15:17 | |
| yoctozepto | nice, thanks for insights | 15:18 |
| noxoid | anytime | 15:19 |
| *** KeithMnemonic has joined #openstack-kolla | 15:22 | |
| openstackgerrit | James Kirsch proposed openstack/kolla-ansible master: Add support for self-signed SSL certificates https://review.opendev.org/664517 | 15:25 |
| *** jschlueter has quit IRC | 15:26 | |
| *** ondrejduchon has quit IRC | 15:38 | |
| *** cally725 has joined #openstack-kolla | 15:43 | |
| cally725 | Hi I am using kolla rocky and I have the process beam.smp that takes 220% of the cpu on one of my two controllers any body can help | 15:44 |
| *** jschlueter has joined #openstack-kolla | 15:45 | |
| *** k_mouza has joined #openstack-kolla | 15:46 | |
| yoctozepto | cally725: that is rabbitmq | 15:47 |
| yoctozepto | I assume you mean this is a permanent situation? | 15:47 |
| *** rpittau is now known as rpittau|afk | 15:52 | |
| *** k_mouza has quit IRC | 15:57 | |
| cally725 | well sometimes it comes back to normal | 15:57 |
| cally725 | but now it's been doing this for the last hour after deleteing a stack in in horizon | 15:58 |
| egonzalez | cally725 how many resources does your controller nodes have? | 15:59 |
| yoctozepto | and how complex that stack was | 16:03 |
| *** pcaruana has joined #openstack-kolla | 16:06 | |
| noxoid | curious, does anyone use hash_behaviour=merge with K-A? | 16:07 |
| noxoid | looking at overriding one key from defaults.yml in globals.yml without having to bring over a 30 line block | 16:07 |
| mgoddard | noxoid: which option in particular? | 16:11 |
| noxoid | sec | 16:11 |
| noxoid | https://opendev.org/openstack/kolla-ansible/src/branch/stable/stein/ansible/roles/prometheus/defaults/main.yml#L20 | 16:11 |
| noxoid | i havent looked at it too much to see if theres a different way to expose prometheus, but seems the most straightforward way | 16:12 |
| *** k_mouza has joined #openstack-kolla | 16:12 | |
| noxoid | but since by default ansible replaces the entire hash id have to bring that entire prometheus_service block with it | 16:12 |
| osmanlicilegi | cally725: take a look at https://www.rabbitmq.com/runtime.html#scheduling | 16:13 |
| mgoddard | noxoid: yeah those service definitions aren't really meant to be overridden. | 16:13 |
| noxoid | gotcha | 16:13 |
| noxoid | i may try setting it to merge to see if i spectacularly break anything. if not i can find another way to do what i need | 16:14 |
| noxoid | thanks | 16:14 |
| mgoddard | noxoid: if you think it's useful, we could make it optional (default no) to expose prometheus via the external API | 16:14 |
| noxoid | you thinking a group_vars/all variable like prometheus_server_external? | 16:15 |
| mgoddard | yeah, enable_prometheus_external_api? | 16:15 |
| noxoid | yea something like that would be useful for me in particular. i have two regions and certain infra services like grafana/elasticsearch/etc i dont want to run twice | 16:16 |
| noxoid | also easier to integrate our existing grafana install | 16:17 |
| noxoid | if you think its appropriate i can file a bug report/blueprint/whatever and work on it when i have time | 16:21 |
| mgoddard | it makes sense to me - please file a blueprint | 16:22 |
| noxoid | sure | 16:23 |
| noxoid | a bit of anecdotal evidence but i noticed that once i enabled prometheus-server ceph-mgr doesnt have a massive memory leak | 16:24 |
| mgoddard | does it normally? | 16:27 |
| noxoid | for my clusters yes | 16:27 |
| mgoddard | ok | 16:28 |
| noxoid | my region without prometheus is using 21.6G | 16:28 |
| noxoid | for ceph-mgr | 16:28 |
| mgoddard | ouch | 16:28 |
| noxoid | ive seen it get as high as 40G | 16:28 |
| noxoid | region with prometheus is using 400M | 16:28 |
| mgoddard | must be some exporter with data that gets leaked | 16:28 |
| noxoid | both daemons restarted on sep20 | 16:28 |
| noxoid | yea | 16:28 |
| noxoid | ive just been restarting them every few days | 16:29 |
| noxoid | i havent brought it up cause i REALLY need to switch to external ceph which may have it fixed in a newer release | 16:29 |
| *** BjoernT has quit IRC | 16:31 | |
| mgoddard | worth knowing for other people | 16:33 |
| mjturek | just a heads up, this https://review.opendev.org/#/c/682618/ doesn't seem to be working (see here https://centos.logs.rdoproject.org/tripleo-upstream-containers-build-master-ppc64le/1540/logs/logs/build.log ) | 16:34 |
| mjturek | (sorry for interrupting not related to current discussion) | 16:34 |
| mgoddard | hey mjturek | 16:34 |
| mjturek | hey mgoddard | 16:35 |
| mgoddard | I'll give it a try locally | 16:37 |
| mjturek | mgoddard: https://github.com/openstack/kolla/blob/master/docker/base/Dockerfile.j2#L116 do you happen to know where the rabbitmq-server.repo file comes from? | 16:39 |
| mjturek | I know it's a third party but not sure where it comes from | 16:39 |
| mgoddard | mjturek: yeah, there's a COPY command | 16:39 |
| mjturek | ah, looking | 16:39 |
| mgoddard | mjturek: L100 | 16:41 |
| mgoddard | mjturek: I think tripleo overrides those repo files | 16:41 |
| mjturek | mgoddard: that would make sense, as we hadn't hit the issue hrw was discussing | 16:41 |
| mgoddard | yeah | 16:42 |
| mgoddard | mjturek: could wrap it in test -f /etc/yum.repos.d/rabbitmq_rabbitmq-server.repo | 16:42 |
| *** jschlueter has quit IRC | 16:42 | |
| mgoddard | {% if base_arch == 'ppc64le' %} | 16:42 |
| mgoddard | -RUN sed -i -e 's/\$basearch/x86_64/g' /etc/yum.repos.d/rabbitmq_rabbitmq-server.repo | 16:43 |
| mgoddard | +RUN test -f /etc/yum.repos.d/rabbitmq_rabbitmq-server.repo \ | 16:43 |
| mgoddard | + && sed -i -e 's/\$basearch/x86_64/g' /etc/yum.repos.d/rabbitmq_rabbitmq-server.repo | 16:43 |
| mgoddard | {% endif %} | 16:43 |
| mgoddard | something like that | 16:43 |
| mjturek | that sounds like the right fix mgoddard | 16:43 |
| mgoddard | mjturek: are you going to propose it? | 16:43 |
| mjturek | mgoddard: sure if you'd like | 16:44 |
| mgoddard | mjturek: that'd be great | 16:44 |
| *** jschlueter has joined #openstack-kolla | 16:45 | |
| *** pcaruana has quit IRC | 16:46 | |
| *** k_mouza has quit IRC | 16:50 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Create and grant all keystone roles in service-ks-register https://review.opendev.org/683403 | 16:51 |
| *** BjoernT has joined #openstack-kolla | 16:52 | |
| noxoid | mgoddard, https://blueprints.launchpad.net/kolla-ansible/+spec/expose-prometheus-server-external | 16:53 |
| noxoid | ill probably still test the merge behavior as there are some containers id like to "turn off" as they dont apply to us or are not needed in a multi-region deployment | 16:54 |
| openstackgerrit | Michael Turek proposed openstack/kolla master: ppc64le: Ensure that rabbitmq-server.repo exists before trying sed https://review.opendev.org/683997 | 16:56 |
| mgoddard | noxoid: again, if you think it could be generally useful then we might accept a patch | 16:58 |
| openstackgerrit | Michael Turek proposed openstack/kolla master: ppc64le: Ensure that rabbitmq-server.repo exists before trying sed https://review.opendev.org/683997 | 16:59 |
| noxoid | yep | 16:59 |
| mjturek | thanks for the help mgoddard | 16:59 |
| mgoddard | mjturek: np, thanks for the patch :) | 17:00 |
| *** BjoernT has quit IRC | 17:04 | |
| *** BjoernT has joined #openstack-kolla | 17:07 | |
| *** k_mouza has joined #openstack-kolla | 17:08 | |
| *** BjoernT has quit IRC | 17:10 | |
| *** watersj has joined #openstack-kolla | 17:15 | |
| watersj | How would you go about adding an addtional nic and network to a kolla stein install? | 17:16 |
| *** BjoernT has joined #openstack-kolla | 17:18 | |
| cally725 | Hi I am using kolla rocky and I have the process beam.smp that takes 220% after restarting docker service it is now down to 140% still high but the cluster is now accessible | 17:19 |
| noxoid | watersj, what network are you looking to move over to the additional nic? | 17:20 |
| watersj | for example I have office net and cluster net, I want to add hpc-network so I can dual home some instance | 17:22 |
| noxoid | oh so you want to add another provider network to neutron? | 17:23 |
| watersj | normally I would use a f/w or router/gw but they want dual homed instance. Also this allow me to spin up instances that have access to hpc network too | 17:23 |
| watersj | yes | 17:23 |
| watersj | I have 3 nodes, wasn't sure how you get bridge to bind to "hpc" nic | 17:23 |
| noxoid | https://docs.openstack.org/neutron/stein/configuration/openvswitch-agent.html#ovs.bridge_mappings | 17:26 |
| noxoid | https://opendev.org/openstack/kolla-ansible/src/branch/stable/stein/ansible/roles/neutron/templates/ml2_conf.ini.j2#L89 | 17:27 |
| watersj | ty, | 17:27 |
| noxoid | i havent done specifically what you are doing but i have manually set up neutron before. should just need to add the new mapping there | 17:28 |
| noxoid | and then you will need to add that new mapping to whatever ml2 type you are trying to use it with (flat/vlan/etc) | 17:28 |
| watersj | I using linuxbridge at moment | 17:28 |
| noxoid | ah yea youll want to find the equivalent option for linuxbridge | 17:28 |
| watersj | will do | 17:28 |
| noxoid | just check where physnet1 is in /etc/kolla/neutron and it should guide you | 17:28 |
| noxoid | hopefully that helps | 17:28 |
| *** BjoernT has quit IRC | 17:30 | |
| watersj | looking at that file earlier on each of nodes, of course each nodes interface name is different | 17:30 |
| watersj | saw in documentation where it said to name nics with bonding? i think | 17:30 |
| watersj | i need go back and check | 17:31 |
| noxoid | yea i use bonds for mine | 17:31 |
| noxoid | my provider networks are all named bond0 | 17:31 |
| noxoid | makes it easier :P | 17:31 |
| watersj | 1 nic or multiples? | 17:31 |
| noxoid | multiples | 17:31 |
| noxoid | but you can still use 1 nic in a bond iirc | 17:31 |
| noxoid | if you just want the consistent naming | 17:31 |
| noxoid | also makes it easier if you're using LACP to add additional capacity down the road | 17:32 |
| watersj | ok, my new compute nodes came in w/plenty nics. This should be fun | 17:32 |
| kplant | what noxoid said | 17:33 |
| kplant | hide the annoying nic names with a little ansible and jinja and present easy names like "bond0" | 17:33 |
| openstackgerrit | Viktor Michalek proposed openstack/kolla master: Debian Buster Stein backports repository https://review.opendev.org/683674 | 17:38 |
| *** BjoernT has joined #openstack-kolla | 17:39 | |
| *** jonaspaulo has joined #openstack-kolla | 17:39 | |
| openstackgerrit | Viktor Michalek proposed openstack/kolla master: Debian Buster Stein backports repository https://review.opendev.org/683674 | 17:41 |
| *** BjoernT_ has joined #openstack-kolla | 17:43 | |
| *** BjoernT has quit IRC | 17:45 | |
| *** rouk has quit IRC | 17:50 | |
| *** igordc has joined #openstack-kolla | 18:06 | |
| *** luksky has joined #openstack-kolla | 18:06 | |
| watersj | i'm guessing I should put my completed ml2_conf.ini into deployers /etc/kolla/config/neutron/neutron-linuxbridge-agent. next time I run deply it doesn't kill my configs | 18:08 |
| *** goldyfruit___ has quit IRC | 18:11 | |
| *** k_mouza has quit IRC | 18:13 | |
| *** watersj has quit IRC | 18:32 | |
| openstackgerrit | Dincer Celik proposed openstack/kolla-ansible master: [prometheus] Added documentation for deployment and setting extra options https://review.opendev.org/684090 | 18:34 |
| *** k_mouza has joined #openstack-kolla | 18:40 | |
| *** goldyfruit___ has joined #openstack-kolla | 18:41 | |
| *** igordc has quit IRC | 18:51 | |
| openstackgerrit | Dincer Celik proposed openstack/kolla-ansible master: [prometheus] Added documentation for deployment and setting extra options https://review.opendev.org/684090 | 18:57 |
| *** goldyfruit___ has quit IRC | 19:01 | |
| openstackgerrit | Viktor Michalek proposed openstack/kolla master: Fluentd: td-agent repository for Debian https://review.opendev.org/684119 | 19:03 |
| *** k_mouza has quit IRC | 19:07 | |
| *** k_mouza has joined #openstack-kolla | 19:11 | |
| openstackgerrit | Dincer Celik proposed openstack/kolla-ansible master: [prometheus] Added documentation for deployment and setting extra options https://review.opendev.org/684090 | 19:16 |
| *** mgoddard has quit IRC | 19:17 | |
| openstackgerrit | Viktor Michalek proposed openstack/kolla master: Fluentd: td-agent repository for Debian https://review.opendev.org/684119 | 19:18 |
| openstackgerrit | Dincer Celik proposed openstack/kolla-ansible master: [prometheus] Added documentation for deployment and setting extra options https://review.opendev.org/684090 | 19:19 |
| *** mgoddard has joined #openstack-kolla | 19:20 | |
| *** k_mouza has quit IRC | 19:21 | |
| openstackgerrit | Dincer Celik proposed openstack/kolla-ansible master: [prometheus] Added documentation for deployment and setting extra options https://review.opendev.org/684090 | 19:22 |
| *** goldyfruit has joined #openstack-kolla | 19:23 | |
| *** k_mouza has joined #openstack-kolla | 19:38 | |
| *** k_mouza has quit IRC | 19:40 | |
| *** pcaruana has joined #openstack-kolla | 19:43 | |
| openstackgerrit | James Kirsch proposed openstack/kolla-ansible master: Add support for self-signed SSL certificates https://review.opendev.org/664517 | 19:49 |
| *** dave-mccowan has joined #openstack-kolla | 19:50 | |
| *** kplant has quit IRC | 19:50 | |
| *** pcaruana has quit IRC | 20:12 | |
| *** vmixor has quit IRC | 20:16 | |
| *** ktibi has joined #openstack-kolla | 20:48 | |
| *** ktibi has quit IRC | 20:51 | |
| *** cally725 has quit IRC | 21:06 | |
| *** igordc has joined #openstack-kolla | 21:27 | |
| *** goldyfruit has quit IRC | 21:32 | |
| *** BjoernT_ has quit IRC | 21:34 | |
| *** goldyfruit has joined #openstack-kolla | 21:43 | |
| *** jonaspaulo has quit IRC | 22:07 | |
| openstackgerrit | Merged openstack/kolla stable/rocky: Fix retries when pushing images https://review.opendev.org/683399 | 22:12 |
| openstackgerrit | James Kirsch proposed openstack/kolla-ansible master: Add support for self-signed SSL certificates https://review.opendev.org/664517 | 22:39 |
| *** ivve has quit IRC | 22:40 | |
| *** k_mouza has joined #openstack-kolla | 23:09 | |
| *** k_mouza has quit IRC | 23:13 | |
| *** cah_link1 has joined #openstack-kolla | 23:17 | |
| *** cah_link has quit IRC | 23:19 | |
| *** cah_link1 is now known as cah_link | 23:19 | |
| *** luksky has quit IRC | 23:44 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!