| *** hwoarang has quit IRC | 00:04 | |
| openstackgerrit | Merged openstack/openstack-ansible stable/rocky: Mark (and use) version with the OSA wrapper script https://review.openstack.org/637366 | 00:04 |
|---|---|---|
| *** hwoarang has joined #openstack-ansible | 00:05 | |
| *** macza has quit IRC | 00:06 | |
| *** markvoelker has joined #openstack-ansible | 00:19 | |
| *** tosky has quit IRC | 00:19 | |
| *** cmart has quit IRC | 00:55 | |
| *** logan- has quit IRC | 01:22 | |
| *** logan- has joined #openstack-ansible | 01:23 | |
| NobodyCam | is it possible to setup OSA-aio on a remote host? | 01:28 |
| *** logan- has quit IRC | 01:31 | |
| *** logan- has joined #openstack-ansible | 01:31 | |
| *** mmercer has quit IRC | 01:36 | |
| *** DanyC has quit IRC | 01:39 | |
| jamesdenton | what do you mean? | 01:43 |
| *** sdake has quit IRC | 01:44 | |
| jamesdenton | You can use the AIO plays to setup openstack on other hosts in addition to the AIO node (i.e. using the AIO node as a controller/compute/etc and have additional computes), but it won't handle any of the other networking/bootstrapping; that's only done on the AIO node itself | 01:45 |
| *** hwoarang has quit IRC | 01:47 | |
| *** sdake has joined #openstack-ansible | 01:47 | |
| *** hwoarang has joined #openstack-ansible | 01:48 | |
| *** hwoarang has quit IRC | 01:55 | |
| *** hwoarang has joined #openstack-ansible | 01:56 | |
| cloudnull | lbragstad I cant say I've ever seen that issue | 02:03 |
| cloudnull | however our key rotation policy may not be as aggressive? | 02:04 |
| *** gyee has quit IRC | 02:06 | |
| lbragstad | yeah - possibly | 02:06 |
| cloudnull | is this user using something like a 1 min rotation ? | 02:06 |
| lbragstad | rotations every hour | 02:06 |
| lbragstad | token expiration every 2 hours | 02:06 |
| lbragstad | 3 max keys | 02:06 |
| cloudnull | hum | 02:08 |
| lbragstad | pas-ha apparently hit the issue using an internal openstack deployment they use for ci/cd, which just churns out vms | 02:08 |
| cloudnull | our default policy is daily rotation w/ 7 keys | 02:08 |
| cloudnull | I guess if you were hammering the API during a rotation event we'd see that same issue | 02:09 |
| lbragstad | right | 02:09 |
| cloudnull | no matter the event timing | 02:09 |
| lbragstad | it has to get routed to a host thats in the middle of an rsync operation, too | 02:10 |
| lbragstad | like - exactly in the middle | 02:10 |
| lbragstad | or anytime the new staged key is sync'd, but before the primary key is sync'd | 02:10 |
| cloudnull | seems like this would be best solved client side. like 401, retry | 02:11 |
| lbragstad | yeah | 02:11 |
| lbragstad | a subsequent would work | 02:11 |
| lbragstad | subsequent request* | 02:11 |
| cloudnull | however, maybe they're seeing 10 million 401s all spike up then back to normal work | 02:11 |
| cloudnull | and then they're seeing this every hour | 02:11 |
| lbragstad | otherwise - you'd have to reverse the order in which you copy keys based on the key inde | 02:11 |
| lbragstad | index* | 02:11 |
| lbragstad | if you copy the key with the highest index first, the problem should go away | 02:12 |
| lbragstad | (but we leave that implementation detail to rsync) | 02:12 |
| cloudnull | I wonder if we could do a list of all the files in the dir and then reverse the order in the command? | 02:14 |
| lbragstad | and do a manual copy of each file? | 02:15 |
| cloudnull | something rsync $flags 7 6 5 4 3 2 1 $user@$server:$path | 02:15 |
| lbragstad | or still use rsync somehow? | 02:15 |
| lbragstad | oh - sure | 02:15 |
| cloudnull | IDK if that would preserve the ordering but that would work ? | 02:15 |
| cloudnull | **should work | 02:15 |
| *** DanyC has joined #openstack-ansible | 02:16 | |
| lbragstad | https://pasted.tech/pastes/ec9618b32aa49b7bbca9c4f32a014c83995d79da.raw | 02:18 |
| cloudnull | so if we specify the file ordering and set the flag `--delay-updates` it should go ? | 02:19 |
| lbragstad | sounds like two different solutions | 02:19 |
| lbragstad | one is to explicitly call out which files you want transferred first and use individual rsync commands | 02:20 |
| cloudnull | I guess we could use scp instead of rsync | 02:20 |
| lbragstad | the second is to use --delay-updates | 02:20 |
| cloudnull | that would keep the order | 02:20 |
| lbragstad | --delay-updates sounds like it just minimizes the window of susceptibility | 02:20 |
| lbragstad | you could theoretically still hit the problem, just less likely | 02:21 |
| cloudnull | scp would be slower, and technically just copying the files one at a time, but could be done in a given order. | 02:21 |
| lbragstad | right | 02:21 |
| lbragstad | key distribution isn't time sensitive anyway | 02:21 |
| lbragstad | we don't require that you rush | 02:21 |
| cloudnull | ok. so we could just change https://github.com/openstack/openstack-ansible-os_keystone/blob/master/templates/keystone-fernet-rotate.sh.j2#L34 to use scp and https://github.com/openstack/openstack-ansible-os_keystone/blob/master/templates/keystone-fernet-rotate.sh.j2#L37 to be a list of the files in reverse order | 02:23 |
| cloudnull | and the problem should be solved? | 02:23 |
| lbragstad | i think so? | 02:23 |
| lbragstad | fwiw - i haven't verified this locally | 02:23 |
| cloudnull | so this is the normal sort order | 02:30 |
| cloudnull | https://pasted.tech/pastes/f87e7c2eedc586ea5725e4ec05a5636f9d3c746a | 02:30 |
| cloudnull | and we want it based on the time it was created https://pasted.tech/pastes/fcd8c0f781bc03f864481335733bc1f5411fe4b5 | 02:30 |
| *** cmart has joined #openstack-ansible | 02:44 | |
| *** sdake has quit IRC | 02:48 | |
| *** sdake has joined #openstack-ansible | 02:49 | |
| *** sdake has quit IRC | 02:57 | |
| *** sdake has joined #openstack-ansible | 02:59 | |
| lbragstad | so - if that's the node your rotating from | 03:05 |
| lbragstad | then i think it has to be 0 $highest-index | 03:05 |
| lbragstad | actually - $highest-index should be first (i think) | 03:06 |
| lbragstad | because the key with the highest index is the key encrypting tokens | 03:06 |
| lbragstad | all other keys, including the 0 key, are secondary keys that can only be used to decrypt tokens | 03:07 |
| lbragstad | the only difference between a non-zero secondary key and the 0 key is that the 0 key hasn't had the opportunity to encrypt anything, yet | 03:07 |
| lbragstad | so - that should mean if you transfer the key with the highest index first, you should be able to validate tokens that were *just* encrypted with that key | 03:08 |
| lbragstad | and if you're using scp - you shouldn't be wiping the entire key repository, so other secondary keys should still be available if an older token comes in for validation halfway through the rotation | 03:09 |
| *** alvinstarr has quit IRC | 03:16 | |
| *** markvoelker has quit IRC | 03:20 | |
| cloudnull | sorry was making dinner, back | 03:26 |
| *** spsurya has joined #openstack-ansible | 03:34 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-os_keystone master: Correct fernet token sync race condition https://review.openstack.org/638327 | 03:48 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-os_keystone master: Correct fernet token sync race condition https://review.openstack.org/638327 | 03:49 |
| cloudnull | lbragstad ^ | 03:49 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-os_keystone master: Correct fernet token sync race condition https://review.openstack.org/638327 | 03:51 |
| *** udesale has joined #openstack-ansible | 03:51 | |
| *** hwoarang has quit IRC | 04:12 | |
| *** hwoarang has joined #openstack-ansible | 04:14 | |
| *** markvoelker has joined #openstack-ansible | 04:21 | |
| lbragstad | cloudnull nice - thanks! | 04:25 |
| *** ianychoi has quit IRC | 04:31 | |
| *** jbadiapa has quit IRC | 04:38 | |
| *** jbadiapa has joined #openstack-ansible | 04:38 | |
| *** cmart has quit IRC | 04:44 | |
| *** vnogin has joined #openstack-ansible | 04:48 | |
| *** sdake has quit IRC | 04:50 | |
| *** vnogin has quit IRC | 04:52 | |
| *** sdake has joined #openstack-ansible | 04:54 | |
| *** markvoelker has quit IRC | 04:55 | |
| *** ArchiFleKs has quit IRC | 04:56 | |
| *** cmart has joined #openstack-ansible | 04:59 | |
| *** ArchiFleKs has joined #openstack-ansible | 05:13 | |
| *** sdake has quit IRC | 05:14 | |
| *** sdake has joined #openstack-ansible | 05:15 | |
| *** hwoarang has quit IRC | 05:16 | |
| *** hwoarang has joined #openstack-ansible | 05:18 | |
| *** shyamb has joined #openstack-ansible | 05:19 | |
| *** hwoarang has quit IRC | 05:39 | |
| *** hwoarang has joined #openstack-ansible | 05:41 | |
| *** sdake has quit IRC | 05:48 | |
| *** sdake has joined #openstack-ansible | 05:51 | |
| *** markvoelker has joined #openstack-ansible | 05:52 | |
| *** gokhani has joined #openstack-ansible | 05:53 | |
| *** lbragstad_ has joined #openstack-ansible | 05:53 | |
| *** lbragstad has quit IRC | 05:55 | |
| *** cmart has quit IRC | 05:56 | |
| *** kmrchdn is now known as chandankumar | 06:01 | |
| *** sdake has quit IRC | 06:03 | |
| *** lbragstad has joined #openstack-ansible | 06:04 | |
| *** sdake has joined #openstack-ansible | 06:04 | |
| *** lbragstad_ has quit IRC | 06:04 | |
| *** lbragstad_ has joined #openstack-ansible | 06:09 | |
| *** lbragstad has quit IRC | 06:10 | |
| *** lbragstad has joined #openstack-ansible | 06:16 | |
| *** lbragstad_ has quit IRC | 06:18 | |
| *** markvoelker has quit IRC | 06:25 | |
| *** spsurya has quit IRC | 06:25 | |
| *** sdake has quit IRC | 06:37 | |
| *** shyamb has quit IRC | 06:55 | |
| *** jorti_ has quit IRC | 07:03 | |
| *** jorti has joined #openstack-ansible | 07:05 | |
| *** shyamb has joined #openstack-ansible | 07:06 | |
| fnpanic | hi | 07:15 |
| *** spsurya has joined #openstack-ansible | 07:17 | |
| *** kopecmartin|off is now known as kopecmartin | 07:19 | |
| *** pcaruana has joined #openstack-ansible | 07:19 | |
| *** markvoelker has joined #openstack-ansible | 07:22 | |
| *** cshen has joined #openstack-ansible | 07:27 | |
| *** DanyC has quit IRC | 07:35 | |
| *** DanyC has joined #openstack-ansible | 07:36 | |
| *** shyamb has quit IRC | 07:37 | |
| *** cshen has quit IRC | 07:41 | |
| *** markvoelker has quit IRC | 07:55 | |
| *** cshen has joined #openstack-ansible | 08:08 | |
| *** osackz has quit IRC | 08:08 | |
| *** phasespace has joined #openstack-ansible | 08:11 | |
| *** cshen has quit IRC | 08:22 | |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-os_tempest master: use run_stackviz to generate stackviz report https://review.openstack.org/638360 | 08:22 |
| *** lbragstad has quit IRC | 08:25 | |
| *** tosky has joined #openstack-ansible | 08:26 | |
| *** ivve has joined #openstack-ansible | 08:27 | |
| *** gillesMo has joined #openstack-ansible | 08:34 | |
| *** fghaas has joined #openstack-ansible | 08:36 | |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-os_tempest master: Added tempest.conf for heat_plugin https://review.openstack.org/632021 | 08:39 |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-os_tempest master: Use the correct heat tests https://review.openstack.org/630695 | 08:40 |
| *** DanyC has quit IRC | 08:41 | |
| CeeMac | morning | 08:47 |
| odyssey4me | prometheanfire there is instrumentation in the lxc-container-create role to allow a different distro than the host, ut the osa integrated build does not use it | 08:48 |
| odyssey4me | prometheanfire yes, OSA uses the infra afs mirrors | 08:49 |
| odyssey4me | cjloader cloudnull d34dh0r53 the ping package was added to the wrong list: https://review.openstack.org/#/c/638189/4/vars/redhat-7.yml | 08:49 |
| *** cshen has joined #openstack-ansible | 08:50 | |
| odyssey4me | prometheanfire to use the infra mirrors, though, we override the defaults in the roles via extra-vars in the integrated build, and via set_facts in the tests repo playbooks (for role tests) | 08:52 |
| *** markvoelker has joined #openstack-ansible | 08:52 | |
| *** cshen has quit IRC | 08:55 | |
| *** shyamb has joined #openstack-ansible | 08:58 | |
| *** gillesMo has quit IRC | 09:03 | |
| odyssey4me | evrardjp heh, no need for https://review.openstack.org/637361 / https://review.openstack.org/637359 / https://review.openstack.org/637363 any more, because the use of the env var has merged :) | 09:05 |
| *** electrofelix has joined #openstack-ansible | 09:08 | |
| evrardjp | haha :) | 09:09 |
| evrardjp | good | 09:09 |
| evrardjp | I need to do something on master, because master must look very weird | 09:09 |
| evrardjp | I have to discuss it with mnaser | 09:09 |
| evrardjp | but cool :) | 09:09 |
| *** cshen has joined #openstack-ansible | 09:22 | |
| *** markvoelker has quit IRC | 09:25 | |
| chandankumar | odyssey4me: evrardjp I need some help on this review https://review.openstack.org/#/c/632726/ I am facing this error http://logs.openstack.org/26/632726/12/check/tripleo-ci-centos-7-standalone-os-tempest/d22a454/job-output.txt.gz#_2019-02-19_09_46_00_505465 | 09:32 |
| chandankumar | odyssey4me: evrardjp since it is a action plugin, if I set dependencies under meta/main.yaml it always assumes as a role | 09:33 |
| chandankumar | but I need to set it as a action plugin | 09:33 |
| chandankumar | is it possible to do that? | 09:33 |
| openstackgerrit | Merged openstack/ansible-config_template master: Add multistropt test cases https://review.openstack.org/636603 | 09:35 |
| openstackgerrit | Merged openstack/ansible-config_template master: Remove whitespace before comments https://review.openstack.org/636935 | 09:35 |
| *** shyamb has quit IRC | 09:41 | |
| *** shyamb has joined #openstack-ansible | 09:50 | |
| *** iurygregory has quit IRC | 10:01 | |
| *** iurygregory has joined #openstack-ansible | 10:01 | |
| *** phasespace has quit IRC | 10:09 | |
| *** markvoelker has joined #openstack-ansible | 10:17 | |
| *** spsurya has quit IRC | 10:22 | |
| *** DanyC has joined #openstack-ansible | 10:25 | |
| openstackgerrit | Chandan Kumar proposed openstack/ansible-config_template master: Fixed config_template setup.cfg to treat as a role https://review.openstack.org/638383 | 10:26 |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-os_tempest master: Added dependency of os_tempest role https://review.openstack.org/632726 | 10:27 |
| *** phasespace has joined #openstack-ansible | 10:33 | |
| chandankumar | odyssey4me: jrosser https://review.openstack.org/#/c/638360/ stackviz enable disable | 10:34 |
| *** ChosSimbaOne_Lap has quit IRC | 10:34 | |
| *** DanyC has quit IRC | 10:38 | |
| *** DanyC has joined #openstack-ansible | 10:39 | |
| *** ChosSimbaOne_Lap has joined #openstack-ansible | 10:41 | |
| *** Chosimba1 has joined #openstack-ansible | 10:42 | |
| *** Chosimba1 has quit IRC | 10:44 | |
| *** ChosSimbaOne_Lap has quit IRC | 10:45 | |
| *** ChosSimbaOne has joined #openstack-ansible | 10:45 | |
| *** shyamb has quit IRC | 10:46 | |
| ChosSimbaOne | Hi. I am currently diving into OpenStack-Ansible for setting up our openstack installation. I am installing on top on Ubuntu 18.04 which uses netplan for network setup. All the documentaion relates to the old ifupdown way of doing network in ubuntu. What is recommended with openstack-ansible for ubutnu 18.04? Replace netplan with ifupdown or is there any documentation on setting up with netplan? | 10:48 |
| chandankumar | odyssey4me: I think if i uncomment this line https://git.openstack.org/cgit/openstack/ansible-role-python_venv_build/tree/defaults/main.yml#n22 then I can reuse it as a dependeny in os_tempest | 10:49 |
| *** DanyC has quit IRC | 10:50 | |
| chandankumar | anyway this var getting overwritten so I donot think it will a problem | 10:50 |
| chandankumar | odyssey4me: what you say? | 10:50 |
| *** ChosSimbaOne has quit IRC | 10:52 | |
| *** ChosSimbaOne has joined #openstack-ansible | 10:52 | |
| openstackgerrit | Chandan Kumar proposed openstack/ansible-role-python_venv_build master: Uncomment venv_install_destination_path for using as a role https://review.openstack.org/638393 | 10:54 |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-os_tempest master: Added dependency of os_tempest role https://review.openstack.org/632726 | 10:55 |
| *** stuartgr has joined #openstack-ansible | 10:59 | |
| *** shyamb has joined #openstack-ansible | 11:15 | |
| *** udesale has quit IRC | 11:26 | |
| *** ArchiFleKs has quit IRC | 11:26 | |
| *** ArchiFleKs has joined #openstack-ansible | 11:31 | |
| *** fghaas has quit IRC | 11:39 | |
| *** fghaas has joined #openstack-ansible | 11:54 | |
| *** sdake has joined #openstack-ansible | 12:02 | |
| *** cshen has quit IRC | 12:07 | |
| phasespace | I'm testing openstack ansible upgrades. After the upgrade i see stuff like this in the ceilometer logs: Invalid input: extra keys not allowed @ data[u'flavor_name']. Googling this I see people saying this will be fixed by running ceilometer-upgrade to feed gnocchi with the new resource types. Is that so? Will I break anything by running this? | 12:08 |
| chandankumar | odyssey4me: I think my venv changes broke it | 12:13 |
| jamesdenton | mornin' folks | 12:13 |
| chandankumar | the tripleo jobs | 12:13 |
| CeeMac | morning jamesdenton | 12:14 |
| CeeMac | just the man i was looking for actually :) | 12:15 |
| jamesdenton | alrighty, let's do this | 12:15 |
| CeeMac | remember the other day I was saying about issues on my network, well the plot thickens | 12:16 |
| gundalow | Anyone able to take a look at https://github.com/ansible/ansible/pull/52699 | 12:16 |
| CeeMac | lost network connectivity to one of the nodes early morning around 6/7am (different times on different nodes) | 12:16 |
| CeeMac | and I've been combing through the logs and found this | 12:16 |
| CeeMac | http://paste.openstack.org/show/745596/ | 12:16 |
| CeeMac | any idea whats going on there? | 12:17 |
| jamesdenton | one sec | 12:18 |
| CeeMac | sure | 12:18 |
| CeeMac | sorry to jump on you first thing | 12:18 |
| jamesdenton | that's alright. You're on Ubuntu? With netplan, right? | 12:19 |
| CeeMac | yep | 12:19 |
| CeeMac | seems to occur frequently | 12:19 |
| CeeMac | but only affects some of the nodes | 12:19 |
| jamesdenton | And i'm assuming you're referring to the stop/start? | 12:20 |
| CeeMac | the only way to get network back is to bounce a switch port. The strange thing is, disabling either switchport afterwards doesn't effect network at all. So maybe some arp aging needs factoring in | 12:20 |
| CeeMac | yeah, after the systemd running in system mode | 12:20 |
| CeeMac | kernal log just shows the ports going down then back up (and the various bonds etc) | 12:21 |
| CeeMac | no idea what is causing systemd to do this? | 12:22 |
| jamesdenton | anything in dmseg around the same time? | 12:22 |
| CeeMac | let me check | 12:23 |
| CeeMac | no dmesg | 12:23 |
| CeeMac | hmm | 12:25 |
| *** kmadac has joined #openstack-ansible | 12:25 | |
| CeeMac | dpkg upgrade ran maybe? | 12:25 |
| CeeMac | http://paste.openstack.org/show/745599/ | 12:25 |
| CeeMac | pretty sure only kernel security patches should be getting installed automatically | 12:25 |
| jamesdenton | definitely looks like a new version of systemd (237) might've been installed. | 12:26 |
| *** cshen has joined #openstack-ansible | 12:27 | |
| jamesdenton | If you run something like, netplan apply or systemctl restart systemd-networkd do you lose connectivity? | 12:27 |
| CeeMac | lets find out :) | 12:28 |
| CeeMac | hmmm | 12:28 |
| CeeMac | not that time | 12:28 |
| CeeMac | let me try netplan apply | 12:29 |
| CeeMac | nope | 12:29 |
| CeeMac | long ping reply but no drop | 12:29 |
| jamesdenton | k | 12:29 |
| jamesdenton | how about systemctl daemon-reexec | 12:29 |
| CeeMac | nope | 12:30 |
| CeeMac | that created the systemd running in system mode messages in syslog | 12:30 |
| jamesdenton | If you have prior timestamps, can you check those to see if they correlate to the same event? or a systemd restart ot upgrade? | 12:30 |
| CeeMac | but not the network stop/start | 12:30 |
| jamesdenton | ok | 12:30 |
| CeeMac | when i was losing connectivity during playbook install it was always around systemd upgrade / change | 12:31 |
| CeeMac | let me see how far the logs go back on this server | 12:32 |
| jamesdenton | can you share your netplan config files? | 12:32 |
| CeeMac | sure | 12:32 |
| CeeMac | http://paste.openstack.org/show/745601/ | 12:33 |
| jamesdenton | thanks | 12:34 |
| CeeMac | np | 12:34 |
| jamesdenton | The playbook install you mentioned.. did you reliably experience loss of connectivity during OSA install or something? | 12:34 |
| CeeMac | no issue during the intial install | 12:35 |
| CeeMac | but when I was running minor upgrade to 18.1.3 i lost connectivity to some of the nodes during both host-setup and inf-setup | 12:35 |
| CeeMac | always during the lxc-hosts install apt task | 12:36 |
| CeeMac | which does target systemd | 12:36 |
| jamesdenton | ok, where it might run an 'apt update' or something? | 12:36 |
| jamesdenton | or upgrade, rather | 12:37 |
| CeeMac | yep | 12:37 |
| CeeMac | well, it installs a specific set of packages from a variable iirc | 12:37 |
| CeeMac | and presumably would upgrade them if required | 12:37 |
| jamesdenton | Well, i think any system package is targeted, too, with this: apt-get upgrade -y -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" --force-yes | 12:39 |
| CeeMac | I'll trust you on that :) | 12:39 |
| jamesdenton | well, don't. haha | 12:39 |
| CeeMac | i didnt get quite that far down the rabbit hole :) | 12:39 |
| jamesdenton | as far as your switch config goes, you don't have any special bonding configuration on there, right? Just configured as normal trunk interfaces? | 12:40 |
| CeeMac | just normal trunk interface yeah | 12:40 |
| CeeMac | they're independant switches | 12:40 |
| CeeMac | no cross links, stacks, virtual-chassis or anything | 12:41 |
| jamesdenton | k | 12:41 |
| CeeMac | although they're both in the same blade chassis | 12:41 |
| *** shyamb has quit IRC | 12:41 | |
| jamesdenton | the blade. forgot about the blade. | 12:41 |
| *** ansmith has quit IRC | 12:43 | |
| *** kaiokmo has quit IRC | 12:44 | |
| CeeMac | the other weird thing is keepalived periodically loses the vip and I need to restart the service | 12:45 |
| jamesdenton | as in, the vip disappears from the interface? | 12:45 |
| CeeMac | haproxy blade is set with flat networks though, no bond, can't see anything in logs so far to account fir this | 12:45 |
| CeeMac | yeah | 12:45 |
| CeeMac | but the keepalived and haproxy services are both up | 12:46 |
| jamesdenton | you have three infras? or only the one | 12:46 |
| CeeMac | haprxy is on tin | 12:46 |
| CeeMac | and there is only 1 right now | 12:46 |
| CeeMac | so if network is failing maybe keepalives is "failing over" to nothing, need to look into that more | 12:46 |
| CeeMac | my next job is to double up the tin for each role to make everything fully HA, wanted to get the minor upgrade out of the way first | 12:47 |
| CeeMac | then ran into this weird network issue | 12:47 |
| jamesdenton | so, you might consider removing the 'primary' parameter on the bond | 12:48 |
| jamesdenton | Are you able to successfully test failover manually? Shutting down interfaces, yanking cables, etc | 12:48 |
| CeeMac | i tested shutting down the interface this morning and it seemed ok, i'll test again | 12:49 |
| CeeMac | can't pull cables as theyre hooked in via midplane to blade switches | 12:49 |
| CeeMac | brb | 12:49 |
| jamesdenton | when this happens, do you have an OOB interface you come in on, or use a virtual console or something? | 12:50 |
| jamesdenton | right ok | 12:50 |
| chandankumar | odyssey4me: please have a look at this issue http://logs.openstack.org/26/632726/14/check/tripleo-ci-centos-7-standalone-os-tempest/a4c5081/job-output.txt.gz#_2019-02-21_12_07_39_451133 | 12:53 |
| *** nicolasbock has joined #openstack-ansible | 12:57 | |
| CeeMac | back | 12:58 |
| CeeMac | yeah, i have iLO / virtual console access | 12:58 |
| jamesdenton | CeeMac if/when this happens again, it would be worth doing a tcpdump on the two interfaces of the bond to see if there's any traffic making it | 12:59 |
| jamesdenton | arp, unicast, whatever | 12:59 |
| CeeMac | i only tend to catch it retrospectively unfortunately | 13:00 |
| CeeMac | 6am seems to be a trigger on some of the servers | 13:00 |
| jamesdenton | so networking is already back up? | 13:00 |
| CeeMac | oh, i see what you mean | 13:00 |
| CeeMac | no, it doesn't come back until i bounce the ports | 13:00 |
| *** kaiokmo has joined #openstack-ansible | 13:00 | |
| jamesdenton | k, yeah, before you bounce | 13:00 |
| CeeMac | good plan | 13:01 |
| CeeMac | ok, so i've shut down each interface in turn, waited 10 secs then unshut it | 13:12 |
| CeeMac | no ping drop from server | 13:12 |
| CeeMac | so it must be a software issue? | 13:12 |
| jamesdenton | *shrug* | 13:12 |
| jamesdenton | this is where it gets fun | 13:12 |
| CeeMac | haha | 13:13 |
| CeeMac | yeah | 13:13 |
| CeeMac | ok, i'll ignore it until it goes off overnight again, then i'll take some pcaps | 13:13 |
| *** Mr_Smurf has joined #openstack-ansible | 13:14 | |
| Mr_Smurf | Rocky + Xenial.. supported or not? | 13:14 |
| CeeMac | do you think its worth removing the primary? | 13:14 |
| CeeMac | on the bond? | 13:14 |
| jamesdenton | CeeMac Up to you - you can leave it and do the caps next time it happens, or remove it and see if the issue persists | 13:15 |
| CeeMac | i'll wait then, in the interest of scientific research | 13:15 |
| CeeMac | just going to check the other nodes that have bonds, only 2 went down, but maybe only 2 got updated | 13:16 |
| jamesdenton | Mr_Smurf I believe so? | 13:16 |
| jamesdenton | Mr_Smurf This seems to imply it is: https://docs.openstack.org/project-deploy-guide/openstack-ansible/rocky/targethosts.html | 13:17 |
| *** gillesMo has joined #openstack-ansible | 13:17 | |
| *** markvoelker has quit IRC | 13:17 | |
| CeeMac | pretty sure i was running rocky on xenial on my last dev setup | 13:18 |
| *** markvoelker has joined #openstack-ansible | 13:18 | |
| gillesMo | Hello ! Is it possible to mix Ubuntu releases for compute nodes with OSA 18.x ? I have a Queens deployment, managed with OSA 17.x, I will soon upgrade to OSA 18.x / Rocky but still on Ubuntu 16.04. As I will need more compute nodes, I wonder if I could deploy them on Ubuntu 18.04 ? | 13:20 |
| *** gillesMo has quit IRC | 13:21 | |
| *** gillesMo has joined #openstack-ansible | 13:21 | |
| Mr_Smurf | jamesdenton: ok, thanks.. | 13:21 |
| gillesMo | Hello ! Is it possible to mix Ubuntu releases for compute nodes with OSA 18.x ? I have a Queens deployment, managed with OSA 17.x, I will soon upgrade to OSA 18.x / Rocky but still on Ubuntu 16.04. As I will need more compute nodes, I wonder if I could deploy them on Ubuntu 18.04 ? | 13:22 |
| CeeMac | hmmm, looks like unattended upgrades *is* enabled. I thought it was just for kernel security | 13:22 |
| *** markvoelker has quit IRC | 13:22 | |
| *** sdake has quit IRC | 13:23 | |
| *** sdake has joined #openstack-ansible | 13:24 | |
| jamesdenton | chandankumar i think because you're doing a distro install, neither the venv_pip_packages nor venv_default_pip_packages lists are being populated. From what I'm finding, the related tasks only get triggered on a source install. So there could be a bug with distro builds? | 13:26 |
| jamesdenton | chandankumar https://github.com/openstack/tripleo-quickstart-extras/blob/master/playbooks/multinode-standalone.yml#L57 | 13:26 |
| chandankumar | jamesdenton: I think it is related to stackviz installation in venv | 13:26 |
| chandankumar | jamesdenton: I am working on a patch to move stackviz on rpm | 13:27 |
| chandankumar | jamesdenton: may be this one https://github.com/openstack/openstack-ansible-os_tempest/blob/master/tasks/tempest_install.yml#L32 | 13:27 |
| chandankumar | but I am not sure odyssey4me will like python_venv_fix | 13:27 |
| jamesdenton | Well this is where it's failing: | 13:28 |
| jamesdenton | https://github.com/openstack/ansible-role-python_venv_build/blob/master/tasks/python_venv_install.yml#L76-L78 | 13:28 |
| jamesdenton | and based on the error, i bet both of those lists are empty (default to empty) because they only get populated on a source install | 13:28 |
| *** shyamb has joined #openstack-ansible | 13:35 | |
| *** sdake has quit IRC | 13:36 | |
| *** sdake has joined #openstack-ansible | 13:38 | |
| *** shyamb has quit IRC | 13:44 | |
| *** fghaas has quit IRC | 13:53 | |
| *** ansmith has joined #openstack-ansible | 13:55 | |
| *** lbragstad has joined #openstack-ansible | 13:57 | |
| *** hamzaachi has joined #openstack-ansible | 13:59 | |
| *** fghaas has joined #openstack-ansible | 14:08 | |
| *** sdake has quit IRC | 14:14 | |
| *** vnogin has joined #openstack-ansible | 14:16 | |
| *** markvoelker has joined #openstack-ansible | 14:18 | |
| *** phasespace has quit IRC | 14:21 | |
| *** udesale has joined #openstack-ansible | 14:22 | |
| *** vnogin has quit IRC | 14:30 | |
| *** sdake has joined #openstack-ansible | 14:30 | |
| CeeMac | oh, jamesdenton i just remembered i had another question | 14:33 |
| jamesdenton | sure | 14:33 |
| CeeMac | see when you're using a cinder volume driver to a backend SAN using iSCSI | 14:34 |
| CeeMac | does cinder still maintain a separate front-end iscsi connection to nova for passing volume claims to instances | 14:34 |
| CeeMac | or does it expect the iSCSI backend to be on the br-storage network? | 14:34 |
| *** vnogin has joined #openstack-ansible | 14:37 | |
| jamesdenton | that's a good question, and one i do not know the answer to offhand | 14:37 |
| CeeMac | hmm | 14:37 |
| CeeMac | guess I'll just try it and see then :) | 14:37 |
| jamesdenton | :) | 14:41 |
| *** dave-mccowan has joined #openstack-ansible | 14:48 | |
| mnaser | hi everyone | 14:50 |
| jamesdenton | howdy | 14:51 |
| *** sdake has quit IRC | 14:51 | |
| kaiokmo | heyllo | 14:52 |
| *** markvoelker has quit IRC | 14:53 | |
| fghaas | CeeMac: so far as I know the contract for Cinder is that it'll always directly connect instances (VMs) to the storage backend. cinder-volume isn't normally expected to operate as a proxy of sorts, it's just that with the iSCSI/LVM backend there's no other way to expose the volumes other than cinder-volume doing it by itself. The general expectation is that your compute node will be able to connect to, depending on your choice | 14:53 |
| fghaas | of Cinder backend, the iSCSI storage network, the fibre channel network, the Ceph cluster public network. For IP-based protocols like iSCSI and Ceph it's not required that the backend is _bridged_ into the storage network, but you definitely need at least a routed connection between the compute nodes and the iSCSI backend. Does that help? | 14:53 |
| CeeMac | hi fghaas, thats great thanks | 14:54 |
| CeeMac | i'm going to be running nova and cinder on the same host so hopefully this will match the criteria | 14:55 |
| CeeMac | nova and cinder-volume that is | 14:55 |
| fghaas | what nova though? compute or the api container? | 14:58 |
| CeeMac | compute | 14:58 |
| fghaas | yeah in your case that's not really necessary. If you have an iSCSI-based backend that you're managing with say the 3par or netapp drivers, you can run cinder-volume in a container on your control nodes | 15:00 |
| fghaas | because that's only ever used for provisioning | 15:00 |
| mnaser | fghaas: ^ i've learned the hardway that's not ideal | 15:00 |
| mnaser | some things like create volume from image with certain drivers can result in it trying to mount things via iscsi inside the container | 15:01 |
| mnaser | and that breaks in weird ways | 15:01 |
| fghaas | mnaser: elaborate? You mean because you have to wire up your control nodes to be able to connect to your storage nodes, and for your compute nodes you're presumably already doing that? | 15:01 |
| fghaas | If so, that's a fair point | 15:01 |
| fghaas | Oh! And the stupid netlink issue | 15:02 |
| mnaser | fghaas: cinder-volume process tries to create a volume from image, so it tries to bind/attach it into the container via iscsi to dd the image into it and yeah | 15:02 |
| *** sdake has joined #openstack-ansible | 15:02 | |
| fghaas | You're right. Forget what I said CeeMac, for any iSCSI backend never run cinder-volume in a container | 15:02 |
| *** cshen has quit IRC | 15:02 | |
| fghaas | So, what you'd planned is perfectly reasonable. | 15:03 |
| fghaas | (sorry for forgetting that one, I usually run with Ceph all the way and there that's a no-issue) | 15:03 |
| CeeMac | i think i just about followed that | 15:05 |
| mnaser | yeah ceph is fine | 15:05 |
| CeeMac | to clarify, I've got the host in compute_hosts, metering_compute_hosts and storage_hosts | 15:05 |
| mnaser | because ceph is great | 15:05 |
| openstackgerrit | Shannon Mitchell proposed openstack/openstack-ansible-os_tempest stable/queens: Update workspace tempest.conf on changes https://review.openstack.org/638032 | 15:06 |
| CeeMac | and i'm using the zadara iscsi backend | 15:06 |
| openstackgerrit | Shannon Mitchell proposed openstack/openstack-ansible-os_tempest stable/pike: Update workspace tempest.conf on changes https://review.openstack.org/638036 | 15:06 |
| CeeMac | so it sounds like I'll be ok? | 15:06 |
| *** ArchiFleKs has quit IRC | 15:07 | |
| *** shananigans has joined #openstack-ansible | 15:10 | |
| *** ArchiFleKs has joined #openstack-ansible | 15:11 | |
| *** vnogin has quit IRC | 15:12 | |
| *** dave-mccowan has quit IRC | 15:21 | |
| *** cshen has joined #openstack-ansible | 15:30 | |
| *** dxiri has joined #openstack-ansible | 15:32 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-os_keystone master: Correct fernet token sync race condition https://review.openstack.org/638327 | 15:37 |
| NobodyCam | jamesdenton: Thank you for the reply. I am attempting to setup a single aio node just not from local host | 15:37 |
| cjloader | odyssey4me: do we need to revert? | 15:40 |
| *** dave-mccowan has joined #openstack-ansible | 15:41 | |
| cloudnull | cjloader I'd say just move the package to the noted list | 15:45 |
| *** dave-mccowan has quit IRC | 15:45 | |
| openstackgerrit | Cam J. Loader (cjloader) proposed openstack/openstack-ansible-os_tempest master: Fix redhat iputtils https://review.openstack.org/638444 | 15:46 |
| cjloader | cloudnull: d34dh0r53 odyssey4me ^ | 15:46 |
| *** sdake has quit IRC | 15:48 | |
| *** markvoelker has joined #openstack-ansible | 15:50 | |
| *** sdake has joined #openstack-ansible | 15:50 | |
| *** tosky has quit IRC | 15:53 | |
| *** tosky has joined #openstack-ansible | 15:54 | |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-os_tempest master: use tempest_run_stackviz to generate stackviz report https://review.openstack.org/638360 | 16:06 |
| *** udesale has quit IRC | 16:11 | |
| *** hamzy has quit IRC | 16:15 | |
| spotz | cjloader: you're good | 16:16 |
| cjloader | spotz: ty | 16:17 |
| spotz | NP | 16:18 |
| CeeMac | quick question, host_vars don't seem to be taking under storage_hosts in openstack_user_config.ym; | 16:19 |
| CeeMac | is there a way to debug that? | 16:19 |
| *** cshen has quit IRC | 16:23 | |
| *** DanyC has joined #openstack-ansible | 16:23 | |
| *** markvoelker has quit IRC | 16:23 | |
| *** cmart has joined #openstack-ansible | 16:24 | |
| *** macza has joined #openstack-ansible | 16:24 | |
| chandankumar | odyssey4me: jrosser when you are free feel to take at these series https://review.openstack.org/#/q/topic:os_tempest_deps+(status:open+OR+status:merged) | 16:25 |
| *** defionscode has quit IRC | 16:30 | |
| *** kopecmartin is now known as kopecmartin|off | 16:33 | |
| *** vnogin has joined #openstack-ansible | 16:35 | |
| openstackgerrit | Shannon Mitchell proposed openstack/openstack-ansible-os_tempest master: Update workspace tempest.conf on changes https://review.openstack.org/638014 | 16:36 |
| *** DanyC has quit IRC | 16:38 | |
| *** DanyC has joined #openstack-ansible | 16:39 | |
| *** hamzy has joined #openstack-ansible | 16:40 | |
| *** dave-mccowan has joined #openstack-ansible | 16:47 | |
| *** defionscode has joined #openstack-ansible | 16:47 | |
| *** gyee has joined #openstack-ansible | 16:48 | |
| *** defionscode has quit IRC | 16:51 | |
| *** defionscode has joined #openstack-ansible | 16:53 | |
| cloudnull | CeeMac are you not seeing the var at all, or just not seeing the value your setting ? | 16:56 |
| cloudnull | like maybe there's a conflicting var? | 16:56 |
| CeeMac | none of the settings are being pushed out in the config drop task | 16:56 |
| CeeMac | its like its not being parsed at all | 16:57 |
| CeeMac | i've had a quick look in cinder.conf and there aren't any existing settings that would block it that I can see | 16:57 |
| CeeMac | tried running with -vvv but couldn't see any reference to it pulling in config overrides other than the default ones | 16:58 |
| cloudnull | mind sharing the stanza in the user config? | 16:58 |
| CeeMac | sure | 16:58 |
| CeeMac | http://paste.openstack.org/show/745634/ | 16:59 |
| CeeMac | its the second host in particular | 16:59 |
| CeeMac | pulling config settings from here: https://docs.openstack.org/cinder/rocky/configuration/block-storage/drivers/zadara-volume-driver.html | 17:00 |
| cloudnull | CeeMac and I e3-211-rccn02v is a baremetal volume server ? | 17:01 |
| chandankumar | cloudnull: Hello | 17:01 |
| CeeMac | yes | 17:01 |
| CeeMac | and compute node | 17:01 |
| cloudnull | ok. | 17:01 |
| cloudnull | is E3213V01P01 supposed to be a section in config? | 17:01 |
| chandankumar | cloudnull: you mean https://review.openstack.org/#/c/638393/1/defaults/main.yml > /openstack/venvs/myvenv -> /openstack/venvs/undefined or I missing somehting? | 17:01 |
| *** vnogin has quit IRC | 17:02 | |
| cloudnull | chandankumar yes :) | 17:02 |
| cloudnull | its just a cosmetic nit | 17:02 |
| CeeMac | as i understand it, if i was to have multiple backends I'd need to have unique names | 17:02 |
| openstackgerrit | Chandan Kumar proposed openstack/ansible-role-python_venv_build master: Uncomment venv_install_destination_path for using as a role https://review.openstack.org/638393 | 17:03 |
| CeeMac | the sample used vpsa, I've tried swapping out E3213V01P01 for vpsa as per the example | 17:03 |
| CeeMac | didnt make a difference | 17:03 |
| chandankumar | cloudnull: by the way I love your pets pics on twitter, keep posting :-) | 17:03 |
| cloudnull | CeeMac: I think this would work https://pasted.tech/pastes/879afb6a4b6653e0411e72dd0a2b7e5fd311acc4 | 17:04 |
| CeeMac | lemme give that a go | 17:04 |
| cloudnull | chandankumar ha! thanks! Rest assured, there will be more soon :) | 17:05 |
| CeeMac | does it treat cinder-volume as container even though its metal? | 17:05 |
| cloudnull | no. sadly the key "container_vars" is a throw back to a time long since past. container_vars will apply given variables to all things under a given host, containers or not. | 17:06 |
| CeeMac | ah | 17:07 |
| *** pcaruana has quit IRC | 17:07 | |
| CeeMac | thats not clear in the documentation :) | 17:07 |
| cloudnull | host_vars was added later, which makes a lot more sense, but adds variables to only items that are noted as "physical_hosts" | 17:07 |
| cloudnull | so if the config needs to hit all of the above, container_vars are the way to go | 17:07 |
| CeeMac | which is why the example in the docs uses the nova conf ? | 17:07 |
| chandankumar | CeeMac: :-) | 17:08 |
| chandankumar | cloudnull: :-) | 17:08 |
| cloudnull | the problem is that a lot of these inventory things were written back in Juno | 17:08 |
| cloudnull | and the term container was being used as the recipient of inventory things. | 17:09 |
| CeeMac | right | 17:09 |
| cloudnull | not a fancy cgroup :) | 17:09 |
| cloudnull | so... i guess this is a long way to say, sorry. | 17:10 |
| CeeMac | haha | 17:10 |
| cloudnull | :) | 17:10 |
| CeeMac | no worries | 17:10 |
| CeeMac | there are always foibles in a system, its just a matter of knowing what they are! | 17:10 |
| CeeMac | which is why I keep pestering you lovely people ;) | 17:10 |
| cloudnull | indeed! | 17:10 |
| cloudnull | pester away! | 17:10 |
| * cloudnull enjoys a good pestering | 17:11 | |
| CeeMac | :D | 17:11 |
| cloudnull | did that updated stanza work ? | 17:12 |
| CeeMac | it did! | 17:12 |
| cloudnull | sweet | 17:12 |
| CeeMac | except for the DEFAULT | 17:12 |
| CeeMac | which didnt take | 17:12 |
| cloudnull | hum . | 17:12 |
| CeeMac | wonder if that is conflicting with the glance iscsi settings | 17:12 |
| * cloudnull looking | 17:12 | |
| CeeMac | wonder if i could set that under the backend config? | 17:13 |
| CeeMac | iscsi_protocol isn't listed in cinder.conf though | 17:13 |
| CeeMac | hmmm | 17:13 |
| cloudnull | it should inject the config into the cinder.conf | 17:14 |
| cloudnull | is cinder_cinder_conf_overrides defined elsewhere? | 17:14 |
| CeeMac | nope | 17:15 |
| cloudnull | maybe in a user_*.yml file ? | 17:15 |
| cloudnull | ok | 17:15 |
| CeeMac | the backend has been set | 17:15 |
| CeeMac | and the enabled_backends | 17:15 |
| CeeMac | just not the iscsi_protocol line | 17:15 |
| CeeMac | just seems to have issues with cinder_cinder_conf_overrides | 17:15 |
| CeeMac | wonder if there is a default override variable for the iscsi protocol | 17:16 |
| * CeeMac goes to look | 17:16 | |
| cloudnull | is cinder_cinder_conf_overrides in your inventory anywhere `openstack_inventory.json`? | 17:17 |
| CeeMac | let me check | 17:18 |
| cloudnull | I assume you're running the playbook with a limit of just that one host? | 17:18 |
| cloudnull | any tags? | 17:18 |
| CeeMac | just the host limit, no tags | 17:18 |
| cloudnull | ok | 17:18 |
| * cloudnull goes to try it | 17:18 | |
| CeeMac | os-cinder-install.yml | 17:19 |
| CeeMac | it is in the inventory | 17:20 |
| *** markvoelker has joined #openstack-ansible | 17:20 | |
| CeeMac | http://paste.openstack.org/show/745640/ | 17:20 |
| openstackgerrit | Merged openstack/openstack-ansible-os_keystone master: Add keystone_user_pip_packages variable https://review.openstack.org/638233 | 17:26 |
| CeeMac | mkay | 17:26 |
| CeeMac | making progress, but not quite there yet. Gonna shoot off and revisit with fresh eyes in the morning | 17:26 |
| CeeMac | thanks for the help as usual chaps | 17:26 |
| cloudnull | take care! | 17:27 |
| CeeMac | ttfn! | 17:31 |
| *** dxiri has quit IRC | 17:35 | |
| openstackgerrit | Jacob Wagner proposed openstack/openstack-ansible-ops master: Add ability to deploy designate with BIND9 servers https://review.openstack.org/635611 | 17:35 |
| *** DanyC has quit IRC | 17:42 | |
| *** shardy has quit IRC | 17:49 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_horizon master: Add horizon_user_pip_packages variable https://review.openstack.org/638239 | 17:51 |
| *** markvoelker has quit IRC | 17:53 | |
| *** gillesMo has quit IRC | 18:00 | |
| *** sdake has quit IRC | 18:05 | |
| *** sdake has joined #openstack-ansible | 18:06 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_heat master: Add heat_user_pip_packages variable https://review.openstack.org/638230 | 18:08 |
| *** vnogin has joined #openstack-ansible | 18:15 | |
| *** vnogin has quit IRC | 18:21 | |
| *** cmart has quit IRC | 18:23 | |
| *** cshen has joined #openstack-ansible | 18:30 | |
| *** cshen has quit IRC | 18:34 | |
| partlycloudy | hi folks, i tried to make a deployment using ovs-dvr, but only got br-int created on the target hosts. | 18:36 |
| *** electrofelix has quit IRC | 18:37 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_cinder stable/queens: cinder.conf: add [nova] section, override interface defaults https://review.openstack.org/638206 | 18:38 |
| openstackgerrit | Merged openstack/openstack-ansible-os_cinder stable/rocky: cinder.conf: add [nova] section, override interface defaults https://review.openstack.org/638205 | 18:38 |
| partlycloudy | OSA tag 18.1.4. here are exerpts from openstack_user_config.yml (https://pasted.tech/pastes/38b55873a7a6912a269) and user_variables.yml (https://pasted.tech/pastes/72f71f9f9db2acb7df499daf14393cff1730e570). is anything that i missed? | 18:39 |
| partlycloudy | sorry for the first broken link. here it is: openstack_user_config.yml (https://pasted.tech/pastes/38b55873a7a6912a26972ff58ace655d97da3314) | 18:40 |
| jamesdenton | chandankumar we you able to get past that pip issue? | 18:46 |
| jamesdenton | *were | 18:46 |
| *** markvoelker has joined #openstack-ansible | 18:50 | |
| jamesdenton | partlycloudy do you still have the deployment logs handy? Are you able to see the result of the 'Setup Network Provider Bridges' task? | 18:55 |
| *** vollman has quit IRC | 18:56 | |
| jamesdenton | actually, it may show up as 'Setup External Network Provider Bridge' | 18:56 |
| jamesdenton | oh n/m, i see it | 18:56 |
| jamesdenton | https://github.com/openstack/openstack-ansible-os_neutron/blob/stable/rocky/tasks/providers/ovs_config.yml#L24 | 18:57 |
| jamesdenton | for Rocky, it would only be setup on non-DVR deploy. So you can just set it up by hand with 'ovs-vsctl add-br br-provider' and restart the agent | 18:57 |
| partlycloudy | jamesdenton: thanks james, as always. what about br-tun? do i need to create that as well? | 18:59 |
| jamesdenton | it's possible it will show up after you fix this. i bet the agents are not loading fully | 18:59 |
| jamesdenton | probably erroring out because the bridge does not exist | 18:59 |
| partlycloudy | jamesdenton: yup. i saw the exact error msg, complaining about non-existing br-provider | 19:00 |
| jamesdenton | yeah, you'll need to add it then. It's "fixed" in master | 19:00 |
| jamesdenton | the docs you followed, if 'latest' then those correspond to master moreso than Rocky | 19:01 |
| partlycloudy | i tried master branch last time, but got some errors with ceph (asking about —allow-downgrade etc…), so i switched back to v18. | 19:02 |
| jamesdenton | i noticed you defined provider networks in openstack_user_config.yml and overrides in user_variables.yml. Ideally, it would be one or the other. There were changes in master (Stein) that should allow you to forgo the overrides in user_variables.yml and set it all up in the other file (preferred?). | 19:03 |
| jamesdenton | for Rocky, though, you are probably fine with what you have, even though there's some redundancy there | 19:03 |
| partlycloudy | i see. the redundant part in user_variables.yml was added after the first unsuccessful run. i thought that may be the problem. i will remove it later. :-) | 19:05 |
| *** cmart has joined #openstack-ansible | 19:17 | |
| *** mmercer has joined #openstack-ansible | 19:20 | |
| *** markvoelker has quit IRC | 19:23 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_tempest master: Fix redhat iputtils https://review.openstack.org/638444 | 19:28 |
| *** alvinstarr has joined #openstack-ansible | 19:35 | |
| partlycloudy | jamesdenton: i'm redeploying the whole thing now (the previous build was destroyed just before you came to the rescue) Will let you know how it goes after the manual fix. cheers! | 19:38 |
| *** ArchiFleKs has quit IRC | 19:39 | |
| *** strattao has joined #openstack-ansible | 19:44 | |
| *** fghaas has quit IRC | 19:46 | |
| *** spatel has joined #openstack-ansible | 19:48 | |
| spatel | Good afternoon folks! | 19:49 |
| cloudnull | o/ spatel | 19:49 |
| strattao | Hello all! | 19:49 |
| strattao | I was just looking at the link for the upcoming PTG in Devner, and it still looks a little… sparse. I’m slated to go to the conference and am staying for the PTGs, so I am definitely looking forward to putting some faces to some names. | 19:49 |
| spatel | my current cloud going to touch 250 compute nodes.. so should i continue adding compute nodes? | 19:50 |
| spatel | cloudnull: or jamesdenton ^^ | 19:50 |
| cloudnull | 300-350 is the largest I would go, as a rule. | 19:50 |
| jamesdenton | 351 | 19:50 |
| cloudnull | that said, 500+ does work you just need to scale the infra to meet the demand | 19:50 |
| cloudnull | dedicated network nodes, rabbitmq, etc | 19:51 |
| cloudnull | strattao awesome! I hope that I will be able to attend and see you there. IDK if there's been any PTG planning for denver quite yet. -cc odyssey4me mnaser evrardjp | 19:52 |
| strattao | Good timing on this topic for me! Are there any metrics for how the infra needs to be scaled? Where things break down at scale? | 19:52 |
| strattao | We’ve got some big systems we’re just starting to roll out and would like to get ahead of some of the bottlenecks we’ll be facing | 19:52 |
| spatel | How do i measure that everything looks good at infra | 19:52 |
| strattao | * seconds spatel’s question | 19:53 |
| spatel | like tipping point or rates i should watch etc... on infra nodes? | 19:53 |
| jamesdenton | simultaneous restart of all agents and services and see how long it takes to fall apart </troll> | 19:53 |
| *** ArchiFleKs has joined #openstack-ansible | 19:54 | |
| spatel | jamesdenton: This is what my rabbitMQ looking https://ibb.co/bbLQdRm | 19:55 |
| spatel | This is only infra-02 stats | 19:55 |
| spatel | its pretty much same on all three nodes | 19:55 |
| *** hamzaachi has quit IRC | 19:56 | |
| strattao | cloudnull, I haven’t been to the conference before or the PTGs… do you guys typically meet all day long for all three days? What is the schedule typically like? | 19:56 |
| jamesdenton | spatel i don't really have a point of reference, unfortunately. this is an area i don't focus on much | 19:56 |
| spatel | MySQL stats - https://ibb.co/6DGBDjr | 19:57 |
| spatel | cloudnull: how to scale infra ? ( could you explain what you trying to say ) | 19:58 |
| spatel | oh!! dedicated service node :) | 19:58 |
| spatel | strattao: you and me on same page :) | 19:58 |
| spatel | jamesdenton: other option i have to build new openstack cloud and maintain two cloud :( | 20:00 |
| *** DanyC has joined #openstack-ansible | 20:03 | |
| jamesdenton | at some point, though, you will need to do that | 20:03 |
| cloudnull | strattao: for the PTG its an all day thing, however, some folks participate in multiple projects so they come and go as needed | 20:04 |
| cloudnull | spatel ++ dedicated service nodes. | 20:05 |
| cloudnull | you can begin exploring cells however, +1 to what jamesdenton said, eventually you will need to begin thinking about RegionTwo | 20:05 |
| spatel | how do i isolate services in production ? | 20:06 |
| strattao | So, is the basic approach, just whenever a service gets bogged down, spin it out onto it’s own dedicated hardward and scale up from there then? | 20:06 |
| cloudnull | That's been my approach. | 20:06 |
| cloudnull | I'll spend a good amount of time tuning but if the issue is contention then the next thing to do is to spin it out onto other gear | 20:07 |
| strattao | but you haven’t been involved with anything that’s approaching 500+ node clouds, right? | 20:07 |
| strattao | or anything bigger like 1000+ ;) | 20:08 |
| cloudnull | 500 yes. >500 in a single region, no, not really. | 20:08 |
| *** maxbab has joined #openstack-ansible | 20:08 | |
| strattao | cool, just curious. thx | 20:08 |
| cloudnull | we generally opt for multiple regions at that scale | 20:09 |
| cloudnull | its more to manage, but limits the blast radius | 20:09 |
| cloudnull | with cells v2 we might be able to a lot more but that largely hasn't been explored, at least not by me | 20:10 |
| cloudnull | I know cern is doing a lot with cellsv2 these days | 20:10 |
| cloudnull | Tim Bell is amazing - so we might be able to learn quite a bit from them | 20:11 |
| *** DanyC has quit IRC | 20:15 | |
| *** maxbab has quit IRC | 20:17 | |
| *** maxbab has joined #openstack-ansible | 20:19 | |
| *** maxbab has quit IRC | 20:19 | |
| *** fghaas has joined #openstack-ansible | 20:20 | |
| *** markvoelker has joined #openstack-ansible | 20:21 | |
| admin0 | what is the variable to set if horizon image upload does not work ? | 20:28 |
| admin0 | i know there was some workaround/setting to set this to remote/something | 20:28 |
| *** cshen has joined #openstack-ansible | 20:30 | |
| *** cshen has quit IRC | 20:34 | |
| *** cmart has quit IRC | 20:37 | |
| *** macza has quit IRC | 20:45 | |
| *** macza has joined #openstack-ansible | 20:45 | |
| *** hamzaachi has joined #openstack-ansible | 20:51 | |
| *** markvoelker has quit IRC | 20:55 | |
| *** hamzaachi has quit IRC | 20:55 | |
| *** DanyC has joined #openstack-ansible | 21:08 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-os_heat master: Do not install heat service distro packages for source installs https://review.openstack.org/638511 | 21:10 |
| jrosser | cloudnull: ^ lets see how that goes | 21:11 |
| *** macza has quit IRC | 21:19 | |
| *** spatel has quit IRC | 21:20 | |
| cloudnull | ++ | 21:21 |
| * cloudnull wonders if that has been contributing to our heat issues? | 21:21 | |
| *** kmadac has quit IRC | 21:26 | |
| *** ansmith has quit IRC | 21:36 | |
| cloudnull | pulled that patch and after purge / redeploy - https://pasted.tech/pastes/1b23368f41efaf5be41a9dce6ddec322593d41a7 - things are looking a lot better. | 21:37 |
| cloudnull | how i purged https://pasted.tech/pastes/3e00db6b17effdcb9487158cef274fd56f8235ee | 21:37 |
| *** sm806 has quit IRC | 21:41 | |
| *** sm806 has joined #openstack-ansible | 21:41 | |
| cloudnull | it doesn't look like we have any other service bleeding distro packages into source, https://pasted.tech/pastes/f9b2b26d8e0879baf402e42c5f11c1589f91f71e, but it'd be good to get another set of eyes looking at the same | 21:42 |
| *** markvoelker has joined #openstack-ansible | 21:53 | |
| *** macza has joined #openstack-ansible | 21:53 | |
| *** macza has quit IRC | 21:55 | |
| *** macza has joined #openstack-ansible | 21:55 | |
| *** fghaas has quit IRC | 21:56 | |
| *** cmart has joined #openstack-ansible | 21:59 | |
| *** shananigans has quit IRC | 22:07 | |
| *** ansmith has joined #openstack-ansible | 22:11 | |
| *** fghaas has joined #openstack-ansible | 22:17 | |
| *** hamzy has quit IRC | 22:21 | |
| *** markvoelker has quit IRC | 22:25 | |
| *** sdake has quit IRC | 22:27 | |
| *** cshen has joined #openstack-ansible | 22:30 | |
| *** cshen has quit IRC | 22:35 | |
| *** dave-mccowan has quit IRC | 22:40 | |
| *** dave-mccowan has joined #openstack-ansible | 22:42 | |
| *** dave-mccowan has quit IRC | 22:46 | |
| *** fghaas has quit IRC | 22:49 | |
| openstackgerrit | Jacob Wagner proposed openstack/openstack-ansible-ops master: Add ability to deploy designate with BIND9 servers https://review.openstack.org/635611 | 22:56 |
| NobodyCam | good afternoon OSA folks | 22:59 |
| NobodyCam | could someone point me to any doc about setting up flat networking, is that even possible? | 23:02 |
| *** sdake has joined #openstack-ansible | 23:11 | |
| *** tosky has quit IRC | 23:11 | |
| cloudnull | any cores around want to give https://review.openstack.org/638511 push though | 23:12 |
| cloudnull | we need to backport that with a quickness. | 23:12 |
| cloudnull | NobodyCam: ye sflat networking is totally possible | 23:13 |
| cloudnull | we do that in the gate | 23:13 |
| NobodyCam | oh sweet | 23:14 |
| NobodyCam | happen to have anything thing you could point me to? | 23:14 |
| cloudnull | heres the doc that covers the overview - https://docs.openstack.org/project-deploy-guide/openstack-ansible/newton/app-networking.html#network-appendix | 23:15 |
| NobodyCam | I'm a little list going thru the official doc's | 23:15 |
| cloudnull | in practice we just create the flat network stanza in the user config and ensure there's an ethernet device to attach to | 23:15 |
| * cloudnull getting a couple snippets | 23:15 | |
| NobodyCam | awesome TY cloudnull :) | 23:16 |
| cloudnull | https://github.com/openstack/openstack-ansible/blob/master/etc/openstack_deploy/openstack_user_config.yml.aio.j2#L86-L94 - that'd go in the openstack_user_config | 23:16 |
| cloudnull | note eth12 in that snippet needs to exist on your machine. | 23:17 |
| cloudnull | you can change that to whatever you want however it needs to be an ethernet device | 23:17 |
| cloudnull | if you dont have an ethernet device you want to use with a flat network you can hang a vethpair off one of the bridges and name it eth11 | 23:18 |
| cloudnull | 's/eth11/eth12/' | 23:18 |
| cloudnull | here's an example on creating a veth pair and hanging it off a bridge | 23:19 |
| cloudnull | https://github.com/openstack/openstack-ansible/blob/master/etc/network/interfaces.d/aio_interfaces.cfg#L52-L59 | 23:19 |
| NobodyCam | :) awesome. I'll give it a shot | 23:19 |
| cloudnull | in that example https://github.com/openstack/openstack-ansible/blob/master/etc/network/interfaces.d/aio_interfaces.cfg#L53 creates the "ethernet" device named eth12 | 23:19 |
| cloudnull | which is used here https://github.com/openstack/openstack-ansible/blob/master/etc/openstack_deploy/openstack_user_config.yml.aio.j2#L89-L90 | 23:19 |
| cloudnull | with the one stanza in config rerun the neutron playbook and you should be off to the races with a new flat network type | 23:20 |
| *** markvoelker has joined #openstack-ansible | 23:22 | |
| *** ivve has quit IRC | 23:25 | |
| *** sdake has quit IRC | 23:33 | |
| *** sdake has joined #openstack-ansible | 23:37 | |
| *** gyee has quit IRC | 23:38 | |
| *** sdake has quit IRC | 23:46 | |
| *** sdake has joined #openstack-ansible | 23:47 | |
| *** sdake has quit IRC | 23:51 | |
| *** gyee has joined #openstack-ansible | 23:52 | |
| *** sdake_ has joined #openstack-ansible | 23:52 | |
| *** strattao has quit IRC | 23:54 | |
| *** markvoelker has quit IRC | 23:55 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!