| *** rajinir has quit IRC | 00:04 | |
| *** evgenyl has quit IRC | 00:04 | |
| *** JpMaxMan has quit IRC | 00:06 | |
| *** rajinir has joined #openstack-infra | 00:07 | |
| *** JpMaxMan has joined #openstack-infra | 00:08 | |
| *** evgenyl has joined #openstack-infra | 00:08 | |
| fungi | gitea02 eventually finished, 04 still underway | 00:14 |
|---|---|---|
| *** jamesmcarthur has joined #openstack-infra | 00:22 | |
| *** jamesmcarthur has quit IRC | 00:35 | |
| *** jamesmcarthur has joined #openstack-infra | 00:39 | |
| *** jamesmcarthur has quit IRC | 00:43 | |
| *** jamesmcarthur_ has joined #openstack-infra | 00:44 | |
| *** yamamoto has quit IRC | 01:04 | |
| *** yamamoto has joined #openstack-infra | 01:25 | |
| *** jamesmcarthur_ has quit IRC | 01:38 | |
| *** jamesmcarthur has joined #openstack-infra | 01:41 | |
| *** jamesmcarthur has quit IRC | 01:46 | |
| *** yamamoto has quit IRC | 01:47 | |
| *** yamamoto has joined #openstack-infra | 01:48 | |
| *** jamesmcarthur has joined #openstack-infra | 02:09 | |
| *** yamamoto has quit IRC | 02:44 | |
| *** jamesmcarthur has quit IRC | 02:55 | |
| *** yamamoto has joined #openstack-infra | 03:17 | |
| *** ricolin has joined #openstack-infra | 03:18 | |
| *** jamesmcarthur has joined #openstack-infra | 03:22 | |
| *** yamamoto has quit IRC | 03:28 | |
| *** yamamoto has joined #openstack-infra | 03:28 | |
| *** bhavikdbavishi has joined #openstack-infra | 03:46 | |
| *** jamesmcarthur has quit IRC | 03:56 | |
| *** yamamoto has quit IRC | 04:26 | |
| *** bhavikdbavishi has quit IRC | 04:32 | |
| *** diablo_rojo has joined #openstack-infra | 04:34 | |
| *** jamesmcarthur has joined #openstack-infra | 04:53 | |
| *** jamesmcarthur has quit IRC | 04:57 | |
| *** jamesmcarthur has joined #openstack-infra | 05:03 | |
| *** yamamoto has joined #openstack-infra | 05:06 | |
| *** diablo_rojo has quit IRC | 05:10 | |
| *** yamamoto has quit IRC | 05:20 | |
| *** factor has joined #openstack-infra | 05:40 | |
| *** whoami-rajat has joined #openstack-infra | 05:40 | |
| *** Lucas_Gray has joined #openstack-infra | 06:01 | |
| *** jamesmcarthur has quit IRC | 06:33 | |
| *** yamamoto has joined #openstack-infra | 06:49 | |
| *** shachar has quit IRC | 07:00 | |
| *** shachar has joined #openstack-infra | 07:01 | |
| *** Lucas_Gray has quit IRC | 07:07 | |
| *** Lucas_Gray has joined #openstack-infra | 07:10 | |
| *** slaweq has joined #openstack-infra | 07:38 | |
| *** ricolin_ has joined #openstack-infra | 07:41 | |
| *** ricolin has quit IRC | 07:44 | |
| *** Lucas_Gray has quit IRC | 08:11 | |
| *** yamamoto has quit IRC | 08:19 | |
| *** bhavikdbavishi has joined #openstack-infra | 08:19 | |
| *** bhavikdbavishi has quit IRC | 08:48 | |
| *** ykarel|away has joined #openstack-infra | 08:49 | |
| *** yamamoto has joined #openstack-infra | 08:51 | |
| *** xek has joined #openstack-infra | 08:53 | |
| *** slaweq has quit IRC | 08:54 | |
| *** slaweq has joined #openstack-infra | 08:59 | |
| *** yamamoto has quit IRC | 09:04 | |
| *** slaweq has quit IRC | 09:09 | |
| *** trident has quit IRC | 09:15 | |
| *** trident has joined #openstack-infra | 09:17 | |
| *** ykarel|away has quit IRC | 09:19 | |
| *** whoami-rajat has quit IRC | 10:49 | |
| *** yamamoto has joined #openstack-infra | 11:36 | |
| *** yamamoto has quit IRC | 11:44 | |
| *** yamamoto has joined #openstack-infra | 11:48 | |
| *** ykarel|away has joined #openstack-infra | 11:50 | |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Swap gitea04 into service and bring down gitea05 https://review.opendev.org/673169 | 12:17 |
| *** whoami-rajat has joined #openstack-infra | 12:19 | |
| *** raissa has joined #openstack-infra | 12:27 | |
| *** Anticimex has quit IRC | 12:32 | |
| *** Anticimex has joined #openstack-infra | 12:33 | |
| *** raissa has quit IRC | 12:36 | |
| *** raissa has joined #openstack-infra | 12:36 | |
| *** tosky has joined #openstack-infra | 12:41 | |
| *** raissa has quit IRC | 12:50 | |
| *** sshnaidm|off is now known as sshnaidm | 13:07 | |
| *** yamamoto has quit IRC | 13:16 | |
| *** dciabrin_ has joined #openstack-infra | 13:31 | |
| *** dciabrin has quit IRC | 13:34 | |
| *** yamamoto has joined #openstack-infra | 13:51 | |
| *** yamamoto has quit IRC | 13:52 | |
| *** Goneri has joined #openstack-infra | 13:53 | |
| *** yamamoto has joined #openstack-infra | 13:54 | |
| *** Goneri has quit IRC | 14:01 | |
| *** pkopec has joined #openstack-infra | 14:14 | |
| *** bhavikdbavishi has joined #openstack-infra | 14:21 | |
| *** dciabrin_ has quit IRC | 14:30 | |
| *** whoami-rajat has quit IRC | 14:39 | |
| *** rh-jelabarre has quit IRC | 14:49 | |
| *** yamamoto has quit IRC | 15:02 | |
| *** yamamoto has joined #openstack-infra | 15:03 | |
| *** yamamoto has quit IRC | 15:04 | |
| *** pkopec has quit IRC | 15:23 | |
| *** bhavikdbavishi has quit IRC | 15:23 | |
| *** yamamoto has joined #openstack-infra | 15:36 | |
| *** ykarel|away has quit IRC | 15:42 | |
| *** ykarel|away has joined #openstack-infra | 15:43 | |
| *** yamamoto has quit IRC | 15:56 | |
| *** goldyfruit has joined #openstack-infra | 16:05 | |
| *** dciabrin_ has joined #openstack-infra | 16:07 | |
| *** dciabrin__ has joined #openstack-infra | 16:09 | |
| *** dciabrin_ has quit IRC | 16:12 | |
| *** jamesdenton has quit IRC | 16:13 | |
| *** e0ne has joined #openstack-infra | 16:15 | |
| *** hamzy has quit IRC | 16:18 | |
| *** bobh has joined #openstack-infra | 16:22 | |
| *** Vadmacs has joined #openstack-infra | 16:51 | |
| *** altlogbot_3 has quit IRC | 16:57 | |
| *** altlogbot_1 has joined #openstack-infra | 16:58 | |
| donnyd | fungi: I think i figured out what the deal is | 17:06 |
| donnyd | when the mirror was originally built, it wasn' | 17:06 |
| donnyd | wasn't dualstack | 17:06 |
| donnyd | it was just v4 | 17:06 |
| *** ykarel|away has quit IRC | 17:06 | |
| donnyd | v6 was added on after the fact | 17:06 |
| fungi | ahh, so what's the solution there? | 17:14 |
| fungi | rejigger the neutron network for that tenant? | 17:14 |
| *** ykarel|away has joined #openstack-infra | 17:20 | |
| *** jamesmcarthur has joined #openstack-infra | 17:22 | |
| *** bobh has quit IRC | 17:24 | |
| donnyd | yea, i just need to pull off the v6 network and then put it back on | 17:24 |
| donnyd | so if your script will automatically request a fip, you should be good to go now | 17:27 |
| donnyd | subnet for v4 is public_v4 | 17:28 |
| donnyd | im sorry, network is public_v4, subnet is public_v4_subnet | 17:29 |
| donnyd | be back later tonight | 17:29 |
| *** AJaeger is now known as AJaeger_ | 17:33 | |
| *** Lucas_Gray has joined #openstack-infra | 17:33 | |
| *** e0ne has quit IRC | 17:37 | |
| *** trident has quit IRC | 17:42 | |
| *** e0ne has joined #openstack-infra | 17:43 | |
| *** goldyfruit has quit IRC | 17:43 | |
| *** trident has joined #openstack-infra | 17:46 | |
| *** igordc has joined #openstack-infra | 17:51 | |
| *** Lucas_Gray has quit IRC | 17:51 | |
| *** jamesmcarthur has quit IRC | 17:52 | |
| *** Lucas_Gray has joined #openstack-infra | 17:52 | |
| openstackgerrit | Merged opendev/system-config master: Swap gitea04 into service and bring down gitea05 https://review.opendev.org/673169 | 17:52 |
| fungi | i'll give it a shot, thanks! | 17:53 |
| *** slaweq has joined #openstack-infra | 17:53 | |
| *** slaweq has quit IRC | 17:58 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Don't try to push images when the build failed https://review.opendev.org/673147 | 18:07 |
| *** trident has quit IRC | 18:22 | |
| *** trident has joined #openstack-infra | 18:25 | |
| *** ricolin_ has quit IRC | 18:25 | |
| *** ricolin_ has joined #openstack-infra | 18:26 | |
| *** ricolin_ has quit IRC | 18:33 | |
| *** Vadmacs has quit IRC | 19:04 | |
| *** tosky has quit IRC | 19:30 | |
| *** npochet has quit IRC | 19:34 | |
| *** npochet has joined #openstack-infra | 19:34 | |
| *** TheJulia has quit IRC | 19:35 | |
| *** dougwig has quit IRC | 19:35 | |
| *** philroche has quit IRC | 19:35 | |
| *** dougwig has joined #openstack-infra | 19:35 | |
| *** TheJulia has joined #openstack-infra | 19:35 | |
| *** philroche has joined #openstack-infra | 19:35 | |
| *** hamzy has joined #openstack-infra | 19:38 | |
| *** jbadiapa has joined #openstack-infra | 19:48 | |
| *** hamzy has quit IRC | 19:51 | |
| *** icarusfactor has joined #openstack-infra | 19:53 | |
| *** yamamoto has joined #openstack-infra | 19:54 | |
| *** factor has quit IRC | 19:54 | |
| *** e0ne has quit IRC | 19:59 | |
| *** factor has joined #openstack-infra | 20:00 | |
| *** yamamoto has quit IRC | 20:01 | |
| *** icarusfactor has quit IRC | 20:02 | |
| *** sgw has quit IRC | 20:04 | |
| *** e0ne has joined #openstack-infra | 20:05 | |
| *** hamzy has joined #openstack-infra | 20:16 | |
| *** icarusfactor has joined #openstack-infra | 20:21 | |
| *** factor has quit IRC | 20:23 | |
| *** Lucas_Gray has quit IRC | 20:39 | |
| *** e0ne has quit IRC | 20:53 | |
| *** icarusfactor has quit IRC | 20:55 | |
| fungi | donnyd: when you're back, you probably need to take a look at the nova/neutron logs. i can reproducibly create an instance in "error" state by asking to create it attached to the public_v4 network. opaque exception i get from the sdk is: | 21:21 |
| fungi | openstack.exceptions.SDKException: Error in creating the server (no further information available) | 21:21 |
| fungi | i've left my most recent error instance in there in case you need to prod it | 21:21 |
| fungi | (and if i try to boot one in that tenant without specifying a network it just gives me no ip address/port at all) | 21:22 |
| clarkb | fungi: iirc neutron doesnt allow that | 21:30 |
| openstackgerrit | Jeremy Stanley proposed opendev/zone-opendev.org master: Update IP address for gitea05 https://review.opendev.org/673186 | 21:40 |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Add gitea05 replacement to inventory https://review.opendev.org/673188 | 21:42 |
| fungi | clarkb: in that case i wonder why we have a --network option in the launch script | 21:42 |
| clarkb | fungi: beacuse sometimes you have to select a different network. The issue is in direct attaching to a public floating ip network | 21:48 |
| fungi | ahh | 21:49 |
| fungi | so it's a different kind of network | 21:49 |
| clarkb | that is my assumption given the name. Typically with neutron you either have private networks with a router that you can attach floating IPs from a public network onto or you have provider networks that are used as is basically | 21:50 |
| fungi | i thought the get-me-a-network implementation in shade/sdk automated away detecting when to request a fip | 21:50 |
| clarkb | yes launch node should automatically get a floating IP for you | 21:50 |
| clarkb | assuming there are available fip pools | 21:50 |
| mordred | yes. | 21:50 |
| mordred | but it's possible there is something about the cloud which makes it detect not what you want | 21:51 |
| clarkb | (I didn't do anything special to launch the mirror with regards to networking last time) | 21:51 |
| fungi | well, i think some of the networking changed after the previous mirror was built | 21:51 |
| fungi | but donnyd will have to confirm when he's got time | 21:51 |
| mordred | like if you have a proper direct-attachable public network that can ALSO serve as a source of floating ips - like vexxhost, thank you mnaser for being sane - and you WANT a floating ip for some strange reason, you'll have to be explicit | 21:51 |
| clarkb | mordred: I don't think neutron allows you to direct attach to that network though and that is the source of the error fungi is seeing | 21:52 |
| clarkb | they have to be logically different entities in neutron | 21:52 |
| mordred | clarkb: they do not | 21:52 |
| fungi | anyway, i expect we ought to go ahead and remove the old fn mirror from our inventory, which will somewhat increase our options in being able to boot the replacement | 21:52 |
| mordred | you can TOTALLY configure neutron to allow direct connection and also for the network to be optionally used as a source of fips ... this is how vexxhost works | 21:53 |
| fungi | i'll push up a patch for that while i'm thinking about it | 21:53 |
| mordred | it's how ALL of the clouds *Should* work, but vexxhost is one of the few to have actually gotten it correct | 21:53 |
| clarkb | mordred: ok well it hasn't ever worked in any cloud I've tried that with including devstack. So pmaybe it is a policy thing? | 21:53 |
| mordred | yeah - a network must allow having a router attached to it (routers give you fips) | 21:53 |
| mordred | and a network must also be configured to allow people to get ips directly | 21:54 |
| mordred | and most people do not configure it to do both | 21:54 |
| mordred | because INSANITY | 21:54 |
| mordred | the public network should be a public L2 - and getting ips from a router or directly should be shrug | 21:54 |
| mordred | clarkb: it might be worth getting mnaser to explain it better to us - just so we know | 21:54 |
| clarkb | mordred: did you see I rewrote a chunk of devstack that uses osc to add users, groups, roles, and projects into a python script using sdk and got runtime down from ~100s to ~9s on rax? | 21:55 |
| mordred | but whatever he does is 100% correct and what everyone else should be doing | 21:55 |
| mordred | clarkb: I saw you started - I didn't see the finish - that's AWESOME | 21:55 |
| clarkb | there is a grand total of about 5 minutes spent doing that osc account bootstrapping in devstack today (I only did the keystone specific bits) whcih I think will go to ~15s if we replace it all | 21:55 |
| mordred | clarkb: orders of magnitude are worth doing | 21:55 |
| mordred | sweet | 21:55 |
| mordred | over all of the devstack runs, that's a lot | 21:55 |
| clarkb | and even within a devstack run that takes it from 30 minutes to 25 minues | 21:56 |
| clarkb | which is noticeable | 21:56 |
| mordred | clarkb: we should make sure to get sdk in the list of devstack core cross-gating. it's already critical path, but that could REALLY start blowing chunks :) | 21:56 |
| clarkb | mordred: ya I mean its still early days yet and ianw and the qa team haven't weighed in yet that I've seen | 21:57 |
| clarkb | I bet we get the runtime down even further if the other resources bootstrapping was SDK'd too | 21:57 |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Remove old fortnebula mirror from inventory https://review.opendev.org/673189 | 21:57 |
| clarkb | (things like creating subnets, networks, routers, flavors, etc) | 21:57 |
| fungi | the eventual sdkification of the universe | 21:58 |
| *** igordc has quit IRC | 22:06 | |
| *** igordc has joined #openstack-infra | 22:06 | |
| *** joeguo has joined #openstack-infra | 22:10 | |
| donnyd | I can configure fn to bring public networks directly to your instances, it's all vlans on my end and I use rbacs to make it look external to a tenant | 22:12 |
| *** tosky has joined #openstack-infra | 22:12 | |
| clarkb | donnyd: I don't think we need to if the typical have private network and attach fip flow works | 22:12 |
| donnyd | Saves me a public ip to do it that way, no neutron router required. You use config drives anyways | 22:12 |
| fungi | yeah, provider network is likely fine for this, if it's easier all around | 22:13 |
| fungi | (it often is, i gather) | 22:13 |
| donnyd | 6 in one hand, half dozen in the other to me. Its already configured that way, just need to flip your rbac to access_as_shared | 22:14 |
| donnyd | Ipv4 addresses are a crutch for me anyways, so any I can save is a win in my book. Does the mirror node use it's own FW? | 22:15 |
| clarkb | donnyd: it does | 22:16 |
| donnyd | So it would probably be better to just drop it directly on public | 22:18 |
| donnyd | Like vexxhost does | 22:18 |
| donnyd | Ok, you should be good to drop the mirror directly on the public network | 22:21 |
| *** trident has quit IRC | 22:32 | |
| *** trident has joined #openstack-infra | 22:34 | |
| *** sgw has joined #openstack-infra | 22:43 | |
| *** tkajinam has joined #openstack-infra | 22:52 | |
| fungi | thanks, had the weekly parental call to get out of the way, can try it now | 23:11 |
| *** rcernin has joined #openstack-infra | 23:14 | |
| *** trident has quit IRC | 23:14 | |
| fungi | seems to be working. fingers crossed | 23:15 |
| *** trident has joined #openstack-infra | 23:17 | |
| fungi | donnyd: it worked! only an ipv4 address so far though, what should we do about adding ipv6? | 23:27 |
| *** tosky has quit IRC | 23:29 | |
| donnyd | I can take care of that when I get back | 23:33 |
| donnyd | The v4 vlan only passes v4 traffic | 23:34 |
| donnyd | Not sure if I should do a dual stack on one interface, or have a v4 int and v6 int | 23:35 |
| *** dchen has joined #openstack-infra | 23:39 | |
| donnyd | mordred: if you are using vlan to vxlan, then I would think public to private via os router is the only way to go. FN is CI only and tenant count is low, so vlan works great. Allows me to do either or using neutron rbacs | 23:40 |
| *** dchen has quit IRC | 23:41 | |
| donnyd | I'm sure there are thos e out there that could also do only vxlan, allowing for the same type of network capability | 23:41 |
| *** dchen has joined #openstack-infra | 23:42 | |
| fungi | yeah, and if you don't need >4k vlans or to extend vlans across networks which lack support, then 802.1q works just dandy (and doesn't come with mtu hassles) | 23:42 |
| donnyd | In my experience because my gear doesn't do vxlan offload, it typically performs better as well | 23:43 |
| fungi | yup | 23:43 |
| fungi | in the pre-vxlan days we just made sure our cloud pods didn't have more customer networks than .1q could accommodate | 23:43 |
| *** bobh has joined #openstack-infra | 23:43 | |
| donnyd | So I have the nvme storage up and running, but some truly funny things happen | 23:43 |
| donnyd | One hypervisor balances load via mpio no problem 100k+ iops in an instance... the other two refuse to mpio | 23:44 |
| donnyd | All the exact same config, kernel and equipment | 23:45 |
| donnyd | The storage will do 2m iops without having to try hard | 23:45 |
| donnyd | Blades are such a pita | 23:46 |
| fungi | indeed. are they using some sort of converged network fabric? | 23:48 |
| donnyd | So even though the mirror is up, still waiting on fans to keep the nvme drives cool enough and figuring out why the mpio works for some and not others | 23:48 |
| donnyd | Just eth | 23:48 |
| donnyd | Iscsi | 23:48 |
| donnyd | One hypervisor will properly load balance all links... two will not | 23:49 |
| donnyd | So I will be running that down in my off time till the fans get here | 23:49 |
| fungi | sounds like fun! | 23:50 |
| donnyd | FN is being tuned on the equipment I have to go as fast as possible for this particular use case | 23:50 |
| donnyd | I will get it sort this week and be back to building soon | 23:52 |
| *** jamesmcarthur has joined #openstack-infra | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!