| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] fix venv https://review.opendev.org/726704 | 00:24 |
|---|---|---|
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: Revert "Revert "ensure-tox: use venv to install"" https://review.opendev.org/726705 | 00:24 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] fix venv https://review.opendev.org/726704 | 00:29 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: Revert "Revert "ensure-tox: use venv to install"" https://review.opendev.org/726705 | 00:29 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: ensure-pip: always check for python3-venv on Debuntu https://review.opendev.org/726704 | 00:44 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: Revert "Revert "ensure-tox: use venv to install"" https://review.opendev.org/726705 | 00:44 |
| *** DSpider has quit IRC | 00:46 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] venv probe full path https://review.opendev.org/726715 | 01:00 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: ensure-pip: always check for python3-venv on Debuntu https://review.opendev.org/726704 | 01:10 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: Revert "Revert "ensure-tox: use venv to install"" https://review.opendev.org/726705 | 01:10 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] venv probe full path https://review.opendev.org/726715 | 01:10 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: ensure-pip: use full python3 path https://review.opendev.org/726715 | 03:17 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: ensure-pip: use full python3 path https://review.opendev.org/726715 | 03:40 |
| *** ykarel|away is now known as ykarel | 05:07 | |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: Set python-path for SUSE builds https://review.opendev.org/726728 | 05:35 |
| *** ysandeep|away is now known as ysandeep | 05:42 | |
| *** dpawlik has joined #opendev | 06:10 | |
| *** dpawlik has quit IRC | 06:13 | |
| *** dpawlik has joined #opendev | 06:18 | |
| *** lpetrut has joined #opendev | 06:41 | |
| *** DSpider has joined #opendev | 07:08 | |
| *** tosky has joined #opendev | 07:34 | |
| *** rpittau|afk is now known as rpittau | 07:36 | |
| *** hashar has joined #opendev | 07:38 | |
| *** ralonsoh has joined #opendev | 07:40 | |
| *** avass has joined #opendev | 07:46 | |
| *** dtantsur|afk is now known as dtantsur | 08:04 | |
| *** panda|pto is now known as panda | 08:04 | |
| *** roman_g has joined #opendev | 08:11 | |
| *** shubjero has quit IRC | 08:15 | |
| *** DSpider has quit IRC | 08:17 | |
| *** DSpider has joined #opendev | 08:18 | |
| openstackgerrit | Xinliang Liu proposed openstack/diskimage-builder master: Fix DIB_UBUNTU_KERNEL issue on arm64 https://review.opendev.org/726745 | 08:19 |
| *** ysandeep is now known as ysandeep|lunch | 08:27 | |
| *** ykarel is now known as ykarel|lunch | 08:27 | |
| *** ykarel|lunch is now known as ykarel | 09:00 | |
| *** kevinz has quit IRC | 09:07 | |
| *** kevinz has joined #opendev | 09:07 | |
| *** ysandeep|lunch is now known as ysandeep | 09:11 | |
| *** sshnaidm|off is now known as sshnaidm | 09:15 | |
| *** ykarel is now known as ykarel|mtg | 09:18 | |
| *** priteau has joined #opendev | 09:44 | |
| *** ykarel|mtg is now known as ykarel | 10:05 | |
| *** rpittau is now known as rpittau|bbl | 10:15 | |
| *** ysandeep is now known as ysandeep|brb | 11:21 | |
| *** iurygregory has quit IRC | 11:37 | |
| *** ysandeep|brb is now known as ysandeep | 11:40 | |
| *** rpittau|bbl is now known as rpittau | 11:58 | |
| *** iurygregory has joined #opendev | 11:58 | |
| *** priteau has quit IRC | 12:16 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Remove requiring tox_envlist https://review.opendev.org/726829 | 12:19 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Deprecate default tox_envlist: venv https://review.opendev.org/726830 | 12:20 |
| *** tkajinam has quit IRC | 12:31 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Remove requiring tox_envlist https://review.opendev.org/726829 | 12:35 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Deprecate default tox_envlist: venv https://review.opendev.org/726830 | 12:36 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Do not fail if stestr or testr is not found https://review.opendev.org/726836 | 12:46 |
| *** iurygregory has quit IRC | 13:01 | |
| *** iurygregory has joined #opendev | 13:02 | |
| *** ykarel is now known as ykarel|afk | 13:05 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Deprecate default tox_envlist: venv https://review.opendev.org/726830 | 13:10 |
| smcginnis | I'm seeing job launch failures and problems accessing apt updates. | 13:17 |
| openstackgerrit | Merged zuul/zuul-jobs master: ensure-pip: always check for python3-venv on Debuntu https://review.opendev.org/726704 | 13:20 |
| frickler | smcginnis: do you have an example? | 13:20 |
| smcginnis | frickler: If you filter by "release" in the status view, there are two examples in check right now. | 13:21 |
| smcginnis | Looks like jobs may be launching OK now though. | 13:21 |
| frickler | infra-root: mirror.gra1.ovh.openstack.org seems unresponsive, if s/o has time to take a closer look | 13:24 |
| corvus | a server list says i need to authenticate; i wonder if the account is disabled? | 13:30 |
| *** owalsh has quit IRC | 13:30 | |
| *** owalsh has joined #opendev | 13:31 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Add explanatory comment to tox siblings https://review.opendev.org/726845 | 13:32 |
| fungi | corvus: same for bhs1 region | 13:36 |
| fungi | looks like mirror01.bhs1.ovh.openstack.org may also be down | 13:36 |
| corvus | i've prepared a nodepool change; but since it looks like the entire account may be down, that may be unecessary right now | 13:37 |
| *** dpawlik has quit IRC | 13:38 | |
| fungi | corvus: well, our openstackci account seems disabled but not our openstackjenkins account | 13:39 |
| fungi | so nodepool is booting nodes in the working one but the mirror is offline in the disabled one | 13:40 |
| fungi | so i think your prepared nodepool change is needed | 13:40 |
| corvus | fungi: oh, then i'll push up the change | 13:40 |
| openstackgerrit | James E. Blair proposed openstack/project-config master: Temporarily disable OVH https://review.opendev.org/726848 | 13:40 |
| *** dpawlik has joined #opendev | 13:42 | |
| fungi | we seem to have lost contact with our mirror servers between 12:55 and 13:00 utc | 13:43 |
| fungi | i've reached out to amorin in #openstack-infra | 13:43 |
| fungi | if we don't hear back, we can try e-mail | 13:43 |
| fungi | also pinged rledisez | 13:44 |
| fungi | since they were previously looking into the intermittent keystone errors | 13:44 |
| *** diablo_rojo has joined #opendev | 13:56 | |
| *** ysandeep is now known as ysandeep|afk | 14:02 | |
| fungi | per #openstack-infra, rledisez is passing it along to colleagues and i've privately supplied them with the affected project id | 14:06 |
| *** dtantsur is now known as dtantsur|brb | 14:17 | |
| *** ykarel|afk is now known as ykarel | 14:19 | |
| *** lpetrut has quit IRC | 14:26 | |
| *** jhesketh has quit IRC | 14:35 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: WIP: Remove requiring tox_envlist https://review.opendev.org/726829 | 14:43 |
| openstackgerrit | Merged openstack/project-config master: Temporarily disable OVH https://review.opendev.org/726848 | 14:44 |
| fungi | infra-root: per amorin in #openstack-infra we've been using the wrong flavor for our mirror servers in ovh (hg-7), if we use ssd-osFoundation-3 there instead we won't consume credits | 14:45 |
| fungi | also i deleted a lingering calebb-mirror-update-test instance there, which was almost certainly not in use | 14:46 |
| mordred | fungi: aha | 14:46 |
| fungi | #status log deleted calebb-mirror-update-test server instance in ovh bhs1 region | 14:46 |
| openstackstatus | fungi: finished logging | 14:46 |
| mordred | fungi: so mirror rebuilds are a good idea then | 14:46 |
| openstackgerrit | Merged zuul/zuul-jobs master: Add explanatory comment to tox siblings https://review.opendev.org/726845 | 14:46 |
| fungi | yeah, i'm also trying to work out which of the two mirror instances in gra1 is in use, if it's the newer opendev.org one then i'll go ahead and clean up the openstack.org one | 14:47 |
| clarkb | fungi: the eay to check is via our zuul site vars | 14:47 |
| clarkb | as that has the opendev.org overrides | 14:48 |
| fungi | yeah, or cacti graphs | 14:49 |
| fungi | which indicate it's the old openstack.org instance in use there, the new opendev.org one is not being utilized | 14:50 |
| fungi | clarkb: where do we set those site vars? | 14:55 |
| clarkb | fungi: I always have a hard time finding them. Let me see | 14:56 |
| fungi | i can't seem to find them in base-jobs, either project-config repo, or system-config | 14:56 |
| *** sshnaidm is now known as sshnaidm|afk | 14:57 | |
| clarkb | fungi: https://opendev.org/openstack/project-config/src/branch/master/zuul/site-variables.yaml#L6 | 14:57 |
| fungi | ahh, this is why my git grep was failing | 14:57 |
| fungi | the fact that it's *not* listed there is what makes it use the openstack.org version | 14:58 |
| clarkb | yes | 14:58 |
| fungi | clearly i should have just grepped for any files not including gra1 ;) | 14:58 |
| fungi | thanks! | 14:58 |
| clarkb | however if we arerebuilding anyway maybe we should switch to opendev.org in that cloud now to get ssl | 14:58 |
| clarkb | its a bit more work to set of the acme forwarding but should be done at dome point anyway | 14:59 |
| fungi | so, "conveniently" we have ovh offline at the moment anyway. should we take this opportunity to just blow away the mirrors and rebuild them? | 14:59 |
| *** dtantsur|brb is now known as dtantsur | 14:59 | |
| *** mlavalle has joined #opendev | 14:59 | |
| clarkb | you indicated we needed to use different flavors right? | 14:59 |
| fungi | yeah, even the not-yet-used opendev.org mirror in gra1 is using the "wrong" (billable) flavor | 15:00 |
| clarkb | new flavors requires new instances so ya I think we should try that if possible | 15:00 |
| clarkb | note we'll need volumes which the old servers dont use | 15:00 |
| clarkb | but Im pretty sure ovh allows for volumes? | 15:00 |
| fungi | so shall i just delete the current instances, boot replacement instances and then submit changes to plumb them in the config? | 15:00 |
| fungi | and yes, we're already using cinder volumes | 15:01 |
| clarkb | ah cool and ya I think that is the proper way forward | 15:01 |
| fungi | er, at least the new opendev.org mirror server in gra1 has two cinder volumes anyway | 15:02 |
| fungi | presumably bhs1 supports cinder too though | 15:02 |
| fungi | do i need to submit changes to remove the old servers from our inventory first, or is it fine having them disappear out from under ansible until the change to update inventory entries gets merged? | 15:03 |
| clarkb | Ansible should timeout ssh but sometimes that works less well than expected | 15:05 |
| clarkb | its probably ok, but small chance it js unhappy | 15:06 |
| fungi | #status notice Our CI mirrors in OVH BHS1 and GRA1 regions were offline between 12:55 and 14:35 UTC, any failures there due to unreachable mirrors can safely be rechecked | 15:06 |
| openstackstatus | fungi: sending notice | 15:06 |
| mordred | you can disable them in emergency while waiting on removing them from inventory | 15:06 |
| -openstackstatus- NOTICE: Our CI mirrors in OVH BHS1 and GRA1 regions were offline between 12:55 and 14:35 UTC, any failures there due to unreachable mirrors can safely be rechecked | 15:07 | |
| fungi | mordred: oh, there's a good idea | 15:07 |
| fungi | will do that now | 15:07 |
| clarkb | ++ | 15:07 |
| openstackstatus | fungi: finished sending notice | 15:10 |
| fungi | #status log all ovh mirror servers placed in emergency disable list in preparation for replacement | 15:10 |
| openstackstatus | fungi: finished logging | 15:10 |
| mordred | do we want to boot the new mirrors on focal? if so - I can work on getting the focal image uploaded everywhere | 15:11 |
| mordred | (although maybe I should do that anyway) | 15:11 |
| *** ykarel is now known as ykarel|away | 15:12 | |
| clarkb | mordred: I dont think so as usually we test afs when doing that | 15:14 |
| *** hashar has quit IRC | 15:14 | |
| clarkb | and right now we just need replacement working servers | 15:15 |
| mordred | ++ | 15:16 |
| mordred | clarkb: well - I'm going to get focal uploaded to all the places anyway just so it's there when we want it | 15:16 |
| mordred | (since I did it manually for rax-dfw - might as well ) | 15:16 |
| clarkb | mordred: I think what weve done in the past is bring up a new server next to old. Done basic sanity checking, then if that looks ok we can switch between old and new afs mirrors with dns cname | 15:18 |
| clarkb | then if we notice new is flaky we can quickly back back to old | 15:18 |
| fungi | i'm waiting for traffic levels on the current mirrors to die off before i delete them as we still have some nodes in those regions | 15:19 |
| fungi | but i'll delete the unused mirror01.gra1.ovh.opendev.org and its cinder volumes now | 15:19 |
| fungi | #status log deleted unused mirror01.gra1.ovh.opendev.org server instance and associated main01 and tmpbuild cinder volumes | 15:22 |
| openstackstatus | fungi: finished logging | 15:22 |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Add new non-npm specific javascript jobs https://review.opendev.org/726547 | 15:22 |
| openstackgerrit | Merged zuul/zuul-jobs master: Revert "Revert "ensure-tox: use venv to install"" https://review.opendev.org/726705 | 15:29 |
| openstackgerrit | Merged zuul/zuul-jobs master: ensure-pip: use full python3 path https://review.opendev.org/726715 | 15:29 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Upload focal images to all control plane clouds https://review.opendev.org/726886 | 15:34 |
| mordred | clarkb: ^^ | 15:34 |
| *** ysandeep|afk is now known as ysandeep|away | 15:36 | |
| mordred | clarkb: although - come to think of it - why don't I build a focal arm image on nb03 instead of downloading from canonical | 15:36 |
| clarkb | zuul memory use looks really good (granted it was the weekend): http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=64792&rra_id=all | 15:47 |
| clarkb | mordred: I think I saw out of the corner of my eye friday that the cloud launcher job was failing | 15:47 |
| clarkb | mordred: might want to sort that out before adding more content to that job | 15:48 |
| clarkb | mordred: couple of things on that change though | 15:50 |
| mordred | clarkb: cool and cool | 15:51 |
| tobiash | clarkb: does that host the scheduler and we or just the scheduler? | 15:52 |
| clarkb | tobiash: web + scheduler | 15:53 |
| tobiash | k,thx | 15:53 |
| clarkb | tobiash: with both running under python3.8 with no jemalloc since friday | 15:53 |
| tobiash | cool | 15:53 |
| tobiash | I also tried out zuul-web with py38 with no jemalloc and that looks much better | 15:54 |
| tobiash | we'll switch to py38 with the scheduler soon | 15:54 |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Add new non-npm specific javascript jobs https://review.opendev.org/726547 | 16:01 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Add new non-npm specific javascript jobs https://review.opendev.org/726547 | 16:02 |
| fungi | we've left jemalloc on our executors, but i think we're also running under 3.5 there still | 16:02 |
| clarkb | fungi: correct and its an older version of jemalloc which doesn't seem to have this problem | 16:03 |
| clarkb | my hunch is that its the jemalloc version more than the version of python that hurts us on the newer systems | 16:03 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Add new non-npm specific javascript jobs https://review.opendev.org/726547 | 16:03 |
| fungi | clarkb: i wonder if maybe it's trying to avoid a performance hit from collecting too often | 16:04 |
| clarkb | maybe? you'd think it would balance that against available system memory? but that could explain why sigusr2 seems to force it to go do work | 16:06 |
| *** rpittau is now known as rpittau|afk | 16:09 | |
| mordred | fungi, clarkb: yeah - I think as we update the executors to be on focal we'll strip out jemalloc from the equation | 16:13 |
| clarkb | #status log Restarted ptgbot on eavesdrop.openstack.org as it had netsplit into some alternate reality | 16:14 |
| openstackstatus | clarkb: finished logging | 16:14 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Change the zuul user id when running the base playbook https://review.opendev.org/726490 | 16:17 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Add new non-npm specific javascript jobs https://review.opendev.org/726547 | 16:18 |
| *** dtantsur is now known as dtantsur|afk | 16:19 | |
| openstackgerrit | Merged zuul/zuul-jobs master: cabal-test: add build target job variable https://review.opendev.org/726266 | 16:20 |
| openstackgerrit | Merged zuul/zuul-jobs master: haskell-stack-test: add build target job variable https://review.opendev.org/726267 | 16:20 |
| *** slittle1 has quit IRC | 16:20 | |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Use inventory host lookup for iptables https://review.opendev.org/726472 | 16:24 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Add iptables_extra_allowed_groups https://review.opendev.org/726475 | 16:24 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: tox: allow tox to be upgraded https://review.opendev.org/690057 | 16:25 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Add new non-npm specific javascript jobs https://review.opendev.org/726547 | 16:30 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Extract ensure-javascript-build-tool role https://review.opendev.org/726900 | 16:30 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: tox: allow tox to be upgraded https://review.opendev.org/690057 | 16:32 |
| *** mlavalle has quit IRC | 16:44 | |
| *** mlavalle has joined #opendev | 16:46 | |
| openstackgerrit | James E. Blair proposed opendev/system-config master: DNM: fail zuul tests https://review.opendev.org/726248 | 16:57 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed zuul/zuul-jobs master: tox: allow tox to be upgraded https://review.opendev.org/690057 | 16:58 |
| fungi | okay, so nl04.openstack.org still doesn't seem to have 726848 applied | 16:58 |
| fungi | and it's not in the emergency disable list | 16:59 |
| corvus | the deployment failed: https://zuul.opendev.org/t/openstack/build/71eedea05c2b41c9a9b1bdb803de4c25 | 17:00 |
| fungi | yep, that's where i looked first | 17:00 |
| fungi | will check /var/log/ansible/service-nodepool.yaml.log on bridge.o.o | 17:00 |
| fungi | TASK [nodepool-base : Get zk config] fatal: [nb04.opendev.org]: FAILED! ... No network information facts found | 17:01 |
| fungi | that's nice and cryptic... do we need to gather-facts in that playbook? | 17:02 |
| corvus | that sounds like the thing mordred added already | 17:02 |
| clarkb | ya I think that required us to run against the zk hosts before the nl hosts? | 17:03 |
| mordred | yeah - so - in the library task there we're looking at ansible_default_ipv4 | 17:03 |
| mordred | I'm thinking - perhaps we should update to what corvus is doing in the iptables stuff above | 17:03 |
| mordred | an just use the inventory ip addresses instead of the ansible detected ones | 17:03 |
| clarkb | mordred: that avoids NAT problems too | 17:04 |
| mordred | so - you know, ansible_host instead of ansible_default_ipv4 | 17:04 |
| corvus | yeah that sounds good | 17:04 |
| fungi | so if i reenqueued that change in deploy it would probably work now? or you mean the earlier fix wasn't sufficient? | 17:04 |
| corvus | fungi: earlier insufficient | 17:04 |
| fungi | thanks | 17:04 |
| mordred | we might even be able to get rid of that python | 17:04 |
| corvus | fungi: (i think it relied on putting certain things in all the necessary playbooks) | 17:04 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Use ansible_host instead of ansible_default_ip* for zk https://review.opendev.org/726907 | 17:05 |
| mordred | corvus: I think a followup might be able to replace that ^^ with some jinja | 17:06 |
| corvus | mordred: is there a reason we're using ip addrs there and not hostnames? | 17:07 |
| mordred | yes - the gate doesn't have hostnames | 17:07 |
| mordred | we could use hostnames if we added gate hosts to /etc/hosts | 17:08 |
| clarkb | corvus: wasn't there also that zk bug where explicitly listing IPs was a workaround? | 17:08 |
| clarkb | (I don't recall if that was related or just fixing things that occurred at similar times) | 17:08 |
| corvus | mordred: k; we should add a comment; cause i'm going to keep asking that :) | 17:08 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Use ansible_host instead of ansible_default_ip* for zk https://review.opendev.org/726907 | 17:09 |
| mordred | corvus: done! | 17:09 |
| corvus | re +3 | 17:09 |
| mordred | corvus: we could aternately use the multinode base job for these which would put things into /etc/hots for us | 17:09 |
| corvus | mordred: yeah clarkb has suggested that; but it does lots of other stuff too and i'm not sure i'm comfortable with it as a production simulation | 17:10 |
| mordred | nod | 17:11 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed zuul/zuul-jobs master: yamlint: EOF newlines and comments indent https://review.opendev.org/725516 | 17:12 |
| corvus | we could just run multi-node-hosts-file i guess | 17:12 |
| corvus | oh, the 'bridge' role only runs on 'switch' and 'peers' hosts, so it won't run... | 17:12 |
| corvus | that just leaves multi-node-firewall as a wildcard | 17:12 |
| *** sshnaidm|afk is now known as sshnaidm | 17:12 | |
| mordred | corvus: and I don't think we want that - I think multi-node-hosts-file role in isolation might be a decent idea - because in production we do expect all of our hosts to dns resolve | 17:13 |
| mordred | so it seems like a reasonable way to simulate prod - unless we have any code that avoids /etc/hosts anywhere :) | 17:13 |
| corvus | yeah, i think i'd be comfortable starting down that road | 17:14 |
| mordred | corvus: is it possible to dual-stack a hosts file/ | 17:14 |
| fungi | you can add entries for different addresses with the same name, but i don't know that they'll all be returned | 17:15 |
| corvus | mordred: it's ip -> name, so i don't see why not | 17:15 |
| clarkb | I think its all smart enough to give you the correct ip based on protocol | 17:16 |
| mordred | cool | 17:16 |
| corvus | mordred: stackoverflow says you just enter them twice, one for each protocol :) | 17:16 |
| clarkb | (and ipv6 should be default) | 17:16 |
| fungi | that's convenient | 17:16 |
| openstackgerrit | Merged openstack/project-config master: Finish retiring syntribos repos https://review.opendev.org/726506 | 17:19 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run multi-node-hosts-file in run-base-pre https://review.opendev.org/726910 | 17:20 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed zuul/zuul-jobs master: tox: allow tox to be upgraded https://review.opendev.org/690057 | 17:20 |
| mordred | let's see if that passes tests | 17:20 |
| *** ralonsoh has quit IRC | 18:11 | |
| *** iurygregory has quit IRC | 18:25 | |
| *** roman_g has quit IRC | 18:32 | |
| *** roman_g has joined #opendev | 18:33 | |
| *** iurygregory has joined #opendev | 18:38 | |
| openstackgerrit | Sorin Sbarnea (zbr) proposed zuul/zuul-jobs master: tox: allow tox to be upgraded https://review.opendev.org/690057 | 18:59 |
| *** slittle1 has joined #opendev | 19:17 | |
| fungi | mordred: it won't, because it's stacked on 726907 which is broken on a trivial missing comma. i would have pushed an edit but didn't want to disturb the rest of your stack | 19:18 |
| clarkb | I'm popping out now for a bike ride and lunch. back in a bit | 19:23 |
| fungi | mordred: would you like me to update that stack with the fix for 726907? i'm looking forward to rerunning infra-prod-service-nodepool so we can disable ovh without mucking with the emergency list and hand-edits of the config | 19:24 |
| fungi | i'm holding off a semi-urgent rebuild of the mirror servers there until that happens | 19:25 |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Add new non-npm specific javascript jobs https://review.opendev.org/726547 | 19:29 |
| *** diablo_rojo has quit IRC | 19:30 | |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Extract ensure-javascript-build-tool role https://review.opendev.org/726900 | 19:33 |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Extract ensure-javascript-build-tool role https://review.opendev.org/726900 | 19:35 |
| *** factor has joined #opendev | 19:36 | |
| fungi | mordred: i'm assuming you're okay with it, so i'm pushing a revised stack and reapproving 726907 | 19:37 |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Use ansible_host instead of ansible_default_ip* for zk https://review.opendev.org/726907 | 19:37 |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Run multi-node-hosts-file in run-base-pre https://review.opendev.org/726910 | 19:37 |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Extract ensure-javascript-build-tool role https://review.opendev.org/726900 | 19:56 |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Extract ensure-javascript-build-tool role https://review.opendev.org/726900 | 20:05 |
| *** avass has quit IRC | 20:08 | |
| *** dpawlik has quit IRC | 20:46 | |
| openstackgerrit | Merged opendev/system-config master: Use ansible_host instead of ansible_default_ip* for zk https://review.opendev.org/726907 | 20:55 |
| fungi | yay! | 20:56 |
| fungi | now i guess i can reenqueue 726848,1 into deploy | 20:59 |
| fungi | #status log reenqueued 726848,1 for openstack/project-config into deploy pipeline after fix 726907 merged | 21:03 |
| openstackstatus | fungi: finished logging | 21:03 |
| corvus | fungi: i noticed the deploy of 726907 failed; should we look into that? | 21:14 |
| corvus | "cmd": "/usr/bin/git clone --origin origin file:///home/zuul/src/opendev.org/opendev/ansible-role-puppet /etc/ansible/roles/puppet", | 21:15 |
| corvus | fatal: destination path '/etc/ansible/roles/puppet' already exists and is not an empty directory. | 21:15 |
| corvus | that's on nb03 | 21:16 |
| fungi | hrm, also the reenqueued 726848,1 in deploy failed | 21:16 |
| corvus | mordred: ^ | 21:19 |
| clarkb | that was changed semi recently I think | 21:19 |
| corvus | i'm thinking this is a flaw in install-ansible-roles | 21:19 |
| mordred | yeah. I think it is | 21:19 |
| clarkb | we were installing via galaxy using ansible tooling but then switched to just git cloning it so that we can pull the latest changes | 21:20 |
| mordred | maybe we need a force on that git: | 21:20 |
| clarkb | the issue is the galaxy installs do a copy of the working dir and not a clone | 21:21 |
| corvus | it seems to have one | 21:21 |
| corvus | oh | 21:21 |
| corvus | so we just need a one-time upgrade to add a .git dir? | 21:21 |
| corvus | possibly by just blowing away the directory? :) | 21:21 |
| clarkb | corvus: ya or just move the whole thing aside and have ansible replace it the way it wants? | 21:21 |
| corvus | that sounds reasonable; are all the /etc/ansible/roles dir's handled this way now? | 21:22 |
| clarkb | mordred: ^ | 21:22 |
| mordred | yah | 21:22 |
| mordred | or - yeah - we actually should have only had 2 - puppet and cloud-launcher, right? | 21:22 |
| clarkb | bazelisk-build cloud-launcher exim kerberos-client openafs-client puppet puppet-install set-hostname | 21:22 |
| clarkb | thats the list | 21:23 |
| mordred | oh - so - they come from 2 sources | 21:23 |
| mordred | 2 are git repos | 21:23 |
| mordred | the rest are in the roles/ dir in the root of system-config | 21:23 |
| mordred | those get copy'd - so should be fine | 21:23 |
| corvus | so i can just mv * out of the way? | 21:24 |
| mordred | and I think we decided to use git: for puppet and cloud-launcher so taht they'd use symlnks and not do a copy of the .git dir | 21:24 |
| mordred | corvus: yeah - everything should happily get re-created | 21:24 |
| corvus | #status log moved contents of /etc/ansible/roles on bridge to /etc/ansible/roles/old-2020-05-11 to allow ansible to recreate as git repos | 21:25 |
| openstackstatus | corvus: finished logging | 21:25 |
| corvus | fungi: want to re-enqueue that change now? | 21:25 |
| fungi | sure, can do | 21:28 |
| fungi | does it matter which one we reenqueue? | 21:28 |
| fungi | as long as it runs that job? (is it idempotent?) | 21:28 |
| fungi | like, should i reenqueue the one which merged most recently? | 21:29 |
| clarkb | fungi: it will use the change state | 21:29 |
| clarkb | fungi: so if you enqueue the child it will run with both, if you enqueue the parent only the first set of changes will apply | 21:29 |
| fungi | okay, so best to reenqueue 726907 | 21:29 |
| clarkb | (I think its fine to enqueue either just don't do both out of order) | 21:29 |
| fungi | though 726907,3 was for system-config | 21:30 |
| fungi | it runs the same job | 21:30 |
| fungi | the earlier 726848,1 was for project-config | 21:30 |
| clarkb | ah | 21:30 |
| fungi | so presumably the job will use the latest of the other repo regardless | 21:30 |
| clarkb | yes I believe so | 21:31 |
| fungi | they just happen to both run infra-prod-service-nodepool in deploy | 21:31 |
| fungi | #status log reenqueued 726907,3 for opendev/system-config into deploy pipeline after old .git dir was moved out of the way | 21:32 |
| openstackstatus | fungi: finished logging | 21:32 |
| corvus | continuing our thought about the zuul user... last week we said we would re-id the zuul user to 10001 on all system-config-run jobs so that the zuul nodes in the gate will be able to match the zuul nodes in prod; however, we do the same thing with the zookeeper user on our zk nodes: | 21:35 |
| corvus | https://fa3204066787dd37fd86-ea893277118f144d3b928cfbb4823c04.ssl.cf1.rackcdn.com/726248/4/check/system-config-run-zuul/9df7763/bridge.openstack.org/ara-report/result/3c3e5a36-69cf-4813-aec8-d1794c2f41c0/ | 21:35 |
| corvus | and presumably the same for the nodepool user... | 21:35 |
| corvus | so i don't think that approach is going to work | 21:35 |
| mordred | corvus: oh! yeah - that is ... I didn't think about that | 21:36 |
| clarkb | we use the same uid on all our images? | 21:36 |
| corvus | clarkb: yep | 21:37 |
| mordred | yeah | 21:37 |
| clarkb | I guess we could move zuul only on the zuul servers then | 21:37 |
| mordred | corvus: dumb thought - perhaps what we want to do in our deployment is make a single user on all of the nodes with 10001 that is the user we use to run these services | 21:37 |
| mordred | instead of a zuul, a zk and a nodepool user depending on host | 21:38 |
| *** yuri has joined #opendev | 21:38 | |
| fungi | another hacky workaround i see mentioned in places is to pass the uid/gid in when initializing the container process and chown stuff in the tree at that point | 21:38 |
| corvus | mordred: not a bad idea.... but thinking along a different tack -- how important is it our user be in /etc/passwd? maybe we could define our own standardized series of users in opendev and ignore what's in the containers? | 21:39 |
| mordred | corvus: yeah - and pass in a uid | 21:39 |
| corvus | fungi: yeah that's sort of the direction i'm thinking, except we don't need/want to chown anything (the user shouldn't be writing to anything in the image) | 21:39 |
| mordred | I don't thnk it's important for the user to be in /etc/passwd | 21:40 |
| fungi | that works | 21:40 |
| clarkb | ya our images rely on externally mounted configs and logging and all that | 21:40 |
| corvus | i can't remember how important we thought it was that there be a zuul user in there... | 21:40 |
| fungi | and yeah, as long as any users/groups are referred to strictly by their uid/gid integer values, it's fine | 21:40 |
| mordred | as long as the dirs we mount that need to be written to match the user we tell docker to run the container as | 21:41 |
| corvus | the potential pitfalls i can think of off the top of my head are the fingergw privelege drop and bwrap | 21:41 |
| mordred | all should be fine | 21:41 |
| mordred | corvus: well - we aren't doing executors in docker yet - so that's fine :) | 21:42 |
| mordred | for fingergw - we could tell fingergw to run on a different port and use the docker port expose thing to map it to the correct one and don't to priv drop in fingergw | 21:42 |
| mordred | but that's maybe too complex | 21:42 |
| fungi | another workaround is to create the user(s)/group(s) at container initialization | 21:43 |
| clarkb | is the priv drop target user configurable too? | 21:43 |
| corvus | fungi: that's not possible without running the container as root | 21:44 |
| fungi | good point | 21:44 |
| corvus | fungi: (or else opening a vulnerability in the container; we used to do that in our images but we removed it) | 21:44 |
| fungi | same for if you needed to chown/chgrp i guess | 21:44 |
| corvus | sounds like our best options are: a) create "containeruser" as 10001 and use that for all our containers; b) create zuul, nodepool, zk, etc, as 1000, 1001, 1002, etc and pass in --uid flags and ignore the lack of /etc/passwd entries. | 21:46 |
| corvus | i'm kinda leaning toward (a) just for simplicity | 21:46 |
| mordred | corvus: yeah. I'm not sure having more than one user is really buying us much | 21:47 |
| clarkb | I think a) gets weird if we try to colocate more of these things and want to limit blast radiuses | 21:47 |
| clarkb | but we arne't doing that yet and if we do do that we're probably resolving this problem anyway? | 21:47 |
| mordred | yeah - although a) does make running an AIO test node easier | 21:47 |
| mordred | and yeah | 21:47 |
| fungi | i suppose the main risk with using a similar uid/gid across all containers is that if there's a non-root breakout vulnerability the process has access to the host side files for the processes of other containers | 21:47 |
| fungi | that seems fairly far down the list of things to worry about | 21:48 |
| mordred | yeah - because we don't _Actually_ run these that way | 21:48 |
| clarkb | fungi: can I take the gerrit reviewers plugin off the agenda? We concluded wait until gerrit 2.16 before adding new plugins right? | 21:48 |
| fungi | (also only if they're on the same server) | 21:48 |
| mordred | we're pretty much service-per-machine | 21:48 |
| fungi | clarkb: oh, yes, though i do still owe the ml a follow-up thread on the plan | 21:49 |
| * mordred has to run for the evening - see y'all tomorrows | 21:49 | |
| * fungi adds that to his to do list | 21:49 | |
| fungi | thanks mordred!!! have a great evening | 21:49 |
| corvus | i think that means on the executor we're going to need to run zuul as "container" as well | 21:57 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Run Zuul, Nodepool, and Zookeeper as the "container" user when in containers. https://review.opendev.org/726958 | 21:58 |
| fungi | so the reenqueued 726907,3 also failed... the role 'puppet-install' was not found in /home/zuul/src/opendev.org/opendev/system-config/playbooks/roles:/etc/ansible/roles:/home/zuul/src/opendev.org/opendev/system-config/playbooks The error appears to be in '/home/zuul/src/opendev.org/opendev/system- | 21:58 |
| fungi | config/playbooks/roles/run-puppet/tasks/main.yaml': line 5, column 11, ... | 21:59 |
| fungi | include_role: name: puppet-install | 21:59 |
| corvus | fungi: puppet-install was in the old dir | 21:59 |
| corvus | fungi: how about i move it back for now | 21:59 |
| clarkb | corvus: do we need the uids to match between hosts running zuul services? | 22:00 |
| clarkb | or is that just for consistency and simplicity? | 22:00 |
| fungi | what's normally supposed to put it there, i wonder | 22:00 |
| corvus | clarkb: no, but we use the same role to write out the same zuul.conf everywhere | 22:00 |
| clarkb | corvus: gotcha | 22:00 |
| clarkb | fungi: that was one of the role sthat was moved aside | 22:00 |
| clarkb | fungi: so I'm guessing we don't bootstrap those properly after galaxy was removed? | 22:00 |
| fungi | clarkb: i got that, but what installs it? | 22:00 |
| fungi | yeah, sounds like maybe nothing now | 22:01 |
| corvus | grepping isn't turning up anything other than use of puppet-install | 22:01 |
| fungi | so this probably would have failed similarly on a fresh replacement server | 22:02 |
| corvus | oh there is a puppet-install role in the system-config repo | 22:02 |
| clarkb | ya these are system-config/roles/ roles | 22:03 |
| fungi | and this was a change for system-config, even | 22:04 |
| corvus | oh and they're supposed to be copied into place so we can run ansible without a system-config repo? | 22:04 |
| clarkb | and what we had on bridge were copies not symlinks | 22:04 |
| clarkb | corvus: ya I think we expect proper copies | 22:04 |
| clarkb | the config file that was pulling the other roles from galaxy doesn't list these roles | 22:04 |
| clarkb | I'm still not sure what mechanism was installing these before | 22:04 |
| corvus | i think the install-ansible role does it | 22:06 |
| clarkb | yup just ofund it | 22:07 |
| corvus | which we don't run in the service-nodepool playbook | 22:07 |
| clarkb | "Copy system-config roles into place" is the task name | 22:07 |
| fungi | okay, so we could probably just add install-ansible to service-nodepool? | 22:08 |
| clarkb | fungi: or wait for the hourly job (or enqueue the hourly job?) | 22:08 |
| corvus | i don't think we run that role in production -- only when we bootstrap bridge | 22:08 |
| clarkb | oh | 22:08 |
| clarkb | hrm I would expect we'd keep those roles up to date :/ | 22:08 |
| corvus | me too, maybe we dropped it from the cron script and didn't have a replacement | 22:09 |
| clarkb | agreed on not running it elsewhere | 22:09 |
| clarkb | seems we test it but don't have a prod side applying it | 22:09 |
| corvus | it used to be in bridge.yaml | 22:09 |
| corvus | but hrm, even then, i think that was only run in zuul | 22:10 |
| corvus | yeah, i think this hasn't been done for a long time, if ever. | 22:11 |
| *** sshnaidm is now known as sshnaidm|afk | 22:11 | |
| clarkb | this is me thinking out loud: we can copy those roles by hand now, but maybealso push up a change to run install-ansible in service-bridge.yaml then have mordred look that over tomorrow? | 22:11 |
| corvus | oh, i failed at grep; we did run service-brdige in run_all.sh | 22:13 |
| *** yuri has quit IRC | 22:13 | |
| *** yuri has joined #opendev | 22:13 | |
| clarkb | or maybe we extend install-ansible-roles to cover this set of roles too | 22:14 |
| clarkb | (that might make it easier to understand how this happens in the future) | 22:14 |
| corvus | clarkb: i agree with your 22:11 plan | 22:15 |
| corvus | (i also think moving this to install-ansible-roles might be a good idea, but non-trivial since it's a different mechanism) | 22:16 |
| corvus | (but at least it'd be in a role with the right name :) | 22:16 |
| clarkb | ya the loop in intsall-ansible-modules is wrong for this set of roles | 22:16 |
| clarkb | fungi: ^ what do you think should we go ahead with idea at 22:11? | 22:16 |
| fungi | yes, seems reasonable. skimming, it doesn't seem destructive (and if we used to run it in run_all.sh then i think that confirms) | 22:17 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Add install-ansible to service-bridge https://review.opendev.org/726961 | 22:17 |
| fungi | and sorry, bit of a space cadet at the moment, just finishing up evening chores and cooking/eating dinner | 22:17 |
| corvus | mordred: https://review.opendev.org/726961 | 22:18 |
| corvus | i see all the roles in the dir now | 22:18 |
| corvus | did anyone move them into place, or did ansible just do that? | 22:19 |
| fungi | i did nothing, so must be ansible ghosts | 22:19 |
| corvus | clarkb: ^? | 22:19 |
| fungi | gnomes? | 22:19 |
| clarkb | I did not | 22:19 |
| fungi | i bet they're actually gremlins | 22:19 |
| corvus | i guess maybe something does run that role after all....? | 22:19 |
| clarkb | corvus: must be | 22:19 |
| corvus | i'll just abandon that change | 22:19 |
| fungi | the hourly could have just run since we were looking | 22:19 |
| corvus | fungi: yeah, i just didn't think the hourly ran the role. <shrug> | 22:20 |
| clarkb | infra-prod-install-ansible <- did that job run? | 22:20 |
| * clarkb is looking | 22:20 | |
| corvus | fungi: anyway, want to re-run your re-enqueue? | 22:20 |
| clarkb | https://zuul.opendev.org/t/openstack/build/e0a401e507584688a5ce50d15bc1793b ya I think that was it | 22:21 |
| fungi | clarkb: last ran 2020-05-07T20:40:24 in deploy and took 4 mins 16 secs | 22:21 |
| clarkb | and Ishouldn't dismiss zuul things so quickly in my greps because zuul is running the show | 22:21 |
| fungi | i thought we were looking after that though | 22:21 |
| clarkb | fungi: the one above ran just now | 22:21 |
| fungi | wow, yep | 22:21 |
| fungi | 2020-05-11T21:00:10 | 22:22 |
| fungi | i should have refreshed | 22:22 |
| fungi | okay, 726907,3 reenqueued | 22:22 |
| fungi | clarkb: https://zuul.opendev.org/t/openstack/build/e0a401e507584688a5ce50d15bc1793b was over an hour ago... i'm guessing there was a newer one at 22:00 that just hasn't reported yet | 22:24 |
| clarkb | oh I was off by an hour | 22:25 |
| fungi | it happens | 22:25 |
| clarkb | ya I bet we havne't cmpleted that buildset so no logs yet | 22:25 |
| clarkb | but the job itself has run | 22:25 |
| fungi | right | 22:25 |
| ianw | is there a summary of what's going on, if i can help? | 22:26 |
| fungi | ianw: the start was when we discovered builds in ovh were failing because our mirror servers there spontaneously went offline | 22:27 |
| fungi | that prompted pushing a change to turn max-servers in both ovh regions to 0 | 22:28 |
| ianw | right ok, that's the ones rebuilt with non-billable flavors now? | 22:28 |
| fungi | well, not yet | 22:29 |
| fungi | very nice ovh folk turned everything back on, and let us know that we should use the non-billed flavors | 22:29 |
| fungi | but the max-servers never got applied on nl04 | 22:29 |
| fungi | i was waiting for utilization there to drop to 0 before starting mirror rebuilds | 22:29 |
| fungi | though i did delete the unused mirror instances and volumes in the meantime | 22:29 |
| ianw | ahh ok; that seems like something i can help with if it's getting late for you | 22:30 |
| fungi | at this point we've just been iterating on figuring out why infra-prod-service-nodepool isn't working | 22:30 |
| ianw | also on my todo was to add focal to the mirror testing | 22:30 |
| clarkb | I'm still around for a bit too but allergies have been really bad today (some of that was my own fault going on a bike ride not realizing the air was yellow) | 22:30 |
| fungi | evening is rapidly bearing down on my slice of the globe | 22:30 |
| fungi | so i likely don't have much bandwidth to do mirror rebuilds | 22:31 |
| fungi | but this seems like a good opportunity to switch over to opendev.org mirrors with letsencrypt certs in ovh | 22:31 |
| fungi | i think the non-billed flavors will likely need cinder volumes for apache/afs caches | 22:32 |
| ianw | ok, i can look at that; so both bhs and gra? | 22:34 |
| clarkb | ianw: oui | 22:34 |
| fungi | ianw: thanks! i should be able to take over when i wake up, if there's still more to do | 22:35 |
| fungi | i'll probably also be around for a few more hours to review config changes | 22:35 |
| fungi | just dealing with the usual waning brain function which accompanies sunset | 22:36 |
| fungi | infra-prod-service-nodepool SUCCESS! | 22:36 |
| fungi | max-servers: 0 | 22:37 |
| fungi | now we're cooking with gas | 22:37 |
| fungi | hopefully utilization there will begin to trail off | 22:37 |
| ianw | ahhh ok so we needed that to work to update the config :) | 22:37 |
| fungi | yes ;) | 22:37 |
| *** DSpider has quit IRC | 22:38 | |
| fungi | thanks mordred, corvus, clarkb for working through that! | 22:38 |
| ianw | ok i just need to transition to school supervisor mode ... (i.e. take laptop to dining room table and ensure ipads are used for good and not evil) | 22:39 |
| clarkb | ianw: that sounds very familiar | 22:39 |
| fungi | wait, they're not being taught evil? | 22:39 |
| clarkb | fungi: kids are born with in innate ability to do evil | 22:39 |
| fungi | oh, good point, no need to teach them more of that | 22:40 |
| fungi | they're already experts | 22:40 |
| ianw | i wonder if i can get in one run of the mirrors on focal to see if it's an option ... | 22:46 |
| fungi | no objection here | 22:46 |
| fungi | we debated it, but didn't want to venture there without tapping your current knowledge of the topic | 22:47 |
| fungi | particularly regarding openafsclient et cetera | 22:48 |
| ianw | i think we've got everything ... openafs should be up and i think my change to test in base merged | 22:48 |
| ianw | actually no, but we probably should - https://review.opendev.org/#/c/725676/ | 22:48 |
| fungi | i'll take a look now | 22:48 |
| ianw | that just keeps the basic setup clean | 22:49 |
| fungi | looks like the job still works, 41 tasks changed on the focal node | 22:50 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Add focal testing for mirror nodes https://review.opendev.org/726970 | 22:53 |
| *** tkajinam has joined #opendev | 22:55 | |
| fungi | ianw: i think you got cut-n-paste happy there and missed the critical bit, see inline comment | 22:56 |
| ianw | haha yes was just wondering why it was pulling from bionic repos! | 22:57 |
| clarkb | oh heh | 22:57 |
| clarkb | my brain is already non functioning | 22:57 |
| *** tosky has quit IRC | 22:57 | |
| fungi | i'm clearly not far enough into this beer yet | 22:57 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Add focal testing for mirror nodes https://review.opendev.org/726970 | 22:58 |
| ianw | istr throwing an arm focal test at dib last week too, i think it failed | 22:59 |
| clarkb | ianw: mordred has a change up to upload arm focal control nodes too fwiw | 23:00 |
| clarkb | (it also uploads non arm to all the other clouds too) | 23:00 |
| ianw | via cloud launcher? | 23:01 |
| clarkb | ya | 23:02 |
| ianw | ok 726886 got it | 23:03 |
| ianw | another one if you could take a poke at is https://review.opendev.org/#/c/726040/ | 23:04 |
| ianw | it skips the kubctl stuff on arm64 so the arm64 base test works | 23:04 |
| clarkb | done | 23:04 |
| ianw | only because we have to keep the bridge roles minimally working on arm64, enough to just fire off ansible on hte hosts e care about | 23:04 |
| fungi | makes sense | 23:05 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Upload focal images to all control plane clouds https://review.opendev.org/726886 | 23:09 |
| ianw | clarkb: ^ as a test i could upload to OVH manually and bootstrap a test | 23:10 |
| clarkb | ianw: ya I think its already manually uploaded to rax | 23:12 |
| ianw | ovh has a Ubuntu 20.04 at any rate | 23:13 |
| *** sshnaidm|afk has quit IRC | 23:14 | |
| *** sshnaidm has joined #opendev | 23:15 | |
| *** sshnaidm is now known as sshnaidm|afk | 23:16 | |
| openstackgerrit | Merged opendev/system-config master: Add focal to system-config base job https://review.opendev.org/725676 | 23:16 |
| ianw | so i'll try it with ssd-osFoundation-3 flavor | 23:16 |
| ianw | fungi: was there anything important to consider in creating volumes to attach as far as you know? | 23:18 |
| clarkb | ianw: I think we need ~200GB of disk total for two different mount points. I think a single cinder volume is fine | 23:18 |
| fungi | yeah, basically giving apache and afs 200g combined to spread out in, without overrunning the rootfs | 23:19 |
| clarkb | apache wants 100gb iirc. we set the cache prune limit lower, but because it runs periodically we need headroom to grow into | 23:19 |
| ianw | fungi: umm, what if there is no ssd-osFoundation-3 flavor? | 23:22 |
| clarkb | I think they mentioned htat and were going to fix it, I guess that didn't happen? | 23:23 |
| clarkb | 14:41:00* amorin | dont hesite to send me a mail, I can also enable the ssd-osFoundation-3 flavor on your other tenant (I remmeber you have 2 tenants) | 23:23 |
| clarkb | hrm I may have misread that as "we will add that flavor" but they weren't actually planning on it? | 23:23 |
| ianw | i'm not seeing it on openstackci or openstackzuul | 23:23 |
| fungi | interesting, yeah i hadn't gotten far enough to check that yet | 23:24 |
| clarkb | ssd-osFoundation-3 that is what we filter for in nodepool | 23:24 |
| clarkb | so it must be there in openstackzuul? | 23:24 |
| fungi | maybe you can catch amorin in the emea morning at least | 23:24 |
| ianw | # OS_CLIENT_CONFIG_FILE=/etc/openstack/all-clouds.yaml openstack --os-cloud=openstackzuul-ovh --os-region=GRA1 flavor list | grep osFoundation | 23:25 |
| clarkb | ianw: note there isn't an all-clouds.yaml anymore we can just use the default now | 23:26 |
| fungi | yeah, no need to pass OS_CLIENT_CONFIG_FILE now | 23:28 |
| fungi | also the credentials for openstackjenkins-ovh seem to be incorrect in there | 23:31 |
| fungi | The request you have made requires authentication. (HTTP 401) (Request-ID: req-1c302da4-7997-4810-9f46-49b041042b4f) | 23:31 |
| clarkb | hrm thats different than the http503 we had before (but similar error) | 23:32 |
| clarkb | oh ya is it openstackzuul-ovh or jenkins? | 23:32 |
| clarkb | ianw: ^ your grep might be hiding an error? | 23:32 |
| ianw | indeed, yeah it's jenkins | 23:32 |
| fungi | in the clouds.yaml on bridge it's jenkins | 23:32 |
| fungi | but either the credentials in there are wrong or that account is now disabled too | 23:33 |
| ianw | fungi: the openstackjenkins-ovh account just worked for me, and saw the osFoundation flavors | 23:33 |
| ianw | however, i'm not seeing it in openstackci-ovh | 23:33 |
| fungi | hah, apparently the ~/launch-env/bin/openstack i have isn't authenticating for openstackjenkins-ovh but is authenticating for openstackci-ovh | 23:35 |
| fungi | if i use /usr/local/bin/openstack instead both work | 23:35 |
| fungi | and yeah, i also misread amorin's comment as indicating he had copied that flavor to the other project, but i hadn't yet gotten around to verifying that. oh well | 23:39 |
| ianw | i'll send a mail, cc infra-root | 23:40 |
| clarkb | ianw: thnaks | 23:40 |
| fungi | awesome, thanks a bunch! | 23:41 |
| ianw | Task: letsencrypt-create-certs : Populate service facts | 23:52 |
| ianw | Malformed output discovered from systemd list-unit-files: acpid.service disabled enabled | 23:52 |
| *** kevinz has quit IRC | 23:52 | |
| ianw | that i did not expect ... | 23:52 |
| clarkb | is that ansible complaining aobut list-unit-files output? | 23:52 |
| ianw | yeah, i wonder if ansible itself isn't focal happy | 23:52 |
| ianw | https://github.com/ansible/ansible/issues/68536 | 23:55 |
| ianw | so that puts a bit of a twist in that plan too | 23:56 |
| clarkb | has that made it into a bug fix for a release yet? seems like that would be important for naisble to do (hopefully we don't have to go through upgrading all of ansible just to do focal) | 23:58 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!