| ianw | np, i know it's a holiday :) | 00:02 |
|---|---|---|
| fungi | well, also well into the evening here | 00:25 |
| fungi | so i'm even less responsive than usual | 00:26 |
| *** elod has quit IRC | 00:35 | |
| *** elod has joined #opendev | 00:36 | |
| *** ysandeep|away is now known as ysandeep | 01:19 | |
| *** noonedeadpunk has quit IRC | 01:47 | |
| *** noonedeadpunk has joined #opendev | 01:50 | |
| frickler | ianw: can you take a look at clarkb's comment on https://review.opendev.org/#/c/749604/1 ? that seems to be the only thing keeping the whole stack from getting merged | 05:29 |
| ianw | frickler: i'm pretty sure that in all contexts that runs bridge==localhost ... if you agree we can probably merge | 05:42 |
| *** DSpider has joined #opendev | 05:59 | |
| *** xiaolin has joined #opendev | 06:01 | |
| *** qchris has quit IRC | 06:20 | |
| *** xiaolin has quit IRC | 06:21 | |
| *** xiaolin has joined #opendev | 06:24 | |
| *** qchris has joined #opendev | 06:33 | |
| *** hashar has joined #opendev | 06:53 | |
| *** openstackgerrit has joined #opendev | 06:55 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: bootloader: remove dangling grubenv links https://review.opendev.org/750279 | 06:55 |
| *** andrewbonney has joined #opendev | 07:28 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: bootloader: remove dangling grubenv links https://review.opendev.org/750279 | 07:31 |
| *** tosky has joined #opendev | 07:35 | |
| *** dtantsur|afk is now known as dtantsur | 07:56 | |
| *** moppy has quit IRC | 08:01 | |
| *** moppy has joined #opendev | 08:02 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: bootloader: remove dangling grubenv links https://review.opendev.org/750279 | 08:05 |
| *** tosky has quit IRC | 08:05 | |
| frickler | ianw: I don't feel confident enough to state that, better wait for someone else to double check | 08:09 |
| *** tosky has joined #opendev | 08:10 | |
| *** tosky has quit IRC | 08:14 | |
| *** tosky has joined #opendev | 08:14 | |
| *** priteau has joined #opendev | 08:35 | |
| *** priteau has quit IRC | 09:12 | |
| *** priteau has joined #opendev | 09:14 | |
| *** priteau has quit IRC | 09:17 | |
| *** priteau has joined #opendev | 09:17 | |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/elastic-recheck master: Drop py27 and add py38 jobs https://review.opendev.org/729330 | 09:21 |
| *** xiaolin has quit IRC | 09:23 | |
| *** tosky has quit IRC | 09:39 | |
| *** tosky has joined #opendev | 09:54 | |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/elastic-recheck master: Enable pylint https://review.opendev.org/750312 | 10:25 |
| openstackgerrit | Merged opendev/elastic-recheck master: Drop py27 and add py38 jobs https://review.opendev.org/729330 | 10:35 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/elastic-recheck master: Avoid pip breakage due to keyring https://review.opendev.org/750318 | 10:53 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/elastic-recheck master: Avoid pip breakage due to keyring https://review.opendev.org/750318 | 10:55 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/elastic-recheck master: Enable pylint https://review.opendev.org/750312 | 11:01 |
| *** ysandeep is now known as ysandeep|brb | 11:50 | |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/elastic-recheck master: Enable pylint https://review.opendev.org/750312 | 12:12 |
| *** redrobot has joined #opendev | 12:18 | |
| donnyd | whenever you are around fungi can you check the MTU on for the OE mirror. Should be 1480, but I just wanted to check | 12:21 |
| fungi | donnyd: i think it may be broken. i can't seem to ssh into it at all. getting "/bin/bash: Input/output error" and then the connection closes. i wonder if it's got a problem with its disk | 12:27 |
| donnyd | hrm | 12:32 |
| donnyd | well that is probably not a good thing | 12:32 |
| *** ysandeep|brb is now known as ysandeep | 12:42 | |
| donnyd | fungi: I know why | 12:49 |
| donnyd | My core switch flaps ports whenever I change the speed on just one | 12:49 |
| donnyd | so its likely it just needs a rebootr | 12:50 |
| zbr | there is something wrong with http://logstash.openstack.org/ -- it does fail to load | 12:51 |
| zbr | basically 503 | 12:51 |
| zbr | sadly its gui is not even able to render a proper 503 page | 12:52 |
| donnyd | fungi: I rebooted the mirror | 12:53 |
| donnyd | I think we should probably try to move the OE mirror to use cinder as a backing store, now that I have the issues with it resolved | 12:54 |
| fungi | zbr: yep, sorry, responded to your comment in #openstack-infra about it just now. i think we lost both elasticsearch03 and 04, and the cluster can only handle losing one node at a time | 12:56 |
| fungi | ianw rebooted 03 when i noticed it wasn't responding to ssh around utc midnight | 12:57 |
| fungi | i'm about to do the same to 04 now | 12:57 |
| fungi | donnyd: i can ssh into the oe mirror again, thanks! | 12:58 |
| fungi | ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc fq_codel state UP group default qlen 1000 | 12:58 |
| fungi | so not 1400 | 12:58 |
| fungi | is it supposed to pick that up from dhcp or configdrive? | 12:58 |
| fungi | #status log elasticsearch03 was rebooted 2020-09-08 23:48z after it was found to be hung | 13:00 |
| openstackstatus | fungi: finished logging | 13:00 |
| openstackgerrit | Merged opendev/elastic-recheck master: Avoid pip breakage due to keyring https://review.opendev.org/750318 | 13:01 |
| openstackgerrit | Merged opendev/elastic-recheck master: Enable pylint https://review.opendev.org/750312 | 13:02 |
| fungi | yeah, confirmed it's the same situation with 04, "INFO task blah-blah:blah blocked for more than 120 seconds" | 13:04 |
| fungi | hard rebooting it via nova api now | 13:04 |
| donnyd | 1480 is what it's supposed to be | 13:05 |
| fungi | #status log elasticsearch04 rebooted after it was found to be hung | 13:05 |
| openstackstatus | fungi: finished logging | 13:05 |
| donnyd | thank you for checking fungi | 13:05 |
| fungi | donnyd: oh, cool, yep now i see you said 1480 earlier not 1400. time for glasses i guess | 13:06 |
| fungi | zbr: even after rebooting elasticsearch04 and confirming all 6 are reachable now, i suspect that it will need the index repaired or wiped since it lost two out of the six cluster nodes | 13:17 |
| fungi | i'll try to dig out our documentation on how to do that here shortly | 13:17 |
| zbr | fungi: thanks. ping when ready so I can resume my e-r work (that is how i found it) | 13:18 |
| fungi | i figured, and yep, will do | 13:18 |
| * fungi mutters something about clouds | 13:18 | |
| *** AnonKapes has joined #opendev | 13:28 | |
| *** AnonKapes has quit IRC | 13:30 | |
| fungi | okay, so turns out part of the problem was that elasticsearch did not start successfully on 03 and 04 when they were rebooted. i manually started them with systemctl (after stopping them just to make sure the status was cleaned up correctly) and they seem to be running again now | 13:42 |
| fungi | though the cluster status is currently "red" http://paste.openstack.org/show/797577/ | 13:42 |
| fungi | zbr: kibana is working again, and returning log events, though i don't know how much data might be missing | 13:43 |
| *** hashar is now known as hasharAway | 13:51 | |
| tristanC | dear openstack/opendev operator, would it be possible to take over the pynotedb package on pypi? it seems like the project was aborted | 13:52 |
| fungi | tristanC: yeah, i forget exactly what zara was wanting to use that for, maybe SotK remembers | 13:54 |
| fungi | are you wanting to resume development of https://opendev.org/opendev/pynotedb or starting from scratch elsewhere and just reusing the package name? | 13:55 |
| fungi | (if the latter, then we should properly retire the git repository too) | 13:55 |
| tristanC | i happen to have created another implementation from scratch named pynotedb, without checking if the name was available | 13:57 |
| fungi | tristanC: for the same purpose (python library to interface with a gerrit notedb)? | 13:58 |
| fungi | tristanC: where are you hosting the source code for it? if we retire the repository on opendev we should probably make sure the readme links to where your source code lives so as to avoid confusion | 13:59 |
| * fungi guesses it's in softwarefactory | 13:59 | |
| *** mlavalle has joined #opendev | 14:00 | |
| tristanC | the purpose of my implementation is mostly to manage the All-Users external-ids reference, it's: https://softwarefactory-project.io/r/#/q/project:software-factory/pynotedb | 14:00 |
| fungi | https://softwarefactory-project.io/cgit/software-factory/pynotedb/ | 14:01 |
| fungi | yep, just found it | 14:01 |
| fungi | tristanC: ooh, that rings a bell. yes i think zara may have been developing hers for similar reasons, to make it possible to deploy gerrit from scratch and inject initial accounts/config into notedb | 14:01 |
| fungi | anyway, what we're hosting in opendev is obviously defunct. looks like there was just a cookiecutter template pushed three years ago, so i don't see any problem relinquishing the name on pypi to someone who is actively making the thing we never got around to building | 14:03 |
| fungi | i'm just one voice though. we can try to discuss it in the opendev meeting at 19:00 today, at least during open discussion. you don't need to be there, i'm happy to present the case on your behalf if that's an inconvenient time for you | 14:05 |
| fungi | i expect everyone else will agree, but i don't want to be making that call without conferring with others | 14:06 |
| tristanC | fungi: thank you very much, otherwise no big deal, the library usage changes are not yet merged and it still easy to pick another name | 14:06 |
| fungi | clarkb: i don't recall seeing a meeting agenda go out yet. are we still meeting today? | 14:11 |
| SotK | I'm in favour of handing the name over | 14:11 |
| SotK | and retiring the defunct repo | 14:12 |
| fungi | thanks SotK. if you still need a python library for interfacing with notedb then maybe work with tristanC to see if the design for theirs can be made to meets your needs | 14:12 |
| fungi | er, s/meets/meet/ | 14:13 |
| tristanC | SotK: fungi: thank you, AJaeger: it seems like https://review.opendev.org/#/c/597402/1 could now be restored ^ | 14:14 |
| SotK | fungi: I think that is the approach that makes the most sense, yeah | 14:15 |
| AJaeger | tristanC: I can restore - will you update if needed and propose the changes to remove it completely from opendev, please? | 14:17 |
| tristanC | AJaeger: sure, but i can't click restore | 14:19 |
| clarkb | fungi: yes, I need to semd one this morning. I didnt do it yesterday | 14:22 |
| *** priteau has quit IRC | 14:23 | |
| fungi | no worries, just double checking. thanks! | 14:29 |
| dtantsur | hey folks! can I make zuul checkout difference branches of the same project, i.e. similar to how grenade works? | 14:31 |
| *** ysandeep is now known as ysandeep|afk | 14:37 | |
| tristanC | dtantsur: iiuc that is not supported, but that is likely a question for the #zuul channel | 14:37 |
| clarkb | dtantsur: tristanC well zuul ensures that every branch in a repo has its head set to the correct speculative state | 14:41 |
| dtantsur | yeah, but for a grenade-alike job I need two sets of repos: old and new | 14:41 |
| clarkb | dtantsur: that is all done in the job | 14:41 |
| clarkb | it clones the source repo set up by zuul then checks out the appropriate branches for old and new iirc | 14:42 |
| clarkb | zuul itself doesn't need to do anything to make that happen, it has already handed up the appropriate repo state | 14:42 |
| AJaeger | tristanC: thanks! | 14:42 |
| dtantsur | this way depends-on will only work for "new" branches, right? | 14:42 |
| *** tkajinam has quit IRC | 14:43 | |
| dtantsur | so I cannot Depends-On say stable/ussuri ironic change? | 14:43 |
| clarkb | dtantsur: no, zuul configues all branches with the speculative states | 14:43 |
| dtantsur | ah! so `git checkout stable/ussuri` will give me the change? | 14:43 |
| clarkb | yes | 14:43 |
| dtantsur | and `master` will mean the change in question? | 14:43 |
| clarkb | dtantsur: it depends if the proposed change was to master, but ys | 14:43 |
| dtantsur | yeah, right, assuming ussuri->master | 14:44 |
| dtantsur | thanks clarkb! | 14:44 |
| clarkb | (if anyone is wondering this is why we combine all branches into the same zuul pipeline queue in the gate) | 14:48 |
| *** priteau has joined #opendev | 14:49 | |
| *** Gyuseok_Jung has quit IRC | 14:49 | |
| *** diablo_rojo has joined #opendev | 14:49 | |
| *** Gyuseok_Jung has joined #opendev | 14:50 | |
| *** sgw has joined #opendev | 15:01 | |
| *** mlavalle has quit IRC | 15:04 | |
| *** mlavalle has joined #opendev | 15:07 | |
| fungi | dtantsur: also the "origin" remote is set to have the parent change for each branch | 15:13 |
| fungi | if you end up needing that for anything | 15:13 |
| dtantsur | cool | 15:13 |
| clarkb | usually that comes up in the context of git diffs or when you want to examine specifically the unmerged state | 15:13 |
| *** hasharAway has quit IRC | 15:19 | |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/elastic-recheck master: WIP: Use pytest for queries https://review.opendev.org/750445 | 15:36 |
| *** admin0 has joined #opendev | 15:47 | |
| *** hasharAway has joined #opendev | 15:50 | |
| clarkb | I've approved the dns change for nb03.opendev.org. https://review.opendev.org/#/c/750039/ is the next thing thatneeds to go in and that is the potentially dangerious one if nodepools fix for using server uuids instead of hostnames doesnt work as expected | 15:51 |
| *** hasharAway is now known as hashar | 15:51 | |
| clarkb | (I expect it will be fine but if there is a change to watch closely in this stack for nb03 it is that one) | 15:51 |
| clarkb | *reapproved the dns change. It was already approved but due to not sharing a gate pipeline with its depends on didn't auto enqueue when theparent merged | 15:52 |
| clarkb | https://review.opendev.org/#/c/744821/ would be a useful change ot land for the management of the sshfp records | 15:52 |
| openstackgerrit | Merged opendev/zone-opendev.org master: Add nb03.opendev.org to DNS https://review.opendev.org/750037 | 15:58 |
| clarkb | I'm also double checking setuptools changelogs and I think we're ok but I'm rechecking https://review.opendev.org/#/c/749766/2 now to double check | 16:11 |
| clarkb | if that is still a +1 from zuul I think we should also land https://review.opendev.org/749777 so we don't forget | 16:11 |
| fungi | yeah, my confusion yesterday was that a couple of jobs which had been previously been broken by setuptools failures landed in vexxhost ca-ymq-1 when the mirror there had broken ipv6 routes | 16:18 |
| clarkb | fungi: do you have time to review https://review.opendev.org/#/c/750039/ to enroll nb03.opendev.org into ansible? And if so any thoughts on trying to land that before or after the meeting? | 16:22 |
| fungi | i thought i already had reviewed it. checking | 16:24 |
| *** ysandeep|afk is now known as ysandeep | 16:24 | |
| fungi | aha, yep, i had missed that one. let's land it now | 16:24 |
| fungi | clarkb: out of curiosity, hadn't we blanked out the ipv6 address for the old builder there for a reason? was it just not reachable, or were there intermittent connectivity issues over v6 there? | 16:26 |
| clarkb | dns seems to resolve for me after the previous change merged | 16:26 |
| clarkb | I haven't accepted the ssh host keys because I think we're relying on sshfp for that now | 16:26 |
| clarkb | fungi: do you know if ^ is the case? If not I can manually accept the host key | 16:26 |
| clarkb | I don't recall where the make ssh and ansible honor sshfp discussion got to | 16:27 |
| fungi | i thought we added the -owhatever to the ssh invocations, but yeah i don't entirely recall. checking | 16:28 |
| fungi | hrm, codesearch isn't turning up uses of VerifyHostKeyDNS for us | 16:30 |
| clarkb | I guess I'll manually accept the ssh host key for now and can sync up with ianw later today on sshfp usage | 16:31 |
| clarkb | thats done | 16:31 |
| fungi | this was our last discussion about it in here: http://eavesdrop.openstack.org/irclogs/%23opendev/%23opendev.2020-07-14.log.html | 16:33 |
| fungi | and then a few days later we covered that as soon as you update to glibc 2.31 your local resolution will stop trusting verified assertions from your configured recursive resolver, and so dnssec will be broken for you unless you override it with the trust-ad option in your /etc/resolv.conf | 16:38 |
| fungi | that's a fun one | 16:38 |
| *** lpetrut has joined #opendev | 16:46 | |
| *** dtantsur is now known as dtantsur|afk | 16:47 | |
| clarkb | https://review.opendev.org/#/c/749766/ is still happy with newer setuptools | 17:06 |
| clarkb | can probably land that one and then once nb03.opendev.org is sorted we can do the image side change | 17:06 |
| clarkb | less than a minute to the nb03.opendev.org change merging | 17:21 |
| openstackgerrit | Merged opendev/system-config master: Add nb03.opendev.org https://review.opendev.org/750039 | 17:25 |
| *** hashar has quit IRC | 17:27 | |
| clarkb | thats enqueued a ton of things (I think due to the inventory file changing) we'll have to wait for it to get to nodepool | 17:29 |
| *** ysandeep is now known as ysandeep|away | 17:30 | |
| *** lpetrut has quit IRC | 17:51 | |
| clarkb | is there a way to tell ansible to summarize the failure in its logging? bsae failed and I'm having a hard time finding what actually broke | 18:05 |
| clarkb | ok I think its because nb03.opendev.org was unreachable | 18:06 |
| clarkb | but I can ssh as root to it from bridge | 18:06 |
| clarkb | ok it is a hostkey issue | 18:07 |
| clarkb | I think because we've reused the ip address in that cloud? sshing to the name works but ansible actually sshes to the ip | 18:08 |
| clarkb | I removed the old ip specific ssh host key in known_hosts and added in the new one. | 18:11 |
| clarkb | I think that service-nodepool will be happy when it gets there | 18:11 |
| clarkb | oh except that failure kicked out the run for the change? the hourly run for nodepool has just started but not sure if I updated known hosts quickly enough | 18:11 |
| clarkb | it seems to be doing stuff so maybe I got it early enough | 18:13 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Add dev packages on arm64 for docker-compose installation https://review.opendev.org/750472 | 18:28 |
| clarkb | infra-root ^ I think that is the next thing we need for nb03.opendev.org | 18:28 |
| clarkb | I guess we can test this if we set up an arm64 only job. I'll look into that after our meeting too but I need to get ready for that now | 18:29 |
| *** hashar has joined #opendev | 18:43 | |
| *** priteau has quit IRC | 19:20 | |
| ianw | https://tools.ietf.org/html/rfc6104 is another one of these ipv6 rfc's that spends a long time basically telling you why a bunch of things don't work | 19:43 |
| ianw | the multi-homed host one is similar | 19:43 |
| ianw | it really seems that "manual configuration" is about the only answer | 19:44 |
| ianw | possibly firewalling, but as the rfc points out, can be unreliable if upstream routers change mac addresses, etc. | 19:44 |
| clarkb | https://etherpad.opendev.org/p/vexxhost-mirror-netplan found the netplan investigating I did | 19:49 |
| clarkb | I think one question about ^ I had outstanding was we actually have multiple valid ipv6 gateways on that host right now and I wasn't sure how to express that with netplan | 19:49 |
| clarkb | I think we'll be ok using a single gateway as long as it doesn't have an outage | 19:49 |
| clarkb | mnaser: ^ can you confirm that? if it is an issue to use a single gateway we can probably dig into netplan further | 19:52 |
| fungi | i think the idea is that ipv6 provides route redundancy so you can rely on that instead of vrrp/hsrp/carp for your gateways and not need to worry about careful vip migrations during router maintenances | 19:53 |
| clarkb | oh that was the other failsafe, we should still get working ipv4 with the netplan stuff I put up there as we aren't touching that | 19:59 |
| clarkb | so worst case ipv6 will not work and we can undo ? maybe we should disable vexxhost nowish in nodepool then we can iterate on that? | 19:59 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Add dev packages on arm64 for docker-compose installation https://review.opendev.org/750472 | 20:01 |
| ianw | clarkb: https://review.opendev.org/#/c/749604/ -- are you happy that localhost == bridge in all contexts so we can merge that? | 20:02 |
| clarkb | ianw: yes I think so | 20:02 |
| clarkb | I couldn't come up with a counter example | 20:02 |
| clarkb | infra-root I had bad quoting in https://review.opendev.org/750472 which should be fixed now | 20:02 |
| fungi | something i would have brought up during the meeting if we'd had more time, for generating historical mailing list usage statistics it would help to publish a list of the mailman sites we host and the public archives we have in pipermail for them (whether or not the lists themselves still exist or were retired). i have a 7-line shell script which generates this: but am wondering what the best mechanism would | 20:04 |
| fungi | be to orchestrate it (ansible via our deploy jobs? cron entry in the existing puppet-mailman module?) http://lists.opendev.org/archives.yaml | 20:04 |
| openstackgerrit | Clark Boylan proposed openstack/project-config master: Disable vexxhost for mirror work https://review.opendev.org/750484 | 20:04 |
| ianw | clarkb: ^ we have the same thing in the install ansible roles too. perhaps we should just move it into base for arm64 nodes? | 20:04 |
| clarkb | mnaser: ^ fyi that will impact your vexxhots specific flavors | 20:04 |
| clarkb | ianw: looking | 20:04 |
| mnaser | clarkb: sorry, been a little all over the place | 20:05 |
| mnaser | the single gateway _should_ be ok | 20:05 |
| clarkb | ianw: I think install-ansible only runs on bridge though | 20:05 |
| clarkb | ianw: thats there for the test jobs that runs an arm64 bridge iirc and won't apply to the nodepool-builder? | 20:06 |
| ianw | clarkb: yeah, that's right, i'm saying maybe we should make it apply to all aarch64 servers :) | 20:06 |
| clarkb | ianw: pull it out of install-ansible into base maybe? | 20:09 |
| ianw | clarkb: it's an option, anyway ... but given that we try to do everything in containers, having docker-compose installed probably means we've covered most things pip installs anyway | 20:10 |
| clarkb | ya | 20:10 |
| clarkb | also looking at base the way its currently split up there isn't a clear place to put that without making a new role which seems heavyweight for this | 20:10 |
| clarkb | https://netplan.io/reference/ is the netplan docs. We may have to set up manual routes to use multiple gateways | 20:32 |
| clarkb | there is a v4 example showing that and I expect converting to ipv6 would work | 20:33 |
| clarkb | I'll edit the etherpad with that | 20:33 |
| *** hashar has quit IRC | 20:38 | |
| clarkb | doesn't seem like I can say it is the default route though? | 20:39 |
| clarkb | does that mean I should set the metric higher so that the other routes will win over the catch all :: dest there? | 20:39 |
| clarkb | maybe we should do it with the gateway6 option to keep it simple | 20:39 |
| *** andrewbonney has quit IRC | 20:39 | |
| openstackgerrit | Merged opendev/system-config master: install-ansible: move install_modules.sh to puppet-setup-ansible https://review.opendev.org/749604 | 20:42 |
| openstackgerrit | Merged opendev/system-config master: launch: move old scripts out of top-level https://review.opendev.org/749605 | 20:42 |
| openstackgerrit | Merged opendev/system-config master: Update README.rst https://review.opendev.org/749615 | 20:42 |
| openstackgerrit | Merged opendev/system-config master: docs: Update some of sysadmin details https://review.opendev.org/749630 | 20:42 |
| fungi | clarkb: yeah, "default" is usually just an alias for the wildcard/full cidr route | 20:58 |
| fungi | so creating a route for :: should be handled the same way | 20:58 |
| clarkb | ah ok in that case I think the etherpad is correct if people want to review it and if that looks good we can land https://review.opendev.org/750484 and try it | 20:58 |
| fungi | though as you noted, ipv6 also has route priorities, those are taken into account after shortest prefix | 20:58 |
| clarkb | I tried to match what is on the mirror already | 20:59 |
| fungi | er, longest prefix | 20:59 |
| fungi | if you have a route for a::/8 and a route for a:b::/16 then the second is preferred for destinations where it applies, even if the first has a higher priority | 21:00 |
| clarkb | got it | 21:00 |
| fungi | consider it like other traditional routing protocols. most specific match/longest prefix is preferred, followed by your path length and local priority rules | 21:07 |
| clarkb | fungi: does that netplan config in the etherpad look good to you then? | 21:09 |
| clarkb | I mean as much as we grok netplan :) | 21:09 |
| fungi | i'm very green to netplan as well (did we really need *yet another* way to configure networking?!?), but will take a look | 21:13 |
| fungi | i guess interface hardware addresses shouldn't change just from something like live migration right? | 21:14 |
| fungi | we'll probably have to adjust that any time we replace the instance though | 21:15 |
| fungi | granted, we'd need to fix the static v6 address when that happened too | 21:16 |
| clarkb | yes and yes | 21:16 |
| fungi | yeah, i think what's there looks reasonable | 21:17 |
| clarkb | neutron fixes the l2 address with the port and that follows the host on migration | 21:17 |
| clarkb | also I was sort of thinking we'd manually apply that for now | 21:17 |
| fungi | yep | 21:17 |
| fungi | keeping in mind that this is a workaround until the neutron bug or whatever it is can be identified and eliminated in that provider | 21:18 |
| fungi | so hopefully not something we have to keep in place for years | 21:18 |
| clarkb | I probably won't touch that until nb03 is happy (fix for that is in the gate) | 21:22 |
| clarkb | but I'm happy to help drive the netplan change along too | 21:23 |
| corvus | fungi: you have a lot of notes on the opendev.org change, along with a +2 | 21:29 |
| corvus | fungi: how do you want to proceed? | 21:29 |
| fungi | corvus: i can push those notes as a followup change, feel free to go ahead and approve | 21:30 |
| fungi | adding to my to do list for tomorrow | 21:30 |
| corvus | k. seems like iterate forward is good here | 21:30 |
| openstackgerrit | Merged opendev/system-config master: Add dev packages on arm64 for docker-compose installation https://review.opendev.org/750472 | 22:09 |
| openstackgerrit | Mohammed Naser proposed opendev/system-config master: Add ceph octopus mirrors https://review.opendev.org/750519 | 22:35 |
| mnaser | ^ hopefully i didnt miss something there | 22:35 |
| *** auristor has quit IRC | 22:38 | |
| *** auristor has joined #opendev | 22:41 | |
| clarkb | 2020-09-08 22:53:29,838 INFO nodepool.builder.BuildWorker.0: Building image ubuntu-xenial-arm64 | 22:54 |
| clarkb | that is what nb03.opendev.org says :) | 22:54 |
| ianw | wooo! thanks clarkb | 22:54 |
| *** tosky has quit IRC | 22:55 | |
| clarkb | that took longer than I had hoped, but ya super excited that we've now got a fully multi arch container pipeline that we consume ourselves | 22:55 |
| openstackgerrit | Merged opendev/system-config master: Explain "why opendev" on opendev.org index page https://review.opendev.org/748263 | 22:56 |
| *** tkajinam has joined #opendev | 22:57 | |
| *** tkajinam has quit IRC | 22:57 | |
| *** tkajinam has joined #opendev | 22:58 | |
| *** mlavalle has quit IRC | 22:58 | |
| clarkb | nb03's image build is pulling in git repos now, so far looks like I would expect it | 23:00 |
| clarkb | hrm apache2 failed to restart on nb03.opendev.org (which is how we load in the LE certs?) | 23:10 |
| clarkb | looking into that now. The nodepool-builder side of things looks finethough | 23:10 |
| clarkb | Sep 08 22:53:29 nb03 apachectl[32731]: SSLCertificateFile: file '/etc/letsencrypt-certs/nb03.opendev.org/nb03.opendev.org.cer' does not exist or is empty | 23:11 |
| clarkb | ianw: I think the issue is that the LE playbook hasn't run against nb03.opendev.org | 23:12 |
| clarkb | possibly due to the earlier ssh key issue? | 23:12 |
| clarkb | we run that job in periodic. I'm inclined to just let it run tonight (my time) and have it correct itself | 23:14 |
| clarkb | ianw: ^ any concerns with that? | 23:14 |
| clarkb | mostly asking you as I think you may be around when that runs? if not thats fine too and I'll check it tomorrow morning | 23:17 |
| clarkb | error: invalid command 'bdist_wheel' <- we got that from the image build trying to intsall os-testr in a venv in the image | 23:34 |
| clarkb | it used setuptools 49.6.0 and pip 8.someting (xenials pip) | 23:35 |
| clarkb | is that familiar to anyone? | 23:35 |
| clarkb | do we need to install the python-wheel package in the image first so that its present when making the venv? | 23:36 |
| clarkb | note this is all within the new VM image chroot context I don't think the nodepool-builder docker image makes a difference here | 23:36 |
| clarkb | I think it must not show up on x86 because there are x86 wheelsfor those packages already? | 23:36 |
| clarkb | future, voluptious, pyyaml, pyperclip, and prettytable | 23:37 |
| clarkb | hrm no thoes pacakges only have sdists (or at least the first two are sdist only) | 23:37 |
| clarkb | ya the same error occurs on nb03.openstack.org so not a docker regression | 23:39 |
| clarkb | ianw: I think we fix this by adding python3-wheel to ensure-venv's package list in dib? | 23:41 |
| ianw | clarkb: hrm, you can probably just manually run the LE playbook to get the certs | 23:43 |
| ianw | or i can | 23:43 |
| ianw | i'm pretty sure python3-wheel is in ensure-venv | 23:44 |
| clarkb | ok that wheel error happens on x86 too I think its just noise | 23:44 |
| ianw | this is *in the chroot* though? | 23:44 |
| clarkb | yes | 23:44 |
| clarkb | 2020-09-08 23:33:18.951 | DEBUG diskimage_builder.block_device.utils [-] exec_sudo: sudo: sgdisk: command not found exec_sudo /usr/local/lib/python3.7/site-packages/diskimage_builder/block_device/utils.py:135 <- that is the actual reason the build fails | 23:44 |
| clarkb | the os-testr installation failures don't seem to cause the builds to fail normally (we should still fix that and I think python3-wheel in the ensure-venv element is how we can do that) | 23:45 |
| clarkb | sgdisk will be a nodepool-builder imgae issue though I expect | 23:45 |
| clarkb | we only hit the sgdisk path on the uefi and gpt partitioning branch | 23:46 |
| clarkb | which we only do for arm64 is my current theory | 23:46 |
| fungi | venvs don't get wheel added to them by default, you have to explicitly install wheel into them if you want to do bdist_wheel | 23:46 |
| fungi | and pip will fall back to directly installing files instead of its default of making wheels and installing those | 23:47 |
| clarkb | fungi: this is with venv not virtualenv on ubuntu (so has debians split up packaging) | 23:47 |
| clarkb | well pip isn't falling back there | 23:47 |
| fungi | mm, may also depend on the package being installed | 23:47 |
| clarkb | anyway the sgdisk issue is the real cause of the failure I think so trying to figure out how to intsall that on the container image now | 23:48 |
| clarkb | its the gdisk package looks like | 23:48 |
| clarkb | fungi: if I create a virtualenv using virtualenv 20.something locally it has wheel in it | 23:53 |
| ianw | clarkb: ahhh, yes we will need that when making gpt partitions | 23:53 |
| ianw | i actually hit that trying the x86 efi testing and didn't fix it :/ | 23:53 |
| clarkb | if I create it with python3 -m venv I don't get a wheel | 23:53 |
| fungi | yep, that's what i was remembering | 23:54 |
| clarkb | ianw: no worries, change is prposed to nodepool now | 23:54 |
| ianw | sorry, i'm getting periodic internet dropouts here :/ | 23:55 |
| clarkb | I'm now installing python3-wheel to see if venv behavior changes but this is on tumbleweed not ubuntu | 23:56 |
| clarkb | still no wheel | 23:57 |
| ianw | clarkb: did you just hand-install sgdisk to see if that works too? | 23:57 |
| ianw | (in the container) | 23:57 |
| clarkb | ianw: I did not. Should I? | 23:57 |
| ianw | it might be worth it to see if any other issue lurks quickly | 23:58 |
| ianw | if you have a window open | 23:58 |
| clarkb | ok I'll do that now, you may want to check logs later today to see if there are new issues | 23:58 |
| ianw | ok, i hope not but you never know :) | 23:59 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!