| ianw | ... ok, nobody (mirror-update, an executor, afs01.dfw) sees any of the mirror volumes in vldb. i think that means we're back to consistency at least | 00:07 |
|---|---|---|
| ianw | https://www.openafs.org/frameset/dl/openafs/1.6.16/ChangeLog ... | 00:10 |
| ianw | This avoids writing a loopback address into the CellServDB on systems | 00:11 |
| ianw | such as Debian, which map the address 127.0.1.1 to the hostname in the | 00:11 |
| ianw | /etc/hosts file. | 00:11 |
| ianw | afs01 runs 1.16.15 | 00:11 |
| fungi | seems like that would have been worth debian/ubuntu backporting | 00:16 |
| fungi | did you get rid of the centos-7 x86 volume too? | 00:17 |
| ianw | fungi ... <insert expletive> | 00:18 |
| ianw | yes | 00:18 |
| ianw | so, "vos create" on afs01 gets it *wrong* ... http://paste.openstack.org/show/794606/ | 00:18 |
| fungi | just did an ls and only saw the xenial/bionic x86 volumes so thought you might have | 00:18 |
| clarkb | ianw: that is without a netrestrict though right? | 00:19 |
| ianw | that is without netrestrict, just "as is" | 00:19 |
| ianw | so, at least that's the root cause here i guess | 00:19 |
| ianw | i feel like doing it on mirror-update say, it would work | 00:19 |
| fungi | and that's without altering /etc/hosts on afs01.dfw i guess | 00:20 |
| clarkb | well fungi's got it wrong which wsa 1.8.x | 00:20 |
| clarkb | the first time around | 00:20 |
| ianw | creating it on mirror-update seems to work ... http://paste.openstack.org/show/794607/ | 00:21 |
| ianw | let me add the r/o mirrors | 00:21 |
| ianw | fungi: do you see the centos 7 x86-64 volume back? | 00:25 |
| fungi | response is taking a while for me | 00:27 |
| fungi | the rw volume is back, yeah | 00:27 |
| fungi | no ro replica yet | 00:27 |
| ianw | fungi: i see http://paste.openstack.org/show/794608/ | 00:28 |
| fungi | if i `ls /afs/openstack.org/mirror/wheel` from my workstation i only see the xenial and bionic replicas | 00:30 |
| ianw | fungi: can you run "fs checkvolumes" ? | 00:31 |
| fungi | that probably did it, taking a while to ls now | 00:31 |
| ianw | auristor explained it once here http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2019-12-04.log.html#t2019-12-04T01:52:57 | 00:32 |
| clarkb | so when you recreated it it had all of the old data again? | 00:35 |
| clarkb | rm_work: we've been asked if you can share the email addr you used | 00:36 |
| ianw | clarkb: no, i've managed to wipe that :( | 00:36 |
| rm_work | sure will PM | 00:37 |
| clarkb | ianw: ah ok that is what I thought but wanted to confirm | 00:37 |
| ianw | i think we've established that creating the volumes on mirror-update works | 00:37 |
| fungi | my ls is still... waiting | 00:37 |
| fungi | but i expect it to complete eventually | 00:37 |
| ianw | eventually consistent might be the best term for this. like a week later it might be consistent :) | 00:38 |
| fungi | ugh | 00:39 |
| fungi | ls: cannot access '/afs/openstack.org/mirror/wheel': Input/output error | 00:39 |
| fungi | trying again now | 00:39 |
| ianw | huh, that's the top level (rather than saying centos-7-x86_64 was inaccessible) | 00:40 |
| fungi | ls: cannot access '/afs/openstack.org/mirror/wheel': Connection timed out | 00:42 |
| fungi | eventually inconsistent? ;) | 00:42 |
| fungi | i wonder if i still have some old 127.0.1.1 entries cached | 00:42 |
| fungi | trying fs checkservers | 00:43 |
| ianw | fungi: what does examine say on mirror.wheel.cent7x86? | 00:43 |
| ianw | sorry mirror.wheel.cent7x64 | 00:43 |
| fungi | http://paste.openstack.org/show/794609/ | 00:44 |
| fungi | `fs checkservers` says "These servers unavailable due to network or server problems: dhole.yuggoth.org afs01.dfw.openstack.org." | 00:45 |
| fungi | that first one is my workstation itself where i ran the command, it shouldn't be a server | 00:45 |
| ianw | yeah, that looks like the same issue coming back ... | 00:46 |
| fungi | but it's what resolves to 127.0.1.1 locally (since it's a debian machine and that's a default /etc/hosts entry) | 00:46 |
| clarkb | I've got to figure out dinner now | 00:46 |
| ianw | but your examine on the new volume looks correct (no 127 entries) | 00:46 |
| ianw | fungi, if you do listvoldb do you see any old entries? | 00:47 |
| fungi | listvldb i assume | 00:47 |
| ianw | ah yeah | 00:47 |
| fungi | it only shows me the rw volume for centos7x64: http://paste.openstack.org/show/794610/ | 00:49 |
| ianw | oh, i think that's a red-herring and an old volume | 00:52 |
| ianw | Creation Wed Mar 23 15:07:15 2016 | 00:52 |
| ianw | yep, that's i think from when we first tried this. i'll remove that volume to avoid confusion | 00:52 |
| ianw | it's the wheel.mirror. prefixed ones | 00:53 |
| fungi | ahh, yeah | 00:53 |
| fungi | yeah, that looks normal, i think: http://paste.openstack.org/show/794611/ | 00:54 |
| ianw | that seems right. i'm going to recreate the other volumes in the same way | 00:56 |
| fungi | my inability to ls the ro fs could be some local problem | 01:00 |
| fungi | hey! it just now worked | 01:00 |
| *** Meiyan has joined #opendev | 01:01 | |
| ianw | http://mirror.iad.rax.opendev.org/wheel/ looks like it's working | 01:08 |
| ianw | acls and quotas are setup | 01:09 |
| ianw | i think i can trigger the periodic builds to try it | 01:09 |
| aannuusshhkkaa | It's working! I was able to register my account.. Thanks guys! | 01:12 |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: Don't release centos7 aarch64 volume https://review.opendev.org/735025 | 01:13 |
| ianw | ok some builds are running, i don't think automated release will work due to ^ but it's a start ... | 01:14 |
| openstackgerrit | Merged openstack/project-config master: Don't release centos7 aarch64 volume https://review.opendev.org/735025 | 01:33 |
| *** xiaolin has joined #opendev | 01:39 | |
| openstackgerrit | Prakash Ramchandran proposed opendev/irc-meetings master: updating irc channel to openstack-interop https://review.opendev.org/735034 | 01:57 |
| openstackgerrit | sebastian marcet proposed opendev/puppet-openstackid master: Upgraded memory limit to 64mb https://review.opendev.org/735036 | 02:08 |
| *** xiaolin has quit IRC | 02:28 | |
| ianw | fungi: centos7 build finished so i did a manual release and it's back https://mirror.iad.rax.opendev.org/wheel/centos-7-x86_64/ | 02:37 |
| *** xiaolin has joined #opendev | 03:31 | |
| ianw | so i doubt the arm builds are going to get there | 03:53 |
| ianw | 102:1591838794.000 1469.000035999400build_env/bin/pip --verbose --exists-action=i wheel -c /tmp/upper-constraints.txt -w /opt/wheel/workspace 'scipy===1.4.1' | 03:53 |
| ianw | 561:1591839516.000 1941.0000145841200build_env/bin/pip --verbose --exists-action=i wheel -c /tmp/upper-constraints.txt -w /opt/wheel/workspace 'scikit-learn===0.23.1 | 03:53 |
| ianw | the third column is build time(s) | 03:54 |
| ianw | an hour or so on these two ... | 03:54 |
| ianw | they're not building in parallel (from pip wheel i mean) ... it's uncear if their setup supports a "-j" type thing | 03:55 |
| ianw | hrw/fungi: ^ | 03:56 |
| openstackgerrit | Tim Burke proposed opendev/bindep master: Tolerate missing bindep.txt when using --profiles https://review.opendev.org/734719 | 04:00 |
| *** ykarel|away is now known as ykarel | 04:37 | |
| openstackgerrit | Merged openstack/project-config master: Add openstack/magnum and openstack/magnum-tempest-plugin in required-projects https://review.opendev.org/734863 | 04:47 |
| openstackgerrit | Merged openstack/project-config master: gerritbot: more notifications in the oslo channel https://review.opendev.org/734827 | 04:59 |
| *** sgw has quit IRC | 05:04 | |
| openstackgerrit | Merged openstack/project-config master: gerritbot: more notifications in the cinder channel https://review.opendev.org/734792 | 05:07 |
| *** ysandeep|away is now known as ysandeep | 05:21 | |
| *** ysandeep is now known as ysandeep|brb | 05:38 | |
| *** ysandeep|brb is now known as ysandeep | 06:04 | |
| ianw | hrw: one to keep an eye on -- https://review.opendev.org/735054 wheel-build-cache: skip scipy / scikit-learn on arm64 for now | 06:07 |
| ianw | some thoughts; i *think* these setup.py files might support '-j' but again i think we'd need to hack the builder script to understand what wheels understand that argument and what don't | 06:08 |
| ianw | although we run in parallel, each branch runs sequentially; so we do have to add the longest build for each branch together to get the total runtime. | 06:09 |
| ianw | it may well be possible to rewrite the script to also work on things across branches in parallel. when it timed out, it was sitting there with one cpu pegged and otherwise doing nothing | 06:09 |
| ianw | make had this all sorted 40 years ago ... but ... yay python packaging evironment i guess | 06:10 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: openafs-client: Use PPA for all Ubuntu https://review.opendev.org/735055 | 06:17 |
| ianw | i've just run a manual release, so happily *all* the x86_64 wheel caches are now populated (http://mirror.iad.rax.opendev.org/wheel/) | 06:22 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: AFS: add note on volume creation servers https://review.opendev.org/735061 | 06:38 |
| AJaeger | hrw: ^ | 06:44 |
| AJaeger | hrw: sorry, ianw pinged you already earlier, ignore my ping, please | 06:45 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Remove virtualenv activation https://review.opendev.org/734414 | 06:49 |
| *** hashar has joined #opendev | 07:14 | |
| *** rpittau|afk is now known as rpittau | 07:18 | |
| *** sshnaidm|afk is now known as sshnaidm | 07:23 | |
| *** tosky has joined #opendev | 07:38 | |
| *** factor has quit IRC | 07:46 | |
| *** factor has joined #opendev | 07:47 | |
| *** ykarel is now known as ykarel|lunch | 07:53 | |
| xiaolin | frickler: mordred: fungi: I review the chat log, thanks. As we are not already cloud operators, we prefer to donate compute resources. | 07:55 |
| *** moppy has quit IRC | 08:01 | |
| *** moppy has joined #opendev | 08:02 | |
| *** jaicaa has quit IRC | 08:13 | |
| *** jaicaa has joined #opendev | 08:16 | |
| *** DSpider has joined #opendev | 08:31 | |
| openstackgerrit | Sagi Shnaidman proposed zuul/zuul-jobs master: Add jobs for testing ensure-ansible https://review.opendev.org/734584 | 08:39 |
| *** ykarel|lunch is now known as ykarel | 08:56 | |
| *** rpittau is now known as rpittau|bbl | 09:15 | |
| *** factor has quit IRC | 09:16 | |
| *** factor has joined #opendev | 09:16 | |
| openstackgerrit | Merged opendev/irc-meetings master: updating irc channel to openstack-interop https://review.opendev.org/735034 | 09:48 |
| ianw | AJaeger/hrw : still times out :( going to try some other things with https://review.opendev.org/735092 [wip] reduce arm64 builds, collect logs after | 09:51 |
| *** Meiyan has quit IRC | 09:52 | |
| openstackgerrit | Sagi Shnaidman proposed zuul/zuul-jobs master: Add jobs for testing ensure-ansible https://review.opendev.org/734584 | 10:20 |
| *** nautics889 has joined #opendev | 10:29 | |
| fungi | xiaolin: makes sense. we're also not cloud operators, so we'd have to find a cloud operator willing to run those donated resources for us, unless you already have a relationship with one | 11:52 |
| *** rpittau|bbl is now known as rpittau | 11:54 | |
| fungi | ianw: as discussed before, the arm wheel jobs are going to have to build a *lot* more wheels, because our equivalent x86 builds take advantage of the fact that there are a lot of prebuilt manylinux1 wheels for that architecture already served from pypi | 11:54 |
| mordred | mnaser: ^^ do you think vexxhost would have any interest in working with xiaolin? | 12:14 |
| *** tkajinam has quit IRC | 12:25 | |
| *** mnaser has quit IRC | 12:32 | |
| *** mnaser has joined #opendev | 12:33 | |
| *** ykarel is now known as ykarel|afk | 13:00 | |
| mnaser | mordred: yeah — xiaolin feel free to ping me or email at mnaser@vexxhost.com | 13:06 |
| xiaolin | thanks, I will ping you tomorrow | 13:08 |
| *** owalsh_ is now known as owalsh | 13:20 | |
| *** hashar has quit IRC | 13:28 | |
| openstackgerrit | Carlos Goncalves proposed openstack/diskimage-builder master: Install yum-utils on Red Hat family hosts https://review.opendev.org/735136 | 13:41 |
| *** sgw has joined #opendev | 13:43 | |
| *** olaph has joined #opendev | 13:47 | |
| *** ykarel|afk is now known as ykarel | 13:57 | |
| openstackgerrit | Joshua Hesketh proposed opendev/bindep master: Allow underscores in profile names https://review.opendev.org/735142 | 14:08 |
| openstackgerrit | Joshua Hesketh proposed opendev/bindep master: Allow underscores in profile names https://review.opendev.org/735142 | 14:16 |
| openstackgerrit | Joshua Hesketh proposed opendev/bindep master: Allow uppercase letters in profiles https://review.opendev.org/735143 | 14:16 |
| *** iurygregory_ has joined #opendev | 14:19 | |
| *** iurygregory has quit IRC | 14:19 | |
| *** iurygregory_ is now known as iurygregory | 14:19 | |
| openstackgerrit | Sagi Shnaidman proposed zuul/zuul-jobs master: Add jobs for testing ensure-ansible https://review.opendev.org/734584 | 14:29 |
| *** mlavalle has joined #opendev | 14:31 | |
| *** _mlavalle_1 has joined #opendev | 14:34 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Add linting rule to enforce no-same-owner policy https://review.opendev.org/727642 | 14:35 |
| *** mlavalle has quit IRC | 14:37 | |
| *** _mlavalle_1 has quit IRC | 14:37 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Add linting rule to enforce no-same-owner policy https://review.opendev.org/727642 | 14:39 |
| openstackgerrit | Sagi Shnaidman proposed zuul/zuul-jobs master: Add jobs for testing ensure-ansible https://review.opendev.org/734584 | 14:51 |
| *** factor has quit IRC | 14:52 | |
| openstackgerrit | sebastian marcet proposed opendev/system-config master: OpenstackId v3.0.12 https://review.opendev.org/735022 | 14:54 |
| openstackgerrit | Carlos Goncalves proposed openstack/diskimage-builder master: Install yum and yum-utils on Red Hat family hosts https://review.opendev.org/735136 | 15:00 |
| openstackgerrit | Joshua Hesketh proposed opendev/bindep master: Allow underscores in profile names https://review.opendev.org/735142 | 15:07 |
| openstackgerrit | Joshua Hesketh proposed opendev/bindep master: Allow uppercase letters in profiles https://review.opendev.org/735143 | 15:07 |
| openstackgerrit | Joshua Hesketh proposed opendev/bindep master: Ensure virtualenv is installed for all hosts https://review.opendev.org/735167 | 15:07 |
| *** mlavalle has joined #opendev | 15:15 | |
| *** sshnaidm_ has joined #opendev | 15:15 | |
| *** sshnaidm has quit IRC | 15:16 | |
| *** hashar has joined #opendev | 15:23 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Terraform roles and jobs. https://review.opendev.org/733675 | 15:26 |
| *** ykarel is now known as ykarel|away | 15:27 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Terraform roles and jobs. https://review.opendev.org/733675 | 15:28 |
| openstackgerrit | Merged opendev/system-config master: OpenstackId v3.0.12 https://review.opendev.org/735022 | 15:36 |
| *** panda has quit IRC | 15:51 | |
| *** panda has joined #opendev | 15:53 | |
| *** rpittau is now known as rpittau|afk | 15:58 | |
| *** ysandeep is now known as ysandeep|away | 15:59 | |
| *** iurygregory has quit IRC | 16:15 | |
| *** nautics889 has quit IRC | 16:33 | |
| *** nautics889 has joined #opendev | 16:35 | |
| *** sshnaidm_ is now known as sshnaidm | 16:46 | |
| *** sshnaidm is now known as sshnaidm|afk | 16:46 | |
| clarkb | I've sorted out the etherpad problem | 17:11 |
| clarkb | 104.239.168.111 as etherpad.opendev.org is running a local build (to that test node) of etherpad 1.8.4 with the headings plugin and my fix. I need to do PR to github now | 17:11 |
| fungi | https://etherpad.opendev.org/p/clarkb-test looks right to me now | 17:14 |
| fungi | with "104.239.168.111 etherpad.opendev.org" in my /etc/hosts | 17:14 |
| clarkb | wow ok so develop has already changed that section of css | 17:15 |
| clarkb | so my fix is no longer valid, but maybe they fixed this some other way | 17:15 |
| clarkb | now I'm trying to figureo ut if they publish pre release images | 17:16 |
| clarkb | looks like they do so I'll deploy that now | 17:16 |
| clarkb | ok its running etherpad/etherpad:latest now and that does seem to fix it differently | 17:19 |
| clarkb | I'll update the bug with all of that and note that the next release will probably be fine | 17:19 |
| clarkb | https://github.com/ether/etherpad-lite/issues/4106 has been updated | 17:23 |
| clarkb | I think we can update the span css on 1.8.4 if we want to while waiting for 1.8.5 | 17:23 |
| clarkb | I'll go ahead and push that update as it gives us the ability to deploy. I also asked upstream if they think that will cause any problems as an interim solution for us | 17:24 |
| mordred | ++ | 17:25 |
| * mordred will now eat sandwich | 17:26 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Test etherpad with testinfra https://review.opendev.org/734880 | 17:27 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Upgrade Etherpad to 1.8.4 https://review.opendev.org/729029 | 17:27 |
| *** hashar has quit IRC | 17:41 | |
| clarkb | fungi: I'm going to transition to renaming prep shortly. I never ended up doing breakfast so will do that first. You don't happen to know where the plan ehterpad is from last time? I think we've simplified the process even further now with the github replication removal too | 17:42 |
| *** nautics889 has quit IRC | 17:44 | |
| fungi | i don't, no... i can likely find it though, just a sec | 17:44 |
| fungi | the last one we did was on 2020-03-20 | 17:45 |
| fungi | according to the status log anyway | 17:46 |
| fungi | and the irc logs from that day mention https://etherpad.opendev.org/p/gerrit-2020-03-20 | 17:46 |
| *** nautics889 has joined #opendev | 17:46 | |
| fungi | i guess we mashed the renames and containerization cut-over into one window | 17:47 |
| fungi | so that may be a terrible example | 17:47 |
| fungi | though i suppose we can just ignore most of what's in there | 17:47 |
| auristor | after yesterday's afs adventures with fileserver address 172.0.1.1 the VLDB was left with a damaged fileserver entry that OpenAFS tooling (vos listaddr) won't display. I see it with the AuriStorFS tooling (vos listfs). Someone should try "vos changeaddr 127.0.1.1 -remove -localauth" and see if it can be safely deleted. | 17:48 |
| fungi | auristor: Removed server 127.0.1.1 from the VLDB | 17:49 |
| fungi | thanks for the heads up! | 17:49 |
| auristor | removal confirmed | 17:49 |
| fungi | #status log cleaned up dangling fileserver entry with `vos changeaddr 127.0.1.1 -remove -localauth` as suggested by auristor | 17:49 |
| openstackstatus | fungi: finished logging | 17:49 |
| *** mlavalle has quit IRC | 17:50 | |
| openstackgerrit | Drew Walters proposed openstack/project-config master: Add missing project to Airship doc job https://review.opendev.org/734874 | 18:02 |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Update openSUSE Mirrors https://review.opendev.org/735207 | 18:08 |
| *** factor has joined #opendev | 18:08 | |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Update openSUSE Mirrors for configure-os-mirrors https://review.opendev.org/735207 | 18:09 |
| *** dtroyer has quit IRC | 18:12 | |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Update openSUSE Mirrors for configure-os-mirrors https://review.opendev.org/735207 | 18:14 |
| *** jaicaa_ has joined #opendev | 18:22 | |
| *** jaicaa has quit IRC | 18:22 | |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: DNM: Debug openSUSE mirror loop https://review.opendev.org/735209 | 18:23 |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Update openSUSE Mirrors for configure-os-mirrors https://review.opendev.org/735207 | 18:35 |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: DNM: Debug openSUSE mirror loop https://review.opendev.org/735209 | 18:35 |
| openstackgerrit | Clark Boylan proposed opendev/project-config master: Record 20200612 project renames https://review.opendev.org/735211 | 18:46 |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Update openSUSE Mirrors for configure-os-mirrors https://review.opendev.org/735207 | 18:47 |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: DNM: Debug openSUSE mirror loop https://review.opendev.org/735209 | 18:47 |
| clarkb | https://etherpad.opendev.org/p/gerrit-project-renames-20200612 starting to build notes there for the renames | 18:47 |
| clarkb | I'm looking at the rename playbook now and it appears to be complete and updated for the new docker setup | 18:49 |
| clarkb | one thing that stands out to me is that we don't do anything other than move secret keys for zuul. I take it that once gerrit is done we can land the rename changes that update teh zuul config and everything will be happy after that? | 18:49 |
| clarkb | mordred: corvus ^ | 18:49 |
| corvus | clarkb: theoretically i think | 18:51 |
| mordred | clarkb: yes, I think that's right | 18:52 |
| mordred | infra-root: I'm booting a VM called review-test.opendev.org to start testing gerrit upgrades | 18:53 |
| clarkb | cool I think https://etherpad.opendev.org/p/gerrit-project-renames-20200612 may have all the bits it needs now if people can take a look at that and the linked changes | 18:54 |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Update openSUSE Mirrors for configure-os-mirrors https://review.opendev.org/735207 | 18:55 |
| corvus | i'm going to re-enqueue the latest zuul release to check on the new dockerhub job | 18:57 |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Update openSUSE Mirrors for configure-os-mirrors https://review.opendev.org/735207 | 19:12 |
| jrosser | hi, question about the arm64 nodes... would you expect that they would be setup the same as the equivalent x86 nodes in terms of which pip/setuptools/... are installed? | 19:14 |
| jrosser | i'm seeing some odd stuff and thought i'd ask first before digging | 19:15 |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Update openSUSE Mirrors for configure-os-mirrors https://review.opendev.org/735207 | 19:15 |
| clarkb | jrosser: they've already been converted to the "cleaner" install | 19:17 |
| clarkb | jrosser: we should be converting the x86 nodes over any day now | 19:17 |
| AJaeger | jrosser: https://review.opendev.org/#/c/734428/ is the change to make them the same ;) | 19:17 |
| jrosser | oh interesting, that would explain it | 19:17 |
| jrosser | it's our little bit of pre-run cleanup that fails so yes i understand now | 19:18 |
| AJaeger | jrosser: 734428 has a link to the mail with more details | 19:18 |
| clarkb | and we said we'd merge it like ~3 days ago so I'm hoping that it really does land real soon now :) | 19:18 |
| AJaeger | clarkb: it has two +2s, what are we waiting for? | 19:19 |
| clarkb | AJaeger: I think mostly for ianw to have some ability to help people debug once the images update? | 19:19 |
| clarkb | AJaeger: maybe we alnd that today as soon as ianw's day starts adn take it from there | 19:19 |
| AJaeger | would be ok for me - let's see what ianw thinks when he shows up | 19:20 |
| fungi | i'm around tomorrow and over the weekend to help with broken jobs too | 19:23 |
| clarkb | ya I don't mean ianw has to do it all, more that having someone with the knowledge around would be good :) | 19:24 |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Update openSUSE Mirrors for configure-os-mirrors https://review.opendev.org/735207 | 19:28 |
| *** dtroyer has joined #opendev | 19:40 | |
| *** mrunge has quit IRC | 19:57 | |
| mnaser | i hate that i'm asking this | 20:01 |
| mnaser | but is there a way to get around port 29418 being blocked to submit patches to gerrit | 20:02 |
| fungi | https | 20:03 |
| fungi | mnaser: you can push via https with your "web password" | 20:03 |
| fungi | (generate one in your preferences in the webui if you don't have one already) | 20:04 |
| fungi | i want to say we have this documented in the opendev manual too, looking now | 20:04 |
| fungi | mnaser: see https://docs.opendev.org/opendev/infra-manual/latest/developers.html#accessing-gerrit-over-https | 20:05 |
| fungi | and if that doesn't work, let us know so we can troubleshoot/fix/update docs | 20:06 |
| mnaser | fungi: ah perfect, i will try it now | 20:07 |
| mnaser | seems straightforward | 20:08 |
| fungi | i hope so. and if it's not, let's make it straightforward | 20:09 |
| *** mrunge has joined #opendev | 20:12 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Update guideline docs for os specific tasks https://review.opendev.org/731606 | 20:24 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Update guideline docs for os specific tasks https://review.opendev.org/731606 | 20:25 |
| clarkb | thinking about the renaming more. Is it sufficient to put review in the emergency file? Seems like we also want to avoid running things on gitea and storyboard to avoid recreating things that have been renamed? | 20:44 |
| clarkb | should we install disable things entirely touching that file in zuul's homedir? | 20:44 |
| fungi | yeah, i think that's what we did last time | 20:46 |
| clarkb | s/install/instead/ | 20:47 |
| fungi | it used to be that cgit stuff was dependent on the gerrit puppet completing | 20:47 |
| fungi | but not so with gitea | 20:47 |
| clarkb | k /me updates etherpad | 20:47 |
| fungi | and also in the beforetime we simply stopped our puppet apply wheel entirely | 20:48 |
| fungi | so this would be similar | 20:48 |
| clarkb | hrm there is actually another issue with that | 20:50 |
| clarkb | now that we use the actual commits we land in the zuul run ansible pausing isn't sufficient in all cases as we can just pick up the old content after unpausing | 20:51 |
| clarkb | or did we switch to always using latest project-config? that was a thing right? | 20:51 |
| clarkb | mordred: ^ | 20:51 |
| fungi | i believe we did that | 20:52 |
| clarkb | ah ok pausing may be sufficient then thanks | 20:52 |
| mordred | uh | 20:53 |
| mordred | yeah - I think we do use latest project-config always now | 20:53 |
| mordred | although I think we also fixed zuul so that it's possible to go back to patchset p-c | 20:53 |
| mordred | but we have not yet done that | 20:53 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Add bit more info on disabling ansible runs https://review.opendev.org/735246 | 20:55 |
| clarkb | mordred: I don't think we want each p-c change due to the race I described aboe | 20:55 |
| clarkb | mordred: basically if we do each p-c change in sequence we have to ensure that we've compltely flused the queue, set DISABLE-ANSIBLE, not merge anything else until the behind the scenes updates are complete | 20:56 |
| clarkb | then merge the matching content first | 20:56 |
| *** moppy has quit IRC | 20:57 | |
| clarkb | I mean its something we can work with, but not needing to worry about it is nice too :) | 20:57 |
| *** moppy has joined #opendev | 20:58 | |
| mordred | oh - yeah - good point | 20:59 |
| mordred | I think maybe p-c always using latest is right even with the fixed zuul thing | 20:59 |
| mordred | because p-c describes a point in time - not a sequence of software steps | 20:59 |
| mordred | it's kind of like the service-types for openstack - an old copy is NEVER correct in ANY case | 21:00 |
| mordred | because tip describes the whole pictures | 21:00 |
| mordred | (or the timezone database) | 21:00 |
| clarkb | ++ | 21:00 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Add test-build-container-image-release job https://review.opendev.org/735249 | 21:00 |
| mordred | fungi: https://opendev.org/zuul/zuul/commit/c12f3fe1defe8b61d59061363c9c04fb520dae18 <-- if you look at that, you'll note gitea doesn't know about corvus' key | 21:13 |
| mordred | I wonder if there is a way we can add a keyring to what gitea knows about | 21:13 |
| corvus | i wonder too, but if i had to guess, i'd guess it's keys attached to individual users | 21:13 |
| corvus | ie, gitea users | 21:13 |
| mordred | yeah- I imagine that's true | 21:13 |
| corvus | even if it is true, we could collect keys and add them to some fake user via our automation.... | 21:14 |
| mordred | still - it seems like a $something that $something should be able to $something with to display a message other than "crap unknown!" | 21:14 |
| mordred | yeah | 21:14 |
| clarkb | I think I saw changes around that in 1.12 too | 21:14 |
| clarkb | (they added gpg to the container images for example) | 21:14 |
| clarkb | not sure if any of that makes it easier for us to use | 21:15 |
| fungi | yes, that's my expectation as well. their pgp key rendering is likely connected with local gitea user accounts who have uploaded public keys. but maybe there's a keyring which can be updated independently? | 21:15 |
| mordred | it would be neat to be able to verify that a key used to sign something is part of a known trust-chain for the service | 21:15 |
| mordred | or something | 21:16 |
| mordred | rather than simply "this has been signed by a key that a user uploaded" | 21:16 |
| fungi | though since it's not the source of truth, that would only be advisory, not enforced | 21:16 |
| mordred | yeah | 21:16 |
| clarkb | keybase or is it zoom now? | 21:16 |
| clarkb | but ya having something a bit stronger than I created a mordred account and uploaded a key and said I'm mordred would be good | 21:17 |
| mordred | wait - what? keybase got bought by zoom? | 21:17 |
| mordred | clarkb: ++ | 21:17 |
| mordred | TIL | 21:17 |
| fungi | at least it shows "GPG Key ID: FE65468ED7FB3137" | 21:17 |
| fungi | a full fingerprint would be better, but ui folks hate long hex strings | 21:18 |
| mordred | incidentally - I have so-far failed at making a new gerrit test server | 21:19 |
| mordred | because right after I said that I experiences some REALLY weird network issues here that I only receently resolved | 21:19 |
| mordred | notably - a speficic laptop+SSID+access-point decided to stop doing dhcp right. unfortunately it was my laptop on my main SSID | 21:19 |
| mordred | fortunately it's the secondary AP - but I got lost down a hole of wat | 21:20 |
| mordred | because seriously - wat? | 21:20 |
| corvus | mordred: yeah, the keybase thing is i think basically viewed as an aquihire: zoom: "people say our security sucks, we need security people" keybase: "we don't have a business model" zoom: "perfect, we'll hire you by buying your company" | 21:21 |
| mordred | corvus: neat! now - if zoom actually lets the keybase people fix their security, that'll be nice | 21:21 |
| * mordred imagines the "wait, you want us to change what? no, we can't do that, because it's not how we work" conversation looming | 21:22 | |
| mordred | "can't you fix our security without changing anything?" | 21:22 |
| corvus | mordred: it would, though the cynic in me says that as soon as the deal was done and they could tell the new york times they hired smart security people then it was mission accomplished | 21:22 |
| mordred | corvus: yeah. and the keybase people are likely fine not actually working on zoom security as long as they can continue to fund keybase | 21:23 |
| corvus | mordred: i don't know details, but i wouldn't count on keybase existing | 21:24 |
| clarkb | semi related, zoom's web client uses webassembly to ship all the codec and related stuff for the audio and video and doesn't use webrtc | 21:24 |
| corvus | the pr that came out was not "we like keybase and keybase is important to us" | 21:24 |
| clarkb | (its an interesting we do things the zoom way item) | 21:24 |
| corvus | clarkb: we what? | 21:25 |
| clarkb | corvus: no we as in us, but we as in zoom | 21:26 |
| corvus | oh gotcha | 21:26 |
| clarkb | responding to mrodred comment about them doing things their way | 21:26 |
| fungi | though the upshot is similar performance with my systems vs jitsi-meet | 21:44 |
| ianw | fungi: that's a good point on having a lot more wheels for arm. it looks like just latest two branches completes in ~ 1.5 hours, so i think that's where to start | 21:48 |
| fungi | sounds reasonable | 21:50 |
| ianw | AJaeger / clarkb: i figured i'd merge the pip-and-virtualenv on my monday morning, do builds and basic sanity while it's quiet | 21:51 |
| clarkb | ianw: wfm | 21:52 |
| ianw | clarkb /fungi : https://review.opendev.org/735061 was the note i got to with the afs localhost stuff. it's not hugely satisfying, i don't know how much time we want to invest in it, i think that time would be better spent getting a 1.8 migration plan | 21:54 |
| ianw | oh nice now i see auristor's comment too. perhaps we should just write things to say "specify IP address rather than hostname" | 21:56 |
| clarkb | ianw: the note there from auristor about NAT is worth incorporating too along with specifying an ip addr | 21:57 |
| clarkb | ya that ++ | 21:57 |
| ianw | ok i'll rewrite to that | 21:58 |
| ianw | we should still do a 1.8 migration plan :) on my todo :) | 21:58 |
| clarkb | ++ to that too | 21:58 |
| clarkb | I think the awrning about NAT would be good to include as well as the /etc/hosts thing since we end up in a lot of clouds with nat | 21:59 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Add test-build-container-image-release job https://review.opendev.org/735249 | 22:07 |
| *** nautics889 has quit IRC | 22:16 | |
| mordred | clarkb: ooh. I just got an error booting review-test that we don't get in testing :) | 22:18 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Fix launch-node to work with the new inventory reorg https://review.opendev.org/735259 | 22:22 |
| mordred | clarkb: aha! it's because we reference inventory | 22:22 |
| mordred | clarkb: ^^ | 22:22 |
| clarkb | from launch? | 22:22 |
| mordred | clarkb: yup | 22:22 |
| mordred | we didnt' update the paths in launch.py | 22:22 |
| clarkb | k we've got remote doctoring today and in person tomorrow. will look when remote veraion is done | 22:23 |
| mordred | and thus aren't getting the shared group vars | 22:23 |
| mordred | cool | 22:23 |
| clarkb | one thing that hasnt changed is the sit and wait aspect of this :) | 22:24 |
| *** tkajinam has joined #opendev | 22:47 | |
| mordred | corvus, fungi, ianw: mind +A on https://review.opendev.org/#/c/735259/ ? | 22:53 |
| mordred | ianw: thanks! | 22:58 |
| mordred | (turns out it's important for launch-node :) ) | 22:58 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: openafs-client: Use PPA for all Ubuntu https://review.opendev.org/735055 | 22:58 |
| mordred | ianw: ^^ neat | 22:59 |
| ianw | yeah it makes things a bit more consistent | 22:59 |
| openstackgerrit | Merged opendev/system-config master: Fix launch-node to work with the new inventory reorg https://review.opendev.org/735259 | 23:10 |
| *** tosky has quit IRC | 23:10 | |
| openstackgerrit | Merged opendev/system-config master: Add puppet3 tests to xenial arm64 https://review.opendev.org/734698 | 23:12 |
| openstackgerrit | Prakash Ramchandran proposed opendev/irc-meetings master: Updated the irc channel https://review.opendev.org/735265 | 23:35 |
| clarkb | ianw: mordred: if you've got a quick moment for reviews https://review.opendev.org/#/c/734880/ is an easy one that came up from doing the etherpad upgrade testing. and https://review.opendev.org/735019 came up doing the python2 audit stuff | 23:36 |
| openstackgerrit | Merged openstack/project-config master: Install git-review under python3 for proposed updates https://review.opendev.org/735019 | 23:45 |
| *** DSpider has quit IRC | 23:57 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!