| mordred | ianw: cool! | 00:02 |
|---|---|---|
| ianw | except it failed to build, but we'll get there :) | 00:02 |
| *** DSpider has quit IRC | 00:03 | |
| mordred | one step at a time ;) | 00:03 |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-jobs master: helm-template: enable using values file https://review.opendev.org/721365 | 00:11 |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-jobs master: helm-template: allow users to disable wait-for-pods https://review.opendev.org/721369 | 00:11 |
| ianw | it built against focal | 00:15 |
| *** factor has joined #opendev | 00:20 | |
| ianw | yay, it seems to be working in a manual test | 00:34 |
| fungi | yay! | 00:34 |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-jobs master: helm-template: enable using values file https://review.opendev.org/721365 | 00:35 |
| ianw | nope, spoke too soon ... it gets further though :/ | 00:35 |
| *** dmsimard1 has joined #opendev | 00:50 | |
| *** dmsimard has quit IRC | 00:51 | |
| *** dmsimard1 is now known as dmsimard | 00:52 | |
| ianw | ok it works with a combination of pull request 26 + 27 ... sigh i don't want to think about it too much | 00:57 |
| *** mlavalle has quit IRC | 01:03 | |
| *** factor has quit IRC | 02:49 | |
| *** factor has joined #opendev | 03:15 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [wip] switch func tests to containers https://review.opendev.org/721511 | 03:18 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [wip] switch func tests to containers https://review.opendev.org/721511 | 03:57 |
| *** ysandeep|afk is now known as ysandeep | 04:20 | |
| *** olaph has quit IRC | 04:22 | |
| *** ykarel|away is now known as ykarel | 04:45 | |
| *** ykarel is now known as ykarel|afk | 05:16 | |
| *** sgw has quit IRC | 05:22 | |
| AJaeger | sgw, config-core, FYI, the docs jobs now fail if no HTML is generated, see https://review.opendev.org/721221 - and that's what sgw run into. That's on purpose, we had a few broken repos that failed in publishing | 05:32 |
| *** ykarel|afk is now known as ykarel | 05:33 | |
| *** sgw has joined #opendev | 05:40 | |
| frickler | corvus: the deploy on meetpad failed https://zuul.opendev.org/t/openstack/build/6b06e2b3356a4422a8ed2b88aa9b9074 , the log on bridge points to some ensure-pip issue | 05:50 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/project-config master: Normalize projects.yaml https://review.opendev.org/721527 | 06:06 |
| *** dpawlik has joined #opendev | 06:08 | |
| openstackgerrit | Merged openstack/project-config master: Normalize projects.yaml https://review.opendev.org/721527 | 06:37 |
| *** rpittau|afk is now known as rpittau | 07:03 | |
| *** DSpider has joined #opendev | 07:09 | |
| *** olaph has joined #opendev | 07:20 | |
| *** olaph has quit IRC | 07:29 | |
| *** tosky has joined #opendev | 07:44 | |
| *** ralonsoh has joined #opendev | 07:57 | |
| *** ysandeep is now known as ysandeep|lunch | 08:17 | |
| openstackgerrit | Paul Albertella proposed zuul/zuul-jobs master: Add Bazel build and install roles https://review.opendev.org/693513 | 09:08 |
| *** ykarel is now known as ykarel|lunch | 09:14 | |
| openstackgerrit | Paul Albertella proposed zuul/zuul-jobs master: Add Bazel build and install roles https://review.opendev.org/693513 | 09:26 |
| *** ysandeep|lunch is now known as ysandeep | 09:39 | |
| *** sshnaidm|afk is now known as sshnaidm | 09:54 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Use cached 'tox_executable' in fetch-tox-output https://review.opendev.org/721192 | 10:01 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Use cached 'tox_executable' in fetch-tox-output https://review.opendev.org/721192 | 10:06 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Use cached 'tox_executable' in fetch-tox-output https://review.opendev.org/721192 | 10:10 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Use cached 'tox_executable' in fetch-tox-output https://review.opendev.org/721192 | 10:11 |
| openstackgerrit | Maciej Józefczyk proposed openstack/project-config master: Retire openstack/networking-ovn repository https://review.opendev.org/721565 | 10:18 |
| *** ykarel|lunch is now known as ykarel | 10:20 | |
| *** rpittau is now known as rpittau|bbl | 10:31 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: fetch-sphinx-tarball: install bzip2 https://review.opendev.org/721571 | 10:42 |
| hrw | heh. forgot to try to catch ianw during his afternoon. | 10:44 |
| hrw | ianw: can you look what is missing to get centos8/aarch64 on zuul? | 10:44 |
| openstackgerrit | Maciej Józefczyk proposed openstack/project-config master: Retire openstack/networking-ovn repository https://review.opendev.org/721565 | 10:55 |
| *** ysandeep is now known as ysandeep|afk | 11:01 | |
| openstackgerrit | Jan Zerebecki proposed openstack/diskimage-builder master: Retry git clone/fetch on timeout https://review.opendev.org/721581 | 11:29 |
| *** ysandeep|afk is now known as ysandeep | 11:32 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: WIP: Add testing of fetch-sphinx-tarball role https://review.opendev.org/721584 | 11:36 |
| *** hashar has joined #opendev | 11:52 | |
| openstackgerrit | Jan Zerebecki proposed openstack/diskimage-builder master: Retry zypper when refresh failed https://review.opendev.org/721587 | 11:53 |
| openstackgerrit | Jan Zerebecki proposed openstack/diskimage-builder master: Retry zypper when refresh failed https://review.opendev.org/721587 | 12:01 |
| openstackgerrit | Paul Albertella proposed zuul/zuul-jobs master: Add Bazel build and ensure roles https://review.opendev.org/693513 | 12:11 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: WIP: Add testing of fetch-sphinx-tarball role https://review.opendev.org/721584 | 12:12 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: WIP: Add testing of fetch-sphinx-tarball role https://review.opendev.org/721584 | 12:13 |
| openstackgerrit | Jan Zerebecki proposed openstack/diskimage-builder master: Retry git clone/fetch on timeout https://review.opendev.org/721581 | 12:18 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: Make linting use of find portable https://review.opendev.org/721595 | 12:28 |
| *** rpittau|bbl is now known as rpittau | 12:31 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: WIP: Add testing of fetch-sphinx-tarball role https://review.opendev.org/721584 | 12:33 |
| ttx | corvus, fungi: more on my refs/changes cleaning adventures. Apparently it is the enormous number of refs in refs/changes that prevents me from cleaning them up. It's very likely that the per-repo mirroring would just fail for anything more active than release-test. So cleaning up changes before we switch to per-repo replication is a must-do rather than a nice-to-have. I found a way to do it in smaller | 12:49 |
| ttx | chunks, which hopefully should work even on larger repos. It's pretty slow but should work. Testing now. | 12:49 |
| ttx | (Basically github fails when you do a push that results in deleting thousands of references) | 12:51 |
| corvus | ttx: ack | 12:51 |
| corvus | frickler: that's really weird, i wonder what's different about that in prod vs test? | 12:52 |
| ttx | It's going to be very long, but the only alternative would be to continue to explicitly push refs/changes in the git replication zuul job. | 12:53 |
| corvus | frickler: the 'pip' ansible module is failing with 'ImportError: No module named pkg_resources' it's running under /usr/bin/python (python2) | 12:54 |
| corvus | ttx: zuul doesn't have all the refs changes, so that would make the replication job take a long time | 12:54 |
| ttx | ok so it's probably the only way to do it. Running a larger tset now | 12:55 |
| ttx | test* | 12:55 |
| corvus | ttx: ++ | 12:55 |
| openstackgerrit | Jan Zerebecki proposed openstack/diskimage-builder master: Retry zypper when refresh failed https://review.opendev.org/721587 | 12:57 |
| openstackgerrit | Jan Zerebecki proposed openstack/diskimage-builder master: Retry git clone/fetch on timeout https://review.opendev.org/721581 | 12:59 |
| *** ykarel is now known as ykarel|afk | 13:08 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: Add testing of fetch-sphinx-tarball role https://review.opendev.org/721584 | 13:10 |
| openstackgerrit | Jan Zerebecki proposed openstack/diskimage-builder master: Retry zypper when refresh failed https://review.opendev.org/721587 | 13:15 |
| openstackgerrit | Jan Zerebecki proposed openstack/diskimage-builder master: Retry git clone/fetch on timeout https://review.opendev.org/721581 | 13:17 |
| *** mlavalle has joined #opendev | 13:58 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Move cloud-init removal to its own playbook https://review.opendev.org/721106 | 14:03 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Just move cloud-init removal into base-server https://review.opendev.org/721107 | 14:03 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Remove some extra bits from site.pp https://review.opendev.org/721101 | 14:03 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Split codesearch into its own playbook https://review.opendev.org/721102 | 14:03 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Split eavesdrop into its own playbook https://review.opendev.org/721098 | 14:03 |
| mordred | clarkb, corvus: ^^ that last one goes ahead and docker/ansiblizes accessbot | 14:08 |
| mordred | it also splits running of accessbot from installing it - so we can just not run the run playbook in the gate | 14:08 |
| *** roman_g has joined #opendev | 14:10 | |
| openstackgerrit | Monty Taylor proposed openstack/project-config master: Run accessbot script on accessbot channels update https://review.opendev.org/721627 | 14:11 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Split eavesdrop into its own playbook https://review.opendev.org/721098 | 14:13 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Split eavesdrop into its own playbook https://review.opendev.org/721098 | 14:14 |
| *** ykarel|afk is now known as ykarel | 14:22 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Move in-tree hiera settings to ansible vars https://review.opendev.org/721629 | 14:26 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Move in-tree hiera settings to ansible vars https://review.opendev.org/721629 | 14:37 |
| openstackgerrit | Merged zuul/zuul-jobs master: Use main.yaml, not .yml https://review.opendev.org/721245 | 14:42 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add new etherpad to cacti https://review.opendev.org/721633 | 14:44 |
| openstackgerrit | Sorin Sbarnea proposed openstack/project-config master: Enable promote to unarchive gz archives in addition to bz2 https://review.opendev.org/721652 | 15:10 |
| *** dzho has joined #opendev | 15:18 | |
| *** ysandeep is now known as ysandeep|away | 15:40 | |
| *** ykarel is now known as ykarel|away | 15:40 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Split codesearch into its own playbook https://review.opendev.org/721102 | 15:44 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Split eavesdrop into its own playbook https://review.opendev.org/721098 | 15:44 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Move in-tree hiera settings to ansible vars https://review.opendev.org/721629 | 15:44 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add new etherpad to cacti https://review.opendev.org/721633 | 15:44 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Stop cloning a bunch of puppet modules we don't use https://review.opendev.org/720892 | 15:44 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run set-hostnames in run-base https://review.opendev.org/721676 | 15:44 |
| *** markmcclain has joined #opendev | 15:56 | |
| mordred | clarkb: so - that stack is green much further up now | 15:57 |
| clarkb | mordred: k I've been distracted by like 10 things already this morning. Going to try and reset than catch up on that work | 15:58 |
| mordred | corvus, fungi: if you're bored - https://review.opendev.org/#/c/721107/ and parents are green and landable | 15:58 |
| mordred | clarkb: it's eversomuch fun | 15:58 |
| mordred | clarkb: also: https://review.opendev.org/#/c/717620/ and https://review.opendev.org/#/c/720527/ and https://review.opendev.org/#/c/720709/ are ready ... although obviously do not land them - we'll need to do a coordinated rollout | 15:59 |
| mordred | actually - maybe if I take out the docker compose up we can land them and then just to the restarts | 16:00 |
| clarkb | mordred: I don't think we want the ups anyway beacuse we land a lot of zuul changes | 16:01 |
| mordred | clarkb: yeah ... well, we don't on scheduler for sure... but we do need them in the gate. maybe I should do the gerrit trick and add a start variable | 16:02 |
| mordred | set it in the gate, don't set it in prod | 16:02 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run nodepool launchers with ansible and containers https://review.opendev.org/720527 | 16:07 |
| mordred | clarkb, corvus : ^^ like that - if you like that approach, I'll apply it to the zuul patch too | 16:08 |
| corvus | mordred: it looks like you did the hostname/cloud-init moves in 3 changes when you probably could have done them in one. i've +2d them, but i dunno if maybe you want to think about squashing them? | 16:09 |
| corvus | mordred: 721676, 721107, 721106 | 16:09 |
| corvus | mordred: i'm fine with them as-is, but considering how test intensive they're likely to be, could take a while to land | 16:10 |
| mordred | corvus: yeah - actually - if y'all like the final version there I can totally squash that | 16:10 |
| mordred | why don't I just go ahead and squash them - I think the end result is solid yeah? | 16:11 |
| corvus | mordred: i do like the final version, i almost suggested that on the first change | 16:11 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Rearrange set-hostnames and cloud-init removal https://review.opendev.org/721106 | 16:14 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Remove some extra bits from site.pp https://review.opendev.org/721101 | 16:14 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Split codesearch into its own playbook https://review.opendev.org/721102 | 16:14 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Split eavesdrop into its own playbook https://review.opendev.org/721098 | 16:14 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Move in-tree hiera settings to ansible vars https://review.opendev.org/721629 | 16:14 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add new etherpad to cacti https://review.opendev.org/721633 | 16:14 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Stop cloning a bunch of puppet modules we don't use https://review.opendev.org/720892 | 16:14 |
| mordred | corvus: thereyago | 16:14 |
| clarkb | mordred: ensuring that cloud-init isn't present should noop on our images. Was that causing problems (because it shouldn't) | 16:21 |
| corvus | clarkb: regardless, we still run it in 721106. i think the commit message just has extra text now. | 16:23 |
| clarkb | hrm good point I think I'm reading too much into the rearrangement | 16:24 |
| mordred | clarkb: no - it wasn't causing problems - it just seemed inappropriate in a playbook called "set-hostnames" | 16:24 |
| clarkb | mordred: left a note on a different thing on why I Think this wasn't part of base | 16:24 |
| corvus | clarkb: expensive in test or prod? | 16:25 |
| corvus | also, i'm not sure i've ever seen autoremove clean up old kernels | 16:25 |
| clarkb | corvus: prod | 16:25 |
| mordred | clarkb: I can take out that autoremove if you want | 16:25 |
| corvus | clarkb: well, that would just happen once, right? if it happens at all? | 16:25 |
| clarkb | corvus: it would happen everytime we upgrade kernels | 16:26 |
| corvus | clarkb: right, but i mean, it'll take a couple of seconds? | 16:26 |
| corvus | and keep our disks from filling up? | 16:26 |
| clarkb | corvus: no its usually several minutes per kernel because it rebuilds initramfs things | 16:26 |
| clarkb | (I think its initramfs) | 16:26 |
| corvus | (if it happens at all, which i'd love for it to, but i'm still skeptical) | 16:26 |
| clarkb | corvus: what are we skeptical about? | 16:27 |
| corvus | i'm still skeptical autoremove will actually remove old kernels | 16:27 |
| corvus | i have lots of machines where i have to run a special script | 16:27 |
| corvus | but let's ignore that for now | 16:27 |
| clarkb | it does for me on my xenial machine. Let me update and autoremove to dobuel check | 16:27 |
| corvus | because i want to be wrong on that :) | 16:27 |
| corvus | let's assume it does happen: i think it's worth it and we should do it | 16:27 |
| mordred | I just ran apt-get autoremove on review-dev | 16:27 |
| mordred | it was a noop | 16:28 |
| mordred | there are more than one kernel there | 16:28 |
| corvus | mordred: more than 2? | 16:28 |
| mordred | yes | 16:28 |
| corvus | (current and latest are generally what we should end up with) | 16:28 |
| mordred | 173, 176 and 177 | 16:28 |
| clarkb | mordred: are we booted on 173? | 16:29 |
| mordred | yes | 16:29 |
| corvus | looks like something may be cleaning them already | 16:29 |
| clarkb | ya so it won't remove 173 | 16:29 |
| corvus | (unattended-upgrades?) | 16:29 |
| clarkb | corvus: and yes I agree that implies something is already cleaning them | 16:29 |
| mordred | yeah - so - I think it's safe - and probably a good hygiene thing | 16:29 |
| mordred | btw - I'm excited about how the puppet manifest split is turning out | 16:31 |
| fungi | mordred: revisiting one of the backup cron errors from review-dev, it looks like mysql-client isn't installed on it (but is installed on the production server). do we need to add that package, or are we supposed to not be backing up that server? | 16:31 |
| mordred | fungi: hrm. you know - maybe we should update the backup cron to run mysqldump in the mariadb container | 16:32 |
| mordred | with an exec | 16:32 |
| mordred | that said - we DO have mysql-client on review | 16:32 |
| fungi | yeah, right now its running outside | 16:32 |
| fungi | and yes, it's working on production | 16:32 |
| clarkb | mordred: there is no mariadb container on review(-dev) | 16:32 |
| clarkb | I think that is the issue | 16:33 |
| fungi | ahh | 16:33 |
| mordred | clarkb: oh - we need to redeploy review-dev don't we? | 16:33 |
| mordred | wait - no - i'm stupid | 16:33 |
| clarkb | mordred: well its still using trove is the reason | 16:33 |
| mordred | yeah | 16:33 |
| mordred | let's add the mysql-client | 16:34 |
| clarkb | corvus: fwiw just confirmed on my local xenial machine that autoremove does remove kernels | 16:34 |
| clarkb | corvus: note that ubuntu default partitioning is notorious for not allocating enough space in /boot for 3 kernels (or even 2 sometimes) | 16:34 |
| clarkb | possible that your need for special script is driven by ^ | 16:34 |
| clarkb | (I manually partition /boot on my ubuntu machines now because the installer never gives it enough room) | 16:34 |
| fungi | same, i split /boot and /boot/efi even | 16:35 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Install mysql-client on gerrit servers https://review.opendev.org/721689 | 16:35 |
| clarkb | fungi: ya the installer does too, but it gives /boot/efi like 10x the space when it needs 1/100th the room | 16:36 |
| fungi | so that /boot can be formatted differently from the limited filesystems efi supports | 16:36 |
| mordred | fungi, clarkb : ^^ | 16:36 |
| fungi | clarkb: yep, my /boot/efi is ~1% utilized according to df | 16:36 |
| mordred | oh good! the logstash filters job just started failing because apparently installing cryptography now isn't going to use wheels! | 16:38 |
| fungi | mordred: i think i also accidentally deleted some related cronspam about bup failing on review-dev, but will try to hold onto the next one | 16:38 |
| mordred | https://zuul.opendev.org/t/openstack/build/81779d15dc634e3ea756c306851c560b | 16:38 |
| mordred | or something | 16:38 |
| clarkb | mordred: usually they fix that within a fwe minutes I think | 16:38 |
| fungi | oh, or i bet i can check syslog ;) | 16:38 |
| mordred | ok. so I just got unlucky it sounds like | 16:38 |
| clarkb | (they do upload wheel after sdists occasionally and that causes fallout) | 16:38 |
| mordred | sigh | 16:38 |
| clarkb | it would be nice if it was a thing to always upload a wheel first | 16:38 |
| clarkb | but since wheels are optional you cna't really enforce that in pypi | 16:38 |
| *** rpittau is now known as rpittau|afk | 16:39 | |
| clarkb | mordred: https://pypi.org/project/cryptography/2.9.1/#files note the wheels vs tar.gz timestamp | 16:39 |
| clarkb | mordred: you got luck! | 16:39 |
| fungi | hrm, syslog seems to indicate bup ran successfully, so maybe it's just noisy | 16:39 |
| mordred | is zuul having a sad? | 16:40 |
| clarkb | mordred: it won't let me ssh | 16:41 |
| *** jrosser has quit IRC | 16:41 | |
| fungi | clarkb: lots of projects build many wheels for different environments and interpreters, some of which may come way after the release depending on their build infrastructure | 16:41 |
| clarkb | (we saw this with bridge the other day too) | 16:41 |
| clarkb | live migrations maybe? | 16:41 |
| clarkb | fungi: ya they should all upload wheels first is my point | 16:41 |
| clarkb | but its not something that is considered by people doing releases | 16:41 |
| mordred | nm | 16:41 |
| mordred | it's back | 16:41 |
| *** mnaser has quit IRC | 16:41 | |
| *** vblando has quit IRC | 16:42 | |
| fungi | right, though that effectively means avoiding uploading your source code until possibly days later when your build infrastructure catches up | 16:42 |
| *** Open10K8S has quit IRC | 16:42 | |
| *** diablo_rojo_phon has quit IRC | 16:42 | |
| clarkb | fungi: yup, but then your users don't break | 16:42 |
| *** Open10K8S has joined #opendev | 16:42 | |
| clarkb | mordred: oh I may not have my ssh key loaded now that I double check | 16:42 |
| clarkb | pebkac | 16:43 |
| fungi | load average and memory utilization look reasonable on the scheduler | 16:43 |
| fungi | nothing at all in dmesg since yesterday, so doesn't look like the vm paused | 16:44 |
| *** jrosser has joined #opendev | 16:44 | |
| *** vblando has joined #opendev | 16:44 | |
| mordred | I think maybe I just caught it doing a reconfig or something | 16:44 |
| *** mnaser has joined #opendev | 16:44 | |
| fungi | infra-root: new pip beta is up if anyone wants to take it for a spin: https://www.mail-archive.com/distutils-sig@python.org/msg28968.html | 16:45 |
| *** mnaser has quit IRC | 16:46 | |
| *** mnaser has joined #opendev | 16:47 | |
| *** mnaser has quit IRC | 16:49 | |
| *** mnaser has joined #opendev | 16:50 | |
| mordred | clarkb: jeez. the carnage of dead jobs from that cryptography release :) | 16:51 |
| corvus | mordred: looks like some test failures coming in on that stack | 16:51 |
| mordred | corvus: yeah - I believe they're all from the cryptography release | 16:52 |
| mordred | (they released tarball before wheels) | 16:53 |
| mordred | corvus: left a follow up comment on https://review.opendev.org/#/c/721098/ to your comment | 16:53 |
| *** hashar has quit IRC | 17:01 | |
| openstackgerrit | Merged opendev/system-config master: Remove old etherpad.openstack.org https://review.opendev.org/717492 | 17:05 |
| openstackgerrit | Merged opendev/system-config master: Remove unused gerrit puppet things https://review.opendev.org/714001 | 17:05 |
| openstackgerrit | Merged opendev/system-config master: Remove global variables from manifest/site.pp https://review.opendev.org/720800 | 17:05 |
| mordred | woot! | 17:07 |
| openstackgerrit | Monty Taylor proposed opendev/zone-opendev.org master: Remove etherpad-dev CNAME https://review.opendev.org/721693 | 17:08 |
| mordred | infra-root: I think we're good to delete the old etherpad.openstack.org and etherpad-dev.openstack.org servers ... do we have any concerns with that? | 17:09 |
| fungi | nope, i haven't left anything there | 17:09 |
| AJaeger | clarkb, mordred, regarding etherpad-dev: Are we fine to take everything done for them? Then I can merge https://review.opendev.org/720900 to remove the puppet modules. | 17:10 |
| corvus | mordred: wfm | 17:10 |
| AJaeger | mordred: will you empty the repos or shall I help you with those? | 17:10 |
| mordred | AJaeger: I can do it - unless you have a script already or something | 17:11 |
| AJaeger | mordred: I have no script... Just seeing that you're busy with other fires and therefore offered to help... | 17:11 |
| clarkb | mordred: I think thats ok. Dont' forget there are trove instances assoicated with them, maybe clean those up a little later? | 17:12 |
| mordred | clarkb: yeah - I think let's leave them for now just in case? | 17:13 |
| fungi | you can snapshot the trove instances to back them up if you want to delete | 17:13 |
| mordred | I think we're far enough down the road that falling back to them is unlikely | 17:13 |
| AJaeger | mordred, clarkb , approved the first retirement change... | 17:13 |
| mordred | AJaeger: \o/ | 17:13 |
| AJaeger | mordred: ok, you go ahead with emptying repo - let me fix your second project-config change | 17:13 |
| fungi | also holding onto an archival snapshot is probably nicer to the provider than keeping an unused trove instance running | 17:13 |
| mordred | ++ | 17:14 |
| AJaeger | mordred: I'll fix 720901 now... | 17:14 |
| mordred | clarkb: actually - for openstackci and gerrit and zuul - should we maybe just leave them as-is - I think people may still be using them to run 3pci? | 17:14 |
| mordred | or should we maybe tag their current tip with an eol tag, then land the retirement patch so that people are, they can pin to the eol patch? | 17:15 |
| clarkb | mordred: ya we can alawys tag after the fact too if necessary | 17:16 |
| mordred | good point | 17:17 |
| mordred | ok. I'm going to delete the old servers | 17:17 |
| mordred | infra-root: also - we have kube-master and 3 kube minion servers in rax - I think those can be deleted too - we're not doing anything with them | 17:19 |
| corvus | mordred: i don't remember us setting up a k8s in rax | 17:20 |
| mordred | we didn't | 17:20 |
| mordred | I think we booted servers in anticipation of doing so and then never used them | 17:20 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Retire unused puppet modules https://review.opendev.org/720901 | 17:21 |
| AJaeger | clarkb: could you put https://review.opendev.org/#/c/721312/ on your review queue to remove an unused job from system-config, please? | 17:22 |
| fungi | mordred: my recollection is that while there are probably still rather a lot of ci systems out there deployed from our puppet modules, few if any were continuously consuming the updates from those modules' git repositories and rarely if ever upgraded (usually just when something in the software broke which we had a corresponding behavior change in the modules for) | 17:23 |
| mordred | fungi: ++ | 17:24 |
| openstackgerrit | Merged openstack/project-config master: Stop running jobs on unused puppet repos https://review.opendev.org/720900 | 17:26 |
| corvus | mordred: i'm still confused on https://review.opendev.org/721098 fwiw | 17:26 |
| AJaeger | mordred: before we can fully retire the repos, you need to update https://review.opendev.org/#/c/720892/ - and get it merged | 17:29 |
| mordred | corvus: so - the existing state of the world in the puppet of eavesdrop is that we create a jenkins user on eavesdrop using the ::jenkins::jenkinsuser puppet module. the private key for that user is then used as a zuul secret in the yaml2ical publication job | 17:30 |
| mordred | corvus: I didnt' want to port "create a jenkins" user to the ansible | 17:30 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Retire unused puppet modules https://review.opendev.org/720901 | 17:30 |
| mordred | so figured this would be a good time to switch to using zuul deployment keys instead of putting an ssh private key into zuul as a secret | 17:30 |
| mordred | so the change here is to create a zuul user for the yaml2ical job to use instead of the jenkins user it was using, and then to update the job in project-config to use add_host along with per-project ssh key to connect to that zuul user | 17:31 |
| corvus | mordred: oooh, it's the jenkins user. got it. | 17:32 |
| mordred | \o/ | 17:33 |
| corvus | mordred: will this require a change to the yaml2ical job, or is the 'jenkins' user key already the project-config key? | 17:33 |
| mordred | corvus: https://review.opendev.org/#/c/721099/ | 17:34 |
| corvus | mordred: got it :) | 17:34 |
| mordred | I shoudl maybe fix that patch :) | 17:34 |
| corvus | yeah, we'll want to land that not too long after the other one :) | 17:34 |
| mordred | ooh - that job might be being run on irc-meetings changes | 17:35 |
| mordred | which I don't think we want as much | 17:35 |
| mordred | maybe we remove it from irc-meetings and run publication hourly from project-config | 17:36 |
| openstackgerrit | Monty Taylor proposed openstack/project-config master: Use zuul deployment keys for yaml2ical https://review.opendev.org/721099 | 17:41 |
| mordred | corvus: maybe something like that would be safer ^^ | 17:41 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Split codesearch into its own playbook https://review.opendev.org/721102 | 17:47 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Split eavesdrop into its own playbook https://review.opendev.org/721098 | 17:48 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Move in-tree hiera settings to ansible vars https://review.opendev.org/721629 | 17:48 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add new etherpad to cacti https://review.opendev.org/721633 | 17:48 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Stop cloning a bunch of puppet modules we don't use https://review.opendev.org/720892 | 17:48 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run nodepool launchers with ansible and containers https://review.opendev.org/720527 | 17:49 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run zookeeper cluster in nodepool jobs https://review.opendev.org/720709 | 17:50 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run nodepool launchers with ansible and containers https://review.opendev.org/720527 | 17:50 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: POC: download-artifacts: provide a dictionary with tests https://review.opendev.org/721703 | 17:52 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: POC: download-artifacts: provide a dictionary with tests https://review.opendev.org/721703 | 17:54 |
| mordred | #status log deleted old etherpad.openstack.org and etherpad-dev.openstack.org servers | 17:54 |
| openstackstatus | mordred: finished logging | 17:54 |
| fungi | thanks mordred!!! | 17:55 |
| mnaser | mordred: Open10K8S is trying to use the image at https://review.opendev.org/#/c/713953/ -- is there any reason why we can't rebase and merge? (should we use it out-of-tree so we can land code and if it seems stable enough, we can psuh it back after?) | 17:59 |
| mordred | mnaser: oh - yeah - we should rebase that | 18:00 |
| mordred | mnaser: lemme do that real quick | 18:00 |
| openstackgerrit | Merged opendev/system-config master: Remove system-config-puppet-beaker-rspec-puppet-4-centos-7-infra https://review.opendev.org/721312 | 18:09 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add a uwsgi-base container image https://review.opendev.org/713953 | 18:11 |
| mordred | mnaser: ^^ thereyougo | 18:11 |
| mnaser | mordred: ok awesome, i'll try to provide a "it works" signal.. | 18:12 |
| mnaser | thinking out loud tho that might be hard for the images we build, bleh! | 18:13 |
| mnaser | because opendev/system-config is not in the vexxhost tenant | 18:13 |
| *** ralonsoh has quit IRC | 18:16 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Rearrange set-hostnames and cloud-init removal https://review.opendev.org/721106 | 18:17 |
| mordred | clarkb, corvus : sorry ^^ I left out a change when I extracted that from the etherpad patch | 18:17 |
| openstackgerrit | Sorin Sbarnea proposed opendev/base-jobs master: Enable promote to unarchive gz archives in addition to bz2 https://review.opendev.org/721706 | 18:19 |
| mordred | corvus, clarkb: also - https://review.opendev.org/#/c/720527/ is green now - and I think it should be safe to land | 18:25 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Rearrange set-hostnames and cloud-init removal https://review.opendev.org/721106 | 18:26 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Remove some extra bits from site.pp https://review.opendev.org/721101 | 18:26 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Split codesearch into its own playbook https://review.opendev.org/721102 | 18:26 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Split eavesdrop into its own playbook https://review.opendev.org/721098 | 18:26 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Move in-tree hiera settings to ansible vars https://review.opendev.org/721629 | 18:26 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add new etherpad to cacti https://review.opendev.org/721633 | 18:26 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Stop cloning a bunch of puppet modules we don't use https://review.opendev.org/720892 | 18:26 |
| clarkb | mordred: there are like 3 stacks now? do we have a preference in priority for one over the other? I've been distracted by testing my nodepool change (which we should land btw to address inap issue, I can restart launcher if it lands) as well as devstack thing in #openstack-qa | 18:27 |
| mordred | clarkb: the nodepool one is likely the simplest - and shouldn't cause things to restart ... | 18:29 |
| mordred | clarkb: so - if we land it, then we could use the need to restart launchers anyway to pick up your change as the opportunity to restart them | 18:29 |
| clarkb | k I'll try to take a quick look at it (but also need to prep for meeting) | 18:30 |
| mordred | clarkb: (a quick re-review on https://review.opendev.org/721106 would be nice - I just missed a single line) | 18:30 |
| clarkb | mordred: done on the quick rereview | 18:30 |
| mordred | clarkb: thanks! | 18:31 |
| frickler | corvus: mordred: it was passing on the previous patch that got merged and I cannot find what changed over the weekend /var/log/ansible/service-meetpad.yaml.log.2020-04-18T18\:54\:06Z vs. /var/log/ansible/service-meetpad.yaml.log.2020-04-20T18\:33\:15Z | 18:32 |
| mordred | frickler: looking | 18:33 |
| mordred | frickler: Failed to import the required Python library (setuptools) on meetpad01's Python /usr/bin/python. Please read module documentation and install in the appropriate location. If the required library is installed, but Ansible is using the wrong Python interpreter, please consult the documentation on ansible_python_interpreter | 18:34 |
| mordred | frickler: we rolled out the clarkb change to install docker-compose from pip | 18:35 |
| clarkb | mordred: frickler do we need to set python3 as the ansible_python_interpreter? | 18:35 |
| mordred | perhaps we shoudl be setting python_interpreter there | 18:35 |
| mordred | yeah | 18:35 |
| mordred | clarkb: I thnik so | 18:35 |
| clarkb | I want to say bionic has python3 but not python2 | 18:35 |
| mordred | python3 is solid on that host | 18:35 |
| clarkb | (I don't know why it cares all of a sudden) | 18:35 |
| mordred | python2 is there | 18:35 |
| mordred | but yeah - I'm confused - maybe something accidentlaly installed python2 where it wasn't installed before? | 18:36 |
| mordred | clarkb, frickler: I'll push up a patch real quic | 18:37 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Set ansible_python_interpreter to python3 on meetpad https://review.opendev.org/721707 | 18:39 |
| mordred | clarkb, frickler :^^ | 18:39 |
| frickler | mordred: oh, unattended upgrades http://paste.openstack.org/show/792499/ | 18:39 |
| mordred | frickler: ah yes - because before we had docker-compose installed which pulled in extra things | 18:40 |
| mordred | but since we don't install that from distro anymore, u-a cleaned up for us (yay) | 18:41 |
| mordred | frickler: glad that explains why it worked then stopped working | 18:41 |
| frickler | that also explains why one run passed even after the removal of docker-compose, it failed after the nightly cron | 18:41 |
| prometheanfire | mordred: ianw mind tagging a new glean release (if https://review.opendev.org/367487 can't get merged beforehand) | 18:41 |
| openstackgerrit | Monty Taylor proposed opendev/ansible-role-puppet master: Stop logging puppet to syslog https://review.opendev.org/721710 | 18:44 |
| mordred | sigh | 18:44 |
| clarkb | mordred: what is wrong with syslog? | 18:45 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Stop logging puppet to syslog https://review.opendev.org/721711 | 18:45 |
| mordred | clarkb: it makes looking at job output hard | 18:45 |
| clarkb | mordred: well the reason we syslogged it was to have the logs on the host iirc | 18:46 |
| mordred | yeah- but I think these days it's more of a pita now that we have more and more logging on bridge because of ansible | 18:46 |
| mordred | it makes the ansible easy to see and the puppet hard | 18:46 |
| clarkb | mordred: ya I don't really care either way just wanted to call out that on host logs was an explicit design choice there before | 18:49 |
| clarkb | mordred: my only other concern is that the stdout/stderr logs may be less verbose than syslog by default? | 18:49 |
| clarkb | I think we get the most verbose set of logs in syslog | 18:49 |
| mordred | clarkb: do we? the docs on logdest don't call that out - but they're also super sparse | 18:50 |
| clarkb | mordred: I think default for stdout is just errors? | 18:51 |
| clarkb | maybe warnings but I know we get all the info stuff in syslog too | 18:51 |
| mordred | clarkb: I could stick that in the tree after the eavesdrop patch and we could see the difference | 18:51 |
| clarkb | mordred: ++ | 18:51 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Stop logging puppet to syslog https://review.opendev.org/721711 | 18:53 |
| mordred | clarkb: there - that'll be sure to trigger it and we can see what it's like | 18:53 |
| mordred | it's almost like this whole testing system works ;) | 18:53 |
| mordred | corvus: if you have a sec, I'm getting a containers issue in https://zuul.opendev.org/t/openstack/build/4e475259a12b4e9383cd1afdb18760cb | 18:55 |
| mnaser | mordred: hmm, ran into "fatal error: stdio.h: No such file or directory" before? i'm assuming because of some of the clean ups done, its missing that | 18:55 |
| mnaser | gcc is installed and i dont feel like adding build-essential in there | 18:56 |
| mordred | corvus: and it makes me think I've got the job set up wrong - but I did parent it on system-config-run-containers | 18:56 |
| mordred | mnaser: you need somethign else ... one sec | 18:56 |
| mnaser | libc6-dev maybeh? | 18:56 |
| mordred | yup | 18:57 |
| mordred | libc6-dev | 18:57 |
| mordred | mnaser: nodepool's bindep is pretty minimal and a good one to look at for the basics | 18:57 |
| mordred | (it has a few more than you need for other thigns - but it's been minimized pretty strongly) | 18:58 |
| mnaser | ok cool i'll add-as-i-go | 18:58 |
| clarkb | if you split build from consumption you should be able to safely add build essential and everything else you need without worrying about size too much | 18:58 |
| clarkb | I think the bindep env is called "compile" | 18:59 |
| mordred | that's right | 18:59 |
| mordred | clarkb: but you actually don't need build-essential even in compile - because it pulls in all the stuff you need to build debian packages | 18:59 |
| openstackgerrit | Mohammed Naser proposed opendev/system-config master: Add a uwsgi-base container image https://review.opendev.org/713953 | 18:59 |
| mordred | most of the time gcc and libc6-dev is all you need | 18:59 |
| mnaser | yeah that's true i guess | 18:59 |
| mnaser | it was just a matter of keeping it quick'n'efficent | 18:59 |
| mordred | I like avoiding it - even though ti's just build stage, because it is a waste | 18:59 |
| mordred | yup | 19:00 |
| clarkb | mordred: ya definitely just use what you need just wanted to call out we've addressed the underlying concern pretty well | 19:00 |
| mordred | ++ | 19:00 |
| mnaser | btw, i wanted to propose at some point to add git to python-builder | 19:00 |
| mnaser | it seems like 99% of the times you end up having to isntall it | 19:00 |
| mnaser | and if you dont then assemble will | 19:00 |
| mordred | it should already be in builder | 19:00 |
| mnaser | hm | 19:00 |
| corvus | meeting time in #opendev-meeting | 19:01 |
| mordred | it's added after the bindep stuff is installed so that if you have git in your bindep file you end up with it in your final image | 19:01 |
| mnaser | at the time i had to install it if i tired to do a git inside of it | 19:01 |
| mnaser | (trying to build openstack images using python-builder/python-base by cloning repos into it) | 19:01 |
| ianw | hrw: hrm, i think the centos-8 arm nodes should be ready? | 19:01 |
| mordred | mnaser: assemble will install it - but it's not there before assemble | 19:01 |
| mnaser | mordred: yeah i kinda needed it to be there before assemble in my case, but its a weird unicorn case | 19:02 |
| mordred | mnaser: I highly recommend cloning the repos externally and pointing builder at them | 19:02 |
| mnaser | mordred: yes i was going to follow up with ideally id like to do COPY from teh zuul checked out repos | 19:02 |
| mordred | yup | 19:02 |
| mnaser | mordred: but i wasn't sure _yet_ how to bake that into the dockerfile while still making it buildable locally | 19:02 |
| mordred | because we explicitly don't have git in the builder image - becaues if we do it winds up excluded from the final image and that's bad :) | 19:02 |
| ianw | hrw: did you just try a centos-8-arm64 node type? | 19:03 |
| mordred | mnaser: I think for these you might just have to have the instructions be "cd src/opendev.org/openstack/nova ; docker build . -f ~/src/openev.org/vexxhost/openstack-images/docker/nova/Dockerfile" | 19:03 |
| mnaser | mordred: ah yes that seems a bit reasonable | 19:04 |
| mnaser | or maybe you could do docker build inside nova but.. one day :) | 19:04 |
| mordred | one day :) | 19:04 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Split eavesdrop into its own playbook https://review.opendev.org/721098 | 19:07 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Stop logging puppet to syslog https://review.opendev.org/721711 | 19:07 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Move in-tree hiera settings to ansible vars https://review.opendev.org/721629 | 19:07 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add new etherpad to cacti https://review.opendev.org/721633 | 19:07 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Stop cloning a bunch of puppet modules we don't use https://review.opendev.org/720892 | 19:07 |
| hrw | ianw: o, indeed. started working. | 19:15 |
| hrw | ianw: sorry for bothering | 19:16 |
| prometheanfire | mordred: ianw should I make a release request to releases for glean? | 19:18 |
| fungi | prometheanfire: no, we can just push a tag | 19:20 |
| corvus | fungi: when you said "this is true" in your comment, what does "this" refer to? | 19:28 |
| corvus | i totally get that the two comments form a conversation :) | 19:28 |
| corvus | i just don't understand the antecedent | 19:28 |
| fungi | corvus: "this" == "Its possible this is no longer a concern..." | 19:32 |
| fungi | sorry | 19:32 |
| openstackgerrit | Merged zuul/zuul-jobs master: Use cached 'tox_executable' in fetch-tox-output https://review.opendev.org/721192 | 19:32 |
| prometheanfire | fungi: ah, cool, wasn't totally sure how it was managed | 19:36 |
| fungi | it's no longer in openstack at all | 19:36 |
| fungi | so wouldn't be handled by the openstack release team regardless | 19:36 |
| clarkb | mordred: root screen has run flock -n /var/run/reprepro/ubuntu.lock bash in window 0 | 19:40 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Split eavesdrop into its own playbook https://review.opendev.org/721098 | 19:45 |
| ianw | hrw: excellent :) lmn ... it seemed too easy! | 19:45 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Stop logging puppet to syslog https://review.opendev.org/721711 | 19:47 |
| ianw | prometheanfire: i can do a tag soon. will this fix the builds in the dib gate? | 19:48 |
| prometheanfire | ianw: It along with the DIB patch will help I think | 19:51 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Split eavesdrop into its own playbook https://review.opendev.org/721098 | 19:55 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Stop logging puppet to syslog https://review.opendev.org/721711 | 19:55 |
| corvus | mordred: re https://review.opendev.org/721707 -- if we just remove python2 on the host, then testing == prod again, right? | 19:55 |
| mordred | yeah - I think that's the better idea | 19:56 |
| mordred | doing that on meetpad now | 19:56 |
| corvus | okay, i -2d the change because our streams crossed and frickler +3d it | 19:57 |
| mordred | ++ | 19:57 |
| mordred | corvus: when you have a sec, https://review.opendev.org/#/c/721106/ needs a re-review - I missed a line when I split it out earlier | 19:59 |
| clarkb | infra-root I have manually held all the affected lockfiles in https://review.opendev.org/#/c/720718/2/modules/openstack_project/manifests/mirror_update.pp via a root shell on mirror-update.openstack.org. I am going to approve this change now | 20:00 |
| corvus | mordred: +3 | 20:00 |
| openstackgerrit | Merged zuul/zuul-jobs master: Make linting use of find portable https://review.opendev.org/721595 | 20:00 |
| mordred | corvus: thanks | 20:00 |
| mordred | clarkb: ++ | 20:00 |
| clarkb | also I've bumped the quotas for ubuntu-ports and ubuntu mirrors as they both add focal | 20:00 |
| clarkb | once that change lands and applies we can then manaully run all of those sync commands, then manually vos release -localauth on the afs servers | 20:01 |
| clarkb | I'll work on all that over the course of today | 20:01 |
| corvus | clarkb: thanks! | 20:01 |
| corvus | mordred: did you remove python2? i'll manually run the meetpad playbook if so | 20:01 |
| mordred | corvus: I did | 20:01 |
| ianw | clarkb: ++ can give a hand if it gets late for you | 20:01 |
| clarkb | ianw: thanks! | 20:01 |
| corvus | <2001:4800:7819:104:be76:4eff:fe04:892f> Failed to connect to the host via ssh: /bin/sh: 1: /usr/bin/python: not found | 20:02 |
| corvus | mordred: i thought it was supposed to try python3 in that case? | 20:02 |
| mordred | corvus: neat. I thought it was too | 20:02 |
| corvus | mordred: oh, maybe i need to refresh the fact cache? | 20:02 |
| fungi | depends on the version of ansible, right? | 20:02 |
| mordred | oh - maybe so | 20:02 |
| mordred | corvus: yeah - /usr/bin/python is definitely in the fact cache for meetpad | 20:03 |
| ianw | i think that container based functional tests for platforms of interest are failing due to test configuration at this point, particularly volume maps, rather than issues with dib/containers as such. investigating | 20:04 |
| mordred | as is discovered_interpreter_python | 20:04 |
| corvus | mordred: i tried "-m setup" and "-m gather_facts" and no joy | 20:04 |
| clarkb | ianw: I've stuck a copy of reprepro-mirror-update in my homedir on mirror-update.openstack.org and commented out the vos release | 20:04 |
| corvus | mordred: got any hints? | 20:04 |
| clarkb | ianw: I'll be using that script once reprepro configs update | 20:04 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Retire i18n-specs repo https://review.opendev.org/721719 | 20:05 |
| mordred | corvus: why don't we try moving the fact cache file out of the way | 20:05 |
| corvus | mordred: https://docs.openstack.org/openstack-ansible/12.2.6/install-guide/ops-troubleshooting-ansiblecachedfacts.html#forcing-regeneration-of-cached-facts | 20:05 |
| corvus | mordred: our osa friends appear to recommend that | 20:05 |
| mordred | + | 20:05 |
| mordred | want me to do that? | 20:05 |
| corvus | mordred: i will | 20:05 |
| mordred | k | 20:05 |
| clarkb | and with that I'm going to get a bike ride in as everything should be stable at the moment | 20:06 |
| clarkb | back in an hour or so | 20:06 |
| corvus | mordred: that made everything better | 20:07 |
| mordred | corvus: woot! | 20:07 |
| clarkb | corvus: mordred ok you know what my explicit trigger of replication for openstack/requirements may have fixed the review/p/ thing | 20:08 |
| clarkb | because I agree ti seems up to date nwo | 20:08 |
| clarkb | mordred: so maybe we just need to fix those new repos and then retrigger replication for all of /opt/git? | 20:09 |
| clarkb | and really popping out now. back soon | 20:09 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Retire i18n-specs repo https://review.opendev.org/721719 | 20:10 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Finish retiring i18n-specs https://review.opendev.org/721722 | 20:10 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Finish retiring i18n-specs https://review.opendev.org/721722 | 20:13 |
| ianw | prometheanfire / mordred: i just pushed glean 1.18.0 ... only functional change really is the gentoo resolv.conf thing | 20:14 |
| corvus | frickler: okay meetpad seems to be working again | 20:16 |
| corvus | er, restarting again real quick | 20:16 |
| openstackgerrit | Merged opendev/system-config master: Rearrange set-hostnames and cloud-init removal https://review.opendev.org/721106 | 20:18 |
| openstackgerrit | Merged opendev/system-config master: Remove some extra bits from site.pp https://review.opendev.org/721101 | 20:18 |
| openstackgerrit | Merged opendev/system-config master: Start mirroring focal, stop mirroring trusty https://review.opendev.org/720718 | 20:19 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [wip] switch func tests to containers https://review.opendev.org/721511 | 20:27 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [wip] strip env vars in yum-minimal chroot calls https://review.opendev.org/721726 | 20:27 |
| mordred | clarkb: yeah - they're definitely not owned right - but I don't know why _why_ they aren't owned right, becaues jeepyb sure does do a chown | 20:35 |
| mordred | I'm just going to fix them for now - let's keep an eye out next time we make a new project - the system was known to be in an inconsistent state, so I don't know that debugging why the three repos created during that time are weird is worth it | 20:38 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [wip] strip env vars in yum-minimal chroot calls https://review.opendev.org/721726 | 20:46 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [wip] switch func tests to containers https://review.opendev.org/721511 | 20:46 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Split eavesdrop into its own playbook https://review.opendev.org/721098 | 20:54 |
| *** DSpider has quit IRC | 20:57 | |
| openstackgerrit | Monty Taylor proposed opendev/puppet-hound master: Update git url to hound-search https://review.opendev.org/721729 | 20:59 |
| mordred | clarkb, fungi, corvus : ^^ sigh | 20:59 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [wip] strip env vars in yum-minimal chroot calls https://review.opendev.org/721726 | 20:59 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [wip] switch func tests to containers https://review.opendev.org/721511 | 20:59 |
| mordred | clarkb: incidentally: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_31b/721711/5/check/system-config-run-codesearch/31b107e/bridge.openstack.org/ara-report/result/a673a179-9ab6-41db-8ade-de5952604587/ | 21:01 |
| openstackgerrit | Monty Taylor proposed opendev/ansible-role-puppet master: Add color parameter https://review.opendev.org/721732 | 21:05 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Stop logging puppet to syslog https://review.opendev.org/721711 | 21:07 |
| mordred | clarkb: that's the results of the "log puppet to stdout" change. there's sadly some ansi color - so there's the abilitu to disable so that it's readable | 21:07 |
| mordred | infra-root: logstash-worker13 seems down | 21:13 |
| mordred | (noticed this from a failed infra-prod-base) | 21:14 |
| corvus | it appears to have gone down on april 18 | 21:14 |
| mordred | neat. | 21:15 |
| mordred | corvus: I'm gonna look at its console on rax - and if there's no smoking gun issue, I figure just restarting it yeah? | 21:17 |
| corvus | mordred: sgtm; i confirm no ssh from here | 21:17 |
| mordred | kk | 21:17 |
| fungi | at best you'll probably see console messages about cpu cycle skips | 21:18 |
| fungi | probably another live migration gone wrong | 21:18 |
| mordred | it OOMed | 21:19 |
| fungi | nice | 21:19 |
| fungi | oom'ed hard enough to take ssh offline? | 21:19 |
| fungi | or just an outright panic? | 21:19 |
| mordred | rebooting | 21:19 |
| corvus | and snmp | 21:19 |
| mordred | oomed and then there were jdb stuck tasks | 21:19 |
| corvus | ah | 21:20 |
| fungi | were the timestamps close together? otherwise the oom could have been from some other time | 21:20 |
| mordred | the rax showed last updated april 18 - so I'm also guessing live migrate - that then caused something to get stuck - and the oom is just. symptom | 21:20 |
| fungi | regardless, yeah, needs reboot | 21:20 |
| mordred | sshing again | 21:20 |
| mordred | working | 21:20 |
| fungi | thanks! | 21:20 |
| mordred | fungi, corvus: do y'all have opinions on the puppet-log-to-stdout patch? | 21:20 |
| mordred | ah - I see you do! | 21:21 |
| * mordred goes to read | 21:21 | |
| corvus | mordred: i just wrote a followup comment too | 21:21 |
| fungi | i had opinions? i've already forgotten if so | 21:21 |
| mordred | corvus: I agree with your statement | 21:22 |
| mordred | I would prefer both too - but I also agree, I don't see that as an option | 21:22 |
| corvus | basically, there have been times recently where the only way i could find out what happened on a host was on the hosts syslog, so i'd have more warm fuzzies if that could remain while we got used to how it's sposed to work :) | 21:22 |
| mordred | ++ | 21:22 |
| mordred | I mean - it's a variable - we could set it differently in the gate | 21:23 |
| corvus | nah, let's try the thing that makes the most sense, and change it if it's bonkers | 21:23 |
| mordred | which is where my main concern is - it's harder to find the logs amongst the syslog in the gate jobs | 21:23 |
| mordred | kk | 21:23 |
| corvus | so frowny face +2 | 21:23 |
| mordred | fungi: https://review.opendev.org/#/c/721732/ and https://review.opendev.org/#/c/721711/ if you feel you might have an opinion | 21:23 |
| mordred | https://zuul.opendev.org/t/openstack/build/31b107e4fef348ba9c4d002d2d689a9e is a job with the logging to stdout - albeit still with ansi codes | 21:24 |
| mordred | but you can click through to the ara report and then see the output in the failed task | 21:24 |
| fungi | mordred: my opinion is yes | 21:25 |
| fungi | at least on the first one | 21:25 |
| mordred | \o/ | 21:25 |
| * mordred is writing so many puppet related patches | 21:26 | |
| fungi | 721711 is somewhat of a lack of visibility for folks who have ssh access to some servers but not bridge | 21:26 |
| fungi | also, like corvus, i've relied on this in the past but agree it won't prevent me from being able to investigate at least | 21:27 |
| corvus | fungi: but if we can verify the log is clean, we could make that public | 21:27 |
| fungi | yep, true | 21:27 |
| corvus | (by including it in the job) | 21:27 |
| fungi | so potentially improved visibility for people with no ssh access | 21:27 |
| fungi | which on balance is nicer | 21:27 |
| fungi | also, what's the story for launch-node? | 21:27 |
| fungi | will the puppet output come back to stdout of the calling terminal now? | 21:27 |
| fungi | previously we had to --keep and then ssh into a broken launched node and look in syslog to find out what puppet did (or didn't) | 21:28 |
| mordred | fungi: I'm gonna say "yes" | 21:28 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: cabal-test: add first haskell job https://review.opendev.org/721735 | 21:28 |
| fungi | if that's how launch-node starts behaving, might prevent needing to rerun with --keep | 21:29 |
| fungi | so actually nice | 21:29 |
| *** dpawlik has quit IRC | 21:34 | |
| *** sgw has quit IRC | 21:34 | |
| clarkb | corvus: ya restarting those workers is generally the best thing if its sad | 21:39 |
| clarkb | infra-root it seems that the focal update is on mirror-update now | 21:39 |
| clarkb | I'm going to start with some of the smaller repo as we should be able to clear them out quickly | 21:40 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: cabal-test: add first haskell job https://review.opendev.org/721735 | 21:41 |
| clarkb | infra-root that results in http://paste.openstack.org/show/792506/ so I'm trying to figure out clearvanished now | 21:42 |
| clarkb | this was on ceph hammer | 21:42 |
| openstackgerrit | Merged opendev/ansible-role-puppet master: Add color parameter https://review.opendev.org/721732 | 21:43 |
| clarkb | infra-root: `k5start -t -f /etc/reprepro.keytab service/reprepro -- reprepro --confdir /etc/reprepro/debian-ceph-hammer clearvanished` appears to be what I want goign for htat if i don't hear objection | 21:46 |
| ianw | clarkb: that seems about right, although i have to re-learn all the reprepro magic whenever i have to touch it | 21:47 |
| clarkb | nwo it says There are still packages in 'trusty|main|amd64', not removing (give --delete to do so)! | 21:48 |
| clarkb | so I'm going to give it the --delete | 21:48 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [wip] strip env vars in yum-minimal chroot calls https://review.opendev.org/721726 | 21:49 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [wip] switch func tests to containers https://review.opendev.org/721511 | 21:49 |
| clarkb | and now it says (dumpunreferenced lists such files, use deleteunreferenced to delete them.) | 21:49 |
| clarkb | so I think the sequence is reprepro --delete clearvanished && reprepro deleteunreferenced | 21:49 |
| clarkb | I'm doing that now | 21:49 |
| clarkb | (still only operating on the small ceph repo so that if I get it wrong rebuilding is easy | 21:50 |
| clarkb | you have to do reprepro --nokeepunreferencedfiles deleteunreferenced otherwise it doesn't actually delete unreferenced files | 21:52 |
| clarkb | so now I'm gonna rerun reprepro normally and it should be happy since trusty has been removed | 21:52 |
| *** sgw has joined #opendev | 21:53 | |
| prometheanfire | ianw: looks like the gentoo dib build is passing, so, good sign | 21:53 |
| clarkb | alright that looks happy so now I'm gonna vos release it and we can double check the trusty cleanup results | 21:54 |
| openstackgerrit | Merged zuul/zuul-jobs master: helm-template: allow users to disable wait-for-pods https://review.opendev.org/721369 | 21:55 |
| fungi | wow, reprepro is such an example of arcane incantations | 21:55 |
| clarkb | infra-root https://mirror.dfw.rax.opendev.org/ceph-deb-hammer/ was my first victim. Can you check if that looks like trusty cleaned up properly? | 21:56 |
| clarkb | looks like we may need to manually remove https://mirror.dfw.rax.opendev.org/ceph-deb-hammer/dists/trusty/main/binary-amd64/ ? | 21:56 |
| clarkb | (I think the man page actually warns of this) | 21:56 |
| clarkb | "Do not forget to remove all exported package indices manually." | 21:56 |
| clarkb | but also those are much smaller can probably be done whenever we like without much worry | 21:57 |
| clarkb | (rather than needing to incorporate it as part of the manual flow but I can as well) | 21:57 |
| ianw | hrm, i thought pretty much what was described in https://docs.openstack.org/infra/system-config/reprepro.html would get rid of it | 21:57 |
| ianw | the bit at "In some situations where things are very out of sync, it may be easier to remove and replace an entire section of the repository." anwyay ... but ... yeah, reprepro is all black magic ... | 21:57 |
| fungi | well, don't forget reprepro keeps a stateful database in, i think, /var/lib about what it's synced | 21:58 |
| fungi | so just deleting the files which were mirrored by hand can get it all nice and confused | 21:59 |
| clarkb | ya I think what ianw links to is how we keep the reprepro db in sync | 21:59 |
| clarkb | to deltee packages I had to run the deleteunreferenced command | 21:59 |
| clarkb | https://mirror.dfw.rax.opendev.org/ceph-deb-hammer/pool/main/c/ceph/ shows no more trusty packages so I think that was working | 21:59 |
| fungi | ahh, yeah, our doc talks about the db files too | 21:59 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: cabal-test: add first haskell job https://review.opendev.org/721735 | 22:00 |
| fungi | anyway, if you're going for clean slate, you have to delete those too | 22:00 |
| clarkb | should I manually delete the indexes as suggested by the manpage then rerun reprepro and a vos release? | 22:00 |
| clarkb | fungi: those? | 22:00 |
| openstackgerrit | Nate Johnston proposed openstack/project-config master: Retire the Congress project https://review.opendev.org/721742 | 22:00 |
| fungi | clarkb: no, i suspect the manpage is talking about empty package indices | 22:00 |
| clarkb | fungi: any idea how I get the trusty indices to go away? | 22:01 |
| fungi | but i'm not in a good state to try to digest it at the moment (i've only ever had to troubleshoot it once, when we were trying to add debian/buster backports i think, and got really turned around) | 22:01 |
| clarkb | https://mirror.dfw.rax.opendev.org/ceph-deb-hammer/dists/trusty/ basically all that | 22:01 |
| clarkb | my reading of the manpage is that I have to manually delete that | 22:02 |
| fungi | oh, hrm, yeah those aren't empty | 22:02 |
| clarkb | and I'm happy to rm that tree and then rerun reprepro to make sure its still happy | 22:02 |
| fungi | you could move the trusty dir out of the way and rerun reprepro to see if ti complains or recreates it | 22:02 |
| clarkb | ok let me try that | 22:03 |
| clarkb | I renamed trusty to trusty-old, reran reprepro. It seems to have completed successfully and not error'd. There is no new trusty/ dir under dists | 22:04 |
| clarkb | now I will rm -rf trusty-old/ ? | 22:04 |
| clarkb | I'll rerun reprepro after that just to be double sure | 22:04 |
| openstackgerrit | Nate Johnston proposed opendev/irc-meetings master: Remove the congress team meeting https://review.opendev.org/721744 | 22:06 |
| fungi | clarkb: sounds fine, yep | 22:06 |
| clarkb | https://mirror.dfw.rax.opendev.org/ceph-deb-hammer/dists/ all done that looks good to me | 22:07 |
| fungi | i guess it just doesn't clean up package metadata files or remove any entries from them if you tell it to delete the packages | 22:07 |
| clarkb | alright I'll proceed with the other smaller repos and get a complete process written down | 22:07 |
| openstackgerrit | Monty Taylor proposed opendev/puppet-hound master: Update git url to hound-search https://review.opendev.org/721729 | 22:09 |
| mordred | corvus, fungi : ^^ sorry - puppet lint - re-review plz? | 22:09 |
| mordred | clarkb: reading scrollback | 22:09 |
| corvus | mordred: i will happily approve changes both with and without the :: :) | 22:10 |
| corvus | because i have no idea | 22:10 |
| fungi | i will even more happily approve changes removing puppet though ;) | 22:11 |
| mordred | fwiw: https://github.com/voxpupuli/puppet-lint-absolute_classname-check#relative-class-name-inclusion | 22:11 |
| clarkb | infra-root https://etherpad.opendev.org/p/tMfB9VRcqe7NhS9a4-ZX thats the process I think | 22:11 |
| mordred | clarkb: looks like things are going decently for you? | 22:11 |
| clarkb | mordred: yup I think I have a process that works now | 22:12 |
| clarkb | I just need to work through the list. I'll leave ubuntu and ubuntu-ports for last as they are scariest :) | 22:12 |
| clarkb | I want things to go wrong in the smaller repos | 22:12 |
| mordred | clarkb: lgtm | 22:12 |
| fungi | clarkb: what's with the custom /home/clarkb/reprepro-mirror-update ? | 22:13 |
| clarkb | fungi: it doesn't vos release because we want to do that on the afs server with localauth | 22:14 |
| fungi | oh, yep! | 22:14 |
| mordred | corvus: if you have a sec: at the end of the failed task here: https://zuul.opendev.org/t/openstack/build/8b290fa9faf44657b5367a1e22e49e2f/console | 22:14 |
| fungi | good call | 22:14 |
| clarkb | also note that the deb ceph jewel repo also has an old jessie repo in it | 22:15 |
| clarkb | gonna clean that up with trusty because may as well | 22:15 |
| mordred | corvus: the system-config-build-image-accessbot job builds opendevorg/accessbot - but that docker pull is not finding it | 22:16 |
| fungi | good call | 22:16 |
| mordred | clarkb: ++ | 22:16 |
| mordred | corvus: this is for https://review.opendev.org/#/c/721098 | 22:16 |
| clarkb | jewel hasn't actually updated in a while because of the jessie thing so this is a good exercise of all the stuff | 22:18 |
| mordred | corvus: do I need to explicitly put a latest tag in there? | 22:19 |
| mordred | corvus: OH | 22:21 |
| mordred | wait - no | 22:21 |
| mordred | corvus: yeah - pull-from-intermediate-registry is running on localhost - not on the jobs' eavesdrop.o.o host | 22:22 |
| mordred | corvus: do we need to do things to make sure the docker we install with system-config on the job's eavesdrop.o.o is configured to pull from the buildset registry? I thought we'd already worked on this before | 22:24 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: cabal-test: add first haskell job https://review.opendev.org/721735 | 22:26 |
| corvus | mordred: erm, let me page stuff in :) | 22:30 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Split eavesdrop into its own playbook https://review.opendev.org/721098 | 22:32 |
| mordred | corvus: I'm taking a stab at something there - but definitely page things in because I think there's still maybe a fundamental thing here | 22:33 |
| clarkb | ceph jewel and ubuntu cloud archive are vos releasing now | 22:33 |
| clarkb | then I'll do ports | 22:33 |
| clarkb | then I'll do ubuntu | 22:33 |
| clarkb | I expect these to take signficiantly more time | 22:33 |
| mordred | corvus: what I *think* I've determined is that we have an issue because we don't run use-buildset-registry on our nodes | 22:33 |
| mordred | so nodes we have that are using images aren't getting speculative images becuase the docker we install as part of our production ansible is not configured to talk to the buildset registry | 22:34 |
| clarkb | alright those two are done done now | 22:34 |
| corvus | mordred: i think we do -- https://zuul.opendev.org/t/openstack/build/8b290fa9faf44657b5367a1e22e49e2f/console#1/0/41/eavesdrop01.openstack.org | 22:34 |
| corvus | mordred: we run it in run-base-pre | 22:35 |
| mordred | corvus: oh - gotcha | 22:35 |
| clarkb | note: ubuntu-ports does not remove any dists only adds focal | 22:35 |
| clarkb | so I'm skipping the first two steps and going straight ot running our script | 22:36 |
| mordred | corvus: so my latest patchset is poo | 22:36 |
| clarkb | Warning parsing /etc/reprepro/ubuntu-ports/updates, line 13: unknown architecture 'amd64' will be ignored! | 22:37 |
| clarkb | I think that is because ports is everything but amd64 | 22:37 |
| clarkb | (so thats all fine and can be ignored) | 22:37 |
| clarkb | also I did bump quota on ubuntu-ports by a fair bit so that should be fine. Can reduce it down if I gave it too mcuh room after | 22:39 |
| mordred | clarkb: I think so - it's just a quota, not an actual FS allocation | 22:39 |
| corvus | mordred: i think we reinstall docker | 22:39 |
| mordred | corvus: yeah - because we install docker as part of the playbook | 22:40 |
| corvus | mordred: it might overwrite the config | 22:40 |
| mordred | sigh | 22:40 |
| corvus | or actually, that might not be a reinstall | 22:40 |
| mordred | it's the first install | 22:40 |
| corvus | yeah; so we should verify the daemon config | 22:40 |
| mordred | there is no docker when we run use-buildset-registry | 22:40 |
| mordred | is it safe to pull back to the log server? | 22:40 |
| corvus | yep | 22:41 |
| mordred | k. adding it to the mix | 22:41 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Split eavesdrop into its own playbook https://review.opendev.org/721098 | 22:42 |
| corvus | mordred: add /etc/hosts too | 22:43 |
| corvus | https://zuul.opendev.org/t/openstack/build/8b290fa9faf44657b5367a1e22e49e2f/console#1/0/39/eavesdrop01.openstack.org | 22:43 |
| corvus | that's what the docker config should be; i just noticed it relies on /etc/hosts | 22:43 |
| mordred | corvus: oh - you know what - I think we write out /etc/hosts | 22:44 |
| mordred | lemme check if it does lineinfile | 22:44 |
| mordred | corvus: nope! that would be it | 22:44 |
| corvus | ah in set-hostname | 22:44 |
| mordred | yeah | 22:45 |
| mordred | maybe we should move set-hostname to the pre-playbook | 22:45 |
| corvus | mordred: and we weren't using that role earlier right (before your change?) | 22:45 |
| mordred | and run it before use-buildset registry | 22:45 |
| mordred | yup, that's right | 22:45 |
| corvus | cool, then we only just broke it :) | 22:45 |
| corvus | mordred: yeah, i kinda like that, since it's something we usually run in launch node | 22:46 |
| mordred | corvus: oh - except wanna hear somethign funny? | 22:46 |
| corvus | mordred: and maybe add a note that use-buildset-registry wants to run after it | 22:46 |
| mordred | corvus: it's in playbooks/roles, not roles - so it's not accessible by zuul | 22:46 |
| mordred | and the pre playbook is run by zuul | 22:46 |
| mordred | lemme try symlink | 22:46 |
| clarkb | I'm realizing this might take all day. So I'm going to start ubuntu concurrent with ubuntu-ports | 22:46 |
| corvus | mordred: the pre playbook is in playbooks/zuul -- is that not good enough? | 22:47 |
| openstackgerrit | Merged opendev/puppet-hound master: Update git url to hound-search https://review.opendev.org/721729 | 22:47 |
| corvus | mordred: (would that only allow access to playbooks/zuul/roles, not playbooks/roles ?) | 22:47 |
| mordred | yup | 22:47 |
| clarkb | ianw: ^ fyi, I'll try pop in over the evening and check on things but chances are you may end up needing to do vos release later? or maybe I'll even do that in the morning | 22:47 |
| mordred | we could move it to playbooks | 22:47 |
| mordred | or - let's start with symlink - verify it fixes the issue - then we can take a step back and think about what the right way to organize is | 22:48 |
| corvus | k | 22:48 |
| corvus | mordred: could also probably do lineinfine :) | 22:48 |
| mordred | corvus: yeah | 22:49 |
| mordred | corvus: but I think we overwrite on purpose because we don't know the incoming state of hostnames from our cloud providers | 22:49 |
| ianw | clarkb: so it's syncing ubuntu? let me pull up a window | 22:49 |
| clarkb | ianw: not yet its doing the cleanup bits now | 22:50 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Split eavesdrop into its own playbook https://review.opendev.org/721098 | 22:50 |
| clarkb | ianw: root screen window 0 is ubuntu and window 1 is ubuntu-ports | 22:50 |
| clarkb | also I see that we sync source packages | 22:50 |
| mordred | corvus: trying symlink. I moved the role to roles and then symlinked to it from playbooks/roles | 22:50 |
| clarkb | we maybe don't want to do that? | 22:50 |
| corvus | mordred: good point, then that sounds best | 22:50 |
| mordred | clarkb: I think we did that on purpose for a reason? | 22:50 |
| clarkb | mordred: oh maybe if it is on purpose I'll just accept it will take longer :) | 22:51 |
| mordred | clarkb: well - I mean - I think - but I don't remember what the reason was | 22:51 |
| clarkb | 106.9k trusty packages are about to be deleted | 22:51 |
| clarkb | I'm largely trusting reprepro here fwiw (and the fact it seemed happy on the 3 previous repos) | 22:52 |
| ianw | clarkb: so i see ubuntu-ports running, is there another window for ubuntu? | 22:53 |
| ianw | oh sorry, i see your prior comment now | 22:53 |
| clarkb | ianw: windows 2,3,4 can all be closed which will release their locks and allow those volumes to be updated normally | 22:53 |
| clarkb | ianw: I'll do that once I've got ubuntu running the normal update script | 22:53 |
| ianw | ok | 22:54 |
| clarkb | ianw: I'm hoping I can reduce the problem set to "run vos release command on afs01 once this script finishes) | 22:54 |
| ianw | ok, i won't go touching anything yet :) | 22:56 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: cabal-test: add initial haskell job https://review.opendev.org/721735 | 22:56 |
| ianw | the first thing we need to move this to ansible is gpg import stuff ... i started looking around to see if that was a solved problem a while ago, ansible didn't have anything built in | 22:57 |
| clarkb | fwiw the kernel versions it is deleting look good | 22:57 |
| clarkb | (they are all reaelly old trusty era versions) | 22:57 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Stop logging puppet to syslog https://review.opendev.org/721711 | 22:58 |
| mordred | ianw: I feel like we've got some gpg import code somewhere that can be cargo culted | 22:58 |
| ianw | the repropro roles are pretty much just templating | 22:58 |
| mordred | ianw: ah - no - all we have is apt-key imports, which are different | 22:59 |
| mordred | ianw: roles/add-gpgkey in zuul-jobs | 23:00 |
| mordred | ianw: I knew I'd seen something somewhere | 23:00 |
| mordred | ianw: don't know if that'll be useful - but maybe it's helpful | 23:01 |
| ianw | cool, yeah i'll have to context switch it back in but it was something like that | 23:03 |
| *** tosky has quit IRC | 23:03 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: cabal-test: add initial haskell job https://review.opendev.org/721735 | 23:05 |
| ianw | ok, i think i've fixed the problem with the functional-containers build for yum-minimal (https://review.opendev.org/721726) -- using wrong TMPDIR when inside initial chroot | 23:05 |
| ianw | why this doesn't fail in a non-container environment ... i don't know | 23:05 |
| ianw | i've noticed some other stuff that can be cleaned up in the job now though, so will beat it all into order today | 23:06 |
| mordred | ianw: sweet | 23:08 |
| mordred | ianw: so - do you think after that's in shape that just installing deboostrap from the ppa you made is the next step in the debootstrap thing? | 23:08 |
| * mordred has only half-paged in the stuff you did - is curious | 23:09 | |
| ianw | mordred: i think so ... for getting debuntu working with the status quo minimal elements | 23:11 |
| clarkb | ianw: https://etherpad.opendev.org/p/tMfB9VRcqe7NhS9a4-ZX I'm taking notes there on current states of things | 23:12 |
| ianw | mordred: no issues looking deeper into the container based build elements -- but personally i don't think we should try bringing in that, and container builders all at the same time | 23:12 |
| mordred | yeah - I think fixing one thing at a time is a good idea | 23:13 |
| ianw | it's always better if we have something to clearly roll-back to | 23:13 |
| mordred | and in general - I'd like to be able to say "the zuul/nodepool-builder images generally work" - which they generally do other than this | 23:13 |
| ianw | i.e. what did *not* happen with the suse stuff last week or so | 23:14 |
| mordred | yeah | 23:14 |
| ianw | i think so -- soon we'll have centos,fedora,xenial,bionic,suse in dib gate and in experimental buster,gentoo using the builder images | 23:15 |
| clarkb | ianw: I've closed the other windows and removed their assocaited locks so the screen is very focused now on the remaining repos | 23:15 |
| ianw | that should be enough distributions for anyone :) | 23:15 |
| ianw | ok, i'm watching a tail of the ports update log | 23:17 |
| mordred | ianw: ++ | 23:18 |
| mordred | ianw: who would ever need more that 640k of RAM??? | 23:18 |
| mordred | zomg! | 23:19 |
| mordred | clarkb, ianw : https://review.opendev.org/#/c/721102 is green! it's our first "split a puppeted service into its own playbook" | 23:19 |
| clarkb | ianw: I think I'm going to start dinner early. That way I can come back to the mirrors when they've gotten further along | 23:32 |
| clarkb | ianw: both seem to be happily chugging through the packages right now and quota headroom looks fine | 23:32 |
| clarkb | I'll check in periodically through my evening and will be sure to let you know when I'm permanently afk | 23:33 |
| clarkb | the etherpad should be up to date onw where we are right now | 23:33 |
| ianw | clarkb: ok, thanks ... i have a window up with the logs so will keep watching too | 23:34 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Split eavesdrop into its own playbook https://review.opendev.org/721098 | 23:36 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Stop logging puppet to syslog https://review.opendev.org/721711 | 23:37 |
| *** mlavalle has quit IRC | 23:37 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: Add remove-zuul-sshkey https://review.opendev.org/680712 | 23:45 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!