| clarkb | ianw: that whole stack lgtm now. But we probabl want ot double hceck the correct versions are installed before approving | 00:08 |
|---|---|---|
| ianw | yeah i'll pull out the log references | 00:09 |
| opendevreview | Ian Wienand proposed opendev/system-config master: system-config-run-borg-backup: rename hosts to distro https://review.opendev.org/c/opendev/system-config/+/852685 | 00:12 |
| opendevreview | Ian Wienand proposed opendev/system-config master: create-venv: add role; use in install-borg https://review.opendev.org/c/opendev/system-config/+/852487 | 00:15 |
| opendevreview | Ian Wienand proposed opendev/system-config master: system-config-run: bump base timeout to 3600 https://review.opendev.org/c/opendev/system-config/+/852479 | 00:15 |
| opendevreview | Ian Wienand proposed opendev/system-config master: install-borg: update to borg 1.1.18 https://review.opendev.org/c/opendev/system-config/+/852488 | 00:15 |
| opendevreview | Ian Wienand proposed opendev/system-config master: system-config-run-borg-backup: add jammy test host https://review.opendev.org/c/opendev/system-config/+/852489 | 00:15 |
| opendevreview | Ian Wienand proposed opendev/system-config master: gate-groups: remove old backup group https://review.opendev.org/c/opendev/system-config/+/852684 | 00:15 |
| opendevreview | Ian Wienand proposed opendev/system-config master: system-config-run-borg-backup: rename hosts to distro https://review.opendev.org/c/opendev/system-config/+/852685 | 00:15 |
| ianw | sigh, typo | 00:15 |
| *** clarkb is now known as Guest166 | 00:18 | |
| Guest166 | hrm I don't want to load the passwd manager to reauth this evening. I'll be a guest for now | 00:19 |
| *** dasm|off is now known as Guest167 | 00:20 | |
| fungi | we're very welcoming to guests | 00:21 |
| Guest166 | seems like my connectivity to oftc has been much flakier lately | 00:21 |
| fungi | me too apparently | 00:31 |
| fungi | looks like it's probably rolling server restarts | 00:31 |
| *** rlandy is now known as rlandy|out | 00:45 | |
| *** ysandeep|out is now known as ysandeep | 01:23 | |
| opendevreview | Ian Wienand proposed openstack/project-config master: nodepool: update package maps for Rocky 9 https://review.opendev.org/c/openstack/project-config/+/852518 | 01:33 |
| opendevreview | Merged openstack/project-config master: project-config-grafyaml: drop job https://review.opendev.org/c/openstack/project-config/+/851954 | 01:49 |
| opendevreview | Merged opendev/system-config master: create-venv: add role; use in install-borg https://review.opendev.org/c/opendev/system-config/+/852487 | 02:11 |
| *** ysandeep is now known as ysandeep|afk | 02:44 | |
| fungi | ianw: looks like infra-prod-service-borg-backup failed in deploy for 852487 | 03:32 |
| fungi | i haven't looked at the logs yet | 03:32 |
| ianw | sigh, looking | 04:44 |
| ianw | The task includes an option with an undefined variable. The error was: 'ansible.vars.hostvars.HostVarsVars object' has no attribute 'borg_user' | 04:46 |
| ianw | which is weird ... | 04:46 |
| ianw | "item": "lists.openstack.org" | 04:47 |
| ianw | oooohhhh i think i see | 04:53 |
| *** ysandeep|afk is now known as ysandeep | 04:57 | |
| ianw | oh for heavens sake .. it's translate01 | 05:01 |
| ianw | the venvs look broken on some of these old hosts. i'm probably going to have to punt on this one until tomorrow | 05:22 |
| *** marios is now known as marios|ruck | 05:37 | |
| *** pojadhav is now known as pojadhav|rover | 05:43 | |
| *** prometheanfire is now known as Guest194 | 07:00 | |
| *** ysandeep is now known as ysandeep|afk | 07:09 | |
| *** Guest194 is now known as Guest200 | 07:28 | |
| *** jpena|off is now known as jpena | 07:36 | |
| *** ysandeep|afk is now known as ysandeep | 07:37 | |
| *** Adri2000_ is now known as Adri2000 | 08:29 | |
| *** ysandeep is now known as ysandeep|lunch | 09:34 | |
| *** rlandy|out is now known as rlandy | 10:37 | |
| *** ysandeep|lunch is now known as ysandeep | 11:06 | |
| *** dviroel|out is now known as dviroel | 11:24 | |
| *** bhagyashris_ is now known as bhagyashris | 11:32 | |
| fungi | broken at some point in the past and we've been needing to blow away and recreate them? | 11:42 |
| Tengu | y | 11:50 |
| *** gthiemon1e is now known as gthiemonge | 12:09 | |
| fungi | we should probably brace for an impending setuptools 64.0.0 release with the new pep 660 editable installs implementation, since i wouldn't be surprised if some of our communities turn up corner cases the maintainers aren't expecting | 12:35 |
| fungi | https://discuss.python.org/t/16904 | 12:36 |
| opendevreview | Simon Westphahl proposed zuul/zuul-jobs master: Supply missing error message on S3 upload failure https://review.opendev.org/c/zuul/zuul-jobs/+/852383 | 13:24 |
| *** ysandeep is now known as ysandeep|break | 13:30 | |
| *** ysandeep|break is now known as ysandeep | 14:34 | |
| *** ysandeep is now known as ysandeep|out | 14:47 | |
| *** dviroel is now known as dviroel|lunch | 15:38 | |
| *** marios|ruck is now known as marios|out | 15:49 | |
| *** Guest200 is now known as prometheanfire | 15:59 | |
| *** pojadhav|rover is now known as pojadhav|out | 16:22 | |
| *** dviroel|lunch is now known as dviroel | 16:47 | |
| *** Guest166 is now known as clarkb | 16:50 | |
| *** jpena is now known as jpena|off | 16:52 | |
| fungi | finally have a moment to dig into the borg venvs on xenial servers... they do look thoroughly borg^Hked | 18:14 |
| fungi | calling `pip list` in then results in a cascade of tracebacks | 18:14 |
| fungi | f'spec_for_{name}', | 18:14 |
| fungi | SyntaxError: invalid syntax | 18:14 |
| clarkb | fungi: they probably had the same issue we fixed for bionic but since the condition matched only bionic they broke | 18:14 |
| clarkb | I think we need to rebuild them with pip and setuptools caps for python3.5 | 18:14 |
| clarkb | basically do the same thing we did for bionic | 18:15 |
| fungi | i think we ended up installing the bionic (py36) version on xenial (py35) yes | 18:15 |
| fungi | agreed, should be a pretty straightforward patch | 18:15 |
| clarkb | I think its worse than that. We install python3.7 versions on xenial | 18:15 |
| clarkb | eg latest | 18:15 |
| fungi | er, yeah but not py35 versions at any rate | 18:15 |
| fungi | i'll see what we need to pin back to for that | 18:15 |
| fungi | of course, we don't have python3-venv installed, so i'm not sure how these venvs/virtualenvs could even have been created | 18:16 |
| fungi | is there some magic back door to make a venv? | 18:17 |
| clarkb | is virtualenv installed? I think once the venv exists ansible doesn't care about the install comamnd so our attempts to use -m venv would just be ignored | 18:18 |
| fungi | no, at least not in the module search path for python3 | 18:18 |
| clarkb | maybe it used python2 virtualenv? | 18:19 |
| fungi | the installed libpython3.5-stdlib does seem to provide /usr/lib/python3.5/venv though, it's the lack of ensurepip which seems to be at issue | 18:19 |
| fungi | fungi@storyboard01:~$ /opt/borg/bin/python --version | 18:19 |
| fungi | Python 3.5.2 | 18:19 |
| fungi | better still, the python3-venv package actually is installed | 18:22 |
| fungi | aha, we had pending upgrades for python 3.5 | 18:25 |
| fungi | unfortunately, we're getting 401 unauthorized from esm.ubuntu.com | 18:26 |
| clarkb | fungi: I guess we should email them about extending the existing setup? | 18:27 |
| clarkb | we're down to just a handful of nodes now | 18:27 |
| fungi | oh, i guess maybe it expired? | 18:27 |
| clarkb | yes I think they gave us a year and that year ran out. | 18:27 |
| clarkb | In theory we can ask them to extend it and then it will just work again? I dunno the documentation and info on that stuff is very light (probably intentionally) | 18:28 |
| fungi | yeah, so in short i don't have access to any python3.5 systems i can easily create a venv on, so i'll inspect package metadata instead | 18:37 |
| fungi | i even deleted the old 3.5 build on my workstation to free up space a while back | 18:37 |
| clarkb | fungi: docker run -it --rm ubuntu:xenial should still work | 18:38 |
| fungi | just append whatever commands i want to run in a xenial container after that? | 18:38 |
| clarkb | fungi: based on pip changelog pip<21 | 18:38 |
| clarkb | fungi: ya I would usually run bash | 18:39 |
| clarkb | and that will give you a shell prompt that you can interact with | 18:39 |
| clarkb | `docker run -it --rm ubuntu:xenial bash` is the full command | 18:39 |
| clarkb | then when you exit bash docker will clean up the container | 18:39 |
| fungi | oh, yep it started a root shell for me | 18:39 |
| fungi | so looks like pip<21 setuptools<51 | 18:40 |
| clarkb | we may need to manually clear out the old venv and let ansible rebuild it after we update our ansible. Or manually fix it up and then let ansible enforce that state going forward | 18:44 |
| clarkb | basically I don't expect the current state to be recoverable using the ansible tooling unless we add some rm's to the ansible too | 18:44 |
| fungi | yeah, i expect that as well | 18:48 |
| fungi | what's the ansible syntax if i want to compare against a list of strings? | 18:49 |
| fungi | right now our fallthrough is "when: ansible_distribution_release != 'bionic'" | 18:50 |
| fungi | and i'd like it to be neither xenial nor bionic | 18:50 |
| fungi | ahh, i can do this instead i think: | 18:51 |
| fungi | when: ansible_distribution_versi | 18:51 |
| fungi | on is version('18.04', '>=') | 18:51 |
| fungi | er, i guess i want >=20.04 for the fallthrough | 18:52 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Also pin pip/setuptools when creating Xenial venvs https://review.opendev.org/c/opendev/system-config/+/852786 | 18:55 |
| fungi | clarkb: ianw: ^ that maybe? | 18:55 |
| Clark[m] | That looks right. CI should provide useful feedback | 18:57 |
| clarkb | fungi: that didn't actually run the backup test job. I think we should update the job to trigger on updates to that create-venv role (at least temproarily) | 19:33 |
| clarkb | do you want to make that update or shoudl I push that? | 19:33 |
| fungi | i can give it a shot | 19:33 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Also pin pip/setuptools when creating Xenial venvs https://review.opendev.org/c/opendev/system-config/+/852786 | 19:35 |
| fungi | clarkb: ^ | 19:35 |
| clarkb | cool zuul has enqueued the borg job this time | 19:36 |
| fungi | perfecto | 19:36 |
| clarkb | we should be able to confirm it installs the right packages on bionic (we don't have a xenial test node in that job) which is probably good enough | 19:37 |
| *** rlandy is now known as rlandy|mtg | 20:16 | |
| fungi | system-config-run-borg-backup TIMED_OUT | 20:27 |
| clarkb | fungi: https://zuul.opendev.org/t/openstack/build/a1d1e7206e224a8baa487e4c405915c2/log/job-output.txt#21159 I think that shows the bionic host installing older pip and setuptools nad that focal and jammy install the latest and greatest | 20:57 |
| clarkb | +2 from me | 20:58 |
| *** rlandy|mtg is now known as rlandy | 21:03 | |
| clarkb | it seems that error has gone away now that I'm doing the alembic check | 21:41 |
| clarkb | but hard to say if I'm just getting lucky or that is helping | 21:41 |
| opendevreview | Alfredo Moralejo proposed opendev/system-config master: Use rackspace mirror to sync centos stream repos https://review.opendev.org/c/opendev/system-config/+/852793 | 21:42 |
| *** dviroel is now known as dviroel|out | 21:48 | |
| clarkb | randomly my brightness change buttons work again | 22:02 |
| fungi | freaky | 22:02 |
| ianw | thanks for looking at the venv stuff! | 22:05 |
| ianw | that was my assumption too that we'd pulled in later versions | 22:05 |
| ianw | on the timeout, the change to bump the timeouts hasn't merged yet because of ... timeouts :) | 22:07 |
| fungi | d'oh! | 22:08 |
| fungi | i would have recheck-spammed it in earlier today if i'd remembered | 22:08 |
| fungi | we could also bypass zuul to merge that | 22:08 |
| fungi | falls into the catch-22 category where that is marginally acceptable | 22:09 |
| *** rlandy is now known as rlandy|bbl | 22:17 | |
| ianw | i'll give it one more chance otherwise take corrective action | 22:21 |
| clarkb | I suspect that I may have misinterpreted the SMTP_HOST variable's purpose and that is why ianw had issues signing up | 22:31 |
| clarkb | I think its meant to designate the source ip for sending email and it sends directly? | 22:32 |
| clarkb | so setting it to localhost is just all wrong | 22:32 |
| fungi | checking whether that has a direct analog in mm2 | 22:34 |
| clarkb | nevermind here it says https://github.com/maxking/docker-mailman/blob/d928d36b97fab6fac2a6295ef5822549a68ed0c8/README.md it wants to send through the host itself | 22:37 |
| fungi | there's a SMTPHOST='localhost' in Defaults.py which we don't seem to override | 22:37 |
| clarkb | the web readme says it is the ip to send from | 22:37 |
| clarkb | ok so I've probably got that bit correct. Will have to dig in properly tomorrow | 22:37 |
| fungi | also SMTP_HELO_HOST=DEFAULT_URL_HOST which may be more of what you were thinking of | 22:38 |
| clarkb | ianw: oh interesting note on the issue being upstream and possibly deserving further investigation | 22:39 |
| clarkb | er previously I mean and maybe this time. We don't know about this time | 22:39 |
| * fungi grumbles at another timeout and rechecks again | 23:15 | |
| ianw | so basically i'll manually remove /opt/borg venv on xenial hosts and let them get recreated | 23:27 |
| clarkb | once that change lands | 23:28 |
| ianw | yep | 23:28 |
| ianw | that's going to be the other thing about upgrading borg to 1.2 series,; it's 3.8+ | 23:29 |
| clarkb | so we'd need everything on focal or newre. Which is in progress but progress is slow :) | 23:30 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!