Friday, 2021-11-05

opendevreviewIan Wienand proposed openstack/project-config master: nodepool elements: use yaml.safe_load  https://review.opendev.org/c/openstack/project-config/+/81677400:26
*** odyssey4me is now known as Guest495100:56
*** rlandy|ruck|afk is now known as rlandy|ruck00:56
opendevreviewMerged openstack/project-config master: nodepool elements: use yaml.safe_load  https://review.opendev.org/c/openstack/project-config/+/81677401:08
*** fzzf2 is now known as fzzf02:34
*** sshnaidm is now known as sshnaidm|off03:06
*** frenzyfriday|sick is now known as frenzy_friday04:25
opendevreviewIan Wienand proposed openstack/project-config master: infra-package-needs: skip haveged start on 9-stream  https://review.opendev.org/c/openstack/project-config/+/81678206:41
opendevreviewMerged openstack/project-config master: infra-package-needs: skip haveged start on 9-stream  https://review.opendev.org/c/openstack/project-config/+/81678207:00
*** amoralej|off is now known as amoralej09:37
amoralejhi after, adding cs9 nodes in nodepool, i see the centos-9-stream labels in https://zuul.openstack.org/labels09:38
amoraleji also see an image was properly created in https://nb02.opendev.org/centos-9-stream-0000000015.log09:38
amoralejbut when i try to use it in jobs in https://review.opendev.org/c/x/packstack/+/81679609:38
amoraleji get node_failure and i can't see any log message09:39
amoralejany hint?09:39
amoralejam i missing some step? do i need to wait to have the images pushed to the providers?09:41
opendevreviewdaniel.pawlik proposed openstack/ci-log-processing master: Initial project commit  https://review.opendev.org/c/openstack/ci-log-processing/+/81560409:54
frickleramoralej: where did you see that node_failure? I see three nodes building now, they might have similar issues getting ready like fedora had/has though09:56
frickleroh, actually it looks like they are failing to boot. guess someone will need to debug, not sure how much progress we had with fedora. ianw might know09:58
opendevreviewAlfredo Moralejo proposed openstack/project-config master: Fix haveged installation in CentOS7  https://review.opendev.org/c/openstack/project-config/+/81681310:07
amoralejfrickler, yep, it's working now10:08
amoralejmaybe just a timing issue10:08
amoraleji see node started and failed later but couldn't upload logs10:09
*** jpena|off is now known as jpena10:36
*** dviroel|out is now known as dviroel|rover10:38
*** rlandy is now known as rlandy|ruck10:43
amoralejfyi, i broke infra-package-needs element for centos7 and image is failing to build. I sent https://review.opendev.org/c/openstack/project-config/+/816813 to fix it11:19
*** jcapitao is now known as jcapitao_lunch11:50
*** jcapitao_lunch is now known as jcapitao12:31
*** amoralej is now known as amoralej|lunch13:48
dpawlikfungi, clarkb: hey, so 2 vcpus, 2GB ram should be enough for the beginning to run gearman-worker, gearman-client and logscraper. If HA is required, same instance with shared one volume or maybe some pacemaker rule to check systemd service if its alive/host if is alive. All services should be running  inside the container... I prefer to use centos 814:16
dpawlikstream + podman, but it is a loose proposition :)14:16
fungidpawlik: cool, i'll check what images are available there. when you say 2 vcpus/2GB ram for gearman-worker, gearman-client and logscraper, are you wanting a separate vm for each of those or planning to start with all three on the same vm? also what hostname(s) do you prefer?14:18
clarkbdpawlik: fwiw the way gearman works you shouldn't need pacemaker or a shared volume. The idea is that each gearman job is independent14:20
clarkbthen you can just ensure the services are running via your init system or similar14:20
clarkbnote I would not use centos 8 as it is EOL in less than 2 months...14:20
dpawlikfungi: all services can be put on one host so far, it should be working normally, but I can be surprised. Dev test != production 14:20
dpawlikclarkb: so the logscraper was doing "checkpoint" file that was puting dates where it should start, so if the host will disappear second host can read the file and start scraping from some point, instead pushing same data...14:21
fungidpawlik: when you say you prefer centos 8, do you mean stream 8?14:22
dpawlikclarkb: c8stream should be running for a while14:23
dpawlikfungi: about hostname, ah, that is always hard14:23
clarkbdpawlik: ok but you said centos 8 whcih != centos 8 stream14:24
clarkb(I'm trying to make people be more specific about that because it is important right now)14:24
dpawlikclarkb: I wrote c8 stream ;)14:24
clarkbdpawlik: oh the irc message length broke it14:24
clarkbI'm sorry I misread that as a result14:24
clarkb(the line ends as "centos 8")14:24
dpawlik"logs" in latin are "acta" but acta word is not very nice in current world14:25
clarkbdpawlik: if your system is running as a singleton with essentially a lock file then refactoring gearman out probably makes sense. gearman isn't ideal in that setup since it expects that all the jobs can run at all times if they have workers available14:25
dpawlikso I will leave the hostname in your hands :)14:26
*** amoralej|lunch is now known as amoralej14:26
dpawlikclarkb: yeah. I spoke yesterday with TristanC about the logscraper and that the gearman worker/client can be deprecated in the future in the "logs workflow" and he have a new idea for the future: http://lists.zuul-ci.org/pipermail/zuul-discuss/2021-November/001744.html14:28
dpawlikIMHO currently we should go like it was designed right now14:29
clarkbdpawlik: ya I'm trying to respond to that email next. I really don't like that idea personally14:29
clarkbThere are a lot of problems with making elasticsearch tightly coupled to your CI system. I'm hoping to get that email out soon today14:29
dpawlikaha14:30
fungii'm not opposed to zuul growing a feature to push logs into an elasticsearch, but opendev likely wouldn't make use of it14:31
fungiin order to minimize the footprint of systems we're responsible for managing14:32
fungi(and managing includes integration points in that case)14:32
* dpawlik I will have something to read on Monday morning 14:34
fungidpawlik: in absence of suggestions, i'll just call it logscraper01.openstack.org14:43
fungidpawlik: what ssh key do you want granted access to the server?14:44
dpawlikfungi: some of those: https://github.com/danpawlik.keys14:45
fungiwill do, thanks14:45
dpawlikcan be first, can be second... I use both14:45
dpawlikwill do ansible playbook + finish role on monday14:45
dpawlikthanks fungi14:54
fungiclarkb: i'm trying to upload a centos 8 stream genericcloud qcow2 to vexxhost, but openstack image create is telling me "No such file or directory" for the local file i'm passing with the --file option. it definitely exists though as i tab-completed it, have you ever seen that behavior before?14:56
dpawliknope, allways was working 15:00
fungitrying to install newer osc into a venv on bridge but it breaks trying to build cryptography's rust extensions15:11
fungiaah, the venv module was using pip 9 by default, i think that had something to do with it15:13
fungiokay, newer openstackclient seems to be uploading the image just fine now15:14
clarkbya I hadn't seen that before15:17
clarkbNote that it is preferable to use raw files on vexxhost iirc15:17
fungidoesn't seem like rh makes those available for download, at least not that i could find15:25
clarkbqemu-img has a simple conversion process if you want to convert. It probably isn't as big of a deal in this case since its a small number of nodes. vexxhost complains more when we do lots of nodepool uploads and boots15:28
opendevreviewAndre Aranha proposed openstack/openstack-zuul-jobs master: Enable support for fips on the jobs  https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/81685515:32
opendevreviewAndre Aranha proposed openstack/openstack-zuul-jobs master: Enable support for fips on the jobs  https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/81685515:33
fungii'm also not sure we have enough free space on bridge for me to convert an centos image to raw anyway15:47
fungialso fun, we're just about at disk quota in sjc115:49
fungi"quota is 2000G and 1922G has been consumed"15:49
fungii'll make the root volume 40gb for now, but that only leaves us ~38gb remaining15:50
fungishould i be putting this in ca-ymq-1 instead?15:50
clarkbfungi: dpawlik: ok I just sent my novel. Apologies for the length on that email15:50
clarkbfungi: I want to say that is where mnaser_ has said we should launch stuff these days15:51
fungisjc1 or ca-ymq-1?15:51
clarkbca-ymq-115:51
fungiahh, can do15:51
clarkbHe just did a DC upgrade and its all shiny there or something along those lines15:51
opendevreviewAndre Aranha proposed openstack/openstack-zuul-jobs master: Enable support for fips on the jobs  https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/81685516:20
opendevreviewAndre Aranha proposed openstack/openstack-zuul-jobs master: Enable support for fips on the jobs  https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/81685516:24
fungidpawlik: i told cloud-init to create a passwordless dpawlik account on logscraper01.openstack.org with your keys authorized and passwordless sudo access, please test it out and let me know if you have trouble getting in, but the console log seems to indicate the account was created successfully at first boot16:24
clarkbfungi: huh how do you do that?16:26
clarkb(I thought most of the cloud-init config for creating users was baked into the images, maybe it grew new features or I missed them entirely)16:26
fungiclarkb: create a yaml file like this https://paste.opendev.org/show/81081616:28
fungipass it with --user-data=cloud-init.yaml (or whatever you called the file) in the openstack server create command16:28
clarkbah user data16:29
fungithat's how i've been doing it for my personal systems anyway and it's worked for me16:29
dpawlikfungi: works well. Thank you!16:29
fungiawesome, since we have more available disk there i gave it a 100gb rootfs, assuming you'll probably be pooling some large batches of files on it16:30
dpawlikfungi: I will read email later or just leave it for reading during Monday coffee :)16:30
fungiof course, have a great weekend!16:30
dpawlikack16:30
dpawlikyou too !16:30
*** jpena is now known as jpena|off17:22
*** amoralej is now known as amoralej|off17:42
*** rlandy|ruck is now known as rlandy|ruck|brb18:59
*** rlandy|ruck|brb is now known as rlandy|ruck19:24
ade_leefungi, hey - is there a pip problem  -- I had jobs starting to fail like this -- https://zuul.opendev.org/t/openstack/build/c0e34834b73844b7a245008c5ab0b91a 21:07
ade_lee?21:07
ade_leethis is setuptools problem ..21:08
rlandy|ruckyep - we have tox failures ... https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d41/807465/14/check/openstack-tox-py38/d415d42/job-output.txt21:11
rlandy|ruck2021-11-05 20:44:07.405487 | ubuntu-focal | The conflict is caused by:21:12
rlandy|ruck2021-11-05 20:44:07.405497 | ubuntu-focal |     jsonschema 3.2.0 depends on setuptools21:12
rlandy|ruck2021-11-05 20:44:07.405516 | ubuntu-focal |     The user requested (constraint) setuptools===58.5.221:12
rlandy|ruck2021-11-05 20:44:07.405526 | ubuntu-focal |21:12
rlandy|ruckhttps://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f45/815384/2/gate/openstack-tox-py36/f45f8dc/job-output.txt21:14
ade_leerlandy|ruck, yay ..21:15
rlandy|ruckade_lee: reporting bug for tracking21:16
rlandy|ruckhappy friday afternoon21:16
ade_leerlandy|ruck, yeah , I think this is a sign ..21:17
rlandy|ruckade_lee: go - run like the wind ... it's weekend time21:18
rlandy|ruckwill forward you the bug link21:18
ade_leerlandy|ruck, thanks21:18
rlandy|ruckade_lee: https://bugs.launchpad.net/tripleo/+bug/195001621:23
ade_leerlandy|ruck, ack my failure looks a little different, but the result is the same21:24
ade_leerlandy|ruck, I think its related to an update in global requirements recently21:25
rlandy|ruckade_lee: pls comment on the bug21:25
rlandy|ruckso we keep the tracking in one place21:25
ade_leerlandy|ruck, ack - doing so now .21:25
rlandy|ruckneed to log off now21:27
rlandy|ruckwill deal with it on sunday if nobody gets to it before21:27
*** jonher_ is now known as jonher21:28
*** yoctozepto8 is now known as yoctozepto21:28
fungisorry, was eating, but catching up on the filures now21:28
fungifirst guess, a new jsonschema release doesn't like setuptools being pinned21:29
ade_leefungi, I just updated the bug -- I ran into a different failure on ocatavia jobs21:30
fungijsonschema 4.2.1 "Released: about 6 hours ago"21:30
fungialso setuptools 58.5.3 was released yesterday, shortly after the constrained 58.5.221:34
fungii wonder if that's a clue21:34
fungilooks like the setuptools constraints bump happened when https://review.opendev.org/816611 merged at 08:13:56 utc yesterday, so that's been in place for more than a day and a half21:37
ade_leefungi, not sure why - but I had jobs get beyond this point until just this afternoon21:38
fungia pbr constraints update merged today at 17:2121:40
fungiin theory we can reproduce the other error by trying to install tempest with constraints into a py38 venv21:49
fungiyours is going to be harder to reproduce without using devstack, since it's installing a new setuptools into an already dirtied global envitonment21:51
ade_leefungi, johnsom just told me he's seeing similar in designate jobs21:52
fungia link would be helpful, so i can see if it's one of the two already observed cases, or yet a third one21:52
johnsomhttps://zuul.opendev.org/t/openstack/build/d4ea4cb87413463c83b19f6d833c6bb421:53
fungithough i expect what's happening is we're somehow sending pip into dependency hell21:53
ade_leefungi, happy Friday afternoon :) and sorry ..21:53
johnsomI had hoped it was just a pypi/index issue, so went for some exercise, but maybe it's not as simple.21:54
ade_leejohnsom, yeah that looks like my failure in octavia21:54
fungiyeah, the designate failure is like the other one ade_lee linked, devstack trying to install/upgrade/downgrade a specific setuptools version in the dirtied devstack global environemnt21:54
fungiit's possible the tripleo one has the same underlying cause, and looks like it may be easier to isolate since it's an initial install into a venv21:55
fungii'm attempting to reproduce that one on my workstation now21:57
fungiSuccessfully installed PrettyTable-2.4.0 PyYAML-6.0 argparse-1.4.0 attrs-21.2.0 autopage-0.4.0 bcrypt-3.2.0 certifi-2021.10.8 cffi-1.15.0 charset-normalizer-2.0.7 cliff-3.9.0 cmd2-2.2.0 colorama-0.4.4 coverage-6.1.1 cryptography-35.0.0 debtcollector-2.3.0 entrypoints-0.3 extras-1.0.0 fasteners-0.14.1 fixtures-3.0.0 flake8-3.7.9 flake8-import-order-0.11 future-0.18.2 hacking-3.0.121:58
fungiidna-3.3 iso8601-0.1.16 jsonschema-3.2.0 linecache2-1.0.0 mccabe-0.6.1 monotonic-1.6 msgpack-1.0.2 netaddr-0.8.0 netifaces-0.11.0 oslo.concurrency-4.5.0 oslo.config-8.7.1 oslo.context-3.4.0 oslo.i18n-5.1.0 oslo.log-4.6.1 oslo.serialization-4.2.0 oslo.utils-4.11.0 oslotest-4.5.0 packaging-21.2 paramiko-2.8.0 pbr-5.7.0 pycodestyle-2.5.0 pycparser-2.20 pyflakes-2.1.1 pyinotify-0.9.621:58
fungipynacl-1.4.0 pyparsing-2.4.7 pyperclip-1.8.2 pyrsistent-0.18.0 python-dateutil-2.8.2 python-subunit-1.4.0 pytz-2021.3 requests-2.26.0 rfc3986-1.5.0 setuptools-58.5.2 six-1.16.0 stestr-3.2.1 stevedore-3.5.0 testtools-2.5.0 traceback2-1.4.0 unittest2-1.1.0 urllib3-1.26.7 voluptuous-0.12.2 wcwidth-0.2.5 wrapt-1.13.321:58
fungiwell that's an unfortunate success21:59
johnsomlol21:59
johnsomI know that feeling21:59
clarkbthis kind of looks like the "we got stale data from pypi cdn" problem22:09
clarkb58.5.2 was released a couple of days ago and is present in the simple index listings from pypi directly and our ovh gra1 proxy cache22:09
clarkbthis means they didn't remove the package file we're pinned to at least. But maybe it isn't present if we get some stale cdn backend instead (it seems this happens more frequently with new packages I think due to the delay in updating that backup backend)22:10
fungioh, good point, i wonder if these ran exclusively in iweb mtl01 and vexxhost ca-ymq-122:10
clarkbthe failure I'm looking at is ovh gra1 so different continent22:10
clarkbbut still possible for that to have failures I guess22:10
opendevreviewMerged openstack/project-config master: Fix haveged installation in CentOS7  https://review.opendev.org/c/openstack/project-config/+/81681322:12
fungiovh-gra1, ovh-gra1, ovh-gra122:12
fungii think i see a pattern, yeah22:13
johnsomYeah, it had the feeling of a bad pypi index issue thingy to me. Later runs passed, etc.22:13
fungiso yes, i think clarkb is onto something, a fastly cdn endpoint in the vicinity of the ovh-gra1 data center with stale/fallback data could explain this22:14
clarkbwe should maybe suggest to pypa that pip should list the available versions when it makes that error message22:14
clarkbI've got a policy of avoiding that repo though so I'll defer to someone else on it22:15
fungii've done a `curl -XPURGE https://pypi.org/simple/setuptools`22:16
fungiin hopes that might wake it up22:16
fungialso did the same with a trailing / just in case22:17
fungiand did it from a shell on mirror.gra1.ovh.opendev.org in case that matters (though i don't think it's supposed to)22:18
clarkbya I think that is supposed to indicate to the entire CDN they need to refresh22:18
clarkbit is theoretically possible they refresh from the backup backend though :/22:19
fungiyep22:19
fungiespecially if there's transatlantic connection issues or something22:19
johnsomI needed to recheck my patch anyway, so I'll let you know if it pops up again22:24
clarkbfungi: looking at the PBR PEP517 thing briefly before I call it a week I notice two things it sets the package version to 0.0.0 and the sdist looks remarkably complete.22:27
clarkbhowever, it seems that installing the resulting tar.gz or whl files doesn't do what we expect. For example import bindep after intsallation fails because pbr isn't installed even though pbr is listed as a requirement22:28
clarkbya ok it didn't install parsley either22:29
clarkbBut the code seems to have been packaged and installed just without deps for some reason22:29
clarkboh and without entrypoints22:29
clarkbaha the wheel is very incomplete22:30
clarkbthe pbr.json record says that the commit is for the last bindep release and not the latest checkout22:32
clarkbya I think what might be happening is we aren't trigger the pbr'y things to generate authors, requirements, entrypoints etc via pyproject.toml22:36
clarkbif I compare the two sdists one built with pyproject.toml and the other setup.py those types of things seem to be the main difference22:36
clarkbya the requires file has all the reqirements in it. Almost like pbr isn't firing with enough pbr stuff via the pep 517 entrypoints22:37
fungimaybe the build entrypoint doesn't encompass as much as we expected?22:38
clarkbya I suspect that is what it going on. pbr/build.py just calls setuptools.build_meta.build_sdist as an example22:39
clarkbwhereas normal pbr instantiation does something a lot more than that iirc (though I don't recallexactly how it takes over setuptools setup() yet)22:39
clarkbfungi: it seems that the way pbr works is by overriding commands? So PBR definitely won't work when commands are stopped being called22:42
fungiwe'll probably need a new wrapper in pbr which calls all those things and use that as the build backend22:42
clarkbya rather than using setuptools build_meta22:43
fungiit may also need to duplicate some setuptools functionality i guess, if it needs to figure out which tasks are appropriate to perform22:43
clarkbwow I've paged back in a lot of python packaging magic just now. So roughly the way it works is pbr sets an extrypoint for distutils.setup_keywords to extend the arguments accepted by setup()22:49
clarkbthat points at pbr.core.pbr which is a function called with the value of that keyboard in setup() and that takes over everythin22:49
clarkbso ya I think we need to refactor pbr.build to call into pbr.core.pbr (or extract pieces of that that it can call)22:50
fungipbr.really_build ;)22:50
clarkblike for build sdist we need to make a subset of the things pbr() runs for an sdist build22:50
clarkbits possible we might get away with simply passing the pbr processed configs to build_meta.build_sdist22:51
clarkbso that version and requirements etc are all set properly22:51
clarkbfungi: https://github.com/pypa/setuptools/blob/main/setuptools/build_meta.py#L239-L276 I think this may have worked properly when mordred first wrote it due to that22:57
clarkbit appears that setuptools at some point replaced the build meta implementation22:57
clarkbobviously not ideal to depend on that and use it but might be a good stopgap if we can test that and see that it works. Let me try it really quickly22:58
funginot unsurprising22:58
clarkbthough build_sdist doesn't exist in the legacy bridge22:58
clarkbso maybe not22:58
clarkboh!22:59
clarkbI think you must still have a setup.py?22:59
clarkbif thats it then wow but ok22:59
clarkbok ya progress. If I put the pbr setup.py back I get what looks like a good sdist but not a good wheel23:01
clarkbI totally misinterpreted how this would be implemented23:01
clarkbhowever making a wheel from that sdist is still a problem23:02
clarkbI'll haev to pick this up again later. But this feels like progress at least. sdists seem to work more like I expect now. Version is set and there is a requires file in the sdist with the requirements23:04
clarkbSomething must be missing for making a proper wheel though23:04
clarkbstephenfin: ^ fyi23:04
clarkbfungi: heh and now if I pip install ./ in the bindep dir after restoring the setup.py it works. This makesme worry it isn't actually going through the pep517 system23:09
clarkbI wonder if there is a way to tell. In any case I suspect this is actuall 80% of the way there and we need to sort out why build doesn't work23:09
fungiclarkb: i want to say pep 517 is about making an sdist from a source tree23:11
fungior maybe it's too late on a friday23:11
fungiahh, it's both according to the hooks list: https://www.python.org/dev/peps/pep-0517/#mandatory-hooks23:13
fungii love that pbr features prominently in an arbitrary example in the recommendations for build frontends section23:15

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!