| openstackgerrit | Merged opendev/system-config master: Docs: Update main page for OpenDev https://review.opendev.org/714323 | 02:22 |
|---|---|---|
| clarkb | that change should exercise the new restart logic | 03:04 |
| clarkb | mordred: ^ I wonder if there is a good way to verify that gerrit responded as expected. maybe via replication logs | 03:05 |
| openstackgerrit | Kevin Zhao proposed openstack/project-config master: Linaro US: Change bionic job to 16G RAM https://review.opendev.org/714346 | 03:08 |
| openstackgerrit | Kevin Zhao proposed openstack/project-config master: Linaro US: Change bionic job to 16G RAM https://review.opendev.org/714346 | 05:04 |
| openstackgerrit | sebastian marcet proposed opendev/puppet-openstackid master: Fixing results from ZAP Scanning Report https://review.opendev.org/714351 | 05:05 |
| openstackgerrit | Kevin Zhao proposed openstack/project-config master: Linaro US: Change bionic job to 16G RAM https://review.opendev.org/714346 | 05:41 |
| openstackgerrit | Kevin Zhao proposed openstack/project-config master: Linaro US: Add a 16GB RAM label for bionic https://review.opendev.org/714346 | 06:03 |
| AJaeger | config-core, please review https://review.opendev.org/714174 https://review.opendev.org/714066 | 06:09 |
| *** DSpider has joined #opendev | 07:44 | |
| *** dpawlik has joined #opendev | 07:44 | |
| *** ralonsoh has joined #opendev | 07:59 | |
| *** tosky has joined #opendev | 08:32 | |
| *** rpittau|afk is now known as rpittau | 08:52 | |
| *** ralonsoh has quit IRC | 09:44 | |
| *** ralonsoh has joined #opendev | 09:44 | |
| *** bcafarel has left #opendev | 09:55 | |
| *** dpawlik has quit IRC | 10:11 | |
| *** dpawlik has joined #opendev | 10:13 | |
| *** roman_g has joined #opendev | 10:33 | |
| *** ouroboros8 has joined #opendev | 11:21 | |
| *** rpittau is now known as rpittau|bbl | 11:47 | |
| *** roman_g has quit IRC | 12:46 | |
| mordred | clarkb: good question | 13:03 |
| *** rpittau|bbl is now known as rpittau | 13:08 | |
| *** roman_g has joined #opendev | 13:35 | |
| AJaeger | infra-root, two small fixes for review, please: https://review.opendev.org/700901 (pbrx), https://review.opendev.org/702246 (netlify-sandbox), | 13:45 |
| mordred | fungi: could I interrupt your morning with https://review.opendev.org/#/c/714319/ ? | 13:48 |
| mordred | fungi: and from thence afterwords unto https://review.opendev.org/#/c/714305/, which verily doth depend on it? | 13:49 |
| mordred | s/from// | 13:49 |
| fungi | sure, just a sec | 13:50 |
| mordred | thanks! | 13:50 |
| AJaeger | thanks, mordred | 13:52 |
| AJaeger | why is netlify-sandbox in opendev? None of us is core there | 13:53 |
| corvus | AJaeger: we can add ourselves | 13:55 |
| corvus | it's a repo to test out the netlify/gerrit integration that we hope to (someday) use for web site cms | 13:56 |
| AJaeger | mmh, readme says " StarlingX website at starlingx.io"? | 13:56 |
| corvus | yeah, that's the test data | 13:56 |
| AJaeger | Ah! | 13:56 |
| corvus | i think so far we haven't had to approve much (it's a sandbox) | 13:57 |
| AJaeger | my change was just updating .gitreview - wanted to cleanup my queue | 13:57 |
| mordred | yeah. we've got 4 or 5 steps we need to work on | 13:57 |
| corvus | i'm sure we can bootstrap ourselves to land a gitreview change | 13:57 |
| AJaeger | thanks for background | 13:57 |
| corvus | say i've got a repo with a debian dir like this: https://github.com/jitsi/jitsi-meet/tree/master/debian | 13:59 |
| corvus | what's the best way to build the packages? | 13:59 |
| corvus | cd into that and run 'dpkg-buildpackage' ? | 13:59 |
| corvus | run debian/rules? | 14:00 |
| corvus | debuild? | 14:01 |
| mordred | I normally use debbuild instead of dpkg-buildpackage | 14:01 |
| mordred | yeah | 14:01 |
| mordred | you might want to run it with -uc -us - which will tell it not to try to sign the changes or source | 14:01 |
| mordred | (if you're not the author of the most recent change in the changelog) | 14:02 |
| corvus | yes i want that, thanks! | 14:02 |
| mordred | there's also pdebuild which will use pbuilder to make a a chroot and then build it in there | 14:02 |
| corvus | i'll probably just throw all this into a container | 14:02 |
| mordred | but that's like 10 year old info - one would imagine there woudl be something similar to pdebuild which would use docker nowadays | 14:02 |
| corvus | since that's what i'm trying to build anyway | 14:03 |
| corvus | the jitsi-meet docker images are really just installs of the project's debian packages | 14:03 |
| corvus | (which seems weird to me, sort of a "worst of both worlds" situation; but i don't want to reinvent their system) | 14:03 |
| mordred | corvus: https://github.com/bureado/pbocker | 14:04 |
| mordred | corvus: might be a small thiung to look at that you might be able to steal commands from | 14:04 |
| corvus | mordred: thanks! | 14:06 |
| fungi | i second the preference for debbuild over dpkg-buildpackage | 14:08 |
| fungi | and yeah, pdebuild (using pbuilder) or cdebuild (using cowbuilder) is useful if you want packages consistently built in a minimal chroot | 14:09 |
| fungi | pbocker does look like an interesting modernization. pbuilder traditionally bootstrapped and the tarred up the chroot for future reuse | 14:10 |
| fungi | things folks use docker for these days | 14:11 |
| fungi | mordred: 714307 depends on an outdated patchset of its parent change | 14:14 |
| corvus | fungi, mordred: does http://paste.openstack.org/show/791025/ mean anything to you? | 14:24 |
| mordred | corvus: yes - debian expects the directory to be named with the version | 14:24 |
| mordred | or, rather, for the package name | 14:25 |
| mordred | (version is optional) | 14:25 |
| corvus | oh, it seems to behave differently if i run it in the top level dir | 14:25 |
| mordred | so try putting it into a dir called "jitsi-meet-web" instead of "jitsi-meet" | 14:25 |
| corvus | instead of debian | 14:25 |
| mordred | yes - sorry | 14:25 |
| mordred | you should run in the top level, not in debian/ | 14:25 |
| corvus | mordred: well, i mean, i'm just trying to do what upstream does | 14:25 |
| corvus | ok i'll do that | 14:25 |
| fungi | debuild generally expects the "debian" directory to be a subdir of your pwd, creates temporary build results in ./debian/build/ and writes package files in ../ | 14:27 |
| corvus | a build-depends would have been nice :/ | 14:27 |
| mordred | corvus: they don't have build-depends in the debian/control file? | 14:28 |
| fungi | yeah, if they do you ought to be able to tell apt-source to parse that for you | 14:28 |
| openstackgerrit | Merged opendev/system-config master: Split gitea and gerrit services from manage-projects https://review.opendev.org/713995 | 14:28 |
| clarkb | fungi: AJaeger looking at those review requests the first one in the list is for #openvswitch. They use meetbot, but don't have accessbot set up and now don't want accessbot with default registration requirement? | 14:28 |
| fungi | er, i mean apt-get source | 14:29 |
| mordred | fungi: one of my long-standing gripes is that there isn't a "please install the build depends for this debian/control file" command - at least not in debian. there is something in ubuntu-dev-scripts I think | 14:29 |
| corvus | mordred: only https://github.com/jitsi/jitsi-meet/blob/master/debian/control#L6 | 14:29 |
| corvus | none of the 'node' 'jdk' etc stuff | 14:29 |
| mordred | wow | 14:29 |
| clarkb | is that captured by their docker file? | 14:30 |
| fungi | mordred: oh, actually i was thinking of `sudo mk-build-deps -i | 14:31 |
| fungi | ` | 14:31 |
| fungi | (from devscripts) | 14:31 |
| corvus | clarkb: no, they don't build debs with docker (i don't know how they build them) | 14:31 |
| mordred | fungi: ah - does that exist now? | 14:31 |
| fungi | yeah | 14:31 |
| mordred | probably build them on someone's laptop | 14:31 |
| fungi | "Given a package name and/or control file, mk-build-deps will use equivs to generate a binary package which may be installed to satisfy all the build dependencies of the given package." | 14:32 |
| clarkb | corvus: https://download.jitsi.org/stable/ another option may just be to grab their pacakges? | 14:32 |
| clarkb | though it doesn't look like that is a proper repo :/ | 14:32 |
| corvus | clarkb: that is what their docker file does; my goal is to build the system with a (2 line) patch. | 14:33 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Remove registry mirror settings https://review.opendev.org/714307 | 14:33 |
| corvus | i want to replicate the existing docker images as much as possible, so i figure that means i should start by building debian packages, then install those into my new docker images | 14:34 |
| mordred | fungi: thanks ^^ it was also actually semantically wrong too | 14:34 |
| mordred | corvus: you are very close to nerdsniping me in to wanting to build the underlying zuul support for speculative deb repos to do the thing you're talking about systemically | 14:34 |
| openstackgerrit | Merged opendev/system-config master: Use dev subdir on review-dev for project-config things https://review.opendev.org/714264 | 14:34 |
| openstackgerrit | Merged opendev/system-config master: Install utility scripts for running jeepyb commands https://review.opendev.org/714267 | 14:34 |
| corvus | mordred: don't let me stop you -- but i don't think ill have any image composition here | 14:35 |
| corvus | i'm curious what build platform they use that has node 10 and opnjdk 8 | 14:41 |
| openstackgerrit | Merged openstack/project-config master: check-release-approval: handle no-review case https://review.opendev.org/714066 | 14:44 |
| fungi | corvus: i have node 10 and openjsk 8 installed on my sid/unstable workstation | 14:44 |
| mordred | I think debian buster has those | 14:45 |
| corvus | oh, i thought it didn't have jdk 8 | 14:46 |
| mordred | oh - no - buster has openjdk 11 | 14:46 |
| mordred | weird | 14:46 |
| mordred | yeah - maybe they're using the upstream nodejs package repos for their node packages | 14:47 |
| fungi | yeah, openjdk8 seems to be in unstable but held out of testing from buster onwards | 14:47 |
| clarkb | bionic has openjdk8 and nodejs8 iirc | 14:47 |
| mordred | yeah | 14:47 |
| mordred | so bionic + node10 upstream would give those 2 | 14:48 |
| fungi | "Migration status for openjdk-8 (- to 8u242-b08-1): BLOCKED: Rejected/violates migration policy/introduces a regression" | 14:48 |
| fungi | https://tracker.debian.org/pkg/openjdk-8 | 14:48 |
| corvus | their runtime docker images are stretch, so maybe they use upstream node? | 14:48 |
| fungi | for the record as to why it's not in buster or testing | 14:48 |
| mordred | corvus: ++ | 14:48 |
| mordred | corvus: yeah - I think use upstream node | 14:48 |
| mordred | corvus: that's what we're doing in our gerrit images | 14:48 |
| corvus | and that won't show up as a runtime dependency anyway, so shouldn't be a big deal | 14:49 |
| fungi | and yeah, latest nodejs in stretch is 8 | 14:49 |
| fungi | not 10 | 14:49 |
| *** ouroboros8 has quit IRC | 14:49 | |
| fungi | so they must do something | 14:49 |
| fungi | and while both of those are technically available in debian/sid(unstable), i doubt that's what they're using | 14:50 |
| corvus | yeah, seems unlikely if they're producing docker images running on stretch | 14:51 |
| AJaeger | clarkb, yes, that's my understanding as well - my ping was before that - fungi, this is again about https://review.opendev.org/714174 - what shall we do? | 14:53 |
| AJaeger | clarkb, fungi: we had in the past accessbot as pre-requisite for meetbot - is that a hard requirement? | 14:58 |
| clarkb | AJaeger: yes, but I think we may be able to update the configs for certain channels to drop the registration requirement? | 14:59 |
| clarkb | we may also want to consider dropping it globally? | 14:59 |
| clarkb | we get some spam in the unregistered channel but not much | 14:59 |
| clarkb | of course if we drop the requirement that may make us a more attractive location | 14:59 |
| AJaeger | clarkb: yes, we could fine tune here. If we drop it globally, we might have to revert and then it's back again | 15:09 |
| clarkb | ya, I think that is my biggest concern. If we drop the requirement we'll become interesting again | 15:10 |
| mnaser | hi infra, can we get a hold on https://review.opendev.org/#/c/714340/ for 'openstack-operator:functional' | 15:18 |
| mnaser | it seems to be working in local testing.. so it'd be nice to have an environemnt once it breaks | 15:19 |
| openstackgerrit | Merged opendev/system-config master: Set zuul_work_dir for system-config-*-image https://review.opendev.org/714319 | 15:22 |
| openstackgerrit | Merged opendev/system-config master: Put gerrit image jobs into a project-template https://review.opendev.org/714304 | 15:22 |
| clarkb | mnaser: can we fix logging first? | 15:27 |
| clarkb | (the job indicates there are pods which have not started, but none of the pods have logs) | 15:27 |
| mnaser | clarkb: the logging actually works fine in other jobs which have not failed, we had pod logs | 15:27 |
| mnaser | i am just unsure at what i'm missing from logs to be able to figure out why those pods dont have logs | 15:28 |
| clarkb | mnaser: right probably because the pods have started :) | 15:28 |
| clarkb | I think there is a gap between logging "pods have not started" and "which pods have not started and why" | 15:28 |
| clarkb | I would start with "which pods did not start" | 15:28 |
| mnaser | that is a much easier exercise in figuring out what to capture logs if i have an actual environment with logs that arent running (seeing what state they are, introspecting the commands to output) | 15:29 |
| mnaser | and probably a lot less useful of ~20m of CI time for every iteration till this fails | 15:29 |
| clarkb | I guess I'm saying there are no logs | 15:29 |
| openstackgerrit | Radosław Piliszek proposed openstack/project-config master: Cache CirrOS 0.5.1 https://review.opendev.org/714475 | 15:29 |
| clarkb | starting with something is a good idea? | 15:29 |
| clarkb | (I don't want holding the job to result in not being able to debug this in the future) | 15:30 |
| mnaser | i'm not sure what do you mean by no logs | 15:30 |
| mnaser | kubectl logs? they won't have anything if the pod isn't up | 15:30 |
| clarkb | mnaser: the job says "pods failed to start" then if you go to the pod logs there is no info | 15:30 |
| mnaser | i'm trying to get an autohold so i _never_ have to autohold again | 15:30 |
| clarkb | you are missing output from k8s saying "pods xyz failed for reasons abc | 15:30 |
| mnaser | ok so i should just iterate and spend 20 minutes of ci time on every attempt till i get it right | 15:31 |
| clarkb | I beliee that info is typically available as airship had the same issue a while back | 15:31 |
| mnaser | it'd be nice if airship published their logging stuff to zuul-jobs then :) | 15:31 |
| clarkb | you'd have to take that up with them | 15:32 |
| clarkb | and yes, for many moons the expectation is that jobs add appropriate logging when failures are obtuse | 15:33 |
| clarkb | then if that doesn't clarify things we hold nodes | 15:33 |
| clarkb | this has been true since the early devstack days | 15:33 |
| clarkb | (because it helps us build jobs that properly log around failure conditions) | 15:33 |
| mnaser | the only thing is that will hurt by spending 20 minutes of CI time on every iteration | 15:33 |
| mnaser | so i can borrow a node for a 15 minutes to introspect and generate a kubectl command that will log things out | 15:34 |
| clarkb | if we are going to be sure to add additional logging to the jobs after I'll go ahead and hold the node | 15:37 |
| clarkb | thats really the bit I don't want to miss | 15:37 |
| mnaser | clarkb: i absolutely have no interest in ever wanting to hold instances tbh | 15:38 |
| mnaser | its easier and faster for me to _not_ have to ask for holds | 15:38 |
| mnaser | and the way to avoid it is by adding the missing gaps for logs when introspecting failures | 15:38 |
| mnaser | (and also trying to avoid blindly just logging $world too) | 15:38 |
| clarkb | mnaser: ok its set | 15:39 |
| mnaser | clarkb: thank you for being understanding | 15:39 |
| clarkb | fwiw this seems to be a common issue with k8s | 15:40 |
| clarkb | your idea of adding that bootstrapping to zuul-jobs is a good one "appropriate logging for when k8s things fail) | 15:40 |
| mnaser | clarkb: i absolutely will end up with a depends-on for a zuul/zuul-jobs change from this :) | 15:40 |
| mnaser | maybe like "gather all info" or something | 15:41 |
| clarkb | seems like beacuse k8s retries things until they work (at least that seemed to be the airship problem) if they fail intermittently locally but consistently in the gate just miss the data | 15:41 |
| mnaser | collect-kubernetes-logs .. i dunno, ill find the right place | 15:41 |
| openstackgerrit | Artom Lifshitz proposed openstack/project-config master: Add whitebox-tempest-plugin under QA https://review.opendev.org/714478 | 15:41 |
| clarkb | ya being able to skim k8s for error logs in particular would be useful I think | 15:41 |
| openstackgerrit | Radosław Piliszek proposed openstack/project-config master: Cache CirrOS 0.5.1 for AArch64 too https://review.opendev.org/714481 | 15:49 |
| clarkb | config-core ^ that is probably a good one to land | 15:51 |
| AJaeger | clarkb: so, we bake both AArch64 and x86-64 images into all our media? | 15:58 |
| clarkb | AJaeger: ya, until we do the improvements that ianw would like to do | 15:59 |
| clarkb | I'm not quite sure what that looks like yet, but I'm pretty confident we can sort it out | 15:59 |
| clarkb | AJaeger: these images are like 12MB so it should be fine | 15:59 |
| openstackgerrit | Artom Lifshitz proposed openstack/project-config master: Add whitebox-tempest-plugin under QA https://review.opendev.org/714478 | 15:59 |
| AJaeger | clarkb: that small? Ok... | 16:01 |
| clarkb | AJaeger: ya its basically busybox with scripts taht pretent to be cloud-init and dropbear | 16:01 |
| clarkb | they've done a good job minimzing the images | 16:01 |
| AJaeger | clarkb: do you want to +2A https://review.opendev.org/714475 - that's the x86-64 part. I'll approve the AArch64 now ;) | 16:02 |
| AJaeger | I knew they were good - just not that good ;) | 16:02 |
| clarkb | done thanks | 16:03 |
| AJaeger | we just had a rename slot last week - and now 714478 comes along ;( | 16:06 |
| clarkb | AJaeger: well with releases schedule things for openstack and having just done it that one may just have to wait :) | 16:06 |
| mordred | AJaeger: perhaps we should suggest that they simply move the repo to the openstack tenant for now so that it can sit alongside tempest ... but yeah, we'll need to wait for release schedule to actually do the rename | 16:08 |
| AJaeger | it's in the openstack tenant already, just not in the namespace | 16:09 |
| AJaeger | so, nothing is blocking them to work on it ;) | 16:10 |
| mordred | cool | 16:10 |
| mordred | clarkb: got a sec for https://review.opendev.org/#/c/714305/ ? | 16:12 |
| openstackgerrit | Merged openstack/project-config master: Cache CirrOS 0.5.1 https://review.opendev.org/714475 | 16:12 |
| openstackgerrit | Merged openstack/project-config master: Cache CirrOS 0.5.1 for AArch64 too https://review.opendev.org/714481 | 16:12 |
| AJaeger | clarkb: could you review an old change of yoursr that I updated, please? https://review.opendev.org/650083 | 16:13 |
| clarkb | mordred: AJaeger yup I'll get to those as soon as my meeting is over | 16:14 |
| AJaeger | sure, thanks | 16:14 |
| mnaser | 714340 which has a hold on it has failed, can i get my keys please? https://github.com/mnaser.keys | 16:24 |
| mordred | mnaser: on it | 16:24 |
| mordred | mnaser: done | 16:25 |
| mnaser | thanks mordred | 16:26 |
| *** hashar has joined #opendev | 16:33 | |
| mnaser | mordred, clarkb: thanks, we caught the failure, so we're looking at adding kubelet logs + a kubectl describe of all resources inside the cluster (similar to collect-container-logs) | 16:34 |
| mnaser | this should probably give us all the info that we need, Openk10s still using that vm to test | 16:34 |
| mnaser | and we'll probably have a patch up to zuul/zuul-jobs for both hopefully | 16:34 |
| mordred | mnaser: what's an Openk10s? I feel like I'm not cool enough to know the latest trendy things :) | 16:36 |
| mnaser | mordred: someone's IRC nickname which is a play on words based on their name :P | 16:36 |
| mordred | AH | 16:36 |
| mordred | neat | 16:36 |
| mordred | I was thinking it was some new weird openstack k8s hybrid craziness :) | 16:37 |
| mnaser | also what they're working on :p | 16:37 |
| mordred | clarkb: actually, https://review.opendev.org/#/c/714307/ first (it needs to go in first cause depends-on) | 16:56 |
| mordred | corvus: ^^ if you have a sec to look at that one | 16:56 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Build jitsi-meet images https://review.opendev.org/714494 | 16:59 |
| corvus | okay, that worked locally (at least before i squashed all the RUN lines) | 17:00 |
| mordred | corvus: what are apt-dpkg-wrap and apt-cleanup - are those things I should know about? | 17:04 |
| corvus | mordred: no, they're local to that image | 17:04 |
| corvus | they get copied in from one of the 'rootfs' dirs | 17:04 |
| corvus | they are simple wrapper scripts that do what you expect | 17:05 |
| mordred | ah! gotcha | 17:05 |
| corvus | (that would be cool if they were some kind of universally accessible helper tool; that's what i thought/hoped at first too) | 17:05 |
| openstackgerrit | Artom Lifshitz proposed openstack/project-config master: Add whitebox-tempest-plugin under QA https://review.opendev.org/714478 | 17:06 |
| *** rpittau is now known as rpittau|afk | 17:08 | |
| openstackgerrit | Artom Lifshitz proposed openstack/project-config master: Add whitebox-tempest-plugin under QA https://review.opendev.org/714478 | 17:12 |
| openstackgerrit | Merged openstack/project-config master: Clean up infra gerritbot irc channel configs https://review.opendev.org/650083 | 17:13 |
| openstackgerrit | Merged opendev/system-config master: Remove registry mirror settings https://review.opendev.org/714307 | 17:23 |
| openstackgerrit | Saul Wold proposed openstack/project-config master: Add Rook to StarlingX https://review.opendev.org/713650 | 17:28 |
| *** hashar has quit IRC | 17:30 | |
| mordred | Just going to leave this here for everyone: https://www.youtube.com/watch?v=KKNaYlzssbc | 17:33 |
| *** dpawlik has quit IRC | 17:37 | |
| *** lpetrut has joined #opendev | 17:40 | |
| clarkb | mordred: corvus looks like the upstream jitsi images are not like the ones we already build? They appear to produce somewhat all in one containers | 17:43 |
| clarkb | I don't think that is a major issue, but something to be aware of because we'll end up with a web setup running nginx, and the app, and certbot, etc | 17:43 |
| clarkb | also I expect we need to update https://review.opendev.org/#/c/714494/1/docker/jitsi-meet/web/rootfs/defaults/config.js ? | 17:45 |
| mordred | corvus: welp - I have run the failing task from the openstackclient promote job (the one that fails getting a JWT token) locally and it succeeded | 17:45 |
| mordred | but that was with the password set directly. I guess the next step is to decrypt the zuul secret myself and see what's different? | 17:46 |
| mordred | clarkb: do we have a doc that points out of how to do that decrypt by hand? I have not yet had to do this debugging step | 17:47 |
| clarkb | mordred: ya fungi wrote a thing up | 17:47 |
| clarkb | though not sure if it merged /me looks | 17:47 |
| clarkb | mordred: https://review.opendev.org/#/c/637969/ it did merge | 17:48 |
| mordred | clarkb: cool thanks! | 17:48 |
| corvus | mordred: it probably has a trailing newline | 17:50 |
| mordred | corvus: yeah - it does look that way. does echo secret | zuul_encrypt not do it right I guess? | 17:53 |
| fungi | mordred: echo -n | 17:53 |
| fungi | if you don't want a trailing newline | 17:53 |
| mordred | nod | 17:54 |
| mordred | corvus: so I guess it worked for the intial push because that's using creds written to a config file for docker - but not for the retag since that's passing the creds to the uri module | 17:57 |
| fungi | that sounds likely | 17:58 |
| mordred | yeah - upload uses docker login on the command line - so it doesn't look like that would allow setting a password that validly had a trailing newline in its contents | 17:59 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Build jitsi-meet images https://review.opendev.org/714494 | 17:59 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: jitsi-meet: open etherpad on join https://review.opendev.org/714505 | 17:59 |
| mordred | would it be reasonable to add a strip to the uri password parameter? | 17:59 |
| corvus | mordred: yes, or we could change the default for encrypt-secret to strip | 18:00 |
| mordred | corvus: I don't suppose leading or trailing whitespace are usually valid tokens in these contexts | 18:00 |
| mordred | corvus: maybe both? | 18:00 |
| corvus | mordred: wfm | 18:01 |
| fungi | which makes it harder if folks want to start or end a secret with whitespace, but i suppose if it's clearly documented and there's a --no-strip-whitespace or something then it's probably less confusing than the current behavior | 18:01 |
| corvus | fungi: yeah, there's a --strip option; i think we should just invert the option/default | 18:01 |
| fungi | (because you almost never want to include whitespace at the start or end of a secret) | 18:01 |
| corvus | or at least default strip for stdin | 18:01 |
| corvus | and default to nostrip for file | 18:01 |
| corvus | (but that could be confusing) | 18:01 |
| *** dpawlik has joined #opendev | 18:02 | |
| fungi | that seems reasonable | 18:02 |
| fungi | but yeah, harder to explain | 18:02 |
| fungi | and harder to explain means more likely for folks to trip over | 18:02 |
| * mordred will make some patches | 18:03 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Trim whitespace from uri password for docker promote https://review.opendev.org/714506 | 18:03 |
| fungi | mordred: also corvus's https://review.opendev.org/639771 may be worth looking at | 18:04 |
| mordred | fungi: cool | 18:08 |
| mordred | fungi: btw - in your docs it says "no need to dedent" - I did not find that to be true on zuul.o.o | 18:08 |
| fungi | looking | 18:08 |
| fungi | mordred: did you remove the leading - from the list items? | 18:09 |
| fungi | that's necessary, but removing the whitespace shouldn't be | 18:09 |
| mordred | I did | 18:09 |
| fungi | base64 -d is supposed to eat whitespace | 18:10 |
| mordred | if you do: "cat testsecret | base64 -d | sudo openssl rsautl -decrypt -oaep -inkey /var/lib/zuul/keys/secrets/project/gerrit/openstack/python-openstackclient/0.pem" | 18:10 |
| mordred | in /root on zuul.o.o you can verify | 18:10 |
| mordred | fungi: I get 140095120840344:error:04065072:rsa routines:RSA_EAY_PRIVATE_DECRYPT:padding check failed:rsa_eay.c:602: | 18:11 |
| mordred | also - those docs aren't in system-config any more ... did they move? | 18:12 |
| fungi | indeed, `cat testsecret | base64 -d >/dev/null` reports "base64: invalid input" but `sed 's/^ *//' testsecret | base64 -d >/dev/null` works | 18:12 |
| mordred | I'd fix the docs - but they don't seem to exist any more :) | 18:13 |
| clarkb | they may have got eaten by the removal of the v3 migration docs | 18:14 |
| mordred | ah - good point | 18:14 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Re-add secret decrypting docs https://review.opendev.org/714509 | 18:16 |
| mordred | clarkb, fungi: ^^ there - and updated to include the sed command | 18:17 |
| fungi | actually it got eaten by https://review.opendev.org/678733 | 18:17 |
| fungi | which ripped out a bunch of the stuff i'd already updated our zuul docs with | 18:17 |
| mordred | would it be better to add that seciton to the zuul docs instead? | 18:17 |
| fungi | looks like that change assumed the zuul.rst hadn't already been updated for v3 | 18:18 |
| fungi | and then replaced it with the zuulv3 migration doc | 18:18 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Add meetpad server https://review.opendev.org/714238 | 18:18 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Build jitsi-meet images https://review.opendev.org/714494 | 18:18 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: jitsi-meet: open etherpad on join https://review.opendev.org/714505 | 18:18 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Use our jitsi-meet image for meetpad https://review.opendev.org/714510 | 18:18 |
| fungi | which we then ripped out more recently | 18:18 |
| corvus | okay, i think that stack of 4 changes should completely implement the meetpad spec | 18:18 |
| mordred | corvus: can you explain making opendev-buildset-registry have requires: jitsi-meet-container-image? is that on purpose? | 18:20 |
| corvus | mordred: yes, and we should probably do it for everything; that's where pull-from-intermediate-registry is run, so if we build an image in a previous change, we need that job to see the artifact and pull it so that it's in the intermediate registry | 18:21 |
| mordred | ah | 18:21 |
| corvus | mordred: i expect that stack would fail without it, because #3 builds an image without running the 'run-meetpad' job, and #4 runs 'run-meetpad' without building an image | 18:21 |
| mordred | corvus: yeah - we should totally update our other jobs to do that | 18:22 |
| corvus | mordred: so without that, i would expect #3 to pass, and #4 to fail because there is no image on dockerhub yet | 18:22 |
| corvus | mordred: in the other cases, we're silently falling back to dockerhub | 18:22 |
| corvus | oh, that reminds me, i need to qualify image names in dockerfile | 18:22 |
| corvus | and docker-compose | 18:22 |
| corvus | upstream didn't do that, but i think that's a change we should carry | 18:22 |
| mordred | corvus: requires is already soft right? so it's safe to require and not have something in the stack providing it | 18:23 |
| corvus | mordred: correct | 18:23 |
| mordred | cool | 18:23 |
| corvus | so i think the final state should be to just have a really long list on that job | 18:23 |
| mordred | I think we're fine on the gerrit changes _for_now_ because they're all triggering build at the same time | 18:23 |
| corvus | yeah. i think we should do that once all the gerrit changes have landed | 18:24 |
| mordred | yup | 18:24 |
| corvus | because right now, it's going to be a big merge conflict | 18:24 |
| mordred | so big | 18:24 |
| corvus | mordred: FROM docker.io/library/debian:stretch-slim as build | 18:26 |
| corvus | like that? | 18:26 |
| mordred | corvus: yup | 18:27 |
| mordred | corvus: "Unable to freeze job graph: Project opendev/jeepyb is not allowed to run job system-config-upload-image-gerrit-base | 18:28 |
| mordred | " | 18:28 |
| mordred | I don't have an allowed-projects in system-config ... | 18:28 |
| corvus | mordred: it's probably implied because of the secret | 18:28 |
| mordred | oh. hrm. | 18:29 |
| corvus | mordred: if the 2 projects trust each other with the secret, you can attach it to the pipeline in a config project | 18:29 |
| mordred | ok. I think that's what I have to do in this case | 18:30 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Add meetpad server https://review.opendev.org/714238 | 18:31 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Build jitsi-meet images https://review.opendev.org/714494 | 18:32 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: jitsi-meet: open etherpad on join https://review.opendev.org/714505 | 18:32 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Use our jitsi-meet image for meetpad https://review.opendev.org/714510 | 18:32 |
| openstackgerrit | Monty Taylor proposed openstack/project-config master: Run gerrit image jobs on jeepyb patches https://review.opendev.org/714516 | 18:32 |
| openstackgerrit | Monty Taylor proposed opendev/jeepyb master: Run pep8 job in-repo https://review.opendev.org/714305 | 18:34 |
| openstackgerrit | Monty Taylor proposed opendev/jeepyb master: Clean up some more python3 things https://review.opendev.org/714318 | 18:34 |
| mordred | corvus, clarkb: https://review.opendev.org/714516 should be the right way to run gerrit image jobs on jeepyb | 18:35 |
| openstackgerrit | Monty Taylor proposed opendev/jeepyb master: Run pep8 job in-repo https://review.opendev.org/714305 | 18:35 |
| openstackgerrit | Monty Taylor proposed opendev/jeepyb master: Clean up some more python3 things https://review.opendev.org/714318 | 18:35 |
| openstackgerrit | Monty Taylor proposed opendev/jeepyb master: Run pep8 job in-repo https://review.opendev.org/714305 | 18:36 |
| openstackgerrit | Monty Taylor proposed opendev/jeepyb master: Clean up some more python3 things https://review.opendev.org/714318 | 18:36 |
| corvus | mordred: just thinking out loud: you might be able to add opendev-buildset-registry to the project-template; it should be harmless to have it in there twice (at least as long as we're not setting any attributes like 'dependencies') | 18:37 |
| corvus | i'm not sure i'm ready to advocate that | 18:37 |
| corvus | mordred: +2 as written | 18:37 |
| mordred | corvus: yeah - I had a similar thought and also wasn't 100% ready for that :) | 18:37 |
| * clarkb is catching up on new set of rules for living | 18:41 | |
| mordred | corvus: you got new rules? | 18:41 |
| clarkb | ya bunch of businesses have to close now, but also all sports courts, pools, playgrounds, and skate parks | 18:42 |
| mordred | nod | 18:43 |
| clarkb | which honeslty is a good thing based on my non scientific observations of neighborhoods while biking | 18:43 |
| clarkb | playgrounds and sports courts in particular have been packed | 18:43 |
| mordred | I was sad about the tennis court closure here - because tennis is played at a distance ... but there's still a human who has to collect the money from people, so it makes sense | 18:44 |
| clarkb | mordred: I've seen a lot of basketball and volleyball, neither of which are very distant | 18:46 |
| clarkb | also lots of kids on playgrounds | 18:47 |
| mordred | yah | 18:48 |
| mordred | clarkb: well, when you're back, https://review.opendev.org/#/c/714516/ would be helpful | 18:49 |
| clarkb | ya Ithink I've gotten through it now. No real change for us since we've been social distant already and I get to work from home everyday | 18:49 |
| mordred | yah | 18:51 |
| clarkb | mordred: is pep8 defined in repo? or are not worrying about pep8 anymore? | 18:52 |
| mordred | clarkb: https://review.opendev.org/#/c/714305/ | 18:53 |
| mordred | (which could also do with a +A :) ) | 18:53 |
| mordred | https://review.opendev.org/#/c/714318/ is the ultimate prize here fwiw | 18:53 |
| mordred | clarkb: woot. thanks! | 18:55 |
| mordred | clarkb: I think once that lands we'll have an image we can do a restart on once we're ready to try that | 18:55 |
| openstackgerrit | Merged opendev/system-config master: Re-add secret decrypting docs https://review.opendev.org/714509 | 18:56 |
| fungi | i guess gerritbot hasn't caught up on the infra-manual rename yet | 18:57 |
| fungi | https://review.opendev.org/714518 Restore Zuul v3 Migration Guide bits on secrets | 18:58 |
| fungi | oh right, because review.o.o is still in the emergency disable file | 18:59 |
| fungi | so that part of the rename change never got applied to ir | 19:00 |
| fungi | er, to it | 19:00 |
| openstackgerrit | Merged openstack/project-config master: Run gerrit image jobs on jeepyb patches https://review.opendev.org/714516 | 19:02 |
| mordred | fungi: hopefully soon we'll undisable that ... | 19:05 |
| fungi | yeah, not an inconvenience, was just confused for a moment and thought gerritbot might be broken until i realized what was going on | 19:12 |
| *** lpetrut has quit IRC | 19:12 | |
| *** diablo_rojo__ has joined #opendev | 19:13 | |
| *** ralonsoh has quit IRC | 19:16 | |
| openstackgerrit | Merged opendev/jeepyb master: Run pep8 job in-repo https://review.opendev.org/714305 | 19:21 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add system-config to required-projects for gerrit images https://review.opendev.org/714519 | 19:21 |
| mordred | clarkb, fungi, corvus: ^^ sigh/whoops | 19:22 |
| openstackgerrit | Oleksandr Kozachenko proposed zuul/zuul-jobs master: add role for collecting the kubernetes pod&kubelet logs https://review.opendev.org/714521 | 19:34 |
| *** hashar has joined #opendev | 19:37 | |
| *** diablo_rojo__ has quit IRC | 19:49 | |
| *** diablo_rojo__ has joined #opendev | 19:51 | |
| openstackgerrit | Oleksandr Kozachenko proposed zuul/zuul-jobs master: add role for collecting the kubernetes pod&kubelet logs https://review.opendev.org/714521 | 19:56 |
| openstackgerrit | Oleksandr Kozachenko proposed zuul/zuul-jobs master: add role for collecting the kubernetes pod&kubelet logs https://review.opendev.org/714521 | 20:05 |
| corvus | mordred: oy, i was wrong; we only run the pull role on the docker image build job which is unfortunate for our system-run jobs | 20:17 |
| corvus | i think we can solve this either by running pull in the base job, or making a new almost-base job which adds that onto base (then parent the system-run jobs to that, then we also don't need to whole big list of images on the buildset-registry job) | 20:19 |
| corvus | i'll work on the second thing since it's more self-contained | 20:20 |
| corvus | oh actually, opendev-build-docker-image-base fits that description but just has a weird name | 20:21 |
| mordred | corvus: I agree - the second thing sounds more correcter | 20:21 |
| fungi | #status log manually deleted old /afs/.openstack.org/project/tarballs.opendev.org/openstack/openstackid which was previously copied and switched to .../osf/openstackid/ | 20:23 |
| openstackstatus | fungi: finished logging | 20:23 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Use our jitsi-meet image for meetpad https://review.opendev.org/714510 | 20:24 |
| corvus | so we may only need that for it to work | 20:24 |
| corvus | mordred: then with that we can start adding requires: entries to "systtem-config-run-foo" jobs, which i think will make a lot of sense | 20:25 |
| corvus | mordred: in fact, if that change works, i think i might pull that part out and just do a standalone change to reparent that and start doing that. i think we can land that without throwing a wrench into ether gerrit or meetpad. | 20:26 |
| mordred | corvus: yeah - the requires entries on the run-foo jobs make more logical sense | 20:26 |
| mordred | yeah | 20:26 |
| mordred | corvus: is there any reason to not just put the pull into the base job? like - could we protect it with a check to see if a docker exists? | 20:27 |
| openstackgerrit | Oleksandr Kozachenko proposed zuul/zuul-jobs master: add role for collecting the kubernetes pod&kubelet logs https://review.opendev.org/714521 | 20:28 |
| corvus | mordred: i think we can do that, it will just take a few changes to do (at least 2 changes to zuul-jobs, and then a nice base/base-test series in base-jobs) | 20:29 |
| mordred | corvus: cool. I think that's ultimately worthwhile | 20:29 |
| ianw | clarkb: apropos arm64 did you see https://review.opendev.org/#/c/714346/ to add a new larger memory label? | 20:42 |
| openstackgerrit | Merged opendev/system-config master: Add system-config to required-projects for gerrit images https://review.opendev.org/714519 | 20:42 |
| ianw | i feel like we might have done this for hrw at some point? i have dejavu we discussed it before | 20:43 |
| clarkb | ianw: I hadnt, but ya I think we knew going in with arm64 we'd have to be flexible with sizing since its a different system | 20:43 |
| *** dpawlik has quit IRC | 20:43 | |
| ianw | yeah, i have not i admit looked at the details of why it needs more memory. now i think of it, we should probably add it as a task to see if we can reduce it if/when things are working devstack wise | 20:45 |
| fungi | https://pyfound.blogspot.com/2020/03/new-pip-resolver-to-roll-out-this-year.html | 20:45 |
| ianw | i'll have to find the story ... first some breakfast :) | 20:45 |
| corvus | a change of mine just hit a bunch of retries on several jobs; i investigated one of them, and it appears to have been a host unreachable due to ssh host key changing error | 20:46 |
| corvus | i don't see anything systemically wrong on the graphs right now | 20:47 |
| ianw | fungi: that sounds good ... i guess if you're in the pip game, you already expect some level of release-to-release ... instability; if you don't, you're in for a surprise | 20:47 |
| corvus | maybe call this a data point and see if we get another one | 20:47 |
| mordred | corvus: you're a data point | 20:48 |
| clarkb | corvus: we see that periodically in rax | 20:48 |
| fungi | ianw: we've been lamenting the lack of a resolver in pip since... well since for ever basically | 20:48 |
| clarkb | corvus: from an outside perspective it seems like it will happen for a short ish period of time then they clean up the bad ips | 20:49 |
| clarkb | and it goes away | 20:49 |
| corvus | clarkb: just happened on ovh-bhs1 | 20:49 |
| fungi | some release jobs hit the same thing late last week | 20:49 |
| clarkb | oh thats new then | 20:49 |
| fungi | but i forget which provider | 20:49 |
| clarkb | being a different cloud I mean | 20:49 |
| corvus | i'm also seeing weird errors here: 20:43 < clarkb> ianw: I hadnt, but ya I think we knew going in with arm64 we'd have to be flexible with sizing since its a different system | 20:51 |
| corvus | 20:45 < ianw> yeah, i have not i admit looked at the details of why it needs more memory. now i think of it, we should probably add it as a task to see if we can reduce it if/when things are working devstack | 20:51 |
| corvus | wise | 20:51 |
| corvus | 20:45 < fungi> https://pyfound.blogspot.com/2020/03/new-pip-resolver-to-roll-out-this-year.html | 20:51 |
| corvus | ga | 20:51 |
| corvus | i can apparently not copy/paste from the zuul console stream? | 20:52 |
| ianw | last host key weirdness from my notes i saw was @ http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2020-02-20.log.html#t2020-02-20T00:15:42 in rax | 20:52 |
| ianw | so there's another data point :) | 20:52 |
| corvus | welp | 20:52 |
| corvus | you'll just have to take my word that there's a permission denied error | 20:53 |
| corvus | that seems unrelated to the later unreachable error | 20:53 |
| mordred | fungi: maybe we shoudl start running pip check in devstack after we're installed everything | 20:53 |
| mordred | fungi: just to be sure | 20:53 |
| *** dpawlik has joined #opendev | 20:53 | |
| corvus | it seems very strange to have 2 errors like that | 20:53 |
| corvus | if you look at https://663c0c5f7cea157bd14c-2e4c3ae56b094a6634f1d9ca742de644.ssl.cf2.rackcdn.com/714510/3/check/system-config-run-dns/f7dd2c1/job-output.txt you can see the "Permission denied error" | 20:54 |
| corvus | that later also encountered the unreachable error, but that seems to be after the upload? | 20:55 |
| corvus | on "get df disk usage" ? | 20:55 |
| clarkb | corvus: there are quite a few layers to this based on previous investigation | 20:55 |
| fungi | ah, the recent release job failure i'm finding from thursday hit in inap: https://zuul.opendev.org/t/openstack/build/66b1173250724747bc9b7630277a5b1a | 20:55 |
| clarkb | corvus: things that use the control peristent connection tend to be most reliable | 20:55 |
| corvus | clarkb: i think i'm going to assume this is something different since it's in ovh | 20:55 |
| clarkb | but then you'll get similar errors if ansibel can't exec things on the remote side potentialyl due to running out of disk | 20:55 |
| corvus | and there are 2 errors involved | 20:56 |
| clarkb | which is where the df check came in | 20:56 |
| clarkb | corvus: I just mean in general | 20:56 |
| clarkb | ansible errors aren't always what they seem at first glance | 20:56 |
| corvus | clarkb: if we see the "host key" error from ssh, is that reliable? | 20:56 |
| clarkb | corvus: yes I believe the host key mismatch errors are reliable | 20:57 |
| clarkb | the permission denied can be more generic iirc | 20:57 |
| openstackgerrit | Merged openstack/project-config master: Linaro US: Add a 16GB RAM label for bionic https://review.opendev.org/714346 | 20:58 |
| mnaser | clarkb: as promised - https://0638708d2f7cd0df14a1-68980240b68f224c2b4e2b88589c97a3.ssl.cf2.rackcdn.com/714340/3/check/openstack-operator:functional/4136641/ :) | 20:58 |
| corvus | clarkb: how about this theory: the permission denied is a red herring because these are system-config tests which rewrite our keys. they run after everything else in the cleanup playbook and so errors normally go unnoticed. they are still harmless. | 20:58 |
| clarkb | corvus: ah ya that could be | 20:59 |
| corvus | so maybe what we should do there is add some failed_when lines so that the next time there's an error, we don't spin wheels :) | 20:59 |
| corvus | so the *actual* problem is the permission denied for the docker logs | 20:59 |
| mnaser | infra-root: hold on 0015407770 can be removed, thanks | 21:00 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Use openstackclient from container https://review.opendev.org/711057 | 21:01 |
| mordred | mnaser: cool | 21:01 |
| corvus | okay, i think i see the problem; i think i'll fix it by doing that whole pull in the base job thing | 21:02 |
| mordred | corvus: ++ | 21:03 |
| mordred | corvus, clarkb, mnaser: we currently have opendevorg/python-base pinned to python:3.7 - the latest python image is on 3.8.2 ... how should we approach updating that? | 21:09 |
| clarkb | I want to say we tried updating but something broke under it? | 21:09 |
| mordred | yeah - although I think that might have been stretch->buster related? | 21:10 |
| clarkb | personally I have no issue with sticking to older python | 21:10 |
| mordred | maybe we should make an opendevorg/python-base:3.7 and an opendevorg/python-base:3.8? | 21:10 |
| clarkb | that would probably help us transition | 21:10 |
| clarkb | since we don't have to do it all at once | 21:10 |
| mordred | clarkb: well - I don't either - but at _some_ point updating is probably not a bad idea | 21:10 |
| mordred | kk. I'll make a patch to make a 3.7 and a 3.8 | 21:11 |
| mordred | we can tag :latest still at 3.7 for now since that's all we've had - but maybe go around and either make sure we're pinning consumption-side or that we're ok bumping with a latest bump | 21:12 |
| mordred | corvus: can provides: be a list? | 21:17 |
| corvus | mordred: yes | 21:19 |
| mordred | awesome | 21:19 |
| *** dpawlik has quit IRC | 21:20 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Disable recommends in python-base and python-builder https://review.opendev.org/711073 | 21:31 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Start making 3.8 python images https://review.opendev.org/714532 | 21:31 |
| mordred | clarkb, corvus: I think that should do the right thing with getting us new versions but without breaking anything existing | 21:31 |
| clarkb | mordred: zuul says that has a syntax error | 21:34 |
| mordred | clarkb: zuul is almost certainly right | 21:34 |
| mordred | more importantly ... | 21:34 |
| mordred | putting the system config jobs on jeepyb in project-config did not work: https://review.opendev.org/#/c/714318/ | 21:34 |
| mordred | still getting Unable to freeze job graph: Project opendev/jeepyb is not allowed to run job system-config-upload-image-gerrit-2.13 | 21:35 |
| clarkb | is it a final job? | 21:35 |
| clarkb | or protected? | 21:35 |
| openstackgerrit | Merged zuul/zuul-jobs master: add role for collecting the kubernetes pod&kubelet logs https://review.opendev.org/714521 | 21:35 |
| mordred | nope | 21:36 |
| *** hashar has quit IRC | 21:37 | |
| corvus | mordred: maybe that needs an explicit allowed-projects? | 21:38 |
| mordred | corvus: should I put it on each leaf job I suppose? | 21:39 |
| corvus | mordred: hang tight a minute, i'm reading code | 21:39 |
| corvus | mordred: there is a distinct possibility that the bug here is the job is in a project template | 21:40 |
| mordred | oh! well that's certainly fun | 21:41 |
| corvus | i'm still reading, but that's my current theory | 21:41 |
| corvus | no that's a bad theory | 21:43 |
| corvus | oooh | 21:45 |
| corvus | how about this one: the project-template is not in a config-repo | 21:45 |
| fungi | did we recently manage to remove puppet-beaker-rspec-puppet-4-infra in the opendev tenant without removing references to it? http://zuul.opendev.org/t/opendev/config-errors | 21:46 |
| corvus | mordred: yep, i think that's the bug. the thing that attaches internally to say "it's okay to run this cross-repo" is attached to the project template itself, not the use of the project template in the config project. i think that's a bug, because we may not necessarily want to have a project-template like that defined in a config project. we want the config project to be the one granting permission, | 21:48 |
| corvus | not necessarily defining the template itself. | 21:48 |
| clarkb | fungi: more likely something moved into that tenant that tries to run it? | 21:49 |
| clarkb | fungi: opendev/sytem-config is primarily served by the openstack tenant at this point | 21:49 |
| corvus | mordred: immediate fixes: don't use a project-template or move the project-template to project-config. | 21:49 |
| mordred | corvus: ah - yes, this makes sense | 21:49 |
| mordred | corvus: yeah. I think I'll just skip using the project template for now | 21:49 |
| mordred | fungi: and what clarkb said - the opendev tenant has an incomplete view of system-config and is frequently unhappy | 21:50 |
| mordred | those legacy jobs are one of teh reasons we can't fully move it | 21:50 |
| mordred | since we dont' have openstack-zuul-jobs legacy-base in the opendev tenant (nor should we) | 21:50 |
| clarkb | we're slowly getting there | 21:50 |
| clarkb | we deleted a bunch of puppet jobs for centos and trusty that we didn't need anymore | 21:50 |
| mordred | I've actually got it on my personal cleanup list to go unlegacy those (I'd like to just delete them - but unlegacy-ing them shouldn't be hard) | 21:50 |
| mordred | then we shoudl be able to move most of opendev repos into opendev tenant :) | 21:51 |
| fungi | clarkb: ahh, okay, yeah makes sense | 21:52 |
| openstackgerrit | Monty Taylor proposed openstack/project-config master: Apply gerrit jobs directly to jeepyb https://review.opendev.org/714540 | 21:53 |
| mordred | clarkb, corvus, fungi: ^^ that shoudl work around the zuul issue corvus identitied | 21:53 |
| clarkb | mordred: needs yamling better | 21:54 |
| clarkb | mordred: the trailing : needs removing | 21:54 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Allow pull-from-registry to be used in base jobs https://review.opendev.org/714541 | 21:54 |
| clarkb | or maybe add in dependencies like the others? | 21:55 |
| openstackgerrit | Monty Taylor proposed openstack/project-config master: Apply gerrit jobs directly to jeepyb https://review.opendev.org/714540 | 22:00 |
| mordred | clarkb: or - I need to properly cut-and-paste | 22:00 |
| openstackgerrit | James E. Blair proposed opendev/base-jobs master: Add opendev-buildset-registry-consumer job https://review.opendev.org/714542 | 22:06 |
| corvus | mordred: I started the base-jobs approach in 714541, but i'm kind of leaning toward the explicit parent approach in 714542, mostly just to keep the base job simple. | 22:06 |
| corvus | (there are some trade-offs either way) | 22:06 |
| mordred | nod | 22:06 |
| corvus | oh no i have trailing whitespace | 22:06 |
| ianw | fungi / clarkb : (as the original reviewers on :) https://review.opendev.org/#/c/482203/ do you think we still need NODEPOOL_TARBALLS_PROXY ? i'm not seeing it used anywhere from codesearch | 22:07 |
| openstackgerrit | James E. Blair proposed opendev/base-jobs master: Add opendev-buildset-registry-consumer job https://review.opendev.org/714542 | 22:07 |
| clarkb | ianw: we push a lot of bw through tarballs.o.o based on the goaccess reports | 22:07 |
| clarkb | ianw: that said I'm not sure of what jobs need to use that caching proxy so not sure where to start in making that better | 22:08 |
| ianw | yeah, well if anything is using, it's not, it would seem, being used via the NODEPOOL_TARBALLS_PROXY env we setup | 22:08 |
| fungi | some projects are/were using tarballs.o.o to cache artifacts used by subsequent jobs | 22:09 |
| fungi | also a number of ci jobs consume things like service vm images from tarballs.o.o | 22:09 |
| ianw | ... but do they do that via the reverse proxy on a mirror node? | 22:10 |
| clarkb | ya I think te service vm images may be the biggest usage of it now | 22:10 |
| fungi | i don't know that any of them switched to using it | 22:10 |
| clarkb | ianw: probably not if that didn't end up getting fully realized | 22:10 |
| fungi | and now that it's on afs anyway, we could do it more like we do distro package caches | 22:10 |
| corvus | clarkb: can you go ahead and look at 714542? (it's in base-jobs, so i can't depends-on it) | 22:11 |
| clarkb | corvus: thats a simplification for downstream jobs? | 22:12 |
| corvus | clarkb: it's for downstream jobs, but not a simplification -- it has to run in base-jobs because it uses secrets | 22:12 |
| clarkb | oh right | 22:12 |
| corvus | would we be okay making all of the system-config-run jobs depend on the buildset registry job? | 22:16 |
| corvus | only about half of them use images right now, so half actually don't have to wait for it and can run quicker | 22:16 |
| *** DSpider has quit IRC | 22:16 | |
| clarkb | some things like bridge and the nameservers may never get that relationship | 22:16 |
| clarkb | maybe we want to have a second parent job to capture it for those that do? | 22:16 |
| mordred | it doesn't bother me to have them all wait - but I'm fine with whatever other people want | 22:17 |
| corvus | yeah... or... maybe we should land both changes, and use a job variable to allow pull to no-op for the jobs that don't | 22:18 |
| corvus | https://review.opendev.org/714541 is the other | 22:18 |
| openstackgerrit | Merged opendev/base-jobs master: Add opendev-buildset-registry-consumer job https://review.opendev.org/714542 | 22:19 |
| openstackgerrit | Ian Wienand proposed opendev/base-jobs master: Remove env var for tarballs mirror https://review.opendev.org/714543 | 22:19 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Use our jitsi-meet image for meetpad https://review.opendev.org/714510 | 22:20 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Remove /tarballs proxy from mirrors https://review.opendev.org/714544 | 22:21 |
| clarkb | corvus: mordred I feel like I'm missing some fundamental background thing for these changes | 22:21 |
| clarkb | haven't we been running these jobs with a buildset registry (which forwards to the intermedaite registry) already? | 22:22 |
| corvus | mordred, clarkb: ^ okay, 714510 is what it would look like if we did that. take a look at the run-meetpad job -- we'd just set that variable on the jobs which use it, and do nothing on the jobs that don't. | 22:22 |
| corvus | clarkb: kindasorta | 22:22 |
| mordred | clarkb: we have - but in the case of the system-config-run-* jobs ... | 22:22 |
| mordred | since they don't build the images themselves, there has been an incomplete config | 22:22 |
| clarkb | where were they getting the images before? | 22:23 |
| corvus | (they have been silently falling back to using images from dockerhub, rather than images from previous changes) | 22:23 |
| mordred | dockerhub | 22:23 |
| clarkb | they were still pulling them from the buildset registry right? | 22:23 |
| mordred | yeah | 22:23 |
| mordred | if the change in question built an image - yes - if a change above it in the stack built the image - no | 22:23 |
| clarkb | I see so it is the intermediate registry integration specifically that was missing | 22:24 |
| mordred | clarkb: so - we were pulling from the buildset registry, but missing things only from previous changes from the intermediate registry | 22:24 |
| mordred | yeah | 22:24 |
| clarkb | even if we were running ab uildset registry (I thought those roles always populated the buildset registry from the intermediate registry but I guess not) | 22:24 |
| corvus | the case where we miss a change is pretty rare, we *usually* build and use images together and call it done. but i just made a stack that built an image without using it, then a second change that used it without building it. | 22:25 |
| clarkb | corvus: rather than what https://review.opendev.org/#/c/714541/ does should we just require the buildset registry always then have two sets of parent jobs for system-config-run those with and without docker images? | 22:26 |
| corvus | clarkb: they populate the buildset registry from artifacts zuul tells it about, so the buildset registry job itself, since it's standalone, doesn't get any artifact data, so it doesn't pull anything. it's only the build jobs (or, with these changes, now also the run jobs) which get the artifact data and pull them. | 22:26 |
| corvus | clarkb: then we would have 2 different system-config-run jobs | 22:27 |
| corvus | (and system-config-run isn't trivial, so i don't really want to duplicate it) | 22:28 |
| corvus | (we could use a mix-in parent) | 22:28 |
| clarkb | ya thats what I'm thinking 3 base jobs essentially (or maybe just 2) | 22:28 |
| corvus | (but that's more typing than this) | 22:28 |
| clarkb | basemost does system-config run things without docker then another that does with docker things | 22:28 |
| corvus | clarkb: without using multiple inheritance, we'd have to duplicate this: https://opendev.org/opendev/system-config/src/branch/master/.zuul.yaml#L695-L716 | 22:29 |
| corvus | we could do that without duplication if we did use multiple inheritance though; let me mock that up and see if we like it | 22:30 |
| mordred | I think I prefer just running the pull in more places even if we don't need it everywhere - once we start hitting multi-inheritance it adds a comprehension cost | 22:31 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Start making 3.8 python images https://review.opendev.org/714532 | 22:32 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: WIP: multi inheritance system-config-run-containers https://review.opendev.org/714547 | 22:32 |
| corvus | clarkb, mordred: that's better than i thought it would be | 22:33 |
| corvus | if i'm right about this, that change should actually pass tests, even though its parent will fail (because the zuul-jobs change would need to land first since it's used in a config-project) | 22:34 |
| corvus | clarkb: is that sort of what you were expecting, or have i misunderstood? | 22:36 |
| clarkb | corvus: the shape of it wasn't quite what I expected (because I hadn't fully appreciated that we need to parent the base-jobs job) but the functionality is what I expected | 22:38 |
| clarkb | then ya jobs that use containers can parent to the containers job and those that don't don't | 22:38 |
| mordred | corvus: I agree - it's not as bad as I thought | 22:39 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Start making 3.8 python images https://review.opendev.org/714532 | 22:40 |
| corvus | clarkb: yeah, *something* has to be parented to *something* in opendev/base-jobs because that's where the pull role has to run | 22:40 |
| mordred | corvus: mind +Aing this: https://review.opendev.org/#/c/714540/ real quick? | 22:40 |
| corvus | mordred: +3; hopefully that will work, confirm the bug, then we can fix zuul | 22:41 |
| mordred | corvus: thx | 22:41 |
| mordred | corvus: and yah | 22:41 |
| mordred | also - hopefully it will result in a working gerrit image | 22:42 |
| clarkb | corvus: I think I like 714547 because it makes the behavior explicit | 22:43 |
| clarkb | corvus: its a bit more mind warpy in the config itself, but we don't have to rely on special behavior from the roles | 22:43 |
| corvus | we can probably address the mind-warp with comments | 22:44 |
| mordred | yeah - I think I like 714547 too | 22:44 |
| mordred | although I wasn't expecting to | 22:44 |
| corvus | i'll wait for some green on 547, and if it's heading the right direction, i'll squash it into 510 (which, as predicted, has started failing) | 22:45 |
| clarkb | and ya I agree comments should help with the config understanding | 22:45 |
| corvus | i'll put 541 back in WIP (we could still land it anyway of course) | 22:46 |
| *** persia has quit IRC | 22:46 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Start making 3.8 python images https://review.opendev.org/714532 | 22:47 |
| *** persia_ has joined #opendev | 22:47 | |
| clarkb | I don't know if this has been said yet but I always read meetpad as meatpad. It doesn't help that my local butcher is "The Meating Place" | 22:49 |
| mordred | clarkb: I tnik that's one of the things I like about it | 22:50 |
| ianw | root@bridge:~# /home/ianw/env/bin/openstack --os-cloud=openstackci-rax --os-region=DFW volume list | 22:53 |
| ianw | Invalid volume client version '1'. must be one of: 2, 3 | 22:53 |
| ianw | have i upgraded into a version of openstack client that doesn't support rax or something? | 22:53 |
| clarkb | ianw: sort of | 22:54 |
| clarkb | oh wait | 22:55 |
| clarkb | wow | 22:55 |
| clarkb | so we pinned the version to version 1 in our clouds.yaml | 22:55 |
| clarkb | because thats the one that worked and now its an invalid version? | 22:55 |
| clarkb | mordred: ^ we probably don't watn to do that if we intend on these things to keep working with clouds in the wild | 22:55 |
| mordred | maybe we should unpin that? | 22:55 |
| clarkb | mordred: we can't | 22:55 |
| mordred | one sec | 22:55 |
| clarkb | rax only does volume api v1 | 22:55 |
| fungi | ianw: the openstackclient at ~fungi/launch-env/bin/openstack is still working if you need a point of comparison | 22:55 |
| clarkb | there was like a whole afternoon of figuring this out between keystone api versions and cinder | 22:56 |
| fungi | with OS_CLIENT_CONFIG_FILE=/etc/openstack/all-clouds.yaml | 22:56 |
| clarkb | because they both have to be pinned back | 22:56 |
| ianw | fungi: ahh, 3.19.0 v 4.0.0 | 22:56 |
| mordred | clarkb: it might take me a while to fix osc in that respect - we haven't changed osc to use sdk for cinder yet | 22:56 |
| openstackgerrit | Merged openstack/project-config master: Apply gerrit jobs directly to jeepyb https://review.opendev.org/714540 | 22:56 |
| mordred | but it's obviously on the list :) | 22:56 |
| ianw | # Try a small import to check if cinderclient v1 is supported | 22:58 |
| ianw | i am guessing this fails | 22:58 |
| ianw | https://opendev.org/openstack/python-openstackclient/src/branch/master/openstackclient/volume/client.py#L44 | 22:58 |
| mordred | clarkb: rax supports v2 | 22:59 |
| clarkb | mordred: it didn't work | 22:59 |
| mordred | >>> c.block_storage.get('https://dfw.blockstorage.api.rackspacecloud.com/').json() | 22:59 |
| mordred | {'versions': [{'status': 'SUPPORTED', 'updated': '2014-06-28T12:20:21Z', 'id': 'v1.0', 'links': [{'href': 'http://dfw.blockstorage.api.rackspacecloud.com/v1/', 'rel': 'self'}]}, {'status': 'CURRENT', 'updated': '2012-11-21T11:33:21Z', 'id': 'v2.0', 'links': [{'href': 'http://dfw.blockstorage.api.rackspacecloud.com/v2/', 'rel': 'self'}]}]} | 22:59 |
| ianw | I get "public endpoint for volumev2 service in DFW region not found" from cmd line | 22:59 |
| clarkb | ianw: ya that | 22:59 |
| mordred | ignore the cmd line for a sec . | 22:59 |
| mordred | let's figure out what actually works and then what we need to do to configure things to use | 22:59 |
| mordred | it | 23:00 |
| clarkb | mordred: my understanding of the issue was their catalog does not list the v2 api | 23:00 |
| clarkb | and that is how tools like osc decide whattehy can talk to | 23:00 |
| clarkb | if the service itself does offer it then the catalog hasn't been updated | 23:00 |
| clarkb | (and that was sorted out after pinning the keystone api back first) | 23:01 |
| clarkb | (because only keysteon v2 works) | 23:01 |
| mordred | yes - that's right - there is a missing catalog entry | 23:01 |
| mordred | however - we should just put in an endpoint override | 23:01 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Use our jitsi-meet image for meetpad https://review.opendev.org/714510 | 23:01 |
| mordred | since block-storage v2 is there | 23:01 |
| mordred | and that's much more supportable | 23:02 |
| corvus | mordred, clarkb: it looked like it was shaping up as expected, so i updated 510 to use multiple-inheritance. | 23:02 |
| mordred | (I agree, we shouldnt' have osc not supporting v1 - and we'll fix that) | 23:02 |
| clarkb | mordred: can we do that separately for each region? | 23:02 |
| mordred | clarkb: yes | 23:02 |
| mordred | clarkb: we can also use a substitution variable on region name | 23:02 |
| mordred | but I think doing it per-region is likely more readble | 23:02 |
| mordred | clarkb, ianw: I need to afk for a minute - but I really want to get this sorted properly so that we don't have to remember to use an old install of osc that's sitting /root when we talk to rax | 23:03 |
| mordred | so - I'll get a patch up when I get back if y'all haven't gotten to is already | 23:04 |
| clarkb | mordred: I'm still not sure how to express this the way you are suggesting fwiw | 23:04 |
| clarkb | so if you are able to push that might be a good thing | 23:04 |
| ianw | ok ... we are sayign we need something in our clouds config to override this? | 23:04 |
| mordred | yes | 23:04 |
| clarkb | ianw: ya per region cinder endpoint overrides to ignore what the catalog is telling us | 23:05 |
| clarkb | ianw: I don't know how to do that but apparently it is possible :) | 23:05 |
| ianw | alright, i'll do some reading too :) | 23:05 |
| mordred | clarkb: https://docs.openstack.org/openstacksdk/latest/user/config/configuration.html#per-region-settings | 23:07 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: WIP override rax block storage endpoint https://review.opendev.org/714553 | 23:08 |
| mordred | clarkb, ianw: something like that | 23:09 |
| mordred | of course -we should probably just put that into the vendor settings for rax too | 23:09 |
| clarkb | we'll want to https those and then add it to all the clouds.yaml files with rax regions | 23:10 |
| mordred | yeah | 23:10 |
| mordred | that's just the real quick "this is what it should look like" | 23:10 |
| mordred | but there may be more rabbit hole for me to go down :) | 23:10 |
| clarkb | yup that helps a lot specifically the key for the override | 23:10 |
| clarkb | (that doesn't seem to be documented in any of the docs I can find) | 23:10 |
| mordred | I just linked to the docs for it | 23:11 |
| mordred | see right before the patch | 23:11 |
| mordred | oh - the endpoint_override? | 23:11 |
| mordred | nod | 23:11 |
| mordred | that's a standard keystoneauth setting - we should get that document in a better place - great point | 23:11 |
| mordred | clarkb: fwiw - this: https://opendev.org/openstack/openstacksdk/src/branch/master/openstack/config/vendors/conoha.json#L5 is also possible | 23:12 |
| mordred | when we find something that's working for us, we should update https://opendev.org/openstack/openstacksdk/src/branch/master/openstack/config/vendors/rackspace.json too | 23:12 |
| mordred | ok - I'm gonna afk - will follow up on this | 23:12 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: WIP override rax block storage endpoint https://review.opendev.org/714553 | 23:14 |
| clarkb | that ps touches all the files it needs to I think | 23:14 |
| clarkb | I haven't tested this yet | 23:14 |
| clarkb | ianw: ^ is that something you might be able to do since you've got the newer non working install? | 23:15 |
| ianw | yep | 23:15 |
| ianw | clarkb:hrm, it does no, a priori, appear to work. debugging a bit now | 23:19 |
| ianw | being careful not to paste passwords! :) | 23:20 |
| *** diablo_rojo__ is now known as diablo_rojo | 23:21 | |
| ianw | - name: DFW | 23:22 |
| ianw | values: | 23:22 |
| ianw | block_storage_endpoint_override: https://dfw.blockstorage.api.rackspacecloud.com/v2/ | 23:22 |
| ianw | i think this is typo free | 23:22 |
| clarkb | https://docs.openstack.org/openstacksdk/latest/user/config/configuration.html#per-region-settings and seems to match this doc | 23:23 |
| clarkb | perhaps block_storage_endpoint_override isn't quite what we need? | 23:23 |
| ianw | Instantiating volume client: <class 'cinderclient.v2.client.Client'> | 23:27 |
| ianw | Making authentication request to https://identity.api.rackspacecloud.com/v2.0/tokens | 23:27 |
| ianw | https://identity.api.rackspacecloud.com:443 "POST /v2.0/tokens HTTP/1.1" 200 1790 | 23:27 |
| ianw | public endpoint for volumev2 service in DFW region not found | 23:27 |
| ianw | it doesn't seem like we're at the point of even talking to the blockstorage api? | 23:28 |
| clarkb | ya that looks like it is still interrogating the catalog from keystone and deciding it can't do the thing | 23:28 |
| clarkb | possible this is a bug in the tooling where it still checks even with an override | 23:29 |
| ianw | it's not clear to me how that setting translates into the config dict @ https://opendev.org/openstack/openstacksdk/src/branch/master/openstack/config/cloud_region.py#L216 | 23:44 |
| clarkb | ianw: https://opendev.org/openstack/openstacksdk/src/branch/master/openstack/config/cloud_region.py#L463-L464 there maybe? | 23:46 |
| ianw | that's looking for a config section though? more like endpoint_override: block_storage: https://... ? | 23:47 |
| clarkb | ianw: https://opendev.org/openstack/openstacksdk/src/branch/master/openstack/config/cloud_region.py#L382-L385 no its doing weird concatenation of terms | 23:48 |
| ianw | clarkb: instrumenting it, https://opendev.org/openstack/openstacksdk/src/branch/master/openstack/config/cloud_region.py#L390 only gets called with a key of "interface" | 23:53 |
| clarkb | ianw: so it never calls the block-storage version of that, interesting | 23:53 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!