| *** jamesmcarthur has joined #zuul | 00:06 | |
| *** jamesmcarthur has quit IRC | 00:18 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Add support for selecting the kubernetes version https://review.opendev.org/688743 | 00:28 |
|---|---|---|
| *** igordc has quit IRC | 01:47 | |
| *** spsurya has joined #zuul | 01:59 | |
| *** bhavikdbavishi has joined #zuul | 03:24 | |
| *** bhavikdbavishi1 has joined #zuul | 03:27 | |
| *** bhavikdbavishi has quit IRC | 03:28 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 03:28 | |
| *** raukadah is now known as chandankumar | 04:29 | |
| *** pcaruana has joined #zuul | 04:37 | |
| *** pcaruana has quit IRC | 05:14 | |
| *** saneax has joined #zuul | 05:52 | |
| *** pcaruana has joined #zuul | 06:13 | |
| *** hashar has joined #zuul | 06:19 | |
| *** tosky has joined #zuul | 07:17 | |
| *** themroc has joined #zuul | 07:19 | |
| *** jpena|off is now known as jpena | 07:39 | |
| *** bolg has joined #zuul | 07:39 | |
| *** bhavikdbavishi has quit IRC | 07:40 | |
| bolg | tobiash: I was toying yesterday with the mandatory SQL connection (https://review.opendev.org/688755) a bit. This would need to adapt most of the tests which use ZuulTestCase to ZuulDBTest case. Just wanted to check if that is what is meant in https://review.opendev.org/c/621479 since this would be mean significant efforts. | 07:45 |
| bolg | *ZuulDBTestCase | 07:45 |
| tobiash | bolg: actually I started with this already a few months ago: https://review.opendev.org/630472 | 07:47 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Enforce sql connections for scheduler and web https://review.opendev.org/630472 | 07:49 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Enforce sql connections for scheduler and web https://review.opendev.org/630472 | 07:54 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: WIP: Add spec for scale out scheduler https://review.opendev.org/621479 | 07:56 |
| bolg | tohiash: ok missed that, i will look at it | 08:00 |
| *** themr0c has joined #zuul | 08:02 | |
| *** themroc has quit IRC | 08:03 | |
| fbo | Hi @zuul-maint is there any blocker that prevent the Elasticsearch reporter to be merged ? https://review.opendev.org/#/c/644927/ | 08:06 |
| bolg | tobiash: maybe we should sync sometime on the topic of scale-out-scheduler | 08:09 |
| *** jangutter has joined #zuul | 08:14 | |
| *** gtema has joined #zuul | 08:21 | |
| *** themr0c has quit IRC | 08:38 | |
| *** themr0c has joined #zuul | 08:38 | |
| *** avass has joined #zuul | 08:48 | |
| *** hashar has quit IRC | 08:50 | |
| openstackgerrit | Felix Schmidt proposed zuul/zuul master: Build GitHub URLs for tags correctly https://review.opendev.org/688882 | 08:59 |
| *** hashar has joined #zuul | 09:01 | |
| *** themr0c has quit IRC | 09:09 | |
| *** panda is now known as panda|drappt | 09:23 | |
| *** hashar has quit IRC | 09:57 | |
| openstackgerrit | Felix Schmidt proposed zuul/zuul master: Build GitHub URLs for tags correctly https://review.opendev.org/688882 | 10:02 |
| *** gtema has quit IRC | 10:08 | |
| *** themroc has joined #zuul | 10:13 | |
| *** fsvsbs has joined #zuul | 10:14 | |
| *** fdegir has quit IRC | 10:16 | |
| *** fdegir has joined #zuul | 10:17 | |
| *** themr0c has joined #zuul | 10:17 | |
| *** themroc has quit IRC | 10:18 | |
| *** themr0c has quit IRC | 10:20 | |
| *** themr0c has joined #zuul | 10:20 | |
| *** hashar has joined #zuul | 10:29 | |
| *** themroc has joined #zuul | 10:34 | |
| *** armstrongs has joined #zuul | 10:34 | |
| *** themr0c has quit IRC | 10:34 | |
| *** bhavikdbavishi has joined #zuul | 10:35 | |
| *** bhavikdbavishi1 has joined #zuul | 10:38 | |
| *** bhavikdbavishi has quit IRC | 10:39 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 10:39 | |
| *** themr0c has joined #zuul | 10:44 | |
| *** themroc has quit IRC | 10:45 | |
| *** themr0c has quit IRC | 10:57 | |
| *** rfolco|ruck is now known as rfolco|brb | 11:01 | |
| *** jpena is now known as jpena|lunch | 11:27 | |
| *** markw18 has joined #zuul | 11:36 | |
| markw18 | Hi there, | 11:37 |
| markw18 | Apologies still typing I just hit enter by mistake | 11:37 |
| markw18 | I was just wondering what the rational was around this commit: https://opendev.org/zuul/zuul/commit/d014e3512548939178683a47eaefa4a1a7b79573 we are currently using the hashicorp_vault plugins to lookup secrets within our Zuul jobs, however after updating to the latest version of Zuul we found the plugin to be blocked. | 11:38 |
| markw18 | We would really love it if there was a way of having a whitelist of allowed lookup plugins as it's really only the vault lookup plugins that we depend upon | 11:40 |
| markw18 | Failing that are there any plans on the cards for directly offering support for looking up secrets from vault actually within the Zuul app itself ? | 11:41 |
| AJaeger | markw18: better look at https://review.opendev.org/#/c/681041/ | 11:41 |
| AJaeger | So, shouldn't this *Fix* using hashi_vault? | 11:42 |
| markw18 | No this actually blocks the usage of the vault plugin, so we now see messages like the following: | 11:43 |
| markw18 | An unhandled exception occurred while running the lookup plugin 'hashi_vault'. Error was a <class 'ansible.errors.AnsibleError'>, original message: Use of lookup modules that perform local actions on the executor is forbidden. | 11:43 |
| AJaeger | ah - so better go back to the orginal broken prevent | 11:43 |
| AJaeger | markw18: https://review.openstack.org/454236 is the change you want to look at | 11:44 |
| markw18 | Yes thanks :) we are able to do that for now but we were just wondering what the plans were long term ? I.E is there plans to add a whitelist of allowed plugins and or adding the support for looking up secrets from hashicorp vault directly in Zuul ? | 11:45 |
| AJaeger | markw18: sorry, can't answer that, you might need to wait until others are awake | 11:45 |
| markw18 | No worries thanks anyway much appreciated ! | 11:46 |
| fungi | markw18: if you look at the other plugins allowed, you'll see many of them need to stub out dangerous methods which allow things like writing files to disk. would probably need to audit that plugin for similar risks | 11:47 |
| fungi | https://review.opendev.org/662870 is one i did not long ago because we wanted to be able to use the password lookup plugin to generate random passwords in jobs | 11:48 |
| openstackgerrit | Merged zuul/zuul master: Remove python-path auto release note https://review.opendev.org/688809 | 11:49 |
| markw18 | Ok, in the case of the hasicorp vault plugin it shouldn't be doing anything other than looking up the secrets and not writing to disk etc, thanks for the PR link I will have a read | 11:50 |
| fungi | markw18: https://opendev.org/zuul/zuul/commit/19def7045e170b0bb50f8d8d0090ff10a9c003cf may be an easier way to read it | 11:52 |
| fungi | for some reason gerrit isn't displaying the zuul/ansible/base/lookup/password.py addition (it replaces a symlink, so that's probably why) | 11:54 |
| *** rfolco|brb is now known as rfolco|ruck | 11:55 | |
| fungi | the password lookup module had functions which allowed specifying a file path, and writing the result to a temporary file | 11:55 |
| fungi | those seem innocuous, but could in theory be used to replace or read arbitrary files as the user running ansible | 11:56 |
| markw18 | Perfect thanks :) much appreciated | 11:56 |
| fungi | so just need to read through the vault plugin and make sure the things its able to do are tightly scoped and that we don't expose functions or parameters which could cause it to inadvertently do more | 11:57 |
| fungi | though we've had discussions about dropping the current plugin filtering entirely and just relying on bubblewrap's sandboxing as the only layer of security there | 11:58 |
| *** themr0c has joined #zuul | 11:59 | |
| fungi | it would be easier on everyone, but requires we put an increased amount of faith in the process and filesystem isolation measures bwrap provides | 11:59 |
| markw18 | Indeed, although I'd say if possible then yes that would be great | 12:00 |
| zbr | is there a generic way to define an env variable for a zuul job? | 12:12 |
| *** jamesmcarthur has joined #zuul | 12:13 | |
| *** jamesmcarthur has joined #zuul | 12:13 | |
| *** rlandy has joined #zuul | 12:14 | |
| zbr | nevermind, i found one way of doing it, i was confused by tox_environment variable with is an shell env, not the tox env :p | 12:16 |
| zbr | very confusing name | 12:17 |
| *** jamesmcarthur has quit IRC | 12:26 | |
| *** jpena|lunch is now known as jpena | 12:31 | |
| jangutter | (wall-of-text-start) Found two interesting entertaining digressions yesterday and today just posting them here for chuckle values. | 12:41 |
| jangutter | First one: we're running software-factory and zuul, and due to the scl python there are persistent ansible warnings for long unix domain socket paths. | 12:41 |
| jangutter | Symlinks don't help, since the full path is resolved. The workaround: set job_dir=/var/lib/zuul/b to save 3 extra characters. | 12:41 |
| jangutter | Second one: we've been scratching our head with Ansible unreachable errors. Turns out, if your script returns 255, it looks like an ssh error. | 12:41 |
| jangutter | Workaround, _don't return -1 as an error, return 1_. (wall-of-text-end) | 12:42 |
| openstackgerrit | David Shrewsbury proposed zuul/nodepool master: Pull minikube log data https://review.opendev.org/688775 | 12:45 |
| Shrews | jangutter: yeah, i've encountered the too-long-socket-path error in zuul in the recent past. I forget what problem I was seeing, but it was not obvious based on the error | 12:52 |
| *** hashar has quit IRC | 12:52 | |
| *** jamesmcarthur has joined #zuul | 12:52 | |
| jangutter | Shrews: to be honest, I think it's a warning, not a problem - but since it was the last error we saw, we thought that was the cause. | 12:53 |
| Shrews | it created some sort of error for me. like i said though, don't recall the context :/ | 12:54 |
| jangutter | Shrews: Ansible seems to fall back to some other working strategy. It makes a bit of noise in the logs though. | 12:54 |
| Shrews | swest: we seem to have a race in your new test_static_waiting_handler_order test: https://f029a83c8590bcf2444a-fd1d77df4b8e8d30c5a944f1596a25a0.ssl.cf5.rackcdn.com/688775/11/check/tox-py36/ad10859/testr_results.html.gz | 13:24 |
| Shrews | swest: any chance you can look into that? | 13:24 |
| *** michael-beaver has joined #zuul | 13:28 | |
| *** saneax has quit IRC | 13:40 | |
| *** bhavikdbavishi has quit IRC | 13:41 | |
| *** jangutter_ has joined #zuul | 13:57 | |
| *** brendangalloway has joined #zuul | 13:58 | |
| *** jangutter_ has quit IRC | 14:00 | |
| *** jangutter has quit IRC | 14:00 | |
| *** jangutter has joined #zuul | 14:01 | |
| *** swest has quit IRC | 14:13 | |
| corvus | i'll tag 88708660f774e506681921bb315c07c25d78e0e5 as zuul 3.11.0 -- sound good? | 14:28 |
| *** mgoddard has quit IRC | 14:28 | |
| *** mgoddard has joined #zuul | 14:30 | |
| fbo | I'm getting zuul-build-image and zuul-quick-start job failure on a patch that seems to not be related https://review.opendev.org/#/c/687351/ . Is there an issue with those jobs atm ? | 14:34 |
| corvus | fbo: could be an issue with the intermediate registry; i'll take a look in a minute | 14:34 |
| *** chandankumar is now known as raukadah | 14:35 | |
| fbo | corvus: thanks | 14:36 |
| *** jangutter has quit IRC | 14:39 | |
| *** jangutter has joined #zuul | 14:40 | |
| *** jangutter_ has joined #zuul | 14:44 | |
| *** jangutter has quit IRC | 14:47 | |
| corvus | fbo: 679938 had a post-failure on zuul-build-image, and it looks like the later changes were trying to get the images from the failed build; i think if we recheck them now, they'll find the new successful build. i've done that. | 14:49 |
| corvus | (this is probably something about the provides/requires that we could improve) | 14:49 |
| *** avass has quit IRC | 14:52 | |
| fbo | ok thanks we'll see if the jobs pass | 14:52 |
| *** jangutter has joined #zuul | 14:59 | |
| clarkb | corvus: yes looks like opendev is running b768ece and the only changes after that to 88708660f774e506681921bb315c07c25d78e0e5 are bindep and release note changes | 15:00 |
| clarkb | corvus: 88708660f774e506681921bb315c07c25d78e0e5 as 3.11.0 lgtm | 15:01 |
| *** jangutter_ has quit IRC | 15:03 | |
| corvus | cool, i'll push it then | 15:03 |
| *** panda|drappt is now known as panda | 15:09 | |
| *** mattw4 has joined #zuul | 15:20 | |
| openstackgerrit | David Shrewsbury proposed zuul/nodepool master: WIP: Cleanup k8s playbook https://review.opendev.org/688961 | 15:37 |
| *** hashar has joined #zuul | 15:43 | |
| *** rfolco|ruck has quit IRC | 15:49 | |
| *** themr0c has quit IRC | 15:52 | |
| Shrews | hrm, the nodepool k8s job seems to always fail in ovh | 15:58 |
| *** jpena is now known as jpena|off | 16:00 | |
| Shrews | tristanC: can you glean any useful info from https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_b7b/688961/1/check/nodepool-functional-k8s/b7b20d5/minikube.txt ? | 16:00 |
| clarkb | Failed to list *v1.Namespace: Get https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: connect: no route to host | 16:01 |
| clarkb | a networking problem I think | 16:01 |
| clarkb | dumping ip addr and ip route output might help | 16:02 |
| Shrews | coredns (whatever that is) seems to be crashing? | 16:02 |
| corvus | clarkb: this is from the start of the job https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_b7b/688961/1/check/nodepool-functional-k8s/b7b20d5/zuul-info/zuul-info.ubuntu-xenial.txt | 16:02 |
| corvus | clarkb: i guess you mean after the minikube crash? | 16:02 |
| clarkb | corvus: yes after k8s has modified things | 16:03 |
| tristanC | Shrews: coredns was also crashing in successfull logs | 16:03 |
| *** bolg has quit IRC | 16:03 | |
| Shrews | tristanC: oh neat :) | 16:03 |
| tristanC | Shrews: looking for the pod id, this line seems suspicious: kubelet[7911]: W1016 15:49:12.115521 7911 docker_sandbox.go:394] failed to read pod IP from plugin/docker: Couldn't find network status for main-0000000001/pod-fedora through plugin: invalid network status for | 16:05 |
| clarkb | is it possible the /32 network for the host confuses k8s? | 16:05 |
| tristanC | unfortunately it seems like the line is truncated right before valuable information :) | 16:05 |
| clarkb | (that is a known network difference on ovh) | 16:05 |
| tristanC | Shrews: iiuc it is a docker related issue, do we collect docker logs too? | 16:06 |
| Shrews | tristanC: doesn't appear we do | 16:07 |
| corvus | Shrews, tristanC: there's some code in system-config to grab all docker logs | 16:09 |
| Shrews | corvus: oh excellent. i'll see about adding that to the job. thx | 16:09 |
| corvus | Shrews: playbooks/zuul/run-base-post.yaml | 16:10 |
| *** gtema has joined #zuul | 16:11 | |
| *** markw18 has quit IRC | 16:12 | |
| *** rfolco has joined #zuul | 16:15 | |
| Shrews | i think that's probably doing what 'minikube logs' is already doing? just pulling the container logs | 16:15 |
| *** rfolco has quit IRC | 16:18 | |
| tristanC | Shrews: is the FN ipv6 issue solved for the openshift integration test? | 16:18 |
| *** rfolco has joined #zuul | 16:18 | |
| Shrews | tristanC: not sure. i haven't noticed any failures and haven't actively been looking for successes | 16:19 |
| *** hashar has quit IRC | 16:19 | |
| *** rfolco is now known as rfolco|ruck | 16:20 | |
| clarkb | Shrews: tristanC we believe the underlying network issue is fixed | 16:20 |
| clarkb | whether or not that was the only issue with that job on that platform I don't know | 16:20 |
| tristanC | clarkb: ianw: thanks a lot for working on the fedora ipv6 issue! | 16:28 |
| *** gtema has quit IRC | 16:28 | |
| Shrews | ah, the entire /var/logs/docker contents are pulled in the system-config playbook. i think that's what we need | 16:29 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Fix skipping child jobs with soft dependencies https://review.opendev.org/688971 | 16:30 |
| corvus | clarkb, tobiash: ^ that bug was introduced in the just-release 3.11.0 -- maybe we want to land that and cut 3.11.1 ? | 16:33 |
| corvus | also, you can see the bug in action now at http://zuul.opendev.org/t/openstack/status | 16:33 |
| corvus | (system-config changes in check) | 16:33 |
| clarkb | reviewing it now | 16:36 |
| *** hashar has joined #zuul | 16:38 | |
| tosky | uhm, my https://review.opendev.org/#/c/674334/ heavily conflicts with tristanC's https://review.opendev.org/#/c/681882/5 | 16:40 |
| corvus | tosky: yesterday we were discussing reworking tristanC's patches... it sounds to me like that may be the approach we take, so i think you may be clear to proceed, and the new versions of tristanC's patches should be simpler | 16:42 |
| tobiash | corvus, clarkb: lgtm and ++ for release | 16:42 |
| corvus | tobiash, clarkb: thanks -- i'll wait for that to land -- should we restart opendev zuul before releasing 3.11.1, or do we think this is tested enough? | 16:43 |
| clarkb | the test case should cover it pretty well I think. But happy to restart opendev zuul as long as openstack release team is done with their release jobs | 16:44 |
| clarkb | (I think they are but we should confirm) | 16:44 |
| tobiash | since we have test cases for soft deps, hard deps and now this case I guess the test cases are probably enough | 16:45 |
| tobiash | but if you feel more comfortable testing it in opendev go ahead | 16:45 |
| *** mattw4 has quit IRC | 16:57 | |
| *** ryanpetrello has joined #zuul | 17:00 | |
| ryanpetrello | o/ fungi | 17:00 |
| *** mattw4 has joined #zuul | 17:00 | |
| ryanpetrello | any chance you have control over the contents of https://superuser.openstack.org/articles/zuul-community-answers-frequent-questions-from-ansiblefest/ ? | 17:00 |
| ryanpetrello | specifically, "Tower, on the other hand, is meant to trigger arbitrary Ansible playbooks on demand via arbitrary human inputs." | 17:01 |
| clarkb | ryanpetrello: we know the people that can update that | 17:01 |
| ryanpetrello | as of very recently, this isn’t true, AWX/Tower has an HTTP API you can integrate with, and it also has direct API support for triggering playbook runs via webhook events from github and gitlab | 17:01 |
| clarkb | ryanpetrello: what would be a more appropriate differentiation? | 17:01 |
| ryanpetrello | zuul's control over this is much more robust and configurable via source control like other zuul things | 17:02 |
| clarkb | (also I totally solicited feedback on that on the zuul mailing list :) ) | 17:02 |
| ryanpetrello | AWX/Tower is controlled by an API (you wouldn't put the config in source control, for instance) | 17:02 |
| ryanpetrello | yea, it's a _very_ recent change we've made to AWX | 17:02 |
| ryanpetrello | like, in the past month | 17:02 |
| ryanpetrello | but it *is* a bit of functionality we're proud of, and it's a use case AWX/Tower users have wanted for some time, so I'm just interested in having accurate info out there | 17:03 |
| ryanpetrello | zuul definitely has the upper hand here in terms of robustness | 17:03 |
| ryanpetrello | but AWX/Tower *does* have some support for webhook integration with both Github and Gitlab today | 17:03 |
| ryanpetrello | (we're zuul users ourselves, because it's awesome :D) | 17:04 |
| fungi | as far as our faq in the zuul docs is concerned, we could likely do a better job there of suggesting where zuul/tower integration opportunities lie, and how they can be complimentary, to defuse the competitive image some folks might imagine | 17:05 |
| clarkb | ryanpetrello: what about "Historically Tower has primarily been used to trigger Ansible playbooks via arbitrary human inputs. Recently Tower has grown an HTTP API and support for GitHub and Gitlab webhooks. While these new features overlap with Zuul you will get a more robust CI experience through Zuul as it has built in support for features like gating." is that better? | 17:05 |
| fungi | "but don't these do the same thing? nope, in fact here's what you can do if you combine them..." | 17:05 |
| *** michael-beaver has quit IRC | 17:08 | |
| clarkb | ryanpetrello: basically I'm happy to provide edits to the superuser team, but am not quite sure what the best way to position this is. Zuul and Tower are different ( as evidenced by tower using zuul ) | 17:08 |
| clarkb | unrlated to tower https://twitter.com/ZuulCI/status/1184480187251068928 | 17:09 |
| *** jamesmcarthur has quit IRC | 17:09 | |
| *** calebb has joined #zuul | 17:09 | |
| corvus | and in fact, we have discussed having opendev's systems managed by awx and having that triggered by zuul -- that's the kind of 'use both systems for complete coverage' that i think is suggested by fungi's comment | 17:10 |
| corvus | sometimes you need to compare apples and oranges, and it's hard. :) | 17:13 |
| fungi | i often compare them while eating fruit salad | 17:14 |
| fungi | apples and oranges can go very well together | 17:14 |
| corvus | ++ | 17:14 |
| *** jangutter has quit IRC | 17:17 | |
| ryanpetrello | @clarkb I think if it mentioned how zuul had more robust support for handling events from source control, that would be most accurate | 17:26 |
| ryanpetrello | but AWX/Tower does have *some* support, so the current text blurb is inaccurate | 17:26 |
| ryanpetrello | @clarkb your suggestion above looks good to me | 17:26 |
| clarkb | ryanpetrello: ok let me make an etherpad with the entire paragraph just to confirm and then I'll get that to the superuser team | 17:27 |
| ryanpetrello | perhaps: | 17:27 |
| ryanpetrello | Recently Tower has grown HTTP API support for GitHub and Gitlab webhooks | 17:27 |
| ryanpetrello | (we've had an HTTP API _itself_ for years) | 17:27 |
| clarkb | rgr | 17:27 |
| ryanpetrello | thanks! | 17:27 |
| openstackgerrit | David Shrewsbury proposed zuul/nodepool master: WIP: k8s logging https://review.opendev.org/688961 | 17:29 |
| clarkb | ryanpetrello: https://etherpad.openstack.org/p/SC1gfnxUNe how does that look | 17:31 |
| clarkb | ryanpetrello: I tried to incorporate the bit about source code events too | 17:36 |
| ryanpetrello | yep, I think that's a great summary | 17:37 |
| ryanpetrello | thanks for putting that together | 17:37 |
| clarkb | np passing that along now | 17:37 |
| fungi | thanks for the update ryanpetrello! | 17:39 |
| fungi | it's unfortunately inevitable that articles are already slightly outdated by the time they go to print | 17:39 |
| fungi | so errata to bring them current at time of publication is always appreciated | 17:40 |
| fungi | that's just a point-in-time article though, so longer-term we very much appreciate whatever adjustments you can propose to the faq we're trying to maintain as a living document here: https://zuul-ci.org/docs/faq.html (granted that question isn't included in it yet) | 17:41 |
| raukadah | fungi: is it possible to get a feature wise comparison with jenkins vs zuul in faq, it is common question always got asked | 17:42 |
| *** michael-beaver has joined #zuul | 17:43 | |
| fungi | raukadah: i think we're working out who can get zuul entries added to https://en.wikipedia.org/wiki/Comparison_of_continuous_integration_software so that will probably be something useful to start from | 17:44 |
| fungi | we also want to get https://en.wikipedia.org/wiki/Gated_commit updated to mention zuul, i think | 17:44 |
| fungi | mostly came down to working out who we know that's already a wikipedia editor | 17:45 |
| clarkb | ryanpetrello: update made to the article | 17:46 |
| ryanpetrello | thx @clarkb | 17:46 |
| raukadah | fungi: thanks :-) | 17:46 |
| *** jangutter has joined #zuul | 17:49 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Fix skipping child jobs with soft dependencies https://review.opendev.org/688971 | 17:52 |
| corvus | clarkb, tobiash: ^ fixed pep8 and added a release note (due to our informal 'no releases without release notes' rule) | 17:53 |
| *** jangutter has quit IRC | 17:53 | |
| *** spsurya has quit IRC | 17:58 | |
| openstackgerrit | David Shrewsbury proposed zuul/nodepool master: Grab docker logs in k8s job https://review.opendev.org/688961 | 18:04 |
| *** mattw4 has quit IRC | 18:13 | |
| *** mattw4 has joined #zuul | 18:15 | |
| *** hashar has quit IRC | 18:18 | |
| *** hashar has joined #zuul | 18:18 | |
| *** jangutter has joined #zuul | 18:22 | |
| *** jamesmcarthur has joined #zuul | 18:23 | |
| *** mattw4 has quit IRC | 18:23 | |
| *** mattw4 has joined #zuul | 18:25 | |
| corvus | tristanC: https://review.opendev.org/688578 is passing now (yay) -- what do you think we should do about https://review.opendev.org/688743 ? it looks like crio has major.minor version packages, but minikube wants major.minor.micro if you specify the version. should we just leave it hardcoded and update occasionally? or maybe use latest k8s, do a package listing to find the latest crio, and use that? | 18:26 |
| Shrews | tobiash: i don't see swest around anymore, but i've seen a couple failures today for the test introduced in https://review.opendev.org/687271. should we revert or wait for swest to have a look? | 18:28 |
| tobiash | Shrews: if it's urgent, revert, id it can wait until tomorrow, he'd probably fix it | 18:30 |
| Shrews | tobiash: ok, let's wait then. wasn't sure of his availability | 18:31 |
| Shrews | tobiash: thx | 18:31 |
| tobiash | I'll notify him | 18:31 |
| tristanC | corvus: fedora package is also using major.minor... We could leave the hardcoded but this may break in the future. Though since we pull k8s and cri-o from different source, there isn't much we can do about it | 18:32 |
| tristanC | and perhaps the cri interface is getting more and more stable and mixing an older cri-o with a newer kubelet may also work well | 18:33 |
| tristanC | corvus: i'll add a quick sanity check (e.g. spawn a pause pod) for cri-o to make sure it's working | 18:35 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: install-kubernetes: add sanity check for crio https://review.opendev.org/689001 | 18:40 |
| openstackgerrit | Merged zuul/zuul-registry master: Implement namespacing https://review.opendev.org/687613 | 18:41 |
| openstackgerrit | Merged zuul/zuul-registry master: Run docker and podman push/pull tests https://review.opendev.org/687692 | 18:52 |
| openstackgerrit | Merged zuul/zuul master: Fix skipping child jobs with soft dependencies https://review.opendev.org/688971 | 18:53 |
| openstackgerrit | David Shrewsbury proposed zuul/nodepool master: Remove unused k8s log config https://review.opendev.org/688961 | 19:03 |
| Shrews | tristanC: corvus: oh, heh, i just realized that the minikube logs also contain the docker service logs | 19:03 |
| clarkb | with all this k8s activity it might be a good time to take a look at https://review.opendev.org/#/c/677787/ my understanding from mnaser's comments is that this is necessary if you want workload in k8s to be able to resolve dns records outside of the cluster | 19:04 |
| clarkb | tristanC: ^ you may have thoughts on that | 19:04 |
| corvus | yeah i stacked my change on that; lgtm | 19:05 |
| AJaeger | #status log openSUSE 42.3 images have been removed from Zuul | 19:11 |
| openstackstatus | AJaeger: finished logging | 19:11 |
| clarkb | AJaeger: you can #success that one too if you want :) | 19:11 |
| AJaeger | ;) | 19:13 |
| AJaeger | #success openSUSE 42.3 images have been removed from Zuul | 19:13 |
| openstackstatus | AJaeger: Added success to Success page (https://wiki.openstack.org/wiki/Successes) | 19:13 |
| Shrews | lgtm too | 19:13 |
| AJaeger | argh, should have done this in #openstack-infra... sorry for the noise here | 19:13 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: install-kubernetes: add sanity check for crio https://review.opendev.org/689001 | 19:16 |
| tristanC | Shrews: then we may need a lower log level... the current logs doesn't seem to tell why the main-0000000001 pod didn't transition from Pending to Running. beside that kubelet warning about plugin/docker network failure, which doesn't seem to correlate with docker logs | 19:26 |
| openstackgerrit | David Shrewsbury proposed zuul/nodepool master: Remove unused playbook https://review.opendev.org/689014 | 19:26 |
| Shrews | tristanC: which logs do you want to see? | 19:27 |
| Shrews | syslog? | 19:28 |
| tristanC | Shrews: i guess docker, kube-apiserver and kubelet | 19:28 |
| Shrews | tristanC: those are all there in the minikube.txt | 19:28 |
| tristanC | Shrews: yes but it's too quiet, docker should at least tell when it start or stop a container | 19:29 |
| openstackgerrit | Merged zuul/zuul-jobs master: Allow for overriding dns resolvers in install-kubernetes https://review.opendev.org/677787 | 19:30 |
| *** tosky has quit IRC | 19:30 | |
| Shrews | looks like minikube has a -v option for logging level, but i don't know if that affects docker. there is very little documentation on it | 19:32 |
| tristanC | Shrews: iiuc, pod creation goes through kube-apiserver <-> kubelet <-> docker, in the minikube.txt there is currently only one reference of the nodepool pod id in apiserver and one in kubelet. Debug log about container creation, status, etc... would help figure out why the job failed. | 19:33 |
| *** armstrongs has quit IRC | 19:34 | |
| tristanC | iirc the kube services argument should be --v=2 | 19:34 |
| Shrews | https://minikube.sigs.k8s.io/docs/tasks/debug/ | 19:34 |
| Shrews | i guess --v=7 is preferable | 19:35 |
| Shrews | still no idea if that affects docker itself | 19:35 |
| Shrews | corvus: would you want to add that option to 688578 to see what effect it has? or shall i try it seperately? | 19:36 |
| corvus | Shrews: i think 578 is ready to land as-is (further work around versions i discussed with tristanC can be a followup) so why don't you stack on top of that? | 19:39 |
| Shrews | alrighty | 19:40 |
| openstackgerrit | David Shrewsbury proposed zuul/zuul-jobs master: Increase minikube logging output to maximum https://review.opendev.org/689021 | 19:43 |
| corvus | Shrews: i'm restarting right now, so that may need a recheck ^ | 19:44 |
| Shrews | *nod* | 19:45 |
| mnaser | did we wanna look at https://review.opendev.org/#/c/688446/ and ill try to test it out locally after and figure out some basics tests ? | 19:47 |
| *** michael-beaver has quit IRC | 19:53 | |
| corvus | mnaser: lgtm -- let's put a note in #openstack-infra real quick since it's an opendev project | 19:53 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: install-kubernetes: add sanity check for crio https://review.opendev.org/689001 | 19:55 |
| mnaser | corvus: thanks for that, my next ideal thing is getting tagged releases of zuul | 19:57 |
| mnaser | inside dockerhub | 19:57 |
| corvus | mnaser: ++ i think we just need to add plain old "build an image on tag" jobs -- i don't think we should try to use the promoted images for that (it's too complicated) | 19:58 |
| mnaser | okay, ill try to get into doing that and then ask nicely to enqueue that job on a tag | 19:59 |
| fungi | mapping the merge commit from the branch back to an artifact from a change merging in the gate pipeline is presumably easier once opendev's zuul pushes merges to gerrit, right? | 20:00 |
| fungi | then we can in theory tie the git tag target commit to an artifact from when a particular change merged | 20:01 |
| corvus | fungi: yeah, that may make the problem easier to solve (it *is* tractable now, but i don't think we need to let perfect be the enemy of good here) | 20:01 |
| fungi | agreed | 20:01 |
| *** hashar has quit IRC | 20:01 | |
| fungi | more thinking it's probably not worth having a precise solution to the problem now if we know it's likely to get a good deal easier in the future | 20:02 |
| corvus | (zuul is back up) | 20:02 |
| fungi | so post pipeline build as an interim stop-gap seems fine | 20:02 |
| mnaser | ooo | 20:07 |
| mnaser | i might be able to get away with some neat workaround | 20:07 |
| * mnaser hack | 20:07 | |
| openstackgerrit | Mohammed Naser proposed zuul/zuul master: DNM: Tag both 'latest' and version https://review.opendev.org/689033 | 20:14 |
| *** jamesmcarthur has quit IRC | 20:14 | |
| mnaser | ^ here is an attempt that might just crash and burn terribly | 20:14 |
| *** igordc has joined #zuul | 20:15 | |
| *** jamesmcarthur has joined #zuul | 20:16 | |
| *** jamesmcarthur has quit IRC | 20:18 | |
| *** mattw4 has quit IRC | 20:19 | |
| mnaser | hmm | 20:21 |
| mnaser | with python we kinda rely on pbr to help us do the versioning there | 20:21 |
| *** mattw4 has joined #zuul | 20:22 | |
| tristanC | corvus: seems like cri-o is working as expected, well done! ( https://zuul.opendev.org/t/zuul/build/bcbf04342e9e4a70a0267f15aef584d9/console#1/0/48/ubuntu-bionic ) | 20:22 |
| corvus | tristanC: \o/ next i'll add a job to zuul-registry to use that, then openshift after that | 20:23 |
| mnaser | wild idea: how do we feel about adding a `zuul_return` with the version from the build-python job | 20:25 |
| corvus | mnaser: that sounds clever | 20:26 |
| clarkb | another option is to ask pbr what the version is | 20:26 |
| clarkb | (download the image then pbr freeze in it) | 20:26 |
| mnaser | right but that means i would have to make a custom "build-python-docker-job" | 20:27 |
| mnaser | and i guess the upload jobs are trusted so i wouldn't be able to modify those in repo | 20:27 |
| clarkb | either way tagged versions would have to be handled separately I think (but the tag itslef is that data so that is easy) | 20:27 |
| mnaser | i have a bit of a very "oblivious" attempt here https://review.opendev.org/#/c/689033/ | 20:28 |
| *** pcaruana has quit IRC | 20:28 | |
| corvus | mnaser: i'm not sure what your idea is -- but originally i was just thinking that we stick build-docker-image in to the release pipeline and the tag is {{zuul.tag}} (or whatever it's called) | 20:28 |
| mnaser | but its naive, with the other one, we'd actually get an image for every single version (say like 3.11.0dev10 or what not) | 20:28 |
| clarkb | corvus: ya that is sort of what I figured too | 20:29 |
| clarkb | (thats the special tag case I mention) | 20:29 |
| mnaser | yep thats pretty much exactly what i did in there, but i was hoping `zuul.project.checkout` might get me a branch *or* tag | 20:29 |
| mnaser | i mean we've always historically had master only but, yeah | 20:29 |
| mnaser | mainly to avoid the copy pasta of all of that docker_images section | 20:29 |
| clarkb | mnaser: I think the problem with it is even the jobs that run against master will now be tagged "master" | 20:30 |
| corvus | what about something like {{ defined(zuul.tag) | ternary(zuul.tag, 'latest') }} ? | 20:30 |
| clarkb | (I think you may be right that the tag is the checkout on release jobs) | 20:30 |
| mnaser | yeah from an openstack world i was like "we get tip of branches for free!" | 20:31 |
| mnaser | and yeah that's what i was thinking, but i was wondering "{{ zuul.tag | default('latest') }}" works | 20:31 |
| corvus | yeah that's shorter :) | 20:31 |
| corvus | we could even make that the default in the base docker jobs i think | 20:31 |
| mnaser | but that would imply zuul always has tag | 20:31 |
| * mnaser checks | 20:32 | |
| corvus | zuul.tag should only exist for tag items: https://zuul-ci.org/docs/zuul/user/jobs.html#var-zuul.tag | 20:32 |
| mnaser | yeah zuul.tag isn't defined at all so likely that would fail for "missing in dict" -- i think | 20:32 |
| mnaser | so my fanciness might not work | 20:32 |
| corvus | well, there's definitely a way to do that :) | 20:33 |
| openstackgerrit | Mohammed Naser proposed zuul/zuul master: Tag both 'latest' and version https://review.opendev.org/689033 | 20:35 |
| mnaser | based on my small testing this might work | 20:35 |
| corvus | hrm, i think we're still missing something, since the upload job is meant to work with promote -- it's going to try to put a change number in the tag and fail... | 20:38 |
| corvus | i'm not sure we have a plain docker-push role... | 20:39 |
| mnaser | corvus: im wondering if we can just push :latest + :<short-commit-id> in gate and and reuse the promote image bits to just retag :<short-commit-id> to :<version-id> | 20:44 |
| corvus | mnaser: that would be great, but that's the hard bit -- mapping tag to short commit | 20:50 |
| mnaser | {{ zuul.tag }} + {{ zuul.newrev }} + some introspection ? | 20:50 |
| mnaser | "If the item was enqueued as the result of a tag being created, the new git sha of the tag will be included here." unless i'm missing something | 20:51 |
| clarkb | if you want the version to show up correctly in the dashboard you need to rebuild the image on tag rather than promoting | 20:51 |
| corvus | mnaser: we know the sha of the tag -- that may be a commit that zuul has never seen (this is why fungi was saying that zuul push would help with this) | 20:51 |
| corvus | mnaser: because merge commits are generated by gerrit | 20:52 |
| clarkb | The simplest thing is likely to do a build and publish on tag | 20:52 |
| *** jamesmcarthur has joined #zuul | 20:53 | |
| *** brendangalloway has quit IRC | 20:54 | |
| corvus | Shrews, ianw: zuul 3.11.1 is out -- we can have the nodepool release note point to that and then cut a nodepool release | 21:03 |
| mnaser | corvus: https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/build-docker-image/tasks/main.yaml#L16-L19 i think we might be ok regarding the issue you mentioned | 21:05 |
| mnaser | ah wait, i see what you mean | 21:06 |
| * mnaser goes to find the jobs that consume the images | 21:07 | |
| mnaser | well, when we run this in check/gate -- it tries to pull 'latest' -- which is still the right thing, and in 'release' when it will run with the tag, well, we dont have any jobs that consume that.. so it should be ok(tm) | 21:08 |
| mnaser | so my non-gating brain says: SUCCESS | 21:08 |
| *** igordc has quit IRC | 21:19 | |
| *** igordc has joined #zuul | 21:29 | |
| *** jangutter has quit IRC | 21:40 | |
| Shrews | corvus: so i guess we just need https://review.opendev.org/682797 updated with that release note modification, then we can merge it and release? | 21:47 |
| Shrews | I expect we might want to run that in production for a day or so first. | 21:48 |
| corvus | Shrews: that's my understanding -- and yeah, maybe we want to exercise that in opendev? | 21:49 |
| Shrews | I think that would be best | 21:50 |
| *** mattw4 has quit IRC | 21:55 | |
| *** mattw4 has joined #zuul | 21:56 | |
| mordred | fbo: since the pagure driver has landed - I wonder if it would make sense to add a pagure connection to opendev zuul so that we could add a functional test between zuul and pagure - and so that you could use depends-on footers on patches that are still waiting on upstream patches to land? | 22:01 |
| mordred | fbo: was looking at https://review.opendev.org/#/c/685116 - and there was a moment where it was marked WIP waiting on the upstream patches to land, which is what made me think of it | 22:02 |
| mordred | although at this point the list of outstanding changes needed in pagure seems to be smaller - so maybe it's not worth it anymore | 22:03 |
| corvus | would still be cool :) | 22:03 |
| mordred | yah | 22:05 |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: Set default python-path to "auto" https://review.opendev.org/682797 | 22:18 |
| ianw | corvus: ^ thanks ... updated releasenotes -- i think the reflects the reality now | 22:18 |
| corvus | ianw: lgtm -- tobiash ^ | 22:23 |
| mordred | corvus: that lgtm- do you want to wait for an ack from tobiash ? | 22:28 |
| corvus | mordred: yeah let's | 22:32 |
| mordred | kk | 22:33 |
| *** jamesmcarthur has quit IRC | 22:34 | |
| openstackgerrit | Merged zuul/zuul master: Build GitHub URLs for tags correctly https://review.opendev.org/688882 | 22:34 |
| mordred | corvus: fwiw - I think the gerrit job is starting to make me grok what zuul might want to do with submodules - assuming there is a good way to get the submodule info from python in the mergers | 22:34 |
| mordred | it seems that if a repo has a submodule that's also on a connection zuul knows about - it should imply a required-projects entry - and if there is a branch listed, it should imply an override-checkout entry on the required-projects entry - and then finally when repos are being prepared on disk, putting the zuul prepared repo into the submodule target location | 22:37 |
| mordred | then - doing the equiv of git submodule update --init for any submodules zuul *doesn't* know about the source repos of when preparing the repo | 22:38 |
| openstackgerrit | Luigi Toscano proposed zuul/zuul-jobs master: fetch-subunit-output: collect additional subunits (2nd try) https://review.opendev.org/674334 | 22:42 |
| *** sgw has quit IRC | 22:51 | |
| openstackgerrit | Merged zuul/zuul master: Add the process environment to zuul.conf parser https://review.opendev.org/612824 | 22:51 |
| SpamapS | mordred: +1 for how submodules would play nice with zuul | 22:56 |
| corvus | mordred: the zuul-internal part of that sounds good -- i think i need to think a bit more about the submodule --init part (and look into what existing submodule roles/jobs do) | 23:17 |
| *** sgw has joined #zuul | 23:18 | |
| mordred | corvus: ++ | 23:19 |
| mordred | corvus: doing submodule --init on unknown submodules could also be skipped - and maybe we say if you want to do zuul + submodules you need all the submodules to be in zuul's config (the main thing is that the repo doesn't have a valid remote, so you can't do submodule init in job itself if the submodule has a relative path) | 23:20 |
| mordred | but yeah - bears more thinking on the specifics | 23:20 |
| *** rfolco has joined #zuul | 23:25 | |
| *** rfolco|ruck has quit IRC | 23:25 | |
| *** mattw4 has quit IRC | 23:26 | |
| *** rfolco has quit IRC | 23:34 | |
| *** rfolco has joined #zuul | 23:34 | |
| corvus | yeah, it's those edge cases i'm not trusting my brain to come up with right now. :) | 23:40 |
| mordred | corvus: ++ | 23:44 |
| *** rlandy has quit IRC | 23:48 | |
| *** rfolco has quit IRC | 23:48 | |
| SpamapS | Lots of little details to work out, but I think the mode of comparing the submodule remotes to the remotes in connection configs, and using those as required-projects, is a good start. | 23:49 |
| SpamapS | Perhaps with an additional step of having zuul's git prep code take care of putting those required projects dirs in the right places. | 23:49 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!