| pabelanger | tristanC: dmsimard: our tenant in sf.io is broken? | 01:47 |
|---|---|---|
| pabelanger | has anything change on zuul side? | 01:47 |
| pabelanger | https://ansible-network.softwarefactory-project.io/zuul/config-errors | 01:48 |
| pabelanger | no pipelines are loaded either | 01:48 |
| pabelanger | base-minimal is in ansible-network/zuul-config | 01:49 |
| pabelanger | it's almost like ansible-network/zuul-config isn't loaded | 01:50 |
| pabelanger | but nothing has changed on our side | 01:50 |
| pabelanger | Last reconfigured: Fri Jan 25 2019 18:12:37 GMT-0500 (Eastern Standard Time) | 01:52 |
| pabelanger | I suspect there is something around that timeframe to why ansible-network/zuul-config didn't load | 01:52 |
| tristanC | pabelanger: looking... | 01:56 |
| pabelanger | hopefully not 3.5 related :) | 01:57 |
| tristanC | that's odd, config looks good, nothing in logs. it seems like zuul-config project isn't getting loaded properly | 02:02 |
| pabelanger | I know there was some updates to github driver around reloads, let me find commit | 02:02 |
| pabelanger | tristanC: http://git.zuul-ci.org/cgit/zuul/commit/?id=aad0e17ddd64851ca4c9e77185c74bebcc049fba | 02:05 |
| tristanC | ansible is also affected, but no config-errors are logged: https://ansible.softwarefactory-project.io/zuul/status | 02:05 |
| tristanC | i'll try a service reload first | 02:07 |
| pabelanger | odd | 02:07 |
| pabelanger | I think we have base / base-minimal there too | 02:07 |
| tristanC | pabelanger: in ansible, both base and base-minimal are in the zuul-config repo | 02:08 |
| pabelanger | k | 02:08 |
| tristanC | reconfiguration is in progress, let's see if zuul-config gets loaded | 02:09 |
| pabelanger | tyty | 02:09 |
| tristanC | hum, so ansible-networ/zuul-config doesn't appeared in logs after reconfiguration | 02:18 |
| tristanC | i mean, during the reconfiguration event | 02:18 |
| tristanC | also, https://softwarefactory-project.io/zuul/t/ansible-test/status does work, im' looking for difference between the two tenants | 02:19 |
| pabelanger | tristanC: we do use branch protection | 02:19 |
| tristanC | pabelanger: but not for zuul-config right? | 02:19 |
| pabelanger | no, we have it enabled there too | 02:20 |
| pabelanger | tristanC: I am not sure on ansible/zuul-config, not admin. If I was to guess, i don't think they have it setup | 02:21 |
| tristanC | for ansible/zuul-config, all branches except the master gets loaded during reconfiguration... | 02:23 |
| pabelanger | wow, they have branches there | 02:23 |
| pabelanger | that isn't good | 02:23 |
| pabelanger | will have to delete them next week | 02:24 |
| tristanC | iirc, doesn't zuul just ignore later down the road non master branch for config projects? | 02:24 |
| pabelanger | not sure TBH, never tried branching config-project | 02:24 |
| tristanC | pabelanger: back to aad0e17, iiuc, this may be causing zuul to cache an empty project-config for the protected branch of zuul-configs, and then the content gets ignored by reconfigure? | 02:27 |
| tristanC | in logs, i don't see any "Submitting cat job" for ansible-network/zuul-config | 02:28 |
| pabelanger | tristanC: want me to disable branch protection and test? | 02:29 |
| tristanC | perhaps i could add a debug log in the "if not tpc.load_classes" and restat the scheduler | 02:29 |
| pabelanger | sure | 02:29 |
| tristanC | though that's odd, because zuul-config should have load_classes | 02:30 |
| tristanC | ok, restarting scheduler now with the extra debug log | 02:33 |
| tristanC | arg, no luck, ansible-network/zuul-config doesn't seems to be assigned that cacheUnparsedConfig | 02:38 |
| pabelanger | tristanC: lets test with branch protection removed | 02:39 |
| tristanC | fwiw, here are grep ansible-network output after the restart: https://softwarefactory-project.io/paste/show/1399/ | 02:40 |
| pabelanger | tristanC: and ansible-network/zuul-config is listed in tenant_config? | 02:41 |
| tristanC | yes | 02:41 |
| pabelanger | odd | 02:41 |
| pabelanger | I'm going to disable branch protection now, then we can reconfigure and see what happens | 02:42 |
| pabelanger | I don't think it will fix anything, but a data point | 02:42 |
| tristanC | here is the 3 tenants configuration with config-project on github: https://softwarefactory-project.io/paste/show/1400/ | 02:42 |
| pabelanger | okay, branch protection removed on ansible-network/zuul-config | 02:42 |
| tristanC | alright, testing reload first | 02:43 |
| tristanC | this time it did 02:43:34,347 INFO zuul.TenantParser: Loading configuration from ansible-network/zuul-config/zuul.d/pipelines.yaml@master | 02:44 |
| pabelanger | wow | 02:44 |
| pabelanger | once zuul is finished, I'll add it back | 02:45 |
| tristanC | pabelanger: i don't think this behavior was introduced by zuul-3.5.0, pretty sure ansible tenant was working after the update | 02:46 |
| pabelanger | maybe 3.4.0 then? | 02:47 |
| pabelanger | tristanC: we've had branch protection for a while on the repo | 02:47 |
| tristanC | pabelanger: could it be that github api recently changed and branch protection affects how the apps can list project? | 02:47 |
| pabelanger | could be | 02:47 |
| pabelanger | what version of github3 do we have? | 02:47 |
| pabelanger | 1.0.2 we released recently | 02:48 |
| pabelanger | tristanC: I guess we have branch protection on ansible/zuul-config, can confirm monday | 02:49 |
| tristanC | 1.2.0 | 02:49 |
| pabelanger | k, we'll have to confirm with tobiash next week, for now I can leave branch protection off | 02:51 |
| pabelanger | tristanC: have to run, will check backlog in a few hours. Thanks again for helping on your saturday | 02:52 |
| tristanC | pabelanger: alright, thanks for the ping too, this went unnoticed :) | 02:52 |
| tristanC | i'll try: https://softwarefactory-project.io/r/14881 Bump to github3 to 1.3.0 | 02:56 |
| tristanC | pabelanger: ansible/zuul-config is protected according to https://api.github.com/repos/ansible/zuul-config/branches | 03:17 |
| tristanC | ok i think i understand what's going on. when 'exclude-protected-branch' isn't set, zuul query github branches with protected=0, e.g.: https://api.github.com/repos/ansible/zuul-config/branches?protected=0 | 03:40 |
| tristanC | and i guess that github is now only returning unprotected branch now, resulting in zuul skipping protected branch | 03:41 |
| tristanC | pabelanger: so i think https://review.openstack.org/633314 should fix the issue, alternatively setting exclude-unprotected-branches on ansibles' zuul-config would do the trick too | 03:51 |
| tristanC | so, i fixed ansible tenant by applying: https://github.com/ansible/zuul-config/pull/47 | 04:07 |
| *** bhavikdbavishi has joined #softwarefactory | 04:09 | |
| tristanC | pabelanger: and ansible-network should work if we apply https://softwarefactory-project.io/r/14882 after re-enabling branch protection on ansible-network/zuul-config | 04:10 |
| tristanC | otherwise, I think we need https://review.openstack.org/633314 to restore the previous behavior (e.g. zuul load protected branch when not using the exclude-unprotected-branches option) | 04:15 |
| *** bhavikdbavishi has quit IRC | 04:38 | |
| *** bhavikdbavishi has joined #softwarefactory | 06:52 | |
| *** bhavikdbavishi has quit IRC | 07:11 | |
| *** bhavikdbavishi has joined #softwarefactory | 12:47 | |
| *** bhavikdbavishi has quit IRC | 13:36 | |
| pabelanger | tristanC: wow, so github changes API it seems | 16:32 |
| pabelanger | tristanC: yah, I think we want to land your patch, or discuss it upstream, but also agree exclude-protected-branch should also be set | 16:32 |
| pabelanger | thanks for digging into this | 16:32 |
| *** sfbender has quit IRC | 22:05 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!