| *** kumarmn has joined #openstack-release | 00:04 | |
| *** hongbin has quit IRC | 00:12 | |
| *** kumarmn has quit IRC | 00:14 | |
| *** kumarmn has joined #openstack-release | 00:14 | |
| *** sree has joined #openstack-release | 00:15 | |
| *** kumarmn has quit IRC | 00:18 | |
| *** kumarmn has joined #openstack-release | 00:18 | |
| openstackgerrit | Terry Wilson proposed openstack/releases master: Release ovsdbapp 0.4.2 https://review.openstack.org/539587 | 00:19 |
|---|---|---|
| *** sree has quit IRC | 00:20 | |
| *** kumarmn has quit IRC | 00:21 | |
| *** kumarmn has joined #openstack-release | 00:21 | |
| *** zhurong has joined #openstack-release | 00:38 | |
| *** kumarmn has quit IRC | 00:40 | |
| *** kumarmn has joined #openstack-release | 00:41 | |
| *** kumarmn has quit IRC | 00:45 | |
| *** sree has joined #openstack-release | 00:53 | |
| *** sree has quit IRC | 01:01 | |
| *** armax has joined #openstack-release | 01:07 | |
| *** lbragstad has quit IRC | 01:10 | |
| *** kumarmn has joined #openstack-release | 01:12 | |
| *** sree has joined #openstack-release | 01:13 | |
| *** armax has quit IRC | 01:13 | |
| *** kumarmn has quit IRC | 01:17 | |
| *** kumarmn has joined #openstack-release | 01:18 | |
| *** sree has quit IRC | 01:18 | |
| *** armax has joined #openstack-release | 01:20 | |
| *** kumarmn has quit IRC | 01:23 | |
| *** sree has joined #openstack-release | 01:24 | |
| *** sree has quit IRC | 01:29 | |
| *** lbragstad has joined #openstack-release | 01:31 | |
| *** kumarmn has joined #openstack-release | 01:39 | |
| *** ricolin has joined #openstack-release | 01:39 | |
| *** sree has joined #openstack-release | 01:49 | |
| *** sree has quit IRC | 01:53 | |
| *** hongbin has joined #openstack-release | 01:59 | |
| *** gcb has joined #openstack-release | 02:16 | |
| *** yamamoto has joined #openstack-release | 02:37 | |
| *** kumarmn has quit IRC | 02:48 | |
| *** kumarmn has joined #openstack-release | 02:48 | |
| *** kumarmn has quit IRC | 02:51 | |
| *** kumarmn has joined #openstack-release | 02:51 | |
| *** kumarmn has quit IRC | 03:02 | |
| *** kumarmn has joined #openstack-release | 03:03 | |
| *** kumarmn has quit IRC | 03:08 | |
| *** rosmaita has quit IRC | 03:17 | |
| *** sree has joined #openstack-release | 03:19 | |
| *** sree has quit IRC | 03:20 | |
| *** sree has joined #openstack-release | 03:20 | |
| *** iyamahat has quit IRC | 03:37 | |
| *** dave-mccowan has quit IRC | 03:48 | |
| *** ekcs has quit IRC | 04:01 | |
| *** coolsvap has joined #openstack-release | 04:07 | |
| *** harlowja has quit IRC | 04:12 | |
| *** ykarel|away has joined #openstack-release | 04:27 | |
| *** harlowja has joined #openstack-release | 04:41 | |
| *** hongbin has quit IRC | 04:47 | |
| *** zhurong has quit IRC | 04:55 | |
| *** sree has quit IRC | 05:31 | |
| *** sree has joined #openstack-release | 05:40 | |
| *** zhurong has joined #openstack-release | 06:02 | |
| *** harlowja has quit IRC | 06:12 | |
| *** sree has quit IRC | 06:24 | |
| *** iyamahat has joined #openstack-release | 06:46 | |
| openstackgerrit | Chandan Kumar proposed openstack/releases master: Release python-tempestconf-1.1.4 https://review.openstack.org/538840 | 07:00 |
| *** zhurong has quit IRC | 07:14 | |
| *** ykarel|away is now known as ykarel | 07:18 | |
| *** iyamahat has quit IRC | 07:22 | |
| *** zhurong has joined #openstack-release | 07:23 | |
| *** iyamahat has joined #openstack-release | 07:37 | |
| *** claudiub has joined #openstack-release | 07:44 | |
| *** claudiub|3 has quit IRC | 07:47 | |
| *** sree has joined #openstack-release | 07:47 | |
| *** pcaruana has joined #openstack-release | 07:47 | |
| *** sree has quit IRC | 07:51 | |
| *** kumarmn has joined #openstack-release | 08:04 | |
| *** kumarmn has quit IRC | 08:08 | |
| *** amoralej|off is now known as amoralej | 08:09 | |
| ttx | ack on tagging needing to happen on a point in master that includes all the patches that landed in stable/pike i(f any) | 08:13 |
| *** d0ugal has joined #openstack-release | 08:28 | |
| *** jpich has joined #openstack-release | 08:39 | |
| openstackgerrit | Witold Bedyk proposed openstack/releases master: Release monasca-agent 2.2.2 https://review.openstack.org/539862 | 08:45 |
| *** shardy has joined #openstack-release | 08:54 | |
| *** pbandark has joined #openstack-release | 09:05 | |
| *** pcaruana has quit IRC | 09:19 | |
| *** pcaruana has joined #openstack-release | 09:19 | |
| *** zhurong has quit IRC | 09:25 | |
| *** zhurong has joined #openstack-release | 09:33 | |
| *** yamahata has quit IRC | 09:36 | |
| *** iyamahat has quit IRC | 09:36 | |
| *** lvdombrkr has joined #openstack-release | 10:10 | |
| *** alexchadin has joined #openstack-release | 10:14 | |
| *** lvdombrkr89 has joined #openstack-release | 10:19 | |
| *** lvdombrkr has quit IRC | 10:22 | |
| *** zhurong has quit IRC | 10:34 | |
| *** wolverineav has joined #openstack-release | 10:36 | |
| *** samP has quit IRC | 10:45 | |
| *** samP has joined #openstack-release | 10:50 | |
| *** lucas-afk is now known as lucasagomes | 10:56 | |
| *** mtreinish_ has joined #openstack-release | 11:08 | |
| *** mtreinish has quit IRC | 11:11 | |
| *** sdake has quit IRC | 11:11 | |
| *** ianw has quit IRC | 11:11 | |
| *** TheJulia has quit IRC | 11:11 | |
| *** mtreinish_ is now known as mtreinish | 11:11 | |
| *** sdake has joined #openstack-release | 11:11 | |
| *** sdake has quit IRC | 11:11 | |
| *** sdake has joined #openstack-release | 11:11 | |
| *** ianw has joined #openstack-release | 11:12 | |
| *** TheJulia has joined #openstack-release | 11:12 | |
| *** pbandark has quit IRC | 11:15 | |
| *** pbandark has joined #openstack-release | 11:18 | |
| *** gcb has quit IRC | 11:25 | |
| *** gcb has joined #openstack-release | 11:25 | |
| *** samP has quit IRC | 11:30 | |
| *** samP has joined #openstack-release | 11:35 | |
| *** pbandark has quit IRC | 11:41 | |
| *** pbandark has joined #openstack-release | 11:45 | |
| *** sree has joined #openstack-release | 11:45 | |
| *** alexchadin has quit IRC | 11:50 | |
| *** alexchadin has joined #openstack-release | 11:51 | |
| *** amoralej is now known as amoralej|off | 11:52 | |
| *** tellesnobrega has quit IRC | 11:53 | |
| *** tellesnobrega has joined #openstack-release | 11:54 | |
| *** tenobreg has joined #openstack-release | 11:58 | |
| *** tellesnobrega has quit IRC | 12:02 | |
| *** tenobreg has quit IRC | 12:03 | |
| *** kumarmn has joined #openstack-release | 12:04 | |
| *** kumarmn has quit IRC | 12:09 | |
| *** ykarel_ has joined #openstack-release | 12:09 | |
| *** pkovar has joined #openstack-release | 12:10 | |
| *** ykarel has quit IRC | 12:12 | |
| *** samP has quit IRC | 12:13 | |
| *** dave-mccowan has joined #openstack-release | 12:14 | |
| *** dtantsur|afk is now known as dtantsur | 12:18 | |
| *** samP has joined #openstack-release | 12:18 | |
| *** ykarel__ has joined #openstack-release | 12:30 | |
| *** root__ has joined #openstack-release | 12:31 | |
| *** root__ has quit IRC | 12:31 | |
| *** root__ has joined #openstack-release | 12:31 | |
| *** pkovar has quit IRC | 12:31 | |
| *** ykarel_ has quit IRC | 12:33 | |
| *** lvdombrkr89 has quit IRC | 12:33 | |
| *** samP has quit IRC | 12:43 | |
| *** samP has joined #openstack-release | 12:46 | |
| *** samP has quit IRC | 12:48 | |
| *** samP has joined #openstack-release | 12:53 | |
| *** samP has quit IRC | 12:54 | |
| *** samP has joined #openstack-release | 13:00 | |
| *** ykarel__ is now known as ykarel | 13:00 | |
| *** rosmaita has joined #openstack-release | 13:05 | |
| *** sree has quit IRC | 13:08 | |
| *** sree has joined #openstack-release | 13:09 | |
| *** shardy is now known as shardy_mtg | 13:12 | |
| *** alexchadin has quit IRC | 13:12 | |
| *** pkovar has joined #openstack-release | 13:12 | |
| *** samP has quit IRC | 13:13 | |
| smcginnis | ttx: That was a sad realization. | 13:13 |
| smcginnis | ttx: Did you see the ML post and patch for the current plan? | 13:13 |
| smcginnis | Just waking up so I haven't checked if there were any comments on either. | 13:13 |
| *** sree has quit IRC | 13:14 | |
| *** edmondsw has joined #openstack-release | 13:15 | |
| openstackgerrit | Monty Taylor proposed openstack/releases master: Release 0.11.2 of openstacksdk for queens https://review.openstack.org/539943 | 13:15 |
| openstackgerrit | Monty Taylor proposed openstack/releases master: Release 1.29.0 of os-client-config for queens https://review.openstack.org/539595 | 13:17 |
| *** samP has joined #openstack-release | 13:17 | |
| *** samP has quit IRC | 13:20 | |
| *** alexchadin has joined #openstack-release | 13:21 | |
| *** coolsvap has quit IRC | 13:22 | |
| *** samP has joined #openstack-release | 13:24 | |
| *** shardy_mtg is now known as shardy | 13:31 | |
| openstackgerrit | Witold Bedyk proposed openstack/releases master: Release monasca-ui 1.5.2 for stable/ocata https://review.openstack.org/539952 | 13:43 |
| *** tenobreg has joined #openstack-release | 13:51 | |
| *** yamamoto has quit IRC | 13:55 | |
| mordred | smcginnis: 539595 is ready to go (sorry, forgot to update it when the patch landed yesterday) | 13:57 |
| ttx | smcginnis: I missed the therad | 13:57 |
| smcginnis | mordred: ack | 13:57 |
| mordred | smcginnis: and 539943 fixes a fun bug that the sahara team ran across when a tempest test of theirs pulled in an unconstrained version of sdk | 13:57 |
| smcginnis | ttx: Let me find that. | 13:58 |
| ttx | oh I see it now | 13:58 |
| ttx | long list of prefixes | 13:58 |
| *** ykarel is now known as ykarel|afk | 13:58 | |
| smcginnis | yeah... | 13:58 |
| *** tenobreg is now known as tellesnobrega | 13:59 | |
| ttx | I saw the patch though | 13:59 |
| *** alexchadin has quit IRC | 14:02 | |
| *** bobh has joined #openstack-release | 14:02 | |
| *** kumarmn has joined #openstack-release | 14:03 | |
| *** samP has quit IRC | 14:06 | |
| *** ykarel|afk has quit IRC | 14:07 | |
| openstackgerrit | Merged openstack/releases master: Release 1.29.0 of os-client-config for queens https://review.openstack.org/539595 | 14:07 |
| dhellmann | ttx, smcginnis : https://review.openstack.org/539692 will fix the rsync errors in our static-publish jobs | 14:09 |
| smcginnis | Oh, great! | 14:09 |
| smcginnis | Nice to see doing something like that is so straight forward. | 14:10 |
| *** root__ is now known as lvdombrkr | 14:10 | |
| ttx | oh, nice | 14:10 |
| *** samP has joined #openstack-release | 14:11 | |
| *** jhesketh has quit IRC | 14:12 | |
| *** jhesketh has joined #openstack-release | 14:13 | |
| dhellmann | it doesn't guarantee order, though, so we still need to be careful when we approve a bunch of things all at once | 14:14 |
| *** iyamahat has joined #openstack-release | 14:14 | |
| dhellmann | corvus has some plans to fix that in the not too distant future | 14:14 |
| *** yamahata has joined #openstack-release | 14:16 | |
| *** yamamoto has joined #openstack-release | 14:17 | |
| smcginnis | We never had this issue in the past from what I remember. I wonder why it's different now. | 14:19 |
| dhellmann | no idea | 14:21 |
| *** pkovar1 has joined #openstack-release | 14:25 | |
| *** pkovar has quit IRC | 14:25 | |
| *** samP has quit IRC | 14:30 | |
| *** lucasagomes is now known as lucas-hungry | 14:32 | |
| *** samP has joined #openstack-release | 14:34 | |
| openstackgerrit | Doug Hellmann proposed openstack/releases master: final queens library releases https://review.openstack.org/539657 | 14:34 |
| openstackgerrit | Doug Hellmann proposed openstack/releases master: final queens branches for libraries https://review.openstack.org/539681 | 14:34 |
| dhellmann | rebased | 14:35 |
| *** mriedem is now known as mriedem_afk | 14:35 | |
| smcginnis | I did not see any objects or concerns raised. Shall we move that along? | 14:37 |
| *** sree has joined #openstack-release | 14:44 | |
| dhellmann | smcginnis : sure | 14:44 |
| smcginnis | I'll take one last look through once zuul +1s, then hit go. | 14:45 |
| smcginnis | Then once that is done, I will manually squash all of the u-c patches for the requirements updates. | 14:46 |
| *** hongbin has joined #openstack-release | 14:46 | |
| openstackgerrit | Merged openstack/releases master: Release 0.11.2 of openstacksdk for queens https://review.openstack.org/539943 | 14:46 |
| *** nikhil has joined #openstack-release | 14:47 | |
| smcginnis | OK, bombs away. | 14:51 |
| * dhellmann puts on his doughboy | 14:54 | |
| openstackgerrit | Merged openstack/releases master: add semaphore for publishing to the releases website https://review.openstack.org/539692 | 14:56 |
| *** mriedem_afk is now known as mriedem | 14:58 | |
| smcginnis | dhellmann: Problem with the semaphore? http://lists.openstack.org/pipermail/release-job-failures/2018-February/000751.html | 15:05 |
| dhellmann | smcginnis : ugh, I guess so | 15:07 |
| *** samP has quit IRC | 15:07 | |
| dhellmann | pabelanger , corvus : ^^ | 15:07 |
| dhellmann | I'll revert that | 15:08 |
| smcginnis | Guess I should have done the more important patch first. | 15:08 |
| openstackgerrit | Doug Hellmann proposed openstack/releases master: Revert "add semaphore for publishing to the releases website" https://review.openstack.org/539980 | 15:10 |
| pabelanger | wonder if we should consider having zuul -1 a patch that does try to modify final jobs before merging. I am not sure if that is possible to check however | 15:10 |
| dhellmann | well, that one was supposed to protect us from all the post updates conflicting | 15:10 |
| pabelanger | not the first time it has happened | 15:10 |
| pabelanger | dhellmann: yah, we'll need to add that into project-config some how | 15:10 |
| dhellmann | pabelanger : yeah, it would be nice if there was a way to check those things before merging | 15:10 |
| dhellmann | it looks like we need a custom version of that job, which I was hoping to avoid because it seems like that stuff changes frequently | 15:11 |
| dhellmann | maybe we can split the job in project-config into one that has all the details and another that is "final" | 15:11 |
| dhellmann | but I'm not sure about the point of that, so maybe just remove the "final" flag | 15:12 |
| dhellmann | I don't know when it's appropriate to use that or not | 15:12 |
| *** sree has quit IRC | 15:13 | |
| openstackgerrit | Merged openstack/releases master: final queens library releases https://review.openstack.org/539657 | 15:13 |
| *** samP has joined #openstack-release | 15:14 | |
| dhellmann | smcginnis : so we should be able to revert the library releases and then rerun it, I think? | 15:14 |
| dhellmann | assuming it fails with the same freeze error | 15:14 |
| smcginnis | dhellmann: Yes, I think that should be safe. | 15:14 |
| dhellmann | ok, let me prepare that | 15:14 |
| *** pkovar1 has quit IRC | 15:14 | |
| smcginnis | Unless there's a way to just re-enqueue that after the revert, since it didn't actually get anywhere. | 15:14 |
| fungi | the revert will trigger the same jobs, right? shouldn't have to reenqueue anything unless they're not idempotent? | 15:15 |
| dhellmann | fungi : right | 15:16 |
| openstackgerrit | Doug Hellmann proposed openstack/releases master: Revert "final queens library releases" https://review.openstack.org/539982 | 15:17 |
| openstackgerrit | Doug Hellmann proposed openstack/releases master: final queens library releases, attempt 2 https://review.openstack.org/539983 | 15:17 |
| fungi | and yeah, since the rsync contention/race isn't going to be trivially solvable with a semaphore (unless i'm missing a simpler option there besides the ones you mentioned) it may be easier to just live with hitting it occasionally in large batches of release approvals until we have the new pipeline manager for idempotent publishers | 15:18 |
| smcginnis | fungi: Not sure what you mean by "the revert will trigger the same jobs" | 15:19 |
| dhellmann | yeah, we always had the ordering issue so as long as an actual failure of that job doesn't prevent something else from working right we can just do what we've done before and save one patch to go through on its own to ensure the published data is right | 15:19 |
| fungi | worth noting, we _did_ have this race pretty much constantly in zuul v2 publication jobs as well, the difference was it tended to happen silently so you only noticed when the site looked out of date by a commit (or several) | 15:20 |
| dhellmann | smcginnis : the original patch landed but the jobs never ran because of the misconfiguration, so reverting and then reverting again will trigger the jobs easily without any manual intervention | 15:20 |
| dhellmann | right, the race was always there. the rsync conflict feels new | 15:20 |
| dhellmann | though the conflict is caused by the race we never saw errors before | 15:20 |
| smcginnis | OK, the reverting then re-reverting is the key part. It sounded like just reverting the job would do something, so I was confused by that. | 15:21 |
| *** AJaeger has joined #openstack-release | 15:21 | |
| dhellmann | ah, yeah, we need the double revert | 15:21 |
| fungi | oh, i was simply surmising that the 539980 revert would cause your post jobs to be rerun | 15:21 |
| dhellmann | I filed those in a separate series from the one to revert the semaphore | 15:21 |
| smcginnis | I think we can live with the failures for now. | 15:21 |
| fungi | without needing to revert and then rerevert the releases themselves | 15:21 |
| dhellmann | I've already approved the semaphore revert | 15:22 |
| dhellmann | fungi : no, the job looks at deliverable file changes to know what to do and there were none in the patch that added the semaphore | 15:22 |
| dhellmann | so we have to redo the original patch with the real data in it | 15:22 |
| fungi | ahh, i thought it looked at the repo state rather than the diff | 15:23 |
| fungi | okay, so yes you're right you would need to revert and rerevert the release requests | 15:23 |
| *** sree has joined #openstack-release | 15:24 | |
| dhellmann | no, it looks at the diff | 15:24 |
| dhellmann | otherwise it would take hours to run the job | 15:24 |
| fungi | great point | 15:24 |
| *** pkovar has joined #openstack-release | 15:24 | |
| fungi | also, i'm not quite sure what would happen if i tried to reenqueue the release request change(s) since they're from a point in the branch which had the broken job configuration. we might just end up failing with the same config error in that case | 15:24 |
| dhellmann | good point | 15:25 |
| smcginnis | True | 15:27 |
| *** AJaeger has left #openstack-release | 15:28 | |
| *** samP has quit IRC | 15:33 | |
| *** sree has quit IRC | 15:36 | |
| *** sree has joined #openstack-release | 15:37 | |
| *** samP has joined #openstack-release | 15:37 | |
| *** lucas-hungry is now known as lucasagomes | 15:38 | |
| corvus | dhellmann: you could try setting the parent job protected, defining the child job with the semaphore in the same project and setting it final | 15:38 |
| corvus | https://docs.openstack.org/infra/zuul/user/config.html#attr-job.protected | 15:39 |
| dhellmann | corvus : ok, thanks, we'll look at that | 15:39 |
| corvus | or just wait, as fungi suggests :) | 15:40 |
| prometheanfire | that mass update coming today? | 15:40 |
| dhellmann | prometheanfire : yeah, working on it | 15:41 |
| *** sree has quit IRC | 15:41 | |
| prometheanfire | :D | 15:41 |
| corvus | pabelanger: and yeah, the job inheritance hierarchy is determined at runtime (because it's determined by what variants match the change in question) so it's not generally possible to evaluate things like modifying final jobs. we could, potentially, evaluate a subset of conditions -- for example, evaluate that all possible job variants include a final job and therefore this could never be | 15:42 |
| corvus | allowed. that may help in many cases, but would not be 100% conclusive. | 15:42 |
| fungi | in this case we didn't deadlock the project (or zuul itself) because we could still propose and merge the config revert, so i think what we have for testing is probably sufficient | 15:43 |
| fungi | being able to identify cases like that in advance of merging them would still be nice, but it's serviceable as-is | 15:43 |
| prometheanfire | currently I'm on my last two reviews, both are gating and expexted to pass | 15:44 |
| pabelanger | corvus: understood | 15:46 |
| openstackgerrit | Merged openstack/releases master: Revert "add semaphore for publishing to the releases website" https://review.openstack.org/539980 | 15:48 |
| *** armax has quit IRC | 15:55 | |
| *** claudiub|2 has joined #openstack-release | 15:57 | |
| *** claudiub has quit IRC | 16:00 | |
| openstackgerrit | Merged openstack/releases master: Revert "final queens library releases" https://review.openstack.org/539982 | 16:00 |
| *** ricolin has quit IRC | 16:10 | |
| *** samP has quit IRC | 16:14 | |
| *** lucasagomes has quit IRC | 16:15 | |
| *** lucasagomes has joined #openstack-release | 16:18 | |
| *** samP has joined #openstack-release | 16:18 | |
| openstackgerrit | Merged openstack/releases master: final queens library releases, attempt 2 https://review.openstack.org/539983 | 16:22 |
| *** ykarel|afk has joined #openstack-release | 16:28 | |
| *** iyamahat has quit IRC | 16:36 | |
| *** yamahata has quit IRC | 16:38 | |
| *** lvdombrkr89 has joined #openstack-release | 16:41 | |
| *** lvdombrkr89 has quit IRC | 16:42 | |
| *** lvdombrkr has quit IRC | 16:44 | |
| dhellmann | smcginnis : it looks like the release jobs are starting to run; so far so good | 16:44 |
| smcginnis | Keeping one eye on it. | 16:45 |
| smcginnis | And fingers crossed. | 16:45 |
| *** pcaruana has quit IRC | 16:45 | |
| *** shardy has quit IRC | 16:52 | |
| prometheanfire | :D | 16:54 |
| *** d0ugal has quit IRC | 17:03 | |
| *** samP has quit IRC | 17:07 | |
| *** harlowja has joined #openstack-release | 17:09 | |
| *** samP has joined #openstack-release | 17:13 | |
| smcginnis | Got a MODULE_FAILURE on python-senlinclient release: http://logs.openstack.org/f8/f84d8220a3df4421c1cfa7ee7b1e551b57c3505d/tag/publish-openstack-releasenotes/49c0e16/job-output.txt.gz#_2018-02-01_17_02_21_681332 | 17:15 |
| smcginnis | Just the release notes job though. | 17:15 |
| dhellmann | I just sent email to the list about that one | 17:17 |
| dhellmann | rsync couldn't find a file, so I suspect it's the race condition issue | 17:17 |
| *** iyamahat has joined #openstack-release | 17:17 | |
| smcginnis | Oh, did I not look high enough? | 17:17 |
| dhellmann | smcginnis : http://logs.openstack.org/f8/f84d8220a3df4421c1cfa7ee7b1e551b57c3505d/tag/publish-openstack-releasenotes/49c0e16/ara/result/f3abf1b3-1c58-4f19-9b32-9952c239991b/ | 17:18 |
| dhellmann | I've started using ara instead of trying to parse the giant log file myself | 17:19 |
| dhellmann | once I figured out which things on the screen were useful to click on it proved to be a lot easier to use | 17:19 |
| * smcginnis needs to do that more | 17:19 | |
| smcginnis | Yeah, looks like maybe the same rsync issue. | 17:19 |
| *** d0ugal has joined #openstack-release | 17:20 | |
| dhellmann | shall we approve https://review.openstack.org/539681 and create the branches? | 17:20 |
| smcginnis | Yep, we're ready now I think. Done. | 17:21 |
| *** ykarel|afk has quit IRC | 17:22 | |
| openstackgerrit | Merged openstack/release-test master: Update reno for stable/queens https://review.openstack.org/514679 | 17:22 |
| *** yamahata has joined #openstack-release | 17:27 | |
| *** jtomasek_ has quit IRC | 17:31 | |
| openstackgerrit | Merged openstack/releases master: final queens branches for libraries https://review.openstack.org/539681 | 17:33 |
| *** jpich has quit IRC | 17:45 | |
| *** Qiming has quit IRC | 17:49 | |
| *** Qiming has joined #openstack-release | 17:49 | |
| *** iyamahat has quit IRC | 17:52 | |
| *** iyamahat has joined #openstack-release | 17:54 | |
| *** ekcs has joined #openstack-release | 17:57 | |
| smcginnis | OK, looks like everything has completed and looks OK. I will work on the requirements patch after I grab some food. | 18:11 |
| *** armax has joined #openstack-release | 18:16 | |
| *** yamamoto has quit IRC | 18:17 | |
| *** openstackgerrit has quit IRC | 18:18 | |
| *** harlowja has quit IRC | 18:29 | |
| *** dtantsur is now known as dtantsur|afk | 18:45 | |
| *** samP has quit IRC | 18:48 | |
| *** samP has joined #openstack-release | 18:52 | |
| dhellmann | whew | 18:54 |
| *** bobh has quit IRC | 18:59 | |
| *** samP has quit IRC | 19:04 | |
| *** annabelleB has joined #openstack-release | 19:05 | |
| *** samP has joined #openstack-release | 19:07 | |
| *** harlowja has joined #openstack-release | 19:09 | |
| *** harlowja_ has joined #openstack-release | 19:11 | |
| *** harlowja has quit IRC | 19:14 | |
| *** yamamoto has joined #openstack-release | 19:18 | |
| *** armax_ has joined #openstack-release | 19:23 | |
| *** armax has quit IRC | 19:23 | |
| *** armax_ is now known as armax | 19:23 | |
| *** nikhil has quit IRC | 19:26 | |
| *** yamamoto has quit IRC | 19:29 | |
| *** bobh has joined #openstack-release | 19:29 | |
| *** samP has quit IRC | 19:33 | |
| *** bobh has quit IRC | 19:34 | |
| *** samP has joined #openstack-release | 19:37 | |
| *** annabelleB has quit IRC | 19:39 | |
| *** annabelleB has joined #openstack-release | 19:42 | |
| *** pkovar has quit IRC | 19:42 | |
| *** annabelleB has quit IRC | 19:49 | |
| *** bobh has joined #openstack-release | 19:53 | |
| *** iyamahat_ has joined #openstack-release | 19:54 | |
| *** lucasagomes is now known as lucas-afk | 19:55 | |
| *** iyamahat has quit IRC | 19:56 | |
| *** annabelleB has joined #openstack-release | 19:58 | |
| *** bobh has quit IRC | 19:59 | |
| ttx | yay | 20:01 |
| prometheanfire | probably not worth moving it forward in the queue (seems more tame) | 20:03 |
| *** bobh has joined #openstack-release | 20:04 | |
| *** samP has quit IRC | 20:07 | |
| *** bobh has quit IRC | 20:09 | |
| *** samP has joined #openstack-release | 20:11 | |
| *** pbandark has quit IRC | 20:12 | |
| *** d0ugal has quit IRC | 20:13 | |
| *** bobh has joined #openstack-release | 20:16 | |
| *** bobh has quit IRC | 20:21 | |
| *** d0ugal has joined #openstack-release | 20:25 | |
| *** bobh has joined #openstack-release | 20:31 | |
| *** bobh has quit IRC | 20:36 | |
| mordred | smcginnis: speaking of stale pypi things - I saw you merged the 0.11.2 release patch from sdk but I don't see it on pypi - I assume you already have lists of release jobs that went sideways, but thought I'd mention in case you don't | 20:39 |
| smcginnis | mordred: I did not see a failure for that one. That's not just missing from our bandersnatch issues? | 20:40 |
| *** bobh has joined #openstack-release | 20:40 | |
| mordred | smcginnis: no - missing from pypi itself | 20:41 |
| smcginnis | Hmm, yeah, just saw that. | 20:41 |
| smcginnis | Looks like it wasn't tagged either. | 20:44 |
| mordred | oh - so maybe issue with the tag job? what's the job that submits tags called? | 20:45 |
| *** bobh has quit IRC | 20:45 | |
| smcginnis | tag-releases | 20:46 |
| smcginnis | Still digging. | 20:46 |
| mordred | hrm. yah - I don't see any immediate failures. WEIRD | 20:46 |
| smcginnis | I'm not even seeing any logs. | 20:48 |
| *** bobh has joined #openstack-release | 20:48 | |
| smcginnis | dhellmann: I may need your expertise tracking this down. ^^ | 20:48 |
| *** samP has quit IRC | 20:48 | |
| smcginnis | Though I think we will probably just want to revert and redo. | 20:49 |
| dhellmann | smcginnis : looking | 20:49 |
| dhellmann | the tag doesn't seem to have been applied | 20:50 |
| smcginnis | dhellmann: I don't even see logs for it. | 20:51 |
| dhellmann | and yes without any logs who knows what happened | 20:51 |
| dhellmann | yeah, if the job didn't fail the logs wouldn't have been saved | 20:51 |
| smcginnis | Oh right, I forgot about that change. | 20:51 |
| smcginnis | No release-announce either. It just disappeared. | 20:52 |
| smcginnis | I will revert and redo that one. | 20:52 |
| *** openstackgerrit has joined #openstack-release | 20:52 | |
| openstackgerrit | Doug Hellmann proposed openstack/releases master: save the logs from tag-releases https://review.openstack.org/540134 | 20:52 |
| openstackgerrit | Sean McGinnis proposed openstack/releases master: Revert "Release 0.11.2 of openstacksdk for queens" https://review.openstack.org/540135 | 20:52 |
| *** bobh has quit IRC | 20:53 | |
| *** samP has joined #openstack-release | 20:53 | |
| dhellmann | smcginnis : let's merge that zuul config change before re-approving it ^^ | 20:54 |
| smcginnis | dhellmann: Syntax error. | 20:54 |
| dhellmann | bah | 20:55 |
| openstackgerrit | Doug Hellmann proposed openstack/releases master: save the logs from tag-releases https://review.openstack.org/540134 | 20:55 |
| prometheanfire | smcginnis: why do we need 0.11.2? | 20:56 |
| smcginnis | mordred can probably explain, but I believe there were neutron errors. | 20:57 |
| prometheanfire | ah, k | 20:57 |
| smcginnis | mordred: Or can we stay with 0.11.1 for now? | 20:57 |
| *** samP has quit IRC | 21:03 | |
| mordred | smcginnis: 0.11.2 would be great - the bug that was found in osc's neutron support is a real bug in sdk and will make several normal actions with osc broken | 21:03 |
| *** bobh has joined #openstack-release | 21:03 | |
| smcginnis | mordred: OK, sounds like we will want this included then. | 21:03 |
| mordred | by that I mean we should get 0.11.2 into the list by queens release ... but it's not like a 'must get it in today' if there are other issues | 21:03 |
| smcginnis | Well, I think we must get it in today, otherwise it's way past the freeze and won't make it in queens. | 21:04 |
| mordred | gotcha! then yes - it's important to get it in today - I think tosky would be sad otherwise | 21:05 |
| smcginnis | Now if the gate and bandersnatch will just cooperate. | 21:06 |
| mordred | smcginnis: should I go ahead and submit a FFE for it? | 21:06 |
| *** yamahata has quit IRC | 21:06 | |
| *** iyamahat_ has quit IRC | 21:06 | |
| smcginnis | mordred: I have a blanket one out there. Want to just reply to that with the version number change and I'll update my batch patch with .2. | 21:07 |
| smcginnis | Assuming prometheanfire is still OK with this. | 21:07 |
| prometheanfire | just add it :P | 21:07 |
| prometheanfire | to the mass update | 21:07 |
| smcginnis | What's one more, right? :) | 21:07 |
| smcginnis | I do have the package in there, but I believe I have it with .1. | 21:07 |
| *** bobh has quit IRC | 21:08 | |
| prometheanfire | :D | 21:08 |
| smcginnis | Oh right, I don't have it in there because the .1 update was already going through. Easy enough to add it though. | 21:09 |
| prometheanfire | first review of the last (now abandoned) was 0.11.0, what's currently in gate is 0.11.1 | 21:09 |
| *** samP has joined #openstack-release | 21:09 | |
| mordred | smcginnis: if you rebase that change on top of the exisitng 0.11.1 constraint update ... https://review.openstack.org/#/c/539682/ | 21:09 |
| prometheanfire | mordred: yep | 21:09 |
| smcginnis | prometheanfire: I wonder if we should abandon that 0.11.1 at this point. | 21:09 |
| mordred | prometheanfire, smcginnis: k. replied to the blanket FFE | 21:10 |
| ttx | smcginnis: how are things going ? | 21:11 |
| smcginnis | ttx: Oh, the typical room on fire, but not bad. | 21:11 |
| smcginnis | ttx: Shouldn't you be asleep? Or travelling again this week? | 21:11 |
| * mordred pours some more gas on smcginnis | 21:11 | |
| smcginnis | :) | 21:11 |
| prometheanfire | smcginnis: ya, we probably should if we are targeting 0.11.2 instead | 21:11 |
| ttx | no, catching up since I took a few hours off this afternoon | 21:12 |
| smcginnis | ttx: Quick update is other than openstacksdk, I believe everything for queens is out. | 21:12 |
| smcginnis | ttx: FFE for the requirements bump is OKd and I have a batch of all of them in one to minimize things. | 21:12 |
| smcginnis | Just waiting for an out of sync pypi mirror to catch up and a redo of the openstacksdk 0.11.2 release to go through. | 21:13 |
| ttx | smcginnis: ok well keep me posted if there is anything more to do in teh European morning | 21:13 |
| smcginnis | ttx: Thanks, will do. | 21:13 |
| ttx | looks like there may not be much | 21:13 |
| prometheanfire | yarp | 21:13 |
| smcginnis | ttx: Hopefully not. | 21:13 |
| *** samP has quit IRC | 21:13 | |
| openstackgerrit | Merged openstack/releases master: Revert "Release 0.11.2 of openstacksdk for queens" https://review.openstack.org/540135 | 21:16 |
| openstackgerrit | Merged openstack/releases master: save the logs from tag-releases https://review.openstack.org/540134 | 21:16 |
| openstackgerrit | Sean McGinnis proposed openstack/releases master: Release 0.11.2 of openstacksdk (take 2) https://review.openstack.org/540140 | 21:16 |
| *** bobh has joined #openstack-release | 21:16 | |
| *** samP has joined #openstack-release | 21:19 | |
| *** ianychoi_ has joined #openstack-release | 21:21 | |
| *** bobh has quit IRC | 21:21 | |
| *** ianychoi has quit IRC | 21:24 | |
| dhellmann | ugh | 21:26 |
| dhellmann | "unable to modify final job" | 21:26 |
| openstackgerrit | Doug Hellmann proposed openstack/releases master: Revert "save the logs from tag-releases" https://review.openstack.org/540144 | 21:27 |
| dhellmann | smcginnis : ^^ | 21:27 |
| dhellmann | those failures don't give any indication at all which patches failed | 21:28 |
| smcginnis | dhellmann: I'm assuming they were leftovers from reverting the older patch? | 21:29 |
| dhellmann | yeah, one may have been the patch to add that flag and then the other would have been the revert | 21:30 |
| dhellmann | so I think we can ignore the failures | 21:30 |
| dhellmann | corvus , fungi : is there some way to tell what patch failed for this notification? http://lists.openstack.org/pipermail/release-job-failures/2018-February/000756.html | 21:30 |
| smcginnis | I'm willing to try without reverting that save logs change. | 21:30 |
| dhellmann | no, we need to revert that otherwise the post jobs aren't going to run | 21:30 |
| dhellmann | pabelanger : maybe you can give us a hint? ^ | 21:31 |
| *** bobh has joined #openstack-release | 21:31 | |
| fungi | i expect we need to check the scheduler logs | 21:32 |
| corvus | that's definitely an oversight; the reporter has the information about the item being processed, so we can improve it to include that in the future. but yes, at the moment, as fungi says we'll need to check the logs | 21:32 |
| *** samP has quit IRC | 21:32 | |
| corvus | fungi: you on that, or should i? | 21:32 |
| fungi | i'm checking logs at the moment | 21:33 |
| fungi | 2018-02-01 21:19:15,920 ERROR zuul.Pipeline.openstack.release-post: Error freezing job graph for <QueueItem 0x7fba39e55240 for <Branch 0x7fbd891570b8 refs/heads/master updated 93c69d3eb576bc55eaa8d42f02a9a715fffee69e..594e3d953b5fe2c0742927d545e7fe03c9dd4dd9> in release-post> | 21:33 |
| fungi | looks to be the one | 21:34 |
| *** kumarmn_ has joined #openstack-release | 21:34 | |
| dhellmann | ok, that's the one to add the bad flag | 21:34 |
| fungi | so commit 594e3d9 i guess | 21:34 |
| dhellmann | there's another one in http://lists.openstack.org/pipermail/release-job-failures/2018-February/000756.html | 21:34 |
| dhellmann | which *should* be the commit to revert the tag request for openstacksdk | 21:34 |
| smcginnis | 594e3d9 was the merge commit for "save the logs" | 21:35 |
| fungi | wrong url? that's the same message number you linked earlier | 21:35 |
| dhellmann | oops: http://lists.openstack.org/pipermail/release-job-failures/2018-February/000757.html | 21:36 |
| fungi | yeah, just spotted it | 21:36 |
| fungi | so i guess we have two which reported at the same time | 21:36 |
| dhellmann | corvus : when did the log handling default change? we just had a tag-releases job pass without any logs earlier today (that's what prompted my patch) | 21:36 |
| smcginnis | The revert merged right about the same time. | 21:36 |
| dhellmann | yeah | 21:37 |
| dhellmann | ok, so we can safely ignore those | 21:37 |
| fungi | 2018-02-01 21:18:46,506 ERROR zuul.Pipeline.openstack.release-post: Error freezing job graph for <QueueItem 0x7fba39e55c50 for <Branch 0x7fbb017ad7b8 refs/heads/master updated dc0a857d6f28df9201ecee70ab3fc960f948ff33..93c69d3eb576bc55eaa8d42f02a9a715fffee69e> in release-post> | 21:37 |
| openstackgerrit | James E. Blair proposed openstack/releases master: Remove log handling override https://review.openstack.org/540148 | 21:37 |
| *** bobh has quit IRC | 21:37 | |
| *** kumarmn has quit IRC | 21:37 | |
| corvus | dhellmann: friday i believe | 21:37 |
| *** samP has joined #openstack-release | 21:37 | |
| dhellmann | corvus : it changed back to true? because if that's the case we're missing some data somewhere. | 21:37 |
| corvus | dhellmann: yeah, any missing logs at this point is an error; we can run it down if you have clues | 21:38 |
| dhellmann | smcginnis : do you still have the reference to the one where the logs were gone? | 21:38 |
| *** bobh has joined #openstack-release | 21:38 | |
| fungi | those were the only two i found in the scheduler log for the past few hours (since the earlier ones around 15:15z) | 21:38 |
| dhellmann | corvus : either d7e2e06b671bd19950f61d71cd15f48179994791 or the merge commit 645d669f46e1396cf9daab78ef39ee19bff9bf0a | 21:39 |
| dhellmann | I expected to find logs at http://logs.openstack.org/d7/d7e2e06b671bd19950f61d71cd15f48179994791/ or http://logs.openstack.org/64/645d669f46e1396cf9daab78ef39ee19bff9bf0a/ and got 404 errors at both | 21:40 |
| dhellmann | fungi : thanks, I think we're safe to continue retrying after reverting my patch that broke the zuul config for that job | 21:41 |
| *** samP has quit IRC | 21:41 | |
| corvus | dhellmann: for release-post pipeline? it hit a final modification error | 21:41 |
| corvus | dhellmann: oh wait | 21:42 |
| corvus | i may have read a line wrong | 21:42 |
| dhellmann | that's possible we've had 2 of those this afternoon | 21:42 |
| corvus | heh, okay, i did read a line wrong, but the right line also says the same thing | 21:42 |
| corvus | 2018-02-01 15:03:30,111 DEBUG zuul.SMTPReporter: Report change <Branch 0x7fbb00c686d8 refs/heads/master updated 47a33ea0a02465c9b327f93eb4899ad9d903d8be..645d669f46e1396cf9daab78ef39ee19bff9bf0a>, params {'to': 'release-job-failures@lists.openstack.org', 'from': 'zuul@openstack.org', 'subject': 'release-post job for {change.project} failed'}, message: Unable to freeze job graph: Unable to modify | 21:42 |
| corvus | final job <Job publish-static branches: None source: openstack-infra/project-config/zuul.d/jobs.yaml@master#26> attribute semaphore=static-publish-releases with variant <Job publish-static branches: {MatchAll:{ImpliedBranchMatcher:master}} source: openstack/releases/.zuul.yaml@master#13> | 21:42 |
| corvus | i think that's the right one | 21:42 |
| dhellmann | ok | 21:42 |
| dhellmann | is it possible to detect that error (modifying a final job) before merging the patch? | 21:43 |
| dhellmann | I know some of these cases aren't, but that one seems like it could be | 21:43 |
| dhellmann | s/aren't/can't be spotted/ | 21:43 |
| *** bobh has quit IRC | 21:43 | |
| corvus | dhellmann: not reliably (see convo with pabelanger at 15:42). we might be able to do it *unreliably* which still may be helpful | 21:43 |
| smcginnis | Sorry, it was 645d669f46e1396cf9daab78ef39ee19bff9bf0a I was looking at before, but I see you got that. | 21:44 |
| dhellmann | ok | 21:44 |
| *** samP has joined #openstack-release | 21:45 | |
| corvus | (basically, it's hard to be sure that it will succeed in all circumstances. but it's easier (not easy, but easier) to determine that it will fail in all circumstances) | 21:46 |
| dhellmann | that sounds like it would still be helpful some of the time | 21:46 |
| dhellmann | I guess if it's not very easy though that's just something I can pay closer attention to when I start making patches | 21:46 |
| dhellmann | at some point we wanted to move the definition of that job into the releases repo anyway, and that would make it easier to modify it in these cases | 21:47 |
| smcginnis | Not too many jobs in queue right now, but those patches are just sitting waiting. | 21:50 |
| *** bobh has joined #openstack-release | 21:53 | |
| dhellmann | odd | 21:57 |
| *** samP has quit IRC | 21:57 | |
| *** bobh has quit IRC | 21:59 | |
| *** samP has joined #openstack-release | 22:02 | |
| *** bobh has joined #openstack-release | 22:15 | |
| *** bobh has quit IRC | 22:18 | |
| *** bobh has joined #openstack-release | 22:18 | |
| *** mriedem is now known as mriedem_parent | 22:20 | |
| *** hongbin has quit IRC | 22:23 | |
| *** dave-mccowan has quit IRC | 22:30 | |
| annabelleB | hey smcginnis—I saw the release highlights page was still empty, so I was thinking I would push some example highlights (things from the roadmap that I believe to still be true) to 4 projects to get the party started. Think that’s a good idea? | 22:37 |
| *** kumarmn_ has quit IRC | 22:37 | |
| smcginnis | annabelleB: Sure, we could do that to get things started. The PTLs can always modify it later if they'd like. | 22:38 |
| smcginnis | annabelleB: That might be nice to get something out there. | 22:38 |
| annabelleB | smcginnis: was thinking the same—they can always take my 2 lines out | 22:38 |
| smcginnis | annabelleB: Works for me. :) | 22:38 |
| *** wolverin_ has joined #openstack-release | 22:38 | |
| openstackgerrit | Anne Bertucio proposed openstack/releases master: Add Queens cycle-highlights messages https://review.openstack.org/540171 | 22:39 |
| *** wolverineav has quit IRC | 22:39 | |
| * mordred has learned a new thing | 22:39 | |
| *** iyamahat has joined #openstack-release | 22:39 | |
| mordred | ooh - I have learned two things - about cycle-highlights, and that cinder multiattach landed! | 22:40 |
| smcginnis | \o/ | 22:41 |
| *** wolverin_ has quit IRC | 22:41 | |
| openstackgerrit | Monty Taylor proposed openstack/releases master: Add cycle-highlights for openstacksdk for queens https://review.openstack.org/540173 | 22:44 |
| openstackgerrit | Merged openstack/releases master: Revert "save the logs from tag-releases" https://review.openstack.org/540144 | 22:45 |
| *** samP has quit IRC | 22:45 | |
| smcginnis | annabelleB: Awesome, I was going to do that, but I think it makes it much more visible coming from you. | 22:49 |
| *** samP has joined #openstack-release | 22:49 | |
| annabelleB | happy to! | 22:49 |
| annabelleB | a good topic to revisit in Dublin too | 22:50 |
| smcginnis | annabelleB: Ooh, yeah. If we have a nicely filled in example for queens we can show at a lunch time session or something, that would really help getting it out there. | 22:52 |
| annabelleB | i’d love that! | 22:53 |
| *** annabelleB has quit IRC | 22:53 | |
| *** kumarmn has joined #openstack-release | 22:55 | |
| *** edmondsw has quit IRC | 22:56 | |
| *** yamahata has joined #openstack-release | 22:58 | |
| *** kumarmn has quit IRC | 22:58 | |
| *** kumarmn has joined #openstack-release | 22:58 | |
| *** annabelleB has joined #openstack-release | 22:58 | |
| *** samP has quit IRC | 23:02 | |
| *** samP has joined #openstack-release | 23:07 | |
| *** bobh has quit IRC | 23:13 | |
| *** wolverineav has joined #openstack-release | 23:14 | |
| *** samP has quit IRC | 23:14 | |
| *** annabelleB has quit IRC | 23:16 | |
| *** annabelleB has joined #openstack-release | 23:16 | |
| *** samP has joined #openstack-release | 23:18 | |
| *** inc0 has quit IRC | 23:22 | |
| *** inc0 has joined #openstack-release | 23:22 | |
| openstackgerrit | Merged openstack/releases master: Release 0.11.2 of openstacksdk (take 2) https://review.openstack.org/540140 | 23:25 |
| *** samP has quit IRC | 23:34 | |
| *** wolverineav has quit IRC | 23:36 | |
| *** samP has joined #openstack-release | 23:38 | |
| *** wolverineav has joined #openstack-release | 23:39 | |
| *** samP has quit IRC | 23:43 | |
| *** wolverineav has quit IRC | 23:45 | |
| *** annabelleB has quit IRC | 23:47 | |
| *** annabelleB has joined #openstack-release | 23:47 | |
| *** samP has joined #openstack-release | 23:49 | |
| *** samP has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!