| *** armax has quit IRC | 03:49 | |
| *** armax has joined #openstack-release | 03:53 | |
| *** armax has quit IRC | 04:13 | |
| *** ykarel has joined #openstack-release | 05:05 | |
| *** whoami-rajat__ has joined #openstack-release | 05:25 | |
| *** evrardjp has quit IRC | 05:33 | |
| *** evrardjp has joined #openstack-release | 05:33 | |
| *** dirk has quit IRC | 07:09 | |
| *** dirk has joined #openstack-release | 07:10 | |
| *** snapiri has quit IRC | 08:15 | |
| openstackgerrit | Rodolfo Alonso proposed openstack/releases master: Bump neutron-lib to 2.8.0 https://review.opendev.org/c/openstack/releases/+/769882 | 08:26 |
|---|---|---|
| hberaud | The latest job failure (kolla) [1] seems related to a docker pull limit[2]. [1] http://lists.openstack.org/pipermail/openstack-discuss/2021-January/019721.html [2] http://lists.openstack.org/pipermail/openstack-discuss/2020-December/019148.html | 08:29 |
| *** rpittau|afk is now known as rpittau | 08:32 | |
| *** ykarel is now known as ykarel|lunch | 08:39 | |
| *** tkajinam has quit IRC | 08:55 | |
| *** tosky has joined #openstack-release | 08:56 | |
| *** jbadiapa has joined #openstack-release | 09:23 | |
| openstackgerrit | Rodolfo Alonso proposed openstack/releases master: Release openstacksdk 0.53.0 https://review.opendev.org/c/openstack/releases/+/769897 | 09:28 |
| *** brinzhang has quit IRC | 09:38 | |
| *** ykarel|lunch is now known as ykarel | 09:54 | |
| *** dtantsur|afk is now known as dtantsur | 09:57 | |
| *** whoami-rajat__ has quit IRC | 10:13 | |
| *** ttx has quit IRC | 10:16 | |
| *** ttx has joined #openstack-release | 10:17 | |
| ykarel | hberaud, kolla tarballs are also missing too, is that also an known issue? | 10:47 |
| ykarel | https://tarballs.opendev.org/openstack/kolla/?C=M;O=D | 10:47 |
| ykarel | recent releases are not in tarballs, i see release job passed for those though but no tarball | 10:47 |
| ykarel | our jobs which relies on those tarballs failed as those are missing https://review.rdoproject.org/r/31499 https://review.rdoproject.org/r/31498 https://review.rdoproject.org/r/31497 | 10:49 |
| ykarel | may be this is same issue related to AFS which was hit during release time that fungi fixed | 10:51 |
| ykarel | ex log release job https://7d350f20fd3e3446f74e-13e23d0c55ae164572ef3fa72875b3c8.ssl.cf2.rackcdn.com/6ea8c0062c08050a3037d0df379d2c62b84746e1/release/release-openstack-python/2ed64df/job-output.txt | 10:51 |
| openstackgerrit | Pierre Riteau proposed openstack/releases master: Release blazar-tempest-plugin for wallaby https://review.opendev.org/c/openstack/releases/+/769524 | 10:54 |
| hberaud | ykarel: have you seen => http://lists.openstack.org/pipermail/openstack-discuss/2021-January/019721.html ? | 11:05 |
| ykarel | hberaud, yes i saw | 11:06 |
| ykarel | is that cuased tarballs missing issue? | 11:06 |
| *** e0ne has joined #openstack-release | 11:11 | |
| hberaud | ykarel: Indeed missing tarballs don't seems related to the ML thread issue, lemme check more deeply | 11:11 |
| ykarel | hberaud, okk | 11:12 |
| ykarel | seems more infra related | 11:12 |
| hberaud | ykarel: lemme know if you find something | 11:12 |
| ykarel | hberaud, i just recall similar issue during release time | 11:13 |
| hberaud | ack | 11:14 |
| ykarel | hberaud, http://eavesdrop.openstack.org/irclogs/%23openstack-release/%23openstack-release.2020-10-07.log.html more context | 11:15 |
| openstackgerrit | Marios Andreou proposed openstack/releases master: Release victoria tripleo-ipsec os-collect-config os-refresh-config https://review.opendev.org/c/openstack/releases/+/769915 | 12:39 |
| hberaud | ykarel: ack thanks | 12:50 |
| *** armax has joined #openstack-release | 13:08 | |
| *** whoami-rajat___ has joined #openstack-release | 13:18 | |
| hberaud | ykarel: I think I got it | 13:19 |
| hberaud | ykarel: yesterday fungi noticed that the AFS server was done during ~20 minutes | 13:20 |
| hberaud | ykarel: and the timestamp of the merge seems to match this period | 13:21 |
| hberaud | ykarel: http://eavesdrop.openstack.org/irclogs/%23openstack-release/%23openstack-release.2021-01-07.log.html#t2021-01-07T16:48:57 | 13:22 |
| hberaud | ykarel: kolla 10.2.0 was released at the same time https://review.opendev.org/c/openstack/releases/+/769324 | 13:23 |
| ykarel | hberaud, okk cool, so those jobs need to rerun now? | 13:24 |
| ykarel | all three kolla releases | 13:24 |
| hberaud | ykarel: I think yes, also mgoddard proposed to reenqueue the jobs for the failing docker jobs http://lists.openstack.org/pipermail/openstack-discuss/2021-January/019722.html | 13:26 |
| hberaud | the 3 tarballs are missing? | 13:26 |
| hberaud | ykarel: ^ | 13:26 |
| hberaud | kolla-ansible is missing too | 13:27 |
| hberaud | and kayobe 8.1.0 too | 13:27 |
| ykarel | hberaud, yes openstack-kolla-9.3.0-1 , openstack-kolla-10.2.0-1, openstack-kolla-11.0.0-0.2.0rc2 this what rdo automation caught | 13:27 |
| ykarel | kolla-ansible and kayobe we don't package so not caught on our side | 13:28 |
| hberaud | yes it match my search | 13:28 |
| ykarel | ok please reenque these as well | 13:28 |
| hberaud | ah so we also need to reenqueue things for https://review.opendev.org/c/openstack/releases/+/769325 and https://review.opendev.org/c/openstack/releases/+/769322 | 13:29 |
| hberaud | so to summarize I don't think that's exactly the same problem that we met in october (c.f http://eavesdrop.openstack.org/irclogs/%23openstack-release/%23openstack-release.2020-10-07.log.html ) but apprently it is an AFS issue too | 13:31 |
| ykarel | okk | 13:32 |
| hberaud | ykarel: I just sent a reply about this on the ML thread related to kolla failing jobs http://lists.openstack.org/pipermail/openstack-discuss/2021-January/019729.html | 13:45 |
| openstackgerrit | Rodolfo Alonso proposed openstack/releases master: Release neutron-lib to 2.8.0 https://review.opendev.org/c/openstack/releases/+/769882 | 13:50 |
| ykarel | hberaud, Thanks, so someone from infra will enque those | 13:55 |
| fungi | ykarel: hberaud: it was more than a 20-minute outage. more like 10-11 hours. there weren't any releases in that timeframe, but if this is about missing branch tarballs yes that could have been related | 13:57 |
| fungi | which tarballs were missing exactly? | 13:58 |
| *** dtantsur is now known as dtantsur|brb | 14:00 | |
| hberaud | fungi: all the tarballs generated by the gerrit patches listed there => http://lists.openstack.org/pipermail/openstack-discuss/2021-January/019729.html | 14:02 |
| hberaud | fungi: - https://review.opendev.org/c/openstack/releases/+/769325 | 14:03 |
| hberaud | - https://review.opendev.org/c/openstack/releases/+/769322 | 14:03 |
| hberaud | - https://review.opendev.org/c/openstack/releases/+/769324 | 14:03 |
| hberaud | for kolla: 9.3.0, 10.2.0, 11.0.0.0rc2 | 14:04 |
| hberaud | for kolla-ansible: 9.3.0, 10.2.0, 11.0.0.0rc2 | 14:05 |
| hberaud | for kayobe: 7.2.0, 8.1.0, 9.0.0.0rc2 | 14:06 |
| hberaud | However AFAIK I didn't noticed "[Release-job-failures]" automatic threads related to tarballs | 14:08 |
| fungi | did the upload jobs fail, or are they just not appearing on the tarballs site? i can check in a moment but it's possible the volume releasing is broken if it was mid-replication when the active server hung yesterday, so may just need the read-only volume re-replicated | 14:31 |
| fungi | there may be other volumes in a similar state i need to correct if so | 14:32 |
| hberaud | fungi: For now (and without too digging) I haven't noticed related jobs fail | 14:33 |
| fungi | got it, in that case it's almost certainly just a stale read-only replica. i'll get that and any others updating again shortly and let you know for sure | 14:34 |
| hberaud | ack thanks | 14:38 |
| fungi | yep, that's the problem. it's there on the read-write volume /afs/.openstack.org/project/tarballs.opendev.org/openstack/kolla/kolla-9.3.0.tar.gz just not the read-only volume | 14:44 |
| hberaud | ok | 14:44 |
| hberaud | ykarel: FYI ^ | 14:45 |
| *** rpittau is now known as rpittau|afk | 14:46 | |
| fungi | there was a hung cronjob for our periodic vos release runs from yesterday which i've killed, will see if it hits any errors when its triggered again in a couple minutes | 14:48 |
| ykarel | hberaud, Thanks | 14:49 |
| hberaud | fungi: ack | 14:50 |
| fungi | it's releasing the docs volume first, that tends to have churn so will take a few minutes | 14:52 |
| hberaud | ack | 14:53 |
| *** e0ne has quit IRC | 14:54 | |
| fungi | hberaud: ykarel: tarballs volume release finished at 15:15:40 and i see 9.3.0 artifacts at https://tarballs.opendev.org/openstack/kolla/ | 15:18 |
| fungi | presumably the other missing ones are present now as well | 15:18 |
| hberaud | 10.2.0 & 11.0.0.0rc2 are there too | 15:19 |
| fungi | doesn't appear we had any corrupt replicas, just the volume release process was hung do to it trying to talk to an unresponsive server | 15:20 |
| hberaud | kolla-ansible LGTM too https://tarballs.opendev.org/openstack/kolla-ansible/ | 15:20 |
| fungi | er, due to | 15:20 |
| hberaud | same thing for kayobe LGTM too https://tarballs.opendev.org/openstack/kayobe/ | 15:22 |
| hberaud | it explain why we didn't get release job failure | 15:23 |
| hberaud | s/failure/failures/ | 15:23 |
| hberaud | ykarel: can we `recheck` on RDO? | 15:24 |
| ykarel | hberaud, sure doing | 15:25 |
| hberaud | thx | 15:25 |
| *** e0ne has joined #openstack-release | 15:27 | |
| *** dtantsur|brb is now known as dtantsur | 15:45 | |
| ykarel | hberaud, RDO jobs passed now | 16:13 |
| hberaud | awesome | 16:14 |
| *** ykarel_ has joined #openstack-release | 16:26 | |
| *** ykarel has quit IRC | 16:29 | |
| fungi | i also spotted the ml thread and replied with accurate details | 16:32 |
| prometheanfire | https://review.opendev.org/769875 ? | 16:38 |
| prometheanfire | just want to verify that's good, happy, just surprised | 16:38 |
| *** whoami-rajat___ is now known as whoami-rajat__ | 16:39 | |
| hberaud | fungi: thanks | 16:51 |
| hberaud | prometheanfire: I replied directly on the patch | 17:00 |
| prometheanfire | cool | 17:00 |
| hberaud | prometheanfire: AFAIK these changes LGTM | 17:01 |
| *** ykarel_ is now known as ykarel | 17:21 | |
| *** sboyron has quit IRC | 17:39 | |
| *** sboyron has joined #openstack-release | 17:41 | |
| *** ykarel has quit IRC | 17:42 | |
| *** hberaud has quit IRC | 18:21 | |
| *** hberaud has joined #openstack-release | 18:22 | |
| *** dtantsur is now known as dtantsur|afk | 18:43 | |
| *** whoami-rajat__ has quit IRC | 18:58 | |
| *** e0ne has quit IRC | 20:24 | |
| *** sboyron has quit IRC | 20:36 | |
| *** diablo_rojo__ has joined #openstack-release | 20:59 | |
| *** slaweq has quit IRC | 21:13 | |
| *** brtknr has joined #openstack-release | 22:49 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!