Monday, 2020-06-15

*** e0ne has quit IRC00:02
*** tetsuro has joined #openstack-release00:38
*** slaweq has joined #openstack-release00:38
*** slaweq has quit IRC00:52
*** slaweq has joined #openstack-release00:58
*** slaweq has quit IRC01:03
*** ykarel|away is now known as ykarel03:55
*** evrardjp has quit IRC04:33
*** evrardjp has joined #openstack-release04:33
*** ricolin has joined #openstack-release04:42
*** udesale has joined #openstack-release04:44
*** jtomasek has joined #openstack-release04:49
*** irclogbot_2 has quit IRC06:05
*** irclogbot_3 has joined #openstack-release06:07
*** tetsuro_ has joined #openstack-release06:10
*** tetsuro has quit IRC06:13
*** priteau has joined #openstack-release06:34
openstackgerritMerged openstack/releases master: Release Octavia stable/stein 4.1.2  https://review.opendev.org/73528706:35
openstackgerritMerged openstack/releases master: Release Octavia stable/ussuri 6.0.1  https://review.opendev.org/73526606:39
*** ykarel is now known as ykarel|afk06:56
*** slaweq has joined #openstack-release06:57
*** ykarel|afk is now known as ykarel07:00
*** vishalmanchanda has joined #openstack-release07:03
*** jbadiapa has joined #openstack-release07:18
*** tosky has joined #openstack-release07:28
*** amoralej|off is now known as amoralej07:30
-openstackstatus- NOTICE: uWSGI made a new release that breaks devstack, please refrain from rechecking until a devstack fix is merged.07:43
*** dtantsur|afk is now known as dtantsur07:51
*** rpittau|afk is now known as rpittau08:00
*** ykarel is now known as ykarel|lunch08:04
ttxWe got a bunch of AFS errors impacting releases these last days... Probably better to hold on releases until that's fixed08:31
openstackgerritMerged openstack/releases master: Release ironic-lib 4.2.1 for ussuri  https://review.opendev.org/73487508:33
ttxLooks like that one passed ok ^08:48
*** ykarel|lunch is now known as ykarel08:49
ttxBut we have the following issues open:08:50
ttxpython-troveclient 4.1.0 - tag OK, tarball KO, needs reenqueue of tag event08:51
ttxcliff 3.3.0 - tag OK, tarball KO, needs reenqueue of tag event08:52
ttxoctavia 4.1.2 - tag OK, atrball KO, needs reenqueue of tag event08:53
ttxoctavia 6.0.1 - tag OK, tarball OK, needs reenqueue of tag event08:54
ttxerr08:54
ttxoctavia 6.0.1 - tag OK, tarball KO, needs reenqueue of tag event08:54
openstackgerritMerged openstack/releases master: Release nova for stable/train  https://review.opendev.org/73495209:01
openstackgerritMerged openstack/releases master: Release python-novaclient for stable/train  https://review.opendev.org/73499009:05
openstackgerritMerged openstack/releases master: Release barbican for stable/train  https://review.opendev.org/73492009:06
openstackgerritMerged openstack/releases master: Release python-neutronclient for stable/train  https://review.opendev.org/73498909:08
openstackgerritMerged openstack/releases master: Release tacker for stable/train  https://review.opendev.org/73500409:08
openstackgerritMerged openstack/releases master: Release manila for stable/train  https://review.opendev.org/73493909:08
openstackgerritMerged openstack/releases master: Release zun for stable/train  https://review.opendev.org/73501609:08
*** e0ne has joined #openstack-release09:09
*** priteau has quit IRC09:11
openstackgerritMerged openstack/releases master: Release manila-ui for stable/train  https://review.opendev.org/73493809:11
openstackgerritMerged openstack/releases master: Release zun-ui for stable/train  https://review.opendev.org/73501509:11
openstackgerritMerged openstack/releases master: Release sushy-tools 0.10.0  https://review.opendev.org/73483709:11
openstackgerritMerged openstack/releases master: Rally is a service  https://review.opendev.org/73533709:11
openstackgerritMerged openstack/releases master: Release trove for stable/train  https://review.opendev.org/73501109:11
openstackgerritMerged openstack/releases master: Release watcher for stable/train  https://review.opendev.org/73501309:11
openstackgerritMerged openstack/releases master: Release heat-dashboard for stable/train  https://review.opendev.org/73492609:14
*** priteau has joined #openstack-release09:21
openstackgerritMerged openstack/releases master: Release murano for stable/train  https://review.opendev.org/73494509:23
openstackgerritMerged openstack/releases master: Release ceilometer for stable/train  https://review.opendev.org/73492209:23
openstackgerritMerged openstack/releases master: Release aodh for stable/train  https://review.opendev.org/73491809:24
openstackgerritMerged openstack/releases master: Release vitrage for stable/train  https://review.opendev.org/73501209:24
openstackgerritMerged openstack/releases master: Release glance for stable/train  https://review.opendev.org/73492509:25
openstackgerritMerged openstack/releases master: Release python-glanceclient for stable/train  https://review.opendev.org/73498709:25
openstackgerritMerged openstack/releases master: Release os-win for stable/train  https://review.opendev.org/73495709:25
openstackgerritMerged openstack/releases master: Release ironic-lib for stable/train  https://review.opendev.org/73492909:28
ttxNew AFS errors for sushy-tools 0.10.0 and python-neutronclient 6.14.1 release notes publication09:36
openstackgerritMerged openstack/releases master: Release ironic-inspector for stable/train  https://review.opendev.org/73492809:39
openstackgerritMerged openstack/releases master: Release zaqar for stable/train  https://review.opendev.org/73501409:39
openstackgerritMerged openstack/releases master: Release python-octaviaclient for stable/train  https://review.opendev.org/73499109:40
openstackgerritMerged openstack/releases master: Release neutron-vpnaas-dashboard for stable/train  https://review.opendev.org/73494909:40
ttxand os-win 4.3.309:41
*** tetsuro_ has quit IRC09:42
*** rpittau is now known as rpittau|bbl10:03
cgoncalvesttx, anything octavia folks can help with?10:05
*** ykarel is now known as ykarel|afk11:30
ttxcgoncalves: no, we just need to trigger the job again... which requires infra help. Thanks for the offer to help!12:01
*** amoralej is now known as amoralej|lunch12:03
*** rpittau|bbl is now known as rpittau12:04
ttxzaqar 9.0.1 - tag OK, tarball KO, needs reenqueue of tag event12:05
openstackgerritThierry Carrez proposed openstack/releases master: propose-library-branches: remove unused --all option  https://review.opendev.org/73557312:20
openstackgerritThierry Carrez proposed openstack/releases master: Trailing is a deliverable type, not a release model  https://review.opendev.org/73557412:20
ttxfungi: we have random AFS sync issues causing some release changes to fail. We'll have to reenqueue a number of tag events once we are reasonably sure it's under control (python-troveclient 4.1.0, cliff 3.3.0, octavia 4.1.2, octavia 6.0.1, zaqar 9.0.1)12:24
*** udesale_ has joined #openstack-release12:42
*** udesale has quit IRC12:44
*** ykarel|afk is now known as ykarel12:46
*** priteau has quit IRC12:54
smcginnisWe have a number of release approvals after failures. Anyone know that status of things?12:54
*** priteau has joined #openstack-release12:55
fungii'm still catching up for the morning, but thought the impact from the afs outage was over as of 13:20 yesterday13:03
smcginnisfungi: Looks like there may be another issue. Several release job failures so far this morning.13:03
fungithough i see from the status log the mirror in inap had become unresponsive and was rebooted just before 07:00 utc13:04
fungiyeah, i can try to dig into a more recent failure while half paying attention in the image encryption meeting13:04
smcginnisfungi: I see a comment from ianw less than an hour ago about some issues.13:04
fungiyeah, looks like some of the mirrors servers may have gotten into a hung state after the afs01.dfw outage13:11
ttxsmcginnis: we had some releases processed successfully, but still a higher-than-normal rate of failures, most of which cam be traced back to AFS errors13:15
ttxI listed the half-done releases in my 12:24 utc message above, we should finish those13:16
ttxthe rest is less urgent (docs upload fails, release notes upload fails)13:16
*** amoralej|lunch is now known as amoralej13:20
smcginnisI did one of the propose-update-constraints manually yesterday, but I know we're still missing the tarball for that one.13:22
smcginnisIf I can take care of some $DAY_JOB things quick, I will try to get these added to the etherpad so we can track the status. Unless you've captured it somewhere else ttx.13:23
ttxNo, I just tracked the ones we should reenqueue13:23
fungidocs and release notes publication errors are likely related to afs write problems... were there any after 13:20 yesterday?13:35
fungidid the other failures look like package retrieval errors?13:35
ttxyes, several today and in the last hour13:35
ttxhttps://zuul.opendev.org/t/openstack/build/7cd903749128484d8518a92b14ff993113:35
fungii'll need to take a look at some of those too, in that case13:35
ttx17 minutes ago13:35
ttxThere were package retrieval errors over the weekend, but that's not what we see now13:36
fungii wonder if afs state is hung client side on some/all executors since that's where the writes happen in jobs13:36
ttxsome jobs passed, so I was thinking one of the nodes being hung up13:37
fungifungi@ze04:~$ ls /afs/.openstack.org/docs/13:37
fungils: cannot access '/afs/.openstack.org/docs/': No such file or directory13:37
fungiso there's at least one. still going through them13:38
fungithat's the only one i found, and the job inventory does say that's the executor that failed build ran on13:53
fungianyway, i have a feeling that was the cause of all the publication errors in the past 24 hours. statistically speaking it should have only affected ~8% of builds since 11 of the 12 executors seem fine13:55
fungiand i've taken it out of rotation for now while we investigate that situation13:55
fungiso what did the other failures look like then? were they also hitting the "Synchronize files to AFS" task or something wholly different?13:58
ttxrecent failures were all linked to AFS14:03
ttxjust different jobs failing when they push data up14:04
*** priteau has quit IRC14:12
fungiokay, in that case they probably all hit ze04 but i should probably double-check that14:14
*** armax has joined #openstack-release14:30
*** ykarel is now known as ykarel|away14:56
openstackgerritMerged openstack/release-test master: Update master for stable/3.1  https://review.opendev.org/73417115:06
*** e0ne has quit IRC15:51
*** e0ne_ has joined #openstack-release15:51
*** e0ne_ has quit IRC15:58
*** rpittau is now known as rpittau|afk16:17
*** dtantsur is now known as dtantsur|afk16:22
*** udesale_ has quit IRC16:26
*** e0ne has joined #openstack-release16:32
*** diablo_rojo has joined #openstack-release16:33
*** e0ne has quit IRC16:34
*** e0ne has joined #openstack-release16:34
*** ricolin has quit IRC16:36
*** diablo_rojo has quit IRC16:39
*** e0ne has quit IRC16:54
*** diablo_rojo has joined #openstack-release16:59
*** amoralej is now known as amoralej|off17:32
fungiokay, so afs-related fallout seems to be under control for some hours now... no new reports of problems. i'll work up the list of inventory urls for the reenqueues ttx suggested: python-troveclient 4.1.0, cliff 3.3.0, octavia 4.1.2, octavia 6.0.1, zaqar 9.0.119:06
fungii'm skeptical reenqueuing https://zuul.opendev.org/t/openstack/build/e1bfe640d24c4a418e14945320d96693 will help as it seems to be complaining about finding a suitable version of pbr on pypi (that build was a failure instead of a post_failure like the others too). might be a stable branch deps issue19:12
smcginnisfungi: I went through the failures and tried to capture the info in19:14
smcginnisCopy past is borked on my machine apparently.19:14
smcginnishttps://etherpad.opendev.org/p/victoria-relmgt-tracking19:14
smcginnisThere it goes.19:14
smcginnisfungi: Most I don't think will help to reenqueue.19:15
smcginnisThe release-openstack-python ones I think will see that the tag exists and not do anything else.19:15
smcginnisfungi: We have several that I think are only missing the tarball on tarballs.o.o.19:16
fungiare you sure octavia 4.1.2 failed on tarball upload? the error looks to me like it failed to build the wheel19:16
smcginnisNot sure if we need to do the announcements, but that would be nice if we can just enqueue those directly since they were skipped with the other failures.19:16
smcginnisI'll look again.19:17
fungican't really enqueue individual jobs, we can reenqueue git refs19:17
smcginnisfungi: If we reenqueue the release job and it skips on the tag already existing, do you know if that will trigger the announce job?19:18
fungithe announcement is triggered by the tag ref getting enqueued, i believe, so would not19:18
smcginnisSo I guess we would need to do that manually, or just skip it for these.19:19
smcginnisThat octavia 4.1.2 one - you're right. That failed earlier than I thought. From the retry errors right before the failure, that actually looks like a connectivity issue.19:20
smcginnisSo I think that one can get reenqueued.19:20
fungiand i agree, reenqueuing the ones where the pypi upload succeeded also won't work, we probably need to manyally copy the artifacts from pypi into afs19:20
fungialso if we didn't, the copies on tarballs.o.o would have different checksums than the ones on pypi19:21
clarkbmaybe we should update the jobs to write to afs before pypi19:21
clarkbsince we can always modify afs but can't modify pypi19:21
smcginnisMaybe19:21
smcginnisOctavia 4.1.2 did not get pushed up to pypi. https://pypi.org/project/octavia/#history19:21
fungiyes, i've wanted that for a while, that used to be the order and it was reversed when paul reimplemented the jobs for zuul v319:22
smcginnisSo one reenqueue, several tarball copies.19:22
fungibecause also if we have to copy sdist/wheel files from pypi we don't have the signatures19:22
smcginnisI manually did the requirements updates that were missed.19:22
*** vishalmanchanda has quit IRC19:22
smcginnisAnd we just lose out on some release announcements.19:22
fungisudo zuul enqueue-ref --tenant=openstack --trigger=gerrit --pipeline=release --project=openstack/octavia --ref=refs/tags/4.1.2 --newrev=0812a1231e3501207236d43fc51ce167c731bb0b19:22
fungirunning that ^ now19:22
smcginnis++19:23
smcginnisCan we regenerate the signatures?19:23
smcginnisOr we just have a few that are missing?19:23
clarkbyou can make new signatures but it is harder to know what you are signing is actually legit19:24
clarkb(we may be able to cross check hashes between zuul logs and pypi contents to be fairly confident in that though)19:24
*** e0ne has joined #openstack-release19:24
*** e0ne has quit IRC19:25
*** e0ne has joined #openstack-release19:25
fungiyeah, i could manually re-sign those with the appropriate key if that was desirable, though the date/time wouldn't match when the release jobs ran19:25
smcginnisfungi, clarkb: What do you guys think is best? No signature, or signature with that mismatch?19:27
clarkbconsidering pip doesn't verify signatures I don't know that it is super important to correct19:28
clarkbunless distros are pulling from pypi and verifying from there (I think they pull fomr us instead)19:28
fungii think if we double-check checksums against build logs then signing isn't particularly disingenuous, though it will take a bit of time to do manually19:29
smcginnisI'm happy to defer that decision to the two of you. :)19:29
smcginnismugsie: I proposed some stable/train releases for anything that had commits. Not a release task, but thought it might help teams remember to do those stable releases.19:30
fungii could probably do it later this week, but have been a bit overwhelmed today and didn't get much of a weekend19:30
smcginnismugsie: There are a couple designate ones out there. My plan was to abandon the ones that didn't get any response by the end of this week.19:30
smcginnismugsie: So if those would be useful, would appreciate an ack on those.19:31
mugsiesmcginnis: thanks - will look this evening, or in the morning19:31
smcginnismugsie: Thanks.19:31
smcginnisfungi: Yeah, it looks like you've been busy. An answer of "yeah we could do that, but I don't have time" is perfectly fine to me.19:31
fungiwatching the octavia release job, looks like it just succeeded19:35
smcginnisGreat, thanks fungi19:36
cgoncalvesyep, https://tarballs.openstack.org/octavia/octavia-4.1.2.tar.gz19:36
cgoncalvesthanks for fixing it, folks!19:36
johnsommugsie Let me know if you need help with the designate stable stuff. My two stable patches are still sitting with +2 BTW19:37
fungii'll work out what we need to copy from pypi for the others, put together the checksum list for them, and work out the manual signing process for later in the week19:37
smcginnisoctavia 4.1.2 now listed on pypi too.19:37
mugsiejohnsom: will do, thanks. I voted on them right?19:37
johnsommugsie You did19:37
mugsiesmcginnis: I ack'd them - but I don;t have the voting ack anymore19:41
smcginnismugsie: Thanks!19:48
smcginnismugsie: You are still listed as the release liaison for designate.19:48
mugsieyeah - previously I had the "PTL Approval" vote button (I thought that was from that)19:49
smcginnisAh, there's a job that checks that and will vote on the patch once it sees PTL or liaison ack. So we're all set.19:50
*** e0ne has quit IRC20:06
*** evrardjp has quit IRC20:11
*** evrardjp has joined #openstack-release20:13
openstackgerritSean McGinnis proposed openstack/releases master: Release blazar-nova for victoria-1 milestone  https://review.opendev.org/73567720:28
openstackgerritSean McGinnis proposed openstack/releases master: Release ceilometermiddleware for victoria-1 milestone  https://review.opendev.org/73567820:29
openstackgerritSean McGinnis proposed openstack/releases master: Release heat-translator for victoria-1 milestone  https://review.opendev.org/73567920:30
openstackgerritSean McGinnis proposed openstack/releases master: Release keystoneauth for victoria-1 milestone  https://review.opendev.org/73568020:30
openstackgerritSean McGinnis proposed openstack/releases master: Release kuryr for victoria-1 milestone  https://review.opendev.org/73568220:31
*** e0ne has joined #openstack-release20:32
openstackgerritSean McGinnis proposed openstack/releases master: Release mistral-extra for victoria-1 milestone  https://review.opendev.org/73568320:33
openstackgerritSean McGinnis proposed openstack/releases master: Release mistral-lib for victoria-1 milestone  https://review.opendev.org/73568420:33
openstackgerritSean McGinnis proposed openstack/releases master: Release monasca-common for victoria-1 milestone  https://review.opendev.org/73568520:33
openstackgerritSean McGinnis proposed openstack/releases master: Release monasca-statsd for victoria-1 milestone  https://review.opendev.org/73568620:33
openstackgerritSean McGinnis proposed openstack/releases master: Release neutron-lib for victoria-1 milestone  https://review.opendev.org/73568720:34
openstackgerritSean McGinnis proposed openstack/releases master: Release openstacksdk for victoria-1 milestone  https://review.opendev.org/73568820:35
openstackgerritSean McGinnis proposed openstack/releases master: Release os-ken for victoria-1 milestone  https://review.opendev.org/73568920:35
openstackgerritSean McGinnis proposed openstack/releases master: Release os-vif for victoria-1 milestone  https://review.opendev.org/73569020:36
openstackgerritSean McGinnis proposed openstack/releases master: Release os-win for victoria-1 milestone  https://review.opendev.org/73569120:36
openstackgerritSean McGinnis proposed openstack/releases master: Release ovn-octavia-provider for victoria-1 milestone  https://review.opendev.org/73569220:37
openstackgerritSean McGinnis proposed openstack/releases master: Release pycadf for victoria-1 milestone  https://review.opendev.org/73569320:38
openstackgerritSean McGinnis proposed openstack/releases master: Release python-adjutantclient for victoria-1 milestone  https://review.opendev.org/73569420:38
openstackgerritSean McGinnis proposed openstack/releases master: Release python-aodhclient for victoria-1 milestone  https://review.opendev.org/73569520:40
openstackgerritSean McGinnis proposed openstack/releases master: Release python-barbicanclient for victoria-1 milestone  https://review.opendev.org/73569620:40
openstackgerritSean McGinnis proposed openstack/releases master: Release python-blazarclient for victoria-1 milestone  https://review.opendev.org/73569720:40
openstackgerritSean McGinnis proposed openstack/releases master: Release python-brick-cinderclient-ext for victoria-1 milestone  https://review.opendev.org/73569820:40
openstackgerritSean McGinnis proposed openstack/releases master: Release python-cinderclient for victoria-1 milestone  https://review.opendev.org/73569920:41
openstackgerritSean McGinnis proposed openstack/releases master: Release python-cyborgclient for victoria-1 milestone  https://review.opendev.org/73570220:42
openstackgerritSean McGinnis proposed openstack/releases master: Release python-freezerclient for victoria-1 milestone  https://review.opendev.org/73570320:42
openstackgerritSean McGinnis proposed openstack/releases master: Release python-glanceclient for victoria-1 milestone  https://review.opendev.org/73570420:43
openstackgerritSean McGinnis proposed openstack/releases master: Release python-heatclient for victoria-1 milestone  https://review.opendev.org/73570520:43
openstackgerritSean McGinnis proposed openstack/releases master: Release python-ironic-inspector-client for victoria-1 milestone  https://review.opendev.org/73570620:44
openstackgerritSean McGinnis proposed openstack/releases master: Release python-ironicclient for victoria-1 milestone  https://review.opendev.org/73570720:44
openstackgerritSean McGinnis proposed openstack/releases master: Release python-keystoneclient for victoria-1 milestone  https://review.opendev.org/73570920:45
openstackgerritSean McGinnis proposed openstack/releases master: Release python-manilaclient for victoria-1 milestone  https://review.opendev.org/73571020:45
openstackgerritSean McGinnis proposed openstack/releases master: Release python-masakariclient for victoria-1 milestone  https://review.opendev.org/73571120:46
openstackgerritSean McGinnis proposed openstack/releases master: Release python-mistralclient for victoria-1 milestone  https://review.opendev.org/73571220:46
openstackgerritSean McGinnis proposed openstack/releases master: Release python-monascaclient for victoria-1 milestone  https://review.opendev.org/73571320:46
openstackgerritSean McGinnis proposed openstack/releases master: Release python-muranoclient for victoria-1 milestone  https://review.opendev.org/73571420:47
openstackgerritSean McGinnis proposed openstack/releases master: Release python-neutronclient for victoria-1 milestone  https://review.opendev.org/73571520:47
openstackgerritSean McGinnis proposed openstack/releases master: Release python-novaclient for victoria-1 milestone  https://review.opendev.org/73571620:47
openstackgerritSean McGinnis proposed openstack/releases master: Release python-octaviaclient for victoria-1 milestone  https://review.opendev.org/73571720:48
openstackgerritSean McGinnis proposed openstack/releases master: Release python-pankoclient for victoria-1 milestone  https://review.opendev.org/73571820:48
openstackgerritSean McGinnis proposed openstack/releases master: Release python-qinlingclient for victoria-1 milestone  https://review.opendev.org/73571920:48
openstackgerritSean McGinnis proposed openstack/releases master: Release python-saharaclient for victoria-1 milestone  https://review.opendev.org/73572020:48
openstackgerritSean McGinnis proposed openstack/releases master: Release python-searchlightclient for victoria-1 milestone  https://review.opendev.org/73572120:49
openstackgerritSean McGinnis proposed openstack/releases master: Release python-senlinclient for victoria-1 milestone  https://review.opendev.org/73572220:49
openstackgerritSean McGinnis proposed openstack/releases master: Release python-solumclient for victoria-1 milestone  https://review.opendev.org/73572320:49
openstackgerritSean McGinnis proposed openstack/releases master: Release python-swiftclient for victoria-1 milestone  https://review.opendev.org/73572420:49
openstackgerritSean McGinnis proposed openstack/releases master: Release python-tackerclient for victoria-1 milestone  https://review.opendev.org/73572520:50
openstackgerritSean McGinnis proposed openstack/releases master: Release python-tricircleclient for victoria-1 milestone  https://review.opendev.org/73572620:50
openstackgerritSean McGinnis proposed openstack/releases master: Release python-vitrageclient for victoria-1 milestone  https://review.opendev.org/73572720:50
openstackgerritSean McGinnis proposed openstack/releases master: Release python-watcherclient for victoria-1 milestone  https://review.opendev.org/73572820:50
openstackgerritSean McGinnis proposed openstack/releases master: Release python-zaqarclient for victoria-1 milestone  https://review.opendev.org/73572920:51
openstackgerritSean McGinnis proposed openstack/releases master: Release python-zunclient for victoria-1 milestone  https://review.opendev.org/73573020:51
openstackgerritSean McGinnis proposed openstack/releases master: Release sushy-cli for victoria-1 milestone  https://review.opendev.org/73573120:51
openstackgerritSean McGinnis proposed openstack/releases master: Release sushy for victoria-1 milestone  https://review.opendev.org/73573220:52
openstackgerritSean McGinnis proposed openstack/releases master: Release tosca-parser for victoria-1 milestone  https://review.opendev.org/73573420:52
openstackgerritSean McGinnis proposed openstack/releases master: Release glance-store for victoria-1 milestone  https://review.opendev.org/73573520:56
*** slaweq has quit IRC21:01
*** slaweq has joined #openstack-release21:07
*** slaweq has quit IRC21:11
*** e0ne has quit IRC21:12
*** gmann is now known as gmann_afk21:39
*** tosky has quit IRC23:04

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!