| openstackgerrit | Clark Boylan proposed zuul/zuul master: Make paused status bar blue https://review.opendev.org/655588 | 00:02 |
|---|---|---|
| openstackgerrit | Clark Boylan proposed zuul/zuul master: Tiny cleanup in change panel js https://review.opendev.org/655589 | 00:02 |
| clarkb | I think that should make the paused bar blue and I think I found a valid cleanup too | 00:02 |
| *** Weifan has joined #openstack-infra | 00:03 | |
| fungi | ooh, neat, we get e-mail from the docker registry gc cron | 00:03 |
| fungi | i hadn't noticed these before just now | 00:03 |
| corvus | hopefully it didn't just delete an orphan layer from one of our jobs | 00:04 |
| *** jamesmcarthur has quit IRC | 00:04 | |
| *** dave-mccowan has quit IRC | 00:05 | |
| fungi | blobs /docker/registry/v2/blobs/sha256/78/78b43f9b8784fa68dd97dcf055ac03735d10f8a78ca52b5e0d174f1aacf984d5 and /docker/registry/v2/blobs/sha256/d9/d9b93e201cee5b09d7af25ba9ba5e420f54b0581e32cc91e967c1db5e2562b15 it says | 00:07 |
| fungi | if those mean anything to anybody | 00:07 |
| corvus | well the good news is that zuul-upload-image succeeded | 00:09 |
| corvus | the bad news is that zuul-upload-image succeeded so we don't get to find out what's up with the EOF thing. also tox-py36 failed so it didn't merge. | 00:10 |
| corvus | the good news is since it didn't merge, we have another convenient test change | 00:10 |
| corvus | i'll re-enqueue both again | 00:10 |
| *** slaweq has joined #openstack-infra | 00:11 | |
| *** diablo_rojo has quit IRC | 00:21 | |
| *** dikonoor has joined #openstack-infra | 00:21 | |
| *** slaweq has quit IRC | 00:24 | |
| *** gyee has quit IRC | 00:24 | |
| *** Goneri has quit IRC | 00:30 | |
| *** jamesmcarthur has joined #openstack-infra | 00:33 | |
| mriedem | i just wanted to say, we'll probably have a working multi-cell ci job for nova by tomorrow https://review.opendev.org/#/c/655222/ and it was surprisingly easy due to zuulv3 and the zuulv3-ification of the devstack/tempest jobs by the QA team (andreaf/gmann/others) | 00:35 |
| mriedem | sdague said a couple of years ago when talking about enabling a multi-cell job with devstack-gate something along the lines of "i think we've reached our limits for this kind of thing with d-g and will be needing to ansible this all up to be sane" | 00:36 |
| *** igordc has quit IRC | 00:40 | |
| *** mattw4 has quit IRC | 00:42 | |
| *** yamamoto has joined #openstack-infra | 00:46 | |
| *** yamamoto has quit IRC | 00:47 | |
| *** yamamoto has joined #openstack-infra | 00:48 | |
| *** rlandy|bbl is now known as rlandy | 00:48 | |
| *** michael-beaver has quit IRC | 00:49 | |
| *** jamesmcarthur has quit IRC | 00:50 | |
| *** Weifan has quit IRC | 00:51 | |
| *** jamesmcarthur has joined #openstack-infra | 00:52 | |
| *** jamesmcarthur has quit IRC | 00:53 | |
| *** jamesmcarthur has joined #openstack-infra | 00:54 | |
| *** jamesmcarthur has quit IRC | 00:56 | |
| *** jamesmcarthur has joined #openstack-infra | 00:56 | |
| *** whoami-rajat has joined #openstack-infra | 01:02 | |
| *** ricolin has joined #openstack-infra | 01:05 | |
| *** diablo_rojo has joined #openstack-infra | 01:07 | |
| *** armax has quit IRC | 01:10 | |
| *** mriedem has quit IRC | 01:11 | |
| *** slaweq has joined #openstack-infra | 01:13 | |
| *** slaweq has quit IRC | 01:24 | |
| openstackgerrit | melanie witt proposed openstack/project-config master: Fix ceph failure rate grafana dashboard https://review.opendev.org/655591 | 01:32 |
| *** markvoelker has quit IRC | 01:34 | |
| *** ykarel|away has joined #openstack-infra | 01:37 | |
| *** smarcet has joined #openstack-infra | 01:37 | |
| *** diablo_rojo has quit IRC | 01:49 | |
| openstackgerrit | melanie witt proposed openstack/project-config master: Fix ceph failure rate grafana dashboard https://review.opendev.org/655591 | 01:50 |
| *** apetrich has quit IRC | 01:57 | |
| *** jamesmcarthur has quit IRC | 02:00 | |
| *** jamesmcarthur has joined #openstack-infra | 02:01 | |
| *** jamesmcarthur has quit IRC | 02:02 | |
| *** jamesmcarthur has joined #openstack-infra | 02:02 | |
| *** ykarel|away has quit IRC | 02:06 | |
| *** slaweq has joined #openstack-infra | 02:11 | |
| *** smarcet has quit IRC | 02:14 | |
| *** smarcet has joined #openstack-infra | 02:16 | |
| *** Emine has quit IRC | 02:16 | |
| *** slaweq has quit IRC | 02:24 | |
| *** dikonoor has quit IRC | 02:25 | |
| *** Weifan has joined #openstack-infra | 02:32 | |
| *** ykarel|away has joined #openstack-infra | 02:34 | |
| *** Weifan has quit IRC | 02:35 | |
| *** smarcet has quit IRC | 02:36 | |
| openstackgerrit | Trinh Nguyen proposed openstack/project-config master: Migrate telemetry to storyboard https://review.opendev.org/655600 | 02:38 |
| *** armax has joined #openstack-infra | 02:41 | |
| *** ricolin has quit IRC | 02:46 | |
| rm_work | hey, anyone know what the senlin channel is? | 02:51 |
| rm_work | oh wait i can look it up on the wiki prolly | 02:51 |
| rm_work | ahh, just #senlin | 02:51 |
| *** hongbin has joined #openstack-infra | 02:51 | |
| *** jamesmcarthur has quit IRC | 02:52 | |
| *** jamesmcarthur_ has joined #openstack-infra | 02:53 | |
| *** armax has quit IRC | 02:53 | |
| *** bgmccollum has quit IRC | 02:59 | |
| *** bgmccollum has joined #openstack-infra | 03:04 | |
| *** bhavikdbavishi has joined #openstack-infra | 03:05 | |
| *** smcginnis has quit IRC | 03:06 | |
| *** bgmccollum has quit IRC | 03:07 | |
| *** bhavikdbavishi1 has joined #openstack-infra | 03:08 | |
| *** bhavikdbavishi has quit IRC | 03:09 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 03:09 | |
| *** auristor has quit IRC | 03:11 | |
| *** dikonoor has joined #openstack-infra | 03:11 | |
| *** hongbin has quit IRC | 03:12 | |
| *** jamesmcarthur_ has quit IRC | 03:14 | |
| *** slaweq has joined #openstack-infra | 03:16 | |
| *** jamesmcarthur has joined #openstack-infra | 03:16 | |
| *** roman_g has quit IRC | 03:18 | |
| openstackgerrit | Jason Lee proposed opendev/storyboard master: WIP: Migrates Users to Blueprints, Writer Script WIP #2 https://review.opendev.org/654812 | 03:21 |
| *** bhavikdbavishi has quit IRC | 03:21 | |
| *** auristor has joined #openstack-infra | 03:24 | |
| *** slaweq has quit IRC | 03:24 | |
| *** ekultails has joined #openstack-infra | 03:28 | |
| *** ykarel|away is now known as ykarel | 03:30 | |
| *** ekultails has quit IRC | 03:35 | |
| *** markvoelker has joined #openstack-infra | 03:35 | |
| *** Nisha_Agarwal has joined #openstack-infra | 03:45 | |
| *** michael-beaver has joined #openstack-infra | 03:46 | |
| *** bhavikdbavishi has joined #openstack-infra | 03:46 | |
| *** bhavikdbavishi has quit IRC | 03:49 | |
| *** psachin has joined #openstack-infra | 03:52 | |
| *** udesale has joined #openstack-infra | 03:55 | |
| *** jamesmcarthur has quit IRC | 03:58 | |
| *** ricolin has joined #openstack-infra | 04:10 | |
| *** pcaruana has joined #openstack-infra | 04:11 | |
| *** slaweq has joined #openstack-infra | 04:13 | |
| *** ykarel is now known as ykarel|afk | 04:14 | |
| *** ykarel|afk has quit IRC | 04:19 | |
| *** slaweq has quit IRC | 04:24 | |
| *** ykarel|afk has joined #openstack-infra | 04:35 | |
| *** ykarel|afk is now known as ykarel | 04:36 | |
| *** sthussey has quit IRC | 04:42 | |
| *** pcaruana has quit IRC | 04:43 | |
| *** uberjay has quit IRC | 05:03 | |
| *** e0ne has joined #openstack-infra | 05:05 | |
| *** kukacz has quit IRC | 05:06 | |
| *** kukacz has joined #openstack-infra | 05:08 | |
| *** e0ne has quit IRC | 05:08 | |
| *** uberjay has joined #openstack-infra | 05:08 | |
| *** quiquell|off is now known as quiquell | 05:09 | |
| *** slaweq has joined #openstack-infra | 05:11 | |
| *** slaweq has quit IRC | 05:21 | |
| *** gagehugo has quit IRC | 05:23 | |
| *** slaweq has joined #openstack-infra | 05:27 | |
| *** gagehugo has joined #openstack-infra | 05:31 | |
| *** Nisha_Agarwal has quit IRC | 05:36 | |
| *** adriant has quit IRC | 05:38 | |
| *** adriant has joined #openstack-infra | 05:38 | |
| *** slaweq has quit IRC | 05:41 | |
| *** Nisha_ has joined #openstack-infra | 05:51 | |
| *** michael-beaver has quit IRC | 05:55 | |
| *** slaweq has joined #openstack-infra | 05:56 | |
| *** slaweq has quit IRC | 06:02 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/project-config master: Normalize projects.yaml https://review.opendev.org/655615 | 06:04 |
| *** igordc has joined #openstack-infra | 06:08 | |
| *** electrofelix has joined #openstack-infra | 06:10 | |
| *** slaweq has joined #openstack-infra | 06:11 | |
| *** ramishra has joined #openstack-infra | 06:14 | |
| *** slaweq has quit IRC | 06:16 | |
| *** pcaruana has joined #openstack-infra | 06:20 | |
| *** iurygregory has joined #openstack-infra | 06:34 | |
| *** yamamoto has quit IRC | 06:39 | |
| *** slaweq has joined #openstack-infra | 06:44 | |
| *** AJaeger has quit IRC | 06:47 | |
| *** quiquell is now known as quiquell|brb | 06:48 | |
| *** apetrich has joined #openstack-infra | 06:52 | |
| *** igordc has quit IRC | 06:53 | |
| *** ricolin has quit IRC | 06:53 | |
| *** evrardjp has quit IRC | 06:55 | |
| *** evrardjp has joined #openstack-infra | 06:55 | |
| *** AJaeger has joined #openstack-infra | 06:56 | |
| *** evrardjp has quit IRC | 06:58 | |
| *** evrardjp has joined #openstack-infra | 06:58 | |
| *** aaronsheffield has quit IRC | 07:02 | |
| *** amoralej|off is now known as amoralej | 07:04 | |
| *** ccamacho has joined #openstack-infra | 07:08 | |
| *** ccamacho has quit IRC | 07:09 | |
| *** ccamacho has joined #openstack-infra | 07:10 | |
| *** dpawlik has quit IRC | 07:14 | |
| *** yamamoto has joined #openstack-infra | 07:16 | |
| *** happyhemant has joined #openstack-infra | 07:18 | |
| openstackgerrit | Merged openstack/project-config master: Normalize projects.yaml https://review.opendev.org/655615 | 07:19 |
| amorin | hey all | 07:21 |
| AJaeger | morning amorin! We have some dns issues with ovh-bhs1 ;( | 07:23 |
| amorin | AJaeger: ok, can you tell me more? do you have any instance running affected, so I can checl? | 07:23 |
| amorin | check* | 07:23 |
| AJaeger | amorin: could you come back during US time and ask, please? WE disabled it for now... | 07:24 |
| *** zbr has joined #openstack-infra | 07:24 | |
| *** Nisha_ has quit IRC | 07:24 | |
| AJaeger | The issue we noticed is that git.openstack.org could not be resolved | 07:24 |
| amorin | AJaeger: no problem, I'll be there full day | 07:24 |
| amorin | ok in the meantime, I will do some checks | 07:24 |
| amorin | also, Shrews asked me to delete some instances from GRA1 | 07:24 |
| amorin | which were stuck in deleting or something like that | 07:25 |
| AJaeger | WE disabled last week Thursday, reenabled yesterday - and disabled again after this log: http://logs.openstack.org/69/655369/1/gate/kolla-build-ubuntu-source/509c989/job-output.txt.gz#_2019-04-24_18_43_31_074787 | 07:25 |
| amorin | nice, thnaks | 07:25 |
| AJaeger | infra-root, anybody around to work with amorin on OVH DNS problems? | 07:25 |
| amorin | do you have any idea of the /etc/resolv.conf file there? | 07:26 |
| AJaeger | no, sorry. fungi, mnaser and clarkb might be able to help better - but are in US/Canada timezones | 07:27 |
| *** ricolin has joined #openstack-infra | 07:28 | |
| amorin | ok no problem | 07:28 |
| *** yamamoto has quit IRC | 07:28 | |
| AJaeger | amorin: http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2019-04-18.log.html#t2019-04-18T22:18:02 was from Thursday | 07:29 |
| AJaeger | clarkb said "dig git.openstack.org @1.1.1.1 works most of the time but I managed to get it to hang" | 07:30 |
| AJaeger | amorin: thanks for looking into that - hope you had a good long weekend! | 07:30 |
| *** yamamoto has joined #openstack-infra | 07:31 | |
| *** slaweq has quit IRC | 07:31 | |
| *** slaweq has joined #openstack-infra | 07:32 | |
| amorin | AJaeger: ok | 07:32 |
| *** kjackal has joined #openstack-infra | 07:32 | |
| *** slaweq has quit IRC | 07:34 | |
| *** slaweq has joined #openstack-infra | 07:34 | |
| *** slaweq has quit IRC | 07:35 | |
| *** slaweq has joined #openstack-infra | 07:36 | |
| *** slaweq has quit IRC | 07:36 | |
| *** dpawlik has joined #openstack-infra | 07:36 | |
| *** slaweq has joined #openstack-infra | 07:37 | |
| *** slaweq has quit IRC | 07:38 | |
| *** bhavikdbavishi has joined #openstack-infra | 07:38 | |
| *** slaweq has joined #openstack-infra | 07:38 | |
| *** jpena|off is now known as jpena | 07:43 | |
| *** kopecmartin|off is now known as kopecmartin | 07:46 | |
| *** ykarel is now known as ykarel|lunch | 07:49 | |
| *** jpich has joined #openstack-infra | 07:55 | |
| *** rpittau|afk is now known as rpittau | 07:58 | |
| *** ramishra_ has joined #openstack-infra | 07:59 | |
| *** ramishra has quit IRC | 08:02 | |
| *** ramishra_ is now known as ramishra | 08:04 | |
| *** ralonsoh has joined #openstack-infra | 08:05 | |
| amorin | funny,I spwaned 20 instances, doing some dig git.openstack.org against 1.1.1.1 and some of them are failing like clarkb said, doing the same with 8.8.8.8, and all of them are working | 08:05 |
| *** quiquell|brb is now known as quiquell | 08:07 | |
| *** e0ne has joined #openstack-infra | 08:08 | |
| *** e0ne has quit IRC | 08:08 | |
| *** e0ne has joined #openstack-infra | 08:08 | |
| amorin | 1.0.0.1 is working better | 08:10 |
| amorin | cc AJaeger | 08:10 |
| *** zhangfei has joined #openstack-infra | 08:12 | |
| *** derekh has joined #openstack-infra | 08:14 | |
| *** roman_g has joined #openstack-infra | 08:14 | |
| *** zbr is now known as zbr|rover | 08:18 | |
| AJaeger | amorin: thanks. | 08:18 |
| AJaeger | Now I wonder whether we should change DNS server | 08:19 |
| *** hrw has joined #openstack-infra | 08:24 | |
| hrw | morning | 08:24 |
| hrw | Can http://mirror.london.linaro-london.openstack.org/debian repos get signed? Debian Buster refuses to use them | 08:27 |
| hrw | http://logs.openstack.org/59/557659/37/experimental/kolla-build-debian-source-arm64/15cf546/logs/build/000_FAILED_base.txt.gz | 08:27 |
| *** zigo has quit IRC | 08:27 | |
| *** gmann has quit IRC | 08:28 | |
| *** yamamoto has quit IRC | 08:29 | |
| *** yamamoto has joined #openstack-infra | 08:30 | |
| *** Lucas_Gray has joined #openstack-infra | 08:32 | |
| *** lucasagomes has joined #openstack-infra | 08:33 | |
| *** ykarel|lunch is now known as ykarel | 08:39 | |
| *** tkajinam has quit IRC | 08:54 | |
| *** rcernin has quit IRC | 08:58 | |
| *** tobias-urdin has quit IRC | 09:01 | |
| *** tobias-urdin has joined #openstack-infra | 09:17 | |
| *** dims has quit IRC | 09:20 | |
| AJaeger | hrw: See https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/configure-mirrors/templates/etc/apt/apt.conf.d/99unauthenticated.j2 - we disable the signing. | 09:21 |
| AJaeger | Why is that not working for Debian? | 09:22 |
| AJaeger | hrw: we don't want to sign our mirror that is a non-trivial task and not really needed | 09:22 |
| *** dims has joined #openstack-infra | 09:26 | |
| *** tobias-urdin has quit IRC | 09:27 | |
| *** Lucas_Gray has quit IRC | 09:29 | |
| *** yamamoto has quit IRC | 09:31 | |
| *** dims has quit IRC | 09:33 | |
| *** dtantsur|afk is now known as dtantsur | 09:33 | |
| *** dims has joined #openstack-infra | 09:34 | |
| *** tobias-urdin has joined #openstack-infra | 09:40 | |
| *** zbr|rover has quit IRC | 09:42 | |
| *** zbr has joined #openstack-infra | 09:43 | |
| *** dklyle has quit IRC | 09:50 | |
| *** dklyle has joined #openstack-infra | 09:50 | |
| *** kota_ has quit IRC | 09:51 | |
| *** kota_ has joined #openstack-infra | 09:52 | |
| *** yamamoto has joined #openstack-infra | 10:07 | |
| *** ccamacho has quit IRC | 10:11 | |
| *** yamamoto has quit IRC | 10:11 | |
| *** bhavikdbavishi has quit IRC | 10:20 | |
| *** yamamoto has joined #openstack-infra | 10:22 | |
| openstackgerrit | Will Szumski proposed openstack/pbr master: Fix white space handling in file names https://review.opendev.org/629161 | 10:22 |
| *** yamamoto has quit IRC | 10:35 | |
| *** yamamoto has joined #openstack-infra | 10:48 | |
| *** dpawlik has quit IRC | 10:52 | |
| *** jpena is now known as jpena|lunch | 10:55 | |
| hrw | AJaeger: ok | 11:05 |
| hrw | AJaeger: will check then what is going on there. thanks | 11:05 |
| *** aaronsheffield has joined #openstack-infra | 11:08 | |
| *** yamamoto has quit IRC | 11:09 | |
| *** Lucas_Gray has joined #openstack-infra | 11:10 | |
| hrw | AJaeger: Buster is not fine with that ;( | 11:12 |
| *** rfolco|ruck is now known as rfolco|ruck|doct | 11:12 | |
| hrw | AJaeger: 'Acquire::AllowInsecureRepositories' is new flag | 11:12 |
| *** yamamoto has joined #openstack-infra | 11:21 | |
| *** Lucas_Gray has quit IRC | 11:24 | |
| *** dpawlik has joined #openstack-infra | 11:25 | |
| *** zhangfei has quit IRC | 11:29 | |
| *** zbr has quit IRC | 11:31 | |
| *** zbr has joined #openstack-infra | 11:32 | |
| *** yamamoto has quit IRC | 11:32 | |
| *** udesale has quit IRC | 11:33 | |
| *** yamamoto has joined #openstack-infra | 11:36 | |
| *** bhavikdbavishi has joined #openstack-infra | 11:44 | |
| *** yamamoto has quit IRC | 11:46 | |
| *** yamamoto has joined #openstack-infra | 11:49 | |
| *** dpawlik has quit IRC | 11:52 | |
| *** zigo has joined #openstack-infra | 11:52 | |
| *** panda is now known as panda|lunch | 11:59 | |
| *** yamamoto has quit IRC | 12:00 | |
| *** rh-jelabarre has joined #openstack-infra | 12:03 | |
| *** fresta has quit IRC | 12:07 | |
| *** jcoufal has joined #openstack-infra | 12:12 | |
| *** markvoelker has quit IRC | 12:14 | |
| *** markvoelker has joined #openstack-infra | 12:15 | |
| *** dpawlik has joined #openstack-infra | 12:15 | |
| *** rlandy has joined #openstack-infra | 12:20 | |
| *** derekh has quit IRC | 12:23 | |
| *** tosky has joined #openstack-infra | 12:25 | |
| *** vabada has joined #openstack-infra | 12:28 | |
| *** kranthikirang has joined #openstack-infra | 12:29 | |
| *** altlogbot_1 has quit IRC | 12:32 | |
| *** yamamoto has joined #openstack-infra | 12:36 | |
| *** lseki has joined #openstack-infra | 12:37 | |
| *** yamamoto has quit IRC | 12:38 | |
| *** nicolasbock has joined #openstack-infra | 12:38 | |
| *** yamamoto has joined #openstack-infra | 12:38 | |
| *** altlogbot_3 has joined #openstack-infra | 12:38 | |
| fungi | also, not signing the indices on our mirrors helps dissuade folks from relying on them outside our ci system | 12:43 |
| fungi | we used to have problems with people pointing their *production* machines at our package mirrors and then freaking out when we would take them down for maintenance or rename/move them | 12:44 |
| *** derekh has joined #openstack-infra | 12:44 | |
| fungi | amorin: that sounds like basically what we were experiencing too, yes. i'm guessing some routing issue between bhs1 and those netblocks but really have no insight into that | 12:46 |
| fungi | also possible 8.8.8.8 and 1.1.1.1 have blocked access from there due to abuse, or are aggressively throttling because of the request volume they see from those address ranges | 12:47 |
| *** altlogbot_3 has quit IRC | 12:47 | |
| *** altlogbot_0 has joined #openstack-infra | 12:48 | |
| *** kgiusti has joined #openstack-infra | 12:54 | |
| *** udesale has joined #openstack-infra | 12:56 | |
| *** eharney has quit IRC | 13:00 | |
| amorin | fungi: 8.8.8.8 and 1.0.0.1 were working perfectly this morning while I was testing | 13:01 |
| amorin | only 1.1.1.1 was affected by those timeouts | 13:01 |
| amorin | I am also suspecting some abuse blocking system | 13:01 |
| fungi | amorin: oh, i misread when you said "doing the same with 8.8.8.8" and thought you meant it was failing the same as 1.1.1.1, but now i see you said they were all working | 13:01 |
| amorin | yup | 13:02 |
| fungi | so 1.1.1.1 is problematic from bhs1 but 8.8.8.8 is not, sounds like? | 13:02 |
| amorin | yes | 13:02 |
| amorin | is the DNS server a parameter on your side ? | 13:02 |
| amorin | based on region? | 13:02 |
| amorin | I mean, can it be updated easily, only for BHS1? | 13:03 |
| fungi | it's set consistently across all places we upload, modulo selecting ipv6 over ipv4 when usable | 13:03 |
| fungi | we could perhaps look into swapping 1.1.1.1 for 1.0.0.1 | 13:03 |
| fungi | clarkb: ^ if you end up being around this morning (i know you have travel to prepare for) | 13:04 |
| *** dikonoor has quit IRC | 13:06 | |
| clarkb | we switched off opendns for similar reloability issuesiirc. Moving to 1.0.0.1 should be fine | 13:06 |
| clarkb | need to update our images then the base job that cleans up the config | 13:06 |
| *** ykarel is now known as ykarel|mtg | 13:08 | |
| openstackgerrit | Dirk Mueller proposed opendev/system-config master: Add mirroring for Stein packages https://review.opendev.org/655686 | 13:09 |
| amorin | fungi: clarkb ok | 13:09 |
| *** smcginnis has joined #openstack-infra | 13:10 | |
| openstackgerrit | Jeremy Stanley proposed openstack/project-config master: Switch from 1.1.1.1 to 1.0.0.1 https://review.opendev.org/655687 | 13:16 |
| fungi | clarkb: amorin: AJaeger: ^ | 13:16 |
| fungi | part 1 | 13:17 |
| *** rfolco|ruck|doct is now known as rfolco|ruck | 13:18 | |
| openstackgerrit | Jeremy Stanley proposed opendev/base-jobs master: Switch from 1.1.1.1 to 1.0.0.1 https://review.opendev.org/655689 | 13:20 |
| fungi | and part 2 ^ which i'll wip awaiting image rebuilds/uploads | 13:20 |
| *** ricolin has quit IRC | 13:21 | |
| *** mriedem has joined #openstack-infra | 13:22 | |
| *** altlogbot_0 has quit IRC | 13:23 | |
| *** michael-beaver has joined #openstack-infra | 13:24 | |
| *** altlogbot_2 has joined #openstack-infra | 13:27 | |
| *** altlogbot_2 has quit IRC | 13:28 | |
| AJaeger | thanks, fungi | 13:30 |
| mordred | fungi: +2 on both | 13:30 |
| *** altlogbot_0 has joined #openstack-infra | 13:32 | |
| *** eharney has joined #openstack-infra | 13:41 | |
| *** altlogbot_0 has quit IRC | 13:42 | |
| openstackgerrit | Merged openstack/project-config master: Switch from 1.1.1.1 to 1.0.0.1 https://review.opendev.org/655687 | 13:47 |
| fungi | zuul scheduler memory utilization looks way better since yesterday's restart | 13:47 |
| fungi | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=64792&rra_id=all | 13:48 |
| tinwood | hello. So after the recent changes, the renamed x/nova-lxd project is no longer pushing to github.com/openstack/nova-lxd ... is it/should it be going somewhere else or do I need to config something? thanks | 13:52 |
| fungi | tinwood: it's updating on https://opendev.org/x/nova-lxd but we have a script which we can use to transfer https://github.com/openstack/nova-lxd to another github organization of your choosing and there's a zuul job you can add to push new commits (and branches and tags) to it as soon as they merge | 13:54 |
| fungi | the org transfer is only actually necessary if you want github to set up transparent redirects so that old urls and git remotes will get forwarded to the new location there | 13:55 |
| tinwood | fungi, is it "allowed" to still be mirrored at github.com/openstack/nova-lxd or does it need to be mirrored elsewhere? (i.e. not in the openstack org)? | 13:56 |
| *** altlogbot_2 has joined #openstack-infra | 13:56 | |
| fungi | tinwood: the openstack tc has decided that going forward the openstack namespace will only be used for official openstack deliverable repositories, so any unofficial repos there will need to be transferred, deleted or archived | 13:57 |
| fungi | tinwood: also, if you would like a custom namespace for it on opendev.org we can rename that independently of anything github-related, though we'll need to coordniate to do that after the ptg since i don't think any of us have time to do it in the next week-ish | 13:57 |
| tinwood | fungi, right, that does make sense why it stopped then. Thanks for that; I'll have a discussion in our charms group to work out where to put it. it might go to openstack-charmers instead, but I'm not sure yet. thanks very much for your help. Is there an example of the job I will (eventually) need to add? | 13:58 |
| clarkb | ara is using it, should be able to find it there | 14:01 |
| mordred | https://opendev.org/recordsansible/ara-web/src/branch/master/.zuul.yaml#L40-L50 | 14:01 |
| mordred | tinwood: ^^ | 14:01 |
| fungi | yeah, that's where i was hunting | 14:01 |
| clarkb | also dmsimard sent email to the openstack-discuss list about how to set it up | 14:01 |
| mordred | along with https://opendev.org/recordsansible/ara-web/src/branch/master/.zuul.yaml#L60-L62 | 14:01 |
| fungi | aha, ara itself didn't have it yet but ara-web does | 14:01 |
| mordred | yeah | 14:01 |
| tinwood | mordred, fungi that's brilliant; thanks. | 14:01 |
| *** panda|lunch is now known as panda | 14:02 | |
| tinwood | (also clarkb)! | 14:02 |
| fungi | tinwood: to be clear, official openstack projects will be using something like that job to replicate to github in the near future too, but for ease of our migration maintenance we left their gerrit replication in place in the meantime | 14:02 |
| tinwood | fungi, ok | 14:02 |
| fungi | level playing field and all | 14:02 |
| mordred | you're a level playing field | 14:03 |
| tinwood | :) | 14:03 |
| mordred | fungi, clarkb: how is that ara-web replication job working | 14:03 |
| mordred | ? | 14:04 |
| mordred | it references ara_git_mirror_credentials which I don't see defined in the ara-web repo | 14:04 |
| clarkb | fungi: re zuul memory I reverted two suspect changes we could live without (leaving two other suspect changes we probably need) If memory holds up we will need to dig into those two changes that were reverted to figure out what is goig on | 14:04 |
| mordred | clarkb: ++ | 14:04 |
| clarkb | mordred: I'm not sure. dmsimard should know | 14:05 |
| mordred | kk | 14:05 |
| *** dpawlik has quit IRC | 14:06 | |
| fungi | possible he hasn't finished adding them yet | 14:06 |
| mordred | good point | 14:06 |
| rpittau | hi mordred, sorry to bug you, just wondering why https://review.opendev.org/655445 was not approved as the other backports. Asking because we're currently blocked by x/pyghmi and that's the last fix :/ | 14:07 |
| mordred | rpittau: I don't know - I'm guessing it's just that corvus and dtroyer both hit +2 at the same time and so neither saw that they were second | 14:08 |
| mordred | clarkb: ^^ you have +A powers there | 14:08 |
| rpittau | oh..... | 14:08 |
| clarkb | I can take a look in a few | 14:09 |
| rpittau | thanks! | 14:09 |
| clarkb | but then Im doi g yardwork and running errands and packing and stuff | 14:09 |
| dtroyer | rpittau, clarkb: got it | 14:10 |
| mordred | yeah. it's not going to be the MOST productive couple of days | 14:10 |
| clarkb | thanks | 14:10 |
| rpittau | thanks all, much appreciated :) | 14:10 |
| *** Goneri has joined #openstack-infra | 14:14 | |
| *** e0ne has quit IRC | 14:16 | |
| *** yamamoto has quit IRC | 14:18 | |
| corvus | has anyone looked into the regitry failures? | 14:26 |
| *** gmann has joined #openstack-infra | 14:26 | |
| *** armax has joined #openstack-infra | 14:26 | |
| corvus | i'll start there | 14:28 |
| AJaeger | corvus: they fail randomly - I did a couple of rechecks, sometimes they passed and py35/py36 failed instead ;( | 14:28 |
| corvus | AJaeger: yeah, we put autoholds in place yesterday and they caught some nodes | 14:29 |
| corvus | so the next step is to identify if any of those are useful | 14:29 |
| AJaeger | corvus: cool! | 14:30 |
| fungi | the unit test failures looked like individual test timeouts, though i haven't yet had time to correlate them | 14:30 |
| AJaeger | and py36 is failing again on 654238 ;( | 14:31 |
| corvus | fungi: yeah, that's a known problem | 14:31 |
| fungi | argh | 14:31 |
| AJaeger | fungi, corvus, want to requeue again once everything is finished? Or how to move forward with these ? | 14:31 |
| corvus | things got a lot worse with the multi-ansible work | 14:32 |
| corvus | here's one node we caught: http://logs.openstack.org/38/654238/3/gate/zuul-upload-image/221aaa2/ara-report/result/9bc3c6c6-3df7-4bd5-a895-a32b36247448/ | 14:32 |
| corvus | that's a new error | 14:32 |
| fungi | AJaeger: i promoted 654238,3 so it'll just restart testing | 14:32 |
| *** jpena|lunch is now known as jpena | 14:33 | |
| fungi | ugly, but faster than alternatives (other than bypassing testing entirely) | 14:33 |
| AJaeger | fungi: thanks! | 14:33 |
| dmsimard | reading backlog | 14:36 |
| dmsimard | mordred, fungi, clarkb: according to zuul builds page, upload-git-mirror hasn't had the opportunity to run for ara-infra or ara-web yet | 14:38 |
| dmsimard | It is entirely in the realm of possibility that I have not set them up properly | 14:38 |
| dmsimard | ara works for sure, though | 14:38 |
| fungi | dmsimard: where is that job added to the ara repo? i couldn't find it | 14:39 |
| dmsimard | though I did notice something | 14:39 |
| dmsimard | when the opendev .gitreview patch was pushed, that didn't trigger the post pipeline | 14:39 |
| fungi | it wouldn't | 14:40 |
| fungi | we didn't push those through gerrit | 14:40 |
| fungi | that would have taken faaaaar too long | 14:40 |
| dmsimard | fungi: it's in the feature/1.0 branch: https://opendev.org/recordsansible/ara/src/branch/feature/1.0/.zuul.d/zuul.yaml#L115-L173 | 14:40 |
| fungi | oh! got it. i was looking in master | 14:41 |
| dmsimard | master has been mostly frozen but I should probably make sure the job is there as well to prevent issues in the future | 14:41 |
| fungi | tinwood: ^ see dmsimard's link for a better working example | 14:41 |
| tinwood | fungi, thanks | 14:42 |
| corvus | fungi, mordred, AJaeger, clarkb: i'm going to start an etherpad to keep notes on the registry issue debugging | 14:43 |
| corvus | https://etherpad.openstack.org/p/akSFrd8Oh7 | 14:43 |
| fungi | the ssh_key secret would be encrypted to your zuul project per https://docs.openstack.org/infra/manual/zuulv3.html#secret-variables | 14:44 |
| fungi | tinwood: ^ | 14:44 |
| fungi | thanks corvus! | 14:44 |
| tinwood | fungi, ah, I see. to be able to push to a nominated repo. | 14:46 |
| fungi | yep | 14:46 |
| *** electrofelix has quit IRC | 14:48 | |
| *** electrofelix has joined #openstack-infra | 14:48 | |
| *** electrofelix has quit IRC | 14:49 | |
| openstackgerrit | Ben Nemec proposed openstack/pbr master: Stop using pbr sphinx integration https://review.opendev.org/655565 | 14:51 |
| corvus | mordred: we might have logs from the registry that you could correlated with that error | 14:52 |
| mordred | corvus: nod | 14:52 |
| dtroyer | I am trying to figure out what is causing a POST_FAILURE on a devstack job where devstack runs to completion: http://logs.openstack.org/42/655542/1/gate/flock-devstack-config/b5401da/ (from review https://review.opendev.org/#/c/655542/). This has happened a couple of times this week and I'm not sure I know where to look. | 14:54 |
| corvus | mordred: the second error is the one we kept seeing yesterday | 14:55 |
| corvus | mordred: the third is the same as the first -- the bad blob | 14:59 |
| *** iurygregory has quit IRC | 15:00 | |
| *** _erlon_ has joined #openstack-infra | 15:00 | |
| fungi | btw, zuul-upload-image hit another post_failure on 654238,3: http://logs.openstack.org/38/654238/3/gate/zuul-upload-image/bbbabdf/ | 15:01 |
| fungi | re-promoting now | 15:01 |
| *** sthussey has joined #openstack-infra | 15:03 | |
| *** Goneri has quit IRC | 15:05 | |
| dmsimard | fwiw, just confirmed that upload-git-mirror still works for ara, will double check for ara-web and ara-infra | 15:06 |
| *** ykarel|mtg is now known as ykarel | 15:06 | |
| dmsimard | mordred: so the secret for ara_git_mirror_credentials should be redefined in both ara-web and ara-infra then ? | 15:07 |
| fungi | dmsimard: yes | 15:07 |
| fungi | dmsimard: if you want, jroll just sent a message to the openstack-discuss ml which mentions that job if you're keen to follow up with an example and indication that it's already working: http://lists.openstack.org/pipermail/openstack-discuss/2019-April/005629.html | 15:07 |
| fungi | dtroyer: i think it's this: http://logs.openstack.org/42/655542/1/gate/flock-devstack-config/b5401da/ara-report/result/aeee74d0-e49f-4f36-9861-99711650c2cf/ "Timeout (32s) waiting for privilege escalation prompt:" | 15:11 |
| dtroyer | fungi: ok, thanks. not that there is anything I can do about that (is there??) but I'm getting frustrated devs who just want to make jobs non-voting because of that… | 15:12 |
| fungi | here's where it appears in the console stream: http://logs.openstack.org/42/655542/1/gate/flock-devstack-config/b5401da/job-output.txt.gz#_2019-04-24_21_07_42_508531 | 15:13 |
| *** quiquell is now known as quiquell|off | 15:14 | |
| fungi | https://github.com/ansible/ansible/issues/14426 might be relevant | 15:15 |
| fungi | seeing some suggestions that it could be related to systemd deciding that services started on-demand by the login process are possibly broken and throttling them | 15:18 |
| *** Goneri has joined #openstack-infra | 15:18 | |
| fungi | dmsimard: maybe you've seen this behavior before? | 15:19 |
| *** igordc has joined #openstack-infra | 15:19 | |
| fungi | unfortunately it's resulting in a failure to collect other logs from the problem node, so can't go digging in syslog to find out | 15:20 |
| *** jamesmcarthur has joined #openstack-infra | 15:20 | |
| fungi | could be that iterating over a massive list of files with that particular action is a bad idea and we should find a more efficient solution to that | 15:21 |
| *** lpetrut has joined #openstack-infra | 15:25 | |
| *** jamesmcarthur has quit IRC | 15:32 | |
| *** jamesmcarthur has joined #openstack-infra | 15:34 | |
| fungi | looks like if zuul-quick-start succeeds for 654238,3 we'll need to reenqueue 55491,2 (it's currently hit another unit test timeout) | 15:35 |
| *** yamamoto has joined #openstack-infra | 15:35 | |
| *** liuyulong has quit IRC | 15:37 | |
| fungi | er, reenqueue 655491,2 i mean | 15:40 |
| fungi | though if it hasn't finished testing by the time the one ahead of it merges, i'll just promote it so we don't have to wait for it to report first | 15:40 |
| *** fresta has joined #openstack-infra | 15:42 | |
| *** yamamoto has quit IRC | 15:42 | |
| *** ramishra has quit IRC | 15:42 | |
| *** efried has quit IRC | 15:43 | |
| *** efried has joined #openstack-infra | 15:43 | |
| pabelanger | fungi: it looks like it might have passed | 15:45 |
| fungi | yeah, i'm watching the console stream for the registry upload | 15:45 |
| fungi | it's just wrapping up now | 15:45 |
| openstackgerrit | Merged zuul/zuul master: Update references for opendev https://review.opendev.org/654238 | 15:45 |
| fungi | there we go | 15:45 |
| pabelanger | need to look at zuul_console to see why we don't output LOOP tasks | 15:45 |
| fungi | restarting builds for 655491,2 now | 15:45 |
| fungi | oh, looks like it ended up kicked out on a post_failure so need to reenqueue it | 15:47 |
| fungi | and done | 15:47 |
| AJaeger | \o/ | 15:48 |
| openstackgerrit | sebastian marcet proposed osf/openstackid-resources master: Updated validation for slides https://review.opendev.org/655724 | 15:53 |
| openstackgerrit | Merged osf/openstackid-resources master: Updated validation for slides https://review.opendev.org/655724 | 15:54 |
| *** jcoufal has quit IRC | 15:56 | |
| *** ykarel is now known as ykarel|away | 15:57 | |
| *** weshay|rover is now known as weshay | 15:59 | |
| openstackgerrit | sebastian marcet proposed osf/openstackid-resources master: Updated validation for slides https://review.opendev.org/655725 | 16:00 |
| *** amoralej is now known as amoralej|off | 16:00 | |
| openstackgerrit | Merged osf/openstackid-resources master: Updated validation for slides https://review.opendev.org/655725 | 16:01 |
| *** rpittau is now known as rpittau|afk | 16:05 | |
| *** ykarel|away has quit IRC | 16:08 | |
| AJaeger | config-core, could you review https://review.opendev.org/655591 and https://review.opendev.org/#/c/651668/ , please? | 16:08 |
| imacdonn | hi infra ... I need to ask a stupid question. Apologies in advance, but I've timed out searching for the answer.... | 16:11 |
| *** kjackal has quit IRC | 16:11 | |
| imacdonn | so I have this cinder third party CI system .. it uses devstack master, but sets CINDER_BRANCH (in local.conf) to ${GERRIT_REFSPEC} for the change to be tested | 16:11 |
| imacdonn | this no longer works, since the opendev migration (I guess?): | 16:12 |
| imacdonn | + functions-common:git_timed:626 : timeout -s SIGINT 0 git fetch https://git.openstack.org/openstack/cinder.git refs/changes/41/627941/38 | 16:12 |
| imacdonn | fatal: Couldn't find remote ref refs/changes/41/627941/38 | 16:12 |
| fungi | imacdonn: yes, for the moment refs/changes isn't being redirected to opendev.org (where git.openstack.org is redirected to) | 16:13 |
| fungi | you would need to fetch refs/changes from https://review.opendev.org/p/openstack/cinder.git for now | 16:13 |
| imacdonn | fungi: OK .... but, I'm not doing that; devstack is .... | 16:14 |
| fungi | imacdonn: oh, there are recent fixes to devstack for that | 16:14 |
| fungi | some for stable branches may have just merged this morning | 16:14 |
| fungi | i think the fixes were for that, at least | 16:14 |
| imacdonn | I'm using devstack master, though | 16:15 |
| *** dklyle has quit IRC | 16:15 | |
| *** dklyle has joined #openstack-infra | 16:15 | |
| *** adriancz has quit IRC | 16:15 | |
| mordred | we have a WIP patch to fix the refs/changes/ thing in the gitea layer | 16:17 |
| openstackgerrit | sebastian marcet proposed osf/openstackid-resources master: Updated validation for slides https://review.opendev.org/655733 | 16:17 |
| clarkb | also CI systems are expected to set error on clone iirc | 16:17 |
| clarkb | but maybe that is more a thing we do and less uni ersal | 16:18 |
| *** jamesmcarthur has quit IRC | 16:18 | |
| *** sshnaidm|off has quit IRC | 16:18 | |
| *** pcaruana has quit IRC | 16:19 | |
| mordred | corvus: I added some notes to the etherpad - they may be largely useless | 16:19 |
| *** quiquell|off has quit IRC | 16:20 | |
| imacdonn | mordred: is it completely broken until then? i.e. no way to get refs/changes in devstack? | 16:20 |
| fungi | ahh, got it, so devstack assumes CINDER_BRANCH will be a ref available on the origin remote, but some third-party ci systems pass the change ref as CINDER_BRANCH (or similar)? | 16:20 |
| corvus | gitea update: the git clone redirect patch has merged upstream and should be included in 1.9.0. i've opened a PR for refs/changes: https://github.com/go-gitea/gitea/pull/6758 | 16:20 |
| mordred | corvus: but - tl;dr - error 1 seems to be "bad size" - and then 2 is an EOF from the buildset registry - this makes me think perhaps both problems are actually somehow linked to a flaky buildset registry and the intermediate registry is actually fine now | 16:20 |
| *** Lucas_Gray has joined #openstack-infra | 16:20 | |
| mordred | corvus: should we go ahead and put yoru refs/changes patch into jeblair/1.8.0-opendev ? | 16:20 |
| openstackgerrit | Merged osf/openstackid-resources master: Updated validation for slides https://review.opendev.org/655733 | 16:20 |
| mordred | corvus: if that very-unsubstantiated-hypothesis holds, perhaps a retry: on the skopeo task would be worthwhile? | 16:21 |
| corvus | mordred: it takes 2 hours to replicate nova into gitea, even with my change. so i think that will take some planning. | 16:21 |
| mordred | corvus: nod | 16:21 |
| *** igordc has quit IRC | 16:21 | |
| corvus | mordred: yeah, maybe our socat thing isn't working? | 16:21 |
| corvus | that seems surprising to me... it's so simple | 16:21 |
| *** lucasagomes has quit IRC | 16:22 | |
| mordred | corvus: oh - right - because in this case localhost is actually socat | 16:22 |
| fungi | restarted gate testing for 655491,2 again due to yet more timeoutexceptions in unit tests | 16:22 |
| mordred | corvus: but yeah - it could be read-issues over socat rather leading to EOF/data-validation | 16:22 |
| imacdonn | fungi: I wonder if I can force devstack by setting GIT_BASE <insert=chin-scratching-emoji> | 16:23 |
| corvus | mordred: these are held nodes -- we could hop on an executor, run socat, and do a bunch of skopeo copy to local disk operations | 16:23 |
| mordred | corvus: on the other, out of curiosity - is that replicating all of nova AND refs/changes/* - or replicating refs/changes into gitea with nova-minus-refs/changes already there? | 16:23 |
| corvus | did that make sense or should i use more words? | 16:23 |
| mordred | corvus: I think that made sense | 16:23 |
| corvus | mordred: 'git push --mirror' -- everything | 16:23 |
| fungi | imacdonn: it may be worth bringing up with the devstack maintainers in #openstack-qa | 16:23 |
| *** igordc has joined #openstack-infra | 16:23 | |
| corvus | mordred: i believe the time is related to the number of refs, so i don't expect substantial improvement | 16:24 |
| mordred | corvus: nod. I mean, I doubt it'll be significantly less - yeah | 16:24 |
| imacdonn | fungi: OK, thanks | 16:24 |
| fungi | imacdonn: i would personally expect CINDER_BRANCH to be separate from the change you're testing, and instead have your ci system attempt to merge the change ref into that branch before starting tests | 16:24 |
| hrw | fungi: I am fine with whatever policies infra has as it is not my job to change them but to obey them. (signed repos thing). Pointing out just a fact that current way used in zuul jobs applies for Debian:stretch (stable) and not good for Debian:buster (testing, soon stable) | 16:24 |
| corvus | mordred: because git-receive-pack still processes each ref and calls the gitea post-receive hook, even though with my change that hook now noops. | 16:24 |
| corvus | that still takes (a very small amount of) time | 16:24 |
| fungi | hrw: yep, understood that part, and thanks for bringing it to our attention | 16:25 |
| hrw | fungi: thanks | 16:25 |
| imacdonn | fungi: it's basically following https://wiki.openstack.org/wiki/Cinder/tested-3rdParty-drivers - at least as far as the CINDER_BRANCH part | 16:26 |
| fungi | a very small amount of time multiplied by a very large amount of git objects | 16:26 |
| imacdonn | fungi: CINDER_REPO might be interesting ... I'm not currently setting that | 16:27 |
| corvus | mordred: you want to pick one of those nodes, grab it's ip and which executor originally was talking to it (we may as well make the test as close as possible) and set up a screen there? | 16:27 |
| fungi | imacdonn: yep, it's possible that document relied on convenience of assuming the origin also had a complete copy of unreviewed changes in gerrit | 16:27 |
| mordred | corvus: yes - but I need to fix my coffee situation first - I will do that in about 5 minutes | 16:28 |
| *** jpich has quit IRC | 16:28 | |
| corvus | k | 16:28 |
| fungi | i need to go find lunch, but if 655491,2 finally manages to merge i should be available again soonish to do executor restarts | 16:29 |
| * fungi disappears for a bit | 16:29 | |
| hrw | fungi, AJaeger: can I get pinged once that Debian apt thing gets implemented? will then resume work on adding arm64/debian ci job to kolla project. | 16:30 |
| *** jamesmcarthur has joined #openstack-infra | 16:31 | |
| corvus | mordred: i have a screen session running as root on ze01 | 16:31 |
| *** ykarel|away has joined #openstack-infra | 16:32 | |
| corvus | mordred: i've chosen the last build/host in the list | 16:32 |
| mordred | corvus: I am on it now - but still not 100% back | 16:33 |
| corvus | mordred: ack. i've got 2 windows. the first is now running socat | 16:34 |
| corvus | i copied this command: http://logs.openstack.org/91/655491/1/gate/zuul-upload-image/fbff568/ara-report/result/679c7f1d-676b-4610-befd-38553d43d192/ | 16:34 |
| corvus | getting ready to run skopeo in the second | 16:34 |
| mordred | k. I'm back - what's the switch-window command again? | 16:35 |
| corvus | mordred: c-a n | 16:35 |
| mordred | c-a something yeah? | 16:35 |
| mordred | awesome | 16:35 |
| openstackgerrit | Merged openstack/project-config master: Fix ceph failure rate grafana dashboard https://review.opendev.org/655591 | 16:35 |
| corvus | sweet! instafail | 16:35 |
| openstackgerrit | Fatih Degirmenci proposed opendev/glean master: Sync when writing the file https://review.opendev.org/652238 | 16:35 |
| mordred | yay! | 16:36 |
| corvus | 2019/04/25 16:35:50 socat[13207] E connect(5, AF=10 [2607:ff68:0100:0054:f816:3eff:fea9:51df]:5000, 28): Permission denied | 16:36 |
| mordred | 2019/04/25 16:35:50 socat[13208] E connect(5, AF=10 [2607:ff68:0100:0054:f816:3eff:fea9:51df]:5000,| | 16:37 |
| mordred | 28): Permission denied | 16:37 |
| mordred | :) | 16:37 |
| mordred | your paste was more fully formed than mine | 16:37 |
| *** zbr is now known as zbr|rover | 16:37 | |
| corvus | is it due to the firewall? if so, how did this ever work? | 16:37 |
| mordred | maybe? I've got this: | 16:38 |
| mordred | https://superuser.com/questions/282976/ssh-over-ipv6-got-permission-denied-error | 16:39 |
| *** jcoufal has joined #openstack-infra | 16:39 | |
| mordred | which isn't the same thing - but is similarly an ipv6 permission error with a socat reference in the answer | 16:39 |
| mordred | https://unix.stackexchange.com/questions/413278/socat-gives-error-read6-0xf97acc0-8192-permission-denied has no answers | 16:39 |
| mordred | well - that telnet certainly didn't work | 16:40 |
| *** pcaruana has joined #openstack-infra | 16:40 | |
| corvus | how about i try opening up the port in iptables | 16:40 |
| mordred | ++ | 16:40 |
| corvus | is the buildset registry using host networking? | 16:41 |
| corvus | looks like no: | 16:41 |
| corvus | ports: | 16:41 |
| corvus | - "5001:5000" | 16:41 |
| mordred | is there a reason we're not using host networking | 16:42 |
| mordred | ? | 16:42 |
| corvus | so.... | 16:42 |
| mordred | otherwise, do we need to tell docker to "expose" that port? | 16:42 |
| * mordred goes tso read | 16:42 | |
| corvus | mordred: yeah sorry that's what i meant by that paste -- we are telling docker to expose that port | 16:42 |
| mordred | ah - gotcha | 16:43 |
| corvus | mordred: https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/run-buildset-registry/tasks/main.yaml#L86 | 16:43 |
| corvus | i see iptables rules for docker regarding port 5000; i do not see ip6tables rules for docker and port 5000 | 16:43 |
| corvus | this is a limestone host so only has ip6 | 16:43 |
| mordred | well - that would certainly explain that | 16:44 |
| mordred | I mean - lack of ip6 rules would certainly explain connection refused | 16:44 |
| corvus | perhaps this only works on ipv4 because of that? | 16:44 |
| *** gyee has joined #openstack-infra | 16:44 | |
| corvus | and we didn't notice this because it only worked on ipv4 because of the *other* docker problem | 16:44 |
| mordred | yeah | 16:44 |
| mordred | so we maybe need to also open 5000 for ip6 outside of docker - assuming it's not going to do *anything* with ip6 rules | 16:45 |
| corvus | yeah, let's start by opening this port, but i think there's a good chance that if docker isn't doing ipv6 port forwarding rules, it still won't work. | 16:46 |
| corvus | but lets get to that point, and then see if the solution is something like "tell docker to ipv6 too" or "use host networking" | 16:46 |
| mordred | if we switch to host networking - does docker get out of the business of trying to do iptables rules? | 16:46 |
| mordred | yeah | 16:46 |
| mordred | ++ | 16:46 |
| * corvus works on iptables incantations | 16:47 | |
| mordred | corvus: honestly, since it's port 5000 - host networking seems like it might be better anyway - less pieces in the middle of the chain | 16:47 |
| mordred | but - I agree, one step at a time | 16:48 |
| corvus | mordred: hrm, that's progress on the iptables front | 16:49 |
| mordred | yes, I agree | 16:50 |
| *** udesale has quit IRC | 16:50 | |
| corvus | i added rules to v4 and v6; let me double check it was the v6 one that made the difference | 16:50 |
| *** jpena is now known as jpena|off | 16:50 | |
| rpioso | After a change has been merged, can comments be sent to the owner and reviewers? Gerrit is permitting me to create draft comments. | 16:51 |
| *** igordc has quit IRC | 16:52 | |
| corvus | mordred: okay, i think we confirmed that the EOF issue is that ip6tables rule | 16:52 |
| *** Goneri has quit IRC | 16:52 | |
| mordred | \o/ | 16:52 |
| mordred | corvus: --src-tls-verify=false will disable tls verification | 16:52 |
| imacdonn | fungi: looks like CINDER_REPO=https://review.opendev.org/p/openstack/cinder.git made it all better ... thanks for helping me find my way ;) | 16:52 |
| *** kopecmartin is now known as kopecmartin|off | 16:54 | |
| mordred | corvus: yes, that | 16:54 |
| corvus | hrm, i wonder what the unauthorized is about | 16:55 |
| *** mattw4 has joined #openstack-infra | 16:55 | |
| mordred | yeah | 16:55 |
| corvus | the buildset registry is reporting an auth error | 16:55 |
| corvus | ha | 16:56 |
| corvus | the password serious had a period in it | 16:56 |
| mordred | hahah | 16:56 |
| mordred | ok - so that seemed to work much betterer | 16:57 |
| corvus | yeah | 16:57 |
| corvus | mordred: i'm really curious about the bad blob error | 16:57 |
| mordred | unfortunately it's not widely applicable enough to support my earlier hypothesis that the other error is from the same cause | 16:57 |
| mordred | yeah | 16:57 |
| corvus | mordred: that happened on limestone too, which should be in the same situation | 16:57 |
| mordred | best I can tell it's length validation related | 16:57 |
| corvus | but it also happened on gra1 which should be ipv4 i think? | 16:57 |
| mordred | corvus: yeah - but also on ovh-gra1 | 16:57 |
| mordred | yeah | 16:57 |
| corvus | it's probably worth repeating this on one or both of those; which should we start with? | 16:58 |
| mordred | corvus: I think ovh-gra1? wait ... how did we get a bad blob error on limestone at all? | 16:58 |
| mordred | we shouldnm't have been able to connect | 16:58 |
| corvus | mordred: yeah, that's why i think that one is interesting too | 16:58 |
| corvus | i think maybe i want to start there | 16:59 |
| corvus | since it's more like changing one variable than two | 16:59 |
| corvus | (there == limestone) | 16:59 |
| mordred | yeah | 16:59 |
| corvus | that job ran on ze06; i'm going to start a new screen session there | 17:00 |
| corvus | oops, wrong job | 17:01 |
| corvus | make that ze03 | 17:01 |
| corvus | sorry, i may have exited a screen, but i've got one up now on ze03 | 17:02 |
| mordred | I'm in | 17:02 |
| corvus | mordred: oh i think there's an error in the etherpad | 17:03 |
| mordred | yeah? | 17:03 |
| corvus | mordred: i think that first job ran in rax-org | 17:03 |
| corvus | ord | 17:03 |
| *** sshnaidm has joined #openstack-infra | 17:03 | |
| corvus | mordred: double check me on that | 17:03 |
| *** sshnaidm has quit IRC | 17:03 | |
| mordred | corvus: yes. I agree weith you | 17:04 |
| mordred | I'm not sure what I checked originally | 17:04 |
| corvus | mordred: that's great news! it means that ipv6 errors with EOF and ipv4 errors with validation | 17:04 |
| mordred | ++ | 17:04 |
| corvus | let's just continue with this host then | 17:04 |
| corvus | i'm in the next window now | 17:05 |
| mordred | yup | 17:05 |
| corvus | okay, i'm weirded out that iptables doesn't matter here but i'll just file that away | 17:05 |
| mordred | :) | 17:06 |
| corvus | i'm logging into the host to look at the registry logs | 17:06 |
| mordred | well - doesn't docker properly set iptables for ipv4? | 17:06 |
| corvus | mordred: yeah, but i'm surprised it does so in a way that bypasses our ingress filters... | 17:06 |
| corvus | very helpful of it | 17:06 |
| corvus | okay, so the registry saw my telnet as a failed tls handshake, that looks good | 17:07 |
| corvus | i'll try skopeo now | 17:07 |
| mordred | ++ | 17:07 |
| corvus | that's annoying | 17:08 |
| mordred | yeah | 17:08 |
| *** derekh has quit IRC | 17:08 | |
| mordred | corvus: so - the failed job was copying to the intermediate registry | 17:09 |
| corvus | oh | 17:09 |
| mordred | skopeo --insecure-policy copy docker://127.0.0.1:38143/zuul/zuul-executor:latest docker://insecure-ci-registry.opendev.org:5000/zuul/zuul-executor:221aaa239c584e3885f3a2e3371a7ee2_latest | 17:09 |
| corvus | derp | 17:09 |
| mordred | corvus: also - pull from zuul-executor | 17:10 |
| corvus | mordred: oh thx | 17:11 |
| mordred | ye | 17:11 |
| mordred | yes | 17:11 |
| *** Lucas_Gray has quit IRC | 17:12 | |
| corvus | grumble | 17:12 |
| mordred | yeah | 17:12 |
| corvus | that's probably going to keep working | 17:13 |
| mordred | well - the copy is an idempotent operation ... maybe there's just occasional network derps and a retries would help it? | 17:13 |
| corvus | yeah. i agree, retries:3 or however that's spelled might be called for here | 17:13 |
| mordred | since it DID skip the ones that worked before and then we watched it transfer the one that broke | 17:13 |
| mordred | corvus: yeah | 17:13 |
| *** dtantsur is now known as dtantsur|afk | 17:14 | |
| corvus | i'm looking at the buildset registry logs from the build | 17:14 |
| mordred | corvus: I'll make a retries: patch so we've got it | 17:15 |
| corvus | mordred: maybe do that for both skopeos -- push and pull | 17:15 |
| mordred | yeah | 17:16 |
| *** happyhemant has quit IRC | 17:17 | |
| corvus | nothing's jumping out at me in the buildset registry log (though there are a bunch of expected errors (blob not found) and something could easily be hiding among them) | 17:18 |
| *** ricolin has joined #openstack-infra | 17:20 | |
| *** ralonsoh has quit IRC | 17:20 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Add retries to skopeo copy operations https://review.opendev.org/655739 | 17:20 |
| mordred | corvus: I *think* that's the right way to do that in a loop context | 17:21 |
| mordred | dmsimard: ^^ you're good with the ansibles, right? | 17:21 |
| mordred | oh - I guess I can do is success | 17:21 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Add retries to skopeo copy operations https://review.opendev.org/655739 | 17:22 |
| *** Weifan has joined #openstack-infra | 17:24 | |
| openstackgerrit | James E. Blair proposed openstack/project-config master: Allow incoming access to port 5000 on test nodes https://review.opendev.org/655741 | 17:24 |
| corvus | mordred: for the ipv6 issue... one way to address that would be to open the port on all our images/build nodes ^ | 17:24 |
| corvus | mordred: or we could do it in the opendev buildset registry job | 17:25 |
| corvus | since that's opendev specific, i think we can encode information like that in it (we can't put it in the role though) | 17:25 |
| corvus | let me try that change to see how it looks | 17:26 |
| *** psachin has quit IRC | 17:28 | |
| mordred | corvus: yeah. I think in the opendev buildset registry job | 17:28 |
| *** Lucas_Gray has joined #openstack-infra | 17:28 | |
| openstackgerrit | James E. Blair proposed opendev/base-jobs master: Open the firewall port for the buildset registry https://review.opendev.org/655744 | 17:32 |
| corvus | mordred: i think ^ should do it | 17:33 |
| mordred | corvus: ++ | 17:33 |
| *** lpetrut has quit IRC | 17:34 | |
| corvus | AJaeger, fungi: are you around to review https://review.opendev.org/655739 ? | 17:34 |
| *** igordc has joined #openstack-infra | 17:35 | |
| openstackgerrit | sebastian marcet proposed osf/openstackid-resources master: Fixed multipart form request for slide updates (PUT) https://review.opendev.org/655745 | 17:35 |
| *** jcoufal has quit IRC | 17:35 | |
| openstackgerrit | Merged osf/openstackid-resources master: Fixed multipart form request for slide updates (PUT) https://review.opendev.org/655745 | 17:37 |
| AJaeger | corvus: on it... | 17:42 |
| AJaeger | corvus: too slow ;( | 17:43 |
| *** igordc has quit IRC | 17:43 | |
| *** ijw has joined #openstack-infra | 17:43 | |
| *** Lucas_Gray has quit IRC | 17:43 | |
| corvus | AJaeger: how about https://review.opendev.org/655744 ? :) | 17:44 |
| mordred | infra-root: I need to step away for a few - biab | 17:44 |
| *** igordc has joined #openstack-infra | 17:45 | |
| *** Weifan has quit IRC | 17:46 | |
| *** ijw has quit IRC | 17:48 | |
| AJaeger | pabelanger: why did you remove your +A from 655744? I can add mine - just wondering... | 17:49 |
| *** ijw has joined #openstack-infra | 17:49 | |
| pabelanger | AJaeger: I didn't see 2nd patch | 17:49 |
| pabelanger | was just about to say I prefer 655744 over other | 17:49 |
| pabelanger | if other infra-roots wanted to look | 17:49 |
| AJaeger | corvus: do we need https://review.opendev.org/#/c/655741/ as well as 744? | 17:50 |
| AJaeger | sorry, now caught up on reviews | 17:51 |
| corvus | AJaeger, no it's an alternate to 744 | 17:51 |
| corvus | seems that so far everyone who has a preference like 744, so i think approving that now and wiping or abandoning the other is good. if we decide we want the other, we can move to it, but it will take a few days to do. | 17:51 |
| corvus | they are compatible (we merge the jobs change, and later the image change, and then remove the jobs change) if we wanted | 17:52 |
| *** Weifan has joined #openstack-infra | 17:53 | |
| pabelanger | +1 | 17:53 |
| *** Weifan has quit IRC | 17:53 | |
| *** nicolasbock has quit IRC | 17:53 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Add retries to skopeo copy operations https://review.opendev.org/655739 | 17:54 |
| *** Weifan has joined #openstack-infra | 17:54 | |
| AJaeger | corvus: great | 17:54 |
| *** Weifan has quit IRC | 17:59 | |
| *** igordc has quit IRC | 18:00 | |
| *** hwoarang has quit IRC | 18:00 | |
| *** bhavikdbavishi has quit IRC | 18:00 | |
| openstackgerrit | Merged opendev/base-jobs master: Open the firewall port for the buildset registry https://review.opendev.org/655744 | 18:01 |
| *** hwoarang has joined #openstack-infra | 18:01 | |
| *** nicolasbock has joined #openstack-infra | 18:02 | |
| corvus | we have 2 zuul changes in check that could exercise that; i think 655474 is too old to get that, but i'm optimistic that 655491 will | 18:04 |
| *** igordc has joined #openstack-infra | 18:06 | |
| *** ijw has quit IRC | 18:08 | |
| *** ijw has joined #openstack-infra | 18:09 | |
| *** ijw has quit IRC | 18:18 | |
| *** ijw has joined #openstack-infra | 18:18 | |
| *** lpetrut has joined #openstack-infra | 18:18 | |
| *** ijw has joined #openstack-infra | 18:19 | |
| rpioso | corvus: After a change has been merged, can comments be sent to the owner and reviewers? The Gerrit web UI is permitting me to create draft comments, but I don't see a Reply button. | 18:19 |
| *** ijw has joined #openstack-infra | 18:19 | |
| *** tosky has quit IRC | 18:20 | |
| corvus | rpioso: comments may be left on changes after they have merged | 18:20 |
| rpioso | corvus: How are they sent to the owner and other reviewers? | 18:20 |
| rpioso | Presently, they're marked as draft comments. | 18:20 |
| corvus | rpioso: draft comments are not. but if you publish them, they will appear as normal comments. | 18:20 |
| openstackgerrit | James E. Blair proposed opendev/base-jobs master: sudo add iptables rules https://review.opendev.org/655759 | 18:21 |
| corvus | AJaeger, mordred: ^ | 18:21 |
| rpioso | corvus: How are they published? There's no Reply... button displayed. | 18:21 |
| corvus | rpioso: what url are you at? | 18:22 |
| openstackgerrit | sebastian marcet proposed osf/openstackid-resources master: Fix error on multipart parsing https://review.opendev.org/655760 | 18:22 |
| openstackgerrit | Merged osf/openstackid-resources master: Fix error on multipart parsing https://review.opendev.org/655760 | 18:22 |
| AJaeger | Jeffrey4l: +2A | 18:23 |
| AJaeger | Jeffrey4l: sorry, complete wrong autocompltion | 18:23 |
| AJaeger | trying again... | 18:23 |
| AJaeger | corvus: +2A | 18:23 |
| *** corvus is now known as jeblair | 18:23 | |
| jeblair | AJaeger: thanks! | 18:23 |
| *** jeblair is now known as corvus | 18:23 | |
| AJaeger | ;) | 18:23 |
| *** lpetrut has quit IRC | 18:23 | |
| AJaeger | mordred: do we still need https://review.opendev.org/580871 or time to abandon? | 18:24 |
| openstackgerrit | James E. Blair proposed openstack/project-config master: Don't announce osf/ projects in openstack-infra/opendev https://review.opendev.org/655761 | 18:25 |
| rpioso | corvus: Never mind. Chrome hides the button when it's not full screen. | 18:26 |
| rpioso | It was half screen on a 23" display :-( | 18:27 |
| corvus | helpful! | 18:27 |
| fungi | back and checking in before i try to wrestle my yard into submission... looks like 655491,2 failed out on more unit test timeouts? | 18:27 |
| AJaeger | corvus: time to abandon https://review.opendev.org/587178 and https://review.opendev.org/556885 ? | 18:28 |
| corvus | fungi: fixes for registry errors are in progress and currently broken; i'll recheck that soon. | 18:28 |
| corvus | AJaeger: i'm not in a position to deal with that right now | 18:29 |
| corvus | AJaeger: we can WIP those if that would help | 18:29 |
| AJaeger | corvus: I can ping in a few weeks again ;) no worries... | 18:29 |
| corvus | currently, my stack overfloweth | 18:30 |
| fungi | taking a look at 655739 now | 18:31 |
| fungi | wow, there's more scrollback than i thought, that's already merged. continuing | 18:31 |
| fungi | 655744 too | 18:31 |
| corvus | fungi: post-merge reviews still welcome :) | 18:32 |
| openstackgerrit | Merged opendev/base-jobs master: sudo add iptables rules https://review.opendev.org/655759 | 18:32 |
| corvus | okay, i'm going to re-enqueue some changes now | 18:32 |
| fungi | well, my post-merge activities also have to compete with yardwork, so... | 18:33 |
| fungi | i figure if it's broken i'll just find out when problems surface | 18:33 |
| *** ijw has quit IRC | 18:35 | |
| *** ijw has joined #openstack-infra | 18:35 | |
| *** ijw has quit IRC | 18:37 | |
| *** ijw has joined #openstack-infra | 18:38 | |
| *** ijw has quit IRC | 18:41 | |
| *** ijw has joined #openstack-infra | 18:42 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: WIP add repl https://review.opendev.org/579962 | 18:45 |
| *** ijw has quit IRC | 18:47 | |
| *** ijw has joined #openstack-infra | 18:48 | |
| openstackgerrit | Paul Belanger proposed zuul/zuul master: Support Ansible 2.8 https://review.opendev.org/631933 | 18:48 |
| *** ijw has quit IRC | 18:54 | |
| *** ijw has joined #openstack-infra | 18:54 | |
| *** jamesmcarthur has quit IRC | 19:02 | |
| *** e0ne has joined #openstack-infra | 19:06 | |
| *** jamesmcarthur has joined #openstack-infra | 19:14 | |
| *** lpetrut has joined #openstack-infra | 19:14 | |
| openstackgerrit | Jason Lee proposed opendev/storyboard master: WIP: Adds Blueprints to DB, Errors in Task Creation https://review.opendev.org/654812 | 19:16 |
| *** ijw has quit IRC | 19:17 | |
| *** jamesmcarthur has quit IRC | 19:17 | |
| *** e0ne has quit IRC | 19:17 | |
| *** igordc has quit IRC | 19:17 | |
| *** ijw has joined #openstack-infra | 19:17 | |
| *** jcoufal has joined #openstack-infra | 19:17 | |
| *** jamesmcarthur has joined #openstack-infra | 19:18 | |
| *** ijw has quit IRC | 19:20 | |
| *** ijw has joined #openstack-infra | 19:20 | |
| *** ijw has quit IRC | 19:21 | |
| *** ijw has joined #openstack-infra | 19:21 | |
| *** ykarel|away has quit IRC | 19:22 | |
| corvus | AJaeger, mordred: progress! the most recent run of 655491 ran the new iptables roles successfully | 19:24 |
| corvus | s/roles/tasks/ | 19:24 |
| *** eharney has quit IRC | 19:25 | |
| corvus | that was on ipv4, so we haven't demonstrated conclusively that solves the v6 problem (EOF), but we've at least unbroken ourselves | 19:25 |
| mordred | corvus: \o/ | 19:25 |
| corvus | mordred: your retries fixed the blob issue! | 19:25 |
| corvus | mordred: http://logs.openstack.org/91/655491/2/gate/zuul-upload-image/8b4d7af/ara-report/result/8ccff3b6-849c-446b-8a47-91803af172b2/ | 19:26 |
| mordred | corvus: double-\o/ | 19:26 |
| AJaeger | corvus, \o/ | 19:26 |
| mordred | corvus: that's very exciting | 19:26 |
| corvus | how should i read "Retries 4" in the ara output? | 19:26 |
| corvus | it looks like that output is saying there were 2 attempts; failed on the first passed on the second | 19:27 |
| corvus | they are numbered "Attempts" 1 and 2 | 19:27 |
| mordred | corvus: I can come up with no explanation of how to read retries 4 | 19:29 |
| corvus | mordred: i'm going to assume ansible is bad at math | 19:29 |
| mordred | unless it's a bug that's somehow adding attempt 1 to retries | 19:29 |
| mordred | yeah | 19:29 |
| corvus | it looks like all the other copies succeeded without retries | 19:29 |
| mordred | cool. so sometimes something just derps | 19:30 |
| *** jamesmcarthur has quit IRC | 19:32 | |
| corvus | i'm assuming someone keeps enqueing 655491 into gate despite check failures, right? | 19:37 |
| corvus | (that's fine, it's just -- it's not me and wanted to make sure that's what's happening) | 19:38 |
| *** yamamoto has joined #openstack-infra | 19:40 | |
| fungi | corvus: yes, that's me ;) | 19:42 |
| fungi | sorry, trying to at least keep an eye on something useful in between laps with the mower | 19:42 |
| fungi | and cheering on the registry troubleshooting from the sidelines | 19:43 |
| corvus | i'd like to point out that hideci is making it all but impossible for the zuul team to do it's work | 19:44 |
| *** yamamoto has quit IRC | 19:44 | |
| *** samueldmq has joined #openstack-infra | 19:44 | |
| corvus | we're having conversations in #zuul which we should not have to be having because most of the team can't even see zuul's output on its own code review system | 19:45 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: WIP: Fix requirements loop warning https://review.opendev.org/655781 | 19:45 |
| fungi | as in having to unhide the results? yeah, maybe we should trim that back to not filter out our gating ci system | 19:46 |
| fungi | and yeah, trying to follow along with conversation there too | 19:47 |
| corvus | there are so many problems with it and no one seems interested in fixing them. maybe we can exclude zuul for now and remove it entirely after the gerrit upgrade (where we should be able to use the reporting system plugin) | 19:49 |
| *** igordc has joined #openstack-infra | 19:49 | |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Do not hide Zuul comments https://review.opendev.org/655782 | 19:50 |
| corvus | fungi: there's that ^ | 19:50 |
| mnaser | corvus: is this some js-fu that I can help with? | 19:50 |
| corvus | mnaser: indeed it is | 19:50 |
| mnaser | oh I see that file | 19:51 |
| mnaser | corvus: where do I have to read to get context? | 19:51 |
| mnaser | maybe I can help fix quickly.. | 19:51 |
| corvus | mnaser: there are 2 problems that we identified in a recent #zuul conversation | 19:52 |
| corvus | the first is only going to be addressed by not hiding zuul comments, so i think 655782 is important regardless | 19:52 |
| corvus | the second is that if a job is reported without a log link, it doesn't match the regex that causes it to appear in the table at the top | 19:53 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: WIP: Fix requirements loop warning https://review.opendev.org/655781 | 19:53 |
| corvus | see https://screenshots.firefox.com/gOvoTW5JW5PvymIT/review.opendev.org | 19:53 |
| mnaser | corvus: ah yes, I've ran into that several times too in other context (i.e. RETRY_LIMIT with finger:// url might do the same) | 19:55 |
| mnaser | I think that's the same issue there? | 19:55 |
| corvus | mnaser: probably | 19:55 |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Don't hide Zuul CI comments https://review.opendev.org/655783 | 19:55 |
| mnaser | https://usercontent.irccloud-cdn.com/file/FF60bE1g/image.png | 19:55 |
| fungi | corvus: ^ a stab at it | 19:55 |
| mnaser | 1 changes set apart | 19:56 |
| fungi | oh, you were also writing one | 19:56 |
| mnaser | almost the same commit id | 19:56 |
| mnaser | :P | 19:56 |
| fungi | i'll abandon | 19:56 |
| corvus | fungi's is better | 19:56 |
| mnaser | s/id/title/ | 19:56 |
| fungi | oh, okay | 19:56 |
| fungi | we have two convergent solutions for comparison ;) | 19:56 |
| fungi | i needed an excuse to sit down for a few minutes anyway | 19:57 |
| fungi | and fwiw i do more often than not end up un-hiding ci comments just so i can see when zuul leaves a new comment | 20:00 |
| fungi | (even when i expect its results to also be exposed in the ci table) | 20:00 |
| mnaser | ok so | 20:02 |
| mnaser | I've narrowed it down here | 20:02 |
| mnaser | https://opendev.org/opendev/system-config/src/branch/master/modules/openstack_project/manifests/review.pp#L176-L180 | 20:02 |
| fungi | there's a bunch which looks like it could be cleaned up in hideci.js, but our time is probably better spent on newer gerrit's ci integration feature than trying to polish that obsolete thing too heavily | 20:03 |
| corvus | i'm way overdue for lunch; biab. | 20:03 |
| fungi | enjoy! | 20:03 |
| mnaser | I think its failing to parse that to convert it to the url that is annotated with span/etc | 20:04 |
| mnaser | which then makes hideci fail | 20:04 |
| fungi | i'm going to continue vacillating between cutting my lawn and reenqueuing the zuul regression revert so we can hopefully unblock pypi uploads | 20:04 |
| *** _erlon_ has quit IRC | 20:09 | |
| openstackgerrit | sebastian marcet proposed osf/openstackid-resources master: Fix on multiparse request data * bug fixin * refactoring https://review.opendev.org/655786 | 20:10 |
| openstackgerrit | Merged osf/openstackid-resources master: Fix on multiparse request data * bug fixin * refactoring https://review.opendev.org/655786 | 20:12 |
| *** eharney has joined #openstack-infra | 20:18 | |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: A reporter for Elasticsearch with the capability to index build and buildset results in an index. https://review.opendev.org/644927 | 20:18 |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: A reporter for Elasticsearch with the capability to index build and buildset results in an index. https://review.opendev.org/644927 | 20:19 |
| *** Weifan has joined #openstack-infra | 20:19 | |
| *** eernst has joined #openstack-infra | 20:22 | |
| *** dave-mccowan has joined #openstack-infra | 20:26 | |
| *** eernst has quit IRC | 20:26 | |
| *** kgiusti has left #openstack-infra | 20:30 | |
| *** panda is now known as panda|off | 20:30 | |
| *** jamesmcarthur has joined #openstack-infra | 20:32 | |
| *** igordc has quit IRC | 20:36 | |
| *** Weifan has quit IRC | 20:41 | |
| *** eernst has joined #openstack-infra | 20:43 | |
| *** eernst has quit IRC | 20:45 | |
| *** kranthikirang has quit IRC | 20:45 | |
| *** eernst_ has joined #openstack-infra | 20:45 | |
| *** eharney has quit IRC | 20:46 | |
| *** pcaruana has quit IRC | 20:48 | |
| *** ricolin has quit IRC | 20:49 | |
| *** gmann is now known as gmann_afk | 21:01 | |
| *** Weifan has joined #openstack-infra | 21:03 | |
| openstackgerrit | sebastian marcet proposed osf/openstackid-resources master: Fixed bool parsing on multipart form https://review.opendev.org/655796 | 21:09 |
| openstackgerrit | Merged osf/openstackid-resources master: Fixed bool parsing on multipart form https://review.opendev.org/655796 | 21:10 |
| *** Goneri has joined #openstack-infra | 21:14 | |
| *** ijw_ has joined #openstack-infra | 21:15 | |
| *** smarcet has joined #openstack-infra | 21:15 | |
| *** ijw has quit IRC | 21:18 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Fix requirements loop warning https://review.opendev.org/655781 | 21:24 |
| *** Weifan has quit IRC | 21:29 | |
| *** Goneri has quit IRC | 21:33 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Don't repeat the etc/alias setup for buildset registry pushes https://review.opendev.org/655802 | 21:33 |
| *** ijw_ has quit IRC | 21:34 | |
| corvus | that one can wait until next week; it's just an optimization. | 21:34 |
| *** ijw has joined #openstack-infra | 21:34 | |
| *** ijw has quit IRC | 21:35 | |
| *** ijw has joined #openstack-infra | 21:36 | |
| *** rlandy has quit IRC | 21:36 | |
| *** Goneri has joined #openstack-infra | 21:37 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Fix requirements loop warning https://review.opendev.org/655781 | 21:40 |
| *** Weifan has joined #openstack-infra | 21:42 | |
| *** jamesmcarthur has quit IRC | 21:44 | |
| *** eharney has joined #openstack-infra | 21:44 | |
| *** Goneri has quit IRC | 21:46 | |
| *** Goneri has joined #openstack-infra | 21:47 | |
| *** Lucas_Gray has joined #openstack-infra | 21:48 | |
| *** jamesmcarthur has joined #openstack-infra | 21:51 | |
| *** jamesmcarthur has quit IRC | 21:53 | |
| *** jamesmcarthur has joined #openstack-infra | 21:56 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Halve stestr concurrency https://review.opendev.org/655804 | 21:56 |
| *** slaweq has quit IRC | 21:57 | |
| *** Goneri has quit IRC | 21:57 | |
| corvus | clarkb: zuul has been backlogged all day so far with no appreciable memory increase | 21:58 |
| *** Weifan has quit IRC | 21:58 | |
| fungi | yeah, it's been holding steady | 21:59 |
| fungi | so i think one of the missing commits is likely the culprit | 21:59 |
| *** ccamacho has joined #openstack-infra | 21:59 | |
| clarkb | ok that implies on or both of the reverts are to blame | 21:59 |
| fungi | yeah, could be both i suppose | 21:59 |
| corvus | did jobs for 655491 just restart? | 22:00 |
| *** imacdonn has quit IRC | 22:02 | |
| *** imacdonn has joined #openstack-infra | 22:02 | |
| fungi | yeah, the py35 unit tests hit test timeouts again so i "promoted" it as the only change in the queue | 22:03 |
| corvus | yes, tox-docs finished at 21:26 and 21:59 | 22:03 |
| corvus | ah ok | 22:03 |
| corvus | fungi: registry jobs okay? | 22:03 |
| fungi | seemed to be | 22:03 |
| fungi | so i think that's fixed | 22:03 |
| *** hwoarang has quit IRC | 22:03 | |
| fungi | it's just the slow unit tests which are problematic now | 22:04 |
| corvus | k; i've been checking on the reports to keep an eye on that. if you promote, let me know if you see anything other than a tox failure, since i won't see it otherwise. | 22:04 |
| fungi | happy to! | 22:04 |
| *** jamesmcarthur has quit IRC | 22:04 | |
| *** kaiokmo has quit IRC | 22:05 | |
| *** ijw has quit IRC | 22:06 | |
| *** ijw has joined #openstack-infra | 22:07 | |
| *** hwoarang has joined #openstack-infra | 22:07 | |
| *** ijw has quit IRC | 22:09 | |
| *** ijw has joined #openstack-infra | 22:10 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: DNM: exercise halving concurrency https://review.opendev.org/655805 | 22:11 |
| *** slaweq has joined #openstack-infra | 22:11 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: DNM: exercise halving concurrency https://review.opendev.org/655805 | 22:12 |
| *** jcoufal has quit IRC | 22:16 | |
| *** lpetrut has quit IRC | 22:26 | |
| *** jamesmcarthur has joined #openstack-infra | 22:27 | |
| *** ijw has quit IRC | 22:29 | |
| *** ijw has joined #openstack-infra | 22:29 | |
| *** ijw has joined #openstack-infra | 22:31 | |
| *** ijw has joined #openstack-infra | 22:31 | |
| *** jamesmcarthur has quit IRC | 22:32 | |
| fungi | corvus: enqueuing 655491,2 into the gate again, this time hit unit test timeouts under py36 instead of py35 but everything else passed | 22:35 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Fix race in test_job_pause_pre_skipped_child https://review.opendev.org/655808 | 22:36 |
| corvus | fungi: that's a fix for the error it hit ^ | 22:36 |
| *** slaweq has quit IRC | 22:37 | |
| fungi | ooh! | 22:38 |
| fungi | thanks | 22:38 |
| fungi | oh, yeah, i wonder if this isn't the reason for most of them | 22:39 |
| openstackgerrit | melanie witt proposed openstack/project-config master: Add glance, tempest, and nova stable* to ceph grafana https://review.opendev.org/655809 | 22:41 |
| *** smarcet has quit IRC | 22:41 | |
| *** Weifan has joined #openstack-infra | 22:44 | |
| corvus | fungi: i think at least half the failures are due to resource contention | 22:44 |
| *** smarcet has joined #openstack-infra | 22:44 | |
| corvus | fungi: however, i have seen this test fail before | 22:44 |
| corvus | usually if a bunch of tests fail and there are messages about zookeeper or gearman disconnections, it's contention | 22:45 |
| *** smarcet has quit IRC | 22:45 | |
| corvus | if one test fails, it's probably a race | 22:45 |
| openstackgerrit | Merged zuul/nodepool master: Update devstack settings and docs for opendev https://review.opendev.org/654230 | 22:45 |
| fungi | new life experience: debugging python source code over a voice call while pushing a mower around the lawn | 22:46 |
| *** Weifan has quit IRC | 22:48 | |
| *** whoami-rajat has quit IRC | 22:51 | |
| mordred | fungi: I have never debugged python source code while pushing a lawn mower | 22:57 |
| *** tkajinam has joined #openstack-infra | 23:01 | |
| *** lathiat has quit IRC | 23:02 | |
| *** rcernin has joined #openstack-infra | 23:03 | |
| *** Lucas_Gray has quit IRC | 23:04 | |
| mordred | corvus: I had a thought re: refs/changes and the long replication - which is that we could add the patch to our gitea, then make a script that we run that pushes refs/changes refs/notes to gitea outside of the gerrit context - just in the background - then once it has caught up, remove the gerrit exclusion. it doesn't really help with making a new gitea server - but maybe for that we just need to | 23:07 |
| mordred | rsync the filesystem of an existing one and copy the mysql db | 23:07 |
| *** lathiat has joined #openstack-infra | 23:16 | |
| *** lseki has quit IRC | 23:16 | |
| *** slaweq has joined #openstack-infra | 23:16 | |
| fungi | i've moved on from yardwork to laundry, and 655491,2 is getting really close to merging | 23:18 |
| fungi | actually passed all its tox jobs this time | 23:18 |
| fungi | just needs the quickstart and image uploads to finish without incident | 23:18 |
| *** jamesmcarthur has joined #openstack-infra | 23:18 | |
| *** harlowja has quit IRC | 23:19 | |
| mordred | fingers crossed | 23:19 |
| openstackgerrit | Merged zuul/zuul master: Revert "Prepend path with bin dir of ansible virtualenv" https://review.opendev.org/655491 | 23:20 |
| clarkb | thereit is | 23:20 |
| *** Weifan has joined #openstack-infra | 23:21 | |
| fungi | woo!!! | 23:23 |
| fungi | i guess once puppet rolls that out i may be freed up enough for executor restarts | 23:23 |
| *** slaweq has quit IRC | 23:24 | |
| openstackgerrit | Jason Lee proposed opendev/storyboard master: RC1: Release Candidate 1 https://review.opendev.org/654812 | 23:28 |
| corvus | mordred: i think when we make a new gitea server, maybe it's okay to bring it up and let it sit for a day getting replicated to before we add it to the LB | 23:28 |
| smcginnis | \o/ | 23:28 |
| corvus | mordred: if we're okay with that, then i think the complexity around the current situation is just making sure we do something in a way that doesn't take out the existing system for any length of time. | 23:29 |
| *** Weifan has quit IRC | 23:29 | |
| corvus | mordred: so, yeah, i think the background script idea may be a good one for getting through the current state | 23:30 |
| corvus | the gitea folks haven't been in quite so much of a hurry to review the refs change yet, though they have seen it and thrown some labels/milestones at it | 23:31 |
| mordred | corvus: yeah - I think I'm ok with that - as I also think it's hopefully a short to medium term thing - and eventually we'll back to a cluster where adding/deleting nodes != needing to re-replicate | 23:38 |
| corvus | ya that too | 23:40 |
| *** yamamoto has joined #openstack-infra | 23:42 | |
| *** yamamoto has quit IRC | 23:46 | |
| *** jamesmcarthur has quit IRC | 23:48 | |
| *** jamesmcarthur has joined #openstack-infra | 23:49 | |
| mordred | ok - it's that time for me - see y'all tomorry | 23:50 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Fix race in test_job_pause_pre_skipped_child https://review.opendev.org/655808 | 23:54 |
| *** gyee has quit IRC | 23:57 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
