| openstackgerrit | Ian Wienand proposed opendev/system-config master: mirror01.dfw.rax.opendev.org : use python3 for ansible https://review.opendev.org/660538 | 00:03 |
|---|---|---|
| *** bobh has joined #openstack-infra | 00:08 | |
| *** gyee has quit IRC | 00:10 | |
| *** hamzy has joined #openstack-infra | 00:10 | |
| *** altlogbot_1 has quit IRC | 00:10 | |
| *** Goneri has joined #openstack-infra | 00:10 | |
| *** altlogbot_0 has joined #openstack-infra | 00:11 | |
| *** bobh has quit IRC | 00:18 | |
| *** mattw4 has quit IRC | 00:20 | |
| *** mattw4 has joined #openstack-infra | 00:20 | |
| *** rcernin has quit IRC | 00:21 | |
| *** sreejithp_ has quit IRC | 00:21 | |
| *** rcernin has joined #openstack-infra | 00:21 | |
| *** sreejithp has joined #openstack-infra | 00:22 | |
| *** sreejithp_ has joined #openstack-infra | 00:25 | |
| *** mattw4 has quit IRC | 00:25 | |
| *** ijw has joined #openstack-infra | 00:26 | |
| *** sreejithp has quit IRC | 00:28 | |
| *** ijw has quit IRC | 00:28 | |
| *** ijw has joined #openstack-infra | 00:29 | |
| *** michael-beaver has quit IRC | 00:29 | |
| *** ijw has quit IRC | 00:29 | |
| *** ijw has joined #openstack-infra | 00:29 | |
| *** armax has joined #openstack-infra | 00:30 | |
| *** ijw has quit IRC | 00:35 | |
| *** ijw_ has joined #openstack-infra | 00:35 | |
| *** igordc has quit IRC | 00:36 | |
| *** sreejithp_ has quit IRC | 00:37 | |
| *** sreejithp has joined #openstack-infra | 00:37 | |
| *** ijw_ has quit IRC | 00:37 | |
| *** ijw has joined #openstack-infra | 00:38 | |
| *** zhurong has joined #openstack-infra | 00:39 | |
| *** jamesdenton has joined #openstack-infra | 00:44 | |
| *** bobh has joined #openstack-infra | 00:48 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Replace skip with errors=ignore https://review.opendev.org/660541 | 00:48 |
| *** ijw has quit IRC | 00:54 | |
| *** ijw has joined #openstack-infra | 00:55 | |
| *** ijw has quit IRC | 00:58 | |
| *** ijw has joined #openstack-infra | 00:58 | |
| *** bobh has quit IRC | 01:08 | |
| *** ijw has quit IRC | 01:11 | |
| *** ijw has joined #openstack-infra | 01:11 | |
| *** ijw has joined #openstack-infra | 01:12 | |
| *** betherly has quit IRC | 01:17 | |
| *** diablo_rojo has quit IRC | 01:21 | |
| *** rcernin has quit IRC | 01:22 | |
| *** rcernin has joined #openstack-infra | 01:22 | |
| openstackgerrit | Merged opendev/system-config master: mirror01.dfw.rax.opendev.org : use python3 for ansible https://review.opendev.org/660538 | 01:26 |
| *** gregoryo has joined #openstack-infra | 01:28 | |
| *** happyhemant has quit IRC | 01:28 | |
| *** bobh has joined #openstack-infra | 01:30 | |
| *** Goneri has quit IRC | 01:31 | |
| *** Goneri has joined #openstack-infra | 01:33 | |
| *** ijw has quit IRC | 01:38 | |
| *** bobh has quit IRC | 01:42 | |
| *** apetrich has quit IRC | 01:57 | |
| *** rlandy has joined #openstack-infra | 02:06 | |
| *** sreejithp has quit IRC | 02:07 | |
| ianw | interesting, i don't think the mirror dns changes have applied | 02:13 |
| *** rh-jelabarre has quit IRC | 02:13 | |
| *** rlandy has quit IRC | 02:15 | |
| ianw | May 22 02:02:40 adns1 named[21836]: client @0x7f2434560ee0 104.239.140.165#51948/key tsig (opendev.org): transfer of 'opendev.org/IN': AXFR started: TSIG tsig (serial 1555627256) | 02:15 |
| ianw | current serial is 1558399857 | 02:15 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zone-opendev.org master: Put dots on end of acme.opendev.org https://review.opendev.org/660543 | 02:17 |
| openstackgerrit | Merged openstack-infra/zone-opendev.org master: Put dots on end of acme.opendev.org https://review.opendev.org/660543 | 02:18 |
| *** dangtrinhnt has quit IRC | 02:21 | |
| *** dangtrinhnt has joined #openstack-infra | 02:22 | |
| *** dangtrinhnt has quit IRC | 02:23 | |
| *** ykarel|away has joined #openstack-infra | 02:24 | |
| ianw | ummm, i'm very worried dns is broken somehow | 02:30 |
| clarkb | ianw is the adns servernot sending transfers? | 02:33 |
| clarkb | or could be the base.yaml split broke dns updates | 02:34 |
| ianw | /var/lib/bind/zones/opendev.org/zone.db on adns1 looks ok | 02:36 |
| clarkb | is the symptom that your updates arent happening? | 02:39 |
| ianw | a minute ago i couldn't lookup www.opendev.org | 02:40 |
| ianw | but yeah, mirror.dfw.rax.opendev.org doesn't seem to exist either | 02:40 |
| clarkb | I cantlookup www but opendev.org resolves | 02:42 |
| clarkb | do we just not have a www record? | 02:42 |
| ianw | hrm, ok, maybe i've just got myself worried over a non-existent record :) | 02:42 |
| clarkb | I confirm that mirror record seems to not exist though | 02:43 |
| clarkb | implying no updates? | 02:43 |
| clarkb | is bind running on adns1? | 02:43 |
| *** ricolin has joined #openstack-infra | 02:44 | |
| ianw | it is, it's reporting | 02:44 |
| ianw | serial: 1558491424 | 02:44 |
| ianw | signed serial: 1555627256 | 02:44 |
| clarkb | oh a signing issue? | 02:44 |
| ianw | the signed serial seems out of date; seems like it should be after the last serial | 02:45 |
| clarkb | ya I think that implies we stopped signing (could be related to base.yaml sllit?) | 02:45 |
| *** nicolasbock has quit IRC | 02:45 | |
| ianw | May 22 02:42:40 adns1 named[21836]: zone acme.opendev.org/IN (unsigned): ixfr-from-differences: new serial (1558470372) out of range [1558470373 - 3705954019] | 02:45 |
| ianw | May 22 02:42:40 adns1 named[21836]: zone acme.opendev.org/IN (unsigned): not loaded due to errors. | 02:45 |
| ianw | argh, oh wait, that's acme | 02:47 |
| *** jamesmcarthur has joined #openstack-infra | 02:50 | |
| ianw | May 22 02:51:08 adns1 named[11161]: zone opendev.org/IN (unsigned): loading from master file /var/lib/bind/zones/opendev.org/zone.db failed: CNAME and other data | 02:51 |
| ianw | May 22 02:51:08 adns1 named[11161]: zone opendev.org/IN (unsigned): not loaded due to errors. | 02:51 |
| ianw | i must have typo'd something ? | 02:52 |
| clarkb | diff against the last working serial? | 02:53 |
| ianw | dns_master_load: ./opendev.org/zone.db:41: graphite.opendev.org: CNAME and other data | 02:53 |
| ianw | dns_master_load: ./opendev.org/zone.db:81: mirror.dfw.rax.opendev.org: CNAME and other data | 02:53 |
| ianw | i think that's the error | 02:53 |
| clarkb | oh right | 02:53 |
| clarkb | cnames must be unique | 02:54 |
| *** xinranwang has joined #openstack-infra | 02:56 | |
| ianw | i'm not sure how they're not? | 02:56 |
| ianw | oh, these IN CAA records i guess ... | 02:57 |
| ianw | that must mean you can't have an IN CAA record to a CNAME, only an A / AAAA ? | 02:57 |
| clarkb | the CAA recirds ya | 02:58 |
| clarkb | yup | 02:58 |
| clarkb | So either we switvh to A/AAAA for drop the CAA records | 02:58 |
| ianw | if a domain name is a CNAME ... then the CA looks for record sets at the CNAME target | 02:59 |
| ianw | i think we point it at the 01 servers | 02:59 |
| clarkb | huh | 02:59 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zone-opendev.org master: Use A/AAAA records for CAA https://review.opendev.org/660549 | 03:01 |
| ianw | also, i will look at getting named-checkzone running as a ci job, which would have caught this | 03:02 |
| logan- | catching up on backscroll.. glad to see that firewall fix appears to have worked clarkb. and ianw: i was also surprised not to find discussion on the journal size topic, especially now that whole disk images are becoming more common for things like bare metal deploys. i'll be interested to see how it affects the image sizes :) | 03:02 |
| ianw | clarkb / fungi : ^^ dns fix ... fungi I think I said to move the CAA records and was completely wrong about that, mea culpa | 03:02 |
| clarkb | ianw +2'd | 03:03 |
| clarkb | logan-: ya and you cant change it without unmounting the fs apparently | 03:03 |
| clarkb | logan-: which means for / you are kinda stuck with what the image sets | 03:03 |
| logan- | yep | 03:03 |
| ianw | yeah i feel like it has to be contiguous so it goes down, but not up | 03:03 |
| *** Goneri has quit IRC | 03:05 | |
| clarkb | logan- I think the firewall wouldve been way more obvious if ipv4 didnt magically work | 03:06 |
| clarkb | oh well lesson learned and hopeefully bug fixed | 03:06 |
| *** bhavikdbavishi has joined #openstack-infra | 03:08 | |
| *** factor has quit IRC | 03:12 | |
| *** factor has joined #openstack-infra | 03:12 | |
| openstackgerrit | Merged openstack-infra/zone-opendev.org master: Use A/AAAA records for CAA https://review.opendev.org/660549 | 03:16 |
| ianw | getting some lunch, will come back and check on all that ^ | 03:17 |
| *** whoami-rajat has joined #openstack-infra | 03:20 | |
| *** bhavikdbavishi1 has joined #openstack-infra | 03:31 | |
| *** bhavikdbavishi has quit IRC | 03:32 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 03:32 | |
| *** ramishra has joined #openstack-infra | 03:38 | |
| *** psachin has joined #openstack-infra | 03:38 | |
| *** armax has quit IRC | 03:43 | |
| *** armax has joined #openstack-infra | 03:44 | |
| *** jamesmcarthur has quit IRC | 03:45 | |
| *** ykarel|away is now known as ykarel | 03:58 | |
| *** armax has quit IRC | 04:01 | |
| *** sreejithp has joined #openstack-infra | 04:03 | |
| *** jamesmcarthur has joined #openstack-infra | 04:04 | |
| *** jamesmcarthur has quit IRC | 04:06 | |
| *** sreejithp has quit IRC | 04:07 | |
| *** Lucas_Gray has joined #openstack-infra | 04:11 | |
| *** udesale has joined #openstack-infra | 04:11 | |
| *** markvoelker has joined #openstack-infra | 04:14 | |
| *** markvoelker has quit IRC | 04:18 | |
| ianw | zone opendev.org/IN (unsigned): journal rollforward failed: journal out of sync with zone | 04:39 |
| ianw | it says that happens because you manually edited a zone; i don't believe i did :/ | 04:39 |
| *** slaweq has joined #openstack-infra | 04:40 | |
| ianw | removed the jnl files, restarted bind manually and it seems up | 04:46 |
| openstackgerrit | Merged zuul/zuul master: pass-to-parent: Fix passing multiple secrets to parent https://review.opendev.org/660105 | 04:51 |
| *** ykarel has quit IRC | 04:53 | |
| *** ykarel has joined #openstack-infra | 05:11 | |
| *** Lucas_Gray has quit IRC | 05:18 | |
| *** dpawlik has joined #openstack-infra | 05:18 | |
| *** ajo has quit IRC | 05:27 | |
| *** portdirect has quit IRC | 05:27 | |
| *** nickv1985 has quit IRC | 05:28 | |
| *** jrosser has quit IRC | 05:29 | |
| *** mwhahaha has quit IRC | 05:30 | |
| *** adriancz has quit IRC | 05:31 | |
| *** rm_work has quit IRC | 05:31 | |
| *** dustinc has quit IRC | 05:31 | |
| *** fresta has quit IRC | 05:32 | |
| *** fresta has joined #openstack-infra | 05:33 | |
| *** ajo has joined #openstack-infra | 05:33 | |
| *** nickv1985 has joined #openstack-infra | 05:33 | |
| *** mwhahaha has joined #openstack-infra | 05:33 | |
| *** adriancz has joined #openstack-infra | 05:33 | |
| *** portdirect has joined #openstack-infra | 05:33 | |
| *** jrosser has joined #openstack-infra | 05:34 | |
| *** dustinc has joined #openstack-infra | 05:34 | |
| AJaeger | config-core, please review https://review.opendev.org/659976 and https://review.opendev.org/659975 | 05:34 |
| *** rm_work has joined #openstack-infra | 05:34 | |
| *** quiquell has joined #openstack-infra | 05:37 | |
| *** lpetrut has joined #openstack-infra | 05:40 | |
| *** imacdonn has quit IRC | 05:41 | |
| *** imacdonn has joined #openstack-infra | 05:41 | |
| *** roman_g has quit IRC | 05:43 | |
| *** sarob has joined #openstack-infra | 05:43 | |
| *** raukadah is now known as chandankumar | 05:44 | |
| AJaeger | zuul-jobs reviewers, config-core, please have a look at https://review.opendev.org/648658 https://review.opendev.org/648815 https://review.opendev.org/632620 | 05:50 |
| *** jcoufal has joined #openstack-infra | 05:56 | |
| *** diablo_rojo has joined #openstack-infra | 06:01 | |
| *** kopecmartin|off is now known as kopecmartin | 06:06 | |
| *** markvoelker has joined #openstack-infra | 06:15 | |
| *** sarob has quit IRC | 06:17 | |
| *** dpawlik has quit IRC | 06:27 | |
| *** pgaxatte has joined #openstack-infra | 06:27 | |
| *** janki has joined #openstack-infra | 06:27 | |
| *** dtantsur|afk is now known as dtantsur | 06:28 | |
| *** e0ne has joined #openstack-infra | 06:29 | |
| *** liuyulong has quit IRC | 06:38 | |
| *** jcoufal has quit IRC | 06:39 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: letsencrypt : use date call for serial number https://review.opendev.org/660576 | 06:45 |
| ianw | infra-root: ^ i might have to emergency merge that, because we're in a loop failing to authenticate the new mirror to letsencrypt (using up our failure quota). although eventually i guess ansible_date_time will update, not sure how long it will remain cached | 06:46 |
| jhesketh | ianw: seems reasonable to me | 06:47 |
| AJaeger | ianw: agreed, go ahead and emergency merge... | 06:48 |
| ianw | jhesketh: 2 hours of my life debugging through the stack why the zone wasn't reloading i won't get back :) | 06:49 |
| *** markvoelker has quit IRC | 06:49 | |
| jhesketh | good find though :-) | 06:49 |
| AJaeger | #thanks ianw for debugging and fixing zone update problems | 06:50 |
| openstackstatus | AJaeger: Added your thanks to Thanks page (https://wiki.openstack.org/wiki/Thanks) | 06:50 |
| * AJaeger is optimistic that this fixes it ;) | 06:50 | |
| ianw | AJaeger: :) ... hit up https://mirror.dfw.rax.opendev.org in a few hours, and we'll see :) | 06:51 |
| *** diablo_rojo has quit IRC | 06:52 | |
| *** e0ne has quit IRC | 06:57 | |
| *** ralonsoh has joined #openstack-infra | 07:02 | |
| *** ralonsoh has quit IRC | 07:03 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: web: remove SafeLoader left-over from ZuulJSONEncoder https://review.opendev.org/659026 | 07:03 |
| *** ralonsoh has joined #openstack-infra | 07:03 | |
| *** ginopc has joined #openstack-infra | 07:08 | |
| *** jcoufal has joined #openstack-infra | 07:09 | |
| openstackgerrit | Ildiko Vancsa proposed opendev/system-config master: Adding new ML for the NBMP group https://review.opendev.org/660485 | 07:16 |
| *** pcaruana has joined #openstack-infra | 07:17 | |
| *** jcoufal has quit IRC | 07:17 | |
| *** tesseract has joined #openstack-infra | 07:18 | |
| *** rpittau|afk is now known as rpittau | 07:19 | |
| *** tosky has joined #openstack-infra | 07:21 | |
| *** roman_g has joined #openstack-infra | 07:25 | |
| *** hwoarang has quit IRC | 07:29 | |
| *** sarob has joined #openstack-infra | 07:34 | |
| openstackgerrit | Merged zuul/zuul-jobs master: add-build-sshkey: remove previously authorized build-sshkey https://review.opendev.org/632620 | 07:36 |
| *** hwoarang has joined #openstack-infra | 07:37 | |
| *** apetrich has joined #openstack-infra | 07:39 | |
| *** jpena|off is now known as jpena | 07:44 | |
| openstackgerrit | Merged opendev/system-config master: letsencrypt : use date call for serial number https://review.opendev.org/660576 | 07:44 |
| *** bhavikdbavishi has quit IRC | 07:44 | |
| *** markvoelker has joined #openstack-infra | 07:46 | |
| *** ykarel is now known as ykarel|lunch | 07:52 | |
| *** xinranwang has quit IRC | 07:55 | |
| *** hwoarang has quit IRC | 07:58 | |
| *** hwoarang has joined #openstack-infra | 08:00 | |
| *** ociuhandu has quit IRC | 08:00 | |
| *** yboaron_ has joined #openstack-infra | 08:00 | |
| *** ociuhandu has joined #openstack-infra | 08:02 | |
| *** sreejithp has joined #openstack-infra | 08:04 | |
| *** sarob has quit IRC | 08:07 | |
| *** sreejithp has quit IRC | 08:08 | |
| *** kashyap has left #openstack-infra | 08:08 | |
| *** dpawlik has joined #openstack-infra | 08:08 | |
| *** lucasagomes has joined #openstack-infra | 08:10 | |
| *** e0ne has joined #openstack-infra | 08:12 | |
| *** pkopec has joined #openstack-infra | 08:15 | |
| *** pkopec has quit IRC | 08:16 | |
| *** pkopec has joined #openstack-infra | 08:17 | |
| *** markvoelker has quit IRC | 08:18 | |
| *** gregoryo has quit IRC | 08:20 | |
| *** tkajinam has quit IRC | 08:21 | |
| openstackgerrit | Luigi Toscano proposed zuul/zuul-jobs master: stage-output: fix dotfiles handling https://review.opendev.org/648658 | 08:23 |
| *** ccamacho has joined #openstack-infra | 08:23 | |
| *** Lucas_Gray has joined #openstack-infra | 08:24 | |
| openstackgerrit | Luigi Toscano proposed zuul/zuul-jobs master: stage-output: fix dotfiles handling https://review.opendev.org/648658 | 08:25 |
| *** bhavikdbavishi has joined #openstack-infra | 08:31 | |
| *** gfidente|afk is now known as gfidente | 08:33 | |
| *** ykarel|lunch is now known as ykarel | 08:38 | |
| *** hwoarang has quit IRC | 08:44 | |
| *** hwoarang has joined #openstack-infra | 08:46 | |
| ianw | yay, it worked! https://mirror.dfw.rax.opendev.org/ | 08:46 |
| ianw | i had to manually reload bind9 and clear the journal after that seq number update. i don't know why | 08:47 |
| *** panda is now known as panda|rover | 08:50 | |
| *** derekh has joined #openstack-infra | 08:51 | |
| *** priteau has joined #openstack-infra | 08:55 | |
| *** electrofelix has joined #openstack-infra | 09:00 | |
| *** Lucas_Gray has quit IRC | 09:04 | |
| openstackgerrit | Merged opendev/irc-meetings master: Update Kolla meeting chair to mgoddard https://review.opendev.org/660344 | 09:08 |
| openstackgerrit | Merged opendev/irc-meetings master: Update neutron-ci agenda and chairs https://review.opendev.org/660443 | 09:08 |
| *** Lucas_Gray has joined #openstack-infra | 09:10 | |
| *** rcernin has quit IRC | 09:10 | |
| *** markvoelker has joined #openstack-infra | 09:15 | |
| openstackgerrit | Adam Spiers proposed openstack/cookiecutter master: Fix outdated references to git.openstack.org/cgit/ https://review.opendev.org/657444 | 09:16 |
| *** ricolin has quit IRC | 09:40 | |
| openstackgerrit | Nir Magnezi proposed openstack/diskimage-builder master: Add version-less RHEL element for RHEL7 and RHEL8 https://review.opendev.org/643731 | 09:42 |
| *** pgaxatte has quit IRC | 09:42 | |
| openstackgerrit | Nir Magnezi proposed openstack/diskimage-builder master: Deprecate rhel7 in favor of rhel https://review.opendev.org/647710 | 09:43 |
| *** lifeless_ is now known as lifeless | 09:46 | |
| *** flaper87 has quit IRC | 09:48 | |
| *** markvoelker has quit IRC | 09:49 | |
| *** bhavikdbavishi has quit IRC | 10:23 | |
| *** bhavikdbavishi has joined #openstack-infra | 10:23 | |
| *** bhavikdbavishi has quit IRC | 10:28 | |
| *** yboaron_ has quit IRC | 10:28 | |
| *** yboaron_ has joined #openstack-infra | 10:29 | |
| *** diablo_rojo has joined #openstack-infra | 10:36 | |
| fungi | ianw: sorry for not catching that! that was probably the reason i proposed the caa for graphite01 instead of graphite originally, but yes i should have remembered that when you asked me to change it | 10:40 |
| fungi | and also if we tried restarting bind on the adns in the job, that should fail in such situations (cname and other data would cause it to refuse to load a required zone in that case) | 10:41 |
| *** bhavikdbavishi has joined #openstack-infra | 10:43 | |
| *** markvoelker has joined #openstack-infra | 10:46 | |
| *** diablo_rojo has quit IRC | 10:56 | |
| *** toabctl has joined #openstack-infra | 10:58 | |
| *** nicolasbock has joined #openstack-infra | 11:06 | |
| *** ykarel is now known as ykarel|afk | 11:12 | |
| *** panda|rover is now known as panda|rover|eat | 11:16 | |
| *** markvoelker has quit IRC | 11:18 | |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Upgrade formatting of the patch series. https://review.opendev.org/660683 | 11:27 |
| *** pkopec has quit IRC | 11:29 | |
| *** pkopec has joined #openstack-infra | 11:30 | |
| *** udesale has quit IRC | 11:30 | |
| *** sreejithp has joined #openstack-infra | 11:30 | |
| *** udesale has joined #openstack-infra | 11:31 | |
| *** jpena is now known as jpena|lunch | 11:31 | |
| *** ginopc has quit IRC | 11:32 | |
| *** sshnaidm|afk is now known as sshnaidm | 11:35 | |
| *** sreejithp has quit IRC | 11:35 | |
| *** rfolco is now known as rfolco|dentist | 11:39 | |
| *** iurygregory_ has joined #openstack-infra | 11:39 | |
| *** iurygregory has quit IRC | 11:40 | |
| *** _erlon_ has joined #openstack-infra | 11:45 | |
| *** bhavikdbavishi has quit IRC | 11:49 | |
| *** bhavikdbavishi has joined #openstack-infra | 11:51 | |
| *** bhavikdbavishi has quit IRC | 11:56 | |
| *** rlandy has joined #openstack-infra | 11:57 | |
| *** apetrich has quit IRC | 12:01 | |
| *** rh-jelabarre has joined #openstack-infra | 12:01 | |
| *** panda|rover|eat is now known as panda|rover | 12:06 | |
| *** eharney has quit IRC | 12:10 | |
| *** bobh has joined #openstack-infra | 12:10 | |
| *** apetrich has joined #openstack-infra | 12:14 | |
| *** markvoelker has joined #openstack-infra | 12:16 | |
| *** ykarel|afk is now known as ykarel | 12:18 | |
| *** apetrich has quit IRC | 12:19 | |
| *** bobh has quit IRC | 12:19 | |
| *** rfarr__ has joined #openstack-infra | 12:21 | |
| *** hwoarang has quit IRC | 12:23 | |
| *** hwoarang has joined #openstack-infra | 12:25 | |
| *** rfarr_ has joined #openstack-infra | 12:27 | |
| *** psachin has quit IRC | 12:27 | |
| *** iurygregory_ is now known as iurygregory | 12:28 | |
| *** markvoelker has quit IRC | 12:29 | |
| *** rfarr__ has quit IRC | 12:29 | |
| *** jpena|lunch is now known as jpena | 12:33 | |
| *** persia has quit IRC | 12:34 | |
| *** apetrich has joined #openstack-infra | 12:35 | |
| *** persia has joined #openstack-infra | 12:35 | |
| *** janki has quit IRC | 12:35 | |
| openstackgerrit | Adam Coldrick proposed opendev/storyboard master: Install compatible version of python-openid when using Python 2 https://review.opendev.org/660701 | 12:39 |
| *** bhavikdbavishi has joined #openstack-infra | 12:52 | |
| *** jamesmcarthur has joined #openstack-infra | 12:54 | |
| *** ricolin has joined #openstack-infra | 12:55 | |
| *** zul has joined #openstack-infra | 13:02 | |
| *** igordc has joined #openstack-infra | 13:06 | |
| *** priteau has quit IRC | 13:09 | |
| *** bhavikdbavishi has quit IRC | 13:10 | |
| *** adriancz has quit IRC | 13:10 | |
| openstackgerrit | Merged opendev/storyboard master: Install compatible version of python-openid when using Python 2 https://review.opendev.org/660701 | 13:11 |
| *** mriedem has joined #openstack-infra | 13:14 | |
| *** lseki has joined #openstack-infra | 13:21 | |
| *** pkopec has quit IRC | 13:23 | |
| *** aaronsheffield has joined #openstack-infra | 13:25 | |
| *** ginopc has joined #openstack-infra | 13:28 | |
| *** rfolco|dentist is now known as rfolco | 13:29 | |
| *** Goneri has joined #openstack-infra | 13:29 | |
| *** dtantsur is now known as dtantsur|brb | 13:29 | |
| *** sreejithp has joined #openstack-infra | 13:33 | |
| *** eharney has joined #openstack-infra | 13:36 | |
| *** sreejithp has quit IRC | 13:37 | |
| *** sreejithp has joined #openstack-infra | 13:38 | |
| *** hwoarang has quit IRC | 13:45 | |
| *** liuyulong has joined #openstack-infra | 13:46 | |
| *** hwoarang has joined #openstack-infra | 13:46 | |
| *** kaiokmo has joined #openstack-infra | 13:49 | |
| *** igordc has quit IRC | 13:51 | |
| *** sarob has joined #openstack-infra | 13:53 | |
| zigo | fungi: Where may I see the logs of the Buster build in the infra? | 13:55 |
| *** bobh has joined #openstack-infra | 13:55 | |
| zigo | (thanks for the +2w btw) | 13:55 |
| *** rfarr_ has quit IRC | 13:57 | |
| fungi | just a sec, finding it | 13:59 |
| fungi | zigo: https://nb01.openstack.org/debian-buster-0000000001.log | 13:59 |
| fungi | 2019-05-21 23:43:56.486 | Build completed successfully | 14:00 |
| fungi | yay! | 14:00 |
| fungi | now... hopefully it's right | 14:00 |
| mordred | fungi: who needs right when you have right now? | 14:00 |
| fungi | zigo: also you can fetch https://nb01.openstack.org/images/debian-buster-0000000001.qcow2 if you want to inspect it, though be forewarned it's 9.1gib | 14:01 |
| fungi | (mostly thanks to the git caches we include) | 14:02 |
| zigo | fungi: Great, thanks so much! :) | 14:03 |
| zigo | I don't really mind 9GB from here in the office... :P | 14:03 |
| *** ginopc has quit IRC | 14:04 | |
| fungi | you can also find checksums in the same directory if you want them | 14:08 |
| zigo | fungi: Is the image safe to boot in my own cloud? | 14:09 |
| zigo | There's no cloud-init, right? | 14:09 |
| fungi | it uses glean, which will expect you to provide metadata via configdrive | 14:09 |
| fungi | though for network access it can rely on dhcp and/or ipv6 ra | 14:10 |
| zigo | Right, so I just need to use the --config-drive param. | 14:10 |
| fungi | yeah | 14:10 |
| openstackgerrit | Adam Spiers proposed openstack/cookiecutter master: Fix outdated references to git.openstack.org/cgit/ https://review.opendev.org/657444 | 14:11 |
| *** quiquell has quit IRC | 14:12 | |
| fungi | and ssh keys you provide should automatically be authorized for the root user, i believe | 14:12 |
| *** lpetrut has quit IRC | 14:12 | |
| *** ginopc has joined #openstack-infra | 14:14 | |
| openstackgerrit | Adam Spiers proposed openstack/cookiecutter master: Add a testenv for lower-constraints https://review.opendev.org/660732 | 14:15 |
| *** jamesmcarthur has quit IRC | 14:16 | |
| openstackgerrit | Matt McEuen proposed openstack/project-config master: Add granular ACLs for Airship Projects https://review.opendev.org/660461 | 14:17 |
| *** jamesmcarthur has joined #openstack-infra | 14:18 | |
| *** jamesmcarthur has quit IRC | 14:20 | |
| *** quiquell has joined #openstack-infra | 14:21 | |
| *** roman_g has quit IRC | 14:22 | |
| mriedem | i noticed the compute api-ref was last published on march 26 https://developer.openstack.org/api-ref/compute/ | 14:25 |
| mriedem | is that when the opendev migration happened? | 14:25 |
| *** sarob has quit IRC | 14:27 | |
| zigo | fungi: Would you use glean as a package if I do that? | 14:28 |
| *** dpawlik has quit IRC | 14:29 | |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Pagure driver - https://pagure.io/pagure/ https://review.opendev.org/604404 | 14:29 |
| *** dtantsur|brb is now known as dtantsur | 14:29 | |
| *** bobh has quit IRC | 14:30 | |
| *** ykarel is now known as ykarel|afk | 14:31 | |
| *** jamesmcarthur has joined #openstack-infra | 14:41 | |
| *** armax has joined #openstack-infra | 14:41 | |
| *** jamesmcarthur has quit IRC | 14:41 | |
| *** jamesmcarthur has joined #openstack-infra | 14:42 | |
| *** udesale has quit IRC | 14:44 | |
| *** udesale has joined #openstack-infra | 14:44 | |
| *** mriedem is now known as hansmoleman | 14:44 | |
| *** tosky has quit IRC | 14:45 | |
| *** sarob has joined #openstack-infra | 14:46 | |
| *** tosky has joined #openstack-infra | 14:46 | |
| *** quiquell has quit IRC | 14:53 | |
| *** pkopec has joined #openstack-infra | 14:54 | |
| *** pkopec is now known as pkopec|afk | 14:54 | |
| fungi | zigo: probably not since we want to be able to continuously deploy it from source. i guess we could autobuild debian packages for it on each commit, but would probably want to do the same with rpms for centos/suse, emerge for gentoo, et cetera | 14:56 |
| fungi | and we want a consistent version installed on distros of various vintages | 14:56 |
| *** lpetrut has joined #openstack-infra | 14:57 | |
| fungi | basically because we sometimes find that to bring a new provider online we have to make adjustments to glean or add features to support unique environments | 14:57 |
| fungi | and so we need to be able to add that support to every image for every distro/version we build | 14:57 |
| fungi | using python packages for it at least gives us a single format which works on a variety of distros of different ages | 14:58 |
| *** roman_g has joined #openstack-infra | 14:59 | |
| *** hansmoleman is now known as mriedem | 15:00 | |
| *** ginopc has quit IRC | 15:05 | |
| *** ykarel|afk is now known as ykarel | 15:06 | |
| *** rfarr__ has joined #openstack-infra | 15:07 | |
| *** jtomasek has joined #openstack-infra | 15:11 | |
| *** roman_g has quit IRC | 15:12 | |
| logan- | 2nd job in ovh-bhs i have seen in the last 24h that hung for a long time trying to clone from opendev.org: http://zuul.openstack.org/stream/c497f0643085400eab96f8d98469b735?logfile=console.log | 15:14 |
| *** roman_g has joined #openstack-infra | 15:14 | |
| openstackgerrit | Sean McGinnis proposed openstack/openstack-zuul-jobs master: Fix typo in tempest-dsvm-full-hdfs https://review.opendev.org/660507 | 15:16 |
| zigo | fungi: https://salsa.debian.org/openstack-team/libs/glean <--- Packaged anyway! :) | 15:16 |
| zigo | I may try it ... | 15:16 |
| fungi | zigo: yeah, feel free to package it for debian, i just don't think the opendev infrastructure would likely be able to take advantage of that is all | 15:19 |
| fungi | i need to step out for lunch and some errands, but expect to be back in plenty of time for the storyboard meeting | 15:19 |
| corvus | i'm back at home and resettling | 15:20 |
| *** hwoarang has quit IRC | 15:20 | |
| *** diablo_rojo has joined #openstack-infra | 15:20 | |
| *** hwoarang has joined #openstack-infra | 15:21 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Annotate builds with event id https://review.opendev.org/658895 | 15:25 |
| *** ginopc has joined #openstack-infra | 15:26 | |
| *** dciabrin_ is now known as dciabrin | 15:28 | |
| *** eharney has quit IRC | 15:28 | |
| *** gyee has joined #openstack-infra | 15:32 | |
| *** ginux has joined #openstack-infra | 15:35 | |
| *** ginopc has quit IRC | 15:35 | |
| *** ginux is now known as ginopc | 15:35 | |
| *** nhicher has quit IRC | 15:35 | |
| mordred | corvus: yay! | 15:36 |
| mordred | corvus: you'll be happy to hear that we figured out the gitea 1.8.0 behavior change and have fixed it | 15:37 |
| mordred | corvus: so https://review.opendev.org/#/c/655522 is now ready to land | 15:37 |
| *** lpetrut has quit IRC | 15:38 | |
| mordred | corvus: (that also led to a PR to gitea docs which has been landed) | 15:39 |
| *** ginopc has quit IRC | 15:39 | |
| *** david-lyle has joined #openstack-infra | 15:39 | |
| *** dklyle has quit IRC | 15:40 | |
| corvus | mordred: oh awesome! | 15:40 |
| *** david-lyle is now known as dklyle | 15:41 | |
| corvus | mordred: related: my gitea patch for refs has landed upstream: https://github.com/go-gitea/gitea/pull/6758 | 15:42 |
| *** lucasagomes has quit IRC | 15:43 | |
| corvus | mordred: so maybe we should land the 1.8.0 patch, then maybe look into running master? | 15:43 |
| *** lucasagomes has joined #openstack-infra | 15:43 | |
| *** lucasagomes has quit IRC | 15:43 | |
| *** lmiccini has quit IRC | 15:43 | |
| *** lucasagomes has joined #openstack-infra | 15:44 | |
| mordred | corvus: yes to 1.8.0 ... master seems maybe scary - but also maybe not super horrible since we wouldn't build new images if they don't pass the docker-compose test | 15:44 |
| mordred | it's possible template updates could cause our UI to regress unexpectedly | 15:44 |
| *** lmiccini has joined #openstack-infra | 15:44 | |
| mordred | but that probably won't happen *frequently* | 15:44 |
| openstackgerrit | Sean McGinnis proposed openstack/openstack-zuul-jobs master: Fix typo in tempest-dsvm-full-hdfs https://review.opendev.org/660507 | 15:45 |
| corvus | mordred: or we could just pull the refs patch into our 1.8.0 branch | 15:45 |
| corvus | mordred: mostly, i didn't want to do something that would interact badly with upstream, but since they accepted it, i think adding it to our local fork until 1.9.0 would be fine | 15:45 |
| mordred | corvus: yeah. or even pull master into our 1.8.0 branch as a one-time sync and then move back to tags when they make them | 15:46 |
| mordred | but yeah - I'd love to go ahead and start running your refs patch so we can start on replicating refs | 15:46 |
| mordred | corvus: we could see if the gitea folks would be interested in zuul running system-config-run-gitea on gitea PRs :) | 15:47 |
| corvus | ok i'll put that on the list; meanwhile, 655522+3 | 15:47 |
| mordred | along with running master | 15:47 |
| *** kjackal has joined #openstack-infra | 15:47 | |
| corvus | er, i'll put "refs patch" on the list for later this week | 15:48 |
| corvus | speaking of talking to the gitea folks, i haven't gotten a response to my email to lunny | 15:48 |
| *** sarob has quit IRC | 15:49 | |
| mordred | darn | 15:49 |
| corvus | may have to try wechat or something | 15:49 |
| mordred | yeah | 15:49 |
| openstackgerrit | Andreas Jaeger proposed openstack/openstack-zuul-jobs master: Fix confusing statement in tempest-dsvm-full-hdfs https://review.opendev.org/660507 | 15:49 |
| mordred | oh - just had a thought - should we redo the metadata for the Zuul app on github to call it "OpenDev Zuul" ? | 15:50 |
| corvus | yes! | 15:50 |
| *** ramishra has quit IRC | 15:51 | |
| mordred | I'm going to do that ... although I think we might also want to create a new app, since the URL for people to use to add the app to their things is "https://github.com/apps/openstack-zuul" - but we can handle that later | 15:51 |
| mordred | corvus: we can upload a logo - should we upload the zuul logo or the opendev logo? I think maybe the opendev logo | 15:54 |
| *** dklyle has quit IRC | 15:55 | |
| *** david-lyle has joined #openstack-infra | 15:55 | |
| *** iurygregory has quit IRC | 15:55 | |
| *** Lucas_Gray has quit IRC | 15:55 | |
| corvus | mordred: ++opendev | 15:55 |
| mordred | done | 15:56 |
| mordred | corvus: oh! that makes the url https://github.com/apps/opendev-zuul ... neat | 15:56 |
| corvus | that looks lovely | 15:58 |
| *** derekh has quit IRC | 15:59 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Update references to OpenDev Zuul https://review.opendev.org/660771 | 16:00 |
| *** ykarel is now known as ykarel|away | 16:01 | |
| mordred | corvus: https://github.com/ansible/ansible/pull/56453#issuecomment-492731316 | 16:01 |
| *** jpena is now known as jpena|off | 16:02 | |
| mordred | infra-root: it still shows "openstack-infra" as the owner of the app - should we maybe make a new gh org, "opendev-zuul", and transfer the App into that org? (we can't make an opendev one because there's that dude whose name is opendev) | 16:04 |
| corvus | mordred: or "opendev-org" or "opendevorg" (ala dockerhub?) | 16:05 |
| *** jtomasek has quit IRC | 16:05 | |
| mordred | ah - yeah - that's smarter | 16:05 |
| *** hwoarang has quit IRC | 16:07 | |
| *** hwoarang has joined #openstack-infra | 16:09 | |
| *** tesseract has quit IRC | 16:10 | |
| *** jbadiapa has quit IRC | 16:10 | |
| *** igordc has joined #openstack-infra | 16:12 | |
| *** tesseract has joined #openstack-infra | 16:14 | |
| *** tesseract has quit IRC | 16:14 | |
| *** ykarel|away has quit IRC | 16:15 | |
| *** dklyle has joined #openstack-infra | 16:15 | |
| *** david-lyle has quit IRC | 16:15 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Update references to OpenDev Zuul https://review.opendev.org/660771 | 16:16 |
| mordred | corvus: done and updated docs ^^ | 16:16 |
| mordred | (I had already created the opendevorg org apparently. I have now added openstackadmin to it as owner) | 16:17 |
| *** rpittau is now known as rpittau|afk | 16:18 | |
| mordred | infra-root: while I'm doing github things - should we rename the openstackadmin account to opendevadmin? | 16:19 |
| corvus | mordred: it seems like we're probably going to have less and less openstacky stuff over time, so i lean toward yes, but we probably want more input on that | 16:20 |
| *** jpena|off is now known as jpena | 16:22 | |
| AJaeger | mriedem: the index.rst of https://developer.openstack.org/api-ref/compute/ was last updated on 26th of March - but published on the 7th of May the last time (view page info). Still a bit long ago, let's check logs... | 16:22 |
| AJaeger | mriedem: see http://zuul.openstack.org/build/2bf755e43b794f1e8df872e4b32452ec for last run | 16:23 |
| *** jbadiapa has joined #openstack-infra | 16:24 | |
| *** david-lyle has joined #openstack-infra | 16:24 | |
| AJaeger | mriedem: I think https://review.opendev.org/659975 and https://review.opendev.org/659976 will fix it - need config-core review... | 16:24 |
| *** kopecmartin is now known as kopecmartin|off | 16:25 | |
| *** dklyle has quit IRC | 16:25 | |
| mordred | corvus: agree | 16:26 |
| mriedem | AJaeger: ok thanks | 16:26 |
| mordred | corvus: also - I could see the openstack project wanting an account like openstackadmin to manage replication things and whatnot | 16:27 |
| clarkb | corvus: we also think we fixedthe limestone docker problems | 16:33 |
| clarkb | firewall neededport 5001 open toi | 16:33 |
| * clarkb goes back to sitting on river | 16:33 | |
| mordred | clarkb: before you go sit on the river - any opinions on renaming openstackadmin to opendevadmin in github? | 16:34 |
| clarkb | can you rename an account? or would be making a new one? | 16:35 |
| clarkb | making a new one seems fine just dont forget to make it owner on all the things | 16:35 |
| corvus | clarkb: huh? i thought we did that? | 16:36 |
| corvus | clarkb: https://review.opendev.org/655744 | 16:37 |
| corvus | oh | 16:37 |
| corvus | 5001 | 16:37 |
| corvus | derp | 16:37 |
| corvus | clarkb: great news! :) | 16:37 |
| mordred | clarkb: you can rename it | 16:37 |
| corvus | (how in the world did we all miss that the first time :) | 16:38 |
| *** udesale has quit IRC | 16:38 | |
| mordred | clarkb: and since it doesn't have repos directly under it, as I read it it should have no impact | 16:38 |
| *** mattw4 has joined #openstack-infra | 16:39 | |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Pagure driver - https://pagure.io/pagure/ https://review.opendev.org/604404 | 16:40 |
| *** david-lyle has quit IRC | 16:40 | |
| *** udesale has joined #openstack-infra | 16:43 | |
| prometheanfire | iirc logs are not kept forever, is there a way I can save this without just coppying the text? | 16:44 |
| prometheanfire | http://logs.openstack.org/54/660254/2/check/cross-nova-functional/a0617a9/testr_results.html.gz | 16:44 |
| prometheanfire | want to submit it to fasteners upstream | 16:45 |
| Shrews | wget http://logs.openstack.org/54/660254/2/check/cross-nova-functional/a0617a9/testr_results.html.gz | 16:46 |
| Shrews | ?? | 16:46 |
| prometheanfire | I meant still hosted and all | 16:46 |
| prometheanfire | browsable, linkable | 16:46 |
| *** dklyle has joined #openstack-infra | 16:48 | |
| *** Lucas_Gray has joined #openstack-infra | 16:48 | |
| *** e0ne has quit IRC | 16:51 | |
| *** diablo_rojo has quit IRC | 16:51 | |
| ricolin | HI all, I'm facing a issue when I try to add a test job to self-healing-sig repo. For some reason, I can't find new directory which I added it that patch http://logs.openstack.org/65/651965/14/check/self-healing-devstack-base/f5670fc/job-output.txt.gz#_2019-05-03_20_25_40_081727 Anyone can guide me on what I doing wrong here? https://review.opendev.org/#/c/651965/14/playbooks/devstack/functional/run.yaml | 16:51 |
| *** Swami has joined #openstack-infra | 16:53 | |
| mordred | ricolin: I can't see anything, but then I've paged most of devstack-gate out of my brain ... AJaeger, do you still know how that works enough to spot an error? | 16:55 |
| mordred | prometheanfire: no, we don't have any mechanism for that at the moment, although it's an interesting use case | 16:56 |
| openstackgerrit | Merged zuul/zuul master: Install latest git-review from PyPI in quickstart https://review.opendev.org/659674 | 16:57 |
| openstackgerrit | Merged opendev/system-config master: Use local fork of gitea and upgrade to 1.8.0 https://review.opendev.org/655522 | 16:58 |
| prometheanfire | mordred: yarp, I did make https://github.com/harlowja/fasteners/issues/36 in the mean time | 17:00 |
| openstackgerrit | Merged zuul/zuul-jobs master: Delete files in dest that don't exist https://review.opendev.org/648815 | 17:01 |
| *** bobh has joined #openstack-infra | 17:06 | |
| *** hjensas has joined #openstack-infra | 17:06 | |
| notmyname | ssl for opendev.org down for everyone or just me? | 17:06 |
| * hjensas has ssl issues with opendev.org as well. | 17:07 | |
| bnemec | notmyname: I'm also here wondering about that | 17:07 |
| notmyname | both git and a browser aren't working for me | 17:08 |
| bnemec | Likewise | 17:09 |
| bnemec | It was working for me 10 minutes ago, FWIW. | 17:10 |
| *** electrofelix has quit IRC | 17:10 | |
| pabelanger | https://review.opendev.org/655522/ | 17:10 |
| pabelanger | just merged, 11mins ago | 17:10 |
| pabelanger | maybe related | 17:10 |
| pabelanger | infra-root: ^ opendev.org seems down | 17:10 |
| corvus | neat, 522 seems very likely to be related | 17:11 |
| corvus | adb916bc6863 opendevorg/gitea:latest "/usr/bin/entrypoint…" 11 minutes ago Restarting (1) 54 seconds ago giteadocker_gitea-web_1 | 17:11 |
| *** udesale has quit IRC | 17:12 | |
| corvus | 2019/05/22 17:11:40 ...s/setting/setting.go:738:NewContext() [F] error saving generating JWT secret to custom config: open /custom/conf/app.ini: permission denied | 17:12 |
| *** Lucas_Gray has quit IRC | 17:12 | |
| corvus | that's very strange since that patch should have taken care of that | 17:12 |
| mordred | that's the thing we fixed in 522 | 17:12 |
| mordred | yeah | 17:12 |
| *** ricolin has quit IRC | 17:13 | |
| corvus | i do not see the oauth2 thing in the app.ini on disk | 17:13 |
| mordred | I don't either | 17:13 |
| corvus | i think we should manually edit that for now, then figure out what went wrong | 17:14 |
| mordred | I agree | 17:14 |
| *** chandankumar is now known as raukadah | 17:14 | |
| mordred | wanna divy up hosts? | 17:14 |
| corvus | mordred: sure, i'll take 1-4, you take 5,7,8 | 17:14 |
| mordred | kk | 17:14 |
| corvus | 6 is out | 17:14 |
| *** bhavikdbavishi has joined #openstack-infra | 17:15 | |
| mordred | corvus: 5-8 done | 17:16 |
| corvus | mordred: once edited, the auto restart should take care of things | 17:16 |
| mordred | yeah | 17:16 |
| corvus | done | 17:16 |
| mordred | now - why did this break? | 17:17 |
| *** jpena is now known as jpena|off | 17:17 | |
| mordred | corvus: I'm still getting site down | 17:17 |
| corvus | there we go | 17:18 |
| mordred | am I just too impatient? | 17:18 |
| mordred | yup | 17:18 |
| corvus | mordred: gitea01 still took 3 minutes to start up... i'm not sure what it's doing here: | 17:19 |
| corvus | 2019/05/22 17:15:46 models/repo.go:136:NewRepoContext() [I] Git Version: 2.20.1 | 17:19 |
| corvus | 2019/05/22 17:18:20 routers/init.go:37:checkRunMode() [I] Run Mode: Production | 17:19 |
| corvus | maybe we should look into that sometime | 17:19 |
| mordred | me either | 17:19 |
| mordred | ++ | 17:19 |
| corvus | but i think we're good now | 17:19 |
| corvus | bnemec, notmyname, hjensas: ^ thanks! | 17:19 |
| notmyname | I'm still getting an error with `git fetch orgiin` | 17:19 |
| notmyname | $ git fetch origin | 17:19 |
| notmyname | fatal: unable to access 'https://opendev.org/openstack/swift/': LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to opendev.org:443 | 17:19 |
| hjensas | I was able to clone a repo, but then when 'git remote update' I get: fatal: unable to access 'https://opendev.org/openstack/python-tripleoclient.git/': OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to opendev.org:443 | 17:20 |
| *** bobh has quit IRC | 17:20 | |
| hjensas | oh, on retry it succeded. | 17:20 |
| notmyname | just started working here too | 17:21 |
| mordred | corvus: it's a race that will be fixed by running playbooks with zuul | 17:21 |
| corvus | my guess is the load balancer was putting backends back in too early, since they were accepting connections but not responding... | 17:21 |
| mordred | corvus: system-config on bridge is not updated yet | 17:21 |
| mordred | but the docker images were published | 17:21 |
| mordred | so the cron-job ansible run told docker compose to pull and restart - but the ansible wasn't ready for it yet | 17:22 |
| corvus | mordred: ah! this is a tricky time for us. | 17:22 |
| openstackgerrit | Merged openstack/cookiecutter master: Fix outdated references to git.openstack.org/cgit/ https://review.opendev.org/657444 | 17:22 |
| mordred | corvus: yah | 17:22 |
| corvus | mordred: in some cases (like this) we could make config/image changes separately. obviously that's fragile, but could be something to keep in mind until we finish the transition | 17:23 |
| *** lucasagomes has quit IRC | 17:23 | |
| mordred | yeah. we could also maybe go ahead and start running the services that are split out from zuul and remove them from teh cron, now that we've landed the first split-out patch | 17:24 |
| corvus | yeah, i think we need the python version of gitea project creation for that, right? | 17:25 |
| *** kskels has joined #openstack-infra | 17:25 | |
| *** bdodd has quit IRC | 17:25 | |
| corvus | or, at least, that will allow us to do the most sensible and least hacky version of that. | 17:25 |
| mordred | oh - yeah | 17:25 |
| kskels | hi clarkb an infra team - is there a way to re-trigger/recheck zuul post job? or would it be waiting on another merge coming in? | 17:26 |
| mordred | corvus: I was going to work on that after upgrading status.o.o off of trusty | 17:26 |
| *** jtomasek has joined #openstack-infra | 17:27 | |
| corvus | kskels: if it's not too much of a burden to wait for another commit, that would be best. but if it's important and there isn't going to be another merge soon, we can retrigger (it's a manual process) | 17:27 |
| kskels | it would likely be a wait.. if you can re-trigger without too much work then I can test and see if adding "write" permissions to gerrit user worked.. then I could move on to apply the same for all other like 14 airship projects.. http://zuul.openstack.org/builds?job_name=treasuremap-upload-git-mirror | 17:29 |
| kskels | if it's too much trouble.. I'll wait. Thank you!! | 17:29 |
| kskels | https://review.opendev.org/#/c/660780/ | 17:30 |
| kskels | is the PS | 17:30 |
| corvus | kskels: oh, i'm sorry i thought maybe there was an infrastructure problem or something... we're trying to get out of the business of being responsible for github replication, so retriggering jobs for that doesn't seem like something we should be doing | 17:32 |
| corvus | (if we were going to be responsible for github replication, it would be better for us to have just kept the old system with gerrit) | 17:32 |
| kskels | no worries, thank you! | 17:33 |
| *** eharney has joined #openstack-infra | 17:33 | |
| *** kskels has left #openstack-infra | 17:34 | |
| *** Swami has quit IRC | 17:35 | |
| *** samueldmq has joined #openstack-infra | 17:50 | |
| *** bobh has joined #openstack-infra | 17:57 | |
| *** yboaron_ has quit IRC | 18:00 | |
| *** yboaron_ has joined #openstack-infra | 18:01 | |
| *** nhicher has joined #openstack-infra | 18:04 | |
| dtroyer | We're seeing an error at https://opendev.org/starlingx/integ/ via browser but not git clone, etc: "template: repo/view_list:82:38: executing "repo/view_list" at <$commit.Summary>: invalid value; expected string" I have not found this in any other repos so far…still checking | 18:10 |
| *** Goneri has quit IRC | 18:11 | |
| *** bdodd has joined #openstack-infra | 18:11 | |
| corvus | dtroyer: fascinating mordred ^ | 18:11 |
| mordred | uh | 18:12 |
| corvus | (i'm digging in to it) | 18:12 |
| mordred | I agree - that does not look good | 18:12 |
| corvus | dtroyer: any chance you can confirm whether this worked before, let's say, 2 hours ago? :) | 18:13 |
| dtroyer | I've asked in #starlingx, not heard back yet... | 18:13 |
| corvus | view_list.tmpl does not seem to be one of the ones we have modified locally | 18:13 |
| corvus | i don't see anything weird about the first line of the 50 most recent commit messages in that repo | 18:16 |
| dtroyer | corvus: it was first reported about 20 minutes ago…we're not sure when it was last known to work... | 18:16 |
| *** jtomasek has quit IRC | 18:23 | |
| *** jtomasek_ has joined #openstack-infra | 18:23 | |
| corvus | the logs don't seem to have any info other than there was a 500 error for that url | 18:25 |
| logan- | would someone mind enqueueing https://review.opendev.org/#/c/660688/ in gate please? it got stuck for over an hour earlier in the gate trying to clone from opendev.org, and eventually timed out, and then i was waiting for a recheck to go through the check queue and one of the tests has been waiting in check for a retry node for over an hour because of the opendev.org issues a bit ago. | 18:26 |
| *** bobh has quit IRC | 18:28 | |
| corvus | dtroyer, mordred: i can reproduce that with a local gitea, will try to find more info | 18:28 |
| mordred | corvus: oh good! | 18:29 |
| *** tosky has quit IRC | 18:30 | |
| *** tosky has joined #openstack-infra | 18:30 | |
| openstackgerrit | Matt McEuen proposed openstack/project-config master: Add granular ACLs for Airship Projects https://review.opendev.org/660461 | 18:33 |
| *** gfidente is now known as gfidente|afk | 18:35 | |
| corvus | dtroyer, mordred: i'm rebuilding gitea with more debug info and will pick this back up after lunch | 18:36 |
| dtroyer | corvus: ok, thank you | 18:36 |
| *** dpawlik has joined #openstack-infra | 18:36 | |
| *** jamesmcarthur has quit IRC | 18:38 | |
| *** bhavikdbavishi has quit IRC | 18:41 | |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Pagure driver - https://pagure.io/pagure/ https://review.opendev.org/604404 | 18:43 |
| *** slaweq has quit IRC | 18:44 | |
| *** sshnaidm is now known as sshnaidm|afk | 18:45 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Log github requests with annotated events https://review.opendev.org/660800 | 18:57 |
| fungi | dtroyer: corvus: mordred: that looks similar to this ref which turned up broken in a release audit but for which i wasn't able to find a cause: https://opendev.org/openstack/python-freezerclient/commit/cc9bbeca28029bdac53aada21e1c29dfdbebe601 | 18:57 |
| fungi | it's the tag ref corresponding to the most recent successful build at https://zuul.opendev.org/t/openstack/builds?job_name=release-openstack-python&project=openstack%2Fpython-freezerclient | 18:58 |
| fungi | maybe the cause for both is related | 18:58 |
| fungi | granted it's a different function and a different variable, but a similar resulting error | 18:59 |
| *** smarcet has joined #openstack-infra | 19:01 | |
| mordred | fungi: when I look at that page in gitea - it shows: tags/1.5.1^0 | 19:01 |
| mordred | which makes me think maybe a null got appended to the tag? | 19:01 |
| *** ramishra has joined #openstack-infra | 19:02 | |
| mordred | oh - hrm. maybe that's just a gitea display weirdness - shows up here too: | 19:02 |
| mordred | https://opendev.org/zuul/zuul/commit/88af9a0f9fc7c7eff2883ddeec98eea73ad12572 | 19:02 |
| openstackgerrit | Adam Coldrick proposed opendev/system-config master: Update key for SotK https://review.opendev.org/660805 | 19:09 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Log github requests with annotated events https://review.opendev.org/660800 | 19:12 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Annotate logs around build completion and cancellation https://review.opendev.org/660806 | 19:12 |
| *** raissa has joined #openstack-infra | 19:12 | |
| openstackgerrit | Merged opendev/system-config master: Update references to OpenDev Zuul https://review.opendev.org/660771 | 19:14 |
| *** e0ne has joined #openstack-infra | 19:15 | |
| *** EmilienM is now known as EmilienM|bbl | 19:15 | |
| *** EmilienM|bbl is now known as EmilienM | 19:16 | |
| *** bobh has joined #openstack-infra | 19:16 | |
| *** sarob has joined #openstack-infra | 19:17 | |
| *** flaper87 has joined #openstack-infra | 19:21 | |
| fungi | any other infra-root available for a quick ssh key change in https://review.opendev.org/660805 ? | 19:21 |
| Shrews | fungi: +3d | 19:22 |
| fungi | thanks! | 19:22 |
| *** smarcet has quit IRC | 19:23 | |
| *** jtomasek_ has quit IRC | 19:24 | |
| *** smarcet has joined #openstack-infra | 19:25 | |
| *** jtomasek has joined #openstack-infra | 19:25 | |
| *** quiquell has joined #openstack-infra | 19:26 | |
| *** quiquell has quit IRC | 19:26 | |
| *** slaweq has joined #openstack-infra | 19:27 | |
| *** ramishra has quit IRC | 19:27 | |
| openstackgerrit | Douglas Mendizábal proposed openstack/project-config master: End Project Gating https://review.opendev.org/660808 | 19:28 |
| *** sarob has quit IRC | 19:32 | |
| *** smarcet has quit IRC | 19:33 | |
| *** Goneri has joined #openstack-infra | 19:37 | |
| *** jamesmcarthur has joined #openstack-infra | 19:39 | |
| *** jtomasek has quit IRC | 19:42 | |
| *** jtomasek_ has joined #openstack-infra | 19:43 | |
| *** smarcet has joined #openstack-infra | 19:47 | |
| AJaeger | mordred: ricolin is touching legacy devstack-gate hooks - I swapped those out as well but will quickly look. Rewriting from scratch is my recommendation ;) | 19:50 |
| AJaeger | config-core, the api jobs are broken due to sphinx/python 3, I think https://review.opendev.org/659975 and https://review.opendev.org/659976 will fix it - could you review, please? | 19:51 |
| *** jcoufal has joined #openstack-infra | 20:00 | |
| *** jcoufal has quit IRC | 20:00 | |
| *** jcoufal has joined #openstack-infra | 20:01 | |
| *** jtomasek_ has quit IRC | 20:02 | |
| rpioso | I'm proposing new entries in driverlog -- https://review.opendev.org/#/c/660534 | 20:03 |
| *** jtomasek has joined #openstack-infra | 20:03 | |
| rpioso | How can I determine the Ironic CI's username/ID, success pattern, and failure pattern? | 20:04 |
| openstackgerrit | Douglas Mendizábal proposed openstack/project-config master: End Project Gating https://review.opendev.org/660808 | 20:04 |
| *** ralonsoh has quit IRC | 20:07 | |
| *** liuyulong has quit IRC | 20:11 | |
| fungi | rpioso: are you the one running the "ironic ci" you describe? the username would be whatever the username is for the gerrit account it's using to leave comments | 20:15 |
| fungi | or so i assume. we don't manage driverlog (i think the mirantis folks running stackalytics do?) | 20:15 |
| logan- | https://review.opendev.org/#/c/660688/ just failed with RETRY_LIMIT on a job that has been retrying for hours (see my message above). i was able to catch the console on the last one while it was running and it seems like the node just died in the middle of it http://paste.openstack.org/raw/751956/ | 20:16 |
| *** bobh has quit IRC | 20:16 | |
| fungi | looks like it was in the middle of installing python packages from remote git sources into containers? | 20:18 |
| logan- | not installing anything, just cloning repos there | 20:18 |
| logan- | i just rechecked it again, but it seems like maybe something is still not quite right with the opendev git | 20:19 |
| corvus | logan-: a problem with the opendev git servers would not cause a zuul worker node to go offline | 20:20 |
| *** pcaruana has quit IRC | 20:20 | |
| *** michael-beaver has joined #openstack-infra | 20:21 | |
| corvus | logan-: i don't suppose you have the top of the build log handy? | 20:21 |
| logan- | corvus: sadly no :( i accidentally clicked a link in there and the back button does not restore it :( | 20:22 |
| corvus | it may be useful for someone to check the logs on ze07 and see which node provider that was in | 20:22 |
| logan- | rax-ord or rax-iad I cannot remember which | 20:23 |
| logan- | related question though, does nodepool schedule the retries to the same cloud? | 20:23 |
| logan- | that IP traceroutes to rax-ord btw | 20:24 |
| corvus | logan-: generally no (unless it depends on a paused job; that's the only time zuul expresses a provider preference to nodepool) | 20:25 |
| *** pkopec|afk has quit IRC | 20:25 | |
| logan- | gotcha | 20:25 |
| logan- | the executor chart on http://grafana.openstack.org/d/T6vSHcSik/zuul-status?orgId=1 looks weird | 20:25 |
| logan- | nodepool clouds like http://grafana.openstack.org/d/WFOSH5Siz/nodepool-limestone?orgId=1 show a similar drop to almost 0 used nodes around the same time | 20:26 |
| corvus | that sounds like a zookeeper connection drop, which usually means a memory problem | 20:27 |
| corvus | the scheduler was last restarted on may 16 at commit ee4b6b1 | 20:28 |
| openstackgerrit | Dirk Mueller proposed openstack/openstack-zuul-jobs master: Switch to using TOX_CONSTRAINTS_FILE https://review.opendev.org/660823 | 20:28 |
| corvus | infra-root: is any other infra-root around to deal with zuul while i continue to deal with gitea? | 20:28 |
| openstackgerrit | Matt McEuen proposed openstack/project-config master: Add granular ACLs for Airship Projects https://review.opendev.org/660461 | 20:28 |
| Shrews | corvus: just a scheduler restart? i can deal with that | 20:30 |
| Shrews | lemme check the mem use first | 20:30 |
| logan- | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=392&rra_id=all | 20:30 |
| corvus | Shrews: thx | 20:30 |
| openstackgerrit | Dirk Mueller proposed zuul/zuul-jobs master: Switch to TOX_CONSTRAINTS_FILE https://review.opendev.org/657886 | 20:31 |
| Shrews | him, seems likely: http://cacti.openstack.org/cacti/graph.php?action=zoom&local_graph_id=64792&rra_id=0&view_type=tree&graph_start=1558470669&graph_end=1558557069 | 20:31 |
| openstackgerrit | Matt McEuen proposed openstack/project-config master: Add granular ACLs for Airship Projects https://review.opendev.org/660461 | 20:33 |
| Shrews | corvus: any need to wait on anything after stopping the scheduler? or is init.d/zuul-scheduler restart enough? | 20:34 |
| *** whoami-rajat has quit IRC | 20:34 | |
| *** e0ne has quit IRC | 20:34 | |
| corvus | Shrews: nope, but there's a bug in pid handling for zuul web, so you'll need to manually remove its pidfile and restart it | 20:35 |
| Shrews | k | 20:35 |
| corvus | (and its also a bug that it needs to be restarted) | 20:35 |
| *** jcoufal has quit IRC | 20:36 | |
| * mordred has eaten lunch and is back ... anything I need to jump in on? | 20:36 | |
| Shrews | him, zuul-scheduler does not seem to want to stop | 20:38 |
| *** mattw4 has quit IRC | 20:39 | |
| *** e0ne has joined #openstack-infra | 20:40 | |
| *** mattw4 has joined #openstack-infra | 20:40 | |
| rpioso | fungi: No, I don’t run it. However, I am a beneficiary of it. | 20:40 |
| *** sarob has joined #openstack-infra | 20:40 | |
| fungi | yeah, sorry, stepped away for a moment but back now | 20:40 |
| fungi | so we think there was a zk connection drop in addition to whatever is wrong with that gitea git ref? | 20:41 |
| Shrews | ok, there it goes | 20:41 |
| *** e0ne has quit IRC | 20:41 | |
| logan- | appears to be a zuul scheduler memory leak fungi http://cacti.openstack.org/cacti/graph.php?action=zoom&local_graph_id=64792&rra_id=0&view_type=tree&graph_start=1558470669&graph_end=1558557069 | 20:42 |
| fungi | oh, huh yep that's a dramatic increase today | 20:42 |
| *** smarcet has quit IRC | 20:43 | |
| Shrews | fungi: yeah, restarted scheduler and web | 20:43 |
| fungi | we've definitely seem memory pressure cause sudden zk connectivity issues | 20:43 |
| Shrews | fungi: anything else too restart? | 20:43 |
| Shrews | s/too/to/ | 20:43 |
| fungi | i think that'll be all you need to restart | 20:44 |
| Shrews | i guess the fingergw is ok | 20:44 |
| fungi | and looks like this was the first restart in ~6 days? so interesting that memory use just skyrocketed in the last 24 hours | 20:44 |
| Shrews | yeah | 20:44 |
| pabelanger | Shrews: yah, usually save queues, restart, then reenqueue gate, then check | 20:44 |
| Shrews | pabelanger: ah, knew i forgot something (the queues) | 20:45 |
| pabelanger | oh | 20:45 |
| Shrews | oopsie | 20:45 |
| pabelanger | Hmm, I think they are still on disk | 20:45 |
| pabelanger | we have a crontab still? | 20:45 |
| fungi | yeah, there is a periodic dump to /var i think, looking | 20:45 |
| fungi | /var/lib/zuul/backup/openstack-zuul-tenant_status_*.json | 20:46 |
| pabelanger | Shrews: https://docs.openstack.org/infra/system-config/zuul.html#restarts gives some good info too | 20:47 |
| pabelanger | but we don't seem to have docs about crontab status that fungi linked | 20:47 |
| fungi | timestamp on /var/run/zuul/scheduler.pid is 20:41z so we want the last dump before that | 20:47 |
| fungi | which i think is either /var/lib/zuul/backup/openstack-zuul-tenant_status_1558557601.json or /var/lib/zuul/backup/openstack-zuul-tenant_status_1558557661.json | 20:48 |
| fungi | nope. earlier | 20:49 |
| fungi | /var/lib/zuul/backup/openstack-zuul-tenant_status_1558557361.json is the last one with a nontrivial file size | 20:49 |
| Shrews | fungi: how do we use that? | 20:50 |
| fungi | great question... shell history says `python zuul-changes-static.py file:///var/lib/zuul/backup/openstack-zuul-tenant_status_1558557361.json openstack gate >gate.sh` et cetera | 20:51 |
| fungi | s/gate/check/g to export the check pipeline similarly | 20:51 |
| *** dpawlik has quit IRC | 20:51 | |
| fungi | those should give you gate.sh and check.sh scripts you can run to reenqueue whatever was running at the time it was stopped (or moments before anyway) | 20:52 |
| Shrews | that script does not appear to be a thing | 20:52 |
| Shrews | at least not in /opt/zuul/tools | 20:53 |
| fungi | i think it'll be in /opt/zuul/tools/ | 20:53 |
| fungi | hrm | 20:53 |
| fungi | possible that /opt/zuul/tools/zuul-changes.py can handle file:// urls now | 20:54 |
| fungi | (or always could?) | 20:54 |
| pabelanger | yah, always could | 20:55 |
| Shrews | hrm, don't think it handles files | 20:58 |
| corvus | i think there's a modified version of that in my homedir | 20:59 |
| fungi | urllib2.URLError: <urlopen error [Errno 20] Not a directory: '/var/lib/zuul/backup/openstack-zuul-tenant_status_1558557361.json/api/info'> | 21:00 |
| fungi | yeah, seems the vanilla one indeed does not | 21:00 |
| corvus | dmsimard has a year-wip patch to support files https://review.opendev.org/536622 | 21:01 |
| dmsimard | :/ | 21:02 |
| fungi | the reason for the -1 there looks incorrect, at least without some alteration to the script | 21:02 |
| corvus | i don't know what needs to be done, but the end goal should be that "file://" is supported | 21:03 |
| Shrews | that seems to work | 21:03 |
| dmsimard | it was the distinction between --file and --url in the patch that was problematic because really file:// is an url | 21:03 |
| fungi | ahh | 21:03 |
| dmsimard | so we could do away with the args altogether and always use urls | 21:03 |
| corvus | yes that | 21:03 |
| fungi | got it | 21:04 |
| Shrews | fungi: using that, i have check.sh and gate.sh scripts from openstack-zuul-tenant_status_1558557361.json | 21:04 |
| Shrews | re-enqueuing using those now... | 21:05 |
| *** smarcet has joined #openstack-infra | 21:11 | |
| corvus | dtroyer, mordred: re integ and gitea -- i have more information: when gitea is displaying the commit message summary of each file in the root of the repo directory, it collects a list of git tree entries and associated commits. for some of the entries in the root of integ, the commit object is nil, and that is not supposed to happen. i do not (yet) know why that is the case. i have also found | 21:11 |
| corvus | that our gitea is perfectly happy to display the commit information for the troublesome objects in other contexts (for instance, "doc/" is one of the problematic entries on the first page, but https://opendev.org/starlingx/integ/src/branch/master/doc works fine). i have also found that while i can reproduce this with a local build of our gitea image, i can not reproduce it with try.gitea.io which | 21:11 |
| corvus | is running upstream master: https://try.gitea.io/corvus2/integ | 21:11 |
| dtroyer | corvus: do we need to do some repo surgery to clean it up? | 21:12 |
| corvus | dtroyer: i have no information at this point to suggest that there is a problem with the repo itself | 21:13 |
| dtroyer | coreycb: ok, thanks | 21:13 |
| corvus | dtroyer, mordred: given all that, i am somewhat inclined to build an opendev gitea image based on master and see if it works, and if it does, upgrade opendev to that (since that is the most forward looking thing). alternatively, i can attempt to find the difference between our version of 1.8.0 and upstream master (perhaps there is a fix upstream for this bug, or, perhaps one of our local changes | 21:14 |
| corvus | on 1.8.0 has broken it) | 21:14 |
| mordred | corvus: I am in favor of the idea of the gitea image based on master since we were talking about doing that earlier today anyway | 21:14 |
| corvus | dtroyer: this is where it would be really helpful to know if it was working before today (as it would eliminate any changes between the last two local builds) | 21:14 |
| dtroyer | corvus: I know it was OK on Friday, I'll see if anyone can confirm for this morning | 21:15 |
| corvus | dtroyer: i think that may be enough | 21:15 |
| corvus | dtroyer: if it worked friday and not today, then the breaking change is between opendev-1.7.6 and opendev-1.8.0, so it does lend credence to the idea that something upstream broke and there is a fix in master | 21:16 |
| * mordred agrees with corvus | 21:17 | |
| corvus | found it | 21:17 |
| corvus | https://github.com/go-gitea/gitea/commit/6d2f0e555e60812f0e61be1fe1ef1c5e38a3acee | 21:17 |
| fungi | aha, so it's runaway template parsing on commit message content? | 21:18 |
| Shrews | check and gate jobs re-enqueued | 21:20 |
| Shrews | not sure if we lost any | 21:20 |
| fungi | Shrews: close enough, i'm sure | 21:20 |
| fungi | the error on https://opendev.org/openstack/python-freezerclient/commit/cc9bbeca28029bdac53aada21e1c29dfdbebe601 is also from "template" but has existed since migration, leading me to wonder if there's a similar bug in repo/diff/box | 21:21 |
| dtroyer | Shrews: FWIW the two I was watching have re-appeared | 21:21 |
| fungi | maybe for longer | 21:21 |
| Shrews | dtroyer: awesome | 21:21 |
| *** smarcet has quit IRC | 21:23 | |
| corvus | that commit alone is insufficient to fix things, so i'm trying a build of upstream master now | 21:24 |
| mordred | I still don't see any errors on that page | 21:24 |
| mordred | fungi: ^^ that link totally works fine for me with no errors | 21:24 |
| pabelanger | same | 21:25 |
| fungi | huh | 21:26 |
| *** jamesmcarthur has quit IRC | 21:26 | |
| fungi | mordred: wow! so it does. i still had this response cached in a browser tab after ttx pointed it out before the summit... template: repo/diff/box:101:80: executing "repo/diff/box" at <$.BeforeSourcePath>: invalid value; expected string | 21:27 |
| fungi | and force refreshing that tab continues to return that error | 21:27 |
| mordred | fascinating | 21:28 |
| fungi | but opening it in a browser on another computer loads fine... | 21:28 |
| fungi | plugging that url into a new tab also loads fine | 21:28 |
| fungi | following the "cc9bbec" link in the change column for the most recent successful build listed at https://zuul.opendev.org/t/openstack/builds?job_name=release-openstack-python&project=openstack%2Fpython-freezerclient is also working now | 21:29 |
| fungi | but was how the error was previously observed | 21:29 |
| fungi | so maybe it was a bug in opendev-1.7.6 which was fixed in opendev-1.8.0? | 21:30 |
| fungi | and somethnig about the persistent javascriptness of gitea is causing that old tab to still return the error message | 21:31 |
| fungi | yeah, after loading the same url in a new tab successfully, reloading the old tab now gets me the correct content... weird | 21:32 |
| mordred | fungi: it's best not to ask questions - the computers don't want you to know their secrets, and they'll stop at nothing to keep it that way | 21:32 |
| *** slaweq has quit IRC | 21:35 | |
| aspiers | interesting, stackalytics broke on nova https://www.stackalytics.com/?metric=commits&release=train&project_type=all&module=nova | 21:36 |
| aspiers | who's the Mirantis contact? | 21:36 |
| aspiers | or whoever maintains it | 21:37 |
| fungi | folks have just been e-mailing openstack-discuss and hoping someone sees it | 21:37 |
| fungi | they replied on the ml over the weekend that they'd finally done a redeploy/reindex so maybe that was what triggered it? could be worth replying to that message | 21:38 |
| aspiers | OK, thanks | 21:38 |
| aspiers | fungi: done | 21:39 |
| *** kjackal has quit IRC | 21:40 | |
| fungi | hah, i was looking for the most recent message in that thread to link for you... and it's yours! ;) | 21:40 |
| aspiers | XD | 21:40 |
| corvus | mordred, fungi, dtroyer: local build of upstream master seems to have the problem fixed. mordred: i've poked around a bit at various urls; do you have a more thorough procedure for determining necessary template updates? | 21:40 |
| aspiers | fungi: quickest list spammer in the west ... of Europe | 21:41 |
| *** mattw4 has quit IRC | 21:41 | |
| openstackgerrit | Merged opendev/system-config master: Update key for SotK https://review.opendev.org/660805 | 21:42 |
| *** mattw4 has joined #openstack-infra | 21:42 | |
| *** Goneri has quit IRC | 21:43 | |
| *** gfidente|afk has quit IRC | 21:43 | |
| *** jamesmcarthur has joined #openstack-infra | 21:45 | |
| corvus | by the way, we got "blame" support in gitea with the upgrade to 1.8.0 | 21:45 |
| aspiers | corvus: wooo, nice | 21:46 |
| corvus | eg: https://opendev.org/opendev/system-config/blame/branch/master/docker/gitea/Dockerfile | 21:46 |
| *** jtomasek has quit IRC | 21:47 | |
| *** mriedem has quit IRC | 21:48 | |
| ianw | sorry i'll be starting today a little later; is there anything i need to help with for the next 30 minutes or so? | 21:49 |
| *** rh-jelabarre has quit IRC | 21:50 | |
| aspiers | persia: have you seen https://github.com/google/git-appraise ? this seems quite like that thing you were proposing to me ages ago | 21:51 |
| aspiers | corvus: I miss equivalent functionality to the 't' shortcut in GitHub | 21:52 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Upgrade gitea to current master (pre-1.9.0) https://review.opendev.org/660844 | 21:52 |
| *** mriedem has joined #openstack-infra | 21:52 | |
| aspiers | I find that frequently useful | 21:52 |
| corvus | ianw: i think all fires are out | 21:53 |
| corvus | dtroyer, mordred, fungi: ^ gitea upgrade | 21:53 |
| corvus | aspiers: what's a "t"? | 21:53 |
| *** Goneri has joined #openstack-infra | 21:53 | |
| aspiers | corvus: try pressing it :) | 21:53 |
| corvus | pretend, for the sake of argument, i'm not a big github user | 21:53 |
| aspiers | https://github.com/openstack/nova for example | 21:53 |
| aspiers | substring path matching | 21:54 |
| corvus | aspiers: do you like hacking in go? i've found the gitea authors very accepting of new features :) | 21:54 |
| aspiers | sadly no go (haha) experience yet | 21:55 |
| aspiers | I'm interpreting "i'm not a big github user" as a boast cunningly designed to make me jealous | 21:55 |
| aspiers | one day maybe I'll be able to say that | 21:56 |
| *** Weifan has joined #openstack-infra | 21:56 | |
| mordred | corvus: +2 from me | 21:57 |
| dtroyer | corvus: thanks for a quick response! also, I look forward to our new 'git blame' overlord^H^H^H^H^H^H^H^Hfunctionality… | 21:58 |
| *** rlandy is now known as rlandy|bbl | 21:59 | |
| Weifan | Hi, I have a question about the github repo. | 22:01 |
| Weifan | Originally our repo was under openstack, and now it is moved to x. | 22:01 |
| Weifan | We decided that we would just keep the one on opendev, and stop updating github. | 22:01 |
| Weifan | However, we want to add a redirect/link on the original github repo to opendev. | 22:01 |
| Weifan | Is there any guidlines for it? | 22:01 |
| *** sreejithp has quit IRC | 22:02 | |
| fungi | we can transfer to openstack/whatever github repo to a different github org under your control | 22:03 |
| *** sreejithp has joined #openstack-infra | 22:03 | |
| *** Weifan has quit IRC | 22:03 | |
| fungi | github has no features i'm aware of for redirecting to somewhere other than github | 22:03 |
| openstackgerrit | Merged zuul/zuul master: Annotate node request processing with event id https://review.opendev.org/658889 | 22:04 |
| *** Weifan has joined #openstack-infra | 22:04 | |
| fungi | Weifan: not sure if you saw what i was saying before you disconnected | 22:04 |
| fungi | Weifan: we can transfer to openstack/whatever github repo to a different github org under your control | 22:04 |
| fungi | Weifan: github has no features i'm aware of for redirecting to somewhere other than github | 22:04 |
| Weifan | so how should we start the proccess? | 22:04 |
| corvus | we can certainly delete the x/ repo | 22:05 |
| *** rfarr__ has quit IRC | 22:05 | |
| corvus | i'm not sure if we could do something like merging a "retirement" commit to it though | 22:06 |
| Weifan | nah.. we want to keep the x | 22:06 |
| Weifan | we just want to retire the github repo | 22:06 |
| corvus | sorry, i meant delete the x/ from github | 22:06 |
| corvus | oh, er, we didn't create any x/ repos on github | 22:06 |
| Weifan | i think we people search for our plugin, they find github first, so we want to put link there | 22:07 |
| corvus | Weifan: we can delete the openstack/ version of the repo | 22:07 |
| fungi | yeah, that would be very simple | 22:07 |
| corvus | i believe that will happen for all of the repos which moved from openstack/ to x/ soon anyway | 22:07 |
| mordred | maybe this is a "archive the openstack/ version and update the description to point to the opendev.org url" | 22:07 |
| weshay | sorry to distrub, is the retry_limit issue known? http://zuul.openstack.org/builds?result=RETRY_LIMIT | 22:07 |
| mordred | so that people searching for something on github - or who are used to the old github mirror have a path to find the correct thing | 22:08 |
| corvus | mordred: my understanding is that one of the goals was to remove all repos from openstack/ which were not openstack | 22:08 |
| mordred | corvus: I'm guessing the gh archive feature probably isn't removal enough | 22:09 |
| corvus | mordred: i would assume that means no archived non-openstack repos... but i admit that's a grey area? | 22:09 |
| mordred | yeah. maybe we should experiment with what archiving a repo looks like | 22:09 |
| mordred | and see if providing an archive-with-pointer would be a good or a bad idea | 22:09 |
| Weifan | when are you planning to delete the repos? personally i feel transfer is better as we would have more control, but i am not sure how complicated the process is | 22:09 |
| corvus | mordred: yeah, from an optics pov, it seems like it may not reflect what openstack wants to project, as i understand it (as in, it would be a whole lot of retired repos unrelated to openstack) | 22:10 |
| *** slaweq has joined #openstack-infra | 22:11 | |
| corvus | Weifan: i don't think a date is set yet, but i think it should be done within a few months at the most. | 22:11 |
| fungi | yeah, the up-side to gh archiving is that it doesn't prevent us from transferring an org in case a project pops up months after asking for that | 22:11 |
| corvus | fungi: then hopefully mordred's idea will work out :) | 22:12 |
| Weifan | i am afraid that if people find that the repo is deleted without noticed or without update for months, they would assume the repo is no longer maintained | 22:12 |
| fungi | downside of course is that the repo still exists and just grows a yellow-on-brown banner in their webui indicating the repo is archived (based on screenshots i saw) | 22:12 |
| mordred | corvus, fungi: maybe transferring to somewhere like stackforge/ - then archiving - would achieve all of the goals - remove things from openstack/ namespace, still have the content show up in searches for peple, then have a pointer to the opendev repos | 22:12 |
| fungi | hrm, another option... | 22:13 |
| corvus | mordred: that could work | 22:13 |
| fungi | we could (eventually) delete repos, but add new empty repos and transfer them on request? | 22:13 |
| fungi | that would effectively replace the nonexistent repo with a redirect to another org | 22:14 |
| *** raissa has quit IRC | 22:14 | |
| mordred | well - I think if we moved them to somewhere people are unlikely to browse to - with a gh transfer - then old links will still work, and we can do the retire-commit trick before we archive witha pointer | 22:14 |
| *** sreejithp has quit IRC | 22:14 | |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Switch to upstream and upgrade gitea to current master (pre-1.9.0) https://review.opendev.org/660848 | 22:15 |
| mordred | and, amusingly enough, non-openstack-things-in-openstack would have been in stackforge/ if we'd stayed with that model, so it's not SUPER inapproriate | 22:15 |
| corvus | mordred, fungi: ^ that commit is an alternate to the previous one | 22:15 |
| *** slaweq has quit IRC | 22:15 | |
| mordred | corvus: actually - I think I like that with a pin | 22:16 |
| mordred | corvus: it does give us an ugly version - but we ARE running off of a sha from master, so it's accurate | 22:16 |
| rpioso | Does the gitea web UI offer a permalink feature similar to GitHub's? | 22:16 |
| mordred | and it makes it easy for us to bump the sha if we need to between now and 1.9.0 | 22:16 |
| mordred | rpioso: what do you mean? | 22:16 |
| mordred | do you mean the redirect thing on repo renames? | 22:17 |
| logan- | rpioso: it does, in a slightly less intuitive way. view a file, click "permalink" in the file header, and then select lines | 22:18 |
| rpioso | mordred: It provides a URI to code which is "permanently" valid. | 22:18 |
| corvus | logan-: as a non-native github user, i found that more intuitive than "select the lines, then click '...' and then click 'copy permalink'" | 22:20 |
| corvus | i have never used "copy permalink" before | 22:20 |
| * mordred too - and me either | 22:20 | |
| corvus | so you just performed a ui experiment on me. it took me between 1 and 2 minutes of poking through the ui to find that. | 22:21 |
| fungi | intuitive for me (also as a non-github user) is to browse the tree as of a particular commit and ling from there | 22:21 |
| corvus | sorry i didn't know it was going to be a ui experiment, otherwise i would have started a recording :) | 22:21 |
| rpioso | logan: Thank you! | 22:21 |
| corvus | fungi: yeah, that's how i've done that before | 22:22 |
| rpioso | fungi: And that appears to be what Gitea's web UI does. The Tree: displays the commit's SHA. | 22:25 |
| *** jamesmcarthur has quit IRC | 22:26 | |
| fungi | makes sense to me. that's how you refer to a permanent state in a git repository | 22:31 |
| fungi | (where permanent depends on it at least being in the history of a head, tag or named ref so the garbage collector doesn't eventually clean it up) | 22:32 |
| *** kskels has joined #openstack-infra | 22:32 | |
| rpioso | fungi: I refer to that as "permanent" :-) | 22:32 |
| *** Goneri has quit IRC | 22:32 | |
| kskels | clarkb: fyi - github replication works now! this is great -thanks for the support and samples from ara project!! | 22:33 |
| fungi | rpioso: i guess i just tend to dwell on the impermanence of all things | 22:33 |
| rpioso | fungi: It's software. | 22:34 |
| mordred | kskels: \o/ | 22:37 |
| Weifan | fungi: if we want to transfer the repo, how should we process? or should we just wait for all the repos to be "archived"? | 22:39 |
| mordred | Weifan: where do you want to transfer the repo? | 22:39 |
| Weifan | transfer to our organization | 22:40 |
| Weifan | https://github.com/openstack/networking-bigswitch this one, to bigswitch | 22:40 |
| mordred | ah. nod. I'm not sure what we've decided on process-wise for doing that ... fungi, do we have a process decided for that? | 22:41 |
| *** diablo_rojo has joined #openstack-infra | 22:42 | |
| *** diablo_rojo has quit IRC | 22:47 | |
| fungi | Weifan: mordred: we just did this for airship's repos, via this mailing list thread: http://lists.openstack.org/pipermail/openstack-infra/2019-May/006376.html | 22:53 |
| *** kskels has left #openstack-infra | 22:57 | |
| *** mriedem has quit IRC | 22:59 | |
| *** igordc has quit IRC | 23:01 | |
| Weifan | fungi: in the email, it says that openstackadmin needs to be added as an adimin of the organization. Is this necessary? I believe it is already an admin of the repo that we want to transfer. | 23:02 |
| fungi | Weifan: the account used to perform the transfer has to temporarily be an admin of both source and destination orgs, because of github's permission model | 23:03 |
| fungi | since it's a one-step process and there's no separate proposal and acceptance phases to the transaction | 23:04 |
| Weifan | shouldn't it be sending an email or notification the receiver? | 23:05 |
| fungi | it does not do any of those things | 23:06 |
| *** tkajinam has joined #openstack-infra | 23:06 | |
| fungi | it basically allows you to transfer the repository if you're an admin of the old and new org, otherwise it refuses | 23:06 |
| fungi | hrm, at least that's the way the process we'd been using works | 23:08 |
| Weifan | is it any different transferring to a private account? | 23:08 |
| fungi | but this documentation implies a proposal/acceptance: https://help.github.com/en/articles/transferring-a-repository | 23:08 |
| mordred | fungi: yeah - I just did an app transfer and when I did it it said "this will have to wait for the receiving party to accept" | 23:09 |
| mordred | and an email was sent to the admin email of the receiving org | 23:09 |
| mordred | fungi: so it's possible this got better since last we looked | 23:09 |
| fungi | agreed... i wonder if this mechanism is accessible via the api | 23:10 |
| mordred | we could try transferring networking-bigswitch without the openstackadmin grant first to test it ... ooh, it would be awesome if it was accessible via the API | 23:10 |
| *** tosky has quit IRC | 23:11 | |
| fungi | the way we'd been doing it was via scripted api interaction | 23:11 |
| *** slaweq has joined #openstack-infra | 23:11 | |
| *** tosky has joined #openstack-infra | 23:11 | |
| Weifan | yep..we can try that | 23:11 |
| fungi | mordred: https://opendev.org/opendev/system-config/src/branch/master/tools/github-org-transfer.py | 23:12 |
| *** slaweq has quit IRC | 23:15 | |
| mordred | fungi: BOOYAH | 23:19 |
| mordred | https://developer.github.com/v3/repos/#transfer-a-repository | 23:19 |
| *** rcernin has joined #openstack-infra | 23:20 | |
| fungi | neat! | 23:21 |
| fungi | this must be a very recent improvement | 23:21 |
| fungi | makes things easier on us | 23:21 |
| *** sarob has quit IRC | 23:23 | |
| *** sarob has joined #openstack-infra | 23:23 | |
| *** sarob has quit IRC | 23:23 | |
| Weifan | :) | 23:24 |
| *** mattw4 has quit IRC | 23:26 | |
| *** zhurong has quit IRC | 23:30 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: Skip file matcher for pipeline using timer trigger https://review.opendev.org/660856 | 23:33 |
| Weifan | so, is there anything needed for me to proceed, or should I just inform our admin to to watch for the incoming email? | 23:34 |
| *** tosky has quit IRC | 23:34 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: ansible-config: pin ara to <1.0.0 https://review.opendev.org/660859 | 23:38 |
| *** michael-beaver has quit IRC | 23:41 | |
| mordred | Weifan: I thnik that - although it might be a couple of days (we should update our script first, and it's dinner time) | 23:42 |
| mordred | (just inform your admin to watch for the incoming email) | 23:42 |
| Weifan | mordred: thanks:) | 23:42 |
| *** michael-beaver has joined #openstack-infra | 23:51 | |
| dirk | config-core: could I get a 2nd review on https://review.opendev.org/#/c/660823/ ? | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!