| clarkb | ianw: and or corvus have a moment for https://review.opendev.org/#/c/674930/ ? I'm happy to approve that when I return thursday if you just want to review it and not approve it | 00:00 |
|---|---|---|
| corvus | sent mail to openstack-discuss | 00:01 |
| *** sthussey has quit IRC | 00:01 | |
| *** betherly has quit IRC | 00:01 | |
| corvus | clarkb: +2 because i'm about to eod | 00:02 |
| clarkb | ya me too | 00:02 |
| clarkb | thanks | 00:02 |
| ianw | clarkb: lgtm, does it need a restart? | 00:02 |
| clarkb | ianw: I believe that docker-compose will do a restart automagically | 00:02 |
| clarkb | ianw: then you'll want to confirm that gerrit replication continues to function afterwards | 00:03 |
| ianw | hrm, yeah let's wait until more than just me around in case of unexpected issues :) | 00:03 |
| clarkb | `docker ps -a` will show you the age of the container instance on the gitea backends. I'm totally happy to monitor that myself too | 00:03 |
| *** yamamoto has quit IRC | 00:04 | |
| fungi | corvus: draft e-mail lgtm, i made a tiny typo correction (which involved deleting a definite article which was superfluous... "We have re-implemented) | 00:04 |
| fungi | er, stray newline... it was "We have re-implemented the these on the Zuul build page" | 00:05 |
| fungi | (in case you're not re-copying from the etherpad, since you wouldn't spot the deletion) | 00:05 |
| *** logan- has quit IRC | 00:12 | |
| ianw | corvus: re one page; yes i'd consider the status and log links pretty equal in "reasons i'm looking at the page" so having everything discoverable by scrolling on the page (rather than having to click to unhide a tab, say) sounds great | 00:12 |
| *** logan- has joined #openstack-infra | 00:13 | |
| *** mattw4 has quit IRC | 00:16 | |
| corvus | fungi: i'm afraid i sent it out with the extra word, sorry | 00:16 |
| *** betherly has joined #openstack-infra | 00:17 | |
| fungi | ah, no big deal | 00:19 |
| fungi | sorry i didn't get a chance to read it sooner | 00:19 |
| *** betherly has quit IRC | 00:22 | |
| *** markvoelker has joined #openstack-infra | 00:23 | |
| *** michael-beaver has quit IRC | 00:28 | |
| *** hongbin has joined #openstack-infra | 00:34 | |
| openstackgerrit | Merged opendev/system-config master: Add vexxhost backup server https://review.opendev.org/674550 | 00:38 |
| *** gregoryo has joined #openstack-infra | 00:41 | |
| *** jamesmcarthur_ has quit IRC | 00:47 | |
| *** armax has quit IRC | 00:50 | |
| *** yamamoto has joined #openstack-infra | 00:52 | |
| *** jamesmcarthur has joined #openstack-infra | 01:01 | |
| *** ricolin has joined #openstack-infra | 01:02 | |
| *** jamesmcarthur has quit IRC | 01:04 | |
| corvus | lists.o.o is pretty swappy; i think mailman needs to be restarted. | 01:09 |
| corvus | i'm going to stop the others first, then openstack, then restart them all | 01:10 |
| corvus | openstack is the one using lots of ram and is sitting in iowait | 01:10 |
| corvus | also we have no cacti graphs for that host :( | 01:12 |
| *** igordc has quit IRC | 01:13 | |
| *** igordc has joined #openstack-infra | 01:15 | |
| *** ricolin_ has joined #openstack-infra | 01:17 | |
| *** betherly has joined #openstack-infra | 01:19 | |
| *** ricolin has quit IRC | 01:20 | |
| *** gregoryo has quit IRC | 01:22 | |
| *** igordc has quit IRC | 01:23 | |
| corvus | whatever the incoming qrunner for openstack is doing, it's slow | 01:23 |
| *** betherly has quit IRC | 01:24 | |
| *** eernst has joined #openstack-infra | 01:25 | |
| corvus | something with the transparency list | 01:25 |
| *** pcaruana has quit IRC | 01:26 | |
| fungi | oh fun. it's probably getting spammed | 01:27 |
| fungi | it also probably needs to be retired | 01:27 |
| corvus | yeah, it looks like it's processing an incoming message for that list | 01:27 |
| fungi | the transparency working group has been defunct for many years | 01:28 |
| corvus | with a large base64-encoded body | 01:28 |
| *** eernst has quit IRC | 01:29 | |
| corvus | i'm going to kill the process, delete the message, and lockfiles | 01:30 |
| fungi | eww | 01:30 |
| fungi | thanks! | 01:30 |
| fungi | i'll ask around about retiring that ml too. its last actual post was in may 2015 | 01:30 |
| *** gyee has quit IRC | 01:33 | |
| corvus | okay, everything should be up and running again now | 01:36 |
| fungi | thanks corvus! looks like the backlog is starting to arrive | 01:36 |
| *** markvoelker has quit IRC | 01:39 | |
| *** bobh has joined #openstack-infra | 01:40 | |
| *** markvoelker has joined #openstack-infra | 01:40 | |
| *** betherly has joined #openstack-infra | 01:48 | |
| *** gregoryo has joined #openstack-infra | 01:50 | |
| *** bhavikdbavishi has joined #openstack-infra | 01:50 | |
| *** betherly has quit IRC | 01:53 | |
| *** jamesmcarthur has joined #openstack-infra | 01:56 | |
| *** jamesmcarthur has quit IRC | 01:58 | |
| *** jamesmcarthur has joined #openstack-infra | 01:59 | |
| *** haleyb has quit IRC | 02:01 | |
| *** smarcet has joined #openstack-infra | 02:06 | |
| *** bhavikdbavishi1 has joined #openstack-infra | 02:07 | |
| *** apetrich has quit IRC | 02:09 | |
| *** bhavikdbavishi has quit IRC | 02:09 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 02:09 | |
| dmsimard | I was mostly offline today and haven't caught up with the full backlog yet but I can help tomorrow | 02:16 |
| *** smarcet has quit IRC | 02:21 | |
| *** jamesmcarthur has quit IRC | 02:22 | |
| *** jamesmcarthur has joined #openstack-infra | 02:22 | |
| *** jamesmcarthur has quit IRC | 02:25 | |
| *** jamesmcarthur has joined #openstack-infra | 02:26 | |
| *** ricolin_ is now known as ricolin | 02:27 | |
| *** betherly has joined #openstack-infra | 02:29 | |
| *** betherly has quit IRC | 02:34 | |
| *** ykarel|away has joined #openstack-infra | 02:44 | |
| *** yamamoto has quit IRC | 02:44 | |
| *** bhavikdbavishi has quit IRC | 02:44 | |
| *** yamamoto has joined #openstack-infra | 02:45 | |
| *** igordc has joined #openstack-infra | 02:49 | |
| *** betherly has joined #openstack-infra | 02:50 | |
| *** bobh has quit IRC | 02:53 | |
| *** betherly has quit IRC | 02:55 | |
| *** ykarel|away has quit IRC | 02:56 | |
| *** jamesmcarthur has quit IRC | 02:56 | |
| *** xinranwang has joined #openstack-infra | 03:01 | |
| *** ykarel|away has joined #openstack-infra | 03:02 | |
| *** jamesmcarthur has joined #openstack-infra | 03:12 | |
| *** markvoelker has quit IRC | 03:14 | |
| *** ykarel|away has quit IRC | 03:16 | |
| *** whoami-rajat has joined #openstack-infra | 03:19 | |
| *** pabelanger has quit IRC | 03:20 | |
| *** ramishra has joined #openstack-infra | 03:27 | |
| *** bhavikdbavishi has joined #openstack-infra | 03:32 | |
| *** ykarel|away has joined #openstack-infra | 03:35 | |
| *** psachin has joined #openstack-infra | 03:36 | |
| *** markvoelker has joined #openstack-infra | 03:38 | |
| *** ianychoi_ has joined #openstack-infra | 03:38 | |
| *** ianychoi has quit IRC | 03:42 | |
| *** ykarel|away has quit IRC | 03:42 | |
| *** betherly has joined #openstack-infra | 03:42 | |
| *** markvoelker has quit IRC | 03:43 | |
| *** pcaruana has joined #openstack-infra | 03:44 | |
| *** betherly has quit IRC | 03:47 | |
| *** hongbin has quit IRC | 03:49 | |
| *** ykarel|away has joined #openstack-infra | 03:50 | |
| *** dchen has quit IRC | 03:52 | |
| *** dchen has joined #openstack-infra | 03:54 | |
| *** jamesmcarthur has quit IRC | 04:00 | |
| *** diablo_rojo has joined #openstack-infra | 04:04 | |
| *** udesale has joined #openstack-infra | 04:04 | |
| openstackgerrit | Andreas Jaeger proposed opendev/puppet-openstackci master: Handle robots.txt on logs.o.o https://review.opendev.org/674920 | 04:21 |
| AJaeger | ianw: this is what you had in mind, isn't it? Much simpler ;) ^ | 04:21 |
| *** jaosorior has quit IRC | 04:22 | |
| *** markvoelker has joined #openstack-infra | 04:24 | |
| ianw | AJaeger: I think so :) there seems to be stray \ at the end, and i think the \( after help/\* needs to go? | 04:24 |
| *** jamesmcarthur has joined #openstack-infra | 04:30 | |
| *** ykarel|away has quit IRC | 04:32 | |
| AJaeger | indeed ;( | 04:34 |
| openstackgerrit | Andreas Jaeger proposed opendev/puppet-openstackci master: Handle robots.txt on logs.o.o https://review.opendev.org/674920 | 04:34 |
| *** markvoelker has quit IRC | 04:38 | |
| *** rh-jelabarre has quit IRC | 04:39 | |
| *** jamesmcarthur has quit IRC | 04:55 | |
| *** jamesmcarthur has joined #openstack-infra | 05:01 | |
| *** ykarel|away has joined #openstack-infra | 05:02 | |
| *** betherly has joined #openstack-infra | 05:05 | |
| *** ykarel|away is now known as ykarel | 05:05 | |
| *** guoqiao has quit IRC | 05:08 | |
| *** icarusfactor has joined #openstack-infra | 05:08 | |
| *** factor has quit IRC | 05:08 | |
| *** guoqiao has joined #openstack-infra | 05:09 | |
| *** betherly has quit IRC | 05:10 | |
| *** raukadah is now known as chandankumar | 05:11 | |
| *** diablo_rojo has quit IRC | 05:11 | |
| *** kopecmartin|off is now known as kopecmartin | 05:23 | |
| *** betherly has joined #openstack-infra | 05:26 | |
| *** igordc has quit IRC | 05:31 | |
| *** betherly has quit IRC | 05:31 | |
| *** jamesmcarthur has quit IRC | 05:35 | |
| *** tdasilva_ has quit IRC | 05:36 | |
| *** jtomasek has joined #openstack-infra | 05:43 | |
| *** jtomasek has quit IRC | 05:43 | |
| *** ociuhandu has joined #openstack-infra | 05:49 | |
| *** ociuhandu has quit IRC | 05:53 | |
| *** dpawlik has joined #openstack-infra | 05:56 | |
| *** icarusfactor has quit IRC | 06:01 | |
| *** jtomasek has joined #openstack-infra | 06:01 | |
| *** dpawlik has quit IRC | 06:01 | |
| *** factor has joined #openstack-infra | 06:02 | |
| *** jamesmcarthur has joined #openstack-infra | 06:03 | |
| *** gregoryo has quit IRC | 06:06 | |
| *** jtomasek has quit IRC | 06:06 | |
| *** jtomasek has joined #openstack-infra | 06:08 | |
| *** xinranwang has quit IRC | 06:10 | |
| *** ramishra has quit IRC | 06:12 | |
| *** markvoelker has joined #openstack-infra | 06:18 | |
| *** betherly has joined #openstack-infra | 06:18 | |
| *** ykarel_ has joined #openstack-infra | 06:18 | |
| *** ykarel has quit IRC | 06:21 | |
| *** markvoelker has quit IRC | 06:22 | |
| *** betherly has quit IRC | 06:23 | |
| *** udesale has quit IRC | 06:24 | |
| *** udesale has joined #openstack-infra | 06:24 | |
| *** udesale has quit IRC | 06:27 | |
| *** udesale has joined #openstack-infra | 06:27 | |
| *** jaosorior has joined #openstack-infra | 06:28 | |
| *** pgaxatte has joined #openstack-infra | 06:35 | |
| *** dchen has quit IRC | 06:35 | |
| *** apetrich has joined #openstack-infra | 06:37 | |
| *** jamesmcarthur has quit IRC | 06:37 | |
| *** dchen has joined #openstack-infra | 06:40 | |
| *** dchen has quit IRC | 06:42 | |
| *** dchen has joined #openstack-infra | 06:42 | |
| *** dchen has quit IRC | 06:43 | |
| *** dchen has joined #openstack-infra | 06:44 | |
| *** dchen has quit IRC | 06:45 | |
| *** ccamacho has joined #openstack-infra | 06:45 | |
| *** dchen has joined #openstack-infra | 06:45 | |
| *** dpawlik has joined #openstack-infra | 06:46 | |
| openstackgerrit | Slawek Kaplonski proposed openstack/project-config master: Add new Neutron jobs to Grafana dashboard https://review.opendev.org/675036 | 06:48 |
| *** slaweq has joined #openstack-infra | 06:48 | |
| *** betherly has joined #openstack-infra | 06:50 | |
| *** betherly has quit IRC | 06:54 | |
| *** dtantsur|afk is now known as dtantsur | 06:55 | |
| *** odicha has joined #openstack-infra | 06:58 | |
| *** yamamoto has quit IRC | 07:00 | |
| *** ykarel_ is now known as ykarel|meeting | 07:00 | |
| *** rcernin has quit IRC | 07:03 | |
| *** ginopc has joined #openstack-infra | 07:04 | |
| *** rpittau|afk is now known as rpittau | 07:07 | |
| *** ramishra has joined #openstack-infra | 07:08 | |
| *** yamamoto has joined #openstack-infra | 07:09 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Use wait for empty update queue before accepting merges https://review.opendev.org/675039 | 07:10 |
| *** udesale has quit IRC | 07:10 | |
| *** sshnaidm|afk is now known as sshnaidm|bbl | 07:17 | |
| *** tosky has joined #openstack-infra | 07:22 | |
| *** tesseract has joined #openstack-infra | 07:26 | |
| *** ralonsoh has joined #openstack-infra | 07:29 | |
| *** guoqiao has quit IRC | 07:34 | |
| *** jamesmcarthur has joined #openstack-infra | 07:35 | |
| *** guoqiao has joined #openstack-infra | 07:37 | |
| *** lpetrut has joined #openstack-infra | 07:38 | |
| *** Goneri has joined #openstack-infra | 07:38 | |
| *** jistr is now known as jistr|afk | 07:42 | |
| *** spsurya has joined #openstack-infra | 07:50 | |
| *** guoqiao has quit IRC | 07:52 | |
| *** dchen has quit IRC | 07:52 | |
| *** dchen has joined #openstack-infra | 07:53 | |
| *** iurygregory has joined #openstack-infra | 07:53 | |
| *** ralonsoh has quit IRC | 07:54 | |
| *** pkopec has joined #openstack-infra | 07:57 | |
| *** Goneri has quit IRC | 07:58 | |
| *** udesale has joined #openstack-infra | 08:00 | |
| openstackgerrit | Merged zuul/zuul master: Update ara to >=0.16.5 to support ansible 2.8 https://review.opendev.org/674958 | 08:01 |
| *** rascasoft has quit IRC | 08:01 | |
| *** ykarel has joined #openstack-infra | 08:02 | |
| *** jtomasek has quit IRC | 08:02 | |
| *** linshiyx has joined #openstack-infra | 08:02 | |
| *** linshiyx has quit IRC | 08:03 | |
| *** cgoncalves has quit IRC | 08:04 | |
| *** rascasoft has joined #openstack-infra | 08:04 | |
| *** lucasagomes has joined #openstack-infra | 08:04 | |
| *** cgoncalves has joined #openstack-infra | 08:04 | |
| *** ykarel|meeting has quit IRC | 08:04 | |
| *** ykarel is now known as ykarel|lunch | 08:06 | |
| *** jamesmcarthur has quit IRC | 08:08 | |
| *** Lucas_Gray has joined #openstack-infra | 08:12 | |
| *** mauro|call has quit IRC | 08:13 | |
| *** mauro|call has joined #openstack-infra | 08:14 | |
| *** dchen has quit IRC | 08:17 | |
| *** e0ne has joined #openstack-infra | 08:18 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Create /etc/machine-id for RHEL images https://review.opendev.org/675056 | 08:20 |
| *** panda has quit IRC | 08:22 | |
| *** panda has joined #openstack-infra | 08:23 | |
| *** roman_g has quit IRC | 08:23 | |
| *** ralonsoh has joined #openstack-infra | 08:26 | |
| *** derekh has joined #openstack-infra | 08:29 | |
| *** ralonsoh has quit IRC | 08:35 | |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Rework a cache invalidation issue https://review.opendev.org/674425 | 08:38 |
| *** jamesmcarthur has joined #openstack-infra | 08:42 | |
| *** betherly has joined #openstack-infra | 08:45 | |
| *** ralonsoh has joined #openstack-infra | 08:47 | |
| *** Lucas_Gray has quit IRC | 08:48 | |
| *** markvoelker has joined #openstack-infra | 08:50 | |
| *** jtomasek has joined #openstack-infra | 08:52 | |
| *** Lucas_Gray has joined #openstack-infra | 08:53 | |
| *** markvoelker has quit IRC | 08:55 | |
| *** mauro|call has quit IRC | 08:58 | |
| *** mauro|call has joined #openstack-infra | 09:00 | |
| *** Lucas_Gray has quit IRC | 09:01 | |
| *** Lucas_Gray has joined #openstack-infra | 09:02 | |
| *** tkajinam has quit IRC | 09:02 | |
| *** ykarel|lunch is now known as ykarel | 09:02 | |
| *** rascasoft has quit IRC | 09:03 | |
| *** ociuhandu has joined #openstack-infra | 09:03 | |
| *** rascasoft has joined #openstack-infra | 09:06 | |
| *** roman_g has joined #openstack-infra | 09:08 | |
| openstackgerrit | Merged opendev/irc-meetings master: Free up unused Dragonflow meeting slot https://review.opendev.org/674303 | 09:13 |
| *** ociuhandu has quit IRC | 09:13 | |
| *** mauro|call is now known as takamatsu | 09:14 | |
| *** ociuhandu has joined #openstack-infra | 09:16 | |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Add reference pipelines file for Gerrit driver https://review.opendev.org/672683 | 09:20 |
| *** ociuhandu has quit IRC | 09:25 | |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: A reporter for Elasticsearch https://review.opendev.org/644927 | 09:27 |
| *** ociuhandu has joined #openstack-infra | 09:29 | |
| *** gfidente has joined #openstack-infra | 09:36 | |
| *** bhavikdbavishi has quit IRC | 09:39 | |
| *** sshnaidm|bbl is now known as sshnaidm | 09:42 | |
| *** jistr|afk is now known as jistr | 09:54 | |
| *** jamesmcarthur has quit IRC | 09:57 | |
| *** ricolin_ has joined #openstack-infra | 10:01 | |
| *** ricolin has quit IRC | 10:03 | |
| *** odicha has quit IRC | 10:06 | |
| *** odicha has joined #openstack-infra | 10:06 | |
| *** n-saito has quit IRC | 10:12 | |
| *** yamamoto has quit IRC | 10:18 | |
| *** roman_g has quit IRC | 10:25 | |
| *** roman_g has joined #openstack-infra | 10:27 | |
| *** udesale has quit IRC | 10:31 | |
| *** pgaxatte has quit IRC | 10:34 | |
| openstackgerrit | Sagi Shnaidman proposed zuul/zuul-jobs master: Don't install centos repos on RHEL https://review.opendev.org/674572 | 10:36 |
| *** yamamoto has joined #openstack-infra | 10:41 | |
| *** yamamoto has quit IRC | 10:45 | |
| *** smarcet has joined #openstack-infra | 10:47 | |
| *** guoqiao has joined #openstack-infra | 10:48 | |
| *** sean-k-mooney has quit IRC | 10:52 | |
| *** rfolco has joined #openstack-infra | 10:54 | |
| *** rfolco|ruck has quit IRC | 10:54 | |
| *** yamamoto has joined #openstack-infra | 10:59 | |
| *** smarcet has quit IRC | 11:02 | |
| *** ociuhandu has quit IRC | 11:07 | |
| *** Wryhder has joined #openstack-infra | 11:16 | |
| *** Lucas_Gray has quit IRC | 11:17 | |
| *** Wryhder is now known as Lucas_Gray | 11:17 | |
| *** Lucas_Gray has quit IRC | 11:21 | |
| *** kjackal has joined #openstack-infra | 11:23 | |
| *** iurygregory has quit IRC | 11:29 | |
| *** iurygregory has joined #openstack-infra | 11:30 | |
| *** Lucas_Gray has joined #openstack-infra | 11:31 | |
| *** jamesmcarthur has joined #openstack-infra | 11:38 | |
| *** markvoelker has joined #openstack-infra | 11:49 | |
| *** bhavikdbavishi has joined #openstack-infra | 11:58 | |
| *** pgaxatte has joined #openstack-infra | 11:58 | |
| *** ekultails has joined #openstack-infra | 11:59 | |
| *** ociuhandu has joined #openstack-infra | 12:09 | |
| *** markvoelker has quit IRC | 12:11 | |
| *** guoqiao has quit IRC | 12:12 | |
| *** bhavikdbavishi1 has joined #openstack-infra | 12:13 | |
| *** rlandy has joined #openstack-infra | 12:13 | |
| *** ociuhandu has quit IRC | 12:14 | |
| *** rh-jelabarre has joined #openstack-infra | 12:14 | |
| *** bhavikdbavishi has quit IRC | 12:14 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 12:14 | |
| *** jamesmcarthur has quit IRC | 12:16 | |
| *** markvoelker has joined #openstack-infra | 12:16 | |
| sshnaidm | cores, please review: https://review.opendev.org/#/c/674572 | 12:18 |
| *** ociuhandu has joined #openstack-infra | 12:19 | |
| sshnaidm | mordred, ianw ^^ | 12:19 |
| *** ociuhandu has quit IRC | 12:22 | |
| *** ociuhandu has joined #openstack-infra | 12:23 | |
| *** kaisers has joined #openstack-infra | 12:28 | |
| *** rfolco has quit IRC | 12:28 | |
| *** Lucas_Gray has quit IRC | 12:34 | |
| *** michael-beaver has joined #openstack-infra | 12:35 | |
| *** jistr is now known as jistr|call | 12:37 | |
| *** ociuhandu has quit IRC | 12:38 | |
| *** haleyb has joined #openstack-infra | 12:39 | |
| *** ociuhandu has joined #openstack-infra | 12:39 | |
| *** zbr has quit IRC | 12:46 | |
| *** ociuhandu has quit IRC | 12:46 | |
| *** ociuhandu has joined #openstack-infra | 12:48 | |
| *** ricolin_ is now known as ricolin | 12:48 | |
| *** jamesmcarthur has joined #openstack-infra | 12:49 | |
| *** aaronsheffield has joined #openstack-infra | 12:52 | |
| *** pabelanger has joined #openstack-infra | 12:57 | |
| *** hrw has left #openstack-infra | 12:58 | |
| *** kjackal has quit IRC | 13:01 | |
| *** kjackal has joined #openstack-infra | 13:01 | |
| *** zbr has joined #openstack-infra | 13:02 | |
| *** smarcet has joined #openstack-infra | 13:05 | |
| *** Lucas_Gray has joined #openstack-infra | 13:09 | |
| *** mriedem has joined #openstack-infra | 13:10 | |
| *** jamesmcarthur has quit IRC | 13:13 | |
| *** jamesmcarthur has joined #openstack-infra | 13:14 | |
| *** jcoufal has joined #openstack-infra | 13:15 | |
| *** ykarel is now known as ykarel|afk | 13:15 | |
| *** ykarel_ has joined #openstack-infra | 13:19 | |
| *** sthussey has joined #openstack-infra | 13:19 | |
| *** udesale has joined #openstack-infra | 13:19 | |
| *** ykarel_ has quit IRC | 13:21 | |
| *** ykarel|afk has quit IRC | 13:21 | |
| AJaeger | fungi: what do you think of https://review.opendev.org/674920 for removing robots.txt from logs.o.o for docs jobs? | 13:22 |
| fungi | i need to take a look at it in a bit, on my way out the door to run a quick errand | 13:23 |
| fungi | pulled it up for when i get back | 13:23 |
| AJaeger | fungi: thanks, take your time running errands ;) | 13:25 |
| openstackgerrit | Merged openstack/project-config master: Add new Neutron jobs to Grafana dashboard https://review.opendev.org/675036 | 13:26 |
| *** smarcet has quit IRC | 13:28 | |
| openstackgerrit | Jeff Liu proposed zuul/zuul-operator master: WIP: Add zuul-operator-functional-openshift job https://review.opendev.org/674355 | 13:38 |
| *** roman_g has quit IRC | 13:42 | |
| *** jamesmcarthur has quit IRC | 13:44 | |
| *** jaosorior has quit IRC | 13:44 | |
| *** factor has quit IRC | 13:44 | |
| *** factor has joined #openstack-infra | 13:45 | |
| *** jamesmcarthur has joined #openstack-infra | 13:46 | |
| *** ociuhandu has quit IRC | 13:48 | |
| *** ociuhandu has joined #openstack-infra | 13:49 | |
| *** Lucas_Gray has quit IRC | 13:49 | |
| *** electrofelix has quit IRC | 13:50 | |
| *** jamesmcarthur has quit IRC | 13:51 | |
| *** jamesmcarthur has joined #openstack-infra | 13:53 | |
| *** Lucas_Gray has joined #openstack-infra | 13:56 | |
| *** tesseract has quit IRC | 13:56 | |
| *** tesseract has joined #openstack-infra | 13:57 | |
| *** whoami-rajat has quit IRC | 13:57 | |
| fungi | it was quick | 13:58 |
| fungi | reviewing now | 13:58 |
| *** smrcascao has quit IRC | 13:59 | |
| *** tesseract has quit IRC | 14:00 | |
| *** tesseract has joined #openstack-infra | 14:01 | |
| fungi | AJaeger: small mistake, see inline comment, the path is wrong | 14:03 |
| *** ociuhandu has quit IRC | 14:03 | |
| openstackgerrit | Andreas Jaeger proposed opendev/puppet-openstackci master: Handle robots.txt on logs.o.o https://review.opendev.org/674920 | 14:05 |
| AJaeger | fungi: ARgh ;( Glad that you caught that! | 14:05 |
| AJaeger | and updated | 14:05 |
| *** ociuhandu has joined #openstack-infra | 14:06 | |
| *** slaweq has quit IRC | 14:07 | |
| donnyd | Well I was posting in openstack-meeting and didnt even realize it | 14:08 |
| donnyd | So I have the compiled data on where the nodepool instances are headed on the public interwebs | 14:08 |
| *** smarcet has joined #openstack-infra | 14:08 | |
| donnyd | Just trying to figure out how to present it in a way that doesn't cough up topologies | 14:08 |
| donnyd | Ok, I think I have it sanitized enough. Now should I send it somewhere? | 14:08 |
| *** slaweq has joined #openstack-infra | 14:08 | |
| *** betherly has quit IRC | 14:10 | |
| *** ociuhandu has quit IRC | 14:11 | |
| fungi | what form is it in? suitable for a paste.openstack.org link in here or a message to one of the mailing lists? | 14:11 |
| *** tiffanie has joined #openstack-infra | 14:12 | |
| *** dpawlik has quit IRC | 14:15 | |
| tiffanie | hi I'm trying to add stories into storyboard and I'm getting this error... Also 1 task out of 4 got created which is weird https://usercontent.irccloud-cdn.com/file/laZrhc1Q/Capture.PNG | 14:17 |
| *** ociuhandu has joined #openstack-infra | 14:20 | |
| *** dtantsur is now known as dtantsur|bbl | 14:21 | |
| *** smarcet has quit IRC | 14:23 | |
| *** ociuhandu has quit IRC | 14:25 | |
| *** Lucas_Gray has quit IRC | 14:27 | |
| *** Lucas_Gray has joined #openstack-infra | 14:29 | |
| *** betherly has joined #openstack-infra | 14:29 | |
| *** ykarel has joined #openstack-infra | 14:34 | |
| *** smarcet has joined #openstack-infra | 14:35 | |
| *** jistr|call is now known as jistr | 14:36 | |
| *** betherly has quit IRC | 14:39 | |
| *** Wryhder has joined #openstack-infra | 14:40 | |
| donnyd | fungi: I can do csv, copy paste, or I can shuttle the data off to somewhere else in realtime | 14:41 |
| *** Lucas_Gray has quit IRC | 14:41 | |
| *** Wryhder is now known as Lucas_Gray | 14:41 | |
| fungi | donnyd: how about https://ethercalc.openstack.org/ ? will that work? | 14:42 |
| fungi | tiffanie: there is currently a race condition when trying to create a story with more than one initial task. if you create the story with only one task and then add the rest to the created story it generally works | 14:43 |
| donnyd | I think i can surely make that work | 14:43 |
| tiffanie | ok it works, thanks | 14:45 |
| *** ociuhandu has joined #openstack-infra | 14:45 | |
| *** mattw4 has joined #openstack-infra | 14:46 | |
| *** altlogbot_3 has quit IRC | 14:46 | |
| *** altlogbot_3 has joined #openstack-infra | 14:47 | |
| *** betherly has joined #openstack-infra | 14:47 | |
| *** yamamoto has quit IRC | 14:48 | |
| donnyd | fungi: its 42mb worth of data, but we should be able to get it sorted on ethercalc | 14:50 |
| fungi | tiffanie: hopefully we'll get that bug fixed in the near future, sorry for the inconvenience there | 14:50 |
| *** jamesmcarthur has quit IRC | 14:51 | |
| fungi | donnyd: ooh, that's more than i was guessing, but might be a good test of ethercalc's abilities. or we order the data on some useful axis first and include only the most significant entries | 14:51 |
| donnyd | Well I already have it sorted if you want to see that data | 14:52 |
| fungi | any early observations, or anything anomalous jump out at you? | 14:52 |
| donnyd | uh oh.. I may have broken the ethercalcs | 14:52 |
| donnyd | I am getting unavailable | 14:53 |
| fungi | heh, sounds like it was a good test then | 14:53 |
| fungi | me too, i'll take a look | 14:53 |
| *** ianychoi_ is now known as ianychoi | 14:54 | |
| *** yamamoto has joined #openstack-infra | 14:54 | |
| *** mattw4 has quit IRC | 14:55 | |
| donnyd | https://www.irccloud.com/pastebin/vqjh6eST/ | 14:56 |
| *** factor has quit IRC | 14:57 | |
| *** factor has joined #openstack-infra | 14:58 | |
| donnyd | This data is not including actual throughput #'s, so just because an IP gets hit a lot doesn't mean it adds up to a significant amount of data | 14:59 |
| corvus | fungi: are you looking at ethercalc? | 15:00 |
| fungi | corvus: yep, though only just getting to the point of figuring out how the service is started and where it logs | 15:00 |
| donnyd | 2001:470:e045:2:f816:3eff:fee6:691d is the mirror and so is 108.44.198.37 | 15:00 |
| corvus | yeah i'm not there either | 15:00 |
| corvus | fungi: it looks like some process should be listening on 8000 but it isn't | 15:01 |
| donnyd | sorry, didn't mean to break it | 15:01 |
| fungi | corvus: yep, i managed to at least get that far looking at the apache config | 15:01 |
| fungi | donnyd: no apologies needed, you didn't introduce whatever bug crashed it | 15:01 |
| fungi | systemctl knows of an ethercalc.service | 15:02 |
| *** mattw4 has joined #openstack-infra | 15:02 | |
| corvus | yep and has an error logged | 15:02 |
| corvus | http://paste.openstack.org/show/755625/ | 15:03 |
| corvus | fungi: you want to try restarting it? | 15:03 |
| *** smarcet has quit IRC | 15:03 | |
| *** factor has quit IRC | 15:03 | |
| fungi | that stacktrace also looks like what i found once i remembered how to use journalctl | 15:03 |
| fungi | and yeah, i can give it a stop and a start now | 15:04 |
| fungi | i see it listening again | 15:04 |
| fungi | though apache is still saying the service is unavailable | 15:05 |
| fungi | now it's working again | 15:05 |
| fungi | just takes a bit for apache to notice maybe, or the service isn't actually responding when it initially binds to the socket | 15:06 |
| *** mattw4 has quit IRC | 15:08 | |
| *** mattw4 has joined #openstack-infra | 15:09 | |
| *** smarcet has joined #openstack-infra | 15:09 | |
| *** rlandy is now known as rlandy|rover | 15:13 | |
| *** chandankumar is now known as raukadah | 15:13 | |
| fungi | donnyd: yeah, are those counts the total number of packets over some duration? | 15:13 |
| *** jaosorior has joined #openstack-infra | 15:14 | |
| *** iurygregory has quit IRC | 15:14 | |
| fungi | or number of tcp sessions initiated? | 15:14 |
| fungi | i guess there's udp in there too | 15:14 |
| fungi | those port 53 entries are likely overall a small portion of the bandwidth, yes | 15:15 |
| donnyd | total number of connections initiated | 15:15 |
| donnyd | so these are coming off the fw logs | 15:15 |
| fungi | got it, so some sort of state tracking | 15:15 |
| donnyd | anything that is a new connection would get logged | 15:15 |
| fungi | i guess we can basically ignore the first 8 lines in that case as they're dns and communication with the mirror | 15:16 |
| donnyd | yep | 15:16 |
| donnyd | we could provide faster dns access at the mirror and that would likely help a little | 15:17 |
| raukadah | Zuul Job cores, please have a look at this review https://review.opendev.org/#/c/674572/ , thanks! | 15:17 |
| *** smarcet has quit IRC | 15:17 | |
| *** jamesmcarthur has joined #openstack-infra | 15:18 | |
| fungi | grr, that irccloud page is doing something to sabotage copying to a buffer | 15:18 |
| fungi | aha. switching it to raw view works | 15:19 |
| donnyd | next highest is a rdo ip | 15:19 |
| openstackgerrit | Merged opendev/puppet-openstackci master: Handle robots.txt on logs.o.o https://review.opendev.org/674920 | 15:19 |
| fungi | donnyd: 54.81.116.189 ? | 15:19 |
| donnyd | yep | 15:19 |
| fungi | interesting, it looks like an aws ec2 instance according to reverse dns | 15:20 |
| fungi | how did you work out that it was rdo? | 15:20 |
| donnyd | http://54.81.116.189 | 15:20 |
| fungi | hah, yep | 15:20 |
| *** yamamoto has quit IRC | 15:21 | |
| fungi | sometimes i get tunnel vision and overlook the simple solutions | 15:21 |
| *** yamamoto has joined #openstack-infra | 15:21 | |
| donnyd | https://52.70.175.131/ is the next | 15:21 |
| fungi | so, yes, 54.81.116.189 is probably where tripleo is accessing all the rdo package builds from | 15:21 |
| donnyd | which has a docker.io name in cn on the cert | 15:22 |
| fungi | right, probably some jobs going straight to dockerhub and bypassing our proxy | 15:22 |
| fungi | assuming you've filtered out the mirror server as a source address | 15:22 |
| fungi | amusingly also an aws ec2 instance | 15:23 |
| donnyd | i haven't gotten that far in the query, but i am not getting raw output from the fw.. .i am picking on individual interfaces | 15:23 |
| donnyd | and the mirrors tenant network isn't one of them | 15:23 |
| fungi | good enough | 15:23 |
| donnyd | also it uses the edge fw only for ipv46requests | 15:23 |
| donnyd | ipv6 | 15:23 |
| *** kjackal has quit IRC | 15:24 | |
| fungi | interesting that multicast is so high there | 15:24 |
| *** kjackal has joined #openstack-infra | 15:24 | |
| donnyd | apparently its popular | 15:24 |
| donnyd | LOL | 15:24 |
| fungi | 104.18.125.25 seems to be a cloudflare endpoint so that's going to be hard to identify | 15:25 |
| donnyd | I can dig a little deeper to see what its asking for at cloudflare | 15:25 |
| donnyd | next up is another docker.io ip | 15:26 |
| fungi | 54.165.149.19 ? | 15:26 |
| donnyd | yea | 15:26 |
| fungi | the cert i saw on it just has "amazon" as the cn | 15:26 |
| *** slaweq_ has joined #openstack-infra | 15:27 | |
| fungi | oh, nope, it's docker.io for the cn | 15:27 |
| fungi | amazon was the issuer | 15:27 |
| donnyd | yea | 15:27 |
| fungi | i guess amazon has a ca for their customers | 15:27 |
| *** slaweq has quit IRC | 15:29 | |
| fungi | interesting that 104.18.121.25:443 is in there twice | 15:30 |
| *** factor has joined #openstack-infra | 15:30 | |
| fungi | oh, nope it's not | 15:30 |
| fungi | but again cloudflare | 15:30 |
| *** smarcet has joined #openstack-infra | 15:30 | |
| fungi | 52.2.186.244 is dockerhub again | 15:31 |
| fungi | i'm starting to see a pattern | 15:31 |
| *** pgaxatte has quit IRC | 15:32 | |
| fungi | 34.233.151.211 also dockerhub | 15:32 |
| fungi | as are 54.175.43.85 and 52.87.94.70 and 54.88.231.116 | 15:34 |
| donnyd | yea, maybe we can speed things up by doing something about dockerhub pulls | 15:34 |
| fungi | so my take is that an rdo package proxy might help (could at least speed up tripleo jobs), and more projects could stand to configure their jobs to hit our dockerhub proxy | 15:35 |
| fungi | but from a bandwidth utilization perspective we probably need flow accounting to really get a good picture of where the hot spots are | 15:36 |
| donnyd | I can probably get that data too | 15:37 |
| *** odicha has quit IRC | 15:37 | |
| fungi | in my experience, connection count doesn't correlate much to bandwidth utilization (nor does packet rate, though it tends to be a little closer at least) | 15:39 |
| fungi | the volume of dns queries is also higher than i would have expected, leading me to wonder whether 1. our unbound caches on the nodes don't mitigate it as much as we think, or 2. there are a lot of jobs which are bypassing unbound for lookups (for example with docker containers going straight to google dns or something) | 15:40 |
| donnyd | well the tenants default dns is set to 8.8.8.8 | 15:41 |
| donnyd | should i set it to something else? | 15:41 |
| fungi | nah, the job nodes ignore that anyway | 15:41 |
| fungi | we install unbound on them as caching forwarders for google dns and... opendns i think? | 15:42 |
| fungi | so some hits to those are expected because unbound will ask them on cache misses | 15:42 |
| fungi | and each node is starting with a cold cache | 15:43 |
| fungi | the main goal is to avoid them asking for the same records more than once over the course of a build | 15:43 |
| donnyd | I would think dns is pretty cheap anyways | 15:45 |
| fungi | yep. we've talked about possible ways to prime unbound's cache, but the options we came up with are either fragile or deemed unlikely to make a significant impact | 15:45 |
| fungi | and as you say, dns is the definition of a lightweight protoco | 15:45 |
| fungi | l | 15:45 |
| *** roman_g has joined #openstack-infra | 15:46 | |
| fungi | so focus there has generally been more on stability than performance | 15:46 |
| *** sreejithp has joined #openstack-infra | 15:46 | |
| fungi | (reducing repeated queries for the same records lowers the risk of a job failing on a dns lookup timeout or similar) | 15:46 |
| donnyd | I would think fixing the sheer volume of dockerhuby things could make a positive impact on job performance | 15:47 |
| donnyd | maybe mitm dockerhub | 15:47 |
| donnyd | which will be really quite challenging, but may pay off in less failed or faster jobs | 15:48 |
| fungi | even just proxying dockerhub is a challenge, so that would be a trick for sure | 15:48 |
| *** diablo_rojo has joined #openstack-infra | 15:48 | |
| fungi | their protocol is a rats nest of indirection | 15:48 |
| fungi | and identifying which addresses are theirs would be a constant game of whack-a-mole | 15:49 |
| donnyd | It would be great if we could get haproxy to cache some things for us in tcp mode | 15:49 |
| *** dtantsur|bbl is now known as dtantsur | 15:49 | |
| donnyd | because it will take care of grabbing the right certs from the backend | 15:50 |
| *** icarusfactor has joined #openstack-infra | 15:50 | |
| donnyd | just need it to saves the containers | 15:50 |
| *** apetrich has quit IRC | 15:51 | |
| fungi | the challenge there is mostly configuring the various operating systems to include it in their ca cert trusts, and also the various tools which provide their own trust chain and ignore the one from the operating system | 15:51 |
| fungi | (like python requests) | 15:51 |
| *** jamesmcarthur has quit IRC | 15:52 | |
| donnyd | yea, its a challenge for sure. And also something that we would have to keep up with | 15:52 |
| *** factor has quit IRC | 15:52 | |
| *** apetrich has joined #openstack-infra | 15:53 | |
| fungi | i suspect the tech debt that brings wouldn't be offset by the performance improvements, and we'll get more benefit from helping projects configure their jobs to go through the existing dockerhub proxy | 15:53 |
| donnyd | Im willing to bet you are on the money | 15:53 |
| *** armax has joined #openstack-infra | 15:55 | |
| *** icarusfactor has quit IRC | 15:55 | |
| *** whoami-rajat has joined #openstack-infra | 15:55 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Render console in js https://review.opendev.org/674368 | 15:55 |
| *** ralonsoh has quit IRC | 15:57 | |
| *** ralonsoh has joined #openstack-infra | 15:58 | |
| *** tesseract has quit IRC | 15:58 | |
| *** jbadiapa has joined #openstack-infra | 16:00 | |
| *** smarcet has quit IRC | 16:01 | |
| *** tesseract has joined #openstack-infra | 16:02 | |
| *** smarcet has joined #openstack-infra | 16:04 | |
| *** yamamoto has quit IRC | 16:05 | |
| *** gyee has joined #openstack-infra | 16:05 | |
| *** lucasagomes has quit IRC | 16:06 | |
| *** yamamoto has joined #openstack-infra | 16:06 | |
| *** yamamoto has quit IRC | 16:06 | |
| donnyd | fungi: so there are some things that are going through the registry | 16:07 |
| *** rh-jelabarre has quit IRC | 16:07 | |
| *** yamamoto has joined #openstack-infra | 16:07 | |
| donnyd | I am seeing a large amount of retransmissions though | 16:07 |
| *** igordc has joined #openstack-infra | 16:07 | |
| fungi | suggestive of packet loss somewhere | 16:08 |
| fungi | in what direction? | 16:08 |
| donnyd | possibly. I also don't know how to run ntoppng | 16:08 |
| donnyd | so it may be alerting me on nothing | 16:08 |
| *** yamamoto has quit IRC | 16:11 | |
| *** rh-jelabarre has joined #openstack-infra | 16:11 | |
| *** Lucas_Gray has quit IRC | 16:12 | |
| *** kjackal has quit IRC | 16:14 | |
| *** betherly has quit IRC | 16:18 | |
| *** ociuhandu has quit IRC | 16:22 | |
| *** jamesmcarthur has joined #openstack-infra | 16:22 | |
| *** ociuhandu has joined #openstack-infra | 16:22 | |
| *** jbadiapa has quit IRC | 16:23 | |
| *** ricolin_ has joined #openstack-infra | 16:23 | |
| *** ricolin has quit IRC | 16:26 | |
| *** betherly has joined #openstack-infra | 16:28 | |
| *** ociuhandu has quit IRC | 16:28 | |
| *** markvoelker has quit IRC | 16:28 | |
| *** ricolin_ is now known as ricolin | 16:29 | |
| *** psachin has quit IRC | 16:30 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Usability tweaks for the build page console https://review.opendev.org/675147 | 16:31 |
| *** betherly has quit IRC | 16:32 | |
| *** fghaas has quit IRC | 16:35 | |
| *** factor has joined #openstack-infra | 16:35 | |
| *** markvoelker has joined #openstack-infra | 16:38 | |
| *** gfidente has quit IRC | 16:38 | |
| *** tesseract has quit IRC | 16:39 | |
| *** ricolin has quit IRC | 16:40 | |
| *** lpetrut has quit IRC | 16:41 | |
| *** iurygregory has joined #openstack-infra | 16:41 | |
| *** markvoelker has quit IRC | 16:42 | |
| *** markvoelker has joined #openstack-infra | 16:44 | |
| *** ociuhandu has joined #openstack-infra | 16:44 | |
| openstackgerrit | Mohammed Naser proposed openstack/project-config master: projects: add vexxhost/ansible-role-openmanage https://review.opendev.org/675154 | 16:48 |
| *** ociuhandu has quit IRC | 16:50 | |
| *** igordc has quit IRC | 16:51 | |
| *** smarcet has quit IRC | 16:51 | |
| *** smarcet has joined #openstack-infra | 16:51 | |
| *** smarcet has left #openstack-infra | 16:52 | |
| *** dtantsur is now known as dtantsur|afk | 16:55 | |
| *** rpittau is now known as rpittau|afk | 16:56 | |
| *** diablo_rojo__ has joined #openstack-infra | 16:56 | |
| *** diablo_rojo has quit IRC | 16:56 | |
| *** ramishra has quit IRC | 16:57 | |
| *** ykarel is now known as ykarel|away | 16:57 | |
| *** derekh has quit IRC | 16:59 | |
| fungi | do we have request logging for gitea's internal httpd? all i can find looks like error logging | 17:00 |
| *** ociuhandu has joined #openstack-infra | 17:01 | |
| *** betherly has joined #openstack-infra | 17:03 | |
| openstackgerrit | Jeremy Stanley proposed opendev/zone-opendev.org master: Temporarily remove AAAA RR for fortnebula mirror https://review.opendev.org/675156 | 17:06 |
| fungi | donnyd: ^ | 17:06 |
| *** e0ne has quit IRC | 17:06 | |
| fungi | infra-root: ^ | 17:06 |
| *** betherly has quit IRC | 17:08 | |
| mnaser | .purely out of curiosity. | 17:09 |
| mnaser | https://review.opendev.org/#/c/621127/ | 17:09 |
| mnaser | why/how is that updated 23 minutes ago | 17:09 |
| *** igordc has joined #openstack-infra | 17:10 | |
| *** hwoarang has quit IRC | 17:11 | |
| mnaser | and wondering/hoping if its possible to land https://review.opendev.org/#/c/675154/ -- just a simple opendev project addition | 17:12 |
| *** jamesmcarthur has quit IRC | 17:12 | |
| fungi | i think if someone deletes their vote on a change, it's not reflected in the commentary timeline but still touches the autodatestamp column for whatever table(s) that's coming from | 17:12 |
| *** hwoarang has joined #openstack-infra | 17:13 | |
| *** pkopec is now known as pkopec|afk | 17:13 | |
| fungi | or maybe it's addition/deletion of requested reviewers which does it | 17:13 |
| fungi | i don't know whether there's a sufficient audit trail to track that down | 17:13 |
| *** udesale has quit IRC | 17:14 | |
| fungi | mnaser: i've approved 675154. assuming you've done enough of these that you know you're starting with an empty repo (other than .gitreview file) since you didn't include an import url | 17:15 |
| *** altlogbot_3 has quit IRC | 17:17 | |
| mnaser | thanks fungi and mordred :) | 17:21 |
| *** altlogbot_1 has joined #openstack-infra | 17:23 | |
| *** betherly has joined #openstack-infra | 17:24 | |
| corvus | fungi, donnyd: i take we we expect the pnat in fn to handle all the node->mirror connections? | 17:25 |
| openstackgerrit | Merged openstack/project-config master: projects: add vexxhost/ansible-role-openmanage https://review.opendev.org/675154 | 17:26 |
| *** ginopc has quit IRC | 17:27 | |
| fungi | corvus: that's a good question, but i believe they're routed internally since the mirror uses an rfc-1918 address which is behind a fip | 17:28 |
| *** jamesmcarthur has joined #openstack-infra | 17:28 | |
| fungi | though... the address in dns is the fip so i don't know what's involved in the triangle there | 17:28 |
| fungi | i guess they do have to trombone through the router, but it's not clear whether there's any translation of the job node addresses in that | 17:28 |
| *** betherly has quit IRC | 17:29 | |
| fungi | donnyd: ^ can you clarify what the internal v4 routing situation is for that communication? | 17:29 |
| *** ociuhandu has quit IRC | 17:29 | |
| donnyd | yea, it should be ok | 17:30 |
| corvus | i went ahead and +3d, but just wanted to double check that we weren't about to fill up a nat table or something | 17:30 |
| *** ykarel|away has quit IRC | 17:30 | |
| fungi | is there any pat/overload nat of node addresses when communicating with the mirror over v4? | 17:30 |
| *** altlogbot_1 has quit IRC | 17:31 | |
| donnyd | If i was running 1k servers i would be worried.. but 70 -100 should be fine | 17:31 |
| fungi | i guess we'll know soon enough once that takes effect, since we can check the apache logs | 17:31 |
| fungi | if it does end up being disruptive donnyd can drop the quota or we can lower max-servers in config while we investigate further | 17:32 |
| *** jamesmcarthur has quit IRC | 17:32 | |
| fungi | thanks corvus! | 17:32 |
| *** jamesmcarthur has joined #openstack-infra | 17:34 | |
| *** altlogbot_0 has joined #openstack-infra | 17:35 | |
| openstackgerrit | Merged opendev/zone-opendev.org master: Temporarily remove AAAA RR for fortnebula mirror https://review.opendev.org/675156 | 17:35 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Don't always show expansion option on build console https://review.opendev.org/675163 | 17:36 |
| donnyd | so we should know in the next few hours if it gets any better | 17:37 |
| *** mattw4 has quit IRC | 17:38 | |
| *** ociuhandu has joined #openstack-infra | 17:39 | |
| *** jamesmcarthur has quit IRC | 17:39 | |
| donnyd | also I would like to note, that with proper monitoring of the mirror now running. I am seeing the numbers one would expect | 17:41 |
| fungi | great! | 17:43 |
| *** ociuhandu has quit IRC | 17:43 | |
| fungi | you mean as far as aggregate throughput? | 17:44 |
| *** betherly has joined #openstack-infra | 17:44 | |
| mnaser | ok i've dug for way too long on this one | 17:46 |
| mnaser | how often does the crons run (or around when?) for infra? | 17:46 |
| donnyd | yea fungi | 17:47 |
| donnyd | around 300M/s | 17:47 |
| donnyd | thats about what a single thread can do through the edge fw | 17:47 |
| *** jamesmcarthur has joined #openstack-infra | 17:49 | |
| *** betherly has quit IRC | 17:49 | |
| AJaeger | mnaser: the usual cron for project creation etc? I think every 30 mins - and it needs nearly as long. So, wait an hour after merge is my rule of thumb ;) | 17:49 |
| fungi | mnaser: it's a bit erratic. every 15 minutes the cronjob is triggered. if there is not a config management update in progress already then a new one is initiated which updates the state of configuration it's working from and begins to push stuff out to the servers. last time i looked the runs were taking around 30 minutes to complete so on a slow day updates are happening ~ every 45 minutes i think | 17:49 |
| mnaser | oh okay, i just have most of my code written up so me wants to punt it :p | 17:50 |
| mnaser | merged at :26 so ill wait a little longer | 17:50 |
| *** dpawlik has joined #openstack-infra | 17:50 | |
| *** smarcet has joined #openstack-infra | 17:50 | |
| AJaeger | thanks for correction, fungi | 17:51 |
| fungi | eventually when we get to the point where we're able to continuously deploy things via zuul jobs, it should be faaar more immediate | 17:51 |
| fungi | right now it's looking like a 45-minute cadence, yes | 17:53 |
| fungi | the current run began at 17:45z and the one before it at 17:00z | 17:53 |
| fungi | 675154 merged at 17:26z so should get applied by the current run which is due to finish probably shortly after 18:15z (runs seem to be taking anywhere from less than a minute over 30 to as much as 40) | 17:55 |
| fungi | however the gerrit/gitea updates can happen anywhere in there | 17:56 |
| *** jamesmcarthur has quit IRC | 18:00 | |
| *** kopecmartin is now known as kopecmartin|off | 18:00 | |
| donnyd | https://usercontent.irccloud-cdn.com/file/vaSnVgjG/Screenshot%20from%202019-08-07%2014-01-00.png | 18:01 |
| *** jamesmcarthur has joined #openstack-infra | 18:01 | |
| *** sgw has quit IRC | 18:03 | |
| *** betherly has joined #openstack-infra | 18:04 | |
| fungi | yeah, looks like that's dockerhub traffic going through the apache cache | 18:04 |
| donnyd | so it looks to me like some jobs grab containers from the mirror | 18:04 |
| fungi | https://opendev.org/opendev/system-config/src/branch/master/modules/openstack_project/templates/mirror.vhost.erb#L351 | 18:04 |
| *** Vadmacs has joined #openstack-infra | 18:05 | |
| fungi | some jobs do indeed grab containers from the mirror | 18:05 |
| donnyd | if more jobs would, they would likely see better performance... 500M/s is much mo betta than you get with docker hub | 18:07 |
| fungi | yup, that's definitely the reason we added it | 18:08 |
| donnyd | fungi: check out the flows tables now | 18:09 |
| *** betherly has quit IRC | 18:09 | |
| mnaser | hmm | 18:09 |
| mnaser | has the new project code been changed recently? | 18:09 |
| donnyd | no / low retransmissions | 18:10 |
| mnaser | okay, nevermind, it looks like the project gets created initially and then rewritten with .gitreview | 18:10 |
| * mnaser saw it pop up without .gitreview | 18:10 | |
| fungi | donnyd: looks like the dns update took effect too | 18:11 |
| fungi | yeah | 18:11 |
| fungi | so i guess correlated | 18:11 |
| donnyd | yea, all the new instances are moving over | 18:11 |
| fungi | oof, getting some real gnarly lightning out over the water at the moment. my connection could go at any moment (or my power for that matter) | 18:12 |
| fungi | just a heads-up in case i go silent for a while | 18:12 |
| donnyd | im in the same boat | 18:12 |
| fungi | i recommend coming ashore | 18:12 |
| donnyd | and my generator is still not hooked up | 18:12 |
| donnyd | LOL | 18:12 |
| fungi | mostly just crossing my fingers for no more waterspouts | 18:13 |
| *** diablo_rojo__ is now known as diablo_rojo | 18:13 | |
| *** bhavikdbavishi has quit IRC | 18:16 | |
| mnaser | http://zuul.opendev.org/t/opendev/status | 18:17 |
| mnaser | job is in "error" status | 18:17 |
| * mnaser has never seen that status before | 18:17 | |
| * donnyd crosses fingers its not FN | 18:17 | |
| mnaser | i wonder if the "job" wasn't installed yet | 18:17 |
| mnaser | http://zuul.opendev.org/t/opendev/jobs | 18:17 |
| mnaser | or somehow the repo wasnt registered yet | 18:18 |
| mnaser | the link also seems to be broken, it points to http://zuul.opendev.org/t/opendev/ansible-role-openmanage-integration-centos-7 which simply just redirets back to / | 18:18 |
| donnyd | also shouldn't it be listed here ? http://zuul.opendev.org/t/opendev/builds?project=vexxhost/ansible-role-openmanage | 18:20 |
| donnyd | Not Found (404) http://zuul.opendev.org/api/tenant/opendev/job/ansible-role-openmanage-integration-centos-7 | 18:21 |
| *** sgw has joined #openstack-infra | 18:21 | |
| mnaser | ya cause it hasnt merged yet | 18:21 |
| donnyd | ansible-role-openmanage-integration-centos-7 ansible-role-openmanage-integration-centos-7 : ERROR Unable to find playbook /var/lib/zuul/builds/709e26dcbc4540428bbce3753a55c3c3/untrusted/project_0/opendev.org/vexxhost/ansible-role-openmanage/tests/post-run.yaml | 18:21 |
| fungi | possible you raced zuul updating? | 18:22 |
| mnaser | maybe | 18:22 |
| *** spsurya has quit IRC | 18:22 | |
| mnaser | no i think i goofed | 18:22 |
| fungi | a recheck may suddenly work | 18:22 |
| mnaser | ERROR Unable to find playbook /var/lib/zuul/builds/709e26dcbc4540428bbce3753a55c3c3/untrusted/project_0/opendev.org/vexxhost/ansible-role-openmanage/tests/post-run.yaml | 18:22 |
| fungi | ahh | 18:22 |
| fungi | does that playbook not exist in the repo? | 18:23 |
| fungi | (or in the change?) | 18:23 |
| *** betherly has joined #openstack-infra | 18:25 | |
| fungi | ahh, yep, wasn't included. so that would be why | 18:25 |
| *** betherly has quit IRC | 18:30 | |
| *** Vadmacs has quit IRC | 18:32 | |
| *** mattw4 has joined #openstack-infra | 18:33 | |
| *** mattw4 has quit IRC | 18:44 | |
| *** mattw4 has joined #openstack-infra | 18:44 | |
| *** betherly has joined #openstack-infra | 18:45 | |
| *** e0ne has joined #openstack-infra | 18:49 | |
| zbr | AJaeger: not sure about you comment on https://review.opendev.org/#/c/667698/7 -- you want that sentence removed or you want the rule to be updated? | 18:50 |
| *** betherly has quit IRC | 18:50 | |
| *** smarcet has quit IRC | 18:55 | |
| *** jamesmcarthur has quit IRC | 19:03 | |
| AJaeger | zbr: "Remove the 206 rule" is not part of that change, so either add it back in or update commit message | 19:04 |
| AJaeger | zbr: you don't update tox.ini | 19:05 |
| zbr | ok. so either one is ok. just wanted to know. | 19:05 |
| AJaeger | zbr: I prefer to update tox.ini as it was initially... | 19:06 |
| AJaeger | zbr: and if not, let's do it as followup for once ansible-lint is ready | 19:06 |
| zbr | glad we are on the same page | 19:06 |
| *** jcoufal_ has joined #openstack-infra | 19:15 | |
| *** jcoufal has quit IRC | 19:16 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: Be consistent about spaces before and after vars https://review.opendev.org/667698 | 19:16 |
| *** jamesmcarthur has joined #openstack-infra | 19:16 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: Be consistent about spaces before and after vars https://review.opendev.org/667698 | 19:17 |
| *** jcoufal_ has quit IRC | 19:19 | |
| *** smarcet has joined #openstack-infra | 19:24 | |
| *** smarcet has quit IRC | 19:27 | |
| *** smarcet has joined #openstack-infra | 19:29 | |
| *** dave-mccowan has joined #openstack-infra | 19:32 | |
| *** jamesmcarthur has quit IRC | 19:35 | |
| *** lpetrut has joined #openstack-infra | 19:35 | |
| *** jamesmcarthur has joined #openstack-infra | 19:36 | |
| *** vesper11 has quit IRC | 19:37 | |
| *** betherly has joined #openstack-infra | 19:46 | |
| *** e0ne has quit IRC | 19:46 | |
| *** betherly has quit IRC | 19:51 | |
| *** dpawlik has quit IRC | 19:53 | |
| *** betherly has joined #openstack-infra | 19:55 | |
| *** Goneri has joined #openstack-infra | 20:00 | |
| *** betherly has quit IRC | 20:00 | |
| *** markvoelker has quit IRC | 20:01 | |
| *** lpetrut has quit IRC | 20:01 | |
| *** factor has quit IRC | 20:03 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Don't always show expansion option on build console https://review.opendev.org/675163 | 20:03 |
| *** markvoelker has joined #openstack-infra | 20:03 | |
| *** factor has joined #openstack-infra | 20:03 | |
| *** kjackal has joined #openstack-infra | 20:04 | |
| *** factor has quit IRC | 20:05 | |
| *** factor has joined #openstack-infra | 20:05 | |
| *** betherly has joined #openstack-infra | 20:16 | |
| *** whoami-rajat has quit IRC | 20:17 | |
| *** smarcet has joined #openstack-infra | 20:21 | |
| *** betherly has quit IRC | 20:21 | |
| *** factor has quit IRC | 20:32 | |
| *** factor has joined #openstack-infra | 20:32 | |
| mordred | corvus, fungi: in written english, is Open Source intended to always be capitalized? Like "I was working on an Open Source project" ? | 20:33 |
| *** factor has quit IRC | 20:35 | |
| *** factor has joined #openstack-infra | 20:36 | |
| corvus | mordred: this is a delightful rabbit hole! | 20:36 |
| mordred | corvus: do you mean Rabbit Hole? | 20:36 |
| corvus | if we're to consider potential authorities on the subject, the OSI comes to mind | 20:36 |
| * Shrews and SpamapS and AJaeger are fond of excessive capitalization | 20:36 | |
| mordred | corvus: wow - https://opensource.org/ inconsistently applies capitalization right on the main page | 20:37 |
| mordred | or - rather, I guess they don't - they only capitalize when in the context of Open Source Initiative or Open Source Definition | 20:37 |
| corvus | on their about page ( https://opensource.org/about ) they have "Open Source" and "Open-Source". the OSD itself ( https://opensource.org/osd ) has "Open source" (and by implication, may have "open source" if not at the start of a sentence) and "open-source". | 20:37 |
| mordred | all other occurances are lower | 20:37 |
| corvus | mordred: yep | 20:37 |
| mordred | well - the hyphens definitely seem wrong | 20:38 |
| corvus | mordred: so, as a potential authority, they are not especially useful, except to say, maybe it doesn't matter that much. | 20:38 |
| fungi | i tend to think that when it's used descriptively it should follow normal sentence capitalization rules | 20:38 |
| corvus | mordred: or, perhaps, they're the most correct :) | 20:38 |
| donnyd | oPen souRce is not a proper noun | 20:38 |
| mordred | although - there's also this: "Please contact the OSI if you are interested in having us help you develop, arrange, or conduct educational conferences, programs, courses of instruction, and/or online educational seminars relating to Open Source (non-proprietary) software." | 20:39 |
| donnyd | so its lower case | 20:39 |
| fungi | much like free software as a description is lower-case, but as part of the name of the fsf it uses proper noun casing | 20:39 |
| mordred | yeah | 20:39 |
| corvus | i think capitalizing it might support their trademark assertions (which i think they've largely abandoned that?) | 20:39 |
| donnyd | correct fungi . The OSI is a proper noun because its an organization | 20:39 |
| donnyd | proper noun - a name used for an individual person, place, or organization, spelled with initial capital letters, e.g., Larry, Mexico, and Boston Red Sox. | 20:40 |
| mordred | in German, all Nouns are Proper | 20:40 |
| *** Goneri has quit IRC | 20:40 | |
| fungi | on the other hand, we could just embrace the germanic origins of the english language and go with german capitalization rules, which i've never been able to completely figure out | 20:41 |
| mordred | fungi: they're really easy | 20:41 |
| mordred | "is it a noun? if yes, capitalize it. if no, do not capitalize it" | 20:41 |
| fungi | nice. i probably knew that at one point, but university was decades ago now | 20:41 |
| fungi | oh so many beers ago | 20:41 |
| donnyd | mordred: You are going to need to be more specific on your dialect of English | 20:41 |
| mordred | Wir trinken grossen Bieren | 20:42 |
| fungi | ja | 20:42 |
| mordred | so it follows that Beer should always be capitalized, which is probably the reason for the general rule :) | 20:43 |
| AJaeger | mordred: In German, it's Open Source ;) We always capitalize nouns. | 20:43 |
| mordred | AJaeger: this is probably why I want to capitalize it :) | 20:43 |
| AJaeger | and now I finished reading backscroll and see you discussed it already... | 20:44 |
| AJaeger | means, I can go back to my beer ;) | 20:44 |
| corvus | mordred: on the hyphenation question -- it's a compound adjective. (opensource, open-source, open source) are all potentially correct. i don't think that 'opensource' has achieved consensus yet, which leaves the last two. hyphenated is the least ambiguous, but i think the unhyphenated form is common enough to be used (and may be more common than hyphenated) | 20:44 |
| fungi | i concur | 20:45 |
| AJaeger | happy to add an entry to https://docs.openstack.org/doc-contrib-guide/writing-style/word-choice.html ;) | 20:45 |
| fungi | i would prefer if open-source were more common, but open source seems to have won out instead | 20:45 |
| fungi | (as an adjectival form) | 20:45 |
| donnyd | https://www.merriam-webster.com/dictionary/open-source | 20:45 |
| corvus | yeah, if i'm using google n-gram correctly, "open source" is more common than "open source" | 20:45 |
| corvus | grr | 20:46 |
| donnyd | even if you punch in opensource it adds the hypen back in there | 20:46 |
| corvus | yeah, if i'm using google n-gram correctly, "open source" is more common than "open-source" | 20:46 |
| donnyd | https://www.macmillandictionary.com/us/spellcheck/american/?q=opensource | 20:46 |
| fungi | it's not in my scrabble dictionary either | 20:47 |
| fungi | also it would be hard to fit on the board | 20:48 |
| corvus | mordred: i think i'd employ "open source" but i won't blink at "open-source" | 20:48 |
| mordred | AJaeger: given the collective human-hours spent on the topic so far, it seems wrong _not_ to add an entry there :) | 20:48 |
| donnyd | two out of three add the hyphen https://www.dictionary.com/browse/open-source | 20:48 |
| donnyd | this is my favorite so far oh-puh n-sawrs | 20:49 |
| corvus | yeah, that reminds me of the days of "open sores" | 20:49 |
| AJaeger | mordred: happy to +2 an addition. But you know, it will lead to 100s of changes to all OpenDev projects to write open source correctly ;) | 20:49 |
| mordred | AJaeger: that should make the patch bots super happy | 20:50 |
| corvus | AJaeger: donnyd keeps asing for more test load :) | 20:50 |
| fungi | yep, this is why i have a personal preference for the hyphenated form, it's grammatically correct... but the internets care not for grammar | 20:50 |
| donnyd | corvus: but only on friday afternoons | 20:50 |
| corvus | "please only submit hyphenation update patches on friday afternoons" | 20:50 |
| fungi | i still capitalize Web and hyphenate E-mail in formal use | 20:51 |
| AJaeger | yeah, a short job for a friday afternoon for a script - and a long stresstest ;) | 20:51 |
| corvus | fungi: https://www-cs-faculty.stanford.edu/~knuth/email.html | 20:51 |
| *** tdasilva has joined #openstack-infra | 20:51 | |
| AJaeger | Lol, the title really fits ;) | 20:51 |
| corvus | (there's a pgraph about that at the bottom) | 20:52 |
| fungi | corvus: too soon! ;) | 20:52 |
| *** guoqiao has joined #openstack-infra | 20:52 | |
| corvus | fungi: you're going to wait until it's finally replaced by google wave, yeah? | 20:52 |
| corvus | (google-wave?) | 20:53 |
| fungi | pretty sure that already happened, right? | 20:53 |
| mordred | corvus: yet another reason to aspire to be knuth | 20:53 |
| mordred | I think I've also reached the point where I've had email for long enough | 20:54 |
| * AJaeger does not have a secretary like he has ;/ | 20:54 | |
| mordred | AJaeger: I think the secretary in this case could simply be /dev/null | 20:54 |
| mordred | it's also web scale | 20:54 |
| corvus | i'm rather happy to be able to communicate with folks in our community over email | 20:55 |
| donnyd | you could just use mongodb mordred https://www.youtube.com/watch?v=b2F-DItXtZs | 20:55 |
| fungi | /dev/null is my copilot | 20:56 |
| * AJaeger sends /dev/zero to talk to your /dev/null | 20:57 | |
| *** ralonsoh has quit IRC | 20:58 | |
| * donnyd apparently killed this fun conversation | 20:59 | |
| AJaeger | fear I send a dos ;/ | 21:00 |
| * AJaeger waves good night | 21:01 | |
| fungi | good-night! don't let the fork-bombs bite! | 21:02 |
| AJaeger | only midges ;) | 21:03 |
| *** yamamoto has joined #openstack-infra | 21:03 | |
| donnyd | fungi: do you need me to leave open the network stats ? | 21:05 |
| *** slaweq_ has quit IRC | 21:06 | |
| fungi | donnyd: nah, i need to move on to all manner of other things anyway | 21:06 |
| donnyd | ok cool. I will keep them running just in case we need the data later on | 21:07 |
| *** diablo_rojo has quit IRC | 21:07 | |
| *** diablo_rojo has joined #openstack-infra | 21:08 | |
| donnyd | might be nice to have some longer term stats on what exactly the jobs are doing | 21:08 |
| *** yamamoto has quit IRC | 21:08 | |
| fungi | one more observation is that the stray dockerhub traffic seemed to be coming from instances with "centos-7" in their names, so that could give us some idea of what's hitting it. we we want to dig deeper i can correlate instances with zuul logs to identify what jobs they ran for what projects | 21:08 |
| *** jcoufal has joined #openstack-infra | 21:08 | |
| fungi | er, if we want to dig deeper i mean | 21:08 |
| donnyd | It seems to me the majority of the issues are the tempest jobs that timeout | 21:09 |
| donnyd | but they aren't just timing out here, they seem to do it everywhere | 21:09 |
| * donnyd needs the foods | 21:10 | |
| *** slaweq_ has joined #openstack-infra | 21:11 | |
| pabelanger | there looks to be a few projects running non-voting jobs in gate. swift, barbican and python-tripleo for example | 21:12 |
| fungi | pabelanger: yeah, i have a to do item i need to get around to either writing a story or just trying to implement an option you can set on a pipeline to simply omit non-voting jobs from being executed | 21:17 |
| *** slaweq_ has quit IRC | 21:17 | |
| pabelanger | fungi: yah, only looked because I was getting impatient waiting for a node to come online for testing | 21:18 |
| fungi | because being able to reuse lists of jobs in multiple pipelines or not have to break out the content of a project-template defined somewhere else just to be able to set a check job non-voting would save us some compute time, but more importantly would allow us to stop caring about it | 21:18 |
| *** markvoelker has quit IRC | 21:23 | |
| *** vesper11 has joined #openstack-infra | 21:24 | |
| *** jcoufal has quit IRC | 21:25 | |
| *** auristor has quit IRC | 21:26 | |
| *** jtomasek has quit IRC | 21:26 | |
| *** tjgresha has quit IRC | 21:28 | |
| *** jamesmcarthur has quit IRC | 21:29 | |
| *** jamesmcarthur has joined #openstack-infra | 21:29 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Adjust results headings in build console page https://review.opendev.org/675203 | 21:30 |
| *** sreejithp has quit IRC | 21:30 | |
| corvus | fungi: sgtm | 21:31 |
| *** tjgresha has joined #openstack-infra | 21:31 | |
| *** tjgresha has quit IRC | 21:32 | |
| *** jamesmcarthur has quit IRC | 21:34 | |
| *** factor has quit IRC | 21:35 | |
| *** factor has joined #openstack-infra | 21:35 | |
| *** markvoelker has joined #openstack-infra | 21:38 | |
| *** jamesmcarthur has joined #openstack-infra | 21:40 | |
| *** diablo_rojo has quit IRC | 21:46 | |
| *** markvoelker has quit IRC | 21:50 | |
| timburke | pabelanger, i'd note that swift's one non-voting gate job (swift-upload-image) is to do with publishing container images to dockerhub -- we did that to ensure that dockerhub issues wouldn't prevent us from being able to land code. maybe there's a better way to achieve that, though? | 21:57 |
| *** pkopec|afk has quit IRC | 22:00 | |
| *** diablo_rojo has joined #openstack-infra | 22:05 | |
| *** diablo_rojo has quit IRC | 22:10 | |
| *** slaweq_ has joined #openstack-infra | 22:11 | |
| *** rcernin has joined #openstack-infra | 22:15 | |
| *** slaweq_ has quit IRC | 22:16 | |
| *** markvoelker has joined #openstack-infra | 22:19 | |
| *** rlandy|rover is now known as rlandy|rover|bbl | 22:22 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Scroll log line anchor into view https://review.opendev.org/675220 | 22:23 |
| corvus | timburke: could probably put the role in an ansible rescue block, so if there's any error, the job still succeeds. but of course, you may go longer without noticing in that case. | 22:24 |
| *** mriedem has quit IRC | 22:24 | |
| corvus | timburke: i think you found a compelling reason to allow non-voting jobs in gate :) | 22:24 |
| fungi | the trick there is noticing when the gate job failed, i expect | 22:27 |
| corvus | yeah, it's not easy, but it's still way easier than if it says "SUCCESS" :) | 22:28 |
| fungi | yup | 22:29 |
| corvus | just testing out that change ^ about scrolling to log lines... | 22:31 |
| corvus | it seems to work fine, but in testing it, i found a devstack job, and scrolled 1/4 of the way down, and the line number i got was #27777 | 22:31 |
| corvus | the last line is #115574 | 22:31 |
| corvus | maybe it's just me... but maybe devstack jobs are outputting a bit much to the console log? | 22:32 |
| fungi | yup | 22:32 |
| corvus | i thought there was a thing that happened a while back that just output summary info to the console, and the rest of the stuff went to a file | 22:33 |
| fungi | yeah, i really don't know. would have to ask #openstack-qa | 22:34 |
| fungi | i agree that seems excessive by a few orders of magnitude | 22:35 |
| *** smarcet has quit IRC | 22:37 | |
| *** markvoelker has quit IRC | 22:38 | |
| *** ekultails has quit IRC | 22:38 | |
| *** jamesmcarthur has quit IRC | 22:41 | |
| *** smarcet has joined #openstack-infra | 22:42 | |
| *** jamesmcarthur has joined #openstack-infra | 22:42 | |
| *** smarcet has quit IRC | 22:44 | |
| *** notmyname has quit IRC | 22:45 | |
| *** jamesmcarthur has quit IRC | 22:47 | |
| *** jamesmcarthur has joined #openstack-infra | 22:50 | |
| *** markvoelker has joined #openstack-infra | 22:53 | |
| *** jamesmcarthur has quit IRC | 22:57 | |
| *** tdasilva has quit IRC | 23:02 | |
| *** tdasilva has joined #openstack-infra | 23:02 | |
| *** aaronsheffield has quit IRC | 23:02 | |
| *** dchen has joined #openstack-infra | 23:05 | |
| *** markvoelker has quit IRC | 23:08 | |
| *** yamamoto has joined #openstack-infra | 23:08 | |
| *** slaweq_ has joined #openstack-infra | 23:11 | |
| *** michael-beaver has quit IRC | 23:14 | |
| *** betherly has joined #openstack-infra | 23:15 | |
| *** notmyname has joined #openstack-infra | 23:15 | |
| *** slaweq_ has quit IRC | 23:16 | |
| ianw | corvus / fungi: yeah, usually when you look at it there's something or other that's popped up and floods the logs and can be culled; e.g. just from now -> https://review.opendev.org/675226 oscwrap: make a little quieter | 23:18 |
| ianw | i feel like the core code does pretty good balance between tracing and not showing every loop iteration, but it's a constant battle | 23:18 |
| *** betherly has quit IRC | 23:20 | |
| *** tosky has quit IRC | 23:20 | |
| *** smarcet has joined #openstack-infra | 23:22 | |
| *** auristor has joined #openstack-infra | 23:23 | |
| *** smarcet has quit IRC | 23:27 | |
| *** jamesmcarthur has joined #openstack-infra | 23:28 | |
| johnsom | In our jobs we use DIB to build the amphora image. When DIB pip installs the requirements I see it going out to https://files.pythonhosted.org which seems to fail regularly. Is there a mirror configuration we are missing? https://logs.opendev.org/59/674659/3/check/octavia-v2-dsvm-scenario/2d46c1a/controller/logs/dib-build/amphora-x64-haproxy.qcow2_log.txt.gz#_2019-08-06_00_57_43_349 | 23:32 |
| *** jamesmcarthur has quit IRC | 23:33 | |
| *** igordc has quit IRC | 23:35 | |
| *** betherly has joined #openstack-infra | 23:37 | |
| *** markvoelker has joined #openstack-infra | 23:38 | |
| ianw | johnsom: the pip inside the chroot won't be configured to use any local mirroring, i don't believe dib has anything to set that up | 23:38 |
| ianw | we have some elements to setup the chroot to use the distro mirrors (apt/yum) inside the chroot, but they don't set pip ... they probably *could* though, with sufficient fiddling | 23:39 |
| johnsom | Yeah, I saw this: DIB_PYPI_MIRROR_URL and wondered if we should be setting that to some infra location. | 23:40 |
| ianw | hrm, yeah i'm not sure, i've never tried using the pypi element | 23:42 |
| *** markvoelker has quit IRC | 23:43 | |
| *** jamesmcarthur has joined #openstack-infra | 23:43 | |
| johnsom | Me either. | 23:43 |
| johnsom | I'm willing to experiment if we even have mirrors for the python stuff. I don't know if we do or not. | 23:44 |
| ianw | it doesn't look like it's had any updates since 2013, so it's either perfect, or in a state of some bitrot ... taking bets :) | 23:45 |
| *** betherly has quit IRC | 23:45 | |
| *** jamesmcarthur has quit IRC | 23:46 | |
| ianw | we do, for gate jobs you're going to want to get the mirrors from the mirror_info script -> https://opendev.org/openstack/openstack-zuul-jobs/src/branch/master/roles/mirror-info/templates/mirror_info.sh.j2#L51 | 23:46 |
| ianw | you can checkout the openstack-ci-mirrors element, which we use in the gate to set distro mirrors -> https://opendev.org/openstack/diskimage-builder/src/branch/master/diskimage_builder/elements/openstack-ci-mirrors/environment.d/11-dib-distribution-mirror.bash | 23:48 |
| johnsom | $100 on rot | 23:48 |
| fungi | anytime you bet against bitrot, it's a sucker bet | 23:49 |
| johnsom | Ok, thanks for the info. I may play around with this stuff and see if I can improve our image build success. | 23:50 |
| ianw | so it might be as easy as setting DIB_PYPI_MIRROR_URL=$NODEPOOL_PYPI_MIRROR in 11-dib-distribution-mirror.bash, and including the pypi element | 23:51 |
| *** tdasilva has quit IRC | 23:51 | |
| *** mattw4 has quit IRC | 23:51 | |
| ianw | depending on what you do with the image, you also want to clear it out so the final image isn't referring to mirrors | 23:52 |
| *** tdasilva has joined #openstack-infra | 23:52 | |
| *** sthussey has quit IRC | 23:53 | |
| fungi | yeah, the majority of our job interaction with python package downloading hits our caching pypi proxy, which significantly shields us from random pypi/fastly connectivity problems | 23:54 |
| fungi | but replicating the default configs we install on job nodes into things like chroots and containers is tricky to remember (or even figure out) | 23:55 |
| johnsom | I'm guessing that is not a transparent cache that snags requests like https://files.pythonhosted.org | 23:55 |
| *** tdasilva has quit IRC | 23:55 | |
| *** tdasilva has joined #openstack-infra | 23:56 | |
| ianw | johnsom: no it's not transparent, for jobs it's setup by https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/configure-mirrors where you can inspect the pip and other config files installed | 23:58 |
| ianw | so yeah, the trick is then propagating that info into whatever nested thing is being built too, that's where it's up to the job | 23:59 |
| johnsom | Ok, I might poke around at this tomorrow and see what I get. | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
