| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: Add ensure-rust role https://review.opendev.org/746423 | 00:06 |
|---|---|---|
| ianw | seems it's just out of sync http://mirror.aarnet.edu.au/pub/fedora/linux/updates/31/Everything/x86_64/repodata/repomd.xml | 00:16 |
| ianw | doesn't line up | 00:16 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: Add ensure-rust role https://review.opendev.org/746423 | 00:19 |
| ianw | mnaser: fyi the POST_FAILURES for vexxhost seem to be back with ^ | 00:23 |
| ianw | also seems like we've lost the log streamer on some executors | 00:26 |
| ianw | although it's saying | 00:27 |
| ianw | Build ID 575534f236fd445498760c09dad0c525 not found | 00:27 |
| ianw | ... rather than just end of stream ... that's odd i think | 00:27 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: Add ensure-rust role https://review.opendev.org/746423 | 00:30 |
| ianw | mnaser: update, it worked now. so ... intermittent maybe? | 00:53 |
| *** DSpider has quit IRC | 01:52 | |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder master: update gentoo to allow building arm64 images https://review.opendev.org/746000 | 03:42 |
| prometheanfire | ianw: ^ :D | 03:42 |
| prometheanfire | fungi: added arm64 systemd support now too | 03:42 |
| ianw | cool | 03:55 |
| *** raukadah is now known as chkumar|rover | 04:20 | |
| *** ysandeep|away is now known as ysandeep | 04:34 | |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder master: update gentoo to allow building arm64 images https://review.opendev.org/746000 | 05:28 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/project-config master: Normalize projects.yaml https://review.opendev.org/746439 | 06:15 |
| openstackgerrit | Merged openstack/project-config master: Normalize projects.yaml https://review.opendev.org/746439 | 07:06 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [wip] Fedora 32 support https://review.opendev.org/737217 | 07:06 |
| *** DSpider has joined #opendev | 07:23 | |
| *** hashar has joined #opendev | 07:49 | |
| *** dtantsur|afk is now known as dtantsur | 07:55 | |
| *** tosky has joined #opendev | 07:56 | |
| *** moppy has quit IRC | 08:01 | |
| *** moppy has joined #opendev | 08:01 | |
| *** hashar has quit IRC | 08:10 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [wip] Fedora 32 support https://review.opendev.org/737217 | 08:31 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Fedora 32 support https://review.opendev.org/737217 | 08:48 |
| *** hashar has joined #opendev | 08:49 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Fedora 32 support https://review.opendev.org/737217 | 09:17 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Fedora 32 support https://review.opendev.org/737217 | 09:40 |
| *** tkajinam has quit IRC | 09:55 | |
| *** priteau has joined #opendev | 10:40 | |
| *** ryohayakawa has quit IRC | 10:53 | |
| *** tkajinam has joined #opendev | 11:27 | |
| *** weshay_ is now known as weshay|ruck | 11:36 | |
| *** Baragaki has left #opendev | 12:35 | |
| openstackgerrit | Thierry Carrez proposed opendev/system-config master: Make ask.openstack.org read-only https://review.opendev.org/746497 | 13:08 |
| *** dtantsur has quit IRC | 13:32 | |
| openstackgerrit | Merged openstack/project-config master: Retire networking-l2gw and networking-l2gw-tempest-plugin https://review.opendev.org/738030 | 13:36 |
| *** dtantsur has joined #opendev | 13:38 | |
| *** hashar has quit IRC | 13:51 | |
| *** hashar has joined #opendev | 13:54 | |
| openstackgerrit | Benjamin Schanzel proposed zuul/zuul-jobs master: Multi Node/Context support for mirroring Git workspaces via kubectl https://review.opendev.org/746508 | 13:56 |
| *** tkajinam has quit IRC | 14:06 | |
| mnaser | infra-root: i think maybe an gitea instance is unhappy | 14:16 |
| mnaser | http://lists.openstack.org/pipermail/openstack-discuss/2020-August/016568.html | 14:17 |
| corvus | mnaser: i tried "git clone https://gitea08.opendev.org:3000/openstack/tripleo-heat-templates.git" for all 8 instances and the clone at least started | 14:21 |
| corvus | i didn't let them all run to completion | 14:21 |
| mnaser | corvus: ok, perhaps it is something at their side then :\ | 14:21 |
| mnaser | i would have done that but i dont remember the port numbers heh | 14:22 |
| corvus | i had to check in the system-config repo :) | 14:22 |
| *** sshnaidm is now known as sshnaidm|MYG | 14:27 | |
| *** sshnaidm|MYG is now known as sshnaidm|mtg | 14:27 | |
| clarkb | one trick for identifying the backend node you are talking to is to inspect the ssl cert and look at the altnames | 14:28 |
| fungi | yeah, i only just approved that message through list moderation, haven't had time to look into it yet | 14:29 |
| *** chkumar|rover is now known as raukadah | 14:45 | |
| clarkb | corvus: if you get a chance I put together https://etherpad.opendev.org/p/gerrit-upgrade-luca-questions after doing more testing Friday. Also https://review.opendev.org/#/c/745595/ makes javamelody not throw exceptions and the url is functional on 2.13, 2.14, 2.15, and 2.16 in my testing and the codemirror-editor error pop up doesn't happen on 2.16 | 14:47 |
| clarkb | corvus: for the email I'll send that out this morning if we're happy with it | 14:47 |
| AJaeger | config-core, please review https://review.opendev.org/745193 - new project addition... | 14:50 |
| *** qchris has quit IRC | 14:57 | |
| openstackgerrit | Merged openstack/project-config master: New Project Request: hostconfig-operator https://review.opendev.org/745193 | 15:02 |
| *** qchris has joined #opendev | 15:09 | |
| *** mlavalle has joined #opendev | 15:15 | |
| *** ysandeep is now known as ysandeep|away | 15:42 | |
| *** priteau has quit IRC | 15:57 | |
| openstackgerrit | Riccardo Pittau proposed openstack/project-config master: Do not update upper constraints for intermediate branches https://review.opendev.org/746553 | 16:00 |
| *** sshnaidm|mtg is now known as sshnaidm | 16:08 | |
| *** dtantsur is now known as dtantsur|afk | 16:59 | |
| *** hashar is now known as hasharAway | 17:10 | |
| *** hipr_c has joined #opendev | 17:36 | |
| *** hipr_c has joined #opendev | 17:57 | |
| fungi | ianw: revisiting the f31 mirror issues, we're still seeing failures from that. i notice that the source we're mirroring from is out of date compared to the one you linked: https://pubmirror2.math.uh.edu/fedora-buffet/fedora/linux/updates/31/Everything/x86_64/repodata/?C=M;O=A | 18:05 |
| fungi | no idea if/when that will be corrected. maybe we need to pick a different rsync source? | 18:06 |
| fungi | looks like prior to pubmirror2.math.uh.edu we used pubmirror1.math.uh.edu but both seem to be in roughly the same state | 18:17 |
| fungi | and before that we used mirrors.kernel.org but they don't seem to mirror anything later than f29 | 18:17 |
| fungi | mirrors.mit.edu has both fedora and epel with current content (and aarch64 too) | 18:22 |
| fungi | and is listed as having rsync available | 18:22 |
| fungi | same for mirror.math.princeton.edu | 18:26 |
| fungi | however neither mirrors fedora atomic that i can find | 18:34 |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Switch Fedora mirror to mirrors.mit.edu https://review.opendev.org/746587 | 18:51 |
| fungi | infra-root: ^ that's my attempt at fixing broken f31 jobs. if it merges soon i'm happy to watch the mirror logs and make sure things get back on track | 18:52 |
| *** smcginnis has quit IRC | 19:19 | |
| *** smcginnis has joined #opendev | 19:21 | |
| clarkb | fungi: +2d but didn't approve | 19:29 |
| clarkb | I'll let you approve when ready | 19:29 |
| corvus | lookin | 19:30 |
| corvus | is there a more upstream place we can mirror from? | 19:30 |
| fungi | also tibbs responded in #fedora-admin and acknowledged those servers are struggling to catch up and probably still need a few dys | 19:30 |
| corvus | just wondering what the policy on mirroring from a more canonical location might be; considering the amount of redhat product that goes through our system, mirroring more directly might be in everyone's interest? | 19:32 |
| clarkb | corvus: if you get a chance to look at https://etherpad.opendev.org/p/gerrit-upgrade-luca-questions and https://review.opendev.org/#/c/745595/ for gerrit things today that would be most excellent | 19:32 |
| corvus | clarkb: will do | 19:32 |
| fungi | corvus: i concur. i wonder where the best place is to find out/ask | 19:32 |
| fungi | https://fedoraproject.org/wiki/Infrastructure/Mirroring seems to indicate (unsurprisingly) that to be able to pull from the "master" servers, your mirror has to appear in the official download mirrors list | 19:37 |
| corvus | ok, that might be a bit much for us to bite off | 19:37 |
| fungi | so if we don't want to be operating an official mirror, we might need to have a discussion with someone about an exception | 19:37 |
| *** hasharAway is now known as hashar | 19:58 | |
| openstackgerrit | Merged opendev/system-config master: Switch Fedora mirror to mirrors.mit.edu https://review.opendev.org/746587 | 19:58 |
| corvus | fungi: ^ | 19:59 |
| clarkb | that should all happen pretty automatically since we do the localhost vos releases for the fedora mirror | 20:00 |
| clarkb | though if yo uwant to speed it up you can run before the next cron job I guess | 20:01 |
| fungi | yeah, i was just going to wait, and then check logs. juggling plenty of other stuff at the same time | 20:02 |
| fungi | but i have a change which is an easy reproducer of the problem, so can recheck that once i see that the mirror log indicates we pulled with no errors | 20:03 |
| *** hashar has quit IRC | 20:10 | |
| clarkb | infra-root https://review.opendev.org/#/c/746335/ is another gerrit related change bug fix I discovered when doing local testing (its a bug in our ansible for gerrit) | 20:15 |
| clarkb | probalby not urgent but would be good to land evebtually so that we don't have to debug that if it pops up on a rebuild | 20:15 |
| corvus | clarkb: msg to luca looks good; only thing i might consider adding is being more explicit that we know not to use notedb in 2.15 (though you did link directly to my saying that in footnote 0) | 20:19 |
| clarkb | corvus: how does that little edit look? | 20:21 |
| corvus | (just so we don't get a treatise back about how we shouldn't do that; lgtm) | 20:21 |
| clarkb | cool I'll get that sent out. I'll cc people too so that they are included in the thread | 20:22 |
| corvus | clarkb: docker change +3 | 20:22 |
| clarkb | tyty | 20:22 |
| clarkb | now updating the infra meeting agenda. Anything to add to that? | 20:35 |
| fungi | ptg/forum maybe | 20:39 |
| clarkb | ++ | 20:40 |
| clarkb | thanks for the reminder | 20:40 |
| clarkb | hrm the docker images change failed on an arm64 thing /me looks into that | 20:47 |
| clarkb | base | W: Failed to fetch https://mirror.regionone.linaro-us.opendev.org/ubuntu-ports/dists/focal/InRelease Connection failed [IP: 139.178.85.143 443] | 20:48 |
| clarkb | iirc we expect ipv6 in that cloud | 20:48 |
| clarkb | so its odd that we would try ipv4, maybe ipv6 failed and we are falling back? | 20:48 |
| clarkb | we do have a AAAA record for that mirror | 20:49 |
| clarkb | the server pings but I'm having trouble getting https to respond | 20:50 |
| clarkb | hrm it finally loaded | 20:50 |
| clarkb | afs: Lost contact with file server 23.253.73.143 in cell openstack.org (code -1) (all multi-homed ip addresses down for the server) | 20:53 |
| clarkb | lots of stuff like that in dmesg and I'm guessing that is the cause | 20:54 |
| clarkb | its all three afs file servers that end up in the log that way | 20:57 |
| clarkb | that makes me think the problem is on the linaro cloud side of things | 20:57 |
| clarkb | at the current moment I can ping all three of them from the mirror iwth no lost packets | 20:58 |
| clarkb | fungi: I'll put https://review.opendev.org/#/c/746497/ on the agenda too | 21:00 |
| fungi | great idea | 21:03 |
| corvus | clarkb: btw luca is generally in london timezone if you wanted to do a voice thing | 21:07 |
| clarkb | thats good to know, thanks | 21:08 |
| * clarkb has had a lot of early mornings lately | 21:08 | |
| corvus | ya | 21:08 |
| clarkb | corvus: reading luca's response I think we should probably strongly ocnsider a 2.16 upgrade without notedb, then regroup, then do notedb | 21:10 |
| corvus | +1 to that | 21:11 |
| clarkb | also looks like I should test an upgrade through to 3.0 next in order to check the db cleanups (mostly I just want to see that we aren't accidentally relying on the db when we expect not to) | 21:14 |
| clarkb | but thats good gives me a few things to push on on our side | 21:14 |
| clarkb | I kept some pings running on the linaro mirror and they lost no packets but dmesg shows that afs lost contact again | 21:40 |
| clarkb | it seems that things generally work but then afs doesn't for some reason? | 21:40 |
| clarkb | fungi: ^ any ideas on what may be going on there? I guessit could be port specific? or we're relying on some stateful firewall treatment of udp packets? | 21:41 |
| clarkb | maybe I should switch to a udp ping | 21:41 |
| fungi | yeah, also make sure it's ipv4 | 21:42 |
| fungi | obviously | 21:42 |
| clarkb | yup ws ipv4 | 21:42 |
| clarkb | I pinged the v4 addrs directly, didn't use names | 21:42 |
| clarkb | udp pings may need mtr? Doesn't seem like ping does them | 21:44 |
| fungi | hping can, i think | 21:46 |
| fungi | checking | 21:46 |
| fungi | hping3 package probably | 21:47 |
| fungi | "hping3 is a network tool able to send custom ICMP/UDP/TCP packets and to display target replies like ping does with ICMP replies." | 21:47 |
| clarkb | I'm installing mtr since I'm familar with it | 21:50 |
| fungi | ahh, i'd never really tried with mtr | 21:50 |
| clarkb | mtr's udp pings don't seem to reliably get to the fileservers | 21:54 |
| fungi | iptables rules? | 21:55 |
| clarkb | maybe, I can try setting hte port. | 21:55 |
| fungi | we don't seem to set the source ports, so it just needs to be one of the allowed destination ports | 21:56 |
| fungi | like afs3-fileserver/udp (7000/udp) | 21:56 |
| * clarkb tries port 7000 | 21:57 | |
| clarkb | ya still not helping | 21:58 |
| clarkb | `mtr --udp -P 7000 $fielserverip` fwiw | 21:58 |
| fungi | another problem you're going to run into is that udp datagrams are, by their very nature, typically unacknowledged by the recipient | 21:59 |
| fungi | probably best would be to add a temporary rule allowing access to a random unused udp port and then try pinging that | 21:59 |
| clarkb | ya | 21:59 |
| clarkb | I wonder how afs decides things ar eunreachable? it must be doing its own ping pongs? | 22:00 |
| fungi | though the answer you get back is typically going to be icmp port unreachable in that case, so you'd need to test in both directions as it's only udp outbound not inbound | 22:00 |
| clarkb | maybe a better appraoch would be to tcpdump for those on both sides | 22:00 |
| ianw | fungi: thanks for looking, did the fedora change roll out? | 22:09 |
| ianw | yeah, it looks like it's timed out | 22:10 |
| ianw | i'll restart in a screen with no timeout | 22:10 |
| fungi | ahh, the mirror pulse timed out you mean? | 22:11 |
| fungi | makes sense, there's several days of backlog | 22:11 |
| fungi | thanks, i was about to go checking the log for that | 22:11 |
| ianw | ok it's running in a screen | 22:12 |
| fungi | awesome | 22:13 |
| ianw | i get the feeling this is going to rewrite the whole mirror :/ | 22:14 |
| fungi | ugh | 22:15 |
| fungi | is it blowing away/mirroring the wrong files? | 22:15 |
| ianw | not sure if y'all saw the latest on the pyca wheels; we discovered a problem -- centos and ubuntu use different page sizes | 22:15 |
| fungi | wha? | 22:16 |
| fungi | for arm64/aarch64? | 22:16 |
| ianw | the manylinux aarch64 spec doesn't specify pagesize, which is a problem it's going to have to solve | 22:16 |
| clarkb | thats a thing? | 22:16 |
| fungi | wow, that's a blocker | 22:16 |
| ianw | on arm, yep | 22:16 |
| clarkb | ianw: do you have any idea why the linaro mirror loses connectivity to the afs fileservers? | 22:17 |
| clarkb | I'm sort of poking at that but starting somewhat from scratch | 22:17 |
| ianw | it's going to have to specify a 64k page size to be compatible everywhere it would seem | 22:17 |
| ianw | so it's unfortunately a bit of a blocker for generic wheels | 22:18 |
| ianw | clarkb: yeah ... been through that before and no particular answer. even spent a while with auristor looking at it, let me see if i can find log reference | 22:18 |
| ianw | fungi: so far it's sitting at "receiving incremental file list" and being very slow about it :/ | 22:20 |
| fungi | whee https://www.kernel.org/doc/html/latest/arm64/memory.html | 22:20 |
| clarkb | is the issue alignments? | 22:21 |
| clarkb | so a 4k system would be ok with our "package" built with 64k alignments? | 22:21 |
| ianw | yeah, the issue is alignment | 22:22 |
| ianw | so choosing the biggest should be compatible. but given the wide variety of ways things get built in python ... urgh | 22:23 |
| ianw | i think this problem will be fairly constant | 22:23 |
| diablo_rojo_phon | You probably are aware already, but it seems ethercalc is down? | 22:23 |
| fungi | i was not aware... checking | 22:24 |
| diablo_rojo_phon | Oh well then. In trying to make a new one I get a Service Unavailable message. | 22:24 |
| diablo_rojo_phon | And in trying to access ones already created it has a disconnected from server..reconnecting message | 22:25 |
| ianw | somebody should write a thesis on how the cost of tlb reloads starts to outweigh the benefits of reduced miss rates ... oh https://ts.data61.csiro.au/publications/theses_public/08/Wienand:me.pdf :) | 22:25 |
| auristor | ianw clarkb: I looked quickly through my logs of openstack-infra and I couldn't find any details. | 22:25 |
| fungi | apache proxies to the ethercalc service listening on 8000/tcp on the loopback, and nothing is currently listening on that port. checking the logs to see what killed it | 22:26 |
| auristor | clarkb: any openafs messages in dmesg? what does "rxdebug localhost 7001" output show? | 22:27 |
| clarkb | auristor: dmesg just shows lost contact then file server is back up | 22:28 |
| clarkb | auristor: do I run rxdebug on the client or server? | 22:28 |
| fungi | looks like ethercalc tripped over some js exception and terminated at 17:58:42 utc | 22:30 |
| ianw | clarkb/auristor: this is the last time i remember looking into it fyi http://eavesdrop.openstack.org/irclogs/%23opendev/%23opendev.2020-07-20.log.html#t2020-07-20T03:25:56 | 22:30 |
| clarkb | auristor: running that on the client it says: http://paste.openstack.org/show/796896/ | 22:31 |
| clarkb | which seems happy | 22:31 |
| fungi | #status restarted ethercalc service on ethercalc02.o.o following unexplained crash at 17:58:42 utc | 22:31 |
| openstackstatus | fungi: unknown command | 22:31 |
| fungi | #status log restarted ethercalc service on ethercalc02.o.o following unexplained crash at 17:58:42 utc | 22:31 |
| openstackstatus | fungi: finished logging | 22:31 |
| fungi | diablo_rojo_phon: ^ give it another try? | 22:32 |
| ianw | clarkb: iirc that was paired with a lot of failures to github, a notoriously ipv4 only service too | 22:32 |
| fungi | i'm digging deeper in the logs of its death throes now | 22:32 |
| clarkb | ianw: oh ya I rememebr the github side of things | 22:33 |
| diablo_rojo_phon | Trying now. | 22:33 |
| diablo_rojo_phon | All good. | 22:34 |
| diablo_rojo_phon | Thanks fungi ! | 22:34 |
| ianw | rsync: opendir "/linux/development/rawhide/Everything/x86_64/os/Packages/b/.~tmp~" (in fedora) failed: Permission denied (13) ... i don't know what that's about (fedora mirror process) | 22:35 |
| fungi | infra-root: looks like maybe a csv export attempt may have crashed the ethercalc process somehow: http://paste.openstack.org/show/796897 | 22:35 |
| fungi | worth keeping an eye on, possible that's been fixed in a newer version than we're running | 22:35 |
| clarkb | ianw: that looks like rsync writing to a tmpdir so it can do atomic moves | 22:35 |
| clarkb | ianw: why that would get a permission denied I don't know. We don't have a competing rsync do we? | 22:35 |
| fungi | we could use --inplace instead with rsync, we don't point anyone at the r/w volume while it's being written to | 22:36 |
| fungi | but yeah, i agree, check for other colliding rsync runs for the same tree | 22:36 |
| ianw | yeah, it's not on our side ... | 22:37 |
| *** mlavalle has quit IRC | 22:38 | |
| fungi | oh, or maybe that's rsync failing to read? | 22:39 |
| fungi | maybe we should specify an exclude for that pattern | 22:39 |
| ianw | Well the "Massachvsetts Institvte of Technology" mirror is living up to it's olde-timey name and about as fast as a horse drawn carriage | 22:39 |
| clarkb | oh ya it could be upstream having that tmp dir because it too is syncing | 22:39 |
| clarkb | then we try to read it and fail | 22:39 |
| fungi | could indeed be rsync writing to a tempfile on the remote system | 22:39 |
| fungi | and not making it world-readable | 22:39 |
| fungi | and then we try to mirror it | 22:40 |
| openstackgerrit | Merged opendev/system-config master: Gerrit image cleanups/fixes https://review.opendev.org/745595 | 22:40 |
| fungi | ianw: speed wise, yeah, no clue what bandwidth sipb has for those mirrors. hu.edu was convenient in that it was in the same state as mirror-update and an 8ms rtt away | 22:40 |
| ianw | there's a rax ord mirror too | 22:41 |
| fungi | mit is thousands of miles from rax-dfw | 22:41 |
| fungi | having seen issues in the past with rackspace's mirror, i was hesitant to try | 22:41 |
| fungi | also rax-ord (chicago) is nearly as far geographically as mit (boston) from rax-dfw (dallas) | 22:42 |
| ianw | yeah i figured they'd at least have dedicated links ... but i agree, i can't imagine it gets much attention | 22:43 |
| fungi | not that geographic distance necessarily has much bearing on it these days, i've seen traceroutes trombone across the continent and back again to go just down the street | 22:43 |
| ianw | ... there's always that facebook mirror yum picked the other day ... | 22:43 |
| fungi | does facebook's mirror have rsync open? | 22:44 |
| fungi | (he asked, half expecting he already knew the answer) | 22:44 |
| ianw | http://mirror.facebook.net/centos-debuginfo/FOOTER.html | 22:45 |
| ianw | "Due to lack of available bandwidth, we currently don't offer rsync to download/synchronize content from centos.org for debuginfo pkgs. Also worth knowing that this mirror is running on limited available bandwidth too. | 22:45 |
| ianw | " | 22:45 |
| ianw | ohh poor facebook mirror ... if only it had somewhere it could get bandwidth | 22:45 |
| * fungi sighs. so close | 22:45 | |
| fungi | you had one job, internet. ONE job | 22:45 |
| ianw | actually, maybe egg on my face(book) ... i think that file is synced to the mirror from the upstream and is saying to *use* the fb mirror | 22:47 |
| fungi | but yeah, if the rackspace mirror has rsync open and it's current, then worth a try | 22:47 |
| ianw | the FB mirror does seem to have rsync open | 22:48 |
| ianw | ... this may not be a bad option ... if it's behind whatever cdn they use | 22:48 |
| fungi | http://mirror.facebook.net/fedora/linux/updates/31/Everything/x86_64/repodata/ looks reasonable | 22:49 |
| fungi | ianw: note that rsync is unlikely to be going through a "cdn" | 22:50 |
| fungi | at least not a traditional http(s) cdn at any rate | 22:50 |
| ianw | true | 22:51 |
| ianw | fungi: did you kill anything in the screen : "sync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at io.c(513) [receiver=3.1.3]" ? | 22:52 |
| fungi | nope, i haven't even attached to it | 22:52 |
| ianw | damn it maybe i forgot to set the no timeout flag | 22:53 |
| fungi | that's do it, the timeout wrapper uses kill | 22:53 |
| ianw | i did | 22:53 |
| ianw | maybe i'll restart it against this fb mirror and see if it's faster, you can pretty much tell by just watching it | 22:53 |
| fungi | yeah, i should have just hacked up the script and tested in place | 22:54 |
| *** tosky has quit IRC | 22:57 | |
| *** tkajinam has joined #opendev | 23:07 | |
| ianw | it's going, honestly about the same speed | 23:08 |
| *** DSpider has quit IRC | 23:15 | |
| fungi | maybe check the cacti graphs for bandwith utilization on mirror-update? | 23:37 |
| fungi | could be the slowness is at our end courtesy of rackspace's per-flavor bandwidth caps | 23:37 |
| *** Dmitrii-Sh has quit IRC | 23:37 | |
| fungi | or don't. looks like we didn't think to add it to cacti. working on a patch now | 23:38 |
| *** Dmitrii-Sh has joined #opendev | 23:39 | |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Add mirror-update.opendev.org to Cacti https://review.opendev.org/746618 | 23:44 |
| ianw | i should probably dig out the credentials and delete some of the old servers there too | 23:45 |
| ianw | like the old git ones are confusing | 23:45 |
| fungi | i concur | 23:47 |
| ianw | ok i'm into cacti admin interface | 23:56 |
| ianw | from my notes, last time i cleared out hosts was 2019-08-28 | 23:56 |
| ianw | i.e. i have completely forgotten what to do | 23:57 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!