| ianw | ok, back to where i was last wednesday, i think borg backups are fixed on ethercalc now, so we can monitor that for the next bit | 00:04 |
|---|---|---|
| *** tosky has quit IRC | 00:28 | |
| *** prometheanfire has quit IRC | 00:44 | |
| *** prometheanfire has joined #opendev | 00:45 | |
| *** DSpider has quit IRC | 01:15 | |
| *** ysandeep|away is now known as ysandeep|holiday | 05:37 | |
| *** manpreet has joined #opendev | 06:22 | |
| *** fressi has joined #opendev | 06:22 | |
| *** marios has joined #opendev | 06:37 | |
| *** sboyron has joined #opendev | 06:55 | |
| *** cap has joined #opendev | 06:57 | |
| *** sshnaidm|off is now known as sshnaidm | 07:00 | |
| *** sshnaidm is now known as sshnaidm|rover | 07:01 | |
| *** ralonsoh has joined #opendev | 07:03 | |
| *** slaweq has joined #opendev | 07:09 | |
| *** slaweq has quit IRC | 07:36 | |
| *** eolivare has joined #opendev | 07:44 | |
| *** rpittau|afk is now known as rpittau | 07:57 | |
| *** slaweq has joined #opendev | 08:12 | |
| *** andrewbonney has joined #opendev | 08:16 | |
| *** pto has joined #opendev | 09:06 | |
| *** webmariner has quit IRC | 09:07 | |
| *** slaweq has quit IRC | 09:12 | |
| *** sboyron has quit IRC | 09:13 | |
| *** sboyron has joined #opendev | 09:20 | |
| *** pto has quit IRC | 09:23 | |
| *** DSpider has joined #opendev | 10:24 | |
| *** hashar has joined #opendev | 10:34 | |
| *** tosky has joined #opendev | 11:25 | |
| *** auristor has quit IRC | 11:43 | |
| *** auristor has joined #opendev | 11:50 | |
| *** Green_Bird has joined #opendev | 11:50 | |
| *** slaweq has joined #opendev | 11:52 | |
| *** persia_ is now known as persia | 11:52 | |
| *** lpetrut has joined #opendev | 12:10 | |
| *** diablo_rojo has joined #opendev | 12:39 | |
| *** sshnaidm|rover has quit IRC | 12:50 | |
| fungi | clarkb: looks like our etherpad and videoconference links are still at their defaults, i'll update them to meetpad with our planning pad now | 12:52 |
| clarkb | fungi: thanks | 12:56 |
| *** sshnaidm has joined #opendev | 12:59 | |
| *** sshnaidm is now known as sshnaidm|rover | 13:00 | |
| *** lpetrut has quit IRC | 13:09 | |
| *** lpetrut has joined #opendev | 13:10 | |
| *** fressi has quit IRC | 13:11 | |
| *** fressi has joined #opendev | 13:11 | |
| *** d34dh0r53 has joined #opendev | 13:51 | |
| *** tobiash has quit IRC | 14:03 | |
| *** tobiash_ has joined #opendev | 14:03 | |
| *** ralonsoh_ has joined #opendev | 14:06 | |
| *** ralonsoh has quit IRC | 14:07 | |
| *** tobiash_ is now known as tobiash | 14:55 | |
| *** mlavalle has joined #opendev | 14:55 | |
| *** hashar is now known as hasharOffsite | 15:04 | |
| *** ralonsoh_ is now known as ralonsoh | 15:08 | |
| *** fressi has quit IRC | 15:15 | |
| *** lpetrut has quit IRC | 15:20 | |
| *** ralonsoh has quit IRC | 16:06 | |
| *** mlavalle has quit IRC | 16:30 | |
| *** Green_Bird has quit IRC | 16:30 | |
| *** whoami-rajat__ has joined #opendev | 16:39 | |
| clarkb | I've cleaned up one more stale db backup for etherpad | 16:50 |
| clarkb | there is one remaining which I'll do tomorrow after tonight's backups run | 16:51 |
| *** mlavalle has joined #opendev | 17:00 | |
| *** rpittau is now known as rpittau|afk | 17:09 | |
| *** sboyron has quit IRC | 17:13 | |
| *** sboyron has joined #opendev | 17:15 | |
| *** tosky has quit IRC | 17:17 | |
| *** snbuback has quit IRC | 17:18 | |
| *** sboyron has quit IRC | 17:24 | |
| *** eolivare has quit IRC | 17:26 | |
| *** marios is now known as marios|out | 17:34 | |
| *** marios|out has quit IRC | 17:55 | |
| *** webmariner has joined #opendev | 18:08 | |
| *** hamalq has joined #opendev | 18:23 | |
| cloudnull | o/ - looks like Fedora 33 has removed the ssh-rsa w/ SHA1 which makes it so folks, like myself, aren't able to "git-review" without updating ssh config, I upgraded last Friday. | 18:39 |
| cloudnull | there's a BZ for it - https://bugzilla.redhat.com/show_bug.cgi?id=1884920 - however, I suspect that's just the way it is. Any chance this is known and we're implementing git servers with SHA2 sometime soon? | 18:39 |
| openstack | bugzilla.redhat.com bug 1884920 in openssh "Cannot ssh into CentOS 6 using ssh key authentication" [Low,New] - Assigned to jjelen | 18:39 |
| cloudnull | figured I'd drop a line here, given I suspect folks will be running into this soon, given F33 is imminent. | 18:39 |
| clarkb | cloudnull: so there are a few things about that. First is that openssh hasn't dropped it directly just deprecated it. Fedora has decided to take a stricter approach. Second is you can enable sha1 on specific hosts which is what I think we're currently suggesting to f33 users | 18:40 |
| clarkb | the last thing is that yes we've been working through a gerrit upgrade whcih will address the problem on the server side | 18:40 |
| clarkb | but that got interrupted last week and we need to pick things back up again after the ptg and see if our previous upgrade planning still makes sense | 18:41 |
| cloudnull | yup, I commented in the BZ w/ the specific config I'm using which re-enables SHA1 for review.opendev.org - just took me a bit to figure out why git-review was failing. | 18:42 |
| clarkb | we haev successfully tested an upgrade of gerrit to 3.2 using a snapshot of production from october 1, but we want to rerun through that and we also have to take a lengthy outage to do the notedb conversion | 18:42 |
| cloudnull | ++ | 18:42 |
| clarkb | largely a matter of being confident in he upgrade process and then finding a reasonable time to take a long outage | 18:43 |
| cloudnull | ++ | 18:43 |
| cloudnull | I suspect there will be a good influx of similar queries tomorrow - when F33 is official | 18:43 |
| cloudnull | https://www.phoronix.com/scan.php?page=news_item&px=Fedora-33-Next-Week#:~:text=At%20Thursday's%20meeting%20it%20was,33%20on%20Tuesday%2C%2027%20October. | 18:44 |
| clarkb | and in 6 months everyone will be wondering why they have no disk but df shows plenty of free space :P | 18:45 |
| clarkb | then everyone will learn about btrfs balancing | 18:45 |
| clarkb | anyway an upgrade will hopefully happen soon for some value of soon. Just need to get through the PTG first and test another upgrade | 18:45 |
| cloudnull | as a former OpenSUSE user that one wont be a head scratcher for me :P | 18:46 |
| cloudnull | thanks clarkb, just figured I'd drop a line about it. | 18:46 |
| clarkb | one thought i had this morning was to announce the november 20-22 outage window under the assumption we'll rerun testing and be happy with it then simply push it back if we feel we aren't ready | 18:47 |
| clarkb | infra-root ^ we can discuss this more in our 23:00-01:00 block but I'd be curious to hear what others think of that | 18:49 |
| *** andrewbonney has quit IRC | 18:51 | |
| *** zbr has quit IRC | 18:56 | |
| *** zbr has joined #opendev | 18:56 | |
| fungi | sounds fine to me, though i would like to entertain mnaser's suggestion in case it allows us to fit the entire upgrade into a single day | 18:58 |
| clarkb | ya trying that next week really quickly might be good next step | 18:58 |
| clarkb | I doubt we'll make much headway on the testing side of things this week with the ptg | 18:58 |
| clarkb | my concern with that is we'd have to add a bunch of steps to copy state first (we're already going to backup things so thats probably not going to add significant overhead, more just another thing to get wrong) | 19:00 |
| clarkb | but we can test it and see how painful it is | 19:00 |
| *** zbr has quit IRC | 19:02 | |
| fungi | i' | 19:03 |
| fungi | grr, return key too close to apostrophe on this keyboard, i want a new keyboard layout which puts the return key somewhere in wisconsin | 19:04 |
| fungi | i'm happy to help/drive gerrit upgrade test repeat next week | 19:04 |
| clarkb | that would be great. I imagine I can help/drive too but I think the more eyeballs we have on the process the more likely we are to catch problems early | 19:05 |
| fungi | agreed, getting more folks involved would be awesome | 19:05 |
| corvus | fungi, clarkb: what was mnaser's suggestion? | 19:16 |
| *** zbr has joined #opendev | 19:16 | |
| clarkb | corvus: he suggseted taht we could boot a surrogate gerrit in vexxhost on one of their new fast flavors with raid 0 ssds and use that to do the notedb migration | 19:17 |
| clarkb | (perhaps even the rest of the upgrade too) then copy that back into place on the production node | 19:17 |
| corvus | ooh | 19:17 |
| clarkb | we would need to test that that is actually after, but if it is may represent a good way to reduce outage window time | 19:18 |
| clarkb | s/after/faster/ | 19:18 |
| clarkb | starting to sketch out what an upgrade announcement might look like at https://etherpad.opendev.org/p/lAv5xqj0oNUjZKeV4vxg | 19:23 |
| fungi | yeah, i think for the vexxhost experiment, we should basically copy our initial state there, then repeat all our upgrade steps and time each one | 19:33 |
| fungi | also time the data transfer into and out of vexxhost obviously since that will add possibly a little deficit to the total outage | 19:33 |
| corvus | rsync should be effective to reduce that on the git repos | 19:38 |
| corvus | probably won't make a big deal with a db dump | 19:38 |
| fungi | agreed, we don't need to add any initial rsync for git repos into the final total, just a nearly-empty last rsync after stopping the service | 19:39 |
| fungi | however given the aggressive git gc performed (more than once) during upgrades, i doubt the rsync back out will save much time over a full copy | 19:40 |
| fungi | but i'd probably use rsync again in that direction too just for consistency | 19:40 |
| fungi | we also might want to disable periodic gc/repacking once we start the first rsync so that we don't create more work for ourselves | 19:41 |
| *** tosky has joined #opendev | 20:43 | |
| *** jentoio has quit IRC | 20:45 | |
| *** erbarr has quit IRC | 20:45 | |
| *** melwitt has quit IRC | 20:45 | |
| *** cmurphy has quit IRC | 20:45 | |
| *** erbarr has joined #opendev | 20:45 | |
| *** jentoio has joined #opendev | 20:45 | |
| *** cmurphy has joined #opendev | 20:45 | |
| TheJulia | is gerrit happy? getting 502 proxy errors? | 21:11 |
| ianw | TheJulia: were you looking at 756434 ? | 21:18 |
| TheJulia | no | 21:22 |
| TheJulia | 757141 | 21:22 |
| ianw | http://cacti.openstack.org/cacti/graph.php?action=zoom&local_graph_id=25&rra_id=0&view_type=tree&graph_start=1603739090&graph_end=1603739090&graph_height=120&graph_width=500&title_font_size=12 | 21:23 |
| ianw | CPU usage has again spiked. i'm seeing occasional 502 responses | 21:23 |
| ianw | (in the logs) | 21:24 |
| ianw | it's another google cloud hosted ip walking all changes in the same way we've seen before | 21:26 |
| ianw | i've blocked it in iptables | 21:30 |
| TheJulia | ugh | 21:31 |
| clarkb | fwiw I think it was ttx that suggested it may be bitergia, we may want to reach out to them and double check? | 21:44 |
| clarkb | another idea I threw out there was that we could see if we can force auth for access then we would get contact info? | 21:45 |
| clarkb | infra-root thoughts on an infra meeting tomorrow? are we wanting to do that or are we swamped with PTG? | 21:47 |
| fungi | i'm not so sure we need the weekly meeting when we also have ptg (though i technically don't have any conflicts for that hour) | 21:49 |
| ianw | I could file something in the google abuse form? i can say that rather than blocking their queries we'd rather work with the owners and give some contact details | 21:53 |
| clarkb | ianw: ya if it isn't bitergia that may be the next step. It seems bitergia.com is hosted on google so wouldn't be surprised if it is them | 21:54 |
| ianw | do we have a contact, other than like a form on the webpage? | 21:55 |
| clarkb | ianw: I think ttx does? | 21:55 |
| ianw | there is a person on twitter who says they work for them who is followed by a lot of openstacky-techy people i follow | 22:00 |
| ianw | oh, haha, that's the ceo | 22:01 |
| clarkb | I think I may cancel the infra meeting. The way our ptg schedule is set up the meeting is in an awkward spot where if you go to the meeting being awake for the next time block will be painful | 22:09 |
| tosky | I know can anyone please band that annoying troll on #openstack-meeting-alt? | 22:10 |
| tosky | and possibly ban the username, which has been reused despite being connected through a web IRC client | 22:10 |
| *** slaweq has quit IRC | 22:10 | |
| clarkb | let me see, I'm not in that channel so don't have scrollback | 22:10 |
| fungi | i can | 22:13 |
| ianw | i'm in there and just kicked them | 22:13 |
| fungi | thanks ianw | 22:13 |
| clarkb | thanks! | 22:13 |
| tosky | thanks clarkb | 22:14 |
| ianw | not sure it isn't some sort of GPT-3 bot | 22:14 |
| tosky | that maybe the case | 22:14 |
| *** hasharOffsite has quit IRC | 22:14 | |
| ianw | did it pass or fail the Turing test? :) | 22:15 |
| tosky | as far I can see, it's the second time I've seen that user trying going with that trolling behavior | 22:15 |
| fungi | reminder to our irc patrolling volunteers, our suggestions for dealing with similar cases are here: https://docs.opendev.org/opendev/system-config/latest/irc.html#banning-disruptive-users | 22:15 |
| tosky | connected several times with the same mask/IP | 22:15 |
| corvus | ianw is turing test proctor | 22:15 |
| *** slaweq has joined #opendev | 22:19 | |
| *** slaweq has quit IRC | 22:35 | |
| *** slaweq has joined #opendev | 22:37 | |
| *** qchris has quit IRC | 22:42 | |
| *** slaweq has quit IRC | 22:47 | |
| *** tkajinam has quit IRC | 22:55 | |
| *** tkajinam has joined #opendev | 22:55 | |
| *** qchris has joined #opendev | 22:56 | |
| fungi | i'll be a couple minutes late, the kettle's not up to a boil quite yet | 23:01 |
| *** mlavalle has quit IRC | 23:03 | |
| ianw | fungi: https://www.kmart.com.au/product/instant-hot-water-dispenser/3139334 | 23:04 |
| ianw | life changing | 23:04 |
| fungi | yeah. i've been tempted but i don't drink tea all that often | 23:05 |
| *** tosky has quit IRC | 23:28 | |
| *** fressi has joined #opendev | 23:39 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!