| ianw | clarkb / corvus : fyi https://github.com/pyca/cryptography/pull/5386#issuecomment-681203556 ... we're in a pretty good position generating the wheels I think | 01:14 |
|---|---|---|
| *** sgw has quit IRC | 01:14 | |
| clarkb | ianw: that means the elfpatch thing was sorted out? | 01:14 |
| ianw | clarkb: not quite but i think the cryptography images are carrying a patch | 01:15 |
| clarkb | ah | 01:16 |
| *** sgw has joined #opendev | 01:33 | |
| *** xiaolin has joined #opendev | 01:59 | |
| *** ysandeep|away is now known as ysandeep | 02:39 | |
| ianw | Error from server (Forbidden): error when creating "test-pod.yaml": pods "test" is forbidden: error looking up service account default/default: serviceaccount "default" not found | 03:37 |
| ianw | hrm, this is in the zuul-jobs kubernets something something test :/ | 03:37 |
| ianw | kevinz: not sure if you've seen https://review.opendev.org/#/c/747063/ | 03:38 |
| ianw | kevinz: in short, if you start a container on a guest and try to access pypi/pythonhosted via ssl it just hangs | 03:38 |
| ianw | kevinz: basically fastly CDN. turning down the MTU makes it work | 03:40 |
| ianw | it's got to be a container, and i guess behind the default NAT networking or whatever docker sets up | 03:40 |
| ianw | https://files.pythonhosted.org/packages/40/0a/9b47124720dba8c80ed9f57aa38986b16c9a0fbccbd3bf1da3120158e6f9/cryptography-3.1-cp35-abi3-manylinux2014_aarch64.whl | 04:47 |
| ianw | we did it! :) | 04:47 |
| *** DSpider has joined #opendev | 04:51 | |
| *** bhagyashris|away is now known as bhagyashris | 05:02 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/project-config master: Normalize projects.yaml https://review.opendev.org/748361 | 06:06 |
| *** ysandeep is now known as ysandeep|afk | 06:47 | |
| *** yoctozepto has quit IRC | 07:34 | |
| *** dtantsur|afk is now known as dtantsur | 07:40 | |
| *** tosky has joined #opendev | 07:47 | |
| *** ysandeep|afk is now known as ysandeep | 07:55 | |
| *** moppy has quit IRC | 08:01 | |
| *** moppy has joined #opendev | 08:01 | |
| openstackgerrit | Merged zuul/zuul-jobs master: bindep: Fixed runtime warnings https://review.opendev.org/747781 | 08:24 |
| openstackgerrit | Mark Goddard proposed openstack/project-config master: kolla-cli: deprecation - retiring master branch https://review.opendev.org/748259 | 08:47 |
| openstackgerrit | Stephen Finucane proposed openstack/project-config master: Add Backport-Candidate label for nova deliverables https://review.opendev.org/748377 | 08:49 |
| openstackgerrit | Stephen Finucane proposed openstack/project-config master: Add Backport-Candidate label for placement deliverables https://review.opendev.org/748378 | 08:49 |
| *** yoctozepto has joined #opendev | 08:55 | |
| *** andrewbonney has joined #opendev | 08:59 | |
| openstackgerrit | Stephen Finucane proposed openstack/project-config master: Drop +2/-2 Backport-Candidate label values for oslo https://review.opendev.org/748384 | 09:13 |
| openstackgerrit | Stephen Finucane proposed openstack/project-config master: Consistently set the stable ref permissions https://review.opendev.org/748385 | 09:13 |
| *** hashar has joined #opendev | 09:45 | |
| openstackgerrit | Michal Nasiadka proposed openstack/diskimage-builder master: Add cloud-init-disable-growpart https://review.opendev.org/748402 | 10:20 |
| openstackgerrit | Michal Nasiadka proposed openstack/diskimage-builder master: Fix grubenv link in latest grub2 CentOS packages (EFI) https://review.opendev.org/748157 | 10:25 |
| *** sshnaidm|afk is now known as sshnaidm | 10:30 | |
| *** jaicaa has quit IRC | 10:41 | |
| *** jaicaa has joined #opendev | 10:44 | |
| *** hashar is now known as hasharLunch | 10:48 | |
| *** ysandeep is now known as ysandeep|afk | 11:13 | |
| *** ysandeep|afk is now known as ysandeep | 11:34 | |
| *** xiaolin has quit IRC | 11:37 | |
| *** hasharLunch is now known as hashar | 12:01 | |
| *** ysandeep is now known as ysandeep|brb | 12:50 | |
| *** ysandeep|brb is now known as ysandeep | 13:02 | |
| openstackgerrit | Thierry Carrez proposed opendev/system-config master: Explain "why opendev" on opendev.org index page https://review.opendev.org/748263 | 14:22 |
| *** ysandeep is now known as ysandeep|away | 14:34 | |
| AJaeger | mnaser: please review https://review.opendev.org/#/c/748210/ as this is vexxhost related | 14:52 |
| clarkb | ianw: !! success | 14:52 |
| *** qchris has quit IRC | 14:57 | |
| AJaeger | config-core, please review https://review.opendev.org/#/c/748273 | 14:57 |
| *** andrewbonney has quit IRC | 14:59 | |
| openstackgerrit | Merged openstack/project-config master: Normalize projects.yaml https://review.opendev.org/748361 | 15:03 |
| *** qchris has joined #opendev | 15:10 | |
| openstackgerrit | Clark Boylan proposed opendev/base-jobs master: Set file modes explicitly https://review.opendev.org/748478 | 16:05 |
| clarkb | next up zuul-jobs though that is quite a bit more involved | 16:06 |
| *** ysandeep|away is now known as ysandeep | 16:14 | |
| openstackgerrit | Merged openstack/project-config master: Remove legacy-tempest-dsvm-neutron-{ipv6only,serviceipv6} https://review.opendev.org/748273 | 16:20 |
| *** dtantsur is now known as dtantsur|afk | 16:23 | |
| *** sshnaidm is now known as sshnaidm|afk | 16:24 | |
| openstackgerrit | Clark Boylan proposed opendev/base-jobs master: Set file modes explicitly https://review.opendev.org/748478 | 16:27 |
| openstackgerrit | Clark Boylan proposed zuul/zuul-jobs master: WIP: Address ansible-lint E208 https://review.opendev.org/748480 | 16:29 |
| *** tosky has quit IRC | 16:45 | |
| openstackgerrit | Sorin Sbarnea (zbr) proposed zuul/zuul-jobs master: Partial address ansible-lint E208 https://review.opendev.org/748480 | 17:21 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed zuul/zuul-jobs master: More E208 mode fixes https://review.opendev.org/748498 | 17:41 |
| *** kevinz has quit IRC | 17:58 | |
| *** tosky has joined #opendev | 18:35 | |
| openstackgerrit | Sorin Sbarnea (zbr) proposed zuul/zuul-jobs master: More E208 mode fixes https://review.opendev.org/748498 | 19:47 |
| clarkb | corvus: if you have a moment https://review.opendev.org/#/c/729966/ zbr asked for rereview on that change. I think the logging at least will help us identify when that happens and help with debugging. If we are happy with it I can include it in the gerritlib releaes needed for jeepyb branch handling | 20:52 |
| clarkb | if not I'll make the release without it | 20:52 |
| zbr | clarkb: in fact event == select.POLLIN is a clear but, because that is a bitmas | 20:59 |
| zbr | event should never be checked with == | 20:59 |
| corvus | zbr: i continue to disagree with you on that as i wrote in the comment on the change. but i'm not going to argue any more. i've left a +1 on the change to indicate i'm not opposed, but i'm not going to +w it since i don't have time to deal with potential fallout | 21:21 |
| corvus | clarkb: ^ | 21:21 |
| clarkb | ya I expect its going to be something with OSX's tcp stack and understanding how to make it happy woul dbe a good thing | 21:22 |
| clarkb | but I don't have an OSX setup to test on so adding logging seems like a reasonable intermediate step | 21:23 |
| *** hashar has quit IRC | 21:23 | |
| corvus | agreed; and if we accept "osx sends a pollpri that's safe to ignore for some reason no one understands" as a true statement, that change lgtm. my lack of +w is due to the small chance something in that stamement is wrong, or merging and deploying this reveals some new issue. just being extra conservative due to limited time. | 21:25 |
| clarkb | wfm thanks for looking | 21:25 |
| * JayF is on an OSX machine and is happy to do a thing if it's helpful | 21:28 | |
| clarkb | JayF: basically gerritlib's ssh connections get POLLPRI events back at them when running on pyton3.8 on osX | 21:35 |
| clarkb | JayF: we don't know if there is an important signal in that or not | 21:35 |
| clarkb | JayF: https://review.opendev.org/#/c/729966/ adds logging you could test and see if the log show up if connecting to review.opendev.org's event stream | 21:35 |
| JayF | well, if you have a thing and need it run with debug logging on OSX I'm happy to. I have nearly-zero low level OSX knowledge | 21:35 |
| JayF | I'm assuming the test case would be like a 10 line python script using that code, for someone who knew gerritlib? | 21:36 |
| clarkb | JayF: ya it would basically be pip install that checkout, then import gerritlib ; g = gerritlib.Gerrit('review.opendev.org', 'yourusername') ; g.startWatching() | 21:38 |
| JayF | k, give me one sec | 21:38 |
| JayF | installing 3.8.1 via pyenv to test | 21:39 |
| JayF | hmm no Gerrit in gerritlib | 21:43 |
| * JayF looks | 21:43 | |
| JayF | aha seems to be gerritlib.gerrit.Gerrit, perhaps | 21:44 |
| JayF | ...or not | 21:44 |
| JayF | import gerritlib.gerrit; gerrit.Gerrit | 21:45 |
| clarkb | aha sorry | 21:46 |
| JayF | that was... underwhelming | 21:46 |
| JayF | gist incoming | 21:46 |
| JayF | https://gist.github.com/jayofdoom/0cf08be224179f75291ebd4c9ca0ebfa | 21:46 |
| JayF | perhaps I'm not popular enough on gerrit? lol | 21:46 |
| clarkb | JayF: its because the startWatching() call starts a daemon thread that fills a queue objet | 21:47 |
| clarkb | you can add a while True: g.getEvent(); to block and pull those off | 21:47 |
| JayF | so I probably need to consume that queue and print things, or put in a busy loop | 21:47 |
| JayF | yep | 21:47 |
| JayF | I'm getting borderline-spammed with `POLLPRI event 3 received, see https://github.com/paramiko/paramiko/issues/1694` | 21:48 |
| clarkb | JayF: cool are you getting events too? | 21:49 |
| clarkb | (I'm curious if the underlying event stream manages to function) | 21:49 |
| JayF | I'm looking now, restarted it with the getEvent wrapped in a print | 21:49 |
| JayF | looks like it tbh | 21:49 |
| JayF | but only one so far | 21:49 |
| clarkb | will depend on people doing gerrit things | 21:49 |
| clarkb | I guess its good the behavior is reproduceable | 21:50 |
| JayF | get to work, $people! :D | 21:50 |
| clarkb | for debugging it I guess we need to write some code to read the side channel data | 21:50 |
| JayF | my hunch is it's going to print those for every event it gets | 21:50 |
| JayF | you got a simple PR for me to +1 to create an event? lol | 21:50 |
| JayF | oh, there goes another | 21:50 |
| JayF | looks like I'm getting that debug log followed by the event pretty reliably | 21:51 |
| JayF | where datapoints=2 | 21:51 |
| JayF | although TBF I wouldn't know if I was missing events | 21:51 |
| JayF | I have certainly had zero cases of *getting* an event without also getting that debug log | 21:51 |
| JayF | oh wow, there it goes, yep, can confirm, lots of data points | 21:52 |
| clarkb | ha | 21:52 |
| clarkb | something probably merged | 21:52 |
| clarkb | creates a lot of events | 21:52 |
| clarkb | on linux that should mean there is out of band data | 21:53 |
| clarkb | on osx I wonder if it can mean something else | 21:53 |
| clarkb | are osx man pages a thing that exist and if so are they on the internets? | 21:53 |
| JayF | pasted those comments into the pr for you | 21:53 |
| JayF | what do you want, a man 2 poll or something? | 21:53 |
| clarkb | ya | 21:54 |
| clarkb | curious what it says about POLLPRI | 21:54 |
| JayF | POLLPRI High priority data may be read without blocking. | 21:54 |
| clarkb | wow so does that mean we'll basically always get it on osx? | 21:54 |
| JayF | going to gist the whole manpage for you | 21:54 |
| clarkb | JayF: I found an iphone manpage thatsays that too | 21:55 |
| JayF | added as a comment here https://gist.github.com/jayofdoom/0cf08be224179f75291ebd4c9ca0ebfa | 21:55 |
| clarkb | the internet says if you get a pollpri then pollin will also be set | 21:58 |
| JayF | I'll keep that venv alive in case you all have something else you want me to run on it later | 21:58 |
| clarkb | random internet person says pollpri does not mean out of band data on osx https://github.com/python-trio/trio/issues/61#issuecomment-546781689 | 22:00 |
| fungi | https://man.openbsd.org/poll.2#POLLPRI | 22:01 |
| fungi | it's not just osx/darwin, seems to maybe be a general bsdism? | 22:01 |
| clarkb | I wonder if osx is identifying the traffic as ssh somehow (ports or whatever) and deciding it should all be priority because it may be an interactive session? | 22:02 |
| fungi | though that manpage does go on to also say that "...on OpenBSD, the POLLPRI and POLLRDBAND flags may be used to detect when out-of-band socket data may be read without blocking." | 22:03 |
| clarkb | corvus: going back to your original comment concerned about getting pollpri and pollin together apparently that is normal | 22:04 |
| clarkb | maybe not necessarily always the case though | 22:05 |
| clarkb | (which would be important to understand if were gonna change that loop I think) | 22:05 |
| clarkb | ya so POLLIN means there is any data to read | 22:07 |
| JayF | POLLPRI means there's high-priority data to read? | 22:07 |
| clarkb | ya so POLLIN should aways be set if POLLPRI is set I think | 22:07 |
| clarkb | so maybe it is fine to do & POLLIN|POLLPRI | 22:08 |
| clarkb | we wouldn't read the priority data first but we'd get to it whatever it may be on osx | 22:08 |
| clarkb | however I think POLLPRI on linux isn't necessarily something that means there is regular stream data to read | 22:09 |
| clarkb | which makes me think we need to differentiate there to properly handle things | 22:09 |
| clarkb | actually what if we checked if bitmask == POLLIN or bitmask == POLLPRI & POLLIN | 22:10 |
| *** mtreinish has quit IRC | 22:10 | |
| clarkb | then we'd be checking that there is data to read and could read (and ignore the out of band on linux and whatever it means on osx) | 22:10 |
| clarkb | the problem with == POLLIN was that we were skipping events on osx because of the extra mask but the problem with an | is that it could be either or (also on that last bitmask == check it should be a | not a &) | 22:11 |
| * clarkb is trying to compose thoughts and will leave a response on the change | 22:13 | |
| *** mtreinish has joined #opendev | 22:16 | |
| clarkb | JayF: if you're still able to help can you do a packet capture of that traffic and see if the URG flag is set | 22:25 |
| clarkb | I wonder if on OSX that is what it means but on linux its like meh you lose | 22:25 |
| clarkb | JayF: that should all be in the tcp framing outside of the encrypted bits so you'll see it just doing a naive capture | 22:26 |
| clarkb | JayF: and we would expect that on the packets from gerrit to you | 22:26 |
| JayF | hmm, I don't have all the tools for that installed | 22:27 |
| JayF | let me see how tough it is | 22:27 |
| JayF | and disconnect from vpn lol | 22:27 |
| clarkb | wireshark will do it and I think is cross platform | 22:28 |
| clarkb | osx likely has a tcpdump too | 22:28 |
| JayF | yeah, just gotta brew install wireshark and disconnnect from VPN so I don't set off klaxons at Verizon SOC :D | 22:28 |
| clarkb | now I'm going to sanity check that my local python3.8 can stream events on linux without POLLPRI | 22:29 |
| clarkb | (it may also be a python3.8 thing?) | 22:29 |
| JayF | I'm surprised it's not a "what headers python compiles against" thing | 22:30 |
| JayF | homebrew appears to be downloading approximately half of hte internet | 22:30 |
| clarkb | I'm sorry :( | 22:31 |
| JayF | eh, I don't mind | 22:31 |
| clarkb | its not really that important right this instant if you hav ebetter things to do | 22:31 |
| JayF | just annoyed when things don't install instantly | 22:32 |
| JayF | I have unreasonable expectations | 22:32 |
| clarkb | confirmed the old code seems to work on linux with python3.8 so it likely an osx thing not a python version thing | 22:33 |
| JayF | this all you needed? | 22:34 |
| JayF | 15:33:43.490580 IP review01.openstack.org.29418 > 192.168.4.52.53053: Flags [P.], seq 3648:3920, ack 1, win 33, options [nop,nop,TS val 1138593608 ecr 651404337], length 272 | 22:34 |
| JayF | 15:33:43.490728 IP 192.168.4.52.53053 > review01.openstack.org.29418: Flags [.], ack 3920, win 2043, options [nop,nop,TS val 651404840 ecr 1138593608], length 0 | 22:34 |
| JayF | I don't know if URG is a tcp flag or not, doubtful because it didn't sound familiar | 22:34 |
| JayF | but my network-foo is extremely dusty | 22:34 |
| corvus | i hope there's a tcp flag called 'urg' | 22:34 |
| clarkb | ok commnt left on the change with my new thoughts | 22:35 |
| JayF | clarkb: was that what you neede? | 22:36 |
| JayF | I have one loaded up in wireshark now too, but don't see much URG-y there | 22:36 |
| clarkb | ya URG is a flag along with SYN ACK FIN etc | 22:36 |
| JayF | > The URG flag is used to inform a receiving station that certain data within a segment is urgent and should be prioritized. | 22:36 |
| JayF | TIL | 22:36 |
| clarkb | is the P flag there for PSH? | 22:36 |
| clarkb | I wonder if that is it | 22:36 |
| JayF | Urgent - not set | 22:36 |
| JayF | Push | 22:37 |
| clarkb | Flags [P.] <- it may be that osx bubbles that up as POLLPRI? | 22:37 |
| JayF | that makes a lot of sense | 22:37 |
| JayF | push is like, tcp_nodelay | 22:37 |
| JayF | and reading that as POLLPRI would not seem crazy to me | 22:37 |
| clarkb | ya | 22:37 |
| JayF | The sending application informs TCP that data should be sent immediately. | 22:37 |
| JayF | The PSH flag in the TCP header informs the receiving host that the data should be pushed up to the receiving application immediately. | 22:37 |
| JayF | from https://packetlife.net/blog/2011/mar/2/tcp-flags-psh-and-urg/#:~:text=To%20summarize%2C%20TCP's%20push%20capability,to%20the%20receiving%20application%20immediately. | 22:38 |
| clarkb | JayF: do you think youcan add that info to the change (your capture bits and the theory that osx exposes psh that way) | 22:38 |
| clarkb | not yet sure what that means from a client perspective but I bet that is it | 22:39 |
| clarkb | and I'm being pushed out the door to do a family walk. Back in a bit. Thanks again for helping debug that on osx | 22:39 |
| JayF | I'll most likely be a pumpkin when you return :) glad to help | 22:40 |
| JayF | I know these issues will bounce around with people guessing about behavior | 22:40 |
| JayF | when I can provide real data, I'm happy to | 22:40 |
| clarkb | I think the next step is to do a similar packet capture on linux (whichI can do) and see if the PSH flag is set there too | 22:40 |
| clarkb | and if so we'll need to handle the poll events differently for different platforms potentially | 22:41 |
| JayF | oh that'd be nuts | 22:41 |
| JayF | if it's an ssh client difference | 22:41 |
| JayF | well, no, it'd have to be a server difference, right? if it's a tcp flag? | 22:41 |
| clarkb | ya its coming from the server but it may be acting differently based on the cleint (I really expect its sending PSH in all cases and linux just ignores it for being silly and osx does not) | 22:42 |
| clarkb | if it sends PSH in all cases I think linux should keep the existing behavior to fail if there is out of band data | 22:42 |
| clarkb | and osx can mask POLLPRI in too | 22:42 |
| JayF | what is "out of band" data? | 22:42 |
| JayF | no, nevermind | 22:42 |
| JayF | go walk | 22:42 |
| clarkb | JayF: its what you get when URG arrives | 22:42 |
| clarkb | URG implies PSH but PSH does not imply URG | 22:43 |
| clarkb | I think that is why linux is acting different here | 22:43 |
| clarkb | and ya popping out now. Back in a bit | 22:43 |
| *** DSpider has quit IRC | 23:15 | |
| clarkb | ok just tested and the PSH happen on linux too | 23:18 |
| clarkb | but they start with the local side | 23:18 |
| clarkb | I wonder if that is a paramiko thing at the end of the day but then linux and osx expose it differently | 23:18 |
| *** tosky has quit IRC | 23:59 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!