| fungi | #status ok The Gerrit service at review.opendev.org is back up and running; for outage details see analysis here: http://lists.opendev.org/pipermail/service-announce/2020-October/000011.html | 00:34 |
|---|---|---|
| openstackstatus | fungi: sending ok | 00:34 |
| *** ChanServ changes topic to "OpenDev is a space for collaborative Open Source software development | https://opendev.org/ | channel logs http://eavesdrop.openstack.org/irclogs/%23opendev/" | 00:35 | |
| -openstackstatus- NOTICE: The Gerrit service at review.opendev.org is back up and running; for outage details see analysis here: http://lists.opendev.org/pipermail/service-announce/2020-October/000011.html | 00:35 | |
| openstackstatus | fungi: finished sending ok | 00:40 |
| *** whoami-rajat__ has joined #opendev | 00:44 | |
| *** chandankumar is now known as raukadah | 00:47 | |
| SotK | big thanks to you all for resolving what sounds like a very painful day, your work is very appreciated! | 00:48 |
| fungi | SotK: that day was a week long :/ | 00:50 |
| SotK | even worse :( | 00:50 |
| portdirect | fungi: thanks so much | 00:50 |
| portdirect | people like you, and all the infra team, are the backbone of openstack | 00:51 |
| portdirect | :) | 00:51 |
| fungi | but you're welcome! i'm just glad to have the opportunity to collaborate with all of you on these systems | 00:51 |
| *** owalsh has quit IRC | 01:08 | |
| *** owalsh has joined #opendev | 01:09 | |
| ianw | kevinz: looks like some issues with the linaro API https://us.linaro.cloud:5000: HTTPSConn | 01:54 |
| ianw | ectionPool(host='us.linaro.cloud', port=5000) | 01:54 |
| jbryce | portdirect: totally agree. the level of effort has been really impressive | 02:15 |
| *** DSpider has quit IRC | 02:34 | |
| melwitt | +1, thank you for all the work you do. we <3 infra | 03:23 |
| *** hamalq has quit IRC | 03:46 | |
| *** lajoskatona has joined #opendev | 04:02 | |
| *** lajoskatona has left #opendev | 04:02 | |
| *** fressi has joined #opendev | 04:17 | |
| *** fressi has quit IRC | 04:41 | |
| *** marios has joined #opendev | 05:04 | |
| *** pto has joined #opendev | 05:28 | |
| *** sboyron has joined #opendev | 06:36 | |
| *** rpittau|afk is now known as rpittau | 06:42 | |
| *** ralonsoh has joined #opendev | 06:45 | |
| *** eolivare has joined #opendev | 06:49 | |
| *** slaweq has joined #opendev | 06:53 | |
| *** andrewbonney has joined #opendev | 07:06 | |
| noonedeadpunk | seems gerritbot has been affected and down at the moment | 07:32 |
| noonedeadpunk | (probably token needs to be updated?) | 07:32 |
| ianw | ahh, probably | 07:33 |
| *** slaweq has quit IRC | 07:34 | |
| *** slaweq has joined #opendev | 07:35 | |
| *** openstackgerrit has quit IRC | 07:38 | |
| ianw | ok, should be back | 07:39 |
| *** slaweq has quit IRC | 07:40 | |
| *** tosky has joined #opendev | 07:42 | |
| *** sshnaidm is now known as sshnaidm|afk | 07:43 | |
| *** slaweq has joined #opendev | 07:44 | |
| *** ttx has quit IRC | 07:48 | |
| noonedeadpunk | yep, it is, thanks! | 07:48 |
| *** ttx has joined #opendev | 07:51 | |
| *** priteau has joined #opendev | 07:54 | |
| *** pto has quit IRC | 07:55 | |
| slaweq | ianw: hi | 07:59 |
| slaweq | ianw: since today my ssh key added in gerrit don't work anymore, is that due to this yesterday outage and should I create new key in gerrit? | 07:59 |
| slaweq | I read Your email but I understood from it that ssh keys added before 1.10 should be ok | 07:59 |
| ianw | slaweq: that's right, you didn't update it since then? | 08:00 |
| ianw | are you sure you're not using HTTP? all HTTP API keys have been cleared | 08:00 |
| *** nautik has joined #opendev | 08:05 | |
| slaweq | ianw: yes, here is how I have it configured http://paste.openstack.org/show/799242/ | 08:08 |
| *** slaweq has quit IRC | 08:08 | |
| *** roman_g has joined #opendev | 08:10 | |
| *** slaweq has joined #opendev | 08:11 | |
| slaweq | ianw: and I didn't change it since looong time for sure :) | 08:12 |
| *** gnuoy has quit IRC | 08:19 | |
| *** gnuoy has joined #opendev | 08:20 | |
| *** slaweq has quit IRC | 08:35 | |
| *** slaweq has joined #opendev | 08:36 | |
| *** slaweq has quit IRC | 08:37 | |
| *** slaweq has joined #opendev | 08:37 | |
| *** sshnaidm|afk is now known as sshnaidm | 08:39 | |
| *** fressi has joined #opendev | 08:43 | |
| frickler | that was the known F33 ssh config issue, not related to our incident | 08:43 |
| slaweq | frickler: ianw: thank You a lot for Your help, it was issue with Fedora 33 ssh config as frickler pointed me. It's now fixed | 08:43 |
| *** priteau has quit IRC | 10:39 | |
| *** DSpider has joined #opendev | 11:00 | |
| cgoncalves | kevinz, hey! is Linaro cloud still under maintenance? Zuul reported NODE_FAILURE twice in https://review.opendev.org/#/c/747629/ | 11:04 |
| *** priteau has joined #opendev | 11:15 | |
| *** priteau has quit IRC | 11:26 | |
| *** mkalcok has quit IRC | 11:38 | |
| *** mkalcok has joined #opendev | 11:39 | |
| *** priteau has joined #opendev | 12:00 | |
| *** mattd01 has joined #opendev | 12:04 | |
| frickler | infra-root: sean-k-mooney noticed a problem with a job on limestone, because the node had no IPv4 address at all and thus devstack failed to find the interface with the default route https://zuul.opendev.org/t/openstack/build/d01bced0e20f4bcca40237bb21093661/log/job-output.txt#1947 | 12:49 |
| frickler | is that intended (no v4 at all)? guess we would have to tune devstack for that scenario | 12:50 |
| fungi | frickler: no, the expectation is that it has some rfc 1918 address which then uses pat/overflow nat to reach v4 addresses on the internet | 12:50 |
| fungi | if it didn't get an ipv4 address, maybe there was a problem with dhcpd | 12:51 |
| frickler | fungi: o.k., let's wait and see whether that was a one-off or we see this more often | 12:53 |
| sean-k-mooney | ill let ye know if i see this again | 12:53 |
| sean-k-mooney | for now im goign to just fix the ordering in the world dump script | 12:54 |
| *** slaweq has quit IRC | 13:08 | |
| *** slaweq has joined #opendev | 13:12 | |
| *** openstackgerrit has joined #opendev | 13:12 | |
| openstackgerrit | zbr proposed zuul/zuul-jobs master: WIP: Add ensure-vagrant role https://review.opendev.org/759046 | 13:12 |
| openstackgerrit | zbr proposed zuul/zuul-jobs master: Improve errors from updat-test-platforms https://review.opendev.org/759050 | 13:17 |
| *** ysandeep is now known as ysandeep|ruck | 13:18 | |
| *** slittle1 has joined #opendev | 13:46 | |
| corvus | i'm replying to mnaser's email | 14:03 |
| *** ysandeep|ruck is now known as ysandeep|ruck|af | 14:04 | |
| fungi | thanks, i did as well earlier | 14:16 |
| fungi | i'm trying to reply on the ironic and tripleo threads on openstack-discuss currently | 14:16 |
| fungi | as well as questions which came up in the cinder weekly meeting underway just now | 14:16 |
| corvus | oh, i'll check my reply against yours | 14:17 |
| fungi | i can see this is going to be most of my day (probably most of my month, and maybe the rest of my year) | 14:17 |
| corvus | welp i didn't say anything that fungi didn't say, so i will discard my reply :) | 14:19 |
| fungi | sorry about that! | 14:21 |
| corvus | np i was only about half done :) | 14:21 |
| *** ysandeep|ruck|af is now known as ysandeep|ruck | 14:21 | |
| *** sshnaidm is now known as sshnaidm|rover | 14:22 | |
| *** leuben has joined #opendev | 14:47 | |
| leuben | Hello I have intermittent network issues with FIP access using neutron (DVR). Can someone guide me to the proper channel to discuss it further ? | 14:50 |
| AJaeger | leuben: #openstack in general or #openstack-neutron. | 14:52 |
| *** ysandeep|ruck is now known as ysandeep|away | 14:57 | |
| *** mkalcok has quit IRC | 14:59 | |
| mnaser | and i was waiting all morning for corvus reply :) | 15:10 |
| *** mlavalle has joined #opendev | 15:11 | |
| *** raukadah is now known as chandankumar | 15:12 | |
| corvus | mnaser: thanks for being 'that guy'. i'm glad you care. :) | 15:19 |
| mnaser | corvus: thanks for you and fungi for hearing me out that this honestly comes from a place of care, and not frustration, cause i know what being on the other end of this this like, it's not fun | 15:20 |
| fungi | mnaser: you're not "that guy" at all, you're basically echoing what most of us are thinking, we just need some time to recover before we get into deep debate on our options i think | 15:21 |
| mnaser | fungi: agreed, it's not like monday was any easier, so hopefully things do ease up in the next few days -- and the ptg might be in a very good time | 15:22 |
| fungi | all the points you raise are great | 15:22 |
| corvus | ++ | 15:22 |
| *** dmsimard1 has joined #opendev | 15:30 | |
| *** dmsimard has quit IRC | 15:32 | |
| *** dmsimard1 is now known as dmsimard | 15:32 | |
| *** noonedeadpunk has quit IRC | 15:51 | |
| *** marios has quit IRC | 15:56 | |
| *** lyarwood has quit IRC | 16:01 | |
| *** fressi has quit IRC | 16:03 | |
| *** rpittau is now known as rpittau|afk | 16:04 | |
| *** noonedeadpunk_ has joined #opendev | 16:13 | |
| *** tosky has quit IRC | 16:16 | |
| *** portdirect has quit IRC | 16:19 | |
| *** portdirect has joined #opendev | 16:20 | |
| *** hashar has joined #opendev | 16:20 | |
| *** portdirect has quit IRC | 16:21 | |
| *** portdirect has joined #opendev | 16:21 | |
| *** hamalq has joined #opendev | 16:27 | |
| *** fressi has joined #opendev | 16:28 | |
| *** hamalq has quit IRC | 16:29 | |
| *** hamalq has joined #opendev | 16:30 | |
| *** eolivare has quit IRC | 16:30 | |
| *** noonedeadpunk_ has quit IRC | 16:33 | |
| *** noonedeadpunk has joined #opendev | 16:39 | |
| roman_g | OpenDev Infra team, you rock! Did a great job on resolving Gerrit/security issue. Thank you! | 16:40 |
| fungi | roman_g: it's arguably the most important service we run, so we took it extremely seriously | 16:45 |
| fungi | even though there's been no sign of successful tampering, we're trying to be as cautious and transparent as possible | 16:45 |
| roman_g | Thank you for that. | 16:46 |
| fungi | having the openid of one of our administrative users come under the control of an untrusted party is essentially one of our worst nightmares | 16:46 |
| fungi | so we're also busy planning for how to make that less risky going forward | 16:47 |
| roman_g | Has the case been reported to the police or such? | 16:47 |
| fungi | i expect it to be heavily discussed during our ptg sessions next week | 16:47 |
| fungi | roman_g: well, for starters, jurisdiction is hard to establish in cases where this occurs across oceans, but also we can't be certain that the identity we've found isn't just a smokescreen or false flag trying to smear someone else and implicate them | 16:48 |
| fungi | ultimately the unauthorized access occurred in the openid provider, which we also don't control, so any legal action may need to be initiated on their behalf or on behalf of the user of that platform | 16:49 |
| roman_g | And that's second question. If you have cooperated with provider and collected evidence. | 16:50 |
| fungi | yep, we've been in touch with the launchpad admins since the first few hours into the incident, and they were instrumental in helping us identify other avenues of intrusion | 16:51 |
| roman_g | Great. | 16:51 |
| roman_g | Gerrit version which is run by you supports GPG signing, I would suggest to make code sign a requirement for infra repositories. This is easy to implement and would add additional security layer. | 16:52 |
| roman_g | Other than that I have nothing to add. | 16:52 |
| *** fressi has quit IRC | 16:52 | |
| clarkb | ya were were talking about that yesterday. I don't think you can enforce such a requirement on a per repo basis (I think newer gerrit allows setting it for all repos?) | 16:52 |
| roman_g | You did a great job and deserve at least congrats and warm hugs. | 16:52 |
| fungi | yep, we've started discussing that as well. thankfully it's also configurable on a per-project basis | 16:52 |
| clarkb | its definitely something to look at | 16:52 |
| clarkb | fungi: oh cool I didn't realize that | 16:52 |
| fungi | clarkb: at least last i looked it was a setting similar to cla or cco enforcement | 16:53 |
| fungi | but we do need to double-check that | 16:53 |
| roman_g | Even if it's not possible to enforce per-repo via API/GUI, either Zuul merger and/or Gerrit Prolog rules could allow to make it enforced per-repo. | 16:55 |
| fungi | yep, absolutely worth investigating. also if it's not yet enforceable per-repo, it might be a feature we can work with the gerrit upstream maintainers to add | 16:58 |
| openstackgerrit | wes hayutin proposed zuul/zuul-jobs master: wip, allow rdo repos to be turned off for openvswitch install https://review.opendev.org/759107 | 17:01 |
| *** andrewbonney has quit IRC | 17:10 | |
| fungi | tristanC: did we miss adding someone from sf to https://pypi.org/project/pynotedb/ or were we waiting to find out who to add? | 17:11 |
| fungi | with our upcoming upgrade to newer gerrit, we're eager to try out your in progress lib for that | 17:11 |
| fungi | i got a taste yesterday of seeing how painful it was to find and roll back changes to authorized_keys in the user refs in All-Users.git | 17:12 |
| fungi | imagining all of gerrit's db content being similarly harder to query/update | 17:13 |
| sshnaidm|rover | I see many retry_limits in jobs, is something going on? For example: https://zuul.opendev.org/t/openstack/status/change/758236,2 | 17:20 |
| *** sshnaidm|rover is now known as sshnaidm|afk | 17:24 | |
| fungi | there were some reports earlier of limestone nodes with no ipv4 addresses at all, looking to see if this is more of that, maybe we need to turn that provider down for a bit | 17:28 |
| *** ralonsoh has quit IRC | 17:28 | |
| clarkb | https://5ecd7f1ed77bcef94b3b-fbb316944f0ca23c676e132d61555672.ssl.cf2.rackcdn.com/754223/6/check/neutron-ovn-tripleo-ci-centos-8-containers-multinode/ed9a30e/job-output.txt that job failed to ssh near th eend of the job | 17:30 |
| clarkb | another one that didn't log logfiles shows a similar failure to ssh | 17:30 |
| clarkb | (but it only reports unreachable via ansible exit code 4 I think) | 17:30 |
| clarkb | usually this is a case of arp conflicts? | 17:30 |
| fungi | or network connectivity problems to some provider, or to rackspace's dfw region where the executors are hosted | 17:31 |
| fungi | the more characteristic arp overwrite failures manifest as host key mismatches | 17:32 |
| clarkb | the one in the link shows permission denied so maybe in that case the job changed things under zuul | 17:33 |
| *** priteau has quit IRC | 18:01 | |
| fungi | clarkb: this was the option i was thinking of, so not enforcing validation of signed commits (though gerrit allows users to push them), but rather validating signed pushes on a per-project basis... https://gerrit-documentation.storage.googleapis.com/Documentation/3.2.3/config-project-config.html#receive.requireSignedPush | 18:07 |
| *** mattd01 has quit IRC | 18:07 | |
| fungi | coupled with keys stored in user preferences | 18:08 |
| fungi | again, a possibly useful feature, though not one which would have helped with this case | 18:08 |
| fungi | since the attacker could also have simply added their key and signed the push with that | 18:09 |
| *** fressi has joined #opendev | 18:17 | |
| *** iurygregory has quit IRC | 18:18 | |
| corvus | fungi: right, but that's more easily auditable... essentially, the audit suspected diffs process becomes verifying signatures are valid and the keys belong to their authors. point well taken that you wouldn't be able to trust gerrit for that last part, but asking ppl on a ml to verify their fingerprints in a keyring is probably not too hard. | 18:19 |
| fungi | i also agree that cryptographically signed commits could help in an audit. if a commit in question has a signature which verifies against the committer's key and you have some confirmation that key was not altered, then you can (mostly, modulo attacks on sha-1) rule out tampering | 18:19 |
| *** iurygregory has joined #opendev | 18:20 | |
| fungi | corvus: well, don't confuse signed pushes with signed commits. they're different things. one signs the action, the other the material | 18:20 |
| corvus | oh i am doing that | 18:20 |
| corvus | are we not talking about signed commits? | 18:20 |
| fungi | we started talking about signed commits. then i realized the enforcement configuration i was remembering in gerrit is for signed pushes | 18:21 |
| corvus | oh. i see the value for signed commits; i'm unaware of how signed pushes would help. that's new to me. | 18:21 |
| fungi | still, even voluntarily signing commits and keeping the relevant keys somewhere (like in the user preferences gerrit happens to use for confirming signed pushes) would give us the ability to narrow the scope of possible tampering | 18:22 |
| corvus | the effort for verification of signed commits is also on the order of the number of authors, whereas inspecting content is on the order of number of commits. so as your exposure period increases, signed commits become increasingly advantageous over the status quo. | 18:23 |
| fungi | corvus: some background: https://github.com/git/git/commit/a85b377 | 18:23 |
| *** iurygregory has quit IRC | 18:24 | |
| *** iurygregory has joined #opendev | 18:25 | |
| corvus | fungi: i wonder to what extent a signed push can imply validation of commit contents. ie, if we had signed pushes, since those push certs all reference git shas, can we then infer that if all the git shas which appeared in the repo have valid corresponding push certs indicating they were pushed to gerrit, that all the contents are unmodified since the push and truly originated with the push signer? | 18:27 |
| fungi | possibly. i too need to learn more about push certificates, whether they are retained, and how they could be accessed later in an audit | 18:28 |
| fungi | or whether they are discarded after validating the push action | 18:28 |
| *** lyarwood has joined #opendev | 18:29 | |
| fungi | at least with commit signatures, those are served as part of the commits themselves | 18:29 |
| corvus | at any rate, i suspect that the best case scenario is that if they can, then still only after a layer of indirection, so it's more difficult to verify. seems like maybe a good enhancement after signed commits are the norm. :) | 18:29 |
| fungi | yes, at a minimum they look like complimentary features | 18:29 |
| fungi | especially since they'll probably usually (in the case of our projects anyway) leverage the same keys for both constructs | 18:30 |
| *** mattd01 has joined #opendev | 18:30 | |
| *** fressi has quit IRC | 18:30 | |
| fungi | for workflows where pushing commits from other contributors is the norm, the keys are likely to differ | 18:30 |
| corvus | good point | 18:31 |
| corvus | fungi: thanks for making me aware of it so i don't confuse them as we talk about what options we might pursue; i suspect we may need to watch out for that and inform/remind people that both things exist. | 18:31 |
| fungi | the reminder was partly to myself, because i had clearly misremembered gerrit having a configuration option for requiring commit signatures, when in actuality it's push certificates it can enforce | 18:33 |
| *** fressi has joined #opendev | 18:37 | |
| *** fressi has quit IRC | 18:56 | |
| *** tosky has joined #opendev | 19:12 | |
| *** leuben has quit IRC | 19:39 | |
| *** priteau has joined #opendev | 20:02 | |
| *** whoami-rajat__ has quit IRC | 20:06 | |
| *** mattd01 has left #opendev | 20:20 | |
| *** hashar has quit IRC | 20:55 | |
| *** sboyron has quit IRC | 20:55 | |
| *** slaweq has quit IRC | 21:24 | |
| tristanC | fungi: could you please add https://pypi.org/user/softwarefactory/ (for pynotedb) | 21:47 |
| fungi | tristanC: yep, gimme a sec | 21:48 |
| fungi | tristanC: i have invited that user to become an owner | 21:51 |
| fungi | once you accept, feel free to remove openstackci | 21:51 |
| tristanC | fungi: alright, thanks a lot | 21:51 |
| fungi | #status log handed off ownership of https://pypi.org/project/pynotedb/ to softwarefactory account | 21:52 |
| openstackstatus | fungi: finished logging | 21:52 |
| tristanC | fungi: and well, the existing implementation is working well for us, we use it to automate gerrit upgrade to v3.x, you can find function to list user references and access their account config, for example: https://softwarefactory-project.io/cgit/software-factory/pynotedb/tree/pynotedb/__init__.py#n195 | 21:53 |
| fungi | tristanC: awesome, we're going to need something like that following our coming upgrade. glad to hear you have working code already! | 21:54 |
| *** roman_g has quit IRC | 22:00 | |
| tristanC | fungi: upload-pypi job ran successfully, thanks again! | 22:04 |
| *** tosky has quit IRC | 22:10 | |
| *** tosky has joined #opendev | 22:11 | |
| fungi | tristanC: awesome, we also got the ownership change notifications, so all yours now | 22:14 |
| *** qchris has quit IRC | 22:42 | |
| *** tosky has quit IRC | 22:51 | |
| *** qchris has joined #opendev | 22:54 | |
| *** hamalq has quit IRC | 22:57 | |
| *** mlavalle has quit IRC | 22:58 | |
| *** hamalq has joined #opendev | 23:01 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!