| *** tobiash has joined #opendev | 07:51 | |
| *** clarkb has joined #opendev | 20:01 | |
| *** ianw has joined #opendev | 20:01 | |
| fungi | did the announcement say we were going to have text discussion in here in parallel with the conference call? | 20:01 |
|---|---|---|
| corvus | i think i mentioned it in the second email | 20:01 |
| fungi | thanks, that's what i thought i recalled reading | 20:02 |
| corvus | i don't expect to be able to type a lot, but it might be nice to share links, etc, and have an out-of-band channel | 20:02 |
| clarkb | hrm I said I had it working on android but now I don't | 20:03 |
| clarkb | computers how do they work | 20:03 |
| *** jhesketh has joined #opendev | 20:03 | |
| ianw | clarkb: did you just "dial" 6561@pbx.openstack.org" ?\ | 20:04 |
| *** mordred has joined #opendev | 20:04 | |
| fungi | clarkb: i use the mute button on my phone to silence my loud keyboard, but i guess that won't help for corvus | 20:05 |
| clarkb | ianw: I told mizudroid to connect to pbx.openstack.org as server, username is 'conference', and password is foo | 20:05 |
| clarkb | ianw: then it fails and in the phone pad uri thing that comes up I typed in sip:conference@pbx.openstack.org | 20:05 |
| clarkb | and followed the prompt to join 6561 | 20:06 |
| ianw | ok i can hear something about renovations | 20:06 |
| *** Shrews has joined #opendev | 20:07 | |
| ianw | ok, i've muted to avoid background noise cool | 20:07 |
| clarkb | I'm ok with audio recording | 20:07 |
| fungi | glad this didn't end up being tomorrow. my roof is getting replaced starting at 8am local | 20:08 |
| corvus | we're recording the audio, and we'll be recording the terminal session (starting shortly) too | 20:09 |
| fungi | it's like corvus is shaking a can of gravel ;) | 20:09 |
| clarkb | not too loud | 20:09 |
| ianw | hmm, sounds like a 1987 era model M ... light grey, ps/2 | 20:10 |
| corvus | telnet bridge.openstack.org 8888 | 20:10 |
| fungi | i see a terminal session! | 20:10 |
| fungi | ianw: i have that one | 20:10 |
| jhesketh | it was very slow for me, but I'm there | 20:11 |
| clarkb | I see the session | 20:11 |
| jhesketh | (I'm assuming my mute is working and nobody can hear me sneezing? :-p) | 20:12 |
| Shrews | jhesketh: didn't hear you | 20:13 |
| fungi | as if anyone would accidentally expose an important password and then have to change it quickly ;) | 20:13 |
| mordred | https://github.com/infraly/k8s-on-openstack | 20:15 |
| mordred | https://review.openstack.org/#/c/626965 | 20:15 |
| ianw | so when you ssh in, you're actually in a different container to gitea, but with views of most of the shared volumes? | 20:31 |
| fungi | this is the "special" ssh service gitea needs to be able to support git+ssh protocol, right? | 20:32 |
| corvus | fungi: yep | 20:34 |
| fungi | just making sure "ssh in a container" wasn't generally going to be our management pattern for other systems or something | 20:35 |
| corvus | yeah, neither of those containers runs a regular ssh service; the only way into them with a shell is via k8s | 20:35 |
| ianw | ahh, right that makes sense | 20:35 |
| fungi | presumably there is an sshd also running in the primary non-container context | 20:35 |
| corvus | fungi, ianw: yes | 20:36 |
| corvus | i think we can demo some things to illustrate this in a sec | 20:36 |
| fungi | did monty suddenly explode? | 20:37 |
| fungi | oh, he's still there | 20:37 |
| fungi | thanks! | 20:45 |
| jhesketh | why are we using cephfs instead of a cinder volume? | 20:51 |
| jhesketh | okay, I follow why we want the shared storage, but it was mentioned earlier that we can attach cinder volumes to a container so I'm unsure why that wouldn't work? | 20:53 |
| clarkb | before I forget, on the topic of sorting out logs things the OSH team's test jobs would restart failed conatiners/pods over and over and over and not record the logs from the failed instances. We should see if we can have the log rotation store instances of failed pod/container logs longer than the other stuff | 20:54 |
| jhesketh | ah okay, I appear to have made a bad assumption that you can attach the same cinder volume to multiple pods/containers | 20:55 |
| jhesketh | thanks for the answers :-) | 20:55 |
| fungi | cinder multi-attach is only half the solution if you want multiple systems writing | 20:55 |
| fungi | usually that's for multiple readers and then you have a protocol to pass around the "write token" so systems know when it's their turn to flush | 20:56 |
| * jhesketh nods | 20:58 | |
| clarkb | questions/thoughts as we go through this. Can we restrict access to the k8s api to say just the bridge? is that desireable? If we delete the master node will the minions keep running as is or is that an outage? | 21:04 |
| fungi | also reminds me that i want to understand enough of this to know whether things like https://groups.google.com/forum/#!topic/kubernetes-dev/P0ghX_DViy8 are something we need to worry about | 21:07 |
| clarkb | fungi: potentially access to the db via that method would be bad | 21:08 |
| corvus | back at 21:26 utc | 21:21 |
| clarkb | for when we get back one of the things that interests me is how our config management through code review would work with such a system. k8s gives you the flexibility to do adhoc edits to your configs or you can reapply templates and it does fancy diffs to figure out what the equivalent adhoc patch to apply is. I kind of figure we'll rely on applying entire yaml templates which can be automated from code | 21:23 |
| clarkb | review, but then we do also need some way to reconcile or audit any out of band adhoc edits (so that we can capture them or remove them if necessary?) that might be looking way far down the road though | 21:23 |
| corvus | i think we can talk about that as part of the next thing, since i have ansible written already to do the gitea deployment | 21:24 |
| mordred | clarkb: yes - for the most part our interaction has been in teh form of applying entire yaml templates ... and we have ansible to do that for us as well | 21:24 |
| mordred | corvus: I believe we were typing at the same time :) | 21:24 |
| corvus | and said the ~same thing :) | 21:25 |
| mordred | clarkb: I'd also think that we'd want to consider out-of-band ad-hoc edits in the same way as manual changes to servers at the moment - occasionally necessary, but not a general daily practice and one that needs to be resolved once the need has passed | 21:26 |
| mordred | incidentally, SpamapS made a tool while at godaddy that he uses to get a diff between a yaml he's about to apply and the existing state of the server for previewing things | 21:27 |
| corvus | we're back! | 21:27 |
| clarkb | mordred: ++ the difference is puppet or ansible will often undo what I did adhoc | 21:27 |
| mordred | I have not yet used it myself, but he says it has been very helful | 21:27 |
| clarkb | where as k8s will preserve them aiui through fancy 3 way diffing | 21:27 |
| mordred | clarkb: yeah - well, so will kubectl potentially :) | 21:27 |
| fungi | yeah, back just don't feel like taking my phone off mute ;) | 21:27 |
| clarkb | Im in fungi's boat | 21:27 |
| mordred | ./kubectl api-resources --verbs=list --namespaced -o name | xargs -n1 ./kubectl get --show-kind --ignore-not-found -n rook-ceph | 21:28 |
| fungi | also, agree with mordred's reply above about treating manual configuration modification the same way as we do now: avoid them, and communicate when you can't | 21:29 |
| fungi | i don't think we can prevent them any more than we can today, and trying to accommodate them seems like a lot of work just to prop up incidental needs | 21:30 |
| clarkb | the difference is that k8s preserves them forward through the future | 21:33 |
| clarkb | it doesn't overwrite unless you later modify the specific attribute that was changed before | 21:33 |
| clarkb | aiui (it does fancy 3 way diffs and revision control of yaml to do that) | 21:33 |
| fungi | clarkb: got it, so we may want to reapply a complete manifest of configuration every time? | 21:34 |
| clarkb | maybe? not sure if that is an option. | 21:34 |
| fungi | yeah, the broadcast worked fine on my terminal | 21:40 |
| clarkb | my clietn seems to have dropepd me from the call | 21:40 |
| clarkb | I'm back | 21:40 |
| ianw | clarkb: that android client ... i seem to have to keep touching it to keep it in the foreground i think | 21:40 |
| fungi | that's one needy client | 21:41 |
| clarkb | ianw: ya I'm going to have to find another one after this | 21:42 |
| clarkb | but this is the first one I've found that will do direct connections like this | 21:42 |
| ianw | covus: so that inventory file ... the master / minion servers in there you've created? | 21:42 |
| fungi | score! | 21:43 |
| ianw | oh, ok thanks i see now. sorry was looking at inventory up one level and getting confused | 21:44 |
| jhesketh | I've got to drop off unfortunately. I'll catch the rest in the recording. Thanks so much for the demo! | 22:16 |
| corvus | ETOOMANYCONTAINERS | 22:18 |
| fungi | yes, this has been great, thanks again! unfortunately i have to go pick christine up from work now but please do post the link to the audio recording once it's up somewhere | 22:18 |
| mordred | clarkb: https://kubernetes.io/docs/concepts/cluster-administration/logging/#cluster-level-logging-architectures | 22:51 |
| ianw | thanks for this overview, been very helpful to understand what's going on | 22:52 |
| clarkb | mordred: reading that it seems like the logs should go into journalctl and then whatever rotation we have for that would be used? | 23:01 |
| clarkb | I think that means we should have everything go to syslog on ubuntu xenial as long as rsyslog is installed too (but kubelet likely won't find the logs tehre if you run kubectl log) | 23:01 |
| clarkb | corvus: ^ I think by default journactl on ubuntu runs with a ring buffer because it doesn't save to disk. We can change that and maybe make things better logging wise | 23:02 |
| corvus | i think that explains/matches the behavior i've seen | 23:02 |
| mordred | clarkb: https://github.com/infraly/k8s-on-openstack/blob/266f9b08cfab106e67d571adb1d8c59aaa73ce11/roles/kubeadm/tasks/main.yaml#L65 | 23:07 |
| mordred | sorry - the one above that | 23:07 |
| mordred | https://github.com/infraly/k8s-on-openstack/blob/266f9b08cfab106e67d571adb1d8c59aaa73ce11/roles/kubeadm/tasks/main.yaml#L52 | 23:07 |
| mordred | so the current deploy configures the docker to log to journald | 23:08 |
| clarkb | perfect then ya I think we just make jrounald persist its data with some rotation | 23:08 |
| clarkb | which is a config flag in /etc/journad.conf or something | 23:08 |
| mordred | clarkb: ++ | 23:14 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!