| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [dnm] letsencrypt prototype implementation https://review.openstack.org/636759 | 00:15 |
|---|---|---|
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Fix build-docker-image when using buildset_registry https://review.openstack.org/637650 | 00:25 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [dnm] letsencrypt prototype implementation https://review.openstack.org/636759 | 00:42 |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Don't request PR issue data https://review.openstack.org/636728 | 00:59 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-preview master: WIP: test docker registry https://review.openstack.org/637037 | 01:01 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: WIP: Run zuul-preview https://review.openstack.org/637654 | 01:21 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: amqp: add basic trigger https://review.openstack.org/637458 | 01:32 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: amqp: add message informations to the job variables https://review.openstack.org/637666 | 02:27 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: amqp: add message informations to the job variables https://review.openstack.org/637666 | 03:08 |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: [dnm] testing devstack fix https://review.openstack.org/637669 | 03:11 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add triggers information to pipeline list https://review.openstack.org/637670 | 03:13 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: webtrigger: add initial driver and event https://review.openstack.org/555153 | 04:01 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: webtrigger: add web route and rpclistener https://review.openstack.org/554839 | 04:01 |
| *** Shrews has quit IRC | 04:17 | |
| *** Shrews has joined #opendev | 04:19 | |
| *** Shrews has quit IRC | 04:24 | |
| *** Shrews has joined #opendev | 04:25 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: amqp: add message informations to the job variables https://review.openstack.org/637666 | 04:57 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [dnm] letsencrypt prototype implementation https://review.openstack.org/636759 | 05:22 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Keep git after ironic-agent post https://review.openstack.org/637162 | 05:32 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul-jobs master: install-kubernetes: fix kube config permission https://review.openstack.org/637682 | 05:48 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: DNM: test install-kubernetes fix https://review.openstack.org/637683 | 05:49 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [dnm] letsencrypt prototype implementation https://review.openstack.org/636759 | 06:19 |
| openstackgerrit | Ian Wienand proposed openstack-infra/infra-specs master: letsencrypt spec https://review.openstack.org/587283 | 06:26 |
| *** bgmccollum has quit IRC | 06:53 | |
| *** bgmccollum has joined #opendev | 07:14 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul-jobs master: install-kubernetes: fix minikube config permission https://review.openstack.org/637682 | 07:32 |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/project-config master: Add tripleo-ci members as irc ops in oooq https://review.openstack.org/634438 | 07:42 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul-jobs master: install-kubernetes: fix minikube config permission https://review.openstack.org/637682 | 08:19 |
| openstackgerrit | zhongshengping proposed openstack/os-testr master: add python 3.7 unit test job https://review.openstack.org/637749 | 09:06 |
| openstackgerrit | zhongshengping proposed openstack/os-performance-tools master: add python 3.7 unit test job https://review.openstack.org/637759 | 09:06 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: CLI: fail if trying to enqueue/dequeue a change for the wrong project https://review.openstack.org/636662 | 09:57 |
| openstackgerrit | Merged openstack/diskimage-builder master: Keep git after ironic-agent post https://review.openstack.org/637162 | 10:08 |
| openstackgerrit | Jan Kundrát proposed openstack-infra/git-review master: Support usernames that contain '@' and ssh Git URLs https://review.openstack.org/428700 | 10:11 |
| openstackgerrit | Simon Westphahl proposed openstack-infra/zuul master: Show animated progress bar in preparation phase https://review.openstack.org/637810 | 10:11 |
| openstackgerrit | Simon Westphahl proposed openstack-infra/zuul master: Log to job output when running Ansible setup https://review.openstack.org/637813 | 10:24 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add jobs list filter https://review.openstack.org/633652 | 10:25 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: CLI: fail if trying to enqueue/dequeue a change for the wrong project https://review.openstack.org/636662 | 10:42 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: config: add statsd-server config parameter https://review.openstack.org/535560 | 10:53 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: CLI: fail if trying to enqueue/dequeue a change for the wrong project https://review.openstack.org/636662 | 11:01 |
| openstackgerrit | Merged openstack-infra/bindep master: Fix bindep --brief for arch linux https://review.openstack.org/637421 | 11:01 |
| openstackgerrit | Brendan proposed openstack-infra/zuul-jobs master: Use zuul_workspace_root variable for Git workspace prep https://review.openstack.org/636870 | 12:10 |
| openstackgerrit | Nir Magnezi proposed openstack/diskimage-builder master: [wip] rhel8 beta support https://review.openstack.org/623137 | 13:40 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: CLI: fail if trying to enqueue/dequeue a change for the wrong project https://review.openstack.org/636662 | 14:32 |
| openstackgerrit | Sean McGinnis proposed openstack-infra/openstack-zuul-jobs master: Add per-cycle Python job templates https://review.openstack.org/637866 | 14:58 |
| openstackgerrit | Merged openstack-infra/nodepool master: Properly handle TaskManagerStopped exception https://review.openstack.org/636393 | 15:26 |
| openstackgerrit | Miguel Lavalle proposed openstack-infra/irc-meetings master: Propose a new time for L3 subteam meeting https://review.openstack.org/637900 | 15:34 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: install-kubernetes: fix minikube config permission https://review.openstack.org/637682 | 15:45 |
| openstackgerrit | Quique Llorente proposed openstack-infra/zuul master: Ignore files at timer trigger https://review.openstack.org/637916 | 15:51 |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Rename project to project_name in getPullBySha https://review.openstack.org/637218 | 15:55 |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Test GithubShaCache https://review.openstack.org/637228 | 15:55 |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Switch to LRU based sha to PR cache https://review.openstack.org/637615 | 15:55 |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Switch to LRU based sha to PR cache https://review.openstack.org/637615 | 16:04 |
| *** dmsimard has joined #opendev | 16:06 | |
| openstackgerrit | Jan Kundrát proposed openstack-infra/nodepool master: Implement a Runc driver https://review.openstack.org/535556 | 16:16 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Fix typo in build api endpoint https://review.openstack.org/636227 | 16:28 |
| openstackgerrit | Jan Kundrát proposed openstack-infra/nodepool master: Implement a Runc driver https://review.openstack.org/535556 | 16:45 |
| * fungi is around | 17:03 | |
| corvus | clarkb, mordred: around? | 17:03 |
| clarkb | hello | 17:03 |
| corvus | let's use https://etherpad.openstack.org/p/ctzZNmV9CE if we need it | 17:03 |
| corvus | so the question at issue is, given https://github.com/go-gitea/gitea/issues/5798 should we continue to plan to deploy on k8s or just use plain docker | 17:05 |
| * mordred is here | 17:05 | |
| corvus | mordred: ooh, nice respnse on your comment there :) | 17:05 |
| mordred | \o/ | 17:05 |
| mordred | I think that's an excellent plan | 17:05 |
| corvus | oh, and just to make it clear, my hope is that no matter what, we eventually get to the system that we thought we were about to run -- a shared-everything multi-master running in k8s | 17:06 |
| corvus | so whatever we're talking about here today, i think should be considered temporary until that happens | 17:06 |
| clarkb | if that is the end goal I think I'd slightly prefer shared nothing in k8s | 17:06 |
| clarkb | then transition to shared everything in k8s | 17:06 |
| corvus | and based on mordred's comment, it looks like there's a roadmap to get there | 17:06 |
| clarkb | as that will give us familiarity with the tooling | 17:06 |
| fungi | yeah, i leaned that direction when we discussed it last week | 17:06 |
| fungi | or was that over the weekend? | 17:07 |
| corvus | i didn't do much discussing over the weekend :) | 17:07 |
| fungi | yeah, my notes say that was friday | 17:08 |
| corvus | okay i was about to write some stuff about how it might be easier to transition from shared-nothing docker to shared-everything k8s....but i think the real thing is that, if we did shared-nothing docker, i would like to keep the k8s cluster running | 17:08 |
| fungi | so anyway, shared nothing in kubernetes struck me as the least reinventing if the end goal is shared everything in kubernetes | 17:08 |
| clarkb | and I expect the transition will be take one of the shared nothings and turn it into a shared everything and delete the other shared nothings | 17:08 |
| clarkb | which in theory k8s makes easier for us when we get there | 17:09 |
| *** SotK has joined #opendev | 17:09 | |
| corvus | so that basically, we would continue to work on the k8s cluster even while using docker | 17:09 |
| corvus | the big question for me regarding shared-nothing-k8s is -- how do we set up replication? | 17:09 |
| clarkb | corvus: oh I was thinking the opposite re transition. We'd essentially tack on an elasticsearch then run more pods | 17:09 |
| clarkb | corvus: I think we can have a load balancer read only frontend for user facing http(s). Then have load balancer do 1:1 to individual ssh pods and set up gerrit to push to each of those | 17:10 |
| fungi | what needs replicating under shared-nothing if we start uot just providing read-only repository browsing and searching? just the git repos themselves i guess? | 17:10 |
| corvus | fungi: yeah, in a shared nothing system, we need gerrit to replicate to (let's use our current number) 8 different locations | 17:11 |
| mordred | corvus: yeah - sort of what clarkb said - I think we'll want to expose port 22 of each individual gitea pod via a non-loadbalancer service | 17:11 |
| corvus | so, does anyone know how to do that in k8s? :) | 17:11 |
| mordred | corvus: and then put each of those into the gerrit | 17:11 |
| mordred | corvus: I can put it on my learn-how-to list if that's where we go - I think we just want to add some service objects with a clusterip type | 17:12 |
| fungi | oh, so it uses the same ssh port for serving git+ssh as for pushing into, i guess that's the challenge? | 17:12 |
| corvus | mordred: i guess that's a different kind of service, but still something special that's going to cause k8s-on-openstack to get a fip from neutron? | 17:12 |
| mordred | corvus: yeah. I think so - obviously need to test that theory | 17:12 |
| clarkb | fungi: I don't think we planned to serve git+ssh generally | 17:13 |
| clarkb | fungi: its merely a way to push replicated git repos from gerrit into gitea I think | 17:13 |
| corvus | right | 17:13 |
| fungi | oh, i see | 17:13 |
| corvus | in all cases, the ssh port is only for gerrit | 17:14 |
| fungi | okay, so we don't need both lb and non-lb for ssh in that case | 17:14 |
| corvus | in shared-everything, our load balancer has 3 ports, 2 of them public http(s), and one of them is ssh, used only by gerrit. they all map to any of the pods, because any pod can handle a git push | 17:14 |
| clarkb | except as I noted in the etherpad I think the high level concept of a load balancer is how you expose public IPs in k8s even if not really load balancing anything | 17:14 |
| corvus | in shared-nothing, we still have the lb with http(s) to all pods, but each pod needs its own incoming ssh listener for gerrit | 17:15 |
| mordred | clarkb: well - actually, that's concept is the "service" I believe - one of the types of Service is LoadBalancer | 17:15 |
| clarkb | mordred: ya `kubectl expose deployment hello-world --type=LoadBalancer --name=my-service` from docs at https://kubernetes.io/docs/tutorials/stateless-application/expose-external-ip-address/ | 17:16 |
| clarkb | basically our terminology is conflicting here. K8s calls it a load balancer even if you don't load balance anything | 17:16 |
| clarkb | you just set up that service per singleton deployment | 17:16 |
| corvus | if we have an octavia lb for each pod, that's rather heavyweight, isn't it? | 17:17 |
| mordred | http://git.openstack.org/cgit/openstack-infra/system-config/tree/kubernetes/gitea/k8s/service.yaml <-- that's how we do the load balancer | 17:17 |
| mordred | http://git.openstack.org/cgit/openstack-infra/system-config/tree/kubernetes/percona-xtradb-cluster/service-percona.yaml is a non-loadbalancer service | 17:18 |
| clarkb | corvus: yes it is possible we'd want to manage that external to k8s in the k8s shared-nothing setup similar to how we'd do it with the docker only setup | 17:18 |
| mordred | although it's just exposing that service locally to the cluster | 17:18 |
| corvus | mordred: right, though that's only internal | 17:18 |
| clarkb | https://kubernetes.io/docs/concepts/services-networking/#external-ips may be what we want which is the non load balancer option | 17:21 |
| mordred | reading through things real quick on https://kubernetes.io/docs/concepts/services-networking/service/ - we might just want to have some ansible splat a floating ip on to internal k8s address (I do think spinning up 8 octavia load balancers starts to get a bit silly) | 17:21 |
| clarkb | mordred: ya then expose that via external ips? | 17:21 |
| mordred | yeah. potentially so | 17:21 |
| corvus | i added a second question -- which is whether we can even individually address the pods with a service at all... | 17:22 |
| clarkb | corvus: I think you can via external-ips and node affinity/pinning | 17:22 |
| mordred | well - I think what we probably want is a statefulset similar to how the percona xtradb cluster is set up | 17:23 |
| clarkb | corvus: you'd tell k8s to expose the pods on these external IPs then map pod1 to host1+ip1 and pod2 to host2+ip2 | 17:23 |
| mordred | so we tell k8s "I want 8 of these, but I want _these_ 8 - and I want them to come back in this shape when they go away and come back" | 17:23 |
| clarkb | ++ | 17:23 |
| mordred | then we give the statefulset the pod spec that looks like our current gitea pod | 17:24 |
| mordred | now - the thing is - this is going to be pretty radically different in shape than our current gitea k8s | 17:24 |
| mordred | and migrating from it to shared disk k8s might not be any easier than migrating from docker-compose | 17:24 |
| mordred | so I think what we'd be getting familiar with is running a something in k8s - but the shape of the something is gonna be pretty different | 17:25 |
| corvus | i suspect that if we started with k8s, we would just migrate to a second k8s cluster when they fix the index issue... | 17:25 |
| mordred | yeah - and maybe take that opportunity to deploy rook with CSI rather than FlexVolume | 17:26 |
| corvus | there's rook to consider too -- by the time they fix that, the new rook....yeah that | 17:26 |
| clarkb | considering rook doesn't do in place upgrades that may be the best approach anyway | 17:26 |
| corvus | that probably wants a redployment anyway, considering how invasive it is | 17:26 |
| mordred | or CNS ... or whatever | 17:26 |
| mordred | yah | 17:27 |
| corvus | so either way, we're talking about migrating to a new k8s cluster when ES becomes an index option | 17:27 |
| fungi | i'm not familiar with what filesystems gitea relies on... does it keep the git repos in a native git disk format or is it keeping them in some sort of gitea-specific format? | 17:28 |
| corvus | fungi: native git | 17:28 |
| fungi | oh, but we can't push directly onto the filesystem because it relies on its ssh interface to trigger indexing, right? | 17:29 |
| mordred | yah | 17:29 |
| corvus | fungi: yep | 17:29 |
| fungi | and there's no way to push into just one of them yet have them all share the same index off the filesystem, because they all want to obtain a write lock on the index | 17:30 |
| openstackgerrit | Merged openstack-infra/zuul master: Fix typo in build api endpoint https://review.openstack.org/636227 | 17:30 |
| fungi | and presumably wouldn't know when to reread the index into memory i suppose | 17:30 |
| mordred | yah. that's the issue that they're looking to solve by adding optional support for elastic | 17:30 |
| clarkb | fungi: ya if they supported read only instances it would work that way, but tehy don't | 17:30 |
| corvus | there is one thing we haven't looked into... | 17:31 |
| clarkb | (elasticsearch and multi master is better fix though) | 17:31 |
| corvus | the idea of still having one git filesystem, but having separate indexes, and somehow getting the triggering event to all of the pods individually | 17:31 |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Switch to LRU based sha to PR cache https://review.openstack.org/637615 | 17:32 |
| corvus | i think that would basically mean replacing a gitea component with some kind of shim we write ourselves. | 17:32 |
| corvus | it sounds like a terrible idea to me. :) | 17:32 |
| corvus | but it might be possible. | 17:32 |
| mordred | corvus: yeah. it does. but it is worth mentioning :) | 17:32 |
| mordred | corvus: becuase I'm pretty sure the indexing event is triggered with a rest payload | 17:32 |
| clarkb | I wonder what would happen if the gitea process flock failed | 17:32 |
| clarkb | maybe that is our shim, mounting the fs in such a way that they can't lock for writing but maybe if well behaved would read for reading | 17:33 |
| mordred | corvus: so we could, potentially, put in a git repo hook something something that hit all 8 of them | 17:33 |
| mordred | clarkb: I believe we can configure them to each have different index files | 17:33 |
| clarkb | (I doubt it is that well behaved but it is surprising it worked in the old buggy code given the comments in bleve/boltdb around the dangers of multiwriters) | 17:33 |
| mordred | clarkb: and they would attempt to flock that separate index file | 17:33 |
| clarkb | mordred: ya we can configure that. | 17:33 |
| corvus | mordred: yeah. i think we'd probably just replace the gitea ssh command with a reimplementiation that hit all of them instead of localhost. | 17:33 |
| clarkb | mordred: btu then we have to replicate the push events | 17:33 |
| fungi | oh, an ssh multiplexer? | 17:33 |
| clarkb | mordred: I was thinking maybe theywould just give up on the flock and open for reading if all but one get a read only fs | 17:34 |
| mordred | corvus: yeah. and we could potentially tempalte that with ansible when we push things out | 17:34 |
| corvus | fungi: no, i think we'd still have one ssh listener, but the ssh "command=" that we run would be our own, and not gitea. | 17:34 |
| fungi | aha | 17:34 |
| corvus | currently it's "command=gitea serv --user=1" or something... | 17:34 |
| corvus | so we'd do "command=/usr/local/bin/hit-all-gitea-urls.py" :) | 17:34 |
| mordred | corvus: I think that woudl take at least as much time to get right as figuring out the right statefulset incantations to get the full shared nothing with 8 ips ... and I don't think any of us would grow super useful knowledge as a result | 17:35 |
| corvus | but i think that command is doing the git recv-pack stuff as well, so it's not exactly straightforward. | 17:35 |
| fungi | and all that really solves is the kubernetes load-balancing configuration difference and having just one replication destination in gerrit configs | 17:36 |
| clarkb | another terrible idea: monkeypatch the fix out | 17:36 |
| mordred | hah | 17:36 |
| clarkb | it will probably work until it doesn't :) | 17:36 |
| corvus | clarkb: yeah... like, it may work until we restart :) | 17:37 |
| mordred | I mean - we ARE building our own images - so we _could_ do that ... but oy. | 17:37 |
| mordred | we coudl also just go back to running 1.6.0 and stay there until a release with elastic | 17:38 |
| clarkb | mordred: 1.7.x includes security fixes | 17:38 |
| clarkb | they are telling everyone to upgrade | 17:38 |
| mordred | yeah - but are they relevant to our use of gitea | 17:39 |
| clarkb | mordred: that I don't know | 17:39 |
| corvus | mordred: i think i have the same worry about that though -- it works until it doesn't | 17:39 |
| mordred | like -w e don't let users log in - nobody is using issues or prs or user accounts or ssh uploads | 17:39 |
| mordred | corvus: fair | 17:39 |
| clarkb | corvus: ya if the bug is present in 1.6.0 it isn't any better than patching out the fix | 17:39 |
| clarkb | fwiw I think the shared nothing appraoch is likely our best option in the near term | 17:39 |
| clarkb | and mostly a decision of whether or not we want ot k8s it. I think if the k8sing isn't too bad for shared nothing we should probably do that given our end goal is to k8s shared everything | 17:40 |
| corvus | also, we don't even really know how well it was working... did each pod really use a shared index? or did each pod have its own index, and we just happened not to notice when we did searches. and were we reindexing the whole system on every startup? | 17:40 |
| corvus | my gut still says "the docker stuff is simple and ready to go (with testing); the k8s approach means a lot of work to figure out how to make it work like docker". i'm not at all opposed to it, if someone wants to jump in and figure that stuff out; i'm personally a little less enthused because of the complexity/longevity ratio here. but i admit, we'd probably be more expert at k8s by the time we're done. | 17:40 |
| corvus | :) | 17:40 |
| corvus | i guess what i'm looking for is someone to say "i will figure out the open questions about shared-nothing k8s". if someone who isn't me wants to do that, i'm on board and will be happy to help. | 17:42 |
| clarkb | re how it was working: My reading of the code is every process deleted the shared index on start then created a new index which allowed it to get a different file to flock. They were all essentially updating transient indexes. This oprobably means that you could get different search results from different backends | 17:42 |
| fungi | before this discussion, i didn't realize that switching out the deployment model entirely was easier than adjusting the existing one for a different configuration | 17:42 |
| openstackgerrit | Merged openstack-infra/zuul master: Don't request PR issue data https://review.openstack.org/636728 | 17:42 |
| fungi | nor was i aware/remembering we had docker-based deployment independent from kubernetes | 17:43 |
| mordred | I'm still torn on which is the best - I think doing k8s gets us more k8s knowledge which is good and where we want to be - but otoh compose is ready to go (with testing) so there's no more dev to do | 17:43 |
| clarkb | fungi: the new docker image registry is docker based deployment without k8s | 17:43 |
| corvus | fungi: we have 1 service in production with independent docker already :) | 17:43 |
| corvus | if you haven't seen it, here's the docker patch: https://review.openstack.org/637330 | 17:44 |
| clarkb | a good compromise is probably to keep the k8s up and running, use it to work on deploying gitea how we want it in k8s, but deploy gitea for near term prod with the docker compose work that is already done | 17:44 |
| mordred | so - in a vacuum I'd vote for shared-nothing-k8s - but compose being ready to go and shared-nothing-k8s needing to be written I think tips me back to "let's just use compose as a temporary" | 17:44 |
| mordred | clarkb: ++ | 17:44 |
| corvus | yeah, i'm all for keeping the k8s cluster up, and continuing work on integrating the k8s-on-openstack into our ansible, and figuring out testing :) | 17:45 |
| mordred | yeah | 17:45 |
| fungi | what service am i forgetting we're already running in production with docker? | 17:45 |
| mordred | like - let's keep the ansible going even - maybe turn gitea deployment replicas back down to 1? | 17:46 |
| corvus | fungi: insecure-ci-registry.opendev.org | 17:46 |
| corvus | the intermediate docker image registry | 17:46 |
| clarkb | mordred: ya thats a good step 0 given what we know about modern gitea | 17:46 |
| mordred | that way the service will still be running there - and we can poke at it by ip | 17:46 |
| clarkb | mordred: and upgrade it and do all the other things that are still useful without many pods sharing everything | 17:46 |
| clarkb | I like that idea | 17:46 |
| mordred | yah | 17:47 |
| mordred | and keep gerrit replicating to it even | 17:47 |
| mordred | cause why not | 17:47 |
| corvus | fungi: whatcha think ^? | 17:47 |
| fungi | makes sense | 17:48 |
| fungi | i'm cool with that plan | 17:48 |
| corvus | okay, i think we have consensus on: proceed with docker, keep k8s running and continue to integrate it, move to k8s when gitea speaks ES. | 17:48 |
| fungi | i like that the docker-compose path is already tested | 17:49 |
| fungi | (continuously tested i mean) | 17:49 |
| mordred | ++ | 17:49 |
| clarkb | thinking out loud about this generally, this is a good real world illustration of how "just put it in k8s" might not actually help solve real problems | 17:49 |
| corvus | next i'm supposed to make a list of outstanding tasks for opendev-gerrit, but i think i'm going to do that after we get the docker-compose stuff into semi-production (like we thought we had k8s until friday) | 17:49 |
| clarkb | so we probably shouldn't expect all our other services to map into k8s necessarily | 17:50 |
| clarkb | corvus: wfm | 17:50 |
| mordred | clarkb: ++ I think that's a great thing for everyone to always remember | 17:50 |
| fungi | clarkb: ahh, but "put it in kubernetes" fans assume your services are already "cloud-native" which, i think, means they would also recommend rewriting gitea to not keep its own data or something like that | 17:50 |
| corvus | pesky data | 17:51 |
| mordred | so pesky | 17:51 |
| mordred | otoh - I think the elastic support is going to be a really nice addition to gitea out of this | 17:51 |
| clarkb | who needs state | 17:51 |
| corvus | okay, https://review.openstack.org/637330 is ready for +3 then. i can spin up some servers after it lands. | 17:51 |
| corvus | mordred: yeah, and running elastic in k8s sounds like a good idea :) | 17:51 |
| clarkb | cool I'll review once I'm caught up on my zuul changes tobish is kindly reviewing | 17:51 |
| fungi | out of curiosity, why is this xenial instead of bionic? | 17:53 |
| fungi | or is the comment in playbooks/roles/gitea/templates/docker-compose.yaml.j2 just copy-paste of staleness? | 17:54 |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Clarify project vs repository in getPullBySha https://review.openstack.org/637956 | 17:54 |
| mordred | corvus: the host-networking patch at the end of that stack has a sad, but +2 on the first 2 | 17:55 |
| clarkb | fungi: I think our k8s work was all xenial because they don't have bionic packages | 17:55 |
| clarkb | fungi: but this docker compose work could be bionic likely | 17:55 |
| fungi | also, i suspect we're going to get a nonzero number of concerned researchers letting us know about the test credentials they think we accidentally published in playbooks/zuul/templates/group_vars/gitea.yaml.j2 | 17:55 |
| mordred | fungi: :) | 17:56 |
| corvus | mordred: yes! it has a sad because i still haven't finished speculative docker containers :( | 17:58 |
| corvus | mordred: until i do that, we'll have to land the patch ahead of it blind, publish the container, recheck and hope for the best | 17:58 |
| mordred | corvus: ah - nod | 17:59 |
| corvus | fungi: it is bionic. but i'm still running xenial on my workstation so all my testing is with docker-compose version 2. and we don't need anything later yet. | 17:59 |
| corvus | mordred: but it's still good news that the test is failing as expected. | 18:00 |
| fungi | got it | 18:00 |
| openstackgerrit | Merged openstack-infra/zuul master: Add dockerized test setup https://review.openstack.org/636424 | 18:21 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Docs addition on decrypting Zuul secrets https://review.openstack.org/637969 | 18:43 |
| openstackgerrit | Merged openstack-infra/zuul master: Rename project to project_name in getPullBySha https://review.openstack.org/637218 | 18:51 |
| openstackgerrit | Merged openstack-infra/zuul master: Test GithubShaCache https://review.openstack.org/637228 | 18:51 |
| openstackgerrit | Merged openstack-infra/zuul master: Switch to LRU based sha to PR cache https://review.openstack.org/637615 | 18:51 |
| openstackgerrit | Merged openstack-infra/zuul master: Clarify project vs repository in getPullBySha https://review.openstack.org/637956 | 18:55 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Remove reference to localhost in zuul_return docs https://review.openstack.org/637974 | 19:09 |
| openstackgerrit | Merged openstack-infra/system-config master: Add new openstackid01 host to inventory and cacti https://review.openstack.org/637633 | 19:22 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Switch artifact return back to list https://review.openstack.org/637979 | 19:23 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Have gitea sshd listen on 222 https://review.openstack.org/637333 | 19:46 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Use host networking for gitea https://review.openstack.org/637334 | 19:46 |
| openstackgerrit | Merged openstack-infra/project-config master: Add cinderlib project https://review.openstack.org/637613 | 20:00 |
| *** openstackgerrit has quit IRC | 20:09 | |
| *** openstackgerrit has joined #opendev | 20:27 | |
| openstackgerrit | Merged openstack-infra/system-config master: Deploy gitea with docker-compose https://review.openstack.org/637330 | 20:27 |
| openstackgerrit | Jan Kundrát proposed openstack-infra/zuul master: More parallelism for git clones and checkouts https://review.openstack.org/637996 | 20:37 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Combine zuul.artifacts lists in zuul_return https://review.openstack.org/638005 | 20:55 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Use list form of zuul artifact return https://review.openstack.org/638007 | 21:00 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Allow nb*.openstack.org to talk to graphite https://review.openstack.org/638011 | 21:07 |
| openstackgerrit | Merged openstack-infra/zuul master: Switch artifact return back to list https://review.openstack.org/637979 | 21:49 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Use list form of zuul artifact return https://review.openstack.org/638007 | 21:52 |
| openstackgerrit | Merged openstack-infra/zuul master: Combine zuul.artifacts lists in zuul_return https://review.openstack.org/638005 | 21:56 |
| mnaser | mordred, clarkb, corvus: you can use a concept of a 'nodeport' in k8s to export a pod at a specific port on a server | 21:57 |
| mnaser | https://kubernetes.io/docs/concepts/services-networking/service/#nodeport | 21:57 |
| mnaser | the only problem is the port range is not the ideal one to use | 21:57 |
| mordred | mnaser: yeah. I think going half-way with this stuff gets really squirrely | 21:58 |
| mordred | still - good link thanks | 21:58 |
| mnaser | mordred: imho the best approach i've been doing is use docker as a pkg manager | 21:59 |
| mnaser | which paves the way because it means you do maybe 80% of the work at containerizing your workload | 21:59 |
| mnaser | and then when you figure out the best k8s strategy (is there any?!) you can leverage teh same work you already did | 21:59 |
| mnaser | ive been using a lot of docker_container with ansible to orchestrate all this stuff | 22:00 |
| mordred | mnaser: yah - that's basically where we've gotten to as well | 22:01 |
| mordred | mnaser: docker-as-packaging, ansible to deploy/run | 22:02 |
| mnaser | yep! | 22:02 |
| mordred | mnaser: means you can do a bunch of your work in the build steps | 22:02 |
| mnaser | really quick, easily repeatable | 22:02 |
| mordred | mnaser: have you seen all the super sexy image promotion stuff corvus has been doing? | 22:02 |
| mnaser | a little bit but not too much | 22:03 |
| mordred | mnaser: tl;dr build image in gate, gate test it - then once it lands, push the exact image you used in the gate up to your image registry | 22:03 |
| mnaser | mordred: so i assume using artifact passing | 22:03 |
| mordred | mnaser: aka - speculative container images | 22:03 |
| corvus | this is used currently for zuul images | 22:03 |
| mnaser | because different jobs do this, right? | 22:04 |
| mnaser | one in gate and one in post, or is it just one job | 22:04 |
| corvus | well, the really speculative stuff is still to come, where we can build a container image based one one change, and use it in another change, all in the check pipeline | 22:04 |
| mordred | yeah. that stuff is super cool | 22:05 |
| corvus | mnaser: one job in gate, another in "promote" | 22:05 |
| corvus | mnaser: the next step will be a third job in "check" | 22:05 |
| mordred | but the stuff we're going now pushes the image to docker hub but with a 'gate_' tag on it - so then the promote pipeline can retag it | 22:05 |
| mordred | ++ | 22:05 |
| mnaser | ok i see | 22:05 |
| mnaser | so passing of the artifacting doesnt happen by say, push to swift | 22:05 |
| mnaser | and then download from swift afterwards | 22:05 |
| corvus | super simple example: this change https://review.openstack.org/637334 depends on this container image change: https://review.openstack.org/637333 | 22:06 |
| corvus | it's failing now, because the container isn't actually built | 22:06 |
| corvus | but when i finish speculative containers, that first change would be able to pass before the second one lands | 22:06 |
| mnaser | aaaaaah | 22:06 |
| openstackgerrit | Clark Boylan proposed openstack-infra/infra-specs master: Update priority effort Gerrit topics https://review.openstack.org/638024 | 22:07 |
| mnaser | really neat. | 22:07 |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Enable DIB_SHOW_IMAGE_USAGE for build https://review.openstack.org/638031 | 22:42 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Run an haproxy load balancer for gitea https://review.openstack.org/638033 | 22:45 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Allow extra data in artifact schema validation https://review.openstack.org/638038 | 22:59 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-zuul master: Use jemalloc on Zuul v3 executors https://review.openstack.org/638044 | 23:16 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Run an haproxy load balancer for gitea https://review.openstack.org/638033 | 23:19 |
| openstackgerrit | Merged openstack-infra/zuul master: Allow extra data in artifact schema validation https://review.openstack.org/638038 | 23:33 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!