| *** d34dh0r53 has joined #openstack-infra | 00:14 | |
| *** zxiiro has joined #openstack-infra | 00:14 | |
| *** hongbin has quit IRC | 00:35 | |
| *** hongbin has joined #openstack-infra | 00:57 | |
| *** redrobot has quit IRC | 01:08 | |
| *** dannins has joined #openstack-infra | 02:15 | |
| *** hongbin has quit IRC | 02:23 | |
| *** hongbin has joined #openstack-infra | 02:31 | |
| *** ramishra has joined #openstack-infra | 02:59 | |
| *** dave-mccowan has quit IRC | 03:23 | |
| *** psachin has joined #openstack-infra | 03:41 | |
| *** ramishra has quit IRC | 03:59 | |
| *** ykarel|away has joined #openstack-infra | 04:05 | |
| *** zzzeek has quit IRC | 04:15 | |
| *** zzzeek has joined #openstack-infra | 04:17 | |
| *** hongbin has quit IRC | 04:20 | |
| *** ramishra has joined #openstack-infra | 04:21 | |
| *** ykarel|away is now known as ykarel | 04:21 | |
| *** hongbin has joined #openstack-infra | 04:24 | |
| *** hongbin has quit IRC | 04:29 | |
| *** rcernin has quit IRC | 04:31 | |
| *** evrardjp has quit IRC | 04:33 | |
| *** evrardjp has joined #openstack-infra | 04:33 | |
| *** zxiiro has quit IRC | 04:35 | |
| *** hongbin has joined #openstack-infra | 04:38 | |
| *** rcernin has joined #openstack-infra | 04:40 | |
| *** vishalmanchanda has joined #openstack-infra | 05:15 | |
| *** hongbin has quit IRC | 05:29 | |
| *** hongbin has joined #openstack-infra | 05:31 | |
| *** hongbin has quit IRC | 05:32 | |
| *** ociuhandu has joined #openstack-infra | 05:35 | |
| *** hongbin has joined #openstack-infra | 05:36 | |
| *** rcernin has quit IRC | 05:39 | |
| *** ociuhandu has quit IRC | 05:40 | |
| *** hongbin has quit IRC | 05:40 | |
| *** abhishekk has joined #openstack-infra | 05:47 | |
| *** rcernin has joined #openstack-infra | 06:01 | |
| *** matt_kosut has joined #openstack-infra | 06:07 | |
| *** Lucas_Gray has joined #openstack-infra | 06:08 | |
| *** lmiccini has joined #openstack-infra | 06:08 | |
| *** rcernin has quit IRC | 06:20 | |
| *** eolivare has joined #openstack-infra | 06:22 | |
| *** Lucas_Gray has quit IRC | 06:22 | |
| *** ralonsoh has joined #openstack-infra | 06:24 | |
| *** jtomasek has joined #openstack-infra | 06:44 | |
| *** andrewbonney has joined #openstack-infra | 06:50 | |
| *** bhagyashris is now known as bhagyashri|rover | 06:55 | |
| *** hashar has joined #openstack-infra | 07:01 | |
| *** slaweq has joined #openstack-infra | 07:03 | |
| *** tosky has joined #openstack-infra | 07:03 | |
| *** jcapitao has joined #openstack-infra | 07:07 | |
| *** gfidente has joined #openstack-infra | 07:27 | |
| *** xek has joined #openstack-infra | 07:37 | |
| *** rcernin has joined #openstack-infra | 07:39 | |
| *** jtomasek has quit IRC | 07:41 | |
| *** jtomasek has joined #openstack-infra | 07:50 | |
| *** jpena|off is now known as jpena | 07:50 | |
| *** rcernin has quit IRC | 07:52 | |
| cgoncalves | AJaeger, good morning. could you kindly review https://review.opendev.org/#/c/751829/ ? thanks! | 07:58 |
|---|---|---|
| *** lucasagomes has joined #openstack-infra | 08:00 | |
| *** Lucas_Gray has joined #openstack-infra | 08:13 | |
| *** auristor has quit IRC | 08:40 | |
| *** derekh has joined #openstack-infra | 08:42 | |
| *** xek has quit IRC | 08:46 | |
| *** ociuhandu has joined #openstack-infra | 08:53 | |
| *** psachin has quit IRC | 09:11 | |
| *** jtomasek has quit IRC | 09:35 | |
| *** jtomasek has joined #openstack-infra | 09:48 | |
| *** sshnaidm|afk is now known as sshnaidm | 09:52 | |
| *** ramishra has quit IRC | 10:05 | |
| *** nightmare_unreal has joined #openstack-infra | 10:22 | |
| *** Wryhder has joined #openstack-infra | 10:28 | |
| *** Lucas_Gray has quit IRC | 10:29 | |
| *** Wryhder is now known as Lucas_Gray | 10:29 | |
| *** jcapitao is now known as jcapitao_lunch | 10:33 | |
| *** ramishra has joined #openstack-infra | 10:45 | |
| *** Tengu has quit IRC | 11:05 | |
| *** Tengu has joined #openstack-infra | 11:07 | |
| *** ramishra_ has joined #openstack-infra | 11:10 | |
| *** ramishra has quit IRC | 11:12 | |
| *** Lucas_Gray has quit IRC | 11:13 | |
| *** dchen has quit IRC | 11:30 | |
| *** jpena is now known as jpena|lunch | 11:30 | |
| *** auristor has joined #openstack-infra | 11:35 | |
| *** artom has joined #openstack-infra | 11:39 | |
| *** smarcet has joined #openstack-infra | 11:39 | |
| *** xek has joined #openstack-infra | 11:42 | |
| *** xek has quit IRC | 11:55 | |
| *** ysandeep is now known as ysandeep|brb | 11:59 | |
| *** rfolco has joined #openstack-infra | 12:04 | |
| openstackgerrit | Merged openstack/project-config master: Add 'check arm64' trigger to check-arm64 pipeline https://review.opendev.org/751829 | 12:05 |
| *** jcapitao_lunch is now known as jcapitao | 12:09 | |
| *** rlandy has joined #openstack-infra | 12:09 | |
| *** vishalmanchanda has quit IRC | 12:11 | |
| *** ysandeep|brb is now known as ysandeep | 12:14 | |
| *** smarcet has quit IRC | 12:18 | |
| *** smarcet has joined #openstack-infra | 12:19 | |
| *** priteau has joined #openstack-infra | 12:22 | |
| *** rfolco is now known as rfolco|ruck | 12:28 | |
| *** jpena|lunch is now known as jpena | 12:32 | |
| *** dave-mccowan has joined #openstack-infra | 12:35 | |
| *** ykarel_ has joined #openstack-infra | 12:39 | |
| *** ykarel has quit IRC | 12:43 | |
| sean-k-mooney | o/ | 12:45 |
| sean-k-mooney | what does infra use to deploy and manage zuul/nodepool ectra these days | 12:46 |
| sean-k-mooney | im thinking of bringin back my personal third party ci and wondering what is a good way to deploy it | 12:46 |
| sean-k-mooney | i previously did it with custom k8s yaml files mainly to learn k8s but i assume infra uses ansible to deploy zuul et al | 12:47 |
| *** xek has joined #openstack-infra | 12:48 | |
| *** Goneri has joined #openstack-infra | 12:50 | |
| *** xek has quit IRC | 12:59 | |
| fungi | sean-k-mooney: we publish images under opendevorg on dockerhub and deploy/manage them with ansible playbooks in the opendev/system-config repo | 13:04 |
| fungi | our dockerfiles are in that repo too | 13:05 |
| sean-k-mooney | fungi: ah that repo is the one i was lookign for | 13:05 |
| sean-k-mooney | and ya i was using your docker image in by k8s files before | 13:05 |
| sean-k-mooney | i used the docker compose example as a base when i was trying to learn the basicis of k8s | 13:06 |
| sean-k-mooney | so do you use zuul to deploy zuul? | 13:07 |
| fungi | we're still using docker-compose, in part because we're driving deployment with zuul itself these days | 13:07 |
| sean-k-mooney | ya i was just reading the Continuous Deployment part | 13:08 |
| sean-k-mooney | in the readme | 13:08 |
| fungi | yep, a mix of merge-triggered deployment jobs and periodic ones | 13:08 |
| sean-k-mooney | that sound useful but im debating if i want to try and set it up the same way or just get it working quickly | 13:08 |
| sean-k-mooney | i know in the long run doing it that way would be better | 13:08 |
| sean-k-mooney | but its a time trade off now or when i eventully have to fix it | 13:09 |
| fungi | right now we funnel all the deployment through a bastion (it allows the system-config project key from zuul to run stuff) and then rely on nested ansible run on the bastion by those jobs so that we're not restricted to what plugins are available on the executors | 13:09 |
| sean-k-mooney | bridge.openstack.org is the bastion host | 13:11 |
| fungi | correct | 13:12 |
| fungi | like the bridge of a ship | 13:12 |
| fungi | the command center | 13:12 |
| sean-k-mooney | so you bootstrap that i guess by running the palybooks on your laptop then drive everything else form there | 13:13 |
| fungi | it's not something we rebuild with enough frequency to entirely automate, but that's more or less the process yeah | 13:14 |
| fungi | get enough of the server stood up to install ansible, clone system-config, apply playbooks locally | 13:14 |
| sean-k-mooney | ya ok im just trying to figure out what is the most effective way to redploy a third party ci and have it be maintainable with relitivly low interaction | 13:15 |
| fungi | we also don't really orchestrate creation of servers, we just fire semi-automated scripts to build the servers where we'll run our containers | 13:15 |
| sean-k-mooney | ya i can just spawn a few vms via horizon or the openstack cli | 13:15 |
| sean-k-mooney | that not really a big deal | 13:15 |
| fungi | the launch directory in system-config has the scripts we use for that, but sure horizon would do the trick too | 13:15 |
| *** lbragstad has joined #openstack-infra | 13:16 | |
| sean-k-mooney | are the role assuming k8s is deployed | 13:16 |
| sean-k-mooney | or is that just optional for other services | 13:16 |
| fungi | we've not been using kubernetes (yet anyway) | 13:17 |
| sean-k-mooney | cool | 13:17 |
| sean-k-mooney | oh the lauch script is just using the sdk that might work too but that the simplest bit anyway | 13:18 |
| fungi | we did briefly try setting up our gitea cluster in kubernetes on magnum at vexxhost, but ran into issues trying to handle shared filesystems (we were trying to do ceph/rook)... ultimately we worked out that gitea doesn't currently like sharing its backend anyway so we dropped back to something simpler for the time being | 13:18 |
| sean-k-mooney | ya the zuul executor and mergers also need indepent file systems too for each instance right | 13:19 |
| fungi | but you might find some remnants of that experiment in the system-config repo... if you run across mentions of kubernetes that's probably what it was from | 13:19 |
| sean-k-mooney | so that there git operations dont conflict | 13:19 |
| fungi | yes, well they need independent file trees at least. they could be parallel directories on the same fs technically | 13:19 |
| sean-k-mooney | oh ya that basically what i ment. | 13:20 |
| sean-k-mooney | off the top of your head to you know roughly the size of our git repos on disk? | 13:20 |
| sean-k-mooney | the way the tempest and devstack jobs are written you basically need all of openstack | 13:21 |
| sean-k-mooney | not quite but it was a lot of work for not much gain the last time to get the minimal subset | 13:21 |
| *** eharney has joined #openstack-infra | 13:34 | |
| frickler | sean-k-mooney: all of opendev.org seems to be 8.1G, that's what "du -h" on a testing node says | 13:35 |
| sean-k-mooney | ok cool i rememebred it being resonably small | 13:36 |
| sean-k-mooney | i tought i was close the 20G range but that should be fine | 13:36 |
| sean-k-mooney | the last time i deploy it i mounted a 200G cinder volume with i used for zuul and node pool storeage and it seamed fine | 13:37 |
| frickler | well, not sure I'd still count near 10G image size as reasonable | 13:37 |
| sean-k-mooney | you dont need to precache them in the image techincally the tempest jobs can use just the ubuntu cloud image | 13:38 |
| sean-k-mooney | since i was running everything form the same server i didnt mind just coping evertying from zull to the vms on the same cloud | 13:39 |
| sean-k-mooney | 10G is not horrible but also not trivally small | 13:39 |
| *** dwilde has joined #openstack-infra | 13:44 | |
| *** d34dh0r53 has quit IRC | 13:44 | |
| *** ramishra has joined #openstack-infra | 13:44 | |
| fungi | frickler: was that on review or ze/zm*? | 13:45 |
| *** ramishra_ has quit IRC | 13:48 | |
| fungi | but yeah, ze01 has been running for 44 days, and `du -sh /var/lib/zuul/executor-git/opendev.org/openstack/` there comes in at 7.9G | 13:49 |
| fungi | which is probably fairly close to steady-state | 13:49 |
| fungi | (though will continue to grow over time as openstack grows of course) | 13:50 |
| *** xek has joined #openstack-infra | 13:51 | |
| *** smarcet has quit IRC | 13:52 | |
| frickler | fungi: that was on a held node that I happened to be logged into by chance. /opt/git/opendev.org | 13:53 |
| fungi | oh, on the images, yeah | 13:54 |
| *** smarcet has joined #openstack-infra | 13:54 | |
| fungi | i thought sean-k-mooney was asking how much room you needed on an executor/merger | 13:54 |
| *** jeremyfreudberg has joined #openstack-infra | 13:55 | |
| *** eharney has quit IRC | 13:55 | |
| sean-k-mooney | fungi: well i was | 13:55 |
| *** jeremyfreudberg has left #openstack-infra | 13:55 | |
| sean-k-mooney | but they will be similar | 13:55 |
| fungi | certainly within the same order of magnitude anyway | 13:56 |
| sean-k-mooney | yep so basically multiple that by 3 1 for nodepool 1 for merger and 1 for executor | 13:56 |
| sean-k-mooney | i proably dont need the merger but i like having it | 13:56 |
| *** eharney has joined #openstack-infra | 13:58 | |
| *** ykarel_ is now known as ykarel|away | 13:58 | |
| *** zxiiro has joined #openstack-infra | 14:15 | |
| fungi | it's technically possible to use a shared copy of git repositories with copy-on-write, but tricky to set up and very hard to update the shared copy and reset the higher layers without taking services offline temporarily, i think? | 14:20 |
| *** hashar has quit IRC | 14:27 | |
| *** ykarel|away has quit IRC | 14:29 | |
| *** ajya|afk is now known as ajya | 14:29 | |
| *** hemna has quit IRC | 14:30 | |
| *** hemna has joined #openstack-infra | 14:30 | |
| *** hongbin has joined #openstack-infra | 14:31 | |
| *** eharney has quit IRC | 14:35 | |
| *** jtomasek has quit IRC | 14:45 | |
| *** eharney has joined #openstack-infra | 14:47 | |
| *** dklyle has joined #openstack-infra | 14:56 | |
| *** hashar has joined #openstack-infra | 14:57 | |
| *** xek has quit IRC | 15:01 | |
| *** armax has joined #openstack-infra | 15:10 | |
| *** ysandeep is now known as ysandeep|away | 15:10 | |
| *** priteau has quit IRC | 15:10 | |
| *** smarcet has quit IRC | 15:16 | |
| *** redrobot has joined #openstack-infra | 15:23 | |
| openstackgerrit | Carlos Goncalves proposed openstack/project-config master: Linaro US: Add a 16GB RAM label for CentOS 8 https://review.opendev.org/753036 | 15:30 |
| *** lmiccini has quit IRC | 15:31 | |
| *** smarcet has joined #openstack-infra | 15:32 | |
| *** hongbin has quit IRC | 15:38 | |
| *** hongbin has joined #openstack-infra | 15:44 | |
| *** ralonsoh has quit IRC | 15:57 | |
| *** lucasagomes has quit IRC | 16:03 | |
| *** vishalmanchanda has joined #openstack-infra | 16:10 | |
| *** hashar has quit IRC | 16:18 | |
| *** jcapitao has quit IRC | 16:20 | |
| *** ociuhandu has quit IRC | 16:25 | |
| *** dwilde has quit IRC | 16:30 | |
| *** d34dh0r53 has joined #openstack-infra | 16:30 | |
| *** Ajohn has joined #openstack-infra | 16:31 | |
| *** _erlon_ has joined #openstack-infra | 16:37 | |
| *** gyee has joined #openstack-infra | 16:40 | |
| *** jpena is now known as jpena|off | 16:56 | |
| *** andrewbonney has quit IRC | 16:59 | |
| *** derekh has quit IRC | 17:00 | |
| *** gfidente has quit IRC | 17:02 | |
| *** zbr is now known as zbr|pto | 17:07 | |
| *** smarcet has quit IRC | 17:13 | |
| *** dtantsur is now known as dtantsur|afk | 17:31 | |
| *** smarcet has joined #openstack-infra | 17:39 | |
| openstackgerrit | Clark Boylan proposed openstack/openstack-zuul-jobs master: Don't add zuul/zuul as required project on linting https://review.opendev.org/753098 | 17:43 |
| *** eolivare has quit IRC | 17:43 | |
| *** hamalq has joined #openstack-infra | 17:57 | |
| *** hongbin has quit IRC | 18:05 | |
| *** hongbin has joined #openstack-infra | 18:11 | |
| openstackgerrit | Merged openstack/openstack-zuul-jobs master: Don't add zuul/zuul as required project on linting https://review.opendev.org/753098 | 18:15 |
| *** jtomasek has joined #openstack-infra | 18:28 | |
| *** artom has quit IRC | 18:32 | |
| *** artom has joined #openstack-infra | 18:32 | |
| *** ociuhandu has joined #openstack-infra | 18:34 | |
| openstackgerrit | Merged openstack/project-config master: Remove old nodepool builder configs https://review.opendev.org/752741 | 18:37 |
| *** ociuhandu has quit IRC | 18:39 | |
| *** ociuhandu has joined #openstack-infra | 18:42 | |
| *** nightmare_unreal has quit IRC | 18:51 | |
| *** xek has joined #openstack-infra | 18:56 | |
| *** hashar has joined #openstack-infra | 19:02 | |
| *** hongbin has quit IRC | 19:05 | |
| *** vishalmanchanda has quit IRC | 19:11 | |
| *** jtomasek has quit IRC | 19:25 | |
| *** ociuhandu has quit IRC | 19:32 | |
| *** Ajohn has quit IRC | 19:43 | |
| *** Ajohn has joined #openstack-infra | 19:52 | |
| *** hashar has quit IRC | 19:55 | |
| *** slaweq has quit IRC | 20:07 | |
| *** smarcet has quit IRC | 20:12 | |
| *** slaweq has joined #openstack-infra | 20:12 | |
| *** smarcet has joined #openstack-infra | 20:13 | |
| *** hongbin has joined #openstack-infra | 20:16 | |
| *** ociuhandu has joined #openstack-infra | 20:25 | |
| *** Ajohn has quit IRC | 20:35 | |
| *** Ajohn has joined #openstack-infra | 20:43 | |
| *** xek has quit IRC | 20:54 | |
| *** ociuhandu has quit IRC | 20:59 | |
| *** matt_kosut has quit IRC | 21:00 | |
| *** matt_kosut has joined #openstack-infra | 21:01 | |
| *** _erlon_ has quit IRC | 21:04 | |
| *** matt_kosut has quit IRC | 21:05 | |
| *** slaweq has quit IRC | 21:06 | |
| *** tbachman has quit IRC | 21:17 | |
| *** rfolco|ruck has quit IRC | 21:22 | |
| *** kaiokmo has joined #openstack-infra | 21:36 | |
| TheJulia | w/r/t VMs for devstack, specifically grenade. Did something change with /opt, do we now have less? | 21:36 |
| TheJulia | https://f3b9509bf703d6407c73-eb4966d6f73a1195abf0bef83b33defe.ssl.cf1.rackcdn.com/753094/1/check/ironic-grenade/e6884e9/job-output.txt | 21:37 |
| fungi | TheJulia: on focal we stopped being able to fallocate a swap file, so if the test node doesn't have an ephemeral disk for us to put a swap partition on we dd /dev/zero out to a swapfile on the rootfs now | 21:38 |
| fungi | that could have an impact on available /opt space (if /opt is part of the rootfs, which on those sorts of nodes it will be) | 21:39 |
| clarkb | the linux kernel broke the rule of having users notice changes | 21:40 |
| TheJulia | Did nodes always only have 1gb of swap, curious because we've seen lots of failures on ironic suddenly that are very perplexing because we've randomly started OOMing it seems | 21:40 |
| fungi | it used to be 8 | 21:40 |
| TheJulia | oh wow | 21:40 |
| clarkb | though that was fallocated so "8" | 21:40 |
| clarkb | we told it it had 8 then hoped that if it tried to use 8 there would actually be 8 left for it :) | 21:41 |
| fungi | we compromised by reducing the swap size default so as not to rob as much space on the fs | 21:41 |
| fungi | there is a simple job var you can add to increase it to whatever size you like though | 21:41 |
| TheJulia | okay, this explains a lot | 21:41 |
| TheJulia | fungi: oh? | 21:41 |
| * fungi checks the discussion from last week... just a sec | 21:41 | |
| fungi | configure_swap_size | 21:43 |
| fungi | you can poke that into codesearch for examples, dulek just did it to the ovn-octavia jobs last week | 21:43 |
| clarkb | I'll also suggest that if you need more than 1GB of swap thinking about multinode jobs or reducing memory consumption are worthwhile | 21:43 |
| clarkb | one of the issues there in particular is we seem to become our own noisy neighbor for disk iops | 21:44 |
| clarkb | and swap has a big impact on that | 21:44 |
| fungi | ahh, it was kuryr | 21:44 |
| fungi | TheJulia: https://opendev.org/openstack/kuryr-kubernetes/src/branch/master/.zuul.d/base.yaml#L35 | 21:44 |
| fungi | and yeah, i agree, a little swap is healthy to free up ram for buffer/cache use and increase performance when there's stuff which isn't heavily accessed and can be paged out for little cost | 21:45 |
| fungi | as soon as you start paging out active memory, you're doomed | 21:45 |
| fungi | DOOMED | 21:45 |
| TheJulia | yeah, we were getting awfully close | 21:47 |
| TheJulia | like 26MB in with 1gb of actual ram free and then a vm started | 21:49 |
| * fungi just likes saying DOOMED | 21:50 | |
| * TheJulia sighs | 21:51 | |
| fungi | sorry, having an invader zim moment | 21:51 |
| TheJulia | dtantsur|afk: fyi^^^ | 21:51 |
| *** dave-mccowan has quit IRC | 21:58 | |
| *** dave-mccowan has joined #openstack-infra | 22:01 | |
| *** smarcet has quit IRC | 22:07 | |
| *** tbachman has joined #openstack-infra | 22:08 | |
| *** tbachman has quit IRC | 22:13 | |
| TheJulia | so just for my sanity/knowledge since space free on /opt now seems to be further constrained since we seeminly failed on our fifth VM's disk creation, how much space should we expect to have at the worst in /opt? | 22:25 |
| clarkb | I think its ~60GB | 22:26 |
| *** tbachman has joined #openstack-infra | 22:30 | |
| fungi | though if you don't need swap you could set configure_swap_size to 0 or at least a very tiny amount | 22:42 |
| fungi | or it may be possible to simply disable the configure-swap role | 22:42 |
| fungi | gmann: ^ this is also notable fallout from the focal migration work, in case it comes up elsewhere | 22:44 |
| *** tosky has quit IRC | 22:49 | |
| *** kaiokmo has quit IRC | 22:59 | |
| TheJulia | Additional question: does this impact ussuri test nodes by default? | 23:00 |
| *** tbachman has quit IRC | 23:00 | |
| *** rcernin has joined #openstack-infra | 23:01 | |
| clarkb | TheJulia: the swap change was made globally I think | 23:02 |
| clarkb | because its a linux kernel thing and we didnt want to deal with figuring if its safe in vertain scenarios or not | 23:02 |
| TheJulia | ok | 23:03 |
| * TheJulia goes and gets a beverage | 23:03 | |
| *** zzzeek has quit IRC | 23:10 | |
| *** rcernin has quit IRC | 23:11 | |
| *** rcernin has joined #openstack-infra | 23:11 | |
| *** zzzeek has joined #openstack-infra | 23:13 | |
| *** eharney has quit IRC | 23:15 | |
| *** zzzeek has quit IRC | 23:17 | |
| *** zzzeek has joined #openstack-infra | 23:20 | |
| fungi | TheJulia: clarkb: it affects any jobs which use the configure-swap role, regardless of branch or node type: https://review.opendev.org/750941 | 23:23 |
| fungi | but that's not all jobs, of course | 23:23 |
| fungi | mainly devstack-based jobs i think? | 23:24 |
| TheJulia | yeah | 23:25 |
| fungi | aha, i misremembered, apparently it was fedora which tripped over it, not ubuntu focal | 23:25 |
| fungi | merged ~11 days ago | 23:25 |
| *** dchen has joined #openstack-infra | 23:34 | |
| *** eharney has joined #openstack-infra | 23:37 | |
| *** eharney has quit IRC | 23:50 | |
| *** eharney has joined #openstack-infra | 23:51 | |
| *** rlandy has quit IRC | 23:52 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!