| ianw | clarkb: oh, how confusing. it seems that xenial hosts have docker setup with journalctl logging, while bionic hosts have json-file | 00:23 |
|---|---|---|
| clarkb | that is unexpected | 00:23 |
| ianw | that's why eavesdrop is spamming itself to syslog, and the other hosts are not recording their output | 00:24 |
| *** openstackgerrit has joined #opendev | 00:32 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: docker: install rsyslog to capture container output https://review.opendev.org/756605 | 00:32 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Add initial borg backup server https://review.opendev.org/756607 | 00:55 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: docker: install rsyslog to capture container output https://review.opendev.org/756605 | 01:08 |
| *** tkajinam has quit IRC | 01:52 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: docker: install rsyslog to capture container output https://review.opendev.org/756605 | 01:53 |
| *** tkajinam has joined #opendev | 01:53 | |
| *** hamalq has quit IRC | 02:08 | |
| *** hamalq has joined #opendev | 02:09 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Add containerfile element https://review.opendev.org/700083 | 02:26 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: WIP: boot test of containerfile image https://review.opendev.org/722148 | 02:31 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: WIP: boot test of containerfile image https://review.opendev.org/722148 | 02:32 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: docker: install rsyslog to capture container output https://review.opendev.org/756605 | 02:36 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Add containerfile element https://review.opendev.org/700083 | 02:38 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: WIP: boot test of containerfile image https://review.opendev.org/722148 | 02:38 |
| *** hamalq has quit IRC | 03:01 | |
| *** hamalq has joined #opendev | 03:01 | |
| *** hamalq has quit IRC | 03:04 | |
| *** hamalq has joined #opendev | 03:05 | |
| *** ysandeep|away is now known as ysandeep | 03:16 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: docker: install rsyslog to capture container output https://review.opendev.org/756605 | 03:33 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Remove Ubuntu Xenial ARM64 base testing https://review.opendev.org/756627 | 03:33 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Remove rsyslogd xconsole workaround https://review.opendev.org/756628 | 03:33 |
| ianw | This configuration file is valid at least on Ubuntu Oneiric, | 03:34 |
| ianw | Precise and Quantal, which covers the releases we currently use. | 03:34 |
| ianw | it survived until i started fiddling on focal nodes, so pretty good | 03:34 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: WIP: boot test of containerfile image https://review.opendev.org/722148 | 03:43 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: ARM64 : run base test on Focal https://review.opendev.org/756629 | 03:59 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: ARM64 : run base test on Focal https://review.opendev.org/756629 | 04:09 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Remove rsyslogd xconsole workaround https://review.opendev.org/756628 | 04:09 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: docker: install rsyslog to capture container output https://review.opendev.org/756605 | 04:09 |
| *** hamalq has quit IRC | 04:23 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: WIP: boot test of containerfile image https://review.opendev.org/722148 | 04:49 |
| *** ykarel has joined #opendev | 04:55 | |
| *** marios has joined #opendev | 05:56 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Remove rsyslogd xconsole workaround https://review.opendev.org/756628 | 06:20 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: docker: install rsyslog to capture container output https://review.opendev.org/756605 | 06:20 |
| *** Eighth_Doctor has quit IRC | 06:35 | |
| *** mordred has quit IRC | 06:35 | |
| *** ralonsoh has joined #opendev | 06:37 | |
| *** Eighth_Doctor has joined #opendev | 06:44 | |
| *** eolivare has joined #opendev | 06:52 | |
| *** hamalq has joined #opendev | 06:56 | |
| *** hamalq_ has joined #opendev | 06:58 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: WIP: boot test of containerfile image https://review.opendev.org/722148 | 07:00 |
| *** hamalq has quit IRC | 07:01 | |
| *** slaweq has joined #opendev | 07:01 | |
| *** hashar has joined #opendev | 07:02 | |
| *** fressi has joined #opendev | 07:04 | |
| *** mordred has joined #opendev | 07:09 | |
| *** fressi has quit IRC | 07:17 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: docker: install rsyslog to capture container output https://review.opendev.org/756605 | 07:21 |
| *** fressi has joined #opendev | 07:23 | |
| *** rpittau|afk is now known as rpittau | 07:27 | |
| *** andrewbonney has joined #opendev | 07:27 | |
| *** hamalq_ has quit IRC | 07:33 | |
| *** auristor has quit IRC | 07:33 | |
| *** ttx has quit IRC | 07:33 | |
| *** hamalq has joined #opendev | 07:34 | |
| *** auristor has joined #opendev | 07:38 | |
| openstackgerrit | Javier Peña proposed openstack/diskimage-builder master: Ensure yum-utils is installed in epel element https://review.opendev.org/756010 | 07:50 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: docker: install rsyslog to capture container output https://review.opendev.org/756605 | 07:51 |
| *** tosky has joined #opendev | 07:54 | |
| *** moppy has quit IRC | 08:01 | |
| *** moppy has joined #opendev | 08:02 | |
| *** ttx has joined #opendev | 08:02 | |
| *** DSpider has joined #opendev | 08:05 | |
| *** roman_g has joined #opendev | 08:19 | |
| *** hamalq has quit IRC | 08:26 | |
| *** hamalq has joined #opendev | 08:34 | |
| *** eolivare has quit IRC | 08:39 | |
| *** eolivare has joined #opendev | 08:39 | |
| *** fressi has quit IRC | 08:54 | |
| *** ysandeep is now known as ysandeep|lunch | 08:55 | |
| *** fressi has joined #opendev | 09:01 | |
| *** johnsom has quit IRC | 09:25 | |
| *** johnsom has joined #opendev | 09:25 | |
| *** ysandeep|lunch is now known as ysandeep | 09:27 | |
| *** dulek has quit IRC | 09:58 | |
| *** dulek has joined #opendev | 09:58 | |
| *** hamalq has quit IRC | 10:03 | |
| *** hashar has quit IRC | 10:21 | |
| openstackgerrit | Carlos Goncalves proposed openstack/project-config master: Add nested-virt-centos-8 label https://review.opendev.org/756674 | 10:28 |
| openstackgerrit | Carlos Goncalves proposed openstack/project-config master: Add nested-virt-centos-8-stream label https://review.opendev.org/756674 | 10:32 |
| *** paramite has joined #opendev | 11:06 | |
| *** lpetrut has joined #opendev | 11:25 | |
| *** hashar has joined #opendev | 12:49 | |
| openstackgerrit | Nicolas Alvarez proposed openstack/project-config master: Add initial files to project-config repo. https://review.opendev.org/756717 | 14:02 |
| openstackgerrit | Nicolas Alvarez proposed openstack/project-config master: Add initial files to project-config repo. https://review.opendev.org/756717 | 14:04 |
| *** lpetrut has quit IRC | 14:21 | |
| *** hashar has quit IRC | 14:27 | |
| openstackgerrit | Javier Peña proposed openstack/project-config master: Create acls for rpm-packaging-release group https://review.opendev.org/756722 | 14:29 |
| *** slaweq has quit IRC | 14:38 | |
| *** slaweq has joined #opendev | 14:42 | |
| openstackgerrit | Nicolas Alvarez proposed openstack/project-config master: Add SNMP Armada app to StarlingX. https://review.opendev.org/756717 | 14:50 |
| openstackgerrit | Jeremy Stanley proposed opendev/storyboard master: Be explicit about base container image https://review.opendev.org/715229 | 14:52 |
| *** ykarel is now known as ykarel|away | 14:57 | |
| *** priteau has joined #opendev | 14:58 | |
| *** mlavalle has joined #opendev | 15:03 | |
| *** ysandeep is now known as ysandeep|away | 15:04 | |
| openstackgerrit | Merged openstack/project-config master: Create acls for rpm-packaging-release group https://review.opendev.org/756722 | 15:12 |
| *** ykarel|away has quit IRC | 15:24 | |
| *** hashar has joined #opendev | 15:43 | |
| *** marios is now known as marios|out | 15:55 | |
| *** rpittau is now known as rpittau|afk | 16:01 | |
| *** marios|out has quit IRC | 16:06 | |
| *** yoctozepto has quit IRC | 16:16 | |
| *** yoctozepto has joined #opendev | 16:17 | |
| *** eolivare has quit IRC | 16:17 | |
| *** eolivare has joined #opendev | 16:17 | |
| *** fressi has quit IRC | 16:20 | |
| *** eolivare_ has joined #opendev | 16:21 | |
| *** eolivare has quit IRC | 16:23 | |
| clarkb | for half a second I was thinking we could make our own dib chroot bases using builders of various distros in zuul jobs. Then I realized thats essentially what the container bootstrapping idea is and now I like the idea even more :) | 16:56 |
| *** mlavalle has quit IRC | 17:08 | |
| *** mlavalle has joined #opendev | 17:09 | |
| *** auristor has quit IRC | 17:17 | |
| *** auristor has joined #opendev | 17:23 | |
| *** Topner has joined #opendev | 18:03 | |
| *** Topner has quit IRC | 18:07 | |
| *** roman_g has quit IRC | 18:10 | |
| *** qchris has quit IRC | 18:21 | |
| *** qchris has joined #opendev | 18:55 | |
| *** andrewbonney has quit IRC | 19:00 | |
| *** roman_g has joined #opendev | 19:20 | |
| *** priteau has quit IRC | 19:31 | |
| *** ralonsoh has quit IRC | 19:50 | |
| *** iurygregory has quit IRC | 20:00 | |
| *** iurygregory has joined #opendev | 20:02 | |
| *** slaweq has quit IRC | 20:26 | |
| *** roman_g has quit IRC | 20:53 | |
| *** roman_g has joined #opendev | 20:53 | |
| *** roman_g has quit IRC | 20:54 | |
| *** roman_g has joined #opendev | 20:54 | |
| *** roman_g has quit IRC | 20:54 | |
| *** roman_g has joined #opendev | 20:55 | |
| *** roman_g has joined #opendev | 20:56 | |
| *** roman_g has quit IRC | 20:56 | |
| ianw | yeah i'll keep poking, i feel like there was talk the base element was working but not the bootloader bits, maybe? i'm sure we can get something working | 20:57 |
| *** roman_g has joined #opendev | 20:57 | |
| *** roman_g has joined #opendev | 20:57 | |
| *** roman_g has quit IRC | 20:58 | |
| *** roman_g has joined #opendev | 20:58 | |
| clarkb | I don't recall | 20:58 |
| *** roman_g has quit IRC | 20:58 | |
| clarkb | the notedb migration is not fast if anyone is wondering | 20:59 |
| clarkb | its been running for 4.5 hours now | 20:59 |
| *** roman_g has joined #opendev | 20:59 | |
| clarkb | it complained about one change so far because it doesn't have any patchsets and it bascially says meh in that case | 20:59 |
| *** roman_g has quit IRC | 20:59 | |
| clarkb | (I think we can live with that | 20:59 |
| *** roman_g has joined #opendev | 21:00 | |
| *** roman_g has quit IRC | 21:00 | |
| fungi | that change is similarly inaccessible in production anyway | 21:00 |
| *** roman_g has joined #opendev | 21:01 | |
| *** roman_g has quit IRC | 21:01 | |
| clarkb | The rough upgrade plan I'm starting to formulate in my head is long weekend outage with upgrade and checkpoint at 2.16 pre notedb so that we can fall back there if necessary, but try and get to 3.2 | 21:01 |
| *** roman_g has joined #opendev | 21:01 | |
| *** roman_g has quit IRC | 21:02 | |
| fungi | or 3.3 if it's ga by then | 21:02 |
| clarkb | on day one we can do that 2.13 backups, and similar prep work, then upgrade to 2.16 pretty quickly giving time to turn on 2.16 but maybe not expose it publicly and if working do another set of backups there. Then kick off notedb migration and go enjoy the evening/sleep | 21:02 |
| clarkb | then hopeflly on day 2 we can do the 2.16 -> 3.2 upgrade process | 21:03 |
| clarkb | (granted still have more to learn about the upgrade at this point but that is roughly what I'm thinking right now) | 21:03 |
| fungi | and then roll back to 2.16 if absolutely necessary | 21:11 |
| clarkb | ya | 21:11 |
| ianw | ++ it will be a big relief to have it upgraded | 21:13 |
| ianw | fungi/clarkb: if you have sec to check the borg backup server addition that i didn't miss anything would be great -> https://review.opendev.org/756607 | 21:15 |
| ianw | if no objections i might try borg backups on review-test? | 21:15 |
| clarkb | ianw: I think using review-test would be fine but I've put it in emergency to avoid having its docker-compose file updated or gerrit config files updated | 21:16 |
| clarkb | so you may need to manually trigger things also might be good to wait for when we're done thrashing the disk :) | 21:16 |
| clarkb | I can give you an all clear for that assuming this finishes before I go to bed :) | 21:17 |
| ianw | ok, no rush :) | 21:18 |
| ianw | clarkb: i ended up with a small stack from https://review.opendev.org/#/c/756605/ to implement collecting docker container logs | 21:19 |
| ianw | i think it's quite compelling for gitea, where currently we're using json logging and we destroy the container logs when they're refreshed | 21:19 |
| ianw | since we're trying to capture backtraces etc. | 21:20 |
| ianw | (that's -1 because it looks for the gitea ssh log, which isn't there .. i think it only gets created when a line is logged so we might need to exercise port 222 in the test) | 21:20 |
| clarkb | reading the commit message that sounds great | 21:20 |
| clarkb | I'll try to make time to review it in a bit | 21:20 |
| ianw | haha no, it will probably work now that i've spelt "/var/log/containers" correctly | 21:22 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: docker: install rsyslog to capture container output https://review.opendev.org/756605 | 21:23 |
| fungi | sometimes that does matter | 21:23 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: docker: install rsyslog to capture container output https://review.opendev.org/756605 | 21:25 |
| *** eolivare_ has quit IRC | 21:29 | |
| ianw | it would be pretty easy to send the container logs somewhere central too | 21:32 |
| ianw | splunk was a big deal when i was at vmware, basically it's logstash | 21:33 |
| *** hamalq has joined #opendev | 21:58 | |
| clarkb | ianw: in your logging change instead of needing to edit docker-compose for every container can we set up our docker json config file to do that for all containers by default? or does the docker- prefix addition require we do it more explicitly? | 22:10 |
| *** hashar has quit IRC | 22:11 | |
| ianw | clarkb: yeah, i think to have the consistent docker-<sensible-container-name> we need to do it for each container individually | 22:13 |
| clarkb | and I guess without a consistent prefix we can't have the auto dumping to specific files | 22:13 |
| clarkb | hrm | 22:13 |
| clarkb | I guess I can live with the extra config then | 22:14 |
| *** qchris has quit IRC | 22:42 | |
| clarkb | based on how long the reindex at the end of the note db migration is taking compared to the one we did as part of the 2.16 upgrade I think the aggressive git gc really makes a big difference | 22:43 |
| clarkb | unfortunately we can't do a gc here because its all baked into the one command | 22:43 |
| clarkb | we are at 63% and it started about an hour and 20 minutes ago. The reindex we did as part of the 2.16 upgrade took 35 minutes | 22:44 |
| clarkb | it has also complained about a couple changes but look at them in prod I think the problems were preexisting | 22:44 |
| clarkb | https://review.opendev.org/#/c/463526/ is one example and if you load them up and try to view a list of patchsets you never get one | 22:44 |
| clarkb | the reindexing seems to fail on patchsets that it can't find | 22:45 |
| clarkb | (the change is abandoned too which makes me wonder if something weird happened there) | 22:45 |
| clarkb | when this is done my next step is going to be another round of gc'ing so that the 3.0 3.1 and 3.2 upgrades can go more smoothly | 22:46 |
| ianw | what is it with the specs repo ... i remember a bunch of issues there at some point | 22:46 |
| ianw | 2018-03-07 nova-specs recovery | 22:47 |
| ianw | http://lists.openstack.org/pipermail/openstack-dev/2017-June/119166.html | 22:47 |
| clarkb | ya it could be related to that I suppose | 22:47 |
| clarkb | iirc they corrupted the repo somehow and github complained but gerrit didn't | 22:47 |
| clarkb | maybe newer gerrit now is mad about it | 22:47 |
| clarkb | also as soon as i start talking about this stuff it complains about https://review.opendev.org/#/c/628296/ | 22:47 |
| clarkb | only a mergeability check on ^ though | 22:48 |
| ianw | that's a huge bunch of renames | 22:48 |
| clarkb | ya I bet it hit a timeout and isn't a real error | 22:49 |
| ianw | yeah, i'd consider nova-specs a very suspect repo | 22:49 |
| clarkb | because we're running with like 16 threads and the disk is very busy right now reindexing a timeout or 3 isn't unexpected | 22:49 |
| clarkb | we can always trigger an online reindex later and have it update things more slowly once we're steady state post upgrade | 22:50 |
| clarkb | I plan to be around this evenign due to the rax db outages | 22:51 |
| clarkb | so I'll also try to get the 3.2 upgrade path on -test as far as I can | 22:51 |
| clarkb | then hopefully tomorrow can be spent double checking repos like nova-specs and openstack/openstack as well as beginning the testing of replication to gitea (I plan to hold a system-config-run-gitea job node and replicate to it) and other things like cla signing and general git review comaptibility and change pushing | 22:52 |
| clarkb | so many things | 22:52 |
| *** qchris has joined #opendev | 22:53 | |
| clarkb | I also need to go through my notes and start drafting WIP changes for the things we need to update in gerrit config (disable change.move for example), the apache config, and I forget what else | 22:53 |
| *** mlavalle has quit IRC | 22:54 | |
| clarkb | latest thing it is mad about is https://review.opendev.org/#/c/11544/ which is a zuul change that also errors trying to load it in prod | 22:58 |
| clarkb | seems like so far every issue (of which we've had maybe 6 out of 550k) has a semi reasonable explanation even if it isn't ideal | 22:58 |
| *** tosky has quit IRC | 22:59 | |
| clarkb | https://review.opendev.org/#/c/22128/ is another with the patchset discovery problem | 23:01 |
| clarkb | ps2 seems to not exist for ^ | 23:02 |
| clarkb | there are fewer than 600 changes to reindex now I assume all for nova | 23:51 |
| *** DSpider has quit IRC | 23:55 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!