| openstackgerrit | Steve Baker proposed openstack/diskimage-builder master: Add element block-device-efi-lvm https://review.opendev.org/c/openstack/diskimage-builder/+/790192 | 00:03 |
|---|---|---|
| mnaser | fungi: sorry, never got around responding thinking its an OSH issue, but it seems to not be the case. i'm seeing the same builds randomly failing, and it's failing during a `pip wheel uwsgi`. i'm not able to replicate it locally, but also noticed it is failing in a non-deterministic way :( | 00:30 |
| mnaser | only thing that i've managed to see repeat is that it almost always fails (when it does) in rax-iad -- but that doesn't mean much. | 00:30 |
| mnaser | i cant accurately say it is an issue there | 00:31 |
| *** _mlavalle_1 has quit IRC | 00:43 | |
| *** _mlavalle_2 has joined #opendev | 00:43 | |
| *** hemanth_n has joined #opendev | 00:55 | |
| fungi | mnaser: got it. and not using a prebuilt wheel i suppose? | 01:00 |
| mnaser | fungi: we use both pypi and add the infra wheels as extra indexes when we do this | 01:01 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: certcheck: cleanup letsencrypt domains https://review.opendev.org/c/opendev/system-config/+/780140 | 01:02 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: system-config-roles: only match jobs on roles tested https://review.opendev.org/c/opendev/system-config/+/778593 | 01:06 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: gerrit docker: match some more files https://review.opendev.org/c/opendev/system-config/+/778613 | 01:06 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Remove obsolete Bazel spawn strategies https://review.opendev.org/c/opendev/system-config/+/778404 | 01:06 |
| fungi | mnaser: and i guess we don't prebuild a wheel for that for the platform where the job runs? | 01:08 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Remove some unnecessary debug: statements https://review.opendev.org/c/opendev/system-config/+/717874 | 01:12 |
| ianw | anyone have an issue with making the host files names consistently have .yaml on the end with -> https://review.opendev.org/c/opendev/system-config/+/786011 ? | 01:17 |
| fungi | seems like a good idea to me | 01:24 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: borg-backup hosts: use exact names https://review.opendev.org/c/opendev/system-config/+/780144 | 01:25 |
| openstackgerrit | Merged opendev/grafyaml master: Add import of json files https://review.opendev.org/c/opendev/grafyaml/+/737900 | 01:31 |
| openstackgerrit | Merged opendev/gerritlib master: docker-compose: update header syntax https://review.opendev.org/c/opendev/gerritlib/+/765898 | 01:40 |
| openstackgerrit | Merged opendev/gerritlib master: Add ignore events filter https://review.opendev.org/c/opendev/gerritlib/+/785262 | 01:40 |
| openstackgerrit | Merged opendev/gerritlib master: Update documentation https://review.opendev.org/c/opendev/gerritlib/+/769241 | 01:40 |
| openstackgerrit | Merged opendev/system-config master: certcheck: cleanup letsencrypt domains https://review.opendev.org/c/opendev/system-config/+/780140 | 01:47 |
| openstackgerrit | Merged opendev/gear master: Create SSL context using PROTOCOL_TLS, fallback to highest supported version https://review.opendev.org/c/opendev/gear/+/741288 | 02:14 |
| openstackgerrit | Merged opendev/gear master: Update testing to Python 3.9 and linters https://review.opendev.org/c/opendev/gear/+/780103 | 02:15 |
| *** pongboom has joined #opendev | 02:20 | |
| openstackgerrit | Merged opendev/system-config master: host_vars : add .yaml extension https://review.opendev.org/c/opendev/system-config/+/786011 | 02:42 |
| openstackgerrit | Merged opendev/system-config master: Remove some unnecessary debug: statements https://review.opendev.org/c/opendev/system-config/+/717874 | 03:03 |
| openstackgerrit | Merged opendev/system-config master: system-config-roles: only match jobs on roles tested https://review.opendev.org/c/opendev/system-config/+/778593 | 03:03 |
| openstackgerrit | Merged opendev/system-config master: gerrit docker: match some more files https://review.opendev.org/c/opendev/system-config/+/778613 | 03:04 |
| openstackgerrit | Merged opendev/system-config master: Remove obsolete Bazel spawn strategies https://review.opendev.org/c/opendev/system-config/+/778404 | 03:34 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Add containerfile element https://review.opendev.org/c/openstack/diskimage-builder/+/700083 | 03:59 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: WIP: boot test of containerfile image https://review.opendev.org/c/openstack/diskimage-builder/+/722148 | 03:59 |
| *** ykarel|away has joined #opendev | 04:25 | |
| *** marios has joined #opendev | 05:06 | |
| openstackgerrit | Merged openstack/diskimage-builder master: setup.cfg: Replace dashes with underscores https://review.opendev.org/c/openstack/diskimage-builder/+/789688 | 05:21 |
| frickler | mnaser: do you have some logs of a failing job? can this be related to the pypi sni changes? (i.e. running with old pip?) | 05:45 |
| frickler | mnaser: also, while you're here, any update on the v6 situation? | 05:46 |
| *** ralonsoh has joined #opendev | 05:46 | |
| *** ykarel|away has quit IRC | 05:47 | |
| *** ykarel|away has joined #opendev | 05:47 | |
| *** ysandeep|away is now known as ysandeep | 06:02 | |
| *** slaweq has joined #opendev | 06:05 | |
| *** cloudnull has quit IRC | 06:23 | |
| *** jhesketh has quit IRC | 06:28 | |
| *** vishalmanchanda has joined #opendev | 06:34 | |
| *** cloudnull has joined #opendev | 06:41 | |
| *** fressi has joined #opendev | 06:51 | |
| *** fressi has quit IRC | 06:57 | |
| *** fressi has joined #opendev | 06:58 | |
| *** rpittau|afk is now known as rpittau | 07:05 | |
| *** amoralej|off is now known as amoralej | 07:08 | |
| *** andrewbonney has joined #opendev | 07:11 | |
| *** fressi has quit IRC | 07:13 | |
| *** hashar has joined #opendev | 07:15 | |
| *** fressi has joined #opendev | 07:35 | |
| *** tosky has joined #opendev | 07:48 | |
| *** ykarel|away is now known as ykarel | 07:48 | |
| *** jpena|off is now known as jpena | 07:51 | |
| *** dtantsur|afk is now known as dtantsur | 07:58 | |
| *** mgoddard has joined #opendev | 08:04 | |
| *** ysandeep is now known as ysandeep|lunch | 08:17 | |
| *** xinliang has joined #opendev | 08:32 | |
| *** marios has quit IRC | 08:50 | |
| *** jhesketh has joined #opendev | 09:01 | |
| *** lpetrut has joined #opendev | 09:41 | |
| *** marios has joined #opendev | 09:48 | |
| *** ysandeep|lunch is now known as ysandeep | 10:06 | |
| *** brinzhang_ has quit IRC | 10:11 | |
| *** hashar has quit IRC | 10:30 | |
| openstackgerrit | Benjamin Schanzel proposed zuul/zuul-jobs master: Allow Specifying Log File Retention in s3 Uploader https://review.opendev.org/c/zuul/zuul-jobs/+/790050 | 10:36 |
| *** xinliang has quit IRC | 10:54 | |
| *** jpena is now known as jpena|lunch | 11:30 | |
| openstackgerrit | Benjamin Schanzel proposed zuul/zuul-jobs master: Allow Specifying Log File Retention in s3 Uploader https://review.opendev.org/c/zuul/zuul-jobs/+/790050 | 11:46 |
| *** cenne is now known as cenn | 11:59 | |
| *** hemanth_n has quit IRC | 12:05 | |
| *** ysandeep is now known as ysandeep|brb | 12:05 | |
| *** brinzhang has joined #opendev | 12:14 | |
| *** fressi has quit IRC | 12:16 | |
| *** jpena|lunch is now known as jpena | 12:27 | |
| *** ysandeep|brb is now known as ysandeep | 12:29 | |
| *** pongboom has quit IRC | 12:51 | |
| *** amoralej is now known as amoralej|lunch | 12:54 | |
| *** amoralej|lunch is now known as amoralej | 13:18 | |
| *** elod is now known as elod_afk | 13:23 | |
| openstackgerrit | Artem Goncharov proposed openstack/project-config master: Modify OpenStackSDK acl https://review.opendev.org/c/openstack/project-config/+/790244 | 13:39 |
| *** whoami-rajat has joined #opendev | 13:39 | |
| *** ykarel has quit IRC | 13:44 | |
| *** mkowalski has joined #opendev | 13:44 | |
| *** ykarel has joined #opendev | 13:46 | |
| *** d34dh0r53 has quit IRC | 14:12 | |
| *** d34dh0r53 has joined #opendev | 14:18 | |
| *** avass has quit IRC | 14:20 | |
| *** avass has joined #opendev | 14:21 | |
| openstackgerrit | Carmen Rata proposed openstack/project-config master: Add audit armada app project to StarlingX https://review.opendev.org/c/openstack/project-config/+/790250 | 14:25 |
| *** rpittau is now known as rpittau|afk | 14:29 | |
| openstackgerrit | Carmen Rata proposed openstack/project-config master: Add audit armada app project to StarlingX https://review.opendev.org/c/openstack/project-config/+/790250 | 14:43 |
| *** mlavalle has joined #opendev | 14:58 | |
| *** _mlavalle_2 has quit IRC | 15:02 | |
| clarkb | corvus: if you get a chance can you check my comment on https://review.opendev.org/c/opendev/system-config/+/789622/18/playbooks/roles/mailman/tasks/main.yaml I think that may be the last thing I haven't really been able to udnerstand fully | 15:32 |
| clarkb | I suspect that my naive just run newlist implementation is fine and it was puppet that was doing extra stuff that I just don't have, but not sure | 15:33 |
| *** hashar has joined #opendev | 15:39 | |
| fungi | clarkb: mass opencloud has this which might be interesting to dissect: https://github.com/CCI-MOC/moc-ansible-mailman | 15:41 |
| *** amoralej is now known as amoralej|off | 15:46 | |
| *** lpetrut has quit IRC | 15:46 | |
| clarkb | fungi: that looks a bit more complete. I think we can probably convert the mailman-list role I wrote to using that rather than execing list_lists and newlist | 15:48 |
| clarkb | but also its all collections and stuff and I have no idea what that means for us | 15:49 |
| corvus | clarkb: i don't think you're missing anything; the original mailman puppet module assumed it should create /etc/alias entries, but we don't do that because we have exim determining what addresses to use, and doing verp. | 15:50 |
| clarkb | got it | 15:50 |
| fungi | clarkb: if we want to use that collection, i would consider that a potential future improvement, but what you have now seems fine. maybe add a todo comment so we don't forget it exists? | 15:52 |
| clarkb | good idea. I can also clean up the todo that corvus indicates is not a todo | 15:53 |
| *** marios is now known as marios|out | 15:54 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Ansible mailman configs https://review.opendev.org/c/opendev/system-config/+/789622 | 15:56 |
| clarkb | there we go ^ no code changes, just comment updates | 15:56 |
| *** ysandeep is now known as ysandeep|away | 16:01 | |
| *** marios|out has quit IRC | 16:04 | |
| *** jpena is now known as jpena|off | 16:20 | |
| *** ykarel has quit IRC | 16:23 | |
| *** ykarel has joined #opendev | 16:36 | |
| *** ykarel has quit IRC | 16:38 | |
| *** hashar has quit IRC | 17:02 | |
| *** dtantsur is now known as dtantsur|afk | 17:12 | |
| *** d34dh0r53 has quit IRC | 17:23 | |
| *** vishalmanchanda has quit IRC | 17:23 | |
| *** d34dh0r53 has joined #opendev | 17:28 | |
| *** ralonsoh has quit IRC | 17:34 | |
| *** brinzhang_ has joined #opendev | 17:53 | |
| *** brinzhang has quit IRC | 17:56 | |
| clarkb | infra-root based on https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/zuul-scheduler/tasks/main.yaml#L69-L71 I think we'll only start zuul's scheduler if we set zuul_scheduler_start and we seem to only set that in testing. I think this means I can safely boot up a zuul02.opendev.org and have ansible do its thing. Then we'll control the flip from 01 to 02 as we migrate | 18:14 |
| clarkb | data and update dns etc | 18:14 |
| clarkb | does that sound right to everyone else? | 18:14 |
| clarkb | I'm also looking at the size of the existing server and wondering if we can change it a bit. In particular it seems liek we may not need as much memory or cpu as we've currently got. Maybe a 16GB memory instance is sufficient? We'll hit OOMs faster if they occur but that may actually be a nice thing? | 18:16 |
| clarkb | Also we seem to have a large volume attached for zuul logs. Currently that consumes 14GB of disk. I don't know if we need to attach a dedicated volume for this on the new host | 18:17 |
| clarkb | aha /var/log/zuul is the ephemeral drive, so ya we can just attach whatever ephemeral we get and should be fine | 18:17 |
| clarkb | corvus: ^ does that seem ok to you? any other considerations I should be aware of? | 18:18 |
| clarkb | looks like zuul_web_start is similar to zuul_scheduler_start (I think having web start would be ok though) | 18:20 |
| *** knikolla has quit IRC | 18:32 | |
| *** knikolla has joined #opendev | 18:35 | |
| *** vishalmanchanda has joined #opendev | 18:38 | |
| *** andrewbonney has quit IRC | 19:10 | |
| corvus | clarkb: yes -- though to be honest, i wouldn't mind if we had too much memory for the next few months; i feel like chances of memory leaks are high as we do the v5 work | 19:16 |
| corvus | clarkb: we should not need to sync anything from the old to the new server; the filesystem keystorage is only backup now | 19:17 |
| corvus | (zk is primary) | 19:17 |
| clarkb | corvus: gotcha so stick with the same sized instance in that case. Also keys being primary in zk makes it easy. We can just stop the old scheduler, start the new one and run the queue restore script on the new one | 19:31 |
| clarkb | I'll start working on that early next week | 19:31 |
| corvus | yep | 19:31 |
| clarkb | corvus: will zuul write out the keys locally when it finds them in zk? | 19:31 |
| corvus | yep | 19:32 |
| clarkb | cool | 19:32 |
| corvus | so we'll end up with copies on 02 as well | 19:32 |
| corvus | we'll eventually stop that, but probably only after we've made an export script for backup purposes | 19:32 |
| clarkb | makes sense | 19:32 |
| fungi | we can scale down the scheduler size when we scale out to redundant schedulers too | 19:51 |
| fungi | so it's not like this is our only opportunity | 19:51 |
| *** artom has quit IRC | 20:01 | |
| *** artom has joined #opendev | 20:02 | |
| *** Tengu has quit IRC | 20:04 | |
| *** Tengu has joined #opendev | 20:04 | |
| *** vishalmanchanda has quit IRC | 20:58 | |
| *** mkowalski has quit IRC | 21:59 | |
| *** mkowalski has joined #opendev | 22:00 | |
| *** tosky has quit IRC | 23:08 | |
| *** mlavalle has quit IRC | 23:59 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!