| *** jamesmcarthur has joined #openstack-infra | 00:05 | |
| *** jamesmcarthur has quit IRC | 00:09 | |
| *** jamesmcarthur has joined #openstack-infra | 00:15 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Revert "Fixed use of flake8" https://review.opendev.org/682257 | 00:20 |
|---|---|---|
| *** jamesmcarthur has quit IRC | 00:25 | |
| *** ociuhandu has joined #openstack-infra | 00:31 | |
| *** dychen has joined #openstack-infra | 00:34 | |
| *** ociuhandu has quit IRC | 00:35 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Only install doc requirements if needed https://review.opendev.org/681991 | 00:58 |
| *** jamesmcarthur has joined #openstack-infra | 00:59 | |
| *** e0ne has joined #openstack-infra | 01:00 | |
| *** dchen has quit IRC | 01:02 | |
| *** dchen has joined #openstack-infra | 01:03 | |
| *** e0ne has quit IRC | 01:05 | |
| *** ccamacho has quit IRC | 01:09 | |
| *** jamesmcarthur has quit IRC | 01:10 | |
| *** slaweq_ has joined #openstack-infra | 01:11 | |
| *** slaweq_ has quit IRC | 01:16 | |
| *** zhurong has joined #openstack-infra | 01:20 | |
| *** factor has quit IRC | 01:23 | |
| *** factor has joined #openstack-infra | 01:25 | |
| *** factor has quit IRC | 01:28 | |
| *** factor has joined #openstack-infra | 01:28 | |
| *** factor has quit IRC | 01:31 | |
| *** factor has joined #openstack-infra | 01:31 | |
| *** dingyichen has joined #openstack-infra | 01:32 | |
| *** dchen has quit IRC | 01:34 | |
| *** lbragstad has quit IRC | 01:39 | |
| *** markvoelker has joined #openstack-infra | 01:53 | |
| *** markvoelker has quit IRC | 01:58 | |
| *** factor has quit IRC | 02:03 | |
| *** factor has joined #openstack-infra | 02:04 | |
| *** factor has quit IRC | 02:07 | |
| *** factor has joined #openstack-infra | 02:07 | |
| *** larainema has joined #openstack-infra | 02:08 | |
| *** factor has quit IRC | 02:10 | |
| *** factor has joined #openstack-infra | 02:10 | |
| *** dklyle has quit IRC | 02:11 | |
| *** david-lyle has joined #openstack-infra | 02:11 | |
| *** factor has quit IRC | 02:14 | |
| *** factor has joined #openstack-infra | 02:15 | |
| *** lbragstad has joined #openstack-infra | 02:18 | |
| *** HuyNguyen has joined #openstack-infra | 02:25 | |
| *** dchen has joined #openstack-infra | 02:36 | |
| *** dingyichen has quit IRC | 02:37 | |
| *** ykarel|away has joined #openstack-infra | 02:45 | |
| *** rcernin has quit IRC | 02:59 | |
| *** jamesmcarthur has joined #openstack-infra | 03:10 | |
| *** slaweq_ has joined #openstack-infra | 03:11 | |
| *** slaweq_ has quit IRC | 03:15 | |
| *** rh-jelabarre has joined #openstack-infra | 03:19 | |
| *** markvoelker has joined #openstack-infra | 03:24 | |
| *** markvoelker has quit IRC | 03:29 | |
| *** psachin has joined #openstack-infra | 03:31 | |
| *** ramishra has joined #openstack-infra | 03:33 | |
| *** factor has quit IRC | 03:37 | |
| *** factor has joined #openstack-infra | 03:38 | |
| *** factor has quit IRC | 03:41 | |
| *** factor has joined #openstack-infra | 03:41 | |
| *** factor has quit IRC | 03:42 | |
| *** factor has joined #openstack-infra | 03:43 | |
| *** factor has quit IRC | 03:45 | |
| *** factor has joined #openstack-infra | 03:46 | |
| *** rcernin has joined #openstack-infra | 03:46 | |
| *** whoami-rajat has joined #openstack-infra | 03:47 | |
| *** jhesketh has joined #openstack-infra | 03:50 | |
| *** jamesmcarthur has quit IRC | 04:01 | |
| *** jamesmcarthur has joined #openstack-infra | 04:04 | |
| *** udesale has joined #openstack-infra | 04:11 | |
| *** jamesmcarthur has quit IRC | 04:17 | |
| *** ykarel|away has quit IRC | 04:35 | |
| AJaeger | thanks, ianw ^ Will you cut a new release of dib? | 04:47 |
| AJaeger | I need this for https://review.opendev.org/#/c/682212/ (not necessarily today ;)... | 04:48 |
| *** factor has quit IRC | 04:49 | |
| *** factor has joined #openstack-infra | 04:50 | |
| *** factor has quit IRC | 04:52 | |
| *** factor has joined #openstack-infra | 04:53 | |
| AJaeger | ianw: https://review.opendev.org/682222 and the stack starting at https://review.opendev.org/682211 should make switching from static to AFS easier | 04:53 |
| *** ykarel has joined #openstack-infra | 04:56 | |
| *** factor has quit IRC | 04:56 | |
| *** factor has joined #openstack-infra | 04:57 | |
| *** bhavikdbavishi has joined #openstack-infra | 05:10 | |
| *** pcaruana has joined #openstack-infra | 05:11 | |
| *** rh-jelabarre has quit IRC | 05:24 | |
| ianw | AJaeger: yep, will do when it merges | 05:27 |
| AJaeger | thanks! | 05:32 |
| *** slaweq_ has joined #openstack-infra | 05:36 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Revert "Fixed use of flake8" https://review.opendev.org/682257 | 05:37 |
| AJaeger | config-core, please review https://review.opendev.org/682205 , https://review.opendev.org/682222, https://review.opendev.org/682225, https://review.opendev.org/682118 , and the stack starting at https://review.opendev.org/682211 | 05:45 |
| *** roman_g has joined #openstack-infra | 05:47 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul master: Support nodes setting 'auto' python path https://review.opendev.org/682275 | 05:53 |
| *** piotrowskim has joined #openstack-infra | 06:01 | |
| *** janki has joined #openstack-infra | 06:01 | |
| *** factor has quit IRC | 06:02 | |
| *** kopecmartin|off is now known as kopecmartin | 06:16 | |
| *** jamesmcarthur has joined #openstack-infra | 06:18 | |
| *** Garyx has quit IRC | 06:22 | |
| *** Garyx has joined #openstack-infra | 06:22 | |
| *** jamesmcarthur has quit IRC | 06:23 | |
| *** owalsh has quit IRC | 06:23 | |
| *** logan- has quit IRC | 06:23 | |
| *** owalsh has joined #openstack-infra | 06:23 | |
| *** markvoelker has joined #openstack-infra | 06:24 | |
| *** logan- has joined #openstack-infra | 06:26 | |
| *** markvoelker has quit IRC | 06:30 | |
| AJaeger | frickler: could you review https://review.opendev.org/678573 again, please? Is that fine now? | 06:36 |
| *** sshnaidm_ is now known as sshnaidm|rover | 06:37 | |
| *** janki has quit IRC | 06:39 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul master: Support nodes setting 'auto' python path https://review.opendev.org/682275 | 06:40 |
| *** trident has quit IRC | 06:48 | |
| ianw | AJaeger: 2.27.1 dib should fix the issue, thanks! | 06:51 |
| *** slaweq_ is now known as slaweq | 06:52 | |
| *** rpittau|afk is now known as rpittau | 06:55 | |
| AJaeger | thanks, ianw ! Will recheck in a bit to be sure | 06:56 |
| *** trident has joined #openstack-infra | 06:57 | |
| *** janki has joined #openstack-infra | 07:00 | |
| *** e0ne has joined #openstack-infra | 07:01 | |
| *** pgaxatte has joined #openstack-infra | 07:01 | |
| *** trident has quit IRC | 07:03 | |
| *** e0ne has quit IRC | 07:06 | |
| *** tosky has joined #openstack-infra | 07:08 | |
| AJaeger | ianw: 2.27.1 fixed the issue, https://review.opendev.org/682212 is passing now. thanks | 07:11 |
| *** trident has joined #openstack-infra | 07:12 | |
| *** jklare has quit IRC | 07:17 | |
| *** jklare has joined #openstack-infra | 07:18 | |
| *** ykarel is now known as ykarel|lunch | 07:28 | |
| *** lajoskatona has joined #openstack-infra | 07:30 | |
| *** ralonsoh has joined #openstack-infra | 07:33 | |
| *** jpena|off is now known as jpena | 07:34 | |
| *** FlorianFa has quit IRC | 07:36 | |
| *** sshnaidm|rover is now known as sshnaidm|afk | 07:39 | |
| lajoskatona | Hi, I would like to see if with extra memory networking-odl tempest jobs are passing, and got the advice from donnyd that FortNebula has expanded labels (see: http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2019-09-13.log.html#t2019-09-13T10:38:42) | 07:39 |
| lajoskatona | but to tell the truth I can't make it working in my patch (see https://review.opendev.org/672925 ), if anyone can help, that would be really helpful | 07:41 |
| *** ociuhandu has joined #openstack-infra | 07:41 | |
| *** FlorianFa has joined #openstack-infra | 07:41 | |
| *** ociuhandu has quit IRC | 07:42 | |
| *** apetrich has joined #openstack-infra | 07:44 | |
| AJaeger | lajoskatona: you need a nodeset | 07:44 |
| AJaeger | lajoskatona: leave the job as is, and just add a new one as child like https://opendev.org/openstack/tempest/src/branch/master/.zuul.yaml#L319 and use a nodeset, you need to define the nodeset first | 07:46 |
| lajoskatona | AJaeger: Hi, so add a patch which defines the nodeset, and rebase https://review.opendev.org/672925 on top of it and use the nodeset? | 07:50 |
| *** panda has quit IRC | 07:56 | |
| *** janki has quit IRC | 07:57 | |
| *** janki has joined #openstack-infra | 07:57 | |
| *** panda has joined #openstack-infra | 07:58 | |
| *** xek_ has joined #openstack-infra | 08:05 | |
| *** HuyNguyen2 has joined #openstack-infra | 08:05 | |
| AJaeger | lajoskatona: you can add the nodeset as part of 672925 | 08:08 |
| *** HuyNguyen has quit IRC | 08:09 | |
| *** ricolin has joined #openstack-infra | 08:09 | |
| *** lpetrut has joined #openstack-infra | 08:10 | |
| *** pkopec has joined #openstack-infra | 08:11 | |
| *** dchen has quit IRC | 08:18 | |
| *** xek_ has quit IRC | 08:19 | |
| *** ociuhandu has joined #openstack-infra | 08:20 | |
| *** dychen has quit IRC | 08:20 | |
| *** markvoelker has joined #openstack-infra | 08:27 | |
| *** rcosnita has joined #openstack-infra | 08:27 | |
| *** HuyNguyen has joined #openstack-infra | 08:28 | |
| *** HuyNguyen2 has quit IRC | 08:31 | |
| *** HuyNguyen2 has joined #openstack-infra | 08:31 | |
| *** markvoelker has quit IRC | 08:32 | |
| *** HuyNguyen has quit IRC | 08:33 | |
| *** derekh has joined #openstack-infra | 08:38 | |
| lajoskatona | AJaeger: thanks I check it | 08:41 |
| *** sshnaidm|afk is now known as sshnaidm|rover | 08:43 | |
| *** xenos76 has joined #openstack-infra | 08:43 | |
| *** e0ne has joined #openstack-infra | 08:47 | |
| *** ykarel|lunch is now known as ykarel | 08:49 | |
| *** ociuhandu has quit IRC | 08:51 | |
| *** panda is now known as panda|ruck | 08:59 | |
| *** kaisers has quit IRC | 09:05 | |
| *** kaisers has joined #openstack-infra | 09:08 | |
| openstackgerrit | Luka Peschke proposed opendev/irc-meetings master: Update CloudKitty Team Meeting https://review.opendev.org/682310 | 09:20 |
| *** mgoddard has quit IRC | 09:26 | |
| *** mgoddard has joined #openstack-infra | 09:28 | |
| *** dtantsur|afk is now known as dtantsur | 09:32 | |
| openstackgerrit | Merged opendev/irc-meetings master: Update CloudKitty Team Meeting https://review.opendev.org/682310 | 09:37 |
| *** e0ne has quit IRC | 09:40 | |
| *** e0ne has joined #openstack-infra | 09:40 | |
| *** jaosorior has joined #openstack-infra | 09:45 | |
| *** ricolin has quit IRC | 09:52 | |
| *** ociuhandu has joined #openstack-infra | 10:04 | |
| *** xek_ has joined #openstack-infra | 10:06 | |
| *** bhavikdbavishi has left #openstack-infra | 10:10 | |
| *** gfidente has joined #openstack-infra | 10:13 | |
| dtantsur | hi folks! how do I debug errors like this http://zuul.openstack.org/build/ad8061d08fb544b283c29d9823b1a0e5 ? there doesn't seem to be any logs. | 10:17 |
| AJaeger | dtantsur: where does the link come from? | 10:22 |
| AJaeger | periodic job? | 10:22 |
| AJaeger | post - found it now... | 10:22 |
| dtantsur | AJaeger: post: http://zuul.openstack.org/builds?job_name=ironic-python-agent-build-image-dib-centos7 | 10:22 |
| dtantsur | aha | 10:22 |
| frickler | AJaeger: post. I'll check zuul logs | 10:22 |
| AJaeger | frickler: this is a new job | 10:24 |
| AJaeger | dtantsur: post-run playbook is not in tree | 10:25 |
| AJaeger | frickler, dtantsur, it fails since playbooks/ironic-python-build-image/post.yaml is the post-run playbook but that file is not in git | 10:26 |
| dtantsur | oh wow.. it seems to be in-tree locally | 10:26 |
| AJaeger | dtantsur: playbooks/ironic-python-agent-build-image/post.yaml | 10:27 |
| AJaeger | is there - wrong name? | 10:27 |
| dtantsur | https://github.com/openstack/ironic-python-agent-builder/blob/master/playbooks/ironic-python-agent-build-image/post.yaml | 10:27 |
| AJaeger | yes, python-*agent* but .zuul.yaml talks about *python* without agent | 10:28 |
| dtantsur | seems like this file, am I missing something? | 10:28 |
| *** markvoelker has joined #openstack-infra | 10:28 | |
| dtantsur | daaaaamn | 10:28 |
| *** udesale has quit IRC | 10:28 | |
| dtantsur | great catch AJaeger | 10:28 |
| AJaeger | so, fix .zuul.yaml and you should be good ;) | 10:28 |
| dtantsur | thanks :) | 10:28 |
| AJaeger | you're welcome | 10:29 |
| *** ociuhandu has quit IRC | 10:33 | |
| *** markvoelker has quit IRC | 10:33 | |
| *** ociuhandu has joined #openstack-infra | 10:45 | |
| *** pgaxatte has quit IRC | 10:47 | |
| *** ociuhandu has quit IRC | 10:50 | |
| *** tesseract has joined #openstack-infra | 10:51 | |
| *** dchen has joined #openstack-infra | 11:01 | |
| *** ociuhandu has joined #openstack-infra | 11:01 | |
| *** ociuhandu has quit IRC | 11:03 | |
| *** ociuhandu has joined #openstack-infra | 11:04 | |
| *** lucasagomes has joined #openstack-infra | 11:06 | |
| *** tesseract has quit IRC | 11:09 | |
| *** ociuhandu has quit IRC | 11:10 | |
| *** ociuhandu has joined #openstack-infra | 11:12 | |
| mordred | morning AJaeger - I'm still catching up with email backlog and whantot - anything you need me to look at? | 11:20 |
| AJaeger | mordred: morning, how was your vacation? | 11:20 |
| AJaeger | mordred - this is what I asked for earlier to config-core: please review https://review.opendev.org/682205 , https://review.opendev.org/682222, https://review.opendev.org/682225, https://review.opendev.org/682118 - the usual backlogto move slowly forward ;) | 11:21 |
| mordred | AJaeger: it was great! we saw the aurora borealis last night! | 11:23 |
| mordred | and also did a scuba dive above the arctic circle for the first time :) | 11:24 |
| AJaeger | that must have been cold ;) | 11:25 |
| AJaeger | aurora borealis? Wow! Great! | 11:25 |
| *** dtantsur is now known as dtantsur|bbl | 11:26 | |
| mordred | AJaeger: it was not bad cold-wise. turns out if you have proper drysuit gear it's fine (I was concerned about this before the dive) | 11:27 |
| AJaeger | ;) | 11:27 |
| mordred | and yes - we were VERY excited. the aurora was a bit faint still, so our cameras actually saw it better than we did - but it was still *SUPER* cool | 11:27 |
| * AJaeger can imagine... | 11:28 | |
| *** jpena is now known as jpena|lunch | 11:30 | |
| openstackgerrit | Merged opendev/base-jobs master: Add nodeset for debian-buster https://review.opendev.org/682118 | 11:32 |
| openstackgerrit | Merged zuul/zuul master: Better error handling for the PagureAPIClient https://review.opendev.org/679746 | 11:35 |
| openstackgerrit | Merged openstack/openstack-zuul-jobs master: Start train translations https://review.opendev.org/682205 | 11:36 |
| *** ociuhandu has quit IRC | 11:38 | |
| *** ccamacho has joined #openstack-infra | 11:39 | |
| *** udesale has joined #openstack-infra | 11:45 | |
| openstackgerrit | Merged zuul/zuul master: Remove auto-add of localhost from unit test inventory https://review.opendev.org/681641 | 11:45 |
| donnyd | mordred: pictures or it didn't happen | 11:48 |
| donnyd | :) | 11:49 |
| *** rh-jelabarre has joined #openstack-infra | 11:50 | |
| *** njohnston has joined #openstack-infra | 11:51 | |
| *** rcosnita has quit IRC | 11:53 | |
| *** weshay has joined #openstack-infra | 11:53 | |
| *** dchen has quit IRC | 11:53 | |
| *** gfidente has quit IRC | 11:55 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Add tests for manifest generation for missing files https://review.opendev.org/678573 | 11:56 |
| *** xek_ has quit IRC | 12:00 | |
| *** ociuhandu has joined #openstack-infra | 12:00 | |
| *** ociuhandu has quit IRC | 12:00 | |
| *** ociuhandu has joined #openstack-infra | 12:01 | |
| *** pgaxatte has joined #openstack-infra | 12:07 | |
| openstackgerrit | Merged zuul/zuul master: Discuss executor-only jobs, add unit-test https://review.opendev.org/679184 | 12:08 |
| *** soniya29 has joined #openstack-infra | 12:11 | |
| *** dave-mccowan has joined #openstack-infra | 12:13 | |
| *** goldyfruit___ has quit IRC | 12:15 | |
| *** rfolco has joined #openstack-infra | 12:23 | |
| *** rlandy has joined #openstack-infra | 12:25 | |
| *** lbragstad has quit IRC | 12:32 | |
| *** jpena|lunch is now known as jpena | 12:33 | |
| *** gfidente has joined #openstack-infra | 12:35 | |
| mordred | donnyd: https://imgur.com/a/KJWOsQA | 12:35 |
| *** rcernin has quit IRC | 12:35 | |
| donnyd | mordred: I was starting to think I made you mad. | 12:37 |
| donnyd | Lol | 12:37 |
| donnyd | Wow, that is just amazing | 12:37 |
| *** kjackal has joined #openstack-infra | 12:37 | |
| donnyd | I just got a new lens for my DSLR | 12:37 |
| donnyd | Hoping to be able to get the milky way some night this week | 12:38 |
| *** rosmaita has quit IRC | 12:38 | |
| mordred | oh neat! | 12:40 |
| *** prometheanfire has quit IRC | 12:40 | |
| mordred | it was faint - so each of those frames is a 2 second exposure :) | 12:40 |
| *** prometheanfire has joined #openstack-infra | 12:41 | |
| mordred | but - like - even faint, holy crap it was cool in person | 12:41 |
| AJaeger | nice! | 12:43 |
| AJaeger | donnyd: thanks for pushing mordred ;) | 12:43 |
| *** HuyNguyen2 has quit IRC | 12:44 | |
| donnyd | mordred: what do you use? | 12:44 |
| donnyd | I was thinking about getting a full frame , because my 7D is a crop sensor... but when I bought it ... I didn't know that actually mattered | 12:46 |
| mordred | donnyd: I'm using a Sony RX10 IV - I used to have a Canon Eos 5 with a bunch of lenses, but travelling with it all was ... heavy | 12:46 |
| mordred | s/5/5d/ | 12:46 |
| mordred | the RX10 has a 24-600mm zoom - so for a compact unit it's *super* versatile | 12:47 |
| *** jamesmcarthur has joined #openstack-infra | 12:48 | |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Store autohold requests in zookeeper https://review.opendev.org/661114 | 12:48 |
| donnyd | mordred: It surely looks it | 12:49 |
| donnyd | AJaeger: NP, its what I am here for | 12:50 |
| donnyd | I haven't heard any complaints about FN in a while so I am guess we must have fixed the issues for good this time | 12:50 |
| mordred | \o/ | 12:50 |
| *** rosmaita has joined #openstack-infra | 12:51 | |
| *** nhicher has joined #openstack-infra | 12:52 | |
| AJaeger | great! | 12:53 |
| AJaeger | and thanks, donnyd ! | 12:53 |
| *** rcernin has joined #openstack-infra | 12:59 | |
| *** markvoelker has joined #openstack-infra | 13:04 | |
| *** kjackal has quit IRC | 13:05 | |
| clarkb | morning. Not quite here yet but should be at computer soon. We are still on for project renames ya? if so anyone want to disable the puppet ansible cron on bridge? | 13:05 |
| clarkb | I'll get repos.yaml copiedto bridge and an edited rename playbook with no reference to the deleted org creation file soon | 13:06 |
| mordred | clarkb: oh my - I'm back from vacation on a renames day? | 13:07 |
| clarkb | mordred: yes, you agreed to it a few weeks ago too :) | 13:07 |
| * mordred feels like he knew this was going to be the case ... yeah | 13:07 | |
| mordred | clarkb: I don't really know how computers work again yet | 13:07 |
| *** mriedem has joined #openstack-infra | 13:08 | |
| *** markvoelker has quit IRC | 13:09 | |
| clarkb | mordred: soundslike a goof vacation | 13:09 |
| clarkb | *good | 13:09 |
| *** Goneri has joined #openstack-infra | 13:11 | |
| mordred | ++ | 13:11 |
| frickler | infra-root: I'm seeing log streaming issues that seem related to ze05, but I'm not sure how to debug further. e.g. http://zuul.openstack.org/stream/00bd6308b2ec48a7897cf2eea4bbc39e?logfile=console.log | 13:11 |
| clarkb | frickler: check if two zuul-executor procrsses are running on that host | 13:12 |
| clarkb | chances are it OOM'd and the child zuul-executor process which acts asthe finger server waskilled | 13:12 |
| frickler | clarkb: zuul 2788 0.0 0.0 0 0 ? Z Sep05 1:16 [zuul-executor] <defunct> | 13:12 |
| clarkb | yup that must be it then. If we restart the executor that s hould fix it. You have to stop, wait for it to finish, then start | 13:13 |
| frickler | will it terminate running jobs? in that case I'd rather wait a bit for my jobs to finish | 13:14 |
| fungi | clarkb: yeah, i had a reminder for myself to disable the cron but was just getting to it now | 13:14 |
| fungi | what was the link to the etherpad again? | 13:15 |
| fungi | #status log disabled run_all.sh on bridge in preparation for project renames maintenance | 13:15 |
| openstackstatus | fungi: finished logging | 13:15 |
| clarkb | fungi: I'll have to dig it out. Give me a few | 13:16 |
| fungi | no worries. happy to send a status notice shortly too | 13:17 |
| fungi | i can dig the url out of channel logs | 13:17 |
| fungi | https://etherpad.openstack.org/p/project-renames-2019-09-10 | 13:17 |
| fungi | found it ^ | 13:18 |
| *** eharney has joined #openstack-infra | 13:18 | |
| *** KeithMnemonic has joined #openstack-infra | 13:18 | |
| fungi | status log The Gerrit service on review.opendev.org will be offline briefly starting at 14:00 UTC for maintenance: http://lists.openstack.org/pipermail/openstack-discuss/2019-September/009064.html | 13:23 |
| fungi | something like that? | 13:23 |
| fungi | er, status notice i guess | 13:23 |
| mordred | yeah - notice | 13:24 |
| fungi | i discovered that the announcement went to all the project discussion lists except the infra list. heh | 13:24 |
| mordred | but otherwise, awesome | 13:24 |
| fungi | we probably ought to announce future maintenance on the infra ml too | 13:24 |
| clarkb | ++ | 13:26 |
| fungi | #status notice The Gerrit service on review.opendev.org will be offline briefly starting at 14:00 UTC (that's roughly 30 minutes from now) for maintenance: http://lists.openstack.org/pipermail/openstack-discuss/2019-September/009064.html | 13:27 |
| openstackstatus | fungi: sending notice | 13:27 |
| mordred | clarkb, fungi: the etherpad mentions repos-first and repos-second - but I only see repos.yaml on disk | 13:27 |
| clarkb | mordred: ++ let me split out the first project there as the test one | 13:27 |
| * mordred is not sure in which direction that is an issue | 13:27 | |
| mordred | cool | 13:27 |
| -openstackstatus- NOTICE: The Gerrit service on review.opendev.org will be offline briefly starting at 14:00 UTC (that's roughly 30 minutes from now) for maintenance: http://lists.openstack.org/pipermail/openstack-discuss/2019-September/009064.html | 13:28 | |
| clarkb | mordred: done | 13:28 |
| fungi | --- end run @ 2019-09-16T13:28:34+00:00 --- | 13:28 |
| fungi | ansible has wrapped up | 13:28 |
| clarkb | mordred: fungi also you may want to check the edits in /root/renames/20190916/system-config | 13:29 |
| clarkb | git diff will show them to you | 13:29 |
| clarkb | basically its stop managing gerrit we'll do that by hand per the etherpad and don't manage gitea orgs because that file does not exist anymore | 13:29 |
| fungi | thanks, looking | 13:29 |
| clarkb | re gitea orgs we aren't creating any new ones this rename so that should be safe | 13:29 |
| *** beekneemech is now known as bnemec | 13:30 | |
| openstackstatus | fungi: finished sending notice | 13:30 |
| clarkb | the other todo is reviewing https://review.opendev.org/#/c/662530/ and https://review.opendev.org/#/c/669298/ | 13:31 |
| fungi | the diff also comments out gerrit group renames | 13:32 |
| fungi | i guess we don't have any of those this time either | 13:32 |
| clarkb | yup there are no gerrit group renames either | 13:32 |
| fungi | diff lgtm then | 13:32 |
| clarkb | I think if this time goes smoothly with no replication on start and systemd successfully stops and starts gerrit then the next time we do this we can do the far more automated thing | 13:32 |
| clarkb | and merge the changes once zuul +1's them etc | 13:32 |
| fungi | image encryption meeting just wrapped up so i'm going to disappear for a couple minutes and then come back to help prep asap | 13:32 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Expose volumes for the gerrit hooks and static dirs https://review.opendev.org/678875 | 13:33 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run a gerrit container (test only) https://review.opendev.org/630406 | 13:33 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Use latest bazel https://review.opendev.org/682363 | 13:33 |
| mordred | clarkb: files look reasonable | 13:34 |
| *** xek has joined #openstack-infra | 13:35 | |
| *** whoami-rajat has quit IRC | 13:36 | |
| * clarkb makes tea, but I think we are pretty well set. Even took notes in the therpad re the two things we should confirm work as expected above so that we can simplify future renames | 13:37 | |
| *** aaronsheffield has joined #openstack-infra | 13:37 | |
| corvus | o/ | 13:38 |
| clarkb | corvus: good morning https://etherpad.openstack.org/p/project-renames-2019-09-10 is our notes etherpad. I've got the files set up on bridge pre the etherpad if you want to review /root/renames/20190916/system-config diff and /root/renames/20190916/*.yaml | 13:39 |
| clarkb | there are also the two changes to review on the etherpad. Other than that I think we are pretty well set to do this at the scheduled time | 13:40 |
| corvus | clarkb, mordred, frickler: that oom on ze05 suggests that the json log rework was not enough to prevent ooms. we'll probably have to merge one of clarkb's patches to either terminate or restart. | 13:40 |
| mordred | corvus: boo | 13:40 |
| clarkb | corvus: re those changes we should discuss which method is preferred then I'll clena up the corresponding change and take it from there | 13:40 |
| *** signed8bit has joined #openstack-infra | 13:44 | |
| *** pcaruana has quit IRC | 13:45 | |
| frickler | corvus: clarkb: there are some oom traces in dmesg on ze05 if you want to take a look. I will defer from restart the executor until we decide how to proceed properly | 13:46 |
| *** rcernin has quit IRC | 13:49 | |
| corvus | frickler: it can be restarted at your convenience; i don't think we need any more debug info | 13:50 |
| corvus | frickler: at this point, we may as well wait until we shut down the rest of the system | 13:50 |
| *** goldyfruit___ has joined #openstack-infra | 13:50 | |
| corvus | oh right, we don't stop zuul anymore | 13:51 |
| corvus | but we still may as well wait until gerrit is down | 13:51 |
| *** iurygregory has joined #openstack-infra | 13:51 | |
| *** soniya29 has quit IRC | 13:54 | |
| clarkb | if someone elseis willing to drive that would be great | 13:58 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-sphinx master: Switch to zuul-release-python https://review.opendev.org/682366 | 13:58 |
| fungi | i'm now in place. we need a status alert next? | 14:00 |
| clarkb | sure. | 14:00 |
| clarkb | Then we need to merge teh changes | 14:00 |
| fungi | #status alert The Gerrit service on review.opendev.org is offline briefly for maintenance: http://lists.openstack.org/pipermail/openstack-discuss/2019-September/009064.html | 14:01 |
| openstackstatus | fungi: sending alert | 14:01 |
| fungi | following normal approval/gating? | 14:01 |
| clarkb | https://review.opendev.org/#/c/662530/3 and https://review.opendev.org/#/c/669298/3 | 14:01 |
| clarkb | no they are not able to pass testing because the projects don't exist yet | 14:01 |
| *** markvoelker has joined #openstack-infra | 14:02 | |
| fungi | ahh, yes, i'll add myself to bootstrappers and do that now | 14:02 |
| clarkb | (we merge ahead of time to ensure things are replicated, the fix for replication on gerrit start should address this but this process will confirm. I'm taking notes on that at the top of the etherpad) | 14:02 |
| corvus | i'll restart ze05 | 14:02 |
| *** ykarel is now known as ykarel|afk | 14:03 | |
| -openstackstatus- NOTICE: The Gerrit service on review.opendev.org is offline briefly for maintenance: http://lists.openstack.org/pipermail/openstack-discuss/2019-September/009064.html | 14:04 | |
| *** ChanServ changes topic to "The Gerrit service on review.opendev.org is offline briefly for maintenance: http://lists.openstack.org/pipermail/openstack-discuss/2019-September/009064.html" | 14:04 | |
| *** kjackal has joined #openstack-infra | 14:04 | |
| paladox | corvus my change was merged upstream, so the config for 3.1 will work now! :) | 14:05 |
| clarkb | I'll do the stop and start of gerritso thta I can double check the systemd stuff works | 14:05 |
| clarkb | will wait for us to have changes merged and replicated and ready to run the rename playbook before I do that though | 14:06 |
| *** pcaruana has joined #openstack-infra | 14:07 | |
| openstackstatus | fungi: finished sending alert | 14:08 |
| *** sthussey has joined #openstack-infra | 14:08 | |
| openstackgerrit | Merged openstack/project-config master: Rename x/kayobe* to openstack/ https://review.opendev.org/669298 | 14:08 |
| openstackgerrit | Merged openstack/project-config master: Rename x/ansible-role-cloud-launcher -> opendev/ https://review.opendev.org/662530 | 14:08 |
| *** janki has quit IRC | 14:09 | |
| clarkb | those commits are on all 8 gitea backends | 14:10 |
| fungi | just a heads up, i didn't comment out the cronjob for run_cloud_launcher.sh so the ansible processes for that are showing up in the process table on bridge at the moment. that should be fine though | 14:10 |
| clarkb | ya that should be fine | 14:10 |
| mordred | yah | 14:10 |
| clarkb | fungi: do you want to start a root screen for running the ansible playbooks? | 14:10 |
| fungi | done | 14:11 |
| fungi | feel free to screen -x as root and join | 14:11 |
| clarkb | I'm on | 14:11 |
| *** markvoelker has quit IRC | 14:11 | |
| clarkb | I'm ready to stop gerrit on review.o.o as soon as everyone else is ready | 14:12 |
| fungi | i guess we need to double-check those commits are now present in gitea? | 14:12 |
| *** markvoelker has joined #openstack-infra | 14:12 | |
| clarkb | fungi: I did that, but if someone else wants to do so that would be great | 14:12 |
| fungi | ahh, cool, i see now you said that 2 minutes ago ;) | 14:12 |
| corvus | i did as well | 14:12 |
| fungi | perfect | 14:12 |
| * fungi is still ingesting his caffeine dose | 14:13 | |
| clarkb | I did https://gitea01.opendev.org:3000/openstack/project-config/commits/branch/master and ran through 01 to 08 | 14:13 |
| corvus | no pending replication tasks | 14:13 |
| fungi | so should be clear to stop gerrit, yes | 14:13 |
| clarkb | ok proceeding with that now | 14:13 |
| clarkb | its done systemctl stop gerrit took a bit of time but seems to have worked | 14:15 |
| clarkb | fungi: I think we can proceed with renaming the first project | 14:15 |
| fungi | i have queued up the command from the pad | 14:16 |
| corvus | ++ | 14:16 |
| clarkb | that looks correct to me | 14:16 |
| fungi | okay, proceeding | 14:16 |
| mordred | yeah | 14:16 |
| clarkb | I didn't catch where it wrote out the status of the variou stasks but that looked like it was happy? | 14:17 |
| *** markvoelker has quit IRC | 14:17 | |
| clarkb | I guess I can open cloud launcher in opendev.org and see if I redirect | 14:18 |
| fungi | looks right, yeah | 14:18 |
| corvus | no failed tasks | 14:18 |
| clarkb | it seems to redirect for me in gitea | 14:18 |
| clarkb | https://opendev.org/x/ansible-role-cloud-launcher | 14:19 |
| fungi | same for me | 14:19 |
| mordred | I agree with the redirect | 14:19 |
| fungi | i've queued up the next command from the pad now | 14:20 |
| corvus | ++ | 14:20 |
| clarkb | fungi: I think I'm ready for it | 14:20 |
| fungi | running | 14:20 |
| mordred | that looked not unhappy | 14:21 |
| clarkb | https://opendev.org/x/kayobe redirects for me as expected | 14:21 |
| fungi | yeah, everything seems to have run cleanly | 14:21 |
| clarkb | I can start gerrit up whenever yall are ready | 14:22 |
| fungi | lgtm, ready | 14:22 |
| clarkb | corvus: mordred ^ | 14:22 |
| corvus | clarkb: ++ | 14:22 |
| corvus | no groups need to be renamed this time, correct? | 14:22 |
| AJaeger | kayobe has now some broken zuul files, I'm fixing... | 14:22 |
| clarkb | corvus: correct the acl files were renamed but their contents did not change | 14:22 |
| AJaeger | (required-projects) | 14:23 |
| clarkb | renamed from eg x/kayobe.config to openstack/kayobe.config | 14:23 |
| clarkb | any other last concerns to check before I start gerrit or should I just go for it at this point? | 14:23 |
| corvus | still ++ | 14:24 |
| mordred | ++ | 14:24 |
| clarkb | k starting gerrit now | 14:24 |
| clarkb | AJaeger: thanks! | 14:24 |
| clarkb | gerrit is up according to systemctl | 14:24 |
| * mordred looks sideways at systemctl questioningly | 14:25 | |
| clarkb | web ui is up for me | 14:25 |
| mordred | me too | 14:25 |
| frickler | submitting a new ps worked fine, too | 14:26 |
| clarkb | gerrit show queue doesn't show 18k items | 14:26 |
| mordred | woot! | 14:26 |
| fungi | yep, seems to be working for mw | 14:26 |
| clarkb | so the replication config change seems to be working | 14:26 |
| fungi | me | 14:26 |
| mordred | we learned things | 14:26 |
| clarkb | next up is triggering reindexing | 14:26 |
| clarkb | I can do that as I'm already on review.o.o | 14:26 |
| fungi | and yeah, no more replication delays after a gerrit restart now | 14:26 |
| *** ykarel_ has joined #openstack-infra | 14:26 | |
| AJaeger | fix for kayobe: https://review.opendev.org/682371 | 14:27 |
| fungi | this will be awesome | 14:27 |
| clarkb | starting with the accounts index | 14:27 |
| *** ykarel_ has quit IRC | 14:27 | |
| fungi | release team no longer has to wait two hours after a gerrit restart before it's safe to approve release requests (though they may still need to wait one hour after eating before going for a swim) | 14:28 |
| *** armax has joined #openstack-infra | 14:28 | |
| corvus | show-queue shows the reindex tasks | 14:28 |
| corvus | we've gone from 25k to 5k quickly | 14:28 |
| clarkb | I've run both account and change reindex commands | 14:28 |
| fungi | the gerrit "error" log will also log a timestamp when reindexes start and end | 14:28 |
| corvus | now i see 2k project indexes; that'll go less quickly | 14:29 |
| corvus | i should check on trevor's changes... | 14:29 |
| *** ykarel|afk has quit IRC | 14:30 | |
| fungi | when should we lift the status alert? | 14:30 |
| clarkb | fungi: I think I'm happy with the state of things | 14:30 |
| clarkb | I've written notes in the etherpad for next time. I think replication and systemd stuff worked great | 14:31 |
| clarkb | that leaves the todo of addressing the org creation code that was deleted | 14:31 |
| *** jamesmcarthur has quit IRC | 14:31 | |
| mordred | corvus: yeah - I look forward to trevors patches | 14:32 |
| fungi | status ok The Gerrit outage portion of the current maintenance is complete and the service is back on line, however reindexing for renamed repositories is still underway and some Zuul job fixes are in the process of being applied | 14:32 |
| fungi | that work? | 14:32 |
| clarkb | mordred: the yaml file included at https://opendev.org/opendev/system-config/src/branch/master/playbooks/gitea-rename-tasks.yaml#L24-L27 was deleted when you rewrote that stuff into python | 14:32 |
| clarkb | fungi: ++ | 14:32 |
| clarkb | mordred: I'm not sure how we want to handle that in the future. Maybe undelete that file and only use it for renames ? | 14:32 |
| fungi | #status ok The Gerrit outage portion of the current maintenance is complete and the service is back on line, however reindexing for renamed repositories is still underway and some Zuul job fixes are in the process of being applied | 14:32 |
| openstackstatus | fungi: sending ok | 14:32 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Fix gerritbot sorting https://review.opendev.org/682375 | 14:32 |
| *** jamesmcarthur has joined #openstack-infra | 14:33 | |
| AJaeger | the rename change had one tiny bug (wrong sort), cleanup change ^ | 14:34 |
| clarkb | AJaeger: +2'd | 14:34 |
| mordred | clarkb: hrm. | 14:34 |
| mordred | clarkb: yeah- maybe so | 14:35 |
| AJaeger | and .gitreview update for opendev/ansible-role-cloud-launcher: https://review.opendev.org/682372 | 14:35 |
| *** ChanServ changes topic to "Discussion of OpenStack Developer and Community Infrastructure | docs http://docs.openstack.org/infra/ | bugs https://storyboard.openstack.org/ | source https://opendev.org/opendev/ | channel logs http://eavesdrop.openstack.org/irclogs/%23openstack-infra/" | 14:35 | |
| -openstackstatus- NOTICE: The Gerrit outage portion of the current maintenance is complete and the service is back on line, however reindexing for renamed repositories is still underway and some Zuul job fixes are in the process of being applied | 14:35 | |
| corvus | mordred: trevor's change merged! | 14:36 |
| fungi | are all the post-outage fixes using the same review topic? | 14:36 |
| corvus | mordred, clarkb, fungi: https://gerrit-review.googlesource.com/c/gerrit/+/234894 | 14:36 |
| corvus | that will speed up reindexing | 14:36 |
| clarkb | fungi: looks like it | 14:36 |
| clarkb | corvus: excellent | 14:36 |
| *** goldyfruit_ has joined #openstack-infra | 14:36 | |
| fungi | cool, as soon as my gertty finishes restarting i'll take a look | 14:36 |
| AJaeger | Zuul is throwing errors for the new repos, http://zuul.opendev.org/t/openstack/config-errors . Should be fixed once the config is reloaded | 14:37 |
| clarkb | ok notes collected on the items I wanted to call out for improvements next time. Feel free to add any additional items to the etherpad around that if you think o them | 14:37 |
| corvus | that's in stable/2.13 -- we could even apply it to our current gerrit if we want. i've also confirmed it's been merged up through all the branches to master, so any future gerrit releases will have it. | 14:37 |
| clarkb | AJaeger: ya once we get puppet to run and it will reload the config | 14:37 |
| mordred | corvus: woot! | 14:37 |
| corvus | AJaeger: yeah, it'll be a while, the instructions have us waiting for an ansible/puppet pulse to trigger zuul's reconfig | 14:38 |
| clarkb | I can't think of anything else to confirm before reenabling puppet myself. We checked opendev backends for the important changes | 14:38 |
| corvus | so up to 40m from now | 14:38 |
| AJaeger | mgoddard: before Zuul is ready to use the new kayobe repos, we need to do a few more fixes. | 14:38 |
| mordred | corvus: I have not been able to get 2.13 to rebuild - so I think we shouldn't spend too much effort on that - but I'm excited about making sure it's in our 2.15 builds | 14:38 |
| mgoddard | AJaeger: ack | 14:38 |
| corvus | mordred: yeah, i think that's the key that will make the upgrades go faster | 14:38 |
| mordred | corvus: ++ | 14:39 |
| clarkb | fungi: mordred corvus do we want to reenable the cron? | 14:39 |
| mgoddard | AJaeger: let me know if I can help | 14:39 |
| clarkb | it will run in ~6 minutes if we reenable it before then | 14:39 |
| corvus | clarkb: ++ | 14:39 |
| *** jamesmcarthur has quit IRC | 14:39 | |
| *** goldyfruit___ has quit IRC | 14:39 | |
| fungi | i can uncomment it now | 14:39 |
| clarkb | fungi: thank you | 14:39 |
| openstackstatus | fungi: finished sending ok | 14:39 |
| fungi | and done | 14:40 |
| mordred | clarkb: ++ | 14:40 |
| fungi | #status log run_all.sh cronjob on bridge uncommented again | 14:40 |
| openstackstatus | fungi: finished logging | 14:40 |
| AJaeger | mgoddard: just wait a bit longer ;) Once everything is clear, you might need to recheck my changes and then they'll merge... Just wanted to point out that Zuul does not know about the repos yet, so approving these changes is great (and pushing to gerrit) but nothing is running yet. | 14:41 |
| mgoddard | AJaeger: got it, thanks | 14:41 |
| clarkb | mgoddard: the way you'll know if zuul is happy is the orange bell in the top right of the zuul status page should go away | 14:41 |
| clarkb | if you click on it you'll get details that correspond to what AJaeger said above | 14:42 |
| clarkb | and once puppet does its thing that should be fixed | 14:42 |
| mgoddard | clarkb: I see it, thanks | 14:42 |
| *** rcosnita has joined #openstack-infra | 14:43 | |
| clarkb | ansipuppet has started | 14:45 |
| fungi | right on cue | 14:46 |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Add caching of autohold requests https://review.opendev.org/663412 | 14:47 |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Add autohold-info CLI command https://review.opendev.org/662487 | 14:47 |
| fungi | were there any infra-side changes we needed to review besides the gerritbot config sort order mistake? | 14:48 |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Record held node IDs with autohold request https://review.opendev.org/662498 | 14:48 |
| clarkb | fungi: the opendev/ansible-role-cloud-launcher git review update change too | 14:49 |
| clarkb | I think that is it | 14:49 |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Auto-delete expired autohold requests https://review.opendev.org/663762 | 14:50 |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Mark nodes as USED when deleting autohold https://review.opendev.org/664060 | 14:50 |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Add autohold delete/info commands to web API https://review.opendev.org/679057 | 14:50 |
| fungi | corvus: not in any way urgent, but do you have any ideas for how to profile memory utilization for gertty? if i leave it running for a few days it will gobble 2gb+ vmem, which isn't a huge pain just means i need to remember to restart it every few days | 14:50 |
| clarkb | 'use errors=ignore instead of skip' I guess that is a warning we can cleanup | 14:50 |
| AJaeger | fungi: one for opendev/ansible-role-cloud-launcher: https://review.opendev.org/682372 | 14:51 |
| AJaeger | fungi, I'm not aware of other changes | 14:51 |
| corvus | fungi: i don't offhand :( | 14:52 |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Auto-delete expired autohold requests https://review.opendev.org/663762 | 14:52 |
| AJaeger | mgoddard: you need to backport the kayobe changes to all active branches. I only updated master, so, please do the rest (especially the job change as needed!) | 14:52 |
| fungi | corvus: no worries, i have a suspicion it starts cropping up after i do things like accidentally follow a link to a nova change and then it keeps that around in recently-seen changes | 14:53 |
| mgoddard | AJaeger: will do | 14:53 |
| clarkb | dns servers are getting ansibled now | 15:00 |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Auto-delete expired autohold requests https://review.opendev.org/663762 | 15:00 |
| clarkb | now afs | 15:01 |
| openstackgerrit | Merged openstack/project-config master: Fix gerritbot sorting https://review.opendev.org/682375 | 15:01 |
| *** jamesmcarthur has joined #openstack-infra | 15:02 | |
| *** goldyfruit___ has joined #openstack-infra | 15:03 | |
| clarkb | now gitea | 15:03 |
| *** goldyfruit_ has quit IRC | 15:05 | |
| *** rcosnita has quit IRC | 15:07 | |
| *** rcosnita_ has joined #openstack-infra | 15:07 | |
| clarkb | gitea is done | 15:07 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Use latest bazel https://review.opendev.org/682363 | 15:07 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Expose volumes for the gerrit hooks and static dirs https://review.opendev.org/678875 | 15:07 |
| clarkb | seems to hvae been happy | 15:07 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run a gerrit container (test only) https://review.opendev.org/630406 | 15:07 |
| clarkb | review.o.o should happen next | 15:07 |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Mark nodes as USED when deleting autohold https://review.opendev.org/664060 | 15:07 |
| *** jamesmcarthur has quit IRC | 15:07 | |
| *** openstackgerrit has quit IRC | 15:08 | |
| fungi | looks like gerritbot got reconfigured ;) | 15:08 |
| clarkb | faeb56b3d9b03ab56b751e6adb8de4bfc6a11922 is the config review.o.o is running with which ya is the gerritbot fix | 15:08 |
| *** openstackgerrit has joined #openstack-infra | 15:08 | |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Add autohold delete/info commands to web API https://review.opendev.org/679057 | 15:08 |
| *** xek has quit IRC | 15:08 | |
| fungi | and even better, is still working | 15:08 |
| *** bdodd has joined #openstack-infra | 15:09 | |
| clarkb | manage projects is running and seems to be doing what I expect of it | 15:10 |
| clarkb | puppet is done on review.o.o | 15:12 |
| clarkb | everything I can see so far looks good | 15:12 |
| *** ykarel has joined #openstack-infra | 15:12 | |
| clarkb | ~1450 reindexing tasks now | 15:16 |
| *** cmorpheus is now known as cmurphy | 15:19 | |
| clarkb | and now 3 reindexing tasks | 15:19 |
| clarkb | nova neutron and openstack manuals I'm guessing | 15:19 |
| clarkb | puppet ran on zuul01 and the scheduler was asked to reload | 15:21 |
| clarkb | I think it is working on processing that now | 15:21 |
| *** mattw4 has joined #openstack-infra | 15:21 | |
| openstackgerrit | Artom Lifshitz proposed openstack/project-config master: Make whitebox use Storyboard https://review.opendev.org/651942 | 15:22 |
| *** dtantsur|bbl is now known as dtantsur | 15:23 | |
| clarkb | ansipuppet is done now | 15:24 |
| *** VedaAnnayappa has joined #openstack-infra | 15:24 | |
| VedaAnnayappa | hi | 15:25 |
| clarkb | hello | 15:25 |
| VedaAnnayappa | I am facing an issue with psycopg2, I am getting " Error: pg_config executable not found." | 15:27 |
| AJaeger | VedaAnnayappa: you need a bindep.txt file with postgres packages in it. That was a change months ago and communicated on openstack-discuss mailing list | 15:28 |
| clarkb | VedaAnnayappa: the pg_config utility is something that you get with postgres itself. I think that means psycopg2 is missing system deps | 15:28 |
| clarkb | and ya AJaeger has the fix | 15:28 |
| AJaeger | Zuul was reconfigured, I still see the error | 15:28 |
| VedaAnnayappa | can u point me to the changeset | 15:29 |
| AJaeger | oh, now different ones ;) | 15:29 |
| AJaeger | tests running for kayobe - great | 15:29 |
| clarkb | AJaeger: looks likeerrors that should be fixed by the kayobe fix changes | 15:29 |
| clarkb | VedaAnnayappa: the changesset that fixes the problem? | 15:30 |
| clarkb | VedaAnnayappa: I don't know of any projects that had to fix it off of the top of my head. AJaeger may | 15:30 |
| AJaeger | mgoddard: you can merge the changes now... | 15:30 |
| AJaeger | VedaAnnayappa: just check for bindep.txt files with codesearch.openstack.org | 15:30 |
| VedaAnnayappa | by adding libpq-dev==9.0.0 | 15:30 |
| *** whoami-rajat has joined #openstack-infra | 15:30 | |
| VedaAnnayappa | in lower constraints , doesnt it helps | 15:30 |
| mgoddard | AJaeger: excellent | 15:30 |
| AJaeger | VedaAnnayappa, bindep.txt | 15:31 |
| *** e0ne has quit IRC | 15:31 | |
| AJaeger | VedaAnnayappa: copy from nova/bindep.txt for example | 15:31 |
| fungi | examples: http://codesearch.openstack.org/?q=postgres&i=nope&files=bindep.txt | 15:32 |
| *** lajoskatona has quit IRC | 15:33 | |
| VedaAnnayappa | just by adding this file is sufficient or i need to go any other action too? | 15:34 |
| VedaAnnayappa | this issue occurs only for py3.7 | 15:34 |
| *** pgaxatte has quit IRC | 15:36 | |
| clarkb | VedaAnnayappa: I expect that should fix the pg_config problem, but its possible that revleas more issues | 15:38 |
| *** lbragstad has joined #openstack-infra | 15:41 | |
| VedaAnnayappa | how can i run the gate without pushing for review | 15:41 |
| AJaeger | VedaAnnayappa: https://docs.openstack.org/bindep has details | 15:41 |
| *** jamesmcarthur has joined #openstack-infra | 15:42 | |
| *** kjackal has quit IRC | 15:43 | |
| fungi | VedaAnnayappa: probably the reason it's only happening with python 3.7 is that some python library you're installing has c extensions built against postgres headers and lacks a prebuilt wheel for that interpreter version so pip is trying to rebuild a wheel for you from source, which involves compiling those extensions locally | 15:43 |
| *** jamesmcarthur has quit IRC | 15:45 | |
| *** jamesmcarthur has joined #openstack-infra | 15:45 | |
| *** gyee has joined #openstack-infra | 15:47 | |
| VedaAnnayappa | i wonder, it works for py36 and not for py37, will try with bindep.txt , hope it passes gate check | 15:47 |
| *** ccamacho has quit IRC | 15:52 | |
| clarkb | Looks like we are just waiting on kayobe zuul fixes and the reindexin for nova and openstack manuals | 15:54 |
| clarkb | I'm going to find breakfast as everything else looks happy | 15:54 |
| openstackgerrit | Andreas Jaeger proposed opendev/ansible-role-cloud-launcher master: Ignore ansible-lint errors https://review.opendev.org/682394 | 15:55 |
| AJaeger | mgoddard: please approve https://review.opendev.org/#/c/682371/ and backport to all branches, so that the config errors disappear. I hope it passes ;) | 15:56 |
| *** xek has joined #openstack-infra | 15:57 | |
| *** ccamacho has joined #openstack-infra | 15:58 | |
| AJaeger | mgoddard: otherwise everything looks fine from the Zuul side | 15:58 |
| *** mattw4 has quit IRC | 15:58 | |
| *** kjackal has joined #openstack-infra | 15:59 | |
| mgoddard | AJaeger: will do | 15:59 |
| openstackgerrit | Merged openstack/project-config master: Make whitebox use Storyboard https://review.opendev.org/651942 | 15:59 |
| mgoddard | Thanks everyone :) | 15:59 |
| fungi | apparently psycopg2 publishes a separate psycopg2-binary package to pypi which provides manylinux1 wheels: https://pypi.org/project/psycopg2-binary/#files | 16:02 |
| fungi | that's kinda a strange way to go about it | 16:02 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Use latest bazel https://review.opendev.org/682363 | 16:02 |
| AJaeger | infra-root, config-core, can we publish our documents always? That simplifies our jobs: https://review.opendev.org/682222 - most repos already publish after each merge, those are the stray ones. | 16:06 |
| clarkb | mnaser: I mentioned the double accounting of cinder quotas for our boot from volume instances in sjc1 to the cinder team last week and they seemed interested in us filing a bug if more info could be provided | 16:06 |
| clarkb | mnaser: unfortunately we don't relaly have the info necessary to do that but I'm happy to help in any way I can if you are willing to look at that and put some info together for them | 16:07 |
| clarkb | AJaeger: looking | 16:07 |
| AJaeger | config-core, a trivial change to update the Zuul grafana board to add promote jobs: https://review.opendev.org/#/c/682225/ | 16:08 |
| *** armax has quit IRC | 16:08 | |
| AJaeger | infra-root, for ansible-role-cloud-launcher, we first need to silence ansible-lint, please review https://review.opendev.org/682394 | 16:09 |
| AJaeger | thanks, clarkb ! | 16:09 |
| *** e0ne has joined #openstack-infra | 16:11 | |
| *** iurygregory has quit IRC | 16:12 | |
| *** rcosnita_ has quit IRC | 16:12 | |
| ianychoi | AJaeger, thank you for dealing with wrong permission on project-creation for normal users on Zanata, and disabling rocky translation! | 16:13 |
| AJaeger | ianychoi: you're welcome! | 16:13 |
| ianychoi | :) | 16:13 |
| *** markvoelker has joined #openstack-infra | 16:13 | |
| AJaeger | ianychoi: I also created manila-ui train version. That was the first one created. Will you take care of rest? | 16:13 |
| *** larainema has quit IRC | 16:14 | |
| *** lpetrut has quit IRC | 16:16 | |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Auto-delete expired autohold requests https://review.opendev.org/663762 | 16:16 |
| *** lpetrut has joined #openstack-infra | 16:16 | |
| ianychoi | AJaeger, aha on Zanata... I see! And I think I can followup tonight (I am traveling in US now - so maybe 12-13 hours later) | 16:17 |
| *** slaweq has quit IRC | 16:17 | |
| *** slaweq has joined #openstack-infra | 16:17 | |
| AJaeger | ianychoi: no urgency, manila-ui is so far the only repo that is translated and has train branch. The rest can come anytime... | 16:18 |
| AJaeger | ianychoi: safe travels | 16:18 |
| *** markvoelker has quit IRC | 16:18 | |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Mark nodes as USED when deleting autohold https://review.opendev.org/664060 | 16:18 |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Add autohold delete/info commands to web API https://review.opendev.org/679057 | 16:18 |
| *** kopecmartin is now known as kopecmartin|off | 16:18 | |
| *** e0ne has quit IRC | 16:18 | |
| openstackgerrit | Merged openstack/project-config master: Always publish infra documents https://review.opendev.org/682222 | 16:19 |
| ianychoi | AJaeger, thank you :) | 16:19 |
| *** e0ne has joined #openstack-infra | 16:19 | |
| *** e0ne has quit IRC | 16:19 | |
| openstackgerrit | Merged opendev/ansible-role-cloud-launcher master: Ignore ansible-lint errors https://review.opendev.org/682394 | 16:19 |
| *** e0ne has joined #openstack-infra | 16:20 | |
| AJaeger | config-core, now we can merge https://review.opendev.org/682223 - please review. | 16:20 |
| openstackgerrit | Merged openstack/project-config master: Add promote pipeline to grafana https://review.opendev.org/682225 | 16:21 |
| *** e0ne has quit IRC | 16:24 | |
| *** ykarel is now known as ykarel|away | 16:24 | |
| *** ccamacho has quit IRC | 16:27 | |
| *** goldyfruit_ has joined #openstack-infra | 16:28 | |
| *** e0ne has joined #openstack-infra | 16:29 | |
| *** e0ne has quit IRC | 16:29 | |
| *** jcoufal has joined #openstack-infra | 16:30 | |
| *** lpetrut has quit IRC | 16:31 | |
| *** efried has joined #openstack-infra | 16:31 | |
| *** goldyfruit___ has quit IRC | 16:31 | |
| efried | o/ | 16:31 |
| efried | This is probably the wrong place to ask this question, but... | 16:31 |
| efried | When do we fork ussuri? | 16:31 |
| *** pcrews has joined #openstack-infra | 16:31 | |
| *** ociuhandu has quit IRC | 16:31 | |
| *** udesale has quit IRC | 16:32 | |
| clarkb | efried: smcginnis can likely ansewr that for you (release team does that branching) | 16:33 |
| cmurphy | efried: rc1 if iirc from last cycle | 16:33 |
| efried | #openstack-releases, yeah, that would have made sense as a place to ask <rolls eyes>. Thanks clarkb cmurphy. | 16:33 |
| *** ykarel|away has quit IRC | 16:34 | |
| smcginnis | efried: What cmurphy said. | 16:34 |
| efried | #openstack-release, rather. I get that wrong EVERY SINGLE TIME. | 16:34 |
| efried | thanks smcginnis | 16:34 |
| smcginnis | efried: I always want to put an "s" at the end too. ;) | 16:34 |
| cmurphy | #openstack-doc is my kryptonite on that one | 16:35 |
| *** markvoelker has joined #openstack-infra | 16:36 | |
| fungi | that's the channel where you can ask the openstack doctor | 16:37 |
| fungi | "the openstack doc is IN" | 16:37 |
| cmurphy | well now i'm never going to forget it | 16:38 |
| *** gfidente has quit IRC | 16:39 | |
| AJaeger | even more fun: The doc mailing list was openstack-docs, the channel is #openstack-doc - confused me quite a lot ;( | 16:40 |
| *** markvoelker has quit IRC | 16:40 | |
| AJaeger | fungi, time for a small cleanup? removing an unused template: https://review.opendev.org/682223 | 16:41 |
| smcginnis | Anyone know if there is any kind of gerrit API I could use to be able to add a PTL and release liaison automatically to a review? | 16:41 |
| fungi | AJaeger: yep, looks safe, thanks! | 16:42 |
| fungi | smcginnis: there is indeed. more than one in fact | 16:42 |
| fungi | you can add requested reviewers via both the http and ssh apis | 16:43 |
| fungi | which would you prefer to use? | 16:43 |
| fungi | i can dig up docs | 16:43 |
| gouthamr | "AJaeger ianychoi: no urgency, manila-ui is so far the only repo that is translated and has train branch. The rest can come anytime..." | 16:43 |
| *** VedaAnnayappa has quit IRC | 16:43 | |
| openstackgerrit | Merged opendev/ansible-role-cloud-launcher master: Fix .gitreview after rename https://review.opendev.org/682372 | 16:43 |
| smcginnis | Not sure yet. If you can point me at some docs, that would be great. If I get some time, I might see if I can play around with it and come up wiht a script we can use for releases. | 16:43 |
| gouthamr | ^ sorry for the paste, and the alert AJaeger ianychoi :/ was wondering if there's anything i/we can do to fix | 16:44 |
| *** rlandy is now known as rlandy|brb | 16:44 | |
| *** ociuhandu has joined #openstack-infra | 16:45 | |
| paladox | mordred i wrote these drafts emails for our list https://phabricator.wikimedia.org/P9110 and https://phabricator.wikimedia.org/P9108 (which i've just sent the first one just now) | 16:45 |
| fungi | smcginnis: https://review.opendev.org/Documentation/rest-api-changes.html#add-reviewer | 16:47 |
| AJaeger | gouthamr: all is fine! You were just the first to get a stable/train branch and thus a stable-train translation branch created :) | 16:48 |
| smcginnis | fungi: Thanks! | 16:48 |
| AJaeger | gouthamr: we need to manually create branches for translation - so additional repos will be done later | 16:48 |
| fungi | smcginnis: the other option is https://review.opendev.org/Documentation/cmd-set-reviewers.html | 16:48 |
| clarkb | smcginnis: fungi: worth noting that a lot of people ignore the emails that that results in | 16:49 |
| clarkb | (like me) | 16:49 |
| gouthamr | ah thank you for the explanation AJaeger! | 16:49 |
| *** sshnaidm|rover is now known as sshnaidm|afk | 16:49 | |
| openstackgerrit | Merged openstack/openstack-zuul-jobs master: Remove release-tox-docs-infra https://review.opendev.org/682223 | 16:49 |
| clarkb | gerrit email volume is just too great to use that as an organization method | 16:49 |
| smcginnis | clarkb: Yeah, I know. But it's been moderately successful for some proposed releases once we've manually added them. | 16:50 |
| *** ociuhandu has quit IRC | 16:50 | |
| AJaeger | clarkb, mordred, corvus: shall we merge https://review.opendev.org/674334 - or do you have an idea how to test it? | 16:50 |
| clarkb | AJaeger: shrews pointed out an example test that we could copy and modify to test this role | 16:50 |
| clarkb | I forget which one it was though | 16:51 |
| clarkb | Shrews: ^ do you still remember? | 16:51 |
| clarkb | nova is the only project that is still reindexing | 16:51 |
| Shrews | clarkb: AJaeger: i think i suggested fetch-output, but clarkb mentioned having to integrate subunit, iirc | 16:51 |
| Shrews | i don't know enough about subunit to offer more suggestion beyond that | 16:52 |
| clarkb | ya would need a couple real subunit file examples to merge | 16:52 |
| clarkb | can copy them out of any job | 16:52 |
| AJaeger | Shrews: did you left that comment in review? anybody wants to help tosky with it? | 16:53 |
| clarkb | corvus: AJaeger https://review.opendev.org/#/c/682091/ is another one. I'm happy to approve that as is and monitor if others are or am open to testing ideas | 16:54 |
| clarkb | and nova is done reindexing. I think that was the last thing on our end | 16:55 |
| Shrews | AJaeger: i left another comment | 16:55 |
| fungi | clarkb: smcginnis: yeah, it also shows them on some of the gerrit dashboard views though (i don't see those either though) | 16:55 |
| AJaeger | thanks, Shrews | 16:56 |
| AJaeger | clarkb: I'm fine merging if it gets monitored | 16:56 |
| corvus | clarkb: +3 | 16:57 |
| fungi | clarkb: smcginnis: i happen to have a fairly complex gerrit e-mail filter set up on my last hop mta to filter out everything except comments and votes from human reviewers, and that's been a useful compromise for me | 16:57 |
| corvus | clarkb: i think merge and monitor is really the only way to go with that | 16:57 |
| clarkb | corvus: AJaeger thanks. I am around to help keep an eye on it too | 16:57 |
| *** jpena is now known as jpena|off | 16:58 | |
| fungi | clarkb: smcginnis: so i don't see the "so-and-so has added you as a reviewer" messages but i do see comments on those changes from other reviewers after i get added | 16:58 |
| clarkb | fungi: ah | 16:58 |
| AJaeger | fungi: clever | 16:58 |
| fungi | and depending on the content of the comments, i can choose to jump into a review discussion | 16:59 |
| *** rpittau is now known as rpittau|afk | 16:59 | |
| fungi | more often than not i quickly skim them when checking my inboxes | 16:59 |
| AJaeger | config-core, please review these changes to convert some more jobs to promote, it will make moving from static to AFS easier: https://review.opendev.org/#/q/status:open++topic:promote-project-config | 16:59 |
| *** derekh has quit IRC | 17:00 | |
| openstackgerrit | Amy Marrich (spotz) proposed opendev/system-config master: Retire the Women-of-OpenStack list https://review.opendev.org/682417 | 17:00 |
| AJaeger | corvus: are you ready for the next steps on https://review.opendev.org/#/q/status:open+topic:prepare-workspace-git - then let's ask for review of https://review.opendev.org/680703 | 17:01 |
| *** slaweq has quit IRC | 17:01 | |
| efried | um, help. I have gerrit doing a really weird thing. | 17:01 |
| efried | https://review.opendev.org/#/c/681060/ | 17:01 |
| efried | a couple hours ago, stephenfin posted new patch sets for the top four patches of this series ^. Only the first of those four (https://review.opendev.org/#/c/671801/51) had an actual delta; the other three were rebases accordingly. | 17:02 |
| efried | I approved the changed one and re+W'd the rebased ones. | 17:02 |
| efried | But now two of the rebased ones are showing up with yellow dots in "Related Changes" on the right. | 17:03 |
| fungi | efried: it claims to have abandoned parents | 17:03 |
| *** jamesmcarthur has quit IRC | 17:03 | |
| fungi | 681060 says its parent commit is 3e8758d254a52dd8bf4f07ff2d64f5e9a69dc919 | 17:04 |
| AJaeger | efried: I guess that some other changes merged as well and this might get a conflict | 17:04 |
| *** kjackal has quit IRC | 17:04 | |
| efried | which is what https://review.opendev.org/#/c/671801/51 is at, no?? | 17:04 |
| efried | fungi: ^ | 17:04 |
| fungi | https://review.opendev.org/#/c/681445/1 | 17:04 |
| fungi | ahh, no | 17:04 |
| fungi | i was looking at the wrong change in the series | 17:05 |
| fungi | you're right | 17:05 |
| fungi | the yellow dots say "not current" | 17:05 |
| clarkb | if I had to guess it has to do with reindexing happening when those changes were being made | 17:05 |
| clarkb | I would ignore it if the change is queued in the gate | 17:05 |
| efried | right, and to make current you can do one of several things, all of which I've done (including using a different browser) and the yellow dot is still showing up. | 17:06 |
| efried | afaik you also can't have +W on a not-current change normally, rite? | 17:06 |
| fungi | efried: oh! yep, i bet clarkb is correct | 17:06 |
| efried | clarkb: well, except what I was actually trying to do was rebase https://review.opendev.org/#/c/681397/10 | 17:06 |
| fungi | nova takes longer to reindex than all our other repos | 17:06 |
| efried | but it thinks it's already based on latest | 17:06 |
| clarkb | efried: rebase manually then? | 17:06 |
| fungi | it's probably still reindexing after the maintenance | 17:06 |
| clarkb | fungi: its done now but was definitely in progress when nova was making changes to that stack | 17:07 |
| efried | mm, /me tries... | 17:07 |
| fungi | aha, okay | 17:07 |
| efried | can't imagine why a 30-odd patch series would cause anybody indexing confusion :P | 17:07 |
| clarkb | I wonder if we lose indexing events or if reindexing supercedes previously processed events | 17:07 |
| efried | so gerrit thinks https://review.opendev.org/#/c/674895/42 is "current" (according to dots), even though there's a 43 in existence. | 17:08 |
| efried | 43 has the yellow dot... and the +2+W | 17:08 |
| clarkb | efried: yes likely because we had to reindex everything after renaming projects this morning | 17:08 |
| *** slaweq has joined #openstack-infra | 17:08 | |
| fungi | speaking of, if the reindexing is complete then we're likely done with the maintenance so i'll close out the screen session and remove myself from the project bootstrappers group | 17:08 |
| clarkb | fungi: ++ | 17:09 |
| fungi | and done | 17:09 |
| efried | manual rebase isn't working either | 17:09 |
| clarkb | it is possible that retriggering the reindexing would fix it but potentially cause the problem to happen to other changes, but I think that this only affects rendering | 17:09 |
| *** e0ne has joined #openstack-infra | 17:09 | |
| clarkb | efried: how does that fail? | 17:09 |
| openstackgerrit | Merged zuul/zuul-jobs master: Retry container gets in upload-logs-swift https://review.opendev.org/682091 | 17:09 |
| efried | "Current branch review/luyao_zhong/bp/virtual-persistent-memory is up to date." | 17:10 |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Remove outdated TODO https://review.opendev.org/682421 | 17:10 |
| clarkb | efried: what command did you run? | 17:10 |
| efried | I did | 17:10 |
| efried | git review -d <messed-up top> | 17:10 |
| efried | git branch -m top | 17:10 |
| efried | git review -d <one I want to rebase> | 17:10 |
| efried | git rebase top | 17:10 |
| efried | and got that output | 17:10 |
| efried | messed up top: 674895 | 17:11 |
| efried | one I want to rebase: 681397 | 17:11 |
| tosky | re https://review.opendev.org/#/c/674334/ and the comments, which test does execute fetch-output.yaml? | 17:11 |
| efried | I don't want to push it, in case I wind up f'ing up the whole series and kicking out all my hard-earned successful checks | 17:11 |
| openstackgerrit | Amy Marrich (spotz) proposed opendev/irc-meetings master: Update UC meeting time and day https://review.opendev.org/682422 | 17:12 |
| clarkb | efried: I think git review is pull 674895,42 instead of 674895,43 | 17:13 |
| *** e0ne has quit IRC | 17:13 | |
| efried | clarkb: yes, I just checked that, you're right. | 17:13 |
| efried | okay, trying again with explicit psnums | 17:13 |
| *** mattw4 has joined #openstack-infra | 17:14 | |
| AJaeger | tosky: I think https://opendev.org/zuul/zuul-jobs/src/branch/master/zuul-tests.d/general-roles-jobs.yaml#L98 - but let me double check | 17:14 |
| *** dtantsur is now known as dtantsur|afk | 17:14 | |
| clarkb | efried: ya if I manually pull with git fetch (from the download drop down) and rebase that it works | 17:14 |
| AJaeger | tosky: yes, that looks like the test | 17:15 |
| efried | clarkb: Works... and gives a merge conflict? | 17:15 |
| clarkb | one sec I may have got things mixed up still | 17:15 |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Retire elections-committee, openstack-content MLs https://review.opendev.org/682425 | 17:15 |
| *** ralonsoh has quit IRC | 17:16 | |
| *** e0ne has joined #openstack-infra | 17:16 | |
| clarkb | efried: yes it conflicts | 17:16 |
| *** xek_ has joined #openstack-infra | 17:16 | |
| efried | okay, maybe that's legit then. Ima check all the commit hashes to be sure, then see about resolving those... | 17:17 |
| *** ramishra has quit IRC | 17:17 | |
| efried | thanks for the help | 17:17 |
| clarkb | conflicts in the unitests | 17:17 |
| *** VedaAnnayappa has joined #openstack-infra | 17:18 | |
| VedaAnnayappa | Hi Again | 17:18 |
| *** xek has quit IRC | 17:18 | |
| VedaAnnayappa | I see that the py37 test if timing out | 17:18 |
| VedaAnnayappa | https://review.opendev.org/#/c/678482/10..11 | 17:19 |
| VedaAnnayappa | can some one help in this regards | 17:19 |
| AJaeger | VedaAnnayappa: what is the log file saying? Are all packages now available? | 17:19 |
| tosky | AJaeger: but shouldn't have it run in https://review.opendev.org/#/c/674334/ , or was it added later, or should it be explicitly added when it covers a certain role? | 17:20 |
| clarkb | VedaAnnayappa: AJaeger my guess is that a test is crashing and causing the test suite to not exit | 17:21 |
| *** Goneri has quit IRC | 17:21 | |
| clarkb | VedaAnnayappa: most openstack projects use a timeout fixture to timeout individual tests you might start iwth ensuring that is in place to try and catch the problem test cases | 17:21 |
| *** e0ne has quit IRC | 17:21 | |
| AJaeger | tosky: Shrews suggested to add a test similar to that one that is then run as part 674334. | 17:22 |
| * AJaeger needs to step out for a bit | 17:22 | |
| *** psachin has quit IRC | 17:25 | |
| *** weshay has quit IRC | 17:25 | |
| VedaAnnayappa | Ajaeger, All the tests are passing as per log file | 17:27 |
| *** jamesmcarthur has joined #openstack-infra | 17:27 | |
| clarkb | VedaAnnayappa: those testnames are printed on completion of the test | 17:28 |
| clarkb | so fi there is anothe rtest case that causes problems it won't be printed | 17:28 |
| *** goldyfruit_ has quit IRC | 17:28 | |
| clarkb | which is why I suggseted double checking on the test timeout fixture to try and identify one that may cause that | 17:28 |
| *** ykarel|away has joined #openstack-infra | 17:29 | |
| *** rlandy|brb is now known as rlandy | 17:29 | |
| fungi | yeah, looks like something hung after https://zuul.opendev.org/t/openstack/build/c8f500d7d3204b2e9bd230cbded4b082/log/job-output.txt#1068 | 17:38 |
| fungi | there's a 30-minute gap before RUN END RESULT_TIMED_OUT: [untrusted : opendev.org/zuul/zuul-jobs/playbooks/tox/run.yaml@master] | 17:38 |
| fungi | odds are one of the tests deadlocks under python 3.7, and the easiest way to find it will be to run tox locally with python 3.7 and bisect the test set | 17:41 |
| fungi | might be nice if stestr trapped sigterm et al and spewed the list of still-running tests to stderr or something | 17:42 |
| fungi | tristanC: this is not urgent, but i noticed that since you originally created the #openstack-election irc channel on freenode, you're the only one with administrative control over it. when you get time, can i trouble you to run this? `/msg chanserv access #openstack-election add openstackinfra +AFRefiorstv` | 17:47 |
| *** lajoskatona has joined #openstack-infra | 17:47 | |
| tristanC | fungi: sure, ChanServ(ChanServ@services.)- Flags +AFRefiorstv were set on openstackinfra in #openstack-election. | 17:49 |
| *** VedaAnnayappa has quit IRC | 17:49 | |
| fungi | tristanC: thanks a bunch!!! | 17:50 |
| openstackgerrit | Eric Fried proposed opendev/elastic-recheck master: Add query for test_fail_set_az NVH bug 1844174 https://review.opendev.org/682439 | 17:52 |
| openstack | bug 1844174 in OpenStack Compute (nova) "test_fail_set_az fails intermittently with "AssertionError: OpenStackApiException not raised by _set_az_aggregate"" [Undecided,New] https://launchpad.net/bugs/1844174 | 17:52 |
| *** ccamacho has joined #openstack-infra | 17:53 | |
| openstackgerrit | Jeremy Stanley proposed openstack/project-config master: Report openstack/election in #openstack-election https://review.opendev.org/682440 | 17:55 |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Log and report infra status in #openstack-election https://review.opendev.org/682442 | 17:58 |
| *** Goneri has joined #openstack-infra | 18:02 | |
| chandankumar | so the wait is over, centos8 is coming on 24th Sept https://twitter.com/CentOSProject/status/1173652996305170432 | 18:03 |
| clarkb | there have been no post failure result jobs in the openstack tenant since the change to swift upload logs merged | 18:03 |
| clarkb | (as a sanity check on that change) | 18:03 |
| clarkb | chandankumar: cool | 18:03 |
| * chandankumar think we can add now centos8 as a part of centos8 as a supported distros for openstack. | 18:04 | |
| *** goldyfruit has joined #openstack-infra | 18:05 | |
| fungi | chandankumar: that *may* be the 7.7 release they've been working on | 18:05 |
| fungi | they said that's a higher priority than finishing 8.0 | 18:05 |
| clarkb | re asserting centos8 support I'd personally like to see working test jobs on it before we say it is supported | 18:06 |
| fungi | yeah, the idea is that if we have usable images for it at the start of the openstack ussuri release cycle then official openstack projects will be told to get it working for them | 18:06 |
| clarkb | in particular the python version shuffle may cause some hiccups that we have to sort through | 18:07 |
| fungi | not that we'd assert it's supported, but that we'd assert that it's a target for supporting by the ussuri release | 18:07 |
| clarkb | gotcha | 18:07 |
| fungi | in theory half a year should be sufficient to work out the kinks | 18:07 |
| chandankumar | I have added a comment here https://review.opendev.org/#/c/681242/2 may be we can reconsider our decision | 18:08 |
| clarkb | fungi: if we get images built in that time frame too | 18:08 |
| fungi | if not, hopefully openstack can figure that out fairly quickly | 18:08 |
| fungi | yup | 18:08 |
| clarkb | (fedora 30 took a while) | 18:08 |
| *** jamesmcarthur has quit IRC | 18:08 | |
| fungi | but the other twist is that with python 3 being included directly in 7.7 (supposedly), openstack could still theoretically keep centos as a supported distro while continuing to drop its python 2.7 requirement | 18:11 |
| efried | clarkb: What is this "test timeout fixture" of which you speak? | 18:16 |
| efried | (Trying to help VedaAnnayappa) | 18:16 |
| efried | ...I think I see it... | 18:18 |
| zbr | fungi: clarkb: i would be bit more reserved regarding usability of py3 from centos7.7, read https://bugs.centos.org/view.php?id=16389 | 18:19 |
| fungi | efried: https://opendev.org/openstack/nova/src/branch/master/nova/tests/fixtures.py#L233-L259 | 18:20 |
| fungi | it's passed into test setup | 18:20 |
| zbr | but i will do my best to add centros8 jobs once we get access to it | 18:20 |
| efried | fungi: ack, thx | 18:20 |
| fungi | zbr: yeah, hard to really know until we have 7.7 and/or 8.0 how viable it will be. there's every chance openstack will have to drop centos "support" officially for the ussuri release and add it again in v | 18:21 |
| jrosser | openstack-ansible fails hard on centos7.7 - I tried that, and the missing selinux library is a showstopper | 18:21 |
| zbr | what i do not like is that I do no see many others commenting on https://bugs.centos.org/view.php?id=16389 -- i am sure that with a bit of peer pressure we can get the libselinux compiled for it. | 18:23 |
| zbr | if I am not wrong it involved uncommenting one line | 18:23 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Use latest bazel https://review.opendev.org/682363 | 18:24 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Fix files matcher for gerrit base image https://review.opendev.org/682449 | 18:24 |
| mordred | corvus: ^^ enjoy that | 18:24 |
| corvus | mordred: can you touch something in that directory so we get a build with that too? | 18:25 |
| efried | fungi, clarkb: nova-powervm tests appear to be based on nova's NoDBTestCase, which is setting up that timeout fixture. I did some pawing around, and it appears as though 309 out of 400 test cases are completing. I noticed on the last run that all 400 tests in py36 completed in under a minute, whereas the 309 tests stopped talking after just over three minutes in py37. Could this be hitting an overall test suite timeout? | 18:25 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: zk: use pathlib.Path to define ZooKeeper paths https://review.opendev.org/682450 | 18:25 |
| *** jamesmcarthur has joined #openstack-infra | 18:25 | |
| *** jamesmcarthur has quit IRC | 18:25 | |
| openstackgerrit | Merged openstack/project-config master: Report openstack/election in #openstack-election https://review.opendev.org/682440 | 18:27 |
| fungi | zbr: chandankumar: looks like another comment was added suggesting it is 8.0 | 18:28 |
| fungi | so... fingers crossed | 18:28 |
| *** ociuhandu has joined #openstack-infra | 18:28 | |
| *** jcoufal_ has joined #openstack-infra | 18:29 | |
| fungi | efried: the log from the failing build on the linked change shows the listing of completed tests just stop for ~30 minutes, and then the configured build timeout for the job kicks in and closes it up | 18:30 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Fix files matcher for gerrit base image https://review.opendev.org/682449 | 18:30 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Use latest bazel https://review.opendev.org/682363 | 18:30 |
| mordred | corvus: yes, I have now done this | 18:30 |
| corvus | mordred: +3 (seems a trivial enough fix to single-core) | 18:30 |
| *** e0ne has joined #openstack-infra | 18:30 | |
| fungi | efried: so it seems like something is hanging while tests are being iterated over | 18:30 |
| efried | fungi: ah, I see that now. | 18:31 |
| efried | okay, perhaps time to poke at some of these deprecation warnings that are showing up... | 18:31 |
| fungi | so we're trying to infer the cause from the silence there | 18:31 |
| sean-k-mooney | i have a fix for one of them locally i think | 18:31 |
| efried | yeesh, those look pretty harmless | 18:32 |
| *** jcoufal has quit IRC | 18:32 | |
| sean-k-mooney | the sqlalchem migrate one | 18:32 |
| fungi | efried: but yes, trying to recreate that behavior locally is what i would suggest next | 18:32 |
| efried | well... that's totally reasonable | 18:32 |
| zbr | fungi: based on all the work we did with fedora28 and rhel8, my hopes are high regarding how well centos-8 will work. | 18:32 |
| sean-k-mooney | this https://bugs.launchpad.net/sqlalchemy-migrate/+bug/1814288 | 18:32 |
| openstack | Launchpad bug 1814288 in sqlalchemy-migrate "DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead" [Medium,Confirmed] | 18:32 |
| fungi | you might at least be able to bisect the test set to narrow it down in only a handful of steps | 18:32 |
| efried | gotta find me a py37 interpreter... | 18:33 |
| mordred | efried: pyenv is your friend | 18:33 |
| fungi | efried: ubuntu bionic (18.04 lts) has one packaged and is what we use in zuul builds | 18:33 |
| sean-k-mooney | i think ubunut has py37. my version does anyway but i choudl have got that form source | 18:33 |
| mordred | or that | 18:33 |
| mordred | or docker | 18:33 |
| fungi | but yeah, i just compile python 3.7 (and lots of others) locally from source and maintain separate trees for them with make altinstall | 18:34 |
| mordred | docker run -it --rm python:3.7 /bin/bash | 18:34 |
| Shrews | fungi: that's hard core man | 18:34 |
| sean-k-mooney | there is a tool for it too pyenv or soemthing that automates it for your | 18:34 |
| efried | I have one. | 18:34 |
| mordred | sean-k-mooney: pyenv, yeah | 18:34 |
| mordred | that's what I use | 18:34 |
| fungi | Shrews: it's really not. it's ~4 commands to build from a git tag in the python source repo | 18:34 |
| corvus | fungi: yes, but if i can't "curl | sudo bash" it i'm not interested | 18:35 |
| * mordred is very pleased with pyenv- specifically that he can set a default python version for an arbitrary directory - but also likes fungi's method | 18:35 | |
| ullbeking | good evening all. I've been scrolling through my backlog of this channel, and I noticed the following comment by efried: | 18:35 |
| *** ociuhandu has quit IRC | 18:36 | |
| ullbeking | 19:25 <efried> fungi, clarkb: nova-powervm tests appear to be based on nova's NoDBTestCase, which is setting up that timeout fixture [...] | 18:36 |
| ullbeking | I'm _completely_ out of my depth here and a complete newbie | 18:36 |
| sean-k-mooney | mordred: there is some tox integration with pyenv too but i have not used really to know if it works well or not | 18:36 |
| ullbeking | but I AM interesting in POWER | 18:36 |
| efried | ullbeking: powervm specifically or ppc64le in general? | 18:36 |
| ullbeking | so my question is how much of the backlog concerns OpenStack on POWER? | 18:37 |
| *** markvoelker has joined #openstack-infra | 18:37 | |
| fungi | Shrews: the source code is https://github.com/python/cpython and i simply checkout the release i want and basically `./configure --enable-optimizations --prefix /home/fungi/lib/cpython/$TAG; make profile-opt; make test; make altinstall; ln -sf ../lib/cpython/$TAG/bin/python3.7 ~/bin/` | 18:37 |
| efried | ullbeking: come join me in #openstack-powervm and we can dig in a bit. | 18:37 |
| mordred | sean-k-mooney: same | 18:37 |
| ullbeking | efried: I'm currently running Debian ppc64le on POWER9 but I am also still interested in powerpc generally, which is why I still hack on/with G4 and G5 Apple computers | 18:38 |
| Shrews | fungi: that meets my definition of "hard core" | 18:38 |
| fungi | mordred: yeah, i'm willing to bet pyenv does basically the same thing i do | 18:38 |
| sean-k-mooney | mordred: i orginally install it so i could run all the env people were using in there repos when i was trying to use centos | 18:38 |
| mordred | fungi: please tell me you type that from memory every time | 18:38 |
| ullbeking | 19:37 <efried> ullbeking: come join me in #openstack-powervm and we can dig in a bit. | 18:38 |
| ullbeking | cool, thanks! now that is a channel that sounds like I should be in :D | 18:38 |
| mordred | fungi: yeah. I think it's pretty much teh same | 18:38 |
| fungi | mordred: i have a script, hence the $TAG | 18:38 |
| fungi | the options to pass vary a little depending on the python version, but they've been mostly stable since 3.4 or 3.5 | 18:39 |
| fungi | i think i also stopped being able to reliably build 3.4.x on debian/sid | 18:39 |
| mordred | fungi: the main difference is it installs a shims called "python" and "python3.6" etc in $PATH that then exec the right linked version given preferences | 18:39 |
| fungi | because of depending on a too-old libssl | 18:39 |
| sean-k-mooney | fungi: pyenve allows you to have a . file in your git repo to automaticaly change your interpreter in that repo too | 18:40 |
| sean-k-mooney | if you have the tox plugin tox can then use that file to choees which version of py3 for example to use | 18:40 |
| sean-k-mooney | that said i only spend like 15mins playing with it so i havent used that enough to know if it works well or not | 18:41 |
| fungi | sean-k-mooney: sounds neat, though i generally rely on tox to just declare minor python versions it cares about and i only keep one build of each minor version around because they take up a fair amount of disk | 18:41 |
| fungi | (some commands i elided include blowing away the older trees of the same minor version before the altinstall) | 18:42 |
| clarkb | I'm on tumbleweed where curses weirdness makes python sad. Its so weird to me that linux distros don't push their fixes for that upstream when upstream carries stuff for all the random platforms that are not linux | 18:42 |
| *** markvoelker has quit IRC | 18:42 | |
| clarkb | I just use docker images if I need a specific python anymore | 18:42 |
| *** xek has joined #openstack-infra | 18:42 | |
| sean-k-mooney | clarkb: ya i have done that alot too its my normal go to | 18:42 |
| fungi | yeah, that's one of the reasons i personally test against builds of vanilla python source instead of distro packaged interpreters | 18:43 |
| sean-k-mooney | but basically i started using pyenv to get a moderen python on centos7 | 18:43 |
| fungi | so i can work out if a problem is stemming from (or fixed by) a distro patch | 18:43 |
| fungi | sean-k-mooney: hah, i started compiling mine for the opposite version. wanting to have old versions of python to test on debian's unstable version | 18:44 |
| fungi | er, opposite reason | 18:44 |
| sean-k-mooney | well i jsut move my devstackenvs to ubuntu becaue devstack + distro python 3 on centos was sad | 18:45 |
| *** xek_ has quit IRC | 18:45 | |
| fungi | makes sense | 18:45 |
| sean-k-mooney | by the way any update on centos 8 vs 7.7 for Ussuri? | 18:45 |
| fungi | here's hoping centos 7.7 and/or 8.0 solve that | 18:45 |
| sean-k-mooney | i think centos8 is ment to be out in about 2 weeks | 18:46 |
| fungi | sean-k-mooney: that's sort of where this tangent started | 18:46 |
| fungi | https://twitter.com/CentOSProject/status/1173652996305170432 | 18:46 |
| *** ykarel|away has quit IRC | 18:46 | |
| sean-k-mooney | ah cool | 18:46 |
| fungi | sean-k-mooney: and also https://review.opendev.org/681242 | 18:46 |
| fungi | that change is an option of last resort, in case we need it | 18:47 |
| sean-k-mooney | ya well hopefully on the 24th we will have centos8 but that will be python 3 only i think | 18:48 |
| sean-k-mooney | so some things might break | 18:48 |
| sean-k-mooney | im expecting devstack to not work out of the box | 18:48 |
| fungi | well, that's fine for ussuri, openstack's claimed python 2.7 support is not important for that release | 18:48 |
| fungi | and we'll have a full release cycle to work through the challenges (hopefully) | 18:49 |
| sean-k-mooney | its more then not imporant right | 18:49 |
| sean-k-mooney | we are removing support in 2.7 | 18:49 |
| sean-k-mooney | *for 2.1 | 18:49 |
| sean-k-mooney | i cant type right now :( | 18:49 |
| zbr | sean-k-mooney: not sure who told you it will not have py27. centos8 will have py27, but not installed by default. it is still there. | 18:49 |
| fungi | sean-k-mooney: i'm not sure who "we" is but the official word for openstack is that it's okay for projects to drop support for python 2.7 for the ussuri release, not that they're required to stop testing it | 18:50 |
| fungi | openstack as a whole will no longer claim to support running with python 2.7 | 18:50 |
| fungi | some projects (i gather swift is one) will still want to test that their releases work with python 2.7 | 18:50 |
| sean-k-mooney | well i know several nova devs including me want to start removing some of the 2.7 support | 18:51 |
| fungi | that's fine | 18:51 |
| fungi | it's officially allowed | 18:51 |
| sean-k-mooney | that said we are not going to do it as lets just remove this because its cool | 18:51 |
| sean-k-mooney | we are just going to work in removal of 2.7 special casing into or normal work | 18:52 |
| *** openstackgerrit has quit IRC | 18:52 | |
| zbr | re py27 i will be very happy to spend few hours between xmas and January 1st to remove any py27 job/support from the few projects i manage. i am not planning to release any kind of py27 patch after the big day. if something breaks, they will need to fork it. | 18:53 |
| sean-k-mooney | zbr: well on RHEL8 there is no defualt python binary and python3.6 is installed by defualt and aviable as python3 | 18:53 |
| sean-k-mooney | so i assume cento8 would do the same | 18:53 |
| zbr | indeed | 18:54 |
| sean-k-mooney | technically on rhel8 you can get python2 via an appstream | 18:54 |
| sean-k-mooney | but i dont think it will be offically supported | 18:54 |
| sean-k-mooney | just avaiable if you need it for some reason | 18:54 |
| sean-k-mooney | i have personally set python to be python3 | 18:54 |
| sean-k-mooney | i belive arch and or debian default /usr/bin/python to link to python3 also but that is an opt in step on rhel | 18:55 |
| zbr | sean-k-mooney: I can only imagine how fun it will be for those working for JPM to see how everyone is dropping support for it right away. | 18:56 |
| sean-k-mooney | i think /usr/bin/python being python is the right thing personally but making it optional is also ok. | 18:56 |
| zbr | they could for entire pypa and keep supporting it :D | 18:56 |
| zbr | re https://news.efinancialcareers.com/uk-en/3002059/banks-python-2-vs-python-3 | 18:56 |
| sean-k-mooney | most of the compat stuff is not a problem but some of it is nasty. | 18:57 |
| sean-k-mooney | what i suspect will happen is we will rely on thing like dict being orderd or stop using six.iteritems() | 18:58 |
| sean-k-mooney | and you will get either slow behavior or non determisitc behavior as a result | 18:59 |
| *** rlandy is now known as rlandy|biab | 18:59 | |
| zbr | sean-k-mooney: yeah, but for those that did not work to migrate and hoped that it will be another cobol it could prove to be an issue once security issues are discovered in unsupported code. | 18:59 |
| *** lucasagomes has quit IRC | 18:59 | |
| zbr | sean-k-mooney: syntax/six and unicode is "easy" compared with other more subtile stuff. | 18:59 |
| sean-k-mooney | i have 0 sympaty for jpm. they are active contiuptor to c++ langage developemnt they have the capastiy to do the same for python | 19:00 |
| sean-k-mooney | zbr: right but dicts being ordered is only true form 3.6 on. so thre will be subtel bugs on py27 when we start relying on it | 19:01 |
| *** goldyfruit_ has joined #openstack-infra | 19:07 | |
| *** pkopec has quit IRC | 19:08 | |
| *** goldyfruit has quit IRC | 19:09 | |
| fungi | i expect banks relying on python 2.7 will just use older distro release series which are still covered under their vendors' support for years to come, or contract third parties who are willing to provide that support for a fee. this is not all that different from companies still relying on other aging technologies. it's quite common in fact | 19:13 |
| clarkb | well and rhel8 has python2 in it | 19:14 |
| clarkb | they are good for another 10 years or so at least | 19:14 |
| clarkb | bionic too | 19:14 |
| clarkb | but bionic is a year older so will eol python2 there a year before rhel8 | 19:14 |
| fungi | yep. much ado about nothing | 19:15 |
| fungi | but it makes for great clickbait headlines, apparently | 19:16 |
| *** openstackgerrit has joined #openstack-infra | 19:18 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: zk: use pathlib.Path to define ZooKeeper paths https://review.opendev.org/682450 | 19:18 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: Store a list of held node per held build in hold request https://review.opendev.org/682466 | 19:18 |
| fungi | even debian's python maintainers haven't decided for certain whether or not they'll continue to ship a python 2.7 interpreter for their next major release, which is roughly two years away | 19:19 |
| mordred | corvus: sigh. I need to squash those - because the bazel version is needed to fix the image builds | 19:19 |
| *** e0ne has quit IRC | 19:20 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Fix files matcher and bazel for gerrit base image https://review.opendev.org/682449 | 19:20 |
| corvus | mordred: oh yeah we should have seen that coming | 19:22 |
| corvus | fungi, clarkb: can you look at https://review.opendev.org/682449 ? (i think that needs another core now) | 19:22 |
| clarkb | looking | 19:23 |
| corvus | mordred: cool, your change landed in 2.14, so we're just waiting for the merge train to leave the station | 19:24 |
| paladox | and it's now in 2.15 corvus :) | 19:24 |
| corvus | woo woo! | 19:24 |
| paladox | https://gerrit-review.googlesource.com/c/gerrit/+/237592 | 19:24 |
| clarkb | as a heads up I may have to take the kids to swim class today at 3ish local time | 19:24 |
| mordred | paladox: woot! | 19:24 |
| * mordred will go ahead and make a patch to remove the hack | 19:24 | |
| paladox | :) | 19:25 |
| *** dciabrin has quit IRC | 19:25 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Remove bazel version hack https://review.opendev.org/682469 | 19:26 |
| corvus | mordred: should be a self-testing change :) | 19:26 |
| mordred | corvus: yah! in fact - I currently expect it to pass for 2.15 and fail for 2.16 and 3.0 - assuming the merge train hasn't landed there yet | 19:29 |
| efried | fungi, clarkb: interesting data point on the nova-powervm py37 timeout: (1) many tests are hitting the OS_TEST_TIMEOUT, which is configured at 60s, but (2) those tests are being marked as passing. (3) this is only happening on py37; those same tests run sub-second in py27,py36. | 19:29 |
| efried | So at this point I surmise that we've got a bundle of those tests jamming up the thread pool until eventually we hit the overall job timeout. If we waited long enough, they would pass. | 19:29 |
| *** jcoufal_ has quit IRC | 19:30 | |
| *** whoami-rajat has quit IRC | 19:30 | |
| *** ccamacho has quit IRC | 19:31 | |
| fungi | makes sense | 19:32 |
| *** jamesmcarthur has joined #openstack-infra | 19:33 | |
| *** dciabrin has joined #openstack-infra | 19:34 | |
| *** jtomasek has quit IRC | 19:34 | |
| paladox | mordred https://gerrit-review.googlesource.com/c/gerrit/+/237593 | 19:37 |
| AJaeger | mgoddard: please remember you need to backport the kaybe change to *all* branches that have a Zuul configuration. | 19:37 |
| efried | fungi: but but but... why (2)?? and why (3)?? | 19:38 |
| fungi | efried: not sure about (2) but (3) could be livelocks/deadlocks which only surface under 3.7 | 19:39 |
| efried | fungi: where can I find out more? | 19:39 |
| *** kjackal has joined #openstack-infra | 19:40 | |
| fungi | by turning on/adding more verbose debugging to the affected tests and comparing 3.6 to 3.7 | 19:40 |
| *** lajoskatona has quit IRC | 19:41 | |
| fungi | tests hitting the timeout fixture but still passing sounds bad. like either the fixture is applied/written wrong or they're exposing a bug in testtools bad | 19:41 |
| fungi | but i don't have deep knowledge there, i'm afraid, so that's pure speculation | 19:41 |
| efried | fungi: Okay, cool. I think I've probably hit my limit for expenditure of time/effort on this since nova-powervm no longer pays the bills. How "required" is py37 support for Train? Do they have to have this fixed immediately, or will "during ussuri" do? | 19:43 |
| fungi | efried: "Based on the available Python runtimes for the supported Linux distributions, all Python-based projects must target and test against, at a minimum: Python 2.7, Python 3.6, Python 3.7" https://governance.openstack.org/tc/reference/runtimes/train.html#python-runtime-for-train | 19:44 |
| efried | ack | 19:44 |
| fungi | efried: as for how much of a "must" that must is, you'd have to ask the tc | 19:45 |
| *** armstrong has joined #openstack-infra | 19:47 | |
| *** igordc has joined #openstack-infra | 19:51 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-sphinx master: Switch to zuul-release-python https://review.opendev.org/682366 | 19:52 |
| sean-k-mooney | clarkb: rhel8 has python2 via an appstream with no support from redhat | 19:52 |
| sean-k-mooney | e.g. it will not be securety patched as far as i know and any issues that result form its use will be up to the third party vender that integrated with that appstream to fix | 19:53 |
| clarkb | sean-k-mooney: ok ubuntu bionic with 10 years of support instead then :) | 19:53 |
| sean-k-mooney | or you pay for extended support | 19:54 |
| sean-k-mooney | but ya 18.04 is a good bet or stay on rhel7 | 19:55 |
| sean-k-mooney | fungi: i thnink that was a fairly strong must or they would have said should | 19:56 |
| *** rlandy|biab is now known as rlandy | 19:57 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Add autogenerated tag to Gerrit reviews https://review.opendev.org/682473 | 20:02 |
| *** jamesmcarthur has quit IRC | 20:15 | |
| *** jamesmcarthur has joined #openstack-infra | 20:16 | |
| openstackgerrit | Merged opendev/elastic-recheck master: Add query for test_fail_set_az NVH bug 1844174 https://review.opendev.org/682439 | 20:18 |
| openstack | bug 1844174 in OpenStack Compute (nova) "test_fail_set_az fails intermittently with "AssertionError: OpenStackApiException not raised by _set_az_aggregate"" [Undecided,New] https://launchpad.net/bugs/1844174 | 20:18 |
| clarkb | http://zuul.openstack.org/build/371120e96e77444fa9e2e25f5bf709ed/log/job-output.txt#1773 is the only post failure since updating the swift upload script | 20:20 |
| clarkb | that looks unrelated | 20:20 |
| clarkb | I think we are good on that change | 20:20 |
| *** jamesmcarthur has quit IRC | 20:21 | |
| jrosser | i'm having a hard time reading some log files, like this as an example https://zuul.opendev.org/t/openstack/build/6b63793de68d40bdbc6b327666dff1cd/log/logs/host/python_venv_build.log.txt.gz | 20:35 |
| jrosser | oh and of course now i paste that it loads :( | 20:35 |
| jrosser | taken a very very long time | 20:36 |
| *** ociuhandu has joined #openstack-infra | 20:36 | |
| clarkb | jrosser: can you link to the raw file location? | 20:38 |
| fungi | it's 18.2 megabytes in size according to the raw view | 20:38 |
| fungi | https://0f670d7cc05fd015a578-3a8c583e482dda2bd52f78367bc4d48a.ssl.cf2.rackcdn.com/682446/1/check/openstack-ansible-deploy-aio_metal-debian-stable/6b63793/logs/host/python_venv_build.log.txt.gz | 20:38 |
| jrosser | yes its massive | 20:38 |
| jrosser | 260k lines | 20:38 |
| *** markvoelker has joined #openstack-infra | 20:38 | |
| *** pcaruana has quit IRC | 20:38 | |
| fungi | so loading the cooked view is basically waiting for all that content to be parsed and annotated | 20:38 |
| fungi | for very large logs you're likely better off going with the raw link | 20:39 |
| jrosser | indeed - i had no idea it was quite that large, which was a bit indistiguishable from broken | 20:39 |
| fungi | since all the parsing is being done browser-side with js | 20:39 |
| clarkb | 82MB uncompressed | 20:39 |
| jrosser | i think i'm spoilt in the office with >> DSL internet | 20:40 |
| clarkb | so ya likely just the size of the file | 20:40 |
| jrosser | and same not true back at home :) | 20:40 |
| clarkb | jrosser: well browsers tend to have a hard time with files that large too | 20:40 |
| fungi | a "loading...please wait" there might be nice | 20:40 |
| clarkb | I often open log files with vim for that reason | 20:40 |
| fungi | yup, vim can take urls directly | 20:40 |
| jrosser | i also, in error, tried to wget the non raw url which also "looks like it should work" | 20:41 |
| fungi | and is excellent for discerning line numbers, filtering, searching with regular expressions, et cetera | 20:41 |
| *** ociuhandu has quit IRC | 20:41 | |
| fungi | ahh, yeah the non-raw url is going to need something which an interpret javascript | 20:42 |
| fungi | so, not wget ;) | 20:42 |
| *** markvoelker has quit IRC | 20:43 | |
| *** jamesmcarthur has joined #openstack-infra | 20:45 | |
| *** trident has quit IRC | 20:47 | |
| *** markvoelker has joined #openstack-infra | 20:49 | |
| clarkb | fungi: ++ on a "please wait while we render this log file" spinner thing | 20:51 |
| *** xek has quit IRC | 20:52 | |
| *** eharney has quit IRC | 20:53 | |
| *** trident has joined #openstack-infra | 20:59 | |
| *** markvoelker has quit IRC | 21:00 | |
| ianychoi | gouthamr, sorry I am seeing your message now but it is pretty fine - As I18n PTL I will try to ask again soon for translators to complete translation activities before your final release. | 21:01 |
| *** lpetrut has joined #openstack-infra | 21:06 | |
| *** panda|ruck has quit IRC | 21:10 | |
| *** e0ne has joined #openstack-infra | 21:12 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Use robot_comments in Gerrit https://review.opendev.org/682487 | 21:13 |
| *** lpetrut has quit IRC | 21:14 | |
| *** panda has joined #openstack-infra | 21:16 | |
| *** rkukura has joined #openstack-infra | 21:19 | |
| gouthamr | ianychoi: thank you! :) | 21:20 |
| *** e0ne has quit IRC | 21:23 | |
| *** markvoelker has joined #openstack-infra | 21:35 | |
| *** kjackal has quit IRC | 21:37 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Use robot_comments in Gerrit https://review.opendev.org/682487 | 21:37 |
| *** Sundar has joined #openstack-infra | 21:39 | |
| Sundar | I am trying to update os-acc so as to retire it. I can access https://review.opendev.org/#/q/status:open+project:openstack/os-acc but attempts to do 'git review' fail with permission error. | 21:41 |
| Sundar | $ git remote -v | 21:41 |
| Sundar | I am able to do that for other projects from the same environment. | 21:42 |
| Sundar | How do we fix the permission error? | 21:42 |
| fungi | Sundar: the project isn't marked read-only at least: https://review.opendev.org/#/admin/projects/openstack/os-acc | 21:44 |
| clarkb | Sundar: can you paste the output of git review -v ? | 21:44 |
| *** harlowja has quit IRC | 21:45 | |
| fungi | ...on http://paste.openstack.org/ and then give us the url | 21:45 |
| *** markvoelker has quit IRC | 21:45 | |
| Sundar | Now that I reviewed the paste above, I see that I spelled gerrit with two t's. Sorry for the false alarm. Tough to see that when you've been staring at it for 30 minutes | 21:47 |
| Sundar | . | 21:47 |
| *** rcernin has joined #openstack-infra | 21:47 | |
| fungi | indeed, glad you figured it out! | 21:47 |
| fungi | sometimes just starting to troubleshoot it with someone else will make things suddenly obvious which have been eluding you | 21:48 |
| fungi | sort of like telling your rubber duck | 21:48 |
| Sundar | fungi: Absolutely! Somebody else created os-acc, and I haven't been active in it, so I was inclined to believe in the permissions error. Thanks for helping me along. :) | 21:51 |
| fungi | no worries | 21:51 |
| Sundar | FWIW, I am working to remove os-acc: https://review.opendev.org/#/c/682498/ | 21:52 |
| *** slaweq has quit IRC | 21:58 | |
| *** eharney has joined #openstack-infra | 21:58 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Use robot_comments in Gerrit https://review.opendev.org/682487 | 21:59 |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Clean up logging for abandoned IRC channels https://review.opendev.org/682500 | 22:09 |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Log and report infra status in #openstack-election https://review.opendev.org/682442 | 22:11 |
| *** armstrong has quit IRC | 22:12 | |
| *** pkopec has joined #openstack-infra | 22:17 | |
| *** markvoelker has joined #openstack-infra | 22:26 | |
| *** zhurong has quit IRC | 22:29 | |
| *** pkopec has quit IRC | 22:30 | |
| *** tosky has quit IRC | 22:35 | |
| *** goldyfruit_ has quit IRC | 22:50 | |
| *** tkajinam has joined #openstack-infra | 22:54 | |
| *** mriedem has quit IRC | 23:07 | |
| *** Tengu has quit IRC | 23:11 | |
| *** Sundar has quit IRC | 23:14 | |
| *** dchen has joined #openstack-infra | 23:21 | |
| *** eharney has quit IRC | 23:25 | |
| *** olaph has joined #openstack-infra | 23:29 | |
| *** jamesmcarthur has quit IRC | 23:37 | |
| *** sthussey has quit IRC | 23:37 | |
| *** rlandy has quit IRC | 23:43 | |
| *** prometheanfire has quit IRC | 23:52 | |
| *** prometheanfire has joined #openstack-infra | 23:53 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!