| *** dpawlik has joined #openstack-infra | 00:00 | |
| ianw | ahh, yeah i wondered if that would work :) | 00:02 |
|---|---|---|
| clarkb | ianw: I expect it will work with the extra quota :) | 00:03 |
| clarkb | er quotes | 00:04 |
| *** dpawlik has quit IRC | 00:04 | |
| *** zigo has quit IRC | 00:05 | |
| *** mriedem_away has quit IRC | 00:10 | |
| *** eernst_ has joined #openstack-infra | 00:31 | |
| *** smarcet has joined #openstack-infra | 00:33 | |
| *** eernst_ has quit IRC | 00:35 | |
| *** longkb has joined #openstack-infra | 00:35 | |
| *** icey has quit IRC | 00:36 | |
| *** Tim_ok has quit IRC | 00:36 | |
| *** dayou_ has joined #openstack-infra | 00:40 | |
| *** dayou_ has quit IRC | 00:43 | |
| openstackgerrit | Matthew Thode proposed openstack-infra/glean master: ignore ip6gre when setting up interfaces https://review.openstack.org/493443 | 00:59 |
| *** hongbin has joined #openstack-infra | 01:00 | |
| *** icey has joined #openstack-infra | 01:02 | |
| openstackgerrit | Merged openstack-infra/glean master: Close file in safe_open https://review.openstack.org/585574 | 01:03 |
| *** rkukura has quit IRC | 01:09 | |
| *** mrsoul has quit IRC | 01:24 | |
| *** jamesmcarthur has joined #openstack-infra | 01:28 | |
| *** jamesmcarthur has quit IRC | 01:33 | |
| prometheanfire | hmm, dat gate time | 01:40 |
| *** smarcet has quit IRC | 01:40 | |
| *** diablo_rojo has quit IRC | 01:48 | |
| *** graphene has quit IRC | 01:55 | |
| *** roman_g has quit IRC | 01:59 | |
| *** felipemonteiro has joined #openstack-infra | 02:01 | |
| *** felipemonteiro has quit IRC | 02:02 | |
| *** harlowja has quit IRC | 02:13 | |
| *** felipemonteiro has joined #openstack-infra | 02:28 | |
| *** Bhujay has joined #openstack-infra | 02:29 | |
| *** dave-mccowan has quit IRC | 02:29 | |
| *** Bhujay has quit IRC | 02:30 | |
| *** Bhujay has joined #openstack-infra | 02:30 | |
| *** jamesmcarthur has joined #openstack-infra | 02:35 | |
| *** felipemonteiro has quit IRC | 02:43 | |
| *** psachin has joined #openstack-infra | 02:43 | |
| *** annp has quit IRC | 02:45 | |
| *** aidin has quit IRC | 02:47 | |
| *** imacdonn has quit IRC | 02:50 | |
| *** dave-mccowan has joined #openstack-infra | 02:50 | |
| *** imacdonn has joined #openstack-infra | 02:50 | |
| *** ijw has joined #openstack-infra | 03:03 | |
| *** ijw has quit IRC | 03:03 | |
| *** armax has quit IRC | 03:11 | |
| *** felipemonteiro has joined #openstack-infra | 03:11 | |
| *** annp has joined #openstack-infra | 03:13 | |
| abelur | Is Gerrit - https://review.openstack.org down? | 03:15 |
| StevenK | Loads for me, what are you seeing? | 03:16 |
| *** ramishra has joined #openstack-infra | 03:18 | |
| *** vipul has quit IRC | 03:20 | |
| *** jamesmcarthur has quit IRC | 03:23 | |
| *** harlowja has joined #openstack-infra | 03:23 | |
| *** jamesmcarthur has joined #openstack-infra | 03:24 | |
| *** jamesmcarthur has quit IRC | 03:28 | |
| *** felipemonteiro has quit IRC | 03:30 | |
| *** ykarel|away has joined #openstack-infra | 03:39 | |
| *** harlowja has quit IRC | 03:41 | |
| *** Bhujay has quit IRC | 03:41 | |
| *** ykarel|away is now known as ykarel | 03:49 | |
| *** udesale has joined #openstack-infra | 03:57 | |
| *** dpawlik has joined #openstack-infra | 04:00 | |
| *** dpawlik has quit IRC | 04:05 | |
| *** vivsoni has quit IRC | 04:08 | |
| ykarel | is this pypi push failure already known? | 04:14 |
| ykarel | which resulted in no tarball for tagged releases | 04:14 |
| ykarel | http://logs.openstack.org/a1/a14990915a4225a198d0671ba1e465b8b24eba70/release/release-openstack-python/ad30dab/job-output.txt.gz#_2018-09-24_15_03_19_301119 | 04:15 |
| ykarel | seeing too many POST_FAILURES: http://zuul.openstack.org/builds.html?job_name=release-openstack-python | 04:15 |
| ykarel | recently there are some passes so seems issue is fixed, but will the failed job will rerun? | 04:16 |
| ykarel | fungi, smcginnis any idea ^^ | 04:16 |
| prometheanfire | ykarel: missing releases are a known problem | 04:17 |
| ykarel | prometheanfire, Thanks for confirming, so the issue fixed? | 04:17 |
| prometheanfire | ykarel: I think so, re-releases are scheduled for 8-12 hours from now | 04:18 |
| *** vivsoni has joined #openstack-infra | 04:18 | |
| ykarel | prometheanfire, Thanks | 04:18 |
| prometheanfire | nova released fine at least | 04:19 |
| *** ediardo has quit IRC | 04:19 | |
| *** rcernin has quit IRC | 04:24 | |
| *** e0ne has joined #openstack-infra | 04:28 | |
| *** hongbin has quit IRC | 04:33 | |
| *** lathiat_ has quit IRC | 04:35 | |
| *** dayou has quit IRC | 04:36 | |
| *** dayou has joined #openstack-infra | 04:36 | |
| *** lathiat has joined #openstack-infra | 04:37 | |
| *** Bhujay has joined #openstack-infra | 04:37 | |
| *** rcernin has joined #openstack-infra | 04:37 | |
| AJaeger_ | infra-root, grafana has no content since 2:00, see http://grafana.openstack.org/d/T6vSHcSik/zuul-status?orgId=1 | 04:41 |
| *** yamamoto has quit IRC | 04:46 | |
| *** yamamoto has joined #openstack-infra | 04:46 | |
| *** vivsoni has quit IRC | 04:48 | |
| *** agopi_ has joined #openstack-infra | 04:50 | |
| *** e0ne has quit IRC | 04:51 | |
| *** ykarel has quit IRC | 04:51 | |
| *** agopi|training has quit IRC | 04:53 | |
| *** rkukura has joined #openstack-infra | 04:58 | |
| *** vivsoni has joined #openstack-infra | 04:58 | |
| prometheanfire | woot, full pass https://review.openstack.org/602439 | 05:01 |
| *** ykarel has joined #openstack-infra | 05:07 | |
| *** e0ne has joined #openstack-infra | 05:09 | |
| *** e0ne has quit IRC | 05:10 | |
| *** rcernin_ has joined #openstack-infra | 05:17 | |
| *** lbragstad has quit IRC | 05:18 | |
| *** rcernin has quit IRC | 05:19 | |
| *** jaosorior has joined #openstack-infra | 05:20 | |
| ianw | AJaeger_: hmm bummer ... | 05:26 |
| ianw | it looks to me like statsd isn't running | 05:27 |
| ianw | did we merge something? | 05:27 |
| ianw | the config file is borked | 05:29 |
| ianw | http://paste.openstack.org/show/730682/ | 05:29 |
| *** vivsoni has quit IRC | 05:33 | |
| *** hashar has joined #openstack-infra | 05:41 | |
| *** pcaruana has joined #openstack-infra | 05:41 | |
| *** chkumar|off is now known as chkumar|ruck | 05:42 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-graphite master: Fix config for ipv6 https://review.openstack.org/604972 | 05:43 |
| ianw | AJaeger_: ^ I guess puppet decided to restart it for some reason, and hit that. i'll probably put it in as an emergency fix | 05:43 |
| *** vivsoni has joined #openstack-infra | 05:44 | |
| ianw | i guess the rspec doesn't actually check the service started ... can look at that | 05:44 |
| *** quique|rover|off is now known as quiquell|rover | 05:45 | |
| *** vivsoni has quit IRC | 05:49 | |
| *** vivsoni has joined #openstack-infra | 05:49 | |
| openstackgerrit | Merged openstack-infra/glean master: Add option to ignore config drive interfaces info https://review.openstack.org/604193 | 05:52 |
| abelur | StevenK: sorry, missed your message | 05:54 |
| abelur | StevenK: OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to git.openstack.org:443 | 05:54 |
| AJaeger_ | thanks, ianw | 05:58 |
| *** rkukura has quit IRC | 05:59 | |
| *** Bhujay has quit IRC | 06:00 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Block twine 1.12.0 when we install it https://review.openstack.org/604862 | 06:05 |
| *** bandini has joined #openstack-infra | 06:10 | |
| StevenK | abelur: From a google, that sounds like the connection is being reset while you're waiting for data | 06:14 |
| *** Bhujay has joined #openstack-infra | 06:14 | |
| StevenK | abelur: What does an mtr to review.openstack.org show? | 06:14 |
| *** Bhujay has quit IRC | 06:15 | |
| *** Bhujay has joined #openstack-infra | 06:16 | |
| abelur | StevenK: this looks like an issue with my isp :/ | 06:18 |
| abelur | sorry for all the noise | 06:18 |
| *** slaweq has joined #openstack-infra | 06:18 | |
| StevenK | abelur: No worries, happy to help | 06:19 |
| abelur | StevenK: thank you :) | 06:20 |
| *** dpawlik has joined #openstack-infra | 06:21 | |
| *** aojea has joined #openstack-infra | 06:24 | |
| *** jtomasek has joined #openstack-infra | 06:27 | |
| *** jtomasek has quit IRC | 06:27 | |
| *** jtomasek has joined #openstack-infra | 06:27 | |
| *** aojeagarcia has joined #openstack-infra | 06:29 | |
| *** aojea has quit IRC | 06:32 | |
| *** Bhujay has quit IRC | 06:32 | |
| *** Bhujay has joined #openstack-infra | 06:33 | |
| ianw | arggh, puppet restarted statsd again. i'm approving the config file fix, and will look at the rspec, i'll put it in emergency for now | 06:42 |
| ianw | #status log graphite.o.o in emergency until merge of https://review.openstack.org/604972 | 06:43 |
| openstackstatus | ianw: finished logging | 06:43 |
| openstackgerrit | Markus Hosch proposed openstack-infra/zuul master: Add support for authentication/STARTTLS to SMTP https://review.openstack.org/603833 | 06:48 |
| openstackgerrit | Merged openstack-infra/glean master: ignore ip6gre when setting up interfaces https://review.openstack.org/493443 | 06:48 |
| ianw | clarkb: ^ so i think we're now in a position to release glean, but in the mean time it's also 5pm here :) so yeah ... following the sun it's almost down here :) | 06:49 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-sphinx master: Add attr_overview directive https://review.openstack.org/604980 | 06:49 |
| *** roman_g has joined #openstack-infra | 06:54 | |
| *** lpetrut has joined #openstack-infra | 06:54 | |
| *** lpetrut has quit IRC | 06:56 | |
| *** lpetrut has joined #openstack-infra | 06:56 | |
| *** strigazi has joined #openstack-infra | 06:56 | |
| *** slaweq has quit IRC | 06:58 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: Add overview of config options https://review.openstack.org/604984 | 07:01 |
| openstackgerrit | Merged openstack-infra/puppet-graphite master: Fix config for ipv6 https://review.openstack.org/604972 | 07:02 |
| *** rcernin_ has quit IRC | 07:05 | |
| *** icey has quit IRC | 07:05 | |
| *** gfidente has joined #openstack-infra | 07:09 | |
| *** quiquell|rover is now known as quique|rover|brb | 07:10 | |
| *** ykarel is now known as ykarel|lunch | 07:25 | |
| *** xinliang has joined #openstack-infra | 07:29 | |
| *** psachin has quit IRC | 07:41 | |
| *** alexchadin has joined #openstack-infra | 07:42 | |
| *** quique|rover|brb is now known as quiquell|rover | 07:42 | |
| *** agopi__ has joined #openstack-infra | 07:46 | |
| *** agopi_ has quit IRC | 07:49 | |
| *** psachin has joined #openstack-infra | 07:49 | |
| *** e0ne has joined #openstack-infra | 07:52 | |
| *** shardy has joined #openstack-infra | 07:52 | |
| openstackgerrit | Ilya Shakhat proposed openstack-infra/project-config master: Move os-faults jobs to project repository https://review.openstack.org/604993 | 07:52 |
| *** jpena|off is now known as jpena | 07:53 | |
| *** e0ne has quit IRC | 07:53 | |
| *** tosky has joined #openstack-infra | 07:54 | |
| *** jamesmcarthur has joined #openstack-infra | 08:09 | |
| *** jamesmcarthur has quit IRC | 08:13 | |
| *** bhavikdbavishi has joined #openstack-infra | 08:14 | |
| *** jcoufal has joined #openstack-infra | 08:22 | |
| *** ginopc has joined #openstack-infra | 08:26 | |
| *** jamesmcarthur has joined #openstack-infra | 08:30 | |
| *** bdodd has quit IRC | 08:30 | |
| *** bdodd has joined #openstack-infra | 08:32 | |
| *** jamesmcarthur has quit IRC | 08:34 | |
| *** Dobroslaw has joined #openstack-infra | 08:35 | |
| *** ykarel|lunch is now known as ykarel | 08:35 | |
| *** dtantsur|afk is now known as dtantsur | 08:38 | |
| *** derekh has joined #openstack-infra | 08:38 | |
| AJaeger_ | config-core, dhellmann's python3-first change for nova team is ready, please review: https://review.openstack.org/#/c/601406 . Still has WIP but I expect dhellmann to remove it soon | 08:48 |
| *** slaweq has joined #openstack-infra | 08:49 | |
| *** evrardjp_ has joined #openstack-infra | 08:50 | |
| *** jamesmcarthur has joined #openstack-infra | 08:50 | |
| *** evrardjp has quit IRC | 08:53 | |
| *** neith has joined #openstack-infra | 08:53 | |
| *** priteau has joined #openstack-infra | 08:54 | |
| *** jamesmcarthur has quit IRC | 08:55 | |
| *** e0ne has joined #openstack-infra | 08:59 | |
| *** panda|off is now known as panda | 09:01 | |
| *** dpawlik has quit IRC | 09:05 | |
| *** dpawlik has joined #openstack-infra | 09:06 | |
| *** sshnaidm has joined #openstack-infra | 09:07 | |
| *** owalsh_ is now known as owalsh | 09:11 | |
| *** jamesmcarthur has joined #openstack-infra | 09:11 | |
| *** alexchadin has quit IRC | 09:12 | |
| *** evrardjp_ is now known as evrardjp | 09:15 | |
| *** alexchadin has joined #openstack-infra | 09:16 | |
| *** jamesmcarthur has quit IRC | 09:16 | |
| *** Emine has joined #openstack-infra | 09:18 | |
| *** janki has joined #openstack-infra | 09:20 | |
| *** pbourke has quit IRC | 09:21 | |
| *** pbourke has joined #openstack-infra | 09:22 | |
| *** vivsoni has quit IRC | 09:22 | |
| *** roman_g has quit IRC | 09:27 | |
| *** roman_g has joined #openstack-infra | 09:29 | |
| *** bhavikdbavishi1 has joined #openstack-infra | 09:29 | |
| *** sshnaidm is now known as sshnaidm|afk | 09:29 | |
| *** bhavikdbavishi has quit IRC | 09:29 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 09:29 | |
| *** jamesmcarthur has joined #openstack-infra | 09:32 | |
| *** vivsoni has joined #openstack-infra | 09:32 | |
| *** jamesmcarthur has quit IRC | 09:36 | |
| *** sshnaidm|afk is now known as sshnaidm|off | 09:37 | |
| *** sshnaidm|off has quit IRC | 09:42 | |
| *** bhavikdbavishi has quit IRC | 09:54 | |
| *** graphene has joined #openstack-infra | 09:55 | |
| openstackgerrit | Michal Nasiadka proposed openstack-infra/project-config master: Fix not working kolla graphs https://review.openstack.org/605026 | 09:57 |
| *** longkb has quit IRC | 09:59 | |
| *** electrofelix has joined #openstack-infra | 10:00 | |
| *** janki has quit IRC | 10:07 | |
| *** e0ne has quit IRC | 10:12 | |
| *** jamesmcarthur has joined #openstack-infra | 10:13 | |
| *** jamesmcarthur has quit IRC | 10:17 | |
| *** bhavikdbavishi has joined #openstack-infra | 10:23 | |
| *** Bhujay has quit IRC | 10:31 | |
| *** Bhujay has joined #openstack-infra | 10:32 | |
| *** yamamoto has quit IRC | 10:32 | |
| *** jamesmcarthur has joined #openstack-infra | 10:34 | |
| *** annp has quit IRC | 10:36 | |
| mnasiadka | What is the easiest way to get notifications about failed periodic zuul jobs? | 10:37 |
| *** jamesmcarthur has quit IRC | 10:38 | |
| AJaeger_ | mnasiadka: for stable ones: http://lists.openstack.org/pipermail/openstack-stable-maint/2018-September/date.html | 10:42 |
| AJaeger_ | mnasiadka: otherwise, you have to pull, e.g. using health (status.openstack.org/openstack-health/ ) or https://zuul.openstack.org/builds.html | 10:42 |
| *** rcernin_ has joined #openstack-infra | 10:43 | |
| mnasiadka | AJaeger_: thanks | 10:47 |
| *** stephenfin_ is now known as stephenfin | 10:48 | |
| *** janki has joined #openstack-infra | 10:50 | |
| *** rcernin_ has quit IRC | 10:52 | |
| *** alexchadin has quit IRC | 10:52 | |
| *** dhill_ has quit IRC | 10:55 | |
| *** florianf has joined #openstack-infra | 10:56 | |
| *** yamamoto has joined #openstack-infra | 10:57 | |
| *** e0ne has joined #openstack-infra | 10:58 | |
| *** jcoufal has quit IRC | 11:01 | |
| *** janki has quit IRC | 11:07 | |
| *** pcaruana has quit IRC | 11:15 | |
| *** jamesmcarthur has joined #openstack-infra | 11:15 | |
| *** udesale has quit IRC | 11:17 | |
| *** smarcet has joined #openstack-infra | 11:19 | |
| *** panda is now known as panda|afk | 11:19 | |
| *** jamesmcarthur has quit IRC | 11:20 | |
| *** e0ne_ has joined #openstack-infra | 11:22 | |
| *** e0ne has quit IRC | 11:25 | |
| *** dtantsur is now known as dtantsur|bbl | 11:25 | |
| *** jpena is now known as jpena|lunch | 11:38 | |
| *** boden has joined #openstack-infra | 11:45 | |
| *** quiquell|rover is now known as quique|rover|lch | 11:49 | |
| *** hashar is now known as hasharAway | 11:50 | |
| *** ginopc has quit IRC | 11:50 | |
| chkumar|ruck | ssbarnea|bkp: did we also added twine check feature in release jobs where tarballs getting uploaded to pypi? | 11:51 |
| *** ginopc has joined #openstack-infra | 11:51 | |
| chkumar|ruck | for checking rendering readme file on pypi | 11:51 |
| *** bhavikdbavishi1 has joined #openstack-infra | 11:51 | |
| *** bhavikdbavishi has quit IRC | 11:53 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 11:53 | |
| *** alexchadin has joined #openstack-infra | 11:56 | |
| *** jamesmcarthur has joined #openstack-infra | 11:57 | |
| *** jamesmcarthur has quit IRC | 12:01 | |
| *** aidin has joined #openstack-infra | 12:01 | |
| *** dhill_ has joined #openstack-infra | 12:01 | |
| mordred | chkumar|ruck: no - cause once it gets to that stage it's too late | 12:02 |
| mordred | chkumar|ruck: however, it wouldn't be a terrible thing to add to a project's pep8 tox env - or maybe it's a thing we should consider adding to our pep8 job in general | 12:02 |
| chkumar|ruck | mordred: I will take care of that | 12:02 |
| *** rh-jelabarre has joined #openstack-infra | 12:05 | |
| *** ginopc has quit IRC | 12:06 | |
| *** psachin has quit IRC | 12:07 | |
| *** bhavikdbavishi has quit IRC | 12:07 | |
| jaosorior | Any zuul admin, could we prioritize these two patches in the tripleo queue https://review.openstack.org/#/c/604977/ https://review.openstack.org/#/c/604976/ | 12:12 |
| jaosorior | ? | 12:12 |
| ssbarnea | chkumar|ruck++ testing this as part of tox-linters (pep8) would be ideal | 12:12 |
| AJaeger_ | jaosorior: ask for an "infra-root" ^ | 12:13 |
| *** ginopc has joined #openstack-infra | 12:13 | |
| *** AJaeger_ is now known as AJaeger | 12:13 | |
| jaosorior | AJaeger_: thanks | 12:13 |
| AJaeger | jaosorior: that's the magic to ask for admins here ;) | 12:13 |
| jaosorior | gotcha :D | 12:15 |
| jaosorior | sooo... any infra-root around? could we prioritize these two patches in the tripleo queue https://review.openstack.org/#/c/604977/ https://review.openstack.org/#/c/604976/ | 12:15 |
| *** yamamoto has quit IRC | 12:22 | |
| *** trown|outtypewww is now known as trown | 12:23 | |
| *** tpsilva has joined #openstack-infra | 12:24 | |
| *** vivsoni has quit IRC | 12:26 | |
| *** kgiusti has joined #openstack-infra | 12:30 | |
| *** Bhujay has quit IRC | 12:31 | |
| *** Bhujay has joined #openstack-infra | 12:32 | |
| *** jcoufal has joined #openstack-infra | 12:32 | |
| *** Bhujay has quit IRC | 12:33 | |
| *** Bhujay has joined #openstack-infra | 12:33 | |
| *** lpetrut has quit IRC | 12:33 | |
| *** agopi__ has quit IRC | 12:34 | |
| *** lbragstad has joined #openstack-infra | 12:35 | |
| *** alexchadin has quit IRC | 12:36 | |
| *** lpetrut has joined #openstack-infra | 12:36 | |
| *** alexchadin has joined #openstack-infra | 12:37 | |
| *** alexchadin has quit IRC | 12:37 | |
| *** aidin has quit IRC | 12:37 | |
| *** alexchadin has joined #openstack-infra | 12:37 | |
| *** alexchadin has quit IRC | 12:38 | |
| *** alexchadin has joined #openstack-infra | 12:38 | |
| *** alexchadin has quit IRC | 12:38 | |
| *** alexchadin has joined #openstack-infra | 12:39 | |
| *** alexchadin has quit IRC | 12:39 | |
| *** jpena|lunch is now known as jpena | 12:40 | |
| *** lbragstad has quit IRC | 12:47 | |
| *** ginopc has quit IRC | 12:48 | |
| *** graphene has quit IRC | 12:49 | |
| *** panda|afk is now known as panda | 12:49 | |
| *** alexchadin has joined #openstack-infra | 12:49 | |
| *** ramishra has quit IRC | 12:50 | |
| *** graphene has joined #openstack-infra | 12:50 | |
| *** kashyap has joined #openstack-infra | 12:51 | |
| kashyap | AJaeger: Hi there. | 12:51 |
| kashyap | AJaeger: Will you be able to confirm / deny from a SLES point of view here: http://lists.openstack.org/pipermail/openstack-dev/2018-September/135007.html | 12:51 |
| kashyap | AJaeger: Refer to question (c). | 12:52 |
| *** ginopc has joined #openstack-infra | 12:52 | |
| *** ramishra has joined #openstack-infra | 12:54 | |
| *** alexchadin has quit IRC | 12:54 | |
| *** ssbarnea has quit IRC | 12:56 | |
| AJaeger | dirk, cmurphy, can you help kashyap, please? ^ | 12:57 |
| jaosorior | Also... if any infra-root is around, could we also prioritize this patch in the tripleo queue https://review.openstack.org/604878 ? | 12:58 |
| *** yamamoto has joined #openstack-infra | 12:59 | |
| *** lbragstad has joined #openstack-infra | 12:59 | |
| cmurphy | AJaeger: kashyap looks like a yes afaict | 13:02 |
| kashyap | cmurphy: Hi there. Guessed as much; and I'm pretty sure it's a yes, given the timeframe. But can you please say so on the list, please? | 13:02 |
| cmurphy | kashyap: yes | 13:03 |
| *** bobh has joined #openstack-infra | 13:03 | |
| kashyap | cmurphy: And if you can ACK this, would be great too: https://review.openstack.org/#/c/605060/1 | 13:03 |
| kashyap | Thanks! | 13:03 |
| *** agopi__ has joined #openstack-infra | 13:05 | |
| *** ssbarnea has joined #openstack-infra | 13:06 | |
| fungi | mordred: chkumar|ruck: on one of my projects, i have a testenv:dist section in my tox.ini which does `python setup.py check --restructuredtext --strict` followed by `python setup.py bdist_wheel sdist` (those seemed closely related to me). in openstack projects where we don't generally directly test wheel and tarball builds it may make sense to add as a test in the docs env or pep8 yes | 13:07 |
| *** quique|rover|lch is now known as quiquell|rover | 13:09 | |
| *** lbragstad has quit IRC | 13:09 | |
| fungi | jaosorior: 604878,3 should be at the front of the tripleo shared gate queue now | 13:10 |
| AJaeger | mordred, chkumar|ruck, fungi , we have the job test-release-openstack-python3 that does some testing, it is part of publish-to-pypi template. | 13:13 |
| jaosorior | fungi: thanks! | 13:13 |
| AJaeger | http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul.d/jobs.yaml#n255 is the job | 13:14 |
| fungi | great! i guess we added that somewhat recently | 13:15 |
| AJaeger | fungi: during last few months... | 13:16 |
| AJaeger | not sure exactly when | 13:16 |
| *** alexchadin has joined #openstack-infra | 13:16 | |
| fungi | yeah, i still have recent memories of projects merging changes which broke release artifact generation and we didn't find out until a release was tagged | 13:17 |
| fungi | so good to see we've plugged that hole in test coverage | 13:17 |
| *** yamamoto has quit IRC | 13:22 | |
| *** xinliang has quit IRC | 13:22 | |
| *** sthussey has joined #openstack-infra | 13:22 | |
| *** graphene has quit IRC | 13:23 | |
| *** lbragstad has joined #openstack-infra | 13:23 | |
| *** graphene has joined #openstack-infra | 13:24 | |
| *** lbragstad has quit IRC | 13:24 | |
| *** lbragstad has joined #openstack-infra | 13:24 | |
| *** graphene has quit IRC | 13:25 | |
| *** graphene has joined #openstack-infra | 13:26 | |
| *** lbragstad has quit IRC | 13:26 | |
| *** lbragstad_ has joined #openstack-infra | 13:26 | |
| *** lbragstad_ has quit IRC | 13:26 | |
| *** lbragstad has joined #openstack-infra | 13:27 | |
| *** icey has joined #openstack-infra | 13:28 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Fix validation issue on links presentation creation https://review.openstack.org/605075 | 13:37 |
| *** hasharAway has quit IRC | 13:37 | |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Fix validation issue on links presentation creation https://review.openstack.org/605075 | 13:38 |
| *** yamamoto has joined #openstack-infra | 13:42 | |
| *** yamamoto has quit IRC | 13:42 | |
| *** yamamoto has joined #openstack-infra | 13:45 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Move glare legacy jobs in-repo https://review.openstack.org/605076 | 13:48 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove migrated legacy-glare-dsvm job https://review.openstack.org/605077 | 13:49 |
| *** smarcet has quit IRC | 13:56 | |
| *** evrardjp has quit IRC | 14:03 | |
| *** evrardjp has joined #openstack-infra | 14:03 | |
| *** dtantsur|bbl is now known as dtantsur | 14:04 | |
| *** alexchadin has quit IRC | 14:07 | |
| *** e0ne_ has quit IRC | 14:10 | |
| *** e0ne has joined #openstack-infra | 14:10 | |
| *** alexchadin has joined #openstack-infra | 14:12 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: WIP: Try to fix single cloud config reload https://review.openstack.org/605086 | 14:13 |
| *** roman_g has quit IRC | 14:14 | |
| openstackgerrit | Slawek Kaplonski proposed openstack-infra/openstack-zuul-jobs master: New role to connect br-infra and br-ex https://review.openstack.org/605087 | 14:17 |
| *** alexchadin has quit IRC | 14:17 | |
| AJaeger | slaweq: that role can go directly to devstack repo, can't it? ^ | 14:17 |
| slaweq | AJaeger: I don't know | 14:18 |
| slaweq | I though that it's good place to put this role here | 14:18 |
| slaweq | but if not, I can propose it in devstack | 14:18 |
| *** josephrsandoval has joined #openstack-infra | 14:18 | |
| *** alexchadin has joined #openstack-infra | 14:19 | |
| AJaeger | slaweq: so, let's ask differently: Who is going to use it? If that is part of a single job/some jobs in same repo, move it there as well. | 14:19 |
| slaweq | AJaeger: I think it will be used only by neutron jobs defined in neutron-tempest-plugin repo | 14:19 |
| AJaeger | slaweq: then add it there | 14:19 |
| slaweq | AJaeger: so do You think that it should be in roles/ directory in this repo? right? | 14:20 |
| AJaeger | yes. | 14:20 |
| *** quiquell|rover is now known as quique|rover|off | 14:20 | |
| AJaeger | config-core, dhellmann's python3-first change for nova team is ready, please review: https://review.openstack.org/#/c/601406 . Still has WIP but I expect dhellmann to remove it soon. PLease also review https://review.openstack.org/605076 https://review.openstack.org/604993 and https://review.openstack.org/604688 | 14:25 |
| openstackgerrit | Merged openstack-dev/pbr master: remove pypy jobs https://review.openstack.org/591504 | 14:25 |
| *** mriedem has joined #openstack-infra | 14:26 | |
| *** dmsimard has quit IRC | 14:31 | |
| *** dmsimard has joined #openstack-infra | 14:32 | |
| *** agopi__ is now known as agopi | 14:33 | |
| *** bhavikdbavishi has joined #openstack-infra | 14:33 | |
| *** vivsoni has joined #openstack-infra | 14:42 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: Proposed spec: tenant-scoped admin web API https://review.openstack.org/562321 | 14:50 |
| clarkb | slaweq: AJaeger ya I would put that either in the neutron repo or the devstack repo, since it is specific to testing neutron with devstack | 14:51 |
| clarkb | mordred: you good with a new release of glean at this point? then we need to rebuild images, then restart nodepool launchers, then update our config to set the instance property | 14:52 |
| slaweq | clarkb: AJaeger: ok, thx for tips, I just pushed it and will see if that will work | 14:52 |
| *** dpawlik has quit IRC | 14:52 | |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder master: enable caching for gentoo builds https://review.openstack.org/604268 | 14:55 |
| *** hwoarang has quit IRC | 14:56 | |
| mordred | clarkb: yes! I think we should definitely do that | 14:56 |
| *** dpawlik has joined #openstack-infra | 14:57 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Add opendev nameservers https://review.openstack.org/605092 | 14:57 |
| clarkb | mordred: I think the new version should be 1.12.0 as we add the new feature | 14:57 |
| clarkb | 1.11.1 was the last tag | 14:57 |
| *** alexchadin has quit IRC | 14:57 | |
| clarkb | I've got a call in a couple minutes but can push that after | 14:57 |
| *** ykarel is now known as ykarel|away | 14:57 | |
| fungi | funny these are the twine version numbers we were talking about yesterday | 15:00 |
| *** felipemonteiro has joined #openstack-infra | 15:00 | |
| clarkb | fungi: you just made me very afraid for glean 1.12.0 :) | 15:01 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Add zone-opendev.org project https://review.openstack.org/605095 | 15:02 |
| corvus | clarkb, fungi, mordred: ^ should that be in opendev/ or openstack-infra/ ? | 15:02 |
| *** Bhujay has quit IRC | 15:03 | |
| fungi | you ask a very existential question! | 15:03 |
| pabelanger | an exciting question | 15:03 |
| fungi | will that confuse gerrit's replication setup? | 15:03 |
| *** david-lyle has quit IRC | 15:04 | |
| *** lpetrut has quit IRC | 15:04 | |
| clarkb | it will try to push to github | 15:04 |
| clarkb | should be fine for cgit | 15:04 |
| *** dklyle has joined #openstack-infra | 15:05 | |
| *** jamesmcarthur has joined #openstack-infra | 15:05 | |
| *** graphene has quit IRC | 15:05 | |
| *** hwoarang has joined #openstack-infra | 15:06 | |
| *** felipemonteiro has quit IRC | 15:06 | |
| mordred | can we do a negative matcher in the replication config? | 15:07 |
| *** graphene has joined #openstack-infra | 15:08 | |
| mordred | oh - I guess probably not for only github | 15:08 |
| *** boden has quit IRC | 15:08 | |
| mordred | maybe we should get around to writing the zuul job version of replication | 15:09 |
| clarkb | I think we can configure a retry limit on gerrit | 15:09 |
| clarkb | maybe if we set that low enough say 3 attempts we'll be ok? | 15:09 |
| corvus | if it's easier to do openstack-infra/ for now, that's fine. presumably we'd be moving a bunch of stuff eventually anyway. just figured we might reduce some initial discordancy :) | 15:09 |
| clarkb | corvus: that is probably easiest considering the git replication | 15:10 |
| openstackgerrit | Chandan Kumar proposed openstack-infra/project-config master: Added twine check functionality to python-tarball playbook https://review.openstack.org/605096 | 15:11 |
| *** udesale has joined #openstack-infra | 15:12 | |
| chkumar|ruck | fungi: mordred AJaeger ^6 I am not sure I have done changes at the right place, but before running twince check, i need somewhere to do python setup.sdist | 15:12 |
| openstackgerrit | Chandan Kumar proposed openstack-infra/project-config master: Added twine check functionality to python-tarball playbook https://review.openstack.org/605096 | 15:13 |
| *** karimsye has joined #openstack-infra | 15:13 | |
| *** dave-mccowan has quit IRC | 15:14 | |
| openstackgerrit | Merged openstack-infra/project-config master: Move os-faults jobs to project repository https://review.openstack.org/604993 | 15:15 |
| *** hwoarang has quit IRC | 15:17 | |
| *** chkumar|ruck is now known as chkumar|off | 15:17 | |
| *** fuentess has joined #openstack-infra | 15:21 | |
| *** ykarel|away has quit IRC | 15:22 | |
| *** dpawlik has quit IRC | 15:25 | |
| *** jtomasek has quit IRC | 15:26 | |
| *** hwoarang has joined #openstack-infra | 15:26 | |
| mwhahaha | question for you folks, have you locked down lookup() in ansible on the zuul executors? re: http://logs.openstack.org/37/580037/60/check/tripleo-ci-centos-7-standalone/1e03908/job-output.txt.gz#_2018-09-25_05_22_46_279743 | 15:28 |
| *** armax has joined #openstack-infra | 15:30 | |
| *** yamamoto has quit IRC | 15:31 | |
| *** yamamoto has joined #openstack-infra | 15:32 | |
| *** yamamoto has quit IRC | 15:32 | |
| clarkb | mwhahaha: I think that is zuul protecting itself from potentially unsafe actiosn on the executor | 15:33 |
| mwhahaha | yea https://github.com/openstack-infra/zuul/blob/b4a4718b4fe38997336fe707e9a628376aa4f465/zuul/ansible/lookup/_banned.py | 15:35 |
| mwhahaha | odd that it was working a few months ago | 15:36 |
| * mwhahaha shrugs and goes off to figure out a different way to get this information | 15:36 | |
| mwhahaha | would really like to be able to fetch a url :/ | 15:36 |
| *** hwoarang has quit IRC | 15:36 | |
| *** aidin has joined #openstack-infra | 15:37 | |
| *** e0ne has quit IRC | 15:37 | |
| *** dave-mccowan has joined #openstack-infra | 15:38 | |
| corvus | mwhahaha: i'm confused -- it looks like that playbook should be running on the "primary" host, not the executor (localhost) | 15:38 |
| corvus | oh, right, lookup plugins | 15:39 |
| corvus | (they always run on the controller) nevermind | 15:39 |
| *** rh-jelabarre has quit IRC | 15:39 | |
| *** che-arne has quit IRC | 15:41 | |
| *** rh-jelabarre has joined #openstack-infra | 15:42 | |
| *** yamamoto has joined #openstack-infra | 15:44 | |
| fuentess | clarkb: hi, do you think it is still good time to add an item in the agenda for the infra meeting? To talk about the next steps for Kata? | 15:46 |
| clarkb | fuentess: sure | 15:46 |
| fuentess | clarkb: cool, thanks | 15:46 |
| *** hwoarang has joined #openstack-infra | 15:47 | |
| mordred | mwhahaha: get_url isn't blocked - if all you want to do is fetch a url | 15:48 |
| *** yamamoto has quit IRC | 15:49 | |
| clarkb | fuentess: I added it to the agenda, https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting | 15:49 |
| fuentess | clarkb: ohh nice, thanks :) | 15:49 |
| fuentess | clarkb: btw, the meeting is in about 1 hour 10 mins? | 15:50 |
| mwhahaha | mordred: corvus I think there is a reason I had to use lookup. U dig deeper | 15:50 |
| clarkb | fuentess: no ut is in 3 hours and 10 minutes (1900UTC currently 15:50UTC) | 15:50 |
| mwhahaha | Er I'll dig deeper | 15:50 |
| mwhahaha | Stupid autocorrect | 15:50 |
| fuentess | clarkb: got it, thanks | 15:51 |
| *** aojeagarcia has quit IRC | 15:52 | |
| mwhahaha | I think it's cause I want the data from the URL lookup as a var | 15:52 |
| *** graphene has quit IRC | 15:52 | |
| mwhahaha | Which get url does not do | 15:52 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Add zone-opendev.org project https://review.openstack.org/605095 | 15:52 |
| mwhahaha | I guess I could use a temp file | 15:52 |
| *** graphene has joined #openstack-infra | 15:53 | |
| *** aidin has quit IRC | 15:56 | |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder master: enable caching for gentoo builds https://review.openstack.org/604268 | 15:58 |
| *** ykarel|away has joined #openstack-infra | 15:59 | |
| *** ykarel|away is now known as ykarel | 15:59 | |
| clarkb | I'm making the glean release now | 15:59 |
| *** sshnaidm|off has joined #openstack-infra | 16:02 | |
| *** bobh_ has joined #openstack-infra | 16:03 | |
| *** ginopc has quit IRC | 16:03 | |
| clarkb | jobs are queued in zuul | 16:04 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: web: add tenant and project scoped, JWT-protected actions https://review.openstack.org/576907 | 16:04 |
| *** _d34dh0r53_ is now known as d34dh0r53 | 16:05 | |
| *** sshnaidm|off has quit IRC | 16:06 | |
| *** bobh has quit IRC | 16:06 | |
| *** sshnaidm has joined #openstack-infra | 16:06 | |
| corvus | clarkb: glean gets us ovh right? | 16:07 |
| *** pcaruana has joined #openstack-infra | 16:07 | |
| clarkb | corvus: yes | 16:07 |
| clarkb | corvus: there are a couple steps after the glean release, but it is the major one. We need to rebuild images after glean release and restart launchers to support setting instance properties. Then configure ovh to set the instance property | 16:08 |
| *** felipemonteiro has joined #openstack-infra | 16:08 | |
| corvus | we can do the last 2 now | 16:09 |
| clarkb | yup. I need to eat something for breakfast but can help with that after | 16:09 |
| *** Emine has quit IRC | 16:12 | |
| *** dave-mccowan has quit IRC | 16:13 | |
| *** jcoufal has quit IRC | 16:16 | |
| *** josephrsandoval has quit IRC | 16:16 | |
| *** yamamoto has joined #openstack-infra | 16:21 | |
| clarkb | ok I think I will start with restarting the launchers | 16:21 |
| fungi | i'm being dragged away to get some lunch, but plan to do some experimenting with the its-storyboard configuration on review-dev when i get back to see if i can get the task footer links working | 16:22 |
| fungi | bbiab | 16:22 |
| clarkb | nodepool==3.2.1.dev38 # git sha e12640e is the version installed on all four launchers which includes the property setting feature | 16:23 |
| clarkb | I'll start with nl04 since that is where ovh bhs1 is launched | 16:23 |
| clarkb | Shrews: ^ fyi | 16:23 |
| *** florianf has quit IRC | 16:23 | |
| Shrews | okie | 16:24 |
| *** udesale has quit IRC | 16:27 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config master: Glean config on OVH nodes https://review.openstack.org/605121 | 16:29 |
| clarkb | corvus: ^ that is the config side which should be safe to apply now | 16:29 |
| *** rkukura has joined #openstack-infra | 16:29 | |
| * clarkb checks on nl04 and will restart launchers on 01-03 if it appears happy | 16:29 | |
| clarkb | seems at least one node in gra1 went from building to ready to in use via nl04 | 16:31 |
| clarkb | we might need to enqueue https://review.openstack.org/605121 to the gate to avoid waiting forever for it to be tested | 16:32 |
| clarkb | I think this is reasonable as it is part of adding capacity to process other changes more quickly | 16:32 |
| clarkb | if any other infra-root or config-core is willing to review it I will do the enqueue | 16:33 |
| clarkb | (it has one +2 already) | 16:33 |
| mnaser | clarkb: lgtm | 16:34 |
| mnaser | good luck :> | 16:34 |
| clarkb | mnaser: thanks | 16:34 |
| clarkb | https://pypi.org/project/glean/ has new version now | 16:37 |
| * clarkb finishes launcher restarts then will queue image builds | 16:37 | |
| clarkb | corvus: mordred: did you see https://review.openstack.org/#/c/604932/ ? | 16:39 |
| clarkb | that tries to be smarter about ssh key config | 16:39 |
| *** dklyle has quit IRC | 16:40 | |
| dhellmann | config-core: https://review.openstack.org/#/c/601406/3 has 2 +2 votes and I've removed my WIP so we're ready to go | 16:40 |
| clarkb | dhellmann: approved | 16:40 |
| dhellmann | clarkb : thanks! | 16:41 |
| *** jtomasek has joined #openstack-infra | 16:42 | |
| clarkb | all four launchers are running latest nodepool code. | 16:43 |
| *** dklyle has joined #openstack-infra | 16:44 | |
| *** dtantsur is now known as dtantsur|afk | 16:45 | |
| *** rkukura has quit IRC | 16:47 | |
| clarkb | centos-7 image happend to build right after glean 1.12.0 was released. Reading the build log 1.12.0 was installed. I've queued image builds for all of our remaining x86 images | 16:48 |
| *** zaro_ has joined #openstack-infra | 16:48 | |
| *** zaro_ has quit IRC | 16:49 | |
| mordred | clarkb: woot | 16:49 |
| mordred | clarkb: and nice on the ssh key config | 16:49 |
| mordred | in fact ... | 16:49 |
| clarkb | mordred: its clearly broken since the tests are not working :) but I think it gets us the auto updating that corvus is interested in if we fix it | 16:49 |
| openstackgerrit | Nate Johnston proposed openstack-infra/project-config master: Show only neutron-fullstack-python36 in neutron grafana https://review.openstack.org/605128 | 16:50 |
| *** zaro has joined #openstack-infra | 16:51 | |
| *** dtantsur|afk has quit IRC | 16:51 | |
| jackatrack_ | clarkb: Yesterday I took a quick look at the stats.py code, and it appears some of the states are initialized to zero here https://github.com/openstack-infra/nodepool/blob/e12640ecfbe7b4af1e81463f7217bb9553ac365f/nodepool/stats.py#L101-L105 but these states (w/ labels as keys) aren't initialized https://github.com/openstack-infra/nodepool/blob/e12640ecfbe7b4af1e81463f7217bb9553ac365f/nodepool/stats.py#L114-L121 | 16:51 |
| openstackgerrit | Merged openstack-infra/project-config master: Glean config on OVH nodes https://review.openstack.org/605121 | 16:51 |
| clarkb | jackatrack_: ah, that else maybe needs to be set to 0 instead of 1? | 16:52 |
| *** panda is now known as panda|off | 16:52 | |
| clarkb | corvus: ^ | 16:52 |
| jackatrack_ | clarkb: I think it needs to be 1 still (because it is setting the nodes with a changed state), but we need to initialize all possible keys to 0 first | 16:54 |
| *** tdasilva has joined #openstack-infra | 16:54 | |
| corvus | jackatrack_: yeah, i think you're right -- we need to add all the labels to the line 101-105 section | 16:56 |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder master: enable caching for gentoo builds https://review.openstack.org/604268 | 16:56 |
| dtroyer | clarkb, infra: I'd appreciate it if we could have a look at https://review.openstack.org/599048 and https://review.openstack.org/599054 for docs.starlingx.io, Is there anything I need to do to keep things going? | 16:56 |
| jackatrack_ | corvus: right, just loop through each label and state and set to zero | 16:56 |
| *** dtantsur has joined #openstack-infra | 16:57 | |
| corvus | jackatrack_: the provider argument should have information about its labels | 16:57 |
| *** dtantsur is now known as dtantsur|afk | 16:58 | |
| jackatrack_ | I can probably make the change but this is a pretty low priority for me, I might have to get back to it down the road. I just wanted to point it out in case you guys wanted to fix it before I get a chance. | 16:59 |
| clarkb | dtroyer: we need https://review.openstack.org/#/c/601302/ I had hoped that corvus would review that as he set up the zuul ci website hosting under that system | 17:00 |
| openstackgerrit | Merged openstack-infra/project-config master: remove job settings for nova repositories https://review.openstack.org/601406 | 17:00 |
| *** derekh has quit IRC | 17:00 | |
| clarkb | dtroyer: would be a good sanity check to make sure we aren't adding that functionality unnecessarily by missing something | 17:00 |
| corvus | i will review that now (was out last week) | 17:00 |
| clarkb | corvus: the other two changes dtroyer linked give context | 17:00 |
| dtroyer | corvus, clarkb: thanks, I'm off to lunch, ping me if I can help somehow | 17:01 |
| *** ykarel is now known as ykarel|away | 17:01 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: WIP - Pagure driver https://review.openstack.org/604404 | 17:03 |
| *** hwoarang has quit IRC | 17:05 | |
| *** trown is now known as trown|lunch | 17:08 | |
| jackatrack_ | clarkb, corvus: do you think you might try fixing the stats.py issue, or would you prefer I make the adjustments? if so it'll be a few weeks at the earliest. thanks | 17:09 |
| clarkb | jackatrack_: as the person that noticed the issue and debugged it I think it would be great if you were also able to push the fix. Is there some hurdle other than time you need to clear? I'm happy to help however I can | 17:10 |
| *** diablo_rojo has joined #openstack-infra | 17:12 | |
| *** priteau has quit IRC | 17:13 | |
| *** roman_g has joined #openstack-infra | 17:14 | |
| jackatrack_ | clarkb: I think the only hurdle besides time is, I assume I need to go through the whole openstack sign up process and get a gerrit account, etc. I'm not sure what all is involved. | 17:15 |
| clarkb | jackatrack_: you need to get a gerrit account (which implies a launchpad/ubuntu one account if you don't have one yet). Otherwise that is it, you shouldn't need an openstack foundation membership/account or to sign a cla for nodepool | 17:16 |
| *** smarcet has joined #openstack-infra | 17:16 | |
| *** ramishra has quit IRC | 17:16 | |
| *** rkukura has joined #openstack-infra | 17:18 | |
| *** hwoarang has joined #openstack-infra | 17:19 | |
| corvus | jackatrack_: http://git.zuul-ci.org/cgit/nodepool/tree/README.rst#n31 attempts to cover it | 17:19 |
| *** anteaya has joined #openstack-infra | 17:21 | |
| *** jamesmcarthur has quit IRC | 17:21 | |
| *** boden has joined #openstack-infra | 17:22 | |
| jackatrack_ | clarkb, corvus: okay that's helpful! I'll get back with you when I'm able to get started. Thanks! | 17:24 |
| corvus | jackatrack_: thank you! | 17:25 |
| openstackgerrit | Merged openstack-infra/system-config master: Allow project website volume path to be overridden https://review.openstack.org/601302 | 17:28 |
| *** felipemonteiro has quit IRC | 17:28 | |
| *** jpena is now known as jpena|off | 17:29 | |
| *** njohnston has quit IRC | 17:31 | |
| *** njohnston has joined #openstack-infra | 17:32 | |
| *** shardy has quit IRC | 17:34 | |
| *** jamesmcarthur has joined #openstack-infra | 17:34 | |
| *** gfidente has quit IRC | 17:36 | |
| *** harlowja has joined #openstack-infra | 17:39 | |
| *** mdbooth has joined #openstack-infra | 17:40 | |
| *** panda|off has quit IRC | 17:41 | |
| mdbooth | Folks, is there a known issue with zuul today? | 17:41 |
| clarkb | mdbooth: I am not aware of any issues with zuul. We are running at lower capacity due to a cloud upgrade and jobs have been flaky in general (see my email to the dev list about that last week) | 17:42 |
| clarkb | mdbooth: are you seeing something specific? | 17:43 |
| mdbooth | clarkb: Nah, just wondering about the huge queue times. | 17:43 |
| mdbooth | clarkb: NM, guess it's just a temporary capacity issue, thanks. | 17:44 |
| clarkb | mdbooth: well it is that and the flakyness of the jobs | 17:44 |
| mdbooth | Incidentally, any idea when it might clear? | 17:44 |
| * mdbooth should look for the email | 17:45 | |
| *** panda has joined #openstack-infra | 17:45 | |
| clarkb | mdbooth: we are rebuilding all of our images right now to address new behavior in the cloud after the upgrade. Hopefully have that region online later today | 17:45 |
| mdbooth | "Zuul job backlog" 19th Sept | 17:45 |
| * mdbooth reads | 17:45 | |
| mdbooth | clarkb: Thanks | 17:45 |
| *** jamesmcarthur has quit IRC | 17:46 | |
| *** panda is now known as panda|off | 17:47 | |
| *** anteaya has quit IRC | 17:47 | |
| fungi | amorin: speaking of the dhcp flag, i wonder what's also involved in getting the static ipv6 address info exposed in configdrive/metadata service, since that would allow consumers using cloud-init or glean to make use of v6 addressing and routing automatically | 17:48 |
| *** ykarel has joined #openstack-infra | 17:49 | |
| * fungi is also a paying ovh customer with a vested interest in not needing to manually configure his ipv6 interface addresses and routes | 17:49 | |
| *** ykarel|away has quit IRC | 17:49 | |
| fungi | clearly the information is available to something given it gets presented in the neutron api responses | 17:50 |
| *** jamesmcarthur has joined #openstack-infra | 17:55 | |
| AJaeger | corvus, fungi, mordred, clarkb, a policy question for periodic master only jobs - see https://review.openstack.org/#/c/595808/1 and https://review.openstack.org/#/c/603554/ , please. The infra-manual states already to keep these kinds of jobs in project-config, see https://docs.openstack.org/infra/manual/creators.html#central-config-exceptions. Should we ignore this? Be stronger? | 17:59 |
| corvus | AJaeger: if going in-tree, that's the way to do it. i think the idea of having them in project-config is to reduce the extra work needed to maintain a master-only job in a branched repo. i guess if a project doesn't mind that extra work, that's probably okay... | 18:01 |
| AJaeger | corvus: I don't like change https://review.openstack.org/#/c/595808/1 - it's stable/rocky. I would have removed the jobs instead of adding branches: master to them | 18:03 |
| corvus | AJaeger: oh i missed that was on rocky, i thought that was on master | 18:03 |
| clarkb | corvus: AJaeger ya I think we set up the policy to allow the jobs in project-config (projects weren't forced to maintain that in tree with the branch changes over time) | 18:04 |
| AJaeger | corvus: there's a similar change for master... | 18:04 |
| *** bhavikdbavishi has quit IRC | 18:04 | |
| *** alishamohanty_ has joined #openstack-infra | 18:04 | |
| clarkb | its easier this way. If projects do something different I'm not sure we need to care too much? it was more about allowing projects to do the easy thing, saying we would review those changes in project-config than a mandate | 18:04 |
| AJaeger | for me it's just confusing to see in stable/rocky a .zuul.yaml file that has "branches: master". If that gets ignored, I would just remove it like I do with https://review.openstack.org/#/c/603554/ . If it's a nop, I will live with it ;) | 18:06 |
| *** njohnston has quit IRC | 18:07 | |
| *** njohnston has joined #openstack-infra | 18:07 | |
| clarkb | bhs1 has a glean 1.12.0 xenial image in it now. I'm testing a boot there. If that works then all we need to wait for is the other images to be uploaded and I think we can turn on bhs1 again | 18:08 |
| corvus | AJaeger: it's a condition that can never be satisfied -- the project stanza has a branch matcher of rocky, so it only takes effect on the rocky branch, and then those jobs will only run on the master branch. so a change/branch would have to be both master and rocky to satisfy it. | 18:08 |
| corvus | AJaeger: so not "ignored" as such, but certainly... inoperative. | 18:09 |
| AJaeger | corvus: so a "noop" basically - something we can ignore and I should not spend time on? ;) | 18:10 |
| AJaeger | clarkb: eager to hear your results ;) | 18:11 |
| corvus | AJaeger: yeah, it won't affect their ability to add or remove jobs from master, so i think the only downside is that it's unused cruft. | 18:12 |
| AJaeger | thanks for explanation. | 18:13 |
| corvus | i would just not recommend that people do that generally because it's a little mind bending :) | 18:13 |
| clarkb | xenial with glean changes seems to work on bhs1 | 18:14 |
| clarkb | I can talk http to the bhs1 mirror and the netmask is 255.255.255.255 and dhcp is used to configure ens3 | 18:14 |
| clarkb | infra-root `ssh root@158.69.64.237` if you want to double check yourself | 18:14 |
| clarkb | I'll delete my test nodes before we enable that region again | 18:14 |
| AJaeger | corvus: people will copy stuff... brian has proposed this change now for all glance repos... | 18:17 |
| AJaeger | clarkb: woot! | 18:17 |
| prometheanfire | ping for reviews?https://review.openstack.org/#/c/604677/ https://review.openstack.org/#/c/604688/ both just need a second | 18:20 |
| * AJaeger takes ...677, the other one has my +2 already | 18:20 | |
| prometheanfire | AJaeger: yep, thanks | 18:21 |
| *** alishamohanty_ is now known as alishamohanty | 18:22 | |
| prometheanfire | once 667 is done https://review.openstack.org/#/c/602439/ will be ready | 18:22 |
| prometheanfire | AJaeger: actually, https://review.openstack.org/#/c/602439/ should be reviewable already due to the deps on | 18:22 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Manage user ssh keys from urls https://review.openstack.org/604932 | 18:35 |
| *** trown|lunch is now known as trown | 18:36 | |
| AJaeger | team, regarding high load, I think that we have more jobs running now since on master we run python35 and python36 for all python3-first repos. And teams add additional py3 jobs without removing jobs... | 18:36 |
| mordred | AJaeger: I would happily remove all the python2 jobs from the sdk repo ... but I fear someone would get mad | 18:39 |
| mordred | AJaeger: (I agree) | 18:39 |
| clarkb | AJaeger: I've also seen (non scientific) evidence that the py3k jobs are less reliable than python2 jobs. Growing pains I think | 18:40 |
| clarkb | mordred: AJaeger likely the place we want to get to is the inverse of today. One or two python2 sanity checks then the majority of testing on python3.x where .x represents some reasonable python3 state | 18:41 |
| AJaeger | I fear there's not much we can do here in general and have to life with that part - or any ideas? | 18:41 |
| clarkb | I think for right now 3.6 is probably that reasonable state? | 18:41 |
| AJaeger | clarkb: so, remove py35? | 18:41 |
| clarkb | AJaeger: possibly. The potential drawback to that is python3.5 is actually a pretty "vanilla" python3 whereas 3.6 and 3.7 start adding a bunch of stuff that isn't backward compat | 18:42 |
| clarkb | so it is possible we will quickly find we no longer support 3.5 if we do that | 18:42 |
| mordred | clarkb: yah - but if projects also have 2.7 in the mix, they won't be able to get that stuff in | 18:42 |
| clarkb | mordred: that is a good point | 18:42 |
| clarkb | asnycio and the annotations stuff all difficult to make happy with 2.7 too | 18:42 |
| mordred | I think I'd be happy killing 3.5 from the sdk jobs and just doing 2.7 and 3.6 | 18:43 |
| AJaeger | so either py27 and 3.6 - or 3.5 and 3.6 ; | 18:43 |
| mordred | AJaeger: yah | 18:43 |
| *** graphene has quit IRC | 18:43 | |
| * mordred makes a patch to sdk | 18:43 | |
| AJaeger | dhellmann: FYI ^ - any other ideas? | 18:44 |
| clarkb | infra meeting in about 15 minutes, see you all over in #openstack-meeting for that | 18:46 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: Proposed spec: tenant-scoped admin web API https://review.openstack.org/562321 | 18:51 |
| *** gouthamr has quit IRC | 18:52 | |
| mordred | AJaeger: https://review.openstack.org/604517 Clean up python3 test and remove duplicate jobs | 18:52 |
| *** dmellado has quit IRC | 18:53 | |
| *** stevebaker has quit IRC | 18:53 | |
| AJaeger | mordred: that adds voting jobs only to check but not gate | 18:54 |
| *** fuentess_ has joined #openstack-infra | 18:55 | |
| *** fuentess has quit IRC | 18:55 | |
| *** bobh_ has quit IRC | 18:55 | |
| *** fuentess_ has quit IRC | 18:56 | |
| *** fuentess has joined #openstack-infra | 18:56 | |
| *** bobh has joined #openstack-infra | 18:56 | |
| *** aidin has joined #openstack-infra | 18:57 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: web: add tenant and project scoped, JWT-protected actions https://review.openstack.org/576907 | 18:59 |
| mwhahaha | if you have a second, please take a moment for this repo addition: https://review.openstack.org/#/c/603489/ | 18:59 |
| mordred | AJaeger: yah - it was intentional - I want to notice shade/os-client-config api breakages in check - but I'm not worried about one of those slipping in at the gate step | 18:59 |
| clarkb | infra meeting over in #openstack-meeting nowish | 18:59 |
| AJaeger | mordred: better add a comment for those... | 18:59 |
| mordred | AJaeger: good call. | 19:00 |
| *** bobh has quit IRC | 19:01 | |
| *** aidin has quit IRC | 19:01 | |
| jroll | I have a new project I'm creating that doesn't have code yet - should I start writing code so I can claim the name on pypi (per project creator's guide) or can that be done after the repository is created? | 19:02 |
| *** ykarel has quit IRC | 19:05 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Speed up build list query under mysql https://review.openstack.org/605170 | 19:05 |
| *** toabctl has quit IRC | 19:05 | |
| clarkb | jroll: you can squat the name without any code | 19:07 |
| *** toabctl has joined #openstack-infra | 19:07 | |
| jroll | clarkb: maybe I'm dumb, but I can't find a "create project" button anywhere on pypi.org | 19:07 |
| jroll | oh wait, twine has a register command, that will work | 19:08 |
| jroll | thanks | 19:08 |
| *** tomaw- is now known as tomaw | 19:08 | |
| *** bobh has joined #openstack-infra | 19:11 | |
| *** gouthamr has joined #openstack-infra | 19:12 | |
| fungi | jroll: you only need enough code to be able to compile a sdist with metadata contaniing a project name and version number (which can also just be 0) | 19:12 |
| fungi | you don't even need to import that into the initial repo | 19:12 |
| jroll | fungi: yeah. the project creator's guide made it sound like a PKG-INFO file was sufficient, but seems I need setup.py | 19:13 |
| *** ssbarnea|bkp has quit IRC | 19:14 | |
| jroll | got it now. thanks to both of you as always :) | 19:14 |
| mwhahaha | have there been any changes to the amount of time allowed in the pre/post part of job runs? I thought those timeouts used to be independent of the overall timeout but were still set to 3 hours or whatever | 19:14 |
| *** ssbarnea|bkp has joined #openstack-infra | 19:14 | |
| *** bobh has quit IRC | 19:15 | |
| clarkb | mwhahaha: pre + run is one timeout. post is a separate time | 19:15 |
| clarkb | we haven't chagned those defaults since we split pre+run from pre+run+post | 19:15 |
| mwhahaha | i thought pre was sperate from run | 19:15 |
| clarkb | it isn't | 19:15 |
| mwhahaha | then why did we have 9 hours timeouts when we first switched to zuulv3 | 19:16 |
| mwhahaha | i thought it was 3+3+3 | 19:16 |
| fungi | jroll: yeah, with the switch from the cheeseshop to warehouse, pypi ceased providing any means of "registering" a project without uploading a sdist or wheel (or egg i suppose) | 19:16 |
| jroll | fungi: indeed, this is my first time doing it post-switch | 19:16 |
| fungi | mwhahaha: because their timeouts used to be separate | 19:16 |
| mwhahaha | right that's my point, when did that change | 19:16 |
| fungi | mwhahaha: could have been even longer in fact if you had multiple playbooks per phase, since the timeout was simply implemented as an ansible playbook timeout in the beginning | 19:17 |
| mwhahaha | cause i'm not sure i like pre eating into our job timeouts if pre+run are combined and maxed to 3 hours? | 19:17 |
| clarkb | mwhahaha: months ago iirc. I want to say sometime around vancouver summit? | 19:17 |
| fungi | i would need to bisect zuul commits or hunt down the announcement which went to the -dev ml about it | 19:17 |
| mwhahaha | i'm assuming around Feb 20th if "[openstack-dev] [all][infra] Some new Zuul features" is that announcement | 19:18 |
| fungi | well, not bisect commits but at least go looking in the changelog/release notes/somewhere | 19:18 |
| fungi | sounds about right | 19:18 |
| mwhahaha | so sounds like we need to adjust out post timeout then | 19:19 |
| mwhahaha | what's the max for pre/run? still 3 hours? | 19:19 |
| fungi | i don't know if we set an upper-bound. it's whatever you configure as the "timeout" attribute for your job | 19:20 |
| prometheanfire | iirc, yes | 19:20 |
| mwhahaha | it used to have an upper limit of 3 hours | 19:20 |
| fungi | it might still be that. checking | 19:20 |
| AJaeger | fungi, mwhahaha http://git.openstack.org/cgit/openstack-infra/zuul/tree/doc/source/admin/tenants.rst#n210 - 3h and I don't think we change that... | 19:26 |
| fungi | mwhahaha: https://zuul-ci.org/docs/zuul/admin/tenants.html#attr-tenant.max-job-timeout says the default is 10800 seconds | 19:27 |
| fungi | i can't find that we're overriding it anywhere | 19:27 |
| mwhahaha | yea sounds about right | 19:27 |
| mwhahaha | i still am not sure it's ok for pre/run to be a shared timeout | 19:27 |
| fungi | i think it was set to that as a default on the auspices that nobody would likely ever want a job to keep running longer than that | 19:28 |
| clarkb | mwhahaha: the idea there is pre and run constitute the work of the job, we give that an allowance. Then post is for cleanup and log collection etc which we want to happen even if the previous steps timeout | 19:28 |
| mwhahaha | clarkb: yea i get that, except projects are penalized time wise by infra setup run time in pre | 19:28 |
| fungi | mainly we didn't want a single timeout covering the entirety of the job (like we had with zuul v2 + jenkins) because that stopped things like log collection in post from happening | 19:29 |
| mwhahaha | so we've always been operating under the assumption that the job timeout length is really the "run" phase | 19:29 |
| fungi | mwhahaha: i was about to say something nasty, but will instead just say i hope you don't feel that your project having access to a massive free ci infrastructure is seen as a penalty of any kind | 19:29 |
| *** jamesmcarthur has quit IRC | 19:29 | |
| fungi | job timeouts are also instituted to help the administrators of the system guard against projects or even specific changes monopolizing available resources. tripleo already accounts for something like half the cpu time across our resource pool | 19:31 |
| openstackgerrit | Jim Rollenhagen proposed openstack-infra/project-config master: Add new project: sardonic https://review.openstack.org/605193 | 19:32 |
| mwhahaha | right and we're trying to minimize that by dropping out scenarios into smaller faster jobs | 19:32 |
| *** electrofelix has quit IRC | 19:32 | |
| mwhahaha | but we can't merge anything because we keep hitting timeouts because the rules have changes and we don't have any container related infrastructure avaiable upstream so the mirrors are OK but not ideal | 19:32 |
| AJaeger | mwhahaha: I don't think it makes for load a difference whether you have two jobs each 60 mins or one job for 120 mins | 19:32 |
| mwhahaha | AJaeger: except we can reduce the amount we run if we can properly address what services are tested | 19:33 |
| AJaeger | or what do you mean with "dropping out scenarios into smaller faster jobs"? | 19:33 |
| jroll | creating a new project is super easy these days, hats off to you infra folks :) | 19:33 |
| AJaeger | thanks, jroll | 19:33 |
| AJaeger | mwhahaha: ah... | 19:33 |
| mwhahaha | AJaeger: so reducing the number of 2 node multinode jobs to single 1 node jobs | 19:33 |
| AJaeger | mwhahaha: I see. | 19:34 |
| mwhahaha | so a 3 hours 2 node jobs is much more costly than 2 1hour 1 node jobs | 19:34 |
| mwhahaha | and it's on our roadmap (hopefully this cycle) to move to more 1 node jobs rather than multinode | 19:35 |
| AJaeger | mwhahaha: the rules changed half a year ago, so I think it's unreasonable to argue that this makes a difference for problems in the last weeks... | 19:35 |
| mwhahaha | of course, it's all dependent on actually being able to murge stuff | 19:35 |
| AJaeger | but I like your approach | 19:35 |
| mwhahaha | we've been hitting this for months but it's gotten especially bad and we're still unsure why because we haven't merged things | 19:36 |
| mwhahaha | we've also been migrating to native zuulv3 so we might have screwed something up along the way | 19:36 |
| mwhahaha | trying to trackdown what and why, which is why i'm asking about timeout rules | 19:36 |
| mwhahaha | the last time we sat down and looked at this, we had 3+3+3 and now it's more 3+.5(adjustable) | 19:37 |
| *** e0ne has joined #openstack-infra | 19:41 | |
| openstackgerrit | Merged openstack-infra/system-config master: Update cache-stats.sh to include the port 80 proxy https://review.openstack.org/604897 | 19:43 |
| *** bobh has joined #openstack-infra | 19:44 | |
| prometheanfire | googling for a neomutt screenshot pulls one up that has openstack stuff :D | 19:46 |
| njohnston | that's pretty funny :-) | 19:48 |
| openstackgerrit | Anne Bertucio proposed openstack-infra/system-config master: Creates 'embargo-notice' list https://review.openstack.org/605212 | 19:58 |
| clarkb | I'm going to find lunch now. The last images needed for ovh are building now | 20:02 |
| *** trown is now known as trown|outtypewww | 20:03 | |
| fungi | gotta run a quick errand but will brb | 20:04 |
| mordred | ssbarnea, corvus: sorry - the bug spray dude arrived and I had to interact with him for a second and I missed the end of the meeting where you were talking about the output | 20:05 |
| *** gouthamr_ has joined #openstack-infra | 20:05 | |
| ssbarnea | clarkb: corvus : i updated the example from https://sbarnea.com/f/ansible-output/ -- click on "0" (vervosity level) for default / debug / yaml and compare them. how easy is to read the result of "ls -la" should be self-explanatory. | 20:05 |
| *** e0ne has quit IRC | 20:05 | |
| mordred | ssbarnea: a thing where we might be missing each other here- is that we are not using the default callback plugin | 20:06 |
| mordred | so changing _anything_ is not going to be an easy/simple fix | 20:06 |
| ssbarnea | corvus: regarding your question about colapsable console output. yes this would be cool feature, but unless we already have a CR in late stages of review we are speaking about potential features, and not one that can easily be implemented. | 20:07 |
| *** dmellado has joined #openstack-infra | 20:07 | |
| openstackgerrit | Hongbin Lu proposed openstack-infra/project-config master: Use storyboard for os-ken project https://review.openstack.org/605224 | 20:07 |
| corvus | ssbarnea: that's why it's important for us to have a conversation about what you're trying to achieve before we talk about how to achieve it | 20:07 |
| corvus | ssbarnea: what do you mean CR in laste stages of review? | 20:07 |
| corvus | 'late' rather | 20:08 |
| ssbarnea | corvus: i want to see readable output lines from shell/command in near feature, as less than a month from now. | 20:09 |
| corvus | ssbarnea: gotcha, thanks. | 20:09 |
| ssbarnea | i know that colapsable could be better, probably much better. still, it would be ok for me to have 10% of functionality tomorrow, than having 90% in 6mo ;) | 20:10 |
| dmsimard | It's hard for me to even read ansible output anymore when I have ara available but I'm biased a little bit :p | 20:10 |
| mordred | ssbarnea: so, it's not http://logs.openstack.org/85/574285/6/check/openstack-tox-py27/d399524/job-output.txt.gz#_2018-09-25_17_20_07_149723 that you're focused on | 20:10 |
| corvus | ssbarnea: yeah, or maybe both. we just need to know things like that so we know what to work on in the future. :) | 20:10 |
| mordred | ssbarnea: do you happen to have (or can find) a bad example in an existing zuul log you could link to? | 20:11 |
| corvus | clarkb: voip.ms balance is low | 20:12 |
| *** janki has joined #openstack-infra | 20:12 | |
| ssbarnea | corvus: output you linked looks ok, stdout lines are readable, by humans. only colors would make it better. | 20:13 |
| ssbarnea | i clearly need to setup my own zuul instance to play a bit more with it. i just picked own hw this morning so soon I will manage to setup a private infra. | 20:16 |
| mordred | yes - that's the best way to have all the fun :) | 20:18 |
| *** e0ne has joined #openstack-infra | 20:19 | |
| ssbarnea | mordred: ... more or less, I got my hw and now I need to wait for a vga 2 hdmi converter as I am unable to configure the server due to lack of monitor (i still need to configure the ipmi). hw was never my favourite thing, still we cannot run sw without it. | 20:25 |
| dmsimard | ssbarnea: what? I thought that was all the hype about serverless :P | 20:34 |
| *** gouthamr has quit IRC | 20:38 | |
| *** gouthamr_ is now known as gouthamr | 20:39 | |
| *** eernst has joined #openstack-infra | 20:40 | |
| *** pcaruana has quit IRC | 20:43 | |
| *** e0ne has quit IRC | 20:43 | |
| *** kgiusti has left #openstack-infra | 20:46 | |
| dhellmann | AJaeger , mordred , clarkb : I think it's likely safe to drop the 3.5 jobs if the distros are all going to include 3.6, but I'm not sure what info we have about that for RH yet | 20:46 |
| mordred | dhellmann: if a project is testing with 2.7 and 3.6 - shouldn't 3.5 be fairly covered? 2.7 won't let anyone land new syntax in 3.6 - and the other main change I can think of (ordered dict by default) should also be covered by 2.7 testing | 20:48 |
| mordred | dhellmann: although on the other hand, if it's not tested it's broken I guess :) | 20:48 |
| dhellmann | mordred : probably? I guess the changes we found that broke things were in 3.7 | 20:48 |
| mordred | yah - 3.7 is fun | 20:48 |
| fungi | okay, back now | 20:54 |
| clarkb | corvus: remind me the outstanding todo there is to have foundation create a new account with direct billing then we can transition the number over? | 20:58 |
| *** jtomasek has quit IRC | 21:00 | |
| *** fuentess has quit IRC | 21:01 | |
| corvus | clarkb: probably not a big deal if we change the number | 21:02 |
| *** fuentess has joined #openstack-infra | 21:02 | |
| clarkb | corvus: in that case, new account with direct payment, create new number, update pbx config? I'll prod scott/lsell about it again | 21:03 |
| corvus | clarkb: sounds good | 21:03 |
| fungi | changing the number seems entirely fine to me. we have to update the asterisk config anyway for the new credentials right? | 21:04 |
| fungi | aside from that, it's a wiki edit i guess | 21:04 |
| clarkb | corvus: there are three locked nodes in ovh bhs1 from before the upgrade. I cannot nodepool delete them because they are locked | 21:05 |
| *** Qiming has quit IRC | 21:05 | |
| clarkb | corvus: any idea why they may be locked persistently like this? and how to undo that? | 21:05 |
| clarkb | Shrews: ^ | 21:05 |
| ianw | did the zuul api status json stop responding? | 21:05 |
| ianw | oh, maybe it's just me | 21:06 |
| clarkb | ianw: I think it does this when it ends up running expensive reconfigurations? | 21:06 |
| *** Qiming has joined #openstack-infra | 21:06 | |
| clarkb | I seem to recall running into this pre ptg and it would go away for a few minutes then come back | 21:07 |
| clarkb | and after checking the logs it was busy with a large reconfiguration or a large number of reconfigurations | 21:07 |
| corvus | i have a suspicion we're maxing out zuul-web due to the size of the status json | 21:07 |
| clarkb | it just rendered for me | 21:08 |
| clarkb | oh wow 318 items in check weeee | 21:08 |
| clarkb | the good news is I am just minutes away from pushing a change to turn on ovh bhs1 | 21:08 |
| corvus | even just hitting http://zuul.openstack.org/api/info is very slow | 21:08 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config master: Revert "Disable OVH BHS1 region" https://review.openstack.org/605234 | 21:08 |
| *** janki has quit IRC | 21:08 | |
| clarkb | the last image finished uploading | 21:08 |
| clarkb | I tested the ubuntu xenial image did work | 21:08 |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: Add overview of config options https://review.openstack.org/604984 | 21:09 |
| clarkb | we've got unittest coverage of all the distros | 21:09 |
| clarkb | so I'm pretty confident that will work | 21:09 |
| clarkb | as before if I can get a couple +2's I will direct enqueue to the gate | 21:09 |
| clarkb | I guess I should double check centos-7 works too since its our other big usage distro | 21:09 |
| corvus | there are 5k outstanding node requests | 21:12 |
| ianw | clarkb: thanks for pushing it all ahead | 21:12 |
| *** stevebaker has joined #openstack-infra | 21:12 | |
| *** fuentess has quit IRC | 21:17 | |
| clarkb | `ssh root@158.69.64.191` to see a centos node working | 21:18 |
| clarkb | I'll delete that shortly though to free capacity for zuul | 21:18 |
| clarkb | ianw: oh sorry didn't realize you already have a revert up | 21:18 |
| clarkb | enqueing the revert into the gate is not fast | 21:19 |
| clarkb | but I have started that command | 21:19 |
| clarkb | ok should be enqueued now | 21:20 |
| *** weshay has joined #openstack-infra | 21:22 | |
| weshay | is zuul up? | 21:24 |
| clarkb | weshay: yes | 21:24 |
| clarkb | it is busy and maybe a little overwhelmed serving the json status file, but it is up and processing jobs | 21:24 |
| weshay | ok.. cool | 21:25 |
| weshay | it's not rendering the website atm.. but thanks clarkb | 21:25 |
| clarkb | weshay: ya we think it is having a hard time serving the json file (it is large) for all the updates. It shoudl render if you are patient | 21:25 |
| weshay | k | 21:26 |
| weshay | thanks | 21:26 |
| *** slaweq has quit IRC | 21:27 | |
| *** slaweq has joined #openstack-infra | 21:27 | |
| fungi | the zuul-web process has a cpu pegged | 21:30 |
| fungi | one of the scheduler processes is also spiking up to 100% of a cpu with regularity, but zuul-web is not budging from 100% of the one it's claimed | 21:31 |
| *** bobh has quit IRC | 21:31 | |
| corvus | 9209 is the scheduler, 9211 is geard | 21:32 |
| fungi | yeah, geard isn't especially busy by comparison | 21:33 |
| openstackgerrit | Merged openstack-infra/project-config master: Revert "Disable OVH BHS1 region" https://review.openstack.org/605234 | 21:33 |
| * clarkb checks when next puppet pulse is | 21:33 | |
| clarkb | puppet just started and we update the project-config to latest when puppet runs on those nodes so we should see this update in the next 30-45 minutes | 21:34 |
| openstackgerrit | Merged openstack-dev/pbr master: Remove my_ip from generated wsgi script https://review.openstack.org/601325 | 21:34 |
| fungi | actually not sure whether zuul-web is really just using one cpu since the per-processor top breakdown shows a fairly even (if undulating) distribution across them all | 21:34 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-sphinx master: Add example and type options to attributes https://review.openstack.org/604267 | 21:34 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-sphinx master: Add attr_overview directive https://review.openstack.org/604980 | 21:34 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-sphinx master: Use OrderedDict for object tracking https://review.openstack.org/605240 | 21:34 |
| fungi | but certainly a strange coincidence none of them seem to go more than a percent or two above 100 | 21:34 |
| mnaser | clarkb: is there any projects right now requiring gpu support in openstack ci? | 21:34 |
| fungi | mnaser: by requiring you mean requesting or faking? | 21:35 |
| mnaser | i mean.. both possibly? | 21:35 |
| fungi | we don't currently have any gpu-capable instances in any of our providers afaik | 21:35 |
| corvus | fungi: the gil will tend to cause a python process that spends a lot of time in the interpreter (rather than in syscalls) to peak at 1 cpu. | 21:35 |
| fungi | so none of them are requiring it (yet anyway) | 21:35 |
| *** smarcet has quit IRC | 21:36 | |
| fungi | i expect cyborg, at least, would love to have something like that available | 21:36 |
| clarkb | mnaser: ya I don't think anyone is requiring it. nova might be interested for testing some of the directed io (whatever the hip name for it is now) code paths | 21:36 |
| mnaser | so we're working on making gpus available in our public cloud | 21:36 |
| *** smarcet has joined #openstack-infra | 21:37 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: Add overview of config options https://review.openstack.org/604984 | 21:37 |
| mnaser | by that i mean, we're waiting for the hardware to arrive. i can provide maybe 1 or 2 gpu instances | 21:37 |
| corvus | fungi: by which i suspect we're spending a lot of time serializing/deserializing 3.7MB json blobs | 21:37 |
| fungi | yes, that sounds likely | 21:37 |
| fungi | we have/had caching for that in apache right? | 21:38 |
| mnaser | it'll likely be a 6 core, 64gb memory, 250gb nvme pci-e local storage, 1 gpu (k80) | 21:38 |
| mnaser | but i'm wondering if there is a use case to start with | 21:38 |
| corvus | fungi: yes, though i suspect that the lifetime is so short that it's ineffective | 21:39 |
| clarkb | mnaser: outside of testing the software itself some individuals like devananda were interested in using gpu enabled instances to do machine learning on test logs | 21:39 |
| clarkb | mnaser: we might be able to leverage that type of isntance for those workloads. tristanC is another good person to talk to about that | 21:39 |
| corvus | mnaser, clarkb: tristanC: ^ | 21:40 |
| *** Qiming has quit IRC | 21:40 | |
| fungi | oh, neat | 21:40 |
| clarkb | corvus: jinx | 21:41 |
| fungi | can elasticsearch offload queries to gpu? | 21:41 |
| clarkb | fungi: no, this would likely be external to elasticsearch | 21:41 |
| clarkb | fungi: what the gpu excells at is training on large datasets | 21:41 |
| clarkb | identify the anomalous log line type stuff | 21:42 |
| clarkb | it is also good at running the trained model on new data | 21:42 |
| corvus | clarkb, fungi: https://github.com/elastic/elasticsearch/issues/19148 https://issues.apache.org/jira/browse/LUCENE-7745 | 21:42 |
| mnaser | anyways: i'd prefer it more of CI need rather than a long running instance doing work all the time (maybe a job based periodic trigger) | 21:42 |
| samueldmq | hi, is there any know issue happening to zuul right now? | 21:43 |
| samueldmq | http://zuul.openstack.org/ is not working properly to me | 21:43 |
| mnaser | just because these instances are really expensive so at least with jobs it means that they can somehow shares resources through zuul | 21:43 |
| *** Qiming has joined #openstack-infra | 21:43 | |
| mnaser | samueldmq: loads for me :( | 21:43 |
| mnaser | oh you know what | 21:43 |
| clarkb | mnaser: it could potentially be set up to run periodically (to do the training runs) then if the model is quick enough on cpu we use cpu to apply the trained model | 21:43 |
| mnaser | nope, only the UI loads, no data is showing up | 21:43 |
| clarkb | mnaser: but I think that is all unexplored territory | 21:43 |
| samueldmq | mnaser: exactly | 21:43 |
| clarkb | samueldmq: mnaser see scrollback. The tldr is that zuul can't keep up with the size of the status reporting blob | 21:44 |
| clarkb | zuul is running and executing jobs though | 21:44 |
| samueldmq | clarkb: awesome. thx | 21:45 |
| clarkb | we are about to add more capacity to nodepool too which will hopefully help with the backlog | 21:45 |
| *** eernst has quit IRC | 21:46 | |
| fungi | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=64792&rra_id=all indicates we ran out of room for more cache memory on the server ~3 hours ago | 21:46 |
| *** eernst has joined #openstack-infra | 21:46 | |
| mnaser | clarkb: oh cool, new donor or? | 21:47 |
| samueldmq | clarkb: nice. has the capacity been diminished lately? I'm noticing some patches are taking several hours t o run | 21:47 |
| clarkb | I wrote http://lists.openstack.org/pipermail/openstack-dev/2018-September/134867.html last week to try and explain why we are seeing this behavior | 21:47 |
| clarkb | if you are interested in helping to make the situation better starting there is a good place | 21:47 |
| fungi | mnaser: finally got a viable workaround for unanticipated behavior changes when ovh upgraded to newton | 21:47 |
| mnaser | yeah ovh is a bit of a big donor | 21:47 |
| mnaser | i think we're up to 100 (50 in each region).. hoping to add more soon | 21:47 |
| clarkb | samueldmq: yes last week a cloud upgraded which resulted in new unexpected interactions between neutron, config drive, and how VMs configure networking in that cloud | 21:48 |
| clarkb | this morning we got all the pieces together to work around that and new images have been built and deployed (and as soon as puppet runs we will use that region again) | 21:48 |
| clarkb | samueldmq: the other half of the story is that our testing hasn't been very reliable either | 21:49 |
| samueldmq | clarkb: kk | 21:49 |
| *** boden has quit IRC | 21:49 | |
| clarkb | the combination of fewer resources and flaky gate means a backlog | 21:49 |
| samueldmq | clarkb: do we use shade in nodepool? | 21:49 |
| clarkb | samueldmq: yes, nodepool is the origination of shade actually | 21:49 |
| * samueldmq nods | 21:49 | |
| clarkb | we (mostly mordred) took nodepool and made a library out of all the openstack api interactions it learned and called it shade | 21:50 |
| corvus | clarkb, samueldmq: though now we actually use openstacksdk (which is basically shade) | 21:50 |
| clarkb | ya we use the shade bits in openstacksdk | 21:50 |
| samueldmq | ++ | 21:50 |
| samueldmq | yes I'm familiar with that. I'm actually doing some work in that project too | 21:50 |
| *** agopi has quit IRC | 21:52 | |
| fungi | analyzing the apache access logs, the server is only getting ~1 request per second | 21:55 |
| fungi | over the past hour | 21:56 |
| fungi | though nearly one in three are for /api/status | 21:56 |
| fungi | so basically every 3 seconds on average it's being asked for a copy of that json blob | 21:57 |
| *** eernst has quit IRC | 21:57 | |
| clarkb | I want to say we have attempted at caching in it apache too, but I think the ttl is incredibly short? | 21:57 |
| clarkb | but that may be a way to alleviate the strain on zuul web? | 21:57 |
| fungi | 8MByte max size per cache entry | 21:58 |
| *** eernst has joined #openstack-infra | 21:58 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Web: don't update the status cache more than once https://review.openstack.org/605243 | 21:58 |
| fungi | no, wait, /api/status is CacheMaxFileSize 10000000 | 21:58 |
| corvus | clarkb, fungi: ^ may be relevant | 21:58 |
| clarkb | bhs1 is building nodes now | 21:59 |
| fungi | oh, indeed that could help | 21:59 |
| corvus | fungi: zuul-web sets the cache expiry that apache uses based on its own cache update time, so once it return a fresh blob, apache should use it for the next ~5 seconds, then all subsequent requests will be apache cache misses, and they'll all hit zuul-web which (due to above oversight) will all line up to get the data via gearman. first one to respond makes the apache cache good again, for 5 | 22:01 |
| corvus | seconds. | 22:01 |
| fungi | how do we get status output from geard these days? | 22:01 |
| corvus | fungi: https://docs.openstack.org/infra/system-config/zuul.html#sysadmin | 22:02 |
| fungi | yeah, for some reason that's returning no data for me | 22:02 |
| corvus | fungi: 'status' wfm | 22:03 |
| corvus | zuul:status_get001 | 22:03 |
| corvus | with none of those in the queue, my bug may not be a big contributor | 22:04 |
| fungi | on zuul01 if i `nc localhost 4730` and then enter "status" it immediately returns to my shell prompt again. same with piping | 22:04 |
| corvus | fungi: it requires ssl | 22:04 |
| fungi | oh! | 22:04 |
| corvus | the command is in the docs i linked | 22:04 |
| *** scarab_ has joined #openstack-infra | 22:05 | |
| fungi | indeed it is. i was looking at the one just a little ways down | 22:05 |
| corvus | welp that's out of date | 22:05 |
| fungi | the fourth command block is using netcat | 22:05 |
| fungi | my eye immediately came to rest on that for whatever reason (i had actually pulled up the docs before you linked them and was copying from that page but thought it might be outdated) | 22:05 |
| fungi | i'll push up a patch for the docs | 22:06 |
| *** scarab_ has quit IRC | 22:06 | |
| *** scarab_ has joined #openstack-infra | 22:06 | |
| fungi | but yeah, the reason i was trying to check the gearman queue was indeed to see whether there was a pileup of status_get | 22:06 |
| clarkb | possibly related, ze07 may be afk | 22:07 |
| clarkb | (ssh isn't happening very quickly to it currently) | 22:07 |
| corvus | i did a sigusr2 on zuul-web which worked but seems to have killed it as a side effect | 22:07 |
| corvus | will restart | 22:07 |
| clarkb | noticed this while following up on jobs running on bhs1 | 22:07 |
| *** scarab_ has quit IRC | 22:08 | |
| fungi | clarkb: rackspace opened a ticket to let us know ze02 got rebooted earlier today, but i've seen nothing about ze07 | 22:08 |
| clarkb | hrm this reminds me I set up imap to that email account on laptop but not desktop I'll have to fix that | 22:09 |
| clarkb | what I see so far of bhs1 is that it is working as expected now | 22:09 |
| *** mriedem has quit IRC | 22:10 | |
| corvus | i'm going to try manually running a zuul-web with my patch applied and see how it works | 22:13 |
| fungi | fwiw, killing zuul-web didn't really clear out any cache memory, looks like that's being held by the scheduler process | 22:14 |
| clarkb | looks like the openstack gate just had a huge reset of the entire queue \o/ | 22:19 |
| corvus | doesn't seem to make a big difference | 22:19 |
| clarkb | I think I'm happy with bhs1 now. IF I can help with the zuul web stuff let me know. Otherwise I'll be writing this forum proposal | 22:19 |
| corvus | i'm not worried about zuul-web; i'd like it to be faster, but it's not urgent -- it's contained and not affecting the scheduler | 22:20 |
| corvus | (so it's something to think about and fix, but not drop everything) | 22:20 |
| clarkb | rgr | 22:20 |
| corvus | yay distributed systems :) | 22:21 |
| corvus | actually -- i do see zuul-web dropping to <100% cpu some now. it may be a small improvement. | 22:21 |
| *** bobh has joined #openstack-infra | 22:23 | |
| *** Qiming has quit IRC | 22:23 | |
| *** jamesmcarthur has joined #openstack-infra | 22:23 | |
| corvus | clarkb, fungi: where did you end up on ze07? | 22:24 |
| clarkb | corvus: I pushed it under the stack of getting forum topic submitted since this is time sensitive | 22:24 |
| clarkb | corvus: it does not ping nor does it ssh. I would probably check the console and see if there is anything interesting and if not then reboot it | 22:25 |
| corvus | clarkb: ok. it's probably not critical either :) | 22:25 |
| clarkb | and if there is maybe still reboot it | 22:25 |
| *** Qiming has joined #openstack-infra | 22:26 | |
| corvus | i've restarted the production zuul-web | 22:27 |
| fungi | i haven't looked at ze07 yet | 22:28 |
| fungi | will check the console | 22:29 |
| *** smarcet has quit IRC | 22:32 | |
| *** agopi has joined #openstack-infra | 22:32 | |
| *** Qiming has quit IRC | 22:33 | |
| fungi | grr, forgot rackspace doesn't support `openstack console log show` | 22:34 |
| clarkb | ya have to go to the web ui | 22:35 |
| fungi | will check that on the computer where i have java installed once i'm done stuffing my face | 22:37 |
| *** agopi has quit IRC | 22:37 | |
| clarkb | "presentation social summary is required" | 22:37 |
| clarkb | that seems unnecessary for forum sessions | 22:38 |
| fungi | you can give it an antisocial summary instead | 22:39 |
| ianw | ze07 is shutoff | 22:40 |
| fungi | good eye! | 22:40 |
| ianw | or at least marked as so | 22:40 |
| ianw | but yet the only option under "manage" is to reboot, rather than start, not sure if that's normal | 22:41 |
| fungi | updated 2018-09-14T01:46:12Z | 22:41 |
| clarkb | I tried my hand at turning the abstract into a one liner. I'm sure anne could've made it better | 22:41 |
| fungi | something tells me it hasn't been offline for 11 days | 22:41 |
| *** Qiming has joined #openstack-infra | 22:42 | |
| ianw | the only other one we have shutdown is review.o.o and that only has "reboot" (as opposed to say, start) too | 22:42 |
| clarkb | ianw: I think that may be a rax specific thing. They don't have stop/start in the api | 22:43 |
| fungi | guess i'll issue an openstack server reboot | 22:43 |
| clarkb | ianw: to shutdown you have to run shutdown in the instance itself | 22:43 |
| ianw | fungi: i've got it up, want me to do it? | 22:43 |
| clarkb | this reminds me we should pick a day to remove the old review.o.o | 22:43 |
| fungi | hah! | 22:43 |
| fungi | Cannot 'reboot' instance 538cc4d3-0c0f-402c-a95b-25e6b8f5bdb2 while it is in vm_state stopped (HTTP 409) (Request-ID: req-1f337367-605f-4675-9cc5-c584cb1953ab) | 22:43 |
| fungi | ianw: give it a try | 22:43 |
| clarkb | I wonder if that is a rax administrative state | 22:43 |
| fungi | guessing it's just a terminology mismatch | 22:43 |
| clarkb | cloudnull: ^ maybe you can help with that | 22:43 |
| ianw | i'm on the website as was looking for console | 22:44 |
| fungi | yeah, try it from the dashboard | 22:44 |
| ianw | dunno, says active now | 22:44 |
| ianw | after i "rebooted" it | 22:44 |
| fungi | the api probably wants start or something | 22:44 |
| ianw | ok, seems alive. weird | 22:45 |
| fungi | :modified-shrug-emoticon-with-cloud: | 22:45 |
| clarkb | I'm suddenly unsure if that is a legit emoji and fungi has embraced that style of communication | 22:46 |
| fungi | :sarcastic-descriptive-text: | 22:47 |
| fungi | :emulated-emjoi-string-employed-mockingly: | 22:48 |
| *** dklyle has quit IRC | 22:48 | |
| clarkb | I actually do enjoy imagining what the :string-here: is trying to tell me | 22:49 |
| clarkb | :slightly_smiling_face: vs :smile: | 22:49 |
| corvus | ze07 seems to be doing things according to grafana | 22:50 |
| corvus | wow, despite the backlog, 605243 got results in 25 minutes | 22:51 |
| *** tpsilva has quit IRC | 22:52 | |
| clarkb | corvus: I think what happened is changes that already had their node requests processed enough to be assigned to a rough provider are queued, but new changes are running on bhs1? | 22:53 |
| clarkb | thats a theory, there is defnitely a line cutting behavior in the check queue though | 22:53 |
| corvus | clarkb: well, jobs share a provider, but not all the jobs in a change. so an early change with a job that hasn't gotten an assignment yet can still end up on bhs1 | 22:55 |
| clarkb | the neutron functional job writes a log file for every test | 22:55 |
| corvus | and they are *generally* handled in order | 22:55 |
| corvus | but, if most of the jobs in the queue are multinode jobs waiting on more nodes, then it would behave more like what you describe | 22:55 |
| clarkb | corvus: ah ya single node would filter out more quickly as being easier to provision | 22:56 |
| *** felipemonteiro has joined #openstack-infra | 23:03 | |
| *** rcernin has joined #openstack-infra | 23:07 | |
| *** bobh has quit IRC | 23:08 | |
| *** jamesmcarthur has quit IRC | 23:10 | |
| *** tosky has quit IRC | 23:11 | |
| *** eernst has quit IRC | 23:18 | |
| *** dave-mccowan has joined #openstack-infra | 23:18 | |
| *** agopi has joined #openstack-infra | 23:22 | |
| *** Tim_ok has joined #openstack-infra | 23:37 | |
| *** felipemonteiro has quit IRC | 23:41 | |
| openstackgerrit | Eric Kao proposed openstack-infra/irc-meetings master: Change congress meeting time https://review.openstack.org/605274 | 23:42 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!