| *** mattw4 has quit IRC | 00:05 | |
| *** yamamoto has joined #openstack-infra | 00:14 | |
| *** yamamoto has quit IRC | 00:18 | |
| clarkb | digging into the gitea timeout requests thing a bit more and I think the macaron context type is not interface compatible with the net.http Context interface (it doesn't have any channels to signal Doneness or any channels at all) | 00:30 |
|---|---|---|
| *** gyee has quit IRC | 00:32 | |
| *** aaronsheffield has quit IRC | 00:32 | |
| clarkb | I think macaron may not support these out of band cancellations as a result (at least not with the std lib interface and not with anything I see in macaron itself) | 00:34 |
| *** bobh has joined #openstack-infra | 00:40 | |
| *** armax has joined #openstack-infra | 00:50 | |
| *** markvoelker has joined #openstack-infra | 00:50 | |
| *** markvoelker has quit IRC | 00:55 | |
| clarkb | I see that macaron includes the request in its context which should hvae the std context in it. I may be able to write a handler that replaces that context with a context that times out | 01:09 |
| *** markvoelker has joined #openstack-infra | 01:36 | |
| *** markvoelker has quit IRC | 01:40 | |
| *** ijw has quit IRC | 01:49 | |
| *** bobh has quit IRC | 02:00 | |
| *** yamamoto has joined #openstack-infra | 02:04 | |
| *** yamamoto has quit IRC | 02:08 | |
| *** guoqiao has joined #openstack-infra | 02:19 | |
| *** slaweq has quit IRC | 02:21 | |
| *** markvoelker has joined #openstack-infra | 02:36 | |
| *** markvoelker has quit IRC | 02:40 | |
| *** goldyfruit_ has quit IRC | 03:20 | |
| *** ricolin has joined #openstack-infra | 03:31 | |
| *** dklyle has joined #openstack-infra | 03:35 | |
| *** factor has joined #openstack-infra | 03:38 | |
| *** bhavikdbavishi has joined #openstack-infra | 03:42 | |
| *** ricolin has quit IRC | 03:43 | |
| *** ricolin has joined #openstack-infra | 03:48 | |
| *** rh-jelabarre has quit IRC | 04:02 | |
| *** markvoelker has joined #openstack-infra | 04:20 | |
| *** markvoelker has quit IRC | 04:25 | |
| *** bhavikdbavishi has quit IRC | 04:28 | |
| *** ricolin has quit IRC | 04:36 | |
| *** xenos76 has joined #openstack-infra | 04:45 | |
| *** ricolin has joined #openstack-infra | 04:50 | |
| *** ociuhandu has joined #openstack-infra | 04:57 | |
| *** ociuhandu has quit IRC | 05:01 | |
| *** dklyle has quit IRC | 05:03 | |
| *** slaweq has joined #openstack-infra | 05:11 | |
| *** slaweq has quit IRC | 05:15 | |
| *** ricolin has quit IRC | 05:18 | |
| *** xenos76 has quit IRC | 05:45 | |
| *** slaweq has joined #openstack-infra | 06:11 | |
| *** slaweq has quit IRC | 06:15 | |
| *** xenos76 has joined #openstack-infra | 06:48 | |
| *** slaweq has joined #openstack-infra | 06:55 | |
| *** ociuhandu has joined #openstack-infra | 07:00 | |
| *** ociuhandu has quit IRC | 07:06 | |
| *** slaweq has quit IRC | 07:17 | |
| *** e0ne has joined #openstack-infra | 07:33 | |
| *** e0ne has quit IRC | 07:39 | |
| *** yamamoto has joined #openstack-infra | 07:56 | |
| *** slaweq has joined #openstack-infra | 07:59 | |
| *** xenos76 has quit IRC | 08:03 | |
| *** yolanda__ is now known as yolanda | 08:08 | |
| *** slaweq has quit IRC | 08:17 | |
| *** jtomasek has joined #openstack-infra | 08:23 | |
| *** yamamoto has quit IRC | 08:25 | |
| *** xenos76 has joined #openstack-infra | 08:26 | |
| *** roman_g has joined #openstack-infra | 08:31 | |
| *** slaweq has joined #openstack-infra | 08:46 | |
| *** slaweq has quit IRC | 08:51 | |
| *** yamamoto has joined #openstack-infra | 08:53 | |
| *** yamamoto has quit IRC | 08:57 | |
| *** roman_g has quit IRC | 09:19 | |
| *** markvoelker has joined #openstack-infra | 09:35 | |
| *** markvoelker has quit IRC | 09:40 | |
| yoctozepto | hey infra, from time to time our jobs get nodes with no ipv4 addresses: https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/logs_05/676905/3/check/kolla-ansible-centos-source-upgrade/229c1c1/ara-report/ | 10:06 |
| yoctozepto | is this to be expected? | 10:06 |
| yoctozepto | this is most painful if it hits on voting or gate because there is a loss of lots of time as we cannot recover from this normally | 10:07 |
| *** guoqiao has quit IRC | 10:07 | |
| *** guoqiao has joined #openstack-infra | 10:08 | |
| yoctozepto | we plan to support ipv6-only deploys soon so we should be able to mitigate this issue but still it would be better for the jobs to not start if they cannot allocate addresses rather than fail | 10:09 |
| yoctozepto | please let me know | 10:09 |
| *** ociuhandu has joined #openstack-infra | 10:09 | |
| *** ociuhandu has quit IRC | 10:13 | |
| *** xenos76 has quit IRC | 10:27 | |
| AJaeger | yoctozepto: getting no ipv4 is valid, read https://docs.openstack.org/infra/manual/testing.html | 10:32 |
| yoctozepto | AJaeger: so there are no private addresses ever assigned? | 10:35 |
| yoctozepto | because we don't really require them public | 10:35 |
| yoctozepto | the error for us is that | 10:38 |
| yoctozepto | we do an assertion | 10:38 |
| yoctozepto | nodepool.private_ipv4 in ansible_all_ipv4_addresses | 10:38 |
| yoctozepto | and get | 10:38 |
| yoctozepto | The nodepool private IP address 10.4.70.78 is not assigned | 10:38 |
| yoctozepto | for example | 10:38 |
| yoctozepto | (we added the assertion because otherwise our jobs break funny because of that) | 10:38 |
| yoctozepto | not having a private IPv4 address is fine ;-) | 10:39 |
| yoctozepto | not* fine | 10:39 |
| yoctozepto | :D | 10:39 |
| *** noorul has joined #openstack-infra | 11:01 | |
| openstackgerrit | Sorin Sbarnea proposed openstack/diskimage-builder master: Fixed use of flake8 https://review.opendev.org/677059 | 11:10 |
| *** altlogbot_2 has quit IRC | 11:10 | |
| *** altlogbot_3 has joined #openstack-infra | 11:11 | |
| *** noorul has quit IRC | 11:15 | |
| *** noorul has joined #openstack-infra | 11:15 | |
| *** noorul has quit IRC | 11:25 | |
| *** yamamoto has joined #openstack-infra | 11:35 | |
| *** ociuhandu has joined #openstack-infra | 11:39 | |
| *** yamamoto has quit IRC | 11:40 | |
| *** noorul has joined #openstack-infra | 11:54 | |
| *** ociuhandu has quit IRC | 11:57 | |
| *** noorul has quit IRC | 12:00 | |
| *** jtomasek has quit IRC | 12:22 | |
| *** ociuhandu has joined #openstack-infra | 12:23 | |
| *** ociuhandu has quit IRC | 12:38 | |
| *** bhavikdbavishi has joined #openstack-infra | 12:39 | |
| *** bhavikdbavishi has quit IRC | 12:45 | |
| *** bhavikdbavishi has joined #openstack-infra | 13:03 | |
| *** markvoelker has joined #openstack-infra | 13:10 | |
| *** bobh has joined #openstack-infra | 13:13 | |
| *** ociuhandu has joined #openstack-infra | 13:24 | |
| *** markvoelker has quit IRC | 13:24 | |
| *** markvoelker has joined #openstack-infra | 13:25 | |
| openstackgerrit | Donny Davis proposed openstack/project-config master: Local NVME drives on FN complete - remove boot from volume https://review.opendev.org/677062 | 13:28 |
| *** ociuhandu has quit IRC | 13:28 | |
| clarkb | yoctozepto: AJaeger all of our hosta should either have a public or a private ipv4 address currently even on ipv6 only clouds (they get private ipv4 for NATing outbound ipv4 traffic) | 13:29 |
| *** markvoelker has quit IRC | 13:30 | |
| clarkb | but you may not have both asin the case of jobs on ovh there are only public addresses | 13:30 |
| donnyd | clarkb: I pulled the boot from volume parameter from FN. I *finally* got all my nvme drives locally installed and functional | 13:33 |
| donnyd | only took 3 weeks | 13:33 |
| *** bobh has quit IRC | 13:35 | |
| *** guoqiao has quit IRC | 13:38 | |
| *** ianychoi_ has joined #openstack-infra | 13:39 | |
| *** ianychoi has quit IRC | 13:40 | |
| *** xenos76 has joined #openstack-infra | 13:48 | |
| *** ijw has joined #openstack-infra | 13:51 | |
| *** bobh has joined #openstack-infra | 13:51 | |
| *** ijw has quit IRC | 13:55 | |
| *** bobh has quit IRC | 14:08 | |
| *** bobh has joined #openstack-infra | 14:09 | |
| *** noorul has joined #openstack-infra | 14:12 | |
| *** e0ne has joined #openstack-infra | 14:13 | |
| *** bobh has quit IRC | 14:13 | |
| *** e0ne has quit IRC | 14:19 | |
| *** bhavikdbavishi has quit IRC | 14:29 | |
| *** noorul has quit IRC | 14:31 | |
| *** noorul has joined #openstack-infra | 14:32 | |
| *** noorul has quit IRC | 14:38 | |
| *** noorul has joined #openstack-infra | 14:38 | |
| *** markvoelker has joined #openstack-infra | 14:41 | |
| AJaeger | clarkb: thanks for correction | 14:41 |
| *** bhavikdbavishi has joined #openstack-infra | 14:41 | |
| AJaeger | donnyd: so, want to have 677062 merged quickly? Hope some config-core is around ... | 14:42 |
| *** markvoelker has quit IRC | 14:45 | |
| * mnaser on mobile and checks | 14:45 | |
| mnaser | Lgtm thanks donnyd and AJaeger | 14:48 |
| donnyd | Well load is pretty low right now, so whenever people are up and around it would be good to get it merged | 14:54 |
| donnyd | mnaser: though it was painful to get local nvme storage, I think you are right and it will be worth it | 14:54 |
| donnyd | blades are so painful to work with for things like that | 14:54 |
| openstackgerrit | Merged openstack/project-config master: Local NVME drives on FN complete - remove boot from volume https://review.opendev.org/677062 | 14:59 |
| *** xenos76 has quit IRC | 15:01 | |
| *** xenos76 has joined #openstack-infra | 15:03 | |
| *** yolanda has quit IRC | 15:06 | |
| *** bhavikdbavishi has quit IRC | 15:07 | |
| *** cmorpheus is now known as cmurphy | 15:09 | |
| donnyd | Thank AJaeger mnaser | 15:10 |
| donnyd | Thanks AJaeger mnaser | 15:10 |
| *** factor has quit IRC | 15:11 | |
| *** factor has joined #openstack-infra | 15:11 | |
| AJaeger | thank you, donnyd for the cloud resources! | 15:12 |
| logan- | yoctozepto: something is wrong there.. the cloud assigned an ipv4 but it never came up on the node. | 15:15 |
| *** e0ne has joined #openstack-infra | 15:18 | |
| *** e0ne has quit IRC | 15:19 | |
| *** yolanda has joined #openstack-infra | 15:20 | |
| logan- | it seems like the logstash results stop around the time logging moved to object storage | 15:20 |
| logan- | http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22TASK%20%5Bconfigure-unbound%20:%20Set%20IPv4%20nameservers%5D%5C%22&from=5d | 15:20 |
| *** icarusfactor has joined #openstack-infra | 15:24 | |
| *** factor has quit IRC | 15:24 | |
| *** markvoelker has joined #openstack-infra | 15:40 | |
| *** markvoelker has quit IRC | 15:50 | |
| *** factor has joined #openstack-infra | 15:53 | |
| *** icarusfactor has quit IRC | 15:54 | |
| *** yamamoto has joined #openstack-infra | 15:54 | |
| *** ianychoi has joined #openstack-infra | 15:54 | |
| *** bobh has joined #openstack-infra | 15:57 | |
| *** ianychoi_ has quit IRC | 15:57 | |
| *** ianychoi_ has joined #openstack-infra | 15:58 | |
| *** yamamoto has quit IRC | 15:58 | |
| *** icarusfactor has joined #openstack-infra | 15:59 | |
| *** e0ne has joined #openstack-infra | 16:00 | |
| *** bobh has quit IRC | 16:01 | |
| *** factor has quit IRC | 16:01 | |
| *** ianychoi has quit IRC | 16:01 | |
| *** ianychoi_ has quit IRC | 16:04 | |
| *** icarusfactor has quit IRC | 16:13 | |
| *** icarusfactor has joined #openstack-infra | 16:13 | |
| *** bhavikdbavishi has joined #openstack-infra | 16:24 | |
| *** e0ne has quit IRC | 16:31 | |
| *** noorul has quit IRC | 16:33 | |
| *** bobh has joined #openstack-infra | 16:34 | |
| *** e0ne has joined #openstack-infra | 16:34 | |
| *** bobh has quit IRC | 16:40 | |
| *** e0ne has quit IRC | 16:50 | |
| *** ociuhandu has joined #openstack-infra | 17:17 | |
| *** ociuhandu has quit IRC | 17:18 | |
| *** ociuhandu has joined #openstack-infra | 17:18 | |
| *** ociuhandu has quit IRC | 17:22 | |
| efried | infra_root <== is that a tag that gets people's attention? | 17:40 |
| *** bobh has joined #openstack-infra | 17:40 | |
| efried | clarkb: are you "on duty"? | 17:43 |
| *** bobh has quit IRC | 17:44 | |
| efried | is it infra-core? | 17:44 |
| AJaeger | efried: infra-root it is ;) | 17:46 |
| AJaeger | and nobody is on duty... | 17:46 |
| efried | AJaeger: Thanks. I need someone who can 911 approve a requirements fix | 17:46 |
| AJaeger | efried: better leave a message... | 17:46 |
| AJaeger | efried: there's a requirements team - did you leave a message in their channel? | 17:47 |
| efried | https://review.opendev.org/#/c/677070/ -- see http://lists.openstack.org/pipermail/openstack-discuss/2019-August/008553.html | 17:47 |
| AJaeger | efried: I can't help | 17:47 |
| efried | I shotgun-pinged some people there. | 17:47 |
| efried | I'll leave this info there as well. | 17:47 |
| efried | Thanks AJaeger | 17:47 |
| *** e0ne has joined #openstack-infra | 17:53 | |
| yoctozepto | AJaeger, clarkb, logan-: so can you fix it somehow now that we know it's bad? :-) | 18:30 |
| yoctozepto | here are some more failed jobs | 18:30 |
| yoctozepto | those without assert yet | 18:30 |
| yoctozepto | https://zuul.opendev.org/t/openstack/build/b6f2d45aeb374ae687e8ca9a11f9cd51 | 18:30 |
| yoctozepto | https://zuul.opendev.org/t/openstack/build/7a5a349d3cfe49fd9801e97164f6f4ba | 18:30 |
| *** yamamoto has joined #openstack-infra | 18:36 | |
| *** yamamoto has quit IRC | 18:41 | |
| clarkb | efried: is the gap that we only test constraints against python27 and 36? | 18:50 |
| clarkb | yoctozepto: the first failure was due to api_interface_name being undefined? | 18:50 |
| clarkb | I'm not surehow that relates to asserts | 18:50 |
| clarkb | it would be helpful if you can link specifically to where you think the test node is misconfigured | 18:53 |
| *** ociuhandu has joined #openstack-infra | 19:25 | |
| *** hongbin has joined #openstack-infra | 19:29 | |
| logan- | clarkb: heres what went wrong in that job. you can see the cloud assigned a private_ipv4 and it is in the inventory here https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/logs_05/676905/3/check/kolla-ansible-centos-source-upgrade/229c1c1/zuul-info/inventory.yaml | 19:32 |
| logan- | but in the facts gathered from the host you can see ansible_all_ipv4_addresses is empty https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/logs_05/676905/3/check/kolla-ansible-centos-source-upgrade/229c1c1/zuul-info/host-info.primary.yaml | 19:33 |
| logan- | i think this is likely the cause of https://bugs.launchpad.net/tripleo/+bug/1839845 also | 19:33 |
| openstack | Launchpad bug 1839845 in tripleo "TripleO gate jobs run on limestone nodes are failing "to retrieve repo file from https://trunk.rdoproject.org/centos7-<release>/current-tripleo/delorean.repo" after 10 retries " [High,Triaged] | 19:33 |
| *** ociuhandu has quit IRC | 19:40 | |
| clarkb | ovh doesnt do private addressesthough | 19:44 |
| clarkb | maybe Im confusing log storage with test node region /me looks more closely | 19:44 |
| logan- | ya job ran on limestone | 19:44 |
| *** pkopec has joined #openstack-infra | 19:45 | |
| clarkb | logan-: ipv4 should be set by dhcp in your cloud or statically from config drive info? | 19:47 |
| logan- | either way should work i think | 19:48 |
| clarkb | ok we'll have to debug where that is failing on centos images then (andpossibly others if more widespread) | 19:49 |
| *** bhavikdbavishi has quit IRC | 19:56 | |
| *** bhavikdbavishi has joined #openstack-infra | 19:57 | |
| fungi | clarkb: sorry, hadn't caught up in here before replying to the latest listserv postmaster questions, but i concur with your analysis. we'll see what they come back with now that i realize it was a >1-month-old problem | 20:01 |
| yoctozepto | logan-: thanks for clarifying that to clarkb | 20:07 |
| yoctozepto | I was not precise about asserts | 20:07 |
| *** bhavikdbavishi has quit IRC | 20:07 | |
| yoctozepto | I meant that some older jobs we have don't have the nice assert to catch this situation | 20:07 |
| yoctozepto | so it results in various later internal failures | 20:07 |
| yoctozepto | the previous previous job had the assert failure | 20:07 |
| yoctozepto | that was what I was about | 20:08 |
| yoctozepto | and the problem is | 20:08 |
| yoctozepto | when you look at hosts facts | 20:08 |
| yoctozepto | there are no IPv4 addresses assigned whatsoever | 20:08 |
| yoctozepto | not even the nodepool private one that is promised by Zuul | 20:08 |
| yoctozepto | hence problems for us | 20:08 |
| *** Lucas_Gray has joined #openstack-infra | 20:09 | |
| *** pcrews has joined #openstack-infra | 20:18 | |
| *** mattw4 has joined #openstack-infra | 20:22 | |
| *** yamamoto has joined #openstack-infra | 20:27 | |
| *** mattw4 has quit IRC | 20:27 | |
| *** mattw4 has joined #openstack-infra | 20:27 | |
| *** yamamoto has quit IRC | 20:31 | |
| *** Lucas_Gray has quit IRC | 20:33 | |
| *** Lucas_Gray has joined #openstack-infra | 20:34 | |
| *** pkopec has quit IRC | 20:36 | |
| *** ociuhandu has joined #openstack-infra | 20:50 | |
| *** ociuhandu has quit IRC | 21:30 | |
| *** smcginnis has joined #openstack-infra | 21:32 | |
| *** markvoelker has joined #openstack-infra | 21:40 | |
| *** markvoelker has quit IRC | 21:45 | |
| *** prometheanfire has joined #openstack-infra | 21:52 | |
| *** prometheanfire has quit IRC | 22:00 | |
| *** e0ne has quit IRC | 22:02 | |
| *** goldyfruit_ has joined #openstack-infra | 22:14 | |
| *** xenos76 has quit IRC | 22:21 | |
| *** ianychoi has joined #openstack-infra | 22:23 | |
| *** markvoelker has joined #openstack-infra | 22:31 | |
| *** armax has joined #openstack-infra | 22:32 | |
| *** prometheanfire has joined #openstack-infra | 22:34 | |
| *** markvoelker has quit IRC | 22:42 | |
| *** markvoelker has joined #openstack-infra | 22:43 | |
| *** markvoelker has quit IRC | 22:47 | |
| *** hongbin has quit IRC | 22:53 | |
| *** ociuhandu has joined #openstack-infra | 23:00 | |
| *** ociuhandu has quit IRC | 23:05 | |
| *** Lucas_Gray has quit IRC | 23:13 | |
| *** armax has quit IRC | 23:33 | |
| *** ijw has joined #openstack-infra | 23:45 | |
| *** ijw has quit IRC | 23:49 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!