| openstackgerrit | Clark Boylan proposed opendev/system-config master: Manage opendev.org cert with LE https://review.opendev.org/694181 | 00:03 |
|---|---|---|
| clarkb | fungi: ianw can you check the comments I left on https://review.opendev.org/#/c/694181/1 I think those are my major open questions | 00:05 |
| *** kjackal has quit IRC | 00:05 | |
| clarkb | otherwise I think the shape of the thing is correct and this should work | 00:05 |
| *** dchen has quit IRC | 00:11 | |
| openstackgerrit | Clark Boylan proposed opendev/zone-opendev.org master: Add gitea acme records https://review.opendev.org/694182 | 00:13 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Manage opendev.org cert with LE https://review.opendev.org/694181 | 00:13 |
| *** dchen has joined #openstack-infra | 00:18 | |
| openstackgerrit | Clark Boylan proposed opendev/base-jobs master: Add cleanup task timeouts to production playbooks https://review.opendev.org/694183 | 00:25 |
| clarkb | config-core I believe ^ to be tested now | 00:26 |
| ianw | clarkb: should we perhaps try it with one host first? | 00:28 |
| clarkb | ianw: I did think about that. I think the bit of code I delete in the roles/gitea/tasks/main.yaml file makes that a bit tricky (though I suppsoe as long as we don't provision any new giteas we can safely remove that bit and test on a single host) | 00:28 |
| clarkb | That seems like a reasonable idea. I can go ahead and split the change into two for that | 00:29 |
| ianw | it looks about right to me, but yeah, i'd hate to just wipe out opendev.org in one go :) | 00:29 |
| *** tosky has quit IRC | 00:36 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Manage opendev.org cert with LE https://review.opendev.org/694181 | 00:36 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Manage opendev.org with LE on all giteas https://review.opendev.org/694184 | 00:36 |
| clarkb | ianw: ^ splitting is done | 00:37 |
| *** mriedem has quit IRC | 00:38 | |
| ianw | do we have gitea testinfra tests? | 00:40 |
| clarkb | yes we do project creation in them and check all that iirc | 00:40 |
| clarkb | ianw: also can you check my comment on https://review.opendev.org/#/c/694182/1 I think I need to respin the zone update too | 00:40 |
| ianw | cool, in that case i think this should be self testing too | 00:40 |
| clarkb | ya I think I've convinced myself I need two zone updates | 00:41 |
| clarkb | new patchset/change incoming | 00:41 |
| ianw | clarkb: what's the race? | 00:42 |
| ianw | doesn't seem that an opendev.org cert will work unless _acme-challenge is setup for it? | 00:43 |
| clarkb | ianw: with the CAA records | 00:45 |
| ianw | ohhh, right, ok | 00:45 |
| clarkb | since we are doing one host first we don't want to assert that opendev.org is always CAA's by LE until all 8 are done | 00:45 |
| openstackgerrit | Clark Boylan proposed opendev/zone-opendev.org master: Add gitea acme records https://review.opendev.org/694182 | 00:46 |
| openstackgerrit | Clark Boylan proposed opendev/zone-opendev.org master: Add cert issuer validation to opendev.org https://review.opendev.org/694185 | 00:46 |
| clarkb | that stack should handle the race | 00:46 |
| clarkb | an upside to this method of cert issuance is we will be able to clone from https://gitea01.opendev.org:3000 with ssl verification when this is done :) | 00:47 |
| clarkb | that has been super useful for identifying unhappy gitea's but I've always had to disable verification (GIT_SSL_NO_VERIFY=1 iirc) | 00:48 |
| ianw | gitea_https = host.socket("tcp://0.0.0.0:3000") | 00:48 |
| ianw | that should be up & with a self-signed cert for testinfra | 00:48 |
| ianw | you could try curl'ing from it as an extra test, but i guess if it's listening it's found it's cert | 00:49 |
| clarkb | oh except that we use a host called gitea99 | 00:49 |
| clarkb | in testinfra | 00:49 |
| clarkb | and I don't think I configured LE stuff for that host in my all giteas change | 00:49 |
| clarkb | I'm working on an update to actually configure gitea99 too | 00:51 |
| ianw | clarkb: yeah, i think it will probably fail then, as gitea_tls_key isn't written out any more, and LE roles aren't generated the fake cert | 00:51 |
| *** yamamoto has joined #openstack-infra | 00:57 | |
| *** goldyfruit has joined #openstack-infra | 00:58 | |
| *** kopecmartin is now known as kopecmartin|off | 00:59 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Manage opendev.org cert with LE https://review.opendev.org/694181 | 00:59 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Manage opendev.org with LE on all giteas https://review.opendev.org/694184 | 00:59 |
| clarkb | ianw: ^ I think that may solve the testing problem | 00:59 |
| *** zhurong has quit IRC | 01:00 | |
| clarkb | first chnage manages gitea01 in prod and gitea99 in test. Then followup manages all of them | 01:00 |
| clarkb | ++ to cleaning things up (just noticed your ps3 comments) I'll get that up then go find dinner | 01:01 |
| ianw | i really like how this should be gate tested :) it's a good setup | 01:02 |
| *** yamamoto has quit IRC | 01:02 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Manage opendev.org cert with LE https://review.opendev.org/694181 | 01:04 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Manage opendev.org with LE on all giteas https://review.opendev.org/694184 | 01:04 |
| clarkb | that should be caught up on all outstanding comments that I have seen (here and in review) | 01:05 |
| clarkb | I need to find dinner now, but hopefully we get back useful test results next | 01:05 |
| clarkb | actually there are a few potential issues :( | 01:06 |
| clarkb | first if LE runs after gitea in the order of roles then we may not be able to write the certs to the locations we need them in | 01:06 |
| clarkb | I think we can address that by having roles/gitea copy the certs from LE location in addition to having handler write them out | 01:07 |
| clarkb | the other is a bug that I'll push new PSs momentarily to fix | 01:07 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Manage opendev.org cert with LE https://review.opendev.org/694181 | 01:08 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Manage opendev.org with LE on all giteas https://review.opendev.org/694184 | 01:08 |
| clarkb | I mismatched the cert and handler names | 01:08 |
| clarkb | that is now fixed. For the other thing I kinda want to wait on test results to see if it is actually a problem | 01:09 |
| ianw | np | 01:09 |
| clarkb | I think this is why the restart apache handlers check that apache is running before restarting it (avoids failing on the bootstrap problem) | 01:10 |
| clarkb | I think what I want to do is have my handler write out the /var/gitea/certs dir if not already present and check if docker/gitea are running prior to running the restart | 01:11 |
| clarkb | but I really do need to find food now so can pick that up tomorrow | 01:11 |
| clarkb | and ya huge ++ to the testing here | 01:11 |
| *** diablo_rojo__ has quit IRC | 01:16 | |
| *** Goneri has quit IRC | 01:20 | |
| *** ricolin has joined #openstack-infra | 01:25 | |
| *** kjackal has joined #openstack-infra | 01:26 | |
| fungi | ianw: if you're done running edit-secrets on bridge can you exit it? keeping that emacs session open holds a gpg-agent running so nobody else can start one | 01:28 |
| ianw | fungi: sorry, yes exited | 01:28 |
| fungi | no worries, thanks! | 01:29 |
| ianw | fungi: that was to make secret for dib docker image upload; if you have a sec could you look in on https://review.opendev.org/#/c/694178/ which would allow testing of those potential images in nodepool | 01:30 |
| *** eernst has quit IRC | 01:30 | |
| fungi | yep, looking | 01:30 |
| *** hwoarang has quit IRC | 01:34 | |
| ianw | thanks, i think that gets automatically reloaded? we'll see anyway | 01:35 |
| *** hwoarang has joined #openstack-infra | 01:36 | |
| fungi | tenant configuration does get automatically reloaded, but by ansible/puppet after the new files gets deployed | 01:36 |
| fungi | so it's not instant at merge | 01:37 |
| ianw | cool; it's been a docker image building crash course lately :) | 01:37 |
| *** kjackal has quit IRC | 01:38 | |
| fungi | hopefully not a docker image build crashing course | 01:41 |
| ianw | haha, yes plenty of that too :) | 01:42 |
| openstackgerrit | Merged openstack/project-config master: zuul layout: include dib in zuul tenant for jobs https://review.opendev.org/694178 | 01:43 |
| *** gyee has quit IRC | 01:46 | |
| *** kjackal has joined #openstack-infra | 01:50 | |
| *** amoralej|off has quit IRC | 01:50 | |
| *** ociuhandu has joined #openstack-infra | 01:54 | |
| *** radez has quit IRC | 01:54 | |
| *** jpena|off has quit IRC | 01:54 | |
| *** ociuhandu has quit IRC | 01:59 | |
| *** rkukura has quit IRC | 02:04 | |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: [dnm] testing docker registry with dib https://review.opendev.org/694177 | 02:06 |
| ianw | cool, config works at leats | 02:07 |
| *** kjackal has quit IRC | 02:11 | |
| *** ykarel has joined #openstack-infra | 02:13 | |
| *** rlandy|rover|bbl has quit IRC | 02:14 | |
| *** rlandy|rover|bbl has joined #openstack-infra | 02:15 | |
| *** ykarel has quit IRC | 02:21 | |
| *** ociuhandu has joined #openstack-infra | 02:28 | |
| *** rh-jelabarre has quit IRC | 02:30 | |
| *** ociuhandu has quit IRC | 02:32 | |
| *** roman_g has quit IRC | 02:33 | |
| *** kjackal has joined #openstack-infra | 02:41 | |
| *** yamamoto has joined #openstack-infra | 02:47 | |
| *** yamamoto has quit IRC | 02:49 | |
| *** yamamoto has joined #openstack-infra | 02:49 | |
| *** xinranwang has joined #openstack-infra | 02:50 | |
| *** kjackal has quit IRC | 02:59 | |
| *** ramishra has joined #openstack-infra | 03:08 | |
| *** rh-jelabarre has joined #openstack-infra | 03:09 | |
| *** tinwood has quit IRC | 03:10 | |
| *** yamamoto has quit IRC | 03:11 | |
| *** tinwood has joined #openstack-infra | 03:12 | |
| *** yamamoto has joined #openstack-infra | 03:12 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Add a Dockerfile and related jobs https://review.opendev.org/693971 | 03:22 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Update bindep.txt for some missing dependencies https://review.opendev.org/693975 | 03:22 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Dockerfile: install vhd-util https://review.opendev.org/693976 | 03:22 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Start a buildset registry with docker image creation https://review.opendev.org/694175 | 03:22 |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: [dnm] testing docker registry with dib https://review.opendev.org/694177 | 03:26 |
| *** psachin has joined #openstack-infra | 03:38 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Add a Dockerfile and related jobs https://review.opendev.org/693971 | 03:39 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Update bindep.txt for some missing dependencies https://review.opendev.org/693975 | 03:39 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Dockerfile: install vhd-util https://review.opendev.org/693976 | 03:39 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Start a buildset registry with docker image creation https://review.opendev.org/694175 | 03:40 |
| *** rlandy|rover|bbl is now known as rlandy|rover | 03:43 | |
| *** udesale has joined #openstack-infra | 03:50 | |
| *** dchen has quit IRC | 03:50 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Manage opendev.org cert with LE https://review.opendev.org/694181 | 03:52 |
| ianw | clarkb: ^ just added the letsencrypt playbook to the gitea test | 03:52 |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: [dnm] testing docker registry with dib https://review.opendev.org/694177 | 04:14 |
| *** rlandy|rover has quit IRC | 04:15 | |
| *** ykarel has joined #openstack-infra | 04:24 | |
| *** ociuhandu has joined #openstack-infra | 04:25 | |
| *** ykarel_ has joined #openstack-infra | 04:26 | |
| *** raukadah is now known as chandankumar | 04:27 | |
| *** dchen has joined #openstack-infra | 04:28 | |
| *** ykarel has quit IRC | 04:29 | |
| *** ociuhandu has quit IRC | 04:30 | |
| tonyb | Is there any chnace the the (nova) git repos on 38.108.68.124 are ... confised? I've been tryign to git remote update (nova) for like most of the day. I don't *think* the problem is local | 04:42 |
| *** yamamoto has quit IRC | 04:45 | |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: Dockerfile: create APP_DIR https://review.opendev.org/693646 | 04:47 |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: docker: Base nodepool images on diskimage-builder base image https://review.opendev.org/694177 | 04:47 |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: [wip] move openstack testing to use containerised daemon https://review.opendev.org/693464 | 04:47 |
| ianw | tonyb: hrm, so that's the load-balancer, it will be sent out to the backends from there | 04:48 |
| ianw | do you have a timestamp maybe? | 04:49 |
| tonyb | Date: Thu, 14 Nov 2019 04:49:02 GMT | 04:49 |
| tonyb | from the headers of the current "POST /openstack/nova/git-upload-pack HTTP/1.1" | 04:50 |
| *** ociuhandu has joined #openstack-infra | 04:50 | |
| *** dtantsur|afk has quit IRC | 04:50 | |
| *** kjackal has joined #openstack-infra | 04:50 | |
| *** ykarel_ is now known as ykarel|mt | 04:51 | |
| *** ykarel|mt is now known as ykarel|mtg | 04:51 | |
| ianw | hrm, i'm not sure where the logs go with haproxy containerised .. looking | 04:52 |
| ianw | yeah, i'm not sure how to correlate that back to the server it went to | 04:54 |
| *** goldyfruit has quit IRC | 04:54 | |
| *** ociuhandu has quit IRC | 04:54 | |
| tonyb | Okay | 04:55 |
| tonyb | Can I tweak my config to hit a specific server? | 04:55 |
| ianw | unfortunately the "docker log" output isn't timestampped | 04:55 |
| *** dtantsur has joined #openstack-infra | 04:55 | |
| ianw | tonyb: yeah, you can go to https://gitea0[1-8].opendev.org:3000/ | 04:56 |
| ianw | but the https cert isn't valid (clarkb is working on that :) | 04:56 |
| tonyb | ianw: I have a couple of cookies which might help? ... no idea what is in docker log | 04:57 |
| ianw | it's just showing various up/down lost connections to the backends | 04:57 |
| ianw | but with no timestamp, that doesn't help that either | 04:58 |
| tonyb | Ahh okay | 04:58 |
| tonyb | So without changing anything git fetch gerrit; git remote update 'fixed' it | 04:59 |
| tonyb | I guess grabbing the objects from gerrit via ssh meant that they weren't needed via origin/https | 04:59 |
| *** kjackal has quit IRC | 05:00 | |
| *** ociuhandu has joined #openstack-infra | 05:02 | |
| ianw | bah, i see what's going on | 05:06 |
| ianw | it sends logs to /dev/log | 05:06 |
| ianw | which ends up in /var/log/syslog on the host | 05:06 |
| ianw | i don't think that's really what we want | 05:06 |
| tonyb | Yeah does make them a little harder to find/colate | 05:07 |
| ianw | tonyb: any of these you? http://paste.openstack.org/show/786091/ | 05:07 |
| tonyb | ianw: I know clone and update are different operations but I'll try cloning direct (not via the LB) and see if one fails | 05:08 |
| tonyb | ianw: nope | 05:09 |
| tonyb | 121.45.215.66 | 05:09 |
| ianw | hrm, that's all for that second on the lb ... time skew? | 05:09 |
| tonyb | Yeah possibly | 05:10 |
| ianw | Nov 14 04:58:09 gitea-lb01 haproxy[6]: ::ffff:121.45.215.66:33178 [14/Nov/2019:04:58:04.272] balance_git_https balance_git_https/gitea06.opendev.org 1/0/4744 12441077 -- 82/82/81/20/0 0/0 | 05:10 |
| ianw | Nov 14 04:58:40 gitea-lb01 haproxy[6]: ::ffff:121.45.215.66:33180 [14/Nov/2019:04:58:36.951] balance_git_https balance_git_https/gitea06.opendev.org 1/0/3852 12441077 -- 79/78/77/21/0 0/0 | 05:10 |
| ianw | all requests from that host are going to gitea06 | 05:11 |
| tonyb | okay | 05:11 |
| tonyb | that's good data | 05:11 |
| *** sshnaidm|ruck has quit IRC | 05:11 | |
| *** sshnaidm_ has joined #openstack-infra | 05:12 | |
| ianw | 2019/11/13 04:58:06 routers/repo/http.go:430:serviceRPC() [E] Fail to serve RPC(upload-pack): exit status 128 | 05:12 |
| *** ociuhandu has quit IRC | 05:12 | |
| ianw | there's a lot of this, not sure if it's a red herring or not | 05:12 |
| tonyb | Well it was upload-pack that failed/hung | 05:13 |
| ianw | 2019/11/14 04:58:40 routers/repo/http.go:430:serviceRPC() [E] Fail to serve RPC(upload-pack): exit status 128 - {[102 97 116 97 108 58 32 116 104 101 32 114 101 109 111 116 101 32 101 110 100 32 104 117 110 103 32 117 112 32 117 110 101 120 112 101 99 116 101 100 108 121 10] 0 0} | 05:13 |
| ianw | right at that time | 05:13 |
| ianw | yep, i think gitea06 is sick, for some reason | 05:14 |
| tonyb | I'm trying a clone now | 05:14 |
| *** dchen has quit IRC | 05:15 | |
| *** dchen has joined #openstack-infra | 05:16 | |
| tonyb | Hmm that worked but as I said before clone is a different operation | 05:16 |
| *** rh-jelabarre has quit IRC | 05:16 | |
| ianw | i can't really find any issues or hints about this | 05:18 |
| *** yamamoto has joined #openstack-infra | 05:19 | |
| ianw | it started when the gitea server started | 05:19 |
| tonyb | Perhaps it isn't a big problem | 05:19 |
| ianw | something is wrong, i'm running out of time to investigate fully right now | 05:20 |
| ianw | i can file something and take it out of rotation for the A team to look at | 05:20 |
| tonyb | okay, that sounds fair | 05:20 |
| tonyb | ianw: I didn't mean to drop this on you | 05:21 |
| *** yamamoto has quit IRC | 05:24 | |
| *** yamamoto has joined #openstack-infra | 05:27 | |
| *** ociuhandu has joined #openstack-infra | 05:30 | |
| ianw | #status log gitea06 showing upload-pack errors per : https://storyboard.openstack.org/#!/story/2006849. i have disabled it in the load balancer so we can investigate | 05:34 |
| openstackstatus | ianw: finished logging | 05:34 |
| ianw | infra-root: ^ i don't want to destroy potential debugging state on this when i don't know what i'm doing | 05:34 |
| tonyb | ianw: Thanks | 05:35 |
| *** ociuhandu has quit IRC | 05:36 | |
| ianw | tonyb: that should at least get you sent to a different server now. if nothing else comes up, i can just try "turn it off and on again" tomorrow | 05:36 |
| clarkb | anything in dmesg about the filesystem? | 05:36 |
| ianw | clarkb: not recently, but the server has been up for 100+ days and there is old stuff | 05:37 |
| clarkb | we've seen sad filesystems lead to sadgitea but other than that I do t have immediate ideas | 05:37 |
| ianw | [Fri Jul 26 19:41:14 2019] INFO: task kworker/u16:2:10089 blocked for more than 120 seconds | 05:38 |
| ianw | that's the last one | 05:38 |
| clarkb | unlikely that then | 05:38 |
| clarkb | docker logs $container will get you gitea logs | 05:39 |
| clarkb | that may shed light on the errirs? | 05:39 |
| clarkb | oh maybe that us what you pasted previously | 05:41 |
| clarkb | the failed to serve RPC | 05:41 |
| clarkb | was it git repo spwcific? | 05:41 |
| tonyb | clarkb: I only saw it in nova | 05:42 |
| tonyb | but I was able to clone nova from that server | 05:43 |
| clarkb | maybe we fsck the nova repo theb and see if that turns anything up on that server | 05:44 |
| ianw | yeah, it doesn't have a repo, just that long message | 05:49 |
| *** igordc has joined #openstack-infra | 05:55 | |
| *** ricolin has quit IRC | 06:00 | |
| *** soniya29 has joined #openstack-infra | 06:04 | |
| *** surpatil has joined #openstack-infra | 06:18 | |
| *** ykarel|mtg is now known as ykarel | 06:22 | |
| *** ykarel is now known as ykarel|afk | 06:23 | |
| *** jtomasek has joined #openstack-infra | 06:41 | |
| *** jtomasek has quit IRC | 06:41 | |
| *** jtomasek has joined #openstack-infra | 06:42 | |
| *** apetrich has joined #openstack-infra | 06:51 | |
| *** rcernin has quit IRC | 06:53 | |
| *** yamamoto has quit IRC | 07:04 | |
| *** ociuhandu has joined #openstack-infra | 07:04 | |
| AJaeger | config-core, could you review this cleanup, please? Remove old jobs/template: https://review.opendev.org/692639 and https://review.opendev.org/692640 . Note it removes it from a dead repo... | 07:05 |
| *** pkopec has joined #openstack-infra | 07:07 | |
| *** jaicaa has quit IRC | 07:13 | |
| *** ociuhandu has quit IRC | 07:13 | |
| *** ociuhandu has joined #openstack-infra | 07:13 | |
| *** jaicaa has joined #openstack-infra | 07:14 | |
| openstackgerrit | Merged opendev/base-jobs master: Add cleanup task timeouts to production playbooks https://review.opendev.org/694183 | 07:15 |
| *** ociuhandu has quit IRC | 07:16 | |
| *** ociuhandu has joined #openstack-infra | 07:17 | |
| *** ociuhandu has quit IRC | 07:22 | |
| *** ociuhandu has joined #openstack-infra | 07:24 | |
| *** ociuhandu has quit IRC | 07:33 | |
| *** ociuhandu has joined #openstack-infra | 07:34 | |
| *** ociuhandu has quit IRC | 07:35 | |
| *** ociuhandu has joined #openstack-infra | 07:36 | |
| *** ociuhandu has quit IRC | 07:36 | |
| *** ociuhandu has joined #openstack-infra | 07:37 | |
| *** pgaxatte has joined #openstack-infra | 07:37 | |
| *** igordc has quit IRC | 07:39 | |
| *** yamamoto has joined #openstack-infra | 07:43 | |
| *** ociuhandu has quit IRC | 07:45 | |
| *** ykarel|afk is now known as ykarel | 07:46 | |
| *** elod is now known as elod_off | 07:47 | |
| *** yamamoto has quit IRC | 07:48 | |
| *** kopecmartin|off is now known as kopecmartin | 07:52 | |
| ianw | clarkb: dropped a note in the story, but fsck on the nova repo seems ok... | 07:53 |
| openstackgerrit | Merged openstack/project-config master: Remove puppet-module-unit-jobs from puppet-n1k-vsm https://review.opendev.org/692639 | 08:02 |
| *** xek has joined #openstack-infra | 08:05 | |
| *** pleia2 has quit IRC | 08:08 | |
| *** pleia2 has joined #openstack-infra | 08:09 | |
| *** roman_g has joined #openstack-infra | 08:15 | |
| *** tosky has joined #openstack-infra | 08:16 | |
| *** tesseract has joined #openstack-infra | 08:17 | |
| *** ykarel_ has joined #openstack-infra | 08:19 | |
| AJaeger | thanks, ianw ! | 08:20 |
| *** dmellado has quit IRC | 08:21 | |
| *** ykarel has quit IRC | 08:21 | |
| *** dmellado has joined #openstack-infra | 08:23 | |
| *** dmellado has quit IRC | 08:25 | |
| openstackgerrit | Merged openstack/openstack-zuul-jobs master: Remove puppet-module-unit-jobs https://review.opendev.org/692640 | 08:26 |
| *** dmellado has joined #openstack-infra | 08:27 | |
| *** tkajinam has quit IRC | 08:29 | |
| *** ociuhandu has joined #openstack-infra | 08:30 | |
| *** slaweq has joined #openstack-infra | 08:30 | |
| *** udesale has quit IRC | 08:35 | |
| *** udesale has joined #openstack-infra | 08:37 | |
| *** lucasagomes has joined #openstack-infra | 08:47 | |
| *** ociuhandu has quit IRC | 08:51 | |
| *** ociuhandu has joined #openstack-infra | 08:52 | |
| *** ralonsoh has joined #openstack-infra | 08:54 | |
| *** rpittau|afk is now known as rpittau | 08:57 | |
| *** slaweq has quit IRC | 09:01 | |
| *** ccamacho has joined #openstack-infra | 09:06 | |
| *** udesale has quit IRC | 09:08 | |
| *** slaweq has joined #openstack-infra | 09:09 | |
| *** jpena has joined #openstack-infra | 09:09 | |
| *** udesale has joined #openstack-infra | 09:09 | |
| *** ykarel_ is now known as ykarel|lunch | 09:09 | |
| *** ociuhandu has quit IRC | 09:10 | |
| *** ociuhandu has joined #openstack-infra | 09:11 | |
| *** tinwood has quit IRC | 09:12 | |
| *** tinwood has joined #openstack-infra | 09:13 | |
| openstackgerrit | Daniel Bengtsson proposed openstack/cookiecutter master: Use python -m pip instead pip command. https://review.opendev.org/694069 | 09:16 |
| *** trident has quit IRC | 09:16 | |
| *** dmellado has quit IRC | 09:20 | |
| *** dmellado has joined #openstack-infra | 09:24 | |
| *** sshnaidm_ is now known as sshnaidm|ruck | 09:26 | |
| *** trident has joined #openstack-infra | 09:26 | |
| *** ociuhandu has quit IRC | 09:34 | |
| *** ociuhandu has joined #openstack-infra | 09:35 | |
| *** ociuhandu has quit IRC | 09:35 | |
| *** ociuhandu has joined #openstack-infra | 09:36 | |
| *** ociuhandu has quit IRC | 09:37 | |
| *** ociuhandu has joined #openstack-infra | 09:37 | |
| *** xinranwang has quit IRC | 09:39 | |
| *** yamamoto has joined #openstack-infra | 09:40 | |
| *** gfidente|afk is now known as gfidente | 09:43 | |
| *** ociuhandu has quit IRC | 09:45 | |
| *** kaisers has quit IRC | 09:49 | |
| *** roman_g has quit IRC | 09:50 | |
| *** iurygregory has joined #openstack-infra | 09:53 | |
| *** ociuhandu has joined #openstack-infra | 09:54 | |
| *** pcaruana has joined #openstack-infra | 09:54 | |
| *** Lucas_Gray has joined #openstack-infra | 10:01 | |
| *** ociuhandu has quit IRC | 10:03 | |
| *** ralonsoh has quit IRC | 10:03 | |
| *** ralonsoh has joined #openstack-infra | 10:03 | |
| *** kaisers has joined #openstack-infra | 10:05 | |
| *** lpetrut has joined #openstack-infra | 10:06 | |
| *** yamamoto has quit IRC | 10:06 | |
| *** lpetrut has quit IRC | 10:07 | |
| *** lpetrut has joined #openstack-infra | 10:07 | |
| *** yamamoto has joined #openstack-infra | 10:10 | |
| *** ociuhandu has joined #openstack-infra | 10:12 | |
| *** yamamoto has quit IRC | 10:15 | |
| *** ociuhandu has quit IRC | 10:19 | |
| *** ociuhandu has joined #openstack-infra | 10:22 | |
| *** ociuhandu has quit IRC | 10:23 | |
| *** aedc has joined #openstack-infra | 10:23 | |
| *** ociuhandu has joined #openstack-infra | 10:24 | |
| *** lucasagomes has quit IRC | 10:25 | |
| *** lucasagomes has joined #openstack-infra | 10:28 | |
| *** ociuhandu has quit IRC | 10:29 | |
| *** dpawlik has joined #openstack-infra | 10:34 | |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Add optional support for circular dependencies https://review.opendev.org/685354 | 10:35 |
| *** gibi_off has joined #openstack-infra | 10:42 | |
| gibi_off | hi! do we already have a bug reported for the requirement-check job failures http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22venv%2Fbin%2Fpip3%5C%22%20AND%20message%3A%5C%22No%20such%20file%20or%20directory%3A%5C%22 ? | 10:44 |
| openstackgerrit | Slawek Kaplonski proposed openstack/openstack-zuul-jobs master: Drop old neutron-grenade job from integrated-gate template https://review.opendev.org/694244 | 10:45 |
| AJaeger | gibi_off: is that related to https://review.opendev.org/693967 ? | 10:49 |
| gibi_off | AJaeger: based on the change it could be related. Let me recheck my patch that was showing the symptoms | 10:51 |
| gibi_off | AJaeger: let's see if https://review.opendev.org/#/c/683609 greens up now | 10:53 |
| *** ykarel|lunch is now known as ykarel | 10:57 | |
| gibi_off | AJaeger: requirements-check still failing https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_bc8/683609/5/check/requirements-check/bc891eb/job-output.txt | 11:00 |
| gibi_off | AJaeger: but now with different error | 11:00 |
| gibi_off | ModuleNotFoundError: No module named 'openstack_requirements' | 11:00 |
| AJaeger | gibi_off: best ask requirements team -> #openstack-requirements | 11:02 |
| gibi_off | AJaeger: ack thanks | 11:03 |
| frickler | gibi_off: AJaeger: https://review.opendev.org/#/c/694248/1/playbooks/requirements-check.yaml | 11:04 |
| gibi_off | frickler: thanks | 11:06 |
| *** udesale has quit IRC | 11:14 | |
| openstackgerrit | Stephen Finucane proposed openstack/cookiecutter master: Stop configuring 'install_command' https://review.opendev.org/694069 | 11:15 |
| openstackgerrit | Merged openstack/cookiecutter master: Stop configuring 'install_command' https://review.opendev.org/694069 | 11:16 |
| *** Lucas_Gray has quit IRC | 11:19 | |
| *** aedc has quit IRC | 11:20 | |
| openstackgerrit | Slawek Kaplonski proposed openstack/openstack-zuul-jobs master: Run neutron-grenade job only for branches before Ussuri https://review.opendev.org/694244 | 11:20 |
| *** ociuhandu has joined #openstack-infra | 11:21 | |
| *** jaosorior has joined #openstack-infra | 11:25 | |
| *** ociuhandu has quit IRC | 11:27 | |
| *** dpawlik has quit IRC | 11:30 | |
| *** dpawlik has joined #openstack-infra | 11:37 | |
| *** rfolco has joined #openstack-infra | 11:37 | |
| *** verdurin has quit IRC | 11:45 | |
| *** Lucas_Gray has joined #openstack-infra | 11:46 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Remove unused job https://review.opendev.org/693990 | 11:47 |
| *** Lucas_Gray has quit IRC | 11:48 | |
| *** Lucas_Gray has joined #openstack-infra | 11:51 | |
| *** lucasagomes has quit IRC | 11:55 | |
| *** verdurin has joined #openstack-infra | 11:59 | |
| *** dave-mccowan has joined #openstack-infra | 12:00 | |
| *** jpena is now known as jpena|lunch | 12:00 | |
| *** trident has quit IRC | 12:09 | |
| *** pgaxatte has quit IRC | 12:12 | |
| *** trident has joined #openstack-infra | 12:18 | |
| *** tesseract has quit IRC | 12:28 | |
| *** tesseract has joined #openstack-infra | 12:29 | |
| *** surpatil has quit IRC | 12:35 | |
| AJaeger | config-core, please review https://review.opendev.org/#/c/688822/ https://review.opendev.org/694244 https://review.opendev.org/#/c/690718/ https://review.opendev.org/691140 https://review.opendev.org/#/c/693455/ and https://review.opendev.org/693462 | 12:41 |
| AJaeger | mnaser: for static.o.o, please review https://review.opendev.org/681582 and stack on top of it. 681582 can merge directly IMHO. I let you drive the rest... | 12:42 |
| *** rfolco has quit IRC | 12:51 | |
| *** jtomasek has quit IRC | 12:51 | |
| *** psachin has quit IRC | 12:51 | |
| *** jistr has quit IRC | 12:52 | |
| *** irclogbot_0 has quit IRC | 12:52 | |
| *** piotrowskim has quit IRC | 12:52 | |
| *** prometheanfire has quit IRC | 12:52 | |
| *** Adri2000 has quit IRC | 12:52 | |
| *** nicholas has quit IRC | 12:52 | |
| *** arif-ali has quit IRC | 12:52 | |
| *** d34dh0r53 has quit IRC | 12:52 | |
| *** Buggys has quit IRC | 12:52 | |
| *** harlowja has quit IRC | 12:52 | |
| *** paladox has quit IRC | 12:52 | |
| *** rakhmerov has quit IRC | 12:52 | |
| *** jklare has quit IRC | 12:52 | |
| *** fresta has quit IRC | 12:52 | |
| *** rpioso has quit IRC | 12:52 | |
| *** ianw has quit IRC | 12:52 | |
| *** calbers has quit IRC | 12:52 | |
| *** kota_ has quit IRC | 12:52 | |
| *** davidlenwell has quit IRC | 12:52 | |
| *** cyberpear has quit IRC | 12:52 | |
| *** petevg has quit IRC | 12:52 | |
| *** SergeyLukjanov has quit IRC | 12:52 | |
| *** dulek has quit IRC | 12:52 | |
| *** amotoki has quit IRC | 12:52 | |
| *** zaro has quit IRC | 12:52 | |
| *** Anticimex has quit IRC | 12:52 | |
| *** calebb has quit IRC | 12:52 | |
| *** coreycb has quit IRC | 12:52 | |
| *** clayg has quit IRC | 12:52 | |
| *** davecore has quit IRC | 12:52 | |
| *** knikolla has quit IRC | 12:52 | |
| *** ildikov has quit IRC | 12:52 | |
| *** aedc has joined #openstack-infra | 12:53 | |
| *** openstackstatus has quit IRC | 12:53 | |
| *** rfolco has joined #openstack-infra | 12:53 | |
| *** jtomasek has joined #openstack-infra | 12:53 | |
| *** psachin has joined #openstack-infra | 12:53 | |
| *** arif-ali has joined #openstack-infra | 12:53 | |
| *** jistr has joined #openstack-infra | 12:53 | |
| *** irclogbot_0 has joined #openstack-infra | 12:53 | |
| *** piotrowskim has joined #openstack-infra | 12:53 | |
| *** nicholas has joined #openstack-infra | 12:53 | |
| *** prometheanfire has joined #openstack-infra | 12:53 | |
| *** Adri2000 has joined #openstack-infra | 12:53 | |
| *** rakhmerov has joined #openstack-infra | 12:53 | |
| *** d34dh0r53 has joined #openstack-infra | 12:53 | |
| *** Buggys has joined #openstack-infra | 12:53 | |
| *** harlowja has joined #openstack-infra | 12:53 | |
| *** paladox has joined #openstack-infra | 12:53 | |
| *** jklare has joined #openstack-infra | 12:53 | |
| *** fresta has joined #openstack-infra | 12:53 | |
| *** rpioso has joined #openstack-infra | 12:53 | |
| *** ianw has joined #openstack-infra | 12:53 | |
| *** dulek has joined #openstack-infra | 12:53 | |
| *** calbers has joined #openstack-infra | 12:53 | |
| *** kota_ has joined #openstack-infra | 12:53 | |
| *** davidlenwell has joined #openstack-infra | 12:53 | |
| *** cyberpear has joined #openstack-infra | 12:53 | |
| *** petevg has joined #openstack-infra | 12:53 | |
| *** SergeyLukjanov has joined #openstack-infra | 12:53 | |
| *** calebb has joined #openstack-infra | 12:53 | |
| *** amotoki has joined #openstack-infra | 12:53 | |
| *** zaro has joined #openstack-infra | 12:53 | |
| *** Anticimex has joined #openstack-infra | 12:53 | |
| *** ildikov has joined #openstack-infra | 12:53 | |
| *** coreycb has joined #openstack-infra | 12:53 | |
| *** clayg has joined #openstack-infra | 12:53 | |
| *** davecore has joined #openstack-infra | 12:53 | |
| *** knikolla has joined #openstack-infra | 12:53 | |
| *** rlandy has joined #openstack-infra | 12:54 | |
| *** rlandy is now known as rlandy|rover | 12:55 | |
| *** rlandy|rover is now known as rlandy|rover|mtg | 13:03 | |
| *** rh-jelabarre has joined #openstack-infra | 13:04 | |
| *** jpena|lunch is now known as jpena | 13:08 | |
| *** kopecmartin is now known as kopecmartin|half | 13:09 | |
| *** kopecmartin|half is now known as kopecmartin|pto | 13:09 | |
| *** yamamoto has joined #openstack-infra | 13:16 | |
| *** yamamoto has quit IRC | 13:20 | |
| yoctozepto | infra-root, started getting logs with no listing access to them: https://openstack.fortnebula.com:13808/v1/AUTH_e8fd161dc34c421a979a9e6421f823e9/zuul_opendev_logs_8b9/694235/1/gate/kolla-build-ubuntu-source/8b9769d/ | 13:23 |
| yoctozepto | Web Listing Disabled | 13:24 |
| yoctozepto | :D | 13:24 |
| donnyd | Works on my phone | 13:24 |
| yoctozepto | odd, it's random | 13:24 |
| yoctozepto | i have been hitting refresh now | 13:25 |
| yoctozepto | but needs it at every level... | 13:25 |
| yoctozepto | https://pastebin.com/6c4zLn65 | 13:25 |
| *** ykarel is now known as ykarel|afk | 13:26 | |
| yoctozepto | donnyd: most tries it does not work | 13:26 |
| yoctozepto | you must have been very lucky :D | 13:26 |
| donnyd | https://usercontent.irccloud-cdn.com/file/MWiqciOe/Screenshot_20191114-082714_Chrome.jpg | 13:27 |
| donnyd | Just tried it again | 13:28 |
| yoctozepto | geographical problem? | 13:28 |
| donnyd | Maybe | 13:29 |
| yoctozepto | north-e Poland here | 13:29 |
| yoctozepto | north-eastern* | 13:29 |
| donnyd | What does curl say? | 13:29 |
| Shrews | i hit both cases as well (from US). looks sort of like an apache issue and just depends on which apache process you hit | 13:29 |
| donnyd | Driving right now, so will be able to take a deeper look | 13:29 |
| donnyd | In about 50 minutes | 13:30 |
| yoctozepto | donnyd: the same | 13:30 |
| yoctozepto | thanks, guys | 13:30 |
| donnyd | Try the other endpoint | 13:30 |
| yoctozepto | hope to get it fixed because it is hard to navigate now ;p | 13:30 |
| donnyd | https://swift.fortnebula.com | 13:31 |
| donnyd | No port | 13:31 |
| yoctozepto | donnyd: yeah, with port it had cert issues | 13:31 |
| yoctozepto | mhm, it works, thanks! | 13:32 |
| yoctozepto | NAH LOL | 13:32 |
| yoctozepto | two dirs up and be gone | 13:32 |
| yoctozepto | trolololo | 13:32 |
| yoctozepto | endpoint did not help | 13:32 |
| yoctozepto | :-( | 13:32 |
| donnyd | Oh I bet its cert swap time | 13:33 |
| donnyd | Its not fully automated | 13:33 |
| *** yamamoto has joined #openstack-infra | 13:33 | |
| yoctozepto | still fails randomly, argh | 13:33 |
| donnyd | Yea that is what happened last time too | 13:34 |
| yoctozepto | coffee time | 13:34 |
| *** ociuhandu has joined #openstack-infra | 13:34 | |
| *** pgaxatte has joined #openstack-infra | 13:36 | |
| *** hrw has joined #openstack-infra | 13:37 | |
| hrw | morning | 13:37 |
| hrw | I clicked in zuul to get info about failed job and was greeted with 'ask admins' like page: https://openstack.fortnebula.com:13808/v1/AUTH_e8fd161dc34c421a979a9e6421f823e9/zuul_opendev_logs_8b9/694235/1/gate/kolla-build-ubuntu-source/8b9769d/ | 13:37 |
| hrw | so I ask for enabling web listings | 13:38 |
| hrw | http://zuul.openstack.org/status/change/694235,1 is the job I clicked | 13:38 |
| *** ociuhandu has quit IRC | 13:40 | |
| Shrews | hrw: yes, this was just reported. fortnebula admin will be looking at it soon | 13:40 |
| hrw | Shrews: thank you. | 13:40 |
| Shrews | hrw: if you hit reload on the page a few times, you might eventually get the listings | 13:42 |
| hrw | good to know ;) | 13:43 |
| * hrw out | 13:43 | |
| *** mriedem has joined #openstack-infra | 13:45 | |
| *** udesale has joined #openstack-infra | 13:46 | |
| *** hwoarang has quit IRC | 13:50 | |
| *** hwoarang has joined #openstack-infra | 13:56 | |
| *** ykarel|afk has quit IRC | 13:57 | |
| *** ykarel has joined #openstack-infra | 13:57 | |
| *** yamamoto has quit IRC | 13:58 | |
| *** ociuhandu has joined #openstack-infra | 14:05 | |
| *** ociuhandu has quit IRC | 14:05 | |
| *** ociuhandu has joined #openstack-infra | 14:06 | |
| *** Lucas_Gray has quit IRC | 14:12 | |
| *** Lucas_Gray has joined #openstack-infra | 14:13 | |
| openstackgerrit | Daniel Bengtsson proposed openstack/cookiecutter master: Update the python 3 version in envlist attribut. https://review.opendev.org/694302 | 14:14 |
| *** psachin has quit IRC | 14:14 | |
| *** ociuhandu has quit IRC | 14:18 | |
| *** ociuhandu has joined #openstack-infra | 14:18 | |
| *** radez has joined #openstack-infra | 14:19 | |
| openstackgerrit | Daniel Bengtsson proposed openstack/cookiecutter master: Update the python version in envlist attribut. https://review.opendev.org/694302 | 14:23 |
| *** eharney has joined #openstack-infra | 14:27 | |
| *** goldyfruit has joined #openstack-infra | 14:30 | |
| *** yamamoto has joined #openstack-infra | 14:32 | |
| openstackgerrit | Hervé Beraud proposed openstack/cookiecutter master: Remove py27 support for future projects https://review.opendev.org/694312 | 14:33 |
| *** ociuhandu has quit IRC | 14:34 | |
| *** ociuhandu has joined #openstack-infra | 14:35 | |
| *** yamamoto has quit IRC | 14:38 | |
| *** ociuhandu has quit IRC | 14:42 | |
| *** lucasagomes has joined #openstack-infra | 14:44 | |
| donnyd | infra-root I think there is a deeper issue with FN swift | 14:47 |
| donnyd | I think we need to disable uploads until I have a chance to look deeper at it | 14:48 |
| *** Goneri has joined #openstack-infra | 14:48 | |
| donnyd | unfortunately I am not home today, so it will have to be when I get back | 14:48 |
| *** chandankumar is now known as raukadah | 14:51 | |
| *** ricolin has joined #openstack-infra | 14:52 | |
| *** Lucas_Gray has quit IRC | 14:54 | |
| *** hrw has left #openstack-infra | 14:55 | |
| fungi | donnyd: i'll get the change pushed up for that asap | 14:55 |
| fungi | thanks for the heads up! | 14:55 |
| donnyd | i am going to try to enable container listings, but I have changed nothing on my end since we first set this up | 14:55 |
| donnyd | all of the older listings work fine | 14:56 |
| donnyd | trying to figure out where the breakage is at | 14:56 |
| donnyd | I am also not at a machine where I can fix the pushed up. Sorry to be needy this morning | 14:59 |
| donnyd | fungi: do you think you could push that up? | 14:59 |
| *** Lucas_Gray has joined #openstack-infra | 15:00 | |
| openstackgerrit | Jeremy Stanley proposed opendev/base-jobs master: Temporarily disable log uploads to fortnebula https://review.opendev.org/694322 | 15:03 |
| fungi | donnyd: yeah, was just digging around to remind myself where to set it | 15:03 |
| *** soniya29 has quit IRC | 15:04 | |
| fungi | infra-root: i'm going to self-approve that ^ | 15:04 |
| AJaeger | fungi: you got my +2A ;) | 15:05 |
| AJaeger | fungi: LGTM | 15:05 |
| donnyd | Thanks fungi | 15:06 |
| donnyd | I will get to the bottom of it | 15:06 |
| fungi | no worries donnyd, and thanks! | 15:07 |
| fungi | also, did anybody tell you that you were a superuser award nominee in shanghai as well as a contributor award recipient? | 15:08 |
| *** Lucas_Gray has quit IRC | 15:08 | |
| AJaeger | donnyd: https://superuser.openstack.org/awards/ | 15:11 |
| *** udesale has quit IRC | 15:11 | |
| donnyd | AJaeger: I think i lost to baidu | 15:12 |
| donnyd | but even being considered is a win in my book | 15:12 |
| donnyd | fungi: no, I had no idea | 15:13 |
| *** udesale has joined #openstack-infra | 15:13 | |
| openstackgerrit | Merged opendev/base-jobs master: Temporarily disable log uploads to fortnebula https://review.opendev.org/694322 | 15:13 |
| donnyd | I <3 this community for even thinking of me to nominate me for an award | 15:14 |
| openstackgerrit | Thierry Carrez proposed openstack/ptgbot master: Add common navbar and display MOTD on all pages https://review.opendev.org/694326 | 15:15 |
| openstackgerrit | Thierry Carrez proposed opendev/puppet-ptgbot master: Deploy new logo.png and motd.js files https://review.opendev.org/694327 | 15:15 |
| AJaeger | donnyd: fortnebula should be disabled now - I think only for future jobs running but not those already started | 15:16 |
| *** lucasagomes has quit IRC | 15:17 | |
| donnyd | AJaeger: I think you are right. I also have been seeing some jobs have the dreaded POST_FAILURE | 15:17 |
| donnyd | so they will need to be resubmitted as well | 15:17 |
| donnyd | maybe want to let the peoples know | 15:17 |
| *** dpawlik has quit IRC | 15:19 | |
| openstackgerrit | Merged openstack/cookiecutter master: Update the python version in envlist attribut. https://review.opendev.org/694302 | 15:19 |
| *** rlandy|rover|mtg is now known as rlandy|rover | 15:28 | |
| *** lucasagomes has joined #openstack-infra | 15:31 | |
| *** ykarel is now known as ykarel|away | 15:34 | |
| *** pgaxatte has quit IRC | 15:41 | |
| *** michael-beaver has joined #openstack-infra | 15:43 | |
| *** goldyfruit_ has joined #openstack-infra | 15:43 | |
| *** goldyfruit has quit IRC | 15:45 | |
| *** lucasagomes has quit IRC | 15:45 | |
| *** lucasagomes has joined #openstack-infra | 15:45 | |
| *** lucasagomes has quit IRC | 15:45 | |
| *** lucasagomes has joined #openstack-infra | 15:45 | |
| *** ociuhandu has joined #openstack-infra | 15:49 | |
| *** odicha has joined #openstack-infra | 15:49 | |
| *** ociuhandu has quit IRC | 15:50 | |
| *** ociuhandu has joined #openstack-infra | 15:52 | |
| *** ociuhandu has quit IRC | 15:57 | |
| frickler | status notice some jobs have been failing with POST_FAILURE due to an issue with one of our log storage providers, those should be safe to recheck now | 15:58 |
| frickler | fungi: AJaeger: donnyd: ^^ like that? | 15:58 |
| fungi | frickler: oh, so there have been post_failures caused by that? did anyone confirm from executor logs yet? | 15:59 |
| fungi | so far the only thing i saw correlated for certain was that some logs uploaded to fn were intermittently not browseable | 16:00 |
| frickler | fungi: I didn't, I was just assuming that connection based on donnyd's comment | 16:00 |
| fungi | we should probably double-check that the post_failures aren't a separate problem before we go saying they've stopped | 16:02 |
| frickler | actually, a lot of the ones I now checked have other issues. more like the well-known ssh host failure. those should still be safe to recheck, but well | 16:02 |
| frickler | so maybe better just leave it at that. /me needs to leave now anyway | 16:03 |
| donnyd | frickler: thank you | 16:03 |
| frickler | donnyd: without the leading "#" what I wrote was just a draft, nothing has been sent yet | 16:04 |
| donnyd | Oh | 16:05 |
| fungi | yeah, if we think we've identified that the post_failure cause was related to swift issues in fn then i'm fine sending a status notice like that, but if we haven't confirmed they're related then it may be premature to suggest they're behind us now | 16:07 |
| fungi | i see a nova-next post_failure result from 14:31:33z at http://zuul.opendev.org/t/openstack/builds | 16:10 |
| *** gyee has joined #openstack-infra | 16:11 | |
| fungi | http://zuul.opendev.org/t/openstack/builds?result=post_failure gives a clearer picture | 16:11 |
| fungi | if it were directly related to fn swift i'm surprised we didn't see more between 14:31 and 15:13 when the job changes merged | 16:12 |
| fungi | or beyond for that matter, as those playbooks would have only taken effect for jobs started *after* the merge time | 16:13 |
| *** xek_ has joined #openstack-infra | 16:14 | |
| *** xek has quit IRC | 16:14 | |
| fungi | browsing through the build results, only a few have log urls in fn and most are for other providers | 16:15 |
| fungi | so i don't immediately see any correlation | 16:15 |
| *** ccamacho has quit IRC | 16:19 | |
| *** ricolin has quit IRC | 16:20 | |
| timburke | on logs no longer being browse-able, i wonder if older jobs were generating index.html files that more-recent jobs dropped... | 16:20 |
| *** odicha has quit IRC | 16:21 | |
| timburke | would want to look at container listings for new and old jobs to confirm | 16:21 |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Gitlab - Minimal reporter ables to comment on MR https://review.opendev.org/694346 | 16:27 |
| fungi | the manifest should confirm. i'll checl | 16:28 |
| fungi | er, check | 16:28 |
| fungi | but that wouldn't explain it only happening in fn i don't think | 16:29 |
| fungi | nor it working intermittently when reloading the same log url multiple times | 16:29 |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Gitlab - Minimal reporter ables to comment on MR https://review.opendev.org/694346 | 16:31 |
| *** udesale has quit IRC | 16:33 | |
| fungi | d'oh, i won't check... the manifests are uploaded to the log store, not kept in the db | 16:33 |
| *** iurygregory has quit IRC | 16:35 | |
| fungi | okay, on retrying i got the manifest to load, but it seems it's more than just indices getting a 404 | 16:36 |
| fungi | swift simply provides a special error on an index 404 is all | 16:36 |
| fungi | also, comparing to other working build logs, zuul doesn't include the generated directory index files in the manifest as far as i can tell | 16:37 |
| fungi | so that line of inquiry is a dead end (but also unlikely to be the cause as it seems we're simply seeing intermittent 404s for all manner of objects served from fn's swift, not merely the directory indices) | 16:39 |
| fungi | anyway, i think that squarely rules out issues with the jobs themselves | 16:40 |
| *** iurygregory has joined #openstack-infra | 16:48 | |
| *** sshnaidm|ruck is now known as sshnaidm|off | 16:56 | |
| *** KeithMnemonic has joined #openstack-infra | 17:00 | |
| *** armstrong has joined #openstack-infra | 17:01 | |
| *** ociuhandu has joined #openstack-infra | 17:01 | |
| *** yamamoto has joined #openstack-infra | 17:02 | |
| *** rpittau is now known as rpittau|afk | 17:04 | |
| *** yamamoto has quit IRC | 17:06 | |
| *** ociuhandu has quit IRC | 17:07 | |
| *** rkukura has joined #openstack-infra | 17:13 | |
| *** tesseract has quit IRC | 17:15 | |
| *** rkukura has quit IRC | 17:22 | |
| *** ociuhandu has joined #openstack-infra | 17:25 | |
| *** aedc has quit IRC | 17:28 | |
| mriedem | i've noticed a spike in package mirror issues on inap nodes in the last 2 days: | 17:40 |
| mriedem | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22ERROR%3A%20Could%20not%20install%20packages%20due%20to%20an%20EnvironmentError%3A%20('Connection%20broken%3A%20IncompleteRead(0%20bytes%20read)'%2C%20IncompleteRead(0%20bytes%20read))%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=7d | 17:40 |
| *** ykarel|away has quit IRC | 17:41 | |
| fungi | that looks like it could either be network issues between job nodes and the mirror server, or between the mirror server and the internet | 17:43 |
| fungi | but i'll check cacti graphs and system logs for any sign of a problem on the server | 17:43 |
| openstackgerrit | Matt Riedemann proposed opendev/elastic-recheck master: Add query for inap package mirror bug 1852617 https://review.opendev.org/694372 | 17:43 |
| openstack | bug 1852617 in OpenStack-Gate "ERROR: Could not install packages due to an EnvironmentError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read)) on inap nodes" [Undecided,New] https://launchpad.net/bugs/1852617 | 17:43 |
| fungi | oog, this doesn't look good: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=67952&rra_id=all | 17:46 |
| fungi | according to historic graphs the old openstack.org mirror server there had separate 100gb cinder volumes for /var/cache/apache and /var/cache/openafs | 17:48 |
| *** tosky has quit IRC | 17:49 | |
| fungi | but those were never added to the rebuilt mirror server so it's trying to record a couple hundred gigabytes of caches on a 150gb rootfs :( | 17:49 |
| fungi | i'll push up a change to temporarily disable builds there until we can add volumes to that mirror | 17:50 |
| openstackgerrit | Jeremy Stanley proposed openstack/project-config master: Revert "Enable inap-mtl01" https://review.opendev.org/694373 | 17:52 |
| fungi | infra-root: config-core: i'm self-approving that ^ | 17:52 |
| *** michael-beaver has quit IRC | 17:52 | |
| clarkb | fungi: the mirror cache is only like 80gb iirc | 17:55 |
| clarkb | hrn maybe afs caching is bigger than I remember too | 17:56 |
| *** jtomasek has quit IRC | 18:00 | |
| *** ociuhandu has quit IRC | 18:00 | |
| *** lucasagomes has quit IRC | 18:01 | |
| *** jpena is now known as jpena|off | 18:04 | |
| openstackgerrit | Merged opendev/elastic-recheck master: Add query for inap package mirror bug 1852617 https://review.opendev.org/694372 | 18:05 |
| openstack | bug 1852617 in OpenStack-Gate "ERROR: Could not install packages due to an EnvironmentError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read)) on inap nodes" [Undecided,Confirmed] https://launchpad.net/bugs/1852617 | 18:05 |
| *** ociuhandu has joined #openstack-infra | 18:05 | |
| fungi | well, the cacti graph showed it running out of space on the rootfs | 18:06 |
| fungi | which would easily explain clusters of failures serving cache content | 18:07 |
| fungi | if it tried to refresh and couldn't write | 18:07 |
| clarkb | ya I'm mostly just surprised 150gb wasnt enough given what I thought our cache limits were | 18:12 |
| *** yamamoto has joined #openstack-infra | 18:13 | |
| *** lpetrut has quit IRC | 18:15 | |
| *** yamamoto has quit IRC | 18:18 | |
| openstackgerrit | Merged openstack/project-config master: Revert "Enable inap-mtl01" https://review.opendev.org/694373 | 18:18 |
| AJaeger | config-core, please review https://review.opendev.org/#/c/688822/ https://review.opendev.org/694244 https://review.opendev.org/#/c/690718/ https://review.opendev.org/691140 https://review.opendev.org/#/c/693455/ and https://review.opendev.org/693462 | 18:19 |
| *** ssbarnea has quit IRC | 18:22 | |
| *** jtomasek has joined #openstack-infra | 18:24 | |
| *** dave-mccowan has quit IRC | 18:26 | |
| *** ralonsoh has quit IRC | 18:28 | |
| *** dave-mccowan has joined #openstack-infra | 18:32 | |
| *** artom has quit IRC | 18:32 | |
| *** jaosorior has quit IRC | 18:34 | |
| *** xek__ has joined #openstack-infra | 18:45 | |
| *** ociuhandu has quit IRC | 18:45 | |
| fungi | clarkb: well, that's 150gb rootfs, so also files for the operating system, logging, et cetera | 18:45 |
| fungi | and also reserved overhead for uid 0 | 18:45 |
| *** ociuhandu has joined #openstack-infra | 18:46 | |
| *** xek_ has quit IRC | 18:47 | |
| fungi | looks like it has ~4% reserved for uid0 | 18:47 |
| fungi | also remember apache cache will spike over its "limit" briefly between cullings | 18:48 |
| fungi | which is why i wouldn't put it on the rootfs anyway unless there was lots of room for temporarily spillage | 18:49 |
| clarkb | ya we expect ~100gb for apache and set the limit to 80gb or so to account for that I think | 18:51 |
| *** ociuhandu has quit IRC | 18:52 | |
| *** yamamoto has joined #openstack-infra | 18:53 | |
| *** igordc has joined #openstack-infra | 18:54 | |
| *** yamamoto has quit IRC | 18:58 | |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: WIP: Use correct ansible version var name https://review.opendev.org/694388 | 19:10 |
| *** rkukura has joined #openstack-infra | 19:12 | |
| fungi | looks like /var/cache/openafs is using 42gb and /var/cache/apache2 has 86gb currently | 19:21 |
| fungi | that's 128 out of the 136gb currently in use, so the os is only ~8gb | 19:23 |
| openstackgerrit | Paul Belanger proposed zuul/zuul master: Don't search github api for projects not in Zuul https://review.opendev.org/694390 | 19:24 |
| clarkb | oh wow afs cache usage is a lot higher than I remember (I expected 5gb or so) | 19:25 |
| clarkb | I guess that is a good thing though and implies quicker afs | 19:26 |
| fungi | current apache logs are in the 2gb neighborhood at rotation so could be temporarily doubling on disk while compressing too | 19:28 |
| fungi | and /var/log/apache2 as a whole is >4gb there | 19:28 |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: WIP: Use correct ansible version var name https://review.opendev.org/694388 | 19:30 |
| openstackgerrit | Paul Belanger proposed zuul/zuul master: Don't search github api for projects not in Zuul https://review.opendev.org/694390 | 19:31 |
| *** Goneri has quit IRC | 19:44 | |
| *** zbr is now known as zbr|ooo | 19:46 | |
| *** kjackal has joined #openstack-infra | 19:49 | |
| *** mriedem has quit IRC | 19:55 | |
| *** mriedem has joined #openstack-infra | 19:57 | |
| *** ociuhandu has joined #openstack-infra | 20:22 | |
| *** eharney has quit IRC | 20:22 | |
| *** eharney has joined #openstack-infra | 20:24 | |
| *** ociuhandu has quit IRC | 20:30 | |
| *** eharney has quit IRC | 20:30 | |
| *** kjackal has quit IRC | 20:35 | |
| *** Goneri has joined #openstack-infra | 20:43 | |
| *** slaweq has quit IRC | 20:52 | |
| *** tosky has joined #openstack-infra | 20:57 | |
| *** slaweq has joined #openstack-infra | 21:09 | |
| *** eharney has joined #openstack-infra | 21:17 | |
| *** slaweq has quit IRC | 21:18 | |
| *** goldyfruit_ has quit IRC | 21:23 | |
| *** rcernin has joined #openstack-infra | 21:28 | |
| *** slaweq has joined #openstack-infra | 21:33 | |
| *** kjackal has joined #openstack-infra | 21:34 | |
| *** armstrong has quit IRC | 21:40 | |
| *** slaweq has quit IRC | 21:41 | |
| *** igordc has quit IRC | 21:45 | |
| *** jaosorior has joined #openstack-infra | 21:45 | |
| *** igordc has joined #openstack-infra | 21:45 | |
| *** yamamoto has joined #openstack-infra | 21:48 | |
| *** rfolco has quit IRC | 21:55 | |
| *** goldyfruit_ has joined #openstack-infra | 21:55 | |
| *** slaweq has joined #openstack-infra | 22:01 | |
| *** ociuhandu has joined #openstack-infra | 22:02 | |
| *** jaosorior has quit IRC | 22:02 | |
| *** jaosorior has joined #openstack-infra | 22:03 | |
| *** iurygregory has quit IRC | 22:05 | |
| *** slaweq has quit IRC | 22:05 | |
| *** pkopec has quit IRC | 22:06 | |
| *** slaweq has joined #openstack-infra | 22:06 | |
| *** eharney has quit IRC | 22:08 | |
| *** ociuhandu has quit IRC | 22:11 | |
| *** kjackal has quit IRC | 22:13 | |
| *** kjackal has joined #openstack-infra | 22:14 | |
| ianw | i'll have another look at gitea06 now | 22:15 |
| ianw | i'm just going to reboot it first | 22:16 |
| *** slaweq has quit IRC | 22:18 | |
| ianw | alright, still failing | 22:18 |
| mnaser | ianw: failing to reboot or failing in $some-system-level-thing ? | 22:22 |
| ianw | sorry, still having the same failure discussed with "upload-pack" errors | 22:23 |
| ianw | i think there might be a step missing in "restore the gitea database" -- https://docs.openstack.org/infra/system-config/gitea.html | 22:25 |
| ianw | not sure where /root/gitea-mariadb.sql comes from | 22:26 |
| ianw | i guess i could remove the repos and try the create step | 22:26 |
| *** slaweq has joined #openstack-infra | 22:29 | |
| *** kaisers1 has joined #openstack-infra | 22:32 | |
| *** kaisers has quit IRC | 22:33 | |
| *** slaweq has quit IRC | 22:34 | |
| clarkb | ianw: the db is backedup to that location on each gitea host | 22:34 |
| clarkb | (sorry I've ended up being afk today due to a cold) | 22:34 |
| ianw | clarkb: hrm, not on gitea06 ...? | 22:35 |
| clarkb | hrm check crontab for the backup one liner | 22:36 |
| clarkb | maybe the path is wrong in the docs | 22:36 |
| ianw | ok, yeah there's a backup put into /var/backup ... i can fix that | 22:37 |
| ianw | there's a "exercise health checks on all repositories button" which i've clicked | 22:38 |
| ianw | Failed to health check repository (/data/git/repositories/openstack/nova.git): signal: killed | 22:38 |
| *** slaweq has joined #openstack-infra | 22:39 | |
| *** slaweq has quit IRC | 22:45 | |
| *** slaweq has joined #openstack-infra | 22:48 | |
| *** yamamoto has quit IRC | 22:49 | |
| *** jaosorior has quit IRC | 22:51 | |
| *** tkajinam has joined #openstack-infra | 22:54 | |
| openstackgerrit | Miguel Lavalle proposed opendev/irc-meetings master: Remove Neutron performance sub-team meeting https://review.opendev.org/694419 | 22:54 |
| *** slaweq has quit IRC | 23:00 | |
| *** dave-mccowan has quit IRC | 23:02 | |
| *** Lucas_Gray has joined #openstack-infra | 23:04 | |
| *** slaweq has joined #openstack-infra | 23:11 | |
| *** rlandy|rover is now known as rlandy|rover|bia | 23:14 | |
| *** KeithMnemonic has quit IRC | 23:15 | |
| *** slaweq has quit IRC | 23:16 | |
| *** dave-mccowan has joined #openstack-infra | 23:17 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Update gitea docs https://review.opendev.org/694427 | 23:22 |
| ianw | i've deleted the git repos and am replicating them again | 23:24 |
| *** Lucas_Gray has quit IRC | 23:26 | |
| *** xek__ has quit IRC | 23:27 | |
| *** Lucas_Gray has joined #openstack-infra | 23:30 | |
| ianw | it's still giving the same error | 23:31 |
| *** ociuhandu has joined #openstack-infra | 23:31 | |
| *** slaweq has joined #openstack-infra | 23:31 | |
| *** ociuhandu has quit IRC | 23:35 | |
| *** slaweq has quit IRC | 23:39 | |
| *** dchen has quit IRC | 23:40 | |
| ianw | i'm out of ideas short of rebuilding the whole server | 23:45 |
| *** Lucas_Gray has quit IRC | 23:47 | |
| *** dchen has joined #openstack-infra | 23:49 | |
| *** slaweq has joined #openstack-infra | 23:53 | |
| *** slaweq has quit IRC | 23:57 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
