| ianw | elasticsearch05.openstack.org appears to have gone bye-bye, and has gummed up the works on bridge | 00:08 |
|---|---|---|
| ianw | console is unresponsive, i'll reboot it | 00:09 |
| *** ryohayakawa has joined #opendev | 00:15 | |
| ianw | #status log rebooted unresponsive elasticsearch05.openstack.org | 00:22 |
| openstackstatus | ianw: finished logging | 00:22 |
| ianw | i have also cleared out lots of hung ansible stuff on bridge related to it | 00:23 |
| ianw | ERROR: for grafana b'manifest for opendevorg/grafana:latest not found: manifest unknown: manifest unknown' | 00:24 |
| ianw | manifest for opendevorg/grafana:latest not found: manifest unknown: manifest unknown | 00:24 |
| ianw | i'm guessing something has changed because the granfa change doesn't seem to like pulling from the intermediate registry any more | 00:25 |
| ianw | ... maybe the :latest tag? | 00:26 |
| ianw | https://hub.docker.com/r/opendevorg/grafana/tags ... change_737397_latest | 00:30 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Update grafana-container https://review.opendev.org/739384 | 00:37 |
| clarkb | ianw: the promote job should add the latest tag | 00:41 |
| clarkb | maybe that failedfor some reason? | 00:41 |
| ianw | yeah i don't see it @ https://review.opendev.org/#/c/737397/ | 00:42 |
| ianw | hrm, maybe it got dropped | 00:44 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Update grafana-container https://review.opendev.org/739384 | 00:49 |
| ianw | yeah the job was there, but not added to the run list | 00:49 |
| *** DSpider has quit IRC | 01:23 | |
| *** xiaolin has joined #opendev | 01:25 | |
| openstackgerrit | Merged opendev/system-config master: Update grafana-container https://review.opendev.org/739384 | 01:35 |
| openstackgerrit | Merged openstack/diskimage-builder master: add more python selections to gentoo https://review.opendev.org/735448 | 03:00 |
| *** ykarel|away is now known as ykarel | 04:13 | |
| *** bhagyashris|afk is now known as bhagyashris | 04:24 | |
| *** sgw1 has quit IRC | 04:28 | |
| *** raukadah is now known as chandankumar | 04:42 | |
| *** ysandeep|away is now known as ysandeep | 04:43 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Revert "Make ipa centos8 job non-voting" https://review.opendev.org/738728 | 05:37 |
| openstackgerrit | Merged opendev/system-config master: Grafana container deployment https://review.opendev.org/737406 | 05:56 |
| *** marios has joined #opendev | 05:58 | |
| *** sshnaidm|off is now known as sshnaidm|ruck | 06:17 | |
| *** ysandeep is now known as ysandeep|brb | 06:21 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Add host keys to inventory; give host key in launch-node script https://review.opendev.org/739412 | 06:33 |
| *** ysandeep|brb is now known as ysandeep | 06:34 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] add host keys on bridge https://review.opendev.org/739414 | 06:52 |
| *** tosky has joined #opendev | 07:25 | |
| *** dtantsur|afk is now known as dtantsur | 07:25 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] add host keys on bridge https://review.opendev.org/739414 | 07:33 |
| *** fressi has joined #opendev | 07:57 | |
| *** moppy has quit IRC | 08:01 | |
| *** moppy has joined #opendev | 08:01 | |
| *** DSpider has joined #opendev | 08:17 | |
| *** marios has quit IRC | 08:21 | |
| *** ysandeep is now known as ysandeep|lunch | 08:23 | |
| openstackgerrit | Carlos Goncalves proposed openstack/diskimage-builder master: Install yum and yum-utils on Red Hat family hosts https://review.opendev.org/735136 | 08:33 |
| *** persia has quit IRC | 08:44 | |
| *** persia has joined #opendev | 08:45 | |
| *** bhagyashris is now known as bhagyashris|afk | 08:50 | |
| *** hashar has joined #opendev | 08:53 | |
| *** ysandeep|lunch is now known as ysandeep | 08:56 | |
| *** marios has joined #opendev | 09:16 | |
| *** bhagyashris|afk is now known as bhagyashris | 09:19 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] add host keys on bridge https://review.opendev.org/739414 | 09:27 |
| openstackgerrit | Riccardo Pittau proposed openstack/diskimage-builder master: Convert multi line if statement to case https://review.opendev.org/734479 | 09:45 |
| *** tkajinam has quit IRC | 10:37 | |
| openstackgerrit | Radosław Piliszek proposed openstack/diskimage-builder master: [CI] Do not run functionality tests on docs change https://review.opendev.org/739367 | 10:38 |
| sshnaidm|ruck | folks, please merge https://review.opendev.org/#/c/739061 it prevents retry_limits in jobs | 10:51 |
| openstackgerrit | yatin proposed zuul/zuul-jobs master: [DNM] Test task in a role https://review.opendev.org/739463 | 11:02 |
| *** lpetrut has joined #opendev | 11:26 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Install yum and yum-utils on Red Hat family hosts https://review.opendev.org/735136 | 11:27 |
| *** ryohayakawa has quit IRC | 12:08 | |
| *** priteau has joined #opendev | 12:21 | |
| openstackgerrit | Rafael Folco proposed openstack/diskimage-builder master: Enable py3 on dib centos/rhel 7 https://review.opendev.org/736421 | 12:28 |
| *** ysandeep is now known as ysandeep|afk | 12:38 | |
| tristanC | clarkb: it seems like you have been looking into persistent-firewall already, could you please have a look at the proposed fix here: https://review.opendev.org/739061 | 12:39 |
| *** ysandeep|afk is now known as ysandeep | 12:56 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Stop bringing the test environment into the mocks https://review.opendev.org/735309 | 13:28 |
| *** redrobot has joined #opendev | 13:34 | |
| *** lpetrut has quit IRC | 14:32 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Build multi-arch python-base/python-builder https://review.opendev.org/726263 | 14:38 |
| mordred | corvus: with your updates to multi-arch images - I think we should be in good shape to land that ^^ now yeah? | 14:45 |
| *** ykarel is now known as ykarel|away | 14:46 | |
| *** mlavalle has joined #opendev | 14:52 | |
| *** factor has joined #opendev | 14:57 | |
| *** rajinir_ has joined #opendev | 15:02 | |
| *** hashar has quit IRC | 15:06 | |
| mordred | parking restrictions have been suspended in new orleans - this means it's about to be very wet | 15:11 |
| *** ysandeep is now known as ysandeep|away | 15:22 | |
| *** sshnaidm|ruck is now known as sshnaidm|mtg | 15:24 | |
| *** factor has quit IRC | 15:25 | |
| *** icarusfactor has joined #opendev | 15:25 | |
| *** sgw1 has joined #opendev | 15:27 | |
| corvus | mordred: +2! :) | 15:44 |
| corvus | tristanC: (cc clarkb, ianw, fungi) i've also been working with mwhahaha on trying to find the root cause of the iptables issue; we think it may be an errno being returned by the ansible command plugin, and mwhahaha suspects that it's a return value from a fcntl call, but we're still trying to catch it in the act. | 15:49 |
| corvus | sshnaidm|mtg: ^ fyi | 15:50 |
| corvus | on friday we were using 739061 to try to catch the issue in action | 15:51 |
| *** marios is now known as marios|out | 15:52 | |
| *** hashar has joined #opendev | 15:52 | |
| *** marios|out has quit IRC | 15:54 | |
| *** slittle1 has quit IRC | 16:12 | |
| *** priteau has quit IRC | 16:14 | |
| *** sgw1 has quit IRC | 16:21 | |
| tristanC | corvus: it sounds like 739061 would be helpfull in anycase, though i'm not entirely sure why retrying would work | 16:22 |
| *** sgw1 has joined #opendev | 16:23 | |
| *** sgw1 has quit IRC | 16:31 | |
| *** sgw1 has joined #opendev | 16:34 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/project-config master: Add "Review-Priority" label to qa projects https://review.opendev.org/739550 | 16:45 |
| openstackgerrit | Ghanshyam Mann proposed openstack/project-config master: Add "Review-Priority" label to qa projects https://review.opendev.org/739550 | 16:47 |
| *** dtantsur is now known as dtantsur|afk | 16:54 | |
| *** slittle1 has joined #opendev | 17:06 | |
| corvus | tristanC: yeah, i don't know that we've seen retrying actually work, but it seems plausible that it could; certainly unlikely to hurt. i'm not sure if folks have set depends-on to that yet. | 17:09 |
| *** priteau has joined #opendev | 17:22 | |
| *** hashar has quit IRC | 17:22 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Make persist-iptables more robust https://review.opendev.org/739061 | 17:33 |
| *** priteau has quit IRC | 18:07 | |
| openstackgerrit | Merged openstack/diskimage-builder master: add musl profile to gentoo https://review.opendev.org/737394 | 18:52 |
| openstackgerrit | Merged openstack/diskimage-builder master: Switch to newer openstackdocstheme and reno versions https://review.opendev.org/732490 | 18:52 |
| *** xiaolin has quit IRC | 18:56 | |
| *** sshnaidm|mtg is now known as sshnaidm|ruck | 19:01 | |
| *** icarusfactor has quit IRC | 19:16 | |
| *** shtepanie has joined #opendev | 19:18 | |
| mnaser | so this might sound a little wild, but.. is there any projects that do cross-tenant third-party-ci inside opendev? | 19:26 |
| mnaser | for example, if the vexxhost tenant wants to do 3pci for openstack/keystone so that they get a signal if they break vexxhost/openstack-operator | 19:26 |
| mnaser | i mean, i could totally do normal 3pci, but it seems like it might more sense in the same context/space/place | 19:27 |
| mordred | mnaser: this is a use case that folks had a while ago (before zuul v3) and I believe is why jhesketh added multi-connection support. my brain is having a tough time paging in current context | 19:35 |
| mordred | corvus: ^^ | 19:35 |
| mnaser | mordred: i mean, i think it probably is very likely possible, though i guess it may require vexxhost getting its own pipeline config which maybe a little bit more than opendev wants to deal with | 19:38 |
| mnaser | (understandable-y so, but if it seems like an idea folks are behind, i'm happy to do it for future tenants reference) | 19:38 |
| corvus | i think the tenant config part is less of an issue than the connection part. it would need a new gerrit connection | 19:41 |
| corvus | (in order to leave a vote as a different user) | 19:42 |
| corvus | it would definitely be extra work/complexity/etc for the opendev admin team, though probably less so than the total human cost of vexxhost running another zuul just for this. :) | 19:43 |
| mordred | corvus: nod - so that's still the situation - we'd want a second zuul account that opendev's zuul would use for that purpose | 19:49 |
| mnaser | corvus: i mean, we can easily start doing 3pci but the idea was that if it made sense to 'pave the way' for future possibilities of tenants interacting with each other | 19:49 |
| mnaser | if that's not a use case we're looking to handle at some point, then yeah, it could be interesting to check it out | 19:50 |
| mnaser | i'm thinking especially since maybe if airship has its own tenant and wants to vote on i dunno, openstack-helm for example | 19:50 |
| mnaser | mordred: perhaps one single extra opendev connection that would do 3pci on all other tenants.. if that makes sense -- but that perhaps doubles the work the zuul-scheduler needs to do | 19:51 |
| fungi | also every time this comes up, the idea of having reporting via multiple user accounts seems like a workaround for our (more accurately i guess gerrit's) data model not being able to adequately represent the information we want to convey | 20:06 |
| fungi | are there aspects of the new checks api which would allow us to do this with a single account? | 20:06 |
| fungi | (assuming zuul had a different means of grouping vexxhost vs openstack builds for the same items and pipelines) | 20:07 |
| fungi | like if we had some way of indicating that a given buildset came from the gating tenant vs merely a non-blocking advisory tenant | 20:10 |
| mnaser | fungi: well i mean, technically, we _could_ have zuul report twice, one for another pipeline called vexxhost-3pci and it would enqueue at once? similar to our arm64 pipelnie today? | 20:10 |
| corvus | fungi: i don't think checks is advantageous here; the main issue is that a single column gets a single vote per user (checks would be the same, a single check gets a single result) | 20:10 |
| corvus | yeah, a key think about arm64 is that it doesn't vote | 20:11 |
| fungi | in the current plan is a check mapped 1:1 to a pipeline/buildset? (i gather subchecks are the 1:1 mapping to an individual job/build)? | 20:12 |
| mnaser | i think my initial intention was non-voting external 3pci too | 20:12 |
| corvus | fungi: yes (pipeline) | 20:12 |
| corvus | mnaser: if you're not concerned with leaving a vote, then a second account isn't necessary | 20:13 |
| corvus | i am told, however, that "people ignore results without votes" | 20:13 |
| fungi | i'm told that people ignore third-party ci results with votes too | 20:13 |
| fungi | hence tripleo folks asking last week whether they could make their rdo ci block tripleo changes from merging | 20:14 |
| corvus | yep | 20:14 |
| corvus | mnaser: maybe if you want to pursue this, that's a good way to start? fewer technical hurdles, mostly policy/procedural ones, and could potentially be upgraded to voting later with addition of another account. | 20:15 |
| mnaser | corvus, fungi: heh, well, yeah, i guess it can be ignored. i'm hoping to try and ask nicely to 'watch' for the signal | 20:17 |
| mnaser | i guess my + side is that these jobs will probably finish faster than the normal check pipeline so maybe eyes will be on them :) | 20:18 |
| *** hashar has joined #opendev | 20:54 | |
| openstackgerrit | Oleksandr Kozachenko proposed openstack/project-config master: Add openstack/devstack-plugin-ceph in vexxhost tenant https://review.opendev.org/739602 | 20:56 |
| openstackgerrit | Oleksandr Kozachenko proposed openstack/project-config master: Add openstack/devstack-plugin-ceph in vexxhost tenant https://review.opendev.org/739602 | 21:03 |
| *** xiaolin has joined #opendev | 21:03 | |
| *** xiaolin has quit IRC | 21:11 | |
| donnyd | soooo I was on-site today and just got back... my air handler is pretty much done.. going to need to turn OE off for a while.. not sure when I can get it fixed | 21:13 |
| openstackgerrit | Donny Davis proposed openstack/project-config master: Air Handler is failed - turn OE off for now https://review.opendev.org/739605 | 21:15 |
| mnaser | donnyd: sorry to hear | 21:15 |
| donnyd | mnaser: so goes being a little guy I guess | 21:16 |
| donnyd | I have kept it chugging along for the last year or so when it crapped the bed last summer | 21:17 |
| mnaser | donnyd: appreciate every bit of providing infra though, every build minute helps :) | 21:17 |
| donnyd | I will now need to come up with a plan to find/build a new one - repairing the old one isn't going to cut it anymore I guess | 21:18 |
| donnyd | I am happy to contribute where and when I can.. this community is important to me | 21:18 |
| fungi | donnyd: your help is invaluable and MUCH appreciated! | 21:20 |
| fungi | having run both commercial and home data centers, i can agree that cooling is probably tied with power distribution for the hardest problems to solve | 21:21 |
| donnyd | it has been working ok till this morning.. but it finally just died | 21:26 |
| donnyd | I will let you guys know when its back up | 21:26 |
| fungi | good luck! i've been struggling with our residential unit here too, we just paid rather too much to have it all replaced a few weeks ago and it's still not quite working how we'd like | 21:28 |
| donnyd | I have a 15K bill to get it fixed by a professional - which unfortunately is not in the cards at the moment | 21:29 |
| donnyd | THankfully the whole system isn't failed - just the part that powers the D/C is unable to cope | 21:30 |
| fungi | yep, not good. that's roughly what the replacement for the entire hvac for our house ran (including rerunning ducts) | 21:31 |
| openstackgerrit | Merged openstack/project-config master: Air Handler is failed - turn OE off for now https://review.opendev.org/739605 | 21:35 |
| *** Eighth_Doctor is now known as Eleventh_Doctor | 22:16 | |
| *** Eleventh_Doctor is now known as Eighth_Doctor | 22:29 | |
| *** tkajinam has joined #opendev | 22:42 | |
| *** shtepanie has quit IRC | 22:48 | |
| *** tosky has quit IRC | 22:49 | |
| ianw | review-test.opendev.org was out of space, which was causing the base job to fail | 22:54 |
| ianw | track_upstream.log was full of failure to connect messages which seems to be the culprit, i have removed it | 22:54 |
| fungi | i thought we merged a change to skip review-test in those jobs? | 22:55 |
| ianw | 8.4Gtrack_upstream.log | 22:56 |
| ianw | umm, it's full again? something screwy going on here | 23:00 |
| ianw | # docker ps | wc -l | 23:01 |
| ianw | 210 | 23:02 |
| ianw | that doesn't seem right | 23:02 |
| *** mlavalle has quit IRC | 23:04 | |
| ianw | well i've stopped all 210 containers and some space has freed up. clearly something got in a loop | 23:05 |
| ianw | it doesn't seem the container starts on boot. i'll just leave it for now; it is at least not going to affect the base playbook run now | 23:08 |
| ianw | clarkb: i know you're not around but is there a meeting tomorrow? | 23:09 |
| clarkb | ianw: I havemt seen anyone volunteer to host it yet | 23:10 |
| ianw | clarkb: ok, i can, just to keep something in the record. i will send an agenda | 23:12 |
| fungi | i'll be around, could be a good opportunity to talk about what to do with the iptables rules for china unicom | 23:24 |
| *** factor has joined #opendev | 23:31 | |
| *** hrw has quit IRC | 23:44 | |
| *** hrw has joined #opendev | 23:44 | |
| *** hashar has quit IRC | 23:56 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!