| fungi | yeah, i expected that might happen | 00:03 |
|---|---|---|
| fungi | i also mentioned it as a possibility during the openstack tc meeting earlier today (yesterday utc), in case it's also happening to any of openstack's jobs which might have relied on those labels | 00:04 |
| corvus | NeilHanlon: is there a bug in zuul docs? | 00:04 |
| opendevreview | Ian Wienand proposed opendev/system-config master: docs: reorganise around a open infrastructure overview https://review.opendev.org/c/opendev/system-config/+/830785 | 01:17 |
| opendevreview | Ian Wienand proposed opendev/system-config master: docs: reorganise around a open infrastructure overview https://review.opendev.org/c/opendev/system-config/+/830785 | 01:18 |
| *** wxy-xiyuan_ is now known as wxy-xiyuan | 01:45 | |
| corvus | executor restart complete...i'm inclined to see if we can get some of the other changes landed and restart the schedulers after that tomorrow | 01:47 |
| NeilHanlon | corvus: I think so, but it may have just been my interpretation. https://zuul-ci.org/docs/zuul/latest/config/job.html#attr-job.nodeset says that nodesets specified as a dictionary in a job (instead of a string) need not specify the 'name' key as they are interpreted as anonymous pools. However the Nodeset docs say both are required, and indeed a | 03:54 |
| NeilHanlon | zuul run will error if the name key doesn't exist (e.g.: https://review.opendev.org/c/openstack/openstack-ansible/+/823573/11#message-244350641dffe814a1c472301b171b7b7bf5cda6) | 03:54 |
| NeilHanlon | i'd be happy to fixup docs and/or file a bug if needed | 03:55 |
| *** frenzy_friday is now known as frenzyfriday|ruck | 04:38 | |
| *** ysandeep|out is now known as ysandeep | 04:47 | |
| *** frenzyfriday|ruck is now known as frenzyfriday|rover | 05:40 | |
| *** ysandeep is now known as ysandeep|mtg | 07:13 | |
| *** jpena|off is now known as jpena | 08:10 | |
| *** ysandeep|mtg is now known as ysandeep|lunch | 08:45 | |
| mnasiadka | hello | 09:59 |
| mnasiadka | Is there a way to get newer Ansible version than 2.9 for Zuul executed playbooks? | 10:00 |
| *** rlandy_ is now known as rlandy|ruck | 11:12 | |
| *** bhagyashris|ruck is now known as bhagyashris | 11:36 | |
| *** pojadhav is now known as pojadhav|brb | 13:09 | |
| fungi | mnasiadka: at the moment, zuul only supports ansible 2.8 and 2.9: https://opendev.org/zuul/zuul/src/branch/master/zuul/lib/ansible-config.conf | 13:19 |
| mnasiadka | fungi: ok then, thank you - I'll need to live with ansible_os_family: "Rocky" instead of "RedHat" ;-) | 13:19 |
| fungi | mnasiadka: adding a newer ansible version to zuul will entail a change something along the lines of https://review.opendev.org/674854 which was what added support for 2.9 | 13:20 |
| mnasiadka | fungi: doesn't look bad, I'll think about adding 2.11 - thanks | 13:23 |
| fungi | probably we should add 2.10 first or at the same time since i don't think zuul has ever supported a discontiguous series of minor versions, but i guess we can hash that out in review | 13:24 |
| opendevreview | yatin proposed zuul/zuul-jobs master: [multi-node-bridge] Add script to configure connectivity https://review.opendev.org/c/zuul/zuul-jobs/+/762650 | 13:44 |
| mnasiadka | fungi: https://review.opendev.org/c/zuul/zuul/+/757452 - seems there was an approach, but abandoned. | 13:46 |
| fungi | yeah, looks like it was mostly passing tests but that was over a year ago when it was last tested | 13:53 |
| fungi | i'm happy to restore that change if you want to work on updating it | 13:54 |
| mnasiadka | sure, why not, can learn something new ;) | 13:54 |
| fungi | it's restored now | 13:56 |
| mnasiadka | thanks, rebased - let's see | 14:04 |
| *** pojadhav|brb is now known as pojadhav|afk | 14:09 | |
| *** iurygregory_ is now known as iurygregory | 14:28 | |
| mnasiadka | fungi: seems Zuul is stripping 0 from 2.10 - and tries to run ansible 2.1 - https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_aba/757452/7/check/zuul-stream-functional-2.10/aba5a83/job-output.txt - any idea why? | 14:29 |
| fungi | mnasiadka: yaml is probably interpreting that version field as a float instead of a str | 14:33 |
| fungi | wrap teh value in quotes | 14:33 |
| mnasiadka | ok, makes sense | 14:33 |
| fungi | we ran into a similar situation elsewhere recently trying to add python 3.10 jobs | 14:37 |
| fungi | we've apparently not done a great job of quoting version strings in configs | 14:37 |
| fungi | it's really only a problem for two-component versions since they look like floats, three-component versions get treated as strings because they contain more than one . | 14:38 |
| * NeilHanlon is still sad the patch to backport rocky into 2.9 didn't make it | 14:41 | |
| corvus | NeilHanlon: ah, i could see how that would be confusing. the docs are trying to say that the first name attribute must be absent: https://zuul-ci.org/docs/zuul/latest/config/nodeset.html#attr-nodeset.name | 14:48 |
| corvus | NeilHanlon: not the second one which applies to the nodes: https://zuul-ci.org/docs/zuul/latest/config/nodeset.html#attr-nodeset.nodes.name | 14:48 |
| corvus | NeilHanlon: ie, the "nodeset" should not have a name since it's anonymous, but the nodes within the nodeset should. | 14:48 |
| NeilHanlon | ahhh, yes. okay that makes sense lol | 14:49 |
| corvus | maybe if we added the word nodeset before name there... or just added a parenthetical ("but the nodes still need names").... or, added a note in the nodeset side.... | 14:50 |
| corvus | NeilHanlon: how's this look? remote: https://review.opendev.org/c/zuul/zuul/+/831959 Clarify anonymous nodeset docs [NEW] | 14:53 |
| NeilHanlon | yep, I think that clears it up! | 14:55 |
| *** dviroel is now known as dviroel|lunch | 14:56 | |
| opendevreview | yatin proposed zuul/zuul-jobs master: [multi-node-bridge] Add script to configure connectivity https://review.opendev.org/c/zuul/zuul-jobs/+/762650 | 15:06 |
| corvus | i'm going to complete the zuul rolling restart now (schedulers+web) | 15:44 |
| corvus | this will entail both of the schedulers and the web service going offline for a few minutes; we will miss some gerrit events, but it shouldn't be a long outage. existing queue states will remain. | 15:45 |
| fungi | thanks corvus! | 15:49 |
| corvus | first scheduler is up; web and 2nd scheduler are on their way | 15:53 |
| corvus | i'm going to restart the nodepool launchers now | 15:53 |
| corvus | that's done | 15:54 |
| opendevreview | Merged openstack/diskimage-builder master: Correctly create DIB_ENV variable and dib_environment file https://review.opendev.org/c/openstack/diskimage-builder/+/828899 | 15:55 |
| corvus | #status log restarted zuul and nodepool launchers; schedulers are at bb2b38c4be8e2592dd2fb7f1f4b631436338ec98 executors a few commits behind, and launchers at ac35b630dfbba7c6af90398b3ea3c82f14eabbde | 15:56 |
| opendevstatus | corvus: finished logging | 15:56 |
| fungi | awesome | 15:57 |
| corvus | cool, the node request time graph on https://grafana.opendev.org/d/2e89fb78e5/zuul-performance-metrics?orgId=1&var-tenant=openstack is alive now | 15:58 |
| fungi | whoa | 16:00 |
| corvus | and everybody is back and running now | 16:00 |
| fungi | that node request time graph is going to be interesting for seeing how different pipeline priorities compare | 16:02 |
| corvus | yeah that should show up there | 16:03 |
| opendevreview | yatin proposed zuul/zuul-jobs master: [multi-node-bridge] Add script to configure connectivity https://review.opendev.org/c/zuul/zuul-jobs/+/762650 | 16:12 |
| *** dviroel|lunch is now known as dviroel | 16:20 | |
| clarkb | looks like we are still a go for landing https://review.opendev.org/c/opendev/system-config/+/831462 today to update the zuul-registry deployment on insecure-ci-regitry's user? corvus is that something you might be interested in revieweing given connection to zuul? Otherwise I think I'll approve it after my meeting this morning | 16:23 |
| clarkb | corvus: one thing I notice on the performance metrics is the really good compression ratio | 16:26 |
| corvus | clarkb: nothing jumps out at me; lgtm thanks :) | 16:26 |
| corvus | clarkb: yeah, i'm wondering if it's "too" good, but i can't find an error in the code | 16:26 |
| *** ysandeep is now known as ysandeep|out | 16:31 | |
| *** marios is now known as marios|out | 16:52 | |
| clarkb | I have approved https://review.opendev.org/c/opendev/system-config/+/831462 and will monitor it as it does its thing | 16:55 |
| fungi | thanks! | 17:00 |
| *** jpena is now known as jpena|off | 17:06 | |
| opendevreview | Merged opendev/system-config master: Adds support for running zuul-registry as a non-root user https://review.opendev.org/c/opendev/system-config/+/831462 | 17:16 |
| clarkb | that is behind the hourly deploy jobs | 17:28 |
| opendevreview | James E. Blair proposed openstack/project-config master: Add more stats to zuul performance metrics dashboard https://review.opendev.org/c/openstack/project-config/+/832120 | 17:45 |
| corvus | infra-root: ^ if you have a second for a quick review, that would be nice to get out there | 17:45 |
| corvus | i'm also wondering if reconfiguration_time is something we should add to the main zuul status page.. but maybe let's see what it looks like here first | 17:46 |
| fungi | lgtm | 17:48 |
| corvus | on the ad-hoc graph i'm looking at, it looks like openstack reconfigures about every 5 minutes... and takes 2.5 minutes to do so. | 17:48 |
| corvus | that's a wee bit more often than i would have expected. | 17:48 |
| fungi | yeah i had no idea it was that frequent | 17:49 |
| opendevreview | Merged openstack/project-config master: Add more stats to zuul performance metrics dashboard https://review.opendev.org/c/openstack/project-config/+/832120 | 18:12 |
| clarkb | 831462 triggered more jobs than I expected. I'm guessing due to the group vars update. We might want to look at those files specifications again | 18:14 |
| clarkb | It shouldn't be a problem. Just takes longer to run the job we're actually interested in | 18:14 |
| opendevreview | yatin proposed zuul/zuul-jobs master: [multi-node-bridge] Add script to configure connectivity https://review.opendev.org/c/zuul/zuul-jobs/+/762650 | 18:15 |
| clarkb | oh heh the hourly jobs appear to have updated the insecure-ci-registry so this whole time I've been waiting for the job to run it was already done | 18:22 |
| clarkb | the process is up and running as the expected user | 18:23 |
| clarkb | I guess I should recheck a change that willtalk to it | 18:23 |
| clarkb | https://review.opendev.org/c/opendev/system-config/+/828184 has been rechecked and show talk to it | 18:23 |
| clarkb | all that to say initial indications are this is happy but double checking with actual jobs now | 18:25 |
| corvus | what's the status of the new lb? | 18:32 |
| corvus | looks like the config changes merged... i'm guessing no one has launched a vm yet and that's next? | 18:33 |
| clarkb | corvus: ya its been on my todo list to try and do that but I keep getting distracted | 18:33 |
| clarkb | but ya we need to boot the instance, test it, then update dns to point at the lb instead of zuul02 | 18:33 |
| clarkb | Next week I'll be afk a bunch for meetings. But if it isn't done by week after I can try and put it higher on the priority queue | 18:34 |
| corvus | i'm guessing we want it in rax-dfw? | 18:40 |
| clarkb | corvus: ya that should match the region of the schedulers | 18:40 |
| corvus | lemme see if i can kick that off now | 18:40 |
| corvus | we have an 8gb vm for gitea-lb... do we want to scale that down for zuul? | 18:41 |
| corvus | omg yes. | 18:42 |
| clarkb | corvus: I think we can. The main consideration there is network bw and I think that scales with flavor size | 18:42 |
| clarkb | corvus: however zuul web network traffic is pretty small compared to git traffic | 18:42 |
| corvus | oh yeah that must be why we did that. i'll consult the rax tables. | 18:42 |
| fungi | is that the case in vexxhost? | 18:42 |
| clarkb | I'm not sure if it is the case in vexxhost | 18:42 |
| fungi | i thought gitea-lb was in vexxhost, so that doesn't quite explain why we used an 8gb flavor there | 18:43 |
| corvus | we're using basically no cpu and ram on gitealb | 18:44 |
| fungi | but yes, if we're creating the zuul lb in rackspace we'll want to be mindful of the flavor-specific bandwidth | 18:44 |
| clarkb | corvus: ya haproxy is extremely efficient. We are limited by the application not haproxy | 18:44 |
| corvus | ah rxtx factor is what we're looknig for i think | 18:45 |
| fungi | yep, that's it | 18:45 |
| corvus | we don't have a way to break out web traffic from zuul02... but we can probably subtract zuul01 from zuul02 and get a rough estimate of req bandwidth | 18:46 |
| fungi | should be more than accurate for that, yes. i expect we'll want to have a lot of breathing room anyway | 18:48 |
| corvus | back-of-napkin math says: zuul02 outbound bandwidth average 2.5mbps, zuul01 is 1.5; so we need 1mbps :) | 18:48 |
| corvus | so literally any flavor they have should work | 18:48 |
| corvus | want to try a 2GB instance? 2vcpus, 240mbps, 80gb disk | 18:49 |
| clarkb | wfm | 18:50 |
| clarkb | I think typcially we have used the "performance" flavors which have smaller disks which may be helpful here since we don't need much disk | 18:50 |
| corvus | sorry, we'd do performance flavor, so 40gb disk and 400mbps. vcpu/ram is the same | 18:50 |
| clarkb | sounds great | 18:50 |
| corvus | making it so | 18:51 |
| corvus | focal still the image du jour? | 18:53 |
| clarkb | yes | 18:55 |
| fungi | 240mbps sounds like more than plenty | 19:01 |
| fungi | even if we missed a MB/s to Mbps conversion in there it's still an order of magnitude beyond what we need | 19:02 |
| corvus | then 400mbps is even better :) | 19:02 |
| fungi | yep, wfm | 19:02 |
| fungi | moar betterer | 19:02 |
| corvus | running unattended upgrades is taking a wee bit of time | 19:13 |
| corvus | good, that was the prompt it needed to finish :) | 19:13 |
| corvus | the rdns scripts don't seem to work... | 19:15 |
| corvus | ERROR: 'response' | 19:15 |
| opendevreview | James E. Blair proposed opendev/system-config master: Add zuul-lb01 to inventory https://review.opendev.org/c/opendev/system-config/+/832127 | 19:17 |
| clarkb | corvus: it responds with error but the records are created | 19:18 |
| opendevreview | James E. Blair proposed opendev/zone-opendev.org master: Add zuul-lb01 https://review.opendev.org/c/opendev/zone-opendev.org/+/832128 | 19:19 |
| corvus | clarkb: error: success | 19:19 |
| fungi | that's marvellous | 19:19 |
| corvus | straight from the ministry of truth | 19:20 |
| fungi | i figured it was ministry of information retrieval | 19:21 |
| fungi | the truth will MAKE you free | 19:21 |
| elodilles | sorry, fyi, we now had again the issue with the create-yoga patches: they were not enqueued to the check queue (i mean they were, but no job matched for them if i remember correctly the error). do you need to leave them as they are to be able to debug those, or is that OK if I 'recheck' them? | 19:24 |
| elodilles | ( https://review.opendev.org/q/topic:create-yoga+is:open ) | 19:24 |
| fungi | elodilles: you can recheck them | 19:24 |
| elodilles | fungi: ack, thanks | 19:24 |
| fungi | we expect it's a race condition with layouts updating from the branch creation | 19:25 |
| fungi | which has been there for a while, but may have worsened when we started running more than one scheduler | 19:25 |
| elodilles | ack, thanks for the details! | 19:26 |
| opendevreview | Merged opendev/zone-opendev.org master: Add zuul-lb01 https://review.opendev.org/c/opendev/zone-opendev.org/+/832128 | 19:34 |
| fungi | jentoio: clarkb: looks like deployment of the updated registry container config finally happened | 19:36 |
| clarkb | fungi: ya I discovered it actually happend a previously via the hourly job | 19:37 |
| fungi | oh, awesome | 19:38 |
| clarkb | anyway I recheked the gitea 1.16.3 change to see that it handles things happily and so far it seems fine | 19:38 |
| opendevreview | Merged opendev/system-config master: Add zuul-lb01 to inventory https://review.opendev.org/c/opendev/system-config/+/832127 | 19:55 |
| corvus | zuul-lb service playbook is running | 20:24 |
| corvus | and now seems to be proxying | 20:24 |
| clarkb | it redirected me to zuul.o.o so seems it hit the backend | 20:25 |
| clarkb | I guess I need to set up /etc/hosts override to check it properly | 20:25 |
| corvus | i went to https://zuul-lb01.opendev.org/tenants and got a cert warning but otherwise works | 20:26 |
| clarkb | my /etc/hosts override shows that it seems to owrk from here as well | 20:26 |
| fungi | i put "2001:4800:7818:104:be76:4eff:fe02:f30f zuul.opendev.org" in my /etc/hosts and went to https://zuul.opendev.org/ with my browser, no problems | 20:27 |
| clarkb | note only zuul02 is in the balance pool currnetly | 20:27 |
| clarkb | and zuul01 is not running a zuul-web but changing that should be straightforward. We put a cert on zuul01 already iirc | 20:28 |
| fungi | yep, that's the one the cert says i got | 20:28 |
| fungi | zuul02 i mean | 20:28 |
| corvus | working on a change for zuul01 now | 20:28 |
| opendevreview | James E. Blair proposed opendev/system-config master: Run zuul-web on zuul01 and add to load balancer https://review.opendev.org/c/opendev/system-config/+/832138 | 20:32 |
| corvus | i think we can go ahead and manually start the zuul-web process on 01; i'll go ahead and do that | 20:32 |
| clarkb | does it go through apache? | 20:33 |
| corvus | yes | 20:33 |
| corvus | i think we were only not running the actual zuul-web service | 20:34 |
| clarkb | gotcha | 20:34 |
| opendevreview | James E. Blair proposed opendev/zone-opendev.org master: Point zuul.o.o at the lb https://review.opendev.org/c/opendev/zone-opendev.org/+/832139 | 20:35 |
| corvus | i think we should be able to land those changes at any time and in any order | 20:36 |
| clarkb | corvus: fwiw the discussion about haproxy checks prompted me to look at that for gitea. If you think that would be helpful for zuul as well I can look into an update for that too | 20:36 |
| clarkb | basically we do an http check agsint the backend which checks both apache and the srevice behind it are functional | 20:36 |
| corvus | yeah that sounds like it would be better | 20:37 |
| corvus | especially since zuul-web can take a long time to start | 20:38 |
| clarkb | ok I'll look at that | 20:38 |
| corvus | looks like we're just doing tcp checks now | 20:38 |
| corvus | clarkb: thank | 20:38 |
| corvus | s | 20:38 |
| corvus | in other news, the points on the grafana dashboard look ridiculous, i'll figure out how to to make them less fisher price | 20:39 |
| fungi | yeah, those are some large dots relative to the spacing | 20:39 |
| *** dviroel is now known as dviroel|out | 20:43 | |
| opendevreview | James E. Blair proposed openstack/project-config master: Fix reconfiguration time graph https://review.opendev.org/c/openstack/project-config/+/832140 | 20:43 |
| corvus | that fixes an oops on one of the graphs and also the dots | 20:44 |
| clarkb | corvus: small issue on the zuul01 change one sec | 20:45 |
| clarkb | and posted | 20:45 |
| corvus | ports! | 20:45 |
| clarkb | I noticed it putting the checks change together | 20:46 |
| opendevreview | James E. Blair proposed opendev/system-config master: Run zuul-web on zuul01 and add to load balancer https://review.opendev.org/c/opendev/system-config/+/832138 | 20:46 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Do more robust checks against zuul-web with haproxy https://review.opendev.org/c/opendev/system-config/+/832141 | 20:52 |
| clarkb | I think ^ should do it for the checks | 20:52 |
| corvus | clarkb: cool, lgtm. another option would be to have apache proxy the /health/ready endpoint (which is on a separate backend port) and check that. but we intentionally don't start cherrypy until we're ready anyway, so this should be equivalent. | 20:56 |
| fungi | corvus: inline question on 832138 | 21:08 |
| opendevreview | James E. Blair proposed opendev/system-config master: Run zuul-web on zuul01 and add to load balancer https://review.opendev.org/c/opendev/system-config/+/832138 | 21:11 |
| corvus | thx fixed | 21:11 |
| fungi | thanks! | 21:11 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Do more robust checks against zuul-web with haproxy https://review.opendev.org/c/opendev/system-config/+/832141 | 21:13 |
| clarkb | should we hold off on landing the dns update until those two change sland nd are confirmed working? | 21:13 |
| fungi | i guess we could, just for a chance to double-check | 21:16 |
| corvus | eh, i'm not too worried. either way :) | 21:17 |
| clarkb | ya I gues its simple to fix if something has a sad | 21:18 |
| corvus | and the worst case scenario of zuul-web being semi-inaccessible on a friday afternoon isn't terrible | 21:19 |
| fungi | agreed | 21:20 |
| clarkb | my typing is extra bad today because I'm on the laptop ensuring everything works before I dpeend on it next week. Unfortunately the network card in this thing has very variable rtt to my AP. I might swap it out with an intel ac200 in the future | 21:21 |
| jentoio | fungi: cool, glad to see it role finally | 21:34 |
| corvus | clarkb: if you have a quick sec for this dashboard fix that'd be swell: https://review.opendev.org/832140 | 21:40 |
| clarkb | oh yup missed it earlier | 21:40 |
| corvus | fungi: and if you have a sec for https://review.opendev.org/832139 that'd be groovy | 21:40 |
| opendevreview | Merged openstack/project-config master: Fix reconfiguration time graph https://review.opendev.org/c/openstack/project-config/+/832140 | 21:55 |
| fungi | done | 22:04 |
| opendevreview | Merged opendev/zone-opendev.org master: Point zuul.o.o at the lb https://review.opendev.org/c/opendev/zone-opendev.org/+/832139 | 22:08 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Do more robust checks against zuul-web with haproxy https://review.opendev.org/c/opendev/system-config/+/832141 | 22:18 |
| clarkb | corvus: fungi ^ I got the variable scoping wrong there. Yay for testing | 22:18 |
| fungi | oh neat | 22:18 |
| corvus | nice! | 22:19 |
| clarkb | I see dns has updated for me | 22:43 |
| fungi | for me as well | 22:44 |
| fungi | webui still solid here | 22:45 |
| fungi | if somewhat quiet. but it's the weekend | 22:45 |
| opendevreview | Merged opendev/system-config master: Run zuul-web on zuul01 and add to load balancer https://review.opendev.org/c/opendev/system-config/+/832138 | 22:53 |
| fungi | corvus: the smaller dots look much better | 23:06 |
| corvus | dns updated for me, looks good | 23:20 |
| corvus | i think i'll try shutting down zuul-web on 01 and see how the lb responds | 23:20 |
| clarkb | corvus: I don't know if thta has applied yet? | 23:21 |
| corvus | yeah, logs are saying only 02 at this point | 23:21 |
| clarkb | its running jobs for that change now | 23:21 |
| clarkb | so should be soon | 23:21 |
| opendevreview | James E. Blair proposed opendev/system-config master: Allow zuul-lb to send stats to graphite https://review.opendev.org/c/opendev/system-config/+/832148 | 23:26 |
| opendevreview | James E. Blair proposed openstack/project-config master: Add zuul load balancer dashboard https://review.opendev.org/c/openstack/project-config/+/832149 | 23:29 |
| corvus | clarkb: fungi something to do while waiting :) | 23:29 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Don't run infra-prod-run-refstack on all group var updates https://review.opendev.org/c/opendev/system-config/+/832150 | 23:31 |
| clarkb | ^ bugged me looking at the jobs running for the lb updates | 23:32 |
| clarkb | corvus: the bits per second entries have a scale of 8 does that mean we are converting bytes to bits? | 23:33 |
| corvus | erm, i copied the existing dashboard and s/git/zuul/ :) | 23:37 |
| corvus | btw, while i'm looking at that, does anyone know how to make grafana not look like a clickbait news site? | 23:38 |
| clarkb | ya isn't it great they give a feed of their blog in the software? | 23:39 |
| corvus | "Learn these 10 secret tricks to grafana the most from your grafana" or whatever | 23:39 |
| clarkb | https://www.reddit.com/r/grafana/comments/fn0f2d/disable_news_panel_grafana_66/fl7i2rh/ I'm not sure how to parse that | 23:40 |
| clarkb | sounds like if you have an account you change your account's preferences | 23:41 |
| clarkb | but not sure how to drop it from the main page | 23:41 |
| corvus | clarkb: confirmed that haproxy reports bytes/sec and our dashboard translates to bits/s | 23:41 |
| corvus | so we can straight up see that opendev.org looks like it does 100mbit continuous | 23:42 |
| clarkb | the load balancer should be getting updated to do both 01 and 02 nowish | 23:42 |
| clarkb | ya I think it just started the replacement process with the new config | 23:43 |
| corvus | and peaks around 400mbps | 23:43 |
| clarkb | I got what looked like a lack of css on first reload after the lb updated. I did a hrad refresh and it seemed fine | 23:44 |
| clarkb | I am talking to 01 according to the cert | 23:44 |
| corvus | i am also talking to 01 | 23:44 |
| corvus | some people are getting 02 though, i see log entries for both | 23:46 |
| corvus | i'm going to shut down 01 now | 23:46 |
| clarkb | the tcp check doesn't quite do what we want there. Thats ok | 23:47 |
| clarkb | (it was a known issue) | 23:47 |
| clarkb | since apache is listening on those ports the tcp check passes and we get a 500 error | 23:47 |
| clarkb | my chang ewhich should land soon should fix that | 23:47 |
| corvus | oh has that change not landed yet? | 23:47 |
| corvus | hah whoops ok | 23:47 |
| clarkb | not yet. I can manually disable 01 in haproxy | 23:47 |
| clarkb | that will fix it | 23:47 |
| clarkb | we just have to manually reenable it again | 23:48 |
| corvus | nah, i mostly wanted to verify your change :) | 23:48 |
| corvus | i thought they were both going in together | 23:48 |
| corvus | 01 is coming back up now | 23:48 |
| clarkb | ah | 23:48 |
| corvus | i shut down apache on 01 | 23:49 |
| corvus | it did take the finger server out correctly :) | 23:49 |
| corvus | and 01 is out now because apache is down. so the tcp check works. | 23:49 |
| clarkb | cool | 23:50 |
| fungi | we could redirect from the root grafana page to the dashboards index | 23:59 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!