| opendevreview | Ian Wienand proposed opendev/system-config master: centos-mirror: add dry run mode https://review.opendev.org/c/opendev/system-config/+/797364 | 00:08 |
|---|---|---|
| opendevreview | Ian Wienand proposed opendev/system-config master: centos-mirror: exclude ppc64le https://review.opendev.org/c/opendev/system-config/+/797365 | 00:08 |
| *** ykarel|away is now known as ykarel | 04:45 | |
| *** ricolin_ is now known as ricolin | 05:05 | |
| *** marios is now known as marios|ruck | 05:15 | |
| *** ysandeep|out is now known as ysandeep | 06:11 | |
| ianw | so it turns out importing the db from the old trove instance has some sort of primary key clash @ https://bugs.chromium.org/p/gerrit/issues/detail?id=14690 | 06:33 |
| ianw | see if that has an easy answer, if not, treating it as ephemeral data is an option i guess | 06:34 |
| ianw | but, in general the deployment of the local db, startup of gerrit, etc. is all good on review02. so we're making progress and getting a little closer to cut over | 06:40 |
| ianw | i hope to have a full checklist for review @ https://etherpad.opendev.org/p/gerrit-upgrade-2021 in short order | 06:40 |
| *** gthiemon1e is now known as gthiemonge | 06:46 | |
| opendevreview | Merged openstack/project-config master: Enable publish-openstack-python-tarball job https://review.opendev.org/c/openstack/project-config/+/797049 | 06:51 |
| *** rpittau|afk is now known as rpittau | 07:06 | |
| *** jpena|off is now known as jpena | 07:37 | |
| opendevreview | Guillaume Chauvel proposed zuul/zuul-jobs master: Update run-buildset-registry for readability https://review.opendev.org/c/zuul/zuul-jobs/+/762447 | 07:45 |
| *** ykarel is now known as ykarel|lunch | 07:50 | |
| opendevreview | chandan kumar proposed openstack/project-config master: Add publish-to-pypi job template for tripleo-operator-ansible https://review.opendev.org/c/openstack/project-config/+/797415 | 08:48 |
| *** ykarel|lunch is now known as ykarel | 09:08 | |
| *** dviroel|out is now known as dviroel | 11:20 | |
| *** jpena is now known as jpena|lunch | 11:36 | |
| *** ykarel_ is now known as ykarel | 12:02 | |
| *** prometheanfire is now known as Guest184 | 12:25 | |
| *** osmanlicilegi is now known as Guest187 | 12:25 | |
| *** ChanServ changes topic to "OpenDev is a space for collaborative Open Source software development | https://opendev.org/ | channel logs http://eavesdrop.openstack.org/irclogs/%23opendev/" | 12:29 | |
| *** jpena|lunch is now known as jpena | 12:31 | |
| *** marios|ruck is now known as marios|ruck|call | 13:10 | |
| *** marios|ruck|call is now known as marios|ruck | 13:25 | |
| TheJulia | Out of curiosity, I've been seeing an increase in the number of random post_failures on ironic jobs. Are these tracked/trended at the opendev level? Is there anyway we can make post-failure less faily? | 13:26 |
| *** marios|ruck is now known as marios|ruck|call | 13:34 | |
| opendevreview | Tristan Cacqueray proposed opendev/gear master: Prevent AttributeError: 'Server' object has no attribute 'statsd' https://review.opendev.org/c/opendev/gear/+/797451 | 13:42 |
| fungi | TheJulia: if they're truly random, then probably not. if there are some patterns you can discern, that might give us a clue as to what can be made more robust. we do generally track counts of each result type for each pipeline and project in our grafana via statsd reporting from the scheduler, but that's just raw counts/times. the builds api/webui on https://zuul.opendev.org can be queried for | 13:51 |
| fungi | more detail around what jobs are getting what results, but if you want to look at the logs of any you find there we only keep a month of those | 13:51 |
| TheJulia | fungi: primarily in ironic we're seeing stuff like https://zuul.opendev.org/t/openstack/build/d3159791bcca4ac89aceddbec6c43611 where we're wondering if the path detection script is just failing oddly somehow https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/fetch-subunit-output/files/find-testr.sh | 13:53 |
| TheJulia | rpittau: ^^ | 13:54 |
| rpittau | ack | 13:54 |
| rpittau | just checked a succesful job and the command has only one line | 13:54 |
| rpittau | https://zuul.opendev.org/t/openstack/build/506ca94ba84a4ef38c467577e9c8b455/console | 13:54 |
| rpittau | the failing ones has 2 | 13:54 |
| rpittau | https://zuul.opendev.org/t/openstack/build/d3159791bcca4ac89aceddbec6c43611/console | 13:54 |
| rpittau | since the script is taking stdout_lines[0] it ends up with an empty entry in the second case | 13:55 |
| rpittau | here I mean -> https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/fetch-subunit-output/tasks/main.yaml#L68 | 13:55 |
| opendevreview | Rodolfo Alonso proposed openstack/project-config master: Make explicit the network backend used in the CI jobs https://review.opendev.org/c/openstack/project-config/+/797454 | 13:59 |
| *** marios|ruck|call is now known as marios|ruck | 14:00 | |
| fungi | TheJulia: rpittau: comparing the succeeding and failing examples, i see the successful one runs `/opt/stack/tempest/.tox/venv/bin/stestr last --subunit >>/tmp/subunit...` but the failing one runs just `last --subunit >>/tmp/subunit...` | 14:01 |
| fungi | maybe it's testr detection breaking | 14:01 |
| rpittau | fungi: yeah, because the stdout output is on 2 lines and the first one is empty | 14:02 |
| fungi | rpittau: aha, yep, just spotted that as well, now i see what you were saying about two lines earlier | 14:03 |
| rpittau | fungi: yeah, sorry, maybe a bit confusing :) | 14:03 |
| fungi | so it's this script: https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/fetch-subunit-output/files/find-testr.sh | 14:05 |
| rpittau | yep | 14:05 |
| fungi | the last loop in that script seems to guard against returning empty lines | 14:06 |
| fungi | suggesting it's some other command run prior to that emitting a blank line, maybe | 14:07 |
| rpittau | fungi: I' mstarting to think the issue is not on the bash script but in the ansible module, what if we add a strip() after stdout? | 14:20 |
| rpittau | oh that's stdout_lines actually | 14:22 |
| fungi | yes, the stdout from running that script | 14:25 |
| fungi | the script seems to try to avoid emitting blank lines, so it would be good to figure out where the blank is coming from... | 14:26 |
| *** ykarel is now known as ykarel|away | 14:35 | |
| *** marios|ruck is now known as marios|ruck|call | 14:44 | |
| rpittau | fungi: maybe use a filter to create a new list from stdout_lines and exclude empty entries ? | 14:44 |
| *** marios|ruck|call is now known as marios|ruck | 14:44 | |
| fungi | rpittau: yeah, i mean ways to work around it are numerous, the bigger question is where the blank is coming from. i'm not personally a fan of code which accumulates a pile of workarounds for unidentified misbehaviors | 14:50 |
| fungi | it's clear that the `type` command can emit blank lines when queried about a nonexistent name, but the script has a filter of sorts already to deal with that | 14:51 |
| fungi | maybe that test -n isn't working as intended? maybe the entry itself has an embedded newline for some reason? and under what circumstances does it arise? | 14:52 |
| opendevreview | Merged zuul/zuul-jobs master: Update run-buildset-registry for readability https://review.opendev.org/c/zuul/zuul-jobs/+/762447 | 14:53 |
| fungi | or could something else run as a part of that script be generating blank lines before it ever reaches that loop? | 14:53 |
| *** ysandeep is now known as ysandeep|out | 14:57 | |
| *** Guest184 is now known as prometheanfire | 15:10 | |
| *** gthiemon1e is now known as gthiemonge | 15:38 | |
| rpittau | fungi: I agree on the "pile of workarounds", just this little issue is getting annoying as it breaks hours of successful CI runs. | 15:38 |
| rpittau | Anyway, as we saw it looks like what's breaking here is that stdout of the command contains a newline, I think it's ansible prepending that for some reason and "unfortunately" not all the times, kind of hard to verify, I don't see debug ansible info in the jobs | 15:38 |
| fungi | rpittau: we do collect the ansible json stream as well and publish it next to the text output log, maybe it will have more clues | 15:40 |
| rpittau | oh didn't see that! | 15:40 |
| rpittau | yeah, there you go | 15:41 |
| rpittau | "stdout": "\r\n/opt/stack/tempest/.tox/venv/bin/stestr\r\n" | 15:41 |
| rpittau | I blame ansible | 15:41 |
| rpittau | :) | 15:41 |
| fungi | yeesh | 15:42 |
| fungi | well, that could just be a representation of what's coming out of the script | 15:42 |
| fungi | we could add some debugging from the script to stderr i suppose, which shouldn't conflict with parsing the stdout, but since this seems to occur infrequently it would be hard to leverage without merging it to the role | 15:43 |
| *** marios|ruck is now known as marios|out | 16:00 | |
| *** rpittau is now known as rpittau|afk | 17:11 | |
| gmann | clarkb: fungi Need some help on ELK service help things, mainly two question | 17:24 |
| fungi | sure | 17:24 |
| gmann | clarkb: fungi 1. ELK +e-r are used in opendev as well as in other open infra project also right not just openstack | 17:24 |
| fungi | (clarkb is out for the next few days, but i'm happy to answer) | 17:24 |
| fungi | gmann: i don't know about "used" but we index logs from any builds which match the logstash filters we've got loaded | 17:25 |
| fungi | so we're potentially indexing logs for more than just openstack projects | 17:25 |
| gmann | ok which is from all the projects opendev support | 17:25 |
| gmann | ok | 17:25 |
| fungi | though openstack's ci data accounts for the vast majority of what we're indexing | 17:25 |
| gmann | yeah | 17:26 |
| gmann | fungi: 2nd question is about skillset requirement for maintainers, we listed 'developers and system administrators ' in this https://governance.openstack.org/tc/reference/upstream-investment-opportunities/2021/community-infrastructure-elk-maintainer.html | 17:27 |
| fungi | well, not necessarily "all projects." not all open infrastructure foundation projects use zuul, and of the ones which do some of them don't use zuul exclusively and also have other ci systems. but also not all the projects hosted in opendev and using opendev's zuul deployment are part of an open infrastructure project, we have plenty of wholly independent projects we're hosting as well | 17:27 |
| gmann | I see | 17:27 |
| gmann | I think that skillset requirement is correct, we still need developer skill not only system admin/ops only as we need to maintain code for e-r especially | 17:28 |
| fungi | the skillset is going to need to be people capable of designing and running a replacement system from the ground up, i think. the current deployment is going to be more work to update to a maintainable state than it would be to recreate from scratch with new technologies | 17:28 |
| fungi | we have basic design documents explaining how the current implementation was intended to work, but a replacement wouldn't necessarily need to follow the same architecture (and should probably differ on a number of key points really) | 17:29 |
| gmann | ok, for ELK we can say developer skill not required as such but for e-r we need developer on code-maintenance part | 17:31 |
| fungi | i can't really speak to "developers" vs "systems administrators" since i don't think there's a clear distinction. at least when i first entered the workforce you needed to be able to write software to be any good at administering systems, and if you couldn't admin things then writing software was especially challenging | 17:32 |
| *** jpena is now known as jpena|off | 17:32 | |
| fungi | a good systems administrator is a developer and a good developer is a systems administrator. they're one and the same | 17:32 |
| fungi | in this particular case it's systems software, so if you don't understand systems then the software isn't going to make sense | 17:33 |
| gmann | yeah, that's more right interpretation. | 17:33 |
| fungi | also e-r and elk are implementation details, any of that can be changed in the redesigned system | 17:34 |
| gmann | true | 17:34 |
| fungi | and yeah, the systems-administrators-are-developers idea has been rebranded "devops" in recent years, but isn't even remotely a new idea, it's how things have worked for many decades | 17:34 |
| gmann | yeah, devops is right term I can add. | 17:35 |
| fungi | the tools have changed, but the basic skills haven't | 17:35 |
| gmann | thanks fungi for clarifying the requirement. I think this is enough info for now. | 17:35 |
| *** iurygregory_ is now known as iurygregory | 18:39 | |
| opendevreview | Jeremy Stanley proposed opendev/infra-manual master: Switch docs from referencing Freenode to OFTC https://review.opendev.org/c/opendev/infra-manual/+/797531 | 19:37 |
| opendevreview | Ildiko Vancsa proposed opendev/infra-manual master: Update the documentation with the recent IRC changes https://review.opendev.org/c/opendev/infra-manual/+/797532 | 19:44 |
| fungi | i'm stepping out to meet some folks for dinner, but should be back in a couple of hours tops | 19:45 |
| *** dviroel is now known as dviroel|out | 20:59 | |
| opendevreview | James E. Blair proposed openstack/project-config master: Add zuul/gating.dev repo https://review.opendev.org/c/openstack/project-config/+/797543 | 22:01 |
| opendevreview | James E. Blair proposed openstack/project-config master: Adding gating.dev zone repo https://review.opendev.org/c/openstack/project-config/+/797544 | 22:06 |
| opendevreview | James E. Blair proposed openstack/project-config master: Adding gating.dev zone repo https://review.opendev.org/c/openstack/project-config/+/797544 | 22:08 |
| opendevreview | James E. Blair proposed opendev/system-config master: Add gating.dev zone to ADNS https://review.opendev.org/c/opendev/system-config/+/797545 | 22:11 |
| ianw | fungi: if you have a sec to loko over https://review.opendev.org/c/opendev/system-config/+/797365 and the files it deletes in the referenced dry-run log, that would be great | 22:13 |
| ianw | that's the ppc removal | 22:13 |
| fungi | aha, thanks! | 22:15 |
| opendevreview | James E. Blair proposed openstack/project-config master: Adding gating.dev zone repo https://review.opendev.org/c/openstack/project-config/+/797544 | 22:18 |
| ianw | this patch review table created by gerrit is very weird | 22:23 |
| ianw | https://gerrit.googlesource.com/gerrit/+/refs/heads/master/java/com/google/gerrit/server/schema/JdbcAccountPatchReviewStore.java#193 | 22:23 |
| ianw | i can not see how it would not be setting the primary key constraints correctly | 22:24 |
| ianw | although we should have executed https://gerrit.googlesource.com/gerrit/+/refs/heads/master/java/com/google/gerrit/server/schema/MariaDBAccountPatchReviewStore.java#57 | 22:25 |
| fungi | could it be the dump from our old db which is the problem? | 22:25 |
| ianw | well this is using the inbuilt gerrit tool to migrate that db | 22:25 |
| fungi | maybe it had a different primary key in the past and migrations didn't undo it? | 22:25 |
| fungi | oh | 22:25 |
| fungi | i will admit to still finding relational databases a bit of witchcraft | 22:26 |
| ianw | https://bugs.chromium.org/p/gerrit/issues/detail?id=14690#c1 | 22:26 |
| ianw | i can't see what is making our table | 22:26 |
| corvus | ianw: i like "show create table foo" for things like this too | 22:27 |
| ianw | we must have created the "generic table" as it's file_name is a varchar(4096) -> https://gerrit.googlesource.com/gerrit/+/refs/heads/master/java/com/google/gerrit/server/schema/JdbcAccountPatchReviewStore.java#198 | 22:28 |
| ianw | not the mariadb version, which is varchar(255) -> https://gerrit.googlesource.com/gerrit/+/refs/heads/master/java/com/google/gerrit/server/schema/MariaDBAccountPatchReviewStore.java#62 | 22:29 |
| corvus | ianw: there's a "file_id" column in mysql which is apparently part of the primary key? | 22:29 |
| corvus | and that is omitted from the mariadb schema | 22:29 |
| ianw | yeah, and not in the above queries that should be creating the table | 22:30 |
| corvus | so, to be clear, it seems like a straightforward case where the old schema does not match the new one. i think in this case a migration script should ignore duplicate key errors on insert and continue | 22:32 |
| ianw | yeah, i wonder when it changed. they must have moved from having a separate auto inc for the primary key to making the primary key that combo of fields | 22:33 |
| corvus | (and we know they've played fast and loose with this table in upgrades before; it's not surprising (but disappointing) to me that our schema doesn't match what's in the source code) | 22:34 |
| ianw | i'm guessing this has to do with our old database not using innodb tables | 22:40 |
| ianw | Creation of the account_patch_reviews table failed on mysql with the | 22:40 |
| ianw | file_name column being set to varchar(4096), see [1]. | 22:40 |
| ianw | mysql has a limit of 767 bytes max key length for InnoDb tables. | 22:40 |
| ianw | since our trove db has a file_name column of 4096 it must be different | 22:41 |
| fungi | so was it myisam on the trove instance then? | 22:41 |
| ianw | it's still unclear to me why it chose a different primary key | 22:42 |
| ianw | (on the old trove side) | 22:42 |
| fungi | i still strongly suspect it's a (perhaps lost to the annals of time) gap in migrations for gerrit | 22:43 |
| corvus | i mean, it didn't choose it... it's very likely what fungi said | 22:43 |
| fungi | we've been carrying over the same databases since, i dunno, gerrit 2.1? | 22:43 |
| fungi | they didn't take upgrades nearly as seriously once upon a time | 22:43 |
| corvus | they really don't care about this table; they also say if anything goes wrong with it, just drop it and start over | 22:44 |
| fungi | part of our upgrade plan was also "if something goes wrong with that db, we'll just wipe it" | 22:44 |
| fungi | it's really just checkmarks for filenames in reviews to signal whether you've "seen" them | 22:45 |
| corvus | my personal opinion: it's worth spending about 30 minutes total trying to port the data over; otherwise i'd drop it :) | 22:45 |
| fungi | i can't remember the last time i've actually relied on that feature in gerrit | 22:45 |
| fungi | if it had a switch to turn the feature off which allowed you to not need a separate database to track those checkmarks, i'd vote for disabling it | 22:46 |
| fungi | ianw: different topic, but before i go pushing a fix, do you agree this is incorrect? https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/limnoria/templates/meetingLocalConfig.py.j2#L4 | 22:53 |
| fungi | should be https://meetings.opendev.org/meetings/ yeah? | 22:54 |
| fungi | just making sure i swing my hammer in the right spot | 22:54 |
| diablo_rojo | So I was digging around looking for where in system config the general opendev ML is setup? I see where the openstack and kata ones are, but not the non project specific ones. | 22:56 |
| diablo_rojo | Any pointers? | 22:56 |
| fungi | diablo_rojo: in the host_vars i pointed you to on the ml thread? | 22:57 |
| fungi | oh i see what you're asking. the "openstack" file i linked is what you're looking for | 22:58 |
| fungi | we have 5 different mailing list "sites" on the lists.openstack.org server | 22:58 |
| fungi | it's not an openstack-specific server, it just dates from a time when all our servers were named in the openstack.org domain | 22:58 |
| diablo_rojo | oh! I missed the reply sorry! | 22:59 |
| fungi | the lists.katacontainers.io server is the odd one out, it only serves that one site, at some point we'll fold it into the multi-site server | 23:00 |
| diablo_rojo | Ah fair enough. I will look a little harder. | 23:00 |
| fungi | well, hopefully my reply explains exactly where to add what you're wanting | 23:01 |
| diablo_rojo | Got it. Sorry! | 23:01 |
| fungi | if it isn't clear, happy to add further details | 23:01 |
| diablo_rojo | I dont actually see the reply on the thread in my inbox for some reason, but I will go look at the archives. | 23:02 |
| fungi | feel free to follow up to the ml thread you started or ask in here, either's fine by me | 23:02 |
| fungi | diablo_rojo: http://lists.opendev.org/pipermail/service-discuss/2021-June/000258.html | 23:03 |
| fungi | maybe your mail provider has something against me | 23:03 |
| diablo_rojo | Perhaps.. though its weird I got Mohammed, but not yours. | 23:05 |
| diablo_rojo | All good now. I found the opendev section. I really don't think a new mailman site is required. | 23:05 |
| fungi | this is why i run my own mailserver. i don't like the idea of some company deciding what messages i do and don't get to see | 23:05 |
| fungi | especially a company with ulterior motives which views me as a product rather than a customer | 23:06 |
| corvus | incidentally, it's pretty easy to set up zuul jobs to publish openedx courses | 23:07 |
| diablo_rojo | fungi, its not normally an issue, but yeah.. a bit arbitrary can be frustrating. | 23:07 |
| diablo_rojo | corvus, oh that is good to know. | 23:07 |
| fungi | it might even be a cool demo to set up | 23:09 |
| diablo_rojo | Yes it would! | 23:10 |
| fungi | having more folks in academia get exposure to the wonders of zuul would be awesome | 23:10 |
| fungi | upstream institute as an open edx course could also be interesting | 23:12 |
| opendevreview | Kendall Nelson proposed opendev/system-config master: Add mailing list for FLOSS MOOC https://review.opendev.org/c/opendev/system-config/+/797549 | 23:16 |
| ianw | fungi: sorry, back from drop-off. umm, yes although it should redirect | 23:16 |
| ianw | select t.change_id, t.patch_set_id, t.account_id, t.file_name from account_patch_reviews as t group by t.change_id, t.patch_set_id, t.account_id, t.file_name having count(*) > 1; | 23:30 |
| ianw | ERROR 126 (HY000): Incorrect key file for table '/var/tmp/#sql_2c2_0.MYI'; try to repair it | 23:30 |
| opendevreview | Merged openstack/project-config master: Add zuul/gating.dev repo https://review.opendev.org/c/openstack/project-config/+/797543 | 23:31 |
| opendevreview | Merged openstack/project-config master: Adding gating.dev zone repo https://review.opendev.org/c/openstack/project-config/+/797544 | 23:31 |
| fungi | ianw: well, it doesn't redirect, but i'll add one. we have a redirect for eavesdrop.openSTACK.org but not eavesdrop.openDEV.org, at least not yet anyway | 23:56 |
| ianw | ohhhh | 23:56 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Correct the meeting base URL for our meetbot https://review.opendev.org/c/opendev/system-config/+/797550 | 23:56 |
| ianw | you're right. now i think about it, i think we don't need eavesdrop.opendev.org at all | 23:57 |
| fungi | however now we have a number of meeting logs with urls for the latter in them | 23:57 |
| ianw | yeah, that's unfortunate and i guess we need to redirect them. | 23:58 |
| ianw | or we could just edit the meeting logs | 23:58 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!