| *** bolg has quit IRC | 00:04 | |
| ianw | my kids school (remote) today is "well-being day" | 00:07 |
|---|---|---|
| ianw | which involves a lot of arts, craft, baking soda and lemon juice and other assorted mess | 00:08 |
| ianw | ... so not sure my well-being will survive :) | 00:08 |
| clarkb | "fatal: Unable to create '/etc/ansible/roles/puppet/.git/index.lock': File exists." <- current issue getting remote puppet else to run | 00:10 |
| clarkb | I'm sure its likely fallout from the killing of old ansible processes | 00:10 |
| clarkb | but I've run out of daytime to debug that further. I can pick it up tomorrow if no one else does | 00:10 |
| clarkb | ianw: ha, we made oobleck the other day as one of my kids watched some science show on netflix that has an episode with it and got obsessed with non newtonian fluids | 00:11 |
| clarkb | its quite the mess | 00:11 |
| ianw | haha, i'm still cleaning that up from the backyard the other week! | 00:11 |
| clarkb | fwiw there are no git processes currently runningon bridge so I think manually removing that index.lock is our best bet (and if not we can always just reclone it) | 00:12 |
| clarkb | but ya I need to figure out dinner now. | 00:12 |
| openstackgerrit | Merged opendev/system-config master: Mirror Puppetlabs puppet for Ubuntu Focal https://review.opendev.org/741299 | 00:12 |
| ianw | i can do that, what am i watching? the puppet-else run right? | 00:12 |
| clarkb | ya | 00:13 |
| clarkb | ianw: look in /var/log/ansible/remote_puppet_else.yaml.log and double check the error message | 00:13 |
| clarkb | there is a bit more to it than just what I quoted that talks about how to fix | 00:13 |
| ianw | hrm /etc/ansible/roles/puppet/.git/index.lock doesn't appear to be there now | 00:14 |
| clarkb | huh maube we just watch and see then? | 00:17 |
| clarkb | maybe the next run will be happy | 00:17 |
| clarkb | maybe double check the timestamps on the logs maybe there is another newer problem? | 00:18 |
| ianw | hang on, i think that's on the remote host? it's not clear which one ... openstackid01.openstack.org maybe | 00:22 |
| clarkb | oh | 00:25 |
| *** mlavalle has quit IRC | 00:27 | |
| ianw | failed: [openstackid01.openstack.org -> localhost] (item=puppet) => | 00:29 |
| ianw | "cmd": "/usr/bin/git reset --hard origin/master --", | 00:29 |
| ianw | "rc": 128 | 00:29 |
| *** mlavalle has joined #opendev | 00:31 | |
| ianw | -rw-r--r-- 1 root root 306K Sep 2 00:27 remote_puppet_else.yaml.log | 00:32 |
| ianw | it just ran (00:33 now) and didn't do that, but one of the elasticsearch still failed | 00:33 |
| ianw | fatal: Unable to create '/etc/ansible/roles/puppet/.git/index.lock': File exists. | 00:34 |
| ianw | boo, that's what it failed on | 00:34 |
| ianw | https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/install-ansible-roles/tasks/main.yaml | 00:38 |
| ianw | https://opendev.org/opendev/system-config/commit/4cb06ec6f2b782bf004eeae58925c86de55e565b | 00:38 |
| ianw | We're running this per-host delegated to localhost. We only want | 00:38 |
| ianw | to run it once, otherwise we have parallel competing git processes. | 00:38 |
| clarkb | ah | 00:38 |
| ianw | it seems like that isn't quite working | 00:39 |
| ianw | hrm, it only appears once in the log | 00:42 |
| ianw | remote_puppet_else.yaml.log.2020-05-13T18:54:31Z:Another git process seems to be running in this repository, e.g | 00:50 |
| ianw | i guess this didn't just start ... | 00:51 |
| *** xiaolin has joined #opendev | 01:09 | |
| fungi | stale lock? | 01:11 |
| fungi | does /etc/ansible/roles/puppet/.git/index.lock have an old date? | 01:11 |
| ianw | fungi: yeah, the problem is the lock isn't there. it must be a runtime thing | 01:13 |
| fungi | oh fun | 01:21 |
| ianw | i am realising i have no idea how the bridge run ansible's find their roles | 01:38 |
| *** xiaolin has quit IRC | 01:50 | |
| ianw | [WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. | 01:53 |
| *** DSpider has joined #opendev | 03:45 | |
| *** fressi has joined #opendev | 04:23 | |
| *** ysandeep|away is now known as ysandeep | 04:59 | |
| *** bhagyashris|away is now known as bhagyashris | 05:27 | |
| *** kevinz has joined #opendev | 05:34 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: puppet: don't run module install steps multiple times https://review.opendev.org/749431 | 05:35 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: puppet: don't run module install steps multiple times https://review.opendev.org/749431 | 06:04 |
| *** diablo_rojo has quit IRC | 06:47 | |
| *** hashar has joined #opendev | 07:07 | |
| openstackgerrit | Merged openstack/project-config master: Create zuul/zuul-client https://review.opendev.org/747127 | 07:18 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: puppet: don't run module install steps multiple times https://review.opendev.org/749431 | 07:28 |
| *** sshnaidm|afk is now known as sshnaidm | 07:32 | |
| *** tosky has joined #opendev | 07:37 | |
| *** ysandeep is now known as ysandeep|lunch | 07:48 | |
| *** moppy has quit IRC | 08:01 | |
| *** moppy has joined #opendev | 08:01 | |
| ttx | re: ask.o.o the change did not seem to catch on -- not sure if it's due to the Zuul deploy jobs failing (since they really never worked), or something else (like it needs a restart to catch on config changes) | 08:08 |
| ttx | I guess that's what y'all were discussing earlier | 08:13 |
| *** calcmandan has quit IRC | 08:14 | |
| * frickler goes to check | 08:15 | |
| *** calcmandan has joined #opendev | 08:16 | |
| *** dtantsur|afk is now known as dtantsur | 08:23 | |
| frickler | ttx: infra-root: seems /opt/system-config/production hasn't been updated on ask.o.o, maybe someone can take a deeper look later | 08:28 |
| *** ysandeep|lunch is now known as ysandeep | 08:33 | |
| * frickler now tries editing askbot config directly, on the first attempt the change doesn't seem to have any effect | 08:41 | |
| frickler | ttx: humm, maybe I had things cached, now it seems to work. please double check if that looks o.k. to you now | 08:44 |
| frickler | maybe we also want to update the help page if we can? https://ask.openstack.org/en/help/ | 08:45 |
| ttx | frickler: It looks ok (for some definition of ok that includes red-over-yellow text) | 08:47 |
| ttx | Re: help, I think that since the message is splashed there too, it's fine | 08:48 |
| ttx | we just can't rewrite all the site so that it stops mentioning that interacting with it is possible | 08:49 |
| *** andrewbonney has joined #opendev | 08:51 | |
| *** ysandeep is now known as ysandeep|brb | 09:50 | |
| *** hashar has quit IRC | 09:53 | |
| *** ysandeep|brb is now known as ysandeep | 10:04 | |
| *** bolg_ has joined #opendev | 10:30 | |
| *** lpetrut has joined #opendev | 10:47 | |
| *** elod has quit IRC | 11:15 | |
| *** elod has joined #opendev | 11:16 | |
| *** ysandeep is now known as ysandeep|brb | 11:36 | |
| *** ysandeep|brb is now known as ysandeep | 11:44 | |
| *** ysandeep is now known as ysandeep|afk | 11:52 | |
| *** mattd01 has joined #opendev | 12:00 | |
| *** ysandeep|afk is now known as ysandeep | 12:11 | |
| chandankumar | Hello #OpenDev. | 12:12 |
| chandankumar | The tripleo ci team is working on jobs to remove dependency on dockerhub | 12:13 |
| chandankumar | in tripleo-ci, apart from openstack projects, we pull ceph images also from docker hub | 12:13 |
| chandankumar | what is the procedure for pushing those same ceph container images on openstack infra registry ? | 12:14 |
| chandankumar | ceph -> https://hub.docker.com/u/ceph | 12:14 |
| weshay|ruck | zbr, fyi ^ | 12:16 |
| chandankumar | and part from that, we also need to promotheorus related images | 12:16 |
| chandankumar | the list is here https://opendev.org/openstack/tripleo-common/src/branch/master/container-images/tripleo_containers.yaml#L214 | 12:16 |
| fungi | chandankumar: which openstack infra registry are you talking about? we don't maintain any persistent registry, we have a semi-persistent registry for interrelated job builds (so that one build can create an image and then another build can consume that particular build's image) | 12:23 |
| chandankumar | fungi: sorry, I thought we host registries, | 12:27 |
| chandankumar | fungi: when we are talking about semi-persistant registry you mean https://opendev.org/zuul/zuul-registry ? | 12:27 |
| fungi | chandankumar: if you mean something like an alternative to dockerhub or quay, no opendev doesn't have any registries of its own just proxies to those services | 12:28 |
| fungi | and yes, the zuul-registry project is the software which we use to implement the semi-persistent passthrough registry for sharing built images with other consuming builds | 12:28 |
| chandankumar | fungi: yes something like an alternative to dockerhub, | 12:28 |
| chandankumar | fungi: cool, it answers my question. thanks :-) | 12:29 |
| fungi | chandankumar: i hear microsoft is going to start providing a dockerhub-like image registry under its github brand, though i expect it will come with all the same drawbacks of actual dockerhub and quay | 12:30 |
| fungi | i suppose what would be a useful service from a ci perspective is a proxyish image registry which speaks dockerhub protocol to clients, but behind the scenes fetches and caches the requested images from major registries | 12:32 |
| fungi | something like apt-proxy does for debian package repositories | 12:32 |
| fungi | i'm not aware of the existence of anything like that though | 12:32 |
| chandankumar | fungi: sounds like a nice project idea :-) | 12:34 |
| sshnaidm | fungi, there is such thing like https://docs.docker.com/registry/recipes/mirror/ | 12:35 |
| sshnaidm | fungi, it acts exactly as squid or whatever cache | 12:36 |
| fungi | neat. maybe we should look into that as an alternative to our registry proxies | 12:36 |
| sshnaidm | first time it pulls from a real registry and saves image in the cache, then provides it from cache only | 12:36 |
| sshnaidm | fungi, it's also transparent for clients | 12:36 |
| fungi | is that the one that can't purge expired images without an outage? | 12:36 |
| sshnaidm | need just to adjust proxy addresses | 12:37 |
| sshnaidm | fungi, that I didn't check, sorry | 12:37 |
| sshnaidm | fungi, I was just removing files on disk | 12:37 |
| fungi | i know we looked at something docker provides and it wasn't capable of having its cache cleaned without downtime | 12:37 |
| sshnaidm | worth to check I think | 12:37 |
| fungi | clarkb would remember, once he's awake | 12:38 |
| fungi | #status log zm07 was rebooted at 01:54 and again at 04:08 by the cloud provider because of unspecified hypervisor host issues | 12:45 |
| openstackstatus | fungi: finished logging | 12:45 |
| zbr | fungi: clarkb: can we proceed with https://review.opendev.org/#/c/747631/ ? | 12:52 |
| fungi | thanks! | 12:55 |
| fungi | zbr: i've approved it, but need to disappear for a meeting and then some errands, so likely won't be around to check on it for the next few hours if it doesn't deploy successfully | 12:56 |
| zbr | no urgency, i was jus trying to cleanup my pile of unmerged reviews. | 12:57 |
| zbr | that one is one of the least important ones | 12:57 |
| frickler | fungi: well the first patch already didn't deploy automatically, so I doubt this one will, see backlog. I deployed the first one manually, so now we have a good opportunity to check the automation again | 12:58 |
| zbr | fungi: when back also take a look at https://review.opendev.org/#/c/748688/ -- adding a new mailing list for monitoring jobs. | 13:00 |
| zbr | the question was if you want to be the owner or not. | 13:00 |
| zbr | that question was for corvus | 13:00 |
| fungi | i can be owner/moderator if needed, i already act as a surrogate for some dozen lists anyway | 13:08 |
| *** fdegir has quit IRC | 13:11 | |
| openstackgerrit | Merged opendev/system-config master: Improved ask read-only message https://review.opendev.org/747631 | 13:26 |
| zbr | corvus: basically https://review.opendev.org/#/c/748688 is waiting for your decision (see clark question). | 13:32 |
| zbr | i do not care who is admin, i only want to see the list created to be able to subscribe to it | 13:32 |
| *** _mlavalle_1 has joined #opendev | 13:44 | |
| *** mlavalle has quit IRC | 13:46 | |
| *** sshnaidm is now known as sshnaidm|bbl | 13:58 | |
| *** ysandeep is now known as ysandeep|away | 13:59 | |
| *** _mlavalle_2 has joined #opendev | 14:14 | |
| *** _mlavalle_1 has quit IRC | 14:17 | |
| *** chandankumar is now known as raukadah | 14:21 | |
| *** fdegir has joined #opendev | 14:34 | |
| AJaeger | config-core, please review https://review.opendev.org/#/c/740560/ https://review.opendev.org/742194 https://review.opendev.org/748693 https://review.opendev.org/748259 https://review.opendev.org/748210 https://review.opendev.org/#/c/748498 https://review.opendev.org/#/c/679306/ https://review.opendev.org/716298 https://review.opendev.org/746827 https://review.opendev.org/#/c/743217 | 14:47 |
| *** _mlavalle3 has joined #opendev | 14:51 | |
| *** _mlavalle_2 has quit IRC | 14:53 | |
| *** _mlavalle3 has quit IRC | 14:53 | |
| *** qchris has quit IRC | 14:57 | |
| zbr | who can help digging an afp deploy failure, one with unfriendly output https://zuul.opendev.org/t/openstack/build/ce97c9438e154d7d8b4c70011d04fba6/console | 15:02 |
| zbr | for some weird reason "cmd" is not even included by default. | 15:02 |
| *** diablo_rojo has joined #opendev | 15:02 | |
| zbr | also the output is redirected, which makes in unaccessible. is that for security reasons? | 15:03 |
| zbr | lack of cmd display is fixed by https://review.opendev.org/#/c/740733/ --- if it ever gets approved. | 15:06 |
| clarkb | frickler: the remote puppet else job was failing due to a git lock file issue. I think ianw tracked it down to the free strategy in ansoble not respecting run once | 15:08 |
| clarkb | frickler: so we have many ansoble threads all trying to update the same git repo at once. This is why ask.o.o wasntupdating | 15:08 |
| clarkb | figuring that out is top of my list this morning. I need some breakfast first though | 15:09 |
| *** qchris has joined #opendev | 15:10 | |
| *** fressi has quit IRC | 15:11 | |
| clarkb | sshnaidm|bbl: fungi yes that us the same tool that need to be off to prune content. However, it seems that document says it does auto pruning when only acting as a pull through cache which I hadnot seen before. Its possible our combined use case for intermediary registry and proxying means we overlooked the pure cache behavior | 15:12 |
| clarkb | however if we run anonymous we'll still be limited to 100 blobs an hour or whatever it is | 15:12 |
| clarkb | the best bet is to see what docker says about CI in partocular which they have promised to do (maybe that is already published) | 15:13 |
| *** mlavalle has joined #opendev | 15:22 | |
| clarkb | fungi: zbr unless remote-puppet-else has been fixed I don't expect that to actually apply | 15:23 |
| clarkb | https://review.opendev.org/#/c/749431/ is the proposed fix for the puppetry from ianw | 15:23 |
| *** tosky has quit IRC | 15:31 | |
| clarkb | zbr: fwiw linter rules that catch problems like the free strategy + run once would be far more helpful than forcing people to noqa shell tasks that run git | 15:34 |
| zbr | clarkb: I can easily write a run_once rule, but I bet it will upset some users too | 15:35 |
| clarkb | zbr: that wouldn't surprise me its just that choosing to use a shell task when there are built ins is more of a matter of choice whereas ansible does not do what you've told it in the free + run once case | 15:36 |
| zbr | the only case where nobody would be upset is if linter would always return 0 | 15:36 |
| clarkb | imo its far more important for linters to highlight issues like the free + run once problem | 15:36 |
| clarkb | which is what the original C linter did. It tried to highlight portability issues aiui | 15:37 |
| zbr | it is not easy to detect this because strategy can be configured using ansible cfg, env vars,... | 15:37 |
| clarkb | "this does not do what you think it does" vs "your code doesn't meet the linter authors idea of good ansible" | 15:38 |
| zbr | i can easily detect places where run_once is used, and if I remember well newer version of ansible is creating runtime warnings when encoutering it anyway | 15:38 |
| zbr | i doubt that depends on strategy, as same play could be run with different strategies in different contex, so run_once is always a danger | 15:38 |
| *** ysandeep|away is now known as ysandeep | 15:45 | |
| zbr | most linters are opinionated, and some of them depend even on a single person taste, but that is not the case for ansbile-lint, which needs at least two people to agree | 15:46 |
| zbr | clarkb: thanks for the idea, i am now writing a new rule... | 15:47 |
| clarkb | ya I'm not necessarily saying that is wrong either. I'm just saying when it comes to the important of rules those that call out non portable actually wrong things are more important than the opinion portion | 15:47 |
| clarkb | *importance of rules | 15:47 |
| clarkb | shelling out to git is extremely useful bceause the git module only covers a fraction of git functionality. Using run once with free is going to do the wrong thing always | 15:48 |
| *** hashar has joined #opendev | 15:48 | |
| zbr | so you think the linter should warn regardless which strategy is used? | 15:50 |
| zbr | as i said, it may be impossible to guess the strategy | 15:50 |
| zbr | especially if someone adds run_once inside a role | 15:51 |
| clarkb | to start maybe just keep it to where you know the free strategy is used (as in our case as it was se tat the play level) | 15:51 |
| clarkb | then based on user feedback maybe expand it | 15:51 |
| zbr | imho, run_once should always be avoided, just because we can consider worst case possible: someone tries to run it using free | 15:52 |
| zbr | imho, adding "# noqa 123" after run_one to confirm that, let me alone, i know what i am doing, is an acceptable "cost" to pay. | 15:53 |
| *** lpetrut has quit IRC | 15:53 | |
| clarkb | ya, I would definitely ensure the rule explainswhy it shoul dbe avoided in that case | 15:54 |
| *** ysandeep is now known as ysandeep|away | 16:02 | |
| zbr | clarkb: would you mind creating a feature request at https://github.com/ansible/ansible-lint/issues ? i am already half way with my implementation | 16:12 |
| clarkb | if you're already writing it do we need a separate issue? | 16:12 |
| zbr | i am writing the rule implementation, not the ticket | 16:16 |
| clarkb | right, I'm asking if a ticket is necessary if someone is already doing the work | 16:16 |
| clarkb | (its not really a request if the work is already done) | 16:16 |
| zbr | just say you would find it useful to be warned about run_once dangers, i can polish it later | 16:16 |
| zbr | it will be a useful place to receive feedback from others | 16:17 |
| clarkb | ok | 16:17 |
| zbr | i am working at the tests now | 16:17 |
| zbr | zuul-jobs has only 7 occurrences, quite easy to fix compared with other issues like the mode= one | 16:18 |
| clarkb | zbr: https://github.com/ansible/ansible-lint/issues/1026 | 16:21 |
| zbr | thanks | 16:22 |
| zbr | https://github.com/ansible/ansible-lint/pull/1028 is ready for reviews. | 16:55 |
| *** dtantsur is now known as dtantsur|afk | 16:57 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Remove dependency on pkg_resources https://review.opendev.org/748737 | 17:06 |
| *** mattd01 has quit IRC | 17:16 | |
| clarkb | AJaeger: if you're still around have you seen any problems with the explicit file mode setting in opendev/base-jobs? | 17:16 |
| clarkb | AJaeger: mostly concerned about things we write to afs like docs | 17:16 |
| clarkb | (if not I thin kwe can proceed to landing the zuul-jobs update next) | 17:16 |
| clarkb | fungi: https://review.opendev.org/#/c/749431/ is a good one to review (though I'm about to head out on a bike ride so feel free to avoid approving if you'd like more eyeballs around when ansible runs puppet) | 17:27 |
| fungi | sure thing | 17:35 |
| yoctozepto | infra-root: ethercalc down again | 17:35 |
| fungi | ugh, checking | 17:35 |
| yoctozepto | fungi: thanks | 17:36 |
| fungi | Sep 2 17:25:29 ethercalc02 bash[29644]: Error: Can't set headers after they are sent. | 17:36 |
| fungi | same as last time | 17:36 |
| fungi | #status log restarted ethercalc service following crash at 17:25:29 | 17:37 |
| openstackstatus | fungi: finished logging | 17:37 |
| fungi | infra-root: this was the full traceback: http://paste.openstack.org/show/797387 | 17:39 |
| fungi | https://github.com/audreyt/ethercalc/issues/16 looks like maybe we need newer nodejs? | 17:40 |
| clarkb | fungi: our version of node shoul dbe newer than that bug | 17:41 |
| fungi | yeah, i concur, just checked | 17:41 |
| fungi | so maybe it's a different bug resulting in a similar error | 17:41 |
| clarkb | its also originating in the redis module | 17:41 |
| clarkb | maybe its a bug there that we need to update | 17:41 |
| fungi | the distro package for nodejs currently installed is 12.18.3~dfsg-4 | 17:41 |
| clarkb | *update the module to fix | 17:41 |
| fungi | we're not installing redis from distro packages | 17:42 |
| fungi | but if memory serves we pin a specific version in puppet | 17:42 |
| fungi | https://opendev.org/opendev/puppet-ethercalc/src/branch/master/manifests/redis.pp#L8 | 17:45 |
| fungi | $redis_version = '2.8.4', | 17:45 |
| clarkb | fungi: thats redis the server not redis the js client lib | 17:47 |
| fungi | oh, great point | 17:47 |
| clarkb | /opt/ethercalc/node_modules/redis/index.js should be installed as part of ethercalcs' npm installation | 17:47 |
| fungi | i suppose that'll be in like a yarn.lock in the ethercalc repo | 17:47 |
| clarkb | ya | 17:47 |
| clarkb | I need to pop out now but can help look more hwne I return | 17:48 |
| fungi | https://github.com/audreyt/ethercalc/blob/master/yarn.lock#L2088-L2090 | 17:48 |
| fungi | https://registry.yarnpkg.com/redis/-/redis-0.12.1.tgz#64df76ad0fc8acebaebd2a0645e8a48fac49185e | 17:49 |
| clarkb | has that been changed since the version we deployed? | 17:51 |
| fungi | hard to say, looks like the yarn.lock file was added two years ago, so more recently than what we had been running before the upgrade | 17:55 |
| fungi | i'll have to dissect their old build system from years ago | 17:55 |
| fungi | https://github.com/audreyt/ethercalc/blob/73afcc6714e8709ebd38c9ef1977aec814b3ffbe/package.json#L18 | 17:57 |
| fungi | "redis": "0.12.x", | 17:57 |
| fungi | so... not changed appreciably i guess? | 17:57 |
| fungi | yeah, that line was last touched 4 years ago | 17:58 |
| fungi | though wasn't actually updated | 17:58 |
| fungi | just reshuffled | 17:58 |
| fungi | here's when it changed: https://github.com/audreyt/ethercalc/commit/82199e293 | 17:59 |
| fungi | appeared in the 0.20151028.0 tag | 18:00 |
| fungi | so we were using the redis js lib version 0.12.x before we upgraded too | 18:00 |
| fungi | if it's a bug introduced in redis, it came between 0.12.0 and 0.12.1 maybe | 18:00 |
| *** mattd01 has joined #opendev | 18:11 | |
| *** hashar has quit IRC | 18:34 | |
| *** DSpider has quit IRC | 18:51 | |
| fungi | i don't know if it's a behavior change in osc or rackspace, but i can't seem to refer to cinder volumes by name any longer | 19:45 |
| fungi | i can only get uuids from volume list, do a volume show on each uuid and then grep out the name to figure out which one it is | 19:45 |
| fungi | it's... tedious, for sure | 19:46 |
| fungi | or could this be a side effect of using the v1 api? | 19:46 |
| *** DSpider has joined #opendev | 19:47 | |
| fungi | for u in `./launch-env/bin/openstack --os-cloud openstackci-rax --os-region-name DFW volume list|grep available|cut -d' ' -f2`;do echo -n $u;./launch-env/bin/openstack --os-cloud openstackci-rax --os-region-name DFW volume show $u|grep name|cut -d'|' -f3|sed 's/ *$//';done | 19:49 |
| fungi | that shouldn't be necessary, fwiw | 19:50 |
| fungi | i could probably tell osc to do --format on volume list to include the name, but why is it omitted to begin with? and why can't i use names in commands? if i don't use uuids, it tells me it can't find any volume matching that name | 19:52 |
| clarkb | maybe smcginnis knows | 19:54 |
| clarkb | if its v1 behavior | 19:54 |
| clarkb | fungi: node redis 1.0.0's changelog doesn't show any breaking changes from 0.12.1 | 19:58 |
| clarkb | maybe we shoul dtry updating? | 19:58 |
| fungi | well, ethercalc's master branch is pinned to 0.12.1 since years | 20:01 |
| clarkb | https://stackoverflow.com/questions/7042340/error-cant-set-headers-after-they-are-sent-to-the-client does seem to point to the redis client lib being the issue though | 20:03 |
| clarkb | basically it isn't do rest things properly | 20:03 |
| clarkb | fungi: another option may be to set that unit to just restart always and let it com eback after failing | 20:03 |
| fungi | yeah, i'm not opposed to upgrading, just wondering if we have to patch the ethercalc codebase to do it | 20:03 |
| fungi | or simply add an extra puppet exec notified from it to upgrade redis after every deployment | 20:04 |
| clarkb | for that I'm not sure | 20:04 |
| smcginnis | Volume names are optional, so I think the background was that UUID was really the preferred way to specify a volume. | 20:05 |
| smcginnis | Some things support looking up the UUID if it can tell a name was provided, but I'm not sure if that is done everywhere. | 20:06 |
| smcginnis | And IIRC, that's implemented in python-cinderclient, so probably not implemented in OSC. | 20:06 |
| smcginnis | Hmm, not list commands have a Name column, but I'm not getting anything on my deployment either. | 20:09 |
| fungi | yeah, i guess the gap here is that i request a volume and provide a name. that request is filled async so i'm not told what uuid it got. now i volume list and i onlt get uuids, not names, so i have to inspect every volume to find out what names they have so i can identify the volume i created | 20:10 |
| smcginnis | API does is supposed to be returning name - https://docs.openstack.org/api-ref/block-storage/v3/index.html?expanded=list-accessible-volumes-detail#id97 | 20:10 |
| fungi | well, again we're pinning to old api major version because rackspace has broken its catalog | 20:11 |
| fungi | so we're relying on the v1 cinder api, in theory | 20:11 |
| fungi | far from ideal, and no fingers are being pointyed | 20:12 |
| fungi | just wondering if i'm missing something obvious | 20:12 |
| *** andrewbonney has quit IRC | 20:12 | |
| fungi | normally i wouldn't even care, but they notified us that the region where most of our control plane resides is scheduled to undergo a lengthy outage for existing volumes, so i'm trying to go through and replace them with new ones | 20:13 |
| smcginnis | This might actually be a bug. Looking at the raw json returned from my v3 API call, it contains "{name: ''}" for all volumes. | 20:14 |
| fungi | a bit of scripting (laggily) works around the blind spot, just curious if i'm "doing it wrong" | 20:14 |
| smcginnis | Oh, for mine it's because my volumes don't have names. | 20:15 |
| fungi | heh | 20:15 |
| fungi | name really is "" | 20:15 |
| smcginnis | Yep, if I create a volume with a name, list does show it. So maybe it is a v1 issue/ | 20:15 |
| fungi | ours have names, and openstack volume show includes a name field. i think i also previously had some success passing a --format string to say to include it | 20:15 |
| fungi | in volume list | 20:15 |
| smcginnis | I'm also not sure if rax ever really switched to using cinder instead of their own special implementation. | 20:16 |
| fungi | ahh, nope, openstack volume list is actually providing a name field, it's just empty for every entry | 20:17 |
| fungi | but if i openstack volume show some_uuid it spits out a field called "name" with the expected value for that volume | 20:18 |
| smcginnis | fungi: Is it included if you do "openstack volume show UUID"? | 20:18 |
| smcginnis | Hmm | 20:18 |
| fungi | could it be case sensitive? | 20:18 |
| smcginnis | Maybe? | 20:18 |
| fungi | no dice. -c column_name ignores "name" but gives me an empty column for "Name" | 20:19 |
| smcginnis | If you openstack --debug volume list, it might be interesting to see if there's anything in the json. | 20:21 |
| fungi | also filtering with --name some_name doesn't actually filter the results, just gives me all of them | 20:21 |
| fungi | yep! it's returned in the json, so maybe this is a parsing problem in osc (or a formatting problem in rax) | 20:22 |
| smcginnis | Can you try a cinder list? | 20:22 |
| fungi | maybe, though i can't use clouds.yaml for that right? | 20:25 |
| fungi | cinderclient doesn't seem to use oscc so --os-cloud and --os-region-name are ignored | 20:25 |
| fungi | and it's not finding my creds | 20:25 |
| fungi | i'll find the docs on how i used to pass those values pre-osc | 20:26 |
| smcginnis | Yeah, you need to either use env vars or pass them in explicitly. | 20:26 |
| fungi | https://docs.openstack.org/python-cinderclient/latest/cli/index.html#description | 20:29 |
| fungi | nostalgic | 20:29 |
| smcginnis | :) | 20:30 |
| fungi | now if only the auth url weren't baked into osc. gotta look that up | 20:33 |
| clarkb | fungi: I'm firmly in front of the computer again now if you have time to review https://review.opendev.org/#/c/749431/ | 20:33 |
| clarkb | also clouds.yaml is the best | 20:33 |
| clarkb | I'd be a lot more sympathetic to teams not wanting to do osc if they added clouds.yaml support to the alternatives | 20:33 |
| fungi | smcginnis: yep, `cinder --os-volume-api-version=1 list` gives me actual volume names | 20:36 |
| fungi | (v2 and 3 give me endpoint not found errors) | 20:37 |
| *** DSpider has quit IRC | 20:37 | |
| fungi | supposedly we can use a block storage endpoint override in osc to get working v2 in rackspace, but i've not had luck with that | 20:38 |
| fungi | but yeah, if i were to place bets, it's that osc has regressed on its ability to parse names out of cinder v1 api responses | 20:39 |
| fungi | mordred: ^ next time you're around, that might be interesting to you | 20:40 |
| fungi | odds are it's really the sdk doing that bit, but i really don't know | 20:40 |
| fungi | infra-root: i have mounted four new cinder volumes on afs01.dfw, added them to the main vg, and am using pvmove to swap them out | 20:47 |
| fungi | i forgot to start the first pvmove under screen, but will do subsequent ones under a root screen session. the pvmove manpage explains how to resume an interrupted move if it comes to that | 20:48 |
| fungi | it's at 2% complete for the first of four moves already, so probably won't last into tomorrow | 20:48 |
| fungi | i'll keep tabs on it while poking at other work | 20:49 |
| clarkb | thanks! | 20:49 |
| fungi | we could run them in parallel, but the manpage warns against doing that if you have logical volumes spanning more than one pv, and also i'm not sure what our storage i/o bandwidth looks like on that instance anyway | 20:50 |
| fungi | also not doing it in parallel with afs02.dfw because if something happens... well, you know | 20:50 |
| fungi | we have until around the middle of next month to get through these anyway | 20:50 |
| * fungi goes back into his do-stuff-while-feeling-like-not-getting-enough-done hole | 20:52 | |
| fungi | the plan for volume replacements on the afs servers is to pvmove all of them, then detach all the old volumes | 20:54 |
| fungi | i don't 100% trust the device names returned by the nova api there | 20:54 |
| fungi | worried that i might detach the wrong device due to a mismatch somewhere in the xen guest api layer | 20:55 |
| clarkb | fungi: the sysfs or is it devfs can expose uuids of the volumes | 20:55 |
| clarkb | though now that I've written that that may be a kvm only feature | 20:55 |
| fungi | well, also "uuid" is vague there | 20:55 |
| clarkb | the uuid in cinder is mapped through in the kvm case | 20:56 |
| clarkb | its really useful | 20:56 |
| fungi | cinder's volume uuid is not the device's uuid afaik | 20:56 |
| clarkb | it is with kvm | 20:56 |
| fungi | oh, neat, kvm added that? | 20:56 |
| clarkb | pretty sure they force the uuid value such that it all works | 20:56 |
| clarkb | but I only ever get to interact iwth thta in devstack usually and its been a while | 20:56 |
| * fungi checks, while not getting hopes up | 20:57 | |
| fungi | yeah, the uuids returned by blkid don't match the uuids returned by nova/cinder | 20:59 |
| fungi | oh, wait, those are partition uuids | 21:00 |
| fungi | but yeah, none of the uuids listed under /dev/disk/by-uuid/ match cinder uuids either | 21:01 |
| fungi | and also those are only partitions too | 21:01 |
| fungi | not the raw devices | 21:01 |
| openstackgerrit | Ghanshyam Mann proposed openstack/project-config master: Final step for networking-l2gw and networking-l2gw-tempest-plugin retirement https://review.opendev.org/738040 | 21:02 |
| fungi | lsblk also doesn't show any uuids for raw block devices, only for their partitions | 21:03 |
| clarkb | huh its the /dev/disk/by-uuid/ paths I know works on kvm | 21:04 |
| clarkb | but I guess xen can't do that | 21:04 |
| fungi | fungi@afs01:~$ sudo blkid /dev/xvdj1 | 21:04 |
| fungi | /dev/xvdj1: UUID="NrzuNc-1Ksr-8Cnf-WCYO-w77X-FIih-WWZflF" TYPE="LVM2_member" PARTUUID="182dbd0a-01" | 21:04 |
| fungi | fungi@afs01:~$ sudo blkid /dev/xvdj | 21:04 |
| fungi | /dev/xvdj: PTUUID="182dbd0a" PTTYPE="dos" | 21:05 |
| fungi | no uuid for the disk, just for the partitions | 21:05 |
| *** prometheanfire has quit IRC | 21:59 | |
| *** paladox has quit IRC | 21:59 | |
| *** paladox has joined #opendev | 22:00 | |
| *** prometheanfire has joined #opendev | 22:04 | |
| *** mattd01 has quit IRC | 22:26 | |
| clarkb | fwiw I've checked if docker has published the promised post on CI use of docker hub and it isn't up yet as far as I can tell | 22:29 |
| clarkb | diablo_rojo: for https://review.opendev.org/#/c/742194/3/playbooks/maintain-github-mirror/github_manager.py does our PR closer only interact with openstack repos? (I sort of assume so just because of permissions) | 22:33 |
| clarkb | fungi: ^ you may know? | 22:33 |
| clarkb | this has all been updated semi recently /me tries to figur eit out | 22:33 |
| clarkb | well the job that runs that playbook is called maintain-github-openstack-mirror which is openstack specific enough for me I think | 22:34 |
| clarkb | mnaser: https://review.opendev.org/#/c/748210/1 may interest you if you aren't already aware of the amd epyc + centos8 nested virt issues. I think you can approv ethat one if you want too | 22:38 |
| *** sshnaidm|bbl is now known as sshnaidm|afk | 22:47 | |
| openstackgerrit | Merged openstack/project-config master: Updates Message on PR Close https://review.opendev.org/742194 | 22:49 |
| openstackgerrit | Merged openstack/project-config master: Add os_senlin to zuul projects https://review.opendev.org/748693 | 22:49 |
| openstackgerrit | Merged openstack/project-config master: kolla-cli: deprecation - retiring master branch https://review.opendev.org/748259 | 22:49 |
| ianw | sorry, running a bit late today, but here now | 22:52 |
| clarkb | ianw: I've been trying to get another review on the puppte change today without much success. I did leave some notes if you want to address them | 22:53 |
| clarkb | I do think that is the proper fix based on the ansible warnings htough | 22:53 |
| ianw | clarkb: yeah noticed, will respin with those notes | 22:53 |
| ianw | infra-root: tangentially related; https://review.opendev.org/#/q/topic:s-c-ansible-devel-testing will fix the ara reporting artifacts for system-config, along with the -devel job | 22:54 |
| clarkb | ianw: are my previous reviews on ^ still good? or do Ineed to do another pass? | 22:55 |
| ianw | umm, i think mostly good, https://review.opendev.org/#/c/747596/ got respun with the new project | 22:56 |
| ianw | https://review.opendev.org/#/c/747328/ is probably the one that i'd mostly want another eye on, that's dropping test-requirements.txt from system-config and moving to more targeted lists | 22:56 |
| clarkb | ianw: and I guess that when we install the release of ansible from pypi it will include all those collections ya? its just when we install from source we have to build up the extra bits that got split out? | 22:58 |
| ianw | yeah, pypi's "ansible" i believe includes all these. when you install from the -devel branch, you now install "ansible-base" | 22:58 |
| ianw | clarkb: i actually considered calling it puppet-setup-roles or puppet-setup-config or something more specific, do you want me to do that as well as update the readme? | 23:00 |
| clarkb | maybe puppet-setup-with-ansible ? | 23:01 |
| clarkb | really its the intersection of the two we're trying to make happy | 23:01 |
| clarkb | and less one or the other in isolation | 23:01 |
| ianw | another thing i noticed, we don't need the install_modules.sh/modules.env at the top-level any more? | 23:03 |
| ianw | i wasn't sure if the apply jobs or something might use it | 23:03 |
| clarkb | we install the modules on bridge then the ansible puppet role copies them onto the remote hosts | 23:06 |
| ianw | to be concrete, just because i get confused ... ansible-role-puppet isn't synced remotely, right? only the modules | 23:06 |
| clarkb | that way we keep things in sync for each pass of ansible running puppet apply | 23:06 |
| openstackgerrit | Merged openstack/project-config master: Add nested-virt-ubuntu-focal label https://review.opendev.org/743217 | 23:06 |
| clarkb | correct only the puppte modules are synced | 23:06 |
| *** mlavalle has quit IRC | 23:07 | |
| clarkb | the ansible role itself is only running from bridge (in the ansible way of thinking) | 23:07 |
| clarkb | it may do remote things but its execution context is from ansible-playbook on bridge | 23:07 |
| ianw | right | 23:07 |
| ianw | this is all probably good info to capture :) | 23:07 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: puppet: don't run module install steps multiple times https://review.opendev.org/749431 | 23:26 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: install-ansible: move install_modules.sh to puppet-setup-ansible https://review.opendev.org/749604 | 23:34 |
| ianw | if you have a in-repo symlink, does gitea serve up the linked file? | 23:37 |
| clarkb | no idea :) | 23:37 |
| clarkb | you could test it with opendev/sandbox really quickly? | 23:38 |
| ianw | the only place that seems to want to grab install_modules.sh and run it is https://opendev.org/x/infra-ansible/src/branch/master | 23:43 |
| clarkb | you mean in addition to system-config? | 23:43 |
| ianw | that's the thing, we have two copies. ansible deploys one and we have the top level | 23:44 |
| clarkb | oh I see | 23:46 |
| clarkb | I thought we were still using the top level but doing so would make ansible more difficult | 23:46 |
| fungi | clarkb: yes, the pr closer is an openstack job, credentials are only authorized for that org | 23:50 |
| ianw | i think they still get pulled in via apply tests | 23:50 |
| fungi | we figure other projects can add similar jobs with their creds instead if they want that, but this one is okay to have openstack-specific messaging | 23:51 |
| clarkb | fungi: ya I ended up approving it once I discovered it was used in a very openstack specific job | 23:51 |
| fungi | if you're looking for a test of gitea and symlinks, we use a ton in zuul's in-repo ansible fork to replace modules with links to a disabled one | 23:52 |
| fungi | also the first pvmove is nearly done. just a few more minutes and i can start the next | 23:52 |
| ianw | ps, i'm wrong, the modules.env in ansible are symlinks back to the top-level ones | 23:54 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: launch: move old scripts out of top-level https://review.opendev.org/749605 | 23:59 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!