Friday, 2020-08-21

*** ildikov has joined #opendev00:01
*** dviroel has joined #opendev00:01
*** aannuusshhkkaa has joined #opendev00:01
ianwi'll double check the dib before restoring builds00:05
*** aannuusshhkkaa has quit IRC00:13
*** aannuusshhkkaa has joined #opendev00:16
openstackgerritIan Wienand proposed opendev/system-config master: Fix ansible-devel job for Ansible 2.10 changes  https://review.opendev.org/71644900:23
*** sgw has quit IRC00:27
ianw$ sudo docker exec -it nodepool-builder-compose_nodepool-builder_1 disk-image-create --version00:30
ianw3.2.100:30
fungilooks good00:33
ianwdmsimard: re 716449 ^^ have you had any thoughts about the requirements.txt for the stable branch re ansible/ansible-base?  i guess it's not possible to say "ansible | ansible-base" :/00:35
ianw(ara requirements, to be clear :)00:35
fungiftr, you can also `cd /etc/nodepool-builder-compose;sudo docker-compose exec nodepool-builder disk-image-create --version`00:36
ianwok, my ssh-agent socket seems to have disappeared again.  something is up00:36
openstackgerritMerged openstack/project-config master: Revert "Pause all image builds"  https://review.opendev.org/74731200:40
openstackgerritIan Wienand proposed opendev/system-config master: Fix ansible-devel job for Ansible 2.10 changes  https://review.opendev.org/71644901:02
openstackgerritIan Wienand proposed opendev/system-config master: Fix ansible-devel job for Ansible 2.10 changes  https://review.opendev.org/71644901:31
openstackgerritIan Wienand proposed opendev/system-config master: Fix ansible-devel job for Ansible 2.10 changes  https://review.opendev.org/71644901:41
openstackgerritMerged zuul/zuul-jobs master: Revert "Ensure git is installed in prepare-workspace-git role"  https://review.opendev.org/74723801:44
dmsimardianw: ara master has jobs that test again devel and I haven't seen them failing recently02:05
dmsimardansible isn't a dependency for >=1.0 but it could be an issue for 0.x02:06
dmsimardIf it doesn't work it's a bug I guess :p02:08
ianwdmsimard: yeah it's obscure ... it's our system-config test running against ansible -devel branch.  ara wants to bring in "ansible" which breaks the install from git checkout which is "ansible-base"02:25
ianw"the install" being "the ansible install"02:25
openstackgerritIan Wienand proposed opendev/system-config master: Fix ansible-devel job for Ansible 2.10 changes  https://review.opendev.org/71644902:47
openstackgerritIan Wienand proposed zuul/zuul-jobs master: tox: include command output in log/error  https://review.opendev.org/74732502:57
openstackgerritIan Wienand proposed opendev/system-config master: Fix ansible-devel job for Ansible 2.10 changes  https://review.opendev.org/71644903:06
openstackgerritIan Wienand proposed opendev/system-config master: Fix ansible-devel job for Ansible 2.10 changes  https://review.opendev.org/71644903:59
openstackgerritIan Wienand proposed opendev/system-config master: [wip] drop test-requirements.txt  https://review.opendev.org/74732803:59
*** ykarel|away has joined #opendev04:26
*** ykarel|away is now known as ykarel04:26
openstackgerritIan Wienand proposed opendev/system-config master: [wip] drop test-requirements.txt  https://review.opendev.org/74732804:31
openstackgerritIan Wienand proposed opendev/system-config master: Fix ansible-devel job for Ansible 2.10 changes  https://review.opendev.org/71644904:31
openstackgerritIan Wienand proposed opendev/system-config master: [wip] collect tox logs  https://review.opendev.org/74732904:31
*** raukadah is now known as chandankumar04:41
*** ysandeep|away is now known as ysandeep04:56
*** sgw has joined #opendev05:15
openstackgerritIan Wienand proposed opendev/system-config master: tox: drop test-requirements.txt  https://review.opendev.org/74732806:31
openstackgerritIan Wienand proposed opendev/system-config master: Collect tox logs  https://review.opendev.org/74732906:31
openstackgerritIan Wienand proposed opendev/system-config master: Fix ansible-devel job for Ansible 2.10 changes  https://review.opendev.org/71644906:31
*** sgw has quit IRC06:38
*** weshay|ruck has quit IRC06:38
*** dulek has quit IRC06:38
*** danpawlik has quit IRC06:38
*** icey has quit IRC06:38
*** fbo|pto has quit IRC06:38
*** SotK has quit IRC06:38
*** noonedeadpunk has quit IRC06:38
*** sgw has joined #opendev06:43
*** dulek has joined #opendev06:43
*** weshay|ruck has joined #opendev06:43
*** fbo|pto has joined #opendev06:43
*** danpawlik has joined #opendev06:43
*** icey has joined #opendev06:43
*** SotK has joined #opendev06:43
*** noonedeadpunk has joined #opendev06:43
fricklerianw: are we green now with the dib/git/apt mess or is there still something I should look at?06:45
ianwfrickler: i think it's all good.  revert went in, new builds done, images are building/uploading06:49
*** andrewbonney has joined #opendev07:07
*** DSpider has joined #opendev07:14
*** hashar has joined #opendev07:17
yoctozeptoinfra-root: could I get https://review.opendev.org/747185 in? (pretty please) :-)07:23
yoctozeptoalso please let me know how to handle the launchpad part07:30
yoctozeptoianw: thanks :-)07:31
*** dtantsur|afk is now known as dtantsur07:31
*** priteau has joined #opendev07:40
openstackgerritMerged openstack/project-config master: Add openstack/etcd3gw  https://review.opendev.org/74718507:49
AJaegeryoctozepto: you need to configure launchpad yourself, there's nothing we do - everything is documented in Creator's Guide07:53
yoctozeptoAJaeger: ok, I'll look in that other guide then07:55
yoctozeptoAJaeger: actually no, I looked in it exactly https://docs.opendev.org/opendev/infra-manual/latest/creators.html07:56
yoctozeptoAJaeger: no mention of launchpad :-(07:56
*** moppy has quit IRC08:01
*** moppy has joined #opendev08:03
AJaegeryoctozepto: ah, we removed that at one point and have only storyboard in there.08:09
AJaegeryoctozepto: see change Ibec66e8dd952ea33a2f84578e41b33d5afab4451 for how to set it up08:11
yoctozeptoAJaeger: I know sb is the way to go, but, considering the general resistance, I guess it would be nice to restore parts of this :-)08:17
*** tosky has joined #opendev08:22
*** lpetrut has joined #opendev08:22
*** fressi has joined #opendev08:23
*** fressi has quit IRC08:27
yoctozeptoAJaeger: i think i did good: https://launchpad.net/python-etcd3gw08:29
*** hashar has quit IRC08:33
*** hashar has joined #opendev08:37
*** fressi has joined #opendev08:43
AJaegeryoctozepto: I hope so - my last setup for launchpad was ages ago. Let's see how it works ;)08:44
*** danpawlik has quit IRC08:45
*** danpawlik has joined #opendev08:47
*** hashar has quit IRC08:47
yoctozeptoAJaeger: :-) would have to ask bnemec to add me as a driver and a bug supervisor to test further now08:49
yoctozepto;d08:49
*** hashar has joined #opendev08:53
*** openstackgerrit has quit IRC08:54
*** openstackgerrit has joined #opendev08:59
openstackgerritMerged opendev/irc-meetings master: Update CloudKitty meeting information  https://review.opendev.org/74725608:59
yoctozeptodoes it usually take long to mirror those repos? I can't see http://opendev.org/openstack/etcd3gw yet09:06
AJaegeryoctozepto: check https://zuul.opendev.org/t/openstack/status - the deploy queue still shows the entry09:20
AJaegeryoctozepto: at this time it takes so long due to the many periodic jobs09:20
AJaegerincluding our daily infra jobs09:20
AJaegeryoctozepto: https://zuul.opendev.org/t/openstack/status/change/747185,309:21
yoctozeptoAJaeger: ah, missed that one; thanks, understood09:55
yoctozeptodid ethercalc die just now? I was editing https://ethercalc.openstack.org/kolla-infra-service-matrix but now it keeps reconnecting09:57
AJaegerinfra-root, I get a 503 on ethercalc ^09:59
fricklerAJaeger: yoctozepto: ethercalc service had crashed, restarted it10:18
frickler#status log restarted crashed ethercalc service, log info at http://paste.openstack.org/show/797024/10:19
openstackstatusfrickler: finished logging10:19
fricklerinfra-root: from the logs before the ethercalc crash and the pads mentioned there, it looks like our service is being used by some non opendev-related consumers, not sure if we'd want to monitor that or rather don't care10:22
yoctozeptofrickler: thanks10:29
*** jhesketh has quit IRC10:53
*** jhesketh has joined #opendev10:55
*** hashar has quit IRC11:47
*** ysandeep is now known as ysandeep|afk11:47
AJaegerthanks, frickler !11:51
*** ysandeep|afk is now known as ysandeep11:59
*** hashar has joined #opendev12:02
*** rosmaita has joined #opendev12:45
*** ysandeep is now known as ysandeep|mtg13:03
fungifrickler: that's the same exportCSV error we've seen twice recently13:10
fungilikely a known bug and i wouldn't be surprised if upgrading fixes it, just haven't had time to look13:11
*** ysandeep|mtg is now known as ysandeep13:46
fungii've started the ethercalc service again. it crashed once more on the same error at 13:12:1013:50
fungihttps://github.com/audreyt/ethercalc/pull/52113:51
fungiassuming it's the same error (certainly looks the same) then the fix merged to master and was included in version 0.20170704.0 over three years ago13:52
fungiunfortunately the javascript nodejs is running is compiled from livescript, so we can't just hot-patch the main.js we'll have to patch the source tree and then rebuild it13:57
*** mlavalle has joined #opendev14:01
fungipuppet-ethercalc says we're installing $ethercalc_version= '0.20161220.1',14:02
fungithough apparently the author doesn't use git tags much, 0.20170704.0 is the most recent tag in the github repo but there's a 0.20200505.0 package at https://www.npmjs.com/package/ethercalc14:06
*** ykarel is now known as ykarel|away14:11
*** lpetrut has quit IRC14:15
*** ysandeep is now known as ysandeep|away14:34
*** ykarel|away has quit IRC14:51
*** chandankumar is now known as raukadah14:52
*** fressi has quit IRC14:54
*** qchris has quit IRC14:57
*** ysandeep|away is now known as ysandeep15:06
*** qchris has joined #opendev15:10
*** ysandeep is now known as ysandeep|away15:10
clarkbfollowing up on the nodepool builders all 4 are running and we have images as new as 9 hours ago15:10
clarkbeverything should be back to normal now without pauses etc15:10
clarkb(thank you ianw for getting that done)15:11
*** tkajinam has quit IRC15:26
openstackgerritClark Boylan proposed openstack/diskimage-builder master: Don't remove packages that are requested to be installed  https://review.opendev.org/74722015:30
clarkbyoctozepto: ^ I think there was actually a bug in the old branching behavior that I noticed when you pointed ou the redundant branch15:30
clarkbyoctozepto: I think that ps fixes it and addresses your comment15:30
*** rosmaita has left #opendev15:52
yoctozeptoclarkb: thanks, now I read it even differently; could you address my comments now?15:53
*** mlavalle has quit IRC15:59
*** tosky has quit IRC16:00
clarkbyoctozepto: responded. Hopefully that helps16:01
*** mlavalle has joined #opendev16:02
yoctozeptoclarkb: one clear, one to go ;d16:21
yoctozeptoI don't deny something might be escaping me16:21
clarkbyoctozepto: hopefully latest comment helps. Basically there is a behavior that I'm trying to preserve as much as possible16:41
clarkbwhether or not that existing behavior is actually critical I don't know16:41
*** dtantsur is now known as dtantsur|afk16:50
*** hashar has quit IRC16:59
*** priteau has quit IRC17:07
yoctozeptoclarkb: ok, the example was great; I guess I'm just a little tired! I left +1 (not a core yet) and some future ideas17:20
clarkbyoctozepto: fwiw I had to run the script locaally and generate files to grok the format17:35
fungitotally weird collision of topics today, i went looking for a geekcode decoder, and the geekcode.pm module on cpan was written by the ethercalc author17:39
clarkbfungi: I think the ethercalc author was/is also secretary of technology of taiwan17:39
fungiyikes17:40
clarkb(I may have gotten the exact title wrong but cabinet level type equivalent)17:40
fungi"taiwan's digital minister"17:40
fungianyway, looking at the latest ethercalc readme we need at least nodejs 0.8 and redis 2.417:42
fungiwhich we're still easily running, it looks like17:44
yoctozeptonodejs 0.8? lol17:45
yoctozeptoit remembers dinosaurs17:45
fungiit recommends 4.x but supports 0.8+17:45
yoctozeptodoes it have no deps?17:45
yoctozeptomany libs dropped support for anything pre-8.x a long time ago17:46
yoctozeptoso I'd worry about its security stance17:46
fungihttps://www.npmjs.com/package/ethercalc lists 13 deps17:47
fungibut yeah, the yarn.lock file hasn't been touched in 2 years17:48
yoctozeptofungi: optimist was dropped a long time ago17:48
yoctozeptoI mean one of the deps17:48
fungieven though the most recent release of ethercalc is only 3 months old17:48
yoctozeptoI do hope nobody dropped any optimists :-)17:48
yoctozeptothey deserve to live17:49
fungiso not regenerating the deps list when making new releases, looks like17:49
yoctozeptowell, you can't really regenerate for deps which are long retired ;-)17:49
fungisure17:49
yoctozeptozappajs looks bad as duck as well17:50
yoctozeptoso it's 7 years old lib and a 3 yu old lib alreay17:50
yoctozeptouuid-pure 9 years17:50
yoctozeptowho gives more17:50
yoctozeptojust burn it17:50
fungianyway, my point was it looks like we can probably upgrade to latest ethercalc on the current server without touching our config management otherwise, and get the fix to the annoying csvexport crash17:51
clarkbjust because a lib is old doesn't necessarily mean it needs updates. I mean half those libs are probably a single function :P17:51
yoctozeptoclarkb: sure, cors is a known stable lib17:51
clarkbfungi: any idea if we have to migrate the data at all? I Guess we can just dump the redis data (I think we back it up daily) then upgrade and if it doesn't work roll back17:51
yoctozeptoclarkb: but optimist is openly retired17:51
yoctozeptoclarkb: zappajs is a whole framework17:52
clarkbyoctozepto: but also what are our choices there? turn off the service? I'm not entirely opposed17:52
clarkbI'm not going to maintain a bunch of js libs. I don't have enough time in the day as it is17:52
yoctozeptoclarkb: true that, I mean just know the consequences17:52
fungiit's been fairly low-maintenance until this past week when we started having someone/something triggering this 3-years-fixed-upstream bug17:52
yoctozeptoI'm using ethercalc from time to time but might move to something else if needed17:52
yoctozeptofungi: is it used much in general?17:53
yoctozeptoi mean ethercalc17:53
yoctozeptoin opendev17:53
clarkbyoctozepto: not like etherpad17:53
clarkbits a much smaller niche17:53
yoctozeptoclarkb: i figured17:53
fungiit's being used for ptg timeslot coordination, for example17:53
clarkbya it is used though17:53
yoctozeptoi see17:53
fungiso i don't want to turn it off out from under people with no warning, but similarly i don't want to have to keep watching it to see if it's crashed yet again17:54
yoctozeptolet's then just cross finger and upgrade for now17:55
fungiand fixing it today is probably better since i'm out all next week meaning even fewer people around to find out it's down and restart it17:55
clarkbfungi: ya do a db dump first so wecan rollback if necessary and update imo17:55
openstackgerritJeremy Stanley proposed opendev/puppet-ethercalc master: Upgrade to latest EtherCalc (0.20200505.0)  https://review.opendev.org/74745317:59
fungiclarkb: yoctozepto: ^17:59
corvusfungi: plan lgtm17:59
funginow we get to find out what bitrot that module has accumulated in over a year since its last merged change17:59
fungii'm expecting random jobs to fail and need fixes inserted ahead of that18:00
fungiso will keep tabs on zuul results18:00
*** shtepanie has joined #opendev18:02
yoctozeptofungi: I understand I should not be working on my sheet now ;d18:04
corvusyoctozepto: i think you have a little while before excitement happens :)18:04
yoctozeptocorvus: it's been a wild CI ride week, I like excitement18:05
fungiyeah, for starters, to completely expect to need to push more changes to fix 1.5 years of accumulated bitrot in puppet-ethercalc before we can hope to merge that18:07
fungihopefully they'll be straightforward issues, i expect we've seen most of them already in our other modules at this point18:08
fungiand then once jobs are succeeding, i'll make an additional db dump before we approve18:08
fungias added insurance in case we have to roll back18:09
clarkbI just fixed the beaker rspec ssh issue and that got jobs passing for another erpo18:12
clarkbI expect that means it will be just known issues like the :: lint problem18:12
*** andrewbonney has quit IRC18:27
openstackgerritClark Boylan proposed opendev/puppet-ethercalc master: Upgrade to latest EtherCalc (0.20200505.0)  https://review.opendev.org/74745319:14
clarkbfungi: ^ that should fix the linter errors19:14
fungithanks! i was just circling back around to check19:15
fungiby the way, how awesome is it that we had a version of puppet-lint which told us we should put :: in front of everything, and then a later version which said we should remove :: from in front of everything? i don't know if i've mentioned yet how much i love undoing something which didn't seem necessary to add in the first place19:17
yoctozeptofungi: like puppet itself19:19
yoctozepto:-)19:19
fungiin fairness, puppet was better than no configuration management, and i'm not sorry we used it19:24
corvussee change #1 :)19:24
fungibut we've well outgrown it and it's a questionable fit for our current use cases19:24
corvusplz can haz config managed ssh? kthxbye19:25
yoctozeptocorvus: where19:27
corvusyoctozepto: https://review.opendev.org/#/c/1/19:27
corvusyoctozepto: the first change in gerrit was adding some really basic puppet config19:27
corvusbecause, as fungi says, it was (much) better than nothing19:28
fungialso a prescient change to implement in light of modern situations like https://www.guardicore.com/2020/08/fritzfrog-p2p-botnet-infects-ssh-servers/19:29
*** moppy has quit IRC19:29
*** yoctozepto1 has joined #opendev19:30
fungiat this point anyone who is running a public ssh with even moderately weak passwords is completely pwn3d and has been for rather a while19:30
*** moppy has joined #opendev19:30
yoctozepto1corvus: "Also Puppet should be capitalized" :D19:31
*** yoctozepto has quit IRC19:32
*** yoctozepto1 is now known as yoctozepto19:32
corvusyeah, it's kind of a graffiti magnet, but that's okay :)19:32
*** jhesketh_ has joined #opendev19:38
*** shtepanie has quit IRC19:42
*** jhesketh has quit IRC19:42
*** mlavalle has quit IRC19:42
*** mlavalle has joined #opendev19:48
clarkbfungi: I've +2'd the ethercalc change but not approved it19:55
clarkbfungi: I'm going to ride the bike shortly so won't be around if it lands soon19:56
clarkbbut feel free to land it when you are able to watch it (and probably do  db dump?)19:56
fungiclarkb: thanks, i'm going to start the dump now20:05
fungioh, actually we don't need to create the dump, redis does that continually20:08
fungi-rw-rw---- 1 redis redis 31078548 Aug 21 20:06 /var/lib/redis/dump.rdb20:09
fungijust need to back it up20:10
openstackgerritMerged opendev/system-config master: Run service-eavesdrop hourly  https://review.opendev.org/74618120:12
clarkbok popping out now. Back in $timeperiod20:17
*** tosky has joined #opendev20:30
openstackgerritMerged opendev/puppet-ethercalc master: Upgrade to latest EtherCalc (0.20200505.0)  https://review.opendev.org/74745320:37
fungiand now we wait for the hourly puppet-else run20:38
fungilooks like the run which was in progress did not update it (or at least didn't restart the service) but there's a fresh one queued up as a few minutes ago21:04
fungihopefully it'll finish around the time i finish eating dinner21:05
*** owalsh has quit IRC21:25
fungilooks like it updated, but didn't restart the service22:05
fungimade one more redis dump copy and am manually restarting ethercalc now22:07
fungiand it's working again22:08
fungianyone please feel free to check existing sheets you have urls for handy, and/or try creating new sheets22:08
corvusi just created https://ethercalc.openstack.org/wwcw19levbug22:09
fungicurrent ptg scheduling sheet still has content: https://ethercalc.openstack.org/126u8ek25noy22:10
fungier, no that's the previous one22:10
fungibut looks right22:10
corvusi see lots of familiar code words so i guess that works22:11
fungithis is the newer one, also still looks right: https://ethercalc.openstack.org/7xp2pcbh1ncb22:11
clarkbthe opendev timeslots look correct on ^22:30
*** mlavalle has quit IRC23:02
*** DSpider has quit IRC23:40
*** tosky has quit IRC23:48

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!