Wednesday, 2020-09-09

*** Goneri has quit IRC00:22
*** hamalq has quit IRC00:31
*** piotrowskim has quit IRC00:43
*** Roamer` has quit IRC01:06
*** d34dh0r53 has quit IRC01:20
*** tetsuro has joined #openstack-infra01:23
*** gyee has quit IRC01:25
*** tetsuro has quit IRC01:25
*** ricolin_ has joined #openstack-infra01:31
*** rcernin has quit IRC02:29
*** rlandy has joined #openstack-infra03:03
*** armax has quit IRC03:10
*** armax has joined #openstack-infra03:10
*** armax has quit IRC03:10
*** stevebaker has quit IRC03:14
*** zzzeek has quit IRC03:18
*** zzzeek has joined #openstack-infra03:19
*** rcernin has joined #openstack-infra03:29
*** rcernin has quit IRC03:29
*** rcernin has joined #openstack-infra03:29
*** rlandy has quit IRC03:31
*** zzzeek has quit IRC03:55
*** zzzeek has joined #openstack-infra03:58
*** yankcrime has quit IRC03:58
*** systemd has quit IRC03:58
*** systemd has joined #openstack-infra03:58
*** ykarel|away has joined #openstack-infra04:19
*** ykarel|away is now known as ykarel04:26
*** evrardjp has quit IRC04:33
*** evrardjp has joined #openstack-infra04:33
*** zzzeek has quit IRC04:45
*** zzzeek has joined #openstack-infra04:48
*** matt_kosut has joined #openstack-infra05:05
openstackgerritOleksandr Kozachenko proposed openstack/project-config master: Add monasca projects in vexxhost tenant  https://review.opendev.org/75056105:09
*** zzzeek has quit IRC05:17
*** zzzeek has joined #openstack-infra05:18
*** viks____ has joined #openstack-infra05:34
*** ykarel has quit IRC05:40
*** ysandeep|away is now known as ysandeep05:42
*** ykarel has joined #openstack-infra05:46
*** lmiccini has joined #openstack-infra05:56
zer0c00lcan the public access firehose.openstack.org? if so where to do i get a username/password?06:05
zer0c00lI am trying to setup a local bot for gerritevents at my company.06:05
zer0c00li'd like to subscribe to germqtt.06:07
zer0c00lSo far i get 'unauthorized'06:07
zer0c00l mosquitto_sub -h firehose.openstack.org --topic 'gerrit'06:07
zer0c00lConnection error: Connection Refused: not authorised.06:07
*** eolivare has joined #openstack-infra06:08
*** eolivare has quit IRC06:10
*** vishalmanchanda has joined #openstack-infra06:14
*** xek has joined #openstack-infra06:24
*** ralonsoh has joined #openstack-infra06:34
*** dklyle has quit IRC06:39
*** gfidente has joined #openstack-infra06:47
*** slaweq has joined #openstack-infra06:51
*** ramishra has quit IRC06:51
openstackgerritMerged openstack/project-config master: Add monasca projects in vexxhost tenant  https://review.opendev.org/75056106:56
*** jcapitao has joined #openstack-infra07:00
*** jtomasek has joined #openstack-infra07:01
*** xek has quit IRC07:01
*** eolivare has joined #openstack-infra07:02
*** andrewbonney has joined #openstack-infra07:06
*** hashar has joined #openstack-infra07:17
*** ramishra has joined #openstack-infra07:24
*** ykarel has quit IRC07:28
*** ykarel has joined #openstack-infra07:29
*** jpena|off is now known as jpena07:30
*** ysandeep is now known as ysandeep|lunch07:40
*** tosky has joined #openstack-infra07:57
*** lmiccini has quit IRC08:02
*** lmiccini has joined #openstack-infra08:05
*** Roamer` has joined #openstack-infra08:06
*** lucasagomes has joined #openstack-infra08:10
*** ykarel has quit IRC08:18
*** ykarel has joined #openstack-infra08:19
*** ykarel has quit IRC08:20
*** slaweq has quit IRC08:20
*** rcernin has quit IRC08:20
*** ykarel has joined #openstack-infra08:21
*** slaweq has joined #openstack-infra08:29
*** Lucas_Gray has joined #openstack-infra08:30
*** derekh has joined #openstack-infra08:32
*** ykarel has quit IRC08:34
*** ykarel has joined #openstack-infra08:35
*** ianychoi__ has quit IRC08:40
*** slaweq has quit IRC08:49
*** ysandeep|lunch is now known as ysandeep08:55
*** Tengu has quit IRC09:10
*** Tengu has joined #openstack-infra09:11
*** ykarel_ has joined #openstack-infra09:13
*** ykarel has quit IRC09:14
*** Tengu has quit IRC09:17
*** Tengu has joined #openstack-infra09:18
*** Lucas_Gray has quit IRC09:25
*** lxkong has joined #openstack-infra09:27
*** lxkong has quit IRC09:34
*** Lucas_Gray has joined #openstack-infra09:34
*** dtantsur|afk is now known as dtantsur09:35
*** ricolin_ has quit IRC09:38
*** hashar has quit IRC09:57
*** ralonsoh has quit IRC10:01
*** ralonsoh has joined #openstack-infra10:02
*** ykarel_ is now known as ykarel10:19
*** jcapitao is now known as jcapitao_lunch10:42
*** priteau has joined #openstack-infra10:42
*** rledisez has quit IRC10:50
*** rcernin has joined #openstack-infra10:52
*** rledisez has joined #openstack-infra10:52
mtreinishzer0c00l: there should be public access to the firehose, the mqtt broker should be configured for public read access. My guess is that something is up with the server, an infra-root admin will have to check11:14
*** ykarel has quit IRC11:15
*** ykarel has joined #openstack-infra11:16
*** svyas is now known as svyas|afk11:18
*** rcernin has quit IRC11:28
*** jpena is now known as jpena|lunch11:30
*** lucasagomes has quit IRC11:32
*** lucasagomes has joined #openstack-infra11:32
openstackgerritOleksandr Kozachenko proposed openstack/project-config master: Add monasca-tempest-plugin in vexxhost tenant  https://review.opendev.org/75062711:37
*** rcernin has joined #openstack-infra11:45
*** svyas|afk is now known as svyas11:52
*** jcapitao_lunch is now known as jcapitao11:56
*** xek has joined #openstack-infra11:59
*** rfolco has joined #openstack-infra12:02
*** svyas is now known as svyas|AFK12:04
*** svyas|AFK is now known as svyas|afk12:04
*** Goneri has joined #openstack-infra12:07
*** jamesdenton has quit IRC12:08
*** thiagop has joined #openstack-infra12:09
*** rlandy has joined #openstack-infra12:09
*** jamesdenton has joined #openstack-infra12:10
openstackgerritTristan Cacqueray proposed openstack/project-config master: pynotedb: end project gating  https://review.opendev.org/75063412:17
openstackgerritTristan Cacqueray proposed openstack/project-config master: pynotedb: remove project from infrastructure systems  https://review.opendev.org/75063512:17
*** jhesketh has quit IRC12:17
*** jhesketh has joined #openstack-infra12:17
*** derekh has quit IRC12:31
*** jpena|lunch is now known as jpena12:32
openstackgerritLee Yarwood proposed openstack/project-config master: WIP Add Fedora 32 builds  https://review.opendev.org/75064212:36
openstackgerritSean McGinnis proposed openstack/project-config master: Set neutron-lib stable ACLs  https://review.opendev.org/75064312:39
*** rcernin has quit IRC12:41
*** piotrowskim has joined #openstack-infra12:45
openstackgerritAndreas Jaeger proposed openstack/project-config master: Fix networking-l2gw missed change  https://review.opendev.org/75064512:48
fungimtreinish: yep, i replied to zer0c00l over in #opendev and started looking into it, but something's apparently broken, still trying to work out exactly what (service is running, ports are allowed through the fw, config is being passed on the command line, acl and password file are correctly referenced in the config)12:51
*** lyarwood has joined #openstack-infra12:53
*** ykarel_ has joined #openstack-infra12:56
*** ykarel has quit IRC12:58
mtreinishfungi: I think it might be a server config issue (was there a mosquitto version update?) when I connected with result code 5. Is there anything in the logs?13:03
mtreinish*when I connected with a python script it returned 'result code 5' which means its at least talking to the server13:04
*** derekh has joined #openstack-infra13:06
*** ykarel_ has quit IRC13:09
*** ykarel has joined #openstack-infra13:12
*** rcernin has joined #openstack-infra13:14
*** xek has quit IRC13:15
fungimtreinish: the log is mostly flooded with "OpenSSL Error: error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol" and less frequently "OpenSSL Error: error:140780E5:SSL routines:ssl23_read:ssl handshake failure"13:17
fungieverything else in the log is the usual startup messages (starting version, config loaded, opening listen sockets, opening websockets, saving in-memory db)13:18
mtreinishhmm, that might mean it's trying to enforce ssl connections on the open read only ports which are supposed to be without ssl13:21
fungicurrent installed version is the 1.6.4-0mosquitto1~xenial1 ubuntu package from august 201913:22
fungimtreinish: yeah, i wondered if tat could be what's happening, so i tested connecting to 1883/tcp with openssl s_client, but it doesn't get a server cert13:22
fungior tls handshake i guess13:23
fungialso the config matches what the manpage says it should have to listen for plaintext connections on 188313:23
fungiit would help to know the last time someone successfully connected13:28
mtreinishhmm, yeah looking at the tcpdump of the traffic I think it's an acl issue. (wireshark knows mqtt :) )13:28
mtreinishcode 5 is apparently not authorized13:29
fungithe ssl errors go back at least a week, which is all the log retention we're configured for13:29
fungiinfra_service.acl was last modified in 201713:30
fungiand the path to it in the loaded mosquitto.conf is correct13:30
*** jhesketh has quit IRC13:31
fungithere is also a /etc/mosquitto/conf.d directory, but it's essentially empty (just has a readme in it)13:31
fungiand mosquitto.conf hasn't been modified since 201813:32
mtreinishyeah, that's why I was asking about the version. Since I assumed the configs hadn't really changed. The last version I remember playing with was 1.4.x (which I remember we had to backport a cve fix on the ubuntu package)13:33
mtreinishreading the latest man page I think I figured it out, is allow_anonymous set to true?13:36
fungi1.5.6 seems to have come with a bunch of security updates regarding password file, acl file, et cetera13:36
mtreinishit looks like it defaults to false now, which is a change from what I see in the puppet-mosquitto template file (which I just copied from a default config and templatized)13:37
funginot explicitly set, and the comments in the config are clearly from before the flipped the default. good eye!13:37
fungii'll try ot on the server real fast13:37
fungilooks like that may have done it13:39
fungii'm at least not getting rejected now13:39
mtreinishyep, I get return code 0 now13:39
fungithough there's also no events being streamed. i may need to go kick the pubclients13:39
mtreinishI'll push up a review to puppet-mosquitto uncommenting that line13:40
*** rcernin has quit IRC13:41
fungimtreinish: thanks! maybe also clear out or fix the misleadingly incorrect comment immediately above it about the default value while you're in there?13:42
mtreinishyep, I'll update the comment from the text in the man page13:43
fungii've restarted germqtt, and hey! there are events now13:45
mtreinishfungi: https://review.opendev.org/#/c/750659/13:49
fungiexcellent, thanks again13:49
fungii've restarted lpmqtt and mqtt_statsd now too13:50
mtreinishcool, thanks. I see all the events now too13:51
mtreinishheh, I guess this has been broken since the mosquitto version got updated. I guess nobody was really using it before now :P13:51
mtreinish(or just silently giving up when they encountered errors)13:51
fungiquite possible13:52
fungiwhich version flipped the default? i can get a rough guess from that as to when it broke13:53
mtreinishbased on the changelog it looks like it was changed in 1.5: https://github.com/eclipse/mosquitto/blob/master/ChangeLog.txt#L786-L79113:57
*** slaweq has joined #openstack-infra14:03
fungilooks like the first backport of 1.5.x to xenial was in 1.5.3-0mosquitto1~xenial from Tue, 25 Sep 2018 16:29:40 +010014:04
fungiaccording to /usr/share/doc/mosquitto/changelog.Debian.gz14:04
fungiso going on two years ago14:04
fungithough maybe the package upgrade didn't restart the service, in which case it may have not taken effect until the server was rebooted in march of 201914:07
fungi(current uptime is 555 days)14:07
*** d34dh0r53 has joined #openstack-infra14:11
*** ricolin_ has joined #openstack-infra14:11
*** ricolin_ has quit IRC14:13
*** armax has joined #openstack-infra14:21
*** xek has joined #openstack-infra14:34
*** outbrito has joined #openstack-infra14:40
*** thiagop has quit IRC14:43
*** dklyle has joined #openstack-infra14:46
*** outbrito is now known as thiagop14:51
*** rfolco is now known as rfolco|ruck14:53
*** d34dh0r53 has quit IRC14:54
*** d34dh0r53 has joined #openstack-infra14:54
mtreinishlol, ok much longer than I was hoping :)14:58
*** slaweq has quit IRC15:04
*** bhagyashris is now known as bhagyashris|ruck15:11
*** bhagyashris|ruck is now known as bhagyashris|rove15:11
*** lpetrut has joined #openstack-infra15:12
*** akahat|rover is now known as akahat15:20
*** jhesketh has joined #openstack-infra15:23
*** Lucas_Gray has quit IRC15:29
*** Lucas_Gray has joined #openstack-infra15:30
*** ramishra has quit IRC15:30
*** zzzeek has quit IRC15:31
*** zzzeek has joined #openstack-infra15:32
*** lmiccini has quit IRC15:36
*** zzzeek has quit IRC15:37
*** zzzeek has joined #openstack-infra15:38
*** zzzeek has quit IRC15:42
*** zzzeek has joined #openstack-infra15:45
*** ramishra has joined #openstack-infra15:46
AJaegerconfig-core, please review https://review.opendev.org/750645 to fix Zuul config errors in openstack/project-config after the changes to networking-l2gw15:48
*** zzzeek has quit IRC15:50
openstackgerritMerged openstack/openstack-zuul-jobs master: Migrate openstack-tox jobs to run on Ubuntu Focal  https://review.opendev.org/73832215:50
*** zzzeek has joined #openstack-infra15:51
*** ysandeep is now known as ysandeep|away15:51
openstackgerritGhanshyam Mann proposed openstack/openstack-zuul-jobs master: Migrate unit tests jobs to run on Ubuntu Focal  https://review.opendev.org/73832315:53
gmannfungi: ^^ updated cmt msg15:54
*** zzzeek has quit IRC15:56
fungigmann: i've approve it, but you'll probably have to rebase the 5 changes which were parented on it before they'll be able to merge15:56
fungier, i've approveD15:56
gmannsure15:56
openstackgerritGhanshyam Mann proposed openstack/openstack-zuul-jobs master: Migrate openstack-tox-cover jobs on Ubuntu Focal  https://review.opendev.org/73832415:57
openstackgerritGhanshyam Mann proposed openstack/openstack-zuul-jobs master: Migrate functional tests jobs to Ubuntu Focal  https://review.opendev.org/73832515:57
openstackgerritGhanshyam Mann proposed openstack/openstack-zuul-jobs master: Migrate openstack-tox-docs jobs to Ubuntu Focal  https://review.opendev.org/73832615:57
*** ykarel is now known as ykarel|away15:57
openstackgerritGhanshyam Mann proposed openstack/openstack-zuul-jobs master: Pin ubuntu bionic for tox-py27 and tox-py36 in no-constraints template  https://review.opendev.org/73832715:57
openstackgerritGhanshyam Mann proposed openstack/openstack-zuul-jobs master: Pin nodejs6 & nodejs8 jobs on ubuntu-bionic  https://review.opendev.org/73832815:57
gmanndone ^^15:57
*** lpetrut has quit IRC15:59
*** zzzeek has joined #openstack-infra15:59
*** zzzeek has quit IRC16:04
*** zzzeek has joined #openstack-infra16:05
*** lucasagomes has quit IRC16:05
*** Lucas_Gray has quit IRC16:12
*** Lucas_Gray has joined #openstack-infra16:20
*** ykarel|away has quit IRC16:31
*** jcapitao has quit IRC16:36
openstackgerritMerged openstack/openstack-zuul-jobs master: Migrate unit tests jobs to run on Ubuntu Focal  https://review.opendev.org/73832316:42
openstackgerritMerged openstack/openstack-zuul-jobs master: Migrate openstack-tox-cover jobs on Ubuntu Focal  https://review.opendev.org/73832416:42
*** ramishra has quit IRC16:43
*** Lucas_Gray has quit IRC16:51
*** dosaboy has quit IRC16:51
*** dtantsur is now known as dtantsur|afk16:55
*** slaweq has joined #openstack-infra16:57
*** Lucas_Gray has joined #openstack-infra16:58
*** derekh has quit IRC17:00
*** gfidente is now known as gfidente|afk17:02
*** Lucas_Gray has quit IRC17:03
*** Lucas_Gray has joined #openstack-infra17:07
*** jpena is now known as jpena|off17:11
*** slaweq has quit IRC17:12
*** andrewbonney has quit IRC17:14
openstackgerritMerged openstack/project-config master: Disable vexxhost for mirror work  https://review.opendev.org/75048417:25
*** Lucas_Gray has quit IRC17:31
*** gyee has joined #openstack-infra17:33
openstackgerritClark Boylan proposed openstack/project-config master: Revert "Disable vexxhost for mirror work"  https://review.opendev.org/75076517:52
*** ralonsoh has quit IRC18:05
fricklergmann: looks to me like this is some fallout of tox on focal https://zuul.opendev.org/t/openstack/build/ed4a21ac37b6446782d5ce01fb65be3218:09
gmannfrickler: yeah, need to fix that, doing18:10
gmanni thought bcafarel has patch up for that18:11
*** hashar has joined #openstack-infra18:13
*** hashar has quit IRC18:31
*** zzzeek has quit IRC18:33
*** zzzeek has joined #openstack-infra18:34
*** eolivare has quit IRC18:38
*** zzzeek has quit IRC18:41
*** zzzeek has joined #openstack-infra18:42
fungii've approved the rest of that stack now too18:47
fungiso we may see more fallout18:47
fungiany idea why that's changed pyflakes/flake8 behavior?18:48
*** zzzeek has quit IRC18:50
*** zzzeek has joined #openstack-infra18:50
fungihttps://github.com/PyCQA/pyflakes/issues/36718:51
fungiso it's that the version of pyflakes we've pinned in constraints doesn't work on python 3.8 which we've just switched to, so we need to use 2.1.0 or later on focal18:53
gmannyeah this one https://bugs.launchpad.net/hacking/+bug/188629618:54
openstackLaunchpad bug 1886296 in hacking "pyflakes till 2.1.0 not compatible with py3.8 (ubuntu focal)" [Undecided,In progress] - Assigned to Ghanshyam Mann (ghanshyammann)18:54
gmanni have taken care this in hacking 3.2.0 so projects moving to new hacking do not need to add it explicitly18:54
gmannfungi: thanks for approving those18:54
fungino sweat18:54
openstackgerritMerged openstack/openstack-zuul-jobs master: Migrate functional tests jobs to Ubuntu Focal  https://review.opendev.org/73832518:55
*** zzzeek has quit IRC18:55
fungigmann: frickler: thanks, so any project which is blocked on that error needs to switch to newer hacking?18:55
*** zzzeek has joined #openstack-infra18:56
fungi(or newer pyflakes if not using hacking)18:56
gmannyeah18:56
fungii guess we don't actually control those with constraints, so it's up to individual projects18:56
gmannnewer hacking might need more work on pep8 chekcs but depends on project to project18:56
gmanntrue18:56
*** priteau has quit IRC18:57
*** hashar has joined #openstack-infra18:58
AJaegerfungi: yes, hacking and dependencies are not in constraints so that projects can upgrade on thwir own scheculde18:59
openstackgerritMerged openstack/openstack-zuul-jobs master: Migrate openstack-tox-docs jobs to Ubuntu Focal  https://review.opendev.org/73832619:00
openstackgerritMerged openstack/openstack-zuul-jobs master: Pin ubuntu bionic for tox-py27 and tox-py36 in no-constraints template  https://review.opendev.org/73832719:00
openstackgerritMerged openstack/openstack-zuul-jobs master: Pin nodejs6 & nodejs8 jobs on ubuntu-bionic  https://review.opendev.org/73832819:00
openstackgerritMerged openstack/project-config master: Fix networking-l2gw missed change  https://review.opendev.org/75064519:00
fungiwhich means that failure is really just an incentive for people to get around to updating ;)19:01
fungigmann: that might warrant a note to openstack-discuss19:01
AJaegerfungi: don't share any secret plans ;)19:01
gmannfungi: its in on Focal migration thread updates . I will update with base patches are merged now19:03
*** zzzeek has quit IRC19:03
openstackgerritAndreas Jaeger proposed openstack/project-config master: pynotedb: remove project from infrastructure systems  https://review.opendev.org/75063519:03
fungigmann: oh, great. somehow i missed it when following that ml thread19:04
openstackgerritAndreas Jaeger proposed openstack/project-config master: pynotedb: remove project from infrastructure systems  https://review.opendev.org/75063519:04
*** zzzeek has joined #openstack-infra19:05
openstackgerritMerged openstack/project-config master: Add monasca-tempest-plugin in vexxhost tenant  https://review.opendev.org/75062719:10
*** slaweq has joined #openstack-infra19:17
*** zzzeek has quit IRC19:21
*** zzzeek has joined #openstack-infra19:23
*** piotrowskim has quit IRC19:34
openstackgerritMerged openstack/project-config master: Retire the devstack-plugin-zmq project  https://review.opendev.org/74872419:36
*** tosky has quit IRC19:37
openstackgerritMerged openstack/project-config master: pynotedb: end project gating  https://review.opendev.org/75063419:43
*** vishalmanchanda has quit IRC19:45
openstackgerritMerged openstack/project-config master: Revert "Disable vexxhost for mirror work"  https://review.opendev.org/75076519:51
*** xek has quit IRC20:14
*** openstackgerrit has quit IRC20:17
*** Lucas_Gray has joined #openstack-infra20:21
*** stevebaker has joined #openstack-infra20:32
*** matt_kosut has quit IRC20:50
*** slaweq has quit IRC20:51
*** matt_kosut has joined #openstack-infra20:51
*** stevebaker_ has joined #openstack-infra20:52
*** stevebaker has quit IRC20:52
*** gfidente|afk has quit IRC20:53
*** slaweq has joined #openstack-infra20:55
*** matt_kosut has quit IRC20:57
*** slaweq has quit IRC20:59
*** rfolco|ruck has quit IRC21:03
*** hashar has quit IRC21:06
*** thiagop has quit IRC21:12
*** slaweq has joined #openstack-infra21:26
* TheJulia pokes zuul21:27
clarkbTheJulia: its being restarted to pick up some updates21:27
TheJuliaack21:27
clarkbmight be a minute as there is a db migration21:27
smcginnisThat explains it. :)21:28
TheJuliadid jobs get killed or are they still running?21:28
clarkbthey will be restarted21:28
TheJuliaok21:28
*** slaweq has quit IRC21:31
fungiif you want to actually help poke zuul on a regular basis, i have the sign-up sheet over here somewhere ;)21:42
*** mmethot has quit IRC21:42
fungioh, nevermind, i gave it to clarkb years ago21:43
clarkbdo you want it back?21:43
fungino, not really21:43
fungiyou can just nail it to a telephone pole21:43
fungi"help wanted: sysadmins inquire within"21:44
*** mmethot has joined #openstack-infra21:46
*** carloss has joined #openstack-infra21:55
*** jtomasek has quit IRC21:56
*** dchen has quit IRC22:02
*** dchen has joined #openstack-infra22:03
*** Tengu has quit IRC22:06
*** Tengu has joined #openstack-infra22:07
*** Lucas_Gray has quit IRC22:13
*** openstackgerrit has joined #openstack-infra22:16
openstackgerritClark Boylan proposed openstack/project-config master: Pause image builds on nb03.openstack.org  https://review.opendev.org/75082222:16
*** auristor has quit IRC22:45
*** auristor has joined #openstack-infra22:49
*** rcernin has joined #openstack-infra22:54
*** rcernin has quit IRC22:56
*** rcernin has joined #openstack-infra22:56
*** zzzeek has quit IRC23:17
*** zzzeek has joined #openstack-infra23:19
*** Goneri has quit IRC23:21
*** dosaboy has joined #openstack-infra23:28
*** zzzeek has quit IRC23:38
*** zzzeek has joined #openstack-infra23:40
*** zzzeek has quit IRC23:59

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!