Tuesday, 2021-05-11

*** hamalq has quit IRC00:06
*** dviroel has quit IRC00:12
*** timburke has joined #openstack-infra00:26
openstackgerritAde Lee proposed openstack/openstack-zuul-jobs master: Add FIPS job  https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/79053400:43
*** gyee has quit IRC01:06
*** __ministry has joined #openstack-infra01:12
*** ysandeep|SL is now known as ysandeep01:54
*** whoami-rajat has quit IRC02:18
*** zxiiro has quit IRC02:19
*** ysandeep is now known as ysandeep|afk03:23
*** carloss has quit IRC03:49
*** lajoskatona has joined #openstack-infra03:56
*** rlandy has quit IRC03:57
*** lajoskatona has quit IRC04:00
*** ykarel has joined #openstack-infra04:53
*** ralonsoh has joined #openstack-infra05:10
*** fdegir has quit IRC05:26
*** fdegir has joined #openstack-infra05:26
*** cgoncalves has quit IRC05:27
*** cgoncalves has joined #openstack-infra05:27
*** bnemec has quit IRC05:27
*** bnemec has joined #openstack-infra05:29
*** vishalmanchanda has joined #openstack-infra06:22
*** dklyle has quit IRC06:39
*** ysandeep|afk is now known as ysandeep06:48
*** jcapitao has joined #openstack-infra06:49
*** sboyron has joined #openstack-infra06:49
*** sshnaidm|afk has quit IRC07:03
*** sshnaidm has joined #openstack-infra07:20
*** hashar has joined #openstack-infra07:22
*** andrewbonney has joined #openstack-infra07:23
*** ramishra has quit IRC07:24
*** ramishra has joined #openstack-infra07:25
*** ociuhandu has joined #openstack-infra07:35
*** rpittau|afk is now known as rpittau07:38
*** tosky has joined #openstack-infra07:39
*** dasm has quit IRC07:41
*** mgoddard has quit IRC07:45
*** ociuhandu has quit IRC07:45
*** ociuhandu has joined #openstack-infra07:46
*** mgoddard has joined #openstack-infra07:50
*** ociuhandu has quit IRC07:50
*** jpena|off is now known as jpena07:51
*** dasm has joined #openstack-infra08:00
*** lucasagomes has joined #openstack-infra08:11
*** ociuhandu has joined #openstack-infra08:16
*** ociuhandu has quit IRC08:17
*** ociuhandu has joined #openstack-infra08:17
*** lpetrut has joined #openstack-infra08:22
*** rcernin has quit IRC08:32
*** dciabrin has quit IRC08:36
*** dciabrin has joined #openstack-infra08:36
openstackgerritLajos Katona proposed openstack/project-config master: Rename x/tap-as-a-service to openstack/tap-as-a-service  https://review.opendev.org/c/openstack/project-config/+/79009308:43
*** gfidente|afk is now known as gfidente09:01
*** rcernin has joined #openstack-infra09:09
*** ykarel_ has joined #openstack-infra09:19
*** ykarel has quit IRC09:21
*** rcernin has quit IRC09:27
*** ykarel_ is now known as ykarel09:28
*** rcernin has joined #openstack-infra09:34
*** ociuhandu has quit IRC09:52
*** rcernin has quit IRC09:53
*** ykarel is now known as ykarel|lunch09:54
*** ykarel_ has joined #openstack-infra09:58
*** ykarel|lunch has quit IRC10:00
*** ociuhandu has joined #openstack-infra10:10
*** dtantsur|afk is now known as dtantsur10:15
*** lpetrut has quit IRC10:30
*** rcernin has joined #openstack-infra10:44
*** rcernin has quit IRC10:48
*** carloss has joined #openstack-infra10:55
*** ykarel_ is now known as ykarel10:56
*** lpetrut has joined #openstack-infra10:56
*** jcapitao is now known as jcapitao_lunch10:57
*** ysandeep is now known as ysandeep|afk11:11
*** dviroel has joined #openstack-infra11:11
*** jpena is now known as jpena|lunch11:15
*** ociuhandu has quit IRC11:21
*** ociuhandu has joined #openstack-infra11:22
*** ociuhandu has quit IRC11:26
*** lpetrut has quit IRC11:27
*** gfidente has quit IRC11:37
*** ysandeep|afk is now known as ysandeep11:40
*** rlandy has joined #openstack-infra11:40
*** gfidente has joined #openstack-infra11:41
*** jcapitao_lunch is now known as jcapitao11:46
sean-k-mooneyclarkb: quick question how do you currently ingest the logs into logstation are the files parsed form txt or do you convert them to json first?11:53
*** jpena|lunch is now known as jpena11:56
*** ociuhandu has joined #openstack-infra11:58
*** ociuhandu has quit IRC12:02
*** dpawlik has quit IRC12:12
*** ykarel_ has joined #openstack-infra12:18
*** ociuhandu has joined #openstack-infra12:19
*** dpawlik3 has joined #openstack-infra12:20
*** ykarel has quit IRC12:20
*** dpawlik3 is now known as dpawlik12:26
sean-k-mooneyoh https://opendev.org/openstack/logstash-filters/src/branch/master/filters/openstack-filters.conf is how you ingest them today12:27
*** nweinber has joined #openstack-infra12:35
*** ociuhandu has quit IRC12:43
*** jaicaa has quit IRC12:47
*** jaicaa has joined #openstack-infra12:47
*** dwilde has joined #openstack-infra12:53
*** dwilde has quit IRC13:04
*** ociuhandu has joined #openstack-infra13:09
*** rcernin has joined #openstack-infra13:13
*** ociuhandu has quit IRC13:14
*** ociuhandu has joined #openstack-infra13:23
*** dwilde has joined #openstack-infra13:25
*** rcernin has quit IRC13:26
*** dwilde has quit IRC13:34
*** Roamer` has quit IRC13:38
*** ykarel_ has quit IRC13:42
*** ykarel_ has joined #openstack-infra13:44
*** stephenfin has quit IRC13:56
*** dwilde has joined #openstack-infra13:57
*** jcapitao has quit IRC13:58
*** dwilde has quit IRC13:59
*** dwilde has joined #openstack-infra13:59
*** jcapitao has joined #openstack-infra13:59
*** iurygregory has quit IRC14:22
*** iurygregory has joined #openstack-infra14:22
*** happyhemant has joined #openstack-infra14:24
*** ykarel_ is now known as ykarel14:25
fungisean-k-mooney: yep, the reason it's in its own repo is that openstack operators ostensibly had an interest in collaborating on those filters so they could be used in production environments too, though i have no idea to what extent that actually happened in the end14:26
*** hashar is now known as hasharAway14:29
sean-k-mooneygodaddy had there own set https://github.com/godaddy/openstack-logstash14:34
sean-k-mooneybut ya not sure14:34
*** stephenfin has joined #openstack-infra14:34
*** ykarel has quit IRC14:42
gmannfungi: clarkb mnaser can you check this quick one once have time -https://review.opendev.org/c/openstack/project-config/+/79005614:46
clarkbsean-k-mooney: fungi: yup its the same logs you see in the log dir for jobs. We have thought about doing json but then we'd need two copies of the logs which isn't great (because humans don't read json as easily)14:53
openstackgerritMerged openstack/project-config master: Retire puppet-glare - Step 1: End project Gating  https://review.opendev.org/c/openstack/project-config/+/79005614:56
sean-k-mooneyclarkb: ya after debuging customer issue with the logs in json format i agre it very human unfriendly15:04
sean-k-mooneyclarkb: there is apprently a tool in olso to convert form json to text but that is still proably more complex then we want15:05
clarkbsean-k-mooney: that is good to know about the tool, but I agree that would just make it less likely epople debug failures15:05
sean-k-mooneywell i was thinking we would log in json the convert to text and only store the text in swift15:06
sean-k-mooneybut ya more complex and proably not worth the effort15:06
clarkbwell we need to store what we index (at least in the current system)15:07
fungiany time i need to read json i end up leaning on SpamapS's https://pypi.org/project/flamel/15:07
fungiit's indispensable15:08
sean-k-mooneyfungi: the json format puts every line into a json object15:08
sean-k-mooneyeven if it converted to yaml its pretty hard to digest15:08
fungioh, ew15:09
clarkbya its there to simplify ingestion to elasticsearch15:09
*** ociuhandu has quit IRC15:09
clarkbbut terrible for humans15:09
clarkbyou can basically skip the logstash step if you do this15:09
clarkbas the data is preparsed15:09
sean-k-mooneythere is also nataiveish fluentd supprot in oslo log15:09
*** dave-mccowan has quit IRC15:10
sean-k-mooneyits just a tweak to the normal json output15:10
sean-k-mooneyclarkb: i can proably find it but did ye just use the tail plugin to load the log files or did ye do somehting smarter15:11
sean-k-mooneyoh are ye taking it via tcp https://opendev.org/opendev/system-config/src/branch/master/modules/openstack_project/templates/logstash/input.conf.erb15:12
clarkbsean-k-mooney: its a bit more complicated because when this all originated the way ELK recommended it was via redis and that worked for about 2 files before it fell over and OOM'd15:13
sean-k-mooneyhehe i see15:13
clarkbsean-k-mooney: what we do is zuul jobs submit a gearman job to a gearman server. We then have small python daemons that look for gearman jobs on that server that will download the file and push them via tcp to logstash15:13
sean-k-mooneyoh is that code somewhere15:14
*** dklyle has joined #openstack-infra15:14
clarkbthe gearman stuff is an implementation detail. I think doing the push over tcp like that is probably sane though15:14
clarkbsean-k-mooney: https://opendev.org/opendev/puppet-log_processor/src/branch/master/files/log-gearman-worker.py15:14
sean-k-mooneyi was considering using fluentd since kolla can deploy that isntead of logstach and pushign over http or tcp15:14
clarkbfluentd cannot do the necessary data parsing last I checked15:15
*** dave-mccowan has joined #openstack-infra15:15
sean-k-mooneyit has regex parsers but not sure if it can fully work15:15
sean-k-mooneyi could just use logstash too for what its worth15:15
clarkbdepending on the use case that is a good or a bad thing. For us it is a bad thing because we've got 50 different file formats and they all need massaging15:15
fungifor a replacement, i would probably reconsider the zuul job injecting things into geard approach, fwiw15:15
clarkbsean-k-mooney: oh if they added proper parsing then fluentd may work15:15
sean-k-mooneythere is a sampel opsnestack parser https://github.com/fluent/fluent-bit/blob/master/conf/parsers_openstack.conf15:16
sean-k-mooneyim not sure if that actully works15:16
clarkboh wow they basically reimplemtned logstash after asserting the logstash approach was wrong :)15:16
sean-k-mooneythey have parser plugins15:17
*** ociuhandu has joined #openstack-infra15:17
fungiat this point, zuul's api is robust and a new implementation could probably query zuul to find completed builds and get the log urls that way15:17
clarkbsean-k-mooney: ya so the main formats we handle are: the console log, "oslofmt" basically openstack services except swift, syslog/journald, apache combined log format15:17
clarkbfungi: yup15:17
clarkbswift uses apache combined log format15:18
sean-k-mooneyfungi: well that was kind of what i was thinking. i was contempleatin runnign my own zuul instance that would listen for comments form upstream zuul for -1 in a pipline15:18
sean-k-mooneythat would trigger a jobs that would downwload the output, process it if need and uploaded it to fulentd15:19
sean-k-mooneyim not sure that is a good idea or not15:19
sean-k-mooneybut basically wanted to fiture out how to injest logs first15:19
fungisean-k-mooney: thinking about it, i would probably not even worry about event streams15:20
fungitreat the zul builds api as a queue, and poll it periodically to check for new work15:20
fungier, zuul i mean15:21
sean-k-mooneyoh well i was thinking of having zuul anyway15:21
fungithough zul isn't in this channel at the moment so i don't need to apologize for the ping ;)15:21
sean-k-mooneybut i could have it run in a perodic instead15:21
*** ociuhandu has quit IRC15:22
fungiahh, yeah that's a great point, a second zuul to react to reports in gerrit and use that to coordinate the indexing work15:22
*** hjensas is now known as hjensas|afk15:22
*** ociuhandu has joined #openstack-infra15:22
fungithough not all builds are reported in gerrit15:23
sean-k-mooneylike the perodics15:23
fungizactly15:23
fungialso post, tag, et cetera15:23
sean-k-mooneyya gate and check are though which woudl be a start15:23
sean-k-mooneyok ill think about it anyway15:24
sean-k-mooneythere isnt a zuul zuul triger is there :P15:24
sean-k-mooneyhavign zuul trigger form extrenal zuul will only end bardly eventually :)15:25
funginot sure what you mean by "zuul zuul trigger"15:26
clarkbthere is a zuul driver but I think that may be for internal events, I don't think there is a way to expose those externally (yet?)15:26
clarkbfungi: one zuul tells another zuul to do work15:26
fungiyeah, i mainly am curious what channel such a trigger would use15:26
sean-k-mooneyclarkb: yep15:26
fungiprobably a webhook sort of driver15:27
sean-k-mooneyfungi: well i was thinkign of havign it check the zuul reset api for complete jobs or something like that15:27
sean-k-mooneyanyway it was not intended as a real idea because a cron job would proably work jsut as well15:27
sean-k-mooneyanyway im going to redploy my home infrastucure later this week and once i have a clean slate ill see how best i can use it.15:31
sean-k-mooneythe zuul api might be a bettere way to go or i might realise that this is too much work to actully run locally15:33
*** ysandeep is now known as ysandeep|away15:48
*** lucasagomes has quit IRC15:58
*** jcapitao has quit IRC15:58
*** ykarel has joined #openstack-infra16:00
*** hasharAway is now known as hashar16:02
*** lbragstad has quit IRC16:04
*** ociuhandu_ has joined #openstack-infra16:13
*** jamesden_ has joined #openstack-infra16:16
*** lbragstad has joined #openstack-infra16:16
*** ociuhandu has quit IRC16:17
*** jamesdenton has quit IRC16:17
*** ociuhandu_ has quit IRC16:18
*** dwilde has quit IRC16:18
*** ociuhandu has joined #openstack-infra16:25
*** ralonsoh has quit IRC16:25
*** dwilde has joined #openstack-infra16:25
*** ykarel has quit IRC16:29
*** ociuhandu has quit IRC16:30
*** dtantsur is now known as dtantsur|afk16:32
*** zxiiro has joined #openstack-infra16:33
*** sboyron has quit IRC16:34
*** dwilde has quit IRC16:35
*** rpittau is now known as rpittau|afk16:36
*** dwilde has joined #openstack-infra16:41
sean-k-mooneyclarkb: fungi  if ye are around i have a different question.16:48
sean-k-mooneyteh review priority lable16:49
fungiyah16:49
sean-k-mooneyis that sticky16:49
sean-k-mooneye.g. will it persist across patch revisions16:49
fungiwhether or not a label persists across reviews is configurable per label (and per project)16:49
sean-k-mooneyya so i was looking for the dfintion in https://github.com/openstack/project-config16:49
sean-k-mooneybut i have not found it yet16:50
fungithe various copy options listed at https://review.opendev.org/Documentation/config-labels.html#label_layout control those behaviors16:50
sean-k-mooneyperhaps its not public or loaded htere but i did not see the project.config file16:51
sean-k-mooneyyep16:51
sean-k-mooneyso maybe defining https://review.opendev.org/Documentation/config-labels.html#label_copyAnyScore16:51
sean-k-mooneyso in https://review.opendev.org/c/openstack/project-config/+/78752316:52
sean-k-mooneyi could set label_copyAnyScore = true16:52
sean-k-mooneywould i have to do that in each file by redfining the lable16:53
fungiwe don't define a review-priority label globally, any project which uses one defines that within their own project acl16:53
sean-k-mooneyoh ok so that is what i missed16:54
sean-k-mooneyok so ill copy the lable defition form one fo the other projects and configure it thanks16:54
fungisean-k-mooney: example is https://opendev.org/openstack/project-config/src/branch/master/gerrit/acls/openstack/cinder.config#L25-L35\16:55
fungier, minus the stray \ at the end of that line16:55
fungiwhich just happens to be adjacent to my return key, enough that my fat fingers added it at the last moment16:56
clarkbnow that openstack has the meta config it may be worth considering doing a global openstack review priority, but everyone has done it differently so may not be easy17:06
clarkbanother option is to tag things instead of using reviews17:07
*** jpena is now known as jpena|off17:07
*** andrewbonney has quit IRC17:07
fungiyeah, getting consensus between teams on how those new labels are used would be a necessary first step17:09
fungione that i would be happy to leave to the tc (who would likely delegate the fact finding to a volunteer pop-up team)17:10
*** gfidente is now known as gfidente|afk17:11
sean-k-mooneyat least the nova usecase is 2 fold. one directing active review capasity(core and non core) and two, potentailly  providing an async way for contibutors to singal to cores that they would like something to be a priortiy17:20
sean-k-mooneyill admit i was more thinkign of the former case which seams to be what cinder is using it for17:20
fungisean-k-mooney: and yeah, recommend also considering hashtags as an alternative implementation17:21
sean-k-mooneyhastags?17:21
sean-k-mooneycan you point me at an example17:21
fungisean-k-mooney: the quick intro is at https://review.opendev.org/Documentation/intro-user.html#hashtags17:22
sean-k-mooneyis that similar to the topic?17:23
sean-k-mooneyoh its sperate17:23
fungiyou can have more than one on a change, unlike the review topic field17:23
sean-k-mooneyoh i see it in the ui17:23
sean-k-mooneythere is no input bug17:23
sean-k-mooneyah17:24
sean-k-mooneyADD HASHTAG17:24
sean-k-mooneydose not show up in dark mode17:24
fungisean-k-mooney: it has to be enabled, https://opendev.org/openstack/project-config/src/branch/master/gerrit/acls/zuul/zuul.config#L617:24
fungii think it will error when you try to set one otherwise17:24
fungithough we've been talking about enabling it system-wide17:25
sean-k-mooneyoh acutlly it does17:25
sean-k-mooneyhttps://review.opendev.org/c/openstack/project-config/+/787523 has it17:25
sean-k-mooneybut i dont see it on https://review.opendev.org/c/openstack/neutron/+/766277/17:25
sean-k-mooneyoh that mereged17:26
sean-k-mooneyok so its by acl17:26
sean-k-mooneyfungi: i could see that as useful for catagorising patches17:26
*** dklyle has quit IRC17:26
sean-k-mooneyas in "networking" "live-migration" "ui" ectra17:27
*** dklyle has joined #openstack-infra17:27
clarkbya we've been talking about enabling it gerrit wide. Probably not a bad followup to fungi's All-Project acl cleanups17:27
clarkbsean-k-mooney: you can also set 'priority' and 'back-port'17:27
*** hashar is now known as hasharAway17:27
sean-k-mooneyyes you could17:27
clarkbit gives you a bit of freedom where the voting mechanism is rigid17:27
sean-k-mooneyi would see it like the tags we have in launchpad17:28
sean-k-mooneyi can raise this with the wider nova tema as a possible alternitive implemation or we could use both in paralle once we defien what we want to achive17:29
sean-k-mooneyfor example we coudl have a need-priorty tag and then have core review that or somehting but that17:30
clarkbI think ironic is doing both. I personally think that will just cause confusion but I'm happy for others to use the features how they want17:30
sean-k-mooneyclarkb: does the hashtags feature supprot proptign for existing tags17:32
sean-k-mooneyas you can proably tell if i have to spell thing consitenlty well its proably not going to happen :)17:33
clarkbI'm not sure. I haven't really used it myself yet17:34
clarkbyou can search by them though which is a big thing for most I think17:34
*** dwilde has quit IRC17:34
*** dwilde has joined #openstack-infra17:48
*** lpetrut has joined #openstack-infra17:55
*** lpetrut has quit IRC17:57
*** dwilde has quit IRC18:05
*** dwilde has joined #openstack-infra18:07
*** happyhemant has quit IRC18:23
*** whoami-rajat_ has joined #openstack-infra18:33
*** vishalmanchanda has quit IRC18:34
*** gyee has joined #openstack-infra18:45
*** hamalq has joined #openstack-infra18:50
*** yoctozepto has quit IRC19:13
*** yoctozepto6 has joined #openstack-infra19:13
*** dwilde has quit IRC19:39
*** dwilde has joined #openstack-infra19:57
*** nweinber has quit IRC20:00
*** hamalq has quit IRC20:09
*** rpioso has quit IRC20:09
*** lxkong has quit IRC20:09
*** srwilkers has quit IRC20:09
*** Adri2000 has quit IRC20:09
*** erbarr has quit IRC20:11
*** hamalq has joined #openstack-infra20:13
*** rpioso has joined #openstack-infra20:13
*** lxkong has joined #openstack-infra20:13
*** srwilkers has joined #openstack-infra20:13
*** Adri2000 has joined #openstack-infra20:13
*** erbarr has joined #openstack-infra20:13
*** JanZerebecki[m] has quit IRC20:16
*** dirk has quit IRC20:17
*** wolsen has quit IRC20:17
*** hasharAway has quit IRC20:27
mtreinishclarkb, fungi: do you think it would be possible to get an archive of the subunit2sql db before it's retired?20:29
mtreinishnot that I've been actively doing much dev on subunit2sql lately, but having such a large dataset was useful in the past for testing various things on the project20:29
fungimtreinish: probably, yeah20:29
fungithe real trick is finding scratch space to do the mysqldump20:30
fungi(it's not small)20:30
fungiit's not like we're turning any of this off tomorrow though, just getting the conversation going, setting expectations, working out logistics20:31
*** dirk has joined #openstack-infra20:42
mtreinishI've got plenty of space locally, I remember we looked into this for a db migration a long time ago and can't remember what we ended up doing20:45
mtreinishI seem to remember the trove db has very tight restrictions on connecting to it, I wonder if we can use the readonly proxy with mysqldump20:45
*** JanZerebecki[m] has joined #openstack-infra20:46
fungiprobably?20:46
fungiworth testing that anyway20:47
mtreinishI'll have to reread the man page and give it a go20:47
*** wolsen has joined #openstack-infra20:48
fungimysqldump -h <hostname> -p -u <username> <database> | xz > dump.sql.xz20:48
fungisomething like that should work20:49
fungiit'll prompt interactively for the "password"20:49
mtreinishah, right I remeber this now. It requires more permissions than the basic read we gave the public user:20:51
mtreinishmysqldump: Got error: 1044: "Access denied for user 'query'@'%' to database 'subunit2sql'" when using LOCK TABLES20:52
fungiahh, okay20:52
fungiyou should be able to tell mysqldump not to try to lock?20:52
mtreinishyeah, looks like --skip-lock-tables20:53
mtreinishit seems to be working20:53
fungiright on20:53
*** JanZerebecki[m] has quit IRC20:57
*** dirk has quit IRC21:02
*** wolsen has quit IRC21:03
*** dwilde has quit IRC21:03
*** slaweq has quit IRC21:19
*** JanZerebecki[m] has joined #openstack-infra21:26
*** wolsen has joined #openstack-infra21:29
*** dirk has joined #openstack-infra21:30
*** brtknr has quit IRC21:47
*** brtknr has joined #openstack-infra21:47
*** dhill has quit IRC21:58
*** dhill has joined #openstack-infra22:00
*** rcernin has joined #openstack-infra22:05
*** rcernin has quit IRC22:27
*** rcernin has joined #openstack-infra23:02
*** dave-mccowan has quit IRC23:02
*** rcernin has quit IRC23:02
*** rcernin has joined #openstack-infra23:02
*** whoami-rajat_ is now known as whoami-rajat23:05
*** dave-mccowan has joined #openstack-infra23:06
*** tosky has quit IRC23:24
*** zzzeek has quit IRC23:34
*** zzzeek has joined #openstack-infra23:35
*** zxiiro has quit IRC23:38

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!