Thursday, 2020-10-22

*** ysandeep|away is now known as ysandeep|ruck00:18
*** seongsoocho has quit IRC00:43
*** seongsoocho has joined #opendev00:43
*** hamalq has quit IRC00:44
*** mrunge has quit IRC01:05
*** sgw has left #opendev01:55
*** DSpider has quit IRC02:02
*** ysandeep|ruck is now known as ysandeep|afk02:11
*** mrunge has joined #opendev02:17
*** brinzhang has quit IRC02:31
*** ysandeep|afk is now known as ysandeep05:14
*** ysandeep is now known as ysandeep|ruck05:14
*** whoami-rajat__ has joined #opendev05:16
cgoncalvesianw, hey! is Linaro cloud still under maintenance or is there an issue in Zuul (NODE_FAILURE)?05:46
ianwcgoncalves: yeah, i think there's something up with the API05:52
ianwkevinz: ^05:53
ianwif kevinz doesn't pop up soon i'll see if i can find anyone else05:53
cgoncalvesthank you05:53
cgoncalvesbtw, can one define a multi-arch nodeset? say, 1x x86_64, 1x arm6405:55
cgoncalvesI'd love to test that in Octavia CI05:55
*** rchurch has quit IRC06:01
*** rchurch has joined #opendev06:03
ianwcgoncalves: unfortunately not, it's mostly that nodepool will not schedule across providers.  so the arm64 cloud doesn't have any x86 resources and thus allocation will fail06:06
*** sboyron has joined #opendev06:06
ianwwill not schedule multi-node jobs across providers i mean06:06
cgoncalvesunderstood06:07
ianwif you can find us a cloud that can do both, we could :)06:07
*** lpetrut has joined #opendev06:09
cgoncalvesor if multi-nodes could be scheduled across providers :D06:09
cgoncalvesmaybe mnaser surprises us by announcing arm64 support at vexxhost ;)06:11
*** slaweq has joined #opendev06:21
*** noonedeadpunk has quit IRC06:31
*** noonedeadpunk has joined #opendev06:33
*** noonedeadpunk has quit IRC06:34
*** eolivare has joined #opendev06:35
*** noonedeadpunk has joined #opendev06:39
*** DSpider has joined #opendev06:39
*** mkalcok has joined #opendev06:40
*** slaweq has quit IRC06:44
*** slaweq has joined #opendev06:46
*** roman_g has joined #opendev06:53
*** rpittau|afk is now known as rpittau06:54
*** ralonsoh has joined #opendev07:05
*** DSpider has quit IRC07:21
*** andrewbonney has joined #opendev07:24
*** DjeufackZane has joined #opendev07:25
*** tosky has joined #opendev07:44
*** fressi has joined #opendev07:44
*** guillaumec has quit IRC07:48
*** guillaumec has joined #opendev07:49
*** mfq001_ has joined #opendev08:05
*** mfq001_ has quit IRC08:08
*** ysandeep|ruck is now known as ysandeep|rck|afk08:12
*** ysandeep|rck|afk is now known as ysandep|ruck|afk08:13
*** sshnaidm|afk is now known as sshnaidm|rover08:29
*** DjeufackZane has quit IRC08:33
*** mabuaisha76 has joined #opendev08:36
*** mabuaisha76 has quit IRC08:36
*** mabuaisha has joined #opendev08:39
*** iurygregory has quit IRC08:57
*** jaicaa has quit IRC09:22
*** jaicaa has joined #opendev09:23
*** iurygregory has joined #opendev09:28
*** ysandep|ruck|afk is now known as ysandep|ruck09:28
*** fressi has quit IRC09:43
*** fressi has joined #opendev10:48
rpittauhi all! Related to the recent issue with gerrit, we found two recent identical commits (merged) in python-ironicclient: https://review.opendev.org/#/c/751649/ https://review.opendev.org/#/c/753922/11:02
rpittauis it just a weird coincidence ?11:02
*** fressi has quit IRC11:09
fricklerrpittau: I think that that is indeed a coincidence. I have seen similar patches in other projects, but hopefully managed to flag duplicates most of the time. if you merge both, the second merge commit should essentially come out as a no-op11:14
rpittaufrickler: ok, thanks, what I found awkward is that the second one is actually not empty and the sequence of approvals is exactly the same11:15
fricklerrpittau: you don't see that it is actually empty because of the merge commits. if you do "git rebase -i a5485dc29fe551e4cb5feaad52cd93d67b0ab53e" on the current master, you'll see that the second patch becomes an empty patch that git-rebase complains about11:20
rpittaufrickler: ah, thanks!11:20
fricklerI'm not sure whether we could have a way for gerrit or maybe a zuul job to detect this situation and prevent a merge, maybe someone else has an idea for that11:21
*** marios has joined #opendev11:54
sshnaidm|roverclarkb, fungi hi, folks, we still see today a lot of retry_limits in zuul12:05
sshnaidm|rovercan't even see which cloud it is12:05
sshnaidm|roverysandep|ruck, ^12:05
fungisshnaidm|rover: is it impacting a specific set of jobs? maybe only your integration tests and not your unit tests or static analysis jobs?12:06
sshnaidm|roverfungi, I don't see any specifics, both tripleo and non tripleo, puppet, kolla, others12:07
sshnaidm|roverfor example octavia: https://zuul.opendev.org/t/openstack/status/change/752858,112:08
sshnaidm|rovertripleo: https://zuul.opendev.org/t/openstack/status/change/758937,212:08
sshnaidm|rovercinder: https://zuul.opendev.org/t/openstack/status/change/759234,112:10
*** fressi has joined #opendev12:11
ysandep|rucksshnaidm|rover, ack thanks for raising this12:13
fungilooking at one, ze03 hit a RESULT_UNREACHABLE condition partway through a build in inap-mtl0112:15
fungiaccording to debug logs on the executor where it ran12:15
fungii guess i can repeat that with some others and see if there's a correlation to either ze03 or to inap-mtl0112:15
fungiso far found two which ran from ze03 and one from ze06 but all ran in inap-mtl0112:18
fungihere's one which ran from ze12 but also in inap-mtl0112:19
fungize04 and inap-mtl0112:20
fungiyeah, i guess we need to turn down inap-mtl01 until the connectivity or rogue instance issue there is resolved12:20
fungii'll work on a config patch12:20
openstackgerritJeremy Stanley proposed openstack/project-config master: Temporarily stop booting nodes in inap-mtl01  https://review.opendev.org/75925112:24
fungiconfig-core: ^ that's probably fairly urgent to quell the retry_limit situation sshnaidm|rover mentioned12:24
sshnaidm|roverfungi, thanks!12:25
AJaegerfungi: single-core approved.12:58
fungithanks!13:00
openstackgerritMerged openstack/project-config master: Temporarily stop booting nodes in inap-mtl01  https://review.opendev.org/75925113:08
openstackgerritAndreas Jaeger proposed openstack/project-config master: Remove openstack/monasca-analytics  https://review.opendev.org/75199313:20
*** lpetrut has quit IRC13:22
AJaegerfungi: the deploy job fails:  infra-prod-service-nodepool https://zuul.opendev.org/t/openstack/build/476e200ccee54f21b7ca0221f43bca4b13:34
fungiwe might have a full filesystem on one of the builders again13:37
*** sshnaidm|rover is now known as sshnaidm|afk13:54
*** sgw has joined #opendev14:06
*** marios has quit IRC14:07
*** marios has joined #opendev14:09
*** DSpider has joined #opendev14:14
*** sshnaidm|afk is now known as sshnaidm|rover14:48
*** mlavalle has joined #opendev15:00
*** auristor has quit IRC15:14
*** ysandep|ruck is now known as ysandep|away15:18
*** auristor has joined #opendev15:27
*** sboyron has quit IRC15:28
*** d34dh0r53 has joined #opendev15:29
clarkbdo we need to manually deploy the inap disabling?15:39
clarkbfungi: AJaeger ^ I'm trying to catch up this morning, let me know if I can be helpful by doing that15:39
fungioh, sorry, i got sidetracked by more damage control and comms15:42
fungiclarkb: AJaeger: we should look into the deployment failure, but it does not at least seem to have prevented us from updating the config on nl0315:43
fungimax-servers is 0 there for inap-mtl01 as of 13:24z15:44
clarkbnb03 is network unreachable15:44
clarkbI'm betting that is the ansible error, but haven't checked ansible logs yet15:44
clarkbnb03.opendev.org           : ok=0    changed=0    unreachable=1    failed=0    skipped=0    rescued=0    ignored=015:48
*** DjeufackZane has joined #opendev15:48
clarkbthere are no other failed or unreachables in the summary so I think that is it15:48
AJaegerI see, thanks for checking15:49
*** rpittau is now known as rpittau|afk15:57
*** hamalq has joined #opendev16:07
*** hamalq has quit IRC16:10
*** hamalq has joined #opendev16:10
*** roman_g has quit IRC16:17
openstackgerritJeremy Stanley proposed opendev/system-config master: Add service-incident@opendev mailing list  https://review.opendev.org/75929316:19
*** marios is now known as marios|out16:20
*** valleedelisle has joined #opendev16:37
*** eolivare has quit IRC16:44
*** DjeufackZane has quit IRC16:51
*** ralonsoh has quit IRC17:08
*** mabuaisha has quit IRC17:08
*** valleedelisle has quit IRC17:13
openstackgerritIgor D.C. proposed ttygroup/gertty master: Make browser open on Windows when running in cygwin  https://review.opendev.org/67120217:20
*** mkalcok has quit IRC17:22
*** valleedelisle has joined #opendev17:24
*** andrewbonney has quit IRC17:26
*** sshnaidm|rover is now known as sshnaidm|off17:45
*** ricolin has quit IRC17:54
*** marios|out has quit IRC18:31
johnsomHi opendev team. It appears that the patch -> storyboard linkage is not working.18:54
johnsomExample patch: https://review.opendev.org/#/c/759164/18:54
johnsomExample Story: https://storyboard.openstack.org/#!/story/200826818:54
clarkbjohnsom: fungi reset the api key used for that communication and gerrit won't notice it until we restart it18:54
johnsomPushing the patch isn't updating the status on the story18:54
clarkbfungi: ^ is that restart something you think we should just go ahead and do?18:54
johnsomAh, ok. Just thought I would mention that it was broken.18:55
fungijohnsom: yep, it'll need a gerrit restart, i was hoping to save that for the weekend18:55
johnsomNo real rush IMO18:55
fungiyeah, sorry, we didn't think to clear the storyboard access tokens until after we'd brought gerrit back up, and i was hesitant to take it back down for that even briefly after such a lengthy outage18:56
melwittclarkb: I'm working on the log-gearman-worker and I was wondering, are there urls to get to the gzip or deflate compressed files in the gate to test reading?19:38
*** fressi has quit IRC19:39
clarkbmelwitt: its just the files that you get from zuul's log indexes19:41
clarkbmelwitt: in the past we gzipped then stored them on disk that way but now we use swift and set content encodings and hte web server sorts it out19:42
clarkbas a result that code can probably be simplified to only handle how swift serves the files (which is what you'll see from zuul log links today)19:42
melwittclarkb: oh I see. does "zuul's log indexes" just mean the normal files under logs/ dir "raw" view when looking at a build result?19:45
clarkbya the raw view19:45
melwittok. I tested with a random c-vol file and found it returns encoding as "gzip" but it's just text19:46
clarkbif you do that in your browser it is because the browser is decoding for you19:46
clarkbwhat curl/wget see should be closer to what requests will do19:46
melwittI'm using requests and I get the content type or encoding back as gzip but it's not actually compressed at all19:47
clarkbhuh I wonder if swift is giving it back as plain text if you don't send an accept-encoding gzip19:48
clarkbbut then leaves the encoding as gzip for some reason19:48
melwittis it because of this? https://opendev.org/opendev/puppet-log_processor/src/branch/master/files/log-gearman-worker.py#L32919:48
clarkbya if you set the accept encoding you should get it back as proper gzip iirc19:49
clarkbthen you'll hvae to decode it locally19:49
clarkb(the benefit of this is faster data transfer at cost of cpu time)19:49
melwittoh, hm. yeah when I do that it always fails to read the header because there is no header, it's just a text file. it's possible I'm doing something wrong19:49
clarkbmaybe see if curl does similar?19:49
melwittI'm using requests with stream=True19:50
clarkboh I wonder if the stream=True is magic19:50
clarkbthat could be19:50
melwittyeah I was just thinking it might be19:50
clarkbits very likely the stream implies a decode19:50
melwittyeah, could be. I'll see if I can find out. thanks for the hints19:51
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Add nim roles and job  https://review.opendev.org/74786520:00
timburkeclarkb, fwiw, stock swift will just send back whatever was stored, with no encoding or decoding. iirc, the rax cdn *might* do some transformations based on what headers the client sent?20:00
clarkbthat wouldn't surprise me20:01
fungiyeah, seems more likely that the cdn is making fancy with the responses20:01
fungiwe've seen this in other ways, like when we end up with double compression (gzip and deflate)20:02
melwittI tried looking in the browser development tools and see this for a c-vol.txt:20:12
melwittContent-Encoding: gzip20:13
melwittContent-Length: 33331220:13
melwittContent-Type: text/plain20:13
melwittso encoding is saying gzip but type is saying plain text20:13
clarkbya so the over the wire format was gzp but then the decoded rendering to the user in the browser should be plain text20:13
melwittah ok, I'll try with curl then. sorry I don't know much about this20:15
clarkbmelwitt: the worst one is deflate because its gzip wihtout the header20:16
melwitthehe yeah. I have learned that because of trying to do this :)20:16
melwittyep you called it, requests is doing fanciness20:17
melwittI didn't expect it because I didn't get from the doc that it does any decoding unless you pass decode_unicode=True https://requests.readthedocs.io/en/latest/api/#requests.Response.iter_content and I didn't pass that20:19
melwittwhen I use curl the file is indeed a gzip binary file20:20
melwittah it says it here https://requests.readthedocs.io/en/master/user/quickstart/#raw-response-content mystery solved20:24
melwittkewl20:25
*** slaweq has quit IRC20:26
fungiclarkb: i notice that the 13:00-15:00z ptg session on monday also conflicts with the openstack security sig and the openstack qa team, so that might not be a great time to discuss either openstack tact sig or gerrit security breach topics20:44
clarkbthats the first slot right?20:44
clarkbI'm game to try juggling or adding time to our schedule next week, do you think that would help? like maybe we can lurk those sessions then do ours later or someting (of course that starts to get later for frickler and other EU timezone folks)20:45
fungiyeah, the first slot20:45
fungii mean, alternatively, maybe it's a good opportunity to discuss either (or both) of those topics if they overlap with those other groups agendas and they want to just all jump in the same room for a bit20:46
clarkbya that may be a good approach if they don't mind the change to agenda20:46
clarkbI had planned to do ptg prep things tomorrow, I'll think about it more over the next little while20:47
fungii get that, i'm putting my personal schedule together now because christine is getting tired of cooking all this week and wants to know when i'm available to cook and run errands next week so we can schedule things like grocery pickup20:48
fungiheh, and for the 23:00 utc block monday i'm double-booked with the openinfra labs session20:50
clarkbanother approach would be to schedule new blocks for the gerrit discussion20:50
clarkbgmann: ^ do you have any thoughts about the qa team conflict?20:52
clarkbgmann: would you be interested in sort of lumping together?20:52
fungijust a heads up that i may skip the 05:00-07:00z slot on wednesday, that's, like, i don't even20:59
fungiwe'll see how i feel next week20:59
fungi1-3am local time is not a window of time i'm often conscious20:59
clarkbya I figured you would21:00
clarkbI tried to have ~2 slots that worked for most people and were just reasonable enough for me to do all 321:00
clarkbfungi: any idea who to ask about possible combined conversation monday from the security sig?21:01
clarkbmaybe we do an hour out of the block ocmbined so we can still talk about less overlapping topics21:01
fungiclarkb: gagehugo in #openstack-security is the sig chair, though i can also ask if you want21:03
clarkbif you'r ealready in there that would be great21:03
fungiwould you want to talk about security breach stuff then?21:03
clarkbya I think we could dive right into that since its likely to be the topic at the top of everyones mind21:03
fungii can ask if it's better to do that on an overlap time and combine, or a non-conflicting time21:04
clarkb++21:04
fungii've asked him in #openstack-security but he may be out for the night so may not hear back until tomorrow21:07
clarkbthanks21:07
*** slaweq has joined #opendev21:08
fungihe's in cdt i think so dunno21:09
fungimight still be at the keyboard21:09
fungiclarkb: gagehugo says he's game. which part of that slot would you want to talk about it in? maybe the second hour?21:13
clarkbya probably the second so that we can settle in first21:13
fungiright, it's (obviously) also the security sig's (and everyone else's) first hour of the ptg, so they'll want to do the same i expect21:14
fungisuggested in #openstack-security and gagehugo acknowledged/assented to that plan21:22
clarkbsuddenly I realize there may be a logistical challenge if they intend on using zoom and not meetpad but I think we can sort that out easil enough21:23
*** slaweq has quit IRC21:27
funginah, they can just say to join the opendev team's meetpad and we're good. if anyone can't make it into the meetpad we're the best people to help them, i expect21:30
clarkbgood point21:31
fungithe security sig generally has abominably low turnout for ptg sessions, what with most people not giving a tinker's dam about security until it bites them on the posterior21:32
fungialso let's not forget to scale out jvbs tomorrow ;)21:32
* fungi sets himself a reminder21:33
*** hamalq_ has joined #opendev21:48
*** hamalq has quit IRC21:51
*** odyssey4me has quit IRC21:53
*** odyssey4me has joined #opendev21:53
*** mlavalle has quit IRC22:15
*** qizhangapp has joined #opendev22:15
qizhangappAnyone has trouble to login to https://review.opendev.org/?22:16
clarkbqizhangapp: hello, I was just looking at your email22:17
fungii also just replied to your e-mail, hello!22:17
clarkbfungi is ahead of me :)22:18
fungii included the steps you would usually experience logging into that webpage22:18
fungiclarkb: i had a head start, since i moderated the post through to the ml ;)22:18
clarkbya it would be good to be clear on what specific login action is failing22:18
qizhangappHello! I actually got an error saying "Invalid OpenID transaction" after I click login button22:20
fungithat's definitely strange, i don't think i've experienced that error before22:21
fungiqizhangapp: by "login button" you mean the "Sign In" link on the review.opendev.org page, or the "Yes, log me in" button on the login.ubuntu.com page?22:22
qizhangappSorry. It's the "Sign In" button.22:25
fungiqizhangapp: try going to https://login.ubuntu.com/ directly and try logging out and back in there22:25
qizhangappI got in! I forgot that I enabled cookies blocker earlier today. Now I disabled it and I was able to login. Thank you!22:28
*** DjeufackZane has joined #opendev22:28
fungiqizhangapp: great! i'm glad you got it worked out22:29
fungiand now you know were to find us if you have any other problems ;)22:29
qizhangappyeah:-)22:29
*** qchris has quit IRC22:42
*** tosky has quit IRC22:54
*** qchris has joined #opendev22:55
*** DjeufackZane has quit IRC22:58
*** tbogue has quit IRC23:04
*** qizhangapp has quit IRC23:11
gmannclarkb: fungi QA slots are until 15 UTC to have Asia side TZ, can we do from 15-16UTC ?23:24
clarkbgmann: I know I can but not sure abut everyone else. I think we can run long if we need to23:24
fungithe d&i wg discussions pick up at 15:00z so i was hoping to help with those23:28
gmannclarkb: fungi how much time you think those topic will need ? like 30 min? or full1 hr23:28
gmannfull 1 hr23:28
clarkbgmann: I imagine we could talk about it for days :) but time with other group swould likely be for their benefit23:29
fungiyeah, like is there some qa-specific facet of it we should discuss?23:29
gmann:), i was thinking last 30 min. 14.30 UTC onwards we can merge QA room into that..23:30
clarkbyou had mentioned them as a conflict so I thought maybe? but ya its possible the overlap there is less strong23:30
fungii have a feeling for opendev it's going to be a lot of answering questions about the situation and what future mitigations we're entertaining (which we'll probably get into deeper detail discussing later in the week)23:30
fungii mostly mentioned the qa team as a major component of openstack's tact sig, alongside many of the opendev contributors23:31
clarkbgotcha23:31
fungiso if we have tact sig related discussions we should maybe either give the qa team a chance to sync up with us in an overlapping period, or defer those to a session when there's no qa team conflict so they can join23:32
gmannyeah, actually we tried to move QA one later in days like after Wed or so but that did not work out as other projects sessions23:32
gmannok23:32
clarkbI guess this is the downside to picking timeslots super early :) you have to rely on others to avoid conflicts rather than the other way around23:32
fungibasically, if we're going to talk about anything of interest to the qa team, we need to make sure we're not doing it at a time when they plan to be talking about something else in another room ;)23:32
fungiwhether that means they join us for part of their conflicting session or we get it into a different part of our agenda at one of our times which don't overlap with theirs23:33
clarkbfrom our original ptg planning doc I don't think we had a ton of topics that overlapped iwth ci23:35
clarkbthey tended to be more service oriented iirc23:35
clarkbbut I haven't looked at it in like a week23:35
gmannseems like on Monday later one at 23.00 UTC i have to run policy-popup at the same time at opendev otherwise that time we could have jump in that time23:37
gmann14.30 UTC onwards we can definitly23:38
gmannmerge23:38
clarkbok maybe lets play it by ear and see how we do. Did the qa team have any specifc things to bring up with us?23:39
clarkb(as I mentioned I think the list of topics I wrote down didn't have significant overlap, but more than happy for people to add things to the list)23:39
gmannok. that works too. I do not find any specific topic for now but if it comes up we can jump in or talk at runtime23:40
gmannone is distro testing one which i added in TC slot there it will be good idea to have QA and Tact SIG for discussion23:42
gmannL38 - https://etherpad.opendev.org/p/tc-wallaby-ptg23:42
fungii've added all the openstack tc slots to my schedule and plan to catch them if possible23:43
gmannnice23:43
gmannfrom QA side i did not see any specific thing for now23:43
clarkbyoctozepto's proposed time for that should work for me too23:43
gmann+123:44
gmannnot sure when mnaser is planning to schedule topic wise time but Friday is good time for that.23:45

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!