Wednesday, 2020-10-28

*** zzzeek has quit IRC00:16
*** zzzeek has joined #openstack-infra00:17
*** hamalq has quit IRC00:26
*** yamamoto has quit IRC00:38
*** yamamoto has joined #openstack-infra00:39
*** yamamoto has quit IRC00:44
*** yamamoto has joined #openstack-infra00:44
*** Goneri has quit IRC01:03
*** yamamoto has quit IRC01:08
*** yamamoto has joined #openstack-infra01:45
*** yamamoto has quit IRC01:57
*** stevebaker has quit IRC02:18
*** stevebaker has joined #openstack-infra02:45
*** gyee has quit IRC02:57
*** yamamoto has joined #openstack-infra03:08
*** tkajinam has quit IRC03:10
*** stevebaker has quit IRC03:21
*** lamt has quit IRC03:22
*** arne_wiebalck has quit IRC03:22
*** arne_wiebalck has joined #openstack-infra03:23
*** yonglihe has quit IRC03:23
*** yonglihe has joined #openstack-infra03:25
*** yamamoto has quit IRC03:35
*** psachin has joined #openstack-infra03:36
*** psachin has joined #openstack-infra03:37
*** rcernin has quit IRC03:48
*** rcernin has joined #openstack-infra03:53
*** yamamoto has joined #openstack-infra04:13
*** sean-k-mooney has quit IRC04:16
*** sean-k-mooney has joined #openstack-infra04:16
*** stevebaker has joined #openstack-infra04:21
*** ykarel has joined #openstack-infra04:24
*** jamesdenton has quit IRC05:23
*** jamesdenton has joined #openstack-infra05:24
*** matt_kosut has joined #openstack-infra05:26
*** evrardjp has quit IRC05:33
*** evrardjp has joined #openstack-infra05:33
*** openstackgerrit has quit IRC05:46
*** ricolin_ has joined #openstack-infra05:56
*** ricolin_ has quit IRC05:58
*** ricolin has joined #openstack-infra05:59
*** ysandeep|away is now known as ysandeep|ruck06:08
*** dklyle has quit IRC06:32
*** lmiccini has joined #openstack-infra06:45
*** tkajinam has joined #openstack-infra06:53
*** yamamoto has quit IRC06:59
*** sshnaidm|afk is now known as sshnaidm|rover07:00
*** eolivare has joined #openstack-infra07:22
*** rcernin has quit IRC07:30
*** sboyron has joined #openstack-infra07:30
*** yamamoto has joined #openstack-infra07:39
*** ykarel has quit IRC07:44
*** ralonsoh has joined #openstack-infra07:45
*** ykarel has joined #openstack-infra07:45
*** rpittau|afk is now known as rpittau07:51
*** yamamoto has quit IRC07:53
*** rcernin has joined #openstack-infra07:53
*** slaweq has joined #openstack-infra08:00
*** jcapitao has joined #openstack-infra08:02
*** lmiccini has quit IRC08:03
*** hberaud has joined #openstack-infra08:07
*** andrewbonney has joined #openstack-infra08:10
*** lpetrut has joined #openstack-infra08:11
*** rcernin has quit IRC08:15
*** lmiccini has joined #openstack-infra08:17
*** yolanda has joined #openstack-infra08:19
*** ociuhandu has joined #openstack-infra08:38
*** hashar has joined #openstack-infra08:42
*** ociuhandu has quit IRC08:48
*** ykarel_ has joined #openstack-infra08:49
*** ykarel has quit IRC08:52
*** jpena|off is now known as jpena08:54
*** tosky has joined #openstack-infra08:57
*** rchanter has joined #openstack-infra08:58
*** ysandeep|ruck is now known as ysandeep|lunch09:00
*** krah has quit IRC09:04
*** yamamoto has joined #openstack-infra09:07
*** lucasagomes has joined #openstack-infra09:12
*** ociuhandu has joined #openstack-infra09:21
*** openstack has quit IRC09:21
*** openstack has joined #openstack-infra09:22
*** ChanServ sets mode: +o openstack09:22
*** yamamoto has quit IRC09:23
*** gfidente has joined #openstack-infra09:24
*** logan- has quit IRC09:27
*** logan- has joined #openstack-infra09:27
*** lajoskatona has joined #openstack-infra09:28
lajoskatonaHi, I need some help regarding networking-l2gw, which was recently moved to x/networking-l2gw09:29
lajoskatonaThe pypi owner is still openstackci, is there a way to change that?09:30
*** rcernin has joined #openstack-infra09:36
AJaegerlajoskatona: openstackci needs to be one owner so that we can push to it when you tag a release.09:39
AJaegerlajoskatona: for what do you need access?09:39
lajoskatonaAJaeger: I added a tag (17.0.0) to x/networking-l2gw, but my assumption was that it has no affect for projects in x/09:42
lajoskatonaAJaeger: I mean no automatic push will happen09:42
*** yamamoto has joined #openstack-infra09:43
AJaegerlajoskatona: the push will happen if you configured it correctly09:45
AJaegerlet me check...09:45
*** ysandeep|lunch is now known as ysandeep|ruck09:45
lajoskatonaAJaeger: thanks09:45
AJaeger publish-to-pypi should be part of your job setup - but is not. Send a patch for project-config to add it.09:46
AJaegerlajoskatona: one that is in, push 17.0.1 and have it appear automatically on pypi09:48
*** rchanter has quit IRC09:50
*** ykarel_ is now known as ykarel09:51
lajoskatonaAJaeger: Add to https://opendev.org/openstack/project-config/src/branch/master/zuul.d/projects.yaml as I see, am I right<09:53
lajoskatona?09:53
AJaegerlajoskatona: yes!09:54
lajoskatonaAJaeger: thanks09:54
*** rcernin has quit IRC09:56
*** yamamoto has quit IRC09:58
*** openstackgerrit has joined #openstack-infra09:59
openstackgerritLajos Katona proposed openstack/project-config master: Add publish-to-pypi template to networking-l2gw  https://review.opendev.org/76009609:59
AJaegerlajoskatona: only publish-to-pypi, not the other template10:01
AJaegerYou have already openstack-python3-victoria-jobs-neutron in repo and don't need that10:01
openstackgerritLajos Katona proposed openstack/project-config master: Add publish-to-pypi template to networking-l2gw  https://review.opendev.org/76009610:03
lajoskatonaAJaeger: ok, I updated10:05
*** yamamoto has joined #openstack-infra10:09
*** yamamoto has quit IRC10:09
*** psachin has quit IRC10:18
*** psachin has joined #openstack-infra10:27
*** ysandeep|ruck is now known as ysandeep|brb10:31
AJaegerLGTM now. config-core, please review ^10:36
*** psachin has quit IRC10:45
*** psachin has joined #openstack-infra10:47
*** psachin has quit IRC10:49
*** psachin has joined #openstack-infra10:50
*** slaweq has quit IRC10:51
*** ysandeep|brb is now known as ysandeep|ruck10:54
*** slaweq has joined #openstack-infra10:55
*** rcernin has joined #openstack-infra11:16
*** lpetrut has quit IRC11:25
*** lpetrut has joined #openstack-infra11:26
*** lpetrut has quit IRC11:26
*** lpetrut has joined #openstack-infra11:27
*** another_larsks has joined #openstack-infra11:29
*** larsks has quit IRC11:31
*** yamamoto has joined #openstack-infra11:31
*** jcapitao is now known as jcapitao_lunch11:31
* kmarc is away: "Auto-away"11:38
*** dmellado has quit IRC11:38
*** dmellado has joined #openstack-infra11:39
*** ykarel has quit IRC11:47
*** ykarel has joined #openstack-infra11:49
*** zzzeek has quit IRC11:49
*** zzzeek has joined #openstack-infra11:52
*** yamamoto has quit IRC11:56
* kmarc is back (gone 00:28:21)12:06
*** rlandy has joined #openstack-infra12:07
* kmarc is away: "Auto-away"12:16
*** ykarel_ has joined #openstack-infra12:16
*** eolivare has quit IRC12:21
*** jpena is now known as jpena|lunch12:25
*** jcapitao_lunch is now known as jcapitao12:27
*** yamamoto has joined #openstack-infra12:31
*** rfolco has joined #openstack-infra12:34
*** lajoskatona has left #openstack-infra12:38
fricklerkmarc: could you please disable your auto-away stuff for all openstack channels?12:40
* kmarc sorry, checking...12:43
*** rcernin has quit IRC12:43
kmarcfrickler: disabled12:45
*** yamamoto has quit IRC12:47
*** hashar has quit IRC12:50
*** hashar has joined #openstack-infra12:56
fricklerkmarc: thanks13:00
openstackgerritMerged openstack/project-config master: Use internal address for RAX ORD  https://review.opendev.org/75997213:10
*** thiago__ has quit IRC13:12
*** tdasilva_ has joined #openstack-infra13:13
*** tdasilva_ has quit IRC13:14
*** tdasilva has joined #openstack-infra13:14
openstackgerritMerged openstack/project-config master: Add publish-to-pypi template to networking-l2gw  https://review.opendev.org/76009613:17
*** lmiccini has quit IRC13:18
*** eolivare has joined #openstack-infra13:24
*** sshnaidm|rover has quit IRC13:25
*** jpena|lunch is now known as jpena13:27
*** ykarel_ has quit IRC13:31
*** sshnaidm|rover has joined #openstack-infra13:39
*** psachin has quit IRC13:40
*** sshnaidm|rover is now known as sshnaidm|mtg13:44
*** lmiccini has joined #openstack-infra14:02
*** d34dh0r53 has quit IRC14:07
*** d34dh0r53 has joined #openstack-infra14:11
*** thiagop has joined #openstack-infra14:20
*** dklyle has joined #openstack-infra14:24
openstackgerritThierry Carrez proposed openstack/ptgbot master: No longer display location  https://review.opendev.org/76015114:33
*** yamamoto has joined #openstack-infra14:47
*** Goneri has joined #openstack-infra14:48
*** Goneri has quit IRC14:52
*** psachin has joined #openstack-infra14:54
*** dklyle has quit IRC14:58
openstackgerritMerged openstack/ptgbot master: No longer display location  https://review.opendev.org/76015115:01
*** hberaud has quit IRC15:02
*** yamamoto has quit IRC15:06
*** ykarel is now known as ykarel|away15:07
*** dklyle has joined #openstack-infra15:08
*** lpetrut has quit IRC15:19
*** gyee has joined #openstack-infra15:24
*** ykarel|away has quit IRC15:41
*** hberaud has joined #openstack-infra15:45
*** takamatsu has quit IRC15:45
*** tdasilva_ has joined #openstack-infra15:46
*** hberaud has quit IRC15:47
*** hberaud has joined #openstack-infra15:47
*** tdasilva has quit IRC15:49
*** takamatsu has joined #openstack-infra15:56
*** sauloasilva1 has joined #openstack-infra16:05
*** sauloasilva1 has quit IRC16:05
*** sauloasilva1 has joined #openstack-infra16:06
*** chandankumar is now known as raukadah16:23
*** ysandeep|ruck is now known as ysandeep|away16:29
dtantsurseeing plenty of retry limits because of mirror resolution problems. is it known/temporary?16:31
dtantsurCould not resolve 'mirror.ord-int.rax.opendev.org'16:32
dtantsurexample: https://d777cc90efd99de28a0e-63e0619ac9734be8bebe39a119169a2b.ssl.cf2.rackcdn.com/744434/23/check/bifrost-integration-redfish-uefi-ubuntu-focal/fbfd63e/job-output.txt16:32
clarkbthat isnt a known issue but a bug in our deployment if using the internal IP16:33
fricklerna, that's a bug16:33
clarkbshould be mirror-int.ord.rax.opendev.org I think16:33
*** lmiccini has quit IRC16:33
clarkbfrickler: do you know if there is a fix for that yet?16:33
fricklerclarkb: there is no fix, I'll put something up16:33
clarkbthanks16:33
openstackgerritJens Harbott (frickler) proposed openstack/project-config master: Fix the internal rax mirror name  https://review.opendev.org/76018316:36
fricklerinfra-root: ^^ I'm off now, feel free to amend if it turns out I did something wrong. maybe someone wants to deploy this before it merges, too16:38
clarkbhrm digging that name doesn't return a record either16:40
clarkbfungi: any idea what the record is?16:40
*** lucasagomes has quit IRC16:41
clarkbfrickler: fungi if I lookup the names against ns1 and ns2 they resolve. However, the cname points to the public ip not private ip which https://review.opendev.org/#/c/759974/ should've fixed16:44
clarkbit seems we're still having problems syncing the zone/16:45
fungimirror-int.ord.rax.opendev.org is an alias for mirror01-int.ord.rax.opendev.org.16:45
fungimirror01-int.ord.rax.opendev.org has address 10.209.128.5716:45
fungithat looks right to me16:45
fungiresult of running `host mirror-int.ord.rax.opendev.org`16:45
clarkbugh ya I'm tired I dug the public name and it gave me the public ip back :/16:46
clarkbstill not sure why it isn't resolving against my local dns though16:47
clarkboh I wonder if I'm filtering 10 net addresses16:47
clarkbthat would be the most amazingly awesome self fail win if so16:47
fungigranted that's going to take a few minutes to land and roll out... should we manually turn down rax-ord for a bit?16:48
clarkbwfm if you want to do that16:49
*** hamalq has joined #openstack-infra16:51
fungijust did16:52
fungido i need to reload nodepool after editing the config or will it reread it? i want to say it rereads between operations16:52
clarkbit rereads it between passes thorugh its operating loop16:52
fungii also temporarily put nl01 in the emergency disable list16:53
openstackgerritMerged openstack/project-config master: Fix the internal rax mirror name  https://review.opendev.org/76018316:55
clarkbthere is another issue with the ord mirror. We haven't managed to issue a new cert with the new alt name16:55
AJaegerclarkb: didn't https://review.opendev.org/#/c/759971/ issued the new certs?16:56
fungiit raced on dns replication seemed like16:56
clarkbya it failed and we think https://review.opendev.org/#/c/759988/ is the fix16:56
clarkbI thin kwe may need to rerun the LE playbook and it will correct itself?16:56
fungiand i guess it doesn't get retried unless the cert is at risk of expiring16:56
AJaegerclarkb, fungi, please double check https://review.opendev.org/#/c/759971/1/inventory/service/host_vars/mirror01.ord.rax.opendev.org.yaml - I think there's an error in it16:57
clarkboh ya lets land a fix for that then it should go through normal processing16:58
clarkbwe want mirror-int.org there not a duplicate16:58
clarkbI'll write that change16:58
fungialso we were already down inap-mtl01, so we're going to be struggling on capacity with our largest rax region out too. anybody have bandwidth to try to run down the disconnects in inap, or just want to turn it back on and hope for the best?16:59
clarkbI don't think there is much we can do to debug that on our side, iirc we've seen that before in inap and its a neutron problem with arp fights17:01
clarkbwe can turn it on and hope that neutron is happier there now and if it isn't something something dogfooding17:01
fungiyeah, was more wondering if anybody had logstash queries we could use to determine if the problem was recurring, but i guess with the current state of the logstash cluster that's probably not going to get us timely feedback17:02
clarkbwe could restart everything on the logstash side then we'd get data until it degraded17:02
clarkbwhich may be good enough for that particular use case17:03
*** ociuhandu has quit IRC17:03
fungito find it last time i basically had to pull up inventories for builds i saw in retry_limit state to determine where they ran17:03
fungiand it seemed to be 100% inap based on a sample of ~10 failed builds17:03
fungibut it was hardly scientific17:04
*** yamamoto has joined #openstack-infra17:05
*** rpittau is now known as rpittau|afk17:09
*** dtantsur is now known as dtantsur|afk17:10
openstackgerritMatthias Runge proposed openstack/project-config master: Create telemetry group and include  https://review.opendev.org/76019017:16
*** tosky has quit IRC17:20
*** yamamoto has quit IRC17:22
*** eolivare has quit IRC17:25
*** jcapitao has quit IRC17:31
*** hberaud has quit IRC17:43
*** sshnaidm|mtg is now known as sshnaidm|rover17:46
*** ykarel|away has joined #openstack-infra17:47
*** thiagop has quit IRC17:47
*** thiagop has joined #openstack-infra17:50
*** andrewbonney has quit IRC17:57
*** ykarel|away has quit IRC17:59
*** ralonsoh has quit IRC18:00
*** jamesmcarthur has joined #openstack-infra18:01
*** jpena is now known as jpena|off18:05
*** psachin has quit IRC18:22
*** hashar has quit IRC18:28
*** jamesmcarthur has quit IRC18:29
*** gfidente is now known as gfidente|afk18:31
*** psyton has quit IRC18:47
openstackgerritMatthias Runge proposed openstack/project-config master: Create telemetry group and include  https://review.opendev.org/76019018:52
*** sshnaidm_ has joined #openstack-infra18:57
*** sshnaidm|rover has quit IRC19:00
*** sshnaidm_ is now known as sshnaidm|rover19:06
*** kwazar has quit IRC19:17
*** ianychoi__ has joined #openstack-infra19:17
*** mtreinish has quit IRC19:19
*** mtreinish has joined #openstack-infra19:20
*** ianychoi_ has quit IRC19:21
*** yamamoto has joined #openstack-infra19:22
*** sshnaidm|rover is now known as sshnaidm|afk19:28
*** mtreinish has quit IRC19:29
*** mtreinish has joined #openstack-infra19:29
*** hashar has joined #openstack-infra19:39
*** jamesmcarthur has joined #openstack-infra19:40
*** hberaud has joined #openstack-infra19:46
*** yamamoto has quit IRC19:46
openstackgerritJeremy Stanley proposed openstack/project-config master: Revert "Temporarily stop booting nodes in inap-mtl01"  https://review.opendev.org/76022019:55
*** vesper has quit IRC20:02
openstackgerritDonny Davis proposed openstack/project-config master: Disable OE, edge router is about to buy the farm  https://review.opendev.org/76022720:12
*** vesper11 has joined #openstack-infra20:21
*** piotrowskim has quit IRC20:23
*** yolanda has quit IRC20:24
*** yolanda has joined #openstack-infra20:25
*** vesper has joined #openstack-infra20:27
*** another_larsks is now known as larsks20:27
*** vesper11 has quit IRC20:30
*** tosky has joined #openstack-infra20:39
*** vesper11 has joined #openstack-infra20:48
*** vesper has quit IRC20:48
openstackgerritMerged openstack/project-config master: Disable OE, edge router is about to buy the farm  https://review.opendev.org/76022720:56
*** gfidente|afk is now known as gfidente21:04
*** rfolco has quit IRC21:06
*** jamesmcarthur has quit IRC21:26
*** yamamoto has joined #openstack-infra21:44
*** slaweq has quit IRC21:48
*** rfolco has joined #openstack-infra21:57
*** dansmith has quit IRC21:59
*** rm_work has quit IRC21:59
*** jrosser has quit IRC21:59
*** zaro has quit IRC21:59
*** zer0c00l has quit IRC21:59
*** zer0c00l has joined #openstack-infra21:59
*** rm_work has joined #openstack-infra21:59
*** jrosser has joined #openstack-infra21:59
*** zaro has joined #openstack-infra21:59
*** dansmith has joined #openstack-infra21:59
*** yamamoto has quit IRC22:01
*** rfolco has quit IRC22:03
*** slaweq has joined #openstack-infra22:04
*** dychen has joined #openstack-infra22:04
*** dchen has quit IRC22:06
*** slaweq has quit IRC22:14
*** matt_kosut has quit IRC22:14
*** yamamoto has joined #openstack-infra22:14
*** matt_kosut has joined #openstack-infra22:14
*** matt_kosut has quit IRC22:19
*** hberaud has quit IRC22:26
*** rfolco has joined #openstack-infra22:30
*** rlandy is now known as rlandy|bbl22:33
*** rfolco has quit IRC22:35
*** rcernin has joined #openstack-infra22:46
*** gfidente has quit IRC22:48
*** thiagop has quit IRC22:59
*** ociuhandu has joined #openstack-infra23:05
*** ociuhandu has quit IRC23:09
*** sboyron has quit IRC23:10
*** hashar has quit IRC23:28

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!