Wednesday, 2020-07-15

*** manfly000 has joined #opendev00:12
*** ryohayakawa has joined #opendev00:13
*** manfly000 has quit IRC00:15
*** sgw1 has quit IRC02:23
*** sgw1 has joined #opendev02:54
*** redrobot has quit IRC03:36
*** sgw1 has quit IRC04:04
*** sgw1 has joined #opendev04:06
*** ysandeep|away is now known as ysandeep|rover04:32
*** sgw1 has quit IRC04:48
*** sgw1 has joined #opendev04:52
*** marios has joined #opendev05:32
*** osmanlicilegi has quit IRC06:02
*** zbr|ruck has quit IRC07:02
*** zbr|ruck has joined #opendev07:03
*** zbr has quit IRC07:06
*** DSpider has joined #opendev07:13
ysandeep|roverHey o/ zbr|ruck c7 train promotion is failing - http://38.102.83.109/centos7_train.log , it seems failing for "neutron-metadata-agent-ovn" - stderr_lines": ["unexpected end of JSON input", "unexpected end of JSON input"] , if you have pointers on this please let me know.07:40
ysandeep|roverchandankumar, ^^07:40
*** zbr has joined #opendev07:42
*** dtantsur|afk is now known as dtantsur07:45
*** tosky has joined #opendev07:46
*** elod_off is now known as elod07:51
*** moppy has quit IRC08:01
*** moppy has joined #opendev08:03
AJaegerysandeep|rover: are you sure you're in the right channel?08:04
ysandeep|roverAJaeger, oops sry my channel sort order changed08:05
AJaeger;)08:11
*** ysandeep|rover has left #opendev08:22
*** dmsimard has quit IRC08:25
*** dmsimard has joined #opendev08:26
*** ysandeep|rover has joined #opendev08:29
*** bhagyashris is now known as bhagyashris|lunc08:43
*** bbezak has joined #opendev08:50
*** ysandeep|rover is now known as ysandeep|lunch08:57
*** dougsz has joined #opendev09:04
dougszMore than 8GB RAM on a Zuul worker, is that possible?09:05
*** ysandeep|lunch is now known as ysandeep09:17
*** ysandeep is now known as ysandeep|rover09:24
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Deprecate default tox_envlist: venv  https://review.opendev.org/72683009:32
*** ryohayakawa has quit IRC10:07
*** sshnaidm is now known as sshnaidm|afk10:24
*** bhagyashris|lunc is now known as bhagyashris10:37
*** roman_g has joined #opendev10:43
*** tosky has quit IRC11:05
*** tosky has joined #opendev11:10
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Deprecate default tox_envlist: venv  https://review.opendev.org/72683011:11
fungidougsz: you can create a swap partition or swapfile11:46
fungidougsz: or you can use multiple nodes11:46
fungiassuming you're asking about opendev's deployment of zuul, and not the zuul project generally11:47
fungiwe document the general limitations of our test nodes at https://docs.opendev.org/opendev/infra-manual/latest/testing.html11:48
fungiwe had two providers supplying a limited number of 16gb flavor instances but one of them is offline (possibly for months still) and the other is having trouble placing them so backlogged and returning frequent node error results11:50
fungiso i would not consider them generally available11:50
dougszfungi: thanks - That's very helpful. It is indeed for opendev zuul. Trying the more nodes approach, but swap and fine tuning may also be feasible.12:11
*** marios has quit IRC12:14
fungidougsz: also it bears pointing out that keeping your tests runnable in under an 8gb footprint makes it easier for devs to be able to run them on their local systems too12:18
mordredinfra-root: could I please get review on https://review.opendev.org/#/c/737023/ ? gerrit upgrade testing is blocked pending review of that patch12:20
*** tkajinam has quit IRC12:24
mordredfungi: thanks!12:33
*** vapjes has joined #opendev12:35
openstackgerritMerged opendev/system-config master: Stop cloning k8s-on-openstack  https://review.opendev.org/74095612:39
dougszfungi: Agreed - the stack is very big in this case (Monasca), but could make sense to split into two smaller scenarios12:47
*** sshnaidm|afk is now known as sshnaidm12:53
*** marios has joined #opendev12:56
*** ysandeep|rover is now known as ysandeep|away13:10
*** redrobot has joined #opendev13:48
openstackgerritThierry Carrez proposed openstack/project-config master: More helpful description for openstack/openstack  https://review.opendev.org/74121314:00
openstackgerritMerged openstack/project-config master: More helpful description for openstack/openstack  https://review.opendev.org/74121314:28
*** mlavalle has joined #opendev14:32
fungi#status log deleted configuration for openstack-infra mailing list for the lists.openstack.org site, leaving archives intact15:01
openstackstatusfungi: finished logging15:01
openstackgerritMerged opendev/system-config master: Forward openstack-infra ML to openstack-discuss  https://review.opendev.org/73915215:22
openstackgerritVladimir Blando proposed openstack/project-config master: added 2 vexxhost repos  https://review.opendev.org/74124715:27
corvusmordred: lgtm; did not +w since i'm assuming that hasn't actually run for realz yet and you may want to watch the first one15:34
mordredcorvus: ++ thanks!15:35
openstackgerritVladimir Blando proposed openstack/project-config master: added 2 vexxhost repos  https://review.opendev.org/74124715:41
clarkbinfra-root if there are no objections I'll approve the gitea v1.12.2 upgrade in a couple minutes15:48
mordredclarkb: do it15:49
fungiplease go for it15:50
openstackgerritClark Boylan proposed opendev/system-config master: Allow setting Gitea repo branch on project creation  https://review.opendev.org/74126416:00
*** shtepanie has joined #opendev16:00
clarkbthats sort of the natural followon to the gitea upgrade but I've added notes in the cahnge it shouldn't be used until gerrit/jeepyb do similar16:00
*** marios is now known as marios|out16:00
clarkb(I'm  not sure I'll have time for the gerrit side in the immediate future)16:00
clarkbbut I figure one step at a time and eventually we'll get there16:02
*** marios|out has quit IRC16:03
clarkbcorvus: mordred now that the zuul registry fix has landed (and should have been applied to intermediate registry) is the next step for us something to do with python-base?16:07
clarkbneed any reviews there or?16:07
openstackgerritMerged zuul/zuul-jobs master: upload-afs-synchronize: expand documentation  https://review.opendev.org/74105116:07
corvusclarkb: oh, let me just re-update that change; i'll do that now16:08
corvusi think we just recheck that a bit16:08
openstackgerritJames E. Blair proposed opendev/system-config master: Build multi-arch python-base/python-builder  https://review.opendev.org/72626316:09
clarkbthe jeepyb side of things is actually not too bad, need a small pdate to gerritlib then an update to jeepyb and we should be able to support that default-branch flag (create-project does what we need so don't have to approximate it with other commands)16:10
corvusclarkb, mordred, ianw: ^ that's back to PS4 (the "real" version); so we recheck that a few times, make sure nothing is weird, then that's ready to merge16:10
clarkbI may be able to write that when I'm not distracted by gitea16:10
clarkbcorvus: k16:10
mordred++16:10
mordredthen once that has landed we can land the nodepool change16:11
*** vblando has joined #opendev16:12
Open10K8SHi team16:12
Open10K8SPlease check this PS.16:12
Open10K8Shttps://review.opendev.org/#/c/74124716:12
Open10K8Swaiting for review :)16:12
mordredoh - actually - we already think we build multi-arch for nodepool.16:13
corvusOpen10K8S: happy to!  you can probably put that all on one line in irc in the future :)16:13
clarkbmordred: ya and it works for x86 because the mixup would go in that direction16:13
Open10K8Scorvus: ok16:13
clarkbmordred: but on the arm side I would expect it to fail16:13
mordredcorvus: https://review.opendev.org/#/c/726477/ is the stack of thinking I had for nodepool about how we could run dib in arm on an x86 nodepool-builder instance16:13
mordredclarkb: ^^16:14
corvusmnaser: i'll leave 741247 for you to +316:14
clarkbmnaser: corvus: Open10K8S that may be a good sanity check after gitea is upgraded16:14
clarkbif we can hold off for ~however long the upgrade will take (an hour maybe?)16:14
corvusclarkb: that sounds reasonable; maybe put a WIP-1 on it?16:14
corvus(i'm going to be afk shopping in 1 hr, so i'll abstain)16:15
clarkbdone I've WIP'd16:15
Open10K8Sclarkb: ok16:15
mnasercorvus, clarkb: +2'd from my side that it's ok from a vexhost-tenant pov and will wait for the gitea upgrade16:16
openstackgerritMonty Taylor proposed zuul/zuul-jobs master: Add a job for publishing a site to netlify  https://review.opendev.org/73904716:18
clarkbmnaser: Open10K8S thanks for the patience16:18
clarkbI can approve it once the gitea backends are done since I'm watching those anyway16:18
clarkbchange is currently in the gate so should happen soon enough16:19
Open10K8Sclarkb: ok, thank you16:19
openstackgerritTobias Urdin proposed opendev/system-config master: Mirror Ceph Octopus for Debian based  https://review.opendev.org/74126916:32
clarkbmordred: can you check my comment on https://review.opendev.org/#/c/741264/1 thats me thinking out loud a bit but since you've reviewed it your thoughts would be appreciated16:33
mordredclarkb: oh - I think that's a non-terrible idea16:34
*** dougsz has quit IRC16:34
clarkblooking at the jeepyb side is what made me think of that because we've got ~3 tools here that could all change at different rates but we want them in sync16:34
clarkbso relying on various tool defaults may not be explicit enough for us16:34
*** osmanlicilegi has joined #opendev16:35
mordredyah16:43
openstackgerritMerged opendev/system-config master: Update to gitea v1.12.2  https://review.opendev.org/74071617:05
clarkbgitea image has been promoted. Now waiting for the deploy job to run17:08
clarkbhttps://gitea01.opendev.org:3000/ is upgraded17:14
clarkbI'm looking at repos now17:14
clarkbit seems to be working and happy17:16
clarkb02 is done, 03 is in progress. Should be done in just a few minutes17:16
*** dtantsur is now known as dtantsur|afk17:18
openstackgerritClark Boylan proposed opendev/system-config master: Allow setting Gitea repo branch on project creation  https://review.opendev.org/74126417:24
openstackgerritClark Boylan proposed opendev/gerritlib master: Add branches arg to createProject  https://review.opendev.org/74127717:26
openstackgerritClark Boylan proposed opendev/jeepyb master: Set repo HEAD on gerrit project creation  https://review.opendev.org/74127917:27
clarkbI expect there to be many unexpected issues with those threee changes :)17:27
clarkbGiteas are all done upgrading and each one reports the expected version to me via the web ui17:28
clarkbI'll approve the vexxhost repos additions from Open10K8S now17:28
openstackgerritDaniel Pawlik proposed opendev/system-config master: Added development/rawhide image  https://review.opendev.org/74128717:35
openstackgerritGuillaume Chauvel proposed opendev/gear master: Create SSL context using PROTOCOL_TLS: highest supported version  https://review.opendev.org/74128817:36
openstackgerritMerged openstack/project-config master: added 2 vexxhost repos  https://review.opendev.org/74124717:42
clarkbinfra-root related to the branch stuff above and the planned gerrit upgrade we need to sunset gerrit's /p/ mirror17:49
clarkba `grep '/p/' gerrit-ssl-access.log | cut -d' ' -f 1 | sort -u | wc -l` says we have 20 ish IPs hitting that path today17:50
clarkbif I can identify people responsible for those I figure we can reach out direclty to them and point them at opendev.org instead? and also send an announcement of this change to service-announce. THen we can either let the upgrade disable it or disable it in apache vhost early depending on timing17:51
clarkbLooks like at least half or so are third party CI systems17:55
clarkbmost for cinder but also oen or two for neutron likely17:55
clarkbthen the others are harder to identify17:55
openstackgerritClark Boylan proposed opendev/gerritlib master: Add branches arg to createProject  https://review.opendev.org/74127717:59
*** qchris has quit IRC18:09
openstackgerritClark Boylan proposed opendev/gerritlib master: Add branches arg to createProject  https://review.opendev.org/74127718:12
*** dougsz has joined #opendev18:13
clarkbOpen10K8S: looks like your new repos were created successfully (it was a minor upgrade for gitea but wanted to confirm)18:15
Open10K8Sclarkb: yeah, surely18:15
Open10K8Sclarkb: thank you18:15
*** qchris has joined #opendev18:21
*** tosky has quit IRC18:25
openstackgerritTobias Urdin proposed opendev/system-config master: Mirror Puppetlabs puppet for Ubuntu Focal  https://review.opendev.org/74129918:25
*** tosky has joined #opendev18:25
*** dougsz has quit IRC18:34
*** roman_g has quit IRC19:58
clarkbcorvus: I've been digging through some cinder third party ci logs to try and identify from specific jobs which may be talking to https://review.o.o/p/ and haven't had much luck, I'm wondering if that config is on the zuul side with the merges and not in the jobs themselves. Is there any way to tell from the external sources if that is the case?20:00
clarkboh found one with GIT_BASE that is https://review.openstack.org/p in d-g so I guess that is at least part of it20:01
clarkbinfra-root https://etherpad.opendev.org/p/E6m-M-3fTLwse2RkrDQL thinking I'll send something like that to service-announce then point third party CI operators to it as I can identify them?20:11
clarkbmordred: corvus ^ you might want to double check I've properly represented what is happening to /p/ (and if disabling it makes any sense)20:11
corvusclarkb: lgtm20:41
openstackgerritSean McGinnis proposed openstack/project-config master: Switch dragonflow ACL to retired  https://review.opendev.org/74131721:01
*** fdegir has quit IRC22:16
*** fdegir has joined #opendev22:16
*** logan- has quit IRC22:17
*** logan- has joined #opendev22:19
*** vapjes has quit IRC22:33
corvusmordred: thanks for the rechecks on 726263; i +3d it22:42
openstackgerritJames E. Blair proposed opendev/system-config master: Revert "Revert "Add Zookeeper TLS support""  https://review.opendev.org/74133522:46
corvusinfra-root: ^ i think the next thing for zk tls is to merge that and then restart all of zuul/nodepool.  we can do that without a a zk cluster restart since it's listening on both ports.  so we can restart all of nodepool first, confirm it's working, then restart all of zuul.22:47
fungionce more around22:47
clarkbchange lgtm22:47
fungilgtm22:48
*** tosky has quit IRC22:48
fungii'm around if folks want to work on restarts today (battling the lawn some more, but still around)22:48
clarkbwe finally sorted out alcohol delivery in this state so I'm getting an order shortly22:49
fungiwow, i still have to go risk my life at the communal booze well22:49
fungicongrats22:50
mordredcorvus: yay!22:50
mordredclarkb: that's wonderful!22:50
clarkbfungi: its a bit weird in that its only local stuff becuse to get normal stuff you still need the liquor store22:50
corvusclarkb: thought you made yer omn?22:50
clarkbbut we have good gin and I'm trying a whiskey that is supposed to be good22:50
clarkbcorvus: I haven't brewed in a long time. I drink way to much when I do that22:51
fungiapparently his backwoods still is on the fritz22:51
fungialso there's the whole going blind from the traces of antifreeze in the old radiator you use for those22:51
corvusspeaking of brewing...22:52
corvuswe got some of this: https://themalamarket.com/collections/all/products/zhongba-360-light-soy-sauce-naturally-brewed-1-year22:52
clarkbfungi: when corvus and I did the whiskey museum thing in dublin they were telling us that the origin of wakes in ireland was from people drinking bad booze. You'd have a wake to be sure they were dead befure putting them in the ground22:52
clarkbcorvus: they add mushrooms, I'm sold22:53
corvusthere is much umami in that soy sauce.  it's quite good... :)22:53
fungii use an unpasteurized soy sauce... looking now to see if i can find a writeup22:55
*** tkajinam has joined #opendev22:55
*** tkajinam has quit IRC22:55
*** xiaolin has quit IRC22:56
fungi"Ohsawa Nama Shoyu, Organic and Aged in 150 Year Cedar Kegs"22:57
*** tkajinam has joined #opendev22:58
fungiwe usually order it by the liter22:58
fungisometimes it gets agitated in shipping and explodes. not so good22:58
*** mlavalle has quit IRC22:59
fungithe yeasts in it will definitely wake back up given the chance23:00
corvusgot it: don't order anything else with it.23:00
*** DSpider has quit IRC23:00
fungiexactly23:04
fungiwe keep it in the refrigerator and it does fairly well, but don't shake it vigorously to redistribute the sediment23:05
fungibut yeah, traditionally brewed and aged, soy sauce is some amazing stuff23:06
openstackgerritMerged opendev/system-config master: Build multi-arch python-base/python-builder  https://review.opendev.org/72626323:19
ianwthe hosts added for backup to vexxhost yesterday with https://review.opendev.org/#/c/740827/2/inventory/service/groups.yaml appear to be working23:37
ianwi'll do something manual for the wiki host23:38
ianwunfortunately, in looking at a new server, bup is not packaged for focal23:40
ianwfunig: bup should close files more aggressively during various operations23:43
ianw  which, for example, should return space to the filesystem sooner23:43
ianw  during midx operations, whether explicit as `bup midx ...` or23:43
ianw  implicit during other operations like `bup save`.  Previously, it23:43
ianw  could leave deleted files open, preventing their space from being23:43
ianw  returned until bup exited."23:43
ianwfungi: ^23:43
*** ysandeep|away is now known as ysandeep23:54

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!